[RFC] Yocto Autobuilder and LAVA Integration

[RFC] Yocto Autobuilder and LAVA Integration

[Anibal Limon]

[-- Attachment #1: Type: text/plain, Size: 2121 bytes --]

Hi,

We know the need to execute OE testimage over real HW not only QEMU,

I'm aware that currently there is an implementation on the Yocto
Autobuilder Helper
, this initial implementation looks pretty well separating parts for
template generation [1] and the script to send jobs to LAVA [2].

There are some limitations.

- Requires that the boards are accessible trough SSH (same network?) by the
Autobuilder, so no distributed LAB testing.
- LAVA doesn't know about test results because the execution is injected
via SSH.

In order to do a distributed Boards testing the Yocto Autobuilder needs to
publish in some place accessible the artifacts (image, kernel, etc) to
flash/boot the board and the test suite expected to execute.

Currently there is a functionality called testexport (not too
used/maintained) that allows you to export the test suite.

I created a simple LAVA test definition that allows run testimage (oe-test
runtime) in my own LAVA LAB, is very simplistic only has a regex to parse
results and uses lava-target-ip and lava-echo-ipv4 to get target and server
IP addresses.

In this way the LAVA server handles all the testing and finally the Yocto
Autobuilder can get/poll an event to know what was the actual result of the
job and the job could be send to different LAVA LAB's.

Some of the tasks, I identified,  (if is accepted)

- Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , move
the EXTRA_PLAIN_CMDS to a class.
-  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
templates across boards.
- Poky/OE: Review/fix test-export or provide other mechanism to export the
test suite.

Cheers,
Anibal

[1]
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava-templates/generate-jobconfig.jinja2
[2]
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava
[3]
https://github.com/alimon/test-definitions/commit/4691b67daca26658b669ac0e79e4f27cbf6ed88d
[4]
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/config-intelqa-x86_64-lava.json#n127

[-- Attachment #2: Type: text/html, Size: 3166 bytes --]

Re: [RFC] Yocto Autobuilder and LAVA Integration

[richard.purdie]

Hi Anibal,

On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> We know the need to execute OE testimage over real HW not only QEMU, 
> 
> I'm aware that currently there is an implementation on the Yocto
> Autobuilder Helper 
> , this initial implementation looks pretty well separating parts for
> template generation [1] and the script to send jobs to LAVA [2].
> 
> There are some limitations.
> 
> - Requires that the boards are accessible trough SSH (same network?)
> by the Autobuilder, so no distributed LAB testing.
> - LAVA doesn't know about test results because the execution is
> injected via SSH.
> 
> In order to do a distributed Boards testing the Yocto Autobuilder
> needs to publish in some place accessible the artifacts (image,
> kernel, etc) to flash/boot the board and the test suite expected to
> execute. 
> 
> Currently there is a functionality called testexport (not too
> used/maintained) that allows you to export the test suite.

I continue to have mixed feelings about testexport. It adds complexity
but I'm not sure its actually worth it.

An alternative would be to specify a set of commit hashes for the
configuration under test (poky or oe-core+bitbake and any other
layers), then have LAVA obtain those pieces and run the tests directly.

Its worth considering that we already now have two difference pieces of
code trying to package up the build system/layers, eSDK and testexport.
Ideally if we had some kind of standardised layer setup/configuration
approach we'd then just have a config file to share, then the tools
could recreate the environment and allow the tests to be run there
without testexport. Layer-setup is itself a harder subject but for
example the layer setup code in autobuilder-helper could easily be
reused as things stand today...

In fact the more I think about it, the more I think we may want to do
it that way...

> I created a simple LAVA test definition that allows run testimage
> (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> get target and server IP addresses.
> 
> In this way the LAVA server handles all the testing and finally the
> Yocto Autobuilder can get/poll an event to know what was the actual
> result of the job and the job could be send to different LAVA LAB's. 

That does sound useful and is likely a way we could end up doing this.
Its probably worth highlighting that we now have a way of summarising
the result of the test in the form of the json file the tests all
generate. Sharing that back to the Yocto autobuilder would give us the
test results we need.

> Some of the tasks, I identified,  (if is accepted)
> 
> - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior
> , move the EXTRA_PLAIN_CMDS to a class.
> -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA
> job templates across boards.
> - Poky/OE: Review/fix test-export or provide other mechanism to
> export the test suite.

I think some of these are also independent of each other and good
things to work on regardless...

I would like to hear feedback from those at Intel using LAVA who
submitted the existing code.

Cheers,

Richard

Re: [RFC] Yocto Autobuilder and LAVA Integration

[Chan, Aaron Chun Yew]

Hi Anibal/RP,

> In order to do a distributed Boards testing the Yocto Autobuilder 
> needs to publish in some place accessible the artifacts (image, 
> kernel, etc) to flash/boot the board and the test suite expected to 
> execute.

[Reply] That is correct, since Linaro have this in place to use https://archive.validation.linaro.org/directories/ and I have look into this as well, we can leverage on this 
              but I am up for any suggestion you might have. So the idea here is that we have a placeholder to store the publish artifacts remotely and deploy using 
              native curl command with token access. Then based on your LAVA job definitions we can instruct LAVA to source the images via https.
              Having said that, the deploy stage in LAVA must have some capabilities to read a token file in the LAVA job defintion and pick up the binaries from public repo (git LFS).

              In order for Board Distributed Tests to happen, there are 2 items in my wish lists

              1. Public hosting of binary repository - with access control 
              2. Ease Handshaking between two(2) different systems CI (e.g. Jenkins/Autobuilder) with LAVA 
                   a. Exchange build property (metadata) - includes hardware info, system info 
                   b. Test reporting results
               
> I created a simple LAVA test definition that allows run testimage 
> (oe-test runtime) in my own LAVA LAB, is very simplistic only has a 
> regex to parse results and uses lava-target-ip and lava-echo-ipv4 to 
> get target and server IP addresses.

[Reply] Although the lava test shell have these capabilities to use lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the way we retrieve the Ipv4
              address is reading the logs from LAVA thru XML-RPC and grep the pattern matching string which contains IP even before the HW get initialize entirely then parse
              IP back to the Yocto Autobuilder. 

              http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs

> Some of the tasks, I identified,  (if is accepted)
> 
> - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , 
> move the EXTRA_PLAIN_CMDS to a class.
> - Poky/OE: Review/fix test-export or provide other mechanism to export 
> the test suite. > - Poky/OE: Review/fix test-export or provide other mechanism to export 
> the test suite.

[Reply] I would like to understand further what is the implementation here and how it addresses the problems that we have today. I believe in the past, Tim has tried
              to enable testexport and transfer the testexport into the DUT but it was not very successful and we found breakage.

> -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job 
> templates across boards.

[Reply] I couldn’t be more supportive on this having a common LAVA job template across boards but I would like to stress this, we don’t exactly know how 
              community will define their own LAVA job definition, therefore what I had in mind as per today is to create a placeholde where LAVA job templates 
              can be define and other boards/community can reuse the same template if it fits their use cases. In general the templates we have today are 
              created to fit into Yocto Project use cases.  

Lastly  there are some works I've done on provisiong QEMU on LAVA sourceing from Yocto Project public releases, I am looking at where we can upstream this
https://github.com/lab-github/yoctoproject-lava-test-shell

Thanks!

Cheers,
Aaron Chan
Open Source Technology Center Intel 

-----Original Message-----
From: richard.purdie@linuxfoundation.org [mailto:richard.purdie@linuxfoundation.org] 
Sent: Thursday, November 8, 2018 6:45 AM
To: Anibal Limon <anibal.limon@linaro.org>; yocto@yoctoproject.org
Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org>; Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com>
Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration

Hi Anibal,

On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> We know the need to execute OE testimage over real HW not only QEMU,
> 
> I'm aware that currently there is an implementation on the Yocto 
> Autobuilder Helper , this initial implementation looks pretty well 
> separating parts for template generation [1] and the script to send 
> jobs to LAVA [2].
> 
> There are some limitations.
> 
> - Requires that the boards are accessible trough SSH (same network?) 
> by the Autobuilder, so no distributed LAB testing.
> - LAVA doesn't know about test results because the execution is 
> injected via SSH.
> 
> In order to do a distributed Boards testing the Yocto Autobuilder 
> needs to publish in some place accessible the artifacts (image, 
> kernel, etc) to flash/boot the board and the test suite expected to 
> execute.
> 
> Currently there is a functionality called testexport (not too
> used/maintained) that allows you to export the test suite.

I continue to have mixed feelings about testexport. It adds complexity but I'm not sure its actually worth it.

An alternative would be to specify a set of commit hashes for the configuration under test (poky or oe-core+bitbake and any other layers), then have LAVA obtain those pieces and run the tests directly.

Its worth considering that we already now have two difference pieces of code trying to package up the build system/layers, eSDK and testexport.
Ideally if we had some kind of standardised layer setup/configuration approach we'd then just have a config file to share, then the tools could recreate the environment and allow the tests to be run there without testexport. Layer-setup is itself a harder subject but for example the layer setup code in autobuilder-helper could easily be reused as things stand today...

In fact the more I think about it, the more I think we may want to do it that way...

> I created a simple LAVA test definition that allows run testimage 
> (oe-test runtime) in my own LAVA LAB, is very simplistic only has a 
> regex to parse results and uses lava-target-ip and lava-echo-ipv4 to 
> get target and server IP addresses.
> 
> In this way the LAVA server handles all the testing and finally the 
> Yocto Autobuilder can get/poll an event to know what was the actual 
> result of the job and the job could be send to different LAVA LAB's.

That does sound useful and is likely a way we could end up doing this.
Its probably worth highlighting that we now have a way of summarising the result of the test in the form of the json file the tests all generate. Sharing that back to the Yocto autobuilder would give us the test results we need.

> Some of the tasks, I identified,  (if is accepted)
> 
> - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , 
> move the EXTRA_PLAIN_CMDS to a class.
> -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job 
> templates across boards.
> - Poky/OE: Review/fix test-export or provide other mechanism to export 
> the test suite.

I think some of these are also independent of each other and good things to work on regardless...

I would like to hear feedback from those at Intel using LAVA who submitted the existing code.

Cheers,

Richard

Re: [RFC] Yocto Autobuilder and LAVA Integration

[Anibal Limon]

[-- Attachment #1: Type: text/plain, Size: 9193 bytes --]

On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew <
aaron.chun.yew.chan@intel.com> wrote:

> Hi Anibal/RP,
>
> > In order to do a distributed Boards testing the Yocto Autobuilder
> > needs to publish in some place accessible the artifacts (image,
> > kernel, etc) to flash/boot the board and the test suite expected to
> > execute.
>
> [Reply] That is correct, since Linaro have this in place to use
> https://archive.validation.linaro.org/directories/ and I have look into
> this as well, we can leverage on this
>               but I am up for any suggestion you might have. So the idea
> here is that we have a placeholder to store the publish artifacts remotely
> and deploy using
>               native curl command with token access. Then based on your
> LAVA job definitions we can instruct LAVA to source the images via https.
>               Having said that, the deploy stage in LAVA must have some
> capabilities to read a token file in the LAVA job defintion and pick up the
> binaries from public repo (git LFS).
>
>               In order for Board Distributed Tests to happen, there are 2
> items in my wish lists
>
>               1. Public hosting of binary repository - with access control
>

For publish the artifacts (Rootfs, Kernel image, Test suite), if there is a
public build a token isn't needed like targeting some boards already
commercialized and can be published anywhere like in
http://downloads.yoctoproject.org.


>               2. Ease Handshaking between two(2) different systems CI
> (e.g. Jenkins/Autobuilder) with LAVA
>                    a. Exchange build property (metadata) - includes
> hardware info, system info
>

You can add meta-data to a LAVA test definition.

                   b. Test reporting results
>

For notify job results LAVA test definition support the notify block in
test jobs or you can poll the API using for both needs a LAVA token.



>
> > I created a simple LAVA test definition that allows run testimage
> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> > get target and server IP addresses.
>
> [Reply] Although the lava test shell have these capabilities to use
> lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the
> way we retrieve the Ipv4
>               address is reading the logs from LAVA thru XML-RPC and grep
> the pattern matching string which contains IP even before the HW get
> initialize entirely then parse
>               IP back to the Yocto Autobuilder.
>
>
> http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs


Yes, that's my idea the Yocto Autobuilder dosen't need to know about
particular network configuration in tha LAVA server for execute the job, in
this way the Yocto Autobuilder can communicate with LAVA server to retrieve
the testing job results, and in a case that needs to debug the board LAVA
support hacking sessions to allow connect to the board outside the LAB.

https://validation.linaro.org/static/docs/v2/hacking-session.html


>
>
> > Some of the tasks, I identified,  (if is accepted)
> >
> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> > move the EXTRA_PLAIN_CMDS to a class.
> > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > the test suite. > - Poky/OE: Review/fix test-export or provide other
> mechanism to export
> > the test suite.
>
> [Reply] I would like to understand further what is the implementation here
> and how it addresses the problems that we have today. I believe in the
> past, Tim has tried
>               to enable testexport and transfer the testexport into the
> DUT but it was not very successful and we found breakage.
>

Agree, The testexport functionality is on not usage so there are some bugs
on it.


>
> > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> > templates across boards.
>
> [Reply] I couldn’t be more supportive on this having a common LAVA job
> template across boards but I would like to stress this, we don’t exactly
> know how
>               community will define their own LAVA job definition,
> therefore what I had in mind as per today is to create a placeholde where
> LAVA job templates
>               can be define and other boards/community can reuse the same
> template if it fits their use cases. In general the templates we have today
> are
>               created to fit into Yocto Project use cases.
>

Agree, I not mean a single template but a manner to add easily new LAVA
templates for boards in Yocto Autobuilder, this involves some base LAVA job
templates and a set of scripts to
generate the final template, like you are doing. For example there are
different ways to deploy a board but the login process is the same for
core-image's (login as root wo passwd).

Cheers,
Anibal


>
> Lastly  there are some works I've done on provisiong QEMU on LAVA
> sourceing from Yocto Project public releases, I am looking at where we can
> upstream this
> https://github.com/lab-github/yoctoproject-lava-test-shell
>
> Thanks!
>
> Cheers,
> Aaron Chan
> Open Source Technology Center Intel
>
> -----Original Message-----
> From: richard.purdie@linuxfoundation.org [mailto:
> richard.purdie@linuxfoundation.org]
> Sent: Thursday, November 8, 2018 6:45 AM
> To: Anibal Limon <anibal.limon@linaro.org>; yocto@yoctoproject.org
> Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org>; Chan, Aaron Chun Yew <
> aaron.chun.yew.chan@intel.com>
> Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
>
> Hi Anibal,
>
> On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> > We know the need to execute OE testimage over real HW not only QEMU,
> >
> > I'm aware that currently there is an implementation on the Yocto
> > Autobuilder Helper , this initial implementation looks pretty well
> > separating parts for template generation [1] and the script to send
> > jobs to LAVA [2].
> >
> > There are some limitations.
> >
> > - Requires that the boards are accessible trough SSH (same network?)
> > by the Autobuilder, so no distributed LAB testing.
> > - LAVA doesn't know about test results because the execution is
> > injected via SSH.
> >
> > In order to do a distributed Boards testing the Yocto Autobuilder
> > needs to publish in some place accessible the artifacts (image,
> > kernel, etc) to flash/boot the board and the test suite expected to
> > execute.
> >
> > Currently there is a functionality called testexport (not too
> > used/maintained) that allows you to export the test suite.
>
> I continue to have mixed feelings about testexport. It adds complexity but
> I'm not sure its actually worth it.
>
> An alternative would be to specify a set of commit hashes for the
> configuration under test (poky or oe-core+bitbake and any other layers),
> then have LAVA obtain those pieces and run the tests directly.
>
> Its worth considering that we already now have two difference pieces of
> code trying to package up the build system/layers, eSDK and testexport.
> Ideally if we had some kind of standardised layer setup/configuration
> approach we'd then just have a config file to share, then the tools could
> recreate the environment and allow the tests to be run there without
> testexport. Layer-setup is itself a harder subject but for example the
> layer setup code in autobuilder-helper could easily be reused as things
> stand today...
>
> In fact the more I think about it, the more I think we may want to do it
> that way...
>
> > I created a simple LAVA test definition that allows run testimage
> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> > get target and server IP addresses.
> >
> > In this way the LAVA server handles all the testing and finally the
> > Yocto Autobuilder can get/poll an event to know what was the actual
> > result of the job and the job could be send to different LAVA LAB's.
>
> That does sound useful and is likely a way we could end up doing this.
> Its probably worth highlighting that we now have a way of summarising the
> result of the test in the form of the json file the tests all generate.
> Sharing that back to the Yocto autobuilder would give us the test results
> we need.
>
> > Some of the tasks, I identified,  (if is accepted)
> >
> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> > move the EXTRA_PLAIN_CMDS to a class.
> > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> > templates across boards.
> > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > the test suite.
>
> I think some of these are also independent of each other and good things
> to work on regardless...
>
> I would like to hear feedback from those at Intel using LAVA who submitted
> the existing code.
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 11961 bytes --]

Re: [RFC] Yocto Autobuilder and LAVA Integration

[Anibal Limon]

[-- Attachment #1: Type: text/plain, Size: 10247 bytes --]

+ Yang Wang

Yang and me have been discussing about this Integration work (bugzilla [1])
and trying to breakdown the tasks needed.

Nico: Yang and me talk about will be if Yocto Project can get a token to
access staging LAVA instance in order to test the integration. [2]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13016
[2] https://staging.validation.linaro.org/

On Fri, 9 Nov 2018 at 14:59, Anibal Limon <anibal.limon@linaro.org> wrote:

>
>
> On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew <
> aaron.chun.yew.chan@intel.com> wrote:
>
>> Hi Anibal/RP,
>>
>> > In order to do a distributed Boards testing the Yocto Autobuilder
>> > needs to publish in some place accessible the artifacts (image,
>> > kernel, etc) to flash/boot the board and the test suite expected to
>> > execute.
>>
>> [Reply] That is correct, since Linaro have this in place to use
>> https://archive.validation.linaro.org/directories/ and I have look into
>> this as well, we can leverage on this
>>               but I am up for any suggestion you might have. So the idea
>> here is that we have a placeholder to store the publish artifacts remotely
>> and deploy using
>>               native curl command with token access. Then based on your
>> LAVA job definitions we can instruct LAVA to source the images via https.
>>               Having said that, the deploy stage in LAVA must have some
>> capabilities to read a token file in the LAVA job defintion and pick up the
>> binaries from public repo (git LFS).
>>
>>               In order for Board Distributed Tests to happen, there are 2
>> items in my wish lists
>>
>>               1. Public hosting of binary repository - with access
>> control
>>
>
> For publish the artifacts (Rootfs, Kernel image, Test suite), if there is
> a public build a token isn't needed like targeting some boards already
> commercialized and can be published anywhere like in
> http://downloads.yoctoproject.org.
>
>
>>               2. Ease Handshaking between two(2) different systems CI
>> (e.g. Jenkins/Autobuilder) with LAVA
>>                    a. Exchange build property (metadata) - includes
>> hardware info, system info
>>
>
> You can add meta-data to a LAVA test definition.
>
>                    b. Test reporting results
>>
>
> For notify job results LAVA test definition support the notify block in
> test jobs or you can poll the API using for both needs a LAVA token.
>
>
>
>>
>> > I created a simple LAVA test definition that allows run testimage
>> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
>> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
>> > get target and server IP addresses.
>>
>> [Reply] Although the lava test shell have these capabilities to use
>> lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the
>> way we retrieve the Ipv4
>>               address is reading the logs from LAVA thru XML-RPC and grep
>> the pattern matching string which contains IP even before the HW get
>> initialize entirely then parse
>>               IP back to the Yocto Autobuilder.
>>
>>
>> http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs
>
>
> Yes, that's my idea the Yocto Autobuilder dosen't need to know about
> particular network configuration in tha LAVA server for execute the job, in
> this way the Yocto Autobuilder can communicate with LAVA server to retrieve
> the testing job results, and in a case that needs to debug the board LAVA
> support hacking sessions to allow connect to the board outside the LAB.
>
> https://validation.linaro.org/static/docs/v2/hacking-session.html
>
>
>>
>>
>> > Some of the tasks, I identified,  (if is accepted)
>> >
>> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
>> > move the EXTRA_PLAIN_CMDS to a class.
>> > - Poky/OE: Review/fix test-export or provide other mechanism to export
>> > the test suite. > - Poky/OE: Review/fix test-export or provide other
>> mechanism to export
>> > the test suite.
>>
>> [Reply] I would like to understand further what is the implementation
>> here and how it addresses the problems that we have today. I believe in the
>> past, Tim has tried
>>               to enable testexport and transfer the testexport into the
>> DUT but it was not very successful and we found breakage.
>>
>
> Agree, The testexport functionality is on not usage so there are some bugs
> on it.
>

Yang commented me that He is using testexport but I agree that isn't not
has full functionality as testimage, so in any case a mechanism to use the
test suite + artifacts is needed, could
be making a copy of full build environment (bitbake + layers + config) and
compress in order be able to execute inside LAVA.


>
>
>>
>> > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
>> > templates across boards.
>>
>> [Reply] I couldn’t be more supportive on this having a common LAVA job
>> template across boards but I would like to stress this, we don’t exactly
>> know how
>>               community will define their own LAVA job definition,
>> therefore what I had in mind as per today is to create a placeholde where
>> LAVA job templates
>>               can be define and other boards/community can reuse the same
>> template if it fits their use cases. In general the templates we have today
>> are
>>               created to fit into Yocto Project use cases.
>>
>
> Agree, I not mean a single template but a manner to add easily new LAVA
> templates for boards in Yocto Autobuilder, this involves some base LAVA job
> templates and a set of scripts to
> generate the final template, like you are doing. For example there are
> different ways to deploy a board but the login process is the same for
> core-image's (login as root wo passwd).
>
> Cheers,
> Anibal
>
>
>>
>> Lastly  there are some works I've done on provisiong QEMU on LAVA
>> sourceing from Yocto Project public releases, I am looking at where we can
>> upstream this
>> https://github.com/lab-github/yoctoproject-lava-test-shell
>>
>> Thanks!
>>
>> Cheers,
>> Aaron Chan
>> Open Source Technology Center Intel
>>
>> -----Original Message-----
>> From: richard.purdie@linuxfoundation.org [mailto:
>> richard.purdie@linuxfoundation.org]
>> Sent: Thursday, November 8, 2018 6:45 AM
>> To: Anibal Limon <anibal.limon@linaro.org>; yocto@yoctoproject.org
>> Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org>; Chan, Aaron Chun Yew
>> <aaron.chun.yew.chan@intel.com>
>> Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
>>
>> Hi Anibal,
>>
>> On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
>> > We know the need to execute OE testimage over real HW not only QEMU,
>> >
>> > I'm aware that currently there is an implementation on the Yocto
>> > Autobuilder Helper , this initial implementation looks pretty well
>> > separating parts for template generation [1] and the script to send
>> > jobs to LAVA [2].
>> >
>> > There are some limitations.
>> >
>> > - Requires that the boards are accessible trough SSH (same network?)
>> > by the Autobuilder, so no distributed LAB testing.
>> > - LAVA doesn't know about test results because the execution is
>> > injected via SSH.
>> >
>> > In order to do a distributed Boards testing the Yocto Autobuilder
>> > needs to publish in some place accessible the artifacts (image,
>> > kernel, etc) to flash/boot the board and the test suite expected to
>> > execute.
>> >
>> > Currently there is a functionality called testexport (not too
>> > used/maintained) that allows you to export the test suite.
>>
>> I continue to have mixed feelings about testexport. It adds complexity
>> but I'm not sure its actually worth it.
>>
>> An alternative would be to specify a set of commit hashes for the
>> configuration under test (poky or oe-core+bitbake and any other layers),
>> then have LAVA obtain those pieces and run the tests directly.
>>
>> Its worth considering that we already now have two difference pieces of
>> code trying to package up the build system/layers, eSDK and testexport.
>> Ideally if we had some kind of standardised layer setup/configuration
>> approach we'd then just have a config file to share, then the tools could
>> recreate the environment and allow the tests to be run there without
>> testexport. Layer-setup is itself a harder subject but for example the
>> layer setup code in autobuilder-helper could easily be reused as things
>> stand today...
>>
>> In fact the more I think about it, the more I think we may want to do it
>> that way...
>>
>> > I created a simple LAVA test definition that allows run testimage
>> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
>> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
>> > get target and server IP addresses.
>> >
>> > In this way the LAVA server handles all the testing and finally the
>> > Yocto Autobuilder can get/poll an event to know what was the actual
>> > result of the job and the job could be send to different LAVA LAB's.
>>
>> That does sound useful and is likely a way we could end up doing this.
>> Its probably worth highlighting that we now have a way of summarising the
>> result of the test in the form of the json file the tests all generate.
>> Sharing that back to the Yocto autobuilder would give us the test results
>> we need.
>>
>> > Some of the tasks, I identified,  (if is accepted)
>> >
>> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
>> > move the EXTRA_PLAIN_CMDS to a class.
>> > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
>> > templates across boards.
>> > - Poky/OE: Review/fix test-export or provide other mechanism to export
>> > the test suite.
>>
>> I think some of these are also independent of each other and good things
>> to work on regardless...
>>
>> I would like to hear feedback from those at Intel using LAVA who
>> submitted the existing code.
>>
>> Cheers,
>>
>> Richard
>>
>>

[-- Attachment #2: Type: text/html, Size: 13715 bytes --]

Re: [RFC] Yocto Autobuilder and LAVA Integration

[Aaron Chan]

[-- Attachment #1: Type: text/plain, Size: 12906 bytes --]

Hi Anibal.

Hope that all is well with you and good to hear from someone from the community.

We are maintaining our own LAVA server/dispatcher and only the administrator can create a user account for you.
With the access, each user can create their own authentication tokens. Steps are:

API (tab) --> Authentication Tokens --> New (+) --> Enter the Description of new token --> Copy the token

You can define the server URL and token into the yoctoabb/config.py like this:

“lava-server”: {
    “server”:  “https://staging.validation.linaro.org/”,
    “token”: <New Generated User Token>
},
“artifactorial”: {
   “server”: “https://archive.validation.linaro.org/artifacts/team/qa/2019/11/24/12/28/”,
   “token”: <New Generated User Token>
}

To summarize on what had just mentioned about a year ago, the concept is that if your hardware is located on a remote site,
we need to have user access to the LAVA server in order to schedule task/jobs and to retrieve the IP addr on the device
which allow host to connect to the device on the network where both servers can communicate with each other.

We had also done publishing the artifacts into Artifactorial (similar to https://archive.validation.linaro.org/directories/),
Artifactorial uses curl command to upload/download artifacts store on the remote server, we can  definitely integrate
a python script using pycurl.
However the setback on Artifactorial is that it creates a timestamp based on /<path>/<year>/<month>/<day>/<hour>/<minute>/<seconds>
which can be tricky at times which automation may require to handle as we do not want to pick up the wrong image and flash into the
hardware (e.g. beaglebone).

On another approach you can bring up your own LAVA server thru Docker - https://hub.docker.com/r/lavasoftware/lava-server
We had also explore this option in the past and its working for us. This way you can have access control and miniture board farm which you
can run tests on, if we do not have the hardware which you require you may still need to have access to staging linaro LAVA server.

Lastly, you may consider to have access to LAVA dispatcher on Linaro end, as “board_info.json” is generated on the hardware booted on Yocto
will contain board information which maybe helpful in the future. The dispatcher holds the rootfs of the image were local results/data are stored.
For do_testimage the results are already handle by the bitbake framework and does not require the effort to retrieve the results.

Cheers,
Aaron
Open Source Technology Center Intel

From: Anibal Limon <anibal.limon@linaro.org>
Sent: Sunday, November 24, 2019 2:40 AM
To: Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; yocto@yoctoproject.org; Nicolas Dechesne <nicolas.dechesne@linaro.org>; Orling, Timothy T <timothy.t.orling@intel.com>; Sangal, Apoorv <apoorv.sangal@intel.com>
Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration

+ Yang Wang

Yang and me have been discussing about this Integration work (bugzilla [1]) and trying to breakdown the tasks needed.

Nico: Yang and me talk about will be if Yocto Project can get a token to access staging LAVA instance in order to test the integration. [2]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13016
[2] https://staging.validation.linaro.org/

On Fri, 9 Nov 2018 at 14:59, Anibal Limon <anibal.limon@linaro.org<mailto:anibal.limon@linaro.org>> wrote:

On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com<mailto:aaron.chun.yew.chan@intel.com>> wrote:
Hi Anibal/RP,

> In order to do a distributed Boards testing the Yocto Autobuilder
> needs to publish in some place accessible the artifacts (image,
> kernel, etc) to flash/boot the board and the test suite expected to
> execute.

[Reply] That is correct, since Linaro have this in place to use https://archive.validation.linaro.org/directories/ and I have look into this as well, we can leverage on this
              but I am up for any suggestion you might have. So the idea here is that we have a placeholder to store the publish artifacts remotely and deploy using
              native curl command with token access. Then based on your LAVA job definitions we can instruct LAVA to source the images via https.
              Having said that, the deploy stage in LAVA must have some capabilities to read a token file in the LAVA job defintion and pick up the binaries from public repo (git LFS).

              In order for Board Distributed Tests to happen, there are 2 items in my wish lists

              1. Public hosting of binary repository - with access control

For publish the artifacts (Rootfs, Kernel image, Test suite), if there is a public build a token isn't needed like targeting some boards already commercialized and can be published anywhere like in http://downloads.yoctoproject.org.

              2. Ease Handshaking between two(2) different systems CI (e.g. Jenkins/Autobuilder) with LAVA
                   a. Exchange build property (metadata) - includes hardware info, system info

You can add meta-data to a LAVA test definition.

                   b. Test reporting results

For notify job results LAVA test definition support the notify block in test jobs or you can poll the API using for both needs a LAVA token.



> I created a simple LAVA test definition that allows run testimage
> (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> get target and server IP addresses.

[Reply] Although the lava test shell have these capabilities to use lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the way we retrieve the Ipv4
              address is reading the logs from LAVA thru XML-RPC and grep the pattern matching string which contains IP even before the HW get initialize entirely then parse
              IP back to the Yocto Autobuilder.

              http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs

Yes, that's my idea the Yocto Autobuilder dosen't need to know about particular network configuration in tha LAVA server for execute the job, in this way the Yocto Autobuilder can communicate with LAVA server to retrieve the testing job results, and in a case that needs to debug the board LAVA support hacking sessions to allow connect to the board outside the LAB.

https://validation.linaro.org/static/docs/v2/hacking-session.html



> Some of the tasks, I identified,  (if is accepted)
>
> - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> move the EXTRA_PLAIN_CMDS to a class.
> - Poky/OE: Review/fix test-export or provide other mechanism to export
> the test suite. > - Poky/OE: Review/fix test-export or provide other mechanism to export
> the test suite.

[Reply] I would like to understand further what is the implementation here and how it addresses the problems that we have today. I believe in the past, Tim has tried
              to enable testexport and transfer the testexport into the DUT but it was not very successful and we found breakage.

Agree, The testexport functionality is on not usage so there are some bugs on it.

Yang commented me that He is using testexport but I agree that isn't not has full functionality as testimage, so in any case a mechanism to use the test suite + artifacts is needed, could
be making a copy of full build environment (bitbake + layers + config) and compress in order be able to execute inside LAVA.



> -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> templates across boards.

[Reply] I couldn’t be more supportive on this having a common LAVA job template across boards but I would like to stress this, we don’t exactly know how
              community will define their own LAVA job definition, therefore what I had in mind as per today is to create a placeholde where LAVA job templates
              can be define and other boards/community can reuse the same template if it fits their use cases. In general the templates we have today are
              created to fit into Yocto Project use cases.

Agree, I not mean a single template but a manner to add easily new LAVA templates for boards in Yocto Autobuilder, this involves some base LAVA job templates and a set of scripts to
generate the final template, like you are doing. For example there are different ways to deploy a board but the login process is the same for core-image's (login as root wo passwd).

Cheers,
Anibal


Lastly  there are some works I've done on provisiong QEMU on LAVA sourceing from Yocto Project public releases, I am looking at where we can upstream this
https://github.com/lab-github/yoctoproject-lava-test-shell

Thanks!

Cheers,
Aaron Chan
Open Source Technology Center Intel

-----Original Message-----
From: richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org> [mailto:richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org>]
Sent: Thursday, November 8, 2018 6:45 AM
To: Anibal Limon <anibal.limon@linaro.org<mailto:anibal.limon@linaro.org>>; yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org<mailto:nicolas.dechesne@linaro.org>>; Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com<mailto:aaron.chun.yew.chan@intel.com>>
Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration

Hi Anibal,

On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> We know the need to execute OE testimage over real HW not only QEMU,
>
> I'm aware that currently there is an implementation on the Yocto
> Autobuilder Helper , this initial implementation looks pretty well
> separating parts for template generation [1] and the script to send
> jobs to LAVA [2].
>
> There are some limitations.
>
> - Requires that the boards are accessible trough SSH (same network?)
> by the Autobuilder, so no distributed LAB testing.
> - LAVA doesn't know about test results because the execution is
> injected via SSH.
>
> In order to do a distributed Boards testing the Yocto Autobuilder
> needs to publish in some place accessible the artifacts (image,
> kernel, etc) to flash/boot the board and the test suite expected to
> execute.
>
> Currently there is a functionality called testexport (not too
> used/maintained) that allows you to export the test suite.

I continue to have mixed feelings about testexport. It adds complexity but I'm not sure its actually worth it.

An alternative would be to specify a set of commit hashes for the configuration under test (poky or oe-core+bitbake and any other layers), then have LAVA obtain those pieces and run the tests directly.

Its worth considering that we already now have two difference pieces of code trying to package up the build system/layers, eSDK and testexport.
Ideally if we had some kind of standardised layer setup/configuration approach we'd then just have a config file to share, then the tools could recreate the environment and allow the tests to be run there without testexport. Layer-setup is itself a harder subject but for example the layer setup code in autobuilder-helper could easily be reused as things stand today...

In fact the more I think about it, the more I think we may want to do it that way...

> I created a simple LAVA test definition that allows run testimage
> (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> get target and server IP addresses.
>
> In this way the LAVA server handles all the testing and finally the
> Yocto Autobuilder can get/poll an event to know what was the actual
> result of the job and the job could be send to different LAVA LAB's.

That does sound useful and is likely a way we could end up doing this.
Its probably worth highlighting that we now have a way of summarising the result of the test in the form of the json file the tests all generate. Sharing that back to the Yocto autobuilder would give us the test results we need.

> Some of the tasks, I identified,  (if is accepted)
>
> - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> move the EXTRA_PLAIN_CMDS to a class.
> -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> templates across boards.
> - Poky/OE: Review/fix test-export or provide other mechanism to export
> the test suite.

I think some of these are also independent of each other and good things to work on regardless...

I would like to hear feedback from those at Intel using LAVA who submitted the existing code.

Cheers,

Richard

[-- Attachment #2: Type: text/html, Size: 24416 bytes --]

Re: [yocto] [RFC] Yocto Autobuilder and LAVA Integration

[Stephen Lawrence]

Hi,

>From: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> On Behalf Of Aaron Chan via Lists.Yoctoproject.Org
>Sent: 25 November 2019 01:49
>To: Anibal Limon <anibal.limon@linaro.org>
>Cc: yocto@lists.yoctoproject.org
>Subject: Re: [yocto] [RFC] Yocto Autobuilder and LAVA Integration

>On another approach you can bring up your own LAVA server thru Docker - https://hub.docker.com/r/lavasoftware/lava-server
>We had also explore this option in the past and its working for us. This way you can have access control and miniture board farm which you
>can run tests on, if we do not have the hardware which you require you may still need to have access to staging linaro LAVA server.

In the case of the Docker route there is another option. KernelCI maintain lava-docker [1] for the creation of Master and Worker instances via Docker. It works and its actively maintained.

[1] https://github.com/kernelci/lava-docker 

Regards

Steve

Re: [RFC] Yocto Autobuilder and LAVA Integration

[Nicolas Dechesne]

hey,


On Mon, Nov 25, 2019 at 2:49 AM Chan, Aaron Chun Yew
<aaron.chun.yew.chan@intel.com> wrote:
>
> Hi Anibal.
>
>
>
> Hope that all is well with you and good to hear from someone from the community.
>
>
>
> We are maintaining our own LAVA server/dispatcher and only the administrator can create a user account for you.
>
> With the access, each user can create their own authentication tokens. Steps are:
>
>
>
> API (tab) à Authentication Tokens à New (+) à Enter the Description of new token à Copy the token
>
>
>
> You can define the server URL and token into the yoctoabb/config.py like this:
>
>
>
> “lava-server”: {
>
>     “server”:  “https://staging.validation.linaro.org/”,
>
>     “token”: <New Generated User Token>
>
> },
>
> “artifactorial”: {
>
>    “server”: “https://archive.validation.linaro.org/artifacts/team/qa/2019/11/24/12/28/”,
>
>    “token”: <New Generated User Token>
>
> }
>
>
>
> To summarize on what had just mentioned about a year ago, the concept is that if your hardware is located on a remote site,
>
> we need to have user access to the LAVA server in order to schedule task/jobs and to retrieve the IP addr on the device
>
> which allow host to connect to the device on the network where both servers can communicate with each other.

Can you please summarize/explain how the device under test is being
used during the testing? The devices we have in our LAVA lab instance
are generally not accessible from the outside world with SSH. I think
I remember that SSH access was required to run the tests, but I am not
sure about the details.

The way we (Linaro) run tests on our devices in our LAVA lab (that
includes all kind of tests, such as kernel only, but also YP/OE based,
Android, ... ) is that the device under test is controlled via the
serial console, not SSH. The test is driven from a LAVA test case
definition. Do you think we can modify the YP ABB to behave in a
similar way?

We can run OE ptest on the OE images that we built, here is an example
of LAVA test case that shows ptest was run:
https://validation.linaro.org/results/1890697/0_linux-ptest

To run ptest, we are using the following LAVA test definition (snippet):
https://github.com/Linaro/test-definitions/blob/master/automated/linux/ptest/ptest.yaml

which in turns use this test execution script:
https://github.com/Linaro/test-definitions/blob/master/automated/linux/ptest/ptest.py

LAVA was essentially designed to be used with a test definition, and I
was hoping we would find a way to integrate and link YP ABB 'output'
with LAVA in this way.

What do you think?

>
>
>
> We had also done publishing the artifacts into Artifactorial (similar to https://archive.validation.linaro.org/directories/),
>
> Artifactorial uses curl command to upload/download artifacts store on the remote server, we can  definitely integrate
>
> a python script using pycurl.
>
> However the setback on Artifactorial is that it creates a timestamp based on /<path>/<year>/<month>/<day>/<hour>/<minute>/<seconds>
>
> which can be tricky at times which automation may require to handle as we do not want to pick up the wrong image and flash into the
>
> hardware (e.g. beaglebone).
>
>
>
> On another approach you can bring up your own LAVA server thru Docker - https://hub.docker.com/r/lavasoftware/lava-server
>
> We had also explore this option in the past and its working for us. This way you can have access control and miniture board farm which you
>
> can run tests on, if we do not have the hardware which you require you may still need to have access to staging linaro LAVA server.
>
>
>
> Lastly, you may consider to have access to LAVA dispatcher on Linaro end, as “board_info.json” is generated on the hardware booted on Yocto
>
> will contain board information which maybe helpful in the future. The dispatcher holds the rootfs of the image were local results/data are stored.
>
> For do_testimage the results are already handle by the bitbake framework and does not require the effort to retrieve the results.
>
>
>
> Cheers,
>
> Aaron
>
> Open Source Technology Center Intel
>
>
>
> From: Anibal Limon <anibal.limon@linaro.org>
> Sent: Sunday, November 24, 2019 2:40 AM
> To: Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; yocto@yoctoproject.org; Nicolas Dechesne <nicolas.dechesne@linaro.org>; Orling, Timothy T <timothy.t.orling@intel.com>; Sangal, Apoorv <apoorv.sangal@intel.com>
> Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
>
>
>
> + Yang Wang
>
>
>
> Yang and me have been discussing about this Integration work (bugzilla [1]) and trying to breakdown the tasks needed.
>
>
>
> Nico: Yang and me talk about will be if Yocto Project can get a token to access staging LAVA instance in order to test the integration. [2]
>
>
>
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13016
>
> [2] https://staging.validation.linaro.org/
>
>
>
> On Fri, 9 Nov 2018 at 14:59, Anibal Limon <anibal.limon@linaro.org> wrote:
>
>
>
> On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com> wrote:
>
> Hi Anibal/RP,
>
> > In order to do a distributed Boards testing the Yocto Autobuilder
> > needs to publish in some place accessible the artifacts (image,
> > kernel, etc) to flash/boot the board and the test suite expected to
> > execute.
>
> [Reply] That is correct, since Linaro have this in place to use https://archive.validation.linaro.org/directories/ and I have look into this as well, we can leverage on this
>               but I am up for any suggestion you might have. So the idea here is that we have a placeholder to store the publish artifacts remotely and deploy using
>               native curl command with token access. Then based on your LAVA job definitions we can instruct LAVA to source the images via https.
>               Having said that, the deploy stage in LAVA must have some capabilities to read a token file in the LAVA job defintion and pick up the binaries from public repo (git LFS).
>
>               In order for Board Distributed Tests to happen, there are 2 items in my wish lists
>
>               1. Public hosting of binary repository - with access control
>
>
>
> For publish the artifacts (Rootfs, Kernel image, Test suite), if there is a public build a token isn't needed like targeting some boards already commercialized and can be published anywhere like in http://downloads.yoctoproject.org.
>
>
>
>               2. Ease Handshaking between two(2) different systems CI (e.g. Jenkins/Autobuilder) with LAVA
>                    a. Exchange build property (metadata) - includes hardware info, system info
>
>
>
> You can add meta-data to a LAVA test definition.
>
>
>
>                    b. Test reporting results
>
>
>
> For notify job results LAVA test definition support the notify block in test jobs or you can poll the API using for both needs a LAVA token.
>
>
>
>
>
>
> > I created a simple LAVA test definition that allows run testimage
> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> > get target and server IP addresses.
>
> [Reply] Although the lava test shell have these capabilities to use lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the way we retrieve the Ipv4
>               address is reading the logs from LAVA thru XML-RPC and grep the pattern matching string which contains IP even before the HW get initialize entirely then parse
>               IP back to the Yocto Autobuilder.
>
>               http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs
>
>
>
> Yes, that's my idea the Yocto Autobuilder dosen't need to know about particular network configuration in tha LAVA server for execute the job, in this way the Yocto Autobuilder can communicate with LAVA server to retrieve the testing job results, and in a case that needs to debug the board LAVA support hacking sessions to allow connect to the board outside the LAB.
>
>
>
> https://validation.linaro.org/static/docs/v2/hacking-session.html
>
>
>
>
>
> > Some of the tasks, I identified,  (if is accepted)
> >
> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> > move the EXTRA_PLAIN_CMDS to a class.
> > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > the test suite. > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > the test suite.
>
> [Reply] I would like to understand further what is the implementation here and how it addresses the problems that we have today. I believe in the past, Tim has tried
>               to enable testexport and transfer the testexport into the DUT but it was not very successful and we found breakage.
>
>
>
> Agree, The testexport functionality is on not usage so there are some bugs on it.
>
>
>
> Yang commented me that He is using testexport but I agree that isn't not has full functionality as testimage, so in any case a mechanism to use the test suite + artifacts is needed, could
>
> be making a copy of full build environment (bitbake + layers + config) and compress in order be able to execute inside LAVA.
>
>
>
>
>
>
> > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> > templates across boards.
>
> [Reply] I couldn’t be more supportive on this having a common LAVA job template across boards but I would like to stress this, we don’t exactly know how
>               community will define their own LAVA job definition, therefore what I had in mind as per today is to create a placeholde where LAVA job templates
>               can be define and other boards/community can reuse the same template if it fits their use cases. In general the templates we have today are
>               created to fit into Yocto Project use cases.
>
>
>
> Agree, I not mean a single template but a manner to add easily new LAVA templates for boards in Yocto Autobuilder, this involves some base LAVA job templates and a set of scripts to
>
> generate the final template, like you are doing. For example there are different ways to deploy a board but the login process is the same for core-image's (login as root wo passwd).
>
>
>
> Cheers,
>
> Anibal
>
>
>
>
> Lastly  there are some works I've done on provisiong QEMU on LAVA sourceing from Yocto Project public releases, I am looking at where we can upstream this
> https://github.com/lab-github/yoctoproject-lava-test-shell
>
> Thanks!
>
> Cheers,
> Aaron Chan
> Open Source Technology Center Intel
>
> -----Original Message-----
> From: richard.purdie@linuxfoundation.org [mailto:richard.purdie@linuxfoundation.org]
> Sent: Thursday, November 8, 2018 6:45 AM
> To: Anibal Limon <anibal.limon@linaro.org>; yocto@yoctoproject.org
> Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org>; Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com>
> Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
>
> Hi Anibal,
>
> On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> > We know the need to execute OE testimage over real HW not only QEMU,
> >
> > I'm aware that currently there is an implementation on the Yocto
> > Autobuilder Helper , this initial implementation looks pretty well
> > separating parts for template generation [1] and the script to send
> > jobs to LAVA [2].
> >
> > There are some limitations.
> >
> > - Requires that the boards are accessible trough SSH (same network?)
> > by the Autobuilder, so no distributed LAB testing.
> > - LAVA doesn't know about test results because the execution is
> > injected via SSH.
> >
> > In order to do a distributed Boards testing the Yocto Autobuilder
> > needs to publish in some place accessible the artifacts (image,
> > kernel, etc) to flash/boot the board and the test suite expected to
> > execute.
> >
> > Currently there is a functionality called testexport (not too
> > used/maintained) that allows you to export the test suite.
>
> I continue to have mixed feelings about testexport. It adds complexity but I'm not sure its actually worth it.
>
> An alternative would be to specify a set of commit hashes for the configuration under test (poky or oe-core+bitbake and any other layers), then have LAVA obtain those pieces and run the tests directly.
>
> Its worth considering that we already now have two difference pieces of code trying to package up the build system/layers, eSDK and testexport.
> Ideally if we had some kind of standardised layer setup/configuration approach we'd then just have a config file to share, then the tools could recreate the environment and allow the tests to be run there without testexport. Layer-setup is itself a harder subject but for example the layer setup code in autobuilder-helper could easily be reused as things stand today...
>
> In fact the more I think about it, the more I think we may want to do it that way...
>
> > I created a simple LAVA test definition that allows run testimage
> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> > get target and server IP addresses.
> >
> > In this way the LAVA server handles all the testing and finally the
> > Yocto Autobuilder can get/poll an event to know what was the actual
> > result of the job and the job could be send to different LAVA LAB's.
>
> That does sound useful and is likely a way we could end up doing this.
> Its probably worth highlighting that we now have a way of summarising the result of the test in the form of the json file the tests all generate. Sharing that back to the Yocto autobuilder would give us the test results we need.
>
> > Some of the tasks, I identified,  (if is accepted)
> >
> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> > move the EXTRA_PLAIN_CMDS to a class.
> > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> > templates across boards.
> > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > the test suite.
>
> I think some of these are also independent of each other and good things to work on regardless...
>
> I would like to hear feedback from those at Intel using LAVA who submitted the existing code.
>
> Cheers,
>
> Richard

Re: [RFC] Yocto Autobuilder and LAVA Integration

[Anibal Limon]

[-- Attachment #1: Type: text/plain, Size: 15978 bytes --]

On Mon, 25 Nov 2019 at 10:23, Nicolas Dechesne <nicolas.dechesne@linaro.org>
wrote:

> hey,
>
>
> On Mon, Nov 25, 2019 at 2:49 AM Chan, Aaron Chun Yew
> <aaron.chun.yew.chan@intel.com> wrote:
> >
> > Hi Anibal.
> >
> >
> >
> > Hope that all is well with you and good to hear from someone from the
> community.
> >
> >
> >
> > We are maintaining our own LAVA server/dispatcher and only the
> administrator can create a user account for you.
> >
> > With the access, each user can create their own authentication tokens.
> Steps are:
> >
> >
> >
> > API (tab) à Authentication Tokens à New (+) à Enter the Description of
> new token à Copy the token
> >
> >
> >
> > You can define the server URL and token into the yoctoabb/config.py like
> this:
> >
> >
> >
> > “lava-server”: {
> >
> >     “server”:  “https://staging.validation.linaro.org/”,
> >
> >     “token”: <New Generated User Token>
> >
> > },
> >
> > “artifactorial”: {
> >
> >    “server”: “
> https://archive.validation.linaro.org/artifacts/team/qa/2019/11/24/12/28/
> ”,
> >
> >    “token”: <New Generated User Token>
> >
> > }
> >
> >
> >
> > To summarize on what had just mentioned about a year ago, the concept is
> that if your hardware is located on a remote site,
> >
> > we need to have user access to the LAVA server in order to schedule
> task/jobs and to retrieve the IP addr on the device
> >
> > which allow host to connect to the device on the network where both
> servers can communicate with each other.
>
> Can you please summarize/explain how the device under test is being
> used during the testing? The devices we have in our LAVA lab instance
> are generally not accessible from the outside world with SSH. I think
> I remember that SSH access was required to run the tests, but I am not
> sure about the details.
>

Aaron is explaining the current implementation in the Yocto Autobuilder it
relays of being in the same network/access with LAVA and the tests are
driven only by OE testimage and right
it requires to have SSH access details.


>
> The way we (Linaro) run tests on our devices in our LAVA lab (that
> includes all kind of tests, such as kernel only, but also YP/OE based,
> Android, ... ) is that the device under test is controlled via the
> serial console, not SSH. The test is driven from a LAVA test case
> definition. Do you think we can modify the YP ABB to behave in a
> similar way?


> We can run OE ptest on the OE images that we built, here is an example
> of LAVA test case that shows ptest was run:
> https://validation.linaro.org/results/1890697/0_linux-ptest
>
> To run ptest, we are using the following LAVA test definition (snippet):
>
> https://github.com/Linaro/test-definitions/blob/master/automated/linux/ptest/ptest.yaml
>
> which in turns use this test execution script:
>
> https://github.com/Linaro/test-definitions/blob/master/automated/linux/ptest/ptest.py
>
> LAVA was essentially designed to be used with a test definition, and I
> was hoping we would find a way to integrate and link YP ABB 'output'
> with LAVA in this way.
>
> What do you think?
>
> >
> >
> >
> > We had also done publishing the artifacts into Artifactorial (similar to
> https://archive.validation.linaro.org/directories/),
> >
> > Artifactorial uses curl command to upload/download artifacts store on
> the remote server, we can  definitely integrate
> >
> > a python script using pycurl.
> >
> > However the setback on Artifactorial is that it creates a timestamp
> based on /<path>/<year>/<month>/<day>/<hour>/<minute>/<seconds>
> >
> > which can be tricky at times which automation may require to handle as
> we do not want to pick up the wrong image and flash into the
> >
> > hardware (e.g. beaglebone).
> >
> >
> >
> > On another approach you can bring up your own LAVA server thru Docker -
> https://hub.docker.com/r/lavasoftware/lava-server
> >
> > We had also explore this option in the past and its working for us. This
> way you can have access control and miniture board farm which you
> >
> > can run tests on, if we do not have the hardware which you require you
> may still need to have access to staging linaro LAVA server.
> >
> >
> >
> > Lastly, you may consider to have access to LAVA dispatcher on Linaro
> end, as “board_info.json” is generated on the hardware booted on Yocto
> >
> > will contain board information which maybe helpful in the future. The
> dispatcher holds the rootfs of the image were local results/data are stored.
> >
> > For do_testimage the results are already handle by the bitbake framework
> and does not require the effort to retrieve the results.
> >
> >
> >
> > Cheers,
> >
> > Aaron
> >
> > Open Source Technology Center Intel
> >
> >
> >
> > From: Anibal Limon <anibal.limon@linaro.org>
> > Sent: Sunday, November 24, 2019 2:40 AM
> > To: Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com>
> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>;
> yocto@yoctoproject.org; Nicolas Dechesne <nicolas.dechesne@linaro.org>;
> Orling, Timothy T <timothy.t.orling@intel.com>; Sangal, Apoorv <
> apoorv.sangal@intel.com>
> > Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
> >
> >
> >
> > + Yang Wang
> >
> >
> >
> > Yang and me have been discussing about this Integration work (bugzilla
> [1]) and trying to breakdown the tasks needed.
> >
> >
> >
> > Nico: Yang and me talk about will be if Yocto Project can get a token to
> access staging LAVA instance in order to test the integration. [2]
> >
> >
> >
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13016
> >
> > [2] https://staging.validation.linaro.org/
> >
> >
> >
> > On Fri, 9 Nov 2018 at 14:59, Anibal Limon <anibal.limon@linaro.org>
> wrote:
> >
> >
> >
> > On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew <
> aaron.chun.yew.chan@intel.com> wrote:
> >
> > Hi Anibal/RP,
> >
> > > In order to do a distributed Boards testing the Yocto Autobuilder
> > > needs to publish in some place accessible the artifacts (image,
> > > kernel, etc) to flash/boot the board and the test suite expected to
> > > execute.
> >
> > [Reply] That is correct, since Linaro have this in place to use
> https://archive.validation.linaro.org/directories/ and I have look into
> this as well, we can leverage on this
> >               but I am up for any suggestion you might have. So the idea
> here is that we have a placeholder to store the publish artifacts remotely
> and deploy using
> >               native curl command with token access. Then based on your
> LAVA job definitions we can instruct LAVA to source the images via https.
> >               Having said that, the deploy stage in LAVA must have some
> capabilities to read a token file in the LAVA job defintion and pick up the
> binaries from public repo (git LFS).
> >
> >               In order for Board Distributed Tests to happen, there are
> 2 items in my wish lists
> >
> >               1. Public hosting of binary repository - with access
> control
> >
> >
> >
> > For publish the artifacts (Rootfs, Kernel image, Test suite), if there
> is a public build a token isn't needed like targeting some boards already
> commercialized and can be published anywhere like in
> http://downloads.yoctoproject.org.
> >
> >
> >
> >               2. Ease Handshaking between two(2) different systems CI
> (e.g. Jenkins/Autobuilder) with LAVA
> >                    a. Exchange build property (metadata) - includes
> hardware info, system info
> >
> >
> >
> > You can add meta-data to a LAVA test definition.
> >
> >
> >
> >                    b. Test reporting results
> >
> >
> >
> > For notify job results LAVA test definition support the notify block in
> test jobs or you can poll the API using for both needs a LAVA token.
> >
> >
> >
> >
> >
> >
> > > I created a simple LAVA test definition that allows run testimage
> > > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> > > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> > > get target and server IP addresses.
> >
> > [Reply] Although the lava test shell have these capabilities to use
> lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the
> way we retrieve the Ipv4
> >               address is reading the logs from LAVA thru XML-RPC and
> grep the pattern matching string which contains IP even before the HW get
> initialize entirely then parse
> >               IP back to the Yocto Autobuilder.
> >
> >
> http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs
> >
> >
> >
> > Yes, that's my idea the Yocto Autobuilder dosen't need to know about
> particular network configuration in tha LAVA server for execute the job, in
> this way the Yocto Autobuilder can communicate with LAVA server to retrieve
> the testing job results, and in a case that needs to debug the board LAVA
> support hacking sessions to allow connect to the board outside the LAB.
> >
> >
> >
> > https://validation.linaro.org/static/docs/v2/hacking-session.html
> >
> >
> >
> >
> >
> > > Some of the tasks, I identified,  (if is accepted)
> > >
> > > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> > > move the EXTRA_PLAIN_CMDS to a class.
> > > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > > the test suite. > - Poky/OE: Review/fix test-export or provide other
> mechanism to export
> > > the test suite.
> >
> > [Reply] I would like to understand further what is the implementation
> here and how it addresses the problems that we have today. I believe in the
> past, Tim has tried
> >               to enable testexport and transfer the testexport into the
> DUT but it was not very successful and we found breakage.
> >
> >
> >
> > Agree, The testexport functionality is on not usage so there are some
> bugs on it.
> >
> >
> >
> > Yang commented me that He is using testexport but I agree that isn't not
> has full functionality as testimage, so in any case a mechanism to use the
> test suite + artifacts is needed, could
> >
> > be making a copy of full build environment (bitbake + layers + config)
> and compress in order be able to execute inside LAVA.
> >
> >
> >
> >
> >
> >
> > > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> > > templates across boards.
> >
> > [Reply] I couldn’t be more supportive on this having a common LAVA job
> template across boards but I would like to stress this, we don’t exactly
> know how
> >               community will define their own LAVA job definition,
> therefore what I had in mind as per today is to create a placeholde where
> LAVA job templates
> >               can be define and other boards/community can reuse the
> same template if it fits their use cases. In general the templates we have
> today are
> >               created to fit into Yocto Project use cases.
> >
> >
> >
> > Agree, I not mean a single template but a manner to add easily new LAVA
> templates for boards in Yocto Autobuilder, this involves some base LAVA job
> templates and a set of scripts to
> >
> > generate the final template, like you are doing. For example there are
> different ways to deploy a board but the login process is the same for
> core-image's (login as root wo passwd).
> >
> >
> >
> > Cheers,
> >
> > Anibal
> >
> >
> >
> >
> > Lastly  there are some works I've done on provisiong QEMU on LAVA
> sourceing from Yocto Project public releases, I am looking at where we can
> upstream this
> > https://github.com/lab-github/yoctoproject-lava-test-shell
> >
> > Thanks!
> >
> > Cheers,
> > Aaron Chan
> > Open Source Technology Center Intel
> >
> > -----Original Message-----
> > From: richard.purdie@linuxfoundation.org [mailto:
> richard.purdie@linuxfoundation.org]
> > Sent: Thursday, November 8, 2018 6:45 AM
> > To: Anibal Limon <anibal.limon@linaro.org>; yocto@yoctoproject.org
> > Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org>; Chan, Aaron Chun
> Yew <aaron.chun.yew.chan@intel.com>
> > Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
> >
> > Hi Anibal,
> >
> > On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> > > We know the need to execute OE testimage over real HW not only QEMU,
> > >
> > > I'm aware that currently there is an implementation on the Yocto
> > > Autobuilder Helper , this initial implementation looks pretty well
> > > separating parts for template generation [1] and the script to send
> > > jobs to LAVA [2].
> > >
> > > There are some limitations.
> > >
> > > - Requires that the boards are accessible trough SSH (same network?)
> > > by the Autobuilder, so no distributed LAB testing.
> > > - LAVA doesn't know about test results because the execution is
> > > injected via SSH.
> > >
> > > In order to do a distributed Boards testing the Yocto Autobuilder
> > > needs to publish in some place accessible the artifacts (image,
> > > kernel, etc) to flash/boot the board and the test suite expected to
> > > execute.
> > >
> > > Currently there is a functionality called testexport (not too
> > > used/maintained) that allows you to export the test suite.
> >
> > I continue to have mixed feelings about testexport. It adds complexity
> but I'm not sure its actually worth it.
> >
> > An alternative would be to specify a set of commit hashes for the
> configuration under test (poky or oe-core+bitbake and any other layers),
> then have LAVA obtain those pieces and run the tests directly.
> >
> > Its worth considering that we already now have two difference pieces of
> code trying to package up the build system/layers, eSDK and testexport.
> > Ideally if we had some kind of standardised layer setup/configuration
> approach we'd then just have a config file to share, then the tools could
> recreate the environment and allow the tests to be run there without
> testexport. Layer-setup is itself a harder subject but for example the
> layer setup code in autobuilder-helper could easily be reused as things
> stand today...
> >
> > In fact the more I think about it, the more I think we may want to do it
> that way...
> >
> > > I created a simple LAVA test definition that allows run testimage
> > > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> > > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> > > get target and server IP addresses.
> > >
> > > In this way the LAVA server handles all the testing and finally the
> > > Yocto Autobuilder can get/poll an event to know what was the actual
> > > result of the job and the job could be send to different LAVA LAB's.
> >
> > That does sound useful and is likely a way we could end up doing this.
> > Its probably worth highlighting that we now have a way of summarising
> the result of the test in the form of the json file the tests all generate.
> Sharing that back to the Yocto autobuilder would give us the test results
> we need.
> >
> > > Some of the tasks, I identified,  (if is accepted)
> > >
> > > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> > > move the EXTRA_PLAIN_CMDS to a class.
> > > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> > > templates across boards.
> > > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > > the test suite.
> >
> > I think some of these are also independent of each other and good things
> to work on regardless...
> >
> > I would like to hear feedback from those at Intel using LAVA who
> submitted the existing code.
> >
> > Cheers,
> >
> > Richard
>

[-- Attachment #2: Type: text/html, Size: 20992 bytes --]

Re: [RFC] Yocto Autobuilder and LAVA Integration

[Aaron Chan]

[-- Attachment #1: Type: text/plain, Size: 18875 bytes --]

Hi Nicholas

From: Anibal Limon <anibal.limon@linaro.org>
Sent: Tuesday, November 26, 2019 2:48 AM
To: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Cc: Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com>; Richard Purdie <richard.purdie@linuxfoundation.org>; yocto@yoctoproject.org; Orling, Timothy T <timothy.t.orling@intel.com>; Sangal, Apoorv <apoorv.sangal@intel.com>
Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration



On Mon, 25 Nov 2019 at 10:23, Nicolas Dechesne <nicolas.dechesne@linaro.org<mailto:nicolas.dechesne@linaro.org>> wrote:
hey,


On Mon, Nov 25, 2019 at 2:49 AM Chan, Aaron Chun Yew
<aaron.chun.yew.chan@intel.com<mailto:aaron.chun.yew.chan@intel.com>> wrote:
>
> Hi Anibal.
>
>
>
> Hope that all is well with you and good to hear from someone from the community.
>
>
>
> We are maintaining our own LAVA server/dispatcher and only the administrator can create a user account for you.
>
> With the access, each user can create their own authentication tokens. Steps are:
>
>
>
> API (tab) à Authentication Tokens à New (+) à Enter the Description of new token à Copy the token
>
>
>
> You can define the server URL and token into the yoctoabb/config.py like this:
>
>
>
> “lava-server”: {
>
>     “server”:  “https://staging.validation.linaro.org/”,
>
>     “token”: <New Generated User Token>
>
> },
>
> “artifactorial”: {
>
>    “server”: “https://archive.validation.linaro.org/artifacts/team/qa/2019/11/24/12/28/”,
>
>    “token”: <New Generated User Token>
>
> }
>
>
>
> To summarize on what had just mentioned about a year ago, the concept is that if your hardware is located on a remote site,
>
> we need to have user access to the LAVA server in order to schedule task/jobs and to retrieve the IP addr on the device
>
> which allow host to connect to the device on the network where both servers can communicate with each other.

Can you please summarize/explain how the device under test is being
used during the testing? The devices we have in our LAVA lab instance
are generally not accessible from the outside world with SSH. I think
I remember that SSH access was required to run the tests, but I am not
sure about the details.

Aaron is explaining the current implementation in the Yocto Autobuilder it relays of being in the same network/access with LAVA and the tests are driven only by OE testimage and right
it requires to have SSH access details.

[Reply: Aaron] As mentioned by Anibal the Yocto Autobuilder and LAVA has to be on a join network in order for host-client connection to happen. I understand these raises a concern on
security having public to access LAVA dispatchers on its overlays.

The way we (Linaro) run tests on our devices in our LAVA lab (that
includes all kind of tests, such as kernel only, but also YP/OE based,
Android, ... ) is that the device under test is controlled via the
serial console, not SSH. The test is driven from a LAVA test case
definition. Do you think we can modify the YP ABB to behave in a
similar way?

We can run OE ptest on the OE images that we built, here is an example
of LAVA test case that shows ptest was run:
https://validation.linaro.org/results/1890697/0_linux-ptest

To run ptest, we are using the following LAVA test definition (snippet):
https://github.com/Linaro/test-definitions/blob/master/automated/linux/ptest/ptest.yaml

which in turns use this test execution script:
https://github.com/Linaro/test-definitions/blob/master/automated/linux/ptest/ptest.py


[Reply: Aaron] So, the YP ABB does not store test cases, it is a CI/CD build system which orchestates and pull the bitbake, layers, helper scripts to build BSP for target hardware.
                          The test cases are pull into a workspace as part of CI process. Today we store these test cases on ref poky as part of oeqa where Yocto community can contribute to it.

LAVA was essentially designed to be used with a test definition, and I
was hoping we would find a way to integrate and link YP ABB 'output'
with LAVA in this way.

What do you think?

[Reply: Aaron] We had review this option in the past and we talked about it with RP.
The way I look at this as test definitions either reside with the oeqa framework or to migrate these definitions into LAVA test definition (e.g. ptest above).

However, there are several concerns is left unanswered for:

  1.  even if we successfully port over the test definition from oeqa into LAVA test definition as form YAML config, the oeqa code will not be exercise since is couple with poky framework would be a concern.
  2.  we relied on resulttool reporting tool to generate the automated test cases result which is stored on the host machine during execution. These results will be push into Git for archiving.
  3.  There are some community test suites (e.g. Kernel selftest, LTP, benchmarking) were results are generated in the target hardware required to be retrieve, analyzed and process in a human readable format (not display in serial console).
  4.  If a target hardware run tests and for some reason hung up/crashes, the test which we run for days could be lost and we ended up rerunning those tests as the previous test results cache into the rootfs were discarded.

Is there a way we can solve this ?

  1.  There are test cases which exercise SSH and the ability to perform package update hosting on a local server.
  2.  Others

The way I see it there are (2) approaches and previous Yocto Autobuilder PoC was based on (1)


  1.  simpleremote – host machine/client connection where client is target hardware and has dependency on the host. In order for the do_testimage to happen, the host machine must be able to

know where it connects to by identifying the hardware IP assigned to it and its able to connect to it of course. 😊

  1.  testexport – the concept is same as LAVA test definition where the test case content are exported and run independently on the target hardware. There are no dependency on the host side.

Results/logs are generated on the target hardware and needs to be retrieve or push to a temporarily storage facility (artifactorial, git, etc).

Let me know what you think.

Cheers,
Aaron

>
>
>
> We had also done publishing the artifacts into Artifactorial (similar to https://archive.validation.linaro.org/directories/),
>
> Artifactorial uses curl command to upload/download artifacts store on the remote server, we can  definitely integrate
>
> a python script using pycurl.
>
> However the setback on Artifactorial is that it creates a timestamp based on /<path>/<year>/<month>/<day>/<hour>/<minute>/<seconds>
>
> which can be tricky at times which automation may require to handle as we do not want to pick up the wrong image and flash into the
>
> hardware (e.g. beaglebone).
>
>
>
> On another approach you can bring up your own LAVA server thru Docker - https://hub.docker.com/r/lavasoftware/lava-server
>
> We had also explore this option in the past and its working for us. This way you can have access control and miniture board farm which you
>
> can run tests on, if we do not have the hardware which you require you may still need to have access to staging linaro LAVA server.
>
>
>
> Lastly, you may consider to have access to LAVA dispatcher on Linaro end, as “board_info.json” is generated on the hardware booted on Yocto
>
> will contain board information which maybe helpful in the future. The dispatcher holds the rootfs of the image were local results/data are stored.
>
> For do_testimage the results are already handle by the bitbake framework and does not require the effort to retrieve the results.
>
>
>
> Cheers,
>
> Aaron
>
> Open Source Technology Center Intel
>
>
>
> From: Anibal Limon <anibal.limon@linaro.org<mailto:anibal.limon@linaro.org>>
> Sent: Sunday, November 24, 2019 2:40 AM
> To: Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com<mailto:aaron.chun.yew.chan@intel.com>>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org>>; yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>; Nicolas Dechesne <nicolas.dechesne@linaro.org<mailto:nicolas.dechesne@linaro.org>>; Orling, Timothy T <timothy.t.orling@intel.com<mailto:timothy.t.orling@intel.com>>; Sangal, Apoorv <apoorv.sangal@intel.com<mailto:apoorv.sangal@intel.com>>
> Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
>
>
>
> + Yang Wang
>
>
>
> Yang and me have been discussing about this Integration work (bugzilla [1]) and trying to breakdown the tasks needed.
>
>
>
> Nico: Yang and me talk about will be if Yocto Project can get a token to access staging LAVA instance in order to test the integration. [2]
>
>
>
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13016
>
> [2] https://staging.validation.linaro.org/
>
>
>
> On Fri, 9 Nov 2018 at 14:59, Anibal Limon <anibal.limon@linaro.org<mailto:anibal.limon@linaro.org>> wrote:
>
>
>
> On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com<mailto:aaron.chun.yew.chan@intel.com>> wrote:
>
> Hi Anibal/RP,
>
> > In order to do a distributed Boards testing the Yocto Autobuilder
> > needs to publish in some place accessible the artifacts (image,
> > kernel, etc) to flash/boot the board and the test suite expected to
> > execute.
>
> [Reply] That is correct, since Linaro have this in place to use https://archive.validation.linaro.org/directories/ and I have look into this as well, we can leverage on this
>               but I am up for any suggestion you might have. So the idea here is that we have a placeholder to store the publish artifacts remotely and deploy using
>               native curl command with token access. Then based on your LAVA job definitions we can instruct LAVA to source the images via https.
>               Having said that, the deploy stage in LAVA must have some capabilities to read a token file in the LAVA job defintion and pick up the binaries from public repo (git LFS).
>
>               In order for Board Distributed Tests to happen, there are 2 items in my wish lists
>
>               1. Public hosting of binary repository - with access control
>
>
>
> For publish the artifacts (Rootfs, Kernel image, Test suite), if there is a public build a token isn't needed like targeting some boards already commercialized and can be published anywhere like in http://downloads.yoctoproject.org.
>
>
>
>               2. Ease Handshaking between two(2) different systems CI (e.g. Jenkins/Autobuilder) with LAVA
>                    a. Exchange build property (metadata) - includes hardware info, system info
>
>
>
> You can add meta-data to a LAVA test definition.
>
>
>
>                    b. Test reporting results
>
>
>
> For notify job results LAVA test definition support the notify block in test jobs or you can poll the API using for both needs a LAVA token.
>
>
>
>
>
>
> > I created a simple LAVA test definition that allows run testimage
> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> > get target and server IP addresses.
>
> [Reply] Although the lava test shell have these capabilities to use lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the way we retrieve the Ipv4
>               address is reading the logs from LAVA thru XML-RPC and grep the pattern matching string which contains IP even before the HW get initialize entirely then parse
>               IP back to the Yocto Autobuilder.
>
>               http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs
>
>
>
> Yes, that's my idea the Yocto Autobuilder dosen't need to know about particular network configuration in tha LAVA server for execute the job, in this way the Yocto Autobuilder can communicate with LAVA server to retrieve the testing job results, and in a case that needs to debug the board LAVA support hacking sessions to allow connect to the board outside the LAB.
>
>
>
> https://validation.linaro.org/static/docs/v2/hacking-session.html
>
>
>
>
>
> > Some of the tasks, I identified,  (if is accepted)
> >
> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> > move the EXTRA_PLAIN_CMDS to a class.
> > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > the test suite. > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > the test suite.
>
> [Reply] I would like to understand further what is the implementation here and how it addresses the problems that we have today. I believe in the past, Tim has tried
>               to enable testexport and transfer the testexport into the DUT but it was not very successful and we found breakage.
>
>
>
> Agree, The testexport functionality is on not usage so there are some bugs on it.
>
>
>
> Yang commented me that He is using testexport but I agree that isn't not has full functionality as testimage, so in any case a mechanism to use the test suite + artifacts is needed, could
>
> be making a copy of full build environment (bitbake + layers + config) and compress in order be able to execute inside LAVA.
>
>
>
>
>
>
> > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> > templates across boards.
>
> [Reply] I couldn’t be more supportive on this having a common LAVA job template across boards but I would like to stress this, we don’t exactly know how
>               community will define their own LAVA job definition, therefore what I had in mind as per today is to create a placeholde where LAVA job templates
>               can be define and other boards/community can reuse the same template if it fits their use cases. In general the templates we have today are
>               created to fit into Yocto Project use cases.
>
>
>
> Agree, I not mean a single template but a manner to add easily new LAVA templates for boards in Yocto Autobuilder, this involves some base LAVA job templates and a set of scripts to
>
> generate the final template, like you are doing. For example there are different ways to deploy a board but the login process is the same for core-image's (login as root wo passwd).
>
>
>
> Cheers,
>
> Anibal
>
>
>
>
> Lastly  there are some works I've done on provisiong QEMU on LAVA sourceing from Yocto Project public releases, I am looking at where we can upstream this
> https://github.com/lab-github/yoctoproject-lava-test-shell
>
> Thanks!
>
> Cheers,
> Aaron Chan
> Open Source Technology Center Intel
>
> -----Original Message-----
> From: richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org> [mailto:richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org>]
> Sent: Thursday, November 8, 2018 6:45 AM
> To: Anibal Limon <anibal.limon@linaro.org<mailto:anibal.limon@linaro.org>>; yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
> Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org<mailto:nicolas.dechesne@linaro.org>>; Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com<mailto:aaron.chun.yew.chan@intel.com>>
> Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
>
> Hi Anibal,
>
> On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> > We know the need to execute OE testimage over real HW not only QEMU,
> >
> > I'm aware that currently there is an implementation on the Yocto
> > Autobuilder Helper , this initial implementation looks pretty well
> > separating parts for template generation [1] and the script to send
> > jobs to LAVA [2].
> >
> > There are some limitations.
> >
> > - Requires that the boards are accessible trough SSH (same network?)
> > by the Autobuilder, so no distributed LAB testing.
> > - LAVA doesn't know about test results because the execution is
> > injected via SSH.
> >
> > In order to do a distributed Boards testing the Yocto Autobuilder
> > needs to publish in some place accessible the artifacts (image,
> > kernel, etc) to flash/boot the board and the test suite expected to
> > execute.
> >
> > Currently there is a functionality called testexport (not too
> > used/maintained) that allows you to export the test suite.
>
> I continue to have mixed feelings about testexport. It adds complexity but I'm not sure its actually worth it.
>
> An alternative would be to specify a set of commit hashes for the configuration under test (poky or oe-core+bitbake and any other layers), then have LAVA obtain those pieces and run the tests directly.
>
> Its worth considering that we already now have two difference pieces of code trying to package up the build system/layers, eSDK and testexport.
> Ideally if we had some kind of standardised layer setup/configuration approach we'd then just have a config file to share, then the tools could recreate the environment and allow the tests to be run there without testexport. Layer-setup is itself a harder subject but for example the layer setup code in autobuilder-helper could easily be reused as things stand today...
>
> In fact the more I think about it, the more I think we may want to do it that way...
>
> > I created a simple LAVA test definition that allows run testimage
> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> > get target and server IP addresses.
> >
> > In this way the LAVA server handles all the testing and finally the
> > Yocto Autobuilder can get/poll an event to know what was the actual
> > result of the job and the job could be send to different LAVA LAB's.
>
> That does sound useful and is likely a way we could end up doing this.
> Its probably worth highlighting that we now have a way of summarising the result of the test in the form of the json file the tests all generate. Sharing that back to the Yocto autobuilder would give us the test results we need.
>
> > Some of the tasks, I identified,  (if is accepted)
> >
> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> > move the EXTRA_PLAIN_CMDS to a class.
> > -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> > templates across boards.
> > - Poky/OE: Review/fix test-export or provide other mechanism to export
> > the test suite.
>
> I think some of these are also independent of each other and good things to work on regardless...
>
> I would like to hear feedback from those at Intel using LAVA who submitted the existing code.
>
> Cheers,
>
> Richard

[-- Attachment #2: Type: text/html, Size: 34917 bytes --]