Building and deploying test binaries

Building and deploying test binaries

[Connor Imes]

[-- Attachment #1: Type: text/plain, Size: 1805 bytes --]

Hi all,

We have some applications that are meant exclusively for testing, and so do not want to include them by default in our image’s rootfs.  It also does not seem appropriate to build a separate image for testing.  After much searching, it seems like the JSON-based approach described here [1] is appropriate to automatically load/remove the binaries at runtime for use during testing.  I’ve written some recipes to build our test binaries, then specified their packages as dependencies using JSON files for our Python oeqa runtime test cases.  If I build the recipes with bitbake, the tests then execute successfully when I run `bitbake <image> -c testimage`.

[1] https://www.yoctoproject.org/docs/2.6/dev-manual/dev-manual.html#installing-packages-in-the-dut-without-the-package-manager

The problem is that we currently have to specify each of these recipes to bitbake so they can built prior to testing, thus requiring a priori knowledge of which recipes our tests depend on.  This manual dependency management is error-prone and won’t scale well.  Is there a way for bitbake to discover these dependencies automatically prior to running `testimage` (which subsequently fails if the dependencies specified in the JSON files aren’t already built)?  Ideally we should be able to manage these recipes similarly to others used to build and populate the rootfs - fetch sources while we’re online, then build offline, then run the tests anytime thereafter.  Something like:
  bitbake <image> -c testimage --runall="fetch"  # download sources (currently fetches some things, but not JSON-specified recipes?)
  bitbake <image> -c testimage --runall="build"  # build/install/etc all test-related recipes
  bitbake <image> -c testimage  # do the actual runtime test

Thanks,
-Connor

[-- Attachment #2: Type: text/html, Size: 4266 bytes --]

Re: Building and deploying test binaries

[Alexander Kanavin]

I'd say a special image for testing is *exactly* what you need. You
might take the existing image recipe, and include it wholesale into
your testing image recipe, then add additional recipes as needed. Then
the testing image is guaranteed to be an extension of the plain image.
Example:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb

Generally, creating various types of images serving various purposes
is not only ok, but is encouraged - YP isn't able to carry this
message across to users very well :-/

Alex

On Fri, 8 Mar 2019 at 21:36, Connor Imes <cimes@isi.edu> wrote:
>
> Hi all,
>
>
>
> We have some applications that are meant exclusively for testing, and so do not want to include them by default in our image’s rootfs.  It also does not seem appropriate to build a separate image for testing.  After much searching, it seems like the JSON-based approach described here [1] is appropriate to automatically load/remove the binaries at runtime for use during testing.  I’ve written some recipes to build our test binaries, then specified their packages as dependencies using JSON files for our Python oeqa runtime test cases.  If I build the recipes with bitbake, the tests then execute successfully when I run `bitbake <image> -c testimage`.
>
>
>
> [1] https://www.yoctoproject.org/docs/2.6/dev-manual/dev-manual.html#installing-packages-in-the-dut-without-the-package-manager
>
>
>
> The problem is that we currently have to specify each of these recipes to bitbake so they can built prior to testing, thus requiring a priori knowledge of which recipes our tests depend on.  This manual dependency management is error-prone and won’t scale well.  Is there a way for bitbake to discover these dependencies automatically prior to running `testimage` (which subsequently fails if the dependencies specified in the JSON files aren’t already built)?  Ideally we should be able to manage these recipes similarly to others used to build and populate the rootfs - fetch sources while we’re online, then build offline, then run the tests anytime thereafter.  Something like:
>
>   bitbake <image> -c testimage --runall="fetch"  # download sources (currently fetches some things, but not JSON-specified recipes?)
>
>   bitbake <image> -c testimage --runall="build"  # build/install/etc all test-related recipes
>
>   bitbake <image> -c testimage  # do the actual runtime test
>
>
>
> Thanks,
>
> -Connor
>
> --
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: Building and deploying test binaries

[Connor Imes]

Thanks for your response, Alex.  I can certainly see how creating a separate image is straightforward and useful for many teams - it fits very nicely into the layered approach that the OpenEmbedded framework offers.  I also agree that it has not been communicated particularly well.  There doesn't seem to be much documentation on our use case - where tests rely on cross-compiled applications not intended for the final rootfs.  The existence of the JSON-based approach seems to indicate that others have encountered similar requirements.  So my original question still remains, which is to find out if bitbake (or another tool) can automatically manage these JSON-specified test dependencies as part of the build lifecycle.

I'm still exploring options, but there are some reasons to prefer not having a separate test image.  If you're curious, some thoughts follow...

Probably first and foremost is that using a separate image means you're not directly testing the artifact you really care about - the original image on which the test image is based.  Trusting that the test results from the test image apply to your original image is predicated on the assumption that the test image is an exact replica of the original, modulo the addition of non-disruptive test recipes.  That is to say, you must assume that the test recipes don't conflict with each other or modify the system behavior in unexpected ways.  Additionally, while it is conceptually straightforward to just check for the addition of recipes (ignoring unexpected side effects), it could still be a slippery slope - it's easy to start making other changes, as the example you present actually does  (e.g., changing the overhead factor and assuming more memory).  We could debate the impacts of such changes on a case-by-case basis, but ultimately every change moves you further away from the baseline, adding risk.

For us, having to manage multiple images also makes things more challenging, in that the Yocto artifacts we produce are only a part of our embedded system.  As such, we have other scripts and components that would then need to account for multiple images, depending on the scenario.  Certainly this can be done, but for simplicity, it's easier not to require that, esp. when there's only one final rootfs needed from our poky configuration.

Cheers,
-Connor

On 3/8/19, 2:26 PM, "Alexander Kanavin" <alex.kanavin@gmail.com> wrote:

    I'd say a special image for testing is *exactly* what you need. You
    might take the existing image recipe, and include it wholesale into
    your testing image recipe, then add additional recipes as needed. Then
    the testing image is guaranteed to be an extension of the plain image.
    Example:
    
    http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb
    
    Generally, creating various types of images serving various purposes
    is not only ok, but is encouraged - YP isn't able to carry this
    message across to users very well :-/
    
    Alex
    
    On Fri, 8 Mar 2019 at 21:36, Connor Imes <cimes@isi.edu> wrote:
    >
    > Hi all,
    >
    >
    >
    > We have some applications that are meant exclusively for testing, and so do not want to include them by default in our image’s rootfs.  It also does not seem appropriate to build a separate image for testing.  After much searching, it seems like the JSON-based approach described here [1] is appropriate to automatically load/remove the binaries at runtime for use during testing.  I’ve written some recipes to build our test binaries, then specified their packages as dependencies using JSON files for our Python oeqa runtime test cases.  If I build the recipes with bitbake, the tests then execute successfully when I run `bitbake <image> -c testimage`.
    >
    >
    >
    > [1] https://www.yoctoproject.org/docs/2.6/dev-manual/dev-manual.html#installing-packages-in-the-dut-without-the-package-manager
    >
    >
    >
    > The problem is that we currently have to specify each of these recipes to bitbake so they can built prior to testing, thus requiring a priori knowledge of which recipes our tests depend on.  This manual dependency management is error-prone and won’t scale well.  Is there a way for bitbake to discover these dependencies automatically prior to running `testimage` (which subsequently fails if the dependencies specified in the JSON files aren’t already built)?  Ideally we should be able to manage these recipes similarly to others used to build and populate the rootfs - fetch sources while we’re online, then build offline, then run the tests anytime thereafter.  Something like:
    >
    >   bitbake <image> -c testimage --runall="fetch"  # download sources (currently fetches some things, but not JSON-specified recipes?)
    >
    >   bitbake <image> -c testimage --runall="build"  # build/install/etc all test-related recipes
    >
    >   bitbake <image> -c testimage  # do the actual runtime test
    >
    >
    >
    > Thanks,
    >
    > -Connor
    >
    > --
    > _______________________________________________
    > poky mailing list
    > poky@yoctoproject.org
    > https://lists.yoctoproject.org/listinfo/poky
    

Re: Building and deploying test binaries

[Alexander Kanavin]

To be honest, to me there is almost no difference between test binaries placed into an image when it is created and test images transferred to the same image with ssh during a test run. In both cases you have to trust that they won’t affect the runtime behavior of the rest of the image. Yocto supports the former case well (e.g. the ptest framework to package and run tests per-recipe), the latter, not so much. 

Alex

> On 8 Mar 2019, at 23.23, Connor Imes <cimes@isi.edu> wrote:
> 
> Thanks for your response, Alex.  I can certainly see how creating a separate image is straightforward and useful for many teams - it fits very nicely into the layered approach that the OpenEmbedded framework offers.  I also agree that it has not been communicated particularly well.  There doesn't seem to be much documentation on our use case - where tests rely on cross-compiled applications not intended for the final rootfs.  The existence of the JSON-based approach seems to indicate that others have encountered similar requirements.  So my original question still remains, which is to find out if bitbake (or another tool) can automatically manage these JSON-specified test dependencies as part of the build lifecycle.
> 
> I'm still exploring options, but there are some reasons to prefer not having a separate test image.  If you're curious, some thoughts follow...
> 
> Probably first and foremost is that using a separate image means you're not directly testing the artifact you really care about - the original image on which the test image is based.  Trusting that the test results from the test image apply to your original image is predicated on the assumption that the test image is an exact replica of the original, modulo the addition of non-disruptive test recipes.  That is to say, you must assume that the test recipes don't conflict with each other or modify the system behavior in unexpected ways.  Additionally, while it is conceptually straightforward to just check for the addition of recipes (ignoring unexpected side effects), it could still be a slippery slope - it's easy to start making other changes, as the example you present actually does  (e.g., changing the overhead factor and assuming more memory).  We could debate the impacts of such changes on a case-by-case basis, but ultimately every change moves you further away from the baseline, adding risk.
> 
> For us, having to manage multiple images also makes things more challenging, in that the Yocto artifacts we produce are only a part of our embedded system.  As such, we have other scripts and components that would then need to account for multiple images, depending on the scenario.  Certainly this can be done, but for simplicity, it's easier not to require that, esp. when there's only one final rootfs needed from our poky configuration.
> 
> Cheers,
> -Connor
> 
> On 3/8/19, 2:26 PM, "Alexander Kanavin" <alex.kanavin@gmail.com> wrote:
> 
>    I'd say a special image for testing is *exactly* what you need. You
>    might take the existing image recipe, and include it wholesale into
>    your testing image recipe, then add additional recipes as needed. Then
>    the testing image is guaranteed to be an extension of the plain image.
>    Example:
> 
>    http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb
> 
>    Generally, creating various types of images serving various purposes
>    is not only ok, but is encouraged - YP isn't able to carry this
>    message across to users very well :-/
> 
>    Alex
> 
>>    On Fri, 8 Mar 2019 at 21:36, Connor Imes <cimes@isi.edu> wrote:
>> 
>> Hi all,
>> 
>> 
>> 
>> We have some applications that are meant exclusively for testing, and so do not want to include them by default in our image’s rootfs.  It also does not seem appropriate to build a separate image for testing.  After much searching, it seems like the JSON-based approach described here [1] is appropriate to automatically load/remove the binaries at runtime for use during testing.  I’ve written some recipes to build our test binaries, then specified their packages as dependencies using JSON files for our Python oeqa runtime test cases.  If I build the recipes with bitbake, the tests then execute successfully when I run `bitbake <image> -c testimage`.
>> 
>> 
>> 
>> [1] https://www.yoctoproject.org/docs/2.6/dev-manual/dev-manual.html#installing-packages-in-the-dut-without-the-package-manager
>> 
>> 
>> 
>> The problem is that we currently have to specify each of these recipes to bitbake so they can built prior to testing, thus requiring a priori knowledge of which recipes our tests depend on.  This manual dependency management is error-prone and won’t scale well.  Is there a way for bitbake to discover these dependencies automatically prior to running `testimage` (which subsequently fails if the dependencies specified in the JSON files aren’t already built)?  Ideally we should be able to manage these recipes similarly to others used to build and populate the rootfs - fetch sources while we’re online, then build offline, then run the tests anytime thereafter.  Something like:
>> 
>>  bitbake <image> -c testimage --runall="fetch"  # download sources (currently fetches some things, but not JSON-specified recipes?)
>> 
>>  bitbake <image> -c testimage --runall="build"  # build/install/etc all test-related recipes
>> 
>>  bitbake <image> -c testimage  # do the actual runtime test
>> 
>> 
>> 
>> Thanks,
>> 
>> -Connor
>> 
>> --
>> _______________________________________________
>> poky mailing list
>> poky@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/poky
> 
> 

Re: Building and deploying test binaries

[Alexander Kanavin]

> On 9 Mar 2019, at 0.11, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> 
> To be honest, to me there is almost no difference between test binaries placed into an image when it is created and test images transferred

Test *binaries* :)

Alex