proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Ana Guerrero Lopez]

[-- Attachment #1: Type: text/plain, Size: 4943 bytes --]


Hi everybody,

I'll add the information from this email in the README.md of the pull
request but it's difficult to discuss over GitHub, so I thought a mail
could make things easier.

At https://github.com/kernelci/kernelci-build-staging/pull/24I have
sent a proposal to build easily Debian images for every test suite
in arm64, armhf, x86 and x86_64.

The code has two parts, on one side, jenkins shared libraries that
are not included in the pull request: 
https://github.com/ana/kernelci-jenkins-libs
And on the other side, what's in the pull request: three small Jenkinsfiles
with the parameters for every job and the files to create the debian
image with debos.


While the shared libraries could be in the same repository that the
Jenkinsfile, I have created a separate repository for several reasons:

     * shared libraries must follow this directory structure
https://jenkins.io/doc/book/pipeline/shared-libraries/#directory-structure

     * if there are more shared libraries used in kernelci, it's good to
     have all of them in a separate repository, so they can be used
     from different git repositories or even from different branches
     *  You can add the shared library directly to the jenkins server
configuration  and use the in the pipeline syntax checker directly
     when playing/testing with the pipelines.

However, if the plan is to have all the Jenkinsfile in the same git
repository, then having an external repository for the shared libraries
is probably a bad idea.

The pull requests includes three things: three jenkinsfiles, debos files
and two Dockerfiles.

The jenkinsfiles are the smallest possible since all the code creating
the pipeline is in the shared library. There are two parts: one with the
job name - that will be used by the resulting images, the destination arches
and the run-time dependencies that need to be added to the image.
There is also the debos file name but this should be removed if we always
use the same debos configuration.
The second part "build_test_suite" is for building the test suite code.
This is on purpose a shell script that must create a cpio.gz tarball
with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
The idea is to be able to add and modify quickly test suites without
knowing too much about jenkins.

The debos files are under the debos directory. I'm not going to enter into
details here, as said in the pull request: debos is a wrapper using
debootstrap and qemu that makes easier to create images and to tweak them
with a set of pre-defined actions [0] following the list of actions
specified listed in a YAML file (debos/stretch.yaml).
debos is the standard tool in collabora for creating images, it's still
young and we're adding new features and fixing bugs as they appear in
our projects.

[0] https://godoc.org/github.com/go-debos/debos/actions

About the two dockerfiles: Dockerfile_debos that's a debian
base image with debos installed and Dockerfile_debian that provides
a debian container with the build dependencies to build the test suites.
As mentioned in the pull request, this second dockerfile should be updated
to have the container installing the required build dependencies for every
test suite.

Combining all the elements above, every jenkinsfile creates a pipeline
with 3 stages for every one of the arches where it runs:
1) debos creates a stripped down debian image installing the runtime
dependencies for running the test suite (listed in extra_packages)
2) build the test suites (shell script build_test_suite)
3) merge the results of the two previous stages and create the final image.

In the case of Jenkinsfile_basic, given that build_test_suite is empty,
it only runs the stage 1. There is a missing stage 4 that would be 
uploading
the final images to kerneci.org

I hope this helps to understand my pull request, especially for those
less familiar with jenkins pipelines.

Finally, a few comment for people who are more familiar with jenkins
pipelines. As you might know, there is a lot of ways to create jenkins
pipelines and my proposal isn't the most common used way for what
I have seen although it's based in an example from jenkinsci at:
https://github.com/jenkinsci/pipeline-examples/tree/master/global-library-examples/global-function

I chose this approach because I found it's the easiest for people for
adding new images for test suites without having to know a lot of about
jenkins.
In case, you're curious, I worked first in an approach using declarative
pipelines with small functions for every step. For example, the jenkinsfile
for building the igt test suite:
https://gitlab.collabora.com/ana/kernelci-build-staging/blob/3557b105b04883800c6d6320a53f44e366ec43bd/Jenkinsfile
The shared functions are at 
https://gitlab.collabora.com/ana/kernelci-jenkins-libs

Cheers,
Ana






[-- Attachment #2: Type: text/html, Size: 14125 bytes --]

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[khilman]

"Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:

> Hi everybody,
>
> I'll add the information from this email in the README.md of the pull
> request but it's difficult to discuss over GitHub, so I thought a mail
> could make things easier.
>
> At https://github.com/kernelci/kernelci-build-staging/pull/24I have
> sent a proposal to build easily Debian images for every test suite
> in arm64, armhf, x86 and x86_64.
>
> The code has two parts, on one side, jenkins shared libraries that
> are not included in the pull request:
> https://github.com/ana/kernelci-jenkins-libs

The code in the actual PR is intimately linked with the shared libs, so
IMO, there needs to be agreement on the shared libs before we can merge
this PR.

Also, at first glance, the shared libs need some review.  The first
thing that looks strange is the arch mappings:

    def debian_arch =  ["arm64": "armhf",
                        "armel": "arm64",
                        "x86": "i386",
                        "x86_64": "amd64"]

It looks to me like the arm64 and armhf are swapped.  The same looks
true for 'toolchain_arch'.

For example, the Jenkins log shows that the 'armel' build is passing
architecture:arm64 to debos:

   [Build image for armel] + debos -t architecture:arm64 -t basename:20180507.2/armel -t extra_packages:initramfs-tools jenkins/debian/debos/stretch.yaml

(I'm not super familiar with Jenkins pipelines, so I took a bit of time
to setup a local Jenkins pipeline so I could test this and have a better
idea what's actually happening.)

> And on the other side, what's in the pull request: three small Jenkinsfiles
> with the parameters for every job and the files to create the debian
> image with debos.
>
> While the shared libraries could be in the same repository that the
> Jenkinsfile, I have created a separate repository for several reasons:
>
>     * shared libraries must follow this directory structure
> https://jenkins.io/doc/book/pipeline/shared-libraries/#directory-structure
>
>     * if there are more shared libraries used in kernelci, it's good to
>     have all of them in a separate repository, so they can be used
>     from different git repositories or even from different branches
>     *  You can add the shared library directly to the jenkins server
> configuration  and use the in the pipeline syntax checker directly
>     when playing/testing with the pipelines.
>
> However, if the plan is to have all the Jenkinsfile in the same git
> repository, then having an external repository for the shared libraries
> is probably a bad idea.

I'm not sure what Jenkins best practices are here, but for kernelCI,
keeping them separate (at least in the long term) sounds like a good
idea.  However, in order to get the structure of the first few jobs
agreed upon and merged, it might make sense to keep them all in the same
repo for ease of review.

Also, Tomeu seems to have separate project for building a tiny
ramdisk/initramfs that doesn't use the shared libs at all.

IMO, what I think would be very helpful at least for initial review and
discussion, is to see an initial PR that only has the "basic" build, and
ideally also generates a minimal, tiny ramdisk from the same build
(e.g. with 'update-initramfs -c -k min' )

> The pull requests includes three things: three jenkinsfiles, debos files
> and two Dockerfiles.
>
> The jenkinsfiles are the smallest possible since all the code creating
> the pipeline is in the shared library. There are two parts: one with the
> job name - that will be used by the resulting images, the destination arches
> and the run-time dependencies that need to be added to the image.
> There is also the debos file name but this should be removed if we always
> use the same debos configuration.
> The second part "build_test_suite" is for building the test suite code.
> This is on purpose a shell script that must create a cpio.gz tarball
> with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
> The idea is to be able to add and modify quickly test suites without
> knowing too much about jenkins.

I'm not sure about the "build_test_suite" approach.

AFAICT, both the _igt and _v4l2 jobs are basically doing the same thing
as "basic", and then essentially installing a bunch more files on top.

Instead of all the rootfs duplication, couldn't the exact same thing be
accomplished by just having one "basic" rootfs image, and then passing
overlay tarballs to LAVA for IGT and V4L2?

IOW, I'm not sure I'm fully understanding the need for completely
separate rootfs for IGT and V4L2.

> The debos files are under the debos directory. I'm not going to enter into
> details here, as said in the pull request: debos is a wrapper using
> debootstrap and qemu that makes easier to create images and to tweak them
> with a set of pre-defined actions [0] following the list of actions
> specified listed in a YAML file (debos/stretch.yaml).
> debos is the standard tool in collabora for creating images, it's still
> young and we're adding new features and fixing bugs as they appear in
> our projects.

debos seems easy enough to understand so far.

One question/request for debos: in addition to creating a .cpio.gz,
could it also create filesystem images (e.g. ext2/3/4), and optionally
compress them?

One reason for this is if we want to use security attributes on the
rootfs, we can't use .cpio.gz, or NFS mounts.  We have to create a
"real" filesystem and use it as a ramdisk, or over the network with NBD.

> About the two dockerfiles: Dockerfile_debos that's a debian
> base image with debos installed and Dockerfile_debian that provides
> a debian container with the build dependencies to build the test suites.
> As mentioned in the pull request, this second dockerfile should be updated
> to have the container installing the required build dependencies for every
> test suite.
>
> Combining all the elements above, every jenkinsfile creates a pipeline
> with 3 stages for every one of the arches where it runs:
> 1) debos creates a stripped down debian image installing the runtime
> dependencies for running the test suite (listed in extra_packages)
> 2) build the test suites (shell script build_test_suite)
> 3) merge the results of the two previous stages and create the final image.

Again, I question the need for this "merge" step.  I think this should
just create another tarball that can be passed to LAVA as an (optional)
overlay when the specific test jobs require it.

Another reason to avoid the merge is if/when we want the base rootfs to
be have security attributes.  If that's the case, I don't think a simple
unpack/repack is going to work.

> In the case of Jenkinsfile_basic, given that build_test_suite is empty,
> it only runs the stage 1. There is a missing stage 4 that would be
> uploading
> the final images to kerneci.org

Tomeu's nfsroot Jenkinsfile has an example of how to do that using
the 'push-source.py' tool:
https://github.com/tomeuv/kernelci-build-staging/blob/nfsroot/jenkins/debian/Jenkinsfile

> I hope this helps to understand my pull request, especially for those
> less familiar with jenkins pipelines.

Yes, thanks for the detaile explanation.

I'm a little more familiar with pipelines now after I setup my own
Jenkins in order to test this PR locally.

Kevin

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[tomeu.vizoso]

Hi,

below I give my opinion on a few comments, but it's Ana who is leading 
now this work.

On 05/08/2018 01:09 AM, Kevin Hilman wrote:
 > "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
 >> And on the other side, what's in the pull request: three small 
Jenkinsfiles
 >> with the parameters for every job and the files to create the debian
 >> image with debos.
 >>
 >> While the shared libraries could be in the same repository that the
 >> Jenkinsfile, I have created a separate repository for several reasons:
 >>
 >>      * shared libraries must follow this directory structure
 >> 
https://jenkins.io/doc/book/pipeline/shared-libraries/#directory-structure
 >>
 >>      * if there are more shared libraries used in kernelci, it's good to
 >>      have all of them in a separate repository, so they can be used
 >>      from different git repositories or even from different branches
 >>      *  You can add the shared library directly to the jenkins server
 >> configuration  and use the in the pipeline syntax checker directly
 >>      when playing/testing with the pipelines.
 >>
 >> However, if the plan is to have all the Jenkinsfile in the same git
 >> repository, then having an external repository for the shared libraries
 >> is probably a bad idea.
 >
 > I'm not sure what Jenkins best practices are here, but for kernelCI,
 > keeping them separate (at least in the long term) sounds like a good
 > idea.  However, in order to get the structure of the first few jobs
 > agreed upon and merged, it might make sense to keep them all in the same
 > repo for ease of review.
 >
 > Also, Tomeu seems to have separate project for building a tiny
 > ramdisk/initramfs that doesn't use the shared libs at all.

Yes, but there was a massive amount of pipeline definition that would 
have to be duplicated between jobs.

 > IMO, what I think would be very helpful at least for initial review and
 > discussion, is to see an initial PR that only has the "basic" build, and
 > ideally also generates a minimal, tiny ramdisk from the same build
 > (e.g. with 'update-initramfs -c -k min' )
 >
 >> The pull requests includes three things: three jenkinsfiles, debos files
 >> and two Dockerfiles.
 >>
 >> The jenkinsfiles are the smallest possible since all the code creating
 >> the pipeline is in the shared library. There are two parts: one with the
 >> job name - that will be used by the resulting images, the destination 
arches
 >> and the run-time dependencies that need to be added to the image.
 >> There is also the debos file name but this should be removed if we always
 >> use the same debos configuration.
 >> The second part "build_test_suite" is for building the test suite code.
 >> This is on purpose a shell script that must create a cpio.gz tarball
 >> with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
 >> The idea is to be able to add and modify quickly test suites without
 >> knowing too much about jenkins.
 >
 > I'm not sure about the "build_test_suite" approach.
 >
 > AFAICT, both the _igt and _v4l2 jobs are basically doing the same thing
 > as "basic", and then essentially installing a bunch more files on top.

The difference is only in the dependencies. Both test suites are in the 
fat side and have several dependencies that otherwise aren't needed. That 
said, a basic image that contains all of them might still not be too big.

 > Instead of all the rootfs duplication, couldn't the exact same thing be
 > accomplished by just having one "basic" rootfs image, and then passing
 > overlay tarballs to LAVA for IGT and V4L2?

TTBOMK, with the move to LAVA v2 we lost the ability to apply random 
overlays to the initramfs, other than modules and the LAVA test helpers.

 > IOW, I'm not sure I'm fully understanding the need for completely
 > separate rootfs for IGT and V4L2.

It's just that priority was given to come up with the smallest possible 
images for each test suite. I'm concerned that some subsystems may have 
gone with for example python for their test suite and that could make it 
more difficult to have a single base image.

 >> The debos files are under the debos directory. I'm not going to enter 
into
 >> details here, as said in the pull request: debos is a wrapper using
 >> debootstrap and qemu that makes easier to create images and to tweak them
 >> with a set of pre-defined actions [0] following the list of actions
 >> specified listed in a YAML file (debos/stretch.yaml).
 >> debos is the standard tool in collabora for creating images, it's still
 >> young and we're adding new features and fixing bugs as they appear in
 >> our projects.
 >
 > debos seems easy enough to understand so far.
 >
 > One question/request for debos: in addition to creating a .cpio.gz,
 > could it also create filesystem images (e.g. ext2/3/4), and optionally
 > compress them?

Yep, that's the usual scenario.

 > One reason for this is if we want to use security attributes on the
 > rootfs, we can't use .cpio.gz, or NFS mounts.  We have to create a
 > "real" filesystem and use it as a ramdisk, or over the network with NBD.
 >
 >> About the two dockerfiles: Dockerfile_debos that's a debian
 >> base image with debos installed and Dockerfile_debian that provides
 >> a debian container with the build dependencies to build the test suites.
 >> As mentioned in the pull request, this second dockerfile should be 
updated
 >> to have the container installing the required build dependencies for 
every
 >> test suite.
 >>
 >> Combining all the elements above, every jenkinsfile creates a pipeline
 >> with 3 stages for every one of the arches where it runs:
 >> 1) debos creates a stripped down debian image installing the runtime
 >> dependencies for running the test suite (listed in extra_packages)
 >> 2) build the test suites (shell script build_test_suite)
 >> 3) merge the results of the two previous stages and create the final 
image.
 >
 > Again, I question the need for this "merge" step.  I think this should
 > just create another tarball that can be passed to LAVA as an (optional)
 > overlay when the specific test jobs require it.
 >
 > Another reason to avoid the merge is if/when we want the base rootfs to
 > be have security attributes.  If that's the case, I don't think a simple
 > unpack/repack is going to work.

Why wouldn't that work, ooi?

 >> In the case of Jenkinsfile_basic, given that build_test_suite is empty,
 >> it only runs the stage 1. There is a missing stage 4 that would be
 >> uploading
 >> the final images to kerneci.org
 >
 > Tomeu's nfsroot Jenkinsfile has an example of how to do that using
 > the 'push-source.py' tool:
 > 
https://github.com/tomeuv/kernelci-build-staging/blob/nfsroot/jenkins/debian/Jenkinsfile
 >
 >> I hope this helps to understand my pull request, especially for those
 >> less familiar with jenkins pipelines.
 >
 > Yes, thanks for the detaile explanation.
 >
 > I'm a little more familiar with pipelines now after I setup my own
 > Jenkins in order to test this PR locally.

Thanks for looking at this!

Tomeu

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Ana Guerrero Lopez]

Hi!

I have made a new pull request (PR#25), more details below.

On 08/05/18 01:09, Kevin Hilman wrote:
 > "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:

 > Also, at first glance, the shared libs need some review.  The first
 > thing that looks strange is the arch mappings:
 >
 >     def debian_arch =  ["arm64": "armhf",
 >                         "armel": "arm64",
 >                         "x86": "i386",
 >                         "x86_64": "amd64"]
 >
 > It looks to me like the arm64 and armhf are swapped.  The same looks
 > true for 'toolchain_arch'.

My bad, I had this fixed locally this but I never pushed the commit.

I have changed armel to arm to be more consistent with the linux/arch/XX
naming. If you happen to be using already armel in kernelci, please tell
me and I'll rename it back to armel.


 >> However, if the plan is to have all the Jenkinsfile in the same git
 >> repository, then having an external repository for the shared libraries
 >> is probably a bad idea.
 >
 > I'm not sure what Jenkins best practices are here, but for kernelCI,
 > keeping them separate (at least in the long term) sounds like a good
 > idea.  However, in order to get the structure of the first few jobs
 > agreed upon and merged, it might make sense to keep them all in the same
 > repo for ease of review.

Yeah, I agree. It's everything in the same repository now.

 > Also, Tomeu seems to have separate project for building a tiny
 > ramdisk/initramfs that doesn't use the shared libs at all.

For only one job it makes sense to have it all in the same Jenkinsfile.
But soon I'm expecting to have very similar jobs and we'll end up
copy-pasting the same blocks of code.

Anyway, for making things easier, right now there is only a function
(buildImage.groovy) and I have merged there the code from 
getDockerArgs.groovy
and setPipelineVersion.groovy
We'll see later when we add more pipelines which functions can be reused
and need to be split in different files for re-use.

You can see I kept the 3 lines in the top of Jenkinsfile_basic importing
the library from the same repository. This step is necessary unless
we declare the library in Jenkins (globally or when creating the pipeline).
This article has a example very similar to what we're doing and explain 
how to do it:
https://dev.to/jalogut/centralise-jenkins-pipelines-configuration-using-shared-libraries

We don't need to do it now, but I wanted to mention it because most
of the examples you'll find use this method and use the call @Library..
in the Jenkinsfile.

 > IMO, what I think would be very helpful at least for initial review and
 > discussion, is to see an initial PR that only has the "basic" build, and
 > ideally also generates a minimal, tiny ramdisk from the same build
 > (e.g. with 'update-initramfs -c -k min' )

The new pull request has only the basic build. I have modified the debos
files to create the tiny ramdisk and the Jenkinsfile to store it.



 > I'm not sure about the "build_test_suite" approach.
 >
 > AFAICT, both the _igt and _v4l2 jobs are basically doing the same thing
 > as "basic", and then essentially installing a bunch more files on top.
 >
 > Instead of all the rootfs duplication, couldn't the exact same thing be
 > accomplished by just having one "basic" rootfs image, and then passing
 > overlay tarballs to LAVA for IGT and V4L2?
 >
 > IOW, I'm not sure I'm fully understanding the need for completely
 > separate rootfs for IGT and V4L2.

Besides what tomeu said in his mail, IGT and V4L2 were bad examples to
compare here because they have almost the same dependencies.

Cheers,

Ana

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Kevin Hilman]

"Tomeu Vizoso" <tomeu.vizoso@collabora.com> writes:

> Hi,
>
> below I give my opinion on a few comments, but it's Ana who is leading
> now this work.
>
> On 05/08/2018 01:09 AM, Kevin Hilman wrote:
>> "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
>>> And on the other side, what's in the pull request: three small 
> Jenkinsfiles
>>> with the parameters for every job and the files to create the debian
>>> image with debos.
>>>
>>> While the shared libraries could be in the same repository that the
>>> Jenkinsfile, I have created a separate repository for several reasons:
>>>
>>>      * shared libraries must follow this directory structure
>>> 
> https://jenkins.io/doc/book/pipeline/shared-libraries/#directory-structure
>>>
>>>      * if there are more shared libraries used in kernelci, it's good to
>>>      have all of them in a separate repository, so they can be used
>>>      from different git repositories or even from different branches
>>>      *  You can add the shared library directly to the jenkins server
>>> configuration  and use the in the pipeline syntax checker directly
>>>      when playing/testing with the pipelines.
>>>
>>> However, if the plan is to have all the Jenkinsfile in the same git
>>> repository, then having an external repository for the shared libraries
>>> is probably a bad idea.
>>
>> I'm not sure what Jenkins best practices are here, but for kernelCI,
>> keeping them separate (at least in the long term) sounds like a good
>> idea.  However, in order to get the structure of the first few jobs
>> agreed upon and merged, it might make sense to keep them all in the same
>> repo for ease of review.
>>
>> Also, Tomeu seems to have separate project for building a tiny
>> ramdisk/initramfs that doesn't use the shared libs at all.
>
> Yes, but there was a massive amount of pipeline definition that would
> have to be duplicated between jobs.
>
>> IMO, what I think would be very helpful at least for initial review and
>> discussion, is to see an initial PR that only has the "basic" build, and
>> ideally also generates a minimal, tiny ramdisk from the same build
>> (e.g. with 'update-initramfs -c -k min' )
>>
>>> The pull requests includes three things: three jenkinsfiles, debos files
>>> and two Dockerfiles.
>>>
>>> The jenkinsfiles are the smallest possible since all the code creating
>>> the pipeline is in the shared library. There are two parts: one with the
>>> job name - that will be used by the resulting images, the
>>> destination 
> arches
>>> and the run-time dependencies that need to be added to the image.
>>> There is also the debos file name but this should be removed if we always
>>> use the same debos configuration.
>>> The second part "build_test_suite" is for building the test suite code.
>>> This is on purpose a shell script that must create a cpio.gz tarball
>>> with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
>>> The idea is to be able to add and modify quickly test suites without
>>> knowing too much about jenkins.
>>
>> I'm not sure about the "build_test_suite" approach.
>>
>> AFAICT, both the _igt and _v4l2 jobs are basically doing the same thing
>> as "basic", and then essentially installing a bunch more files on top.
>
> The difference is only in the dependencies. Both test suites are in
> the fat side and have several dependencies that otherwise aren't
> needed. That said, a basic image that contains all of them might still
> not be too big.

IMO, it's better to go for a single, shared base image with
dependencies.  Building a new rootfs for every test suite sounds like a
scalability problem to me.

>> Instead of all the rootfs duplication, couldn't the exact same thing be
>> accomplished by just having one "basic" rootfs image, and then passing
>> overlay tarballs to LAVA for IGT and V4L2?
>
> TTBOMK, with the move to LAVA v2 we lost the ability to apply random
> overlays to the initramfs, other than modules and the LAVA test
> helpers.
>
>> IOW, I'm not sure I'm fully understanding the need for completely
>> separate rootfs for IGT and V4L2.
>
> It's just that priority was given to come up with the smallest
> possible images for each test suite. I'm concerned that some
> subsystems may have gone with for example python for their test suite
> and that could make it more difficult to have a single base image.

I'm not sure to understand the priority of the smallest possible rootfs
image for each test suite.  Why not the smallest possible initrd that
can pivot to a "real", and possibly large, rootfs on MMC/NFS whatever
than has the dependencies for some set of tests?

Then, using LAVA test shell, you write the test jobs in a way that
LAVA will fetch/install the test-specific bits that are in the test
definiton.  LAVA test shell can fetch stuff from git, download arbitrary
tarballs, etc. and either overlay them on the rootfs (default), or make
them available for the target to fetch (e.g. wget) in the case where the
rootfs has security attributes.

>>> The debos files are under the debos directory. I'm not going to
>>> enter
> into
>>> details here, as said in the pull request: debos is a wrapper using
>>> debootstrap and qemu that makes easier to create images and to tweak them
>>> with a set of pre-defined actions [0] following the list of actions
>>> specified listed in a YAML file (debos/stretch.yaml).
>>> debos is the standard tool in collabora for creating images, it's still
>>> young and we're adding new features and fixing bugs as they appear in
>>> our projects.
>>
>> debos seems easy enough to understand so far.
>>
>> One question/request for debos: in addition to creating a .cpio.gz,
>> could it also create filesystem images (e.g. ext2/3/4), and optionally
>> compress them?
>
> Yep, that's the usual scenario.
>
>> One reason for this is if we want to use security attributes on the
>> rootfs, we can't use .cpio.gz, or NFS mounts.  We have to create a
>> "real" filesystem and use it as a ramdisk, or over the network with NBD.
>>
>>> About the two dockerfiles: Dockerfile_debos that's a debian
>>> base image with debos installed and Dockerfile_debian that provides
>>> a debian container with the build dependencies to build the test suites.
>>> As mentioned in the pull request, this second dockerfile should be 
> updated
>>> to have the container installing the required build dependencies
>>> for 
> every
>>> test suite.
>>>
>>> Combining all the elements above, every jenkinsfile creates a pipeline
>>> with 3 stages for every one of the arches where it runs:
>>> 1) debos creates a stripped down debian image installing the runtime
>>> dependencies for running the test suite (listed in extra_packages)
>>> 2) build the test suites (shell script build_test_suite)
>>> 3) merge the results of the two previous stages and create the
>>> final 
> image.
>>
>> Again, I question the need for this "merge" step.  I think this should
>> just create another tarball that can be passed to LAVA as an (optional)
>> overlay when the specific test jobs require it.
>>
>> Another reason to avoid the merge is if/when we want the base rootfs to
>> be have security attributes.  If that's the case, I don't think a simple
>> unpack/repack is going to work.
>
> Why wouldn't that work, ooi?

It might, but it smells funny (and I haven't tried it.)

>>> In the case of Jenkinsfile_basic, given that build_test_suite is empty,
>>> it only runs the stage 1. There is a missing stage 4 that would be
>>> uploading
>>> the final images to kerneci.org
>>
>> Tomeu's nfsroot Jenkinsfile has an example of how to do that using
>> the 'push-source.py' tool:
>> 
> https://github.com/tomeuv/kernelci-build-staging/blob/nfsroot/jenkins/debian/Jenkinsfile
>>
>>> I hope this helps to understand my pull request, especially for those
>>> less familiar with jenkins pipelines.
>>
>> Yes, thanks for the detaile explanation.
>>
>> I'm a little more familiar with pipelines now after I setup my own
>> Jenkins in order to test this PR locally.
>
> Thanks for looking at this!
>
> Tomeu

Kevin

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Kevin Hilman]

"Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:

> I have made a new pull request (PR#25), more details below.

Thanks, I've made a few comments there.

> On 08/05/18 01:09, Kevin Hilman wrote:
>> "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
>
>> Also, at first glance, the shared libs need some review.  The first
>> thing that looks strange is the arch mappings:
>>
>>     def debian_arch =  ["arm64": "armhf",
>>                         "armel": "arm64",
>>                         "x86": "i386",
>>                         "x86_64": "amd64"]
>>
>> It looks to me like the arm64 and armhf are swapped.  The same looks
>> true for 'toolchain_arch'.
>
> My bad, I had this fixed locally this but I never pushed the commit.
>
> I have changed armel to arm to be more consistent with the linux/arch/XX
> naming. If you happen to be using already armel in kernelci, please tell
> me and I'll rename it back to armel.

We're already using armel in kernelCI on purpose, because we still have
a handful of ARMv5 boards.  As I suggested on your PR, maybe we could
build armel and armhf so both are available.

>>> However, if the plan is to have all the Jenkinsfile in the same git
>>> repository, then having an external repository for the shared libraries
>>> is probably a bad idea.
>>
>> I'm not sure what Jenkins best practices are here, but for kernelCI,
>> keeping them separate (at least in the long term) sounds like a good
>> idea.  However, in order to get the structure of the first few jobs
>> agreed upon and merged, it might make sense to keep them all in the same
>> repo for ease of review.
>
> Yeah, I agree. It's everything in the same repository now.
>
>> Also, Tomeu seems to have separate project for building a tiny
>> ramdisk/initramfs that doesn't use the shared libs at all.
>
> For only one job it makes sense to have it all in the same Jenkinsfile.
> But soon I'm expecting to have very similar jobs and we'll end up
> copy-pasting the same blocks of code.
>
> Anyway, for making things easier, right now there is only a function
> (buildImage.groovy) and I have merged there the code from
> getDockerArgs.groovy
> and setPipelineVersion.groovy
> We'll see later when we add more pipelines which functions can be reused
> and need to be split in different files for re-use.

Thanks, that makes it easier to review all together, and doesn't have a
dependency on an external repo for the libs.

> You can see I kept the 3 lines in the top of Jenkinsfile_basic importing
> the library from the same repository. This step is necessary unless
> we declare the library in Jenkins (globally or when creating the pipeline).
> This article has a example very similar to what we're doing and
> explain how to do it:
> https://dev.to/jalogut/centralise-jenkins-pipelines-configuration-using-shared-libraries
> We don't need to do it now, but I wanted to mention it because most
> of the examples you'll find use this method and use the call @Library..
> in the Jenkinsfile.

OK, when we start doing it that way, please include that in link in the
changelog as background.

And, while you're fixing the changelog, s/var/vars/.  (I couldn't figure
out how to comment on the changelog in github.)

>> IMO, what I think would be very helpful at least for initial review and
>> discussion, is to see an initial PR that only has the "basic" build, and
>> ideally also generates a minimal, tiny ramdisk from the same build
>> (e.g. with 'update-initramfs -c -k min' )
>
> The new pull request has only the basic build. I have modified the debos
> files to create the tiny ramdisk and the Jenkinsfile to store it.

Very nice, thank you.

Kevin

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Tomeu Vizoso]

On 05/10/2018 03:04 AM, Kevin Hilman wrote:
> "Tomeu Vizoso" <tomeu.vizoso@collabora.com> writes:
> 
>> Hi,
>>
>> below I give my opinion on a few comments, but it's Ana who is leading
>> now this work.
>>
>> On 05/08/2018 01:09 AM, Kevin Hilman wrote:
>>> "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
>>
>>> IMO, what I think would be very helpful at least for initial review and
>>> discussion, is to see an initial PR that only has the "basic" build, and
>>> ideally also generates a minimal, tiny ramdisk from the same build
>>> (e.g. with 'update-initramfs -c -k min' )
>>>
>>>> The pull requests includes three things: three jenkinsfiles, debos files
>>>> and two Dockerfiles.
>>>>
>>>> The jenkinsfiles are the smallest possible since all the code creating
>>>> the pipeline is in the shared library. There are two parts: one with the
>>>> job name - that will be used by the resulting images, the
>>>> destination
>> arches
>>>> and the run-time dependencies that need to be added to the image.
>>>> There is also the debos file name but this should be removed if we always
>>>> use the same debos configuration.
>>>> The second part "build_test_suite" is for building the test suite code.
>>>> This is on purpose a shell script that must create a cpio.gz tarball
>>>> with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
>>>> The idea is to be able to add and modify quickly test suites without
>>>> knowing too much about jenkins.
>>>
>>> I'm not sure about the "build_test_suite" approach.
>>>
>>> AFAICT, both the _igt and _v4l2 jobs are basically doing the same thing
>>> as "basic", and then essentially installing a bunch more files on top.
>>
>> The difference is only in the dependencies. Both test suites are in
>> the fat side and have several dependencies that otherwise aren't
>> needed. That said, a basic image that contains all of them might still
>> not be too big.
> 
> IMO, it's better to go for a single, shared base image with
> dependencies.  Building a new rootfs for every test suite sounds like a
> scalability problem to me.

Well, only the fatter test suites would need their own rootfs. So far 
only IGT and V4L2 have a fair amount of dependencies. But still we could 
probably build a single image for both that isn't too big to be used as a 
ramdisk.

>>> Instead of all the rootfs duplication, couldn't the exact same thing be
>>> accomplished by just having one "basic" rootfs image, and then passing
>>> overlay tarballs to LAVA for IGT and V4L2?
>>
>> TTBOMK, with the move to LAVA v2 we lost the ability to apply random
>> overlays to the initramfs, other than modules and the LAVA test
>> helpers.
>>
>>> IOW, I'm not sure I'm fully understanding the need for completely
>>> separate rootfs for IGT and V4L2.
>>
>> It's just that priority was given to come up with the smallest
>> possible images for each test suite. I'm concerned that some
>> subsystems may have gone with for example python for their test suite
>> and that could make it more difficult to have a single base image.
> 
> I'm not sure to understand the priority of the smallest possible rootfs
> image for each test suite.  Why not the smallest possible initrd that
> can pivot to a "real", and possibly large, rootfs on MMC/NFS whatever
> than has the dependencies for some set of tests?

The main problem with NFS is that we should be testing that subsystems 
keep working properly across suspend/resume cycles, and often the network 
device will go away when the machine resumes. When the network device 
comes back and userspace tries to setup the network device, it finds out 
that the files it needs to do so aren't available. A second problem is 
that at some point we'll want to use the network for functional and 
performance testing.

The problems with secondary storage are:

- it would greatly increase the test duration,
- we cannot use it when running destructive tests such as those in fio,
- it would greatly increase lab admin costs specially for devices without 
internal storage in which SD cards or USB devices have to be used instead 
of MMC, and
- because of the above, we would be getting false positives when media 
starts failing.

Memory is basically the only medium that is:

- fast to deploy,
- always there,
- reliable,
- tests will never interfere with.

The biggest downside is that it's a more scarce resource, thus the 
priority given to reducing the sizes of the ramdisks.

> Then, using LAVA test shell, you write the test jobs in a way that
> LAVA will fetch/install the test-specific bits that are in the test
> definiton.  LAVA test shell can fetch stuff from git, download arbitrary
> tarballs, etc. and either overlay them on the rootfs (default), or make
> them available for the target to fetch (e.g. wget) in the case where the
> rootfs has security attributes.

For scalability reasons I think we should move as much work as possible 
to happen outside the DUTs. We're going to have a very limited time 
budget to test as much as possible in the DUTs and if we do anything else 
we're going to hit limits very fast and I'm afraid the coverage will be 
greatly reduced.

>>> One reason for this is if we want to use security attributes on the
>>> rootfs, we can't use .cpio.gz, or NFS mounts.  We have to create a
>>> "real" filesystem and use it as a ramdisk, or over the network with NBD.
>>>
>>>> About the two dockerfiles: Dockerfile_debos that's a debian
>>>> base image with debos installed and Dockerfile_debian that provides
>>>> a debian container with the build dependencies to build the test suites.
>>>> As mentioned in the pull request, this second dockerfile should be
>> updated
>>>> to have the container installing the required build dependencies
>>>> for
>> every
>>>> test suite.
>>>>
>>>> Combining all the elements above, every jenkinsfile creates a pipeline
>>>> with 3 stages for every one of the arches where it runs:
>>>> 1) debos creates a stripped down debian image installing the runtime
>>>> dependencies for running the test suite (listed in extra_packages)
>>>> 2) build the test suites (shell script build_test_suite)
>>>> 3) merge the results of the two previous stages and create the
>>>> final
>> image.
>>>
>>> Again, I question the need for this "merge" step.  I think this should
>>> just create another tarball that can be passed to LAVA as an (optional)
>>> overlay when the specific test jobs require it.
>>>
>>> Another reason to avoid the merge is if/when we want the base rootfs to
>>> be have security attributes.  If that's the case, I don't think a simple
>>> unpack/repack is going to work.
>>
>> Why wouldn't that work, ooi?
> 
> It might, but it smells funny (and I haven't tried it.)

Ok, guess we can try and do things differently if needed.

Cheers,

Tomeu

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Kevin Hilman]

Tomeu Vizoso <tomeu.vizoso@collabora.com> writes:

> On 05/10/2018 03:04 AM, Kevin Hilman wrote:
>> "Tomeu Vizoso" <tomeu.vizoso@collabora.com> writes:
>>
>>> Hi,
>>>
>>> below I give my opinion on a few comments, but it's Ana who is leading
>>> now this work.
>>>
>>> On 05/08/2018 01:09 AM, Kevin Hilman wrote:
>>>> "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
>>>
>>>> IMO, what I think would be very helpful at least for initial review and
>>>> discussion, is to see an initial PR that only has the "basic" build, and
>>>> ideally also generates a minimal, tiny ramdisk from the same build
>>>> (e.g. with 'update-initramfs -c -k min' )
>>>>
>>>>> The pull requests includes three things: three jenkinsfiles, debos files
>>>>> and two Dockerfiles.
>>>>>
>>>>> The jenkinsfiles are the smallest possible since all the code creating
>>>>> the pipeline is in the shared library. There are two parts: one with the
>>>>> job name - that will be used by the resulting images, the
>>>>> destination
>>> arches
>>>>> and the run-time dependencies that need to be added to the image.
>>>>> There is also the debos file name but this should be removed if we always
>>>>> use the same debos configuration.
>>>>> The second part "build_test_suite" is for building the test suite code.
>>>>> This is on purpose a shell script that must create a cpio.gz tarball
>>>>> with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
>>>>> The idea is to be able to add and modify quickly test suites without
>>>>> knowing too much about jenkins.
>>>>
>>>> I'm not sure about the "build_test_suite" approach.
>>>>
>>>> AFAICT, both the _igt and _v4l2 jobs are basically doing the same thing
>>>> as "basic", and then essentially installing a bunch more files on top.
>>>
>>> The difference is only in the dependencies. Both test suites are in
>>> the fat side and have several dependencies that otherwise aren't
>>> needed. That said, a basic image that contains all of them might still
>>> not be too big.
>>
>> IMO, it's better to go for a single, shared base image with
>> dependencies.  Building a new rootfs for every test suite sounds like a
>> scalability problem to me.
>
> Well, only the fatter test suites would need their own rootfs. So far
> only IGT and V4L2 have a fair amount of dependencies. But still we
> could probably build a single image for both that isn't too big to be
> used as a ramdisk.

OK, I think we should start with that.

>>>> Instead of all the rootfs duplication, couldn't the exact same thing be
>>>> accomplished by just having one "basic" rootfs image, and then passing
>>>> overlay tarballs to LAVA for IGT and V4L2?
>>>
>>> TTBOMK, with the move to LAVA v2 we lost the ability to apply random
>>> overlays to the initramfs, other than modules and the LAVA test
>>> helpers.
>>>
>>>> IOW, I'm not sure I'm fully understanding the need for completely
>>>> separate rootfs for IGT and V4L2.
>>>
>>> It's just that priority was given to come up with the smallest
>>> possible images for each test suite. I'm concerned that some
>>> subsystems may have gone with for example python for their test suite
>>> and that could make it more difficult to have a single base image.
>>
>> I'm not sure to understand the priority of the smallest possible rootfs
>> image for each test suite.  Why not the smallest possible initrd that
>> can pivot to a "real", and possibly large, rootfs on MMC/NFS whatever
>> than has the dependencies for some set of tests?
>
> The main problem with NFS is that we should be testing that subsystems
> keep working properly across suspend/resume cycles, and often the
> network device will go away when the machine resumes. When the network
> device comes back and userspace tries to setup the network device, it
> finds out that the files it needs to do so aren't available. A second
> problem is that at some point we'll want to use the network for
> functional and performance testing.
>
> The problems with secondary storage are:
>
> - it would greatly increase the test duration,
> - we cannot use it when running destructive tests such as those in fio,
> - it would greatly increase lab admin costs specially for devices
> without internal storage in which SD cards or USB devices have to be
> used instead of MMC, and
> - because of the above, we would be getting false positives when media
> starts failing.
>
> Memory is basically the only medium that is:
>
> - fast to deploy,
> - always there,
> - reliable,
> - tests will never interfere with.
>
> The biggest downside is that it's a more scarce resource, thus the
> priority given to reducing the sizes of the ramdisks.

Well summarized, and I agree.

One other downside is that production systems don't (generally) have
their primary rootfs in memory, but usually on secondary storage, so
you don't end up testing how production systems actually work. 

But I agree, for scalability, secondary storage is a major PITA.
However, maybe for a subset of devices that we want to test that way, we
should make it easy.

So, what about this: From a single base debian image, we create

1) minimal (~2M) initramfs

This should have pivot capabilities, and this is what Ana is doing today
with 'update-initramfs -c -k min'

2) small "basic" ramdisk

This is the base image *after* the strip/crush steps.  Basically, the
"basic" that Ana is building today.

3) full rootfs for secondary storage (e.g. MMC, NFS, NBD)

The base image without all the strip/crush removed, so it still has the
ability to install packages etc.

This should be available in a few formats: .cpio.gz, .tar.xz, .ext4.xz

This would also be the pivot destination when using an initramfs.

>> Then, using LAVA test shell, you write the test jobs in a way that
>> LAVA will fetch/install the test-specific bits that are in the test
>> definiton.  LAVA test shell can fetch stuff from git, download arbitrary
>> tarballs, etc. and either overlay them on the rootfs (default), or make
>> them available for the target to fetch (e.g. wget) in the case where the
>> rootfs has security attributes.
>
> For scalability reasons I think we should move as much work as
> possible to happen outside the DUTs. We're going to have a very
> limited time budget to test as much as possible in the DUTs and if we
> do anything else we're going to hit limits very fast and I'm afraid
> the coverage will be greatly reduced.

In general, I agree.  For speed/scalability, this defniitely slows
things down.  However, I want to make sure it remains in the realm of
usecases for our rootfs images.

For example, there are cases where you want to test a production (or
product-like) rootfs that may not have all the test dependencies on it.
The additional catch is the rootfs may have secure attributes, so you
can't just let LAVA do overlays on it.  In these cases, the DUT has to
do fetch the test overlay and unpack it *after* mounting the rootfs.

Again, this may not be the normal case that scales up well, but we
should allow it.  It also makes it easy to write tests that actually
install their own dependencies before running.

Kevin

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Tomeu Vizoso]

On 05/10/2018 05:41 PM, Kevin Hilman wrote:
> Tomeu Vizoso <tomeu.vizoso@collabora.com> writes:
> 
>> On 05/10/2018 03:04 AM, Kevin Hilman wrote:
>>> "Tomeu Vizoso" <tomeu.vizoso@collabora.com> writes:
>>>
>>>> Hi,
>>>>
>>>> below I give my opinion on a few comments, but it's Ana who is leading
>>>> now this work.
>>>>
>>>> On 05/08/2018 01:09 AM, Kevin Hilman wrote:
>>>>> "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
>>>>
>>>>> IMO, what I think would be very helpful at least for initial review and
>>>>> discussion, is to see an initial PR that only has the "basic" build, and
>>>>> ideally also generates a minimal, tiny ramdisk from the same build
>>>>> (e.g. with 'update-initramfs -c -k min' )
>>>>>
>>>>>> The pull requests includes three things: three jenkinsfiles, debos files
>>>>>> and two Dockerfiles.
>>>>>>
>>>>>> The jenkinsfiles are the smallest possible since all the code creating
>>>>>> the pipeline is in the shared library. There are two parts: one with the
>>>>>> job name - that will be used by the resulting images, the
>>>>>> destination
>>>> arches
>>>>>> and the run-time dependencies that need to be added to the image.
>>>>>> There is also the debos file name but this should be removed if we always
>>>>>> use the same debos configuration.
>>>>>> The second part "build_test_suite" is for building the test suite code.
>>>>>> This is on purpose a shell script that must create a cpio.gz tarball
>>>>>> with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
>>>>>> The idea is to be able to add and modify quickly test suites without
>>>>>> knowing too much about jenkins.
>>>>>
>>>>> I'm not sure about the "build_test_suite" approach.
>>>>>
>>>>> AFAICT, both the _igt and _v4l2 jobs are basically doing the same thing
>>>>> as "basic", and then essentially installing a bunch more files on top.
>>>>
>>>> The difference is only in the dependencies. Both test suites are in
>>>> the fat side and have several dependencies that otherwise aren't
>>>> needed. That said, a basic image that contains all of them might still
>>>> not be too big.
>>>
>>> IMO, it's better to go for a single, shared base image with
>>> dependencies.  Building a new rootfs for every test suite sounds like a
>>> scalability problem to me.
>>
>> Well, only the fatter test suites would need their own rootfs. So far
>> only IGT and V4L2 have a fair amount of dependencies. But still we
>> could probably build a single image for both that isn't too big to be
>> used as a ramdisk.
> 
> OK, I think we should start with that.

What do you think about that, Ana?

>>>>> Instead of all the rootfs duplication, couldn't the exact same thing be
>>>>> accomplished by just having one "basic" rootfs image, and then passing
>>>>> overlay tarballs to LAVA for IGT and V4L2?
>>>>
>>>> TTBOMK, with the move to LAVA v2 we lost the ability to apply random
>>>> overlays to the initramfs, other than modules and the LAVA test
>>>> helpers.
>>>>
>>>>> IOW, I'm not sure I'm fully understanding the need for completely
>>>>> separate rootfs for IGT and V4L2.
>>>>
>>>> It's just that priority was given to come up with the smallest
>>>> possible images for each test suite. I'm concerned that some
>>>> subsystems may have gone with for example python for their test suite
>>>> and that could make it more difficult to have a single base image.
>>>
>>> I'm not sure to understand the priority of the smallest possible rootfs
>>> image for each test suite.  Why not the smallest possible initrd that
>>> can pivot to a "real", and possibly large, rootfs on MMC/NFS whatever
>>> than has the dependencies for some set of tests?
>>
>> The main problem with NFS is that we should be testing that subsystems
>> keep working properly across suspend/resume cycles, and often the
>> network device will go away when the machine resumes. When the network
>> device comes back and userspace tries to setup the network device, it
>> finds out that the files it needs to do so aren't available. A second
>> problem is that at some point we'll want to use the network for
>> functional and performance testing.
>>
>> The problems with secondary storage are:
>>
>> - it would greatly increase the test duration,
>> - we cannot use it when running destructive tests such as those in fio,
>> - it would greatly increase lab admin costs specially for devices
>> without internal storage in which SD cards or USB devices have to be
>> used instead of MMC, and
>> - because of the above, we would be getting false positives when media
>> starts failing.
>>
>> Memory is basically the only medium that is:
>>
>> - fast to deploy,
>> - always there,
>> - reliable,
>> - tests will never interfere with.
>>
>> The biggest downside is that it's a more scarce resource, thus the
>> priority given to reducing the sizes of the ramdisks.
> 
> Well summarized, and I agree.
> 
> One other downside is that production systems don't (generally) have
> their primary rootfs in memory, but usually on secondary storage, so
> you don't end up testing how production systems actually work.

Yep, Collabora's current focus is towards mostly-black-box testing of 
individual subsystems, but I sure think at some point we'll see 
downstreams testing mainline (or even next) in their production hardware, 
with their own battery of full-system tests on their userspace.

We'll be happy to help those companies do that as part of kernelci, but I 
think we need to make mainline a bit more robust and stable before we do 
that, thus why we focus first on per-subsystem sanity testing.

> But I agree, for scalability, secondary storage is a major PITA.
> However, maybe for a subset of devices that we want to test that way, we
> should make it easy.
> 
> So, what about this: From a single base debian image, we create
> 
> 1) minimal (~2M) initramfs
> 
> This should have pivot capabilities, and this is what Ana is doing today
> with 'update-initramfs -c -k min'
> 
> 2) small "basic" ramdisk
> 
> This is the base image *after* the strip/crush steps.  Basically, the
> "basic" that Ana is building today.
> 
> 3) full rootfs for secondary storage (e.g. MMC, NFS, NBD)
> 
> The base image without all the strip/crush removed, so it still has the
> ability to install packages etc.
> 
> This should be available in a few formats: .cpio.gz, .tar.xz, .ext4.xz
> 
> This would also be the pivot destination when using an initramfs.

Sounds good!

>>> Then, using LAVA test shell, you write the test jobs in a way that
>>> LAVA will fetch/install the test-specific bits that are in the test
>>> definiton.  LAVA test shell can fetch stuff from git, download arbitrary
>>> tarballs, etc. and either overlay them on the rootfs (default), or make
>>> them available for the target to fetch (e.g. wget) in the case where the
>>> rootfs has security attributes.
>>
>> For scalability reasons I think we should move as much work as
>> possible to happen outside the DUTs. We're going to have a very
>> limited time budget to test as much as possible in the DUTs and if we
>> do anything else we're going to hit limits very fast and I'm afraid
>> the coverage will be greatly reduced.
> 
> In general, I agree.  For speed/scalability, this defniitely slows
> things down.  However, I want to make sure it remains in the realm of
> usecases for our rootfs images.
> 
> For example, there are cases where you want to test a production (or
> product-like) rootfs that may not have all the test dependencies on it.
> The additional catch is the rootfs may have secure attributes, so you
> can't just let LAVA do overlays on it.  In these cases, the DUT has to
> do fetch the test overlay and unpack it *after* mounting the rootfs.
> 
> Again, this may not be the normal case that scales up well, but we
> should allow it.  It also makes it easy to write tests that actually
> install their own dependencies before running.

Agreed.

Cheers,

Tomeu

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Ana Guerrero Lopez]

On 10/05/18 03:15, Kevin Hilman wrote:
 > "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
 >
 >> I have made a new pull request (PR#25), more details below.

I updated the PR, a couple of comments below for those not following
closely in GitHub.

[...]
 >> I have changed armel to arm to be more consistent with the linux/arch/XX
 >> naming. If you happen to be using already armel in kernelci, please tell
 >> me and I'll rename it back to armel.
 >
 > We're already using armel in kernelCI on purpose, because we still have
 > a handful of ARMv5 boards.  As I suggested on your PR, maybe we could
 > build armel and armhf so both are available.

We have now 5 architectures with the following mapping:

     def debian_arch =  ["armhf": "armhf",
                         "armel": "armel",
                         "arm64": "arm64",
                         "x86": "i386",
                         "x86_64": "amd64"]

producing 5 different images for every arch!


[...]
 >> You can see I kept the 3 lines in the top of Jenkinsfile_basic importing
 >> the library from the same repository. This step is necessary unless
 >> we declare the library in Jenkins (globally or when creating the 
pipeline).
 >> This article has a example very similar to what we're doing and
 >> explain how to do it:
 >> 
https://dev.to/jalogut/centralise-jenkins-pipelines-configuration-using-shared-libraries
 >> We don't need to do it now, but I wanted to mention it because most
 >> of the examples you'll find use this method and use the call @Library..
 >> in the Jenkinsfile.
 >
 > OK, when we start doing it that way, please include that in link in the
 > changelog as background.
 >
 > And, while you're fixing the changelog, s/var/vars/. (I couldn't figure
 > out how to comment on the changelog in github.)

I fixed the typo in the changelog and updated var/README.md with the
information on how to use the shared libraries.


Finally, I've added a last commit with a new stage to upload the generated
images to kernelci.org. Apart from any other problem that needs fixing,
I consider the pull request ready to be merged.

Cheers,
Ana

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Ana Guerrero Lopez]

On 11/05/18 07:23, Tomeu Vizoso wrote:
 > On 05/10/2018 05:41 PM, Kevin Hilman wrote:
 >> Tomeu Vizoso <tomeu.vizoso@collabora.com> writes:
 >>
 >>> On 05/10/2018 03:04 AM, Kevin Hilman wrote:
 >>>> "Tomeu Vizoso" <tomeu.vizoso@collabora.com> writes:
 >>>>
 >>>>> Hi,
 >>>>>
 >>>>> below I give my opinion on a few comments, but it's Ana who is 
leading
 >>>>> now this work.
 >>>>>
 >>>>> On 05/08/2018 01:09 AM, Kevin Hilman wrote:
 >>>>>> "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
 >>>>>
 >>>>>> IMO, what I think would be very helpful at least for initial 
review and
 >>>>>> discussion, is to see an initial PR that only has the "basic" 
build, and
 >>>>>> ideally also generates a minimal, tiny ramdisk from the same build
 >>>>>> (e.g. with 'update-initramfs -c -k min' )
 >>>>>>
 >>>>>>> The pull requests includes three things: three jenkinsfiles, 
debos files
 >>>>>>> and two Dockerfiles.
 >>>>>>>
 >>>>>>> The jenkinsfiles are the smallest possible since all the code 
creating
 >>>>>>> the pipeline is in the shared library. There are two parts: one 
with the
 >>>>>>> job name - that will be used by the resulting images, the
 >>>>>>> destination
 >>>>> arches
 >>>>>>> and the run-time dependencies that need to be added to the image.
 >>>>>>> There is also the debos file name but this should be removed if 
we always
 >>>>>>> use the same debos configuration.
 >>>>>>> The second part "build_test_suite" is for building the test 
suite code.
 >>>>>>> This is on purpose a shell script that must create a cpio.gz 
tarball
 >>>>>>> with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
 >>>>>>> The idea is to be able to add and modify quickly test suites 
without
 >>>>>>> knowing too much about jenkins.
 >>>>>>
 >>>>>> I'm not sure about the "build_test_suite" approach.
 >>>>>>
 >>>>>> AFAICT, both the _igt and _v4l2 jobs are basically doing the 
same thing
 >>>>>> as "basic", and then essentially installing a bunch more files 
on top.
 >>>>>
 >>>>> The difference is only in the dependencies. Both test suites are in
 >>>>> the fat side and have several dependencies that otherwise aren't
 >>>>> needed. That said, a basic image that contains all of them might 
still
 >>>>> not be too big.
 >>>>
 >>>> IMO, it's better to go for a single, shared base image with
 >>>> dependencies.  Building a new rootfs for every test suite sounds 
like a
 >>>> scalability problem to me.
 >>>
 >>> Well, only the fatter test suites would need their own rootfs. So far
 >>> only IGT and V4L2 have a fair amount of dependencies. But still we
 >>> could probably build a single image for both that isn't too big to be
 >>> used as a ramdisk.
 >>
 >> OK, I think we should start with that.
 >
 > What do you think about that, Ana?

I'm fine having a single image with these two test suites, if eventually
this not convenient we can always change it. However, now we have more
or less agreed on a job creating the "basic" images, how do you envisage
we should be creating the images with the test suites?

My proposal is installing the build and run dependencies via extra_packages
in the Jenkinsfile and then building the test suite directly against the
new created image with script run by debos directly. This will avoid
the merge step, although the strip/crush might need a bit of work
for making sure the resulting ramdisk is still small.
The advantage is adding/modifying the test suites doesn't require a lot
of knowledge of jenkins or even debos.

[...]
 >> But I agree, for scalability, secondary storage is a major PITA.
 >> However, maybe for a subset of devices that we want to test that way, we
 >> should make it easy.
 >>
 >> So, what about this: From a single base debian image, we create
 >>
 >> 1) minimal (~2M) initramfs
 >>
 >> This should have pivot capabilities, and this is what Ana is doing today
 >> with 'update-initramfs -c -k min'
 >>
 >> 2) small "basic" ramdisk
 >>
 >> This is the base image *after* the strip/crush steps.  Basically, the
 >> "basic" that Ana is building today.
 >>
 >> 3) full rootfs for secondary storage (e.g. MMC, NFS, NBD)
 >>
 >> The base image without all the strip/crush removed, so it still has the
 >> ability to install packages etc.
 >>
 >> This should be available in a few formats: .cpio.gz, .tar.xz, .ext4.xz
 >>
 >> This would also be the pivot destination when using an initramfs.
 >
 > Sounds good!

Yup. I added the images listed in 3). So right now we have for every arch:
1) minimal initramfs: initrd.cpio.gz
2) small basic ramdisk:  rootfs.cpio.gz
3) full rootfs in several formats:  full.rootfs.cpio.gz ,
full.rootfs.tar.xz  and  rootfs.ext4.xz

Cheers,
Ana

Re: [kernelci] proposal to build Debian images for every test suite (pull #24/kernelci-build-staging)

[Tomeu Vizoso]

On 05/16/2018 06:46 PM, Ana Guerrero Lopez wrote:
> 
> 
> 
> On 11/05/18 07:23, Tomeu Vizoso wrote:
>  > On 05/10/2018 05:41 PM, Kevin Hilman wrote:
>  >> Tomeu Vizoso <tomeu.vizoso@collabora.com> writes:
>  >>
>  >>> On 05/10/2018 03:04 AM, Kevin Hilman wrote:
>  >>>> "Tomeu Vizoso" <tomeu.vizoso@collabora.com> writes:
>  >>>>
>  >>>>> Hi,
>  >>>>>
>  >>>>> below I give my opinion on a few comments, but it's Ana who is 
> leading
>  >>>>> now this work.
>  >>>>>
>  >>>>> On 05/08/2018 01:09 AM, Kevin Hilman wrote:
>  >>>>>> "Ana Guerrero Lopez" <ana.guerrero@collabora.com> writes:
>  >>>>>
>  >>>>>> IMO, what I think would be very helpful at least for initial 
> review and
>  >>>>>> discussion, is to see an initial PR that only has the "basic" 
> build, and
>  >>>>>> ideally also generates a minimal, tiny ramdisk from the same build
>  >>>>>> (e.g. with 'update-initramfs -c -k min' )
>  >>>>>>
>  >>>>>>> The pull requests includes three things: three jenkinsfiles, 
> debos files
>  >>>>>>> and two Dockerfiles.
>  >>>>>>>
>  >>>>>>> The jenkinsfiles are the smallest possible since all the code 
> creating
>  >>>>>>> the pipeline is in the shared library. There are two parts: one 
> with the
>  >>>>>>> job name - that will be used by the resulting images, the
>  >>>>>>> destination
>  >>>>> arches
>  >>>>>>> and the run-time dependencies that need to be added to the image.
>  >>>>>>> There is also the debos file name but this should be removed if 
> we always
>  >>>>>>> use the same debos configuration.
>  >>>>>>> The second part "build_test_suite" is for building the test 
> suite code.
>  >>>>>>> This is on purpose a shell script that must create a cpio.gz 
> tarball
>  >>>>>>> with the name rootfs-built-${NAME}-${ARCH}.cpio.gz
>  >>>>>>> The idea is to be able to add and modify quickly test suites 
> without
>  >>>>>>> knowing too much about jenkins.
>  >>>>>>
>  >>>>>> I'm not sure about the "build_test_suite" approach.
>  >>>>>>
>  >>>>>> AFAICT, both the _igt and _v4l2 jobs are basically doing the same 
> thing
>  >>>>>> as "basic", and then essentially installing a bunch more files on 
> top.
>  >>>>>
>  >>>>> The difference is only in the dependencies. Both test suites are in
>  >>>>> the fat side and have several dependencies that otherwise aren't
>  >>>>> needed. That said, a basic image that contains all of them might 
> still
>  >>>>> not be too big.
>  >>>>
>  >>>> IMO, it's better to go for a single, shared base image with
>  >>>> dependencies.  Building a new rootfs for every test suite sounds 
> like a
>  >>>> scalability problem to me.
>  >>>
>  >>> Well, only the fatter test suites would need their own rootfs. So far
>  >>> only IGT and V4L2 have a fair amount of dependencies. But still we
>  >>> could probably build a single image for both that isn't too big to be
>  >>> used as a ramdisk.
>  >>
>  >> OK, I think we should start with that.
>  >
>  > What do you think about that, Ana?
> 
> I'm fine having a single image with these two test suites, if eventually
> this not convenient we can always change it. However, now we have more
> or less agreed on a job creating the "basic" images, how do you envisage
> we should be creating the images with the test suites?
> 
> My proposal is installing the build and run dependencies via extra_packages
> in the Jenkinsfile and then building the test suite directly against the
> new created image with script run by debos directly. This will avoid
> the merge step, although the strip/crush might need a bit of work
> for making sure the resulting ramdisk is still small.
> The advantage is adding/modifying the test suites doesn't require a lot
> of knowledge of jenkins or even debos.

Sounds good to me, assuming that making sure that build dependencies 
aren't in the final image isn't cumbersome to those adding more images.

Thanks,

Tomeu

> [...]
>  >> But I agree, for scalability, secondary storage is a major PITA.
>  >> However, maybe for a subset of devices that we want to test that way, we
>  >> should make it easy.
>  >>
>  >> So, what about this: From a single base debian image, we create
>  >>
>  >> 1) minimal (~2M) initramfs
>  >>
>  >> This should have pivot capabilities, and this is what Ana is doing today
>  >> with 'update-initramfs -c -k min'
>  >>
>  >> 2) small "basic" ramdisk
>  >>
>  >> This is the base image *after* the strip/crush steps.  Basically, the
>  >> "basic" that Ana is building today.
>  >>
>  >> 3) full rootfs for secondary storage (e.g. MMC, NFS, NBD)
>  >>
>  >> The base image without all the strip/crush removed, so it still has the
>  >> ability to install packages etc.
>  >>
>  >> This should be available in a few formats: .cpio.gz, .tar.xz, .ext4.xz
>  >>
>  >> This would also be the pivot destination when using an initramfs.
>  >
>  > Sounds good!
> 
> Yup. I added the images listed in 3). So right now we have for every arch:
> 1) minimal initramfs: initrd.cpio.gz
> 2) small basic ramdisk:  rootfs.cpio.gz
> 3) full rootfs in several formats:  full.rootfs.cpio.gz ,
> full.rootfs.tar.xz  and  rootfs.ext4.xz
> 
> Cheers,
> Ana
> 
> 
> 
> 
> 
>