[cip-dev] Testing Dependencies

[cip-dev] Testing Dependencies

[Patryk Mungai Ndungu]

Hello,

Some dependencies will be required by some tests. As discussed in the TSC meeting today, there are a few options on how to do this:

1) All the package dependencies for all tests included in one CIP-Core filesystem
2) Install the dependency packages during the test (this will require a package manager in the RFS, and an internet connection to the device under test)
	- Static packages binaries could be stored on S3 bucket

There are 2 different types of dependencies. First, the test might require specific packages to be installed in the filesystem. Secondly, a test might require a test file/binary to run (eg. kselftests, ltp tests).
Linaro store static test binaries (such as cross-compiled kselftests) on a server, and download these during a test (in the test script). They also download and install packages (using apt-get) during the test.

Are there any preferences in which of the above options to use?
	- We have to bear in mind, not all labs may provide an internet connection to the boards

Are there any package managers that should be included in CIP-Core? (apt-get, pip, yum, dpkg etc.)

Any other comments?

Kind Regards,
Patryk

[cip-dev] Testing Dependencies

[daniel.sangorrin at toshiba.co.jp]

Hello Patryk,

> -----Original Message-----
> From: cip-dev-bounces at lists.cip-project.org <cip-dev-bounces@lists.cip-project.org> On Behalf Of Patryk
> Mungai Ndungu
> Sent: Tuesday, March 19, 2019 2:42 AM
> To: cip-dev at lists.cip-project.org
> Subject: [cip-dev] Testing Dependencies
> 
> Hello,
> 
> Some dependencies will be required by some tests. As discussed in the TSC meeting today, there are a few
> options on how to do this:
> 
> 1) All the package dependencies for all tests included in one CIP-Core filesystem

This can't always be achieved. For example, Deby v1 does not have many recipes.
Having said that, if Deby v2 becomes perfectly compatible with Yocto/OE, adding tests to the image will get easier.

> 2) Install the dependency packages during the test (this will require a package manager in the RFS, and an
> internet connection to the device under test)

This works well for the generic profile, but not for the tiny profile.

3)
> 	- Static packages binaries could be stored on S3 bucket

This option should work for both profiles.

- Other options:
4) Prepare a multi-node job where one node (actually a LXC container) contains Fuego. This is being developed by Linaro at the moment.
5) We can just use or create our own shell scripts for testing the basic behavior of most commands. There are a few test definitions in Fuego that can be easily adapted.

> There are 2 different types of dependencies. First, the test might require specific packages to be installed in the
> filesystem. Secondly, a test might require a test file/binary to run (eg. kselftests, ltp tests).
> Linaro store static test binaries (such as cross-compiled kselftests) on a server, and download these during a test
> (in the test script). They also download and install packages (using apt-get) during the test.
> 
> Are there any preferences in which of the above options to use?
> 	- We have to bear in mind, not all labs may provide an internet connection to the boards

I prefer to separate the test build step from the test run step.
The test build step can be a combination of:
- tests and dependencies installed in the image
- tests built using the SDK (cross-compiled)

> Are there any package managers that should be included in CIP-Core? (apt-get, pip, yum, dpkg etc.)

In ISAR, apt-get should be installed by default.
Deby v1 does not support apt-get. It supports dpkg, but almost a tar xzvf

> Any other comments?

Yes, I think we should clarify two things:

- What type of problems we want to find
  - Option 1: regressions. In this case, we can use always the same tests and test versions
  - Option 2: bugs. In this case, it is important to update the test version (e.g. run the latest LTP) continuously.
- What we want to test
  - Kernel regressions: LTP (including OpenPOSIX), kselftests, iperf3, IOZone, rt-tests, xfstests..
  - Kernel 0-day bugs: we need a fuzzy tool such as syzkaller
  - Black-box regression testing (busybox, findutils, etc..): custom shell scripts should be enough

In my opinion, we should focus on regression tests only.
#Although updating the LTP version from time to time might be important.

Thanks,
Daniel

[cip-dev] Testing Dependencies

[Patryk Mungai Ndungu]

Hello Daniel,

Thank you for your feedback.

> -----Original Message-----
> From: daniel.sangorrin at toshiba.co.jp <daniel.sangorrin@toshiba.co.jp>
> Sent: 19 March 2019 00:50
> To: Patryk Mungai Ndungu <patryk.mungai-ndungu.kx@renesas.com>; cip-
> dev at lists.cip-project.org
> Subject: RE: Testing Dependencies
> 
> Hello Patryk,
> 
> > -----Original Message-----
> > From: cip-dev-bounces at lists.cip-project.org <cip-dev-bounces@lists.cip-
> project.org> On Behalf Of Patryk
> > Mungai Ndungu
> > Sent: Tuesday, March 19, 2019 2:42 AM
> > To: cip-dev at lists.cip-project.org
> > Subject: [cip-dev] Testing Dependencies
> >
> > Hello,
> >
> > Some dependencies will be required by some tests. As discussed in the TSC
> meeting today, there are a few
> > options on how to do this:
> >
> > 1) All the package dependencies for all tests included in one CIP-Core
> filesystem
> 
> This can't always be achieved. For example, Deby v1 does not have many
> recipes.
> Having said that, if Deby v2 becomes perfectly compatible with Yocto/OE,
> adding tests to the image will get easier.
> 
> > 2) Install the dependency packages during the test (this will require a
> package manager in the RFS, and an
> > internet connection to the device under test)
> 
> This works well for the generic profile, but not for the tiny profile.
> 
> 3)
> > 	- Static packages binaries could be stored on S3 bucket
> 
> This option should work for both profiles.
> 

This makes sense to me too.

> - Other options:
> 4) Prepare a multi-node job where one node (actually a LXC container)
> contains Fuego. This is being developed by Linaro at the moment.

Perhaps we can migrate to this once we have more of the infrastructure and tests set up?
As you mention later, this looks to be the better option when it comes to testing for bugs.

> 5) We can just use or create our own shell scripts for testing the basic
> behavior of most commands. There are a few test definitions in Fuego that
> can be easily adapted.

I assume this is primarily for CIP-Core testing? In which case great! I think it would be nice to show CIP-Core tests in LAVA too.

> 
> > There are 2 different types of dependencies. First, the test might require
> specific packages to be installed in the
> > filesystem. Secondly, a test might require a test file/binary to run (eg.
> kselftests, ltp tests).
> > Linaro store static test binaries (such as cross-compiled kselftests) on a
> server, and download these during a test
> > (in the test script). They also download and install packages (using apt-get)
> during the test.
> >
> > Are there any preferences in which of the above options to use?
> > 	- We have to bear in mind, not all labs may provide an internet
> connection to the boards
> 
> I prefer to separate the test build step from the test run step.
> The test build step can be a combination of:
> - tests and dependencies installed in the image
> - tests built using the SDK (cross-compiled)
> 

Makes sense.

> > Are there any package managers that should be included in CIP-Core? (apt-
> get, pip, yum, dpkg etc.)
> 
> In ISAR, apt-get should be installed by default.
> Deby v1 does not support apt-get. It supports dpkg, but almost a tar xzvf

Okay. If/when the Renesas iwg20m board is supported in the ISAR build (CIP-Core generic profile?), I'll switch to using that image.

> 
> > Any other comments?
> 
> Yes, I think we should clarify two things:
> 
> - What type of problems we want to find
>   - Option 1: regressions. In this case, we can use always the same tests and
> test versions
>   - Option 2: bugs. In this case, it is important to update the test version (e.g.
> run the latest LTP) continuously.
> - What we want to test
>   - Kernel regressions: LTP (including OpenPOSIX), kselftests, iperf3, IOZone,
> rt-tests, xfstests..
>   - Kernel 0-day bugs: we need a fuzzy tool such as syzkaller
>   - Black-box regression testing (busybox, findutils, etc..): custom shell scripts
> should be enough
> 
> In my opinion, we should focus on regression tests only.
> #Although updating the LTP version from time to time might be important.

I agree. At least for now, before we have the CI build system setup.

So to summarise:
- Images should have all tools dependencies installed during build
- If not, it would be helpful if CIP-Core images have apt-get installed (use generic profile built with ISAR)
- Any binaries for tests should be stored in the S3 bucket
- As our focus is regression testing, these binaries should remain static
- We can look into running Kernel 0-day bug tests and multi-node jobs later

Thanks,
Patryk

> 
> Thanks,
> Daniel

[cip-dev] Testing Dependencies

[daniel.sangorrin at toshiba.co.jp]

> -----Original Message-----
> From: Patryk Mungai Ndungu <patryk.mungai-ndungu.kx@renesas.com>
> > 5) We can just use or create our own shell scripts for testing the basic
> > behavior of most commands. There are a few test definitions in Fuego that
> > can be easily adapted.
> 
> I assume this is primarily for CIP-Core testing? In which case great! I think it would be nice to show CIP-Core tests
> in LAVA too.

Yes, that's right.

> So to summarise:
> - Images should have all tools dependencies installed during build
> - If not, it would be helpful if CIP-Core images have apt-get installed (use generic profile built with ISAR)
> - Any binaries for tests should be stored in the S3 bucket
> - As our focus is regression testing, these binaries should remain static
> - We can look into running Kernel 0-day bug tests and multi-node jobs later

Well summarized.

Thanks,
Daniel

> 
> Thanks,
> Patryk
> 
> >
> > Thanks,
> > Daniel