Questions to CI Testing the RT-Kernel

Questions to CI Testing the RT-Kernel

[Gylstorff Quirin]

Dear all,

we are currently looking to setup a test lab based on LAVA[1] for 
CI-testing the RT-Kernel.

Kernelci executes the combination cyclictest with hackbench in the 
background[2].

Our questions are:

 From what I found the RT-Project also uses the combination of 
cyclictest with hackbench for testing or are there currently any other 
tests specifically for real-time executed?

Are there recommended tests that should be added to the CI testing of a 
RT-Kernel?

[1]: https://docs.lavasoftware.org/lava/index.html
[2]: 
https://github.com/kernelci/kernelci-core/blob/main/config/lava/preempt-rt/preempt-rt.jinja2

-- 

With best regards,
Quirin Gylstorff

Siemens AG

Re: Questions to CI Testing the RT-Kernel

[John Kacur]

There is a program called rteval

It uses cyclictest as a measurement module.
It uses hackbench as a load module, and it also compiles the kernel as
a load module.
Finally it can uses stress-ng as an optional load module - the support
for that is fairly new and developing.

git clone git://git.kernel.org/pub/scm/utils/rteval/rteval.git
To run from git: su c './rteval cmd'

You can also get it in Fedora with dnf

John

On Mon, 8 Nov 2021 at 08:52, Gylstorff Quirin
<quirin.gylstorff@siemens.com> wrote:
>
> Dear all,
>
> we are currently looking to setup a test lab based on LAVA[1] for
> CI-testing the RT-Kernel.
>
> Kernelci executes the combination cyclictest with hackbench in the
> background[2].
>
> Our questions are:
>
>  From what I found the RT-Project also uses the combination of
> cyclictest with hackbench for testing or are there currently any other
> tests specifically for real-time executed?
>
> Are there recommended tests that should be added to the CI testing of a
> RT-Kernel?
>
> [1]: https://docs.lavasoftware.org/lava/index.html
> [2]:
> https://github.com/kernelci/kernelci-core/blob/main/config/lava/preempt-rt/preempt-rt.jinja2
>
> --
>
> With best regards,
> Quirin Gylstorff
>
> Siemens AG

Re: Questions to CI Testing the RT-Kernel

[Daniel Wagner]

Hi Quirin,

On Mon, Nov 08, 2021 at 11:28:15AM +0100, Gylstorff Quirin wrote:
> From what I found the RT-Project also uses the combination of cyclictest
> with hackbench for testing or are there currently any other tests
> specifically for real-time executed?

I run most of the rt-tests with some background load. Usually it's a
hackbench. Since recently, I let run stress-ng (disabling a bunch of
stressors which trigger OOM situation for example).

For your CI setup, I would recommend trying to get real workloads from
your projects. hackbench is a very artificial workload for and only
exercises a small part of the kernel.

> Are there recommended tests that should be added to the CI testing of a
> RT-Kernel?

As mentioned, rt-tests is a good start. IIRC, ltp has also some RT tests
but I don't know their exact status. We (SUSE) did recently review
the ltp tests and most of them are covered by rt-tests, IIRC.

BTW, a bunch of links you might be interested if you are planing
to use LAVA:

https://github.com/kernelci/kernelci-core/issues/474
https://github.com/Linaro/test-definitions/pull/85
https://github.com/Linaro/test-definitions/pull/201

HTH,
Daniel

Re: Questions to CI Testing the RT-Kernel

[Michael Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 1226 bytes --]

On 08/11/2021 10:28, Gylstorff Quirin wrote:
> Dear all,
> 
> we are currently looking to setup a test lab based on LAVA[1] for CI-testing the
> RT-Kernel.
> 
> Kernelci executes the combination cyclictest with hackbench in the background[2].
> 
> Our questions are:
> 
> From what I found the RT-Project also uses the combination of cyclictest with
> hackbench for testing or are there currently any other tests specifically for
> real-time executed?
> 
> Are there recommended tests that should be added to the CI testing of a RT-Kernel?
> 
> [1]: https://docs.lavasoftware.org/lava/index.html
> [2]:
> https://github.com/kernelci/kernelci-core/blob/main/config/lava/preempt-rt/preempt-rt.jinja2
> 
> 
Probably worthwhile contacting the KernelCI project
(https://foundation.kernelci.org/) who I see are quite active (I shadow the IRC
channel as I know two members personally).

I believe they have a central 'dashboard' where you can submit your test results,
which may prove valuable to other users (either via KCIDB or other methods, see below).

https://foundation.kernelci.org/blog/2020/11/05/notes-from-oss-elc-europe-2020/ is
quite an interesting article on the roadmap of the project.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Questions to CI Testing the RT-Kernel

[Sebastian Andrzej Siewior]

On 2021-11-08 11:28:15 [+0100], Gylstorff Quirin wrote:
> Dear all,
Hi,

> Our questions are:
> 
> From what I found the RT-Project also uses the combination of cyclictest
> with hackbench for testing or are there currently any other tests
> specifically for real-time executed?
> 
> Are there recommended tests that should be added to the CI testing of a
> RT-Kernel?

In general you want to have two categories:
- A debug run with lots of debug switches enabled like lockdep,
  debug-objects and what not.

- A "production" run where all of those expensive debug swiches are off.
  That "sleeping while atomic" is cheap and could remain enabled.

In these two types of config you want to run tests. As mentioned here
hackbench, stress-ng, … is a good start. You want also make sure that
you "use" all of your hardware that could be used in "production"
including disk, i2c, gpio, … and what else might be there.

While doing this, you don't want see any warnings in your debug config.
In your production config you want also to measure the responsibility/
latency of the system. cyclictest is good and simple but depending on
your usecase it might also include the latency of IPC communication,
ping-ping of network communication, etc.

Sebastian