[LTP] runltp-ng and possible ltp-pan deprecation

[LTP] runltp-ng and possible ltp-pan deprecation

[Cyril Hrubis]

Hi!
As you may have know some time ago I have started to work on runltp-ng a
script to replace runltp and ltp-pan.

The new executor is different in several key aspects, for example:

- it can and is supposed to run on a different machine from the system
  under test (although it still supports running on the same machine)

- there are different backends, it can run tests locally, over ssh, or
  start a VM with qemu

- produces different output formats currently html or json

We have been using runltp-ng in SUSE for a while and it works for us
quite well and I think that we need a larger user base now to iron out
all the quirks. So unless someone strongly disagrees I think that it's
time to make it officialy supported test executor for LTP.

I would like to eventually get rid of runltp and ltp-pan, but if that
happens it should be done once everyone switched over to a different
solution and it's probably still at least a year or two from now.

To remind you the code still lives in my private repository at:

https://github.com/metan-ucw/runltp-ng

For a start I would like to move the code somewhere to the
linux-test-project github just to make it more visible. Does everyone
agree to that?

Also there are two different options how to do that. Either we put it
into runltp-ng directory to the LTP git root or we may as well create a
separate repository under the linux-test-project and make it a ltp git
submodule. Any preferencies to that?

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] runltp-ng and possible ltp-pan deprecation

[Tim.Bird]

> -----Original Message-----
> From: ltp <ltp-bounces+tim.bird=sony.com@lists.linux.it> On Behalf Of Cyril Hrubis
> 
> Hi!
> As you may have know some time ago I have started to work on runltp-ng a
> script to replace runltp and ltp-pan.
> 
> The new executor is different in several key aspects, for example:
> 
> - it can and is supposed to run on a different machine from the system
>   under test (although it still supports running on the same machine)
> 
> - there are different backends, it can run tests locally, over ssh, or
>   start a VM with qemu
> 
> - produces different output formats currently html or json
> 
> We have been using runltp-ng in SUSE for a while and it works for us
> quite well and I think that we need a larger user base now to iron out
> all the quirks. So unless someone strongly disagrees I think that it's
> time to make it officialy supported test executor for LTP.
> 
> I would like to eventually get rid of runltp and ltp-pan, but if that
> happens it should be done once everyone switched over to a different
> solution and it's probably still at least a year or two from now.
> 
> To remind you the code still lives in my private repository at:
> 
> https://github.com/metan-ucw/runltp-ng
> 
> For a start I would like to move the code somewhere to the
> linux-test-project github just to make it more visible. Does everyone
> agree to that?

Sounds great to me!  I'm very interested in the new functionality.

It sounds like I might be interested in adding a backend for running
tests over a Fuego transport.  Fuego mostly ends up using
ssh, but it provides other transports as well, so I'll be
interested to see what your backend API for test-running looks like.

> 
> Also there are two different options how to do that. Either we put it
> into runltp-ng directory to the LTP git root or we may as well create a
> separate repository under the linux-test-project and make it a ltp git
> submodule. Any preferencies to that?

If this is intended to be run with LTP only, then my preference would be to
put it into the LTP repository.
 -- Tim

[LTP] runltp-ng and possible ltp-pan deprecation

[Cyril Hrubis]

Hi!
> > For a start I would like to move the code somewhere to the
> > linux-test-project github just to make it more visible. Does everyone
> > agree to that?
> 
> Sounds great to me!  I'm very interested in the new functionality.
> 
> It sounds like I might be interested in adding a backend for running
> tests over a Fuego transport.  Fuego mostly ends up using
> ssh, but it provides other transports as well, so I'll be
> interested to see what your backend API for test-running looks like.

Backend is basically implemented with these functions:

* foo_init() that parses backend specific parameters
             and prepares a handle that is used later on

* foo_start() that boots machine, connects to SUT, ...

	      This gets SUT into a state where we can execute commands
	      on (remote) shell and stores pair of file descriptors to
	      the backend handle

	      for qemu it boots the machine and logs in

	      for ssh it connects to remote machine

	      for sh (local execution) it just forks shell

* foo_stop() that does the oposite of foo_start()

             Closes connection, poweroff qemu OS, etc.

* foo_force_stop() that attempts to force stop the machine after it
                   stopped responding so that we can bring it up again
		   and continue with next test

		   for qemu it just kills the process, we do have
		   overlay for the disks in place, so we always start
		   with a clean state

		   for ssh it's expected that user supplies a script

		     - when I was testing the test runner I had a script
		       that toggled relay to reset the machine

		     - for public cloud testing we pass a short script
		       that requests the machine to be rebooted


If machine crashes we do force_stop() followed by a start() to bring it
back to a defined state.

Everything else is build on a top of these functions. The testcases are
executed using a (remote) shell and the output is parsed, which is not
optimal for a several reasons and may change later on. I guess that the
general API will not change in this case, the difference would be that
once machine starts we would start a binary and talk to it via the file
descriptors instead of shell.

We are also working on parallel test execution, which will require such
binary running on the SUT that will dispatch testcases and report
results, but that haven't been finished yet.

Also in the short term we would like to make use of the metadata.json
produced by LTP build to make smarter decisions, probably easiest and
quite usefull would be adjusting timeout expectations. LTP generally
consists of two types of tests, most of the syscall test are fast and
take only miliseconds, then there are tests that run for minutes or even
scale its runtime based on test timeout. And this is all multiplied by
tests that do several iterations, for instance for each filesystem. I
really need to sit down and think how all this should work toghether.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] runltp-ng and possible ltp-pan deprecation

[Petr Vorel]

Hi Cyril,

Thank you for bringing the discussion to upstream runltp-ng.

I'd prefer to add the project to ltp git, in runltp-ng/ in root directory.
And add either a symlink runltp-ng/runltp-ng to the root directory or have a
shell script which runs it:
perl -I$(readlink -f $0)/runltp-ng $(readlink -f $0)/runltp-ng/runltp-ng $@

Kind regards,
Petr

[LTP] runltp-ng and possible ltp-pan deprecation

[Xiao Yang]

On 4/22/21 4:30 PM, Petr Vorel wrote:
> Hi Cyril,
>
> Thank you for bringing the discussion to upstream runltp-ng.
>
> I'd prefer to add the project to ltp git, in runltp-ng/ in root directory.

+1

I also prefer to add it into ltp directly.

Best Regards,

Xiao Yang

> And add either a symlink runltp-ng/runltp-ng to the root directory or have a
> shell script which runs it:
> perl -I$(readlink -f $0)/runltp-ng $(readlink -f $0)/runltp-ng/runltp-ng $@
>
> Kind regards,
> Petr
>
>

[LTP] runltp-ng and possible ltp-pan deprecation

[Li Wang]

Hi Cyril,

Cyril Hrubis <chrubis@suse.cz> wrote:

>
> Hi!
> As you may have know some time ago I have started to work on runltp-ng a
> script to replace runltp and ltp-pan.
>
> The new executor is different in several key aspects, for example:
>
> - it can and is supposed to run on a different machine from the system
>   under test (although it still supports running on the same machine)
>
> - there are different backends, it can run tests locally, over ssh, or
>   start a VM with qemu
>
> - produces different output formats currently html or json
>

Very nice features. Thanks!

>
> Also there are two different options how to do that. Either we put it
> into runltp-ng directory to the LTP git root or we may as well create a
> separate repository under the linux-test-project and make it a ltp git
> submodule. Any preferencies to that?
>

I prefer the LTP git root as well.

Btw, I opened a PR for some tiny issues fixing, plz help review.
https://github.com/metan-ucw/runltp-ng/pull/28

-- 
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210426/57080db9/attachment.htm>

[LTP] runltp-ng and possible ltp-pan deprecation

[Anders Roxell]

On Thu, 22 Apr 2021 at 10:30, Petr Vorel <pvorel@suse.cz> wrote:
>
> Hi Cyril,
>
> Thank you for bringing the discussion to upstream runltp-ng.
>
> I'd prefer to add the project to ltp git

I agree with that too.
Also if we can seamlessly move from runltp to runltp-ng, since we
don't want the behavior to change.
Or we need to support both for a release or two and then remove runltp.


Cheers,
Anders

>, in runltp-ng/ in root directory.
> And add either a symlink runltp-ng/runltp-ng to the root directory or have a
> shell script which runs it:
> perl -I$(readlink -f $0)/runltp-ng $(readlink -f $0)/runltp-ng/runltp-ng $@
>
> Kind regards,
> Petr
>
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] runltp-ng and possible ltp-pan deprecation

[Cyril Hrubis]

Hi!
> Also if we can seamlessly move from runltp to runltp-ng, since we
> don't want the behavior to change.
> Or we need to support both for a release or two and then remove runltp.

Do not worry, I do plan to carry both for more than one release in order
to give everyone enough time for the transition.

-- 
Cyril Hrubis
chrubis@suse.cz