[ANNOUNCE v2] ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

[ANNOUNCE v2] ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

[Steven Rostedt]

I previously posted the announcement about ktest (and shamelessly
promoted it at both Kernel Summit and Linux Plumbers), but I just
updated it with a new feature.

  TEST_TYPE = config_bisect

(See http://marc.info/?l=linux-kernel&m=128829496215347 for original
post)

With the new config_bisect, the config is tested in halves. To handle
strange dependencies, the following is done.

First the "bad-config" needs to be prepared for the current working git
tree. As with all other tests, a "min-config" may be specified and that
will be override any config settings in the "bad-config". Since this may
produce new settings to be set (or removed) some preparation needs to be
done. This is done automatically with ktest:

1) overrides all configs with what is in the MIN_CONFIG or ADD_CONFIG
settings.

2) Runs the result through a make oldconfig to see if what may have been
added or removed (it reports these).

3) creates what is about to be tested and again runs the make oldconfig
to see if anything else might have been removed by dependencies.

All of the above changes are reported by the tool.


Finally, it gets into the bisect. It starts with a list of configs to
test. It cuts it in half, and then creates a .config file, and runs that
through oldconfig. If none of the configs in the list are enabled
(because they all depend on configs in the other half) then the second
half of the list is tried. If there's still no configs set, there must
be a circular dependency and the test fails here (but this should never
happen).

It compiles, and performs the type of test specified by the
CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all
the configs that were enabled in the .config are removed from the
configs to test and added to the good configs to always enable. This
allows for configs in the other half to be enabled if they depended on
these configs.

If the test fails, then we have a new subset of configs that is known to
cause the failure. Any config that was not set in the .config for this
run is removed from the test and will not be enabled again.

This iterates until we have one config left and that config will be
reported.

Because of dependencies and selects, the config reported may not be the
only config to cause the error. If you find that you can still produce
an error after disabling the config it reports, then simply disable it
and run the bisect again. Most likely it will turn up the problem. I may
automate this part in the future.

Anyway, you can download the latest ktest from:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ktest.git

I need to write up a man page and perhaps more documentation. The
sample.conf that comes in that repo has loads of information on how to
do the tests.


Have fun!

-- Steve

Re: [ANNOUNCE v2] ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

[Jon Masters]

On Mon, 2010-11-08 at 18:45 -0500, Steven Rostedt wrote:

> It compiles, and performs the type of test specified by the
> CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all
> the configs that were enabled in the .config are removed from the
> configs to test and added to the good configs to always enable. This
> allows for configs in the other half to be enabled if they depended on
> these configs.

Haven't had time to play, but I was thinking about this when bisecting
an i915 bug last week. In that case none of the kernels failed to boot,
they just had busted modelines. Do you have support in this tool for
testing some condition on the system after it boots? I would guess I
probably would have needed something more heavyweight (that could run a
script to look for busted modes on the target after booting).

Jon.

Re: [ANNOUNCE v2] ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

[Steven Rostedt]

On Thu, 2010-11-18 at 02:03 -0500, Jon Masters wrote:
> On Mon, 2010-11-08 at 18:45 -0500, Steven Rostedt wrote:
> 
> > It compiles, and performs the type of test specified by the
> > CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all
> > the configs that were enabled in the .config are removed from the
> > configs to test and added to the good configs to always enable. This
> > allows for configs in the other half to be enabled if they depended on
> > these configs.
> 
> Haven't had time to play, but I was thinking about this when bisecting
> an i915 bug last week. In that case none of the kernels failed to boot,
> they just had busted modelines. Do you have support in this tool for
> testing some condition on the system after it boots? I would guess I
> probably would have needed something more heavyweight (that could run a
> script to look for busted modes on the target after booting).

If you set:

BISECT_TYPE = test
TEST = ssh user@host test_script

Where the test_script resides on the test target, and will exit with 0
on success and non-0 otherwise, it should work.

I could also change the code to give you access to the bootlog (actually
you already have it, but it's in the temp directory).

-- Steve

Re: [ANNOUNCE v2] ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

[Steven Rostedt]

Question:

Has anyone tried this out? Does anyone feel that I should get this ready
to be included in 2.6.38 tools directory?

Just want to know if I should go ahead and prepare it, or if it will
just be a waste of my time.

Thanks,

-- Steve


On Mon, 2010-11-08 at 18:45 -0500, Steven Rostedt wrote:
> I previously posted the announcement about ktest (and shamelessly
> promoted it at both Kernel Summit and Linux Plumbers), but I just
> updated it with a new feature.
> 
>   TEST_TYPE = config_bisect
> 
> (See http://marc.info/?l=linux-kernel&m=128829496215347 for original
> post)
> 
> With the new config_bisect, the config is tested in halves. To handle
> strange dependencies, the following is done.
> 
> First the "bad-config" needs to be prepared for the current working git
> tree. As with all other tests, a "min-config" may be specified and that
> will be override any config settings in the "bad-config". Since this may
> produce new settings to be set (or removed) some preparation needs to be
> done. This is done automatically with ktest:
> 
> 1) overrides all configs with what is in the MIN_CONFIG or ADD_CONFIG
> settings.
> 
> 2) Runs the result through a make oldconfig to see if what may have been
> added or removed (it reports these).
> 
> 3) creates what is about to be tested and again runs the make oldconfig
> to see if anything else might have been removed by dependencies.
> 
> All of the above changes are reported by the tool.
> 
> 
> Finally, it gets into the bisect. It starts with a list of configs to
> test. It cuts it in half, and then creates a .config file, and runs that
> through oldconfig. If none of the configs in the list are enabled
> (because they all depend on configs in the other half) then the second
> half of the list is tried. If there's still no configs set, there must
> be a circular dependency and the test fails here (but this should never
> happen).
> 
> It compiles, and performs the type of test specified by the
> CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all
> the configs that were enabled in the .config are removed from the
> configs to test and added to the good configs to always enable. This
> allows for configs in the other half to be enabled if they depended on
> these configs.
> 
> If the test fails, then we have a new subset of configs that is known to
> cause the failure. Any config that was not set in the .config for this
> run is removed from the test and will not be enabled again.
> 
> This iterates until we have one config left and that config will be
> reported.
> 
> Because of dependencies and selects, the config reported may not be the
> only config to cause the error. If you find that you can still produce
> an error after disabling the config it reports, then simply disable it
> and run the bisect again. Most likely it will turn up the problem. I may
> automate this part in the future.
> 
> Anyway, you can download the latest ktest from:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ktest.git
> 
> I need to write up a man page and perhaps more documentation. The
> sample.conf that comes in that repo has loads of information on how to
> do the tests.
> 
> 
> Have fun!
> 
> -- Steve
> 

Re: [ANNOUNCE v2] ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

[Vivek Goyal]

On Thu, Nov 18, 2010 at 09:08:37AM -0500, Steven Rostedt wrote:
> Question:
> 
> Has anyone tried this out? Does anyone feel that I should get this ready
> to be included in 2.6.38 tools directory?
> 
> Just want to know if I should go ahead and prepare it, or if it will
> just be a waste of my time.

I had done some testing only on initial internal version but did not
get a chance to do more testing later.

But I believe it is worth to put effort to keep this script going. I am
especially interested in part where I can just automate compilation and
testing of a patch on a remote machine.

Thanks
Vivek

> 
> Thanks,
> 
> -- Steve
> 
> 
> On Mon, 2010-11-08 at 18:45 -0500, Steven Rostedt wrote:
> > I previously posted the announcement about ktest (and shamelessly
> > promoted it at both Kernel Summit and Linux Plumbers), but I just
> > updated it with a new feature.
> > 
> >   TEST_TYPE = config_bisect
> > 
> > (See http://marc.info/?l=linux-kernel&m=128829496215347 for original
> > post)
> > 
> > With the new config_bisect, the config is tested in halves. To handle
> > strange dependencies, the following is done.
> > 
> > First the "bad-config" needs to be prepared for the current working git
> > tree. As with all other tests, a "min-config" may be specified and that
> > will be override any config settings in the "bad-config". Since this may
> > produce new settings to be set (or removed) some preparation needs to be
> > done. This is done automatically with ktest:
> > 
> > 1) overrides all configs with what is in the MIN_CONFIG or ADD_CONFIG
> > settings.
> > 
> > 2) Runs the result through a make oldconfig to see if what may have been
> > added or removed (it reports these).
> > 
> > 3) creates what is about to be tested and again runs the make oldconfig
> > to see if anything else might have been removed by dependencies.
> > 
> > All of the above changes are reported by the tool.
> > 
> > 
> > Finally, it gets into the bisect. It starts with a list of configs to
> > test. It cuts it in half, and then creates a .config file, and runs that
> > through oldconfig. If none of the configs in the list are enabled
> > (because they all depend on configs in the other half) then the second
> > half of the list is tried. If there's still no configs set, there must
> > be a circular dependency and the test fails here (but this should never
> > happen).
> > 
> > It compiles, and performs the type of test specified by the
> > CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all
> > the configs that were enabled in the .config are removed from the
> > configs to test and added to the good configs to always enable. This
> > allows for configs in the other half to be enabled if they depended on
> > these configs.
> > 
> > If the test fails, then we have a new subset of configs that is known to
> > cause the failure. Any config that was not set in the .config for this
> > run is removed from the test and will not be enabled again.
> > 
> > This iterates until we have one config left and that config will be
> > reported.
> > 
> > Because of dependencies and selects, the config reported may not be the
> > only config to cause the error. If you find that you can still produce
> > an error after disabling the config it reports, then simply disable it
> > and run the bisect again. Most likely it will turn up the problem. I may
> > automate this part in the future.
> > 
> > Anyway, you can download the latest ktest from:
> > 
> > git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ktest.git
> > 
> > I need to write up a man page and perhaps more documentation. The
> > sample.conf that comes in that repo has loads of information on how to
> > do the tests.
> > 
> > 
> > Have fun!
> > 
> > -- Steve
> > 
> 

Re: [ANNOUNCE v2] ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

[Steven Rostedt]

[ Finally remembered to remove that annoying autotest@test.kernel.org
  from the Cc list ]

On Thu, 2010-11-18 at 09:26 -0500, Vivek Goyal wrote:
> On Thu, Nov 18, 2010 at 09:08:37AM -0500, Steven Rostedt wrote:
> > Question:
> > 
> > Has anyone tried this out? Does anyone feel that I should get this ready
> > to be included in 2.6.38 tools directory?
> > 
> > Just want to know if I should go ahead and prepare it, or if it will
> > just be a waste of my time.
> 
> I had done some testing only on initial internal version but did not
> get a chance to do more testing later.
> 
> But I believe it is worth to put effort to keep this script going. I am
> especially interested in part where I can just automate compilation and
> testing of a patch on a remote machine.

Thanks for the reply! I'll start setting up a git tree that places all
the changes into the tools directory.

-- Steve