Re: automated test? (was Re: Linux 2.6.17.7)

[Andrew de Quincey <adq_dvb@lidskialf.net>]

On Wednesday 26 July 2006 14:02, Adrian Bunk wrote:
> On Tue, Jul 25, 2006 at 09:47:43AM -0700, David Lang wrote:
> > On Tue, 25 Jul 2006, Andrew de Quincey wrote:
> > >On Tuesday 25 July 2006 10:55, Arnaud Patard wrote:
> > >>Greg KH <gregkh@suse.de> writes:
> > >>
> > >>Hi,
> > >>
> > >>>We (the -stable team) are announcing the release of the 2.6.17.7
> > >>> kernel.
> > >>
> > >>Sorry, but doesn't compile if DVB_BUDGET_AV is set :(
> > >>
> > >>>Andrew de Quincey:
> > >>>      v4l/dvb: Fix budget-av frontend detection
> > >
> > >In fact it is just this patch causing the problem:
> >
> > <SNIP>
> >
> > >Sorry, I had so much work going on in that area I must have diffed the
> > >wrong
> > >kernel when I created this patch. :(
> >
> > is it reasonable to have an aotomated test figure out what config options
> > are relavent to a patch (or patchset) and test compile all the
> > combinations to catch this sort of mistake?
>
> If you think about it, you'll notice it's definitely not reasonable:
>
> #include <linux/module.h> brings you a dependency on 5 config options.
> #include <linux/pci.h> brings you a dependency on 6 config options.
>
> By only including these two headers you are at 2048 combinations.
> The number of valid configurations will be lower, but 500 test compiles
> sound realistically.
>
> With have a dozen #include's you might need more than a million test
> compiles.
>
> With a dozen #include's, you might need a trilion [1] test compiles.
>
>
> Compile errors are quickly catched and don't cause any serious problem.
>
> What bothers me more is that noone tested this patch against the kernel
> it was applied against.
>
> The submitter didn't test it works (he didn't even test the compilation).

Yes I did - I didn't test the final generated patch unfortunately since I 
assumed it worked. The kernel I _meant_ to diff against worked perfectly :(