[PATCH 1/8] ktest: Clarify config file usage

[PATCH 1/8] ktest: Clarify config file usage

[Scott Wood]

Simply telling a new user to edit "the config file" without giving any
hints on where that file should go, what it should be named, or where
a template can be found, is not particularly helpful.

Signed-off-by: Scott Wood <swood@redhat.com>
---
 tools/testing/ktest/ktest.pl    | 3 ++-
 tools/testing/ktest/sample.conf | 3 +++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 0c8b61f8398e..cdf6974bc462 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -1140,7 +1140,8 @@ sub __read_config {
 sub get_test_case {
 	print "What test case would you like to run?\n";
 	print " (build, install or boot)\n";
-	print " Other tests are available but require editing the config file\n";
+	print " Other tests are available but require editing ktest.conf\n";
+	print " (see tools/testing/ktest/sample.conf)\n";
 	my $ans = <STDIN>;
 	chomp $ans;
 	$default{"TEST_TYPE"} = $ans;
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 6c58cd8bbbae..7636236b87d3 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -1,6 +1,9 @@
 #
 # Config file for ktest.pl
 #
+# Place your customized version of this, named ktest.conf, in the
+# working directory that ktest.pl is run from.
+#
 # Note, all paths must be absolute
 #
 
-- 
2.9.4

[PATCH 2/8] ktest: Set buildonly=1 for CONFIG_BISECT_TYPE=build

[Scott Wood]

Rather than adding a third copy of the same logic, rework it to cover
all three buildonly cases at once.

In the future, please consider using the same variable to perform the
same function regardless of context...

Signed-off-by: Scott Wood <swood@redhat.com>
---
 tools/testing/ktest/ktest.pl | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index cdf6974bc462..36e1b91259cd 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -718,22 +718,14 @@ sub set_value {
 
     my $prvalue = process_variables($rvalue);
 
-    if ($buildonly && $lvalue =~ /^TEST_TYPE(\[.*\])?$/ && $prvalue ne "build") {
+    if ($lvalue =~ /^(TEST|BISECT|CONFIG_BISECT)_TYPE(\[.*\])?$/ &&
+	$prvalue !~ /^(config_|)bisect$/ &&
+	$prvalue !~ /^build$/ &&
+	$buildonly) {
+
 	# Note if a test is something other than build, then we
 	# will need other mandatory options.
 	if ($prvalue ne "install") {
-	    # for bisect, we need to check BISECT_TYPE
-	    if ($prvalue ne "bisect") {
-		$buildonly = 0;
-	    }
-	} else {
-	    # install still limits some mandatory options.
-	    $buildonly = 2;
-	}
-    }
-
-    if ($buildonly && $lvalue =~ /^BISECT_TYPE(\[.*\])?$/ && $prvalue ne "build") {
-	if ($prvalue ne "install") {
 	    $buildonly = 0;
 	} else {
 	    # install still limits some mandatory options.
-- 
2.9.4

[PATCH 3/8] ktest: Set do_not_reboot=y for CONFIG_BISECT_TYPE=build

[Scott Wood]

Currently setting do_not_reboot is triggered by simple builds and bisect
builds, but not config bisect builds.

Signed-off-by: Scott Wood <swood@redhat.com>
---
 tools/testing/ktest/ktest.pl | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 36e1b91259cd..7d537bdf462b 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -1397,7 +1397,8 @@ sub do_not_reboot {
 
     return $test_type eq "build" || $no_reboot ||
 	($test_type eq "patchcheck" && $opt{"PATCHCHECK_TYPE[$i]"} eq "build") ||
-	($test_type eq "bisect" && $opt{"BISECT_TYPE[$i]"} eq "build");
+	($test_type eq "bisect" && $opt{"BISECT_TYPE[$i]"} eq "build") ||
+	($test_type eq "config_bisect" && $opt{"CONFIG_BISECT_TYPE[$i]"} eq "build");
 }
 
 sub dodie {
-- 
2.9.4

[PATCH 4/8] ktest: Separate out config bisect logic

[Scott Wood]

From: Scott Wood <oss@buserror.net>

This allows config bisecting to be performed when building on the machine
being tested -- all required state is saved in the filesystem rather than
requiring one script instance to run for the duration of the bisecting.

Plus, while ktest appears very featureful, it is also a bit complicated
to set up, which can be a barrier to accessing the config bisect
functionality.  Separating it is more in line with traditional Unix
philosophy of small tools working together rather than a big monolith.

Signed-off-by: Scott Wood <swood@redhat.com>
---
 tools/testing/ktest/config-bisect.pl | 280 +++++++++++++++++++++++++++++++++++
 1 file changed, 280 insertions(+)
 create mode 100755 tools/testing/ktest/config-bisect.pl

diff --git a/tools/testing/ktest/config-bisect.pl b/tools/testing/ktest/config-bisect.pl
new file mode 100755
index 000000000000..1768d145ced8
--- /dev/null
+++ b/tools/testing/ktest/config-bisect.pl
@@ -0,0 +1,280 @@
+#!/usr/bin/perl -w
+#
+# Copyright 2010 - Steven Rostedt <srostedt@redhat.com>, Red Hat Inc.
+# Licensed under the terms of the GNU GPL License version 2
+#
+
+use strict;
+
+my $outputdir;
+
+sub make_oldconfig {
+	if (system "\$MAKE O=$outputdir olddefconfig") {
+		# Perhaps olddefconfig doesn't exist in this version of
+		# the kernel; try oldnoconfig
+		print "olddefconfig failed, trying make oldnoconfig\n";
+		if (system "\$MAKE O=$outputdir oldnoconfig") {
+			print "oldnoconfig failed, trying yes '' | make oldconfig\n";
+			# try a yes '' | oldconfig
+			system "yes '' | \$MAKE O=$outputdir oldconfig" == 0 or
+				die "failed make config oldconfig";
+		}
+	}
+}
+
+sub assign_configs {
+	my ($hash, $config) = @_;
+
+	print "Reading configs from $config\n";
+
+	open (IN, $config)
+		or die "Failed to read $config";
+
+	while (<IN>) {
+		chomp;
+		if (/^((CONFIG\S*)=.*)/) {
+			${$hash}{$2} = $1;
+		} elsif (/^(# (CONFIG\S*) is not set)/) {
+			${$hash}{$2} = $1;
+		}
+	}
+
+	close(IN);
+}
+
+sub save_config {
+	my ($pc, $file) = @_;
+
+	my %configs = %{$pc};
+
+	print "Saving configs into $file\n";
+
+	open(OUT, ">$file") or die "Can not write to $file";
+
+	foreach my $config (keys %configs) {
+		print OUT "$configs{$config}\n";
+	}
+	close(OUT);
+
+	make_oldconfig;
+}
+
+# compare two config hashes, and return configs with different vals.
+# It returns B's config values, but you can use A to see what A was.
+sub diff_config_vals {
+	my ($pa, $pb) = @_;
+
+	# crappy Perl way to pass in hashes.
+	my %a = %{$pa};
+	my %b = %{$pb};
+
+	my %ret;
+
+	foreach my $item (keys %a) {
+		if (defined($b{$item}) && $b{$item} ne $a{$item}) {
+			$ret{$item} = $b{$item};
+		}
+	}
+
+	return %ret;
+}
+
+# compare two config hashes and return the configs in B but not A
+sub diff_configs {
+	my ($pa, $pb) = @_;
+
+	my %ret;
+
+	# crappy Perl way to pass in hashes.
+	my %a = %{$pa};
+	my %b = %{$pb};
+
+	foreach my $item (keys %b) {
+		if (!defined($a{$item})) {
+			$ret{$item} = $b{$item};
+		}
+	}
+
+	return %ret;
+}
+
+# return if two configs are equal or not
+# 0 is equal +1 b has something a does not
+# +1 if a and b have a different item.
+# -1 if a has something b does not
+sub compare_configs {
+	my ($pa, $pb) = @_;
+
+	my %ret;
+
+	# crappy Perl way to pass in hashes.
+	my %a = %{$pa};
+	my %b = %{$pb};
+
+	foreach my $item (keys %b) {
+		if (!defined($a{$item})) {
+			return 1;
+		}
+		if ($a{$item} ne $b{$item}) {
+			return 1;
+		}
+	}
+
+	foreach my $item (keys %a) {
+		if (!defined($b{$item})) {
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+
+sub process_new_config {
+	my ($tc, $nc, $gc, $bc, $ofile) = @_;
+
+	my %tmp_config = %{$tc};
+	my %good_configs = %{$gc};
+	my %bad_configs = %{$bc};
+
+	my %new_configs;
+
+	my $runtest = 1;
+	my $ret;
+
+	save_config \%tmp_config, $ofile;
+	assign_configs \%new_configs, $ofile;
+
+	$ret = compare_configs \%new_configs, \%bad_configs;
+	if (!$ret) {
+		print "New config equals bad config, try next test\n";
+		$runtest = 0;
+	}
+
+	if ($runtest) {
+		$ret = compare_configs \%new_configs, \%good_configs;
+		if (!$ret) {
+			print "New config equals good config, try next test\n";
+			$runtest = 0;
+		}
+	}
+
+	%{$nc} = %new_configs;
+
+	return $runtest;
+}
+
+sub run_config_bisect {
+	my ($outfile, $pgood, $pbad) = @_;
+
+	my %good_configs = %{$pgood};
+	my %bad_configs = %{$pbad};
+
+	my %diff_configs = diff_config_vals \%good_configs, \%bad_configs;
+	my %b_configs = diff_configs \%good_configs, \%bad_configs;
+	my %g_configs = diff_configs \%bad_configs, \%good_configs;
+
+	my @diff_arr = keys %diff_configs;
+	my $len_diff = $#diff_arr + 1;
+
+	my @b_arr = keys %b_configs;
+	my $len_b = $#b_arr + 1;
+
+	my @g_arr = keys %g_configs;
+	my $len_g = $#g_arr + 1;
+
+	my $runtest = 1;
+	my %new_configs;
+	my $ret;
+
+	# First, lets get it down to a single subset.
+	# Is the problem with a difference in values?
+	# Is the problem with a missing config?
+	# Is the problem with a config that breaks things?
+
+	# Enable all of one set and see if we get a new bad
+	# or good config.
+
+	# first set the good config to the bad values.
+
+	print "d=$len_diff g=$len_g b=$len_b\n";
+
+	# first lets enable things in bad config that are enabled in good config
+
+	if ($len_diff > 0) {
+		if ($len_b > 0 || $len_g > 0) {
+			my %tmp_config = %bad_configs;
+
+			print "Set tmp config to be bad config with good config values\n";
+			foreach my $item (@diff_arr) {
+				$tmp_config{$item} = $good_configs{$item};
+			}
+
+			$runtest = process_new_config \%tmp_config,
+				\%new_configs, \%good_configs,
+				\%bad_configs, $outfile;
+		}
+	}
+
+	if (!$runtest && $len_diff > 0) {
+
+		if ($len_diff == 1) {
+			process_failed $diff_arr[0];
+			return 1;
+		}
+		my %tmp_config = %bad_configs;
+
+		my $half = int($#diff_arr / 2);
+		my @tophalf = @diff_arr[0 .. $half];
+
+		print "Settings bisect with top half:\n";
+		print "Set tmp config to be bad config with some good config values\n";
+		foreach my $item (@tophalf) {
+			$tmp_config{$item} = $good_configs{$item};
+		}
+
+		$runtest = process_new_config \%tmp_config, \%new_configs,
+			\%good_configs, \%bad_configs, $outfile;
+
+		if (!$runtest) {
+			my %tmp_config = %bad_configs;
+
+			print "Try bottom half\n";
+
+			my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
+
+			foreach my $item (@bottomhalf) {
+				$tmp_config{$item} = $good_configs{$item};
+			}
+
+			$runtest = process_new_config \%tmp_config,
+				\%new_configs, \%good_configs,
+				\%bad_configs, $outfile;
+		}
+	}
+
+}
+
+sub cb_by_file {
+	my ($outfile, $goodfile, $badfile) = @_;
+	my (%good, %bad);
+
+	assign_configs \%good, $goodfile;
+	assign_configs \%bad, $badfile;
+
+	run_config_bisect $outfile, \%good, \%bad;
+}
+
+if (!defined($ARGV[2])) {
+	print "Usage: $0 <outputdir> <good> <bad>\n";
+	exit 1;
+}
+
+if (!defined($ENV{MAKE})) {
+	$ENV{MAKE} = "make";
+}
+
+$outputdir = $ARGV[0];
+
+cb_by_file("${outputdir}/.config", $ARGV[1], $ARGV[2]);
+exit 0;
-- 
2.9.4

[PATCH 5/8] ktest/config_bisect: Simplify bisect logic

[Scott Wood]

From: Scott Wood <oss@buserror.net>

There's no need to diff the configs three different ways or to do things
like "first lets enable things in bad config that are enabled in good
config".  If "undefined" is treated as just another value that a config
can have, then all the cases reduce to "a problem with a difference in
values".

Plus, the old logic would fail with some inputs, as it confuses a symbol
not appearing in the .config file (and thus in the hash) with a symbol
not being selected (and thus being present in the hash with a value of "#
CONFIG_FOO is not set").  diff_configs checks for the former, but
assuming the input configs have been run through make oldconfig, any
symbols that are not present at all are not selectable and thus are side
effects of some other difference -- not relevant to the bisection.

If the bisection proceeds to the point where there are none of these
side effects left, len_b and len_g will both be zero.  If len_diff is
greater than one at this point, a new config to try will not be produced.

Since b_arr and g_arr are irrelevant to the bisection, remove them and
only calculate differences in symbols that are present in both hashes, and
proceed directly to the logic that deals with differences in value.

While we're at it, have diff_config_vals return an array of keys rather
than a hash, since the keys are the only thing ever looked at.

Signed-off-by: Scott Wood <swood@redhat.com>
---
 tools/testing/ktest/config-bisect.pl | 108 ++++++++---------------------------
 1 file changed, 23 insertions(+), 85 deletions(-)

diff --git a/tools/testing/ktest/config-bisect.pl b/tools/testing/ktest/config-bisect.pl
index 1768d145ced8..63a3a940b5de 100755
--- a/tools/testing/ktest/config-bisect.pl
+++ b/tools/testing/ktest/config-bisect.pl
@@ -60,7 +60,6 @@ sub save_config {
 }
 
 # compare two config hashes, and return configs with different vals.
-# It returns B's config values, but you can use A to see what A was.
 sub diff_config_vals {
 	my ($pa, $pb) = @_;
 
@@ -68,34 +67,15 @@ sub diff_config_vals {
 	my %a = %{$pa};
 	my %b = %{$pb};
 
-	my %ret;
+	my @ret;
 
 	foreach my $item (keys %a) {
 		if (defined($b{$item}) && $b{$item} ne $a{$item}) {
-			$ret{$item} = $b{$item};
+			push @ret, $item;
 		}
 	}
 
-	return %ret;
-}
-
-# compare two config hashes and return the configs in B but not A
-sub diff_configs {
-	my ($pa, $pb) = @_;
-
-	my %ret;
-
-	# crappy Perl way to pass in hashes.
-	my %a = %{$pa};
-	my %b = %{$pb};
-
-	foreach my $item (keys %b) {
-		if (!defined($a{$item})) {
-			$ret{$item} = $b{$item};
-		}
-	}
-
-	return %ret;
+	return @ret;
 }
 
 # return if two configs are equal or not
@@ -170,89 +150,47 @@ sub run_config_bisect {
 	my %good_configs = %{$pgood};
 	my %bad_configs = %{$pbad};
 
-	my %diff_configs = diff_config_vals \%good_configs, \%bad_configs;
-	my %b_configs = diff_configs \%good_configs, \%bad_configs;
-	my %g_configs = diff_configs \%bad_configs, \%good_configs;
-
-	my @diff_arr = keys %diff_configs;
+	my @diff_arr = diff_config_vals \%good_configs, \%bad_configs;
 	my $len_diff = $#diff_arr + 1;
 
-	my @b_arr = keys %b_configs;
-	my $len_b = $#b_arr + 1;
-
-	my @g_arr = keys %g_configs;
-	my $len_g = $#g_arr + 1;
-
 	my $runtest = 1;
 	my %new_configs;
 	my $ret;
 
-	# First, lets get it down to a single subset.
-	# Is the problem with a difference in values?
-	# Is the problem with a missing config?
-	# Is the problem with a config that breaks things?
-
-	# Enable all of one set and see if we get a new bad
-	# or good config.
-
-	# first set the good config to the bad values.
-
-	print "d=$len_diff g=$len_g b=$len_b\n";
+	print "d=$len_diff\n";
 
-	# first lets enable things in bad config that are enabled in good config
+	if ($len_diff <= 1) {
+		print "$0: No more bisecting possible\n";
+		exit 2;
+	}
 
-	if ($len_diff > 0) {
-		if ($len_b > 0 || $len_g > 0) {
-			my %tmp_config = %bad_configs;
+	my %tmp_config = %bad_configs;
 
-			print "Set tmp config to be bad config with good config values\n";
-			foreach my $item (@diff_arr) {
-				$tmp_config{$item} = $good_configs{$item};
-			}
+	my $half = int($#diff_arr / 2);
+	my @tophalf = @diff_arr[0 .. $half];
 
-			$runtest = process_new_config \%tmp_config,
-				\%new_configs, \%good_configs,
-				\%bad_configs, $outfile;
-		}
+	print "Settings bisect with top half:\n";
+	foreach my $item (@tophalf) {
+		$tmp_config{$item} = $good_configs{$item};
 	}
 
-	if (!$runtest && $len_diff > 0) {
+	$runtest = process_new_config \%tmp_config, \%new_configs,
+		\%good_configs, \%bad_configs, $outfile;
 
-		if ($len_diff == 1) {
-			process_failed $diff_arr[0];
-			return 1;
-		}
+	if (!$runtest) {
 		my %tmp_config = %bad_configs;
 
-		my $half = int($#diff_arr / 2);
-		my @tophalf = @diff_arr[0 .. $half];
+		print "Try bottom half\n";
 
-		print "Settings bisect with top half:\n";
-		print "Set tmp config to be bad config with some good config values\n";
-		foreach my $item (@tophalf) {
+		my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
+
+		foreach my $item (@bottomhalf) {
 			$tmp_config{$item} = $good_configs{$item};
 		}
 
 		$runtest = process_new_config \%tmp_config, \%new_configs,
-			\%good_configs, \%bad_configs, $outfile;
-
-		if (!$runtest) {
-			my %tmp_config = %bad_configs;
-
-			print "Try bottom half\n";
-
-			my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
-
-			foreach my $item (@bottomhalf) {
-				$tmp_config{$item} = $good_configs{$item};
-			}
-
-			$runtest = process_new_config \%tmp_config,
-				\%new_configs, \%good_configs,
-				\%bad_configs, $outfile;
-		}
+				\%good_configs, \%bad_configs, $outfile;
 	}
-
 }
 
 sub cb_by_file {
-- 
2.9.4

[PATCH 6/8] ktest/config-bisect: Try harder to find a new config

[Scott Wood]

From: Scott Wood <oss@buserror.net>

It's possible that the arbitrarily chosen half of the config differences
fails to make any actual difference in the output config, due to options
that are present in .config but not independently changeable (e.g. the
mutually-exclusive symbols of a "choice" kconfig construct, or a symbol
that has been "select"ed by another).

Try harder to generate a new and different config by randomizing the key
order and bisection percentage, and repeating a few times.

Explicitly randomizing also makes it possible to skip a broken generated
config by simply calling the script again, even on older versions of Perl
that didn't automatically randomize hash order.

Signed-off-by: Scott Wood <swood@redhat.com>
---
 tools/testing/ktest/config-bisect.pl | 41 +++++++++++++++++++++---------------
 1 file changed, 24 insertions(+), 17 deletions(-)

diff --git a/tools/testing/ktest/config-bisect.pl b/tools/testing/ktest/config-bisect.pl
index 63a3a940b5de..54f27b45079f 100755
--- a/tools/testing/ktest/config-bisect.pl
+++ b/tools/testing/ktest/config-bisect.pl
@@ -5,6 +5,7 @@
 #
 
 use strict;
+use List::Util qw(shuffle);
 
 my $outputdir;
 
@@ -153,9 +154,11 @@ sub run_config_bisect {
 	my @diff_arr = diff_config_vals \%good_configs, \%bad_configs;
 	my $len_diff = $#diff_arr + 1;
 
-	my $runtest = 1;
+	my $rand_tries = 10;
+	my $runtest = 0;
 	my %new_configs;
 	my $ret;
+	my $count = 0;
 
 	print "d=$len_diff\n";
 
@@ -164,33 +167,37 @@ sub run_config_bisect {
 		exit 2;
 	}
 
-	my %tmp_config = %bad_configs;
-
-	my $half = int($#diff_arr / 2);
-	my @tophalf = @diff_arr[0 .. $half];
-
-	print "Settings bisect with top half:\n";
-	foreach my $item (@tophalf) {
-		$tmp_config{$item} = $good_configs{$item};
-	}
+	while ($runtest == 0 && ++$count <= $rand_tries) {
+		my %tmp_config = %bad_configs;
 
-	$runtest = process_new_config \%tmp_config, \%new_configs,
-		\%good_configs, \%bad_configs, $outfile;
+		# If we fail to generate a new config (due to the top half
+		# configs being unchangeable without the bottom half configs)
+		# then try a few random permutations, and if those fail try
+		# each option one at a time.
 
-	if (!$runtest) {
-		my %tmp_config = %bad_configs;
+		if ($count <= $rand_tries) {
+			@diff_arr = shuffle(@diff_arr);
+		}
 
-		print "Try bottom half\n";
+		my $half = int($#diff_arr / 2);
+		if ($count > 2) {
+			$half = int(rand($#diff_arr));
+		}
 
-		my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
+		my @tophalf = @diff_arr[0 .. $half];
 
-		foreach my $item (@bottomhalf) {
+		foreach my $item (@tophalf) {
 			$tmp_config{$item} = $good_configs{$item};
 		}
 
 		$runtest = process_new_config \%tmp_config, \%new_configs,
 				\%good_configs, \%bad_configs, $outfile;
 	}
+
+	if ($runtest == 0) {
+		print "$0: No more bisecting possible\n";
+		exit 2;
+	}
 }
 
 sub cb_by_file {
-- 
2.9.4

[PATCH 7/8] ktest: Add simple config-bisect frontend

[Scott Wood]

From: Scott Wood <oss@buserror.net>

Add a friendly, git-bisect-like frontend to config-bisect.pl.  Unlike
ktest.pl, this frontend requires no configuration beyond specifying
the kernel output directory and (if cross-compiling) $ARCH, and does not
need to run continuously for the entire bisection (and thus can be used
to bisect on the machine testing the kernels).

Signed-off-by: Scott Wood <swood@redhat.com>
---
Is the ktest directory the right place for this, or should it (along with
config-bisect.pl) go elsewhere since it no longer depends on the ktest
infrastructure?  If so, where?

 tools/testing/ktest/config-bisect.sh | 120 +++++++++++++++++++++++++++++++++++
 1 file changed, 120 insertions(+)
 create mode 100755 tools/testing/ktest/config-bisect.sh

diff --git a/tools/testing/ktest/config-bisect.sh b/tools/testing/ktest/config-bisect.sh
new file mode 100755
index 000000000000..6032e6736109
--- /dev/null
+++ b/tools/testing/ktest/config-bisect.sh
@@ -0,0 +1,120 @@
+#!/bin/bash
+
+BACKEND=$(dirname $BASH_SOURCE)/config-bisect.pl
+
+usage() {
+	echo Usage:
+	echo "$0 <outputdir> command [args]"
+	echo commands:
+	echo    reset
+	echo    init
+	echo    diff
+	echo    bad [conf]
+	echo    good [conf]
+	echo    skip
+	echo If conf is unspecified, \".config\" is used.
+	echo If cross-compiling, pass the ARCH environment variable.
+}
+
+reset() {
+	rm -rf $STATE
+}
+
+init() {
+	reset
+	mkdir -p $STATE/bad
+	mkdir -p $STATE/good
+	echo 0 > $STATE/good/idx
+	echo 0 > $STATE/bad/idx
+}
+
+next() {
+	if ! [ -d $STATE ]; then
+		echo $0: No config-bisect in progress -- initializing
+		init
+	fi
+
+	IDX=$(cat $STATE/$1/idx)
+	IDX=$(expr $IDX + 1)
+
+	CONF=$2
+	if [ -z "$CONF" ]; then
+		CONF=$O/.config
+	fi
+
+	cp $CONF $STATE/$1/$IDX
+	echo $IDX > $STATE/$1/idx
+}
+
+show_diff() {
+	GI=$(cat $STATE/good/idx)
+	BI=$(cat $STATE/bad/idx)
+
+	if [ $GI != 0 ] && [ $BI != 0 ]; then
+		diff -u $STATE/good/$GI $STATE/bad/$BI
+	else
+		echo $0: cannot diff without at least one good and one bad
+	fi
+}
+
+genconf() {
+	GI=$(cat $STATE/good/idx)
+	BI=$(cat $STATE/bad/idx)
+
+	echo good index $GI, bad index $BI
+
+	if [ $GI != 0 ] && [ $BI != 0 ]; then
+		$BACKEND $O $STATE/good/$GI $STATE/bad/$BI
+
+		case $? in
+		0)
+			;;
+		2)
+			echo Failing config diff:
+			show_diff
+			;;
+		*)
+			echo $0: error in backend
+			exit 1;
+			;;
+		esac
+	fi
+}
+
+if [ -z "$1" -o -z "$2" ]; then
+	usage
+	exit 1
+fi
+
+O=$1
+STATE=$O/.config-bisect
+
+case $2 in
+init)
+	init
+	;;
+reset)
+	reset
+	;;
+diff)
+	show_diff
+	;;
+bad)
+	next bad $3
+	genconf
+	;;
+good)
+	next good $3
+	genconf
+	;;
+skip)
+	# The options chosen are randomized, so we'll get a different
+	# config just by re-running the config bisect backend with the
+	# same inputs.
+	genconf
+	;;
+*)
+	usage
+	exit 1
+	;;
+esac
-- 
2.9.4

[PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl

[Scott Wood]

Reduce code duplication and take advantage of bisection logic
improvements by calling config-bisect.pl.

The output of make oldconfig is now copied directly to the desired file,
rather than doing assign_configs+save_config, in order to preserve the
ordering so that diffing the configs at the end will provide useful
output.

Signed-off-by: Scott Wood <swood@redhat.com>
---
 tools/testing/ktest/ktest.pl | 247 +++++--------------------------------------
 1 file changed, 29 insertions(+), 218 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 7d537bdf462b..cd1f4b5b76b7 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -3086,76 +3086,6 @@ sub create_config {
     make_oldconfig;
 }
 
-# compare two config hashes, and return configs with different vals.
-# It returns B's config values, but you can use A to see what A was.
-sub diff_config_vals {
-    my ($pa, $pb) = @_;
-
-    # crappy Perl way to pass in hashes.
-    my %a = %{$pa};
-    my %b = %{$pb};
-
-    my %ret;
-
-    foreach my $item (keys %a) {
-	if (defined($b{$item}) && $b{$item} ne $a{$item}) {
-	    $ret{$item} = $b{$item};
-	}
-    }
-
-    return %ret;
-}
-
-# compare two config hashes and return the configs in B but not A
-sub diff_configs {
-    my ($pa, $pb) = @_;
-
-    my %ret;
-
-    # crappy Perl way to pass in hashes.
-    my %a = %{$pa};
-    my %b = %{$pb};
-
-    foreach my $item (keys %b) {
-	if (!defined($a{$item})) {
-	    $ret{$item} = $b{$item};
-	}
-    }
-
-    return %ret;
-}
-
-# return if two configs are equal or not
-# 0 is equal +1 b has something a does not
-# +1 if a and b have a different item.
-# -1 if a has something b does not
-sub compare_configs {
-    my ($pa, $pb) = @_;
-
-    my %ret;
-
-    # crappy Perl way to pass in hashes.
-    my %a = %{$pa};
-    my %b = %{$pb};
-
-    foreach my $item (keys %b) {
-	if (!defined($a{$item})) {
-	    return 1;
-	}
-	if ($a{$item} ne $b{$item}) {
-	    return 1;
-	}
-    }
-
-    foreach my $item (keys %a) {
-	if (!defined($b{$item})) {
-	    return -1;
-	}
-    }
-
-    return 0;
-}
-
 sub run_config_bisect_test {
     my ($type) = @_;
 
@@ -3168,166 +3098,51 @@ sub run_config_bisect_test {
     return $ret;
 }
 
-sub process_failed {
-    my ($config) = @_;
+sub config_bisect_end {
+    my ($good, $bad) = @_;
 
     doprint "\n\n***************************************\n";
-    doprint "Found bad config: $config\n";
+    doprint "No more config bisecting possible.\n";
+    doprint `diff -u $good $bad`;
     doprint "***************************************\n\n";
 }
 
-# used for config bisecting
-my $good_config;
-my $bad_config;
-
-sub process_new_config {
-    my ($tc, $nc, $gc, $bc) = @_;
-
-    my %tmp_config = %{$tc};
-    my %good_configs = %{$gc};
-    my %bad_configs = %{$bc};
-
-    my %new_configs;
-
-    my $runtest = 1;
-    my $ret;
-
-    create_config "tmp_configs", \%tmp_config;
-    assign_configs \%new_configs, $output_config;
-
-    $ret = compare_configs \%new_configs, \%bad_configs;
-    if (!$ret) {
-	doprint "New config equals bad config, try next test\n";
-	$runtest = 0;
-    }
-
-    if ($runtest) {
-	$ret = compare_configs \%new_configs, \%good_configs;
-	if (!$ret) {
-	    doprint "New config equals good config, try next test\n";
-	    $runtest = 0;
-	}
-    }
-
-    %{$nc} = %new_configs;
-
-    return $runtest;
-}
-
 sub run_config_bisect {
-    my ($pgood, $pbad) = @_;
-
-    my $type = $config_bisect_type;
-
-    my %good_configs = %{$pgood};
-    my %bad_configs = %{$pbad};
-
-    my %diff_configs = diff_config_vals \%good_configs, \%bad_configs;
-    my %b_configs = diff_configs \%good_configs, \%bad_configs;
-    my %g_configs = diff_configs \%bad_configs, \%good_configs;
-
-    my @diff_arr = keys %diff_configs;
-    my $len_diff = $#diff_arr + 1;
-
-    my @b_arr = keys %b_configs;
-    my $len_b = $#b_arr + 1;
-
-    my @g_arr = keys %g_configs;
-    my $len_g = $#g_arr + 1;
-
-    my $runtest = 1;
-    my %new_configs;
+    my ($good, $bad) = @_;
+    my $cmd;
     my $ret;
 
-    # First, lets get it down to a single subset.
-    # Is the problem with a difference in values?
-    # Is the problem with a missing config?
-    # Is the problem with a config that breaks things?
-
-    # Enable all of one set and see if we get a new bad
-    # or good config.
+    run_command "$builddir/tools/testing/ktest/config-bisect.pl $outputdir $good $bad";
 
-    # first set the good config to the bad values.
-
-    doprint "d=$len_diff g=$len_g b=$len_b\n";
-
-    # first lets enable things in bad config that are enabled in good config
-
-    if ($len_diff > 0) {
-	if ($len_b > 0 || $len_g > 0) {
-	    my %tmp_config = %bad_configs;
-
-	    doprint "Set tmp config to be bad config with good config values\n";
-	    foreach my $item (@diff_arr) {
-		$tmp_config{$item} = $good_configs{$item};
-	    }
-
-	    $runtest = process_new_config \%tmp_config, \%new_configs,
-			    \%good_configs, \%bad_configs;
-	}
+    if ($run_command_status == 2) {
+        config_bisect_end $good, $bad;
+        return 1;
     }
 
-    if (!$runtest && $len_diff > 0) {
-
-	if ($len_diff == 1) {
-	    process_failed $diff_arr[0];
-	    return 1;
-	}
-	my %tmp_config = %bad_configs;
-
-	my $half = int($#diff_arr / 2);
-	my @tophalf = @diff_arr[0 .. $half];
-
-	doprint "Settings bisect with top half:\n";
-	doprint "Set tmp config to be bad config with some good config values\n";
-	foreach my $item (@tophalf) {
-	    $tmp_config{$item} = $good_configs{$item};
-	}
-
-	$runtest = process_new_config \%tmp_config, \%new_configs,
-			    \%good_configs, \%bad_configs;
-
-	if (!$runtest) {
-	    my %tmp_config = %bad_configs;
-
-	    doprint "Try bottom half\n";
-
-	    my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
-
-	    foreach my $item (@bottomhalf) {
-		$tmp_config{$item} = $good_configs{$item};
-	    }
-
-	    $runtest = process_new_config \%tmp_config, \%new_configs,
-			    \%good_configs, \%bad_configs;
-	}
+    if ($run_command_status != 0) {
+        dodie "config-bisect.pl failed";
     }
 
-    if ($runtest) {
-	$ret = run_config_bisect_test $type;
-	if ($ret) {
-	    doprint "NEW GOOD CONFIG\n";
-	    %good_configs = %new_configs;
-	    run_command "mv $good_config ${good_config}.last";
-	    save_config \%good_configs, $good_config;
-	    %{$pgood} = %good_configs;
-	} else {
-	    doprint "NEW BAD CONFIG\n";
-	    %bad_configs = %new_configs;
-	    run_command "mv $bad_config ${bad_config}.last";
-	    save_config \%bad_configs, $bad_config;
-	    %{$pbad} = %bad_configs;
-	}
-	return 0;
+    $ret = run_config_bisect_test $config_bisect_type;
+    if ($ret) {
+        doprint "NEW GOOD CONFIG\n";
+        $cmd = "mv $good ${good}.last && mv $output_config $good";
+    } else {
+        doprint "NEW BAD CONFIG\n";
+        $cmd = "mv $bad ${bad}.last && mv $output_config $bad";
     }
 
-    fail "Hmm, need to do a mix match?\n";
-    return -1;
+    system($cmd) == 0 or dodie $cmd;
+
+    return 0;
 }
 
 sub config_bisect {
     my ($i) = @_;
 
+    my $good_config;
+    my $bad_config;
+
     my $type = $config_bisect_type;
     my $ret;
 
@@ -3358,18 +3173,14 @@ sub config_bisect {
     doprint "Run good configs through make oldconfig\n";
     assign_configs \%tmp_configs, $good_config;
     create_config "$good_config", \%tmp_configs;
-    assign_configs \%good_configs, $output_config;
+    $good_config = "$tmpdir/good_config";
+    system("cp $output_config $good_config") == 0 or dodie "cp good config";
 
     doprint "Run bad configs through make oldconfig\n";
     assign_configs \%tmp_configs, $bad_config;
     create_config "$bad_config", \%tmp_configs;
-    assign_configs \%bad_configs, $output_config;
-
-    $good_config = "$tmpdir/good_config";
     $bad_config = "$tmpdir/bad_config";
-
-    save_config \%good_configs, $good_config;
-    save_config \%bad_configs, $bad_config;
+    system("cp $output_config $bad_config") == 0 or dodie "cp bad config";
 
     if (defined($config_bisect_check) && $config_bisect_check ne "0") {
 	if ($config_bisect_check ne "good") {
@@ -3393,7 +3204,7 @@ sub config_bisect {
     }
 
     do {
-	$ret = run_config_bisect \%good_configs, \%bad_configs;
+	$ret = run_config_bisect $good_config, $bad_config;
 	print_times;
     } while (!$ret);
 
-- 
2.9.4

Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl

[Scott Wood]

On Sun, 2017-07-16 at 19:16 -0500, Scott Wood wrote:
> Reduce code duplication and take advantage of bisection logic
> improvements by calling config-bisect.pl.
> 
> The output of make oldconfig is now copied directly to the desired
> file,
> rather than doing assign_configs+save_config, in order to preserve the
> ordering so that diffing the configs at the end will provide useful
> output.
> 
> Signed-off-by: Scott Wood <swood@redhat.com>
> ---
>  tools/testing/ktest/ktest.pl | 247 +++++-----------------------------

Any comments on this patchset?

-Scott

Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl

[Steven Rostedt]

On Thu, 14 Sep 2017 14:41:12 -0700
Scott Wood <swood@redhat.com> wrote:

> On Sun, 2017-07-16 at 19:16 -0500, Scott Wood wrote:
> > Reduce code duplication and take advantage of bisection logic
> > improvements by calling config-bisect.pl.
> > 
> > The output of make oldconfig is now copied directly to the desired
> > file,
> > rather than doing assign_configs+save_config, in order to preserve the
> > ordering so that diffing the configs at the end will provide useful
> > output.
> > 
> > Signed-off-by: Scott Wood <swood@redhat.com>
> > ---
> >  tools/testing/ktest/ktest.pl | 247 +++++-----------------------------  
> 
> Any comments on this patchset?

Finally got around to looking at it. Sorry for the late reply.

I didn't see anything to concerning in patches 1-7 and applied them.
I'll start testing them soon too.

-- Steve

Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl

[Steven Rostedt]

On Sun, 16 Jul 2017 19:16:30 -0500
Scott Wood <swood@redhat.com> wrote:

> Reduce code duplication and take advantage of bisection logic
> improvements by calling config-bisect.pl.
> 
> The output of make oldconfig is now copied directly to the desired file,
> rather than doing assign_configs+save_config, in order to preserve the
> ordering so that diffing the configs at the end will provide useful
> output.

The reason I never did this, was that I copy ktest.pl all over the
place :-/ I need it to be a stand alone. I don't copy config-bisect
around. Not sure how to deal with this.

-- Steve

Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl

[Scott Wood]

On Wed, 2017-10-04 at 15:18 -0400, Steven Rostedt wrote:
> On Sun, 16 Jul 2017 19:16:30 -0500
> Scott Wood <swood@redhat.com> wrote:
> 
> > Reduce code duplication and take advantage of bisection logic
> > improvements by calling config-bisect.pl.
> > 
> > The output of make oldconfig is now copied directly to the desired
> > file,
> > rather than doing assign_configs+save_config, in order to preserve
> > the
> > ordering so that diffing the configs at the end will provide useful
> > output.
> 
> The reason I never did this, was that I copy ktest.pl all over the
> place :-/ I need it to be a stand alone. I don't copy config-bisect
> around. Not sure how to deal with this.

By "this" I assume you mean using the external config-bisect code,
rather than the part about copying oldconfig output?

The options I can see are to copy both files as a group (possibly making
that easier by renaming config-bisect.pl to ktest-conf-bisect.pl, so you
can "cp tools/testing/ktest/ktest*.pl <dest>"), or just living with code
duplication.  In a previous discussion you suggested you preferred that
latter option, though in that case the config bisect logic changes
should ideally be done before the fork.

It should also be noted that ktest.pl only depends on config-bisect.pl
if a config bisect is being performed, so other ktest.pl functions still
work standalone.

-Scott

Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl

[Steven Rostedt]

On Wed, 04 Oct 2017 15:24:23 -0500
Scott Wood <swood@redhat.com> wrote:

> On Wed, 2017-10-04 at 15:18 -0400, Steven Rostedt wrote:
> > On Sun, 16 Jul 2017 19:16:30 -0500
> > Scott Wood <swood@redhat.com> wrote:
> >   
> > > Reduce code duplication and take advantage of bisection logic
> > > improvements by calling config-bisect.pl.
> > > 
> > > The output of make oldconfig is now copied directly to the desired
> > > file,
> > > rather than doing assign_configs+save_config, in order to preserve
> > > the
> > > ordering so that diffing the configs at the end will provide useful
> > > output.  
> > 
> > The reason I never did this, was that I copy ktest.pl all over the
> > place :-/ I need it to be a stand alone. I don't copy config-bisect
> > around. Not sure how to deal with this.  
> 
> By "this" I assume you mean using the external config-bisect code,
> rather than the part about copying oldconfig output?
> 
> The options I can see are to copy both files as a group (possibly making
> that easier by renaming config-bisect.pl to ktest-conf-bisect.pl, so you
> can "cp tools/testing/ktest/ktest*.pl <dest>"), or just living with code
> duplication.  In a previous discussion you suggested you preferred that
> latter option, though in that case the config bisect logic changes
> should ideally be done before the fork.

I don't want to rename it. I like the simply "config-bisect.pl" name.

> 
> It should also be noted that ktest.pl only depends on config-bisect.pl
> if a config bisect is being performed, so other ktest.pl functions still
> work standalone.

I thought about this too, and may be able to live with that. Perhaps we
should make the config-bisect internals into something that can be a
perl library, and be able to place that someplace that both could use?

-- Steve

Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl

[Steven Rostedt]

On Thu, 05 Oct 2017 18:18:54 -0500
Scott Wood <swood@redhat.com> wrote:

> On Thu, 2017-10-05 at 08:50 -0400, Steven Rostedt wrote:
> > On Wed, 04 Oct 2017 15:24:23 -0500
> > Scott Wood <swood@redhat.com> wrote:
> >   
> > > It should also be noted that ktest.pl only depends on config-
> > > bisect.pl
> > > if a config bisect is being performed, so other ktest.pl functions
> > > still
> > > work standalone.  
> > 
> > I thought about this too, and may be able to live with that. Perhaps
> > we
> > should make the config-bisect internals into something that can be a
> > perl library, and be able to place that someplace that both could use?  
> 
> I'm not familiar with how Perl libraries work -- how would they solve
> this problem?  If you're talking about having to install the library
> into some location outside the kernel tree, that seems contrary to the
> desire to have a lightweight tool that can be copied around.  If not,
> how would it be any better than depending on a callable script?
>

I'm thinking about installing it in the tree, and then using variables
to find it. I'm not sure how perl works either, but as an interpreted
language, I'm sure there's gotta be a way to add default paths to look
for it. Perhaps there's a way to include other perl scripts.

I'll have to play around with this to come up with better ideas.

-- Steve

Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl

[Scott Wood]

On Thu, 2017-10-05 at 08:50 -0400, Steven Rostedt wrote:
> On Wed, 04 Oct 2017 15:24:23 -0500
> Scott Wood <swood@redhat.com> wrote:
> 
> > It should also be noted that ktest.pl only depends on config-
> > bisect.pl
> > if a config bisect is being performed, so other ktest.pl functions
> > still
> > work standalone.
> 
> I thought about this too, and may be able to live with that. Perhaps
> we
> should make the config-bisect internals into something that can be a
> perl library, and be able to place that someplace that both could use?

I'm not familiar with how Perl libraries work -- how would they solve
this problem?  If you're talking about having to install the library
into some location outside the kernel tree, that seems contrary to the
desire to have a lightweight tool that can be copied around.  If not,
how would it be any better than depending on a callable script?

-Scott