[PATCH 0/4] Improvements to bitbake-selftest

[PATCH 0/4] Improvements to bitbake-selftest

[Markus Lehtonen]

This patchset aims at making it easier to debug and develop bitbake unit tests.
It makes it possible to have finer control of which tests to run and improves
the console output, especially for failed tests. One patch also allows
preserving temporary directories for deeper debugging.

The following changes since commit 3ff1c66e6f336e5de7dcbc983a97fcd19ddc6b81:

  bitbake: Update version to 1.31.1 (2016-08-18 10:05:26 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib marquiz/bitbake/unittest
  http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/bitbake/unittest


Markus Lehtonen (4):
  bitbake-selftest: utilize unittest.main better
  bitbake-selftest: add help text for env variable(s)
  bitbake-selftest: introduce BB_TMPDIR_NOCLEAN
  bitbake-selftest: enable bitbake logging to stdout

 bin/bitbake-selftest  | 61 ++++++++++++++++++++++++++++++++-------------------
 lib/bb/tests/fetch.py |  5 ++++-
 2 files changed, 43 insertions(+), 23 deletions(-)

-- 
2.6.6

[PATCH 1/4] bitbake-selftest: utilize unittest.main better

[Markus Lehtonen]

This simplifies the script, and, gives new features. It is now possible
to run single test functions, for example. This is nice when writing new
test cases.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 bin/bitbake-selftest | 30 ++++++++----------------------
 1 file changed, 8 insertions(+), 22 deletions(-)

diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
index 1e6f35e..1e615cc 100755
--- a/bin/bitbake-selftest
+++ b/bin/bitbake-selftest
@@ -25,31 +25,17 @@ try:
 except RuntimeError as exc:
     sys.exit(str(exc))
 
-def usage():
-    print('usage: [BB_SKIP_NETTESTS=yes] %s [-v] [testname1 [testname2]...]' % os.path.basename(sys.argv[0]))
-
-verbosity = 1
-
-tests = sys.argv[1:]
-if '-v' in sys.argv:
-    tests.remove('-v')
-    verbosity = 2
-
-if tests:
-    if '--help' in sys.argv[1:]:
-        usage()
-        sys.exit(0)
-else:
-    tests = ["bb.tests.codeparser",
-             "bb.tests.cow",
-             "bb.tests.data",
-             "bb.tests.fetch",
-             "bb.tests.parse",
-             "bb.tests.utils"]
+tests = ["bb.tests.codeparser",
+         "bb.tests.cow",
+         "bb.tests.data",
+         "bb.tests.fetch",
+         "bb.tests.parse",
+         "bb.tests.utils"]
 
 for t in tests:
     t = '.'.join(t.split('.')[:3])
     __import__(t)
 
-unittest.main(argv=["bitbake-selftest"] + tests, verbosity=verbosity)
 
+if __name__ == '__main__':
+        unittest.main(defaultTest=tests)
-- 
2.6.6

[PATCH 2/4] bitbake-selftest: add help text for env variable(s)

[Markus Lehtonen]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 bin/bitbake-selftest | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
index 1e615cc..25905d7 100755
--- a/bin/bitbake-selftest
+++ b/bin/bitbake-selftest
@@ -37,5 +37,17 @@ for t in tests:
     __import__(t)
 
 
+ENV_HELP = """\
+Environment variables:
+  BB_SKIP_NETTESTS      set to 'yes' in order to skip tests using network
+                        connection
+"""
+
+class main(unittest.main):
+    def _print_help(self, *args, **kwargs):
+        super(main, self)._print_help(*args, **kwargs)
+        print(ENV_HELP)
+
+
 if __name__ == '__main__':
-        unittest.main(defaultTest=tests)
+        main(defaultTest=tests)
-- 
2.6.6

[PATCH 3/4] bitbake-selftest: introduce BB_TMPDIR_NOCLEAN

[Markus Lehtonen]

Set this env variable to 'yes' to preserve temporary directories used by
the fetcher tests. Useful for debugging tests.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 bin/bitbake-selftest  | 1 +
 lib/bb/tests/fetch.py | 5 ++++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
index 25905d7..1e00e33 100755
--- a/bin/bitbake-selftest
+++ b/bin/bitbake-selftest
@@ -41,6 +41,7 @@ ENV_HELP = """\
 Environment variables:
   BB_SKIP_NETTESTS      set to 'yes' in order to skip tests using network
                         connection
+  BB_TMPDIR_NOCLEAN     set to 'yes' to preserve test tmp directories
 """
 
 class main(unittest.main):
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 272078f..e8c416a 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -360,7 +360,10 @@ class FetcherTest(unittest.TestCase):
 
     def tearDown(self):
         os.chdir(self.origdir)
-        bb.utils.prunedir(self.tempdir)
+        if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
+            print("Not cleaning up %s. Please remove manually." % self.tempdir)
+        else:
+            bb.utils.prunedir(self.tempdir)
 
 class MirrorUriTest(FetcherTest):
 
-- 
2.6.6

[PATCH 4/4] bitbake-selftest: enable bitbake logging to stdout

[Markus Lehtonen]

Now you get the bb logger output for failed tests. This helps debugging
problems. Also, all stdout/stderr data for successful tests is silenced
which makes for less cluttered console output.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 bin/bitbake-selftest | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
index 1e00e33..380e003 100755
--- a/bin/bitbake-selftest
+++ b/bin/bitbake-selftest
@@ -37,6 +37,24 @@ for t in tests:
     __import__(t)
 
 
+# Set-up logging
+class StdoutStreamHandler(logging.StreamHandler):
+    """Special handler so that unittest is able to capture stdout"""
+    def __init__(self):
+        # Override __init__() because we don't want to set self.stream here
+        logging.Handler.__init__(self)
+
+    @property
+    def stream(self):
+        # We want to dynamically write wherever sys.stdout is pointing to
+        return sys.stdout
+
+
+handler = StdoutStreamHandler()
+bb.logger.addHandler(handler)
+bb.logger.setLevel(logging.DEBUG)
+
+
 ENV_HELP = """\
 Environment variables:
   BB_SKIP_NETTESTS      set to 'yes' in order to skip tests using network
@@ -51,4 +69,4 @@ class main(unittest.main):
 
 
 if __name__ == '__main__':
-        main(defaultTest=tests)
+        main(defaultTest=tests, buffer=True)
-- 
2.6.6

Re: [PATCH 1/4] bitbake-selftest: utilize unittest.main better

[Richard Purdie]

On Thu, 2016-08-18 at 19:55 +0300, Markus Lehtonen wrote:
> This simplifies the script, and, gives new features. It is now
> possible
> to run single test functions, for example. This is nice when writing
> new
> test cases.
> 
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
>  bin/bitbake-selftest | 30 ++++++++----------------------
>  1 file changed, 8 insertions(+), 22 deletions(-)
> 
> diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
> index 1e6f35e..1e615cc 100755
> --- a/bin/bitbake-selftest
> +++ b/bin/bitbake-selftest
> @@ -25,31 +25,17 @@ try:
>  except RuntimeError as exc:
>      sys.exit(str(exc))
>  
> -def usage():
> -    print('usage: [BB_SKIP_NETTESTS=yes] %s [-v] [testname1
> [testname2]...]' % os.path.basename(sys.argv[0]))

Could we at least preserve the usage message which is helpful as I can
never remember BB_SKIP_NETTESTS=yes without prompting...

Cheers,

Richard

Re: [PATCH 1/4] bitbake-selftest: utilize unittest.main better

[Markus Lehtonen]

On 18/08/16 20:03, "Richard Purdie" <richard.purdie@linuxfoundation.org>
wrote:

>On Thu, 2016-08-18 at 19:55 +0300, Markus Lehtonen wrote:
>> This simplifies the script, and, gives new features. It is now
>> possible
>> to run single test functions, for example. This is nice when writing
>> new
>> test cases.
>> 
>> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>> ---
>>  bin/bitbake-selftest | 30 ++++++++----------------------
>>  1 file changed, 8 insertions(+), 22 deletions(-)
>> 
>> diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
>> index 1e6f35e..1e615cc 100755
>> --- a/bin/bitbake-selftest
>> +++ b/bin/bitbake-selftest
>> @@ -25,31 +25,17 @@ try:
>>  except RuntimeError as exc:
>>      sys.exit(str(exc))
>>  
>> -def usage():
>> -    print('usage: [BB_SKIP_NETTESTS=yes] %s [-v] [testname1
>> [testname2]...]' % os.path.basename(sys.argv[0]))
>
>Could we at least preserve the usage message which is helpful as I can
>never remember BB_SKIP_NETTESTS=yes without prompting...

I didn't find an easy way to add that to the usage string. However, patch
#2 ("bitbake-selftest: add help text for env variable(s)") adds a note
about BB_SKIP_NETTESTS that gets printed if you use --help. Would this be
ok?

Thanks,
  Markus

Re: [PATCH 1/4] bitbake-selftest: utilize unittest.main better

[Richard Purdie]

On Fri, 2016-08-19 at 08:41 +0300, Markus Lehtonen wrote:
> On 18/08/16 20:03, "Richard Purdie" <
> richard.purdie@linuxfoundation.org>
> wrote:
> 
> > On Thu, 2016-08-18 at 19:55 +0300, Markus Lehtonen wrote:
> > > This simplifies the script, and, gives new features. It is now
> > > possible
> > > to run single test functions, for example. This is nice when
> > > writing
> > > new
> > > test cases.
> > > 
> > > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> > > ---
> > >  bin/bitbake-selftest | 30 ++++++++----------------------
> > >  1 file changed, 8 insertions(+), 22 deletions(-)
> > > 
> > > diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
> > > index 1e6f35e..1e615cc 100755
> > > --- a/bin/bitbake-selftest
> > > +++ b/bin/bitbake-selftest
> > > @@ -25,31 +25,17 @@ try:
> > >  except RuntimeError as exc:
> > >      sys.exit(str(exc))
> > >  
> > > -def usage():
> > > -    print('usage: [BB_SKIP_NETTESTS=yes] %s [-v] [testname1
> > > [testname2]...]' % os.path.basename(sys.argv[0]))
> > 
> > Could we at least preserve the usage message which is helpful as I
> > can
> > never remember BB_SKIP_NETTESTS=yes without prompting...
> 
> I didn't find an easy way to add that to the usage string. However,
> patch
> #2 ("bitbake-selftest: add help text for env variable(s)") adds a
> note
> about BB_SKIP_NETTESTS that gets printed if you use --help. Would
> this be
> ok?

Yes, when I sent this I was reading through the patches in order and
hadn't got to 2/4, sorry. I should have read ahead! This should be ok.

Cheers,

Richard