* [PATCH] test-manual code block, link, and format update
@ 2020-09-14 23:11 Mark Morton
2020-09-15 6:08 ` [docs] " Nicolas Dechesne
0 siblings, 1 reply; 3+ messages in thread
From: Mark Morton @ 2020-09-14 23:11 UTC (permalink / raw)
To: docs
Signed-off-by: Mark Morton <mark.morton@windriver.com>
---
documentation/conf.py | 4 +-
.../test-manual/test-manual-intro.rst | 266 +++++++++++-------
.../test-manual-understand-autobuilder.rst | 102 ++++---
3 files changed, 226 insertions(+), 146 deletions(-)
diff --git a/documentation/conf.py b/documentation/conf.py
index e1a7e5e3d..e825fbf6e 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -42,8 +42,8 @@ sys.path.insert(0, os.path.abspath('sphinx'))
extensions = [
'sphinx.ext.autosectionlabel',
'sphinx.ext.extlinks',
- 'sphinx.ext.intersphinx',
- 'yocto-vars'
+ 'sphinx.ext.intersphinx'
+ # 'yocto-vars'
]
autosectionlabel_prefix_document = True
diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst
index 2f11e2beb..d68ab73f4 100644
--- a/documentation/test-manual/test-manual-intro.rst
+++ b/documentation/test-manual/test-manual-intro.rst
@@ -23,26 +23,25 @@ project core.
Currently, the Yocto Project Test Environment Manual has no projected
release date. This manual is a work-in-progress and is being initially
-loaded with information from the `README <>`__ files and notes from key
+loaded with information from the README files and notes from key
engineers:
-- *``yocto-autobuilder2``:* This
- ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
+- *yocto-autobuilder2:* This
+ `README.md <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md>`__
is the main README which detials how to set up the Yocto Project
Autobuilder. The ``yocto-autobuilder2`` repository represents the
Yocto Project's console UI plugin to Buildbot and the configuration
necessary to configure Buildbot to perform the testing the project
requires.
-- *``yocto-autobuilder-helper``:* This
- ```README`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README
+- *yocto-autobuilder-helper:* This `README <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README/>`__
and repository contains Yocto Project Autobuilder Helper scripts and
configuration. The ``yocto-autobuilder-helper`` repository contains
the "glue" logic that defines which tests to run and how to run them.
As a result, it can be used by any Continuous Improvement (CI) system
to run builds, support getting the correct code revisions, configure
builds and layers, run builds, and collect results. The code is
- independent of any CI system, which means the code can work Buildbot,
+ independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
Jenkins, or others. This repository has a branch per release of the
project defining the tests to run on a per release basis.
@@ -66,9 +65,9 @@ that the Yocto Project customizes using code from the
resulting UI plug-in allows you to visualize builds in a way suited to
the project's needs.
-A ``helper`` layer provides configuration and job management through
+A helper layer provides configuration and job management through
scripts found in the ``yocto-autobuilder-helper`` repository. The
-``helper`` layer contains the bulk of the build configuration
+helper layer contains the bulk of the build configuration
information and is release-specific, which makes it highly customizable
on a per-project basis. The layer is CI system-agnostic and contains a
number of Helper scripts that can generate build configurations from
@@ -124,18 +123,21 @@ thefollowing types of tests:
ensure we have no single point of failure and can ensure the
different distros work effectively.
-- *eSDK Testing:* Image tests initiated through the following command:
- $ bitbake image -c testsdkext The tests utilize the ``testsdkext``
- class and the ``do_testsdkext`` task.
+- *eSDK Testing:* Image tests initiated through the following command::
+
+ $ bitbake image -c testsdkext
+
+ The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task.
- *Feature Testing:* Various scenario-based tests are run through the
- `OpenEmbedded
- Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__
- (oe-selftest). We test oe-selftest on each of the main distrubutions
+ :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions
we support.
-- *Image Testing:* Image tests initiated through the following command:
- $ bitbake image -c testimage The tests utilize the
+- *Image Testing:* Image tests initiated through the following command::
+
+ $ bitbake image -c testimage
+
+ The tests utilize the
:ref:`testimage* <ref-classes-testimage*>`
classes and the
:ref:`ref-tasks-testimage` task.
@@ -147,8 +149,8 @@ thefollowing types of tests:
- *Package Testing:* A Package Test (ptest) runs tests against packages
built by the OpenEmbedded build system on the target machine. See the
- "`Testing Packages With
- ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section
+ :ref:`Testing Packages With
+ ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section
in the Yocto Project Development Tasks Manual and the
":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more
information on Ptest.
@@ -159,8 +161,8 @@ thefollowing types of tests:
the ``do_testsdk`` task.
- *Unit Testing:* Unit tests on various components of the system run
- through ``oe-selftest`` and
- ```bitbake-selftest`` <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__.
+ through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and
+ :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`.
- *Automatic Upgrade Helper:* This target tests whether new versions of
software are available and whether we can automatically upgrade to
@@ -174,36 +176,43 @@ How Tests Map to Areas of Code
Tests map into the codebase as follows:
-- *bitbake-selftest*:
+- *bitbake-selftest:*
These tests are self-contained and test BitBake as well as its APIs,
which include the fetchers. The tests are located in
``bitbake/lib/*/tests``.
- From within the BitBake repository, run the following: $
- bitbake-selftest
+ From within the BitBake repository, run the following::
+
+ $ bitbake-selftest
To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
- variable when running "bitbake-selftest" as follows: $
- BB_SKIP_NETTEST=yes bitbake-selftest
+ variable when running "bitbake-selftest" as follows::
+
+ $ BB_SKIP_NETTEST=yes bitbake-selftest
The default output is quiet and just prints a summary of what was
- run. To see more information, there is a verbose option:$
- bitbake-selftest -v
+ run. To see more information, there is a verbose option::
+
+ $ bitbake-selftest -v
Use this option when you wish to skip tests that access the network,
which are mostly necessary to test the fetcher modules. To specify
individual test modules to run, append the test module name to the
"bitbake-selftest" command. For example, to specify the tests for the
- bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can
- also specify individual tests by defining the full name and module
- plus the class path of the test, for example: $ bitbake-selftest
- bb.tests.data.TestOverrides.test_one_override
+ bb.data.module, run::
+
+ $ bitbake-selftest bb.test.data.module
+
+ You can also specify individual tests by defining the full name and module
+ plus the class path of the test, for example::
+
+ $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
The tests are based on `Python
unittest <https://docs.python.org/3/library/unittest.html>`__.
-- *oe-selftest*:
+- *oe-selftest:*
- These tests use OE to test the workflows, which include testing
specific features, behaviors of tasks, and API unit tests.
@@ -219,16 +228,25 @@ Tests map into the codebase as follows:
- The code for the tests resides in
``meta/lib/oeqa/selftest/cases/``.
- - To run all the tests, enter the following command: $ oe-selftest
- -a
+ - To run all the tests, enter the following command::
+
+ $ oe-selftest -a
- To run a specific test, use the following command form where
- testname is the name of the specific test: $ oe-selftest -r
- testname For example, the following command would run the tinfoil
- getVar API test:$ oe-selftest -r
- tinfoil.TinfoilTests.test_getvarIt is also possible to run a set
+ testname is the name of the specific test::
+
+ $ oe-selftest -r <testname>
+
+ For example, the following command would run the tinfoil
+ getVar API test::
+
+ $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
+
+ It is also possible to run a set
of tests. For example the following command will run all of the
- tinfoil tests:$ oe-selftest -r tinfoil
+ tinfoil tests::
+
+ $ oe-selftest -r tinfoil
- *testimage:*
@@ -240,10 +258,13 @@ Tests map into the codebase as follows:
- You need to set the
:term:`IMAGE_CLASSES`
- variable as follows: IMAGE_CLASSES += "testimage"
+ variable as follows::
+
+ IMAGE_CLASSES += "testimage"
- - Run the tests using the following command form: $ bitbake image -c
- testimage
+ - Run the tests using the following command form::
+
+ $ bitbake image -c testimage
- *testsdk:*
@@ -252,8 +273,9 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
- - Run the test using the following command form: $ bitbake image -c
- testsdk
+ - Run the test using the following command form::
+
+ $ bitbake image -c testsdk
- *testsdk_ext:*
@@ -262,8 +284,9 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/esdk``.
- - To run the tests, use the following command form: $ bitbake image
- -c testsdkext
+ - To run the tests, use the following command form::
+
+ $ bitbake image -c testsdkext
- *oe-build-perf-test:*
@@ -272,8 +295,11 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/buildperf``.
- - To run the tests, use the following command form: $
- oe-build-perf-test optionsThe command takes a number of options,
+ - To run the tests, use the following command form::
+
+ $ oe-build-perf-test <options>
+
+ The command takes a number of options,
such as where to place the test results. The Autobuilder Helper
Scripts include the ``build-perf-test-wrapper`` script with
examples of how to use the oe-build-perf-test from the command
@@ -308,14 +334,20 @@ directory.
``bitbake-selftest``
--------------------
-A simple test example from ``lib/bb/tests/data.py`` is: class
-DataExpansions(unittest.TestCase): def setUp(self): self.d =
-bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] =
-"value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def
-test_one_var(self): val = self.d.expand("${foo}")
-self.assertEqual(str(val), "value_of_foo")
+A simple test example from ``lib/bb/tests/data.py`` is::
-In this example, a ```DataExpansions`` <>`__ class of tests is created,
+ class DataExpansions(unittest.TestCase):
+ def setUp(self):
+ self.d = bb.data.init()
+ self.d["foo"] = "value_of_foo"
+ self.d["bar"] = "value_of_bar"
+ self.d["value_of_foo"] = "value_of_'value_of_foo'"
+
+ def test_one_var(self):
+ val = self.d.expand("${foo}")
+ self.assertEqual(str(val), "value_of_foo")
+
+In this example, a ``DataExpansions`` class of tests is created,
derived from standard python unittest. The class has a common ``setUp``
function which is shared by all the tests in the class. A simple test is
then added to test that when a variable is expanded, the correct value
@@ -334,14 +366,15 @@ These tests are more complex due to the setup required behind the scenes
for full builds. Rather than directly using Python's unittest, the code
wraps most of the standard objects. The tests can be simple, such as
testing a command from within the OE build environment using the
-following example:class BitbakeLayers(OESelftestTestCase): def
-test_bitbakelayers_showcrossdepends(self): result =
-runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in
-result.output, msg = "No dependencies were shown. bitbake-layers
-show-cross-depends output: %s"% result.output)
+following example::
+
+ class BitbakeLayers(OESelftestTestCase):
+ def test_bitbakelayers_showcrossdepends(self):
+ result = runCmd('bitbake-layers show-cross-depends')
+ self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
-creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived
+creates a testcase from the ``OESelftestTestCase`` class, derived
from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
and checks the output to ensure it contains something we know should be
here.
@@ -367,7 +400,7 @@ so tests within a given test class should always run in the same build,
while tests in different classes or modules may be split into different
builds. There is no data store available for these tests since the tests
launch the ``bitbake`` command and exist outside of its context. As a
-result, common bitbake library functions (bb.*) are also unavailable.
+result, common bitbake library functions (bb.\*) are also unavailable.
.. _testimage-example:
@@ -378,14 +411,18 @@ These tests are run once an image is up and running, either on target
hardware or under QEMU. As a result, they are assumed to be running in a
target image environment, as opposed to a host build environment. A
simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
-the following:class PythonTest(OERuntimeTestCase):
-@OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core'])
-def test_python3(self): cmd = "python3 -c \\"import codecs;
-print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output =
-self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output
-self.assertEqual(status, 0, msg=msg)
-
-In this example, the ```OERuntimeTestCase`` <>`__ class wraps
+the following::
+
+ class PythonTest(OERuntimeTestCase):
+ @OETestDepends(['ssh.SSHTest.test_ssh'])
+ @OEHasPackage(['python3-core'])
+ def test_python3(self):
+ cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+ status, output = self.target.run(cmd)
+ msg = 'Exit status was not 0. Output: %s' % output
+ self.assertEqual(status, 0, msg=msg)
+
+In this example, the ``OERuntimeTestCase`` class wraps
``unittest.TestCase``. Within the test, ``self.target`` represents the
target system, where commands can be run on it using the ``run()``
method.
@@ -402,19 +439,30 @@ the image.
These tests are run against built extensible SDKs (eSDKs). The tests can
assume that the eSDK environment has already been setup. An example from
-``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class
-DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls):
-myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst =
-os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src,
-cls.myapp_dst) subprocess.check_output(['git', 'init', '.'],
-cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'],
-cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m',
-"'test commit'"], cwd=cls.myapp_dst) @classmethod def
-tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def
-\_test_devtool_build(self, directory): self._run('devtool add myapp %s'
-% directory) try: self._run('devtool build myapp') finally:
-self._run('devtool reset myapp') def test_devtool_build_make(self):
-self._test_devtool_build(self.myapp_dst)In this example, the ``devtool``
+``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
+
+ class DevtoolTest(OESDKExtTestCase):
+ @classmethod def setUpClass(cls):
+ myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
+ cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
+ shutil.copytree(myapp_src, cls.myapp_dst)
+ subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
+ subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
+ subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.myapp_dst)
+ def _test_devtool_build(self, directory):
+ self._run('devtool add myapp %s' % directory)
+ try:
+ self._run('devtool build myapp')
+ finally:
+ self._run('devtool reset myapp')
+ def test_devtool_build_make(self):
+ self._test_devtool_build(self.myapp_dst)
+
+In this example, the ``devtool``
command is tested to see whether a sample application can be built with
the ``devtool build`` command within the eSDK.
@@ -426,14 +474,20 @@ the ``devtool build`` command within the eSDK.
These tests are run against built SDKs. The tests can assume that an SDK
has already been extracted and its environment file has been sourced. A
simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
-following:class Python3Test(OESDKTestCase): def setUp(self): if not
-(self.tc.hasHostPackage("nativesdk-python3-core") or
-self.tc.hasHostPackage("python3-core-native")): raise
-unittest.SkipTest("No python3 package in the SDK") def
-test_python3(self): cmd = "python3 -c \\"import codecs;
-print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd)
-self.assertEqual(output, "Hello, world\n")In this example, if
-nativesdk-python3-core has been installed into the SDK, the code runs
+following::
+
+ class Python3Test(OESDKTestCase):
+ def setUp(self):
+ if not (self.tc.hasHostPackage("nativesdk-python3-core") or
+ self.tc.hasHostPackage("python3-core-native")):
+ raise unittest.SkipTest("No python3 package in the SDK")
+
+ def test_python3(self):
+ cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+ output = self._run(cmd)
+ self.assertEqual(output, "Hello, world\n")
+
+In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
the python3 interpreter with a basic command to check it is working
correctly. The test would only run if python3 is installed in the SDK.
@@ -444,17 +498,25 @@ correctly. The test would only run if python3 is installed in the SDK.
The performance tests usually measure how long operations take and the
resource utilisation as that happens. An example from
-``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class
-Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake
--p)""" # Drop all caches and parse self.rm_cache()
-oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no
-caches)') # Drop tmp/cache
-oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no
-tmp/cache)') # Parse with fully cached data
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p
-(cached)')This example shows how three specific parsing timings are
+``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
+
+ class Test3(BuildPerfTestCase):
+ def test3(self):
+ """Bitbake parsing (bitbake -p)"""
+ # Drop all caches and parse
+ self.rm_cache()
+ oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
+ 'bitbake -p (no caches)')
+ # Drop tmp/cache
+ oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
+ 'bitbake -p (no tmp/cache)')
+ # Parse with fully cached data
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
+ 'bitbake -p (cached)')
+
+This example shows how three specific parsing timings are
measured, with and without various caches, to show how BitBake’s parsing
performance trends over time.
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst
index 8d3c84415..0d987d245 100644
--- a/documentation/test-manual/test-manual-understand-autobuilder.rst
+++ b/documentation/test-manual/test-manual-understand-autobuilder.rst
@@ -20,32 +20,48 @@ each of those targets on a seperate buildbot worker. To understand the
configuration, you need to look at the entry on ``config.json`` file
within the ``yocto-autobuilder-helper`` repository. The targets are
defined in the ‘overrides’ section, a quick example could be qemux86-64
-which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" :
-"arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic
-wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from
-the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" :
-true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato
-core-image-sato-dev core-image-sato-sdk core-image-minimal
-core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS"
-: "core-image-minimal:do_testimage core-image-sato:do_testimage
-core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2"
-: { "SDKMACHINE" : "x86_64", "BBTARGETS" :
-"core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext
-core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" :
-"core-image-sato:do_testsdk core-image-minimal:do_testsdkext
-core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false,
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest
-${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" :
-["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can
-see that “qemux86-64” is a three step build where the
-``bitbake BBTARGETS`` would be run, then ``bitbake
- SANITYTARGETS`` for each step; all for
+which looks like::
+
+ "qemux86-64" : {
+ "MACHINE" : "qemux86-64",
+ "TEMPLATE" : "arch-qemu",
+ "step1" : {
+ "extravars" : [
+ "IMAGE_FSTYPES_append = ' wic wic.bmap'"
+ ]
+ }
+ },
+
+And to expand that, you need the “arch-qemu” entry from
+the “templates” section, which looks like::
+
+ "arch-qemu" : {
+ "BUILDINFO" : true,
+ "BUILDHISTORY" : true,
+ "step1" : {
+ "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
+ "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
+ },
+ "step2" : {
+ "SDKMACHINE" : "x86_64",
+ "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
+ "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
+ },
+ "step3" : {
+ "BUILDHISTORY" : false,
+ "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
+ "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
+ }
+ },
+
+Combining these two entries you can see that “qemux86-64” is a three step build where the
+``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step
1 an extra variable is added to the ``auto.conf`` file to enable wic
image generation.
While not every detail of this is covered here, you can see how the
-templating mechanism allows quite complex configurations to be built up
+template mechanism allows quite complex configurations to be built up
yet allows duplication and repetition to be kept to a minimum.
The different build targets are designed to allow for parallelisation,
@@ -70,7 +86,7 @@ roughly consist of:
This cleans out any previous build. Old builds are left around to
allow easier debugging of failed builds. For additional information,
- see ```clobberdir`` <#test-clobberdir>`__.
+ see :ref:`test-clobberdir`.
2. *Obtain yocto-autobuilder-helper*
@@ -94,7 +110,7 @@ roughly consist of:
from the ``layerinfo.json`` file to help understand the
configuration. It will also use a local cache of repositories to
speed up the clone checkouts. For additional information, see
- `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__.
+ :ref:`test-autobuilder-clone-cache`.
This step has two possible modes of operation. If the build is part
of a parent build, its possible that all the repositories needed may
@@ -137,8 +153,7 @@ special script that moves files to a special location, rather than
deleting them. Files in this location are deleted by an ``rm`` command,
which is run under ``ionice -c 3``. For example, the deletion only
happens when there is idle IO capacity on the Worker. The Autobuilder
-Worker Janitor runs this deletion. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor runs this deletion. See :ref:`test-autobuilder-worker-janitor`.
.. _test-autobuilder-clone-cache:
@@ -150,8 +165,8 @@ on the Autobuilder. We therefore have a stash of commonly used
repositories pre-cloned on the Workers. Data is fetched from these
during clones first, then "topped up" with later revisions from any
upstream when necesary. The cache is maintained by the Autobuilder
-Worker Janitor. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor. See :ref:`Autobuilder Worker
+Janitor`.
.. _test-autobuilder-worker-janitor:
@@ -159,8 +174,8 @@ Autobuilder Worker Janitor
--------------------------
This is a process running on each Worker that performs two basic
-operations, including background file deletion at IO idle (see `Target
-Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and
+operations, including background file deletion at IO idle (see Target
+Execution: clobberdir :ref:`test-list-tgt-exec-clobberdir`) and
maintainenance of a cache of cloned repositories to improve the speed
the system can checkout repositories.
@@ -216,9 +231,6 @@ are general setup steps that are run once and include:
For each step that is configured in ``config.json``, it will perform the
following:
-## WRITER's question: What does "logging in as stepXa" and others refer
-to below? ##
-
1. Add any layers that are specified using the
``bitbake-layers add-layer`` command (logging as stepXa)
@@ -235,9 +247,8 @@ to below? ##
6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
BitBake build environment (logging as stepXd)
-7. Remove any layers added in `step
- 1 <#test-run-config-add-layers-step>`__ using the
- ``bitbake-layers remove-layer`` command (logging as stepXa)
+7. Remove any layers added in step
+ 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
Once the execution steps above complete, ``run-config`` executes a set
of post-build steps, including:
@@ -252,7 +263,7 @@ of post-build steps, including:
generated to the remote server.
4. Cleanup the build directory using
- ```clobberdir`` <#test-clobberdir>`__ if the build was successful,
+ :ref:`test-clobberdir` if the build was successful,
else rename it to “build-renamed” for potential future debugging.
.. _test-deploying-yp-autobuilder:
@@ -279,11 +290,18 @@ The standard ``config.json`` minimally attempts to allow substitution of
the paths. The Helper script repository includes a
``local-example.json`` file to show how you could override these from a
separate configuration file. Pass the following into the environment of
-the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As
-another example, you could also pass the following into the
-environment:$ ABHELPER_JSON="config.json /some/location/local.json"One
-issue users often run into is validation of the ``config.json`` files. A
+the Autobuilder::
+
+ $ ABHELPER_JSON="config.json local-example.json"
+
+As another example, you could also pass the following into the
+environment::
+
+ $ ABHELPER_JSON="config.json /some/location/local.json"
+
+One issue users often run into is validation of the ``config.json`` files. A
tip for minimizing issues from invalid json files is to use a Git
``pre-commit-hook.sh`` script to verify the JSON file before committing
-it. Create a symbolic link as follows:$ ln -s
-../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
+it. Create a symbolic link as follows::
+
+ $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
--
2.17.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [docs] [PATCH] test-manual code block, link, and format update
2020-09-14 23:11 [PATCH] test-manual code block, link, and format update Mark Morton
@ 2020-09-15 6:08 ` Nicolas Dechesne
2020-09-15 19:06 ` Mark Morton
0 siblings, 1 reply; 3+ messages in thread
From: Nicolas Dechesne @ 2020-09-15 6:08 UTC (permalink / raw)
To: Mark Morton; +Cc: docs
[-- Attachment #1: Type: text/plain, Size: 34334 bytes --]
hi Mark,
many thanks for your help!
On Tue, Sep 15, 2020 at 1:16 AM Mark Morton <Mark.Morton@windriver.com>
wrote:
> Signed-off-by: Mark Morton <mark.morton@windriver.com>
> ---
> documentation/conf.py | 4 +-
> .../test-manual/test-manual-intro.rst | 266 +++++++++++-------
> .../test-manual-understand-autobuilder.rst | 102 ++++---
> 3 files changed, 226 insertions(+), 146 deletions(-)
>
> diff --git a/documentation/conf.py b/documentation/conf.py
> index e1a7e5e3d..e825fbf6e 100644
> --- a/documentation/conf.py
> +++ b/documentation/conf.py
> @@ -42,8 +42,8 @@ sys.path.insert(0, os.path.abspath('sphinx'))
> extensions = [
> 'sphinx.ext.autosectionlabel',
> 'sphinx.ext.extlinks',
> - 'sphinx.ext.intersphinx',
> - 'yocto-vars'
> + 'sphinx.ext.intersphinx'
> + # 'yocto-vars'
> ]
>
This extension doesn't work with Python2. To be honest, I don't think i
want to bother with Python2 at this point. I am not even sure how well
Sphinx is expected to work with Python2, so we should make clear that our
doc system requires Python3, and we need to write clear instructions for
users who have Python3 on their host as 'default' vs users who still have
Python2 (which is the problem you faced)
autosectionlabel_prefix_document = True
>
> diff --git a/documentation/test-manual/test-manual-intro.rst
> b/documentation/test-manual/test-manual-intro.rst
> index 2f11e2beb..d68ab73f4 100644
> --- a/documentation/test-manual/test-manual-intro.rst
> +++ b/documentation/test-manual/test-manual-intro.rst
> @@ -23,26 +23,25 @@ project core.
>
> Currently, the Yocto Project Test Environment Manual has no projected
> release date. This manual is a work-in-progress and is being initially
> -loaded with information from the `README <>`__ files and notes from key
> +loaded with information from the README files and notes from key
> engineers:
>
> -- *``yocto-autobuilder2``:* This
> - ```README.md`
> http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
> +- *yocto-autobuilder2:* This
> + `README.md <
> http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
> >`__
>
that works. However for links to git trees, we are using the :yocto_git:
extlink , such as (from the BSP guide):
:yocto_git:`README.md </cgit.cgi/meta-raspberrypi/tree/README.md>`
We should standardize on that.
I think i will eventually make sure we don't even need /cgit.git in the
link name, but for now it's needed.
is the main README which detials how to set up the Yocto Project
> Autobuilder. The ``yocto-autobuilder2`` repository represents the
> Yocto Project's console UI plugin to Buildbot and the configuration
> necessary to configure Buildbot to perform the testing the project
> requires.
>
> -- *``yocto-autobuilder-helper``:* This
> - ```README`
> http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README
> +- *yocto-autobuilder-helper:* This `README <
> http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README/
> >`__
> and repository contains Yocto Project Autobuilder Helper scripts and
> configuration. The ``yocto-autobuilder-helper`` repository contains
> the "glue" logic that defines which tests to run and how to run them.
> As a result, it can be used by any Continuous Improvement (CI) system
> to run builds, support getting the correct code revisions, configure
> builds and layers, run builds, and collect results. The code is
> - independent of any CI system, which means the code can work Buildbot,
> + independent of any CI system, which means the code can work `Buildbot <
> https://docs.buildbot.net/0.9.15.post1/>`__,
>
this is a new link you created which was not in the original Docbook,
right? Perhaps best to have that in a separate patch if you think it's
needed.
> Jenkins, or others. This repository has a branch per release of the
> project defining the tests to run on a per release basis.
>
> @@ -66,9 +65,9 @@ that the Yocto Project customizes using code from the
> resulting UI plug-in allows you to visualize builds in a way suited to
> the project's needs.
>
> -A ``helper`` layer provides configuration and job management through
> +A helper layer provides configuration and job management through
>
Why? 'helper' is inside a <filename> tag in docbook to make it verbatim in
the output, why did you remove it?
scripts found in the ``yocto-autobuilder-helper`` repository. The
> -``helper`` layer contains the bulk of the build configuration
> +helper layer contains the bulk of the build configuration
> information and is release-specific, which makes it highly customizable
> on a per-project basis. The layer is CI system-agnostic and contains a
> number of Helper scripts that can generate build configurations from
> @@ -124,18 +123,21 @@ thefollowing types of tests:
> ensure we have no single point of failure and can ensure the
> different distros work effectively.
>
> -- *eSDK Testing:* Image tests initiated through the following command:
> - $ bitbake image -c testsdkext The tests utilize the ``testsdkext``
> - class and the ``do_testsdkext`` task.
> +- *eSDK Testing:* Image tests initiated through the following command::
>
does that work? I've always done ": ::" to keep the semi column, and I
thought the :: needed a space before.
> +
> + $ bitbake image -c testsdkext
>
Indentation is not typical here. you need 3 space starting from the
previous indentation level.
> +
> + The tests utilize the ``testsdkext`` class and the ``do_testsdkext``
> task.
>
this sentence above is no in the code-block, it should not be at the same
indentation level as the code block.
>
> - *Feature Testing:* Various scenario-based tests are run through the
> - `OpenEmbedded
> - Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__
> - (oe-selftest). We test oe-selftest on each of the main distrubutions
> + :ref:`OpenEmbedded Self test (oe-selftest)
> <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test
> oe-selftest on each of the main distrubutions
> we support.
>
> -- *Image Testing:* Image tests initiated through the following command:
> - $ bitbake image -c testimage The tests utilize the
> +- *Image Testing:* Image tests initiated through the following command::
> +
> + $ bitbake image -c testimage
> +
> + The tests utilize the
>
Need to fix indentation here
:ref:`testimage* <ref-classes-testimage*>`
> classes and the
> :ref:`ref-tasks-testimage` task.
> @@ -147,8 +149,8 @@ thefollowing types of tests:
>
> - *Package Testing:* A Package Test (ptest) runs tests against packages
> built by the OpenEmbedded build system on the target machine. See the
> - "`Testing Packages With
> - ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section
> + :ref:`Testing Packages With
> + ptest <dev-manual/dev-manual-common-tasks:Testing Packages With
> ptest>` section
> in the Yocto Project Development Tasks Manual and the
> ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more
> information on Ptest.
> @@ -159,8 +161,8 @@ thefollowing types of tests:
> the ``do_testsdk`` task.
>
> - *Unit Testing:* Unit tests on various components of the system run
> - through ``oe-selftest`` and
> - ```bitbake-selftest``
> <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__.
> + through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing
> and Quality Assurance>` and
> + :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality
> Assurance>`.
>
the link is on bitbake-selftest, not oe-selftest (you added it).
>
> - *Automatic Upgrade Helper:* This target tests whether new versions of
> software are available and whether we can automatically upgrade to
> @@ -174,36 +176,43 @@ How Tests Map to Areas of Code
>
> Tests map into the codebase as follows:
>
> -- *bitbake-selftest*:
> +- *bitbake-selftest:*
>
> These tests are self-contained and test BitBake as well as its APIs,
> which include the fetchers. The tests are located in
> ``bitbake/lib/*/tests``.
>
> - From within the BitBake repository, run the following: $
> - bitbake-selftest
> + From within the BitBake repository, run the following::
> +
> + $ bitbake-selftest
>
indentation and use of :: are not right (please check all other occurrences)
>
> To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
> - variable when running "bitbake-selftest" as follows: $
> - BB_SKIP_NETTEST=yes bitbake-selftest
> + variable when running "bitbake-selftest" as follows::
> +
> + $ BB_SKIP_NETTEST=yes bitbake-selftest
>
> The default output is quiet and just prints a summary of what was
> - run. To see more information, there is a verbose option:$
> - bitbake-selftest -v
> + run. To see more information, there is a verbose option::
> +
> + $ bitbake-selftest -v
>
> Use this option when you wish to skip tests that access the network,
> which are mostly necessary to test the fetcher modules. To specify
> individual test modules to run, append the test module name to the
> "bitbake-selftest" command. For example, to specify the tests for the
> - bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can
> - also specify individual tests by defining the full name and module
> - plus the class path of the test, for example: $ bitbake-selftest
> - bb.tests.data.TestOverrides.test_one_override
> + bb.data.module, run::
> +
> + $ bitbake-selftest bb.test.data.module
> +
> + You can also specify individual tests by defining the full name and
> module
> + plus the class path of the test, for example::
> +
> + $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
>
> The tests are based on `Python
> unittest <https://docs.python.org/3/library/unittest.html>`__.
>
> -- *oe-selftest*:
> +- *oe-selftest:*
>
> - These tests use OE to test the workflows, which include testing
> specific features, behaviors of tasks, and API unit tests.
> @@ -219,16 +228,25 @@ Tests map into the codebase as follows:
> - The code for the tests resides in
> ``meta/lib/oeqa/selftest/cases/``.
>
> - - To run all the tests, enter the following command: $ oe-selftest
> - -a
> + - To run all the tests, enter the following command::
> +
> + $ oe-selftest -a
>
> - To run a specific test, use the following command form where
> - testname is the name of the specific test: $ oe-selftest -r
> - testname For example, the following command would run the tinfoil
> - getVar API test:$ oe-selftest -r
> - tinfoil.TinfoilTests.test_getvarIt is also possible to run a set
> + testname is the name of the specific test::
> +
> + $ oe-selftest -r <testname>
> +
> + For example, the following command would run the tinfoil
> + getVar API test::
> +
> + $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
> +
> + It is also possible to run a set
> of tests. For example the following command will run all of the
> - tinfoil tests:$ oe-selftest -r tinfoil
> + tinfoil tests::
> +
> + $ oe-selftest -r tinfoil
>
> - *testimage:*
>
> @@ -240,10 +258,13 @@ Tests map into the codebase as follows:
>
> - You need to set the
> :term:`IMAGE_CLASSES`
> - variable as follows: IMAGE_CLASSES += "testimage"
> + variable as follows::
> +
> + IMAGE_CLASSES += "testimage"
>
> - - Run the tests using the following command form: $ bitbake image -c
> - testimage
> + - Run the tests using the following command form::
> +
> + $ bitbake image -c testimage
>
> - *testsdk:*
>
> @@ -252,8 +273,9 @@ Tests map into the codebase as follows:
>
> - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
>
> - - Run the test using the following command form: $ bitbake image -c
> - testsdk
> + - Run the test using the following command form::
> +
> + $ bitbake image -c testsdk
>
> - *testsdk_ext:*
>
> @@ -262,8 +284,9 @@ Tests map into the codebase as follows:
>
> - The code for these tests resides in ``meta/lib/oeqa/esdk``.
>
> - - To run the tests, use the following command form: $ bitbake image
> - -c testsdkext
> + - To run the tests, use the following command form::
> +
> + $ bitbake image -c testsdkext
>
> - *oe-build-perf-test:*
>
> @@ -272,8 +295,11 @@ Tests map into the codebase as follows:
>
> - The code for these tests resides in ``meta/lib/oeqa/buildperf``.
>
> - - To run the tests, use the following command form: $
> - oe-build-perf-test optionsThe command takes a number of options,
> + - To run the tests, use the following command form::
> +
> + $ oe-build-perf-test <options>
> +
> + The command takes a number of options,
> such as where to place the test results. The Autobuilder Helper
> Scripts include the ``build-perf-test-wrapper`` script with
> examples of how to use the oe-build-perf-test from the command
> @@ -308,14 +334,20 @@ directory.
> ``bitbake-selftest``
> --------------------
>
> -A simple test example from ``lib/bb/tests/data.py`` is: class
> -DataExpansions(unittest.TestCase): def setUp(self): self.d =
> -bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] =
> -"value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def
> -test_one_var(self): val = self.d.expand("${foo}")
> -self.assertEqual(str(val), "value_of_foo")
> +A simple test example from ``lib/bb/tests/data.py`` is::
>
> -In this example, a ```DataExpansions`` <>`__ class of tests is created,
> + class DataExpansions(unittest.TestCase):
> + def setUp(self):
> + self.d = bb.data.init()
> + self.d["foo"] = "value_of_foo"
> + self.d["bar"] = "value_of_bar"
> + self.d["value_of_foo"] = "value_of_'value_of_foo'"
> +
> + def test_one_var(self):
> + val = self.d.expand("${foo}")
> + self.assertEqual(str(val), "value_of_foo")
> +
> +In this example, a ``DataExpansions`` class of tests is created,
> derived from standard python unittest. The class has a common ``setUp``
> function which is shared by all the tests in the class. A simple test is
> then added to test that when a variable is expanded, the correct value
> @@ -334,14 +366,15 @@ These tests are more complex due to the setup
> required behind the scenes
> for full builds. Rather than directly using Python's unittest, the code
> wraps most of the standard objects. The tests can be simple, such as
> testing a command from within the OE build environment using the
> -following example:class BitbakeLayers(OESelftestTestCase): def
> -test_bitbakelayers_showcrossdepends(self): result =
> -runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in
> -result.output, msg = "No dependencies were shown. bitbake-layers
> -show-cross-depends output: %s"% result.output)
> +following example::
> +
> + class BitbakeLayers(OESelftestTestCase):
> + def test_bitbakelayers_showcrossdepends(self):
> + result = runCmd('bitbake-layers show-cross-depends')
> + self.assertTrue('aspell' in result.output, msg = "No
> dependencies were shown. bitbake-layers show-cross-depends output: %s"%
> result.output)
>
> This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
> -creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived
> +creates a testcase from the ``OESelftestTestCase`` class, derived
> from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
> and checks the output to ensure it contains something we know should be
> here.
> @@ -367,7 +400,7 @@ so tests within a given test class should always run
> in the same build,
> while tests in different classes or modules may be split into different
> builds. There is no data store available for these tests since the tests
> launch the ``bitbake`` command and exist outside of its context. As a
> -result, common bitbake library functions (bb.*) are also unavailable.
> +result, common bitbake library functions (bb.\*) are also unavailable.
>
> .. _testimage-example:
>
> @@ -378,14 +411,18 @@ These tests are run once an image is up and running,
> either on target
> hardware or under QEMU. As a result, they are assumed to be running in a
> target image environment, as opposed to a host build environment. A
> simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
> -the following:class PythonTest(OERuntimeTestCase):
> -@OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core'])
> -def test_python3(self): cmd = "python3 -c \\"import codecs;
> -print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output =
> -self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output
> -self.assertEqual(status, 0, msg=msg)
> -
> -In this example, the ```OERuntimeTestCase`` <>`__ class wraps
> +the following::
> +
> + class PythonTest(OERuntimeTestCase):
> + @OETestDepends(['ssh.SSHTest.test_ssh'])
> + @OEHasPackage(['python3-core'])
> + def test_python3(self):
> + cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb,
> jbeyq', 'rot13'))\""
> + status, output = self.target.run(cmd)
> + msg = 'Exit status was not 0. Output: %s' % output
> + self.assertEqual(status, 0, msg=msg)
> +
> +In this example, the ``OERuntimeTestCase`` class wraps
> ``unittest.TestCase``. Within the test, ``self.target`` represents the
> target system, where commands can be run on it using the ``run()``
> method.
> @@ -402,19 +439,30 @@ the image.
>
> These tests are run against built extensible SDKs (eSDKs). The tests can
> assume that the eSDK environment has already been setup. An example from
> -``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class
> -DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls):
> -myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst =
> -os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src,
> -cls.myapp_dst) subprocess.check_output(['git', 'init', '.'],
> -cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'],
> -cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m',
> -"'test commit'"], cwd=cls.myapp_dst) @classmethod def
> -tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def
> -\_test_devtool_build(self, directory): self._run('devtool add myapp %s'
> -% directory) try: self._run('devtool build myapp') finally:
> -self._run('devtool reset myapp') def test_devtool_build_make(self):
> -self._test_devtool_build(self.myapp_dst)In this example, the ``devtool``
> +``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
> +
> + class DevtoolTest(OESDKExtTestCase):
> + @classmethod def setUpClass(cls):
> + myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
> + cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
> + shutil.copytree(myapp_src, cls.myapp_dst)
> + subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
> + subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
> + subprocess.check_output(['git', 'commit', '-m', "'test
> commit'"], cwd=cls.myapp_dst)
> +
> + @classmethod
> + def tearDownClass(cls):
> + shutil.rmtree(cls.myapp_dst)
> + def _test_devtool_build(self, directory):
> + self._run('devtool add myapp %s' % directory)
> + try:
> + self._run('devtool build myapp')
> + finally:
> + self._run('devtool reset myapp')
> + def test_devtool_build_make(self):
> + self._test_devtool_build(self.myapp_dst)
> +
> +In this example, the ``devtool``
> command is tested to see whether a sample application can be built with
> the ``devtool build`` command within the eSDK.
>
> @@ -426,14 +474,20 @@ the ``devtool build`` command within the eSDK.
> These tests are run against built SDKs. The tests can assume that an SDK
> has already been extracted and its environment file has been sourced. A
> simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
> -following:class Python3Test(OESDKTestCase): def setUp(self): if not
> -(self.tc.hasHostPackage("nativesdk-python3-core") or
> -self.tc.hasHostPackage("python3-core-native")): raise
> -unittest.SkipTest("No python3 package in the SDK") def
> -test_python3(self): cmd = "python3 -c \\"import codecs;
> -print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd)
> -self.assertEqual(output, "Hello, world\n")In this example, if
> -nativesdk-python3-core has been installed into the SDK, the code runs
> +following::
> +
> + class Python3Test(OESDKTestCase):
> + def setUp(self):
> + if not (self.tc.hasHostPackage("nativesdk-python3-core") or
> + self.tc.hasHostPackage("python3-core-native")):
> + raise unittest.SkipTest("No python3 package in the SDK")
> +
> + def test_python3(self):
> + cmd = "python3 -c \\"import codecs;
> print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
> + output = self._run(cmd)
> + self.assertEqual(output, "Hello, world\n")
> +
> +In this example, if nativesdk-python3-core has been installed into the
> SDK, the code runs
> the python3 interpreter with a basic command to check it is working
> correctly. The test would only run if python3 is installed in the SDK.
>
> @@ -444,17 +498,25 @@ correctly. The test would only run if python3 is
> installed in the SDK.
>
> The performance tests usually measure how long operations take and the
> resource utilisation as that happens. An example from
> -``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class
> -Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake
> --p)""" # Drop all caches and parse self.rm_cache()
> -oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
> -self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no
> -caches)') # Drop tmp/cache
> -oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
> -self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no
> -tmp/cache)') # Parse with fully cached data
> -self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p
> -(cached)')This example shows how three specific parsing timings are
> +``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
> +
> + class Test3(BuildPerfTestCase):
> + def test3(self):
> + """Bitbake parsing (bitbake -p)"""
> + # Drop all caches and parse
> + self.rm_cache()
> + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'),
> True)
> + self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
> + 'bitbake -p (no caches)')
> + # Drop tmp/cache
> + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'),
> True)
> + self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
> + 'bitbake -p (no tmp/cache)')
> + # Parse with fully cached data
> + self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
> + 'bitbake -p (cached)')
> +
> +This example shows how three specific parsing timings are
> measured, with and without various caches, to show how BitBake’s parsing
> performance trends over time.
>
> diff --git
> a/documentation/test-manual/test-manual-understand-autobuilder.rst
> b/documentation/test-manual/test-manual-understand-autobuilder.rst
> index 8d3c84415..0d987d245 100644
> --- a/documentation/test-manual/test-manual-understand-autobuilder.rst
> +++ b/documentation/test-manual/test-manual-understand-autobuilder.rst
> @@ -20,32 +20,48 @@ each of those targets on a seperate buildbot worker.
> To understand the
> configuration, you need to look at the entry on ``config.json`` file
> within the ``yocto-autobuilder-helper`` repository. The targets are
> defined in the ‘overrides’ section, a quick example could be qemux86-64
> -which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" :
> -"arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic
> -wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from
> -the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" :
> -true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato
> -core-image-sato-dev core-image-sato-sdk core-image-minimal
> -core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS"
> -: "core-image-minimal:do_testimage core-image-sato:do_testimage
> -core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2"
> -: { "SDKMACHINE" : "x86_64", "BBTARGETS" :
> -"core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext
> -core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" :
> -"core-image-sato:do_testsdk core-image-minimal:do_testsdkext
> -core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false,
> -"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest
> -${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" :
> -["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can
> -see that “qemux86-64” is a three step build where the
> -``bitbake BBTARGETS`` would be run, then ``bitbake
> - SANITYTARGETS`` for each step; all for
> +which looks like::
> +
> + "qemux86-64" : {
> + "MACHINE" : "qemux86-64",
> + "TEMPLATE" : "arch-qemu",
> + "step1" : {
> + "extravars" : [
> + "IMAGE_FSTYPES_append = ' wic wic.bmap'"
> + ]
> + }
> + },
> +
> +And to expand that, you need the “arch-qemu” entry from
> +the “templates” section, which looks like::
> +
> + "arch-qemu" : {
> + "BUILDINFO" : true,
> + "BUILDHISTORY" : true,
> + "step1" : {
> + "BBTARGETS" : "core-image-sato core-image-sato-dev
> core-image-sato-sdk core-image-minimal core-image-minimal-dev
> core-image-sato:do_populate_sdk",
> + "SANITYTARGETS" : "core-image-minimal:do_testimage
> core-image-sato:do_testimage core-image-sato-sdk:do_testimage
> core-image-sato:do_testsdk"
> + },
> + "step2" : {
> + "SDKMACHINE" : "x86_64",
> + "BBTARGETS" : "core-image-sato:do_populate_sdk
> core-image-minimal:do_populate_sdk_ext
> core-image-sato:do_populate_sdk_ext",
> + "SANITYTARGETS" : "core-image-sato:do_testsdk
> core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
> + },
> + "step3" : {
> + "BUILDHISTORY" : false,
> + "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1
> oe-selftest ${HELPERSTMACHTARGS} -j 15"],
> + "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
> + }
> + },
> +
> +Combining these two entries you can see that “qemux86-64” is a three step
> build where the
> +``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for
> each step; all for
> ``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step
> 1 an extra variable is added to the ``auto.conf`` file to enable wic
> image generation.
>
> While not every detail of this is covered here, you can see how the
> -templating mechanism allows quite complex configurations to be built up
> +template mechanism allows quite complex configurations to be built up
>
why? let's stick to the original .xml file, we can fix the doc later.
> yet allows duplication and repetition to be kept to a minimum.
>
> The different build targets are designed to allow for parallelisation,
> @@ -70,7 +86,7 @@ roughly consist of:
>
> This cleans out any previous build. Old builds are left around to
> allow easier debugging of failed builds. For additional information,
> - see ```clobberdir`` <#test-clobberdir>`__.
> + see :ref:`test-clobberdir`.
>
It works. However we are now using autosectionlabel which creates sections
labels for headings automatically. So I would like to get rid of all custom
labels, as much as possible. the following is preferred:
:ref:`test-manual/test-manual-understand-autobuilder:clobberdir>`
I know we've left some links like that in other manuals. So it's not
strictly required to change it now.
> 2. *Obtain yocto-autobuilder-helper*
>
> @@ -94,7 +110,7 @@ roughly consist of:
> from the ``layerinfo.json`` file to help understand the
> configuration. It will also use a local cache of repositories to
> speed up the clone checkouts. For additional information, see
> - `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__.
> + :ref:`test-autobuilder-clone-cache`.
>
same here (and probably in other places)
>
> This step has two possible modes of operation. If the build is part
> of a parent build, its possible that all the repositories needed may
> @@ -137,8 +153,7 @@ special script that moves files to a special location,
> rather than
> deleting them. Files in this location are deleted by an ``rm`` command,
> which is run under ``ionice -c 3``. For example, the deletion only
> happens when there is idle IO capacity on the Worker. The Autobuilder
> -Worker Janitor runs this deletion. See `Autobuilder Worker
> -Janitor <#test-autobuilder-worker-janitor>`__.
> +Worker Janitor runs this deletion. See
> :ref:`test-autobuilder-worker-janitor`.
>
> .. _test-autobuilder-clone-cache:
>
> @@ -150,8 +165,8 @@ on the Autobuilder. We therefore have a stash of
> commonly used
> repositories pre-cloned on the Workers. Data is fetched from these
> during clones first, then "topped up" with later revisions from any
> upstream when necesary. The cache is maintained by the Autobuilder
> -Worker Janitor. See `Autobuilder Worker
> -Janitor <#test-autobuilder-worker-janitor>`__.
> +Worker Janitor. See :ref:`Autobuilder Worker
> +Janitor`.
>
> .. _test-autobuilder-worker-janitor:
>
> @@ -159,8 +174,8 @@ Autobuilder Worker Janitor
> --------------------------
>
> This is a process running on each Worker that performs two basic
> -operations, including background file deletion at IO idle (see `Target
> -Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and
> +operations, including background file deletion at IO idle (see Target
> +Execution: clobberdir :ref:`test-list-tgt-exec-clobberdir`) and
>
the link and text look wrong here. the link is not on the 'text'
> maintainenance of a cache of cloned repositories to improve the speed
> the system can checkout repositories.
>
> @@ -216,9 +231,6 @@ are general setup steps that are run once and include:
> For each step that is configured in ``config.json``, it will perform the
> following:
>
> -## WRITER's question: What does "logging in as stepXa" and others refer
> -to below? ##
>
you can keep comment with the ".. xxx" syntax.
> -
> 1. Add any layers that are specified using the
> ``bitbake-layers add-layer`` command (logging as stepXa)
>
> @@ -235,9 +247,8 @@ to below? ##
> 6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
> BitBake build environment (logging as stepXd)
>
> -7. Remove any layers added in `step
> - 1 <#test-run-config-add-layers-step>`__ using the
> - ``bitbake-layers remove-layer`` command (logging as stepXa)
> +7. Remove any layers added in step
> + 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
>
you removed a link here, no?
>
> Once the execution steps above complete, ``run-config`` executes a set
> of post-build steps, including:
> @@ -252,7 +263,7 @@ of post-build steps, including:
> generated to the remote server.
>
> 4. Cleanup the build directory using
> - ```clobberdir`` <#test-clobberdir>`__ if the build was successful,
> + :ref:`test-clobberdir` if the build was successful,
> else rename it to “build-renamed” for potential future debugging.
>
> .. _test-deploying-yp-autobuilder:
> @@ -279,11 +290,18 @@ The standard ``config.json`` minimally attempts to
> allow substitution of
> the paths. The Helper script repository includes a
> ``local-example.json`` file to show how you could override these from a
> separate configuration file. Pass the following into the environment of
> -the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As
> -another example, you could also pass the following into the
> -environment:$ ABHELPER_JSON="config.json /some/location/local.json"One
> -issue users often run into is validation of the ``config.json`` files. A
> +the Autobuilder::
> +
> + $ ABHELPER_JSON="config.json local-example.json"
> +
> +As another example, you could also pass the following into the
> +environment::
> +
> + $ ABHELPER_JSON="config.json /some/location/local.json"
> +
> +One issue users often run into is validation of the ``config.json``
> files. A
> tip for minimizing issues from invalid json files is to use a Git
> ``pre-commit-hook.sh`` script to verify the JSON file before committing
> -it. Create a symbolic link as follows:$ ln -s
> -../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
> +it. Create a symbolic link as follows::
> +
> + $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
> --
> 2.17.1
>
>
>
[-- Attachment #2: Type: text/html, Size: 42833 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [docs] [PATCH] test-manual code block, link, and format update
2020-09-15 6:08 ` [docs] " Nicolas Dechesne
@ 2020-09-15 19:06 ` Mark Morton
0 siblings, 0 replies; 3+ messages in thread
From: Mark Morton @ 2020-09-15 19:06 UTC (permalink / raw)
To: Nicolas Dechesne; +Cc: docs
[-- Attachment #1: Type: text/plain, Size: 34617 bytes --]
From: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Sent: Monday, September 14, 2020 11:08 PM
To: Morton, Mark <Mark.Morton@windriver.com>
Cc: docs@lists.yoctoproject.org
Subject: Re: [docs] [PATCH] test-manual code block, link, and format update
hi Mark,
many thanks for your help!
On Tue, Sep 15, 2020 at 1:16 AM Mark Morton <Mark.Morton@windriver.com<mailto:Mark.Morton@windriver.com>> wrote:
Signed-off-by: Mark Morton <mark.morton@windriver.com<mailto:mark.morton@windriver.com>>
---
documentation/conf.py | 4 +-
.../test-manual/test-manual-intro.rst | 266 +++++++++++-------
.../test-manual-understand-autobuilder.rst | 102 ++++---
3 files changed, 226 insertions(+), 146 deletions(-)
diff --git a/documentation/conf.py b/documentation/conf.py
index e1a7e5e3d..e825fbf6e 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -42,8 +42,8 @@ sys.path.insert(0, os.path.abspath('sphinx'))
extensions = [
'sphinx.ext.autosectionlabel',
'sphinx.ext.extlinks',
- 'sphinx.ext.intersphinx',
- 'yocto-vars'
+ 'sphinx.ext.intersphinx'
+ # 'yocto-vars'
]
This extension doesn't work with Python2. To be honest, I don't think i want to bother with Python2 at this point. I am not even sure how well Sphinx is expected to work with Python2, so we should make clear that our doc system requires Python3, and we need to write clear instructions for users who have Python3 on their host as 'default' vs users who still have Python2 (which is the problem you faced)
autosectionlabel_prefix_document = True
diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst
index 2f11e2beb..d68ab73f4 100644
--- a/documentation/test-manual/test-manual-intro.rst
+++ b/documentation/test-manual/test-manual-intro.rst
@@ -23,26 +23,25 @@ project core.
Currently, the Yocto Project Test Environment Manual has no projected
release date. This manual is a work-in-progress and is being initially
-loaded with information from the `README <>`__ files and notes from key
+loaded with information from the README files and notes from key
engineers:
-- *``yocto-autobuilder2``:* This
- ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
+- *yocto-autobuilder2:* This
+ `README.md <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md>`__
that works. However for links to git trees, we are using the :yocto_git: extlink , such as (from the BSP guide):
:yocto_git:`README.md </cgit.cgi/meta-raspberrypi/tree/README.md>`
We should standardize on that.
I think i will eventually make sure we don't even need /cgit.git in the link name, but for now it's needed.
I updated both links to use :yocto-git:, however because of the Python2 issue, it does not generate in my output. If you have issues, let me know.
is the main README which detials how to set up the Yocto Project
Autobuilder. The ``yocto-autobuilder2`` repository represents the
Yocto Project's console UI plugin to Buildbot and the configuration
necessary to configure Buildbot to perform the testing the project
requires.
-- *``yocto-autobuilder-helper``:* This
- ```README`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README
+- *yocto-autobuilder-helper:* This `README <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README/>`__
and repository contains Yocto Project Autobuilder Helper scripts and
configuration. The ``yocto-autobuilder-helper`` repository contains
the "glue" logic that defines which tests to run and how to run them.
As a result, it can be used by any Continuous Improvement (CI) system
to run builds, support getting the correct code revisions, configure
builds and layers, run builds, and collect results. The code is
- independent of any CI system, which means the code can work Buildbot,
+ independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
this is a new link you created which was not in the original Docbook, right? Perhaps best to have that in a separate patch if you think it's needed.
It was in the original – It actually converted just like that with no additional help from me.
Jenkins, or others. This repository has a branch per release of the
project defining the tests to run on a per release basis.
@@ -66,9 +65,9 @@ that the Yocto Project customizes using code from the
resulting UI plug-in allows you to visualize builds in a way suited to
the project's needs.
-A ``helper`` layer provides configuration and job management through
+A helper layer provides configuration and job management through
Why? 'helper' is inside a <filename> tag in docbook to make it verbatim in the output, why did you remove it?
Not really sure why – I thought it was general editing, but on closer inspection, I should have left it in, as it’s the name of a valid layer.
scripts found in the ``yocto-autobuilder-helper`` repository. The
-``helper`` layer contains the bulk of the build configuration
+helper layer contains the bulk of the build configuration
information and is release-specific, which makes it highly customizable
on a per-project basis. The layer is CI system-agnostic and contains a
number of Helper scripts that can generate build configurations from
@@ -124,18 +123,21 @@ thefollowing types of tests:
ensure we have no single point of failure and can ensure the
different distros work effectively.
-- *eSDK Testing:* Image tests initiated through the following command:
- $ bitbake image -c testsdkext The tests utilize the ``testsdkext``
- class and the ``do_testsdkext`` task.
+- *eSDK Testing:* Image tests initiated through the following command::
does that work? I've always done ": ::" to keep the semi column, and I thought the :: needed a space before.
It’s shorthand, and works so long as the sentence ends with a colon (:)
+
+ $ bitbake image -c testsdkext
Indentation is not typical here. you need 3 space starting from the previous indentation level.
I missed this (and the following one) – good catch.
+
+ The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task.
this sentence above is no in the code-block, it should not be at the same indentation level as the code block.
- *Feature Testing:* Various scenario-based tests are run through the
- `OpenEmbedded
- Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__
- (oe-selftest). We test oe-selftest on each of the main distrubutions
+ :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions
we support.
-- *Image Testing:* Image tests initiated through the following command:
- $ bitbake image -c testimage The tests utilize the
+- *Image Testing:* Image tests initiated through the following command::
+
+ $ bitbake image -c testimage
+
+ The tests utilize the
Need to fix indentation here
Done
:ref:`testimage* <ref-classes-testimage*>`
classes and the
:ref:`ref-tasks-testimage` task.
@@ -147,8 +149,8 @@ thefollowing types of tests:
- *Package Testing:* A Package Test (ptest) runs tests against packages
built by the OpenEmbedded build system on the target machine. See the
- "`Testing Packages With
- ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section
+ :ref:`Testing Packages With
+ ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section
in the Yocto Project Development Tasks Manual and the
":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more
information on Ptest.
@@ -159,8 +161,8 @@ thefollowing types of tests:
the ``do_testsdk`` task.
- *Unit Testing:* Unit tests on various components of the system run
- through ``oe-selftest`` and
- ```bitbake-selftest`` <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__.
+ through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and
+ :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`.
the link is on bitbake-selftest, not oe-selftest (you added it).
They both point to the same location, so to be consistent, I added a link for both, and rearranged them to be in alphabetical order.
- *Automatic Upgrade Helper:* This target tests whether new versions of
software are available and whether we can automatically upgrade to
@@ -174,36 +176,43 @@ How Tests Map to Areas of Code
Tests map into the codebase as follows:
-- *bitbake-selftest*:
+- *bitbake-selftest:*
These tests are self-contained and test BitBake as well as its APIs,
which include the fetchers. The tests are located in
``bitbake/lib/*/tests``.
- From within the BitBake repository, run the following: $
- bitbake-selftest
+ From within the BitBake repository, run the following::
+
+ $ bitbake-selftest
indentation and use of :: are not right (please check all other occurrences)
Per our discussion, I checked indentation only. It is three spaces, but I see a slight indentation in the output. This might need a CSS tweak.
To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
- variable when running "bitbake-selftest" as follows: $
- BB_SKIP_NETTEST=yes bitbake-selftest
+ variable when running "bitbake-selftest" as follows::
+
+ $ BB_SKIP_NETTEST=yes bitbake-selftest
The default output is quiet and just prints a summary of what was
- run. To see more information, there is a verbose option:$
- bitbake-selftest -v
+ run. To see more information, there is a verbose option::
+
+ $ bitbake-selftest -v
Use this option when you wish to skip tests that access the network,
which are mostly necessary to test the fetcher modules. To specify
individual test modules to run, append the test module name to the
"bitbake-selftest" command. For example, to specify the tests for the
- bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can
- also specify individual tests by defining the full name and module
- plus the class path of the test, for example: $ bitbake-selftest
- bb.tests.data.TestOverrides.test_one_override
+ bb.data.module, run::
+
+ $ bitbake-selftest bb.test.data.module
+
+ You can also specify individual tests by defining the full name and module
+ plus the class path of the test, for example::
+
+ $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
The tests are based on `Python
unittest <https://docs.python.org/3/library/unittest.html>`__.
-- *oe-selftest*:
+- *oe-selftest:*
- These tests use OE to test the workflows, which include testing
specific features, behaviors of tasks, and API unit tests.
@@ -219,16 +228,25 @@ Tests map into the codebase as follows:
- The code for the tests resides in
``meta/lib/oeqa/selftest/cases/``.
- - To run all the tests, enter the following command: $ oe-selftest
- -a
+ - To run all the tests, enter the following command::
+
+ $ oe-selftest -a
- To run a specific test, use the following command form where
- testname is the name of the specific test: $ oe-selftest -r
- testname For example, the following command would run the tinfoil
- getVar API test:$ oe-selftest -r
- tinfoil.TinfoilTests.test_getvarIt is also possible to run a set
+ testname is the name of the specific test::
+
+ $ oe-selftest -r <testname>
+
+ For example, the following command would run the tinfoil
+ getVar API test::
+
+ $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
+
+ It is also possible to run a set
of tests. For example the following command will run all of the
- tinfoil tests:$ oe-selftest -r tinfoil
+ tinfoil tests::
+
+ $ oe-selftest -r tinfoil
- *testimage:*
@@ -240,10 +258,13 @@ Tests map into the codebase as follows:
- You need to set the
:term:`IMAGE_CLASSES`
- variable as follows: IMAGE_CLASSES += "testimage"
+ variable as follows::
+
+ IMAGE_CLASSES += "testimage"
- - Run the tests using the following command form: $ bitbake image -c
- testimage
+ - Run the tests using the following command form::
+
+ $ bitbake image -c testimage
- *testsdk:*
@@ -252,8 +273,9 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
- - Run the test using the following command form: $ bitbake image -c
- testsdk
+ - Run the test using the following command form::
+
+ $ bitbake image -c testsdk
- *testsdk_ext:*
@@ -262,8 +284,9 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/esdk``.
- - To run the tests, use the following command form: $ bitbake image
- -c testsdkext
+ - To run the tests, use the following command form::
+
+ $ bitbake image -c testsdkext
- *oe-build-perf-test:*
@@ -272,8 +295,11 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/buildperf``.
- - To run the tests, use the following command form: $
- oe-build-perf-test optionsThe command takes a number of options,
+ - To run the tests, use the following command form::
+
+ $ oe-build-perf-test <options>
+
+ The command takes a number of options,
such as where to place the test results. The Autobuilder Helper
Scripts include the ``build-perf-test-wrapper`` script with
examples of how to use the oe-build-perf-test from the command
@@ -308,14 +334,20 @@ directory.
``bitbake-selftest``
--------------------
-A simple test example from ``lib/bb/tests/data.py`` is: class
-DataExpansions(unittest.TestCase): def setUp(self): self.d =
-bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] =
-"value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def
-test_one_var(self): val = self.d.expand("${foo}")
-self.assertEqual(str(val), "value_of_foo")
+A simple test example from ``lib/bb/tests/data.py`` is::
-In this example, a ```DataExpansions`` <>`__ class of tests is created,
+ class DataExpansions(unittest.TestCase):
+ def setUp(self):
+ self.d = bb.data.init()
+ self.d["foo"] = "value_of_foo"
+ self.d["bar"] = "value_of_bar"
+ self.d["value_of_foo"] = "value_of_'value_of_foo'"
+
+ def test_one_var(self):
+ val = self.d.expand("${foo}")
+ self.assertEqual(str(val), "value_of_foo")
+
+In this example, a ``DataExpansions`` class of tests is created,
derived from standard python unittest. The class has a common ``setUp``
function which is shared by all the tests in the class. A simple test is
then added to test that when a variable is expanded, the correct value
@@ -334,14 +366,15 @@ These tests are more complex due to the setup required behind the scenes
for full builds. Rather than directly using Python's unittest, the code
wraps most of the standard objects. The tests can be simple, such as
testing a command from within the OE build environment using the
-following example:class BitbakeLayers(OESelftestTestCase): def
-test_bitbakelayers_showcrossdepends(self): result =
-runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in
-result.output, msg = "No dependencies were shown. bitbake-layers
-show-cross-depends output: %s"% result.output)
+following example::
+
+ class BitbakeLayers(OESelftestTestCase):
+ def test_bitbakelayers_showcrossdepends(self):
+ result = runCmd('bitbake-layers show-cross-depends')
+ self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
-creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived
+creates a testcase from the ``OESelftestTestCase`` class, derived
from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
and checks the output to ensure it contains something we know should be
here.
@@ -367,7 +400,7 @@ so tests within a given test class should always run in the same build,
while tests in different classes or modules may be split into different
builds. There is no data store available for these tests since the tests
launch the ``bitbake`` command and exist outside of its context. As a
-result, common bitbake library functions (bb.*) are also unavailable.
+result, common bitbake library functions (bb.\*) are also unavailable.
.. _testimage-example:
@@ -378,14 +411,18 @@ These tests are run once an image is up and running, either on target
hardware or under QEMU. As a result, they are assumed to be running in a
target image environment, as opposed to a host build environment. A
simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
-the following:class PythonTest(OERuntimeTestCase):
-@OETestDepends(['ssh.SSHTest.test_ssh'])<mailto:-@OETestDepends(['ssh.SSHTest.test_ssh'])> @OEHasPackage(['python3-core'])
-def test_python3(self): cmd = "python3 -c \\"import<file://%22import> codecs;
-print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output =
-self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output
-self.assertEqual(status, 0, msg=msg)
-
-In this example, the ```OERuntimeTestCase`` <>`__ class wraps
+the following::
+
+ class PythonTest(OERuntimeTestCase):
+ @OETestDepends(['ssh.SSHTest.test_ssh'])
+ @OEHasPackage(['python3-core'])
+ def test_python3(self):
+ cmd = "python3 -c \\"import<file://%22import> codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+ status, output = self.target.run(cmd)
+ msg = 'Exit status was not 0. Output: %s' % output
+ self.assertEqual(status, 0, msg=msg)
+
+In this example, the ``OERuntimeTestCase`` class wraps
``unittest.TestCase``. Within the test, ``self.target`` represents the
target system, where commands can be run on it using the ``run()``
method.
@@ -402,19 +439,30 @@ the image.
These tests are run against built extensible SDKs (eSDKs). The tests can
assume that the eSDK environment has already been setup. An example from
-``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class
-DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls):
-myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst =
-os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src,
-cls.myapp_dst) subprocess.check_output(['git', 'init', '.'],
-cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'],
-cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m',
-"'test commit'"], cwd=cls.myapp_dst) @classmethod def
-tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def
-\_test_devtool_build(self, directory): self._run('devtool add myapp %s'
-% directory) try: self._run('devtool build myapp') finally:
-self._run('devtool reset myapp') def test_devtool_build_make(self):
-self._test_devtool_build(self.myapp_dst)In this example, the ``devtool``
+``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
+
+ class DevtoolTest(OESDKExtTestCase):
+ @classmethod def setUpClass(cls):
+ myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
+ cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
+ shutil.copytree(myapp_src, cls.myapp_dst)
+ subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
+ subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
+ subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.myapp_dst)
+ def _test_devtool_build(self, directory):
+ self._run('devtool add myapp %s' % directory)
+ try:
+ self._run('devtool build myapp')
+ finally:
+ self._run('devtool reset myapp')
+ def test_devtool_build_make(self):
+ self._test_devtool_build(self.myapp_dst)
+
+In this example, the ``devtool``
command is tested to see whether a sample application can be built with
the ``devtool build`` command within the eSDK.
@@ -426,14 +474,20 @@ the ``devtool build`` command within the eSDK.
These tests are run against built SDKs. The tests can assume that an SDK
has already been extracted and its environment file has been sourced. A
simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
-following:class Python3Test(OESDKTestCase): def setUp(self): if not
-(self.tc.hasHostPackage("nativesdk-python3-core") or
-self.tc.hasHostPackage("python3-core-native")): raise
-unittest.SkipTest("No python3 package in the SDK") def
-test_python3(self): cmd = "python3 -c \\"import<file://%22import> codecs;
-print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd)
-self.assertEqual(output, "Hello, world\n")In this example, if
-nativesdk-python3-core has been installed into the SDK, the code runs
+following::
+
+ class Python3Test(OESDKTestCase):
+ def setUp(self):
+ if not (self.tc.hasHostPackage("nativesdk-python3-core") or
+ self.tc.hasHostPackage("python3-core-native")):
+ raise unittest.SkipTest("No python3 package in the SDK")
+
+ def test_python3(self):
+ cmd = "python3 -c \\"import<file://%22import> codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+ output = self._run(cmd)
+ self.assertEqual(output, "Hello, world\n")
+
+In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
the python3 interpreter with a basic command to check it is working
correctly. The test would only run if python3 is installed in the SDK.
@@ -444,17 +498,25 @@ correctly. The test would only run if python3 is installed in the SDK.
The performance tests usually measure how long operations take and the
resource utilisation as that happens. An example from
-``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class
-Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake
--p)""" # Drop all caches and parse self.rm_cache()
-oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no
-caches)') # Drop tmp/cache
-oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no
-tmp/cache)') # Parse with fully cached data
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p
-(cached)')This example shows how three specific parsing timings are
+``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
+
+ class Test3(BuildPerfTestCase):
+ def test3(self):
+ """Bitbake parsing (bitbake -p)"""
+ # Drop all caches and parse
+ self.rm_cache()
+ oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
+ 'bitbake -p (no caches)')
+ # Drop tmp/cache
+ oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
+ 'bitbake -p (no tmp/cache)')
+ # Parse with fully cached data
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
+ 'bitbake -p (cached)')
+
+This example shows how three specific parsing timings are
measured, with and without various caches, to show how BitBake’s parsing
performance trends over time.
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst
index 8d3c84415..0d987d245 100644
--- a/documentation/test-manual/test-manual-understand-autobuilder.rst
+++ b/documentation/test-manual/test-manual-understand-autobuilder.rst
@@ -20,32 +20,48 @@ each of those targets on a seperate buildbot worker. To understand the
configuration, you need to look at the entry on ``config.json`` file
within the ``yocto-autobuilder-helper`` repository. The targets are
defined in the ‘overrides’ section, a quick example could be qemux86-64
-which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" :
-"arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic
-wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from
-the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" :
-true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato
-core-image-sato-dev core-image-sato-sdk core-image-minimal
-core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS"
-: "core-image-minimal:do_testimage core-image-sato:do_testimage
-core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2"
-: { "SDKMACHINE" : "x86_64", "BBTARGETS" :
-"core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext
-core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" :
-"core-image-sato:do_testsdk core-image-minimal:do_testsdkext
-core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false,
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest
-${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" :
-["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can
-see that “qemux86-64” is a three step build where the
-``bitbake BBTARGETS`` would be run, then ``bitbake
- SANITYTARGETS`` for each step; all for
+which looks like::
+
+ "qemux86-64" : {
+ "MACHINE" : "qemux86-64",
+ "TEMPLATE" : "arch-qemu",
+ "step1" : {
+ "extravars" : [
+ "IMAGE_FSTYPES_append = ' wic wic.bmap'"
+ ]
+ }
+ },
+
+And to expand that, you need the “arch-qemu” entry from
+the “templates” section, which looks like::
+
+ "arch-qemu" : {
+ "BUILDINFO" : true,
+ "BUILDHISTORY" : true,
+ "step1" : {
+ "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
+ "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
+ },
+ "step2" : {
+ "SDKMACHINE" : "x86_64",
+ "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
+ "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
+ },
+ "step3" : {
+ "BUILDHISTORY" : false,
+ "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
+ "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
+ }
+ },
+
+Combining these two entries you can see that “qemux86-64” is a three step build where the
+``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step
1 an extra variable is added to the ``auto.conf`` file to enable wic
image generation.
While not every detail of this is covered here, you can see how the
-templating mechanism allows quite complex configurations to be built up
+template mechanism allows quite complex configurations to be built up
why? let's stick to the original .xml file, we can fix the doc later.
It’s just more grammatically correct. I feel like I should have caught this the first time around ☺
yet allows duplication and repetition to be kept to a minimum.
The different build targets are designed to allow for parallelisation,
@@ -70,7 +86,7 @@ roughly consist of:
This cleans out any previous build. Old builds are left around to
allow easier debugging of failed builds. For additional information,
- see ```clobberdir`` <#test-clobberdir>`__.
+ see :ref:`test-clobberdir`.
It works. However we are now using autosectionlabel which creates sections labels for headings automatically. So I would like to get rid of all custom labels, as much as possible. the following is preferred:
:ref:`test-manual/test-manual-understand-autobuilder:clobberdir>`
I’ve updated all these instances
I know we've left some links like that in other manuals. So it's not strictly required to change it now.
2. *Obtain yocto-autobuilder-helper*
@@ -94,7 +110,7 @@ roughly consist of:
from the ``layerinfo.json`` file to help understand the
configuration. It will also use a local cache of repositories to
speed up the clone checkouts. For additional information, see
- `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__.
+ :ref:`test-autobuilder-clone-cache`.
same here (and probably in other places)
I updated all of them
This step has two possible modes of operation. If the build is part
of a parent build, its possible that all the repositories needed may
@@ -137,8 +153,7 @@ special script that moves files to a special location, rather than
deleting them. Files in this location are deleted by an ``rm`` command,
which is run under ``ionice -c 3``. For example, the deletion only
happens when there is idle IO capacity on the Worker. The Autobuilder
-Worker Janitor runs this deletion. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor runs this deletion. See :ref:`test-autobuilder-worker-janitor`.
.. _test-autobuilder-clone-cache:
@@ -150,8 +165,8 @@ on the Autobuilder. We therefore have a stash of commonly used
repositories pre-cloned on the Workers. Data is fetched from these
during clones first, then "topped up" with later revisions from any
upstream when necesary. The cache is maintained by the Autobuilder
-Worker Janitor. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor. See :ref:`Autobuilder Worker
+Janitor`.
.. _test-autobuilder-worker-janitor:
@@ -159,8 +174,8 @@ Autobuilder Worker Janitor
--------------------------
This is a process running on each Worker that performs two basic
-operations, including background file deletion at IO idle (see `Target
-Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and
+operations, including background file deletion at IO idle (see Target
+Execution: clobberdir :ref:`test-list-tgt-exec-clobberdir`) and
the link and text look wrong here. the link is not on the 'text'
It was to an old anchor link for the step that did not convert. I updated the link to point to the main section.
maintainenance of a cache of cloned repositories to improve the speed
the system can checkout repositories.
@@ -216,9 +231,6 @@ are general setup steps that are run once and include:
For each step that is configured in ``config.json``, it will perform the
following:
-## WRITER's question: What does "logging in as stepXa" and others refer
-to below? ##
you can keep comment with the ".. xxx" syntax.
It was no longer necessary – should I add it back in?
-
1. Add any layers that are specified using the
``bitbake-layers add-layer`` command (logging as stepXa)
@@ -235,9 +247,8 @@ to below? ##
6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
BitBake build environment (logging as stepXd)
-7. Remove any layers added in `step
- 1 <#test-run-config-add-layers-step>`__ using the
- ``bitbake-layers remove-layer`` command (logging as stepXa)
+7. Remove any layers added in step
+ 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
you removed a link here, no?
Yes – it pointed to a step, so I just used the step number instead, as it’s part of the same procedure.
Once the execution steps above complete, ``run-config`` executes a set
of post-build steps, including:
@@ -252,7 +263,7 @@ of post-build steps, including:
generated to the remote server.
4. Cleanup the build directory using
- ```clobberdir`` <#test-clobberdir>`__ if the build was successful,
+ :ref:`test-clobberdir` if the build was successful,
else rename it to “build-renamed” for potential future debugging.
.. _test-deploying-yp-autobuilder:
@@ -279,11 +290,18 @@ The standard ``config.json`` minimally attempts to allow substitution of
the paths. The Helper script repository includes a
``local-example.json`` file to show how you could override these from a
separate configuration file. Pass the following into the environment of
-the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As
-another example, you could also pass the following into the
-environment:$ ABHELPER_JSON="config.json /some/location/local.json"One
-issue users often run into is validation of the ``config.json`` files. A
+the Autobuilder::
+
+ $ ABHELPER_JSON="config.json local-example.json"
+
+As another example, you could also pass the following into the
+environment::
+
+ $ ABHELPER_JSON="config.json /some/location/local.json"
+
+One issue users often run into is validation of the ``config.json`` files. A
tip for minimizing issues from invalid json files is to use a Git
``pre-commit-hook.sh`` script to verify the JSON file before committing
-it. Create a symbolic link as follows:$ ln -s
-../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
+it. Create a symbolic link as follows::
+
+ $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
--
2.17.1
[-- Attachment #2: Type: text/html, Size: 57735 bytes --]
^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-09-15 19:07 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-14 23:11 [PATCH] test-manual code block, link, and format update Mark Morton
2020-09-15 6:08 ` [docs] " Nicolas Dechesne
2020-09-15 19:06 ` Mark Morton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.