[PATCH v3] test-manual: add initial reproducible builds documentation

[PATCH v3] test-manual: add initial reproducible builds documentation

[Michael Opdenacker]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Quentin Schulz <foss@0leil.net>
---
 documentation/test-manual/index.rst           |   1 +
 .../test-manual/reproducible-builds.rst       | 134 ++++++++++++++++++
 2 files changed, 135 insertions(+)
 create mode 100644 documentation/test-manual/reproducible-builds.rst

diff --git a/documentation/test-manual/index.rst b/documentation/test-manual/index.rst
index e2198c4c39..d122b5e270 100644
--- a/documentation/test-manual/index.rst
+++ b/documentation/test-manual/index.rst
@@ -13,6 +13,7 @@ Yocto Project Test Environment Manual
    intro
    test-process
    understand-autobuilder
+   reproducible-builds
    history
 
 .. include:: /boilerplate.rst
diff --git a/documentation/test-manual/reproducible-builds.rst b/documentation/test-manual/reproducible-builds.rst
new file mode 100644
index 0000000000..f8140d7890
--- /dev/null
+++ b/documentation/test-manual/reproducible-builds.rst
@@ -0,0 +1,134 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+*******************
+Reproducible Builds
+*******************
+
+================
+How we define it
+================
+
+The Yocto Project defines reproducibility as where a given input build
+configuration will give the same binary output regardless of when it is built
+(now or in 5 years time), regardless of the path on the filesystem the build is
+run in, and regardless of the distro and tools on the underlying host system the
+build is running on.
+
+==============
+Why it matters
+==============
+
+The project aligns with the `Reproducible Builds project
+<https://reproducible-builds.org/>`_, which shares information about why
+reproducibility matters. The primary focus of the project is the ability to
+detect security issues being introduced. However, from a Yocto Project
+perspective, it is also hugely important that our builds are deterministic. When
+you build a given input set of metadata, we expect you to get consistent output.
+This has always been a key focus but, since release 3.1 ("dunfell"), it is now true
+down to the binary level including timestamps.
+
+For example, at some point in the future life of a product, you find that you
+need to rebuild to add a security fix. If this happens, only the components that
+have been modified should change at the binary level. This would lead to much
+easier and clearer bounds on where validation is needed.
+
+This also gives an additional benefit to the project builds themselves, our hash
+equivalence for :ref:`Shared State <overview-manual/concepts:Shared State>`
+object reuse works much more effectively when the binary output remains the
+same.
+
+.. note::
+
+   We strongly advise you to make sure your project builds reproducibly
+   before finalizing your production images. It would be too late if you
+   only address this issue when the first updates are required.
+
+===================
+How we implement it
+===================
+
+There are many different aspects to build reproducibility, but some particular
+things we do within the build system to ensure reproducibility include:
+
+-  Adding mappings to the compiler options to ensure debug filepaths are mapped
+   to consistent target compatible paths. This is done through the
+   ``DEBUG_PREFIX_MAP`` variable which sets the ``-fmacro-prefix-map`` and
+   ``-fdebug-prefix-map`` compiler options correctly to map to target paths.
+-  Being explicit about recipe dependencies and their configuration (no floating
+   configure options or host dependencies creeping in). In particular this means
+   making sure :term:`PACKAGECONFIG` coverage covers configure options which may
+   otherwise try and auto-detect host dependencies.
+-  Using recipe specific sysroots to isolate recipes so they only see their
+   dependencies. These are visible as ``recipe-sysroot`` and
+   ``recipe-sysroot-native`` directories within the :term:`WORKDIR` of a given
+   recipe and are populated only with the dependencies a recipe has.
+-  Build images from a reduced package set: only packages from recipes the image
+   depends upon.
+-  Filtering the tools available from the host's ``PATH`` to only a specific set
+   of tools, set using the :term:`HOSTTOOLS` variable.
+
+=========================================
+Can we prove the project is reproducible?
+=========================================
+
+Yes, we can prove it and we regularly test this on the Autobuilder. At the
+time of writing (release 3.3, "hardknott"), :term:`OpenEmbedded-Core (OE-Core)`
+is 100% reproducible for all its recipes (i.e. world builds) apart from the Go
+language and Ruby documentation packages. Unfortunately, the current
+implementation of the Go language has fundamental reproducibility problems as
+it always depends upon the paths it is built in.
+
+.. note::
+
+   Only BitBake and :term:`OpenEmbedded-Core (OE-Core)`, which is the ``meta``
+   layer in Poky, guarantee complete reproducibility. The moment you add
+   another layer, this warranty is voided, because of additional configuration
+   files, ``bbappend`` files, overridden classes, etc.
+
+To run our automated selftest, as we use in our CI on the Autobuilder, you can
+run::
+
+   oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
+
+This defaults to including a ``world`` build so, if other layers are added, it would
+also run the tests for recipes in the additional layers. The first build will be
+run using :ref:`Shared State <overview-manual/concepts:Shared State>` if
+available, the second build explicitly disables
+:ref:`Shared State <overview-manual/concepts:Shared State>` and builds on the
+specific host the build is running on. This means we can test reproducibility
+builds between different host distributions over time on the Autobuilder.
+
+If ``OEQA_DEBUGGING_SAVED_OUTPUT`` is set, any differing packages will be saved
+here. The test is also able to run the ``diffoscope`` command on the output to
+generate HTML files showing the differences between the packages, to aid
+debugging. On the Autobuilder, these appear under
+https://autobuilder.yocto.io/pub/repro-fail/ in the form ``oe-reproducible +
+<date> + <random ID>``, e.g. ``oe-reproducible-20200202-1lm8o1th``.
+
+The project's current reproducibility status can be seen at
+https://www.yoctoproject.org/reproducible-build-results/
+
+You can also check the reproducibility status on supported host distributions:
+
+-  CentOS: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-centos
+-  Debian: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-debian
+-  Fedora: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-fedora
+-  Ubuntu: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-ubuntu
+
+===============================
+Can I test my layer or recipes?
+===============================
+
+Once again, you can run a ``world`` test using the
+:ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>`
+command provided above. This functionality is implemented
+in `meta/lib/oeqa/selftest/cases/reproducible.py
+<https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_.
+
+You could subclass the test and change ``targets`` to a different target.
+
+You may also change ``sstate_targets`` which would allow you to "pre-cache" some
+set of recipes before the test, meaning they are excluded from reproducibility
+testing. As a practical example, you could set ``sstate_targets`` to
+``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would
+run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
-- 
2.25.1

Re: [docs] [PATCH v3] test-manual: add initial reproducible builds documentation

[Quentin Schulz]

On Wed, Jun 09, 2021 at 04:47:28PM +0200, Michael Opdenacker wrote:
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> Reviewed-by: Quentin Schulz <foss@0leil.net>
> ---
>  documentation/test-manual/index.rst           |   1 +
>  .../test-manual/reproducible-builds.rst       | 134 ++++++++++++++++++
>  2 files changed, 135 insertions(+)
>  create mode 100644 documentation/test-manual/reproducible-builds.rst
> 
> diff --git a/documentation/test-manual/index.rst b/documentation/test-manual/index.rst
> index e2198c4c39..d122b5e270 100644
> --- a/documentation/test-manual/index.rst
> +++ b/documentation/test-manual/index.rst
> @@ -13,6 +13,7 @@ Yocto Project Test Environment Manual
>     intro
>     test-process
>     understand-autobuilder
> +   reproducible-builds
>     history
>  
>  .. include:: /boilerplate.rst
> diff --git a/documentation/test-manual/reproducible-builds.rst b/documentation/test-manual/reproducible-builds.rst
> new file mode 100644
> index 0000000000..f8140d7890
> --- /dev/null
> +++ b/documentation/test-manual/reproducible-builds.rst
> @@ -0,0 +1,134 @@
> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
> +
> +*******************
> +Reproducible Builds
> +*******************
> +
> +================
> +How we define it
> +================
> +
> +The Yocto Project defines reproducibility as where a given input build
> +configuration will give the same binary output regardless of when it is built
> +(now or in 5 years time), regardless of the path on the filesystem the build is
> +run in, and regardless of the distro and tools on the underlying host system the
> +build is running on.
> +
> +==============
> +Why it matters
> +==============
> +
> +The project aligns with the `Reproducible Builds project
> +<https://reproducible-builds.org/>`_, which shares information about why
> +reproducibility matters. The primary focus of the project is the ability to
> +detect security issues being introduced. However, from a Yocto Project
> +perspective, it is also hugely important that our builds are deterministic. When
> +you build a given input set of metadata, we expect you to get consistent output.
> +This has always been a key focus but, since release 3.1 ("dunfell"), it is now true

Just picking nits here, but I could see a reference to:
https://docs.yoctoproject.org/ref-manual/migration-3.1.html#reproducible-builds-now-enabled-by-default

> +down to the binary level including timestamps.
> +
> +For example, at some point in the future life of a product, you find that you
> +need to rebuild to add a security fix. If this happens, only the components that
> +have been modified should change at the binary level. This would lead to much
> +easier and clearer bounds on where validation is needed.
> +
> +This also gives an additional benefit to the project builds themselves, our hash
> +equivalence for :ref:`Shared State <overview-manual/concepts:Shared State>`
> +object reuse works much more effectively when the binary output remains the
> +same.
> +
> +.. note::
> +
> +   We strongly advise you to make sure your project builds reproducibly
> +   before finalizing your production images. It would be too late if you
> +   only address this issue when the first updates are required.
> +
> +===================
> +How we implement it
> +===================
> +
> +There are many different aspects to build reproducibility, but some particular
> +things we do within the build system to ensure reproducibility include:
> +
> +-  Adding mappings to the compiler options to ensure debug filepaths are mapped
> +   to consistent target compatible paths. This is done through the
> +   ``DEBUG_PREFIX_MAP`` variable which sets the ``-fmacro-prefix-map`` and
> +   ``-fdebug-prefix-map`` compiler options correctly to map to target paths.
> +-  Being explicit about recipe dependencies and their configuration (no floating
> +   configure options or host dependencies creeping in). In particular this means
> +   making sure :term:`PACKAGECONFIG` coverage covers configure options which may
> +   otherwise try and auto-detect host dependencies.
> +-  Using recipe specific sysroots to isolate recipes so they only see their
> +   dependencies. These are visible as ``recipe-sysroot`` and
> +   ``recipe-sysroot-native`` directories within the :term:`WORKDIR` of a given
> +   recipe and are populated only with the dependencies a recipe has.
> +-  Build images from a reduced package set: only packages from recipes the image
> +   depends upon.
> +-  Filtering the tools available from the host's ``PATH`` to only a specific set
> +   of tools, set using the :term:`HOSTTOOLS` variable.
> +
> +=========================================
> +Can we prove the project is reproducible?
> +=========================================
> +
> +Yes, we can prove it and we regularly test this on the Autobuilder. At the
> +time of writing (release 3.3, "hardknott"), :term:`OpenEmbedded-Core (OE-Core)`
> +is 100% reproducible for all its recipes (i.e. world builds) apart from the Go
> +language and Ruby documentation packages. Unfortunately, the current
> +implementation of the Go language has fundamental reproducibility problems as
> +it always depends upon the paths it is built in.
> +
> +.. note::
> +
> +   Only BitBake and :term:`OpenEmbedded-Core (OE-Core)`, which is the ``meta``
> +   layer in Poky, guarantee complete reproducibility. The moment you add
> +   another layer, this warranty is voided, because of additional configuration
> +   files, ``bbappend`` files, overridden classes, etc.
> +
> +To run our automated selftest, as we use in our CI on the Autobuilder, you can
> +run::
> +
> +   oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
> +
> +This defaults to including a ``world`` build so, if other layers are added, it would
> +also run the tests for recipes in the additional layers. The first build will be
> +run using :ref:`Shared State <overview-manual/concepts:Shared State>` if
> +available, the second build explicitly disables
> +:ref:`Shared State <overview-manual/concepts:Shared State>` and builds on the
> +specific host the build is running on. This means we can test reproducibility
> +builds between different host distributions over time on the Autobuilder.
> +
> +If ``OEQA_DEBUGGING_SAVED_OUTPUT`` is set, any differing packages will be saved
> +here. The test is also able to run the ``diffoscope`` command on the output to
> +generate HTML files showing the differences between the packages, to aid
> +debugging. On the Autobuilder, these appear under
> +https://autobuilder.yocto.io/pub/repro-fail/ in the form ``oe-reproducible +
> +<date> + <random ID>``, e.g. ``oe-reproducible-20200202-1lm8o1th``.
> +
> +The project's current reproducibility status can be seen at
> +https://www.yoctoproject.org/reproducible-build-results/
> +
> +You can also check the reproducibility status on supported host distributions:
> +
> +-  CentOS: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-centos
> +-  Debian: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-debian
> +-  Fedora: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-fedora
> +-  Ubuntu: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-ubuntu
> +
> +===============================
> +Can I test my layer or recipes?
> +===============================
> +
> +Once again, you can run a ``world`` test using the
> +:ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>`
> +command provided above. This functionality is implemented
> +in `meta/lib/oeqa/selftest/cases/reproducible.py
> +<https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_.
> +
> +You could subclass the test and change ``targets`` to a different target.
> +
> +You may also change ``sstate_targets`` which would allow you to "pre-cache" some
> +set of recipes before the test, meaning they are excluded from reproducibility
> +testing. As a practical example, you could set ``sstate_targets`` to
> +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would
> +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
> -- 

I'm not sure this section has its place in our documentation since it
seems bound to the current implementation and explains modifications of
the code.

I'd vote it out.

Cheers,
Quentin

Re: [docs] [PATCH v3] test-manual: add initial reproducible builds documentation

[Richard Purdie]

On Wed, 2021-06-09 at 17:55 +0200, Quentin Schulz wrote:
> On Wed, Jun 09, 2021 at 04:47:28PM +0200, Michael Opdenacker wrote:
> 
> > +
> > +You could subclass the test and change ``targets`` to a different target.
> > +
> > +You may also change ``sstate_targets`` which would allow you to "pre-cache" some
> > +set of recipes before the test, meaning they are excluded from reproducibility
> > +testing. As a practical example, you could set ``sstate_targets`` to
> > +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would
> > +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
> > -- 
> 
> I'm not sure this section has its place in our documentation since it
> seems bound to the current implementation and explains modifications of
> the code.
> 
> I'd vote it out.

Doesn't most of the manual document the current implementation?

When this wasn't here, you wanted an example! I do agree we should give people 
some kind of a hint about how to write their own tests. We're likely going to end 
up adding more example, not less?

Cheers,

Richard

Re: [docs] [PATCH v3] test-manual: add initial reproducible builds documentation

[Quentin Schulz]

Hi Richard,

On Wed, Jun 09, 2021 at 05:30:56PM +0100, Richard Purdie wrote:
> On Wed, 2021-06-09 at 17:55 +0200, Quentin Schulz wrote:
> > On Wed, Jun 09, 2021 at 04:47:28PM +0200, Michael Opdenacker wrote:
> > 
> > > +
> > > +You could subclass the test and change ``targets`` to a different target.
> > > +
> > > +You may also change ``sstate_targets`` which would allow you to "pre-cache" some
> > > +set of recipes before the test, meaning they are excluded from reproducibility
> > > +testing. As a practical example, you could set ``sstate_targets`` to
> > > +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would
> > > +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
> > > -- 
> > 
> > I'm not sure this section has its place in our documentation since it
> > seems bound to the current implementation and explains modifications of
> > the code.
> > 
> > I'd vote it out.
> 
> Doesn't most of the manual document the current implementation?
> 
> When this wasn't here, you wanted an example! I do agree we should give people 
> some kind of a hint about how to write their own tests. We're likely going to end 
> up adding more example, not less?
> 

Because I understood the original section as "it's very simple to do
this, look into this file", so I assumed some arguments could be passed
to make it easy to re-use. I wasn't expecting changes in a Python
script.

As opposed to classes, variables and/or tasks, which to me seems to be
kind of an ABI, so it is maintained and watched for changes and if
there's any, usually listed in the migration manual, I would assume the
content of Python scripts aren't expected to stay "stable" or their
change be documented otherwise?

Here the suggested example is about modifying variables that are used only
in a given Python class, which could change over time, including the name
of the variables to be changed.

This also kind of encourages people to modify code in layers they don't
have push access to, which is something that we usually don't want
people to do IIRC?

I'm not sure I managed to convey my thoughts better. I just expected a
simple command to be able to test my recipes, or via a configuration
script.

Anyway, no big deal, I'm happy to see the reproducible test procedure is
being documented :)

Cheers,
Quentin

Re: [docs] [PATCH v3] test-manual: add initial reproducible builds documentation

[Richard Purdie]

On Thu, 2021-06-10 at 10:54 +0200, Quentin Schulz wrote:
> Hi Richard,
> 
> On Wed, Jun 09, 2021 at 05:30:56PM +0100, Richard Purdie wrote:
> > On Wed, 2021-06-09 at 17:55 +0200, Quentin Schulz wrote:
> > > On Wed, Jun 09, 2021 at 04:47:28PM +0200, Michael Opdenacker wrote:
> > > 
> > > > +
> > > > +You could subclass the test and change ``targets`` to a different target.
> > > > +
> > > > +You may also change ``sstate_targets`` which would allow you to "pre-cache" some
> > > > +set of recipes before the test, meaning they are excluded from reproducibility
> > > > +testing. As a practical example, you could set ``sstate_targets`` to
> > > > +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would
> > > > +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
> > > > -- 
> > > 
> > > I'm not sure this section has its place in our documentation since it
> > > seems bound to the current implementation and explains modifications of
> > > the code.
> > > 
> > > I'd vote it out.
> > 
> > Doesn't most of the manual document the current implementation?
> > 
> > When this wasn't here, you wanted an example! I do agree we should give people 
> > some kind of a hint about how to write their own tests. We're likely going to end 
> > up adding more example, not less?
> > 
> 
> Because I understood the original section as "it's very simple to do
> this, look into this file", so I assumed some arguments could be passed
> to make it easy to re-use. I wasn't expecting changes in a Python
> script.

I think the confusion here is the definition of simple :).

At least in theory, you can create your own test case in your own layer
and in that test, inherit and subclass the original test (standard python). 
In that subclassed test, you could change those two variables and you 
should then have your own test case.

Would that work? Should do, at least mostly but I will admit I've not tried it.
Adding a new test in your layer and also subclassing an existing test is 
something we should document somewhere in the test manual ultimately. That
is partly why I continued to expand on what I meant here.

> As opposed to classes, variables and/or tasks, which to me seems to be
> kind of an ABI, so it is maintained and watched for changes and if
> there's any, usually listed in the migration manual, I would assume the
> content of Python scripts aren't expected to stay "stable" or their
> change be documented otherwise?
> 
> Here the suggested example is about modifying variables that are used only
> in a given Python class, which could change over time, including the name
> of the variables to be changed.

I agree it is a bit risky to have these in the manual, equally, the manual
does have worked examples and I believe there is a good case for having them
here as this is something we want to teach people how to do.

FWIW that test was designed to be sub-classed so there is at least some
intent to allow people to do this. As such we'd probably try to maintain 
some kind of compatibility if we can.

> This also kind of encourages people to modify code in layers they don't
> have push access to, which is something that we usually don't want
> people to do IIRC?

The key missing detail is you can put test code in your own layer. I
can even point at an example:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta-yocto-bsp/lib/oeqa/selftest/cases/systemd_boot.py

(which is also how we test this stays working)

> I'm not sure I managed to convey my thoughts better. I just expected a
> simple command to be able to test my recipes, or via a configuration
> script.
> 
> Anyway, no big deal, I'm happy to see the reproducible test procedure is
> being documented :)

Thanks for the review, it is helpful to have have a fresher perspective on
this!

Cheers,

Richard

Re: [docs] [PATCH v3] test-manual: add initial reproducible builds documentation

[Quentin Schulz]

Hi Richard,

On Thu, Jun 10, 2021 at 10:14:11AM +0100, Richard Purdie wrote:
> On Thu, 2021-06-10 at 10:54 +0200, Quentin Schulz wrote:
> > Hi Richard,
> > 
> > On Wed, Jun 09, 2021 at 05:30:56PM +0100, Richard Purdie wrote:
> > > On Wed, 2021-06-09 at 17:55 +0200, Quentin Schulz wrote:
> > > > On Wed, Jun 09, 2021 at 04:47:28PM +0200, Michael Opdenacker wrote:
> > > > 
> > > > > +
> > > > > +You could subclass the test and change ``targets`` to a different target.
> > > > > +
> > > > > +You may also change ``sstate_targets`` which would allow you to "pre-cache" some
> > > > > +set of recipes before the test, meaning they are excluded from reproducibility
> > > > > +testing. As a practical example, you could set ``sstate_targets`` to
> > > > > +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would
> > > > > +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
> > > > > -- 
> > > > 
> > > > I'm not sure this section has its place in our documentation since it
> > > > seems bound to the current implementation and explains modifications of
> > > > the code.
> > > > 
> > > > I'd vote it out.
> > > 
> > > Doesn't most of the manual document the current implementation?
> > > 
> > > When this wasn't here, you wanted an example! I do agree we should give people 
> > > some kind of a hint about how to write their own tests. We're likely going to end 
> > > up adding more example, not less?
> > > 
> > 
> > Because I understood the original section as "it's very simple to do
> > this, look into this file", so I assumed some arguments could be passed
> > to make it easy to re-use. I wasn't expecting changes in a Python
> > script.
> 
> I think the confusion here is the definition of simple :).
> 
> At least in theory, you can create your own test case in your own layer
> and in that test, inherit and subclass the original test (standard python). 
> In that subclassed test, you could change those two variables and you 
> should then have your own test case.
> 
> Would that work? Should do, at least mostly but I will admit I've not tried it.
> Adding a new test in your layer and also subclassing an existing test is 
> something we should document somewhere in the test manual ultimately. That
> is partly why I continued to expand on what I meant here.

I re-read the previous mail and I think my brain just ignored the
following sentence:
You could subclass the test and change ``targets`` to a different target.

Which should have put me on the right track. I was missing this bit in
my understanding. And now, I happily concede that my comments didn't
make much sense considering what you suggested :)

> 
> > As opposed to classes, variables and/or tasks, which to me seems to be
> > kind of an ABI, so it is maintained and watched for changes and if
> > there's any, usually listed in the migration manual, I would assume the
> > content of Python scripts aren't expected to stay "stable" or their
> > change be documented otherwise?
> > 
> > Here the suggested example is about modifying variables that are used only
> > in a given Python class, which could change over time, including the name
> > of the variables to be changed.
> 
> I agree it is a bit risky to have these in the manual, equally, the manual
> does have worked examples and I believe there is a good case for having them
> here as this is something we want to teach people how to do.
> 
> FWIW that test was designed to be sub-classed so there is at least some
> intent to allow people to do this. As such we'd probably try to maintain 
> some kind of compatibility if we can.
> 

Agreed. I would try to make sure subclassing works since you seem to
insinuate you're not 100% sure it works right now.

I would very much see a "code" example, with the correct path in the
layer (if that matters in some ways to bitbake) somewhere in the docs
then. If it does not make sense in this section, at least refer to where
it will lie.

> > This also kind of encourages people to modify code in layers they don't
> > have push access to, which is something that we usually don't want
> > people to do IIRC?
> 
> The key missing detail is you can put test code in your own layer. I
> can even point at an example:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/tree/meta-yocto-bsp/lib/oeqa/selftest/cases/systemd_boot.py
> 
> (which is also how we test this stays working)
> 

Indeed, thanks for clarifying this, it makes sense to me now.

Cheers,
Quentin

Re: [docs] [PATCH v3] test-manual: add initial reproducible builds documentation

[Michael Opdenacker]

Hello Quentin,

On 6/9/21 5:55 PM, Quentin Schulz wrote:
>> +The project aligns with the `Reproducible Builds project
>> +<https://reproducible-builds.org/>`_, which shares information about why
>> +reproducibility matters. The primary focus of the project is the ability to
>> +detect security issues being introduced. However, from a Yocto Project
>> +perspective, it is also hugely important that our builds are deterministic. When
>> +you build a given input set of metadata, we expect you to get consistent output.
>> +This has always been a key focus but, since release 3.1 ("dunfell"), it is now true
> Just picking nits here, but I could see a reference to:
> https://docs.yoctoproject.org/ref-manual/migration-3.1.html#reproducible-builds-now-enabled-by-default


Good idea. I implemented it.

Thanks!
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [PATCH v3] test-manual: add initial reproducible builds documentation

[Michael Opdenacker]

Hello,

Commenting my own patch, before others do :-}

On 6/9/21 4:47 PM, Michael Opdenacker wrote:
> +The project's current reproducibility status can be seen at
> +https://www.yoctoproject.org/reproducible-build-results/

Oops, my bad, I could have used the :yocto_home: macro here. Will do.

> +
> +You can also check the reproducibility status on supported host distributions:
> +
> +-  CentOS: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-centos
> +-  Debian: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-debian
> +-  Fedora: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-fedora
> +-  Ubuntu: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-ubuntu


Same here, will use :yocto_ab:

> +
> +===============================
> +Can I test my layer or recipes?
> +===============================
> +
> +Once again, you can run a ``world`` test using the
> +:ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>`
> +command provided above. This functionality is implemented
> +in `meta/lib/oeqa/selftest/cases/reproducible.py
> +<https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_.


Same here, will use :yocto_git:

Pfooh, that was just in time before Quentin and Nico catch those ;-)

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com