Re: [docs] [PATCH v3] test-manual: add initial reproducible builds documentation

["Quentin Schulz" <quentin.schulz@streamunlimited.com>]

Hi Richard,

On Wed, Jun 09, 2021 at 05:30:56PM +0100, Richard Purdie wrote:
> On Wed, 2021-06-09 at 17:55 +0200, Quentin Schulz wrote:
> > On Wed, Jun 09, 2021 at 04:47:28PM +0200, Michael Opdenacker wrote:
> > 
> > > +
> > > +You could subclass the test and change ``targets`` to a different target.
> > > +
> > > +You may also change ``sstate_targets`` which would allow you to "pre-cache" some
> > > +set of recipes before the test, meaning they are excluded from reproducibility
> > > +testing. As a practical example, you could set ``sstate_targets`` to
> > > +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would
> > > +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
> > > -- 
> > 
> > I'm not sure this section has its place in our documentation since it
> > seems bound to the current implementation and explains modifications of
> > the code.
> > 
> > I'd vote it out.
> 
> Doesn't most of the manual document the current implementation?
> 
> When this wasn't here, you wanted an example! I do agree we should give people 
> some kind of a hint about how to write their own tests. We're likely going to end 
> up adding more example, not less?
> 

Because I understood the original section as "it's very simple to do
this, look into this file", so I assumed some arguments could be passed
to make it easy to re-use. I wasn't expecting changes in a Python
script.

As opposed to classes, variables and/or tasks, which to me seems to be
kind of an ABI, so it is maintained and watched for changes and if
there's any, usually listed in the migration manual, I would assume the
content of Python scripts aren't expected to stay "stable" or their
change be documented otherwise?

Here the suggested example is about modifying variables that are used only
in a given Python class, which could change over time, including the name
of the variables to be changed.

This also kind of encourages people to modify code in layers they don't
have push access to, which is something that we usually don't want
people to do IIRC?

I'm not sure I managed to convey my thoughts better. I just expected a
simple command to be able to test my recipes, or via a configuration
script.

Anyway, no big deal, I'm happy to see the reproducible test procedure is
being documented :)

Cheers,
Quentin