Re: [docs] [PATCH v3] test-manual: add initial reproducible builds documentation

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Thu, 2021-06-10 at 10:54 +0200, Quentin Schulz wrote:
> Hi Richard,
> 
> On Wed, Jun 09, 2021 at 05:30:56PM +0100, Richard Purdie wrote:
> > On Wed, 2021-06-09 at 17:55 +0200, Quentin Schulz wrote:
> > > On Wed, Jun 09, 2021 at 04:47:28PM +0200, Michael Opdenacker wrote:
> > > 
> > > > +
> > > > +You could subclass the test and change ``targets`` to a different target.
> > > > +
> > > > +You may also change ``sstate_targets`` which would allow you to "pre-cache" some
> > > > +set of recipes before the test, meaning they are excluded from reproducibility
> > > > +testing. As a practical example, you could set ``sstate_targets`` to
> > > > +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would
> > > > +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
> > > > -- 
> > > 
> > > I'm not sure this section has its place in our documentation since it
> > > seems bound to the current implementation and explains modifications of
> > > the code.
> > > 
> > > I'd vote it out.
> > 
> > Doesn't most of the manual document the current implementation?
> > 
> > When this wasn't here, you wanted an example! I do agree we should give people 
> > some kind of a hint about how to write their own tests. We're likely going to end 
> > up adding more example, not less?
> > 
> 
> Because I understood the original section as "it's very simple to do
> this, look into this file", so I assumed some arguments could be passed
> to make it easy to re-use. I wasn't expecting changes in a Python
> script.

I think the confusion here is the definition of simple :).

At least in theory, you can create your own test case in your own layer
and in that test, inherit and subclass the original test (standard python). 
In that subclassed test, you could change those two variables and you 
should then have your own test case.

Would that work? Should do, at least mostly but I will admit I've not tried it.
Adding a new test in your layer and also subclassing an existing test is 
something we should document somewhere in the test manual ultimately. That
is partly why I continued to expand on what I meant here.

> As opposed to classes, variables and/or tasks, which to me seems to be
> kind of an ABI, so it is maintained and watched for changes and if
> there's any, usually listed in the migration manual, I would assume the
> content of Python scripts aren't expected to stay "stable" or their
> change be documented otherwise?
> 
> Here the suggested example is about modifying variables that are used only
> in a given Python class, which could change over time, including the name
> of the variables to be changed.

I agree it is a bit risky to have these in the manual, equally, the manual
does have worked examples and I believe there is a good case for having them
here as this is something we want to teach people how to do.

FWIW that test was designed to be sub-classed so there is at least some
intent to allow people to do this. As such we'd probably try to maintain 
some kind of compatibility if we can.

> This also kind of encourages people to modify code in layers they don't
> have push access to, which is something that we usually don't want
> people to do IIRC?

The key missing detail is you can put test code in your own layer. I
can even point at an example:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta-yocto-bsp/lib/oeqa/selftest/cases/systemd_boot.py

(which is also how we test this stays working)

> I'm not sure I managed to convey my thoughts better. I just expected a
> simple command to be able to test my recipes, or via a configuration
> script.
> 
> Anyway, no big deal, I'm happy to see the reproducible test procedure is
> being documented :)

Thanks for the review, it is helpful to have have a fresher perspective on
this!

Cheers,

Richard