[PATCH 1/2] test-manual: Add extra detail to YP Compatible section

[PATCH 1/2] test-manual: Add extra detail to YP Compatible section

[Richard Purdie]

Add a note about documenting where a layer doesn't support 'core'
functionality.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 documentation/test-manual/yocto-project-compatible.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/documentation/test-manual/yocto-project-compatible.rst b/documentation/test-manual/yocto-project-compatible.rst
index a7897469f..b83a0f12b 100644
--- a/documentation/test-manual/yocto-project-compatible.rst
+++ b/documentation/test-manual/yocto-project-compatible.rst
@@ -115,6 +115,11 @@ Here are key best practices the program tries to encourage:
    user changes a configuration setting to activate the layer, by selecting
    a :term:`MACHINE`, a :term:`DISTRO` or a :term:`DISTRO_FEATURES` setting.
 
+-  Layers should be documenting where they don’t support normal 'core'
+   functionality such as where debug symbols are disabled or missing, where
+   development headers and on-target library usage may not work or where
+   functionality like the SDK/eSDK would not be execpted to work.
+
 The project does test the compatibility status of the core project layers on
 its :doc:`Autobuilder </test-manual/understand-autobuilder>`.
 
-- 
2.30.2

[PATCH 2/2] migration-3.4: Add extra notes to override sytnax changes

[Richard Purdie]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 documentation/migration-guides/migration-3.4.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/documentation/migration-guides/migration-3.4.rst b/documentation/migration-guides/migration-3.4.rst
index 6fa1ab20c..d42ee8851 100644
--- a/documentation/migration-guides/migration-3.4.rst
+++ b/documentation/migration-guides/migration-3.4.rst
@@ -56,6 +56,12 @@ context being specific tasks in that case. Tune overrides are another special
 case where some code does use them as overrides but some does not. We plan to try
 and make the tune code use overrides more consistently in the future.
 
+There are some variables which do not use override syntax which include the
+suffix to variables in layer.conf files such as BBFILE_PATTERN, :term:`SRCREV`_xxx
+where xxx is a name from :term:`SRC_URI` and PREFERRED_VERSION_xxx. In particular,
+layer.conf suffixes may be the same as a :term:`DISTRO` override causing some
+confusion. We do plan to try and improve consistency as these issues are identified.
+
 To help with migration of layers there is a script in OE-Core. Once configured
 with the overrides used by a layer, this can be run as::
 
-- 
2.30.2

Re: [docs] [PATCH 1/2] test-manual: Add extra detail to YP Compatible section

[Michael Opdenacker]

Hi Richard,

Thanks for the documentation patch!

On 8/4/21 11:47 AM, Richard Purdie wrote:
> Add a note about documenting where a layer doesn't support 'core'
> functionality.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  documentation/test-manual/yocto-project-compatible.rst | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/documentation/test-manual/yocto-project-compatible.rst b/documentation/test-manual/yocto-project-compatible.rst
> index a7897469f..b83a0f12b 100644
> --- a/documentation/test-manual/yocto-project-compatible.rst
> +++ b/documentation/test-manual/yocto-project-compatible.rst
> @@ -115,6 +115,11 @@ Here are key best practices the program tries to encourage:
>     user changes a configuration setting to activate the layer, by selecting
>     a :term:`MACHINE`, a :term:`DISTRO` or a :term:`DISTRO_FEATURES` setting.
>  
> +-  Layers should be documenting where they don’t support normal 'core'
> +   functionality such as where debug symbols are disabled or missing, where
> +   development headers and on-target library usage may not work or where


Replaced "or" by "and" here, as we are dealing with a list of examples,
rather than with a list of alternatives.
https://forum.oxforddictionaries.com/en/discussion/27/such-as-or-or-and


> +   functionality like the SDK/eSDK would not be execpted to work.


s/execpted/expected/

This sounded like a system call, too much work with glibc, I guess ;-)
I took care of this change too, and will merge the change into
master-next (though extra reviews are still possible).

Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH 1/2] test-manual: Add extra detail to YP Compatible section

[Quentin Schulz]

Hi Richard,

On Wed, Aug 04, 2021 at 12:04:34PM +0200, Michael Opdenacker wrote:
> Hi Richard,
> 
> Thanks for the documentation patch!
> 
> On 8/4/21 11:47 AM, Richard Purdie wrote:
> > Add a note about documenting where a layer doesn't support 'core'
> > functionality.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  documentation/test-manual/yocto-project-compatible.rst | 5 +++++
> >  1 file changed, 5 insertions(+)
> >
> > diff --git a/documentation/test-manual/yocto-project-compatible.rst b/documentation/test-manual/yocto-project-compatible.rst
> > index a7897469f..b83a0f12b 100644
> > --- a/documentation/test-manual/yocto-project-compatible.rst
> > +++ b/documentation/test-manual/yocto-project-compatible.rst
> > @@ -115,6 +115,11 @@ Here are key best practices the program tries to encourage:
> >     user changes a configuration setting to activate the layer, by selecting
> >     a :term:`MACHINE`, a :term:`DISTRO` or a :term:`DISTRO_FEATURES` setting.
> >  
> > +-  Layers should be documenting where they don’t support normal 'core'
> > +   functionality such as where debug symbols are disabled or missing, where
> > +   development headers and on-target library usage may not work or where
>

Do we currently have an example layer that is following this suggested
best practice? Because I'm struggling to understand exactly what I'm
expected to do if I were a layer maintainer.

Do you want the maintainer to keep a list of recipes/packages provided
in their layer(s), in e.g. the README, that are missing debug symbols or
"invalid" headers/libraries?

Thanks,
Quentin

Re: [docs] [PATCH 1/2] test-manual: Add extra detail to YP Compatible section

[Richard Purdie]

On Wed, 2021-08-04 at 12:04 +0200, Michael Opdenacker wrote:
> Hi Richard,
> 
> Thanks for the documentation patch!
> 
> On 8/4/21 11:47 AM, Richard Purdie wrote:
> > Add a note about documenting where a layer doesn't support 'core'
> > functionality.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  documentation/test-manual/yocto-project-compatible.rst | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/documentation/test-manual/yocto-project-compatible.rst b/documentation/test-manual/yocto-project-compatible.rst
> > index a7897469f..b83a0f12b 100644
> > --- a/documentation/test-manual/yocto-project-compatible.rst
> > +++ b/documentation/test-manual/yocto-project-compatible.rst
> > @@ -115,6 +115,11 @@ Here are key best practices the program tries to encourage:
> >     user changes a configuration setting to activate the layer, by selecting
> >     a :term:`MACHINE`, a :term:`DISTRO` or a :term:`DISTRO_FEATURES` setting.
> >  
> > 
> > +-  Layers should be documenting where they don’t support normal 'core'
> > +   functionality such as where debug symbols are disabled or missing, where
> > +   development headers and on-target library usage may not work or where
> 
> 
> Replaced "or" by "and" here, as we are dealing with a list of examples,
> rather than with a list of alternatives.
> https://forum.oxforddictionaries.com/en/discussion/27/such-as-or-or-and

I think it was more correct with "or". From your link, "If you are choosing from 
a list of alternatives, then OR precedes the last item." from that link applies 
here since you are not choosing a list of items and only one of the list may apply. 

> > +   functionality like the SDK/eSDK would not be execpted to work.
> 
> 
> s/execpted/expected/
> 
> This sounded like a system call, too much work with glibc, I guess ;-)
> I took care of this change too, and will merge the change into
> master-next (though extra reviews are still possible).

Thanks :)

Cheers,

Richard

Re: [docs] [PATCH 1/2] test-manual: Add extra detail to YP Compatible section

[Richard Purdie]

On Wed, 2021-08-04 at 12:13 +0200, Quentin Schulz wrote:
> Hi Richard,
> 
> On Wed, Aug 04, 2021 at 12:04:34PM +0200, Michael Opdenacker wrote:
> > Hi Richard,
> > 
> > Thanks for the documentation patch!
> > 
> > On 8/4/21 11:47 AM, Richard Purdie wrote:
> > > Add a note about documenting where a layer doesn't support 'core'
> > > functionality.
> > > 
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > ---
> > >  documentation/test-manual/yocto-project-compatible.rst | 5 +++++
> > >  1 file changed, 5 insertions(+)
> > > 
> > > diff --git a/documentation/test-manual/yocto-project-compatible.rst b/documentation/test-manual/yocto-project-compatible.rst
> > > index a7897469f..b83a0f12b 100644
> > > --- a/documentation/test-manual/yocto-project-compatible.rst
> > > +++ b/documentation/test-manual/yocto-project-compatible.rst
> > > @@ -115,6 +115,11 @@ Here are key best practices the program tries to encourage:
> > >     user changes a configuration setting to activate the layer, by selecting
> > >     a :term:`MACHINE`, a :term:`DISTRO` or a :term:`DISTRO_FEATURES` setting.
> > >  
> > > 
> > > +-  Layers should be documenting where they don’t support normal 'core'
> > > +   functionality such as where debug symbols are disabled or missing, where
> > > +   development headers and on-target library usage may not work or where
> > 
> 
> Do we currently have an example layer that is following this suggested
> best practice? Because I'm struggling to understand exactly what I'm
> expected to do if I were a layer maintainer.

I only know of one layer at present where SDKs were broken, -dbg info was dropped etc. 
and that was a genuine mistake. We (as in the TSC) wanted to make it clear that this 
was not intended behaviour in a YP Compat layer. So no, we don't have an example.

> Do you want the maintainer to keep a list of recipes/packages provided
> in their layer(s), in e.g. the README, that are missing debug symbols or
> "invalid" headers/libraries?

I don't think it needs to be a list, just a statement that the layer is not 
intended to have debug symbols present, or support building an SDK.

Cheers,

Richard

Re: [docs] [PATCH 2/2] migration-3.4: Add extra notes to override sytnax changes

[Quentin Schulz]

Hi Richard,

On Wed, Aug 04, 2021 at 10:47:14AM +0100, Richard Purdie wrote:
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  documentation/migration-guides/migration-3.4.rst | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/documentation/migration-guides/migration-3.4.rst b/documentation/migration-guides/migration-3.4.rst
> index 6fa1ab20c..d42ee8851 100644
> --- a/documentation/migration-guides/migration-3.4.rst
> +++ b/documentation/migration-guides/migration-3.4.rst
> @@ -56,6 +56,12 @@ context being specific tasks in that case. Tune overrides are another special
>  case where some code does use them as overrides but some does not. We plan to try
>  and make the tune code use overrides more consistently in the future.
>  
> +There are some variables which do not use override syntax which include the
> +suffix to variables in layer.conf files such as BBFILE_PATTERN, :term:`SRCREV`_xxx

s/BBFILE_PATTERN/:term:`BBFILE_PATTERN`/

Styling might be better with :term:`SRCREV```_xxx`` (if that works,
might need a "\ " before or something to separate the tick quotes?)?
Maybe worth a try.

It's the first time I see this SRCREV_xxx variable. It does not seem to
be documented except in bitbake/lib/bb/fetch2/git.py. I think we should
document this.

I'm failing to see exactly what the usecase depicted in the
"documentation" in git.py is for, but I imagine a plausible scenario
could be the following?

SRC_URI = " \
	   git://some-git-repo;name=foo \
	   git://some-other-git-repo;name=bar \
	   "

SRCREV_foo = "abcdef1234567890"
SRCREV_bar = "1234567890abcdef"

> +where xxx is a name from :term:`SRC_URI` and PREFERRED_VERSION_xxx. In particular,

s/xxx/``xxx``/ ?

s/PREFERRED_VERSION_xxx/:term:`PREFERRED_VERSION`_xxx/ (if style
suggestion above is chosen, to be applied to this too).

> +layer.conf suffixes may be the same as a :term:`DISTRO` override causing some

s/layer.conf/``layer.conf``/

> +confusion. We do plan to try and improve consistency as these issues are identified.
> +

Is there going to be backwards compatibility issues when supporting e.g.
PREFERRED_VERSION_xxx to PREFERRED_VERSION:xxx?

We can't really afford to break compatibility every time we "fix" those,
is there any plan for this already?

I also think we should keep in mind to update this section every time we
encounter one of those "inconsistencies"?

Thanks,
Quentin

Re: [docs] [PATCH 2/2] migration-3.4: Add extra notes to override sytnax changes

[Richard Purdie]

On Wed, 2021-08-04 at 12:29 +0200, Quentin Schulz wrote:
> Hi Richard,
> 
> On Wed, Aug 04, 2021 at 10:47:14AM +0100, Richard Purdie wrote:
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  documentation/migration-guides/migration-3.4.rst | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/documentation/migration-guides/migration-3.4.rst b/documentation/migration-guides/migration-3.4.rst
> > index 6fa1ab20c..d42ee8851 100644
> > --- a/documentation/migration-guides/migration-3.4.rst
> > +++ b/documentation/migration-guides/migration-3.4.rst
> > @@ -56,6 +56,12 @@ context being specific tasks in that case. Tune overrides are another special
> >  case where some code does use them as overrides but some does not. We plan to try
> >  and make the tune code use overrides more consistently in the future.
> >  
> > 
> > +There are some variables which do not use override syntax which include the
> > +suffix to variables in layer.conf files such as BBFILE_PATTERN, :term:`SRCREV`_xxx
> 
> s/BBFILE_PATTERN/:term:`BBFILE_PATTERN`/
> 
> Styling might be better with :term:`SRCREV```_xxx`` (if that works,
> might need a "\ " before or something to separate the tick quotes?)?
> Maybe worth a try.
> 
> It's the first time I see this SRCREV_xxx variable. It does not seem to
> be documented except in bitbake/lib/bb/fetch2/git.py. I think we should
> document this.
> 
> I'm failing to see exactly what the usecase depicted in the
> "documentation" in git.py is for, but I imagine a plausible scenario
> could be the following?
> 
> SRC_URI = " \
> 	   git://some-git-repo;name=foo \
> 	   git://some-other-git-repo;name=bar \
> 	   "
> 
> SRCREV_foo = "abcdef1234567890"
> SRCREV_bar = "1234567890abcdef"

That is how it is used, yes. Also SRCREV_FORMAT needs to be set in general
with this.

> > +where xxx is a name from :term:`SRC_URI` and PREFERRED_VERSION_xxx. In particular,
> 
> s/xxx/``xxx``/ ?
> 
> s/PREFERRED_VERSION_xxx/:term:`PREFERRED_VERSION`_xxx/ (if style
> suggestion above is chosen, to be applied to this too).
> 
> > +layer.conf suffixes may be the same as a :term:`DISTRO` override causing some
> 
> s/layer.conf/``layer.conf``/
> 
> > +confusion. We do plan to try and improve consistency as these issues are identified.
> > +
> 
> Is there going to be backwards compatibility issues when supporting e.g.
> PREFERRED_VERSION_xxx to PREFERRED_VERSION:xxx?

We're explicitly not changing that right now.  For specific instances like this we 
can likely warn on old usage and convert dynamically for a bit so they are "easier"
to fix later.

> We can't really afford to break compatibility every time we "fix" those,
> is there any plan for this already?
> 
> I also think we should keep in mind to update this section every time we
> encounter one of those "inconsistencies"?

Yes, I suspect we'll have more updates to this section as people convert layers and
we find the corner cases. I want to document the ones we know about whilst we know 
about them and remember!

Cheers,

Richard

Re: [docs] [PATCH 2/2] migration-3.4: Add extra notes to override sytnax changes

[Michael Opdenacker]

Hi Quentin,

Thanks for the review!

On 8/4/21 12:29 PM, Quentin Schulz wrote:
> s/BBFILE_PATTERN/:term:`BBFILE_PATTERN`/
>
> Styling might be better with :term:`SRCREV```_xxx`` (if that works,
> might need a "\ " before or something to separate the tick quotes?)?
> Maybe worth a try.

Fixed. Indeed, a "\ " was needed to make it work.

> s/xxx/``xxx``/ ?
>
> s/PREFERRED_VERSION_xxx/:term:`PREFERRED_VERSION`_xxx/ (if style
> suggestion above is chosen, to be applied to this too).


Done

>
>> +layer.conf suffixes may be the same as a :term:`DISTRO` override causing some
> s/layer.conf/``layer.conf``/


Done. 

I also fixed the typo in the title.

I pushed the changes to "master-next". See the built result in
http://docs.yoctoproject.org/next/migration-guides/migration-3.4.html#override-syntax-changes

Once again, further reviews are still possible. I can also add
"Reviewed-by" statements ;-)

Thanks again,

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH 1/2] test-manual: Add extra detail to YP Compatible section

[Michael Opdenacker]

Hi Richard,

On 8/4/21 12:15 PM, Richard Purdie wrote:
>>> +-  Layers should be documenting where they don’t support normal 'core'
>>> +   functionality such as where debug symbols are disabled or missing, where
>>> +   development headers and on-target library usage may not work or where
>>
>> Replaced "or" by "and" here, as we are dealing with a list of examples,
>> rather than with a list of alternatives.
>> https://forum.oxforddictionaries.com/en/discussion/27/such-as-or-or-and
> I think it was more correct with "or". From your link, "If you are choosing from 
> a list of alternatives, then OR precedes the last item." from that link applies 
> here since you are not choosing a list of items and only one of the list may apply. 


Thanks for double checking this. I considered this as a list of examples
rather than a list of alternatives, but indeed, you need to understand
the exact meaning to know which case you are in.
I fixed this in my commit in "master-next".

Thanks again,

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com