Sstate cache for version 4.0.999?

Sstate cache for version 4.0.999?

[Michael Opdenacker]

Greetings,

In https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html, we
suggest to use:

SSTATE_MIRRORS ?= "file://.* https://sstate.yoctoproject.org/4.0.999/PATH;downloadfilename=PATH"

However, https://sstate.yoctoproject.org/4.0.999/ doesn't seem to exist.
Could we create it?

Thanks in advance
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] Sstate cache for version 4.0.999?

[Quentin Schulz]

Hi Michael,

On 4/27/22 11:34, Michael Opdenacker via lists.yoctoproject.org wrote:
> Greetings,
> 
> In https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_brief-2Dyoctoprojectqs_index.html&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=I0aiuJJTct1SDB_0N89D6xAIxno9IDPyMeI3dhnbF9v4zDEMeK9uUCv3-sR7N9oB&s=Iu7yw2VPisaC_25nQTj_glXbledzMo8_Mc8ek--miks&e= , we
> suggest to use:
> 
> SSTATE_MIRRORS ?= "file://.* https://urldefense.proofpoint.com/v2/url?u=https-3A__sstate.yoctoproject.org_4.0.999_PATH-3Bdownloadfilename-3DPATH&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=I0aiuJJTct1SDB_0N89D6xAIxno9IDPyMeI3dhnbF9v4zDEMeK9uUCv3-sR7N9oB&s=CmYf2gQjYpJvBW3gs6yY_DCrZoFCMw0CgOVkyWIChu8&e= "
> 
> However, https://urldefense.proofpoint.com/v2/url?u=https-3A__sstate.yoctoproject.org_4.0.999_&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=I0aiuJJTct1SDB_0N89D6xAIxno9IDPyMeI3dhnbF9v4zDEMeK9uUCv3-sR7N9oB&s=XBO9beJxxsKb69CRj38-OHCYV5aFixLdie0ruVYc1vo&e=  doesn't seem to exist.
> Could we create it?
> 

Sorry for the botched HTML links, Microsoft O365 just rewrites them 
before the mail lands in my inbox....

4.0.999 represents the kirkstone branch. It is what's between 4.0 and 
4.0.1 tags.

I personally do not want to use 4.0.1 until it is released otherwise it 
may confuse user as to what's actually released and "usable".

I don't know what the sstate cache URL should be for the current 
kirkstone branch (or any branch that is not a tag) though as I don't see 
kirkstone or 4.0.1 subpaths... Could it be that we put sstate-cache of 
kirkstone branch in 4.0 until we have a 4.0.1 tag? Not sure how to 
explain it, but I guess the autobuilder will just put the sstate-cache 
of the current state of the release branch in the last release tag 
sstate-cache directory?

As a side note, the current default docs being the kirkstone (4.0.999) 
branch is something I deem temporary. The issue is that the migration 
manuals and release notes of kirkstone 4.0 *tag* are not up-to-date. The 
ones in kirkstone branch are up-to-date.

I need to write something on the wiki to explain the issue and 
requirements so we can work on fixing this sooner than later.

Cheers,
Quentin

Re: [docs] Sstate cache for version 4.0.999?

[Richard Purdie]

On Wed, 2022-04-27 at 11:34 +0200, Michael Opdenacker via lists.yoctoproject.org
wrote:
> Greetings,
> 
> In https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html, we
> suggest to use:
> 
> SSTATE_MIRRORS ?= "file://.* https://sstate.yoctoproject.org/4.0.999/PATH;downloadfilename=PATH"
> 
> However, https://sstate.yoctoproject.org/4.0.999/ doesn't seem to exist.
> Could we create it?

It needs to be 4.0 for all releases in the kirkstone series.

Cheers,

Richard

Re: [docs] Sstate cache for version 4.0.999?

[Michael Opdenacker]

Hi Richard,

On 4/27/22 12:07, Richard Purdie wrote:
> On Wed, 2022-04-27 at 11:34 +0200, Michael Opdenacker via lists.yoctoproject.org
> wrote:
>> Greetings,
>>
>> In https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html, we
>> suggest to use:
>>
>> SSTATE_MIRRORS ?= "file://.* https://sstate.yoctoproject.org/4.0.999/PATH;downloadfilename=PATH"
>>
>> However, https://sstate.yoctoproject.org/4.0.999/ doesn't seem to exist.
>> Could we create it?
> It needs to be 4.0 for all releases in the kirkstone series.


Unfortunately, I see no variable in poky.yaml.in for this. Could we have
something like a "DISTRO_SERIES" variable which would be "DISTRO" but
without the point release number?

Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] Sstate cache for version 4.0.999?

[Richard Purdie]

On Wed, 2022-04-27 at 12:23 +0200, Michael Opdenacker wrote:
> Hi Richard,
> 
> On 4/27/22 12:07, Richard Purdie wrote:
> > On Wed, 2022-04-27 at 11:34 +0200, Michael Opdenacker via lists.yoctoproject.org
> > wrote:
> > > Greetings,
> > > 
> > > In https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html, we
> > > suggest to use:
> > > 
> > > SSTATE_MIRRORS ?= "file://.* https://sstate.yoctoproject.org/4.0.999/PATH;downloadfilename=PATH"
> > > 
> > > However, https://sstate.yoctoproject.org/4.0.999/ doesn't seem to exist.
> > > Could we create it?
> > It needs to be 4.0 for all releases in the kirkstone series.
> 
> 
> Unfortunately, I see no variable in poky.yaml.in for this. Could we have
> something like a "DISTRO_SERIES" variable which would be "DISTRO" but
> without the point release number?

Or a specific PUBLIC_SSTATE_VERSION variable or similar?

Cheers,

Richard

Re: [docs] Sstate cache for version 4.0.999?

[Quentin Schulz]

Hi all,

On 4/27/22 12:24, Richard Purdie wrote:
> On Wed, 2022-04-27 at 12:23 +0200, Michael Opdenacker wrote:
>> Hi Richard,
>>
>> On 4/27/22 12:07, Richard Purdie wrote:
>>> On Wed, 2022-04-27 at 11:34 +0200, Michael Opdenacker via lists.yoctoproject.org
>>> wrote:
>>>> Greetings,
>>>>
>>>> In https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_brief-2Dyoctoprojectqs_index.html&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=MjQeHR9DHOQsVFvva-_ZotfsR7POIz1TJygkVPGsEn2-goaA86V8vUdPTZ332Qpa&s=CODaCOFarKzmVKW6oZ6KXyqtBSKtrUnzsHftrVzOvuo&e= , we
>>>> suggest to use:
>>>>
>>>> SSTATE_MIRRORS ?= "file://.* https://urldefense.proofpoint.com/v2/url?u=https-3A__sstate.yoctoproject.org_4.0.999_PATH-3Bdownloadfilename-3DPATH&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=MjQeHR9DHOQsVFvva-_ZotfsR7POIz1TJygkVPGsEn2-goaA86V8vUdPTZ332Qpa&s=l8IYvM7AWJwXbzc2csFiERnsApA-RPSvBI9onBSHdXU&e= "
>>>>
>>>> However, https://urldefense.proofpoint.com/v2/url?u=https-3A__sstate.yoctoproject.org_4.0.999_&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=MjQeHR9DHOQsVFvva-_ZotfsR7POIz1TJygkVPGsEn2-goaA86V8vUdPTZ332Qpa&s=OJMppY4I4Tdj_tKNPvS5Tzq52W-bscW1iuxv4y3hpfc&e=  doesn't seem to exist.
>>>> Could we create it?
>>> It needs to be 4.0 for all releases in the kirkstone series.
>>

Mmm.. we do have 3.1.x directories on sstate.yoctoproject.org so I 
assume we should just point to the latest tag (3.1.15, 4.0 until 4.0.1 
is released) of a given series instead of the original tag (3.1, 4.0).

>>
>> Unfortunately, I see no variable in poky.yaml.in for this. Could we have
>> something like a "DISTRO_SERIES" variable which would be "DISTRO" but
>> without the point release number?
> 
> Or a specific PUBLIC_SSTATE_VERSION variable or similar?
> 

I think we actually should have DISTRO set to the latest tag available 
in the series and not have the YOCTO_DOC_VERSION used in sphinx files 
(it's necessary for conf.py though).

What do you think?

Cheers,
Quentin

Re: [docs] Sstate cache for version 4.0.999?

[Richard Purdie]

On Wed, 2022-04-27 at 13:38 +0200, Quentin Schulz wrote:
> Hi all,
> 
> On 4/27/22 12:24, Richard Purdie wrote:
> > On Wed, 2022-04-27 at 12:23 +0200, Michael Opdenacker wrote:
> > > Hi Richard,
> > > 
> > > On 4/27/22 12:07, Richard Purdie wrote:
> > > > On Wed, 2022-04-27 at 11:34 +0200, Michael Opdenacker via lists.yoctoproject.org
> > > > wrote:
> > > > > Greetings,
> > > > > 
> > > > > In https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_brief-2Dyoctoprojectqs_index.html&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=MjQeHR9DHOQsVFvva-_ZotfsR7POIz1TJygkVPGsEn2-goaA86V8vUdPTZ332Qpa&s=CODaCOFarKzmVKW6oZ6KXyqtBSKtrUnzsHftrVzOvuo&e= , we
> > > > > suggest to use:
> > > > > 
> > > > > SSTATE_MIRRORS ?= "file://.* https://urldefense.proofpoint.com/v2/url?u=https-3A__sstate.yoctoproject.org_4.0.999_PATH-3Bdownloadfilename-3DPATH&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=MjQeHR9DHOQsVFvva-_ZotfsR7POIz1TJygkVPGsEn2-goaA86V8vUdPTZ332Qpa&s=l8IYvM7AWJwXbzc2csFiERnsApA-RPSvBI9onBSHdXU&e= "
> > > > > 
> > > > > However, https://urldefense.proofpoint.com/v2/url?u=https-3A__sstate.yoctoproject.org_4.0.999_&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=MjQeHR9DHOQsVFvva-_ZotfsR7POIz1TJygkVPGsEn2-goaA86V8vUdPTZ332Qpa&s=OJMppY4I4Tdj_tKNPvS5Tzq52W-bscW1iuxv4y3hpfc&e=  doesn't seem to exist.
> > > > > Could we create it?
> > > > It needs to be 4.0 for all releases in the kirkstone series.
> > > 
> 
> Mmm.. we do have 3.1.x directories on sstate.yoctoproject.org so I 
> assume we should just point to the latest tag (3.1.15, 4.0 until 4.0.1 
> is released) of a given series instead of the original tag (3.1, 4.0).

Hmm, you're right. I'm confusing the backend non-release build configuration
which is by release series with the published sstate which is done per release.

I'm not convinced it should be by release but you're right about how it works
today. We should look at tweaking that as I'm not sure the current setup is
great.

> 
> > > 
> > > Unfortunately, I see no variable in poky.yaml.in for this. Could we have
> > > something like a "DISTRO_SERIES" variable which would be "DISTRO" but
> > > without the point release number?
> > 
> > Or a specific PUBLIC_SSTATE_VERSION variable or similar?
> > 
> 
> I think we actually should have DISTRO set to the latest tag available 
> in the series and not have the YOCTO_DOC_VERSION used in sphinx files 
> (it's necessary for conf.py though).
> 
> What do you think?

I'm ok with that.

Cheers,

Richard

Re: [docs] Sstate cache for version 4.0.999?

[Michael Opdenacker]

Hi Quentin,

On 4/27/22 11:59, Quentin Schulz wrote:
> As a side note, the current default docs being the kirkstone (4.0.999)
> branch is something I deem temporary. The issue is that the migration
> manuals and release notes of kirkstone 4.0 *tag* are not up-to-date.
> The ones in kirkstone branch are up-to-date.


I guess that is what causes the 4.0 documentation to be out of date. For
example, in
https://docs.yoctoproject.org/4.0/migration-guides/migration-4.0.html#release-notes-for-4-0-kirkstone,
there is still the temporary section called "Changes for release notes"
(https://docs.yoctoproject.org/4.0/migration-guides/migration-4.0.html#changes-for-release-notes).

>
> I need to write something on the wiki to explain the issue and
> requirements so we can work on fixing this sooner than later.


Do we need a 4.0 patch for run-docs-build?
Sorry, got lost with all the recent changes.
Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] Sstate cache for version 4.0.999?

[Quentin Schulz]

Hi Michael,

On April 27, 2022 6:07:41 PM GMT+02:00, "Michael Opdenacker via lists.yoctoproject.org" <michael.opdenacker=bootlin.com@lists.yoctoproject.org> wrote:
>Hi Quentin,
>
>On 4/27/22 11:59, Quentin Schulz wrote:
>> As a side note, the current default docs being the kirkstone (4.0.999)
>> branch is something I deem temporary. The issue is that the migration
>> manuals and release notes of kirkstone 4.0 *tag* are not up-to-date.
>> The ones in kirkstone branch are up-to-date.
>
>
>I guess that is what causes the 4.0 documentation to be out of date. For
>example, in
>https://docs.yoctoproject.org/4.0/migration-guides/migration-4.0.html#release-notes-for-4-0-kirkstone,
>there is still the temporary section called "Changes for release notes"
>(https://docs.yoctoproject.org/4.0/migration-guides/migration-4.0.html#changes-for-release-notes).
>

Yes. The tagged commit for 4.0 didn't include (some of?) last week's work which is why it's missing from the release notes.

>>
>> I need to write something on the wiki to explain the issue and
>> requirements so we can work on fixing this sooner than later.
>
>
>Do we need a 4.0 patch for run-docs-build?

It's not technically one patch but more like 6. I know which ones to backport, it was just decided to use kirkstone branch as default page instead of 4.0 tag. This can be revisited I guess?

We can patch if you want but the point is that release and migration manuals are generated from the docs at a given commit which is bad and makes it impossible to be up-to-date. For example, some fix was sent recently for 3.4 migration manual but it won't make it to 3.4 release unless we manually patch it on yp-autobuilder-helper script. This is bad.

Also, in 3.1 docs, we only have migration manuals and release notes for <3.1 if I'm not mistaken.

I would personally very much be in favor of having migration manuals and release notes pages common to all versions of the docs.

I emitted the idea of using the transition branch for this purpose. What I suggested is to have a migration directory for each release where you only have the migration manual from release n-1 to release n and release notes from release n. That's it.
What is cool with this is that we don't need to remove references or :term: of older migration manuals once the variables disappear from Bitbake or Yocto glossaries (an issue we recently had). It also directly links to the correct version of Bitbake docs for the links. E.g. if a variable disappears in n+1, we need to remove the ref/term to it in release n-2, n-1 and n migration manuals while they shouldn't be.

The not so nice thing about this is that the migration manuals and release notes are not part of a given release of the docs anymore, which is not really acceptable.

>Sorry, got lost with all the recent changes.

Yeah, we went on a commit spree last week for the docs git repo and run-docs-build. It should have fixed a few issues and improved some things hopefully.

Cheers,
Quentin

Re: [docs] Sstate cache for version 4.0.999?

[Michael Opdenacker]

Hi Quentin,

Thanks for the clarifications!

On 4/27/22 19:14, Quentin Schulz wrote:
> Hi Michael,
>
> On April 27, 2022 6:07:41 PM GMT+02:00, "Michael Opdenacker via lists.yoctoproject.org" <michael.opdenacker=bootlin.com@lists.yoctoproject.org> wrote:
>> Hi Quentin,
>>
>> On 4/27/22 11:59, Quentin Schulz wrote:
>>> As a side note, the current default docs being the kirkstone (4.0.999)
>>> branch is something I deem temporary. The issue is that the migration
>>> manuals and release notes of kirkstone 4.0 *tag* are not up-to-date.
>>> The ones in kirkstone branch are up-to-date.
>>
>> I guess that is what causes the 4.0 documentation to be out of date. For
>> example, in
>> https://docs.yoctoproject.org/4.0/migration-guides/migration-4.0.html#release-notes-for-4-0-kirkstone,
>> there is still the temporary section called "Changes for release notes"
>> (https://docs.yoctoproject.org/4.0/migration-guides/migration-4.0.html#changes-for-release-notes).
>>
> Yes. The tagged commit for 4.0 didn't include (some of?) last week's work which is why it's missing from the release notes.
>
>>> I need to write something on the wiki to explain the issue and
>>> requirements so we can work on fixing this sooner than later.
>>
>> Do we need a 4.0 patch for run-docs-build?
> It's not technically one patch but more like 6. I know which ones to backport, it was just decided to use kirkstone branch as default page instead of 4.0 tag. This can be revisited I guess?
>
> We can patch if you want but the point is that release and migration manuals are generated from the docs at a given commit which is bad and makes it impossible to be up-to-date. For example, some fix was sent recently for 3.4 migration manual but it won't make it to 3.4 release unless we manually patch it on yp-autobuilder-helper script. This is bad.
>
> Also, in 3.1 docs, we only have migration manuals and release notes for <3.1 if I'm not mistaken.
>
> I would personally very much be in favor of having migration manuals and release notes pages common to all versions of the docs.
>
> I emitted the idea of using the transition branch for this purpose. What I suggested is to have a migration directory for each release where you only have the migration manual from release n-1 to release n and release notes from release n. That's it.
> What is cool with this is that we don't need to remove references or :term: of older migration manuals once the variables disappear from Bitbake or Yocto glossaries (an issue we recently had). It also directly links to the correct version of Bitbake docs for the links. E.g. if a variable disappears in n+1, we need to remove the ref/term to it in release n-2, n-1 and n migration manuals while they shouldn't be.
>
> The not so nice thing about this is that the migration manuals and release notes are not part of a given release of the docs anymore, which is not really acceptable.


Interesting idea indeed. Don't hesitate to propose patches when you have
time :)
Thanks again

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com