master/main branch renaming and bitbake

master/main branch renaming and bitbake

[colin walters]

(Sorry I think this isn't the right list, but I happen to still be subscribed to this one so...)

Can someone please tell me there's a fix for yocto/bitbake failing if the master branch is renamed?

https://github.com/ostreedev/ostree/issues/2360

Re: [OE-core] master/main branch renaming and bitbake

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

For ostree, yes:
http://git.openembedded.org/meta-openembedded/log/?h=master-next

For the generic case, no. It's not a good idea to start guessing what the
upstream did.

Alex


On Wed, 12 May 2021 at 22:34, colin walters <walters@verbum.org> wrote:

> (Sorry I think this isn't the right list, but I happen to still be
> subscribed to this one so...)
>
> Can someone please tell me there's a fix for yocto/bitbake failing if the
> master branch is renamed?
>
> https://github.com/ostreedev/ostree/issues/2360
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 1123 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

meta-oe/master
https://lists.openembedded.org/g/openembedded-devel/message/91252
meta-oe/hardknott
https://lists.openembedded.org/g/openembedded-devel/message/91253
meta-oe/gatesgarth
https://lists.openembedded.org/g/openembedded-devel/message/91254
meta-oe/dunfell
https://lists.openembedded.org/g/openembedded-devel/message/91255

It will be a while until it's merged in these still supported branches, for
older branches it would be responsibility of the unlucky people using
unsupported release to fix it on their own with .bbappend.

On Wed, May 12, 2021 at 10:34 PM colin walters <walters@verbum.org> wrote:

> (Sorry I think this isn't the right list, but I happen to still be
> subscribed to this one so...)
>
> Can someone please tell me there's a fix for yocto/bitbake failing if the
> master branch is renamed?
>
> https://github.com/ostreedev/ostree/issues/2360
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 1629 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[colin walters]

On Wed, May 12, 2021, at 4:40 PM, Alexander Kanavin wrote:
> For ostree, yes:
> http://git.openembedded.org/meta-openembedded/log/?h=master-next
> 
> For the generic case, no. It's not a good idea to start guessing what 
> the upstream did.

What is the goal of the `branch=` specification?  I can understand it being *informative* for humans specifically when things like non-master/main branches e.g. `release-4.x`, `lts-`, `stable-` etc. are involved.

But why is bitbake explicitly checking it?  Is it to validate what the human expressed, or is it to try to cover some security aspect?  Something else?

Re: [OE-core] master/main branch renaming and bitbake

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

On Wed, 12 May 2021 at 22:44, Colin Walters <walters@verbum.org> wrote:

> On Wed, May 12, 2021, at 4:40 PM, Alexander Kanavin wrote:
> > For ostree, yes:
> > http://git.openembedded.org/meta-openembedded/log/?h=master-next
> >
> > For the generic case, no. It's not a good idea to start guessing what
> > the upstream did.
>
> What is the goal of the `branch=` specification?  I can understand it
> being *informative* for humans specifically when things like
> non-master/main branches e.g. `release-4.x`, `lts-`, `stable-` etc. are
> involved.
>
> But why is bitbake explicitly checking it?  Is it to validate what the
> human expressed, or is it to try to cover some security aspect?  Something
> else?
>

To ensure the specified revision is actually linked with the specified
branch, and not just free-floating, and catch branch renames, deletions and
force pushes. You can override that behaviour with nobranch=1, at your own
risk - free-floating revisions can be 'garbage collected'.

Yes, it's annoying when the upstream renames branches, but it's no
different than upstreams clearing up old tarballs, or disappearing
altogether. Ensure you have proper download mirrors.

Alex

[-- Attachment #2: Type: text/html, Size: 1711 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1662 bytes --]

And by the way, another reason to check that revision is linked to a branch
is when SRCREV is updated - we need some reassurance that the updated
SRCREV comes from the same branch as previous SRCREV, or that if the branch
has changed, it's explicitly visible in the diff and explained in the
commit message.

Alex

On Wed, 12 May 2021 at 23:17, Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> On Wed, 12 May 2021 at 22:44, Colin Walters <walters@verbum.org> wrote:
>
>> On Wed, May 12, 2021, at 4:40 PM, Alexander Kanavin wrote:
>> > For ostree, yes:
>> > http://git.openembedded.org/meta-openembedded/log/?h=master-next
>> >
>> > For the generic case, no. It's not a good idea to start guessing what
>> > the upstream did.
>>
>> What is the goal of the `branch=` specification?  I can understand it
>> being *informative* for humans specifically when things like
>> non-master/main branches e.g. `release-4.x`, `lts-`, `stable-` etc. are
>> involved.
>>
>> But why is bitbake explicitly checking it?  Is it to validate what the
>> human expressed, or is it to try to cover some security aspect?  Something
>> else?
>>
>
> To ensure the specified revision is actually linked with the specified
> branch, and not just free-floating, and catch branch renames, deletions and
> force pushes. You can override that behaviour with nobranch=1, at your own
> risk - free-floating revisions can be 'garbage collected'.
>
> Yes, it's annoying when the upstream renames branches, but it's no
> different than upstreams clearing up old tarballs, or disappearing
> altogether. Ensure you have proper download mirrors.
>
> Alex
>

[-- Attachment #2: Type: text/html, Size: 2448 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Andre McCurdy]

On Wed, May 12, 2021 at 2:23 PM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> And by the way, another reason to check that revision is linked to a branch is when SRCREV is updated - we need some reassurance that the updated SRCREV comes from the same branch as previous SRCREV, or that if the branch has changed, it's explicitly visible in the diff and explained in the commit message.

None of the answers given so far seem very convincing...

If the git revision that a recipe wants is available on an unexpected
branch in the upstream repo then it's not really different from a tar
file being fetched from a mirror rather than whatever is in SRC_URI.
If we want the fetcher to fail as an indication that an upstream git
repo has renamed branches then don't we also want it to fail if a tar
file disappears from an upstream server? It seems odd that one should
be a fatal error and the other to be something we try to cover up and
hide from the end user.

Anyway, for recipes which don't explicitly specify a branch in SRC_URI
it would seem quite reasonable for the fetcher to check what the
default branch is set to in the upstream repo and search for the
required git revision in that branch (rather than rely on a hardcoded
default of "master" as we do now). Going forward, there's going to be
less standardisation on what upstream repos call their default branch,
so we're either going to have to explicitly specify a branch in more
and more recipes or teach the fetcher to automatically figure out what
the default branch in the upstream repo is.

> Alex
>
> On Wed, 12 May 2021 at 23:17, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>>
>> On Wed, 12 May 2021 at 22:44, Colin Walters <walters@verbum.org> wrote:
>>>
>>> On Wed, May 12, 2021, at 4:40 PM, Alexander Kanavin wrote:
>>> > For ostree, yes:
>>> > http://git.openembedded.org/meta-openembedded/log/?h=master-next
>>> >
>>> > For the generic case, no. It's not a good idea to start guessing what
>>> > the upstream did.
>>>
>>> What is the goal of the `branch=` specification?  I can understand it being *informative* for humans specifically when things like non-master/main branches e.g. `release-4.x`, `lts-`, `stable-` etc. are involved.
>>>
>>> But why is bitbake explicitly checking it?  Is it to validate what the human expressed, or is it to try to cover some security aspect?  Something else?
>>
>>
>> To ensure the specified revision is actually linked with the specified branch, and not just free-floating, and catch branch renames, deletions and force pushes. You can override that behaviour with nobranch=1, at your own risk - free-floating revisions can be 'garbage collected'.
>>
>> Yes, it's annoying when the upstream renames branches, but it's no different than upstreams clearing up old tarballs, or disappearing altogether. Ensure you have proper download mirrors.
>>
>> Alex
>
>
> 
>

Re: [OE-core] master/main branch renaming and bitbake

[Richard Purdie]

On Wed, 2021-05-12 at 17:11 -0700, Andre McCurdy wrote:
> On Wed, May 12, 2021 at 2:23 PM Alexander Kanavin
> <alex.kanavin@gmail.com> wrote:
> > 
> > And by the way, another reason to check that revision is linked to a 
> > branch is when SRCREV is updated - we need some reassurance that the 
> > updated SRCREV comes from the same branch as previous SRCREV, or that
> > if the branch has changed, it's explicitly visible in the diff and explained in the commit message.
> 
> None of the answers given so far seem very convincing...

I had a look at the code to try and remind myself why it is doing this. 
The best answer I found is that it does support filtered fetching of 
remotes, i.e. it can and will only pull the branch it wants/needs rather
than a full repo clone. In the case of a small repo, it doesn't matter
much. For a large repo it can make a significant difference to the speed.
We also don't run "test" commands against the remote repo, we figure out
what we want and then get it with small numbers of commands.

Another reason we have the branch name is so that we can ls-remote the repo
when using AUTOREV for SRCREV. We've tried to make it so that once a url
is established in SRC_URI, you can make it use AUTOREV without changes to 
the url itself. The AUTOREV code path is in parsing which does mean we
need to be efficient about the revision resolution.

> If the git revision that a recipe wants is available on an unexpected
> branch in the upstream repo then it's not really different from a tar
> file being fetched from a mirror rather than whatever is in SRC_URI.
> If we want the fetcher to fail as an indication that an upstream git
> repo has renamed branches then don't we also want it to fail if a tar
> file disappears from an upstream server? It seems odd that one should
> be a fatal error and the other to be something we try to cover up and
> hide from the end user.

The fetcher is consistent. It will fail in both cases unless a source
mirror is available with the sources in. If a source mirror is available,
it should work.

> Anyway, for recipes which don't explicitly specify a branch in SRC_URI
> it would seem quite reasonable for the fetcher to check what the
> default branch is set to in the upstream repo and search for the
> required git revision in that branch (rather than rely on a hardcoded
> default of "master" as we do now). Going forward, there's going to be
> less standardisation on what upstream repos call their default branch,
> so we're either going to have to explicitly specify a branch in more
> and more recipes or teach the fetcher to automatically figure out what
> the default branch in the upstream repo is.

Is explicitly specifying the branch along with the repo url such a big 
problem? We already have to provide the url, it isn't like we guess that
too.

Cheers,

Richard

Re: [OE-core] master/main branch renaming and bitbake

[colin walters]

On Thu, May 13, 2021, at 3:08 AM, Richard Purdie wrote:
>
> I had a look at the code to try and remind myself why it is doing this. 

Thanks!

> The best answer I found is that it does support filtered fetching of 
> remotes, i.e. it can and will only pull the branch it wants/needs rather
> than a full repo clone. In the case of a small repo, it doesn't matter
> much. For a large repo it can make a significant difference to the speed.

This is with `git clone --single-branch`?  Makes sense.

> We also don't run "test" commands against the remote repo, we figure out
> what we want and then get it with small numbers of commands.

I think what I'd argue is that in the case where the remote branch is deleted,
the tooling should fall back to listing remote branches, sort by
most recently updated, and try finding the commit on that.
And in practice, explicitly using `main` first in that list would make sense.

> Is explicitly specifying the branch along with the repo url such a big 
> problem? We already have to provide the url, it isn't like we guess that
> too.

The problem is that your build system is penalizing upstream projects that are trying to migrate
to using `main`.

Re: [OE-core] master/main branch renaming and bitbake

[Bruce Ashfield]

On Thu, May 13, 2021 at 9:27 AM colin walters <walters@verbum.org> wrote:
>
>
>
> On Thu, May 13, 2021, at 3:08 AM, Richard Purdie wrote:
> >
> > I had a look at the code to try and remind myself why it is doing this.
>
> Thanks!
>
> > The best answer I found is that it does support filtered fetching of
> > remotes, i.e. it can and will only pull the branch it wants/needs rather
> > than a full repo clone. In the case of a small repo, it doesn't matter
> > much. For a large repo it can make a significant difference to the speed.
>
> This is with `git clone --single-branch`?  Makes sense.
>
> > We also don't run "test" commands against the remote repo, we figure out
> > what we want and then get it with small numbers of commands.
>
> I think what I'd argue is that in the case where the remote branch is deleted,
> the tooling should fall back to listing remote branches, sort by
> most recently updated, and try finding the commit on that.
> And in practice, explicitly using `main` first in that list would make sense.

The tools really shouldn't be making decisions like this on a recipes'
behalf. It is complex and just doesn't need to exist when we can
simply specify the branch.

>
> > Is explicitly specifying the branch along with the repo url such a big
> > problem? We already have to provide the url, it isn't like we guess that
> > too.
>
> The problem is that your build system is penalizing upstream projects that are trying to migrate
> to using `main`.
>

Similarly those same projects are completely deleting those branches,
versus no longer making them the default. Which is penalizing the
existing users.

Bruce



>
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] master/main branch renaming and bitbake

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 13 maj 2021 09:09
> To: Andre McCurdy <armccurdy@gmail.com>; Alexander Kanavin
> <alex.kanavin@gmail.com>
> Cc: Colin Walters <walters@verbum.org>; OE-core <openembedded-
> core@lists.openembedded.org>
> Subject: Re: [OE-core] master/main branch renaming and bitbake
> 
> On Wed, 2021-05-12 at 17:11 -0700, Andre McCurdy wrote:
> > On Wed, May 12, 2021 at 2:23 PM Alexander Kanavin
> > <alex.kanavin@gmail.com> wrote:
> > >
> > > And by the way, another reason to check that revision is linked to a
> > > branch is when SRCREV is updated - we need some reassurance that the
> > > updated SRCREV comes from the same branch as previous SRCREV, or that
> > > if the branch has changed, it's explicitly visible in the diff and
> explained in the commit message.
> >
> > None of the answers given so far seem very convincing...
> 
> I had a look at the code to try and remind myself why it is doing this.
> The best answer I found is that it does support filtered fetching of
> remotes, i.e. it can and will only pull the branch it wants/needs rather
> than a full repo clone. In the case of a small repo, it doesn't matter
> much. For a large repo it can make a significant difference to the speed.
> We also don't run "test" commands against the remote repo, we figure out
> what we want and then get it with small numbers of commands.
> 
> Another reason we have the branch name is so that we can ls-remote the
> repo
> when using AUTOREV for SRCREV. We've tried to make it so that once a url
> is established in SRC_URI, you can make it use AUTOREV without changes to
> the url itself. The AUTOREV code path is in parsing which does mean we
> need to be efficient about the revision resolution.

Wouldn't using "HEAD" instead still maintain the same properties as using 
"master", but with the added benefit that it just works for repositories 
that decide to use some other branch than "master" as their main branch?
E.g., using `git clone --single-branch` (without -b) fetches HEAD and 
whatever branch it references, which is what we want, is it not?

> > If the git revision that a recipe wants is available on an unexpected
> > branch in the upstream repo then it's not really different from a tar
> > file being fetched from a mirror rather than whatever is in SRC_URI.
> > If we want the fetcher to fail as an indication that an upstream git
> > repo has renamed branches then don't we also want it to fail if a tar
> > file disappears from an upstream server? It seems odd that one should
> > be a fatal error and the other to be something we try to cover up and
> > hide from the end user.
> 
> The fetcher is consistent. It will fail in both cases unless a source
> mirror is available with the sources in. If a source mirror is available,
> it should work.
> 
> > Anyway, for recipes which don't explicitly specify a branch in SRC_URI
> > it would seem quite reasonable for the fetcher to check what the
> > default branch is set to in the upstream repo and search for the
> > required git revision in that branch (rather than rely on a hardcoded
> > default of "master" as we do now). Going forward, there's going to be
> > less standardisation on what upstream repos call their default branch,
> > so we're either going to have to explicitly specify a branch in more
> > and more recipes or teach the fetcher to automatically figure out what
> > the default branch in the upstream repo is.
> 
> Is explicitly specifying the branch along with the repo url such a big
> problem? We already have to provide the url, it isn't like we guess that
> too.

One problem I have with the way the branch is specified is that it is a 
URL parameter. This means that when I want to add a bbappend to specify 
another SRCREV, it becomes problematic if that SRCREV is on a different 
branch than what the base recipe uses. It basically means one has to 
use SRC_URI_remove to remove the original URL for the repository, and 
then add a new version of the URL with SRC_URI_prepend. This obviously 
is unrelated to the discussion about main vs. master, but I would very 
much like to see a new variable, e.g., SRC_BRANCH, being added to 
complement SRCREV. There is an edge case for recipes that use multiple 
Git URLs in the SRC_URI, but they are rare, so having to specify the 
branch using the branch= parameter in the unlikely event that those 
repositories need different branches should be acceptable.

> Cheers,
> 
> Richard

//Peter

Re: master/main branch renaming and bitbake

[Benjamin Gilbert]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

On Thu, May 13, 2021 at 06:36 AM, Bruce Ashfield wrote:

> 
> 
>> The problem is that your build system is penalizing upstream projects that
>> are trying to migrate
>> to using `main`.
> 
> Similarly those same projects are completely deleting those branches,
> versus no longer making them the default. Which is penalizing the
> existing users.

Deleting the branch penalizes existing developers, which projects might be willing to do.  Git branch names don't usually have any effect on users.

--Benjamin Gilbert

[-- Attachment #2: Type: text/html, Size: 582 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Bruce Ashfield]

On Thu, May 13, 2021 at 3:02 PM Benjamin Gilbert <bgilbert@backtick.net> wrote:
>
> On Thu, May 13, 2021 at 06:36 AM, Bruce Ashfield wrote:
>
> The problem is that your build system is penalizing upstream projects that are trying to migrate
> to using `main`.
>
> Similarly those same projects are completely deleting those branches,
> versus no longer making them the default. Which is penalizing the
> existing users.
>
> Deleting the branch penalizes existing developers, which projects might be willing to do.  Git branch names don't usually have any effect on users.
>

I'm not sure I see the distinction you are trying to draw.

We are somewhere between developers and users, but again .. it's not
really a relevant distinction.

Bruce

> --Benjamin Gilbert
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: master/main branch renaming and bitbake

[Benjamin Gilbert]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On Thu, May 13, 2021 at 12:04 PM, Bruce Ashfield wrote:

> 
> 
>> Deleting the branch penalizes existing developers, which projects might be
>> willing to do. Git branch names don't usually have any effect on users.
> 
> I'm not sure I see the distinction you are trying to draw.

Deleting the branch normally affects people who are in the project's developer community and pay attention to its announcements.  Those folks can then simply update their local checkouts and move on with their lives.  Everyone else just clones the default branch and maybe checks out a particular tag or commit before building a package.  That workflow still works after the default branch is renamed.  Bitbake's additional checks are unusual, and impose a long-term compatibility constraint on upstream projects that they didn't sign up for.

--Benjamin Gilbert

[-- Attachment #2: Type: text/html, Size: 917 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Bruce Ashfield]

On Thu, May 13, 2021 at 3:25 PM Benjamin Gilbert <bgilbert@backtick.net> wrote:
>
> On Thu, May 13, 2021 at 12:04 PM, Bruce Ashfield wrote:
>
> Deleting the branch penalizes existing developers, which projects might be willing to do. Git branch names don't usually have any effect on users.
>
> I'm not sure I see the distinction you are trying to draw.
>
> Deleting the branch normally affects people who are in the project's developer community and pay attention to its announcements.  Those folks can then simply update their local checkouts and move on with their lives.  Everyone else just clones the default branch and maybe checks out a particular tag or commit before building a package.  That workflow still works after the default branch is renamed.  Bitbake's additional checks are unusual, and impose a long-term compatibility constraint on upstream projects that they didn't sign up for.

By "user",  I simply meant consumer. (I'm aware of what a developer is
and how they typically work).

We'll agree to disagree on the finer points, since arguing semantics
of good repository management/behaviour is not a good use of time.

Cheers,

Bruce

>
> --Benjamin Gilbert
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] master/main branch renaming and bitbake

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

On Thu, 13 May 2021 at 21:25, Benjamin Gilbert <bgilbert@backtick.net>
wrote:

> Deleting the branch normally affects people who are in the project's
> developer community and pay attention to its announcements.  Those folks
> can then simply update their local checkouts and move on with their lives.
> Everyone else just clones the default branch and maybe checks out a
> particular tag or commit before building a package.  That workflow still
> works after the default branch is renamed.  Bitbake's additional checks are
> unusual, and impose a long-term compatibility constraint on upstream
> projects that they didn't sign up for.
>

They're not useless though. As a layer maintainer I absolutely want those
checks, as I want to be aware of any branch configuration changes upstream,
and to ensure I'm not accepting a change in source revision that points to
an unattached commit, or one from a branch that's not supposed to be used
at all.

Alex

[-- Attachment #2: Type: text/html, Size: 1298 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Konrad Weihmann]

And IMHO deleting branches upstream is just bad practice. From a 
Yocto/OE perspective I have to consider that people might want to build 
stuff in 5yrs or even 10yrs from now.
a more reasonable pattern would be to branch off from master to main, 
make main the new default but leave master untouched.

But this hard renaming of branches causes headaches when maintaining layers.

Also I'd like to point out that there are plenty of examples where it's 
just not the master>main transition, but projects deleting whole release 
branches without a trace - so for me these extra security measures do 
make sense and are really welcome.

On 13.05.21 21:36, Alexander Kanavin wrote:
> On Thu, 13 May 2021 at 21:25, Benjamin Gilbert <bgilbert@backtick.net 
> <mailto:bgilbert@backtick.net>> wrote:
> 
>     Deleting the branch normally affects people who are in the project's
>     developer community and pay attention to its announcements.  Those
>     folks can then simply update their local checkouts and move on with
>     their lives.  Everyone else just clones the default branch and maybe
>     checks out a particular tag or commit before building a package. 
>     That workflow still works after the default branch is renamed. 
>     Bitbake's additional checks are unusual, and impose a long-term
>     compatibility constraint on upstream projects that they didn't sign
>     up for.
> 
> 
> They're not useless though. As a layer maintainer I absolutely want 
> those checks, as I want to be aware of any branch configuration changes 
> upstream, and to ensure I'm not accepting a change in source revision 
> that points to an unattached commit, or one from a branch that's not 
> supposed to be used at all.
> 
> Alex
> 
> 
> 
> 

Re: master/main branch renaming and bitbake

[Benjamin Gilbert]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Thu, May 13, 2021 at 12:36 PM, Alexander Kanavin wrote:

> 
> They're not useless though. As a layer maintainer I absolutely want those
> checks, as I want to be aware of any branch configuration changes
> upstream, and to ensure I'm not accepting a change in source revision that
> points to an unattached commit, or one from a branch that's not supposed
> to be used at all.
> 

Yup, if downstream packagers want to manually review such cases, no objection from me.  My concern is upstream getting bug reports that say "please reinstate this branch and never delete it again or you'll break our builds".  See https://github.com/ostreedev/ostree/issues/2360 and https://github.com/coreos/go-systemd/issues/371 for examples.

Best,
--Benjamin Gilbert

[-- Attachment #2: Type: text/html, Size: 996 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Konrad Weihmann]

On 13.05.21 21:54, Benjamin Gilbert wrote:
> On Thu, May 13, 2021 at 12:36 PM, Alexander Kanavin wrote:
> 
>     They're not useless though. As a layer maintainer I absolutely want
>     those checks, as I want to be aware of any branch configuration
>     changes upstream, and to ensure I'm not accepting a change in source
>     revision that points to an unattached commit, or one from a branch
>     that's not supposed to be used at all.
> 
> Yup, if downstream packagers want to manually review such cases, no 
> objection from me.  My concern is upstream getting bug reports that say 
> "please reinstate this branch and never delete it again or you'll break 
> our builds".  See https://github.com/ostreedev/ostree/issues/2360 
> <https://github.com/ostreedev/ostree/issues/2360> and 
> https://github.com/coreos/go-systemd/issues/371 
> <https://github.com/coreos/go-systemd/issues/371> for examples.

But the genie has been out of the bottle already - revision x was 
published on branch y - so for me changing the (branch, revision) tuple 
is as suspicious as a forced rewrite.

> 
> Best,
> --Benjamin Gilbert
> 
> 
> 
> 

Re: [OE-core] master/main branch renaming and bitbake

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Thu, 13 May 2021 at 21:54, Benjamin Gilbert <bgilbert@backtick.net>
wrote:

> They're not useless though. As a layer maintainer I absolutely want those
> checks, as I want to be aware of any branch configuration changes upstream,
> and to ensure I'm not accepting a change in source revision that points to
> an unattached commit, or one from a branch that's not supposed to be used
> at all.
>
> Yup, if downstream packagers want to manually review such cases, no
> objection from me.  My concern is upstream getting bug reports that say
> "please reinstate this branch and never delete it again or you'll break our
> builds".  See https://github.com/ostreedev/ostree/issues/2360 and
> https://github.com/coreos/go-systemd/issues/371 for examples.
>

It's not that different from deleting or archiving old tarball downloads -
yes it keeps things clean and tidy for upstream, but it will break
someone's automated build that way, no matter how obsolete the code being
downloaded.

Alex

[-- Attachment #2: Type: text/html, Size: 1551 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Richard Purdie]

On Thu, 2021-05-13 at 09:27 -0400, colin walters wrote:
> 
> On Thu, May 13, 2021, at 3:08 AM, Richard Purdie wrote:
> > 
> > I had a look at the code to try and remind myself why it is doing this. 
> 
> Thanks!
> 
> > The best answer I found is that it does support filtered fetching of 
> > remotes, i.e. it can and will only pull the branch it wants/needs rather
> > than a full repo clone. In the case of a small repo, it doesn't matter
> > much. For a large repo it can make a significant difference to the speed.
> 
> This is with `git clone --single-branch`?  Makes sense.
> 
> > We also don't run "test" commands against the remote repo, we figure out
> > what we want and then get it with small numbers of commands.
> 
> I think what I'd argue is that in the case where the remote branch is deleted,
> the tooling should fall back to listing remote branches, sort by
> most recently updated, and try finding the commit on that.
> And in practice, explicitly using `main` first in that list would make sense.
> 
> > Is explicitly specifying the branch along with the repo url such a big 
> > problem? We already have to provide the url, it isn't like we guess that
> > too.
> 
> The problem is that your build system is penalizing upstream projects that 
> are trying to migrate to using `main`.

Some of our users have complained, yes. That isn't a conscious decision on 
"our" part, just a rather unfortunate and unplanned consequence of the fact
we're pretty strict about declaring where we get our sources.

The advice to anyone hitting this issue is to add in the correct branch
to the SRC_URI. It is simple and easy to do, can be in bbappends 
or even changed via anonymous python and similar if necessary. We've already
found the issue with several core recipes, we simply updated them and most 
users didn't notice. I would even likely take that kind of change into older
otherwise unmaintained branches and I think I did so in at least one case in
the past.

There is no intent to peanlize or complain, my intent is just to deal with
it like any other upstream source change, these things happen all the time. 
I am stating that here for any other maintainers to read and suggest they 
follow oe-core's lead.

I appreciate the tooling could do all kinds of magic things. I have a strong
preference for not adding magic into it, or over complicating it, it is already
horrendously complicated and a nightmare to test. I appreciate nobody believes
me, I only do my best to maintain it. The code is here for anyone interested:

http://git.yoctoproject.org/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/git.py

I'd also note you can add ;nobranch=1 to the urls or ;usehead=1. Those do
have side effects, I will not recommend them, or accept them for general use
in layers I maintain, they're considered developer options. I was reminded
recently that we have seen bugs the branch parameter has caught where a 
revision was not where we thought it was so these do catch real world issues.

Cheers,

Richard

Re: [OE-core] master/main branch renaming and bitbake

[Richard Purdie]

On Thu, 2021-05-13 at 14:25 +0000, Peter Kjellerstedt wrote:
> > Another reason we have the branch name is so that we can ls-remote the
> > repo
> > when using AUTOREV for SRCREV. We've tried to make it so that once a url
> > is established in SRC_URI, you can make it use AUTOREV without changes to
> > the url itself. The AUTOREV code path is in parsing which does mean we
> > need to be efficient about the revision resolution.
> 
> Wouldn't using "HEAD" instead still maintain the same properties as using 
> "master", but with the added benefit that it just works for repositories 
> that decide to use some other branch than "master" as their main branch?
> E.g., using `git clone --single-branch` (without -b) fetches HEAD and 
> whatever branch it references, which is what we want, is it not?
> > 

See the ;usehead=1 uri option.

> > 
> One problem I have with the way the branch is specified is that it is a 
> URL parameter. This means that when I want to add a bbappend to specify 
> another SRCREV, it becomes problematic if that SRCREV is on a different 
> branch than what the base recipe uses. It basically means one has to 
> use SRC_URI_remove to remove the original URL for the repository, and 
> then add a new version of the URL with SRC_URI_prepend. This obviously 
> is unrelated to the discussion about main vs. master, but I would very 
> much like to see a new variable, e.g., SRC_BRANCH, being added to 
> complement SRCREV. There is an edge case for recipes that use multiple 
> Git URLs in the SRC_URI, but they are rare, so having to specify the 
> branch using the branch= parameter in the unlikely event that those 
> repositories need different branches should be acceptable.

This is getting a little sidetracked from the original discussion. I believe
anyone changing branch urls like that can probably parameterise the url
themselves and there isn't generic demand to add more complexity...

What I do suspect we need is the ability to change url parameters in 
mirroring url mappings, which could possibly help for a few of these 
cases too. There is an open bug for that, I keep meaning to try and sort
it out.

Cheers,

Richard

Re: master/main branch renaming and bitbake

[Benjamin Gilbert]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

On Thu, May 13, 2021 at 01:07 PM, Alexander Kanavin wrote:

> 
> It's not that different from deleting or archiving old tarball downloads -
> yes it keeps things clean and tidy for upstream, but it will break
> someone's automated build that way, no matter how obsolete the code being
> downloaded.
> 

Tarballs, commits on published branches, and annotated tags are all supposed to be stable artifacts.  But I don't know of any general expectation that a specific branch continues to exist, other than the one created by Bitbake.  GitHub's branch rename tool doesn't leave the old branch behind, not even a symref, so I don't think this is a fringe viewpoint.  And as I said, automated builds usually only care about tags, commits, and the default branch.

I genuinely appreciate the discussion, all.  In the two upstream projects I mentioned, I'm going to recommend that we re-delete the master branches after a reasonable migration period has elapsed.

Best,
--Benjamin Gilbert

[-- Attachment #2: Type: text/html, Size: 1065 bytes --]

Re: master/main branch renaming and bitbake

[Benjamin Gilbert]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Thu, May 13, 2021 at 01:09 PM, Richard Purdie wrote:

> 
> Some of our users have complained, yes. That isn't a conscious decision on
> 
> "our" part, just a rather unfortunate and unplanned consequence of the
> fact
> we're pretty strict about declaring where we get our sources.
> 
> The advice to anyone hitting this issue is to add in the correct branch
> to the SRC_URI. It is simple and easy to do, can be in bbappends
> or even changed via anonymous python and similar if necessary. We've
> already
> found the issue with several core recipes, we simply updated them and most
> 
> users didn't notice. I would even likely take that kind of change into
> older
> otherwise unmaintained branches and I think I did so in at least one case
> in
> the past.
> 
> There is no intent to peanlize or complain, my intent is just to deal with
> 
> it like any other upstream source change, these things happen all the
> time.
> I am stating that here for any other maintainers to read and suggest they
> follow oe-core's lead.

Whoops, just saw this.  Thanks for the position statement, Richard; very much appreciated.

Best,
--Benjamin Gilbert

[-- Attachment #2: Type: text/html, Size: 1243 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

On Thu, 13 May 2021 at 22:34, Benjamin Gilbert <bgilbert@backtick.net>
wrote:

> Tarballs, commits on published branches, and annotated tags are all
> supposed to be stable artifacts.  But I don't know of any general
> expectation that a specific branch continues to exist, other than the one
> created by Bitbake.  GitHub's branch rename tool doesn't leave the old
> branch behind, not even a symref, so I don't think this is a fringe
> viewpoint.  And as I said, automated builds usually only care about tags,
> commits, and the default branch.
>

Just wanted to add that there's a reason Yocto doesn't use tags alone: they
can and do move from one commit to another, and to the best of my knowledge
there only way to tie them to a commit is to specify the commit too.

Alex

[-- Attachment #2: Type: text/html, Size: 1127 bytes --]

Re: [OE-core] master/main branch renaming and bitbake

[colin walters]

On Thu, May 13, 2021, at 4:08 PM, Richard Purdie wrote:
>
> The advice to anyone hitting this issue is to add in the correct branch
> to the SRC_URI. It is simple and easy to do, can be in bbappends 
> or even changed via anonymous python and similar if necessary. We've already
> found the issue with several core recipes, we simply updated them and most 
> users didn't notice. I would even likely take that kind of change into older
> otherwise unmaintained branches and I think I did so in at least one case in
> the past.

What I am uncertain about is: how quickly does that translate into us being able to remove the old branch?

We're not the first project to do this and we won't be the last, so having a solution here will be good.

Hmm actually I notice systemd upstream did the rename a while ago and they're not carrying a `master` branch.  What's the difference between systemd and ostree here?

Is it the use of `SRCREV`?  Or no, in the systemd case is it because it's set to a tag?
https://github.com/openembedded/openembedded-core/blob/2621dbbc1181808f18ca4ae79408d0d5b557670f/meta/recipes-core/systemd/systemd.inc#L18

ostree is also using tags, is the recipe just broken in not using tags?

> I appreciate the tooling could do all kinds of magic things. I have a strong
> preference for not adding magic into it, or over complicating it, it is already
> horrendously complicated and a nightmare to test. I appreciate nobody believes
> me, I only do my best to maintain it. The code is here for anyone interested:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/git.py

Yeah, understood.

> I'd also note you can add ;nobranch=1 to the urls or ;usehead=1. Those do
> have side effects, I will not recommend them, or accept them for general use
> in layers I maintain, they're considered developer options. I was reminded
> recently that we have seen bugs the branch parameter has caught where a 
> revision was not where we thought it was so these do catch real world issues.

Well I hope the result of this discussion is a recommended best practice at least.
If recommending using a tag works, that seems good to me.

Re: [OE-core] master/main branch renaming and bitbake

[Richard Purdie]

On Thu, 2021-05-13 at 17:33 -0400, colin walters wrote:
> 
> On Thu, May 13, 2021, at 4:08 PM, Richard Purdie wrote:
> > 
> > The advice to anyone hitting this issue is to add in the correct branch
> > to the SRC_URI. It is simple and easy to do, can be in bbappends 
> > or even changed via anonymous python and similar if necessary. We've already
> > found the issue with several core recipes, we simply updated them and most 
> > users didn't notice. I would even likely take that kind of change into older
> > otherwise unmaintained branches and I think I did so in at least one case in
> > the past.
> 
> What I am uncertain about is: how quickly does that translate into us being 
> able to remove the old branch?

My personal opinion is see the patches updating the SRC_URIs make the branches
given we know about this one and then do it.

> We're not the first project to do this and we won't be the last, so having a 
> solution here will be good.

I'm suggesting people update the recipes.

> Hmm actually I notice systemd upstream did the rename a while ago and they're 
> not carrying a `master` branch.  What's the difference between systemd and ostree here?
> 
> Is it the use of `SRCREV`?  Or no, in the systemd case is it because it's set to a tag?
> https://github.com/openembedded/openembedded-core/blob/2621dbbc1181808f18ca4ae79408d0d5b557670f/meta/recipes-core/systemd/systemd.inc#L18
> ostree is also using tags, is the recipe just broken in not using tags?

No:

SRCBRANCH = "v247-stable"
SRC_URI = "git://github.com/systemd/systemd-stable.git;protocol=git;branch=${SRCBRANCH}

i.e. there is a specific branch specified.

> > I appreciate the tooling could do all kinds of magic things. I have a strong
> > preference for not adding magic into it, or over complicating it, it is already
> > horrendously complicated and a nightmare to test. I appreciate nobody believes
> > me, I only do my best to maintain it. The code is here for anyone interested:
> > 
> > http://git.yoctoproject.org/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/git.py
> 
> Yeah, understood.
> 
> > I'd also note you can add ;nobranch=1 to the urls or ;usehead=1. Those do
> > have side effects, I will not recommend them, or accept them for general use
> > in layers I maintain, they're considered developer options. I was reminded
> > recently that we have seen bugs the branch parameter has caught where a 
> > revision was not where we thought it was so these do catch real world issues.
> 
> Well I hope the result of this discussion is a recommended best practice at least.
> If recommending using a tag works, that seems good to me.

People should add/update the branch in SRC_URI.

Cheers,

Richard

Re: [OE-core] master/main branch renaming and bitbake

[Andre McCurdy]

On Thu, May 13, 2021 at 3:48 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2021-05-13 at 17:33 -0400, colin walters wrote:
> >
> > On Thu, May 13, 2021, at 4:08 PM, Richard Purdie wrote:
> > >
> > > The advice to anyone hitting this issue is to add in the correct branch
> > > to the SRC_URI. It is simple and easy to do, can be in bbappends
> > > or even changed via anonymous python and similar if necessary. We've already
> > > found the issue with several core recipes, we simply updated them and most
> > > users didn't notice. I would even likely take that kind of change into older
> > > otherwise unmaintained branches and I think I did so in at least one case in
> > > the past.
> >
> > What I am uncertain about is: how quickly does that translate into us being
> > able to remove the old branch?
>
> My personal opinion is see the patches updating the SRC_URIs make the branches
> given we know about this one and then do it.
>
> > We're not the first project to do this and we won't be the last, so having a
> > solution here will be good.
>
> I'm suggesting people update the recipes.
>
> > Hmm actually I notice systemd upstream did the rename a while ago and they're
> > not carrying a `master` branch.  What's the difference between systemd and ostree here?
> >
> > Is it the use of `SRCREV`?  Or no, in the systemd case is it because it's set to a tag?
> > https://github.com/openembedded/openembedded-core/blob/2621dbbc1181808f18ca4ae79408d0d5b557670f/meta/recipes-core/systemd/systemd.inc#L18
> > ostree is also using tags, is the recipe just broken in not using tags?
>
> No:
>
> SRCBRANCH = "v247-stable"
> SRC_URI = "git://github.com/systemd/systemd-stable.git;protocol=git;branch=${SRCBRANCH}
>
> i.e. there is a specific branch specified.
>
> > > I appreciate the tooling could do all kinds of magic things. I have a strong
> > > preference for not adding magic into it, or over complicating it, it is already
> > > horrendously complicated and a nightmare to test. I appreciate nobody believes
> > > me, I only do my best to maintain it. The code is here for anyone interested:
> > >
> > > http://git.yoctoproject.org/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/git.py
> >
> > Yeah, understood.
> >
> > > I'd also note you can add ;nobranch=1 to the urls or ;usehead=1. Those do
> > > have side effects, I will not recommend them, or accept them for general use
> > > in layers I maintain, they're considered developer options. I was reminded
> > > recently that we have seen bugs the branch parameter has caught where a
> > > revision was not where we thought it was so these do catch real world issues.
> >
> > Well I hope the result of this discussion is a recommended best practice at least.
> > If recommending using a tag works, that seems good to me.
>
> People should add/update the branch in SRC_URI.

Could we perhaps have an "official" recommendation on that somewhere
which users etc could be pointed to?

I've just had a discussion along the lines of "but ;nobranch=1 works
and will be more robust than setting ;branch=main if upstream changes
their mind again". From a user's point of view it's not clear at all
that setting branch is the better solution or why.