git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Matthias Schoepfer]

Hi!

We have noticed the following issue: We keep the versions of out 
software by means of tags on the git repositories, i.e. during the 
build, somethings like git describe --tags gets called. In yocto, our 
recipe of SomeLibrary might look then similar to this:

SRC_URI = "git://our.private.gitserver.org/git/SomeLibrary.git;tags=v${PV}"

When we build the image, and *then* apply a tag to HEAD, that was 
already built before, it seems like the tags are not fetched (my guess 
is git fetcher sees that origins HEAD and local HEAD have the same hash 
and is fine with it).

The error is, that while the correct version is built, it will report 
itself with a wrong version.

Is this a bug?

Regards,

   Matthias

-- 
Dr.-Ing. Matthias Schöpfer
matthias.schoepfer@googlemail.com

Re: [yocto] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Denys Dmytriyenko]

Why not use SRCREV?
Also, bitbake has its own mailing list.

Denys


On Thu, Mar 12, 2020 at 02:55:59PM +0100, Matthias Schoepfer via Lists.Yoctoproject.Org wrote:
> Hi!
> 
> We have noticed the following issue: We keep the versions of out
> software by means of tags on the git repositories, i.e. during the
> build, somethings like git describe --tags gets called. In yocto,
> our recipe of SomeLibrary might look then similar to this:
> 
> SRC_URI = "git://our.private.gitserver.org/git/SomeLibrary.git;tags=v${PV}"
> 
> When we build the image, and *then* apply a tag to HEAD, that was
> already built before, it seems like the tags are not fetched (my
> guess is git fetcher sees that origins HEAD and local HEAD have the
> same hash and is fine with it).
> 
> The error is, that while the correct version is built, it will
> report itself with a wrong version.
> 
> Is this a bug?
> 
> Regards,
> 
>   Matthias
> 
> -- 
> Dr.-Ing. Matthias Schöpfer
> matthias.schoepfer@googlemail.com

> 

Re: [yocto] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Matthias Schoepfer]

Hi Denys,

On 3/13/20 5:29 PM, Denys Dmytriyenko wrote:
> Why not use SRCREV?
> Also, bitbake has its own mailing list.

1) Would SRCREV not be affected by this as well? When the same hash is 
read, then no update of the repository (esp. the tags) is done. Or can 
you be sure this is not the case?

2) It allows for very convenient bump of versions. We develop a product, 
so, version bumps are on daily or weekly basis. All that is needed is to 
rename the .bb file. No need to edit SRCREV.

3) I am sorry if I used the wrong mailing list for this. I will repost 
on the other list as well.

Regards,

   Matthias

Re: [yocto] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 581 bytes --]

On Sun, 15 Mar 2020 at 15:04, Matthias Schoepfer via Lists.Yoctoproject.Org
<matthias.schoepfer=googlemail.com@lists.yoctoproject.org> wrote:

> 2) It allows for very convenient bump of versions. We develop a product,
> so, version bumps are on daily or weekly basis. All that is needed is to
> rename the .bb file. No need to edit SRCREV.
>

You can trivially script SRCREV updates ('devtool upgrade'/'devtool
finish'). The disadvantage of using tags is that you lose reproducibility,
as tags can be moved or deleted, and so I wouldn't encourage this practice.

Alex

[-- Attachment #2: Type: text/html, Size: 967 bytes --]

Re: [yocto] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Matthias Schoepfer]

Hi Alexander,

does it solve the Problem of a SRCREV, that has been fetched (for 
example for a _git testbuild) and now gets a tag applied and build with 
the _<version> build. The problem is *not* that the yocto PV package is 
wrong *nor* that the wrong version of the software gets build! The 
problem is, that the tag does not get fetched and our software gets its 
version from the git tags. Since this is very convenient to have the 
version information in one place only, we will not change this.

Thanks and Regards,

   Matthias

On 3/15/20 5:02 PM, Alexander Kanavin wrote:
> On Sun, 15 Mar 2020 at 15:04, Matthias Schoepfer via 
> Lists.Yoctoproject.Org <http://Lists.Yoctoproject.Org> 
> <matthias.schoepfer=googlemail.com@lists.yoctoproject.org 
> <mailto:googlemail.com@lists.yoctoproject.org>> wrote:
> 
>     2) It allows for very convenient bump of versions. We develop a
>     product,
>     so, version bumps are on daily or weekly basis. All that is needed
>     is to
>     rename the .bb file. No need to edit SRCREV.
> 
> 
> You can trivially script SRCREV updates ('devtool upgrade'/'devtool 
> finish'). The disadvantage of using tags is that you lose 
> reproducibility, as tags can be moved or deleted, and so I wouldn't 
> encourage this practice.
> 
> Alex


-- 

Dr.-Ing. Matthias Schoepfer
Mobile: +49 175 2062739
email: matthias.schoepfer@googlemail.com

Re: [yocto] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1804 bytes --]

'devtool upgrade' updates SRCREV to match the version tag that you specify:

devtool upgrade -V <tag>

If you omit -V, it will upgrade to the latest tag. It updates PV as well.
Why is it unsuitable for you?

Alex

On Mon, 16 Mar 2020 at 09:35, Matthias Schöpfer <
matthias.schoepfer@googlemail.com> wrote:

> Hi Alexander,
>
> does it solve the Problem of a SRCREV, that has been fetched (for
> example for a _git testbuild) and now gets a tag applied and build with
> the _<version> build. The problem is *not* that the yocto PV package is
> wrong *nor* that the wrong version of the software gets build! The
> problem is, that the tag does not get fetched and our software gets its
> version from the git tags. Since this is very convenient to have the
> version information in one place only, we will not change this.
>
> Thanks and Regards,
>
>    Matthias
>
> On 3/15/20 5:02 PM, Alexander Kanavin wrote:
> > On Sun, 15 Mar 2020 at 15:04, Matthias Schoepfer via
> > Lists.Yoctoproject.Org <http://Lists.Yoctoproject.Org>
> > <matthias.schoepfer=googlemail.com@lists.yoctoproject.org
> > <mailto:googlemail.com@lists.yoctoproject.org>> wrote:
> >
> >     2) It allows for very convenient bump of versions. We develop a
> >     product,
> >     so, version bumps are on daily or weekly basis. All that is needed
> >     is to
> >     rename the .bb file. No need to edit SRCREV.
> >
> >
> > You can trivially script SRCREV updates ('devtool upgrade'/'devtool
> > finish'). The disadvantage of using tags is that you lose
> > reproducibility, as tags can be moved or deleted, and so I wouldn't
> > encourage this practice.
> >
> > Alex
>
>
> --
>
> Dr.-Ing. Matthias Schoepfer
> Mobile: +49 175 2062739
> email: matthias.schoepfer@googlemail.com
>

[-- Attachment #2: Type: text/html, Size: 2780 bytes --]

Re: [yocto] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Matthias Schoepfer]

Hi Alexander,

thanks for your message. I will try to explain my issue once again, 
since it still seems not clearly formulated. We build our images on CI 
(Jenkins). There, we have a shared download dir, so that we do not have 
to fetch all sources on every build. This gets shared for our 
"experimental" builds, that use latest git master HEAD (yes, evil 
AUTOREV) and our "stable" builds, where only tagged (i.e. no AUTOREV 
versions) are used. Our software uses the git tag to create a version. 
So, when on experimental, you might get something like:

# ourCoolSoftware --version
This is ourCoolSoftware version 1.2.3-3 (gabcdef1)

What happens is, that people check their changes on the experimental 
build. After testing, it is considered for the next release, so they 
attach a version tag to the git repository and update the _<version>.bb 
accordingly. Now they build the image with the "stable" profile. But the 
hash "abcdef1" has not changed. Hence, although we pushed a new tag, 
yocto git fetcher does not update the cached download git, since git 
hashes match (same with SRCREV). Which means, even tough we have a 
recipe ourcoolsoftware_1.2.4.bb with SRCREV="abcdef1..." which will 
build the correct hash, our software checks the local repository, which 
is missing the last tag and will still be reporting a wrong version 
1.2.3-3 for example.

If we were to do a
bitbake -c cleanall ourcoolsoftware && bitbake ourcoolimage

then we get
# ourCoolSoftware --version
This is ourCoolSoftware version 1.2.4 (gabcdef1)

because we artificially force a complete fetch of the repository.

We usually do a clean build (no sstate / tmp / etc.) but we use the 
download dir, because we do not see why we would need to download gcc 
etc. every time. A workaround would be to bitbake -c cleanall our 
packages, but it seems hacky and prone to error (need to keep lists of 
our used software in at least two places). My question is therefore, if 
this is a "bug" and if we were to investigate this and propose a patch, 
if this could this be considered for merge, or if this behaviour is 
intentional and adding a fetch --tags would be considered harmful in any 
way.

Hope, I could bring my point across now.

Thanks and Regards,

  Matthias

On 3/16/20 9:43 AM, Alexander Kanavin wrote:
> 'devtool upgrade' updates SRCREV to match the version tag that you specify:
> 
> devtool upgrade -V <tag>
> 
> If you omit -V, it will upgrade to the latest tag. It updates PV as 
> well. Why is it unsuitable for you?
> 
> Alex
> 
> On Mon, 16 Mar 2020 at 09:35, Matthias Schöpfer 
> <matthias.schoepfer@googlemail.com 
> <mailto:matthias.schoepfer@googlemail.com>> wrote:
> 
>     Hi Alexander,
> 
>     does it solve the Problem of a SRCREV, that has been fetched (for
>     example for a _git testbuild) and now gets a tag applied and build with
>     the _<version> build. The problem is *not* that the yocto PV package is
>     wrong *nor* that the wrong version of the software gets build! The
>     problem is, that the tag does not get fetched and our software gets its
>     version from the git tags. Since this is very convenient to have the
>     version information in one place only, we will not change this.
> 
>     Thanks and Regards,
> 
>         Matthias
> 
>     On 3/15/20 5:02 PM, Alexander Kanavin wrote:
>      > On Sun, 15 Mar 2020 at 15:04, Matthias Schoepfer via
>      > Lists.Yoctoproject.Org <http://Lists.Yoctoproject.Org>
>     <http://Lists.Yoctoproject.Org>
>      > <matthias.schoepfer=googlemail.com@lists.yoctoproject.org
>     <mailto:googlemail.com@lists.yoctoproject.org>
>      > <mailto:googlemail.com@lists.yoctoproject.org
>     <mailto:googlemail.com@lists.yoctoproject.org>>> wrote:
>      >
>      >     2) It allows for very convenient bump of versions. We develop a
>      >     product,
>      >     so, version bumps are on daily or weekly basis. All that is
>     needed
>      >     is to
>      >     rename the .bb file. No need to edit SRCREV.
>      >
>      >
>      > You can trivially script SRCREV updates ('devtool upgrade'/'devtool
>      > finish'). The disadvantage of using tags is that you lose
>      > reproducibility, as tags can be moved or deleted, and so I wouldn't
>      > encourage this practice.
>      >
>      > Alex
> 
> 
>     -- 
> 
>     Dr.-Ing. Matthias Schoepfer
>     Mobile: +49 175 2062739
>     email: matthias.schoepfer@googlemail.com
>     <mailto:matthias.schoepfer@googlemail.com>
> 


-- 

Dr.-Ing. Matthias Schoepfer
Mobile: +49 175 2062739
email: matthias.schoepfer@googlemail.com

Re: [yocto] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 5468 bytes --]

Right, now I see what you mean. The build process does not rely on static
metadata in the source tree to determine the version, and is instead poking
the git checkout to determine what the version is? That's not established
practice. You should write the version into a file in the source tree, then
create a commit, then tag that commit with the version.

Alternatively, you could pass in ${PV} to the build process instead as a
command line parameter.

Alex

On Mon, 16 Mar 2020 at 12:52, Matthias Schöpfer <
matthias.schoepfer@googlemail.com> wrote:

> Hi Alexander,
>
> thanks for your message. I will try to explain my issue once again,
> since it still seems not clearly formulated. We build our images on CI
> (Jenkins). There, we have a shared download dir, so that we do not have
> to fetch all sources on every build. This gets shared for our
> "experimental" builds, that use latest git master HEAD (yes, evil
> AUTOREV) and our "stable" builds, where only tagged (i.e. no AUTOREV
> versions) are used. Our software uses the git tag to create a version.
> So, when on experimental, you might get something like:
>
> # ourCoolSoftware --version
> This is ourCoolSoftware version 1.2.3-3 (gabcdef1)
>
> What happens is, that people check their changes on the experimental
> build. After testing, it is considered for the next release, so they
> attach a version tag to the git repository and update the _<version>.bb
> accordingly. Now they build the image with the "stable" profile. But the
> hash "abcdef1" has not changed. Hence, although we pushed a new tag,
> yocto git fetcher does not update the cached download git, since git
> hashes match (same with SRCREV). Which means, even tough we have a
> recipe ourcoolsoftware_1.2.4.bb with SRCREV="abcdef1..." which will
> build the correct hash, our software checks the local repository, which
> is missing the last tag and will still be reporting a wrong version
> 1.2.3-3 for example.
>
> If we were to do a
> bitbake -c cleanall ourcoolsoftware && bitbake ourcoolimage
>
> then we get
> # ourCoolSoftware --version
> This is ourCoolSoftware version 1.2.4 (gabcdef1)
>
> because we artificially force a complete fetch of the repository.
>
> We usually do a clean build (no sstate / tmp / etc.) but we use the
> download dir, because we do not see why we would need to download gcc
> etc. every time. A workaround would be to bitbake -c cleanall our
> packages, but it seems hacky and prone to error (need to keep lists of
> our used software in at least two places). My question is therefore, if
> this is a "bug" and if we were to investigate this and propose a patch,
> if this could this be considered for merge, or if this behaviour is
> intentional and adding a fetch --tags would be considered harmful in any
> way.
>
> Hope, I could bring my point across now.
>
> Thanks and Regards,
>
>   Matthias
>
> On 3/16/20 9:43 AM, Alexander Kanavin wrote:
> > 'devtool upgrade' updates SRCREV to match the version tag that you
> specify:
> >
> > devtool upgrade -V <tag>
> >
> > If you omit -V, it will upgrade to the latest tag. It updates PV as
> > well. Why is it unsuitable for you?
> >
> > Alex
> >
> > On Mon, 16 Mar 2020 at 09:35, Matthias Schöpfer
> > <matthias.schoepfer@googlemail.com
> > <mailto:matthias.schoepfer@googlemail.com>> wrote:
> >
> >     Hi Alexander,
> >
> >     does it solve the Problem of a SRCREV, that has been fetched (for
> >     example for a _git testbuild) and now gets a tag applied and build
> with
> >     the _<version> build. The problem is *not* that the yocto PV package
> is
> >     wrong *nor* that the wrong version of the software gets build! The
> >     problem is, that the tag does not get fetched and our software gets
> its
> >     version from the git tags. Since this is very convenient to have the
> >     version information in one place only, we will not change this.
> >
> >     Thanks and Regards,
> >
> >         Matthias
> >
> >     On 3/15/20 5:02 PM, Alexander Kanavin wrote:
> >      > On Sun, 15 Mar 2020 at 15:04, Matthias Schoepfer via
> >      > Lists.Yoctoproject.Org <http://Lists.Yoctoproject.Org>
> >     <http://Lists.Yoctoproject.Org>
> >      > <matthias.schoepfer=googlemail.com@lists.yoctoproject.org
> >     <mailto:googlemail.com@lists.yoctoproject.org>
> >      > <mailto:googlemail.com@lists.yoctoproject.org
> >     <mailto:googlemail.com@lists.yoctoproject.org>>> wrote:
> >      >
> >      >     2) It allows for very convenient bump of versions. We develop
> a
> >      >     product,
> >      >     so, version bumps are on daily or weekly basis. All that is
> >     needed
> >      >     is to
> >      >     rename the .bb file. No need to edit SRCREV.
> >      >
> >      >
> >      > You can trivially script SRCREV updates ('devtool
> upgrade'/'devtool
> >      > finish'). The disadvantage of using tags is that you lose
> >      > reproducibility, as tags can be moved or deleted, and so I
> wouldn't
> >      > encourage this practice.
> >      >
> >      > Alex
> >
> >
> >     --
> >
> >     Dr.-Ing. Matthias Schoepfer
> >     Mobile: +49 175 2062739
> >     email: matthias.schoepfer@googlemail.com
> >     <mailto:matthias.schoepfer@googlemail.com>
> >
>
>
> --
>
> Dr.-Ing. Matthias Schoepfer
> Mobile: +49 175 2062739
> email: matthias.schoepfer@googlemail.com
>

[-- Attachment #2: Type: text/html, Size: 7582 bytes --]

Re: [yocto] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed

[Richard Purdie]

On Thu, 2020-03-12 at 14:55 +0100, Matthias Schoepfer via Lists.Yoctoproject.Org wrote:
> We have noticed the following issue: We keep the versions of out 
> software by means of tags on the git repositories, i.e. during the 
> build, somethings like git describe --tags gets called. In yocto,
> our 
> recipe of SomeLibrary might look then similar to this:
> 
> SRC_URI =
> "git://our.private.gitserver.org/git/SomeLibrary.git;tags=v${PV}"
> 
> When we build the image, and *then* apply a tag to HEAD, that was 
> already built before, it seems like the tags are not fetched (my
> guess 
> is git fetcher sees that origins HEAD and local HEAD have the same
> hash 
> and is fine with it).
> 
> The error is, that while the correct version is built, it will
> report 
> itself with a wrong version.
> 
> Is this a bug?

Each of the fetcher backends makes some assumptions about what it
considers to be "the same".

For tarballs from whatever source this is easy and we use the hash of
the tarball.

For git, we work off the hash from git itself. If the hash changes, the
source is taken to be different. Hashes of the same data are taken to
be the same.

What we don't do is account for any tag that represents that hash. If
you have some extra part of the build process which is using that as a
factor for the build you would also have to account for it in the task
hashes. Nothing does that by default.

git tags are a pain as they require network access to evaluate and they
break the mirroring structure as you can't know a given tag was fetched
into the mirror.

So no, I'd not define it as a bug, the bug is you're relying on the tag
data but not adding it to the task hashes.

You'd somehow need to inject the "git tag for revision X" output into
the do_fetch task dependencies.

For autorev this happens via:

meta/classes/base.bbclass:        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}"

so you'd need a new version of get_srcrev() and need to inject that
into the vardeps of the fetch task.

Cheers,

Richard