[PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Tobias Hagelborn]

Change made to enable building offline also with git tag references
in SRC_URI.

If BB_NO_NETWORK is set, fallback to search for tags and revisions
in local DL_DIR / GITDIR in order to avoid network access.

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
---
 bitbake/lib/bb/fetch2/git.py | 36 ++++++++++++++++++++++++------------
 1 file changed, 24 insertions(+), 12 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 9bd87ad..0c08a3c 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -131,12 +131,6 @@ class Git(FetchMethod):
 
         ud.setup_revisons(d)
 
-        for name in ud.names:
-            # Ensure anything that doesn't look like a sha256 checksum/revision is translated into one
-            if not ud.revisions[name] or len(ud.revisions[name]) != 40  or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
-                if ud.revisions[name]:
-                    ud.unresolvedrev[name] = ud.revisions[name]
-                ud.revisions[name] = self.latest_revision(ud, d, name)
 
         gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.'))
         if gitsrcname.startswith('.'):
@@ -156,6 +150,13 @@ class Git(FetchMethod):
 
         ud.localfile = ud.clonedir
 
+        for name in ud.names:
+            # Ensure anything that doesn't look like a sha256 checksum/revision is translated into one
+            if not ud.revisions[name] or len(ud.revisions[name]) != 40  or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
+                if ud.revisions[name]:
+                    ud.unresolvedrev[name] = ud.revisions[name]
+                ud.revisions[name] = self.latest_revision(ud, d, name)
+
     def localpath(self, ud, d):
         return ud.clonedir
 
@@ -345,12 +346,23 @@ class Git(FetchMethod):
         repourl = self._get_repo_url(ud)
         cmd = "%s ls-remote %s %s" % \
               (ud.basecmd, repourl, search)
-        if ud.proto.lower() != 'file':
-            bb.fetch2.check_network_access(d, cmd)
-        output = runfetchcmd(cmd, d, True)
-        if not output:
-            raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
-        return output
+        try:
+            if ud.proto.lower() != 'file':
+                bb.fetch2.check_network_access(d, cmd)
+            output = runfetchcmd(cmd, d, True)
+            if not output:
+                raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
+            return output
+        except bb.fetch2.NetworkAccess as exc:
+            # Fallback to local DL_DIR/GITDIR if BB_NO_NETWORK is set
+            cmd = "%s ls-remote %s://%s %s" % \
+                  (ud.basecmd, 'file', self.localpath(ud, d), search)
+            output = runfetchcmd(cmd, d, True)
+            if output.split() == []:
+                bb.warn('failed: ', cmd)
+            if not output:
+                raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
+            return output
 
     def _latest_revision(self, ud, d, name):
         """
-- 
2.1.4

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Richard Purdie]

On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
> Change made to enable building offline also with git tag references
> in SRC_URI.
> 
> If BB_NO_NETWORK is set, fallback to search for tags and revisions
> in local DL_DIR / GITDIR in order to avoid network access.

I'm not sure I agree with this since you'd get different build results
depending on whether BB_NO_NETWORK is set or not.

I think if BB_NO_NETWORK is set, its reasonable to assume that the
configuration should be locked down and that BB_NO_NETWORK should not
have other side effects like this.

It would also make it very hard to detect unlocked configurations
during testing which currently are quite easily identified.

Cheers,

Richard

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Jérémy Rosen]

On 08/08/2016 11:55, Richard Purdie wrote:
> On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
>> Change made to enable building offline also with git tag references
>> in SRC_URI.
>>
>> If BB_NO_NETWORK is set, fallback to search for tags and revisions
>> in local DL_DIR / GITDIR in order to avoid network access.
> I'm not sure I agree with this since you'd get different build results
> depending on whether BB_NO_NETWORK is set or not.
>
> I think if BB_NO_NETWORK is set, its reasonable to assume that the
> configuration should be locked down and that BB_NO_NETWORK should not
> have other side effects like this.
>
> It would also make it very hard to detect unlocked configurations
> during testing which currently are quite easily identified.

To be honest, this is a real world problem I have on a regular basis...

The problem we have is that our customers want to keep an archive for 
(very) long term support and they test offline mode.

This problem means that we have to set all our SRC_URI to git SHA 
instead of the much more readable and easy to check git tags.


you think to assume that a reference is a branch, but the real-life 
problem is when reference is a tag. Tag don't move so this is
a case of locked configuration, but i'm not sure how to handle it.

Is there a way to have the best of both world somehow ?

Regards
Jérémy Rosen

> Cheers,
>
> Richard

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Tobias Hagelborn]

Hi Richard,

I think I understand your concern. This patch takes whatever is available locally and that means it is a side-effect compared to not setting BB_NO_NETWORK. Something the user might not expect.

I don't know if you would accept the use of a new  flag that the user can set to agree with this side effect.

Something along the lines of "FORCE_LOCAL_TAGS". 
Then the user would have given some consent on that side-affect.
We think it is still very useful to us to be able to build locally with whatever you have fetched in cases where no NW is available and you basically know that few/no tags have moved.


Regards
Tobias


________________________________________
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Sent: Monday, August 8, 2016 11:55 AM
To: Tobias Hagelborn; bitbake-devel@lists.openembedded.org
Subject: Re: [bitbake-devel] [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
> Change made to enable building offline also with git tag references
> in SRC_URI.
>
> If BB_NO_NETWORK is set, fallback to search for tags and revisions
> in local DL_DIR / GITDIR in order to avoid network access.

I'm not sure I agree with this since you'd get different build results
depending on whether BB_NO_NETWORK is set or not.

I think if BB_NO_NETWORK is set, its reasonable to assume that the
configuration should be locked down and that BB_NO_NETWORK should not
have other side effects like this.

It would also make it very hard to detect unlocked configurations
during testing which currently are quite easily identified.

Cheers,

Richard

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]


> On Aug 8, 2016, at 4:24 AM, Jérémy Rosen <jeremy.rosen@smile.fr> wrote:
> 
> 
> On 08/08/2016 11:55, Richard Purdie wrote:
>> On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
>>> Change made to enable building offline also with git tag references
>>> in SRC_URI.
>>> 
>>> If BB_NO_NETWORK is set, fallback to search for tags and revisions
>>> in local DL_DIR / GITDIR in order to avoid network access.
>> I'm not sure I agree with this since you'd get different build results
>> depending on whether BB_NO_NETWORK is set or not.
>> 
>> I think if BB_NO_NETWORK is set, its reasonable to assume that the
>> configuration should be locked down and that BB_NO_NETWORK should not
>> have other side effects like this.
>> 
>> It would also make it very hard to detect unlocked configurations
>> during testing which currently are quite easily identified.
> 
> To be honest, this is a real world problem I have on a regular basis...
> 
> The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
> 
> This problem means that we have to set all our SRC_URI to git SHA instead of the much more readable and easy to check git tags.
> 
> 
> you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is
> a case of locked configuration, but i'm not sure how to handle it.
> 
> Is there a way to have the best of both world somehow ?

Don’t use tags instead rely on SHA numbers in SRCREV and use
BB_GENERATE_MIRROR_TARBALLS like mentioned here

https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F

tags are floating types in git, you can not use them reliably.
You may tool in a method to automate updating the
recipes SRCREVs if you dont want to do it by hand.

That should help hopefully

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Jérémy Rosen]

On 08/08/2016 17:35, Khem Raj wrote:
>> On Aug 8, 2016, at 4:24 AM, Jérémy Rosen <jeremy.rosen@smile.fr> wrote:
>>
>>
>> On 08/08/2016 11:55, Richard Purdie wrote:
>>> On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
>>>> Change made to enable building offline also with git tag references
>>>> in SRC_URI.
>>>>
>>>> If BB_NO_NETWORK is set, fallback to search for tags and revisions
>>>> in local DL_DIR / GITDIR in order to avoid network access.
>>> I'm not sure I agree with this since you'd get different build results
>>> depending on whether BB_NO_NETWORK is set or not.
>>>
>>> I think if BB_NO_NETWORK is set, its reasonable to assume that the
>>> configuration should be locked down and that BB_NO_NETWORK should not
>>> have other side effects like this.
>>>
>>> It would also make it very hard to detect unlocked configurations
>>> during testing which currently are quite easily identified.
>> To be honest, this is a real world problem I have on a regular basis...
>>
>> The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
>>
>> This problem means that we have to set all our SRC_URI to git SHA instead of the much more readable and easy to check git tags.
>>
>>
>> you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is
>> a case of locked configuration, but i'm not sure how to handle it.
>>
>> Is there a way to have the best of both world somehow ?
> Don’t use tags instead rely on SHA numbers in SRCREV and use
> BB_GENERATE_MIRROR_TARBALLS like mentioned here
>
> https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
>
> tags are floating types in git, you can not use them reliably.
> You may tool in a method to automate updating the
> recipes SRCREVs if you dont want to do it by hand.
>
> That should help hopefully
I know... but the fact that NO_NETWORK can't work if any git uses a tag 
seems... not what I would expect. Especially if the tag is available 
locally. I understand what you say, but I think it breaks the purpose of 
NO_NETWORK

If a git reference is floating, then building twice won't produce the 
same result anyway... this sounds more like a QA problem to me than 
something that bitbake should enforce. But i'm really not a BB 
architect, it's just that this patch would solve one of our problem and 
i'd like to see if there is a way to do that without loosing other 
aspects of bitbake...

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 3621 bytes --]


> On Aug 8, 2016, at 8:58 AM, Jérémy Rosen <jeremy.rosen@smile.fr> wrote:
> 
> 
> 
> On 08/08/2016 17:35, Khem Raj wrote:
>>> On Aug 8, 2016, at 4:24 AM, Jérémy Rosen <jeremy.rosen@smile.fr> wrote:
>>> 
>>> 
>>> On 08/08/2016 11:55, Richard Purdie wrote:
>>>> On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
>>>>> Change made to enable building offline also with git tag references
>>>>> in SRC_URI.
>>>>> 
>>>>> If BB_NO_NETWORK is set, fallback to search for tags and revisions
>>>>> in local DL_DIR / GITDIR in order to avoid network access.
>>>> I'm not sure I agree with this since you'd get different build results
>>>> depending on whether BB_NO_NETWORK is set or not.
>>>> 
>>>> I think if BB_NO_NETWORK is set, its reasonable to assume that the
>>>> configuration should be locked down and that BB_NO_NETWORK should not
>>>> have other side effects like this.
>>>> 
>>>> It would also make it very hard to detect unlocked configurations
>>>> during testing which currently are quite easily identified.
>>> To be honest, this is a real world problem I have on a regular basis...
>>> 
>>> The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
>>> 
>>> This problem means that we have to set all our SRC_URI to git SHA instead of the much more readable and easy to check git tags.
>>> 
>>> 
>>> you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is
>>> a case of locked configuration, but i'm not sure how to handle it.
>>> 
>>> Is there a way to have the best of both world somehow ?
>> Don’t use tags instead rely on SHA numbers in SRCREV and use
>> BB_GENERATE_MIRROR_TARBALLS like mentioned here
>> 
>> https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
>> 
>> tags are floating types in git, you can not use them reliably.
>> You may tool in a method to automate updating the
>> recipes SRCREVs if you dont want to do it by hand.
>> 
>> That should help hopefully
> I know... but the fact that NO_NETWORK can't work if any git uses a tag seems... not what I would expect. Especially if the tag is available locally. I understand what you say, but I think it breaks the purpose of NO_NETWORK

There is an assumption here you are making, which is that tags are fixed in git which is not true. However when you use
tags in your recipe you want folks to use that tag and it could be moved/displaced under the hood and user would not notice
some tags like “last_good” “last_sane_build” are moving tags the are practiced. Tags are in same category as AUTOREV

BB_NO_NETWORK means you need to ensure that metadata can operate on disconnected snapshot which is not true for tags.

> 
> If a git reference is floating, then building twice won't produce the same result anyway... this sounds more like a QA problem to me than something that bitbake should enforce. But i'm really not a BB architect, it's just that this patch would solve one of our problem and i'd like to see if there is a way to do that without loosing other aspects of bitbake…
> 

We should be warning about this case as it does today and to make builds more predictable, this patch takes us in reverse direction.
otherwise soon folks have no idea what they are building every developer has his own snapshot think of supporting hundreds of developers
where you have no idea what combination they built to get into trouble that they are seeing. It turns into a nightmare very
quickly.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Olof Johansson]

On 16-08-08 08:35 -0700, Khem Raj wrote:
> tags are floating types in git, you can not use them reliably.

This is not always true. We could choose to trust tags set
according to our processes to not move (e.g. enforced rules that
disallows deletion/updates of tags), and only use tags for these
recipes, and rely on commit object ids eleswhere.

Tags are only a problem in git if people insist on
modifying/deleting them.

-- 
olofjn

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 867 bytes --]


> On Aug 8, 2016, at 9:39 AM, Olof Johansson <olof.johansson@axis.com> wrote:
> 
> On 16-08-08 08:35 -0700, Khem Raj wrote:
>> tags are floating types in git, you can not use them reliably.
> 
> This is not always true.

you are confusing how a tool is specifying its functions with process
that could emulate a different behavior. OE does not have any control
over the internal processes of its users and it should not have.

> We could choose to trust tags set
> according to our processes to not move (e.g. enforced rules that
> disallows deletion/updates of tags), and only use tags for these
> recipes, and rely on commit object ids eleswhere.
> 
> Tags are only a problem in git if people insist on
> modifying/deleting them.

Fact is, its allowed rewrite tags in git, how people use it or not
is irrelevant here.

> 
> --
> olofjn


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Paul Eggleton]

On Mon, 08 Aug 2016 08:35:38 Khem Raj wrote:
> > On Aug 8, 2016, at 4:24 AM, Jérémy Rosen <jeremy.rosen@smile.fr> wrote:
> > 
> > On 08/08/2016 11:55, Richard Purdie wrote:
> >> On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
> >>> Change made to enable building offline also with git tag references
> >>> in SRC_URI.
> >>> 
> >>> If BB_NO_NETWORK is set, fallback to search for tags and revisions
> >>> in local DL_DIR / GITDIR in order to avoid network access.
> >> 
> >> I'm not sure I agree with this since you'd get different build results
> >> depending on whether BB_NO_NETWORK is set or not.
> >> 
> >> I think if BB_NO_NETWORK is set, its reasonable to assume that the
> >> configuration should be locked down and that BB_NO_NETWORK should not
> >> have other side effects like this.
> >> 
> >> It would also make it very hard to detect unlocked configurations
> >> during testing which currently are quite easily identified.
> > 
> > To be honest, this is a real world problem I have on a regular basis...
> > 
> > The problem we have is that our customers want to keep an archive for
> > (very) long term support and they test offline mode.
> > 
> > This problem means that we have to set all our SRC_URI to git SHA instead
> > of the much more readable and easy to check git tags.
> > 
> > 
> > you think to assume that a reference is a branch, but the real-life
> > problem is when reference is a tag. Tag don't move so this is a case of
> > locked configuration, but i'm not sure how to handle it.
> > 
> > Is there a way to have the best of both world somehow ?
> 
> Don’t use tags instead rely on SHA numbers in SRCREV and use
> BB_GENERATE_MIRROR_TARBALLS like mentioned here
> 
> https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source
> _download_mirror_.3F
> 
> tags are floating types in git, you can not use them reliably.
> You may tool in a method to automate updating the
> recipes SRCREVs if you dont want to do it by hand.
> 
> That should help hopefully

FYI, another way to handle this is to enable buildhistory, run your build and 
then use the buildhistory-collect-srcrevs script. This will produce output 
that you can save to .inc file that will lock down the SRCREV values to exact 
hashes, which you can then include and commit with your sources. This allows 
you to keep your tags and even ${AUTOREV} if you want within each of your 
recipes and still have everything locked down for future reproducibility.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Jérémy Rosen]

I am not very good at internal git magic, but how possible would it be 
to use the output of git-describe --long in SRC_URI...

This output contains both the tag name and the (abreviated) SHA, so it's 
both readable for the human user, easy to check and can't change...


On 09/08/2016 05:22, Paul Eggleton wrote:
> On Mon, 08 Aug 2016 08:35:38 Khem Raj wrote:
>>> On Aug 8, 2016, at 4:24 AM, Jérémy Rosen <jeremy.rosen@smile.fr> wrote:
>>>
>>> On 08/08/2016 11:55, Richard Purdie wrote:
>>>> On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
>>>>> Change made to enable building offline also with git tag references
>>>>> in SRC_URI.
>>>>>
>>>>> If BB_NO_NETWORK is set, fallback to search for tags and revisions
>>>>> in local DL_DIR / GITDIR in order to avoid network access.
>>>> I'm not sure I agree with this since you'd get different build results
>>>> depending on whether BB_NO_NETWORK is set or not.
>>>>
>>>> I think if BB_NO_NETWORK is set, its reasonable to assume that the
>>>> configuration should be locked down and that BB_NO_NETWORK should not
>>>> have other side effects like this.
>>>>
>>>> It would also make it very hard to detect unlocked configurations
>>>> during testing which currently are quite easily identified.
>>> To be honest, this is a real world problem I have on a regular basis...
>>>
>>> The problem we have is that our customers want to keep an archive for
>>> (very) long term support and they test offline mode.
>>>
>>> This problem means that we have to set all our SRC_URI to git SHA instead
>>> of the much more readable and easy to check git tags.
>>>
>>>
>>> you think to assume that a reference is a branch, but the real-life
>>> problem is when reference is a tag. Tag don't move so this is a case of
>>> locked configuration, but i'm not sure how to handle it.
>>>
>>> Is there a way to have the best of both world somehow ?
>> Don’t use tags instead rely on SHA numbers in SRCREV and use
>> BB_GENERATE_MIRROR_TARBALLS like mentioned here
>>
>> https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source
>> _download_mirror_.3F
>>
>> tags are floating types in git, you can not use them reliably.
>> You may tool in a method to automate updating the
>> recipes SRCREVs if you dont want to do it by hand.
>>
>> That should help hopefully
> FYI, another way to handle this is to enable buildhistory, run your build and
> then use the buildhistory-collect-srcrevs script. This will produce output
> that you can save to .inc file that will lock down the SRCREV values to exact
> hashes, which you can then include and commit with your sources. This allows
> you to keep your tags and even ${AUTOREV} if you want within each of your
> recipes and still have everything locked down for future reproducibility.
>
> Cheers,
> Paul
>

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Olof Johansson]

On 16-08-08 10:22 -0700, Khem Raj wrote:
> you are confusing how a tool is specifying its functions with
> process that could emulate a different behavior. OE does not
> have any control over the internal processes of its users and
> it should not have.

No, I don't think I'm confusing anything. Are you confusing OE
and bitbake? We maintain and develop an in-house Poky based
distribution, and unlike OE, we know a lot about our
users/developers.

> > Tags are only a problem in git if people insist on
> > modifying/deleting them.
> 
> Fact is, its allowed rewrite tags in git, how people use it or not
> is irrelevant here.

Nope, Git is a protocol; git the tool can't do nothing about a
server refusing to (re)move a tag. Your view of what a git tags
is yours. Tags are meant to be static, otherwise they wouldn't
have been synced to all clones in such a silly namespace.

While I agreed with Tobias' patch ("if you use tags in your
builds, and use BB_NO_NETWORK, you are ok with falling back to
what you have locally"), RP had a point about this making
BB_NO_NETWORK have non-obvious side-effects, and also making it
harder to catch cases where tags aren't wanted (like in community
maintained layers).

But *we* still want to use human readable tags as the only
version string internally (for our layers), where we know we are
in control. So, introducing a variable that lets us opt-in to
this behavior sounds like something that could be accepted?

-- 
olofjn

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Tobias Hagelborn]

Thanks for the tip Paul,

I was not aware of this support-script.
This is a good workaround in the current Bitbake environment.

We strive to make it easier for our users to continue building if going offline and as such a step, I will submit a new patchset with some optional flag to use local tags (so that no unexpected side-effect is default in a-action). This since it would be a little less overhead/steps to take when building offline.

Cheers,
Tobias


________________________________________
From: Paul Eggleton <paul.eggleton@linux.intel.com>
Sent: Tuesday, August 9, 2016 5:22 AM
To: bitbake-devel@lists.openembedded.org
Cc: Khem Raj; Jérémy Rosen; Tobias Hagelborn; Richard Purdie
Subject: Re: [bitbake-devel] [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

On Mon, 08 Aug 2016 08:35:38 Khem Raj wrote:
> > On Aug 8, 2016, at 4:24 AM, Jérémy Rosen <jeremy.rosen@smile.fr> wrote:
> >
> > On 08/08/2016 11:55, Richard Purdie wrote:
> >> On Mon, 2016-08-08 at 08:34 +0000, Tobias Hagelborn wrote:
> >>> Change made to enable building offline also with git tag references
> >>> in SRC_URI.
> >>>
> >>> If BB_NO_NETWORK is set, fallback to search for tags and revisions
> >>> in local DL_DIR / GITDIR in order to avoid network access.
> >>
> >> I'm not sure I agree with this since you'd get different build results
> >> depending on whether BB_NO_NETWORK is set or not.
> >>
> >> I think if BB_NO_NETWORK is set, its reasonable to assume that the
> >> configuration should be locked down and that BB_NO_NETWORK should not
> >> have other side effects like this.
> >>
> >> It would also make it very hard to detect unlocked configurations
> >> during testing which currently are quite easily identified.
> >
> > To be honest, this is a real world problem I have on a regular basis...
> >
> > The problem we have is that our customers want to keep an archive for
> > (very) long term support and they test offline mode.
> >
> > This problem means that we have to set all our SRC_URI to git SHA instead
> > of the much more readable and easy to check git tags.
> >
> >
> > you think to assume that a reference is a branch, but the real-life
> > problem is when reference is a tag. Tag don't move so this is a case of
> > locked configuration, but i'm not sure how to handle it.
> >
> > Is there a way to have the best of both world somehow ?
>
> Don’t use tags instead rely on SHA numbers in SRCREV and use
> BB_GENERATE_MIRROR_TARBALLS like mentioned here
>
> https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source
> _download_mirror_.3F
>
> tags are floating types in git, you can not use them reliably.
> You may tool in a method to automate updating the
> recipes SRCREVs if you dont want to do it by hand.
>
> That should help hopefully

FYI, another way to handle this is to enable buildhistory, run your build and
then use the buildhistory-collect-srcrevs script. This will produce output
that you can save to .inc file that will lock down the SRCREV values to exact
hashes, which you can then include and commit with your sources. This allows
you to keep your tags and even ${AUTOREV} if you want within each of your
recipes and still have everything locked down for future reproducibility.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Barros Pena, Belen]

On 09/08/2016 10:36, "bitbake-devel-bounces@lists.openembedded.org on
behalf of Olof Johansson" <bitbake-devel-bounces@lists.openembedded.org on
behalf of olof.johansson@axis.com> wrote:

>and unlike OE, we know a lot about our
>users/developers.

We actually know a thing or two about our users, but having worked on both
OE and commercial products I have learnt that the degree of variability in
the user base and the kind of constraints you can enforce upon them are
very different in both contexts.

To be fair, OE has made a significant investment on user-centered
approaches. Between other things, by accepting an interaction designer and
user researcher (yours truly) between its contributors, something pretty
much unheard of among FOSS projects making tools for software engineers.

Cheers

Belén

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1914 bytes --]


> On Aug 9, 2016, at 2:36 AM, Olof Johansson <olof.johansson@axis.com> wrote:
> 
> On 16-08-08 10:22 -0700, Khem Raj wrote:
>> you are confusing how a tool is specifying its functions with
>> process that could emulate a different behavior. OE does not
>> have any control over the internal processes of its users and
>> it should not have.
> 
> No, I don't think I'm confusing anything. Are you confusing OE
> and bitbake? We maintain and develop an in-house Poky based
> distribution, and unlike OE, we know a lot about our
> users/developers.

its a typo.
Thats fine. You can maintain this change locally for your use case.

> 
>>> Tags are only a problem in git if people insist on
>>> modifying/deleting them.
>> 
>> Fact is, its allowed rewrite tags in git, how people use it or not
>> is irrelevant here.
> 
> Nope, Git is a protocol; git the tool can't do nothing about a

Too literal, I guess a reader is getting what I was referring to

> server refusing to (re)move a tag. Your view of what a git tags
> is yours. Tags are meant to be static, otherwise they wouldn't
> have been synced to all clones in such a silly namespace.
> 
> While I agreed with Tobias' patch ("if you use tags in your
> builds, and use BB_NO_NETWORK, you are ok with falling back to
> what you have locally"), RP had a point about this making
> BB_NO_NETWORK have non-obvious side-effects, and also making it
> harder to catch cases where tags aren't wanted (like in community
> maintained layers).
> 
> But *we* still want to use human readable tags as the only
> version string internally (for our layers), where we know we are
> in control. So, introducing a variable that lets us opt-in to
> this behavior sounds like something that could be accepted?

If you were to think of a tool to generate the srcrevs from tags, that IMO
can be better solution here. Like Paul suggested.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Olof Johansson]

On 16-08-09 07:34 -0700, Khem Raj wrote:
> Too literal, I guess a reader is getting what I was referring to

No, a reader did not understood what you are talking about.
Quite frankly, you're not being constructive here Khem. Assuming
that I'm confusing things because we have differences in
opinions, and that it doesn't matter that I don't understand what
you are saying as long as you think that other people do is not
very helpful. What's the point in discussing anything with you?

> If you were to think of a tool to generate the srcrevs from
> tags, that IMO can be better solution here. Like Paul
> suggested.

Yes, Paul's suggestion was very constructive (thanks!), and
that's something we might do. But it still forces us to deal with
sha1 revision ids. Would require effort on our part to hide this
from our users and automate it somehow. Some issues that we would
like to handle:

 - changing PV won't have any effect on what source is fetched/used
 - changing SRCREV to another revision won't affect PV

-- 
olofjn

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

Olof

> On Aug 9, 2016, at 8:09 AM, Olof Johansson <olof.johansson@axis.com> wrote:
> 
> On 16-08-09 07:34 -0700, Khem Raj wrote:
>> Too literal, I guess a reader is getting what I was referring to
> 
> No, a reader did not understood what you are talking about.
> Quite frankly, you're not being constructive here Khem. Assuming
> that I'm confusing things because we have differences in
> opinions, and that it doesn't matter that I don't understand what
> you are saying as long as you think that other people do is not
> very helpful. What's the point in discussing anything with you?

I am sorry if its coming across this way.I for one was trying to convey
that there are reasons to not have such a patch e.g. predictable
builds and general support problems with BB_NO_NETWORK.

We have been through this very same problem and decided to use srcrev
collection approach which is similar to buildhistory collection
approach. It was a bit of getting used to, but in the end it was
a very good way.

> 
>> If you were to think of a tool to generate the srcrevs from
>> tags, that IMO can be better solution here. Like Paul
>> suggested.
> 
> Yes, Paul's suggestion was very constructive (thanks!), and
> that's something we might do. But it still forces us to deal with
> sha1 revision ids. Would require effort on our part to hide this
> from our users and automate it somehow. Some issues that we would
> like to handle:
> 
> - changing PV won't have any effect on what source is fetched/used

I believe it should not.

> - changing SRCREV to another revision won't affect PV

If you do not utilize SRCPV in PV variable. SRCREV should not reflect
in the PV

> 
> --
> olofjn


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Olof Johansson]

On 16-08-09 08:54 -0700, Khem Raj wrote:
> I am sorry if its coming across this way.I for one was trying to convey
> that there are reasons to not have such a patch e.g. predictable
> builds and general support problems with BB_NO_NETWORK.
> 
> We have been through this very same problem and decided to use srcrev
> collection approach which is similar to buildhistory collection
> approach. It was a bit of getting used to, but in the end it was
> a very good way.

Thanks for clarifying, this is a reasonable stance, and I can see
where you are coming from. But I'm not advocating that this
behavior should be forced upon anybody. What I'm trying to convey
is that not all environments have issues with volatile tags, and
from my pov, it's reasonably easy to fight people (within an
organization) moving tags if you own the Git infrastructure =).

(Having reproducible builds is another way to alleviate the risks
of using possibly volatile tags. If the build output is
different, then the input obviously is different as well (incl.
implicit inputs like datetime, TZ or locale). Just putting it out
there. People in the Debian community and elsewhere are doing a
lot of work in this area. [1])

> >> If you were to think of a tool to generate the srcrevs from
> >> tags, that IMO can be better solution here. Like Paul
> >> suggested.
> > 
> > Yes, Paul's suggestion was very constructive (thanks!), and
> > that's something we might do. But it still forces us to deal with
> > sha1 revision ids. Would require effort on our part to hide this
> > from our users and automate it somehow. Some issues that we would
> > like to handle:
> > 
> > - changing PV won't have any effect on what source is fetched/used
> 
> I believe it should not.

Yes, that's the heart of our disagreement, I think. We currently
set SRCREV = "${PV}". We believe that in our environment, we are
safe to do this even without syncing with the remote.

> > - changing SRCREV to another revision won't affect PV
> 
> If you do not utilize SRCPV in PV variable. SRCREV should not reflect
> in the PV

Exactly, so that's the problem for me. I'm hesitent to introduce
sha1s into the metadata in our layers as it will make us have two
variables to keep track of the version. I fear that they will
become out of sync.

If somebody tries to only update the SRCREV, the package
manifests and elsewhere will be confusing for people trying to
troubleshoot as the PV is that of the old version (the same way a
moved tag can cause massive confusion). If somebody tries to only
update the PV, the package manifest will look correct, but the
sources is that of something else. Is this something you have any
experience with?

-- 
olofjn

1: https://reproducible-builds.org/

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Richard Purdie]

On Tue, 2016-08-09 at 18:26 +0200, Olof Johansson wrote:
> On 16-08-09 08:54 -0700, Khem Raj wrote:
> > I am sorry if its coming across this way.I for one was trying to
> > convey
> > that there are reasons to not have such a patch e.g. predictable
> > builds and general support problems with BB_NO_NETWORK.
> > 
> > We have been through this very same problem and decided to use
> > srcrev
> > collection approach which is similar to buildhistory collection
> > approach. It was a bit of getting used to, but in the end it was
> > a very good way.
> 
> Thanks for clarifying, this is a reasonable stance, and I can see
> where you are coming from. But I'm not advocating that this
> behavior should be forced upon anybody. What I'm trying to convey
> is that not all environments have issues with volatile tags, and
> from my pov, it's reasonably easy to fight people (within an
> organization) moving tags if you own the Git infrastructure =).

The trouble is that whilst you and others posting on this mailing list
understand this problem (in some cases now its been explained), in the
general case people don't realise that git tags can be rewritten.

If there is an "easy" option to make a failure go away, people will
take it every time, without understanding why it can be a bad idea.
People prefer not to think if they don't need to.

I've watched git tags get rewritten by upstreams who you'd have thought
would have known better, with all kinds of nasty fallout from the
result as few people understood what happened, why or how to fix it.
The usual "solution" is to blame bitbake or sstate, since they're
complex, people don't understand them and they're therefore clearly at
fault and broken.

So my position on this is based on experience of how badly you can get
burnt by it. I was very reluctant to make bitbake behave in the way it
does today originally but in the end various discussions brought me to
that conclusion.

What I'm trying to avoid in the fetcher is too many code paths. We
already have a lot of them, probably too many, and it makes maintaining
and enhancing the fetchers very very hard as you generally end up
trampling on some weird usecase someone relies upon. I therefore in
general don't like magic switches that drastically alter behaviour,
prefering to have one standard path we all use. Add to that the
concerns above and the idea of making this optional for people makes me
very nervous.

Having better descriptions of revisions is a semi-tangential issue.
There have been a few different approaches to using the output of git
describe or counting revisions and injecting it into PV. The
gitpkgv.bbclass was one similar idea and we did end up putting a method
into the fetcher to better enable this where the revision number was
based on numbers of commits in a branch. I'd be ok with extending or
supplementing that idea to work with git describe so that package
output revisions are human readable. I would however want the recipes
and fetcher to primarily work with the real sha revisions as those are
what define a specific revision in git.

Cheers,

Richard

Re: [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote

[Jérémy Rosen]

(this conversation accidentally became private, I'm forwarding it to the 
mailing list...)

On 09/08/2016 17:50, Olof Johansson wrote:
> On 16-08-09 17:16 +0200, Jérémy Rosen wrote:
>> I must admit i'm kinda confused here...
>>
>> what's the point of NO_NETWORK
>>
>> * the "airplane use-case : I'm on a plane with no network access and I want
>> to work on yocto. I built stuff previously and I don't really care about
>> upstream changes. I set NO_NETWORK so bitbake doesn't complain but still
>> build everything : In that case, bitbake needs to use it's locally cached
>> ref positions, and currently doesn't work
>>
>> * the "long term archive" use-case. I have built a local mirror 10years ago
>> and I want to rebuild from scratch. In that case, it makes sense (again) to
>> use the ref from the mirror to find you SHA. Yes, fixed refs would be
>> better, but if the ref have changed, I would argue that using the old
>> position is exactly what you want.
>>
>> * other use-case ? I don't really see...
> There's a variable called BB_ALLOWED_NETWORKS, so if you are
> using BB_NO_NETWORK, we can whitelist one or more hosts or
> networks:
>
>   BB_NO_NETWORK = "1"
>   BB_ALLOWED_NETWORKS = "*.axis.com mirror.example.com"
>
> This introduces a new use case: we can make sure that only some
> network hosts are used during builds. (Like only using your
> internal mirror or your internal Git server.)
Interesting... not a use-case I had thought of
>> Overall, I think that if a SRC_URI points to a ref, NO_NETWORK is set, and
>> we have that ref somewhere in DL_DIR, then using that ref is exactly what we
>> wants, and currently it doesn't work because bitbake insists on using the
>> network even when we tell it explicitely to do so.
>>
>> I understand Tobias's point (that it has side effects) but I don't really
>> understand the idea that any SRC_URI with a ref should always fail with
>> NO_NETWORK...
> So I thought Tobias' patch made sense for the same reason you do,
> but I can live with having to set another variable to opt-in to
> that behavior. Clearly people have mixed opinions about this, but
> having the option to enable the behavior would be enough for us.
It seems that there is an assumption
BB_NO_NETWORK == reproducible build

That assumption is true for my "long term archive" use-case, but
definitely not for my airplane use-case.

  From a user experience point of view, it would make sense to "activate"
(or deactivate) reproducible build explicitely. That could trigger a
number of important QA tests that don't make sense for general consumption

but again, no-network is not reproducible build (they have some common
ideas, but there are cases where you want one but not the other) so the
two concepts should be separated...

> I can see the case for what Richard mentioned, that the upstream
> community layers (or other layers that forbid git tags) can use
> BB_NO_NETWORK to verify that no refs are being used in SRCREV.
> As this is an existing use case, it would probably be a bad idea
> to change it now.
so some sort of BB_NOT_REPRODUCIBLE variable (horrible name, please find
a better one :P )
>
> For your "long term archive" (as well as the airplane) use case,
> I think that you could probably use what Paul mentioned without
> the caveats we are facing; as (if I understood your case
> correctly) you aren't actively doing development on the sources,
> just reproducing builds offline.
yes for "long term" but "airplane" can't make that assumption. it's just
a dev on the move, or without general internet access...
more generally, I understand BB_NO_NETWORK as telling bitbake that no
network is not an error and it should work around it.

>
> We would like to avoid having two version strings (one being
> informational only), as this will probably lead to confusion as
> people are actually updating the recipes:
>
> Q: "why didn't my modification make it into the build?"
> A: "because you only updated PV, not SRCREV"
>
> Q: "why does the package manifest claim that this is the old
>      version even though my changes are in?"
> A: "because you only updated SRCREV, not PV"
>
I don't really understand that last part, I must admit I only use
pvserver, so I don't touch PV ususally.
I completely understand that having a proper, reproducible build
requires that you change all floating revs to fixed revs (including,
arguably, any git tag) but I don't understand why NO_NETWORK should
force fixed rev on the user...

it's mainly a problem of discussing the use-case for no-network... mine
doesn't work if you force no-floating-rev on me.

Nobody discussed the idea of using git-describe as a way around (or a
way to prettyfy) the SRCREV freeze. This way of decribing a commit is
both human readable and guaranteed unique (it contains a shortened SHA)

Hope this helps
Jeremy

> Hope this helps in clarifying a little :)
>