[PATCH 1/1] go.bbclass: Allow network in do_compile

[PATCH 1/1] go.bbclass: Allow network in do_compile

[Andrei Gherzan]

From: Andrei Gherzan <andrei.gherzan@huawei.com>

Compile pulls in the go.mod list requiring network. Without this, do
compile would fail with a similar error to the following:

dial tcp: lookup proxy.golang.org: Temporary failure in name resolution

Signed-off-by: Andrei Gherzan <andrei.gherzan@huawei.com>
---
 meta/classes/go.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/go.bbclass b/meta/classes/go.bbclass
index 9c4c92bffd..dc246c72cb 100644
--- a/meta/classes/go.bbclass
+++ b/meta/classes/go.bbclass
@@ -113,6 +113,7 @@ go_do_compile() {
 }
 do_compile[dirs] =+ "${GOTMPDIR}"
 do_compile[cleandirs] = "${B}/bin ${B}/pkg"
+do_compile[network] = "1"
 
 go_do_install() {
 	install -d ${D}${libdir}/go/src/${GO_IMPORT}
-- 
2.25.1

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Bruce Ashfield]

On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>
> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>
> Compile pulls in the go.mod list requiring network. Without this, do
> compile would fail with a similar error to the following:
>
> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution

This is something that needs to be carried in your own layers, IMHO it
isn't appropriate for core.

It isn't about the fetching, it is the entire gap in functionality
that we are missing if go starts fetching dependencies during compile.

Bruce

>
> Signed-off-by: Andrei Gherzan <andrei.gherzan@huawei.com>
> ---
>  meta/classes/go.bbclass | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/meta/classes/go.bbclass b/meta/classes/go.bbclass
> index 9c4c92bffd..dc246c72cb 100644
> --- a/meta/classes/go.bbclass
> +++ b/meta/classes/go.bbclass
> @@ -113,6 +113,7 @@ go_do_compile() {
>  }
>  do_compile[dirs] =+ "${GOTMPDIR}"
>  do_compile[cleandirs] = "${B}/bin ${B}/pkg"
> +do_compile[network] = "1"
>
>  go_do_install() {
>         install -d ${D}${libdir}/go/src/${GO_IMPORT}
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#162517): https://lists.openembedded.org/g/openembedded-core/message/162517
> Mute This Topic: https://lists.openembedded.org/mt/89464905/1050810
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Bruce Ashfield]

On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
lists.openembedded.org
<bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>
> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
> >
> > From: Andrei Gherzan <andrei.gherzan@huawei.com>
> >
> > Compile pulls in the go.mod list requiring network. Without this, do
> > compile would fail with a similar error to the following:
> >
> > dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>
> This is something that needs to be carried in your own layers, IMHO it
> isn't appropriate for core.
>
> It isn't about the fetching, it is the entire gap in functionality
> that we are missing if go starts fetching dependencies during compile.

A further thought is that if this is for go.mod issues, there is the
go-mod.bbclass.

Perhaps enabling it in that class and doing a bbwarn about go fetching
dependencies would be appropriate ?

Otherwise, someone may not know that this is happening and that a no
network configuration has no chance of working.

Cheers,

Bruce

>
> Bruce
>
> >
> > Signed-off-by: Andrei Gherzan <andrei.gherzan@huawei.com>
> > ---
> >  meta/classes/go.bbclass | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/meta/classes/go.bbclass b/meta/classes/go.bbclass
> > index 9c4c92bffd..dc246c72cb 100644
> > --- a/meta/classes/go.bbclass
> > +++ b/meta/classes/go.bbclass
> > @@ -113,6 +113,7 @@ go_do_compile() {
> >  }
> >  do_compile[dirs] =+ "${GOTMPDIR}"
> >  do_compile[cleandirs] = "${B}/bin ${B}/pkg"
> > +do_compile[network] = "1"
> >
> >  go_do_install() {
> >         install -d ${D}${libdir}/go/src/${GO_IMPORT}
> > --
> > 2.25.1
> >
> >
> >
> >
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#162519): https://lists.openembedded.org/g/openembedded-core/message/162519
> Mute This Topic: https://lists.openembedded.org/mt/89464905/1050810
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Andrei Gherzan]

On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
> lists.openembedded.org
> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>>
>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>> >
>> > From: Andrei Gherzan <andrei.gherzan@huawei.com>
>> >
>> > Compile pulls in the go.mod list requiring network. Without this, do
>> > compile would fail with a similar error to the following:
>> >
>> > dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>>
>> This is something that needs to be carried in your own layers, IMHO it
>> isn't appropriate for core.
>>
>> It isn't about the fetching, it is the entire gap in functionality
>> that we are missing if go starts fetching dependencies during compile.
>
> A further thought is that if this is for go.mod issues, there is the
> go-mod.bbclass.
>
> Perhaps enabling it in that class and doing a bbwarn about go fetching
> dependencies would be appropriate ?
>
> Otherwise, someone may not know that this is happening and that a no
> network configuration has no chance of working.
>

I reckon that is reasonable. I'll personally go down the recipe level to workaround this change but understanding and agreeing with the reasoning behind this change, I want to invest a bit into trying to find a proper solution in the core. Bruce, I know you invested a fair amount of time into this already. Would you be willing to sync up and see how we can work together in tackling this? 

Added Zyga to CC as he is also interested in this as part of his go development activities.

Thanks,
Andrei

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Richard Purdie]

On Tue, 2022-03-01 at 11:42 +0000, Andrei Gherzan wrote:
> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
> > On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
> > lists.openembedded.org
> > <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
> > > 
> > > On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
> > > > 
> > > > From: Andrei Gherzan <andrei.gherzan@huawei.com>
> > > > 
> > > > Compile pulls in the go.mod list requiring network. Without this, do
> > > > compile would fail with a similar error to the following:
> > > > 
> > > > dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
> > > 
> > > This is something that needs to be carried in your own layers, IMHO it
> > > isn't appropriate for core.
> > > 
> > > It isn't about the fetching, it is the entire gap in functionality
> > > that we are missing if go starts fetching dependencies during compile.
> > 
> > A further thought is that if this is for go.mod issues, there is the
> > go-mod.bbclass.
> > 
> > Perhaps enabling it in that class and doing a bbwarn about go fetching
> > dependencies would be appropriate ?
> > 
> > Otherwise, someone may not know that this is happening and that a no
> > network configuration has no chance of working.
> > 
> 
> I reckon that is reasonable. I'll personally go down the recipe level to workaround 
> this change but understanding and agreeing with the reasoning behind this change, 
> I want to invest a bit into trying to find a proper solution in the core. Bruce, 
> I know you invested a fair amount of time into this already. Would you be willing 
> to sync up and see how we can work together in tackling this? 
> 
> Added Zyga to CC as he is also interested in this as part of his go development activities.

I'd like to ask that anyone working around this does leave a bb.warn() in the
code so that the issue is visible. We need to make sure there is awareness of
the problem and hopefully that will encourage people not to forget about it.

Cheers,

Richard

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Bruce Ashfield]

On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>
> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
> > On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
> > lists.openembedded.org
> > <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
> >>
> >> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
> >> >
> >> > From: Andrei Gherzan <andrei.gherzan@huawei.com>
> >> >
> >> > Compile pulls in the go.mod list requiring network. Without this, do
> >> > compile would fail with a similar error to the following:
> >> >
> >> > dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
> >>
> >> This is something that needs to be carried in your own layers, IMHO it
> >> isn't appropriate for core.
> >>
> >> It isn't about the fetching, it is the entire gap in functionality
> >> that we are missing if go starts fetching dependencies during compile.
> >
> > A further thought is that if this is for go.mod issues, there is the
> > go-mod.bbclass.
> >
> > Perhaps enabling it in that class and doing a bbwarn about go fetching
> > dependencies would be appropriate ?
> >
> > Otherwise, someone may not know that this is happening and that a no
> > network configuration has no chance of working.
> >
>
> I reckon that is reasonable. I'll personally go down the recipe level to workaround this change but understanding and agreeing with the reasoning behind this change, I want to invest a bit into trying to find a proper solution in the core. Bruce, I know you invested a fair amount of time into this already. Would you be willing to sync up and see how we can work together in tackling this?
>

Definitely, more ideas are good. In fact, I think there are probably
several approaches that can co-exist, depending on what a
recipe/developer needs.

I'm in the Eastern time zone here, and will try and grab folks on IRC
to have a level set

Bruce

> Added Zyga to CC as he is also interested in this as part of his go development activities.
>
> Thanks,
> Andrei



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH 1/1] go.bbclass: Allow network in do_compile

[lukas.funke]

[-- Attachment #1: Type: text/plain, Size: 3756 bytes --]

On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:

> 
> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
> 
>> 
>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>> 
>>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>>> lists.openembedded.org
>>> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>>> 
>>>> 
>>>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>>>> 
>>>> 
>>>>> 
>>>>> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>>>>> 
>>>>> Compile pulls in the go.mod list requiring network. Without this, do
>>>>> compile would fail with a similar error to the following:
>>>>> 
>>>>> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>>>> 
>>>> This is something that needs to be carried in your own layers, IMHO it
>>>> isn't appropriate for core.
>>>> 
>>>> It isn't about the fetching, it is the entire gap in functionality
>>>> that we are missing if go starts fetching dependencies during compile.
>>> 
>>> A further thought is that if this is for go.mod issues, there is the
>>> go-mod.bbclass.
>>> 
>>> Perhaps enabling it in that class and doing a bbwarn about go fetching
>>> dependencies would be appropriate ?
>>> 
>>> Otherwise, someone may not know that this is happening and that a no
>>> network configuration has no chance of working.
>> 
>> I reckon that is reasonable. I'll personally go down the recipe level to
>> workaround this change but understanding and agreeing with the reasoning
>> behind this change, I want to invest a bit into trying to find a proper
>> solution in the core. Bruce, I know you invested a fair amount of time
>> into this already. Would you be willing to sync up and see how we can work
>> together in tackling this?
> 
> Definitely, more ideas are good. In fact, I think there are probably
> several approaches that can co-exist, depending on what a
> recipe/developer needs.
> 
> I'm in the Eastern time zone here, and will try and grab folks on IRC
> to have a level set
> 
> Bruce
> 
> 
>> Added Zyga to CC as he is also interested in this as part of his go
>> development activities.
>> 
>> Thanks,
>> Andrei
> 
> 
> 
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II

The problem in allowing downloads during compile (e.g. by go) is, that it leads to non-reproducable builds. I'm currently facing the same issue and would like to have a reproducable go *offline* build.
I would like to propose two ideas to workaround the go-compile fetching issue:

First:
- Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies into go projects 'vendor' folder. This forces go to compile offline. However, one have to generate the 'modules.txt' file in the vendor folder 'manually' during unpack. This is error prone, as there is no official documentation how this format should look like. Anyway, I've tried this approach and it works for me.

Second:
- Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher) and unpack the dependencies into a local (workdir) go-path. This seemed a good solution for me as the go-path is well defined. But for some reason 'go'  fetches the zip-files during compile into it's download-cache AGAIN, even if the source is already unpacked in the go-path. I'll assume this is required to verify the source files integrity?! With this approach one have to adapt 'go' to suppress this download behaviour.

Please let me know your opinion on these two approaches.

[-- Attachment #2: Type: text/html, Size: 3982 bytes --]

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Khem Raj]

On 3/1/22 7:54 AM, lukas.funke@weidmueller.com wrote:
> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
> 
>     On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com>
>     wrote:
> 
> 
>         On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
> 
>             On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>             lists.openembedded.org
>             <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
> 
> 
>                 On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan
>                 <andrei@gherzan.com> wrote:
> 
> 
>                     From: Andrei Gherzan <andrei.gherzan@huawei.com>
> 
>                     Compile pulls in the go.mod list requiring network.
>                     Without this, do
>                     compile would fail with a similar error to the
>                     following:
> 
>                     dial tcp: lookup proxy.golang.org: Temporary failure
>                     in name resolution
> 
>                 This is something that needs to be carried in your own
>                 layers, IMHO it
>                 isn't appropriate for core.
> 
>                 It isn't about the fetching, it is the entire gap in
>                 functionality
>                 that we are missing if go starts fetching dependencies
>                 during compile.
> 
>             A further thought is that if this is for go.mod issues,
>             there is the
>             go-mod.bbclass.
> 
>             Perhaps enabling it in that class and doing a bbwarn about
>             go fetching
>             dependencies would be appropriate ?
> 
>             Otherwise, someone may not know that this is happening and
>             that a no
>             network configuration has no chance of working.
> 
>         I reckon that is reasonable. I'll personally go down the recipe
>         level to workaround this change but understanding and agreeing
>         with the reasoning behind this change, I want to invest a bit
>         into trying to find a proper solution in the core. Bruce, I know
>         you invested a fair amount of time into this already. Would you
>         be willing to sync up and see how we can work together in
>         tackling this?
> 
>     Definitely, more ideas are good. In fact, I think there are probably
>     several approaches that can co-exist, depending on what a
>     recipe/developer needs.
> 
>     I'm in the Eastern time zone here, and will try and grab folks on IRC
>     to have a level set
> 
>     Bruce
> 
>         Added Zyga to CC as he is also interested in this as part of his
>         go development activities.
> 
>         Thanks,
>         Andrei
> 
> 
> 
>     -- 
>     - Thou shalt not follow the NULL pointer, for chaos and madness await
>     thee at its end
>     - "Use the force Harry" - Gandalf, Star Trek II
> 
> The problem in allowing downloads during compile (e.g. by go) is, that 
> it leads to non-reproducable builds. I'm currently facing the same issue 
> and would like to have a reproducable go *offline* build.
> I would like to propose two ideas to workaround the go-compile fetching 
> issue:
> First:
> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. 
> by writing a seperate go fetcher or a wget-fetcher) and unpack the 
> dependencies into go projects 'vendor' folder. This forces go to compile 
> offline. However, one have to generate the 'modules.txt' file in the 
> vendor folder 'manually' during unpack. This is error prone, as there is 
> no official documentation how this format should look like. Anyway, I've 
> tried this approach and it works for me.
> Second:
> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. 
> by writing a seperate go fetcher) and unpack the dependencies into a 
> local (workdir) go-path. This seemed a good solution for me as the 
> go-path is well defined. But for some reason 'go'  fetches the zip-files 
> during compile into it's download-cache AGAIN, even if the source is 
> already unpacked in the go-path. I'll assume this is required to verify 
> the source files integrity?! With this approach one have to adapt 'go' 
> to suppress this download behaviour.
> Please let me know your opinion on these two approaches.
> 


Second approach is more tenable, go community has larger plans for go 
modules, so it will be good for bitbake to delegate work to it rather 
than doing parallel duplicate work, that will mean we will have to keep 
up with go mod changes always.

> 
> 
> 

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Bruce Ashfield]

On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
>
> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>
> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>
> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
> lists.openembedded.org
> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>
>
> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>
> Compile pulls in the go.mod list requiring network. Without this, do
> compile would fail with a similar error to the following:
>
> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>
> This is something that needs to be carried in your own layers, IMHO it
> isn't appropriate for core.
>
> It isn't about the fetching, it is the entire gap in functionality
> that we are missing if go starts fetching dependencies during compile.
>
> A further thought is that if this is for go.mod issues, there is the
> go-mod.bbclass.
>
> Perhaps enabling it in that class and doing a bbwarn about go fetching
> dependencies would be appropriate ?
>
> Otherwise, someone may not know that this is happening and that a no
> network configuration has no chance of working.
>
> I reckon that is reasonable. I'll personally go down the recipe level to workaround this change but understanding and agreeing with the reasoning behind this change, I want to invest a bit into trying to find a proper solution in the core. Bruce, I know you invested a fair amount of time into this already. Would you be willing to sync up and see how we can work together in tackling this?
>
> Definitely, more ideas are good. In fact, I think there are probably
> several approaches that can co-exist, depending on what a
> recipe/developer needs.
>
> I'm in the Eastern time zone here, and will try and grab folks on IRC
> to have a level set
>
> Bruce
>
> Added Zyga to CC as he is also interested in this as part of his go development activities.
>
> Thanks,
> Andrei
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> The problem in allowing downloads during compile (e.g. by go) is, that it leads to non-reproducable builds. I'm currently facing the same issue and would like to have a reproducable go *offline* build.
> I would like to propose two ideas to workaround the go-compile fetching issue:
>
> First:
> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies into go projects 'vendor' folder. This forces go to compile offline. However, one have to generate the 'modules.txt' file in the vendor folder 'manually' during unpack. This is error prone, as there is no official documentation how this format should look like. Anyway, I've tried this approach and it works for me.
>
> Second:
> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher) and unpack the dependencies into a local (workdir) go-path. This seemed a good solution for me as the go-path is well defined. But for some reason 'go'  fetches the zip-files during compile into it's download-cache AGAIN, even if the source is already unpacked in the go-path. I'll assume this is required to verify the source files integrity?! With this approach one have to adapt 'go' to suppress this download behaviour.
>

I've been doing offline builds using a constructed vendor/ directory
and generated modules.txt.

The only difference between what I have working and what you are
suggesting (type 1), is that I've gone directly to the sources and
constructed the vendor directory using the OE git fetcher. That allows
all functionality to continue to work that is part of OEcore, and the
build to continue.  Switching out the git fetches for tarballs would
be possible, I just wasn't sure how to use the proxied modules (and I
wanted the history for debug).

I've never had any issues with the modules.txt, as I generate it at
the same time as the git fetch lines for the SRC_URI. I've also not
been using information from the go.mod directly from go.proxy.org, it
is information I've generated from a clone of the project and dumped
via go mod. There's likely improvements I can do there, but with what
I'm doing, I'm going directly to the source of the projects and doing
clones, which keeps everything clear of the go infrastructure.

I have a utility that I'm still cleaning up that generates the SRC_URI
lines, as well as the modules.txt, when I resolve a few nagging
issues, I'll make the WIP scripts available.

Other projects (BSD, etc), have been doing different sorts of
constructed vendor directories, but they are similar in approach.

For the short term (i.e. the upcoming release), that is pretty much
all we can do. There isn't enough time to implement a new go fetcher
backend for bitbake.

In the end, how we fetch and place the dependencies is a transport, so
whether or not we fetch them ourselves, or let go do it, that part is
largely the same.

For now (short term), I favour vendor/, as it is workable, but not
perfect. It isn't exactly efficient or pretty, but at least it seems
to produce correct output, and allows all of the project capabilities
to work. And of course, the approach will continue to work regardless
of development on other go.mod elements.

Cheers,

Bruce

> Please let me know your opinion on these two approaches.
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#162562): https://lists.openembedded.org/g/openembedded-core/message/162562
> Mute This Topic: https://lists.openembedded.org/mt/89464905/1050810
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Andrei Gherzan]

Mar 1, 2022 20:15:52 Bruce Ashfield <bruce.ashfield@gmail.com>:

> On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
>>
>> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>>
>> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>>
>>
>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>>
>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>> lists.openembedded.org
>> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>>
>>
>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>>
>>
>> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>>
>> Compile pulls in the go.mod list requiring network. Without this, do
>> compile would fail with a similar error to the following:
>>
>> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>>
>> This is something that needs to be carried in your own layers, IMHO it
>> isn't appropriate for core.
>>
>> It isn't about the fetching, it is the entire gap in functionality
>> that we are missing if go starts fetching dependencies during compile.
>>
>> A further thought is that if this is for go.mod issues, there is the
>> go-mod.bbclass.
>>
>> Perhaps enabling it in that class and doing a bbwarn about go fetching
>> dependencies would be appropriate ?
>>
>> Otherwise, someone may not know that this is happening and that a no
>> network configuration has no chance of working.
>>
>> I reckon that is reasonable. I'll personally go down the recipe level to workaround this change but understanding and agreeing with the reasoning behind this change, I want to invest a bit into trying to find a proper solution in the core. Bruce, I know you invested a fair amount of time into this already. Would you be willing to sync up and see how we can work together in tackling this?
>>
>> Definitely, more ideas are good. In fact, I think there are probably
>> several approaches that can co-exist, depending on what a
>> recipe/developer needs.
>>
>> I'm in the Eastern time zone here, and will try and grab folks on IRC
>> to have a level set
>>
>> Bruce
>>
>> Added Zyga to CC as he is also interested in this as part of his go development activities.
>>
>> Thanks,
>> Andrei
>>
>>
>>
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>>
>> The problem in allowing downloads during compile (e.g. by go) is, that it leads to non-reproducable builds. I'm currently facing the same issue and would like to have a reproducable go *offline* build.
>> I would like to propose two ideas to workaround the go-compile fetching issue:
>>
>> First:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies into go projects 'vendor' folder. This forces go to compile offline. However, one have to generate the 'modules.txt' file in the vendor folder 'manually' during unpack. This is error prone, as there is no official documentation how this format should look like. Anyway, I've tried this approach and it works for me.
>>
>> Second:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher) and unpack the dependencies into a local (workdir) go-path. This seemed a good solution for me as the go-path is well defined. But for some reason 'go'  fetches the zip-files during compile into it's download-cache AGAIN, even if the source is already unpacked in the go-path. I'll assume this is required to verify the source files integrity?! With this approach one have to adapt 'go' to suppress this download behaviour.
>>
>
> I've been doing offline builds using a constructed vendor/ directory
> and generated modules.txt.
>
> The only difference between what I have working and what you are
> suggesting (type 1), is that I've gone directly to the sources and
> constructed the vendor directory using the OE git fetcher. That allows
> all functionality to continue to work that is part of OEcore, and the
> build to continue.  Switching out the git fetches for tarballs would
> be possible, I just wasn't sure how to use the proxied modules (and I
> wanted the history for debug).
>
> I've never had any issues with the modules.txt, as I generate it at
> the same time as the git fetch lines for the SRC_URI. I've also not
> been using information from the go.mod directly from go.proxy.org, it
> is information I've generated from a clone of the project and dumped
> via go mod. There's likely improvements I can do there, but with what
> I'm doing, I'm going directly to the source of the projects and doing
> clones, which keeps everything clear of the go infrastructure.
>
> I have a utility that I'm still cleaning up that generates the SRC_URI
> lines, as well as the modules.txt, when I resolve a few nagging
> issues, I'll make the WIP scripts available.
>
> Other projects (BSD, etc), have been doing different sorts of
> constructed vendor directories, but they are similar in approach.
>
> For the short term (i.e. the upcoming release), that is pretty much
> all we can do. There isn't enough time to implement a new go fetcher
> backend for bitbake.
>
> In the end, how we fetch and place the dependencies is a transport, so
> whether or not we fetch them ourselves, or let go do it, that part is
> largely the same.
>
> For now (short term), I favour vendor/, as it is workable, but not
> perfect. It isn't exactly efficient or pretty, but at least it seems
> to produce correct output, and allows all of the project capabilities
> to work. And of course, the approach will continue to work regardless
> of development on other go.mod elements.

After reflecting on this for a while I reckon this is the fastest way forward while addressing the reproducibility issue. I'm wondering what we can do in terms of compliance? Maybe we can turn the script you were talking about into a recipe generator that also deals with this by querying the licenses of all the dependencies (direct and indirect).

That being said, how can I help? It seems that there is an existing WIP state on this. Can I take something from it? Maybe help with cleaning up the script?
--
Andrei Gherzan
gpg: rsa4096/D4D94F67AD0E9640

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Bruce Ashfield]

On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> Mar 1, 2022 20:15:52 Bruce Ashfield <bruce.ashfield@gmail.com>:
>
> > On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
> >>
> >> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
> >>
> >> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
> >>
> >>
> >> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
> >>
> >> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
> >> lists.openembedded.org
> >> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
> >>
> >>
> >> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
> >>
> >>
> >> From: Andrei Gherzan <andrei.gherzan@huawei.com>
> >>
> >> Compile pulls in the go.mod list requiring network. Without this, do
> >> compile would fail with a similar error to the following:
> >>
> >> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
> >>
> >> This is something that needs to be carried in your own layers, IMHO it
> >> isn't appropriate for core.
> >>
> >> It isn't about the fetching, it is the entire gap in functionality
> >> that we are missing if go starts fetching dependencies during compile.
> >>
> >> A further thought is that if this is for go.mod issues, there is the
> >> go-mod.bbclass.
> >>
> >> Perhaps enabling it in that class and doing a bbwarn about go fetching
> >> dependencies would be appropriate ?
> >>
> >> Otherwise, someone may not know that this is happening and that a no
> >> network configuration has no chance of working.
> >>
> >> I reckon that is reasonable. I'll personally go down the recipe level to workaround this change but understanding and agreeing with the reasoning behind this change, I want to invest a bit into trying to find a proper solution in the core. Bruce, I know you invested a fair amount of time into this already. Would you be willing to sync up and see how we can work together in tackling this?
> >>
> >> Definitely, more ideas are good. In fact, I think there are probably
> >> several approaches that can co-exist, depending on what a
> >> recipe/developer needs.
> >>
> >> I'm in the Eastern time zone here, and will try and grab folks on IRC
> >> to have a level set
> >>
> >> Bruce
> >>
> >> Added Zyga to CC as he is also interested in this as part of his go development activities.
> >>
> >> Thanks,
> >> Andrei
> >>
> >>
> >>
> >> --
> >> - Thou shalt not follow the NULL pointer, for chaos and madness await
> >> thee at its end
> >> - "Use the force Harry" - Gandalf, Star Trek II
> >>
> >> The problem in allowing downloads during compile (e.g. by go) is, that it leads to non-reproducable builds. I'm currently facing the same issue and would like to have a reproducable go *offline* build.
> >> I would like to propose two ideas to workaround the go-compile fetching issue:
> >>
> >> First:
> >> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies into go projects 'vendor' folder. This forces go to compile offline. However, one have to generate the 'modules.txt' file in the vendor folder 'manually' during unpack. This is error prone, as there is no official documentation how this format should look like. Anyway, I've tried this approach and it works for me.
> >>
> >> Second:
> >> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher) and unpack the dependencies into a local (workdir) go-path. This seemed a good solution for me as the go-path is well defined. But for some reason 'go'  fetches the zip-files during compile into it's download-cache AGAIN, even if the source is already unpacked in the go-path. I'll assume this is required to verify the source files integrity?! With this approach one have to adapt 'go' to suppress this download behaviour.
> >>
> >
> > I've been doing offline builds using a constructed vendor/ directory
> > and generated modules.txt.
> >
> > The only difference between what I have working and what you are
> > suggesting (type 1), is that I've gone directly to the sources and
> > constructed the vendor directory using the OE git fetcher. That allows
> > all functionality to continue to work that is part of OEcore, and the
> > build to continue.  Switching out the git fetches for tarballs would
> > be possible, I just wasn't sure how to use the proxied modules (and I
> > wanted the history for debug).
> >
> > I've never had any issues with the modules.txt, as I generate it at
> > the same time as the git fetch lines for the SRC_URI. I've also not
> > been using information from the go.mod directly from go.proxy.org, it
> > is information I've generated from a clone of the project and dumped
> > via go mod. There's likely improvements I can do there, but with what
> > I'm doing, I'm going directly to the source of the projects and doing
> > clones, which keeps everything clear of the go infrastructure.
> >
> > I have a utility that I'm still cleaning up that generates the SRC_URI
> > lines, as well as the modules.txt, when I resolve a few nagging
> > issues, I'll make the WIP scripts available.
> >
> > Other projects (BSD, etc), have been doing different sorts of
> > constructed vendor directories, but they are similar in approach.
> >
> > For the short term (i.e. the upcoming release), that is pretty much
> > all we can do. There isn't enough time to implement a new go fetcher
> > backend for bitbake.
> >
> > In the end, how we fetch and place the dependencies is a transport, so
> > whether or not we fetch them ourselves, or let go do it, that part is
> > largely the same.
> >
> > For now (short term), I favour vendor/, as it is workable, but not
> > perfect. It isn't exactly efficient or pretty, but at least it seems
> > to produce correct output, and allows all of the project capabilities
> > to work. And of course, the approach will continue to work regardless
> > of development on other go.mod elements.
>
> After reflecting on this for a while I reckon this is the fastest way forward while addressing the reproducibility issue. I'm wondering what we can do in terms of compliance? Maybe we can turn the script you were talking about into a recipe generator that also deals with this by querying the licenses of all the dependencies (direct and indirect).

That was my rough plan, generate a recipe or have it generate  an
include that recipes pull in, there are some repeating patterns go
modules, so there is some re-use to be found.

I roughed out a process for it to work with k3s, and have a working
updated recipe that creates a vendor/ directory and doesn't touch the
network during the actual build.

There's definitely efficiencies to be found, as the first fetch is
quite long, and there's some i/o required as the fetches secondarily
shuffled into place that go expects in a vendor directory.

I'm trying to complete a second recipe with the generated SRC_URI
entries now (nerdctl) and I ran into an issue with the script where
some repeated fetches were breaking the vendor directory creation. I
need to spend time with that on Thursday, but after I sort that out,
I can remove the curse words from the script and do a bit of cleanup.
There's plenty of bugs, and alternate ways things can operate (maybe
some of the packaged go modules versus git, etc, etc), but since those
choices don't required bitbake/fetcher or other core changes, we have
a bit of time to iterate on a workable approach.

Bruce

>
> That being said, how can I help? It seems that there is an existing WIP state on this. Can I take something from it? Maybe help with cleaning up the script?
> --
> Andrei Gherzan
> gpg: rsa4096/D4D94F67AD0E9640



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Andrei Gherzan]

On Thu, 3 Mar 2022, at 03:34, Bruce Ashfield wrote:
> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>>
>>
>> Mar 1, 2022 20:15:52 Bruce Ashfield <bruce.ashfield@gmail.com>:
>>
>> > On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
>> >>
>> >> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>> >>
>> >> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>> >>
>> >>
>> >> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>> >>
>> >> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>> >> lists.openembedded.org
>> >> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>> >>
>> >>
>> >> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>> >>
>> >>
>> >> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>> >>
>> >> Compile pulls in the go.mod list requiring network. Without this, do
>> >> compile would fail with a similar error to the following:
>> >>
>> >> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>> >>
>> >> This is something that needs to be carried in your own layers, IMHO it
>> >> isn't appropriate for core.
>> >>
>> >> It isn't about the fetching, it is the entire gap in functionality
>> >> that we are missing if go starts fetching dependencies during compile.
>> >>
>> >> A further thought is that if this is for go.mod issues, there is the
>> >> go-mod.bbclass.
>> >>
>> >> Perhaps enabling it in that class and doing a bbwarn about go fetching
>> >> dependencies would be appropriate ?
>> >>
>> >> Otherwise, someone may not know that this is happening and that a no
>> >> network configuration has no chance of working.
>> >>
>> >> I reckon that is reasonable. I'll personally go down the recipe level to workaround this change but understanding and agreeing with the reasoning behind this change, I want to invest a bit into trying to find a proper solution in the core. Bruce, I know you invested a fair amount of time into this already. Would you be willing to sync up and see how we can work together in tackling this?
>> >>
>> >> Definitely, more ideas are good. In fact, I think there are probably
>> >> several approaches that can co-exist, depending on what a
>> >> recipe/developer needs.
>> >>
>> >> I'm in the Eastern time zone here, and will try and grab folks on IRC
>> >> to have a level set
>> >>
>> >> Bruce
>> >>
>> >> Added Zyga to CC as he is also interested in this as part of his go development activities.
>> >>
>> >> Thanks,
>> >> Andrei
>> >>
>> >>
>> >>
>> >> --
>> >> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> >> thee at its end
>> >> - "Use the force Harry" - Gandalf, Star Trek II
>> >>
>> >> The problem in allowing downloads during compile (e.g. by go) is, that it leads to non-reproducable builds. I'm currently facing the same issue and would like to have a reproducable go *offline* build.
>> >> I would like to propose two ideas to workaround the go-compile fetching issue:
>> >>
>> >> First:
>> >> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies into go projects 'vendor' folder. This forces go to compile offline. However, one have to generate the 'modules.txt' file in the vendor folder 'manually' during unpack. This is error prone, as there is no official documentation how this format should look like. Anyway, I've tried this approach and it works for me.
>> >>
>> >> Second:
>> >> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher) and unpack the dependencies into a local (workdir) go-path. This seemed a good solution for me as the go-path is well defined. But for some reason 'go'  fetches the zip-files during compile into it's download-cache AGAIN, even if the source is already unpacked in the go-path. I'll assume this is required to verify the source files integrity?! With this approach one have to adapt 'go' to suppress this download behaviour.
>> >>
>> >
>> > I've been doing offline builds using a constructed vendor/ directory
>> > and generated modules.txt.
>> >
>> > The only difference between what I have working and what you are
>> > suggesting (type 1), is that I've gone directly to the sources and
>> > constructed the vendor directory using the OE git fetcher. That allows
>> > all functionality to continue to work that is part of OEcore, and the
>> > build to continue.  Switching out the git fetches for tarballs would
>> > be possible, I just wasn't sure how to use the proxied modules (and I
>> > wanted the history for debug).
>> >
>> > I've never had any issues with the modules.txt, as I generate it at
>> > the same time as the git fetch lines for the SRC_URI. I've also not
>> > been using information from the go.mod directly from go.proxy.org, it
>> > is information I've generated from a clone of the project and dumped
>> > via go mod. There's likely improvements I can do there, but with what
>> > I'm doing, I'm going directly to the source of the projects and doing
>> > clones, which keeps everything clear of the go infrastructure.
>> >
>> > I have a utility that I'm still cleaning up that generates the SRC_URI
>> > lines, as well as the modules.txt, when I resolve a few nagging
>> > issues, I'll make the WIP scripts available.
>> >
>> > Other projects (BSD, etc), have been doing different sorts of
>> > constructed vendor directories, but they are similar in approach.
>> >
>> > For the short term (i.e. the upcoming release), that is pretty much
>> > all we can do. There isn't enough time to implement a new go fetcher
>> > backend for bitbake.
>> >
>> > In the end, how we fetch and place the dependencies is a transport, so
>> > whether or not we fetch them ourselves, or let go do it, that part is
>> > largely the same.
>> >
>> > For now (short term), I favour vendor/, as it is workable, but not
>> > perfect. It isn't exactly efficient or pretty, but at least it seems
>> > to produce correct output, and allows all of the project capabilities
>> > to work. And of course, the approach will continue to work regardless
>> > of development on other go.mod elements.
>>
>> After reflecting on this for a while I reckon this is the fastest way forward while addressing the reproducibility issue. I'm wondering what we can do in terms of compliance? Maybe we can turn the script you were talking about into a recipe generator that also deals with this by querying the licenses of all the dependencies (direct and indirect).
>
> That was my rough plan, generate a recipe or have it generate  an
> include that recipes pull in, there are some repeating patterns go
> modules, so there is some re-use to be found.
>
> I roughed out a process for it to work with k3s, and have a working
> updated recipe that creates a vendor/ directory and doesn't touch the
> network during the actual build.
>
> There's definitely efficiencies to be found, as the first fetch is
> quite long, and there's some i/o required as the fetches secondarily
> shuffled into place that go expects in a vendor directory.
>
> I'm trying to complete a second recipe with the generated SRC_URI
> entries now (nerdctl) and I ran into an issue with the script where
> some repeated fetches were breaking the vendor directory creation. I
> need to spend time with that on Thursday, but after I sort that out,
> I can remove the curse words from the script and do a bit of cleanup.
> There's plenty of bugs, and alternate ways things can operate (maybe
> some of the packaged go modules versus git, etc, etc), but since those
> choices don't required bitbake/fetcher or other core changes, we have
> a bit of time to iterate on a workable approach.

That sounds good. Keep me in the loop and once you have a patch set pushed, I try it out and squeeze some of the bugs you'll leave in intentionally :)

Cheers,
Andrei

Re: [PATCH 1/1] go.bbclass: Allow network in do_compile

[lukas.funke]

[-- Attachment #1: Type: text/plain, Size: 8804 bytes --]

On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote:

> 
> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan <andrei@gherzan.com> wrote:
> 
>> 
>> Mar 1, 2022 20:15:52 Bruce Ashfield <bruce.ashfield@gmail.com>:
>> 
>> 
>>> On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
>>> 
>>>> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>>>> 
>>>> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>>>> 
>>>> 
>>>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>>>> 
>>>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>>>> lists.openembedded.org
>>>> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>>>> 
>>>> 
>>>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>>>> 
>>>> 
>>>> 
>>>> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>>>> 
>>>> Compile pulls in the go.mod list requiring network. Without this, do
>>>> compile would fail with a similar error to the following:
>>>> 
>>>> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>>>> 
>>>> This is something that needs to be carried in your own layers, IMHO it
>>>> isn't appropriate for core.
>>>> 
>>>> It isn't about the fetching, it is the entire gap in functionality
>>>> that we are missing if go starts fetching dependencies during compile.
>>>> 
>>>> A further thought is that if this is for go.mod issues, there is the
>>>> go-mod.bbclass.
>>>> 
>>>> Perhaps enabling it in that class and doing a bbwarn about go fetching
>>>> dependencies would be appropriate ?
>>>> 
>>>> Otherwise, someone may not know that this is happening and that a no
>>>> network configuration has no chance of working.
>>>> 
>>>> I reckon that is reasonable. I'll personally go down the recipe level to
>>>> workaround this change but understanding and agreeing with the reasoning
>>>> behind this change, I want to invest a bit into trying to find a proper
>>>> solution in the core. Bruce, I know you invested a fair amount of time
>>>> into this already. Would you be willing to sync up and see how we can work
>>>> together in tackling this?
>>>> 
>>>> Definitely, more ideas are good. In fact, I think there are probably
>>>> several approaches that can co-exist, depending on what a
>>>> recipe/developer needs.
>>>> 
>>>> I'm in the Eastern time zone here, and will try and grab folks on IRC
>>>> to have a level set
>>>> 
>>>> Bruce
>>>> 
>>>> Added Zyga to CC as he is also interested in this as part of his go
>>>> development activities.
>>>> 
>>>> Thanks,
>>>> Andrei
>>>> 
>>>> 
>>>> 
>>>> --
>>>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>>>> thee at its end
>>>> - "Use the force Harry" - Gandalf, Star Trek II
>>>> 
>>>> The problem in allowing downloads during compile (e.g. by go) is, that it
>>>> leads to non-reproducable builds. I'm currently facing the same issue and
>>>> would like to have a reproducable go *offline* build.
>>>> I would like to propose two ideas to workaround the go-compile fetching
>>>> issue:
>>>> 
>>>> First:
>>>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>>>> writing a seperate go fetcher or a wget-fetcher) and unpack the
>>>> dependencies into go projects 'vendor' folder. This forces go to compile
>>>> offline. However, one have to generate the 'modules.txt' file in the
>>>> vendor folder 'manually' during unpack. This is error prone, as there is
>>>> no official documentation how this format should look like. Anyway, I've
>>>> tried this approach and it works for me.
>>>> 
>>>> Second:
>>>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>>>> writing a seperate go fetcher) and unpack the dependencies into a local
>>>> (workdir) go-path. This seemed a good solution for me as the go-path is
>>>> well defined. But for some reason 'go' fetches the zip-files during
>>>> compile into it's download-cache AGAIN, even if the source is already
>>>> unpacked in the go-path. I'll assume this is required to verify the source
>>>> files integrity?! With this approach one have to adapt 'go' to suppress
>>>> this download behaviour.
>>> 
>>> I've been doing offline builds using a constructed vendor/ directory
>>> and generated modules.txt.
>>> 
>>> The only difference between what I have working and what you are
>>> suggesting (type 1), is that I've gone directly to the sources and
>>> constructed the vendor directory using the OE git fetcher. That allows
>>> all functionality to continue to work that is part of OEcore, and the
>>> build to continue. Switching out the git fetches for tarballs would
>>> be possible, I just wasn't sure how to use the proxied modules (and I
>>> wanted the history for debug).
>>> 
>>> I've never had any issues with the modules.txt, as I generate it at
>>> the same time as the git fetch lines for the SRC_URI. I've also not
>>> been using information from the go.mod directly from go.proxy.org, it
>>> is information I've generated from a clone of the project and dumped
>>> via go mod. There's likely improvements I can do there, but with what
>>> I'm doing, I'm going directly to the source of the projects and doing
>>> clones, which keeps everything clear of the go infrastructure.
>>> 
>>> I have a utility that I'm still cleaning up that generates the SRC_URI
>>> lines, as well as the modules.txt, when I resolve a few nagging
>>> issues, I'll make the WIP scripts available.
>>> 
>>> Other projects (BSD, etc), have been doing different sorts of
>>> constructed vendor directories, but they are similar in approach.
>>> 
>>> For the short term (i.e. the upcoming release), that is pretty much
>>> all we can do. There isn't enough time to implement a new go fetcher
>>> backend for bitbake.
>>> 
>>> In the end, how we fetch and place the dependencies is a transport, so
>>> whether or not we fetch them ourselves, or let go do it, that part is
>>> largely the same.
>>> 
>>> For now (short term), I favour vendor/, as it is workable, but not
>>> perfect. It isn't exactly efficient or pretty, but at least it seems
>>> to produce correct output, and allows all of the project capabilities
>>> to work. And of course, the approach will continue to work regardless
>>> of development on other go.mod elements.
>> 
>> After reflecting on this for a while I reckon this is the fastest way
>> forward while addressing the reproducibility issue. I'm wondering what we
>> can do in terms of compliance? Maybe we can turn the script you were
>> talking about into a recipe generator that also deals with this by
>> querying the licenses of all the dependencies (direct and indirect).
> 
> That was my rough plan, generate a recipe or have it generate an
> include that recipes pull in, there are some repeating patterns go
> modules, so there is some re-use to be found.
> 
> I roughed out a process for it to work with k3s, and have a working
> updated recipe that creates a vendor/ directory and doesn't touch the
> network during the actual build.
> 
> There's definitely efficiencies to be found, as the first fetch is
> quite long, and there's some i/o required as the fetches secondarily
> shuffled into place that go expects in a vendor directory.
> 
> I'm trying to complete a second recipe with the generated SRC_URI
> entries now (nerdctl) and I ran into an issue with the script where
> some repeated fetches were breaking the vendor directory creation. I
> need to spend time with that on Thursday, but after I sort that out,
> I can remove the curse words from the script and do a bit of cleanup.
> There's plenty of bugs, and alternate ways things can operate (maybe
> some of the packaged go modules versus git, etc, etc), but since those
> choices don't required bitbake/fetcher or other core changes, we have
> a bit of time to iterate on a workable approach.
> 
> Bruce

Bruce, I'm looking forward to review your approach. My biggest concern in fetching the imports from source via git is, that an 'import' my not necessarily relate to a git repository. 'go' supports multiple backends (e.g. hg, svn, etc.). That said, an import-path cannot be transformed to a git SRC_URI in a 1:1 manner. That's why I ended up downloading the modules from golang-proxy.

Lukas

> 
> 
> 
>> That being said, how can I help? It seems that there is an existing WIP
>> state on this. Can I take something from it? Maybe help with cleaning up
>> the script?
>> --
>> Andrei Gherzan
>> gpg: rsa4096/D4D94F67AD0E9640
> 
> 
> 
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II

[-- Attachment #2: Type: text/html, Size: 8794 bytes --]

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Bruce Ashfield]

On Thu, Mar 3, 2022 at 10:13 AM <lukas.funke@weidmueller.com> wrote:
>
> On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote:
>
> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> Mar 1, 2022 20:15:52 Bruce Ashfield <bruce.ashfield@gmail.com>:
>
> On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
>
> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>
> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>
> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
> lists.openembedded.org
> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>
>
> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>
> Compile pulls in the go.mod list requiring network. Without this, do
> compile would fail with a similar error to the following:
>
> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>
> This is something that needs to be carried in your own layers, IMHO it
> isn't appropriate for core.
>
> It isn't about the fetching, it is the entire gap in functionality
> that we are missing if go starts fetching dependencies during compile.
>
> A further thought is that if this is for go.mod issues, there is the
> go-mod.bbclass.
>
> Perhaps enabling it in that class and doing a bbwarn about go fetching
> dependencies would be appropriate ?
>
> Otherwise, someone may not know that this is happening and that a no
> network configuration has no chance of working.
>
> I reckon that is reasonable. I'll personally go down the recipe level to workaround this change but understanding and agreeing with the reasoning behind this change, I want to invest a bit into trying to find a proper solution in the core. Bruce, I know you invested a fair amount of time into this already. Would you be willing to sync up and see how we can work together in tackling this?
>
> Definitely, more ideas are good. In fact, I think there are probably
> several approaches that can co-exist, depending on what a
> recipe/developer needs.
>
> I'm in the Eastern time zone here, and will try and grab folks on IRC
> to have a level set
>
> Bruce
>
> Added Zyga to CC as he is also interested in this as part of his go development activities.
>
> Thanks,
> Andrei
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> The problem in allowing downloads during compile (e.g. by go) is, that it leads to non-reproducable builds. I'm currently facing the same issue and would like to have a reproducable go *offline* build.
> I would like to propose two ideas to workaround the go-compile fetching issue:
>
> First:
> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies into go projects 'vendor' folder. This forces go to compile offline. However, one have to generate the 'modules.txt' file in the vendor folder 'manually' during unpack. This is error prone, as there is no official documentation how this format should look like. Anyway, I've tried this approach and it works for me.
>
> Second:
> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher) and unpack the dependencies into a local (workdir) go-path. This seemed a good solution for me as the go-path is well defined. But for some reason 'go' fetches the zip-files during compile into it's download-cache AGAIN, even if the source is already unpacked in the go-path. I'll assume this is required to verify the source files integrity?! With this approach one have to adapt 'go' to suppress this download behaviour.
>
> I've been doing offline builds using a constructed vendor/ directory
> and generated modules.txt.
>
> The only difference between what I have working and what you are
> suggesting (type 1), is that I've gone directly to the sources and
> constructed the vendor directory using the OE git fetcher. That allows
> all functionality to continue to work that is part of OEcore, and the
> build to continue. Switching out the git fetches for tarballs would
> be possible, I just wasn't sure how to use the proxied modules (and I
> wanted the history for debug).
>
> I've never had any issues with the modules.txt, as I generate it at
> the same time as the git fetch lines for the SRC_URI. I've also not
> been using information from the go.mod directly from go.proxy.org, it
> is information I've generated from a clone of the project and dumped
> via go mod. There's likely improvements I can do there, but with what
> I'm doing, I'm going directly to the source of the projects and doing
> clones, which keeps everything clear of the go infrastructure.
>
> I have a utility that I'm still cleaning up that generates the SRC_URI
> lines, as well as the modules.txt, when I resolve a few nagging
> issues, I'll make the WIP scripts available.
>
> Other projects (BSD, etc), have been doing different sorts of
> constructed vendor directories, but they are similar in approach.
>
> For the short term (i.e. the upcoming release), that is pretty much
> all we can do. There isn't enough time to implement a new go fetcher
> backend for bitbake.
>
> In the end, how we fetch and place the dependencies is a transport, so
> whether or not we fetch them ourselves, or let go do it, that part is
> largely the same.
>
> For now (short term), I favour vendor/, as it is workable, but not
> perfect. It isn't exactly efficient or pretty, but at least it seems
> to produce correct output, and allows all of the project capabilities
> to work. And of course, the approach will continue to work regardless
> of development on other go.mod elements.
>
> After reflecting on this for a while I reckon this is the fastest way forward while addressing the reproducibility issue. I'm wondering what we can do in terms of compliance? Maybe we can turn the script you were talking about into a recipe generator that also deals with this by querying the licenses of all the dependencies (direct and indirect).
>
> That was my rough plan, generate a recipe or have it generate an
> include that recipes pull in, there are some repeating patterns go
> modules, so there is some re-use to be found.
>
> I roughed out a process for it to work with k3s, and have a working
> updated recipe that creates a vendor/ directory and doesn't touch the
> network during the actual build.
>
> There's definitely efficiencies to be found, as the first fetch is
> quite long, and there's some i/o required as the fetches secondarily
> shuffled into place that go expects in a vendor directory.
>
> I'm trying to complete a second recipe with the generated SRC_URI
> entries now (nerdctl) and I ran into an issue with the script where
> some repeated fetches were breaking the vendor directory creation. I
> need to spend time with that on Thursday, but after I sort that out,
> I can remove the curse words from the script and do a bit of cleanup.
> There's plenty of bugs, and alternate ways things can operate (maybe
> some of the packaged go modules versus git, etc, etc), but since those
> choices don't required bitbake/fetcher or other core changes, we have
> a bit of time to iterate on a workable approach.
>
> Bruce
>
> Bruce, I'm looking forward to review your approach. My biggest concern in fetching the imports from source via git is, that an 'import' my not necessarily relate to a git repository. 'go' supports multiple backends (e.g. hg, svn, etc.). That said, an import-path cannot be transformed to a git SRC_URI in a 1:1 manner. That's why I ended up downloading the modules from golang-proxy.
>

I just used git as an example. Any supported OE fetcher can be used. I
haven't run into any source bases that can't be resolved to git so
far, but the generation of other SRC_URI entries is relatively
trivial. Since this is static information and part of the recipe, it
can all be sorted out ahead after the original generation of the
source locations.

Experience with supporting some of the larger go applications has
shown me that the source has made it easier for hot fixes to address
CVE issues. We can easily bump an individual SRCREV or bring in a
patch. It's solvable no matter what the approach, that's just how
we've solved it with the source repos.

I was browsing around the proxy docs, and the API to get modules was
clear to me, do you have a link to an example or a document that
describes it ?

Bruce

> Lukas
>
>
> That being said, how can I help? It seems that there is an existing WIP state on this. Can I take something from it? Maybe help with cleaning up the script?
> --
> Andrei Gherzan
> gpg: rsa4096/D4D94F67AD0E9640
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#162658): https://lists.openembedded.org/g/openembedded-core/message/162658
> Mute This Topic: https://lists.openembedded.org/mt/89464905/1050810
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH 1/1] go.bbclass: Allow network in do_compile

[lukas.funke]

[-- Attachment #1: Type: text/plain, Size: 10231 bytes --]

On Thu, Mar 3, 2022 at 05:46 PM, Bruce Ashfield wrote:

> 
> On Thu, Mar 3, 2022 at 10:13 AM <lukas.funke@weidmueller.com> wrote:
> 
>> On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote:
>> 
>> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>> 
>> 
>> Mar 1, 2022 20:15:52 Bruce Ashfield <bruce.ashfield@gmail.com>:
>> 
>> On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
>> 
>> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>> 
>> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>> 
>> 
>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>> 
>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>> lists.openembedded.org
>> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>> 
>> 
>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>> 
>> 
>> 
>> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>> 
>> Compile pulls in the go.mod list requiring network. Without this, do
>> compile would fail with a similar error to the following:
>> 
>> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>> 
>> This is something that needs to be carried in your own layers, IMHO it
>> isn't appropriate for core.
>> 
>> It isn't about the fetching, it is the entire gap in functionality
>> that we are missing if go starts fetching dependencies during compile.
>> 
>> A further thought is that if this is for go.mod issues, there is the
>> go-mod.bbclass.
>> 
>> Perhaps enabling it in that class and doing a bbwarn about go fetching
>> dependencies would be appropriate ?
>> 
>> Otherwise, someone may not know that this is happening and that a no
>> network configuration has no chance of working.
>> 
>> I reckon that is reasonable. I'll personally go down the recipe level to
>> workaround this change but understanding and agreeing with the reasoning
>> behind this change, I want to invest a bit into trying to find a proper
>> solution in the core. Bruce, I know you invested a fair amount of time
>> into this already. Would you be willing to sync up and see how we can work
>> together in tackling this?
>> 
>> Definitely, more ideas are good. In fact, I think there are probably
>> several approaches that can co-exist, depending on what a
>> recipe/developer needs.
>> 
>> I'm in the Eastern time zone here, and will try and grab folks on IRC
>> to have a level set
>> 
>> Bruce
>> 
>> Added Zyga to CC as he is also interested in this as part of his go
>> development activities.
>> 
>> Thanks,
>> Andrei
>> 
>> 
>> 
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>> 
>> The problem in allowing downloads during compile (e.g. by go) is, that it
>> leads to non-reproducable builds. I'm currently facing the same issue and
>> would like to have a reproducable go *offline* build.
>> I would like to propose two ideas to workaround the go-compile fetching
>> issue:
>> 
>> First:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>> writing a seperate go fetcher or a wget-fetcher) and unpack the
>> dependencies into go projects 'vendor' folder. This forces go to compile
>> offline. However, one have to generate the 'modules.txt' file in the
>> vendor folder 'manually' during unpack. This is error prone, as there is
>> no official documentation how this format should look like. Anyway, I've
>> tried this approach and it works for me.
>> 
>> Second:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>> writing a seperate go fetcher) and unpack the dependencies into a local
>> (workdir) go-path. This seemed a good solution for me as the go-path is
>> well defined. But for some reason 'go' fetches the zip-files during
>> compile into it's download-cache AGAIN, even if the source is already
>> unpacked in the go-path. I'll assume this is required to verify the source
>> files integrity?! With this approach one have to adapt 'go' to suppress
>> this download behaviour.
>> 
>> I've been doing offline builds using a constructed vendor/ directory
>> and generated modules.txt.
>> 
>> The only difference between what I have working and what you are
>> suggesting (type 1), is that I've gone directly to the sources and
>> constructed the vendor directory using the OE git fetcher. That allows
>> all functionality to continue to work that is part of OEcore, and the
>> build to continue. Switching out the git fetches for tarballs would
>> be possible, I just wasn't sure how to use the proxied modules (and I
>> wanted the history for debug).
>> 
>> I've never had any issues with the modules.txt, as I generate it at
>> the same time as the git fetch lines for the SRC_URI. I've also not
>> been using information from the go.mod directly from go.proxy.org, it
>> is information I've generated from a clone of the project and dumped
>> via go mod. There's likely improvements I can do there, but with what
>> I'm doing, I'm going directly to the source of the projects and doing
>> clones, which keeps everything clear of the go infrastructure.
>> 
>> I have a utility that I'm still cleaning up that generates the SRC_URI
>> lines, as well as the modules.txt, when I resolve a few nagging
>> issues, I'll make the WIP scripts available.
>> 
>> Other projects (BSD, etc), have been doing different sorts of
>> constructed vendor directories, but they are similar in approach.
>> 
>> For the short term (i.e. the upcoming release), that is pretty much
>> all we can do. There isn't enough time to implement a new go fetcher
>> backend for bitbake.
>> 
>> In the end, how we fetch and place the dependencies is a transport, so
>> whether or not we fetch them ourselves, or let go do it, that part is
>> largely the same.
>> 
>> For now (short term), I favour vendor/, as it is workable, but not
>> perfect. It isn't exactly efficient or pretty, but at least it seems
>> to produce correct output, and allows all of the project capabilities
>> to work. And of course, the approach will continue to work regardless
>> of development on other go.mod elements.
>> 
>> After reflecting on this for a while I reckon this is the fastest way
>> forward while addressing the reproducibility issue. I'm wondering what we
>> can do in terms of compliance? Maybe we can turn the script you were
>> talking about into a recipe generator that also deals with this by
>> querying the licenses of all the dependencies (direct and indirect).
>> 
>> That was my rough plan, generate a recipe or have it generate an
>> include that recipes pull in, there are some repeating patterns go
>> modules, so there is some re-use to be found.
>> 
>> I roughed out a process for it to work with k3s, and have a working
>> updated recipe that creates a vendor/ directory and doesn't touch the
>> network during the actual build.
>> 
>> There's definitely efficiencies to be found, as the first fetch is
>> quite long, and there's some i/o required as the fetches secondarily
>> shuffled into place that go expects in a vendor directory.
>> 
>> I'm trying to complete a second recipe with the generated SRC_URI
>> entries now (nerdctl) and I ran into an issue with the script where
>> some repeated fetches were breaking the vendor directory creation. I
>> need to spend time with that on Thursday, but after I sort that out,
>> I can remove the curse words from the script and do a bit of cleanup.
>> There's plenty of bugs, and alternate ways things can operate (maybe
>> some of the packaged go modules versus git, etc, etc), but since those
>> choices don't required bitbake/fetcher or other core changes, we have
>> a bit of time to iterate on a workable approach.
>> 
>> Bruce
>> 
>> Bruce, I'm looking forward to review your approach. My biggest concern in
>> fetching the imports from source via git is, that an 'import' my not
>> necessarily relate to a git repository. 'go' supports multiple backends
>> (e.g. hg, svn, etc.). That said, an import-path cannot be transformed to a
>> git SRC_URI in a 1:1 manner. That's why I ended up downloading the modules
>> from golang-proxy.
> 
> I just used git as an example. Any supported OE fetcher can be used. I
> haven't run into any source bases that can't be resolved to git so
> far, but the generation of other SRC_URI entries is relatively
> trivial. Since this is static information and part of the recipe, it
> can all be sorted out ahead after the original generation of the
> source locations.
> 
> Experience with supporting some of the larger go applications has
> shown me that the source has made it easier for hot fixes to address
> CVE issues. We can easily bump an individual SRCREV or bring in a
> patch. It's solvable no matter what the approach, that's just how
> we've solved it with the source repos.
> 
> I was browsing around the proxy docs, and the API to get modules was
> clear to me, do you have a link to an example or a document that
> describes it ?

Bruce, sorry for the late reply. The only documentation I found regarding the mapping 'import-path' <> 'src-uri' is this one: Go Modules Reference - The Go Programming Language ( https://go.dev/ref/mod#vcs-find )
The mapping seems to be clear. That said, I would agree that it is possible to download the sources directly from git (or any other vcs) and unpack it to the vendor folder.

Lukas

> 
> 
> Bruce
> 
> 
>> Lukas
>> 
>> 
>> That being said, how can I help? It seems that there is an existing WIP
>> state on this. Can I take something from it? Maybe help with cleaning up
>> the script?
>> --
>> Andrei Gherzan
>> gpg: rsa4096/D4D94F67AD0E9640
>> 
>> 
>> 
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>> 
>> 
>> 
> 
> 
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II

[-- Attachment #2: Type: text/html, Size: 10470 bytes --]

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

[Bruce Ashfield]

On Wed, Mar 9, 2022 at 3:10 AM <lukas.funke@weidmueller.com> wrote:
>
> On Thu, Mar 3, 2022 at 05:46 PM, Bruce Ashfield wrote:
>
> On Thu, Mar 3, 2022 at 10:13 AM <lukas.funke@weidmueller.com> wrote:
>
> On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote:
>
> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> Mar 1, 2022 20:15:52 Bruce Ashfield <bruce.ashfield@gmail.com>:
>
> On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
>
> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>
> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>
> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
> lists.openembedded.org
> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>
>
> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>
>
> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>
> Compile pulls in the go.mod list requiring network. Without this, do
> compile would fail with a similar error to the following:
>
> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>
> This is something that needs to be carried in your own layers, IMHO it
> isn't appropriate for core.
>
> It isn't about the fetching, it is the entire gap in functionality
> that we are missing if go starts fetching dependencies during compile.
>
> A further thought is that if this is for go.mod issues, there is the
> go-mod.bbclass.
>
> Perhaps enabling it in that class and doing a bbwarn about go fetching
> dependencies would be appropriate ?
>
> Otherwise, someone may not know that this is happening and that a no
> network configuration has no chance of working.
>
> I reckon that is reasonable. I'll personally go down the recipe level to workaround this change but understanding and agreeing with the reasoning behind this change, I want to invest a bit into trying to find a proper solution in the core. Bruce, I know you invested a fair amount of time into this already. Would you be willing to sync up and see how we can work together in tackling this?
>
> Definitely, more ideas are good. In fact, I think there are probably
> several approaches that can co-exist, depending on what a
> recipe/developer needs.
>
> I'm in the Eastern time zone here, and will try and grab folks on IRC
> to have a level set
>
> Bruce
>
> Added Zyga to CC as he is also interested in this as part of his go development activities.
>
> Thanks,
> Andrei
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> The problem in allowing downloads during compile (e.g. by go) is, that it leads to non-reproducable builds. I'm currently facing the same issue and would like to have a reproducable go *offline* build.
> I would like to propose two ideas to workaround the go-compile fetching issue:
>
> First:
> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies into go projects 'vendor' folder. This forces go to compile offline. However, one have to generate the 'modules.txt' file in the vendor folder 'manually' during unpack. This is error prone, as there is no official documentation how this format should look like. Anyway, I've tried this approach and it works for me.
>
> Second:
> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by writing a seperate go fetcher) and unpack the dependencies into a local (workdir) go-path. This seemed a good solution for me as the go-path is well defined. But for some reason 'go' fetches the zip-files during compile into it's download-cache AGAIN, even if the source is already unpacked in the go-path. I'll assume this is required to verify the source files integrity?! With this approach one have to adapt 'go' to suppress this download behaviour.
>
> I've been doing offline builds using a constructed vendor/ directory
> and generated modules.txt.
>
> The only difference between what I have working and what you are
> suggesting (type 1), is that I've gone directly to the sources and
> constructed the vendor directory using the OE git fetcher. That allows
> all functionality to continue to work that is part of OEcore, and the
> build to continue. Switching out the git fetches for tarballs would
> be possible, I just wasn't sure how to use the proxied modules (and I
> wanted the history for debug).
>
> I've never had any issues with the modules.txt, as I generate it at
> the same time as the git fetch lines for the SRC_URI. I've also not
> been using information from the go.mod directly from go.proxy.org, it
> is information I've generated from a clone of the project and dumped
> via go mod. There's likely improvements I can do there, but with what
> I'm doing, I'm going directly to the source of the projects and doing
> clones, which keeps everything clear of the go infrastructure.
>
> I have a utility that I'm still cleaning up that generates the SRC_URI
> lines, as well as the modules.txt, when I resolve a few nagging
> issues, I'll make the WIP scripts available.
>
> Other projects (BSD, etc), have been doing different sorts of
> constructed vendor directories, but they are similar in approach.
>
> For the short term (i.e. the upcoming release), that is pretty much
> all we can do. There isn't enough time to implement a new go fetcher
> backend for bitbake.
>
> In the end, how we fetch and place the dependencies is a transport, so
> whether or not we fetch them ourselves, or let go do it, that part is
> largely the same.
>
> For now (short term), I favour vendor/, as it is workable, but not
> perfect. It isn't exactly efficient or pretty, but at least it seems
> to produce correct output, and allows all of the project capabilities
> to work. And of course, the approach will continue to work regardless
> of development on other go.mod elements.
>
> After reflecting on this for a while I reckon this is the fastest way forward while addressing the reproducibility issue. I'm wondering what we can do in terms of compliance? Maybe we can turn the script you were talking about into a recipe generator that also deals with this by querying the licenses of all the dependencies (direct and indirect).
>
> That was my rough plan, generate a recipe or have it generate an
> include that recipes pull in, there are some repeating patterns go
> modules, so there is some re-use to be found.
>
> I roughed out a process for it to work with k3s, and have a working
> updated recipe that creates a vendor/ directory and doesn't touch the
> network during the actual build.
>
> There's definitely efficiencies to be found, as the first fetch is
> quite long, and there's some i/o required as the fetches secondarily
> shuffled into place that go expects in a vendor directory.
>
> I'm trying to complete a second recipe with the generated SRC_URI
> entries now (nerdctl) and I ran into an issue with the script where
> some repeated fetches were breaking the vendor directory creation. I
> need to spend time with that on Thursday, but after I sort that out,
> I can remove the curse words from the script and do a bit of cleanup.
> There's plenty of bugs, and alternate ways things can operate (maybe
> some of the packaged go modules versus git, etc, etc), but since those
> choices don't required bitbake/fetcher or other core changes, we have
> a bit of time to iterate on a workable approach.
>
> Bruce
>
> Bruce, I'm looking forward to review your approach. My biggest concern in fetching the imports from source via git is, that an 'import' my not necessarily relate to a git repository. 'go' supports multiple backends (e.g. hg, svn, etc.). That said, an import-path cannot be transformed to a git SRC_URI in a 1:1 manner. That's why I ended up downloading the modules from golang-proxy.
>
> I just used git as an example. Any supported OE fetcher can be used. I
> haven't run into any source bases that can't be resolved to git so
> far, but the generation of other SRC_URI entries is relatively
> trivial. Since this is static information and part of the recipe, it
> can all be sorted out ahead after the original generation of the
> source locations.
>
> Experience with supporting some of the larger go applications has
> shown me that the source has made it easier for hot fixes to address
> CVE issues. We can easily bump an individual SRCREV or bring in a
> patch. It's solvable no matter what the approach, that's just how
> we've solved it with the source repos.
>
> I was browsing around the proxy docs, and the API to get modules was
> clear to me, do you have a link to an example or a document that
> describes it ?
>
> Bruce, sorry for the late reply. The only documentation I found regarding the mapping 'import-path' <> 'src-uri' is this one: Go Modules Reference - The Go Programming Language
> The mapping seems to be clear. That said, I would agree that it is possible to download the sources directly from git (or any other vcs) and unpack it to the vendor folder.
>

Right, I had read that section, but actually overlooked the practical
aspect of using it to do a download.

I have a working prototype done at this point, but I've had to hand
edit the mod.dep/dependency file in a couple of situations when I
couldn't discover the source repo for the depdency (you can see the
admittedly nasty recipes on meta-virtualizations master-next branch).

In that situation, or in a scenario where the dependencies are really
large, there's a lot of I/O, etc, it would be nice to fallback to
pulling the module from the proxy (since it is lighter weight, but
doesn't give us development/patching/licensing flexibility that a
source fetch brings for 'free').

I'm putting in the fallback now, but I'm out of the office next week,
so time is short and nothing is "documented" yet.

If there's interest, it would be good to sync the week of March 21st,
and go over the WIP and see if there's gaps, etc, and plot a short
term path.

Thoughts ?

Bruce

> Lukas
>
>
> Bruce
>
> Lukas
>
>
> That being said, how can I help? It seems that there is an existing WIP state on this. Can I take something from it? Maybe help with cleaning up the script?
> --
> Andrei Gherzan
> gpg: rsa4096/D4D94F67AD0E9640
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#162951): https://lists.openembedded.org/g/openembedded-core/message/162951
> Mute This Topic: https://lists.openembedded.org/mt/89464905/1050810
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH 1/1] go.bbclass: Allow network in do_compile

[lukas.funke]

[-- Attachment #1: Type: text/plain, Size: 12230 bytes --]

On Wed, Mar 9, 2022 at 07:43 PM, Bruce Ashfield wrote:

> 
> On Wed, Mar 9, 2022 at 3:10 AM <lukas.funke@weidmueller.com> wrote:
> 
>> On Thu, Mar 3, 2022 at 05:46 PM, Bruce Ashfield wrote:
>> 
>> On Thu, Mar 3, 2022 at 10:13 AM <lukas.funke@weidmueller.com> wrote:
>> 
>> On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote:
>> 
>> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>> 
>> 
>> Mar 1, 2022 20:15:52 Bruce Ashfield <bruce.ashfield@gmail.com>:
>> 
>> On Tue, Mar 1, 2022 at 10:54 AM <lukas.funke@weidmueller.com> wrote:
>> 
>> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>> 
>> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <andrei@gherzan.com> wrote:
>> 
>> 
>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>> 
>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>> lists.openembedded.org
>> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>> 
>> 
>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <andrei@gherzan.com> wrote:
>> 
>> 
>> 
>> From: Andrei Gherzan <andrei.gherzan@huawei.com>
>> 
>> Compile pulls in the go.mod list requiring network. Without this, do
>> compile would fail with a similar error to the following:
>> 
>> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>> 
>> This is something that needs to be carried in your own layers, IMHO it
>> isn't appropriate for core.
>> 
>> It isn't about the fetching, it is the entire gap in functionality
>> that we are missing if go starts fetching dependencies during compile.
>> 
>> A further thought is that if this is for go.mod issues, there is the
>> go-mod.bbclass.
>> 
>> Perhaps enabling it in that class and doing a bbwarn about go fetching
>> dependencies would be appropriate ?
>> 
>> Otherwise, someone may not know that this is happening and that a no
>> network configuration has no chance of working.
>> 
>> I reckon that is reasonable. I'll personally go down the recipe level to
>> workaround this change but understanding and agreeing with the reasoning
>> behind this change, I want to invest a bit into trying to find a proper
>> solution in the core. Bruce, I know you invested a fair amount of time
>> into this already. Would you be willing to sync up and see how we can work
>> together in tackling this?
>> 
>> Definitely, more ideas are good. In fact, I think there are probably
>> several approaches that can co-exist, depending on what a
>> recipe/developer needs.
>> 
>> I'm in the Eastern time zone here, and will try and grab folks on IRC
>> to have a level set
>> 
>> Bruce
>> 
>> Added Zyga to CC as he is also interested in this as part of his go
>> development activities.
>> 
>> Thanks,
>> Andrei
>> 
>> 
>> 
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>> 
>> The problem in allowing downloads during compile (e.g. by go) is, that it
>> leads to non-reproducable builds. I'm currently facing the same issue and
>> would like to have a reproducable go *offline* build.
>> I would like to propose two ideas to workaround the go-compile fetching
>> issue:
>> 
>> First:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>> writing a seperate go fetcher or a wget-fetcher) and unpack the
>> dependencies into go projects 'vendor' folder. This forces go to compile
>> offline. However, one have to generate the 'modules.txt' file in the
>> vendor folder 'manually' during unpack. This is error prone, as there is
>> no official documentation how this format should look like. Anyway, I've
>> tried this approach and it works for me.
>> 
>> Second:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>> writing a seperate go fetcher) and unpack the dependencies into a local
>> (workdir) go-path. This seemed a good solution for me as the go-path is
>> well defined. But for some reason 'go' fetches the zip-files during
>> compile into it's download-cache AGAIN, even if the source is already
>> unpacked in the go-path. I'll assume this is required to verify the source
>> files integrity?! With this approach one have to adapt 'go' to suppress
>> this download behaviour.
>> 
>> I've been doing offline builds using a constructed vendor/ directory
>> and generated modules.txt.
>> 
>> The only difference between what I have working and what you are
>> suggesting (type 1), is that I've gone directly to the sources and
>> constructed the vendor directory using the OE git fetcher. That allows
>> all functionality to continue to work that is part of OEcore, and the
>> build to continue. Switching out the git fetches for tarballs would
>> be possible, I just wasn't sure how to use the proxied modules (and I
>> wanted the history for debug).
>> 
>> I've never had any issues with the modules.txt, as I generate it at
>> the same time as the git fetch lines for the SRC_URI. I've also not
>> been using information from the go.mod directly from go.proxy.org, it
>> is information I've generated from a clone of the project and dumped
>> via go mod. There's likely improvements I can do there, but with what
>> I'm doing, I'm going directly to the source of the projects and doing
>> clones, which keeps everything clear of the go infrastructure.
>> 
>> I have a utility that I'm still cleaning up that generates the SRC_URI
>> lines, as well as the modules.txt, when I resolve a few nagging
>> issues, I'll make the WIP scripts available.
>> 
>> Other projects (BSD, etc), have been doing different sorts of
>> constructed vendor directories, but they are similar in approach.
>> 
>> For the short term (i.e. the upcoming release), that is pretty much
>> all we can do. There isn't enough time to implement a new go fetcher
>> backend for bitbake.
>> 
>> In the end, how we fetch and place the dependencies is a transport, so
>> whether or not we fetch them ourselves, or let go do it, that part is
>> largely the same.
>> 
>> For now (short term), I favour vendor/, as it is workable, but not
>> perfect. It isn't exactly efficient or pretty, but at least it seems
>> to produce correct output, and allows all of the project capabilities
>> to work. And of course, the approach will continue to work regardless
>> of development on other go.mod elements.
>> 
>> After reflecting on this for a while I reckon this is the fastest way
>> forward while addressing the reproducibility issue. I'm wondering what we
>> can do in terms of compliance? Maybe we can turn the script you were
>> talking about into a recipe generator that also deals with this by
>> querying the licenses of all the dependencies (direct and indirect).
>> 
>> That was my rough plan, generate a recipe or have it generate an
>> include that recipes pull in, there are some repeating patterns go
>> modules, so there is some re-use to be found.
>> 
>> I roughed out a process for it to work with k3s, and have a working
>> updated recipe that creates a vendor/ directory and doesn't touch the
>> network during the actual build.
>> 
>> There's definitely efficiencies to be found, as the first fetch is
>> quite long, and there's some i/o required as the fetches secondarily
>> shuffled into place that go expects in a vendor directory.
>> 
>> I'm trying to complete a second recipe with the generated SRC_URI
>> entries now (nerdctl) and I ran into an issue with the script where
>> some repeated fetches were breaking the vendor directory creation. I
>> need to spend time with that on Thursday, but after I sort that out,
>> I can remove the curse words from the script and do a bit of cleanup.
>> There's plenty of bugs, and alternate ways things can operate (maybe
>> some of the packaged go modules versus git, etc, etc), but since those
>> choices don't required bitbake/fetcher or other core changes, we have
>> a bit of time to iterate on a workable approach.
>> 
>> Bruce
>> 
>> Bruce, I'm looking forward to review your approach. My biggest concern in
>> fetching the imports from source via git is, that an 'import' my not
>> necessarily relate to a git repository. 'go' supports multiple backends
>> (e.g. hg, svn, etc.). That said, an import-path cannot be transformed to a
>> git SRC_URI in a 1:1 manner. That's why I ended up downloading the modules
>> from golang-proxy.
>> 
>> I just used git as an example. Any supported OE fetcher can be used. I
>> haven't run into any source bases that can't be resolved to git so
>> far, but the generation of other SRC_URI entries is relatively
>> trivial. Since this is static information and part of the recipe, it
>> can all be sorted out ahead after the original generation of the
>> source locations.
>> 
>> Experience with supporting some of the larger go applications has
>> shown me that the source has made it easier for hot fixes to address
>> CVE issues. We can easily bump an individual SRCREV or bring in a
>> patch. It's solvable no matter what the approach, that's just how
>> we've solved it with the source repos.
>> 
>> I was browsing around the proxy docs, and the API to get modules was
>> clear to me, do you have a link to an example or a document that
>> describes it ?
>> 
>> Bruce, sorry for the late reply. The only documentation I found regarding
>> the mapping 'import-path' <> 'src-uri' is this one: Go Modules Reference -
>> The Go Programming Language
>> The mapping seems to be clear. That said, I would agree that it is
>> possible to download the sources directly from git (or any other vcs) and
>> unpack it to the vendor folder.
> 
> Right, I had read that section, but actually overlooked the practical
> aspect of using it to do a download.
> 
> I have a working prototype done at this point, but I've had to hand
> edit the mod.dep/dependency file in a couple of situations when I
> couldn't discover the source repo for the depdency (you can see the
> admittedly nasty recipes on meta-virtualizations master-next branch).
> 
> In that situation, or in a scenario where the dependencies are really
> large, there's a lot of I/O, etc, it would be nice to fallback to
> pulling the module from the proxy (since it is lighter weight, but
> doesn't give us development/patching/licensing flexibility that a
> source fetch brings for 'free').
> 
> I'm putting in the fallback now, but I'm out of the office next week,
> so time is short and nothing is "documented" yet.
> 
> If there's interest, it would be good to sync the week of March 21st,
> and go over the WIP and see if there's gaps, etc, and plot a short
> term path.
> 
> Thoughts ?

Regarding the source fetching / modulepath translation, this link might also be helpfull for you:

https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/vcs/vcs.go

"RepoRootForImportPath" implements some parts of the translation. For most modules there is a lookup-table with regex-entries that implement the actual translation. The go-documentation lacks some detail ;-)

Go implements the source fetching the other way around: first proxy then vcs. However, in OE your proposed solution might be more flexible regarding patching, security etc.

I'm looking forward to review your solution.

Lukas

> 
> 
> Bruce
> 
> 
>> Lukas
>> 
>> 
>> Bruce
>> 
>> Lukas
>> 
>> 
>> That being said, how can I help? It seems that there is an existing WIP
>> state on this. Can I take something from it? Maybe help with cleaning up
>> the script?
>> --
>> Andrei Gherzan
>> gpg: rsa4096/D4D94F67AD0E9640
>> 
>> 
>> 
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>> 
>> 
>> 
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>> 
>> 
>> 
> 
> 
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II

[-- Attachment #2: Type: text/html, Size: 12671 bytes --]