#golang: go fetches dependencies in compile phase

#golang: go fetches dependencies in compile phase

[juergen.landwehr]

[-- Attachment #1: Type: text/html, Size: 768 bytes --]

Re: [yocto] #golang: go fetches dependencies in compile phase

[Konrad Weihmann]

Well AFAIK your observation is correct, go-mod.bbclass doesn't work a in 
BB_NO_NETWORK-safe manner.
What you could do is to provide all the dependencies manually and add 
them via DEPENDS, which is grant the behavior expected.

The bad thing about go is that chances of circular dependencies of go 
modules are relatively high.

I've also seen a few approaches using vendoring to escape this, but I'm 
afraid none of the code is pretty well tested nor publicly available.

If you'd ask me, I would go the long painful road of providing each 
required go module as a recipe of it's own and inject it into the final 
recipe - this also makes it pretty visible what you're pulling into your 
build in terms of licensing a.s.o.

On 09.04.21 15:53, Juergen Landwehr wrote:
> Hi,
> we are developing an application in go and use the "go-mod" bb-class 
> from poky. This works fine.
> However, the problem is, that the dependencies in go.mod are now fetched 
> in the compile phase and not in the fetch phase.
> This is not allowed in our project setup and kind of contradicts the 
> Yocto paradigmn of having a fetch and a compile phase.
> Is there a way to avoid this or some other solution that we could use?
> Thanks,
> Jürgen

#golang: go fetches dependencies in compile phase

[Juergen Landwehr]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

Hello Konrad,

thanks for your reply.

It is interesting that you mention vendoring, because we tried this approach as well. We started to develop a "go-mod-vendor.bbclass", which retrieves the
application source code in "do_fetch_append" and then calls "go mod vendor".

The big problem was, that in that phase there is not yet a go executable available for the recipe, so we downloaded our own go version, which makes this approach a bit ugly.

But there is already a go version installed on the build host, right? In "poky/meta/recipes-devtools/go" I found a "go-native_1.14.bb" recipe.
So can we use this go exectuable somehow by adding a dependency (e.g. DEPENDS="go-native") and then use a path to the go exectuable that is guaranteed to work?

The other alternative (defining a recipe for each dependent go module) sounds really painful. Especially as we have quite a few dependencies, which have dependencies as well.
I see the point, that doing it that way makes it more visible and more Yocto like, but thinking about it gives me headaches already :)

Thanks again.

[-- Attachment #2: Type: text/html, Size: 1158 bytes --]

Re: [yocto] #golang: go fetches dependencies in compile phase

[Robert Berger]

On 09/04/2021 16:53, Juergen Landwehr wrote:
> Hi,
> we are developing an application in go and use the "go-mod" bb-class 
> from poky. This works fine.
> However, the problem is, that the dependencies in go.mod are now fetched 
> in the compile phase and not in the fetch phase.

Alexander (whom you cc'ed here as well) wrote a long time ago on the 
mailing list[1] about this stuff.

We have a couple of issues with those "modern" languages like go, which 
behave quite differently from good old Unix stuff. So the paradigm on 
which the whole Bitbake/OpenEmbedded/Yocto system was built does not 
quite apply for those.

Don't get me wrong, you can also do stupidities like git clone via make 
or cmake, but this is most likely not the way those tools should be used.

1) "Guaranteed __NOT__ reproducible builds"(C)

Go pulls in dependencies by itself, which might explain why it's in the 
compile phase and not it the fetch phase.

This leads to many interesting issues, like "guaranteed __NOT__ 
reproducible builds"(C). The problem is, that you don't have any 
influence on dependencies of dependencies. Someone changes somewhere a 
dependency on github and the party starts.

1.1) One sane solution to "please give me always the same input" is what 
Konrad already mentioned in combination with a local golang proxy.

2) Open Source License Compliance

If you pull in all those funny dependencies and also link things 
statically, like with go, Open Source License Compliance is getting very 
interesting.

People typically just use the license of the top level "main" entry 
point. In reality you would need to check all the dependency tree for 
licenses. You should also check if it's legally allowed to combine the 
licenses and in case it's allowed what are implications of GPLvx, 
LGPLv2, LGPLv3 without exceptions for your product.

[1] https://www.yoctoproject.org/pipermail/yocto/2017-March/035028.html

Regards,

Robert

#golang: go fetches dependencies in compile phase

[Juergen Landwehr]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

Hi Robert,

thanks for your thoughts.

I see your point and the last thing I want is "NOT reproducable builds".

But dependency management in go is not that arbitrary as it may seem. Dependencies and their version is stored in "go.mod". To ensure reproducable builds, hashes for each dependency and version are stored in "go.sum". Both files are in git and together with a local golang proxy, this should ensure reproducable builds, right?

To ensure that licences are valid and did not change over time, we developed a little tool, that goes through all dependencies and creates the following output, which we insert into our recipe:

LICENSE = "Apache-2.0 & MIT & MIT & BSD-2-Clause & BSD-3-Clause & ...

LIC_FILES_CHKSUM = " \
file://${S}/src/${GO_IMPORT}/vendor/github.com/coreos/go-oidc/LICENSE;md5=d2794c0df5b907fdace235a619d80314 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/go-playground/locales/LICENSE;md5=3ccbda375ee345400ad1da85ba522301 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/go-playground/universal-translator/LICENSE;md5=2e2b21ef8f61057977d27c727c84bef1 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/godbus/dbus/v5/LICENSE;md5=09042bd5c6c96a2b9e45ddf1bc517eed \
file://${S}/src/${GO_IMPORT}/vendor/github.com/golang/gddo/LICENSE;md5=4c728948788b1a02f33ae4e906546eef \
... This is a manual step and whenever dependencies change we have to create this list again and add it to the recipe, but it contains all the required license information and makes them visible in the recipe.

It might pollute the recipe a bit, but luckily we do not have thousands of dependencies like some npm projects. So I think it is still manageable. And is it much less work, than defining a recipe for each dependency.

So we would
* guarantee reproducable builds
* use the programming language (in our case golang) to handle dependency management
* ensure that licenses are visible and correct

I mean it is not perfect and still a compromise, but it should work without breaking reproducable builds or causing license issues?
What do you think?

Regards,
Jürgen

PS: Thanks for the link. I was not aware of this discussion (it is quite a bit to read:))

[-- Attachment #2: Type: text/html, Size: 2485 bytes --]

Re: [yocto] #golang: go fetches dependencies in compile phase

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

On Mon, 12 Apr 2021 at 13:47, Juergen Landwehr <juergen.landwehr@gmx.de>
wrote:

> But dependency management in go is not that arbitrary as it may seem.
> Dependencies and their version is stored in "go.mod". To ensure
> reproducable builds, hashes for each dependency and version are stored in
> "go.sum". Both files are in git and together with a local golang proxy,
> this should ensure reproducable builds, right?
>

Reproducibility means anyone can run a build at any point in the future
even if the upstream repositories are gone, so all inputs must be stored in
a local download cache, which is the other thing SRC_URI guarantees, in
addition to verifying integrity of the inputs.

Alex

[-- Attachment #2: Type: text/html, Size: 1042 bytes --]

#golang: go fetches dependencies in compile phase

[Juergen Landwehr]

[-- Attachment #1: Type: text/plain, Size: 563 bytes --]

Hi Alex,

OK, understood.

If the "local download cache path" is well-known (this is not by any chance $DL_DIR?), then we could create a tar from the vendor directory (which is created when you call "go mod vendor" and contains all the downloaded dependencies) and put this tar file into the download cache.

Before actually calling "go mod vendor", we would first check, if there is a tar file for the vendor directory in the download cache and if so simply unpack the tar.

Does this make sense, or am I too naive and this is just nonsense?

Jürgen

[-- Attachment #2: Type: text/html, Size: 609 bytes --]

Re: [yocto] #golang: go fetches dependencies in compile phase

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

I'd suggest you place that tarball into some artefact storage, and have it
listed in SRC_URI. Then the standard Yocto mechanism for fetching,
checksumming, caching and unpacking tarballs kicks in, so you only need to
make sure the tree is set up correctly after unpacking, maybe with some
simple post-processing.

Alex

On Mon, 12 Apr 2021 at 16:02, Juergen Landwehr <juergen.landwehr@gmx.de>
wrote:

> Hi Alex,
>
> OK, understood.
>
> If the "local download cache path" is well-known (this is not by any
> chance $DL_DIR?), then we could create a tar from the vendor directory
> (which is created when you call "go mod vendor" and contains all the
> downloaded dependencies) and put this tar file into the download cache.
>
> Before actually calling "go mod vendor", we would first check, if there is
> a tar file for the vendor directory in the download cache and if so simply
> unpack the tar.
>
> Does this make sense, or am I too naive and this is just nonsense?
>
> Jürgen
> 
>
>

[-- Attachment #2: Type: text/html, Size: 1360 bytes --]

Re: [yocto] #golang: go fetches dependencies in compile phase

[Robert Berger]

Hi,

My comments are inline.

On 12/04/2021 14:47, Juergen Landwehr wrote:
> Hi Robert,
> 
> thanks for your thoughts.
> 
> I see your point and the last thing I want is "NOT reproducable builds".
> 
> But dependency management in go is not that arbitrary as it may seem. 

... if everybody does what they are supposed to - which is not the case.

> Dependencies and their version is stored in "go.mod". To ensure 
> reproducable builds, hashes for each dependency and version are stored 
> in "go.sum". Both files are in git and together with a local golang 
> proxy, this should ensure reproducable builds, right?

This does not sound too bad. This would also mean, that if the outside 
repo dies you can still build and that you know what's on your proxy.

> 
> To ensure that licences are valid and did not change over time, we 
> developed a little tool, that goes through all dependencies and creates 
> the following output, which we insert into our recipe:
> 
> LICENSE = "Apache-2.0 & MIT & MIT & BSD-2-Clause & BSD-3-Clause & ...

Here you

1) Totally lost which License comes from which module and I hope 2 times 
MIT is just a typo.

Of course if you are really serious about licensing you should use a 
tool like fossology, upload all your sources there and inspect them.
You will be surprised.

There are a couple of tools out there which scan your sources and some 
which claim to do stuff with golang as well.

2) Do you check if the licenses are compatible?

MIT and Apache are compatible:

some googling:

"Both the Apache License and the MIT license are permissive, so 
incorporating MIT licensed code into your Apache licensed project is 
certainly allowed. Just be sure to attribute the original author for the 
parts your incorporated and include a copy of the MIT License terms, as 
required by the license."

Apache and BSD should also be OK:

some googling:

"In both of them you must:

     Include copyright

But in Apache, unlike BSD you must:

     Include License
     State Changes
     Include Notice
"

> 
> LIC_FILES_CHKSUM = " \
> file://${S}/src/${GO_IMPORT}/vendor/github.com/coreos/go-oidc/LICENSE;md5=d2794c0df5b907fdace235a619d80314 \
> file://${S}/src/${GO_IMPORT}/vendor/github.com/go-playground/locales/LICENSE;md5=3ccbda375ee345400ad1da85ba522301 \
> file://${S}/src/${GO_IMPORT}/vendor/github.com/go-playground/universal-translator/LICENSE;md5=2e2b21ef8f61057977d27c727c84bef1 \
> file://${S}/src/${GO_IMPORT}/vendor/github.com/godbus/dbus/v5/LICENSE;md5=09042bd5c6c96a2b9e45ddf1bc517eed \
> file://${S}/src/${GO_IMPORT}/vendor/github.com/golang/gddo/LICENSE;md5=4c728948788b1a02f33ae4e906546eef \
> ...
> 
> This is a manual step and whenever dependencies change we have to create 
> this list again and add it to the recipe, but it contains all the 
> required license information and makes them visible in the recipe.

really all?

You search through every single file of a go module and it's 
dependencies? Or just the license text, if there is one?

> 
> It might pollute the recipe a bit, but luckily we do not have thousands 
> of dependencies like some npm projects. So I think it is still 
> manageable. And is it much less work, than defining a recipe for each 
> dependency.
> 
> So we would
> * guarantee reproducable builds
> * use the programming language (in our case golang) to handle dependency 
> management
> * ensure that licenses are visible and correct
> 
> I mean it is not perfect and still a compromise, but it should work 
> without breaking reproducable builds or causing license issues?
> What do you think?
> 
> Regards,
> Jürgen
> 
> PS: Thanks for the link. I was not aware of this discussion (it is quite 
> a bit to read:))

Regards,

Robert

> 
> 
> 
>