[Buildroot] [PATCH v2 0/6] Fixes for go language support

[Buildroot] [PATCH v2 0/6] Fixes for go language support

[Geoff Levand]

Hi,

This series includes a number of fixup for the go language support.  All the
buildbot problems pointed out by Thomas [1] should be resolved by this series.

As requested, I pulled out the enhancment patches to post as a seperate
series [2].  Please consider all patches in this series to apply to master.

[1] http://lists.busybox.net/pipermail/buildroot/2016-April/159336.html
[2] https://github.com/glevand/buildroot--buildroot/tree/for-merge-go-enhancements

-Geoff

The following changes since commit 3f3342a581970977f28620f17391d8aaf2204fec:

  Update for 2016.05-rc2 (2016-05-17 16:11:24 +0200)

are available in the git repository at:

  git at github.com:glevand/buildroot--buildroot.git for-merge-go-fixes

for you to fetch changes up to ec83a5927025fc759d5bfc26ee01aabde3658a7c:

  package/go: Set file timestamp (2016-05-19 15:17:20 -0700)

----------------------------------------------------------------
Geoff Levand (6):
      package/go: Build host tools with host CC
      package/go: Add HOST_GO_TARGET_ENV
      package/flannel: Use HOST_GO_TARGET_ENV
      package/go: Add HOST_GO_CGO_ENABLED
      package/flannel: Add BR2_TOOLCHAIN_HAS_THREADS
      package/go: Set file timestamp

 package/flannel/Config.in  |  4 +++
 package/flannel/flannel.mk |  2 +-
 package/go/go.mk           | 66 ++++++++++++++++++++++++++++++++++++++++++----
 3 files changed, 66 insertions(+), 6 deletions(-)

-- 
2.5.0

[Buildroot] [PATCH v2 2/6] package/go: Add HOST_GO_TARGET_ENV

[Geoff Levand]

For the convenience of package makefiles define the new
make variables HOST_GO_TOOLDIR and HOST_GO_TARGET_ENV.

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
 package/go/go.mk | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/package/go/go.mk b/package/go/go.mk
index 5acd75f..5e4ee09 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -35,6 +35,15 @@ endif
 HOST_GO_DEPENDENCIES = host-go-bootstrap
 HOST_GO_ROOT = $(HOST_DIR)/usr/lib/go
 
+# For the convienience of target packages.
+HOST_GO_TOOLDIR = $(HOST_GO_ROOT)/pkg/tool/linux_$(GO_GOARCH)
+HOST_GO_TARGET_ENV = \
+	GOARCH=$(GO_GOARCH) \
+	GOROOT="$(HOST_GO_ROOT)" \
+	CC=$(TARGET_CC) \
+	CXX=$(TARGET_CXX) \
+	GOTOOLDIR="$(HOST_GO_TOOLDIR)"
+
 # The go build system doesn't have the notion of cross compiling, but just the
 # notion of architecture.  When the host and target architectures are different
 # it expects to be given a target cross compiler in CC_FOR_TARGET.  When the
-- 
2.5.0

[Buildroot] [PATCH v2 4/6] package/go: Add HOST_GO_CGO_ENABLED

[Geoff Levand]

The go compiler's cgo support uses threads.  If BR2_TOOLCHAIN_HAS_THREADS is
set, build in cgo support for any go programs that may need it.  Note that
any target package needing cgo support must include
'depends on BR2_TOOLCHAIN_HAS_THREADS' in its config file.

Fixes build errors like these:

  error: #warning requested reentrant code

  http://autobuild.buildroot.net/results/42a8d07101d8d954511d1c884ecb66e8d861899e

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
 package/go/go.mk | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/package/go/go.mk b/package/go/go.mk
index 5e4ee09..8565a86 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -44,6 +44,16 @@ HOST_GO_TARGET_ENV = \
 	CXX=$(TARGET_CXX) \
 	GOTOOLDIR="$(HOST_GO_TOOLDIR)"
 
+# The go compiler's cgo support uses threads.  If BR2_TOOLCHAIN_HAS_THREADS is
+# set, build in cgo support for any go programs that may need it.  Note that
+# any target package needing cgo support must include
+# 'depends on BR2_TOOLCHAIN_HAS_THREADS' in its config file.
+ifeq (BR2_TOOLCHAIN_HAS_THREADS,y)
+	HOST_GO_CGO_ENABLED = 1
+else
+	HOST_GO_CGO_ENABLED = 0
+endif
+
 # The go build system doesn't have the notion of cross compiling, but just the
 # notion of architecture.  When the host and target architectures are different
 # it expects to be given a target cross compiler in CC_FOR_TARGET.  When the
@@ -62,7 +72,7 @@ HOST_GO_MAKE_ENV = \
 	GOARCH=$(GO_GOARCH) \
 	$(if $(GO_GOARM),GOARM=$(GO_GOARM)) \
 	GOOS=linux \
-	CGO_ENABLED=1 \
+	CGO_ENABLED=$(HOST_GO_CGO_ENABLED) \
 	CC=$(HOSTCC_NOCCACHE)
 
 HOST_GO_TARGET_CC = \
-- 
2.5.0

[Buildroot] [PATCH v2 6/6] package/go: Set file timestamp

[Geoff Levand]

Set all file timestamps to prevent the go compiler from rebuilding any
built in packages when programs are built.

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
 package/go/go.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/package/go/go.mk b/package/go/go.mk
index 8565a86..a2babab 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -119,6 +119,10 @@ define HOST_GO_INSTALL_CMDS
 	# There is a known issue which requires the go sources to be installed
 	# https://golang.org/issue/2775
 	cp -a $(@D)/src $(HOST_GO_ROOT)/
+
+	# Set all file timestamps to prevent the go compiler from rebuilding any
+	# built in packages when programs are built.
+	find $(HOST_GO_ROOT) -type f -exec touch -r $(HOST_GO_TMP)/bin/go {} \;
 endef
 
 $(eval $(host-generic-package))
-- 
2.5.0

[Buildroot] [PATCH v2 1/6] package/go: Build host tools with host CC

[Geoff Levand]

The go build system doesn't have the notion of cross compiling, but just the
notion of architecture.  When the host and target architectures are different
it expects to be given a target cross compiler in CC_FOR_TARGET.  When the
architectures are the same it will use CC_FOR_TARGET for both host and target
compilation.  To work around this limitation, when the host and target
architectures are the same build a set of host binaries and tools with
CC_FOR_TARGET set to the host compiler.  Also, the go build system is not
compatible with ccache, so use HOSTCC_NOCCACHE.  See
https://github.com/golang/go/issues/11685.

Fixes build errors like these:

  host/usr/bin/go: No such file or directory

  http://autobuild.buildroot.net/results/6664189a6f3a815978e8d0a1d7ef408ca47e2874/

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
 package/go/go.mk | 41 +++++++++++++++++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 4 deletions(-)

diff --git a/package/go/go.mk b/package/go/go.mk
index a9e16dd..5acd75f 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -35,6 +35,16 @@ endif
 HOST_GO_DEPENDENCIES = host-go-bootstrap
 HOST_GO_ROOT = $(HOST_DIR)/usr/lib/go
 
+# The go build system doesn't have the notion of cross compiling, but just the
+# notion of architecture.  When the host and target architectures are different
+# it expects to be given a target cross compiler in CC_FOR_TARGET.  When the
+# architectures are the same it will use CC_FOR_TARGET for both host and target
+# compilation.  To work around this limitation, when the host and target
+# architectures are the same build a set of host binaries and tools with
+# CC_FOR_TARGET set to the host compiler.  Also, the go build system is not
+# compatible with ccache, so use HOSTCC_NOCCACHE.  See
+# https://github.com/golang/go/issues/11685.
+
 HOST_GO_MAKE_ENV = \
 	GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_ROOT) \
 	GOROOT_FINAL=$(HOST_GO_ROOT) \
@@ -44,16 +54,39 @@ HOST_GO_MAKE_ENV = \
 	$(if $(GO_GOARM),GOARM=$(GO_GOARM)) \
 	GOOS=linux \
 	CGO_ENABLED=1 \
+	CC=$(HOSTCC_NOCCACHE)
+
+HOST_GO_TARGET_CC = \
 	CC_FOR_TARGET=$(TARGET_CC) \
 	CXX_FOR_TARGET=$(TARGET_CXX)
 
+HOST_GO_HOST_CC = \
+	CC_FOR_TARGET=$(HOSTCC_NOCCACHE) \
+	CXX_FOR_TARGET=$(HOSTCC_NOCCACHE)
+
+HOST_GO_TMP = $(@D)/host-go-tmp
+
+ifneq ($(ARCH),$(HOSTARCH))
 define HOST_GO_BUILD_CMDS
-	cd $(@D)/src && $(HOST_GO_MAKE_ENV) ./make.bash
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
+	mkdir -p $(HOST_GO_TMP)
+	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
+	mv $(@D)/bin/ $(HOST_GO_TMP)/
 endef
+else
+define HOST_GO_BUILD_CMDS
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
+	mkdir -p $(HOST_GO_TMP)
+	cp -a $(@D)/pkg/tool $(HOST_GO_TMP)/
+	cp -a $(@D)/bin/ $(HOST_GO_TMP)/
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
+	cp -a $(@D)/pkg/tool $(HOST_GO_TMP)/
+endef
+endif
 
 define HOST_GO_INSTALL_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/bin/go $(HOST_GO_ROOT)/bin/go
-	$(INSTALL) -D -m 0755 $(@D)/bin/gofmt $(HOST_GO_ROOT)/bin/gofmt
+	$(INSTALL) -D -m 0755 $(HOST_GO_TMP)/bin/go $(HOST_GO_ROOT)/bin/go
+	$(INSTALL) -D -m 0755 $(HOST_GO_TMP)/bin/gofmt $(HOST_GO_ROOT)/bin/gofmt
 
 	ln -sf ../lib/go/bin/go $(HOST_DIR)/usr/bin/
 	ln -sf ../lib/go/bin/gofmt $(HOST_DIR)/usr/bin/
@@ -62,7 +95,7 @@ define HOST_GO_INSTALL_CMDS
 
 	mkdir -p $(HOST_GO_ROOT)/pkg
 	cp -a $(@D)/pkg/include $(@D)/pkg/linux_* $(HOST_GO_ROOT)/pkg/
-	cp -a $(@D)/pkg/tool $(HOST_GO_ROOT)/pkg/
+	cp -a $(HOST_GO_TMP)/tool $(HOST_GO_ROOT)/pkg/
 
 	# There is a known issue which requires the go sources to be installed
 	# https://golang.org/issue/2775
-- 
2.5.0

[Buildroot] [PATCH v2 3/6] package/flannel: Use HOST_GO_TARGET_ENV

[Geoff Levand]

Use the newly added HOST_GO_TARGET_ENV variable to pickup the
correct go environment for package builds.

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
 package/flannel/flannel.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/flannel/flannel.mk b/package/flannel/flannel.mk
index 876efe6..b5f61a0 100644
--- a/package/flannel/flannel.mk
+++ b/package/flannel/flannel.mk
@@ -14,9 +14,9 @@ FLANNEL_LICENSE_FILES = LICENSE
 FLANNEL_DEPENDENCIES = host-go
 
 FLANNEL_MAKE_ENV = \
+	$(HOST_GO_TARGET_ENV) \
 	GOBIN="$(@D)/bin" \
 	GOPATH="$(@D)/gopath" \
-	GOARCH=$(GO_GOARCH) \
 	CGO_ENABLED=1
 
 FLANNEL_GLDFLAGS = \
-- 
2.5.0

[Buildroot] [PATCH v2 5/6] package/flannel: Add BR2_TOOLCHAIN_HAS_THREADS

[Geoff Levand]

flannel uses the cgo package, so needs a toolchain with thread
support.

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
 package/flannel/Config.in | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/package/flannel/Config.in b/package/flannel/Config.in
index c6a84ef..32d9f66 100644
--- a/package/flannel/Config.in
+++ b/package/flannel/Config.in
@@ -1,6 +1,10 @@
+comment "flannel needs a toolchain w/ threads"
+	depends on !BR2_TOOLCHAIN_HAS_THREADS
+
 config BR2_PACKAGE_FLANNEL
 	bool "flannel"
 	depends on BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS
+	depends on BR2_TOOLCHAIN_HAS_THREADS
 	help
 	  Flannel is a virtual network that gives a subnet to each
 	  host for use with container runtimes.
-- 
2.5.0

[Buildroot] [PATCH v2 1/6] package/go: Build host tools with host CC

[Thomas Petazzoni]

Hello,

Thanks for this new version.

On Thu, 19 May 2016 22:25:02 +0000, Geoff Levand wrote:

> +HOST_GO_TMP = $(@D)/host-go-tmp
> +
> +ifneq ($(ARCH),$(HOSTARCH))
>  define HOST_GO_BUILD_CMDS
> -	cd $(@D)/src && $(HOST_GO_MAKE_ENV) ./make.bash
> +	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> +	mkdir -p $(HOST_GO_TMP)
> +	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> +	mv $(@D)/bin/ $(HOST_GO_TMP)/

So here you are moving what was built to HOST_GO_TMP.

>  endef
> +else
> +define HOST_GO_BUILD_CMDS
> +	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
> +	mkdir -p $(HOST_GO_TMP)
> +	cp -a $(@D)/pkg/tool $(HOST_GO_TMP)/
> +	cp -a $(@D)/bin/ $(HOST_GO_TMP)/
> +	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> +	cp -a $(@D)/pkg/tool $(HOST_GO_TMP)/

And here you're using a copy. Why?

Also, in this last case, you are copying pkg/tool/ twice, once after
the build with HOST_GO_HOST_CC, and once after the build with
HOST_GO_TARGET_CC. It seems like the first copy is useless.

Could you comment on this?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2 6/6] package/go: Set file timestamp

[Thomas Petazzoni]

Hello,

On Thu, 19 May 2016 22:25:02 +0000, Geoff Levand wrote:
> Set all file timestamps to prevent the go compiler from rebuilding any
> built in packages when programs are built.
> 
> Signed-off-by: Geoff Levand <geoff@infradead.org>
> ---
>  package/go/go.mk | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/package/go/go.mk b/package/go/go.mk
> index 8565a86..a2babab 100644
> --- a/package/go/go.mk
> +++ b/package/go/go.mk
> @@ -119,6 +119,10 @@ define HOST_GO_INSTALL_CMDS
>  	# There is a known issue which requires the go sources to be installed
>  	# https://golang.org/issue/2775
>  	cp -a $(@D)/src $(HOST_GO_ROOT)/
> +
> +	# Set all file timestamps to prevent the go compiler from rebuilding any
> +	# built in packages when programs are built.
> +	find $(HOST_GO_ROOT) -type f -exec touch -r $(HOST_GO_TMP)/bin/go {} \;
>  endef

That's weird, why is it needed? Why are timestamps wrong in the first
place?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2 6/6] package/go: Set file timestamp

[Christian Stewart]

Geoff,

Same question. I'm not sure if the timestamps are even considered by the
compiler. If so, do you have a reference to how this works?

Thanks,
Christian

On Tue, May 24, 2016, 12:13 PM Thomas Petazzoni <
thomas.petazzoni@free-electrons.com> wrote:

> Hello,
>
> On Thu, 19 May 2016 22:25:02 +0000, Geoff Levand wrote:
> > Set all file timestamps to prevent the go compiler from rebuilding any
> > built in packages when programs are built.
> >
> > Signed-off-by: Geoff Levand <geoff@infradead.org>
> > ---
> >  package/go/go.mk | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/package/go/go.mk b/package/go/go.mk
> > index 8565a86..a2babab 100644
> > --- a/package/go/go.mk
> > +++ b/package/go/go.mk
> > @@ -119,6 +119,10 @@ define HOST_GO_INSTALL_CMDS
> >       # There is a known issue which requires the go sources to be
> installed
> >       # https://golang.org/issue/2775
> >       cp -a $(@D)/src $(HOST_GO_ROOT)/
> > +
> > +     # Set all file timestamps to prevent the go compiler from
> rebuilding any
> > +     # built in packages when programs are built.
> > +     find $(HOST_GO_ROOT) -type f -exec touch -r $(HOST_GO_TMP)/bin/go
> {} \;
> >  endef
>
> That's weird, why is it needed? Why are timestamps wrong in the first
> place?
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160524/dbcac3da/attachment.html>

[Buildroot] [PATCH v2 1/6] package/go: Build host tools with host CC

[Geoff Levand]

Hi,

On Tue, 2016-05-24 at 21:11 +0200, Thomas Petazzoni wrote:
> On Thu, 19 May 2016 22:25:02 +0000, Geoff Levand wrote:
> 
> > +HOST_GO_TMP = $(@D)/host-go-tmp
> > +
> > +ifneq ($(ARCH),$(HOSTARCH))
> >  define HOST_GO_BUILD_CMDS
> > -> > 	> > cd $(@D)/src && $(HOST_GO_MAKE_ENV) ./make.bash
> > +> > 	> > cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> > +> > 	> > mkdir -p $(HOST_GO_TMP)
> > +> > 	> > mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> > +> > 	> > mv $(@D)/bin/ $(HOST_GO_TMP)/
> 
> So here you are moving what was built to HOST_GO_TMP.
> 
> >  endef
> > +else
> > +define HOST_GO_BUILD_CMDS
> > +> > 	> > cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
> > +> > 	> > mkdir -p $(HOST_GO_TMP)
> > +> > 	> > cp -a $(@D)/pkg/tool $(HOST_GO_TMP)/
> > +> > 	> > cp -a $(@D)/bin/ $(HOST_GO_TMP)/
> > +> > 	> > cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> > +> > 	> > cp -a $(@D)/pkg/tool $(HOST_GO_TMP)/
> 
> And here you're using a copy. Why?

We can use mv here also.

> Also, in this last case, you are copying pkg/tool/ twice, once after
> the build with HOST_GO_HOST_CC, and once after the build with
> HOST_GO_TARGET_CC. It seems like the first copy is useless.

This got messed up, this last cp line should not be there.  We
want the binaries and tools build with the host compiler to be
installed.  Everything else installed should be build with the
target compiler.

I'll post a follow up patch.

-Geoff

[Buildroot] [PATCH v2.1 1/6] package/go: Build host tools with host CC

[Geoff Levand]

The go build system doesn't have the notion of cross compiling, but just the
notion of architecture.  When the host and target architectures are different
it expects to be given a target cross compiler in CC_FOR_TARGET.  When the
architectures are the same it will use CC_FOR_TARGET for both host and target
compilation.  To work around this limitation, when the host and target
architectures are the same build a set of host binaries and tools with
CC_FOR_TARGET set to the host compiler.  Also, the go build system is not
compatible with ccache, so use HOSTCC_NOCCACHE.  See
https://github.com/golang/go/issues/11685.

Fixes build errors like these:

  host/usr/bin/go: No such file or directory

  http://autobuild.buildroot.net/results/6664189a6f3a815978e8d0a1d7ef408ca47e2874/

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
 package/go/go.mk | 40 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 36 insertions(+), 4 deletions(-)

diff --git a/package/go/go.mk b/package/go/go.mk
index a9e16dd..45c29fe 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -35,6 +35,16 @@ endif
 HOST_GO_DEPENDENCIES = host-go-bootstrap
 HOST_GO_ROOT = $(HOST_DIR)/usr/lib/go
 
+# The go build system doesn't have the notion of cross compiling, but just the
+# notion of architecture.  When the host and target architectures are different
+# it expects to be given a target cross compiler in CC_FOR_TARGET.  When the
+# architectures are the same it will use CC_FOR_TARGET for both host and target
+# compilation.  To work around this limitation, when the host and target
+# architectures are the same build a set of host binaries and tools with
+# CC_FOR_TARGET set to the host compiler.  Also, the go build system is not
+# compatible with ccache, so use HOSTCC_NOCCACHE.  See
+# https://github.com/golang/go/issues/11685.
+
 HOST_GO_MAKE_ENV = \
 	GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_ROOT) \
 	GOROOT_FINAL=$(HOST_GO_ROOT) \
@@ -44,16 +54,38 @@ HOST_GO_MAKE_ENV = \
 	$(if $(GO_GOARM),GOARM=$(GO_GOARM)) \
 	GOOS=linux \
 	CGO_ENABLED=1 \
+	CC=$(HOSTCC_NOCCACHE)
+
+HOST_GO_TARGET_CC = \
 	CC_FOR_TARGET=$(TARGET_CC) \
 	CXX_FOR_TARGET=$(TARGET_CXX)
 
+HOST_GO_HOST_CC = \
+	CC_FOR_TARGET=$(HOSTCC_NOCCACHE) \
+	CXX_FOR_TARGET=$(HOSTCC_NOCCACHE)
+
+HOST_GO_TMP = $(@D)/host-go-tmp
+
+ifneq ($(ARCH),$(HOSTARCH))
 define HOST_GO_BUILD_CMDS
-	cd $(@D)/src && $(HOST_GO_MAKE_ENV) ./make.bash
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
+	mkdir -p $(HOST_GO_TMP)
+	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
+	mv $(@D)/bin/ $(HOST_GO_TMP)/
 endef
+else
+define HOST_GO_BUILD_CMDS
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
+	mkdir -p $(HOST_GO_TMP)
+	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
+	mv $(@D)/bin/ $(HOST_GO_TMP)/
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
+endef
+endif
 
 define HOST_GO_INSTALL_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/bin/go $(HOST_GO_ROOT)/bin/go
-	$(INSTALL) -D -m 0755 $(@D)/bin/gofmt $(HOST_GO_ROOT)/bin/gofmt
+	$(INSTALL) -D -m 0755 $(HOST_GO_TMP)/bin/go $(HOST_GO_ROOT)/bin/go
+	$(INSTALL) -D -m 0755 $(HOST_GO_TMP)/bin/gofmt $(HOST_GO_ROOT)/bin/gofmt
 
 	ln -sf ../lib/go/bin/go $(HOST_DIR)/usr/bin/
 	ln -sf ../lib/go/bin/gofmt $(HOST_DIR)/usr/bin/
@@ -62,7 +94,7 @@ define HOST_GO_INSTALL_CMDS
 
 	mkdir -p $(HOST_GO_ROOT)/pkg
 	cp -a $(@D)/pkg/include $(@D)/pkg/linux_* $(HOST_GO_ROOT)/pkg/
-	cp -a $(@D)/pkg/tool $(HOST_GO_ROOT)/pkg/
+	cp -a $(HOST_GO_TMP)/tool $(HOST_GO_ROOT)/pkg/
 
 	# There is a known issue which requires the go sources to be installed
 	# https://golang.org/issue/2775
-- 
2.5.0

[Buildroot] [PATCH v2 6/6] package/go: Set file timestamp

[Geoff Levand]

Hi,

On Tue, 2016-05-24 at 21:13 +0200, Thomas Petazzoni wrote:
> On Thu, 19 May 2016 22:25:02 +0000, Geoff Levand wrote:
> > Set all file timestamps to prevent the go compiler from rebuilding any
> > built in packages when programs are built.
>
> That's weird, why is it needed? 

If the times are not set, when a go program is built go may
rebuild some of the installed packages.  This is the same
kind of time based dependecy checking make does.

Without this change I can observe installed packages being
rebuilt when flannel is built on my machine.

> Why are timestamps wrong in the first
> place?

I guess go is built in parallel, and so the artifacts have
different times.  Gentoo does a similar timestamp operation
in the pkg_postinst of its ebuild:

 https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-lang/go/go-9999.ebuild?logsort=cvs&revision=1.30&view=markup

-Geoff

[Buildroot] [PATCH v2.1 1/6] package/go: Build host tools with host CC

[Thomas Petazzoni]

Hello,

On Tue, 24 May 2016 14:34:05 -0700, Geoff Levand wrote:

> +ifneq ($(ARCH),$(HOSTARCH))
>  define HOST_GO_BUILD_CMDS
> -	cd $(@D)/src && $(HOST_GO_MAKE_ENV) ./make.bash
> +	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> +	mkdir -p $(HOST_GO_TMP)
> +	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> +	mv $(@D)/bin/ $(HOST_GO_TMP)/
>  endef
> +else
> +define HOST_GO_BUILD_CMDS
> +	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
> +	mkdir -p $(HOST_GO_TMP)
> +	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> +	mv $(@D)/bin/ $(HOST_GO_TMP)/
> +	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash

Just to make sure I understand correctly: you build everything with
HOST_GO_HOST_CC first, and put the files generated in pkg/tool and bin/
on the side in HOST_GO_TMP, and then rebuild everything again with
HOST_GO_TARGET_CC. Then later on, the installation step will pick up
bin/ and pkg/tool from HOST_GO_TMP, and the rest from $(@D) directly ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2.1 1/6] package/go: Build host tools with host CC

[Geoff Levand]

On Wed, 2016-05-25 at 16:05 +0200, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 24 May 2016 14:34:05 -0700, Geoff Levand wrote:
> 
> > +ifneq ($(ARCH),$(HOSTARCH))
> >  define HOST_GO_BUILD_CMDS
> > -> > 	> > cd $(@D)/src && $(HOST_GO_MAKE_ENV) ./make.bash
> > +> > 	> > cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> > +> > 	> > mkdir -p $(HOST_GO_TMP)
> > +> > 	> > mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> > +> > 	> > mv $(@D)/bin/ $(HOST_GO_TMP)/
> >  endef
> > +else
> > +define HOST_GO_BUILD_CMDS
> > +> > 	> > cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
> > +> > 	> > mkdir -p $(HOST_GO_TMP)
> > +> > 	> > mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> > +> > 	> > mv $(@D)/bin/ $(HOST_GO_TMP)/
> > +> > 	> > cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> 
> Just to make sure I understand correctly: you build everything with
> HOST_GO_HOST_CC first, and put the files generated in pkg/tool and bin/
> on the side in HOST_GO_TMP, and then rebuild everything again with
> HOST_GO_TARGET_CC. Then later on, the installation step will pick up
> bin/ and pkg/tool from HOST_GO_TMP, and the rest from $(@D) directly ?

Yes, that is correct.

bin and tool with HOST_GO_HOST_CC, because those are host binaries.
The rest with HOST_GO_TARGET_CC because those are target binaries.

If you can think of a better way to do this two step build I can try it.

-Geoff

[Buildroot] [PATCH v2.1 1/6] package/go: Build host tools with host CC

[Thomas Petazzoni]

Hello,

On Wed, 25 May 2016 09:41:54 -0700, Geoff Levand wrote:

> > Just to make sure I understand correctly: you build everything with
> > HOST_GO_HOST_CC first, and put the files generated in pkg/tool and bin/
> > on the side in HOST_GO_TMP, and then rebuild everything again with
> > HOST_GO_TARGET_CC. Then later on, the installation step will pick up
> > bin/ and pkg/tool from HOST_GO_TMP, and the rest from $(@D) directly ?  
> 
> Yes, that is correct.

OK, thanks for confirming!

> bin and tool with HOST_GO_HOST_CC, because those are host binaries.
> The rest with HOST_GO_TARGET_CC because those are target binaries.
> 
> If you can think of a better way to do this two step build I can try it.

I was thinking of having the host parts build by host-go and the
target parts by the "go" package, but I'm not sure it's easy to achieve
nor that it will make the stuff clearer.

Now that I think of it, why do we have to have those two cases:

+ifneq ($(ARCH),$(HOSTARCH))
 define HOST_GO_BUILD_CMDS
-	cd $(@D)/src && $(HOST_GO_MAKE_ENV) ./make.bash
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
+	mkdir -p $(HOST_GO_TMP)
+	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
+	mv $(@D)/bin/ $(HOST_GO_TMP)/
 endef
+else
+define HOST_GO_BUILD_CMDS
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
+	mkdir -p $(HOST_GO_TMP)
+	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
+	mv $(@D)/bin/ $(HOST_GO_TMP)/
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
+endef
+endif

Aren't the host binaries always built for the host, and the target
binaries always built for the target? So can't we replace this by just:

+define HOST_GO_BUILD_CMDS
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
+	mkdir -p $(HOST_GO_TMP)
+	mv $(@D)/pkg/tool $(HOST_GO_TMP)/
+	mv $(@D)/bin/ $(HOST_GO_TMP)/
+	cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
+endef

and get in fact the exact same behavior?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2.1 1/6] package/go: Build host tools with host CC

[Geoff Levand]

Hi,

On Wed, 2016-05-25 at 22:59 +0200, Thomas Petazzoni wrote:
> On Wed, 25 May 2016 09:41:54 -0700, Geoff Levand wrote:
> bin and tool with HOST_GO_HOST_CC, because those are host binaries.
> > The rest with HOST_GO_TARGET_CC because those are target binaries.
> > 
> > If you can think of a better way to do this two step build I can try it.
> 
> I was thinking of having the host parts build by host-go and the
> target parts by the "go" package, but I'm not sure it's easy to achieve
> nor that it will make the stuff clearer.

Seems to be more complicated.

> Now that I think of it, why do we have to have those two cases:
> 
> +ifneq ($(ARCH),$(HOSTARCH))
>  define HOST_GO_BUILD_CMDS
> -> 	> cd $(@D)/src && $(HOST_GO_MAKE_ENV) ./make.bash
> +> 	> cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> +> 	> mkdir -p $(HOST_GO_TMP)
> +> 	> mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> +> 	> mv $(@D)/bin/ $(HOST_GO_TMP)/
>  endef
> +else
> +define HOST_GO_BUILD_CMDS
> +> 	> cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
> +> 	> mkdir -p $(HOST_GO_TMP)
> +> 	> mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> +> 	> mv $(@D)/bin/ $(HOST_GO_TMP)/
> +> 	> cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> +endef
> +endif
> 
> Aren't the host binaries always built for the host, and the target
> binaries always built for the target? So can't we replace this by just:
> 
> +define HOST_GO_BUILD_CMDS
> +> 	> cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_HOST_CC) ./make.bash
> +> 	> mkdir -p $(HOST_GO_TMP)
> +> 	> mv $(@D)/pkg/tool $(HOST_GO_TMP)/
> +> 	> mv $(@D)/bin/ $(HOST_GO_TMP)/
> +> 	> cd $(@D)/src && $(HOST_GO_MAKE_ENV) $(HOST_GO_TARGET_CC) ./make.bash
> +endef
> 
> and get in fact the exact same behavior?

There is a subtle difference, when we build with HOST_GO_HOST_CC the
resulting host compiler binaries have the go environment variables CC
and CXX set to the host compilers.  They should be set to the target
cross compilers so that by default go programs are built with the target
cross compiler.  That was one motivation for my follow up patch
'package/go: Add HOST_GO_TARGET_ENV', which sets those to the proper
values for target package builds.

I had planned to look into how to fix this environment problem.  One
idea was to use a custom version of ./make.bash that would do the
right thing.

On thinking about it, it seems it would be best for now to do the
two step build for all configs and expect package maintainers to
either use HOST_GO_TARGET_ENV or set CC and CXX.  That would avoid
the situation where the maintainer does not test with host arch
equal to target arch.

-Geoff

[Buildroot] [PATCH v2.1 1/6] package/go: Build host tools with host CC

[Thomas Petazzoni]

Hello,

On Thu, 26 May 2016 10:24:56 -0700, Geoff Levand wrote:

> There is a subtle difference, when we build with HOST_GO_HOST_CC the
> resulting host compiler binaries have the go environment variables CC
> and CXX set to the host compilers.  They should be set to the target
> cross compilers so that by default go programs are built with the target
> cross compiler.  That was one motivation for my follow up patch
> 'package/go: Add HOST_GO_TARGET_ENV', which sets those to the proper
> values for target package builds.

Ah, indeed.

> I had planned to look into how to fix this environment problem.  One
> idea was to use a custom version of ./make.bash that would do the
> right thing.
> 
> On thinking about it, it seems it would be best for now to do the
> two step build for all configs and expect package maintainers to
> either use HOST_GO_TARGET_ENV or set CC and CXX.  That would avoid
> the situation where the maintainer does not test with host arch
> equal to target arch.

OK. Can you respin the patch quickly to address this? the host-go and
flannel issues are the main remaining build issues we have, and it
would be good to fix them for 2016.05.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com