All of lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [PATCH 0/6]  support for a pre-built Go compiler
@ 2024-04-12 13:26 Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 1/6] DEVELOPERS: add Thomas Perale for 'go' Thomas Perale via buildroot
                   ` (5 more replies)
  0 siblings, 6 replies; 13+ messages in thread
From: Thomas Perale via buildroot @ 2024-04-12 13:26 UTC (permalink / raw)
  To: buildroot
  Cc: Christian Stewart, Geoff Levand, Anisse Astier, Thomas Perale,
	Angelo Compagnucci, Marcin Niestroj, Thomas Petazzoni,
	Romain Naour, Fabio Estevam, Yann E . MORIN

Improvement suggested on the buildroot TODO list (https://www.elinux.org/Buildroot#Todo_list).
This patch adds support for a pre-built version of the Go compiler that
doesn't require users to go through the different compilation stages of
the Go compiler and speeds up the overall build time.

This patch is a new version of the proposal I made last year
(see https://lore.kernel.org/buildroot/CA+h8R2pNRrBsChKS3PHQtT0_rsVF3Wi3LwMTD9jKFoKf1ot2-A@mail.gmail.com/#b)
and a rebase of my previous patch (https://lore.kernel.org/buildroot/20231004142303.203520-5-thomas.perale@essensium.com/T/)
to support new Go 1.22 that require a 3rd bootstrap stage.

It introduces a new directory structure for host compilers with different
providers:

    package/
    ├─ go/                     <-- Virtual package
    │  ├─ go-src/              <-- Build from source option
    │  ├─ go-bin/              <-- Download pre-built option
    │  ├─ go-bootstrap-stage1/ <-- Dependency for go-bootstrap-stage2
    │  ├─ go-bootstrap-stage2/ <-- Dependency for go-bootstrap-stage3
    │  ├─ go-bootstrap-stage3/ <-- Dependency for go-src
    │  ├─ go.mk                <-- Common variables
    │  ├─ Config.in            <-- Common variables/virtual package definition

The subdirectory structure is used to put the `GO_VERSION` variable in
common between the different providers.

This subdirectory proposal should be extended to the Rust compiler case and
NodeJS proposal Thomas Petazzoni did (https://patchwork.ozlabs.org/project/buildroot/list/?series=319967)
to unify the naming of the different compilers options.

Thomas Perale (6):
  DEVELOPERS: add Thomas Perale for 'go'
  package/go: new subdirectory for go variants
  package/go: make host package a virtual package
  package/pkg-golang: select go compiler
  package/go/go-bin: new go provider
  support/testing: add tests for Go providers

 .checkpackageignore                           |  2 +-
 DEVELOPERS                                    |  6 +--
 package/Config.in.host                        |  3 --
 package/balena-engine/Config.in               |  1 +
 package/cni-plugins/Config.in                 |  1 +
 package/containerd/Config.in                  |  1 +
 package/crucible/Config.in                    |  1 +
 package/delve/Config.in                       |  1 +
 package/delve/Config.in.host                  |  1 +
 package/docker-cli/Config.in                  |  1 +
 package/docker-compose/Config.in              |  1 +
 package/docker-engine/Config.in               |  1 +
 package/embiggen-disk/Config.in               |  1 +
 package/flannel/Config.in                     |  1 +
 package/gitlab-runner/Config.in               |  1 +
 package/go/Config.in.host                     | 46 ++++++++++++++++
 package/go/go-bin/Config.in.host              | 14 +++++
 package/go/go-bin/go-bin.hash                 |  8 +++
 package/go/go-bin/go-bin.mk                   | 19 +++++++
 .../go-bootstrap-stage1/Config.in.host        |  0
 .../go-bootstrap-stage1.hash                  |  0
 .../go-bootstrap-stage1.mk                    |  0
 .../go-bootstrap-stage2/Config.in.host        |  0
 .../go-bootstrap-stage2.hash                  |  0
 .../go-bootstrap-stage2.mk                    |  0
 .../go-bootstrap-stage3/Config.in.host        |  0
 .../go-bootstrap-stage3.hash                  |  0
 .../go-bootstrap-stage3.mk                    |  0
 ...explicit-option-for-crosscompilation.patch |  0
 ...ldvcs-false-when-building-go-bootstr.patch |  0
 package/go/{go.hash => go-src/go-src.hash}    |  0
 package/go/go-src/go-src.mk                   | 54 +++++++++++++++++++
 package/go/go.mk                              | 42 ++-------------
 package/gocryptfs/Config.in                   |  1 +
 package/mender-artifact/Config.in.host        |  1 +
 package/mender-connect/Config.in              |  1 +
 package/mender/Config.in                      |  1 +
 package/moby-buildkit/Config.in               |  1 +
 package/moby-buildkit/Config.in.host          |  1 +
 package/nerdctl/Config.in                     |  1 +
 package/runc/Config.in                        |  1 +
 package/runc/Config.in.host                   |  1 +
 package/tinifier/Config.in                    |  1 +
 package/wtfutil/Config.in                     |  1 +
 support/testing/tests/package/test_go.py      | 41 ++++++++++++++
 45 files changed, 213 insertions(+), 45 deletions(-)
 create mode 100644 package/go/go-bin/Config.in.host
 create mode 100644 package/go/go-bin/go-bin.hash
 create mode 100644 package/go/go-bin/go-bin.mk
 rename package/{ => go}/go-bootstrap-stage1/Config.in.host (100%)
 rename package/{ => go}/go-bootstrap-stage1/go-bootstrap-stage1.hash (100%)
 rename package/{ => go}/go-bootstrap-stage1/go-bootstrap-stage1.mk (100%)
 rename package/{ => go}/go-bootstrap-stage2/Config.in.host (100%)
 rename package/{ => go}/go-bootstrap-stage2/go-bootstrap-stage2.hash (100%)
 rename package/{ => go}/go-bootstrap-stage2/go-bootstrap-stage2.mk (100%)
 rename package/{ => go}/go-bootstrap-stage3/Config.in.host (100%)
 rename package/{ => go}/go-bootstrap-stage3/go-bootstrap-stage3.hash (100%)
 rename package/{ => go}/go-bootstrap-stage3/go-bootstrap-stage3.mk (100%)
 rename package/go/{ => go-src}/0001-build.go-explicit-option-for-crosscompilation.patch (100%)
 rename package/go/{ => go-src}/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch (100%)
 rename package/go/{go.hash => go-src/go-src.hash} (100%)
 create mode 100644 package/go/go-src/go-src.mk
 create mode 100644 support/testing/tests/package/test_go.py

--
2.44.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 13+ messages in thread

* [Buildroot] [PATCH 1/6] DEVELOPERS: add Thomas Perale for 'go'
  2024-04-12 13:26 [Buildroot] [PATCH 0/6] support for a pre-built Go compiler Thomas Perale via buildroot
@ 2024-04-12 13:26 ` Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 2/6] package/go: new subdirectory for go variants Thomas Perale via buildroot
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 13+ messages in thread
From: Thomas Perale via buildroot @ 2024-04-12 13:26 UTC (permalink / raw)
  To: buildroot
  Cc: Christian Stewart, Geoff Levand, Anisse Astier, Thomas Perale,
	Angelo Compagnucci, Marcin Niestroj, Thomas Petazzoni,
	Romain Naour, Fabio Estevam, Yann E . MORIN

Signed-off-by: Thomas Perale <thomas.perale@mind.be>
---
 DEVELOPERS | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/DEVELOPERS b/DEVELOPERS
index 399b2931ff..039b803d50 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -2979,6 +2979,9 @@ F:	package/frotz/
 F:	package/kvm-unit-tests/
 F:	package/xorcurses/
 
+N:	Thomas Perale <thomas.perale@mind.be>
+F:	package/go/
+
 N:	Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 F:	arch/Config.in.arm
 F:	board/beaglev/
-- 
2.44.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [Buildroot] [PATCH 2/6] package/go: new subdirectory for go variants
  2024-04-12 13:26 [Buildroot] [PATCH 0/6] support for a pre-built Go compiler Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 1/6] DEVELOPERS: add Thomas Perale for 'go' Thomas Perale via buildroot
@ 2024-04-12 13:26 ` Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 3/6] package/go: make host package a virtual package Thomas Perale via buildroot
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 13+ messages in thread
From: Thomas Perale via buildroot @ 2024-04-12 13:26 UTC (permalink / raw)
  To: buildroot
  Cc: Christian Stewart, Geoff Levand, Anisse Astier, Thomas Perale,
	Angelo Compagnucci, Marcin Niestroj, Thomas Petazzoni,
	Romain Naour, Fabio Estevam, Yann E . MORIN

Move every go compiler-related packages into a newly created
package/go/ subdirectory.

This subdirectory structure moves the GO_VERSION variable into the
common package/go/go.mk file. In the next commits, host-go will be
turned into a virtual-package and the common GO_VERSION force the
providers to use the same Go compiler version.
Common variables to all providers are kept in package/go/go.mk and
package/go/Config.in.host.
Also, the subdirectory structure forces the evaluation of the common
GO_VERSION before the providers access it.

Signed-off-by: Thomas Perale <thomas.perale@mind.be>
---
 .checkpackageignore                           |  2 +-
 DEVELOPERS                                    |  3 --
 package/Config.in.host                        |  3 --
 package/go/Config.in.host                     |  4 ++
 .../go-bootstrap-stage1/Config.in.host        |  0
 .../go-bootstrap-stage1.hash                  |  0
 .../go-bootstrap-stage1.mk                    |  0
 .../go-bootstrap-stage2/Config.in.host        |  0
 .../go-bootstrap-stage2.hash                  |  0
 .../go-bootstrap-stage2.mk                    |  0
 .../go-bootstrap-stage3/Config.in.host        |  0
 .../go-bootstrap-stage3.hash                  |  0
 .../go-bootstrap-stage3.mk                    |  0
 package/go/go.mk                              | 40 +-------------
 ...explicit-option-for-crosscompilation.patch |  0
 ...ldvcs-false-when-building-go-bootstr.patch |  0
 package/go/{ => go}/go.hash                   |  0
 package/go/go/go.mk                           | 53 +++++++++++++++++++
 18 files changed, 60 insertions(+), 45 deletions(-)
 rename package/{ => go}/go-bootstrap-stage1/Config.in.host (100%)
 rename package/{ => go}/go-bootstrap-stage1/go-bootstrap-stage1.hash (100%)
 rename package/{ => go}/go-bootstrap-stage1/go-bootstrap-stage1.mk (100%)
 rename package/{ => go}/go-bootstrap-stage2/Config.in.host (100%)
 rename package/{ => go}/go-bootstrap-stage2/go-bootstrap-stage2.hash (100%)
 rename package/{ => go}/go-bootstrap-stage2/go-bootstrap-stage2.mk (100%)
 rename package/{ => go}/go-bootstrap-stage3/Config.in.host (100%)
 rename package/{ => go}/go-bootstrap-stage3/go-bootstrap-stage3.hash (100%)
 rename package/{ => go}/go-bootstrap-stage3/go-bootstrap-stage3.mk (100%)
 rename package/go/{ => go}/0001-build.go-explicit-option-for-crosscompilation.patch (100%)
 rename package/go/{ => go}/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch (100%)
 rename package/go/{ => go}/go.hash (100%)
 create mode 100644 package/go/go/go.mk

diff --git a/.checkpackageignore b/.checkpackageignore
index 1b336ce7a3..b3eab26071 100644
--- a/.checkpackageignore
+++ b/.checkpackageignore
@@ -465,7 +465,7 @@ package/glorytun/0002-aegis256.c-fix-aarch64-build-with-uclibc.patch lib_patch.U
 package/gnu-efi/0001-Make.defaults-don-t-override-ARCH-when-cross-compili.patch lib_patch.Upstream
 package/gnupg/0001-build-Always-use-EXTERN_UNLESS_MAIN_MODULE-pattern.patch lib_patch.Upstream
 package/gnuplot/0001-configure-add-without-demo-option.patch lib_patch.Upstream
-package/go/0001-build.go-explicit-option-for-crosscompilation.patch lib_patch.Upstream
+package/go/go/0001-build.go-explicit-option-for-crosscompilation.patch lib_patch.Upstream
 package/gob2/0001-dont-include-from-prefix.patch lib_patch.Upstream
 package/gobject-introspection/0001-disable-tests.patch lib_patch.Upstream
 package/gobject-introspection/0002-Add-rpath-links-to-ccompiler.patch lib_patch.Upstream
diff --git a/DEVELOPERS b/DEVELOPERS
index 039b803d50..51fc4962c9 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -603,9 +603,6 @@ F:	package/docker-engine/
 F:	package/embiggen-disk/
 F:	package/fuse-overlayfs/
 F:	package/go/
-F:	package/go-bootstrap-stage1/
-F:	package/go-bootstrap-stage2/
-F:	package/go-bootstrap-stage3/
 F:	package/gocryptfs/
 F:	package/mbpfan/
 F:	package/moby-buildkit/
diff --git a/package/Config.in.host b/package/Config.in.host
index 9543a22ffc..2c481cfae7 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -44,9 +44,6 @@ menu "Host utilities"
 	source "package/genpart/Config.in.host"
 	source "package/gnupg/Config.in.host"
 	source "package/go/Config.in.host"
-	source "package/go-bootstrap-stage1/Config.in.host"
-	source "package/go-bootstrap-stage2/Config.in.host"
-	source "package/go-bootstrap-stage3/Config.in.host"
 	source "package/google-breakpad/Config.in.host"
 	source "package/gptfdisk/Config.in.host"
 	source "package/imagemagick/Config.in.host"
diff --git a/package/go/Config.in.host b/package/go/Config.in.host
index 0d89e875ad..7edf45850d 100644
--- a/package/go/Config.in.host
+++ b/package/go/Config.in.host
@@ -31,3 +31,7 @@ config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
 	bool
 	default y
 	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
+
+source "package/go/go-bootstrap-stage1/Config.in.host"
+source "package/go/go-bootstrap-stage2/Config.in.host"
+source "package/go/go-bootstrap-stage3/Config.in.host"
diff --git a/package/go-bootstrap-stage1/Config.in.host b/package/go/go-bootstrap-stage1/Config.in.host
similarity index 100%
rename from package/go-bootstrap-stage1/Config.in.host
rename to package/go/go-bootstrap-stage1/Config.in.host
diff --git a/package/go-bootstrap-stage1/go-bootstrap-stage1.hash b/package/go/go-bootstrap-stage1/go-bootstrap-stage1.hash
similarity index 100%
rename from package/go-bootstrap-stage1/go-bootstrap-stage1.hash
rename to package/go/go-bootstrap-stage1/go-bootstrap-stage1.hash
diff --git a/package/go-bootstrap-stage1/go-bootstrap-stage1.mk b/package/go/go-bootstrap-stage1/go-bootstrap-stage1.mk
similarity index 100%
rename from package/go-bootstrap-stage1/go-bootstrap-stage1.mk
rename to package/go/go-bootstrap-stage1/go-bootstrap-stage1.mk
diff --git a/package/go-bootstrap-stage2/Config.in.host b/package/go/go-bootstrap-stage2/Config.in.host
similarity index 100%
rename from package/go-bootstrap-stage2/Config.in.host
rename to package/go/go-bootstrap-stage2/Config.in.host
diff --git a/package/go-bootstrap-stage2/go-bootstrap-stage2.hash b/package/go/go-bootstrap-stage2/go-bootstrap-stage2.hash
similarity index 100%
rename from package/go-bootstrap-stage2/go-bootstrap-stage2.hash
rename to package/go/go-bootstrap-stage2/go-bootstrap-stage2.hash
diff --git a/package/go-bootstrap-stage2/go-bootstrap-stage2.mk b/package/go/go-bootstrap-stage2/go-bootstrap-stage2.mk
similarity index 100%
rename from package/go-bootstrap-stage2/go-bootstrap-stage2.mk
rename to package/go/go-bootstrap-stage2/go-bootstrap-stage2.mk
diff --git a/package/go-bootstrap-stage3/Config.in.host b/package/go/go-bootstrap-stage3/Config.in.host
similarity index 100%
rename from package/go-bootstrap-stage3/Config.in.host
rename to package/go/go-bootstrap-stage3/Config.in.host
diff --git a/package/go-bootstrap-stage3/go-bootstrap-stage3.hash b/package/go/go-bootstrap-stage3/go-bootstrap-stage3.hash
similarity index 100%
rename from package/go-bootstrap-stage3/go-bootstrap-stage3.hash
rename to package/go/go-bootstrap-stage3/go-bootstrap-stage3.hash
diff --git a/package/go-bootstrap-stage3/go-bootstrap-stage3.mk b/package/go/go-bootstrap-stage3/go-bootstrap-stage3.mk
similarity index 100%
rename from package/go-bootstrap-stage3/go-bootstrap-stage3.mk
rename to package/go/go-bootstrap-stage3/go-bootstrap-stage3.mk
diff --git a/package/go/go.mk b/package/go/go.mk
index 0e39ad9f8c..104a4a73fb 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -5,14 +5,7 @@
 ################################################################################
 
 GO_VERSION = 1.22.2
-GO_SITE = https://storage.googleapis.com/golang
-GO_SOURCE = go$(GO_VERSION).src.tar.gz
 
-GO_LICENSE = BSD-3-Clause
-GO_LICENSE_FILES = LICENSE
-GO_CPE_ID_VENDOR = golang
-
-HOST_GO_DEPENDENCIES = host-go-bootstrap-stage3
 HOST_GO_GOPATH = $(HOST_DIR)/share/go-path
 HOST_GO_HOST_CACHE = $(HOST_DIR)/share/host-go-cache
 HOST_GO_ROOT = $(HOST_DIR)/lib/go
@@ -96,15 +89,6 @@ else
 HOST_GO_CGO_ENABLED = 0
 endif
 
-HOST_GO_CROSS_ENV = \
-	CC_FOR_TARGET="$(TARGET_CC)" \
-	CXX_FOR_TARGET="$(TARGET_CXX)" \
-	GOOS="linux" \
-	GOARCH=$(GO_GOARCH) \
-	$(if $(GO_GO386),GO386=$(GO_GO386)) \
-	$(if $(GO_GOARM),GOARM=$(GO_GOARM)) \
-	GO_ASSUME_CROSSCOMPILING=1
-
 else # !BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 # host-go can still be used to build packages for the host. No need to set all
 # the arch stuff since we will not be cross-compiling.
@@ -123,27 +107,7 @@ HOST_GO_HOST_ENV = \
 	CGO_CXXFLAGS="$(HOST_CXXFLAGS)" \
 	CGO_LDFLAGS="$(HOST_LDFLAGS)"
 
-# The go build system is not compatible with ccache, so use
-# HOSTCC_NOCCACHE.  See https://github.com/golang/go/issues/11685.
-HOST_GO_MAKE_ENV = \
-	GO111MODULE=off \
-	GOCACHE=$(HOST_GO_HOST_CACHE) \
-	GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_STAGE3_ROOT) \
-	GOROOT_FINAL=$(HOST_GO_ROOT) \
-	GOROOT="$(@D)" \
-	GOBIN="$(@D)/bin" \
-	GOOS=linux \
-	CC=$(HOSTCC_NOCCACHE) \
-	CXX=$(HOSTCXX_NOCCACHE) \
-	CGO_ENABLED=$(HOST_GO_CGO_ENABLED) \
-	$(HOST_GO_CROSS_ENV)
-
-define HOST_GO_BUILD_CMDS
-	cd $(@D)/src && \
-		$(HOST_GO_MAKE_ENV) ./make.bash $(if $(VERBOSE),-v)
-endef
-
-define HOST_GO_INSTALL_CMDS
+define GO_BINARIES_INSTALL
 	$(INSTALL) -D -m 0755 $(@D)/bin/go $(HOST_GO_ROOT)/bin/go
 	$(INSTALL) -D -m 0755 $(@D)/bin/gofmt $(HOST_GO_ROOT)/bin/gofmt
 
@@ -165,4 +129,4 @@ define HOST_GO_INSTALL_CMDS
 	find $(HOST_GO_ROOT) -type f -exec touch -r $(@D)/bin/go {} \;
 endef
 
-$(eval $(host-generic-package))
+include $(sort $(wildcard package/go/*/*.mk))
diff --git a/package/go/0001-build.go-explicit-option-for-crosscompilation.patch b/package/go/go/0001-build.go-explicit-option-for-crosscompilation.patch
similarity index 100%
rename from package/go/0001-build.go-explicit-option-for-crosscompilation.patch
rename to package/go/go/0001-build.go-explicit-option-for-crosscompilation.patch
diff --git a/package/go/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch b/package/go/go/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch
similarity index 100%
rename from package/go/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch
rename to package/go/go/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch
diff --git a/package/go/go.hash b/package/go/go/go.hash
similarity index 100%
rename from package/go/go.hash
rename to package/go/go/go.hash
diff --git a/package/go/go/go.mk b/package/go/go/go.mk
new file mode 100644
index 0000000000..c3c40922f4
--- /dev/null
+++ b/package/go/go/go.mk
@@ -0,0 +1,53 @@
+################################################################################
+#
+# go
+#
+################################################################################
+
+GO_SITE = https://storage.googleapis.com/golang
+GO_SOURCE = go$(GO_VERSION).src.tar.gz
+
+GO_LICENSE = BSD-3-Clause
+GO_LICENSE_FILES = LICENSE
+GO_CPE_ID_VENDOR = golang
+
+HOST_GO_DEPENDENCIES = host-go-bootstrap-stage3
+
+ifeq ($(BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS),y)
+
+HOST_GO_CROSS_ENV = \
+	CC_FOR_TARGET="$(TARGET_CC)" \
+	CXX_FOR_TARGET="$(TARGET_CXX)" \
+	GOOS="linux" \
+	GOARCH=$(GO_GOARCH) \
+	$(if $(GO_GO386),GO386=$(GO_GO386)) \
+	$(if $(GO_GOARM),GOARM=$(GO_GOARM)) \
+	GO_ASSUME_CROSSCOMPILING=1
+
+endif
+
+# The go build system is not compatible with ccache, so use
+# HOSTCC_NOCCACHE.  See https://github.com/golang/go/issues/11685.
+HOST_GO_MAKE_ENV = \
+	GO111MODULE=off \
+	GOCACHE=$(HOST_GO_HOST_CACHE) \
+	GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_STAGE3_ROOT) \
+	GOROOT_FINAL=$(HOST_GO_ROOT) \
+	GOROOT="$(@D)" \
+	GOBIN="$(@D)/bin" \
+	GOOS=linux \
+	CC=$(HOSTCC_NOCCACHE) \
+	CXX=$(HOSTCXX_NOCCACHE) \
+	CGO_ENABLED=$(HOST_GO_CGO_ENABLED) \
+	$(HOST_GO_CROSS_ENV)
+
+define HOST_GO_BUILD_CMDS
+	cd $(@D)/src && \
+		$(HOST_GO_MAKE_ENV) ./make.bash $(if $(VERBOSE),-v)
+endef
+
+define HOST_GO_INSTALL_CMDS
+	$(GO_BINARIES_INSTALL)
+endef
+
+$(eval $(host-generic-package))
-- 
2.44.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [Buildroot] [PATCH 3/6] package/go: make host package a virtual package
  2024-04-12 13:26 [Buildroot] [PATCH 0/6] support for a pre-built Go compiler Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 1/6] DEVELOPERS: add Thomas Perale for 'go' Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 2/6] package/go: new subdirectory for go variants Thomas Perale via buildroot
@ 2024-04-12 13:26 ` Thomas Perale via buildroot
  2024-04-12 20:00   ` Yann E. MORIN
  2024-04-12 13:26 ` [Buildroot] [PATCH 4/6] package/pkg-golang: select go compiler Thomas Perale via buildroot
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 13+ messages in thread
From: Thomas Perale via buildroot @ 2024-04-12 13:26 UTC (permalink / raw)
  To: buildroot
  Cc: Christian Stewart, Geoff Levand, Anisse Astier, Thomas Perale,
	Angelo Compagnucci, Marcin Niestroj, Thomas Petazzoni,
	Romain Naour, Fabio Estevam, Yann E . MORIN

Turns host-go into a virtual package, which has two providers:

 - host-go-src, which builds host-go from source based on the same logic
   that was previously used in package/go/go/go.mk, now moved to
   package/go/go-src/go-src.mk to remove any ambiguity on the role of
   the package.

 - host-go-bin, which will be introduced in the next commit and installs
   a pre-built Go compiler.

The usage of a virtual package enables support for pre-built Go
compiler, which reduces the build time for of host-go.
A similar solution is proposed for host-rust.

By default, host-go is built from sources to keep the same behavior as
the former version.

Signed-off-by: Thomas Perale <thomas.perale@mind.be>
---
 .checkpackageignore                           |  2 +-
 package/go/Config.in.host                     | 32 +++++++++++++++++++
 ...explicit-option-for-crosscompilation.patch |  0
 ...ldvcs-false-when-building-go-bootstr.patch |  0
 package/go/{go/go.hash => go-src/go-src.hash} |  0
 package/go/{go/go.mk => go-src/go-src.mk}     | 25 ++++++++-------
 package/go/go.mk                              |  2 ++
 7 files changed, 48 insertions(+), 13 deletions(-)
 rename package/go/{go => go-src}/0001-build.go-explicit-option-for-crosscompilation.patch (100%)
 rename package/go/{go => go-src}/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch (100%)
 rename package/go/{go/go.hash => go-src/go-src.hash} (100%)
 rename package/go/{go/go.mk => go-src/go-src.mk} (68%)

diff --git a/.checkpackageignore b/.checkpackageignore
index b3eab26071..69681bfeaa 100644
--- a/.checkpackageignore
+++ b/.checkpackageignore
@@ -465,7 +465,7 @@ package/glorytun/0002-aegis256.c-fix-aarch64-build-with-uclibc.patch lib_patch.U
 package/gnu-efi/0001-Make.defaults-don-t-override-ARCH-when-cross-compili.patch lib_patch.Upstream
 package/gnupg/0001-build-Always-use-EXTERN_UNLESS_MAIN_MODULE-pattern.patch lib_patch.Upstream
 package/gnuplot/0001-configure-add-without-demo-option.patch lib_patch.Upstream
-package/go/go/0001-build.go-explicit-option-for-crosscompilation.patch lib_patch.Upstream
+package/go/go-src/0001-build.go-explicit-option-for-crosscompilation.patch lib_patch.Upstream
 package/gob2/0001-dont-include-from-prefix.patch lib_patch.Upstream
 package/gobject-introspection/0001-disable-tests.patch lib_patch.Upstream
 package/gobject-introspection/0002-Add-rpath-links-to-ccompiler.patch lib_patch.Upstream
diff --git a/package/go/Config.in.host b/package/go/Config.in.host
index 7edf45850d..a213ee94bf 100644
--- a/package/go/Config.in.host
+++ b/package/go/Config.in.host
@@ -32,6 +32,38 @@ config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
 	default y
 	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
 
+config BR2_PACKAGE_HOST_GO
+	bool "host go compiler"
+	depends on BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
+	help
+	  Compiler for the Go language
+
+	  https://go.dev
+
+if BR2_PACKAGE_HOST_GO
+
+choice
+	prompt "Go compiler variant"
+	default BR2_PACKAGE_HOST_GO_SRC
+	help
+	  Select a Go compiler variant.
+
+	  Default to 'host-go-src'.
+
+config BR2_PACKAGE_HOST_GO_SRC
+	bool "host go (source)"
+	help
+	  This package will build the go compiler for the host.
+
+endchoice
+
+config BR2_PACKAGE_PROVIDES_HOST_GO
+	string
+	# Default to host-go-src
+	default "host-go-src" if BR2_PACKAGE_HOST_GO_SRC
+
+endif
+
 source "package/go/go-bootstrap-stage1/Config.in.host"
 source "package/go/go-bootstrap-stage2/Config.in.host"
 source "package/go/go-bootstrap-stage3/Config.in.host"
diff --git a/package/go/go/0001-build.go-explicit-option-for-crosscompilation.patch b/package/go/go-src/0001-build.go-explicit-option-for-crosscompilation.patch
similarity index 100%
rename from package/go/go/0001-build.go-explicit-option-for-crosscompilation.patch
rename to package/go/go-src/0001-build.go-explicit-option-for-crosscompilation.patch
diff --git a/package/go/go/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch b/package/go/go-src/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch
similarity index 100%
rename from package/go/go/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch
rename to package/go/go-src/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch
diff --git a/package/go/go/go.hash b/package/go/go-src/go-src.hash
similarity index 100%
rename from package/go/go/go.hash
rename to package/go/go-src/go-src.hash
diff --git a/package/go/go/go.mk b/package/go/go-src/go-src.mk
similarity index 68%
rename from package/go/go/go.mk
rename to package/go/go-src/go-src.mk
index c3c40922f4..9d73c7c692 100644
--- a/package/go/go/go.mk
+++ b/package/go/go-src/go-src.mk
@@ -1,21 +1,22 @@
 ################################################################################
 #
-# go
+# go-src
 #
 ################################################################################
 
-GO_SITE = https://storage.googleapis.com/golang
-GO_SOURCE = go$(GO_VERSION).src.tar.gz
+GO_SRC_SITE = https://storage.googleapis.com/golang
+GO_SRC_SOURCE = go$(GO_VERSION).src.tar.gz
 
-GO_LICENSE = BSD-3-Clause
-GO_LICENSE_FILES = LICENSE
-GO_CPE_ID_VENDOR = golang
+GO_SRC_LICENSE = BSD-3-Clause
+GO_SRC_LICENSE_FILES = LICENSE
+GO_SRC_CPE_ID_VENDOR = golang
 
-HOST_GO_DEPENDENCIES = host-go-bootstrap-stage3
+HOST_GO_SRC_PROVIDES = host-go
+HOST_GO_SRC_DEPENDENCIES = host-go-bootstrap-stage3
 
 ifeq ($(BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS),y)
 
-HOST_GO_CROSS_ENV = \
+HOST_GO_SRC_CROSS_ENV = \
 	CC_FOR_TARGET="$(TARGET_CC)" \
 	CXX_FOR_TARGET="$(TARGET_CXX)" \
 	GOOS="linux" \
@@ -28,7 +29,7 @@ endif
 
 # The go build system is not compatible with ccache, so use
 # HOSTCC_NOCCACHE.  See https://github.com/golang/go/issues/11685.
-HOST_GO_MAKE_ENV = \
+HOST_GO_SRC_MAKE_ENV = \
 	GO111MODULE=off \
 	GOCACHE=$(HOST_GO_HOST_CACHE) \
 	GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_STAGE3_ROOT) \
@@ -41,12 +42,12 @@ HOST_GO_MAKE_ENV = \
 	CGO_ENABLED=$(HOST_GO_CGO_ENABLED) \
 	$(HOST_GO_CROSS_ENV)
 
-define HOST_GO_BUILD_CMDS
+define HOST_GO_SRC_BUILD_CMDS
 	cd $(@D)/src && \
-		$(HOST_GO_MAKE_ENV) ./make.bash $(if $(VERBOSE),-v)
+		$(HOST_GO_SRC_MAKE_ENV) ./make.bash $(if $(VERBOSE),-v)
 endef
 
-define HOST_GO_INSTALL_CMDS
+define HOST_GO_SRC_INSTALL_CMDS
 	$(GO_BINARIES_INSTALL)
 endef
 
diff --git a/package/go/go.mk b/package/go/go.mk
index 104a4a73fb..21cdb29f8f 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -129,4 +129,6 @@ define GO_BINARIES_INSTALL
 	find $(HOST_GO_ROOT) -type f -exec touch -r $(@D)/bin/go {} \;
 endef
 
+$(eval $(host-virtual-package))
+
 include $(sort $(wildcard package/go/*/*.mk))
-- 
2.44.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [Buildroot] [PATCH 4/6] package/pkg-golang: select go compiler
  2024-04-12 13:26 [Buildroot] [PATCH 0/6] support for a pre-built Go compiler Thomas Perale via buildroot
                   ` (2 preceding siblings ...)
  2024-04-12 13:26 ` [Buildroot] [PATCH 3/6] package/go: make host package a virtual package Thomas Perale via buildroot
@ 2024-04-12 13:26 ` Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 5/6] package/go/go-bin: new go provider Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 6/6] support/testing: add tests for Go providers Thomas Perale via buildroot
  5 siblings, 0 replies; 13+ messages in thread
From: Thomas Perale via buildroot @ 2024-04-12 13:26 UTC (permalink / raw)
  To: buildroot
  Cc: Christian Stewart, Geoff Levand, Anisse Astier, Thomas Perale,
	Angelo Compagnucci, Marcin Niestroj, Thomas Petazzoni,
	Romain Naour, Fabio Estevam, Yann E . MORIN

Select the 'go' virtual package in every package built by the Go
compiler. This maintains the correct dependencies in 'menuconfig'.
The same pattern is currently used for Rust packages.

Signed-off-by: Thomas Perale <thomas.perale@mind.be>
---
 package/balena-engine/Config.in        | 1 +
 package/cni-plugins/Config.in          | 1 +
 package/containerd/Config.in           | 1 +
 package/crucible/Config.in             | 1 +
 package/delve/Config.in                | 1 +
 package/delve/Config.in.host           | 1 +
 package/docker-cli/Config.in           | 1 +
 package/docker-compose/Config.in       | 1 +
 package/docker-engine/Config.in        | 1 +
 package/embiggen-disk/Config.in        | 1 +
 package/flannel/Config.in              | 1 +
 package/gitlab-runner/Config.in        | 1 +
 package/gocryptfs/Config.in            | 1 +
 package/mender-artifact/Config.in.host | 1 +
 package/mender-connect/Config.in       | 1 +
 package/mender/Config.in               | 1 +
 package/moby-buildkit/Config.in        | 1 +
 package/moby-buildkit/Config.in.host   | 1 +
 package/nerdctl/Config.in              | 1 +
 package/runc/Config.in                 | 1 +
 package/runc/Config.in.host            | 1 +
 package/tinifier/Config.in             | 1 +
 package/wtfutil/Config.in              | 1 +
 23 files changed, 23 insertions(+)

diff --git a/package/balena-engine/Config.in b/package/balena-engine/Config.in
index 02f694515e..5966b33f58 100644
--- a/package/balena-engine/Config.in
+++ b/package/balena-engine/Config.in
@@ -5,6 +5,7 @@ config BR2_PACKAGE_BALENA_ENGINE
 	depends on !BR2_TOOLCHAIN_USES_UCLIBC # no fexecve
 	depends on BR2_USE_MMU # util-linux
 	select BR2_PACKAGE_CGROUPFS_MOUNT if !BR2_PACKAGE_SYSTEMD # runtime
+	select BR2_PACKAGE_HOST_GO
 	select BR2_PACKAGE_IPTABLES # runtime
 	select BR2_PACKAGE_UTIL_LINUX # runtime
 	select BR2_PACKAGE_UTIL_LINUX_BINARIES # runtime
diff --git a/package/cni-plugins/Config.in b/package/cni-plugins/Config.in
index 25e6ff534d..6d4ca0a5b4 100644
--- a/package/cni-plugins/Config.in
+++ b/package/cni-plugins/Config.in
@@ -5,6 +5,7 @@ config BR2_PACKAGE_CNI_PLUGINS
 	depends on BR2_TOOLCHAIN_HAS_THREADS
 	depends on !BR2_TOOLCHAIN_USES_UCLIBC # no fexecve
 	depends on BR2_USE_MMU
+	select BR2_PACKAGE_HOST_GO
 	help
 	  Container Networking Interface plugins.
 
diff --git a/package/containerd/Config.in b/package/containerd/Config.in
index 2dd04b109c..f014c56e10 100644
--- a/package/containerd/Config.in
+++ b/package/containerd/Config.in
@@ -5,6 +5,7 @@ config BR2_PACKAGE_CONTAINERD
 	depends on BR2_TOOLCHAIN_HAS_THREADS
 	depends on !BR2_TOOLCHAIN_USES_UCLIBC # runc
 	depends on BR2_USE_MMU # util-linux
+	select BR2_PACKAGE_HOST_GO
 	select BR2_PACKAGE_RUNC if !BR2_PACKAGE_CRUN # runtime dependency
 	select BR2_PACKAGE_UTIL_LINUX # runtime dependency
 	select BR2_PACKAGE_UTIL_LINUX_BINARIES
diff --git a/package/crucible/Config.in b/package/crucible/Config.in
index b27eb31ae5..1e1b29c3df 100644
--- a/package/crucible/Config.in
+++ b/package/crucible/Config.in
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_CRUCIBLE
 	bool "crucible"
 	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  Crucible is a One-Time-Programmable (OTP) fusing tool for the
 	  i.MX family.
diff --git a/package/delve/Config.in b/package/delve/Config.in
index 9428e4016a..288dc1ea15 100644
--- a/package/delve/Config.in
+++ b/package/delve/Config.in
@@ -7,6 +7,7 @@ config BR2_PACKAGE_DELVE_ARCH_SUPPORTS
 	default y if BR2_x86_64
 	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
 
 config BR2_PACKAGE_DELVE
 	bool "delve"
diff --git a/package/delve/Config.in.host b/package/delve/Config.in.host
index dab5ca8914..0938ab0659 100644
--- a/package/delve/Config.in.host
+++ b/package/delve/Config.in.host
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_HOST_DELVE
 	bool "host delve"
 	depends on BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  Delve is a debugger for the Go programming language.
 
diff --git a/package/docker-cli/Config.in b/package/docker-cli/Config.in
index 6f187c9d22..52e7f7c871 100644
--- a/package/docker-cli/Config.in
+++ b/package/docker-cli/Config.in
@@ -3,6 +3,7 @@ config BR2_PACKAGE_DOCKER_CLI
 	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 	depends on BR2_TOOLCHAIN_HAS_THREADS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  Docker is a platform to build, ship,
 	  and run applications as lightweight containers.
diff --git a/package/docker-compose/Config.in b/package/docker-compose/Config.in
index 59128f7a19..f3560e402b 100644
--- a/package/docker-compose/Config.in
+++ b/package/docker-compose/Config.in
@@ -4,6 +4,7 @@ config BR2_PACKAGE_DOCKER_COMPOSE
 	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 	depends on BR2_TOOLCHAIN_HAS_THREADS
 	depends on BR2_PACKAGE_DOCKER_CLI
+	select BR2_PACKAGE_HOST_GO
 	help
 	  Multi-container applications with the Docker CLI.
 
diff --git a/package/docker-engine/Config.in b/package/docker-engine/Config.in
index d48ffe6484..49a002a647 100644
--- a/package/docker-engine/Config.in
+++ b/package/docker-engine/Config.in
@@ -9,6 +9,7 @@ config BR2_PACKAGE_DOCKER_ENGINE
 	depends on BR2_TOOLCHAIN_HEADERS_AT_LEAST_3_17 # libseccomp
 	select BR2_PACKAGE_CGROUPFS_MOUNT if !BR2_PACKAGE_SYSTEMD # runtime dependency
 	select BR2_PACKAGE_CONTAINERD # runtime dependency
+	select BR2_PACKAGE_HOST_GO
 	select BR2_PACKAGE_IPTABLES # runtime dependency
 	select BR2_PACKAGE_LIBSECCOMP
 	help
diff --git a/package/embiggen-disk/Config.in b/package/embiggen-disk/Config.in
index 667d71c86e..c323fb1b6d 100644
--- a/package/embiggen-disk/Config.in
+++ b/package/embiggen-disk/Config.in
@@ -5,6 +5,7 @@ config BR2_PACKAGE_EMBIGGEN_DISK
 	depends on BR2_TOOLCHAIN_HAS_THREADS
 	depends on !BR2_TOOLCHAIN_USES_UCLIBC # no fexecve
 	depends on BR2_USE_MMU # util-linux
+	select BR2_PACKAGE_HOST_GO
 	select BR2_PACKAGE_UTIL_LINUX # sfdisk
 	select BR2_PACKAGE_UTIL_LINUX_BINARIES # sfdisk
 	help
diff --git a/package/flannel/Config.in b/package/flannel/Config.in
index eb8d798f48..f37cfaa9cc 100644
--- a/package/flannel/Config.in
+++ b/package/flannel/Config.in
@@ -8,6 +8,7 @@ config BR2_PACKAGE_FLANNEL
 	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 	depends on BR2_TOOLCHAIN_HAS_THREADS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  Flannel is a virtual network that gives a subnet to each
 	  host for use with container runtimes.
diff --git a/package/gitlab-runner/Config.in b/package/gitlab-runner/Config.in
index 4289613ab9..cfe8c941bc 100644
--- a/package/gitlab-runner/Config.in
+++ b/package/gitlab-runner/Config.in
@@ -9,6 +9,7 @@ config BR2_PACKAGE_GITLAB_RUNNER
 	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # bash
 	select BR2_PACKAGE_CA_CERTIFICATES # runtime
 	select BR2_PACKAGE_GIT # runtime
+	select BR2_PACKAGE_HOST_GO
 	select BR2_PACKAGE_LIBCURL # runtime
 	select BR2_PACKAGE_LIBCURL_CURL # runtime
 	select BR2_PACKAGE_LIBCURL_FORCE_TLS # runtime
diff --git a/package/gocryptfs/Config.in b/package/gocryptfs/Config.in
index 2630cafdcf..2c70b9b21b 100644
--- a/package/gocryptfs/Config.in
+++ b/package/gocryptfs/Config.in
@@ -2,6 +2,7 @@ config BR2_PACKAGE_GOCRYPTFS
 	bool "gocryptfs"
 	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	depends on BR2_TOOLCHAIN_HAS_THREADS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  gocryptfs is an encrypted FUSE overlay filesystem.
 
diff --git a/package/mender-artifact/Config.in.host b/package/mender-artifact/Config.in.host
index c7be0ed261..39353b23e7 100644
--- a/package/mender-artifact/Config.in.host
+++ b/package/mender-artifact/Config.in.host
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_HOST_MENDER_ARTIFACT
 	bool "host mender-artifact"
 	depends on BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  The mender-artifact tool is a CLI implementation of the
 	  Mender artifacts library.
diff --git a/package/mender-connect/Config.in b/package/mender-connect/Config.in
index 0845910880..d478e468c2 100644
--- a/package/mender-connect/Config.in
+++ b/package/mender-connect/Config.in
@@ -6,6 +6,7 @@ config BR2_PACKAGE_MENDER_CONNECT
 	depends on BR2_USE_WCHAR # libglib2 -> gettext
 	depends on BR2_USE_MMU # dbus -> fork()
 	select BR2_PACKAGE_DBUS # runtime
+	select BR2_PACKAGE_HOST_GO
 	select BR2_PACKAGE_LIBGLIB2
 	select BR2_PACKAGE_MENDER # runtime
 	select BR2_PACKAGE_OPENSSL
diff --git a/package/mender/Config.in b/package/mender/Config.in
index 7d768cebec..4831548f76 100644
--- a/package/mender/Config.in
+++ b/package/mender/Config.in
@@ -3,6 +3,7 @@ config BR2_PACKAGE_MENDER
 	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 	depends on BR2_TOOLCHAIN_HAS_THREADS
+	select BR2_PACKAGE_HOST_GO
 	select BR2_PACKAGE_OPENSSL
 	select BR2_PACKAGE_OPENSSL_FORCE_LIBOPENSSL
 	select BR2_PACKAGE_LIBOPENSSL_ENABLE_MD4
diff --git a/package/moby-buildkit/Config.in b/package/moby-buildkit/Config.in
index 93c852bcdb..1a0b599097 100644
--- a/package/moby-buildkit/Config.in
+++ b/package/moby-buildkit/Config.in
@@ -6,6 +6,7 @@ config BR2_PACKAGE_MOBY_BUILDKIT
 	depends on !BR2_TOOLCHAIN_USES_UCLIBC
 	depends on BR2_USE_MMU
 	select BR2_PACKAGE_CGROUPFS_MOUNT if !BR2_PACKAGE_SYSTEMD # runtime dependency
+	select BR2_PACKAGE_HOST_GO
 	help
 	  BuildKit is a toolkit for converting source code to build
 	  artifacts in an efficient, expressive and repeatable manner.
diff --git a/package/moby-buildkit/Config.in.host b/package/moby-buildkit/Config.in.host
index a3b18548f8..d4eb0d74c3 100644
--- a/package/moby-buildkit/Config.in.host
+++ b/package/moby-buildkit/Config.in.host
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_HOST_MOBY_BUILDKIT
 	bool "host moby-buildkit"
 	depends on BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  BuildKit is a toolkit for converting source code to build
 	  artifacts in an efficient expressive and repeatable manner.
diff --git a/package/nerdctl/Config.in b/package/nerdctl/Config.in
index 969b9bb40e..062b36ab85 100644
--- a/package/nerdctl/Config.in
+++ b/package/nerdctl/Config.in
@@ -6,6 +6,7 @@ config BR2_PACKAGE_NERDCTL
 	depends on !BR2_TOOLCHAIN_USES_UCLIBC # containerd
 	depends on BR2_USE_MMU # util-linux
 	select BR2_PACKAGE_CONTAINERD # runtime
+	select BR2_PACKAGE_HOST_GO
 	help
 	  Docker-compatible CLI for containerd, controlling runc.
 
diff --git a/package/runc/Config.in b/package/runc/Config.in
index fe527b1954..73eac72527 100644
--- a/package/runc/Config.in
+++ b/package/runc/Config.in
@@ -4,6 +4,7 @@ config BR2_PACKAGE_RUNC
 	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 	depends on BR2_TOOLCHAIN_HAS_THREADS
 	depends on !BR2_TOOLCHAIN_USES_UCLIBC # no fexecve
+	select BR2_PACKAGE_HOST_GO
 	help
 	  runC is a CLI tool for spawning and running containers
 	  according to the OCI specification.
diff --git a/package/runc/Config.in.host b/package/runc/Config.in.host
index 4b2555e029..78193cb499 100644
--- a/package/runc/Config.in.host
+++ b/package/runc/Config.in.host
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_HOST_RUNC
 	bool "host runc"
 	depends on BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  runC is a CLI tool for spawning and running containers
 	  according to the OCI specification.
diff --git a/package/tinifier/Config.in b/package/tinifier/Config.in
index fbadfe6bd9..eae8b8f499 100644
--- a/package/tinifier/Config.in
+++ b/package/tinifier/Config.in
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_TINIFIER
 	bool "tinifier"
 	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  CLI tool for images compressing
 
diff --git a/package/wtfutil/Config.in b/package/wtfutil/Config.in
index 8c3efcea5d..3fd8c41bab 100644
--- a/package/wtfutil/Config.in
+++ b/package/wtfutil/Config.in
@@ -2,6 +2,7 @@ config BR2_PACKAGE_WTFUTIL
 	bool "wtfutil"
 	depends on BR2_USE_MMU  # fork()
 	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
 	help
 	  WTF is the personal information dashboard for your terminal.
 
-- 
2.44.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [Buildroot] [PATCH 5/6] package/go/go-bin: new go provider
  2024-04-12 13:26 [Buildroot] [PATCH 0/6] support for a pre-built Go compiler Thomas Perale via buildroot
                   ` (3 preceding siblings ...)
  2024-04-12 13:26 ` [Buildroot] [PATCH 4/6] package/pkg-golang: select go compiler Thomas Perale via buildroot
@ 2024-04-12 13:26 ` Thomas Perale via buildroot
  2024-04-12 13:26 ` [Buildroot] [PATCH 6/6] support/testing: add tests for Go providers Thomas Perale via buildroot
  5 siblings, 0 replies; 13+ messages in thread
From: Thomas Perale via buildroot @ 2024-04-12 13:26 UTC (permalink / raw)
  To: buildroot
  Cc: Christian Stewart, Geoff Levand, Anisse Astier, Thomas Perale,
	Angelo Compagnucci, Marcin Niestroj, Thomas Petazzoni,
	Romain Naour, Fabio Estevam, Yann E . MORIN

This package provides a pre-built version for the host-go virtual
package introduced in the previous commits.

host-go-bin install a pre-built version of the Go compiler.

The menuconfig entry for host-go is updated to expose the host-go-bin
provider. The dependencies are set as such as if host-go-src does not
support the host architecture, it will automatically fall back to
host-go-bin and vice versa.

Signed-off-by: Thomas Perale <thomas.perale@mind.be>
---
 package/go/Config.in.host        | 14 ++++++++++++--
 package/go/go-bin/Config.in.host | 14 ++++++++++++++
 package/go/go-bin/go-bin.hash    |  8 ++++++++
 package/go/go-bin/go-bin.mk      | 19 +++++++++++++++++++
 4 files changed, 53 insertions(+), 2 deletions(-)
 create mode 100644 package/go/go-bin/Config.in.host
 create mode 100644 package/go/go-bin/go-bin.hash
 create mode 100644 package/go/go-bin/go-bin.mk

diff --git a/package/go/Config.in.host b/package/go/Config.in.host
index a213ee94bf..c98e41483d 100644
--- a/package/go/Config.in.host
+++ b/package/go/Config.in.host
@@ -30,7 +30,7 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS || BR2_PACKAGE_HOST_GO_BIN_HOST_ARCH_SUPPORTS
 
 config BR2_PACKAGE_HOST_GO
 	bool "host go compiler"
@@ -44,7 +44,8 @@ if BR2_PACKAGE_HOST_GO
 
 choice
 	prompt "Go compiler variant"
-	default BR2_PACKAGE_HOST_GO_SRC
+	default BR2_PACKAGE_HOST_GO_SRC if BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
+	default BR2_PACKAGE_HOST_GO_BIN if BR2_PACKAGE_HOST_GO_BIN_HOST_ARCH_SUPPORTS
 	help
 	  Select a Go compiler variant.
 
@@ -52,18 +53,27 @@ choice
 
 config BR2_PACKAGE_HOST_GO_SRC
 	bool "host go (source)"
+	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
 	help
 	  This package will build the go compiler for the host.
 
+config BR2_PACKAGE_HOST_GO_BIN
+	bool "host go (pre-built)"
+	depends on BR2_PACKAGE_HOST_GO_BIN_HOST_ARCH_SUPPORTS
+	help
+	  This package will install pre-built versions of the compiler
+
 endchoice
 
 config BR2_PACKAGE_PROVIDES_HOST_GO
 	string
 	# Default to host-go-src
 	default "host-go-src" if BR2_PACKAGE_HOST_GO_SRC
+	default "host-go-bin" if BR2_PACKAGE_HOST_GO_BIN
 
 endif
 
+source "package/go/go-bin/Config.in.host"
 source "package/go/go-bootstrap-stage1/Config.in.host"
 source "package/go/go-bootstrap-stage2/Config.in.host"
 source "package/go/go-bootstrap-stage3/Config.in.host"
diff --git a/package/go/go-bin/Config.in.host b/package/go/go-bin/Config.in.host
new file mode 100644
index 0000000000..b259afa197
--- /dev/null
+++ b/package/go/go-bin/Config.in.host
@@ -0,0 +1,14 @@
+config BR2_PACKAGE_HOST_GO_BIN_HOST_ARCH
+	string "Translate the HOSTARCH into the architecture name used by the Go compiler"
+	default armv6l   if BR2_HOSTARCH = "arm"
+	default arm64    if BR2_HOSTARCH = "aarch64"
+	default ppc64le  if BR2_HOSTARCH = "powerpc64le"
+	default s390x    if BR2_HOSTARCH = "s390x"
+	default 386      if BR2_HOSTARCH = "x86"
+	default amd64    if BR2_HOSTARCH = "x86_64"
+	default ""
+
+config BR2_PACKAGE_HOST_GO_BIN_HOST_ARCH_SUPPORTS
+	bool
+	default y
+	depends on BR2_PACKAGE_HOST_GO_BIN_HOST_ARCH != ""
diff --git a/package/go/go-bin/go-bin.hash b/package/go/go-bin/go-bin.hash
new file mode 100644
index 0000000000..298503c156
--- /dev/null
+++ b/package/go/go-bin/go-bin.hash
@@ -0,0 +1,8 @@
+# sha256 checksum from https://go.dev/dl/
+sha256  586d9eb7fe0489ab297ad80dd06414997df487c5cf536c490ffeaa8d8f1807a7  go1.22.2.linux-386.tar.gz
+sha256  5901c52b7a78002aeff14a21f93e0f064f74ce1360fce51c6ee68cd471216a17  go1.22.2.linux-amd64.tar.gz
+sha256  36e720b2d564980c162a48c7e97da2e407dfcc4239e1e58d98082dfa2486a0c1  go1.22.2.linux-arm64.tar.gz
+sha256  9243dfafde06e1efe24d59df6701818e6786b4adfdf1191098050d6d023c5369  go1.22.2.linux-armv6l.tar.gz
+sha256  32ea3dfef75d6a4d42a28a315148ba54c4f6f9a8dc93a522d421d13df0e6c3b7  go1.22.2.linux-ppc64le.tar.gz
+sha256  2b39019481c28c560d65e9811a478ae10e3ef765e0f59af362031d386a71bfef  go1.22.2.linux-s390x.tar.gz
+sha256  2d36597f7117c38b006835ae7f537487207d8ec407aa9d9980794b2030cbc067  LICENSE
diff --git a/package/go/go-bin/go-bin.mk b/package/go/go-bin/go-bin.mk
new file mode 100644
index 0000000000..6c5c1b6485
--- /dev/null
+++ b/package/go/go-bin/go-bin.mk
@@ -0,0 +1,19 @@
+################################################################################
+#
+# go-bin
+#
+################################################################################
+
+GO_BIN_SITE = https://go.dev/dl
+GO_BIN_LICENSE = BSD-3-Clause
+GO_BIN_LICENSE_FILES = LICENSE
+
+HOST_GO_BIN_PROVIDES = host-go
+
+HOST_GO_BIN_SOURCE = go$(GO_VERSION).linux-$(call qstrip, $(BR2_PACKAGE_HOST_GO_BIN_HOST_ARCH)).tar.gz
+
+define HOST_GO_BIN_INSTALL_CMDS
+	$(GO_BINARIES_INSTALL)
+endef
+
+$(eval $(host-generic-package))
-- 
2.44.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [Buildroot] [PATCH 6/6] support/testing: add tests for Go providers
  2024-04-12 13:26 [Buildroot] [PATCH 0/6] support for a pre-built Go compiler Thomas Perale via buildroot
                   ` (4 preceding siblings ...)
  2024-04-12 13:26 ` [Buildroot] [PATCH 5/6] package/go/go-bin: new go provider Thomas Perale via buildroot
@ 2024-04-12 13:26 ` Thomas Perale via buildroot
  5 siblings, 0 replies; 13+ messages in thread
From: Thomas Perale via buildroot @ 2024-04-12 13:26 UTC (permalink / raw)
  To: buildroot
  Cc: Christian Stewart, Geoff Levand, Anisse Astier, Thomas Perale,
	Angelo Compagnucci, Marcin Niestroj, Thomas Petazzoni,
	Romain Naour, Fabio Estevam, Yann E . MORIN

Test the two providers of host-go to build a Go package.

- TestGoBin tests host-go-bin
- TestGoSource tests host-go-src

The tests consist of building and installing a Go package in the root
file system of an ARM vexpress QEMU system.
The tests pass if the program runs on the target.

Signed-off-by: Thomas Perale <thomas.perale@mind.be>
---
 support/testing/tests/package/test_go.py | 41 ++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 support/testing/tests/package/test_go.py

diff --git a/support/testing/tests/package/test_go.py b/support/testing/tests/package/test_go.py
new file mode 100644
index 0000000000..305f8aeb3b
--- /dev/null
+++ b/support/testing/tests/package/test_go.py
@@ -0,0 +1,41 @@
+import os
+
+import infra.basetest
+
+
+class TestGoBase(infra.basetest.BRTest):
+
+    def login(self):
+        cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-initrd", cpio_file])
+        self.emulator.login()
+
+
+class TestGoBin(TestGoBase):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        """
+        BR2_TARGET_ROOTFS_CPIO=y
+        BR2_PACKAGE_HOST_GO=y
+        BR2_PACKAGE_HOST_GO_BIN=y
+        BR2_PACKAGE_TINIFIER=y
+        """
+
+    def test_run(self):
+        self.login()
+        self.assertRunOk("tinifier -h")
+
+
+class TestGoSource(TestGoBase):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        """
+        BR2_TARGET_ROOTFS_CPIO=y
+        BR2_PACKAGE_HOST_GO=y
+        BR2_PACKAGE_HOST_GO_SRC=y
+        BR2_PACKAGE_TINIFIER=y
+        """
+
+    def test_run(self):
+        self.login()
+        self.assertRunOk("tinifier -h")
-- 
2.44.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* Re: [Buildroot] [PATCH 3/6] package/go: make host package a virtual package
  2024-04-12 13:26 ` [Buildroot] [PATCH 3/6] package/go: make host package a virtual package Thomas Perale via buildroot
@ 2024-04-12 20:00   ` Yann E. MORIN
  2024-04-14  6:28     ` Christian Stewart via buildroot
  2024-04-15 12:59     ` Thomas Perale via buildroot
  0 siblings, 2 replies; 13+ messages in thread
From: Yann E. MORIN @ 2024-04-12 20:00 UTC (permalink / raw)
  To: Thomas Perale
  Cc: Thomas Petazzoni, Geoff Levand, Anisse Astier,
	Angelo Compagnucci, Marcin Niestroj, Christian Stewart,
	buildroot, Romain Naour, Fabio Estevam, Thomas Perale

Thomas, All,

Thanks for this series, it really looks good! :-)

I anyway have a few comments and questions, please see below.

On 2024-04-12 15:26 +0200, Thomas Perale via buildroot spake thusly:
> Turns host-go into a virtual package, which has two providers:

With this patch, there is a single provider, so the commit log is not
correct. Instead, it should state something like:

    Turn host-go into a virtual package, which currently has a single
    provider, the existing host-go package. In later commits, we'll
    add a prebuilt, binary host go compiler as another provider.

>  - host-go-src, which builds host-go from source based on the same logic
>    that was previously used in package/go/go/go.mk, now moved to
>    package/go/go-src/go-src.mk to remove any ambiguity on the role of
>    the package.
> 
>  - host-go-bin, which will be introduced in the next commit and installs
>    a pre-built Go compiler.
> 
> The usage of a virtual package enables support for pre-built Go

s/usage/use/

> compiler, which reduces the build time for of host-go.
> A similar solution is proposed for host-rust.
> 
> By default, host-go is built from sources to keep the same behavior as
> the former version.

Since this commit only introduces a single provider, this comment does
not make sense, and belongs to the commit that adds the prebuilt go.

Not related to this patch, but still relevant to the discussion: it
would be even better if we could also have the possibility that the
stage-3 has the option of being a prebuilt go.

Indeed, it would still be interesting to be able to build the final go
from source, but use a prebuild bootstrap.

So that would allow for three situations:

  - full source: stage-1, stage-2, stage-3, and final go, all built from
    sources;

  - prebuilt stage-3 (so not stage-1 and no stage-2), but final built
    from sources;

  - no bootstrap at all, and prebuilt final go.

Of course, this is _not_ required for this series to progress.

> Signed-off-by: Thomas Perale <thomas.perale@mind.be>
> ---

> diff --git a/package/go/Config.in.host b/package/go/Config.in.host
> index 7edf45850d..a213ee94bf 100644
> --- a/package/go/Config.in.host
> +++ b/package/go/Config.in.host
> @@ -32,6 +32,38 @@ config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
>  	default y
>  	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
>  
> +config BR2_PACKAGE_HOST_GO

So now, all go packages are broken, because none selects
BR2_PACKAGE_HOST_GO, which is required for the virtual-paclage infra.

To "fix" that issue, there are two options:

  - fix all package in this very patch, but it is a bit huge, or

  - introduce BR2_PACKAGE_HOST_GO first and have packages select it;
    basically that's your next patch, but it should come first:

        config BR2_PACKAGE_HOST_GO
            bool
            depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS

        config BR2_PACKAGE_BALENA_ENGINE
            depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
            select BR2_PACKAGE_HOST_GO

    and then this patch comes in to introduce the host variant.

This way, we have a series that is bisectable.

Also, I see that you modeled the solution after the rust one, and this
is good.

However, I don't think there is a need for a prompt for
BR2_PACKAGE_HOST_GO. It can very well be a prompt-less symbol, as
packages need to select it anyway.

Also, there is (I think) no reason to enable BR2_PACKAGE_HOST_GO when no
package needs it. Post-build/fakeroot/image scripts should not need the
go compiler either, so I don;t think there is an actual need for a
prompt...

[--SNIP--]
> diff --git a/package/go/go/go.mk b/package/go/go-src/go-src.mk
> similarity index 68%
> rename from package/go/go/go.mk
> rename to package/go/go-src/go-src.mk
> index c3c40922f4..9d73c7c692 100644
> --- a/package/go/go/go.mk
> +++ b/package/go/go-src/go-src.mk
> @@ -1,21 +1,22 @@
>  ################################################################################
>  #
> -# go
> +# go-src

I haven't reviewed the s/GO/GO_SRC/ transition; that's mostly
mechanical, so I don;t expect too much surprise.

Except one point:if any of those symbol is used elsewhere, they need to
be fixed as well... But unlike rust, there does not seem like there are
packages that build go stuff as part of another ecosystem (à-la
python+rust).

Hmm... HOST_GO_COMMON_ENV is used in the pkg-golang infra, but I could
not spot anything else on a cursory look. "It should be fine!"

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [Buildroot] [PATCH 3/6] package/go: make host package a virtual package
  2024-04-12 20:00   ` Yann E. MORIN
@ 2024-04-14  6:28     ` Christian Stewart via buildroot
  2024-04-28 17:29       ` Arnout Vandecappelle via buildroot
  2024-04-15 12:59     ` Thomas Perale via buildroot
  1 sibling, 1 reply; 13+ messages in thread
From: Christian Stewart via buildroot @ 2024-04-14  6:28 UTC (permalink / raw)
  To: Yann E. MORIN
  Cc: Geoff Levand, Anisse Astier, Angelo Compagnucci, Marcin Niestroj,
	Thomas Petazzoni, buildroot, Romain Naour, Fabio Estevam,
	Thomas Perale, Thomas Perale

Hi all,

On Fri, Apr 12, 2024 at 1:01 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Thanks for this series, it really looks good! :-)

I agree that it looks good & having the go-bootstrap below the go/
package is nice.

Haven't fully reviewed it yet but will try it.

> I anyway have a few comments and questions, please see below.
>

> Not related to this patch, but still relevant to the discussion: it
> would be even better if we could also have the possibility that the
> stage-3 has the option of being a prebuilt go.

Using stage-3 as pre-built and host-go itself as compiled from source
using the pre-built toolchain as the bootstrap one is ideal.

This is so that we can control which version of Go is used for the
build for reproducibility.

Otherwise if the major version of Go installed on the host machine is
different, it might produce different results, particularly when using
cgo.

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [Buildroot] [PATCH 3/6] package/go: make host package a virtual package
  2024-04-12 20:00   ` Yann E. MORIN
  2024-04-14  6:28     ` Christian Stewart via buildroot
@ 2024-04-15 12:59     ` Thomas Perale via buildroot
  1 sibling, 0 replies; 13+ messages in thread
From: Thomas Perale via buildroot @ 2024-04-15 12:59 UTC (permalink / raw)
  To: Yann E. MORIN
  Cc: Arnout Vandecappelle, Thomas Petazzoni, Geoff Levand,
	Anisse Astier, Angelo Compagnucci, Marcin Niestroj,
	Christian Stewart, buildroot, Romain Naour, Fabio Estevam,
	Thomas Perale

Yann, Christian,

Thanks for your comments I will soon send a V2 of the patch.

> Not related to this patch, but still relevant to the discussion: it
> would be even better if we could also have the possibility that the
> stage-3 has the option of being a prebuilt go.
>
> Indeed, it would still be interesting to be able to build the final go
> from source, but use a prebuild bootstrap.
This is a good suggestion, I will add this to my TODO list and introduce 
it in a future patch series.

> Also, I see that you modeled the solution after the rust one, and this
> is good.
Do you think it's worth moving rust packages to a subdirectory structure 
in the future ?

Also, another point of the buildroot TODO list is to add support for 
prebuilt nodejs. If I get time to work on that patch series it will 
probably follow the same subdir structure.

Regards,
PERALE Thomas

On 4/12/24 10:00 PM, Yann E. MORIN wrote:
> Thomas, All,
>
> Thanks for this series, it really looks good! :-)
>
> I anyway have a few comments and questions, please see below.
>
> On 2024-04-12 15:26 +0200, Thomas Perale via buildroot spake thusly:
>> Turns host-go into a virtual package, which has two providers:
> With this patch, there is a single provider, so the commit log is not
> correct. Instead, it should state something like:
>
>      Turn host-go into a virtual package, which currently has a single
>      provider, the existing host-go package. In later commits, we'll
>      add a prebuilt, binary host go compiler as another provider.
>
>>   - host-go-src, which builds host-go from source based on the same logic
>>     that was previously used in package/go/go/go.mk, now moved to
>>     package/go/go-src/go-src.mk to remove any ambiguity on the role of
>>     the package.
>>
>>   - host-go-bin, which will be introduced in the next commit and installs
>>     a pre-built Go compiler.
>>
>> The usage of a virtual package enables support for pre-built Go
> s/usage/use/
>
>> compiler, which reduces the build time for of host-go.
>> A similar solution is proposed for host-rust.
>>
>> By default, host-go is built from sources to keep the same behavior as
>> the former version.
> Since this commit only introduces a single provider, this comment does
> not make sense, and belongs to the commit that adds the prebuilt go.
>
> Not related to this patch, but still relevant to the discussion: it
> would be even better if we could also have the possibility that the
> stage-3 has the option of being a prebuilt go.
>
> Indeed, it would still be interesting to be able to build the final go
> from source, but use a prebuild bootstrap.
>
> So that would allow for three situations:
>
>    - full source: stage-1, stage-2, stage-3, and final go, all built from
>      sources;
>
>    - prebuilt stage-3 (so not stage-1 and no stage-2), but final built
>      from sources;
>
>    - no bootstrap at all, and prebuilt final go.
>
> Of course, this is _not_ required for this series to progress.
>
>> Signed-off-by: Thomas Perale <thomas.perale@mind.be>
>> ---
>> diff --git a/package/go/Config.in.host b/package/go/Config.in.host
>> index 7edf45850d..a213ee94bf 100644
>> --- a/package/go/Config.in.host
>> +++ b/package/go/Config.in.host
>> @@ -32,6 +32,38 @@ config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
>>   	default y
>>   	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
>>   
>> +config BR2_PACKAGE_HOST_GO
> So now, all go packages are broken, because none selects
> BR2_PACKAGE_HOST_GO, which is required for the virtual-paclage infra.
>
> To "fix" that issue, there are two options:
>
>    - fix all package in this very patch, but it is a bit huge, or
>
>    - introduce BR2_PACKAGE_HOST_GO first and have packages select it;
>      basically that's your next patch, but it should come first:
>
>          config BR2_PACKAGE_HOST_GO
>              bool
>              depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
>
>          config BR2_PACKAGE_BALENA_ENGINE
>              depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
>              select BR2_PACKAGE_HOST_GO
>
>      and then this patch comes in to introduce the host variant.
>
> This way, we have a series that is bisectable.
>
> Also, I see that you modeled the solution after the rust one, and this
> is good.
>
> However, I don't think there is a need for a prompt for
> BR2_PACKAGE_HOST_GO. It can very well be a prompt-less symbol, as
> packages need to select it anyway.
>
> Also, there is (I think) no reason to enable BR2_PACKAGE_HOST_GO when no
> package needs it. Post-build/fakeroot/image scripts should not need the
> go compiler either, so I don;t think there is an actual need for a
> prompt...
>
> [--SNIP--]
>> diff --git a/package/go/go/go.mk b/package/go/go-src/go-src.mk
>> similarity index 68%
>> rename from package/go/go/go.mk
>> rename to package/go/go-src/go-src.mk
>> index c3c40922f4..9d73c7c692 100644
>> --- a/package/go/go/go.mk
>> +++ b/package/go/go-src/go-src.mk
>> @@ -1,21 +1,22 @@
>>   ################################################################################
>>   #
>> -# go
>> +# go-src
> I haven't reviewed the s/GO/GO_SRC/ transition; that's mostly
> mechanical, so I don;t expect too much surprise.
>
> Except one point:if any of those symbol is used elsewhere, they need to
> be fixed as well... But unlike rust, there does not seem like there are
> packages that build go stuff as part of another ecosystem (à-la
> python+rust).
>
> Hmm... HOST_GO_COMMON_ENV is used in the pkg-golang infra, but I could
> not spot anything else on a cursory look. "It should be fine!"
>
> Regards,
> Yann E. MORIN.
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [Buildroot] [PATCH 3/6] package/go: make host package a virtual package
  2024-04-14  6:28     ` Christian Stewart via buildroot
@ 2024-04-28 17:29       ` Arnout Vandecappelle via buildroot
  2024-04-28 19:18         ` Christian Stewart via buildroot
  0 siblings, 1 reply; 13+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2024-04-28 17:29 UTC (permalink / raw)
  To: Christian Stewart, Yann E. MORIN
  Cc: Geoff Levand, Anisse Astier, Angelo Compagnucci, Marcin Niestroj,
	Thomas Petazzoni, buildroot, Romain Naour, Fabio Estevam,
	Thomas Perale, Thomas Perale



On 14/04/2024 08:28, Christian Stewart via buildroot wrote:
> Hi all,
> 
> On Fri, Apr 12, 2024 at 1:01 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>> Thanks for this series, it really looks good! :-)
> 
> I agree that it looks good & having the go-bootstrap below the go/
> package is nice.
> 
> Haven't fully reviewed it yet but will try it.
> 
>> I anyway have a few comments and questions, please see below.
>>
> 
>> Not related to this patch, but still relevant to the discussion: it
>> would be even better if we could also have the possibility that the
>> stage-3 has the option of being a prebuilt go.
> 
> Using stage-3 as pre-built and host-go itself as compiled from source
> using the pre-built toolchain as the bootstrap one is ideal.
> 
> This is so that we can control which version of Go is used for the
> build for reproducibility.
> 
> Otherwise if the major version of Go installed on the host machine is
> different, it might produce different results, particularly when using
> cgo.

  The "system" go, i.e. the one installed on the build machine, is never used. 
This patch series gives the choice to either *download* the go binary, or build 
it from source.

  I don't think we want to have the option to use the system go in Buildroot. We 
do have that for things like cmake and gzip, but it's a bit painful to maintain. 
We also have it for python3, but only for a very small subset of packages (none 
of them Python packages). And since a prebuilt binary of go is anyway available 
for almost all platforms supported by Go, there is absolutely no reason not to 
just download it to be sure.

  Regards,
  Arnout

> 
> Best regards,
> Christian Stewart
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [Buildroot] [PATCH 3/6] package/go: make host package a virtual package
  2024-04-28 17:29       ` Arnout Vandecappelle via buildroot
@ 2024-04-28 19:18         ` Christian Stewart via buildroot
  0 siblings, 0 replies; 13+ messages in thread
From: Christian Stewart via buildroot @ 2024-04-28 19:18 UTC (permalink / raw)
  To: Arnout Vandecappelle
  Cc: Geoff Levand, Anisse Astier, Thomas Perale, Angelo Compagnucci,
	Marcin Niestroj, Thomas Petazzoni, Buildroot Mailing List,
	Romain Naour, Fabio Estevam, Yann E. MORIN, Thomas Perale


[-- Attachment #1.1: Type: text/plain, Size: 2337 bytes --]

Yann,

On Sun, Apr 28, 2024, 10:29 AM Arnout Vandecappelle <arnout@mind.be> wrote:

>
>
> On 14/04/2024 08:28, Christian Stewart via buildroot wrote:
> > Hi all,
> >
> > On Fri, Apr 12, 2024 at 1:01 PM Yann E. MORIN <yann.morin.1998@free.fr>
> wrote:
> >> Thanks for this series, it really looks good! :-)
> >
> > I agree that it looks good & having the go-bootstrap below the go/
> > package is nice.
> >
> > Haven't fully reviewed it yet but will try it.
> >
> >> I anyway have a few comments and questions, please see below.
> >>
> >
> >> Not related to this patch, but still relevant to the discussion: it
> >> would be even better if we could also have the possibility that the
> >> stage-3 has the option of being a prebuilt go.
> >
> > Using stage-3 as pre-built and host-go itself as compiled from source
> > using the pre-built toolchain as the bootstrap one is ideal.
> >
> > This is so that we can control which version of Go is used for the
> > build for reproducibility.
> >
> > Otherwise if the major version of Go installed on the host machine is
> > different, it might produce different results, particularly when using
> > cgo.
>
>   The "system" go, i.e. the one installed on the build machine, is never
> used.
> This patch series gives the choice to either *download* the go binary, or
> build
> it from source.
>
>   I don't think we want to have the option to use the system go in
> Buildroot. We
> do have that for things like cmake and gzip, but it's a bit painful to
> maintain.
> We also have it for python3, but only for a very small subset of packages
> (none
> of them Python packages). And since a prebuilt binary of go is anyway
> available
> for almost all platforms supported by Go, there is absolutely no reason
> not to
> just download it to be sure.
>

Using the host go compiler on the system to build host-go is absolutely
viable (I already use it all the time in my fork).

It's better than just downloading a pre-built go for multiple reasons:
namely, when doing a full from-source build, and I almost always prefer
building the compiler myself instead of trusting binaries downloaded over
the internet.

It's not hard to add, I already have a patch on the mailing list for it and
it's 5 lines of changes.

Thanks,
Christian

[-- Attachment #1.2: Type: text/html, Size: 3089 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 13+ messages in thread

* [Buildroot] [PATCH 0/6] support for a pre-built Go compiler
@ 2023-10-04 14:22 Thomas Perale via buildroot
  0 siblings, 0 replies; 13+ messages in thread
From: Thomas Perale via buildroot @ 2023-10-04 14:22 UTC (permalink / raw)
  To: buildroot
  Cc: Thomas Petazzoni, Geoff Levand, Fabio Estevam, Anisse Astier,
	Yann E . MORIN, Angelo Compagnucci, Marcin Niestroj,
	Christian Stewart, Romain Naour, Thomas Perale

From: Thomas Perale <thomas.perale@mind.be>

Improvement suggested on the buildroot TODO list (https://www.elinux.org/Buildroot#Todo_list).
This patch adds support for a pre-built version of the Go compiler that
doesn't requires users to go through the different compilation stages of
the Go compiler and speeds up the overall build time.
This is a new version of a patch I proposed for the same feature last
year (see https://lore.kernel.org/buildroot/CA+h8R2pNRrBsChKS3PHQtT0_rsVF3Wi3LwMTD9jKFoKf1ot2-A@mail.gmail.com/#b)
and with a fix of the typo I made on the [PATCH 6/6].

It introduces a new directory structure for host compilers with different
providers:

    package/
    ├─ go/                     <-- Virtual package
    │  ├─ go-src/              <-- Build from source option
    │  ├─ go-bin/              <-- Download pre-built option
    │  ├─ go-bootstrap-stage1/ <-- Dependency for go-bootstrap-stage2
    │  ├─ go-bootstrap-stage2/ <-- Dependency for go-src
    │  ├─ go.mk                <-- Common variables
    │  ├─ Config.in            <-- Common variables/virtual package definition

The subdirectory structure is used to put the `GO_VERSION` variable in
common between the different provider.

This subdirectory proposal should be extended to the Rust compiler case and
NodeJS proposal Thomas Petazzoni did (https://patchwork.ozlabs.org/project/buildroot/list/?series=319967)
to unify the naming of for the different compilers options.

Thomas Perale (6):
  DEVELOPERS: add Thomas Perale for 'go'
  package/go: new subdirectory for go variants
  package/go: make host package a virtual package
  package/pkg-golang: select go compiler
  package/go/go-bin: new go provider
  support/testing: add tests for Go providers

 .checkpackageignore                           |  2 +-
 DEVELOPERS                                    |  5 +-
 package/Config.in.host                        |  2 -
 package/balena-engine/Config.in               |  1 +
 package/cni-plugins/Config.in                 |  1 +
 package/containerd/Config.in                  |  1 +
 package/crucible/Config.in                    |  1 +
 package/delve/Config.in                       |  1 +
 package/delve/Config.in.host                  |  1 +
 package/docker-cli/Config.in                  |  1 +
 package/docker-compose/Config.in              |  1 +
 package/docker-engine/Config.in               |  1 +
 package/embiggen-disk/Config.in               |  1 +
 package/flannel/Config.in                     |  1 +
 package/gitlab-runner/Config.in               |  1 +
 package/go/Config.in.host                     | 44 +++++++++++++++
 package/go/go-bin/Config.in.host              | 14 +++++
 package/go/go-bin/go-bin.hash                 |  8 +++
 package/go/go-bin/go-bin.mk                   | 19 +++++++
 .../go-bootstrap-stage1/Config.in.host        |  0
 .../go-bootstrap-stage1.hash                  |  0
 .../go-bootstrap-stage1.mk                    |  0
 .../go-bootstrap-stage2/Config.in.host        |  0
 .../go-bootstrap-stage2.hash                  |  0
 .../go-bootstrap-stage2.mk                    |  0
 ...explicit-option-for-crosscompilation.patch |  0
 ...ldvcs-false-when-building-go-bootstr.patch |  0
 package/go/{go.hash => go-src/go-src.hash}    |  0
 package/go/go-src/go-src.mk                   | 54 +++++++++++++++++++
 package/go/go.mk                              | 42 ++-------------
 package/gocryptfs/Config.in                   |  1 +
 package/mender-artifact/Config.in.host        |  1 +
 package/mender-connect/Config.in              |  1 +
 package/mender/Config.in                      |  1 +
 package/moby-buildkit/Config.in               |  1 +
 package/moby-buildkit/Config.in.host          |  1 +
 package/nerdctl/Config.in                     |  1 +
 package/runc/Config.in                        |  1 +
 package/runc/Config.in.host                   |  1 +
 package/tinifier/Config.in                    |  1 +
 package/wtfutil/Config.in                     |  1 +
 support/testing/tests/package/test_go.py      | 41 ++++++++++++++
 42 files changed, 211 insertions(+), 43 deletions(-)
 create mode 100644 package/go/go-bin/Config.in.host
 create mode 100644 package/go/go-bin/go-bin.hash
 create mode 100644 package/go/go-bin/go-bin.mk
 rename package/{ => go}/go-bootstrap-stage1/Config.in.host (100%)
 rename package/{ => go}/go-bootstrap-stage1/go-bootstrap-stage1.hash (100%)
 rename package/{ => go}/go-bootstrap-stage1/go-bootstrap-stage1.mk (100%)
 rename package/{ => go}/go-bootstrap-stage2/Config.in.host (100%)
 rename package/{ => go}/go-bootstrap-stage2/go-bootstrap-stage2.hash (100%)
 rename package/{ => go}/go-bootstrap-stage2/go-bootstrap-stage2.mk (100%)
 rename package/go/{ => go-src}/0001-build.go-explicit-option-for-crosscompilation.patch (100%)
 rename package/go/{ => go-src}/0002-cmd-dist-set-buildvcs-false-when-building-go-bootstr.patch (100%)
 rename package/go/{go.hash => go-src/go-src.hash} (100%)
 create mode 100644 package/go/go-src/go-src.mk
 create mode 100644 support/testing/tests/package/test_go.py

-- 
2.42.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2024-04-28 19:18 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-12 13:26 [Buildroot] [PATCH 0/6] support for a pre-built Go compiler Thomas Perale via buildroot
2024-04-12 13:26 ` [Buildroot] [PATCH 1/6] DEVELOPERS: add Thomas Perale for 'go' Thomas Perale via buildroot
2024-04-12 13:26 ` [Buildroot] [PATCH 2/6] package/go: new subdirectory for go variants Thomas Perale via buildroot
2024-04-12 13:26 ` [Buildroot] [PATCH 3/6] package/go: make host package a virtual package Thomas Perale via buildroot
2024-04-12 20:00   ` Yann E. MORIN
2024-04-14  6:28     ` Christian Stewart via buildroot
2024-04-28 17:29       ` Arnout Vandecappelle via buildroot
2024-04-28 19:18         ` Christian Stewart via buildroot
2024-04-15 12:59     ` Thomas Perale via buildroot
2024-04-12 13:26 ` [Buildroot] [PATCH 4/6] package/pkg-golang: select go compiler Thomas Perale via buildroot
2024-04-12 13:26 ` [Buildroot] [PATCH 5/6] package/go/go-bin: new go provider Thomas Perale via buildroot
2024-04-12 13:26 ` [Buildroot] [PATCH 6/6] support/testing: add tests for Go providers Thomas Perale via buildroot
  -- strict thread matches above, loose matches on Subject: below --
2023-10-04 14:22 [Buildroot] [PATCH 0/6] support for a pre-built Go compiler Thomas Perale via buildroot

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.