All of lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [[PATCH v2 0/2] mender-artifact support
@ 2018-08-27 11:37 Mirza Krak
  2018-08-27 11:37 ` [Buildroot] [[PATCH v2 1/2] package/pkg-golang: add support for building host packages Mirza Krak
                   ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Mirza Krak @ 2018-08-27 11:37 UTC (permalink / raw)
  To: buildroot

First patch adds "host" support in pkg-golang.mk.

Second patch adds a new package, mender-artifact. This is a host tool
to generate update images in Mender Artifact format compatible with
Mender client and the Mender server. More information about the Mender
artifact format can be found here:

    https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format.md

Patch 2 depends on:

    package/pkg-golang: add support for building host packages

All changes are based on review from Thomas Petazzoni, thank you for the review!

Changes in v2:
    - "fixup" patch 3 (DEVELOPERS update) in to patch 2.

See individual patches for more info on updates from v1 -> v2

Mirza Krak (2):
  package/pkg-golang: add support for building host packages
  package/mender-artifact: initial support

 DEVELOPERS                                   |  1 +
 docs/manual/adding-packages-golang.txt       |  5 ++--
 package/Config.in.host                       |  1 +
 package/mender-artifact/Config.in.host       | 20 +++++++++++++++
 package/mender-artifact/mender-artifact.hash | 28 +++++++++++++++++++++
 package/mender-artifact/mender-artifact.mk   | 21 ++++++++++++++++
 package/pkg-golang.mk                        | 37 +++++++++++++++++++++++++---
 7 files changed, 108 insertions(+), 5 deletions(-)
 create mode 100644 package/mender-artifact/Config.in.host
 create mode 100644 package/mender-artifact/mender-artifact.hash
 create mode 100644 package/mender-artifact/mender-artifact.mk

--
2.11.0

^ permalink raw reply	[flat|nested] 9+ messages in thread
* [Buildroot] [[PATCH v2 1/2] package/pkg-golang: add support for building host packages
  2018-08-27 11:37 [Buildroot] [[PATCH v2 0/2] mender-artifact support Mirza Krak
@ 2018-08-27 11:37 ` Mirza Krak
  2018-09-05 15:26   ` Angelo Compagnucci
  2018-08-27 11:37 ` [Buildroot] [[PATCH v2 2/2] package/mender-artifact: initial support Mirza Krak
  2018-09-02 13:25 ` [Buildroot] [[PATCH v2 0/2] mender-artifact support Thomas Petazzoni
  2 siblings, 1 reply; 9+ messages in thread
From: Mirza Krak @ 2018-08-27 11:37 UTC (permalink / raw)
  To: buildroot

With this you can add:

    $(eval $(host-golang-package))

to a package .mk file to build for host.

Signed-off-by: Mirza Krak <mirza.krak@northern.tech>
---

Changes in v2:
    - Added entry in Buildroot manual
    - Install to $(HOST_DIR)/bin instead of $(HOST_DIR)/usr/bin
    - Created "GO_COMMON_ENV" to be shared between GO_TARGET_ENV and GO_HOST_ENV
    - Dropped accidental change of comment
    - Removed a empty line

 docs/manual/adding-packages-golang.txt |  5 +++--
 package/pkg-golang.mk                  | 37 +++++++++++++++++++++++++++++++---
 2 files changed, 37 insertions(+), 5 deletions(-)

diff --git a/docs/manual/adding-packages-golang.txt b/docs/manual/adding-packages-golang.txt
index efcf696867..94520480ce 100644
--- a/docs/manual/adding-packages-golang.txt
+++ b/docs/manual/adding-packages-golang.txt
@@ -50,8 +50,9 @@ and +BR2_PACKAGE_HOST_GO_CGO_LINKING_SUPPORTS+ because Buildroot will
 automatically add a dependency on +host-go+ to such packages.

 The main macro of the Go package infrastructure is
-+golang-package+. It is similar to the +generic-package+ macro. Only
-target packages are supported with +golang-package+.
++golang-package+. It is similar to the +generic-package+ macro. The
+ability to build host packages is also available, with the
++host-golang-package+ macro.

 Just like the generic infrastructure, the Go infrastructure works
 by defining a number of variables before calling the +golang-package+.
diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index 6eacd14180..878228ce74 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -25,12 +25,18 @@ GO_BIN = $(HOST_DIR)/bin/go

 # We pass an empty GOBIN, otherwise "go install: cannot install
 # cross-compiled binaries when GOBIN is set"
-GO_TARGET_ENV = \
-	$(HOST_GO_TARGET_ENV) \
+GO_COMMON_ENV = \
 	PATH=$(BR_PATH) \
 	GOBIN= \
 	CGO_ENABLED=$(HOST_GO_CGO_ENABLED)

+GO_TARGET_ENV = \
+	$(HOST_GO_TARGET_ENV) \
+	$(GO_COMMON_ENV)
+
+GO_HOST_ENV = \
+	$(GO_COMMON_ENV)
+
 ################################################################################
 # inner-golang-package -- defines how the configuration, compilation and
 # installation of a Go package should be done, implements a few hooks to tune
@@ -44,7 +50,6 @@ GO_TARGET_ENV = \
 #             packages
 #  argument 4 is the type (target or host)
 #
-# NOTE Only type target is supported at the moment
 ################################################################################

 define inner-golang-package
@@ -96,6 +101,8 @@ endif
 # Build step. Only define it if not already defined by the package .mk
 # file.
 ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+# Build package for target
 define $(2)_BUILD_CMDS
 	$$(foreach d,$$($(2)_BUILD_TARGETS),\
 		cd $$($(2)_SRC_PATH); \
@@ -107,6 +114,20 @@ define $(2)_BUILD_CMDS
 			./$$(d)
 	)
 endef
+else
+# Build package for host
+define $(2)_BUILD_CMDS
+	$$(foreach d,$$($(2)_BUILD_TARGETS),\
+		cd $$($(2)_SRC_PATH); \
+		$$(GO_HOST_ENV) \
+			GOPATH="$$(@D)/$$($(2)_WORKSPACE)" \
+			$$($(2)_GO_ENV) \
+			$$(GO_BIN) build -v $$($(2)_BUILD_OPTS) \
+			-o $$(@D)/bin/$$(or $$($(2)_BIN_NAME),$$(notdir $$(d))) \
+			./$$(d)
+	)
+endef
+endif
 endif

 # Target installation step. Only define it if not already defined by the
@@ -119,6 +140,15 @@ define $(2)_INSTALL_TARGET_CMDS
 endef
 endif

+# Host installation step
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+	$$(foreach d,$$($(2)_INSTALL_BINS),\
+		$(INSTALL) -D -m 0755 $$(@D)/bin/$$(d) $(HOST_DIR)/bin/$$(d)
+	)
+endef
+endif
+
 # Call the generic package infrastructure to generate the necessary make
 # targets
 $(call inner-generic-package,$(1),$(2),$(3),$(4))
@@ -130,3 +160,4 @@ endef # inner-golang-package
 ################################################################################

 golang-package = $(call inner-golang-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
+host-golang-package = $(call inner-golang-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host)
--
2.11.0

^ permalink raw reply related	[flat|nested] 9+ messages in thread
* [Buildroot] [[PATCH v2 2/2] package/mender-artifact: initial support
  2018-08-27 11:37 [Buildroot] [[PATCH v2 0/2] mender-artifact support Mirza Krak
  2018-08-27 11:37 ` [Buildroot] [[PATCH v2 1/2] package/pkg-golang: add support for building host packages Mirza Krak
@ 2018-08-27 11:37 ` Mirza Krak
  2018-09-02 13:47   ` Thomas Petazzoni
  2018-09-02 13:25 ` [Buildroot] [[PATCH v2 0/2] mender-artifact support Thomas Petazzoni
  2 siblings, 1 reply; 9+ messages in thread
From: Mirza Krak @ 2018-08-27 11:37 UTC (permalink / raw)
  To: buildroot

mender-artifact is a host tool to generate update images
in the Mender artifact file format.

Example usage:

    $ mender-artifact write rootfs-image \
         --update rootfs.ext4 \
         --output-path rootfs.mender \
         --artifact-name "release-v1.0.0" \
         --device-type "beaglebone"

Above will generate a Mender artifact called "rootfs.mender"
containing the "rootfs.ext4" image along with meta-data.

One can read-out the meta-data with the following command:

    $ mender-artifact read rootfs.mender
      Mender artifact:
      Name: release-v1.0.0
      Format: mender
      Version: 2
      Signature: no signature
      Compatible devices: '[beaglebone]'

    Updates:
      0000:
        Type:   rootfs-image
        Files:
          name:     rootfs.ext4
          size:     52428800
          modified: 2018-08-27 09:10:55 +0200 CEST
          checksum: e70b113fb0964a810a3043586eb4fc1c48e684ba78b02ba65fead4aa3e540d87

Signed-off-by: Mirza Krak <mirza.krak@northern.tech>
---

Changes in v2:
  - Merged DEVELOPERS change and fixed indentation
  - Fixed indentation in Config.in.host and order (check-package warnings)
  - Added a more descriptive help section in Config.in.host with URL to upstream
  - Clarified comments in mender-artifact.hash around license hash and
    how they are generated
  - Updated commit message to a bit more descriptive and with examples.

 DEVELOPERS                                   |  1 +
 package/Config.in.host                       |  1 +
 package/mender-artifact/Config.in.host       | 20 ++++++++++++++++++++
 package/mender-artifact/mender-artifact.hash | 28 ++++++++++++++++++++++++++++
 package/mender-artifact/mender-artifact.mk   | 21 +++++++++++++++++++++
 5 files changed, 71 insertions(+)
 create mode 100644 package/mender-artifact/Config.in.host
 create mode 100644 package/mender-artifact/mender-artifact.hash
 create mode 100644 package/mender-artifact/mender-artifact.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index a6d76dd9c4..3f8390285e 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -1439,6 +1439,7 @@ F:	package/shadowsocks-libev/

 N:	Mirza Krak <mirza.krak@northern.tech>
 F:	package/mender/
+F:	package/mender-artifact/

 N:	Morgan Delestre <m.delestre@sinters.fr>
 F:	package/monkey/
diff --git a/package/Config.in.host b/package/Config.in.host
index 7838ffc219..2b85d2b189 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -30,6 +30,7 @@ menu "Host utilities"
 	source "package/jsmin/Config.in.host"
 	source "package/lpc3250loader/Config.in.host"
 	source "package/lttng-babeltrace/Config.in.host"
+	source "package/mender-artifact/Config.in.host"
 	source "package/mfgtools/Config.in.host"
 	source "package/mkpasswd/Config.in.host"
 	source "package/mtd/Config.in.host"
diff --git a/package/mender-artifact/Config.in.host b/package/mender-artifact/Config.in.host
new file mode 100644
index 0000000000..d2deabc783
--- /dev/null
+++ b/package/mender-artifact/Config.in.host
@@ -0,0 +1,20 @@
+config BR2_PACKAGE_HOST_MENDER_ARTIFACT
+	bool "host mender-artifact"
+	depends on BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_GO_CGO_LINKING_SUPPORTS
+	depends on BR2_TOOLCHAIN_HAS_THREADS
+	help
+	  The mender-artifact tool is a CLI implementation of the
+	  Mender artifacts library.
+
+	  A Mender artifact can be recognized by its .mender suffix.
+	  Mender artifacts can contain binaries, metadata, checksums,
+	  signatures and scripts that are used during a deployment.
+	  The artifact format acts as a wrapper, and uses the tar
+	  format to bundle several files into one.
+
+	  In its simplest form, an artifact contains just a rootfs
+	  image, along with its checksum, id and device type
+	  compatibility.
+
+	  https://github.com/mendersoftware/mender-artifact
diff --git a/package/mender-artifact/mender-artifact.hash b/package/mender-artifact/mender-artifact.hash
new file mode 100644
index 0000000000..96a2cbc6a1
--- /dev/null
+++ b/package/mender-artifact/mender-artifact.hash
@@ -0,0 +1,28 @@
+# Locally computed:
+sha256 2a0322d8707c8ea7cada12c8f96144382264c898cadc35e8058bb9ea6bf8b041 mender-artifact-2.2.0.tar.gz
+
+# License hash extracted from LIC_FILES_CHKSUM.sha256 using the
+# following command:
+#
+#     sed '/^[A-Za-z0-9_]/s/^/sha256  /' LIC_FILES_CHKSUM.sha256
+
+# Apache-2.0 license.
+sha256 b40930bbcf80744c86c46a12bc9da056641d722716c378f5659b9e555ef833e1 LICENSE
+sha256 ceb1b36ff073bd13d9806d4615b931707768ca9023805620acc32dd1cfc2f680 vendor/github.com/mendersoftware/mendertesting/LICENSE
+
+# BSD 2 Clause licenses.
+sha256 8d427fd87bc9579ea368fde3d49f9ca22eac857f91a9dec7e3004bdfab7dee86 vendor/github.com/pkg/errors/LICENSE
+
+# BSD 3 Clause licenses.
+sha256 2eb550be6801c1ea434feba53bf6d12e7c71c90253e0a9de4a4f46cf88b56477 vendor/github.com/pmezard/go-difflib/LICENSE
+
+# ISC licenses.
+sha256 3525392c6db3b804af76980b2c560ee9ec1abdadd907d76a26091df7c78f3a25 vendor/github.com/davecgh/go-spew/LICENSE
+
+# MIT licenses.
+sha256 402f39eed8a1851385d0703999aa9f23d067c2ea3e15c63c074e389cbf8f8f8f vendor/github.com/stretchr/testify/LICENSE
+sha256 402f39eed8a1851385d0703999aa9f23d067c2ea3e15c63c074e389cbf8f8f8f vendor/github.com/stretchr/testify/LICENCE.txt
+sha256 da277af11b85227490377fbcac6afccc68be560c4fff36ac05ca62de55345fd7 vendor/github.com/urfave/cli/LICENSE
+
+# sha256 of all licenses files combined (locally computed)
+sha256 a9d3dea05c34f2bf7eb3fa677dafe09e9a80edd024cb31da750616c10f2bbcbb LIC_FILES_CHKSUM.sha256
diff --git a/package/mender-artifact/mender-artifact.mk b/package/mender-artifact/mender-artifact.mk
new file mode 100644
index 0000000000..590895de8c
--- /dev/null
+++ b/package/mender-artifact/mender-artifact.mk
@@ -0,0 +1,21 @@
+################################################################################
+#
+# mender-artifact
+#
+################################################################################
+
+MENDER_ARTIFACT_VERSION = 2.2.0
+MENDER_ARTIFACT_SITE = $(call github,mendersoftware,mender-artifact,$(MENDER_ARTIFACT_VERSION))
+MENDER_ARTIFACT_LICENSE = Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, MIT
+MENDER_ARTIFACT_LICENSE_FILES = \
+	LICENSE \
+	LIC_FILES_CHKSUM.sha256 \
+	vendor/github.com/mendersoftware/mendertesting/LICENSE \
+	vendor/github.com/pkg/errors/LICENSE \
+	vendor/github.com/pmezard/go-difflib/LICENSE \
+	vendor/github.com/davecgh/go-spew/LICENSE \
+	vendor/github.com/stretchr/testify/LICENSE \
+	vendor/github.com/stretchr/testify/LICENCE.txt \
+	vendor/github.com/urfave/cli/LICENSE
+
+$(eval $(host-golang-package))
--
2.11.0

^ permalink raw reply related	[flat|nested] 9+ messages in thread
* [Buildroot] [[PATCH v2 0/2] mender-artifact support
  2018-08-27 11:37 [Buildroot] [[PATCH v2 0/2] mender-artifact support Mirza Krak
  2018-08-27 11:37 ` [Buildroot] [[PATCH v2 1/2] package/pkg-golang: add support for building host packages Mirza Krak
  2018-08-27 11:37 ` [Buildroot] [[PATCH v2 2/2] package/mender-artifact: initial support Mirza Krak
@ 2018-09-02 13:25 ` Thomas Petazzoni
  2018-09-02 20:00   ` Mirza Krak
  2 siblings, 1 reply; 9+ messages in thread
From: Thomas Petazzoni @ 2018-09-02 13:25 UTC (permalink / raw)
  To: buildroot

Hello Mirza,

On Mon, 27 Aug 2018 13:37:26 +0200, Mirza Krak wrote:

> Mirza Krak (2):
>   package/pkg-golang: add support for building host packages
>   package/mender-artifact: initial support

Thanks for this patch series. It looks mostly good from a review
perspective, except one aspect on PATCH 2/2, to which I'll reply
separately.

However, the host-mender-artifact package doesn't build here. With the
following deconfig:

BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2018.05.tar.bz2"
BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_1=y
BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
# BR2_TARGET_ROOTFS_TAR is not set
BR2_PACKAGE_HOST_MENDER_ARTIFACT=y

I get the following error:

>>> host-mender-artifact 2.2.0 Building
cd /home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/_gopath/src///; PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/usr/local/bin:/usr/bin:/bin:/home/thomas/.rvm/bin:/usr/local/sbin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/home/thomas/.rvm/bin" GOBIN= CGO_ENABLED=1 GOPATH="/home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/_gopath"  /home/thomas/projets/buildroot/output/host/bin/go build -v -ldflags "" -tags "" -o /home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/bin/host-mender-artifact ./.
can't load package: package .: no Go files in /home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/_gopath/src
make[1]: *** [package/pkg-generic.mk:232: /home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/.stamp_built] Error 1
make: *** [Makefile:79: _all] Error 2

Could you have a look into this ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

^ permalink raw reply	[flat|nested] 9+ messages in thread
* [Buildroot] [[PATCH v2 2/2] package/mender-artifact: initial support
  2018-08-27 11:37 ` [Buildroot] [[PATCH v2 2/2] package/mender-artifact: initial support Mirza Krak
@ 2018-09-02 13:47   ` Thomas Petazzoni
  2018-09-12 19:26     ` Mirza Krak
  0 siblings, 1 reply; 9+ messages in thread
From: Thomas Petazzoni @ 2018-09-02 13:47 UTC (permalink / raw)
  To: buildroot

Hello Mirza,

Overall it looks good, except for one aspect, which will require a bit
of discussion.

On Mon, 27 Aug 2018 13:37:28 +0200, Mirza Krak wrote:

> diff --git a/package/mender-artifact/Config.in.host b/package/mender-artifact/Config.in.host
> new file mode 100644
> index 0000000000..d2deabc783
> --- /dev/null
> +++ b/package/mender-artifact/Config.in.host
> @@ -0,0 +1,20 @@
> +config BR2_PACKAGE_HOST_MENDER_ARTIFACT
> +	bool "host mender-artifact"
> +	depends on BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS
> +	depends on BR2_PACKAGE_HOST_GO_CGO_LINKING_SUPPORTS
> +	depends on BR2_TOOLCHAIN_HAS_THREADS

I am not happy about those dependencies, because they are dependencies
meant to be used for *target* packages, not for *host* packages.

BR2_TOOLCHAIN_HAS_THREADS is easy: it's not needed for host packages,
as we assume the build machine has thread support.

For the other dependencies, it's a bit more complicated, but I believe
somewhat similar to the situation we have with the Rust support.

Right now, BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS indicates if the
currently selected target architecture is supported by Go. This is fine
for target packages.

But apparently, host-go provides really two things:

 (1) A Go compiler capable of building code for the host machine.

 (2) A Go compiler capable of building code for the target machine.

For Go target packages, We are currently using (2), and for packages
that rely on (2), using BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS make sense.

However, with your support for host packages, what we rely on is (1).
And for that BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS doesn't make a lot of
sense.

For example, BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS says that Go is not
available on the ARC architecture. But for building a host package, we
really don't care about this: what matters is that Go is capable of
building code for the x86 or x86-64 build machine.

Therefore, we should probably use the same strategy as Rust, which is
proposed in patch http://patchwork.ozlabs.org/patch/961158/:

 - The BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS is changed to only indicate if
   Go supports the host/build system architecture. In our case, it
   should probably just be:

	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS

   All host Go packages should depend on this option.

 - A new BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS that indicates if Go
   supports the target system architecture. It should be:

	depends on BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS
        depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
                || BR2_i386 || BR2_x86_64 || BR2_powerpc64le \
                || BR2_mips64 || BR2_mips64el
        depends on !BR2_ARM_CPU_ARMV4
        # MIPS R6 support in Go has not yet been developed.
        depends on !BR2_MIPS_CPU_MIPS64R6

Of course we can only do this if host-go builds correctly regardless of
whether the target architecture is supported or not. What Rust does is:

 - If the target is architecture is supported by Rust, it installs
   support for building code for both the host and the target
   architectures.

 - If the target is architecture is not supported by Rust, it only
   installs support for building code for the host architecture.

Do you know how Go supports this ? I have no idea how Go is told to
support this or that architecture. Could you investigate this ?

Best regards,

Thomas Petazzoni
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

^ permalink raw reply	[flat|nested] 9+ messages in thread
* [Buildroot] [[PATCH v2 0/2] mender-artifact support
  2018-09-02 13:25 ` [Buildroot] [[PATCH v2 0/2] mender-artifact support Thomas Petazzoni
@ 2018-09-02 20:00   ` Mirza Krak
  2018-09-02 20:12     ` Thomas Petazzoni
  0 siblings, 1 reply; 9+ messages in thread
From: Mirza Krak @ 2018-09-02 20:00 UTC (permalink / raw)
  To: buildroot

On Sun, Sep 2, 2018 at 3:25 PM, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Mirza,
>
> On Mon, 27 Aug 2018 13:37:26 +0200, Mirza Krak wrote:
>
> > Mirza Krak (2):
> >   package/pkg-golang: add support for building host packages
> >   package/mender-artifact: initial support
>
> Thanks for this patch series. It looks mostly good from a review
> perspective, except one aspect on PATCH 2/2, to which I'll reply
> separately.
>
> However, the host-mender-artifact package doesn't build here. With the
> following deconfig:
>
> BR2_arm=y
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
> BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
> BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2018.05.tar.bz2"
> BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
> BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_1=y
> BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
> # BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
> BR2_TOOLCHAIN_EXTERNAL_CXX=y
> BR2_INIT_NONE=y
> BR2_SYSTEM_BIN_SH_NONE=y
> # BR2_PACKAGE_BUSYBOX is not set
> # BR2_TARGET_ROOTFS_TAR is not set
> BR2_PACKAGE_HOST_MENDER_ARTIFACT=y
>
> I get the following error:
>
> >>> host-mender-artifact 2.2.0 Building
> cd /home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/_gopath/src///; PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/usr/local/bin:/usr/bin:/bin:/home/thomas/.rvm/bin:/usr/local/sbin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/home/thomas/.rvm/bin" GOBIN= CGO_ENABLED=1 GOPATH="/home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/_gopath"  /home/thomas/projets/buildroot/output/host/bin/go build -v -ldflags "" -tags "" -o /home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/bin/host-mender-artifact ./.
> can't load package: package .: no Go files in /home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/_gopath/src
> make[1]: *** [package/pkg-generic.mk:232: /home/thomas/projets/buildroot/output/build/host-mender-artifact-2.2.0/.stamp_built] Error 1
> make: *** [Makefile:79: _all] Error 2
>
> Could you have a look into this ?

This looks like this patch [1] is missing. Or at least it was a
similar error that I got before that was fixed.

[1]. https://git.buildroot.net/buildroot/commit/?id=4eccbe3ef975fa7dcc8984fa9a19c778752d70af

-- 
Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

^ permalink raw reply	[flat|nested] 9+ messages in thread
* [Buildroot] [[PATCH v2 0/2] mender-artifact support
  2018-09-02 20:00   ` Mirza Krak
@ 2018-09-02 20:12     ` Thomas Petazzoni
  0 siblings, 0 replies; 9+ messages in thread
From: Thomas Petazzoni @ 2018-09-02 20:12 UTC (permalink / raw)
  To: buildroot

Hello,

On Sun, 2 Sep 2018 22:00:17 +0200, Mirza Krak wrote:

> > Could you have a look into this ?  
> 
> This looks like this patch [1] is missing. Or at least it was a
> similar error that I got before that was fixed.
> 
> [1]. https://git.buildroot.net/buildroot/commit/?id=4eccbe3ef975fa7dcc8984fa9a19c778752d70af

Aaah, yes, my bad, sorry for not thinking about this. I tested your
host-golang-package + mender-artifact patches on top of the next
branch, while commit 4eccbe3ef975fa7dcc8984fa9a19c778752d70af was made
to the master branch, because it was a fix.

I will retry with this fixed merged into the next branch as well.

Sorry about that :-/

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

^ permalink raw reply	[flat|nested] 9+ messages in thread
* [Buildroot] [[PATCH v2 1/2] package/pkg-golang: add support for building host packages
  2018-08-27 11:37 ` [Buildroot] [[PATCH v2 1/2] package/pkg-golang: add support for building host packages Mirza Krak
@ 2018-09-05 15:26   ` Angelo Compagnucci
  0 siblings, 0 replies; 9+ messages in thread
From: Angelo Compagnucci @ 2018-09-05 15:26 UTC (permalink / raw)
  To: buildroot

On Mon, Aug 27, 2018 at 1:37 PM, Mirza Krak <mirza.krak@northern.tech> wrote:
> With this you can add:
>
>     $(eval $(host-golang-package))
>
> to a package .mk file to build for host.
>
> Signed-off-by: Mirza Krak <mirza.krak@northern.tech>
> ---

Acked-by: Angelo Compagnucci <angelo@amarulasolutions.com>
Tested-by: Angelo Compagnucci <angelo@amarulasolutions.com>

>
> Changes in v2:
>     - Added entry in Buildroot manual
>     - Install to $(HOST_DIR)/bin instead of $(HOST_DIR)/usr/bin
>     - Created "GO_COMMON_ENV" to be shared between GO_TARGET_ENV and GO_HOST_ENV
>     - Dropped accidental change of comment
>     - Removed a empty line
>
>  docs/manual/adding-packages-golang.txt |  5 +++--
>  package/pkg-golang.mk                  | 37 +++++++++++++++++++++++++++++++---
>  2 files changed, 37 insertions(+), 5 deletions(-)
>
> diff --git a/docs/manual/adding-packages-golang.txt b/docs/manual/adding-packages-golang.txt
> index efcf696867..94520480ce 100644
> --- a/docs/manual/adding-packages-golang.txt
> +++ b/docs/manual/adding-packages-golang.txt
> @@ -50,8 +50,9 @@ and +BR2_PACKAGE_HOST_GO_CGO_LINKING_SUPPORTS+ because Buildroot will
>  automatically add a dependency on +host-go+ to such packages.
>
>  The main macro of the Go package infrastructure is
> -+golang-package+. It is similar to the +generic-package+ macro. Only
> -target packages are supported with +golang-package+.
> ++golang-package+. It is similar to the +generic-package+ macro. The
> +ability to build host packages is also available, with the
> ++host-golang-package+ macro.
>
>  Just like the generic infrastructure, the Go infrastructure works
>  by defining a number of variables before calling the +golang-package+.
> diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
> index 6eacd14180..878228ce74 100644
> --- a/package/pkg-golang.mk
> +++ b/package/pkg-golang.mk
> @@ -25,12 +25,18 @@ GO_BIN = $(HOST_DIR)/bin/go
>
>  # We pass an empty GOBIN, otherwise "go install: cannot install
>  # cross-compiled binaries when GOBIN is set"
> -GO_TARGET_ENV = \
> -       $(HOST_GO_TARGET_ENV) \
> +GO_COMMON_ENV = \
>         PATH=$(BR_PATH) \
>         GOBIN= \
>         CGO_ENABLED=$(HOST_GO_CGO_ENABLED)
>
> +GO_TARGET_ENV = \
> +       $(HOST_GO_TARGET_ENV) \
> +       $(GO_COMMON_ENV)
> +
> +GO_HOST_ENV = \
> +       $(GO_COMMON_ENV)
> +
>  ################################################################################
>  # inner-golang-package -- defines how the configuration, compilation and
>  # installation of a Go package should be done, implements a few hooks to tune
> @@ -44,7 +50,6 @@ GO_TARGET_ENV = \
>  #             packages
>  #  argument 4 is the type (target or host)
>  #
> -# NOTE Only type target is supported at the moment
>  ################################################################################
>
>  define inner-golang-package
> @@ -96,6 +101,8 @@ endif
>  # Build step. Only define it if not already defined by the package .mk
>  # file.
>  ifndef $(2)_BUILD_CMDS
> +ifeq ($(4),target)
> +# Build package for target
>  define $(2)_BUILD_CMDS
>         $$(foreach d,$$($(2)_BUILD_TARGETS),\
>                 cd $$($(2)_SRC_PATH); \
> @@ -107,6 +114,20 @@ define $(2)_BUILD_CMDS
>                         ./$$(d)
>         )
>  endef
> +else
> +# Build package for host
> +define $(2)_BUILD_CMDS
> +       $$(foreach d,$$($(2)_BUILD_TARGETS),\
> +               cd $$($(2)_SRC_PATH); \
> +               $$(GO_HOST_ENV) \
> +                       GOPATH="$$(@D)/$$($(2)_WORKSPACE)" \
> +                       $$($(2)_GO_ENV) \
> +                       $$(GO_BIN) build -v $$($(2)_BUILD_OPTS) \
> +                       -o $$(@D)/bin/$$(or $$($(2)_BIN_NAME),$$(notdir $$(d))) \
> +                       ./$$(d)
> +       )
> +endef
> +endif
>  endif
>
>  # Target installation step. Only define it if not already defined by the
> @@ -119,6 +140,15 @@ define $(2)_INSTALL_TARGET_CMDS
>  endef
>  endif
>
> +# Host installation step
> +ifndef $(2)_INSTALL_CMDS
> +define $(2)_INSTALL_CMDS
> +       $$(foreach d,$$($(2)_INSTALL_BINS),\
> +               $(INSTALL) -D -m 0755 $$(@D)/bin/$$(d) $(HOST_DIR)/bin/$$(d)
> +       )
> +endef
> +endif
> +
>  # Call the generic package infrastructure to generate the necessary make
>  # targets
>  $(call inner-generic-package,$(1),$(2),$(3),$(4))
> @@ -130,3 +160,4 @@ endef # inner-golang-package
>  ################################################################################
>
>  golang-package = $(call inner-golang-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
> +host-golang-package = $(call inner-golang-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host)
> --
> 2.11.0
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 9+ messages in thread
* [Buildroot] [[PATCH v2 2/2] package/mender-artifact: initial support
  2018-09-02 13:47   ` Thomas Petazzoni
@ 2018-09-12 19:26     ` Mirza Krak
  0 siblings, 0 replies; 9+ messages in thread
From: Mirza Krak @ 2018-09-12 19:26 UTC (permalink / raw)
  To: buildroot

On Sun, Sep 2, 2018 at 3:47 PM, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Mirza,
>
> Overall it looks good, except for one aspect, which will require a bit
> of discussion.
>
> On Mon, 27 Aug 2018 13:37:28 +0200, Mirza Krak wrote:
>
> > diff --git a/package/mender-artifact/Config.in.host b/package/mender-artifact/Config.in.host
> > new file mode 100644
> > index 0000000000..d2deabc783
> > --- /dev/null
> > +++ b/package/mender-artifact/Config.in.host
> > @@ -0,0 +1,20 @@
> > +config BR2_PACKAGE_HOST_MENDER_ARTIFACT
> > +     bool "host mender-artifact"
> > +     depends on BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS
> > +     depends on BR2_PACKAGE_HOST_GO_CGO_LINKING_SUPPORTS
> > +     depends on BR2_TOOLCHAIN_HAS_THREADS
>
> I am not happy about those dependencies, because they are dependencies
> meant to be used for *target* packages, not for *host* packages.
>
> BR2_TOOLCHAIN_HAS_THREADS is easy: it's not needed for host packages,
> as we assume the build machine has thread support.
>
> For the other dependencies, it's a bit more complicated, but I believe
> somewhat similar to the situation we have with the Rust support.
>
> Right now, BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS indicates if the
> currently selected target architecture is supported by Go. This is fine
> for target packages.
>
> But apparently, host-go provides really two things:
>
>  (1) A Go compiler capable of building code for the host machine.
>
>  (2) A Go compiler capable of building code for the target machine.
>
> For Go target packages, We are currently using (2), and for packages
> that rely on (2), using BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS make sense.
>
> However, with your support for host packages, what we rely on is (1).
> And for that BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS doesn't make a lot of
> sense.
>
> For example, BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS says that Go is not
> available on the ARC architecture. But for building a host package, we
> really don't care about this: what matters is that Go is capable of
> building code for the x86 or x86-64 build machine.
>
> Therefore, we should probably use the same strategy as Rust, which is
> proposed in patch http://patchwork.ozlabs.org/patch/961158/:
>
>  - The BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS is changed to only indicate if
>    Go supports the host/build system architecture. In our case, it
>    should probably just be:
>
>         depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
>
>    All host Go packages should depend on this option.
>
>  - A new BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS that indicates if Go
>    supports the target system architecture. It should be:
>
>         depends on BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS
>         depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
>                 || BR2_i386 || BR2_x86_64 || BR2_powerpc64le \
>                 || BR2_mips64 || BR2_mips64el
>         depends on !BR2_ARM_CPU_ARMV4
>         # MIPS R6 support in Go has not yet been developed.
>         depends on !BR2_MIPS_CPU_MIPS64R6
>
> Of course we can only do this if host-go builds correctly regardless of
> whether the target architecture is supported or not. What Rust does is:
>
>  - If the target is architecture is supported by Rust, it installs
>    support for building code for both the host and the target
>    architectures.
>
>  - If the target is architecture is not supported by Rust, it only
>    installs support for building code for the host architecture.
>
> Do you know how Go supports this ? I have no idea how Go is told to
> support this or that architecture. Could you investigate this ?

My priorities have shifted a bit but I plan on getting back on this
one (need to investigate as I do not know the answers to above).

If someone else holds the answers please feel free to jump in.

-- 
Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-09-12 19:26 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-27 11:37 [Buildroot] [[PATCH v2 0/2] mender-artifact support Mirza Krak
2018-08-27 11:37 ` [Buildroot] [[PATCH v2 1/2] package/pkg-golang: add support for building host packages Mirza Krak
2018-09-05 15:26   ` Angelo Compagnucci
2018-08-27 11:37 ` [Buildroot] [[PATCH v2 2/2] package/mender-artifact: initial support Mirza Krak
2018-09-02 13:47   ` Thomas Petazzoni
2018-09-12 19:26     ` Mirza Krak
2018-09-02 13:25 ` [Buildroot] [[PATCH v2 0/2] mender-artifact support Thomas Petazzoni
2018-09-02 20:00   ` Mirza Krak
2018-09-02 20:12     ` Thomas Petazzoni

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.