* [Buildroot] [PATCH 0 of 5] toolchain-external: support downloading custom toolchains + related misc. fixes @ 2011-08-02 18:33 Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 1 of 5] toolchain-external: allow downloading a custom toolchain Thomas De Schampheleire ` (4 more replies) 0 siblings, 5 replies; 16+ messages in thread From: Thomas De Schampheleire @ 2011-08-02 18:33 UTC (permalink / raw) To: buildroot This patch series originates from the need to be able to use a custom external (buildroot) toolchain, downloaded from an scp location. Support for scp is not included in this series (it was submitted previously, but needs to be retriggered) but the prerequisites relevant for toolchain-external are. While I was at it, a few CodeSourcery typos were fixed as well. Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> --- For the patches related to host-tar, I would like to get some feedback regarding the best way to handle this. I found an earlier thread (http://lists.busybox.net/pipermail/buildroot/2011-March/042191.html - alsa-utils and nctp troubles with 2011.02) where the tar issue was left to rest because it was attributed to an old distribution. However, RedHat/CentOS still ship these old versions of tar, so I think the problem remains. The proposed changes do work and fix my problems, but may need refinement (as described in the patch comment). package/Makefile.package.in | 9 ++++++++- package/tar/tar.mk | 3 +++ toolchain/toolchain-external/Config.in | 6 +++--- toolchain/toolchain-external/ext-tool.mk | 8 ++++---- 4 files changed, 18 insertions(+), 8 deletions(-) ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 1 of 5] toolchain-external: allow downloading a custom toolchain 2011-08-02 18:33 [Buildroot] [PATCH 0 of 5] toolchain-external: support downloading custom toolchains + related misc. fixes Thomas De Schampheleire @ 2011-08-02 18:33 ` Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 2 of 5] Support URI schemes in DOWNLOAD function, when not using gentargets Thomas De Schampheleire ` (3 subsequent siblings) 4 siblings, 0 replies; 16+ messages in thread From: Thomas De Schampheleire @ 2011-08-02 18:33 UTC (permalink / raw) To: buildroot Custom external toolchains currently already have to be extracted by the user. This patch adds support for downloading a custom toolchain in the form of a tarball, which will be extracted by buildroot. Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> diff --git a/toolchain/toolchain-external/Config.in b/toolchain/toolchain-external/Config.in --- a/toolchain/toolchain-external/Config.in +++ b/toolchain/toolchain-external/Config.in @@ -189,12 +189,23 @@ config BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD bool "Download toolchain automatically" default y - depends on !BR2_TOOLCHAIN_EXTERNAL_CUSTOM help When enabled, Buildroot will automatically download and install the selected external toolchain. When disabled, Buildroot will use a pre-installed toolchain. +config BR2_TOOLCHAIN_EXTERNAL_SITE + string "Toolchain download site" + depends on BR2_TOOLCHAIN_EXTERNAL_CUSTOM && BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD + help + Base URL at which the custom toolchain can be downloaded. + +config BR2_TOOLCHAIN_EXTERNAL_SOURCE + string "Toolchain filename" + depends on BR2_TOOLCHAIN_EXTERNAL_CUSTOM && BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD + help + Filename of the toolchain package to download. + config BR2_TOOLCHAIN_EXTERNAL_PATH string "Toolchain path" default "/path/to/toolchain/usr" diff --git a/toolchain/toolchain-external/ext-tool.mk b/toolchain/toolchain-external/ext-tool.mk --- a/toolchain/toolchain-external/ext-tool.mk +++ b/toolchain/toolchain-external/ext-tool.mk @@ -160,6 +160,10 @@ TOOLCHAIN_EXTERNAL_DEPENDENCIES = $(STAMP_DIR)/ext-toolchain-checked endif +ifeq ($(BR2_TOOLCHAIN_EXTERNAL_CUSTOM),y) +TOOLCHAIN_EXTERNAL_DEPENDENCIES += $(STAMP_DIR)/ext-toolchain-checked +endif + ifeq ($(BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM2009Q1),y) TOOLCHAIN_EXTERNAL_SITE=http://www.codesourcery.com/sgpp/lite/arm/portal/package4571/public/arm-none-linux-gnueabi/ TOOLCHAIN_EXTERNAL_SOURCE=arm-2009q1-203-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2 @@ -197,11 +201,16 @@ TOOLCHAIN_EXTERNAL_SOURCE_2 = blackfin-toolchain-uclibc-full-2010R1-RC4.i386.tar.bz2 TOOLCHAIN_EXTERNAL_SOURCE = $(TOOLCHAIN_EXTERNAL_SOURCE_1) $(TOOLCHAIN_EXTERNAL_SOURCE_2) else +# Custom toolchain +TOOLCHAIN_EXTERNAL_SITE=$(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_SITE)) +TOOLCHAIN_EXTERNAL_SOURCE=$(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_SOURCE)) # A value must be set (even if unused), otherwise the # $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE) rule would override the main # $(DL_DIR) rule +ifeq (,$(TOOLCHAIN_EXTERNAL_SOURCE)) TOOLCHAIN_EXTERNAL_SOURCE=none endif +endif # Special handling for Blackfin toolchain, because of the split in two # tarballs, and the organization of tarball contents. The tarballs ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 2 of 5] Support URI schemes in DOWNLOAD function, when not using gentargets 2011-08-02 18:33 [Buildroot] [PATCH 0 of 5] toolchain-external: support downloading custom toolchains + related misc. fixes Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 1 of 5] toolchain-external: allow downloading a custom toolchain Thomas De Schampheleire @ 2011-08-02 18:33 ` Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 3 of 5] toolchain-external: fix some typos CodeSoucery -> CodeSourcery Thomas De Schampheleire ` (2 subsequent siblings) 4 siblings, 0 replies; 16+ messages in thread From: Thomas De Schampheleire @ 2011-08-02 18:33 UTC (permalink / raw) To: buildroot When using the gentargets/autotools/cmake infrastructure, the DOWNLOAD function uses an appropriate download method based on $(PKG)_SITE_METHOD, which is autodetected based on the URI if none was explicitly set. When the DOWNLOAD function is called directly, when not using gentargets, the SITE_METHOD is not autodetected, and thus the download defaults to wget. This patch adds URI scheme detection directly to the DOWNLOAD method, in case no SITE_METHOD was detected yet. Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> diff --git a/package/Makefile.package.in b/package/Makefile.package.in --- a/package/Makefile.package.in +++ b/package/Makefile.package.in @@ -184,12 +184,19 @@ # $(call DOWNLOAD,$(FOO_SITE),$(FOO_SOURCE)) ################################################################################ +geturischeme=$(firstword $(subst ://, ,$(call qstrip,$(1)))) + define DOWNLOAD $(Q)if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \ $(call $(DL_MODE)_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ; \ fi ; \ if test -n "$(1)" ; then \ - case "$($(PKG)_SITE_METHOD)" in \ + if test -z "$($(PKG)_SITE_METHOD)" ; then \ + scheme="$(call geturischeme,$(1))" ; \ + else \ + scheme="$($(PKG)_SITE_METHOD)" ; \ + fi ; \ + case "$$scheme" in \ git) $($(DL_MODE)_GIT) && exit ;; \ svn) $($(DL_MODE)_SVN) && exit ;; \ bzr) $($(DL_MODE)_BZR) && exit ;; \ ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 3 of 5] toolchain-external: fix some typos CodeSoucery -> CodeSourcery 2011-08-02 18:33 [Buildroot] [PATCH 0 of 5] toolchain-external: support downloading custom toolchains + related misc. fixes Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 1 of 5] toolchain-external: allow downloading a custom toolchain Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 2 of 5] Support URI schemes in DOWNLOAD function, when not using gentargets Thomas De Schampheleire @ 2011-08-02 18:33 ` Thomas De Schampheleire 2011-08-04 19:47 ` Thomas Petazzoni 2011-08-04 19:57 ` Peter Korsgaard 2011-08-02 18:33 ` [Buildroot] [PATCH 4 of 5] tar: create a host version Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains Thomas De Schampheleire 4 siblings, 2 replies; 16+ messages in thread From: Thomas De Schampheleire @ 2011-08-02 18:33 UTC (permalink / raw) To: buildroot Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> diff --git a/toolchain/toolchain-external/Config.in b/toolchain/toolchain-external/Config.in --- a/toolchain/toolchain-external/Config.in +++ b/toolchain/toolchain-external/Config.in @@ -120,7 +120,7 @@ - MIPS32 little endian soft float uclibc config BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_POWERPC201009 - bool "CodeSoucery PowerPC 2010.09" + bool "CodeSourcery PowerPC 2010.09" depends on BR2_powerpc select BR2_TOOLCHAIN_EXTERNAL_GLIBC select BR2_INSTALL_LIBSTDCPP @@ -138,7 +138,7 @@ - 970 glibc hard-float, 64 bits config BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_SH201009 - bool "CodeSoucery SH 2010.09" + bool "CodeSourcery SH 2010.09" depends on BR2_sh4a || BR2_sh4aeb select BR2_TOOLCHAIN_EXTERNAL_GLIBC select BR2_INSTALL_LIBSTDCPP @@ -153,7 +153,7 @@ - SH4A, uClibc, big endian config BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_SH2A_201009 - bool "CodeSoucery SH 2010.09" + bool "CodeSourcery SH 2010.09" depends on BR2_sh2a select BR2_TOOLCHAIN_EXTERNAL_UCLIBC select BR2_INSTALL_LIBSTDCPP ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 3 of 5] toolchain-external: fix some typos CodeSoucery -> CodeSourcery 2011-08-02 18:33 ` [Buildroot] [PATCH 3 of 5] toolchain-external: fix some typos CodeSoucery -> CodeSourcery Thomas De Schampheleire @ 2011-08-04 19:47 ` Thomas Petazzoni 2011-08-04 19:57 ` Peter Korsgaard 1 sibling, 0 replies; 16+ messages in thread From: Thomas Petazzoni @ 2011-08-04 19:47 UTC (permalink / raw) To: buildroot Le Tue, 02 Aug 2011 20:33:17 +0200, Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit : > Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Acked-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Peter: this one should go in 2011.08, it's just fixes. The other patches need discussion, so I'd prefer to handle them post 2011.08. Thomas -- Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 3 of 5] toolchain-external: fix some typos CodeSoucery -> CodeSourcery 2011-08-02 18:33 ` [Buildroot] [PATCH 3 of 5] toolchain-external: fix some typos CodeSoucery -> CodeSourcery Thomas De Schampheleire 2011-08-04 19:47 ` Thomas Petazzoni @ 2011-08-04 19:57 ` Peter Korsgaard 1 sibling, 0 replies; 16+ messages in thread From: Peter Korsgaard @ 2011-08-04 19:57 UTC (permalink / raw) To: buildroot >>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes: Thomas> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Committed, thanks. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 4 of 5] tar: create a host version 2011-08-02 18:33 [Buildroot] [PATCH 0 of 5] toolchain-external: support downloading custom toolchains + related misc. fixes Thomas De Schampheleire ` (2 preceding siblings ...) 2011-08-02 18:33 ` [Buildroot] [PATCH 3 of 5] toolchain-external: fix some typos CodeSoucery -> CodeSourcery Thomas De Schampheleire @ 2011-08-02 18:33 ` Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains Thomas De Schampheleire 4 siblings, 0 replies; 16+ messages in thread From: Thomas De Schampheleire @ 2011-08-02 18:33 UTC (permalink / raw) To: buildroot Some systems, like RedHat/CentOS, are even today shipped with ancient versions of GNU tar, like 1.15. This can cause trouble, for example when unpacking archives containing hardlinks with the --strip-components option. To avoid such problems, let buildroot create a host-tar that can be used when unpacking. Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> diff --git a/package/tar/tar.mk b/package/tar/tar.mk --- a/package/tar/tar.mk +++ b/package/tar/tar.mk @@ -8,3 +8,6 @@ TAR_SITE = $(BR2_GNU_MIRROR)/tar $(eval $(call AUTOTARGETS,package,tar)) +$(eval $(call AUTOTARGETS,package,tar,host)) + +HOST_TAR = $(HOST_DIR)/usr/bin/tar ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-08-02 18:33 [Buildroot] [PATCH 0 of 5] toolchain-external: support downloading custom toolchains + related misc. fixes Thomas De Schampheleire ` (3 preceding siblings ...) 2011-08-02 18:33 ` [Buildroot] [PATCH 4 of 5] tar: create a host version Thomas De Schampheleire @ 2011-08-02 18:33 ` Thomas De Schampheleire 2011-08-19 9:11 ` Thomas De Schampheleire 4 siblings, 1 reply; 16+ messages in thread From: Thomas De Schampheleire @ 2011-08-02 18:33 UTC (permalink / raw) To: buildroot Using the system tar with --strip-components causes problems when its version is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15. To avoid such problems, always use host-tar instead of tar to unpack toolchains. Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> --- To be honest, I'm not sure what the best way is to handle this. The same problems with tar could occur in other places than just external toolchains. So maybe we should use host-tar in more places. On the other hand, most users have a recent, working, tar version on there system. It may thus make more sense to detect the system tar version and only use host-tar if it's too old. diff --git a/toolchain/toolchain-external/ext-tool.mk b/toolchain/toolchain-external/ext-tool.mk --- a/toolchain/toolchain-external/ext-tool.mk +++ b/toolchain/toolchain-external/ext-tool.mk @@ -155,7 +155,7 @@ endif ifeq ($(BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD),y) -TOOLCHAIN_EXTERNAL_DEPENDENCIES = $(TOOLCHAIN_EXTERNAL_DIR)/.extracted +TOOLCHAIN_EXTERNAL_DEPENDENCIES = host-tar $(TOOLCHAIN_EXTERNAL_DIR)/.extracted else TOOLCHAIN_EXTERNAL_DEPENDENCIES = $(STAMP_DIR)/ext-toolchain-checked endif @@ -227,9 +227,9 @@ $(TOOLCHAIN_EXTERNAL_DIR)/.extracted: $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_1) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_2) mkdir -p $(@D) $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE_1))) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_1) | \ - $(TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - + $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE_2))) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_2) | \ - $(TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - + $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - ifeq ($(TOOLCHAIN_EXTERNAL_PREFIX),bfin-uclinux) rm -rf $(TOOLCHAIN_EXTERNAL_DIR)/bfin-linux-uclibc mv $(TOOLCHAIN_EXTERNAL_DIR)/bfin-uclinux $(TOOLCHAIN_EXTERNAL_DIR)/tmp @@ -249,7 +249,7 @@ $(TOOLCHAIN_EXTERNAL_DIR)/.extracted: $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE) mkdir -p $(@D) - $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE))) $^ | $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) - + $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE))) $^ | $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) - $(Q)touch $@ endif ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-08-02 18:33 ` [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains Thomas De Schampheleire @ 2011-08-19 9:11 ` Thomas De Schampheleire 0 siblings, 0 replies; 16+ messages in thread From: Thomas De Schampheleire @ 2011-08-19 9:11 UTC (permalink / raw) To: buildroot On Tue, Aug 2, 2011 at 8:33 PM, Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> wrote: > Using the system tar with --strip-components causes problems when its version is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15. To avoid such problems, always use host-tar instead of tar to unpack toolchains. > > Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> > > --- > To be honest, I'm not sure what the best way is to handle this. The same problems with tar could occur in other places than just external toolchains. So maybe we should use host-tar in more places. > On the other hand, most users have a recent, working, tar version on there system. It may thus make more sense to detect the system tar version and only use host-tar if it's too old. > > diff --git a/toolchain/toolchain-external/ext-tool.mk b/toolchain/toolchain-external/ext-tool.mk > --- a/toolchain/toolchain-external/ext-tool.mk > +++ b/toolchain/toolchain-external/ext-tool.mk > @@ -155,7 +155,7 @@ > ?endif > > ?ifeq ($(BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD),y) > -TOOLCHAIN_EXTERNAL_DEPENDENCIES = $(TOOLCHAIN_EXTERNAL_DIR)/.extracted > +TOOLCHAIN_EXTERNAL_DEPENDENCIES = host-tar $(TOOLCHAIN_EXTERNAL_DIR)/.extracted It turns out that this has the annoying side-effect of re-running the 'ext-toolchain-installed' step each time, because host-tar is not an existing file. One can workaround this as follows: include package/tar/tar.mk $(STAMP_DIR)/tar-installed: $(BUILD_DIR)/host-tar-$(TAR_VERSION)/.stamp_host_installed $(Q)touch $@ TOOLCHAIN_EXTERNAL_DEPENDENCIES = $(STAMP_DIR)tar-installed $(TOOLCHAIN_EXTERNAL_DIR)/.extracted This is not very beautiful because you need to include package/tar/tar.mk to get to $(TAR_VERSION). Is there a better way? Thanks, Thomas > ?else > ?TOOLCHAIN_EXTERNAL_DEPENDENCIES = $(STAMP_DIR)/ext-toolchain-checked > ?endif > @@ -227,9 +227,9 @@ > ?$(TOOLCHAIN_EXTERNAL_DIR)/.extracted: $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_1) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_2) > ? ? ? ?mkdir -p $(@D) > ? ? ? ?$(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE_1))) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_1) | \ > - ? ? ? ? ? ? ? $(TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - > + ? ? ? ? ? ? ? $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - > ? ? ? ?$(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE_2))) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_2) | \ > - ? ? ? ? ? ? ? $(TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - > + ? ? ? ? ? ? ? $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - > ?ifeq ($(TOOLCHAIN_EXTERNAL_PREFIX),bfin-uclinux) > ? ? ? ?rm -rf $(TOOLCHAIN_EXTERNAL_DIR)/bfin-linux-uclibc > ? ? ? ?mv $(TOOLCHAIN_EXTERNAL_DIR)/bfin-uclinux $(TOOLCHAIN_EXTERNAL_DIR)/tmp > @@ -249,7 +249,7 @@ > > ?$(TOOLCHAIN_EXTERNAL_DIR)/.extracted: $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE) > ? ? ? ?mkdir -p $(@D) > - ? ? ? $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE))) $^ | $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) - > + ? ? ? $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE))) $^ | $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) - > ? ? ? ?$(Q)touch $@ > ?endif > > ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 0 of 5] toolchain-external: improve support for custom toolchains @ 2011-09-21 6:55 Thomas De Schampheleire 2011-09-21 6:55 ` [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains Thomas De Schampheleire 0 siblings, 1 reply; 16+ messages in thread From: Thomas De Schampheleire @ 2011-09-21 6:55 UTC (permalink / raw) To: buildroot This patch series adds the ability to download and unpack a custom external toolchain (instead of expecting the user to do that), and adds a configuration option to copy extra external libraries to the target. (Note that if you apply this on top of my scp/hg support patches, you should take care in package/Makefile.package.in about a potentially double geturischeme definition.) Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> --- package/Makefile.package.in | 10 +++++++++- package/tar/tar.mk | 3 +++ toolchain/toolchain-external/Config.in | 21 ++++++++++++++++++++- toolchain/toolchain-external/ext-tool.mk | 21 +++++++++++++++++---- 4 files changed, 49 insertions(+), 6 deletions(-) ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-09-21 6:55 [Buildroot] [PATCH 0 of 5] toolchain-external: improve support for custom toolchains Thomas De Schampheleire @ 2011-09-21 6:55 ` Thomas De Schampheleire 2011-09-21 8:02 ` Thomas Petazzoni 0 siblings, 1 reply; 16+ messages in thread From: Thomas De Schampheleire @ 2011-09-21 6:55 UTC (permalink / raw) To: buildroot Some toolchains, like the one built with buildroot itself, use hardlinks (for example to link between the c++ and g++ binary). Unpacking such a toolchain with the --strip-components options does not work correctly if the system tar is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15. To avoid such problems, always use host-tar instead of tar to unpack toolchains. Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> --- toolchain/toolchain-external/ext-tool.mk | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-) diff --git a/toolchain/toolchain-external/ext-tool.mk b/toolchain/toolchain-external/ext-tool.mk --- a/toolchain/toolchain-external/ext-tool.mk +++ b/toolchain/toolchain-external/ext-tool.mk @@ -156,7 +156,10 @@ endif ifeq ($(BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD),y) -TOOLCHAIN_EXTERNAL_DEPENDENCIES = $(TOOLCHAIN_EXTERNAL_DIR)/.extracted + ifeq (,$(wildcard $(HOST_DIR)/usr/bin/tar)) + TOOLCHAIN_EXTERNAL_DEPENDENCIES = host-tar + endif + TOOLCHAIN_EXTERNAL_DEPENDENCIES += $(TOOLCHAIN_EXTERNAL_DIR)/.extracted else TOOLCHAIN_EXTERNAL_DEPENDENCIES = $(STAMP_DIR)/ext-toolchain-checked endif @@ -228,9 +231,9 @@ $(TOOLCHAIN_EXTERNAL_DIR)/.extracted: $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_1) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_2) mkdir -p $(@D) $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE_1))) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_1) | \ - $(TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - + $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE_2))) $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE_2) | \ - $(TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - + $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=3 --hard-dereference -C $(@D) $(TAR_OPTIONS) - ifeq ($(TOOLCHAIN_EXTERNAL_PREFIX),bfin-uclinux) rm -rf $(TOOLCHAIN_EXTERNAL_DIR)/bfin-linux-uclibc mv $(TOOLCHAIN_EXTERNAL_DIR)/bfin-uclinux $(TOOLCHAIN_EXTERNAL_DIR)/tmp @@ -250,7 +253,7 @@ $(TOOLCHAIN_EXTERNAL_DIR)/.extracted: $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE) mkdir -p $(@D) - $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE))) $^ | $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) - + $(INFLATE$(suffix $(TOOLCHAIN_EXTERNAL_SOURCE))) $^ | $(HOST_TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) - $(Q)touch $@ endif ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-09-21 6:55 ` [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains Thomas De Schampheleire @ 2011-09-21 8:02 ` Thomas Petazzoni 2011-09-21 8:35 ` Thomas De Schampheleire 0 siblings, 1 reply; 16+ messages in thread From: Thomas Petazzoni @ 2011-09-21 8:02 UTC (permalink / raw) To: buildroot Hello, Le Wed, 21 Sep 2011 08:55:35 +0200, Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit : > Some toolchains, like the one built with buildroot itself, use hardlinks (for > example to link between the c++ and g++ binary). Unpacking such a toolchain > with the --strip-components options does not work correctly if the system tar > is too old (<1.17). Even recent releases of RedHat/CentOS still ship with > tar 1.15. > To avoid such problems, always use host-tar instead of tar to unpack > toolchains. I'm sorry but we need to find a better solution for this. I really don't want all users to pay the price of building tar to extract the external toolchain just because some crappy distribution has an old version of tar. Can we make this conditional on the version of tar ? I'd also like some comment in the code suggesting to remove this crap 1 or 2 years from now, once those RedHat/CentOS distributions will have been upgraded. Also, this proposal only takes care of using the compiled host tar for toolchain extraction. Why don't we also use it to extract all other packages (except tar itself, of course...) ? Thomas -- Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-09-21 8:02 ` Thomas Petazzoni @ 2011-09-21 8:35 ` Thomas De Schampheleire 2011-09-21 10:54 ` Michael S. Zick 0 siblings, 1 reply; 16+ messages in thread From: Thomas De Schampheleire @ 2011-09-21 8:35 UTC (permalink / raw) To: buildroot Hi, On Wed, Sep 21, 2011 at 10:02 AM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Hello, > > Le Wed, 21 Sep 2011 08:55:35 +0200, > Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit : > >> Some toolchains, like the one built with buildroot itself, use hardlinks (for >> example to link between the c++ and g++ binary). Unpacking such a toolchain >> with the --strip-components options does not work correctly if the system tar >> is too old (<1.17). Even recent releases of RedHat/CentOS still ship with >> tar 1.15. >> To avoid such problems, always use host-tar instead of tar to unpack >> toolchains. > > I'm sorry but we need to find a better solution for this. I really > don't want all users to pay the price of building tar to extract > the external toolchain just because some crappy distribution has an old > version of tar. Can we make this conditional on the version of tar ? > I'd also like some comment in the code suggesting to remove this crap 1 > or 2 years from now, once those RedHat/CentOS distributions will have > been upgraded. > > Also, this proposal only takes care of using the compiled host tar for > toolchain extraction. Why don't we also use it to extract all other > packages (except tar itself, of course...) ? I agree with both your points. In fact, I raised the same doubts in my original patches: http://lists.busybox.net/pipermail/buildroot/2011-August/044784.html ---- To be honest, I'm not sure what the best way is to handle this. The same problems with tar could occur in other places than just external toolchains. So maybe we should use host-tar in more places. On the other hand, most users have a recent, working, tar version on there system. It may thus make more sense to detect the system tar version and only use host-tar if it's too old. ---- Regarding deprecating this change after some time: I'm not sure whether this is still necessary if we make this conditional on the system tar version. A user that is still on some old corporate system with RedHat 5, may still need this patch in two or three years. I'll see if I can update this patch with an automatic detection. How would you go about to use the host-tar for all packages? I need to add an extra autotargets call to package/tar/tar.mk, but I also need to redefine TAR somewhere (currently it's defined in package/Makefile.package.in). But TAR_STRIP_COMPONENTS uses TAR with a := dependency, and package/tar/tar.mk is only included after package/Makefile.package.in, so I would need to add the conditional check in two places. Unless I move the check to a new file that is included from both tar.mk and Makefile.package.in. In this case, where will I place the file? How do you see this? Thanks for your comments, Thomas > > Thomas > -- > Thomas Petazzoni, Free Electrons > Kernel, drivers, real-time and embedded Linux > development, consulting, training and support. > http://free-electrons.com > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-09-21 8:35 ` Thomas De Schampheleire @ 2011-09-21 10:54 ` Michael S. Zick 2011-09-21 12:16 ` Thomas De Schampheleire 2011-09-22 20:02 ` Arnout Vandecappelle 0 siblings, 2 replies; 16+ messages in thread From: Michael S. Zick @ 2011-09-21 10:54 UTC (permalink / raw) To: buildroot On Wed September 21 2011, Thomas De Schampheleire wrote: > Hi, > > On Wed, Sep 21, 2011 at 10:02 AM, Thomas Petazzoni > <thomas.petazzoni@free-electrons.com> wrote: > > Hello, > > > > Le Wed, 21 Sep 2011 08:55:35 +0200, > > Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit : > > > >> Some toolchains, like the one built with buildroot itself, use hardlinks (for > >> example to link between the c++ and g++ binary). Unpacking such a toolchain > >> with the --strip-components options does not work correctly if the system tar > >> is too old (<1.17). Even recent releases of RedHat/CentOS still ship with > >> tar 1.15. > >> To avoid such problems, always use host-tar instead of tar to unpack > >> toolchains. > > > > I'm sorry but we need to find a better solution for this. I really > > don't want all users to pay the price of building tar to extract > > the external toolchain just because some crappy distribution has an old > > version of tar. Can we make this conditional on the version of tar ? > > I'd also like some comment in the code suggesting to remove this crap 1 > > or 2 years from now, once those RedHat/CentOS distributions will have > > been upgraded. > > > > Also, this proposal only takes care of using the compiled host tar for > > toolchain extraction. Why don't we also use it to extract all other > > packages (except tar itself, of course...) ? > > I agree with both your points. > A suggestion here - In that build system dependencies code/file, the one recently extended to also list rsync... Perhaps extend that with a bit of infrastructure that can check not only if present (with which) but also the version? That should help keep the build environment sane, with a message to the user if something is too old; And should relieve the maintainers of the task of remembering to "re-correct" the build system someday in the future. Mike > In fact, I raised the same doubts in my > original patches: > http://lists.busybox.net/pipermail/buildroot/2011-August/044784.html > ---- > To be honest, I'm not sure what the best way is to handle this. The > same problems with tar could occur in other places than just external > toolchains. So maybe we should use host-tar in more places. > On the other hand, most users have a recent, working, tar version on > there system. It may thus make more sense to detect the system tar > version and only use host-tar if it's too old. > ---- > > > Regarding deprecating this change after some time: I'm not sure > whether this is still necessary if we make this conditional on the > system tar version. A user that is still on some old corporate system > with RedHat 5, may still need this patch in two or three years. > > I'll see if I can update this patch with an automatic detection. > > How would you go about to use the host-tar for all packages? I need to > add an extra autotargets call to package/tar/tar.mk, but I also need > to redefine TAR somewhere (currently it's defined in > package/Makefile.package.in). But TAR_STRIP_COMPONENTS uses TAR with a > := dependency, and package/tar/tar.mk is only included after > package/Makefile.package.in, so I would need to add the conditional > check in two places. > Unless I move the check to a new file that is included from both > tar.mk and Makefile.package.in. In this case, where will I place the > file? > How do you see this? > > Thanks for your comments, > Thomas > > > > > Thomas > > -- > > Thomas Petazzoni, Free Electrons > > Kernel, drivers, real-time and embedded Linux > > development, consulting, training and support. > > http://free-electrons.com > > _______________________________________________ > > buildroot mailing list > > buildroot at busybox.net > > http://lists.busybox.net/mailman/listinfo/buildroot > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot > > ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-09-21 10:54 ` Michael S. Zick @ 2011-09-21 12:16 ` Thomas De Schampheleire 2011-09-22 20:02 ` Arnout Vandecappelle 1 sibling, 0 replies; 16+ messages in thread From: Thomas De Schampheleire @ 2011-09-21 12:16 UTC (permalink / raw) To: buildroot Hi, On Wed, Sep 21, 2011 at 12:54 PM, Michael S. Zick <minimod@morethan.org> wrote: > On Wed September 21 2011, Thomas De Schampheleire wrote: >> Hi, >> >> On Wed, Sep 21, 2011 at 10:02 AM, Thomas Petazzoni >> <thomas.petazzoni@free-electrons.com> wrote: >> > Hello, >> > >> > Le Wed, 21 Sep 2011 08:55:35 +0200, >> > Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit : >> > >> >> Some toolchains, like the one built with buildroot itself, use hardlinks (for >> >> example to link between the c++ and g++ binary). Unpacking such a toolchain >> >> with the --strip-components options does not work correctly if the system tar >> >> is too old (<1.17). Even recent releases of RedHat/CentOS still ship with >> >> tar 1.15. >> >> To avoid such problems, always use host-tar instead of tar to unpack >> >> toolchains. >> > >> > I'm sorry but we need to find a better solution for this. I really >> > don't want all users to pay the price of building tar to extract >> > the external toolchain just because some crappy distribution has an old >> > version of tar. Can we make this conditional on the version of tar ? >> > I'd also like some comment in the code suggesting to remove this crap 1 >> > or 2 years from now, once those RedHat/CentOS distributions will have >> > been upgraded. >> > >> > Also, this proposal only takes care of using the compiled host tar for >> > toolchain extraction. Why don't we also use it to extract all other >> > packages (except tar itself, of course...) ? >> >> I agree with both your points. >> > > A suggestion here - > In that build system dependencies code/file, the one recently extended > to also list rsync... > > Perhaps extend that with a bit of infrastructure that can check not > only if present (with which) but also the version? > > That should help keep the build environment sane, with a message > to the user if something is too old; > And should relieve the maintainers of the task of remembering to > "re-correct" the build system someday in the future. Thanks for the hint! I will definitely look into that dependencies file.. > > Mike >> In fact, I raised the same doubts in my >> original patches: >> http://lists.busybox.net/pipermail/buildroot/2011-August/044784.html >> ---- >> To be honest, I'm not sure what the best way is to handle this. The >> same problems with tar could occur in other places than just external >> toolchains. So maybe we should use host-tar in more places. >> On the other hand, most users have a recent, working, tar version on >> there system. It may thus make more sense to detect the system tar >> version and only use host-tar if it's too old. >> ---- >> >> >> Regarding deprecating this change after some time: I'm not sure >> whether this is still necessary if we make this conditional on the >> system tar version. A user that is still on some old corporate system >> with RedHat 5, may still need this patch in two or three years. >> >> I'll see if I can update this patch with an automatic detection. >> >> How would you go about to use the host-tar for all packages? I need to >> add an extra autotargets call to package/tar/tar.mk, but I also need >> to redefine TAR somewhere (currently it's defined in >> package/Makefile.package.in). But TAR_STRIP_COMPONENTS uses TAR with a >> := dependency, and package/tar/tar.mk is only included after >> package/Makefile.package.in, so I would need to add the conditional >> check in two places. >> Unless I move the check to a new file that is included from both >> tar.mk and Makefile.package.in. In this case, where will I place the >> file? >> How do you see this? >> >> Thanks for your comments, >> Thomas >> >> > >> > Thomas >> > -- >> > Thomas Petazzoni, Free Electrons >> > Kernel, drivers, real-time and embedded Linux >> > development, consulting, training and support. >> > http://free-electrons.com >> > _______________________________________________ >> > buildroot mailing list >> > buildroot at busybox.net >> > http://lists.busybox.net/mailman/listinfo/buildroot >> _______________________________________________ >> buildroot mailing list >> buildroot at busybox.net >> http://lists.busybox.net/mailman/listinfo/buildroot >> >> > > > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot > ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-09-21 10:54 ` Michael S. Zick 2011-09-21 12:16 ` Thomas De Schampheleire @ 2011-09-22 20:02 ` Arnout Vandecappelle 2011-09-23 6:18 ` Thomas De Schampheleire 1 sibling, 1 reply; 16+ messages in thread From: Arnout Vandecappelle @ 2011-09-22 20:02 UTC (permalink / raw) To: buildroot On Wednesday 21 September 2011 12:54:55, Michael S. Zick wrote: > That should help keep the build environment sane, with a message > to the user if something is too old; > And should relieve the maintainers of the task of remembering to > "re-correct" the build system someday in the future. How does that help the users who are stuck on RH5? Buildroot would still not work for them (although with a clearer explanation why not), so they'd have to implement Thomas's patches after all. Regards, Arnout -- Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286540 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 31BB CF53 8660 6F88 345D 54CC A836 5879 20D7 CF43 ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains 2011-09-22 20:02 ` Arnout Vandecappelle @ 2011-09-23 6:18 ` Thomas De Schampheleire 0 siblings, 0 replies; 16+ messages in thread From: Thomas De Schampheleire @ 2011-09-23 6:18 UTC (permalink / raw) To: buildroot On Thu, Sep 22, 2011 at 10:02 PM, Arnout Vandecappelle <arnout@mind.be> wrote: > > On Wednesday 21 September 2011 12:54:55, Michael S. Zick wrote: >> That should help keep the build environment sane, with a message >> to the user if something is too old; >> And should relieve the maintainers of the task of remembering to >> "re-correct" the build system someday in the future. > > ?How does that help the users who are stuck on RH5? ?Buildroot would still not > work for them (although with a clearer explanation why not), so they'd have to > implement Thomas's patches after all. The way I interpreted this was to add a check to dependencies.sh on the system tar version. If it's too old, we go ahead and build our own host-tar and use it as $(TAR). If it is recent enough, we don't change anything. I still need to try that out though. Best regards, Thomas ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2011-09-23 6:18 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-08-02 18:33 [Buildroot] [PATCH 0 of 5] toolchain-external: support downloading custom toolchains + related misc. fixes Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 1 of 5] toolchain-external: allow downloading a custom toolchain Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 2 of 5] Support URI schemes in DOWNLOAD function, when not using gentargets Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 3 of 5] toolchain-external: fix some typos CodeSoucery -> CodeSourcery Thomas De Schampheleire 2011-08-04 19:47 ` Thomas Petazzoni 2011-08-04 19:57 ` Peter Korsgaard 2011-08-02 18:33 ` [Buildroot] [PATCH 4 of 5] tar: create a host version Thomas De Schampheleire 2011-08-02 18:33 ` [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains Thomas De Schampheleire 2011-08-19 9:11 ` Thomas De Schampheleire 2011-09-21 6:55 [Buildroot] [PATCH 0 of 5] toolchain-external: improve support for custom toolchains Thomas De Schampheleire 2011-09-21 6:55 ` [Buildroot] [PATCH 5 of 5] toolchain-external: use host-tar instead of tar to unpack toolchains Thomas De Schampheleire 2011-09-21 8:02 ` Thomas Petazzoni 2011-09-21 8:35 ` Thomas De Schampheleire 2011-09-21 10:54 ` Michael S. Zick 2011-09-21 12:16 ` Thomas De Schampheleire 2011-09-22 20:02 ` Arnout Vandecappelle 2011-09-23 6:18 ` Thomas De Schampheleire
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.