[Buildroot] [PATCH 0/2] Allow only downloading sources with crosstool-NG backend

[Buildroot] [PATCH 0/2] Allow only downloading sources with crosstool-NG backend

[Yann E. MORIN]

Hello All!

This patch series allows one to call 'make source' for later off-line build,
even when the toolchain backend is crosstool-NG.

  [PATCH 1/2] toolchain/crosstool-ng: allow pre-downloading the sources
  [PATCH 2/2] source: print messages when all sources downloaded

Regards,
Yann E. MORIN

[Buildroot] [PATCH 1/2] toolchain/crosstool-ng: allow pre-downloading the sources

[Yann E. MORIN]

On "make source", just download the required sources, and do not
build the entire toolchain. This requires a new patch against
crosstool-NG (got from upstream).

The "uclibc-source" rule requires an installed and configured
crosstool-NG, so it has to be after the variables declaration.
For consistency, move down the main rule "uclibc".

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
---
 ...-break-on-existing-symlinks-and-build.log.patch |   76 ++++++++++++++++++++
 toolchain/toolchain-crosstool-ng/crosstool-ng.mk   |   17 +++--
 2 files changed, 88 insertions(+), 5 deletions(-)
 create mode 100644 toolchain/toolchain-crosstool-ng/002-dont-break-on-existing-symlinks-and-build.log.patch

diff --git a/toolchain/toolchain-crosstool-ng/002-dont-break-on-existing-symlinks-and-build.log.patch b/toolchain/toolchain-crosstool-ng/002-dont-break-on-existing-symlinks-and-build.log.patch
new file mode 100644
index 0000000..2bb613f
--- /dev/null
+++ b/toolchain/toolchain-crosstool-ng/002-dont-break-on-existing-symlinks-and-build.log.patch
@@ -0,0 +1,76 @@
+
+# HG changeset patch
+# User "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
+# Date 1310507544 -7200
+# Node ID 1ebc2248cc60230cd53ff94ae8f8f1e3261461a3
+# Parent  f1c730da41f3b22ca1a1e1b14592da4b7c078797
+scripts, cc/gcc: do not fail on existing symlinks or build.log
+
+If the user builds a toolchain over an existing one, so, without removing
+CT_PREFIX_DIR, the build fails as the symlinks already exist, as does the
+build.log.
+
+This can also happen (for build.log) if the user first ran in download-
+or extract-only.
+
+Patch (with no SoB) originally from:
+  Phil Wilshire <phil.wilshire@overturenetworks.com>
+
+Modified by me as it did not apply cleanly.
+
+Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
+
+diff -r f1c730da41f3 -r 1ebc2248cc60 scripts/build/cc/gcc.sh
+--- a/scripts/build/cc/gcc.sh	Tue Jul 12 23:21:27 2011 +0200
++++ b/scripts/build/cc/gcc.sh	Tue Jul 12 23:52:24 2011 +0200
+@@ -386,7 +386,7 @@
+     # check whether compiler has an extension
+     file="$( ls -1 "${core_prefix_dir}/bin/${CT_TARGET}-gcc."* 2>/dev/null || true )"
+     [ -z "${file}" ] || ext=".${file##*.}"
+-    CT_DoExecLog ALL ln -sv "${CT_TARGET}-gcc${ext}" "${core_prefix_dir}/bin/${CT_TARGET}-cc${ext}"
++    CT_DoExecLog ALL ln -sfv "${CT_TARGET}-gcc${ext}" "${core_prefix_dir}/bin/${CT_TARGET}-cc${ext}"
+ 
+     CT_EndStep
+ }
+@@ -630,7 +630,7 @@
+     # check whether compiler has an extension
+     file="$( ls -1 "${CT_PREFIX_DIR}/bin/${CT_TARGET}-gcc."* 2>/dev/null || true )"
+     [ -z "${file}" ] || ext=".${file##*.}"
+-    CT_DoExecLog ALL ln -sv "${CT_TARGET}-gcc${ext}" "${CT_PREFIX_DIR}/bin/${CT_TARGET}-cc${ext}"
++    CT_DoExecLog ALL ln -sfv "${CT_TARGET}-gcc${ext}" "${CT_PREFIX_DIR}/bin/${CT_TARGET}-cc${ext}"
+ 
+     CT_EndStep
+ }
+diff -r f1c730da41f3 -r 1ebc2248cc60 scripts/build/internals.sh
+--- a/scripts/build/internals.sh	Tue Jul 12 23:21:27 2011 +0200
++++ b/scripts/build/internals.sh	Tue Jul 12 23:52:24 2011 +0200
+@@ -86,14 +86,14 @@
+     for t in "${CT_TARGET}-"*; do
+         if [ -n "${CT_TARGET_ALIAS}" ]; then
+             _t=$(echo "$t" |sed -r -e 's/^'"${CT_TARGET}"'-/'"${CT_TARGET_ALIAS}"'-/;')
+-            CT_DoExecLog ALL ln -sv "${t}" "${_t}"
++            CT_DoExecLog ALL ln -sfv "${t}" "${_t}"
+         fi
+         if [ -n "${CT_TARGET_ALIAS_SED_EXPR}" ]; then
+             _t=$(echo "$t" |sed -r -e "${CT_TARGET_ALIAS_SED_EXPR}")
+             if [ "${_t}" = "${t}" ]; then
+                 CT_DoLog WARN "The sed expression '${CT_TARGET_ALIAS_SED_EXPR}' has no effect on '${t}'"
+             else
+-                CT_DoExecLog ALL ln -sv "${t}" "${_t}"
++                CT_DoExecLog ALL ln -sfv "${t}" "${_t}"
+             fi
+         fi
+     done
+diff -r f1c730da41f3 -r 1ebc2248cc60 scripts/crosstool-NG.sh.in
+--- a/scripts/crosstool-NG.sh.in	Tue Jul 12 23:21:27 2011 +0200
++++ b/scripts/crosstool-NG.sh.in	Tue Jul 12 23:52:24 2011 +0200
+@@ -604,7 +604,7 @@
+ # moment... Consign all ouptut to oblivion...
+ CT_DoLog INFO "Finishing installation (may take a few seconds)..."
+ exec >/dev/null 2>&1
+-
++rm -f ${CT_PREFIX_DIR}/build.log.bz2
+ if [ "${CT_LOG_TO_FILE}" = "y" ]; then
+     cp "${tmp_log_file}" "${CT_PREFIX_DIR}/build.log"
+     if [ "${CT_LOG_FILE_COMPRESS}" = y ]; then
+
diff --git a/toolchain/toolchain-crosstool-ng/crosstool-ng.mk b/toolchain/toolchain-crosstool-ng/crosstool-ng.mk
index 2d1b51c..d3fcc65 100644
--- a/toolchain/toolchain-crosstool-ng/crosstool-ng.mk
+++ b/toolchain/toolchain-crosstool-ng/crosstool-ng.mk
@@ -6,11 +6,6 @@
 # except that it is not pre-built.
 
 #-----------------------------------------------------------------------------
-# 'uclibc' is the target to depend on to get the toolchain and prepare
-# the staging directory and co.
-uclibc: dependencies $(STAMP_DIR)/ct-ng-toolchain-installed
-
-#-----------------------------------------------------------------------------
 # Internal variables
 
 # Crostool-NG hard-coded configuration options
@@ -33,6 +28,18 @@ PATH=$(HOST_PATH) $(CTNG_DIR)/ct-ng -C $(CTNG_DIR) --no-print-directory $(1)
 endef
 
 #-----------------------------------------------------------------------------
+# 'uclibc' is the target to depend on to get the toolchain and prepare
+# the staging directory and co.
+uclibc: dependencies $(STAMP_DIR)/ct-ng-toolchain-installed
+
+# 'uclibc-source' is the target used by the infra structure to mean
+# "we just want to download the toolchain's sources, not build it"
+# For crosstool-NG, we need it to be configured before we can download;
+# then we have to override a config option to just do the download
+uclibc-source: $(CTNG_DIR)/.config
+	$(Q)$(call ctng,build CT_ONLY_DOWNLOAD=y)
+
+#-----------------------------------------------------------------------------
 # Installing the libs to target/ and staging/
 
 #--------------
-- 
1.7.2.5

[Buildroot] [PATCH 2/2] source: print messages when all sources downloaded

[Yann E. MORIN]

When downloading all sources for later off-line build (eg. make source),
and the toolchain backend is crosstool-NG, the last messages the user
wil see is something like:
  [INFO ]  Retrieving needed toolchain components' tarballs: done in 98.02s (at 01:39)
  [INFO ]  Build completed at 20110713.000605
  [INFO ]  (elapsed: 1:39.21)
  [INFO ]  Finishing installation (may take a few seconds)...

which is pretty uninformative, here; crosstool-NG is just saying that
it just finished what it was asked for, that is download the sources,
but there is no separate message for that.

To avoid confusion, add an explicit message that makes it clear that
downloading the sources was sucessful, and nothing more.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
---
 Makefile |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/Makefile b/Makefile
index 2f3802c..3021add 100644
--- a/Makefile
+++ b/Makefile
@@ -487,6 +487,9 @@ target-purgelocales:
 endif
 
 source: $(TARGETS_SOURCE) $(HOST_SOURCE)
+	@echo ""
+	@echo "*"
+	@echo "* Finished downloading the required packages' sources"
 
 _source-check:
 	$(MAKE) DL_MODE=SOURCE_CHECK $(EXTRAMAKEARGS) source
-- 
1.7.2.5

[Buildroot] [PATCH 1/2] toolchain/crosstool-ng: allow pre-downloading the sources

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@anciens.enib.fr> writes:

 Yann> On "make source", just download the required sources, and do not
 Yann> build the entire toolchain. This requires a new patch against
 Yann> crosstool-NG (got from upstream).

 Yann> The "uclibc-source" rule requires an installed and configured
 Yann> crosstool-NG, so it has to be after the variables declaration.
 Yann> For consistency, move down the main rule "uclibc".

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/2] source: print messages when all sources downloaded

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@anciens.enib.fr> writes:

 Yann> When downloading all sources for later off-line build (eg. make source),
 Yann> and the toolchain backend is crosstool-NG, the last messages the user
 Yann> wil see is something like:
 Yann>   [INFO ]  Retrieving needed toolchain components' tarballs: done in 98.02s (at 01:39)
 Yann>   [INFO ]  Build completed at 20110713.000605
 Yann>   [INFO ]  (elapsed: 1:39.21)
 Yann>   [INFO ]  Finishing installation (may take a few seconds)...

 Yann> which is pretty uninformative, here; crosstool-NG is just saying that
 Yann> it just finished what it was asked for, that is download the sources,
 Yann> but there is no separate message for that.

 Yann> To avoid confusion, add an explicit message that makes it clear that
 Yann> downloading the sources was sucessful, and nothing more.

I agree it isn't optimal, but this breaks E.G. make external-deps

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/2] source: print messages when all sources downloaded

[Yann E. MORIN]

Peter, All,

On Wednesday 13 July 2011 08:37:26 Peter Korsgaard wrote:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@anciens.enib.fr> writes:
[--SNIP--]
>  Yann> To avoid confusion, add an explicit message that makes it clear that
>  Yann> downloading the sources was sucessful, and nothing more.
> 
> I agree it isn't optimal, but this breaks E.G. make external-deps

Indeed. We could print the message only if DL_MODE==DOWNLOAD. Would that
be OK with you?

Incidentally, external-deps and source-check will not be really complete
when crostool-NG is the backend, as there is no way to ask such a thing
to crosstool-NG.

Note that source-check is currently borked due to some unconfigured
defconfig files for bootloaders and kernel. Eg. for barebox:
  $ make -C ../br.upstream O=$(pwd) source-check
  [--SNIP-]
  make O=/home/ymorin/dev/buildroot/O _source-check
  make[1]: Entering directory `/home/ymorin/dev/buildroot/br.upstream'
  boot/barebox/barebox.mk:60: *** No Barebox defconfig file. Check your
  BR2_TARGET_BAREBOX_BOARD_DEFCONFIG setting.  Stop.
  make[1]: Leaving directory `/home/ymorin/dev/buildroot/br.upstream'
  make: *** [source-check] Error 2
  make: Leaving directory `/home/ymorin/dev/buildroot/br.upstream'

Setting the barebox defconfig to non-empty now breaks on u-boot, still
because the defconfig entry is empty. And then it is the Linux kernel
that breaks for the same reason. :-/

I'll push a patch shortly...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/2] source: print messages when all sources downloaded

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@anciens.enib.fr> writes:

Hi,

 Yann> Indeed. We could print the message only if DL_MODE==DOWNLOAD. Would that
 Yann> be OK with you?

We could, but I'm not really sure if we should. We don't do anything
like that for any of the other make targets.

 Yann> Incidentally, external-deps and source-check will not be really complete
 Yann> when crostool-NG is the backend, as there is no way to ask such a thing
 Yann> to crosstool-NG.

Ok, also not by playing games with a replacement wget/curl/whatever that
just prints basename $1 or similar?

 Yann> Note that source-check is currently borked due to some unconfigured
 Yann> defconfig files for bootloaders and kernel. Eg. for barebox:
 Yann>   $ make -C ../br.upstream O=$(pwd) source-check
 Yann>   [--SNIP-]
 Yann>   make O=/home/ymorin/dev/buildroot/O _source-check
 Yann>   make[1]: Entering directory `/home/ymorin/dev/buildroot/br.upstream'
 Yann>   boot/barebox/barebox.mk:60: *** No Barebox defconfig file. Check your
 Yann>   BR2_TARGET_BAREBOX_BOARD_DEFCONFIG setting.  Stop.
 Yann>   make[1]: Leaving directory `/home/ymorin/dev/buildroot/br.upstream'
 Yann>   make: *** [source-check] Error 2
 Yann>   make: Leaving directory `/home/ymorin/dev/buildroot/br.upstream'

Yes, we currently require a buildable .config for make
source/source-check/external-deps, which I don't really think is big
issue for normal use cases. I do hit it when I update the
sources.buildroot.net mirror using make allyesconfig && make source, so
we could try to get rid of that restriction if it doesn't make the
linux/barebox/uboot Makefiles too ugly.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/2] source: print messages when all sources downloaded

[Yann E. MORIN]

Peter, All,

On Wednesday 13 July 2011 20:42:05 Peter Korsgaard wrote:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@anciens.enib.fr> writes:
>  Yann> Indeed. We could print the message only if DL_MODE==DOWNLOAD. Would that
>  Yann> be OK with you?
> 
> We could, but I'm not really sure if we should. We don't do anything
> like that for any of the other make targets.

OK, I'll ditch this change, then...

>  Yann> Incidentally, external-deps and source-check will not be really complete
>  Yann> when crostool-NG is the backend, as there is no way to ask such a thing
>  Yann> to crosstool-NG.
> 
> Ok, also not by playing games with a replacement wget/curl/whatever that
> just prints basename $1 or similar?

Nope, there's curently no way.

I could add that, but I'm not sure how much it would make sense in the
crosstool-NG case by itself.

Also, if buildroot wanted crosstool-NG to print the list of its required
sources, it would mean building crosstool-NG, which first requires building
host-gawk and host-automake anyway... Which I find ugly...

>  Yann> Note that source-check is currently borked due to some unconfigured
>  Yann> defconfig files for bootloaders and kernel. Eg. for barebox:
[--SNIP--]
> Yes, we currently require a buildable .config for make
> source/source-check/external-deps, which I don't really think is big
> issue for normal use cases. I do hit it when I update the
> sources.buildroot.net mirror using make allyesconfig && make source, so
> we could try to get rid of that restriction if it doesn't make the
> linux/barebox/uboot Makefiles too ugly.

What I'm doing here is to have the _DEFCONFIG options to default to
'(none)' so it satisfies the checks.

It is the responsibility of the user to set correct values in place of the
defaults. But that does not really change from what's required today, the
user is already responsible for that.

Of course, it'll be up to you to accept or ditch such a change, if you don't
mind setting those manually. After all, it's only three options. For now...
:-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'