[Buildroot] [PATCH/autobuild 1/8] autobuild-run: fix typo in help text

[Buildroot] [PATCH/autobuild 1/8] autobuild-run: fix typo in help text

[Arnout Vandecappelle]

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 scripts/autobuild-run | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 37c2b12..64da088 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -86,7 +86,7 @@ Options:
   -s, --submitter SUBMITTER      name/machine of submitter
                                  Defaults to %(--submitter)s.
   --http-url URL                 URL of resource to submit your results.
-                                 Defaults to $(--http-url)s.
+                                 Defaults to %(--http-url)s.
   --http-login LOGIN             username to send results with
                                  Not set by default.
   --http-password PASSWORD       password to send results with (for security
-- 
2.11.0

[Buildroot] [PATCH/autobuild 2/8] autobuild-run: add '-d, --debug' option

[Arnout Vandecappelle]

While changing and debugging the autobuild-run script, it is convenient
to be able to see the log directly on stdout rather than opening the
log file. So provide an option for this.

The option is called '--debug' with the idea that other debugging
features added later can reuse the same option.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 scripts/autobuild-run | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 64da088..2f0463d 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -101,6 +101,7 @@ Options:
                                  Defaults to %(--pid-file)s.
   -c, --config CONFIG            path to configuration file
                                  Not set by default.
+  -d, --debug                    Send log output to stdout instead of log file
   --toolchains-url URL           URL of toolchain configuration file
 
 Format of the configuration file:
@@ -800,7 +801,10 @@ def run_instance(**kwargs):
     if not os.path.exists(idir):
         os.mkdir(idir)
 
-    kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+")
+    if kwargs['debug']:
+        kwargs['log'] = sys.stdout
+    else:
+        kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+")
     log_write(kwargs['log'], "INFO: instance started")
 
     while True:
@@ -926,7 +930,8 @@ def main():
                 nice = (args['--nice'] or 0),
                 toolchains_url = args['--toolchains-url'],
                 upload = upload,
-                buildpid = buildpid
+                buildpid = buildpid,
+                debug = args['--debug']
             ))
         p.start()
         processes.append(p)
-- 
2.11.0

[Buildroot] [PATCH/autobuild 2/8] autobuild-run: add '-d, --debug' option

[Thomas Petazzoni]

Hello,

On Sun, 9 Apr 2017 22:51:22 +0200, Arnout Vandecappelle
(Essensium/Mind) wrote:

> -    kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+")
> +    if kwargs['debug']:
> +        kwargs['log'] = sys.stdout
> +    else:
> +        kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+")
>      log_write(kwargs['log'], "INFO: instance started")

How does this works when you have several instances running in
parallel?

Everything gets mixed up on stdout?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH/autobuild 2/8] autobuild-run: add '-d, --debug' option

[Arnout Vandecappelle]

On 10-04-17 10:22, Thomas Petazzoni wrote:
> Hello,
> 
> On Sun, 9 Apr 2017 22:51:22 +0200, Arnout Vandecappelle
> (Essensium/Mind) wrote:
> 
>> -    kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+")
>> +    if kwargs['debug']:
>> +        kwargs['log'] = sys.stdout
>> +    else:
>> +        kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+")
>>      log_write(kwargs['log'], "INFO: instance started")
> 
> How does this works when you have several instances running in
> parallel?
> 
> Everything gets mixed up on stdout?

 Indeed. I assume if you're debugging, you're not going to run several instances
in parallel.

 Anyway, if you don't want this feature, just drop it. It's something I used
while debugging the script and I thought could be useful for others as well.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH/autobuild 3/8] autobuild-run: use 'olddefconfig' instead of 'oldconfig'

[Arnout Vandecappelle]

Back in the days that autobuild-run was created, we didn't have an
olddefconfig target so we had to pipe yes into oldconfig instead.
Nowadays, however, we can use the olddefconfig target.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 scripts/autobuild-run | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 2f0463d..42276db 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -558,8 +558,8 @@ def gen_config(**kwargs):
 
     devnull = open(os.devnull, "w")
 
-    ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \
-                           (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull)
+    ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "olddefconfig"],
+                          stdout=devnull, stderr=devnull)
     if ret != 0:
         log_write(log, "ERROR: cannot oldconfig")
         return -1
-- 
2.11.0

[Buildroot] [PATCH/autobuild 4/8] autobuild-run: remove redundant 'make oldconfig'

[Arnout Vandecappelle]

After running 'make randpackageconfig', and additional 'make oldconfig'
is done. However, this is redundant, since randpackageconfig already
does an olddefconfig.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 scripts/autobuild-run | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 42276db..1df1ce0 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -586,12 +586,6 @@ def gen_config(**kwargs):
         if fixup_config(**kwargs):
             break
 
-    ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \
-                           (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull)
-    if ret != 0:
-        log_write(log, "ERROR: cannot oldconfig")
-        return -1
-
     ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"],
                           stdout=devnull, stderr=devnull)
     if ret != 0:
-- 
2.11.0

[Buildroot] [PATCH/autobuild 4/8] autobuild-run: remove redundant 'make oldconfig'

[Thomas Petazzoni]

Hello,

On Sun, 9 Apr 2017 22:51:24 +0200, Arnout Vandecappelle
(Essensium/Mind) wrote:
> After running 'make randpackageconfig', and additional 'make oldconfig'
> is done. However, this is redundant, since randpackageconfig already
> does an olddefconfig.
> 
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Well, this oldconfig is not exactly right after randpackageconfig.

We do this:

 - randpackageconfig
 - fixup_config()
 - oldconfig

The fixup_config() function potentially does some changes to
the .config file, and that's why we re-run oldconfig afterwards. This
ensures that the build will not wait indefinitely while asking the user
the value of a config option.

Therefore, I wouldn't qualify this oldconfig as "redundant".

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH/autobuild 4/8] autobuild-run: remove redundant 'make oldconfig'

[Arnout Vandecappelle]

On 10-04-17 10:25, Thomas Petazzoni wrote:
> Hello,
> 
> On Sun, 9 Apr 2017 22:51:24 +0200, Arnout Vandecappelle
> (Essensium/Mind) wrote:
>> After running 'make randpackageconfig', and additional 'make oldconfig'
>> is done. However, this is redundant, since randpackageconfig already
>> does an olddefconfig.
>>
>> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> 
> Well, this oldconfig is not exactly right after randpackageconfig.
> 
> We do this:
> 
>  - randpackageconfig
>  - fixup_config()
>  - oldconfig
> 
> The fixup_config() function potentially does some changes to
> the .config file, and that's why we re-run oldconfig afterwards. This
> ensures that the build will not wait indefinitely while asking the user
> the value of a config option.

 D'oh, I missed that. Fortunately somebody reviews my crap! :-)

 So instead I'll replace it with olddefconfig and squash with the preceding
patch, OK?

 Regards,
 Arnout

> 
> Therefore, I wouldn't qualify this oldconfig as "redundant".
> 
> Best regards,
> 
> Thomas
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH/autobuild 4/8] autobuild-run: remove redundant 'make oldconfig'

[Thomas Petazzoni]

Hello,

On Mon, 10 Apr 2017 10:46:54 +0200, Arnout Vandecappelle wrote:

>  So instead I'll replace it with olddefconfig and squash with the preceding
> patch, OK?

Yup, seen your v2, looks good to me.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH/autobuild 5/8] autobuild-run: check that toolchain config lines are still present

[Arnout Vandecappelle]

Some lines from the toolchain config may be removed due to dependency
issues. Currently this is covered by explicit conditions in the
autobuild-run script, e.g. checking that libc is not glibc before
enabling BR2_STATIC_LIBS. However, that binds this script pretty
tightly to the logic in Buildroot itself.

So instead, just check that the toolchain configuration is still valid
after running 'olddefconfig', and discard the configuration if it
isn't.  This is similar to how it's done in the test-pkg script.

We also report all the missing lines in the log file.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 scripts/autobuild-run | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 1df1ce0..50808b6 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -366,6 +366,16 @@ def is_toolchain_usable(**kwargs):
     with open(os.path.join(outputdir, ".config")) as configf:
         configlines = configf.readlines()
 
+    # Check that the toolchain configuration is still present
+    # Report all the missing ones
+    toolchaincomplete = True
+    for toolchainline in kwargs['config']:
+        if toolchainline not in configlines:
+            log_write(log, "WARN: missing toolchain config line: %s" % toolchainline[:-1])
+            toolchaincomplete = False
+    if not toolchaincomplete:
+        return False
+
     # The latest Linaro toolchains on x86-64 hosts requires glibc
     # 2.14+ on the host.
     if platform.machine() == 'x86_64':
@@ -564,7 +574,7 @@ def gen_config(**kwargs):
         log_write(log, "ERROR: cannot oldconfig")
         return -1
 
-    if not is_toolchain_usable(**kwargs):
+    if not is_toolchain_usable(config=config["contents"], **kwargs):
         return -1
 
     # Now, generate the random selection of packages, and fixup
-- 
2.11.0

[Buildroot] [PATCH/autobuild 5/8] autobuild-run: check that toolchain config lines are still present

[Thomas Petazzoni]

Hello,

On Sun, 9 Apr 2017 22:51:25 +0200, Arnout Vandecappelle
(Essensium/Mind) wrote:
> Some lines from the toolchain config may be removed due to dependency
> issues. Currently this is covered by explicit conditions in the
> autobuild-run script, e.g. checking that libc is not glibc before
> enabling BR2_STATIC_LIBS. However, that binds this script pretty
> tightly to the logic in Buildroot itself.

Seems like a good idea!

> +    # Check that the toolchain configuration is still present
> +    # Report all the missing ones
> +    toolchaincomplete = True
> +    for toolchainline in kwargs['config']:
> +        if toolchainline not in configlines:
> +            log_write(log, "WARN: missing toolchain config line: %s" % toolchainline[:-1])

We should remove this warning, it's not a warning at all. It is
a perfectly normal situation that a configuration gets rejected here,
for example when glibc+static is used. So having a warning everytime a
completely regular situation occurs doesn't seem like a good idea.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH/autobuild 5/8] autobuild-run: check that toolchain config lines are still present

[Arnout Vandecappelle]

On 10-04-17 10:28, Thomas Petazzoni wrote:
> Hello,
> 
> On Sun, 9 Apr 2017 22:51:25 +0200, Arnout Vandecappelle
> (Essensium/Mind) wrote:
>> Some lines from the toolchain config may be removed due to dependency
>> issues. Currently this is covered by explicit conditions in the
>> autobuild-run script, e.g. checking that libc is not glibc before
>> enabling BR2_STATIC_LIBS. However, that binds this script pretty
>> tightly to the logic in Buildroot itself.
> 
> Seems like a good idea!
> 
>> +    # Check that the toolchain configuration is still present
>> +    # Report all the missing ones
>> +    toolchaincomplete = True
>> +    for toolchainline in kwargs['config']:
>> +        if toolchainline not in configlines:
>> +            log_write(log, "WARN: missing toolchain config line: %s" % toolchainline[:-1])
> 
> We should remove this warning, it's not a warning at all. It is
> a perfectly normal situation that a configuration gets rejected here,
> for example when glibc+static is used. So having a warning everytime a
> completely regular situation occurs doesn't seem like a good idea.

 Yes of course. I needed that for debugging but it indeed shouldn't be there in
production.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH/autobuild 6/8] autobuild-run: remove check for glibc

[Arnout Vandecappelle]

Now that we check the validity of the toolchain config after running
'make olddefconfig', it is no longer necessary to have an explicit
check of glibc before enabling BR2_STATIC_LIBS.

This will be necessary to be able to use the in-tree toolchain configs.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 scripts/autobuild-run | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 50808b6..2aa89b4 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -559,7 +559,7 @@ def gen_config(**kwargs):
         configlines.append("BR2_INIT_SYSTEMD=y\n")
     elif randint(0, 20) == 0:
         configlines.append("BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV=y\n")
-    if config["libc"] != "glibc" and randint(0, 20) == 0:
+    if randint(0, 20) == 0:
         configlines.append("BR2_STATIC_LIBS=y\n")
 
     # Write out the configuration file
-- 
2.11.0

[Buildroot] [PATCH/autobuild 7/8] autobuild-run: simplify, don't make a dict of config

[Arnout Vandecappelle]

A toolchain config was defined as a dict with keys url, hostarch,
libc and contents. Of these, only contents and libc were actually
used outside of get_toolchain_configs(). And the use of libc has
been removed in the previous patch.

So there is no reason anymore to make config a dict. Just replace
it with its contents.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 scripts/autobuild-run | 26 +++++++++++---------------
 1 file changed, 11 insertions(+), 15 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 2aa89b4..d17e089 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -252,11 +252,8 @@ class SystemInfo:
 def get_toolchain_configs(**kwargs):
     """Fetch and return the possible toolchain configurations
 
-    This function returns an array of dictionaries, with for each toolchain:
-        - url: the URL of the toolchain defconfig
-        - libc: the C library used by the toolchain
-        - hostarch: the host architecture for which the toolchain is built
-        - contents: an array of lines of the defconfig
+    This function returns an array of toolchain configurations. Each
+    toolchain configuration is itself an array of lines of the defconfig.
     """
     toolchains_url = kwargs['toolchains_url']
 
@@ -271,31 +268,30 @@ def get_toolchain_configs(**kwargs):
 
     for row in csv.reader(l):
         config = {}
-        config["url"] = row[0]
-        config["hostarch"] = row[1]
+        url = row[0]
+        config_hostarch = row[1]
         keep = False
 
         # Keep all toolchain configs that work regardless of the host
         # architecture
-        if config['hostarch'] == "any":
+        if config_hostarch == "any":
             keep = True
 
         # Keep all toolchain configs that can work on the current host
         # architecture
-        if hostarch == config["hostarch"]:
+        if hostarch == config_hostarch:
             keep = True
 
         # Assume that x86 32 bits toolchains work on x86_64 build
         # machines
-        if hostarch == 'x86_64' and config["hostarch"] == "x86":
+        if hostarch == 'x86_64' and config_hostarch == "x86":
             keep = True
 
         if not keep:
             continue
 
-        config["libc"] = row[2]
-        with urlopen_closing(config["url"]) as r:
-            config["contents"] = decode_byte_list(r.readlines())
+        with urlopen_closing(url) as r:
+            config = decode_byte_list(r.readlines())
         configs.append(config)
     return configs
 
@@ -547,7 +543,7 @@ def gen_config(**kwargs):
     i = randint(0, len(configs) - 1)
     config = configs[i]
 
-    configlines = config["contents"]
+    configlines = config
 
     # Amend the configuration with a few things.
     configlines.append("BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y\n")
@@ -574,7 +570,7 @@ def gen_config(**kwargs):
         log_write(log, "ERROR: cannot oldconfig")
         return -1
 
-    if not is_toolchain_usable(config=config["contents"], **kwargs):
+    if not is_toolchain_usable(config=config, **kwargs):
         return -1
 
     # Now, generate the random selection of packages, and fixup
-- 
2.11.0

[Buildroot] [PATCH/autobuild 8/8] autobuild-run: use in-tree toolchain configs

[Arnout Vandecappelle]

Instead of having the list of toolchain configs as a CSV file, they
are now maintained within the buildroot tree itself. Therefore,
autobuild-run must get the toolchain configs from there.

The toolchains-url option is replaced with a toolchains-path option.
This path is relative to the buildroot directory and defaults to
support/config-fragments/autobuild. It is possible to override this
with a relative or absolute path to the directory containing the
toolchain configs.

The toolchain configs no longer contain the hostarch information.  For
the predefined external toolchains, this is handled by checking after
the olddefconfig: because of "depends on HOSTARCH=...", the toolchain
config will have been removed. For the custom external toolchains, we
instead rely on the URL: it ends with -HOSTARCH.tar.gz.

Since we don't fetch from a URL anymore, the exception handling
introduced in 5f1016ab0 is no longer needed.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
Clearly, at the moment, the custom external toolchain tarball names
*don't* end with -HOSTARCH, so the hostarch check doesn't actually
work. However, I think this is the most elegent solution. Other
alternatives are:
- Adding a dummy config option BR2_HOSTARCH=... to the toolchain
  config fragments. This works pretty nicely, because then it will
  be covered automatically by the check that all the toolchain config
  lines are still there. However, this requires manual handling when
  generating the toolchain configs (they are no longer the result of
  "make savedefconfig"). Also, it is not enough for the x86 case
  (i.e. the ctng toolchains): these also work on x86_64 autobuilders,
  so special handling for that case would be needed.
- As it happens, all br-* toolchains are x86_64, and all ctng
  toolchains are x86, so we could just do string matching on those to
  derive the hostarch. However, that would be too much of a hack for
  my comfort.

Any other ideas are welcome!

If this solution is chosen, I propose to replace
http://patchwork.ozlabs.org/patch/748187/ with the renamed tarballs
immediately while applying.
---
 scripts/autobuild-run | 62 +++++++++++++++------------------------------------
 1 file changed, 18 insertions(+), 44 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index d17e089..f920251 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -67,7 +67,7 @@ defaults = {
     '--nice': 0,
     '--pid-file': '/tmp/buildroot-autobuild.pid',
     '--http-url': 'http://autobuild.buildroot.org/submit/',
-    '--toolchains-url': 'http://autobuild.buildroot.org/toolchains/configs/toolchain-configs.csv',
+    '--toolchains-path': 'support/config-fragments/autobuild',
 }
 
 doc = """autobuild-run - run Buildroot autobuilder
@@ -102,7 +102,8 @@ Options:
   -c, --config CONFIG            path to configuration file
                                  Not set by default.
   -d, --debug                    Send log output to stdout instead of log file
-  --toolchains-url URL           URL of toolchain configuration file
+  --toolchains-path URL          Path to the toolchain configuration files
+                                 (relative to buildroot dir).
 
 Format of the configuration file:
 
@@ -169,7 +170,7 @@ else:
     encode_str = _identity
 
 MAX_DURATION = 60 * 60 * 8
-VERSION = 1
+VERSION = 2
 
 def log_write(logf, msg):
     logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg))
@@ -255,44 +256,20 @@ def get_toolchain_configs(**kwargs):
     This function returns an array of toolchain configurations. Each
     toolchain configuration is itself an array of lines of the defconfig.
     """
-    toolchains_url = kwargs['toolchains_url']
+    toolchains_path = kwargs['toolchains_path']
 
-    with urlopen_closing(toolchains_url) as r:
-        l = decode_byte_list(r.readlines())
-    configs = []
-
-    (_, _, _, _, hostarch) = os.uname()
-    # ~2015 distros report x86 when on a 32bit install
-    if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
-        hostarch = 'x86'
-
-    for row in csv.reader(l):
-        config = {}
-        url = row[0]
-        config_hostarch = row[1]
-        keep = False
-
-        # Keep all toolchain configs that work regardless of the host
-        # architecture
-        if config_hostarch == "any":
-            keep = True
-
-        # Keep all toolchain configs that can work on the current host
-        # architecture
-        if hostarch == config_hostarch:
-            keep = True
-
-        # Assume that x86 32 bits toolchains work on x86_64 build
-        # machines
-        if hostarch == 'x86_64' and config_hostarch == "x86":
-            keep = True
-
-        if not keep:
-            continue
+    if os.path.isabs(toolchains_path):
+        toolchains_dir = toolchains_path
+    else:
+        idir = "instance-%d" % kwargs['instance']
+        toolchains_dir = os.path.join(idir, "buildroot", toolchains_path)
 
-        with urlopen_closing(url) as r:
-            config = decode_byte_list(r.readlines())
-        configs.append(config)
+    configs = []
+    for configfile in os.listdir(toolchains_dir):
+        if configfile.endswith('.config'):
+            with open(os.path.join(toolchains_dir, configfile)) as r:
+                config = decode_byte_list(r.readlines())
+            configs.append(config)
     return configs
 
 def prepare_build(**kwargs):
@@ -535,10 +512,7 @@ def gen_config(**kwargs):
     log_write(log, "INFO: generate the configuration")
 
     # Select a random toolchain configuration
-    try:
-        configs = get_toolchain_configs(**kwargs)
-    except:
-        return -1
+    configs = get_toolchain_configs(**kwargs)
 
     i = randint(0, len(configs) - 1)
     config = configs[i]
@@ -928,7 +902,7 @@ def main():
                 submitter = args['--submitter'],
                 make_opts = (args['--make-opts'] or ''),
                 nice = (args['--nice'] or 0),
-                toolchains_url = args['--toolchains-url'],
+                toolchains_path = args['--toolchains-path'],
                 upload = upload,
                 buildpid = buildpid,
                 debug = args['--debug']
-- 
2.11.0

[Buildroot] [PATCH/autobuild 8/8] autobuild-run: use in-tree toolchain configs

[Arnout Vandecappelle]

 Ho, turns out I sent this out before amending, so this patch is NOT correct...

 I'll quickly send out an update for proper review, but it'll probably need a v3
as well.

 Regards,
 Arnout

On 09-04-17 22:51, Arnout Vandecappelle (Essensium/Mind) wrote:
> Instead of having the list of toolchain configs as a CSV file, they
> are now maintained within the buildroot tree itself. Therefore,
> autobuild-run must get the toolchain configs from there.
> 
> The toolchains-url option is replaced with a toolchains-path option.
> This path is relative to the buildroot directory and defaults to
> support/config-fragments/autobuild. It is possible to override this
> with a relative or absolute path to the directory containing the
> toolchain configs.
> 
> The toolchain configs no longer contain the hostarch information.  For
> the predefined external toolchains, this is handled by checking after
> the olddefconfig: because of "depends on HOSTARCH=...", the toolchain
> config will have been removed. For the custom external toolchains, we
> instead rely on the URL: it ends with -HOSTARCH.tar.gz.
> 
> Since we don't fetch from a URL anymore, the exception handling
> introduced in 5f1016ab0 is no longer needed.
> 
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
> Clearly, at the moment, the custom external toolchain tarball names
> *don't* end with -HOSTARCH, so the hostarch check doesn't actually
> work. However, I think this is the most elegent solution. Other
> alternatives are:
> - Adding a dummy config option BR2_HOSTARCH=... to the toolchain
>   config fragments. This works pretty nicely, because then it will
>   be covered automatically by the check that all the toolchain config
>   lines are still there. However, this requires manual handling when
>   generating the toolchain configs (they are no longer the result of
>   "make savedefconfig"). Also, it is not enough for the x86 case
>   (i.e. the ctng toolchains): these also work on x86_64 autobuilders,
>   so special handling for that case would be needed.
> - As it happens, all br-* toolchains are x86_64, and all ctng
>   toolchains are x86, so we could just do string matching on those to
>   derive the hostarch. However, that would be too much of a hack for
>   my comfort.
> 
> Any other ideas are welcome!
> 
> If this solution is chosen, I propose to replace
> http://patchwork.ozlabs.org/patch/748187/ with the renamed tarballs
> immediately while applying.
> ---
>  scripts/autobuild-run | 62 +++++++++++++++------------------------------------
>  1 file changed, 18 insertions(+), 44 deletions(-)
> 
> diff --git a/scripts/autobuild-run b/scripts/autobuild-run
> index d17e089..f920251 100755
> --- a/scripts/autobuild-run
> +++ b/scripts/autobuild-run
> @@ -67,7 +67,7 @@ defaults = {
>      '--nice': 0,
>      '--pid-file': '/tmp/buildroot-autobuild.pid',
>      '--http-url': 'http://autobuild.buildroot.org/submit/',
> -    '--toolchains-url': 'http://autobuild.buildroot.org/toolchains/configs/toolchain-configs.csv',
> +    '--toolchains-path': 'support/config-fragments/autobuild',
>  }
>  
>  doc = """autobuild-run - run Buildroot autobuilder
> @@ -102,7 +102,8 @@ Options:
>    -c, --config CONFIG            path to configuration file
>                                   Not set by default.
>    -d, --debug                    Send log output to stdout instead of log file
> -  --toolchains-url URL           URL of toolchain configuration file
> +  --toolchains-path URL          Path to the toolchain configuration files
> +                                 (relative to buildroot dir).
>  
>  Format of the configuration file:
>  
> @@ -169,7 +170,7 @@ else:
>      encode_str = _identity
>  
>  MAX_DURATION = 60 * 60 * 8
> -VERSION = 1
> +VERSION = 2
>  
>  def log_write(logf, msg):
>      logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg))
> @@ -255,44 +256,20 @@ def get_toolchain_configs(**kwargs):
>      This function returns an array of toolchain configurations. Each
>      toolchain configuration is itself an array of lines of the defconfig.
>      """
> -    toolchains_url = kwargs['toolchains_url']
> +    toolchains_path = kwargs['toolchains_path']
>  
> -    with urlopen_closing(toolchains_url) as r:
> -        l = decode_byte_list(r.readlines())
> -    configs = []
> -
> -    (_, _, _, _, hostarch) = os.uname()
> -    # ~2015 distros report x86 when on a 32bit install
> -    if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
> -        hostarch = 'x86'
> -
> -    for row in csv.reader(l):
> -        config = {}
> -        url = row[0]
> -        config_hostarch = row[1]
> -        keep = False
> -
> -        # Keep all toolchain configs that work regardless of the host
> -        # architecture
> -        if config_hostarch == "any":
> -            keep = True
> -
> -        # Keep all toolchain configs that can work on the current host
> -        # architecture
> -        if hostarch == config_hostarch:
> -            keep = True
> -
> -        # Assume that x86 32 bits toolchains work on x86_64 build
> -        # machines
> -        if hostarch == 'x86_64' and config_hostarch == "x86":
> -            keep = True
> -
> -        if not keep:
> -            continue
> +    if os.path.isabs(toolchains_path):
> +        toolchains_dir = toolchains_path
> +    else:
> +        idir = "instance-%d" % kwargs['instance']
> +        toolchains_dir = os.path.join(idir, "buildroot", toolchains_path)
>  
> -        with urlopen_closing(url) as r:
> -            config = decode_byte_list(r.readlines())
> -        configs.append(config)
> +    configs = []
> +    for configfile in os.listdir(toolchains_dir):
> +        if configfile.endswith('.config'):
> +            with open(os.path.join(toolchains_dir, configfile)) as r:
> +                config = decode_byte_list(r.readlines())
> +            configs.append(config)
>      return configs
>  
>  def prepare_build(**kwargs):
> @@ -535,10 +512,7 @@ def gen_config(**kwargs):
>      log_write(log, "INFO: generate the configuration")
>  
>      # Select a random toolchain configuration
> -    try:
> -        configs = get_toolchain_configs(**kwargs)
> -    except:
> -        return -1
> +    configs = get_toolchain_configs(**kwargs)
>  
>      i = randint(0, len(configs) - 1)
>      config = configs[i]
> @@ -928,7 +902,7 @@ def main():
>                  submitter = args['--submitter'],
>                  make_opts = (args['--make-opts'] or ''),
>                  nice = (args['--nice'] or 0),
> -                toolchains_url = args['--toolchains-url'],
> +                toolchains_path = args['--toolchains-path'],
>                  upload = upload,
>                  buildpid = buildpid,
>                  debug = args['--debug']
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] autobuild-run: use in-tree toolchain configs

[Arnout Vandecappelle]

Instead of having the list of toolchain configs as a CSV file, they
are now maintained within the buildroot tree itself. Therefore,
autobuild-run must get the toolchain configs from there.

The toolchains-url option is replaced with a toolchains-path option.
This path is relative to the buildroot directory and defaults to
support/config-fragments/autobuild. It is possible to override this
with a relative or absolute path to the directory containing the
toolchain configs.

The toolchain configs no longer contain the hostarch information.  For
the predefined external toolchains, this is handled by checking after
the olddefconfig: because of "depends on HOSTARCH=...", the toolchain
config will have been removed. For the custom external toolchains, we
instead rely on the URL: it ends with -HOSTARCH.tar.gz.

Since we don't fetch from a URL anymore, the exception handling
introduced in 5f1016ab0 is no longer needed.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
Clearly, at the moment, the custom external toolchain tarball names
*don't* end with -HOSTARCH, so the hostarch check doesn't actually
work. However, I think this is the most elegent solution. Other
alternatives are:
- Adding a dummy config option BR2_HOSTARCH=... to the toolchain
  config fragments. This works pretty nicely, because then it will
  be covered automatically by the check that all the toolchain config
  lines are still there. However, this requires manual handling when
  generating the toolchain configs (they are no longer the result of
  "make savedefconfig"). Also, it is not enough for the x86 case
  (i.e. the ctng toolchains): these also work on x86_64 autobuilders,
  so special handling for that case would be needed.
- As it happens, all br-* toolchains are x86_64, and all ctng
  toolchains are x86, so we could just do string matching on those to
  derive the hostarch. However, that would be too much of a hack for
  my comfort.

Any other ideas are welcome!

If this solution is chosen, I propose to replace
http://patchwork.ozlabs.org/patch/748187/ with the renamed tarballs
immediately while applying.
---
 scripts/autobuild-run | 69 ++++++++++++++++++++++++---------------------------
 1 file changed, 32 insertions(+), 37 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index d17e089..7012902 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -67,7 +67,7 @@ defaults = {
     '--nice': 0,
     '--pid-file': '/tmp/buildroot-autobuild.pid',
     '--http-url': 'http://autobuild.buildroot.org/submit/',
-    '--toolchains-url': 'http://autobuild.buildroot.org/toolchains/configs/toolchain-configs.csv',
+    '--toolchains-path': 'support/config-fragments/autobuild',
 }
 
 doc = """autobuild-run - run Buildroot autobuilder
@@ -102,7 +102,8 @@ Options:
   -c, --config CONFIG            path to configuration file
                                  Not set by default.
   -d, --debug                    Send log output to stdout instead of log file
-  --toolchains-url URL           URL of toolchain configuration file
+  --toolchains-path URL          Path to the toolchain configuration files
+                                 (relative to buildroot dir).
 
 Format of the configuration file:
 
@@ -169,7 +170,7 @@ else:
     encode_str = _identity
 
 MAX_DURATION = 60 * 60 * 8
-VERSION = 1
+VERSION = 2
 
 def log_write(logf, msg):
     logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg))
@@ -255,10 +256,14 @@ def get_toolchain_configs(**kwargs):
     This function returns an array of toolchain configurations. Each
     toolchain configuration is itself an array of lines of the defconfig.
     """
-    toolchains_url = kwargs['toolchains_url']
+    toolchains_path = kwargs['toolchains_path']
+
+    if os.path.isabs(toolchains_path):
+        toolchains_dir = toolchains_path
+    else:
+        idir = "instance-%d" % kwargs['instance']
+        toolchains_dir = os.path.join(idir, "buildroot", toolchains_path)
 
-    with urlopen_closing(toolchains_url) as r:
-        l = decode_byte_list(r.readlines())
     configs = []
 
     (_, _, _, _, hostarch) = os.uname()
@@ -266,33 +271,26 @@ def get_toolchain_configs(**kwargs):
     if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
         hostarch = 'x86'
 
-    for row in csv.reader(l):
-        config = {}
-        url = row[0]
-        config_hostarch = row[1]
-        keep = False
-
-        # Keep all toolchain configs that work regardless of the host
-        # architecture
-        if config_hostarch == "any":
+    for configfile in os.listdir(toolchains_dir):
+        if configfile.endswith('.config'):
             keep = True
-
-        # Keep all toolchain configs that can work on the current host
-        # architecture
-        if hostarch == config_hostarch:
-            keep = True
-
-        # Assume that x86 32 bits toolchains work on x86_64 build
-        # machines
-        if hostarch == 'x86_64' and config_hostarch == "x86":
-            keep = True
-
-        if not keep:
-            continue
-
-        with urlopen_closing(url) as r:
-            config = decode_byte_list(r.readlines())
-        configs.append(config)
+            with open(os.path.join(toolchains_dir, configfile)) as r:
+                config = decode_byte_list(r.readlines())
+                for configline in config:
+                    # If the toolchain is for a specific arch, check it. The prebuilt toolchains
+                    # are only for x86 or x86_64. If the URL doesn't specify an arch, assume it
+                    # is OK.
+                    m = re.match(r'^BR2_TOOLCHAIN_EXTERNAL_URL=".*(x86|x86_64)\.tar', configline)
+                    if m:
+                        configarch = m.group(1)
+                        if configarch != hostarch:
+                            keep = False
+                        # Assume that x86 32 bits toolchains work on x86_64
+                        # build machines
+                        if configarch == 'x86' and hostarch == 'x86_64':
+                            keep = True
+            if keep:
+                configs.append(config)
     return configs
 
 def prepare_build(**kwargs):
@@ -535,10 +533,7 @@ def gen_config(**kwargs):
     log_write(log, "INFO: generate the configuration")
 
     # Select a random toolchain configuration
-    try:
-        configs = get_toolchain_configs(**kwargs)
-    except:
-        return -1
+    configs = get_toolchain_configs(**kwargs)
 
     i = randint(0, len(configs) - 1)
     config = configs[i]
@@ -928,7 +923,7 @@ def main():
                 submitter = args['--submitter'],
                 make_opts = (args['--make-opts'] or ''),
                 nice = (args['--nice'] or 0),
-                toolchains_url = args['--toolchains-url'],
+                toolchains_path = args['--toolchains-path'],
                 upload = upload,
                 buildpid = buildpid,
                 debug = args['--debug']
-- 
2.11.0