[yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[gmane]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: 0001-deal-only-with-built-toolchains-cp-also-md5-and-mani.patch --]
[-- Type: text/x-diff; name=0001-deal-only-with-built-toolchains-cp-also-md5-and-mani.patch, Size: 2971 bytes --]

From 8f28afbfc21dc84f076b0cc620e578f9607d1465 Mon Sep 17 00:00:00 2001
From: Robert Berger <robert.berger@ReliableEmbeddedSystems.com> 
Date: Mon, 10 Oct 2016 06:26:06 +0000
Subject: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also
 md5 and manifests

Signed-off-by: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>
---
 .../autobuilder/buildsteps/PublishArtifacts.py             | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py b/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
index 633600c..46d7b40 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
@@ -154,14 +154,16 @@ class PublishArtifacts(ShellCommand):
                 elif artifact == "toolchain":
                     artifact_name, deploy_image_dir = self.getDeployNames(artifact, buildername)
                     command=command+self.generateMD5cmd(artifact, deploy_image_dir)
-                    command=command+"mkdir -p " + os.path.join(DEST, X86TC_PUBLISH_DIR) + ";"
+                    command=command+"if [ -e " + os.path.join(deploy_image_dir, "poky-*i686-core-image*.sh ") + " ]; then " + \
+                                    "mkdir -p " + os.path.join(DEST, X86TC_PUBLISH_DIR) + "; "
                     command=command+"cp -R --no-dereference --preserve=links " + \
-                                    os.path.join(deploy_image_dir, "poky-*i686-core-image*.sh ") + \
-                                    os.path.join(DEST, X86TC_PUBLISH_DIR) + pipeline + ";"
-                    command=command+"mkdir -p " + os.path.join(DEST, X8664TC_PUBLISH_DIR) + ";"
+                                    os.path.join(deploy_image_dir, "poky-*i686-core-image* ") + \
+                                    os.path.join(DEST, X86TC_PUBLISH_DIR) + pipeline + "; fi;"
+                    command=command+"if [ -e " + os.path.join(deploy_image_dir, "poky-*x86_64-core-image*.sh ") + " ]; then " + \
+                                    "mkdir -p " + os.path.join(DEST, X8664TC_PUBLISH_DIR) + "; "
                     command=command+"cp -R --no-dereference --preserve=links " + \
-                                    os.path.join(deploy_image_dir, "poky-*x86_64-core-image*.sh ") + \
-                                    os.path.join(DEST, X8664TC_PUBLISH_DIR) + pipeline + ";"
+                                    os.path.join(deploy_image_dir, "poky-*x86_64-core-image* ") + \
+                                    os.path.join(DEST, X8664TC_PUBLISH_DIR) + pipeline + "; fi;"
                 elif artifact == "uninative":
                     artifact_name, deploy_image_dir = self.getDeployNames(artifact, buildername)
                     command=command+self.generateMD5cmd(artifact, deploy_image_dir)
-- 
2.7.4

Re: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[Bill Randle]

Hi Robert,

Then "normal" method to send patches is via direct insertion into the
email, rather than an attachment (see git-send-email). Also, it is
missing a commit comment as the first text in the patch file.
Otherwise, it looks good to me now.

    -Bill

2016-10-09 23:44 GMT-07:00  <gmane@reliableembeddedsystems.com>:
>

Re: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[Beth 'pidge' Flanagan]

On Wed, 2016-10-12 at 07:28 -0700, Bill Randle wrote:
> Hi Robert,
> 
> Then "normal" method to send patches is via direct insertion into the
> email, rather than an attachment (see git-send-email). Also, it is
> missing a commit comment as the first text in the patch file.
> Otherwise, it looks good to me now.
> 
>     -Bill

I'm going to pull this but please, in the future, follow the above.

-b



> 
> 2016-10-09 23:44 GMT-07:00  <gmane@reliableembeddedsystems.com>:
> > 
> > 

Re: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[Beth 'pidge' Flanagan]

On Mon, 2016-10-10 at 01:44 -0500, gmane@reliableembeddedsystems.com
wrote:
> 


A few notes (not picking on this patch but it does point out some
design failures and I want to at least get this on folks radar).

I've been unhappy with how DeployArtifacts work for sometime.

Long ago, when I wrote DeployArtifacts, it was "get it working, we'll
refactor later" and that refactor never happened. So I'm up for ideas.

I would like a deploy infrastructure that:

a. Is distro/image name/etc agnostic (aka, poky and image names and
paths hardcoded. ick.).
b. Is perhaps config file driven.
c. Had much less of the cruft that exists withing DeployArtifacts.

So. Thoughts on this? I'd be willing to hear suggestions on how to do
this before I go and spend some time ripping this apart post release?

-b

-- 
Beth 'pidge' Flanagan <pidge@toganlabs.com>
toganlabs.com

Re: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[Lock, Joshua G]

Can you help me understand why you needed to create this patch?

We've run into some issues recently where toolchains we expected to be
built weren't and the PublishArtifacts buildstep failing because
they're missing is useful. With this change we'll no longer get that,
right?

Thanks,

Joshua
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[Joshua Lock]

On Wed, 2016-10-12 at 17:29 +0200, Beth 'pidge' Flanagan wrote:
> On Mon, 2016-10-10 at 01:44 -0500, gmane@reliableembeddedsystems.com
> wrote:
> > 
> > 
> 
> 
> A few notes (not picking on this patch but it does point out some
> design failures and I want to at least get this on folks radar).
> 
> I've been unhappy with how DeployArtifacts work for sometime.
> 
> Long ago, when I wrote DeployArtifacts, it was "get it working, we'll
> refactor later" and that refactor never happened. So I'm up for
> ideas.
> 
> I would like a deploy infrastructure that:
> 
> a. Is distro/image name/etc agnostic (aka, poky and image names and
> paths hardcoded. ick.).
> b. Is perhaps config file driven.
> c. Had much less of the cruft that exists withing DeployArtifacts.
> 
> So. Thoughts on this? I'd be willing to hear suggestions on how to do
> this before I go and spend some time ripping this apart post release?

It's quite useful to have a list of expected output of the build so
that this step can fail if a component which should be published is
missing.

I'd been planning to take a look at rewriting this build step after the
release and my initial thoughts were towards something config file
driven.

Thanks,

Joshua

Re: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[gmane]

Hi,

On 2016-10-13 16:29, Lock, Joshua G wrote:
> Can you help me understand why you needed to create this patch?
> 
> We've run into some issues recently where toolchains we expected to be
> built weren't and the PublishArtifacts buildstep failing because
> they're missing is useful. With this change we'll no longer get that,
> right?

Yes that's right (and not intended).

I would hope that you'll be able to detect such kind of problems before 
the PublishArtifacts buildstep because there should be some error in 
previous steps. The toolchain did not build!

I made this patch because I just want to build a 64 bit toolchain as 
opposed to both (like I used to do with some older version of Y-AB) and 
don't want the PublishArtifacts step to fail just because there is no 
32-bit toolchain. There is no 32 bit toolchain on purpose.

As a bonus the patch also copies md5sums and friends over and not just 
the .sh files.

I do not want PublishArtifacts toolchain(s) to find out that something 
went wrong in previous build steps, but I would want it to copy over 
whatever was built (and not to fail if something I did not build is 
missing).

As a quick and dirty alternative I would also be happy with a solution 
based on additional PublishArtifacts steps like:

toolchain_i686 for 32 bit only   (only the 32 bit toolchain)
toolchain_x86_64 for 64 bit only (only the 64 bit toolchain)

if that helps.

Then in my config file I just use toolchain_x86_64 instead of toolchain 
in my PublishArtifacts step to indicate that I just want the 64 bit 
version.

To publish both you could use toolchain_i686 and toolchain_x86_64 in the 
PublishArtifacts step.

I find it a bit confusing to write toolchain and to expect various 64 
bit and 32 bit toolchains to be published.

BTW what happens here[1] if one toolchain does not build?

I think even without my patch you'll not be able to detect the problem, 
since BuildToolchainImages is being called 4 times so two 32 bit and two 
64 bit toolchains should be built. You'll just get an error in 
PublishArtifacts before my patch if no 32 bit or no 64 bit toolchain is 
being built. Nobody counts how many should be there.

So I would say the best would be to count how many toolchains should be 
built (count BuildToolchainImages in combination with SDKMACHINE) and 
check if that's the case in PublishArtifacts. No idea if and how this 
can be done.

> 
> Thanks,
> 
> Joshua

Regards,

Robert

[1] 
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/tree/buildset-config.controller/nightly-arm.conf

> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.

Re: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[Joshua Lock]

On Thu, 2016-10-13 at 17:38 -0500, gmane@reliableembeddedsystems.com
wrote:
> Hi,
> 
> On 2016-10-13 16:29, Lock, Joshua G wrote:
> > 
> > Can you help me understand why you needed to create this patch?
> > 
> > We've run into some issues recently where toolchains we expected to
> > be
> > built weren't and the PublishArtifacts buildstep failing because
> > they're missing is useful. With this change we'll no longer get
> > that,
> > right?
> 
> Yes that's right (and not intended).
> 
> I would hope that you'll be able to detect such kind of problems
> before 
> the PublishArtifacts buildstep because there should be some error in 
> previous steps. The toolchain did not build!


Indeed, but software is buggy. This happened very recently on the
public builders:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=10275

Toolchains *were* being built but they were being unstaged from the
deploy directory by an unfortunate bug. We had errors logged due to
failing cp the buildstep didn't fail.

> I made this patch because I just want to build a 64 bit toolchain as 
> opposed to both (like I used to do with some older version of Y-AB)
> and 
> don't want the PublishArtifacts step to fail just because there is
> no 
> 32-bit toolchain. There is no 32 bit toolchain on purpose.
> 
> As a bonus the patch also copies md5sums and friends over and not
> just 
> the .sh files.

This is a reasonable goal, but something I'd rather see addressed in
the proposed PublishArtifacts rewrite. As we're trying to release morty
right now I'd like to avoid changes to the AB behaviour until after the
release.

Regards,

Joshua

Re: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

[gmane]

Hi,

On 2016-10-14 04:27, Joshua Lock wrote:
> 
> This is a reasonable goal, but something I'd rather see addressed in
> the proposed PublishArtifacts rewrite. As we're trying to release morty
> right now I'd like to avoid changes to the AB behaviour until after the
> release.

I'm absolutely happy with this and eagerly waiting for morty.
Please keep me updated so, as my time permits, I can test the new Y-AB 
version.

> 
> Regards,
> 
> Joshua

Regards,

Robert