[review-request][PATCH 0/3][V2] Fix SDK downloads for builds

[review-request][PATCH 0/3][V2] Fix SDK downloads for builds

[Elliot Smith]

V2:

Rebased on toaster-next and fixed conflicts.

V1:

Artifacts could not be downloaded for a build as they were not being listed.
It was not possible to download them manually either, as the artifact download
code was broken for SDK artifacts produced by command line builds.

Fix how toaster.bbclass hooks onto bitbake's build tasks, so that the SDK
artifacts are picked up after do_populate_sdk runs.

Fix Toaster code for returning SDK artifacts in HTTP responses.

Changes since b53df3c (toaster-next) are in
git://git.yoctoproject.org/poky-contrib, elliot/toaster/sdk_artifacts-7603
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/sdk_artifacts-7603

NB one of the commits on this branch (fbc2024) has been submitted to oe-core:
http://lists.openembedded.org/pipermail/openembedded-core/2016-January/115366.html
This patch set contains the other commits on the branch.

Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7603

To test:

1. Ensure that your database has no build artifacts in it, either by
removing the database, clearing the relevant table with SQL via the command line
(delete from orm_buildartifact), or removing the objects via the Django shell
(orm.models.BuildArtifact.objects.all().delete()).

Note that the current behaviour, where we only associate SDK artifacts with the
first build which produces them, has not been modified. So if you have already
run a build and have files in tmp/deploy/sdk associated with it, subsequent
builds will not be associated with those files.

2. Start a new build from the command line or Toaster, invoking the populate_sdk
target:

# command line
bitbake -c populate_sdk core-image-minimal

# Toaster (NB you must use the "Local" release)
run a build like "core-image-minimal:populate_sdk"

3. View the build dashboard for the build when it completes and ensure it
contains an "Other artifacts" section, as shown in the bug report.

4. Click on one of the "Other artifact" links and ensure that the artifact
downloads correctly.

As this code slightly modifies all other artifact downloads, you could also
check downloads for the following, too:

* cooker logs
* task logs
* image files
* licence manifests

Elliot Smith (3):
  toaster: toasterui: listen for bb.event.MetadataEvent
  toaster: toasterui: log OSErrorException metadata events
  toaster: toastergui: make artifact download more robust

 bitbake/lib/bb/ui/toasterui.py                     |  5 +-
 .../toastergui/templates/unavailable_artifact.html | 13 ++-
 bitbake/lib/toaster/toastergui/views.py            | 97 +++++++++-------------
 3 files changed, 51 insertions(+), 64 deletions(-)

--
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[review-request][PATCH 1/3] toaster: toasterui: listen for bb.event.MetadataEvent

[Elliot Smith]

The event mask for toasterui doesn't include MetadataEvents.
This means that we're missing the ArtifactFileSize event
(among others), which is the one we use to populate the SDK
artifact table.

Add that event type to the toasterui event mask so we can
record SDK artifacts as they are created.

[YOCTO #7603]

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 0265df7..19be623 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -99,7 +99,8 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
               "bb.command.CommandExit", "bb.command.CommandCompleted",  "bb.cooker.CookerExit",
               "bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
               "bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
-              "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent"]
+              "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
+              "bb.event.MetadataEvent"]
 
 def main(server, eventHandler, params):
     # set to a logging.FileHandler instance when a build starts;
-- 
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[review-request][PATCH 2/3] toaster: toasterui: log OSErrorException metadata events

[Elliot Smith]

OSErrors occurring in toaster.bbclass are converted to
OSErrorException metadata events. They were then being swallowed
as unprocessed events by toasterui, which made them difficult
to spot.

Explicitly catch OSErrorException events and log them so they
are easier to spot and debug.

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 19be623..32b1889 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -366,6 +366,8 @@ def main(server, eventHandler, params):
                     buildinfohelper.store_license_manifest_path(event)
                 elif event.type == "SetBRBE":
                     buildinfohelper.brbe = buildinfohelper._get_data_from_event(event)
+                elif event.type == "OSErrorException":
+                    logger.error(event)
                 else:
                     logger.error("Unprocessed MetadataEvent %s ", str(event))
                 continue
-- 
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[review-request][PATCH 3/3] toaster: toastergui: make artifact download more robust

[Elliot Smith]

When an artifact download is requested, Toaster goes through a
convoluted series of conditions to decide which file to push
to the response. In the case of build artifact downloads for
command line builds, this caused an ugly exception, as command
line builds don't have a build request.

To simplify and catch more corner cases, remove the code which
fetches files via the build environment (we only support the local
build environment anyway). Then push all requests along a single
path, catching any missing file errors, missing object errors
or poorly-formed URLs in a single except clause which always returns
a valid response.

Also modify the text on the "unavailable artifact" page so it
says that the artifact doesn't exist, rather than it "no longer"
exists (exceptions may occur because an invalid artifact was
requested, rather than an artifact which was removed).

[YOCTO #7603]

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 .../toastergui/templates/unavailable_artifact.html | 13 ++-
 bitbake/lib/toaster/toastergui/views.py            | 97 +++++++++-------------
 2 files changed, 47 insertions(+), 63 deletions(-)

diff --git a/bitbake/lib/toaster/toastergui/templates/unavailable_artifact.html b/bitbake/lib/toaster/toastergui/templates/unavailable_artifact.html
index 0301a6c..2d3d02c 100644
--- a/bitbake/lib/toaster/toastergui/templates/unavailable_artifact.html
+++ b/bitbake/lib/toaster/toastergui/templates/unavailable_artifact.html
@@ -3,15 +3,14 @@
 {% load humanize %}
 {% load static %}
 
-{% block title %} Build artifact no longer exists - Toaster {% endblock %}
+{% block title %} Build artifact does not exist - Toaster {% endblock %}
 
 {% block pagecontent %}
-
-<div class="row-fluid air">
-    <div class="alert alert-info span8 lead">
-        <p"> The build artifact you are trying to download no longer exists.</p>
-        <p><a href="javascript:window.history.back()">Back to previous page</a></p>
+    <div class="row-fluid air">
+        <div class="alert alert-info span8 lead">
+            <p>The build artifact you are trying to download does not exist.</p>
+            <p><a href="javascript:window.history.back()">Back to previous page</a></p>
+        </div>
     </div>
-</div>
 {% endblock %}
 
diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py
index 59e16b2..995937a 100755
--- a/bitbake/lib/toaster/toastergui/views.py
+++ b/bitbake/lib/toaster/toastergui/views.py
@@ -35,7 +35,7 @@ from orm.models import BitbakeVersion, CustomImageRecipe
 from bldcontrol import bbcontroller
 from django.views.decorators.cache import cache_control
 from django.core.urlresolvers import reverse, resolve
-from django.core.exceptions import MultipleObjectsReturned
+from django.core.exceptions import MultipleObjectsReturned, ObjectDoesNotExist
 from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
 from django.http import HttpResponseBadRequest, HttpResponseNotFound
 from django.utils import timezone
@@ -2575,78 +2575,63 @@ if True:
 
         return context
 
-    def _file_name_for_artifact(b, artifact_type, artifact_id):
+    def _file_names_for_artifact(build, artifact_type, artifact_id):
+        """
+        Return a tuple (file path, file name for the download response) for an
+        artifact of type artifact_type with ID artifact_id for build; if
+        artifact type is not supported, returns (None, None)
+        """
         file_name = None
-        # Target_Image_File file_name
-        if artifact_type == "imagefile":
-            file_name = Target_Image_File.objects.get(target__build = b, pk = artifact_id).file_name
+        response_file_name = None
+
+        if artifact_type == "cookerlog":
+            file_name = build.cooker_log_path
+            response_file_name = "cooker.log"
+
+        elif artifact_type == "imagefile":
+            file_name = Target_Image_File.objects.get(target__build = build, pk = artifact_id).file_name
 
         elif artifact_type == "buildartifact":
-            file_name = BuildArtifact.objects.get(build = b, pk = artifact_id).file_name
+            file_name = BuildArtifact.objects.get(build = build, pk = artifact_id).file_name
 
-        elif artifact_type ==  "licensemanifest":
-            file_name = Target.objects.get(build = b, pk = artifact_id).license_manifest_path
+        elif artifact_type == "licensemanifest":
+            file_name = Target.objects.get(build = build, pk = artifact_id).license_manifest_path
 
         elif artifact_type == "tasklogfile":
-            file_name = Task.objects.get(build = b, pk = artifact_id).logfile
+            file_name = Task.objects.get(build = build, pk = artifact_id).logfile
 
         elif artifact_type == "logmessagefile":
-            file_name = LogMessage.objects.get(build = b, pk = artifact_id).pathname
-        else:
-            raise Exception("FIXME: artifact type %s not implemented" % (artifact_type))
+            file_name = LogMessage.objects.get(build = build, pk = artifact_id).pathname
 
-        return file_name
+        if file_name and not response_file_name:
+            response_file_name = os.path.basename(file_name)
 
+        return (file_name, response_file_name)
 
     def build_artifact(request, build_id, artifact_type, artifact_id):
-        if artifact_type in ["cookerlog"]:
-            try:
-                build = Build.objects.get(pk = build_id)
-                file_name = build.cooker_log_path
+        """
+        View which returns a build artifact file as a response
+        """
+        file_name = None
+        response_file_name = None
+
+        try:
+            build = Build.objects.get(pk = build_id)
+            file_name, response_file_name = _file_names_for_artifact(
+                build, artifact_type, artifact_id
+            )
+
+            if file_name and response_file_name:
                 fsock = open(file_name, "r")
                 content_type = MimeTypeFinder.get_mimetype(file_name)
 
                 response = HttpResponse(fsock, content_type = content_type)
 
-                disposition = 'attachment; filename=cooker.log'
-                response['Content-Disposition'] = disposition
+                disposition = "attachment; filename=" + response_file_name
+                response["Content-Disposition"] = disposition
 
                 return response
-            except IOError:
-                context = {
-                    'build' : Build.objects.get(pk = build_id),
-                }
-                return render(request, "unavailable_artifact.html", context)
-
-        else:
-            # retrieve the artifact directly from the build environment
-            return _get_be_artifact(request, build_id, artifact_type, artifact_id)
-
-
-    def _get_be_artifact(request, build_id, artifact_type, artifact_id):
-        try:
-            b = Build.objects.get(pk=build_id)
-            if b.buildrequest is None or b.buildrequest.environment is None:
-                raise Exception("Artifact not available for download (missing build request or build environment)")
-
-            file_name = _file_name_for_artifact(b, artifact_type, artifact_id)
-            fsock = None
-            content_type='application/force-download'
-
-            if file_name is None:
-                raise Exception("Could not handle artifact %s id %s" % (artifact_type, artifact_id))
             else:
-                content_type = MimeTypeFinder.get_mimetype(file_name)
-                fsock = b.buildrequest.environment.get_artifact(file_name)
-                file_name = os.path.basename(file_name) # we assume that the build environment system has the same path conventions as host
-
-            response = HttpResponse(fsock, content_type = content_type)
-
-            # returns a file from the environment
-            response['Content-Disposition'] = 'attachment; filename=' + file_name
-            return response
-        except IOError:
-            context = {
-                'build' : Build.objects.get(pk = build_id),
-            }
-            return render(request, "unavailable_artifact.html", context)
+                return render(request, "unavailable_artifact.html")
+        except ObjectDoesNotExist, IOError:
+            return render(request, "unavailable_artifact.html")
-- 
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [review-request][PATCH 0/3][V2] Fix SDK downloads for builds

[Barros Pena, Belen]

[-- Attachment #1: Type: text/plain, Size: 4035 bytes --]



On 15/01/2016 16:30, "toaster-bounces@yoctoproject.org on behalf of Elliot
Smith" <toaster-bounces@yoctoproject.org on behalf of
elliot.smith@intel.com> wrote:

>V2:
>
>Rebased on toaster-next and fixed conflicts.

Hi Elliot,

Somehow this is not working for me. I built an sdk for core-image-minimal
from Toaster on a clean build directory, but I cannot see the sdk
artifacts shown in the build dashboard, although they are in
/build/tmp/deploy/sdk. I've attached a screenshot of what I see.

Am I doing something wrong?

Thanks!

Belén


>
>V1:
>
>Artifacts could not be downloaded for a build as they were not being
>listed.
>It was not possible to download them manually either, as the artifact
>download
>code was broken for SDK artifacts produced by command line builds.
>
>Fix how toaster.bbclass hooks onto bitbake's build tasks, so that the SDK
>artifacts are picked up after do_populate_sdk runs.
>
>Fix Toaster code for returning SDK artifacts in HTTP responses.
>
>Changes since b53df3c (toaster-next) are in
>git://git.yoctoproject.org/poky-contrib, elliot/toaster/sdk_artifacts-7603
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/sd
>k_artifacts-7603
>
>NB one of the commits on this branch (fbc2024) has been submitted to
>oe-core:
>http://lists.openembedded.org/pipermail/openembedded-core/2016-January/115
>366.html
>This patch set contains the other commits on the branch.
>
>Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7603
>
>To test:
>
>1. Ensure that your database has no build artifacts in it, either by
>removing the database, clearing the relevant table with SQL via the
>command line
>(delete from orm_buildartifact), or removing the objects via the Django
>shell
>(orm.models.BuildArtifact.objects.all().delete()).
>
>Note that the current behaviour, where we only associate SDK artifacts
>with the
>first build which produces them, has not been modified. So if you have
>already
>run a build and have files in tmp/deploy/sdk associated with it,
>subsequent
>builds will not be associated with those files.
>
>2. Start a new build from the command line or Toaster, invoking the
>populate_sdk
>target:
>
># command line
>bitbake -c populate_sdk core-image-minimal
>
># Toaster (NB you must use the "Local" release)
>run a build like "core-image-minimal:populate_sdk"
>
>3. View the build dashboard for the build when it completes and ensure it
>contains an "Other artifacts" section, as shown in the bug report.
>
>4. Click on one of the "Other artifact" links and ensure that the artifact
>downloads correctly.
>
>As this code slightly modifies all other artifact downloads, you could
>also
>check downloads for the following, too:
>
>* cooker logs
>* task logs
>* image files
>* licence manifests
>
>Elliot Smith (3):
>  toaster: toasterui: listen for bb.event.MetadataEvent
>  toaster: toasterui: log OSErrorException metadata events
>  toaster: toastergui: make artifact download more robust
>
> bitbake/lib/bb/ui/toasterui.py                     |  5 +-
> .../toastergui/templates/unavailable_artifact.html | 13 ++-
> bitbake/lib/toaster/toastergui/views.py            | 97
>+++++++++-------------
> 3 files changed, 51 insertions(+), 64 deletions(-)
>
>--
>Elliot Smith
>Software Engineer
>Intel OTC
>
>---------------------------------------------------------------------
>Intel Corporation (UK) Limited
>Registered No. 1134945 (England)
>Registered Office: Pipers Way, Swindon SN3 1RJ
>VAT No: 860 2173 47
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster


[-- Attachment #2: sdk_build_dashboard.png --]
[-- Type: image/png, Size: 194334 bytes --]

Re: [review-request][PATCH 0/3][V2] Fix SDK downloads for builds

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 4661 bytes --]

On 16 January 2016 at 12:23, Barros Pena, Belen <belen.barros.pena@intel.com
> wrote:

>
> On 15/01/2016 16:30, "toaster-bounces@yoctoproject.org on behalf of Elliot
> Smith" <toaster-bounces@yoctoproject.org on behalf of
> elliot.smith@intel.com> wrote:
>
> >V2:
> >
> >Rebased on toaster-next and fixed conflicts.
>
> Hi Elliot,
>
> Somehow this is not working for me. I built an sdk for core-image-minimal
> from Toaster on a clean build directory, but I cannot see the sdk
> artifacts shown in the build dashboard, although they are in
> /build/tmp/deploy/sdk. I've attached a screenshot of what I see.
>

I don't think I mentioned it in my review request, but you must build
against the *local* release, otherwise the changes to bitbake (in
meta/classes/toaster.bbclass) won't be active.

Elliot


>
> Am I doing something wrong?
>
> Thanks!
>
> Belén
>
>
> >
> >V1:
> >
> >Artifacts could not be downloaded for a build as they were not being
> >listed.
> >It was not possible to download them manually either, as the artifact
> >download
> >code was broken for SDK artifacts produced by command line builds.
> >
> >Fix how toaster.bbclass hooks onto bitbake's build tasks, so that the SDK
> >artifacts are picked up after do_populate_sdk runs.
> >
> >Fix Toaster code for returning SDK artifacts in HTTP responses.
> >
> >Changes since b53df3c (toaster-next) are in
> >git://git.yoctoproject.org/poky-contrib,
> elliot/toaster/sdk_artifacts-7603
> >
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/sd
> >k_artifacts-7603
> >
> >NB one of the commits on this branch (fbc2024) has been submitted to
> >oe-core:
> >
> http://lists.openembedded.org/pipermail/openembedded-core/2016-January/115
> >366.html
> >This patch set contains the other commits on the branch.
> >
> >Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7603
> >
> >To test:
> >
> >1. Ensure that your database has no build artifacts in it, either by
> >removing the database, clearing the relevant table with SQL via the
> >command line
> >(delete from orm_buildartifact), or removing the objects via the Django
> >shell
> >(orm.models.BuildArtifact.objects.all().delete()).
> >
> >Note that the current behaviour, where we only associate SDK artifacts
> >with the
> >first build which produces them, has not been modified. So if you have
> >already
> >run a build and have files in tmp/deploy/sdk associated with it,
> >subsequent
> >builds will not be associated with those files.
> >
> >2. Start a new build from the command line or Toaster, invoking the
> >populate_sdk
> >target:
> >
> ># command line
> >bitbake -c populate_sdk core-image-minimal
> >
> ># Toaster (NB you must use the "Local" release)
> >run a build like "core-image-minimal:populate_sdk"
> >
> >3. View the build dashboard for the build when it completes and ensure it
> >contains an "Other artifacts" section, as shown in the bug report.
> >
> >4. Click on one of the "Other artifact" links and ensure that the artifact
> >downloads correctly.
> >
> >As this code slightly modifies all other artifact downloads, you could
> >also
> >check downloads for the following, too:
> >
> >* cooker logs
> >* task logs
> >* image files
> >* licence manifests
> >
> >Elliot Smith (3):
> >  toaster: toasterui: listen for bb.event.MetadataEvent
> >  toaster: toasterui: log OSErrorException metadata events
> >  toaster: toastergui: make artifact download more robust
> >
> > bitbake/lib/bb/ui/toasterui.py                     |  5 +-
> > .../toastergui/templates/unavailable_artifact.html | 13 ++-
> > bitbake/lib/toaster/toastergui/views.py            | 97
> >+++++++++-------------
> > 3 files changed, 51 insertions(+), 64 deletions(-)
> >
> >--
> >Elliot Smith
> >Software Engineer
> >Intel OTC
> >
> >---------------------------------------------------------------------
> >Intel Corporation (UK) Limited
> >Registered No. 1134945 (England)
> >Registered Office: Pipers Way, Swindon SN3 1RJ
> >VAT No: 860 2173 47
> >
> >This e-mail and any attachments may contain confidential material for
> >the sole use of the intended recipient(s). Any review or distribution
> >by others is strictly prohibited. If you are not the intended
> >recipient, please contact the sender and delete all copies.
> >
> >--
> >_______________________________________________
> >toaster mailing list
> >toaster@yoctoproject.org
> >https://lists.yoctoproject.org/listinfo/toaster
>
>


-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 6797 bytes --]

Re: [review-request][PATCH 0/3][V2] Fix SDK downloads for builds

[Barros Pena, Belen]

On 18/01/2016 11:17, "Smith, Elliot" <elliot.smith@intel.com> wrote:

>On 16 January 2016 at 12:23, Barros Pena, Belen
><belen.barros.pena@intel.com> wrote:
>
>
>On 15/01/2016 16:30, "toaster-bounces@yoctoproject.org on behalf of Elliot
>Smith" <toaster-bounces@yoctoproject.org on behalf of
>elliot.smith@intel.com> wrote:
>
>>V2:
>>
>>Rebased on toaster-next and fixed conflicts.
>
>Hi Elliot,
>
>Somehow this is not working for me. I built an sdk for core-image-minimal
>from Toaster on a clean build directory, but I cannot see the sdk
>artifacts shown in the build dashboard, although they are in
>/build/tmp/deploy/sdk. I've attached a screenshot of what I see.
>
>
>
>
>I don't think I mentioned it in my review request, but you must build
>against the local release, otherwise the changes to bitbake (in
>meta/classes/toaster.bbclass) won't be active.

Right, that was the problem. If I build with the local release I can see
the sdk artifacts listed.

Thanks!

Belén


>
>
>
>Elliot
>
> 
>
>
>Am I doing something wrong?
>
>Thanks!
>
>Belén
>
>
>>
>>V1:
>>
>>Artifacts could not be downloaded for a build as they were not being
>>listed.
>>It was not possible to download them manually either, as the artifact
>>download
>>code was broken for SDK artifacts produced by command line builds.
>>
>>Fix how toaster.bbclass hooks onto bitbake's build tasks, so that the SDK
>>artifacts are picked up after do_populate_sdk runs.
>>
>>Fix Toaster code for returning SDK artifacts in HTTP responses.
>>
>>Changes since b53df3c (toaster-next) are in
>>git://git.yoctoproject.org/poky-contrib
>><http://git.yoctoproject.org/poky-contrib>,
>>elliot/toaster/sdk_artifacts-7603
>>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/s
>>d
>>k_artifacts-7603
>>
>>NB one of the commits on this branch (fbc2024) has been submitted to
>>oe-core:
>>http://lists.openembedded.org/pipermail/openembedded-core/2016-January/11
>>5
>>366.html
>>This patch set contains the other commits on the branch.
>>
>>Related bug: 
>https://bugzilla.yoctoproject.org/show_bug.cgi?id=7603
><https://bugzilla.yoctoproject.org/show_bug.cgi?id=7603>
>>
>>To test:
>>
>>1. Ensure that your database has no build artifacts in it, either by
>>removing the database, clearing the relevant table with SQL via the
>>command line
>>(delete from orm_buildartifact), or removing the objects via the Django
>>shell
>>(orm.models.BuildArtifact.objects.all().delete()).
>>
>>Note that the current behaviour, where we only associate SDK artifacts
>>with the
>>first build which produces them, has not been modified. So if you have
>>already
>>run a build and have files in tmp/deploy/sdk associated with it,
>>subsequent
>>builds will not be associated with those files.
>>
>>2. Start a new build from the command line or Toaster, invoking the
>>populate_sdk
>>target:
>>
>># command line
>>bitbake -c populate_sdk core-image-minimal
>>
>># Toaster (NB you must use the "Local" release)
>>run a build like "core-image-minimal:populate_sdk"
>>
>>3. View the build dashboard for the build when it completes and ensure it
>>contains an "Other artifacts" section, as shown in the bug report.
>>
>>4. Click on one of the "Other artifact" links and ensure that the
>>artifact
>>downloads correctly.
>>
>>As this code slightly modifies all other artifact downloads, you could
>>also
>>check downloads for the following, too:
>>
>>* cooker logs
>>* task logs
>>* image files
>>* licence manifests
>>
>>Elliot Smith (3):
>>  toaster: toasterui: listen for bb.event.MetadataEvent
>>  toaster: toasterui: log OSErrorException metadata events
>>  toaster: toastergui: make artifact download more robust
>>
>> bitbake/lib/bb/ui/toasterui.py                     |  5 +-
>> .../toastergui/templates/unavailable_artifact.html | 13 ++-
>> bitbake/lib/toaster/toastergui/views.py            | 97
>>+++++++++-------------
>> 3 files changed, 51 insertions(+), 64 deletions(-)
>>
>>--
>>Elliot Smith
>>Software Engineer
>>Intel OTC
>>
>>---------------------------------------------------------------------
>>Intel Corporation (UK) Limited
>>Registered No. 1134945 (England)
>>Registered Office: Pipers Way, Swindon SN3 1RJ
>>VAT No: 860 2173 47
>>
>>This e-mail and any attachments may contain confidential material for
>>the sole use of the intended recipient(s). Any review or distribution
>>by others is strictly prohibited. If you are not the intended
>>recipient, please contact the sender and delete all copies.
>>
>
>
>>--
>>_______________________________________________
>>toaster mailing list
>>toaster@yoctoproject.org
>>https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
>
>
>
>
>-- 
>Elliot Smith
>Software Engineer
>Intel Open Source Technology Centre
>
>
>
>

Re: [review-request][PATCH 0/3][V2] Fix SDK downloads for builds

[Ed Bartosh]

Hi,

I've sent this patchset upstream & updated toaster-next.

On Mon, Jan 18, 2016 at 11:37:51AM +0000, Barros Pena, Belen wrote:
> 
> 
> On 18/01/2016 11:17, "Smith, Elliot" <elliot.smith@intel.com> wrote:
> 
> >On 16 January 2016 at 12:23, Barros Pena, Belen
> ><belen.barros.pena@intel.com> wrote:
> >
> >
> >On 15/01/2016 16:30, "toaster-bounces@yoctoproject.org on behalf of Elliot
> >Smith" <toaster-bounces@yoctoproject.org on behalf of
> >elliot.smith@intel.com> wrote:
> >
> >>V2:
> >>
> >>Rebased on toaster-next and fixed conflicts.
> >
> >Hi Elliot,
> >
> >Somehow this is not working for me. I built an sdk for core-image-minimal
> >from Toaster on a clean build directory, but I cannot see the sdk
> >artifacts shown in the build dashboard, although they are in
> >/build/tmp/deploy/sdk. I've attached a screenshot of what I see.
> >
> >
> >
> >
> >I don't think I mentioned it in my review request, but you must build
> >against the local release, otherwise the changes to bitbake (in
> >meta/classes/toaster.bbclass) won't be active.
> 
> Right, that was the problem. If I build with the local release I can see
> the sdk artifacts listed.
> 
> Thanks!
> 
> Belén
> 
> 
> >
> >
> >
> >Elliot
> >
> > 
> >
> >
> >Am I doing something wrong?
> >
> >Thanks!
> >
> >Belén
> >
> >
> >>
> >>V1:
> >>
> >>Artifacts could not be downloaded for a build as they were not being
> >>listed.
> >>It was not possible to download them manually either, as the artifact
> >>download
> >>code was broken for SDK artifacts produced by command line builds.
> >>
> >>Fix how toaster.bbclass hooks onto bitbake's build tasks, so that the SDK
> >>artifacts are picked up after do_populate_sdk runs.
> >>
> >>Fix Toaster code for returning SDK artifacts in HTTP responses.
> >>
> >>Changes since b53df3c (toaster-next) are in
> >>git://git.yoctoproject.org/poky-contrib
> >><http://git.yoctoproject.org/poky-contrib>,
> >>elliot/toaster/sdk_artifacts-7603
> >>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/s
> >>d
> >>k_artifacts-7603
> >>
> >>NB one of the commits on this branch (fbc2024) has been submitted to
> >>oe-core:
> >>http://lists.openembedded.org/pipermail/openembedded-core/2016-January/11
> >>5
> >>366.html
> >>This patch set contains the other commits on the branch.
> >>
> >>Related bug: 
> >https://bugzilla.yoctoproject.org/show_bug.cgi?id=7603
> ><https://bugzilla.yoctoproject.org/show_bug.cgi?id=7603>
> >>
> >>To test:
> >>
> >>1. Ensure that your database has no build artifacts in it, either by
> >>removing the database, clearing the relevant table with SQL via the
> >>command line
> >>(delete from orm_buildartifact), or removing the objects via the Django
> >>shell
> >>(orm.models.BuildArtifact.objects.all().delete()).
> >>
> >>Note that the current behaviour, where we only associate SDK artifacts
> >>with the
> >>first build which produces them, has not been modified. So if you have
> >>already
> >>run a build and have files in tmp/deploy/sdk associated with it,
> >>subsequent
> >>builds will not be associated with those files.
> >>
> >>2. Start a new build from the command line or Toaster, invoking the
> >>populate_sdk
> >>target:
> >>
> >># command line
> >>bitbake -c populate_sdk core-image-minimal
> >>
> >># Toaster (NB you must use the "Local" release)
> >>run a build like "core-image-minimal:populate_sdk"
> >>
> >>3. View the build dashboard for the build when it completes and ensure it
> >>contains an "Other artifacts" section, as shown in the bug report.
> >>
> >>4. Click on one of the "Other artifact" links and ensure that the
> >>artifact
> >>downloads correctly.
> >>
> >>As this code slightly modifies all other artifact downloads, you could
> >>also
> >>check downloads for the following, too:
> >>
> >>* cooker logs
> >>* task logs
> >>* image files
> >>* licence manifests
> >>
> >>Elliot Smith (3):
> >>  toaster: toasterui: listen for bb.event.MetadataEvent
> >>  toaster: toasterui: log OSErrorException metadata events
> >>  toaster: toastergui: make artifact download more robust
> >>
> >> bitbake/lib/bb/ui/toasterui.py                     |  5 +-
> >> .../toastergui/templates/unavailable_artifact.html | 13 ++-
> >> bitbake/lib/toaster/toastergui/views.py            | 97
> >>+++++++++-------------
> >> 3 files changed, 51 insertions(+), 64 deletions(-)
> >>
> >>--
> >>Elliot Smith
> >>Software Engineer
> >>Intel OTC
> >>
> >>---------------------------------------------------------------------
> >>Intel Corporation (UK) Limited
> >>Registered No. 1134945 (England)
> >>Registered Office: Pipers Way, Swindon SN3 1RJ
> >>VAT No: 860 2173 47
> >>
> >>This e-mail and any attachments may contain confidential material for
> >>the sole use of the intended recipient(s). Any review or distribution
> >>by others is strictly prohibited. If you are not the intended
> >>recipient, please contact the sender and delete all copies.
> >>
> >
> >
> >>--
> >>_______________________________________________
> >>toaster mailing list
> >>toaster@yoctoproject.org
> >>https://lists.yoctoproject.org/listinfo/toaster
> >
> >
> >
> >
> >
> >
> >
> >
> >-- 
> >Elliot Smith
> >Software Engineer
> >Intel Open Source Technology Centre
> >
> >
> >
> >
> 
> -- 
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster

-- 
--
Regards,
Ed

[review-request][PATCH 1/3] toaster: toasterui Listen for bb.event.MetadataEvent

[Elliot Smith]

The event mask for toasterui doesn't include MetadataEvents.
This means that we're missing the ArtifactFileSize event
(among others), which is the one we use to populate the SDK
artifact table.

Add that event type to the toasterui event mask so we can
record SDK artifacts as they are created.

[YOCTO #7603]

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 0265df7..19be623 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -99,7 +99,8 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
               "bb.command.CommandExit", "bb.command.CommandCompleted",  "bb.cooker.CookerExit",
               "bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
               "bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
-              "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent"]
+              "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
+              "bb.event.MetadataEvent"]
 
 def main(server, eventHandler, params):
     # set to a logging.FileHandler instance when a build starts;
-- 
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.