[PATCH 1/2] toasterui: Remove the excessive exception logging

[PATCH 1/2] toasterui: Remove the excessive exception logging

[Michael Wood]

Remove the very verbose log dump from toasterui. This generates several
megabytes of not that useful debug information and actually hinders
finding the original exception.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index ee40110..29aed2e 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -447,13 +447,6 @@ def main(server, eventHandler, params):
             exception_data = traceback.format_exc()
             logger.error("%s\n%s" , e, exception_data)
 
-            _, _, tb = sys.exc_info()
-            if tb is not None:
-                curr = tb
-                while curr is not None:
-                    logger.error("Error data dump %s\n%s\n" , traceback.format_tb(curr,1), pformat(curr.tb_frame.f_locals))
-                    curr = curr.tb_next
-
             # save them to database, if possible; if it fails, we already logged to console.
             try:
                 buildinfohelper.store_log_exception("%s\n%s" % (str(e), exception_data))
-- 
2.1.4

[PATCH 2/2] toaster: buildinfohelper Add additional metadata to the built layer

[Michael Wood]

Add additional metadata to the layer created for build history to be
able to identify the layer and recipe later on. Specifically this is the
branch and release to which the recipe and layer are associated with
enabling differentiation of two recipes which are local release and
master and 'master' release.

[YOCTO #8528]
[YOCTO #8545]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 31 ++++++++++++++++++++-----------
 1 file changed, 20 insertions(+), 11 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 7805086..6b19fda 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -355,12 +355,17 @@ class ORMWrapper(object):
 
             # create a new copy of this layer version as a snapshot for
             # historical purposes
-            layer_copy, c = Layer_Version.objects.get_or_create(build=build_obj,
-                            layer=layer_obj.layer,
-                            commit=layer_version_information['commit'],
-                            local_path = layer_version_information['local_path'],
-                            )
-            logger.info("created new historical layer version %d", layer_copy.pk)
+            layer_copy, c = Layer_Version.objects.get_or_create(
+                build=build_obj,
+                layer=layer_obj.layer,
+                up_branch=layer_obj.up_branch,
+                branch=layer_version_information['branch'],
+                commit=layer_version_information['commit'],
+                local_path=layer_version_information['local_path'],
+            )
+
+            logger.info("created new historical layer version %d",
+                        layer_copy.pk)
 
             self.layer_version_built.append(layer_copy)
 
@@ -581,11 +586,15 @@ class ORMWrapper(object):
                 packagedict[p]['object'].package_dependencies_target.all().delete()
                 packagedict[p]['object'].package_dependencies_source.all().delete()
                 try:
-                    recipe = self._cached_get(Recipe,
-                                              name=built_recipe.name,
-                                              layer_version__build=None,
-                                              file_path=built_recipe.file_path,
-                                              version=built_recipe.version)
+                    recipe = self._cached_get(
+                        Recipe,
+                        name=built_recipe.name,
+                        layer_version__build=None,
+                        layer_version__up_branch=
+                        built_recipe.layer_version.up_branch,
+                        file_path=built_recipe.file_path,
+                        version=built_recipe.version
+                    )
                 except (Recipe.DoesNotExist,
                         Recipe.MultipleObjectsReturned) as e:
                     logger.info("We did not find one recipe for the"
-- 
2.1.4

Re: [PATCH 2/2] toaster: buildinfohelper Add additional metadata to the built layer

[Barros Pena, Belen]

On 10/03/2016 14:28, "toaster-bounces@yoctoproject.org on behalf of
Michael Wood" <toaster-bounces@yoctoproject.org on behalf of
michael.g.wood@intel.com> wrote:

>Add additional metadata to the layer created for build history to be
>able to identify the layer and recipe later on. Specifically this is the
>branch and release to which the recipe and layer are associated with
>enabling differentiation of two recipes which are local release and
>master and 'master' release.
>
>[YOCTO #8528]

I can't get this to work, I'm afraid. This is what I did:

* Created a project with the local release
* Built core-image-minimal
* When the build finished, went to the image recipes page

It still shows 0 image recipes.

>[YOCTO #8545]

This totally works :)

Thanks!

Belén

>
>Signed-off-by: Michael Wood <michael.g.wood@intel.com>
>---
> bitbake/lib/bb/ui/buildinfohelper.py | 31 ++++++++++++++++++++-----------
> 1 file changed, 20 insertions(+), 11 deletions(-)
>
>diff --git a/bitbake/lib/bb/ui/buildinfohelper.py
>b/bitbake/lib/bb/ui/buildinfohelper.py
>index 7805086..6b19fda 100644
>--- a/bitbake/lib/bb/ui/buildinfohelper.py
>+++ b/bitbake/lib/bb/ui/buildinfohelper.py
>@@ -355,12 +355,17 @@ class ORMWrapper(object):
> 
>             # create a new copy of this layer version as a snapshot for
>             # historical purposes
>-            layer_copy, c =
>Layer_Version.objects.get_or_create(build=build_obj,
>-                            layer=layer_obj.layer,
>-                            commit=layer_version_information['commit'],
>-                            local_path =
>layer_version_information['local_path'],
>-                            )
>-            logger.info("created new historical layer version %d",
>layer_copy.pk)
>+            layer_copy, c = Layer_Version.objects.get_or_create(
>+                build=build_obj,
>+                layer=layer_obj.layer,
>+                up_branch=layer_obj.up_branch,
>+                branch=layer_version_information['branch'],
>+                commit=layer_version_information['commit'],
>+                local_path=layer_version_information['local_path'],
>+            )
>+
>+            logger.info("created new historical layer version %d",
>+                        layer_copy.pk)
> 
>             self.layer_version_built.append(layer_copy)
> 
>@@ -581,11 +586,15 @@ class ORMWrapper(object):
>                 
>packagedict[p]['object'].package_dependencies_target.all().delete()
>                 
>packagedict[p]['object'].package_dependencies_source.all().delete()
>                 try:
>-                    recipe = self._cached_get(Recipe,
>-                                              name=built_recipe.name,
>-                                              layer_version__build=None,
>-                 
>file_path=built_recipe.file_path,
>-                 
>version=built_recipe.version)
>+                    recipe = self._cached_get(
>+                        Recipe,
>+                        name=built_recipe.name,
>+                        layer_version__build=None,
>+                        layer_version__up_branch=
>+                        built_recipe.layer_version.up_branch,
>+                        file_path=built_recipe.file_path,
>+                        version=built_recipe.version
>+                    )
>                 except (Recipe.DoesNotExist,
>                         Recipe.MultipleObjectsReturned) as e:
>                     logger.info("We did not find one recipe for the"
>-- 
>2.1.4
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: [PATCH 1/2] toasterui: Remove the excessive exception logging

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1785 bytes --]

I will upstream this patch of the series; but if I understand correctly,
there are still a couple of issues to deal with in the first patch, so I
won't upstream that yet.

Elliot

On 10 March 2016 at 14:28, Michael Wood <michael.g.wood@intel.com> wrote:

> Remove the very verbose log dump from toasterui. This generates several
> megabytes of not that useful debug information and actually hinders
> finding the original exception.
>
> Signed-off-by: Michael Wood <michael.g.wood@intel.com>
> ---
>  bitbake/lib/bb/ui/toasterui.py | 7 -------
>  1 file changed, 7 deletions(-)
>
> diff --git a/bitbake/lib/bb/ui/toasterui.py
> b/bitbake/lib/bb/ui/toasterui.py
> index ee40110..29aed2e 100644
> --- a/bitbake/lib/bb/ui/toasterui.py
> +++ b/bitbake/lib/bb/ui/toasterui.py
> @@ -447,13 +447,6 @@ def main(server, eventHandler, params):
>              exception_data = traceback.format_exc()
>              logger.error("%s\n%s" , e, exception_data)
>
> -            _, _, tb = sys.exc_info()
> -            if tb is not None:
> -                curr = tb
> -                while curr is not None:
> -                    logger.error("Error data dump %s\n%s\n" ,
> traceback.format_tb(curr,1), pformat(curr.tb_frame.f_locals))
> -                    curr = curr.tb_next
> -
>              # save them to database, if possible; if it fails, we already
> logged to console.
>              try:
>                  buildinfohelper.store_log_exception("%s\n%s" % (str(e),
> exception_data))
> --
> 2.1.4
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 2628 bytes --]

Re: [PATCH 1/2] toasterui: Remove the excessive exception logging

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]

Upstreamed to bitbake-devel and added to toaster-next.

Thanks.
Elliot

On 10 March 2016 at 14:28, Michael Wood <michael.g.wood@intel.com> wrote:

> Remove the very verbose log dump from toasterui. This generates several
> megabytes of not that useful debug information and actually hinders
> finding the original exception.
>
> Signed-off-by: Michael Wood <michael.g.wood@intel.com>
> ---
>  bitbake/lib/bb/ui/toasterui.py | 7 -------
>  1 file changed, 7 deletions(-)
>
> diff --git a/bitbake/lib/bb/ui/toasterui.py
> b/bitbake/lib/bb/ui/toasterui.py
> index ee40110..29aed2e 100644
> --- a/bitbake/lib/bb/ui/toasterui.py
> +++ b/bitbake/lib/bb/ui/toasterui.py
> @@ -447,13 +447,6 @@ def main(server, eventHandler, params):
>              exception_data = traceback.format_exc()
>              logger.error("%s\n%s" , e, exception_data)
>
> -            _, _, tb = sys.exc_info()
> -            if tb is not None:
> -                curr = tb
> -                while curr is not None:
> -                    logger.error("Error data dump %s\n%s\n" ,
> traceback.format_tb(curr,1), pformat(curr.tb_frame.f_locals))
> -                    curr = curr.tb_next
> -
>              # save them to database, if possible; if it fails, we already
> logged to console.
>              try:
>                  buildinfohelper.store_log_exception("%s\n%s" % (str(e),
> exception_data))
> --
> 2.1.4
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 2526 bytes --]

[v2 PATCH] toaster: buildinfohelper Add additional metadata to the built layer

[Michael Wood]

Add additional metadata to the layer created for build history to be
able to identify the layer and recipe later on. Specifically this is the
branch and release to which the recipe and layer are associated with
enabling differentiation of two recipes which are local release and
master and 'master' release.

[YOCTO #8528]
[YOCTO #8545]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 34 +++++++++++++++++++++++-----------
 1 file changed, 23 insertions(+), 11 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 28c7353..80ac800 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -355,12 +355,17 @@ class ORMWrapper(object):
 
             # create a new copy of this layer version as a snapshot for
             # historical purposes
-            layer_copy, c = Layer_Version.objects.get_or_create(build=build_obj,
-                            layer=layer_obj.layer,
-                            commit=layer_version_information['commit'],
-                            local_path = layer_version_information['local_path'],
-                            )
-            logger.info("created new historical layer version %d", layer_copy.pk)
+            layer_copy, c = Layer_Version.objects.get_or_create(
+                build=build_obj,
+                layer=layer_obj.layer,
+                up_branch=layer_obj.up_branch,
+                branch=layer_version_information['branch'],
+                commit=layer_version_information['commit'],
+                local_path=layer_version_information['local_path'],
+            )
+
+            logger.info("created new historical layer version %d",
+                        layer_copy.pk)
 
             self.layer_version_built.append(layer_copy)
 
@@ -581,11 +586,15 @@ class ORMWrapper(object):
                 packagedict[p]['object'].package_dependencies_target.all().delete()
                 packagedict[p]['object'].package_dependencies_source.all().delete()
                 try:
-                    recipe = self._cached_get(Recipe,
-                                              name=built_recipe.name,
-                                              layer_version__build=None,
-                                              file_path=built_recipe.file_path,
-                                              version=built_recipe.version)
+                    recipe = self._cached_get(
+                        Recipe,
+                        name=built_recipe.name,
+                        layer_version__build=None,
+                        layer_version__up_branch=
+                        built_recipe.layer_version.up_branch,
+                        file_path=built_recipe.file_path,
+                        version=built_recipe.version
+                    )
                 except (Recipe.DoesNotExist,
                         Recipe.MultipleObjectsReturned) as e:
                     logger.info("We did not find one recipe for the"
@@ -1293,6 +1302,9 @@ class BuildInfoHelper(object):
                 for cls in event._depgraph['pn'][pn]['inherits']:
                     if cls.endswith('/image.bbclass'):
                         recipe.is_image = True
+                        recipe_info['is_image'] = True
+                        # Save the is_image state to the relevant recipe objects
+                        self.orm_wrapper.get_update_recipe_object(recipe_info)
                         break
             if recipe.is_image:
                 for t in self.internal_state['targets']:
-- 
2.1.4

Re: [v2 PATCH] toaster: buildinfohelper Add additional metadata to the built layer

[Barros Pena, Belen]

On 17/03/2016 18:27, "toaster-bounces@yoctoproject.org on behalf of
Michael Wood" <toaster-bounces@yoctoproject.org on behalf of
michael.g.wood@intel.com> wrote:

>Add additional metadata to the layer created for build history to be
>able to identify the layer and recipe later on. Specifically this is the
>branch and release to which the recipe and layer are associated with
>enabling differentiation of two recipes which are local release and
>master and 'master' release.
>
>[YOCTO #8528]
>[YOCTO #8545]

This totally works:

* Images now come up in the image recipes page of local projects after you
build them
* The build information includes the layer branch

But it prompts the question of why we are seeing different branches for
the core layers: meta-poky and meta-yocto-bsp show master, but my
openembedded-core shows a different brach (a development one I was using
previously). I know this has nothing to do with this patch, but just
wondering why this is happening, and whether is a toaster thing or a
bitbake thing.

Thanks!

Belén
 
>
>Signed-off-by: Michael Wood <michael.g.wood@intel.com>
>---
> bitbake/lib/bb/ui/buildinfohelper.py | 34
>+++++++++++++++++++++++-----------
> 1 file changed, 23 insertions(+), 11 deletions(-)
>
>diff --git a/bitbake/lib/bb/ui/buildinfohelper.py
>b/bitbake/lib/bb/ui/buildinfohelper.py
>index 28c7353..80ac800 100644
>--- a/bitbake/lib/bb/ui/buildinfohelper.py
>+++ b/bitbake/lib/bb/ui/buildinfohelper.py
>@@ -355,12 +355,17 @@ class ORMWrapper(object):
> 
>             # create a new copy of this layer version as a snapshot for
>             # historical purposes
>-            layer_copy, c =
>Layer_Version.objects.get_or_create(build=build_obj,
>-                            layer=layer_obj.layer,
>-                            commit=layer_version_information['commit'],
>-                            local_path =
>layer_version_information['local_path'],
>-                            )
>-            logger.info("created new historical layer version %d",
>layer_copy.pk)
>+            layer_copy, c = Layer_Version.objects.get_or_create(
>+                build=build_obj,
>+                layer=layer_obj.layer,
>+                up_branch=layer_obj.up_branch,
>+                branch=layer_version_information['branch'],
>+                commit=layer_version_information['commit'],
>+                local_path=layer_version_information['local_path'],
>+            )
>+
>+            logger.info("created new historical layer version %d",
>+                        layer_copy.pk)
> 
>             self.layer_version_built.append(layer_copy)
> 
>@@ -581,11 +586,15 @@ class ORMWrapper(object):
>                 
>packagedict[p]['object'].package_dependencies_target.all().delete()
>                 
>packagedict[p]['object'].package_dependencies_source.all().delete()
>                 try:
>-                    recipe = self._cached_get(Recipe,
>-                                              name=built_recipe.name,
>-                                              layer_version__build=None,
>-                 
>file_path=built_recipe.file_path,
>-                 
>version=built_recipe.version)
>+                    recipe = self._cached_get(
>+                        Recipe,
>+                        name=built_recipe.name,
>+                        layer_version__build=None,
>+                        layer_version__up_branch=
>+                        built_recipe.layer_version.up_branch,
>+                        file_path=built_recipe.file_path,
>+                        version=built_recipe.version
>+                    )
>                 except (Recipe.DoesNotExist,
>                         Recipe.MultipleObjectsReturned) as e:
>                     logger.info("We did not find one recipe for the"
>@@ -1293,6 +1302,9 @@ class BuildInfoHelper(object):
>                 for cls in event._depgraph['pn'][pn]['inherits']:
>                     if cls.endswith('/image.bbclass'):
>                         recipe.is_image = True
>+                        recipe_info['is_image'] = True
>+                        # Save the is_image state to the relevant recipe
>objects
>+                 
>self.orm_wrapper.get_update_recipe_object(recipe_info)
>                         break
>             if recipe.is_image:
>                 for t in self.internal_state['targets']:
>-- 
>2.1.4
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: [v2 PATCH] toaster: buildinfohelper Add additional metadata to the built layer

[Michael Wood]

On 21/03/16 14:58, Barros Pena, Belen wrote:
>
> On 17/03/2016 18:27, "toaster-bounces@yoctoproject.org on behalf of
> Michael Wood" <toaster-bounces@yoctoproject.org on behalf of
> michael.g.wood@intel.com> wrote:
>
>> Add additional metadata to the layer created for build history to be
>> able to identify the layer and recipe later on. Specifically this is the
>> branch and release to which the recipe and layer are associated with
>> enabling differentiation of two recipes which are local release and
>> master and 'master' release.
>>
>> [YOCTO #8528]
>> [YOCTO #8545]
> This totally works:
>
> * Images now come up in the image recipes page of local projects after you
> build them
> * The build information includes the layer branch
>
> But it prompts the question of why we are seeing different branches for
> the core layers: meta-poky and meta-yocto-bsp show master, but my
> openembedded-core shows a different brach (a development one I was using
> previously). I know this has nothing to do with this patch, but just
> wondering why this is happening, and whether is a toaster thing or a
> bitbake thing.

I guess you can check in the toaster cloned layers directory to see if 
what it's reporting is correct first off. Secondly I suspect that this 
is caused by cloning 'self', if you do a new poky git clone and switch 
branches to contrib/catdog and then ask it to build something in 
openembedded-core it will go 'oh I don't need to go and clone 
openembedded-core from the internet because I know there is a copy in my 
directory /meta/ so I will copy myself/this instead' and of course it 
copies it as-is which probably means the git repo is still checked out 
as contrib/catdog.

Michael

> Thanks!
>
> Belén
>   
>> Signed-off-by: Michael Wood <michael.g.wood@intel.com>
>> ---
>> bitbake/lib/bb/ui/buildinfohelper.py | 34
>> +++++++++++++++++++++++-----------
>> 1 file changed, 23 insertions(+), 11 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/ui/buildinfohelper.py
>> b/bitbake/lib/bb/ui/buildinfohelper.py
>> index 28c7353..80ac800 100644
>> --- a/bitbake/lib/bb/ui/buildinfohelper.py
>> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
>> @@ -355,12 +355,17 @@ class ORMWrapper(object):
>>
>>              # create a new copy of this layer version as a snapshot for
>>              # historical purposes
>> -            layer_copy, c =
>> Layer_Version.objects.get_or_create(build=build_obj,
>> -                            layer=layer_obj.layer,
>> -                            commit=layer_version_information['commit'],
>> -                            local_path =
>> layer_version_information['local_path'],
>> -                            )
>> -            logger.info("created new historical layer version %d",
>> layer_copy.pk)
>> +            layer_copy, c = Layer_Version.objects.get_or_create(
>> +                build=build_obj,
>> +                layer=layer_obj.layer,
>> +                up_branch=layer_obj.up_branch,
>> +                branch=layer_version_information['branch'],
>> +                commit=layer_version_information['commit'],
>> +                local_path=layer_version_information['local_path'],
>> +            )
>> +
>> +            logger.info("created new historical layer version %d",
>> +                        layer_copy.pk)
>>
>>              self.layer_version_built.append(layer_copy)
>>
>> @@ -581,11 +586,15 @@ class ORMWrapper(object):
>>                  
>> packagedict[p]['object'].package_dependencies_target.all().delete()
>>                  
>> packagedict[p]['object'].package_dependencies_source.all().delete()
>>                  try:
>> -                    recipe = self._cached_get(Recipe,
>> -                                              name=built_recipe.name,
>> -                                              layer_version__build=None,
>> -
>> file_path=built_recipe.file_path,
>> -
>> version=built_recipe.version)
>> +                    recipe = self._cached_get(
>> +                        Recipe,
>> +                        name=built_recipe.name,
>> +                        layer_version__build=None,
>> +                        layer_version__up_branch=
>> +                        built_recipe.layer_version.up_branch,
>> +                        file_path=built_recipe.file_path,
>> +                        version=built_recipe.version
>> +                    )
>>                  except (Recipe.DoesNotExist,
>>                          Recipe.MultipleObjectsReturned) as e:
>>                      logger.info("We did not find one recipe for the"
>> @@ -1293,6 +1302,9 @@ class BuildInfoHelper(object):
>>                  for cls in event._depgraph['pn'][pn]['inherits']:
>>                      if cls.endswith('/image.bbclass'):
>>                          recipe.is_image = True
>> +                        recipe_info['is_image'] = True
>> +                        # Save the is_image state to the relevant recipe
>> objects
>> +
>> self.orm_wrapper.get_update_recipe_object(recipe_info)
>>                          break
>>              if recipe.is_image:
>>                  for t in self.internal_state['targets']:
>> -- 
>> 2.1.4
>>
>> -- 
>> _______________________________________________
>> toaster mailing list
>> toaster@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/toaster