[PATCH 0/5] localhostbecontroller fixes

[PATCH 0/5] localhostbecontroller fixes

[Michael Wood]

Fixes for localhostbecontroller to allow file:/// uri for git repositories. Allowing the file:/// uri had an unintended consequence that the layer we generate for the custom image layer/recipes have had a dummy value of file:///_meta-toaster as its giturl so it tried and fails to git clone the dummy url. This fixes this and another issue identified in testing when building a custom image based on a custom image https://bugzilla.yoctoproject.org/show_bug.cgi?id=9206

branch available on poky-contrib michaelw/toaster/localhostbctrl


Michael Wood (5):
  toaster: localhostbecontroller Don't clear out toaster custom layer
    dir
  toaster: orm Add a constant for the CustomImageRecipe's layer name
  toaster: localhostbecontroller Allow file:/// uri type for git repo
  toaster: localhostbecontroller put generated layer in the builddir
  toaster: orm generate CustomImageRecipe contents try secondary path

 bitbake/lib/bb/ui/buildinfohelper.py                    |  2 +-
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 10 +++++-----
 bitbake/lib/toaster/orm/models.py                       | 15 +++++++++++----
 bitbake/lib/toaster/toastergui/views.py                 |  2 +-
 4 files changed, 18 insertions(+), 11 deletions(-)

-- 
2.5.0

[PATCH 1/5] toaster: localhostbecontroller Don't clear out toaster custom layer dir

[Michael Wood]

We may have a recipe which is based on a custom image recipe that has
already been built so keep the recipe file around so that it can be read
by the generate recipe function.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index efd82c3..38e97a2 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -217,9 +217,9 @@ class LocalhostBEController(BuildEnvironmentController):
             raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
 
         # 5. create custom layer and add custom recipes to it
-        layerpath = os.path.join(self.be.sourcedir, "_meta-toaster-custom")
-        if os.path.isdir(layerpath):
-            shutil.rmtree(layerpath) # remove leftovers from previous builds
+        layerpath = os.path.join(self.be.sourcedir,
+                                 "_meta-toaster-custom_",
+                                 bitbake.req.project.name)
         for target in targets:
             try:
                 customrecipe = CustomImageRecipe.objects.get(name=target.target,
-- 
2.5.0

[PATCH 2/5] toaster: orm Add a constant for the CustomImageRecipe's layer name

[Michael Wood]

Use a constant to define the name for the toaster custom images layer
this constant is then used to identify this layer in various places

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py    | 2 +-
 bitbake/lib/toaster/orm/models.py       | 4 ++++
 bitbake/lib/toaster/toastergui/views.py | 2 +-
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 81abede..20ccbc6 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -326,7 +326,7 @@ class ORMWrapper(object):
             # Special case the toaster-custom-images layer which is created
             # on the fly so don't update the values which may cause the layer
             # to be duplicated on a future get_or_create
-            if layer_obj.layer.name == "toaster-custom-images":
+            if layer_obj.layer.name == CustomImageRecipe.LAYER_NAME:
                 return layer_obj
             # We already found our layer version for this build so just
             # update it with the new build information
diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 93b5df3..d451989 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1430,6 +1430,10 @@ class ProjectLayer(models.Model):
         unique_together = (("project", "layercommit"),)
 
 class CustomImageRecipe(Recipe):
+
+    # CustomImageRecipe's belong to layers called:
+    LAYER_NAME = "toaster-custom-images"
+
     search_allowed_fields = ['name']
     base_recipe = models.ForeignKey(Recipe, related_name='based_on_recipe')
     project = models.ForeignKey(Project)
diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py
index f3b8d3c..9baa956 100755
--- a/bitbake/lib/toaster/toastergui/views.py
+++ b/bitbake/lib/toaster/toastergui/views.py
@@ -2392,7 +2392,7 @@ if True:
 
             # create layer 'Custom layer' and verion if needed
             layer = Layer.objects.get_or_create(
-                name="toaster-custom-images",
+                name=CustomImageRecipe.LAYER_NAME,
                 summary="Layer for custom recipes",
                 vcs_url="file:///toaster_created_layer")[0]
 
-- 
2.5.0

[PATCH 3/5] toaster: localhostbecontroller Allow file:/// uri type for git repo

[Michael Wood]

We don't need to skip file:/// uri type locations for git repositories.
If you're using a file:/// uri you should know that it has to be a local
path.

[YOCTO #9200]

igned-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 38e97a2..9deb572 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -128,8 +128,9 @@ class LocalhostBEController(BuildEnvironmentController):
         gitrepos[(bitbake.giturl, bitbake.commit)].append( ("bitbake", bitbake.dirpath) )
 
         for layer in layers:
-            # we don't process local URLs
-            if layer.giturl.startswith("file://"):
+            # We don't need to git clone the layer for the CustomImageRecipe
+            # as it's generated by us layer on if needed
+            if CustomImageRecipe.LAYER_NAME in layer.name:
                 continue
             if not (layer.giturl, layer.commit) in gitrepos:
                 gitrepos[(layer.giturl, layer.commit)] = []
-- 
2.5.0

[PATCH 4/5] toaster: localhostbecontroller put generated layer in the builddir

[Michael Wood]

Move the generated layer for custom recipes into the build directory.
The build directory makes more sense as this layer/recipe is generated
for a particular build/project.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 9deb572..08419f2 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -218,9 +218,8 @@ class LocalhostBEController(BuildEnvironmentController):
             raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
 
         # 5. create custom layer and add custom recipes to it
-        layerpath = os.path.join(self.be.sourcedir,
-                                 "_meta-toaster-custom_",
-                                 bitbake.req.project.name)
+        layerpath = os.path.join(self.be.builddir,
+                                 CustomImageRecipe.LAYER_NAME)
         for target in targets:
             try:
                 customrecipe = CustomImageRecipe.objects.get(name=target.target,
-- 
2.5.0

[PATCH 5/5] toaster: orm generate CustomImageRecipe contents try secondary path

[Michael Wood]

Try a secondary file path if the first does not exist. When we get the
recipe paths and layer information from the layer index it is not a
complete path but we are usually able to reconstruct it. If the complete
real path has been discovered by building then use this instead.

[YOCTO #9206]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/orm/models.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index d451989..b89bfa9 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1515,10 +1515,13 @@ class CustomImageRecipe(Recipe):
                 packages_conf += pkg.name+' '
 
         packages_conf += "\""
-
-        base_recipe = open("%s/%s" %
-                           (self.base_recipe.layer_version.dirpath,
-                            self.base_recipe.file_path), 'r').read()
+        try:
+            base_recipe = open("%s/%s" %
+                               (self.base_recipe.layer_version.dirpath,
+                                self.base_recipe.file_path), 'r').read()
+        except IOError:
+            # The path may now be the full path if the recipe has been built
+            base_recipe = open(self.base_recipe.file_path, 'r').read()
 
         # Add a special case for when the recipe we have based a custom image
         # recipe on requires another recipe.
-- 
2.5.0

Re: [PATCH 0/5] localhostbecontroller fixes

[Brian Avery]

Hi,

There's an issue where I can't add or delete packages from my custom
image after initially creating it.
Steps to reproduce:
1) build core-image-minimal
2) New custom image based on core-image-minimal
3) I get a page like this:
https://www.dropbox.com/s/bigqrsu9d9awct9/Screenshot%202016-03-04%2017.07.00.png?dl=0
4) also, the download recipe file button gives me:
[Errno 2] No such file or directory:
u'recipes-core/images/core-image-minimal.bb'
5) Even if I build the custom image it still says:
Toaster has no package information for mwimage1. To generate package
information, build mwimage1
-b

On Fri, Mar 4, 2016 at 9:58 AM, Michael Wood <michael.g.wood@intel.com> wrote:
>
> Fixes for localhostbecontroller to allow file:/// uri for git repositories. Allowing the file:/// uri had an unintended consequence that the layer we generate for the custom image layer/recipes have had a dummy value of file:///_meta-toaster as its giturl so it tried and fails to git clone the dummy url. This fixes this and another issue identified in testing when building a custom image based on a custom image https://bugzilla.yoctoproject.org/show_bug.cgi?id=9206
>
> branch available on poky-contrib michaelw/toaster/localhostbctrl
>
>
> Michael Wood (5):
>   toaster: localhostbecontroller Don't clear out toaster custom layer
>     dir
>   toaster: orm Add a constant for the CustomImageRecipe's layer name
>   toaster: localhostbecontroller Allow file:/// uri type for git repo
>   toaster: localhostbecontroller put generated layer in the builddir
>   toaster: orm generate CustomImageRecipe contents try secondary path
>
>  bitbake/lib/bb/ui/buildinfohelper.py                    |  2 +-
>  bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 10 +++++-----
>  bitbake/lib/toaster/orm/models.py                       | 15 +++++++++++----
>  bitbake/lib/toaster/toastergui/views.py                 |  2 +-
>  4 files changed, 18 insertions(+), 11 deletions(-)
>
> --
> 2.5.0
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster

Re: [PATCH 0/5] localhostbecontroller fixes

[Barros Pena, Belen]

On 05/03/2016 01:13, "toaster-bounces@yoctoproject.org on behalf of Brian
Avery" <toaster-bounces@yoctoproject.org on behalf of
avery.brian@gmail.com> wrote:

>Hi,
>
>There's an issue where I can't add or delete packages from my custom
>image after initially creating it.

I see the same thing, and also something else: image builds don't seem to
be recognised as such. The "Images" section is missing from the left
navigation (so no packages installed or directory structure information),
and the image files are not shown. Screenshot at

http://imgur.com/Co0d8Wz

I guess both issues are somehow connected?

Cheers

Belén

>Steps to reproduce:
>1) build core-image-minimal
>2) New custom image based on core-image-minimal
>3) I get a page like this:
>https://www.dropbox.com/s/bigqrsu9d9awct9/Screenshot%202016-03-04%2017.07.
>00.png?dl=0
>4) also, the download recipe file button gives me:
>[Errno 2] No such file or directory:
>u'recipes-core/images/core-image-minimal.bb'
>5) Even if I build the custom image it still says:
>Toaster has no package information for mwimage1. To generate package
>information, build mwimage1
>-b
>
>On Fri, Mar 4, 2016 at 9:58 AM, Michael Wood <michael.g.wood@intel.com>
>wrote:
>>
>> Fixes for localhostbecontroller to allow file:/// uri for git
>>repositories. Allowing the file:/// uri had an unintended consequence
>>that the layer we generate for the custom image layer/recipes have had a
>>dummy value of file:///_meta-toaster as its giturl so it tried and fails
>>to git clone the dummy url. This fixes this and another issue identified
>>in testing when building a custom image based on a custom image
>>https://bugzilla.yoctoproject.org/show_bug.cgi?id=9206
>>
>> branch available on poky-contrib michaelw/toaster/localhostbctrl
>>
>>
>> Michael Wood (5):
>>   toaster: localhostbecontroller Don't clear out toaster custom layer
>>     dir
>>   toaster: orm Add a constant for the CustomImageRecipe's layer name
>>   toaster: localhostbecontroller Allow file:/// uri type for git repo
>>   toaster: localhostbecontroller put generated layer in the builddir
>>   toaster: orm generate CustomImageRecipe contents try secondary path
>>
>>  bitbake/lib/bb/ui/buildinfohelper.py                    |  2 +-
>>  bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 10 +++++-----
>>  bitbake/lib/toaster/orm/models.py                       | 15
>>+++++++++++----
>>  bitbake/lib/toaster/toastergui/views.py                 |  2 +-
>>  4 files changed, 18 insertions(+), 11 deletions(-)
>>
>> --
>> 2.5.0
>>
>> --
>> _______________________________________________
>> toaster mailing list
>> toaster@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/toaster
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

[PATCH v2 1/5] toaster: localhostbecontroller Don't clear out toaster custom layer dir

[Michael Wood]

We may have a recipe which is based on a custom image recipe that has
already been built so keep the recipe file around so that it can be read
by the generate recipe function.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index efd82c3..38e97a2 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -217,9 +217,9 @@ class LocalhostBEController(BuildEnvironmentController):
             raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
 
         # 5. create custom layer and add custom recipes to it
-        layerpath = os.path.join(self.be.sourcedir, "_meta-toaster-custom")
-        if os.path.isdir(layerpath):
-            shutil.rmtree(layerpath) # remove leftovers from previous builds
+        layerpath = os.path.join(self.be.sourcedir,
+                                 "_meta-toaster-custom_",
+                                 bitbake.req.project.name)
         for target in targets:
             try:
                 customrecipe = CustomImageRecipe.objects.get(name=target.target,
-- 
2.5.0

[PATCH v2 2/5] toaster: orm Add a constant for the CustomImageRecipe's layer name

[Michael Wood]

Use a constant to define the name for the toaster custom images layer
this constant is then used to identify this layer in various places

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py    | 4 ++--
 bitbake/lib/toaster/orm/models.py       | 4 ++++
 bitbake/lib/toaster/toastergui/views.py | 2 +-
 3 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 81abede..a1af080 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -42,7 +42,7 @@ from orm.models import Variable, VariableHistory
 from orm.models import Package, Package_File, Target_Installed_Package, Target_File
 from orm.models import Task_Dependency, Package_Dependency
 from orm.models import Recipe_Dependency, Provides
-from orm.models import Project, CustomImagePackage
+from orm.models import Project, CustomImagePackage, CustomImageRecipe
 
 from bldcontrol.models import BuildEnvironment, BuildRequest
 
@@ -326,7 +326,7 @@ class ORMWrapper(object):
             # Special case the toaster-custom-images layer which is created
             # on the fly so don't update the values which may cause the layer
             # to be duplicated on a future get_or_create
-            if layer_obj.layer.name == "toaster-custom-images":
+            if layer_obj.layer.name == CustomImageRecipe.LAYER_NAME:
                 return layer_obj
             # We already found our layer version for this build so just
             # update it with the new build information
diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 93b5df3..d451989 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1430,6 +1430,10 @@ class ProjectLayer(models.Model):
         unique_together = (("project", "layercommit"),)
 
 class CustomImageRecipe(Recipe):
+
+    # CustomImageRecipe's belong to layers called:
+    LAYER_NAME = "toaster-custom-images"
+
     search_allowed_fields = ['name']
     base_recipe = models.ForeignKey(Recipe, related_name='based_on_recipe')
     project = models.ForeignKey(Project)
diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py
index bd11892..de029ef 100755
--- a/bitbake/lib/toaster/toastergui/views.py
+++ b/bitbake/lib/toaster/toastergui/views.py
@@ -2392,7 +2392,7 @@ if True:
 
             # create layer 'Custom layer' and verion if needed
             layer = Layer.objects.get_or_create(
-                name="toaster-custom-images",
+                name=CustomImageRecipe.LAYER_NAME,
                 summary="Layer for custom recipes",
                 vcs_url="file:///toaster_created_layer")[0]
 
-- 
2.5.0

[PATCH v2 3/5] toaster: localhostbecontroller Allow file:/// uri type for git repo

[Michael Wood]

We don't need to skip file:/// uri type locations for git repositories.
If you're using a file:/// uri you should know that it has to be a local
path.

[YOCTO #9200]

igned-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 38e97a2..9deb572 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -128,8 +128,9 @@ class LocalhostBEController(BuildEnvironmentController):
         gitrepos[(bitbake.giturl, bitbake.commit)].append( ("bitbake", bitbake.dirpath) )
 
         for layer in layers:
-            # we don't process local URLs
-            if layer.giturl.startswith("file://"):
+            # We don't need to git clone the layer for the CustomImageRecipe
+            # as it's generated by us layer on if needed
+            if CustomImageRecipe.LAYER_NAME in layer.name:
                 continue
             if not (layer.giturl, layer.commit) in gitrepos:
                 gitrepos[(layer.giturl, layer.commit)] = []
-- 
2.5.0

[PATCH v2 4/5] toaster: localhostbecontroller put generated layer in the builddir

[Michael Wood]

Move the generated layer for custom recipes into the build directory.
The build directory makes more sense as this layer/recipe is generated
for a particular build/project.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 9deb572..08419f2 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -218,9 +218,8 @@ class LocalhostBEController(BuildEnvironmentController):
             raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
 
         # 5. create custom layer and add custom recipes to it
-        layerpath = os.path.join(self.be.sourcedir,
-                                 "_meta-toaster-custom_",
-                                 bitbake.req.project.name)
+        layerpath = os.path.join(self.be.builddir,
+                                 CustomImageRecipe.LAYER_NAME)
         for target in targets:
             try:
                 customrecipe = CustomImageRecipe.objects.get(name=target.target,
-- 
2.5.0

[PATCH v2 5/5] toaster: orm generate CustomImageRecipe contents try secondary path

[Michael Wood]

Try a secondary file path if the first does not exist. When we get the
recipe paths and layer information from the layer index it is not a
complete path but we are usually able to reconstruct it. If the complete
real path has been discovered by building then use this instead.

[YOCTO #9206]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/orm/models.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index d451989..b89bfa9 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1515,10 +1515,13 @@ class CustomImageRecipe(Recipe):
                 packages_conf += pkg.name+' '
 
         packages_conf += "\""
-
-        base_recipe = open("%s/%s" %
-                           (self.base_recipe.layer_version.dirpath,
-                            self.base_recipe.file_path), 'r').read()
+        try:
+            base_recipe = open("%s/%s" %
+                               (self.base_recipe.layer_version.dirpath,
+                                self.base_recipe.file_path), 'r').read()
+        except IOError:
+            # The path may now be the full path if the recipe has been built
+            base_recipe = open(self.base_recipe.file_path, 'r').read()
 
         # Add a special case for when the recipe we have based a custom image
         # recipe on requires another recipe.
-- 
2.5.0

Re: [PATCH 0/5] localhostbecontroller fixes

[Michael Wood]

I can't recreate the download error, there is a known issue if you're 
never git cloned a non poky repo where this might happen, but if you've 
built it it should be findable.

The no packages one is because I left off an import in the 
buildinfohelper in [PATCH 2/5] toaster: orm Add a constant for the 
CustomImageRecipe's layer name.

Will send a v2 series.

Michael


On 07/03/16 11:42, Barros Pena, Belen wrote:
>
> On 05/03/2016 01:13, "toaster-bounces@yoctoproject.org on behalf of Brian
> Avery" <toaster-bounces@yoctoproject.org on behalf of
> avery.brian@gmail.com> wrote:
>
>> Hi,
>>
>> There's an issue where I can't add or delete packages from my custom
>> image after initially creating it.
> I see the same thing, and also something else: image builds don't seem to
> be recognised as such. The "Images" section is missing from the left
> navigation (so no packages installed or directory structure information),
> and the image files are not shown. Screenshot at
>
> http://imgur.com/Co0d8Wz
>
> I guess both issues are somehow connected?
>
> Cheers
>
> Belén
>
>> Steps to reproduce:
>> 1) build core-image-minimal
>> 2) New custom image based on core-image-minimal
>> 3) I get a page like this:
>> https://www.dropbox.com/s/bigqrsu9d9awct9/Screenshot%202016-03-04%2017.07.
>> 00.png?dl=0
>> 4) also, the download recipe file button gives me:
>> [Errno 2] No such file or directory:
>> u'recipes-core/images/core-image-minimal.bb'
>> 5) Even if I build the custom image it still says:
>> Toaster has no package information for mwimage1. To generate package
>> information, build mwimage1
>> -b
>>
>> On Fri, Mar 4, 2016 at 9:58 AM, Michael Wood <michael.g.wood@intel.com>
>> wrote:
>>> Fixes for localhostbecontroller to allow file:/// uri for git
>>> repositories. Allowing the file:/// uri had an unintended consequence
>>> that the layer we generate for the custom image layer/recipes have had a
>>> dummy value of file:///_meta-toaster as its giturl so it tried and fails
>>> to git clone the dummy url. This fixes this and another issue identified
>>> in testing when building a custom image based on a custom image
>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=9206
>>>
>>> branch available on poky-contrib michaelw/toaster/localhostbctrl
>>>
>>>
>>> Michael Wood (5):
>>>    toaster: localhostbecontroller Don't clear out toaster custom layer
>>>      dir
>>>    toaster: orm Add a constant for the CustomImageRecipe's layer name
>>>    toaster: localhostbecontroller Allow file:/// uri type for git repo
>>>    toaster: localhostbecontroller put generated layer in the builddir
>>>    toaster: orm generate CustomImageRecipe contents try secondary path
>>>
>>>   bitbake/lib/bb/ui/buildinfohelper.py                    |  2 +-
>>>   bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 10 +++++-----
>>>   bitbake/lib/toaster/orm/models.py                       | 15
>>> +++++++++++----
>>>   bitbake/lib/toaster/toastergui/views.py                 |  2 +-
>>>   4 files changed, 18 insertions(+), 11 deletions(-)
>>>
>>> --
>>> 2.5.0
>>>
>>> --
>>> _______________________________________________
>>> toaster mailing list
>>> toaster@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/toaster
>> -- 
>> _______________________________________________
>> toaster mailing list
>> toaster@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/toaster

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [PATCH v2 3/5] toaster: localhostbecontroller Allow file:/// uri type for git repo

[Barros Pena, Belen]

On 07/03/2016 16:03, "toaster-bounces@yoctoproject.org on behalf of
Michael Wood" <toaster-bounces@yoctoproject.org on behalf of
michael.g.wood@intel.com> wrote:

>We don't need to skip file:/// uri type locations for git repositories.
>If you're using a file:/// uri you should know that it has to be a local
>path.
>
>[YOCTO #9200]

This seems to work for me. I can import and build layers using both
file:/// and without.

Cheers

Belén

>
>igned-off-by: Michael Wood <michael.g.wood@intel.com>
>---
> bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
>diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
>b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
>index 38e97a2..9deb572 100644
>--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
>+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
>@@ -128,8 +128,9 @@ class
>LocalhostBEController(BuildEnvironmentController):
>         gitrepos[(bitbake.giturl, bitbake.commit)].append( ("bitbake",
>bitbake.dirpath) )
> 
>         for layer in layers:
>-            # we don't process local URLs
>-            if layer.giturl.startswith("file://"):
>+            # We don't need to git clone the layer for the
>CustomImageRecipe
>+            # as it's generated by us layer on if needed
>+            if CustomImageRecipe.LAYER_NAME in layer.name:
>                 continue
>             if not (layer.giturl, layer.commit) in gitrepos:
>                 gitrepos[(layer.giturl, layer.commit)] = []
>-- 
>2.5.0
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: [PATCH v2 5/5] toaster: orm generate CustomImageRecipe contents try secondary path

[Barros Pena, Belen]

[-- Attachment #1: Type: text/plain, Size: 3179 bytes --]



On 07/03/2016 16:03, "toaster-bounces@yoctoproject.org on behalf of
Michael Wood" <toaster-bounces@yoctoproject.org on behalf of
michael.g.wood@intel.com> wrote:

>Try a secondary file path if the first does not exist. When we get the
>recipe paths and layer information from the layer index it is not a
>complete path but we are usually able to reconstruct it. If the complete
>real path has been discovered by building then use this instead.
>
>[YOCTO #9206]

I think this is working... I think. This is what I did:

1. Create a custom image called morse-image-1 based on core-image-minimal
and add a package called 'morseapp' from my local imported layer. Build.
This image reported 29 packages.

2. Create a custom image based on morse-image-1 called morse-image-2 and
add a package called 'hello' from my local imported layer. Build. This
image reported 30 packages.

3. Create a custom image based morse-image-2 called morse-image-3 and add
a package called 'hello-mod' from my local imported layer. Build. This
image reported 32 packages.

After doing all this, I went to the 'custom images' page, and there
Toaster says that morse-image-1 has 30 packages (instead of the initial
29). But, if you actually go to the details page for morse-image-1, and
apply the package included filter, it shows and displays 29 packages :/ I
have seen this before, but only on Jethro builds, never on master ones.

The other thing is about the content of the bb files: the "Customisation
Generated by Toaster on" section, which includes SUMMARY, DESCRIPTION,
LICENCE and IMAGE_INSTALL_append, is repeated. It's hard to explain, so
I've attached the files. Maybe this is harmless, but I thought I'd let you
know.

Cheers

Belén


>
>Signed-off-by: Michael Wood <michael.g.wood@intel.com>
>---
> bitbake/lib/toaster/orm/models.py | 11 +++++++----
> 1 file changed, 7 insertions(+), 4 deletions(-)
>
>diff --git a/bitbake/lib/toaster/orm/models.py
>b/bitbake/lib/toaster/orm/models.py
>index d451989..b89bfa9 100644
>--- a/bitbake/lib/toaster/orm/models.py
>+++ b/bitbake/lib/toaster/orm/models.py
>@@ -1515,10 +1515,13 @@ class CustomImageRecipe(Recipe):
>                 packages_conf += pkg.name+' '
> 
>         packages_conf += "\""
>-
>-        base_recipe = open("%s/%s" %
>-                           (self.base_recipe.layer_version.dirpath,
>-                            self.base_recipe.file_path), 'r').read()
>+        try:
>+            base_recipe = open("%s/%s" %
>+                               (self.base_recipe.layer_version.dirpath,
>+                                self.base_recipe.file_path), 'r').read()
>+        except IOError:
>+            # The path may now be the full path if the recipe has been
>built
>+            base_recipe = open(self.base_recipe.file_path, 'r').read()
> 
>         # Add a special case for when the recipe we have based a custom
>image
>         # recipe on requires another recipe.
>-- 
>2.5.0
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster


[-- Attachment #2: morse-image-3_0.1.bb --]
[-- Type: application/octet-stream, Size: 965 bytes --]

# Original recipe morse-image-2 
# Original recipe morse-image-1 
# Original recipe core-image-minimal 
SUMMARY = "A small image just capable of allowing a device to boot."

IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP} ${CORE_IMAGE_EXTRA_INSTALL}"

IMAGE_LINGUAS = " "

LICENSE = "MIT"

inherit core-image

IMAGE_ROOTFS_SIZE ?= "8192"
IMAGE_ROOTFS_EXTRA_SPACE_append = "${@bb.utils.contains("DISTRO_FEATURES", "systemd", " + 4096", "" ,d)}"


# Recipe morse-image-1 
# Customisation Generated by Toaster on 2016-03-08 17:24:11
SUMMARY = ""
DESCRIPTION = ""
LICENSE = "MIT"
IMAGE_INSTALL_append = " morseapp "

# Recipe morse-image-2 
# Customisation Generated by Toaster on 2016-03-08 17:37:31
SUMMARY = ""
DESCRIPTION = ""
LICENSE = "MIT"
IMAGE_INSTALL_append = " hello "

# Recipe morse-image-3 
# Customisation Generated by Toaster on 2016-03-08 17:59:12
SUMMARY = ""
DESCRIPTION = ""
LICENSE = "MIT"
IMAGE_INSTALL_append = " hello-mod "

[-- Attachment #3: morse-image-2_0.1.bb --]
[-- Type: application/octet-stream, Size: 764 bytes --]

# Original recipe morse-image-1 
# Original recipe core-image-minimal 
SUMMARY = "A small image just capable of allowing a device to boot."

IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP} ${CORE_IMAGE_EXTRA_INSTALL}"

IMAGE_LINGUAS = " "

LICENSE = "MIT"

inherit core-image

IMAGE_ROOTFS_SIZE ?= "8192"
IMAGE_ROOTFS_EXTRA_SPACE_append = "${@bb.utils.contains("DISTRO_FEATURES", "systemd", " + 4096", "" ,d)}"


# Recipe morse-image-1 
# Customisation Generated by Toaster on 2016-03-08 17:24:11
SUMMARY = ""
DESCRIPTION = ""
LICENSE = "MIT"
IMAGE_INSTALL_append = " morseapp "

# Recipe morse-image-2 
# Customisation Generated by Toaster on 2016-03-08 17:59:04
SUMMARY = ""
DESCRIPTION = ""
LICENSE = "MIT"
IMAGE_INSTALL_append = " hello "

[-- Attachment #4: morse-image-1_0.1.bb --]
[-- Type: application/octet-stream, Size: 567 bytes --]

# Original recipe core-image-minimal 
SUMMARY = "A small image just capable of allowing a device to boot."

IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP} ${CORE_IMAGE_EXTRA_INSTALL}"

IMAGE_LINGUAS = " "

LICENSE = "MIT"

inherit core-image

IMAGE_ROOTFS_SIZE ?= "8192"
IMAGE_ROOTFS_EXTRA_SPACE_append = "${@bb.utils.contains("DISTRO_FEATURES", "systemd", " + 4096", "" ,d)}"


# Recipe morse-image-1 
# Customisation Generated by Toaster on 2016-03-08 17:58:55
SUMMARY = ""
DESCRIPTION = ""
LICENSE = "MIT"
IMAGE_INSTALL_append = " morseapp "