[PATCH 1/1] toaster: protect against circular Layer dependencies

[PATCH 1/1] toaster: protect against circular Layer dependencies

[Reyna, David]

Hi all,

I found this issue while testing some new Wind River layers, where they were required to be included together which meant that they had an intentional circular dependency.

I think that there are no current layers in the OE Layer Index with a circular dependency, nor can we create a custom one since the current Toaster UI does not allow that. However, these will be coming so I want to preempt that problem upstream with this patch.

  http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=dreyna%2Fcircular_dependencies_10631

- David 

===========

From fdc3e055eb2fcad098711f03d7ca1798f926e5ae Mon Sep 17 00:00:00 2001
From: David Reyna <David.Reyna@windriver.com>
Date: Tue, 15 Nov 2016 21:36:25 -0800
Subject: [PATCH] toaster: protect circular dependencies

Limit the recursion (to say 20 levels) when processing layer dependencies
so that circular dependecies do not cause infinite decent and an
out-of-memory failure. The duplicate found layers are already immediately
filtered in the code.

[YOCTO #10630]

Signed-off-by: David Reyna <David.Reyna@windriver.com>
---
 bitbake/lib/toaster/orm/models.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 4f8510c..061d84e 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1478,17 +1478,21 @@ class Layer_Version(models.Model):
 
     def get_alldeps(self, project_id):
         """Get full list of unique layer dependencies."""
-        def gen_layerdeps(lver, project):
+        def gen_layerdeps(lver, project, depth):
+            if depth==0:
+                return
             for ldep in lver.dependencies.all():
                 yield ldep.depends_on
                 # get next level of deps recursively calling gen_layerdeps
-                for subdep in gen_layerdeps(ldep.depends_on, project):
+                for subdep in gen_layerdeps(ldep.depends_on, project,depth-1):
                     yield subdep
 
         project = Project.objects.get(pk=project_id)
         result = []
         projectlvers = [player.layercommit for player in project.projectlayer_set.all()]
-        for dep in gen_layerdeps(self, project):
+        # protect against infinite layer dependency loops
+        maxdepth=20
+        for dep in gen_layerdeps(self, project, maxdepth):
             # filter out duplicates and layers already belonging to the project
             if dep not in result + projectlvers:
                 result.append(dep)
-- 
1.9.1

Re: [PATCH 1/1] toaster: protect against circular Layer dependencies

[Michael Wood]

On 16/11/16 05:55, Reyna, David wrote:
> Hi all,
>
> I found this issue while testing some new Wind River layers, where they were required to be included together which meant that they had an intentional circular dependency.
>
> I think that there are no current layers in the OE Layer Index with a circular dependency, nor can we create a custom one since the current Toaster UI does not allow that. However, these will be coming so I want to preempt that problem upstream with this patch.
>
>    http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=dreyna%2Fcircular_dependencies_10631
>
> - David
>
> ===========
>
>  From fdc3e055eb2fcad098711f03d7ca1798f926e5ae Mon Sep 17 00:00:00 2001
> From: David Reyna <David.Reyna@windriver.com>
> Date: Tue, 15 Nov 2016 21:36:25 -0800
> Subject: [PATCH] toaster: protect circular dependencies

In the patch subject it's helpful to have a clue as to the module which 
is being affected such as

"toaster: orm gen_layerdeps Protect against circular dependencies"

I'll add this to the contribution guide as I really find it helpful when 
scanning down the git log to see what things have changed and where.

>
> Limit the recursion (to say 20 levels) when processing layer dependencies
> so that circular dependecies do not cause infinite decent and an
> out-of-memory failure. The duplicate found layers are already immediately
> filtered in the code.
>
> [YOCTO #10630]
>
> Signed-off-by: David Reyna <David.Reyna@windriver.com>
> ---
>   bitbake/lib/toaster/orm/models.py | 10 +++++++---
>   1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
> index 4f8510c..061d84e 100644
> --- a/bitbake/lib/toaster/orm/models.py
> +++ b/bitbake/lib/toaster/orm/models.py
> @@ -1478,17 +1478,21 @@ class Layer_Version(models.Model):
>   
>       def get_alldeps(self, project_id):
>           """Get full list of unique layer dependencies."""
> -        def gen_layerdeps(lver, project):
> +        def gen_layerdeps(lver, project, depth):
> +            if depth==0:

Just for future reference make sure that you have white space around the 
operators
"if depth == 0" and not "depth==0" as this generates a warning (E225)
You can check for these using a lint tool such as flake8 (most good 
editors have plugins for this)

I'll fix this on submission upstream. No need to send another revision.

> +                return
>               for ldep in lver.dependencies.all():
>                   yield ldep.depends_on
>                   # get next level of deps recursively calling gen_layerdeps
> -                for subdep in gen_layerdeps(ldep.depends_on, project):
> +                for subdep in gen_layerdeps(ldep.depends_on, project,depth-1):
>                       yield subdep
>   
>           project = Project.objects.get(pk=project_id)
>           result = []
>           projectlvers = [player.layercommit for player in project.projectlayer_set.all()]
> -        for dep in gen_layerdeps(self, project):
> +        # protect against infinite layer dependency loops
> +        maxdepth=20
> +        for dep in gen_layerdeps(self, project, maxdepth):
>               # filter out duplicates and layers already belonging to the project
>               if dep not in result + projectlvers:
>                   result.append(dep)