[PATCH 0/5] michaelw/toaster/tables-misc

[PATCH 0/5] michaelw/toaster/tables-misc

[Michael Wood]

A number of fixes for toaster tables identifed by writing a generic unit test
for this widget.
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=michaelw/toaster/tables-misc

Michael Wood (5):
  toaster: machines table Fix missing layers information needed for
    filter
  toaster: CustomImageRecipe add search_allowed_fields to this model
  toaster: tables Add default_orderby field where it was missing or
    unset
  toaster: tables Fix invalid field name on NewCustomImagesTable
  toaster: toastergui tests Add generic test for ToasterTables widget

 bitbake/lib/toaster/orm/models.py                  |   1 +
 bitbake/lib/toaster/toastergui/tables.py           |  18 +-
 .../toaster/toastergui/templates/machine_btn.html  |  12 +-
 bitbake/lib/toaster/toastergui/tests.py            | 223 ++++++++++++++++++++-
 4 files changed, 235 insertions(+), 19 deletions(-)

-- 
2.5.0

[PATCH 1/5] toaster: machines table Fix missing layers information needed for filter

[Michael Wood]

The current layers information wasn't being passed to the template for
the Select/Add button for the Compatible machines filter.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/toastergui/tables.py                  |  6 ++++--
 bitbake/lib/toaster/toastergui/templates/machine_btn.html | 12 ++++++++++--
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/toaster/toastergui/tables.py b/bitbake/lib/toaster/toastergui/tables.py
index 9c9cda4..be8916c 100644
--- a/bitbake/lib/toaster/toastergui/tables.py
+++ b/bitbake/lib/toaster/toastergui/tables.py
@@ -217,12 +217,10 @@ class MachinesTable(ToasterTable, ProjectFiltersMixin):
     def get_context_data(self, **kwargs):
         context = super(MachinesTable, self).get_context_data(**kwargs)
         context['project'] = Project.objects.get(pk=kwargs['pid'])
-        context['projectlayers'] = map(lambda prjlayer: prjlayer.layercommit.id, ProjectLayer.objects.filter(project=context['project']))
         return context
 
     def setup_filters(self, *args, **kwargs):
         project = Project.objects.get(pk=kwargs['pid'])
-        self.project_layers = project.get_project_layer_versions()
 
         self.add_filter(title="Filter by project machines",
                         name="in_current_project",
@@ -236,6 +234,10 @@ class MachinesTable(ToasterTable, ProjectFiltersMixin):
         self.queryset = prj.get_all_compatible_machines()
         self.queryset = self.queryset.order_by(self.default_orderby)
 
+        self.static_context_extra['current_layers'] = \
+                self.project_layers = \
+                prj.get_project_layer_versions(pk=True)
+
     def setup_columns(self, *args, **kwargs):
 
         self.add_column(title="Machine",
diff --git a/bitbake/lib/toaster/toastergui/templates/machine_btn.html b/bitbake/lib/toaster/toastergui/templates/machine_btn.html
index d2cb55b..7b08f6a9a6 100644
--- a/bitbake/lib/toaster/toastergui/templates/machine_btn.html
+++ b/bitbake/lib/toaster/toastergui/templates/machine_btn.html
@@ -1,6 +1,14 @@
-<a href="{% url 'project' extra.pid %}?setMachine={{data.name}}" class="btn btn-block layer-exists-{{data.layer_version.id}}" style="margin-top: 5px; display:none">
+<a href="{% url 'project' extra.pid %}?setMachine={{data.name}}" class="btn btn-block layer-exists-{{data.layer_version.id}}"
+    {% if data.layer_version.pk not in extra.current_layers %}
+    style="display:none;"
+    {% endif %}
+>
   Select machine</a>
-<button class="btn btn-block layerbtn layer-add-{{data.layer_version.id}}" data-layer='{ "id": {{data.layer_version.id}}, "name":  "{{data.layer_version.layer.name}}", "layerdetailurl": "{%url 'layerdetails' extra.pid data.layer_version.id %}"}' data-directive="add">
+<button class="btn btn-block layerbtn layer-add-{{data.layer_version.id}}" data-layer='{ "id": {{data.layer_version.id}}, "name":  "{{data.layer_version.layer.name}}", "layerdetailurl": "{%url 'layerdetails' extra.pid data.layer_version.id %}"}' data-directive="add"
+    {% if data.layer_version.pk in extra.current_layers %}
+    style="display:none;"
+    {% endif %}
+>
   <i class="icon-plus"></i>
   Add layer
   <i title="" class="icon-question-sign get-help" data-original-title="To enable this machine, you must first add the {{data.layer_version.layer.name}} layer to your project"></i>
-- 
2.5.0

[PATCH 2/5] toaster: CustomImageRecipe add search_allowed_fields to this model

[Michael Wood]

In order to search the model from the UI some fields must be nominated
as searchable.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/orm/models.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 0ac94b9..0174233 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1238,6 +1238,7 @@ class ProjectLayer(models.Model):
         unique_together = (("project", "layercommit"),)
 
 class CustomImageRecipe(models.Model):
+    search_allowed_fields = ['name']
     name = models.CharField(max_length=100)
     base_recipe = models.ForeignKey(Recipe)
     packages = models.ManyToManyField(Package)
-- 
2.5.0

[PATCH 3/5] toaster: tables Add default_orderby field where it was missing or unset

[Michael Wood]

This value is used to set the default ordering of the model that is used
for ToasterTables, it is picked up client side to set the ordering
indicator.

[YOCTO #8695]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/toastergui/tables.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/toaster/toastergui/tables.py b/bitbake/lib/toaster/toastergui/tables.py
index be8916c..44a89d9 100644
--- a/bitbake/lib/toaster/toastergui/tables.py
+++ b/bitbake/lib/toaster/toastergui/tables.py
@@ -295,6 +295,7 @@ class LayerMachinesTable(MachinesTable):
         MachinesTable.setup_queryset(self, *args, **kwargs)
 
         self.queryset = self.queryset.filter(layer_version__pk=int(kwargs['layerid']))
+        self.queryset = self.queryset.order_by(self.default_orderby)
         self.static_context_extra['in_prj'] = ProjectLayer.objects.filter(Q(project=kwargs['pid']) & Q(layercommit=kwargs['layerid'])).count()
 
     def setup_columns(self, *args, **kwargs):
@@ -319,7 +320,6 @@ class RecipesTable(ToasterTable, ProjectFiltersMixin):
     def __init__(self, *args, **kwargs):
         super(RecipesTable, self).__init__(*args, **kwargs)
         self.empty_state = "Toaster has no recipe information. To generate recipe information you can configure a layer source then run a build."
-        self.default_orderby = "name"
 
     build_col = { 'title' : "Build",
             'help_text' : "Add or delete recipes to and from your project",
@@ -356,7 +356,6 @@ class RecipesTable(ToasterTable, ProjectFiltersMixin):
         self.static_context_extra['current_layers'] = self.project_layers
 
         self.queryset = prj.get_all_compatible_recipes()
-        self.queryset = self.queryset.order_by(self.default_orderby)
 
 
     def setup_columns(self, *args, **kwargs):
@@ -414,6 +413,7 @@ class LayerRecipesTable(RecipesTable):
 
     def __init__(self, *args, **kwargs):
         super(LayerRecipesTable, self).__init__(*args, **kwargs)
+        self.default_orderby = "name"
 
     def get_context_data(self, **kwargs):
         context = super(LayerRecipesTable, self).get_context_data(**kwargs)
@@ -425,6 +425,7 @@ class LayerRecipesTable(RecipesTable):
         self.queryset = \
                 Recipe.objects.filter(layer_version__pk=int(kwargs['layerid']))
 
+        self.queryset = self.queryset.order_by(self.default_orderby)
         self.static_context_extra['in_prj'] = ProjectLayer.objects.filter(Q(project=kwargs['pid']) & Q(layercommit=kwargs['layerid'])).count()
 
     def setup_columns(self, *args, **kwargs):
@@ -451,6 +452,7 @@ class CustomImagesTable(ToasterTable):
     def __init__(self, *args, **kwargs):
         super(CustomImagesTable, self).__init__(*args, **kwargs)
         self.title = "Custom images"
+        self.default_orderby = "name"
 
     def get_context_data(self, **kwargs):
         context = super(CustomImagesTable, self).get_context_data(**kwargs)
@@ -462,7 +464,7 @@ class CustomImagesTable(ToasterTable):
     def setup_queryset(self, *args, **kwargs):
         prj = Project.objects.get(pk = kwargs['pid'])
         self.queryset = CustomImageRecipe.objects.filter(project=prj)
-        self.queryset = self.queryset.order_by('name')
+        self.queryset = self.queryset.order_by(self.default_orderby)
 
     def setup_columns(self, *args, **kwargs):
 
@@ -502,11 +504,13 @@ class ImageRecipesTable(RecipesTable):
     def __init__(self, *args, **kwargs):
         super(ImageRecipesTable, self).__init__(*args, **kwargs)
         self.title = "Compatible image recipes"
+        self.default_orderby = "name"
 
     def setup_queryset(self, *args, **kwargs):
         super(ImageRecipesTable, self).setup_queryset(*args, **kwargs)
 
         self.queryset = self.queryset.filter(is_image=True)
+        self.queryset = self.queryset.order_by(self.default_orderby)
 
 
     def setup_columns(self, *args, **kwargs):
-- 
2.5.0

[PATCH 4/5] toaster: tables Fix invalid field name on NewCustomImagesTable

[Michael Wood]

Correct the field name in the NewCustomImagesTable as it is a Recipe
object it's self and not a container.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/toastergui/tables.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/toastergui/tables.py b/bitbake/lib/toaster/toastergui/tables.py
index 44a89d9..74af507 100644
--- a/bitbake/lib/toaster/toastergui/tables.py
+++ b/bitbake/lib/toaster/toastergui/tables.py
@@ -545,7 +545,7 @@ class NewCustomImagesTable(ImageRecipesTable):
                                   "deploy to a machine",
                         hideable=False,
                         orderable=True,
-                        field_name="recipe__name")
+                        field_name="name")
 
         super(ImageRecipesTable, self).setup_columns(*args, **kwargs)
 
-- 
2.5.0

[PATCH 5/5] toaster: toastergui tests Add generic test for ToasterTables widget

[Michael Wood]

For each of the ToasterTables defined test:
 - Data returned
 - Search
 - Order by
 - Limit
 - Paginate
 - Filter

These tests test that the functions on all tables work, it does not
validate the content of the data in the tables themselves, this needs
to be done in table specific tests.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/toastergui/tests.py | 223 ++++++++++++++++++++++++++++++--
 1 file changed, 212 insertions(+), 11 deletions(-)

diff --git a/bitbake/lib/toaster/toastergui/tests.py b/bitbake/lib/toaster/toastergui/tests.py
index 9e6c46a..a631203 100644
--- a/bitbake/lib/toaster/toastergui/tests.py
+++ b/bitbake/lib/toaster/toastergui/tests.py
@@ -33,11 +33,14 @@ from orm.models import CustomImageRecipe, ProjectVariable
 from orm.models import Branch
 
 import toastermain
+import inspect
+import toastergui
 
 from toastergui.tables import SoftwareRecipesTable
 import json
 from bs4 import BeautifulSoup
 import re
+import string
 
 PROJECT_NAME = "test project"
 CLI_BUILDS_PROJECT_NAME = 'Command line builds'
@@ -69,32 +72,68 @@ class ViewTests(TestCase):
         layer = Layer.objects.create(name="base-layer", layer_source=layersrc,
                                      vcs_url="/tmp/")
 
+        layer_two = Layer.objects.create(name="z-layer",
+                                         layer_source=layersrc,
+                                         vcs_url="git://two/")
+
+
         branch = Branch.objects.create(name="master", layer_source=layersrc)
 
-        lver = Layer_Version.objects.create(layer=layer, project=self.project,
-                                            layer_source=layersrc, commit="master",
-                                            up_branch=branch)
+        self.lver = Layer_Version.objects.create(layer=layer,
+                                                 project=self.project,
+                                                 layer_source=layersrc,
+                                                 commit="master",
+                                                 up_branch=branch)
+
+        lver_two = Layer_Version.objects.create(layer=layer_two,
+                                                layer_source=layersrc,
+                                                commit="master",
+                                                up_branch=branch)
+
+        Recipe.objects.create(layer_source=layersrc,
+                              name="z recipe",
+                              version="5.2",
+                              summary="z recipe",
+                              description="G recipe",
+                              license="Z GPL",
+                              section="h section",
+                              layer_version=lver_two)
 
         self.recipe1 = Recipe.objects.create(layer_source=layersrc,
                                              name="base-recipe",
                                              version="1.2",
                                              summary="one recipe",
                                              description="recipe",
-                                             layer_version=lver)
+                                             section="A section",
+                                             license="Apache",
+                                             layer_version=self.lver)
 
-        Machine.objects.create(layer_version=lver, name="wisk",
+        Machine.objects.create(layer_version=self.lver, name="wisk",
                                description="wisking machine")
+        Machine.objects.create(layer_version=self.lver, name="zap",
+                               description="zap machine")
+        Machine.objects.create(layer_version=lver_two, name="xray",
+                               description="xray machine")
+
 
-        ProjectLayer.objects.create(project=self.project, layercommit=lver)
+
+        ProjectLayer.objects.create(project=self.project, layercommit=self.lver)
 
 
         self.customr = CustomImageRecipe.objects.create(\
                            name="custom recipe", project=self.project,
                            base_recipe=self.recipe1)
 
-        self.package = Package.objects.create(name='pkg1', recipe=self.recipe1,
+        CustomImageRecipe.objects.create(name="z custom recipe",
+                                         project=self.project,
+                                         base_recipe=self.recipe1)
+
+        self.package = Package.objects.create(name='pkg1',
+                                              size=999,
+                                              recipe=self.recipe1,
                                               build=build)
 
+        Package.objects.create(name='zpkg1', recipe=self.recipe1, build=build)
 
         # recipe with project for testing AvailableRecipe table
         self.recipe2 = Recipe.objects.create(layer_source=layersrc,
@@ -102,10 +141,35 @@ class ViewTests(TestCase):
                                              version="1.4",
                                              summary="a fancy recipe",
                                              description="fancy recipe",
-                                             layer_version=lver,
+                                             license="MIT",
+                                             layer_version=self.lver,
+                                             section="Z section",
                                              file_path='/home/foo')
 
-        self.assertTrue(lver in self.project.compatible_layerversions())
+        Recipe.objects.create(layer_source=layersrc,
+                              is_image=True,
+                              name="Test image one",
+                              version="1.2",
+                              summary="one recipe",
+                              description="recipe",
+                              section="A",
+                              license="A",
+                              file_path="/one/",
+                              layer_version=self.lver)
+
+        Recipe.objects.create(layer_source=layersrc,
+                              is_image=True,
+                              name="Z Test image two",
+                              version="1.3",
+                              summary="two image recipe",
+                              description="recipe two",
+                              section="B",
+                              license="Z",
+                              file_path="/two/",
+                              layer_version=lver_two)
+
+
+
 
     def test_get_base_call_returns_html(self):
         """Basic test for all-projects view"""
@@ -181,7 +245,6 @@ class ViewTests(TestCase):
 
             return False
 
-        import string
 
         for url in urls:
             results = False
@@ -344,7 +407,7 @@ class ViewTests(TestCase):
         row2 = next(x for x in rows if x['name'] == self.recipe2.name)
 
         self.assertEqual(response.status_code, 200, 'should be 200 OK status')
-        self.assertEqual(len(rows), 2, 'should be 2 recipes')
+        self.assertTrue(row2, 'should be 2 recipes')
 
         # check other columns have been populated correctly
         self.assertEqual(row1['name'], self.recipe1.name)
@@ -360,6 +423,144 @@ class ViewTests(TestCase):
         self.assertEqual(row2['layer_version__layer__name'],
                          self.recipe2.layer_version.layer.name)
 
+    def test_toaster_tables(self):
+        """Test all ToasterTables instances"""
+
+        def get_data(table, options={}):
+            """Send a request and parse the json response"""
+            options['format'] = "json"
+            options['nocache'] = "true"
+            request = RequestFactory().get('/', options)
+            # Add any kwargs that are needed by any of the possible tables
+            response = table.get(request,
+                                 pid=self.project.id,
+                                 layerid=self.lver.pk,
+                                 recipeid=self.recipe1.pk)
+            return json.loads(response.content)
+
+        # Get a list of classes in tables module
+        tables = inspect.getmembers(toastergui.tables, inspect.isclass)
+
+        for name, table_cls in tables:
+            # Filter out the non ToasterTables from the tables module
+            if not issubclass(table_cls, toastergui.widgets.ToasterTable) or \
+                table_cls == toastergui.widgets.ToasterTable:
+                continue
+
+            # Get the table data without any options, this also does the
+            # initialisation of the table i.e. setup_columns,
+            # setup_filters and setup_queryset that we can use later
+            table = table_cls()
+            all_data = get_data(table)
+
+            self.assertTrue(len(all_data['rows']) > 1,
+                            "Cannot test on a table with < 1 row")
+
+            if table.default_orderby:
+                row_one = all_data['rows'][0][table.default_orderby.strip("-")]
+                row_two = all_data['rows'][1][table.default_orderby.strip("-")]
+
+                if '-' in table.default_orderby:
+                    self.assertTrue(row_one >= row_two,
+                                    "Default ordering not working on %s" % name)
+                else:
+                    self.assertTrue(row_one <= row_two,
+                                    "Default ordering not working on %s" % name)
+
+            # Test the column ordering and filtering functionality
+            for column in table.columns:
+                if column['orderable']:
+                    # If a column is orderable test it in both order
+                    # directions ordering on the columns field_name
+                    ascending = get_data(table_cls(),
+                                         {"orderby" : column['field_name']})
+
+                    row_one = ascending['rows'][0][column['field_name']]
+                    row_two = ascending['rows'][1][column['field_name']]
+
+                    self.assertTrue(row_one <= row_two,
+                                    "Ascending sort applied but row 0 is less "
+                                    "than row 1")
+
+                    descending = get_data(table_cls(),
+                                          {"orderby" :
+                                           '-'+column['field_name']})
+
+                    row_one = descending['rows'][0][column['field_name']]
+                    row_two = descending['rows'][1][column['field_name']]
+
+                    self.assertTrue(row_one >= row_two,
+                                    "Descending sort applied but row 0 is "
+                                    "greater than row 1")
+
+                    # If the two start rows are the same we haven't actually
+                    # changed the order
+                    self.assertNotEqual(ascending['rows'][0],
+                                        descending['rows'][0],
+                                        "An orderby %s has not changed the "
+                                        "order of the data in table %s" %
+                                        (column['field_name'], name))
+
+                if column['filter_name']:
+                    # If a filter is available for the column get the filter
+                    # info. This contains what filter actions are defined.
+                    filter_info = get_data(table_cls(),
+                                           {"cmd": "filterinfo",
+                                            "name": column['filter_name']})
+                    self.assertTrue(len(filter_info['filter_actions']) > 0,
+                                    "Filter %s was defined but no actions "
+                                    "added to it" % column['filter_name'])
+
+                    for filter_action in filter_info['filter_actions']:
+                        # filter string to pass as the option
+                        # This is the name of the filter:action
+                        # e.g. project_filter:not_in_project
+                        filter_string = "%s:%s" % (column['filter_name'],
+                                                   filter_action['name'])
+                        # Now get the data with the filter applied
+                        filtered_data = get_data(table_cls(),
+                                                 {"filter" : filter_string})
+                        self.assertEqual(len(filtered_data['rows']),
+                                         int(filter_action['count']),
+                                         "We added a table filter for %s but "
+                                         "the number of rows returned was not "
+                                         "what the filter info said there "
+                                         "would be" % name)
+
+
+            # Test search functionality on the table
+            something_found = False
+            for search in list(string.ascii_letters):
+                search_data = get_data(table_cls(), {'search' : search})
+
+                if len(search_data['rows']) > 0:
+                    something_found = True
+                    break
+
+            self.assertTrue(something_found,
+                            "We went through the whole alphabet and nothing"
+                            " was found for the search of table %s" % name)
+
+            # Test the limit functionality on the table
+            limited_data = get_data(table_cls(), {'limit' : "1"})
+            self.assertEqual(len(limited_data['rows']),
+                             1,
+                             "Limit 1 set on table %s but not 1 row returned"
+                             % name)
+
+            # Test the pagination functionality on the table
+            page_one_data = get_data(table_cls(), {'limit' : "1",
+                                                   "page": "1"})['rows'][0]
+
+            page_two_data = get_data(table_cls(), {'limit' : "1",
+                                                   "page": "2"})['rows'][0]
+
+            self.assertNotEqual(page_one_data,
+                                page_two_data,
+                                "Changed page on table %s but first row is the "
+                                "same as the previous page" % name)
+
+
 class LandingPageTests(TestCase):
     """ Tests for redirects on the landing page """
     # disable bogus pylint message error:
-- 
2.5.0

[PATCH 2/5] toaster: CustomImageRecipe add search_allowed_fields to this model

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

In order to search the model from the UI some fields must be nominated
as searchable.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/toaster/orm/models.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py
index 0ac94b9..0174233 100644
--- a/lib/toaster/orm/models.py
+++ b/lib/toaster/orm/models.py
@@ -1238,6 +1238,7 @@ class ProjectLayer(models.Model):
         unique_together = (("project", "layercommit"),)
 
 class CustomImageRecipe(models.Model):
+    search_allowed_fields = ['name']
     name = models.CharField(max_length=100)
     base_recipe = models.ForeignKey(Recipe)
     packages = models.ManyToManyField(Package)
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.