[PATCH v3 1/8] Toaster: Update toaster-requirements.txt to add pytest and some plugin's

[PATCH v3 1/8] Toaster: Update toaster-requirements.txt to add pytest and some plugin's

[Alassane Yattara]

The pytest framework makes it easy to write small, readable tests, and can scale
to support complex functional testing for applications and libraries.
- Pytest-html is a plugin for pytest that generates a HTML report for test results.
- Pytest-django allows us to test Django project/applications with the pytest testing tool.
- Pytest-env is a pytest plugin that enables us to set environment variables in a pytest.ini or pyproject.toml file
- Pytest-metadata is a plugin for pytest that provides access to test session metadata, required by pytest-html plugin
to provides metadata information in the HTML report like below:

Python 3.10.12
Platform Linux-6.2.0-35-generic-x86_64-with-glibc2.35
Packages:
    - pytest: 7.4.2
    - pluggy: 1.3.0
Plugins	:
    - django: 4.5.2
    - metadata: 3.0.0
    - order: 1.1.0
    - html: 4.0.2
    - env: 1.1.0

Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
---
 toaster-requirements.txt | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/toaster-requirements.txt b/toaster-requirements.txt
index d8e48b7f..19bf6fd2 100644
--- a/toaster-requirements.txt
+++ b/toaster-requirements.txt
@@ -2,3 +2,8 @@ Django>4.2,<4.3
 beautifulsoup4>=4.4.0
 pytz
 django-log-viewer==1.1.7
+pytest==7.4.2
+pytest-django==4.5.2
+pytest-env==1.1.0
+pytest-html==4.0.2
+pytest-metadata==3.0.0
-- 
2.34.1

[PATCH v3 2/8] Toaster: Update orm.models to catch error ProcessLookupError

[Alassane Yattara]

- catch error ProcessLookupError and logs it

Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
---
 lib/toaster/orm/models.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py
index 0d503a51..1098ad3f 100644
--- a/lib/toaster/orm/models.py
+++ b/lib/toaster/orm/models.py
@@ -1389,9 +1389,6 @@ class Machine(models.Model):
         return "Machine " + self.name + "(" + self.description + ")"
 
 
-
-
-
 class BitbakeVersion(models.Model):
 
     name = models.CharField(max_length=32, unique = True)
@@ -1853,6 +1850,8 @@ def signal_runbuilds():
             os.kill(int(pidf.read()), SIGUSR1)
     except FileNotFoundError:
         logger.info("Stopping existing runbuilds: no current process found")
+    except ProcessLookupError:
+        logger.warning("Stopping existing runbuilds: process lookup not found")
 
 class Distro(models.Model):
     search_allowed_fields = ["name", "description", "layer_version__layer__name"]
-- 
2.34.1

[PATCH v3 3/8] Toaster: Bug-fix pytest and Failed: Database access not allowed

[Alassane Yattara]

Remove load and create build environment from tests/functional/functional_helpers
- Testcases in the FunctionalTestCase do not require a build process,
- Also marked setUpClass or class with pytest django_db, db ... not
  working, as declared above of file functional_helpers.py, The database access
  process runs as an external process, separate from the test case process
  and outside the context of pytest.

Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
---
 lib/toaster/tests/functional/functional_helpers.py | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/lib/toaster/tests/functional/functional_helpers.py b/lib/toaster/tests/functional/functional_helpers.py
index c3191f66..b80d403b 100644
--- a/lib/toaster/tests/functional/functional_helpers.py
+++ b/lib/toaster/tests/functional/functional_helpers.py
@@ -15,8 +15,6 @@ import time
 import re
 
 from tests.browser.selenium_helpers_base import SeleniumTestCaseBase
-from tests.builds.buildtest import load_build_environment
-from bldcontrol.models import BuildEnvironment
 from selenium.webdriver.common.by import By
 from selenium.common.exceptions import NoSuchElementException
 
@@ -33,10 +31,6 @@ class SeleniumFunctionalTestCase(SeleniumTestCaseBase):
             raise RuntimeError("Please initialise django with the tests settings:  " \
                 "DJANGO_SETTINGS_MODULE='toastermain.settings_test'")
 
-        if BuildEnvironment.objects.count() == 0:
-            BuildEnvironment.objects.create(betype=BuildEnvironment.TYPE_LOCAL)
-        load_build_environment()
-
         # start toaster
         cmd = "bash -c 'source toaster start'"
         p = subprocess.Popen(
-- 
2.34.1

[PATCH v3 4/8] Toaster: fixed pytest error: Database access not allowed, use the "django_db"

[Alassane Yattara]

Pytest failed on functional/test_functional_basic because database access not allowed
- we should use "django_db" fixture to allowed db access

Note: Pytest-django takes a conservative approach to enabling database access.
By default your tests will fail if they try to access the database.
Only if you explicitly request database access will this be allowed.

https://pytest-django.readthedocs.io/en/latest/helpers.html#pytest-mark-django-db-request-database-access

Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
---
 lib/toaster/tests/functional/test_functional_basic.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/toaster/tests/functional/test_functional_basic.py b/lib/toaster/tests/functional/test_functional_basic.py
index b0def544..7e9be91c 100644
--- a/lib/toaster/tests/functional/test_functional_basic.py
+++ b/lib/toaster/tests/functional/test_functional_basic.py
@@ -8,6 +8,7 @@
 #
 
 import re, time
+import pytest
 from django.urls import reverse
 from tests.functional.functional_helpers import SeleniumFunctionalTestCase
 from orm.models import Project
@@ -16,6 +17,7 @@ from selenium.webdriver.common.by import By
 class FuntionalTestBasic(SeleniumFunctionalTestCase):
 
 #   testcase (1514)
+    @pytest.mark.django_db
     def test_create_slenium_project(self):
         project_name = 'selenium-project'
         self.get(reverse('newproject'))
-- 
2.34.1

[PATCH v3 5/8] Toaster: Bug-fix django.db.utils.IntegrityError: Problem installing fixture

[Alassane Yattara]

toastergui-unittest-data.xml fixture contains a release objects with
name="master" or an release with same name seem existing
- Change release name and fix bitbakeversion instead of bitbake_version

Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
---
 lib/toaster/toastergui/fixtures/toastergui-unittest-data.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/toaster/toastergui/fixtures/toastergui-unittest-data.xml b/lib/toaster/toastergui/fixtures/toastergui-unittest-data.xml
index 1d522f88..df106934 100644
--- a/lib/toaster/toastergui/fixtures/toastergui-unittest-data.xml
+++ b/lib/toaster/toastergui/fixtures/toastergui-unittest-data.xml
@@ -19,9 +19,9 @@
      <field type="CharField" name="description">poky_distro2 description</field>
   </object>
   <object pk="1" model="orm.release">
-     <field type="CharField" name="name">master</field>
+     <field type="CharField" name="name">foo_master</field>
      <field type="CharField" name="description">master project</field>
-     <field to="orm.bitbake_version" name="bitbake_version">1</field>
+     <field to="orm.bitbakeversion" name="bitbake_version">1</field>
   </object>
   <object pk="1" model="orm.project">
     <field type="CharField" name="name">a test project</field>
-- 
2.34.1

[PATCH v3 6/8] Toaster: Move toaster tests requirements in bitbake/lib/toaster/toaster-tests-requirements

[Alassane Yattara]

To make difference between bitbake and toaster python requirements,
a dedicated requirements file is created for toaster in root of toaster dir.

Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
---
 lib/toaster/tests/toaster-tests-requirements.txt | 5 +++++
 toaster-requirements.txt                         | 5 -----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/toaster/tests/toaster-tests-requirements.txt b/lib/toaster/tests/toaster-tests-requirements.txt
index f30ac070..7109c625 100644
--- a/lib/toaster/tests/toaster-tests-requirements.txt
+++ b/lib/toaster/tests/toaster-tests-requirements.txt
@@ -1 +1,6 @@
 selenium>=4.13.0
+pytest==7.4.2
+pytest-django==4.5.2
+pytest-env==1.1.0
+pytest-html==4.0.2
+pytest-metadata==3.0.0
diff --git a/toaster-requirements.txt b/toaster-requirements.txt
index 19bf6fd2..d8e48b7f 100644
--- a/toaster-requirements.txt
+++ b/toaster-requirements.txt
@@ -2,8 +2,3 @@ Django>4.2,<4.3
 beautifulsoup4>=4.4.0
 pytz
 django-log-viewer==1.1.7
-pytest==7.4.2
-pytest-django==4.5.2
-pytest-env==1.1.0
-pytest-html==4.0.2
-pytest-metadata==3.0.0
-- 
2.34.1

Re: [Toaster] [PATCH v3 6/8] Toaster: Move toaster tests requirements in bitbake/lib/toaster/toaster-tests-requirements

[Richard Purdie]

On Thu, 2023-11-09 at 16:55 +0100, Alassane Yattara wrote:
> To make difference between bitbake and toaster python requirements,
> a dedicated requirements file is created for toaster in root of toaster dir.
> 
> Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
> ---
>  lib/toaster/tests/toaster-tests-requirements.txt | 5 +++++
>  toaster-requirements.txt                         | 5 -----
>  2 files changed, 5 insertions(+), 5 deletions(-)

This needs to be squashed into 1/8 and the commit message updated,
like:

https://git.openembedded.org/bitbake/commit/?h=master-next&id=45c3fb79d04ce6b0a886b1b1b0cda2b9dcee1145

so it just creates the new file.

Cheers,

Richard

[PATCH v3 7/8] Toaster: fixed: Tests fail when executed one after the other out of sequence

[Alassane Yattara]

As mentionned in earlier commit, pytest-django takes a conservative approach
to enabling database access. By default our tests will fail if they try to access
the database, Only if we explicitly request database access will this be allowed,
using pytest marks to tell pytest-django our test needs database access.

A side effect of pytest mark, is test_case method marked is execute out of
scope of its module class, which create an inconsistance sequence and make fails
followings tests.

The scope of the ordering is global per default, e.g. tests with lower ordinal
numbers are always executed before tests with higher numbers in the same test session,
regardless of the module and class they reside in. This can be changed by using
the --order-scope option from module pytest-order.

To fix that i added execution order to tests suite using pytest-order.

Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
---
 lib/toaster/tests/functional/test_functional_basic.py | 4 +++-
 lib/toaster/tests/toaster-tests-requirements.txt      | 1 +
 lib/toaster/tests/views/test_views.py                 | 2 ++
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/toaster/tests/functional/test_functional_basic.py b/lib/toaster/tests/functional/test_functional_basic.py
index 7e9be91c..f558cce8 100644
--- a/lib/toaster/tests/functional/test_functional_basic.py
+++ b/lib/toaster/tests/functional/test_functional_basic.py
@@ -8,12 +8,14 @@
 #
 
 import re, time
-import pytest
 from django.urls import reverse
+import pytest
 from tests.functional.functional_helpers import SeleniumFunctionalTestCase
 from orm.models import Project
 from selenium.webdriver.common.by import By
 
+
+@pytest.mark.order("last")
 class FuntionalTestBasic(SeleniumFunctionalTestCase):
 
 #   testcase (1514)
diff --git a/lib/toaster/tests/toaster-tests-requirements.txt b/lib/toaster/tests/toaster-tests-requirements.txt
index 7109c625..71cc0834 100644
--- a/lib/toaster/tests/toaster-tests-requirements.txt
+++ b/lib/toaster/tests/toaster-tests-requirements.txt
@@ -4,3 +4,4 @@ pytest-django==4.5.2
 pytest-env==1.1.0
 pytest-html==4.0.2
 pytest-metadata==3.0.0
+pytest-order==1.1.0
diff --git a/lib/toaster/tests/views/test_views.py b/lib/toaster/tests/views/test_views.py
index 06bf6c20..349881eb 100644
--- a/lib/toaster/tests/views/test_views.py
+++ b/lib/toaster/tests/views/test_views.py
@@ -9,6 +9,7 @@
 
 """Test cases for Toaster GUI and ReST."""
 
+import pytest
 from django.test import TestCase
 from django.test.client import RequestFactory
 from django.urls import reverse
@@ -33,6 +34,7 @@ PROJECT_NAME2 = "test project 2"
 CLI_BUILDS_PROJECT_NAME = 'Command line builds'
 
 
+@pytest.mark.order(1)
 class ViewTests(TestCase):
     """Tests to verify view APIs."""
 
-- 
2.34.1

[PATCH v3 8/8] Toaster: Added pytest.ini file

[Alassane Yattara]

The main reason for using pytest is to be able to generate a positive test report
using the pytest-html plugin.

Integrating Pytest with Tox is a straightforward process, this can be
done using tox.ini instead of pytest.ini used to configure pytest, that
is another reason for using pytest. Tox is a tool that automates testing
across different virtual environments, it can help ensure application
will be tested against multiple Python versions and environments.
https://github.com/pytest-dev/pytest/blob/main/tox.ini

Generated reports create a historical record of test results over time.
This can help track the progress of the application's stability and quality

Documentation and Transparency: Test reports provide us a clear and detailed
documentation of the test results. They show what tests were executed,
which ones passed, and which ones failed. This transparency is critical
for understanding the current state of the application and its test coverage.

Communication: Test reports are an effective means of communication among community
to understand the testing progress and results.

Debugging, Troubleshooting Historical Tracking and Regression Testing:
In case of test failures, a detailed test report can be invaluable for debugging.
It provides information about the specific test case that failed,
the input data used, and any error messages.

Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
---
 lib/toaster/pytest.ini | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
 create mode 100644 lib/toaster/pytest.ini

diff --git a/lib/toaster/pytest.ini b/lib/toaster/pytest.ini
new file mode 100644
index 00000000..f07076b7
--- /dev/null
+++ b/lib/toaster/pytest.ini
@@ -0,0 +1,19 @@
+# -- FILE: pytest.ini (or tox.ini)
+[pytest]
+DJANGO_SETTINGS_MODULE = toastermain.settings_test
+
+python_files = db/test_*.py commands/test_*.py views/test_*.py browser/test_*.py functional/test_*.py
+
+# --create-db - force re creation of the test database
+# https://pytest-django.readthedocs.io/en/latest/database.html#create-db-force-re-creation-of-the-test-database
+
+# --html=report.html --self-contained-html
+# https://docs.pytest.org/en/latest/usage.html#creating-html-reports
+# https://pytest-html.readthedocs.io/en/latest/user_guide.html#creating-a-self-contained-report
+addopts = --create-db --html="Toaster Tests Report.html" --self-contained-html 
+
+# Define environment variables using pytest-env
+# A pytest plugin that enables you to set environment variables in the pytest.ini file.
+# https://pypi.org/project/pytest-env/
+env =
+    TOASTER_BUILDSERVER=1
-- 
2.34.1