[RFC 0/1] acceptance tests: rename acceptance to system

* [RFC 0/1] acceptance tests: rename acceptance to system
@ 2021-05-20 19:53 Willian Rampazzo
  2021-05-20 19:53 ` [RFC 1/1] " Willian Rampazzo
  0 siblings, 1 reply; 21+ messages in thread
From: Willian Rampazzo @ 2021-05-20 19:53 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Niek Linnenbank,
	qemu-arm, Michael Rolnik, Cleber Rosa, Alex Bennée

CI pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/306850452

Conceptually speaking, acceptance tests "are a series of specific tests
conducted by the customer in an attempt to uncover product errors before
accepting the software from the developer. Conducted by the end-user rather
than software engineers, acceptance testing can range from an informal
“test drive” to a planned and systematically executed series of scripted
tests" [1]. Every time Pressman refers to the term "acceptance testing," he
also refers to user's agreement in the final state of an implemented feature.
Today, QEMU is not implementing user acceptance tests as described by Pressman.

There are other three possible terms we could use to describe what is currently
QEMU "acceptance" tests:

  1 - Integration tests:
      - "Integration testing is a systematic technique for constructing the
         software architecture while at the same time conducting tests to
         uncover errors associated with interfacing. The objective is to take
         unit-tested components and build a program structure that has been
         dictated by design." [2]
      * Note: Sommerville does not have a clear definition of integration
        testing. He refers to incremental integration of components inside
        the system testing (see [3]).

  2 - Validation tests:
      - "Validation testing begins at the culmination of integration testing,
         when individual components have been exercised, the software is
         completely assembled as a package, and interfacing errors have been
         uncovered and corrected. At the validation or system level, the
         distinction between different software categories disappears. Testing
         focuses on user-visible actions and user-recognizable output from the
         system." [4]
      - "where you expect the system to perform correctly using a set of test
         cases that reflect the system’s expected use." [5]
      * Note: the definition of "validation testing" from Sommerville reflects
        the same definition found around the Internet, as one of the processes
        inside the "Verification & Validation (V&V)." In this concept,
        validation testing is a high-level definition that covers unit testing,
        functional testing, integration testing, system testing, and acceptance
        testing.

  3 - System tests:
      - "verifies that all elements mesh properly and that overall system
         function and performance is achieved." [6]
      - "involves integrating components to create a version of the system and
         then testing the integrated system. System testing checks that
         components are compatible, interact correctly, and transfer the right
         data at the right time across their interfaces." [7]

The tests implemented inside the QEMU "acceptance" directory depend on the
software completely assembled and, sometimes, on other elements, like operating
system images. In this case, the proposal here is to rename the current
"acceptance" directory to "system."

[1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. p. 430.
[2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 398.
[3] Sommerville, Ian (2016). Software Engineering. p. 240-242.
[4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 407.
[5] Sommerville, Ian (2016). Software Engineering. p. 227.
[6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 377.
[7] Sommerville, Ian (2016). Software Engineering. p. 240.

Signed-off-by: Willian Rampazzo <willianr@redhat.com>

Willian Rampazzo (1):
  acceptance tests: rename acceptance to system

 .gitlab-ci.yml                                | 58 +++++++++----------
 MAINTAINERS                                   | 40 ++++++-------
 configure                                     |  2 +-
 docs/devel/build-system.rst                   |  2 +-
 docs/devel/testing.rst                        | 23 ++++----
 docs/system/arm/orangepi.rst                  |  8 +--
 tests/Makefile.include                        | 14 ++---
 tests/acceptance/README.rst                   | 10 ----
 tests/system/README.rst                       | 10 ++++
 .../avocado_qemu/__init__.py                  |  2 +-
 tests/{acceptance => system}/boot_linux.py    |  0
 .../boot_linux_console.py                     |  0
 tests/{acceptance => system}/boot_xen.py      |  0
 tests/{acceptance => system}/cpu_queries.py   |  0
 .../{acceptance => system}/empty_cpu_model.py |  0
 tests/{acceptance => system}/linux_initrd.py  |  2 +-
 .../linux_ssh_mips_malta.py                   |  0
 .../machine_arm_canona1100.py                 |  0
 .../machine_arm_integratorcp.py               |  0
 .../machine_arm_n8x0.py                       |  0
 tests/{acceptance => system}/machine_avr6.py  |  2 +-
 .../machine_m68k_nextcube.py                  |  0
 .../machine_microblaze.py                     |  0
 .../machine_mips_loongson3v.py                |  0
 .../machine_mips_malta.py                     |  0
 tests/{acceptance => system}/machine_ppc.py   |  0
 .../machine_rx_gdbsim.py                      |  0
 .../machine_s390_ccw_virtio.py                |  0
 .../machine_sparc64_sun4u.py                  |  0
 .../machine_sparc_leon3.py                    |  0
 tests/{acceptance => system}/migration.py     |  0
 tests/{acceptance => system}/multiprocess.py  |  0
 .../pc_cpu_hotplug_props.py                   |  0
 tests/{acceptance => system}/ppc_prep_40p.py  |  0
 tests/{acceptance => system}/replay_kernel.py |  0
 .../reverse_debugging.py                      |  0
 tests/{acceptance => system}/tcg_plugins.py   |  0
 .../{acceptance => system}/tesseract_utils.py |  0
 tests/{acceptance => system}/version.py       |  0
 tests/{acceptance => system}/virtio-gpu.py    |  0
 .../virtio_check_params.py                    |  0
 .../{acceptance => system}/virtio_version.py  |  0
 .../virtiofs_submounts.py                     |  0
 .../virtiofs_submounts.py.data/cleanup.sh     |  0
 .../guest-cleanup.sh                          |  0
 .../virtiofs_submounts.py.data/guest.sh       |  0
 .../virtiofs_submounts.py.data/host.sh        |  0
 tests/{acceptance => system}/vnc.py           |  0
 .../x86_cpu_model_versions.py                 |  0
 49 files changed, 86 insertions(+), 87 deletions(-)
 delete mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/system/README.rst
 rename tests/{acceptance => system}/avocado_qemu/__init__.py (99%)
 rename tests/{acceptance => system}/boot_linux.py (100%)
 rename tests/{acceptance => system}/boot_linux_console.py (100%)
 rename tests/{acceptance => system}/boot_xen.py (100%)
 rename tests/{acceptance => system}/cpu_queries.py (100%)
 rename tests/{acceptance => system}/empty_cpu_model.py (100%)
 rename tests/{acceptance => system}/linux_initrd.py (99%)
 rename tests/{acceptance => system}/linux_ssh_mips_malta.py (100%)
 rename tests/{acceptance => system}/machine_arm_canona1100.py (100%)
 rename tests/{acceptance => system}/machine_arm_integratorcp.py (100%)
 rename tests/{acceptance => system}/machine_arm_n8x0.py (100%)
 rename tests/{acceptance => system}/machine_avr6.py (98%)
 rename tests/{acceptance => system}/machine_m68k_nextcube.py (100%)
 rename tests/{acceptance => system}/machine_microblaze.py (100%)
 rename tests/{acceptance => system}/machine_mips_loongson3v.py (100%)
 rename tests/{acceptance => system}/machine_mips_malta.py (100%)
 rename tests/{acceptance => system}/machine_ppc.py (100%)
 rename tests/{acceptance => system}/machine_rx_gdbsim.py (100%)
 rename tests/{acceptance => system}/machine_s390_ccw_virtio.py (100%)
 rename tests/{acceptance => system}/machine_sparc64_sun4u.py (100%)
 rename tests/{acceptance => system}/machine_sparc_leon3.py (100%)
 rename tests/{acceptance => system}/migration.py (100%)
 rename tests/{acceptance => system}/multiprocess.py (100%)
 rename tests/{acceptance => system}/pc_cpu_hotplug_props.py (100%)
 rename tests/{acceptance => system}/ppc_prep_40p.py (100%)
 rename tests/{acceptance => system}/replay_kernel.py (100%)
 rename tests/{acceptance => system}/reverse_debugging.py (100%)
 rename tests/{acceptance => system}/tcg_plugins.py (100%)
 rename tests/{acceptance => system}/tesseract_utils.py (100%)
 rename tests/{acceptance => system}/version.py (100%)
 rename tests/{acceptance => system}/virtio-gpu.py (100%)
 rename tests/{acceptance => system}/virtio_check_params.py (100%)
 rename tests/{acceptance => system}/virtio_version.py (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/cleanup.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/guest-cleanup.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/guest.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/host.sh (100%)
 rename tests/{acceptance => system}/vnc.py (100%)
 rename tests/{acceptance => system}/x86_cpu_model_versions.py (100%)

-- 
2.31.1

^ permalink raw reply	[flat|nested] 21+ messages in thread