[RFC 0/1] acceptance tests: rename acceptance to system

[RFC 0/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

CI pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/306850452

Conceptually speaking, acceptance tests "are a series of specific tests
conducted by the customer in an attempt to uncover product errors before
accepting the software from the developer. Conducted by the end-user rather
than software engineers, acceptance testing can range from an informal
“test drive” to a planned and systematically executed series of scripted
tests" [1]. Every time Pressman refers to the term "acceptance testing," he
also refers to user's agreement in the final state of an implemented feature.
Today, QEMU is not implementing user acceptance tests as described by Pressman.

There are other three possible terms we could use to describe what is currently
QEMU "acceptance" tests:

  1 - Integration tests:
      - "Integration testing is a systematic technique for constructing the
         software architecture while at the same time conducting tests to
         uncover errors associated with interfacing. The objective is to take
         unit-tested components and build a program structure that has been
         dictated by design." [2]
      * Note: Sommerville does not have a clear definition of integration
        testing. He refers to incremental integration of components inside
        the system testing (see [3]).

  2 - Validation tests:
      - "Validation testing begins at the culmination of integration testing,
         when individual components have been exercised, the software is
         completely assembled as a package, and interfacing errors have been
         uncovered and corrected. At the validation or system level, the
         distinction between different software categories disappears. Testing
         focuses on user-visible actions and user-recognizable output from the
         system." [4]
      - "where you expect the system to perform correctly using a set of test
         cases that reflect the system’s expected use." [5]
      * Note: the definition of "validation testing" from Sommerville reflects
        the same definition found around the Internet, as one of the processes
        inside the "Verification & Validation (V&V)." In this concept,
        validation testing is a high-level definition that covers unit testing,
        functional testing, integration testing, system testing, and acceptance
        testing.

  3 - System tests:
      - "verifies that all elements mesh properly and that overall system
         function and performance is achieved." [6]
      - "involves integrating components to create a version of the system and
         then testing the integrated system. System testing checks that
         components are compatible, interact correctly, and transfer the right
         data at the right time across their interfaces." [7]

The tests implemented inside the QEMU "acceptance" directory depend on the
software completely assembled and, sometimes, on other elements, like operating
system images. In this case, the proposal here is to rename the current
"acceptance" directory to "system."

[1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. p. 430.
[2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 398.
[3] Sommerville, Ian (2016). Software Engineering. p. 240-242.
[4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 407.
[5] Sommerville, Ian (2016). Software Engineering. p. 227.
[6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 377.
[7] Sommerville, Ian (2016). Software Engineering. p. 240.

Signed-off-by: Willian Rampazzo <willianr@redhat.com>

Willian Rampazzo (1):
  acceptance tests: rename acceptance to system

 .gitlab-ci.yml                                | 58 +++++++++----------
 MAINTAINERS                                   | 40 ++++++-------
 configure                                     |  2 +-
 docs/devel/build-system.rst                   |  2 +-
 docs/devel/testing.rst                        | 23 ++++----
 docs/system/arm/orangepi.rst                  |  8 +--
 tests/Makefile.include                        | 14 ++---
 tests/acceptance/README.rst                   | 10 ----
 tests/system/README.rst                       | 10 ++++
 .../avocado_qemu/__init__.py                  |  2 +-
 tests/{acceptance => system}/boot_linux.py    |  0
 .../boot_linux_console.py                     |  0
 tests/{acceptance => system}/boot_xen.py      |  0
 tests/{acceptance => system}/cpu_queries.py   |  0
 .../{acceptance => system}/empty_cpu_model.py |  0
 tests/{acceptance => system}/linux_initrd.py  |  2 +-
 .../linux_ssh_mips_malta.py                   |  0
 .../machine_arm_canona1100.py                 |  0
 .../machine_arm_integratorcp.py               |  0
 .../machine_arm_n8x0.py                       |  0
 tests/{acceptance => system}/machine_avr6.py  |  2 +-
 .../machine_m68k_nextcube.py                  |  0
 .../machine_microblaze.py                     |  0
 .../machine_mips_loongson3v.py                |  0
 .../machine_mips_malta.py                     |  0
 tests/{acceptance => system}/machine_ppc.py   |  0
 .../machine_rx_gdbsim.py                      |  0
 .../machine_s390_ccw_virtio.py                |  0
 .../machine_sparc64_sun4u.py                  |  0
 .../machine_sparc_leon3.py                    |  0
 tests/{acceptance => system}/migration.py     |  0
 tests/{acceptance => system}/multiprocess.py  |  0
 .../pc_cpu_hotplug_props.py                   |  0
 tests/{acceptance => system}/ppc_prep_40p.py  |  0
 tests/{acceptance => system}/replay_kernel.py |  0
 .../reverse_debugging.py                      |  0
 tests/{acceptance => system}/tcg_plugins.py   |  0
 .../{acceptance => system}/tesseract_utils.py |  0
 tests/{acceptance => system}/version.py       |  0
 tests/{acceptance => system}/virtio-gpu.py    |  0
 .../virtio_check_params.py                    |  0
 .../{acceptance => system}/virtio_version.py  |  0
 .../virtiofs_submounts.py                     |  0
 .../virtiofs_submounts.py.data/cleanup.sh     |  0
 .../guest-cleanup.sh                          |  0
 .../virtiofs_submounts.py.data/guest.sh       |  0
 .../virtiofs_submounts.py.data/host.sh        |  0
 tests/{acceptance => system}/vnc.py           |  0
 .../x86_cpu_model_versions.py                 |  0
 49 files changed, 86 insertions(+), 87 deletions(-)
 delete mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/system/README.rst
 rename tests/{acceptance => system}/avocado_qemu/__init__.py (99%)
 rename tests/{acceptance => system}/boot_linux.py (100%)
 rename tests/{acceptance => system}/boot_linux_console.py (100%)
 rename tests/{acceptance => system}/boot_xen.py (100%)
 rename tests/{acceptance => system}/cpu_queries.py (100%)
 rename tests/{acceptance => system}/empty_cpu_model.py (100%)
 rename tests/{acceptance => system}/linux_initrd.py (99%)
 rename tests/{acceptance => system}/linux_ssh_mips_malta.py (100%)
 rename tests/{acceptance => system}/machine_arm_canona1100.py (100%)
 rename tests/{acceptance => system}/machine_arm_integratorcp.py (100%)
 rename tests/{acceptance => system}/machine_arm_n8x0.py (100%)
 rename tests/{acceptance => system}/machine_avr6.py (98%)
 rename tests/{acceptance => system}/machine_m68k_nextcube.py (100%)
 rename tests/{acceptance => system}/machine_microblaze.py (100%)
 rename tests/{acceptance => system}/machine_mips_loongson3v.py (100%)
 rename tests/{acceptance => system}/machine_mips_malta.py (100%)
 rename tests/{acceptance => system}/machine_ppc.py (100%)
 rename tests/{acceptance => system}/machine_rx_gdbsim.py (100%)
 rename tests/{acceptance => system}/machine_s390_ccw_virtio.py (100%)
 rename tests/{acceptance => system}/machine_sparc64_sun4u.py (100%)
 rename tests/{acceptance => system}/machine_sparc_leon3.py (100%)
 rename tests/{acceptance => system}/migration.py (100%)
 rename tests/{acceptance => system}/multiprocess.py (100%)
 rename tests/{acceptance => system}/pc_cpu_hotplug_props.py (100%)
 rename tests/{acceptance => system}/ppc_prep_40p.py (100%)
 rename tests/{acceptance => system}/replay_kernel.py (100%)
 rename tests/{acceptance => system}/reverse_debugging.py (100%)
 rename tests/{acceptance => system}/tcg_plugins.py (100%)
 rename tests/{acceptance => system}/tesseract_utils.py (100%)
 rename tests/{acceptance => system}/version.py (100%)
 rename tests/{acceptance => system}/virtio-gpu.py (100%)
 rename tests/{acceptance => system}/virtio_check_params.py (100%)
 rename tests/{acceptance => system}/virtio_version.py (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/cleanup.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/guest-cleanup.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/guest.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/host.sh (100%)
 rename tests/{acceptance => system}/vnc.py (100%)
 rename tests/{acceptance => system}/x86_cpu_model_versions.py (100%)

-- 
2.31.1

[RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

Conceptually speaking, acceptance tests "are a series of specific tests
conducted by the customer in an attempt to uncover product errors before
accepting the software from the developer. Conducted by the end-user rather
than software engineers, acceptance testing can range from an informal
“test drive” to a planned and systematically executed series of scripted
tests" [1]. Every time Pressman refers to the term "acceptance testing," he
also refers to user's agreement in the final state of an implemented feature.
Today, QEMU is not implementing user acceptance tests as described by Pressman.

There are other three possible terms we could use to describe what is currently
QEMU "acceptance" tests:

  1 - Integration tests:
      - "Integration testing is a systematic technique for constructing the
         software architecture while at the same time conducting tests to
         uncover errors associated with interfacing. The objective is to take
         unit-tested components and build a program structure that has been
         dictated by design." [2]
      * Note: Sommerville does not have a clear definition of integration
        testing. He refers to incremental integration of components inside
        the system testing (see [3]).

  2 - Validation tests:
      - "Validation testing begins at the culmination of integration testing,
         when individual components have been exercised, the software is
         completely assembled as a package, and interfacing errors have been
         uncovered and corrected. At the validation or system level, the
         distinction between different software categories disappears. Testing
         focuses on user-visible actions and user-recognizable output from the
         system." [4]
      - "where you expect the system to perform correctly using a set of test
         cases that reflect the system’s expected use." [5]
      * Note: the definition of "validation testing" from Sommerville reflects
        the same definition found around the Internet, as one of the processes
        inside the "Verification & Validation (V&V)." In this concept,
        validation testing is a high-level definition that covers unit testing,
        functional testing, integration testing, system testing, and acceptance
        testing.

  3 - System tests:
      - "verifies that all elements mesh properly and that overall system
         function and performance is achieved." [6]
      - "involves integrating components to create a version of the system and
         then testing the integrated system. System testing checks that
         components are compatible, interact correctly, and transfer the right
         data at the right time across their interfaces." [7]

The tests implemented inside the QEMU "acceptance" directory depend on the
software completely assembled and, sometimes, on other elements, like operating
system images. In this case, the proposal here is to rename the current
"acceptance" directory to "system."

[1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. p. 430.
[2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 398.
[3] Sommerville, Ian (2016). Software Engineering. p. 240-242.
[4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 407.
[5] Sommerville, Ian (2016). Software Engineering. p. 227.
[6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
    Practitioner’s Approach. Software Engineering, p. 377.
[7] Sommerville, Ian (2016). Software Engineering. p. 240.

Signed-off-by: Willian Rampazzo <willianr@redhat.com>
---
 .gitlab-ci.yml                                | 58 +++++++++----------
 MAINTAINERS                                   | 40 ++++++-------
 configure                                     |  2 +-
 docs/devel/build-system.rst                   |  2 +-
 docs/devel/testing.rst                        | 23 ++++----
 docs/system/arm/orangepi.rst                  |  8 +--
 tests/Makefile.include                        | 14 ++---
 tests/acceptance/README.rst                   | 10 ----
 tests/system/README.rst                       | 10 ++++
 .../avocado_qemu/__init__.py                  |  2 +-
 tests/{acceptance => system}/boot_linux.py    |  0
 .../boot_linux_console.py                     |  0
 tests/{acceptance => system}/boot_xen.py      |  0
 tests/{acceptance => system}/cpu_queries.py   |  0
 .../{acceptance => system}/empty_cpu_model.py |  0
 tests/{acceptance => system}/linux_initrd.py  |  2 +-
 .../linux_ssh_mips_malta.py                   |  0
 .../machine_arm_canona1100.py                 |  0
 .../machine_arm_integratorcp.py               |  0
 .../machine_arm_n8x0.py                       |  0
 tests/{acceptance => system}/machine_avr6.py  |  2 +-
 .../machine_m68k_nextcube.py                  |  0
 .../machine_microblaze.py                     |  0
 .../machine_mips_loongson3v.py                |  0
 .../machine_mips_malta.py                     |  0
 tests/{acceptance => system}/machine_ppc.py   |  0
 .../machine_rx_gdbsim.py                      |  0
 .../machine_s390_ccw_virtio.py                |  0
 .../machine_sparc64_sun4u.py                  |  0
 .../machine_sparc_leon3.py                    |  0
 tests/{acceptance => system}/migration.py     |  0
 tests/{acceptance => system}/multiprocess.py  |  0
 .../pc_cpu_hotplug_props.py                   |  0
 tests/{acceptance => system}/ppc_prep_40p.py  |  0
 tests/{acceptance => system}/replay_kernel.py |  0
 .../reverse_debugging.py                      |  0
 tests/{acceptance => system}/tcg_plugins.py   |  0
 .../{acceptance => system}/tesseract_utils.py |  0
 tests/{acceptance => system}/version.py       |  0
 tests/{acceptance => system}/virtio-gpu.py    |  0
 .../virtio_check_params.py                    |  0
 .../{acceptance => system}/virtio_version.py  |  0
 .../virtiofs_submounts.py                     |  0
 .../virtiofs_submounts.py.data/cleanup.sh     |  0
 .../guest-cleanup.sh                          |  0
 .../virtiofs_submounts.py.data/guest.sh       |  0
 .../virtiofs_submounts.py.data/host.sh        |  0
 tests/{acceptance => system}/vnc.py           |  0
 .../x86_cpu_model_versions.py                 |  0
 49 files changed, 86 insertions(+), 87 deletions(-)
 delete mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/system/README.rst
 rename tests/{acceptance => system}/avocado_qemu/__init__.py (99%)
 rename tests/{acceptance => system}/boot_linux.py (100%)
 rename tests/{acceptance => system}/boot_linux_console.py (100%)
 rename tests/{acceptance => system}/boot_xen.py (100%)
 rename tests/{acceptance => system}/cpu_queries.py (100%)
 rename tests/{acceptance => system}/empty_cpu_model.py (100%)
 rename tests/{acceptance => system}/linux_initrd.py (99%)
 rename tests/{acceptance => system}/linux_ssh_mips_malta.py (100%)
 rename tests/{acceptance => system}/machine_arm_canona1100.py (100%)
 rename tests/{acceptance => system}/machine_arm_integratorcp.py (100%)
 rename tests/{acceptance => system}/machine_arm_n8x0.py (100%)
 rename tests/{acceptance => system}/machine_avr6.py (98%)
 rename tests/{acceptance => system}/machine_m68k_nextcube.py (100%)
 rename tests/{acceptance => system}/machine_microblaze.py (100%)
 rename tests/{acceptance => system}/machine_mips_loongson3v.py (100%)
 rename tests/{acceptance => system}/machine_mips_malta.py (100%)
 rename tests/{acceptance => system}/machine_ppc.py (100%)
 rename tests/{acceptance => system}/machine_rx_gdbsim.py (100%)
 rename tests/{acceptance => system}/machine_s390_ccw_virtio.py (100%)
 rename tests/{acceptance => system}/machine_sparc64_sun4u.py (100%)
 rename tests/{acceptance => system}/machine_sparc_leon3.py (100%)
 rename tests/{acceptance => system}/migration.py (100%)
 rename tests/{acceptance => system}/multiprocess.py (100%)
 rename tests/{acceptance => system}/pc_cpu_hotplug_props.py (100%)
 rename tests/{acceptance => system}/ppc_prep_40p.py (100%)
 rename tests/{acceptance => system}/replay_kernel.py (100%)
 rename tests/{acceptance => system}/reverse_debugging.py (100%)
 rename tests/{acceptance => system}/tcg_plugins.py (100%)
 rename tests/{acceptance => system}/tesseract_utils.py (100%)
 rename tests/{acceptance => system}/version.py (100%)
 rename tests/{acceptance => system}/virtio-gpu.py (100%)
 rename tests/{acceptance => system}/virtio_check_params.py (100%)
 rename tests/{acceptance => system}/virtio_version.py (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/cleanup.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/guest-cleanup.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/guest.sh (100%)
 rename tests/{acceptance => system}/virtiofs_submounts.py.data/host.sh (100%)
 rename tests/{acceptance => system}/vnc.py (100%)
 rename tests/{acceptance => system}/x86_cpu_model_versions.py (100%)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index f718b61fa7..c5de3c9fd5 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -52,7 +52,7 @@ include:
     # Avoid recompiling by hiding ninja with NINJA=":"
     - make NINJA=":" $MAKE_CHECK_ARGS
 
-.acceptance_template: &acceptance_definition
+.system_template: &system_definition
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
@@ -107,15 +107,15 @@ check-system-alpine:
     IMAGE: alpine
     MAKE_CHECK_ARGS: check
 
-acceptance-system-alpine:
+system-test-system-alpine:
   extends: .native_test_job_template
   needs:
     - job: build-system-alpine
       artifacts: true
   variables:
     IMAGE: alpine
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 build-system-ubuntu:
   extends: .native_build_job_template
@@ -141,15 +141,15 @@ check-system-ubuntu:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
-acceptance-system-ubuntu:
+system-test-system-ubuntu:
   extends: .native_test_job_template
   needs:
     - job: build-system-ubuntu
       artifacts: true
   variables:
     IMAGE: ubuntu2004
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 build-system-debian:
   extends: .native_build_job_template
@@ -175,15 +175,15 @@ check-system-debian:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
-acceptance-system-debian:
+system-test-system-debian:
   extends: .native_test_job_template
   needs:
     - job: build-system-debian
       artifacts: true
   variables:
     IMAGE: debian-amd64
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 build-system-fedora:
   extends: .native_build_job_template
@@ -210,15 +210,15 @@ check-system-fedora:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-system-fedora:
+system-test-system-fedora:
   extends: .native_test_job_template
   needs:
     - job: build-system-fedora
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 build-system-centos:
   extends: .native_build_job_template
@@ -245,15 +245,15 @@ check-system-centos:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check
 
-acceptance-system-centos:
+system-test-system-centos:
   extends: .native_test_job_template
   needs:
     - job: build-system-centos
       artifacts: true
   variables:
     IMAGE: centos8
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 build-system-opensuse:
   extends: .native_build_job_template
@@ -278,15 +278,15 @@ check-system-opensuse:
     IMAGE: opensuse-leap
     MAKE_CHECK_ARGS: check
 
-acceptance-system-opensuse:
+system-test-system-opensuse:
   extends: .native_test_job_template
   needs:
     - job: build-system-opensuse
       artifacts: true
   variables:
     IMAGE: opensuse-leap
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 
 build-disabled:
@@ -502,7 +502,7 @@ clang-user:
 # This can be accomplished by using -enable-slirp=git, which avoids the use of
 # a system-wide version of the library
 #
-# Split in three sets of build/check/acceptance to limit the execution time of each
+# Split in three sets of build/check/system to limit the execution time of each
 # job
 build-cfi-aarch64:
   extends: .native_build_job_template
@@ -531,15 +531,15 @@ check-cfi-aarch64:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-cfi-aarch64:
+system-test-cfi-aarch64:
   extends: .native_test_job_template
   needs:
     - job: build-cfi-aarch64
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 build-cfi-ppc64-s390x:
   extends: .native_build_job_template
@@ -568,15 +568,15 @@ check-cfi-ppc64-s390x:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-cfi-ppc64-s390x:
+system-test-cfi-ppc64-s390x:
   extends: .native_test_job_template
   needs:
     - job: build-cfi-ppc64-s390x
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 build-cfi-x86_64:
   extends: .native_build_job_template
@@ -605,15 +605,15 @@ check-cfi-x86_64:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-cfi-x86_64:
+system-test-cfi-x86_64:
   extends: .native_test_job_template
   needs:
     - job: build-cfi-x86_64
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+    MAKE_CHECK_ARGS: check-system
+  <<: *system_definition
 
 tsan-build:
   extends: .native_build_job_template
diff --git a/MAINTAINERS b/MAINTAINERS
index 75e0f2d750..bc8a2dd248 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -178,7 +178,7 @@ S: Maintained
 F: docs/system/target-avr.rst
 F: gdb-xml/avr-cpu.xml
 F: target/avr/
-F: tests/acceptance/machine_avr6.py
+F: tests/system/machine_avr6.py
 
 CRIS TCG CPUs
 M: Edgar E. Iglesias <edgar.iglesias@gmail.com>
@@ -272,7 +272,7 @@ F: target/ppc/
 F: hw/ppc/
 F: include/hw/ppc/
 F: disas/ppc.c
-F: tests/acceptance/machine_ppc.py
+F: tests/system/machine_ppc.py
 
 RISC-V TCG CPUs
 M: Palmer Dabbelt <palmer@dabbelt.com>
@@ -641,7 +641,7 @@ S: Odd Fixes
 F: include/hw/arm/digic.h
 F: hw/*/digic*
 F: include/hw/*/digic*
-F: tests/acceptance/machine_arm_canona1100.py
+F: tests/system/machine_arm_canona1100.py
 F: docs/system/arm/digic.rst
 
 Goldfish RTC
@@ -690,7 +690,7 @@ S: Maintained
 F: hw/arm/integratorcp.c
 F: hw/misc/arm_integrator_debug.c
 F: include/hw/misc/arm_integrator_debug.h
-F: tests/acceptance/machine_arm_integratorcp.py
+F: tests/system/machine_arm_integratorcp.py
 F: docs/system/arm/integratorcp.rst
 
 MCIMX6UL EVK / i.MX6ul
@@ -787,7 +787,7 @@ F: hw/rtc/twl92230.c
 F: include/hw/display/blizzard.h
 F: include/hw/input/tsc2xxx.h
 F: include/hw/misc/cbus.h
-F: tests/acceptance/machine_arm_n8x0.py
+F: tests/system/machine_arm_n8x0.py
 F: docs/system/arm/nseries.rst
 
 Palm
@@ -1123,7 +1123,7 @@ M: Edgar E. Iglesias <edgar.iglesias@gmail.com>
 S: Maintained
 F: hw/microblaze/petalogix_s3adsp1800_mmu.c
 F: include/hw/char/xilinx_uartlite.h
-F: tests/acceptance/machine_microblaze.py
+F: tests/system/machine_microblaze.py
 
 petalogix_ml605
 M: Edgar E. Iglesias <edgar.iglesias@gmail.com>
@@ -1149,8 +1149,8 @@ F: hw/acpi/piix4.c
 F: hw/mips/malta.c
 F: hw/mips/gt64xxx_pci.c
 F: include/hw/southbridge/piix.h
-F: tests/acceptance/linux_ssh_mips_malta.py
-F: tests/acceptance/machine_mips_malta.py
+F: tests/system/linux_ssh_mips_malta.py
+F: tests/system/machine_mips_malta.py
 
 Mipssim
 R: Aleksandar Rikalo <aleksandar.rikalo@syrmia.com>
@@ -1177,7 +1177,7 @@ F: hw/intc/loongson_liointc.c
 F: hw/mips/loongson3_bootp.c
 F: hw/mips/loongson3_bootp.h
 F: hw/mips/loongson3_virt.c
-F: tests/acceptance/machine_mips_loongson3v.py
+F: tests/system/machine_mips_loongson3v.py
 
 Boston
 M: Paul Burton <paulburton@kernel.org>
@@ -1284,7 +1284,7 @@ F: hw/dma/i82374.c
 F: hw/rtc/m48t59-isa.c
 F: include/hw/isa/pc87312.h
 F: include/hw/rtc/m48t59.h
-F: tests/acceptance/ppc_prep_40p.py
+F: tests/system/ppc_prep_40p.py
 
 sPAPR
 M: David Gibson <david@gibson.dropbear.id.au>
@@ -1402,7 +1402,7 @@ R: Yoshinori Sato <ysato@users.sourceforge.jp>
 S: Orphan
 F: docs/system/target-rx.rst
 F: hw/rx/rx-gdbsim.c
-F: tests/acceptance/machine_rx_gdbsim.py
+F: tests/system/machine_rx_gdbsim.py
 
 SH4 Machines
 ------------
@@ -1456,7 +1456,7 @@ F: include/hw/pci-host/sabre.h
 F: hw/pci-bridge/simba.c
 F: include/hw/pci-bridge/simba.h
 F: pc-bios/openbios-sparc64
-F: tests/acceptance/machine_sparc64_sun4u.py
+F: tests/system/machine_sparc64_sun4u.py
 
 Sun4v
 M: Artyom Tarasenko <atar4qemu@gmail.com>
@@ -1472,7 +1472,7 @@ S: Maintained
 F: hw/sparc/leon3.c
 F: hw/*/grlib*
 F: include/hw/*/grlib*
-F: tests/acceptance/machine_sparc_leon3.py
+F: tests/system/machine_sparc_leon3.py
 
 S390 Machines
 -------------
@@ -1488,7 +1488,7 @@ F: include/hw/s390x/
 F: hw/watchdog/wdt_diag288.c
 F: include/hw/watchdog/wdt_diag288.h
 F: default-configs/*/s390x-softmmu.mak
-F: tests/acceptance/machine_s390_ccw_virtio.py
+F: tests/system/machine_s390_ccw_virtio.py
 T: git https://gitlab.com/cohuck/qemu.git s390-next
 T: git https://github.com/borntraeger/qemu.git s390-next
 L: qemu-s390x@nongnu.org
@@ -2045,7 +2045,7 @@ M: Alex Bennée <alex.bennee@linaro.org>
 S: Maintained
 F: hw/core/guest-loader.c
 F: docs/system/guest-loader.rst
-F: tests/acceptance/boot_xen.py
+F: tests/system/boot_xen.py
 
 Intel Hexadecimal Object File Loader
 M: Su Hang <suhang16@mails.ucas.ac.cn>
@@ -2854,8 +2854,8 @@ F: net/filter-replay.c
 F: include/sysemu/replay.h
 F: docs/replay.txt
 F: stubs/replay.c
-F: tests/acceptance/replay_kernel.py
-F: tests/acceptance/reverse_debugging.py
+F: tests/system/replay_kernel.py
+F: tests/system/reverse_debugging.py
 F: qapi/replay.json
 
 IOVA Tree
@@ -2970,7 +2970,7 @@ S: Maintained
 F: docs/devel/tcg-plugins.rst
 F: plugins/
 F: tests/plugin/
-F: tests/acceptance/tcg_plugins.py
+F: tests/system/tcg_plugins.py
 F: contrib/plugins/
 
 AArch64 TCG target
@@ -3348,13 +3348,13 @@ S: Maintained
 F: tests/tcg/Makefile
 F: tests/tcg/Makefile.include
 
-Acceptance (Integration) Testing with the Avocado framework
+System Testing with the Avocado framework
 W: https://trello.com/b/6Qi1pxVn/avocado-qemu
 R: Cleber Rosa <crosa@redhat.com>
 R: Philippe Mathieu-Daudé <philmd@redhat.com>
 R: Wainer dos Santos Moschetta <wainersm@redhat.com>
 S: Odd Fixes
-F: tests/acceptance/
+F: tests/system/
 
 Documentation
 -------------
diff --git a/configure b/configure
index 9470fff09a..70a3ab0095 100755
--- a/configure
+++ b/configure
@@ -6290,7 +6290,7 @@ LINKS="$LINKS pc-bios/s390-ccw/Makefile"
 LINKS="$LINKS roms/seabios/Makefile"
 LINKS="$LINKS pc-bios/qemu-icon.bmp"
 LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit
-LINKS="$LINKS tests/acceptance tests/data"
+LINKS="$LINKS tests/system tests/data"
 LINKS="$LINKS tests/qemu-iotests/check"
 LINKS="$LINKS python"
 LINKS="$LINKS contrib/plugins/Makefile "
diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
index 7ef36f42d0..4ee01d270b 100644
--- a/docs/devel/build-system.rst
+++ b/docs/devel/build-system.rst
@@ -387,7 +387,7 @@ number of dynamically created files listed later.
 
 `tests/Makefile.include`
   Rules for external test harnesses. These include the TCG tests,
-  `qemu-iotests` and the Avocado-based acceptance tests.
+  `qemu-iotests` and the Avocado-based system tests.
 
 `tests/docker/Makefile.include`
   Rules for Docker tests. Like tests/Makefile, this file is included
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 1da4c4e4c4..e60a035bca 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -643,17 +643,16 @@ supported. To start the fuzzer, run
 Alternatively, some command different from "qemu-img info" can be tested, by
 changing the ``-c`` option.
 
-Acceptance tests using the Avocado Framework
-============================================
+System tests using the Avocado Framework
+========================================
 
-The ``tests/acceptance`` directory hosts functional tests, also known
-as acceptance level tests.  They're usually higher level tests, and
-may interact with external resources and with various guest operating
-systems.
+The ``tests/system`` directory hosts system tests. They're usually higher
+level tests, and may interact with external resources and with various
+guest operating systems.
 
 These tests are written using the Avocado Testing Framework (which must
 be installed separately) in conjunction with a the ``avocado_qemu.Test``
-class, implemented at ``tests/acceptance/avocado_qemu``.
+class, implemented at ``tests/system/avocado_qemu``.
 
 Tests based on ``avocado_qemu.Test`` can easily:
 
@@ -685,11 +684,11 @@ Tests based on ``avocado_qemu.Test`` can easily:
 Running tests
 -------------
 
-You can run the acceptance tests simply by executing:
+You can run the system tests simply by executing:
 
 .. code::
 
-  make check-acceptance
+  make check-system
 
 This involves the automatic creation of Python virtual environment
 within the build tree (at ``tests/venv``) which will have all the
@@ -709,7 +708,7 @@ may be invoked by running:
 
  .. code::
 
-  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
+  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/system/
 
 Manual Installation
 -------------------
@@ -727,7 +726,7 @@ Alternatively, follow the instructions on this link:
 Overview
 --------
 
-The ``tests/acceptance/avocado_qemu`` directory provides the
+The ``tests/system/avocado_qemu`` directory provides the
 ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
 class.  Here's a simple usage example:
 
@@ -1011,7 +1010,7 @@ And remove any package you want with::
 
   pip uninstall <package_name>
 
-If you've used ``make check-acceptance``, the Python virtual environment where
+If you've used ``make check-system``, the Python virtual environment where
 Avocado is installed will be cleaned up as part of ``make check-clean``.
 
 .. _checktcg-ref:
diff --git a/docs/system/arm/orangepi.rst b/docs/system/arm/orangepi.rst
index 6f23907fb6..84124b104d 100644
--- a/docs/system/arm/orangepi.rst
+++ b/docs/system/arm/orangepi.rst
@@ -250,14 +250,14 @@ and set the following environment variables before booting:
 Optionally you may save the environment variables to SD card with 'saveenv'.
 To continue booting simply give the 'boot' command and NetBSD boots.
 
-Orange Pi PC acceptance tests
-"""""""""""""""""""""""""""""
+Orange Pi PC system tests
+"""""""""""""""""""""""""
 
-The Orange Pi PC machine has several acceptance tests included.
+The Orange Pi PC machine has several system tests included.
 To run the whole set of tests, build QEMU from source and simply
 provide the following command:
 
 .. code-block:: bash
 
   $ AVOCADO_ALLOW_LARGE_STORAGE=yes avocado --show=app,console run \
-     -t machine:orangepi-pc tests/acceptance/boot_linux_console.py
+     -t machine:orangepi-pc tests/system/boot_linux_console.py
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 8f220e15d1..c580292bb5 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -16,7 +16,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),)
 	@echo " $(MAKE) check-tcg            Run TCG tests"
 	@echo " $(MAKE) check-softfloat      Run FPU emulation tests"
 endif
-	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
+	@echo " $(MAKE) check-system         Run all system tests"
 	@echo
 	@echo " $(MAKE) check-report.tap     Generates an aggregated TAP test report"
 	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
@@ -24,7 +24,7 @@ endif
 	@echo
 	@echo "The following are useful for CI builds"
 	@echo " $(MAKE) check-build          Build most test binaris"
-	@echo " $(MAKE) get-vm-images        Downloads all images used by acceptance tests, according to configured targets (~350 MB each, 1.5 GB max)"
+	@echo " $(MAKE) get-vm-images        Downloads all images used by system tests, according to configured targets (~350 MB each, 1.5 GB max)"
 	@echo
 	@echo
 	@echo "The variable SPEED can be set to control the gtester speed setting."
@@ -83,7 +83,7 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
 
 # Python venv for running tests
 
-.PHONY: check-venv check-acceptance
+.PHONY: check-venv check-system
 
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
@@ -119,19 +119,19 @@ get-vm-image-fedora-31-%: check-venv
 	$(call quiet-command, \
              $(TESTS_VENV_DIR)/bin/python -m avocado vmimage get \
              --distro=fedora --distro-version=31 --arch=$*, \
-	"AVOCADO", "Downloading acceptance tests VM image for $*")
+	"AVOCADO", "Downloading system tests VM image for $*")
 
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
-check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
+check-system: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
             $(AVOCADO_TAGS) \
-            $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
-            "AVOCADO", "tests/acceptance")
+            $(if $(GITLAB_CI),,--failfast) tests/system, \
+            "AVOCADO", "tests/system")
 
 # Consolidated targets
 
diff --git a/tests/acceptance/README.rst b/tests/acceptance/README.rst
deleted file mode 100644
index 89260faed6..0000000000
--- a/tests/acceptance/README.rst
+++ /dev/null
@@ -1,10 +0,0 @@
-============================================
-Acceptance tests using the Avocado Framework
-============================================
-
-This directory contains functional tests, also known as acceptance
-level tests.  They're usually higher level, and may interact with
-external resources and with various guest operating systems.
-
-For more information, please refer to ``docs/devel/testing.rst``,
-section "Acceptance tests using the Avocado Framework".
diff --git a/tests/system/README.rst b/tests/system/README.rst
new file mode 100644
index 0000000000..ece14073c5
--- /dev/null
+++ b/tests/system/README.rst
@@ -0,0 +1,10 @@
+========================================
+System tests using the Avocado Framework
+========================================
+
+This directory contains system tests. They're usually higher level,
+and may interact with external resources and with various guest
+operating systems.
+
+For more information, please refer to ``docs/devel/testing.rst``,
+section "System tests using the Avocado Framework".
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/system/avocado_qemu/__init__.py
similarity index 99%
rename from tests/acceptance/avocado_qemu/__init__.py
rename to tests/system/avocado_qemu/__init__.py
index 83b1741ec8..c04f9901f3 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/system/avocado_qemu/__init__.py
@@ -31,7 +31,7 @@
 BUILD_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
 
 if os.path.islink(os.path.dirname(os.path.dirname(__file__))):
-    # The link to the acceptance tests dir in the source code directory
+    # The link to the system tests dir in the source code directory
     lnk = os.path.dirname(os.path.dirname(__file__))
     #: The QEMU root source directory
     SOURCE_DIR = os.path.dirname(os.path.dirname(os.readlink(lnk)))
diff --git a/tests/acceptance/boot_linux.py b/tests/system/boot_linux.py
similarity index 100%
rename from tests/acceptance/boot_linux.py
rename to tests/system/boot_linux.py
diff --git a/tests/acceptance/boot_linux_console.py b/tests/system/boot_linux_console.py
similarity index 100%
rename from tests/acceptance/boot_linux_console.py
rename to tests/system/boot_linux_console.py
diff --git a/tests/acceptance/boot_xen.py b/tests/system/boot_xen.py
similarity index 100%
rename from tests/acceptance/boot_xen.py
rename to tests/system/boot_xen.py
diff --git a/tests/acceptance/cpu_queries.py b/tests/system/cpu_queries.py
similarity index 100%
rename from tests/acceptance/cpu_queries.py
rename to tests/system/cpu_queries.py
diff --git a/tests/acceptance/empty_cpu_model.py b/tests/system/empty_cpu_model.py
similarity index 100%
rename from tests/acceptance/empty_cpu_model.py
rename to tests/system/empty_cpu_model.py
diff --git a/tests/acceptance/linux_initrd.py b/tests/system/linux_initrd.py
similarity index 99%
rename from tests/acceptance/linux_initrd.py
rename to tests/system/linux_initrd.py
index a249e2f14a..351ca4959d 100644
--- a/tests/acceptance/linux_initrd.py
+++ b/tests/system/linux_initrd.py
@@ -1,4 +1,4 @@
-# Linux initrd acceptance test.
+# Linux initrd system test.
 #
 # Copyright (c) 2018 Red Hat, Inc.
 #
diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/system/linux_ssh_mips_malta.py
similarity index 100%
rename from tests/acceptance/linux_ssh_mips_malta.py
rename to tests/system/linux_ssh_mips_malta.py
diff --git a/tests/acceptance/machine_arm_canona1100.py b/tests/system/machine_arm_canona1100.py
similarity index 100%
rename from tests/acceptance/machine_arm_canona1100.py
rename to tests/system/machine_arm_canona1100.py
diff --git a/tests/acceptance/machine_arm_integratorcp.py b/tests/system/machine_arm_integratorcp.py
similarity index 100%
rename from tests/acceptance/machine_arm_integratorcp.py
rename to tests/system/machine_arm_integratorcp.py
diff --git a/tests/acceptance/machine_arm_n8x0.py b/tests/system/machine_arm_n8x0.py
similarity index 100%
rename from tests/acceptance/machine_arm_n8x0.py
rename to tests/system/machine_arm_n8x0.py
diff --git a/tests/acceptance/machine_avr6.py b/tests/system/machine_avr6.py
similarity index 98%
rename from tests/acceptance/machine_avr6.py
rename to tests/system/machine_avr6.py
index 6baf4e9c7f..acc5608776 100644
--- a/tests/acceptance/machine_avr6.py
+++ b/tests/system/machine_avr6.py
@@ -1,5 +1,5 @@
 #
-# QEMU AVR acceptance tests
+# QEMU AVR system tests
 #
 # Copyright (c) 2019-2020 Michael Rolnik <mrolnik@gmail.com>
 #
diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/system/machine_m68k_nextcube.py
similarity index 100%
rename from tests/acceptance/machine_m68k_nextcube.py
rename to tests/system/machine_m68k_nextcube.py
diff --git a/tests/acceptance/machine_microblaze.py b/tests/system/machine_microblaze.py
similarity index 100%
rename from tests/acceptance/machine_microblaze.py
rename to tests/system/machine_microblaze.py
diff --git a/tests/acceptance/machine_mips_loongson3v.py b/tests/system/machine_mips_loongson3v.py
similarity index 100%
rename from tests/acceptance/machine_mips_loongson3v.py
rename to tests/system/machine_mips_loongson3v.py
diff --git a/tests/acceptance/machine_mips_malta.py b/tests/system/machine_mips_malta.py
similarity index 100%
rename from tests/acceptance/machine_mips_malta.py
rename to tests/system/machine_mips_malta.py
diff --git a/tests/acceptance/machine_ppc.py b/tests/system/machine_ppc.py
similarity index 100%
rename from tests/acceptance/machine_ppc.py
rename to tests/system/machine_ppc.py
diff --git a/tests/acceptance/machine_rx_gdbsim.py b/tests/system/machine_rx_gdbsim.py
similarity index 100%
rename from tests/acceptance/machine_rx_gdbsim.py
rename to tests/system/machine_rx_gdbsim.py
diff --git a/tests/acceptance/machine_s390_ccw_virtio.py b/tests/system/machine_s390_ccw_virtio.py
similarity index 100%
rename from tests/acceptance/machine_s390_ccw_virtio.py
rename to tests/system/machine_s390_ccw_virtio.py
diff --git a/tests/acceptance/machine_sparc64_sun4u.py b/tests/system/machine_sparc64_sun4u.py
similarity index 100%
rename from tests/acceptance/machine_sparc64_sun4u.py
rename to tests/system/machine_sparc64_sun4u.py
diff --git a/tests/acceptance/machine_sparc_leon3.py b/tests/system/machine_sparc_leon3.py
similarity index 100%
rename from tests/acceptance/machine_sparc_leon3.py
rename to tests/system/machine_sparc_leon3.py
diff --git a/tests/acceptance/migration.py b/tests/system/migration.py
similarity index 100%
rename from tests/acceptance/migration.py
rename to tests/system/migration.py
diff --git a/tests/acceptance/multiprocess.py b/tests/system/multiprocess.py
similarity index 100%
rename from tests/acceptance/multiprocess.py
rename to tests/system/multiprocess.py
diff --git a/tests/acceptance/pc_cpu_hotplug_props.py b/tests/system/pc_cpu_hotplug_props.py
similarity index 100%
rename from tests/acceptance/pc_cpu_hotplug_props.py
rename to tests/system/pc_cpu_hotplug_props.py
diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/system/ppc_prep_40p.py
similarity index 100%
rename from tests/acceptance/ppc_prep_40p.py
rename to tests/system/ppc_prep_40p.py
diff --git a/tests/acceptance/replay_kernel.py b/tests/system/replay_kernel.py
similarity index 100%
rename from tests/acceptance/replay_kernel.py
rename to tests/system/replay_kernel.py
diff --git a/tests/acceptance/reverse_debugging.py b/tests/system/reverse_debugging.py
similarity index 100%
rename from tests/acceptance/reverse_debugging.py
rename to tests/system/reverse_debugging.py
diff --git a/tests/acceptance/tcg_plugins.py b/tests/system/tcg_plugins.py
similarity index 100%
rename from tests/acceptance/tcg_plugins.py
rename to tests/system/tcg_plugins.py
diff --git a/tests/acceptance/tesseract_utils.py b/tests/system/tesseract_utils.py
similarity index 100%
rename from tests/acceptance/tesseract_utils.py
rename to tests/system/tesseract_utils.py
diff --git a/tests/acceptance/version.py b/tests/system/version.py
similarity index 100%
rename from tests/acceptance/version.py
rename to tests/system/version.py
diff --git a/tests/acceptance/virtio-gpu.py b/tests/system/virtio-gpu.py
similarity index 100%
rename from tests/acceptance/virtio-gpu.py
rename to tests/system/virtio-gpu.py
diff --git a/tests/acceptance/virtio_check_params.py b/tests/system/virtio_check_params.py
similarity index 100%
rename from tests/acceptance/virtio_check_params.py
rename to tests/system/virtio_check_params.py
diff --git a/tests/acceptance/virtio_version.py b/tests/system/virtio_version.py
similarity index 100%
rename from tests/acceptance/virtio_version.py
rename to tests/system/virtio_version.py
diff --git a/tests/acceptance/virtiofs_submounts.py b/tests/system/virtiofs_submounts.py
similarity index 100%
rename from tests/acceptance/virtiofs_submounts.py
rename to tests/system/virtiofs_submounts.py
diff --git a/tests/acceptance/virtiofs_submounts.py.data/cleanup.sh b/tests/system/virtiofs_submounts.py.data/cleanup.sh
similarity index 100%
rename from tests/acceptance/virtiofs_submounts.py.data/cleanup.sh
rename to tests/system/virtiofs_submounts.py.data/cleanup.sh
diff --git a/tests/acceptance/virtiofs_submounts.py.data/guest-cleanup.sh b/tests/system/virtiofs_submounts.py.data/guest-cleanup.sh
similarity index 100%
rename from tests/acceptance/virtiofs_submounts.py.data/guest-cleanup.sh
rename to tests/system/virtiofs_submounts.py.data/guest-cleanup.sh
diff --git a/tests/acceptance/virtiofs_submounts.py.data/guest.sh b/tests/system/virtiofs_submounts.py.data/guest.sh
similarity index 100%
rename from tests/acceptance/virtiofs_submounts.py.data/guest.sh
rename to tests/system/virtiofs_submounts.py.data/guest.sh
diff --git a/tests/acceptance/virtiofs_submounts.py.data/host.sh b/tests/system/virtiofs_submounts.py.data/host.sh
similarity index 100%
rename from tests/acceptance/virtiofs_submounts.py.data/host.sh
rename to tests/system/virtiofs_submounts.py.data/host.sh
diff --git a/tests/acceptance/vnc.py b/tests/system/vnc.py
similarity index 100%
rename from tests/acceptance/vnc.py
rename to tests/system/vnc.py
diff --git a/tests/acceptance/x86_cpu_model_versions.py b/tests/system/x86_cpu_model_versions.py
similarity index 100%
rename from tests/acceptance/x86_cpu_model_versions.py
rename to tests/system/x86_cpu_model_versions.py
-- 
2.31.1

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Philippe Mathieu-Daudé]

On 5/20/21 9:53 PM, Willian Rampazzo wrote:
> Conceptually speaking, acceptance tests "are a series of specific tests
> conducted by the customer in an attempt to uncover product errors before
> accepting the software from the developer. Conducted by the end-user rather
> than software engineers, acceptance testing can range from an informal
> “test drive” to a planned and systematically executed series of scripted
> tests" [1]. Every time Pressman refers to the term "acceptance testing," he
> also refers to user's agreement in the final state of an implemented feature.
> Today, QEMU is not implementing user acceptance tests as described by Pressman.
> 
> There are other three possible terms we could use to describe what is currently
> QEMU "acceptance" tests:
> 
>   1 - Integration tests:
>       - "Integration testing is a systematic technique for constructing the
>          software architecture while at the same time conducting tests to
>          uncover errors associated with interfacing. The objective is to take
>          unit-tested components and build a program structure that has been
>          dictated by design." [2]
>       * Note: Sommerville does not have a clear definition of integration
>         testing. He refers to incremental integration of components inside
>         the system testing (see [3]).
> 
>   2 - Validation tests:
>       - "Validation testing begins at the culmination of integration testing,
>          when individual components have been exercised, the software is
>          completely assembled as a package, and interfacing errors have been
>          uncovered and corrected. At the validation or system level, the
>          distinction between different software categories disappears. Testing
>          focuses on user-visible actions and user-recognizable output from the
>          system." [4]
>       - "where you expect the system to perform correctly using a set of test
>          cases that reflect the system’s expected use." [5]
>       * Note: the definition of "validation testing" from Sommerville reflects
>         the same definition found around the Internet, as one of the processes
>         inside the "Verification & Validation (V&V)." In this concept,
>         validation testing is a high-level definition that covers unit testing,
>         functional testing, integration testing, system testing, and acceptance
>         testing.
> 
>   3 - System tests:
>       - "verifies that all elements mesh properly and that overall system
>          function and performance is achieved." [6]
>       - "involves integrating components to create a version of the system and
>          then testing the integrated system. System testing checks that
>          components are compatible, interact correctly, and transfer the right
>          data at the right time across their interfaces." [7]
> 
> The tests implemented inside the QEMU "acceptance" directory depend on the
> software completely assembled and, sometimes, on other elements, like operating
> system images. In this case, the proposal here is to rename the current
> "acceptance" directory to "system."

Are user-mode tests using Avocado also system tests?
https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html

> [1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
>     Practitioner’s Approach. p. 430.
> [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
>     Practitioner’s Approach. Software Engineering, p. 398.
> [3] Sommerville, Ian (2016). Software Engineering. p. 240-242.
> [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
>     Practitioner’s Approach. Software Engineering, p. 407.
> [5] Sommerville, Ian (2016). Software Engineering. p. 227.
> [6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
>     Practitioner’s Approach. Software Engineering, p. 377.
> [7] Sommerville, Ian (2016). Software Engineering. p. 240.
> 
> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> ---

> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index f718b61fa7..c5de3c9fd5 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -52,7 +52,7 @@ include:
>      # Avoid recompiling by hiding ninja with NINJA=":"
>      - make NINJA=":" $MAKE_CHECK_ARGS
>  
> -.acceptance_template: &acceptance_definition
> +.system_template: &system_definition

.system_test_template: &system_test_definition ?

> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 8f220e15d1..c580292bb5 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -16,7 +16,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),)
>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>  	@echo " $(MAKE) check-softfloat      Run FPU emulation tests"
>  endif
> -	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
> +	@echo " $(MAKE) check-system         Run all system tests"

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Thomas Huth]

On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote:
> On 5/20/21 9:53 PM, Willian Rampazzo wrote:
>> Conceptually speaking, acceptance tests "are a series of specific tests
>> conducted by the customer in an attempt to uncover product errors before
>> accepting the software from the developer. Conducted by the end-user rather
>> than software engineers, acceptance testing can range from an informal
>> “test drive” to a planned and systematically executed series of scripted
>> tests" [1]. Every time Pressman refers to the term "acceptance testing," he
>> also refers to user's agreement in the final state of an implemented feature.
>> Today, QEMU is not implementing user acceptance tests as described by Pressman.
>>
>> There are other three possible terms we could use to describe what is currently
>> QEMU "acceptance" tests:
>>
>>    1 - Integration tests:
>>        - "Integration testing is a systematic technique for constructing the
>>           software architecture while at the same time conducting tests to
>>           uncover errors associated with interfacing. The objective is to take
>>           unit-tested components and build a program structure that has been
>>           dictated by design." [2]
>>        * Note: Sommerville does not have a clear definition of integration
>>          testing. He refers to incremental integration of components inside
>>          the system testing (see [3]).

After thinking about this for a while, I agree with you that renaming the 
"acceptance" tests to "integration" tests is also not a good idea. When I 
hear "integration" test in the context of the virt stack, I'd rather expect 
a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe 
virt-manager on top and tests them all together. So we should look for a 
different name indeed.

>>    2 - Validation tests:
>>        - "Validation testing begins at the culmination of integration testing,
>>           when individual components have been exercised, the software is
>>           completely assembled as a package, and interfacing errors have been
>>           uncovered and corrected. At the validation or system level, the
>>           distinction between different software categories disappears. Testing
>>           focuses on user-visible actions and user-recognizable output from the
>>           system." [4]
>>        - "where you expect the system to perform correctly using a set of test
>>           cases that reflect the system’s expected use." [5]
>>        * Note: the definition of "validation testing" from Sommerville reflects
>>          the same definition found around the Internet, as one of the processes
>>          inside the "Verification & Validation (V&V)." In this concept,
>>          validation testing is a high-level definition that covers unit testing,
>>          functional testing, integration testing, system testing, and acceptance
>>          testing.
>>
>>    3 - System tests:
>>        - "verifies that all elements mesh properly and that overall system
>>           function and performance is achieved." [6]
>>        - "involves integrating components to create a version of the system and
>>           then testing the integrated system. System testing checks that
>>           components are compatible, interact correctly, and transfer the right
>>           data at the right time across their interfaces." [7]
>>
>> The tests implemented inside the QEMU "acceptance" directory depend on the
>> software completely assembled and, sometimes, on other elements, like operating
>> system images. In this case, the proposal here is to rename the current
>> "acceptance" directory to "system."
> 
> Are user-mode tests using Avocado also system tests?
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html

We've indeed got the problem that the word "system" is a little bit 
overloaded in the context of QEMU. We often talk about "system" when 
referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user 
emulator binaries). For example, the "--disable-system" switch of the 
configure script, or the "build-system" and "check-system" jobs in the 
.gitlab-ci.yml file ... thus this could get quite confusing in the 
.gitlab-ci.yml file afterwards.

So I think renaming "acceptance" to "system" is especially ok if we only 
keep the "softmmu"-related tests in that folder... would it maybe make sense 
to add the linux-user related tests in a separate folder called tests/user/ 
instead, Philippe? And we should likely rename the current build-system and 
check-system jobs in our gitlab-CI to build-softmmu and check-softmmu or so?

Alternatively, what about renaming the "acceptance" tests to "validation" 
instead? That word does not have a duplicated definition in the context of 
QEMU yet, so I think it would be less confusing.

  Thomas

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote:
> > On 5/20/21 9:53 PM, Willian Rampazzo wrote:
> >> Conceptually speaking, acceptance tests "are a series of specific tests
> >> conducted by the customer in an attempt to uncover product errors before
> >> accepting the software from the developer. Conducted by the end-user rather
> >> than software engineers, acceptance testing can range from an informal
> >> “test drive” to a planned and systematically executed series of scripted
> >> tests" [1]. Every time Pressman refers to the term "acceptance testing," he
> >> also refers to user's agreement in the final state of an implemented feature.
> >> Today, QEMU is not implementing user acceptance tests as described by Pressman.
> >>
> >> There are other three possible terms we could use to describe what is currently
> >> QEMU "acceptance" tests:
> >>
> >>    1 - Integration tests:
> >>        - "Integration testing is a systematic technique for constructing the
> >>           software architecture while at the same time conducting tests to
> >>           uncover errors associated with interfacing. The objective is to take
> >>           unit-tested components and build a program structure that has been
> >>           dictated by design." [2]
> >>        * Note: Sommerville does not have a clear definition of integration
> >>          testing. He refers to incremental integration of components inside
> >>          the system testing (see [3]).
>
> After thinking about this for a while, I agree with you that renaming the
> "acceptance" tests to "integration" tests is also not a good idea. When I
> hear "integration" test in the context of the virt stack, I'd rather expect
> a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe
> virt-manager on top and tests them all together. So we should look for a
> different name indeed.
>
> >>    2 - Validation tests:
> >>        - "Validation testing begins at the culmination of integration testing,
> >>           when individual components have been exercised, the software is
> >>           completely assembled as a package, and interfacing errors have been
> >>           uncovered and corrected. At the validation or system level, the
> >>           distinction between different software categories disappears. Testing
> >>           focuses on user-visible actions and user-recognizable output from the
> >>           system." [4]
> >>        - "where you expect the system to perform correctly using a set of test
> >>           cases that reflect the system’s expected use." [5]
> >>        * Note: the definition of "validation testing" from Sommerville reflects
> >>          the same definition found around the Internet, as one of the processes
> >>          inside the "Verification & Validation (V&V)." In this concept,
> >>          validation testing is a high-level definition that covers unit testing,
> >>          functional testing, integration testing, system testing, and acceptance
> >>          testing.
> >>
> >>    3 - System tests:
> >>        - "verifies that all elements mesh properly and that overall system
> >>           function and performance is achieved." [6]
> >>        - "involves integrating components to create a version of the system and
> >>           then testing the integrated system. System testing checks that
> >>           components are compatible, interact correctly, and transfer the right
> >>           data at the right time across their interfaces." [7]
> >>
> >> The tests implemented inside the QEMU "acceptance" directory depend on the
> >> software completely assembled and, sometimes, on other elements, like operating
> >> system images. In this case, the proposal here is to rename the current
> >> "acceptance" directory to "system."
> >
> > Are user-mode tests using Avocado also system tests?
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html
>
> We've indeed got the problem that the word "system" is a little bit
> overloaded in the context of QEMU. We often talk about "system" when
> referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user
> emulator binaries). For example, the "--disable-system" switch of the
> configure script, or the "build-system" and "check-system" jobs in the
> .gitlab-ci.yml file ... thus this could get quite confusing in the
> .gitlab-ci.yml file afterwards.

I agree with you here. After I made the changes to the code, I noticed
QEMU has the "system" word spread all over the place. That may confuse
people looking at the "system tests" without much interaction with
software testing terminology.

>
> So I think renaming "acceptance" to "system" is especially ok if we only
> keep the "softmmu"-related tests in that folder... would it maybe make sense
> to add the linux-user related tests in a separate folder called tests/user/
> instead, Philippe? And we should likely rename the current build-system and
> check-system jobs in our gitlab-CI to build-softmmu and check-softmmu or so?
>

As I mentioned in Philippe's reply, those tests are still considered
system tests because system testing is the software built and
interacting with external test artifacts in software engineering.

> Alternatively, what about renaming the "acceptance" tests to "validation"
> instead? That word does not have a duplicated definition in the context of
> QEMU yet, so I think it would be less confusing.

While at the beginning of your reply, I started thinking if
"validation" would cause less confusion for the QEMU project. Although
validation testing is a broader concept inside the Verification &
Validation process, encompassing unit testing, functional testing,
integration testing, system testing, and acceptance testing, it may be
an option for the QEMU project.

While system testing would be the correct terminology to use, if it
causes more confusion, using a less strict terminology, like
validation testing, is valid, in my opinion.

>
>   Thomas
>

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Philippe Mathieu-Daudé]

On 5/21/21 2:28 PM, Willian Rampazzo wrote:
> On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote:
>>> On 5/20/21 9:53 PM, Willian Rampazzo wrote:
>>>> Conceptually speaking, acceptance tests "are a series of specific tests
>>>> conducted by the customer in an attempt to uncover product errors before
>>>> accepting the software from the developer. Conducted by the end-user rather
>>>> than software engineers, acceptance testing can range from an informal
>>>> “test drive” to a planned and systematically executed series of scripted
>>>> tests" [1]. Every time Pressman refers to the term "acceptance testing," he
>>>> also refers to user's agreement in the final state of an implemented feature.
>>>> Today, QEMU is not implementing user acceptance tests as described by Pressman.
>>>>
>>>> There are other three possible terms we could use to describe what is currently
>>>> QEMU "acceptance" tests:
>>>>
>>>>    1 - Integration tests:
>>>>        - "Integration testing is a systematic technique for constructing the
>>>>           software architecture while at the same time conducting tests to
>>>>           uncover errors associated with interfacing. The objective is to take
>>>>           unit-tested components and build a program structure that has been
>>>>           dictated by design." [2]
>>>>        * Note: Sommerville does not have a clear definition of integration
>>>>          testing. He refers to incremental integration of components inside
>>>>          the system testing (see [3]).
>>
>> After thinking about this for a while, I agree with you that renaming the
>> "acceptance" tests to "integration" tests is also not a good idea. When I
>> hear "integration" test in the context of the virt stack, I'd rather expect
>> a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe
>> virt-manager on top and tests them all together. So we should look for a
>> different name indeed.
>>
>>>>    2 - Validation tests:
>>>>        - "Validation testing begins at the culmination of integration testing,
>>>>           when individual components have been exercised, the software is
>>>>           completely assembled as a package, and interfacing errors have been
>>>>           uncovered and corrected. At the validation or system level, the
>>>>           distinction between different software categories disappears. Testing
>>>>           focuses on user-visible actions and user-recognizable output from the
>>>>           system." [4]
>>>>        - "where you expect the system to perform correctly using a set of test
>>>>           cases that reflect the system’s expected use." [5]
>>>>        * Note: the definition of "validation testing" from Sommerville reflects
>>>>          the same definition found around the Internet, as one of the processes
>>>>          inside the "Verification & Validation (V&V)." In this concept,
>>>>          validation testing is a high-level definition that covers unit testing,
>>>>          functional testing, integration testing, system testing, and acceptance
>>>>          testing.
>>>>
>>>>    3 - System tests:
>>>>        - "verifies that all elements mesh properly and that overall system
>>>>           function and performance is achieved." [6]
>>>>        - "involves integrating components to create a version of the system and
>>>>           then testing the integrated system. System testing checks that
>>>>           components are compatible, interact correctly, and transfer the right
>>>>           data at the right time across their interfaces." [7]
>>>>
>>>> The tests implemented inside the QEMU "acceptance" directory depend on the
>>>> software completely assembled and, sometimes, on other elements, like operating
>>>> system images. In this case, the proposal here is to rename the current
>>>> "acceptance" directory to "system."
>>>
>>> Are user-mode tests using Avocado also system tests?
>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html
>>
>> We've indeed got the problem that the word "system" is a little bit
>> overloaded in the context of QEMU. We often talk about "system" when
>> referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user
>> emulator binaries). For example, the "--disable-system" switch of the
>> configure script, or the "build-system" and "check-system" jobs in the
>> .gitlab-ci.yml file ... thus this could get quite confusing in the
>> .gitlab-ci.yml file afterwards.
> 
> I agree with you here. After I made the changes to the code, I noticed
> QEMU has the "system" word spread all over the place. That may confuse
> people looking at the "system tests" without much interaction with
> software testing terminology.
> 
>>
>> So I think renaming "acceptance" to "system" is especially ok if we only
>> keep the "softmmu"-related tests in that folder... would it maybe make sense
>> to add the linux-user related tests in a separate folder called tests/user/
>> instead, Philippe? And we should likely rename the current build-system and
>> check-system jobs in our gitlab-CI to build-softmmu and check-softmmu or so?
>>
> 
> As I mentioned in Philippe's reply, those tests are still considered
> system tests because system testing is the software built and
> interacting with external test artifacts in software engineering.
> 
>> Alternatively, what about renaming the "acceptance" tests to "validation"
>> instead? That word does not have a duplicated definition in the context of
>> QEMU yet, so I think it would be less confusing.
> 
> While at the beginning of your reply, I started thinking if
> "validation" would cause less confusion for the QEMU project. Although
> validation testing is a broader concept inside the Verification &
> Validation process, encompassing unit testing, functional testing,
> integration testing, system testing, and acceptance testing, it may be
> an option for the QEMU project.
> 
> While system testing would be the correct terminology to use, if it
> causes more confusion, using a less strict terminology, like
> validation testing, is valid, in my opinion.

This works for me:

- tests/system/softmmu
- tests/system/user

Or validation, as you prefer.

Thanks for sharing the background references,

Phil.

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> On 5/21/21 2:28 PM, Willian Rampazzo wrote:
>> On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
>>>
>>> On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote:
>>>> On 5/20/21 9:53 PM, Willian Rampazzo wrote:
>>>>> Conceptually speaking, acceptance tests "are a series of specific tests
>>>>> conducted by the customer in an attempt to uncover product errors before
>>>>> accepting the software from the developer. Conducted by the end-user rather
>>>>> than software engineers, acceptance testing can range from an informal
>>>>> “test drive” to a planned and systematically executed series of scripted
>>>>> tests" [1]. Every time Pressman refers to the term "acceptance testing," he
>>>>> also refers to user's agreement in the final state of an implemented feature.
>>>>> Today, QEMU is not implementing user acceptance tests as described by Pressman.
>>>>>
>>>>> There are other three possible terms we could use to describe what is currently
>>>>> QEMU "acceptance" tests:
>>>>>
>>>>>    1 - Integration tests:
>>>>>        - "Integration testing is a systematic technique for constructing the
>>>>>           software architecture while at the same time conducting tests to
>>>>>           uncover errors associated with interfacing. The objective is to take
>>>>>           unit-tested components and build a program structure that has been
>>>>>           dictated by design." [2]
>>>>>        * Note: Sommerville does not have a clear definition of integration
>>>>>          testing. He refers to incremental integration of components inside
>>>>>          the system testing (see [3]).
>>>
>>> After thinking about this for a while, I agree with you that renaming the
>>> "acceptance" tests to "integration" tests is also not a good idea. When I
>>> hear "integration" test in the context of the virt stack, I'd rather expect
>>> a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe
>>> virt-manager on top and tests them all together. So we should look for a
>>> different name indeed.
>>>
>>>>>    2 - Validation tests:
>>>>>        - "Validation testing begins at the culmination of integration testing,
>>>>>           when individual components have been exercised, the software is
>>>>>           completely assembled as a package, and interfacing errors have been
>>>>>           uncovered and corrected. At the validation or system level, the
>>>>>           distinction between different software categories disappears. Testing
>>>>>           focuses on user-visible actions and user-recognizable output from the
>>>>>           system." [4]
>>>>>        - "where you expect the system to perform correctly using a set of test
>>>>>           cases that reflect the system’s expected use." [5]
>>>>>        * Note: the definition of "validation testing" from Sommerville reflects
>>>>>          the same definition found around the Internet, as one of the processes
>>>>>          inside the "Verification & Validation (V&V)." In this concept,
>>>>>          validation testing is a high-level definition that covers unit testing,
>>>>>          functional testing, integration testing, system testing, and acceptance
>>>>>          testing.
>>>>>
>>>>>    3 - System tests:
>>>>>        - "verifies that all elements mesh properly and that overall system
>>>>>           function and performance is achieved." [6]
>>>>>        - "involves integrating components to create a version of the system and
>>>>>           then testing the integrated system. System testing checks that
>>>>>           components are compatible, interact correctly, and transfer the right
>>>>>           data at the right time across their interfaces." [7]
>>>>>
>>>>> The tests implemented inside the QEMU "acceptance" directory depend on the
>>>>> software completely assembled and, sometimes, on other elements, like operating
>>>>> system images. In this case, the proposal here is to rename the current
>>>>> "acceptance" directory to "system."
>>>>
>>>> Are user-mode tests using Avocado also system tests?
>>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html
>>>
>>> We've indeed got the problem that the word "system" is a little bit
>>> overloaded in the context of QEMU. We often talk about "system" when
>>> referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user
>>> emulator binaries). For example, the "--disable-system" switch of the
>>> configure script, or the "build-system" and "check-system" jobs in the
>>> .gitlab-ci.yml file ... thus this could get quite confusing in the
>>> .gitlab-ci.yml file afterwards.
>> 
>> I agree with you here. After I made the changes to the code, I noticed
>> QEMU has the "system" word spread all over the place. That may confuse
>> people looking at the "system tests" without much interaction with
>> software testing terminology.
>> 
>>>
>>> So I think renaming "acceptance" to "system" is especially ok if we only
>>> keep the "softmmu"-related tests in that folder... would it maybe make sense
>>> to add the linux-user related tests in a separate folder called tests/user/
>>> instead, Philippe? And we should likely rename the current build-system and
>>> check-system jobs in our gitlab-CI to build-softmmu and check-softmmu or so?
>>>
>> 
>> As I mentioned in Philippe's reply, those tests are still considered
>> system tests because system testing is the software built and
>> interacting with external test artifacts in software engineering.
>> 
>>> Alternatively, what about renaming the "acceptance" tests to "validation"
>>> instead? That word does not have a duplicated definition in the context of
>>> QEMU yet, so I think it would be less confusing.
>> 
>> While at the beginning of your reply, I started thinking if
>> "validation" would cause less confusion for the QEMU project. Although
>> validation testing is a broader concept inside the Verification &
>> Validation process, encompassing unit testing, functional testing,
>> integration testing, system testing, and acceptance testing, it may be
>> an option for the QEMU project.
>> 
>> While system testing would be the correct terminology to use, if it
>> causes more confusion, using a less strict terminology, like
>> validation testing, is valid, in my opinion.
>
> This works for me:
>
> - tests/system/softmmu
> - tests/system/user
>
> Or validation, as you prefer.

So what are tests/tcg if not user tests? They *mostly* test
linux-user emulation but of course we have softmmu tests in there as
well. 

>
> Thanks for sharing the background references,
>
> Phil.


-- 
Alex Bennée

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Philippe Mathieu-Daudé]

On 5/21/21 3:03 PM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>> On 5/21/21 2:28 PM, Willian Rampazzo wrote:
>>> On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
>>>>
>>>> On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote:
>>>>> On 5/20/21 9:53 PM, Willian Rampazzo wrote:
>>>>>> Conceptually speaking, acceptance tests "are a series of specific tests
>>>>>> conducted by the customer in an attempt to uncover product errors before
>>>>>> accepting the software from the developer. Conducted by the end-user rather
>>>>>> than software engineers, acceptance testing can range from an informal
>>>>>> “test drive” to a planned and systematically executed series of scripted
>>>>>> tests" [1]. Every time Pressman refers to the term "acceptance testing," he
>>>>>> also refers to user's agreement in the final state of an implemented feature.
>>>>>> Today, QEMU is not implementing user acceptance tests as described by Pressman.
>>>>>>
>>>>>> There are other three possible terms we could use to describe what is currently
>>>>>> QEMU "acceptance" tests:
>>>>>>
>>>>>>    1 - Integration tests:
>>>>>>        - "Integration testing is a systematic technique for constructing the
>>>>>>           software architecture while at the same time conducting tests to
>>>>>>           uncover errors associated with interfacing. The objective is to take
>>>>>>           unit-tested components and build a program structure that has been
>>>>>>           dictated by design." [2]
>>>>>>        * Note: Sommerville does not have a clear definition of integration
>>>>>>          testing. He refers to incremental integration of components inside
>>>>>>          the system testing (see [3]).
>>>>
>>>> After thinking about this for a while, I agree with you that renaming the
>>>> "acceptance" tests to "integration" tests is also not a good idea. When I
>>>> hear "integration" test in the context of the virt stack, I'd rather expect
>>>> a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe
>>>> virt-manager on top and tests them all together. So we should look for a
>>>> different name indeed.
>>>>
>>>>>>    2 - Validation tests:
>>>>>>        - "Validation testing begins at the culmination of integration testing,
>>>>>>           when individual components have been exercised, the software is
>>>>>>           completely assembled as a package, and interfacing errors have been
>>>>>>           uncovered and corrected. At the validation or system level, the
>>>>>>           distinction between different software categories disappears. Testing
>>>>>>           focuses on user-visible actions and user-recognizable output from the
>>>>>>           system." [4]
>>>>>>        - "where you expect the system to perform correctly using a set of test
>>>>>>           cases that reflect the system’s expected use." [5]
>>>>>>        * Note: the definition of "validation testing" from Sommerville reflects
>>>>>>          the same definition found around the Internet, as one of the processes
>>>>>>          inside the "Verification & Validation (V&V)." In this concept,
>>>>>>          validation testing is a high-level definition that covers unit testing,
>>>>>>          functional testing, integration testing, system testing, and acceptance
>>>>>>          testing.
>>>>>>
>>>>>>    3 - System tests:
>>>>>>        - "verifies that all elements mesh properly and that overall system
>>>>>>           function and performance is achieved." [6]
>>>>>>        - "involves integrating components to create a version of the system and
>>>>>>           then testing the integrated system. System testing checks that
>>>>>>           components are compatible, interact correctly, and transfer the right
>>>>>>           data at the right time across their interfaces." [7]
>>>>>>
>>>>>> The tests implemented inside the QEMU "acceptance" directory depend on the
>>>>>> software completely assembled and, sometimes, on other elements, like operating
>>>>>> system images. In this case, the proposal here is to rename the current
>>>>>> "acceptance" directory to "system."
>>>>>
>>>>> Are user-mode tests using Avocado also system tests?
>>>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html
>>>>
>>>> We've indeed got the problem that the word "system" is a little bit
>>>> overloaded in the context of QEMU. We often talk about "system" when
>>>> referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user
>>>> emulator binaries). For example, the "--disable-system" switch of the
>>>> configure script, or the "build-system" and "check-system" jobs in the
>>>> .gitlab-ci.yml file ... thus this could get quite confusing in the
>>>> .gitlab-ci.yml file afterwards.
>>>
>>> I agree with you here. After I made the changes to the code, I noticed
>>> QEMU has the "system" word spread all over the place. That may confuse
>>> people looking at the "system tests" without much interaction with
>>> software testing terminology.
>>>
>>>>
>>>> So I think renaming "acceptance" to "system" is especially ok if we only
>>>> keep the "softmmu"-related tests in that folder... would it maybe make sense
>>>> to add the linux-user related tests in a separate folder called tests/user/
>>>> instead, Philippe? And we should likely rename the current build-system and
>>>> check-system jobs in our gitlab-CI to build-softmmu and check-softmmu or so?
>>>>
>>>
>>> As I mentioned in Philippe's reply, those tests are still considered
>>> system tests because system testing is the software built and
>>> interacting with external test artifacts in software engineering.
>>>
>>>> Alternatively, what about renaming the "acceptance" tests to "validation"
>>>> instead? That word does not have a duplicated definition in the context of
>>>> QEMU yet, so I think it would be less confusing.
>>>
>>> While at the beginning of your reply, I started thinking if
>>> "validation" would cause less confusion for the QEMU project. Although
>>> validation testing is a broader concept inside the Verification &
>>> Validation process, encompassing unit testing, functional testing,
>>> integration testing, system testing, and acceptance testing, it may be
>>> an option for the QEMU project.
>>>
>>> While system testing would be the correct terminology to use, if it
>>> causes more confusion, using a less strict terminology, like
>>> validation testing, is valid, in my opinion.
>>
>> This works for me:
>>
>> - tests/system/softmmu
>> - tests/system/user
>>
>> Or validation, as you prefer.
> 
> So what are tests/tcg if not user tests? They *mostly* test
> linux-user emulation but of course we have softmmu tests in there as
> well. 

I expect a tests/tcg/ to check a specific TCG feature, which doesn't
have to be user-mode specific (IIRC Xtensa does some sysemu checks).
Also, you control the compiler toolchain, flags, etc... so you can
adapt for a specific feature bit to test, use kludges and so on.

I expect tests in tests/system/ (user/softmmu) to user real-world
binaries, which we aren't modifying. Sometime non-public/released
compiler toolchain has been used.

See for example the test referred tests the bFLT loader (beside
testing userland Linux binary for Cortex-M).

Another example is the Sony PlayStation2 binary testing the
O32 ABI and multiple opcodes from the TX79 SIMD core:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg782493.html

Personally I'm not interested in writing a test for a loader or
multiple opcodes when we have pre-built binaries. For the opcodes
coverage I'd use a TCG plugin to confirm the opcodes have been
used.

If you think these tests belong to tests/tcg/, I am OK to put
them they, but I don't think adding the Avocado buildsys
machinery to the already-complex tests/tcg/ Makefiles is going
to help us...

Regards,

Phil.

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Peter Maydell]

On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> If you think these tests belong to tests/tcg/, I am OK to put
> them they, but I don't think adding the Avocado buildsys
> machinery to the already-complex tests/tcg/ Makefiles is going
> to help us...

This does raise the question of what we're actually trying
to distinguish. It seems to me somewhat that what tests/acceptance/
actually contains that makes it interestingly different from other
tests/ stuff is that it's specifically "tests using the Avocado
framework". On that theory we might name it tests/avocado/.

Or we could just leave it as it is -- is the current naming
actually confusing anybody? :-)

thanks
-- PMM

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

On Fri, May 21, 2021 at 11:29 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> > If you think these tests belong to tests/tcg/, I am OK to put
> > them they, but I don't think adding the Avocado buildsys
> > machinery to the already-complex tests/tcg/ Makefiles is going
> > to help us...
>
> This does raise the question of what we're actually trying
> to distinguish. It seems to me somewhat that what tests/acceptance/
> actually contains that makes it interestingly different from other
> tests/ stuff is that it's specifically "tests using the Avocado
> framework". On that theory we might name it tests/avocado/.
>
> Or we could just leave it as it is -- is the current naming
> actually confusing anybody? :-)

Conceptually speaking, it isn't very clear because acceptance tests
are meant to test features requested by the customer during the
requisite phase for a specific software release. The QEMU project does
not have a formal requisite phase where features added to one software
version are discussed with the customers. In this case, the QEMU
acceptance tests are not accepting a use case or requested feature of
a release. Eventually, those tests may not be a blocker to a release
if some of them fail. Thus, calling them acceptance, may confuse
people working with tests.

>
> thanks
> -- PMM
>

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

On Fri, May 21, 2021 at 11:29 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> > If you think these tests belong to tests/tcg/, I am OK to put
> > them they, but I don't think adding the Avocado buildsys
> > machinery to the already-complex tests/tcg/ Makefiles is going
> > to help us...
>
> This does raise the question of what we're actually trying
> to distinguish. It seems to me somewhat that what tests/acceptance/
> actually contains that makes it interestingly different from other
> tests/ stuff is that it's specifically "tests using the Avocado
> framework". On that theory we might name it tests/avocado/.

I think the updated README.rst from this RFC, inside the system
(originally acceptance) folder, is a good description of what these
tests should be: "This directory contains system tests. They're
usually higher level, and may interact with external resources and
with various guest operating systems." I can improve it, if needed.

We are using Avocado Framework as a tool to accomplish the above
description, but I don't think we should strictly use it if there is
another way to accomplish what those tests are supposed to be. Calling
them "avocado" tests may restrict the intent of them, in my opinion.

>
> Or we could just leave it as it is -- is the current naming
> actually confusing anybody? :-)
>
> thanks
> -- PMM
>

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Peter Maydell]

On Fri, 21 May 2021 at 16:13, Willian Rampazzo <wrampazz@redhat.com> wrote:
> On Fri, May 21, 2021 at 11:29 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > This does raise the question of what we're actually trying
> > to distinguish. It seems to me somewhat that what tests/acceptance/
> > actually contains that makes it interestingly different from other
> > tests/ stuff is that it's specifically "tests using the Avocado
> > framework". On that theory we might name it tests/avocado/.
>
> I think the updated README.rst from this RFC, inside the system
> (originally acceptance) folder, is a good description of what these
> tests should be: "This directory contains system tests. They're
> usually higher level, and may interact with external resources and
> with various guest operating systems." I can improve it, if needed.
>
> We are using Avocado Framework as a tool to accomplish the above
> description, but I don't think we should strictly use it if there is
> another way to accomplish what those tests are supposed to be. Calling
> them "avocado" tests may restrict the intent of them, in my opinion.

But the main reason IMHO that we have them in a separate directory is
because that's where we have all the avocado machinery. I think the
sharing of the machinery is what dictates whether a test winds up in
tests/acceptance, tests/qtest, tests/tcg or tests/qemu-iotests
much more than whether it is "usually higher level" or more of a
unit test or whatever. If we ever added some other test framework for
doing system tests, we'd probably want to put it in its own
directory rather than lumping all its support machinery and
build files in together with the avocado based tests.

thanks
-- PMM

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

On Fri, May 21, 2021 at 12:23 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Fri, 21 May 2021 at 16:13, Willian Rampazzo <wrampazz@redhat.com> wrote:
> > On Fri, May 21, 2021 at 11:29 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > > This does raise the question of what we're actually trying
> > > to distinguish. It seems to me somewhat that what tests/acceptance/
> > > actually contains that makes it interestingly different from other
> > > tests/ stuff is that it's specifically "tests using the Avocado
> > > framework". On that theory we might name it tests/avocado/.
> >
> > I think the updated README.rst from this RFC, inside the system
> > (originally acceptance) folder, is a good description of what these
> > tests should be: "This directory contains system tests. They're
> > usually higher level, and may interact with external resources and
> > with various guest operating systems." I can improve it, if needed.
> >
> > We are using Avocado Framework as a tool to accomplish the above
> > description, but I don't think we should strictly use it if there is
> > another way to accomplish what those tests are supposed to be. Calling
> > them "avocado" tests may restrict the intent of them, in my opinion.
>
> But the main reason IMHO that we have them in a separate directory is
> because that's where we have all the avocado machinery. I think the
> sharing of the machinery is what dictates whether a test winds up in
> tests/acceptance, tests/qtest, tests/tcg or tests/qemu-iotests
> much more than whether it is "usually higher level" or more of a
> unit test or whatever. If we ever added some other test framework for
> doing system tests, we'd probably want to put it in its own
> directory rather than lumping all its support machinery and
> build files in together with the avocado based tests.

Okay, I understand your point. With the organization of tests under
the test folder that QEMU has today, I agree with you.

>
> thanks
> -- PMM
>

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Thomas Huth]

On 21/05/2021 16.29, Peter Maydell wrote:
> On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>> If you think these tests belong to tests/tcg/, I am OK to put
>> them they, but I don't think adding the Avocado buildsys
>> machinery to the already-complex tests/tcg/ Makefiles is going
>> to help us...
> 
> This does raise the question of what we're actually trying
> to distinguish. It seems to me somewhat that what tests/acceptance/
> actually contains that makes it interestingly different from other
> tests/ stuff is that it's specifically "tests using the Avocado
> framework". On that theory we might name it tests/avocado/.

I think there are two aspects:

1) These tests are using the avocado framework

2) These tests are downloading other stuff from the internet (unlike the 
other tests that we have)

> Or we could just leave it as it is -- is the current naming
> actually confusing anybody? :-)

Yes, I think "acceptance" is rather confusing. So far they haven't been part 
of your PR acceptance tests (well, now they are part of the gitlab-CI, 
though), and it's also not about tests that have been set up by customers, 
which is what you normally think of when hearing "acceptance tests". So a 
different name would be adequate.

I think I'd vote for either "avocado", "avoqado" or "validation".

  Thomas

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Philippe Mathieu-Daudé]

On 5/21/21 7:14 PM, Thomas Huth wrote:
> On 21/05/2021 16.29, Peter Maydell wrote:
>> On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org>
>> wrote:
>>> If you think these tests belong to tests/tcg/, I am OK to put
>>> them they, but I don't think adding the Avocado buildsys
>>> machinery to the already-complex tests/tcg/ Makefiles is going
>>> to help us...
>>
>> This does raise the question of what we're actually trying
>> to distinguish. It seems to me somewhat that what tests/acceptance/
>> actually contains that makes it interestingly different from other
>> tests/ stuff is that it's specifically "tests using the Avocado
>> framework". On that theory we might name it tests/avocado/.
> 
> I think there are two aspects:
> 
> 1) These tests are using the avocado framework
> 
> 2) These tests are downloading other stuff from the internet (unlike the
> other tests that we have)
> 
>> Or we could just leave it as it is -- is the current naming
>> actually confusing anybody? :-)
> 
> Yes, I think "acceptance" is rather confusing. So far they haven't been
> part of your PR acceptance tests (well, now they are part of the
> gitlab-CI, though), and it's also not about tests that have been set up
> by customers, which is what you normally think of when hearing
> "acceptance tests". So a different name would be adequate.

IIUC the current "acceptance tests" are the ones Peter runs,
which are *gating* the merge process. They can not fail.

The current tests in tests/acceptance/ use Avocado (as said Thomas,
to easily download artifacts) and shouldn't be considered gating;
they could fail.

This is my confusion so far. It should be OK to add tests using the
Avocado framework which might fail and aren't gating.

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

On Fri, May 21, 2021 at 2:14 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 21/05/2021 16.29, Peter Maydell wrote:
> > On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> >> If you think these tests belong to tests/tcg/, I am OK to put
> >> them they, but I don't think adding the Avocado buildsys
> >> machinery to the already-complex tests/tcg/ Makefiles is going
> >> to help us...
> >
> > This does raise the question of what we're actually trying
> > to distinguish. It seems to me somewhat that what tests/acceptance/
> > actually contains that makes it interestingly different from other
> > tests/ stuff is that it's specifically "tests using the Avocado
> > framework". On that theory we might name it tests/avocado/.
>
> I think there are two aspects:
>
> 1) These tests are using the avocado framework
>
> 2) These tests are downloading other stuff from the internet (unlike the
> other tests that we have)
>

After Peter's reply, I noticed QEMU does not organize tests under the
tests folder by software engineering test category but by the
mechanism/machinery the tests run on. This makes me think that we may
need to handle the folders name and the CI jobs name differently:

1 - Change the current "test/acceptance" folder name to "test/(avocado
or avoqado)." Change the "make check-acceptance" to "make
check-validation," and the GitLab CI job names to "validation,"
meaning that, in a promising future, other tests running on a
different framework and acting like validation tests would run in the
same make command and same GitLab CI job.

2 - Change the current "test/acceptance" folder name to "test/(avocado
or avoqado)." Change the "make check-acceptance" to "make
check-(avocado or avoqaco)" and the GitLab CI job names to "(avocado
or avoqado)," meaning that, in a promising future, we can categorize
validation jobs inside the CI and run each of the different validation
tests supported by a framework on its own GitLab CI job.

Personally, I prefer option 2 as it gives more flexibility to decide
how to set a GitLab CI job or run it when testing locally.

> > Or we could just leave it as it is -- is the current naming
> > actually confusing anybody? :-)
>
> Yes, I think "acceptance" is rather confusing. So far they haven't been part
> of your PR acceptance tests (well, now they are part of the gitlab-CI,
> though), and it's also not about tests that have been set up by customers,
> which is what you normally think of when hearing "acceptance tests". So a
> different name would be adequate.
>
> I think I'd vote for either "avocado", "avoqado" or "validation".
>

Even laughing every time I read "avoqado" (and thanks for that), I
liked the idea as there is supplementary code added inside
"tests/acceptance/avocado_qemu" to support the tests, meaning they are
not "pure" avocado.

>   Thomas
>

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 5/21/21 3:03 PM, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>> On 5/21/21 2:28 PM, Willian Rampazzo wrote:
>>>> On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
>>>>>
>>>>> On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote:
>>>>>> On 5/20/21 9:53 PM, Willian Rampazzo wrote:
>>>>>>> Conceptually speaking, acceptance tests "are a series of specific tests
>>>>>>> conducted by the customer in an attempt to uncover product errors before
>>>>>>> accepting the software from the developer. Conducted by the end-user rather
>>>>>>> than software engineers, acceptance testing can range from an informal
>>>>>>> “test drive” to a planned and systematically executed series of scripted
>>>>>>> tests" [1]. Every time Pressman refers to the term "acceptance testing," he
>>>>>>> also refers to user's agreement in the final state of an implemented feature.
>>>>>>> Today, QEMU is not implementing user acceptance tests as described by Pressman.
>>>>>>>
>>>>>>> There are other three possible terms we could use to describe what is currently
>>>>>>> QEMU "acceptance" tests:
>>>>>>>
>>>>>>>    1 - Integration tests:
>>>>>>>        - "Integration testing is a systematic technique for constructing the
>>>>>>>           software architecture while at the same time conducting tests to
>>>>>>>           uncover errors associated with interfacing. The objective is to take
>>>>>>>           unit-tested components and build a program structure that has been
>>>>>>>           dictated by design." [2]
>>>>>>>        * Note: Sommerville does not have a clear definition of integration
>>>>>>>          testing. He refers to incremental integration of components inside
>>>>>>>          the system testing (see [3]).
>>>>>
>>>>> After thinking about this for a while, I agree with you that renaming the
>>>>> "acceptance" tests to "integration" tests is also not a good idea. When I
>>>>> hear "integration" test in the context of the virt stack, I'd rather expect
>>>>> a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe
>>>>> virt-manager on top and tests them all together. So we should look for a
>>>>> different name indeed.
>>>>>
>>>>>>>    2 - Validation tests:
>>>>>>>        - "Validation testing begins at the culmination of integration testing,
>>>>>>>           when individual components have been exercised, the software is
>>>>>>>           completely assembled as a package, and interfacing errors have been
>>>>>>>           uncovered and corrected. At the validation or system level, the
>>>>>>>           distinction between different software categories disappears. Testing
>>>>>>>           focuses on user-visible actions and user-recognizable output from the
>>>>>>>           system." [4]
>>>>>>>        - "where you expect the system to perform correctly using a set of test
>>>>>>>           cases that reflect the system’s expected use." [5]
>>>>>>>        * Note: the definition of "validation testing" from Sommerville reflects
>>>>>>>          the same definition found around the Internet, as one of the processes
>>>>>>>          inside the "Verification & Validation (V&V)." In this concept,
>>>>>>>          validation testing is a high-level definition that covers unit testing,
>>>>>>>          functional testing, integration testing, system testing, and acceptance
>>>>>>>          testing.
>>>>>>>
>>>>>>>    3 - System tests:
>>>>>>>        - "verifies that all elements mesh properly and that overall system
>>>>>>>           function and performance is achieved." [6]
>>>>>>>        - "involves integrating components to create a version of the system and
>>>>>>>           then testing the integrated system. System testing checks that
>>>>>>>           components are compatible, interact correctly, and transfer the right
>>>>>>>           data at the right time across their interfaces." [7]
>>>>>>>
>>>>>>> The tests implemented inside the QEMU "acceptance" directory depend on the
>>>>>>> software completely assembled and, sometimes, on other elements, like operating
>>>>>>> system images. In this case, the proposal here is to rename the current
>>>>>>> "acceptance" directory to "system."
>>>>>>
>>>>>> Are user-mode tests using Avocado also system tests?
>>>>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html
>>>>>
>>>>> We've indeed got the problem that the word "system" is a little bit
>>>>> overloaded in the context of QEMU. We often talk about "system" when
>>>>> referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user
>>>>> emulator binaries). For example, the "--disable-system" switch of the
>>>>> configure script, or the "build-system" and "check-system" jobs in the
>>>>> .gitlab-ci.yml file ... thus this could get quite confusing in the
>>>>> .gitlab-ci.yml file afterwards.
>>>>
>>>> I agree with you here. After I made the changes to the code, I noticed
>>>> QEMU has the "system" word spread all over the place. That may confuse
>>>> people looking at the "system tests" without much interaction with
>>>> software testing terminology.
>>>>
>>>>>
>>>>> So I think renaming "acceptance" to "system" is especially ok if we only
>>>>> keep the "softmmu"-related tests in that folder... would it maybe make sense
>>>>> to add the linux-user related tests in a separate folder called tests/user/
>>>>> instead, Philippe? And we should likely rename the current build-system and
>>>>> check-system jobs in our gitlab-CI to build-softmmu and check-softmmu or so?
>>>>>
>>>>
>>>> As I mentioned in Philippe's reply, those tests are still considered
>>>> system tests because system testing is the software built and
>>>> interacting with external test artifacts in software engineering.
>>>>
>>>>> Alternatively, what about renaming the "acceptance" tests to "validation"
>>>>> instead? That word does not have a duplicated definition in the context of
>>>>> QEMU yet, so I think it would be less confusing.
>>>>
>>>> While at the beginning of your reply, I started thinking if
>>>> "validation" would cause less confusion for the QEMU project. Although
>>>> validation testing is a broader concept inside the Verification &
>>>> Validation process, encompassing unit testing, functional testing,
>>>> integration testing, system testing, and acceptance testing, it may be
>>>> an option for the QEMU project.
>>>>
>>>> While system testing would be the correct terminology to use, if it
>>>> causes more confusion, using a less strict terminology, like
>>>> validation testing, is valid, in my opinion.
>>>
>>> This works for me:
>>>
>>> - tests/system/softmmu
>>> - tests/system/user
>>>
>>> Or validation, as you prefer.
>> 
>> So what are tests/tcg if not user tests? They *mostly* test
>> linux-user emulation but of course we have softmmu tests in there as
>> well. 
>
> I expect a tests/tcg/ to check a specific TCG feature, which doesn't
> have to be user-mode specific (IIRC Xtensa does some sysemu checks).
> Also, you control the compiler toolchain, flags, etc... so you can
> adapt for a specific feature bit to test, use kludges and so on.

Well I won't say there are things that couldn't be tested elsewhere. I
think the initial record/replay tests are probably replaceable by the
acceptance/whatever tests - and possibly the gdbstub tests as well.

> I expect tests in tests/system/ (user/softmmu) to user real-world
> binaries, which we aren't modifying. Sometime non-public/released
> compiler toolchain has been used.

LTP binaries?

>
> See for example the test referred tests the bFLT loader (beside
> testing userland Linux binary for Cortex-M).
>
> Another example is the Sony PlayStation2 binary testing the
> O32 ABI and multiple opcodes from the TX79 SIMD core:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782493.html
>
> Personally I'm not interested in writing a test for a loader or
> multiple opcodes when we have pre-built binaries. For the opcodes
> coverage I'd use a TCG plugin to confirm the opcodes have been
> used.
>
> If you think these tests belong to tests/tcg/, I am OK to put
> them they, but I don't think adding the Avocado buildsys
> machinery to the already-complex tests/tcg/ Makefiles is going
> to help us...

No I wasn't advocating that - it was more a comment on the naming of
things. -ETOOMUCHFRIDAYBIKESHEDDING...

>
> Regards,
>
> Phil.


-- 
Alex Bennée

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Thomas Huth]

On 21/05/2021 14.28, Willian Rampazzo wrote:
> On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
[...]
>> Alternatively, what about renaming the "acceptance" tests to "validation"
>> instead? That word does not have a duplicated definition in the context of
>> QEMU yet, so I think it would be less confusing.
> 
> While at the beginning of your reply, I started thinking if
> "validation" would cause less confusion for the QEMU project. Although
> validation testing is a broader concept inside the Verification &
> Validation process, encompassing unit testing, functional testing,
> integration testing, system testing, and acceptance testing, it may be
> an option for the QEMU project.
> 
> While system testing would be the correct terminology to use, if it
> causes more confusion, using a less strict terminology, like
> validation testing, is valid, in my opinion.

<semi-homorous-friday-bikeshedding>
Or we could come up with a new, artificial name instead (like "qtests"). It 
certainly need a "q" in the name. For example what about:

- avoqado
- valiqation
- aqueptance
- inteqration

?
</semi-homorous-friday-bikeshedding>

  Thomas

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Philippe Mathieu-Daudé]

On 5/21/21 2:42 PM, Thomas Huth wrote:
> On 21/05/2021 14.28, Willian Rampazzo wrote:
>> On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
> [...]
>>> Alternatively, what about renaming the "acceptance" tests to
>>> "validation"
>>> instead? That word does not have a duplicated definition in the
>>> context of
>>> QEMU yet, so I think it would be less confusing.
>>
>> While at the beginning of your reply, I started thinking if
>> "validation" would cause less confusion for the QEMU project. Although
>> validation testing is a broader concept inside the Verification &
>> Validation process, encompassing unit testing, functional testing,
>> integration testing, system testing, and acceptance testing, it may be
>> an option for the QEMU project.
>>
>> While system testing would be the correct terminology to use, if it
>> causes more confusion, using a less strict terminology, like
>> validation testing, is valid, in my opinion.
> 
> <semi-homorous-friday-bikeshedding>
> Or we could come up with a new, artificial name instead (like "qtests").
> It certainly need a "q" in the name. For example what about:
> 
> - avoqado

+1 for 'make check-avoqado' :)

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 21/05/2021 14.28, Willian Rampazzo wrote:
>> On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
> [...]
>>> Alternatively, what about renaming the "acceptance" tests to "validation"
>>> instead? That word does not have a duplicated definition in the context of
>>> QEMU yet, so I think it would be less confusing.
>> While at the beginning of your reply, I started thinking if
>> "validation" would cause less confusion for the QEMU project. Although
>> validation testing is a broader concept inside the Verification &
>> Validation process, encompassing unit testing, functional testing,
>> integration testing, system testing, and acceptance testing, it may be
>> an option for the QEMU project.
>> While system testing would be the correct terminology to use, if it
>> causes more confusion, using a less strict terminology, like
>> validation testing, is valid, in my opinion.
>
> <semi-homorous-friday-bikeshedding>
> Or we could come up with a new, artificial name instead (like
> "qtests"). It certainly need a "q" in the name. For example what
> about:
>
> - avoqado
> - valiqation
> - aqueptance
> - inteqration

make check-quixotic?

>
> ?
> </semi-homorous-friday-bikeshedding>
>
>  Thomas


-- 
Alex Bennée

Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

On Thu, May 20, 2021 at 5:28 PM Philippe Mathieu-Daudé
<philmd@redhat.com> wrote:
>
> On 5/20/21 9:53 PM, Willian Rampazzo wrote:
> > Conceptually speaking, acceptance tests "are a series of specific tests
> > conducted by the customer in an attempt to uncover product errors before
> > accepting the software from the developer. Conducted by the end-user rather
> > than software engineers, acceptance testing can range from an informal
> > “test drive” to a planned and systematically executed series of scripted
> > tests" [1]. Every time Pressman refers to the term "acceptance testing," he
> > also refers to user's agreement in the final state of an implemented feature.
> > Today, QEMU is not implementing user acceptance tests as described by Pressman.
> >
> > There are other three possible terms we could use to describe what is currently
> > QEMU "acceptance" tests:
> >
> >   1 - Integration tests:
> >       - "Integration testing is a systematic technique for constructing the
> >          software architecture while at the same time conducting tests to
> >          uncover errors associated with interfacing. The objective is to take
> >          unit-tested components and build a program structure that has been
> >          dictated by design." [2]
> >       * Note: Sommerville does not have a clear definition of integration
> >         testing. He refers to incremental integration of components inside
> >         the system testing (see [3]).
> >
> >   2 - Validation tests:
> >       - "Validation testing begins at the culmination of integration testing,
> >          when individual components have been exercised, the software is
> >          completely assembled as a package, and interfacing errors have been
> >          uncovered and corrected. At the validation or system level, the
> >          distinction between different software categories disappears. Testing
> >          focuses on user-visible actions and user-recognizable output from the
> >          system." [4]
> >       - "where you expect the system to perform correctly using a set of test
> >          cases that reflect the system’s expected use." [5]
> >       * Note: the definition of "validation testing" from Sommerville reflects
> >         the same definition found around the Internet, as one of the processes
> >         inside the "Verification & Validation (V&V)." In this concept,
> >         validation testing is a high-level definition that covers unit testing,
> >         functional testing, integration testing, system testing, and acceptance
> >         testing.
> >
> >   3 - System tests:
> >       - "verifies that all elements mesh properly and that overall system
> >          function and performance is achieved." [6]
> >       - "involves integrating components to create a version of the system and
> >          then testing the integrated system. System testing checks that
> >          components are compatible, interact correctly, and transfer the right
> >          data at the right time across their interfaces." [7]
> >
> > The tests implemented inside the QEMU "acceptance" directory depend on the
> > software completely assembled and, sometimes, on other elements, like operating
> > system images. In this case, the proposal here is to rename the current
> > "acceptance" directory to "system."
>
> Are user-mode tests using Avocado also system tests?
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html

Yes, they are considered system tests because in software engineering,
system testing means your software built and tested using external
test artifacts, just like your example.

>
> > [1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
> >     Practitioner’s Approach. p. 430.
> > [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
> >     Practitioner’s Approach. Software Engineering, p. 398.
> > [3] Sommerville, Ian (2016). Software Engineering. p. 240-242.
> > [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
> >     Practitioner’s Approach. Software Engineering, p. 407.
> > [5] Sommerville, Ian (2016). Software Engineering. p. 227.
> > [6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
> >     Practitioner’s Approach. Software Engineering, p. 377.
> > [7] Sommerville, Ian (2016). Software Engineering. p. 240.
> >
> > Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> > ---
>
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index f718b61fa7..c5de3c9fd5 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -52,7 +52,7 @@ include:
> >      # Avoid recompiling by hiding ninja with NINJA=":"
> >      - make NINJA=":" $MAKE_CHECK_ARGS
> >
> > -.acceptance_template: &acceptance_definition
> > +.system_template: &system_definition
>
> .system_test_template: &system_test_definition ?

Agreed, keeps the consistency.

>
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 8f220e15d1..c580292bb5 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -16,7 +16,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),)
> >       @echo " $(MAKE) check-tcg            Run TCG tests"
> >       @echo " $(MAKE) check-softfloat      Run FPU emulation tests"
> >  endif
> > -     @echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
> > +     @echo " $(MAKE) check-system         Run all system tests"
>