linux-nvme.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [PATCH blktests v2 00/11] support test case repeat by different conditions
@ 2024-04-16 10:31 Shin'ichiro Kawasaki
  2024-04-16 10:31 ` [PATCH blktests v2 01/11] check: factor out _run_test() Shin'ichiro Kawasaki
                   ` (14 more replies)
  0 siblings, 15 replies; 18+ messages in thread
From: Shin'ichiro Kawasaki @ 2024-04-16 10:31 UTC (permalink / raw)
  To: linux-block; +Cc: linux-nvme, Daniel Wagner, Chaitanya Kulkarni

In the recent discussion for nvme test group [1], two pain points were mentioned
regarding the test case runs.

1) Several test cases in nvme test group do exactly the same test except the
   NVME transport backend set up condition difference (device vs. file). This
   results in duplicate test script codes. It is desired to unify the test cases
   and run them repeatedly with the different conditions.

2) NVME transport types can be specified with nvme_trtype parameter so that the
   same tests can be run for various transport types. However, some test cases
   do not depend on the transport types. They are repeated in multiple runs for
   the various transport types under the exact same conditions. It is desired to
   repeat the test cases only when such repetition is required.

[1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/

One idea to address these pain points is to add the test repeat feature to the
nvme test group. However, Daniel questioned if the feature could be implemented
in the blktests framework. Actually, a similar feature has already been
implemented to repeat some test cases for non-zoned block devices and zoned
block devices. However, this feature is implemented only for the zoned and non-
zoned device conditions. It can not fulfill the desires for nvme test group.

This series proposes to generalize the feature in the blktests framework to
repeat the test cases with different conditions. Introduce a new function
set_conditions() that each test case can define and instruct the framework to
repeat the test case. The first four patches introduce the feature and apply it
to the repetition for non-zoned and zoned block devices. The following seven
patches apply the feature to nvme test group so that the test cases can be
repeated for NVME transport types and backend types in the ideal way. Two of the
seven patches are reused from the work by Daniel. The all patches are listed in
the order that does not lose the test coverage with the default set up.

This series introduces new config parameters NVMET_TRTYPES and
NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When
they are defined in the config file as follows,

  NVMET_TRTYPES="loop rdma tcp"
  NVMET_BLKDEV_TYPES="device file"

the test cases which depend on these parameters are repeated 3 x 2 = 6 times.
For example, nvme/006 is repeated as follows.

nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed]
    runtime  0.148s  ...  0.165s
nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed]
    runtime  0.273s  ...  0.235s
nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target)  [passed]
    runtime  0.162s  ...  0.164s
nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target)   [passed]
    runtime  0.138s  ...  0.134s
nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target)   [passed]
    runtime  0.216s  ...  0.201s
nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target)    [passed]
    runtime  0.154s  ...  0.146s


Changes from v1:
* Renamed NVMET_TR_TYPES to NVMET_TRTYPES
* 1st patch: reflected comments on the list and added Reviewed-by tag
* 5th patch: changed NVMET_TRTYPES from array to variable
* 7th patch: changed NVMET_BLKDEV_TYPES from array to variable
* Reflected other comments on the list


Daniel Wagner (3):
  nvme/rc: add blkdev type environment variable
  nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type
    tests
  nvme/{021,022,025,026,027,028}: do not hard code target blkdev type

Shin'ichiro Kawasaki (8):
  check: factor out _run_test()
  check: support test case repeat by different conditions
  check: use set_conditions() for the CAN_BE_ZONED test cases
  meta/{016,017}: add test cases to check repeated test case runs
  nvme/rc: introduce NVMET_TRTYPES
  nvme/rc: introduce NVMET_BLKDEV_TYPES
  nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES
  nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES

 Documentation/running-tests.md |  16 +++-
 Makefile                       |   3 +-
 check                          | 129 ++++++++++++++++++++++-----------
 common/shellcheck              |   2 +-
 common/zoned                   |  22 ++++++
 new                            |  21 ++++++
 tests/meta/016                 |  29 ++++++++
 tests/meta/016.out             |   2 +
 tests/meta/017                 |  29 ++++++++
 tests/meta/017.out             |   2 +
 tests/nvme/002                 |   4 +
 tests/nvme/003                 |   4 +
 tests/nvme/004                 |   4 +
 tests/nvme/005                 |   4 +
 tests/nvme/006                 |   9 ++-
 tests/nvme/007                 |  28 -------
 tests/nvme/007.out             |   2 -
 tests/nvme/008                 |   8 +-
 tests/nvme/009                 |  36 ---------
 tests/nvme/009.out             |   3 -
 tests/nvme/010                 |   8 +-
 tests/nvme/011                 |  39 ----------
 tests/nvme/011.out             |   3 -
 tests/nvme/012                 |   8 +-
 tests/nvme/013                 |  43 -----------
 tests/nvme/013.out             |   3 -
 tests/nvme/014                 |   8 +-
 tests/nvme/015                 |  48 ------------
 tests/nvme/015.out             |   4 -
 tests/nvme/016                 |   4 +
 tests/nvme/017                 |   4 +
 tests/nvme/018                 |   4 +
 tests/nvme/019                 |   8 +-
 tests/nvme/020                 |  40 ----------
 tests/nvme/020.out             |   4 -
 tests/nvme/021                 |  10 ++-
 tests/nvme/022                 |  10 ++-
 tests/nvme/023                 |   8 +-
 tests/nvme/024                 |  40 ----------
 tests/nvme/024.out             |   2 -
 tests/nvme/025                 |  10 ++-
 tests/nvme/026                 |  10 ++-
 tests/nvme/027                 |  10 ++-
 tests/nvme/028                 |  10 ++-
 tests/nvme/029                 |   4 +
 tests/nvme/030                 |   4 +
 tests/nvme/031                 |   4 +
 tests/nvme/033                 |   4 +
 tests/nvme/034                 |   4 +
 tests/nvme/035                 |   4 +
 tests/nvme/036                 |   4 +
 tests/nvme/037                 |   4 +
 tests/nvme/038                 |   4 +
 tests/nvme/040                 |   4 +
 tests/nvme/041                 |   3 +
 tests/nvme/042                 |   3 +
 tests/nvme/043                 |   3 +
 tests/nvme/044                 |   3 +
 tests/nvme/045                 |   3 +
 tests/nvme/047                 |   4 +
 tests/nvme/048                 |   4 +
 tests/nvme/rc                  |  58 ++++++++++++++-
 62 files changed, 437 insertions(+), 379 deletions(-)
 create mode 100644 common/zoned
 create mode 100755 tests/meta/016
 create mode 100644 tests/meta/016.out
 create mode 100755 tests/meta/017
 create mode 100644 tests/meta/017.out
 delete mode 100755 tests/nvme/007
 delete mode 100644 tests/nvme/007.out
 delete mode 100755 tests/nvme/009
 delete mode 100644 tests/nvme/009.out
 delete mode 100755 tests/nvme/011
 delete mode 100644 tests/nvme/011.out
 delete mode 100755 tests/nvme/013
 delete mode 100644 tests/nvme/013.out
 delete mode 100755 tests/nvme/015
 delete mode 100644 tests/nvme/015.out
 delete mode 100755 tests/nvme/020
 delete mode 100644 tests/nvme/020.out
 delete mode 100755 tests/nvme/024
 delete mode 100644 tests/nvme/024.out

-- 
2.44.0



^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2024-04-22 11:35 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-16 10:31 [PATCH blktests v2 00/11] support test case repeat by different conditions Shin'ichiro Kawasaki
2024-04-16 10:31 ` [PATCH blktests v2 01/11] check: factor out _run_test() Shin'ichiro Kawasaki
2024-04-16 10:31 ` [PATCH blktests v2 02/11] check: support test case repeat by different conditions Shin'ichiro Kawasaki
2024-04-16 10:31 ` [PATCH blktests v2 03/11] check: use set_conditions() for the CAN_BE_ZONED test cases Shin'ichiro Kawasaki
2024-04-16 10:32 ` [PATCH blktests v2 04/11] meta/{016,017}: add test cases to check repeated test case runs Shin'ichiro Kawasaki
2024-04-16 10:32 ` [PATCH blktests v2 05/11] nvme/rc: introduce NVMET_TRTYPES Shin'ichiro Kawasaki
2024-04-18  9:39   ` Sagi Grimberg
2024-04-22 11:35     ` Shinichiro Kawasaki
2024-04-16 10:32 ` [PATCH blktests v2 06/11] nvme/rc: add blkdev type environment variable Shin'ichiro Kawasaki
2024-04-16 10:32 ` [PATCH blktests v2 07/11] nvme/rc: introduce NVMET_BLKDEV_TYPES Shin'ichiro Kawasaki
2024-04-16 10:32 ` [PATCH blktests v2 08/11] nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES Shin'ichiro Kawasaki
2024-04-16 10:32 ` [PATCH blktests v2 09/11] nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES Shin'ichiro Kawasaki
2024-04-16 10:32 ` [PATCH blktests v2 10/11] nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type tests Shin'ichiro Kawasaki
2024-04-16 10:32 ` [PATCH blktests v2 11/11] nvme/{021,022,025,026,027,028}: do not hard code target blkdev type Shin'ichiro Kawasaki
2024-04-17  7:01 ` [PATCH blktests v2 00/11] support test case repeat by different conditions Daniel Wagner
2024-04-18  7:23 ` Chaitanya Kulkarni
2024-04-18  9:40 ` Sagi Grimberg
     [not found] ` <CGME20240418194325epcas5p2c88282729c2cb0715d7343428217de8d@epcas5p2.samsung.com>
2024-04-18 19:36   ` Nitesh Shetty

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).