From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
To: linux-block@vger.kernel.org
Cc: linux-nvme@lists.infradead.org, Daniel Wagner <dwagern@suse.de>,
Chaitanya Kulkarni <kch@nvidia.com>,
Sagi Grimberg <sagi@grimberg.me>
Subject: [PATCH blktests v3 00/15] support test case repeat by different conditions
Date: Wed, 24 Apr 2024 16:59:40 +0900 [thread overview]
Message-ID: <20240424075955.3604997-1-shinichiro.kawasaki@wdc.com> (raw)
In the recent discussion for nvme test group [1], two pain points were mentioned
regarding the test case runs.
1) Several test cases in nvme test group do exactly the same test except the
NVME transport backend set up condition difference (device vs. file). This
results in duplicate test script codes. It is desired to unify the test cases
and run them repeatedly with the different conditions.
2) NVME transport types can be specified with nvme_trtype parameter so that the
same tests can be run for various transport types. However, some test cases
do not depend on the transport types. They are repeated in multiple runs for
the various transport types under the exact same conditions. It is desired to
repeat the test cases only when such repetition is required.
[1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/
One idea to address these pain points is to add the test repeat feature to the
nvme test group. However, Daniel questioned if the feature could be implemented
in the blktests framework. Actually, a similar feature has already been
implemented to repeat some test cases for non-zoned block devices and zoned
block devices. However, this feature is implemented only for the zoned and non-
zoned device conditions. It can not fulfill the desires for nvme test group.
This series proposes to generalize the feature in the blktests framework to
repeat the test cases with different conditions. Introduce a new function
set_conditions() that each test case can define and instruct the framework to
repeat the test case. This series applies this feature to nvme test group so
that the test cases can be repeated for NVME transport types and backend types
in the ideal way. For this purpose, this series introduces new config parameters
NVMET_TRTYPES and NVMET_BLKDEV_TYPES. Taking this chance, it renames other
lowercase config parameters nvme_img_size, nvme_num_iter and use_rxe to
uppercase to follow the guide for environment variables.
The first four patches introduce the feature and apply it to the repetition for
non-zoned and zoned block devices. The fifth patch introduces a helper function
to prepare config parameter rename to uppercase. The following seven patches
apply the test case repeat feature to nvme group. Three of the seven patches are
reused from the work by Daniel. The last three patches rename the config
parameters. The all patches are listed in the order that does not lose the test
coverage with the default set up.
As an example of the repeated test run, let's say NVMET_TRTYPES and
NVMET_BLKDEV_TYPES are defined in the config file as follows:
NVMET_TRTYPES="loop rdma tcp"
NVMET_BLKDEV_TYPES="device file"
In this case, the test cases which depend on these parameters are repeated
3 x 2 = 6 times. For example, nvme/006 is repeated as follows.
nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed]
runtime 0.148s ... 0.165s
nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed]
runtime 0.273s ... 0.235s
nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target) [passed]
runtime 0.162s ... 0.164s
nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target) [passed]
runtime 0.138s ... 0.134s
nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target) [passed]
runtime 0.216s ... 0.201s
nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target) [passed]
runtime 0.154s ... 0.146s
Changes from v2:
* 5th patch: added to prepare for the parameter rename
* 6th patch: modified nvme_trtype description and added NVMET_TRTYPES check
* 13-15th patches: added for the parameter rename
* Applied Reviewed-by tags
* Rebased to the latest master branch tip
Changes from v1:
* Renamed NVMET_TR_TYPES to NVMET_TRTYPES
* 1st patch: reflected comments on the list and added Reviewed-by tag
* 5th patch: changed NVMET_TRTYPES from array to variable
* 7th patch: changed NVMET_BLKDEV_TYPES from array to variable
* Reflected other comments on the list
Daniel Wagner (3):
nvme/rc: add blkdev type environment variable
nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type
tests
nvme/{021,022,025,026,027,028}: do not hard code target blkdev type
Shin'ichiro Kawasaki (12):
check: factor out _run_test()
check: support test case repeat by different conditions
check: use set_conditions() for the CAN_BE_ZONED test cases
meta/{016,017}: add test cases to check repeated test case runs
common/rc: introduce _check_conflict_and_set_default()
nvme/rc: introduce NVMET_TRTYPES
nvme/rc: introduce NVMET_BLKDEV_TYPES
nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES
nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES
nvme/{rc,010,017,031,034,035}: rename nvme_img_size to NVME_IMG_SIZE
nvme/{rc,016,017}: rename nvme_num_iter to NVME_NUM_ITER
nvme/rc,srp/rc,common/multipath-over-rdma: rename use_rxe to USE_RXE
Documentation/running-tests.md | 33 ++++++---
Makefile | 3 +-
check | 129 ++++++++++++++++++++++-----------
common/multipath-over-rdma | 5 +-
common/rc | 27 +++++++
common/shellcheck | 2 +-
common/zoned | 22 ++++++
new | 21 ++++++
tests/meta/016 | 29 ++++++++
tests/meta/016.out | 2 +
tests/meta/017 | 29 ++++++++
tests/meta/017.out | 2 +
tests/nvme/002 | 4 +
tests/nvme/003 | 4 +
tests/nvme/004 | 4 +
tests/nvme/005 | 4 +
tests/nvme/006 | 8 +-
tests/nvme/007 | 27 -------
tests/nvme/007.out | 2 -
tests/nvme/008 | 8 +-
tests/nvme/009 | 36 ---------
tests/nvme/009.out | 3 -
tests/nvme/010 | 10 ++-
tests/nvme/011 | 39 ----------
tests/nvme/011.out | 3 -
tests/nvme/012 | 8 +-
tests/nvme/013 | 43 -----------
tests/nvme/013.out | 3 -
tests/nvme/014 | 8 +-
tests/nvme/015 | 48 ------------
tests/nvme/015.out | 4 -
tests/nvme/016 | 6 +-
tests/nvme/017 | 8 +-
tests/nvme/018 | 4 +
tests/nvme/019 | 8 +-
tests/nvme/020 | 40 ----------
tests/nvme/020.out | 4 -
tests/nvme/021 | 10 ++-
tests/nvme/022 | 10 ++-
tests/nvme/023 | 8 +-
tests/nvme/024 | 40 ----------
tests/nvme/024.out | 2 -
tests/nvme/025 | 10 ++-
tests/nvme/026 | 10 ++-
tests/nvme/027 | 10 ++-
tests/nvme/028 | 10 ++-
tests/nvme/029 | 4 +
tests/nvme/030 | 4 +
tests/nvme/031 | 6 +-
tests/nvme/033 | 4 +
tests/nvme/034 | 6 +-
tests/nvme/035 | 8 +-
tests/nvme/036 | 4 +
tests/nvme/037 | 4 +
tests/nvme/038 | 4 +
tests/nvme/040 | 4 +
tests/nvme/041 | 3 +
tests/nvme/042 | 3 +
tests/nvme/043 | 3 +
tests/nvme/044 | 3 +
tests/nvme/045 | 3 +
tests/nvme/047 | 4 +
tests/nvme/048 | 4 +
tests/nvme/rc | 74 ++++++++++++++++---
tests/srp/rc | 2 +-
65 files changed, 493 insertions(+), 404 deletions(-)
create mode 100644 common/zoned
create mode 100755 tests/meta/016
create mode 100644 tests/meta/016.out
create mode 100755 tests/meta/017
create mode 100644 tests/meta/017.out
delete mode 100755 tests/nvme/007
delete mode 100644 tests/nvme/007.out
delete mode 100755 tests/nvme/009
delete mode 100644 tests/nvme/009.out
delete mode 100755 tests/nvme/011
delete mode 100644 tests/nvme/011.out
delete mode 100755 tests/nvme/013
delete mode 100644 tests/nvme/013.out
delete mode 100755 tests/nvme/015
delete mode 100644 tests/nvme/015.out
delete mode 100755 tests/nvme/020
delete mode 100644 tests/nvme/020.out
delete mode 100755 tests/nvme/024
delete mode 100644 tests/nvme/024.out
--
2.44.0
next reply other threads:[~2024-04-24 8:00 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-24 7:59 Shin'ichiro Kawasaki [this message]
2024-04-24 7:59 ` [PATCH blktests v3 01/15] check: factor out _run_test() Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 02/15] check: support test case repeat by different conditions Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 03/15] check: use set_conditions() for the CAN_BE_ZONED test cases Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 04/15] meta/{016,017}: add test cases to check repeated test case runs Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 05/15] common/rc: introduce _check_conflict_and_set_default() Shin'ichiro Kawasaki
2024-04-24 12:27 ` Daniel Wagner
2024-04-24 7:59 ` [PATCH blktests v3 06/15] nvme/rc: introduce NVMET_TRTYPES Shin'ichiro Kawasaki
2024-04-24 12:28 ` Daniel Wagner
2024-04-24 7:59 ` [PATCH blktests v3 07/15] nvme/rc: add blkdev type environment variable Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 08/15] nvme/rc: introduce NVMET_BLKDEV_TYPES Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 09/15] nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 10/15] nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES Shin'ichiro Kawasaki
2024-04-28 8:58 ` Sagi Grimberg
2024-04-28 10:32 ` Shinichiro Kawasaki
2024-04-28 13:12 ` Sagi Grimberg
2024-04-28 23:58 ` Shinichiro Kawasaki
2024-04-30 12:00 ` Sagi Grimberg
2024-05-03 6:23 ` Shinichiro Kawasaki
2024-05-03 7:33 ` Sagi Grimberg
2024-04-24 7:59 ` [PATCH blktests v3 11/15] nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type tests Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 12/15] nvme/{021,022,025,026,027,028}: do not hard code target blkdev type Shin'ichiro Kawasaki
2024-04-24 7:59 ` [PATCH blktests v3 13/15] nvme/{rc,010,017,031,034,035}: rename nvme_img_size to NVME_IMG_SIZE Shin'ichiro Kawasaki
2024-04-24 12:29 ` Daniel Wagner
2024-04-24 7:59 ` [PATCH blktests v3 14/15] nvme/{rc,016,017}: rename nvme_num_iter to NVME_NUM_ITER Shin'ichiro Kawasaki
2024-04-24 12:30 ` Daniel Wagner
2024-04-24 7:59 ` [PATCH blktests v3 15/15] nvme/rc,srp/rc,common/multipath-over-rdma: rename use_rxe to USE_RXE Shin'ichiro Kawasaki
2024-04-24 12:33 ` Daniel Wagner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240424075955.3604997-1-shinichiro.kawasaki@wdc.com \
--to=shinichiro.kawasaki@wdc.com \
--cc=dwagern@suse.de \
--cc=kch@nvidia.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).