[LTP] [PATCH v2 00/30] Introduce runtime and conver tests

[LTP] [PATCH v2 00/30] Introduce runtime and conver tests

[Cyril Hrubis]

This patchset introduces a concept of runtime, which is described well
enough in the first patch of the series and converts all tests that were
previously using timeout API to runtime API.

The main change to the v1 of the patchset is that the runtime is now per
iteration, e.g. if we run the test for all_filesystem the runtime
applies to a single filesystem only. And at the same time the overall
test timeout is a sum of static timeout and runtime. The static part of
the timeout is meant to cover test teardown and any inaccuracy in the
test runtime accounting.

The patches that convert tests to runtime API fall into these classes:

- fzsync tests
- faster timeout tests
- I/O tests
- timer precision tests
- oom tests
- misc tests that run for some time (controlled by command line
  parameter)

fzsync tests
============

These tests previously defined their runtime based on scale factor that
used slice of the default timeout (300s) for the test runtime.

Now the runtime is explicitly encoded in the test, which I find much
more user friendly because it's clear how long will the test run.

faster timeout tests
====================

There were several tests that set low timeout becuase the test may lock
up. In these cases the timeout was simply removed because the default
value (30s) should be small enough.

I/O tests
=========

The aiodio tests were modified so that their runtime is capped to
30 minutes by default. That means that even on very slow machine they
finish in a reasonable time.

timer precision tests
=====================

These test simply set the test timeout to be 110% of the sum of the
sleep_time * samples. That is to accomodate for the context switches and
upkeeping between the calls in the timer library.


oom tests
=========

Here we just set the .max_iteration_runtime to TST_RUNTIME_UNLIMITED,
which has the very same effect as .timeout = -1 previously had.

misc test
=========

In this case the test code was simplified quite a lot since the runtime
is now passed in a unified way and parsed in the test library.

TODO:

This patchset is reasonably complete in a sense that it removes the
timeout API at the end. Still there are a few things to consider:

- is reusing the -I parameter a good idea? Wouldn't adding new parameter
  (-r) be better?

- there are quite likely tests that run for more than a second or a two
  and should be made runtime aware

- anything else?

Cyril Hrubis (30):
  Introduce a concept of max runtime
  mtest06/mmap1: Convert to runtime
  mtest06/mmap3: Convert to runtime
  mtest01/mtest01: Convert to runtime
  cve/cve-2015-3290: Convert to runtime
  crypto/af_alg02: Convert to runtime.
  crypto/pcrypt_aead01: Convert to runtime
  syscalls/clock_gettime01: Remove useless timeout
  syscalls/fanotify22: Remove useless timeout
  syscalls/gettimeofday02: Convert to runtime.
  syscalls/inotify06: Convert to runtime.
  syscalls/inotify01: Remove now useless timeout
  syscalls/perf_event_open03: Convert to runtime
  syscalls/readv01: Remove now useless timeout
  syscalls/tgkill03: Remove now unused timeout
  syscalls/setsockopt09: Remove now useless timeout
  syscalls/userfaultfd01: Remove now useless timeout
  syscalls/move_pages12: Convert to runtime
  syscalls/rt_sigqueueinfo01: Remove now useless timeout
  mem/mallocstress: Convert to runtime
  mem/{oom,min_free_kbytes}: Convert to runtime
  crypto/af_alg02: Convert to runtime
  fuzzy_sync: Convert to runtime
  ltp-aiodio/dio_sparse,aiodio_sparse: Convert to runtime.
  ltp-aiodio/read_checkzero: Remove
  ltp-aiodio/dio_{truncate,append}: Convert to runtime
  ltp-aiodio/dio_read: Convert to runtime
  ltp-aiodio/aiodio_append: Convert to runtime
  timer_test: Convert to runtime
  tst_test: Remove timeout stubs

 doc/user-guide.txt                            |   4 +
 include/tst_fuzzy_sync.h                      |  19 +--
 include/tst_test.h                            |  31 +++-
 lib/newlib_tests/.gitignore                   |   5 +-
 lib/newlib_tests/test10.c                     |  22 ---
 lib/newlib_tests/test12.c                     |  21 ---
 lib/newlib_tests/test13.c                     |   1 -
 lib/newlib_tests/test_children_cleanup.c      |   1 -
 .../{test18.c => test_runtime01.c}            |  10 +-
 lib/newlib_tests/test_runtime02.c             |  31 ++++
 lib/tst_test.c                                | 144 +++++++++++++-----
 lib/tst_timer_test.c                          |  12 +-
 runtest/ltp-aiodio.part4                      |   2 -
 testcases/cve/cve-2015-3290.c                 |   8 +-
 testcases/kernel/crypto/af_alg02.c            |   4 +-
 testcases/kernel/crypto/af_alg07.c            |   1 +
 testcases/kernel/crypto/pcrypt_aead01.c       |   3 +-
 testcases/kernel/io/ltp-aiodio/.gitignore     |   1 -
 .../kernel/io/ltp-aiodio/aiodio_append.c      |  25 ++-
 .../kernel/io/ltp-aiodio/aiodio_sparse.c      |  26 +++-
 testcases/kernel/io/ltp-aiodio/common.h       |  12 +-
 testcases/kernel/io/ltp-aiodio/dio_append.c   |   4 +
 testcases/kernel/io/ltp-aiodio/dio_read.c     |   9 ++
 testcases/kernel/io/ltp-aiodio/dio_sparse.c   |   6 +-
 testcases/kernel/io/ltp-aiodio/dio_truncate.c |   6 +
 .../kernel/io/ltp-aiodio/read_checkzero.c     |  84 ----------
 testcases/kernel/mem/mtest01/mtest01.c        |  14 +-
 testcases/kernel/mem/mtest06/mmap1.c          |  24 +--
 testcases/kernel/mem/mtest06/mmap3.c          |  12 +-
 testcases/kernel/mem/mtest07/mallocstress.c   |  11 +-
 testcases/kernel/mem/oom/oom01.c              |   2 +-
 testcases/kernel/mem/oom/oom02.c              |   2 +-
 testcases/kernel/mem/oom/oom03.c              |   2 +-
 testcases/kernel/mem/oom/oom04.c              |   2 +-
 testcases/kernel/mem/oom/oom05.c              |   2 +-
 testcases/kernel/mem/thp/thp04.c              |   1 +
 .../kernel/mem/tunable/min_free_kbytes.c      |   2 +-
 testcases/kernel/pty/pty03.c                  |   2 +-
 testcases/kernel/pty/pty05.c                  |   1 +
 testcases/kernel/pty/pty06.c                  |   1 +
 testcases/kernel/pty/pty07.c                  |   1 +
 testcases/kernel/sound/snd_seq01.c            |   2 +-
 testcases/kernel/sound/snd_timer01.c          |   1 +
 testcases/kernel/syscalls/bind/bind06.c       |   2 +-
 .../syscalls/clock_gettime/clock_gettime01.c  |   1 -
 .../kernel/syscalls/fanotify/fanotify22.c     |   1 -
 .../syscalls/gettimeofday/gettimeofday02.c    |  15 +-
 testcases/kernel/syscalls/inotify/inotify06.c |   7 +-
 testcases/kernel/syscalls/inotify/inotify09.c |   1 +
 testcases/kernel/syscalls/inotify/inotify11.c |   1 -
 .../kernel/syscalls/ipc/shmctl/shmctl05.c     |   2 +-
 .../kernel/syscalls/move_pages/move_pages12.c |   4 +-
 .../perf_event_open/perf_event_open03.c       |  15 +-
 testcases/kernel/syscalls/readv/readv01.c     |   1 -
 .../rt_sigqueueinfo/rt_sigqueueinfo01.c       |   1 -
 testcases/kernel/syscalls/sendmsg/sendmsg03.c |   1 +
 .../kernel/syscalls/setsockopt/setsockopt06.c |   2 +-
 .../kernel/syscalls/setsockopt/setsockopt07.c |   1 +
 .../kernel/syscalls/setsockopt/setsockopt09.c |   1 -
 testcases/kernel/syscalls/tgkill/tgkill03.c   |   1 -
 .../syscalls/timerfd/timerfd_settime02.c      |   1 +
 .../syscalls/userfaultfd/userfaultfd01.c      |   1 -
 testcases/kernel/syscalls/writev/writev03.c   |   2 +-
 63 files changed, 339 insertions(+), 296 deletions(-)
 delete mode 100644 lib/newlib_tests/test10.c
 delete mode 100644 lib/newlib_tests/test12.c
 rename lib/newlib_tests/{test18.c => test_runtime01.c} (58%)
 create mode 100644 lib/newlib_tests/test_runtime02.c
 delete mode 100644 testcases/kernel/io/ltp-aiodio/read_checkzero.c

-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 01/30] Introduce a concept of max runtime

[Cyril Hrubis]

This commit introduce a concept of per iteration max test runtime. In
other words test runtime is capped at a certain value in order to make
testruns more deterministic. Test is free to to finish before the
runtime is used up, for example when maximal number of iterations
was reached, but test must stop once the runtime has been used up.

Testcases that run for more than a second or two must check for
remaining runtime by regular calls to tst_remaining_runtime() and should
exit when zero is returned.

The test max runtime must be set either by the .max_iteration_runtime in
the tst_test structure or in the test setup by a call to
tst_set_runtime().

The test timeout is then computed as a sum of DEFAULT_TIMEOUT (currently
set to 30 seconds) and the test runtime. The DEFAULT_TIMEOUT is nothing
more than a safety margin for teardown of the test.

This commit also maps the -I parameter to the test max runtime if
available and introduces LTP_RUNTIME_MUL enviroment variable so that we
have an easy controll over the runtime cap.

Lastly but not least the function related to the timeout are turned into
no-op by this commit and removed after all test are converted to the
runtime API.

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 doc/user-guide.txt                            |   4 +
 include/tst_test.h                            |  27 ++++
 lib/newlib_tests/.gitignore                   |   5 +-
 lib/newlib_tests/test10.c                     |  22 ---
 lib/newlib_tests/test12.c                     |  21 ---
 lib/newlib_tests/test13.c                     |   1 -
 lib/newlib_tests/test_children_cleanup.c      |   1 -
 .../{test18.c => test_runtime01.c}            |  10 +-
 lib/newlib_tests/test_runtime02.c             |  31 ++++
 lib/tst_test.c                                | 151 ++++++++++++++----
 10 files changed, 187 insertions(+), 86 deletions(-)
 delete mode 100644 lib/newlib_tests/test10.c
 delete mode 100644 lib/newlib_tests/test12.c
 rename lib/newlib_tests/{test18.c => test_runtime01.c} (58%)
 create mode 100644 lib/newlib_tests/test_runtime02.c

diff --git a/doc/user-guide.txt b/doc/user-guide.txt
index f41cbc733..d50d2e0cb 100644
--- a/doc/user-guide.txt
+++ b/doc/user-guide.txt
@@ -25,6 +25,10 @@ For running LTP network tests see `testcases/network/README.md`.
 | 'LTP_TIMEOUT_MUL'     | Multiply timeout, must be number >= 1 (> 1 is useful for
                           slow machines to avoid unexpected timeout).
                           Variable is also used in shell tests, but ceiled to int.
+| 'LTP_RUNTIME_MUL'     | Multiplies maximal test iteration runtime. Tests
+                          that run for more than a second or two are capped on
+			  runtime. You can scale the default runtime both up
+			  and down with this multiplier.
 | 'LTP_VIRT_OVERRIDE'   | Overrides virtual machine detection in the test
                           library. Setting it to empty string tell the library
                           that system is not a virtual machine. Other possible
diff --git a/include/tst_test.h b/include/tst_test.h
index dbe303bc8..c084ce4bc 100644
--- a/include/tst_test.h
+++ b/include/tst_test.h
@@ -134,6 +134,8 @@ extern unsigned int tst_variant;
 
 #define TST_NO_HUGEPAGES ((unsigned long)-1)
 
+#define TST_UNLIMITED_RUNTIME (-1)
+
 struct tst_test {
 	/* number of tests available in test() function */
 	unsigned int tcnt;
@@ -236,6 +238,18 @@ struct tst_test {
 
 	/* override default timeout per test run, disabled == -1 */
 	int timeout;
+	/*
+	 * Maximal test runtime in seconds.
+	 *
+	 * Any test that runs for more than a second or two should set this and
+	 * also use tst_remaining_runtime() to exit when runtime was used up.
+	 * Tests may finish sooner, for example if requested number of
+	 * iterations was reached before the runtime runs out.
+	 *
+	 * If test runtime cannot be know in advance it should be set to
+	 * TST_UNLIMITED_RUNTIME.
+	 */
+	int max_iteration_runtime;
 
 	void (*setup)(void);
 	void (*cleanup)(void);
@@ -323,6 +337,19 @@ unsigned int tst_timeout_remaining(void);
 unsigned int tst_multiply_timeout(unsigned int timeout);
 void tst_set_timeout(int timeout);
 
+/*
+ * Returns remaining test runtime. Test that runs for more than a few seconds
+ * should check if they should exit by calling this function regularly.
+ *
+ * The function returns remaining runtime in seconds. If runtime was used up
+ * zero is returned.
+ */
+unsigned int tst_remaining_runtime(void);
+
+/*
+ * Sets runtime per iteration in seconds.
+ */
+void tst_set_runtime(int runtime_per_iteration);
 
 /*
  * Returns path to the test temporary directory in a newly allocated buffer.
diff --git a/lib/newlib_tests/.gitignore b/lib/newlib_tests/.gitignore
index f4414f6a1..59b57d063 100644
--- a/lib/newlib_tests/.gitignore
+++ b/lib/newlib_tests/.gitignore
@@ -7,9 +7,7 @@ test06
 test07
 test08
 test09
-test10
 test11
-test12
 test13
 test14
 test15
@@ -22,7 +20,6 @@ tst_safe_fileops
 tst_res_hexd
 tst_strstatus
 tst_print_result
-test18
 test19
 test20
 test22
@@ -56,3 +53,5 @@ tst_needs_cmds05
 tst_needs_cmds06
 tst_needs_cmds07
 tst_needs_cmds08
+test_runtime01
+test_runtime02
diff --git a/lib/newlib_tests/test10.c b/lib/newlib_tests/test10.c
deleted file mode 100644
index df61908ac..000000000
--- a/lib/newlib_tests/test10.c
+++ /dev/null
@@ -1,22 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Copyright (c) 2016 Linux Test Project
- */
-
-/*
- * Test for watchdog timeout.
- */
-
-#include "tst_test.h"
-
-
-static void do_test(void)
-{
-	sleep(2);
-	tst_res(TPASS, "Not reached");
-}
-
-static struct tst_test test = {
-	.test_all = do_test,
-	.timeout = 1,
-};
diff --git a/lib/newlib_tests/test12.c b/lib/newlib_tests/test12.c
deleted file mode 100644
index b4f0d6303..000000000
--- a/lib/newlib_tests/test12.c
+++ /dev/null
@@ -1,21 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Copyright (c) 2016 Linux Test Project
- */
-
-/*
- * Test for timeout override.
- */
-
-#include "tst_test.h"
-
-static void do_test(void)
-{
-	sleep(1);
-	tst_res(TPASS, "Passed!");
-}
-
-static struct tst_test test = {
-	.timeout = 2,
-	.test_all = do_test,
-};
diff --git a/lib/newlib_tests/test13.c b/lib/newlib_tests/test13.c
index c447dc3dc..83c48f734 100644
--- a/lib/newlib_tests/test13.c
+++ b/lib/newlib_tests/test13.c
@@ -20,7 +20,6 @@ static void do_test(void)
 }
 
 static struct tst_test test = {
-	.timeout = 1,
 	.forks_child = 1,
 	.test_all = do_test,
 };
diff --git a/lib/newlib_tests/test_children_cleanup.c b/lib/newlib_tests/test_children_cleanup.c
index 2b1ca5f9c..4a1313f6d 100644
--- a/lib/newlib_tests/test_children_cleanup.c
+++ b/lib/newlib_tests/test_children_cleanup.c
@@ -39,5 +39,4 @@ static void run(void)
 static struct tst_test test = {
 	.test_all = run,
 	.forks_child = 1,
-	.timeout = 10,
 };
diff --git a/lib/newlib_tests/test18.c b/lib/newlib_tests/test_runtime01.c
similarity index 58%
rename from lib/newlib_tests/test18.c
rename to lib/newlib_tests/test_runtime01.c
index 026435d7d..79e4c7eac 100644
--- a/lib/newlib_tests/test18.c
+++ b/lib/newlib_tests/test_runtime01.c
@@ -9,14 +9,18 @@
 
 static void run(void)
 {
+	int runtime;
+
 	do {
+		runtime = tst_remaining_runtime();
+		tst_res(TINFO, "Remaining runtime %d", runtime);
 		sleep(1);
-	} while (tst_timeout_remaining() >= 4);
+	} while (runtime);
 
-	tst_res(TPASS, "Timeout remaining: %d", tst_timeout_remaining());
+	tst_res(TPASS, "Finished loop!");
 }
 
 static struct tst_test test = {
 	.test_all = run,
-	.timeout = 5
+	.max_iteration_runtime = 5
 };
diff --git a/lib/newlib_tests/test_runtime02.c b/lib/newlib_tests/test_runtime02.c
new file mode 100644
index 000000000..1329743f4
--- /dev/null
+++ b/lib/newlib_tests/test_runtime02.c
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (c) 2021, Linux Test Project
+ */
+/*
+ * This test is set up so that the timeout is not long enough to guarantee
+ * enough runtime for two iterations, i.e. the timeout without offset and after
+ * scaling is too small and the tests ends up with TBROK.
+ *
+ * You can fix this by exporting LTP_MAX_TEST_RUNTIME=10 before executing the
+ * test, in that case the runtime would be divided between iterations and timeout
+ * adjusted so that it provides enough safeguards for the test to finish.
+ */
+
+#include <stdlib.h>
+#include <unistd.h>
+#include "tst_test.h"
+
+static void run(void)
+{
+	while (tst_remaining_runtime())
+		sleep(1);
+
+	tst_res(TPASS, "Timeout remaining: %d", tst_remaining_runtime());
+}
+
+static struct tst_test test = {
+	.test_all = run,
+	.max_iteration_runtime = 5,
+	.test_variants = 2
+};
diff --git a/lib/tst_test.c b/lib/tst_test.c
index 8e258594a..096acef96 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -45,6 +45,8 @@ const char *TCID __attribute__((weak));
 #define GLIBC_GIT_URL "https://sourceware.org/git/?p=glibc.git;a=commit;h="
 #define CVE_DB_URL "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-"
 
+#define DEFAULT_TIMEOUT 30
+
 struct tst_test *tst_test;
 
 static const char *tid;
@@ -63,6 +65,7 @@ struct results {
 	int warnings;
 	int broken;
 	unsigned int timeout;
+	int max_iteration_runtime;
 };
 
 static struct results *results;
@@ -464,6 +467,40 @@ pid_t safe_clone(const char *file, const int lineno,
 	return pid;
 }
 
+static void parse_mul(float *mul, const char *env_name, float min, float max)
+{
+	char *str_mul;
+	int ret;
+
+	if (*mul > 0)
+		return;
+
+	str_mul = getenv(env_name);
+
+	if (!str_mul) {
+		*mul = 1;
+		return;
+	}
+
+	ret = tst_parse_float(str_mul, mul, min, max);
+	if (ret) {
+		tst_brk(TBROK, "Failed to parse %s: %s",
+			env_name, tst_strerrno(ret));
+	}
+}
+
+static int multiply_runtime(void)
+{
+	static float runtime_mul = -1;
+
+	if (tst_test->max_iteration_runtime <= 0)
+		return tst_test->max_iteration_runtime;
+
+	parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
+
+	return tst_test->max_iteration_runtime * runtime_mul;
+}
+
 static struct option {
 	char *optstr;
 	char *help;
@@ -477,6 +514,7 @@ static struct option {
 static void print_help(void)
 {
 	unsigned int i;
+	int timeout, runtime;
 
 	/* see doc/user-guide.txt, which lists also shell API variables */
 	fprintf(stderr, "Environment Variables\n");
@@ -489,10 +527,32 @@ static void print_help(void)
 	fprintf(stderr, "LTP_DEV_FS_TYPE      Filesystem used for testing (default: %s)\n", DEFAULT_FS_TYPE);
 	fprintf(stderr, "LTP_SINGLE_FS_TYPE   Testing only - specifies filesystem instead all supported (for .all_filesystems)\n");
 	fprintf(stderr, "LTP_TIMEOUT_MUL      Timeout multiplier (must be a number >=1)\n");
+	fprintf(stderr, "LTP_RUNTIME_MUL      Runtime multiplier (must be a number >=1)\n");
 	fprintf(stderr, "LTP_VIRT_OVERRIDE    Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
 	fprintf(stderr, "TMPDIR               Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
 	fprintf(stderr, "\n");
 
+	fprintf(stderr, "Timeout and runtime\n");
+	fprintf(stderr, "-------------------\n");
+
+	if (tst_test->max_iteration_runtime) {
+		runtime = multiply_runtime();
+
+		if (runtime == TST_UNLIMITED_RUNTIME) {
+			fprintf(stderr, "Test iteration runtime is not limited\n");
+		} else {
+			fprintf(stderr, "Test iteration runtime cap %ih %im %is\n",
+				runtime/3600, (runtime%3600)/60, runtime % 60);
+		}
+	}
+
+	timeout = tst_multiply_timeout(DEFAULT_TIMEOUT);
+
+	fprintf(stderr, "Test timeout (not including runtime) %ih %im %is\n",
+		timeout/3600, (timeout%3600)/60, timeout % 60);
+
+	fprintf(stderr, "\n");
+
 	fprintf(stderr, "Options\n");
 	fprintf(stderr, "-------\n");
 
@@ -620,7 +680,10 @@ static void parse_opts(int argc, char *argv[])
 			iterations = atoi(optarg);
 		break;
 		case 'I':
-			duration = atof(optarg);
+			if (tst_test->max_iteration_runtime > 0)
+				tst_test->max_iteration_runtime = atoi(optarg);
+			else
+				duration = atof(optarg);
 		break;
 		case 'C':
 #ifdef UCLINUX
@@ -1034,6 +1097,11 @@ static void do_setup(int argc, char *argv[])
 	if (!tst_test)
 		tst_brk(TBROK, "No tests to run");
 
+	if (tst_test->max_iteration_runtime < -1) {
+		tst_brk(TBROK, "Invalid runtime value %i",
+			results->max_iteration_runtime);
+	}
+
 	if (tst_test->tconf_msg)
 		tst_brk(TCONF, "%s", tst_test->tconf_msg);
 
@@ -1404,39 +1472,36 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED)
 }
 
 unsigned int tst_timeout_remaining(void)
+{
+	tst_brk(TBROK, "Stub called!");
+	return 0;
+}
+
+unsigned int tst_remaining_runtime(void)
 {
 	static struct timespec now;
-	unsigned int elapsed;
+	int elapsed;
+
+	if (results->max_iteration_runtime == TST_UNLIMITED_RUNTIME)
+		return UINT_MAX;
+
+	if (results->max_iteration_runtime == 0)
+		tst_brk(TBROK, "Runtime not set!");
 
 	if (tst_clock_gettime(CLOCK_MONOTONIC, &now))
 		tst_res(TWARN | TERRNO, "tst_clock_gettime() failed");
 
-	elapsed = (tst_timespec_diff_ms(now, tst_start_time) + 500) / 1000;
-	if (results->timeout > elapsed)
-		return results->timeout - elapsed;
+	elapsed = tst_timespec_diff_ms(now, tst_start_time) / 1000;
+	if (results->max_iteration_runtime > elapsed)
+		return results->max_iteration_runtime - elapsed;
 
 	return 0;
 }
 
+
 unsigned int tst_multiply_timeout(unsigned int timeout)
 {
-	char *mul;
-	int ret;
-
-	if (timeout_mul == -1) {
-		mul = getenv("LTP_TIMEOUT_MUL");
-		if (mul) {
-			if ((ret = tst_parse_float(mul, &timeout_mul, 1, 10000))) {
-				tst_brk(TBROK, "Failed to parse LTP_TIMEOUT_MUL: %s",
-						tst_strerrno(ret));
-			}
-		} else {
-			timeout_mul = 1;
-		}
-	}
-	if (timeout_mul < 1)
-		tst_brk(TBROK, "LTP_TIMEOUT_MUL must to be int >= 1! (%.2f)",
-				timeout_mul);
+	parse_mul(&timeout_mul, "LTP_TIMEOUT_MUL", 0.099, 10000);
 
 	if (timeout < 1)
 		tst_brk(TBROK, "timeout must to be >= 1! (%d)", timeout);
@@ -1446,37 +1511,48 @@ unsigned int tst_multiply_timeout(unsigned int timeout)
 
 void tst_set_timeout(int timeout)
 {
-	if (timeout == -1) {
+	tst_brk(TBROK, "Stub called!");
+}
+
+static void set_timeout(void)
+{
+	unsigned int timeout = DEFAULT_TIMEOUT;
+
+	if (results->max_iteration_runtime == TST_UNLIMITED_RUNTIME) {
 		tst_res(TINFO, "Timeout per run is disabled");
 		return;
 	}
 
-	if (timeout < 1)
-		tst_brk(TBROK, "timeout must to be >= 1! (%d)", timeout);
+	if (results->max_iteration_runtime < 0) {
+		tst_brk(TBROK, "max_iteration_runtime must to be >= 0! (%d)",
+			results->max_iteration_runtime);
+	}
 
-	results->timeout = tst_multiply_timeout(timeout);
+	results->timeout = tst_multiply_timeout(timeout) + results->max_iteration_runtime;
 
 	tst_res(TINFO, "Timeout per run is %uh %02um %02us",
 		results->timeout/3600, (results->timeout%3600)/60,
 		results->timeout % 60);
+}
 
-	if (getpid() == lib_pid)
-		alarm(results->timeout);
-	else
-		heartbeat();
+void tst_set_runtime(int max_iteration_runtime)
+{
+	results->max_iteration_runtime = max_iteration_runtime;
+	tst_res(TINFO, "Updating max runtime to %uh %02um %02us",
+		max_iteration_runtime/3600, (max_iteration_runtime%3600)/60,
+		max_iteration_runtime % 60);
+	set_timeout();
+	heartbeat();
 }
 
 static int fork_testrun(void)
 {
 	int status;
 
-	if (tst_test->timeout)
-		tst_set_timeout(tst_test->timeout);
-	else
-		tst_set_timeout(300);
-
 	SAFE_SIGNAL(SIGINT, sigint_handler);
 
+	alarm(results->timeout);
+
 	test_pid = fork();
 	if (test_pid < 0)
 		tst_brk(TBROK | TERRNO, "fork()");
@@ -1568,6 +1644,11 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 	SAFE_SIGNAL(SIGALRM, alarm_handler);
 	SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
 
+	if (tst_test->max_iteration_runtime)
+		results->max_iteration_runtime = tst_test->max_iteration_runtime;
+
+	set_timeout();
+
 	if (tst_test->test_variants)
 		test_variants = tst_test->test_variants;
 
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 02/30] mtest06/mmap1: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/mem/mtest06/mmap1.c | 24 ++++++------------------
 1 file changed, 6 insertions(+), 18 deletions(-)

diff --git a/testcases/kernel/mem/mtest06/mmap1.c b/testcases/kernel/mem/mtest06/mmap1.c
index 10c47c35c..72506c37b 100644
--- a/testcases/kernel/mem/mtest06/mmap1.c
+++ b/testcases/kernel/mem/mtest06/mmap1.c
@@ -42,10 +42,8 @@
 
 static int file_size = 1024;
 static int num_iter = 5000;
-static float exec_time = 0.05; /* default is 3 min */
 
 static void *distant_area;
-static char *str_exec_time;
 static jmp_buf jmpbuf;
 static volatile unsigned char *map_address;
 static unsigned long page_sz;
@@ -206,17 +204,10 @@ static void setup(void)
 	SAFE_MUNMAP(distant_area, distant_mmap_size);
 	distant_area += distant_mmap_size / 2;
 
-	if (tst_parse_float(str_exec_time, &exec_time, 0, FLT_MAX)) {
-		tst_brk(TBROK, "Invalid number for exec_time '%s'",
-			str_exec_time);
-	}
-
 	sigptr.sa_sigaction = sig_handler;
 	sigemptyset(&sigptr.sa_mask);
 	sigptr.sa_flags = SA_SIGINFO | SA_NODEFER;
 	SAFE_SIGACTION(SIGSEGV, &sigptr, NULL);
-
-	tst_set_timeout((int)(exec_time * 3600));
 }
 
 static void run(void)
@@ -224,8 +215,8 @@ static void run(void)
 	pthread_t thid[2];
 	int start, last_update;
 
-	start = last_update = tst_timeout_remaining();
-	while (tst_timeout_remaining() > STOP_THRESHOLD) {
+	start = last_update = tst_remaining_runtime();
+	while (tst_remaining_runtime()) {
 		int fd = mkfile(file_size);
 
 		tst_atomic_store(0, &mapcnt);
@@ -240,11 +231,11 @@ static void run(void)
 
 		close(fd);
 
-		if (last_update - tst_timeout_remaining() >= PROGRESS_SEC) {
-			last_update = tst_timeout_remaining();
+		if (last_update - tst_remaining_runtime() >= PROGRESS_SEC) {
+			last_update = tst_remaining_runtime();
 			tst_res(TINFO, "[%03d] mapped: %lu, sigsegv hit: %lu, "
 				"threads spawned: %lu",
-				start - tst_timeout_remaining(),
+				start - last_update,
 				map_count, mapped_sigsegv_count,
 				threads_spawned);
 			tst_res(TINFO, "      repeated_reads: %ld, "
@@ -258,9 +249,6 @@ static void run(void)
 static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
-	.options = (struct tst_option[]) {
-		{"x:", &str_exec_time, "Exec time (hours)"},
-		{}
-	},
+	.max_iteration_runtime = 180,
 	.needs_tmpdir = 1,
 };
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 03/30] mtest06/mmap3: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/mem/mtest06/mmap3.c | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/testcases/kernel/mem/mtest06/mmap3.c b/testcases/kernel/mem/mtest06/mmap3.c
index 23609752f..461131e15 100644
--- a/testcases/kernel/mem/mtest06/mmap3.c
+++ b/testcases/kernel/mem/mtest06/mmap3.c
@@ -23,11 +23,9 @@
 static char *str_loops;
 static char *str_threads;
 static char *map_private;
-static char *str_exec_time;
 
 static int loops = 1000;
 static int threads = 40;
-static float exec_time = 24;
 
 static volatile int sig_caught;
 static int threads_running;
@@ -109,7 +107,7 @@ static void test_mmap(void)
 	long i;
 	pthread_t thids[threads];
 
-	alarm(exec_time * 3600);
+	alarm(tst_remaining_runtime());
 
 	while (!sig_caught) {
 		for (i = 0; i < threads; i++) {
@@ -138,11 +136,6 @@ static void setup(void)
 	if (tst_parse_int(str_threads, &threads, 1, INT_MAX))
 		tst_brk(TBROK, "Invalid number of threads '%s'", str_threads);
 
-	if (tst_parse_float(str_exec_time, &exec_time, 0.0005, 9000))
-		tst_brk(TBROK, "Invalid execution time '%s'", str_exec_time);
-
-	tst_set_timeout(exec_time * 3600 + 300);
-
 	SAFE_SIGNAL(SIGALRM, sig_handler);
 	SAFE_SIGNAL(SIGBUS, sig_handler);
 	SAFE_SIGNAL(SIGSEGV, sig_handler);
@@ -155,7 +148,6 @@ static void setup(void)
 	tst_res(TINFO, "Number of loops %i", loops);
 	tst_res(TINFO, "Number of threads %i", threads);
 	tst_res(TINFO, "MAP_PRIVATE = %i", map_private ? 1 : 0);
-	tst_res(TINFO, "Execution time %fH", exec_time);
 }
 
 static void cleanup(void)
@@ -184,11 +176,11 @@ static struct tst_test test = {
 		{"l:", &str_loops, "Number of map-write-unmap loops"},
 		{"n:", &str_threads, "Number of worker threads"},
 		{"p", &map_private, "Turns on MAP_PRIVATE (default MAP_SHARED)"},
-		{"x:", &str_exec_time, "float Execution time in hours (default 24H)"},
 		{}
 	},
 	.needs_tmpdir = 1,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = test_mmap,
+	.max_iteration_runtime = 86400,
 };
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 04/30] mtest01/mtest01: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/mem/mtest01/mtest01.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/testcases/kernel/mem/mtest01/mtest01.c b/testcases/kernel/mem/mtest01/mtest01.c
index 20cb95191..2991d3d2e 100644
--- a/testcases/kernel/mem/mtest01/mtest01.c
+++ b/testcases/kernel/mem/mtest01/mtest01.c
@@ -41,8 +41,6 @@
 #define ALLOC_THRESHOLD		(6*FIVE_HUNDRED_MB)
 #endif
 
-#define STOP_THRESHOLD 15	/* seconds remaining before reaching timeout */
-
 static pid_t *pid_list;
 static sig_atomic_t children_done;
 static int max_pids;
@@ -137,6 +135,7 @@ static void child_loop_alloc(unsigned long long alloc_bytes)
 {
 	unsigned long bytecount = 0;
 	char *mem;
+	int runtime_rem;
 
 	tst_res(TINFO, "... child %d starting", getpid());
 
@@ -153,12 +152,15 @@ static void child_loop_alloc(unsigned long long alloc_bytes)
 		if (bytecount >= alloc_bytes)
 			break;
 	}
+
+	runtime_rem = tst_remaining_runtime();
+
 	if (dowrite)
 		tst_res(TINFO, "... [t=%d] %lu bytes allocated and used in child %d",
-				tst_timeout_remaining(), bytecount, getpid());
+				runtime_rem, bytecount, getpid());
 	else
 		tst_res(TINFO, "... [t=%d] %lu bytes allocated only in child %d",
-				tst_timeout_remaining(), bytecount, getpid());
+				runtime_rem, bytecount, getpid());
 
 	kill(getppid(), SIGRTMIN);
 	raise(SIGSTOP);
@@ -195,10 +197,9 @@ static void mem_test(void)
 
 	/* wait in the loop for all children finish allocating */
 	while (children_done < pid_cntr) {
-		if (tst_timeout_remaining() < STOP_THRESHOLD) {
+		if (!tst_remaining_runtime()) {
 			tst_res(TWARN,
 				"the remaininig time is not enough for testing");
-
 			break;
 		}
 
@@ -234,6 +235,7 @@ static struct tst_test test = {
 		{"v",  &verbose,     	"Verbose"},
 		{}
 	},
+	.max_iteration_runtime = 300,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = mem_test,
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 05/30] cve/cve-2015-3290: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/cve/cve-2015-3290.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/testcases/cve/cve-2015-3290.c b/testcases/cve/cve-2015-3290.c
index fd1abe136..91fe813eb 100644
--- a/testcases/cve/cve-2015-3290.c
+++ b/testcases/cve/cve-2015-3290.c
@@ -395,10 +395,6 @@ static void *child_thread(void *arg LTP_ATTRIBUTE_UNUSED)
 	return (void *)niter;
 }
 
-#define TIMEOUT		(180)
-#define TIME_TO_GIVEUP	(TIMEOUT - 5)
-#define TIMER_TYPE	CLOCK_MONOTONIC
-
 static void do_child(void)
 {
 	int i, ncpus;
@@ -415,7 +411,7 @@ static void do_child(void)
 	for (i = 0; i < ncpus; i++)
 		SAFE_PTHREAD_CREATE(&threads[i], NULL, child_thread, NULL);
 
-	sleep(TIME_TO_GIVEUP);
+	sleep(tst_remaining_runtime());
 	running = 0;
 
 	for (i = 0; i < ncpus; i++) {
@@ -468,7 +464,7 @@ static struct tst_test test = {
 	.needs_root = 1,
 	.needs_checkpoints = 1,
 	.setup = setup,
-	.timeout = TIMEOUT,
+	.max_iteration_runtime = 180,
 	.test_all = run,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "9b6e6a8334d5"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 06/30] crypto/af_alg02: Convert to runtime.

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/crypto/af_alg02.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testcases/kernel/crypto/af_alg02.c b/testcases/kernel/crypto/af_alg02.c
index 9894ffacd..b96b8b341 100644
--- a/testcases/kernel/crypto/af_alg02.c
+++ b/testcases/kernel/crypto/af_alg02.c
@@ -65,7 +65,7 @@ static void run(void)
 	TST_CHECKPOINT_WAIT(0);
 
 	while (!completed) {
-		if (tst_timeout_remaining() <= 10) {
+		if (tst_remaining_runtime()) {
 			pthread_cancel(thr);
 			tst_brk(TBROK,
 				"Timed out while reading from request socket.");
@@ -77,7 +77,7 @@ static void run(void)
 
 static struct tst_test test = {
 	.test_all = run,
-	.timeout = 20,
+	.runtime_per_iteration = 20,
 	.needs_checkpoints = 1,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "ecaaab564978"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 07/30] crypto/pcrypt_aead01: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/crypto/pcrypt_aead01.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/testcases/kernel/crypto/pcrypt_aead01.c b/testcases/kernel/crypto/pcrypt_aead01.c
index 0609af9f6..d10d744c5 100644
--- a/testcases/kernel/crypto/pcrypt_aead01.c
+++ b/testcases/kernel/crypto/pcrypt_aead01.c
@@ -55,7 +55,7 @@ void run(void)
 		if (TST_RET)
 			tst_brk(TBROK | TRERRNO, "del_alg");
 
-		if (tst_timeout_remaining() < 10) {
+		if (!tst_remaining_runtime()) {
 			tst_res(TINFO, "Time limit reached, stopping at "
 				"%d iterations", i);
 			break;
@@ -75,6 +75,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.cleanup = cleanup,
 	.needs_root = 1,
+	.max_iteration_runtime = 300,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d76c68109f37"},
 		{"CVE", "2017-5754"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 08/30] syscalls/clock_gettime01: Remove useless timeout

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/clock_gettime/clock_gettime01.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/testcases/kernel/syscalls/clock_gettime/clock_gettime01.c b/testcases/kernel/syscalls/clock_gettime/clock_gettime01.c
index 11da39cf0..a67639b05 100644
--- a/testcases/kernel/syscalls/clock_gettime/clock_gettime01.c
+++ b/testcases/kernel/syscalls/clock_gettime/clock_gettime01.c
@@ -126,5 +126,4 @@ static struct tst_test test = {
 	.test_variants = ARRAY_SIZE(variants),
 	.setup = setup,
 	.needs_root = 1,
-	.timeout = 10,
 };
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 09/30] syscalls/fanotify22: Remove useless timeout

[Cyril Hrubis]

I guess that the timeout was there to exit faster in a case that the
test gets stuck. However that is no longer needed as the default timeout
was shortented significantly.

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/fanotify/fanotify22.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/testcases/kernel/syscalls/fanotify/fanotify22.c b/testcases/kernel/syscalls/fanotify/fanotify22.c
index f45c8512d..1105172bb 100644
--- a/testcases/kernel/syscalls/fanotify/fanotify22.c
+++ b/testcases/kernel/syscalls/fanotify/fanotify22.c
@@ -305,7 +305,6 @@ static struct tst_test test = {
 		{"linux-git", "124e7c61deb2"},
 		{}
 	},
-	.timeout = 10,
 	.needs_cmds = (const char *[]) {
 		"debugfs",
 		NULL
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 10/30] syscalls/gettimeofday02: Convert to runtime.

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 .../kernel/syscalls/gettimeofday/gettimeofday02.c | 15 ++-------------
 1 file changed, 2 insertions(+), 13 deletions(-)

diff --git a/testcases/kernel/syscalls/gettimeofday/gettimeofday02.c b/testcases/kernel/syscalls/gettimeofday/gettimeofday02.c
index 948d2d8ec..6f175e79b 100644
--- a/testcases/kernel/syscalls/gettimeofday/gettimeofday02.c
+++ b/testcases/kernel/syscalls/gettimeofday/gettimeofday02.c
@@ -25,8 +25,6 @@
 #include "lapi/syscalls.h"
 
 static volatile sig_atomic_t done;
-static char *str_rtime;
-static int rtime = 10;
 
 static void breakout(int sig)
 {
@@ -37,6 +35,7 @@ static void verify_gettimeofday(void)
 {
 	struct __kernel_old_timeval tv1, tv2;
 	unsigned long long cnt = 0;
+	int rtime = tst_remaining_runtime();
 
 	done = 0;
 
@@ -68,21 +67,11 @@ static void verify_gettimeofday(void)
 
 static void setup(void)
 {
-	if (str_rtime) {
-		rtime = atoi(str_rtime);
-		if (rtime <= 0)
-			tst_brk(TBROK, "Invalid runtime '%s'", str_rtime);
-		tst_set_timeout(rtime + 60);
-	}
-
 	SAFE_SIGNAL(SIGALRM, breakout);
 }
 
 static struct tst_test test = {
 	.setup = setup,
-	.options = (struct tst_option[]) {
-		{"T:", &str_rtime, "Test iteration runtime in seconds"},
-		{},
-	},
+	.max_iteration_runtime = 10,
 	.test_all = verify_gettimeofday,
 };
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 11/30] syscalls/inotify06: Convert to runtime.

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/inotify/inotify06.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/testcases/kernel/syscalls/inotify/inotify06.c b/testcases/kernel/syscalls/inotify/inotify06.c
index 68813769b..10b5de0bc 100644
--- a/testcases/kernel/syscalls/inotify/inotify06.c
+++ b/testcases/kernel/syscalls/inotify/inotify06.c
@@ -87,6 +87,11 @@ static void verify_inotify(void)
 			myinotify_add_watch(inotify_fd, names[i], IN_MODIFY);
 		}
 		SAFE_CLOSE(inotify_fd);
+
+		if (!tst_remaining_runtime()) {
+			tst_res(TINFO, "Test out of runtime, exitting");
+			break;
+		}
 	}
 	/* We survived for given time - test succeeded */
 	tst_res(TPASS, "kernel survived inotify beating");
@@ -108,7 +113,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
-	.timeout = 600,
+	.max_iteration_runtime = 600,
 	.needs_root = 1,
 	.needs_tmpdir = 1,
 	.forks_child = 1,
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 12/30] syscalls/inotify01: Remove now useless timeout

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/inotify/inotify11.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/testcases/kernel/syscalls/inotify/inotify11.c b/testcases/kernel/syscalls/inotify/inotify11.c
index 062b92409..e426f9e1c 100644
--- a/testcases/kernel/syscalls/inotify/inotify11.c
+++ b/testcases/kernel/syscalls/inotify/inotify11.c
@@ -117,7 +117,6 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
-	.timeout = 10,
 	.needs_tmpdir = 1,
 	.forks_child = 1,
 	.cleanup = cleanup,
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 13/30] syscalls/perf_event_open03: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 .../syscalls/perf_event_open/perf_event_open03.c  | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/testcases/kernel/syscalls/perf_event_open/perf_event_open03.c b/testcases/kernel/syscalls/perf_event_open/perf_event_open03.c
index 6dffc8272..38897d1bf 100644
--- a/testcases/kernel/syscalls/perf_event_open/perf_event_open03.c
+++ b/testcases/kernel/syscalls/perf_event_open/perf_event_open03.c
@@ -25,7 +25,7 @@
 
 const int iterations = 12000000;
 static int fd = -1;
-static int timeout;
+static int runtime;
 
 static void setup(void)
 {
@@ -43,12 +43,12 @@ static void setup(void)
 	SAFE_FILE_SCANF(INTEL_PT_PATH, "%d", &ev.type);
 	fd = perf_event_open(&ev, getpid(), -1, -1, 0);
 
-	timeout = tst_timeout_remaining();
+	runtime = tst_remaining_runtime();
 }
 
 /*
  * Check how fast we can do the iterations after 5 seconds of runtime.
- * If the rate is too small to complete for current timeout then
+ * If the rate is too small to complete for current runtime then
  * stop the test.
  */
 static void check_progress(int i)
@@ -67,11 +67,11 @@ static void check_progress(int i)
 	if (elapsed_ms > 5000) {
 		iter_per_ms = (float) i / elapsed_ms;
 		tst_res(TINFO, "rate: %f iters/ms", iter_per_ms);
-		tst_res(TINFO, "needed rate for current test timeout: %f iters/ms",
-			(float) iterations / (timeout * 1000));
+		tst_res(TINFO, "needed rate for current test runtime: %f iters/ms",
+			(float) iterations / (runtime * 1000));
 
-		if (iter_per_ms * 1000 * (timeout - 1) < iterations)
-			tst_brk(TCONF, "System too slow to complete test in specified timeout");
+		if (iter_per_ms * 1000 * (runtime - 1) < iterations)
+			tst_brk(TCONF, "System too slow to complete test in specified runtime");
 	}
 }
 
@@ -108,6 +108,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.needs_root = 1,
+	.max_iteration_runtime = 300,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "7bdb157cdebb"},
 		{"CVE", "2020-25704"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 14/30] syscalls/readv01: Remove now useless timeout

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/readv/readv01.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/testcases/kernel/syscalls/readv/readv01.c b/testcases/kernel/syscalls/readv/readv01.c
index f12b3f09f..2bd3fd5eb 100644
--- a/testcases/kernel/syscalls/readv/readv01.c
+++ b/testcases/kernel/syscalls/readv/readv01.c
@@ -110,7 +110,6 @@ static struct tst_test test = {
 	.test = test_readv,
 	.tcnt = ARRAY_SIZE(testcase_list),
 	.needs_tmpdir = 1,
-	.timeout = 15,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "19f18459330f"},
 		{}
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 15/30] syscalls/tgkill03: Remove now unused timeout

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/tgkill/tgkill03.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/testcases/kernel/syscalls/tgkill/tgkill03.c b/testcases/kernel/syscalls/tgkill/tgkill03.c
index 0002f3278..e46e95f73 100644
--- a/testcases/kernel/syscalls/tgkill/tgkill03.c
+++ b/testcases/kernel/syscalls/tgkill/tgkill03.c
@@ -117,5 +117,4 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test = run,
-	.timeout = 20,
 };
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 16/30] syscalls/setsockopt09: Remove now useless timeout

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/setsockopt/setsockopt09.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/testcases/kernel/syscalls/setsockopt/setsockopt09.c b/testcases/kernel/syscalls/setsockopt/setsockopt09.c
index 3047ec0ff..98f7fd00e 100644
--- a/testcases/kernel/syscalls/setsockopt/setsockopt09.c
+++ b/testcases/kernel/syscalls/setsockopt/setsockopt09.c
@@ -117,7 +117,6 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.timeout = 5,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 17/30] syscalls/userfaultfd01: Remove now useless timeout

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/userfaultfd/userfaultfd01.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/testcases/kernel/syscalls/userfaultfd/userfaultfd01.c b/testcases/kernel/syscalls/userfaultfd/userfaultfd01.c
index 4e178b4f8..2dae2ec78 100644
--- a/testcases/kernel/syscalls/userfaultfd/userfaultfd01.c
+++ b/testcases/kernel/syscalls/userfaultfd/userfaultfd01.c
@@ -121,7 +121,6 @@ static void run(void)
 static struct tst_test test = {
 	.test_all = run,
 	.min_kver = "4.3",
-	.timeout = 20
 };
 
 #else
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 18/30] syscalls/move_pages12: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/move_pages/move_pages12.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testcases/kernel/syscalls/move_pages/move_pages12.c b/testcases/kernel/syscalls/move_pages/move_pages12.c
index 220130f4b..26271eaaf 100644
--- a/testcases/kernel/syscalls/move_pages/move_pages12.c
+++ b/testcases/kernel/syscalls/move_pages/move_pages12.c
@@ -153,7 +153,6 @@ static void do_test(unsigned int n)
 	void *ptr;
 	pid_t cpid = -1;
 	int status;
-	unsigned int twenty_percent = (tst_timeout_remaining() / 5);
 
 	addr = SAFE_MMAP(NULL, tcases[n].tpages * hpsz, PROT_READ | PROT_WRITE,
 		MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
@@ -198,7 +197,7 @@ static void do_test(unsigned int n)
 
 		SAFE_MUNMAP(addr, tcases[n].tpages * hpsz);
 
-		if (tst_timeout_remaining() < twenty_percent)
+		if (!tst_remaining_runtime())
 			break;
 	}
 
@@ -341,6 +340,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.test = do_test,
 	.tcnt = ARRAY_SIZE(tcases),
+	.max_iteration_runtime = 240,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "e66f17ff7177"},
 		{"linux-git", "c9d398fa2378"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 19/30] syscalls/rt_sigqueueinfo01: Remove now useless timeout

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/rt_sigqueueinfo/rt_sigqueueinfo01.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/testcases/kernel/syscalls/rt_sigqueueinfo/rt_sigqueueinfo01.c b/testcases/kernel/syscalls/rt_sigqueueinfo/rt_sigqueueinfo01.c
index 5996e998b..c8b07c077 100644
--- a/testcases/kernel/syscalls/rt_sigqueueinfo/rt_sigqueueinfo01.c
+++ b/testcases/kernel/syscalls/rt_sigqueueinfo/rt_sigqueueinfo01.c
@@ -108,7 +108,6 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.needs_checkpoints = 1,
-	.timeout = 20,
 };
 
 #else
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 20/30] mem/mallocstress: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/mem/mtest07/mallocstress.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/testcases/kernel/mem/mtest07/mallocstress.c b/testcases/kernel/mem/mtest07/mallocstress.c
index fa7494e7d..089e42bc0 100644
--- a/testcases/kernel/mem/mtest07/mallocstress.c
+++ b/testcases/kernel/mem/mtest07/mallocstress.c
@@ -62,7 +62,7 @@ static void my_yield(void)
  *  0: success
  *  1: failure
  */
-int allocate_free(int scheme)
+int allocate_free(int scheme, int threadnum)
 {
 	int loop;
 	const int MAXPTRS = 50;	/* only 42 or so get used on 32 bit machine */
@@ -127,6 +127,11 @@ int allocate_free(int scheme)
 		}
 
 		my_yield();
+
+		if (!tst_remaining_runtime()) {
+			tst_res(TINFO, "Thread [%d]: Test runtime is over, exitting", threadnum);
+			break;
+		}
 	}
 
 	/* Success! */
@@ -141,7 +146,7 @@ void *alloc_mem(void *threadnum)
 	TST_CHECKPOINT_WAIT(0);
 
 	/* thread N will use growth scheme N mod 4 */
-	err = allocate_free(((uintptr_t)threadnum) % 4);
+	err = allocate_free(((uintptr_t)threadnum) % 4, (uintptr_t)threadnum);
 	tst_res(TINFO,
 		"Thread [%d]: allocate_free() returned %d, %s.  Thread exiting.\n",
 		(int)(uintptr_t)threadnum, err,
@@ -189,7 +194,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
-	.timeout = 600,
+	.max_iteration_runtime = 600,
 	.needs_checkpoints = 1,
 	.setup = setup,
 	.cleanup = cleanup,
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 21/30] mem/{oom, min_free_kbytes}: Convert to runtime

[Cyril Hrubis]

This just sets the runtime to TST_UNLIMITED_RUNTIME which disables the
timeout.

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/mem/oom/oom01.c               | 2 +-
 testcases/kernel/mem/oom/oom02.c               | 2 +-
 testcases/kernel/mem/oom/oom03.c               | 2 +-
 testcases/kernel/mem/oom/oom04.c               | 2 +-
 testcases/kernel/mem/oom/oom05.c               | 2 +-
 testcases/kernel/mem/tunable/min_free_kbytes.c | 2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/testcases/kernel/mem/oom/oom01.c b/testcases/kernel/mem/oom/oom01.c
index 258bfd89e..c4dd55e63 100644
--- a/testcases/kernel/mem/oom/oom01.c
+++ b/testcases/kernel/mem/oom/oom01.c
@@ -63,7 +63,7 @@ static void cleanup(void)
 static struct tst_test test = {
 	.needs_root = 1,
 	.forks_child = 1,
-	.timeout = -1,
+	.max_iteration_runtime = TST_UNLIMITED_RUNTIME,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_oom,
diff --git a/testcases/kernel/mem/oom/oom02.c b/testcases/kernel/mem/oom/oom02.c
index 2b9bcb1b2..bc6f9c105 100644
--- a/testcases/kernel/mem/oom/oom02.c
+++ b/testcases/kernel/mem/oom/oom02.c
@@ -70,7 +70,7 @@ static void cleanup(void)
 static struct tst_test test = {
 	.needs_root = 1,
 	.forks_child = 1,
-	.timeout = -1,
+        .max_iteration_runtime = TST_UNLIMITED_RUNTIME,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_oom,
diff --git a/testcases/kernel/mem/oom/oom03.c b/testcases/kernel/mem/oom/oom03.c
index 48a0f20b5..cfbcba615 100644
--- a/testcases/kernel/mem/oom/oom03.c
+++ b/testcases/kernel/mem/oom/oom03.c
@@ -95,7 +95,7 @@ static void cleanup(void)
 static struct tst_test test = {
 	.needs_root = 1,
 	.forks_child = 1,
-	.timeout = -1,
+	.max_iteration_runtime = TST_UNLIMITED_RUNTIME,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_oom,
diff --git a/testcases/kernel/mem/oom/oom04.c b/testcases/kernel/mem/oom/oom04.c
index 6b65383a3..fb70a69ae 100644
--- a/testcases/kernel/mem/oom/oom04.c
+++ b/testcases/kernel/mem/oom/oom04.c
@@ -91,7 +91,7 @@ static void cleanup(void)
 static struct tst_test test = {
 	.needs_root = 1,
 	.forks_child = 1,
-	.timeout = -1,
+	.max_iteration_runtime = TST_UNLIMITED_RUNTIME,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_oom,
diff --git a/testcases/kernel/mem/oom/oom05.c b/testcases/kernel/mem/oom/oom05.c
index 91c9bec83..6a8750353 100644
--- a/testcases/kernel/mem/oom/oom05.c
+++ b/testcases/kernel/mem/oom/oom05.c
@@ -113,7 +113,7 @@ void cleanup(void)
 static struct tst_test test = {
 	.needs_root = 1,
 	.forks_child = 1,
-	.timeout = -1,
+	.max_iteration_runtime = TST_UNLIMITED_RUNTIME,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_oom,
diff --git a/testcases/kernel/mem/tunable/min_free_kbytes.c b/testcases/kernel/mem/tunable/min_free_kbytes.c
index 09741eea3..c9b766c2b 100644
--- a/testcases/kernel/mem/tunable/min_free_kbytes.c
+++ b/testcases/kernel/mem/tunable/min_free_kbytes.c
@@ -231,7 +231,7 @@ static void cleanup(void)
 static struct tst_test test = {
 	.needs_root = 1,
 	.forks_child = 1,
-	.timeout = -1,
+	.max_iteration_runtime = TST_UNLIMITED_RUNTIME,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = min_free_kbytes_test,
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 22/30] crypto/af_alg02: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/crypto/af_alg02.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testcases/kernel/crypto/af_alg02.c b/testcases/kernel/crypto/af_alg02.c
index b96b8b341..9c2f488c4 100644
--- a/testcases/kernel/crypto/af_alg02.c
+++ b/testcases/kernel/crypto/af_alg02.c
@@ -65,7 +65,7 @@ static void run(void)
 	TST_CHECKPOINT_WAIT(0);
 
 	while (!completed) {
-		if (tst_remaining_runtime()) {
+		if (!tst_remaining_runtime()) {
 			pthread_cancel(thr);
 			tst_brk(TBROK,
 				"Timed out while reading from request socket.");
@@ -77,7 +77,7 @@ static void run(void)
 
 static struct tst_test test = {
 	.test_all = run,
-	.runtime_per_iteration = 20,
+	.max_iteration_runtime = 20,
 	.needs_checkpoints = 1,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "ecaaab564978"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 23/30] fuzzy_sync: Convert to runtime

[Cyril Hrubis]

The runtime for fuzzy sync tests are computed as timeout * exec_time_p.
The default timeout was 300 and the default was 0.5.

CC: Richard Palethorpe <rpalethorpe@suse.com>
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 include/tst_fuzzy_sync.h                      | 19 ++++---------------
 testcases/kernel/crypto/af_alg07.c            |  1 +
 testcases/kernel/mem/thp/thp04.c              |  1 +
 testcases/kernel/pty/pty03.c                  |  2 +-
 testcases/kernel/pty/pty05.c                  |  1 +
 testcases/kernel/pty/pty06.c                  |  1 +
 testcases/kernel/pty/pty07.c                  |  1 +
 testcases/kernel/sound/snd_seq01.c            |  2 +-
 testcases/kernel/sound/snd_timer01.c          |  1 +
 testcases/kernel/syscalls/bind/bind06.c       |  2 +-
 testcases/kernel/syscalls/inotify/inotify09.c |  1 +
 .../kernel/syscalls/ipc/shmctl/shmctl05.c     |  2 +-
 testcases/kernel/syscalls/sendmsg/sendmsg03.c |  1 +
 .../kernel/syscalls/setsockopt/setsockopt06.c |  2 +-
 .../kernel/syscalls/setsockopt/setsockopt07.c |  1 +
 .../syscalls/timerfd/timerfd_settime02.c      |  1 +
 testcases/kernel/syscalls/writev/writev03.c   |  2 +-
 17 files changed, 20 insertions(+), 21 deletions(-)

diff --git a/include/tst_fuzzy_sync.h b/include/tst_fuzzy_sync.h
index 4f09ed416..bef424002 100644
--- a/include/tst_fuzzy_sync.h
+++ b/include/tst_fuzzy_sync.h
@@ -160,15 +160,6 @@ struct tst_fzsync_pair {
 	int b_cntr;
 	/** Internal; Used by tst_fzsync_pair_exit() and fzsync_pair_wait() */
 	int exit;
-	/**
-	 * The maximum desired execution time as a proportion of the timeout
-	 *
-	 * A value x so that 0 < x < 1 which decides how long the test should
-	 * be run for (assuming the loop limit is not exceeded first).
-	 *
-	 * Defaults to 0.5 (~150 seconds with default timeout).
-	 */
-	float exec_time_p;
 	/** Internal; The test time remaining on tst_fzsync_pair_reset() */
 	float exec_time_start;
 	/**
@@ -214,7 +205,6 @@ static inline void tst_fzsync_pair_init(struct tst_fzsync_pair *pair)
 	CHK(avg_alpha, 0, 1, 0.25);
 	CHK(min_samples, 20, INT_MAX, 1024);
 	CHK(max_dev_ratio, 0, 1, 0.1);
-	CHK(exec_time_p, 0, 1, 0.5);
 	CHK(exec_loops, 20, INT_MAX, 3000000);
 
 	if (tst_ncpus_available() <= 1)
@@ -291,7 +281,7 @@ static inline void tst_fzsync_pair_reset(struct tst_fzsync_pair *pair,
 	if (run_b)
 		SAFE_PTHREAD_CREATE(&pair->thread_b, 0, run_b, 0);
 
-	pair->exec_time_start = (float)tst_timeout_remaining();
+	pair->exec_time_start = (float)tst_remaining_runtime();
 }
 
 /**
@@ -644,10 +634,9 @@ static inline void tst_fzsync_wait_b(struct tst_fzsync_pair *pair)
  */
 static inline int tst_fzsync_run_a(struct tst_fzsync_pair *pair)
 {
-	float rem_p = 1 - tst_timeout_remaining() / pair->exec_time_start;
+	float rem_p = 1 - tst_remaining_runtime() / pair->exec_time_start;
 
-	if ((pair->exec_time_p * SAMPLING_SLICE < rem_p)
-		&& (pair->sampling > 0)) {
+	if ((SAMPLING_SLICE < rem_p) && (pair->sampling > 0)) {
 		tst_res(TINFO, "Stopped sampling at %d (out of %d) samples, "
 			"sampling time reached 50%% of the total time limit",
 			pair->exec_loop, pair->min_samples);
@@ -655,7 +644,7 @@ static inline int tst_fzsync_run_a(struct tst_fzsync_pair *pair)
 		tst_fzsync_pair_info(pair);
 	}
 
-	if (pair->exec_time_p < rem_p) {
+	if (rem_p >= 1) {
 		tst_res(TINFO,
 			"Exceeded execution time, requesting exit");
 		tst_atomic_store(1, &pair->exit);
diff --git a/testcases/kernel/crypto/af_alg07.c b/testcases/kernel/crypto/af_alg07.c
index ef13ad764..4754c501e 100644
--- a/testcases/kernel/crypto/af_alg07.c
+++ b/testcases/kernel/crypto/af_alg07.c
@@ -125,6 +125,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.min_kver = "4.10.0",
 	.min_cpus = 2,
+	.max_iteration_runtime = 150,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "ff7b11aa481f"},
diff --git a/testcases/kernel/mem/thp/thp04.c b/testcases/kernel/mem/thp/thp04.c
index 985394dc3..77f49e777 100644
--- a/testcases/kernel/mem/thp/thp04.c
+++ b/testcases/kernel/mem/thp/thp04.c
@@ -163,6 +163,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
+	.max_iteration_runtime = 150,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "a8f97366452e"},
 		{"linux-git", "8310d48b125d"},
diff --git a/testcases/kernel/pty/pty03.c b/testcases/kernel/pty/pty03.c
index 71bcb2eb6..f5680e7c5 100644
--- a/testcases/kernel/pty/pty03.c
+++ b/testcases/kernel/pty/pty03.c
@@ -135,7 +135,6 @@ static void do_test(unsigned int n)
 static void setup(void)
 {
 	fzp.min_samples = 20;
-	fzp.exec_time_p = 0.1;
 
 	tst_fzsync_pair_init(&fzp);
 }
@@ -151,6 +150,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.needs_root = 1,
+	.max_iteration_runtime = 30,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "0ace17d568241"},
 		{"CVE", "2020-14416"},
diff --git a/testcases/kernel/pty/pty05.c b/testcases/kernel/pty/pty05.c
index afef051c8..72a2f2c36 100644
--- a/testcases/kernel/pty/pty05.c
+++ b/testcases/kernel/pty/pty05.c
@@ -97,6 +97,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
+	.max_iteration_runtime = 150,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "82f2341c94d27"},
 		{"CVE", "2017-2636"},
diff --git a/testcases/kernel/pty/pty06.c b/testcases/kernel/pty/pty06.c
index cc8eb1f85..37edbc981 100644
--- a/testcases/kernel/pty/pty06.c
+++ b/testcases/kernel/pty/pty06.c
@@ -97,6 +97,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_root = 1,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
+	.max_iteration_runtime = 150,
 	.tags = (const struct tst_tag[]) {
 	    { "linux-git", "ca4463bf8438"},
 	    {}
diff --git a/testcases/kernel/pty/pty07.c b/testcases/kernel/pty/pty07.c
index c63b71c89..0190b581d 100644
--- a/testcases/kernel/pty/pty07.c
+++ b/testcases/kernel/pty/pty07.c
@@ -110,6 +110,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_root = 1,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
+	.max_iteration_runtime = 150,
 	.tags = (const struct tst_tag[]) {
 		{ "linux-git", "6cd1ed50efd8"},
 		{}
diff --git a/testcases/kernel/sound/snd_seq01.c b/testcases/kernel/sound/snd_seq01.c
index c56752230..3ce9550c7 100644
--- a/testcases/kernel/sound/snd_seq01.c
+++ b/testcases/kernel/sound/snd_seq01.c
@@ -123,7 +123,7 @@ static struct tst_test test = {
 	.tcnt = ARRAY_SIZE(testfunc_list),
 	.setup = setup,
 	.cleanup = cleanup,
-	.timeout = 120,
+	.max_iteration_runtime = 60,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d15d662e89fc"},
diff --git a/testcases/kernel/sound/snd_timer01.c b/testcases/kernel/sound/snd_timer01.c
index 51591c18e..4c46c966f 100644
--- a/testcases/kernel/sound/snd_timer01.c
+++ b/testcases/kernel/sound/snd_timer01.c
@@ -139,6 +139,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
+	.max_iteration_runtime = 150,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d11662f4f798"},
 		{"linux-git", "ba3021b2c79b"},
diff --git a/testcases/kernel/syscalls/bind/bind06.c b/testcases/kernel/syscalls/bind/bind06.c
index 1dfc534a7..a2c1d67c1 100644
--- a/testcases/kernel/syscalls/bind/bind06.c
+++ b/testcases/kernel/syscalls/bind/bind06.c
@@ -102,7 +102,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.timeout = 600,
+	.max_iteration_runtime = 300,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
diff --git a/testcases/kernel/syscalls/inotify/inotify09.c b/testcases/kernel/syscalls/inotify/inotify09.c
index fdfc9c078..8364a9e6a 100644
--- a/testcases/kernel/syscalls/inotify/inotify09.c
+++ b/testcases/kernel/syscalls/inotify/inotify09.c
@@ -94,6 +94,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_inotify,
+	.max_iteration_runtime = 150,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d90a10e2444b"},
 		{}
diff --git a/testcases/kernel/syscalls/ipc/shmctl/shmctl05.c b/testcases/kernel/syscalls/ipc/shmctl/shmctl05.c
index 228174c9b..2a7dd3c0f 100644
--- a/testcases/kernel/syscalls/ipc/shmctl/shmctl05.c
+++ b/testcases/kernel/syscalls/ipc/shmctl/shmctl05.c
@@ -106,7 +106,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
-	.timeout = 20,
+	.max_iteration_runtime = 10,
 	.setup = setup,
 	.test_all = do_test,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/sendmsg/sendmsg03.c b/testcases/kernel/syscalls/sendmsg/sendmsg03.c
index c7d72f686..fb9182dec 100644
--- a/testcases/kernel/syscalls/sendmsg/sendmsg03.c
+++ b/testcases/kernel/syscalls/sendmsg/sendmsg03.c
@@ -105,6 +105,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
+	.max_iteration_runtime = 150,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "8f659a03a0ba"},
 		{"CVE", "2017-17712"},
diff --git a/testcases/kernel/syscalls/setsockopt/setsockopt06.c b/testcases/kernel/syscalls/setsockopt/setsockopt06.c
index c6fc2fba9..dafb9665f 100644
--- a/testcases/kernel/syscalls/setsockopt/setsockopt06.c
+++ b/testcases/kernel/syscalls/setsockopt/setsockopt06.c
@@ -46,7 +46,6 @@ static void setup(void)
 	SAFE_FILE_PRINTF("/proc/self/gid_map", "0 %d 1", real_gid);
 
 	fzsync_pair.exec_loops = 100000;
-	fzsync_pair.exec_time_p = 0.9;
 	tst_fzsync_pair_init(&fzsync_pair);
 }
 
@@ -123,6 +122,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
+	.max_iteration_runtime = 270,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
diff --git a/testcases/kernel/syscalls/setsockopt/setsockopt07.c b/testcases/kernel/syscalls/setsockopt/setsockopt07.c
index 147ae3899..cde0ea2d4 100644
--- a/testcases/kernel/syscalls/setsockopt/setsockopt07.c
+++ b/testcases/kernel/syscalls/setsockopt/setsockopt07.c
@@ -136,6 +136,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
+	.max_iteration_runtime = 150,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
 		"CONFIG_NET_NS=y",
diff --git a/testcases/kernel/syscalls/timerfd/timerfd_settime02.c b/testcases/kernel/syscalls/timerfd/timerfd_settime02.c
index bd92ee964..327a1b8e5 100644
--- a/testcases/kernel/syscalls/timerfd/timerfd_settime02.c
+++ b/testcases/kernel/syscalls/timerfd/timerfd_settime02.c
@@ -112,6 +112,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.min_kver = "2.6.25",
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
+	.max_iteration_runtime = 150,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "1e38da300e1e"},
 		{"CVE", "2017-10661"},
diff --git a/testcases/kernel/syscalls/writev/writev03.c b/testcases/kernel/syscalls/writev/writev03.c
index d0c64743b..467e2de6c 100644
--- a/testcases/kernel/syscalls/writev/writev03.c
+++ b/testcases/kernel/syscalls/writev/writev03.c
@@ -49,7 +49,6 @@ static void setup(void)
 	mapfd = SAFE_OPEN(MAPFILE, O_CREAT|O_RDWR|O_TRUNC, 0644);
 	SAFE_WRITE(1, mapfd, buf, BUF_SIZE);
 
-	fzsync_pair.exec_time_p = 0.25;
 	tst_fzsync_pair_init(&fzsync_pair);
 }
 
@@ -146,6 +145,7 @@ static struct tst_test test = {
 	.min_cpus = 2,
 	.setup = setup,
 	.cleanup = cleanup,
+	.max_iteration_runtime = 75,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d4690f1e1cda"},
 		{}
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 24/30] ltp-aiodio/dio_sparse, aiodio_sparse: Convert to runtime.

[Cyril Hrubis]

Change the code so that the test runtime is capped at the maximal
runtime allowed for the test.

For this to work we also have to:

- check the run_child in the io_read() in the inner loop otherwise the
  test will timeout while waiting for the children to finish

- run the AIO writer in a separate process and kill it when we are out
  of runtime in the aiodio_sparse

CC: Andrea Cervesato <andrea.cervesato@suse.de>
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 .../kernel/io/ltp-aiodio/aiodio_sparse.c      | 26 ++++++++++++++++---
 testcases/kernel/io/ltp-aiodio/common.h       |  6 ++++-
 testcases/kernel/io/ltp-aiodio/dio_sparse.c   |  6 ++++-
 3 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/testcases/kernel/io/ltp-aiodio/aiodio_sparse.c b/testcases/kernel/io/ltp-aiodio/aiodio_sparse.c
index 2aa5662bb..d3c27646c 100644
--- a/testcases/kernel/io/ltp-aiodio/aiodio_sparse.c
+++ b/testcases/kernel/io/ltp-aiodio/aiodio_sparse.c
@@ -189,7 +189,7 @@ static void run(void)
 {
 	char *filename = "file.bin";
 	int status;
-	int i;
+	int i, pid;
 
 	*run_child = 1;
 
@@ -200,9 +200,27 @@ static void run(void)
 		}
 	}
 
-	tst_res(TINFO, "Parent create a sparse file");
+	pid = SAFE_FORK();
+	if (!pid) {
+		aiodio_sparse(filename, alignment, writesize, filesize, numaio);
+		return;
+	}
+
+	tst_res(TINFO, "Child %i creates a sparse file", pid);
+
+	for (;;) {
+		if (SAFE_WAITPID(pid, NULL, WNOHANG))
+			break;
 
-	aiodio_sparse(filename, alignment, writesize, filesize, numaio);
+		sleep(1);
+
+		if (!tst_remaining_runtime()) {
+			tst_res(TINFO, "Test out of runtime, exitting");
+			kill(pid, SIGKILL);
+			SAFE_WAITPID(pid, NULL, 0);
+			break;
+		}
+	}
 
 	if (SAFE_WAITPID(-1, &status, WNOHANG))
 		tst_res(TFAIL, "Non zero bytes read");
@@ -229,7 +247,7 @@ static struct tst_test test = {
 		"tmpfs",
 		NULL
 	},
-	.timeout = 1800,
+	.max_iteration_runtime = 1800,
 };
 #else
 TST_TEST_TCONF("test requires libaio and its development packages");
diff --git a/testcases/kernel/io/ltp-aiodio/common.h b/testcases/kernel/io/ltp-aiodio/common.h
index c27c3a0ad..0a242b91e 100644
--- a/testcases/kernel/io/ltp-aiodio/common.h
+++ b/testcases/kernel/io/ltp-aiodio/common.h
@@ -63,7 +63,7 @@ static inline void io_read(const char *filename, int filesize, volatile int *run
 
 	tst_res(TINFO, "child %i reading file", getpid());
 
-	while (*run_child) {
+	for (;;) {
 		off_t offset = 0;
 		char *bufoff;
 
@@ -80,9 +80,13 @@ static inline void io_read(const char *filename, int filesize, volatile int *run
 				}
 				offset += r;
 			}
+
+			if (!*run_child)
+				goto exit;
 		}
 	}
 
+exit:
 	SAFE_CLOSE(fd);
 }
 
diff --git a/testcases/kernel/io/ltp-aiodio/dio_sparse.c b/testcases/kernel/io/ltp-aiodio/dio_sparse.c
index 0039daa8d..dac034158 100644
--- a/testcases/kernel/io/ltp-aiodio/dio_sparse.c
+++ b/testcases/kernel/io/ltp-aiodio/dio_sparse.c
@@ -51,6 +51,10 @@ static void dio_sparse(int fd, int align, long long fs, int ws, long long off)
 	SAFE_LSEEK(fd, off, SEEK_SET);
 
 	for (i = off; i < fs;) {
+		if (!tst_remaining_runtime()) {
+			tst_res(TINFO, "Test runtime is over, exitting");
+			return;
+		}
 		w = SAFE_WRITE(0, fd, bufptr, ws);
 		i += w;
 	}
@@ -135,5 +139,5 @@ static struct tst_test test = {
 		"tmpfs",
 		NULL
 	},
-	.timeout = 1800,
+	.max_iteration_runtime = 1800,
 };
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 25/30] ltp-aiodio/read_checkzero: Remove

[Cyril Hrubis]

Remove leftovers that are not used anymore.

CC: Andrea Cervesato <andrea.cervesato@suse.de>
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 runtest/ltp-aiodio.part4                      |  2 -
 testcases/kernel/io/ltp-aiodio/.gitignore     |  1 -
 .../kernel/io/ltp-aiodio/read_checkzero.c     | 84 -------------------
 3 files changed, 87 deletions(-)
 delete mode 100644 testcases/kernel/io/ltp-aiodio/read_checkzero.c

diff --git a/runtest/ltp-aiodio.part4 b/runtest/ltp-aiodio.part4
index 54019d47b..d88c27a83 100644
--- a/runtest/ltp-aiodio.part4
+++ b/runtest/ltp-aiodio.part4
@@ -59,8 +59,6 @@ ADI009 dio_append
 DIT000 dio_truncate
 DIT001 dio_truncate
 DIT002 dio_truncate
-#Running read_checkzero
-#gread_checkzero
 #Running dio_read
 DOR000 dio_read -n 1 -i 100 -r 512k -w 512k -s 32M
 DOR001 dio_read -n 10 -i 30 -r 512k -w 512k -s 32M
diff --git a/testcases/kernel/io/ltp-aiodio/.gitignore b/testcases/kernel/io/ltp-aiodio/.gitignore
index f5f20d57e..09a49bfbf 100644
--- a/testcases/kernel/io/ltp-aiodio/.gitignore
+++ b/testcases/kernel/io/ltp-aiodio/.gitignore
@@ -7,4 +7,3 @@
 /dio_truncate
 /dio_read
 /dirty
-/read_checkzero
diff --git a/testcases/kernel/io/ltp-aiodio/read_checkzero.c b/testcases/kernel/io/ltp-aiodio/read_checkzero.c
deleted file mode 100644
index b48197a7b..000000000
--- a/testcases/kernel/io/ltp-aiodio/read_checkzero.c
+++ /dev/null
@@ -1,84 +0,0 @@
-/*
- * Copyright (c) 2004 Daniel McNeil <daniel@osdl.org>
- *               2004 Open Source Development Lab
- *   This program is free software;  you can redistribute it and/or modify
- *   it under the terms of the GNU General Public License as published by
- *   the Free Software Foundation; either version 2 of the License, or
- *   (at your option) any later version.
- *
- *   This program is distributed in the hope that it will be useful,
- *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
- *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
- *   the GNU General Public License for more details.
- *
- *   You should have received a copy of the GNU General Public License
- *   along with this program;  if not, write to the Free Software
- *   Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
- *
- * Module: .c
- */
-
-/*
- * Change History:
- *
- * 2/2004  Marty Ridgeway (mridge@us.ibm.com) Changes to adapt to LTP
- *
- */
-#define _GNU_SOURCE
-
-#include <stdlib.h>
-#include <sys/types.h>
-#include <signal.h>
-#include <fcntl.h>
-#include <stdio.h>
-#include <unistd.h>
-
-#include "common_checkzero.h"
-
-int read_eof(char *filename)
-{
-	int fd;
-	int i;
-	int r;
-	char buf[4096];
-
-	if ((fd = open(filename, O_RDWR)) < 0) {
-		fprintf(stderr, "can't open file %s \n", filename);
-		exit(1);
-	}
-
-	for (i = 0; i < 100000; i++) {
-		off_t offset;
-		char *bufoff;
-
-		offset = lseek(fd, 4096, SEEK_END);
-		r = write(fd, "A", 1);
-
-		offset = lseek(fd, offset - 4096, SEEK_SET);
-
-		r = read(fd, buf, 4096);
-		if (r > 0) {
-			if ((bufoff = check_zero(buf, r))) {
-				fprintf(stderr, "non-zero read at offset %p\n",
-					offset + bufoff);
-				exit(1);
-			}
-		}
-	}
-	fprintf(stderr, "read_checkzero done\n");
-	return 0;
-}
-
-int main(int argc, char **argv)
-{
-	if (argc < 2) {
-		printf("You must pass a filename to the test \n");
-		exit(1);
-	}
-
-	char *filename = argv[1];
-
-	read_eof(filename);
-
-	return 0;
-}
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 26/30] ltp-aiodio/dio_{truncate, append}: Convert to runtime

[Cyril Hrubis]

Change the code so that the test runtime is capped at the maximal
runtime allowed for the test, which is set to 30 minutes by default.

The io_append() in common.h has to check for runtime to make this work.

CC: Andrea Cervesato <andrea.cervesato@suse.de>
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/io/ltp-aiodio/common.h       | 6 +++++-
 testcases/kernel/io/ltp-aiodio/dio_append.c   | 4 ++++
 testcases/kernel/io/ltp-aiodio/dio_truncate.c | 6 ++++++
 3 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/testcases/kernel/io/ltp-aiodio/common.h b/testcases/kernel/io/ltp-aiodio/common.h
index 0a242b91e..d9cbd8611 100644
--- a/testcases/kernel/io/ltp-aiodio/common.h
+++ b/testcases/kernel/io/ltp-aiodio/common.h
@@ -44,9 +44,13 @@ static inline void io_append(const char *path, char pattern, int flags, size_t b
 
 	fd = SAFE_OPEN(path, flags, 0666);
 
-	for (i = 0; i < bcount; i++)
+	for (i = 0; i < bcount; i++) {
 		SAFE_WRITE(1, fd, bufptr, bs);
 
+		if (!tst_remaining_runtime())
+			break;
+	}
+
 	free(bufptr);
 	SAFE_CLOSE(fd);
 }
diff --git a/testcases/kernel/io/ltp-aiodio/dio_append.c b/testcases/kernel/io/ltp-aiodio/dio_append.c
index c099793f6..3a950f3c6 100644
--- a/testcases/kernel/io/ltp-aiodio/dio_append.c
+++ b/testcases/kernel/io/ltp-aiodio/dio_append.c
@@ -71,6 +71,9 @@ static void run(void)
 
 	io_append(filename, 0, O_DIRECT | O_WRONLY | O_CREAT, writesize, appends);
 
+	if (!tst_remaining_runtime())
+		tst_res(TINFO, "Test out of runtime, exitting");
+
 	if (SAFE_WAITPID(-1, &status, WNOHANG))
 		tst_res(TFAIL, "Non zero bytes read");
 	else
@@ -87,6 +90,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_tmpdir = 1,
 	.forks_child = 1,
+	.max_iteration_runtime = 1800,
 	.options = (struct tst_option[]) {
 		{"n:", &str_numchildren, "Number of processes (default 16)"},
 		{"w:", &str_writesize, "Write size for each append (default 64K)"},
diff --git a/testcases/kernel/io/ltp-aiodio/dio_truncate.c b/testcases/kernel/io/ltp-aiodio/dio_truncate.c
index 1fbf83de0..b04302a7b 100644
--- a/testcases/kernel/io/ltp-aiodio/dio_truncate.c
+++ b/testcases/kernel/io/ltp-aiodio/dio_truncate.c
@@ -142,6 +142,11 @@ static void run(void)
 			fail = 1;
 			break;
 		}
+
+		if (!tst_remaining_runtime()) {
+			tst_res(TINFO, "Test out of runtime, exitting");
+			break;
+		}
 	}
 
 	if (fail)
@@ -158,6 +163,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_tmpdir = 1,
 	.forks_child = 1,
+	.max_iteration_runtime = 1800,
 	.options = (struct tst_option[]) {
 		{"n:", &str_numchildren, "Number of threads (default 16)"},
 		{"s:", &str_filesize, "Size of file (default 64K)"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 27/30] ltp-aiodio/dio_read: Convert to runtime

[Cyril Hrubis]

And cap the runtime on 30 minutes by default.

CC: Andrea Cervesato <andrea.cervesato@suse.de>
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/io/ltp-aiodio/dio_read.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/testcases/kernel/io/ltp-aiodio/dio_read.c b/testcases/kernel/io/ltp-aiodio/dio_read.c
index f6ed59782..8a16307a7 100644
--- a/testcases/kernel/io/ltp-aiodio/dio_read.c
+++ b/testcases/kernel/io/ltp-aiodio/dio_read.c
@@ -51,6 +51,9 @@ static void do_buffered_writes(int fd, char *bufptr, long long fsize, long long
 			tst_brk(TBROK, "pwrite: wrote %lld bytes out of %lld", w, wsize);
 
 		SAFE_FSYNC(fd);
+
+		if (!tst_remaining_runtime())
+			return;
 	}
 }
 
@@ -75,6 +78,11 @@ static int do_direct_reads(char *filename, char *bufptr, long long fsize, long l
 				goto exit;
 			}
 
+			if (!tst_remaining_runtime()) {
+				tst_res(TINFO, "Test out of runtime, exitting");
+				goto exit;
+			}
+
 			w = pread(fd, bufptr, rsize, offset);
 			if (w < 0)
 				tst_brk(TBROK, "pread: %s", tst_strerrno(-w));
@@ -170,6 +178,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_tmpdir = 1,
 	.forks_child = 1,
+	.max_iteration_runtime = 1800,
 	.options = (struct tst_option[]) {
 		{"n:", &str_numchildren, "Number of threads (default 8)"},
 		{"w:", &str_writesize, "Size of writing blocks (default 32M)"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 28/30] ltp-aiodio/aiodio_append: Convert to runtime

[Cyril Hrubis]

Cap the test runtime on 30 minutes by default.

For this to work we have to run the process that issues the aio requests
in a child process and kill it when runtime is over.

CC: Andrea Cervesato <andrea.cervesato@suse.de>
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 .../kernel/io/ltp-aiodio/aiodio_append.c      | 25 ++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)

diff --git a/testcases/kernel/io/ltp-aiodio/aiodio_append.c b/testcases/kernel/io/ltp-aiodio/aiodio_append.c
index 46cc74ee4..4bb95189c 100644
--- a/testcases/kernel/io/ltp-aiodio/aiodio_append.c
+++ b/testcases/kernel/io/ltp-aiodio/aiodio_append.c
@@ -141,7 +141,7 @@ static void run(void)
 {
 	char *filename = "aiodio_append";
 	int status;
-	int i;
+	int i, pid;
 
 	*run_child = 1;
 
@@ -152,9 +152,27 @@ static void run(void)
 		}
 	}
 
-	tst_res(TINFO, "Parent append to file");
+	pid = SAFE_FORK();
+	if (!pid) {
+		aiodio_append(filename, appends, alignment, writesize, numaio);
+		return;
+	}
+
+	tst_res(TINFO, "Child %i appends to a file", pid);
+
+	for (;;) {
+		if (SAFE_WAITPID(pid, NULL, WNOHANG))
+			break;
 
-	aiodio_append(filename, appends, alignment, writesize, numaio);
+		sleep(1);
+
+		if (!tst_remaining_runtime()) {
+			tst_res(TINFO, "Test out of runtime, exitting");
+			kill(pid, SIGKILL);
+			SAFE_WAITPID(pid, NULL, 0);
+			break;
+		}
+	}
 
 	if (SAFE_WAITPID(-1, &status, WNOHANG))
 		tst_res(TFAIL, "Non zero bytes read");
@@ -172,6 +190,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_tmpdir = 1,
 	.forks_child = 1,
+	.max_iteration_runtime = 1800,
 	.options = (struct tst_option[]) {
 		{"n:", &str_numchildren, "Number of threads (default 16)"},
 		{"s:", &str_writesize, "Size of the file to write (default 64K)"},
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 29/30] timer_test: Convert to runtime

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 lib/tst_timer_test.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/lib/tst_timer_test.c b/lib/tst_timer_test.c
index 32fa55c7c..3ff9f1d65 100644
--- a/lib/tst_timer_test.c
+++ b/lib/tst_timer_test.c
@@ -420,6 +420,9 @@ static struct tst_option options[] = {
 
 static void parse_timer_opts(void)
 {
+	size_t i;
+	long long runtime_us = 0;
+
 	if (str_sleep_time) {
 		if (tst_parse_int(str_sleep_time, &sleep_time, 0, INT_MAX)) {
 			tst_brk(TBROK,
@@ -441,14 +444,17 @@ static void parse_timer_opts(void)
 		if (!sample_cnt)
 			sample_cnt = 500;
 
-		long long timeout = sleep_time * sample_cnt / 1000000;
-
-		tst_set_timeout(timeout + timeout/10);
+		runtime_us = sleep_time * sample_cnt;
 
 		test->test_all = single_timer_test;
 		test->test = NULL;
 		test->tcnt = 0;
+	} else {
+		for (i = 0; i < ARRAY_SIZE(tcases); i++)
+			runtime_us += tcases[i].usec * tcases[i].samples;
 	}
+
+	tst_set_runtime((runtime_us + runtime_us/10)/1000000);
 }
 
 struct tst_test *tst_timer_test_setup(struct tst_test *timer_test)
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 30/30] tst_test: Remove timeout stubs

[Cyril Hrubis]

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 include/tst_test.h |  4 ----
 lib/tst_test.c     | 11 -----------
 2 files changed, 15 deletions(-)

diff --git a/include/tst_test.h b/include/tst_test.h
index c084ce4bc..5a84195c7 100644
--- a/include/tst_test.h
+++ b/include/tst_test.h
@@ -236,8 +236,6 @@ struct tst_test {
 	unsigned int mnt_flags;
 	void *mnt_data;
 
-	/* override default timeout per test run, disabled == -1 */
-	int timeout;
 	/*
 	 * Maximal test runtime in seconds.
 	 *
@@ -333,9 +331,7 @@ const char *tst_strsig(int sig);
  */
 const char *tst_strstatus(int status);
 
-unsigned int tst_timeout_remaining(void);
 unsigned int tst_multiply_timeout(unsigned int timeout);
-void tst_set_timeout(int timeout);
 
 /*
  * Returns remaining test runtime. Test that runs for more than a few seconds
diff --git a/lib/tst_test.c b/lib/tst_test.c
index 096acef96..2ec81ab0e 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -1471,12 +1471,6 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED)
 	}
 }
 
-unsigned int tst_timeout_remaining(void)
-{
-	tst_brk(TBROK, "Stub called!");
-	return 0;
-}
-
 unsigned int tst_remaining_runtime(void)
 {
 	static struct timespec now;
@@ -1509,11 +1503,6 @@ unsigned int tst_multiply_timeout(unsigned int timeout)
 	return timeout * timeout_mul;
 }
 
-void tst_set_timeout(int timeout)
-{
-	tst_brk(TBROK, "Stub called!");
-}
-
 static void set_timeout(void)
 {
 	unsigned int timeout = DEFAULT_TIMEOUT;
-- 
2.35.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 01/30] Introduce a concept of max runtime

[Petr Vorel]

Hi Cyril,

...
>  doc/user-guide.txt                            |   4 +
>  include/tst_test.h                            |  27 ++++
>  lib/newlib_tests/.gitignore                   |   5 +-
>  lib/newlib_tests/test10.c                     |  22 ---
>  lib/newlib_tests/test12.c                     |  21 ---
FYI test12 is run in CI, need to be removed from runtest.sh.

Kind regards,
Petr

+++ lib/newlib_tests/runtest.sh
@@ -1,7 +1,7 @@
 #!/bin/sh
 # Copyright (c) 2021 Petr Vorel <pvorel@suse.cz>
 
-LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18
+LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test15 test18
 tst_needs_cmds01 tst_needs_cmds02 tst_needs_cmds03 tst_needs_cmds06
 tst_needs_cmds07 tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus
 tst_fuzzy_sync03 test_zero_hugepage.sh test_kconfig.sh

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 01/30] Introduce a concept of max runtime

[Petr Vorel]

Hi Cyril,

> ...
> >  doc/user-guide.txt                            |   4 +
> >  include/tst_test.h                            |  27 ++++
> >  lib/newlib_tests/.gitignore                   |   5 +-
> >  lib/newlib_tests/test10.c                     |  22 ---
> >  lib/newlib_tests/test12.c                     |  21 ---
> FYI test12 is run in CI, need to be removed from runtest.sh.

>  .../{test18.c => test_runtime01.c}            |  10 +-
And obviously rename test18 to test_runtime01.
>  lib/newlib_tests/test_runtime02.c             |  31 ++++
And add test_runtime02, which TPASS.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 23/30] fuzzy_sync: Convert to runtime

[Petr Vorel]

Hi Cyril,

> The runtime for fuzzy sync tests are computed as timeout * exec_time_p.
> The default timeout was 300 and the default was 0.5.

> CC: Richard Palethorpe <rpalethorpe@suse.com>
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>  include/tst_fuzzy_sync.h                      | 19 ++++---------------
>  testcases/kernel/crypto/af_alg07.c            |  1 +
>  testcases/kernel/mem/thp/thp04.c              |  1 +
>  testcases/kernel/pty/pty03.c                  |  2 +-
>  testcases/kernel/pty/pty05.c                  |  1 +
>  testcases/kernel/pty/pty06.c                  |  1 +
>  testcases/kernel/pty/pty07.c                  |  1 +
>  testcases/kernel/sound/snd_seq01.c            |  2 +-
>  testcases/kernel/sound/snd_timer01.c          |  1 +
>  testcases/kernel/syscalls/bind/bind06.c       |  2 +-
>  testcases/kernel/syscalls/inotify/inotify09.c |  1 +
>  .../kernel/syscalls/ipc/shmctl/shmctl05.c     |  2 +-
>  testcases/kernel/syscalls/sendmsg/sendmsg03.c |  1 +
>  .../kernel/syscalls/setsockopt/setsockopt06.c |  2 +-
>  .../kernel/syscalls/setsockopt/setsockopt07.c |  1 +
>  .../syscalls/timerfd/timerfd_settime02.c      |  1 +
>  testcases/kernel/syscalls/writev/writev03.c   |  2 +-
>  17 files changed, 20 insertions(+), 21 deletions(-)

FYI you haven't converted fuzzy sync tests.
I suppose these errors must be fix in first commit.

./tst_fuzzy_sync01
tst_test.c:1522: TINFO: Timeout per run is 0h 00m 30s
tst_test.c:1483: TBROK: Runtime not set!

./tst_fuzzy_sync02
tst_test.c:1522: TINFO: Timeout per run is 0h 00m 30s
tst_test.c:1483: TBROK: Runtime not set!

./tst_fuzzy_sync03
tst_test.c:1522: TINFO: Timeout per run is 0h 00m 30s
tst_test.c:1483: TBROK: Runtime not set!
tst_fuzzy_sync03.c:45: TFAIL: Worker performed wrong number of iterations: 0 != 65535


Only tst_fuzzy_sync03.c is run in CI:
https://github.com/pevik/ltp/runs/6280658557?check_suite_focus=true
/__w/ltp/ltp/lib/tst_test.c:1483: TBROK: Runtime not set!
/__w/ltp/ltp/lib/newlib_tests/tst_fuzzy_sync03.c:45: TFAIL: Worker performed wrong number of iterations: 0 != 65535

maybe after fix the other two might be possible to run in CI as well.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 01/30] Introduce a concept of max runtime

[Richard Palethorpe]

Hello Cyril,

"Cyril Hrubis" <chrubis@suse.cz> writes:

> This commit introduce a concept of per iteration max test runtime. In
> other words test runtime is capped at a certain value in order to make
> testruns more deterministic. Test is free to to finish before the
> runtime is used up, for example when maximal number of iterations
> was reached, but test must stop once the runtime has been used up.
>
> Testcases that run for more than a second or two must check for
> remaining runtime by regular calls to tst_remaining_runtime() and should
> exit when zero is returned.
>
> The test max runtime must be set either by the .max_iteration_runtime in
> the tst_test structure or in the test setup by a call to
> tst_set_runtime().
>
> The test timeout is then computed as a sum of DEFAULT_TIMEOUT (currently
> set to 30 seconds) and the test runtime. The DEFAULT_TIMEOUT is nothing
> more than a safety margin for teardown of the test.
>
> This commit also maps the -I parameter to the test max runtime if
> available and introduces LTP_RUNTIME_MUL enviroment variable so that we
> have an easy controll over the runtime cap.
>
> Lastly but not least the function related to the timeout are turned into
> no-op by this commit and removed after all test are converted to the
> runtime API.
>
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>  doc/user-guide.txt                            |   4 +
>  include/tst_test.h                            |  27 ++++
>  lib/newlib_tests/.gitignore                   |   5 +-
>  lib/newlib_tests/test10.c                     |  22 ---
>  lib/newlib_tests/test12.c                     |  21 ---
>  lib/newlib_tests/test13.c                     |   1 -
>  lib/newlib_tests/test_children_cleanup.c      |   1 -
>  .../{test18.c => test_runtime01.c}            |  10 +-
>  lib/newlib_tests/test_runtime02.c             |  31 ++++
>  lib/tst_test.c                                | 151 ++++++++++++++----
>  10 files changed, 187 insertions(+), 86 deletions(-)
>  delete mode 100644 lib/newlib_tests/test10.c
>  delete mode 100644 lib/newlib_tests/test12.c
>  rename lib/newlib_tests/{test18.c => test_runtime01.c} (58%)
>  create mode 100644 lib/newlib_tests/test_runtime02.c
>
> diff --git a/doc/user-guide.txt b/doc/user-guide.txt
> index f41cbc733..d50d2e0cb 100644
> --- a/doc/user-guide.txt
> +++ b/doc/user-guide.txt
> @@ -25,6 +25,10 @@ For running LTP network tests see `testcases/network/README.md`.
>  | 'LTP_TIMEOUT_MUL'     | Multiply timeout, must be number >= 1 (> 1 is useful for
>                            slow machines to avoid unexpected timeout).
>                            Variable is also used in shell tests, but ceiled to int.
> +| 'LTP_RUNTIME_MUL'     | Multiplies maximal test iteration runtime. Tests
> +                          that run for more than a second or two are capped on
> +			  runtime. You can scale the default runtime both up
> +			  and down with this multiplier.
>  | 'LTP_VIRT_OVERRIDE'   | Overrides virtual machine detection in the test
>                            library. Setting it to empty string tell the library
>                            that system is not a virtual machine. Other possible
> diff --git a/include/tst_test.h b/include/tst_test.h
> index dbe303bc8..c084ce4bc 100644
> --- a/include/tst_test.h
> +++ b/include/tst_test.h
> @@ -134,6 +134,8 @@ extern unsigned int tst_variant;
>  
>  #define TST_NO_HUGEPAGES ((unsigned long)-1)
>  
> +#define TST_UNLIMITED_RUNTIME (-1)
> +
>  struct tst_test {
>  	/* number of tests available in test() function */
>  	unsigned int tcnt;
> @@ -236,6 +238,18 @@ struct tst_test {
>  
>  	/* override default timeout per test run, disabled == -1 */
>  	int timeout;
> +	/*
> +	 * Maximal test runtime in seconds.
> +	 *
> +	 * Any test that runs for more than a second or two should set this and
> +	 * also use tst_remaining_runtime() to exit when runtime was used up.
> +	 * Tests may finish sooner, for example if requested number of
> +	 * iterations was reached before the runtime runs out.
> +	 *
> +	 * If test runtime cannot be know in advance it should be set to
                                     ^^known
> +	 * TST_UNLIMITED_RUNTIME.
> +	 */
> +	int max_iteration_runtime;

It's not immediately clear if iteration refers to the inner test loop
(e.g. Fuzzy Sync) or the outer loop performed by adding '-i N'. Perhaps
it would be better to call it max_runtime and document that it is scaled
by '-i N'?

>  
>  	void (*setup)(void);
>  	void (*cleanup)(void);
> @@ -323,6 +337,19 @@ unsigned int tst_timeout_remaining(void);
>  unsigned int tst_multiply_timeout(unsigned int timeout);
>  void tst_set_timeout(int timeout);
>  
> +/*
> + * Returns remaining test runtime. Test that runs for more than a few seconds
> + * should check if they should exit by calling this function regularly.
> + *
> + * The function returns remaining runtime in seconds. If runtime was used up
> + * zero is returned.
> + */
> +unsigned int tst_remaining_runtime(void);
> +
> +/*
> + * Sets runtime per iteration in seconds.
> + */
> +void tst_set_runtime(int runtime_per_iteration);
>  
>  /*
>   * Returns path to the test temporary directory in a newly allocated buffer.
> diff --git a/lib/newlib_tests/.gitignore b/lib/newlib_tests/.gitignore
> index f4414f6a1..59b57d063 100644
> --- a/lib/newlib_tests/.gitignore
> +++ b/lib/newlib_tests/.gitignore
> @@ -7,9 +7,7 @@ test06
>  test07
>  test08
>  test09
> -test10
>  test11
> -test12
>  test13
>  test14
>  test15
> @@ -22,7 +20,6 @@ tst_safe_fileops
>  tst_res_hexd
>  tst_strstatus
>  tst_print_result
> -test18
>  test19
>  test20
>  test22
> @@ -56,3 +53,5 @@ tst_needs_cmds05
>  tst_needs_cmds06
>  tst_needs_cmds07
>  tst_needs_cmds08
> +test_runtime01
> +test_runtime02
> diff --git a/lib/newlib_tests/test10.c b/lib/newlib_tests/test10.c
> deleted file mode 100644
> index df61908ac..000000000
> --- a/lib/newlib_tests/test10.c
> +++ /dev/null
> @@ -1,22 +0,0 @@
> -// SPDX-License-Identifier: GPL-2.0-or-later
> -/*
> - * Copyright (c) 2016 Linux Test Project
> - */
> -
> -/*
> - * Test for watchdog timeout.
> - */
> -
> -#include "tst_test.h"
> -
> -
> -static void do_test(void)
> -{
> -	sleep(2);
> -	tst_res(TPASS, "Not reached");
> -}
> -
> -static struct tst_test test = {
> -	.test_all = do_test,
> -	.timeout = 1,
> -};
> diff --git a/lib/newlib_tests/test12.c b/lib/newlib_tests/test12.c
> deleted file mode 100644
> index b4f0d6303..000000000
> --- a/lib/newlib_tests/test12.c
> +++ /dev/null
> @@ -1,21 +0,0 @@
> -// SPDX-License-Identifier: GPL-2.0-or-later
> -/*
> - * Copyright (c) 2016 Linux Test Project
> - */
> -
> -/*
> - * Test for timeout override.
> - */
> -
> -#include "tst_test.h"
> -
> -static void do_test(void)
> -{
> -	sleep(1);
> -	tst_res(TPASS, "Passed!");
> -}
> -
> -static struct tst_test test = {
> -	.timeout = 2,
> -	.test_all = do_test,
> -};
> diff --git a/lib/newlib_tests/test13.c b/lib/newlib_tests/test13.c
> index c447dc3dc..83c48f734 100644
> --- a/lib/newlib_tests/test13.c
> +++ b/lib/newlib_tests/test13.c
> @@ -20,7 +20,6 @@ static void do_test(void)
>  }
>  
>  static struct tst_test test = {
> -	.timeout = 1,
>  	.forks_child = 1,
>  	.test_all = do_test,
>  };
> diff --git a/lib/newlib_tests/test_children_cleanup.c b/lib/newlib_tests/test_children_cleanup.c
> index 2b1ca5f9c..4a1313f6d 100644
> --- a/lib/newlib_tests/test_children_cleanup.c
> +++ b/lib/newlib_tests/test_children_cleanup.c
> @@ -39,5 +39,4 @@ static void run(void)
>  static struct tst_test test = {
>  	.test_all = run,
>  	.forks_child = 1,
> -	.timeout = 10,
>  };
> diff --git a/lib/newlib_tests/test18.c b/lib/newlib_tests/test_runtime01.c
> similarity index 58%
> rename from lib/newlib_tests/test18.c
> rename to lib/newlib_tests/test_runtime01.c
> index 026435d7d..79e4c7eac 100644
> --- a/lib/newlib_tests/test18.c
> +++ b/lib/newlib_tests/test_runtime01.c
> @@ -9,14 +9,18 @@
>  
>  static void run(void)
>  {
> +	int runtime;
> +
>  	do {
> +		runtime = tst_remaining_runtime();
> +		tst_res(TINFO, "Remaining runtime %d", runtime);
>  		sleep(1);
> -	} while (tst_timeout_remaining() >= 4);
> +	} while (runtime);
>  
> -	tst_res(TPASS, "Timeout remaining: %d", tst_timeout_remaining());
> +	tst_res(TPASS, "Finished loop!");
>  }
>  
>  static struct tst_test test = {
>  	.test_all = run,
> -	.timeout = 5
> +	.max_iteration_runtime = 5
>  };
> diff --git a/lib/newlib_tests/test_runtime02.c b/lib/newlib_tests/test_runtime02.c
> new file mode 100644
> index 000000000..1329743f4
> --- /dev/null
> +++ b/lib/newlib_tests/test_runtime02.c
> @@ -0,0 +1,31 @@
> +// SPDX-License-Identifier: GPL-2.0-or-later
> +/*
> + * Copyright (c) 2021, Linux Test Project
> + */
> +/*
> + * This test is set up so that the timeout is not long enough to guarantee
> + * enough runtime for two iterations, i.e. the timeout without offset and after
> + * scaling is too small and the tests ends up with TBROK.
> + *
> + * You can fix this by exporting LTP_MAX_TEST_RUNTIME=10 before executing the
> + * test, in that case the runtime would be divided between iterations and timeout
> + * adjusted so that it provides enough safeguards for the test to finish.
> + */
> +
> +#include <stdlib.h>
> +#include <unistd.h>
> +#include "tst_test.h"
> +
> +static void run(void)
> +{
> +	while (tst_remaining_runtime())
> +		sleep(1);
> +
> +	tst_res(TPASS, "Timeout remaining: %d", tst_remaining_runtime());
> +}
> +
> +static struct tst_test test = {
> +	.test_all = run,
> +	.max_iteration_runtime = 5,
> +	.test_variants = 2
> +};
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 8e258594a..096acef96 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -45,6 +45,8 @@ const char *TCID __attribute__((weak));
>  #define GLIBC_GIT_URL "https://sourceware.org/git/?p=glibc.git;a=commit;h="
>  #define CVE_DB_URL "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-"
>  
> +#define DEFAULT_TIMEOUT 30
> +
>  struct tst_test *tst_test;
>  
>  static const char *tid;
> @@ -63,6 +65,7 @@ struct results {
>  	int warnings;
>  	int broken;
>  	unsigned int timeout;
> +	int max_iteration_runtime;
>  };
>  
>  static struct results *results;
> @@ -464,6 +467,40 @@ pid_t safe_clone(const char *file, const int lineno,
>  	return pid;
>  }
>  
> +static void parse_mul(float *mul, const char *env_name, float min, float max)
> +{
> +	char *str_mul;
> +	int ret;
> +
> +	if (*mul > 0)
> +		return;
> +
> +	str_mul = getenv(env_name);
> +
> +	if (!str_mul) {
> +		*mul = 1;
> +		return;
> +	}
> +
> +	ret = tst_parse_float(str_mul, mul, min, max);
> +	if (ret) {
> +		tst_brk(TBROK, "Failed to parse %s: %s",
> +			env_name, tst_strerrno(ret));
> +	}
> +}
> +
> +static int multiply_runtime(void)
> +{
> +	static float runtime_mul = -1;
> +
> +	if (tst_test->max_iteration_runtime <= 0)
> +		return tst_test->max_iteration_runtime;

nit; IMO it would be easier to understand if it returned
TST_UNLIMITED_RUNTIME.

> +
> +	parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
> +
> +	return tst_test->max_iteration_runtime * runtime_mul;
> +}
> +
>  static struct option {
>  	char *optstr;
>  	char *help;
> @@ -477,6 +514,7 @@ static struct option {
>  static void print_help(void)
>  {
>  	unsigned int i;
> +	int timeout, runtime;
>  
>  	/* see doc/user-guide.txt, which lists also shell API variables */
>  	fprintf(stderr, "Environment Variables\n");
> @@ -489,10 +527,32 @@ static void print_help(void)
>  	fprintf(stderr, "LTP_DEV_FS_TYPE      Filesystem used for testing (default: %s)\n", DEFAULT_FS_TYPE);
>  	fprintf(stderr, "LTP_SINGLE_FS_TYPE   Testing only - specifies filesystem instead all supported (for .all_filesystems)\n");
>  	fprintf(stderr, "LTP_TIMEOUT_MUL      Timeout multiplier (must be a number >=1)\n");
> +	fprintf(stderr, "LTP_RUNTIME_MUL      Runtime multiplier (must be a number >=1)\n");
>  	fprintf(stderr, "LTP_VIRT_OVERRIDE    Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
>  	fprintf(stderr, "TMPDIR               Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
>  	fprintf(stderr, "\n");
>  
> +	fprintf(stderr, "Timeout and runtime\n");
> +	fprintf(stderr, "-------------------\n");
> +
> +	if (tst_test->max_iteration_runtime) {
> +		runtime = multiply_runtime();
> +
> +		if (runtime == TST_UNLIMITED_RUNTIME) {
> +			fprintf(stderr, "Test iteration runtime is not limited\n");
> +		} else {
> +			fprintf(stderr, "Test iteration runtime cap %ih %im %is\n",
> +				runtime/3600, (runtime%3600)/60, runtime % 60);
> +		}
> +	}
> +
> +	timeout = tst_multiply_timeout(DEFAULT_TIMEOUT);
> +
> +	fprintf(stderr, "Test timeout (not including runtime) %ih %im %is\n",
> +		timeout/3600, (timeout%3600)/60, timeout % 60);
> +
> +	fprintf(stderr, "\n");
> +
>  	fprintf(stderr, "Options\n");
>  	fprintf(stderr, "-------\n");
>  
> @@ -620,7 +680,10 @@ static void parse_opts(int argc, char *argv[])
>  			iterations = atoi(optarg);
>  		break;
>  		case 'I':
> -			duration = atof(optarg);
> +			if (tst_test->max_iteration_runtime > 0)
> +				tst_test->max_iteration_runtime =
> atoi(optarg);

Doesn't this change the semantics of -I? Duration does not seem to be
per iteration, but overall execution time.

> +			else
> +				duration = atof(optarg);
>  		break;
>  		case 'C':
>  #ifdef UCLINUX
> @@ -1034,6 +1097,11 @@ static void do_setup(int argc, char *argv[])
>  	if (!tst_test)
>  		tst_brk(TBROK, "No tests to run");
>  
> +	if (tst_test->max_iteration_runtime < -1) {
> +		tst_brk(TBROK, "Invalid runtime value %i",
> +			results->max_iteration_runtime);
> +	}
> +
>  	if (tst_test->tconf_msg)
>  		tst_brk(TCONF, "%s", tst_test->tconf_msg);
>  
> @@ -1404,39 +1472,36 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED)
>  }
>  
>  unsigned int tst_timeout_remaining(void)
> +{
> +	tst_brk(TBROK, "Stub called!");
> +	return 0;
> +}
> +
> +unsigned int tst_remaining_runtime(void)
>  {
>  	static struct timespec now;
> -	unsigned int elapsed;
> +	int elapsed;
> +
> +	if (results->max_iteration_runtime == TST_UNLIMITED_RUNTIME)
> +		return UINT_MAX;
> +
> +	if (results->max_iteration_runtime == 0)
> +		tst_brk(TBROK, "Runtime not set!");
>  
>  	if (tst_clock_gettime(CLOCK_MONOTONIC, &now))
>  		tst_res(TWARN | TERRNO, "tst_clock_gettime() failed");
>  
> -	elapsed = (tst_timespec_diff_ms(now, tst_start_time) + 500) / 1000;
> -	if (results->timeout > elapsed)
> -		return results->timeout - elapsed;
> +	elapsed = tst_timespec_diff_ms(now, tst_start_time) / 1000;
> +	if (results->max_iteration_runtime > elapsed)
> +		return results->max_iteration_runtime - elapsed;
>  
>  	return 0;
>  }
>  
> +
>  unsigned int tst_multiply_timeout(unsigned int timeout)
>  {
> -	char *mul;
> -	int ret;
> -
> -	if (timeout_mul == -1) {
> -		mul = getenv("LTP_TIMEOUT_MUL");
> -		if (mul) {
> -			if ((ret = tst_parse_float(mul, &timeout_mul, 1, 10000))) {
> -				tst_brk(TBROK, "Failed to parse LTP_TIMEOUT_MUL: %s",
> -						tst_strerrno(ret));
> -			}
> -		} else {
> -			timeout_mul = 1;
> -		}
> -	}
> -	if (timeout_mul < 1)
> -		tst_brk(TBROK, "LTP_TIMEOUT_MUL must to be int >= 1! (%.2f)",
> -				timeout_mul);
> +	parse_mul(&timeout_mul, "LTP_TIMEOUT_MUL", 0.099, 10000);
>  
>  	if (timeout < 1)
>  		tst_brk(TBROK, "timeout must to be >= 1! (%d)", timeout);
> @@ -1446,37 +1511,48 @@ unsigned int tst_multiply_timeout(unsigned int timeout)
>  
>  void tst_set_timeout(int timeout)
>  {
> -	if (timeout == -1) {
> +	tst_brk(TBROK, "Stub called!");
> +}
> +
> +static void set_timeout(void)
> +{
> +	unsigned int timeout = DEFAULT_TIMEOUT;
> +
> +	if (results->max_iteration_runtime == TST_UNLIMITED_RUNTIME) {
>  		tst_res(TINFO, "Timeout per run is disabled");
>  		return;
>  	}
>  
> -	if (timeout < 1)
> -		tst_brk(TBROK, "timeout must to be >= 1! (%d)", timeout);
> +	if (results->max_iteration_runtime < 0) {
> +		tst_brk(TBROK, "max_iteration_runtime must to be >= 0! (%d)",

It can be -1

> +			results->max_iteration_runtime);
> +	}
>  
> -	results->timeout = tst_multiply_timeout(timeout);
> +	results->timeout = tst_multiply_timeout(timeout) + results->max_iteration_runtime;
>  
>  	tst_res(TINFO, "Timeout per run is %uh %02um %02us",
>  		results->timeout/3600, (results->timeout%3600)/60,
>  		results->timeout % 60);
> +}
>  
> -	if (getpid() == lib_pid)
> -		alarm(results->timeout);
> -	else
> -		heartbeat();
> +void tst_set_runtime(int max_iteration_runtime)
> +{
> +	results->max_iteration_runtime = max_iteration_runtime;
> +	tst_res(TINFO, "Updating max runtime to %uh %02um %02us",
> +		max_iteration_runtime/3600, (max_iteration_runtime%3600)/60,
> +		max_iteration_runtime % 60);
> +	set_timeout();
> +	heartbeat();
>  }
>  
>  static int fork_testrun(void)
>  {
>  	int status;
>  
> -	if (tst_test->timeout)
> -		tst_set_timeout(tst_test->timeout);
> -	else
> -		tst_set_timeout(300);
> -
>  	SAFE_SIGNAL(SIGINT, sigint_handler);
>  
> +	alarm(results->timeout);
> +
>  	test_pid = fork();
>  	if (test_pid < 0)
>  		tst_brk(TBROK | TERRNO, "fork()");
> @@ -1568,6 +1644,11 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
>  	SAFE_SIGNAL(SIGALRM, alarm_handler);
>  	SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
>  
> +	if (tst_test->max_iteration_runtime)
> +		results->max_iteration_runtime = tst_test->max_iteration_runtime;
> +
> +	set_timeout();
> +
>  	if (tst_test->test_variants)
>  		test_variants = tst_test->test_variants;


-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 00/30] Introduce runtime and conver tests

[Richard Palethorpe]

Hello, Cyril,

"Cyril Hrubis" <chrubis@suse.cz> writes:

> This patchset introduces a concept of runtime, which is described well
> enough in the first patch of the series and converts all tests that were
> previously using timeout API to runtime API.
>
> The main change to the v1 of the patchset is that the runtime is now per
> iteration, e.g. if we run the test for all_filesystem the runtime
> applies to a single filesystem only. And at the same time the overall
> test timeout is a sum of static timeout and runtime. The static part of
> the timeout is meant to cover test teardown and any inaccuracy in the
> test runtime accounting.
>
> The patches that convert tests to runtime API fall into these classes:
>
> - fzsync tests
> - faster timeout tests
> - I/O tests
> - timer precision tests
> - oom tests
> - misc tests that run for some time (controlled by command line
>   parameter)
>
> fzsync tests
> ============
>
> These tests previously defined their runtime based on scale factor that
> used slice of the default timeout (300s) for the test runtime.
>
> Now the runtime is explicitly encoded in the test, which I find much
> more user friendly because it's clear how long will the test run.
>
> faster timeout tests
> ====================
>
> There were several tests that set low timeout becuase the test may lock
> up. In these cases the timeout was simply removed because the default
> value (30s) should be small enough.
>
> I/O tests
> =========
>
> The aiodio tests were modified so that their runtime is capped to
> 30 minutes by default. That means that even on very slow machine they
> finish in a reasonable time.
>
> timer precision tests
> =====================
>
> These test simply set the test timeout to be 110% of the sum of the
> sleep_time * samples. That is to accomodate for the context switches and
> upkeeping between the calls in the timer library.
>
>
> oom tests
> =========
>
> Here we just set the .max_iteration_runtime to TST_RUNTIME_UNLIMITED,
> which has the very same effect as .timeout = -1 previously had.
>
> misc test
> =========
>
> In this case the test code was simplified quite a lot since the runtime
> is now passed in a unified way and parsed in the test library.
>
> TODO:
>
> This patchset is reasonably complete in a sense that it removes the
> timeout API at the end. Still there are a few things to consider:
>
> - is reusing the -I parameter a good idea? Wouldn't adding new parameter
>   (-r) be better?

Perhaps we could deprecate -I, but convert it to use the new mechanism
while trying to keep the behaviour similar.

I'm not sure we need a global -r option, but unlike -i it's not much
effort to support. It would not surprise me if there are existing tests
which don't work with -i.

>
> - there are quite likely tests that run for more than a second or a two
>   and should be made runtime aware
>
> - anything else?
>
> Cyril Hrubis (30):
>   Introduce a concept of max runtime
>   mtest06/mmap1: Convert to runtime
>   mtest06/mmap3: Convert to runtime
>   mtest01/mtest01: Convert to runtime
>   cve/cve-2015-3290: Convert to runtime
>   crypto/af_alg02: Convert to runtime.
>   crypto/pcrypt_aead01: Convert to runtime
>   syscalls/clock_gettime01: Remove useless timeout
>   syscalls/fanotify22: Remove useless timeout
>   syscalls/gettimeofday02: Convert to runtime.
>   syscalls/inotify06: Convert to runtime.
>   syscalls/inotify01: Remove now useless timeout
>   syscalls/perf_event_open03: Convert to runtime
>   syscalls/readv01: Remove now useless timeout
>   syscalls/tgkill03: Remove now unused timeout
>   syscalls/setsockopt09: Remove now useless timeout
>   syscalls/userfaultfd01: Remove now useless timeout
>   syscalls/move_pages12: Convert to runtime
>   syscalls/rt_sigqueueinfo01: Remove now useless timeout
>   mem/mallocstress: Convert to runtime
>   mem/{oom,min_free_kbytes}: Convert to runtime
>   crypto/af_alg02: Convert to runtime
>   fuzzy_sync: Convert to runtime
>   ltp-aiodio/dio_sparse,aiodio_sparse: Convert to runtime.
>   ltp-aiodio/read_checkzero: Remove
>   ltp-aiodio/dio_{truncate,append}: Convert to runtime
>   ltp-aiodio/dio_read: Convert to runtime
>   ltp-aiodio/aiodio_append: Convert to runtime
>   timer_test: Convert to runtime
>   tst_test: Remove timeout stubs
>
>  doc/user-guide.txt                            |   4 +
>  include/tst_fuzzy_sync.h                      |  19 +--
>  include/tst_test.h                            |  31 +++-
>  lib/newlib_tests/.gitignore                   |   5 +-
>  lib/newlib_tests/test10.c                     |  22 ---
>  lib/newlib_tests/test12.c                     |  21 ---
>  lib/newlib_tests/test13.c                     |   1 -
>  lib/newlib_tests/test_children_cleanup.c      |   1 -
>  .../{test18.c => test_runtime01.c}            |  10 +-
>  lib/newlib_tests/test_runtime02.c             |  31 ++++
>  lib/tst_test.c                                | 144 +++++++++++++-----
>  lib/tst_timer_test.c                          |  12 +-
>  runtest/ltp-aiodio.part4                      |   2 -
>  testcases/cve/cve-2015-3290.c                 |   8 +-
>  testcases/kernel/crypto/af_alg02.c            |   4 +-
>  testcases/kernel/crypto/af_alg07.c            |   1 +
>  testcases/kernel/crypto/pcrypt_aead01.c       |   3 +-
>  testcases/kernel/io/ltp-aiodio/.gitignore     |   1 -
>  .../kernel/io/ltp-aiodio/aiodio_append.c      |  25 ++-
>  .../kernel/io/ltp-aiodio/aiodio_sparse.c      |  26 +++-
>  testcases/kernel/io/ltp-aiodio/common.h       |  12 +-
>  testcases/kernel/io/ltp-aiodio/dio_append.c   |   4 +
>  testcases/kernel/io/ltp-aiodio/dio_read.c     |   9 ++
>  testcases/kernel/io/ltp-aiodio/dio_sparse.c   |   6 +-
>  testcases/kernel/io/ltp-aiodio/dio_truncate.c |   6 +
>  .../kernel/io/ltp-aiodio/read_checkzero.c     |  84 ----------
>  testcases/kernel/mem/mtest01/mtest01.c        |  14 +-
>  testcases/kernel/mem/mtest06/mmap1.c          |  24 +--
>  testcases/kernel/mem/mtest06/mmap3.c          |  12 +-
>  testcases/kernel/mem/mtest07/mallocstress.c   |  11 +-
>  testcases/kernel/mem/oom/oom01.c              |   2 +-
>  testcases/kernel/mem/oom/oom02.c              |   2 +-
>  testcases/kernel/mem/oom/oom03.c              |   2 +-
>  testcases/kernel/mem/oom/oom04.c              |   2 +-
>  testcases/kernel/mem/oom/oom05.c              |   2 +-
>  testcases/kernel/mem/thp/thp04.c              |   1 +
>  .../kernel/mem/tunable/min_free_kbytes.c      |   2 +-
>  testcases/kernel/pty/pty03.c                  |   2 +-
>  testcases/kernel/pty/pty05.c                  |   1 +
>  testcases/kernel/pty/pty06.c                  |   1 +
>  testcases/kernel/pty/pty07.c                  |   1 +
>  testcases/kernel/sound/snd_seq01.c            |   2 +-
>  testcases/kernel/sound/snd_timer01.c          |   1 +
>  testcases/kernel/syscalls/bind/bind06.c       |   2 +-
>  .../syscalls/clock_gettime/clock_gettime01.c  |   1 -
>  .../kernel/syscalls/fanotify/fanotify22.c     |   1 -
>  .../syscalls/gettimeofday/gettimeofday02.c    |  15 +-
>  testcases/kernel/syscalls/inotify/inotify06.c |   7 +-
>  testcases/kernel/syscalls/inotify/inotify09.c |   1 +
>  testcases/kernel/syscalls/inotify/inotify11.c |   1 -
>  .../kernel/syscalls/ipc/shmctl/shmctl05.c     |   2 +-
>  .../kernel/syscalls/move_pages/move_pages12.c |   4 +-
>  .../perf_event_open/perf_event_open03.c       |  15 +-
>  testcases/kernel/syscalls/readv/readv01.c     |   1 -
>  .../rt_sigqueueinfo/rt_sigqueueinfo01.c       |   1 -
>  testcases/kernel/syscalls/sendmsg/sendmsg03.c |   1 +
>  .../kernel/syscalls/setsockopt/setsockopt06.c |   2 +-
>  .../kernel/syscalls/setsockopt/setsockopt07.c |   1 +
>  .../kernel/syscalls/setsockopt/setsockopt09.c |   1 -
>  testcases/kernel/syscalls/tgkill/tgkill03.c   |   1 -
>  .../syscalls/timerfd/timerfd_settime02.c      |   1 +
>  .../syscalls/userfaultfd/userfaultfd01.c      |   1 -
>  testcases/kernel/syscalls/writev/writev03.c   |   2 +-
>  63 files changed, 339 insertions(+), 296 deletions(-)
>  delete mode 100644 lib/newlib_tests/test10.c
>  delete mode 100644 lib/newlib_tests/test12.c
>  rename lib/newlib_tests/{test18.c => test_runtime01.c} (58%)
>  create mode 100644 lib/newlib_tests/test_runtime02.c
>  delete mode 100644 testcases/kernel/io/ltp-aiodio/read_checkzero.c


-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 06/30] crypto/af_alg02: Convert to runtime.

[Richard Palethorpe]

Hello,

"Cyril Hrubis" <chrubis@suse.cz> writes:

> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>  testcases/kernel/crypto/af_alg02.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/testcases/kernel/crypto/af_alg02.c b/testcases/kernel/crypto/af_alg02.c
> index 9894ffacd..b96b8b341 100644
> --- a/testcases/kernel/crypto/af_alg02.c
> +++ b/testcases/kernel/crypto/af_alg02.c
> @@ -65,7 +65,7 @@ static void run(void)
>  	TST_CHECKPOINT_WAIT(0);
>  
>  	while (!completed) {
> -		if (tst_timeout_remaining() <= 10) {
> +		if (tst_remaining_runtime()) {
>  			pthread_cancel(thr);
>  			tst_brk(TBROK,
>  				"Timed out while reading from request socket.");
> @@ -77,7 +77,7 @@ static void run(void)
>  
>  static struct tst_test test = {
>  	.test_all = run,
> -	.timeout = 20,
> +	.runtime_per_iteration = 20,

Isn't this the argument name to a function to set the runtime not the
tst_test member?

Probably they should be given the same name.

>  	.needs_checkpoints = 1,
>  	.tags = (const struct tst_tag[]) {
>  		{"linux-git", "ecaaab564978"},


-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 22/30] crypto/af_alg02: Convert to runtime

[Richard Palethorpe]

"Cyril Hrubis" <chrubis@suse.cz> writes:

> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>  testcases/kernel/crypto/af_alg02.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/testcases/kernel/crypto/af_alg02.c b/testcases/kernel/crypto/af_alg02.c
> index b96b8b341..9c2f488c4 100644
> --- a/testcases/kernel/crypto/af_alg02.c
> +++ b/testcases/kernel/crypto/af_alg02.c
> @@ -65,7 +65,7 @@ static void run(void)
>  	TST_CHECKPOINT_WAIT(0);
>  
>  	while (!completed) {
> -		if (tst_remaining_runtime()) {
> +		if (!tst_remaining_runtime()) {
>  			pthread_cancel(thr);
>  			tst_brk(TBROK,
>  				"Timed out while reading from request socket.");
> @@ -77,7 +77,7 @@ static void run(void)
>  
>  static struct tst_test test = {
>  	.test_all = run,
> -	.runtime_per_iteration = 20,
> +	.max_iteration_runtime = 20,

Ah, you probably want to ammend the previous commit.

>  	.needs_checkpoints = 1,
>  	.tags = (const struct tst_tag[]) {
>  		{"linux-git", "ecaaab564978"},


-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 00/30] Introduce runtime and conver tests

[Petr Vorel]

Hi all,

> > TODO:

> > This patchset is reasonably complete in a sense that it removes the
> > timeout API at the end. Still there are a few things to consider:

> > - is reusing the -I parameter a good idea? Wouldn't adding new parameter
> >   (-r) be better?

But -r is using only when .max_iteration_runtime set, right?
How should it behave on tests which don't set it?
Should -I and -r be mutually exclusive?
Maybe reusing -I would be simpler for users.

> Perhaps we could deprecate -I, but convert it to use the new mechanism
> while trying to keep the behaviour similar.

> I'm not sure we need a global -r option, but unlike -i it's not much
> effort to support. It would not surprise me if there are existing tests
> which don't work with -i.
                        ^^ Do you mean -I ?

Kind regards,
Petr

> > - there are quite likely tests that run for more than a second or a two
> >   and should be made runtime aware

> > - anything else?

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 23/30] fuzzy_sync: Convert to runtime

[Richard Palethorpe]

Hello Cyril,

Cyril Hrubis <chrubis@suse.cz> writes:

> The runtime for fuzzy sync tests are computed as timeout * exec_time_p.
> The default timeout was 300 and the default was 0.5.
>
> CC: Richard Palethorpe <rpalethorpe@suse.com>
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>  include/tst_fuzzy_sync.h                      | 19
> ++++---------------

The fzsync lib/newlib_tests validation tests probably need more than the
default 5 seconds.

Otherwise LGTM

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 23/30] fuzzy_sync: Convert to runtime

[Richard Palethorpe]

Hi,

Petr Vorel <pvorel@suse.cz> writes:

> Hi Cyril,
>
>> The runtime for fuzzy sync tests are computed as timeout * exec_time_p.
>> The default timeout was 300 and the default was 0.5.
>
>> CC: Richard Palethorpe <rpalethorpe@suse.com>
>> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
>> ---
>>  include/tst_fuzzy_sync.h                      | 19 ++++---------------
>>  testcases/kernel/crypto/af_alg07.c            |  1 +
>>  testcases/kernel/mem/thp/thp04.c              |  1 +
>>  testcases/kernel/pty/pty03.c                  |  2 +-
>>  testcases/kernel/pty/pty05.c                  |  1 +
>>  testcases/kernel/pty/pty06.c                  |  1 +
>>  testcases/kernel/pty/pty07.c                  |  1 +
>>  testcases/kernel/sound/snd_seq01.c            |  2 +-
>>  testcases/kernel/sound/snd_timer01.c          |  1 +
>>  testcases/kernel/syscalls/bind/bind06.c       |  2 +-
>>  testcases/kernel/syscalls/inotify/inotify09.c |  1 +
>>  .../kernel/syscalls/ipc/shmctl/shmctl05.c     |  2 +-
>>  testcases/kernel/syscalls/sendmsg/sendmsg03.c |  1 +
>>  .../kernel/syscalls/setsockopt/setsockopt06.c |  2 +-
>>  .../kernel/syscalls/setsockopt/setsockopt07.c |  1 +
>>  .../syscalls/timerfd/timerfd_settime02.c      |  1 +
>>  testcases/kernel/syscalls/writev/writev03.c   |  2 +-
>>  17 files changed, 20 insertions(+), 21 deletions(-)
>
> FYI you haven't converted fuzzy sync tests.
> I suppose these errors must be fix in first commit.
>
> ./tst_fuzzy_sync01
> tst_test.c:1522: TINFO: Timeout per run is 0h 00m 30s
> tst_test.c:1483: TBROK: Runtime not set!

Ah, I see the missing runtime actually causes an error. I would have
expected it to use a default 5 seconds?

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 01/30] Introduce a concept of max runtime

[Cyril Hrubis]

Hi!
> > +	int max_iteration_runtime;
> 
> It's not immediately clear if iteration refers to the inner test loop
> (e.g. Fuzzy Sync) or the outer loop performed by adding '-i N'. Perhaps
> it would be better to call it max_runtime and document that it is scaled
> by '-i N'?

It's not only the -i N paramater, it's carthesian product of:

	-i parameter x all_filesystems x test_variants

And every single instance of that product is limited by the runtime
value, which is the reason I want to have iteration explicitly in the
name. Maybe we should call it instance instead or whatever else that may
be more fitting.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 00/30] Introduce runtime and conver tests

[Richard Palethorpe]

Hello,

Petr Vorel <pvorel@suse.cz> writes:

> Hi all,
>
>> > TODO:
>
>> > This patchset is reasonably complete in a sense that it removes the
>> > timeout API at the end. Still there are a few things to consider:
>
>> > - is reusing the -I parameter a good idea? Wouldn't adding new parameter
>> >   (-r) be better?
>
> But -r is using only when .max_iteration_runtime set, right?
> How should it behave on tests which don't set it?
> Should -I and -r be mutually exclusive?
> Maybe reusing -I would be simpler for users.
>
>> Perhaps we could deprecate -I, but convert it to use the new mechanism
>> while trying to keep the behaviour similar.
>
>> I'm not sure we need a global -r option, but unlike -i it's not much
>> effort to support. It would not surprise me if there are existing tests
>> which don't work with -i.
>                         ^^ Do you mean -I ?

Nope, I mean -i, my point is that global parameters add another failure
mode and -i is the best example of this. However -r has less
implications than -i.

>
> Kind regards,
> Petr
>
>> > - there are quite likely tests that run for more than a second or a two
>> >   and should be made runtime aware
>
>> > - anything else?


-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 23/30] fuzzy_sync: Convert to runtime

[Cyril Hrubis]

Hi!
> Ah, I see the missing runtime actually causes an error. I would have
> expected it to use a default 5 seconds?

Yes I think that having the runtime explicitly written down is better
than some percentage of some value somewhere.

I.e. for most of the fuzzy sync tests what should be done is to
callibrate how long does the test need on an average hardware and put
that number in.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 01/30] Introduce a concept of max runtime

[Richard Palethorpe]

Hello,

Cyril Hrubis <chrubis@suse.cz> writes:

> Hi!
>> > +	int max_iteration_runtime;
>> 
>> It's not immediately clear if iteration refers to the inner test loop
>> (e.g. Fuzzy Sync) or the outer loop performed by adding '-i N'. Perhaps
>> it would be better to call it max_runtime and document that it is scaled
>> by '-i N'?
>
> It's not only the -i N paramater, it's carthesian product of:
>
> 	-i parameter x all_filesystems x test_variants
>
> And every single instance of that product is limited by the runtime
> value, which is the reason I want to have iteration explicitly in the
> name. Maybe we should call it instance instead or whatever else that may
> be more fitting.

I suppose it is the innermost runtime, so I would vote for inner_runtime
or just runtime. However whatever it is called it will require
explanation. What is considered to be an iteration, instance, inner, test
etc. is arbitrary.

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 01/30] Introduce a concept of max runtime

[Jan Stancek]

On Tue, May 3, 2022 at 8:36 PM Cyril Hrubis <chrubis@suse.cz> wrote:
>
> This commit introduce a concept of per iteration max test runtime. In
> other words test runtime is capped at a certain value in order to make
> testruns more deterministic. Test is free to to finish before the
> runtime is used up, for example when maximal number of iterations
> was reached, but test must stop once the runtime has been used up.
>
> Testcases that run for more than a second or two must check for
> remaining runtime by regular calls to tst_remaining_runtime() and should
> exit when zero is returned.
>
> The test max runtime must be set either by the .max_iteration_runtime in
> the tst_test structure or in the test setup by a call to
> tst_set_runtime().
>
> The test timeout is then computed as a sum of DEFAULT_TIMEOUT (currently
> set to 30 seconds) and the test runtime. The DEFAULT_TIMEOUT is nothing
> more than a safety margin for teardown of the test.
>
> This commit also maps the -I parameter to the test max runtime if
> available and introduces LTP_RUNTIME_MUL enviroment variable so that we
> have an easy controll over the runtime cap.
>
> Lastly but not least the function related to the timeout are turned into
> no-op by this commit and removed after all test are converted to the
> runtime API.
>
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>  doc/user-guide.txt                            |   4 +
>  include/tst_test.h                            |  27 ++++
>  lib/newlib_tests/.gitignore                   |   5 +-
>  lib/newlib_tests/test10.c                     |  22 ---
>  lib/newlib_tests/test12.c                     |  21 ---
>  lib/newlib_tests/test13.c                     |   1 -
>  lib/newlib_tests/test_children_cleanup.c      |   1 -
>  .../{test18.c => test_runtime01.c}            |  10 +-
>  lib/newlib_tests/test_runtime02.c             |  31 ++++
>  lib/tst_test.c                                | 151 ++++++++++++++----
>  10 files changed, 187 insertions(+), 86 deletions(-)
>  delete mode 100644 lib/newlib_tests/test10.c
>  delete mode 100644 lib/newlib_tests/test12.c
>  rename lib/newlib_tests/{test18.c => test_runtime01.c} (58%)
>  create mode 100644 lib/newlib_tests/test_runtime02.c
>
> diff --git a/doc/user-guide.txt b/doc/user-guide.txt
> index f41cbc733..d50d2e0cb 100644
> --- a/doc/user-guide.txt
> +++ b/doc/user-guide.txt
> @@ -25,6 +25,10 @@ For running LTP network tests see `testcases/network/README.md`.
>  | 'LTP_TIMEOUT_MUL'     | Multiply timeout, must be number >= 1 (> 1 is useful for
>                            slow machines to avoid unexpected timeout).
>                            Variable is also used in shell tests, but ceiled to int.

Previously, setting LTP_TIMEOUT_MUL meant that test could run twice as long.
After patch, the description above could probably use more detail, to explain
what this variable will affect and what's the difference compared to
LTP_RUNTIME_MUL.

> +| 'LTP_RUNTIME_MUL'     | Multiplies maximal test iteration runtime. Tests
> +                          that run for more than a second or two are capped on
> +                         runtime. You can scale the default runtime both up
> +                         and down with this multiplier.
>  | 'LTP_VIRT_OVERRIDE'   | Overrides virtual machine detection in the test
>                            library. Setting it to empty string tell the library
>                            that system is not a virtual machine. Other possible


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 01/30] Introduce a concept of max runtime

[Cyril Hrubis]

Hi!
> Previously, setting LTP_TIMEOUT_MUL meant that test could run twice as long.
> After patch, the description above could probably use more detail, to explain
> what this variable will affect and what's the difference compared to
> LTP_RUNTIME_MUL.

Indeed more documentation is needed.

The key point of this patchset is that timeout composes of two different
parts, static setup/cleanum timeout and runtime and each of them can be
tuned separately, so something along these lines should be added here as
well.

> > +| 'LTP_RUNTIME_MUL'     | Multiplies maximal test iteration runtime. Tests
> > +                          that run for more than a second or two are capped on
> > +                         runtime. You can scale the default runtime both up
> > +                         and down with this multiplier.
> >  | 'LTP_VIRT_OVERRIDE'   | Overrides virtual machine detection in the test
> >                            library. Setting it to empty string tell the library
> >                            that system is not a virtual machine. Other possible
> 

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 01/30] Introduce a concept of max runtime

[Li Wang]

[-- Attachment #1.1: Type: text/plain, Size: 1362 bytes --]

Richard Palethorpe <rpalethorpe@suse.de> wrote:


> >> > +  int max_iteration_runtime;
> >>
> >> It's not immediately clear if iteration refers to the inner test loop
> >> (e.g. Fuzzy Sync) or the outer loop performed by adding '-i N'. Perhaps
> >> it would be better to call it max_runtime and document that it is scaled
> >> by '-i N'?
> >
> > It's not only the -i N paramater, it's carthesian product of:
> >
> >       -i parameter x all_filesystems x test_variants
> >
> > And every single instance of that product is limited by the runtime
> > value, which is the reason I want to have iteration explicitly in the
> > name. Maybe we should call it instance instead or whatever else that may
> > be more fitting.
>
> I suppose it is the innermost runtime, so I would vote for inner_runtime
> or just runtime. However whatever it is called it will require
>

I do really have the same feelings here. I'm even afraid 'iteration' will
confused people who are not familiar with ltp-library.

From my comprehension, max_iteration_runtime takes effect on every
onefold test iteration, or maybe just directly rename to 'max_test_runtime'
(or max_runtime) with explanation in code comments.



> explanation. What is considered to be an iteration, instance, inner, test
> etc. is arbitrary.
>

Yes, includes "onefold" which hovers in my mind.


-- 
Regards,
Li Wang

[-- Attachment #1.2: Type: text/html, Size: 2589 bytes --]

[-- Attachment #2: Type: text/plain, Size: 60 bytes --]


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 01/30] Introduce a concept of max runtime

[Li Wang]

[-- Attachment #1.1: Type: text/plain, Size: 2130 bytes --]

Cyril Hrubis <chrubis@suse.cz> wrote:


> --- a/doc/user-guide.txt
> +++ b/doc/user-guide.txt
> @@ -25,6 +25,10 @@ For running LTP network tests see
> `testcases/network/README.md`.
>  | 'LTP_TIMEOUT_MUL'     | Multiply timeout, must be number >= 1 (> 1 is
> useful for
>                            slow machines to avoid unexpected timeout).
>                            Variable is also used in shell tests, but
> ceiled to int.
> +| 'LTP_RUNTIME_MUL'     | Multiplies maximal test iteration runtime.
> Tests
>

Seems 'LTP_RUNTIME_MUL'  does not take effect, maybe there is a bug
in saving multiply runtime to results->max_iteration_runtime?

--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -1634,7 +1634,7 @@ void tst_run_tcases(int argc, char *argv[], struct
tst_test *self)
        SAFE_SIGNAL(SIGUSR1, heartbeat_handler);

        if (tst_test->max_iteration_runtime)
-               results->max_iteration_runtime =
tst_test->max_iteration_runtime;
+               results->max_iteration_runtime = multiply_runtime();

        set_timeout();


$ LTP_RUNTIME_MUL=2 ./test_runtime01
tst_test.c:1522: TINFO: Timeout per run is 0h 00m 35s
test_runtime01.c:16: TINFO: Remaining runtime 5
test_runtime01.c:16: TINFO: Remaining runtime 4
test_runtime01.c:16: TINFO: Remaining runtime 3
test_runtime01.c:16: TINFO: Remaining runtime 2
test_runtime01.c:16: TINFO: Remaining runtime 1
test_runtime01.c:16: TINFO: Remaining runtime 0
test_runtime01.c:20: TPASS: Finished loop!

Summary:
passed   1
failed   0
broken   0
skipped  0
warnings 0


+++ b/lib/newlib_tests/test_runtime02.c
> @@ -0,0 +1,31 @@
> +// SPDX-License-Identifier: GPL-2.0-or-later
> +/*
> + * Copyright (c) 2021, Linux Test Project
> + */
> +/*
> + * This test is set up so that the timeout is not long enough to guarantee
> + * enough runtime for two iterations, i.e. the timeout without offset and
> after
> + * scaling is too small and the tests ends up with TBROK.
> + *
> + * You can fix this by exporting LTP_MAX_TEST_RUNTIME=10 before executing
> the
>

I didn't find where to achieve this LTP_MAX_TEST_RUNTIME function in the
patchset.


-- 
Regards,
Li Wang

[-- Attachment #1.2: Type: text/html, Size: 3588 bytes --]

[-- Attachment #2: Type: text/plain, Size: 60 bytes --]


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 01/30] Introduce a concept of max runtime

[Cyril Hrubis]

Hi!
> > I suppose it is the innermost runtime, so I would vote for inner_runtime
> > or just runtime. However whatever it is called it will require
> >
> 
> I do really have the same feelings here. I'm even afraid 'iteration' will
> confused people who are not familiar with ltp-library.
> 
> From my comprehension, max_iteration_runtime takes effect on every
> onefold test iteration, or maybe just directly rename to 'max_test_runtime'
> (or max_runtime) with explanation in code comments.

Okay then, I guess that we will simply go with max_runtime then and
describe that it applies to a single call of the run() function in the
comments.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 01/30] Introduce a concept of max runtime

[Cyril Hrubis]

Hi!
> > --- a/doc/user-guide.txt
> > +++ b/doc/user-guide.txt
> > @@ -25,6 +25,10 @@ For running LTP network tests see
> > `testcases/network/README.md`.
> >  | 'LTP_TIMEOUT_MUL'     | Multiply timeout, must be number >= 1 (> 1 is
> > useful for
> >                            slow machines to avoid unexpected timeout).
> >                            Variable is also used in shell tests, but
> > ceiled to int.
> > +| 'LTP_RUNTIME_MUL'     | Multiplies maximal test iteration runtime.
> > Tests
> >
> 
> Seems 'LTP_RUNTIME_MUL'  does not take effect, maybe there is a bug
> in saving multiply runtime to results->max_iteration_runtime?
> 
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -1634,7 +1634,7 @@ void tst_run_tcases(int argc, char *argv[], struct
> tst_test *self)
>         SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
> 
>         if (tst_test->max_iteration_runtime)
> -               results->max_iteration_runtime =
> tst_test->max_iteration_runtime;
> +               results->max_iteration_runtime = multiply_runtime();
> 
>         set_timeout();

My bad it's actually used in the -h switch but not when the runtime is
actually set. Will fix.

> $ LTP_RUNTIME_MUL=2 ./test_runtime01
> tst_test.c:1522: TINFO: Timeout per run is 0h 00m 35s
> test_runtime01.c:16: TINFO: Remaining runtime 5
> test_runtime01.c:16: TINFO: Remaining runtime 4
> test_runtime01.c:16: TINFO: Remaining runtime 3
> test_runtime01.c:16: TINFO: Remaining runtime 2
> test_runtime01.c:16: TINFO: Remaining runtime 1
> test_runtime01.c:16: TINFO: Remaining runtime 0
> test_runtime01.c:20: TPASS: Finished loop!
> 
> Summary:
> passed   1
> failed   0
> broken   0
> skipped  0
> warnings 0
> 
> 
> +++ b/lib/newlib_tests/test_runtime02.c
> > @@ -0,0 +1,31 @@
> > +// SPDX-License-Identifier: GPL-2.0-or-later
> > +/*
> > + * Copyright (c) 2021, Linux Test Project
> > + */
> > +/*
> > + * This test is set up so that the timeout is not long enough to guarantee
> > + * enough runtime for two iterations, i.e. the timeout without offset and
> > after
> > + * scaling is too small and the tests ends up with TBROK.
> > + *
> > + * You can fix this by exporting LTP_MAX_TEST_RUNTIME=10 before executing
> > the
> >
> 
> I didn't find where to achieve this LTP_MAX_TEST_RUNTIME function in the
> patchset.

That is a leftover from the v1 of the patchset that should be removed.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [Automated-testing] [PATCH v2 01/30] Introduce a concept of max runtime

[Cyril Hrubis]

Hi!
> > +static int multiply_runtime(void)
> > +{
> > +	static float runtime_mul = -1;
> > +
> > +	if (tst_test->max_iteration_runtime <= 0)
> > +		return tst_test->max_iteration_runtime;
> 
> nit; IMO it would be easier to understand if it returned
> TST_UNLIMITED_RUNTIME.

I wanted to keep this as a passthrough for any other possible values we
may add in the future, but I guess that we will not.

> > +
> > +	parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
> > +
> > +	return tst_test->max_iteration_runtime * runtime_mul;
> > +}
> > +
> >  static struct option {
> >  	char *optstr;
> >  	char *help;
> > @@ -477,6 +514,7 @@ static struct option {
> >  static void print_help(void)
> >  {
> >  	unsigned int i;
> > +	int timeout, runtime;
> >  
> >  	/* see doc/user-guide.txt, which lists also shell API variables */
> >  	fprintf(stderr, "Environment Variables\n");
> > @@ -489,10 +527,32 @@ static void print_help(void)
> >  	fprintf(stderr, "LTP_DEV_FS_TYPE      Filesystem used for testing (default: %s)\n", DEFAULT_FS_TYPE);
> >  	fprintf(stderr, "LTP_SINGLE_FS_TYPE   Testing only - specifies filesystem instead all supported (for .all_filesystems)\n");
> >  	fprintf(stderr, "LTP_TIMEOUT_MUL      Timeout multiplier (must be a number >=1)\n");
> > +	fprintf(stderr, "LTP_RUNTIME_MUL      Runtime multiplier (must be a number >=1)\n");
> >  	fprintf(stderr, "LTP_VIRT_OVERRIDE    Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
> >  	fprintf(stderr, "TMPDIR               Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
> >  	fprintf(stderr, "\n");
> >  
> > +	fprintf(stderr, "Timeout and runtime\n");
> > +	fprintf(stderr, "-------------------\n");
> > +
> > +	if (tst_test->max_iteration_runtime) {
> > +		runtime = multiply_runtime();
> > +
> > +		if (runtime == TST_UNLIMITED_RUNTIME) {
> > +			fprintf(stderr, "Test iteration runtime is not limited\n");
> > +		} else {
> > +			fprintf(stderr, "Test iteration runtime cap %ih %im %is\n",
> > +				runtime/3600, (runtime%3600)/60, runtime % 60);
> > +		}
> > +	}
> > +
> > +	timeout = tst_multiply_timeout(DEFAULT_TIMEOUT);
> > +
> > +	fprintf(stderr, "Test timeout (not including runtime) %ih %im %is\n",
> > +		timeout/3600, (timeout%3600)/60, timeout % 60);
> > +
> > +	fprintf(stderr, "\n");
> > +
> >  	fprintf(stderr, "Options\n");
> >  	fprintf(stderr, "-------\n");
> >  
> > @@ -620,7 +680,10 @@ static void parse_opts(int argc, char *argv[])
> >  			iterations = atoi(optarg);
> >  		break;
> >  		case 'I':
> > -			duration = atof(optarg);
> > +			if (tst_test->max_iteration_runtime > 0)
> > +				tst_test->max_iteration_runtime =
> > atoi(optarg);
> 
> Doesn't this change the semantics of -I? Duration does not seem to be
> per iteration, but overall execution time.

That's why I asked if we should override the -I or add a new option.

The thing is that setting overall execution time is simply wrong, as we
figured out previously, since in many cases we do not know in advance
how many variants will the test run until we actually start executing
it.

Also the -I option was kind of behaving like this from the start, since
the duration variable it sets applies on the most inner call (the
testrun() function in tst_test.c).

So all in all this patchset just fixes the -I option for long running
tests so that the runtime is actually propagated to the test itself.

> > +			else
> > +				duration = atof(optarg);
> >  		break;
> >  		case 'C':
> >  #ifdef UCLINUX
> > @@ -1034,6 +1097,11 @@ static void do_setup(int argc, char *argv[])
> >  	if (!tst_test)
> >  		tst_brk(TBROK, "No tests to run");
> >  
> > +	if (tst_test->max_iteration_runtime < -1) {
> > +		tst_brk(TBROK, "Invalid runtime value %i",
> > +			results->max_iteration_runtime);
> > +	}
> > +
> >  	if (tst_test->tconf_msg)
> >  		tst_brk(TCONF, "%s", tst_test->tconf_msg);
> >  
> > @@ -1404,39 +1472,36 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED)
> >  }
> >  
> >  unsigned int tst_timeout_remaining(void)
> > +{
> > +	tst_brk(TBROK, "Stub called!");
> > +	return 0;
> > +}
> > +
> > +unsigned int tst_remaining_runtime(void)
> >  {
> >  	static struct timespec now;
> > -	unsigned int elapsed;
> > +	int elapsed;
> > +
> > +	if (results->max_iteration_runtime == TST_UNLIMITED_RUNTIME)
> > +		return UINT_MAX;
> > +
> > +	if (results->max_iteration_runtime == 0)
> > +		tst_brk(TBROK, "Runtime not set!");
> >  
> >  	if (tst_clock_gettime(CLOCK_MONOTONIC, &now))
> >  		tst_res(TWARN | TERRNO, "tst_clock_gettime() failed");
> >  
> > -	elapsed = (tst_timespec_diff_ms(now, tst_start_time) + 500) / 1000;
> > -	if (results->timeout > elapsed)
> > -		return results->timeout - elapsed;
> > +	elapsed = tst_timespec_diff_ms(now, tst_start_time) / 1000;
> > +	if (results->max_iteration_runtime > elapsed)
> > +		return results->max_iteration_runtime - elapsed;
> >  
> >  	return 0;
> >  }
> >  
> > +
> >  unsigned int tst_multiply_timeout(unsigned int timeout)
> >  {
> > -	char *mul;
> > -	int ret;
> > -
> > -	if (timeout_mul == -1) {
> > -		mul = getenv("LTP_TIMEOUT_MUL");
> > -		if (mul) {
> > -			if ((ret = tst_parse_float(mul, &timeout_mul, 1, 10000))) {
> > -				tst_brk(TBROK, "Failed to parse LTP_TIMEOUT_MUL: %s",
> > -						tst_strerrno(ret));
> > -			}
> > -		} else {
> > -			timeout_mul = 1;
> > -		}
> > -	}
> > -	if (timeout_mul < 1)
> > -		tst_brk(TBROK, "LTP_TIMEOUT_MUL must to be int >= 1! (%.2f)",
> > -				timeout_mul);
> > +	parse_mul(&timeout_mul, "LTP_TIMEOUT_MUL", 0.099, 10000);
> >  
> >  	if (timeout < 1)
> >  		tst_brk(TBROK, "timeout must to be >= 1! (%d)", timeout);
> > @@ -1446,37 +1511,48 @@ unsigned int tst_multiply_timeout(unsigned int timeout)
> >  
> >  void tst_set_timeout(int timeout)
> >  {
> > -	if (timeout == -1) {
> > +	tst_brk(TBROK, "Stub called!");
> > +}
> > +
> > +static void set_timeout(void)
> > +{
> > +	unsigned int timeout = DEFAULT_TIMEOUT;
> > +
> > +	if (results->max_iteration_runtime == TST_UNLIMITED_RUNTIME) {
> >  		tst_res(TINFO, "Timeout per run is disabled");
> >  		return;
> >  	}
> >  
> > -	if (timeout < 1)
> > -		tst_brk(TBROK, "timeout must to be >= 1! (%d)", timeout);
> > +	if (results->max_iteration_runtime < 0) {
> > +		tst_brk(TBROK, "max_iteration_runtime must to be >= 0! (%d)",
> 
> It can be -1

Ah right, fixed the comment.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp