From mboxrd@z Thu Jan 1 00:00:00 1970 From: Petr Vorel Date: Wed, 26 May 2021 17:49:49 +0200 Subject: [LTP] [RFC PATCH 1/1] doc: Split test-writing-guidelines Message-ID: <20210526154949.4473-1-pvorel@suse.cz> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: ltp@lists.linux.it Test Writing Guidelines wiki page is too long, thus split it into 3 parts: 1) generic part (only first chapter, the same URL) 2) C test API (2.2 chapter, 4. Common problems) 3) shell test API Unfortunately this breaks users' bookmarks. Start numbering in headers from 1 on each page (links are broken anyway). NOTE: in order to have '...' formatting as code, main header ====== was needed to add on the page. Signed-off-by: Petr Vorel --- See it: https://github.com/pevik/ltp/wiki/Test-Writing-Guidelines https://github.com/pevik/ltp/wiki/C-Test-API https://github.com/pevik/ltp/wiki/Shell-Test-API Kind regards, Petr doc/c-test-api.txt | 2241 +++++++++++++++++++++++ doc/shell-test-api.txt | 740 ++++++++ doc/test-writing-guidelines.txt | 2993 +------------------------------ 3 files changed, 2993 insertions(+), 2981 deletions(-) create mode 100644 doc/c-test-api.txt create mode 100644 doc/shell-test-api.txt diff --git a/doc/c-test-api.txt b/doc/c-test-api.txt new file mode 100644 index 000000000..8caf9ec67 --- /dev/null +++ b/doc/c-test-api.txt @@ -0,0 +1,2241 @@ +LTP C Test API +============== + +1 Writing a test in C +--------------------- + +1.1 Basic test structure +~~~~~~~~~~~~~~~~~~~~~~~~ + +Let's start with an example, following code is a simple test for a 'getenv()'. + +[source,c] +------------------------------------------------------------------------------- +/* + * This is test for basic functionality of getenv(). + * + * - create an env variable and verify that getenv() can get get it + * - call getenv() with nonexisting variable name, check that it returns NULL + */ + +#include "tst_test.h" + +#define ENV1 "LTP_TEST_ENV" +#define ENV2 "LTP_TEST_THIS_DOES_NOT_EXIST" +#define ENV_VAL "val" + +static void setup(void) +{ + if (setenv(ENV1, ENV_VAL, 1)) + tst_brk(TBROK | TERRNO, "setenv() failed"); +} + +static void test(void) +{ + char *ret; + + ret = getenv(ENV1); + + if (!ret) { + tst_res(TFAIL, "getenv(" ENV1 ") = NULL"); + goto next; + } + + if (!strcmp(ret, ENV_VAL)) { + tst_res(TPASS, "getenv(" ENV1 ") = '"ENV_VAL "'"); + } else { + tst_res(TFAIL, "getenv(" ENV1 ") = '%s', expected '" + ENV_VAL "'", ret); + } + +next: + ret = getenv(ENV2); + + if (ret) + tst_res(TFAIL, "getenv(" ENV2 ") = '%s'", ret); + else + tst_res(TPASS, "getenv(" ENV2 ") = NULL"); +} + +static struct tst_test test = { + .test_all = test, + .setup = setup, +}; +------------------------------------------------------------------------------- + +Each test includes the 'tst_test.h' header and must define the 'struct +tst_test test' structure. + +The overall test initialization is done in the 'setup()' function. + +The overall cleanup is done in a 'cleanup()' function. Here 'cleanup()' is +omitted as the test does not have anything to clean up. If cleanup is set in +the test structure it's called on test exit just before the test library +cleanup. That especially means that cleanup can be called at any point in a +test execution. For example even when a test setup step has failed, therefore +the 'cleanup()' function must be able to cope with unfinished initialization, +and so on. + +The test itself is done in the 'test()' function. The test function must work +fine if called in a loop. + +There are two types of a test function pointers in the test structure. The +first one is a '.test_all' pointer that is used when test is implemented as a +single function. Then there is a '.test' function along with the number of +tests '.tcnt' that allows for more detailed result reporting. If the '.test' +pointer is set the function is called '.tcnt' times with an integer parameter +in range of [0, '.tcnt' - 1]. + +IMPORTANT: Only one of '.test' and '.test_all' can be set at a time. + +Each test has a default timeout set to 300s. The default timeout can be +overridden by setting '.timeout' in the test structure or by calling +'tst_set_timeout()' in the test 'setup()'. There are a few testcases whose run +time may vary arbitrarily, for these timeout can be disabled by setting it to +-1. + +Test can find out how much time (in seconds) is remaining to timeout, +by calling 'tst_timeout_remaining()'. + +LAPI headers +++++++++++++ + +Use our LAPI headers ('include "lapi/foo.h"') to keep compatibility with old +distributions. LAPI header should always include original header. Older linux +headers were problematic, therefore we preferred to use libc headers. There are +still some bugs when combining certain glibc headers with linux headers, see +https://sourceware.org/glibc/wiki/Synchronizing_Headers. + +A word about the cleanup() callback ++++++++++++++++++++++++++++++++++++ + +There are a few rules that needs to be followed in order to write correct +cleanup() callback. + +1. Free only resources that were initialized. Keep in mind that callback can + be executed at any point in the test run. + +2. Make sure to free resources in the reverse order they were + initialized. (Some of the steps may not depend on others and everything + will work if there were swapped but let's keep it in order.) + +The first rule may seem complicated at first however, on the contrary, it's +quite easy. All you have to do is to keep track of what was already +initialized. For example file descriptors needs to be closed only if they were +assigned a valid file descriptor. For most of the things you need to create +extra flag that is set right after successful initialization though. Consider, +for example, test setup below. + +We also prefer cleaning up resources that would otherwise be released on the +program exit. There are two main reasons for this decision. Resources such as +file descriptors and mmaped memory could block umounting a block device in +cases where the test library has mounted a filesystem for the test temporary +directory. Not freeing allocated memory would upset static analysis and tools +such as valgrind and produce false-positives when checking for leaks in the +libc and other low level libraries. + +[source,c] +------------------------------------------------------------------------------- +static int fd0, fd1, mount_flag; + +#define MNTPOINT "mntpoint" +#define FILE1 "mntpoint/file1" +#define FILE2 "mntpoint/file2" + +static void setup(void) +{ + SAFE_MKDIR(MNTPOINT, 0777); + SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL); + SAFE_MOUNT(tst_device->dev, MNTPOINT, tst_device->fs_type, 0, 0); + mount_flag = 1; + + fd0 = SAFE_OPEN(cleanup, FILE1, O_CREAT | O_RDWR, 0666); + fd1 = SAFE_OPEN(cleanup, FILE2, O_CREAT | O_RDWR, 0666); +} +------------------------------------------------------------------------------- + +In this case the 'cleanup()' function may be invoked when any of the 'SAFE_*' +macros has failed and therefore must be able to work with unfinished +initialization as well. Since global variables are initialized to zero we can +just check that fd > 0 before we attempt to close it. The mount function +requires extra flag to be set after device was successfully mounted. + +[source,c] +------------------------------------------------------------------------------- +static void cleanup(void) +{ + if (fd1 > 0) + SAFE_CLOSE(fd1); + + if (fd0 > 0) + SAFE_CLOSE(fd0); + + if (mount_flag && tst_umouont(MNTPOINT)) + tst_res(TWARN | TERRNO, "umount(%s)", MNTPOINT); +} +------------------------------------------------------------------------------- + +IMPORTANT: 'SAFE_MACROS()' used in cleanup *do not* exit the test. Failure + only produces a warning and the 'cleanup()' carries on. This is + intentional as we want to execute as much 'cleanup()' as possible. + +WARNING: Calling tst_brk() in test 'cleanup()' does not exit the test as well + and 'TBROK' is converted to 'TWARN'. + +NOTE: Creation and removal of the test temporary directory is handled in + the test library and the directory is removed recursively. Therefore + we do not have to remove files and directories in the test cleanup. + +1.2 Basic test interface +~~~~~~~~~~~~~~~~~~~~~~~~ + +[source,c] +------------------------------------------------------------------------------- +void tst_res(int ttype, char *arg_fmt, ...); +------------------------------------------------------------------------------- + +Printf-like function to report test result, it's mostly used with ttype: + +|============================== +| 'TPASS' | Test has passed. +| 'TFAIL' | Test has failed. +| 'TINFO' | General message. +| 'TWARN' | Something went wrong but we decided to continue. Mostly used in cleanup functions. +|============================== + +The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print +'errno', 'TST_ERR' respectively. + +[source,c] +------------------------------------------------------------------------------- +void tst_brk(int ttype, char *arg_fmt, ...); +------------------------------------------------------------------------------- + +Printf-like function to report error and exit the test, it can be used with ttype: + +|============================================================ +| 'TBROK' | Something has failed in test preparation phase. +| 'TCONF' | Test is not appropriate for current configuration + (syscall not implemented, unsupported arch, ...) +|============================================================ + +The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print +'errno', 'TST_ERR' respectively. + +There are also 'TST_EXP_*()' macros that can simplify syscall unit tests to a +single line, use them whenever possible. These macros take a function call as +the first parameter and a printf-like format string and parameters as well. +These test macros then expand to a code that runs the call, checks the return +value and errno and reports the test result. + +[source,c] +------------------------------------------------------------------------------- +static void test(void) +{ + ... + TST_EXP_PASS(stat(fname, &statbuf), "stat(%s, ...)", fname); + + if (!TST_PASS) + return; + ... +} +------------------------------------------------------------------------------- + +The 'TST_EXP_PASS()' can be used for calls that return -1 on failure and 0 on +success. It will check for the return value and reports failure if the return +value is not equal to 0. The call also sets the 'TST_PASS' variable to 1 if +the call succeeeded. + +[source,c] +------------------------------------------------------------------------------- +static void test(void) +{ + ... + TST_EXP_FD(open(fname, O_RDONLY), "open(%s, O_RDONLY)", fname); + + SAFE_CLOSE(TST_RET); + ... +} +------------------------------------------------------------------------------- + +The 'TST_EXP_FD()' is the same as 'TST_EXP_PASS()' the only difference is that +the return value is expected to be a file descriptor so the call passes if +positive integer is returned. + +[source,c] +------------------------------------------------------------------------------- +static void test(void) +{ + ... + TST_EXP_FAIL(stat(fname, &statbuf), ENOENT, "stat(%s, ...)", fname); + ... +} +------------------------------------------------------------------------------- + +The 'TST_EXP_FAIL()' is similar to 'TST_EXP_PASS()' but it fails the test if +the call haven't failed with -1 and 'errno' wasn't set to the expected one +passed as the second argument. + +[source,c] +------------------------------------------------------------------------------- +const char *tst_strsig(int sig); +------------------------------------------------------------------------------- + +Return the given signal number's corresponding string. + +[source,c] +------------------------------------------------------------------------------- +const char *tst_strerrno(int err); +------------------------------------------------------------------------------- + +Return the given errno number's corresponding string. Using this function to +translate 'errno' values to strings is preferred. You should not use the +'strerror()' function in the testcases. + +[source,c] +------------------------------------------------------------------------------- +const char *tst_strstatus(int status); +------------------------------------------------------------------------------- + +Returns string describing the status as returned by 'wait()'. + +WARNING: This function is not thread safe. + +[source,c] +------------------------------------------------------------------------------- +void tst_set_timeout(unsigned int timeout); +------------------------------------------------------------------------------- + +Allows for setting timeout per test iteration dynamically in the test setup(), +the timeout is specified in seconds. There are a few testcases whose runtime +can vary arbitrarily, these can disable timeouts by setting it to -1. + +[source,c] +------------------------------------------------------------------------------- +void tst_flush(void); +------------------------------------------------------------------------------- + +Flush output streams, handling errors appropriately. + +This function is rarely needed when you have to flush the output streams +before calling 'fork()' or 'clone()'. Note that the 'SAFE_FORK()' and 'SAFE_CLONE()' +calls this function automatically. See 2.4 FILE buffers and fork() for explanation +why is this needed. + +1.3 Test temporary directory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If '.needs_tmpdir' is set to '1' in the 'struct tst_test' unique test +temporary is created and it's set as the test working directory. Tests *MUST +NOT* create temporary files outside that directory. The flag is not needed to +be set when use these flags: '.all_filesystems', '.format_device', '.mntpoint', +'.mount_device' '.needs_checkpoints', '.needs_device', '.resource_file' +(these flags imply creating temporary directory). + +IMPORTANT: Close all file descriptors (that point to files in test temporary + directory, even the unlinked ones) either in the 'test()' function + or in the test 'cleanup()' otherwise the test may break temporary + directory removal on NFS (look for "NFS silly rename"). + +1.4 Safe macros +~~~~~~~~~~~~~~~ + +Safe macros aim to simplify error checking in test preparation. Instead of +calling system API functions, checking for their return value and aborting the +test if the operation has failed, you just use corresponding safe macro. + +Use them whenever it's possible. + +Instead of writing: + +[source,c] +------------------------------------------------------------------------------- + fd = open("/dev/null", O_RDONLY); + if (fd < 0) + tst_brk(TBROK | TERRNO, "opening /dev/null failed"); +------------------------------------------------------------------------------- + +You write just: + +[source,c] +------------------------------------------------------------------------------- + fd = SAFE_OPEN("/dev/null", O_RDONLY); +------------------------------------------------------------------------------- + +IMPORTANT: The SAFE_CLOSE() function also sets the passed file descriptor to -1 + after it's successfully closed. + +They can also simplify reading and writing of sysfs files, you can, for +example, do: + +[source,c] +------------------------------------------------------------------------------- + SAFE_FILE_SCANF("/proc/sys/kernel/pid_max", "%lu", &pid_max); +------------------------------------------------------------------------------- + +See 'include/tst_safe_macros.h', 'include/tst_safe_stdio.h' and +'include/tst_safe_file_ops.h' and 'include/tst_safe_net.h' for a complete list. + +1.5 Test specific command line options +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +[source,c] +------------------------------------------------------------------------------- +struct tst_option { + char *optstr; + char **arg; + char *help; +}; +------------------------------------------------------------------------------- + +Test specific command line parameters can be passed with the 'NULL' terminated +array of 'struct tst_option'. The 'optstr' is the command line option i.e. "o" +or "o:" if option has a parameter. Only short options are supported. The 'arg' +is where 'optarg' is stored upon match. If option has no parameter it's set to +non-'NULL' value if option was present. The 'help' is a short help string. + +NOTE: The test parameters must not collide with common test parameters defined + in the library the currently used ones are +-i+, +-I+, +-C+, and +-h+. + +[source,c] +------------------------------------------------------------------------------- +int tst_parse_int(const char *str, int *val, int min, int max); +int tst_parse_float(const char *str, float *val, float min, float max); +------------------------------------------------------------------------------- + +Helpers for parsing the strings returned in the 'struct tst_option'. + +Both return zero on success and 'errno', mostly 'EINVAL' or 'ERANGE', on +failure. + +Both functions are no-op if 'str' is 'NULL'. + +The valid range for result includes both 'min' and 'max'. + +.Example Usage +[source,c] +------------------------------------------------------------------------------- +#include +#include "tst_test.h" + +static char *str_threads; +static int threads = 10; + +static struct tst_option options[] = { + {"t:", &str_threads, "Number of threads (default 10)"}, + ... + {NULL, NULL, NULL} +}; + +static void setup(void) +{ + if (tst_parse_int(str_threads, &threads, 1, INT_MAX)) + tst_brk(TBROK, "Invalid number of threads '%s'", str_threads); + + ... +} + +static void test_threads(void) +{ + ... + + for (i = 0; i < threads; i++) { + ... + } + + ... +} + +static struct tst_test test = { + ... + .options = options, + ... +}; +------------------------------------------------------------------------------- + + +1.6 Runtime kernel version detection +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Testcases for newly added kernel functionality require kernel newer than a +certain version to run. All you need to skip a test on older kernels is to +set the '.min_kver' string in the 'struct tst_test' to a minimal required +kernel version, e.g. '.min_kver = "2.6.30"'. + +For more complicated operations such as skipping a test for a certain range +of kernel versions, following functions could be used: + +[source,c] +------------------------------------------------------------------------------- +int tst_kvercmp(int r1, int r2, int r3); + +struct tst_kern_exv { + char *dist_name; + char *extra_ver; +}; + +int tst_kvercmp2(int r1, int r2, int r3, struct tst_kern_exv *vers); +------------------------------------------------------------------------------- + +These two functions are intended for runtime kernel version detection. They +parse the output from 'uname()' and compare it to the passed values. + +The return value is similar to the 'strcmp()' function, i.e. zero means equal, +negative value means that the kernel is older than than the expected value and +positive means that it's newer. + +The second function 'tst_kvercmp2()' allows for specifying per-vendor table of +kernel versions as vendors typically backport fixes to their kernels and the +test may be relevant even if the kernel version does not suggests so. See +'testcases/kernel/syscalls/inotify/inotify04.c' for example usage. + +WARNING: The shell 'tst_kvercmp' maps the result into unsigned integer - the + process exit value. + +1.7 Fork()-ing +~~~~~~~~~~~~~~ + +Be wary that if the test forks and there were messages printed by the +'tst_*()' interfaces, the data may still be in libc/kernel buffers and these +*ARE NOT* flushed automatically. + +This happens when 'stdout' gets redirected to a file. In this case, the +'stdout' is not line buffered, but block buffered. Hence after a fork content +of the buffers will be printed by the parent and each of the children. + +To avoid that you should use 'SAFE_FORK()', 'SAFE_CLONE()' or 'tst_clone()'. + +IMPORTANT: You have to set the '.forks_child' flag in the test structure + if your testcase forks or calls 'SAFE_CLONE()'. + +1.8 Doing the test in the child process +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Results reported by 'tst_res()' are propagated to the parent test process via +block of shared memory. + +Calling 'tst_brk()' causes child process to exit with non-zero exit value. +Which means that it's safe to use 'SAFE_*()' macros in the child processes as +well. + +Children that outlive the 'test()' function execution are waited for in the +test library. Unclean child exit (killed by signal, non-zero exit value, etc.) +will cause the main test process to exit with 'tst_brk()', which especially +means that 'TBROK' propagated from a child process will cause the whole test +to exit with 'TBROK'. + +If a test needs a child that segfaults or does anything else that cause it to +exit uncleanly all you need to do is to wait for such children from the +'test()' function so that it's reaped before the main test exits the 'test()' +function. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +void tst_reap_children(void); +------------------------------------------------------------------------------- + +The 'tst_reap_children()' function makes the process wait for all of its +children and exits with 'tst_brk(TBROK, ...)' if any of them returned +a non zero exit code. + +When using 'SAFE_CLONE' or 'tst_clone', this may not work depending on +the parameters passed to clone. The following call to 'SAFE_CLONE' is +identical to 'fork()', so will work as expected. + +[source,c] +-------------------------------------------------------------------------------- +const struct tst_clone_args args = { + .exit_signal = SIGCHLD, +}; + +SAFE_CLONE(&args); +-------------------------------------------------------------------------------- + +If 'exit_signal' is set to something else, then this will break +'tst_reap_children'. It's not expected that all parameters to clone will +work with the LTP library unless specific action is taken by the test code. + +.Using 'tst_res()' from binaries started by 'exec()' +[source,c] +------------------------------------------------------------------------------- +/* test.c */ +#define _GNU_SOURCE +#include +#include "tst_test.h" + +static void do_test(void) +{ + char *const argv[] = {"test_exec_child", NULL}; + char path[4096]; + + if (tst_get_path("test_exec_child", path, sizeof(path))) + tst_brk(TCONF, "Couldn't find test_exec_child in $PATH"); + + execve(path, argv, environ); + + tst_res(TFAIL | TERRNO, "EXEC!"); +} + +static struct tst_test test = { + .test_all = do_test, + .child_needs_reinit = 1, +}; + +/* test_exec_child.c */ +#define TST_NO_DEFAULT_MAIN +#include "tst_test.h" + +int main(void) +{ + tst_reinit(); + tst_res(TPASS, "Child passed!"); + return 0; +} +------------------------------------------------------------------------------- + +The 'tst_res()' function can be also used from binaries started by 'exec()', +the parent test process has to set the '.child_needs_reinit' flag so that the +library prepares for it and has to make sure the 'LTP_IPC_PATH' environment +variable is passed down, then the very fist thing the program has to call in +'main()' is 'tst_reinit()' that sets up the IPC. + +1.9 Fork() and Parent-child synchronization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As LTP tests are written for Linux, most of the tests involve fork()-ing and +parent-child process synchronization. LTP includes a checkpoint library that +provides wait/wake futex based functions. + +In order to use checkpoints the '.needs_checkpoints' flag in the 'struct +tst_test' must be set to '1', this causes the test library to initialize +checkpoints before the 'test()' function is called. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +TST_CHECKPOINT_WAIT(id) + +TST_CHECKPOINT_WAIT2(id, msec_timeout) + +TST_CHECKPOINT_WAKE(id) + +TST_CHECKPOINT_WAKE2(id, nr_wake) + +TST_CHECKPOINT_WAKE_AND_WAIT(id) +------------------------------------------------------------------------------- + +The checkpoint interface provides pair of wake and wait functions. The 'id' is +unsigned integer which specifies checkpoint to wake/wait for. As a matter of +fact it's an index to an array stored in a shared memory, so it starts on +'0' and there should be enough room for at least of hundred of them. + +The 'TST_CHECKPOINT_WAIT()' and 'TST_CHECKPOINT_WAIT2()' suspends process +execution until it's woken up or until timeout is reached. + +The 'TST_CHECKPOINT_WAKE()' wakes one process waiting on the checkpoint. +If no process is waiting the function retries until it success or until +timeout is reached. + +If timeout has been reached process exits with appropriate error message (uses +'tst_brk()'). + +The 'TST_CHECKPOINT_WAKE2()' does the same as 'TST_CHECKPOINT_WAKE()' but can +be used to wake precisely 'nr_wake' processes. + +The 'TST_CHECKPOINT_WAKE_AND_WAIT()' is a shorthand for doing wake and then +immediately waiting on the same checkpoint. + +Child processes created via 'SAFE_FORK()' are ready to use the checkpoint +synchronization functions, as they inherited the mapped page automatically. + +Child processes started via 'exec()', or any other processes not forked from +the test process must initialize the checkpoint by calling 'tst_reinit()'. + +For the details of the interface, look into the 'include/tst_checkpoint.h'. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +/* + * Waits for process state change. + * + * The state is one of the following: + * + * R - process is running + * S - process is sleeping + * D - process sleeping uninterruptibly + * Z - zombie process + * T - process is traced + */ +TST_PROCESS_STATE_WAIT(pid, state, msec_timeout) +------------------------------------------------------------------------------- + +The 'TST_PROCESS_STATE_WAIT()' waits until process 'pid' is in requested +'state' or timeout is reached. The call polls +/proc/pid/stat+ to get this +information. A timeout of 0 will wait infinitely. + +On timeout -1 is returned and errno set to ETIMEDOUT. + +It's mostly used with state 'S' which means that process is sleeping in kernel +for example in 'pause()' or any other blocking syscall. + +1.10 Signals and signal handlers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you need to use signal handlers, keep the code short and simple. Don't +forget that the signal handler is called asynchronously and can interrupt the +code execution at any place. + +This means that problems arise when global state is changed both from the test +code and signal handler, which will occasionally lead to: + +* Data corruption (data gets into inconsistent state), this may happen, for + example, for any operations on 'FILE' objects. + +* Deadlock, this happens, for example, if you call 'malloc(2)', 'free(2)', + etc. from both the test code and the signal handler at the same time since + 'malloc' has global lock for it's internal data structures. (Be wary that + 'malloc(2)' is used by the libc functions internally too.) + +* Any other unreproducible and unexpected behavior. + +Quite common mistake is to call 'exit(3)' from a signal handler. Note that this +function is not signal-async-safe as it flushes buffers, etc. If you need to +exit a test immediately from a signal handler use '_exit(2)' instead. + +TIP: See 'man 7 signal' for the list of signal-async-safe functions. + +If a signal handler sets a variable, its declaration must be 'volatile', +otherwise compiler may misoptimize the code. This is because the variable may +not be changed in the compiler code flow analysis. There is 'sig_atomic_t' +type defined in C99 but this one *DOES NOT* imply 'volatile' (it's just a +'typedef' to 'int'). So the correct type for a flag that is changed from a +signal handler is either 'volatile int' or 'volatile sig_atomic_t'. + +If a crash (e.g. triggered by signal SIGSEGV) is expected in testing, you +can avoid creation of core files by calling tst_no_corefile() function. +This takes effect for process (and its children) which invoked it, unless +they subsequently modify RLIMIT_CORE. + +Note that LTP library will reap any processes that test didn't reap itself, +and report any non-zero exit code as failure. + +1.11 Kernel Modules +~~~~~~~~~~~~~~~~~~~ + +There are certain cases where the test needs a kernel part and userspace part, +happily, LTP can build a kernel module and then insert it to the kernel on test +start for you. See 'testcases/kernel/device-drivers/block' for details. + +1.12 Useful macros +~~~~~~~~~~~~~~~~~~ + +[source,c] +------------------------------------------------------------------------------- +ARRAY_SIZE(arr) +------------------------------------------------------------------------------- + +Returns the size of statically defined array, i.e. +'(sizeof(arr) / sizeof(*arr))' + +[source,c] +------------------------------------------------------------------------------- +LTP_ALIGN(x, a) +------------------------------------------------------------------------------- + +Aligns the x to be next multiple of a. The a must be power of 2. + +1.13 Filesystem type detection and skiplist +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some tests are known to fail on certain filesystems (you cannot swap on TMPFS, +there are unimplemented 'fcntl()' etc.). + +If your test needs to be skipped on certain filesystems use the +'.skip_filesystems' field in the tst_test structure as follows: + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static struct tst_test test = { + ... + .skip_filesystems = (const char *const []) { + "tmpfs", + "ramfs", + "nfs", + NULL + }, +}; +------------------------------------------------------------------------------- + +When the '.all_filesystem' flag is set the '.skip_filesystems' list is passed +to the function that detects supported filesystems any listed filesystem is +not included in the resulting list of supported filesystems. + +If test needs to adjust expectations based on filesystem type it's also +possible to detect filesystem type at the runtime. This is preferably used +when only subset of the test is not applicable for a given filesystem. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static void run(void) +{ + ... + + switch ((type = tst_fs_type("."))) { + case TST_NFS_MAGIC: + case TST_TMPFS_MAGIC: + case TST_RAMFS_MAGIC: + tst_brk(TCONF, "Subtest not supported on %s", + tst_fs_type_name(type)); + return; + break; + } + + ... +} +------------------------------------------------------------------------------- + +1.14 Thread-safety in the LTP library +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is safe to use library 'tst_res()' function in multi-threaded tests. + +Only the main thread must return from the 'test()' function to the test +library and that must be done only after all threads that may call any library +function has been terminated. That especially means that threads that may call +'tst_brk()' must terminate before the execution of the 'test()' function +returns to the library. This is usually done by the main thread joining all +worker threads at the end of the 'test()' function. Note that the main thread +will never get to the library code in a case that 'tst_brk()' was called from +one of the threads since it will sleep at least in 'pthread_join()' on the +thread that called the 'tst_brk()' till 'exit()' is called by 'tst_brk()'. + +The test-supplied cleanup function runs *concurrently* to the rest of the +threads in a case that cleanup was entered from 'tst_brk()'. Subsequent +threads entering 'tst_brk()' must be suspended or terminated at the start of +the user supplied cleanup function. It may be necessary to stop or exit +the rest of the threads before the test cleans up as well. For example threads +that create new files should be stopped before temporary directory is be +removed. + +Following code example shows thread safe cleanup function example using atomic +increment as a guard. The library calls its cleanup after the execution returns +from the user supplied cleanup and expects that only one thread returns from +the user supplied cleanup to the test library. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static void cleanup(void) +{ + static int flag; + + if (tst_atomic_inc(&flag) != 1) + pthread_exit(NULL); + + /* if needed stop the rest of the threads here */ + + ... + + /* then do cleanup work */ + + ... + + /* only one thread returns to the library */ +} +------------------------------------------------------------------------------- + + +1.15 Testing with a block device +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some tests needs a block device (inotify tests, syscall 'EROFS' failures, +etc.). LTP library contains a code to prepare a testing device. + +If '.needs_device' flag in the 'struct tst_test' is set the 'tst_device' +structure is initialized with a path to a test device and default filesystem +to be used. + +You can also request minimal device size in megabytes by setting +'.dev_min_size' the device is guaranteed to have at least the requested size +then. + +If '.format_device' flag is set the device is formatted with a filesystem as +well. You can use '.dev_fs_type' to override the default filesystem type if +needed and pass additional options to mkfs via '.dev_fs_opts' and +'.dev_extra_opts' pointers. Note that '.format_device' implies '.needs_device' +there is no need to set both. + +If '.mount_device' is set, the device is mounted at '.mntpoint' which is used +to pass a directory name that will be created and used as mount destination. +You can pass additional flags and data to the mount command via '.mnt_flags' +and '.mnt_data' pointers. Note that '.mount_device' implies '.needs_device' +and '.format_device' so there is no need to set the later two. + +If '.needs_rofs' is set, read-only filesystem is mounted at '.mntpoint' this +one is supposed to be used for 'EROFS' tests. + +If '.all_filesystems' is set the test function is executed for all supported +filesystems. Supported filesystems are detected based on existence of the +'mkfs.$fs' helper and on kernel support to mount it. For each supported +filesystem the 'tst_device.fs_type' is set to the currently tested fs type, if +'.format_device' is set the device is formatted as well, if '.mount_device' is +set it's mounted at '.mntpoint'. Also the test timeout is reset for each +execution of the test function. This flag is expected to be used for filesystem +related syscalls that are at least partly implemented in the filesystem +specific code e.g. fallocate(). + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +struct tst_device { + const char *dev; + const char *fs_type; +}; + +extern struct tst_device *tst_device; + +int tst_umount(const char *path); +------------------------------------------------------------------------------- + +In case that 'LTP_DEV' is passed to the test in an environment, the library +checks that the file exists and that it's a block device, if +'.device_min_size' is set the device size is checked as well. If 'LTP_DEV' +wasn't set or if size requirements were not met a temporary file is created +and attached to a free loop device. + +If there is no usable device and loop device couldn't be initialized the test +exits with 'TCONF'. + +The 'tst_umount()' function works exactly as 'umount(2)' but retries several +times on 'EBUSY'. This is because various desktop daemons (gvfsd-trash is known +for that) may be stupid enough to probe all newly mounted filesystem which +results in 'umount(2)' failing with 'EBUSY'. + +IMPORTANT: All testcases should use 'tst_umount()' instead of 'umount(2)' to + umount filesystems. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_find_free_loopdev(const char *path, size_t path_len); +------------------------------------------------------------------------------- + +This function finds a free loopdev and returns the free loopdev minor (-1 for no +free loopdev). If path is non-NULL, it will be filled with free loopdev path. +If you want to use a customized loop device, we can call tst_find_free_loopdev +(NULL, 0) in tests to get a free minor number and then mknod. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +unsigned long tst_dev_bytes_written(const char *dev); +------------------------------------------------------------------------------- + +This function reads test block device stat file (/sys/block//stat) and +returns the bytes written since the last invocation of this function. To avoid +FS deferred IO metadata/cache interference, we suggest doing "syncfs" before the +tst_dev_bytes_written first invocation. And an inline function named tst_dev_sync +is created for that intention. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +voud tst_find_backing_dev(const char *path, char *dev); +------------------------------------------------------------------------------- + +This function finds the block dev that this path belongs to, it uses stat function +to get the major/minor number of the path. Then scan them in "/proc/self/mountinfo" +and list 2th column value after ' - ' string as its block dev if match succeeds. + +1.16 Formatting a device with a filesystem +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static void setup(void) +{ + ... + SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL); + ... +} +------------------------------------------------------------------------------- + +This function takes a path to a device, filesystem type and an array of extra +options passed to mkfs. + +The fs options 'fs_opts' should either be 'NULL' if there are none, or a +'NULL' terminated array of strings such as: ++const char *const opts[] = {"-b", "1024", NULL}+. + +The extra options 'extra_opts' should either be 'NULL' if there are none, or a +'NULL' terminated array of strings such as +{"102400", NULL}+; 'extra_opts' +will be passed after device name. e.g: +mkfs -t ext4 -b 1024 /dev/sda1 102400+ +in this case. + +Note that perfer to store the options which can be passed before or after device +name by 'fs_opts' array. + +1.17 Verifying a filesystem's free space +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some tests have size requirements for the filesystem's free space. If these +requirements are not satisfied, the tests should be skipped. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_fs_has_free(const char *path, unsigned int size, unsigned int mult); +------------------------------------------------------------------------------- + +The 'tst_fs_has_free()' function returns 1 if there is enough space and 0 if +there is not. + +The 'path' is the pathname of any directory/file within a filesystem. + +The 'mult' is a multiplier, one of 'TST_BYTES', 'TST_KB', 'TST_MB' or 'TST_GB'. + +The required free space is calculated by 'size * mult', e.g. +'tst_fs_has_free("/tmp/testfile", 64, TST_MB)' will return 1 if the +filesystem, which '"/tmp/testfile"' is in, has 64MB free space at least, and 0 +if not. + +1.18 Files, directories and fs limits +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some tests need to know the maximum count of links to a regular file or +directory, such as 'rename(2)' or 'linkat(2)' to test 'EMLINK' error. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_fs_fill_hardlinks(const char *dir); +------------------------------------------------------------------------------- + +Try to get maximum count of hard links to a regular file inside the 'dir'. + +NOTE: This number depends on the filesystem 'dir' is on. + +This function uses 'link(2)' to create hard links to a single file until it +gets 'EMLINK' or creates 65535 links. If the limit is hit, the maximum number of +hardlinks is returned and the 'dir' is filled with hardlinks in format +"testfile%i", where i belongs to [0, limit) interval. If no limit is hit or if +'link(2)' failed with 'ENOSPC' or 'EDQUOT', zero is returned and previously +created files are removed. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_fs_fill_subdirs(const char *dir); +------------------------------------------------------------------------------- + +Try to get maximum number of subdirectories in directory. + +NOTE: This number depends on the filesystem 'dir' is on. For current kernel, +subdir limit is not available for all filesystems (available for ext2, ext3, +minix, sysv and more). If the test runs on some other filesystems, like ramfs, +tmpfs, it will not even try to reach the limit and return 0. + +This function uses 'mkdir(2)' to create directories in 'dir' until it gets +'EMLINK' or creates 65535 directories. If the limit is hit, the maximum number +of subdirectories is returned and the 'dir' is filled with subdirectories in +format "testdir%i", where i belongs to [0, limit - 2) interval (because each +newly created dir has two links already - the '.' and the link from parent +dir). If no limit is hit or if 'mkdir(2)' failed with 'ENOSPC' or 'EDQUOT', +zero is returned and previously created directories are removed. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_dir_is_empty(const char *dir, int verbose); +------------------------------------------------------------------------------- + +Returns non-zero if directory is empty and zero otherwise. + +Directory is considered empty if it contains only '.' and '..'. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +void tst_purge_dir(const char *path); +------------------------------------------------------------------------------- + +Deletes the contents of given directory but keeps the directory itself. Useful +for cleaning up the temporary directory and mount points between test cases or +test iterations. Terminates the program with 'TBROK' on error. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_fill_fd(int fd, char pattern, size_t bs, size_t bcount); +------------------------------------------------------------------------------- + +Fill a file with specified pattern using file descriptor. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_prealloc_size_fd(int fd, size_t bs, size_t bcount); +------------------------------------------------------------------------------- + +Preallocate the specified amount of space using 'fallocate()'. Falls back to +'tst_fill_fd()' if 'fallocate()' fails. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_fill_file(const char *path, char pattern, size_t bs, size_t bcount); +------------------------------------------------------------------------------- + +Creates/overwrites a file with specified pattern using file path. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_prealloc_file(const char *path, size_t bs, size_t bcount); +------------------------------------------------------------------------------- + +Create/overwrite a file and preallocate the specified amount of space for it. +The allocated space will not be initialized to any particular content. + +1.19 Getting an unused PID number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some tests require a 'PID', which is not used by the OS (does not belong to +any process within it). For example, kill(2) should set errno to 'ESRCH' if +it's passed such 'PID'. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +pid_t tst_get_unused_pid(void); +------------------------------------------------------------------------------- + +Return a 'PID' value not used by the OS or any process within it. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_get_free_pids(void); +------------------------------------------------------------------------------- + +Returns number of unused pids in the system. Note that this number may be +different once the call returns and should be used only for rough estimates. + +1.20 Running executables +~~~~~~~~~~~~~~~~~~~~~~~~ + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +int tst_cmd(const char *const argv[], + const char *stdout_path, + const char *stderr_path, + enum tst_cmd_flags flags); +------------------------------------------------------------------------------- + +'tst_cmd()' is a wrapper for 'vfork() + execvp()' which provides a way +to execute an external program. + +'argv[]' is a 'NULL' terminated array of strings starting with the program name +which is followed by optional arguments. + +'TST_CMD_PASS_RETVAL' enum 'tst_cmd_flags' makes 'tst_cmd()' +return the program exit code to the caller, otherwise 'tst_cmd()' exit the +tests on failure. 'TST_CMD_TCONF_ON_MISSING' check for program in '$PATH' and exit +with 'TCONF' if not found. + +In case that 'execvp()' has failed and the enum 'TST_CMD_PASS_RETVAL' flag was set, the +return value is '255' if 'execvp()' failed with 'ENOENT' and '254' otherwise. + +'stdout_path' and 'stderr_path' determine where to redirect the program +stdout and stderr I/O streams. + +The 'SAFE_CMD()' macro can be used automatic handling non-zero exits (exits +with 'TBROK') and 'ENOENT' (exits with 'TCONF'). + +.Example +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +const char *const cmd[] = { "ls", "-l", NULL }; + +... + /* Store output of 'ls -l' into log.txt */ + tst_cmd(cmd, "log.txt", NULL, 0); +... +------------------------------------------------------------------------------- + +1.21 Measuring elapsed time and helper functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +[source,c] +------------------------------------------------------------------------------- +#include "tst_timer.h" + +void tst_timer_check(clockid_t clk_id); + +void tst_timer_start(clockid_t clk_id); + +void tst_timer_stop(void); + +struct timespec tst_timer_elapsed(void); + +long long tst_timer_elapsed_ms(void); + +long long tst_timer_elapsed_us(void); + +int tst_timer_expired_ms(long long ms); +------------------------------------------------------------------------------- + +The 'tst_timer_check()' function checks if specified 'clk_id' is suppored and +exits the test with 'TCONF' otherwise. It's expected to be used in test +'setup()' before any resources that needs to be cleaned up are initialized, +hence it does not include a cleanup function parameter. + +The 'tst_timer_start()' marks start time and stores the 'clk_id' for further +use. + +The 'tst_timer_stop()' marks the stop time using the same 'clk_id' as last +call to 'tst_timer_start()'. + +The 'tst_timer_elapsed*()' returns time difference between the timer start and +last timer stop in several formats and units. + +The 'tst_timer_expired_ms()' function checks if the timer started by +'tst_timer_start()' has been running longer than ms milliseconds. The function +returns non-zero if timer has expired and zero otherwise. + +IMPORTANT: The timer functions use 'clock_gettime()' internally which needs to + be linked with '-lrt' on older glibc. Please do not forget to add + 'LDLIBS+=-lrt' in Makefile. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" +#include "tst_timer.h" + +static void setup(void) +{ + ... + tst_timer_check(CLOCK_MONOTONIC); + ... +} + +static void run(void) +{ + ... + tst_timer_start(CLOCK_MONOTONIC); + ... + while (!tst_timer_expired_ms(5000)) { + ... + } + ... +} + +struct tst_test test = { + ... + .setup = setup, + .test_all = run, + ... +}; +------------------------------------------------------------------------------- + +Expiration timer example usage. + +[source,c] +------------------------------------------------------------------------------- +long long tst_timespec_to_us(struct timespec t); +long long tst_timespec_to_ms(struct timespec t); + +struct timeval tst_us_to_timeval(long long us); +struct timeval tst_ms_to_timeval(long long ms); + +int tst_timespec_lt(struct timespec t1, struct timespec t2); + +struct timespec tst_timespec_add_us(struct timespec t, long long us); + +struct timespec tst_timespec_diff(struct timespec t1, struct timespec t2); +long long tst_timespec_diff_us(struct timespec t1, struct timespec t2); +long long tst_timespec_diff_ms(struct timespec t1, struct timespec t2); + +struct timespec tst_timespec_abs_diff(struct timespec t1, struct timespec t2); +long long tst_timespec_abs_diff_us(struct timespec t1, struct timespec t2); +long long tst_timespec_abs_diff_ms(struct timespec t1, struct timespec t2); +------------------------------------------------------------------------------- + +The first four functions are simple inline conversion functions. + +The 'tst_timespec_lt()' function returns non-zero if 't1' is earlier than +'t2'. + +The 'tst_timespec_add_us()' function adds 'us' microseconds to the timespec +'t'. The 'us' is expected to be positive. + +The 'tst_timespec_diff*()' functions returns difference between two times, the +'t1' is expected to be later than 't2'. + +The 'tst_timespec_abs_diff*()' functions returns absolute value of difference +between two times. + +NOTE: All conversions to ms and us rounds the value. + +1.22 Datafiles +~~~~~~~~~~~~~~ + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static const char *const res_files[] = { + "foo", + "bar", + NULL +}; + +static struct tst_test test = { + ... + .resource_files = res_files, + ... +} +------------------------------------------------------------------------------- + +If the test needs additional files to be copied to the test temporary +directory all you need to do is to list their filenames in the +'NULL' terminated array '.resource_files' in the tst_test structure. + +When resource files is set test temporary directory is created automatically, +there is need to set '.needs_tmpdir' as well. + +The test library looks for datafiles first, these are either stored in a +directory called +datafiles+ in the +$PWD+ at the start of the test or in ++$LTPROOT/testcases/data/${test_binary_name}+. If the file is not found the +library looks into +$LTPROOT/testcases/bin/+ and to +$PWD+ at the start of the +test. This ensures that the testcases can copy the file(s) effortlessly both +when test is started from the directory it was compiled in as well as when LTP +was installed. + +The file(s) are copied to the newly created test temporary directory which is +set as the test working directory when the 'test()' functions is executed. + +1.23 Code path tracing +~~~~~~~~~~~~~~~~~~~~~~ + +'tst_res' is a macro, so on when you define a function in one file: + +[source,c] +------------------------------------------------------------------------------- +int do_action(int arg) +{ + ... + + if (ok) { + tst_res(TPASS, "check passed"); + return 0; + } else { + tst_res(TFAIL, "check failed"); + return -1; + } +} +------------------------------------------------------------------------------- + +and call it from another file, the file and line reported by 'tst_res' in this +function will be from the former file. + +'TST_TRACE' can make the analysis of such situations easier. It's a macro which +inserts a call to 'tst_res(TINFO, ...)' in case its argument evaluates to +non-zero. In this call to 'tst_res(TINFO, ...)' the file and line will be +expanded using the actual location of 'TST_TRACE'. + +For example, if this another file contains: + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +if (TST_TRACE(do_action(arg))) { + ... +} +------------------------------------------------------------------------------- + +the generated output may look similar to: + +------------------------------------------------------------------------------- +common.h:9: FAIL: check failed +test.c:8: INFO: do_action(arg) failed +------------------------------------------------------------------------------- + +1.24 Tainted kernels +~~~~~~~~~~~~~~~~~~~~ + +If you need to detect whether a testcase triggers a kernel warning, bug or +oops, the following can be used to detect TAINT_W or TAINT_D: + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static struct tst_test test = { + ... + .taint_check = TST_TAINT_W | TST_TAINT_D, + ... +}; + +void run(void) +{ + ... + if (tst_taint_check() != 0) + tst_res(TFAIL, "kernel has issues"); + else + tst_res(TPASS, "kernel seems to be fine"); +} +------------------------------------------------------------------------------- + +To initialize taint checks, you have to set the taint flags you want to test +for in the 'taint_check' attribute of the tst_test struct. LTP library will +then automatically call 'tst_taint_init()' during test setup. The function +will generate a 'TCONF' if the requested flags are not fully supported on the +running kernel, and 'TBROK' if the kernel is already tainted before executing +the test. + +LTP library will then automatically check kernel taint at the end of testing. +If '.all_filesystems' is set in struct tst_test, taint check will be performed +after each file system and taint will abort testing early with 'TFAIL'. You +can optionally also call 'tst_taint_check()' during 'run()', which returns 0 +or the tainted flags set in '/proc/sys/kernel/tainted' as specified earlier. + +Depending on your kernel version, not all tainted-flags will be supported. + +For reference to tainted kernels, see kernel documentation: +Documentation/admin-guide/tainted-kernels.rst or +https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html + +1.25 Checksums +~~~~~~~~~~~~~~ + +CRC32c checksum generation is supported by LTP. In order to use it, the +test should include 'tst_checksum.h' header, then can call 'tst_crc32c()'. + +1.26 Checking kernel for the driver support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some tests may need specific kernel drivers, either compiled in, or built +as a module. If '.needs_drivers' points to a 'NULL' terminated array of kernel +module names these are all checked and the test exits with 'TCONF' on the +first missing driver. + +Since it relies on modprobe command, the check will be skipped if the command +itself is not available on the system. + +1.27 Saving & restoring /proc|sys values +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +LTP library can be instructed to save and restore value of specified +(/proc|sys) files. This is achieved by initialized tst_test struct +field 'save_restore'. It is a 'NULL' terminated array of strings where +each string represents a file, whose value is saved at the beginning +and restored at the end of the test. Only first line of a specified +file is saved and restored. + +Pathnames can be optionally prefixed to specify how strictly (during +'store') are handled errors: + +* (no prefix) - test ends with 'TCONF', if file doesn't exist +* '?' - test prints info message and continues, + if file doesn't exist or open/read fails +* '!' - test ends with 'TBROK', if file doesn't exist + +'restore' is always strict and will TWARN if it encounters any error. + +[source,c] +------------------------------------------------------------------------------- +static const char *save_restore[] = { + "/proc/sys/kernel/core_pattern", + NULL, +}; + +static void setup(void) +{ + FILE_PRINTF("/proc/sys/kernel/core_pattern", "/mypath"); +} + +static struct tst_test test = { + ... + .setup = setup, + .save_restore = save_restore, +}; +------------------------------------------------------------------------------- + +1.28 Parsing kernel .config +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Generally testcases should attempt to autodetect as much kernel features as +possible based on the currently running kernel. We do have tst_check_driver() +to check if functionality that could be compiled as kernel module is present +on the system, disabled syscalls can be detected by checking for 'ENOSYS' +errno etc. + +However in rare cases core kernel features couldn't be detected based on the +kernel userspace API and we have to resort to parse the kernel .config. + +For this cases the test should set the 'NULL' terminated '.needs_kconfigs' +array of boolean expressions with constraints on the kconfig variables. The +boolean expression consits of variables, two binary operations '&' and '|', +negation '!' and correct sequence of parentesis '()'. Variables are expected +to be in a form of "CONFIG_FOO[=bar]". + +The test will continue to run if all expressions are evaluated to 'True'. +Missing variable is mapped to 'False' as well as variable with different than +specified value, e.g. 'CONFIG_FOO=bar' will evaluate to 'False' if the value +is anything else but 'bar'. If config variable is specified as plain +'CONFIG_FOO' it's evaluated to true it's set to any value (typically =y or =m). + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static const char *kconfigs[] = { + "CONFIG_X86_INTEL_UMIP | CONFIG_X86_UMIP", + NULL +}; + +static struct tst_test test = { + ... + .needs_kconfigs = kconfigs, + ... +}; +------------------------------------------------------------------------------- + +1.29 Changing the Wall Clock Time during test execution +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are some tests that, for different reasons, might need to change the +system-wide clock time. Whenever this happens, it is imperative that the clock +is restored, at the end of test's execution, taking in consideration the amount +of time elapsed during that test. + +In order for that to happen, struct tst_test has a variable called +"restore_wallclock" that should be set to "1" so LTP knows it should: (1) +initialize a monotonic clock during test setup phase and (2) use that monotonic +clock to fix the system-wide clock time at the test cleanup phase. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static void setup(void) +{ + ... +} + +static void run(void) +{ + ... +} + +struct tst_test test = { + ... + .setup = setup, + .test_all = run, + .restore_wallclock = 1, + ... +}; +------------------------------------------------------------------------------- + +1.30 Testing similar syscalls in one test +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In some cases kernel has several very similar syscalls that do either the same +or very similar job. This is most noticeable on i386 where we commonly have +two or three syscall versions. That is because i386 was first platform that +Linux was developed on and because of that most mistakes in API happened there +as well. However this is not limited to i386 at all, it's quite common that +version two syscall has added missing flags parameters or so. + +In such cases it does not make much sense to copy&paste the test code over and +over, rather than that the test library provides support for test variants. +The idea behind test variants is simple, we run the test several times each +time with different syscall variant. + +The implementation consist of test_variants integer that, if set, denotes number +of test variants. The test is then forked and executed test_variants times each +time with different value in global tst_variant variable. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static int do_foo(void) +{ + switch (tst_variant) { + case 0: + return foo(); + case 1: + return syscall(__NR_foo); + } + + return -1; +} + +static void run(void) +{ + ... + + TEST(do_foo); + + ... +} + +static void setup(void) +{ + switch (tst_variant) { + case 0: + tst_res(TINFO, "Testing foo variant 1"); + break; + case 1: + tst_res(TINFO, "Testing foo variant 2"); + break; + } +} + +struct tst_test test = { + ... + .setup = setup, + .test_all = run, + .test_variants = 2, + ... +}; +------------------------------------------------------------------------------- + +1.31 Guarded buffers +~~~~~~~~~~~~~~~~~~~~ + +The test library supports guarded buffers, which are buffers allocated so +that: + +* The end of the buffer is followed by a PROT_NONE page + +* The remainder of the page before the buffer is filled with random canary + data + +Which means that the any access after the buffer will yield a Segmentation +fault or EFAULT depending on if the access happened in userspace or the kernel +respectively. The canary before the buffer will also catch any write access +outside of the buffer. + +The purpose of the patch is to catch off-by-one bugs which happens when +buffers and structures are passed to syscalls. New tests should allocate +guarded buffers for all data passed to the tested syscall which are passed by +a pointer. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static struct foo *foo_ptr; +static struct iovec *iov; +static void *buf_ptr; +static char *id; +... + +static void run(void) +{ + ... + + foo_ptr->bar = 1; + foo_ptr->buf = buf_ptr; + + ... +} + +static void setup(void) +{ + ... + + id = tst_strdup(string); + + ... +} + +static struct tst_test test = { + ... + .bufs = (struct tst_buffers []) { + {&foo_ptr, .size = sizeof(*foo_ptr)}, + {&buf_ptr, .size = BUF_SIZE}, + {&iov, .iov_sizes = (int[]){128, 32, -1}, + {} + } +}; +------------------------------------------------------------------------------- + +Guarded buffers can be allocated on runtime in a test setup() by a +'tst_alloc()' or by 'tst_strdup()' as well as by filling up the .bufs array in +the tst_test structure. + +So far the tst_test structure supports allocating either a plain buffer by +setting up the size or struct iovec, which is allocated recursively including +the individual buffers as described by an '-1' terminated array of buffer +sizes. + +1.32 Adding and removing capabilities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some tests may require the presence or absence of particular +capabilities. Using the API provided by 'tst_capability.h' the test author can +try to ensure that some capabilities are either present or absent during the +test. + +For example; below we try to create a raw socket, which requires +CAP_NET_ADMIN. During setup we should be able to do it, then during run it +should be impossible. The LTP capability library will check before setup that +we have this capability, then after setup it will drop it. + +[source,c] +-------------------------------------------------------------------------------- +#include "tst_test.h" +#include "tst_capability.h" +#include "tst_safe_net.h" + +#include "lapi/socket.h" + +static void run(void) +{ + TEST(socket(AF_INET, SOCK_RAW, 1)); + if (TST_RET > -1) { + tst_res(TFAIL, "Created raw socket"); + } else if (TST_ERR != EPERM) { + tst_res(TFAIL | TTERRNO, + "Failed to create socket for wrong reason"); + } else { + tst_res(TPASS | TTERRNO, "Didn't create raw socket"); + } +} + +static void setup(void) +{ + TEST(socket(AF_INET, SOCK_RAW, 1)); + if (TST_RET < 0) + tst_brk(TCONF | TTERRNO, "We don't have CAP_NET_RAW to begin with"); + + SAFE_CLOSE(TST_RET); +} + +static struct tst_test test = { + .setup = setup, + .test_all = run, + .caps = (struct tst_cap []) { + TST_CAP(TST_CAP_REQ, CAP_NET_RAW), + TST_CAP(TST_CAP_DROP, CAP_NET_RAW), + {} + }, +}; +-------------------------------------------------------------------------------- + +Look at the test struct at the bottom. We have filled in the 'caps' field with +a 'NULL' terminated array containing two 'tst_cap' structs. 'TST_CAP_REQ' +actions are executed before setup and 'TST_CAP_DROP' are executed after +setup. This means it is possible to both request and drop a capability. + +[source,c] +-------------------------------------------------------------------------------- +static struct tst_test test = { + .test_all = run, + .caps = (struct tst_cap []) { + TST_CAP(TST_CAP_REQ, CAP_NET_RAW), + TST_CAP(TST_CAP_DROP, CAP_SYS_ADMIN), + {} + }, +}; +-------------------------------------------------------------------------------- + +Here we request 'CAP_NET_RAW', but drop 'CAP_SYS_ADMIN'. If the capability is +in the permitted set, but not the effective set, the library will try to +permit it. If it is not in the permitted set, then it will fail with 'TCONF'. + +This API does not require 'libcap' to be installed. However it has limited +features relative to 'libcap'. It only tries to add or remove capabilities +from the effective set. This means that tests which need to spawn child +processes may have difficulties ensuring the correct capabilities are +available to the children (see the capabilities (7) manual pages). + +However a lot of problems can be solved by using 'tst_cap_action(struct +tst_cap *cap)' directly which can be called at any time. This also helps if +you wish to drop a capability at the begining of setup. + +1.33 Reproducing race-conditions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If a bug is caused by two tasks in the kernel racing and you wish to create a +regression test (or bug-fix validation test) then the 'tst_fuzzy_sync.h' +library should be used. + +It allows you to specify, in your code, two race windows. One window in each +thread's loop (triggering a race usually requires many iterations). These +windows show fuzzy-sync where the race can happen. They don't need to be +exact, hence the 'fuzzy' part. If the race condition is not immediately +triggered then the library will begin experimenting with different timings. + +[source,c] +-------------------------------------------------------------------------------- +#include "tst_fuzzy_sync.h" + +static struct tst_fzsync_pair fzsync_pair; + +static void setup(void) +{ + tst_fzsync_pair_init(&fzsync_pair); +} + +static void cleanup(void) +{ + tst_fzsync_pair_cleanup(&fzsync_pair); +} + +static void *thread_b(void *arg) +{ + while (tst_fzsync_run_b(&fzsync_pair)) { + + tst_fzsync_start_race_b(&fzsync_pair); + + /* This is the race window for thread B */ + + tst_fzsync_end_race_b(&fzsync_pair); + } + + return arg; +} + +static void thread_a(void) +{ + tst_fzsync_pair_reset(&fzsync_pair, thread_b); + + while (tst_fzsync_run_a(&fzsync_pair)) { + + tst_fzsync_start_race_a(&fzsync_pair); + + /* This is the race window for thread A */ + + tst_fzsync_end_race_a(&fzsync_pair); + } +} + +static struct tst_test test = { + .test_all = thread_a, + .setup = setup, + .cleanup = cleanup, +}; +-------------------------------------------------------------------------------- + +Above is a minimal template for a test using fuzzy-sync. In a simple case, you +just need to put the bits you want to race inbetween 'start_race' and +'end_race'. Meanwhile, any setup you need to do per-iteration goes outside the +windows. + +Fuzzy sync synchronises 'run_a' and 'run_b', which act as barriers, so that +neither thread can progress until the other has caught up with it. There is +also the 'pair_wait' function which can be used to add barriers in other +locations. Of course 'start/end_race_a/b' are also a barriers. + +The library decides how long the test should run for based on the timeout +specified by the user plus some other heuristics. + +For full documentation see the comments in 'include/tst_fuzzy_sync.h'. + +1.34 Reserving hugepages +~~~~~~~~~~~~~~~~~~~~~~~~ + +Many of the LTP tests need to use hugepage in their testing, this allows the +test can reserve hugepages from system only via '.request_hugepages = xx'. + +If set non-zero number of 'request_hugepages', test will try to reserve the +expected number of hugepage for testing in setup phase. If system does not +have enough hpage for using, it will try the best to reserve 80% available +number of hpages. With success test stores the reserved hugepage number in +'tst_hugepages'. For the system without hugetlb supporting, variable +'tst_hugepages' will be set to 0. + +Also, we do cleanup and restore work for the hpages resetting automatically. + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static void run(void) +{ + ... + + if (tst_hugepages == test.request_hugepages) + TEST(do_hpage_test); + else + ... + ... +} + +struct tst_test test = { + .test_all = run, + .request_hugepages = 2, + ... +}; +------------------------------------------------------------------------------- + +or, + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" + +static void run(void) +{ + ... +} + +static void setup(void) +{ +? ? ? ? if (tst_hugepages != test.requested_hugepages) +? ? ? ? ? ? ? ? tst_brk(TCONF, "..."); +} + +struct tst_test test = { + .test_all = run, + .request_hugepages = 2, + ... +}; +------------------------------------------------------------------------------- + +1.35 Checking for required commands +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Required commands can be checked with '.needs_cmds', which points to a 'NULL' +terminated array of strings such as: + +[source,c] +------------------------------------------------------------------------------- +.needs_cmds = (const char *const []) { + "useradd", + "userdel", + NULL +}, +------------------------------------------------------------------------------- + +1.36 Assert sys or proc file value +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Using TST_ASSERT_INT/STR(path, val) to assert that integer value or string stored in +the prefix field of file pointed by path equals to the value passed to this function. + +Also having a similar api pair TST_ASSERT_FILE_INT/STR(path, prefix, val) to assert +the field value of file. + +1.36 Using Control Group +~~~~~~~~~~~~~~~~~~~~~~~~ + +Some LTP tests need specific Control Group configurations. tst_cgroup.h provides +APIs to discover and use CGroups. There are many differences between CGroups API +V1 and V2. We encapsulate the details of configuring CGroups in high-level +functions which follow the V2 kernel API. Allowing one to use CGroups without +caring too much about the current system's configuration. + +Also, the LTP library will automatically mount/umount and configure the CGroup +hierarchies if that is required (e.g. if you run the tests from init with no +system manager). + +[source,c] +------------------------------------------------------------------------------- +#include "tst_test.h" +#include "tst_cgroup.h" + +static const struct tst_cgroup_group *cg; + +static void run(void) +{ + ... + // do test under cgroup + ... +} + +static void setup(void) +{ + tst_cgroup_require("memory", NULL); + cg = tst_cgroup_get_test_group(); + SAFE_CGROUP_PRINTF(cg, "cgroup.procs", "%d", getpid()); + SAFE_CGROUP_PRINTF(cg, "memory.max", "%lu", MEMSIZE); + if (SAFE_CGROUP_HAS(cg, "memory.swap.max")) + SAFE_CGROUP_PRINTF(cg, "memory.swap.max", "%zu", memsw); +} + +static void cleanup(void) +{ + tst_cgroup_cleanup(); +} + +struct tst_test test = { + .setup = setup, + .test_all = run, + .cleanup = cleanup, + ... +}; +------------------------------------------------------------------------------- + +Above, we first ensure the memory controller is available on the +test's CGroup with 'tst_cgroup_require'. We then get a structure, +'cg', which represents the test's CGroup. Note that +'tst_cgroup_get_test_group' should not be called many times, as it is +allocated in a guarded buffer (See section 2.2.31). Therefor it is +best to call it once in 'setup' and not 'run' because 'run' may be +repeated with the '-i' option. + +We then write the current processes PID into 'cgroup.procs', which +moves the current process into the test's CGroup. After which we set +the maximum memory size by writing to 'memory.max'. If the memory +controller is mounted on CGroups V1 then the library will actually +write to 'memory.limit_in_bytes'. As a general rule, if a file exists +on both CGroup versions, then we use the V2 naming. + +Some controller features, such as 'memory.swap', can be +disabled. Therefor we need to check if they exist before accessing +them. This can be done with 'SAFE_CGROUP_HAS' which can be called on +any control file or feature. + +Most tests only require setting a few limits similar to the above. In +such cases the differences between V1 and V2 are hidden. Setup and +cleanup is also mostly hidden. However things can get much worse. + +[source,c] +------------------------------------------------------------------------------- +static const struct tst_cgroup_group *cg; +static const struct tst_cgroup_group *cg_drain; +static struct tst_cgroup_group *cg_child; + +static void run(void) +{ + char buf[BUFSIZ]; + size_t mem = 0; + + cg_child = tst_cgroup_group_mk(cg, "child"); + SAFE_CGROUP_PRINTF(cg_child, "cgroup.procs", "%d", getpid()); + + if (TST_CGROUP_VER(cg, "memory") != TST_CGROUP_V1) + SAFE_CGROUP_PRINT(cg, "cgroup.subtree_control", "+memory"); + if (TST_CGROUP_VER(cg, "cpuset") != TST_CGROUP_V1) + SAFE_CGROUP_PRINT(cg, "cgroup.subtree_control", "+cpuset"); + + if (!SAFE_FORK()) { + SAFE_CGROUP_PRINTF(cg_child, "cgroup.procs", "%d", getpid()); + + if (SAFE_CGROUP_HAS(cg_child, "memory.swap")) + SAFE_CGROUP_SCANF(cg_child, "memory.swap.current", "%zu", &mem); + SAFE_CGROUP_READ(cg_child, "cpuset.mems", buf, sizeof(buf)); + + // Do something with cpuset.mems and memory.current values + ... + + exit(0); + } + + tst_reap_children(); + SAFE_CGROUP_PRINTF(cg_drain, "cgroup.procs", "%d", getpid()); + cg_child = tst_cgroup_group_rm(cg_child); +} + +static void setup(void) +{ + tst_cgroup_require("memory", NULL); + tst_cgroup_require("cpuset", NULL); + + cg = tst_cgroup_get_test_group(); + cg_drain = tst_cgroup_get_drain_group(); +} + +static void cleanup(void) +{ + if (cg_child) { + SAFE_CGROUP_PRINTF(cg_drain, "cgroup.procs", "%d", getpid()); + cg_child = tst_cgroup_group_rm(cg_child); + } + + tst_cgroup_cleanup(); +} + +struct tst_test test = { + .setup = setup, + .test_all = run, + .cleanup = cleanup, + ... +}; +------------------------------------------------------------------------------- + +Starting with setup; we can see here that we also fetch the 'drain' +CGroup. This is a shared group (between parallel tests) which may +contain processes from other tests. It should have default settings and +these should not be changed by the test. It can be used to remove +processes from other CGroups incase the hierarchy root is not +accessible. + +In 'run', we first create a child CGroup with 'tst_cgroup_mk'. As we +create this CGroup in 'run' we should also remove it at the end of +run. We also need to check if it exists and remove it in cleanup as +well. Because there are 'SAFE_' functions which may jump to cleanup. + +We then move the main test process into the child CGroup. This is +important as it means that before we destroy the child CGroup we have +to move the main test process elsewhere. For that we use the 'drain' +group. + +Next we enable the memory and cpuset controller configuration on the +test CGroup's descendants (i.e. 'cg_child'). This allows each child to +have its own settings. The file 'cgroup.subtree_control' does not +exist on V1. Because it is possible to have both V1 and V2 active at +the same time. We can not simply check if 'subtree_control' exists +before writing to it. We have to check if a particular controller is +on V2 before trying to add it to 'subtree_control'. Trying to add a V1 +controller will result in 'ENOENT'. + +We then fork a child process and add this to the child CGroup. Within +the child process we try to read 'memory.swap.current'. It is possible +that the memory controller was compiled without swap support, so it is +necessary to check if 'memory.swap' is enabled. That is unless the +test will never reach the point where 'memory.swap.*' are used without +swap support. + +The parent process waits for the child process to be reaped before +destroying the child CGroup. So there is no need to transfer the child +to drain. However the parent process must be moved otherwise we will +get 'EBUSY' when trying to remove the child CGroup. + +Another example of an edge case is the following. + +[source,c] +------------------------------------------------------------------------------- + if (TST_CGROUP_VER(cg, "memory") == TST_CGROUP_V1) + SAFE_CGROUP_PRINTF(cg, "memory.swap.max", "%lu", ~0UL); + else + SAFE_CGROUP_PRINT(cg, "memory.swap.max", "max"); +------------------------------------------------------------------------------- + +CGroups V2 introduced a feature where 'memory[.swap].max' could be set to +"max". This does not appear to work on V1 'limit_in_bytes' however. For most +tests, simply using a large number is sufficient and there is no need to use +"max". Importantly though, one should be careful to read both the V1 and V2 +kernel docs. The LTP library can not handle all edge cases. It does the minimal +amount of work to make testing on both V1 and V2 feasible. + +1.37 Require minimum numbers of CPU for a testcase +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some tests require more than specific number of CPU. It can be defined with +`.min_cpus = N`. + +1.38 Test tags +~~~~~~~~~~~~~~ + +Test tags are name-value pairs that can hold any test metadata. + +We have additional support for CVE entries, git commit in mainline kernel, +stable kernel or glibc git repository. If a test is a regression test it +should include these tags. They are printed when test fails and exported +into documentation. + +CVE, mainline and stable kernel git commits in a regression test for a kernel bug: +[source,c] +------------------------------------------------------------------------------- +struct tst_test test = { + ... + .tags = (const struct tst_tag[]) { + {"linux-git", "9392a27d88b9"}, + {"linux-git", "ff002b30181d"}, + {"linux-stable-git", "c4a23c852e80"}, + {"CVE", "2020-29373"}, + {} + } +}; +------------------------------------------------------------------------------- + +Glibc git commit in a regression test for a glibc bug: +[source,c] +------------------------------------------------------------------------------- +struct tst_test test = { + ... + .tags = (const struct tst_tag[]) { + {"glibc-git", "574500a108be"}, + {} + } +}; +------------------------------------------------------------------------------- + +2. Common problems +------------------ + +This chapter describes common problems/misuses and less obvious design patters +(quirks) in UNIX interfaces. Read it carefully :) + +2.1 umask() +~~~~~~~~~~~ + +I've been hit by this one several times already... When you create files +with 'open()' or 'creat()' etc, the mode specified as the last parameter *is +not* the mode the file is created with. The mode depends on current 'umask()' +settings which may clear some of the bits. If your test depends on specific +file permissions you need either to change umask to 0 or 'chmod()' the file +afterwards or use 'SAFE_TOUCH()' that does the 'chmod()' for you. + +2.2 access() +~~~~~~~~~~~~ + +If 'access(some_file, W_OK)' is executed by root, it will return success even +if the file doesn't have write permission bits set (the same holds for R_OK +too). For sysfs files you can use 'open()' as a workaround to check file +read/write permissions. It might not work for other filesystems, for these you +have to use 'stat()', 'lstat()' or 'fstat()'. + +2.3 umount() EBUSY +~~~~~~~~~~~~~~~~~~ + +Various desktop daemons (gvfsd-trash is known for that) may be stupid enough +to probe all newly mounted filesystem which results in 'umount(2)' failing +with 'EBUSY'; use 'tst_umount()' described in 1.19 that retries in this case +instead of plain 'umount(2)'. + +2.4 FILE buffers and fork() +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Be vary that if a process calls 'fork(2)' the child process inherits open +descriptors as well as copy of the parent memory so especially if there are +any open 'FILE' buffers with a data in them they may be written both by the +parent and children resulting in corrupted/duplicated data in the resulting +files. + +Also open 'FILE' streams are flushed and closed at 'exit(3)' so if your +program works with 'FILE' streams, does 'fork(2)', and the child may end up +calling 'exit(3)' you will likely end up with corrupted files. + +The solution to this problem is either simply call 'fflush(NULL)' that flushes +all open output 'FILE' streams just before doing 'fork(2)'. You may also use +'_exit(2)' in child processes which does not flush 'FILE' buffers and also +skips 'atexit(3)' callbacks. diff --git a/doc/shell-test-api.txt b/doc/shell-test-api.txt new file mode 100644 index 000000000..21fefc8e0 --- /dev/null +++ b/doc/shell-test-api.txt @@ -0,0 +1,740 @@ +LTP Shell Test API +================== + +1 Writing a testcase in shell +----------------------------- + +LTP supports testcases to be written in a portable shell too. + +There is a shell library modeled closely to the C interface at +'testcases/lib/tst_test.sh'. + +WARNING: All identifiers starting with 'TST_' or 'tst_' are reserved for the + test library. + +1.1 Basic test interface +~~~~~~~~~~~~~~~~~~~~~~~~ + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# This is a basic test for true shell builtin + +TST_TESTFUNC=do_test +. tst_test.sh + +do_test() +{ + true + ret=$? + + if [ $ret -eq 0 ]; then + tst_res TPASS "true returned 0" + else + tst_res TFAIL "true returned $ret" + fi +} + +tst_run +------------------------------------------------------------------------------- + +TIP: To execute this test the 'tst_test.sh' library must be in '$PATH'. If you + are executing the test from a git checkout you can run it as + 'PATH="$PATH:../../lib" ./foo01.sh' + +The shell library expects test setup, cleanup and the test function executing +the test in the '$TST_SETUP', '$TST_CLEANUP' and '$TST_TESTFUNC' variables. + +Both '$TST_SETUP' and '$TST_CLEANUP' are optional. + +The '$TST_TESTFUNC' may be called several times if more than one test +iteration was requested by passing right command line options to the test. + +The '$TST_CLEANUP' may be called even in the middle of the setup and must be +able to clean up correctly even in this situation. The easiest solution for +this is to keep track of what was initialized and act accordingly in the +cleanup. + +WARNING: Similar to the C library, calling 'tst_brk' in the $TST_CLEANUP does + not exit the test and 'TBROK' is converted to 'TWARN'. + +Notice also the 'tst_run' shell API function called@the end of the test that +actually starts the test. + +WARNING: cleanup function is called only after 'tst_run' has been started. +Calling 'tst_brk' in shell libraries, e.g. 'tst_test.sh' or 'tst_net.sh' does +not trigger calling it. + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Example test with tests in separate functions + +TST_TESTFUNC=test +TST_CNT=2 +. tst_test.sh + +test1() +{ + tst_res TPASS "Test $1 passed" +} + +test2() +{ + tst_res TPASS "Test $1 passed" +} + +tst_run +# output: +# foo 1 TPASS: Test 1 passed +# foo 2 TPASS: Test 2 passed +------------------------------------------------------------------------------- + +If '$TST_CNT' is set, the test library looks if there are functions named +'$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are +found they are executed one by one. The test number is passed to it in the '$1'. + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Example test with tests in a single function + +TST_TESTFUNC=do_test +TST_CNT=2 +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed";; + 2) tst_res TPASS "Test $1 passed";; + esac +} + +tst_run +# output: +# foo 1 TPASS: Test 1 passed +# foo 2 TPASS: Test 2 passed +------------------------------------------------------------------------------- + +Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., +the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed +to it in the '$1'. + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Example test with tests in a single function, using $TST_TEST_DATA and +# $TST_TEST_DATA_IFS + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +TST_TEST_DATA_IFS=":" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# foo 1 TPASS: Test 1 passed with data 'foo' +# foo 2 TPASS: Test 1 passed with data 'bar' +# foo 3 TPASS: Test 1 passed with data 'd dd' +------------------------------------------------------------------------------- + +It's possible to pass data for function with '$TST_TEST_DATA'. Optional +'$TST_TEST_DATA_IFS' is used for splitting, default value is space. + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT + +TST_TESTFUNC=do_test +TST_CNT=2 +TST_TEST_DATA="foo bar" +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with data '$2'";; + 2) tst_res TPASS "Test $1 passed with data '$2'";; + esac +} + +tst_run +# output: +# foo 1 TPASS: Test 1 passed with data 'foo' +# foo 2 TPASS: Test 2 passed with data 'foo' +# foo 3 TPASS: Test 1 passed with data 'bar' +# foo 4 TPASS: Test 2 passed with data 'bar' +------------------------------------------------------------------------------- + +'$TST_TEST_DATA' can be used with '$TST_CNT'. If '$TST_TEST_DATA_IFS' not specified, +space as default value is used. Of course, it's possible to use separate functions. + +1.2 Library environment variables and functions for shell +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Similarily to the C library various checks and preparations can be requested +simply by setting right '$TST_NEEDS_FOO'. + +[options="header"] +|============================================================================= +| Variable name | Action done +| 'TST_NEEDS_ROOT' | Exit the test with 'TCONF' unless executed under root. +| | Alternatively the 'tst_require_root' command can be used. +| 'TST_NEEDS_TMPDIR' | Create test temporary directory and cd into it. +| 'TST_NEEDS_DEVICE' | Prepare test temporary device, the path to testing + device is stored in '$TST_DEVICE' variable. + The option implies 'TST_NEEDS_TMPDIR'. +| 'TST_NEEDS_CMDS' | String with command names that has to be present for + the test (see below). +| 'TST_NEEDS_MODULE' | Test module name needed for the test (see below). +| 'TST_NEEDS_DRIVERS'| Checks kernel drivers support for the test. +| 'TST_TIMEOUT' | Maximum timeout set for the test in sec. Must be int >= 1, + or -1 (special value to disable timeout), default is 300. + Variable is meant be set in tests, not by user. + It's an equivalent of `tst_test.timeout` in C, can be set + via 'tst_set_timeout(timeout)' after test has started. +|============================================================================= + +[options="header"] +|============================================================================= +| Function name | Action done +| 'tst_set_timeout(timeout)' | Maximum timeout set for the test in sec. + See 'TST_TIMEOUT' variable. +|============================================================================= + +NOTE: Network tests (see testcases/network/README.md) use additional variables +and functions in 'tst_net.sh'. + +Checking for presence of commands ++++++++++++++++++++++++++++++++++ + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh + +... + +TST_NEEDS_CMDS="modinfo modprobe" +. tst_test.sh + +... + +------------------------------------------------------------------------------- + +Setting '$TST_NEEDS_CMDS' to a string listing required commands will check for +existence each of them and exits the test with 'TCONF' on first missing. + +Alternatively the 'tst_require_cmds()' function can be used to do the same on +runtime, since sometimes we need to the check at runtime too. + +'tst_check_cmds()' can be used for requirements just for a particular test +as it doesn't exit (it issues 'tst_res TCONF'). Expected usage is: + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh + +TST_TESTFUNC=do_test +. tst_test.sh + +do_test() +{ + tst_check_cmds cmd || return + cmd --foo + ... +} + +tst_run +------------------------------------------------------------------------------- + +Locating kernel modules ++++++++++++++++++++++++ + +The LTP build system can build kernel modules as well, setting +'$TST_NEEDS_MODULE' to module name will cause the library to look for the +module in a few possible paths. + +If module was found the path to it will be stored into '$TST_MODPATH' +variable, if module wasn't found the test will exit with 'TCONF'. + +Alternatively the 'tst_require_module()' function can be used to do the same +at runtime. + +1.3 Optional command line parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Optional test command line parameters + +TST_OPTS="af:" +TST_USAGE=usage +TST_PARSE_ARGS=parse_args +TST_TESTFUNC=do_test + +. tst_test.sh + +ALTERNATIVE=0 +MODE="foo" + +usage() +{ + cat << EOF +usage: $0 [-a] [-f ] + +OPTIONS +-a Enable support for alternative foo +-f Specify foo or bar mode +EOF +} + +parse_args() +{ + case $1 in + a) ALTERNATIVE=1;; + f) MODE="$2";; + esac +} + +do_test() +{ + ... +} + +tst_run +------------------------------------------------------------------------------- + +The 'getopts' string for optional parameters is passed in the '$TST_OPTS' +variable. There are a few default parameters that cannot be used by a test, +these can be listed with passing help '-h' option to any test. + +The function that prints the usage is passed in '$TST_USAGE', the help for +the options implemented in the library is appended when usage is printed. + +Lastly the function '$PARSE_ARGS' is called with the option name in the '$1' +and, if option has argument, its value in the '$2'. + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Optional test positional parameters + +TST_POS_ARGS=3 +TST_USAGE=usage +TST_TESTFUNC=do_test + +. tst_test.sh + +usage() +{ + cat << EOF +usage: $0 [min] [max] [size] + +EOF +} + +min="$1" +max="$2" +size="$3" + +do_test() +{ + ... +} + +tst_run +------------------------------------------------------------------------------- + +You can also request a number of positional parameters by setting the +'$TST_POS_ARGS' variable. If you do, these will be available as they were +passed directly to the script in '$1', '$2', ..., '$n'. + +1.4 Useful library functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Retrieving configuration variables +++++++++++++++++++++++++++++++++++ + +You may need to retrieve configuration values such as PAGESIZE, there is +'getconf' but as some system may not have it, you are advised to use +'tst_getconf' instead. Note that it implements subset of 'getconf' +system variables used by the testcases only. + +[source,sh] +------------------------------------------------------------------------------- +# retrieve PAGESIZE +pagesize=`tst_getconf PAGESIZE` +------------------------------------------------------------------------------- + +Sleeping for subsecond intervals +++++++++++++++++++++++++++++++++ + +Albeit there is a sleep command available basically everywhere not all +implementations can support sleeping for less than one second. And most of the +time sleeping for a second is too much. Therefore LTP includes 'tst_sleep' +that can sleep for defined amount of seconds, milliseconds or microseconds. + +[source,sh] +------------------------------------------------------------------------------- +# sleep for 100 milliseconds +tst_sleep 100ms +------------------------------------------------------------------------------- + +Retry a function call multiple times +++++++++++++++++++++++++++++++++++++ + +Sometimes an LTP test needs to retry a function call multiple times because +the system is not ready to process it successfully on the first try. The LTP +library has useful tools to handle the call retry automatically. +'TST_RETRY_FUNC()' will keep retrying for up to 1 second. If you want a custom +time limit use 'TST_RETRY_FN_EXP_BACKOFF()'. Both methods return the value +returned by the last 'FUNC' call. + +The delay between retries starts@1 microsecond and doubles after each call. +The retry loop ends when the function call succeeds or when the next delay +exceeds the specified time (1 second for 'TST_RETRY_FUNC()'). The maximum +delay is multiplied by TST_TIMEOUT_MUL. The total cumulative delay may be up +to twice as long as the adjusted maximum delay. + +The C version of 'TST_RETRY_FUNC()' is a macro which takes two arguments: + +* 'FUNC' is the complete function call with arguments which should be retried + multiple times. +* 'SUCCESS_CHECK' is a macro or function which will validate 'FUNC' return + value. 'FUNC' call was successful if 'SUCCESS_CHECK(ret)' evaluates to + non-zero. + +Both retry methods clear 'errno' before every 'FUNC' call so your +'SUCCESS_CHECK' can look for specific error codes as well. The LTP library +also includes predefined 'SUCCESS_CHECK' macros for the most common call +conventions: + +* 'TST_RETVAL_EQ0()' - The call was successful if 'FUNC' returned 0 or NULL +* 'TST_RETVAL_NOTNULL()' - The call was successful if 'FUNC' returned any + value other than 0 or NULL. +* 'TST_RETVAL_GE0()' - The call was successful if 'FUNC' returned value >= 0. + +[source,c] +------------------------------------------------------------------------------- +/* Keep trying for 1 second */ +TST_RETRY_FUNC(FUNC, SUCCESS_CHECK) + +/* Keep trying for up to 2*N seconds */ +TST_RETRY_FN_EXP_BACKOFF(FUNC, SUCCESS_CHECK, N) +------------------------------------------------------------------------------- + +The shell version of 'TST_RETRY_FUNC()' is simpler and takes slightly +different arguments: + +* 'FUNC' is a string containing the complete function or program call with + arguments. +* 'EXPECTED_RET' is a single expected return value. 'FUNC' call was successful + if the return value is equal to EXPECTED_RET. + +[source,sh] +------------------------------------------------------------------------------- +# Keep trying for 1 second +TST_RETRY_FUNC "FUNC arg1 arg2 ..." "EXPECTED_RET" + +# Keep trying for up to 2*N seconds +TST_RETRY_FN_EXP_BACKOFF "FUNC arg1 arg2 ..." "EXPECTED_RET" "N" +------------------------------------------------------------------------------- + +Checking for integers ++++++++++++++++++++++ + +[source,sh] +------------------------------------------------------------------------------- +# returns zero if passed an integer parameter, non-zero otherwise +tst_is_int "$FOO" +------------------------------------------------------------------------------- + +Checking for integers and floating point numbers +++++++++++++++++++++++++++++++++++++++++++++++++ + +[source,sh] +------------------------------------------------------------------------------- +# returns zero if passed an integer or floating point number parameter, +# non-zero otherwise +tst_is_num "$FOO" +------------------------------------------------------------------------------- + +Obtaining random numbers +++++++++++++++++++++++++ + +There is no '$RANDOM' in portable shell, use 'tst_random' instead. + +[source,sh] +------------------------------------------------------------------------------- +# get random integer between 0 and 1000 (including 0 and 1000) +tst_random 0 1000 +------------------------------------------------------------------------------- + +Formatting device with a filesystem ++++++++++++++++++++++++++++++++++++ + +The 'tst_mkfs' helper will format device with the filesystem. + +[source,sh] +------------------------------------------------------------------------------- +# format test device with ext2 +tst_mkfs ext2 $TST_DEVICE +# default params are $TST_FS_TYPE $TST_DEVICE +tst_mkfs +# optional parameters +tst_mkfs ext4 /dev/device -T largefile +------------------------------------------------------------------------------- + +Mounting and unmounting filesystems ++++++++++++++++++++++++++++++++++++ + +The 'tst_mount' and 'tst_umount' helpers are a safe way to mount/umount +a filesystem. + +The 'tst_mount' mounts '$TST_DEVICE' of '$TST_FS_TYPE' (optional) to +'$TST_MNTPOINT' (defaults to mntpoint), optionally using the +'$TST_MNT_PARAMS'. The '$TST_MNTPOINT' directory is created if it didn't +exist prior to the function call. + +If the path passed (optional, defaults to '$TST_DEVICE') to the 'tst_umount' is +not mounted (present in '/proc/mounts') it's noop. +Otherwise it retries to umount the filesystem a few times on a failure, which +is a workaround since there are a daemons dumb enough to probe all newly +mounted filesystems, which prevents them from umounting shortly after they +were mounted. + +ROD and ROD_SILENT +++++++++++++++++++ + +These functions supply the 'SAFE_MACROS' used in C although they work and are +named differently. + +[source,sh] +------------------------------------------------------------------------------- +ROD_SILENT command arg1 arg2 ... + +# is shorthand for: + +command arg1 arg2 ... > /dev/null 2>&1 +if [ $? -ne 0 ]; then + tst_brk TBROK "..." +fi + + +ROD command arg1 arg2 ... + +# is shorthand for: + +ROD arg1 arg2 ... +if [ $? -ne 0 ]; then + tst_brk TBROK "..." +fi +------------------------------------------------------------------------------- + +WARNING: Keep in mind that output redirection (to a file) happens in the + caller rather than in the ROD function and cannot be checked for + write errors by the ROD function. + +As a matter of a fact doing +ROD echo a > /proc/cpuinfo+ would work just fine +since the 'ROD' function will only get the +echo a+ part that will run just +fine. + +[source,sh] +------------------------------------------------------------------------------- +# Redirect output to a file with ROD +ROD echo foo \> bar +------------------------------------------------------------------------------- + +Note the '>' is escaped with '\', this causes that the '>' and filename are +passed to the 'ROD' function as parameters and the 'ROD' function contains +code to split '$@' on '>' and redirects the output to the file. + +EXPECT_PASS{,_BRK} and EXPECT_FAIL{,_BRK} ++++++++++++++++++++++++++++++++++++++++++ + +[source,sh] +------------------------------------------------------------------------------- +EXPECT_PASS command arg1 arg2 ... [ \> file ] +EXPECT_FAIL command arg1 arg2 ... [ \> file ] +------------------------------------------------------------------------------- + +'EXPECT_PASS' calls 'tst_res TPASS' if the command exited with 0 exit code, +and 'tst_res TFAIL' otherwise. 'EXPECT_FAIL' does vice versa. + +Output redirection rules are the same as for the 'ROD' function. In addition +to that, 'EXPECT_FAIL' always redirects the command's stderr to '/dev/null'. + +There are also 'EXPECT_PASS_BRK' and 'EXPECT_FAIL_BRK', which works the same way +except breaking a test when unexpected action happen. + +It's possible to detect whether expected value happened: +[source,sh] +------------------------------------------------------------------------------- +if ! EXPECT_PASS command arg1 2\> /dev/null; then + continue +fi +------------------------------------------------------------------------------- + +tst_kvcmp ++++++++++ + +This command compares the currently running kernel version given conditions +with syntax similar to the shell test command. + +[source,sh] +------------------------------------------------------------------------------- +# Exit the test if kernel version is older or equal to 2.6.8 +if tst_kvcmp -le 2.6.8; then + tst_brk TCONF "Kernel newer than 2.6.8 is needed" +fi + +# Exit the test if kernel is newer than 3.8 and older than 4.0.1 +if tst_kvcmp -gt 3.8 -a -lt 4.0.1; then + tst_brk TCONF "Kernel must be older than 3.8 or newer than 4.0.1" +fi +------------------------------------------------------------------------------- + +[options="header"] +|======================================================================= +| expression | description +| -eq kver | Returns true if kernel version is equal +| -ne kver | Returns true if kernel version is not equal +| -gt kver | Returns true if kernel version is greater +| -ge kver | Returns true if kernel version is greater or equal +| -lt kver | Returns true if kernel version is lesser +| -le kver | Returns true if kernel version is lesser or equal +| -a | Does logical and between two expressions +| -o | Does logical or between two expressions +|======================================================================= + +The format for kernel version has to either be with one dot e.g. '2.6' or with +two dots e.g. '4.8.1'. + +.tst_fs_has_free +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh + +... + +# whether current directory has 100MB free space at least. +if ! tst_fs_has_free . 100MB; then + tst_brkm TCONF "Not enough free space" +fi + +... +------------------------------------------------------------------------------- + +The 'tst_fs_has_free' shell interface returns 0 if the specified free space is +satisfied, 1 if not, and 2 on error. + +The second argument supports suffixes kB, MB and GB, the default unit is Byte. + +.tst_retry +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh + +... + +# Retry ping command three times +tst_retry "ping -c 1 127.0.0.1" + +if [ $? -ne 0 ]; then + tst_resm TFAIL "Failed to ping 127.0.0.1" +else + tst_resm TPASS "Successfully pinged 127.0.0.1" +fi + +... +------------------------------------------------------------------------------- + +The 'tst_retry' function allows you to retry a command after waiting small +amount of time until it succeeds or until given amount of retries has been +reached (default is three attempts). + +1.5 Restarting daemons +~~~~~~~~~~~~~~~~~~~~~~ + +Restarting system daemons is a complicated task for two reasons. + +* There are different init systems + (SysV init, systemd, etc...) + +* Daemon names are not unified between distributions + (apache vs httpd, cron vs crond, various syslog variations) + +To solve these problems LTP has 'testcases/lib/daemonlib.sh' library that +provides functions to start/stop/query daemons as well as variables that store +correct daemon name. + +.Supported operations +|============================================================================== +| start_daemon() | Starts daemon, name is passed as first parameter. +| stop_daemon() | Stops daemon, name is passed as first parameter. +| restart_daemon() | Restarts daemon, name is passed as first parameter. +| status_daemon() | Detect daemon status (exit code: 0: running, 1: not running). +|============================================================================== + +.Variables with detected names +|============================================================================== +| CROND_DAEMON | Cron daemon name (cron, crond). +| SYSLOG_DAEMON | Syslog daemon name (syslog, syslog-ng, rsyslog). +|============================================================================== + +.Cron daemon restart example +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Cron daemon restart example + +TCID=cron01 +TST_COUNT=1 +. test.sh +. daemonlib.sh + +... + +restart_daemon $CROND_DAEMON + +... + +tst_exit +------------------------------------------------------------------------------- + +1.6 Access to the checkpoint interface +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The shell library provides an implementation of the checkpoint interface +compatible with the C version. All 'TST_CHECKPOINT_*' functions are available. + +In order to initialize checkpoints '$TST_NEEDS_CHECKPOINTS' must be set to '1' +before the inclusion of 'test.sh': + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh + +TST_NEEDS_CHECKPOINTS=1 +. test.sh +------------------------------------------------------------------------------- + +Since both the implementations are compatible, it's also possible to start +a child binary process from a shell test and synchronize with it. This process +must have checkpoints initialized by calling 'tst_reinit()'. diff --git a/doc/test-writing-guidelines.txt b/doc/test-writing-guidelines.txt index c268b8804..0accac704 100644 --- a/doc/test-writing-guidelines.txt +++ b/doc/test-writing-guidelines.txt @@ -5,6 +5,10 @@ This document describes LTP guidelines and LTP test interface and is intended for anybody who want to write or modify a LTP testcase. It's not a definitive guide and it's not, by any means, a substitute for common sense. +NOTE: See also + https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API], + https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API]. + 1. General Rules ---------------- @@ -154,7 +158,9 @@ separate patch. ~~~~~~~~~~~ Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or -any later version). Use `SPDX-License-Identifier: GPL-2.0-or-later`. +any later version). + +Use `SPDX-License-Identifier: GPL-2.0-or-later` 2. Writing a testcase --------------------- @@ -232,2985 +238,11 @@ binaries is added to the '$PATH' and you can execute it just by its name. TIP: If you need to execute such test from the LTP tree, you can add path to current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'. -2.2 Writing a test in C -~~~~~~~~~~~~~~~~~~~~~~~ - -2.2.1 Basic test structure -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Let's start with an example, following code is a simple test for a 'getenv()'. - -[source,c] -------------------------------------------------------------------------------- -/* - * This is test for basic functionality of getenv(). - * - * - create an env variable and verify that getenv() can get get it - * - call getenv() with nonexisting variable name, check that it returns NULL - */ - -#include "tst_test.h" - -#define ENV1 "LTP_TEST_ENV" -#define ENV2 "LTP_TEST_THIS_DOES_NOT_EXIST" -#define ENV_VAL "val" - -static void setup(void) -{ - if (setenv(ENV1, ENV_VAL, 1)) - tst_brk(TBROK | TERRNO, "setenv() failed"); -} - -static void test(void) -{ - char *ret; - - ret = getenv(ENV1); - - if (!ret) { - tst_res(TFAIL, "getenv(" ENV1 ") = NULL"); - goto next; - } - - if (!strcmp(ret, ENV_VAL)) { - tst_res(TPASS, "getenv(" ENV1 ") = '"ENV_VAL "'"); - } else { - tst_res(TFAIL, "getenv(" ENV1 ") = '%s', expected '" - ENV_VAL "'", ret); - } - -next: - ret = getenv(ENV2); - - if (ret) - tst_res(TFAIL, "getenv(" ENV2 ") = '%s'", ret); - else - tst_res(TPASS, "getenv(" ENV2 ") = NULL"); -} - -static struct tst_test test = { - .test_all = test, - .setup = setup, -}; -------------------------------------------------------------------------------- - -Each test includes the 'tst_test.h' header and must define the 'struct -tst_test test' structure. - -The overall test initialization is done in the 'setup()' function. - -The overall cleanup is done in a 'cleanup()' function. Here 'cleanup()' is -omitted as the test does not have anything to clean up. If cleanup is set in -the test structure it's called on test exit just before the test library -cleanup. That especially means that cleanup can be called at any point in a -test execution. For example even when a test setup step has failed, therefore -the 'cleanup()' function must be able to cope with unfinished initialization, -and so on. - -The test itself is done in the 'test()' function. The test function must work -fine if called in a loop. - -There are two types of a test function pointers in the test structure. The -first one is a '.test_all' pointer that is used when test is implemented as a -single function. Then there is a '.test' function along with the number of -tests '.tcnt' that allows for more detailed result reporting. If the '.test' -pointer is set the function is called '.tcnt' times with an integer parameter -in range of [0, '.tcnt' - 1]. - -IMPORTANT: Only one of '.test' and '.test_all' can be set at a time. - -Each test has a default timeout set to 300s. The default timeout can be -overridden by setting '.timeout' in the test structure or by calling -'tst_set_timeout()' in the test 'setup()'. There are a few testcases whose run -time may vary arbitrarily, for these timeout can be disabled by setting it to --1. - -Test can find out how much time (in seconds) is remaining to timeout, -by calling 'tst_timeout_remaining()'. - -LAPI headers -++++++++++++ - -Use our LAPI headers ('include "lapi/foo.h"') to keep compatibility with old -distributions. LAPI header should always include original header. Older linux -headers were problematic, therefore we preferred to use libc headers. There are -still some bugs when combining certain glibc headers with linux headers, see -https://sourceware.org/glibc/wiki/Synchronizing_Headers. - -A word about the cleanup() callback -+++++++++++++++++++++++++++++++++++ - -There are a few rules that needs to be followed in order to write correct -cleanup() callback. - -1. Free only resources that were initialized. Keep in mind that callback can - be executed at any point in the test run. - -2. Make sure to free resources in the reverse order they were - initialized. (Some of the steps may not depend on others and everything - will work if there were swapped but let's keep it in order.) - -The first rule may seem complicated at first however, on the contrary, it's -quite easy. All you have to do is to keep track of what was already -initialized. For example file descriptors needs to be closed only if they were -assigned a valid file descriptor. For most of the things you need to create -extra flag that is set right after successful initialization though. Consider, -for example, test setup below. - -We also prefer cleaning up resources that would otherwise be released on the -program exit. There are two main reasons for this decision. Resources such as -file descriptors and mmaped memory could block umounting a block device in -cases where the test library has mounted a filesystem for the test temporary -directory. Not freeing allocated memory would upset static analysis and tools -such as valgrind and produce false-positives when checking for leaks in the -libc and other low level libraries. - -[source,c] -------------------------------------------------------------------------------- -static int fd0, fd1, mount_flag; - -#define MNTPOINT "mntpoint" -#define FILE1 "mntpoint/file1" -#define FILE2 "mntpoint/file2" - -static void setup(void) -{ - SAFE_MKDIR(MNTPOINT, 0777); - SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL); - SAFE_MOUNT(tst_device->dev, MNTPOINT, tst_device->fs_type, 0, 0); - mount_flag = 1; - - fd0 = SAFE_OPEN(cleanup, FILE1, O_CREAT | O_RDWR, 0666); - fd1 = SAFE_OPEN(cleanup, FILE2, O_CREAT | O_RDWR, 0666); -} -------------------------------------------------------------------------------- - -In this case the 'cleanup()' function may be invoked when any of the 'SAFE_*' -macros has failed and therefore must be able to work with unfinished -initialization as well. Since global variables are initialized to zero we can -just check that fd > 0 before we attempt to close it. The mount function -requires extra flag to be set after device was successfully mounted. - -[source,c] -------------------------------------------------------------------------------- -static void cleanup(void) -{ - if (fd1 > 0) - SAFE_CLOSE(fd1); - - if (fd0 > 0) - SAFE_CLOSE(fd0); - - if (mount_flag && tst_umouont(MNTPOINT)) - tst_res(TWARN | TERRNO, "umount(%s)", MNTPOINT); -} -------------------------------------------------------------------------------- - -IMPORTANT: 'SAFE_MACROS()' used in cleanup *do not* exit the test. Failure - only produces a warning and the 'cleanup()' carries on. This is - intentional as we want to execute as much 'cleanup()' as possible. - -WARNING: Calling tst_brk() in test 'cleanup()' does not exit the test as well - and 'TBROK' is converted to 'TWARN'. - -NOTE: Creation and removal of the test temporary directory is handled in - the test library and the directory is removed recursively. Therefore - we do not have to remove files and directories in the test cleanup. - -2.2.2 Basic test interface -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -[source,c] -------------------------------------------------------------------------------- -void tst_res(int ttype, char *arg_fmt, ...); -------------------------------------------------------------------------------- - -Printf-like function to report test result, it's mostly used with ttype: - -|============================== -| 'TPASS' | Test has passed. -| 'TFAIL' | Test has failed. -| 'TINFO' | General message. -| 'TWARN' | Something went wrong but we decided to continue. Mostly used in cleanup functions. -|============================== - -The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print -'errno', 'TST_ERR' respectively. - -[source,c] -------------------------------------------------------------------------------- -void tst_brk(int ttype, char *arg_fmt, ...); -------------------------------------------------------------------------------- - -Printf-like function to report error and exit the test, it can be used with ttype: - -|============================================================ -| 'TBROK' | Something has failed in test preparation phase. -| 'TCONF' | Test is not appropriate for current configuration - (syscall not implemented, unsupported arch, ...) -|============================================================ - -The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print -'errno', 'TST_ERR' respectively. - -There are also 'TST_EXP_*()' macros that can simplify syscall unit tests to a -single line, use them whenever possible. These macros take a function call as -the first parameter and a printf-like format string and parameters as well. -These test macros then expand to a code that runs the call, checks the return -value and errno and reports the test result. - -[source,c] -------------------------------------------------------------------------------- -static void test(void) -{ - ... - TST_EXP_PASS(stat(fname, &statbuf), "stat(%s, ...)", fname); - - if (!TST_PASS) - return; - ... -} -------------------------------------------------------------------------------- - -The 'TST_EXP_PASS()' can be used for calls that return -1 on failure and 0 on -success. It will check for the return value and reports failure if the return -value is not equal to 0. The call also sets the 'TST_PASS' variable to 1 if -the call succeeeded. - -[source,c] -------------------------------------------------------------------------------- -static void test(void) -{ - ... - TST_EXP_FD(open(fname, O_RDONLY), "open(%s, O_RDONLY)", fname); - - SAFE_CLOSE(TST_RET); - ... -} -------------------------------------------------------------------------------- - -The 'TST_EXP_FD()' is the same as 'TST_EXP_PASS()' the only difference is that -the return value is expected to be a file descriptor so the call passes if -positive integer is returned. - -[source,c] -------------------------------------------------------------------------------- -static void test(void) -{ - ... - TST_EXP_FAIL(stat(fname, &statbuf), ENOENT, "stat(%s, ...)", fname); - ... -} -------------------------------------------------------------------------------- - -The 'TST_EXP_FAIL()' is similar to 'TST_EXP_PASS()' but it fails the test if -the call haven't failed with -1 and 'errno' wasn't set to the expected one -passed as the second argument. - -[source,c] -------------------------------------------------------------------------------- -const char *tst_strsig(int sig); -------------------------------------------------------------------------------- - -Return the given signal number's corresponding string. - -[source,c] -------------------------------------------------------------------------------- -const char *tst_strerrno(int err); -------------------------------------------------------------------------------- - -Return the given errno number's corresponding string. Using this function to -translate 'errno' values to strings is preferred. You should not use the -'strerror()' function in the testcases. - -[source,c] -------------------------------------------------------------------------------- -const char *tst_strstatus(int status); -------------------------------------------------------------------------------- - -Returns string describing the status as returned by 'wait()'. - -WARNING: This function is not thread safe. - -[source,c] -------------------------------------------------------------------------------- -void tst_set_timeout(unsigned int timeout); -------------------------------------------------------------------------------- - -Allows for setting timeout per test iteration dynamically in the test setup(), -the timeout is specified in seconds. There are a few testcases whose runtime -can vary arbitrarily, these can disable timeouts by setting it to -1. - -[source,c] -------------------------------------------------------------------------------- -void tst_flush(void); -------------------------------------------------------------------------------- - -Flush output streams, handling errors appropriately. - -This function is rarely needed when you have to flush the output streams -before calling 'fork()' or 'clone()'. Note that the 'SAFE_FORK()' and 'SAFE_CLONE()' -calls this function automatically. See 3.4 FILE buffers and fork() for explanation -why is this needed. - -2.2.3 Test temporary directory -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If '.needs_tmpdir' is set to '1' in the 'struct tst_test' unique test -temporary is created and it's set as the test working directory. Tests *MUST -NOT* create temporary files outside that directory. The flag is not needed to -be set when use these flags: '.all_filesystems', '.format_device', '.mntpoint', -'.mount_device' '.needs_checkpoints', '.needs_device', '.resource_file' -(these flags imply creating temporary directory). - -IMPORTANT: Close all file descriptors (that point to files in test temporary - directory, even the unlinked ones) either in the 'test()' function - or in the test 'cleanup()' otherwise the test may break temporary - directory removal on NFS (look for "NFS silly rename"). - -2.2.4 Safe macros -^^^^^^^^^^^^^^^^^ - -Safe macros aim to simplify error checking in test preparation. Instead of -calling system API functions, checking for their return value and aborting the -test if the operation has failed, you just use corresponding safe macro. - -Use them whenever it's possible. - -Instead of writing: - -[source,c] -------------------------------------------------------------------------------- - fd = open("/dev/null", O_RDONLY); - if (fd < 0) - tst_brk(TBROK | TERRNO, "opening /dev/null failed"); -------------------------------------------------------------------------------- - -You write just: - -[source,c] -------------------------------------------------------------------------------- - fd = SAFE_OPEN("/dev/null", O_RDONLY); -------------------------------------------------------------------------------- - -IMPORTANT: The SAFE_CLOSE() function also sets the passed file descriptor to -1 - after it's successfully closed. - -They can also simplify reading and writing of sysfs files, you can, for -example, do: - -[source,c] -------------------------------------------------------------------------------- - SAFE_FILE_SCANF("/proc/sys/kernel/pid_max", "%lu", &pid_max); -------------------------------------------------------------------------------- - -See 'include/tst_safe_macros.h', 'include/tst_safe_stdio.h' and -'include/tst_safe_file_ops.h' and 'include/tst_safe_net.h' for a complete list. - -2.2.5 Test specific command line options -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -[source,c] -------------------------------------------------------------------------------- -struct tst_option { - char *optstr; - char **arg; - char *help; -}; -------------------------------------------------------------------------------- - -Test specific command line parameters can be passed with the 'NULL' terminated -array of 'struct tst_option'. The 'optstr' is the command line option i.e. "o" -or "o:" if option has a parameter. Only short options are supported. The 'arg' -is where 'optarg' is stored upon match. If option has no parameter it's set to -non-'NULL' value if option was present. The 'help' is a short help string. - -NOTE: The test parameters must not collide with common test parameters defined - in the library the currently used ones are +-i+, +-I+, +-C+, and +-h+. - -[source,c] -------------------------------------------------------------------------------- -int tst_parse_int(const char *str, int *val, int min, int max); -int tst_parse_float(const char *str, float *val, float min, float max); -------------------------------------------------------------------------------- - -Helpers for parsing the strings returned in the 'struct tst_option'. - -Both return zero on success and 'errno', mostly 'EINVAL' or 'ERANGE', on -failure. - -Both functions are no-op if 'str' is 'NULL'. - -The valid range for result includes both 'min' and 'max'. - -.Example Usage -[source,c] -------------------------------------------------------------------------------- -#include -#include "tst_test.h" - -static char *str_threads; -static int threads = 10; - -static struct tst_option options[] = { - {"t:", &str_threads, "Number of threads (default 10)"}, - ... - {NULL, NULL, NULL} -}; - -static void setup(void) -{ - if (tst_parse_int(str_threads, &threads, 1, INT_MAX)) - tst_brk(TBROK, "Invalid number of threads '%s'", str_threads); - - ... -} - -static void test_threads(void) -{ - ... - - for (i = 0; i < threads; i++) { - ... - } - - ... -} - -static struct tst_test test = { - ... - .options = options, - ... -}; -------------------------------------------------------------------------------- - - -2.2.6 Runtime kernel version detection -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Testcases for newly added kernel functionality require kernel newer than a -certain version to run. All you need to skip a test on older kernels is to -set the '.min_kver' string in the 'struct tst_test' to a minimal required -kernel version, e.g. '.min_kver = "2.6.30"'. - -For more complicated operations such as skipping a test for a certain range -of kernel versions, following functions could be used: - -[source,c] -------------------------------------------------------------------------------- -int tst_kvercmp(int r1, int r2, int r3); - -struct tst_kern_exv { - char *dist_name; - char *extra_ver; -}; - -int tst_kvercmp2(int r1, int r2, int r3, struct tst_kern_exv *vers); -------------------------------------------------------------------------------- - -These two functions are intended for runtime kernel version detection. They -parse the output from 'uname()' and compare it to the passed values. - -The return value is similar to the 'strcmp()' function, i.e. zero means equal, -negative value means that the kernel is older than than the expected value and -positive means that it's newer. - -The second function 'tst_kvercmp2()' allows for specifying per-vendor table of -kernel versions as vendors typically backport fixes to their kernels and the -test may be relevant even if the kernel version does not suggests so. See -'testcases/kernel/syscalls/inotify/inotify04.c' for example usage. - -WARNING: The shell 'tst_kvercmp' maps the result into unsigned integer - the - process exit value. - -2.2.7 Fork()-ing -^^^^^^^^^^^^^^^^ - -Be wary that if the test forks and there were messages printed by the -'tst_*()' interfaces, the data may still be in libc/kernel buffers and these -*ARE NOT* flushed automatically. - -This happens when 'stdout' gets redirected to a file. In this case, the -'stdout' is not line buffered, but block buffered. Hence after a fork content -of the buffers will be printed by the parent and each of the children. - -To avoid that you should use 'SAFE_FORK()', 'SAFE_CLONE()' or 'tst_clone()'. - -IMPORTANT: You have to set the '.forks_child' flag in the test structure - if your testcase forks or calls 'SAFE_CLONE()'. - -2.2.8 Doing the test in the child process -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Results reported by 'tst_res()' are propagated to the parent test process via -block of shared memory. - -Calling 'tst_brk()' causes child process to exit with non-zero exit value. -Which means that it's safe to use 'SAFE_*()' macros in the child processes as -well. - -Children that outlive the 'test()' function execution are waited for in the -test library. Unclean child exit (killed by signal, non-zero exit value, etc.) -will cause the main test process to exit with 'tst_brk()', which especially -means that 'TBROK' propagated from a child process will cause the whole test -to exit with 'TBROK'. - -If a test needs a child that segfaults or does anything else that cause it to -exit uncleanly all you need to do is to wait for such children from the -'test()' function so that it's reaped before the main test exits the 'test()' -function. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -void tst_reap_children(void); -------------------------------------------------------------------------------- - -The 'tst_reap_children()' function makes the process wait for all of its -children and exits with 'tst_brk(TBROK, ...)' if any of them returned -a non zero exit code. - -When using 'SAFE_CLONE' or 'tst_clone', this may not work depending on -the parameters passed to clone. The following call to 'SAFE_CLONE' is -identical to 'fork()', so will work as expected. - -[source,c] --------------------------------------------------------------------------------- -const struct tst_clone_args args = { - .exit_signal = SIGCHLD, -}; - -SAFE_CLONE(&args); --------------------------------------------------------------------------------- - -If 'exit_signal' is set to something else, then this will break -'tst_reap_children'. It's not expected that all parameters to clone will -work with the LTP library unless specific action is taken by the test code. - -.Using 'tst_res()' from binaries started by 'exec()' -[source,c] -------------------------------------------------------------------------------- -/* test.c */ -#define _GNU_SOURCE -#include -#include "tst_test.h" - -static void do_test(void) -{ - char *const argv[] = {"test_exec_child", NULL}; - char path[4096]; - - if (tst_get_path("test_exec_child", path, sizeof(path))) - tst_brk(TCONF, "Couldn't find test_exec_child in $PATH"); - - execve(path, argv, environ); - - tst_res(TFAIL | TERRNO, "EXEC!"); -} - -static struct tst_test test = { - .test_all = do_test, - .child_needs_reinit = 1, -}; - -/* test_exec_child.c */ -#define TST_NO_DEFAULT_MAIN -#include "tst_test.h" - -int main(void) -{ - tst_reinit(); - tst_res(TPASS, "Child passed!"); - return 0; -} -------------------------------------------------------------------------------- - -The 'tst_res()' function can be also used from binaries started by 'exec()', -the parent test process has to set the '.child_needs_reinit' flag so that the -library prepares for it and has to make sure the 'LTP_IPC_PATH' environment -variable is passed down, then the very fist thing the program has to call in -'main()' is 'tst_reinit()' that sets up the IPC. - -2.2.9 Fork() and Parent-child synchronization -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -As LTP tests are written for Linux, most of the tests involve fork()-ing and -parent-child process synchronization. LTP includes a checkpoint library that -provides wait/wake futex based functions. - -In order to use checkpoints the '.needs_checkpoints' flag in the 'struct -tst_test' must be set to '1', this causes the test library to initialize -checkpoints before the 'test()' function is called. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -TST_CHECKPOINT_WAIT(id) - -TST_CHECKPOINT_WAIT2(id, msec_timeout) - -TST_CHECKPOINT_WAKE(id) - -TST_CHECKPOINT_WAKE2(id, nr_wake) - -TST_CHECKPOINT_WAKE_AND_WAIT(id) -------------------------------------------------------------------------------- - -The checkpoint interface provides pair of wake and wait functions. The 'id' is -unsigned integer which specifies checkpoint to wake/wait for. As a matter of -fact it's an index to an array stored in a shared memory, so it starts on -'0' and there should be enough room for at least of hundred of them. - -The 'TST_CHECKPOINT_WAIT()' and 'TST_CHECKPOINT_WAIT2()' suspends process -execution until it's woken up or until timeout is reached. - -The 'TST_CHECKPOINT_WAKE()' wakes one process waiting on the checkpoint. -If no process is waiting the function retries until it success or until -timeout is reached. - -If timeout has been reached process exits with appropriate error message (uses -'tst_brk()'). - -The 'TST_CHECKPOINT_WAKE2()' does the same as 'TST_CHECKPOINT_WAKE()' but can -be used to wake precisely 'nr_wake' processes. - -The 'TST_CHECKPOINT_WAKE_AND_WAIT()' is a shorthand for doing wake and then -immediately waiting on the same checkpoint. - -Child processes created via 'SAFE_FORK()' are ready to use the checkpoint -synchronization functions, as they inherited the mapped page automatically. - -Child processes started via 'exec()', or any other processes not forked from -the test process must initialize the checkpoint by calling 'tst_reinit()'. - -For the details of the interface, look into the 'include/tst_checkpoint.h'. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -/* - * Waits for process state change. - * - * The state is one of the following: - * - * R - process is running - * S - process is sleeping - * D - process sleeping uninterruptibly - * Z - zombie process - * T - process is traced - */ -TST_PROCESS_STATE_WAIT(pid, state, msec_timeout) -------------------------------------------------------------------------------- - -The 'TST_PROCESS_STATE_WAIT()' waits until process 'pid' is in requested -'state' or timeout is reached. The call polls +/proc/pid/stat+ to get this -information. A timeout of 0 will wait infinitely. - -On timeout -1 is returned and errno set to ETIMEDOUT. - -It's mostly used with state 'S' which means that process is sleeping in kernel -for example in 'pause()' or any other blocking syscall. - -2.2.10 Signals and signal handlers -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you need to use signal handlers, keep the code short and simple. Don't -forget that the signal handler is called asynchronously and can interrupt the -code execution at any place. - -This means that problems arise when global state is changed both from the test -code and signal handler, which will occasionally lead to: - -* Data corruption (data gets into inconsistent state), this may happen, for - example, for any operations on 'FILE' objects. - -* Deadlock, this happens, for example, if you call 'malloc(2)', 'free(2)', - etc. from both the test code and the signal handler at the same time since - 'malloc' has global lock for it's internal data structures. (Be wary that - 'malloc(2)' is used by the libc functions internally too.) - -* Any other unreproducible and unexpected behavior. - -Quite common mistake is to call 'exit(3)' from a signal handler. Note that this -function is not signal-async-safe as it flushes buffers, etc. If you need to -exit a test immediately from a signal handler use '_exit(2)' instead. - -TIP: See 'man 7 signal' for the list of signal-async-safe functions. - -If a signal handler sets a variable, its declaration must be 'volatile', -otherwise compiler may misoptimize the code. This is because the variable may -not be changed in the compiler code flow analysis. There is 'sig_atomic_t' -type defined in C99 but this one *DOES NOT* imply 'volatile' (it's just a -'typedef' to 'int'). So the correct type for a flag that is changed from a -signal handler is either 'volatile int' or 'volatile sig_atomic_t'. - -If a crash (e.g. triggered by signal SIGSEGV) is expected in testing, you -can avoid creation of core files by calling tst_no_corefile() function. -This takes effect for process (and its children) which invoked it, unless -they subsequently modify RLIMIT_CORE. - -Note that LTP library will reap any processes that test didn't reap itself, -and report any non-zero exit code as failure. - -2.2.11 Kernel Modules -^^^^^^^^^^^^^^^^^^^^^ - -There are certain cases where the test needs a kernel part and userspace part, -happily, LTP can build a kernel module and then insert it to the kernel on test -start for you. See 'testcases/kernel/device-drivers/block' for details. - -2.2.12 Useful macros -^^^^^^^^^^^^^^^^^^^^^ - -[source,c] -------------------------------------------------------------------------------- -ARRAY_SIZE(arr) -------------------------------------------------------------------------------- - -Returns the size of statically defined array, i.e. -'(sizeof(arr) / sizeof(*arr))' - -[source,c] -------------------------------------------------------------------------------- -LTP_ALIGN(x, a) -------------------------------------------------------------------------------- - -Aligns the x to be next multiple of a. The a must be power of 2. - -2.2.13 Filesystem type detection and skiplist -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some tests are known to fail on certain filesystems (you cannot swap on TMPFS, -there are unimplemented 'fcntl()' etc.). - -If your test needs to be skipped on certain filesystems use the -'.skip_filesystems' field in the tst_test structure as follows: - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static struct tst_test test = { - ... - .skip_filesystems = (const char *const []) { - "tmpfs", - "ramfs", - "nfs", - NULL - }, -}; -------------------------------------------------------------------------------- - -When the '.all_filesystem' flag is set the '.skip_filesystems' list is passed -to the function that detects supported filesystems any listed filesystem is -not included in the resulting list of supported filesystems. - -If test needs to adjust expectations based on filesystem type it's also -possible to detect filesystem type at the runtime. This is preferably used -when only subset of the test is not applicable for a given filesystem. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static void run(void) -{ - ... - - switch ((type = tst_fs_type("."))) { - case TST_NFS_MAGIC: - case TST_TMPFS_MAGIC: - case TST_RAMFS_MAGIC: - tst_brk(TCONF, "Subtest not supported on %s", - tst_fs_type_name(type)); - return; - break; - } - - ... -} -------------------------------------------------------------------------------- - -2.2.14 Thread-safety in the LTP library -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is safe to use library 'tst_res()' function in multi-threaded tests. - -Only the main thread must return from the 'test()' function to the test -library and that must be done only after all threads that may call any library -function has been terminated. That especially means that threads that may call -'tst_brk()' must terminate before the execution of the 'test()' function -returns to the library. This is usually done by the main thread joining all -worker threads at the end of the 'test()' function. Note that the main thread -will never get to the library code in a case that 'tst_brk()' was called from -one of the threads since it will sleep at least in 'pthread_join()' on the -thread that called the 'tst_brk()' till 'exit()' is called by 'tst_brk()'. - -The test-supplied cleanup function runs *concurrently* to the rest of the -threads in a case that cleanup was entered from 'tst_brk()'. Subsequent -threads entering 'tst_brk()' must be suspended or terminated at the start of -the user supplied cleanup function. It may be necessary to stop or exit -the rest of the threads before the test cleans up as well. For example threads -that create new files should be stopped before temporary directory is be -removed. - -Following code example shows thread safe cleanup function example using atomic -increment as a guard. The library calls its cleanup after the execution returns -from the user supplied cleanup and expects that only one thread returns from -the user supplied cleanup to the test library. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static void cleanup(void) -{ - static int flag; - - if (tst_atomic_inc(&flag) != 1) - pthread_exit(NULL); - - /* if needed stop the rest of the threads here */ - - ... - - /* then do cleanup work */ - - ... - - /* only one thread returns to the library */ -} -------------------------------------------------------------------------------- - - -2.2.15 Testing with a block device -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some tests needs a block device (inotify tests, syscall 'EROFS' failures, -etc.). LTP library contains a code to prepare a testing device. - -If '.needs_device' flag in the 'struct tst_test' is set the 'tst_device' -structure is initialized with a path to a test device and default filesystem -to be used. - -You can also request minimal device size in megabytes by setting -'.dev_min_size' the device is guaranteed to have at least the requested size -then. - -If '.format_device' flag is set the device is formatted with a filesystem as -well. You can use '.dev_fs_type' to override the default filesystem type if -needed and pass additional options to mkfs via '.dev_fs_opts' and -'.dev_extra_opts' pointers. Note that '.format_device' implies '.needs_device' -there is no need to set both. - -If '.mount_device' is set, the device is mounted at '.mntpoint' which is used -to pass a directory name that will be created and used as mount destination. -You can pass additional flags and data to the mount command via '.mnt_flags' -and '.mnt_data' pointers. Note that '.mount_device' implies '.needs_device' -and '.format_device' so there is no need to set the later two. - -If '.needs_rofs' is set, read-only filesystem is mounted at '.mntpoint' this -one is supposed to be used for 'EROFS' tests. - -If '.all_filesystems' is set the test function is executed for all supported -filesystems. Supported filesystems are detected based on existence of the -'mkfs.$fs' helper and on kernel support to mount it. For each supported -filesystem the 'tst_device.fs_type' is set to the currently tested fs type, if -'.format_device' is set the device is formatted as well, if '.mount_device' is -set it's mounted at '.mntpoint'. Also the test timeout is reset for each -execution of the test function. This flag is expected to be used for filesystem -related syscalls that are at least partly implemented in the filesystem -specific code e.g. fallocate(). - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -struct tst_device { - const char *dev; - const char *fs_type; -}; - -extern struct tst_device *tst_device; +3. Test Contribution Checklist +------------------------------ -int tst_umount(const char *path); -------------------------------------------------------------------------------- - -In case that 'LTP_DEV' is passed to the test in an environment, the library -checks that the file exists and that it's a block device, if -'.device_min_size' is set the device size is checked as well. If 'LTP_DEV' -wasn't set or if size requirements were not met a temporary file is created -and attached to a free loop device. - -If there is no usable device and loop device couldn't be initialized the test -exits with 'TCONF'. - -The 'tst_umount()' function works exactly as 'umount(2)' but retries several -times on 'EBUSY'. This is because various desktop daemons (gvfsd-trash is known -for that) may be stupid enough to probe all newly mounted filesystem which -results in 'umount(2)' failing with 'EBUSY'. - -IMPORTANT: All testcases should use 'tst_umount()' instead of 'umount(2)' to - umount filesystems. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_find_free_loopdev(const char *path, size_t path_len); -------------------------------------------------------------------------------- - -This function finds a free loopdev and returns the free loopdev minor (-1 for no -free loopdev). If path is non-NULL, it will be filled with free loopdev path. -If you want to use a customized loop device, we can call tst_find_free_loopdev -(NULL, 0) in tests to get a free minor number and then mknod. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -unsigned long tst_dev_bytes_written(const char *dev); -------------------------------------------------------------------------------- - -This function reads test block device stat file (/sys/block//stat) and -returns the bytes written since the last invocation of this function. To avoid -FS deferred IO metadata/cache interference, we suggest doing "syncfs" before the -tst_dev_bytes_written first invocation. And an inline function named tst_dev_sync -is created for that intention. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -voud tst_find_backing_dev(const char *path, char *dev); -------------------------------------------------------------------------------- - -This function finds the block dev that this path belongs to, it uses stat function -to get the major/minor number of the path. Then scan them in "/proc/self/mountinfo" -and list 2th column value after ' - ' string as its block dev if match succeeds. - -2.2.16 Formatting a device with a filesystem -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static void setup(void) -{ - ... - SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL); - ... -} -------------------------------------------------------------------------------- - -This function takes a path to a device, filesystem type and an array of extra -options passed to mkfs. - -The fs options 'fs_opts' should either be 'NULL' if there are none, or a -'NULL' terminated array of strings such as: -+const char *const opts[] = {"-b", "1024", NULL}+. - -The extra options 'extra_opts' should either be 'NULL' if there are none, or a -'NULL' terminated array of strings such as +{"102400", NULL}+; 'extra_opts' -will be passed after device name. e.g: +mkfs -t ext4 -b 1024 /dev/sda1 102400+ -in this case. - -Note that perfer to store the options which can be passed before or after device -name by 'fs_opts' array. - -2.2.17 Verifying a filesystem's free space -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some tests have size requirements for the filesystem's free space. If these -requirements are not satisfied, the tests should be skipped. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_fs_has_free(const char *path, unsigned int size, unsigned int mult); -------------------------------------------------------------------------------- - -The 'tst_fs_has_free()' function returns 1 if there is enough space and 0 if -there is not. - -The 'path' is the pathname of any directory/file within a filesystem. - -The 'mult' is a multiplier, one of 'TST_BYTES', 'TST_KB', 'TST_MB' or 'TST_GB'. - -The required free space is calculated by 'size * mult', e.g. -'tst_fs_has_free("/tmp/testfile", 64, TST_MB)' will return 1 if the -filesystem, which '"/tmp/testfile"' is in, has 64MB free space at least, and 0 -if not. - -2.2.18 Files, directories and fs limits -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some tests need to know the maximum count of links to a regular file or -directory, such as 'rename(2)' or 'linkat(2)' to test 'EMLINK' error. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_fs_fill_hardlinks(const char *dir); -------------------------------------------------------------------------------- - -Try to get maximum count of hard links to a regular file inside the 'dir'. - -NOTE: This number depends on the filesystem 'dir' is on. - -This function uses 'link(2)' to create hard links to a single file until it -gets 'EMLINK' or creates 65535 links. If the limit is hit, the maximum number of -hardlinks is returned and the 'dir' is filled with hardlinks in format -"testfile%i", where i belongs to [0, limit) interval. If no limit is hit or if -'link(2)' failed with 'ENOSPC' or 'EDQUOT', zero is returned and previously -created files are removed. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_fs_fill_subdirs(const char *dir); -------------------------------------------------------------------------------- - -Try to get maximum number of subdirectories in directory. - -NOTE: This number depends on the filesystem 'dir' is on. For current kernel, -subdir limit is not available for all filesystems (available for ext2, ext3, -minix, sysv and more). If the test runs on some other filesystems, like ramfs, -tmpfs, it will not even try to reach the limit and return 0. - -This function uses 'mkdir(2)' to create directories in 'dir' until it gets -'EMLINK' or creates 65535 directories. If the limit is hit, the maximum number -of subdirectories is returned and the 'dir' is filled with subdirectories in -format "testdir%i", where i belongs to [0, limit - 2) interval (because each -newly created dir has two links already - the '.' and the link from parent -dir). If no limit is hit or if 'mkdir(2)' failed with 'ENOSPC' or 'EDQUOT', -zero is returned and previously created directories are removed. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_dir_is_empty(const char *dir, int verbose); -------------------------------------------------------------------------------- - -Returns non-zero if directory is empty and zero otherwise. - -Directory is considered empty if it contains only '.' and '..'. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -void tst_purge_dir(const char *path); -------------------------------------------------------------------------------- - -Deletes the contents of given directory but keeps the directory itself. Useful -for cleaning up the temporary directory and mount points between test cases or -test iterations. Terminates the program with 'TBROK' on error. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_fill_fd(int fd, char pattern, size_t bs, size_t bcount); -------------------------------------------------------------------------------- - -Fill a file with specified pattern using file descriptor. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_prealloc_size_fd(int fd, size_t bs, size_t bcount); -------------------------------------------------------------------------------- - -Preallocate the specified amount of space using 'fallocate()'. Falls back to -'tst_fill_fd()' if 'fallocate()' fails. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_fill_file(const char *path, char pattern, size_t bs, size_t bcount); -------------------------------------------------------------------------------- - -Creates/overwrites a file with specified pattern using file path. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_prealloc_file(const char *path, size_t bs, size_t bcount); -------------------------------------------------------------------------------- - -Create/overwrite a file and preallocate the specified amount of space for it. -The allocated space will not be initialized to any particular content. - -2.2.19 Getting an unused PID number -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some tests require a 'PID', which is not used by the OS (does not belong to -any process within it). For example, kill(2) should set errno to 'ESRCH' if -it's passed such 'PID'. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -pid_t tst_get_unused_pid(void); -------------------------------------------------------------------------------- - -Return a 'PID' value not used by the OS or any process within it. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_get_free_pids(void); -------------------------------------------------------------------------------- - -Returns number of unused pids in the system. Note that this number may be -different once the call returns and should be used only for rough estimates. - -2.2.20 Running executables -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -int tst_cmd(const char *const argv[], - const char *stdout_path, - const char *stderr_path, - enum tst_cmd_flags flags); -------------------------------------------------------------------------------- - -'tst_cmd()' is a wrapper for 'vfork() + execvp()' which provides a way -to execute an external program. - -'argv[]' is a 'NULL' terminated array of strings starting with the program name -which is followed by optional arguments. - -'TST_CMD_PASS_RETVAL' enum 'tst_cmd_flags' makes 'tst_cmd()' -return the program exit code to the caller, otherwise 'tst_cmd()' exit the -tests on failure. 'TST_CMD_TCONF_ON_MISSING' check for program in '$PATH' and exit -with 'TCONF' if not found. - -In case that 'execvp()' has failed and the enum 'TST_CMD_PASS_RETVAL' flag was set, the -return value is '255' if 'execvp()' failed with 'ENOENT' and '254' otherwise. - -'stdout_path' and 'stderr_path' determine where to redirect the program -stdout and stderr I/O streams. - -The 'SAFE_CMD()' macro can be used automatic handling non-zero exits (exits -with 'TBROK') and 'ENOENT' (exits with 'TCONF'). - -.Example -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -const char *const cmd[] = { "ls", "-l", NULL }; - -... - /* Store output of 'ls -l' into log.txt */ - tst_cmd(cmd, "log.txt", NULL, 0); -... -------------------------------------------------------------------------------- - -2.2.21 Measuring elapsed time and helper functions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -[source,c] -------------------------------------------------------------------------------- -#include "tst_timer.h" - -void tst_timer_check(clockid_t clk_id); - -void tst_timer_start(clockid_t clk_id); - -void tst_timer_stop(void); - -struct timespec tst_timer_elapsed(void); - -long long tst_timer_elapsed_ms(void); - -long long tst_timer_elapsed_us(void); - -int tst_timer_expired_ms(long long ms); -------------------------------------------------------------------------------- - -The 'tst_timer_check()' function checks if specified 'clk_id' is suppored and -exits the test with 'TCONF' otherwise. It's expected to be used in test -'setup()' before any resources that needs to be cleaned up are initialized, -hence it does not include a cleanup function parameter. - -The 'tst_timer_start()' marks start time and stores the 'clk_id' for further -use. - -The 'tst_timer_stop()' marks the stop time using the same 'clk_id' as last -call to 'tst_timer_start()'. - -The 'tst_timer_elapsed*()' returns time difference between the timer start and -last timer stop in several formats and units. - -The 'tst_timer_expired_ms()' function checks if the timer started by -'tst_timer_start()' has been running longer than ms milliseconds. The function -returns non-zero if timer has expired and zero otherwise. - -IMPORTANT: The timer functions use 'clock_gettime()' internally which needs to - be linked with '-lrt' on older glibc. Please do not forget to add - 'LDLIBS+=-lrt' in Makefile. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" -#include "tst_timer.h" - -static void setup(void) -{ - ... - tst_timer_check(CLOCK_MONOTONIC); - ... -} - -static void run(void) -{ - ... - tst_timer_start(CLOCK_MONOTONIC); - ... - while (!tst_timer_expired_ms(5000)) { - ... - } - ... -} - -struct tst_test test = { - ... - .setup = setup, - .test_all = run, - ... -}; -------------------------------------------------------------------------------- - -Expiration timer example usage. - -[source,c] -------------------------------------------------------------------------------- -long long tst_timespec_to_us(struct timespec t); -long long tst_timespec_to_ms(struct timespec t); - -struct timeval tst_us_to_timeval(long long us); -struct timeval tst_ms_to_timeval(long long ms); - -int tst_timespec_lt(struct timespec t1, struct timespec t2); - -struct timespec tst_timespec_add_us(struct timespec t, long long us); - -struct timespec tst_timespec_diff(struct timespec t1, struct timespec t2); -long long tst_timespec_diff_us(struct timespec t1, struct timespec t2); -long long tst_timespec_diff_ms(struct timespec t1, struct timespec t2); - -struct timespec tst_timespec_abs_diff(struct timespec t1, struct timespec t2); -long long tst_timespec_abs_diff_us(struct timespec t1, struct timespec t2); -long long tst_timespec_abs_diff_ms(struct timespec t1, struct timespec t2); -------------------------------------------------------------------------------- - -The first four functions are simple inline conversion functions. - -The 'tst_timespec_lt()' function returns non-zero if 't1' is earlier than -'t2'. - -The 'tst_timespec_add_us()' function adds 'us' microseconds to the timespec -'t'. The 'us' is expected to be positive. - -The 'tst_timespec_diff*()' functions returns difference between two times, the -'t1' is expected to be later than 't2'. - -The 'tst_timespec_abs_diff*()' functions returns absolute value of difference -between two times. - -NOTE: All conversions to ms and us rounds the value. - -2.2.22 Datafiles -^^^^^^^^^^^^^^^^ - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static const char *const res_files[] = { - "foo", - "bar", - NULL -}; - -static struct tst_test test = { - ... - .resource_files = res_files, - ... -} -------------------------------------------------------------------------------- - -If the test needs additional files to be copied to the test temporary -directory all you need to do is to list their filenames in the -'NULL' terminated array '.resource_files' in the tst_test structure. - -When resource files is set test temporary directory is created automatically, -there is need to set '.needs_tmpdir' as well. - -The test library looks for datafiles first, these are either stored in a -directory called +datafiles+ in the +$PWD+ at the start of the test or in -+$LTPROOT/testcases/data/${test_binary_name}+. If the file is not found the -library looks into +$LTPROOT/testcases/bin/+ and to +$PWD+ at the start of the -test. This ensures that the testcases can copy the file(s) effortlessly both -when test is started from the directory it was compiled in as well as when LTP -was installed. - -The file(s) are copied to the newly created test temporary directory which is -set as the test working directory when the 'test()' functions is executed. - -2.2.23 Code path tracing -^^^^^^^^^^^^^^^^^^^^^^^^ - -'tst_res' is a macro, so on when you define a function in one file: - -[source,c] -------------------------------------------------------------------------------- -int do_action(int arg) -{ - ... - - if (ok) { - tst_res(TPASS, "check passed"); - return 0; - } else { - tst_res(TFAIL, "check failed"); - return -1; - } -} -------------------------------------------------------------------------------- - -and call it from another file, the file and line reported by 'tst_res' in this -function will be from the former file. - -'TST_TRACE' can make the analysis of such situations easier. It's a macro which -inserts a call to 'tst_res(TINFO, ...)' in case its argument evaluates to -non-zero. In this call to 'tst_res(TINFO, ...)' the file and line will be -expanded using the actual location of 'TST_TRACE'. - -For example, if this another file contains: - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -if (TST_TRACE(do_action(arg))) { - ... -} -------------------------------------------------------------------------------- - -the generated output may look similar to: - -------------------------------------------------------------------------------- -common.h:9: FAIL: check failed -test.c:8: INFO: do_action(arg) failed -------------------------------------------------------------------------------- - -2.2.24 Tainted kernels -^^^^^^^^^^^^^^^^^^^^^^ - -If you need to detect whether a testcase triggers a kernel warning, bug or -oops, the following can be used to detect TAINT_W or TAINT_D: - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static struct tst_test test = { - ... - .taint_check = TST_TAINT_W | TST_TAINT_D, - ... -}; - -void run(void) -{ - ... - if (tst_taint_check() != 0) - tst_res(TFAIL, "kernel has issues"); - else - tst_res(TPASS, "kernel seems to be fine"); -} -------------------------------------------------------------------------------- - -To initialize taint checks, you have to set the taint flags you want to test -for in the 'taint_check' attribute of the tst_test struct. LTP library will -then automatically call 'tst_taint_init()' during test setup. The function -will generate a 'TCONF' if the requested flags are not fully supported on the -running kernel, and 'TBROK' if the kernel is already tainted before executing -the test. - -LTP library will then automatically check kernel taint at the end of testing. -If '.all_filesystems' is set in struct tst_test, taint check will be performed -after each file system and taint will abort testing early with 'TFAIL'. You -can optionally also call 'tst_taint_check()' during 'run()', which returns 0 -or the tainted flags set in '/proc/sys/kernel/tainted' as specified earlier. - -Depending on your kernel version, not all tainted-flags will be supported. - -For reference to tainted kernels, see kernel documentation: -Documentation/admin-guide/tainted-kernels.rst or -https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html - -2.2.25 Checksums -^^^^^^^^^^^^^^^^ - -CRC32c checksum generation is supported by LTP. In order to use it, the -test should include 'tst_checksum.h' header, then can call 'tst_crc32c()'. - -2.2.26 Checking kernel for the driver support -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some tests may need specific kernel drivers, either compiled in, or built -as a module. If '.needs_drivers' points to a 'NULL' terminated array of kernel -module names these are all checked and the test exits with 'TCONF' on the -first missing driver. - -Since it relies on modprobe command, the check will be skipped if the command -itself is not available on the system. - -2.2.27 Saving & restoring /proc|sys values -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -LTP library can be instructed to save and restore value of specified -(/proc|sys) files. This is achieved by initialized tst_test struct -field 'save_restore'. It is a 'NULL' terminated array of strings where -each string represents a file, whose value is saved at the beginning -and restored at the end of the test. Only first line of a specified -file is saved and restored. - -Pathnames can be optionally prefixed to specify how strictly (during -'store') are handled errors: - -* (no prefix) - test ends with 'TCONF', if file doesn't exist -* '?' - test prints info message and continues, - if file doesn't exist or open/read fails -* '!' - test ends with 'TBROK', if file doesn't exist - -'restore' is always strict and will TWARN if it encounters any error. - -[source,c] -------------------------------------------------------------------------------- -static const char *save_restore[] = { - "/proc/sys/kernel/core_pattern", - NULL, -}; - -static void setup(void) -{ - FILE_PRINTF("/proc/sys/kernel/core_pattern", "/mypath"); -} - -static struct tst_test test = { - ... - .setup = setup, - .save_restore = save_restore, -}; -------------------------------------------------------------------------------- - -2.2.28 Parsing kernel .config -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Generally testcases should attempt to autodetect as much kernel features as -possible based on the currently running kernel. We do have tst_check_driver() -to check if functionality that could be compiled as kernel module is present -on the system, disabled syscalls can be detected by checking for 'ENOSYS' -errno etc. - -However in rare cases core kernel features couldn't be detected based on the -kernel userspace API and we have to resort to parse the kernel .config. - -For this cases the test should set the 'NULL' terminated '.needs_kconfigs' -array of boolean expressions with constraints on the kconfig variables. The -boolean expression consits of variables, two binary operations '&' and '|', -negation '!' and correct sequence of parentesis '()'. Variables are expected -to be in a form of "CONFIG_FOO[=bar]". - -The test will continue to run if all expressions are evaluated to 'True'. -Missing variable is mapped to 'False' as well as variable with different than -specified value, e.g. 'CONFIG_FOO=bar' will evaluate to 'False' if the value -is anything else but 'bar'. If config variable is specified as plain -'CONFIG_FOO' it's evaluated to true it's set to any value (typically =y or =m). - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static const char *kconfigs[] = { - "CONFIG_X86_INTEL_UMIP | CONFIG_X86_UMIP", - NULL -}; - -static struct tst_test test = { - ... - .needs_kconfigs = kconfigs, - ... -}; -------------------------------------------------------------------------------- - -2.2.29 Changing the Wall Clock Time during test execution -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There are some tests that, for different reasons, might need to change the -system-wide clock time. Whenever this happens, it is imperative that the clock -is restored, at the end of test's execution, taking in consideration the amount -of time elapsed during that test. - -In order for that to happen, struct tst_test has a variable called -"restore_wallclock" that should be set to "1" so LTP knows it should: (1) -initialize a monotonic clock during test setup phase and (2) use that monotonic -clock to fix the system-wide clock time at the test cleanup phase. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static void setup(void) -{ - ... -} - -static void run(void) -{ - ... -} - -struct tst_test test = { - ... - .setup = setup, - .test_all = run, - .restore_wallclock = 1, - ... -}; -------------------------------------------------------------------------------- - -2.2.30 Testing similar syscalls in one test -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In some cases kernel has several very similar syscalls that do either the same -or very similar job. This is most noticeable on i386 where we commonly have -two or three syscall versions. That is because i386 was first platform that -Linux was developed on and because of that most mistakes in API happened there -as well. However this is not limited to i386 at all, it's quite common that -version two syscall has added missing flags parameters or so. - -In such cases it does not make much sense to copy&paste the test code over and -over, rather than that the test library provides support for test variants. -The idea behind test variants is simple, we run the test several times each -time with different syscall variant. - -The implementation consist of test_variants integer that, if set, denotes number -of test variants. The test is then forked and executed test_variants times each -time with different value in global tst_variant variable. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static int do_foo(void) -{ - switch (tst_variant) { - case 0: - return foo(); - case 1: - return syscall(__NR_foo); - } - - return -1; -} - -static void run(void) -{ - ... - - TEST(do_foo); - - ... -} - -static void setup(void) -{ - switch (tst_variant) { - case 0: - tst_res(TINFO, "Testing foo variant 1"); - break; - case 1: - tst_res(TINFO, "Testing foo variant 2"); - break; - } -} - -struct tst_test test = { - ... - .setup = setup, - .test_all = run, - .test_variants = 2, - ... -}; -------------------------------------------------------------------------------- - -2.2.31 Guarded buffers -^^^^^^^^^^^^^^^^^^^^^^ - -The test library supports guarded buffers, which are buffers allocated so -that: - -* The end of the buffer is followed by a PROT_NONE page - -* The remainder of the page before the buffer is filled with random canary - data - -Which means that the any access after the buffer will yield a Segmentation -fault or EFAULT depending on if the access happened in userspace or the kernel -respectively. The canary before the buffer will also catch any write access -outside of the buffer. - -The purpose of the patch is to catch off-by-one bugs which happens when -buffers and structures are passed to syscalls. New tests should allocate -guarded buffers for all data passed to the tested syscall which are passed by -a pointer. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static struct foo *foo_ptr; -static struct iovec *iov; -static void *buf_ptr; -static char *id; -... - -static void run(void) -{ - ... - - foo_ptr->bar = 1; - foo_ptr->buf = buf_ptr; - - ... -} - -static void setup(void) -{ - ... - - id = tst_strdup(string); - - ... -} - -static struct tst_test test = { - ... - .bufs = (struct tst_buffers []) { - {&foo_ptr, .size = sizeof(*foo_ptr)}, - {&buf_ptr, .size = BUF_SIZE}, - {&iov, .iov_sizes = (int[]){128, 32, -1}, - {} - } -}; -------------------------------------------------------------------------------- - -Guarded buffers can be allocated on runtime in a test setup() by a -'tst_alloc()' or by 'tst_strdup()' as well as by filling up the .bufs array in -the tst_test structure. - -So far the tst_test structure supports allocating either a plain buffer by -setting up the size or struct iovec, which is allocated recursively including -the individual buffers as described by an '-1' terminated array of buffer -sizes. - -2.2.32 Adding and removing capabilities -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some tests may require the presence or absence of particular -capabilities. Using the API provided by 'tst_capability.h' the test author can -try to ensure that some capabilities are either present or absent during the -test. - -For example; below we try to create a raw socket, which requires -CAP_NET_ADMIN. During setup we should be able to do it, then during run it -should be impossible. The LTP capability library will check before setup that -we have this capability, then after setup it will drop it. - -[source,c] --------------------------------------------------------------------------------- -#include "tst_test.h" -#include "tst_capability.h" -#include "tst_safe_net.h" - -#include "lapi/socket.h" - -static void run(void) -{ - TEST(socket(AF_INET, SOCK_RAW, 1)); - if (TST_RET > -1) { - tst_res(TFAIL, "Created raw socket"); - } else if (TST_ERR != EPERM) { - tst_res(TFAIL | TTERRNO, - "Failed to create socket for wrong reason"); - } else { - tst_res(TPASS | TTERRNO, "Didn't create raw socket"); - } -} - -static void setup(void) -{ - TEST(socket(AF_INET, SOCK_RAW, 1)); - if (TST_RET < 0) - tst_brk(TCONF | TTERRNO, "We don't have CAP_NET_RAW to begin with"); - - SAFE_CLOSE(TST_RET); -} - -static struct tst_test test = { - .setup = setup, - .test_all = run, - .caps = (struct tst_cap []) { - TST_CAP(TST_CAP_REQ, CAP_NET_RAW), - TST_CAP(TST_CAP_DROP, CAP_NET_RAW), - {} - }, -}; --------------------------------------------------------------------------------- - -Look at the test struct at the bottom. We have filled in the 'caps' field with -a 'NULL' terminated array containing two 'tst_cap' structs. 'TST_CAP_REQ' -actions are executed before setup and 'TST_CAP_DROP' are executed after -setup. This means it is possible to both request and drop a capability. - -[source,c] --------------------------------------------------------------------------------- -static struct tst_test test = { - .test_all = run, - .caps = (struct tst_cap []) { - TST_CAP(TST_CAP_REQ, CAP_NET_RAW), - TST_CAP(TST_CAP_DROP, CAP_SYS_ADMIN), - {} - }, -}; --------------------------------------------------------------------------------- - -Here we request 'CAP_NET_RAW', but drop 'CAP_SYS_ADMIN'. If the capability is -in the permitted set, but not the effective set, the library will try to -permit it. If it is not in the permitted set, then it will fail with 'TCONF'. - -This API does not require 'libcap' to be installed. However it has limited -features relative to 'libcap'. It only tries to add or remove capabilities -from the effective set. This means that tests which need to spawn child -processes may have difficulties ensuring the correct capabilities are -available to the children (see the capabilities (7) manual pages). - -However a lot of problems can be solved by using 'tst_cap_action(struct -tst_cap *cap)' directly which can be called at any time. This also helps if -you wish to drop a capability at the begining of setup. - -2.2.33 Reproducing race-conditions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If a bug is caused by two tasks in the kernel racing and you wish to create a -regression test (or bug-fix validation test) then the 'tst_fuzzy_sync.h' -library should be used. - -It allows you to specify, in your code, two race windows. One window in each -thread's loop (triggering a race usually requires many iterations). These -windows show fuzzy-sync where the race can happen. They don't need to be -exact, hence the 'fuzzy' part. If the race condition is not immediately -triggered then the library will begin experimenting with different timings. - -[source,c] --------------------------------------------------------------------------------- -#include "tst_fuzzy_sync.h" - -static struct tst_fzsync_pair fzsync_pair; - -static void setup(void) -{ - tst_fzsync_pair_init(&fzsync_pair); -} - -static void cleanup(void) -{ - tst_fzsync_pair_cleanup(&fzsync_pair); -} - -static void *thread_b(void *arg) -{ - while (tst_fzsync_run_b(&fzsync_pair)) { - - tst_fzsync_start_race_b(&fzsync_pair); - - /* This is the race window for thread B */ - - tst_fzsync_end_race_b(&fzsync_pair); - } - - return arg; -} - -static void thread_a(void) -{ - tst_fzsync_pair_reset(&fzsync_pair, thread_b); - - while (tst_fzsync_run_a(&fzsync_pair)) { - - tst_fzsync_start_race_a(&fzsync_pair); - - /* This is the race window for thread A */ - - tst_fzsync_end_race_a(&fzsync_pair); - } -} - -static struct tst_test test = { - .test_all = thread_a, - .setup = setup, - .cleanup = cleanup, -}; --------------------------------------------------------------------------------- - -Above is a minimal template for a test using fuzzy-sync. In a simple case, you -just need to put the bits you want to race inbetween 'start_race' and -'end_race'. Meanwhile, any setup you need to do per-iteration goes outside the -windows. - -Fuzzy sync synchronises 'run_a' and 'run_b', which act as barriers, so that -neither thread can progress until the other has caught up with it. There is -also the 'pair_wait' function which can be used to add barriers in other -locations. Of course 'start/end_race_a/b' are also a barriers. - -The library decides how long the test should run for based on the timeout -specified by the user plus some other heuristics. - -For full documentation see the comments in 'include/tst_fuzzy_sync.h'. - -2.2.34 Reserving hugepages -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Many of the LTP tests need to use hugepage in their testing, this allows the -test can reserve hugepages from system only via '.request_hugepages = xx'. - -If set non-zero number of 'request_hugepages', test will try to reserve the -expected number of hugepage for testing in setup phase. If system does not -have enough hpage for using, it will try the best to reserve 80% available -number of hpages. With success test stores the reserved hugepage number in -'tst_hugepages'. For the system without hugetlb supporting, variable -'tst_hugepages' will be set to 0. - -Also, we do cleanup and restore work for the hpages resetting automatically. - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static void run(void) -{ - ... - - if (tst_hugepages == test.request_hugepages) - TEST(do_hpage_test); - else - ... - ... -} - -struct tst_test test = { - .test_all = run, - .request_hugepages = 2, - ... -}; -------------------------------------------------------------------------------- - -or, - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" - -static void run(void) -{ - ... -} - -static void setup(void) -{ -? ? ? ? if (tst_hugepages != test.requested_hugepages) -? ? ? ? ? ? ? ? tst_brk(TCONF, "..."); -} - -struct tst_test test = { - .test_all = run, - .request_hugepages = 2, - ... -}; -------------------------------------------------------------------------------- - -2.2.35 Checking for required commands -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Required commands can be checked with '.needs_cmds', which points to a 'NULL' -terminated array of strings such as: - -[source,c] -------------------------------------------------------------------------------- -.needs_cmds = (const char *const []) { - "useradd", - "userdel", - NULL -}, -------------------------------------------------------------------------------- - -2.2.36 Assert sys or proc file value -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Using TST_ASSERT_INT/STR(path, val) to assert that integer value or string stored in -the prefix field of file pointed by path equals to the value passed to this function. - -Also having a similar api pair TST_ASSERT_FILE_INT/STR(path, prefix, val) to assert -the field value of file. - -2.2.36 Using Control Group -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some LTP tests need specific Control Group configurations. tst_cgroup.h provides -APIs to discover and use CGroups. There are many differences between CGroups API -V1 and V2. We encapsulate the details of configuring CGroups in high-level -functions which follow the V2 kernel API. Allowing one to use CGroups without -caring too much about the current system's configuration. - -Also, the LTP library will automatically mount/umount and configure the CGroup -hierarchies if that is required (e.g. if you run the tests from init with no -system manager). - -[source,c] -------------------------------------------------------------------------------- -#include "tst_test.h" -#include "tst_cgroup.h" - -static const struct tst_cgroup_group *cg; - -static void run(void) -{ - ... - // do test under cgroup - ... -} - -static void setup(void) -{ - tst_cgroup_require("memory", NULL); - cg = tst_cgroup_get_test_group(); - SAFE_CGROUP_PRINTF(cg, "cgroup.procs", "%d", getpid()); - SAFE_CGROUP_PRINTF(cg, "memory.max", "%lu", MEMSIZE); - if (SAFE_CGROUP_HAS(cg, "memory.swap.max")) - SAFE_CGROUP_PRINTF(cg, "memory.swap.max", "%zu", memsw); -} - -static void cleanup(void) -{ - tst_cgroup_cleanup(); -} - -struct tst_test test = { - .setup = setup, - .test_all = run, - .cleanup = cleanup, - ... -}; -------------------------------------------------------------------------------- - -Above, we first ensure the memory controller is available on the -test's CGroup with 'tst_cgroup_require'. We then get a structure, -'cg', which represents the test's CGroup. Note that -'tst_cgroup_get_test_group' should not be called many times, as it is -allocated in a guarded buffer (See section 2.2.31). Therefor it is -best to call it once in 'setup' and not 'run' because 'run' may be -repeated with the '-i' option. - -We then write the current processes PID into 'cgroup.procs', which -moves the current process into the test's CGroup. After which we set -the maximum memory size by writing to 'memory.max'. If the memory -controller is mounted on CGroups V1 then the library will actually -write to 'memory.limit_in_bytes'. As a general rule, if a file exists -on both CGroup versions, then we use the V2 naming. - -Some controller features, such as 'memory.swap', can be -disabled. Therefor we need to check if they exist before accessing -them. This can be done with 'SAFE_CGROUP_HAS' which can be called on -any control file or feature. - -Most tests only require setting a few limits similar to the above. In -such cases the differences between V1 and V2 are hidden. Setup and -cleanup is also mostly hidden. However things can get much worse. - -[source,c] -------------------------------------------------------------------------------- -static const struct tst_cgroup_group *cg; -static const struct tst_cgroup_group *cg_drain; -static struct tst_cgroup_group *cg_child; - -static void run(void) -{ - char buf[BUFSIZ]; - size_t mem = 0; - - cg_child = tst_cgroup_group_mk(cg, "child"); - SAFE_CGROUP_PRINTF(cg_child, "cgroup.procs", "%d", getpid()); - - if (TST_CGROUP_VER(cg, "memory") != TST_CGROUP_V1) - SAFE_CGROUP_PRINT(cg, "cgroup.subtree_control", "+memory"); - if (TST_CGROUP_VER(cg, "cpuset") != TST_CGROUP_V1) - SAFE_CGROUP_PRINT(cg, "cgroup.subtree_control", "+cpuset"); - - if (!SAFE_FORK()) { - SAFE_CGROUP_PRINTF(cg_child, "cgroup.procs", "%d", getpid()); - - if (SAFE_CGROUP_HAS(cg_child, "memory.swap")) - SAFE_CGROUP_SCANF(cg_child, "memory.swap.current", "%zu", &mem); - SAFE_CGROUP_READ(cg_child, "cpuset.mems", buf, sizeof(buf)); - - // Do something with cpuset.mems and memory.current values - ... - - exit(0); - } - - tst_reap_children(); - SAFE_CGROUP_PRINTF(cg_drain, "cgroup.procs", "%d", getpid()); - cg_child = tst_cgroup_group_rm(cg_child); -} - -static void setup(void) -{ - tst_cgroup_require("memory", NULL); - tst_cgroup_require("cpuset", NULL); - - cg = tst_cgroup_get_test_group(); - cg_drain = tst_cgroup_get_drain_group(); -} - -static void cleanup(void) -{ - if (cg_child) { - SAFE_CGROUP_PRINTF(cg_drain, "cgroup.procs", "%d", getpid()); - cg_child = tst_cgroup_group_rm(cg_child); - } - - tst_cgroup_cleanup(); -} - -struct tst_test test = { - .setup = setup, - .test_all = run, - .cleanup = cleanup, - ... -}; -------------------------------------------------------------------------------- - -Starting with setup; we can see here that we also fetch the 'drain' -CGroup. This is a shared group (between parallel tests) which may -contain processes from other tests. It should have default settings and -these should not be changed by the test. It can be used to remove -processes from other CGroups incase the hierarchy root is not -accessible. - -In 'run', we first create a child CGroup with 'tst_cgroup_mk'. As we -create this CGroup in 'run' we should also remove it at the end of -run. We also need to check if it exists and remove it in cleanup as -well. Because there are 'SAFE_' functions which may jump to cleanup. - -We then move the main test process into the child CGroup. This is -important as it means that before we destroy the child CGroup we have -to move the main test process elsewhere. For that we use the 'drain' -group. - -Next we enable the memory and cpuset controller configuration on the -test CGroup's descendants (i.e. 'cg_child'). This allows each child to -have its own settings. The file 'cgroup.subtree_control' does not -exist on V1. Because it is possible to have both V1 and V2 active at -the same time. We can not simply check if 'subtree_control' exists -before writing to it. We have to check if a particular controller is -on V2 before trying to add it to 'subtree_control'. Trying to add a V1 -controller will result in 'ENOENT'. - -We then fork a child process and add this to the child CGroup. Within -the child process we try to read 'memory.swap.current'. It is possible -that the memory controller was compiled without swap support, so it is -necessary to check if 'memory.swap' is enabled. That is unless the -test will never reach the point where 'memory.swap.*' are used without -swap support. - -The parent process waits for the child process to be reaped before -destroying the child CGroup. So there is no need to transfer the child -to drain. However the parent process must be moved otherwise we will -get 'EBUSY' when trying to remove the child CGroup. - -Another example of an edge case is the following. - -[source,c] -------------------------------------------------------------------------------- - if (TST_CGROUP_VER(cg, "memory") == TST_CGROUP_V1) - SAFE_CGROUP_PRINTF(cg, "memory.swap.max", "%lu", ~0UL); - else - SAFE_CGROUP_PRINT(cg, "memory.swap.max", "max"); -------------------------------------------------------------------------------- - -CGroups V2 introduced a feature where 'memory[.swap].max' could be set to -"max". This does not appear to work on V1 'limit_in_bytes' however. For most -tests, simply using a large number is sufficient and there is no need to use -"max". Importantly though, one should be careful to read both the V1 and V2 -kernel docs. The LTP library can not handle all edge cases. It does the minimal -amount of work to make testing on both V1 and V2 feasible. - -2.2.37 Require minimum numbers of CPU for a testcase -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some tests require more than specific number of CPU. It can be defined with -`.min_cpus = N`. - -2.2.38 Test tags -^^^^^^^^^^^^^^^^ - -Test tags are name-value pairs that can hold any test metadata. - -We have additional support for CVE entries, git commit in mainline kernel, -stable kernel or glibc git repository. If a test is a regression test it -should include these tags. They are printed when test fails and exported -into documentation. - -CVE, mainline and stable kernel git commits in a regression test for a kernel bug: -[source,c] -------------------------------------------------------------------------------- -struct tst_test test = { - ... - .tags = (const struct tst_tag[]) { - {"linux-git", "9392a27d88b9"}, - {"linux-git", "ff002b30181d"}, - {"linux-stable-git", "c4a23c852e80"}, - {"CVE", "2020-29373"}, - {} - } -}; -------------------------------------------------------------------------------- - -Glibc git commit in a regression test for a glibc bug: -[source,c] -------------------------------------------------------------------------------- -struct tst_test test = { - ... - .tags = (const struct tst_tag[]) { - {"glibc-git", "574500a108be"}, - {} - } -}; -------------------------------------------------------------------------------- - -2.3 Writing a testcase in shell -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -LTP supports testcases to be written in a portable shell too. - -There is a shell library modeled closely to the C interface at -'testcases/lib/tst_test.sh'. - -WARNING: All identifiers starting with TST_ or tst_ are reserved for the - test library. - -2.3.1 Basic test interface -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-or-later -# This is a basic test for true shell builtin - -TST_TESTFUNC=do_test -. tst_test.sh - -do_test() -{ - true - ret=$? - - if [ $ret -eq 0 ]; then - tst_res TPASS "true returned 0" - else - tst_res TFAIL "true returned $ret" - fi -} - -tst_run -------------------------------------------------------------------------------- - -TIP: To execute this test the 'tst_test.sh' library must be in '$PATH'. If you - are executing the test from a git checkout you can run it as - 'PATH="$PATH:../../lib" ./foo01.sh' - -The shell library expects test setup, cleanup and the test function executing -the test in the '$TST_SETUP', '$TST_CLEANUP' and '$TST_TESTFUNC' variables. - -Both '$TST_SETUP' and '$TST_CLEANUP' are optional. - -The '$TST_TESTFUNC' may be called several times if more than one test -iteration was requested by passing right command line options to the test. - -The '$TST_CLEANUP' may be called even in the middle of the setup and must be -able to clean up correctly even in this situation. The easiest solution for -this is to keep track of what was initialized and act accordingly in the -cleanup. - -WARNING: Similar to the C library, calling 'tst_brk' in the $TST_CLEANUP does - not exit the test and 'TBROK' is converted to 'TWARN'. - -Notice also the 'tst_run' shell API function called@the end of the test that -actually starts the test. - -WARNING: cleanup function is called only after 'tst_run' has been started. -Calling 'tst_brk' in shell libraries, e.g. 'tst_test.sh' or 'tst_net.sh' does -not trigger calling it. - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-or-later -# Example test with tests in separate functions - -TST_TESTFUNC=test -TST_CNT=2 -. tst_test.sh - -test1() -{ - tst_res TPASS "Test $1 passed" -} - -test2() -{ - tst_res TPASS "Test $1 passed" -} - -tst_run -# output: -# foo 1 TPASS: Test 1 passed -# foo 2 TPASS: Test 2 passed -------------------------------------------------------------------------------- - -If '$TST_CNT' is set, the test library looks if there are functions named -'$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are -found they are executed one by one. The test number is passed to it in the '$1'. - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-or-later -# Example test with tests in a single function - -TST_TESTFUNC=do_test -TST_CNT=2 -. tst_test.sh - -do_test() -{ - case $1 in - 1) tst_res TPASS "Test $1 passed";; - 2) tst_res TPASS "Test $1 passed";; - esac -} - -tst_run -# output: -# foo 1 TPASS: Test 1 passed -# foo 2 TPASS: Test 2 passed -------------------------------------------------------------------------------- - -Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., -the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed -to it in the '$1'. - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-or-later -# Example test with tests in a single function, using $TST_TEST_DATA and -# $TST_TEST_DATA_IFS - -TST_TESTFUNC=do_test -TST_TEST_DATA="foo:bar:d dd" -TST_TEST_DATA_IFS=":" -. tst_test.sh - -do_test() -{ - tst_res TPASS "Test $1 passed with data '$2'" -} - -tst_run -# output: -# foo 1 TPASS: Test 1 passed with data 'foo' -# foo 2 TPASS: Test 1 passed with data 'bar' -# foo 3 TPASS: Test 1 passed with data 'd dd' -------------------------------------------------------------------------------- - -It's possible to pass data for function with '$TST_TEST_DATA'. Optional -'$TST_TEST_DATA_IFS' is used for splitting, default value is space. - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-or-later -# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT - -TST_TESTFUNC=do_test -TST_CNT=2 -TST_TEST_DATA="foo bar" -. tst_test.sh - -do_test() -{ - case $1 in - 1) tst_res TPASS "Test $1 passed with data '$2'";; - 2) tst_res TPASS "Test $1 passed with data '$2'";; - esac -} - -tst_run -# output: -# foo 1 TPASS: Test 1 passed with data 'foo' -# foo 2 TPASS: Test 2 passed with data 'foo' -# foo 3 TPASS: Test 1 passed with data 'bar' -# foo 4 TPASS: Test 2 passed with data 'bar' -------------------------------------------------------------------------------- - -'$TST_TEST_DATA' can be used with '$TST_CNT'. If '$TST_TEST_DATA_IFS' not specified, -space as default value is used. Of course, it's possible to use separate functions. - -2.3.2 Library environment variables and functions for shell -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Similarily to the C library various checks and preparations can be requested -simply by setting right '$TST_NEEDS_FOO'. - -[options="header"] -|============================================================================= -| Variable name | Action done -| 'TST_NEEDS_ROOT' | Exit the test with 'TCONF' unless executed under root. -| | Alternatively the 'tst_require_root' command can be used. -| 'TST_NEEDS_TMPDIR' | Create test temporary directory and cd into it. -| 'TST_NEEDS_DEVICE' | Prepare test temporary device, the path to testing - device is stored in '$TST_DEVICE' variable. - The option implies 'TST_NEEDS_TMPDIR'. -| 'TST_NEEDS_CMDS' | String with command names that has to be present for - the test (see below). -| 'TST_NEEDS_MODULE' | Test module name needed for the test (see below). -| 'TST_NEEDS_DRIVERS'| Checks kernel drivers support for the test. -| 'TST_TIMEOUT' | Maximum timeout set for the test in sec. Must be int >= 1, - or -1 (special value to disable timeout), default is 300. - Variable is meant be set in tests, not by user. - It's an equivalent of `tst_test.timeout` in C, can be set - via 'tst_set_timeout(timeout)' after test has started. -|============================================================================= - -[options="header"] -|============================================================================= -| Function name | Action done -| 'tst_set_timeout(timeout)' | Maximum timeout set for the test in sec. - See 'TST_TIMEOUT' variable. -|============================================================================= - -NOTE: Network tests (see testcases/network/README.md) use additional variables -and functions in 'tst_net.sh'. - -Checking for presence of commands -+++++++++++++++++++++++++++++++++ - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh - -... - -TST_NEEDS_CMDS="modinfo modprobe" -. tst_test.sh - -... - -------------------------------------------------------------------------------- - -Setting '$TST_NEEDS_CMDS' to a string listing required commands will check for -existence each of them and exits the test with 'TCONF' on first missing. - -Alternatively the 'tst_require_cmds()' function can be used to do the same on -runtime, since sometimes we need to the check at runtime too. - -'tst_check_cmds()' can be used for requirements just for a particular test -as it doesn't exit (it issues 'tst_res TCONF'). Expected usage is: - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh - -TST_TESTFUNC=do_test -. tst_test.sh - -do_test() -{ - tst_check_cmds cmd || return - cmd --foo - ... -} - -tst_run -------------------------------------------------------------------------------- - -Locating kernel modules -+++++++++++++++++++++++ - -The LTP build system can build kernel modules as well, setting -'$TST_NEEDS_MODULE' to module name will cause the library to look for the -module in a few possible paths. - -If module was found the path to it will be stored into '$TST_MODPATH' -variable, if module wasn't found the test will exit with 'TCONF'. - -Alternatively the 'tst_require_module()' function can be used to do the same -at runtime. - -2.3.3 Optional command line parameters -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-or-later -# Optional test command line parameters - -TST_OPTS="af:" -TST_USAGE=usage -TST_PARSE_ARGS=parse_args -TST_TESTFUNC=do_test - -. tst_test.sh - -ALTERNATIVE=0 -MODE="foo" - -usage() -{ - cat << EOF -usage: $0 [-a] [-f ] - -OPTIONS --a Enable support for alternative foo --f Specify foo or bar mode -EOF -} - -parse_args() -{ - case $1 in - a) ALTERNATIVE=1;; - f) MODE="$2";; - esac -} - -do_test() -{ - ... -} - -tst_run -------------------------------------------------------------------------------- - -The 'getopts' string for optional parameters is passed in the '$TST_OPTS' -variable. There are a few default parameters that cannot be used by a test, -these can be listed with passing help '-h' option to any test. - -The function that prints the usage is passed in '$TST_USAGE', the help for -the options implemented in the library is appended when usage is printed. - -Lastly the function '$PARSE_ARGS' is called with the option name in the '$1' -and, if option has argument, its value in the '$2'. - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-or-later -# Optional test positional parameters - -TST_POS_ARGS=3 -TST_USAGE=usage -TST_TESTFUNC=do_test - -. tst_test.sh - -usage() -{ - cat << EOF -usage: $0 [min] [max] [size] - -EOF -} - -min="$1" -max="$2" -size="$3" - -do_test() -{ - ... -} - -tst_run -------------------------------------------------------------------------------- - -You can also request a number of positional parameters by setting the -'$TST_POS_ARGS' variable. If you do, these will be available as they were -passed directly to the script in '$1', '$2', ..., '$n'. - -2.3.4 Useful library functions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Retrieving configuration variables -++++++++++++++++++++++++++++++++++ - -You may need to retrieve configuration values such as PAGESIZE, there is -'getconf' but as some system may not have it, you are advised to use -'tst_getconf' instead. Note that it implements subset of 'getconf' -system variables used by the testcases only. - -[source,sh] -------------------------------------------------------------------------------- -# retrieve PAGESIZE -pagesize=`tst_getconf PAGESIZE` -------------------------------------------------------------------------------- - -Sleeping for subsecond intervals -++++++++++++++++++++++++++++++++ - -Albeit there is a sleep command available basically everywhere not all -implementations can support sleeping for less than one second. And most of the -time sleeping for a second is too much. Therefore LTP includes 'tst_sleep' -that can sleep for defined amount of seconds, milliseconds or microseconds. - -[source,sh] -------------------------------------------------------------------------------- -# sleep for 100 milliseconds -tst_sleep 100ms -------------------------------------------------------------------------------- - -Retry a function call multiple times -++++++++++++++++++++++++++++++++++++ - -Sometimes an LTP test needs to retry a function call multiple times because -the system is not ready to process it successfully on the first try. The LTP -library has useful tools to handle the call retry automatically. -'TST_RETRY_FUNC()' will keep retrying for up to 1 second. If you want a custom -time limit use 'TST_RETRY_FN_EXP_BACKOFF()'. Both methods return the value -returned by the last 'FUNC' call. - -The delay between retries starts@1 microsecond and doubles after each call. -The retry loop ends when the function call succeeds or when the next delay -exceeds the specified time (1 second for 'TST_RETRY_FUNC()'). The maximum -delay is multiplied by TST_TIMEOUT_MUL. The total cumulative delay may be up -to twice as long as the adjusted maximum delay. - -The C version of 'TST_RETRY_FUNC()' is a macro which takes two arguments: - -* 'FUNC' is the complete function call with arguments which should be retried - multiple times. -* 'SUCCESS_CHECK' is a macro or function which will validate 'FUNC' return - value. 'FUNC' call was successful if 'SUCCESS_CHECK(ret)' evaluates to - non-zero. - -Both retry methods clear 'errno' before every 'FUNC' call so your -'SUCCESS_CHECK' can look for specific error codes as well. The LTP library -also includes predefined 'SUCCESS_CHECK' macros for the most common call -conventions: - -* 'TST_RETVAL_EQ0()' - The call was successful if 'FUNC' returned 0 or NULL -* 'TST_RETVAL_NOTNULL()' - The call was successful if 'FUNC' returned any - value other than 0 or NULL. -* 'TST_RETVAL_GE0()' - The call was successful if 'FUNC' returned value >= 0. - -[source,c] -------------------------------------------------------------------------------- -/* Keep trying for 1 second */ -TST_RETRY_FUNC(FUNC, SUCCESS_CHECK) - -/* Keep trying for up to 2*N seconds */ -TST_RETRY_FN_EXP_BACKOFF(FUNC, SUCCESS_CHECK, N) -------------------------------------------------------------------------------- - -The shell version of 'TST_RETRY_FUNC()' is simpler and takes slightly -different arguments: - -* 'FUNC' is a string containing the complete function or program call with - arguments. -* 'EXPECTED_RET' is a single expected return value. 'FUNC' call was successful - if the return value is equal to EXPECTED_RET. - -[source,sh] -------------------------------------------------------------------------------- -# Keep trying for 1 second -TST_RETRY_FUNC "FUNC arg1 arg2 ..." "EXPECTED_RET" - -# Keep trying for up to 2*N seconds -TST_RETRY_FN_EXP_BACKOFF "FUNC arg1 arg2 ..." "EXPECTED_RET" "N" -------------------------------------------------------------------------------- - -Checking for integers -+++++++++++++++++++++ - -[source,sh] -------------------------------------------------------------------------------- -# returns zero if passed an integer parameter, non-zero otherwise -tst_is_int "$FOO" -------------------------------------------------------------------------------- - -Checking for integers and floating point numbers -++++++++++++++++++++++++++++++++++++++++++++++++ - -[source,sh] -------------------------------------------------------------------------------- -# returns zero if passed an integer or floating point number parameter, -# non-zero otherwise -tst_is_num "$FOO" -------------------------------------------------------------------------------- - -Obtaining random numbers -++++++++++++++++++++++++ - -There is no '$RANDOM' in portable shell, use 'tst_random' instead. - -[source,sh] -------------------------------------------------------------------------------- -# get random integer between 0 and 1000 (including 0 and 1000) -tst_random 0 1000 -------------------------------------------------------------------------------- - -Formatting device with a filesystem -+++++++++++++++++++++++++++++++++++ - -The 'tst_mkfs' helper will format device with the filesystem. - -[source,sh] -------------------------------------------------------------------------------- -# format test device with ext2 -tst_mkfs ext2 $TST_DEVICE -# default params are $TST_FS_TYPE $TST_DEVICE -tst_mkfs -# optional parameters -tst_mkfs ext4 /dev/device -T largefile -------------------------------------------------------------------------------- - -Mounting and unmounting filesystems -+++++++++++++++++++++++++++++++++++ - -The 'tst_mount' and 'tst_umount' helpers are a safe way to mount/umount -a filesystem. - -The 'tst_mount' mounts '$TST_DEVICE' of '$TST_FS_TYPE' (optional) to -'$TST_MNTPOINT' (defaults to mntpoint), optionally using the -'$TST_MNT_PARAMS'. The '$TST_MNTPOINT' directory is created if it didn't -exist prior to the function call. - -If the path passed (optional, defaults to '$TST_DEVICE') to the 'tst_umount' is -not mounted (present in '/proc/mounts') it's noop. -Otherwise it retries to umount the filesystem a few times on a failure, which -is a workaround since there are a daemons dumb enough to probe all newly -mounted filesystems, which prevents them from umounting shortly after they -were mounted. - -ROD and ROD_SILENT -++++++++++++++++++ - -These functions supply the 'SAFE_MACROS' used in C although they work and are -named differently. - -[source,sh] -------------------------------------------------------------------------------- -ROD_SILENT command arg1 arg2 ... - -# is shorthand for: - -command arg1 arg2 ... > /dev/null 2>&1 -if [ $? -ne 0 ]; then - tst_brk TBROK "..." -fi - - -ROD command arg1 arg2 ... - -# is shorthand for: - -ROD arg1 arg2 ... -if [ $? -ne 0 ]; then - tst_brk TBROK "..." -fi -------------------------------------------------------------------------------- - -WARNING: Keep in mind that output redirection (to a file) happens in the - caller rather than in the ROD function and cannot be checked for - write errors by the ROD function. - -As a matter of a fact doing +ROD echo a > /proc/cpuinfo+ would work just fine -since the 'ROD' function will only get the +echo a+ part that will run just -fine. - -[source,sh] -------------------------------------------------------------------------------- -# Redirect output to a file with ROD -ROD echo foo \> bar -------------------------------------------------------------------------------- - -Note the '>' is escaped with '\', this causes that the '>' and filename are -passed to the 'ROD' function as parameters and the 'ROD' function contains -code to split '$@' on '>' and redirects the output to the file. - -EXPECT_PASS{,_BRK} and EXPECT_FAIL{,_BRK} -+++++++++++++++++++++++++++++++++++++++++ - -[source,sh] -------------------------------------------------------------------------------- -EXPECT_PASS command arg1 arg2 ... [ \> file ] -EXPECT_FAIL command arg1 arg2 ... [ \> file ] -------------------------------------------------------------------------------- - -'EXPECT_PASS' calls 'tst_res TPASS' if the command exited with 0 exit code, -and 'tst_res TFAIL' otherwise. 'EXPECT_FAIL' does vice versa. - -Output redirection rules are the same as for the 'ROD' function. In addition -to that, 'EXPECT_FAIL' always redirects the command's stderr to '/dev/null'. - -There are also 'EXPECT_PASS_BRK' and 'EXPECT_FAIL_BRK', which works the same way -except breaking a test when unexpected action happen. - -It's possible to detect whether expected value happened: -[source,sh] -------------------------------------------------------------------------------- -if ! EXPECT_PASS command arg1 2\> /dev/null; then - continue -fi -------------------------------------------------------------------------------- - -tst_kvcmp -+++++++++ - -This command compares the currently running kernel version given conditions -with syntax similar to the shell test command. - -[source,sh] -------------------------------------------------------------------------------- -# Exit the test if kernel version is older or equal to 2.6.8 -if tst_kvcmp -le 2.6.8; then - tst_brk TCONF "Kernel newer than 2.6.8 is needed" -fi - -# Exit the test if kernel is newer than 3.8 and older than 4.0.1 -if tst_kvcmp -gt 3.8 -a -lt 4.0.1; then - tst_brk TCONF "Kernel must be older than 3.8 or newer than 4.0.1" -fi -------------------------------------------------------------------------------- - -[options="header"] -|======================================================================= -| expression | description -| -eq kver | Returns true if kernel version is equal -| -ne kver | Returns true if kernel version is not equal -| -gt kver | Returns true if kernel version is greater -| -ge kver | Returns true if kernel version is greater or equal -| -lt kver | Returns true if kernel version is lesser -| -le kver | Returns true if kernel version is lesser or equal -| -a | Does logical and between two expressions -| -o | Does logical or between two expressions -|======================================================================= - -The format for kernel version has to either be with one dot e.g. '2.6' or with -two dots e.g. '4.8.1'. - -.tst_fs_has_free -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh - -... - -# whether current directory has 100MB free space at least. -if ! tst_fs_has_free . 100MB; then - tst_brkm TCONF "Not enough free space" -fi - -... -------------------------------------------------------------------------------- - -The 'tst_fs_has_free' shell interface returns 0 if the specified free space is -satisfied, 1 if not, and 2 on error. - -The second argument supports suffixes kB, MB and GB, the default unit is Byte. - -.tst_retry -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh - -... - -# Retry ping command three times -tst_retry "ping -c 1 127.0.0.1" - -if [ $? -ne 0 ]; then - tst_resm TFAIL "Failed to ping 127.0.0.1" -else - tst_resm TPASS "Successfully pinged 127.0.0.1" -fi - -... -------------------------------------------------------------------------------- - -The 'tst_retry' function allows you to retry a command after waiting small -amount of time until it succeeds or until given amount of retries has been -reached (default is three attempts). - -2.3.5 Restarting daemons -^^^^^^^^^^^^^^^^^^^^^^^^ - -Restarting system daemons is a complicated task for two reasons. - -* There are different init systems - (SysV init, systemd, etc...) - -* Daemon names are not unified between distributions - (apache vs httpd, cron vs crond, various syslog variations) - -To solve these problems LTP has 'testcases/lib/daemonlib.sh' library that -provides functions to start/stop/query daemons as well as variables that store -correct daemon name. - -.Supported operations -|============================================================================== -| start_daemon() | Starts daemon, name is passed as first parameter. -| stop_daemon() | Stops daemon, name is passed as first parameter. -| restart_daemon() | Restarts daemon, name is passed as first parameter. -| status_daemon() | Detect daemon status (exit code: 0: running, 1: not running). -|============================================================================== - -.Variables with detected names -|============================================================================== -| CROND_DAEMON | Cron daemon name (cron, crond). -| SYSLOG_DAEMON | Syslog daemon name (syslog, syslog-ng, rsyslog). -|============================================================================== - -.Cron daemon restart example -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-or-later -# Cron daemon restart example - -TCID=cron01 -TST_COUNT=1 -. test.sh -. daemonlib.sh - -... - -restart_daemon $CROND_DAEMON - -... - -tst_exit -------------------------------------------------------------------------------- - -2.3.6 Access to the checkpoint interface -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The shell library provides an implementation of the checkpoint interface -compatible with the C version. All 'TST_CHECKPOINT_*' functions are available. - -In order to initialize checkpoints '$TST_NEEDS_CHECKPOINTS' must be set to '1' -before the inclusion of 'test.sh': - -[source,sh] -------------------------------------------------------------------------------- -#!/bin/sh - -TST_NEEDS_CHECKPOINTS=1 -. test.sh -------------------------------------------------------------------------------- - -Since both the implementations are compatible, it's also possible to start -a child binary process from a shell test and synchronize with it. This process -must have checkpoints initialized by calling 'tst_reinit()'. - -3. Common problems ------------------- - -This chapter describes common problems/misuses and less obvious design patters -(quirks) in UNIX interfaces. Read it carefully :) - -3.1 umask() -~~~~~~~~~~~ - -I've been hit by this one several times already... When you create files -with 'open()' or 'creat()' etc, the mode specified as the last parameter *is -not* the mode the file is created with. The mode depends on current 'umask()' -settings which may clear some of the bits. If your test depends on specific -file permissions you need either to change umask to 0 or 'chmod()' the file -afterwards or use 'SAFE_TOUCH()' that does the 'chmod()' for you. - -3.2 access() -~~~~~~~~~~~ - -If 'access(some_file, W_OK)' is executed by root, it will return success even -if the file doesn't have write permission bits set (the same holds for R_OK -too). For sysfs files you can use 'open()' as a workaround to check file -read/write permissions. It might not work for other filesystems, for these you -have to use 'stat()', 'lstat()' or 'fstat()'. - -3.3 umount() EBUSY -~~~~~~~~~~~~~~~~~~ - -Various desktop daemons (gvfsd-trash is known for that) may be stupid enough -to probe all newly mounted filesystem which results in 'umount(2)' failing -with 'EBUSY'; use 'tst_umount()' described in 2.2.19 that retries in this case -instead of plain 'umount(2)'. - -3.4 FILE buffers and fork() -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Be vary that if a process calls 'fork(2)' the child process inherits open -descriptors as well as copy of the parent memory so especially if there are -any open 'FILE' buffers with a data in them they may be written both by the -parent and children resulting in corrupted/duplicated data in the resulting -files. - -Also open 'FILE' streams are flushed and closed at 'exit(3)' so if your -program works with 'FILE' streams, does 'fork(2)', and the child may end up -calling 'exit(3)' you will likely end up with corrupted files. - -The solution to this problem is either simply call 'fflush(NULL)' that flushes -all open output 'FILE' streams just before doing 'fork(2)'. You may also use -'_exit(2)' in child processes which does not flush 'FILE' buffers and also -skips 'atexit(3)' callbacks. - -4. Test Contribution Checklist ------------------------------- +NOTE: See also + https://github.com/linux-test-project/ltp/wiki/Maintainer-Patch-Review-Checklist[Maintainer Patch Review Checklist]. 1. Test compiles and runs fine (check with `-i 10` too) 2. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl[checkpatch.pl] @@ -3219,8 +251,7 @@ skips 'atexit(3)' callbacks. 4. Test binaries are added into corresponding '.gitignore' files 5. Patches apply over the latest git - -4.1 About .gitignore files +3.1 About .gitignore files ~~~~~~~~~~~~~~~~~~~~~~~~~~ There are numerous '.gitignore' files in the LTP tree. Usually there is a -- 2.31.1