[PATCH 00/17] nolibc: add preliminary self tests

[PATCH 00/17] nolibc: add preliminary self tests

[Willy Tarreau]

Hi Paul,

as previously promised, here comes the nolibc update which introduces the
minimal self-test infrastructure that aims at being reasonably easy to
expand further.

It's based on your branch "dev.2022.06.30b" that contains the previous
minor fixes that aimed at addressing Linus' concerns about the build
process inconsistencies.

The way it works tries to mimmick as much as possible the regular build
process, so that it reuses the same ARCH, CC, CROSS_COMPILE to build the
test program, that will be embedded into an initramfs and the kernel is
(re)built with that initramfs. Then you can decide to run that kernel
under QEMU for the supported archs, and the output of the tests appears
in an output text file in a format that's easily greppable and diffable.
A single target "run" does everything.

By default it will reuse your existing .config (so that developers
continue to use their regular config handling), though it can also
create a known-to-work defconfig for each arch. The reason behind this
is that it took me a moment to figure certain defconfig + machine name
combinations and I found it better to put them there once for all.

I've successfully tested it on arm, arm64, i386, x86_64. riscv64 works
except two syscalls which return unexpected errors, and mips segfaults
in sbrk(). I don't know why yet, but this proves that it's worth having
such a test.

There are not that many tests yet (71), those that have to run can be
filtered either from the program's command line or from a NOLIBC_TEST
environment variable so that it's possible to skip broken ones or to
focus on a few ranges only.

Tests are numerically numbered, and are conveniently handled in a
switch/case statement so that a relative line number assigns the number
to the test. That's convenient because the vast majority of syscall tests
are one-liners. This sometimes slightly upsets check-patch when lines get
moderately long but that significantly improves legibility.

There are expectation for both successes and failures (e.g. -1 ENOTDIR).
I'm sure this can be improved later (and that's the goal). Right now it
covers two test families:
  - syscalls
  - stdlib    (str* functions mostly)

I suspect that over time we might want to split syscalls into different
parts (e.g. core, fs, etc maybe) but I could be wrong.

The program can automatically modulate QEMU's return value on x86 when
QEMU is run with the appropriate options, but for now I'm not using it
as I felt like it didn't bring much value, and the output is more useful.
That's debatable, and maybe some might want to use it in bisect scripts
for example. It's too early to say IMHO.

Oh, I also arranged the code so that the test also builds with glibc. I
noticed that when adding a new test that fails, sometimes it's convenient
to see if it's the nolibc part that's broken or the test. I don't find
this critical but the required includes and ifdefs are there so that it
should be easy to maintain over time as well.

I'm obviously interested in comments, but really, I don't want to
overdesign something for a first step, it remains a very modest test
program and I'd like that it remains easy to hack on it and to contribute
new tests that are deemed useful.

I'm CCing the few who already contributed some patches and/or expressed
interest, as well as Linus who had a first bad experience when trying to
test it, hoping this one will be better. I'm pasting below [1] a copy of
a test on x86_64 below, that's summed up as "71 test(s) passed" at the
end of the "run" target.

If there's no objection, it would be nice to have this with your current
series, as it definitely helps spot and fix the bugs. In parallel I'll see
if I can figure the problems with the two tests that fail each on a
specific arch and I might possibly have a few extra fixes for the current
nolibc.

Thank you!
Willy

[1] example output
----8<----
Running test 'syscall'
0 getpid = 1                             [OK]
1 getppid = 0                            [OK]
5 getpgid_self = 0                       [OK]
6 getpgid_bad = -1 ESRCH                 [OK]
7 kill_0 = 0                             [OK]
8 kill_CONT = 0                          [OK]
9 kill_BADPID = -1 ESRCH                 [OK]
10 sbrk = 0                              [OK]
11 brk = 0                               [OK]
12 chdir_root = 0                        [OK]
13 chdir_dot = 0                         [OK]
14 chdir_blah = -1 ENOENT                [OK]
15 chmod_net = 0                         [OK]
16 chmod_self = -1 EPERM                 [OK]
17 chown_self = -1 EPERM                 [OK]
18 chroot_root = 0                       [OK]
19 chroot_blah = -1 ENOENT               [OK]
20 chroot_exe = -1 ENOTDIR               [OK]
21 close_m1 = -1 EBADF                   [OK]
22 close_dup = 0                         [OK]
23 dup_0 = 3                             [OK]
24 dup_m1 = -1 EBADF                     [OK]
25 dup2_0 = 100                          [OK]
26 dup2_m1 = -1 EBADF                    [OK]
27 dup3_0 = 100                          [OK]
28 dup3_m1 = -1 EBADF                    [OK]
29 execve_root = -1 EACCES               [OK]
30 getdents64_root = 120                 [OK]
31 getdents64_null = -1 ENOTDIR          [OK]
32 gettimeofday_null = 0                 [OK]
38 ioctl_tiocinq = 0                     [OK]
39 ioctl_tiocinq = 0                     [OK]
40 link_root1 = -1 EEXIST                [OK]
41 link_blah = -1 ENOENT                 [OK]
42 link_dir = -1 EPERM                   [OK]
43 link_cross = -1 EXDEV                 [OK]
44 lseek_m1 = -1 EBADF                   [OK]
45 lseek_0 = -1 ESPIPE                   [OK]
46 mkdir_root = -1 EEXIST                [OK]
47 open_tty = 3                          [OK]
48 open_blah = -1 ENOENT                 [OK]
49 poll_null = 0                         [OK]
50 poll_stdout = 1                       [OK]
51 poll_fault = -1 EFAULT                [OK]
52 read_badf = -1 EBADF                  [OK]
53 sched_yield = 0                       [OK]
54 select_null = 0                       [OK]
55 select_stdout = 1                     [OK]
56 select_fault = -1 EFAULT              [OK]
57 stat_blah = -1 ENOENT                 [OK]
58 stat_fault = -1 EFAULT                [OK]
59 symlink_root = -1 EEXIST              [OK]
60 unlink_root = -1 EISDIR               [OK]
61 unlink_blah = -1 ENOENT               [OK]
62 wait_child = -1 ECHILD                [OK]
63 waitpid_min = -1 ESRCH                [OK]
64 waitpid_child = -1 ECHILD             [OK]
65 write_badf = -1 EBADF                 [OK]
66 write_zero = 0                        [OK]
Errors during this test: 0

Running test 'stdlib'
0 getenv_TERM = <linux>                  [OK]
1 getenv_blah = <(null)>                 [OK]
2 setcmp_blah_blah = 0                   [OK]
3 setcmp_blah_blah2 = -50                [OK]
4 setncmp_blah_blah = 0                  [OK]
5 setncmp_blah_blah4 = 0                 [OK]
6 setncmp_blah_blah5 = -53               [OK]
7 setncmp_blah_blah6 = -54               [OK]
8 strchr_foobar_o = <oobar>              [OK]
9 strchr_foobar_z = <(null)>             [OK]
10 strrchr_foobar_o = <obar>             [OK]
11 strrchr_foobar_z = <(null)>           [OK]
Errors during this test: 0

Total number of errors: 0
---->8----


--

Willy Tarreau (17):
  tools/nolibc: make argc 32-bit in riscv startup code
  tools/nolibc: fix build warning in sys_mmap() when my_syscall6 is not
    defined
  tools/nolibc: make sys_mmap() automatically use the right __NR_mmap
    definition
  selftests/nolibc: add basic infrastructure to ease creation of nolibc
    tests
  selftests/nolibc: support a test definition format
  selftests/nolibc: implement a few tests for various syscalls
  selftests/nolibc: add a few tests for some stdlib functions
  selftests/nolibc: exit with poweroff on success when getpid() == 1
  selftests/nolibc: on x86, support exiting with isa-debug-exit
  selftests/nolibc: recreate and populate /dev and /proc if missing
  selftests/nolibc: condition some tests on /proc existence
  selftests/nolibc: support glibc as well
  selftests/nolibc: add a "kernel" target to build the kernel with the
    initramfs
  selftests/nolibc: add a "defconfig" target
  selftests/nolibc: add a "run" target to start the kernel in QEMU
  selftests/nolibc: "sysroot" target installs a local copy of the
    sysroot
  selftests/nolibc: add a "help" target

 MAINTAINERS                                  |   1 +
 tools/include/nolibc/arch-riscv.h            |   2 +-
 tools/include/nolibc/sys.h                   |   4 +-
 tools/testing/selftests/nolibc/Makefile      | 135 ++++
 tools/testing/selftests/nolibc/nolibc-test.c | 757 +++++++++++++++++++
 5 files changed, 896 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/nolibc/Makefile
 create mode 100644 tools/testing/selftests/nolibc/nolibc-test.c

-- 
2.17.5

[PATCH 01/17] tools/nolibc: make argc 32-bit in riscv startup code

[Willy Tarreau]

The "ld a0, 0(sp)" instruction doesn't build on RISCV32 because that
would load a 64-bit value into a 32-bit register. But argc 32-bit,
not 64, so we ought to use "lw" here. Tested on both RISCV32 and
RISCV64.

Cc: Pranith Kumar <bobby.prani@gmail.com>
Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/include/nolibc/arch-riscv.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/include/nolibc/arch-riscv.h b/tools/include/nolibc/arch-riscv.h
index 95e2b7924925..ba04771cb3a3 100644
--- a/tools/include/nolibc/arch-riscv.h
+++ b/tools/include/nolibc/arch-riscv.h
@@ -190,7 +190,7 @@ __asm__ (".section .text\n"
     ".option norelax\n"
     "lla   gp, __global_pointer$\n"
     ".option pop\n"
-    "ld    a0, 0(sp)\n"          // argc (a0) was in the stack
+    "lw    a0, 0(sp)\n"          // argc (a0) was in the stack
     "add   a1, sp, "SZREG"\n"    // argv (a1) = sp
     "slli  a2, a0, "PTRLOG"\n"   // envp (a2) = SZREG*argc ...
     "add   a2, a2, "SZREG"\n"    //             + SZREG (skip null)
-- 
2.17.5

[PATCH 02/17] tools/nolibc: fix build warning in sys_mmap() when my_syscall6 is not defined

[Willy Tarreau]

We return -ENOSYS when there's no syscall6() operation, but we must cast
it to void* to avoid a warning.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/include/nolibc/sys.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/include/nolibc/sys.h b/tools/include/nolibc/sys.h
index 08491070387b..b8c96878c9ce 100644
--- a/tools/include/nolibc/sys.h
+++ b/tools/include/nolibc/sys.h
@@ -692,7 +692,7 @@ void *sys_mmap(void *addr, size_t length, int prot, int flags, int fd,
 {
 #ifndef my_syscall6
 	/* Function not implemented. */
-	return -ENOSYS;
+	return (void *)-ENOSYS;
 #else
 
 	int n;
-- 
2.17.5

[PATCH 03/17] tools/nolibc: make sys_mmap() automatically use the right __NR_mmap definition

[Willy Tarreau]

__NR_mmap2 was used for i386 but it's also needed for other archs such
as RISCV32 or ARM. Let's decide to use it based on the __NR_mmap2
definition as it's not defined on other archs.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/include/nolibc/sys.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/include/nolibc/sys.h b/tools/include/nolibc/sys.h
index b8c96878c9ce..ce3ee03aa679 100644
--- a/tools/include/nolibc/sys.h
+++ b/tools/include/nolibc/sys.h
@@ -697,7 +697,7 @@ void *sys_mmap(void *addr, size_t length, int prot, int flags, int fd,
 
 	int n;
 
-#if defined(__i386__)
+#if defined(__NR_mmap2)
 	n = __NR_mmap2;
 	offset >>= 12;
 #else
-- 
2.17.5

[PATCH 04/17] selftests/nolibc: add basic infrastructure to ease creation of nolibc tests

[Willy Tarreau]

This creates a "nolibc" selftest that intends to test various parts of
the nolibc component, both in terms of build and execution for a given
architecture.

The aim is for it to be as simple to run as a kernel build, by just
passing the compiler (for the build) and the ARCH (for kernel and
execution).

It brings a basic squeleton made of a single C file that will ease testing
and error reporting. The code will be arranged so that it remains easy to
add basic tests for syscalls or library calls that may rely on a condition
to be executed, and whose result is compared to a value or to an error
with a specific errno value.

Tests will just use a relative line number in switch/case statements as
an index, saving the user from having to maintain arrays and complicated
functions which can often just be one-liners.

MAINTAINERS was updated.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 MAINTAINERS                                  |   1 +
 tools/testing/selftests/nolibc/Makefile      |  43 ++
 tools/testing/selftests/nolibc/nolibc-test.c | 395 +++++++++++++++++++
 3 files changed, 439 insertions(+)
 create mode 100644 tools/testing/selftests/nolibc/Makefile
 create mode 100644 tools/testing/selftests/nolibc/nolibc-test.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 4e38d7533cbe..6ab10b235ed0 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -14114,6 +14114,7 @@ M:	Willy Tarreau <w@1wt.eu>
 S:	Maintained
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/wtarreau/nolibc.git
 F:	tools/include/nolibc/
+F:	tools/testing/selftests/nolibc/
 
 NSDEPS
 M:	Matthias Maennich <maennich@google.com>
diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
new file mode 100644
index 000000000000..fd0a67082334
--- /dev/null
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: GPL-2.0
+# Makefile for nolibc tests
+include ../../../scripts/Makefile.include
+
+# we're in ".../tools/testing/selftests/nolibc"
+ifeq ($(srctree),)
+srctree := $(patsubst %/tools/testing/selftests/,%,$(dir $(CURDIR)))
+endif
+
+ifeq ($(ARCH),)
+include $(srctree)/scripts/subarch.include
+ARCH = $(SUBARCH)
+endif
+
+# OUTPUT is only set when run from the main makefile, otherwise
+# it defaults to this nolibc directory.
+OUTPUT ?= $(CURDIR)/
+
+ifeq ($(V),1)
+Q=
+else
+Q=@
+endif
+
+CFLAGS  ?= -Os -fno-ident -fno-asynchronous-unwind-tables
+LDFLAGS := -s
+
+all: nolibc-test
+
+nolibc-test: nolibc-test.c
+	$(QUIET_CC)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ \
+	  -nostdlib -static -include ../../../include/nolibc/nolibc.h $^ -lgcc
+
+initramfs: nolibc-test
+	$(QUIET_MKDIR)mkdir -p initramfs
+	$(call QUIET_INSTALL, initramfs/init)
+	$(Q)cp nolibc-test initramfs/init
+
+clean:
+	$(call QUIET_CLEAN, nolibc-test)
+	$(Q)rm -f nolibc-test
+	$(call QUIET_CLEAN, initramfs)
+	$(Q)rm -rf initramfs
diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
new file mode 100644
index 000000000000..6c050d4381fe
--- /dev/null
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -0,0 +1,395 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* platform-specific include files coming from the compiler */
+#include <limits.h>
+
+/* libc-specific include files
+ * The program may be built in 2 ways:
+ *   $(CC) -nostdlib -include /path/to/nolibc.h => NOLIBC already defined
+ *   $(CC) -nostdlib -I/path/to/nolibc/sysroot
+ */
+#ifndef NOLIBC
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#endif
+
+/* will be used by nolibc by getenv() */
+char **environ;
+
+#define CASE_ERR(err) \
+	case err: return #err
+
+/* returns the error name (e.g. "ENOENT") for common errors, "SUCCESS" for 0,
+ * or the decimal value for less common ones.
+ */
+const char *errorname(int err)
+{
+	switch (err) {
+	case 0: return "SUCCESS";
+	CASE_ERR(EPERM);
+	CASE_ERR(ENOENT);
+	CASE_ERR(ESRCH);
+	CASE_ERR(EINTR);
+	CASE_ERR(EIO);
+	CASE_ERR(ENXIO);
+	CASE_ERR(E2BIG);
+	CASE_ERR(ENOEXEC);
+	CASE_ERR(EBADF);
+	CASE_ERR(ECHILD);
+	CASE_ERR(EAGAIN);
+	CASE_ERR(ENOMEM);
+	CASE_ERR(EACCES);
+	CASE_ERR(EFAULT);
+	CASE_ERR(ENOTBLK);
+	CASE_ERR(EBUSY);
+	CASE_ERR(EEXIST);
+	CASE_ERR(EXDEV);
+	CASE_ERR(ENODEV);
+	CASE_ERR(ENOTDIR);
+	CASE_ERR(EISDIR);
+	CASE_ERR(EINVAL);
+	CASE_ERR(ENFILE);
+	CASE_ERR(EMFILE);
+	CASE_ERR(ENOTTY);
+	CASE_ERR(ETXTBSY);
+	CASE_ERR(EFBIG);
+	CASE_ERR(ENOSPC);
+	CASE_ERR(ESPIPE);
+	CASE_ERR(EROFS);
+	CASE_ERR(EMLINK);
+	CASE_ERR(EPIPE);
+	CASE_ERR(EDOM);
+	CASE_ERR(ERANGE);
+	CASE_ERR(ENOSYS);
+	default:
+		return itoa(err);
+	}
+}
+
+static int pad_spc(int llen, int cnt, const char *fmt, ...)
+{
+	va_list args;
+	int len;
+	int ret;
+
+	for (len = 0; len < cnt - llen; len++)
+		putchar(' ');
+
+	va_start(args, fmt);
+	ret = vfprintf(stdout, fmt, args);
+	va_end(args);
+	return ret < 0 ? ret : ret + len;
+}
+
+/* The tests below are intended to be used by the macroes, which evaluate
+ * expression <expr>, print the status to stdout, and update the "ret"
+ * variable to count failures. The functions themselves return the number
+ * of failures, thus either 0 or 1.
+ */
+
+#define EXPECT_ZR(cond, expr)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_zr(expr, llen); } while (0)
+
+static int expect_zr(int expr, int llen)
+{
+	int ret = !(expr == 0);
+
+	llen += printf(" = %d ", expr);
+	pad_spc(llen, 40, ret ? "[FAIL]\n" : " [OK]\n");
+	return ret;
+}
+
+
+#define EXPECT_NZ(cond, expr, val)			\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_nz(expr, llen; } while (0)
+
+static int expect_nz(int expr, int llen)
+{
+	int ret = !(expr != 0);
+
+	llen += printf(" = %d ", expr);
+	pad_spc(llen, 40, ret ? "[FAIL]\n" : " [OK]\n");
+	return ret;
+}
+
+
+#define EXPECT_EQ(cond, expr, val)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_eq(expr, llen, val); } while (0)
+
+static int expect_eq(int expr, int llen, int val)
+{
+	int ret = !(expr == val);
+
+	llen += printf(" = %d ", expr);
+	pad_spc(llen, 40, ret ? "[FAIL]\n" : " [OK]\n");
+	return ret;
+}
+
+
+#define EXPECT_NE(cond, expr, val)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_ne(expr, llen, val); } while (0)
+
+static int expect_ne(int expr, int llen, int val)
+{
+	int ret = !(expr != val);
+
+	llen += printf(" = %d ", expr);
+	pad_spc(llen, 40, ret ? "[FAIL]\n" : " [OK]\n");
+	return ret;
+}
+
+
+#define EXPECT_GE(cond, expr, val)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_ge(expr, llen, val); } while (0)
+
+static int expect_ge(int expr, int llen, int val)
+{
+	int ret = !(expr >= val);
+
+	llen += printf(" = %d ", expr);
+	pad_spc(llen, 40, ret ? "[FAIL]\n" : " [OK]\n");
+	return ret;
+}
+
+
+#define EXPECT_GT(cond, expr, val)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_gt(expr, llen, val); } while (0)
+
+static int expect_gt(int expr, int llen, int val)
+{
+	int ret = !(expr > val);
+
+	llen += printf(" = %d ", expr);
+	pad_spc(llen, 40, ret ? "[FAIL]\n" : " [OK]\n");
+	return ret;
+}
+
+
+#define EXPECT_LE(cond, expr, val)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_le(expr, llen, val); } while (0)
+
+static int expect_le(int expr, int llen, int val)
+{
+	int ret = !(expr <= val);
+
+	llen += printf(" = %d ", expr);
+	pad_spc(llen, 40, ret ? "[FAIL]\n" : " [OK]\n");
+	return ret;
+}
+
+
+#define EXPECT_LT(cond, expr, val)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_lt(expr, llen, val); } while (0)
+
+static int expect_lt(int expr, int llen, int val)
+{
+	int ret = !(expr < val);
+
+	llen += printf(" = %d ", expr);
+	pad_spc(llen, 40, ret ? "[FAIL]\n" : " [OK]\n");
+	return ret;
+}
+
+
+#define EXPECT_SYSZR(cond, expr)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_syszr(expr, llen); } while (0)
+
+static int expect_syszr(int expr, int llen)
+{
+	int ret = 0;
+
+	if (expr) {
+		ret = 1;
+		llen += printf(" = %d %s ", expr, errorname(errno));
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += printf(" = %d ", expr);
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_SYSEQ(cond, expr, val)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_syseq(expr, llen, val); } while (0)
+
+static int expect_syseq(int expr, int llen, int val)
+{
+	int ret = 0;
+
+	if (expr != val) {
+		ret = 1;
+		llen += printf(" = %d %s ", expr, errorname(errno));
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += printf(" = %d ", expr);
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_SYSNE(cond, expr, val)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_sysne(expr, llen, val); } while (0)
+
+static int expect_sysne(int expr, int llen, int val)
+{
+	int ret = 0;
+
+	if (expr == val) {
+		ret = 1;
+		llen += printf(" = %d %s ", expr, errorname(errno));
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += printf(" = %d ", expr);
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_SYSER(cond, expr, expret, experr)			\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_syserr(expr, expret, experr, llen); } while (0)
+
+static int expect_syserr(int expr, int expret, int experr, int llen)
+{
+	int ret = 0;
+	int _errno = errno;
+
+	llen += printf(" = %d %s ", expr, errorname(_errno));
+	if (expr != expret || _errno != experr) {
+		ret = 1;
+		llen += printf(" != (%d %s) ", expret, errorname(experr));
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_PTRZR(cond, expr)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_ptrzr(expr, llen); } while (0)
+
+static int expect_ptrzr(const void *expr, int llen)
+{
+	int ret = 0;
+
+	llen += printf(" = <%p> ", expr);
+	if (expr) {
+		ret = 1;
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_PTRNZ(cond, expr)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_ptrnz(expr, llen); } while (0)
+
+static int expect_ptrnz(const void *expr, int llen)
+{
+	int ret = 0;
+
+	llen += printf(" = <%p> ", expr);
+	if (!expr) {
+		ret = 1;
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_STRZR(cond, expr)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_strzr(expr, llen); } while (0)
+
+static int expect_strzr(const char *expr, int llen)
+{
+	int ret = 0;
+
+	llen += printf(" = <%s> ", expr);
+	if (expr) {
+		ret = 1;
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_STRNZ(cond, expr)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_strnz(expr, llen); } while (0)
+
+static int expect_strnz(const char *expr, int llen)
+{
+	int ret = 0;
+
+	llen += printf(" = <%s> ", expr);
+	if (!expr) {
+		ret = 1;
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_STREQ(cond, expr, cmp)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_streq(expr, llen, cmp); } while (0)
+
+static int expect_streq(const char *expr, int llen, const char *cmp)
+{
+	int ret = 0;
+
+	llen += printf(" = <%s> ", expr);
+	if (strcmp(expr, cmp) != 0) {
+		ret = 1;
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+
+#define EXPECT_STRNE(cond, expr, cmp)				\
+	do { if (!cond) pad_spc(llen, 40, "[SKIPPED]\n"); else ret += expect_strne(expr, llen, cmp); } while (0)
+
+static int expect_strne(const char *expr, int llen, const char *cmp)
+{
+	int ret = 0;
+
+	llen += printf(" = <%s> ", expr);
+	if (strcmp(expr, cmp) == 0) {
+		ret = 1;
+		llen += pad_spc(llen, 40, "[FAIL]\n");
+	} else {
+		llen += pad_spc(llen, 40, " [OK]\n");
+	}
+	return ret;
+}
+
+/* declare tests based on line numbers. There must be exactly one test per line. */
+#define CASE_TEST(name) \
+	case __LINE__: llen += printf("%d %s", test, #name);
+
+
+int main(int argc, char **argv, char **envp)
+{
+	int min = 0;
+	int max = __INT_MAX__;
+	int ret = 0;
+
+	environ = envp;
+
+	printf("Total number of errors: %d\n", ret);
+	printf("Exiting with status %d\n", !!ret);
+	return !!ret;
+}
-- 
2.17.5

[PATCH 05/17] selftests/nolibc: support a test definition format

[Willy Tarreau]

It now becomes possible to pass a string either in argv[1] or in the
NOLIBC_TEST environment variable (the former having precedence), to
specify which tests to run. The format is:

   testname[:range]*[,testname...]

Where a range is either a single value or the min and max numbers of the
test IDs in a sequence, delimited by a dash. Multiple ranges are possible.
This should provide enough flexibility to focus on certain failing parts
just by playing with the boot command line in a boot loader or in qemu
depending on what is accessible.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 91 ++++++++++++++++++++
 1 file changed, 91 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 6c050d4381fe..49177ea9943c 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -17,6 +17,12 @@
 /* will be used by nolibc by getenv() */
 char **environ;
 
+/* definition of a series of tests */
+struct test {
+	const char *name;              // test name
+	int (*func)(int min, int max); // handler
+};
+
 #define CASE_ERR(err) \
 	case err: return #err
 
@@ -376,19 +382,104 @@ static int expect_strne(const char *expr, int llen, const char *cmp)
 	return ret;
 }
 
+
 /* declare tests based on line numbers. There must be exactly one test per line. */
 #define CASE_TEST(name) \
 	case __LINE__: llen += printf("%d %s", test, #name);
 
 
+/* This is the definition of known test names, with their functions */
+static struct test test_names[] = {
+	/* add new tests here */
+	{ 0 }
+};
+
 int main(int argc, char **argv, char **envp)
 {
 	int min = 0;
 	int max = __INT_MAX__;
 	int ret = 0;
+	int err;
+	int idx;
+	char *test;
 
 	environ = envp;
 
+	/* the definition of a series of tests comes from either argv[1] or the
+	 * "NOLIBC_TEST" environment variable. It's made of a comma-delimited
+	 * series of test names and optional ranges:
+	 *    syscall:5-15[:.*],stdlib:8-10
+	 */
+	test = argv[1];
+	if (!test)
+		test = getenv("NOLIBC_TEST");
+
+	if (test) {
+		char *comma, *colon, *dash, *value;
+
+		do {
+			comma = strchr(test, ',');
+			if (comma)
+				*(comma++) = '\0';
+
+			colon = strchr(test, ':');
+			if (colon)
+				*(colon++) = '\0';
+
+			for (idx = 0; test_names[idx].name; idx++) {
+				if (strcmp(test, test_names[idx].name) == 0)
+					break;
+			}
+
+			if (test_names[idx].name) {
+				/* The test was named, it will be called at least
+				 * once. We may have an optional range at <colon>
+				 * here, which defaults to the full range.
+				 */
+				do {
+					min = 0; max = __INT_MAX__;
+					value = colon;
+					if (value && *value) {
+						colon = strchr(value, ':');
+						if (colon)
+							*(colon++) = '\0';
+
+						dash = strchr(value, '-');
+						if (dash)
+							*(dash++) = '\0';
+
+						/* support :val: :min-max: :min-: :-max: */
+						if (*value)
+							min = atoi(value);
+						if (!dash)
+							max = min;
+						else if (*dash)
+							max = atoi(dash);
+
+						value = colon;
+					}
+
+					/* now's time to call the test */
+					printf("Running test '%s'\n", test_names[idx].name);
+					err = test_names[idx].func(min, max);
+					ret += err;
+					printf("Errors during this test: %d\n\n", err);
+				} while (colon && *colon);
+			} else
+				printf("Ignoring unknown test name '%s'\n", test);
+
+			test = comma;
+		} while (test && *test);
+	} else {
+		/* no test mentioned, run everything */
+		for (idx = 0; test_names[idx].name; idx++) {
+			printf("Running test '%s'\n", test_names[idx].name);
+			err = test_names[idx].func(min, max);
+			ret += err;
+			printf("Errors during this test: %d\n\n", err);
+		}
+	}
+
 	printf("Total number of errors: %d\n", ret);
 	printf("Exiting with status %d\n", !!ret);
 	return !!ret;
-- 
2.17.5

[PATCH 06/17] selftests/nolibc: implement a few tests for various syscalls

[Willy Tarreau]

This adds 63 tests covering about 34 syscalls. Both successes and
failures are tested. Two tests fail when run as unprivileged user
(link_dir which returns EACCESS instead of EPERM, and chroot which
returns EPERM). One test (execve("/")) expects to fail on EACCESS,
but needs to have valid arguments otherwise the kernel will log a
message. And a few tests require /proc to be mounted.

The code is not pretty since all tests are one-liners, sometimes
resulting in long lines, especially when using compount statements to
preset a line, but it's convenient and doesn't obfuscate the code,
which is important to understand what failed.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 110 +++++++++++++++++++
 1 file changed, 110 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 49177ea9943c..dc87832912ce 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -388,9 +388,119 @@ static int expect_strne(const char *expr, int llen, const char *cmp)
 	case __LINE__: llen += printf("%d %s", test, #name);
 
 
+/* used by some syscall tests below */
+int test_getdents64(const char *dir)
+{
+	char buffer[4096];
+	int fd, ret;
+	int err;
+
+	ret = fd = open(dir, O_RDONLY | O_DIRECTORY, 0);
+	if (ret < 0)
+		return ret;
+
+	ret = getdents64(fd, (void *)buffer, sizeof(buffer));
+	err = errno;
+	close(fd);
+
+	errno = err;
+	return ret;
+}
+
+/* Run syscall tests between IDs <min> and <max>.
+ * Return 0 on success, non-zero on failure.
+ */
+int run_syscall(int min, int max)
+{
+	struct stat stat_buf;
+	int test;
+	int tmp;
+	int ret = 0;
+	void *p1, *p2;
+
+	for (test = min; test >= 0 && test <= max; test++) {
+		int llen = 0; // line length
+
+		/* avoid leaving empty lines below, this will insert holes into
+		 * test numbers.
+		 */
+		switch (test + __LINE__ + 1) {
+		CASE_TEST(getpid);            EXPECT_SYSNE(1, getpid(), -1); break;
+		CASE_TEST(getppid);           EXPECT_SYSNE(1, getppid(), -1); break;
+		CASE_TEST(gettid);            EXPECT_SYSNE(1, gettid(), -1); break;
+		CASE_TEST(getpgid_self);      EXPECT_SYSNE(1, getpgid(0), -1); break;
+		CASE_TEST(getpgid_bad);       EXPECT_SYSER(1, getpgid(-1), -1, ESRCH); break;
+		CASE_TEST(kill_0);            EXPECT_SYSZR(1, kill(getpid(), 0)); break;
+		CASE_TEST(kill_CONT);         EXPECT_SYSZR(1, kill(getpid(), 0)); break;
+		CASE_TEST(kill_BADPID);       EXPECT_SYSER(1, kill(INT_MAX, 0), -1, ESRCH); break;
+		CASE_TEST(sbrk);              if ((p1 = p2 = sbrk(4096)) != (void *)-1) p2 = sbrk(-4096); EXPECT_SYSZR(1, (p2 == (void *)-1) || p2 == p1); break;
+		CASE_TEST(brk);               EXPECT_SYSZR(1, brk(sbrk(0))); break;
+		CASE_TEST(chdir_root);        EXPECT_SYSZR(1, chdir("/")); break;
+		CASE_TEST(chdir_dot);         EXPECT_SYSZR(1, chdir(".")); break;
+		CASE_TEST(chdir_blah);        EXPECT_SYSER(1, chdir("/blah"), -1, ENOENT); break;
+		CASE_TEST(chmod_net);         EXPECT_SYSZR(1, chmod("/proc/self/net", 0555)); break;
+		CASE_TEST(chmod_self);        EXPECT_SYSER(1, chmod("/proc/self", 0555), -1, EPERM); break;
+		CASE_TEST(chown_self);        EXPECT_SYSER(1, chown("/proc/self", 0, 0), -1, EPERM); break;
+		CASE_TEST(chroot_root);       EXPECT_SYSZR(1, chroot("/")); break;
+		CASE_TEST(chroot_blah);       EXPECT_SYSER(1, chroot("/proc/self/blah"), -1, ENOENT); break;
+		CASE_TEST(chroot_exe);        EXPECT_SYSER(1, chroot("/proc/self/exe"), -1, ENOTDIR); break;
+		CASE_TEST(close_m1);          EXPECT_SYSER(1, close(-1), -1, EBADF); break;
+		CASE_TEST(close_dup);         EXPECT_SYSZR(1, close(dup(0))); break;
+		CASE_TEST(dup_0);             tmp = dup(0);  EXPECT_SYSNE(1, tmp, -1); close(tmp); break;
+		CASE_TEST(dup_m1);            tmp = dup(-1); EXPECT_SYSER(1, tmp, -1, EBADF); if (tmp != -1) close(tmp); break;
+		CASE_TEST(dup2_0);            tmp = dup2(0, 100);  EXPECT_SYSNE(1, tmp, -1); close(tmp); break;
+		CASE_TEST(dup2_m1);           tmp = dup2(-1, 100); EXPECT_SYSER(1, tmp, -1, EBADF); if (tmp != -1) close(tmp); break;
+		CASE_TEST(dup3_0);            tmp = dup3(0, 100, 0);  EXPECT_SYSNE(1, tmp, -1); close(tmp); break;
+		CASE_TEST(dup3_m1);           tmp = dup3(-1, 100, 0); EXPECT_SYSER(1, tmp, -1, EBADF); if (tmp != -1) close(tmp); break;
+		CASE_TEST(execve_root);       EXPECT_SYSER(1, execve("/", (char*[]){ [0] = "/", [1] = NULL }, NULL), -1, EACCES); break;
+		CASE_TEST(getdents64_root);   EXPECT_SYSNE(1, test_getdents64("/"), -1); break;
+		CASE_TEST(getdents64_null);   EXPECT_SYSER(1, test_getdents64("/dev/null"), -1, ENOTDIR); break;
+		CASE_TEST(gettimeofday_null); EXPECT_SYSZR(1, gettimeofday(NULL, NULL)); break;
+		CASE_TEST(gettimeofday_bad1); EXPECT_SYSER(1, gettimeofday((void *)1, NULL), -1, EFAULT); break;
+		CASE_TEST(gettimeofday_bad2); EXPECT_SYSER(1, gettimeofday(NULL, (void *)1), -1, EFAULT); break;
+		CASE_TEST(gettimeofday_bad2); EXPECT_SYSER(1, gettimeofday(NULL, (void *)1), -1, EFAULT); break;
+		CASE_TEST(ioctl_tiocinq);     EXPECT_SYSZR(1, ioctl(0, TIOCINQ, &tmp)); break;
+		CASE_TEST(ioctl_tiocinq);     EXPECT_SYSZR(1, ioctl(0, TIOCINQ, &tmp)); break;
+		CASE_TEST(link_root1);        EXPECT_SYSER(1, link("/", "/"), -1, EEXIST); break;
+		CASE_TEST(link_blah);         EXPECT_SYSER(1, link("/proc/self/blah", "/blah"), -1, ENOENT); break;
+		CASE_TEST(link_dir);          EXPECT_SYSER(1, link("/", "/blah"), -1, EPERM); break;
+		CASE_TEST(link_cross);        EXPECT_SYSER(1, link("/proc/self/net", "/blah"), -1, EXDEV); break;
+		CASE_TEST(lseek_m1);          EXPECT_SYSER(1, lseek(-1, 0, SEEK_SET), -1, EBADF); break;
+		CASE_TEST(lseek_0);           EXPECT_SYSER(1, lseek(0, 0, SEEK_SET), -1, ESPIPE); break;
+		CASE_TEST(mkdir_root);        EXPECT_SYSER(1, mkdir("/", 0755), -1, EEXIST); break;
+		CASE_TEST(open_tty);          EXPECT_SYSNE(1, tmp = open("/dev/null", 0), -1); if (tmp != -1) close(tmp); break;
+		CASE_TEST(open_blah);         EXPECT_SYSER(1, tmp = open("/proc/self/blah", 0), -1, ENOENT); if (tmp != -1) close(tmp); break;
+		CASE_TEST(poll_null);         EXPECT_SYSZR(1, poll(NULL, 0, 0)); break;
+		CASE_TEST(poll_stdout);       EXPECT_SYSNE(1, ({ struct pollfd fds = { 1, POLLOUT, 0}; poll(&fds, 1, 0); }), -1); break;
+		CASE_TEST(poll_fault);        EXPECT_SYSER(1, poll((void *)1, 1, 0), -1, EFAULT); break;
+		CASE_TEST(read_badf);         EXPECT_SYSER(1, read(-1, &tmp, 1), -1, EBADF); break;
+		CASE_TEST(sched_yield);       EXPECT_SYSZR(1, sched_yield()); break;
+		CASE_TEST(select_null);       EXPECT_SYSZR(1, ({ struct timeval tv = { 0 }; select(0, NULL, NULL, NULL, &tv); })); break;
+		CASE_TEST(select_stdout);     EXPECT_SYSNE(1, ({ fd_set fds; FD_ZERO(&fds); FD_SET(1, &fds); select(2, NULL, &fds, NULL, NULL); }), -1); break;
+		CASE_TEST(select_fault);      EXPECT_SYSER(1, select(1, (void *)1, NULL, NULL, 0), -1, EFAULT); break;
+		CASE_TEST(stat_blah);         EXPECT_SYSER(1, stat("/proc/self/blah", &stat_buf), -1, ENOENT); break;
+		CASE_TEST(stat_fault);        EXPECT_SYSER(1, stat(NULL, &stat_buf), -1, EFAULT); break;
+		CASE_TEST(symlink_root);      EXPECT_SYSER(1, symlink("/", "/"), -1, EEXIST); break;
+		CASE_TEST(unlink_root);       EXPECT_SYSER(1, unlink("/"), -1, EISDIR); break;
+		CASE_TEST(unlink_blah);       EXPECT_SYSER(1, unlink("/proc/self/blah"), -1, ENOENT); break;
+		CASE_TEST(wait_child);        EXPECT_SYSER(1, wait(&tmp), -1, ECHILD); break;
+		CASE_TEST(waitpid_min);       EXPECT_SYSER(1, waitpid(INT_MIN, &tmp, WNOHANG), -1, ESRCH); break;
+		CASE_TEST(waitpid_child);     EXPECT_SYSER(1, waitpid(getpid(), &tmp, WNOHANG), -1, ECHILD); break;
+		CASE_TEST(write_badf);        EXPECT_SYSER(1, write(-1, &tmp, 1), -1, EBADF); break;
+		CASE_TEST(write_zero);        EXPECT_SYSZR(1, write(1, &tmp, 0)); break;
+		case __LINE__:
+			return ret; /* must be last */
+		/* note: do not set any defaults so as to permit holes above */
+		}
+	}
+	return ret;
+}
+
+
 /* This is the definition of known test names, with their functions */
 static struct test test_names[] = {
 	/* add new tests here */
+	{ .name = "syscall",   .func = run_syscall  },
 	{ 0 }
 };
 
-- 
2.17.5

[PATCH 07/17] selftests/nolibc: add a few tests for some libc functions

[Willy Tarreau]

The test series called "stdlib" covers some libc functions (string,
stdlib etc). By default they are automatically run after "syscall"
but may be requested in argument or in variable NOLIBC_TEST.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 35 ++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index dc87832912ce..b928f099431f 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -496,11 +496,46 @@ int run_syscall(int min, int max)
 	return ret;
 }
 
+int run_stdlib(int min, int max)
+{
+	int test;
+	int tmp;
+	int ret = 0;
+	void *p1, *p2;
+
+	for (test = min; test >= 0 && test <= max; test++) {
+		int llen = 0; // line length
+
+		/* avoid leaving empty lines below, this will insert holes into
+		 * test numbers.
+		 */
+		switch (test + __LINE__ + 1) {
+		CASE_TEST(getenv_TERM);        EXPECT_STRNZ(1, getenv("TERM")); break;
+		CASE_TEST(getenv_blah);        EXPECT_STRZR(1, getenv("blah")); break;
+		CASE_TEST(setcmp_blah_blah);   EXPECT_EQ(1, strcmp("blah", "blah"), 0); break;
+		CASE_TEST(setcmp_blah_blah2);  EXPECT_NE(1, strcmp("blah", "blah2"), 0); break;
+		CASE_TEST(setncmp_blah_blah);  EXPECT_EQ(1, strncmp("blah", "blah", 10), 0); break;
+		CASE_TEST(setncmp_blah_blah4); EXPECT_EQ(1, strncmp("blah", "blah4", 4), 0); break;
+		CASE_TEST(setncmp_blah_blah5); EXPECT_NE(1, strncmp("blah", "blah5", 5), 0); break;
+		CASE_TEST(setncmp_blah_blah6); EXPECT_NE(1, strncmp("blah", "blah6", 6), 0); break;
+		CASE_TEST(strchr_foobar_o);    EXPECT_STREQ(1, strchr("foobar", 'o'), "oobar"); break;
+		CASE_TEST(strchr_foobar_z);    EXPECT_STRZR(1, strchr("foobar", 'z')); break;
+		CASE_TEST(strrchr_foobar_o);   EXPECT_STREQ(1, strrchr("foobar", 'o'), "obar"); break;
+		CASE_TEST(strrchr_foobar_z);   EXPECT_STRZR(1, strrchr("foobar", 'z')); break;
+		case __LINE__:
+			return ret; /* must be last */
+		/* note: do not set any defaults so as to permit holes above */
+		}
+	}
+	return ret;
+}
+
 
 /* This is the definition of known test names, with their functions */
 static struct test test_names[] = {
 	/* add new tests here */
 	{ .name = "syscall",   .func = run_syscall  },
+	{ .name = "stdlib",    .func = run_stdlib   },
 	{ 0 }
 };
 
-- 
2.17.5

[PATCH 07/17] selftests/nolibc: add a few tests for some stdlib functions

[Willy Tarreau]

The test series called "stdlib" covers some libc functions (string,
stdlib etc). By default they are automatically run after "syscall"
but may be requested in argument or in variable NOLIBC_TEST.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 35 ++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index dc87832912ce..ab7c4cbdef9b 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -496,11 +496,46 @@ int run_syscall(int min, int max)
 	return ret;
 }
 
+int run_stdlib(int min, int max)
+{
+	int test;
+	int tmp;
+	int ret = 0;
+	void *p1, *p2;
+
+	for (test = min; test >= 0 && test <= max; test++) {
+		int llen = 0; // line length
+
+		/* avoid leaving empty lines below, this will insert holes into
+		 * test numbers.
+		 */
+		switch (test + __LINE__ + 1) {
+		CASE_TEST(getenv_TERM);        EXPECT_STRNZ(1, getenv("TERM")); break;
+		CASE_TEST(getenv_blah);        EXPECT_STRZR(1, getenv("blah")); break;
+		CASE_TEST(strcmp_blah_blah);   EXPECT_EQ(1, strcmp("blah", "blah"), 0); break;
+		CASE_TEST(strcmp_blah_blah2);  EXPECT_NE(1, strcmp("blah", "blah2"), 0); break;
+		CASE_TEST(strncmp_blah_blah);  EXPECT_EQ(1, strncmp("blah", "blah", 10), 0); break;
+		CASE_TEST(strncmp_blah_blah4); EXPECT_EQ(1, strncmp("blah", "blah4", 4), 0); break;
+		CASE_TEST(strncmp_blah_blah5); EXPECT_NE(1, strncmp("blah", "blah5", 5), 0); break;
+		CASE_TEST(strncmp_blah_blah6); EXPECT_NE(1, strncmp("blah", "blah6", 6), 0); break;
+		CASE_TEST(strchr_foobar_o);    EXPECT_STREQ(1, strchr("foobar", 'o'), "oobar"); break;
+		CASE_TEST(strchr_foobar_z);    EXPECT_STRZR(1, strchr("foobar", 'z')); break;
+		CASE_TEST(strrchr_foobar_o);   EXPECT_STREQ(1, strrchr("foobar", 'o'), "obar"); break;
+		CASE_TEST(strrchr_foobar_z);   EXPECT_STRZR(1, strrchr("foobar", 'z')); break;
+		case __LINE__:
+			return ret; /* must be last */
+		/* note: do not set any defaults so as to permit holes above */
+		}
+	}
+	return ret;
+}
+
 
 /* This is the definition of known test names, with their functions */
 static struct test test_names[] = {
 	/* add new tests here */
 	{ .name = "syscall",   .func = run_syscall  },
+	{ .name = "stdlib",    .func = run_stdlib   },
 	{ 0 }
 };
 
-- 
2.17.5

[PATCH 08/17] selftests/nolibc: exit with poweroff on success when getpid() == 1

[Willy Tarreau]

The idea is to ease automated testing under qemu. If the test succeeds
while running as PID 1, indicating the system was booted with init=/test,
let's just power off so that qemu can exit with a successful code. In
other situations it will exit and provoke a panic, which may be caught
for example with CONFIG_PVPANIC.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index ab7c4cbdef9b..5f2173610d03 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -626,6 +626,20 @@ int main(int argc, char **argv, char **envp)
 	}
 
 	printf("Total number of errors: %d\n", ret);
+
+	if (getpid() == 1) {
+		/* we're running as init, there's no other process on the
+		 * system, thus likely started from a VM for a quick check.
+		 * Exiting will provoke a kernel panic that may be reported
+		 * as an error by Qemu or the hypervisor, while stopping
+		 * cleanly will often be reported as a success. This allows
+		 * to use the output of this program for bisecting kernels.
+		 */
+		printf("Leaving init with final status: %d\n", !!ret);
+		if (ret == 0)
+			reboot(LINUX_REBOOT_CMD_POWER_OFF);
+	}
+
 	printf("Exiting with status %d\n", !!ret);
 	return !!ret;
 }
-- 
2.17.5

[PATCH 09/17] selftests/nolibc: on x86, support exiting with isa-debug-exit

[Willy Tarreau]

QEMU, when started with "-device isa-debug-exit -no-reboot" will exit
with status code 2N+1 when N is written to 0x501. This is particularly
convenient for automated tests but this is not portable. As such we
only enable this on x86_64 when pid==1. In addition, this requires an
ioperm() call but in order not to have to define arch-specific syscalls
we just perform the syscall by hand there.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 5f2173610d03..45793b6f2244 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -638,6 +638,15 @@ int main(int argc, char **argv, char **envp)
 		printf("Leaving init with final status: %d\n", !!ret);
 		if (ret == 0)
 			reboot(LINUX_REBOOT_CMD_POWER_OFF);
+#if defined(__x86_64__)
+		/* QEMU started with "-device isa-debug-exit -no-reboot" will
+		 * exit with status code 2N+1 when N is written to 0x501. We
+		 * hard-code the syscall here as it's arch-dependent.
+		 */
+		else if (my_syscall3(__NR_ioperm, 0x501, 1, 1) == 0)
+			asm volatile ("outb %%al, %%dx" :: "d"(0x501), "a"(0));
+		/* if it does nothing, fall back to the regular panic */
+#endif
 	}
 
 	printf("Exiting with status %d\n", !!ret);
-- 
2.17.5

[PATCH 10/17] selftests/nolibc: recreate and populate /dev and /proc if missing

[Willy Tarreau]

Most of the time the program will be run alone in an initramfs. There
is no value in requiring the user to populate /dev and /proc for such
tests, we can do it ourselves, and it participates to the tests at the
same time.

What's done here is that when called as init (getpid()==1) we check
if /dev exists or create it, if /dev/console and /dev/null exists,
otherwise we try to mount a devtmpfs there, and if it fails we fall
back to mknod. The console is reopened if stdout was closed. Finally
/proc is created and mounted if /proc/self cannot be found. This is
sufficient for most tests.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 56 ++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 45793b6f2244..0bbfe0b3b648 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -530,6 +530,54 @@ int run_stdlib(int min, int max)
 	return ret;
 }
 
+/* prepare what needs to be prepared for pid 1 (stdio, /dev, /proc, etc) */
+int prepare(void)
+{
+	struct stat stat_buf;
+
+	/* It's possible that /dev doesn't even exist or was not mounted, so
+	 * we'll try to create it, mount it, or create minimal entries into it.
+	 * We want at least /dev/null and /dev/console.
+	 */
+	if (stat("/dev/.", &stat_buf) == 0 || mkdir("/dev", 0755) == 0) {
+		if (stat("/dev/console", &stat_buf) != 0 ||
+		    stat("/dev/null", &stat_buf) != 0) {
+			/* try devtmpfs first, otherwise fall back to manual creation */
+			if (mount("/dev", "/dev", "devtmpfs", 0, 0) != 0) {
+				mknod("/dev/console", 0600 | S_IFCHR, makedev(5, 1));
+				mknod("/dev/null",    0666 | S_IFCHR, makedev(1, 3));
+			}
+		}
+	}
+
+	/* If no /dev/console was found before calling init, stdio is closed so
+	 * we need to reopen it from /dev/console. If it failed above, it will
+	 * still fail here and we cannot emit a message anyway.
+	 */
+	if (close(dup(1)) == -1) {
+		int fd = open("/dev/console", O_RDWR);
+
+		if (fd >= 0) {
+			if (fd != 0)
+				dup2(fd, 0);
+			if (fd != 1)
+				dup2(fd, 1);
+			if (fd != 2)
+				dup2(fd, 2);
+			if (fd > 2)
+				close(fd);
+			puts("\nSuccessfully reopened /dev/console.");
+		}
+	}
+
+	/* try to mount /proc if not mounted. Silently fail otherwise */
+	if (stat("/proc/.", &stat_buf) == 0 || mkdir("/proc", 0755) == 0) {
+		if (stat("/proc/self", &stat_buf) != 0)
+			mount("/proc", "/proc", "proc", 0, 0);
+	}
+
+	return 0;
+}
 
 /* This is the definition of known test names, with their functions */
 static struct test test_names[] = {
@@ -550,6 +598,14 @@ int main(int argc, char **argv, char **envp)
 
 	environ = envp;
 
+	/* when called as init, it's possible that no console was opened, for
+	 * example if no /dev file system was provided. We'll check that fd#1
+	 * was opened, and if not we'll attempt to create and open /dev/console
+	 * and /dev/null that we'll use for later tests.
+	 */
+	if (getpid() == 1)
+		prepare();
+
 	/* the definition of a series of tests comes from either argv[1] or the
 	 * "NOLIBC_TEST" environment variable. It's made of a comma-delimited
 	 * series of test names and optional ranges:
-- 
2.17.5

[PATCH 11/17] selftests/nolibc: condition some tests on /proc existence

[Willy Tarreau]

If /proc is not available (program run inside a chroot or without
sufficient permissions), it's better to disable the associated tests.
Some will be preserved like the ones which check for a failure to
create some entries there since they're still supposed to fail.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 0bbfe0b3b648..26d61ec910c1 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -413,11 +413,15 @@ int test_getdents64(const char *dir)
 int run_syscall(int min, int max)
 {
 	struct stat stat_buf;
+	int proc;
 	int test;
 	int tmp;
 	int ret = 0;
 	void *p1, *p2;
 
+	/* <proc> indicates whether or not /proc is mounted */
+	proc = stat("/proc", &stat_buf) == 0;
+
 	for (test = min; test >= 0 && test <= max; test++) {
 		int llen = 0; // line length
 
@@ -438,12 +442,12 @@ int run_syscall(int min, int max)
 		CASE_TEST(chdir_root);        EXPECT_SYSZR(1, chdir("/")); break;
 		CASE_TEST(chdir_dot);         EXPECT_SYSZR(1, chdir(".")); break;
 		CASE_TEST(chdir_blah);        EXPECT_SYSER(1, chdir("/blah"), -1, ENOENT); break;
-		CASE_TEST(chmod_net);         EXPECT_SYSZR(1, chmod("/proc/self/net", 0555)); break;
-		CASE_TEST(chmod_self);        EXPECT_SYSER(1, chmod("/proc/self", 0555), -1, EPERM); break;
-		CASE_TEST(chown_self);        EXPECT_SYSER(1, chown("/proc/self", 0, 0), -1, EPERM); break;
+		CASE_TEST(chmod_net);         EXPECT_SYSZR(proc, chmod("/proc/self/net", 0555)); break;
+		CASE_TEST(chmod_self);        EXPECT_SYSER(proc, chmod("/proc/self", 0555), -1, EPERM); break;
+		CASE_TEST(chown_self);        EXPECT_SYSER(proc, chown("/proc/self", 0, 0), -1, EPERM); break;
 		CASE_TEST(chroot_root);       EXPECT_SYSZR(1, chroot("/")); break;
 		CASE_TEST(chroot_blah);       EXPECT_SYSER(1, chroot("/proc/self/blah"), -1, ENOENT); break;
-		CASE_TEST(chroot_exe);        EXPECT_SYSER(1, chroot("/proc/self/exe"), -1, ENOTDIR); break;
+		CASE_TEST(chroot_exe);        EXPECT_SYSER(proc, chroot("/proc/self/exe"), -1, ENOTDIR); break;
 		CASE_TEST(close_m1);          EXPECT_SYSER(1, close(-1), -1, EBADF); break;
 		CASE_TEST(close_dup);         EXPECT_SYSZR(1, close(dup(0))); break;
 		CASE_TEST(dup_0);             tmp = dup(0);  EXPECT_SYSNE(1, tmp, -1); close(tmp); break;
@@ -464,7 +468,7 @@ int run_syscall(int min, int max)
 		CASE_TEST(link_root1);        EXPECT_SYSER(1, link("/", "/"), -1, EEXIST); break;
 		CASE_TEST(link_blah);         EXPECT_SYSER(1, link("/proc/self/blah", "/blah"), -1, ENOENT); break;
 		CASE_TEST(link_dir);          EXPECT_SYSER(1, link("/", "/blah"), -1, EPERM); break;
-		CASE_TEST(link_cross);        EXPECT_SYSER(1, link("/proc/self/net", "/blah"), -1, EXDEV); break;
+		CASE_TEST(link_cross);        EXPECT_SYSER(proc, link("/proc/self/net", "/blah"), -1, EXDEV); break;
 		CASE_TEST(lseek_m1);          EXPECT_SYSER(1, lseek(-1, 0, SEEK_SET), -1, EBADF); break;
 		CASE_TEST(lseek_0);           EXPECT_SYSER(1, lseek(0, 0, SEEK_SET), -1, ESPIPE); break;
 		CASE_TEST(mkdir_root);        EXPECT_SYSER(1, mkdir("/", 0755), -1, EEXIST); break;
-- 
2.17.5

[PATCH 12/17] selftests/nolibc: support glibc as well

[Willy Tarreau]

Adding support for glibc can be useful to distinguish between bugs in
nolibc and bugs in the kernel when a syscall reports an unusual value.

It's not that much work and should not affect the long term
maintainability of the tests. The necessary changes can essentially be
summed up like this:
  - set _GNU_SOURCE a the top to access some definitions
  - many includes added when we know we don't come from nolibc (missing
    the stdio include guard)
  - disable gettid() which is not exposed by glibc
  - disable gettimeofday's support of bad pointers since these  crash
    in glibc
  - add a simple itoa() for errorname(); strerror() is too verbose (no
    way to get short messages). strerrorname_np() was added in modern
    glibc (2.32) to do exactly this but that 's too recent to be usable
    as the default fallback.
  - use the standard ioperm() definition. May be we need to implement
    ioperm() in nolibc if that's useful.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 47 +++++++++++++++++++-
 1 file changed, 45 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 26d61ec910c1..88ee5d0828ec 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -1,17 +1,41 @@
 // SPDX-License-Identifier: GPL-2.0
 
+#define _GNU_SOURCE
+
 /* platform-specific include files coming from the compiler */
 #include <limits.h>
 
 /* libc-specific include files
- * The program may be built in 2 ways:
+ * The program may be built in 3 ways:
  *   $(CC) -nostdlib -include /path/to/nolibc.h => NOLIBC already defined
- *   $(CC) -nostdlib -I/path/to/nolibc/sysroot
+ *   $(CC) -nostdlib -I/path/to/nolibc/sysroot  => _NOLIBC_* guards are present
+ *   $(CC) with default libc                    => NOLIBC* never defined
  */
 #ifndef NOLIBC
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#ifndef _NOLIBC_STDIO_H
+/* standard libcs need more includes */
+#include <linux/reboot.h>
+#include <sys/io.h>
+#include <sys/ioctl.h>
+#include <sys/mount.h>
+#include <sys/reboot.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/sysmacros.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <dirent.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <poll.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdarg.h>
+#include <unistd.h>
+#endif
 #endif
 
 /* will be used by nolibc by getenv() */
@@ -23,6 +47,17 @@ struct test {
 	int (*func)(int min, int max); // handler
 };
 
+#ifndef _NOLIBC_STDLIB_H
+char *itoa(int i)
+{
+	static char buf[12];
+	int ret;
+
+	ret = snprintf(buf, sizeof(buf), "%d", i);
+	return (ret >= 0 && ret < sizeof(buf)) ? buf : "#err";
+}
+#endif
+
 #define CASE_ERR(err) \
 	case err: return #err
 
@@ -431,7 +466,9 @@ int run_syscall(int min, int max)
 		switch (test + __LINE__ + 1) {
 		CASE_TEST(getpid);            EXPECT_SYSNE(1, getpid(), -1); break;
 		CASE_TEST(getppid);           EXPECT_SYSNE(1, getppid(), -1); break;
+#ifdef NOLIBC
 		CASE_TEST(gettid);            EXPECT_SYSNE(1, gettid(), -1); break;
+#endif
 		CASE_TEST(getpgid_self);      EXPECT_SYSNE(1, getpgid(0), -1); break;
 		CASE_TEST(getpgid_bad);       EXPECT_SYSER(1, getpgid(-1), -1, ESRCH); break;
 		CASE_TEST(kill_0);            EXPECT_SYSZR(1, kill(getpid(), 0)); break;
@@ -460,9 +497,11 @@ int run_syscall(int min, int max)
 		CASE_TEST(getdents64_root);   EXPECT_SYSNE(1, test_getdents64("/"), -1); break;
 		CASE_TEST(getdents64_null);   EXPECT_SYSER(1, test_getdents64("/dev/null"), -1, ENOTDIR); break;
 		CASE_TEST(gettimeofday_null); EXPECT_SYSZR(1, gettimeofday(NULL, NULL)); break;
+#ifdef NOLIBC
 		CASE_TEST(gettimeofday_bad1); EXPECT_SYSER(1, gettimeofday((void *)1, NULL), -1, EFAULT); break;
 		CASE_TEST(gettimeofday_bad2); EXPECT_SYSER(1, gettimeofday(NULL, (void *)1), -1, EFAULT); break;
 		CASE_TEST(gettimeofday_bad2); EXPECT_SYSER(1, gettimeofday(NULL, (void *)1), -1, EFAULT); break;
+#endif
 		CASE_TEST(ioctl_tiocinq);     EXPECT_SYSZR(1, ioctl(0, TIOCINQ, &tmp)); break;
 		CASE_TEST(ioctl_tiocinq);     EXPECT_SYSZR(1, ioctl(0, TIOCINQ, &tmp)); break;
 		CASE_TEST(link_root1);        EXPECT_SYSER(1, link("/", "/"), -1, EEXIST); break;
@@ -703,7 +742,11 @@ int main(int argc, char **argv, char **envp)
 		 * exit with status code 2N+1 when N is written to 0x501. We
 		 * hard-code the syscall here as it's arch-dependent.
 		 */
+#if defined(_NOLIBC_SYS_H)
 		else if (my_syscall3(__NR_ioperm, 0x501, 1, 1) == 0)
+#else
+		else if (ioperm(0x501, 1, 1) == 0)
+#endif
 			asm volatile ("outb %%al, %%dx" :: "d"(0x501), "a"(0));
 		/* if it does nothing, fall back to the regular panic */
 #endif
-- 
2.17.5

[PATCH 13/17] selftests/nolibc: add a "kernel" target to build the kernel with the initramfs

[Willy Tarreau]

The "kernel" target rebuilds the kernel with the current config for the
selected arch, with an initramfs containing the nolibc-test utility.

Since image names depend on the architecture, the currently supported
ones are referenced and resolved based on the architecture.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/Makefile | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index fd0a67082334..4a2ab0e73ce2 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -12,6 +12,16 @@ include $(srctree)/scripts/subarch.include
 ARCH = $(SUBARCH)
 endif
 
+# kernel image names by architecture
+IMAGE_i386    = arch/x86/boot/bzImage
+IMAGE_x86     = arch/x86/boot/bzImage
+IMAGE_arm64   = arch/arm64/boot/Image
+IMAGE_arm     = arch/arm/boot/zImage
+IMAGE_mips    = vmlinuz
+IMAGE_riscv   = arch/riscv/boot/Image
+IMAGE         = $(IMAGE_$(ARCH))
+IMAGE_NAME    = $(notdir $(IMAGE))
+
 # OUTPUT is only set when run from the main makefile, otherwise
 # it defaults to this nolibc directory.
 OUTPUT ?= $(CURDIR)/
@@ -36,6 +46,9 @@ initramfs: nolibc-test
 	$(call QUIET_INSTALL, initramfs/init)
 	$(Q)cp nolibc-test initramfs/init
 
+kernel: initramfs
+	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
+
 clean:
 	$(call QUIET_CLEAN, nolibc-test)
 	$(Q)rm -f nolibc-test
-- 
2.17.5

[PATCH 14/17] selftests/nolibc: add a "defconfig" target

[Willy Tarreau]

While most archs will work fine with "make defconfig", not all will
do, and it's not always easy to remember the most suitable choice to
use for a specific architecture.

This adds a "defconfig" target to the Makefile so that one may easily
run "make -C ... defconfig" and make sure to clean and rebuild a fresh
config. This is *not* used by default because we want to preserve the
user's config by default.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/Makefile | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 4a2ab0e73ce2..c104719eae8b 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -22,6 +22,15 @@ IMAGE_riscv   = arch/riscv/boot/Image
 IMAGE         = $(IMAGE_$(ARCH))
 IMAGE_NAME    = $(notdir $(IMAGE))
 
+# default kernel configurations that appear to be usable
+DEFCONFIG_i386    = defconfig
+DEFCONFIG_x86     = defconfig
+DEFCONFIG_arm64   = defconfig
+DEFCONFIG_arm     = multi_v7_defconfig
+DEFCONFIG_mips    = malta_defconfig
+DEFCONFIG_riscv   = defconfig
+DEFCONFIG         = $(DEFCONFIG_$(ARCH))
+
 # OUTPUT is only set when run from the main makefile, otherwise
 # it defaults to this nolibc directory.
 OUTPUT ?= $(CURDIR)/
@@ -46,6 +55,9 @@ initramfs: nolibc-test
 	$(call QUIET_INSTALL, initramfs/init)
 	$(Q)cp nolibc-test initramfs/init
 
+defconfig:
+	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper $(DEFCONFIG) prepare
+
 kernel: initramfs
 	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
 
-- 
2.17.5

[PATCH 15/17] selftests/nolibc: add a "run" target to start the kernel in QEMU

[Willy Tarreau]

The "run" target will build the kernel and start it in QEMU. The
"rerun" target will not have the kernel dependency and will just try
to start QEMU. The QEMU architecture used to start the kernel is
derived from the configured ARCH. This might need to be improved
for archs which include different variants under the same name
(mips vs mipsel, +/-64, riscv32 vs riscv64). This could be tested
for i386, x86, arm, arm64, mips and riscv (the later two reporting
issues on some tests).

It is possible to pass a test specification for nolibc-test in the TEST
variable, which will be passed as-is as NOLIBC_TEST.

On success, the number of successful tests is printed. On failure, failed
lines are individually printed.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/Makefile | 33 +++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index c104719eae8b..7c1f5360f454 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -31,6 +31,27 @@ DEFCONFIG_mips    = malta_defconfig
 DEFCONFIG_riscv   = defconfig
 DEFCONFIG         = $(DEFCONFIG_$(ARCH))
 
+# optional tests to run (default = all)
+TEST =
+
+# QEMU_ARCH: arch names used by qemu
+QEMU_ARCH_i386    = i386
+QEMU_ARCH_x86     = x86_64
+QEMU_ARCH_arm64   = aarch64
+QEMU_ARCH_arm     = arm
+QEMU_ARCH_mips    = mipsel  # works with malta_defconfig
+QEMU_ARCH_riscv   = riscv64
+QEMU_ARCH         = $(QEMU_ARCH_$(ARCH))
+
+# QEMU_ARGS : some arch-specific args to pass to qemu
+QEMU_ARGS_i386    = -M pc -append "console=ttyS0,9600 i8042.noaux panic=-1 $(TEST:%=NOLIBC_TEST=%)"
+QEMU_ARGS_x86     = -M pc -append "console=ttyS0,9600 i8042.noaux panic=-1 $(TEST:%=NOLIBC_TEST=%)"
+QEMU_ARGS_arm64   = -M virt -cpu cortex-a53 -append "panic=-1 $(TEST:%=NOLIBC_TEST=%)"
+QEMU_ARGS_arm     = -M virt -append "panic=-1 $(TEST:%=NOLIBC_TEST=%)"
+QEMU_ARGS_mips    = -M malta -append "panic=-1 $(TEST:%=NOLIBC_TEST=%)"
+QEMU_ARGS_riscv   = -M virt -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
+QEMU_ARGS         = $(QEMU_ARGS_$(ARCH))
+
 # OUTPUT is only set when run from the main makefile, otherwise
 # it defaults to this nolibc directory.
 OUTPUT ?= $(CURDIR)/
@@ -61,8 +82,20 @@ defconfig:
 kernel: initramfs
 	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
 
+# run the tests after building the kernel
+run: kernel
+	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
+	$(Q)grep -w FAIL "$(CURDIR)/run.out" && echo "See all results in $(CURDIR)/run.out" || echo "$$(grep -c ^[0-9].*OK $(CURDIR)/run.out) test(s) passed."
+
+# re-run the tests from an existing kernel
+rerun:
+	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
+	$(Q)grep -w FAIL "$(CURDIR)/run.out" && echo "See all results in $(CURDIR)/run.out" || echo "$$(grep -c ^[0-9].*OK $(CURDIR)/run.out) test(s) passed."
+
 clean:
 	$(call QUIET_CLEAN, nolibc-test)
 	$(Q)rm -f nolibc-test
 	$(call QUIET_CLEAN, initramfs)
 	$(Q)rm -rf initramfs
+	$(call QUIET_CLEAN, run.out)
+	$(Q)rm -rf run.out
-- 
2.17.5

[PATCH 16/17] selftests/nolibc: "sysroot" target installs a local copy of the sysroot

[Willy Tarreau]

It's not convenient to rely on a sysroot built in another directory,
especially when running cross-compilation tests, where one has to
switch back and forth between directories.

Let's make it possible to install the sysroot directly in the test
directory. It's not big and even benefits from being copied by arch
so that it's easier to switch between archs if needed. The new
"sysroot" target does this, it just calls "headers_standalone" from
nolibc to install the sysroot right here.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/Makefile | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 7c1f5360f454..210f5369fdfc 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -67,9 +67,16 @@ LDFLAGS := -s
 
 all: nolibc-test
 
-nolibc-test: nolibc-test.c
+sysroot: sysroot/$(ARCH)/include
+
+sysroot/$(ARCH)/include:
+	$(QUIET_MKDIR)mkdir -p sysroot
+	$(Q)$(MAKE) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone
+	$(Q)mv sysroot/sysroot sysroot/$(ARCH)
+
+nolibc-test: nolibc-test.c sysroot/$(ARCH)/include
 	$(QUIET_CC)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ \
-	  -nostdlib -static -include ../../../include/nolibc/nolibc.h $^ -lgcc
+	  -nostdlib -static -Isysroot/$(ARCH)/include $< -lgcc
 
 initramfs: nolibc-test
 	$(QUIET_MKDIR)mkdir -p initramfs
@@ -93,6 +100,8 @@ rerun:
 	$(Q)grep -w FAIL "$(CURDIR)/run.out" && echo "See all results in $(CURDIR)/run.out" || echo "$$(grep -c ^[0-9].*OK $(CURDIR)/run.out) test(s) passed."
 
 clean:
+	$(call QUIET_CLEAN, sysroot)
+	$(Q)rm -rf sysroot
 	$(call QUIET_CLEAN, nolibc-test)
 	$(Q)rm -f nolibc-test
 	$(call QUIET_CLEAN, initramfs)
-- 
2.17.5

[PATCH 17/17] selftests/nolibc: add a "help" target

[Willy Tarreau]

It presents the supported targets, and becomes the default target to
save the user from having to read the makefile. The "all" target was
placed after it and now points to "run" to do everything since it's
no longer the default one.

Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 tools/testing/selftests/nolibc/Makefile | 27 ++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 210f5369fdfc..69ea659caca9 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -65,7 +65,32 @@ endif
 CFLAGS  ?= -Os -fno-ident -fno-asynchronous-unwind-tables
 LDFLAGS := -s
 
-all: nolibc-test
+help:
+	@echo "Supported targets under selftests/nolibc:"
+	@echo "  all          call the \"run\" target below"
+	@echo "  help         this help"
+	@echo "  sysroot      create the nolibc sysroot here (uses \$$ARCH)"
+	@echo "  nolibc-test  build the executable (uses \$$CC and \$$CROSS_COMPILE)"
+	@echo "  initramfs    prepare the initramfs with nolibc-test"
+	@echo "  defconfig    create a fresh new default config (uses \$$ARCH)"
+	@echo "  kernel       (re)build the kernel with the initramfs (uses \$$ARCH)"
+	@echo "  run          runs the kernel in QEMU after building it (uses \$$ARCH, \$$TEST)"
+	@echo "  rerun        runs a previously prebuilt kernel in QEMU (uses \$$ARCH, \$$TEST)"
+	@echo "  clean        clean the sysroot, initramfs, build and output files"
+	@echo ""
+	@echo "The output file is \"run.out\". Test ranges may be passed using \$$TEST."
+	@echo ""
+	@echo "Currently using the following variables:"
+	@echo "  ARCH          = $(ARCH)"
+	@echo "  CROSS_COMPILE = $(CROSS_COMPILE)"
+	@echo "  CC            = $(CC)"
+	@echo "  OUTPUT        = $(OUTPUT)"
+	@echo "  TEST          = $(TEST)"
+	@echo "  QEMU_ARCH     = $(if $(QEMU_ARCH),$(QEMU_ARCH),UNKNOWN_ARCH) [determined from \$$ARCH]"
+	@echo "  IMAGE_NAME    = $(if $(IMAGE_NAME),$(IMAGE_NAME),UNKNOWN_ARCH) [determined from \$$ARCH]"
+	@echo ""
+
+all: run
 
 sysroot: sysroot/$(ARCH)/include
 
-- 
2.17.5

Re: [PATCH 00/17] nolibc: add preliminary self tests

[Paul E. McKenney]

On Tue, Jul 19, 2022 at 11:44:31PM +0200, Willy Tarreau wrote:
> Hi Paul,
> 
> as previously promised, here comes the nolibc update which introduces the
> minimal self-test infrastructure that aims at being reasonably easy to
> expand further.
> 
> It's based on your branch "dev.2022.06.30b" that contains the previous
> minor fixes that aimed at addressing Linus' concerns about the build
> process inconsistencies.
> 
> The way it works tries to mimmick as much as possible the regular build
> process, so that it reuses the same ARCH, CC, CROSS_COMPILE to build the
> test program, that will be embedded into an initramfs and the kernel is
> (re)built with that initramfs. Then you can decide to run that kernel
> under QEMU for the supported archs, and the output of the tests appears
> in an output text file in a format that's easily greppable and diffable.
> A single target "run" does everything.
> 
> By default it will reuse your existing .config (so that developers
> continue to use their regular config handling), though it can also
> create a known-to-work defconfig for each arch. The reason behind this
> is that it took me a moment to figure certain defconfig + machine name
> combinations and I found it better to put them there once for all.
> 
> I've successfully tested it on arm, arm64, i386, x86_64. riscv64 works
> except two syscalls which return unexpected errors, and mips segfaults
> in sbrk(). I don't know why yet, but this proves that it's worth having
> such a test.

Excellent, thank you!!!

As we often said during my misspent youth, "If it ain't tested, it
don't work."

But I do get "71 test(s) passed." when running on x86.  I will let you
decide whether that constitutes all being well or indicates a bug in
the tests.   ;-)

> There are not that many tests yet (71), those that have to run can be
> filtered either from the program's command line or from a NOLIBC_TEST
> environment variable so that it's possible to skip broken ones or to
> focus on a few ranges only.
> 
> Tests are numerically numbered, and are conveniently handled in a
> switch/case statement so that a relative line number assigns the number
> to the test. That's convenient because the vast majority of syscall tests
> are one-liners. This sometimes slightly upsets check-patch when lines get
> moderately long but that significantly improves legibility.
> 
> There are expectation for both successes and failures (e.g. -1 ENOTDIR).
> I'm sure this can be improved later (and that's the goal). Right now it
> covers two test families:
>   - syscalls
>   - stdlib    (str* functions mostly)
> 
> I suspect that over time we might want to split syscalls into different
> parts (e.g. core, fs, etc maybe) but I could be wrong.

This is a good start, and we can let experience drive any additional
changes that might be required.

> The program can automatically modulate QEMU's return value on x86 when
> QEMU is run with the appropriate options, but for now I'm not using it
> as I felt like it didn't bring much value, and the output is more useful.
> That's debatable, and maybe some might want to use it in bisect scripts
> for example. It's too early to say IMHO.

For the moment, grepping the output works.  And perhaps indefinitely.

> Oh, I also arranged the code so that the test also builds with glibc. I
> noticed that when adding a new test that fails, sometimes it's convenient
> to see if it's the nolibc part that's broken or the test. I don't find
> this critical but the required includes and ifdefs are there so that it
> should be easy to maintain over time as well.

If nothing else, the ability to run against glibc is a good way to test
the test.

> I'm obviously interested in comments, but really, I don't want to
> overdesign something for a first step, it remains a very modest test
> program and I'd like that it remains easy to hack on it and to contribute
> new tests that are deemed useful.

I am good with a simple starting point.

> I'm CCing the few who already contributed some patches and/or expressed
> interest, as well as Linus who had a first bad experience when trying to
> test it, hoping this one will be better. I'm pasting below [1] a copy of
> a test on x86_64 below, that's summed up as "71 test(s) passed" at the
> end of the "run" target.
> 
> If there's no objection, it would be nice to have this with your current
> series, as it definitely helps spot and fix the bugs. In parallel I'll see
> if I can figure the problems with the two tests that fail each on a
> specific arch and I might possibly have a few extra fixes for the current
> nolibc.

This series is now on the -rcu tree's "dev" branch.  I got two almost
identical copies of patch 7, so I took the later of the two.  Please let
me know if I guessed wrong.

							Thanx, Paul

> Thank you!
> Willy
> 
> [1] example output
> ----8<----
> Running test 'syscall'
> 0 getpid = 1                             [OK]
> 1 getppid = 0                            [OK]
> 5 getpgid_self = 0                       [OK]
> 6 getpgid_bad = -1 ESRCH                 [OK]
> 7 kill_0 = 0                             [OK]
> 8 kill_CONT = 0                          [OK]
> 9 kill_BADPID = -1 ESRCH                 [OK]
> 10 sbrk = 0                              [OK]
> 11 brk = 0                               [OK]
> 12 chdir_root = 0                        [OK]
> 13 chdir_dot = 0                         [OK]
> 14 chdir_blah = -1 ENOENT                [OK]
> 15 chmod_net = 0                         [OK]
> 16 chmod_self = -1 EPERM                 [OK]
> 17 chown_self = -1 EPERM                 [OK]
> 18 chroot_root = 0                       [OK]
> 19 chroot_blah = -1 ENOENT               [OK]
> 20 chroot_exe = -1 ENOTDIR               [OK]
> 21 close_m1 = -1 EBADF                   [OK]
> 22 close_dup = 0                         [OK]
> 23 dup_0 = 3                             [OK]
> 24 dup_m1 = -1 EBADF                     [OK]
> 25 dup2_0 = 100                          [OK]
> 26 dup2_m1 = -1 EBADF                    [OK]
> 27 dup3_0 = 100                          [OK]
> 28 dup3_m1 = -1 EBADF                    [OK]
> 29 execve_root = -1 EACCES               [OK]
> 30 getdents64_root = 120                 [OK]
> 31 getdents64_null = -1 ENOTDIR          [OK]
> 32 gettimeofday_null = 0                 [OK]
> 38 ioctl_tiocinq = 0                     [OK]
> 39 ioctl_tiocinq = 0                     [OK]
> 40 link_root1 = -1 EEXIST                [OK]
> 41 link_blah = -1 ENOENT                 [OK]
> 42 link_dir = -1 EPERM                   [OK]
> 43 link_cross = -1 EXDEV                 [OK]
> 44 lseek_m1 = -1 EBADF                   [OK]
> 45 lseek_0 = -1 ESPIPE                   [OK]
> 46 mkdir_root = -1 EEXIST                [OK]
> 47 open_tty = 3                          [OK]
> 48 open_blah = -1 ENOENT                 [OK]
> 49 poll_null = 0                         [OK]
> 50 poll_stdout = 1                       [OK]
> 51 poll_fault = -1 EFAULT                [OK]
> 52 read_badf = -1 EBADF                  [OK]
> 53 sched_yield = 0                       [OK]
> 54 select_null = 0                       [OK]
> 55 select_stdout = 1                     [OK]
> 56 select_fault = -1 EFAULT              [OK]
> 57 stat_blah = -1 ENOENT                 [OK]
> 58 stat_fault = -1 EFAULT                [OK]
> 59 symlink_root = -1 EEXIST              [OK]
> 60 unlink_root = -1 EISDIR               [OK]
> 61 unlink_blah = -1 ENOENT               [OK]
> 62 wait_child = -1 ECHILD                [OK]
> 63 waitpid_min = -1 ESRCH                [OK]
> 64 waitpid_child = -1 ECHILD             [OK]
> 65 write_badf = -1 EBADF                 [OK]
> 66 write_zero = 0                        [OK]
> Errors during this test: 0
> 
> Running test 'stdlib'
> 0 getenv_TERM = <linux>                  [OK]
> 1 getenv_blah = <(null)>                 [OK]
> 2 setcmp_blah_blah = 0                   [OK]
> 3 setcmp_blah_blah2 = -50                [OK]
> 4 setncmp_blah_blah = 0                  [OK]
> 5 setncmp_blah_blah4 = 0                 [OK]
> 6 setncmp_blah_blah5 = -53               [OK]
> 7 setncmp_blah_blah6 = -54               [OK]
> 8 strchr_foobar_o = <oobar>              [OK]
> 9 strchr_foobar_z = <(null)>             [OK]
> 10 strrchr_foobar_o = <obar>             [OK]
> 11 strrchr_foobar_z = <(null)>           [OK]
> Errors during this test: 0
> 
> Total number of errors: 0
> ---->8----
> 
> 
> --
> 
> Willy Tarreau (17):
>   tools/nolibc: make argc 32-bit in riscv startup code
>   tools/nolibc: fix build warning in sys_mmap() when my_syscall6 is not
>     defined
>   tools/nolibc: make sys_mmap() automatically use the right __NR_mmap
>     definition
>   selftests/nolibc: add basic infrastructure to ease creation of nolibc
>     tests
>   selftests/nolibc: support a test definition format
>   selftests/nolibc: implement a few tests for various syscalls
>   selftests/nolibc: add a few tests for some stdlib functions
>   selftests/nolibc: exit with poweroff on success when getpid() == 1
>   selftests/nolibc: on x86, support exiting with isa-debug-exit
>   selftests/nolibc: recreate and populate /dev and /proc if missing
>   selftests/nolibc: condition some tests on /proc existence
>   selftests/nolibc: support glibc as well
>   selftests/nolibc: add a "kernel" target to build the kernel with the
>     initramfs
>   selftests/nolibc: add a "defconfig" target
>   selftests/nolibc: add a "run" target to start the kernel in QEMU
>   selftests/nolibc: "sysroot" target installs a local copy of the
>     sysroot
>   selftests/nolibc: add a "help" target
> 
>  MAINTAINERS                                  |   1 +
>  tools/include/nolibc/arch-riscv.h            |   2 +-
>  tools/include/nolibc/sys.h                   |   4 +-
>  tools/testing/selftests/nolibc/Makefile      | 135 ++++
>  tools/testing/selftests/nolibc/nolibc-test.c | 757 +++++++++++++++++++
>  5 files changed, 896 insertions(+), 3 deletions(-)
>  create mode 100644 tools/testing/selftests/nolibc/Makefile
>  create mode 100644 tools/testing/selftests/nolibc/nolibc-test.c
> 
> -- 
> 2.17.5
> 

Re: [PATCH 00/17] nolibc: add preliminary self tests

[Willy Tarreau]

On Tue, Jul 19, 2022 at 03:49:47PM -0700, Paul E. McKenney wrote:
> But I do get "71 test(s) passed." when running on x86.  I will let you
> decide whether that constitutes all being well or indicates a bug in
> the tests.   ;-)

Technically speaking both are possible, but one is more likely :-)

> > The program can automatically modulate QEMU's return value on x86 when
> > QEMU is run with the appropriate options, but for now I'm not using it
> > as I felt like it didn't bring much value, and the output is more useful.
> > That's debatable, and maybe some might want to use it in bisect scripts
> > for example. It's too early to say IMHO.
> 
> For the moment, grepping the output works.  And perhaps indefinitely.

That's my intuition as well, given that there will always be a bit of
scripting around that anyway.

> This series is now on the -rcu tree's "dev" branch.

Thank you!

> I got two almost
> identical copies of patch 7, so I took the later of the two.  Please let
> me know if I guessed wrong.

Oh you're right, I'm sorry about that. I adjusted one commit message late
and failed to erase the previous one from the directory. In either case
we really don't care but I thought that the one mentioning "stdlib" which
is the term used in the test was better.

Thanks again for your time,
Willy

Re: [PATCH 00/17] nolibc: add preliminary self tests

[Ammar Faizi]

On 7/20/22 4:44 AM, Willy Tarreau wrote:
> I'm obviously interested in comments, but really, I don't want to
> overdesign something for a first step, it remains a very modest test
> program and I'd like that it remains easy to hack on it and to contribute
> new tests that are deemed useful.

I personally hate how the test framework mandates:

   "There must be exactly one test per line."

which makes the test case, for example, one long liner like this:

   if ((p1 = p2 = sbrk(4096)) != (void *)-1) p2 = sbrk(-4096); EXPECT_SYSZR(1, (p2 == (void *)-1) || p2 == p1); break;

that's ugly and hard to read. Can we get rid of this "one test per line" rule?

It would be great if we followed the documented coding style that says:

    "Statements longer than 80 columns should be broken into sensible chunks,
     unless exceeding 80 columns significantly increases readability and does
     not hide information." [1]

What we have here doesn't really increase the readability at all. Maybe
it's too late for 5.20, just for next in case we want to fix it.

Willy?

[1]: https://www.kernel.org/doc/html/v5.15/process/coding-style.html#breaking-long-lines-and-strings

-- 
Ammar Faizi

Re: [PATCH 00/17] nolibc: add preliminary self tests

[Willy Tarreau]

On Wed, Jul 20, 2022 at 11:03:58PM +0700, Ammar Faizi wrote:
> On 7/20/22 4:44 AM, Willy Tarreau wrote:
> > I'm obviously interested in comments, but really, I don't want to
> > overdesign something for a first step, it remains a very modest test
> > program and I'd like that it remains easy to hack on it and to contribute
> > new tests that are deemed useful.
> 
> I personally hate how the test framework mandates:
> 
>   "There must be exactly one test per line."

I know, that's a design choice that makes them trivial to add, because
it's the compiler that assigns the test IDs, and it comes with a non
negligible benefit.

> which makes the test case, for example, one long liner like this:
> 
>   if ((p1 = p2 = sbrk(4096)) != (void *)-1) p2 = sbrk(-4096); EXPECT_SYSZR(1, (p2 == (void *)-1) || p2 == p1); break;
> 
> that's ugly and hard to read. Can we get rid of this "one test per line" rule?

If you find a better solution, I'm open. What I certainly don't want
to do is to have to cross-reference IDs with arrays, nor start to stack
endless if/else that are even more painful to deal with, or have to
renumber everything by hand once in a while.

> It would be great if we followed the documented coding style that says:
> 
>    "Statements longer than 80 columns should be broken into sensible chunks,
>     unless exceeding 80 columns significantly increases readability and does
>     not hide information." [1]

Admittedly this is not core code but debugging code running in userland
to help developers spot bugs in their code which is somewhere else and
well maintained. I personally think that the tradeoff is positive here,
i.e. non-pretty but easily hackable short tests that encourage additions
and variations. The ease of adding tests allowed me to create 71 of them
in a single afternoon and two of them brought me bugs in existing code,
which I think is efficient. But I'm not fond of the approach either, I
just couldn't produce anything as efficient that was prettier, but I'm
quite open to being proven wrong by an alternate proposal.

> What we have here doesn't really increase the readability at all. Maybe
> it's too late for 5.20, just for next in case we want to fix it.

The goal was not to increase *readability* but *writability*. We're
still missing test for most syscalls and I would like them to be added
quickly so that we can continue to add tested code. The readability I
care about is understanding the output. When I'm seeing:

  ...
  29 execve_root = -1 EACCES               [OK]
  30 getdents64_root = -1 EBADF           [FAIL]
  31 getdents64_null = -1 EBADF  != (-1 ENOTDIR) [FAIL]
  32 gettimeofday_null = 0                 [OK]
  ...

on riscv64, I don't have to search long to figure that we did something
wrong with getdents64() on this arch and that the error path works
differently. Similarly, this on mips:

  8 kill_CONT = 0                          [OK]
  9 kill_BADPID = -1 ESRCH                 [OK]
  10 sbrkdo_page_fault(): sending SIGSEGV to init for invalid read access from 0000000a
  epc = 0000000a in init[400000+4000]
  ra  = 0000000a in init[400000+4000]
  Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

tells me that sbrk() definitely doesn't work there.

In both cases I know what and where to look without even having to *read*
that test. This does matter to me, as a developer of the component subject
to the test.

But again, I'm open to better proposals. I reached the limits of my
imagination there, but I do value the ability to "yyp" one line, change
two arguments and gain one extra test for a different combination, and
I really do not want to lose that simplicity. Note that for more complex
tests, it's trivial to add a dedicated function and that's what was done
for getdents64() which also serves as an example.

Willy

Re: [PATCH 00/17] nolibc: add preliminary self tests

[Ammar Faizi]

On 7/20/22 11:20 PM, Willy Tarreau wrote:
> What I certainly don't want to do is to have to cross-reference IDs
> with arrays, nor start to stack endless if/else that are even more
> painful to deal with, or have to renumber everything by hand once in
> a while.

Noted.

> But again, I'm open to better proposals. I reached the limits of my
> imagination there, but I do value the ability to "yyp" one line, change
> two arguments and gain one extra test for a different combination, and
> I really do not want to lose that simplicity. Note that for more complex
> tests, it's trivial to add a dedicated function and that's what was done
> for getdents64() which also serves as an example.

OK, I understand the reason behind this now. I and Fernanda will try
to visit this again at around 5.20-rc. *If* we can find a better
design that matches your requirements, we will send you an RFC to
improve it too.

Thank you!

-- 
Ammar Faizi

Re: [PATCH 00/17] nolibc: add preliminary self tests

[Willy Tarreau]

On Thu, Jul 21, 2022 at 12:05:14AM +0700, Ammar Faizi wrote:
> > But again, I'm open to better proposals. I reached the limits of my
> > imagination there, but I do value the ability to "yyp" one line, change
> > two arguments and gain one extra test for a different combination, and
> > I really do not want to lose that simplicity. Note that for more complex
> > tests, it's trivial to add a dedicated function and that's what was done
> > for getdents64() which also serves as an example.
> 
> OK, I understand the reason behind this now. I and Fernanda will try
> to visit this again at around 5.20-rc. *If* we can find a better
> design that matches your requirements, we will send you an RFC to
> improve it too.

You would be very welcome, thank you!

Willy