From: Zhangjin Wu <falcon@tinylab.org>
To: w@1wt.eu
Cc: arnd@arndb.de, falcon@tinylab.org, linux-kernel@vger.kernel.org,
linux-kselftest@vger.kernel.org, linux-riscv@lists.infradead.org,
thomas@t-8ch.de
Subject: Re: [PATCH 1/4] selftests/nolibc: add a test-report target
Date: Wed, 7 Jun 2023 22:15:02 +0800 [thread overview]
Message-ID: <20230607141502.864645-1-falcon@tinylab.org> (raw)
In-Reply-To: <ZIB792FtG6ibOudp@1wt.eu>
On Wed, Jun 07, 2023 at 01:52:00PM +0800, Zhangjin Wu wrote:
> > Hi, Willy
> >
> (...)
> > >
> > > Ok, thanks, what about this?
> > >
> > > # LOG_REPORT: report the test results
> > > LOG_REPORT := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
> > > END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
> > > printf(" See all results in %s\n", ARGV[1]); }'
> > >
> > > run-user: nolibc-test
> > > $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > > $(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > >
> > > run: kernel
> > > $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> > > $(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > >
> > > rerun:
> > > $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> > > $(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > >
> > > Or we directly add a standalone test report script? something like
> > > tools/testing/selftests/nolibc/report.sh
> > >
> > > #!/bin/sh
> > > #
> > > # report.sh -- report the test results of nolibc-test
> > > #
> > >
> > > LOG_FILE=$1
> > > [ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out"
> > >
> > > awk '
> > > /\[OK\][\r]*$$/{ p++ }
> > > /\[FAIL\][\r]*$$/{ f++ }
> > > /\[SKIPPED\][\r]*$$/{ s++ }
> > >
> > > END {
> > > printf("%d test(s) passed, %d skipped, %d failed.", p, s, f);
> > > printf(" See all results in %s\n", ARGV[1]);
> > > }' $LOG_FILE
> > >
> > > And use it like this:
> > >
> > > LOG_REPORT = $(CURDIR)/report.sh
> > >
> >
> > I plan to renew this patchset, which one of the above methods do you
> > prefer?
>
> IFF it needs to be done I prefer the macro in the Makefile to avoid
> depending on external scripts that are useless outside of the makefile.
> BUT, my point remains that I adopted this so that I could quickly and
> visually check that everything was OK. I'm fine with any other method
> but I do not want to have to carefully read all these lines to make
> sure I'm not mixing a "8" with a "0" (I'm mentioning this one because
> it's exactly the one I had when I decided to add the extra values).
> For example if you prepend "FAILURE: ", "WARNING: ", "SUCCESS: " in
> front of these lines to summarize them depending on the highest level
> encountered (success, skipped, failed), then I'm fine because it's
> easy to check that all lines show the same word.
>
Ok.
> > For the always print statement:
> >
> > printf(" See all results in %s\n", ARGV[1]); }'
>
> Then please put it on its own line without the leading space, this
> will be even more readable.
>
It is a good balance.
This may be more useful if we run this from the kselftest framework,
seems it is not able to run it from the 'kselftest' target currently,
here shares something I have tried:
I tried something like this, seems run-user works, but defconfig and run
not.
diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 4a3a105e1fdf..70693f6501c6 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -83,6 +83,8 @@ CFLAGS ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 \
$(CFLAGS_$(ARCH)) $(CFLAGS_STACKPROTECTOR)
LDFLAGS := -s
+NOLIBC_SUBTARGETS ?= run-user
+
help:
@echo "Supported targets under selftests/nolibc:"
@echo " all call the \"run\" target below"
@@ -110,7 +112,9 @@ help:
@echo " IMAGE_NAME = $(if $(IMAGE_NAME),$(IMAGE_NAME),UNKNOWN_ARCH) [determined from \$$ARCH]"
@echo ""
-all: run
+run_tests: $(NOLIBC_SUBTARGETS)
+
+all: $(NOLIBC_SUBTARGETS)
This can be triggered from the top-level kselftest framework:
$ make -C /path/to/linux-stable kselftest TARGETS=nolibc NOLIBC_SUBTARGETS=run-user
And seems we still not support O= currently either.
> > I will paste the reason why I need it, as mentioned in [1], if you still
> > need a clean test report, I will give up this change ;-)
>
> No worries, I don't want to be annoying if you need something, but I
> don't want to be annoyed by changes either :-)
>
Thanks,
Zhangjin
> thanks,
> Willy
next prev parent reply other threads:[~2023-06-07 14:15 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-05 3:47 [PATCH 0/4] selftests/nolibc: fix up and improve test report Zhangjin Wu
2023-06-05 3:48 ` [PATCH 1/4] selftests/nolibc: add a test-report target Zhangjin Wu
2023-06-05 4:18 ` Willy Tarreau
2023-06-05 6:54 ` Zhangjin Wu
2023-06-07 5:52 ` Zhangjin Wu
2023-06-07 12:45 ` Willy Tarreau
2023-06-07 14:15 ` Zhangjin Wu [this message]
2023-06-05 3:50 ` [PATCH 2/4] selftests/nolibc: allow run test-report directly Zhangjin Wu
2023-06-05 3:57 ` [PATCH 3/4] selftests/nolibc: always print the log file Zhangjin Wu
2023-06-05 4:20 ` Willy Tarreau
2023-06-05 7:05 ` Zhangjin Wu
2023-06-05 3:58 ` [PATCH 4/4] selftests/nolibc: fix up test-report for run target Zhangjin Wu
2023-06-05 4:23 ` Willy Tarreau
2023-06-05 6:32 ` [PATCH 0/4] selftests/nolibc: fix up and improve test report Willy Tarreau
2023-06-05 10:53 ` Zhangjin Wu
2023-06-06 4:50 ` Willy Tarreau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230607141502.864645-1-falcon@tinylab.org \
--to=falcon@tinylab.org \
--cc=arnd@arndb.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=thomas@t-8ch.de \
--cc=w@1wt.eu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).