All of lore.kernel.org
 help / color / mirror / Atom feed
From: chrubis at suse.cz (Cyril Hrubis)
Subject: [RFC PATCH 1/2] selftests: Start shell API
Date: Mon, 8 Apr 2019 13:38:42 +0200	[thread overview]
Message-ID: <20190408113841.GA2418@rei.lan> (raw)
In-Reply-To: <20190406214915.16914-2-pvorel@suse.cz>

Hi!
> +ksft_pass()
> +{
> +	echo "[PASS] $TEST: $1"
> +	exit $KSFT_PASS
> +}
> +
> +ksft_fail()
> +{
> +	echo "[FAIL] $TEST: $1"
> +	exit $KSFT_FAIL
> +}

I think that the main disadvantage here is that these functions call
exit instead of storing the results which leads to a common pattern of
passing the result up the function call chain which is prone to errors.

What I have learned the hard way over the years is that the result
reporting should be separated from the functions that exit the tests and
that the test code should not be trusted with passing the overall test
result at the end. I've seen too many cases where the actuall failure
was ignored becaues the failure was lost on it's way to the main
function.

Another lesson is that tests shouldn't implement the main() function,
that is something that the test library should do, which allows for
resources to be listed in a declarative way instead of calling init
funcitons at the start of the tests. Which means that in LTP you can say
"mount at least 512MB device formatted with ext4 to this mount point"
and all this handled in the test library before the actual test starts.

As the last point this completely misses a cleanup callback support,
i.e. function that is called to clean up if you need to exit in the
middle of a test in a case of an error.

-- 
Cyril Hrubis
chrubis at suse.cz

WARNING: multiple messages have this Message-ID
From: chrubis@suse.cz (Cyril Hrubis)
Subject: [RFC PATCH 1/2] selftests: Start shell API
Date: Mon, 8 Apr 2019 13:38:42 +0200	[thread overview]
Message-ID: <20190408113841.GA2418@rei.lan> (raw)
Message-ID: <20190408113842.hfmf9dtYfXTDzU2pdWvb0VqboeSd9YwPLpVWdV7aMmc@z> (raw)
In-Reply-To: <20190406214915.16914-2-pvorel@suse.cz>

Hi!
> +ksft_pass()
> +{
> +	echo "[PASS] $TEST: $1"
> +	exit $KSFT_PASS
> +}
> +
> +ksft_fail()
> +{
> +	echo "[FAIL] $TEST: $1"
> +	exit $KSFT_FAIL
> +}

I think that the main disadvantage here is that these functions call
exit instead of storing the results which leads to a common pattern of
passing the result up the function call chain which is prone to errors.

What I have learned the hard way over the years is that the result
reporting should be separated from the functions that exit the tests and
that the test code should not be trusted with passing the overall test
result at the end. I've seen too many cases where the actuall failure
was ignored becaues the failure was lost on it's way to the main
function.

Another lesson is that tests shouldn't implement the main() function,
that is something that the test library should do, which allows for
resources to be listed in a declarative way instead of calling init
funcitons at the start of the tests. Which means that in LTP you can say
"mount at least 512MB device formatted with ext4 to this mount point"
and all this handled in the test library before the actual test starts.

As the last point this completely misses a cleanup callback support,
i.e. function that is called to clean up if you need to exit in the
middle of a test in a case of an error.

-- 
Cyril Hrubis
chrubis at suse.cz

  parent reply	other threads:[~2019-04-08 11:38 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-06 21:49 [RFC PATCH 0/2] Kselftest shell (or even C) API pvorel
2019-04-06 21:49 ` Petr Vorel
2019-04-06 21:49 ` [RFC PATCH 1/2] selftests: Start shell API pvorel
2019-04-06 21:49   ` Petr Vorel
2019-04-08 11:06   ` zohar
2019-04-08 11:06     ` Mimi Zohar
2019-04-08 12:22     ` pvorel
2019-04-08 12:22       ` Petr Vorel
2019-04-08 11:38   ` chrubis [this message]
2019-04-08 11:38     ` Cyril Hrubis
2019-04-08 13:07     ` pvorel
2019-04-08 13:07       ` Petr Vorel
2019-04-06 21:49 ` [RFC PATCH 2/2] selftest/kexec: Use kselftest " pvorel
2019-04-06 21:49   ` Petr Vorel
2019-04-08 11:29   ` zohar
2019-04-08 11:29     ` Mimi Zohar
2019-04-08 11:43 ` [RFC PATCH 0/2] Kselftest shell (or even C) API chrubis
2019-04-08 11:43   ` Cyril Hrubis
2019-04-08 13:25   ` pvorel
2019-04-08 13:25     ` Petr Vorel
2019-04-08 12:14 ` zohar
2019-04-08 12:14   ` Mimi Zohar
2019-04-08 12:29   ` pvorel
2019-04-08 12:29     ` Petr Vorel

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190408113841.GA2418@rei.lan \
    --to=unknown@example.com \
    --subject='Re: [RFC PATCH 1/2] selftests: Start shell API' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.