All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jakub Kicinski <kuba@kernel.org>
To: davem@davemloft.net
Cc: netdev@vger.kernel.org, edumazet@google.com, pabeni@redhat.com,
	shuah@kernel.org, petrm@nvidia.com,
	linux-kselftest@vger.kernel.org, willemdebruijn.kernel@gmail.com,
	Jakub Kicinski <kuba@kernel.org>
Subject: [PATCH net-next v3 1/8] selftests: net: fix counting totals when some checks fail
Date: Wed, 17 Apr 2024 16:11:39 -0700	[thread overview]
Message-ID: <20240417231146.2435572-2-kuba@kernel.org> (raw)
In-Reply-To: <20240417231146.2435572-1-kuba@kernel.org>

Totals currently only pay attention to exceptions, if check fails
(say ksft_eq()) the test case will be counted as pass:

  # At /ksft/drivers/net/./ping.py line 18:
  # Check failed 1 != 2
  not ok 1 ping.test_v4
  ok 2 ping.test_v6
  ok 3 ping.test_tcp
  # Totals: pass:3 fail:0 xfail:0 xpass:0 skip:0 error:0
            ^^^^^^^^^^^^^

Pay attention to the result.

Fixes: b86761ff6374 ("selftests: net: add scaffolding for Netlink tests in Python")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
 tools/testing/selftests/net/lib/py/ksft.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/net/lib/py/ksft.py b/tools/testing/selftests/net/lib/py/ksft.py
index 3769b9197213..640dfbf47702 100644
--- a/tools/testing/selftests/net/lib/py/ksft.py
+++ b/tools/testing/selftests/net/lib/py/ksft.py
@@ -106,7 +106,10 @@ KSFT_RESULT = None
             continue
 
         ktap_result(KSFT_RESULT, cnt, case)
-        totals['pass'] += 1
+        if KSFT_RESULT:
+            totals['pass'] += 1
+        else:
+            totals['fail'] += 1
 
     print(
         f"# Totals: pass:{totals['pass']} fail:{totals['fail']} xfail:{totals['xfail']} xpass:0 skip:{totals['skip']} error:0"
-- 
2.44.0


  reply	other threads:[~2024-04-17 23:11 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-04-17 23:11 [PATCH net-next v3 0/8] selftests: drv-net: support testing with a remote system Jakub Kicinski
2024-04-17 23:11 ` Jakub Kicinski [this message]
2024-04-17 23:11 ` [PATCH net-next v3 2/8] selftests: net: set the exit code correctly in Python tests Jakub Kicinski
2024-04-17 23:11 ` [PATCH net-next v3 3/8] selftests: drv-net: define endpoint structures Jakub Kicinski
2024-04-17 23:11 ` [PATCH net-next v3 4/8] selftests: drv-net: factor out parsing of the env Jakub Kicinski
2024-04-17 23:11 ` [PATCH net-next v3 5/8] selftests: drv-net: construct environment for running tests which require an endpoint Jakub Kicinski
2024-04-17 23:11 ` [PATCH net-next v3 6/8] selftests: drv-net: add a trivial ping test Jakub Kicinski
2024-04-17 23:11 ` [PATCH net-next v3 7/8] selftests: net: support matching cases by name prefix Jakub Kicinski
2024-04-18 14:26   ` Willem de Bruijn
2024-04-18 19:06     ` Jakub Kicinski
2024-04-18 19:35       ` Willem de Bruijn
2024-04-17 23:11 ` [PATCH net-next v3 8/8] selftests: drv-net: add a TCP ping test case (and useful helpers) Jakub Kicinski
2024-04-18 14:44   ` Willem de Bruijn
2024-04-18 19:48     ` Jakub Kicinski
2024-04-18 23:20 ` [PATCH net-next v3 0/8] selftests: drv-net: support testing with a remote system patchwork-bot+netdevbpf

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240417231146.2435572-2-kuba@kernel.org \
    --to=kuba@kernel.org \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=petrm@nvidia.com \
    --cc=shuah@kernel.org \
    --cc=willemdebruijn.kernel@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.