linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 1/3] kunit: tool: make TestCounts a dataclass
@ 2022-11-02 16:40 Daniel Latypov
  2022-11-02 16:40 ` [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit Daniel Latypov
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Daniel Latypov @ 2022-11-02 16:40 UTC (permalink / raw)
  To: brendanhiggins, davidgow
  Cc: rmoar, linux-kernel, kunit-dev, linux-kselftest, skhan, Daniel Latypov

Since we're using Python 3.7+, we can use dataclasses to tersen the
code.

It also lets us create pre-populated TestCounts() objects and compare
them in our unit test. (Before, you could only create empty ones).

Signed-off-by: Daniel Latypov <dlatypov@google.com>
---
 tools/testing/kunit/kunit_parser.py    | 25 ++++++++-----------------
 tools/testing/kunit/kunit_tool_test.py |  4 +---
 2 files changed, 9 insertions(+), 20 deletions(-)

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 1ae873e3e341..f022966858f2 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -10,6 +10,7 @@
 # Author: Rae Moar <rmoar@google.com>
 
 from __future__ import annotations
+from dataclasses import dataclass
 import re
 import sys
 
@@ -67,27 +68,17 @@ class TestStatus(Enum):
 	NO_TESTS = auto()
 	FAILURE_TO_PARSE_TESTS = auto()
 
+@dataclass
 class TestCounts:
 	"""
 	Tracks the counts of statuses of all test cases and any errors within
 	a Test.
-
-	Attributes:
-	passed : int - the number of tests that have passed
-	failed : int - the number of tests that have failed
-	crashed : int - the number of tests that have crashed
-	skipped : int - the number of tests that have skipped
-	errors : int - the number of errors in the test and subtests
-	"""
-	def __init__(self):
-		"""Creates TestCounts object with counts of all test
-		statuses and test errors set to 0.
-		"""
-		self.passed = 0
-		self.failed = 0
-		self.crashed = 0
-		self.skipped = 0
-		self.errors = 0
+	"""
+	passed: int = 0
+	failed: int = 0
+	crashed: int = 0
+	skipped: int = 0
+	errors: int = 0
 
 	def __str__(self) -> str:
 		"""Returns the string representation of a TestCounts object."""
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index e2cd2cc2e98f..9fa4babb2506 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -179,9 +179,7 @@ class KUnitParserTest(unittest.TestCase):
 				kunit_parser.extract_tap_lines(
 				file.readlines()))
 		# A missing test plan is not an error.
-		self.assertEqual(0, result.counts.errors)
-		# All tests should be accounted for.
-		self.assertEqual(10, result.counts.total())
+		self.assertEqual(result.counts, kunit_parser.TestCounts(passed=10, errors=0))
 		self.assertEqual(
 			kunit_parser.TestStatus.SUCCESS,
 			result.status)

base-commit: 5aaef24b5c6d4246b2cac1be949869fa36577737
-- 
2.38.1.273.g43a17bfeac-goog


^ permalink raw reply related	[flat|nested] 7+ messages in thread
* [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit
  2022-11-02 16:40 [PATCH 1/3] kunit: tool: make TestCounts a dataclass Daniel Latypov
@ 2022-11-02 16:40 ` Daniel Latypov
  2022-11-03  6:38   ` David Gow
  2022-11-02 16:40 ` [PATCH 3/3] kunit: tool: remove redundant file.close() call in unit test Daniel Latypov
  2022-11-03  6:37 ` [PATCH 1/3] kunit: tool: make TestCounts a dataclass David Gow
  2 siblings, 1 reply; 7+ messages in thread
From: Daniel Latypov @ 2022-11-02 16:40 UTC (permalink / raw)
  To: brendanhiggins, davidgow
  Cc: rmoar, linux-kernel, kunit-dev, linux-kselftest, skhan, Daniel Latypov

Let's verify that the parser isn't reporting any errors for valid
inputs.

This change also
* does result.status checking on one line
* makes sure we consistently do it outside of the `with` block

Signed-off-by: Daniel Latypov <dlatypov@google.com>
---
 tools/testing/kunit/kunit_tool_test.py | 93 +++++++++++---------------
 1 file changed, 38 insertions(+), 55 deletions(-)

diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index 9fa4babb2506..0063773c0fc4 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -133,33 +133,29 @@ class KUnitParserTest(unittest.TestCase):
 		all_passed_log = test_data_path('test_is_test_passed-all_passed.log')
 		with open(all_passed_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-		self.assertEqual(
-			kunit_parser.TestStatus.SUCCESS,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual(result.counts.errors, 0)
 
 	def test_parse_successful_nested_tests_log(self):
 		all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log')
 		with open(all_passed_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-		self.assertEqual(
-			kunit_parser.TestStatus.SUCCESS,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual(result.counts.errors, 0)
 
 	def test_kselftest_nested(self):
 		kselftest_log = test_data_path('test_is_test_passed-kselftest.log')
 		with open(kselftest_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-			self.assertEqual(
-				kunit_parser.TestStatus.SUCCESS,
-				result.status)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual(result.counts.errors, 0)
 
 	def test_parse_failed_test_log(self):
 		failed_log = test_data_path('test_is_test_passed-failure.log')
 		with open(failed_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-		self.assertEqual(
-			kunit_parser.TestStatus.FAILURE,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
+		self.assertEqual(result.counts.errors, 0)
 
 	def test_no_header(self):
 		empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log')
@@ -167,9 +163,8 @@ class KUnitParserTest(unittest.TestCase):
 			result = kunit_parser.parse_run_tests(
 				kunit_parser.extract_tap_lines(file.readlines()))
 		self.assertEqual(0, len(result.subtests))
-		self.assertEqual(
-			kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status)
+		self.assertEqual(result.counts.errors, 1)
 
 	def test_missing_test_plan(self):
 		missing_plan_log = test_data_path('test_is_test_passed-'
@@ -180,9 +175,7 @@ class KUnitParserTest(unittest.TestCase):
 				file.readlines()))
 		# A missing test plan is not an error.
 		self.assertEqual(result.counts, kunit_parser.TestCounts(passed=10, errors=0))
-		self.assertEqual(
-			kunit_parser.TestStatus.SUCCESS,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
 
 	def test_no_tests(self):
 		header_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log')
@@ -190,9 +183,8 @@ class KUnitParserTest(unittest.TestCase):
 			result = kunit_parser.parse_run_tests(
 				kunit_parser.extract_tap_lines(file.readlines()))
 		self.assertEqual(0, len(result.subtests))
-		self.assertEqual(
-			kunit_parser.TestStatus.NO_TESTS,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.NO_TESTS, result.status)
+		self.assertEqual(result.counts.errors, 1)
 
 	def test_no_tests_no_plan(self):
 		no_plan_log = test_data_path('test_is_test_passed-no_tests_no_plan.log')
@@ -203,7 +195,7 @@ class KUnitParserTest(unittest.TestCase):
 		self.assertEqual(
 			kunit_parser.TestStatus.NO_TESTS,
 			result.subtests[0].subtests[0].status)
-		self.assertEqual(1, result.counts.errors)
+		self.assertEqual(result.counts, kunit_parser.TestCounts(passed=1, errors=1))
 
 
 	def test_no_kunit_output(self):
@@ -215,6 +207,7 @@ class KUnitParserTest(unittest.TestCase):
 		print_mock.assert_any_call(StrContains('could not find any KTAP output!'))
 		print_mock.stop()
 		self.assertEqual(0, len(result.subtests))
+		self.assertEqual(result.counts.errors, 1)
 
 	def test_skipped_test(self):
 		skipped_log = test_data_path('test_skip_tests.log')
@@ -222,18 +215,16 @@ class KUnitParserTest(unittest.TestCase):
 			result = kunit_parser.parse_run_tests(file.readlines())
 
 		# A skipped test does not fail the whole suite.
-		self.assertEqual(
-			kunit_parser.TestStatus.SUCCESS,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual(result.counts, kunit_parser.TestCounts(passed=4, skipped=1))
 
 	def test_skipped_all_tests(self):
 		skipped_log = test_data_path('test_skip_all_tests.log')
 		with open(skipped_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
 
-		self.assertEqual(
-			kunit_parser.TestStatus.SKIPPED,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.SKIPPED, result.status)
+		self.assertEqual(result.counts, kunit_parser.TestCounts(skipped=5))
 
 	def test_ignores_hyphen(self):
 		hyphen_log = test_data_path('test_strip_hyphen.log')
@@ -241,9 +232,7 @@ class KUnitParserTest(unittest.TestCase):
 			result = kunit_parser.parse_run_tests(file.readlines())
 
 		# A skipped test does not fail the whole suite.
-		self.assertEqual(
-			kunit_parser.TestStatus.SUCCESS,
-			result.status)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
 		self.assertEqual(
 			"sysctl_test",
 			result.subtests[0].name)
@@ -257,55 +246,49 @@ class KUnitParserTest(unittest.TestCase):
 		prefix_log = test_data_path('test_config_printk_time.log')
 		with open(prefix_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-			self.assertEqual(
-				kunit_parser.TestStatus.SUCCESS,
-				result.status)
-			self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(result.counts.errors, 0)
 
 	def test_ignores_multiple_prefixes(self):
 		prefix_log = test_data_path('test_multiple_prefixes.log')
 		with open(prefix_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-			self.assertEqual(
-				kunit_parser.TestStatus.SUCCESS,
-				result.status)
-			self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(result.counts.errors, 0)
 
 	def test_prefix_mixed_kernel_output(self):
 		mixed_prefix_log = test_data_path('test_interrupted_tap_output.log')
 		with open(mixed_prefix_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-			self.assertEqual(
-				kunit_parser.TestStatus.SUCCESS,
-				result.status)
-			self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(result.counts.errors, 0)
 
 	def test_prefix_poundsign(self):
 		pound_log = test_data_path('test_pound_sign.log')
 		with open(pound_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-			self.assertEqual(
-				kunit_parser.TestStatus.SUCCESS,
-				result.status)
-			self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(result.counts.errors, 0)
 
 	def test_kernel_panic_end(self):
 		panic_log = test_data_path('test_kernel_panic_interrupt.log')
 		with open(panic_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-			self.assertEqual(
-				kunit_parser.TestStatus.TEST_CRASHED,
-				result.status)
-			self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(kunit_parser.TestStatus.TEST_CRASHED, result.status)
+		self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertGreaterEqual(result.counts.errors, 1)
 
 	def test_pound_no_prefix(self):
 		pound_log = test_data_path('test_pound_no_prefix.log')
 		with open(pound_log) as file:
 			result = kunit_parser.parse_run_tests(file.readlines())
-			self.assertEqual(
-				kunit_parser.TestStatus.SUCCESS,
-				result.status)
-			self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
+		self.assertEqual('kunit-resource-test', result.subtests[0].name)
+		self.assertEqual(result.counts.errors, 0)
 
 def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream:
 	return kunit_parser.LineStream(enumerate(strs, start=1))
-- 
2.38.1.273.g43a17bfeac-goog


^ permalink raw reply related	[flat|nested] 7+ messages in thread
* [PATCH 3/3] kunit: tool: remove redundant file.close() call in unit test
  2022-11-02 16:40 [PATCH 1/3] kunit: tool: make TestCounts a dataclass Daniel Latypov
  2022-11-02 16:40 ` [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit Daniel Latypov
@ 2022-11-02 16:40 ` Daniel Latypov
  2022-11-03  6:41   ` David Gow
  2022-11-03  6:37 ` [PATCH 1/3] kunit: tool: make TestCounts a dataclass David Gow
  2 siblings, 1 reply; 7+ messages in thread
From: Daniel Latypov @ 2022-11-02 16:40 UTC (permalink / raw)
  To: brendanhiggins, davidgow
  Cc: rmoar, linux-kernel, kunit-dev, linux-kselftest, skhan, Daniel Latypov

We're using a `with` block above, so the file object is already closed.

Signed-off-by: Daniel Latypov <dlatypov@google.com>
---
 tools/testing/kunit/kunit_tool_test.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index 0063773c0fc4..c41567eaf3c3 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -239,8 +239,6 @@ class KUnitParserTest(unittest.TestCase):
 		self.assertEqual(
 			"example",
 			result.subtests[1].name)
-		file.close()
-
 
 	def test_ignores_prefix_printk_time(self):
 		prefix_log = test_data_path('test_config_printk_time.log')
-- 
2.38.1.273.g43a17bfeac-goog


^ permalink raw reply related	[flat|nested] 7+ messages in thread
* Re: [PATCH 1/3] kunit: tool: make TestCounts a dataclass
  2022-11-02 16:40 [PATCH 1/3] kunit: tool: make TestCounts a dataclass Daniel Latypov
  2022-11-02 16:40 ` [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit Daniel Latypov
  2022-11-02 16:40 ` [PATCH 3/3] kunit: tool: remove redundant file.close() call in unit test Daniel Latypov
@ 2022-11-03  6:37 ` David Gow
  2 siblings, 0 replies; 7+ messages in thread
From: David Gow @ 2022-11-03  6:37 UTC (permalink / raw)
  To: Daniel Latypov
  Cc: brendanhiggins, rmoar, linux-kernel, kunit-dev, linux-kselftest, skhan

[-- Attachment #1: Type: text/plain, Size: 3230 bytes --]

On Thu, Nov 3, 2022 at 12:40 AM Daniel Latypov <dlatypov@google.com> wrote:
>
> Since we're using Python 3.7+, we can use dataclasses to tersen the
> code.
>
> It also lets us create pre-populated TestCounts() objects and compare
> them in our unit test. (Before, you could only create empty ones).
>
> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> ---

Looks good, thanks.

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David

>  tools/testing/kunit/kunit_parser.py    | 25 ++++++++-----------------
>  tools/testing/kunit/kunit_tool_test.py |  4 +---
>  2 files changed, 9 insertions(+), 20 deletions(-)
>
> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> index 1ae873e3e341..f022966858f2 100644
> --- a/tools/testing/kunit/kunit_parser.py
> +++ b/tools/testing/kunit/kunit_parser.py
> @@ -10,6 +10,7 @@
>  # Author: Rae Moar <rmoar@google.com>
>
>  from __future__ import annotations
> +from dataclasses import dataclass
>  import re
>  import sys
>
> @@ -67,27 +68,17 @@ class TestStatus(Enum):
>         NO_TESTS = auto()
>         FAILURE_TO_PARSE_TESTS = auto()
>
> +@dataclass
>  class TestCounts:
>         """
>         Tracks the counts of statuses of all test cases and any errors within
>         a Test.
> -
> -       Attributes:
> -       passed : int - the number of tests that have passed
> -       failed : int - the number of tests that have failed
> -       crashed : int - the number of tests that have crashed
> -       skipped : int - the number of tests that have skipped
> -       errors : int - the number of errors in the test and subtests
> -       """
> -       def __init__(self):
> -               """Creates TestCounts object with counts of all test
> -               statuses and test errors set to 0.
> -               """
> -               self.passed = 0
> -               self.failed = 0
> -               self.crashed = 0
> -               self.skipped = 0
> -               self.errors = 0
> +       """
> +       passed: int = 0
> +       failed: int = 0
> +       crashed: int = 0
> +       skipped: int = 0
> +       errors: int = 0
>
>         def __str__(self) -> str:
>                 """Returns the string representation of a TestCounts object."""
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index e2cd2cc2e98f..9fa4babb2506 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -179,9 +179,7 @@ class KUnitParserTest(unittest.TestCase):
>                                 kunit_parser.extract_tap_lines(
>                                 file.readlines()))
>                 # A missing test plan is not an error.
> -               self.assertEqual(0, result.counts.errors)
> -               # All tests should be accounted for.
> -               self.assertEqual(10, result.counts.total())
> +               self.assertEqual(result.counts, kunit_parser.TestCounts(passed=10, errors=0))
>                 self.assertEqual(
>                         kunit_parser.TestStatus.SUCCESS,
>                         result.status)
>
> base-commit: 5aaef24b5c6d4246b2cac1be949869fa36577737
> --
> 2.38.1.273.g43a17bfeac-goog
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit
  2022-11-02 16:40 ` [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit Daniel Latypov
@ 2022-11-03  6:38   ` David Gow
  2022-11-03 17:45     ` Daniel Latypov
  0 siblings, 1 reply; 7+ messages in thread
From: David Gow @ 2022-11-03  6:38 UTC (permalink / raw)
  To: Daniel Latypov
  Cc: brendanhiggins, rmoar, linux-kernel, kunit-dev, linux-kselftest, skhan

[-- Attachment #1: Type: text/plain, Size: 13119 bytes --]

On Thu, Nov 3, 2022 at 12:40 AM 'Daniel Latypov' via KUnit Development
<kunit-dev@googlegroups.com> wrote:
>
> Let's verify that the parser isn't reporting any errors for valid
> inputs.
>
> This change also
> * does result.status checking on one line
> * makes sure we consistently do it outside of the `with` block
>
> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> ---

Looks good, thanks.

Note that this patch does conflict with "kunit: tool: print summary of
failed tests if a few failed out of a lot":
https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=kunit&id=f19dd011d8de6f0c1d20abea5158aa4f5d9cea44
It's only a context line issue, though.

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David


>  tools/testing/kunit/kunit_tool_test.py | 93 +++++++++++---------------
>  1 file changed, 38 insertions(+), 55 deletions(-)
>
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index 9fa4babb2506..0063773c0fc4 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -133,33 +133,29 @@ class KUnitParserTest(unittest.TestCase):
>                 all_passed_log = test_data_path('test_is_test_passed-all_passed.log')
>                 with open(all_passed_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.SUCCESS,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual(result.counts.errors, 0)
>
>         def test_parse_successful_nested_tests_log(self):
>                 all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log')
>                 with open(all_passed_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.SUCCESS,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual(result.counts.errors, 0)
>
>         def test_kselftest_nested(self):
>                 kselftest_log = test_data_path('test_is_test_passed-kselftest.log')
>                 with open(kselftest_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -                       self.assertEqual(
> -                               kunit_parser.TestStatus.SUCCESS,
> -                               result.status)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual(result.counts.errors, 0)
>
>         def test_parse_failed_test_log(self):
>                 failed_log = test_data_path('test_is_test_passed-failure.log')
>                 with open(failed_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.FAILURE,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
> +               self.assertEqual(result.counts.errors, 0)
>
>         def test_no_header(self):
>                 empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log')
> @@ -167,9 +163,8 @@ class KUnitParserTest(unittest.TestCase):
>                         result = kunit_parser.parse_run_tests(
>                                 kunit_parser.extract_tap_lines(file.readlines()))
>                 self.assertEqual(0, len(result.subtests))
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status)
> +               self.assertEqual(result.counts.errors, 1)
>
>         def test_missing_test_plan(self):
>                 missing_plan_log = test_data_path('test_is_test_passed-'
> @@ -180,9 +175,7 @@ class KUnitParserTest(unittest.TestCase):
>                                 file.readlines()))
>                 # A missing test plan is not an error.
>                 self.assertEqual(result.counts, kunit_parser.TestCounts(passed=10, errors=0))
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.SUCCESS,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
>
>         def test_no_tests(self):
>                 header_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log')
> @@ -190,9 +183,8 @@ class KUnitParserTest(unittest.TestCase):
>                         result = kunit_parser.parse_run_tests(
>                                 kunit_parser.extract_tap_lines(file.readlines()))
>                 self.assertEqual(0, len(result.subtests))
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.NO_TESTS,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.NO_TESTS, result.status)
> +               self.assertEqual(result.counts.errors, 1)
>
>         def test_no_tests_no_plan(self):
>                 no_plan_log = test_data_path('test_is_test_passed-no_tests_no_plan.log')
> @@ -203,7 +195,7 @@ class KUnitParserTest(unittest.TestCase):
>                 self.assertEqual(
>                         kunit_parser.TestStatus.NO_TESTS,
>                         result.subtests[0].subtests[0].status)
> -               self.assertEqual(1, result.counts.errors)
> +               self.assertEqual(result.counts, kunit_parser.TestCounts(passed=1, errors=1))
>
>
>         def test_no_kunit_output(self):
> @@ -215,6 +207,7 @@ class KUnitParserTest(unittest.TestCase):
>                 print_mock.assert_any_call(StrContains('could not find any KTAP output!'))
>                 print_mock.stop()
>                 self.assertEqual(0, len(result.subtests))
> +               self.assertEqual(result.counts.errors, 1)
>
>         def test_skipped_test(self):
>                 skipped_log = test_data_path('test_skip_tests.log')
> @@ -222,18 +215,16 @@ class KUnitParserTest(unittest.TestCase):
>                         result = kunit_parser.parse_run_tests(file.readlines())
>
>                 # A skipped test does not fail the whole suite.
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.SUCCESS,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual(result.counts, kunit_parser.TestCounts(passed=4, skipped=1))
>
>         def test_skipped_all_tests(self):
>                 skipped_log = test_data_path('test_skip_all_tests.log')
>                 with open(skipped_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
>
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.SKIPPED,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.SKIPPED, result.status)
> +               self.assertEqual(result.counts, kunit_parser.TestCounts(skipped=5))
>
>         def test_ignores_hyphen(self):
>                 hyphen_log = test_data_path('test_strip_hyphen.log')
> @@ -241,9 +232,7 @@ class KUnitParserTest(unittest.TestCase):
>                         result = kunit_parser.parse_run_tests(file.readlines())
>
>                 # A skipped test does not fail the whole suite.
> -               self.assertEqual(
> -                       kunit_parser.TestStatus.SUCCESS,
> -                       result.status)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
>                 self.assertEqual(
>                         "sysctl_test",
>                         result.subtests[0].name)
> @@ -257,55 +246,49 @@ class KUnitParserTest(unittest.TestCase):
>                 prefix_log = test_data_path('test_config_printk_time.log')
>                 with open(prefix_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -                       self.assertEqual(
> -                               kunit_parser.TestStatus.SUCCESS,
> -                               result.status)
> -                       self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(result.counts.errors, 0)
>
>         def test_ignores_multiple_prefixes(self):
>                 prefix_log = test_data_path('test_multiple_prefixes.log')
>                 with open(prefix_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -                       self.assertEqual(
> -                               kunit_parser.TestStatus.SUCCESS,
> -                               result.status)
> -                       self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(result.counts.errors, 0)
>
>         def test_prefix_mixed_kernel_output(self):
>                 mixed_prefix_log = test_data_path('test_interrupted_tap_output.log')
>                 with open(mixed_prefix_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -                       self.assertEqual(
> -                               kunit_parser.TestStatus.SUCCESS,
> -                               result.status)
> -                       self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(result.counts.errors, 0)
>
>         def test_prefix_poundsign(self):
>                 pound_log = test_data_path('test_pound_sign.log')
>                 with open(pound_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -                       self.assertEqual(
> -                               kunit_parser.TestStatus.SUCCESS,
> -                               result.status)
> -                       self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(result.counts.errors, 0)
>
>         def test_kernel_panic_end(self):
>                 panic_log = test_data_path('test_kernel_panic_interrupt.log')
>                 with open(panic_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -                       self.assertEqual(
> -                               kunit_parser.TestStatus.TEST_CRASHED,
> -                               result.status)
> -                       self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(kunit_parser.TestStatus.TEST_CRASHED, result.status)
> +               self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertGreaterEqual(result.counts.errors, 1)
>
>         def test_pound_no_prefix(self):
>                 pound_log = test_data_path('test_pound_no_prefix.log')
>                 with open(pound_log) as file:
>                         result = kunit_parser.parse_run_tests(file.readlines())
> -                       self.assertEqual(
> -                               kunit_parser.TestStatus.SUCCESS,
> -                               result.status)
> -                       self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status)
> +               self.assertEqual('kunit-resource-test', result.subtests[0].name)
> +               self.assertEqual(result.counts.errors, 0)
>
>  def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream:
>         return kunit_parser.LineStream(enumerate(strs, start=1))
> --
> 2.38.1.273.g43a17bfeac-goog
>
> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/20221102164005.2516646-2-dlatypov%40google.com.

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: [PATCH 3/3] kunit: tool: remove redundant file.close() call in unit test
  2022-11-02 16:40 ` [PATCH 3/3] kunit: tool: remove redundant file.close() call in unit test Daniel Latypov
@ 2022-11-03  6:41   ` David Gow
  0 siblings, 0 replies; 7+ messages in thread
From: David Gow @ 2022-11-03  6:41 UTC (permalink / raw)
  To: Daniel Latypov
  Cc: brendanhiggins, rmoar, linux-kernel, kunit-dev, linux-kselftest, skhan

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

On Thu, Nov 3, 2022 at 12:40 AM Daniel Latypov <dlatypov@google.com> wrote:
>
> We're using a `with` block above, so the file object is already closed.
>
> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> ---

Nice catch.

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David

>  tools/testing/kunit/kunit_tool_test.py | 2 --
>  1 file changed, 2 deletions(-)
>
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index 0063773c0fc4..c41567eaf3c3 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -239,8 +239,6 @@ class KUnitParserTest(unittest.TestCase):
>                 self.assertEqual(
>                         "example",
>                         result.subtests[1].name)
> -               file.close()
> -
>
>         def test_ignores_prefix_printk_time(self):
>                 prefix_log = test_data_path('test_config_printk_time.log')
> --
> 2.38.1.273.g43a17bfeac-goog
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit
  2022-11-03  6:38   ` David Gow
@ 2022-11-03 17:45     ` Daniel Latypov
  0 siblings, 0 replies; 7+ messages in thread
From: Daniel Latypov @ 2022-11-03 17:45 UTC (permalink / raw)
  To: David Gow
  Cc: brendanhiggins, rmoar, linux-kernel, kunit-dev, linux-kselftest, skhan

On Wed, Nov 2, 2022 at 11:38 PM David Gow <davidgow@google.com> wrote:
>
> On Thu, Nov 3, 2022 at 12:40 AM 'Daniel Latypov' via KUnit Development
> <kunit-dev@googlegroups.com> wrote:
> >
> > Let's verify that the parser isn't reporting any errors for valid
> > inputs.
> >
> > This change also
> > * does result.status checking on one line
> > * makes sure we consistently do it outside of the `with` block
> >
> > Signed-off-by: Daniel Latypov <dlatypov@google.com>
> > ---
>
> Looks good, thanks.
>
> Note that this patch does conflict with "kunit: tool: print summary of
> failed tests if a few failed out of a lot":
> https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=kunit&id=f19dd011d8de6f0c1d20abea5158aa4f5d9cea44
> It's only a context line issue, though.
>
> Reviewed-by: David Gow <davidgow@google.com>

Oh huh, I rebased onto the kunit branch and it managed to merge cleanly.
I guess `git am` is more picky than `git rebase`.

I'll send a v2 that's rebased to avoid issues applying them.

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-11-03 17:47 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-02 16:40 [PATCH 1/3] kunit: tool: make TestCounts a dataclass Daniel Latypov
2022-11-02 16:40 ` [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit Daniel Latypov
2022-11-03  6:38   ` David Gow
2022-11-03 17:45     ` Daniel Latypov
2022-11-02 16:40 ` [PATCH 3/3] kunit: tool: remove redundant file.close() call in unit test Daniel Latypov
2022-11-03  6:41   ` David Gow
2022-11-03  6:37 ` [PATCH 1/3] kunit: tool: make TestCounts a dataclass David Gow

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).