* [PATCH v4 0/2] Documentation: dev-tools: clarify KTAP specification wording
@ 2022-02-10 2:35 frowand.list
2022-02-10 2:35 ` [PATCH v4 1/2] " frowand.list
2022-02-10 2:35 ` [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings frowand.list
0 siblings, 2 replies; 7+ messages in thread
From: frowand.list @ 2022-02-10 2:35 UTC (permalink / raw)
To: Jonathan Corbet, David Gow, Shuah Khan, Kees Cook, Rae Moar,
Tim.Bird, Brendan Higgins
Cc: rmr167, guillaume.tucker, dlatypov, kernelci, kunit-dev,
linux-kselftest, linux-doc, linux-kernel
From: Frank Rowand <frank.rowand@sony.com>
Clarify ambiguous wording in KTAP specification version 1.
Fix build warnings.
Changes since version 3
- Add Reviewed-by: Shuah Khan
- Add patch 2/2 to fix build warnings
Frank Rowand (2):
Documentation: dev-tools: clarify KTAP specification wording
Documentation: dev-tools: fix KTAP specification build warnings
Documentation/dev-tools/ktap.rst | 49 +++++++++++++++++++++-----------
1 file changed, 32 insertions(+), 17 deletions(-)
--
Frank Rowand <frank.rowand@sony.com>
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v4 1/2] Documentation: dev-tools: clarify KTAP specification wording
2022-02-10 2:35 [PATCH v4 0/2] Documentation: dev-tools: clarify KTAP specification wording frowand.list
@ 2022-02-10 2:35 ` frowand.list
2022-02-10 17:43 ` Kees Cook
2022-02-10 2:35 ` [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings frowand.list
1 sibling, 1 reply; 7+ messages in thread
From: frowand.list @ 2022-02-10 2:35 UTC (permalink / raw)
To: Jonathan Corbet, David Gow, Shuah Khan, Kees Cook, Rae Moar,
Tim.Bird, Brendan Higgins
Cc: rmr167, guillaume.tucker, dlatypov, kernelci, kunit-dev,
linux-kselftest, linux-doc, linux-kernel
From: Frank Rowand <frank.rowand@sony.com>
Add the spec version to the title line.
Explain likely source of "Unknown lines".
"Unknown lines" in nested tests are optionally indented.
Add "Unknown lines" items to differences between TAP & KTAP list
Reviewed-by: Tim Bird <Tim.Bird@sony.com>
Reviewed-by: David Gow <davidgow@google.com>
Reviewed-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Frank Rowand <frank.rowand@sony.com>
---
Changes since version 3
- Add reviewed by Shuah
Changes since version 2
- Add missing quote after word: incorrect
- Add Reviewed-by tags
Changes since version 1
- Explain likely source of "Unknown lines"
- "Unknown line" in nested tests are optionally indented
- Add "Unknown lines" items to differences between TAP & KTAP list
Documentation/dev-tools/ktap.rst | 24 ++++++++++++++++++------
1 file changed, 18 insertions(+), 6 deletions(-)
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
index 878530cb9c27..dfb3f10a8b2d 100644
--- a/Documentation/dev-tools/ktap.rst
+++ b/Documentation/dev-tools/ktap.rst
@@ -1,8 +1,8 @@
.. SPDX-License-Identifier: GPL-2.0
-========================================
-The Kernel Test Anything Protocol (KTAP)
-========================================
+===================================================
+The Kernel Test Anything Protocol (KTAP), version 1
+===================================================
TAP, or the Test Anything Protocol is a format for specifying test results used
by a number of projects. It's website and specification are found at this `link
@@ -174,6 +174,13 @@ There may be lines within KTAP output that do not follow the format of one of
the four formats for lines described above. This is allowed, however, they will
not influence the status of the tests.
+This is an important difference from TAP. Kernel tests may print messages
+to the system console or a log file. Both of these destinations may contain
+messages either from unrelated kernel or userspace activity, or kernel
+messages from non-test code that is invoked by the test. The kernel code
+invoked by the test likely is not aware that a test is in progress and
+thus can not print the message as a diagnostic message.
+
Nested tests
------------
@@ -186,10 +193,13 @@ starting with another KTAP version line and test plan, and end with the overall
result. If one of the subtests fail, for example, the parent test should also
fail.
-Additionally, all result lines in a subtest should be indented. One level of
+Additionally, all lines in a subtest should be indented. One level of
indentation is two spaces: " ". The indentation should begin at the version
line and should end before the parent test's result line.
+"Unknown lines" are not considered to be lines in a subtest and thus are
+allowed to be either indented or not indented.
+
An example of a test with two nested subtests:
.. code-block::
@@ -225,9 +235,11 @@ Major differences between TAP and KTAP
--------------------------------------
Note the major differences between the TAP and KTAP specification:
-- yaml and json are not recommended in diagnostic messages
-- TODO directive not recognized
+- yaml and json are not recommended in KTAP diagnostic messages
+- TODO directive not recognized in KTAP
- KTAP allows for an arbitrary number of tests to be nested
+- TAP includes "Unknown lines" in the category of "Anything else"
+- TAP says "Unknown lines" are "incorrect"; KTAP allows "Unknown lines"
The TAP14 specification does permit nested tests, but instead of using another
nested version line, uses a line of the form
--
Frank Rowand <frank.rowand@sony.com>
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings
2022-02-10 2:35 [PATCH v4 0/2] Documentation: dev-tools: clarify KTAP specification wording frowand.list
2022-02-10 2:35 ` [PATCH v4 1/2] " frowand.list
@ 2022-02-10 2:35 ` frowand.list
2022-02-10 7:44 ` David Gow
` (2 more replies)
1 sibling, 3 replies; 7+ messages in thread
From: frowand.list @ 2022-02-10 2:35 UTC (permalink / raw)
To: Jonathan Corbet, David Gow, Shuah Khan, Kees Cook, Rae Moar,
Tim.Bird, Brendan Higgins
Cc: rmr167, guillaume.tucker, dlatypov, kernelci, kunit-dev,
linux-kselftest, linux-doc, linux-kernel
From: Frank Rowand <frank.rowand@sony.com>
Convert "Major differences between TAP and KTAP" from a bullet list
to a table. The bullet list was being formatted as a single
paragraph.
Add missing required argument in code-block directives.
---
Table conversion suggested by Shuah.
Patch 2/2 not previously reviewed, so Reviewed-by tags not provided.
Changes since version 3
- Add this commit (patch 2/2) to the series
Signed-off-by: Frank Rowand <frank.rowand@sony.com>
---
Documentation/dev-tools/ktap.rst | 33 +++++++++++++++++---------------
1 file changed, 18 insertions(+), 15 deletions(-)
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
index dfb3f10a8b2d..5ee735c6687f 100644
--- a/Documentation/dev-tools/ktap.rst
+++ b/Documentation/dev-tools/ktap.rst
@@ -68,7 +68,7 @@ Test case result lines
Test case result lines indicate the final status of a test.
They are required and must have the format:
-.. code-block::
+.. code-block:: none
<result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
@@ -117,32 +117,32 @@ separator.
Example result lines include:
-.. code-block::
+.. code-block:: none
ok 1 test_case_name
The test "test_case_name" passed.
-.. code-block::
+.. code-block:: none
not ok 1 test_case_name
The test "test_case_name" failed.
-.. code-block::
+.. code-block:: none
ok 1 test # SKIP necessary dependency unavailable
The test "test" was SKIPPED with the diagnostic message "necessary dependency
unavailable".
-.. code-block::
+.. code-block:: none
not ok 1 test # TIMEOUT 30 seconds
The test "test" timed out, with diagnostic data "30 seconds".
-.. code-block::
+.. code-block:: none
ok 5 check return code # rcode=0
@@ -202,7 +202,7 @@ allowed to be either indented or not indented.
An example of a test with two nested subtests:
-.. code-block::
+.. code-block:: none
KTAP version 1
1..1
@@ -215,7 +215,7 @@ An example of a test with two nested subtests:
An example format with multiple levels of nested testing:
-.. code-block::
+.. code-block:: none
KTAP version 1
1..2
@@ -234,12 +234,15 @@ An example format with multiple levels of nested testing:
Major differences between TAP and KTAP
--------------------------------------
-Note the major differences between the TAP and KTAP specification:
-- yaml and json are not recommended in KTAP diagnostic messages
-- TODO directive not recognized in KTAP
-- KTAP allows for an arbitrary number of tests to be nested
-- TAP includes "Unknown lines" in the category of "Anything else"
-- TAP says "Unknown lines" are "incorrect"; KTAP allows "Unknown lines"
+================================================== ========= ===============
+Feature TAP KTAP
+================================================== ========= ===============
+yaml and json in diagnosic message ok not recommended
+TODO directive ok not recognized
+allows an arbitrary number of tests to be nested no yes
+"Unknown lines" are in category of "Anything else" yes no
+"Unknown lines" are incorrect allowed
+================================================== ========= ===============
The TAP14 specification does permit nested tests, but instead of using another
nested version line, uses a line of the form
@@ -247,7 +250,7 @@ nested version line, uses a line of the form
Example KTAP output
--------------------
-.. code-block::
+.. code-block:: none
KTAP version 1
1..1
--
Frank Rowand <frank.rowand@sony.com>
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings
2022-02-10 2:35 ` [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings frowand.list
@ 2022-02-10 7:44 ` David Gow
2022-02-10 17:25 ` Shuah Khan
2022-02-10 17:43 ` Kees Cook
2 siblings, 0 replies; 7+ messages in thread
From: David Gow @ 2022-02-10 7:44 UTC (permalink / raw)
To: Frank Rowand
Cc: Jonathan Corbet, Shuah Khan, Kees Cook, Rae Moar, Bird, Tim,
Brendan Higgins, Rae Moar, Guillaume Tucker, Daniel Latypov,
kernelci, KUnit Development, open list:KERNEL SELFTEST FRAMEWORK,
open list:DOCUMENTATION, Linux Kernel Mailing List
[-- Attachment #1: Type: text/plain, Size: 4387 bytes --]
On Thu, Feb 10, 2022 at 10:35 AM <frowand.list@gmail.com> wrote:
>
> From: Frank Rowand <frank.rowand@sony.com>
>
> Convert "Major differences between TAP and KTAP" from a bullet list
> to a table. The bullet list was being formatted as a single
> paragraph.
>
> Add missing required argument in code-block directives.
>
> ---
>
> Table conversion suggested by Shuah.
>
> Patch 2/2 not previously reviewed, so Reviewed-by tags not provided.
>
> Changes since version 3
> - Add this commit (patch 2/2) to the series
>
> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
> ---
The table looks good to me, the "none" argument in code-block
directives is already fixed in another patch.
The table bits are:
Reviewed-by: David Gow <davidgow@google.com>
Cheers,
-- David
> Documentation/dev-tools/ktap.rst | 33 +++++++++++++++++---------------
> 1 file changed, 18 insertions(+), 15 deletions(-)
>
> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> index dfb3f10a8b2d..5ee735c6687f 100644
> --- a/Documentation/dev-tools/ktap.rst
> +++ b/Documentation/dev-tools/ktap.rst
> @@ -68,7 +68,7 @@ Test case result lines
> Test case result lines indicate the final status of a test.
> They are required and must have the format:
>
> -.. code-block::
> +.. code-block:: none
These code-block changes were already submitted in:
https://lore.kernel.org/lkml/20220131003637.14274-1-rdunlap@infradead.org/T/
>
> <result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
>
> @@ -117,32 +117,32 @@ separator.
>
> Example result lines include:
>
> -.. code-block::
> +.. code-block:: none
>
> ok 1 test_case_name
>
> The test "test_case_name" passed.
>
> -.. code-block::
> +.. code-block:: none
>
> not ok 1 test_case_name
>
> The test "test_case_name" failed.
>
> -.. code-block::
> +.. code-block:: none
>
> ok 1 test # SKIP necessary dependency unavailable
>
> The test "test" was SKIPPED with the diagnostic message "necessary dependency
> unavailable".
>
> -.. code-block::
> +.. code-block:: none
>
> not ok 1 test # TIMEOUT 30 seconds
>
> The test "test" timed out, with diagnostic data "30 seconds".
>
> -.. code-block::
> +.. code-block:: none
>
> ok 5 check return code # rcode=0
>
> @@ -202,7 +202,7 @@ allowed to be either indented or not indented.
>
> An example of a test with two nested subtests:
>
> -.. code-block::
> +.. code-block:: none
>
> KTAP version 1
> 1..1
> @@ -215,7 +215,7 @@ An example of a test with two nested subtests:
>
> An example format with multiple levels of nested testing:
>
> -.. code-block::
> +.. code-block:: none
>
> KTAP version 1
> 1..2
> @@ -234,12 +234,15 @@ An example format with multiple levels of nested testing:
> Major differences between TAP and KTAP
> --------------------------------------
>
> -Note the major differences between the TAP and KTAP specification:
> -- yaml and json are not recommended in KTAP diagnostic messages
> -- TODO directive not recognized in KTAP
> -- KTAP allows for an arbitrary number of tests to be nested
> -- TAP includes "Unknown lines" in the category of "Anything else"
> -- TAP says "Unknown lines" are "incorrect"; KTAP allows "Unknown lines"
> +================================================== ========= ===============
> +Feature TAP KTAP
> +================================================== ========= ===============
> +yaml and json in diagnosic message ok not recommended
> +TODO directive ok not recognized
> +allows an arbitrary number of tests to be nested no yes
> +"Unknown lines" are in category of "Anything else" yes no
> +"Unknown lines" are incorrect allowed
> +================================================== ========= ===============
This looks good to me, thanks!
>
> The TAP14 specification does permit nested tests, but instead of using another
> nested version line, uses a line of the form
> @@ -247,7 +250,7 @@ nested version line, uses a line of the form
>
> Example KTAP output
> --------------------
> -.. code-block::
> +.. code-block:: none
>
> KTAP version 1
> 1..1
> --
> Frank Rowand <frank.rowand@sony.com>
>
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings
2022-02-10 2:35 ` [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings frowand.list
2022-02-10 7:44 ` David Gow
@ 2022-02-10 17:25 ` Shuah Khan
2022-02-10 17:43 ` Kees Cook
2 siblings, 0 replies; 7+ messages in thread
From: Shuah Khan @ 2022-02-10 17:25 UTC (permalink / raw)
To: frowand.list, Jonathan Corbet, David Gow, Kees Cook, Rae Moar,
Tim.Bird, Brendan Higgins
Cc: rmr167, guillaume.tucker, dlatypov, kernelci, kunit-dev,
linux-kselftest, linux-doc, linux-kernel, Shuah Khan
On 2/9/22 7:35 PM, frowand.list@gmail.com wrote:
> From: Frank Rowand <frank.rowand@sony.com>
>
> Convert "Major differences between TAP and KTAP" from a bullet list
> to a table. The bullet list was being formatted as a single
> paragraph.
>
> Add missing required argument in code-block directives.
>
> ---
>
> Table conversion suggested by Shuah.
>
> Patch 2/2 not previously reviewed, so Reviewed-by tags not provided.
>
> Changes since version 3
> - Add this commit (patch 2/2) to the series
>
> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
> ---
> Documentation/dev-tools/ktap.rst | 33 +++++++++++++++++---------------
> 1 file changed, 18 insertions(+), 15 deletions(-)
>
> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> index dfb3f10a8b2d..5ee735c6687f 100644
> --- a/Documentation/dev-tools/ktap.rst
> +++ b/Documentation/dev-tools/ktap.rst
> @@ -68,7 +68,7 @@ Test case result lines
> Test case result lines indicate the final status of a test.
> They are required and must have the format:
>
> -.. code-block::
> +.. code-block:: none
>
> <result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
>
> @@ -117,32 +117,32 @@ separator.
>
> Example result lines include:
>
> -.. code-block::
> +.. code-block:: none
>
> ok 1 test_case_name
>
> The test "test_case_name" passed.
>
> -.. code-block::
> +.. code-block:: none
>
> not ok 1 test_case_name
>
> The test "test_case_name" failed.
>
> -.. code-block::
> +.. code-block:: none
>
> ok 1 test # SKIP necessary dependency unavailable
>
> The test "test" was SKIPPED with the diagnostic message "necessary dependency
> unavailable".
>
> -.. code-block::
> +.. code-block:: none
>
> not ok 1 test # TIMEOUT 30 seconds
>
> The test "test" timed out, with diagnostic data "30 seconds".
>
> -.. code-block::
> +.. code-block:: none
>
> ok 5 check return code # rcode=0
>
> @@ -202,7 +202,7 @@ allowed to be either indented or not indented.
>
> An example of a test with two nested subtests:
>
> -.. code-block::
> +.. code-block:: none
>
> KTAP version 1
> 1..1
> @@ -215,7 +215,7 @@ An example of a test with two nested subtests:
>
> An example format with multiple levels of nested testing:
>
> -.. code-block::
> +.. code-block:: none
>
> KTAP version 1
> 1..2
> @@ -234,12 +234,15 @@ An example format with multiple levels of nested testing:
> Major differences between TAP and KTAP
> --------------------------------------
>
> -Note the major differences between the TAP and KTAP specification:
> -- yaml and json are not recommended in KTAP diagnostic messages
> -- TODO directive not recognized in KTAP
> -- KTAP allows for an arbitrary number of tests to be nested
> -- TAP includes "Unknown lines" in the category of "Anything else"
> -- TAP says "Unknown lines" are "incorrect"; KTAP allows "Unknown lines"
> +================================================== ========= ===============
> +Feature TAP KTAP
> +================================================== ========= ===============
> +yaml and json in diagnosic message ok not recommended
> +TODO directive ok not recognized
> +allows an arbitrary number of tests to be nested no yes
> +"Unknown lines" are in category of "Anything else" yes no
> +"Unknown lines" are incorrect allowed
> +================================================== ========= ===============
>
> The TAP14 specification does permit nested tests, but instead of using another
> nested version line, uses a line of the form
> @@ -247,7 +250,7 @@ nested version line, uses a line of the form
>
> Example KTAP output
> --------------------
> -.. code-block::
> +.. code-block:: none
>
> KTAP version 1
> 1..1
>
Table looks good. Thanks for adding it. The two patches could be
collapsed into one perhaps? Either way:
Reviewed-by: Shuah Khan <skhan@linuxfoundation.org>
thanks,
-- Shuah
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v4 1/2] Documentation: dev-tools: clarify KTAP specification wording
2022-02-10 2:35 ` [PATCH v4 1/2] " frowand.list
@ 2022-02-10 17:43 ` Kees Cook
0 siblings, 0 replies; 7+ messages in thread
From: Kees Cook @ 2022-02-10 17:43 UTC (permalink / raw)
To: frowand.list
Cc: Jonathan Corbet, David Gow, Shuah Khan, Rae Moar, Tim.Bird,
Brendan Higgins, rmr167, guillaume.tucker, dlatypov, kernelci,
kunit-dev, linux-kselftest, linux-doc, linux-kernel
On Wed, Feb 09, 2022 at 08:35:18PM -0600, frowand.list@gmail.com wrote:
> From: Frank Rowand <frank.rowand@sony.com>
>
> Add the spec version to the title line.
>
> Explain likely source of "Unknown lines".
>
> "Unknown lines" in nested tests are optionally indented.
>
> Add "Unknown lines" items to differences between TAP & KTAP list
>
> Reviewed-by: Tim Bird <Tim.Bird@sony.com>
> Reviewed-by: David Gow <davidgow@google.com>
> Reviewed-by: Shuah Khan <skhan@linuxfoundation.org>
> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
Thanks for nailing this down!
Reviewed-by: Kees Cook <keescook@chromium.org>
--
Kees Cook
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings
2022-02-10 2:35 ` [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings frowand.list
2022-02-10 7:44 ` David Gow
2022-02-10 17:25 ` Shuah Khan
@ 2022-02-10 17:43 ` Kees Cook
2 siblings, 0 replies; 7+ messages in thread
From: Kees Cook @ 2022-02-10 17:43 UTC (permalink / raw)
To: frowand.list
Cc: Jonathan Corbet, David Gow, Shuah Khan, Rae Moar, Tim.Bird,
Brendan Higgins, rmr167, guillaume.tucker, dlatypov, kernelci,
kunit-dev, linux-kselftest, linux-doc, linux-kernel
On Wed, Feb 09, 2022 at 08:35:19PM -0600, frowand.list@gmail.com wrote:
> From: Frank Rowand <frank.rowand@sony.com>
>
> Convert "Major differences between TAP and KTAP" from a bullet list
> to a table. The bullet list was being formatted as a single
> paragraph.
>
> Add missing required argument in code-block directives.
>
> ---
>
> Table conversion suggested by Shuah.
>
> Patch 2/2 not previously reviewed, so Reviewed-by tags not provided.
>
> Changes since version 3
> - Add this commit (patch 2/2) to the series
>
> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
--
Kees Cook
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-02-10 17:43 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-10 2:35 [PATCH v4 0/2] Documentation: dev-tools: clarify KTAP specification wording frowand.list
2022-02-10 2:35 ` [PATCH v4 1/2] " frowand.list
2022-02-10 17:43 ` Kees Cook
2022-02-10 2:35 ` [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings frowand.list
2022-02-10 7:44 ` David Gow
2022-02-10 17:25 ` Shuah Khan
2022-02-10 17:43 ` Kees Cook
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).