[PATCH v2 1/1] Documentation: dev-tools: clarify KTAP specification wording

[PATCH v2 1/1] Documentation: dev-tools: clarify KTAP specification wording

[frowand.list]

From: Frank Rowand <frank.rowand@sony.com>

Add the spec version to the title line.

Explain likely source of "Unknown lines".

"Unknown lines" in nested tests are optionally indented.

Add "Unknown lines" items to differences between TAP & KTAP list

Signed-off-by: Frank Rowand <frank.rowand@sony.com>
---

Changes since version 1
  - Explain likely source of "Unknown lines"
  - "Unknown line" in nested tests are optionally indented
  - Add "Unknown lines" items to differences between TAP & KTAP list

 Documentation/dev-tools/ktap.rst | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
index 878530cb9c27..9c40c94d3f12 100644
--- a/Documentation/dev-tools/ktap.rst
+++ b/Documentation/dev-tools/ktap.rst
@@ -1,8 +1,8 @@
 .. SPDX-License-Identifier: GPL-2.0
 
-========================================
-The Kernel Test Anything Protocol (KTAP)
-========================================
+===================================================
+The Kernel Test Anything Protocol (KTAP), version 1
+===================================================
 
 TAP, or the Test Anything Protocol is a format for specifying test results used
 by a number of projects. It's website and specification are found at this `link
@@ -174,6 +174,13 @@ There may be lines within KTAP output that do not follow the format of one of
 the four formats for lines described above. This is allowed, however, they will
 not influence the status of the tests.
 
+This is an important difference from TAP.  Kernel tests may print messages
+to the system console or a log file.  Both of these destinations may contain
+messages either from unrelated kernel or userspace activity, or kernel
+messages from non-test code that is invoked by the test.  The kernel code
+invoked by the test likely is not aware that a test is in progress and
+thus can not print the message as a diagnostic message.
+
 Nested tests
 ------------
 
@@ -186,10 +193,13 @@ starting with another KTAP version line and test plan, and end with the overall
 result. If one of the subtests fail, for example, the parent test should also
 fail.
 
-Additionally, all result lines in a subtest should be indented. One level of
+Additionally, all lines in a subtest should be indented. One level of
 indentation is two spaces: "  ". The indentation should begin at the version
 line and should end before the parent test's result line.
 
+"Unknown lines" are not considered to be lines in a subtest and thus are
+allowed to be either indented or not indented.
+
 An example of a test with two nested subtests:
 
 .. code-block::
@@ -225,9 +235,11 @@ Major differences between TAP and KTAP
 --------------------------------------
 
 Note the major differences between the TAP and KTAP specification:
-- yaml and json are not recommended in diagnostic messages
-- TODO directive not recognized
+- yaml and json are not recommended in KTAP diagnostic messages
+- TODO directive not recognized in KTAP
 - KTAP allows for an arbitrary number of tests to be nested
+- TAP includes "Unknown lines" in the category of "Anything else"
+- TAP says "Unknown lines" are "incorrect; KTAP allows "Unknown lines"
 
 The TAP14 specification does permit nested tests, but instead of using another
 nested version line, uses a line of the form
-- 
Frank Rowand <frank.rowand@sony.com>

Re: [PATCH v2 1/1] Documentation: dev-tools: clarify KTAP specification wording

[Frank Rowand]

On 2/7/22 10:28 AM, frowand.list@gmail.com wrote:
> From: Frank Rowand <frank.rowand@sony.com>
> 
> Add the spec version to the title line.
> 
> Explain likely source of "Unknown lines".
> 
> "Unknown lines" in nested tests are optionally indented.
> 
> Add "Unknown lines" items to differences between TAP & KTAP list
> 
> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
> ---

Please do not apply this version (v2).  I was working off the KTAP specification
from a different github repo than I should have been referencing.

v3 upcoming.

-Frank

> 
> Changes since version 1
>   - Explain likely source of "Unknown lines"
>   - "Unknown line" in nested tests are optionally indented
>   - Add "Unknown lines" items to differences between TAP & KTAP list
> 
>  Documentation/dev-tools/ktap.rst | 24 ++++++++++++++++++------
>  1 file changed, 18 insertions(+), 6 deletions(-)
> 
> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> index 878530cb9c27..9c40c94d3f12 100644
> --- a/Documentation/dev-tools/ktap.rst
> +++ b/Documentation/dev-tools/ktap.rst
> @@ -1,8 +1,8 @@
>  .. SPDX-License-Identifier: GPL-2.0
>  
> -========================================
> -The Kernel Test Anything Protocol (KTAP)
> -========================================
> +===================================================
> +The Kernel Test Anything Protocol (KTAP), version 1
> +===================================================
>  
>  TAP, or the Test Anything Protocol is a format for specifying test results used
>  by a number of projects. It's website and specification are found at this `link
> @@ -174,6 +174,13 @@ There may be lines within KTAP output that do not follow the format of one of
>  the four formats for lines described above. This is allowed, however, they will
>  not influence the status of the tests.
>  
> +This is an important difference from TAP.  Kernel tests may print messages
> +to the system console or a log file.  Both of these destinations may contain
> +messages either from unrelated kernel or userspace activity, or kernel
> +messages from non-test code that is invoked by the test.  The kernel code
> +invoked by the test likely is not aware that a test is in progress and
> +thus can not print the message as a diagnostic message.
> +
>  Nested tests
>  ------------
>  
> @@ -186,10 +193,13 @@ starting with another KTAP version line and test plan, and end with the overall
>  result. If one of the subtests fail, for example, the parent test should also
>  fail.
>  
> -Additionally, all result lines in a subtest should be indented. One level of
> +Additionally, all lines in a subtest should be indented. One level of
>  indentation is two spaces: "  ". The indentation should begin at the version
>  line and should end before the parent test's result line.
>  
> +"Unknown lines" are not considered to be lines in a subtest and thus are
> +allowed to be either indented or not indented.
> +
>  An example of a test with two nested subtests:
>  
>  .. code-block::
> @@ -225,9 +235,11 @@ Major differences between TAP and KTAP
>  --------------------------------------
>  
>  Note the major differences between the TAP and KTAP specification:
> -- yaml and json are not recommended in diagnostic messages
> -- TODO directive not recognized
> +- yaml and json are not recommended in KTAP diagnostic messages
> +- TODO directive not recognized in KTAP
>  - KTAP allows for an arbitrary number of tests to be nested
> +- TAP includes "Unknown lines" in the category of "Anything else"
> +- TAP says "Unknown lines" are "incorrect; KTAP allows "Unknown lines"
>  
>  The TAP14 specification does permit nested tests, but instead of using another
>  nested version line, uses a line of the form
> 

Re: [PATCH v2 1/1] Documentation: dev-tools: clarify KTAP specification wording

[Frank Rowand]

On 2/7/22 10:38 AM, Frank Rowand wrote:
> On 2/7/22 10:28 AM, frowand.list@gmail.com wrote:
>> From: Frank Rowand <frank.rowand@sony.com>
>>
>> Add the spec version to the title line.
>>
>> Explain likely source of "Unknown lines".
>>
>> "Unknown lines" in nested tests are optionally indented.
>>
>> Add "Unknown lines" items to differences between TAP & KTAP list
>>
>> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
>> ---
> 
> Please do not apply this version (v2).  I was working off the KTAP specification
                                                                ^^^^ TAP
> from a different github repo than I should have been referencing.

OK to proceed with this version of the patch.  I was referencing the proposed
TAP version 14, which is in a different github repo than the TAP version 13
specification.  Everything in this patch is consistent with TAP version 13,
as well as the proposed TAP version 14.

-Frank

> 
> v3 upcoming.
> 
> -Frank
> 
>>
>> Changes since version 1
>>   - Explain likely source of "Unknown lines"
>>   - "Unknown line" in nested tests are optionally indented
>>   - Add "Unknown lines" items to differences between TAP & KTAP list
>>
>>  Documentation/dev-tools/ktap.rst | 24 ++++++++++++++++++------
>>  1 file changed, 18 insertions(+), 6 deletions(-)
>>
>> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
>> index 878530cb9c27..9c40c94d3f12 100644
>> --- a/Documentation/dev-tools/ktap.rst
>> +++ b/Documentation/dev-tools/ktap.rst
>> @@ -1,8 +1,8 @@
>>  .. SPDX-License-Identifier: GPL-2.0
>>  
>> -========================================
>> -The Kernel Test Anything Protocol (KTAP)
>> -========================================
>> +===================================================
>> +The Kernel Test Anything Protocol (KTAP), version 1
>> +===================================================
>>  
>>  TAP, or the Test Anything Protocol is a format for specifying test results used
>>  by a number of projects. It's website and specification are found at this `link
>> @@ -174,6 +174,13 @@ There may be lines within KTAP output that do not follow the format of one of
>>  the four formats for lines described above. This is allowed, however, they will
>>  not influence the status of the tests.
>>  
>> +This is an important difference from TAP.  Kernel tests may print messages
>> +to the system console or a log file.  Both of these destinations may contain
>> +messages either from unrelated kernel or userspace activity, or kernel
>> +messages from non-test code that is invoked by the test.  The kernel code
>> +invoked by the test likely is not aware that a test is in progress and
>> +thus can not print the message as a diagnostic message.
>> +
>>  Nested tests
>>  ------------
>>  
>> @@ -186,10 +193,13 @@ starting with another KTAP version line and test plan, and end with the overall
>>  result. If one of the subtests fail, for example, the parent test should also
>>  fail.
>>  
>> -Additionally, all result lines in a subtest should be indented. One level of
>> +Additionally, all lines in a subtest should be indented. One level of
>>  indentation is two spaces: "  ". The indentation should begin at the version
>>  line and should end before the parent test's result line.
>>  
>> +"Unknown lines" are not considered to be lines in a subtest and thus are
>> +allowed to be either indented or not indented.
>> +
>>  An example of a test with two nested subtests:
>>  
>>  .. code-block::
>> @@ -225,9 +235,11 @@ Major differences between TAP and KTAP
>>  --------------------------------------
>>  
>>  Note the major differences between the TAP and KTAP specification:
>> -- yaml and json are not recommended in diagnostic messages
>> -- TODO directive not recognized
>> +- yaml and json are not recommended in KTAP diagnostic messages
>> +- TODO directive not recognized in KTAP
>>  - KTAP allows for an arbitrary number of tests to be nested
>> +- TAP includes "Unknown lines" in the category of "Anything else"
>> +- TAP says "Unknown lines" are "incorrect; KTAP allows "Unknown lines"
>>  
>>  The TAP14 specification does permit nested tests, but instead of using another
>>  nested version line, uses a line of the form
>>
> 

RE: [PATCH v2 1/1] Documentation: dev-tools: clarify KTAP specification wording

[Bird, Tim]

> -----Original Message-----
> From: Frank Rowand <frowand.list@gmail.com>
> 
> On 2/7/22 10:38 AM, Frank Rowand wrote:
> > On 2/7/22 10:28 AM, frowand.list@gmail.com wrote:
> >> From: Frank Rowand <frank.rowand@sony.com>
> >>
> >> Add the spec version to the title line.
> >>
> >> Explain likely source of "Unknown lines".
> >>
> >> "Unknown lines" in nested tests are optionally indented.
> >>
> >> Add "Unknown lines" items to differences between TAP & KTAP list
> >>
> >> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
> >> ---
> >
> > Please do not apply this version (v2).  I was working off the KTAP specification
>                                                                 ^^^^ TAP

What???

> > from a different github repo than I should have been referencing.
> 
> OK to proceed with this version of the patch.  I was referencing the proposed
> TAP version 14, which is in a different github repo than the TAP version 13
> specification.  Everything in this patch is consistent with TAP version 13,
> as well as the proposed TAP version 14.

I'm confused by this message.
Will there be a v3 of the patch, or are you saying to just apply the original v2 of
this patch?

For the original (v2) of the patch, which I believe is 
https://lore.kernel.org/all/20220207162813.3091899-1-frowand.list@gmail.com/

you can add a "Reviewed-by" from me.
 -- Tim

Re: [PATCH v2 1/1] Documentation: dev-tools: clarify KTAP specification wording

[Bird, Tim]

> -----Original Message-----
> From: Frank Rowand <frowand.list@gmail.com>
> 
> On 2/7/22 10:38 AM, Frank Rowand wrote:
> > On 2/7/22 10:28 AM, frowand.list@gmail.com wrote:
> >> From: Frank Rowand <frank.rowand@sony.com>
> >>
> >> Add the spec version to the title line.
> >>
> >> Explain likely source of "Unknown lines".
> >>
> >> "Unknown lines" in nested tests are optionally indented.
> >>
> >> Add "Unknown lines" items to differences between TAP & KTAP list
> >>
> >> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
> >> ---
> >
> > Please do not apply this version (v2).  I was working off the KTAP specification
>                                                                 ^^^^ TAP

What???

> > from a different github repo than I should have been referencing.
> 
> OK to proceed with this version of the patch.  I was referencing the proposed
> TAP version 14, which is in a different github repo than the TAP version 13
> specification.  Everything in this patch is consistent with TAP version 13,
> as well as the proposed TAP version 14.

I'm confused by this message.
Will there be a v3 of the patch, or are you saying to just apply the original v2 of
this patch?

For the original (v2) of the patch, which I believe is 
https://lore.kernel.org/all/20220207162813.3091899-1-frowand.list@gmail.com/

you can add a "Reviewed-by" from me.
 -- Tim

Re: [PATCH v2 1/1] Documentation: dev-tools: clarify KTAP specification wording

[Frank Rowand]

On 2/7/22 11:06 AM, Bird, Tim wrote:
> 
> 
>> -----Original Message-----
>> From: Frank Rowand <frowand.list@gmail.com>
>>
>> On 2/7/22 10:38 AM, Frank Rowand wrote:
>>> On 2/7/22 10:28 AM, frowand.list@gmail.com wrote:
>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>
>>>> Add the spec version to the title line.
>>>>
>>>> Explain likely source of "Unknown lines".
>>>>
>>>> "Unknown lines" in nested tests are optionally indented.
>>>>
>>>> Add "Unknown lines" items to differences between TAP & KTAP list
>>>>
>>>> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
>>>> ---
>>>
>>> Please do not apply this version (v2).  I was working off the KTAP specification
>>                                                                 ^^^^ TAP
> 
> What???

I was looking at the proposed TAP v14 from:
  branch 'git14' of https://github.com/isaacs/testanything.github.io.git
while David was looking at
  branch 'tap-14-specification' of https://github.com/TestAnything/Specification/

David did not find the line type of "Anything else" in the TAP v14 he was looking
at, but did find it in TAP v13 (tap-version-13-specification.md) which is in
https://github.com/TestAnything/testanything.github.io
so my reference to TAP "Anything else" in the patch remains valid.  The lack
of "Anything else" in the TAP v14 that David was looking at is why I initially
said to not apply this patch (v2).

> 
>>> from a different github repo than I should have been referencing.
>>
>> OK to proceed with this version of the patch.  I was referencing the proposed
>> TAP version 14, which is in a different github repo than the TAP version 13
>> specification.  Everything in this patch is consistent with TAP version 13,
>> as well as the proposed TAP version 14.
> 
> I'm confused by this message.
> Will there be a v3 of the patch, or are you saying to just apply the original v2 of
> this patch?

Sorry for the confusion.  There is no longer a need for v3 of the patch, so it is
ok to apply v2 if it passes review.

> 
> For the original (v2) of the patch, which I believe is 
> https://lore.kernel.org/all/20220207162813.3091899-1-frowand.list@gmail.com/
> 
> you can add a "Reviewed-by" from me.
>  -- Tim
> 

Re: [PATCH v2 1/1] Documentation: dev-tools: clarify KTAP specification wording

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 3765 bytes --]

On Tue, Feb 8, 2022 at 12:28 AM <frowand.list@gmail.com> wrote:
>
> From: Frank Rowand <frank.rowand@sony.com>
>
> Add the spec version to the title line.
>
> Explain likely source of "Unknown lines".
>
> "Unknown lines" in nested tests are optionally indented.
>
> Add "Unknown lines" items to differences between TAP & KTAP list
>
> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
> ---

Thanks for fixing this. I'm happy with this version.

Reviewed-by: David Gow <davidgow@google.com>

>
> Changes since version 1
>   - Explain likely source of "Unknown lines"
>   - "Unknown line" in nested tests are optionally indented
>   - Add "Unknown lines" items to differences between TAP & KTAP list
>
>  Documentation/dev-tools/ktap.rst | 24 ++++++++++++++++++------
>  1 file changed, 18 insertions(+), 6 deletions(-)
>
> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> index 878530cb9c27..9c40c94d3f12 100644
> --- a/Documentation/dev-tools/ktap.rst
> +++ b/Documentation/dev-tools/ktap.rst
> @@ -1,8 +1,8 @@
>  .. SPDX-License-Identifier: GPL-2.0
>
> -========================================
> -The Kernel Test Anything Protocol (KTAP)
> -========================================
> +===================================================
> +The Kernel Test Anything Protocol (KTAP), version 1
> +===================================================
>
>  TAP, or the Test Anything Protocol is a format for specifying test results used
>  by a number of projects. It's website and specification are found at this `link
> @@ -174,6 +174,13 @@ There may be lines within KTAP output that do not follow the format of one of
>  the four formats for lines described above. This is allowed, however, they will
>  not influence the status of the tests.
>
> +This is an important difference from TAP.  Kernel tests may print messages
> +to the system console or a log file.  Both of these destinations may contain
> +messages either from unrelated kernel or userspace activity, or kernel
> +messages from non-test code that is invoked by the test.  The kernel code
> +invoked by the test likely is not aware that a test is in progress and
> +thus can not print the message as a diagnostic message.
> +
>  Nested tests
>  ------------
>
> @@ -186,10 +193,13 @@ starting with another KTAP version line and test plan, and end with the overall
>  result. If one of the subtests fail, for example, the parent test should also
>  fail.
>
> -Additionally, all result lines in a subtest should be indented. One level of
> +Additionally, all lines in a subtest should be indented. One level of
>  indentation is two spaces: "  ". The indentation should begin at the version
>  line and should end before the parent test's result line.
>
> +"Unknown lines" are not considered to be lines in a subtest and thus are
> +allowed to be either indented or not indented.
> +
>  An example of a test with two nested subtests:
>
>  .. code-block::
> @@ -225,9 +235,11 @@ Major differences between TAP and KTAP
>  --------------------------------------
>
>  Note the major differences between the TAP and KTAP specification:
> -- yaml and json are not recommended in diagnostic messages
> -- TODO directive not recognized
> +- yaml and json are not recommended in KTAP diagnostic messages
> +- TODO directive not recognized in KTAP
>  - KTAP allows for an arbitrary number of tests to be nested
> +- TAP includes "Unknown lines" in the category of "Anything else"
> +- TAP says "Unknown lines" are "incorrect; KTAP allows "Unknown lines"

Nit: unmached quotes around "incorrect".

>
>  The TAP14 specification does permit nested tests, but instead of using another
>  nested version line, uses a line of the form
> --
> Frank Rowand <frank.rowand@sony.com>
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]