Linux-kselftest Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings
@ 2019-08-27  0:37 Shuah Khan
  2019-08-28 19:17 ` shuah
  2019-08-29 17:30 ` Brendan Higgins
  0 siblings, 2 replies; 5+ messages in thread
From: Shuah Khan @ 2019-08-27  0:37 UTC (permalink / raw)
  To: shuah, corbet
  Cc: Shuah Khan, linux-kselftest, linux-doc, linux-kernel, dan.rue,
	anders.roxell

Update to add clarity and recommendations on running newer kselftests
on older kernels vs. matching the kernel and kselftest revisions.

The recommendation is "Match kernel revision and kselftest."

Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
---
Changes since v1: Fixed "WARNING: Title underline too short."

 Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index 25604904fa6e..308506c5e8fa 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki:
 
 https://kselftest.wiki.kernel.org/
 
+Recommendations on running kselftests in Continuous Integration test rings
+==========================================================================
+
+It is recommended that users run Kselftest from the same release. Running
+newer Kselftest on older kernels isn't recommended for the following
+reasons:
+
+- Kselftest from mainline and linux-next might not be stable enough to run
+  on stable kernels.
+- Kselftests detect feature dependencies at run-time and skip tests if a
+  feature and/or configuration they test aren't enabled. Running newer
+  tests on older kernels could result in a few too many skipped/failed
+  conditions. It becomes difficult to evaluate the results.
+- Newer tests provide better coverage. However, users should make a judgement
+  call on coverage vs. run to run consistency and being able to compare
+  run to run results on older kernels.
+
+Recommendations:
+
+Match kernel revision and kselftest. Especially important for LTS and
+Stable kernel Continuous Integration test rings.
+
+Hot-plug tests
+==============
+
 On some systems, hot-plug tests could hang forever waiting for cpu and
 memory to be ready to be offlined. A special hot-plug target is created
 to run the full range of hot-plug tests. In default mode, hot-plug tests run
-- 
2.20.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings
  2019-08-27  0:37 [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings Shuah Khan
@ 2019-08-28 19:17 ` shuah
  2019-08-29 17:38   ` Brendan Higgins
  2019-08-29 17:30 ` Brendan Higgins
  1 sibling, 1 reply; 5+ messages in thread
From: shuah @ 2019-08-28 19:17 UTC (permalink / raw)
  To: Shuah Khan, corbet
  Cc: linux-kselftest, linux-doc, linux-kernel, dan.rue, anders.roxell, shuah

On 8/26/19 6:37 PM, Shuah Khan wrote:
> Update to add clarity and recommendations on running newer kselftests
> on older kernels vs. matching the kernel and kselftest revisions.
> 
> The recommendation is "Match kernel revision and kselftest."
> 
> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
> ---
> Changes since v1: Fixed "WARNING: Title underline too short."

I have a few more changes and would like to make and send a v3 after
the LPC's Testing and Fuzzing kselftest discussion.

Holding off on this patch for now.

thanks,
-- Shuah

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings
  2019-08-27  0:37 [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings Shuah Khan
  2019-08-28 19:17 ` shuah
@ 2019-08-29 17:30 ` Brendan Higgins
  2019-08-29 17:39   ` shuah
  1 sibling, 1 reply; 5+ messages in thread
From: Brendan Higgins @ 2019-08-29 17:30 UTC (permalink / raw)
  To: Shuah Khan
  Cc: shuah, Jonathan Corbet, open list:KERNEL SELFTEST FRAMEWORK,
	open list:DOCUMENTATION, Linux Kernel Mailing List, dan.rue,
	anders.roxell

On Mon, Aug 26, 2019 at 5:37 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
>
> Update to add clarity and recommendations on running newer kselftests
> on older kernels vs. matching the kernel and kselftest revisions.
>
> The recommendation is "Match kernel revision and kselftest."
>
> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

Reviewed-by: Brendan Higgins <brendanhiggins@google.com>

One minor comment below.

> ---
> Changes since v1: Fixed "WARNING: Title underline too short."
>
>  Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
>
> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
> index 25604904fa6e..308506c5e8fa 100644
> --- a/Documentation/dev-tools/kselftest.rst
> +++ b/Documentation/dev-tools/kselftest.rst
> @@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki:
>
>  https://kselftest.wiki.kernel.org/
>
> +Recommendations on running kselftests in Continuous Integration test rings
> +==========================================================================
> +
> +It is recommended that users run Kselftest from the same release. Running
> +newer Kselftest on older kernels isn't recommended for the following
> +reasons:
> +
> +- Kselftest from mainline and linux-next might not be stable enough to run
> +  on stable kernels.
> +- Kselftests detect feature dependencies at run-time and skip tests if a
> +  feature and/or configuration they test aren't enabled. Running newer
> +  tests on older kernels could result in a few too many skipped/failed
> +  conditions. It becomes difficult to evaluate the results.
> +- Newer tests provide better coverage. However, users should make a judgement
> +  call on coverage vs. run to run consistency and being able to compare
> +  run to run results on older kernels.
> +
> +Recommendations:

I think this should be the "note" directive. You already have a list
of recommendations above; this seems like a TLDR section, or an
important note or tip.

> +
> +Match kernel revision and kselftest. Especially important for LTS and
> +Stable kernel Continuous Integration test rings.
> +
> +Hot-plug tests
> +==============
> +
>  On some systems, hot-plug tests could hang forever waiting for cpu and
>  memory to be ready to be offlined. A special hot-plug target is created
>  to run the full range of hot-plug tests. In default mode, hot-plug tests run
> --
> 2.20.1
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings
  2019-08-28 19:17 ` shuah
@ 2019-08-29 17:38   ` Brendan Higgins
  0 siblings, 0 replies; 5+ messages in thread
From: Brendan Higgins @ 2019-08-29 17:38 UTC (permalink / raw)
  To: shuah
  Cc: Shuah Khan, Jonathan Corbet, open list:KERNEL SELFTEST FRAMEWORK,
	open list:DOCUMENTATION, Linux Kernel Mailing List, dan.rue,
	anders.roxell

On Wed, Aug 28, 2019 at 12:18 PM shuah <shuah@kernel.org> wrote:
>
> On 8/26/19 6:37 PM, Shuah Khan wrote:
> > Update to add clarity and recommendations on running newer kselftests
> > on older kernels vs. matching the kernel and kselftest revisions.
> >
> > The recommendation is "Match kernel revision and kselftest."
> >
> > Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
> > ---
> > Changes since v1: Fixed "WARNING: Title underline too short."
>
> I have a few more changes and would like to make and send a v3 after
> the LPC's Testing and Fuzzing kselftest discussion.
>
> Holding off on this patch for now.

Is this just because you are busy, or because you expect what you want
to say to change after the discussion?

If it is because you expect what you want to say here to change, I am
surprised. This seems like pretty good, straightforward advice. From
where I stand this seems like it makes the documentation better
without making anything worse, and so this change should probably be
included.

In anycase, your call. I just don't think anyone is going to dispute
what you are saying here :-)

Cheers!

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings
  2019-08-29 17:30 ` Brendan Higgins
@ 2019-08-29 17:39   ` shuah
  0 siblings, 0 replies; 5+ messages in thread
From: shuah @ 2019-08-29 17:39 UTC (permalink / raw)
  To: Brendan Higgins, Shuah Khan
  Cc: Jonathan Corbet, open list:KERNEL SELFTEST FRAMEWORK,
	open list:DOCUMENTATION, Linux Kernel Mailing List, dan.rue,
	anders.roxell, shuah

On 8/29/19 11:30 AM, Brendan Higgins wrote:
> On Mon, Aug 26, 2019 at 5:37 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
>>
>> Update to add clarity and recommendations on running newer kselftests
>> on older kernels vs. matching the kernel and kselftest revisions.
>>
>> The recommendation is "Match kernel revision and kselftest."
>>
>> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
> 
> Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
> 
> One minor comment below.
> 
>> ---
>> Changes since v1: Fixed "WARNING: Title underline too short."
>>
>>   Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++
>>   1 file changed, 25 insertions(+)
>>
>> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
>> index 25604904fa6e..308506c5e8fa 100644
>> --- a/Documentation/dev-tools/kselftest.rst
>> +++ b/Documentation/dev-tools/kselftest.rst
>> @@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki:
>>
>>   https://kselftest.wiki.kernel.org/
>>
>> +Recommendations on running kselftests in Continuous Integration test rings
>> +==========================================================================
>> +
>> +It is recommended that users run Kselftest from the same release. Running
>> +newer Kselftest on older kernels isn't recommended for the following
>> +reasons:
>> +
>> +- Kselftest from mainline and linux-next might not be stable enough to run
>> +  on stable kernels.
>> +- Kselftests detect feature dependencies at run-time and skip tests if a
>> +  feature and/or configuration they test aren't enabled. Running newer
>> +  tests on older kernels could result in a few too many skipped/failed
>> +  conditions. It becomes difficult to evaluate the results.
>> +- Newer tests provide better coverage. However, users should make a judgement
>> +  call on coverage vs. run to run consistency and being able to compare
>> +  run to run results on older kernels.
>> +
>> +Recommendations:
> 
> I think this should be the "note" directive. You already have a list
> of recommendations above; this seems like a TLDR section, or an
> important note or tip.
> 
>> +
>> +Match kernel revision and kselftest. Especially important for LTS and
>> +Stable kernel Continuous Integration test rings.
>> +
>> +Hot-plug tests
>> +==============
>> +
>>   On some systems, hot-plug tests could hang forever waiting for cpu and
>>   memory to be ready to be offlined. A special hot-plug target is created
>>   to run the full range of hot-plug tests. In default mode, hot-plug tests run
>> --
>> 2.20.1
>>
> 

Thanks Brendan. I am holding off on this patch for now. I will have
to make some updates. I will fold your comment in when I send an
updated patch.

thanks,
-- Shuah

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-27  0:37 [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings Shuah Khan
2019-08-28 19:17 ` shuah
2019-08-29 17:38   ` Brendan Higgins
2019-08-29 17:30 ` Brendan Higgins
2019-08-29 17:39   ` shuah

Linux-kselftest Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-kselftest/0 linux-kselftest/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-kselftest linux-kselftest/ https://lore.kernel.org/linux-kselftest \
		linux-kselftest@vger.kernel.org linux-kselftest@archiver.kernel.org
	public-inbox-index linux-kselftest

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kselftest


AGPL code for this site: git clone https://public-inbox.org/ public-inbox