Re: [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings

From: shuah <shuah@kernel.org>
To: Brendan Higgins <brendanhiggins@google.com>,
	Shuah Khan <skhan@linuxfoundation.org>
Cc: Jonathan Corbet <corbet@lwn.net>,
	"open list:KERNEL SELFTEST FRAMEWORK" 
	<linux-kselftest@vger.kernel.org>,
	"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	dan.rue@linaro.org, anders.roxell@linaro.org,
	shuah <shuah@kernel.org>
Subject: Re: [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings
Date: Thu, 29 Aug 2019 11:39:31 -0600	[thread overview]
Message-ID: <f328adad-7559-a459-fd56-b46795d882c6@kernel.org> (raw)
In-Reply-To: <CAFd5g47jeZGqdhOUEOFGwHqHQ1O0=99Ce+bFhZ-7PCyFT1YO1Q@mail.gmail.com>

On 8/29/19 11:30 AM, Brendan Higgins wrote:
> On Mon, Aug 26, 2019 at 5:37 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
>>
>> Update to add clarity and recommendations on running newer kselftests
>> on older kernels vs. matching the kernel and kselftest revisions.
>>
>> The recommendation is "Match kernel revision and kselftest."
>>
>> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
> 
> Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
> 
> One minor comment below.
> 
>> ---
>> Changes since v1: Fixed "WARNING: Title underline too short."
>>
>>   Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++
>>   1 file changed, 25 insertions(+)
>>
>> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
>> index 25604904fa6e..308506c5e8fa 100644
>> --- a/Documentation/dev-tools/kselftest.rst
>> +++ b/Documentation/dev-tools/kselftest.rst
>> @@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki:
>>
>>   https://kselftest.wiki.kernel.org/
>>
>> +Recommendations on running kselftests in Continuous Integration test rings
>> +==========================================================================
>> +
>> +It is recommended that users run Kselftest from the same release. Running
>> +newer Kselftest on older kernels isn't recommended for the following
>> +reasons:
>> +
>> +- Kselftest from mainline and linux-next might not be stable enough to run
>> +  on stable kernels.
>> +- Kselftests detect feature dependencies at run-time and skip tests if a
>> +  feature and/or configuration they test aren't enabled. Running newer
>> +  tests on older kernels could result in a few too many skipped/failed
>> +  conditions. It becomes difficult to evaluate the results.
>> +- Newer tests provide better coverage. However, users should make a judgement
>> +  call on coverage vs. run to run consistency and being able to compare
>> +  run to run results on older kernels.
>> +
>> +Recommendations:
> 
> I think this should be the "note" directive. You already have a list
> of recommendations above; this seems like a TLDR section, or an
> important note or tip.
> 
>> +
>> +Match kernel revision and kselftest. Especially important for LTS and
>> +Stable kernel Continuous Integration test rings.
>> +
>> +Hot-plug tests
>> +==============
>> +
>>   On some systems, hot-plug tests could hang forever waiting for cpu and
>>   memory to be ready to be offlined. A special hot-plug target is created
>>   to run the full range of hot-plug tests. In default mode, hot-plug tests run
>> --
>> 2.20.1
>>
> 

Thanks Brendan. I am holding off on this patch for now. I will have
to make some updates. I will fold your comment in when I send an
updated patch.

thanks,
-- Shuah