Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan <shuah@kernel.org>]

On 06/22/2017 10:02 PM, Sumit Semwal wrote:
> Hi Shuah,
> 
> On 23 June 2017 at 01:53, Shuah Khan <shuah@kernel.org> wrote:
>> Hi Tom,
>>
>> On 06/22/2017 01:48 PM, Tom Gall wrote:
>>> Hi
>>>
>>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan <shuah@kernel.org> wrote:
>>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote:
>>>>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote:
>>>>>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>>>>>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote:
>>>>>>>>> Hi Kees, Andy,
>>>>>>>>>
>>>>>>>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote:
>>>>>>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>>>>>>>>> feature and test together.
>>>>>>>>>> - This one also seems like a security hole being closed, and the
>>>>>>>>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>>>>>>>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>>>>>>>>> from the subsystem developers here.
>>>>>>>>>
>>>>>>>>> Could you please help us sort this out? Our goal is to help Greg with
>>>>>>>>> testing stable kernels, and currently the seccomp tests fail due to
>>>>>>>>> missing feature (seccomp ptrace hole closure) getting tested via
>>>>>>>>> latest kselftest.
>>>>>>>>>
>>>>>>>>> If you feel the feature isn't a stable candidate, then could you
>>>>>>>>> please help make the test degrade gracefully in its absence?
>>>>
>>>> In some cases, it is not easy to degrade and/or check for a feature.
>>>> Probably several security features could fall in this bucket.
>>>>
>>>>>>>>
>>>>>>>> I don't really want to have that change be a backport -- it's quite
>>>>>>>> invasive across multiple architectures.
>>>>
>>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>>> feature can't be back-ported, the test for that feature will fall in the
>>>> same bucket. It shouldn't be back-ported.
>>>>
>>>>>>>>
>>>>>>>> I would say just add a kernel version check to the test. This is
>>>>>>>> probably not the only selftest that will need such things. :)
>>>>>>>
>>>>>>> Adding release checks to selftests is going to problematic for maintenance.
>>>>>>> Tests should fail gracefully if feature isn't supported in older kernels.
>>>>>>>
>>>>>>> Several tests do that now and please find a way to check for dependencies
>>>>>>> and feature availability and fail the test gracefully. If there is a test
>>>>>>> that can't do that for some reason, we can discuss it, but as a general
>>>>>>> rule, I don't want to see kselftest patches that check release.
>>>>>>
>>>>>> If a future kernel inadvertently loses the new feature and degrades to
>>>>>> the behavior of old kernels, that would be a serious bug and should be
>>>>>> caught.
>>>>
>>>> Agreed. If I understand you correctly, by not testing stable kernels
>>>> with their own selftests, some serious bugs could go undetected.
>>>
>>> Personally I'm a bit skeptical. I think the reasoning is more that the
>>> latest selftests provide more coverage, and therefore should be better
>>> tests, even on older kernels.
>>
>> The assumption that "the latest selftests provide more coverage, and
>> therefore should be better tests, even on older kernels." is incorrect.
>>
>> Selftests in general track the kernel features. In some cases, new
>> tests could be added that provide better coverage on older kernels,
>> however, it is more likely that new tests are added to test new kernel
>> features and enhancements to existing features. Based on the second
>> "enhancements to existing features" it is more important to test newer
>> kernels with older selftests. This does happen in kernel integration
>> cycles during development.
>>
>> As a general rule, testing stable kernels with their own selftests will
>> yield the best results.
>>
> I would have agreed totally, if the selftests and the kernel were in
> sync since forever. But since the kselftests are a comparatively
> recent addition, the number of tests available for features existing
> in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
> misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
> to name a few.

Yes. Several tests have been added since 4.4. Unfortunately these new
additions don't qualify as candidates for stable releases.

> 
> Also, while trying to run kselftests from later kernels with 4.4, we
> only had a few failures for existing features, while most other tests
> ran ok. Just another data point.

That is good to hear. Let's fix the problems as you find them and make
the selftests from newer releases work well (if not perfect) on older
kernels.

4.13 is going to see TAP13 changes as well, so if selftests from newer
kernels work for you, you will see TAB13 benefits.

thanks,
-- Shuah