* seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
@ 2017-06-22 16:18 Sumit Semwal
2017-06-22 16:53 ` Kees Cook
0 siblings, 1 reply; 20+ messages in thread
From: Sumit Semwal @ 2017-06-22 16:18 UTC (permalink / raw)
To: Kees Cook, luto
Cc: Brian Norris, mcgrof, Greg Kroah-Hartman, LKML, stable,
linux-kselftest, Shuah Khan
Hi Kees, Andy,
On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps we'll need some help
> from the subsystem developers here.
Could you please help us sort this out? Our goal is to help Greg with
testing stable kernels, and currently the seccomp tests fail due to
missing feature (seccomp ptrace hole closure) getting tested via
latest kselftest.
If you feel the feature isn't a stable candidate, then could you
please help make the test degrade gracefully in its absence?
Greatly appreciated!
Thanks and best regards,
Sumit
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 16:18 seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] Sumit Semwal @ 2017-06-22 16:53 ` Kees Cook 2017-06-22 17:09 ` Shuah Khan 0 siblings, 1 reply; 20+ messages in thread From: Kees Cook @ 2017-06-22 16:53 UTC (permalink / raw) To: Sumit Semwal Cc: Andy Lutomirski, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, linux-kselftest, Shuah Khan On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: > Hi Kees, Andy, > > On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >> feature and test together. >> - This one also seems like a security hole being closed, and the >> 'feature' could be a candidate for stable backports, but Arnd tried >> that, and it was quite non-trivial. So perhaps we'll need some help >> from the subsystem developers here. > > Could you please help us sort this out? Our goal is to help Greg with > testing stable kernels, and currently the seccomp tests fail due to > missing feature (seccomp ptrace hole closure) getting tested via > latest kselftest. > > If you feel the feature isn't a stable candidate, then could you > please help make the test degrade gracefully in its absence? I don't really want to have that change be a backport -- it's quite invasive across multiple architectures. I would say just add a kernel version check to the test. This is probably not the only selftest that will need such things. :) I'd be happy to review such changes! -Kees -- Kees Cook Pixel Security ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 16:53 ` Kees Cook @ 2017-06-22 17:09 ` Shuah Khan 2017-06-22 17:49 ` Andy Lutomirski 0 siblings, 1 reply; 20+ messages in thread From: Shuah Khan @ 2017-06-22 17:09 UTC (permalink / raw) To: Kees Cook, Sumit Semwal Cc: Andy Lutomirski, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, linux-kselftest, Shuah Khan, Shuah Khan On 06/22/2017 10:53 AM, Kees Cook wrote: > On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >> Hi Kees, Andy, >> >> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>> feature and test together. >>> - This one also seems like a security hole being closed, and the >>> 'feature' could be a candidate for stable backports, but Arnd tried >>> that, and it was quite non-trivial. So perhaps we'll need some help >>> from the subsystem developers here. >> >> Could you please help us sort this out? Our goal is to help Greg with >> testing stable kernels, and currently the seccomp tests fail due to >> missing feature (seccomp ptrace hole closure) getting tested via >> latest kselftest. >> >> If you feel the feature isn't a stable candidate, then could you >> please help make the test degrade gracefully in its absence? > > I don't really want to have that change be a backport -- it's quite > invasive across multiple architectures. > > I would say just add a kernel version check to the test. This is > probably not the only selftest that will need such things. :) Adding release checks to selftests is going to problematic for maintenance. Tests should fail gracefully if feature isn't supported in older kernels. Several tests do that now and please find a way to check for dependencies and feature availability and fail the test gracefully. If there is a test that can't do that for some reason, we can discuss it, but as a general rule, I don't want to see kselftest patches that check release. thanks, -- Shuah ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 17:09 ` Shuah Khan @ 2017-06-22 17:49 ` Andy Lutomirski 2017-06-22 17:50 ` Kees Cook 0 siblings, 1 reply; 20+ messages in thread From: Andy Lutomirski @ 2017-06-22 17:49 UTC (permalink / raw) To: Shuah Khan Cc: Kees Cook, Sumit Semwal, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: > On 06/22/2017 10:53 AM, Kees Cook wrote: >> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>> Hi Kees, Andy, >>> >>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>> feature and test together. >>>> - This one also seems like a security hole being closed, and the >>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>> from the subsystem developers here. >>> >>> Could you please help us sort this out? Our goal is to help Greg with >>> testing stable kernels, and currently the seccomp tests fail due to >>> missing feature (seccomp ptrace hole closure) getting tested via >>> latest kselftest. >>> >>> If you feel the feature isn't a stable candidate, then could you >>> please help make the test degrade gracefully in its absence? >> >> I don't really want to have that change be a backport -- it's quite >> invasive across multiple architectures. >> >> I would say just add a kernel version check to the test. This is >> probably not the only selftest that will need such things. :) > > Adding release checks to selftests is going to problematic for maintenance. > Tests should fail gracefully if feature isn't supported in older kernels. > > Several tests do that now and please find a way to check for dependencies > and feature availability and fail the test gracefully. If there is a test > that can't do that for some reason, we can discuss it, but as a general > rule, I don't want to see kselftest patches that check release. If a future kernel inadvertently loses the new feature and degrades to the behavior of old kernels, that would be a serious bug and should be caught. --Andy ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 17:49 ` Andy Lutomirski @ 2017-06-22 17:50 ` Kees Cook 2017-06-22 19:06 ` Shuah Khan 2017-06-23 1:52 ` Greg Kroah-Hartman 0 siblings, 2 replies; 20+ messages in thread From: Kees Cook @ 2017-06-22 17:50 UTC (permalink / raw) To: Andy Lutomirski Cc: Shuah Khan, Sumit Semwal, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >> On 06/22/2017 10:53 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>> Hi Kees, Andy, >>>> >>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>> feature and test together. >>>>> - This one also seems like a security hole being closed, and the >>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>> from the subsystem developers here. >>>> >>>> Could you please help us sort this out? Our goal is to help Greg with >>>> testing stable kernels, and currently the seccomp tests fail due to >>>> missing feature (seccomp ptrace hole closure) getting tested via >>>> latest kselftest. >>>> >>>> If you feel the feature isn't a stable candidate, then could you >>>> please help make the test degrade gracefully in its absence? >>> >>> I don't really want to have that change be a backport -- it's quite >>> invasive across multiple architectures. >>> >>> I would say just add a kernel version check to the test. This is >>> probably not the only selftest that will need such things. :) >> >> Adding release checks to selftests is going to problematic for maintenance. >> Tests should fail gracefully if feature isn't supported in older kernels. >> >> Several tests do that now and please find a way to check for dependencies >> and feature availability and fail the test gracefully. If there is a test >> that can't do that for some reason, we can discuss it, but as a general >> rule, I don't want to see kselftest patches that check release. > > If a future kernel inadvertently loses the new feature and degrades to > the behavior of old kernels, that would be a serious bug and should be > caught. Right. I really think stable kernels should be tested with their own selftests. If some test is needed in a stable kernel it should be backported to that stable kernel. -Kees -- Kees Cook Pixel Security ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 17:50 ` Kees Cook @ 2017-06-22 19:06 ` Shuah Khan 2017-06-22 19:48 ` Tom Gall 2017-06-23 1:52 ` Greg Kroah-Hartman 1 sibling, 1 reply; 20+ messages in thread From: Shuah Khan @ 2017-06-22 19:06 UTC (permalink / raw) To: Kees Cook, Andy Lutomirski, Sumit Semwal Cc: Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan, Shuah Khan On 06/22/2017 11:50 AM, Kees Cook wrote: > On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >>> On 06/22/2017 10:53 AM, Kees Cook wrote: >>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>> Hi Kees, Andy, >>>>> >>>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>>> feature and test together. >>>>>> - This one also seems like a security hole being closed, and the >>>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>>> from the subsystem developers here. >>>>> >>>>> Could you please help us sort this out? Our goal is to help Greg with >>>>> testing stable kernels, and currently the seccomp tests fail due to >>>>> missing feature (seccomp ptrace hole closure) getting tested via >>>>> latest kselftest. >>>>> >>>>> If you feel the feature isn't a stable candidate, then could you >>>>> please help make the test degrade gracefully in its absence? In some cases, it is not easy to degrade and/or check for a feature. Probably several security features could fall in this bucket. >>>> >>>> I don't really want to have that change be a backport -- it's quite >>>> invasive across multiple architectures. Agreed. The same test for kernel applies to tests as well. If a kernel feature can't be back-ported, the test for that feature will fall in the same bucket. It shouldn't be back-ported. >>>> >>>> I would say just add a kernel version check to the test. This is >>>> probably not the only selftest that will need such things. :) >>> >>> Adding release checks to selftests is going to problematic for maintenance. >>> Tests should fail gracefully if feature isn't supported in older kernels. >>> >>> Several tests do that now and please find a way to check for dependencies >>> and feature availability and fail the test gracefully. If there is a test >>> that can't do that for some reason, we can discuss it, but as a general >>> rule, I don't want to see kselftest patches that check release. >> >> If a future kernel inadvertently loses the new feature and degrades to >> the behavior of old kernels, that would be a serious bug and should be >> caught. Agreed. If I understand you correctly, by not testing stable kernels with their own selftests, some serious bugs could go undetected. > > Right. I really think stable kernels should be tested with their own > selftests. If some test is needed in a stable kernel it should be > backported to that stable kernel. Correct. This is always a safe option. There might be cases that even prevent tests being built, especially if a new feature adds new fields to an existing structure. It appears in some cases, users want to run newer tests on older kernels. Some tests can clearly detect feature support using module presence and/or Kconfig enabled or disabled. These are conditions even on a kernel that supports a new module or new config option. The kernel the test is running on might not have the feature enabled or module might not be present. In these cases, it would be easier to detect and skip the test. However, some features aren't so easy. For example: - a new flag is added to a syscall, and new test is added. It might not be easy to detect that. - We might have some tests that can't detect and skip. Based on this discussion, it is probably accurate to say: 1. It is recommended that selftests from the same release be run on the kernel. 2. Selftests from newer kernels will run on older kernels, user should understand the risks such as some tests might fail and might not detect feature degradation related bugs. 3. Selftests will fail gracefully on older releases if at all possible. Sumit! 1. What are the reasons for testing older kernel with selftests from newer kernels? What are the benefits you see for doing so? I am looking to understand the need/reasons for this use-case. In our previous discussion on this subject, I did say, you should be able to do so with some exceptions. 2. Do you test kernels with the selftests from the same release? 3. Do you find testing with newer selftests to be useful? thanks, -- Shuah ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 19:06 ` Shuah Khan @ 2017-06-22 19:48 ` Tom Gall 2017-06-22 20:23 ` Shuah Khan 2017-06-23 19:03 ` Shuah Khan 0 siblings, 2 replies; 20+ messages in thread From: Tom Gall @ 2017-06-22 19:48 UTC (permalink / raw) To: shuah Cc: Kees Cook, Andy Lutomirski, Sumit Semwal, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan Hi On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan <shuah@kernel.org> wrote: > On 06/22/2017 11:50 AM, Kees Cook wrote: >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >>>> On 06/22/2017 10:53 AM, Kees Cook wrote: >>>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>> Hi Kees, Andy, >>>>>> >>>>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>>>> feature and test together. >>>>>>> - This one also seems like a security hole being closed, and the >>>>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>>>> from the subsystem developers here. >>>>>> >>>>>> Could you please help us sort this out? Our goal is to help Greg with >>>>>> testing stable kernels, and currently the seccomp tests fail due to >>>>>> missing feature (seccomp ptrace hole closure) getting tested via >>>>>> latest kselftest. >>>>>> >>>>>> If you feel the feature isn't a stable candidate, then could you >>>>>> please help make the test degrade gracefully in its absence? > > In some cases, it is not easy to degrade and/or check for a feature. > Probably several security features could fall in this bucket. > >>>>> >>>>> I don't really want to have that change be a backport -- it's quite >>>>> invasive across multiple architectures. > > Agreed. The same test for kernel applies to tests as well. If a kernel > feature can't be back-ported, the test for that feature will fall in the > same bucket. It shouldn't be back-ported. > >>>>> >>>>> I would say just add a kernel version check to the test. This is >>>>> probably not the only selftest that will need such things. :) >>>> >>>> Adding release checks to selftests is going to problematic for maintenance. >>>> Tests should fail gracefully if feature isn't supported in older kernels. >>>> >>>> Several tests do that now and please find a way to check for dependencies >>>> and feature availability and fail the test gracefully. If there is a test >>>> that can't do that for some reason, we can discuss it, but as a general >>>> rule, I don't want to see kselftest patches that check release. >>> >>> If a future kernel inadvertently loses the new feature and degrades to >>> the behavior of old kernels, that would be a serious bug and should be >>> caught. > > Agreed. If I understand you correctly, by not testing stable kernels > with their own selftests, some serious bugs could go undetected. Personally I'm a bit skeptical. I think the reasoning is more that the latest selftests provide more coverage, and therefore should be better tests, even on older kernels. >> >> Right. I really think stable kernels should be tested with their own >> selftests. If some test is needed in a stable kernel it should be >> backported to that stable kernel. > > Correct. This is always a safe option. There might be cases that even > prevent tests being built, especially if a new feature adds new fields > to an existing structure. > > It appears in some cases, users want to run newer tests on older kernels. > Some tests can clearly detect feature support using module presence and/or > Kconfig enabled or disabled. These are conditions even on a kernel that > supports a new module or new config option. The kernel the test is running > on might not have the feature enabled or module might not be present. In > these cases, it would be easier to detect and skip the test. > > However, some features aren't so easy. For example: > > - a new flag is added to a syscall, and new test is added. It might not > be easy to detect that. > - We might have some tests that can't detect and skip. > > Based on this discussion, it is probably accurate to say: > > 1. It is recommended that selftests from the same release be run on the > kernel. > 2. Selftests from newer kernels will run on older kernels, user should > understand the risks such as some tests might fail and might not > detect feature degradation related bugs. > 3. Selftests will fail gracefully on older releases if at all possible. How about gracefully be skipped instead of fail? The later suggests the test case in some situations can detect it's pointless to run something and say as much instead of emitting a failure that would be a waste of time to look into. As another example take tools/testing/selftests/net/psock_fanout.c On 4.9 it'll fail to compile (using master's selftests) because PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for that symbol and the psock_fanout test will compile and run just fine. > Sumit! > > 1. What are the reasons for testing older kernel with selftests from > newer kernels? What are the benefits you see for doing so? I think the presumption is the latest greatest collection of selftests are the best, most complete. > I am looking to understand the need/reasons for this use-case. In our > previous discussion on this subject, I did say, you should be able to > do so with some exceptions. > > 2. Do you test kernels with the selftests from the same release? We have the ability to do either. The new shiny .... it calls. > 3. Do you find testing with newer selftests to be useful? I think it comes down to coverage and again the current perception that latest greatest is better. Quantitatively we haven't collected data to support that position tho it would be interesting to compare say a 4.4-lts and it's selftests directory to a mainline, see how much was new and then find out how much of those new selftests actually work on the older 4.4-lts. > thanks, > -- Shuah -- Regards, Tom Director, Linaro Mobile Group Linaro.org │ Open source software for ARM SoCs irc: tgall_foo | skype : tom_gall "Where's the kaboom!? There was supposed to be an earth-shattering kaboom!" Marvin Martian ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 19:48 ` Tom Gall @ 2017-06-22 20:23 ` Shuah Khan 2017-06-23 4:02 ` Sumit Semwal 2017-06-23 19:03 ` Shuah Khan 1 sibling, 1 reply; 20+ messages in thread From: Shuah Khan @ 2017-06-22 20:23 UTC (permalink / raw) To: Tom Gall Cc: Kees Cook, Andy Lutomirski, Sumit Semwal, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan, Shuah Khan Hi Tom, On 06/22/2017 01:48 PM, Tom Gall wrote: > Hi > > On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan <shuah@kernel.org> wrote: >> On 06/22/2017 11:50 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >>>>> On 06/22/2017 10:53 AM, Kees Cook wrote: >>>>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>> Hi Kees, Andy, >>>>>>> >>>>>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>>>>> feature and test together. >>>>>>>> - This one also seems like a security hole being closed, and the >>>>>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>>>>> from the subsystem developers here. >>>>>>> >>>>>>> Could you please help us sort this out? Our goal is to help Greg with >>>>>>> testing stable kernels, and currently the seccomp tests fail due to >>>>>>> missing feature (seccomp ptrace hole closure) getting tested via >>>>>>> latest kselftest. >>>>>>> >>>>>>> If you feel the feature isn't a stable candidate, then could you >>>>>>> please help make the test degrade gracefully in its absence? >> >> In some cases, it is not easy to degrade and/or check for a feature. >> Probably several security features could fall in this bucket. >> >>>>>> >>>>>> I don't really want to have that change be a backport -- it's quite >>>>>> invasive across multiple architectures. >> >> Agreed. The same test for kernel applies to tests as well. If a kernel >> feature can't be back-ported, the test for that feature will fall in the >> same bucket. It shouldn't be back-ported. >> >>>>>> >>>>>> I would say just add a kernel version check to the test. This is >>>>>> probably not the only selftest that will need such things. :) >>>>> >>>>> Adding release checks to selftests is going to problematic for maintenance. >>>>> Tests should fail gracefully if feature isn't supported in older kernels. >>>>> >>>>> Several tests do that now and please find a way to check for dependencies >>>>> and feature availability and fail the test gracefully. If there is a test >>>>> that can't do that for some reason, we can discuss it, but as a general >>>>> rule, I don't want to see kselftest patches that check release. >>>> >>>> If a future kernel inadvertently loses the new feature and degrades to >>>> the behavior of old kernels, that would be a serious bug and should be >>>> caught. >> >> Agreed. If I understand you correctly, by not testing stable kernels >> with their own selftests, some serious bugs could go undetected. > > Personally I'm a bit skeptical. I think the reasoning is more that the > latest selftests provide more coverage, and therefore should be better > tests, even on older kernels. The assumption that "the latest selftests provide more coverage, and therefore should be better tests, even on older kernels." is incorrect. Selftests in general track the kernel features. In some cases, new tests could be added that provide better coverage on older kernels, however, it is more likely that new tests are added to test new kernel features and enhancements to existing features. Based on the second "enhancements to existing features" it is more important to test newer kernels with older selftests. This does happen in kernel integration cycles during development. As a general rule, testing stable kernels with their own selftests will yield the best results. > >>> >>> Right. I really think stable kernels should be tested with their own >>> selftests. If some test is needed in a stable kernel it should be >>> backported to that stable kernel. >> >> Correct. This is always a safe option. There might be cases that even >> prevent tests being built, especially if a new feature adds new fields >> to an existing structure. >> >> It appears in some cases, users want to run newer tests on older kernels. >> Some tests can clearly detect feature support using module presence and/or >> Kconfig enabled or disabled. These are conditions even on a kernel that >> supports a new module or new config option. The kernel the test is running >> on might not have the feature enabled or module might not be present. In >> these cases, it would be easier to detect and skip the test. >> >> However, some features aren't so easy. For example: >> >> - a new flag is added to a syscall, and new test is added. It might not >> be easy to detect that. >> - We might have some tests that can't detect and skip. >> >> Based on this discussion, it is probably accurate to say: >> >> 1. It is recommended that selftests from the same release be run on the >> kernel. >> 2. Selftests from newer kernels will run on older kernels, user should >> understand the risks such as some tests might fail and might not >> detect feature degradation related bugs. >> 3. Selftests will fail gracefully on older releases if at all possible. > > How about gracefully be skipped instead of fail? Yes. That is the goal and that is what tests do. Tests do detect dependencies on features, modules, config options and decide to skip the test. If a test doesn't do that, it gets fixed. > > The later suggests the test case in some situations can detect it's > pointless to run something and say as much instead of emitting a > failure that would be a waste of time to look into. Right. Please see above. However, correctly detecting dependencies isn't possible in all cases. In some cases, fail is what it can do. > > As another example take tools/testing/selftests/net/psock_fanout.c > On 4.9 it'll fail to compile (using master's selftests) because > PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for > that symbol and the psock_fanout test will compile and run just fine. > >> Sumit! >> >> 1. What are the reasons for testing older kernel with selftests from >> newer kernels? What are the benefits you see for doing so? > > I think the presumption is the latest greatest collection of selftests > are the best, most complete. Not necessarily the case. > >> I am looking to understand the need/reasons for this use-case. In our >> previous discussion on this subject, I did say, you should be able to >> do so with some exceptions. >> >> 2. Do you test kernels with the selftests from the same release? > > We have the ability to do either. The new shiny .... it calls. If the only reason is "shiny", I would say you might not be getting the best results possible. > >> 3. Do you find testing with newer selftests to be useful? > > I think it comes down to coverage and again the current perception > that latest greatest is better. Quantitatively we haven't collected > data to support that position tho it would be interesting to compare > say a 4.4-lts and it's selftests directory to a mainline, see how much > was new and then find out how much of those new selftests actually > work on the older 4.4-lts. > As I explained above, The assumption/perception that "the latest selftests provide more coverage, and therefore should be better tests, even on older kernels." is incorrect. As per collecting data to see if testing newer selftests provide better coverage or not might or might not be worth while exercise. Some releases might include tests for existing features and some might not. The mix might be different. As a general rule "selftests are intended to track and do track features in their release" is a good assumption. It might be useful to fix tests from newer releases so they "never fail" on older releases might not give us the best ROI as whole. These need to be evaluated case by case basis. I would recommend the following approach based on this discussion and now that we understand incorrect assumption and/or mis-perception to be the basis for choosing to test stable kernels with selftests from new releases. 1. Testing stable kernels with their own selftests will yield the best results. 2. Testing stable kernels with newer selftests could be done if user finds that it provides better coverage, knowing that there is no guarantee that it will. thanks, -- Shuah ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 20:23 ` Shuah Khan @ 2017-06-23 4:02 ` Sumit Semwal 2017-06-23 15:36 ` Shuah Khan 0 siblings, 1 reply; 20+ messages in thread From: Sumit Semwal @ 2017-06-23 4:02 UTC (permalink / raw) To: Shuah Khan Cc: Tom Gall, Kees Cook, Andy Lutomirski, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan Hi Shuah, On 23 June 2017 at 01:53, Shuah Khan <shuah@kernel.org> wrote: > Hi Tom, > > On 06/22/2017 01:48 PM, Tom Gall wrote: >> Hi >> >> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan <shuah@kernel.org> wrote: >>> On 06/22/2017 11:50 AM, Kees Cook wrote: >>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >>>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >>>>>> On 06/22/2017 10:53 AM, Kees Cook wrote: >>>>>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>>> Hi Kees, Andy, >>>>>>>> >>>>>>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>>>>>> feature and test together. >>>>>>>>> - This one also seems like a security hole being closed, and the >>>>>>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>>>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>>>>>> from the subsystem developers here. >>>>>>>> >>>>>>>> Could you please help us sort this out? Our goal is to help Greg with >>>>>>>> testing stable kernels, and currently the seccomp tests fail due to >>>>>>>> missing feature (seccomp ptrace hole closure) getting tested via >>>>>>>> latest kselftest. >>>>>>>> >>>>>>>> If you feel the feature isn't a stable candidate, then could you >>>>>>>> please help make the test degrade gracefully in its absence? >>> >>> In some cases, it is not easy to degrade and/or check for a feature. >>> Probably several security features could fall in this bucket. >>> >>>>>>> >>>>>>> I don't really want to have that change be a backport -- it's quite >>>>>>> invasive across multiple architectures. >>> >>> Agreed. The same test for kernel applies to tests as well. If a kernel >>> feature can't be back-ported, the test for that feature will fall in the >>> same bucket. It shouldn't be back-ported. >>> >>>>>>> >>>>>>> I would say just add a kernel version check to the test. This is >>>>>>> probably not the only selftest that will need such things. :) >>>>>> >>>>>> Adding release checks to selftests is going to problematic for maintenance. >>>>>> Tests should fail gracefully if feature isn't supported in older kernels. >>>>>> >>>>>> Several tests do that now and please find a way to check for dependencies >>>>>> and feature availability and fail the test gracefully. If there is a test >>>>>> that can't do that for some reason, we can discuss it, but as a general >>>>>> rule, I don't want to see kselftest patches that check release. >>>>> >>>>> If a future kernel inadvertently loses the new feature and degrades to >>>>> the behavior of old kernels, that would be a serious bug and should be >>>>> caught. >>> >>> Agreed. If I understand you correctly, by not testing stable kernels >>> with their own selftests, some serious bugs could go undetected. >> >> Personally I'm a bit skeptical. I think the reasoning is more that the >> latest selftests provide more coverage, and therefore should be better >> tests, even on older kernels. > > The assumption that "the latest selftests provide more coverage, and > therefore should be better tests, even on older kernels." is incorrect. > > Selftests in general track the kernel features. In some cases, new > tests could be added that provide better coverage on older kernels, > however, it is more likely that new tests are added to test new kernel > features and enhancements to existing features. Based on the second > "enhancements to existing features" it is more important to test newer > kernels with older selftests. This does happen in kernel integration > cycles during development. > > As a general rule, testing stable kernels with their own selftests will > yield the best results. > I would have agreed totally, if the selftests and the kernel were in sync since forever. But since the kselftests are a comparatively recent addition, the number of tests available for features existing in LTS kernels is really quite small. Just as a comparison, 4.4-LTS misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl, to name a few. Also, while trying to run kselftests from later kernels with 4.4, we only had a few failures for existing features, while most other tests ran ok. Just another data point. >> >>>> >>>> Right. I really think stable kernels should be tested with their own >>>> selftests. If some test is needed in a stable kernel it should be >>>> backported to that stable kernel. >>> >>> Correct. This is always a safe option. There might be cases that even >>> prevent tests being built, especially if a new feature adds new fields >>> to an existing structure. >>> >>> It appears in some cases, users want to run newer tests on older kernels. >>> Some tests can clearly detect feature support using module presence and/or >>> Kconfig enabled or disabled. These are conditions even on a kernel that >>> supports a new module or new config option. The kernel the test is running >>> on might not have the feature enabled or module might not be present. In >>> these cases, it would be easier to detect and skip the test. >>> >>> However, some features aren't so easy. For example: >>> >>> - a new flag is added to a syscall, and new test is added. It might not >>> be easy to detect that. >>> - We might have some tests that can't detect and skip. >>> >>> Based on this discussion, it is probably accurate to say: >>> >>> 1. It is recommended that selftests from the same release be run on the >>> kernel. >>> 2. Selftests from newer kernels will run on older kernels, user should >>> understand the risks such as some tests might fail and might not >>> detect feature degradation related bugs. >>> 3. Selftests will fail gracefully on older releases if at all possible. >> >> How about gracefully be skipped instead of fail? > > Yes. That is the goal and that is what tests do. Tests do detect > dependencies on features, modules, config options and decide to skip > the test. If a test doesn't do that, it gets fixed. > >> >> The later suggests the test case in some situations can detect it's >> pointless to run something and say as much instead of emitting a >> failure that would be a waste of time to look into. > > Right. Please see above. However, correctly detecting dependencies > isn't possible in all cases. In some cases, fail is what it can do. > >> >> As another example take tools/testing/selftests/net/psock_fanout.c >> On 4.9 it'll fail to compile (using master's selftests) because >> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for >> that symbol and the psock_fanout test will compile and run just fine. >> >>> Sumit! >>> >>> 1. What are the reasons for testing older kernel with selftests from >>> newer kernels? What are the benefits you see for doing so? >> >> I think the presumption is the latest greatest collection of selftests >> are the best, most complete. > > Not necessarily the case. > >> >>> I am looking to understand the need/reasons for this use-case. In our >>> previous discussion on this subject, I did say, you should be able to >>> do so with some exceptions. >>> >>> 2. Do you test kernels with the selftests from the same release? >> >> We have the ability to do either. The new shiny .... it calls. > > If the only reason is "shiny", I would say you might not be getting > the best results possible. > >> >>> 3. Do you find testing with newer selftests to be useful? >> >> I think it comes down to coverage and again the current perception >> that latest greatest is better. Quantitatively we haven't collected >> data to support that position tho it would be interesting to compare >> say a 4.4-lts and it's selftests directory to a mainline, see how much >> was new and then find out how much of those new selftests actually >> work on the older 4.4-lts. >> > > As I explained above, The assumption/perception that "the latest selftests > provide more coverage, and therefore should be better tests, even on older > kernels." is incorrect. > > As per collecting data to see if testing newer selftests provide better > coverage or not might or might not be worth while exercise. Some releases > might include tests for existing features and some might not. The mix might > be different. As a general rule "selftests are intended to track and do track > features in their release" is a good assumption. > > It might be useful to fix tests from newer releases so they "never fail" on > older releases might not give us the best ROI as whole. These need to be > evaluated case by case basis. > > I would recommend the following approach based on this discussion and now > that we understand incorrect assumption and/or mis-perception to be the > basis for choosing to test stable kernels with selftests from new releases. > > 1. Testing stable kernels with their own selftests will yield the best > results. > 2. Testing stable kernels with newer selftests could be done if user finds > that it provides better coverage, knowing that there is no guarantee that > it will. > > thanks, > -- Shuah Best, Sumit. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-23 4:02 ` Sumit Semwal @ 2017-06-23 15:36 ` Shuah Khan 0 siblings, 0 replies; 20+ messages in thread From: Shuah Khan @ 2017-06-23 15:36 UTC (permalink / raw) To: Sumit Semwal Cc: Tom Gall, Kees Cook, Andy Lutomirski, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan, Shuah Khan On 06/22/2017 10:02 PM, Sumit Semwal wrote: > Hi Shuah, > > On 23 June 2017 at 01:53, Shuah Khan <shuah@kernel.org> wrote: >> Hi Tom, >> >> On 06/22/2017 01:48 PM, Tom Gall wrote: >>> Hi >>> >>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan <shuah@kernel.org> wrote: >>>> On 06/22/2017 11:50 AM, Kees Cook wrote: >>>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >>>>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >>>>>>> On 06/22/2017 10:53 AM, Kees Cook wrote: >>>>>>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>>>> Hi Kees, Andy, >>>>>>>>> >>>>>>>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>>>>>>> feature and test together. >>>>>>>>>> - This one also seems like a security hole being closed, and the >>>>>>>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>>>>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>>>>>>> from the subsystem developers here. >>>>>>>>> >>>>>>>>> Could you please help us sort this out? Our goal is to help Greg with >>>>>>>>> testing stable kernels, and currently the seccomp tests fail due to >>>>>>>>> missing feature (seccomp ptrace hole closure) getting tested via >>>>>>>>> latest kselftest. >>>>>>>>> >>>>>>>>> If you feel the feature isn't a stable candidate, then could you >>>>>>>>> please help make the test degrade gracefully in its absence? >>>> >>>> In some cases, it is not easy to degrade and/or check for a feature. >>>> Probably several security features could fall in this bucket. >>>> >>>>>>>> >>>>>>>> I don't really want to have that change be a backport -- it's quite >>>>>>>> invasive across multiple architectures. >>>> >>>> Agreed. The same test for kernel applies to tests as well. If a kernel >>>> feature can't be back-ported, the test for that feature will fall in the >>>> same bucket. It shouldn't be back-ported. >>>> >>>>>>>> >>>>>>>> I would say just add a kernel version check to the test. This is >>>>>>>> probably not the only selftest that will need such things. :) >>>>>>> >>>>>>> Adding release checks to selftests is going to problematic for maintenance. >>>>>>> Tests should fail gracefully if feature isn't supported in older kernels. >>>>>>> >>>>>>> Several tests do that now and please find a way to check for dependencies >>>>>>> and feature availability and fail the test gracefully. If there is a test >>>>>>> that can't do that for some reason, we can discuss it, but as a general >>>>>>> rule, I don't want to see kselftest patches that check release. >>>>>> >>>>>> If a future kernel inadvertently loses the new feature and degrades to >>>>>> the behavior of old kernels, that would be a serious bug and should be >>>>>> caught. >>>> >>>> Agreed. If I understand you correctly, by not testing stable kernels >>>> with their own selftests, some serious bugs could go undetected. >>> >>> Personally I'm a bit skeptical. I think the reasoning is more that the >>> latest selftests provide more coverage, and therefore should be better >>> tests, even on older kernels. >> >> The assumption that "the latest selftests provide more coverage, and >> therefore should be better tests, even on older kernels." is incorrect. >> >> Selftests in general track the kernel features. In some cases, new >> tests could be added that provide better coverage on older kernels, >> however, it is more likely that new tests are added to test new kernel >> features and enhancements to existing features. Based on the second >> "enhancements to existing features" it is more important to test newer >> kernels with older selftests. This does happen in kernel integration >> cycles during development. >> >> As a general rule, testing stable kernels with their own selftests will >> yield the best results. >> > I would have agreed totally, if the selftests and the kernel were in > sync since forever. But since the kselftests are a comparatively > recent addition, the number of tests available for features existing > in LTS kernels is really quite small. Just as a comparison, 4.4-LTS > misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl, > to name a few. Yes. Several tests have been added since 4.4. Unfortunately these new additions don't qualify as candidates for stable releases. > > Also, while trying to run kselftests from later kernels with 4.4, we > only had a few failures for existing features, while most other tests > ran ok. Just another data point. That is good to hear. Let's fix the problems as you find them and make the selftests from newer releases work well (if not perfect) on older kernels. 4.13 is going to see TAP13 changes as well, so if selftests from newer kernels work for you, you will see TAB13 benefits. thanks, -- Shuah ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 19:48 ` Tom Gall 2017-06-22 20:23 ` Shuah Khan @ 2017-06-23 19:03 ` Shuah Khan 2017-06-23 19:44 ` Tom Gall 1 sibling, 1 reply; 20+ messages in thread From: Shuah Khan @ 2017-06-23 19:03 UTC (permalink / raw) To: Tom Gall Cc: Kees Cook, Andy Lutomirski, Sumit Semwal, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan, Shuah Khan On 06/22/2017 01:48 PM, Tom Gall wrote: > Hi > > On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan <shuah@kernel.org> wrote: >> On 06/22/2017 11:50 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >>>>> On 06/22/2017 10:53 AM, Kees Cook wrote: >>>>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>> Hi Kees, Andy, >>>>>>> >>>>>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>>>>> feature and test together. >>>>>>>> - This one also seems like a security hole being closed, and the >>>>>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>>>>> from the subsystem developers here. >>>>>>> >>>>>>> Could you please help us sort this out? Our goal is to help Greg with >>>>>>> testing stable kernels, and currently the seccomp tests fail due to >>>>>>> missing feature (seccomp ptrace hole closure) getting tested via >>>>>>> latest kselftest. >>>>>>> >>>>>>> If you feel the feature isn't a stable candidate, then could you >>>>>>> please help make the test degrade gracefully in its absence? >> >> In some cases, it is not easy to degrade and/or check for a feature. >> Probably several security features could fall in this bucket. >> >>>>>> >>>>>> I don't really want to have that change be a backport -- it's quite >>>>>> invasive across multiple architectures. >> >> Agreed. The same test for kernel applies to tests as well. If a kernel >> feature can't be back-ported, the test for that feature will fall in the >> same bucket. It shouldn't be back-ported. >> >>>>>> >>>>>> I would say just add a kernel version check to the test. This is >>>>>> probably not the only selftest that will need such things. :) >>>>> >>>>> Adding release checks to selftests is going to problematic for maintenance. >>>>> Tests should fail gracefully if feature isn't supported in older kernels. >>>>> >>>>> Several tests do that now and please find a way to check for dependencies >>>>> and feature availability and fail the test gracefully. If there is a test >>>>> that can't do that for some reason, we can discuss it, but as a general >>>>> rule, I don't want to see kselftest patches that check release. >>>> >>>> If a future kernel inadvertently loses the new feature and degrades to >>>> the behavior of old kernels, that would be a serious bug and should be >>>> caught. >> >> Agreed. If I understand you correctly, by not testing stable kernels >> with their own selftests, some serious bugs could go undetected. > > Personally I'm a bit skeptical. I think the reasoning is more that the > latest selftests provide more coverage, and therefore should be better > tests, even on older kernels. > >>> >>> Right. I really think stable kernels should be tested with their own >>> selftests. If some test is needed in a stable kernel it should be >>> backported to that stable kernel. >> >> Correct. This is always a safe option. There might be cases that even >> prevent tests being built, especially if a new feature adds new fields >> to an existing structure. >> >> It appears in some cases, users want to run newer tests on older kernels. >> Some tests can clearly detect feature support using module presence and/or >> Kconfig enabled or disabled. These are conditions even on a kernel that >> supports a new module or new config option. The kernel the test is running >> on might not have the feature enabled or module might not be present. In >> these cases, it would be easier to detect and skip the test. >> >> However, some features aren't so easy. For example: >> >> - a new flag is added to a syscall, and new test is added. It might not >> be easy to detect that. >> - We might have some tests that can't detect and skip. >> >> Based on this discussion, it is probably accurate to say: >> >> 1. It is recommended that selftests from the same release be run on the >> kernel. >> 2. Selftests from newer kernels will run on older kernels, user should >> understand the risks such as some tests might fail and might not >> detect feature degradation related bugs. >> 3. Selftests will fail gracefully on older releases if at all possible. > > How about gracefully be skipped instead of fail? > > The later suggests the test case in some situations can detect it's > pointless to run something and say as much instead of emitting a > failure that would be a waste of time to look into. > > As another example take tools/testing/selftests/net/psock_fanout.c > On 4.9 it'll fail to compile (using master's selftests) because > PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for > that symbol and the psock_fanout test will compile and run just fine. Unfortunately adding PACKET_FANOUT_FLAG_UNIQUEID isn't correct. This is a new feature that went into 4.12. You don't want to define this for older kernel which will break on newer kernels. This is one concern I have that in am attempt to fix mainline selftest to be able to test older kernels will cause problems. This is a good example of a new test that has dependency on a new define in an include file which will be hard to check - compile will fail on older kernels. The rest of the net tests will compile and run. -- Shuah > >> Sumit! >> >> 1. What are the reasons for testing older kernel with selftests from >> newer kernels? What are the benefits you see for doing so? > > I think the presumption is the latest greatest collection of selftests > are the best, most complete. > >> I am looking to understand the need/reasons for this use-case. In our >> previous discussion on this subject, I did say, you should be able to >> do so with some exceptions. >> >> 2. Do you test kernels with the selftests from the same release? > > We have the ability to do either. The new shiny .... it calls. > >> 3. Do you find testing with newer selftests to be useful? > > I think it comes down to coverage and again the current perception > that latest greatest is better. Quantitatively we haven't collected > data to support that position tho it would be interesting to compare > say a 4.4-lts and it's selftests directory to a mainline, see how much > was new and then find out how much of those new selftests actually > work on the older 4.4-lts. > >> thanks, >> -- Shuah > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-23 19:03 ` Shuah Khan @ 2017-06-23 19:44 ` Tom Gall 0 siblings, 0 replies; 20+ messages in thread From: Tom Gall @ 2017-06-23 19:44 UTC (permalink / raw) To: shuah Cc: Kees Cook, Andy Lutomirski, Sumit Semwal, Brian Norris, Luis R. Rodriguez, Greg Kroah-Hartman, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan Hi Shuah, On Fri, Jun 23, 2017 at 2:03 PM, Shuah Khan <shuah@kernel.org> wrote: > On 06/22/2017 01:48 PM, Tom Gall wrote: >> Hi >> >> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan <shuah@kernel.org> wrote: >>> On 06/22/2017 11:50 AM, Kees Cook wrote: >>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >>>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >>>>>> On 06/22/2017 10:53 AM, Kees Cook wrote: >>>>>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>>> Hi Kees, Andy, >>>>>>>> >>>>>>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>>>>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>>>>>> feature and test together. >>>>>>>>> - This one also seems like a security hole being closed, and the >>>>>>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>>>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>>>>>> from the subsystem developers here. >>>>>>>> >>>>>>>> Could you please help us sort this out? Our goal is to help Greg with >>>>>>>> testing stable kernels, and currently the seccomp tests fail due to >>>>>>>> missing feature (seccomp ptrace hole closure) getting tested via >>>>>>>> latest kselftest. >>>>>>>> >>>>>>>> If you feel the feature isn't a stable candidate, then could you >>>>>>>> please help make the test degrade gracefully in its absence? >>> >>> In some cases, it is not easy to degrade and/or check for a feature. >>> Probably several security features could fall in this bucket. >>> >>>>>>> >>>>>>> I don't really want to have that change be a backport -- it's quite >>>>>>> invasive across multiple architectures. >>> >>> Agreed. The same test for kernel applies to tests as well. If a kernel >>> feature can't be back-ported, the test for that feature will fall in the >>> same bucket. It shouldn't be back-ported. >>> >>>>>>> >>>>>>> I would say just add a kernel version check to the test. This is >>>>>>> probably not the only selftest that will need such things. :) >>>>>> >>>>>> Adding release checks to selftests is going to problematic for maintenance. >>>>>> Tests should fail gracefully if feature isn't supported in older kernels. >>>>>> >>>>>> Several tests do that now and please find a way to check for dependencies >>>>>> and feature availability and fail the test gracefully. If there is a test >>>>>> that can't do that for some reason, we can discuss it, but as a general >>>>>> rule, I don't want to see kselftest patches that check release. >>>>> >>>>> If a future kernel inadvertently loses the new feature and degrades to >>>>> the behavior of old kernels, that would be a serious bug and should be >>>>> caught. >>> >>> Agreed. If I understand you correctly, by not testing stable kernels >>> with their own selftests, some serious bugs could go undetected. >> >> Personally I'm a bit skeptical. I think the reasoning is more that the >> latest selftests provide more coverage, and therefore should be better >> tests, even on older kernels. >> >>>> >>>> Right. I really think stable kernels should be tested with their own >>>> selftests. If some test is needed in a stable kernel it should be >>>> backported to that stable kernel. >>> >>> Correct. This is always a safe option. There might be cases that even >>> prevent tests being built, especially if a new feature adds new fields >>> to an existing structure. >>> >>> It appears in some cases, users want to run newer tests on older kernels. >>> Some tests can clearly detect feature support using module presence and/or >>> Kconfig enabled or disabled. These are conditions even on a kernel that >>> supports a new module or new config option. The kernel the test is running >>> on might not have the feature enabled or module might not be present. In >>> these cases, it would be easier to detect and skip the test. >>> >>> However, some features aren't so easy. For example: >>> >>> - a new flag is added to a syscall, and new test is added. It might not >>> be easy to detect that. >>> - We might have some tests that can't detect and skip. >>> >>> Based on this discussion, it is probably accurate to say: >>> >>> 1. It is recommended that selftests from the same release be run on the >>> kernel. >>> 2. Selftests from newer kernels will run on older kernels, user should >>> understand the risks such as some tests might fail and might not >>> detect feature degradation related bugs. >>> 3. Selftests will fail gracefully on older releases if at all possible. >> >> How about gracefully be skipped instead of fail? >> >> The later suggests the test case in some situations can detect it's >> pointless to run something and say as much instead of emitting a >> failure that would be a waste of time to look into. >> >> As another example take tools/testing/selftests/net/psock_fanout.c >> On 4.9 it'll fail to compile (using master's selftests) because >> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for >> that symbol and the psock_fanout test will compile and run just fine. > > Unfortunately adding PACKET_FANOUT_FLAG_UNIQUEID isn't correct. This is > a new feature that went into 4.12. You don't want to define this for older > kernel which will break on newer kernels. My thought here wasn't to define the symbol, but to detect it. If it's not defined, you know it's not an available feature and thus avoid that portion of the test. > This is one concern I have that in am attempt to fix mainline selftest to > be able to test older kernels will cause problems. As Greg had mentioned, if we stay focused on features and function in kernel, many tests shouldn't care what version of the kernel they are on. Likewise those tests that are for new function need to be compartmentalized to be separate and not be mixed in in such a way that a compile time failure cause parts of a test to not be built that would be perfectly valid should the feature not be available. > This is a good example of a new test that has dependency on a new define > in an include file which will be hard to check - compile will fail on > older kernels. The rest of the net tests will compile and run. To me seeing a bunch of preventable ugly compile errors which don't give guidance if they are stemming from invalid test cases seems like something we should address. That sorta suggests there's some kind of meta that decides to either build or not build a test based on feature existence. And I really mean existence not kernel version. That suggests the following rule, sefltests should always either successfully build or not be built at all. Anyway, I'm going to shut up until I have patches. > -- Shuah > >> >>> Sumit! >>> >>> 1. What are the reasons for testing older kernel with selftests from >>> newer kernels? What are the benefits you see for doing so? >> >> I think the presumption is the latest greatest collection of selftests >> are the best, most complete. >> >>> I am looking to understand the need/reasons for this use-case. In our >>> previous discussion on this subject, I did say, you should be able to >>> do so with some exceptions. >>> >>> 2. Do you test kernels with the selftests from the same release? >> >> We have the ability to do either. The new shiny .... it calls. >> >>> 3. Do you find testing with newer selftests to be useful? >> >> I think it comes down to coverage and again the current perception >> that latest greatest is better. Quantitatively we haven't collected >> data to support that position tho it would be interesting to compare >> say a 4.4-lts and it's selftests directory to a mainline, see how much >> was new and then find out how much of those new selftests actually >> work on the older 4.4-lts. >> >>> thanks, >>> -- Shuah >> >> > -- Regards, Tom Director, Linaro Mobile Group Linaro.org │ Open source software for ARM SoCs irc: tgall_foo | skype : tom_gall "Where's the kaboom!? There was supposed to be an earth-shattering kaboom!" Marvin Martian ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-22 17:50 ` Kees Cook 2017-06-22 19:06 ` Shuah Khan @ 2017-06-23 1:52 ` Greg Kroah-Hartman 2017-06-23 2:40 ` Andy Lutomirski 1 sibling, 1 reply; 20+ messages in thread From: Greg Kroah-Hartman @ 2017-06-23 1:52 UTC (permalink / raw) To: Kees Cook Cc: Andy Lutomirski, Shuah Khan, Sumit Semwal, Brian Norris, Luis R. Rodriguez, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote: > On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: > > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: > >> On 06/22/2017 10:53 AM, Kees Cook wrote: > >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: > >>>> Hi Kees, Andy, > >>>> > >>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: > >>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - > >>>>> feature and test together. > >>>>> - This one also seems like a security hole being closed, and the > >>>>> 'feature' could be a candidate for stable backports, but Arnd tried > >>>>> that, and it was quite non-trivial. So perhaps we'll need some help > >>>>> from the subsystem developers here. > >>>> > >>>> Could you please help us sort this out? Our goal is to help Greg with > >>>> testing stable kernels, and currently the seccomp tests fail due to > >>>> missing feature (seccomp ptrace hole closure) getting tested via > >>>> latest kselftest. > >>>> > >>>> If you feel the feature isn't a stable candidate, then could you > >>>> please help make the test degrade gracefully in its absence? > >>> > >>> I don't really want to have that change be a backport -- it's quite > >>> invasive across multiple architectures. > >>> > >>> I would say just add a kernel version check to the test. This is > >>> probably not the only selftest that will need such things. :) > >> > >> Adding release checks to selftests is going to problematic for maintenance. > >> Tests should fail gracefully if feature isn't supported in older kernels. > >> > >> Several tests do that now and please find a way to check for dependencies > >> and feature availability and fail the test gracefully. If there is a test > >> that can't do that for some reason, we can discuss it, but as a general > >> rule, I don't want to see kselftest patches that check release. > > > > If a future kernel inadvertently loses the new feature and degrades to > > the behavior of old kernels, that would be a serious bug and should be > > caught. > > Right. I really think stable kernels should be tested with their own > selftests. If some test is needed in a stable kernel it should be > backported to that stable kernel. Well, ideally all new features added to the kernel should be able to be detected by userspace somehow if they are present or not. How do you expect a program to know if a feature has "failed" or is just "not enabled/present in this kernel"? Normally with syscalls this is easy, same for sysfs changes. Is seccomp in the bad state where there is no way to detect the two different states here? How is userspace supposed to deal with that? We make fun of glibc having a zillion crazy tests to determine kernel features, and recently, just not wrapping new syscalls at all because they are just frustrated at the compatibility issues over time. Let's not make their life any harder than it has to be please. I don't see how any of the kselftest programs are any different than any other userspace program that wants to use our kernel api, and as such, any version of kselftest should be able to successfully run on any kernel release. If not, then we messed up in how we either wrote the test, or how we added a new kernel api. Neither is acceptable. thanks, greg k-h ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-23 1:52 ` Greg Kroah-Hartman @ 2017-06-23 2:40 ` Andy Lutomirski 2017-06-23 4:05 ` Kees Cook ` (2 more replies) 0 siblings, 3 replies; 20+ messages in thread From: Andy Lutomirski @ 2017-06-23 2:40 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Kees Cook, Andy Lutomirski, Shuah Khan, Sumit Semwal, Brian Norris, Luis R. Rodriguez, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote: >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >> >> On 06/22/2017 10:53 AM, Kees Cook wrote: >> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >> >>>> Hi Kees, Andy, >> >>>> >> >>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >> >>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >> >>>>> feature and test together. >> >>>>> - This one also seems like a security hole being closed, and the >> >>>>> 'feature' could be a candidate for stable backports, but Arnd tried >> >>>>> that, and it was quite non-trivial. So perhaps we'll need some help >> >>>>> from the subsystem developers here. >> >>>> >> >>>> Could you please help us sort this out? Our goal is to help Greg with >> >>>> testing stable kernels, and currently the seccomp tests fail due to >> >>>> missing feature (seccomp ptrace hole closure) getting tested via >> >>>> latest kselftest. >> >>>> >> >>>> If you feel the feature isn't a stable candidate, then could you >> >>>> please help make the test degrade gracefully in its absence? >> >>> >> >>> I don't really want to have that change be a backport -- it's quite >> >>> invasive across multiple architectures. >> >>> >> >>> I would say just add a kernel version check to the test. This is >> >>> probably not the only selftest that will need such things. :) >> >> >> >> Adding release checks to selftests is going to problematic for maintenance. >> >> Tests should fail gracefully if feature isn't supported in older kernels. >> >> >> >> Several tests do that now and please find a way to check for dependencies >> >> and feature availability and fail the test gracefully. If there is a test >> >> that can't do that for some reason, we can discuss it, but as a general >> >> rule, I don't want to see kselftest patches that check release. >> > >> > If a future kernel inadvertently loses the new feature and degrades to >> > the behavior of old kernels, that would be a serious bug and should be >> > caught. >> >> Right. I really think stable kernels should be tested with their own >> selftests. If some test is needed in a stable kernel it should be >> backported to that stable kernel. > > Well, ideally all new features added to the kernel should be able to be > detected by userspace somehow if they are present or not. > > How do you expect a program to know if a feature has "failed" or is just > "not enabled/present in this kernel"? Normally with syscalls this is > easy, same for sysfs changes. Is seccomp in the bad state where there > is no way to detect the two different states here? How is userspace > supposed to deal with that? > > We make fun of glibc having a zillion crazy tests to determine kernel > features, and recently, just not wrapping new syscalls at all because > they are just frustrated at the compatibility issues over time. Let's > not make their life any harder than it has to be please. > > I don't see how any of the kselftest programs are any different than any > other userspace program that wants to use our kernel api, and as such, > any version of kselftest should be able to successfully run on any > kernel release. If not, then we messed up in how we either wrote the > test, or how we added a new kernel api. Neither is acceptable. That's a fair point. We could add SECCOMP_GET_FEATURES that returns a mask of otherwise-difficult-to-probe features. Greg, for context, the issue here is that we made what was arguably a design error in seccomp's interaction with ptrace. After determining that fixing it solved a bunch of problems and didn't break any user programs, we fixed it. There might be new code that relies on the fix being present in the sense that it would be insecure without the fix. The problem is that the fix is moderately intrusive and doesn't seem like a great candidate for backporting, although we could plausibly do it. --Andy ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-23 2:40 ` Andy Lutomirski @ 2017-06-23 4:05 ` Kees Cook 2017-06-24 0:34 ` Luis R. Rodriguez 2017-06-24 4:43 ` Greg Kroah-Hartman 2 siblings, 0 replies; 20+ messages in thread From: Kees Cook @ 2017-06-23 4:05 UTC (permalink / raw) To: Andy Lutomirski Cc: Greg Kroah-Hartman, Shuah Khan, Sumit Semwal, Brian Norris, Luis R. Rodriguez, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Thu, Jun 22, 2017 at 7:40 PM, Andy Lutomirski <luto@kernel.org> wrote: > On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman > <gregkh@linuxfoundation.org> wrote: >> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: >>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: >>> >> On 06/22/2017 10:53 AM, Kees Cook wrote: >>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>> >>>> Hi Kees, Andy, >>> >>>> >>> >>>> On 15 June 2017 at 23:26, Sumit Semwal <sumit.semwal@linaro.org> wrote: >>> >>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>> >>>>> feature and test together. >>> >>>>> - This one also seems like a security hole being closed, and the >>> >>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>> >>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>> >>>>> from the subsystem developers here. >>> >>>> >>> >>>> Could you please help us sort this out? Our goal is to help Greg with >>> >>>> testing stable kernels, and currently the seccomp tests fail due to >>> >>>> missing feature (seccomp ptrace hole closure) getting tested via >>> >>>> latest kselftest. >>> >>>> >>> >>>> If you feel the feature isn't a stable candidate, then could you >>> >>>> please help make the test degrade gracefully in its absence? >>> >>> >>> >>> I don't really want to have that change be a backport -- it's quite >>> >>> invasive across multiple architectures. >>> >>> >>> >>> I would say just add a kernel version check to the test. This is >>> >>> probably not the only selftest that will need such things. :) >>> >> >>> >> Adding release checks to selftests is going to problematic for maintenance. >>> >> Tests should fail gracefully if feature isn't supported in older kernels. >>> >> >>> >> Several tests do that now and please find a way to check for dependencies >>> >> and feature availability and fail the test gracefully. If there is a test >>> >> that can't do that for some reason, we can discuss it, but as a general >>> >> rule, I don't want to see kselftest patches that check release. >>> > >>> > If a future kernel inadvertently loses the new feature and degrades to >>> > the behavior of old kernels, that would be a serious bug and should be >>> > caught. >>> >>> Right. I really think stable kernels should be tested with their own >>> selftests. If some test is needed in a stable kernel it should be >>> backported to that stable kernel. >> >> Well, ideally all new features added to the kernel should be able to be >> detected by userspace somehow if they are present or not. >> >> How do you expect a program to know if a feature has "failed" or is just >> "not enabled/present in this kernel"? Normally with syscalls this is >> easy, same for sysfs changes. Is seccomp in the bad state where there >> is no way to detect the two different states here? How is userspace >> supposed to deal with that? >> >> We make fun of glibc having a zillion crazy tests to determine kernel >> features, and recently, just not wrapping new syscalls at all because >> they are just frustrated at the compatibility issues over time. Let's >> not make their life any harder than it has to be please. >> >> I don't see how any of the kselftest programs are any different than any >> other userspace program that wants to use our kernel api, and as such, >> any version of kselftest should be able to successfully run on any >> kernel release. If not, then we messed up in how we either wrote the >> test, or how we added a new kernel api. Neither is acceptable. > > That's a fair point. We could add SECCOMP_GET_FEATURES that returns a > mask of otherwise-difficult-to-probe features. But that's silly. The self tests includes all kinds of bug fix tests, and adding each of those to some features list seems like crazy overkill. And every API in the kernel is going to do this? > Greg, for context, the issue here is that we made what was arguably a > design error in seccomp's interaction with ptrace. After determining > that fixing it solved a bunch of problems and didn't break any user > programs, we fixed it. There might be new code that relies on the fix > being present in the sense that it would be insecure without the fix. > > The problem is that the fix is moderately intrusive and doesn't seem > like a great candidate for backporting, although we could plausibly do > it. Even if we did all this for seccomp, we're left with the general case. This is not a situation unique to seccomp. Behaviors and bug fix tests are added to selftests over time, and not all of those things are backport-worthy. If a certain test is desired for a stable kernel, we should backport the test. -Kees -- Kees Cook Pixel Security ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-23 2:40 ` Andy Lutomirski 2017-06-23 4:05 ` Kees Cook @ 2017-06-24 0:34 ` Luis R. Rodriguez 2017-06-24 4:45 ` Greg Kroah-Hartman 2017-06-24 4:43 ` Greg Kroah-Hartman 2 siblings, 1 reply; 20+ messages in thread From: Luis R. Rodriguez @ 2017-06-24 0:34 UTC (permalink / raw) To: Andy Lutomirski Cc: Greg Kroah-Hartman, Kees Cook, Shuah Khan, Sumit Semwal, Brian Norris, Jiri Kosina, Luis R. Rodriguez, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote: > On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman > <gregkh@linuxfoundation.org> wrote: > > On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote: > >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski <luto@kernel.org> wrote: > >> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan <shuah@kernel.org> wrote: > >> >> On 06/22/2017 10:53 AM, Kees Cook wrote: > >> Right. I really think stable kernels should be tested with their own > >> selftests. If some test is needed in a stable kernel it should be > >> backported to that stable kernel. > > > > Well, ideally all new features added to the kernel should be able to be > > detected by userspace somehow if they are present or not. > > > > How do you expect a program to know if a feature has "failed" or is just > > "not enabled/present in this kernel"? Normally with syscalls this is > > easy, same for sysfs changes. Is seccomp in the bad state where there > > is no way to detect the two different states here? How is userspace > > supposed to deal with that? > > > > We make fun of glibc having a zillion crazy tests to determine kernel > > features, and recently, just not wrapping new syscalls at all because > > they are just frustrated at the compatibility issues over time. Let's > > not make their life any harder than it has to be please. > > > > I don't see how any of the kselftest programs are any different than any > > other userspace program that wants to use our kernel api, and as such, > > any version of kselftest should be able to successfully run on any > > kernel release. If not, then we messed up in how we either wrote the > > test, or how we added a new kernel api. Neither is acceptable. > > That's a fair point. I agreed with it as well just a few threads ago due to similar issues, however, thinking this over I'm afraid this has some interesting side consequences for fixes and what code goes upstream into kselftest. <-- snip --> > The problem is that the fix is moderately intrusive and doesn't seem > like a great candidate for backporting, although we could plausibly do > it. Such is the case often actually. So taking the position that any kselftest script on linux-next or a future kernel should never break stable implicate that *any* fix going upstream for which there is a respective ksefltest test *must* have a stable upstream fix. Its not obvious to me that everyone is aware of this. What do we do about those cases where we *don't* want a stable fix due to the complexity? Luis ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-24 0:34 ` Luis R. Rodriguez @ 2017-06-24 4:45 ` Greg Kroah-Hartman 2017-06-26 21:44 ` Luis R. Rodriguez 0 siblings, 1 reply; 20+ messages in thread From: Greg Kroah-Hartman @ 2017-06-24 4:45 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Andy Lutomirski, Kees Cook, Shuah Khan, Sumit Semwal, Brian Norris, Jiri Kosina, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote: > So taking the position that any kselftest script on linux-next or a future > kernel should never break stable implicate that *any* fix going upstream for > which there is a respective ksefltest test *must* have a stable upstream fix. What, no! If it's a bug, that kselftest points out, great, it's up to stable or a vendor to backport that fix if they want it. Again, it's just like any other test suite, look at xfstests, no one gets mad when it adds new tests that fail on old kernels, due to bugs there, right? > Its not obvious to me that everyone is aware of this. What do we do about > those cases where we *don't* want a stable fix due to the complexity? That's up to the stable maintainers or the vendors that maintain their own kernel trees. thanks, greg k-h ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-24 4:45 ` Greg Kroah-Hartman @ 2017-06-26 21:44 ` Luis R. Rodriguez 0 siblings, 0 replies; 20+ messages in thread From: Luis R. Rodriguez @ 2017-06-26 21:44 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Luis R. Rodriguez, Andy Lutomirski, Kees Cook, Shuah Khan, Sumit Semwal, Brian Norris, Jiri Kosina, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Sat, Jun 24, 2017 at 06:45:37AM +0200, Greg Kroah-Hartman wrote: > On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote: > > So taking the position that any kselftest script on linux-next or a future > > kernel should never break stable implicate that *any* fix going upstream for > > which there is a respective ksefltest test *must* have a stable upstream fix. > > What, no! If it's a bug, that kselftest points out, great, it's up to > stable or a vendor to backport that fix if they want it. Makes sense! > Again, it's just like any other test suite, look at xfstests, no one > gets mad when it adds new tests that fail on old kernels, due to bugs > there, right? Correct but that means there is an information gap of which test cases fit into this category. An option to run kselftest with the ability to only run tests designed to also include stable would be good, otherwise Sumit Semwal or others would be sending reports or issues for things folks already designed the fix to *not* be a stable candidate. This actually would also prove useful to distributions to ensure to then run kselftest as part of their distribution test suite with all bells and whistles enabled so that they can make a determination of what questionable "fixes" things to fold in somehow. Unless of course we want these discussions to purposely bubble up as an alternative to this kselftest option. It seems rather wasteful though. > > Its not obvious to me that everyone is aware of this. What do we do about > > those cases where we *don't* want a stable fix due to the complexity? > > That's up to the stable maintainers or the vendors that maintain their > own kernel trees. But if the above is already decided having folks send emails about it seems pointless. Luis ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-23 2:40 ` Andy Lutomirski 2017-06-23 4:05 ` Kees Cook 2017-06-24 0:34 ` Luis R. Rodriguez @ 2017-06-24 4:43 ` Greg Kroah-Hartman 2017-07-05 14:59 ` Sumit Semwal 2 siblings, 1 reply; 20+ messages in thread From: Greg Kroah-Hartman @ 2017-06-24 4:43 UTC (permalink / raw) To: Andy Lutomirski Cc: Kees Cook, Shuah Khan, Sumit Semwal, Brian Norris, Luis R. Rodriguez, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote: > Greg, for context, the issue here is that we made what was arguably a > design error in seccomp's interaction with ptrace. After determining > that fixing it solved a bunch of problems and didn't break any user > programs, we fixed it. There might be new code that relies on the fix > being present in the sense that it would be insecure without the fix. > > The problem is that the fix is moderately intrusive and doesn't seem > like a great candidate for backporting, although we could plausibly do > it. That's fine, not all bugfixes that tests are created to find, should be backported. That's up to the stable maintainers, or someone who has a device/vendor tree based on that kernel if they want to do that or not. That has nothing to do with the fact that the test should fail or gracefully degrade. Tests should fail if the action that they are testing fails. They should degrade and not run if the _feature_ they are testing is not present. Yes, sometimes this is hard, like with the seccomp stuff, and will not always work, but that's the rule for our userspace api independant of any testing framework or code. Look at xfstests, no one gets mad when it adds a new test that old kernels fail at. It's up to someone else to either backport the kernel change, if they want it fixed in an old kernel, not to have xfstests just not run it at all! There's nothing different here either. thanks, greg k-h ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] 2017-06-24 4:43 ` Greg Kroah-Hartman @ 2017-07-05 14:59 ` Sumit Semwal 0 siblings, 0 replies; 20+ messages in thread From: Sumit Semwal @ 2017-07-05 14:59 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Andy Lutomirski, Kees Cook, Shuah Khan, Brian Norris, Luis R. Rodriguez, LKML, # 3.4.x, open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan Hi Andy, On 24 June 2017 at 10:13, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote: >> Greg, for context, the issue here is that we made what was arguably a >> design error in seccomp's interaction with ptrace. After determining >> that fixing it solved a bunch of problems and didn't break any user >> programs, we fixed it. There might be new code that relies on the fix >> being present in the sense that it would be insecure without the fix. >> >> The problem is that the fix is moderately intrusive and doesn't seem >> like a great candidate for backporting, although we could plausibly do >> it. > > That's fine, not all bugfixes that tests are created to find, should be > backported. That's up to the stable maintainers, or someone who has a > device/vendor tree based on that kernel if they want to do that or not. > > That has nothing to do with the fact that the test should fail or > gracefully degrade. Tests should fail if the action that they are > testing fails. They should degrade and not run if the _feature_ they > are testing is not present. So, any updates on this yet - getting the seccomp tests to degrade gracefully? I realise you mentioned that the fix could be intrusive; just wanted to know if it was on your radar still. > > Yes, sometimes this is hard, like with the seccomp stuff, and will not > always work, but that's the rule for our userspace api independant of > any testing framework or code. > > Look at xfstests, no one gets mad when it adds a new test that old > kernels fail at. It's up to someone else to either backport the kernel > change, if they want it fixed in an old kernel, not to have xfstests > just not run it at all! There's nothing different here either. > > thanks, > > greg k-h Thanks much, Sumit. ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2017-07-05 15:00 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-06-22 16:18 seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] Sumit Semwal 2017-06-22 16:53 ` Kees Cook 2017-06-22 17:09 ` Shuah Khan 2017-06-22 17:49 ` Andy Lutomirski 2017-06-22 17:50 ` Kees Cook 2017-06-22 19:06 ` Shuah Khan 2017-06-22 19:48 ` Tom Gall 2017-06-22 20:23 ` Shuah Khan 2017-06-23 4:02 ` Sumit Semwal 2017-06-23 15:36 ` Shuah Khan 2017-06-23 19:03 ` Shuah Khan 2017-06-23 19:44 ` Tom Gall 2017-06-23 1:52 ` Greg Kroah-Hartman 2017-06-23 2:40 ` Andy Lutomirski 2017-06-23 4:05 ` Kees Cook 2017-06-24 0:34 ` Luis R. Rodriguez 2017-06-24 4:45 ` Greg Kroah-Hartman 2017-06-26 21:44 ` Luis R. Rodriguez 2017-06-24 4:43 ` Greg Kroah-Hartman 2017-07-05 14:59 ` Sumit Semwal
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).