Linux backports CII badge and run time testing

Linux backports CII badge and run time testing

[Luis R. Rodriguez]

As per the Core Infrastructure Initiative guidelines we now meet the
requirements for a badge, the details of the submission is here:

https://bestpractices.coreinfrastructure.org/projects/111

A lot of it just required updating our documentation to enable folks
to report security issues. If there are things that need to be
adjusted please let me know. A lot of this follows the Linux kernel
submission for a lot of things:

https://bestpractices.coreinfrastructure.org/projects/34

Things we need to improve on though are automated tests specific to
backports against a series of kernels, and also providing a bit of a
description when we make new releases. I realize that is hard, but its
also hard for Linux, we have no reasons no to be able to do that as
well.

As for run time testing, we know folks out there in the industry
already use backports and do their own run time tests against drivers,
and this may be automated, we however need something more, at the very
least a boot.

Looking for something minimal to start off, how about this:

https://github.com/michal42/qemu-boot-test

This doesn't yet load modules but it should be possible to add support
for it. I suppose we'd need to backport some qemu related drivers, and
call it a day to start off with? Anyone oppose to that? Any other
ideas? I know we discussed the possibility of using 0-day somehow, but
0-day builds full kernels, not packages that provide modules, and our
issues also is we would need coverage of testing over a series of
kernels.

If you have further information about how you use backports to do any
regular run time testing, information on that would be appreciated.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: Linux backports CII badge and run time testing

[Jouni Malinen]

On Thu, May 5, 2016 at 1:34 AM, Luis R. Rodriguez
<mcgrof@do-not-panic.com> wrote:

> As for run time testing, we know folks out there in the industry
> already use backports and do their own run time tests against drivers,
> and this may be automated, we however need something more, at the very
> least a boot.

All the automated wpa_supplicant/hostapd testing with mac80211_hwsim
on my server use Backports: http://buildbot.w1.fi/hwsim/

The current model does not focus on testing Backports, so I'm only
updating that manually every now and then while hostap.git updates are
automated (that being the main focus for testing). That said, it would
be trivial to update Backports to the latest snapshot whenever running
the test. In fact, the same server is already generating snapshot
builds of Backports from wireless-testing.git daily.

This is all with a single base kernel version, though, so if you want
more coverage in that front, you'd want to run the same setup against
multiple kernels.

- Jouni
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: Linux backports CII badge and run time testing

[Hauke Mehrtens]

Hi Luis,

Did you apply for this? What are the benefits of this?

On 05/05/2016 12:34 AM, Luis R. Rodriguez wrote:
> As per the Core Infrastructure Initiative guidelines we now meet the
> requirements for a badge, the details of the submission is here:
> 
> https://bestpractices.coreinfrastructure.org/projects/111
> 
> A lot of it just required updating our documentation to enable folks
> to report security issues. If there are things that need to be
> adjusted please let me know. A lot of this follows the Linux kernel
> submission for a lot of things:
> 
> https://bestpractices.coreinfrastructure.org/projects/34
> 
> Things we need to improve on though are automated tests specific to
> backports against a series of kernels, and also providing a bit of a
> description when we make new releases. I realize that is hard, but its
> also hard for Linux, we have no reasons no to be able to do that as
> well.

Doing predictable releases cost some constant effort. I do not know if
we have the resources to do predictable releases.

Security support could also be a problem because then we have to react
pretty fast when someone sees a problem in one of the drivers we ship.
Otherwise fixing the security problems should not be a big deal as I
assume that most security problems will be already fixed in upstream
Linux kernel.

> As for run time testing, we know folks out there in the industry
> already use backports and do their own run time tests against drivers,
> and this may be automated, we however need something more, at the very
> least a boot.

Build testing and testing with simulated drivers like mac80211_hwsim
should be doable, but testing all the device drivers which serve real
hardware is nearly impossible.

> Looking for something minimal to start off, how about this:
> 
> https://github.com/michal42/qemu-boot-test

yes that should be doable to have a qemu boot test which also checks if
loading the modules and unloading works.

> This doesn't yet load modules but it should be possible to add support
> for it. I suppose we'd need to backport some qemu related drivers, and
> call it a day to start off with? Anyone oppose to that? Any other
> ideas? I know we discussed the possibility of using 0-day somehow, but
> 0-day builds full kernels, not packages that provide modules, and our
> issues also is we would need coverage of testing over a series of
> kernels.
> 
> If you have further information about how you use backports to do any
> regular run time testing, information on that would be appreciated.

Hauke
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: Linux backports CII badge and run time testing

[Luis R. Rodriguez]

On Thu, May 5, 2016 at 2:46 AM, Jouni Malinen <jkmalinen@gmail.com> wrote:
> On Thu, May 5, 2016 at 1:34 AM, Luis R. Rodriguez
> <mcgrof@do-not-panic.com> wrote:
>
>> As for run time testing, we know folks out there in the industry
>> already use backports and do their own run time tests against drivers,
>> and this may be automated, we however need something more, at the very
>> least a boot.
>
> All the automated wpa_supplicant/hostapd testing with mac80211_hwsim
> on my server use Backports: http://buildbot.w1.fi/hwsim/

Quite impressive work, thanks!

> The current model does not focus on testing Backports, so I'm only
> updating that manually every now and then while hostap.git updates are
> automated (that being the main focus for testing). That said, it would
> be trivial to update Backports to the latest snapshot whenever running
> the test. In fact, the same server is already generating snapshot
> builds of Backports from wireless-testing.git daily.

Interesting...

> This is all with a single base kernel version, though, so if you want
> more coverage in that front, you'd want to run the same setup against
> multiple kernels.

Indeed, this raises the question of "what to test" exactly, given
backports really is a subset of Linux. The automated tests you have
seem more in line for things perhaps we should get 0-day to consider
embracing so that if a regression is introduced linux-wireless
developers are nagged with the respective commit ID and tests cases
(if this is not done already). Upstream and 0-day seems like a much
more suitable place to test daily updates on the 802.11 front.

Backports-wise we should cover at least basic functionality, but
annotating that if an issue is present on backports with the latest
linux-next release it may also exist on upstream linux-next, and as
such not a backports issue but rather an upstream issue. Its this fine
line of overlap we need a way to somehow remove, and only test things
which ensure the backporting works for an array of kernels.

In lack of 0-day integration for wpa_supplicant / hostapd with
mac80211_hwsim (and leaving aside its debate over doing so or not),
intuitively I think its a fair assumption to make that linux-next
mac80211_hwisim should at the very least be able to load and perhaps
run a series of *basic functionality*. If this is agreeable, and
reducing the test cases is indeed easy and possible, I think a series
of basic tests are indeed called for as reasonable for backports to
integrate for testing as part of its own infrastructure specially
given 802.11 is a major stakeholder. In that sense, perhaps we can
cover basic testing for each subsystem, and for 802.11 then testing
mac80211_hwsim with basic functionality would be our litmus test.

Thoughts? If agreeable then perhap we just need something similar for
each subsystem we decide to take on.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: Linux backports CII badge and run time testing

[Luis R. Rodriguez]

On Sat, May 7, 2016 at 4:13 AM, Hauke Mehrtens <hauke@hauke-m.de> wrote:
> Hi Luis,
>
> Did you apply for this? What are the benefits of this?

Yes -- the benefits are that you push the software project under
application under an honest heavy scrutiny with best practices advised
by Core Infrastructure Initiative (CII). That sounds vague but, the
skinny is, to help reduce attack surfaces in FOSS. To understand the
CII intentions one has to understand how it got spawned... The
heartbleed bug was one trigger that made us start questioning -- WTF
!? How can we do better. The Best Practices are an empirical response
lead through a serious n academic evaluation. I applied for this as a
proactive measure to see where we stand as a project. IMHO every FOSS
should apply and strive to meet the criteria for the badge.
Mathematically, if you really are honest, if you get the badge --
we're in a better place in the community.

> On 05/05/2016 12:34 AM, Luis R. Rodriguez wrote:
>> As per the Core Infrastructure Initiative guidelines we now meet the
>> requirements for a badge, the details of the submission is here:
>>
>> https://bestpractices.coreinfrastructure.org/projects/111
>>
>> A lot of it just required updating our documentation to enable folks
>> to report security issues. If there are things that need to be
>> adjusted please let me know. A lot of this follows the Linux kernel
>> submission for a lot of things:
>>
>> https://bestpractices.coreinfrastructure.org/projects/34
>>
>> Things we need to improve on though are automated tests specific to
>> backports against a series of kernels, and also providing a bit of a
>> description when we make new releases. I realize that is hard, but its
>> also hard for Linux, we have no reasons no to be able to do that as
>> well.
>
> Doing predictable releases cost some constant effort. I do not know if
> we have the resources to do predictable releases.

Same can be said about Linux but hey 0-day came around :D

Now granted for us its different as we also have an array of
kernels... so that Linux * number of kernels we support that needs
testing, but as I noted to Jouni, we really only need certain sanity
tests, and with time perhaps we can zero in further.

> Security support could also be a problem because then we have to react
> pretty fast when someone sees a problem in one of the drivers we ship.

You're right but security-wise the main attack surface really is
Linux, so a security issue that *is* part of Linux, belongs upstream
on Linux, and not backports. So for instance, just as Linus has put
down the law on asking stable fixes to go through him first before any
stable kernels, I think we should also just require security / fixes
to go always upstream, and we never carry any delta, best we can do is
wait for upstream integration to a new release of Linux, or a new
stable release.

The still is a small gap in terms of security that we should cover
though, and that is that about 1-2% of actual backport code we mend
Linux for. If a security issues lies in there *we* should have a
policy for letting people privately report and for us to fix in a
timely manner. That's some room for improvement. Given the attack
surface is about 1-2% of code, and based on an empirical evaluation of
our bugs on kernel.org bugzilla -- I don't think this is a huge
burden, correct me if I'm wrong. Thoughts?

That is to say: I don't see this requiring much effort, specially if
we keep trimming older kernels as we did last time,

> Otherwise fixing the security problems should not be a big deal as I
> assume that most security problems will be already fixed in upstream
> Linux kernel.

Indeed.

>> As for run time testing, we know folks out there in the industry
>> already use backports and do their own run time tests against drivers,
>> and this may be automated, we however need something more, at the very
>> least a boot.
>
> Build testing and testing with simulated drivers like mac80211_hwsim
> should be doable, but testing all the device drivers which serve real
> hardware is nearly impossible.

Agreed.

>> Looking for something minimal to start off, how about this:
>>
>> https://github.com/michal42/qemu-boot-test
>
> yes that should be doable to have a qemu boot test which also checks if
> loading the modules and unloading works.

Cool!

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: Linux backports CII badge and run time testing

[Fengguang Wu]

CC more 0-day developers.

On Wed, May 11, 2016 at 04:14:42PM -0700, Luis R. Rodriguez wrote:
> On Thu, May 5, 2016 at 2:46 AM, Jouni Malinen <jkmalinen@gmail.com> wrote:
> > On Thu, May 5, 2016 at 1:34 AM, Luis R. Rodriguez
> > <mcgrof@do-not-panic.com> wrote:
> >
> >> As for run time testing, we know folks out there in the industry
> >> already use backports and do their own run time tests against drivers,
> >> and this may be automated, we however need something more, at the very
> >> least a boot.
> >
> > All the automated wpa_supplicant/hostapd testing with mac80211_hwsim
> > on my server use Backports: http://buildbot.w1.fi/hwsim/
> 
> Quite impressive work, thanks!
> 
> > The current model does not focus on testing Backports, so I'm only
> > updating that manually every now and then while hostap.git updates are
> > automated (that being the main focus for testing). That said, it would
> > be trivial to update Backports to the latest snapshot whenever running
> > the test. In fact, the same server is already generating snapshot
> > builds of Backports from wireless-testing.git daily.
> 
> Interesting...
> 
> > This is all with a single base kernel version, though, so if you want
> > more coverage in that front, you'd want to run the same setup against
> > multiple kernels.
> 
> Indeed, this raises the question of "what to test" exactly, given
> backports really is a subset of Linux. The automated tests you have
> seem more in line for things perhaps we should get 0-day to consider
> embracing so that if a regression is introduced linux-wireless
> developers are nagged with the respective commit ID and tests cases
> (if this is not done already). Upstream and 0-day seems like a much
> more suitable place to test daily updates on the 802.11 front.
> 
> Backports-wise we should cover at least basic functionality, but
> annotating that if an issue is present on backports with the latest
> linux-next release it may also exist on upstream linux-next, and as
> such not a backports issue but rather an upstream issue. Its this fine
> line of overlap we need a way to somehow remove, and only test things
> which ensure the backporting works for an array of kernels.
> 
> In lack of 0-day integration for wpa_supplicant / hostapd with
> mac80211_hwsim (and leaving aside its debate over doing so or not),
> intuitively I think its a fair assumption to make that linux-next
> mac80211_hwisim should at the very least be able to load and perhaps
> run a series of *basic functionality*. If this is agreeable, and
> reducing the test cases is indeed easy and possible, I think a series
> of basic tests are indeed called for as reasonable for backports to
> integrate for testing as part of its own infrastructure specially
> given 802.11 is a major stakeholder. In that sense, perhaps we can
> cover basic testing for each subsystem, and for 802.11 then testing
> mac80211_hwsim with basic functionality would be our litmus test.
> 
> Thoughts? If agreeable then perhap we just need something similar for
> each subsystem we decide to take on.

FYI 0-day runs simple hwsim tests based on

        git://w1.fi/srv/git/hostap.git

The 0-day test scripts are

        https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/tests/hwsim
        https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/pack/hwsim

We are very interested in hearing improvement suggestions on test coverage,
or be offered new test schemes/scripts to run regularly on new/stable kernels!

Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: Linux backports CII badge and run time testing

[Johannes Berg]

On Thu, 2016-05-12 at 08:06 +0800, Fengguang Wu wrote:

> FYI 0-day runs simple hwsim tests based on
> 
>         git://w1.fi/srv/git/hostap.git
> 
> The 0-day test scripts are
> 
>         https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/tests/hwsim
>         https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/pack/hwsim

All of them? Wow. Can we see the results of that somewhere?

johannes

--
To unsubscribe from this list: send the line "unsubscribe backports" in