Re: [PATCH v2] ci: run SELinux kernel test suite

[William Roberts <bill.c.roberts@gmail.com>]

On Fri, May 29, 2020 at 8:24 AM Stephen Smalley
<stephen.smalley.work@gmail.com> wrote:
>
> On Sun, May 24, 2020 at 12:18 PM William Roberts
> <bill.c.roberts@gmail.com> wrote:
> >
> > On Fri, May 22, 2020 at 2:40 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> > >
> > > On Thu, May 21, 2020 at 4:12 PM William Roberts
> > > <bill.c.roberts@gmail.com> wrote:
> > > > On Thu, May 21, 2020 at 7:58 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> > > > >
> > > > > On Thu, May 21, 2020 at 2:52 PM Stephen Smalley
> > > > > <stephen.smalley.work@gmail.com> wrote:
> > > [...]
> > > > > > Last I looked, his script builds and installs the userspace code on
> > > > > > top of the Fedora libraries and programs (make LIBDIR=... install...)
> > > > > > and then runs the testsuite.  That was my suggestion.
> > > > >
> > > > > Ah, yes, I can see that line now. Sorry, somehow I missed it before.
> > > > >
> > > > > > While it is the
> > > > > > kernel testsuite, it exercises a lot of SELinux userspace
> > > > > > functionality that isn't tested by the userspace tests.
> > > > >
> > > > > OK, I suppose it's better than nothing...
> > > > >
> > > >
> > > > Stephen pointed out the additional ways userspace gets tested, and
> > > > perhaps my title and description
> > > > of the patch could be better. But the main point is to increase the
> > > > test coverage
> > > > and perform the testing steps we expect are done before a release in
> > > > the CI. We should have
> > > > the testing coverage and the confidence to release userspace from master at any
> > > > point. We also have forward facing proof that tests are being executed
> > > > and we can make sure
> > > > nothing regresses.
> > > >
> > > > My ultimate goal here, is to help make sure that if Petr gets hit by a
> > > > bus, releases will
> > > > move forward without worry and without any change in quality among the various
> > > > maintainers.
> > > >
> > > > Additionally, we pick up some cross project testing and can find other
> > > > surprises.
> > >
> > > Ah, so you want to move an integration test, which we would normally
> > > run only before release, down to per-commit testing, which is fine
> > > because it is quite fast... OK, it started to make sense to me now :)
> >
> > Exactly, plus we pick up a little more test coverage on the userspace bits
> > by swapping out what's installed in the VM with the current build and running
> > the tests. The speed is less important, it's just fast enough where our CI isn't
> > going to take years to complete. CI doesn't need to be super snappy per se,
> > but it also cannot take a fortnight.
> >
> > >
> > > If I find time I'll have a closer look at the scripts. I already see
> > > some tiny possible improvements... I have Nicolas's last comments
> > addressed and staged, so ill wait a few days and see what you come
> > back with and re-send a V3.
>
> This is looking good to me.  Only questions I have are:
> 1) Have you confirmed that a testsuite failure within the VM gets
> correctly propagated up and treated as a failure by travis-ci itself?

Yes, when I was testing and didn't have SCTP support, the test suite
would fail and propagate back.

> 2) Have you seen any problems with instability in running of the tests
> due to the additional complexity and time?  I've certainly already
> seen instances where travis-ci of selinux or selinux-testsuite fails
> randomly due to timeouts or something when downloading external
> components.

No, and I did a bunch of additional runs, like 10-12 and never saw additional
failures. I see some of those intermittent travis CI failures with all
my projects
on Travis, but I don't see this as adding any more issues. Timeouts are caused
by nothing sent to stdout for 10 mins (IIRC), so we should be good there.

Im just waiting on Ondrej, he said he had some feedback and I was gonna send
round 3. If I don't here anything back on Monday ill send round 3.