Re: [PATCH v2] ci: run SELinux kernel test suite

[William Roberts <bill.c.roberts@gmail.com>]

On Thu, May 21, 2020 at 7:58 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
>
> On Thu, May 21, 2020 at 2:52 PM Stephen Smalley
> <stephen.smalley.work@gmail.com> wrote:
> > On Thu, May 21, 2020 at 4:51 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> > >
> > > On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
> > > > From: William Roberts <william.c.roberts@intel.com>
> > > >
> > > > The current Travis CI runs the userspace tooling and libraries against
> > > > policy files, but cannot test against an SELinux enabled kernel. Thus,
> > > > some tests are not being done in the CI. Travis, unfortunately only
> > > > provides Ubuntu images, so in order to run against a modern distro with
> > > > SELinux in enforcing mode, we need to launch a KVM with something like
> > > > Fedora.
> > > >
> > > > This patch enables this support by launching a Fedora32 Cloud Image with
> > > > the SELinux userspace library passed on from the Travis clone, it then
> > > > builds and replaces the current SELinux bits on the Fedora32 image and
> > > > runs the SELinux testsuite.
> > > >
> > > > Signed-off-by: William Roberts <william.c.roberts@intel.com>
> > >
> > > From the text above I infer that this patch is intended against the
> > > userspace repo, right?
> > >
> > > If so, I don't quite see the usefulness of running the
> > > selinux-testsuite on every userspace change... It is mainly intended
> > > for testing the kernel and only a small part of its running time is
> > > spent on running (i.e. testing in a sense) the SELinux userspace
> > > programs. Not to mention that in your patch it runs with the userspace
> > > shipped in Fedora and not the version from the given commit...

Totally doable, and you just copy + edit those scripts to achieve
that. Currently we inject the travis repo
with patches, which is the selinux userspace, into the VM image with
virt-sysprep.
But if this was selinux-testsuite ci you would just virt-sysprep
inject it's repo.

> >
> > Last I looked, his script builds and installs the userspace code on
> > top of the Fedora libraries and programs (make LIBDIR=... install...)
> > and then runs the testsuite.  That was my suggestion.
>
> Ah, yes, I can see that line now. Sorry, somehow I missed it before.
>
> > While it is the
> > kernel testsuite, it exercises a lot of SELinux userspace
> > functionality that isn't tested by the userspace tests.
>
> OK, I suppose it's better than nothing...
>

Stephen pointed out the additional ways userspace gets tested, and
perhaps my title and description
of the patch could be better. But the main point is to increase the
test coverage
and perform the testing steps we expect are done before a release in
the CI. We should have
the testing coverage and the confidence to release userspace from master at any
point. We also have forward facing proof that tests are being executed
and we can make sure
nothing regresses.

My ultimate goal here, is to help make sure that if Petr gets hit by a
bus, releases will
move forward without worry and without any change in quality among the various
maintainers.

Additionally, we pick up some cross project testing and can find other
surprises.