Re: [PATCH linux-kselftest/test v1] apparmor: add AppArmor KUnit tests for policy unpack

From: Brendan Higgins <brendanhiggins@google.com>
To: Iurii Zaikin <yzaikin@google.com>
Cc: Kees Cook <keescook@chromium.org>,
	Luis Chamberlain <mcgrof@kernel.org>,
	Alan Maguire <alan.maguire@oracle.com>,
	Matthias Maennich <maennich@google.com>, shuah <shuah@kernel.org>,
	John Johansen <john.johansen@canonical.com>,
	jmorris@namei.org, serge@hallyn.com,
	David Gow <davidgow@google.com>, "Theodore Ts'o" <tytso@mit.edu>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	linux-security-module@vger.kernel.org,
	KUnit Development <kunit-dev@googlegroups.com>,
	"open list:KERNEL SELFTEST FRAMEWORK" 
	<linux-kselftest@vger.kernel.org>,
	Mike Salvatore <mike.salvatore@canonical.com>
Subject: Re: [PATCH linux-kselftest/test v1] apparmor: add AppArmor KUnit tests for policy unpack
Date: Thu, 31 Oct 2019 02:33:32 -0700	[thread overview]
Message-ID: <CAFd5g446cyijzgap9r8nm_202zkUsfdZXrn5E1_Mfe-R+eFb_g@mail.gmail.com> (raw)
In-Reply-To: <CAAXuY3o31iCJwZ+WGHMaK1MgpC0qv=JkJWnzv8Lhym9TnZQvcQ@mail.gmail.com>

On Wed, Oct 30, 2019 at 1:12 PM Iurii Zaikin <yzaikin@google.com> wrote:
>
> > Why can't unit tests live with the code they're testing? They're already
> > logically tied together; what's the harm there? This needn't be the case
> > for ALL tests, etc. The test driver could still live externally. The
> > test in the other .c would just have exported functions... ?
> >
> Curiously enough, this approach has been adopted by D 2.0 where unittests are
> members of the class under test:  https://digitalmars.com/d/2.0/unittest.html

Thanks for pointing this out, Iurii, that actually looks pretty cool.
I still personally prefer keeping tests and code separate, but if we
decide to go the route of mixing tests and code, maybe we might want
to use this as a model.

> but such approach is not mainstream.
> I personally like the idea of testing the lowest level bits in isolation even if
> they are not a part of any interface. I think that specifying the
> interface using
> unit tests and ensuring implementation correctness are complementary but
> I haven't had much luck arguing this with our esteemed colleagues.

So I think this is a very subtle point which is very widely
misunderstood. Most people write code and then write their tests,
following this practice along with only testing public interfaces
often causes people to just not test all of their code, which is
wrong.

The idea of only testing public interfaces is supposed to make people
think more carefully about what the composite layers of the program
is. If you are having difficulty getting decent coverage by only
testing your public interfaces, then it likely tells you that you have
one of two problems:

1) You have code that you don't need, and you should remove it.

2) One of the layers in your program is too think, and you should
introduce a new layer with a new public interface that you can test
through.

I think the second point here is problematic with how C is written in
the kernel. We don't really have any concept of public vs. private
inside the kernel outside of static vs. not static, which is much more
restricted.