linux-kselftest.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Mike Salvatore <mike.salvatore@canonical.com>
To: Brendan Higgins <brendanhiggins@google.com>,
	Iurii Zaikin <yzaikin@google.com>
Cc: Kees Cook <keescook@chromium.org>,
	Luis Chamberlain <mcgrof@kernel.org>,
	Alan Maguire <alan.maguire@oracle.com>,
	Matthias Maennich <maennich@google.com>, shuah <shuah@kernel.org>,
	John Johansen <john.johansen@canonical.com>,
	jmorris@namei.org, serge@hallyn.com,
	David Gow <davidgow@google.com>, Theodore Ts'o <tytso@mit.edu>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	linux-security-module@vger.kernel.org,
	KUnit Development <kunit-dev@googlegroups.com>,
	"open list:KERNEL SELFTEST FRAMEWORK" 
	<linux-kselftest@vger.kernel.org>
Subject: Re: [PATCH linux-kselftest/test v1] apparmor: add AppArmor KUnit tests for policy unpack
Date: Tue, 5 Nov 2019 11:43:16 -0500	[thread overview]
Message-ID: <205525ba-dc2f-34a9-b7dc-4421285535d7@canonical.com> (raw)
In-Reply-To: <CAFd5g446cyijzgap9r8nm_202zkUsfdZXrn5E1_Mfe-R+eFb_g@mail.gmail.com>

>> but such approach is not mainstream.
>> I personally like the idea of testing the lowest level bits in isolation even if
>> they are not a part of any interface. I think that specifying the
>> interface using
>> unit tests and ensuring implementation correctness are complementary but
>> I haven't had much luck arguing this with our esteemed colleagues.

In general, testing public interfaces is preferable, however, I think it's
important to avoid becoming dogmatic. IMHO, it's more important to have tests
that are clear in what they test than to not write tests (or write confusing
tests) in order to adhere to a generalized principle.

> So I think this is a very subtle point which is very widely
> misunderstood. Most people write code and then write their tests,
> following this practice along with only testing public interfaces
> often causes people to just not test all of their code, which is
> wrong.

The very nature of this situation is that the code was written before the tests.

> The idea of only testing public interfaces is supposed to make people
> think more carefully about what the composite layers of the program
> is. If you are having difficulty getting decent coverage by only
> testing your public interfaces, then it likely tells you that you have
> one of two problems:
> 
> 1) You have code that you don't need, and you should remove it.
> 
> 2) One of the layers in your program is too think, and you should
> introduce a new layer with a new public interface that you can test
> through.
> 
> I think the second point here is problematic with how C is written in
> the kernel. We don't really have any concept of public vs. private
> inside the kernel outside of static vs. not static, which is much more
> restricted.

I don't think we can expect developers to refactor large portions of complex
kernel code in order to improve its testability. I imagine this will happen
naturally over time, but I think we need to allow for developers to test
"private" code in the meanwhile.

My opinion is that it's more important to have tests than not. As evidence, I
submit the following commit:
https://github.com/torvalds/linux/commit/156e42996bd84eccb6acf319f19ce0cb140d00e3.

While not a major bug, this bug was discovered as a direct result of writing
these unit tests. So, in summary, I see value in "testing the lowest level bits
in isolation", even if it doesn't necessarily represent the Gold Standard in
Unit Testing.


  parent reply	other threads:[~2019-11-05 16:43 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-18  0:18 Brendan Higgins
2019-10-18  0:33 ` Iurii Zaikin
2019-10-30 18:59   ` Kees Cook
2019-11-06  0:35     ` Brendan Higgins
2019-11-06  0:37       ` Brendan Higgins
2019-10-18  0:43 ` Brendan Higgins
2019-10-18 16:25   ` Theodore Y. Ts'o
2019-10-18 21:41     ` Brendan Higgins
2019-10-30 19:02       ` Kees Cook
2019-10-31  9:01         ` Brendan Higgins
2019-10-18 12:29 ` Luis Chamberlain
2019-10-19 12:56   ` Alan Maguire
2019-10-19 18:36     ` Luis Chamberlain
2019-10-24  0:42     ` Brendan Higgins
2019-10-24 10:15       ` Luis Chamberlain
2019-10-30 19:09         ` Kees Cook
2019-10-30 20:11           ` Iurii Zaikin
2019-10-31  1:40             ` John Johansen
2019-10-31  9:33             ` Brendan Higgins
2019-10-31 18:40               ` Kees Cook
2019-11-05 16:43               ` Mike Salvatore [this message]
2019-11-05 23:59                 ` Brendan Higgins
2019-10-31  1:37           ` John Johansen
2019-10-31  9:17           ` Brendan Higgins
2019-11-01 12:30             ` Alan Maguire
2019-11-05 23:44               ` Brendan Higgins

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=205525ba-dc2f-34a9-b7dc-4421285535d7@canonical.com \
    --to=mike.salvatore@canonical.com \
    --cc=alan.maguire@oracle.com \
    --cc=brendanhiggins@google.com \
    --cc=davidgow@google.com \
    --cc=jmorris@namei.org \
    --cc=john.johansen@canonical.com \
    --cc=keescook@chromium.org \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-security-module@vger.kernel.org \
    --cc=maennich@google.com \
    --cc=mcgrof@kernel.org \
    --cc=serge@hallyn.com \
    --cc=shuah@kernel.org \
    --cc=tytso@mit.edu \
    --cc=yzaikin@google.com \
    --subject='Re: [PATCH linux-kselftest/test v1] apparmor: add AppArmor KUnit tests for policy unpack' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).