Re: [PATCH] Documentation: dev-tools: Add Testing Overview

From: Marco Elver <elver@google.com>
To: Daniel Latypov <dlatypov@google.com>
Cc: David Gow <davidgow@google.com>, Jonathan Corbet <corbet@lwn.net>,
	Shuah Khan <skhan@linuxfoundation.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Dmitry Vyukov <dvyukov@google.com>,
	Brendan Higgins <brendanhiggins@google.com>,
	"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
	KUnit Development <kunit-dev@googlegroups.com>,
	"open list:KERNEL SELFTEST FRAMEWORK" 
	<linux-kselftest@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] Documentation: dev-tools: Add Testing Overview
Date: Mon, 12 Apr 2021 12:43:09 +0200	[thread overview]
Message-ID: <CANpmjNN0sZ1ZrvidqbYaabre_uL14dR0es0Y9Aogq3qyJ9aO2Q@mail.gmail.com> (raw)
In-Reply-To: <CAGS_qxo=BCVfePBcN8NWSAb=htf4pdybcvQuGf2aaWMvfiQMsA@mail.gmail.com>

On Sat, 10 Apr 2021 at 13:53, Daniel Latypov <dlatypov@google.com> wrote:
> On Sat, Apr 10, 2021 at 12:05 AM David Gow <davidgow@google.com> wrote:
[...]
> > +
> > +
> > +Sanitizers
> > +==========
> > +

The "sanitizers" have originally been a group of tools that relied on
compiler instrumentation to perform various dynamic analysis
(initially ASan, TSan, MSan for user space). The term "sanitizer" has
since been broadened to include a few non-compiler based tools such as
GWP-ASan in user space, of which KFENCE is its kernel cousin but it
doesn't have "sanitizer" in its name (because we felt GWP-KASAN was
pushing it with the acronyms ;-)). Also, these days we have HW_TAGS
based KASAN, which doesn't rely on compiler instrumentation but
instead on MTE in Arm64.

Things like kmemleak have never really been called a sanitizer, but
they _are_ dynamic analysis tools.

So to avoid confusion, in particular avoid establishing "sanitizers"
to be synonymous with "dynamic analysis" ("all sanitizers are dynamic
analysis tools, but not all dynamic analysis tools are sanitizers"),
the section here should not be called "Sanitizers" but "Dynamic
Analysis Tools". We could have a subsection "Sanitizers", but I think
it's not necessary.

> > +The kernel also supports a number of sanitizers, which attempt to detect
> > +classes of issues when the occur in a running kernel. These typically
>
> *they occur
>
> > +look for undefined behaviour of some kind, such as invalid memory accesses,
> > +concurrency issues such as data races, or other undefined behaviour like
> > +integer overflows.
> > +
> > +* :doc:`kmemleak` (Kmemleak) detects possible memory leaks.
> > +* :doc:`kasan` detects invalid memory accesses such as out-of-bounds and
> > +  use-after-free errors.
> > +* :doc:`ubsan` detects behaviour that is undefined by the C standard, like
> > +  integer overflows.
> > +* :doc:`kcsan` detects data races.
> > +* :doc:`kfence` is a low-overhead detector of memory issues, which is much
> > +  faster than KASAN and can be used in production.
>
> Hmm, it lives elsewhere, but would also calling out lockdep here be useful?
> I've also not heard anyone call it a sanitizer before, but it fits the
> definition you've given.
>
> Now that I think about it, I've never looked for documentation on it,
> is this the best page?
> https://www.kernel.org/doc/html/latest/locking/lockdep-design.html

Not a "sanitizer" but our sanitizers are all dynamic analysis tools,
and lockdep is also a dynamic analysis tool.

If we want to be pedantic, the kernel has numerous options to add
"instrumentation" (compiler based or explicit) that will detect some
kind of error at runtime. Most of them live in lib/Kconfig.debug. I
think mentioning something like that is in scope of this document, but
we certainly can't mention all debug tools the kernel has to offer.
Mentioning the big ones like above and then referring to
lib/Kconfig.debug is probably fine.

Dmitry recently gave an excellent talk on some of this:
https://www.youtube.com/watch?v=ufcyOkgFZ2Q

Thanks,
-- Marco