Re: [PATCH v2] Add kernel config option for fuzz testing.

[Dmitry Vyukov <dvyukov@google.com>]

On Mon, Mar 9, 2020 at 4:39 PM Jiri Slaby <jslaby@suse.com> wrote:
>
> On 08. 03. 20, 7:52, Greg Kroah-Hartman wrote:
> > On Sat, Mar 07, 2020 at 05:28:22PM +0100, Jiri Slaby wrote:
> >> On 07. 03. 20, 14:58, Tetsuo Handa wrote:
> >>> While syzkaller is finding many bugs, sometimes syzkaller examines
> >>> stupid operations. Currently we prevent syzkaller from examining
> >>> stupid operations by blacklisting syscall arguments and/or disabling
> >>> whole functionality using existing kernel config options, but it is
> >>> a whack-a-mole approach. We need cooperation from kernel side [1].
> >>>
> >>> This patch introduces a kernel config option which allows disabling
> >>> only specific operations. This kernel config option should be enabled
> >>> only when building kernels for fuzz testing.
> >>>
> >>> We discussed possibility of disabling specific operations at run-time
> >>> using some lockdown mechanism [2], but conclusion seems that build-time
> >>> control (i.e. kernel config option) fits better for this purpose.
> >>> Since patches for users of this kernel config option will want a lot of
> >>> explanation [3], this patch provides only kernel config option for them.
> >>>
> >>> [1] https://github.com/google/syzkaller/issues/1622
> >>> [2] https://lkml.kernel.org/r/CACdnJutc7OQeoor6WLTh8as10da_CN=crs79v3Fp0mJTaO=+yw@mail.gmail.com
> >>> [3] https://lkml.kernel.org/r/20191216163155.GB2258618@kroah.com
> >>>
> >>> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> >>> Cc: Dmitry Vyukov <dvyukov@google.com>
> >>> ---
> >>>  lib/Kconfig.debug | 10 ++++++++++
> >>>  1 file changed, 10 insertions(+)
> >>>
> >>> Changes since v1:
> >>>   Drop users of this kernel config option.
> >>>   Update patch description.
> >>>
> >>> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> >>> index 53e786e0a604..e360090e24c5 100644
> >>> --- a/lib/Kconfig.debug
> >>> +++ b/lib/Kconfig.debug
> >>> @@ -2208,4 +2208,14 @@ config HYPERV_TESTING
> >>>
> >>>  endmenu # "Kernel Testing and Coverage"
> >>>
> >>> +config KERNEL_BUILT_FOR_FUZZ_TESTING
> >>> +       bool "Build kernel for fuzz testing"
> >>
> >> If we really want to go this way, I wouldn't limit it for fuzz testing
> >> only. Static analyzers, symbolic executors, formal verifiers, etc. --
> >> all of them should avoid the paths.
> >
> > No, anything that just evaluates the code should be fine, we want static
> > analyzers to be processing those code paths.  Just not to run them as
> > root on a live system.
>
> Even static analyzers generate real-world counter-examples in .c. They
> are ran dynamically to check if the issue is real or if it's only a
> shortcoming of static analysis. Klee is one of those and I used to run
> it on the kernel some time ago. Throwing such generated input results in
> the same weird behavior as using fuzzy testing, while it's still not
> fuzzy testing, but accurate testing.


I am all for naming/framing this as a more generic option (good it at
least does not have SYZ in the name :)).

Re making it a single config vs a set of fine-grained configs. I think
making it fine-grained is a proper way to do it, but the point Tetsuo
raised is very real and painful as well -- when a kernel developer
adds another option, they will not go and update configs on all
external testing systems. This problem is also common for "enable all
boot tests that can run on this kernel", or "configure a 'standard'
debug build". Currently doing these things require all of expertise,
sacred knowledge, checking all configs one-by-one as well as checking
every new kernel patch and that needs to be done by everybody doing
any kernel testing.
I wonder if this can be solved by doing fine-grained configs, but also
adding some umbrella uber-config that will select all of the
individual options. Config system allows this, right? With "select" or
"default if" clauses. What would be better: have the umbrella option
select all individual, or all individual default to y if umbrella is
selected?

FTR, some of the things we would like to disable are collected here:
https://github.com/google/syzkaller/issues/1622

Steve, I am not sure if by lockdown you mean the existing lockdown
mechanism, or just something similar in nature. As Tetsuo pointed, the
possibility of using the existing lockdown mechanism for this was
discussed here (and rejected):
https://lore.kernel.org/lkml/CACdnJutc7OQeoor6WLTh8as10da_CN=crs79v3Fp0mJTaO=+yw@mail.gmail.com/