[Announce] LPC 2018: Testing and Fuzzing Microconference

[Announce] LPC 2018: Testing and Fuzzing Microconference

[Dhaval Giani]

Hi folks,

Sasha and I are pleased to announce the Testing and Fuzzing track at
LPC [ 1 ]. We are planning to continue the discussions from last
year's microconference [2]. Many discussions from the Automated
Testing Summit [3] will also continue, and a final agenda will come up
only soon after that.

Suggested Topics

- Syzbot/syzkaller
- ATS
- Distro/stable testing
- kernelci
- kernelci auto bisection
- Unit testing framework

We look forward to other interesting topics for this microconference
as a reply to this email.

Thanks!
Dhaval and Sasha

[1] https://blog.linuxplumbersconf.org/2018/testing-and-fuzzing-mc/
[2] https://lwn.net/Articles/735034/
[3] https://elinux.org/Automated_Testing_Summit

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Liam R. Howlett]

* Dhaval Giani <dhaval.giani@gmail.com> [180919 13:15]:
> Hi folks,
> 
> Sasha and I are pleased to announce the Testing and Fuzzing track at
> LPC [ 1 ]. We are planning to continue the discussions from last
> year's microconference [2]. Many discussions from the Automated
> Testing Summit [3] will also continue, and a final agenda will come up
> only soon after that.
> 
> Suggested Topics
> 
> - Syzbot/syzkaller
> - ATS
> - Distro/stable testing
> - kernelci
> - kernelci auto bisection
> - Unit testing framework
> 
> We look forward to other interesting topics for this microconference
> as a reply to this email.
> 
> Thanks!
> Dhaval and Sasha
> 
> [1] https://blog.linuxplumbersconf.org/2018/testing-and-fuzzing-mc/
> [2] https://lwn.net/Articles/735034/
> [3] https://elinux.org/Automated_Testing_Summit


Hello,

I have a new way to analyze binaries to detect specific calls without
the need for source.  I would like to discuss Machine Code Trace
(MCTrace) at the Testing and Fuzzing LPC track.  MCTrace intercepts the
application prior to execution and does not rely on a specific user
input. It then decodes the machine instructions to follow all control
flows to their natural conclusions.  This includes control flows that go
beyond the boundaries of the static executable code into shared
libraries. This new technique avoids false positives which could be
produced by static analysis and includes paths that could be missed by
dynamic tracing.  This type of analysis could be useful in both testing
and fuzzing by providing a call graph to a given function.

MCTrace was initially designed to help generate the seccomp() filter
list, which is a whitelist/blacklist of system calls for a specific
application. Seccomp filters easily become outdated when the application
or shared library is updated. This can cause failures or security
issues [ 1 ].  Other potential uses including examining binary blobs,
vulnerability analysis, and debugging.

Thank you,
Liam R. Howlett

[1] https://lwn.net/Articles/738750/

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Sasha Levin]

On Tue, Oct 2, 2018 at 4:44 PM Liam R. Howlett <Liam.Howlett@oracle.com> wrote:
>
> * Dhaval Giani <dhaval.giani@gmail.com> [180919 13:15]:
> > Hi folks,
> >
> > Sasha and I are pleased to announce the Testing and Fuzzing track at
> > LPC [ 1 ]. We are planning to continue the discussions from last
> > year's microconference [2]. Many discussions from the Automated
> > Testing Summit [3] will also continue, and a final agenda will come up
> > only soon after that.
> >
> > Suggested Topics
> >
> > - Syzbot/syzkaller
> > - ATS
> > - Distro/stable testing
> > - kernelci
> > - kernelci auto bisection
> > - Unit testing framework
> >
> > We look forward to other interesting topics for this microconference
> > as a reply to this email.
> >
> > Thanks!
> > Dhaval and Sasha
> >
> > [1] https://blog.linuxplumbersconf.org/2018/testing-and-fuzzing-mc/
> > [2] https://lwn.net/Articles/735034/
> > [3] https://elinux.org/Automated_Testing_Summit
>
>
> Hello,
>
> I have a new way to analyze binaries to detect specific calls without
> the need for source.  I would like to discuss Machine Code Trace
> (MCTrace) at the Testing and Fuzzing LPC track.  MCTrace intercepts the
> application prior to execution and does not rely on a specific user
> input. It then decodes the machine instructions to follow all control
> flows to their natural conclusions.  This includes control flows that go
> beyond the boundaries of the static executable code into shared
> libraries. This new technique avoids false positives which could be
> produced by static analysis and includes paths that could be missed by
> dynamic tracing.  This type of analysis could be useful in both testing
> and fuzzing by providing a call graph to a given function.
>
> MCTrace was initially designed to help generate the seccomp() filter
> list, which is a whitelist/blacklist of system calls for a specific
> application. Seccomp filters easily become outdated when the application
> or shared library is updated. This can cause failures or security
> issues [ 1 ].  Other potential uses including examining binary blobs,
> vulnerability analysis, and debugging.

Hi Liam,

Is MCTrace available anywhere?


--
Thanks,
Sasha

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Dhaval Giani]

On Tue, Oct 2, 2018 at 2:03 PM Sasha Levin <levinsasha928@gmail.com> wrote:
>
> On Tue, Oct 2, 2018 at 4:44 PM Liam R. Howlett <Liam.Howlett@oracle.com> wrote:
> >
> > * Dhaval Giani <dhaval.giani@gmail.com> [180919 13:15]:
> > > Hi folks,
> > >
> > > Sasha and I are pleased to announce the Testing and Fuzzing track at
> > > LPC [ 1 ]. We are planning to continue the discussions from last
> > > year's microconference [2]. Many discussions from the Automated
> > > Testing Summit [3] will also continue, and a final agenda will come up
> > > only soon after that.
> > >
> > > Suggested Topics
> > >
> > > - Syzbot/syzkaller
> > > - ATS
> > > - Distro/stable testing
> > > - kernelci
> > > - kernelci auto bisection
> > > - Unit testing framework
> > >
> > > We look forward to other interesting topics for this microconference
> > > as a reply to this email.
> > >
> > > Thanks!
> > > Dhaval and Sasha
> > >
> > > [1] https://blog.linuxplumbersconf.org/2018/testing-and-fuzzing-mc/
> > > [2] https://lwn.net/Articles/735034/
> > > [3] https://elinux.org/Automated_Testing_Summit
> >
> >
> > Hello,
> >
> > I have a new way to analyze binaries to detect specific calls without
> > the need for source.  I would like to discuss Machine Code Trace
> > (MCTrace) at the Testing and Fuzzing LPC track.  MCTrace intercepts the
> > application prior to execution and does not rely on a specific user
> > input. It then decodes the machine instructions to follow all control
> > flows to their natural conclusions.  This includes control flows that go
> > beyond the boundaries of the static executable code into shared
> > libraries. This new technique avoids false positives which could be
> > produced by static analysis and includes paths that could be missed by
> > dynamic tracing.  This type of analysis could be useful in both testing
> > and fuzzing by providing a call graph to a given function.
> >
> > MCTrace was initially designed to help generate the seccomp() filter
> > list, which is a whitelist/blacklist of system calls for a specific
> > application. Seccomp filters easily become outdated when the application
> > or shared library is updated. This can cause failures or security
> > issues [ 1 ].  Other potential uses including examining binary blobs,
> > vulnerability analysis, and debugging.
>
> Hi Liam,
>
> Is MCTrace available anywhere?
>

Sasha,

McTrace is an early prototype, really needing a lot of feedback. I
will let Liam send more details (some how he got dropped from the cc)

Dhavla

>
> --
> Thanks,
> Sasha

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Liam R. Howlett]

* Sasha Levin <levinsasha928@gmail.com> [181002 17:03]:
> On Tue, Oct 2, 2018 at 4:44 PM Liam R. Howlett <Liam.Howlett@oracle.com> wrote:
> >
> > * Dhaval Giani <dhaval.giani@gmail.com> [180919 13:15]:
> > > Hi folks,
> > >
> > > Sasha and I are pleased to announce the Testing and Fuzzing track at
> > > LPC [ 1 ]. We are planning to continue the discussions from last
> > > year's microconference [2]. Many discussions from the Automated
> > > Testing Summit [3] will also continue, and a final agenda will come up
> > > only soon after that.
> > >
> > > Suggested Topics
> > >
> > > - Syzbot/syzkaller
> > > - ATS
> > > - Distro/stable testing
> > > - kernelci
> > > - kernelci auto bisection
> > > - Unit testing framework
> > >
> > > We look forward to other interesting topics for this microconference
> > > as a reply to this email.
> > >
> > > Thanks!
> > > Dhaval and Sasha
> > >
> > > [1] https://blog.linuxplumbersconf.org/2018/testing-and-fuzzing-mc/
> > > [2] https://lwn.net/Articles/735034/
> > > [3] https://elinux.org/Automated_Testing_Summit
> >
> >
> > Hello,
> >
> > I have a new way to analyze binaries to detect specific calls without
> > the need for source.  I would like to discuss Machine Code Trace
> > (MCTrace) at the Testing and Fuzzing LPC track.  MCTrace intercepts the
> > application prior to execution and does not rely on a specific user
> > input. It then decodes the machine instructions to follow all control
> > flows to their natural conclusions.  This includes control flows that go
> > beyond the boundaries of the static executable code into shared
> > libraries. This new technique avoids false positives which could be
> > produced by static analysis and includes paths that could be missed by
> > dynamic tracing.  This type of analysis could be useful in both testing
> > and fuzzing by providing a call graph to a given function.
> >
> > MCTrace was initially designed to help generate the seccomp() filter
> > list, which is a whitelist/blacklist of system calls for a specific
> > application. Seccomp filters easily become outdated when the application
> > or shared library is updated. This can cause failures or security
> > issues [ 1 ].  Other potential uses including examining binary blobs,
> > vulnerability analysis, and debugging.
> 
> Hi Liam,
> 
> Is MCTrace available anywhere?

Hello Sasha,

I missed this email as I was not CC'ed.

MCTrace is currently a proof-of-concept and the source is not available.
There are a number of instructions that need additional work, but I have
some test applications that can be analyzed.  I'd like to explain the
concept, why it is useful, and debate other potential uses.

Thank you,
Liam

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Sasha Levin]

On Wed, Oct 3, 2018 at 3:16 PM Liam R. Howlett <Liam.Howlett@oracle.com> wrote:
>
> * Sasha Levin <levinsasha928@gmail.com> [181002 17:03]:
> > On Tue, Oct 2, 2018 at 4:44 PM Liam R. Howlett <Liam.Howlett@oracle.com> wrote:
> > >
> > > * Dhaval Giani <dhaval.giani@gmail.com> [180919 13:15]:
> > > > Hi folks,
> > > >
> > > > Sasha and I are pleased to announce the Testing and Fuzzing track at
> > > > LPC [ 1 ]. We are planning to continue the discussions from last
> > > > year's microconference [2]. Many discussions from the Automated
> > > > Testing Summit [3] will also continue, and a final agenda will come up
> > > > only soon after that.
> > > >
> > > > Suggested Topics
> > > >
> > > > - Syzbot/syzkaller
> > > > - ATS
> > > > - Distro/stable testing
> > > > - kernelci
> > > > - kernelci auto bisection
> > > > - Unit testing framework
> > > >
> > > > We look forward to other interesting topics for this microconference
> > > > as a reply to this email.
> > > >
> > > > Thanks!
> > > > Dhaval and Sasha
> > > >
> > > > [1] https://blog.linuxplumbersconf.org/2018/testing-and-fuzzing-mc/
> > > > [2] https://lwn.net/Articles/735034/
> > > > [3] https://elinux.org/Automated_Testing_Summit
> > >
> > >
> > > Hello,
> > >
> > > I have a new way to analyze binaries to detect specific calls without
> > > the need for source.  I would like to discuss Machine Code Trace
> > > (MCTrace) at the Testing and Fuzzing LPC track.  MCTrace intercepts the
> > > application prior to execution and does not rely on a specific user
> > > input. It then decodes the machine instructions to follow all control
> > > flows to their natural conclusions.  This includes control flows that go
> > > beyond the boundaries of the static executable code into shared
> > > libraries. This new technique avoids false positives which could be
> > > produced by static analysis and includes paths that could be missed by
> > > dynamic tracing.  This type of analysis could be useful in both testing
> > > and fuzzing by providing a call graph to a given function.
> > >
> > > MCTrace was initially designed to help generate the seccomp() filter
> > > list, which is a whitelist/blacklist of system calls for a specific
> > > application. Seccomp filters easily become outdated when the application
> > > or shared library is updated. This can cause failures or security
> > > issues [ 1 ].  Other potential uses including examining binary blobs,
> > > vulnerability analysis, and debugging.
> >
> > Hi Liam,
> >
> > Is MCTrace available anywhere?
>
> Hello Sasha,
>
> I missed this email as I was not CC'ed.

Sorry about that, I must have messed something up.

> MCTrace is currently a proof-of-concept and the source is not available.

What is the reason behind it not being available?

> There are a number of instructions that need additional work, but I have
> some test applications that can be analyzed.  I'd like to explain the
> concept, why it is useful, and debate other potential uses.

I have 2 concerns here:

1. This is an interesting new field to explore but since no one is
familiar with how this works, nor anyone can actually play and tinker
with it, I suspect that the ~30 min you'll have to discuss it will be
spent on describing how it works and answering basic questions. This
seems like a better fit for a refereed track session rather than MC.

2. In general, I don't think we can or should discuss a closed source
project. Sure, we can discuss the concept itself, but in that case I
don't see how it will benefit the community.

--
Thanks,
Sasha

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Dan Carpenter]

I would like to talk about Smatch, if I could.  Especially the Spectre
work and the improvements to cross function tracking.

regards,
dan carpenter

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Knut Omang]

On Wed, 2018-09-19 at 10:14 -0700, Dhaval Giani wrote:
> Hi folks,
> 
> Sasha and I are pleased to announce the Testing and Fuzzing track at
> LPC [ 1 ]. We are planning to continue the discussions from last
> year's microconference [2]. Many discussions from the Automated
> Testing Summit [3] will also continue, and a final agenda will come up
> only soon after that.
> 
> Suggested Topics
> 
> - Syzbot/syzkaller
> - ATS
> - Distro/stable testing
> - kernelci
> - kernelci auto bisection
> - Unit testing framework
> 
> We look forward to other interesting topics for this microconference
> as a reply to this email.
> 
> Thanks!
> Dhaval and Sasha
> 
> [1] https://blog.linuxplumbersconf.org/2018/testing-and-fuzzing-mc/
> [2] https://lwn.net/Articles/735034/
> [3] https://elinux.org/Automated_Testing_Summit

Hi,

I'd like to make a short case for my "runchecks" enhancement to 'make C={1,2}' which is
the result of a discussion on the kernel mailing list last winter, to see how it can be
brought forward. It allows simple configuration to run one or more tools like checkpatch,
sparse, smatch and documentation checking with selective suppression of certain classes of
issues, to enable automated testing even when not all issues have been resolved - see
mailing list threads here:

https://lkml.org/lkml/2017/12/16/135
https://patchwork.kernel.org/patch/10174967/

Thanks,
Knut

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Kevin Hilman]

Dhaval Giani <dhaval.giani@gmail.com> writes:

> On Mon, Nov 5, 2018 at 10:05 AM Gustavo Padovan
> <gustavo.padovan@collabora.com> wrote:
>>
>> Hi Dhaval,
>>
>> On 9/19/18 7:13 PM, Dhaval Giani wrote:
>> > Hi folks,
>> >
>> > Sasha and I are pleased to announce the Testing and Fuzzing track at
>> > LPC [ 1 ]. We are planning to continue the discussions from last
>> > year's microconference [2]. Many discussions from the Automated
>> > Testing Summit [3] will also continue, and a final agenda will come up
>> > only soon after that.
>> >
>> > Suggested Topics
>> >
>> > - Syzbot/syzkaller
>> > - ATS
>> > - Distro/stable testing
>> > - kernelci
>> > - kernelci auto bisection
>>
>> Having 2 kernelci talks don't make too much sense, I discussed with
>> Kevin and we thing it would be a good idea to merge them together. Could
>> you do that?
>>
>
> OK, we can make that happen. 45 minutes for the 2 combined topics?
>

Yes, that will be enough.

Thanks,

Kevin

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Dhaval Giani]

On Mon, Nov 5, 2018 at 10:05 AM Gustavo Padovan
<gustavo.padovan@collabora.com> wrote:
>
> Hi Dhaval,
>
> On 9/19/18 7:13 PM, Dhaval Giani wrote:
> > Hi folks,
> >
> > Sasha and I are pleased to announce the Testing and Fuzzing track at
> > LPC [ 1 ]. We are planning to continue the discussions from last
> > year's microconference [2]. Many discussions from the Automated
> > Testing Summit [3] will also continue, and a final agenda will come up
> > only soon after that.
> >
> > Suggested Topics
> >
> > - Syzbot/syzkaller
> > - ATS
> > - Distro/stable testing
> > - kernelci
> > - kernelci auto bisection
>
> Having 2 kernelci talks don't make too much sense, I discussed with
> Kevin and we thing it would be a good idea to merge them together. Could
> you do that?
>

OK, we can make that happen. 45 minutes for the 2 combined topics?

> Thanks,
>
> Gustavo
>
>
> Gustavo Padovan
> Collabora Ltd
>

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Gustavo Padovan]

Hi Dhaval,

On 9/19/18 7:13 PM, Dhaval Giani wrote:
> Hi folks,
>
> Sasha and I are pleased to announce the Testing and Fuzzing track at
> LPC [ 1 ]. We are planning to continue the discussions from last
> year's microconference [2]. Many discussions from the Automated
> Testing Summit [3] will also continue, and a final agenda will come up
> only soon after that.
>
> Suggested Topics
>
> - Syzbot/syzkaller
> - ATS
> - Distro/stable testing
> - kernelci
> - kernelci auto bisection

Having 2 kernelci talks don't make too much sense, I discussed with 
Kevin and we thing it would be a good idea to merge them together. Could 
you do that?

Thanks,

Gustavo


Gustavo Padovan
Collabora Ltd

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Steven Rostedt]

On Thu, 11 Oct 2018 11:41:47 +0200
Dmitry Vyukov <dvyukov@google.com> wrote:

> >
> > Hi Steven, Dhaval,
> >
> > Acknowledged.
> >
> > Then smaller topic that would benefit from discussion are:
> >
> > the main one being:
> >
> > 1. syzbot: developer process; unfixed bugs; bug triage; what's
> > working? what's not? why? how can we make more parts work? collecting
> > feedback  
> 
> +Paul as Plumbers committee representative
> And +Greg is already here as Kernel Summit representative
> 
> Re syzbot talk (around process and feedback).
> I've submitted this as proposal to Plumbers, it did not make it to
> main track but it's in backlog (I am a reserve speaker).
> Sergey suggested to submit this as proposal for Kernel Summit TECH
> TOPIC. It would probably benefit from more kernel maintainers in the
> room.
> 
> Now I am somewhat troubled, if I submit the talk for Kernel Summit and
> then suddenly I materialize as speaker on Plumbers too. It's not a
> good idea to do the same talk twice, right?

Why not, it's been done before :-)

> The questions are:
> 1. Is it a good fit for Kernel Summit TECH TOPIC?
> 2. What should I do if I suddenly accepted to both Summit and
> Plumbers? I probably could prepare a second, different talk for
> Plumbers then (it wasn't announced yet).
> 3. If the talk is accepted to Summit, what should we do with Testing
> MC part? Withdraw?

If this is a discussion topic, and you get accepted as a plumbers talk,
here's what I would do:

1. Have a talk (full presentation) ready to give at the plumbers track.
The Refereed Track is presentation style. Here you will be able to
explain in details what the issues are, and explain your own ideas.

2. For the plumbers MC, have something ready to give a quick overview
of what the issues are and what you think needs to get done. This is a
good way to get feed back from people, and use the time for discussions.

3. If you also get a Kernel Summit track accepted, I find that this is
somewhat in between a full presentation and a full discussion base of a
MC. Here, I would focus on details focusing on the kernel, and what you
would like to discuss with kernel developers specifically (note, I'm
sure the plumbers MC room will also be filled with kernel developers).

In other words, you can have three talks about the same topic, but all
addressing different aspects of that topic in different ways. That's
perfectly fine to do (I did this in Prague, I had 3 talks about printk!)

-- Steve


> 
> 
> > 2. how to increase test coverage/find more bugs, in particular:
> >  - adding manual spot KASAN/KMSAN checks into kernel codebase
> >  - stubbing hardware/external inputs to kernel for testing
> >  - description of kernel interfaces (again)
> >
> > 3. dealing with kernel console output mess
> >  - intermixed/split lines
> >  - understand when/how kernel crashed
> >  - and where is that message
> >  - crash identity  

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Dmitry Vyukov]

On Thu, Oct 11, 2018 at 10:54 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
>>> On Mon, 8 Oct 2018 19:02:51 +0200
>>> Dmitry Vyukov <dvyukov@google.com> wrote:
>>>
>>> > On Wed, Sep 19, 2018 at 7:13 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>>> > > Hi folks,
>>> > >
>>> > > Sasha and I are pleased to announce the Testing and Fuzzing track at
>>> > > LPC [ 1 ]. We are planning to continue the discussions from last
>>> > > year's microconference [2]. Many discussions from the Automated
>>> > > Testing Summit [3] will also continue, and a final agenda will come up
>>> > > only soon after that.
>>> > >
>>> > > Suggested Topics
>>> > >
>>> > > - Syzbot/syzkaller
>>> > > - ATS
>>> > > - Distro/stable testing
>>> > > - kernelci
>>> > > - kernelci auto bisection
>>> > > - Unit testing framework
>>> > >
>>> > > We look forward to other interesting topics for this microconference
>>> > > as a reply to this email.
>>> >
>>> > Hi Dhaval and Sasha,
>>> >
>>> > My syzbot talk wasn't accepted to main track, so I would like to do
>>> > more or less full-fledged talk on the microconf. Is it possible?
>>>
>>> Hi Dmitry,
>>>
>>> Note, microconfs are not for full-fledged talks. They are to be
>>> discussion focused. You can have a 5-10 minute presentation that leads
>>> up to discussion of future work, but we like to refrain from any talks
>>> about what was done if there's nothing to go forward with.
>>
>> Dmitiry,
>>
>> Can you clarify the scope of what you want to discuss during the
>> microconference? Further to what Steven said, we don't want
>> presentations (So 3, maybe 4 slides). We want discussions about future
>> work.
>>
>> Thanks!
>> Dhaval
>
>
> Hi Steven, Dhaval,
>
> Acknowledged.
>
> Then smaller topic that would benefit from discussion are:
>
> the main one being:
>
> 1. syzbot: developer process; unfixed bugs; bug triage; what's
> working? what's not? why? how can we make more parts work? collecting
> feedback

+Paul as Plumbers committee representative
And +Greg is already here as Kernel Summit representative

Re syzbot talk (around process and feedback).
I've submitted this as proposal to Plumbers, it did not make it to
main track but it's in backlog (I am a reserve speaker).
Sergey suggested to submit this as proposal for Kernel Summit TECH
TOPIC. It would probably benefit from more kernel maintainers in the
room.

Now I am somewhat troubled, if I submit the talk for Kernel Summit and
then suddenly I materialize as speaker on Plumbers too. It's not a
good idea to do the same talk twice, right?
The questions are:
1. Is it a good fit for Kernel Summit TECH TOPIC?
2. What should I do if I suddenly accepted to both Summit and
Plumbers? I probably could prepare a second, different talk for
Plumbers then (it wasn't announced yet).
3. If the talk is accepted to Summit, what should we do with Testing
MC part? Withdraw?


> 2. how to increase test coverage/find more bugs, in particular:
>  - adding manual spot KASAN/KMSAN checks into kernel codebase
>  - stubbing hardware/external inputs to kernel for testing
>  - description of kernel interfaces (again)
>
> 3. dealing with kernel console output mess
>  - intermixed/split lines
>  - understand when/how kernel crashed
>  - and where is that message
>  - crash identity

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Dmitry Vyukov]

On Wed, Oct 10, 2018 at 5:47 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> On Mon, Oct 8, 2018 at 11:23 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>>
>> On Mon, 8 Oct 2018 19:02:51 +0200
>> Dmitry Vyukov <dvyukov@google.com> wrote:
>>
>> > On Wed, Sep 19, 2018 at 7:13 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>> > > Hi folks,
>> > >
>> > > Sasha and I are pleased to announce the Testing and Fuzzing track at
>> > > LPC [ 1 ]. We are planning to continue the discussions from last
>> > > year's microconference [2]. Many discussions from the Automated
>> > > Testing Summit [3] will also continue, and a final agenda will come up
>> > > only soon after that.
>> > >
>> > > Suggested Topics
>> > >
>> > > - Syzbot/syzkaller
>> > > - ATS
>> > > - Distro/stable testing
>> > > - kernelci
>> > > - kernelci auto bisection
>> > > - Unit testing framework
>> > >
>> > > We look forward to other interesting topics for this microconference
>> > > as a reply to this email.
>> >
>> > Hi Dhaval and Sasha,
>> >
>> > My syzbot talk wasn't accepted to main track, so I would like to do
>> > more or less full-fledged talk on the microconf. Is it possible?
>>
>> Hi Dmitry,
>>
>> Note, microconfs are not for full-fledged talks. They are to be
>> discussion focused. You can have a 5-10 minute presentation that leads
>> up to discussion of future work, but we like to refrain from any talks
>> about what was done if there's nothing to go forward with.
>
> Dmitiry,
>
> Can you clarify the scope of what you want to discuss during the
> microconference? Further to what Steven said, we don't want
> presentations (So 3, maybe 4 slides). We want discussions about future
> work.
>
> Thanks!
> Dhaval


Hi Steven, Dhaval,

Acknowledged.

Then smaller topic that would benefit from discussion are:

the main one being:

1. syzbot: developer process; unfixed bugs; bug triage; what's
working? what's not? why? how can we make more parts work? collecting
feedback

2. how to increase test coverage/find more bugs, in particular:
 - adding manual spot KASAN/KMSAN checks into kernel codebase
 - stubbing hardware/external inputs to kernel for testing
 - description of kernel interfaces (again)

3. dealing with kernel console output mess
 - intermixed/split lines
 - understand when/how kernel crashed
 - and where is that message
 - crash identity

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Dhaval Giani]

On Mon, Oct 8, 2018 at 11:23 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 8 Oct 2018 19:02:51 +0200
> Dmitry Vyukov <dvyukov@google.com> wrote:
>
> > On Wed, Sep 19, 2018 at 7:13 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> > > Hi folks,
> > >
> > > Sasha and I are pleased to announce the Testing and Fuzzing track at
> > > LPC [ 1 ]. We are planning to continue the discussions from last
> > > year's microconference [2]. Many discussions from the Automated
> > > Testing Summit [3] will also continue, and a final agenda will come up
> > > only soon after that.
> > >
> > > Suggested Topics
> > >
> > > - Syzbot/syzkaller
> > > - ATS
> > > - Distro/stable testing
> > > - kernelci
> > > - kernelci auto bisection
> > > - Unit testing framework
> > >
> > > We look forward to other interesting topics for this microconference
> > > as a reply to this email.
> >
> > Hi Dhaval and Sasha,
> >
> > My syzbot talk wasn't accepted to main track, so I would like to do
> > more or less full-fledged talk on the microconf. Is it possible?
>
> Hi Dmitry,
>
> Note, microconfs are not for full-fledged talks. They are to be
> discussion focused. You can have a 5-10 minute presentation that leads
> up to discussion of future work, but we like to refrain from any talks
> about what was done if there's nothing to go forward with.

Dmitiry,

Can you clarify the scope of what you want to discuss during the
microconference? Further to what Steven said, we don't want
presentations (So 3, maybe 4 slides). We want discussions about future
work.

Thanks!
Dhaval

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Steven Rostedt]

On Mon, 8 Oct 2018 19:02:51 +0200
Dmitry Vyukov <dvyukov@google.com> wrote:

> On Wed, Sep 19, 2018 at 7:13 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> > Hi folks,
> >
> > Sasha and I are pleased to announce the Testing and Fuzzing track at
> > LPC [ 1 ]. We are planning to continue the discussions from last
> > year's microconference [2]. Many discussions from the Automated
> > Testing Summit [3] will also continue, and a final agenda will come up
> > only soon after that.
> >
> > Suggested Topics
> >
> > - Syzbot/syzkaller
> > - ATS
> > - Distro/stable testing
> > - kernelci
> > - kernelci auto bisection
> > - Unit testing framework
> >
> > We look forward to other interesting topics for this microconference
> > as a reply to this email.  
> 
> Hi Dhaval and Sasha,
> 
> My syzbot talk wasn't accepted to main track, so I would like to do
> more or less full-fledged talk on the microconf. Is it possible?

Hi Dmitry,

Note, microconfs are not for full-fledged talks. They are to be
discussion focused. You can have a 5-10 minute presentation that leads
up to discussion of future work, but we like to refrain from any talks
about what was done if there's nothing to go forward with.

-- Steve

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Dmitry Vyukov]

On Wed, Sep 19, 2018 at 7:13 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> Hi folks,
>
> Sasha and I are pleased to announce the Testing and Fuzzing track at
> LPC [ 1 ]. We are planning to continue the discussions from last
> year's microconference [2]. Many discussions from the Automated
> Testing Summit [3] will also continue, and a final agenda will come up
> only soon after that.
>
> Suggested Topics
>
> - Syzbot/syzkaller
> - ATS
> - Distro/stable testing
> - kernelci
> - kernelci auto bisection
> - Unit testing framework
>
> We look forward to other interesting topics for this microconference
> as a reply to this email.

Hi Dhaval and Sasha,

My syzbot talk wasn't accepted to main track, so I would like to do
more or less full-fledged talk on the microconf. Is it possible?

Thanks

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Knut Omang]

On Mon, 2018-09-24 at 15:42 +0200, Dmitry Vyukov wrote:
> On Sat, Sep 22, 2018 at 2:52 PM, Matthew Wilcox <willy@infradead.org> wrote:
> > On Wed, Sep 19, 2018 at 10:13:15AM -0700, Dhaval Giani wrote:
> >> Sasha and I are pleased to announce the Testing and Fuzzing track at
> >> LPC [ 1 ]. We are planning to continue the discussions from last
> >> year's microconference [2]. Many discussions from the Automated
> >> Testing Summit [3] will also continue, and a final agenda will come up
> >> only soon after that.
> >>
> >> Suggested Topics
> >>
> >> - Syzbot/syzkaller
> >> - ATS
> >> - Distro/stable testing
> >> - kernelci
> >> - kernelci auto bisection
> >> - Unit testing framework
> >>
> >> We look forward to other interesting topics for this microconference
> >> as a reply to this email.
> >
> > I would like to talk about the IDA test suite that was recently merged.
> > See lib/test_ida.c.  It can be built as a module (CONFIG_TEST_IDA=m),
> > built-in (=y) or built in userspace (as part of the radix tree test
> > suite for historical reasons) along with the current kernel code.
> >
> > Being able to build the test suite in userspace allows for much more
> > rapid development.  Building it in kernel space offers testing across
> > a wide range of configurations that I don't have access to and can't
> > necessarily simulate well in userspace.
> >
> > My userspace implementation of kmalloc() simulates failures (in a rather
> > heavy-handed way; every non-GFP_KERNEL allocation fails).  That's rather
> > harder to simulate in kernel space.  I'd like there to be a way for a
> > kernel space test suite to ask for kmalloc failures so that the failure
> > paths can be tested.
> 
> Hi Matthew,
> 
> kmalloc fault injection is already implemented with
> CONFIG_FAULT_INJECTION. For original fault injection, you more-or-less
> ask to fail X% of allocations at random. It's great for testing
> servers for stability, but not so well suited for testing. Recently
> I've added so-called "systematic" fault injection
> (/proc/thread-self/fail-nth) which is perfect for unit testing. It
> allows to ask to fail N-th allocation request in the current task. So
> a unit test for a syscall can do:
> 
> for (i = 0;; i++) {
>   write(/proc/thread-self/fail-nth, i);
>   syscall_under_test();
>   if (read(/proc/thread-self/fail-nth) != 0)
>     break;
> }
> 
> which allows to examine each failure site one-by-one systematically
> (without making random processes on the machine fail too).
> 
> 
> > I think the idea of building parts of the core kernel libraries in
> > userspace and testing them there has greater generality than just the
> > IDA, IDR, XArray & radix tree and might profitably be adopted by other
> > parts of lib/.  The userspace simulation of other parts of the kernel
> > may well need to be extended.
> 
> This would be great. Besides providing faster turn-around time, this
> allow us to finally have something like:
> 
> make test [subsystem]
> 
> which will run all tests for the subsystem locally (in parallel,
> giving final OK/FAIL). This is not possible to in-kernel tests,
> because they require a machine and an image, and it's not possible to
> support everything that people use.

With KTF (https://github.com/oracle/ktf) we are using kprobes to inject 
modifying behaviour (by modifying a function's return value or input).

To cater for the needs for failover testing from user space, our approach 
is what we call "hybrid tests", where parts of the test execute in user land and 
parts of the test execute in the kernel. 

One limitation with config based test functionality is that it requires
rebuilding the kernel to enable the functionality. I think having the tests
available for running even on pre-existing kernels can be of great value.

I agree with Matthew in that there's good time savings to be made to be
able to "lift" code out of the kernel and execute it completely in user space.
The challenge is to be able to compile the kernel code in user land unmodified. 
The pieces under lib/ is probably the easiest ones since they have few 
dependencies on other kernel components.

If interesting, I could say a few words in this context about some work I did 
to allow me to run the Infiniband driver I was working on - and also developed 
the precursor of KTF for 
(https://github.com/oracle/linux-uek/tree/uek4/master/drivers/infiniband/hw/sif)
- entirely in user space under valgrind.
I used it a.o. to test the fairly complex page table management driver code for the on-
board MMU. I have wanted to make it available in a similar way as KTF, just haven't had
the time to get back to it yet.

Knut

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Dmitry Vyukov]

On Sat, Sep 22, 2018 at 2:52 PM, Matthew Wilcox <willy@infradead.org> wrote:
> On Wed, Sep 19, 2018 at 10:13:15AM -0700, Dhaval Giani wrote:
>> Sasha and I are pleased to announce the Testing and Fuzzing track at
>> LPC [ 1 ]. We are planning to continue the discussions from last
>> year's microconference [2]. Many discussions from the Automated
>> Testing Summit [3] will also continue, and a final agenda will come up
>> only soon after that.
>>
>> Suggested Topics
>>
>> - Syzbot/syzkaller
>> - ATS
>> - Distro/stable testing
>> - kernelci
>> - kernelci auto bisection
>> - Unit testing framework
>>
>> We look forward to other interesting topics for this microconference
>> as a reply to this email.
>
> I would like to talk about the IDA test suite that was recently merged.
> See lib/test_ida.c.  It can be built as a module (CONFIG_TEST_IDA=m),
> built-in (=y) or built in userspace (as part of the radix tree test
> suite for historical reasons) along with the current kernel code.
>
> Being able to build the test suite in userspace allows for much more
> rapid development.  Building it in kernel space offers testing across
> a wide range of configurations that I don't have access to and can't
> necessarily simulate well in userspace.
>
> My userspace implementation of kmalloc() simulates failures (in a rather
> heavy-handed way; every non-GFP_KERNEL allocation fails).  That's rather
> harder to simulate in kernel space.  I'd like there to be a way for a
> kernel space test suite to ask for kmalloc failures so that the failure
> paths can be tested.

Hi Matthew,

kmalloc fault injection is already implemented with
CONFIG_FAULT_INJECTION. For original fault injection, you more-or-less
ask to fail X% of allocations at random. It's great for testing
servers for stability, but not so well suited for testing. Recently
I've added so-called "systematic" fault injection
(/proc/thread-self/fail-nth) which is perfect for unit testing. It
allows to ask to fail N-th allocation request in the current task. So
a unit test for a syscall can do:

for (i = 0;; i++) {
  write(/proc/thread-self/fail-nth, i);
  syscall_under_test();
  if (read(/proc/thread-self/fail-nth) != 0)
    break;
}

which allows to examine each failure site one-by-one systematically
(without making random processes on the machine fail too).


> I think the idea of building parts of the core kernel libraries in
> userspace and testing them there has greater generality than just the
> IDA, IDR, XArray & radix tree and might profitably be adopted by other
> parts of lib/.  The userspace simulation of other parts of the kernel
> may well need to be extended.

This would be great. Besides providing faster turn-around time, this
allow us to finally have something like:

make test [subsystem]

which will run all tests for the subsystem locally (in parallel,
giving final OK/FAIL). This is not possible to in-kernel tests,
because they require a machine and an image, and it's not possible to
support everything that people use.

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Matthew Wilcox]

On Wed, Sep 19, 2018 at 10:13:15AM -0700, Dhaval Giani wrote:
> Sasha and I are pleased to announce the Testing and Fuzzing track at
> LPC [ 1 ]. We are planning to continue the discussions from last
> year's microconference [2]. Many discussions from the Automated
> Testing Summit [3] will also continue, and a final agenda will come up
> only soon after that.
> 
> Suggested Topics
> 
> - Syzbot/syzkaller
> - ATS
> - Distro/stable testing
> - kernelci
> - kernelci auto bisection
> - Unit testing framework
> 
> We look forward to other interesting topics for this microconference
> as a reply to this email.

I would like to talk about the IDA test suite that was recently merged.
See lib/test_ida.c.  It can be built as a module (CONFIG_TEST_IDA=m),
built-in (=y) or built in userspace (as part of the radix tree test
suite for historical reasons) along with the current kernel code.

Being able to build the test suite in userspace allows for much more
rapid development.  Building it in kernel space offers testing across
a wide range of configurations that I don't have access to and can't
necessarily simulate well in userspace.

My userspace implementation of kmalloc() simulates failures (in a rather
heavy-handed way; every non-GFP_KERNEL allocation fails).  That's rather
harder to simulate in kernel space.  I'd like there to be a way for a
kernel space test suite to ask for kmalloc failures so that the failure
paths can be tested.

I think the idea of building parts of the core kernel libraries in
userspace and testing them there has greater generality than just the
IDA, IDR, XArray & radix tree and might profitably be adopted by other
parts of lib/.  The userspace simulation of other parts of the kernel
may well need to be extended.

I'd also like to do at least kernel-space testing of the page cache.
I have some ideas around that too, but I don't know if they're
sufficiently far advanced to talk about in November.