Any tracing mechanism can track the executed instructions of a user process in the kernel?

Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Dongliang Mu]

Hi all,

I am writing to kindly ask one question: is there any tracing
mechanism in Linux kernel that can trace all the executed instructions
of a user process? If this user process is run on different
processors, traces of this process on different processors should be
also recorded.

Any comment is welcome.

--
My best regards to you.

     No System Is Safe!
     Dongliang Mu

Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Dongliang Mu]

Hi all,

I am writing to kindly ask one question: is there any tracing
mechanism in Linux kernel that can trace all the executed instructions
of a user process? If this user process is run on different
processors, traces of this process on different processors should be
also recorded.

Any comment is welcome.

--
My best regards to you.

     No System Is Safe!
     Dongliang Mu

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Aruna Hewapathirane]

[-- Attachment #1.1: Type: text/plain, Size: 977 bytes --]

On Sun, Oct 17, 2021 at 10:46 PM Dongliang Mu <mudongliangabcd@gmail.com>
wrote:

> Hi all,
>
> I am writing to kindly ask one question: is there any tracing
> mechanism in Linux kernel that can trace all the executed instructions
> of a user process?


Use gdb -p <pid> to attach to the process your interested in
The gdb command: display/i $pc shows you the instruction before it
executes. display $pc shows the line of code before n or s executes it.


> If this user process is run on different
> processors, traces of this process on different processors should be
> also recorded.
>

This am not too sure about, someone more knowledgeable will  probably let
us know ( Valdis ? )

>
> Any comment is welcome.
>
> --
> My best regards to you.
>
>      No System Is Safe!
>      Dongliang Mu
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

[-- Attachment #1.2: Type: text/html, Size: 1934 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Pavel Skripkin]

On 10/18/21 05:45, Dongliang Mu wrote:
> Hi all,
> 
> I am writing to kindly ask one question: is there any tracing
> mechanism in Linux kernel that can trace all the executed instructions
> of a user process? If this user process is run on different
> processors, traces of this process on different processors should be
> also recorded.
> 
>

I think, kernel is not supposed to do that kind of things. I mean, there 
is no such wrapper in the kernel to do this task, AFAIK.

You have an access to all system calls, so you can implement your own 
gdb in the kernel via ptrace() :)


If you need only report about instructions, you can call `perf record` + 
`perf report` via call_usermodehelper() and somehow parse the output of 
these helpers.


> Any comment is welcome.
> 


With regards,
Pavel Skripkin

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Pavel Skripkin]

On 10/18/21 05:45, Dongliang Mu wrote:
> Hi all,
> 
> I am writing to kindly ask one question: is there any tracing
> mechanism in Linux kernel that can trace all the executed instructions
> of a user process? If this user process is run on different
> processors, traces of this process on different processors should be
> also recorded.
> 
>

I think, kernel is not supposed to do that kind of things. I mean, there 
is no such wrapper in the kernel to do this task, AFAIK.

You have an access to all system calls, so you can implement your own 
gdb in the kernel via ptrace() :)


If you need only report about instructions, you can call `perf record` + 
`perf report` via call_usermodehelper() and somehow parse the output of 
these helpers.


> Any comment is welcome.
> 


With regards,
Pavel Skripkin

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[FMDF]

[-- Attachment #1.1: Type: text/plain, Size: 1773 bytes --]

On Mon, 18 Oct 2021, 04:46 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:

> Hi all,
>
> I am writing to kindly ask one question: is there any tracing
> mechanism in Linux kernel that can trace all the executed instructions
> of a user process? If this user process is run on different
> processors, traces of this process on different processors should be
> also recorded.
>

You've not explained in detail what is the purpose of the tracing that you
want to do. Missing this information I can only provide you a list of links
to various tools and methods. Take a look by yourself and try to figure out
what is better suited for your needs...

https://lwn.net/Kernel/Index/#Development_tools-Kernel_tracing
trace-cmd: https://trace-cmd.org
perf ftrace: <Linux source>/tools/perf/Documentation/perf-trace.txt
perf-tools: https://github.com/brendangregg/perf-tools

In particular, the "function_graph" tracer of ftrace seems to be suited for
your purposes:

https://man7.org/linux/man-pages/man1/trace-cmd.1.html
https://man7.org/linux/man-pages/man1/trace-cmd-record.1.html
https://man7.org/linux/man-pages/man1/trace-cmd-report.1.html

Obviously, gdb (and friends) is still invaluable (but it may be difficult
to use, depending on your knowledge and experience, so use it if you really
know that  you need it):

https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html
https://www.kernel.org/doc/html/latest/dev-tools/kgdb.html

Regards,

Fabio M. De Francesco


> Any comment is welcome.
>
> --
> My best regards to you.
>
>      No System Is Safe!
>      Dongliang Mu
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

[-- Attachment #1.2: Type: text/html, Size: 3611 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Dongliang Mu]

On Mon, Oct 18, 2021 at 4:07 PM FMDF <fmdefrancesco@gmail.com> wrote:
>
> On Mon, 18 Oct 2021, 04:46 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:
>>
>> Hi all,
>>
>> I am writing to kindly ask one question: is there any tracing
>> mechanism in Linux kernel that can trace all the executed instructions
>> of a user process? If this user process is run on different
>> processors, traces of this process on different processors should be
>> also recorded.
>
>
> You've not explained in detail what is the purpose of the tracing that you want to do. Missing this information I can only provide you a list of links to various tools and methods. Take a look by yourself and try to figure out what is better suited for your needs...

I want to log all the executed instructions of a user process (e.g.,
poc.c in syzkaller) in the kernel mode and then would like to leverage
backward analysis to capture the root cause of kernel panic/crash.

Therefore, I need the instruction-level tracing mechanisms or tools.

>
> https://lwn.net/Kernel/Index/#Development_tools-Kernel_tracing
> trace-cmd: https://trace-cmd.org

Ftrace is working at the function level. Therefore, ftrace and tracers
based on ftrace are not suitable for my scenario.

> perf ftrace: <Linux source>/tools/perf/Documentation/perf-trace.txt
> perf-tools: https://github.com/brendangregg/perf-tools

I am not familiar with the perf tool and cannot verify if it works for
my scenario.

>
> In particular, the "function_graph" tracer of ftrace seems to be suited for your purposes:
>
> https://man7.org/linux/man-pages/man1/trace-cmd.1.html
> https://man7.org/linux/man-pages/man1/trace-cmd-record.1.html
> https://man7.org/linux/man-pages/man1/trace-cmd-report.1.html
>
> Obviously, gdb (and friends) is still invaluable (but it may be difficult to use, depending on your knowledge and experience, so use it if you really know that  you need it):
>
> https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html
> https://www.kernel.org/doc/html/latest/dev-tools/kgdb.html
>
> Regards,
>
> Fabio M. De Francesco
>
>>
>> Any comment is welcome.
>>
>> --
>> My best regards to you.
>>
>>      No System Is Safe!
>>      Dongliang Mu
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Dongliang Mu]

On Mon, Oct 18, 2021 at 4:07 PM FMDF <fmdefrancesco@gmail.com> wrote:
>
> On Mon, 18 Oct 2021, 04:46 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:
>>
>> Hi all,
>>
>> I am writing to kindly ask one question: is there any tracing
>> mechanism in Linux kernel that can trace all the executed instructions
>> of a user process? If this user process is run on different
>> processors, traces of this process on different processors should be
>> also recorded.
>
>
> You've not explained in detail what is the purpose of the tracing that you want to do. Missing this information I can only provide you a list of links to various tools and methods. Take a look by yourself and try to figure out what is better suited for your needs...

I want to log all the executed instructions of a user process (e.g.,
poc.c in syzkaller) in the kernel mode and then would like to leverage
backward analysis to capture the root cause of kernel panic/crash.

Therefore, I need the instruction-level tracing mechanisms or tools.

>
> https://lwn.net/Kernel/Index/#Development_tools-Kernel_tracing
> trace-cmd: https://trace-cmd.org

Ftrace is working at the function level. Therefore, ftrace and tracers
based on ftrace are not suitable for my scenario.

> perf ftrace: <Linux source>/tools/perf/Documentation/perf-trace.txt
> perf-tools: https://github.com/brendangregg/perf-tools

I am not familiar with the perf tool and cannot verify if it works for
my scenario.

>
> In particular, the "function_graph" tracer of ftrace seems to be suited for your purposes:
>
> https://man7.org/linux/man-pages/man1/trace-cmd.1.html
> https://man7.org/linux/man-pages/man1/trace-cmd-record.1.html
> https://man7.org/linux/man-pages/man1/trace-cmd-report.1.html
>
> Obviously, gdb (and friends) is still invaluable (but it may be difficult to use, depending on your knowledge and experience, so use it if you really know that  you need it):
>
> https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html
> https://www.kernel.org/doc/html/latest/dev-tools/kgdb.html
>
> Regards,
>
> Fabio M. De Francesco
>
>>
>> Any comment is welcome.
>>
>> --
>> My best regards to you.
>>
>>      No System Is Safe!
>>      Dongliang Mu
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Greg KH]

On Mon, Oct 18, 2021 at 04:41:14PM +0800, Dongliang Mu wrote:
> On Mon, Oct 18, 2021 at 4:07 PM FMDF <fmdefrancesco@gmail.com> wrote:
> >
> > On Mon, 18 Oct 2021, 04:46 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> I am writing to kindly ask one question: is there any tracing
> >> mechanism in Linux kernel that can trace all the executed instructions
> >> of a user process? If this user process is run on different
> >> processors, traces of this process on different processors should be
> >> also recorded.
> >
> >
> > You've not explained in detail what is the purpose of the tracing that you want to do. Missing this information I can only provide you a list of links to various tools and methods. Take a look by yourself and try to figure out what is better suited for your needs...
> 
> I want to log all the executed instructions of a user process (e.g.,
> poc.c in syzkaller) in the kernel mode and then would like to leverage
> backward analysis to capture the root cause of kernel panic/crash.
> 
> Therefore, I need the instruction-level tracing mechanisms or tools.

Then use a userspace debugger like gdb, that is what they are designed
for.

good luck!

greg k-h

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Greg KH]

On Mon, Oct 18, 2021 at 04:41:14PM +0800, Dongliang Mu wrote:
> On Mon, Oct 18, 2021 at 4:07 PM FMDF <fmdefrancesco@gmail.com> wrote:
> >
> > On Mon, 18 Oct 2021, 04:46 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> I am writing to kindly ask one question: is there any tracing
> >> mechanism in Linux kernel that can trace all the executed instructions
> >> of a user process? If this user process is run on different
> >> processors, traces of this process on different processors should be
> >> also recorded.
> >
> >
> > You've not explained in detail what is the purpose of the tracing that you want to do. Missing this information I can only provide you a list of links to various tools and methods. Take a look by yourself and try to figure out what is better suited for your needs...
> 
> I want to log all the executed instructions of a user process (e.g.,
> poc.c in syzkaller) in the kernel mode and then would like to leverage
> backward analysis to capture the root cause of kernel panic/crash.
> 
> Therefore, I need the instruction-level tracing mechanisms or tools.

Then use a userspace debugger like gdb, that is what they are designed
for.

good luck!

greg k-h

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Dongliang Mu]

+Brendan Gregg

On Mon, Oct 18, 2021 at 4:41 PM Dongliang Mu <mudongliangabcd@gmail.com> wrote:
>
> On Mon, Oct 18, 2021 at 4:07 PM FMDF <fmdefrancesco@gmail.com> wrote:
> >
> > On Mon, 18 Oct 2021, 04:46 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> I am writing to kindly ask one question: is there any tracing
> >> mechanism in Linux kernel that can trace all the executed instructions
> >> of a user process? If this user process is run on different
> >> processors, traces of this process on different processors should be
> >> also recorded.
> >
> >
> > You've not explained in detail what is the purpose of the tracing that you want to do. Missing this information I can only provide you a list of links to various tools and methods. Take a look by yourself and try to figure out what is better suited for your needs...
>
> I want to log all the executed instructions of a user process (e.g.,
> poc.c in syzkaller) in the kernel mode and then would like to leverage
> backward analysis to capture the root cause of kernel panic/crash.
>
> Therefore, I need the instruction-level tracing mechanisms or tools.
>
> >
> > https://lwn.net/Kernel/Index/#Development_tools-Kernel_tracing
> > trace-cmd: https://trace-cmd.org
>
> Ftrace is working at the function level. Therefore, ftrace and tracers
> based on ftrace are not suitable for my scenario.
>
> > perf ftrace: <Linux source>/tools/perf/Documentation/perf-trace.txt
> > perf-tools: https://github.com/brendangregg/perf-tools
>
> I am not familiar with the perf tool and cannot verify if it works for
> my scenario.
>
> >
> > In particular, the "function_graph" tracer of ftrace seems to be suited for your purposes:
> >
> > https://man7.org/linux/man-pages/man1/trace-cmd.1.html
> > https://man7.org/linux/man-pages/man1/trace-cmd-record.1.html
> > https://man7.org/linux/man-pages/man1/trace-cmd-report.1.html
> >
> > Obviously, gdb (and friends) is still invaluable (but it may be difficult to use, depending on your knowledge and experience, so use it if you really know that  you need it):
> >
> > https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html
> > https://www.kernel.org/doc/html/latest/dev-tools/kgdb.html
> >
> > Regards,
> >
> > Fabio M. De Francesco
> >
> >>
> >> Any comment is welcome.
> >>
> >> --
> >> My best regards to you.
> >>
> >>      No System Is Safe!
> >>      Dongliang Mu
> >>
> >> _______________________________________________
> >> Kernelnewbies mailing list
> >> Kernelnewbies@kernelnewbies.org
> >> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Dongliang Mu]

+Brendan Gregg

On Mon, Oct 18, 2021 at 4:41 PM Dongliang Mu <mudongliangabcd@gmail.com> wrote:
>
> On Mon, Oct 18, 2021 at 4:07 PM FMDF <fmdefrancesco@gmail.com> wrote:
> >
> > On Mon, 18 Oct 2021, 04:46 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> I am writing to kindly ask one question: is there any tracing
> >> mechanism in Linux kernel that can trace all the executed instructions
> >> of a user process? If this user process is run on different
> >> processors, traces of this process on different processors should be
> >> also recorded.
> >
> >
> > You've not explained in detail what is the purpose of the tracing that you want to do. Missing this information I can only provide you a list of links to various tools and methods. Take a look by yourself and try to figure out what is better suited for your needs...
>
> I want to log all the executed instructions of a user process (e.g.,
> poc.c in syzkaller) in the kernel mode and then would like to leverage
> backward analysis to capture the root cause of kernel panic/crash.
>
> Therefore, I need the instruction-level tracing mechanisms or tools.
>
> >
> > https://lwn.net/Kernel/Index/#Development_tools-Kernel_tracing
> > trace-cmd: https://trace-cmd.org
>
> Ftrace is working at the function level. Therefore, ftrace and tracers
> based on ftrace are not suitable for my scenario.
>
> > perf ftrace: <Linux source>/tools/perf/Documentation/perf-trace.txt
> > perf-tools: https://github.com/brendangregg/perf-tools
>
> I am not familiar with the perf tool and cannot verify if it works for
> my scenario.
>
> >
> > In particular, the "function_graph" tracer of ftrace seems to be suited for your purposes:
> >
> > https://man7.org/linux/man-pages/man1/trace-cmd.1.html
> > https://man7.org/linux/man-pages/man1/trace-cmd-record.1.html
> > https://man7.org/linux/man-pages/man1/trace-cmd-report.1.html
> >
> > Obviously, gdb (and friends) is still invaluable (but it may be difficult to use, depending on your knowledge and experience, so use it if you really know that  you need it):
> >
> > https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html
> > https://www.kernel.org/doc/html/latest/dev-tools/kgdb.html
> >
> > Regards,
> >
> > Fabio M. De Francesco
> >
> >>
> >> Any comment is welcome.
> >>
> >> --
> >> My best regards to you.
> >>
> >>      No System Is Safe!
> >>      Dongliang Mu
> >>
> >> _______________________________________________
> >> Kernelnewbies mailing list
> >> Kernelnewbies@kernelnewbies.org
> >> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[FMDF]

[-- Attachment #1.1: Type: text/plain, Size: 939 bytes --]

On Mon, 18 Oct 2021, 11:18 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:

> +Brendan Gregg
>

Good catch!

Take a look at his "System Performance, 2nd ed., ISBN 978-0-13-682015-4.

He has an interesting blog at https://www.brendangregg.com/blog/index.html

As Greg, Pavel, and I said, use GDB if you need it. However I'd still
consider using the "func_graph" tracer of ftrace in order to know where to
attach GDB in the calls chain.

With ftrace you'll also get info about which CPU is running (you wrote that
you need to know this information).

Now I recall that, by using Perf,  somehow I was able to go interactively
down to the assembly code and see where it was stuck in an endless loop.

Unfortunately at this moment I cannot remember how I did that and I cannot
try to reproduce it for you because these days I'm away from my PC (I'm
writing with a smartphone). Please try to figure it out by your own.

Regards,

Fabio




>
>

[-- Attachment #1.2: Type: text/html, Size: 1975 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[FMDF]

[-- Attachment #1.1: Type: text/plain, Size: 439 bytes --]

On Mon, 18 Oct 2021, 11:48 FMDF, <fmdefrancesco@gmail.com> wrote:

Now I recall that, by using Perf,  somehow I was able to go interactively
> down to the assembly code and see where it was stuck in an endless loop.
>

Ah, yes. The loop was somewhere into glibc. So I'm not sure if it is
possible to see the kernel assembly with it (it seems highly not probable).

I guess that you still need GDB and friends for that.

Regards,

Fabio

>

[-- Attachment #1.2: Type: text/html, Size: 1161 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Valdis Klētnieks]

[-- Attachment #1: Type: text/plain, Size: 2986 bytes --]

On Mon, 18 Oct 2021 16:41:14 +0800, Dongliang Mu said:

> I want to log all the executed instructions of a user process (e.g.,
> poc.c in syzkaller) in the kernel mode and then would like to leverage
> backward analysis to capture the root cause of kernel panic/crash.

> Therefore, I need the instruction-level tracing mechanisms or tools.

Tracing just the instructions won't get you where you want to be if
you're going through this approach.

You *also* need to track all the data - the instruction path inside two
different runs of syzkaller may be essentially identical, but pass 2 different
values as the 3rd parameter of a syscall.

You may also have to deal with insane amounts of data - the actual error could
have been minutes or even hours before, or the interaction between two
different processes.

You probably want to take a *really* close look at how prof and friends avoid
infinite regress when code execution drops inside the prof code, because you're
going to hit the same issues.

Or....

You can work smarter rather than harder, and ask yourself what's the minimum
amount and type of additional information to make a significant improvement in
the debugging of system crashes.

For example, 95% of the time, you can figure out what the bug is by merely
looking at the stack traceback. For most of the rest of the cases, simply
capturing the parameter values from the syscall and the basic info for page
faults and other interrupts is probably sufficient, and you can probably
leverage the audit subsystem for most of that. It can already record syscall
parameters, while logging page faults and other interrupts can probably be done
with prof.

At that point, you don't actually *need* every instruction - only tracing
branch and call instructions is sufficient, because you already know that each
instruction between the target of a branch/call and the next branch/call will
be executed.

Similarly, the lockdep code will catch most locking issues. But it won't flag
issues with data that should be protected with a lock, but are bereft of any
locking. So ask yourself: What ways are there to analyze the code and detect
critical sections prone to race conditions? Is there a sparse-on-steroids
approach that wil do the heavy lifting for those? (Note that this isn't an easy
task for the general case, but identifying two or three specific common
patterns and finding a way to detect them may be worthwhile)

And many of the rest of crashes are timing related, and "let's trace every single
instruction" is almost guaranteed to make things slow enough to change/bypass
the timing issue.

So... What's left that would be the most helpful with the least amount of data?

Go look at some threads on linux-kernel.  Look at the kernel bugs that were the
result of a Homer Simpson "D'oh!" moment.  What can we do to make those
bugs less likely to make it into the code in the first place? For the more subtle
bugs, what data finally made the debugging come together?



[-- Attachment #2: Type: application/pgp-signature, Size: 494 bytes --]

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 2986 bytes --]

On Mon, 18 Oct 2021 16:41:14 +0800, Dongliang Mu said:

> I want to log all the executed instructions of a user process (e.g.,
> poc.c in syzkaller) in the kernel mode and then would like to leverage
> backward analysis to capture the root cause of kernel panic/crash.

> Therefore, I need the instruction-level tracing mechanisms or tools.

Tracing just the instructions won't get you where you want to be if
you're going through this approach.

You *also* need to track all the data - the instruction path inside two
different runs of syzkaller may be essentially identical, but pass 2 different
values as the 3rd parameter of a syscall.

You may also have to deal with insane amounts of data - the actual error could
have been minutes or even hours before, or the interaction between two
different processes.

You probably want to take a *really* close look at how prof and friends avoid
infinite regress when code execution drops inside the prof code, because you're
going to hit the same issues.

Or....

You can work smarter rather than harder, and ask yourself what's the minimum
amount and type of additional information to make a significant improvement in
the debugging of system crashes.

For example, 95% of the time, you can figure out what the bug is by merely
looking at the stack traceback. For most of the rest of the cases, simply
capturing the parameter values from the syscall and the basic info for page
faults and other interrupts is probably sufficient, and you can probably
leverage the audit subsystem for most of that. It can already record syscall
parameters, while logging page faults and other interrupts can probably be done
with prof.

At that point, you don't actually *need* every instruction - only tracing
branch and call instructions is sufficient, because you already know that each
instruction between the target of a branch/call and the next branch/call will
be executed.

Similarly, the lockdep code will catch most locking issues. But it won't flag
issues with data that should be protected with a lock, but are bereft of any
locking. So ask yourself: What ways are there to analyze the code and detect
critical sections prone to race conditions? Is there a sparse-on-steroids
approach that wil do the heavy lifting for those? (Note that this isn't an easy
task for the general case, but identifying two or three specific common
patterns and finding a way to detect them may be worthwhile)

And many of the rest of crashes are timing related, and "let's trace every single
instruction" is almost guaranteed to make things slow enough to change/bypass
the timing issue.

So... What's left that would be the most helpful with the least amount of data?

Go look at some threads on linux-kernel.  Look at the kernel bugs that were the
result of a Homer Simpson "D'oh!" moment.  What can we do to make those
bugs less likely to make it into the code in the first place? For the more subtle
bugs, what data finally made the debugging come together?



[-- Attachment #1.2: Type: application/pgp-signature, Size: 494 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[jim.cromie]

On Sun, Oct 17, 2021 at 8:46 PM Dongliang Mu <mudongliangabcd@gmail.com> wrote:
>
> Hi all,
>
> I am writing to kindly ask one question: is there any tracing
> mechanism in Linux kernel that can trace all the executed instructions
> of a user process? If this user process is run on different
> processors, traces of this process on different processors should be
> also recorded.
>
> Any comment is welcome.
>

take a look at rr-project.org

what rr does

rr aspires to be your primary C/C++ debugging tool for Linux,
replacing — well, enhancing — gdb. You record a failure once, then
debug the recording, deterministically, as many times as you want. The
same execution is replayed every time.

rr also provides efficient reverse execution under gdb. Set
breakpoints and data watchpoints and quickly reverse-execute to where
they were hit.

Re: Any tracing mechanism can track the executed instructions of a user process in the kernel?

[jim.cromie]

On Sun, Oct 17, 2021 at 8:46 PM Dongliang Mu <mudongliangabcd@gmail.com> wrote:
>
> Hi all,
>
> I am writing to kindly ask one question: is there any tracing
> mechanism in Linux kernel that can trace all the executed instructions
> of a user process? If this user process is run on different
> processors, traces of this process on different processors should be
> also recorded.
>
> Any comment is welcome.
>

take a look at rr-project.org

what rr does

rr aspires to be your primary C/C++ debugging tool for Linux,
replacing — well, enhancing — gdb. You record a failure once, then
debug the recording, deterministically, as many times as you want. The
same execution is replayed every time.

rr also provides efficient reverse execution under gdb. Set
breakpoints and data watchpoints and quickly reverse-execute to where
they were hit.

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies