* Re: Why do kprobes and uprobes singlestep? [not found] ` <CALCETrU2Rc4ejSoYyWgbk00U8tSc=aZDaj0mm+Ep62wOirZG7g@mail.gmail.com> @ 2021-03-02 20:24 ` Alexei Starovoitov 2021-03-02 21:02 ` Andy Lutomirski 0 siblings, 1 reply; 8+ messages in thread From: Alexei Starovoitov @ 2021-03-02 20:24 UTC (permalink / raw) To: Andy Lutomirski, bpf Cc: Oleg Nesterov, Masami Hiramatsu, Peter Zijlstra, LKML, Anil S Keshavamurthy, David S. Miller, X86 ML, Andrew Cooper On Tue, Mar 2, 2021 at 10:38 AM Andy Lutomirski <luto@kernel.org> wrote: > > Is there something like a uprobe test suite? How maintained / > actively used is uprobe? uprobe+bpf is heavily used in production. selftests/bpf has only one test for it though. Why are you asking? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why do kprobes and uprobes singlestep? 2021-03-02 20:24 ` Why do kprobes and uprobes singlestep? Alexei Starovoitov @ 2021-03-02 21:02 ` Andy Lutomirski 2021-03-03 1:22 ` Alexei Starovoitov 0 siblings, 1 reply; 8+ messages in thread From: Andy Lutomirski @ 2021-03-02 21:02 UTC (permalink / raw) To: Alexei Starovoitov Cc: Andy Lutomirski, bpf, Oleg Nesterov, Masami Hiramatsu, Peter Zijlstra, LKML, Anil S Keshavamurthy, David S. Miller, X86 ML, Andrew Cooper > On Mar 2, 2021, at 12:24 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Tue, Mar 2, 2021 at 10:38 AM Andy Lutomirski <luto@kernel.org> wrote: >> >> Is there something like a uprobe test suite? How maintained / >> actively used is uprobe? > > uprobe+bpf is heavily used in production. > selftests/bpf has only one test for it though. > > Why are you asking? Because the integration with the x86 entry code is a mess, and I want to know whether to mark it BROKEN or how to make sure the any cleanups actually work. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why do kprobes and uprobes singlestep? 2021-03-02 21:02 ` Andy Lutomirski @ 2021-03-03 1:22 ` Alexei Starovoitov 2021-03-03 1:46 ` Andy Lutomirski 0 siblings, 1 reply; 8+ messages in thread From: Alexei Starovoitov @ 2021-03-03 1:22 UTC (permalink / raw) To: Andy Lutomirski Cc: Andy Lutomirski, bpf, Oleg Nesterov, Masami Hiramatsu, Peter Zijlstra, LKML, Anil S Keshavamurthy, David S. Miller, X86 ML, Andrew Cooper On Tue, Mar 2, 2021 at 1:02 PM Andy Lutomirski <luto@amacapital.net> wrote: > > > > On Mar 2, 2021, at 12:24 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > > > On Tue, Mar 2, 2021 at 10:38 AM Andy Lutomirski <luto@kernel.org> wrote: > >> > >> Is there something like a uprobe test suite? How maintained / > >> actively used is uprobe? > > > > uprobe+bpf is heavily used in production. > > selftests/bpf has only one test for it though. > > > > Why are you asking? > > Because the integration with the x86 entry code is a mess, and I want to know whether to mark it BROKEN or how to make sure the any cleanups actually work. Any test case to repro the issue you found? Is it a bug or just messy code? Nowadays a good chunk of popular applications (python, mysql, etc) has USDTs in them. Issues reported with bcc: https://github.com/iovisor/bcc/issues?q=is%3Aissue+USDT Similar thing with bpftrace. Both standard USDT and semaphore based are used in the wild. uprobe for containers has been a long standing feature request. If you can improve uprobe performance that would be awesome. That's another thing that people report often. We optimized it a bit. More can be done. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why do kprobes and uprobes singlestep? 2021-03-03 1:22 ` Alexei Starovoitov @ 2021-03-03 1:46 ` Andy Lutomirski 2021-03-03 2:18 ` Alexei Starovoitov 0 siblings, 1 reply; 8+ messages in thread From: Andy Lutomirski @ 2021-03-03 1:46 UTC (permalink / raw) To: Alexei Starovoitov Cc: Andy Lutomirski, bpf, Oleg Nesterov, Masami Hiramatsu, Peter Zijlstra, LKML, Anil S Keshavamurthy, David S. Miller, X86 ML, Andrew Cooper > On Mar 2, 2021, at 5:22 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Tue, Mar 2, 2021 at 1:02 PM Andy Lutomirski <luto@amacapital.net> wrote: >> >> >>>> On Mar 2, 2021, at 12:24 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: >>> >>> On Tue, Mar 2, 2021 at 10:38 AM Andy Lutomirski <luto@kernel.org> wrote: >>>> >>>> Is there something like a uprobe test suite? How maintained / >>>> actively used is uprobe? >>> >>> uprobe+bpf is heavily used in production. >>> selftests/bpf has only one test for it though. >>> >>> Why are you asking? >> >> Because the integration with the x86 entry code is a mess, and I want to know whether to mark it BROKEN or how to make sure the any cleanups actually work. > > Any test case to repro the issue you found? > Is it a bug or just messy code? Just messy code. > Nowadays a good chunk of popular applications (python, mysql, etc) has > USDTs in them. > Issues reported with bcc: > https://github.com/iovisor/bcc/issues?q=is%3Aissue+USDT > Similar thing with bpftrace. > Both standard USDT and semaphore based are used in the wild. > uprobe for containers has been a long standing feature request. > If you can improve uprobe performance that would be awesome. > That's another thing that people report often. We optimized it a bit. > More can be done. Wait... USDT is much easier to implement well. Are we talking just USDT or are we talking about general uprobes in which almost any instruction can get probed? If the only users that care about uprobes are doing USDT, we could vastly simplify the implementation and probably make it faster, too. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why do kprobes and uprobes singlestep? 2021-03-03 1:46 ` Andy Lutomirski @ 2021-03-03 2:18 ` Alexei Starovoitov 2021-03-03 13:27 ` Oleg Nesterov 2021-03-03 18:11 ` Daniel Xu 0 siblings, 2 replies; 8+ messages in thread From: Alexei Starovoitov @ 2021-03-03 2:18 UTC (permalink / raw) To: Andy Lutomirski Cc: Andy Lutomirski, bpf, Oleg Nesterov, Masami Hiramatsu, Peter Zijlstra, LKML, Anil S Keshavamurthy, David S. Miller, X86 ML, Andrew Cooper On Tue, Mar 2, 2021 at 5:46 PM Andy Lutomirski <luto@amacapital.net> wrote: > > > > On Mar 2, 2021, at 5:22 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > > > On Tue, Mar 2, 2021 at 1:02 PM Andy Lutomirski <luto@amacapital.net> wrote: > >> > >> > >>>> On Mar 2, 2021, at 12:24 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > >>> > >>> On Tue, Mar 2, 2021 at 10:38 AM Andy Lutomirski <luto@kernel.org> wrote: > >>>> > >>>> Is there something like a uprobe test suite? How maintained / > >>>> actively used is uprobe? > >>> > >>> uprobe+bpf is heavily used in production. > >>> selftests/bpf has only one test for it though. > >>> > >>> Why are you asking? > >> > >> Because the integration with the x86 entry code is a mess, and I want to know whether to mark it BROKEN or how to make sure the any cleanups actually work. > > > > Any test case to repro the issue you found? > > Is it a bug or just messy code? > > Just messy code. > > > Nowadays a good chunk of popular applications (python, mysql, etc) has > > USDTs in them. > > Issues reported with bcc: > > https://github.com/iovisor/bcc/issues?q=is%3Aissue+USDT > > Similar thing with bpftrace. > > Both standard USDT and semaphore based are used in the wild. > > uprobe for containers has been a long standing feature request. > > If you can improve uprobe performance that would be awesome. > > That's another thing that people report often. We optimized it a bit. > > More can be done. > > > Wait... USDT is much easier to implement well. Are we talking just USDT or are we talking about general uprobes in which almost any instruction can get probed? If the only users that care about uprobes are doing USDT, we could vastly simplify the implementation and probably make it faster, too. USDTs are driving the majority of uprobe usage. If they can get faster it will increase their adoption even more. There are certainly cases of normal uprobes. They are at the start of the function 99% of the time. Like the following: "uprobe:/lib64/libc.so:malloc(u64 size):size:size,_ret", "uprobe:/lib64/libc.so:free(void *ptr)::ptr", is common despite its overhead. Here is the most interesting and practical usage of uprobes: https://github.com/iovisor/bcc/blob/master/tools/sslsniff.py and the manpage for the tool: https://github.com/iovisor/bcc/blob/master/tools/sslsniff_example.txt uprobe in the middle of the function is very rare. If the kernel starts rejecting uprobes on some weird instructions I suspect no one will complain. Especially if such tightening will come with performance boost for uprobe on a nop and unprobe at the start (which is typically push or alu on %sp). That would be a great step forward. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why do kprobes and uprobes singlestep? 2021-03-03 2:18 ` Alexei Starovoitov @ 2021-03-03 13:27 ` Oleg Nesterov 2021-03-03 18:11 ` Daniel Xu 1 sibling, 0 replies; 8+ messages in thread From: Oleg Nesterov @ 2021-03-03 13:27 UTC (permalink / raw) To: Alexei Starovoitov Cc: Andy Lutomirski, Andy Lutomirski, bpf, Masami Hiramatsu, Peter Zijlstra, LKML, Anil S Keshavamurthy, David S. Miller, X86 ML, Andrew Cooper On 03/02, Alexei Starovoitov wrote: > > Especially if such tightening will come with performance boost for > uprobe on a nop and unprobe at the start (which is typically push or > alu on %sp). > That would be a great step forward. Just in case, nop and push are emulated without additional overhead. Oleg. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why do kprobes and uprobes singlestep? 2021-03-03 2:18 ` Alexei Starovoitov 2021-03-03 13:27 ` Oleg Nesterov @ 2021-03-03 18:11 ` Daniel Xu 2021-03-03 19:14 ` Andy Lutomirski 1 sibling, 1 reply; 8+ messages in thread From: Daniel Xu @ 2021-03-03 18:11 UTC (permalink / raw) To: Alexei Starovoitov Cc: Andy Lutomirski, Andy Lutomirski, bpf, Oleg Nesterov, Masami Hiramatsu, Peter Zijlstra, LKML, Anil S Keshavamurthy, David S. Miller, X86 ML, Andrew Cooper On Tue, Mar 02, 2021 at 06:18:23PM -0800, Alexei Starovoitov wrote: > On Tue, Mar 2, 2021 at 5:46 PM Andy Lutomirski <luto@amacapital.net> wrote: > > > > > > > On Mar 2, 2021, at 5:22 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > > > > > On Tue, Mar 2, 2021 at 1:02 PM Andy Lutomirski <luto@amacapital.net> wrote: > > >> > > >> > > >>>> On Mar 2, 2021, at 12:24 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > >>> > > >>> On Tue, Mar 2, 2021 at 10:38 AM Andy Lutomirski <luto@kernel.org> wrote: > > >>>> > > >>>> Is there something like a uprobe test suite? How maintained / > > >>>> actively used is uprobe? > > >>> > > >>> uprobe+bpf is heavily used in production. > > >>> selftests/bpf has only one test for it though. > > >>> > > >>> Why are you asking? > > >> > > >> Because the integration with the x86 entry code is a mess, and I want to know whether to mark it BROKEN or how to make sure the any cleanups actually work. > > > > > > Any test case to repro the issue you found? > > > Is it a bug or just messy code? > > > > Just messy code. > > > > > Nowadays a good chunk of popular applications (python, mysql, etc) has > > > USDTs in them. > > > Issues reported with bcc: > > > https://github.com/iovisor/bcc/issues?q=is%3Aissue+USDT > > > Similar thing with bpftrace. > > > Both standard USDT and semaphore based are used in the wild. > > > uprobe for containers has been a long standing feature request. > > > If you can improve uprobe performance that would be awesome. > > > That's another thing that people report often. We optimized it a bit. > > > More can be done. > > > > > > Wait... USDT is much easier to implement well. Are we talking just USDT or are we talking about general uprobes in which almost any instruction can get probed? If the only users that care about uprobes are doing USDT, we could vastly simplify the implementation and probably make it faster, too. > > USDTs are driving the majority of uprobe usage. I'd say 50/50 in my experience. Larger userspace applications using bpf for production monitoring tend to use USDT for stability and ABI reasons (hard for bpf to read C++ classes). Bare uprobes (ie not USDT) are used quite often for ad-hoc production debugging. > If they can get faster it will increase their adoption even more. > There are certainly cases of normal uprobes. > They are at the start of the function 99% of the time. > Like the following: > "uprobe:/lib64/libc.so:malloc(u64 size):size:size,_ret", > "uprobe:/lib64/libc.so:free(void *ptr)::ptr", > is common despite its overhead. > > Here is the most interesting and practical usage of uprobes: > https://github.com/iovisor/bcc/blob/master/tools/sslsniff.py > and the manpage for the tool: > https://github.com/iovisor/bcc/blob/master/tools/sslsniff_example.txt > > uprobe in the middle of the function is very rare. > If the kernel starts rejecting uprobes on some weird instructions > I suspect no one will complain. I think it would be great if the kernel could reject mid-instruction uprobes. Unlike with kprobes, you can place uprobes on immediate operands which can cause silent data corruption. See https://github.com/iovisor/bpftrace/pull/803#issuecomment-507693933 for a funny example. To prevent accidental (and silent) data corruption, bpftrace uses a disassembler to ensure uprobes are placed on instruction boundaries. <...> Daniel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why do kprobes and uprobes singlestep? 2021-03-03 18:11 ` Daniel Xu @ 2021-03-03 19:14 ` Andy Lutomirski 0 siblings, 0 replies; 8+ messages in thread From: Andy Lutomirski @ 2021-03-03 19:14 UTC (permalink / raw) To: Daniel Xu Cc: Alexei Starovoitov, Andy Lutomirski, bpf, Oleg Nesterov, Masami Hiramatsu, Peter Zijlstra, LKML, Anil S Keshavamurthy, David S. Miller, X86 ML, Andrew Cooper > On Mar 3, 2021, at 10:11 AM, Daniel Xu <dxu@dxuuu.xyz> wrote: > > On Tue, Mar 02, 2021 at 06:18:23PM -0800, Alexei Starovoitov wrote: >>> On Tue, Mar 2, 2021 at 5:46 PM Andy Lutomirski <luto@amacapital.net> wrote: >>> >>> >>>> On Mar 2, 2021, at 5:22 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: >>>> >>>> On Tue, Mar 2, 2021 at 1:02 PM Andy Lutomirski <luto@amacapital.net> wrote: >>>>> >>>>> >>>>>>> On Mar 2, 2021, at 12:24 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: >>>>>> >>>>>> On Tue, Mar 2, 2021 at 10:38 AM Andy Lutomirski <luto@kernel.org> wrote: >>>>>>> >>>>>>> Is there something like a uprobe test suite? How maintained / >>>>>>> actively used is uprobe? >>>>>> >>>>>> uprobe+bpf is heavily used in production. >>>>>> selftests/bpf has only one test for it though. >>>>>> >>>>>> Why are you asking? >>>>> >>>>> Because the integration with the x86 entry code is a mess, and I want to know whether to mark it BROKEN or how to make sure the any cleanups actually work. >>>> >>>> Any test case to repro the issue you found? >>>> Is it a bug or just messy code? >>> >>> Just messy code. >>> >>>> Nowadays a good chunk of popular applications (python, mysql, etc) has >>>> USDTs in them. >>>> Issues reported with bcc: >>>> https://github.com/iovisor/bcc/issues?q=is%3Aissue+USDT >>>> Similar thing with bpftrace. >>>> Both standard USDT and semaphore based are used in the wild. >>>> uprobe for containers has been a long standing feature request. >>>> If you can improve uprobe performance that would be awesome. >>>> That's another thing that people report often. We optimized it a bit. >>>> More can be done. >>> >>> >>> Wait... USDT is much easier to implement well. Are we talking just USDT or are we talking about general uprobes in which almost any instruction can get probed? If the only users that care about uprobes are doing USDT, we could vastly simplify the implementation and probably make it faster, too. >> >> USDTs are driving the majority of uprobe usage. > > I'd say 50/50 in my experience. Larger userspace applications using bpf > for production monitoring tend to use USDT for stability and ABI reasons > (hard for bpf to read C++ classes). Bare uprobes (ie not USDT) are used > quite often for ad-hoc production debugging. > >> If they can get faster it will increase their adoption even more. >> There are certainly cases of normal uprobes. >> They are at the start of the function 99% of the time. >> Like the following: >> "uprobe:/lib64/libc.so:malloc(u64 size):size:size,_ret", >> "uprobe:/lib64/libc.so:free(void *ptr)::ptr", >> is common despite its overhead. >> >> Here is the most interesting and practical usage of uprobes: >> https://github.com/iovisor/bcc/blob/master/tools/sslsniff.py >> and the manpage for the tool: >> https://github.com/iovisor/bcc/blob/master/tools/sslsniff_example.txt >> >> uprobe in the middle of the function is very rare. >> If the kernel starts rejecting uprobes on some weird instructions >> I suspect no one will complain. > > I think it would be great if the kernel could reject mid-instruction > uprobes. Unlike with kprobes, you can place uprobes on immediate > operands which can cause silent data corruption. See > https://github.com/iovisor/bpftrace/pull/803#issuecomment-507693933 > for a funny example. This can’t be done in general on x86. One cannot look at code and find the instruction boundaries. > > To prevent accidental (and silent) data corruption, bpftrace uses a > disassembler to ensure uprobes are placed on instruction boundaries. > > <...> > > Daniel ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-03-03 22:57 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CALCETrXzXv-V3A3SpN_Pdj_PNG8Gw0AVsZD7+VO-q_xCAu2T2A@mail.gmail.com> [not found] ` <20210301165130.GA5351@redhat.com> [not found] ` <CALCETrU2Rc4ejSoYyWgbk00U8tSc=aZDaj0mm+Ep62wOirZG7g@mail.gmail.com> 2021-03-02 20:24 ` Why do kprobes and uprobes singlestep? Alexei Starovoitov 2021-03-02 21:02 ` Andy Lutomirski 2021-03-03 1:22 ` Alexei Starovoitov 2021-03-03 1:46 ` Andy Lutomirski 2021-03-03 2:18 ` Alexei Starovoitov 2021-03-03 13:27 ` Oleg Nesterov 2021-03-03 18:11 ` Daniel Xu 2021-03-03 19:14 ` Andy Lutomirski
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).