From: Ming Lei <ming.lei@redhat.com>
To: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Ming Lei <tom.leiming@gmail.com>,
"Naveen N. Rao" <naveen.n.rao@linux.ibm.com>,
Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Steven Rostedt <rostedt@goodmis.org>,
linux-block <linux-block@vger.kernel.org>
Subject: Re: kprobe: __blkdev_put probe is missed
Date: Fri, 19 Jun 2020 07:19:01 +0800 [thread overview]
Message-ID: <20200618231901.GA196099@T590> (raw)
In-Reply-To: <20200618225602.3f2cca3f0ed48427fc0a483b@kernel.org>
On Thu, Jun 18, 2020 at 10:56:02PM +0900, Masami Hiramatsu wrote:
> Hi Ming,
>
> On Thu, 18 Jun 2020 20:54:38 +0800
> Ming Lei <ming.lei@redhat.com> wrote:
>
> > On Wed, Jun 17, 2020 at 06:30:39PM +0800, Ming Lei wrote:
> > > Hello Guys,
> > >
> > > I found probe on __blkdev_put is missed, which can be observed
> > > via bcc/perf reliably:
> > >
> > > 1) start trace
> > > - perf probe __blkdev_put
> > > - perf trace -a -e probe:__blkdev_put
>
> Could you dump the kprobe_event as below?
>
> # cat /sys/kernel/tracing/kprobe_events
>
>
> > >
> > > or
> > >
> > > /usr/share/bcc/tools/stackcount __blkdev_put
> > >
> > > 2) run the following command:
> > > blockdev --getbsz /dev/sda1
>
> And dump the kprobe profile?
>
> # cat /sys/kernel/tracing/kprobe_profile
>
> > >
> > > 3) 'perf trace' or stackcount just dumps one trace event, and it
> > > should have been two
> > > __blkdev_put() traces, since one __blkdev_put() is called for
> > > partition(/dev/sda1),
> > > and another is for disk(/dev/sda). If trace_printk() is added in __blkdev_put(),
> > > two events will be captured from ftrace.
> > >
> >
> > The issue can be shown by loading a kprobe module which registers on
> > __blkdev_put(), just by replacing _do_fork with __blkdev_put on
> > samples/kprobes/kprobe_example.c.
>
> Could you tell me what kernel are you using?
>
> I'm using 5.4 on ubuntu and can not reproduce it with kprobe_event.
>
> root@devnote2:/sys/kernel/tracing# uname -a
> Linux devnote2 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> root@devnote2:/sys/kernel/tracing# echo p __blkdev_put > kprobe_events
> root@devnote2:/sys/kernel/tracing# echo 1 > events/kprobes/p___blkdev_put_0/enable
> root@devnote2:/sys/kernel/tracing# cat trace
> # tracer: nop
> #
> # entries-in-buffer/entries-written: 0/0 #P:8
> #
> # _-----=> irqs-off
> # / _----=> need-resched
> # | / _---=> hardirq/softirq
> # || / _--=> preempt-depth
> # ||| / delay
> # TASK-PID CPU# |||| TIMESTAMP FUNCTION
> # | | | |||| | |
> root@devnote2:/sys/kernel/tracing# blockdev --getbsz /dev/nvme0n1
> 4096
> root@devnote2:/sys/kernel/tracing# cat trace
> # tracer: nop
> #
> # entries-in-buffer/entries-written: 1/1 #P:8
> #
> # _-----=> irqs-off
> # / _----=> need-resched
> # | / _---=> hardirq/softirq
> # || / _--=> preempt-depth
> # ||| / delay
> # TASK-PID CPU# |||| TIMESTAMP FUNCTION
> # | | | |||| | |
> <...>-111740 [002] .... 301734.476991: p___blkdev_put_0: (__blkdev_put+0x0/0x1e0)
>
> Hmm, maybe some issue in the latest kernel...?
Hello Masami,
I am testing the latest upstream kernel, your trace actually reproduces
this issue.
After 'blockdev --getbsz /dev/nvme0n1' returns, __blkdev_put() should
have been called two times(one for partition, and the other for disk),
however kprobe trace just shows one time of calling this function.
If trace_printk() is added at the entry of __blkdev_put() manually,
you will see that __blkdev_put() is called two times in 'blockdev
--getbsz /dev/nvme0n1'.
Thanks,
Ming
next prev parent reply other threads:[~2020-06-18 23:19 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-17 10:30 krobe: __blkdev_put probe is missed Ming Lei
2020-06-18 12:54 ` kprobe: " Ming Lei
2020-06-18 13:56 ` Masami Hiramatsu
2020-06-18 23:19 ` Ming Lei [this message]
2020-06-19 5:12 ` Masami Hiramatsu
2020-06-19 7:28 ` Ming Lei
2020-06-19 12:19 ` Steven Rostedt
2020-06-19 13:32 ` Ming Lei
2020-06-19 15:35 ` Masami Hiramatsu
2020-06-19 23:28 ` Ming Lei
2020-06-20 0:59 ` Steven Rostedt
2020-06-20 1:37 ` Masami Hiramatsu
2020-06-22 0:27 ` Ming Lei
2020-06-22 1:34 ` Masami Hiramatsu
2020-06-22 13:01 ` Steven Rostedt
2020-06-22 23:47 ` Masami Hiramatsu
2020-06-23 0:38 ` Masami Hiramatsu
2020-06-23 5:28 ` Masami Hiramatsu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200618231901.GA196099@T590 \
--to=ming.lei@redhat.com \
--cc=anil.s.keshavamurthy@intel.com \
--cc=davem@davemloft.net \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=naveen.n.rao@linux.ibm.com \
--cc=rostedt@goodmis.org \
--cc=tom.leiming@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).