* [PATCH 0/2] ftrace: two fixes with func_probes handling @ 2019-07-04 14:34 Naveen N. Rao 2019-07-04 14:34 ` [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() Naveen N. Rao ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Naveen N. Rao @ 2019-07-04 14:34 UTC (permalink / raw) To: Steven Rostedt; +Cc: linuxppc-dev, linux-kernel Two patches addressing bugs in ftrace function probe handling. The first patch addresses a NULL pointer dereference reported by LTP tests, while the second one is a trivial patch to address a missing check for return value, found by code inspection. - Naveen Naveen N. Rao (2): ftrace: Fix NULL pointer dereference in t_probe_next() ftrace: Check for successful allocation of hash kernel/trace/ftrace.c | 9 +++++++++ 1 file changed, 9 insertions(+) -- 2.22.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() 2019-07-04 14:34 [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao @ 2019-07-04 14:34 ` Naveen N. Rao 2019-08-30 20:16 ` Steven Rostedt 2019-08-30 20:47 ` Steven Rostedt 2019-07-04 14:34 ` [PATCH 2/2] ftrace: Check for successful allocation of hash Naveen N. Rao 2019-08-08 15:15 ` [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao 2 siblings, 2 replies; 9+ messages in thread From: Naveen N. Rao @ 2019-07-04 14:34 UTC (permalink / raw) To: Steven Rostedt; +Cc: linuxppc-dev, linux-kernel LTP testsuite on powerpc results in the below crash: Unable to handle kernel paging request for data at address 0x00000000 Faulting instruction address: 0xc00000000029d800 Oops: Kernel access of bad area, sig: 11 [#1] LE SMP NR_CPUS=2048 NUMA PowerNV ... CPU: 68 PID: 96584 Comm: cat Kdump: loaded Tainted: G W NIP: c00000000029d800 LR: c00000000029dac4 CTR: c0000000001e6ad0 REGS: c0002017fae8ba10 TRAP: 0300 Tainted: G W MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28022422 XER: 20040000 CFAR: c00000000029d90c DAR: 0000000000000000 DSISR: 40000000 IRQMASK: 0 ... NIP [c00000000029d800] t_probe_next+0x60/0x180 LR [c00000000029dac4] t_mod_start+0x1a4/0x1f0 Call Trace: [c0002017fae8bc90] [c000000000cdbc40] _cond_resched+0x10/0xb0 (unreliable) [c0002017fae8bce0] [c0000000002a15b0] t_start+0xf0/0x1c0 [c0002017fae8bd30] [c0000000004ec2b4] seq_read+0x184/0x640 [c0002017fae8bdd0] [c0000000004a57bc] sys_read+0x10c/0x300 [c0002017fae8be30] [c00000000000b388] system_call+0x5c/0x70 The test (ftrace_set_ftrace_filter.sh) is part of ftrace stress tests and the crash happens when the test does 'cat $TRACING_PATH/set_ftrace_filter'. The address points to the second line below, in t_probe_next(), where filter_hash is dereferenced: hash = iter->probe->ops.func_hash->filter_hash; size = 1 << hash->size_bits; This happens due to a race with register_ftrace_function_probe(). A new ftrace_func_probe is created and added into the func_probes list in trace_array under ftrace_lock. However, before initializing the filter, we drop ftrace_lock, and re-acquire it after acquiring regex_lock. If another process is trying to read set_ftrace_filter, it will be able to acquire ftrace_lock during this window and it will end up seeing a NULL filter_hash. Fix this by just checking for a NULL filter_hash in t_probe_next(). If the filter_hash is NULL, then this probe is just being added and we can simply return from here. Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> --- kernel/trace/ftrace.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 7b037295a1f1..0791eafb693d 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -3093,6 +3093,10 @@ t_probe_next(struct seq_file *m, loff_t *pos) hnd = &iter->probe_entry->hlist; hash = iter->probe->ops.func_hash->filter_hash; + + if (!hash) + return NULL; + size = 1 << hash->size_bits; retry: -- 2.22.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() 2019-07-04 14:34 ` [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() Naveen N. Rao @ 2019-08-30 20:16 ` Steven Rostedt 2019-08-30 20:47 ` Steven Rostedt 1 sibling, 0 replies; 9+ messages in thread From: Steven Rostedt @ 2019-08-30 20:16 UTC (permalink / raw) To: Naveen N. Rao; +Cc: linuxppc-dev, linux-kernel On Thu, 4 Jul 2019 20:04:41 +0530 "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> wrote: > LTP testsuite on powerpc results in the below crash: > > Unable to handle kernel paging request for data at address 0x00000000 > Faulting instruction address: 0xc00000000029d800 > Oops: Kernel access of bad area, sig: 11 [#1] > LE SMP NR_CPUS=2048 NUMA PowerNV > ... > CPU: 68 PID: 96584 Comm: cat Kdump: loaded Tainted: G W > NIP: c00000000029d800 LR: c00000000029dac4 CTR: c0000000001e6ad0 > REGS: c0002017fae8ba10 TRAP: 0300 Tainted: G W > MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28022422 XER: 20040000 > CFAR: c00000000029d90c DAR: 0000000000000000 DSISR: 40000000 IRQMASK: 0 > ... > NIP [c00000000029d800] t_probe_next+0x60/0x180 > LR [c00000000029dac4] t_mod_start+0x1a4/0x1f0 > Call Trace: > [c0002017fae8bc90] [c000000000cdbc40] _cond_resched+0x10/0xb0 (unreliable) > [c0002017fae8bce0] [c0000000002a15b0] t_start+0xf0/0x1c0 > [c0002017fae8bd30] [c0000000004ec2b4] seq_read+0x184/0x640 > [c0002017fae8bdd0] [c0000000004a57bc] sys_read+0x10c/0x300 > [c0002017fae8be30] [c00000000000b388] system_call+0x5c/0x70 > > The test (ftrace_set_ftrace_filter.sh) is part of ftrace stress tests > and the crash happens when the test does 'cat > $TRACING_PATH/set_ftrace_filter'. > > The address points to the second line below, in t_probe_next(), where > filter_hash is dereferenced: > hash = iter->probe->ops.func_hash->filter_hash; > size = 1 << hash->size_bits; > > This happens due to a race with register_ftrace_function_probe(). A new > ftrace_func_probe is created and added into the func_probes list in > trace_array under ftrace_lock. However, before initializing the filter, > we drop ftrace_lock, and re-acquire it after acquiring regex_lock. If > another process is trying to read set_ftrace_filter, it will be able to > acquire ftrace_lock during this window and it will end up seeing a NULL > filter_hash. > > Fix this by just checking for a NULL filter_hash in t_probe_next(). If > the filter_hash is NULL, then this probe is just being added and we can > simply return from here. Hmm, this is very subtle. I'll take a deeper look at this to see if we can keep the race from happening. Thanks! -- Steve > > Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> > --- > kernel/trace/ftrace.c | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c > index 7b037295a1f1..0791eafb693d 100644 > --- a/kernel/trace/ftrace.c > +++ b/kernel/trace/ftrace.c > @@ -3093,6 +3093,10 @@ t_probe_next(struct seq_file *m, loff_t *pos) > hnd = &iter->probe_entry->hlist; > > hash = iter->probe->ops.func_hash->filter_hash; > + > + if (!hash) > + return NULL; > + > size = 1 << hash->size_bits; > > retry: ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() 2019-07-04 14:34 ` [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() Naveen N. Rao 2019-08-30 20:16 ` Steven Rostedt @ 2019-08-30 20:47 ` Steven Rostedt 2019-09-04 8:07 ` Naveen N. Rao 1 sibling, 1 reply; 9+ messages in thread From: Steven Rostedt @ 2019-08-30 20:47 UTC (permalink / raw) To: Naveen N. Rao; +Cc: linuxppc-dev, linux-kernel On Thu, 4 Jul 2019 20:04:41 +0530 "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> wrote: > kernel/trace/ftrace.c | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c > index 7b037295a1f1..0791eafb693d 100644 > --- a/kernel/trace/ftrace.c > +++ b/kernel/trace/ftrace.c > @@ -3093,6 +3093,10 @@ t_probe_next(struct seq_file *m, loff_t *pos) > hnd = &iter->probe_entry->hlist; > > hash = iter->probe->ops.func_hash->filter_hash; > + > + if (!hash) > + return NULL; > + > size = 1 << hash->size_bits; > > retry: OK, I added this, but I'm also adding this on top: -- Steve From 372e0d01da71c84dcecf7028598a33813b0d5256 Mon Sep 17 00:00:00 2001 From: "Steven Rostedt (VMware)" <rostedt@goodmis.org> Date: Fri, 30 Aug 2019 16:30:01 -0400 Subject: [PATCH] ftrace: Check for empty hash and comment the race with registering probes The race between adding a function probe and reading the probes that exist is very subtle. It needs a comment. Also, the issue can also happen if the probe has has the EMPTY_HASH as its func_hash. Cc: stable@vger.kernel.org Fixes: 7b60f3d876156 ("ftrace: Dynamically create the probe ftrace_ops for the trace_array") Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org> --- kernel/trace/ftrace.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 80beed2cf0da..6200a6fe10e3 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -3096,7 +3096,11 @@ t_probe_next(struct seq_file *m, loff_t *pos) hash = iter->probe->ops.func_hash->filter_hash; - if (!hash) + /* + * A probe being registered may temporarily have an empty hash + * and it's at the end of the func_probes list. + */ + if (!hash || hash == EMPTY_HASH) return NULL; size = 1 << hash->size_bits; @@ -4324,6 +4328,10 @@ register_ftrace_function_probe(char *glob, struct trace_array *tr, mutex_unlock(&ftrace_lock); + /* + * Note, there's a small window here that the func_hash->filter_hash + * may be NULL or empty. Need to be carefule when reading the loop. + */ mutex_lock(&probe->ops.func_hash->regex_lock); orig_hash = &probe->ops.func_hash->filter_hash; -- 2.20.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() 2019-08-30 20:47 ` Steven Rostedt @ 2019-09-04 8:07 ` Naveen N. Rao 0 siblings, 0 replies; 9+ messages in thread From: Naveen N. Rao @ 2019-09-04 8:07 UTC (permalink / raw) To: Steven Rostedt; +Cc: linuxppc-dev, linux-kernel Steven Rostedt wrote: > On Thu, 4 Jul 2019 20:04:41 +0530 > "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> wrote: > > >> kernel/trace/ftrace.c | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c >> index 7b037295a1f1..0791eafb693d 100644 >> --- a/kernel/trace/ftrace.c >> +++ b/kernel/trace/ftrace.c >> @@ -3093,6 +3093,10 @@ t_probe_next(struct seq_file *m, loff_t *pos) >> hnd = &iter->probe_entry->hlist; >> >> hash = iter->probe->ops.func_hash->filter_hash; >> + >> + if (!hash) >> + return NULL; >> + >> size = 1 << hash->size_bits; >> >> retry: > > OK, I added this, but I'm also adding this on top: Thanks, the additional comments do make this much clearer. Regards, Naveen ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/2] ftrace: Check for successful allocation of hash 2019-07-04 14:34 [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao 2019-07-04 14:34 ` [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() Naveen N. Rao @ 2019-07-04 14:34 ` Naveen N. Rao 2019-08-08 15:15 ` [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao 2 siblings, 0 replies; 9+ messages in thread From: Naveen N. Rao @ 2019-07-04 14:34 UTC (permalink / raw) To: Steven Rostedt; +Cc: linuxppc-dev, linux-kernel In register_ftrace_function_probe(), we are not checking the return value of alloc_and_copy_ftrace_hash(). The subsequent call to ftrace_match_records() may end up dereferencing the same. Add a check to ensure this doesn't happen. Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> --- kernel/trace/ftrace.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 0791eafb693d..0d5f7d4a4936 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -4328,6 +4328,11 @@ register_ftrace_function_probe(char *glob, struct trace_array *tr, old_hash = *orig_hash; hash = alloc_and_copy_ftrace_hash(FTRACE_HASH_DEFAULT_BITS, old_hash); + if (!hash) { + ret = -ENOMEM; + goto out; + } + ret = ftrace_match_records(hash, glob, strlen(glob)); /* Nothing found? */ -- 2.22.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 0/2] ftrace: two fixes with func_probes handling 2019-07-04 14:34 [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao 2019-07-04 14:34 ` [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() Naveen N. Rao 2019-07-04 14:34 ` [PATCH 2/2] ftrace: Check for successful allocation of hash Naveen N. Rao @ 2019-08-08 15:15 ` Naveen N. Rao 2019-08-08 15:17 ` Steven Rostedt 2019-08-30 20:14 ` Steven Rostedt 2 siblings, 2 replies; 9+ messages in thread From: Naveen N. Rao @ 2019-08-08 15:15 UTC (permalink / raw) To: Steven Rostedt; +Cc: linuxppc-dev, linux-kernel Naveen N. Rao wrote: > Two patches addressing bugs in ftrace function probe handling. The first > patch addresses a NULL pointer dereference reported by LTP tests, while > the second one is a trivial patch to address a missing check for return > value, found by code inspection. Steven, Can you please take a look at these patches? Thanks, Naveen ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 0/2] ftrace: two fixes with func_probes handling 2019-08-08 15:15 ` [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao @ 2019-08-08 15:17 ` Steven Rostedt 2019-08-30 20:14 ` Steven Rostedt 1 sibling, 0 replies; 9+ messages in thread From: Steven Rostedt @ 2019-08-08 15:17 UTC (permalink / raw) To: Naveen N. Rao; +Cc: linuxppc-dev, linux-kernel On Thu, 08 Aug 2019 20:45:04 +0530 "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> wrote: > Naveen N. Rao wrote: > > Two patches addressing bugs in ftrace function probe handling. The first > > patch addresses a NULL pointer dereference reported by LTP tests, while > > the second one is a trivial patch to address a missing check for return > > value, found by code inspection. > > Steven, > Can you please take a look at these patches? Thanks for the ping. Yes I will. -- Steve ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 0/2] ftrace: two fixes with func_probes handling 2019-08-08 15:15 ` [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao 2019-08-08 15:17 ` Steven Rostedt @ 2019-08-30 20:14 ` Steven Rostedt 1 sibling, 0 replies; 9+ messages in thread From: Steven Rostedt @ 2019-08-30 20:14 UTC (permalink / raw) To: Naveen N. Rao; +Cc: linuxppc-dev, linux-kernel On Thu, 08 Aug 2019 20:45:04 +0530 "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> wrote: > Naveen N. Rao wrote: > > Two patches addressing bugs in ftrace function probe handling. The first > > patch addresses a NULL pointer dereference reported by LTP tests, while > > the second one is a trivial patch to address a missing check for return > > value, found by code inspection. > > Steven, > Can you please take a look at these patches? > Sorry for the late reply, I've been traveling a lot lately. I'm looking at these now. I'm trying to see how they triggered a bug. -- Steve ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-09-04 8:10 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-07-04 14:34 [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao 2019-07-04 14:34 ` [PATCH 1/2] ftrace: Fix NULL pointer dereference in t_probe_next() Naveen N. Rao 2019-08-30 20:16 ` Steven Rostedt 2019-08-30 20:47 ` Steven Rostedt 2019-09-04 8:07 ` Naveen N. Rao 2019-07-04 14:34 ` [PATCH 2/2] ftrace: Check for successful allocation of hash Naveen N. Rao 2019-08-08 15:15 ` [PATCH 0/2] ftrace: two fixes with func_probes handling Naveen N. Rao 2019-08-08 15:17 ` Steven Rostedt 2019-08-30 20:14 ` Steven Rostedt
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).