* [PATCH] trace: fix race in perf_trace_buf initialization
@ 2019-10-15 6:17 Prateek Sood
2019-10-17 6:53 ` Prateek Sood
2019-10-18 21:12 ` Steven Rostedt
0 siblings, 2 replies; 7+ messages in thread
From: Prateek Sood @ 2019-10-15 6:17 UTC (permalink / raw)
To: rostedt, mingo; +Cc: linux-kernel, kaushalk, Prateek Sood
[ 943.034988] Unable to handle kernel paging request at virtual address 0000003106f2003c
[ 943.043653] Mem abort info:
[ 943.046679] ESR = 0x96000045
[ 943.050428] Exception class = DABT (current EL), IL = 32 bits
[ 943.056643] SET = 0, FnV = 0
[ 943.060168] EA = 0, S1PTW = 0
[ 943.063449] Data abort info:
[ 943.066474] ISV = 0, ISS = 0x00000045
[ 943.070856] CM = 0, WnR = 1
[ 943.074016] user pgtable: 4k pages, 39-bit VAs, pgdp = ffffffc034b9b000
[ 943.081446] [0000003106f2003c] pgd=0000000000000000, pud=0000000000000000
[ 943.088862] Internal error: Oops: 96000045 [#1] PREEMPT SMP
[ 943.141700] Process syz-executor (pid: 18393, stack limit = 0xffffffc093190000)
[ 943.164146] pstate: 80400005 (Nzcv daif +PAN -UAO)
[ 943.169119] pc : __memset+0x20/0x1ac
[ 943.172831] lr : memset+0x3c/0x50
[ 943.176269] sp : ffffffc09319fc50
[ 943.557593] __memset+0x20/0x1ac
[ 943.560953] perf_trace_buf_alloc+0x140/0x1a0
[ 943.565472] perf_trace_sys_enter+0x158/0x310
[ 943.569985] syscall_trace_enter+0x348/0x7c0
[ 943.574413] el0_svc_common+0x11c/0x368
[ 943.578394] el0_svc_handler+0x12c/0x198
[ 943.582459] el0_svc+0x8/0xc
In Ramdumps:
total_ref_count = 3
perf_trace_buf = (
0x0 -> NULL,
0x0 -> NULL,
0x0 -> NULL,
0x0 -> NULL)
event_call in perf_trace_sys_enter()
event_call = 0xFFFFFF900CB511D8 -> (
list = (next = 0xFFFFFF900CB4E2E0, prev = 0xFFFFFF900CB512B0),
class = 0xFFFFFF900CDC8308,
name = 0xFFFFFF900CDDA1D8,
tp = 0xFFFFFF900CDDA1D8,
event = (
node = (next = 0x0, pprev = 0xFFFFFF900CB80210),
list = (next = 0xFFFFFF900CB512E0, prev = 0xFFFFFF900CB4E310),
type = 21,
funcs = 0xFFFFFF900CB51130),
print_fmt = 0xFFFFFF900CB51150,
filter = 0x0,
mod = 0x0,
data = 0x0,
flags = 18,
perf_refcount = 1,
perf_events = 0xFFFFFF8DB8E54158,
prog_array = 0x0,
perf_perm = 0x0)
perf_events added on CPU0
(struct hlist_head *)(0xFFFFFF8DB8E54158+__per_cpu_offset[0]) -> (
first = 0xFFFFFFC0980FD0E0 -> (
next = 0x0,
pprev = 0xFFFFFFBEBFD74158))
Could you please confirm:
1) the race mentioned below exists or not.
2) if following patch fixes it.
>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8
A race condition exists while initialiazing perf_trace_buf from
perf_trace_init() and perf_kprobe_init().
CPU0 CPU1
perf_trace_init()
mutex_lock(&event_mutex)
perf_trace_event_init()
perf_trace_event_reg()
total_ref_count == 0
buf = alloc_percpu()
perf_trace_buf[i] = buf
tp_event->class->reg() //fails perf_kprobe_init()
goto fail perf_trace_event_init()
perf_trace_event_reg()
fail:
total_ref_count == 0
total_ref_count == 0
buf = alloc_percpu()
perf_trace_buf[i] = buf
tp_event->class->reg()
total_ref_count++
free_percpu(perf_trace_buf[i])
perf_trace_buf[i] = NULL
Any subsequent call to perf_trace_event_reg() will observe total_ref_count > 0,
causing the perf_trace_buf to be NULL always. This can result in perf_trace_buf
getting accessed from perf_trace_buf_alloc() without being initialized. Acquiring
event_mutex in perf_kprobe_init() before calling perf_trace_event_init() should
fix this race.
Signed-off-by: Prateek Sood <prsood@codeaurora.org>
---
kernel/trace/trace_event_perf.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
index 4629a61..48ee92c 100644
--- a/kernel/trace/trace_event_perf.c
+++ b/kernel/trace/trace_event_perf.c
@@ -272,9 +272,11 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
goto out;
}
+ mutex_lock(&event_mutex);
ret = perf_trace_event_init(tp_event, p_event);
if (ret)
destroy_local_trace_kprobe(tp_event);
+ mutex_unlock(&event_mutex);
out:
kfree(func);
return ret;
@@ -282,8 +284,10 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
void perf_kprobe_destroy(struct perf_event *p_event)
{
+ mutex_lock(&event_mutex);
perf_trace_event_close(p_event);
perf_trace_event_unreg(p_event);
+ mutex_unlock(&event_mutex);
destroy_local_trace_kprobe(p_event->tp_event);
}
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc.,
is a member of Code Aurora Forum, a Linux Foundation Collaborative Project.
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] trace: fix race in perf_trace_buf initialization
2019-10-15 6:17 [PATCH] trace: fix race in perf_trace_buf initialization Prateek Sood
@ 2019-10-17 6:53 ` Prateek Sood
2019-10-17 12:34 ` Steven Rostedt
2019-10-18 21:12 ` Steven Rostedt
1 sibling, 1 reply; 7+ messages in thread
From: Prateek Sood @ 2019-10-17 6:53 UTC (permalink / raw)
To: rostedt, mingo; +Cc: linux-kernel, kaushalk
On 10/15/19 11:47 AM, Prateek Sood wrote:
> [ 943.034988] Unable to handle kernel paging request at virtual address 0000003106f2003c
> [ 943.043653] Mem abort info:
> [ 943.046679] ESR = 0x96000045
> [ 943.050428] Exception class = DABT (current EL), IL = 32 bits
> [ 943.056643] SET = 0, FnV = 0
> [ 943.060168] EA = 0, S1PTW = 0
> [ 943.063449] Data abort info:
> [ 943.066474] ISV = 0, ISS = 0x00000045
> [ 943.070856] CM = 0, WnR = 1
> [ 943.074016] user pgtable: 4k pages, 39-bit VAs, pgdp = ffffffc034b9b000
> [ 943.081446] [0000003106f2003c] pgd=0000000000000000, pud=0000000000000000
> [ 943.088862] Internal error: Oops: 96000045 [#1] PREEMPT SMP
> [ 943.141700] Process syz-executor (pid: 18393, stack limit = 0xffffffc093190000)
> [ 943.164146] pstate: 80400005 (Nzcv daif +PAN -UAO)
> [ 943.169119] pc : __memset+0x20/0x1ac
> [ 943.172831] lr : memset+0x3c/0x50
> [ 943.176269] sp : ffffffc09319fc50
>
> [ 943.557593] __memset+0x20/0x1ac
> [ 943.560953] perf_trace_buf_alloc+0x140/0x1a0
> [ 943.565472] perf_trace_sys_enter+0x158/0x310
> [ 943.569985] syscall_trace_enter+0x348/0x7c0
> [ 943.574413] el0_svc_common+0x11c/0x368
> [ 943.578394] el0_svc_handler+0x12c/0x198
> [ 943.582459] el0_svc+0x8/0xc
>
> In Ramdumps:
> total_ref_count = 3
> perf_trace_buf = (
> 0x0 -> NULL,
> 0x0 -> NULL,
> 0x0 -> NULL,
> 0x0 -> NULL)
>
> event_call in perf_trace_sys_enter()
> event_call = 0xFFFFFF900CB511D8 -> (
> list = (next = 0xFFFFFF900CB4E2E0, prev = 0xFFFFFF900CB512B0),
> class = 0xFFFFFF900CDC8308,
> name = 0xFFFFFF900CDDA1D8,
> tp = 0xFFFFFF900CDDA1D8,
> event = (
> node = (next = 0x0, pprev = 0xFFFFFF900CB80210),
> list = (next = 0xFFFFFF900CB512E0, prev = 0xFFFFFF900CB4E310),
> type = 21,
> funcs = 0xFFFFFF900CB51130),
> print_fmt = 0xFFFFFF900CB51150,
> filter = 0x0,
> mod = 0x0,
> data = 0x0,
> flags = 18,
> perf_refcount = 1,
> perf_events = 0xFFFFFF8DB8E54158,
> prog_array = 0x0,
> perf_perm = 0x0)
>
> perf_events added on CPU0
> (struct hlist_head *)(0xFFFFFF8DB8E54158+__per_cpu_offset[0]) -> (
> first = 0xFFFFFFC0980FD0E0 -> (
> next = 0x0,
> pprev = 0xFFFFFFBEBFD74158))
>
> Could you please confirm:
> 1) the race mentioned below exists or not.
> 2) if following patch fixes it.
>
>
>> 8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8
>
> A race condition exists while initialiazing perf_trace_buf from
> perf_trace_init() and perf_kprobe_init().
>
> CPU0 CPU1
> perf_trace_init()
> mutex_lock(&event_mutex)
> perf_trace_event_init()
> perf_trace_event_reg()
> total_ref_count == 0
> buf = alloc_percpu()
> perf_trace_buf[i] = buf
> tp_event->class->reg() //fails perf_kprobe_init()
> goto fail perf_trace_event_init()
> perf_trace_event_reg()
> fail:
> total_ref_count == 0
>
> total_ref_count == 0
> buf = alloc_percpu()
> perf_trace_buf[i] = buf
> tp_event->class->reg()
> total_ref_count++
>
> free_percpu(perf_trace_buf[i])
> perf_trace_buf[i] = NULL
>
> Any subsequent call to perf_trace_event_reg() will observe total_ref_count > 0,
> causing the perf_trace_buf to be NULL always. This can result in perf_trace_buf
> getting accessed from perf_trace_buf_alloc() without being initialized. Acquiring
> event_mutex in perf_kprobe_init() before calling perf_trace_event_init() should
> fix this race.
>
> Signed-off-by: Prateek Sood <prsood@codeaurora.org>
> ---
> kernel/trace/trace_event_perf.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
> index 4629a61..48ee92c 100644
> --- a/kernel/trace/trace_event_perf.c
> +++ b/kernel/trace/trace_event_perf.c
> @@ -272,9 +272,11 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
> goto out;
> }
>
> + mutex_lock(&event_mutex);
> ret = perf_trace_event_init(tp_event, p_event);
> if (ret)
> destroy_local_trace_kprobe(tp_event);
> + mutex_unlock(&event_mutex);
> out:
> kfree(func);
> return ret;
> @@ -282,8 +284,10 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
>
> void perf_kprobe_destroy(struct perf_event *p_event)
> {
> + mutex_lock(&event_mutex);
> perf_trace_event_close(p_event);
> perf_trace_event_unreg(p_event);
> + mutex_unlock(&event_mutex);
>
> destroy_local_trace_kprobe(p_event->tp_event);
> }
>
Folks,
Could you please help with this query/patch?
Thanks
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] trace: fix race in perf_trace_buf initialization
2019-10-17 6:53 ` Prateek Sood
@ 2019-10-17 12:34 ` Steven Rostedt
0 siblings, 0 replies; 7+ messages in thread
From: Steven Rostedt @ 2019-10-17 12:34 UTC (permalink / raw)
To: Prateek Sood; +Cc: mingo, linux-kernel, kaushalk, Peter Zijlstra
On Thu, 17 Oct 2019 12:23:49 +0530
Prateek Sood <prsood@codeaurora.org> wrote:
> On 10/15/19 11:47 AM, Prateek Sood wrote:
> > [ 943.034988] Unable to handle kernel paging request at virtual address 0000003106f2003c
> > [ 943.043653] Mem abort info:
> > [ 943.046679] ESR = 0x96000045
> > [ 943.050428] Exception class = DABT (current EL), IL = 32 bits
> > [ 943.056643] SET = 0, FnV = 0
> > [ 943.060168] EA = 0, S1PTW = 0
> > [ 943.063449] Data abort info:
> > [ 943.066474] ISV = 0, ISS = 0x00000045
> > [ 943.070856] CM = 0, WnR = 1
> > [ 943.074016] user pgtable: 4k pages, 39-bit VAs, pgdp = ffffffc034b9b000
> > [ 943.081446] [0000003106f2003c] pgd=0000000000000000, pud=0000000000000000
> > [ 943.088862] Internal error: Oops: 96000045 [#1] PREEMPT SMP
> > [ 943.141700] Process syz-executor (pid: 18393, stack limit = 0xffffffc093190000)
> > [ 943.164146] pstate: 80400005 (Nzcv daif +PAN -UAO)
> > [ 943.169119] pc : __memset+0x20/0x1ac
> > [ 943.172831] lr : memset+0x3c/0x50
> > [ 943.176269] sp : ffffffc09319fc50
> >
> > [ 943.557593] __memset+0x20/0x1ac
> > [ 943.560953] perf_trace_buf_alloc+0x140/0x1a0
> > [ 943.565472] perf_trace_sys_enter+0x158/0x310
> > [ 943.569985] syscall_trace_enter+0x348/0x7c0
> > [ 943.574413] el0_svc_common+0x11c/0x368
> > [ 943.578394] el0_svc_handler+0x12c/0x198
> > [ 943.582459] el0_svc+0x8/0xc
> >
> > In Ramdumps:
> > total_ref_count = 3
> > perf_trace_buf = (
> > 0x0 -> NULL,
> > 0x0 -> NULL,
> > 0x0 -> NULL,
> > 0x0 -> NULL)
> >
> > event_call in perf_trace_sys_enter()
> > event_call = 0xFFFFFF900CB511D8 -> (
> > list = (next = 0xFFFFFF900CB4E2E0, prev = 0xFFFFFF900CB512B0),
> > class = 0xFFFFFF900CDC8308,
> > name = 0xFFFFFF900CDDA1D8,
> > tp = 0xFFFFFF900CDDA1D8,
> > event = (
> > node = (next = 0x0, pprev = 0xFFFFFF900CB80210),
> > list = (next = 0xFFFFFF900CB512E0, prev = 0xFFFFFF900CB4E310),
> > type = 21,
> > funcs = 0xFFFFFF900CB51130),
> > print_fmt = 0xFFFFFF900CB51150,
> > filter = 0x0,
> > mod = 0x0,
> > data = 0x0,
> > flags = 18,
> > perf_refcount = 1,
> > perf_events = 0xFFFFFF8DB8E54158,
> > prog_array = 0x0,
> > perf_perm = 0x0)
> >
> > perf_events added on CPU0
> > (struct hlist_head *)(0xFFFFFF8DB8E54158+__per_cpu_offset[0]) -> (
> > first = 0xFFFFFFC0980FD0E0 -> (
> > next = 0x0,
> > pprev = 0xFFFFFFBEBFD74158))
> >
> > Could you please confirm:
> > 1) the race mentioned below exists or not.
> > 2) if following patch fixes it.
> >
> >
> >> 8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8
> >
> > A race condition exists while initialiazing perf_trace_buf from
> > perf_trace_init() and perf_kprobe_init().
> >
> > CPU0 CPU1
> > perf_trace_init()
> > mutex_lock(&event_mutex)
> > perf_trace_event_init()
> > perf_trace_event_reg()
> > total_ref_count == 0
> > buf = alloc_percpu()
> > perf_trace_buf[i] = buf
> > tp_event->class->reg() //fails perf_kprobe_init()
> > goto fail perf_trace_event_init()
> > perf_trace_event_reg()
> > fail:
> > total_ref_count == 0
> >
> > total_ref_count == 0
> > buf = alloc_percpu()
> > perf_trace_buf[i] = buf
> > tp_event->class->reg()
> > total_ref_count++
> >
> > free_percpu(perf_trace_buf[i])
> > perf_trace_buf[i] = NULL
> >
> > Any subsequent call to perf_trace_event_reg() will observe total_ref_count > 0,
> > causing the perf_trace_buf to be NULL always. This can result in perf_trace_buf
> > getting accessed from perf_trace_buf_alloc() without being initialized. Acquiring
> > event_mutex in perf_kprobe_init() before calling perf_trace_event_init() should
> > fix this race.
> >
> > Signed-off-by: Prateek Sood <prsood@codeaurora.org>
> > ---
> > kernel/trace/trace_event_perf.c | 4 ++++
> > 1 file changed, 4 insertions(+)
> >
> > diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
> > index 4629a61..48ee92c 100644
> > --- a/kernel/trace/trace_event_perf.c
> > +++ b/kernel/trace/trace_event_perf.c
> > @@ -272,9 +272,11 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
> > goto out;
> > }
> >
> > + mutex_lock(&event_mutex);
> > ret = perf_trace_event_init(tp_event, p_event);
> > if (ret)
> > destroy_local_trace_kprobe(tp_event);
> > + mutex_unlock(&event_mutex);
> > out:
> > kfree(func);
> > return ret;
> > @@ -282,8 +284,10 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
> >
> > void perf_kprobe_destroy(struct perf_event *p_event)
> > {
> > + mutex_lock(&event_mutex);
> > perf_trace_event_close(p_event);
> > perf_trace_event_unreg(p_event);
> > + mutex_unlock(&event_mutex);
> >
> > destroy_local_trace_kprobe(p_event->tp_event);
> > }
> >
>
> Folks,
>
> Could you please help with this query/patch?
Added Peter Zijlstra who maintains the kernel side of perf.
-- Steve
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] trace: fix race in perf_trace_buf initialization
2019-10-15 6:17 [PATCH] trace: fix race in perf_trace_buf initialization Prateek Sood
2019-10-17 6:53 ` Prateek Sood
@ 2019-10-18 21:12 ` Steven Rostedt
2019-10-21 4:42 ` Prateek Sood
1 sibling, 1 reply; 7+ messages in thread
From: Steven Rostedt @ 2019-10-18 21:12 UTC (permalink / raw)
To: Prateek Sood; +Cc: mingo, linux-kernel, kaushalk, Peter Zijlstra, Song Liu
Peter told me it was fine, but I'm also adding Song who is the author
of the code.
Also needs the tags (which I'll add)
Cc: stable@vger.kernel.org
Fixes: e12f03d7031a9 ("perf/core: Implement the 'perf_kprobe' PMU")
-- Steve
On Tue, 15 Oct 2019 11:47:25 +0530
Prateek Sood <prsood@codeaurora.org> wrote:
> [ 943.034988] Unable to handle kernel paging request at virtual address 0000003106f2003c
> [ 943.043653] Mem abort info:
> [ 943.046679] ESR = 0x96000045
> [ 943.050428] Exception class = DABT (current EL), IL = 32 bits
> [ 943.056643] SET = 0, FnV = 0
> [ 943.060168] EA = 0, S1PTW = 0
> [ 943.063449] Data abort info:
> [ 943.066474] ISV = 0, ISS = 0x00000045
> [ 943.070856] CM = 0, WnR = 1
> [ 943.074016] user pgtable: 4k pages, 39-bit VAs, pgdp = ffffffc034b9b000
> [ 943.081446] [0000003106f2003c] pgd=0000000000000000, pud=0000000000000000
> [ 943.088862] Internal error: Oops: 96000045 [#1] PREEMPT SMP
> [ 943.141700] Process syz-executor (pid: 18393, stack limit = 0xffffffc093190000)
> [ 943.164146] pstate: 80400005 (Nzcv daif +PAN -UAO)
> [ 943.169119] pc : __memset+0x20/0x1ac
> [ 943.172831] lr : memset+0x3c/0x50
> [ 943.176269] sp : ffffffc09319fc50
>
> [ 943.557593] __memset+0x20/0x1ac
> [ 943.560953] perf_trace_buf_alloc+0x140/0x1a0
> [ 943.565472] perf_trace_sys_enter+0x158/0x310
> [ 943.569985] syscall_trace_enter+0x348/0x7c0
> [ 943.574413] el0_svc_common+0x11c/0x368
> [ 943.578394] el0_svc_handler+0x12c/0x198
> [ 943.582459] el0_svc+0x8/0xc
>
> In Ramdumps:
> total_ref_count = 3
> perf_trace_buf = (
> 0x0 -> NULL,
> 0x0 -> NULL,
> 0x0 -> NULL,
> 0x0 -> NULL)
>
> event_call in perf_trace_sys_enter()
> event_call = 0xFFFFFF900CB511D8 -> (
> list = (next = 0xFFFFFF900CB4E2E0, prev = 0xFFFFFF900CB512B0),
> class = 0xFFFFFF900CDC8308,
> name = 0xFFFFFF900CDDA1D8,
> tp = 0xFFFFFF900CDDA1D8,
> event = (
> node = (next = 0x0, pprev = 0xFFFFFF900CB80210),
> list = (next = 0xFFFFFF900CB512E0, prev = 0xFFFFFF900CB4E310),
> type = 21,
> funcs = 0xFFFFFF900CB51130),
> print_fmt = 0xFFFFFF900CB51150,
> filter = 0x0,
> mod = 0x0,
> data = 0x0,
> flags = 18,
> perf_refcount = 1,
> perf_events = 0xFFFFFF8DB8E54158,
> prog_array = 0x0,
> perf_perm = 0x0)
>
> perf_events added on CPU0
> (struct hlist_head *)(0xFFFFFF8DB8E54158+__per_cpu_offset[0]) -> (
> first = 0xFFFFFFC0980FD0E0 -> (
> next = 0x0,
> pprev = 0xFFFFFFBEBFD74158))
>
> Could you please confirm:
> 1) the race mentioned below exists or not.
> 2) if following patch fixes it.
>
>
> >8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8
>
> A race condition exists while initialiazing perf_trace_buf from
> perf_trace_init() and perf_kprobe_init().
>
> CPU0 CPU1
> perf_trace_init()
> mutex_lock(&event_mutex)
> perf_trace_event_init()
> perf_trace_event_reg()
> total_ref_count == 0
> buf = alloc_percpu()
> perf_trace_buf[i] = buf
> tp_event->class->reg() //fails perf_kprobe_init()
> goto fail perf_trace_event_init()
> perf_trace_event_reg()
> fail:
> total_ref_count == 0
>
> total_ref_count == 0
> buf = alloc_percpu()
> perf_trace_buf[i] = buf
> tp_event->class->reg()
> total_ref_count++
>
> free_percpu(perf_trace_buf[i])
> perf_trace_buf[i] = NULL
>
> Any subsequent call to perf_trace_event_reg() will observe total_ref_count > 0,
> causing the perf_trace_buf to be NULL always. This can result in perf_trace_buf
> getting accessed from perf_trace_buf_alloc() without being initialized. Acquiring
> event_mutex in perf_kprobe_init() before calling perf_trace_event_init() should
> fix this race.
>
> Signed-off-by: Prateek Sood <prsood@codeaurora.org>
> ---
> kernel/trace/trace_event_perf.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
> index 4629a61..48ee92c 100644
> --- a/kernel/trace/trace_event_perf.c
> +++ b/kernel/trace/trace_event_perf.c
> @@ -272,9 +272,11 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
> goto out;
> }
>
> + mutex_lock(&event_mutex);
> ret = perf_trace_event_init(tp_event, p_event);
> if (ret)
> destroy_local_trace_kprobe(tp_event);
> + mutex_unlock(&event_mutex);
> out:
> kfree(func);
> return ret;
> @@ -282,8 +284,10 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
>
> void perf_kprobe_destroy(struct perf_event *p_event)
> {
> + mutex_lock(&event_mutex);
> perf_trace_event_close(p_event);
> perf_trace_event_unreg(p_event);
> + mutex_unlock(&event_mutex);
>
> destroy_local_trace_kprobe(p_event->tp_event);
> }
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] trace: fix race in perf_trace_buf initialization
2019-10-18 21:12 ` Steven Rostedt
@ 2019-10-21 4:42 ` Prateek Sood
2019-10-21 15:44 ` Steven Rostedt
0 siblings, 1 reply; 7+ messages in thread
From: Prateek Sood @ 2019-10-21 4:42 UTC (permalink / raw)
To: Steven Rostedt; +Cc: mingo, linux-kernel, kaushalk, Peter Zijlstra, Song Liu
On 10/19/19 2:42 AM, Steven Rostedt wrote:
>
> Peter told me it was fine, but I'm also adding Song who is the author
> of the code.
>
> Also needs the tags (which I'll add)
>
> Cc: stable@vger.kernel.org
> Fixes: e12f03d7031a9 ("perf/core: Implement the 'perf_kprobe' PMU")
>
> -- Steve
>
>
> On Tue, 15 Oct 2019 11:47:25 +0530
> Prateek Sood <prsood@codeaurora.org> wrote:
>
>> [ 943.034988] Unable to handle kernel paging request at virtual address 0000003106f2003c
>> [ 943.043653] Mem abort info:
>> [ 943.046679] ESR = 0x96000045
>> [ 943.050428] Exception class = DABT (current EL), IL = 32 bits
>> [ 943.056643] SET = 0, FnV = 0
>> [ 943.060168] EA = 0, S1PTW = 0
>> [ 943.063449] Data abort info:
>> [ 943.066474] ISV = 0, ISS = 0x00000045
>> [ 943.070856] CM = 0, WnR = 1
>> [ 943.074016] user pgtable: 4k pages, 39-bit VAs, pgdp = ffffffc034b9b000
>> [ 943.081446] [0000003106f2003c] pgd=0000000000000000, pud=0000000000000000
>> [ 943.088862] Internal error: Oops: 96000045 [#1] PREEMPT SMP
>> [ 943.141700] Process syz-executor (pid: 18393, stack limit = 0xffffffc093190000)
>> [ 943.164146] pstate: 80400005 (Nzcv daif +PAN -UAO)
>> [ 943.169119] pc : __memset+0x20/0x1ac
>> [ 943.172831] lr : memset+0x3c/0x50
>> [ 943.176269] sp : ffffffc09319fc50
>>
>> [ 943.557593] __memset+0x20/0x1ac
>> [ 943.560953] perf_trace_buf_alloc+0x140/0x1a0
>> [ 943.565472] perf_trace_sys_enter+0x158/0x310
>> [ 943.569985] syscall_trace_enter+0x348/0x7c0
>> [ 943.574413] el0_svc_common+0x11c/0x368
>> [ 943.578394] el0_svc_handler+0x12c/0x198
>> [ 943.582459] el0_svc+0x8/0xc
>>
>> In Ramdumps:
>> total_ref_count = 3
>> perf_trace_buf = (
>> 0x0 -> NULL,
>> 0x0 -> NULL,
>> 0x0 -> NULL,
>> 0x0 -> NULL)
>>
>> event_call in perf_trace_sys_enter()
>> event_call = 0xFFFFFF900CB511D8 -> (
>> list = (next = 0xFFFFFF900CB4E2E0, prev = 0xFFFFFF900CB512B0),
>> class = 0xFFFFFF900CDC8308,
>> name = 0xFFFFFF900CDDA1D8,
>> tp = 0xFFFFFF900CDDA1D8,
>> event = (
>> node = (next = 0x0, pprev = 0xFFFFFF900CB80210),
>> list = (next = 0xFFFFFF900CB512E0, prev = 0xFFFFFF900CB4E310),
>> type = 21,
>> funcs = 0xFFFFFF900CB51130),
>> print_fmt = 0xFFFFFF900CB51150,
>> filter = 0x0,
>> mod = 0x0,
>> data = 0x0,
>> flags = 18,
>> perf_refcount = 1,
>> perf_events = 0xFFFFFF8DB8E54158,
>> prog_array = 0x0,
>> perf_perm = 0x0)
>>
>> perf_events added on CPU0
>> (struct hlist_head *)(0xFFFFFF8DB8E54158+__per_cpu_offset[0]) -> (
>> first = 0xFFFFFFC0980FD0E0 -> (
>> next = 0x0,
>> pprev = 0xFFFFFFBEBFD74158))
>>
>> Could you please confirm:
>> 1) the race mentioned below exists or not.
>> 2) if following patch fixes it.
>>
>>
>>> 8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8
>>
>> A race condition exists while initialiazing perf_trace_buf from
>> perf_trace_init() and perf_kprobe_init().
>>
>> CPU0 CPU1
>> perf_trace_init()
>> mutex_lock(&event_mutex)
>> perf_trace_event_init()
>> perf_trace_event_reg()
>> total_ref_count == 0
>> buf = alloc_percpu()
>> perf_trace_buf[i] = buf
>> tp_event->class->reg() //fails perf_kprobe_init()
>> goto fail perf_trace_event_init()
>> perf_trace_event_reg()
>> fail:
>> total_ref_count == 0
>>
>> total_ref_count == 0
>> buf = alloc_percpu()
>> perf_trace_buf[i] = buf
>> tp_event->class->reg()
>> total_ref_count++
>>
>> free_percpu(perf_trace_buf[i])
>> perf_trace_buf[i] = NULL
>>
>> Any subsequent call to perf_trace_event_reg() will observe total_ref_count > 0,
>> causing the perf_trace_buf to be NULL always. This can result in perf_trace_buf
>> getting accessed from perf_trace_buf_alloc() without being initialized. Acquiring
>> event_mutex in perf_kprobe_init() before calling perf_trace_event_init() should
>> fix this race.
>>
>> Signed-off-by: Prateek Sood <prsood@codeaurora.org>
>> ---
>> kernel/trace/trace_event_perf.c | 4 ++++
>> 1 file changed, 4 insertions(+)
>>
>> diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
>> index 4629a61..48ee92c 100644
>> --- a/kernel/trace/trace_event_perf.c
>> +++ b/kernel/trace/trace_event_perf.c
>> @@ -272,9 +272,11 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
>> goto out;
>> }
>>
>> + mutex_lock(&event_mutex);
>> ret = perf_trace_event_init(tp_event, p_event);
>> if (ret)
>> destroy_local_trace_kprobe(tp_event);
>> + mutex_unlock(&event_mutex);
>> out:
>> kfree(func);
>> return ret;
>> @@ -282,8 +284,10 @@ int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
>>
>> void perf_kprobe_destroy(struct perf_event *p_event)
>> {
>> + mutex_lock(&event_mutex);
>> perf_trace_event_close(p_event);
>> perf_trace_event_unreg(p_event);
>> + mutex_unlock(&event_mutex);
>>
>> destroy_local_trace_kprobe(p_event->tp_event);
>> }
>
Hi Song,
Could you please help in this query.
Thanks
Prateek
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] trace: fix race in perf_trace_buf initialization
2019-10-21 4:42 ` Prateek Sood
@ 2019-10-21 15:44 ` Steven Rostedt
2019-10-21 15:51 ` Song Liu
0 siblings, 1 reply; 7+ messages in thread
From: Steven Rostedt @ 2019-10-21 15:44 UTC (permalink / raw)
To: Prateek Sood; +Cc: mingo, linux-kernel, kaushalk, Peter Zijlstra, Song Liu
On Mon, 21 Oct 2019 10:12:43 +0530
Prateek Sood <prsood@codeaurora.org> wrote:
> Hi Song,
>
> Could you please help in this query.
I have it ready to go to Linus. I'll wait a few hours, and if I don't
hear anything I'll send it out.
-- Steve
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] trace: fix race in perf_trace_buf initialization
2019-10-21 15:44 ` Steven Rostedt
@ 2019-10-21 15:51 ` Song Liu
0 siblings, 0 replies; 7+ messages in thread
From: Song Liu @ 2019-10-21 15:51 UTC (permalink / raw)
To: Steven Rostedt
Cc: Prateek Sood, mingo, linux-kernel, kaushalk, Peter Zijlstra
> On Oct 21, 2019, at 8:44 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 21 Oct 2019 10:12:43 +0530
> Prateek Sood <prsood@codeaurora.org> wrote:
>
>> Hi Song,
>>
>> Could you please help in this query.
>
> I have it ready to go to Linus. I'll wait a few hours, and if I don't
> hear anything I'll send it out.
>
Sorry for the late response.
The fix looks good to me.
Acked-by: Song Liu <songliubraving@fb.com>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-10-21 15:52 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-15 6:17 [PATCH] trace: fix race in perf_trace_buf initialization Prateek Sood
2019-10-17 6:53 ` Prateek Sood
2019-10-17 12:34 ` Steven Rostedt
2019-10-18 21:12 ` Steven Rostedt
2019-10-21 4:42 ` Prateek Sood
2019-10-21 15:44 ` Steven Rostedt
2019-10-21 15:51 ` Song Liu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).