From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Jiri Olsa <jolsa@kernel.org>
Cc: lkml <linux-kernel@vger.kernel.org>,
Ingo Molnar <mingo@kernel.org>,
Namhyung Kim <namhyung@kernel.org>,
David Ahern <dsahern@gmail.com>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
Kan Liang <kan.liang@intel.com>, Andi Kleen <ak@linux.intel.com>,
Lukasz Odzioba <lukasz.odzioba@intel.com>,
Wang Nan <wangnan0@huawei.com>
Subject: Re: [PATCH 4/4] perf tools: Fix struct comm_str removal crash
Date: Thu, 19 Jul 2018 12:58:26 -0300 [thread overview]
Message-ID: <20180719155826.GF4070@kernel.org> (raw)
In-Reply-To: <20180719143345.12963-5-jolsa@kernel.org>
Em Thu, Jul 19, 2018 at 04:33:45PM +0200, Jiri Olsa escreveu:
> We occasionaly hit following assert failure in perf top,
> when processing the /proc info in multiple threads.
Namhyung, are you ok with this one?
- Arnaldo
> perf: ...include/linux/refcount.h:109: refcount_inc:
> Assertion `!(!refcount_inc_not_zero(r))' failed.
>
> The gdb backtrace looks like this:
>
> [Switching to Thread 0x7ffff11ba700 (LWP 13749)]
> 0x00007ffff50839fb in raise () from /lib64/libc.so.6
> (gdb)
> #0 0x00007ffff50839fb in raise () from /lib64/libc.so.6
> #1 0x00007ffff5085800 in abort () from /lib64/libc.so.6
> #2 0x00007ffff507c0da in __assert_fail_base () from /lib64/libc.so.6
> #3 0x00007ffff507c152 in __assert_fail () from /lib64/libc.so.6
> #4 0x0000000000535373 in refcount_inc (r=0x7fffdc009be0)
> at ...include/linux/refcount.h:109
> #5 0x00000000005354f1 in comm_str__get (cs=0x7fffdc009bc0)
> at util/comm.c:24
> #6 0x00000000005356bd in __comm_str__findnew (str=0x7fffd000b260 ":2",
> root=0xbed5c0 <comm_str_root>) at util/comm.c:72
> #7 0x000000000053579e in comm_str__findnew (str=0x7fffd000b260 ":2",
> root=0xbed5c0 <comm_str_root>) at util/comm.c:95
> #8 0x000000000053582e in comm__new (str=0x7fffd000b260 ":2",
> timestamp=0, exec=false) at util/comm.c:111
> #9 0x00000000005363bc in thread__new (pid=2, tid=2) at util/thread.c:57
> #10 0x0000000000523da0 in ____machine__findnew_thread (machine=0xbfde38,
> threads=0xbfdf28, pid=2, tid=2, create=true) at util/machine.c:457
> #11 0x0000000000523eb4 in __machine__findnew_thread (machine=0xbfde38,
> ...
>
> The failing assertion is this one:
>
> REFCOUNT_WARN(!refcount_inc_not_zero(r), ...
>
> The problem is that we keep global comm_str_root list, which
> is accessed by multiple threads during the perf top startup
> and following 2 paths can race:
>
> thread 1:
> ...
> thread__new
> comm__new
> comm_str__findnew
> down_write(&comm_str_lock);
> __comm_str__findnew
> comm_str__get
>
> thread 2:
> ...
> comm__override or comm__free
> comm_str__put
> refcount_dec_and_test
> down_write(&comm_str_lock);
> rb_erase(&cs->rb_node, &comm_str_root);
>
> Because thread 2 first decrements the refcnt and only after then it
> removes the struct comm_str from the list, the thread 1 can find this
> object on the list with refcnt equls to 0 and hit the assert.
>
> This patch fixes the thread 1 __comm_str__findnew path, by ignoring
> objects that already dropped the refcnt to 0. For the rest of the
> objects we take the refcnt before comparing its name and release
> it afterwards with comm_str__put, which can also release the object
> completely.
>
> Link: http://lkml.kernel.org/n/tip-vrizt6sw1lu1ybsrl9l0wwln@git.kernel.org
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
> tools/perf/util/comm.c | 15 +++++++++------
> 1 file changed, 9 insertions(+), 6 deletions(-)
>
> diff --git a/tools/perf/util/comm.c b/tools/perf/util/comm.c
> index 7798a2cc8a86..9c4a18991e41 100644
> --- a/tools/perf/util/comm.c
> +++ b/tools/perf/util/comm.c
> @@ -18,11 +18,9 @@ struct comm_str {
> static struct rb_root comm_str_root;
> static struct rw_semaphore comm_str_lock = {.lock = PTHREAD_RWLOCK_INITIALIZER,};
>
> -static struct comm_str *comm_str__get(struct comm_str *cs)
> +static bool comm_str__get(struct comm_str *cs)
> {
> - if (cs)
> - refcount_inc(&cs->refcnt);
> - return cs;
> + return cs ? refcount_inc_not_zero(&cs->refcnt) : false;
> }
>
> static void comm_str__put(struct comm_str *cs)
> @@ -67,9 +65,14 @@ struct comm_str *__comm_str__findnew(const char *str, struct rb_root *root)
> parent = *p;
> iter = rb_entry(parent, struct comm_str, rb_node);
>
> + /*
> + * If we race with comm_str__put, iter->refcnt is 0
> + * and it will be removed within comm_str__put call
> + * shortly, ignore it in this search.
> + */
> cmp = strcmp(str, iter->str);
> - if (!cmp)
> - return comm_str__get(iter);
> + if (!cmp && comm_str__get(iter))
> + return iter;
>
> if (cmp < 0)
> p = &(*p)->rb_left;
> --
> 2.17.1
next prev parent reply other threads:[~2018-07-19 15:58 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-07-19 14:33 [PATCHv2 0/4] perf tools: Fix top crashes Jiri Olsa
2018-07-19 14:33 ` [PATCH 1/4] perf tools: Add threads__get_last_match function Jiri Olsa
2018-07-25 20:50 ` [tip:perf/core] perf machine: " tip-bot for Jiri Olsa
2018-07-19 14:33 ` [PATCH 2/4] perf tools: Add threads__set_last_match function Jiri Olsa
2018-07-25 20:51 ` [tip:perf/core] perf machine: " tip-bot for Jiri Olsa
2018-07-19 14:33 ` [PATCH 3/4] perf tools: Use last_match threads cache only in single thread mode Jiri Olsa
2018-07-25 20:51 ` [tip:perf/core] perf machine: " tip-bot for Jiri Olsa
2018-07-19 14:33 ` [PATCH 4/4] perf tools: Fix struct comm_str removal crash Jiri Olsa
2018-07-19 15:58 ` Arnaldo Carvalho de Melo [this message]
2018-07-19 18:28 ` Arnaldo Carvalho de Melo
2018-07-19 18:31 ` Arnaldo Carvalho de Melo
2018-07-20 1:20 ` Namhyung Kim
2018-07-20 10:17 ` [PATCHv3 " Jiri Olsa
2018-07-20 14:42 ` Arnaldo Carvalho de Melo
2018-07-25 20:52 ` [tip:perf/core] " tip-bot for Jiri Olsa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180719155826.GF4070@kernel.org \
--to=acme@kernel.org \
--cc=a.p.zijlstra@chello.nl \
--cc=ak@linux.intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=dsahern@gmail.com \
--cc=jolsa@kernel.org \
--cc=kan.liang@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lukasz.odzioba@intel.com \
--cc=mingo@kernel.org \
--cc=namhyung@kernel.org \
--cc=wangnan0@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).