From: Jiri Olsa <jolsa@redhat.com>
To: David Miller <davem@davemloft.net>
Cc: acme@kernel.org, linux-kernel@vger.kernel.org,
namhyung@kernel.org, jolsa@kernel.org
Subject: Re: [PATCH RFC] hist lookups
Date: Sun, 4 Nov 2018 21:18:21 +0100 [thread overview]
Message-ID: <20181104201821.GA22049@krava> (raw)
In-Reply-To: <20181102.233003.1814045087128749000.davem@davemloft.net>
On Fri, Nov 02, 2018 at 11:30:03PM -0700, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Wed, 31 Oct 2018 09:08:16 -0700 (PDT)
>
> > From: Jiri Olsa <jolsa@redhat.com>
> > Date: Wed, 31 Oct 2018 16:39:07 +0100
> >
> >> it'd be great to make hist processing faster, but is your main target here
> >> to get the load out of the reader thread, so we dont lose events during the
> >> hist processing?
> >>
> >> we could queue events directly from reader thread into another thread and
> >> keep it (the reader thread) free of processing, focusing only on event
> >> reading/passing
> >
> > Indeed, we could create threads that take samples from the thread processing
> > the ring buffers, and insert them into the histogram.
>
> So I played around with some ideas like this and ran into some dead ends.
>
> I ran each mmap ring's processing in a separate thread.
>
> This doesn't help at all, the problem is that all the threads serialize
> at the pthread lock for the histogram part of the work.
>
> And the histogram part dominates the cost of processing each sample.
yep, it suck.. I was thinking of keeping separate hist objects for
each thread and merge them at the end
>
> Nevertheless I started work on formally threading all of the code that
> the mmap threads operate on, such as symbol processing etc. and while
> doing so I came to the conclusion that pushing the histogram processing
> only to a separate thread poses it's own set of big challenges.
>
> To make this work we would have to make a piece of transient on-stack
> state (the processed event) into allocated persistent state.
>
> These persistent event structures get queued up to the histogram
> thread(s).
>
> Therefore, if the histogram thread(s) can't keep up (and as per my
> experiment above, it is easy to enter this state because the histogram
> code itself is going to run linearly with the histgram lock held),
> this persistent event memory will just get larger and larger.
>
> We would have to find some way to parallelize the histgram code to
> make any kind of threading worthwhile.
do you have some code I could check on?
I'm going to make that separate thread to get the processing out
of the reading thread.. I think we need that in any case, so the
ring buffer is kept free as fast as possible
thanks,
jirka
next prev parent reply other threads:[~2018-11-04 20:18 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-31 5:03 [PATCH RFC] hist lookups David Miller
2018-10-31 12:43 ` Arnaldo Carvalho de Melo
2018-10-31 15:39 ` Jiri Olsa
2018-10-31 16:08 ` David Miller
2018-11-03 6:30 ` David Miller
2018-11-04 20:18 ` Jiri Olsa [this message]
2018-11-05 0:50 ` David Miller
2018-11-05 20:34 ` Jiri Olsa
2018-11-05 22:52 ` David Miller
2018-11-06 3:45 ` David Miller
2018-11-06 4:03 ` David Miller
2018-11-06 4:53 ` David Miller
2018-11-06 11:54 ` Jiri Olsa
2018-11-19 5:26 ` Namhyung Kim
2018-11-19 9:12 ` Jiri Olsa
2018-11-20 1:13 ` Namhyung Kim
2018-11-06 20:42 ` Jiri Olsa
2018-11-07 6:13 ` David Miller
2018-11-07 8:32 ` Jiri Olsa
2018-11-07 19:43 ` Jiri Olsa
2018-11-07 20:01 ` David Miller
2018-11-07 20:28 ` Arnaldo Carvalho de Melo
2018-11-08 6:04 ` David Miller
2018-11-08 7:13 ` Jiri Olsa
2018-11-09 1:07 ` David Miller
2018-11-11 19:41 ` Jiri Olsa
2018-11-11 19:41 ` Jiri Olsa
2018-11-11 22:32 ` David Miller
2018-11-11 22:43 ` Jiri Olsa
2018-11-11 22:58 ` David Miller
2018-11-11 23:08 ` David Miller
2018-11-11 23:26 ` Jiri Olsa
2018-11-11 23:32 ` David Miller
2018-11-13 10:40 ` Jiri Olsa
2018-11-19 4:52 ` David Miller
2018-11-19 6:28 ` Namhyung Kim
2018-11-19 6:33 ` David Miller
2018-11-19 7:16 ` Namhyung Kim
2018-11-19 9:14 ` Jiri Olsa
2018-11-06 11:51 ` Jiri Olsa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181104201821.GA22049@krava \
--to=jolsa@redhat.com \
--cc=acme@kernel.org \
--cc=davem@davemloft.net \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=namhyung@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).