From: "Emilio G. Cota" <cota@braap.org>
To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
Cc: qemu-devel@nongnu.org, "Alex Bennée" <alex.bennee@linaro.org>,
"Pranith Kumar" <bobby.prani@gmail.com>,
"Richard Henderson" <richard.henderson@linaro.org>
Subject: Re: [Qemu-devel] [RFC 6/6] cputlb: dynamically resize TLBs based on use rate
Date: Sun, 7 Oct 2018 21:48:34 -0400 [thread overview]
Message-ID: <20181008014834.GA21299@flamenco> (raw)
In-Reply-To: <197b7649-0bc7-4a9a-021b-a308a1755cf1@amsat.org>
On Sun, Oct 07, 2018 at 19:37:50 +0200, Philippe Mathieu-Daudé wrote:
> On 10/6/18 11:45 PM, Emilio G. Cota wrote:
> > 2. System boot + shutdown, ubuntu 18.04 x86_64:
>
> You can also run the VM tests to build QEMU:
>
> $ make vm-test
Thanks, will give that a look.
> > + if (rate == 100) {
> > + new_size = MIN(desc->size << 2, 1 << TCG_TARGET_TLB_MAX_INDEX_BITS);
> > + } else if (rate > 70) {
> > + new_size = MIN(desc->size << 1, 1 << TCG_TARGET_TLB_MAX_INDEX_BITS);
> > + } else if (rate < 30) {
>
> I wonder if those thresholds might be per TCG_TARGET.
Do you mean to tune the growth rate based on each TCG target?
(max and min are already determined by the TCG target).
The optimal growth rate is mostly dependent on the guest
workload, so I wouldn't expect the TCG target to matter
much.
That said, we could spend quite some time tweaking the
TLB sizing algorithm. But with this RFC I wanted to see
(a) whether this approach is a good idea at all, and (b) show
what 'easy' speedups might look like (because converting
all TCG targets is a pain, so it better be justified).
> Btw the paper used 40% here, did you tried it too?
Yes, I tried several alternatives including what the
paper describes, i.e. (skipping the min/max checks
for simplicity):
if (rate > 70) {
new_size = 2 * old_size;
} else if (rate < 40) {
new_size = old_size / 2;
}
But that didn't give great speedups (see "resizing-paper"
set):
https://imgur.com/a/w3AqHP7
A few points stand out to me:
- We get very different speedups even if we implement
the algorithm they describe (not sure that's exactly
what they implemented though). But there are many
variables that could explain that, e.g. different guest
images (and therefore different TLB flush rates) and
different QEMU baselines (ours is faster than the paper's,
so getting speedups is harder).
- 70/40% use rate for growing/shrinking the TLB does not
seem a great choice, if one wants to avoid a pathological
case that can induce constant resizing. Imagine we got
exactly 70% use rate, and all TLB misses were compulsory
(i.e. a direct-mapped TLB would have not prevented a
single miss). We'd then double the TLB size:
size_new = 2*size_old
But then the use rate will halve:
use_new = 0.7/2 = 0.35
So we'd then end up in a grow-shrink loop!
Picking a "shrink threshold" below 0.70/2=0.35 avoids this.
- Aggressively increasing the TLB size when usage is high
makes sense. However, reducing the size at the same
rate does not make much sense. Imagine the following
scenario with two processes being scheduled: one process
uses a lot of memory, and the other one uses little, but
both are CPU-intensive and therefore being assigned similar
time slices by the scheduler. Ideally you'd resize the TLB
to meet each process' memory demands. However, at flush
time we don't even know what process is running or about
to run, so we have to size the TLB exclusively based on
recent use rates. In this scenario you're probably close
to optimal if you size the TLB to meet the demands of the
most memory-hungry process. You'll lose some extra time
flushing the (now larger) TLB, but your net gain is likely
to be positive given the TLB fills you won't have to do
when the memory-heavy process is scheduled in.
So to me it's quite likely that in the paper they
could have gotten even better results by reducing the
shrink rate, like we did.
Thanks,
Emilio
next prev parent reply other threads:[~2018-10-08 1:48 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-06 21:45 [Qemu-devel] [RFC 0/6] Dynamic TLB sizing Emilio G. Cota
2018-10-06 21:45 ` [Qemu-devel] [RFC 1/6] (XXX) cputlb: separate MMU allocation + run-time sizing Emilio G. Cota
2018-10-08 1:47 ` Richard Henderson
2018-10-06 21:45 ` [Qemu-devel] [RFC 2/6] cputlb: do not evict invalid entries to the vtlb Emilio G. Cota
2018-10-08 2:09 ` Richard Henderson
2018-10-08 14:42 ` Emilio G. Cota
2018-10-08 19:46 ` Richard Henderson
2018-10-08 20:23 ` Emilio G. Cota
2018-10-06 21:45 ` [Qemu-devel] [RFC 3/6] cputlb: track TLB use rates Emilio G. Cota
2018-10-08 2:54 ` Richard Henderson
2018-10-06 21:45 ` [Qemu-devel] [RFC 4/6] tcg: define TCG_TARGET_TLB_MAX_INDEX_BITS Emilio G. Cota
2018-10-08 2:56 ` Richard Henderson
2018-10-06 21:45 ` [Qemu-devel] [RFC 5/6] cpu-defs: define MIN_CPU_TLB_SIZE Emilio G. Cota
2018-10-08 3:01 ` Richard Henderson
2018-10-06 21:45 ` [Qemu-devel] [RFC 6/6] cputlb: dynamically resize TLBs based on use rate Emilio G. Cota
2018-10-07 17:37 ` Philippe Mathieu-Daudé
2018-10-08 1:48 ` Emilio G. Cota [this message]
2018-10-08 13:46 ` Emilio G. Cota
2018-10-08 3:21 ` Richard Henderson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181008014834.GA21299@flamenco \
--to=cota@braap.org \
--cc=alex.bennee@linaro.org \
--cc=bobby.prani@gmail.com \
--cc=f4bug@amsat.org \
--cc=qemu-devel@nongnu.org \
--cc=richard.henderson@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.