From: Julien Grall <julien.grall@arm.com>
To: Stefano Stabellini <sstabellini@kernel.org>
Cc: Andre Przywara <andre.przywara@arm.com>,
Steve Capper <Steve.Capper@arm.com>,
wei.liu2@citrix.com, xen-devel@lists.xen.org
Subject: Re: [for-4.7] xen/arm: Force broadcast of TLB and instruction cache maintenance instructions
Date: Mon, 25 Apr 2016 14:54:22 +0100 [thread overview]
Message-ID: <571E218E.50900@arm.com> (raw)
In-Reply-To: <alpine.DEB.2.10.1604251138530.24872@sstabellini-ThinkPad-X260>
(CC Steve and Andre)
Hi Stefano,
On 25/04/16 11:45, Stefano Stabellini wrote:
> On Mon, 18 Apr 2016, Julien Grall wrote:
>> UP guest usually uses TLB instruction to flush only on the local CPU. The
>> TLB flush won't be broadcasted across all the CPUs within the same
>> innershareable domain.
>>
>> When the vCPU is migrated between different CPUs, it may be rescheduled
>> to a previous CPU where the TLB has not been flushed. The TLB may
>> contain stale entries which will result to translate incorrectly a VA to
>> IPA or even cause TLB conflicts.
>>
>> To avoid a such situation, always set HCR_EL2.FB which will force the
>> broadcast of TLB and instruction cache maintenance instructions.
>> Cheers,
>>
>> Signed-off-by: Julien Grall <julien.grall@arm.com>
>
> Well spotted!
>
> Julien, I was wondering whether we could avoid the HCR_FB by manually
> doing a flush in ctxt_switch_from or context_switch. I am suggesting
> this because I have the feeling that enabling HCR_FB would have a
> negative performance impact.
The performance impact will depend on how much the guest makes use of
local flush instructions.
When HCR.FB is set, the hardware will broadcast the flush (TLBs,
instruction cache or branch predictor) to all the CPUs in the same
innershareable domain. I.e any local flush instructions will be upgraded
to innershareable.
ARM64 Linux kernel is SMP-aware (no possibility to build only for UP),
most of the flush instructions are innershareable. The local flushes are
limited to boot (1 flush per CPU) and when the ASID of a task is
changing. Therefore the impact of setting HCR.FB for ARM64 Linux guest
would be very limited.
ARM32 Linux kernel can be built SMP-aware or only UP-aware. The former,
will make a very limited use of those instructions. The latter will
obviously use only local flush instructions. Therefore, there will be an
impact to set HCR.FB for UP-aware kernel guest.
I have looked quickly at FreeBSD (both ARM64 and ARM32). SMP-aware
kernel will mostly make use of innershareable flush instructions.
UP-aware kernel will only make use of local flush instructions.
However, nothing prevent an SMP-aware kernel to make more often use of
local flush instructions.
In the case that HCR.FB is not set, Xen would need to:
* Flush all the TLBs for the VMID associated to this domain
* Invalidate all the entries from branch predictors (on for AArch32)
* Invalidate all the entries from the instruction cache
Whilst you suggested to do it at every domain context switch, this is
only necessary when the vCPU migrates between 2 physical CPUs.
In any case, not setting HCR.FB will have a big impact on any SMP-aware
Linux/Freebsd kernel as any context switch (or migration) will nuke the
TLBs, the instruction cache and the branch predictor.
The impact of HCR.FB on UP-aware kernel would need to be benchmarked.
But to be honest, I expect most of the kernels, which run in a guest, to
be SMP-aware.
So setting HCR.FB seems to be the best solution. We can revisit it
later, if we notice negative performance impact.
Regards,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2016-04-25 13:54 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-04-18 9:29 [for-4.7] xen/arm: Force broadcast of TLB and instruction cache maintenance instructions Julien Grall
2016-04-25 10:45 ` Stefano Stabellini
2016-04-25 13:54 ` Julien Grall [this message]
2016-04-26 9:45 ` Stefano Stabellini
2016-04-27 10:26 ` Julien Grall
2016-04-25 18:22 ` Konrad Rzeszutek Wilk
2016-04-26 9:36 ` Julien Grall
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=571E218E.50900@arm.com \
--to=julien.grall@arm.com \
--cc=Steve.Capper@arm.com \
--cc=andre.przywara@arm.com \
--cc=sstabellini@kernel.org \
--cc=wei.liu2@citrix.com \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.