From: Andre Przywara <andre.przywara@arm.com>
To: Julien Grall <julien.grall@arm.com>,
Stefano Stabellini <sstabellini@kernel.org>
Cc: xen-devel@lists.xenproject.org,
Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>,
Vijay Kilari <vijay.kilari@gmail.com>,
Shanker Donthineni <shankerd@codeaurora.org>
Subject: Re: [PATCH v10 01/32] ARM: vGIC: avoid rank lock when reading priority
Date: Tue, 6 Jun 2017 18:20:26 +0100 [thread overview]
Message-ID: <c76e32bc-3b3b-e52a-e898-7ca88d2be796@arm.com> (raw)
In-Reply-To: <b6399b51-ab27-786e-4671-d09feb3c7727@arm.com>
Hi,
On 06/06/17 18:11, Julien Grall wrote:
>
>
> On 06/06/17 18:06, Andre Przywara wrote:
>> On 30/05/17 11:47, Julien Grall wrote:
>>> Hi Andre,
>>>
>>> On 26/05/17 18:35, Andre Przywara wrote:
>>>> When reading the priority value of a virtual interrupt, we were taking
>>>> the respective rank lock so far.
>>>> However for forwarded interrupts (Dom0 only so far) this may lead to a
>>>> deadlock with the following call chain:
>>>> - MMIO access to change the IRQ affinity, calling the ITARGETSR handler
>>>> - this handler takes the appropriate rank lock and calls
>>>> vgic_store_itargetsr()
>>>> - vgic_store_itargetsr() will eventually call vgic_migrate_irq()
>>>> - if this IRQ is already in-flight, it will remove it from the old
>>>> VCPU and inject it into the new one, by calling
>>>> vgic_vcpu_inject_irq()
>>>> - vgic_vcpu_inject_irq will call vgic_get_virq_priority()
>>>> - vgic_get_virq_priority() tries to take the rank lock - again!
>>>> It seems like this code path has never been exercised before.
>>>>
>>>> Fix this by avoiding taking the lock in vgic_get_virq_priority()
>>>> (like we
>>>> do in vgic_get_target_vcpu()).
>>>> Actually we are just reading one byte, and priority changes while
>>>> interrupts are handled are a benign race that can happen on real
>>>> hardware
>>>> too. So it looks safe to just use read_atomic() instead.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>> ---
>>>> xen/arch/arm/vgic.c | 8 +-------
>>>> 1 file changed, 1 insertion(+), 7 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>>>> index 83569b0..54b2aad 100644
>>>> --- a/xen/arch/arm/vgic.c
>>>> +++ b/xen/arch/arm/vgic.c
>>>> @@ -227,14 +227,8 @@ struct vcpu *vgic_get_target_vcpu(struct vcpu *v,
>>>> unsigned int virq)
>>>> static int vgic_get_virq_priority(struct vcpu *v, unsigned int virq)
>>>> {
>>>> struct vgic_irq_rank *rank = vgic_rank_irq(v, virq);
>>>> - unsigned long flags;
>>>> - int priority;
>>>> -
>>>> - vgic_lock_rank(v, rank, flags);
>>>> - priority = rank->priority[virq & INTERRUPT_RANK_MASK];
>>>> - vgic_unlock_rank(v, rank, flags);
>>>>
>>>> - return priority;
>>>> + return read_atomic(&rank->priority[virq & INTERRUPT_RANK_MASK]);
>>>
>>> The write in rank->priority will not be atomic (see vgic_reg_update
>>> implementation): the register is first masked, the the priority set.
>>>
>>> So you may end up to read 0 (which is the higher priority) by mistake.
>>>
>>> We should probably think to make vgic_reg_* helper atomic.
>>
>> The patch you sent yesterday may be a bit over the top for this.
>
> Patch which I only sent internally to get feedback. ;)
Ah, indeed, sorry for dragging you into the light ;-)
For the records: I am not saying that your patch was wrong, I am just a
bit reluctant to change such a central function (vgic_reg_update) when
we only care about the priority.
>>
>> What about this one (in xen/arch/arm/vgic-v[23].c):
>>
>> static int vgic_v2_distr_mmio_write(struct vcpu *v, ...
>> {
>> - uint32_t *ipriorityr;
>> + uint32_t *ipriorityr, priority;
>> ....
>> vgic_lock_rank(v, rank, flags);
>> ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8,
>> gicd_reg - GICD_IPRIORITYR,
>> DABT_WORD)];
>> - vgic_reg32_update(ipriorityr, r, info);
>> + priority = ACCESS_ONCE(*ipriorityr);
>> + vgic_reg32_update(&priority, r, info);
>> + ACCESS_ONCE(*ipriorityr) = priority;
>> vgic_unlock_rank(v, rank, flags);
>> ....
>
> What you do is very similar to my patch but open-code it. It might be
> possible to have other place which could take advantage of the a generic
> solution, so I am not convinced this would be the right way.
My idea was to confine this fix to what we care for right now.
I guess eventually it doesn't matter as we will probably get rid of the
rank and its lock anyway.
>>
>> Concurrent writes are protected by the rank lock, and reads are
>> guaranteed to be atomic due to ACCESS_ONCE and the architectural
>> guarantee of an aligned uint32_t access being atomic.
>> We can't do anything about the benign race when the priority gets
>> changed while we read it, but the above should make sure that we either
>> read the old or the new value and nothing in between.
>> The other read accesses (in vgic_v[23]_distr_mmio_read() and in vgic.c
>> here in this patch) get protected by an ACCESS_ONCE().
>>
>> Does that make sense?
>
> I was trying to address the suggestion of Stefano to drop the rank lock
> in some place and also make the solution generic.
>
> I failed to address properly the lock drop. And have some idea how to do
> it.
Looking forward to it ....
Cheers,
Andre.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
next prev parent reply other threads:[~2017-06-06 17:20 UTC|newest]
Thread overview: 96+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-26 17:35 [PATCH v10 00/32] arm64: Dom0 ITS emulation Andre Przywara
2017-05-26 17:35 ` [PATCH v10 01/32] ARM: vGIC: avoid rank lock when reading priority Andre Przywara
2017-05-30 10:47 ` Julien Grall
2017-05-30 21:39 ` Stefano Stabellini
2017-05-31 10:42 ` Julien Grall
2017-06-02 17:44 ` Julien Grall
2017-06-06 17:06 ` Andre Przywara
2017-06-06 17:11 ` Julien Grall
2017-06-06 17:20 ` Andre Przywara [this message]
2017-06-06 17:21 ` Julien Grall
2017-06-06 18:39 ` Stefano Stabellini
2017-05-26 17:35 ` [PATCH v10 02/32] ARM: GICv3: setup number of LPI bits for a GICv3 guest Andre Przywara
2017-05-30 10:54 ` Julien Grall
2017-06-06 10:19 ` Andre Przywara
2017-05-26 17:35 ` [PATCH v10 03/32] ARM: vGIC: move irq_to_pending() calls under the VGIC VCPU lock Andre Przywara
2017-05-30 11:08 ` Julien Grall
2017-05-30 21:46 ` Stefano Stabellini
2017-05-31 10:44 ` Julien Grall
2017-06-06 17:24 ` Andre Przywara
2017-06-06 18:46 ` Stefano Stabellini
2017-06-07 10:49 ` Andre Przywara
2017-05-26 17:35 ` [PATCH v10 04/32] ARM: vGIC: rework gic_remove_from_queues() Andre Przywara
2017-05-30 11:15 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 05/32] ARM: vGIC: introduce gic_remove_irq() Andre Przywara
2017-05-30 11:31 ` Julien Grall
2017-06-06 10:19 ` Andre Przywara
2017-05-26 17:35 ` [PATCH v10 06/32] ARM: GIC: Add checks for NULL pointer pending_irq's Andre Przywara
2017-05-30 11:38 ` Julien Grall
2017-06-06 10:19 ` Andre Przywara
2017-06-07 11:19 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 07/32] ARM: GICv3: introduce separate pending_irq structs for LPIs Andre Przywara
2017-05-26 17:35 ` [PATCH v10 08/32] ARM: GIC: export and extend vgic_init_pending_irq() Andre Przywara
2017-05-26 17:35 ` [PATCH v10 09/32] ARM: vGIC: cache virtual LPI priority in struct pending_irq Andre Przywara
2017-05-26 17:35 ` [PATCH v10 10/32] ARM: vGIC: add LPI VCPU ID to " Andre Przywara
2017-05-26 17:35 ` [PATCH v10 11/32] ARM: GICv3: forward pending LPIs to guests Andre Przywara
2017-05-30 11:56 ` Julien Grall
2017-05-30 22:07 ` Stefano Stabellini
2017-05-31 11:09 ` Julien Grall
2017-05-31 17:56 ` Stefano Stabellini
2017-05-31 18:39 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 12/32] ARM: GICv3: enable ITS and LPIs on the host Andre Przywara
2017-05-30 11:58 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 13/32] ARM: vGICv3: handle virtual LPI pending and property tables Andre Przywara
2017-05-26 17:35 ` [PATCH v10 14/32] ARM: introduce vgic_access_guest_memory() Andre Przywara
2017-05-26 17:35 ` [PATCH v10 15/32] ARM: vGICv3: re-use vgic_reg64_check_access Andre Przywara
2017-05-26 17:35 ` [PATCH v10 16/32] ARM: vGIC: advertise LPI support Andre Przywara
2017-05-30 12:59 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 17/32] ARM: vITS: add command handling stub and MMIO emulation Andre Przywara
2017-06-01 18:13 ` Julien Grall
2017-06-08 9:57 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 18/32] ARM: vITS: introduce translation table walks Andre Przywara
2017-06-02 16:25 ` Julien Grall
2017-06-08 9:35 ` Julien Grall
2017-06-08 9:45 ` Andre Przywara
2017-05-26 17:35 ` [PATCH v10 19/32] ARM: vITS: provide access to struct pending_irq Andre Przywara
2017-06-02 16:32 ` Julien Grall
2017-06-02 16:45 ` Julien Grall
2017-06-06 10:19 ` Andre Przywara
2017-06-06 11:13 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 20/32] ARM: vITS: handle INT command Andre Przywara
2017-06-02 16:37 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 21/32] ARM: vITS: handle MAPC command Andre Przywara
2017-05-26 17:35 ` [PATCH v10 22/32] ARM: vITS: handle CLEAR command Andre Przywara
2017-06-02 16:40 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 23/32] ARM: vITS: handle MAPD command Andre Przywara
2017-06-02 16:46 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 24/32] ARM: GICv3: handle unmapped LPIs Andre Przywara
2017-06-02 16:55 ` Julien Grall
2017-06-02 20:45 ` Stefano Stabellini
2017-06-08 9:45 ` Julien Grall
2017-06-08 13:51 ` Andre Przywara
2017-05-26 17:35 ` [PATCH v10 25/32] ARM: vITS: handle MAPTI/MAPI command Andre Przywara
2017-06-02 17:12 ` Julien Grall
2017-06-07 17:49 ` Andre Przywara
2017-06-12 16:33 ` Julien Grall
2017-06-09 11:17 ` Andre Przywara
2017-06-09 19:14 ` Stefano Stabellini
2017-06-12 16:10 ` Andre Przywara
2017-05-26 17:35 ` [PATCH v10 26/32] ARM: vITS: handle MOVI command Andre Przywara
2017-05-30 22:35 ` Stefano Stabellini
2017-05-31 11:23 ` Julien Grall
2017-05-31 17:53 ` Stefano Stabellini
2017-05-31 18:49 ` Julien Grall
2017-06-02 17:17 ` Julien Grall
2017-06-02 20:36 ` Stefano Stabellini
2017-05-26 17:35 ` [PATCH v10 27/32] ARM: vITS: handle DISCARD command Andre Przywara
2017-06-02 17:21 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 28/32] ARM: vITS: handle INV command Andre Przywara
2017-05-30 22:23 ` Stefano Stabellini
2017-05-26 17:35 ` [PATCH v10 29/32] ARM: vITS: handle INVALL command Andre Przywara
2017-06-02 17:27 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 30/32] ARM: vITS: increase mmio_count for each ITS Andre Przywara
2017-05-26 17:35 ` [PATCH v10 31/32] ARM: vITS: create and initialize virtual ITSes for Dom0 Andre Przywara
2017-06-02 17:31 ` Julien Grall
2017-05-26 17:35 ` [PATCH v10 32/32] ARM: vITS: create ITS subnodes for Dom0 DT Andre Przywara
2017-06-02 17:33 ` Julien Grall
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c76e32bc-3b3b-e52a-e898-7ca88d2be796@arm.com \
--to=andre.przywara@arm.com \
--cc=Vijaya.Kumar@caviumnetworks.com \
--cc=julien.grall@arm.com \
--cc=shankerd@codeaurora.org \
--cc=sstabellini@kernel.org \
--cc=vijay.kilari@gmail.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).