From: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
To: Jan Beulich <jbeulich@suse.com>
Cc: "julien@xen.org" <julien@xen.org>,
"sstabellini@kernel.org" <sstabellini@kernel.org>,
"Oleksandr Tyshchenko" <Oleksandr_Tyshchenko@epam.com>,
"Volodymyr Babchuk" <Volodymyr_Babchuk@epam.com>,
"Artem Mygaiev" <Artem_Mygaiev@epam.com>,
"andrew.cooper3@citrix.com" <andrew.cooper3@citrix.com>,
"george.dunlap@citrix.com" <george.dunlap@citrix.com>,
"paul@xen.org" <paul@xen.org>,
"Bertrand Marquis" <bertrand.marquis@arm.com>,
"Rahul Singh" <rahul.singh@arm.com>,
"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
"Roger Pau Monné" <roger.pau@citrix.com>,
"Oleksandr Andrushchenko" <Oleksandr_Andrushchenko@epam.com>
Subject: Re: [PATCH v6 03/13] vpci: move lock outside of struct vpci
Date: Fri, 4 Feb 2022 12:53:20 +0000 [thread overview]
Message-ID: <deef8468-d607-e49e-f456-6a8013329ca1@epam.com> (raw)
In-Reply-To: <a8fc599c-f620-c4d1-2077-c57068f46a7f@suse.com>
On 04.02.22 14:47, Jan Beulich wrote:
> On 04.02.2022 13:37, Oleksandr Andrushchenko wrote:
>>
>> On 04.02.22 13:37, Jan Beulich wrote:
>>> On 04.02.2022 12:13, Roger Pau Monné wrote:
>>>> On Fri, Feb 04, 2022 at 11:49:18AM +0100, Jan Beulich wrote:
>>>>> On 04.02.2022 11:12, Oleksandr Andrushchenko wrote:
>>>>>> On 04.02.22 11:15, Jan Beulich wrote:
>>>>>>> On 04.02.2022 09:58, Oleksandr Andrushchenko wrote:
>>>>>>>> On 04.02.22 09:52, Jan Beulich wrote:
>>>>>>>>> On 04.02.2022 07:34, Oleksandr Andrushchenko wrote:
>>>>>>>>>> @@ -285,6 +286,12 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only)
>>>>>>>>>> continue;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> + spin_lock(&tmp->vpci_lock);
>>>>>>>>>> + if ( !tmp->vpci )
>>>>>>>>>> + {
>>>>>>>>>> + spin_unlock(&tmp->vpci_lock);
>>>>>>>>>> + continue;
>>>>>>>>>> + }
>>>>>>>>>> for ( i = 0; i < ARRAY_SIZE(tmp->vpci->header.bars); i++ )
>>>>>>>>>> {
>>>>>>>>>> const struct vpci_bar *bar = &tmp->vpci->header.bars[i];
>>>>>>>>>> @@ -303,12 +310,14 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only)
>>>>>>>>>> rc = rangeset_remove_range(mem, start, end);
>>>>>>>>>> if ( rc )
>>>>>>>>>> {
>>>>>>>>>> + spin_unlock(&tmp->vpci_lock);
>>>>>>>>>> printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n",
>>>>>>>>>> start, end, rc);
>>>>>>>>>> rangeset_destroy(mem);
>>>>>>>>>> return rc;
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>> + spin_unlock(&tmp->vpci_lock);
>>>>>>>>>> }
>>>>>>>>> At the first glance this simply looks like another unjustified (in the
>>>>>>>>> description) change, as you're not converting anything here but you
>>>>>>>>> actually add locking (and I realize this was there before, so I'm sorry
>>>>>>>>> for not pointing this out earlier).
>>>>>>>> Well, I thought that the description already has "...the lock can be
>>>>>>>> used (and in a few cases is used right away) to check whether vpci
>>>>>>>> is present" and this is enough for such uses as here.
>>>>>>>>> But then I wonder whether you
>>>>>>>>> actually tested this, since I can't help getting the impression that
>>>>>>>>> you're introducing a live-lock: The function is called from cmd_write()
>>>>>>>>> and rom_write(), which in turn are called out of vpci_write(). Yet that
>>>>>>>>> function already holds the lock, and the lock is not (currently)
>>>>>>>>> recursive. (For the 3rd caller of the function - init_bars() - otoh
>>>>>>>>> the locking looks to be entirely unnecessary.)
>>>>>>>> Well, you are correct: if tmp != pdev then it is correct to acquire
>>>>>>>> the lock. But if tmp == pdev and rom_only == true
>>>>>>>> then we'll deadlock.
>>>>>>>>
>>>>>>>> It seems we need to have the locking conditional, e.g. only lock
>>>>>>>> if tmp != pdev
>>>>>>> Which will address the live-lock, but introduce ABBA deadlock potential
>>>>>>> between the two locks.
>>>>>> I am not sure I can suggest a better solution here
>>>>>> @Roger, @Jan, could you please help here?
>>>>> Well, first of all I'd like to mention that while it may have been okay to
>>>>> not hold pcidevs_lock here for Dom0, it surely needs acquiring when dealing
>>>>> with DomU-s' lists of PCI devices. The requirement really applies to the
>>>>> other use of for_each_pdev() as well (in vpci_dump_msi()), except that
>>>>> there it probably wants to be a try-lock.
>>>>>
>>>>> Next I'd like to point out that here we have the still pending issue of
>>>>> how to deal with hidden devices, which Dom0 can access. See my RFC patch
>>>>> "vPCI: account for hidden devices in modify_bars()". Whatever the solution
>>>>> here, I think it wants to at least account for the extra need there.
>>>> Yes, sorry, I should take care of that.
>>>>
>>>>> Now it is quite clear that pcidevs_lock isn't going to help with avoiding
>>>>> the deadlock, as it's imo not an option at all to acquire that lock
>>>>> everywhere else you access ->vpci (or else the vpci lock itself would be
>>>>> pointless). But a per-domain auxiliary r/w lock may help: Other paths
>>>>> would acquire it in read mode, and here you'd acquire it in write mode (in
>>>>> the former case around the vpci lock, while in the latter case there may
>>>>> then not be any need to acquire the individual vpci locks at all). FTAOD:
>>>>> I haven't fully thought through all implications (and hence whether this is
>>>>> viable in the first place); I expect you will, documenting what you've
>>>>> found in the resulting patch description. Of course the double lock
>>>>> acquire/release would then likely want hiding in helper functions.
>>>> I've been also thinking about this, and whether it's really worth to
>>>> have a per-device lock rather than a per-domain one that protects all
>>>> vpci regions of the devices assigned to the domain.
>>>>
>>>> The OS is likely to serialize accesses to the PCI config space anyway,
>>>> and the only place I could see a benefit of having per-device locks is
>>>> in the handling of MSI-X tables, as the handling of the mask bit is
>>>> likely very performance sensitive, so adding a per-domain lock there
>>>> could be a bottleneck.
>>> Hmm, with method 1 accesses serializing globally is basically
>>> unavoidable, but with MMCFG I see no reason why OSes may not (move
>>> to) permit(ting) parallel accesses, with serialization perhaps done
>>> only at device level. See our own pci_config_lock, which applies to
>>> only method 1 accesses; we don't look to be serializing MMCFG
>>> accesses at all.
>>>
>>>> We could alternatively do a per-domain rwlock for vpci and special case
>>>> the MSI-X area to also have a per-device specific lock. At which point
>>>> it becomes fairly similar to what you propose.
>> @Jan, @Roger
>>
>> 1. d->vpci_lock - rwlock <- this protects vpci
>> 2. pdev->vpci->msix_tbl_lock - rwlock <- this protects MSI-X tables
>> or should it better be pdev->msix_tbl_lock as MSI-X tables don't
>> really depend on vPCI?
> If so, perhaps indeed better the latter. But as said in reply to Roger,
> I'm not convinced (yet) that doing away with the per-device lock is a
> good move. As said there - we're ourselves doing fully parallel MMCFG
> accesses, so OSes ought to be fine to do so, too.
But with pdev->vpci_lock we face ABBA...
>
> Jan
>
>
next prev parent reply other threads:[~2022-02-04 12:53 UTC|newest]
Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-04 6:34 [PATCH v6 00/13] PCI devices passthrough on Arm, part 3 Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 01/13] xen/pci: arm: add stub for is_memory_hole Oleksandr Andrushchenko
2022-02-04 8:51 ` Julien Grall
2022-02-04 9:01 ` Oleksandr Andrushchenko
2022-02-04 9:41 ` Julien Grall
2022-02-04 9:47 ` Oleksandr Andrushchenko
2022-02-04 9:57 ` Julien Grall
2022-02-04 10:35 ` Oleksandr Andrushchenko
2022-02-04 11:00 ` Julien Grall
2022-02-04 11:25 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 02/13] rangeset: add RANGESETF_no_print flag Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 03/13] vpci: move lock outside of struct vpci Oleksandr Andrushchenko
2022-02-04 7:52 ` Jan Beulich
2022-02-04 8:13 ` Oleksandr Andrushchenko
2022-02-04 8:36 ` Jan Beulich
2022-02-04 8:58 ` Oleksandr Andrushchenko
2022-02-04 9:15 ` Jan Beulich
2022-02-04 10:12 ` Oleksandr Andrushchenko
2022-02-04 10:49 ` Jan Beulich
2022-02-04 11:13 ` Roger Pau Monné
2022-02-04 11:37 ` Jan Beulich
2022-02-04 12:37 ` Oleksandr Andrushchenko
2022-02-04 12:47 ` Jan Beulich
2022-02-04 12:53 ` Oleksandr Andrushchenko [this message]
2022-02-04 13:03 ` Jan Beulich
2022-02-04 13:06 ` Roger Pau Monné
2022-02-04 14:43 ` Oleksandr Andrushchenko
2022-02-04 14:57 ` Roger Pau Monné
2022-02-07 11:08 ` Oleksandr Andrushchenko
2022-02-07 12:34 ` Jan Beulich
2022-02-07 12:57 ` Oleksandr Andrushchenko
2022-02-07 13:02 ` Jan Beulich
2022-02-07 12:46 ` Roger Pau Monné
2022-02-07 13:53 ` Oleksandr Andrushchenko
2022-02-07 14:11 ` Jan Beulich
2022-02-07 14:27 ` Roger Pau Monné
2022-02-07 14:33 ` Jan Beulich
2022-02-07 14:35 ` Oleksandr Andrushchenko
2022-02-07 15:11 ` Oleksandr Andrushchenko
2022-02-07 15:26 ` Jan Beulich
2022-02-07 16:07 ` Oleksandr Andrushchenko
2022-02-07 16:15 ` Jan Beulich
2022-02-07 16:21 ` Oleksandr Andrushchenko
2022-02-07 16:37 ` Jan Beulich
2022-02-07 16:44 ` Oleksandr Andrushchenko
2022-02-08 7:35 ` Oleksandr Andrushchenko
2022-02-08 8:57 ` Jan Beulich
2022-02-08 9:03 ` Oleksandr Andrushchenko
2022-02-08 10:50 ` Roger Pau Monné
2022-02-08 11:13 ` Oleksandr Andrushchenko
2022-02-08 13:38 ` Roger Pau Monné
2022-02-08 13:52 ` Oleksandr Andrushchenko
2022-02-08 8:53 ` Jan Beulich
2022-02-08 9:00 ` Oleksandr Andrushchenko
2022-02-08 10:11 ` Roger Pau Monné
2022-02-08 10:32 ` Oleksandr Andrushchenko
2022-02-07 16:08 ` Roger Pau Monné
2022-02-07 16:12 ` Jan Beulich
2022-02-07 14:28 ` Oleksandr Andrushchenko
2022-02-07 14:19 ` Roger Pau Monné
2022-02-07 14:27 ` Oleksandr Andrushchenko
2022-02-04 11:37 ` Oleksandr Andrushchenko
2022-02-04 12:15 ` Roger Pau Monné
2022-02-04 10:57 ` Roger Pau Monné
2022-02-04 6:34 ` [PATCH v6 04/13] vpci: restrict unhandled read/write operations for guests Oleksandr Andrushchenko
2022-02-04 14:11 ` Jan Beulich
2022-02-04 14:24 ` Oleksandr Andrushchenko
2022-02-08 8:00 ` Oleksandr Andrushchenko
2022-02-08 9:04 ` Jan Beulich
2022-02-08 9:09 ` Oleksandr Andrushchenko
2022-02-08 9:05 ` Roger Pau Monné
2022-02-08 9:10 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 05/13] vpci: add hooks for PCI device assign/de-assign Oleksandr Andrushchenko
2022-02-07 16:28 ` Jan Beulich
2022-02-08 8:32 ` Oleksandr Andrushchenko
2022-02-08 9:13 ` Jan Beulich
2022-02-08 9:27 ` Oleksandr Andrushchenko
2022-02-08 9:44 ` Jan Beulich
2022-02-08 9:55 ` Oleksandr Andrushchenko
2022-02-08 10:09 ` Jan Beulich
2022-02-08 10:22 ` Oleksandr Andrushchenko
2022-02-08 10:29 ` Jan Beulich
2022-02-08 10:52 ` Oleksandr Andrushchenko
2022-02-08 11:00 ` Jan Beulich
2022-02-08 11:25 ` Oleksandr Andrushchenko
2022-02-10 8:21 ` Oleksandr Andrushchenko
2022-02-10 9:22 ` Jan Beulich
2022-02-10 9:33 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 06/13] vpci/header: implement guest BAR register handlers Oleksandr Andrushchenko
2022-02-07 17:06 ` Jan Beulich
2022-02-08 8:06 ` Oleksandr Andrushchenko
2022-02-08 9:16 ` Jan Beulich
2022-02-08 9:29 ` Roger Pau Monné
2022-02-08 9:25 ` Roger Pau Monné
2022-02-08 9:31 ` Oleksandr Andrushchenko
2022-02-08 9:48 ` Jan Beulich
2022-02-08 9:57 ` Oleksandr Andrushchenko
2022-02-08 10:15 ` Jan Beulich
2022-02-08 10:29 ` Oleksandr Andrushchenko
2022-02-08 13:58 ` Roger Pau Monné
2022-02-04 6:34 ` [PATCH v6 07/13] vpci/header: handle p2m range sets per BAR Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 08/13] vpci/header: program p2m with guest BAR view Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 09/13] vpci/header: emulate PCI_COMMAND register for guests Oleksandr Andrushchenko
2022-02-04 14:25 ` Jan Beulich
2022-02-08 8:13 ` Oleksandr Andrushchenko
2022-02-08 9:33 ` Jan Beulich
2022-02-08 9:38 ` Oleksandr Andrushchenko
2022-02-08 9:52 ` Jan Beulich
2022-02-08 9:58 ` Oleksandr Andrushchenko
2022-02-08 11:11 ` Roger Pau Monné
2022-02-08 11:29 ` Oleksandr Andrushchenko
2022-02-08 14:09 ` Roger Pau Monné
2022-02-08 14:13 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 10/13] vpci/header: reset the command register when adding devices Oleksandr Andrushchenko
2022-02-04 14:30 ` Jan Beulich
2022-02-04 14:37 ` Oleksandr Andrushchenko
2022-02-07 7:29 ` Jan Beulich
2022-02-07 11:27 ` Oleksandr Andrushchenko
2022-02-07 12:38 ` Jan Beulich
2022-02-07 12:51 ` Oleksandr Andrushchenko
2022-02-07 12:54 ` Jan Beulich
2022-02-07 14:17 ` Oleksandr Andrushchenko
2022-02-07 14:31 ` Jan Beulich
2022-02-07 14:46 ` Oleksandr Andrushchenko
2022-02-07 15:05 ` Jan Beulich
2022-02-07 15:14 ` Oleksandr Andrushchenko
2022-02-07 15:28 ` Jan Beulich
2022-02-07 15:59 ` Oleksandr Andrushchenko
2022-02-10 12:54 ` Oleksandr Andrushchenko
2022-02-10 13:36 ` Jan Beulich
2022-02-10 13:56 ` Oleksandr Andrushchenko
2022-02-10 12:59 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 11/13] vpci: add initial support for virtual PCI bus topology Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 12/13] xen/arm: translate virtual PCI bus topology for guests Oleksandr Andrushchenko
2022-02-04 7:56 ` Jan Beulich
2022-02-04 8:18 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 13/13] xen/arm: account IO handlers for emulated PCI MSI-X Oleksandr Andrushchenko
2022-02-11 15:28 ` Julien Grall
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=deef8468-d607-e49e-f456-6a8013329ca1@epam.com \
--to=oleksandr_andrushchenko@epam.com \
--cc=Artem_Mygaiev@epam.com \
--cc=Oleksandr_Tyshchenko@epam.com \
--cc=Volodymyr_Babchuk@epam.com \
--cc=andrew.cooper3@citrix.com \
--cc=bertrand.marquis@arm.com \
--cc=george.dunlap@citrix.com \
--cc=jbeulich@suse.com \
--cc=julien@xen.org \
--cc=paul@xen.org \
--cc=rahul.singh@arm.com \
--cc=roger.pau@citrix.com \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).