From: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
To: Jan Beulich <jbeulich@suse.com>
Cc: "julien@xen.org" <julien@xen.org>,
"sstabellini@kernel.org" <sstabellini@kernel.org>,
"Oleksandr Tyshchenko" <Oleksandr_Tyshchenko@epam.com>,
"Volodymyr Babchuk" <Volodymyr_Babchuk@epam.com>,
"Artem Mygaiev" <Artem_Mygaiev@epam.com>,
"andrew.cooper3@citrix.com" <andrew.cooper3@citrix.com>,
"george.dunlap@citrix.com" <george.dunlap@citrix.com>,
"paul@xen.org" <paul@xen.org>,
"Bertrand Marquis" <bertrand.marquis@arm.com>,
"Rahul Singh" <rahul.singh@arm.com>,
"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
"Roger Pau Monné" <roger.pau@citrix.com>
Subject: Re: [PATCH v6 03/13] vpci: move lock outside of struct vpci
Date: Mon, 7 Feb 2022 16:07:37 +0000 [thread overview]
Message-ID: <4276fe72-558b-7d95-6430-d4330989dbe8@epam.com> (raw)
In-Reply-To: <21c8125a-2f79-1a4a-494c-db9e4be2a1a6@suse.com>
On 07.02.22 17:26, Jan Beulich wrote:
> On 07.02.2022 16:11, Oleksandr Andrushchenko wrote:
>>
>> On 07.02.22 16:35, Oleksandr Andrushchenko wrote:
>>> On 07.02.22 16:27, Roger Pau Monné wrote:
>>>> On Mon, Feb 07, 2022 at 03:11:03PM +0100, Jan Beulich wrote:
>>>>> On 07.02.2022 14:53, Oleksandr Andrushchenko wrote:
>>>>>> On 07.02.22 14:46, Roger Pau Monné wrote:
>>>>>>> I think the per-domain rwlock seems like a good option. I would do
>>>>>>> that as a pre-patch.
>>>>>> It is. But it seems it won't solve the thing we started this adventure for:
>>>>>>
>>>>>> With per-domain read lock and still ABBA in modify_bars (hope the below
>>>>>> is correctly seen with a monospace font):
>>>>>>
>>>>>> cpu0: vpci_write-> d->RLock -> pdev1->lock -> rom_write -> modify_bars: tmp (pdev2) ->lock
>>>>>> cpu1: vpci_write-> d->RLock pdev2->lock -> cmd_write -> modify_bars: tmp (pdev1) ->lock
>>>>>>
>>>>>> There is no API to upgrade read lock to write lock in modify_bars which could help,
>>>>>> so in both cases vpci_write should take write lock.
>>>>> Hmm, yes, I think you're right: It's not modify_bars() itself which needs
>>>>> to acquire the write lock, but its (perhaps indirect) caller. Effectively
>>>>> vpci_write() would need to take the write lock if the range written
>>>>> overlaps the BARs or the command register.
>>>> I'm confused. If we use a per-domain rwlock approach there would be no
>>>> need to lock tmp again in modify_bars, because we should hold the
>>>> rwlock in write mode, so there's no ABBA?
>>> this is only possible with what you wrote below:
>>>> We will have however to drop the per domain read and vpci locks and
>>>> pick the per-domain lock in write mode.
>>> I think this is going to be unreliable. We need a reliable way to
>>> upgrade read lock to write lock.
>>> Then, we can drop pdev->vpci_lock at all, because we are always
>>> protected with d->rwlock and those who want to free pdev->vpci
>>> will use write lock.
>>>
>>> So, per-domain rwlock with write upgrade implemented minus pdev->vpci
>>> should do the trick
>> Linux doesn't implement write upgrade and it seems for a reason [1]:
>> "Also, you cannot “upgrade” a read-lock to a write-lock, so if you at _any_ time
>> need to do any changes (even if you don’t do it every time), you have to get
>> the write-lock at the very beginning."
>>
>> So, I am not sure we can have the same for Xen...
>>
>> At the moment I see at least two possible ways to solve the issue:
>> 1. Make vpci_write use write lock, thus make all write accesses synchronized
>> for the given domain, read are fully parallel
> 1b. Make vpci_write use write lock for writes to command register and BARs
> only; keep using the read lock for all other writes.
I am not quite sure how to do that. Do you mean something like:
void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
uint32_t data)
[snip]
list_for_each_entry ( r, &pdev->vpci->handlers, node )
{
[snip]
if ( r->needs_write_lock)
write_lock(d->vpci_lock)
else
read_lock(d->vpci_lock)
....
And provide rw as an argument to:
int vpci_add_register(struct vpci *vpci, vpci_read_t *read_handler,
vpci_write_t *write_handler, unsigned int offset,
unsigned int size, void *data, --->>> bool write_path <<<-----)
Is this what you mean?
With the above, if we have d->vpci_lock, I think we can drop
pdev->vpci_lock at all
Thank you,
Oleksandr
P.S. I don't think you mean we just drop the read lock and acquire write lock
as it leads to the mentioned before unreliability.
next prev parent reply other threads:[~2022-02-07 16:08 UTC|newest]
Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-04 6:34 [PATCH v6 00/13] PCI devices passthrough on Arm, part 3 Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 01/13] xen/pci: arm: add stub for is_memory_hole Oleksandr Andrushchenko
2022-02-04 8:51 ` Julien Grall
2022-02-04 9:01 ` Oleksandr Andrushchenko
2022-02-04 9:41 ` Julien Grall
2022-02-04 9:47 ` Oleksandr Andrushchenko
2022-02-04 9:57 ` Julien Grall
2022-02-04 10:35 ` Oleksandr Andrushchenko
2022-02-04 11:00 ` Julien Grall
2022-02-04 11:25 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 02/13] rangeset: add RANGESETF_no_print flag Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 03/13] vpci: move lock outside of struct vpci Oleksandr Andrushchenko
2022-02-04 7:52 ` Jan Beulich
2022-02-04 8:13 ` Oleksandr Andrushchenko
2022-02-04 8:36 ` Jan Beulich
2022-02-04 8:58 ` Oleksandr Andrushchenko
2022-02-04 9:15 ` Jan Beulich
2022-02-04 10:12 ` Oleksandr Andrushchenko
2022-02-04 10:49 ` Jan Beulich
2022-02-04 11:13 ` Roger Pau Monné
2022-02-04 11:37 ` Jan Beulich
2022-02-04 12:37 ` Oleksandr Andrushchenko
2022-02-04 12:47 ` Jan Beulich
2022-02-04 12:53 ` Oleksandr Andrushchenko
2022-02-04 13:03 ` Jan Beulich
2022-02-04 13:06 ` Roger Pau Monné
2022-02-04 14:43 ` Oleksandr Andrushchenko
2022-02-04 14:57 ` Roger Pau Monné
2022-02-07 11:08 ` Oleksandr Andrushchenko
2022-02-07 12:34 ` Jan Beulich
2022-02-07 12:57 ` Oleksandr Andrushchenko
2022-02-07 13:02 ` Jan Beulich
2022-02-07 12:46 ` Roger Pau Monné
2022-02-07 13:53 ` Oleksandr Andrushchenko
2022-02-07 14:11 ` Jan Beulich
2022-02-07 14:27 ` Roger Pau Monné
2022-02-07 14:33 ` Jan Beulich
2022-02-07 14:35 ` Oleksandr Andrushchenko
2022-02-07 15:11 ` Oleksandr Andrushchenko
2022-02-07 15:26 ` Jan Beulich
2022-02-07 16:07 ` Oleksandr Andrushchenko [this message]
2022-02-07 16:15 ` Jan Beulich
2022-02-07 16:21 ` Oleksandr Andrushchenko
2022-02-07 16:37 ` Jan Beulich
2022-02-07 16:44 ` Oleksandr Andrushchenko
2022-02-08 7:35 ` Oleksandr Andrushchenko
2022-02-08 8:57 ` Jan Beulich
2022-02-08 9:03 ` Oleksandr Andrushchenko
2022-02-08 10:50 ` Roger Pau Monné
2022-02-08 11:13 ` Oleksandr Andrushchenko
2022-02-08 13:38 ` Roger Pau Monné
2022-02-08 13:52 ` Oleksandr Andrushchenko
2022-02-08 8:53 ` Jan Beulich
2022-02-08 9:00 ` Oleksandr Andrushchenko
2022-02-08 10:11 ` Roger Pau Monné
2022-02-08 10:32 ` Oleksandr Andrushchenko
2022-02-07 16:08 ` Roger Pau Monné
2022-02-07 16:12 ` Jan Beulich
2022-02-07 14:28 ` Oleksandr Andrushchenko
2022-02-07 14:19 ` Roger Pau Monné
2022-02-07 14:27 ` Oleksandr Andrushchenko
2022-02-04 11:37 ` Oleksandr Andrushchenko
2022-02-04 12:15 ` Roger Pau Monné
2022-02-04 10:57 ` Roger Pau Monné
2022-02-04 6:34 ` [PATCH v6 04/13] vpci: restrict unhandled read/write operations for guests Oleksandr Andrushchenko
2022-02-04 14:11 ` Jan Beulich
2022-02-04 14:24 ` Oleksandr Andrushchenko
2022-02-08 8:00 ` Oleksandr Andrushchenko
2022-02-08 9:04 ` Jan Beulich
2022-02-08 9:09 ` Oleksandr Andrushchenko
2022-02-08 9:05 ` Roger Pau Monné
2022-02-08 9:10 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 05/13] vpci: add hooks for PCI device assign/de-assign Oleksandr Andrushchenko
2022-02-07 16:28 ` Jan Beulich
2022-02-08 8:32 ` Oleksandr Andrushchenko
2022-02-08 9:13 ` Jan Beulich
2022-02-08 9:27 ` Oleksandr Andrushchenko
2022-02-08 9:44 ` Jan Beulich
2022-02-08 9:55 ` Oleksandr Andrushchenko
2022-02-08 10:09 ` Jan Beulich
2022-02-08 10:22 ` Oleksandr Andrushchenko
2022-02-08 10:29 ` Jan Beulich
2022-02-08 10:52 ` Oleksandr Andrushchenko
2022-02-08 11:00 ` Jan Beulich
2022-02-08 11:25 ` Oleksandr Andrushchenko
2022-02-10 8:21 ` Oleksandr Andrushchenko
2022-02-10 9:22 ` Jan Beulich
2022-02-10 9:33 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 06/13] vpci/header: implement guest BAR register handlers Oleksandr Andrushchenko
2022-02-07 17:06 ` Jan Beulich
2022-02-08 8:06 ` Oleksandr Andrushchenko
2022-02-08 9:16 ` Jan Beulich
2022-02-08 9:29 ` Roger Pau Monné
2022-02-08 9:25 ` Roger Pau Monné
2022-02-08 9:31 ` Oleksandr Andrushchenko
2022-02-08 9:48 ` Jan Beulich
2022-02-08 9:57 ` Oleksandr Andrushchenko
2022-02-08 10:15 ` Jan Beulich
2022-02-08 10:29 ` Oleksandr Andrushchenko
2022-02-08 13:58 ` Roger Pau Monné
2022-02-04 6:34 ` [PATCH v6 07/13] vpci/header: handle p2m range sets per BAR Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 08/13] vpci/header: program p2m with guest BAR view Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 09/13] vpci/header: emulate PCI_COMMAND register for guests Oleksandr Andrushchenko
2022-02-04 14:25 ` Jan Beulich
2022-02-08 8:13 ` Oleksandr Andrushchenko
2022-02-08 9:33 ` Jan Beulich
2022-02-08 9:38 ` Oleksandr Andrushchenko
2022-02-08 9:52 ` Jan Beulich
2022-02-08 9:58 ` Oleksandr Andrushchenko
2022-02-08 11:11 ` Roger Pau Monné
2022-02-08 11:29 ` Oleksandr Andrushchenko
2022-02-08 14:09 ` Roger Pau Monné
2022-02-08 14:13 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 10/13] vpci/header: reset the command register when adding devices Oleksandr Andrushchenko
2022-02-04 14:30 ` Jan Beulich
2022-02-04 14:37 ` Oleksandr Andrushchenko
2022-02-07 7:29 ` Jan Beulich
2022-02-07 11:27 ` Oleksandr Andrushchenko
2022-02-07 12:38 ` Jan Beulich
2022-02-07 12:51 ` Oleksandr Andrushchenko
2022-02-07 12:54 ` Jan Beulich
2022-02-07 14:17 ` Oleksandr Andrushchenko
2022-02-07 14:31 ` Jan Beulich
2022-02-07 14:46 ` Oleksandr Andrushchenko
2022-02-07 15:05 ` Jan Beulich
2022-02-07 15:14 ` Oleksandr Andrushchenko
2022-02-07 15:28 ` Jan Beulich
2022-02-07 15:59 ` Oleksandr Andrushchenko
2022-02-10 12:54 ` Oleksandr Andrushchenko
2022-02-10 13:36 ` Jan Beulich
2022-02-10 13:56 ` Oleksandr Andrushchenko
2022-02-10 12:59 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 11/13] vpci: add initial support for virtual PCI bus topology Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 12/13] xen/arm: translate virtual PCI bus topology for guests Oleksandr Andrushchenko
2022-02-04 7:56 ` Jan Beulich
2022-02-04 8:18 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 13/13] xen/arm: account IO handlers for emulated PCI MSI-X Oleksandr Andrushchenko
2022-02-11 15:28 ` Julien Grall
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4276fe72-558b-7d95-6430-d4330989dbe8@epam.com \
--to=oleksandr_andrushchenko@epam.com \
--cc=Artem_Mygaiev@epam.com \
--cc=Oleksandr_Tyshchenko@epam.com \
--cc=Volodymyr_Babchuk@epam.com \
--cc=andrew.cooper3@citrix.com \
--cc=bertrand.marquis@arm.com \
--cc=george.dunlap@citrix.com \
--cc=jbeulich@suse.com \
--cc=julien@xen.org \
--cc=paul@xen.org \
--cc=rahul.singh@arm.com \
--cc=roger.pau@citrix.com \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).