From: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
To: Jan Beulich <jbeulich@suse.com>,
"roger.pau@citrix.com" <roger.pau@citrix.com>
Cc: "julien@xen.org" <julien@xen.org>,
"sstabellini@kernel.org" <sstabellini@kernel.org>,
Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>,
Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>,
Artem Mygaiev <Artem_Mygaiev@epam.com>,
"andrew.cooper3@citrix.com" <andrew.cooper3@citrix.com>,
"george.dunlap@citrix.com" <george.dunlap@citrix.com>,
"paul@xen.org" <paul@xen.org>,
Bertrand Marquis <bertrand.marquis@arm.com>,
Rahul Singh <rahul.singh@arm.com>,
"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
Subject: Re: [PATCH v4 02/11] vpci: cancel pending map/unmap on vpci removal
Date: Tue, 16 Nov 2021 13:27:29 +0000 [thread overview]
Message-ID: <cc7fb79b-5a43-3bdc-4621-097a01982f49@epam.com> (raw)
In-Reply-To: <b8de7b20-539b-23e7-1ee4-9a777be296a3@suse.com>
On 16.11.21 13:38, Jan Beulich wrote:
> On 16.11.2021 09:23, Oleksandr Andrushchenko wrote:
>>
>> On 16.11.21 10:01, Jan Beulich wrote:
>>> On 16.11.2021 08:32, Oleksandr Andrushchenko wrote:
>>>> On 15.11.21 18:56, Jan Beulich wrote:
>>>>> On 05.11.2021 07:56, Oleksandr Andrushchenko wrote:
>>>>>> @@ -165,6 +164,18 @@ bool vpci_process_pending(struct vcpu *v)
>>>>>> return false;
>>>>>> }
>>>>>>
>>>>>> +void vpci_cancel_pending(const struct pci_dev *pdev)
>>>>>> +{
>>>>>> + struct vcpu *v = current;
>>>>>> +
>>>>>> + /* Cancel any pending work now. */
>>>>> Doesn't "any" include pending work on all vCPU-s of the guest, not
>>>>> just current? Is current even relevant (as in: part of the correct
>>>>> domain), considering ...
>>>>>
>>>>>> --- a/xen/drivers/vpci/vpci.c
>>>>>> +++ b/xen/drivers/vpci/vpci.c
>>>>>> @@ -51,6 +51,8 @@ void vpci_remove_device(struct pci_dev *pdev)
>>>>>> xfree(r);
>>>>>> }
>>>>>> spin_unlock(&pdev->vpci->lock);
>>>>>> +
>>>>>> + vpci_cancel_pending(pdev);
>>>>> ... this code path, when coming here from pci_{add,remove}_device()?
>>>>>
>>>>> I can agree that there's a problem here, but I think you need to
>>>>> properly (i.e. in a race free manner) drain pending work.
>>>> Yes, the code is inconsistent with this respect. I am thinking about:
>>>>
>>>> void vpci_cancel_pending(const struct pci_dev *pdev)
>>>> {
>>>> struct domain *d = pdev->domain;
>>>> struct vcpu *v;
>>>>
>>>> /* Cancel any pending work now. */
>>>> domain_lock(d);
>>>> for_each_vcpu ( d, v )
>>>> {
>>>> vcpu_pause(v);
>>>> if ( v->vpci.mem && v->vpci.pdev == pdev)
>>> Nit: Same style issue as in the original patch.
>> Will fix
>>>> {
>>>> rangeset_destroy(v->vpci.mem);
>>>> v->vpci.mem = NULL;
>>>> }
>>>> vcpu_unpause(v);
>>>> }
>>>> domain_unlock(d);
>>>> }
>>>>
>>>> which seems to solve all the concerns. Is this what you mean?
>>> Something along these lines. I expect you will want to make use of
>>> domain_pause_except_self(),
>> Yes, this is what is needed here, thanks. The only question is that
>>
>> int domain_pause_except_self(struct domain *d)
>> {
>> [snip]
>> /* Avoid racing with other vcpus which may want to be pausing us */
>> if ( !spin_trylock(&d->hypercall_deadlock_mutex) )
>> return -ERESTART;
>>
>> so it is not clear what do we do in case of -ERESTART: do we want to spin?
>> Otherwise we will leave the job not done effectively not canceling the
>> pending work. Any idea other then spinning?
> Depends on the call chain you come through. There may need to be some
> rearrangements such that you may be able to preempt the enclosing
> hypercall.
Well, there are three places which may lead to the pending work
needs to be canceled:
MMIO trap -> vpci_write -> vpci_cmd_write -> modify_bars -> vpci_cancel_pending (on modify_bars fail path)
PHYSDEVOP_pci_device_add -> pci_add_device (error path) -> vpci_remove_device -> vpci_cancel_pending
PHYSDEVOP_pci_device_remove -> pci_remove_device -> vpci_remove_device -> vpci_cancel_pending
So, taking into account the MMIO path, I think about the below code
/*
* Cancel any pending work now.
*
* FIXME: this can be called from an MMIO trap handler's error
* path, so we cannot just return an error code here, so upper
* layers can handle it. The best we can do is to still try
* removing the range sets.
*/
while ( (rc = domain_pause_except_self(d)) == -ERESTART )
cpu_relax();
if ( rc )
printk(XENLOG_G_ERR
"Failed to pause vCPUs while canceling vPCI map/unmap for %pp %pd: %d\n",
&pdev->sbdf, pdev->domain, rc);
I am not sure how to handle this otherwise
@Roger, do you see any other good way?
>
> Jan
>
Thank you,
Oleksandr
next prev parent reply other threads:[~2021-11-16 13:28 UTC|newest]
Thread overview: 101+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-05 6:56 [PATCH v4 00/11] PCI devices passthrough on Arm, part 3 Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 01/11] vpci: fix function attributes for vpci_process_pending Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 02/11] vpci: cancel pending map/unmap on vpci removal Oleksandr Andrushchenko
2021-11-15 16:56 ` Jan Beulich
2021-11-16 7:32 ` Oleksandr Andrushchenko
2021-11-16 8:01 ` Jan Beulich
2021-11-16 8:23 ` Oleksandr Andrushchenko
2021-11-16 11:38 ` Jan Beulich
2021-11-16 13:27 ` Oleksandr Andrushchenko [this message]
2021-11-16 14:11 ` Jan Beulich
2021-11-16 13:41 ` Oleksandr Andrushchenko
2021-11-16 14:12 ` Jan Beulich
2021-11-16 14:24 ` Oleksandr Andrushchenko
2021-11-16 14:37 ` Oleksandr Andrushchenko
2021-11-16 16:09 ` Jan Beulich
2021-11-16 18:02 ` Julien Grall
2021-11-18 12:57 ` Oleksandr Andrushchenko
2021-11-17 8:28 ` Jan Beulich
2021-11-18 7:49 ` Oleksandr Andrushchenko
2021-11-18 8:36 ` Jan Beulich
2021-11-18 8:54 ` Oleksandr Andrushchenko
2021-11-18 9:15 ` Jan Beulich
2021-11-18 9:32 ` Oleksandr Andrushchenko
2021-11-18 13:25 ` Jan Beulich
2021-11-18 13:48 ` Oleksandr Andrushchenko
2021-11-18 14:04 ` Roger Pau Monné
2021-11-18 14:14 ` Oleksandr Andrushchenko
2021-11-18 14:35 ` Jan Beulich
2021-11-18 15:11 ` Oleksandr Andrushchenko
2021-11-18 15:16 ` Jan Beulich
2021-11-18 15:21 ` Oleksandr Andrushchenko
2021-11-18 15:41 ` Jan Beulich
2021-11-18 15:46 ` Oleksandr Andrushchenko
2021-11-18 15:53 ` Jan Beulich
2021-11-19 12:34 ` Oleksandr Andrushchenko
2021-11-19 13:00 ` Jan Beulich
2021-11-19 13:16 ` Oleksandr Andrushchenko
2021-11-19 13:25 ` Jan Beulich
2021-11-19 13:34 ` Oleksandr Andrushchenko
2021-11-22 14:21 ` Oleksandr Andrushchenko
2021-11-22 14:37 ` Jan Beulich
2021-11-22 14:45 ` Oleksandr Andrushchenko
2021-11-22 14:57 ` Jan Beulich
2021-11-22 15:02 ` Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 03/11] vpci: make vpci registers removal a dedicated function Oleksandr Andrushchenko
2021-11-15 16:57 ` Jan Beulich
2021-11-16 8:02 ` Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 04/11] vpci: add hooks for PCI device assign/de-assign Oleksandr Andrushchenko
2021-11-15 17:06 ` Jan Beulich
2021-11-16 9:38 ` Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 05/11] vpci/header: implement guest BAR register handlers Oleksandr Andrushchenko
2021-11-19 11:58 ` Jan Beulich
2021-11-19 12:10 ` Oleksandr Andrushchenko
2021-11-19 12:37 ` Jan Beulich
2021-11-19 12:46 ` Oleksandr Andrushchenko
2021-11-19 12:49 ` Jan Beulich
2021-11-19 12:54 ` Oleksandr Andrushchenko
2021-11-19 13:02 ` Jan Beulich
2021-11-19 13:17 ` Oleksandr Andrushchenko
2021-11-23 15:14 ` Oleksandr Andrushchenko
2021-11-24 12:32 ` Roger Pau Monné
2021-11-24 12:36 ` Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 06/11] vpci/header: handle p2m range sets per BAR Oleksandr Andrushchenko
2021-11-19 12:05 ` Jan Beulich
2021-11-19 12:13 ` Oleksandr Andrushchenko
2021-11-19 12:45 ` Jan Beulich
2021-11-19 12:50 ` Oleksandr Andrushchenko
2021-11-19 13:06 ` Jan Beulich
2021-11-19 13:19 ` Oleksandr Andrushchenko
2021-11-19 13:29 ` Jan Beulich
2021-11-19 13:38 ` Oleksandr Andrushchenko
2021-11-19 13:16 ` Jan Beulich
2021-11-19 13:41 ` Oleksandr Andrushchenko
2021-11-19 13:57 ` Jan Beulich
2021-11-19 14:09 ` Oleksandr Andrushchenko
2021-11-22 8:24 ` Jan Beulich
2021-11-22 8:31 ` Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 07/11] vpci/header: program p2m with guest BAR view Oleksandr Andrushchenko
2021-11-19 12:33 ` Jan Beulich
2021-11-19 12:44 ` Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 08/11] vpci/header: emulate PCI_COMMAND register for guests Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 09/11] vpci/header: reset the command register when adding devices Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 10/11] vpci: add initial support for virtual PCI bus topology Oleksandr Andrushchenko
2021-11-18 16:45 ` Jan Beulich
2021-11-24 11:28 ` Oleksandr Andrushchenko
2021-11-24 12:36 ` Roger Pau Monné
2021-11-24 12:43 ` Oleksandr Andrushchenko
2021-11-05 6:56 ` [PATCH v4 11/11] xen/arm: translate virtual PCI bus topology for guests Oleksandr Andrushchenko
2021-11-08 11:10 ` Jan Beulich
2021-11-08 11:16 ` Oleksandr Andrushchenko
2021-11-08 14:23 ` Roger Pau Monné
2021-11-08 15:28 ` Oleksandr Andrushchenko
2021-11-24 11:31 ` Oleksandr Andrushchenko
2021-11-19 13:56 ` [PATCH v4 00/11] PCI devices passthrough on Arm, part 3 Jan Beulich
2021-11-19 14:06 ` Oleksandr Andrushchenko
2021-11-19 14:23 ` Roger Pau Monné
2021-11-19 14:26 ` Oleksandr Andrushchenko
2021-11-20 9:47 ` Roger Pau Monné
2021-11-22 8:22 ` Jan Beulich
2021-11-22 8:34 ` Oleksandr Andrushchenko
2021-11-22 8:44 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cc7fb79b-5a43-3bdc-4621-097a01982f49@epam.com \
--to=oleksandr_andrushchenko@epam.com \
--cc=Artem_Mygaiev@epam.com \
--cc=Oleksandr_Tyshchenko@epam.com \
--cc=Volodymyr_Babchuk@epam.com \
--cc=andrew.cooper3@citrix.com \
--cc=bertrand.marquis@arm.com \
--cc=george.dunlap@citrix.com \
--cc=jbeulich@suse.com \
--cc=julien@xen.org \
--cc=paul@xen.org \
--cc=rahul.singh@arm.com \
--cc=roger.pau@citrix.com \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).