Re: [PATCH v12 01/15] vpci: use per-domain PCI lock to protect vpci structure

From: Stewart Hildebrand <stewart.hildebrand@amd.com>
To: "Roger Pau Monné" <roger.pau@citrix.com>
Cc: <xen-devel@lists.xenproject.org>,
	Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>,
	Jan Beulich <jbeulich@suse.com>,
	"Andrew Cooper" <andrew.cooper3@citrix.com>, Wei Liu <wl@xen.org>,
	George Dunlap <george.dunlap@citrix.com>,
	Julien Grall <julien@xen.org>,
	Stefano Stabellini <sstabellini@kernel.org>,
	Jun Nakajima <jun.nakajima@intel.com>,
	Kevin Tian <kevin.tian@intel.com>, Paul Durrant <paul@xen.org>,
	Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Subject: Re: [PATCH v12 01/15] vpci: use per-domain PCI lock to protect vpci structure
Date: Mon, 15 Jan 2024 10:08:35 -0500	[thread overview]
Message-ID: <26b92079-4358-4fa8-97b3-b2e1ed7277e1@amd.com> (raw)
In-Reply-To: <ZaTyehqxp_q9wY3Y@macbook>

On 1/15/24 03:53, Roger Pau Monné wrote:
> On Fri, Jan 12, 2024 at 12:54:56PM -0500, Stewart Hildebrand wrote:
>> On 1/12/24 08:48, Roger Pau Monné wrote:
>>> On Tue, Jan 09, 2024 at 04:51:16PM -0500, Stewart Hildebrand wrote:
>>>> @@ -202,8 +204,20 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev,
>>>>      struct map_data data = { .d = d, .map = true };
>>>>      int rc;
>>>>  
>>>> +    ASSERT(rw_is_write_locked(&d->pci_lock));
>>>> +
>>>>      while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART )
>>>> +    {
>>>> +        /*
>>>> +         * It's safe to drop and reacquire the lock in this context
>>>> +         * without risking pdev disappearing because devices cannot be
>>>> +         * removed until the initial domain has been started.
>>>> +         */
>>>> +        write_unlock(&d->pci_lock);
>>>>          process_pending_softirqs();
>>>> +        write_lock(&d->pci_lock);
>>>
>>> Hm, I should have noticed before, but we already call
>>> process_pending_softirqs() with the pdev->vpci->lock held here, so it
>>> would make sense to drop it also.
>>
>> I don't quite understand this, maybe I'm missing something. I don't see where we acquire pdev->vpci->lock before calling process_pending_softirqs()?
>>
>> Also, I tried adding
>>
>>     ASSERT(!spin_is_locked(&pdev->vpci->lock));
>>
>> both here in apply_map() and in vpci_process_pending(), and they haven't triggered in either dom0 or domU test cases, tested on both arm and x86.
> 
> I think I was confused.  Are you sure that pdev->vpci->lock is taken
> in the apply_map() call?

I'm sure that it's NOT taken in apply_map(). See the ! in the test ASSERT above.

> I was mistakenly assuming that
> vpci_add_handlers() called the init function with the vpci->lock
> taken, but that doesn't seem to be case with the current code.  That
> leads to apply_map() also being called without the vpci->lock taken.

Right.

> 
> I was wrongly assuming that apply_map() was called with the vpci->lock
> lock taken, and that would need dropping around the
> process_pending_softirqs() call.
> 
> Thanks, Roger.