From: Ethan Zhao <haifeng.zhao@linux.intel.com>
To: "Tian, Kevin" <kevin.tian@intel.com>,
"Liu, Yi L" <yi.l.liu@intel.com>,
"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
"bhelgaas@google.com" <bhelgaas@google.com>,
"robin.murphy@arm.com" <robin.murphy@arm.com>,
"jgg@ziepe.ca" <jgg@ziepe.ca>
Cc: "dwmw2@infradead.org" <dwmw2@infradead.org>,
"will@kernel.org" <will@kernel.org>,
"lukas@wunner.de" <lukas@wunner.de>,
"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>
Subject: Re: [PATCH v12 5/5] iommu/vt-d: improve ITE fault handling if target device isn't present
Date: Tue, 30 Jan 2024 17:13:29 +0800 [thread overview]
Message-ID: <56a9971e-7015-4584-89c7-80056b7ec547@linux.intel.com> (raw)
In-Reply-To: <BN9PR11MB527674172BBA9BDC49A004D08C7D2@BN9PR11MB5276.namprd11.prod.outlook.com>
On 1/30/2024 4:43 PM, Tian, Kevin wrote:
>> From: Ethan Zhao <haifeng.zhao@linux.intel.com>
>> Sent: Tuesday, January 30, 2024 4:16 PM
>>
>> On 1/30/2024 2:22 PM, Tian, Kevin wrote:
>>> Here we need consider two situations.
>>>
>>> One is that the device is not bound to a driver or bound to a driver
>>> which doesn't do active work to the device when it's removed. In
>>> that case one may observe the timeout situation only in the removal
>>> path as the stack dump in your patch02 shows.
>> When iommu_bus_notifier() got called for hotplug removal cases to
>> flush devTLB (ATS invalidation), driver was already unloaded.
>> whatever safe removal or surprise removal. so in theory no active
>> driver working there.
>>
>> pciehp_ist()
>> pciehp_disable_slot()
>> remove_board()
>> pciehp_unconfigure_device()
>> pci_stop_and_remove_bus_device()
>> pci_stop_bus_device()--->here unload driver
>> pci_remove_bus_device()->here qi_flush_dev_iotlb() got called.
> yes, so patch02 can fix this case.
>
>>> patch02 can fix that case by checking whether the device is present
>>> to skip sending the invalidation requests. So the logic being discussed
>>> here doesn't matter.
>>>
>>> The 2nd situation is more tricky. The device might be bound to
>>> a driver which is doing active work to the device with in-fly
>>> ATS invalidation requests. In this case in-fly requests must be aborted
>>> before the driver can be detached from the removed device. Conceptually
>>> a device is removed from the bus only after its driver is detached.
>> Some tricky situations:
>>
>> 1. The ATS invalidation request is issued from driver driver, while it is
>> in handling, device is removed. this momment, the device instance still
>> exists in the bus list. yes, if searching it by BDF, could get it.
> it's searchable between the point where the device is removed and the
> point where the driver is unloaded:
>
> CPU0 CPU1
> (Driver is active) (pciehp handler)
> qi_submit_sync() pciehp_ist()
> ... ...
> loop for completion() { pciehp_unconfigure_device()
> ... pci_dev_set_disconnected()
> if (ITE) { ...
> //find pci_dev from sid pci_remove_bus_device()
> if (pci_dev_is_connected()) device_del()
> break; bus_remove_device()
> } device_remove_driver()
If the device was hot plugin or re-scanned, the device has a PCI_DEV_ADDED flag,
if so the driver unloading work isn't defered to the tail of device_del(), it
is unloaded before pci_remove_bus_device()->device_del(), in pci_stop_dev
pci_stop_bus_device()
pci_stop_dev()
{
if (pci_dev_is_added(dev)) {
device_release_driver(&dev->dev);
}
So the interval the device is searchable, only applied to those devices
not hot plugged, or never be scanned.
Thanks,
Ethan
> .. //wait for driver unload
> }
> ..
> return;
>
> BUS_NOTIFY_REMOVED_DEVICE;
> list_del(&dev->bus_list);
>
> (I didn’t draw the full calling stack on the right hand side)
>
>> 2. The ATS invalidation request is issued from iommu_bus_notifier()
>> for surprise removal reason, as shown in above calltrace, device was
>> already removed from bus list. if searching it by BDF, return NULL.
>>
>> 3. The ATS invlidation request is issued from iommu_bus_notifier()
>> for safe removal, when is in handling, device is removed or link
>> is down. also as #2, device was already removed from bus list.
>> if searching it by BDF. got NULL.
>> ...
>>
>> so, searching device by BDF, only works for the ATS invalidation
>> request is from device driver.
>>
> anything related to bus notifier has been fixed by patch02.
>
> the remaining logic is really for fixing the race invalidation from
> device driver.
next prev parent reply other threads:[~2024-01-30 9:13 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-29 3:49 [PATCH v12 0/5] fix vt-d hard lockup when hotplug ATS capable device Ethan Zhao
2024-01-29 3:49 ` [PATCH v12 1/5] PCI: make pci_dev_is_disconnected() helper public for other drivers Ethan Zhao
2024-01-29 8:50 ` Tian, Kevin
2024-01-30 5:23 ` Ethan Zhao
2024-01-30 5:25 ` Ethan Zhao
2024-01-30 6:23 ` Tian, Kevin
2024-01-29 3:49 ` [PATCH v12 2/5] iommu/vt-d: don't issue ATS Invalidation request when device is disconnected Ethan Zhao
2024-01-29 8:53 ` Tian, Kevin
2024-01-29 9:32 ` Yi Liu
2024-01-30 5:37 ` Ethan Zhao
2024-01-31 4:25 ` Yi Liu
2024-01-31 5:25 ` Ethan Zhao
2024-01-29 3:49 ` [PATCH v12 3/5] iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation callers Ethan Zhao
2024-01-29 9:37 ` Yi Liu
2024-01-30 5:43 ` Ethan Zhao
2024-01-29 3:49 ` [PATCH v12 4/5] iommu/vt-d: pass pdev parameter for qi_check_fault() and refactor callers Ethan Zhao
2024-01-29 8:58 ` Tian, Kevin
2024-01-30 7:30 ` Ethan Zhao
2024-02-08 7:15 ` Dan Carpenter
2024-02-09 2:08 ` Ethan Zhao
2024-01-29 3:49 ` [PATCH v12 5/5] iommu/vt-d: improve ITE fault handling if target device isn't present Ethan Zhao
2024-01-29 9:06 ` Tian, Kevin
2024-01-29 9:21 ` Yi Liu
2024-01-30 5:12 ` Ethan Zhao
2024-01-30 6:22 ` Tian, Kevin
2024-01-30 8:15 ` Ethan Zhao
2024-01-30 8:43 ` Tian, Kevin
2024-01-30 9:13 ` Ethan Zhao [this message]
2024-01-30 9:24 ` Tian, Kevin
2024-01-31 5:42 ` Ethan Zhao
2024-01-30 16:29 ` Jason Gunthorpe
2024-01-31 6:21 ` Baolu Lu
2024-02-01 19:34 ` Jason Gunthorpe
2024-02-15 7:37 ` Baolu Lu
2024-01-29 14:48 ` Baolu Lu
2024-01-30 3:28 ` Tian, Kevin
2024-01-30 8:43 ` Ethan Zhao
2024-01-29 9:33 ` Yi Liu
2024-01-29 5:16 ` [PATCH v12 0/5] fix vt-d hard lockup when hotplug ATS capable device Ethan Zhao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56a9971e-7015-4584-89c7-80056b7ec547@linux.intel.com \
--to=haifeng.zhao@linux.intel.com \
--cc=baolu.lu@linux.intel.com \
--cc=bhelgaas@google.com \
--cc=dwmw2@infradead.org \
--cc=iommu@lists.linux.dev \
--cc=jgg@ziepe.ca \
--cc=kevin.tian@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=lukas@wunner.de \
--cc=robin.murphy@arm.com \
--cc=will@kernel.org \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).