From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
ashok.raj@intel.com, keith.busch@intel.com,
Joerg Roedel <joro@8bytes.org>,
David Woodhouse <dwmw2@infradead.org>,
iommu@lists.linux-foundation.org
Subject: Re: [PATCH v5 4/7] PCI/ATS: Add PRI support for PCIe VF devices
Date: Mon, 19 Aug 2019 15:53:31 -0700 [thread overview]
Message-ID: <20190819225331.GB28404@skuppusw-desk.amr.corp.intel.com> (raw)
In-Reply-To: <20190819141500.GQ253360@google.com>
On Mon, Aug 19, 2019 at 09:15:00AM -0500, Bjorn Helgaas wrote:
> On Thu, Aug 15, 2019 at 03:39:03PM -0700, Kuppuswamy Sathyanarayanan wrote:
> > On 8/15/19 3:20 PM, Bjorn Helgaas wrote:
> > > [+cc Joerg, David, iommu list: because IOMMU drivers are the only
> > > callers of pci_enable_pri() and pci_enable_pasid()]
> > >
> > > On Thu, Aug 01, 2019 at 05:06:01PM -0700, sathyanarayanan.kuppuswamy@linux.intel.com wrote:
> > > > From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> > > >
> > > > When IOMMU tries to enable Page Request Interface (PRI) for VF device
> > > > in iommu_enable_dev_iotlb(), it always fails because PRI support for
> > > > PCIe VF device is currently broken. Current implementation expects
> > > > the given PCIe device (PF & VF) to implement PRI capability before
> > > > enabling the PRI support. But this assumption is incorrect. As per PCIe
> > > > spec r4.0, sec 9.3.7.11, all VFs associated with PF can only use the
> > > > PRI of the PF and not implement it. Hence we need to create exception
> > > > for handling the PRI support for PCIe VF device.
> > > >
> > > > Also, since PRI is a shared resource between PF/VF, following rules
> > > > should apply.
> > > >
> > > > 1. Use proper locking before accessing/modifying PF resources in VF
> > > > PRI enable/disable call.
> > > > 2. Use reference count logic to track the usage of PRI resource.
> > > > 3. Disable PRI only if the PRI reference count (pri_ref_cnt) is zero.
>
> > > Wait, why do we need this at all? I agree the spec says VFs may not
> > > implement PRI or PASID capabilities and that VFs use the PRI and
> > > PASID of the PF.
> > >
> > > But why do we need to support pci_enable_pri() and pci_enable_pasid()
> > > for VFs? There's nothing interesting we can *do* in the VF, and
> > > passing it off to the PF adds all this locking mess. For VFs, can we
> > > just make them do nothing or return -EINVAL? What functionality would
> > > we be missing if we did that?
> >
> > Currently PRI/PASID capabilities are not enabled by default. IOMMU can
> > enable PRI/PASID for VF first (and not enable it for PF). In this case,
> > doing nothing for VF device will break the functionality.
>
> What is the path where we can enable PRI/PASID for VF but not for the
> PF? The call chains leading to pci_enable_pri() go through the
> iommu_ops.add_device interface, which makes me think this is part of
> the device enumeration done by the PCI core, and in that case I would
> think this it should be done for the PF before VFs. But maybe this
> path isn't exercised until a driver does a DMA map or something
> similar?
AFAIK, this path will only get exercised when the device does DMA and
hence there is no specific order in which PRI/PASID is enabled in PF/VF.
In fact, my v2 version of this patch set had a check to ensure PF
PRI/PASID enable is happened before VF attempts PRI/PASID
enable/disable. But I had to remove it in later version of this series
due to failure case reported by one the tester of this code.
>
> Bjorn
--
--
Sathyanarayanan Kuppuswamy
Linux kernel developer
next prev parent reply other threads:[~2019-08-19 22:56 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-02 0:05 [PATCH v5 0/7] Fix PF/VF dependency issue sathyanarayanan.kuppuswamy
2019-08-02 0:05 ` [PATCH v5 1/7] PCI/ATS: Fix pci_prg_resp_pasid_required() dependency issues sathyanarayanan.kuppuswamy
2019-08-12 20:04 ` Bjorn Helgaas
2019-08-12 20:20 ` sathyanarayanan kuppuswamy
2019-08-13 3:51 ` Bjorn Helgaas
2019-08-16 18:06 ` Kuppuswamy Sathyanarayanan
2019-08-02 0:05 ` [PATCH v5 2/7] PCI/ATS: Initialize PRI in pci_ats_init() sathyanarayanan.kuppuswamy
2019-08-12 20:04 ` Bjorn Helgaas
2019-08-12 21:35 ` sathyanarayanan kuppuswamy
2019-08-13 4:10 ` Bjorn Helgaas
2019-08-15 4:46 ` Bjorn Helgaas
2019-08-15 17:30 ` Kuppuswamy Sathyanarayanan
2019-08-16 17:31 ` Bjorn Helgaas
2019-08-02 0:06 ` [PATCH v5 3/7] PCI/ATS: Initialize PASID " sathyanarayanan.kuppuswamy
2019-08-12 20:04 ` Bjorn Helgaas
2019-08-15 4:48 ` Bjorn Helgaas
2019-08-15 4:56 ` Bjorn Helgaas
2019-08-15 17:31 ` Kuppuswamy Sathyanarayanan
2019-08-02 0:06 ` [PATCH v5 4/7] PCI/ATS: Add PRI support for PCIe VF devices sathyanarayanan.kuppuswamy
2019-08-12 20:04 ` Bjorn Helgaas
2019-08-12 21:40 ` sathyanarayanan kuppuswamy
2019-08-13 4:16 ` Bjorn Helgaas
2019-08-15 22:20 ` Bjorn Helgaas
2019-08-15 22:39 ` Kuppuswamy Sathyanarayanan
2019-08-19 14:15 ` Bjorn Helgaas
2019-08-19 22:53 ` Kuppuswamy Sathyanarayanan [this message]
2019-08-19 23:19 ` Bjorn Helgaas
2019-08-28 18:21 ` Kuppuswamy Sathyanarayanan
2019-08-28 18:57 ` Bjorn Helgaas
2019-08-02 0:06 ` [PATCH v5 5/7] PCI/ATS: Add PASID " sathyanarayanan.kuppuswamy
2019-08-12 20:05 ` Bjorn Helgaas
2019-08-13 22:19 ` Kuppuswamy Sathyanarayanan
2019-08-15 5:04 ` Bjorn Helgaas
2019-08-16 1:21 ` Kuppuswamy Sathyanarayanan
2019-08-02 0:06 ` [PATCH v5 6/7] PCI/ATS: Disable PF/VF ATS service independently sathyanarayanan.kuppuswamy
2019-08-02 0:06 ` [PATCH v5 7/7] PCI: Skip Enhanced Allocation (EA) initialization for VF device sathyanarayanan.kuppuswamy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190819225331.GB28404@skuppusw-desk.amr.corp.intel.com \
--to=sathyanarayanan.kuppuswamy@linux.intel.com \
--cc=ashok.raj@intel.com \
--cc=dwmw2@infradead.org \
--cc=helgaas@kernel.org \
--cc=iommu@lists.linux-foundation.org \
--cc=joro@8bytes.org \
--cc=keith.busch@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).