From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc: Lu Baolu <baolu.lu@linux.intel.com>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>,
LKML <linux-kernel@vger.kernel.org>,
Joerg Roedel <joro@8bytes.org>,
David Woodhouse <dwmw2@infradead.org>,
"Alex Williamson" <alex.williamson@redhat.com>,
Jean-Philippe Brucker <jean-philippe@linaro.com>,
"Liu, Yi L" <yi.l.liu@intel.com>,
"Raj, Ashok" <ashok.raj@intel.com>,
Christoph Hellwig <hch@infradead.org>,
"Jonathan Cameron" <jic23@kernel.org>,
Eric Auger <eric.auger@redhat.com>
Subject: RE: [PATCH V10 11/11] iommu/vt-d: Add custom allocator for IOASID
Date: Thu, 2 Apr 2020 02:18:45 +0000 [thread overview]
Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D19D805FAF@SHSMSX104.ccr.corp.intel.com> (raw)
In-Reply-To: <20200401084759.575b38c4@jacob-builder>
> From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Sent: Wednesday, April 1, 2020 11:48 PM
>
> On Sat, 28 Mar 2020 10:22:41 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
> > > From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > > Sent: Saturday, March 21, 2020 7:28 AM
> > >
> > > When VT-d driver runs in the guest, PASID allocation must be
> > > performed via virtual command interface. This patch registers a
> > > custom IOASID allocator which takes precedence over the default
> > > XArray based allocator. The resulting IOASID allocation will always
> > > come from the host. This ensures that PASID namespace is system-
> > > wide.
> > >
> > > Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> > > Signed-off-by: Liu, Yi L <yi.l.liu@intel.com>
> > > Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > > ---
> > > drivers/iommu/intel-iommu.c | 84
> > > +++++++++++++++++++++++++++++++++++++++++++++
> > > include/linux/intel-iommu.h | 2 ++
> > > 2 files changed, 86 insertions(+)
> > >
> > > diff --git a/drivers/iommu/intel-iommu.c
> > > b/drivers/iommu/intel-iommu.c index a76afb0fd51a..c1c0b0fb93c3
> > > 100644 --- a/drivers/iommu/intel-iommu.c
> > > +++ b/drivers/iommu/intel-iommu.c
> > > @@ -1757,6 +1757,9 @@ static void free_dmar_iommu(struct
> intel_iommu
> > > *iommu)
> > > if (ecap_prs(iommu->ecap))
> > > intel_svm_finish_prq(iommu);
> > > }
> > > + if (ecap_vcs(iommu->ecap) && vccap_pasid(iommu->vccap))
> > > +
> > > ioasid_unregister_allocator(&iommu->pasid_allocator); +
> > > #endif
> > > }
> > >
> > > @@ -3291,6 +3294,84 @@ static int copy_translation_tables(struct
> > > intel_iommu *iommu)
> > > return ret;
> > > }
> > >
> > > +#ifdef CONFIG_INTEL_IOMMU_SVM
> > > +static ioasid_t intel_ioasid_alloc(ioasid_t min, ioasid_t max,
> > > void *data)
> >
> > the name is too generic... can we add vcmd in the name to clarify
> > its purpose, e.g. intel_vcmd_ioasid_alloc?
> >
> I feel the intel_ prefix is a natural extension of a generic API, we do
> that for other IOMMU APIs, right?
other IOMMU APIs have no difference between host and guest, but
this one only applies to guest with vcmd interface.
>
> > > +{
> > > + struct intel_iommu *iommu = data;
> > > + ioasid_t ioasid;
> > > +
> > > + if (!iommu)
> > > + return INVALID_IOASID;
> > > + /*
> > > + * VT-d virtual command interface always uses the full 20
> > > bit
> > > + * PASID range. Host can partition guest PASID range based
> > > on
> > > + * policies but it is out of guest's control.
> > > + */
> > > + if (min < PASID_MIN || max > intel_pasid_max_id)
> > > + return INVALID_IOASID;
> > > +
> > > + if (vcmd_alloc_pasid(iommu, &ioasid))
> > > + return INVALID_IOASID;
> > > +
> > > + return ioasid;
> > > +}
> > > +
> > > +static void intel_ioasid_free(ioasid_t ioasid, void *data)
> > > +{
> > > + struct intel_iommu *iommu = data;
> > > +
> > > + if (!iommu)
> > > + return;
> > > + /*
> > > + * Sanity check the ioasid owner is done at upper layer,
> > > e.g. VFIO
> > > + * We can only free the PASID when all the devices are
> > > unbound.
> > > + */
> > > + if (ioasid_find(NULL, ioasid, NULL)) {
> > > + pr_alert("Cannot free active IOASID %d\n", ioasid);
> > > + return;
> > > + }
> >
> > However the sanity check is not done in default_free. Is there a
> > reason why using vcmd adds such new requirement?
> >
> Since we don't support nested guest. This vcmd allocator is only used
> by the guest IOMMU driver not VFIO. We expect IOMMU driver to have
> control of the free()/unbind() ordering.
>
> For default_free, it can come from user space and host VFIO which can
> be out of order. But we will solve that issue with the blocking
> notifier.
>
> > > + vcmd_free_pasid(iommu, ioasid);
> > > +}
> > > +
> > > +static void register_pasid_allocator(struct intel_iommu *iommu)
> > > +{
> > > + /*
> > > + * If we are running in the host, no need for custom
> > > allocator
> > > + * in that PASIDs are allocated from the host system-wide.
> > > + */
> > > + if (!cap_caching_mode(iommu->cap))
> > > + return;
> >
> > is it more accurate to check against vcmd capability?
> >
> I think this is sufficient. The spec says if vcmd is present, we must
> use it but not the other way.
No, what about an vIOMMU implementation reports CM but not
VCMD? I didn't get the rationale why we check an indirect capability
when there is already one well defined for the purpose.
>
> > > +
> > > + if (!sm_supported(iommu)) {
> > > + pr_warn("VT-d Scalable Mode not enabled, no PASID
> > > allocation\n");
> > > + return;
> > > + }
> > > +
> > > + /*
> > > + * Register a custom PASID allocator if we are running in
> > > a guest,
> > > + * guest PASID must be obtained via virtual command
> > > interface.
> > > + * There can be multiple vIOMMUs in each guest but only one
> > > allocator
> > > + * is active. All vIOMMU allocators will eventually be
> > > calling the same
> >
> > which one? the first or last?
> >
> All allocators share the same ops, so first=last. IOASID code will
> inspect the ops function and see if they are shared with others then
> use the same ops.
ok, got you.
>
> > > + * host allocator.
> > > + */
> > > + if (ecap_vcs(iommu->ecap) && vccap_pasid(iommu->vccap)) {
> > > + pr_info("Register custom PASID allocator\n");
> > > + iommu->pasid_allocator.alloc = intel_ioasid_alloc;
> > > + iommu->pasid_allocator.free = intel_ioasid_free;
> > > + iommu->pasid_allocator.pdata = (void *)iommu;
> > > + if
> > > (ioasid_register_allocator(&iommu->pasid_allocator)) {
> > > + pr_warn("Custom PASID allocator failed,
> > > scalable mode disabled\n");
> > > + /*
> > > + * Disable scalable mode on this IOMMU if
> > > there
> > > + * is no custom allocator. Mixing SM
> > > capable vIOMMU
> > > + * and non-SM vIOMMU are not supported.
> > > + */
> > > + intel_iommu_sm = 0;
> >
> > since you register an allocator for every vIOMMU, means previously
> > registered allocators should also be unregistered here?
> >
> True, but it is not necessary for two reasons:
> 1. This should not happen unless something went seriously wrong.
> All vIOMMU shares the same alloc/free function, so they are put under
> the same bucket by IOASID. So the case for the first vIOMMU to succeed
> then fail in later vIOMMU registration should not happen. Unless kernel
> run out of memory etc.
>
> 2. Once SM is disabled, there is no user of ioasid allocator.
>
> > > + }
> > > + }
> > > +}
> > > +#endif
> > > +
> > > static int __init init_dmars(void)
> > > {
> > > struct dmar_drhd_unit *drhd;
> > > @@ -3408,6 +3489,9 @@ static int __init init_dmars(void)
> > > */
> > > for_each_active_iommu(iommu, drhd) {
> > > iommu_flush_write_buffer(iommu);
> > > +#ifdef CONFIG_INTEL_IOMMU_SVM
> > > + register_pasid_allocator(iommu);
> > > +#endif
> > > iommu_set_root_entry(iommu);
> > > iommu->flush.flush_context(iommu, 0, 0, 0,
> > > DMA_CCMD_GLOBAL_INVL);
> > > iommu->flush.flush_iotlb(iommu, 0, 0, 0,
> > > DMA_TLB_GLOBAL_FLUSH);
> > > diff --git a/include/linux/intel-iommu.h
> > > b/include/linux/intel-iommu.h index 9cbf5357138b..9c357a325c72
> > > 100644 --- a/include/linux/intel-iommu.h
> > > +++ b/include/linux/intel-iommu.h
> > > @@ -19,6 +19,7 @@
> > > #include <linux/iommu.h>
> > > #include <linux/io-64-nonatomic-lo-hi.h>
> > > #include <linux/dmar.h>
> > > +#include <linux/ioasid.h>
> > >
> > > #include <asm/cacheflush.h>
> > > #include <asm/iommu.h>
> > > @@ -563,6 +564,7 @@ struct intel_iommu {
> > > #ifdef CONFIG_INTEL_IOMMU_SVM
> > > struct page_req_dsc *prq;
> > > unsigned char prq_name[16]; /* Name for PRQ interrupt */
> > > + struct ioasid_allocator_ops pasid_allocator; /* Custom
> > > allocator for PASIDs */
> > > #endif
> > > struct q_inval *qi; /* Queued invalidation
> > > info */ u32 *iommu_state; /* Store iommu states between suspend and
> > > resume.*/
> > > --
> > > 2.7.4
> >
>
> [Jacob Pan]
next prev parent reply other threads:[~2020-04-02 2:18 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-03-20 23:27 [PATCH V10 00/11] Nested Shared Virtual Address (SVA) VT-d support Jacob Pan
2020-03-20 23:27 ` [PATCH V10 01/11] iommu/vt-d: Move domain helper to header Jacob Pan
2020-03-27 11:48 ` Tian, Kevin
2020-03-20 23:27 ` [PATCH V10 02/11] iommu/uapi: Define a mask for bind data Jacob Pan
2020-03-22 1:29 ` Lu Baolu
2020-03-23 19:37 ` Jacob Pan
2020-03-24 1:50 ` Lu Baolu
2020-03-27 11:50 ` Tian, Kevin
2020-03-27 14:13 ` Auger Eric
2020-03-20 23:27 ` [PATCH V10 03/11] iommu/vt-d: Add a helper function to skip agaw Jacob Pan
2020-03-27 11:53 ` Tian, Kevin
2020-03-29 7:20 ` Lu Baolu
2020-03-30 17:50 ` Jacob Pan
2020-03-20 23:27 ` [PATCH V10 04/11] iommu/vt-d: Use helper function to skip agaw for SL Jacob Pan
2020-03-27 11:55 ` Tian, Kevin
2020-03-27 16:05 ` Auger Eric
2020-03-29 7:35 ` Lu Baolu
2020-03-20 23:27 ` [PATCH V10 05/11] iommu/vt-d: Add nested translation helper function Jacob Pan
2020-03-27 12:21 ` Tian, Kevin
2020-03-29 8:03 ` Lu Baolu
2020-03-30 18:21 ` Jacob Pan
2020-03-31 3:36 ` Tian, Kevin
2020-03-29 11:35 ` Auger Eric
2020-04-01 20:06 ` Jacob Pan
2020-03-20 23:27 ` [PATCH V10 06/11] iommu/vt-d: Add bind guest PASID support Jacob Pan
2020-03-28 8:02 ` Tian, Kevin
2020-03-30 20:51 ` Jacob Pan
2020-03-31 3:43 ` Tian, Kevin
2020-04-01 17:13 ` Jacob Pan
2020-03-29 13:40 ` Auger Eric
2020-03-30 22:53 ` Jacob Pan
2020-03-20 23:27 ` [PATCH V10 07/11] iommu/vt-d: Support flushing more translation cache types Jacob Pan
2020-03-27 14:46 ` Auger Eric
2020-03-30 23:28 ` Jacob Pan
2020-03-31 16:13 ` Jacob Pan
2020-03-31 16:15 ` Auger Eric
2020-03-20 23:27 ` [PATCH V10 08/11] iommu/vt-d: Add svm/sva invalidate function Jacob Pan
2020-03-28 10:01 ` Tian, Kevin
2020-03-29 15:34 ` Auger Eric
2020-03-31 2:49 ` Tian, Kevin
2020-03-31 20:58 ` Jacob Pan
2020-04-01 6:29 ` Tian, Kevin
2020-04-01 7:13 ` Liu, Yi L
2020-04-01 7:32 ` Auger Eric
2020-04-01 16:05 ` Jacob Pan
2020-04-02 15:54 ` Jacob Pan
2020-03-29 16:05 ` Auger Eric
2020-03-31 3:34 ` Tian, Kevin
2020-03-31 21:07 ` Jacob Pan
2020-04-01 6:32 ` Tian, Kevin
2020-03-31 18:13 ` Jacob Pan
2020-04-01 6:24 ` Tian, Kevin
2020-04-01 6:57 ` Liu, Yi L
2020-04-01 16:03 ` Jacob Pan
2020-03-29 16:05 ` Auger Eric
2020-03-31 22:28 ` Jacob Pan
2020-03-20 23:27 ` [PATCH V10 09/11] iommu/vt-d: Cache virtual command capability register Jacob Pan
2020-03-28 10:04 ` Tian, Kevin
2020-03-31 22:33 ` Jacob Pan
2020-03-20 23:27 ` [PATCH V10 10/11] iommu/vt-d: Enlightened PASID allocation Jacob Pan
2020-03-28 10:08 ` Tian, Kevin
2020-03-31 22:37 ` Jacob Pan
2020-03-20 23:27 ` [PATCH V10 11/11] iommu/vt-d: Add custom allocator for IOASID Jacob Pan
2020-03-28 10:22 ` Tian, Kevin
2020-04-01 15:47 ` Jacob Pan
2020-04-02 2:18 ` Tian, Kevin [this message]
2020-04-02 20:28 ` Jacob Pan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=AADFC41AFE54684AB9EE6CBC0274A5D19D805FAF@SHSMSX104.ccr.corp.intel.com \
--to=kevin.tian@intel.com \
--cc=alex.williamson@redhat.com \
--cc=ashok.raj@intel.com \
--cc=baolu.lu@linux.intel.com \
--cc=dwmw2@infradead.org \
--cc=eric.auger@redhat.com \
--cc=hch@infradead.org \
--cc=iommu@lists.linux-foundation.org \
--cc=jacob.jun.pan@linux.intel.com \
--cc=jean-philippe@linaro.com \
--cc=jic23@kernel.org \
--cc=joro@8bytes.org \
--cc=linux-kernel@vger.kernel.org \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).