IOMMU Archive on lore.kernel.org
 help / color / Atom feed
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc: "Raj, Ashok" <ashok.raj@intel.com>,
	Jean-Philippe Brucker <jean-philippe@linaro.com>,
	LKML <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux-foundation.org"
	<iommu@lists.linux-foundation.org>,
	Alex Williamson <alex.williamson@redhat.com>,
	David Woodhouse <dwmw2@infradead.org>,
	Jonathan Cameron <jic23@kernel.org>
Subject: RE: [PATCH 03/10] iommu/ioasid: Introduce per set allocation APIs
Date: Sat, 28 Mar 2020 06:32:29 +0000
Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D19D7F464D@SHSMSX104.ccr.corp.intel.com> (raw)
In-Reply-To: <20200327095923.4454cc7f@jacob-builder>

> From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Sent: Saturday, March 28, 2020 12:59 AM
> 
> On Fri, 27 Mar 2020 08:38:44 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > > Sent: Thursday, March 26, 2020 1:55 AM
> > >
> > > IOASID set defines a group of IDs that share the same token. The
> > > ioasid_set concept helps to do permission checking among users as
> > > in the current code.
> > >
> > > With guest SVA usage, each VM has its own IOASID set. More
> > > functionalities are needed:
> > > 1. Enforce quota, each guest may be assigned limited quota such
> > > that one guest cannot abuse all the system resource.
> > > 2. Stores IOASID mapping between guest and host IOASIDs
> > > 3. Per set operations, e.g. free the entire set
> > >
> > > For each ioasid_set token, a unique set ID is assigned. This makes
> > > reference of the set and data lookup much easier to implement.
> > >
> > > Signed-off-by: Liu Yi L <yi.l.liu@intel.com>
> > > Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > > ---
> > >  drivers/iommu/ioasid.c | 147
> > > +++++++++++++++++++++++++++++++++++++++++++++++++
> > >  include/linux/ioasid.h |  13 +++++
> > >  2 files changed, 160 insertions(+)
> > >
> > > diff --git a/drivers/iommu/ioasid.c b/drivers/iommu/ioasid.c
> > > index 4026e52855b9..27ee57f7079b 100644
> > > --- a/drivers/iommu/ioasid.c
> > > +++ b/drivers/iommu/ioasid.c
> > > @@ -10,6 +10,25 @@
> > >  #include <linux/spinlock.h>
> > >  #include <linux/xarray.h>
> > >
> > > +static DEFINE_XARRAY_ALLOC(ioasid_sets);
> > > +/**
> > > + * struct ioasid_set_data - Meta data about ioasid_set
> > > + *
> > > + * @token:	Unique to identify an IOASID set
> > > + * @xa:		XArray to store subset ID and IOASID
> > > mapping
> >
> > what is a subset? is it a different thing from set?
> >
> Subset is a set, but a subset ID is an ID only valid within the set.
> When we have non-identity Guest-Host PASID mapping, Subset ID is
> the Guest PASID but in more general terms. Or call it "Set Private ID"
> 
> This can be confusing, perhaps I rephrase it as:
> "XArray to store ioasid_set private ID to system-wide IOASID mapping"
> 
> 
> > > + * @size:	Max number of IOASIDs can be allocated within the
> > > set
> >
> > 'size' reads more like 'current size' instead of 'max size'. maybe
> > call it 'max_ioasids' to align with 'nr_ioasids'? or simplify both as
> > 'max' and 'nr'?
> >
> Right, how about max_id and nr_id?

sounds good.

> 
> > > + * @nr_ioasids	Number of IOASIDs allocated in the set
> > > + * @sid		ID of the set
> > > + */
> > > +struct ioasid_set_data {
> > > +	struct ioasid_set *token;
> > > +	struct xarray xa;
> > > +	int size;
> > > +	int nr_ioasids;
> > > +	int sid;
> > > +	struct rcu_head rcu;
> > > +};
> > > +
> > >  struct ioasid_data {
> > >  	ioasid_t id;
> > >  	struct ioasid_set *set;
> > > @@ -388,6 +407,111 @@ void ioasid_free(ioasid_t ioasid)
> > >  EXPORT_SYMBOL_GPL(ioasid_free);
> > >
> > >  /**
> > > + * ioasid_alloc_set - Allocate a set of IOASIDs
> >
> > 'a set of IOASIDS' sounds like 'many IOASIDs'. Just saying 'allocate
> > an IOASID set' is more clear. 😊
> >
> Make sense
> 
> > > + * @token:	Unique token of the IOASID set
> > > + * @quota:	Quota allowed in this set
> > > + * @sid:	IOASID set ID to be assigned
> > > + *
> > > + * Return 0 upon success. Token will be stored internally for
> > > lookup,
> > > + * IOASID allocation within the set and other per set operations
> > > will use
> > > + * the @sid assigned.
> > > + *
> > > + */
> > > +int ioasid_alloc_set(struct ioasid_set *token, ioasid_t quota, int
> > > *sid) +{
> > > +	struct ioasid_set_data *sdata;
> > > +	ioasid_t id;
> > > +	int ret = 0;
> > > +
> > > +	if (quota > ioasid_capacity_avail) {
> > > +		pr_warn("Out of IOASID capacity! ask %d, avail
> > > %d\n",
> > > +			quota, ioasid_capacity_avail);
> > > +		return -ENOSPC;
> > > +	}
> > > +
> > > +	sdata = kzalloc(sizeof(*sdata), GFP_KERNEL);
> > > +	if (!sdata)
> > > +		return -ENOMEM;
> > > +
> > > +	spin_lock(&ioasid_allocator_lock);
> > > +
> > > +	ret = xa_alloc(&ioasid_sets, &id, sdata,
> > > +		       XA_LIMIT(0, ioasid_capacity_avail - quota),
> > > +		       GFP_KERNEL);
> >
> > Interestingly I didn't find the definition of ioasid_sets. and it is
> > not in existing file.
> >
> It is at the beginning of this file
> +static DEFINE_XARRAY_ALLOC(ioasid_sets);

How did I overlook it after several checks... 😊

> 
> > I'm not sure how many sets can be created, but anyway the set
> > namespace is different from ioasid name space. Then why do we
> > use ioasid capability as the limitation for allocating set id here?
> >
> I am assuming the worst case scenario which is one IOASID per set, that
> is why the number of sets are limited by the number of system IOASIDs.

I feel using a static max is simpler and clearer here. Anyway the set id
is never used on hardware so it is not necessary to tie it with dynamic
IOAPIC numbers. 

> 
> > > +	if (ret) {
> > > +		kfree(sdata);
> > > +		goto error;
> > > +	}
> > > +
> > > +	sdata->token = token;
> >
> > given token must be unique, a check on any conflict is required here?
> >
> Right, I will add a check to reject duplicated tokens.
> 
> 	/* Search existing set tokens, reject duplicates */
> 	xa_for_each(&ioasid_sets, index, sdata) {
> 		if (sdata->token == token) {
> 			pr_warn("Token already exists in the set %lu\n",
> index);
> 			ret = -EEXIST;
> 			goto error;
> 		}
> 	}
> 
> 
> 
> 
> > > +	sdata->size = quota;
> > > +	sdata->sid = id;
> > > +
> > > +	/*
> > > +	 * Set Xarray is used to store IDs within the set, get
> > > ready for
> > > +	 * sub-set ID and system-wide IOASID allocation results.
> >
> > looks 'subset' is the same thing as 'set'. let's make it consistent.
> >
> Sounds good, will also rename subset ID to set private ID.
> 
> > > +	 */
> > > +	xa_init_flags(&sdata->xa, XA_FLAGS_ALLOC);
> > > +
> > > +	ioasid_capacity_avail -= quota;
> > > +	*sid = id;
> > > +
> > > +error:
> > > +	spin_unlock(&ioasid_allocator_lock);
> > > +
> > > +	return ret;
> > > +}
> > > +EXPORT_SYMBOL_GPL(ioasid_alloc_set);
> > > +
> > > +/**
> > > + * ioasid_free_set - Free all IOASIDs within the set
> > > + *
> > > + * @sid:		The IOASID set ID to be freed
> > > + * @destroy_set:	Whether to keep the set for further
> > > allocation.
> > > + *			If true, the set will be destroyed.
> > > + *
> > > + * All IOASIDs allocated within the set will be freed upon return.
> > > + */
> > > +void ioasid_free_set(int sid, bool destroy_set)
> > > +{
> >
> > what is the actual usage of just freeing ioasid while keeping the
> > set itself?
> >
> I was thinking users use mm as token can retain the ioasid_set until
> mm being destroyed. This is to support some kind of lazy free.
> 
> > > +	struct ioasid_set_data *sdata;
> > > +	struct ioasid_data *entry;
> > > +	unsigned long index;
> > > +
> > > +	spin_lock(&ioasid_allocator_lock);
> > > +	sdata = xa_load(&ioasid_sets, sid);
> > > +	if (!sdata) {
> > > +		pr_err("No IOASID set found to free %d\n", sid);
> > > +		goto done_unlock;
> > > +	}
> > > +
> > > +	if (xa_empty(&sdata->xa)) {
> > > +		pr_warn("No IOASIDs in the set %d\n", sdata->sid);
> > > +		goto done_destroy;
> > > +	}
> >
> > why is it a warning condition? it is possible that an user has done
> > ioasid_free for all allocated ioasids and then call this function,
> > which is actually the expected normal situation.
> >
> You are right, there is no need to warn. I will put the following
> comment in place.
> 	/* The set is already empty, we just destroy the set if requested */
> 	if (xa_empty(&sdata->xa))
> 		goto done_destroy;
> 
> > > +
> > > +	/* Just a place holder for now */
> > > +	xa_for_each(&sdata->xa, index, entry) {
> > > +		/* Free from per sub-set pool */
> > > +		xa_erase(&sdata->xa, index);
> > > +	}
> >
> > but the placeholder would lead to undesired behavior, not good for
> > bisect. If no support now, then should return an error if any in-use
> > ioasid is not freed.
> >
> Good point, I will return -ENOTSUPP in the place holder. Remove it
> during the API conversion.
> 
> > > +
> > > +done_destroy:
> > > +	if (destroy_set) {
> > > +		xa_erase(&ioasid_sets, sid);
> > > +
> > > +		/* Return the quota back to system pool */
> > > +		ioasid_capacity_avail += sdata->size;
> > > +		kfree_rcu(sdata, rcu);
> > > +	}
> > > +
> > > +done_unlock:
> > > +	spin_unlock(&ioasid_allocator_lock);
> > > +}
> > > +EXPORT_SYMBOL_GPL(ioasid_free_set);
> > > +
> > > +
> > > +/**
> > >   * ioasid_find - Find IOASID data
> > >   * @set: the IOASID set
> > >   * @ioasid: the IOASID to find
> > > @@ -431,6 +555,29 @@ void *ioasid_find(struct ioasid_set *set,
> > > ioasid_t ioasid,
> > >  }
> > >  EXPORT_SYMBOL_GPL(ioasid_find);
> > >
> > > +/**
> > > + * ioasid_find_sid - Retrieve IOASID set ID from an ioasid
> > > + *                   Caller must hold a reference to the set.
> >
> > please unify capitalization around IOASID or ioasid.
> >
> Will do.
> 
> > Thanks
> > Kevin
> >
> > > + *
> > > + * @ioasid: IOASID associated with the set
> > > + *
> > > + * Return IOASID set ID or error
> > > + */
> > > +int ioasid_find_sid(ioasid_t ioasid)
> > > +{
> > > +	struct ioasid_data *ioasid_data;
> > > +	int ret = 0;
> > > +
> > > +	spin_lock(&ioasid_allocator_lock);
> > > +	ioasid_data = xa_load(&active_allocator->xa, ioasid);
> > > +	ret = (ioasid_data) ? ioasid_data->sdata->sid : -ENOENT;
> > > +
> > > +	spin_unlock(&ioasid_allocator_lock);
> > > +
> > > +	return ret;
> > > +}
> > > +EXPORT_SYMBOL_GPL(ioasid_find_sid);
> > > +
> > >  MODULE_AUTHOR("Jean-Philippe Brucker <jean-
> > > philippe.brucker@arm.com>");
> > >  MODULE_AUTHOR("Jacob Pan <jacob.jun.pan@linux.intel.com>");
> > >  MODULE_DESCRIPTION("IO Address Space ID (IOASID) allocator");
> > > diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h
> > > index 9711fa0dc357..be158e03c034 100644
> > > --- a/include/linux/ioasid.h
> > > +++ b/include/linux/ioasid.h
> > > @@ -41,6 +41,9 @@ int ioasid_register_allocator(struct
> > > ioasid_allocator_ops *allocator);
> > >  void ioasid_unregister_allocator(struct ioasid_allocator_ops
> > > *allocator); int ioasid_set_data(ioasid_t ioasid, void *data);
> > >  void ioasid_install_capacity(ioasid_t total);
> > > +int ioasid_alloc_set(struct ioasid_set *token, ioasid_t quota, int
> > > *sid); +void ioasid_free_set(int sid, bool destroy_set);
> > > +int ioasid_find_sid(ioasid_t ioasid);
> > >  #else /* !CONFIG_IOASID */
> > >  static inline ioasid_t ioasid_alloc(struct ioasid_set *set,
> > > ioasid_t min, ioasid_t max, void *private)
> > > @@ -52,6 +55,15 @@ static inline void ioasid_free(ioasid_t ioasid)
> > >  {
> > >  }
> > >
> > > +static inline int ioasid_alloc_set(struct ioasid_set *token,
> > > ioasid_t quota, int *sid)
> > > +{
> > > +	return -ENOTSUPP;
> > > +}
> > > +
> > > +static inline void ioasid_free_set(int sid, bool destroy_set)
> > > +{
> > > +}
> > > +
> > >  static inline void *ioasid_find(struct ioasid_set *set, ioasid_t
> > > ioasid, bool (*getter)(void *))
> > >  {
> > > @@ -75,5 +87,6 @@ static inline int ioasid_set_data(ioasid_t
> > > ioasid, void *data)
> > >  static inline void ioasid_install_capacity(ioasid_t total)
> > >  {
> > >  }
> > > +
> > >  #endif /* CONFIG_IOASID */
> > >  #endif /* __LINUX_IOASID_H */
> > > --
> > > 2.7.4
> >
> 
> [Jacob Pan]
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

  reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-25 17:55 [PATCH 00/10] IOASID extensions for guest SVA Jacob Pan
2020-03-25 17:55 ` [PATCH 01/10] iommu/ioasid: Introduce system-wide capacity Jacob Pan
2020-03-27  8:07   ` Tian, Kevin
2020-03-27 16:08     ` Jacob Pan
2020-04-01 13:45   ` Jean-Philippe Brucker
2020-04-01 22:50     ` Jacob Pan
2020-03-25 17:55 ` [PATCH 02/10] iommu/vt-d: Set IOASID capacity when SVM is enabled Jacob Pan
2020-03-27  8:08   ` Tian, Kevin
2020-03-25 17:55 ` [PATCH 03/10] iommu/ioasid: Introduce per set allocation APIs Jacob Pan
2020-03-26  2:12   ` Lu Baolu
2020-03-26 21:30     ` Jacob Pan
2020-03-27  8:38   ` Tian, Kevin
2020-03-27 16:59     ` Jacob Pan
2020-03-28  6:32       ` Tian, Kevin [this message]
2020-04-01 13:47   ` Jean-Philippe Brucker
2020-04-06 20:02     ` Jacob Pan
2020-04-07 11:01       ` Jean-Philippe Brucker
2020-04-21 21:51         ` Jacob Pan
2020-03-25 17:55 ` [PATCH 04/10] iommu/ioasid: Rename ioasid_set_data to avoid confusion with ioasid_set Jacob Pan
2020-03-27  9:35   ` Tian, Kevin
2020-03-25 17:55 ` [PATCH 05/10] iommu/ioasid: Create an IOASID set for host SVA use Jacob Pan
2020-03-27  9:41   ` Tian, Kevin
2020-03-27 17:28     ` Jacob Pan
2020-03-28  6:33       ` Tian, Kevin
2020-04-01 13:53   ` Jean-Philippe Brucker
2020-04-06 15:33     ` Jacob Pan
2020-04-07 11:01       ` Jean-Philippe Brucker
2020-04-13 22:06         ` Jacob Pan
2020-04-15 15:10           ` Jean-Philippe Brucker
2020-03-25 17:55 ` [PATCH 06/10] iommu/ioasid: Convert to set aware allocations Jacob Pan
2020-03-27  9:54   ` Tian, Kevin
2020-03-27 17:41     ` Jacob Pan
2020-03-28  6:40       ` Tian, Kevin
2020-04-06 20:07         ` Jacob Pan
2020-04-01 13:55   ` Jean-Philippe Brucker
2020-04-01 22:45     ` Jacob Pan
2020-03-25 17:55 ` [PATCH 07/10] iommu/ioasid: Use mutex instead of spinlock Jacob Pan
2020-03-27  9:55   ` Tian, Kevin
2020-04-01 13:58   ` Jean-Philippe Brucker
2020-03-25 17:55 ` [PATCH 08/10] iommu/ioasid: Introduce notifier APIs Jacob Pan
2020-03-27 10:03   ` Tian, Kevin
2020-03-27 18:36     ` Jacob Pan
2020-03-28  6:43       ` Tian, Kevin
2020-03-31 15:13         ` Jacob Pan
2020-04-01 14:00   ` Jean-Philippe Brucker
2020-04-10 15:43     ` Jacob Pan
2020-03-25 17:55 ` [PATCH 09/10] iommu/ioasid: Support ioasid_set quota adjustment Jacob Pan
2020-03-27 10:09   ` Tian, Kevin
2020-03-27 23:30     ` Jacob Pan
2020-03-28  6:44       ` Tian, Kevin
2020-03-25 17:55 ` [PATCH 10/10] iommu/vt-d: Register PASID notifier for status change Jacob Pan
2020-03-27 10:22   ` Tian, Kevin
2020-03-27 23:47     ` Jacob Pan
2020-04-01 14:03 ` [PATCH 00/10] IOASID extensions for guest SVA Jean-Philippe Brucker
2020-04-01 23:38   ` Jacob Pan
2020-04-02 12:26     ` Jean-Philippe Brucker
2020-04-02 16:09       ` Jacob Pan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AADFC41AFE54684AB9EE6CBC0274A5D19D7F464D@SHSMSX104.ccr.corp.intel.com \
    --to=kevin.tian@intel.com \
    --cc=alex.williamson@redhat.com \
    --cc=ashok.raj@intel.com \
    --cc=dwmw2@infradead.org \
    --cc=iommu@lists.linux-foundation.org \
    --cc=jacob.jun.pan@linux.intel.com \
    --cc=jean-philippe@linaro.com \
    --cc=jic23@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

IOMMU Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-iommu/0 linux-iommu/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-iommu linux-iommu/ https://lore.kernel.org/linux-iommu \
		iommu@lists.linux-foundation.org
	public-inbox-index linux-iommu

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.linux-foundation.lists.iommu


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git