IOMMU Archive on lore.kernel.org
 help / color / Atom feed
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc: "Raj, Ashok" <ashok.raj@intel.com>,
	Jean-Philippe Brucker <jean-philippe@linaro.com>,
	LKML <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux-foundation.org"
	<iommu@lists.linux-foundation.org>,
	Alex Williamson <alex.williamson@redhat.com>,
	David Woodhouse <dwmw2@infradead.org>,
	Jonathan Cameron <jic23@kernel.org>
Subject: RE: [PATCH 08/10] iommu/ioasid: Introduce notifier APIs
Date: Sat, 28 Mar 2020 06:43:37 +0000
Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D19D7F76B7@SHSMSX104.ccr.corp.intel.com> (raw)
In-Reply-To: <20200327113646.3d87f17f@jacob-builder>

> From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Sent: Saturday, March 28, 2020 2:37 AM
> 
> On Fri, 27 Mar 2020 10:03:26 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > > Sent: Thursday, March 26, 2020 1:55 AM
> > >
> > > IOASID users fit into the publisher-subscriber pattern, a system
> > > wide blocking notifier chain can be used to inform subscribers of
> > > state changes. Notifier mechanism also abstracts publisher from
> > > knowing the private context each subcriber may have.
> > >
> > > This patch adds APIs and a global notifier chain, a further
> > > optimization might be per set notifier for ioasid_set aware users.
> > >
> > > Usage example:
> > > KVM register notifier block such that it can keep its guest-host
> > > PASID translation table in sync with any IOASID updates.
> > >
> > > VFIO publish IOASID change by performing alloc/free, bind/unbind
> > > operations.
> > >
> > > IOMMU driver gets notified when IOASID is freed by VFIO or core mm
> > > code such that PASID context can be cleaned up.
> >
> > above example looks mixed. You have KVM registers the notifier but
> > finally having IOMMU driver to get notified... 😊
> >
> Right, felt like a tale of two subscribers got mixed. I meant to list a
> few use cases with publisher and subscriber roles separate.
> I will change that to "Usage examples", and explicit state each role.
> 
> > >
> > > Signed-off-by: Liu Yi L <yi.l.liu@intel.com>
> > > Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > > ---
> > >  drivers/iommu/ioasid.c | 61
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++
> > >  include/linux/ioasid.h | 40 +++++++++++++++++++++++++++++++++
> > >  2 files changed, 101 insertions(+)
> > >
> > > diff --git a/drivers/iommu/ioasid.c b/drivers/iommu/ioasid.c
> > > index 8612fe6477dc..27dce2cb5af2 100644
> > > --- a/drivers/iommu/ioasid.c
> > > +++ b/drivers/iommu/ioasid.c
> > > @@ -11,6 +11,22 @@
> > >  #include <linux/xarray.h>
> > >
> > >  static DEFINE_XARRAY_ALLOC(ioasid_sets);
> > > +/*
> > > + * An IOASID could have multiple consumers. When a status change
> > > occurs,
> > > + * this notifier chain is used to keep them in sync. Each consumer
> > > of the
> > > + * IOASID service must register notifier block early to ensure no
> > > events
> > > + * are missed.
> > > + *
> > > + * This is a publisher-subscriber pattern where publisher can
> > > change the
> > > + * state of each IOASID, e.g. alloc/free, bind IOASID to a device
> > > and mm.
> > > + * On the other hand, subscribers gets notified for the state
> > > change and
> > > + * keep local states in sync.
> > > + *
> > > + * Currently, the notifier is global. A further optimization could
> > > be per
> > > + * IOASID set notifier chain.
> > > + */
> > > +static BLOCKING_NOTIFIER_HEAD(ioasid_chain);
> > > +
> > >  /**
> > >   * struct ioasid_set_data - Meta data about ioasid_set
> > >   *
> > > @@ -408,6 +424,7 @@ static void ioasid_free_locked(ioasid_t ioasid)
> > >  {
> > >  	struct ioasid_data *ioasid_data;
> > >  	struct ioasid_set_data *sdata;
> > > +	struct ioasid_nb_args args;
> > >
> > >  	ioasid_data = xa_load(&active_allocator->xa, ioasid);
> > >  	if (!ioasid_data) {
> > > @@ -415,6 +432,13 @@ static void ioasid_free_locked(ioasid_t ioasid)
> > >  		return;
> > >  	}
> > >
> > > +	args.id = ioasid;
> > > +	args.sid = ioasid_data->sdata->sid;
> > > +	args.pdata = ioasid_data->private;
> > > +	args.set_token = ioasid_data->sdata->token;
> > > +
> > > +	/* Notify all users that this IOASID is being freed */
> > > +	blocking_notifier_call_chain(&ioasid_chain, IOASID_FREE,
> > > &args); active_allocator->ops->free(ioasid,
> > > active_allocator->ops->pdata); /* Custom allocator needs additional
> > > steps to free the xa element */ if (active_allocator->flags &
> > > IOASID_ALLOCATOR_CUSTOM) { @@ -624,6 +648,43 @@ int
> > > ioasid_find_sid(ioasid_t ioasid) }
> > >  EXPORT_SYMBOL_GPL(ioasid_find_sid);
> > >
> > > +int ioasid_add_notifier(struct notifier_block *nb)
> > > +{
> > > +	return blocking_notifier_chain_register(&ioasid_chain, nb);
> > > +}
> > > +EXPORT_SYMBOL_GPL(ioasid_add_notifier);
> > > +
> > > +void ioasid_remove_notifier(struct notifier_block *nb)
> > > +{
> > > +	blocking_notifier_chain_unregister(&ioasid_chain, nb);
> > > +}
> > > +EXPORT_SYMBOL_GPL(ioasid_remove_notifier);
> >
> > register/unregister
> >
> Sounds good.
> 
> > > +
> > > +int ioasid_notify(ioasid_t ioasid, enum ioasid_notify_val cmd)
> >
> > add a comment on when this function should be used?
> >
> Sure, how about:
> /**
>  * ioasid_notify - Send notification on a given IOASID for status change.
>  *                 Used by publishers when the status change may affect
>  *                 subscriber's internal state.
>  *
>  * @ioasid:	The IOASID to which the notification will send
>  * @cmd:	The notification event
>  *
>  */

looks good.

> 
> > > +{
> > > +	struct ioasid_data *ioasid_data;
> > > +	struct ioasid_nb_args args;
> > > +	int ret = 0;
> > > +
> > > +	mutex_lock(&ioasid_allocator_lock);
> > > +	ioasid_data = xa_load(&active_allocator->xa, ioasid);
> > > +	if (!ioasid_data) {
> > > +		pr_err("Trying to free unknown IOASID %u\n",
> > > ioasid);
> >
> > why is it fixed to 'free'?
> >
> Good catch, it shouldn;t be just free. It was a relic of early test
> case.
> 
> > > +		mutex_unlock(&ioasid_allocator_lock);
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	args.id = ioasid;
> > > +	args.sid = ioasid_data->sdata->sid;
> > > +	args.pdata = ioasid_data->private;
> >
> > why no token info as did in ioasid_free?
> >
> Good catch, should include token as well. It is better to include all
> the data such that subscribers don't have to do any lookup which may
> cause race.
> 
> > > +
> > > +	ret = blocking_notifier_call_chain(&ioasid_chain, cmd,
> > > &args);
> > > +	mutex_unlock(&ioasid_allocator_lock);
> > > +
> > > +	return ret;
> > > +}
> > > +EXPORT_SYMBOL_GPL(ioasid_notify);
> > > +
> > >  MODULE_AUTHOR("Jean-Philippe Brucker <jean-
> > > philippe.brucker@arm.com>");
> > >  MODULE_AUTHOR("Jacob Pan <jacob.jun.pan@linux.intel.com>");
> > >  MODULE_DESCRIPTION("IO Address Space ID (IOASID) allocator");
> > > diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h
> > > index e19c0ad93bd7..32d032913828 100644
> > > --- a/include/linux/ioasid.h
> > > +++ b/include/linux/ioasid.h
> > > @@ -4,6 +4,7 @@
> > >
> > >  #include <linux/types.h>
> > >  #include <linux/errno.h>
> > > +#include <linux/notifier.h>
> > >
> > >  #define INVALID_IOASID ((ioasid_t)-1)
> > >  #define INVALID_IOASID_SET (-1)
> > > @@ -30,6 +31,27 @@ struct ioasid_allocator_ops {
> > >  	void *pdata;
> > >  };
> > >
> > > +/* Notification data when IOASID status changed */
> > > +enum ioasid_notify_val {
> > > +	IOASID_ALLOC = 1,
> > > +	IOASID_FREE,
> > > +	IOASID_BIND,
> > > +	IOASID_UNBIND,
> > > +};
> >
> > Curious why IOASID_ALLOC is not notified aumatically within
> > ioasid_alloc similar to ioasid_free, while leaving to the publisher?
> > BIND/UNBIND is a publisher thing but a bit strange to see ALLOC/FREE
> > with different policy here.
> >
> I don't see a use case for ALLOC notification yet. Any user does the
> allocation would the be the first and only one know about this IOASID.
> 
> Unless we have set level notifier, which may be interested in a new
> IOASID being allocated within the set.

then remove this type until it is actually required when supporting
set-level notifier? 

> 
> > > +
> > > +/**
> > > + * struct ioasid_nb_args - Argument provided by IOASID core when
> > > notifier
> > > + * is called.
> > > + * @id:		the IOASID being notified
> > > + * @sid:	the IOASID set @id belongs to
> > > + * @pdata:	the private data attached to the IOASID
> > > + */
> > > +struct ioasid_nb_args {
> > > +	ioasid_t id;
> > > +	int sid;
> > > +	struct ioasid_set *set_token;
> > > +	void *pdata;
> > > +};
> > >  /* Shared IOASID set for reserved for host system use */
> > >  extern int system_ioasid_sid;
> > >
> > > @@ -43,11 +65,15 @@ void *ioasid_find(int sid, ioasid_t ioasid, bool
> > > (*getter)(void *));
> > >  int ioasid_register_allocator(struct ioasid_allocator_ops
> > > *allocator); void ioasid_unregister_allocator(struct
> > > ioasid_allocator_ops *allocator); int ioasid_attach_data(ioasid_t
> > > ioasid, void *data); +int ioasid_add_notifier(struct notifier_block
> > > *nb); +void ioasid_remove_notifier(struct notifier_block *nb);
> > >  void ioasid_install_capacity(ioasid_t total);
> > >  int ioasid_alloc_system_set(int quota);
> > >  int ioasid_alloc_set(struct ioasid_set *token, ioasid_t quota, int
> > > *sid); void ioasid_free_set(int sid, bool destroy_set);
> > >  int ioasid_find_sid(ioasid_t ioasid);
> > > +int ioasid_notify(ioasid_t id, enum ioasid_notify_val cmd);
> > > +
> > >  #else /* !CONFIG_IOASID */
> > >  static inline ioasid_t ioasid_alloc(int sid, ioasid_t min,
> > >  				    ioasid_t max, void *private)
> > > @@ -73,6 +99,20 @@ static inline void *ioasid_find(int sid,
> > > ioasid_t ioasid, bool (*getter)(void *)
> > >  	return NULL;
> > >  }
> > >
> > > +static inline int ioasid_add_notifier(struct notifier_block *nb)
> > > +{
> > > +	return -ENOTSUPP;
> > > +}
> > > +
> > > +static inline void ioasid_remove_notifier(struct notifier_block
> > > *nb) +{
> > > +}
> > > +
> > > +int ioasid_notify(ioasid_t ioasid, enum ioasid_notify_val cmd)
> > > +{
> > > +	return -ENOTSUPP;
> > > +}
> > > +
> > >  static inline int ioasid_register_allocator(struct
> > > ioasid_allocator_ops *allocator)
> > >  {
> > >  	return -ENOTSUPP;
> > > --
> > > 2.7.4
> >
> 
> [Jacob Pan]
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

  reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-25 17:55 [PATCH 00/10] IOASID extensions for guest SVA Jacob Pan
2020-03-25 17:55 ` [PATCH 01/10] iommu/ioasid: Introduce system-wide capacity Jacob Pan
2020-03-27  8:07   ` Tian, Kevin
2020-03-27 16:08     ` Jacob Pan
2020-04-01 13:45   ` Jean-Philippe Brucker
2020-04-01 22:50     ` Jacob Pan
2020-03-25 17:55 ` [PATCH 02/10] iommu/vt-d: Set IOASID capacity when SVM is enabled Jacob Pan
2020-03-27  8:08   ` Tian, Kevin
2020-03-25 17:55 ` [PATCH 03/10] iommu/ioasid: Introduce per set allocation APIs Jacob Pan
2020-03-26  2:12   ` Lu Baolu
2020-03-26 21:30     ` Jacob Pan
2020-03-27  8:38   ` Tian, Kevin
2020-03-27 16:59     ` Jacob Pan
2020-03-28  6:32       ` Tian, Kevin
2020-04-01 13:47   ` Jean-Philippe Brucker
2020-04-06 20:02     ` Jacob Pan
2020-04-07 11:01       ` Jean-Philippe Brucker
2020-04-21 21:51         ` Jacob Pan
2020-03-25 17:55 ` [PATCH 04/10] iommu/ioasid: Rename ioasid_set_data to avoid confusion with ioasid_set Jacob Pan
2020-03-27  9:35   ` Tian, Kevin
2020-03-25 17:55 ` [PATCH 05/10] iommu/ioasid: Create an IOASID set for host SVA use Jacob Pan
2020-03-27  9:41   ` Tian, Kevin
2020-03-27 17:28     ` Jacob Pan
2020-03-28  6:33       ` Tian, Kevin
2020-04-01 13:53   ` Jean-Philippe Brucker
2020-04-06 15:33     ` Jacob Pan
2020-04-07 11:01       ` Jean-Philippe Brucker
2020-04-13 22:06         ` Jacob Pan
2020-04-15 15:10           ` Jean-Philippe Brucker
2020-03-25 17:55 ` [PATCH 06/10] iommu/ioasid: Convert to set aware allocations Jacob Pan
2020-03-27  9:54   ` Tian, Kevin
2020-03-27 17:41     ` Jacob Pan
2020-03-28  6:40       ` Tian, Kevin
2020-04-06 20:07         ` Jacob Pan
2020-04-01 13:55   ` Jean-Philippe Brucker
2020-04-01 22:45     ` Jacob Pan
2020-03-25 17:55 ` [PATCH 07/10] iommu/ioasid: Use mutex instead of spinlock Jacob Pan
2020-03-27  9:55   ` Tian, Kevin
2020-04-01 13:58   ` Jean-Philippe Brucker
2020-03-25 17:55 ` [PATCH 08/10] iommu/ioasid: Introduce notifier APIs Jacob Pan
2020-03-27 10:03   ` Tian, Kevin
2020-03-27 18:36     ` Jacob Pan
2020-03-28  6:43       ` Tian, Kevin [this message]
2020-03-31 15:13         ` Jacob Pan
2020-04-01 14:00   ` Jean-Philippe Brucker
2020-04-10 15:43     ` Jacob Pan
2020-03-25 17:55 ` [PATCH 09/10] iommu/ioasid: Support ioasid_set quota adjustment Jacob Pan
2020-03-27 10:09   ` Tian, Kevin
2020-03-27 23:30     ` Jacob Pan
2020-03-28  6:44       ` Tian, Kevin
2020-03-25 17:55 ` [PATCH 10/10] iommu/vt-d: Register PASID notifier for status change Jacob Pan
2020-03-27 10:22   ` Tian, Kevin
2020-03-27 23:47     ` Jacob Pan
2020-04-01 14:03 ` [PATCH 00/10] IOASID extensions for guest SVA Jean-Philippe Brucker
2020-04-01 23:38   ` Jacob Pan
2020-04-02 12:26     ` Jean-Philippe Brucker
2020-04-02 16:09       ` Jacob Pan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AADFC41AFE54684AB9EE6CBC0274A5D19D7F76B7@SHSMSX104.ccr.corp.intel.com \
    --to=kevin.tian@intel.com \
    --cc=alex.williamson@redhat.com \
    --cc=ashok.raj@intel.com \
    --cc=dwmw2@infradead.org \
    --cc=iommu@lists.linux-foundation.org \
    --cc=jacob.jun.pan@linux.intel.com \
    --cc=jean-philippe@linaro.com \
    --cc=jic23@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

IOMMU Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-iommu/0 linux-iommu/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-iommu linux-iommu/ https://lore.kernel.org/linux-iommu \
		iommu@lists.linux-foundation.org
	public-inbox-index linux-iommu

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.linux-foundation.lists.iommu


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git