From: Jacob Pan <jacob.jun.pan@linux.intel.com> To: Joerg Roedel <joro@8bytes.org>, Alex Williamson <alex.williamson@redhat.com>, "Lu Baolu" <baolu.lu@linux.intel.com>, iommu@lists.linux-foundation.org, LKML <linux-kernel@vger.kernel.org>, David Woodhouse <dwmw2@infradead.org>, Jean-Philippe Brucker <jean-philippe@linaro.com> Cc: "Yi Liu" <yi.l.liu@intel.com>, "Tian, Kevin" <kevin.tian@intel.com>, Raj Ashok <ashok.raj@intel.com>, "Christoph Hellwig" <hch@infradead.org>, Jonathan Cameron <jic23@kernel.org>, Eric Auger <eric.auger@redhat.com>, Jacob Pan <jacob.jun.pan@linux.intel.com> Subject: [PATCH 00/10] IOASID extensions for guest SVA Date: Wed, 25 Mar 2020 10:55:21 -0700 [thread overview] Message-ID: <1585158931-1825-1-git-send-email-jacob.jun.pan@linux.intel.com> (raw) IOASID was introduced in v5.5 as a generic kernel allocator service for both PCIe Process Address Space ID (PASID) and ARM SMMU's Sub Stream ID. In addition to basic ID allocation, ioasid_set was introduced as a token that is shared by a group of IOASIDs. This set token can be used for permission checking but lack of some features needed by guest Shared Virtual Address (SVA). In addition, IOASID support for life cycle management is needed among multiple users. This patchset introduces two extensions to the IOASID code, 1. IOASID set operations 2. Notifications for IOASID state synchronization Part #1: IOASIDs used by each VM fits naturally into an ioasid_set. The usage for per set management requires the following features: - Quota enforcement - This is to prevent one VM from abusing the allocator to take all the system IOASIDs. Though VFIO layer can also enforce the quota, but it cannot cover the usage with both guest and host SVA on the same system. - Stores guest IOASID-Host IOASID mapping within the set. To support live migration, IOASID namespace should be owned by the guest. This requires per IOASID set look up between guest and host IOASIDs. This patchset does not introduce non-identity guest-host IOASID lookup, we merely introduce the infrastructure in per set data. - Set level operations, e.g. when a guest terminates, it is likely to free the entire set. Having a single place to manage the set where the IOASIDs are stored makes iteration much easier. New APIs are: - void ioasid_install_capacity(ioasid_t total); Set the system capacity prior to any allocations. On x86, VT-d driver calls this function to set max number of PASIDs, typically 1 million (20 bits). - int ioasid_alloc_system_set(int quota); Host system has a default ioasid_set, during boot it is expected that this default set is allocated with a reasonable quota, e.g. PID_MAX. This default/system set is used for baremetal SVA. - int ioasid_alloc_set(struct ioasid_set *token, ioasid_t quota, int *sid); Allocate a new set with a token, returned sid (set ID) will be used to allocate IOASIDs within the set. Allocation of IOASIDs cannot exceed the quota. - void ioasid_free_set(int sid, bool destroy_set); Free the entire set and notify all users with an option to destroy the set. Set ID can be used for allocation again if not destroyed. - int ioasid_find_sid(ioasid_t ioasid); Look up the set ID from an ioasid. There is no reference held, assuming set has a single owner. - int ioasid_adjust_set(int sid, int quota); Change the quota of the set, new quota cannot be less than the number of IOASIDs already allocated within the set. This is useful when IOASID resource needs to be balanced among VMs. Part #2 Notification service. Since IOASIDs are used by many consumers that follow publisher-subscriber pattern, notification is a natural choice to keep states synchronized. For example, on x86 system, guest PASID allocation and bind call results in VFIO IOCTL that can add and change guest-host PASID states. At the same time, IOMMU driver and KVM need to maintain its own PASID contexts. In this case, VFIO is the publisher within the kernel, IOMMU driver and KVM are the subscribers. This patchset introduces a global blocking notifier chain and APIs to operate on. Not all events nor all IOASIDs are of interests to all subscribers. e.g. KVM is only interested in the IOASIDs within its set. IOMMU driver is not ioasid_set aware. A further optimization could be having both global and per set notifier. But consider the infrequent nature of bind/unbind and relatively long process life cycle, this optimization may not be needed at this time. To register/unregister notification blocks, use these two APIs: - int ioasid_add_notifier(struct notifier_block *nb); - void ioasid_remove_notifier(struct notifier_block *nb) To send notification on an IOASID with one of the commands (FREE, BIND/UNBIND, etc.), use: - int ioasid_notify(ioasid_t id, enum ioasid_notify_val cmd); This work is a result of collaboration with many people: Liu, Yi L <yi.l.liu@intel.com> Wu Hao <hao.wu@intel.com> Ashok Raj <ashok.raj@intel.com> Kevin Tian <kevin.tian@intel.com> Thanks, Jacob Jacob Pan (10): iommu/ioasid: Introduce system-wide capacity iommu/vt-d: Set IOASID capacity when SVM is enabled iommu/ioasid: Introduce per set allocation APIs iommu/ioasid: Rename ioasid_set_data to avoid confusion with ioasid_set iommu/ioasid: Create an IOASID set for host SVA use iommu/ioasid: Convert to set aware allocations iommu/ioasid: Use mutex instead of spinlock iommu/ioasid: Introduce notifier APIs iommu/ioasid: Support ioasid_set quota adjustment iommu/vt-d: Register PASID notifier for status change drivers/iommu/intel-iommu.c | 20 ++- drivers/iommu/intel-svm.c | 89 ++++++++-- drivers/iommu/ioasid.c | 387 +++++++++++++++++++++++++++++++++++++++----- include/linux/intel-iommu.h | 1 + include/linux/ioasid.h | 86 +++++++++- 5 files changed, 522 insertions(+), 61 deletions(-) -- 2.7.4
WARNING: multiple messages have this Message-ID (diff)
From: Jacob Pan <jacob.jun.pan@linux.intel.com> To: Joerg Roedel <joro@8bytes.org>, Alex Williamson <alex.williamson@redhat.com>, "Lu Baolu" <baolu.lu@linux.intel.com>, iommu@lists.linux-foundation.org, LKML <linux-kernel@vger.kernel.org>, David Woodhouse <dwmw2@infradead.org>, Jean-Philippe Brucker <jean-philippe@linaro.com> Cc: "Tian, Kevin" <kevin.tian@intel.com>, Raj Ashok <ashok.raj@intel.com>, Jonathan Cameron <jic23@kernel.org> Subject: [PATCH 00/10] IOASID extensions for guest SVA Date: Wed, 25 Mar 2020 10:55:21 -0700 [thread overview] Message-ID: <1585158931-1825-1-git-send-email-jacob.jun.pan@linux.intel.com> (raw) IOASID was introduced in v5.5 as a generic kernel allocator service for both PCIe Process Address Space ID (PASID) and ARM SMMU's Sub Stream ID. In addition to basic ID allocation, ioasid_set was introduced as a token that is shared by a group of IOASIDs. This set token can be used for permission checking but lack of some features needed by guest Shared Virtual Address (SVA). In addition, IOASID support for life cycle management is needed among multiple users. This patchset introduces two extensions to the IOASID code, 1. IOASID set operations 2. Notifications for IOASID state synchronization Part #1: IOASIDs used by each VM fits naturally into an ioasid_set. The usage for per set management requires the following features: - Quota enforcement - This is to prevent one VM from abusing the allocator to take all the system IOASIDs. Though VFIO layer can also enforce the quota, but it cannot cover the usage with both guest and host SVA on the same system. - Stores guest IOASID-Host IOASID mapping within the set. To support live migration, IOASID namespace should be owned by the guest. This requires per IOASID set look up between guest and host IOASIDs. This patchset does not introduce non-identity guest-host IOASID lookup, we merely introduce the infrastructure in per set data. - Set level operations, e.g. when a guest terminates, it is likely to free the entire set. Having a single place to manage the set where the IOASIDs are stored makes iteration much easier. New APIs are: - void ioasid_install_capacity(ioasid_t total); Set the system capacity prior to any allocations. On x86, VT-d driver calls this function to set max number of PASIDs, typically 1 million (20 bits). - int ioasid_alloc_system_set(int quota); Host system has a default ioasid_set, during boot it is expected that this default set is allocated with a reasonable quota, e.g. PID_MAX. This default/system set is used for baremetal SVA. - int ioasid_alloc_set(struct ioasid_set *token, ioasid_t quota, int *sid); Allocate a new set with a token, returned sid (set ID) will be used to allocate IOASIDs within the set. Allocation of IOASIDs cannot exceed the quota. - void ioasid_free_set(int sid, bool destroy_set); Free the entire set and notify all users with an option to destroy the set. Set ID can be used for allocation again if not destroyed. - int ioasid_find_sid(ioasid_t ioasid); Look up the set ID from an ioasid. There is no reference held, assuming set has a single owner. - int ioasid_adjust_set(int sid, int quota); Change the quota of the set, new quota cannot be less than the number of IOASIDs already allocated within the set. This is useful when IOASID resource needs to be balanced among VMs. Part #2 Notification service. Since IOASIDs are used by many consumers that follow publisher-subscriber pattern, notification is a natural choice to keep states synchronized. For example, on x86 system, guest PASID allocation and bind call results in VFIO IOCTL that can add and change guest-host PASID states. At the same time, IOMMU driver and KVM need to maintain its own PASID contexts. In this case, VFIO is the publisher within the kernel, IOMMU driver and KVM are the subscribers. This patchset introduces a global blocking notifier chain and APIs to operate on. Not all events nor all IOASIDs are of interests to all subscribers. e.g. KVM is only interested in the IOASIDs within its set. IOMMU driver is not ioasid_set aware. A further optimization could be having both global and per set notifier. But consider the infrequent nature of bind/unbind and relatively long process life cycle, this optimization may not be needed at this time. To register/unregister notification blocks, use these two APIs: - int ioasid_add_notifier(struct notifier_block *nb); - void ioasid_remove_notifier(struct notifier_block *nb) To send notification on an IOASID with one of the commands (FREE, BIND/UNBIND, etc.), use: - int ioasid_notify(ioasid_t id, enum ioasid_notify_val cmd); This work is a result of collaboration with many people: Liu, Yi L <yi.l.liu@intel.com> Wu Hao <hao.wu@intel.com> Ashok Raj <ashok.raj@intel.com> Kevin Tian <kevin.tian@intel.com> Thanks, Jacob Jacob Pan (10): iommu/ioasid: Introduce system-wide capacity iommu/vt-d: Set IOASID capacity when SVM is enabled iommu/ioasid: Introduce per set allocation APIs iommu/ioasid: Rename ioasid_set_data to avoid confusion with ioasid_set iommu/ioasid: Create an IOASID set for host SVA use iommu/ioasid: Convert to set aware allocations iommu/ioasid: Use mutex instead of spinlock iommu/ioasid: Introduce notifier APIs iommu/ioasid: Support ioasid_set quota adjustment iommu/vt-d: Register PASID notifier for status change drivers/iommu/intel-iommu.c | 20 ++- drivers/iommu/intel-svm.c | 89 ++++++++-- drivers/iommu/ioasid.c | 387 +++++++++++++++++++++++++++++++++++++++----- include/linux/intel-iommu.h | 1 + include/linux/ioasid.h | 86 +++++++++- 5 files changed, 522 insertions(+), 61 deletions(-) -- 2.7.4 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
next reply other threads:[~2020-03-25 17:49 UTC|newest] Thread overview: 114+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-03-25 17:55 Jacob Pan [this message] 2020-03-25 17:55 ` [PATCH 00/10] IOASID extensions for guest SVA Jacob Pan 2020-03-25 17:55 ` [PATCH 01/10] iommu/ioasid: Introduce system-wide capacity Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 8:07 ` Tian, Kevin 2020-03-27 8:07 ` Tian, Kevin 2020-03-27 16:08 ` Jacob Pan 2020-03-27 16:08 ` Jacob Pan 2020-04-01 13:45 ` Jean-Philippe Brucker 2020-04-01 13:45 ` Jean-Philippe Brucker 2020-04-01 22:50 ` Jacob Pan 2020-04-01 22:50 ` Jacob Pan 2020-03-25 17:55 ` [PATCH 02/10] iommu/vt-d: Set IOASID capacity when SVM is enabled Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 8:08 ` Tian, Kevin 2020-03-27 8:08 ` Tian, Kevin 2020-03-25 17:55 ` [PATCH 03/10] iommu/ioasid: Introduce per set allocation APIs Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-26 2:12 ` Lu Baolu 2020-03-26 2:12 ` Lu Baolu 2020-03-26 21:30 ` Jacob Pan 2020-03-26 21:30 ` Jacob Pan 2020-03-27 8:38 ` Tian, Kevin 2020-03-27 8:38 ` Tian, Kevin 2020-03-27 16:59 ` Jacob Pan 2020-03-27 16:59 ` Jacob Pan 2020-03-28 6:32 ` Tian, Kevin 2020-03-28 6:32 ` Tian, Kevin 2020-04-01 13:47 ` Jean-Philippe Brucker 2020-04-01 13:47 ` Jean-Philippe Brucker 2020-04-06 20:02 ` Jacob Pan 2020-04-06 20:02 ` Jacob Pan 2020-04-07 11:01 ` Jean-Philippe Brucker 2020-04-07 11:01 ` Jean-Philippe Brucker 2020-04-21 21:51 ` Jacob Pan 2020-04-21 21:51 ` Jacob Pan 2020-03-25 17:55 ` [PATCH 04/10] iommu/ioasid: Rename ioasid_set_data to avoid confusion with ioasid_set Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 9:35 ` Tian, Kevin 2020-03-27 9:35 ` Tian, Kevin 2020-03-25 17:55 ` [PATCH 05/10] iommu/ioasid: Create an IOASID set for host SVA use Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 9:41 ` Tian, Kevin 2020-03-27 9:41 ` Tian, Kevin 2020-03-27 17:28 ` Jacob Pan 2020-03-27 17:28 ` Jacob Pan 2020-03-28 6:33 ` Tian, Kevin 2020-03-28 6:33 ` Tian, Kevin 2020-04-01 13:53 ` Jean-Philippe Brucker 2020-04-01 13:53 ` Jean-Philippe Brucker 2020-04-06 15:33 ` Jacob Pan 2020-04-06 15:33 ` Jacob Pan 2020-04-07 11:01 ` Jean-Philippe Brucker 2020-04-07 11:01 ` Jean-Philippe Brucker 2020-04-13 22:06 ` Jacob Pan 2020-04-13 22:06 ` Jacob Pan 2020-04-15 15:10 ` Jean-Philippe Brucker 2020-04-15 15:10 ` Jean-Philippe Brucker 2020-03-25 17:55 ` [PATCH 06/10] iommu/ioasid: Convert to set aware allocations Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 9:54 ` Tian, Kevin 2020-03-27 9:54 ` Tian, Kevin 2020-03-27 17:41 ` Jacob Pan 2020-03-27 17:41 ` Jacob Pan 2020-03-28 6:40 ` Tian, Kevin 2020-03-28 6:40 ` Tian, Kevin 2020-04-06 20:07 ` Jacob Pan 2020-04-06 20:07 ` Jacob Pan 2020-04-01 13:55 ` Jean-Philippe Brucker 2020-04-01 13:55 ` Jean-Philippe Brucker 2020-04-01 22:45 ` Jacob Pan 2020-04-01 22:45 ` Jacob Pan 2020-03-25 17:55 ` [PATCH 07/10] iommu/ioasid: Use mutex instead of spinlock Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 9:55 ` Tian, Kevin 2020-03-27 9:55 ` Tian, Kevin 2020-04-01 13:58 ` Jean-Philippe Brucker 2020-04-01 13:58 ` Jean-Philippe Brucker 2020-03-25 17:55 ` [PATCH 08/10] iommu/ioasid: Introduce notifier APIs Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 10:03 ` Tian, Kevin 2020-03-27 10:03 ` Tian, Kevin 2020-03-27 18:36 ` Jacob Pan 2020-03-27 18:36 ` Jacob Pan 2020-03-28 6:43 ` Tian, Kevin 2020-03-28 6:43 ` Tian, Kevin 2020-03-31 15:13 ` Jacob Pan 2020-03-31 15:13 ` Jacob Pan 2020-04-01 14:00 ` Jean-Philippe Brucker 2020-04-01 14:00 ` Jean-Philippe Brucker 2020-04-10 15:43 ` Jacob Pan 2020-04-10 15:43 ` Jacob Pan 2020-03-25 17:55 ` [PATCH 09/10] iommu/ioasid: Support ioasid_set quota adjustment Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 10:09 ` Tian, Kevin 2020-03-27 10:09 ` Tian, Kevin 2020-03-27 23:30 ` Jacob Pan 2020-03-27 23:30 ` Jacob Pan 2020-03-28 6:44 ` Tian, Kevin 2020-03-28 6:44 ` Tian, Kevin 2020-03-25 17:55 ` [PATCH 10/10] iommu/vt-d: Register PASID notifier for status change Jacob Pan 2020-03-25 17:55 ` Jacob Pan 2020-03-27 10:22 ` Tian, Kevin 2020-03-27 10:22 ` Tian, Kevin 2020-03-27 23:47 ` Jacob Pan 2020-03-27 23:47 ` Jacob Pan 2020-04-01 14:03 ` [PATCH 00/10] IOASID extensions for guest SVA Jean-Philippe Brucker 2020-04-01 14:03 ` Jean-Philippe Brucker 2020-04-01 23:38 ` Jacob Pan 2020-04-01 23:38 ` Jacob Pan 2020-04-02 12:26 ` Jean-Philippe Brucker 2020-04-02 12:26 ` Jean-Philippe Brucker 2020-04-02 16:09 ` Jacob Pan 2020-04-02 16:09 ` Jacob Pan
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1585158931-1825-1-git-send-email-jacob.jun.pan@linux.intel.com \ --to=jacob.jun.pan@linux.intel.com \ --cc=alex.williamson@redhat.com \ --cc=ashok.raj@intel.com \ --cc=baolu.lu@linux.intel.com \ --cc=dwmw2@infradead.org \ --cc=eric.auger@redhat.com \ --cc=hch@infradead.org \ --cc=iommu@lists.linux-foundation.org \ --cc=jean-philippe@linaro.com \ --cc=jic23@kernel.org \ --cc=joro@8bytes.org \ --cc=kevin.tian@intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=yi.l.liu@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.