From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935015AbcJFUnr (ORCPT ); Thu, 6 Oct 2016 16:43:47 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50282 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933011AbcJFUni (ORCPT ); Thu, 6 Oct 2016 16:43:38 -0400 Date: Thu, 6 Oct 2016 14:42:51 -0600 From: Alex Williamson To: Eric Auger Cc: eric.auger.pro@gmail.com, christoffer.dall@linaro.org, marc.zyngier@arm.com, robin.murphy@arm.com, will.deacon@arm.com, joro@8bytes.org, tglx@linutronix.de, jason@lakedaemon.net, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, drjones@redhat.com, linux-kernel@vger.kernel.org, Bharat.Bhushan@freescale.com, pranav.sawargaonkar@gmail.com, p.fedin@samsung.com, iommu@lists.linux-foundation.org, Jean-Philippe.Brucker@arm.com, yehuday@marvell.com, Manish.Jaggi@caviumnetworks.com Subject: Re: [PATCH v13 15/15] vfio/type1: Return the MSI geometry through VFIO_IOMMU_GET_INFO capability chains Message-ID: <20161006144251.46b45117@t450s.home> In-Reply-To: <20161006142040.13cbdb8f@t450s.home> References: <1475743531-4780-1-git-send-email-eric.auger@redhat.com> <1475743531-4780-16-git-send-email-eric.auger@redhat.com> <20161006142040.13cbdb8f@t450s.home> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Thu, 06 Oct 2016 20:42:53 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 6 Oct 2016 14:20:40 -0600 Alex Williamson wrote: > On Thu, 6 Oct 2016 08:45:31 +0000 > Eric Auger wrote: > > > This patch allows the user-space to retrieve the MSI geometry. The > > implementation is based on capability chains, now also added to > > VFIO_IOMMU_GET_INFO. > > > > The returned info comprise: > > - whether the MSI IOVA are constrained to a reserved range (x86 case) and > > in the positive, the start/end of the aperture, > > - or whether the IOVA aperture need to be set by the userspace. In that > > case, the size and alignment of the IOVA window to be provided are > > returned. > > > > In case the userspace must provide the IOVA aperture, we currently report > > a size/alignment based on all the doorbells registered by the host kernel. > > This may exceed the actual needs. > > > > Signed-off-by: Eric Auger > > > > --- > > v11 -> v11: > > - msi_doorbell_pages was renamed msi_doorbell_calc_pages > > > > v9 -> v10: > > - move cap_offset after iova_pgsizes > > - replace __u64 alignment by __u32 order > > - introduce __u32 flags in vfio_iommu_type1_info_cap_msi_geometry and > > fix alignment > > - call msi-doorbell API to compute the size/alignment > > > > v8 -> v9: > > - use iommu_msi_supported flag instead of programmable > > - replace IOMMU_INFO_REQUIRE_MSI_MAP flag by a more sophisticated > > capability chain, reporting the MSI geometry > > > > v7 -> v8: > > - use iommu_domain_msi_geometry > > > > v6 -> v7: > > - remove the computation of the number of IOVA pages to be provisionned. > > This number depends on the domain/group/device topology which can > > dynamically change. Let's rely instead rely on an arbitrary max depending > > on the system > > > > v4 -> v5: > > - move msi_info and ret declaration within the conditional code > > > > v3 -> v4: > > - replace former vfio_domains_require_msi_mapping by > > more complex computation of MSI mapping requirements, especially the > > number of pages to be provided by the user-space. > > - reword patch title > > > > RFC v1 -> v1: > > - derived from > > [RFC PATCH 3/6] vfio: Extend iommu-info to return MSIs automap state > > - renamed allow_msi_reconfig into require_msi_mapping > > - fixed VFIO_IOMMU_GET_INFO > > --- > > drivers/vfio/vfio_iommu_type1.c | 78 ++++++++++++++++++++++++++++++++++++++++- > > include/uapi/linux/vfio.h | 32 ++++++++++++++++- > > 2 files changed, 108 insertions(+), 2 deletions(-) > > > > diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c > > index dc3ee5d..ce5e7eb 100644 > > --- a/drivers/vfio/vfio_iommu_type1.c > > +++ b/drivers/vfio/vfio_iommu_type1.c > > @@ -38,6 +38,8 @@ > > #include > > #include > > #include > > +#include > > +#include > > > > #define DRIVER_VERSION "0.2" > > #define DRIVER_AUTHOR "Alex Williamson " > > @@ -1101,6 +1103,55 @@ static int vfio_domains_have_iommu_cache(struct vfio_iommu *iommu) > > return ret; > > } > > > > +static int compute_msi_geometry_caps(struct vfio_iommu *iommu, > > + struct vfio_info_cap *caps) > > +{ > > + struct vfio_iommu_type1_info_cap_msi_geometry *vfio_msi_geometry; > > + unsigned long order = __ffs(vfio_pgsize_bitmap(iommu)); > > + struct iommu_domain_msi_geometry msi_geometry; > > + struct vfio_info_cap_header *header; > > + struct vfio_domain *d; > > + bool reserved; > > + size_t size; > > + > > + mutex_lock(&iommu->lock); > > + /* All domains have same require_msi_map property, pick first */ > > + d = list_first_entry(&iommu->domain_list, struct vfio_domain, next); > > + iommu_domain_get_attr(d->domain, DOMAIN_ATTR_MSI_GEOMETRY, > > + &msi_geometry); > > + reserved = !msi_geometry.iommu_msi_supported; > > + > > + mutex_unlock(&iommu->lock); > > + > > + size = sizeof(*vfio_msi_geometry); > > + header = vfio_info_cap_add(caps, size, > > + VFIO_IOMMU_TYPE1_INFO_CAP_MSI_GEOMETRY, 1); > > + > > + if (IS_ERR(header)) > > + return PTR_ERR(header); > > + > > + vfio_msi_geometry = container_of(header, > > + struct vfio_iommu_type1_info_cap_msi_geometry, > > + header); > > + > > + vfio_msi_geometry->flags = reserved; > > Use the bit flag VFIO_IOMMU_MSI_GEOMETRY_RESERVED > > > + if (reserved) { > > + vfio_msi_geometry->aperture_start = msi_geometry.aperture_start; > > + vfio_msi_geometry->aperture_end = msi_geometry.aperture_end; > > But maybe nobody has set these, did you intend to use > iommu_domain_msi_aperture_valid(), which you defined early on but never > used? > > > + return 0; > > + } > > + > > + vfio_msi_geometry->order = order; > > I'm tempted to suggest that a user could do the same math on their own > since we provide the supported bitmap already... could it ever not be > the same? > > > + /* > > + * we compute a system-wide requirement based on all the registered > > + * doorbells > > + */ > > + vfio_msi_geometry->size = > > + msi_doorbell_calc_pages(order) * ((uint64_t) 1 << order); > > + > > + return 0; > > +} > > + > > static long vfio_iommu_type1_ioctl(void *iommu_data, > > unsigned int cmd, unsigned long arg) > > { > > @@ -1122,8 +1173,10 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, > > } > > } else if (cmd == VFIO_IOMMU_GET_INFO) { > > struct vfio_iommu_type1_info info; > > + struct vfio_info_cap caps = { .buf = NULL, .size = 0 }; > > + int ret; > > > > - minsz = offsetofend(struct vfio_iommu_type1_info, iova_pgsizes); > > + minsz = offsetofend(struct vfio_iommu_type1_info, cap_offset); > > > > if (copy_from_user(&info, (void __user *)arg, minsz)) > > return -EFAULT; > > @@ -1135,6 +1188,29 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, > > > > info.iova_pgsizes = vfio_pgsize_bitmap(iommu); > > > > + ret = compute_msi_geometry_caps(iommu, &caps); > > + if (ret) > > + return ret; > > + > > + if (caps.size) { > > + info.flags |= VFIO_IOMMU_INFO_CAPS; > > + if (info.argsz < sizeof(info) + caps.size) { > > + info.argsz = sizeof(info) + caps.size; > > + info.cap_offset = 0; > > + } else { > > + vfio_info_cap_shift(&caps, sizeof(info)); > > + if (copy_to_user((void __user *)arg + > > + sizeof(info), caps.buf, > > + caps.size)) { > > + kfree(caps.buf); > > + return -EFAULT; > > + } > > + info.cap_offset = sizeof(info); > > + } > > + > > + kfree(caps.buf); > > + } > > + > > return copy_to_user((void __user *)arg, &info, minsz) ? > > -EFAULT : 0; > > > > diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h > > index 4a9dbc2..8dae013 100644 > > --- a/include/uapi/linux/vfio.h > > +++ b/include/uapi/linux/vfio.h > > @@ -488,7 +488,35 @@ struct vfio_iommu_type1_info { > > __u32 argsz; > > __u32 flags; > > #define VFIO_IOMMU_INFO_PGSIZES (1 << 0) /* supported page sizes info */ > > - __u64 iova_pgsizes; /* Bitmap of supported page sizes */ > > +#define VFIO_IOMMU_INFO_CAPS (1 << 1) /* Info supports caps */ > > + __u64 iova_pgsizes; /* Bitmap of supported page sizes */ > > + __u32 __resv; > > + __u32 cap_offset; /* Offset within info struct of first cap */ > > +}; > > I understand the padding, but not the ordering. Why not end with > padding? > > > + > > +#define VFIO_IOMMU_TYPE1_INFO_CAP_MSI_GEOMETRY 1 > > + > > +/* > > + * The MSI geometry capability allows to report the MSI IOVA geometry: > > + * - either the MSI IOVAs are constrained within a reserved IOVA aperture > > + * whose boundaries are given by [@aperture_start, @aperture_end]. > > + * this is typically the case on x86 host. The userspace is not allowed > > + * to map userspace memory at IOVAs intersecting this range using > > + * VFIO_IOMMU_MAP_DMA. > > + * - or the MSI IOVAs are not requested to belong to any reserved range; > > + * in that case the userspace must provide an IOVA window characterized by > > + * @size and @alignment using VFIO_IOMMU_MAP_DMA with RESERVED_MSI_IOVA flag. > > + */ > > +struct vfio_iommu_type1_info_cap_msi_geometry { > > + struct vfio_info_cap_header header; > > + __u32 flags; > > +#define VFIO_IOMMU_MSI_GEOMETRY_RESERVED (1 << 0) /* reserved geometry */ > > + /* not reserved */ > > + __u32 order; /* iommu page order used for aperture alignment*/ > > + __u64 size; /* IOVA aperture size (bytes) the userspace must provide */ > > + /* reserved */ > > + __u64 aperture_start; > > + __u64 aperture_end; > > Should these be a union? We never set them both. Should the !reserved > case have a flag as well, so the user can positively identify what's > being provided? Actually, is there really any need to fit both of these within the same structure? Part of the idea of the capability chains is we can create a capability for each new thing we want to describe. So, we could simply define a generic reserved IOVA range capability with a 'start' and 'end' and then another capability to define MSI mapping requirements. Thanks, Alex > > }; > > > > #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12) > > @@ -503,6 +531,8 @@ struct vfio_iommu_type1_info { > > * IOVA region that will be used on some platforms to map the host MSI frames. > > * In that specific case, vaddr is ignored. Once registered, an MSI reserved > > * IOVA region stays until the container is closed. > > + * The requirement for provisioning such reserved IOVA range can be checked by > > + * checking the VFIO_IOMMU_TYPE1_INFO_CAP_MSI_GEOMETRY capability. > > */ > > struct vfio_iommu_type1_dma_map { > > __u32 argsz; >