[PATCH 1/2] vfio/type1: Cope with hardware MSI reserved regions

[PATCH 1/2] vfio/type1: Cope with hardware MSI reserved regions

[Robin Murphy]

For ARM-based systems with a GICv3 ITS to provide interrupt isolation,
but hardware limitations which are worked around by having MSIs bypass
SMMU translation (e.g. HiSilicon Hip06/Hip07), VFIO neglects to check
for the IRQ_DOMAIN_FLAG_MSI_REMAP capability, (and thus erroneously
demands unsafe_interrupts) if a software-managed MSI region is absent.

Fix this by always checking for isolation capability at both the IRQ
domain and IOMMU domain levels, rather than predicating that on whether
MSIs require an IOMMU mapping (which was always slightly tenuous logic).

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/vfio/vfio_iommu_type1.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 8549cb111627..2328be628f21 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -1265,8 +1265,8 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
 	INIT_LIST_HEAD(&domain->group_list);
 	list_add(&group->next, &domain->group_list);
 
-	msi_remap = resv_msi ? irq_domain_check_msi_remap() :
-				iommu_capable(bus, IOMMU_CAP_INTR_REMAP);
+	msi_remap = irq_domain_check_msi_remap() ||
+		    iommu_capable(bus, IOMMU_CAP_INTR_REMAP);
 
 	if (!allow_unsafe_interrupts && !msi_remap) {
 		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
-- 
2.12.2.dirty

[PATCH 2/2] vfio/type1: Give hardware MSI regions precedence

[Robin Murphy]

If the IOMMU driver advertises 'real' reserved regions for MSIs, but
still includes the software-managed region as well, we are currently
blind to the former and will configure the IOMMU domain to map MSIs into
the latter, which is unlikely to work as expected.

Since it would take a ridiculous hardware topology for both regions to
be valid (which would be rather difficult to support in general), we
should be safe to assume that the presence of any hardware regions makes
the software region irrelevant. However, the IOMMU driver might still
advertise the software region by default, particularly if the hardware
regions are filled in elsewhere by generic code, so it might not be fair
for VFIO to be super-strict about not mixing them. To that end, make
vfio_iommu_has_sw_msi() robust against the presence of both region types
at once, so that we end up doing what is almost certainly right, rather
than what is almost certainly wrong.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/vfio/vfio_iommu_type1.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 2328be628f21..92155cce926d 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -1169,13 +1169,21 @@ static bool vfio_iommu_has_sw_msi(struct iommu_group *group, phys_addr_t *base)
 	INIT_LIST_HEAD(&group_resv_regions);
 	iommu_get_group_resv_regions(group, &group_resv_regions);
 	list_for_each_entry(region, &group_resv_regions, list) {
+		/*
+		 * The presence of any 'real' MSI regions should take
+		 * precedence over the software-managed one if the
+		 * IOMMU driver happens to advertise both types.
+		 */
+		if (region->type == IOMMU_RESV_MSI) {
+			ret = false;
+			break;
+		}
+
 		if (region->type == IOMMU_RESV_SW_MSI) {
 			*base = region->start;
 			ret = true;
-			goto out;
 		}
 	}
-out:
 	list_for_each_entry_safe(region, next, &group_resv_regions, list)
 		kfree(region);
 	return ret;
-- 
2.12.2.dirty

Re: [PATCH 1/2] vfio/type1: Cope with hardware MSI reserved regions

[Auger Eric]

Hi Robin,
On 27/07/2017 16:54, Robin Murphy wrote:
> For ARM-based systems with a GICv3 ITS to provide interrupt isolation,
> but hardware limitations which are worked around by having MSIs bypass
> SMMU translation (e.g. HiSilicon Hip06/Hip07), VFIO neglects to check
> for the IRQ_DOMAIN_FLAG_MSI_REMAP capability, (and thus erroneously
> demands unsafe_interrupts) if a software-managed MSI region is absent.
> 
> Fix this by always checking for isolation capability at both the IRQ
> domain and IOMMU domain levels, rather than predicating that on whether
> MSIs require an IOMMU mapping (which was always slightly tenuous logic).
> 
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

> ---
>  drivers/vfio/vfio_iommu_type1.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 8549cb111627..2328be628f21 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -1265,8 +1265,8 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  	INIT_LIST_HEAD(&domain->group_list);
>  	list_add(&group->next, &domain->group_list);
>  
> -	msi_remap = resv_msi ? irq_domain_check_msi_remap() :
> -				iommu_capable(bus, IOMMU_CAP_INTR_REMAP);
> +	msi_remap = irq_domain_check_msi_remap() ||
> +		    iommu_capable(bus, IOMMU_CAP_INTR_REMAP);
>  
>  	if (!allow_unsafe_interrupts && !msi_remap) {
>  		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
> 

Re: [PATCH 2/2] vfio/type1: Give hardware MSI regions precedence

[Auger Eric]

Hi Robin,
On 27/07/2017 16:54, Robin Murphy wrote:
> If the IOMMU driver advertises 'real' reserved regions for MSIs, but
> still includes the software-managed region as well, we are currently
> blind to the former and will configure the IOMMU domain to map MSIs into
> the latter, which is unlikely to work as expected.
> 
> Since it would take a ridiculous hardware topology for both regions to
> be valid (which would be rather difficult to support in general), we
> should be safe to assume that the presence of any hardware regions makes
> the software region irrelevant. However, the IOMMU driver might still
> advertise the software region by default, particularly if the hardware
> regions are filled in elsewhere by generic code, so it might not be fair
> for VFIO to be super-strict about not mixing them. To that end, make
> vfio_iommu_has_sw_msi() robust against the presence of both region types
> at once, so that we end up doing what is almost certainly right, rather
> than what is almost certainly wrong.
> 
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric
> ---
>  drivers/vfio/vfio_iommu_type1.c | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 2328be628f21..92155cce926d 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -1169,13 +1169,21 @@ static bool vfio_iommu_has_sw_msi(struct iommu_group *group, phys_addr_t *base)
>  	INIT_LIST_HEAD(&group_resv_regions);
>  	iommu_get_group_resv_regions(group, &group_resv_regions);
>  	list_for_each_entry(region, &group_resv_regions, list) {
> +		/*
> +		 * The presence of any 'real' MSI regions should take
> +		 * precedence over the software-managed one if the
> +		 * IOMMU driver happens to advertise both types.
> +		 */
> +		if (region->type == IOMMU_RESV_MSI) {
> +			ret = false;
> +			break;
> +		}
> +
>  		if (region->type == IOMMU_RESV_SW_MSI) {
>  			*base = region->start;
>  			ret = true;
> -			goto out;
>  		}
>  	}
> -out:
>  	list_for_each_entry_safe(region, next, &group_resv_regions, list)
>  		kfree(region);
>  	return ret;
> 

RE: [PATCH 1/2] vfio/type1: Cope with hardware MSI reserved regions

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Robin Murphy [mailto:robin.murphy@arm.com]
> Sent: Thursday, July 27, 2017 3:54 PM
> To: alex.williamson@redhat.com
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; eric.auger@redhat.com; Shameerali Kolothum
> Thodi; marc.zyngier@arm.com
> Subject: [PATCH 1/2] vfio/type1: Cope with hardware MSI reserved regions
> 
> For ARM-based systems with a GICv3 ITS to provide interrupt isolation,
> but hardware limitations which are worked around by having MSIs bypass
> SMMU translation (e.g. HiSilicon Hip06/Hip07), VFIO neglects to check
> for the IRQ_DOMAIN_FLAG_MSI_REMAP capability, (and thus erroneously
> demands unsafe_interrupts) if a software-managed MSI region is absent.
> 
> Fix this by always checking for isolation capability at both the IRQ
> domain and IOMMU domain levels, rather than predicating that on whether
> MSIs require an IOMMU mapping (which was always slightly tenuous logic).
> 
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
> ---
>  drivers/vfio/vfio_iommu_type1.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vfio/vfio_iommu_type1.c
> b/drivers/vfio/vfio_iommu_type1.c
> index 8549cb111627..2328be628f21 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -1265,8 +1265,8 @@ static int vfio_iommu_type1_attach_group(void
> *iommu_data,
>  	INIT_LIST_HEAD(&domain->group_list);
>  	list_add(&group->next, &domain->group_list);
> 
> -	msi_remap = resv_msi ? irq_domain_check_msi_remap() :
> -				iommu_capable(bus,
> IOMMU_CAP_INTR_REMAP);
> +	msi_remap = irq_domain_check_msi_remap() ||
> +		    iommu_capable(bus, IOMMU_CAP_INTR_REMAP);
> 
>  	if (!allow_unsafe_interrupts && !msi_remap) {
>  		pr_warn("%s: No interrupt remapping support.  Use the
> module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support
> on this platform\n",

Verified these patches(1 and 2) on HiSilicon Hip07 platform along with HW MSI resv
region patches[1]. Please feel free to add, 

Tested-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

Thanks,
Shameer

1. https://www.spinics.net/lists/arm-kernel/msg595847.html

Re: [PATCH 1/2] vfio/type1: Cope with hardware MSI reserved regions

[Alex Williamson]

On Thu, 27 Jul 2017 15:54:13 +0100
Robin Murphy <robin.murphy@arm.com> wrote:

> For ARM-based systems with a GICv3 ITS to provide interrupt isolation,
> but hardware limitations which are worked around by having MSIs bypass
> SMMU translation (e.g. HiSilicon Hip06/Hip07), VFIO neglects to check
> for the IRQ_DOMAIN_FLAG_MSI_REMAP capability, (and thus erroneously
> demands unsafe_interrupts) if a software-managed MSI region is absent.
> 
> Fix this by always checking for isolation capability at both the IRQ
> domain and IOMMU domain levels, rather than predicating that on whether
> MSIs require an IOMMU mapping (which was always slightly tenuous logic).
> 
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
> ---
>  drivers/vfio/vfio_iommu_type1.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

I've pushed these to my next branch with Eric's R-b and Shameerali's
T-b for v4.14.  Thanks,

Alex

> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 8549cb111627..2328be628f21 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -1265,8 +1265,8 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  	INIT_LIST_HEAD(&domain->group_list);
>  	list_add(&group->next, &domain->group_list);
>  
> -	msi_remap = resv_msi ? irq_domain_check_msi_remap() :
> -				iommu_capable(bus, IOMMU_CAP_INTR_REMAP);
> +	msi_remap = irq_domain_check_msi_remap() ||
> +		    iommu_capable(bus, IOMMU_CAP_INTR_REMAP);
>  
>  	if (!allow_unsafe_interrupts && !msi_remap) {
>  		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",