[PATCH v1 0/3] iommu/vt-d: real DMA sub-device info allocation

[PATCH v1 0/3] iommu/vt-d: real DMA sub-device info allocation

[Jon Derrick]

This set adds the support for real DMA sub-devices to have device_domain_info,
leading to the correct domain type being used.

This applies on Joerg's origin/next. This also applies against v5.6.12
and v5.7-rc7 with some API modifications, making it a stable candidate
that fixes the issue reported in [1].

For v5.6.12 and v5.7-rc7, identity_mapping() would return 0 for real DMA
sub-devices due to not having valid device_domain_info, leading to
__intel_map_single() paths. This is a problem if the real DMA device
started in IDENTITY, leading to a NULL Pointer Dereference:

	__intel_map_single()
		domain = find_domain(dev);
			dev = &pci_real_dma_dev(to_pci_dev(dev))->dev;
			info = dev->archdata.iommu;
			return info->domain;

		iommu = domain_get_iommu(domain)
			if (WARN_ON(domain->domain.type != IOMMU_DOMAIN_DMA))
				return NULL;

		cap_zlr(iommu->cap) <-- NULL Pointer Deref

This issue was also fixed by 6fc7020cf298 ("iommu/vt-d: Apply per-device
dma_ops") due to removing identity_mapping() paths.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=207575

Jon Derrick (3):
  iommu/vt-d: Only clear real DMA device's context entries
  iommu/vt-d: Allocate domain info for real DMA sub-devices
  iommu/vt-d: Remove real DMA lookup in find_domain

 drivers/iommu/intel-iommu.c | 31 +++++++++++++++++++++++--------
 include/linux/intel-iommu.h |  1 +
 2 files changed, 24 insertions(+), 8 deletions(-)

-- 
1.8.3.1

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v1 1/3] iommu/vt-d: Only clear real DMA device's context entries

[Jon Derrick]

Domain context mapping can encounter issues with sub-devices of a real
DMA device. A sub-device cannot have a valid context entry due to it
potentially aliasing another device's 16-bit ID. It's expected that
sub-devices of the real DMA device uses the real DMA device's requester
when context mapping.

This is an issue when a sub-device is removed where the context entry is
cleared for all aliases. Other sub-devices are still valid, resulting in
those sub-devices being stranded without valid context entries.

The correct approach is to use the real DMA device when programming the
context entries. The insertion path is correct because device_to_iommu()
will return the bus and devfn of the real DMA device. The removal path
needs to only operate on the real DMA device, otherwise the entire
context entry would be cleared for all sub-devices of the real DMA
device.

This patch also adds a helper to determine if a struct device is a
sub-device of a real DMA device.

Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>
---
 drivers/iommu/intel-iommu.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index ff5a30a..1ff45b2 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -2500,6 +2500,12 @@ static int domain_setup_first_level(struct intel_iommu *iommu,
 					     flags);
 }
 
+static bool dev_is_real_dma_subdevice(struct device *dev)
+{
+	return dev && dev_is_pci(dev) &&
+	       pci_real_dma_dev(to_pci_dev(dev)) != to_pci_dev(dev);
+}
+
 static struct dmar_domain *dmar_insert_one_dev_info(struct intel_iommu *iommu,
 						    int bus, int devfn,
 						    struct device *dev,
@@ -4975,7 +4981,8 @@ static void __dmar_remove_one_dev_info(struct device_domain_info *info)
 					PASID_RID2PASID, false);
 
 		iommu_disable_dev_iotlb(info);
-		domain_context_clear(iommu, info->dev);
+		if (!dev_is_real_dma_subdevice(info->dev))
+			domain_context_clear(iommu, info->dev);
 		intel_pasid_free_table(info->dev);
 	}
 
-- 
1.8.3.1

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1 1/3] iommu/vt-d: Only clear real DMA device's context entries

[Lu Baolu]

On 2020/5/28 0:56, Jon Derrick wrote:
> Domain context mapping can encounter issues with sub-devices of a real
> DMA device. A sub-device cannot have a valid context entry due to it
> potentially aliasing another device's 16-bit ID. It's expected that
> sub-devices of the real DMA device uses the real DMA device's requester
> when context mapping.
> 
> This is an issue when a sub-device is removed where the context entry is
> cleared for all aliases. Other sub-devices are still valid, resulting in
> those sub-devices being stranded without valid context entries.
> 
> The correct approach is to use the real DMA device when programming the
> context entries. The insertion path is correct because device_to_iommu()
> will return the bus and devfn of the real DMA device. The removal path
> needs to only operate on the real DMA device, otherwise the entire
> context entry would be cleared for all sub-devices of the real DMA
> device.
> 
> This patch also adds a helper to determine if a struct device is a
> sub-device of a real DMA device.
> 
> Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>

Fixes: 2b0140c69637e ("iommu/vt-d: Use pci_real_dma_dev() for mapping")
Acked-by: Lu Baolu <baolu.lu@linux.intel.com>

Best regards,
baolu
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v1 2/3] iommu/vt-d: Allocate domain info for real DMA sub-devices

[Jon Derrick]

Sub-devices of a real DMA device might exist on a separate segment than
the real DMA device and its IOMMU. These devices should still have a
valid device_domain_info, but the current dma alias model won't
allocate info for the subdevice.

This patch adds a segment member to struct device_domain_info and uses
the sub-device's BDF so that these sub-devices won't alias to other
devices.

Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>
---
 drivers/iommu/intel-iommu.c | 19 +++++++++++++++----
 include/linux/intel-iommu.h |  1 +
 2 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 1ff45b2..6d39b9b 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -2463,7 +2463,7 @@ static void do_deferred_attach(struct device *dev)
 	struct device_domain_info *info;
 
 	list_for_each_entry(info, &device_domain_list, global)
-		if (info->iommu->segment == segment && info->bus == bus &&
+		if (info->segment == segment && info->bus == bus &&
 		    info->devfn == devfn)
 			return info;
 
@@ -2520,8 +2520,18 @@ static struct dmar_domain *dmar_insert_one_dev_info(struct intel_iommu *iommu,
 	if (!info)
 		return NULL;
 
-	info->bus = bus;
-	info->devfn = devfn;
+	if (!dev_is_real_dma_subdevice(dev)) {
+		info->bus = bus;
+		info->devfn = devfn;
+		info->segment = iommu->segment;
+	} else {
+		struct pci_dev *pdev = to_pci_dev(dev);
+
+		info->bus = pdev->bus->number;
+		info->devfn = pdev->devfn;
+		info->segment = pci_domain_nr(pdev->bus);
+	}
+
 	info->ats_supported = info->pasid_supported = info->pri_supported = 0;
 	info->ats_enabled = info->pasid_enabled = info->pri_enabled = 0;
 	info->ats_qdep = 0;
@@ -2561,7 +2571,8 @@ static struct dmar_domain *dmar_insert_one_dev_info(struct intel_iommu *iommu,
 
 	if (!found) {
 		struct device_domain_info *info2;
-		info2 = dmar_search_domain_by_dev_info(iommu->segment, bus, devfn);
+		info2 = dmar_search_domain_by_dev_info(info->segment, info->bus,
+						       info->devfn);
 		if (info2) {
 			found      = info2->domain;
 			info2->dev = dev;
diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h
index 21633ce..4100bd2 100644
--- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h
@@ -609,6 +609,7 @@ struct device_domain_info {
 	struct list_head auxiliary_domains; /* auxiliary domains
 					     * attached to this device
 					     */
+	u32 segment;		/* PCI segment number */
 	u8 bus;			/* PCI bus number */
 	u8 devfn;		/* PCI devfn number */
 	u16 pfsid;		/* SRIOV physical function source ID */
-- 
1.8.3.1

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1 2/3] iommu/vt-d: Allocate domain info for real DMA sub-devices

[Lu Baolu]

On 2020/5/28 0:56, Jon Derrick wrote:
> Sub-devices of a real DMA device might exist on a separate segment than
> the real DMA device and its IOMMU. These devices should still have a
> valid device_domain_info, but the current dma alias model won't
> allocate info for the subdevice.
> 
> This patch adds a segment member to struct device_domain_info and uses
> the sub-device's BDF so that these sub-devices won't alias to other
> devices.
> 
> Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>

Acked-by: Lu Baolu <baolu.lu@linux.intel.com>

Best regards,
baolu

> ---
>   drivers/iommu/intel-iommu.c | 19 +++++++++++++++----
>   include/linux/intel-iommu.h |  1 +
>   2 files changed, 16 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index 1ff45b2..6d39b9b 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -2463,7 +2463,7 @@ static void do_deferred_attach(struct device *dev)
>   	struct device_domain_info *info;
>   
>   	list_for_each_entry(info, &device_domain_list, global)
> -		if (info->iommu->segment == segment && info->bus == bus &&
> +		if (info->segment == segment && info->bus == bus &&
>   		    info->devfn == devfn)
>   			return info;
>   
> @@ -2520,8 +2520,18 @@ static struct dmar_domain *dmar_insert_one_dev_info(struct intel_iommu *iommu,
>   	if (!info)
>   		return NULL;
>   
> -	info->bus = bus;
> -	info->devfn = devfn;
> +	if (!dev_is_real_dma_subdevice(dev)) {
> +		info->bus = bus;
> +		info->devfn = devfn;
> +		info->segment = iommu->segment;
> +	} else {
> +		struct pci_dev *pdev = to_pci_dev(dev);
> +
> +		info->bus = pdev->bus->number;
> +		info->devfn = pdev->devfn;
> +		info->segment = pci_domain_nr(pdev->bus);
> +	}
> +
>   	info->ats_supported = info->pasid_supported = info->pri_supported = 0;
>   	info->ats_enabled = info->pasid_enabled = info->pri_enabled = 0;
>   	info->ats_qdep = 0;
> @@ -2561,7 +2571,8 @@ static struct dmar_domain *dmar_insert_one_dev_info(struct intel_iommu *iommu,
>   
>   	if (!found) {
>   		struct device_domain_info *info2;
> -		info2 = dmar_search_domain_by_dev_info(iommu->segment, bus, devfn);
> +		info2 = dmar_search_domain_by_dev_info(info->segment, info->bus,
> +						       info->devfn);
>   		if (info2) {
>   			found      = info2->domain;
>   			info2->dev = dev;
> diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h
> index 21633ce..4100bd2 100644
> --- a/include/linux/intel-iommu.h
> +++ b/include/linux/intel-iommu.h
> @@ -609,6 +609,7 @@ struct device_domain_info {
>   	struct list_head auxiliary_domains; /* auxiliary domains
>   					     * attached to this device
>   					     */
> +	u32 segment;		/* PCI segment number */
>   	u8 bus;			/* PCI bus number */
>   	u8 devfn;		/* PCI devfn number */
>   	u16 pfsid;		/* SRIOV physical function source ID */
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1 2/3] iommu/vt-d: Allocate domain info for real DMA sub-devices

[Lu Baolu]

On 2020/5/28 15:01, Lu Baolu wrote:
> On 2020/5/28 0:56, Jon Derrick wrote:
>> Sub-devices of a real DMA device might exist on a separate segment than
>> the real DMA device and its IOMMU. These devices should still have a
>> valid device_domain_info, but the current dma alias model won't
>> allocate info for the subdevice.
>>
>> This patch adds a segment member to struct device_domain_info and uses
>> the sub-device's BDF so that these sub-devices won't alias to other
>> devices.
>>
>> Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>
> 
> Acked-by: Lu Baolu <baolu.lu@linux.intel.com>

And a fix tag, so that it could be picked up for v5.6+ stable kernel.

Fixes: 2b0140c69637e ("iommu/vt-d: Use pci_real_dma_dev() for mapping")

Best regards,
baolu
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v1 3/3] iommu/vt-d: Remove real DMA lookup in find_domain

[Jon Derrick]

By removing the real DMA indirection in find_domain(), we can allow
sub-devices of a real DMA device to have their own valid
device_domain_info. The dmar lookup and context entry removal paths have
been fixed to account for sub-devices.

Fixes: 2b0140c69637 ("iommu/vt-d: Use pci_real_dma_dev() for mapping")
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=207575
Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>
---
 drivers/iommu/intel-iommu.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 6d39b9b..5767882 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -2436,9 +2436,6 @@ struct dmar_domain *find_domain(struct device *dev)
 	if (unlikely(attach_deferred(dev) || iommu_dummy(dev)))
 		return NULL;
 
-	if (dev_is_pci(dev))
-		dev = &pci_real_dma_dev(to_pci_dev(dev))->dev;
-
 	/* No lock here, assumes no domain exit in normal case */
 	info = get_domain_info(dev);
 	if (likely(info))
-- 
1.8.3.1

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1 3/3] iommu/vt-d: Remove real DMA lookup in find_domain

[Lu Baolu]

On 2020/5/28 0:56, Jon Derrick wrote:
> By removing the real DMA indirection in find_domain(), we can allow
> sub-devices of a real DMA device to have their own valid
> device_domain_info. The dmar lookup and context entry removal paths have
> been fixed to account for sub-devices.
> 
> Fixes: 2b0140c69637 ("iommu/vt-d: Use pci_real_dma_dev() for mapping")
> Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=207575
> Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>

Acked-by: Lu Baolu <baolu.lu@linux.intel.com>

Best regards,
baolu

> ---
>   drivers/iommu/intel-iommu.c | 3 ---
>   1 file changed, 3 deletions(-)
> 
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index 6d39b9b..5767882 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -2436,9 +2436,6 @@ struct dmar_domain *find_domain(struct device *dev)
>   	if (unlikely(attach_deferred(dev) || iommu_dummy(dev)))
>   		return NULL;
>   
> -	if (dev_is_pci(dev))
> -		dev = &pci_real_dma_dev(to_pci_dev(dev))->dev;
> -
>   	/* No lock here, assumes no domain exit in normal case */
>   	info = get_domain_info(dev);
>   	if (likely(info))
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1 0/3] iommu/vt-d: real DMA sub-device info allocation

[Joerg Roedel]

On Wed, May 27, 2020 at 10:56:14AM -0600, Jon Derrick wrote:
> Jon Derrick (3):
>   iommu/vt-d: Only clear real DMA device's context entries
>   iommu/vt-d: Allocate domain info for real DMA sub-devices
>   iommu/vt-d: Remove real DMA lookup in find_domain
> 
>  drivers/iommu/intel-iommu.c | 31 +++++++++++++++++++++++--------
>  include/linux/intel-iommu.h |  1 +
>  2 files changed, 24 insertions(+), 8 deletions(-)

Applied, thanks.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu