[PATCH 0/3] MSI: Track device proxying when allocating MSIs

[PATCH 0/3] MSI: Track device proxying when allocating MSIs

[Marc Zyngier]

This series aims to plug a gap found with John's "managed interrupts
for platform devices" series.

We have a couple of cases where we allocate MSIs for a device, but the
endpoint generating the actual interrupts (let's call it the "logical
device") isn't the one being seen generating the MSIs (the "effective
device"):

- PCI devices behind a non transparent bridge
- Devices attached to a wire-to-MSI bridge

For interrupt infrastructures such as the ITS (where the effective
device ID is part of the translation), it is important decouple the
lifetime of the effective device mapping from the interrupt allocation
on behalf of the logical device. Yes, I have a headache too.

To achieve this split, let's tag the such allocations with a new flag
that convey the fact that the effective device isn't the logical
one. This mostly affects the platform-MSI subsystem.

Subsequent patches add handling of this new flag in the GICv3 ITS
(though there may be scope for something more generic in the case of
the last patch).

Marc Zyngier (3):
  platform-msi: Track shared domain allocation
  irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy
    device
  irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI
    bridge

 drivers/base/platform-msi.c              |  7 +++++++
 drivers/irqchip/irq-gic-v3-its-pci-msi.c | 11 ++++++++---
 drivers/irqchip/irq-gic-v3-its.c         |  3 +++
 include/asm-generic/msi.h                |  4 ++++
 4 files changed, 22 insertions(+), 3 deletions(-)

-- 
2.28.0

[PATCH 1/3] platform-msi: Track shared domain allocation

[Marc Zyngier]

We have two flavours of platform-MSI:

- MSIs generated by devices for themselves (the usual case)

- MSIs generated on behalf of other devices, as the generating
  device is some form of bridge (either a wire-to-MSI bridge,
  or even a non-transparent PCI bridge that repaints the PCI
  requester ID).

In the latter case, the underlying interrupt architecture may need
to track this in order to keep the mapping alive even when no MSI
are currently being generated.

Add a set of flags to the generic msi_alloc_info_t structure, as
well as the MSI_ALLOC_FLAGS_PROXY_DEVICE flag that will get
advertized by the platform-MSI code when allocating an irqdomain
for a device.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/base/platform-msi.c | 7 +++++++
 include/asm-generic/msi.h   | 4 ++++
 2 files changed, 11 insertions(+)

diff --git a/drivers/base/platform-msi.c b/drivers/base/platform-msi.c
index c4a17e5edf8b..2c1e2e0c1a59 100644
--- a/drivers/base/platform-msi.c
+++ b/drivers/base/platform-msi.c
@@ -59,9 +59,15 @@ static int platform_msi_init(struct irq_domain *domain,
 	return irq_domain_set_hwirq_and_chip(domain, virq, hwirq,
 					     info->chip, info->chip_data);
 }
+
+static void platform_msi_set_proxy_dev(msi_alloc_info_t *arg)
+{
+	arg->flags |= MSI_ALLOC_FLAGS_PROXY_DEVICE;
+}
 #else
 #define platform_msi_set_desc		NULL
 #define platform_msi_init		NULL
+#define platform_msi_set_proxy_dev(x)	do {} while(0)
 #endif
 
 static void platform_msi_update_dom_ops(struct msi_domain_info *info)
@@ -343,6 +349,7 @@ __platform_msi_create_device_domain(struct device *dev,
 	if (!domain)
 		goto free_priv;
 
+	platform_msi_set_proxy_dev(&data->arg);
 	err = msi_domain_prepare_irqs(domain->parent, dev, nvec, &data->arg);
 	if (err)
 		goto free_domain;
diff --git a/include/asm-generic/msi.h b/include/asm-generic/msi.h
index e6795f088bdd..1010e74cb8e0 100644
--- a/include/asm-generic/msi.h
+++ b/include/asm-generic/msi.h
@@ -22,12 +22,16 @@ struct msi_desc;
 typedef struct msi_alloc_info {
 	struct msi_desc			*desc;
 	irq_hw_number_t			hwirq;
+	unsigned long			flags;
 	union {
 		unsigned long		ul;
 		void			*ptr;
 	} scratchpad[NUM_MSI_ALLOC_SCRATCHPAD_REGS];
 } msi_alloc_info_t;
 
+/* Device generating MSIs is proxying for another device */
+#define MSI_ALLOC_FLAGS_PROXY_DEVICE	(1UL << 0)
+
 #define GENERIC_MSI_DOMAIN_OPS		1
 
 #endif
-- 
2.28.0

[PATCH 2/3] irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy device

[Marc Zyngier]

The ITS already has some notion of "shared" devices. Let's map the
MSI_ALLOC_FLAGS_PROXY_DEVICE flag onto this internal property.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3-its.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 0fec31931e11..b99a8ec6585e 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3488,6 +3488,9 @@ static int its_msi_prepare(struct irq_domain *domain, struct device *dev,
 		goto out;
 	}
 
+	if (info->flags & MSI_ALLOC_FLAGS_PROXY_DEVICE)
+		its_dev->shared = true;
+
 	pr_debug("ITT %d entries, %d bits\n", nvec, ilog2(nvec));
 out:
 	mutex_unlock(&its->dev_alloc_lock);
-- 
2.28.0

[PATCH 3/3] irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI bridge

[Marc Zyngier]

An aliasing PCI bridge is another case where we should flag the
corresponding allocation as "proxied", as MSIs are coming with
the bridge's RID, and not the originating device's.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3-its-pci-msi.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its-pci-msi.c b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
index 87711e0f8014..ad2810c017ed 100644
--- a/drivers/irqchip/irq-gic-v3-its-pci-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
@@ -67,11 +67,16 @@ static int its_pci_msi_prepare(struct irq_domain *domain, struct device *dev,
 	/*
 	 * If pdev is downstream of any aliasing bridges, take an upper
 	 * bound of how many other vectors could map to the same DevID.
+	 * Also tell the ITS that the signalling will come from a proxy
+	 * device, and that special allocation rules apply.
 	 */
 	pci_for_each_dma_alias(pdev, its_get_pci_alias, &alias_dev);
-	if (alias_dev != pdev && alias_dev->subordinate)
-		pci_walk_bus(alias_dev->subordinate, its_pci_msi_vec_count,
-			     &alias_count);
+	if (alias_dev != pdev) {
+		if (alias_dev->subordinate)
+			pci_walk_bus(alias_dev->subordinate,
+				     its_pci_msi_vec_count, &alias_count);
+		info->flags |= MSI_ALLOC_FLAGS_PROXY_DEVICE;
+	}
 
 	/* ITS specific DeviceID, as the core ITS ignores dev. */
 	info->scratchpad[0].ul = pci_msi_domain_get_msi_rid(domain, pdev);
-- 
2.28.0

Re: [PATCH 0/3] MSI: Track device proxying when allocating MSIs

[John Garry]

On 29/11/2020 13:52, Marc Zyngier wrote:
> This series aims to plug a gap found with John's "managed interrupts
> for platform devices" series.
> 
> We have a couple of cases where we allocate MSIs for a device, but the
> endpoint generating the actual interrupts (let's call it the "logical
> device") isn't the one being seen generating the MSIs (the "effective
> device"):
> 
> - PCI devices behind a non transparent bridge
> - Devices attached to a wire-to-MSI bridge
> 
> For interrupt infrastructures such as the ITS (where the effective
> device ID is part of the translation), it is important decouple the
> lifetime of the effective device mapping from the interrupt allocation
> on behalf of the logical device. Yes, I have a headache too.
> 
> To achieve this split, let's tag the such allocations with a new flag
> that convey the fact that the effective device isn't the logical
> one. This mostly affects the platform-MSI subsystem.
> 
> Subsequent patches add handling of this new flag in the GICv3 ITS
> (though there may be scope for something more generic in the case of
> the last patch).

Hi Marc,

Tested-by: John Garry <john.garry@huawei.com>

Thanks!

> 
> Marc Zyngier (3):
>    platform-msi: Track shared domain allocation
>    irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy
>      device
>    irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI
>      bridge
> 
>   drivers/base/platform-msi.c              |  7 +++++++
>   drivers/irqchip/irq-gic-v3-its-pci-msi.c | 11 ++++++++---
>   drivers/irqchip/irq-gic-v3-its.c         |  3 +++
>   include/asm-generic/msi.h                |  4 ++++
>   4 files changed, 22 insertions(+), 3 deletions(-)
> 

[irqchip: irq/irqchip-next] irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy device

[irqchip-bot for Marc Zyngier]

The following commit has been merged into the irq/irqchip-next branch of irqchip:

Commit-ID:     5fe71d271df8c05e1060c0184764eba18b17a96f
Gitweb:        https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/5fe71d271df8c05e1060c0184764eba18b17a96f
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Sun, 29 Nov 2020 13:52:07 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Fri, 11 Dec 2020 14:47:50 

irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy device

The ITS already has some notion of "shared" devices. Let's map the
MSI_ALLOC_FLAGS_PROXY_DEVICE flag onto this internal property.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Tested-by: John Garry <john.garry@huawei.com>
Link: https://lore.kernel.org/r/20201129135208.680293-3-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index d74ef41..c951ad2 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3487,6 +3487,9 @@ static int its_msi_prepare(struct irq_domain *domain, struct device *dev,
 		goto out;
 	}
 
+	if (info->flags & MSI_ALLOC_FLAGS_PROXY_DEVICE)
+		its_dev->shared = true;
+
 	pr_debug("ITT %d entries, %d bits\n", nvec, ilog2(nvec));
 out:
 	mutex_unlock(&its->dev_alloc_lock);

[irqchip: irq/irqchip-next] irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI bridge

[irqchip-bot for Marc Zyngier]

The following commit has been merged into the irq/irqchip-next branch of irqchip:

Commit-ID:     34dd263fce3114147f21698f8e55e05b9e8185bd
Gitweb:        https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/34dd263fce3114147f21698f8e55e05b9e8185bd
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Sun, 29 Nov 2020 13:52:08 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Fri, 11 Dec 2020 14:47:50 

irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI bridge

An aliasing PCI bridge is another case where we should flag the
corresponding allocation as "proxied", as MSIs are coming with
the bridge's RID, and not the originating device's.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Tested-by: John Garry <john.garry@huawei.com>
Link: https://lore.kernel.org/r/20201129135208.680293-4-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its-pci-msi.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its-pci-msi.c b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
index 87711e0..ad2810c 100644
--- a/drivers/irqchip/irq-gic-v3-its-pci-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
@@ -67,11 +67,16 @@ static int its_pci_msi_prepare(struct irq_domain *domain, struct device *dev,
 	/*
 	 * If pdev is downstream of any aliasing bridges, take an upper
 	 * bound of how many other vectors could map to the same DevID.
+	 * Also tell the ITS that the signalling will come from a proxy
+	 * device, and that special allocation rules apply.
 	 */
 	pci_for_each_dma_alias(pdev, its_get_pci_alias, &alias_dev);
-	if (alias_dev != pdev && alias_dev->subordinate)
-		pci_walk_bus(alias_dev->subordinate, its_pci_msi_vec_count,
-			     &alias_count);
+	if (alias_dev != pdev) {
+		if (alias_dev->subordinate)
+			pci_walk_bus(alias_dev->subordinate,
+				     its_pci_msi_vec_count, &alias_count);
+		info->flags |= MSI_ALLOC_FLAGS_PROXY_DEVICE;
+	}
 
 	/* ITS specific DeviceID, as the core ITS ignores dev. */
 	info->scratchpad[0].ul = pci_msi_domain_get_msi_rid(domain, pdev);

[irqchip: irq/irqchip-next] platform-msi: Track shared domain allocation

[irqchip-bot for Marc Zyngier]

The following commit has been merged into the irq/irqchip-next branch of irqchip:

Commit-ID:     91f90daa4fb2b77db7aa25ef2e0206f2e3962665
Gitweb:        https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/91f90daa4fb2b77db7aa25ef2e0206f2e3962665
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Sun, 29 Nov 2020 13:52:06 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Fri, 11 Dec 2020 14:47:50 

platform-msi: Track shared domain allocation

We have two flavours of platform-MSI:

- MSIs generated by devices for themselves (the usual case)

- MSIs generated on behalf of other devices, as the generating
  device is some form of bridge (either a wire-to-MSI bridge,
  or even a non-transparent PCI bridge that repaints the PCI
  requester ID).

In the latter case, the underlying interrupt architecture may need
to track this in order to keep the mapping alive even when no MSI
are currently being generated.

Add a set of flags to the generic msi_alloc_info_t structure, as
well as the MSI_ALLOC_FLAGS_PROXY_DEVICE flag that will get
advertized by the platform-MSI code when allocating an irqdomain
for a device.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Tested-by: John Garry <john.garry@huawei.com>
Link: https://lore.kernel.org/r/20201129135208.680293-2-maz@kernel.org
---
 drivers/base/platform-msi.c | 7 +++++++
 include/asm-generic/msi.h   | 4 ++++
 2 files changed, 11 insertions(+)

diff --git a/drivers/base/platform-msi.c b/drivers/base/platform-msi.c
index c4a17e5..2c1e2e0 100644
--- a/drivers/base/platform-msi.c
+++ b/drivers/base/platform-msi.c
@@ -59,9 +59,15 @@ static int platform_msi_init(struct irq_domain *domain,
 	return irq_domain_set_hwirq_and_chip(domain, virq, hwirq,
 					     info->chip, info->chip_data);
 }
+
+static void platform_msi_set_proxy_dev(msi_alloc_info_t *arg)
+{
+	arg->flags |= MSI_ALLOC_FLAGS_PROXY_DEVICE;
+}
 #else
 #define platform_msi_set_desc		NULL
 #define platform_msi_init		NULL
+#define platform_msi_set_proxy_dev(x)	do {} while(0)
 #endif
 
 static void platform_msi_update_dom_ops(struct msi_domain_info *info)
@@ -343,6 +349,7 @@ __platform_msi_create_device_domain(struct device *dev,
 	if (!domain)
 		goto free_priv;
 
+	platform_msi_set_proxy_dev(&data->arg);
 	err = msi_domain_prepare_irqs(domain->parent, dev, nvec, &data->arg);
 	if (err)
 		goto free_domain;
diff --git a/include/asm-generic/msi.h b/include/asm-generic/msi.h
index e6795f0..1010e74 100644
--- a/include/asm-generic/msi.h
+++ b/include/asm-generic/msi.h
@@ -22,12 +22,16 @@ struct msi_desc;
 typedef struct msi_alloc_info {
 	struct msi_desc			*desc;
 	irq_hw_number_t			hwirq;
+	unsigned long			flags;
 	union {
 		unsigned long		ul;
 		void			*ptr;
 	} scratchpad[NUM_MSI_ALLOC_SCRATCHPAD_REGS];
 } msi_alloc_info_t;
 
+/* Device generating MSIs is proxying for another device */
+#define MSI_ALLOC_FLAGS_PROXY_DEVICE	(1UL << 0)
+
 #define GENERIC_MSI_DOMAIN_OPS		1
 
 #endif