[PATCH 0/6] Implement generic IPI support mechanism

[PATCH 0/6] Implement generic IPI support mechanism

[Qais Yousef]

This RFC series attempts to implement a generic IPI layer for reserving and sending IPI.

It is based on the discussion in this link

	https://lkml.org/lkml/2015/8/26/713

This series deals with points #1 and #2 only. Since I'm not the irq expert, I'm hoping this
series will give me early feedback and drive the discussion further about any potential
tricky points.

I tried to keep changes clean and small, but since this is just an RFC I might have missed
few things.

Thomas I hope I didn't stray far from what you had in mind :-)

My only testing so far is having SMP linux booting.

Qais Yousef (6):
  irqdomain: add new IRQ_DOMAIN_FLAGS_IPI
  irqdomain: add a new send_ipi() to irq_domain_ops
  irqdomain: add struct irq_hwcfg and helper functions
  irq: add a new generic IPI handling code to irq core
  irqchip: mips-gic: add a IPI hierarchy domain
  irqchip: mips-gic: use the new generic IPI API

 arch/mips/kernel/smp-gic.c       |  37 ++--
 drivers/irqchip/Kconfig          |   1 +
 drivers/irqchip/irq-mips-gic.c   | 189 ++++++++++++++++++---
 include/linux/irqchip/mips-gic.h |   3 +-
 include/linux/irqdomain.h        |  67 ++++++++
 kernel/irq/irqdomain.c           | 352 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 601 insertions(+), 48 deletions(-)

-- 
2.1.0

[PATCH 1/6] irqdomain: add new IRQ_DOMAIN_FLAGS_IPI

[Qais Yousef]

This flag will be used to identify an IPI domain.

The idea is to reuse the alloc functions to allocate an IPI if it's set.

We could alternatively add a separate new alloc_ipi() function to struct
domain_ops.

Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
---
 include/linux/irqdomain.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index d3ca79236fb0..9b3dc6c2a3cc 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -153,6 +153,9 @@ enum {
 	/* Core calls alloc/free recursive through the domain hierarchy. */
 	IRQ_DOMAIN_FLAG_AUTO_RECURSIVE	= (1 << 1),
 
+	/* Irq domain is an IPI domain */
+	IRQ_DOMAIN_FLAG_IPI		= (1 << 2),
+
 	/*
 	 * Flags starting from IRQ_DOMAIN_FLAG_NONCORE are reserved
 	 * for implementation specific purposes and ignored by the
@@ -326,6 +329,11 @@ static inline bool irq_domain_is_hierarchy(struct irq_domain *domain)
 {
 	return domain->flags & IRQ_DOMAIN_FLAG_HIERARCHY;
 }
+
+static inline bool irq_domain_is_ipi(struct irq_domain *domain)
+{
+	return domain->flags & IRQ_DOMAIN_FLAG_IPI;
+}
 #else	/* CONFIG_IRQ_DOMAIN_HIERARCHY */
 static inline void irq_domain_activate_irq(struct irq_data *data) { }
 static inline void irq_domain_deactivate_irq(struct irq_data *data) { }
@@ -339,6 +347,11 @@ static inline bool irq_domain_is_hierarchy(struct irq_domain *domain)
 {
 	return false;
 }
+
+static inline bool irq_domain_is_ipi(struct irq_domain *domain)
+{
+	return false;
+}
 #endif	/* CONFIG_IRQ_DOMAIN_HIERARCHY */
 
 #else /* CONFIG_IRQ_DOMAIN */
-- 
2.1.0

[PATCH 2/6] irqdomain: add a new send_ipi() to irq_domain_ops

[Qais Yousef]

For generic ipi core to use. It takes hwirq as its sole argument.
Hopefully this is generic enough? Should we pass something more abstract?

Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
---
 include/linux/irqdomain.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index 9b3dc6c2a3cc..cef9e6158be0 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -92,6 +92,7 @@ struct irq_domain_ops {
 	void (*activate)(struct irq_domain *d, struct irq_data *irq_data);
 	void (*deactivate)(struct irq_domain *d, struct irq_data *irq_data);
 #endif
+	void (*send_ipi)(irq_hw_number_t hwirq);
 };
 
 extern struct irq_domain_ops irq_generic_chip_ops;
-- 
2.1.0

[PATCH 3/6] irqdomain: add struct irq_hwcfg and helper functions

[Qais Yousef]

struct irq_hwcfg is used to describe hardware IPI setup.

>From ThomasG suggestion, there's generic parts and a union of hardware
specific part. Assuming that virq and hwirq are enough to describe different
sort of hardwares as so far they are all that was needed, I'm hoping this
implementation will render the need for a platform specific part unnecessary.

The idea is for the irqdomain alloc function to take a hwirq from the available
list, do the necessary mapping to CPU(s), setting ipi_hwirq cpumask, and then
adding this hwirq to the mapped_hwirq list in ipi_virq struct.

Hopefully the code and documentation is self explanatory.

Next commit implements the actual reserve, destroy, and send IPI using these functions.

Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
---
 include/linux/irqdomain.h |  49 ++++++++++++
 kernel/irq/irqdomain.c    | 193 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 242 insertions(+)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index cef9e6158be0..76a0e7aaa8df 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -100,6 +100,45 @@ extern struct irq_domain_ops irq_generic_chip_ops;
 struct irq_domain_chip_generic;
 
 /**
+ * struct ipi_hwirq - IPI hwirq information object
+ * @link: Element in struct ipi_hwirq list
+ * @hwirq: hardware irq value
+ * @cpumask: cpumask where this hwirq is mapped to
+ */
+struct ipi_hwirq {
+	struct list_head link;
+	irq_hw_number_t hwirq;
+	struct cpumask cpumask;
+};
+
+/**
+ * struct ipi_virq - IPI virq information object
+ * @link: Element in struct ipi_virq list
+ * @virq: alocated linux irq number
+ * @cpumask: cpumask where this virq can send IPIs
+ * @devid: devid of device owning this virq
+ * @mapped_hwirq: list of ipi_hwirq that this virq is mapped to
+ */
+struct ipi_virq {
+	struct list_head link;
+	unsigned int virq;
+	struct cpumask cpumask;
+	void *devid;
+
+	struct list_head mapped_hwirq;
+};
+
+/**
+ * struct irq_hwcfg - IPI hardware allocation and translation object
+ * @available_ipis: a list of hwirq that can be allocated for IPIs
+ * @alloced_virqs: a list of virqs that were allocated and their mapping
+ */
+struct irq_hwcfg {
+	struct list_head available_ipis;
+	struct list_head alloced_virqs;
+};
+
+/**
  * struct irq_domain - Hardware interrupt number translation object
  * @link: Element in global irq_domain list.
  * @name: Name of interrupt domain
@@ -137,6 +176,7 @@ struct irq_domain {
 #ifdef	CONFIG_IRQ_DOMAIN_HIERARCHY
 	struct irq_domain *parent;
 #endif
+	struct irq_hwcfg ipi_hwcfg;
 
 	/* reverse map data. The linear map gets appended to the irq_domain */
 	irq_hw_number_t hwirq_max;
@@ -281,6 +321,15 @@ int irq_domain_xlate_onetwocell(struct irq_domain *d, struct device_node *ctrlr,
 			const u32 *intspec, unsigned int intsize,
 			irq_hw_number_t *out_hwirq, unsigned int *out_type);
 
+/* irq_hwcfg functions */
+extern int irq_domain_put_ipi_hwirq(struct irq_domain *d, struct ipi_hwirq *h);
+extern struct ipi_hwirq *irq_domain_get_ipi_hwirq(struct irq_domain *d);
+extern struct ipi_virq *irq_domain_find_ipi_virq(struct irq_domain *d,
+						unsigned int virq);
+extern int irq_domain_ipi_virq_add_hwirq(struct ipi_virq *v,
+					 struct ipi_hwirq *h);
+extern struct ipi_hwirq *irq_domain_ipi_virq_rm_hwirq(struct ipi_virq *v);
+
 /* V2 interfaces to support hierarchy IRQ domains. */
 extern struct irq_data *irq_domain_get_irq_data(struct irq_domain *domain,
 						unsigned int virq);
diff --git a/kernel/irq/irqdomain.c b/kernel/irq/irqdomain.c
index dc9d27c0c158..6d911fc8fa52 100644
--- a/kernel/irq/irqdomain.c
+++ b/kernel/irq/irqdomain.c
@@ -61,6 +61,8 @@ struct irq_domain *__irq_domain_add(struct device_node *of_node, int size,
 	domain->revmap_size = size;
 	domain->revmap_direct_max_irq = direct_max;
 	irq_domain_check_hierarchy(domain);
+	INIT_LIST_HEAD(&domain->ipi_hwcfg.available_ipis);
+	INIT_LIST_HEAD(&domain->ipi_hwcfg.alloced_virqs);
 
 	mutex_lock(&irq_domain_mutex);
 	list_add(&domain->link, &irq_domain_list);
@@ -749,6 +751,197 @@ static int irq_domain_alloc_descs(int virq, unsigned int cnt,
 	return virq;
 }
 
+/**
+ * irq_domain_put_ipi_hwirq() - return an IPI hwirq to the system
+ * @d: IPI domain to return the hwirq to
+ * @h: ipi_hwirq struct to be returned
+ *
+ * The domain must have a set of hwirqs to be used for IPI which are allocated
+ * by the underlying driver. This function adds this hwirq so it can be used
+ * to reserve an IPI later.
+ *
+ * returns 0 on success and negative errno on failure
+ */
+int irq_domain_put_ipi_hwirq(struct irq_domain *d, struct ipi_hwirq *h)
+{
+	struct list_head *head = &d->ipi_hwcfg.available_ipis;
+
+	cpumask_clear(&h->cpumask);
+	list_add_tail(&h->link, head);
+
+	return 0;
+}
+
+/**
+ * irq_domain_get_ipi_hwirq() - obtain an IPI hwirq from the system
+ * @d: IPI domain to obtain the hwirq from
+ *
+ * The domain contains a set of hwirqs that can be used for IPIs. This function
+ * obtains one from the available set of hwirqs.
+ *
+ * returns a pointer to ipi_hwirq struct on success and NULL on failure
+ */
+struct ipi_hwirq *irq_domain_get_ipi_hwirq(struct irq_domain *d)
+{
+	struct list_head *head = &d->ipi_hwcfg.available_ipis;
+	struct ipi_hwirq *h;
+
+	if (list_empty(head))
+		return NULL;
+
+	h = list_first_entry(head, struct ipi_hwirq, link);
+	list_del(&h->link);
+
+	return h;
+}
+
+/**
+ * irq_domain_add_ipi_virq() - add an allocated virq struct to the IPI domain
+ * @d: IPI domain to act on
+ * @v: ipi_virq struct to add
+ *
+ * The domain keeps a list of allocated (reserved) IPIs. This functions adds
+ * a newly reserved virq to this list.
+ *
+ * returns 0 on success and negative errno on failure
+ */
+static int irq_domain_add_ipi_virq(struct irq_domain *d, struct ipi_virq *v)
+{
+	struct list_head *head = &d->ipi_hwcfg.alloced_virqs;
+
+	list_add_tail(&v->link, head);
+
+	return 0;
+}
+
+/**
+ * irq_domain_add_ipi_virq() - remove an allocated virq struct from the IPI domain
+ * @d: IPI domain to act on
+ * @virq: virq to remove
+ *
+ * The domain keeps a list of allocated (reserved) IPIs. This functions removes
+ * a previously added virq from this list.
+ *
+ * returns a pointer to ipi_virq struct on success and NULL on failure
+ */
+static struct ipi_virq *irq_domain_rm_ipi_virq(struct irq_domain *d,
+					       unsigned int virq)
+{
+	struct ipi_virq *v;
+
+	v = irq_domain_find_ipi_virq(d, virq);
+	if (!v)
+		return NULL;
+
+	list_del(&v->link);
+
+	return v;
+}
+
+/**
+ * irq_domain_find_ipi_virq() - search the IPI domain for a virq
+ * @d: IPI domain to act on
+ * @virq: virq number to find
+ *
+ * The domain keeps a list of allocated (reserved) IPIs. This functions searches
+ * this list for a struct ipi_virq with a matching virq number
+ *
+ * returns a pointer to ipi_virq struct on success and NULL on failure
+ */
+struct ipi_virq *irq_domain_find_ipi_virq(struct irq_domain *d,
+					  unsigned int virq)
+{
+	struct list_head *head = &d->ipi_hwcfg.alloced_virqs;
+	struct ipi_virq *v;
+
+	list_for_each_entry(v, head, link) {
+		if (v->virq == virq)
+			return v;
+	}
+
+	return NULL;
+}
+
+/**
+ * irq_domain_alloc_ipi_virq() - alloc a ipi_virq struct
+ * @virq: virq number of the new ipi_virq struct
+ * @cpumask: cpumask associated with the virq
+ * @devid: devid whose this virq belong too
+ *
+ * Allocates a struct ipi_virq and initialise it
+ *
+ * returns a pointer to ipi_virq struct on success and NULL on failure
+ */
+static struct ipi_virq *irq_domain_alloc_ipi_virq(unsigned int virq,
+				struct cpumask *cpumask, void *devid)
+{
+	struct ipi_virq *v;
+
+	v = kzalloc(sizeof(struct ipi_virq), GFP_KERNEL);
+	if (!v)
+		return NULL;
+
+	v->virq = virq;
+	v->devid = devid;
+	cpumask_copy(&v->cpumask, cpumask);
+	INIT_LIST_HEAD(&v->mapped_hwirq);
+
+	return v;
+}
+
+/**
+ * irq_domain_alloc_ipi_virq() - free a ipi_virq struct
+ * @v: pointer to the ipi_virq struct to be freed
+ *
+ * Frees the memory assiciated with ipi_virq struct
+ */
+static void irq_domain_free_ipi_virq(struct ipi_virq *v)
+{
+	kfree(v);
+}
+
+/**
+ * irq_domain_ipi_virq_add_hwirq() - associated hwirq with a virq
+ * @v: pointer to the ipi_virq struct
+ * @h: pointer to the ipi_hwirq struct
+ *
+ * Associate a hwirq with a virq by adding it to its mapping list
+ *
+ * returns 0 on success and a negative errno on failure
+ */
+int irq_domain_ipi_virq_add_hwirq(struct ipi_virq *v,
+				  struct ipi_hwirq *h)
+{
+	struct list_head *head = &v->mapped_hwirq;
+
+	list_add_tail(&h->link, head);
+
+	return 0;
+}
+
+/**
+ * irq_domain_ipi_virq_rm_hwirq() - remove hwirq from a virq
+ * @v: pointer to the ipi_virq struct
+ *
+ * Removes a hwirq from the mapped hwirq list and returns it.
+ *
+ * returns a pointer to an associated ipi_hwirq struct on success or a NULL on
+ * failure
+ */
+struct ipi_hwirq *irq_domain_ipi_virq_rm_hwirq(struct ipi_virq *v)
+{
+	struct list_head *head = &v->mapped_hwirq;
+	struct ipi_hwirq *h;
+
+	if (list_empty(head))
+		return NULL;
+
+	h = list_first_entry(head, struct ipi_hwirq, link);
+	list_del(&h->link);
+
+	return h;
+}
+
 #ifdef	CONFIG_IRQ_DOMAIN_HIERARCHY
 /**
  * irq_domain_add_hierarchy - Add a irqdomain into the hierarchy
-- 
2.1.0

[PATCH 4/6] irq: add a new generic IPI handling code to irq core

[Qais Yousef]

Add a generic mechanism to dynamically allocate an IPI.

With this change the user can call irq_reserve_ipi() to dynamically allocate an
IPI and use the associate virq to send one to 1 or more cpus.

No irq_get_irq_hwcfg() as I hope we can provide an implementation without
hardware specific part. Hopefully I'm not being too optimistic :)

Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
---
 include/linux/irqdomain.h |   4 ++
 kernel/irq/irqdomain.c    | 161 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 164 insertions(+), 1 deletion(-)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index 76a0e7aaa8df..4bdd7b48d205 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -329,6 +329,10 @@ extern struct ipi_virq *irq_domain_find_ipi_virq(struct irq_domain *d,
 extern int irq_domain_ipi_virq_add_hwirq(struct ipi_virq *v,
 					 struct ipi_hwirq *h);
 extern struct ipi_hwirq *irq_domain_ipi_virq_rm_hwirq(struct ipi_virq *v);
+extern unsigned int irq_reserve_ipi(struct irq_domain *domain,
+				const struct cpumask *dest, void *devid);
+extern void irq_destroy_ipi(unsigned int irq, void *devid);
+extern void irq_send_ipi(unsigned int irq, const struct cpumask *dest, void *devid);
 
 /* V2 interfaces to support hierarchy IRQ domains. */
 extern struct irq_data *irq_domain_get_irq_data(struct irq_domain *domain,
diff --git a/kernel/irq/irqdomain.c b/kernel/irq/irqdomain.c
index 6d911fc8fa52..d38d78a994ca 100644
--- a/kernel/irq/irqdomain.c
+++ b/kernel/irq/irqdomain.c
@@ -873,7 +873,7 @@ struct ipi_virq *irq_domain_find_ipi_virq(struct irq_domain *d,
  * returns a pointer to ipi_virq struct on success and NULL on failure
  */
 static struct ipi_virq *irq_domain_alloc_ipi_virq(unsigned int virq,
-				struct cpumask *cpumask, void *devid)
+				const struct cpumask *cpumask, void *devid)
 {
 	struct ipi_virq *v;
 
@@ -942,6 +942,165 @@ struct ipi_hwirq *irq_domain_ipi_virq_rm_hwirq(struct ipi_virq *v)
 	return h;
 }
 
+/**
+ * irq_reserve_ipi() - setup an IPI to destination cpumask
+ * @domain: IPI domain
+ * @dest: cpumask of cpus to receive the IPI
+ * @devid: devid that requested the reservation
+ *
+ * Allocate a virq that can be used to send IPI to any CPU in dest mask.
+ *
+ * On success it'll return linux irq number and 0 on failure
+ */
+unsigned int irq_reserve_ipi(struct irq_domain *domain,
+			     const struct cpumask *dest, void *devid)
+{
+	int virq, ret;
+	unsigned int nr_irqs;
+	struct ipi_virq *v;
+
+	if (domain == NULL)
+		domain = irq_default_domain; /* need a separate ipi_default_domain? */
+
+	if (domain == NULL) {
+		pr_warn("Must provide a valid IPI domain!\n");
+		return 0;
+	}
+
+	if (!irq_domain_is_ipi(domain)) {
+		pr_warn("Not an IPI domain!\n");
+		return 0;
+	}
+
+	if (cpumask_empty(dest)) {
+		pr_warn("Can't reserve IPI due to empty cpumask\n");
+		return 0;
+	}
+
+	/* always allocate a virq per cpu */
+	nr_irqs = cpumask_weight(dest);
+
+	virq = irq_domain_alloc_descs(-1, nr_irqs, 0, NUMA_NO_NODE);
+	if (virq <= 0) {
+		pr_warn("Can't reserve IPI, failed to alloc descs\n");
+		return 0;
+	}
+
+	v = irq_domain_alloc_ipi_virq(virq, dest, devid);
+	if (!v) {
+		pr_warn("Can't reserve IPI, failed to alloc ipi_virq\n");
+		goto free_descs;
+	}
+
+	/* we are reusing hierarchy alloc function, should we create another one? */
+	virq = __irq_domain_alloc_irqs(domain, virq, nr_irqs, NUMA_NO_NODE,
+					v, true);
+	if (virq <= 0) {
+		pr_warn("Can't reserve IPI, failed to alloc irqs\n");
+		goto free_ipi;
+	}
+
+	ret = irq_domain_add_ipi_virq(domain, v);
+	if (ret) {
+		pr_warn("Can't reserve IPI, failed to add ipi_virq\n");
+		goto free_ipi;
+	}
+
+	return virq;
+
+free_ipi:
+	irq_domain_free_ipi_virq(v);
+free_descs:
+	irq_free_descs(virq, nr_irqs);
+	return 0;
+}
+
+/**
+ * irq_destroy_ipi() - unreserve an IPI that was previously allocated
+ * @irq: linux irq number to be destroyed
+ * @devid: devid that reserved the IPI
+ *
+ * Return an IPI allocated with irq_reserve_ipi() to the system.
+ */
+void irq_destroy_ipi(unsigned int irq, void *devid)
+{
+	struct irq_data *irq_data = irq_get_irq_data(irq);
+	struct irq_domain *domain;
+	struct ipi_virq *v;
+
+	if (!irq || !irq_data)
+		return;
+
+	domain = irq_data->domain;
+	if (WARN_ON(domain == NULL))
+		return;
+
+	if (!irq_domain_is_ipi(domain)) {
+		pr_warn("Not an IPI domain!\n");
+		return;
+	}
+
+	v = irq_domain_find_ipi_virq(domain, irq);
+	if (!v)
+		return;
+
+	if (v->devid != devid) {
+		pr_warn("Only the device that allocated the IPI can destroy it\n");
+		return;
+	}
+
+	irq_domain_free_irqs(irq, cpumask_weight(&v->cpumask));
+
+	v = irq_domain_rm_ipi_virq(domain, irq);
+	irq_domain_free_ipi_virq(v);
+}
+
+/**
+ * irq_send_ipi() - send an IPI to target CPU(s)
+ * @irq: linux irq number from irq_reserve_ipi()
+ * @dest: dest CPU(s), must be the same or a subset of the mask past to
+ *	  irq_reserve_ipi()
+ * @devid: devid that reserved the IPI
+ *
+ * Sends an IPI to all cpus in dest mask
+ */
+void irq_send_ipi(unsigned int irq, const struct cpumask *dest, void *devid)
+{
+	struct irq_data *irq_data = irq_get_irq_data(irq);
+	struct irq_domain *domain;
+	struct ipi_virq *v;
+	struct ipi_hwirq *h;
+
+	if (!irq || !irq_data)
+		return;
+
+	domain = irq_data->domain;
+	if (WARN_ON(domain == NULL))
+		return;
+
+	if (!irq_domain_is_ipi(domain)) {
+		pr_warn("Not an IPI domain!\n");
+		return;
+	}
+
+	v = irq_domain_find_ipi_virq(domain, irq);
+	if (!v)
+		return;
+
+	if (v->devid != devid) {
+		pr_warn("Only the device that allocated the IPI can send one\n");
+		return;
+	}
+
+	if (!cpumask_intersects(&v->cpumask, dest))
+		return;
+
+	list_for_each_entry(h, &v->mapped_hwirq, link) {
+		if (cpumask_intersects(&h->cpumask, dest))
+			domain->ops->send_ipi(h->hwirq);
+	}
+}
+
 #ifdef	CONFIG_IRQ_DOMAIN_HIERARCHY
 /**
  * irq_domain_add_hierarchy - Add a irqdomain into the hierarchy
-- 
2.1.0

[PATCH 5/6] irqchip: mips-gic: add a IPI hierarchy domain

[Qais Yousef]

Add a new ipi domain on top of the normal domain.

The only reason this domain needs to be hierarchal is because we need to use
the alloc function to allocate the IPI.

Should we make the gic IPI a completely different irqchip?

Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
---
 drivers/irqchip/Kconfig        |   1 +
 drivers/irqchip/irq-mips-gic.c | 101 +++++++++++++++++++++++++++++++++++++++--
 2 files changed, 97 insertions(+), 5 deletions(-)

diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index 27b52c8729cd..e07593f4860d 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -167,6 +167,7 @@ config KEYSTONE_IRQ
 
 config MIPS_GIC
 	bool
+	select IRQ_DOMAIN_HIERARCHY
 	select MIPS_CM
 
 config INGENIC_IRQ
diff --git a/drivers/irqchip/irq-mips-gic.c b/drivers/irqchip/irq-mips-gic.c
index af2f16bb8a94..14e99ea0f963 100644
--- a/drivers/irqchip/irq-mips-gic.c
+++ b/drivers/irqchip/irq-mips-gic.c
@@ -33,6 +33,7 @@ static void __iomem *gic_base;
 static struct gic_pcpu_mask pcpu_masks[NR_CPUS];
 static DEFINE_SPINLOCK(gic_lock);
 static struct irq_domain *gic_irq_domain;
+static struct irq_domain *gic_ipi_domain;
 static int gic_shared_intrs;
 static int gic_vpes;
 static unsigned int gic_cpu_pin;
@@ -733,7 +734,7 @@ static int gic_local_irq_domain_map(struct irq_domain *d, unsigned int virq,
 }
 
 static int gic_shared_irq_domain_map(struct irq_domain *d, unsigned int virq,
-				     irq_hw_number_t hw)
+				     irq_hw_number_t hw, unsigned int vpe)
 {
 	int intr = GIC_HWIRQ_TO_SHARED(hw);
 	unsigned long flags;
@@ -743,9 +744,8 @@ static int gic_shared_irq_domain_map(struct irq_domain *d, unsigned int virq,
 
 	spin_lock_irqsave(&gic_lock, flags);
 	gic_map_to_pin(intr, gic_cpu_pin);
-	/* Map to VPE 0 by default */
-	gic_map_to_vpe(intr, 0);
-	set_bit(intr, pcpu_masks[0].pcpu_mask);
+	gic_map_to_vpe(intr, vpe);
+	set_bit(intr, pcpu_masks[vpe].pcpu_mask);
 	spin_unlock_irqrestore(&gic_lock, flags);
 
 	return 0;
@@ -756,7 +756,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int virq,
 {
 	if (GIC_HWIRQ_TO_LOCAL(hw) < GIC_NUM_LOCAL_INTRS)
 		return gic_local_irq_domain_map(d, virq, hw);
-	return gic_shared_irq_domain_map(d, virq, hw);
+	return gic_shared_irq_domain_map(d, virq, hw, 0);
 }
 
 static int gic_irq_domain_xlate(struct irq_domain *d, struct device_node *ctrlr,
@@ -783,6 +783,91 @@ static const struct irq_domain_ops gic_irq_domain_ops = {
 	.xlate = gic_irq_domain_xlate,
 };
 
+static int gic_ipi_domain_xlate(struct irq_domain *d, struct device_node *ctrlr,
+				const u32 *intspec, unsigned int intsize,
+				irq_hw_number_t *out_hwirq,
+				unsigned int *out_type)
+{
+	/*
+	 * There's nothing to translate here. hwirq is dynamically allocated and
+	 * the irq type is always edge triggered.
+	 * */
+	*out_hwirq = 0;
+	*out_type = IRQ_TYPE_EDGE_RISING;
+
+	return 0;
+}
+
+static int gic_ipi_domain_alloc(struct irq_domain *d, unsigned int virq,
+				unsigned int nr_irqs, void *arg)
+{
+	int vpe, ret;
+	struct ipi_virq *v = arg;
+	struct ipi_hwirq *h;
+	irq_hw_number_t hwirq;
+
+	for_each_cpu(vpe, &v->cpumask) {
+		/* we need a GIC shared hwirq for each vpe */
+		h = irq_domain_get_ipi_hwirq(d);
+		if (!h) {
+			ret = -EINVAL;
+			goto error;
+		}
+		cpumask_set_cpu(vpe, &h->cpumask);
+
+		/* add this hwirq to the virq mapping list */
+		ret = irq_domain_ipi_virq_add_hwirq(v, h);
+		if (ret)
+			goto error;
+
+		hwirq = GIC_SHARED_TO_HWIRQ(h->hwirq);
+		ret = irq_domain_set_hwirq_and_chip(d, virq + vpe, hwirq,
+						NULL, NULL);
+		if (ret)
+			goto error;
+
+		ret = gic_shared_irq_domain_map(d, virq + vpe, hwirq, vpe);
+		if (ret)
+			goto error;
+	}
+
+	return 0;
+error:
+	while ((h = irq_domain_ipi_virq_rm_hwirq(v))) {
+		for_each_cpu(vpe, &h->cpumask)
+			clear_bit(h->hwirq, pcpu_masks[vpe].pcpu_mask);
+		irq_domain_put_ipi_hwirq(d, h);
+	}
+
+	return ret;
+}
+
+void gic_ipi_domain_free(struct irq_domain *d, unsigned int virq,
+			 unsigned int nr_irqs)
+{
+	struct ipi_hwirq *h;
+	struct ipi_virq *v;
+	int vpe;
+
+	v = irq_domain_find_ipi_virq(d, virq);
+	if (!v)
+		return;
+
+	/* return all allocated hwirqs to the IPI domain */
+	while ((h = irq_domain_ipi_virq_rm_hwirq(v))) {
+		for_each_cpu(vpe, &h->cpumask)
+			clear_bit(h->hwirq, pcpu_masks[vpe].pcpu_mask);
+		irq_domain_put_ipi_hwirq(d, h);
+	}
+}
+
+static struct irq_domain_ops gic_ipi_domain_ops = {
+	.xlate = gic_ipi_domain_xlate,
+	.alloc = gic_ipi_domain_alloc,
+	.free = gic_ipi_domain_free,
+	.send_ipi = gic_send_ipi,
+};
+
 static void __init __gic_init(unsigned long gic_base_addr,
 			      unsigned long gic_addrspace_size,
 			      unsigned int cpu_vec, unsigned int irqbase,
@@ -842,6 +927,12 @@ static void __init __gic_init(unsigned long gic_base_addr,
 	if (!gic_irq_domain)
 		panic("Failed to add GIC IRQ domain");
 
+	gic_ipi_domain = irq_domain_add_hierarchy(gic_irq_domain, IRQ_DOMAIN_FLAG_IPI,
+						  GIC_NUM_LOCAL_INTRS + gic_shared_intrs,
+						  NULL, &gic_ipi_domain_ops, NULL);
+	if (!gic_ipi_domain)
+		panic("Failed to add GIC IPI domain");
+
 	gic_basic_init();
 
 	gic_ipi_init();
-- 
2.1.0

[PATCH 6/6] irqchip: mips-gic: use the new generic IPI API

[Qais Yousef]

Use the new IPI generic functions to implement smp IPIs.

For simplicity for now, I'm reserving the IPIs from the end of the available
hwirqs. But this could easily be changed to get a list of hwirqs to use as IPIs
from platform code or DT.

The implementation is meant as a demonstration of using the new IPI mechanism.
It is less efficient than before because of the added layer and the only
advantage is we have support for generic IPI reservation scheme.

How can we refactor this better?

Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
---
 arch/mips/kernel/smp-gic.c       | 37 ++++++++---------
 drivers/irqchip/irq-mips-gic.c   | 88 ++++++++++++++++++++++++++++++----------
 include/linux/irqchip/mips-gic.h |  3 +-
 3 files changed, 85 insertions(+), 43 deletions(-)

diff --git a/arch/mips/kernel/smp-gic.c b/arch/mips/kernel/smp-gic.c
index 5f0ab5bcd01e..ef9a80df1fcc 100644
--- a/arch/mips/kernel/smp-gic.c
+++ b/arch/mips/kernel/smp-gic.c
@@ -20,45 +20,40 @@
 
 void gic_send_ipi_single(int cpu, unsigned int action)
 {
-	unsigned long flags;
-	unsigned int intr;
-	unsigned int core = cpu_data[cpu].core;
+	gic_send_ipi_mask(cpumask_of(cpu), action);
+}
 
-	pr_debug("CPU%d: %s cpu %d action %u status %08x\n",
-		 smp_processor_id(), __func__, cpu, action, read_c0_status());
+void gic_send_ipi_mask(const struct cpumask *mask, unsigned int action)
+{
+	unsigned long flags;
+	unsigned int core;
+	int cpu;
 
 	local_irq_save(flags);
 
 	switch (action) {
 	case SMP_CALL_FUNCTION:
-		intr = plat_ipi_call_int_xlate(cpu);
+		gic_send_call_ipi(mask);
 		break;
 
 	case SMP_RESCHEDULE_YOURSELF:
-		intr = plat_ipi_resched_int_xlate(cpu);
+		gic_send_resched_ipi(mask);
 		break;
 
 	default:
 		BUG();
 	}
 
-	gic_send_ipi(intr);
-
 	if (mips_cpc_present() && (core != current_cpu_data.core)) {
-		while (!cpumask_test_cpu(cpu, &cpu_coherent_mask)) {
-			mips_cpc_lock_other(core);
-			write_cpc_co_cmd(CPC_Cx_CMD_PWRUP);
-			mips_cpc_unlock_other();
+		for_each_cpu(cpu, mask) {
+			core = cpu_data[cpu].core;
+			while (!cpumask_test_cpu(cpu, &cpu_coherent_mask)) {
+				mips_cpc_lock_other(core);
+				write_cpc_co_cmd(CPC_Cx_CMD_PWRUP);
+				mips_cpc_unlock_other();
+			}
 		}
 	}
 
 	local_irq_restore(flags);
 }
-
-void gic_send_ipi_mask(const struct cpumask *mask, unsigned int action)
-{
-	unsigned int i;
-
-	for_each_cpu(i, mask)
-		gic_send_ipi_single(i, action);
-}
diff --git a/drivers/irqchip/irq-mips-gic.c b/drivers/irqchip/irq-mips-gic.c
index 14e99ea0f963..ff9b79170e2f 100644
--- a/drivers/irqchip/irq-mips-gic.c
+++ b/drivers/irqchip/irq-mips-gic.c
@@ -15,6 +15,7 @@
 #include <linux/irqchip/mips-gic.h>
 #include <linux/of_address.h>
 #include <linux/sched.h>
+#include <linux/slab.h>
 #include <linux/smp.h>
 
 #include <asm/mips-cm.h>
@@ -39,6 +40,7 @@ static int gic_vpes;
 static unsigned int gic_cpu_pin;
 static unsigned int timer_cpu_pin;
 static struct irq_chip gic_level_irq_controller, gic_edge_irq_controller;
+DECLARE_BITMAP(ipi_intrs, GIC_MAX_INTRS);
 
 static void __gic_irq_dispatch(void);
 
@@ -264,7 +266,7 @@ static void gic_bind_eic_interrupt(int irq, int set)
 		  GIC_VPE_EIC_SS(irq), set);
 }
 
-void gic_send_ipi(unsigned int intr)
+static void gic_send_ipi(irq_hw_number_t intr)
 {
 	gic_write(GIC_REG(SHARED, GIC_SH_WEDGE), GIC_SH_WEDGE_SET(intr));
 }
@@ -328,8 +330,14 @@ static void gic_handle_shared_int(bool chained)
 
 	intr = find_first_bit(pending, gic_shared_intrs);
 	while (intr != gic_shared_intrs) {
-		virq = irq_linear_revmap(gic_irq_domain,
-					 GIC_SHARED_TO_HWIRQ(intr));
+		if (test_bit(intr, ipi_intrs)) {
+			virq = irq_linear_revmap(gic_ipi_domain,
+					GIC_SHARED_TO_HWIRQ(intr));
+		} else {
+			virq = irq_linear_revmap(gic_irq_domain,
+					GIC_SHARED_TO_HWIRQ(intr));
+		}
+
 		if (chained)
 			generic_handle_irq(virq);
 		else
@@ -593,37 +601,74 @@ static struct irqaction irq_call = {
 	.name		= "IPI call"
 };
 
-static __init void gic_ipi_init_one(unsigned int intr, int cpu,
+static __init void gic_ipi_init_one(unsigned int virq,
 				    struct irqaction *action)
 {
-	int virq = irq_create_mapping(gic_irq_domain,
-				      GIC_SHARED_TO_HWIRQ(intr));
-	int i;
-
-	gic_map_to_vpe(intr, cpu);
-	for (i = 0; i < NR_CPUS; i++)
-		clear_bit(intr, pcpu_masks[i].pcpu_mask);
-	set_bit(intr, pcpu_masks[cpu].pcpu_mask);
+	int ret;
 
-	irq_set_irq_type(virq, IRQ_TYPE_EDGE_RISING);
+	ret = irq_set_irq_type(virq, IRQ_TYPE_EDGE_RISING);
+	if (ret) {
+		pr_warn("Failed to set IPI irq type at %d\n", virq);
+		return;
+	}
 
 	irq_set_handler(virq, handle_percpu_irq);
-	setup_irq(virq, action);
+
+	ret = setup_irq(virq, action);
+	if (ret)
+		pr_warn("Failed to setup IPI at %d\n", virq);
 }
 
 static __init void gic_ipi_init(void)
 {
-	int i;
+	int cpu, i;
+	struct ipi_hwirq *h;
 
-	/* Use last 2 * NR_CPUS interrupts as IPIs */
-	gic_resched_int_base = gic_shared_intrs - nr_cpu_ids;
-	gic_call_int_base = gic_resched_int_base - nr_cpu_ids;
+	/* We should get a list of hwirqs to reserve from DT or platform code */
+	for (i = 1; i <= 2 * cpumask_weight(cpu_possible_mask); i++) {
+		h = kzalloc(sizeof(struct ipi_hwirq), GFP_KERNEL);
+		if (!h)
+			goto out_kzalloc;
+		h->hwirq = gic_shared_intrs - i;
+		bitmap_set(ipi_intrs, h->hwirq, 1);
+		irq_domain_put_ipi_hwirq(gic_ipi_domain, h);
+	}
+
+	gic_resched_int_base = irq_reserve_ipi(gic_ipi_domain,
+						cpu_possible_mask, NULL);
+	if (!gic_resched_int_base)
+		goto out_kzalloc;
+	gic_call_int_base = irq_reserve_ipi(gic_ipi_domain,
+						cpu_possible_mask, NULL);
+	if (!gic_call_int_base) {
+		goto out_call;
+	}
+
+	for_each_cpu(cpu, cpu_possible_mask) {
+		gic_ipi_init_one(gic_call_int_base + cpu, &irq_call);
+		gic_ipi_init_one(gic_resched_int_base + cpu, &irq_resched);
+	}
+
+	return;
 
-	for (i = 0; i < nr_cpu_ids; i++) {
-		gic_ipi_init_one(gic_call_int_base + i, i, &irq_call);
-		gic_ipi_init_one(gic_resched_int_base + i, i, &irq_resched);
+out_call:
+	irq_destroy_ipi(gic_resched_int_base, NULL);
+out_kzalloc:
+	while ((h = irq_domain_get_ipi_hwirq(gic_ipi_domain))) {
+		bitmap_clear(ipi_intrs, h->hwirq, 1);
+		kfree(h);
 	}
 }
+
+void gic_send_resched_ipi(const struct cpumask *cpumask)
+{
+	irq_send_ipi(gic_resched_int_base, cpumask, NULL);
+}
+
+void gic_send_call_ipi(const struct cpumask *cpumask)
+{
+	irq_send_ipi(gic_call_int_base, cpumask, NULL);
+}
 #else
 static inline void gic_ipi_init(void)
 {
@@ -857,6 +902,7 @@ void gic_ipi_domain_free(struct irq_domain *d, unsigned int virq,
 	while ((h = irq_domain_ipi_virq_rm_hwirq(v))) {
 		for_each_cpu(vpe, &h->cpumask)
 			clear_bit(h->hwirq, pcpu_masks[vpe].pcpu_mask);
+		bitmap_clear(ipi_intrs, h->hwirq, 1);
 		irq_domain_put_ipi_hwirq(d, h);
 	}
 }
diff --git a/include/linux/irqchip/mips-gic.h b/include/linux/irqchip/mips-gic.h
index 4e6861605050..57e794e7da12 100644
--- a/include/linux/irqchip/mips-gic.h
+++ b/include/linux/irqchip/mips-gic.h
@@ -258,7 +258,8 @@ extern void gic_write_compare(cycle_t cnt);
 extern void gic_write_cpu_compare(cycle_t cnt, int cpu);
 extern void gic_start_count(void);
 extern void gic_stop_count(void);
-extern void gic_send_ipi(unsigned int intr);
+extern void gic_send_resched_ipi(const struct cpumask *cpumask);
+extern void gic_send_call_ipi(const struct cpumask *cpumask);
 extern unsigned int plat_ipi_call_int_xlate(unsigned int);
 extern unsigned int plat_ipi_resched_int_xlate(unsigned int);
 extern int gic_get_c0_compare_int(void);
-- 
2.1.0

Re: [PATCH 2/6] irqdomain: add a new send_ipi() to irq_domain_ops

[Jiang Liu]

On 2015/9/23 22:49, Qais Yousef wrote:
> For generic ipi core to use. It takes hwirq as its sole argument.
> Hopefully this is generic enough? Should we pass something more abstract?
> 
> Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
> ---
>  include/linux/irqdomain.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
> index 9b3dc6c2a3cc..cef9e6158be0 100644
> --- a/include/linux/irqdomain.h
> +++ b/include/linux/irqdomain.h
> @@ -92,6 +92,7 @@ struct irq_domain_ops {
>  	void (*activate)(struct irq_domain *d, struct irq_data *irq_data);
>  	void (*deactivate)(struct irq_domain *d, struct irq_data *irq_data);
>  #endif
> +	void (*send_ipi)(irq_hw_number_t hwirq);
Hi Qais,
	Instead of extending the irq_domain_ops, how about extending
irq_chip instead? If we treat IPI as a sort of irq controller, and
irq_chip is used to encapsulate all irq controller related operations,
and irq_domain_ops is mainly used to allocated resources instead of
operating corresponding hardware.
Thanks!
Gerry

>  };
>  
>  extern struct irq_domain_ops irq_generic_chip_ops;
> 

Re: [PATCH 4/6] irq: add a new generic IPI handling code to irq core

[Jiang Liu]

On 2015/9/23 22:49, Qais Yousef wrote:
> Add a generic mechanism to dynamically allocate an IPI.
> 
> With this change the user can call irq_reserve_ipi() to dynamically allocate an
> IPI and use the associate virq to send one to 1 or more cpus.
> 
> No irq_get_irq_hwcfg() as I hope we can provide an implementation without
> hardware specific part. Hopefully I'm not being too optimistic :)
> 
> Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
> ---
>  include/linux/irqdomain.h |   4 ++
>  kernel/irq/irqdomain.c    | 161 +++++++++++++++++++++++++++++++++++++++++++++-
>  2 files changed, 164 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
> index 76a0e7aaa8df..4bdd7b48d205 100644
> --- a/include/linux/irqdomain.h
> +++ b/include/linux/irqdomain.h
> @@ -329,6 +329,10 @@ extern struct ipi_virq *irq_domain_find_ipi_virq(struct irq_domain *d,
>  extern int irq_domain_ipi_virq_add_hwirq(struct ipi_virq *v,
>  					 struct ipi_hwirq *h);
>  extern struct ipi_hwirq *irq_domain_ipi_virq_rm_hwirq(struct ipi_virq *v);
> +extern unsigned int irq_reserve_ipi(struct irq_domain *domain,
> +				const struct cpumask *dest, void *devid);
> +extern void irq_destroy_ipi(unsigned int irq, void *devid);
> +extern void irq_send_ipi(unsigned int irq, const struct cpumask *dest, void *devid);
>  
>  /* V2 interfaces to support hierarchy IRQ domains. */
>  extern struct irq_data *irq_domain_get_irq_data(struct irq_domain *domain,
> diff --git a/kernel/irq/irqdomain.c b/kernel/irq/irqdomain.c
> index 6d911fc8fa52..d38d78a994ca 100644
> --- a/kernel/irq/irqdomain.c
> +++ b/kernel/irq/irqdomain.c
> @@ -873,7 +873,7 @@ struct ipi_virq *irq_domain_find_ipi_virq(struct irq_domain *d,
>   * returns a pointer to ipi_virq struct on success and NULL on failure
>   */
>  static struct ipi_virq *irq_domain_alloc_ipi_virq(unsigned int virq,
> -				struct cpumask *cpumask, void *devid)
> +				const struct cpumask *cpumask, void *devid)
>  {
>  	struct ipi_virq *v;
>  
> @@ -942,6 +942,165 @@ struct ipi_hwirq *irq_domain_ipi_virq_rm_hwirq(struct ipi_virq *v)
>  	return h;
>  }
>  
> +/**
> + * irq_reserve_ipi() - setup an IPI to destination cpumask
> + * @domain: IPI domain
> + * @dest: cpumask of cpus to receive the IPI
> + * @devid: devid that requested the reservation
> + *
> + * Allocate a virq that can be used to send IPI to any CPU in dest mask.
> + *
> + * On success it'll return linux irq number and 0 on failure
> + */
> +unsigned int irq_reserve_ipi(struct irq_domain *domain,
> +			     const struct cpumask *dest, void *devid)
Hi Qais,
	I have caught the idea why we need "dest" here. Per my
understanding, IPI could be sent to any active CPUs and the target
CPUs are specified when calling send_ipi(). What's the benefit or
usage to use "dest" to define a possible target scope here? And
how cpu hotplug?
Thanks!
Gerry

> +{
> +	int virq, ret;
> +	unsigned int nr_irqs;
> +	struct ipi_virq *v;
> +
> +	if (domain == NULL)
> +		domain = irq_default_domain; /* need a separate ipi_default_domain? */
> +
> +	if (domain == NULL) {
> +		pr_warn("Must provide a valid IPI domain!\n");
> +		return 0;
> +	}
> +
> +	if (!irq_domain_is_ipi(domain)) {
> +		pr_warn("Not an IPI domain!\n");
> +		return 0;
> +	}
> +
> +	if (cpumask_empty(dest)) {
> +		pr_warn("Can't reserve IPI due to empty cpumask\n");
> +		return 0;
> +	}
> +
> +	/* always allocate a virq per cpu */
> +	nr_irqs = cpumask_weight(dest);
> +
> +	virq = irq_domain_alloc_descs(-1, nr_irqs, 0, NUMA_NO_NODE);
> +	if (virq <= 0) {
> +		pr_warn("Can't reserve IPI, failed to alloc descs\n");
> +		return 0;
> +	}
> +
> +	v = irq_domain_alloc_ipi_virq(virq, dest, devid);
> +	if (!v) {
> +		pr_warn("Can't reserve IPI, failed to alloc ipi_virq\n");
> +		goto free_descs;
> +	}
> +
> +	/* we are reusing hierarchy alloc function, should we create another one? */
> +	virq = __irq_domain_alloc_irqs(domain, virq, nr_irqs, NUMA_NO_NODE,
> +					v, true);
> +	if (virq <= 0) {
> +		pr_warn("Can't reserve IPI, failed to alloc irqs\n");
> +		goto free_ipi;
> +	}
> +
> +	ret = irq_domain_add_ipi_virq(domain, v);
> +	if (ret) {
> +		pr_warn("Can't reserve IPI, failed to add ipi_virq\n");
> +		goto free_ipi;
> +	}
> +
> +	return virq;
> +
> +free_ipi:
> +	irq_domain_free_ipi_virq(v);
> +free_descs:
> +	irq_free_descs(virq, nr_irqs);
> +	return 0;
> +}
> +
> +/**
> + * irq_destroy_ipi() - unreserve an IPI that was previously allocated
> + * @irq: linux irq number to be destroyed
> + * @devid: devid that reserved the IPI
> + *
> + * Return an IPI allocated with irq_reserve_ipi() to the system.
> + */
> +void irq_destroy_ipi(unsigned int irq, void *devid)
> +{
> +	struct irq_data *irq_data = irq_get_irq_data(irq);
> +	struct irq_domain *domain;
> +	struct ipi_virq *v;
> +
> +	if (!irq || !irq_data)
> +		return;
> +
> +	domain = irq_data->domain;
> +	if (WARN_ON(domain == NULL))
> +		return;
> +
> +	if (!irq_domain_is_ipi(domain)) {
> +		pr_warn("Not an IPI domain!\n");
> +		return;
> +	}
> +
> +	v = irq_domain_find_ipi_virq(domain, irq);
> +	if (!v)
> +		return;
> +
> +	if (v->devid != devid) {
> +		pr_warn("Only the device that allocated the IPI can destroy it\n");
> +		return;
> +	}
> +
> +	irq_domain_free_irqs(irq, cpumask_weight(&v->cpumask));
> +
> +	v = irq_domain_rm_ipi_virq(domain, irq);
> +	irq_domain_free_ipi_virq(v);
> +}
> +
> +/**
> + * irq_send_ipi() - send an IPI to target CPU(s)
> + * @irq: linux irq number from irq_reserve_ipi()
> + * @dest: dest CPU(s), must be the same or a subset of the mask past to
> + *	  irq_reserve_ipi()
> + * @devid: devid that reserved the IPI
> + *
> + * Sends an IPI to all cpus in dest mask
> + */
> +void irq_send_ipi(unsigned int irq, const struct cpumask *dest, void *devid)
> +{
> +	struct irq_data *irq_data = irq_get_irq_data(irq);
> +	struct irq_domain *domain;
> +	struct ipi_virq *v;
> +	struct ipi_hwirq *h;
> +
> +	if (!irq || !irq_data)
> +		return;
> +
> +	domain = irq_data->domain;
> +	if (WARN_ON(domain == NULL))
> +		return;
> +
> +	if (!irq_domain_is_ipi(domain)) {
> +		pr_warn("Not an IPI domain!\n");
> +		return;
> +	}
> +
> +	v = irq_domain_find_ipi_virq(domain, irq);
> +	if (!v)
> +		return;
> +
> +	if (v->devid != devid) {
> +		pr_warn("Only the device that allocated the IPI can send one\n");
> +		return;
> +	}
> +
> +	if (!cpumask_intersects(&v->cpumask, dest))
> +		return;
> +
> +	list_for_each_entry(h, &v->mapped_hwirq, link) {
> +		if (cpumask_intersects(&h->cpumask, dest))
> +			domain->ops->send_ipi(h->hwirq);
> +	}
> +}
> +
>  #ifdef	CONFIG_IRQ_DOMAIN_HIERARCHY
>  /**
>   * irq_domain_add_hierarchy - Add a irqdomain into the hierarchy
> 

Re: [PATCH 0/6] Implement generic IPI support mechanism

[Jiang Liu]

On 2015/9/23 22:49, Qais Yousef wrote:
> This RFC series attempts to implement a generic IPI layer for reserving and sending IPI.
> 
> It is based on the discussion in this link
> 
> 	https://lkml.org/lkml/2015/8/26/713
> 
> This series deals with points #1 and #2 only. Since I'm not the irq expert, I'm hoping this
> series will give me early feedback and drive the discussion further about any potential
> tricky points.
> 
> I tried to keep changes clean and small, but since this is just an RFC I might have missed
> few things.
> 
> Thomas I hope I didn't stray far from what you had in mind :-)
> 
> My only testing so far is having SMP linux booting.
Hi Qais,
	Thanks for doing this, but the change is a little bigger than
my expectation. Could we achieve this by:
1) extend irq_chip to support send_ipi operation
2) reuse existing irqdomain allocation interfaces to allocate IPI IRQ
3) arch code to create an IPI domain for IPI allocations
4) IRQ core provides some helpers to help arch code to implement IPI
   irqdomain
	I think that may make the change smaller and more clear.
Thanks!
Gerry

> 
> Qais Yousef (6):
>   irqdomain: add new IRQ_DOMAIN_FLAGS_IPI
>   irqdomain: add a new send_ipi() to irq_domain_ops
>   irqdomain: add struct irq_hwcfg and helper functions
>   irq: add a new generic IPI handling code to irq core
>   irqchip: mips-gic: add a IPI hierarchy domain
>   irqchip: mips-gic: use the new generic IPI API
> 
>  arch/mips/kernel/smp-gic.c       |  37 ++--
>  drivers/irqchip/Kconfig          |   1 +
>  drivers/irqchip/irq-mips-gic.c   | 189 ++++++++++++++++++---
>  include/linux/irqchip/mips-gic.h |   3 +-
>  include/linux/irqdomain.h        |  67 ++++++++
>  kernel/irq/irqdomain.c           | 352 +++++++++++++++++++++++++++++++++++++++
>  6 files changed, 601 insertions(+), 48 deletions(-)
> 

Re: [PATCH 2/6] irqdomain: add a new send_ipi() to irq_domain_ops

[Qais Yousef]

On 09/23/2015 05:44 PM, Jiang Liu wrote:
> On 2015/9/23 22:49, Qais Yousef wrote:
>> For generic ipi core to use. It takes hwirq as its sole argument.
>> Hopefully this is generic enough? Should we pass something more abstract?
>>
>> Signed-off-by: Qais Yousef <qais.yousef@imgtec.com>
>> ---
>>   include/linux/irqdomain.h | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
>> index 9b3dc6c2a3cc..cef9e6158be0 100644
>> --- a/include/linux/irqdomain.h
>> +++ b/include/linux/irqdomain.h
>> @@ -92,6 +92,7 @@ struct irq_domain_ops {
>>   	void (*activate)(struct irq_domain *d, struct irq_data *irq_data);
>>   	void (*deactivate)(struct irq_domain *d, struct irq_data *irq_data);
>>   #endif
>> +	void (*send_ipi)(irq_hw_number_t hwirq);
> Hi Qais,
> 	Instead of extending the irq_domain_ops, how about extending
> irq_chip instead? If we treat IPI as a sort of irq controller, and
> irq_chip is used to encapsulate all irq controller related operations,
> and irq_domain_ops is mainly used to allocated resources instead of
> operating corresponding hardware.
> Thanks!
> Gerry
>



Yes true. I was too focused on the reservation part so when it came to 
sending the IPI it just felt natural to add it to irqdomain as it was 
simple to implement like this!

Thanks,
Qais

Re: [PATCH 4/6] irq: add a new generic IPI handling code to irq core

[Qais Yousef]

On 09/23/2015 05:50 PM, Jiang Liu wrote:
> On 2015/9/23 22:49, Qais Yousef wrote:
>>   
>> +/**
>> + * irq_reserve_ipi() - setup an IPI to destination cpumask
>> + * @domain: IPI domain
>> + * @dest: cpumask of cpus to receive the IPI
>> + * @devid: devid that requested the reservation
>> + *
>> + * Allocate a virq that can be used to send IPI to any CPU in dest mask.
>> + *
>> + * On success it'll return linux irq number and 0 on failure
>> + */
>> +unsigned int irq_reserve_ipi(struct irq_domain *domain,
>> +			     const struct cpumask *dest, void *devid)
> Hi Qais,
> 	I have caught the idea why we need "dest" here. Per my
> understanding, IPI could be sent to any active CPUs and the target
> CPUs are specified when calling send_ipi(). What's the benefit or
> usage to use "dest" to define a possible target scope here? And
> how cpu hotplug?
> Thanks!
> Gerry
>


The CPUs we want to send the IPI to are not Linux CPUs only. My use case 
is about sending IPI to audio coprocessor.
So "dest" doesn't have to be part of Linux online CPUs, hence we need to 
specify it so that the underlying controller will know how to map to 
that CPU. I should have put more info in the cover letter, not just the 
link to the discussion, apologies for that.

I'm not sure about cpu hotplug. We could call irq_destroy_ipi() when a 
cpu is hot unplugged, but the current behaviour is to statically reserve 
the IPI and keep them reserved. I think it makes sense to keep it this 
way for SMP IPIs or things will get complicated.

For a coprocessor, if we the 'module is unloaded', I'd expect the 
irq_destroy_ipi() to be called returning the reserved IPI to the pool.

Makes sense?

Thanks,
Qais

Re: [PATCH 0/6] Implement generic IPI support mechanism

[Qais Yousef]

On 09/23/2015 05:54 PM, Jiang Liu wrote:
> On 2015/9/23 22:49, Qais Yousef wrote:
>> This RFC series attempts to implement a generic IPI layer for reserving and sending IPI.
>>
>> It is based on the discussion in this link
>>
>> 	https://lkml.org/lkml/2015/8/26/713
>>
>> This series deals with points #1 and #2 only. Since I'm not the irq expert, I'm hoping this
>> series will give me early feedback and drive the discussion further about any potential
>> tricky points.
>>
>> I tried to keep changes clean and small, but since this is just an RFC I might have missed
>> few things.
>>
>> Thomas I hope I didn't stray far from what you had in mind :-)
>>
>> My only testing so far is having SMP linux booting.
> Hi Qais,
> 	Thanks for doing this, but the change is a little bigger than
> my expectation. Could we achieve this by:
> 1) extend irq_chip to support send_ipi operation
> 2) reuse existing irqdomain allocation interfaces to allocate IPI IRQ
> 3) arch code to create an IPI domain for IPI allocations
> 4) IRQ core provides some helpers to help arch code to implement IPI
>     irqdomain
> 	I think that may make the change smaller and more clear.
> Thanks!
> Gerry
>
>


Can you be more specific about 2 please? I tried to reuse the hierarchy 
irqdomain alloc function. One major difference when allocating IPI than 
a normal irq is that it's dynamic. The caller doesn't know what hwirq 
number it needs. It actually shouldn't.

The idea is for the user to just say 'I want an IPI to a CPUAFFINITY' 
from DT and get a virq in return to send an IPI to the target CPU(s). 
Also I think we need to accommodate the possibility of having more than 
1 IPI controller.

Can you provide more pointers please?

Thanks,
Qais

Re: [PATCH 4/6] irq: add a new generic IPI handling code to irq core

[Thomas Gleixner]

On Thu, 24 Sep 2015, Qais Yousef wrote:
> The CPUs we want to send the IPI to are not Linux CPUs only. My use case is
> about sending IPI to audio coprocessor.
> So "dest" doesn't have to be part of Linux online CPUs, hence we need to
> specify it so that the underlying controller will know how to map to that CPU.
> I should have put more info in the cover letter, not just the link to the
> discussion, apologies for that.
> 
> I'm not sure about cpu hotplug. We could call irq_destroy_ipi() when a cpu is
> hot unplugged, but the current behaviour is to statically reserve the IPI and
> keep them reserved. I think it makes sense to keep it this way for SMP IPIs or
> things will get complicated.

Right. For general IPIs which are destined to all Linux CPUs we keep
them reserved unless the facility which needs them is shut down.

CPU hotplug is just disabling the IPI reception on the particular
core, but does not change the reservation for e.g. the resched IPI.
 
> For a coprocessor, if we the 'module is unloaded', I'd expect the
> irq_destroy_ipi() to be called returning the reserved IPI to the pool.

For any case which shuts down a IPI based facility (coprocessor or
general) we want to return the IPI. Otherwise we leak the IPI on
module removal and allocate a new one on reload.

Thanks,

	tglx

Re: [PATCH 0/6] Implement generic IPI support mechanism

[Thomas Gleixner]

On Thu, 24 Sep 2015, Qais Yousef wrote:
> On 09/23/2015 05:54 PM, Jiang Liu wrote:
> > 	Thanks for doing this, but the change is a little bigger than
> > my expectation. Could we achieve this by:
> > 1) extend irq_chip to support send_ipi operation
> > 2) reuse existing irqdomain allocation interfaces to allocate IPI IRQ
> > 3) arch code to create an IPI domain for IPI allocations
> > 4) IRQ core provides some helpers to help arch code to implement IPI
> >     irqdomain

That's not sufficient as IPIs are different from normal interrupts
because we need an interface to actually send them.

> Can you be more specific about 2 please? I tried to reuse the hierarchy
> irqdomain alloc function. One major difference when allocating IPI than a
> normal irq is that it's dynamic. The caller doesn't know what hwirq number it
> needs. It actually shouldn't.

Right. But we have the same behaviour with e.g. MSI. The caller does
not know a hardware irq number because it is dynamically assigned.
 
> The idea is for the user to just say 'I want an IPI to a CPUAFFINITY' from DT
> and get a virq in return to send an IPI to the target CPU(s). Also I think we
> need to accommodate the possibility of having more than 1 IPI controller.

Having more than one IPI controller is not a problem. It's going to be
a separate IPI domain, which you select from DT or other means.

These IPI domains are implemented like the MSI domain as child
domains of the underlying irq domain.

     [IPI domain] ---> [GIC domain]

like we have on x86

     [MSI domain] ---> [Vector domain]

So you need some infrastructure, which allows you to:

 - allocate IPI(s)

     Requests IPI(s) from a IPI domain. That might be the default IPI
     domain or one that is matched via OF against a list of registered
     domains or one which is known to the caller by other means.

     Now that allocation interface does:

      1) Allocate irq descriptor

         This is required even for IPIs which are targeted to
         coprocessors and cannot be requested from Linux. In that case
         the only purpose is to store the irq chip and the irq domain
         specific data for that virq/hwirq mapping and the irq is
         marked as NOREQUEST.

      2) Allocate the vector/hwirq number block from the IPI domain
      	 
	 Part of the allocation request info is a pointer to the
	 target cpu mask. The weight of the target cpu mask is the
	 number of hwirqs you need to allocate from the underlying
	 domain.

	 For a normal Linux IPI, this will be the number of possible
	 CPUs. For a coprocessor IPI, this will be a single hwirq.

	 We also store that target cpu mask for runtime validation and
	 other usage in the irq descriptor data. We can actually reuse
	 the existing affinity mask for that.

	 Now how these hwirqs are allocated is a domain/architecture
	 specific issue.

	 x86 will just find a vector which is available on all target
	 cpus and mark it as used. That's a single hw irq number.

	 mips and others, which implement IPIs as regular hw interrupt
	 numbers, will allocate a these (consecutive) hw interrupt
	 numbers either from a reserved region or just from the
	 regular space. That's a bunch of hw irq numbers and we need
	 to come up with a proper storage format in the irqdata for
	 that. That might be

	       struct ipi_mapping {
		      unsigned int	nr_hwirqs;
		      unsigned int	cpumap[NR_CPUS];
	       };

	 or some other appropriate storage format like:

	       struct ipi_mapping {
	       	      unsigned int	hwirq_base;
		      unsigned int	cpu_offset;
		      unsigned int	nr_hwirqs;
	       };

	 which is less space consuming, but restricted to consecutive
	 hwirqs which can be mapped to the cpu number linearly:

	 	hwirq = hwirq_base + cpu - cpu_offset;
	 
       The result of this is a single virq number, which has all the
       necessary information stored in the associated irq descriptor
       and the domain specific hierarchical irq_data.

       For normal Linux IPIs that irq is marked as per cpu irq and can
       be requested via request_percpu_irq() and enabled/disabled via
       enable_percpu_irq/disable_percpu_irq on CPU hot[un]plug.

 - A function to send an IPI to a virq number

     That function takes the virq number and a target cpumask as
     argument.

     Actually we want two functions where the one which takes an virq
     number is a wrapper around the other which takes a irq descriptor
     pointer.

     The one which takes the virq number can be exported to drivers,
     the other one is a core/arch code only interface. The reason for
     this is that we want to avoid the irq descriptor lookup for
     regular IPIs, but for drivers this is a NONO.

     int irq_send_ipi(int virq, const struct cpumask *mask)
     {
	struct irq_desc *desc = irq_to_desc(virq);

	if (!desc)
		return -EINVAL;

	return irq_desc_send_ipi(desc, mask);
     }

     Along with a version which sends an IPI to all cpus in the target
     mask:

     int irq_send_ipi_all(int virq)
     {
	struct irq_desc *desc = irq_to_desc(virq);
	struct irq_data *data;

	if (!desc)
		return -EINVAL;

	data = irq_desc_get_irq_data(desc);
	return irq_desc_send_ipi(desc, irq_data_get_affinity_mask(data));
     }
     
     And the internal function:

     int irq_desc_send_ipi(struct irq_desc *desc, const struct cpumask *mask)
     {
	struct irq_data *data = irq_desc_get_irq_data(desc);
	struct irq_chip *chip = irq_data_get_irq_chip(data);

	if (!chip || !chip->send_ipi)
	   	  return -EINVAL;

	/*
	 * Do not validate the mask for IPIs marked global. These are
	 * regular IPIs so we can avoid the operation as their target
	 * mask is the cpu_possible_mask.
	 */
	if (!irqd_is_global_ipi(data)) {
	   if (!cpumask_subset(mask, irq_data_get_affinity_mask(data))
	      	  return -EINVAL;
	}

	chip->send_ipi(data, mask);
	return 0;
     }

     So now the chip specific send_ipi function will deal with the
     underlying implementation details.

     on x86 it uses the selected APIC implementation and sends
     the IPI to the vector stored in the hw irq number to all CPUs
     which are in the mask.

     on mips and others it's a bit different as you need to figure out
     the effective hwirq number for the cpus set in the target mask
     from the stored mapping in the hierarchical irq data. We
     certainly can create common helpers for this. Assume the simple
     mapping format:

     	       struct ipi_mapping {
		      unsigned int	nr_hwirqs;
		      unsigned int	cpumap[];
	       };
     
     then a helper function for the IPI domain irq chip would be:

     void irq_chip_send_ipi(struct irq_data *data, const struct cpumask *mask)
     {
	struct ipi_mapping *map = irq_data_get_irq_chip_data(data);
	struct irq_data *parent = data->parent;
	unsigned int cpu, hwirq;

	for_each_cpu(cpu, mask) {
		hwirq = map->cpumap[cpu];
		/* Deal with gaps */
		if (hwirq == INVALID_HWIRQ)
		   continue;
		parent->chip->send_ipi(parent, cpumask_of(cpu));
	}
     }
     
No linked lists, no magic other stuff. Just a natural extension to the
existing hierarchical irq domain code, which can be reused by all
architectures.

Thanks,

	tglx

Re: [PATCH 0/6] Implement generic IPI support mechanism

[Qais Yousef]

On 09/29/2015 09:48 PM, Thomas Gleixner wrote:
>
> 	 Now how these hwirqs are allocated is a domain/architecture
> 	 specific issue.
>
> 	 x86 will just find a vector which is available on all target
> 	 cpus and mark it as used. That's a single hw irq number.
>
> 	 mips and others, which implement IPIs as regular hw interrupt
> 	 numbers, will allocate a these (consecutive) hw interrupt
> 	 numbers either from a reserved region or just from the
> 	 regular space. That's a bunch of hw irq numbers and we need
> 	 to come up with a proper storage format in the irqdata for
> 	 that. That might be
>
> 	       struct ipi_mapping {
> 		      unsigned int	nr_hwirqs;
> 		      unsigned int	cpumap[NR_CPUS];
> 	       };

Can we use NR_CPUS here? If we run in UP configuration for instance, 
this will be one. The coprocessor could be outside the NR_CPUS range in 
general, no?

How about

                         struct ipi_mapping {
                                 unsigned int        nr_hwirqs;
                                 unsigned int        nr_cpus;
                                 unsigned int        *cpumap;
                         }

where cpumap is dynamically allocated by the controller which has better 
knowledge about the supported cpu range it can talk to?

This made me realise another problem. struct cpumask is dependent on 
NR_CPUS. I can use the generic BITMAP I suppose?

> 	 or some other appropriate storage format like:
>
> 	       struct ipi_mapping {
> 	       	      unsigned int	hwirq_base;
> 		      unsigned int	cpu_offset;
> 		      unsigned int	nr_hwirqs;
> 	       };
>
> 	 which is less space consuming, but restricted to consecutive
> 	 hwirqs which can be mapped to the cpu number linearly:
>
> 	 	hwirq = hwirq_base + cpu - cpu_offset;
> 	
>


This could work without worrying about NR_CPUS but it would be nice not 
to restrict the controller to consecutive hwirqs.

Thanks a lot for the comprehensive pointers!

Thanks,
Qais

Re: [PATCH 0/6] Implement generic IPI support mechanism

[Thomas Gleixner]

On Wed, 30 Sep 2015, Qais Yousef wrote:

> On 09/29/2015 09:48 PM, Thomas Gleixner wrote:
> > 
> > 	 Now how these hwirqs are allocated is a domain/architecture
> > 	 specific issue.
> > 
> > 	 x86 will just find a vector which is available on all target
> > 	 cpus and mark it as used. That's a single hw irq number.
> > 
> > 	 mips and others, which implement IPIs as regular hw interrupt
> > 	 numbers, will allocate a these (consecutive) hw interrupt
> > 	 numbers either from a reserved region or just from the
> > 	 regular space. That's a bunch of hw irq numbers and we need
> > 	 to come up with a proper storage format in the irqdata for
> > 	 that. That might be
> > 
> > 	       struct ipi_mapping {
> > 		      unsigned int	nr_hwirqs;
> > 		      unsigned int	cpumap[NR_CPUS];
> > 	       };
> 
> Can we use NR_CPUS here? If we run in UP configuration for instance, this will
> be one. The coprocessor could be outside the NR_CPUS range in general, no?
> 
> How about
> 
>                         struct ipi_mapping {
>                                 unsigned int        nr_hwirqs;
>                                 unsigned int        nr_cpus;
>                                 unsigned int        *cpumap;
>                         }
> 
> where cpumap is dynamically allocated by the controller which has better
> knowledge about the supported cpu range it can talk to?

Sure. As I said: 'That might be' ....
 
Thanks,

	tglx

Re: [PATCH 0/6] Implement generic IPI support mechanism

[Qais Yousef]

On 09/30/2015 03:03 PM, Thomas Gleixner wrote:
> On Wed, 30 Sep 2015, Qais Yousef wrote:
>
>> On 09/29/2015 09:48 PM, Thomas Gleixner wrote:
>>> 	 Now how these hwirqs are allocated is a domain/architecture
>>> 	 specific issue.
>>>
>>> 	 x86 will just find a vector which is available on all target
>>> 	 cpus and mark it as used. That's a single hw irq number.
>>>
>>> 	 mips and others, which implement IPIs as regular hw interrupt
>>> 	 numbers, will allocate a these (consecutive) hw interrupt
>>> 	 numbers either from a reserved region or just from the
>>> 	 regular space. That's a bunch of hw irq numbers and we need
>>> 	 to come up with a proper storage format in the irqdata for
>>> 	 that. That might be
>>>
>>> 	       struct ipi_mapping {
>>> 		      unsigned int	nr_hwirqs;
>>> 		      unsigned int	cpumap[NR_CPUS];
>>> 	       };
>> Can we use NR_CPUS here? If we run in UP configuration for instance, this will
>> be one. The coprocessor could be outside the NR_CPUS range in general, no?
>>
>> How about
>>
>>                          struct ipi_mapping {
>>                                  unsigned int        nr_hwirqs;
>>                                  unsigned int        nr_cpus;
>>                                  unsigned int        *cpumap;
>>                          }
>>
>> where cpumap is dynamically allocated by the controller which has better
>> knowledge about the supported cpu range it can talk to?
> Sure. As I said: 'That might be' ....
>   
>


OK thanks. I just wanted to make sure I didn't misunderstand anything.

Will try to send an updated version with all the changes soon.

Thanks,
Qais