[PATCH 0/5] Partitioning per-cpu interrupts

[PATCH 0/5] Partitioning per-cpu interrupts

[Marc Zyngier]

We've unfortunately started seeing a situation where percpu interrupts
are partitioned in the system: one arbitrary set of CPUs has an
interrupt connected to a type of device, while another disjoint set of
CPUs has the same interrupt connected to another type of device.
    
This makes it impossible to have a device driver requesting this
interrupt using the current percpu-interrupt abstraction, as the same
interrupt number is now potentially claimed by at least two drivers,
and we forbid interrupt sharing on per-cpu interrupt.

A potential solution to this has been proposed by Will Deacon,
expanding the handling in the core code:

http://lists.infradead.org/pipermail/linux-arm-kernel/2015-November/388800.html

followed by a counter-proposal from Thomas Gleixner, which Will tried
to implement, but ran into issues where the probing code was running
in preemptible context, making the percpu-ness of interrupts difficult
to guarantee.
    
Another approach to this is to turn things upside down. Let's assume
that our system describes all the possible partitions for a given
interrupt, and give each of them a unique identifier. It is then
possible to create a namespace where the affinity identifier itself is
a form of interrupt number. At this point, it becomes easy to
implement a set of partitions as a cascaded irqchip, each affinity
identifier being the secondary HW irq, as outlined in the following
example:

	Aff-0: { cpu0 cpu3 }
	Aff-1: { cpu1 cpu2 }
	Aff-2: { cpu4 cpu5 cpu6 cpu7 }

Let's assume that HW interrupt 1 is partitioned over these 3
affinities. When HW interrupt 1 fires on a given CPU, all it takes is
to find out which affinity this CPU belongs to, which gives us a new
HW interrupt number. Bingo. Of course, this only works as long as you
don't have overlapping affinities (but if you do your system is broken
anyway).
    
This allows us to keep a number of nice properties:

- Each partition results in a separate percpu-interrupt (with a
  restricted affinity), which keeps drivers happy. This alone
  garantees that we do not have to change the programming model for
  per-cpu interrupts.

- Because the underlying interrupt is still per-cpu, the overhead of
  the indirection can be kept pretty minimal.

- The core code can ignore most of that crap.
    
For that purpose, we implement a small library that deals with some of
the boilerplate code, relying on platform-specific drivers to provide
a description of the affinity sets and a set of callbacks. This also
relies on a small change in the irqdomain layer, and now offers a way
for the affinity of a percpu interrupt to be retrieved by a driver.

As an example, the GICv3 driver has been adapted to use this new
feature. Patches on top of v4.6-r3, tested on an arm64 FVP model.

Marc Zyngier (5):
  irqdomain: Allow domain matching on irq_fwspec
  genirq: Allow the affinity of a percpu interrupt to be set/retrieved
  irqchip: Add per-cpu interrupt partitioning library
  irqchip/gic-v3: Add support for partitioned PPIs
  DT: arm,gic-v3: Documment PPI partition support

 .../bindings/interrupt-controller/arm,gic-v3.txt   |  34 ++-
 drivers/irqchip/Kconfig                            |   4 +
 drivers/irqchip/Makefile                           |   1 +
 drivers/irqchip/irq-gic-v3.c                       | 176 +++++++++++++-
 drivers/irqchip/irq-partition-percpu.c             | 256 +++++++++++++++++++++
 include/linux/irq.h                                |   4 +
 include/linux/irqchip/irq-partition-percpu.h       |  59 +++++
 include/linux/irqdesc.h                            |   1 +
 include/linux/irqdomain.h                          |  15 +-
 kernel/irq/irqdesc.c                               |  26 ++-
 kernel/irq/irqdomain.c                             |  19 +-
 11 files changed, 580 insertions(+), 15 deletions(-)
 create mode 100644 drivers/irqchip/irq-partition-percpu.c
 create mode 100644 include/linux/irqchip/irq-partition-percpu.h

-- 
2.1.4

[PATCH 1/5] irqdomain: Allow domain matching on irq_fwspec

[Marc Zyngier]

When iterating over the irq domain list, we try to match a domain
either by calling a match() function or by comparing a number
of fields passed as parameters.

Both approaches are a bit restrictive:
- match() is DT specific and only takes a device node
- the fallback case only deals with the fwnode_handle

It would be useful if we had a per-domain function that would
actually perform the matching check on the whole of the
irq_fwspec structure. This would allow for a domain to triage
matching attempts that need to extend beyond the fwnode.

Let's introduce irq_find_matching_fwspec(), which takes a full
blown irq_fwspec structure, and call into a select() function
implemented by the irqdomain. irq_find_matching_fwnode() is
made a wrapper around irq_find_matching_fwspec in order to
preserve compatibility.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 include/linux/irqdomain.h | 15 ++++++++++++++-
 kernel/irq/irqdomain.c    | 19 ++++++++++---------
 2 files changed, 24 insertions(+), 10 deletions(-)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index 2aed043..997373b 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -96,6 +96,8 @@ enum irq_domain_bus_token {
 struct irq_domain_ops {
 	int (*match)(struct irq_domain *d, struct device_node *node,
 		     enum irq_domain_bus_token bus_token);
+	int (*select)(struct irq_domain *d, struct irq_fwspec *fwspec,
+		      enum irq_domain_bus_token bus_token);
 	int (*map)(struct irq_domain *d, unsigned int virq, irq_hw_number_t hw);
 	void (*unmap)(struct irq_domain *d, unsigned int virq);
 	int (*xlate)(struct irq_domain *d, struct device_node *node,
@@ -211,7 +213,7 @@ struct irq_domain *irq_domain_add_legacy(struct device_node *of_node,
 					 irq_hw_number_t first_hwirq,
 					 const struct irq_domain_ops *ops,
 					 void *host_data);
-extern struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
+extern struct irq_domain *irq_find_matching_fwspec(struct irq_fwspec *fwspec,
 						   enum irq_domain_bus_token bus_token);
 extern void irq_set_default_host(struct irq_domain *host);
 extern int irq_domain_alloc_descs(int virq, unsigned int nr_irqs,
@@ -227,6 +229,17 @@ static inline bool is_fwnode_irqchip(struct fwnode_handle *fwnode)
 	return fwnode && fwnode->type == FWNODE_IRQCHIP;
 }
 
+static inline
+struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
+					    enum irq_domain_bus_token bus_token)
+{
+	struct irq_fwspec fwspec = {
+		.fwnode = fwnode,
+	};
+
+	return irq_find_matching_fwspec(&fwspec, bus_token);
+}
+
 static inline struct irq_domain *irq_find_matching_host(struct device_node *node,
 							enum irq_domain_bus_token bus_token)
 {
diff --git a/kernel/irq/irqdomain.c b/kernel/irq/irqdomain.c
index 3a519a0..503c5b9 100644
--- a/kernel/irq/irqdomain.c
+++ b/kernel/irq/irqdomain.c
@@ -243,14 +243,15 @@ struct irq_domain *irq_domain_add_legacy(struct device_node *of_node,
 EXPORT_SYMBOL_GPL(irq_domain_add_legacy);
 
 /**
- * irq_find_matching_fwnode() - Locates a domain for a given fwnode
- * @fwnode: FW descriptor of the interrupt controller
+ * irq_find_matching_fwspec() - Locates a domain for a given fwspec
+ * @fwspec: FW specifier for an interrupt
  * @bus_token: domain-specific data
  */
-struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
+struct irq_domain *irq_find_matching_fwspec(struct irq_fwspec *fwspec,
 					    enum irq_domain_bus_token bus_token)
 {
 	struct irq_domain *h, *found = NULL;
+	struct fwnode_handle *fwnode = fwspec->fwnode;
 	int rc;
 
 	/* We might want to match the legacy controller last since
@@ -264,7 +265,9 @@ struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
 	 */
 	mutex_lock(&irq_domain_mutex);
 	list_for_each_entry(h, &irq_domain_list, link) {
-		if (h->ops->match)
+		if (h->ops->select && fwspec->param_count)
+			rc = h->ops->select(h, fwspec, bus_token);
+		else if (h->ops->match)
 			rc = h->ops->match(h, to_of_node(fwnode), bus_token);
 		else
 			rc = ((fwnode != NULL) && (h->fwnode == fwnode) &&
@@ -279,7 +282,7 @@ struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
 	mutex_unlock(&irq_domain_mutex);
 	return found;
 }
-EXPORT_SYMBOL_GPL(irq_find_matching_fwnode);
+EXPORT_SYMBOL_GPL(irq_find_matching_fwspec);
 
 /**
  * irq_set_default_host() - Set a "default" irq domain
@@ -574,11 +577,9 @@ unsigned int irq_create_fwspec_mapping(struct irq_fwspec *fwspec)
 	int virq;
 
 	if (fwspec->fwnode) {
-		domain = irq_find_matching_fwnode(fwspec->fwnode,
-						  DOMAIN_BUS_WIRED);
+		domain = irq_find_matching_fwspec(fwspec, DOMAIN_BUS_WIRED);
 		if (!domain)
-			domain = irq_find_matching_fwnode(fwspec->fwnode,
-							  DOMAIN_BUS_ANY);
+			domain = irq_find_matching_fwspec(fwspec, DOMAIN_BUS_ANY);
 	} else {
 		domain = irq_default_domain;
 	}
-- 
2.1.4

[PATCH 2/5] genirq: Allow the affinity of a percpu interrupt to be set/retrieved

[Marc Zyngier]

In order to prepare the genirq layer for the concept of partitionned
percpu interrupts, let's allow an affinity to be associated with
such an interrupt. We introduce:

- irq_set_percpu_devid_partition: flag an interrupt as a percpu-devid
  interrupt, and associate it with an affinity
- irq_get_percpu_devid_partition: allow the affinity of that interrupt
  to be retrieved.

This will allow a driver to discover which CPUs the per-cpu interrupt
can actually fire on.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 include/linux/irq.h     |  4 ++++
 include/linux/irqdesc.h |  1 +
 kernel/irq/irqdesc.c    | 26 +++++++++++++++++++++++++-
 3 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/include/linux/irq.h b/include/linux/irq.h
index c4de623..4d758a7 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -530,6 +530,10 @@ static inline void irq_set_chip_and_handler(unsigned int irq, struct irq_chip *c
 }
 
 extern int irq_set_percpu_devid(unsigned int irq);
+extern int irq_set_percpu_devid_partition(unsigned int irq,
+					  const struct cpumask *affinity);
+extern int irq_get_percpu_devid_partition(unsigned int irq,
+					  struct cpumask *affinity);
 
 extern void
 __irq_set_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
diff --git a/include/linux/irqdesc.h b/include/linux/irqdesc.h
index dcca77c..b51beeb 100644
--- a/include/linux/irqdesc.h
+++ b/include/linux/irqdesc.h
@@ -66,6 +66,7 @@ struct irq_desc {
 	int			threads_handled_last;
 	raw_spinlock_t		lock;
 	struct cpumask		*percpu_enabled;
+	const struct cpumask	*percpu_affinity;
 #ifdef CONFIG_SMP
 	const struct cpumask	*affinity_hint;
 	struct irq_affinity_notify *affinity_notify;
diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c
index 0ccd028..8731e1c 100644
--- a/kernel/irq/irqdesc.c
+++ b/kernel/irq/irqdesc.c
@@ -595,7 +595,8 @@ void __irq_put_desc_unlock(struct irq_desc *desc, unsigned long flags, bool bus)
 		chip_bus_sync_unlock(desc);
 }
 
-int irq_set_percpu_devid(unsigned int irq)
+int irq_set_percpu_devid_partition(unsigned int irq,
+				   const struct cpumask *affinity)
 {
 	struct irq_desc *desc = irq_to_desc(irq);
 
@@ -610,10 +611,33 @@ int irq_set_percpu_devid(unsigned int irq)
 	if (!desc->percpu_enabled)
 		return -ENOMEM;
 
+	if (affinity)
+		desc->percpu_affinity = affinity;
+	else
+		desc->percpu_affinity = cpu_possible_mask;
+
 	irq_set_percpu_devid_flags(irq);
 	return 0;
 }
 
+int irq_set_percpu_devid(unsigned int irq)
+{
+	return irq_set_percpu_devid_partition(irq, NULL);
+}
+
+int irq_get_percpu_devid_partition(unsigned int irq, struct cpumask *affinity)
+{
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	if (!desc || !desc->percpu_enabled)
+		return -EINVAL;
+
+	if (affinity)
+		cpumask_copy(affinity, desc->percpu_affinity);
+
+	return 0;
+}
+
 void kstat_incr_irq_this_cpu(unsigned int irq)
 {
 	kstat_incr_irqs_this_cpu(irq_to_desc(irq));
-- 
2.1.4

[PATCH 3/5] irqchip: Add per-cpu interrupt partitioning library

[Marc Zyngier]

We've unfortunately started seeing a situation where percpu interrupts
are partitioned in the system: one arbitrary set of CPUs has an
interrupt connected to a type of device, while another disjoint
set of CPUs has the same interrupt connected to another type of device.

This makes it impossible to have a device driver requesting this interrupt
using the current percpu-interrupt abstraction, as the same interrupt number
is now potentially claimed by at least two drivers, and we forbid interrupt
sharing on per-cpu interrupt.

A solution to this is to turn things upside down. Let's assume that our
system describes all the possible partitions for a given interrupt, and
give each of them a unique identifier. It is then possible to create
a namespace where the affinity identifier itself is a form of interrupt
number. At this point, it becomes easy to implement a set of partitions
as a cascaded irqchip, each affinity identifier being the HW irq.

This allows us to keep a number of nice properties:
- Each partition results in a separate percpu-interrupt (with a restrictied
  affinity), which keeps drivers happy.
- Because the underlying interrupt is still per-cpu, the overhead of
  the indirection can be kept pretty minimal.
- The core code can ignore most of that crap.

For that purpose, we implement a small library that deals with some of
the boilerplate code, relying on platform-specific drivers to provide
a description of the affinity sets and a set of callbacks.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 drivers/irqchip/Kconfig                      |   3 +
 drivers/irqchip/Makefile                     |   1 +
 drivers/irqchip/irq-partition-percpu.c       | 256 +++++++++++++++++++++++++++
 include/linux/irqchip/irq-partition-percpu.h |  59 ++++++
 4 files changed, 319 insertions(+)
 create mode 100644 drivers/irqchip/irq-partition-percpu.c
 create mode 100644 include/linux/irqchip/irq-partition-percpu.h

diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index 3e12479..ea1836b 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -244,3 +244,6 @@ config IRQ_MXS
 config MVEBU_ODMI
 	bool
 	select GENERIC_MSI_IRQ_DOMAIN
+
+config PARTITION_PERCPU
+	bool
diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
index b03cfcb..e354b00c 100644
--- a/drivers/irqchip/Makefile
+++ b/drivers/irqchip/Makefile
@@ -27,6 +27,7 @@ obj-$(CONFIG_REALVIEW_DT)		+= irq-gic-realview.o
 obj-$(CONFIG_ARM_GIC_V2M)		+= irq-gic-v2m.o
 obj-$(CONFIG_ARM_GIC_V3)		+= irq-gic-v3.o irq-gic-common.o
 obj-$(CONFIG_ARM_GIC_V3_ITS)		+= irq-gic-v3-its.o irq-gic-v3-its-pci-msi.o irq-gic-v3-its-platform-msi.o
+obj-$(CONFIG_PARTITION_PERCPU)		+= irq-partition-percpu.o
 obj-$(CONFIG_HISILICON_IRQ_MBIGEN)	+= irq-mbigen.o
 obj-$(CONFIG_ARM_NVIC)			+= irq-nvic.o
 obj-$(CONFIG_ARM_VIC)			+= irq-vic.o
diff --git a/drivers/irqchip/irq-partition-percpu.c b/drivers/irqchip/irq-partition-percpu.c
new file mode 100644
index 0000000..ccd72c2
--- /dev/null
+++ b/drivers/irqchip/irq-partition-percpu.c
@@ -0,0 +1,256 @@
+/*
+ * Copyright (C) 2016 ARM Limited, All Rights Reserved.
+ * Author: Marc Zyngier <marc.zyngier@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/bitops.h>
+#include <linux/interrupt.h>
+#include <linux/irqchip.h>
+#include <linux/irqchip/chained_irq.h>
+#include <linux/irqchip/irq-partition-percpu.h>
+#include <linux/irqdomain.h>
+#include <linux/seq_file.h>
+#include <linux/slab.h>
+
+struct partition_desc {
+	int				nr_parts;
+	struct partition_affinity	*parts;
+	struct irq_domain		*domain;
+	struct irq_desc			*chained_desc;
+	unsigned long			*bitmap;
+	struct irq_domain_ops		ops;
+};
+
+static bool partition_check_cpu(struct partition_desc *part,
+				unsigned int cpu, unsigned int hwirq)
+{
+	return cpumask_test_cpu(cpu, &part->parts[hwirq].mask);
+}
+
+static void partition_irq_mask(struct irq_data *d)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (partition_check_cpu(part, smp_processor_id(), d->hwirq) &&
+	    chip->irq_mask)
+		chip->irq_mask(data);
+}
+
+static void partition_irq_unmask(struct irq_data *d)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (partition_check_cpu(part, smp_processor_id(), d->hwirq) &&
+	    chip->irq_unmask)
+		chip->irq_unmask(data);
+}
+
+static int partition_irq_set_irqchip_state(struct irq_data *d,
+					   enum irqchip_irq_state which,
+					   bool val)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (partition_check_cpu(part, smp_processor_id(), d->hwirq) &&
+	    chip->irq_set_irqchip_state)
+		return chip->irq_set_irqchip_state(data, which, val);
+
+	return -EINVAL;
+}
+
+static int partition_irq_get_irqchip_state(struct irq_data *d,
+					   enum irqchip_irq_state which,
+					   bool *val)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (partition_check_cpu(part, smp_processor_id(), d->hwirq) &&
+	    chip->irq_get_irqchip_state)
+		return chip->irq_get_irqchip_state(data, which, val);
+
+	return -EINVAL;
+}
+
+static int partition_irq_set_type(struct irq_data *d, unsigned int type)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (chip->irq_set_type)
+		return chip->irq_set_type(data, type);
+
+	return -EINVAL;
+}
+
+static void partition_irq_print_chip(struct irq_data *d, struct seq_file *p)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	seq_printf(p, " %5s-%lu", chip->name, data->hwirq);
+}
+
+static struct irq_chip partition_irq_chip = {
+	.irq_mask		= partition_irq_mask,
+	.irq_unmask		= partition_irq_unmask,
+	.irq_set_type		= partition_irq_set_type,
+	.irq_get_irqchip_state	= partition_irq_get_irqchip_state,
+	.irq_set_irqchip_state	= partition_irq_set_irqchip_state,
+	.irq_print_chip		= partition_irq_print_chip,
+};
+
+static void partition_handle_irq(struct irq_desc *desc)
+{
+	struct partition_desc *part = irq_desc_get_handler_data(desc);
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+	int cpu = smp_processor_id();
+	int hwirq;
+
+	chained_irq_enter(chip, desc);
+
+	for_each_set_bit(hwirq, part->bitmap, part->nr_parts) {
+		if (partition_check_cpu(part, cpu, hwirq))
+			break;
+	}
+
+	if (unlikely(hwirq == part->nr_parts)) {
+		handle_bad_irq(desc);
+	} else {
+		unsigned int irq;
+		irq = irq_find_mapping(part->domain, hwirq);
+		generic_handle_irq(irq);
+	}
+
+	chained_irq_exit(chip, desc);
+}
+
+static int partition_domain_alloc(struct irq_domain *domain, unsigned int virq,
+				  unsigned int nr_irqs, void *arg)
+{
+	int ret;
+	irq_hw_number_t hwirq;
+	unsigned int type;
+	struct irq_fwspec *fwspec = arg;
+	struct partition_desc *part;
+
+	BUG_ON(nr_irqs != 1);
+	ret = domain->ops->translate(domain, fwspec, &hwirq, &type);
+	if (ret)
+		return ret;
+
+	part = domain->host_data;
+
+	set_bit(hwirq, part->bitmap);
+	irq_set_chained_handler_and_data(irq_desc_get_irq(part->chained_desc),
+					 partition_handle_irq, part);
+	irq_set_percpu_devid_partition(virq, &part->parts[hwirq].mask);
+	irq_domain_set_info(domain, virq, hwirq, &partition_irq_chip, part,
+			    handle_percpu_devid_irq, NULL, NULL);
+	irq_set_status_flags(virq, IRQ_NOAUTOEN);
+
+	return 0;
+}
+
+static void partition_domain_free(struct irq_domain *domain, unsigned int virq,
+				  unsigned int nr_irqs)
+{
+	struct irq_data *d;
+
+	BUG_ON(nr_irqs != 1);
+
+	d = irq_domain_get_irq_data(domain, virq);
+	irq_set_handler(virq, NULL);
+	irq_domain_reset_irq_data(d);
+}
+
+int partition_translate_id(struct partition_desc *desc, void *partition_id)
+{
+	struct partition_affinity *part = NULL;
+	int i;
+
+	for (i = 0; i < desc->nr_parts; i++) {
+		if (desc->parts[i].partition_id == partition_id) {
+			part = &desc->parts[i];
+			break;
+		}
+	}
+
+	if (WARN_ON(!part)) {
+		pr_err("Failed to find partition\n");
+		return -EINVAL;
+	}
+
+	return i;
+}
+
+struct partition_desc *partition_create_desc(struct fwnode_handle *fwnode,
+					     struct partition_affinity *parts,
+					     int nr_parts,
+					     int chained_irq,
+					     const struct irq_domain_ops *ops)
+{
+	struct partition_desc *desc;
+	struct irq_domain *d;
+
+	BUG_ON(!ops->select || !ops->translate);
+
+	desc = kzalloc(sizeof(*desc), GFP_KERNEL);
+	if (!desc)
+		return NULL;
+
+	desc->ops = *ops;
+	desc->ops.free = partition_domain_free;
+	desc->ops.alloc = partition_domain_alloc;
+
+	d = irq_domain_create_linear(fwnode, nr_parts, &desc->ops, desc);
+	if (!d)
+		goto out;
+	desc->domain = d;
+
+	desc->bitmap = kzalloc(sizeof(long) * BITS_TO_LONGS(nr_parts),
+			       GFP_KERNEL);
+	if (WARN_ON(!desc->bitmap))
+		goto out;
+
+	desc->chained_desc = irq_to_desc(chained_irq);
+	desc->nr_parts = nr_parts;
+	desc->parts = parts;
+
+	return desc;
+out:
+	if (d)
+		irq_domain_remove(d);
+	kfree(desc);
+
+	return NULL;
+}
+
+struct irq_domain *partition_get_domain(struct partition_desc *dsc)
+{
+	if (dsc)
+		return dsc->domain;
+
+	return NULL;
+}
diff --git a/include/linux/irqchip/irq-partition-percpu.h b/include/linux/irqchip/irq-partition-percpu.h
new file mode 100644
index 0000000..87433a5
--- /dev/null
+++ b/include/linux/irqchip/irq-partition-percpu.h
@@ -0,0 +1,59 @@
+/*
+ * Copyright (C) 2016 ARM Limited, All Rights Reserved.
+ * Author: Marc Zyngier <marc.zyngier@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/fwnode.h>
+#include <linux/cpumask.h>
+#include <linux/irqdomain.h>
+
+struct partition_affinity {
+	cpumask_t			mask;
+	void				*partition_id;
+};
+
+struct partition_desc;
+
+#ifdef CONFIG_PARTITION_PERCPU
+int partition_translate_id(struct partition_desc *desc, void *partition_id);
+struct partition_desc *partition_create_desc(struct fwnode_handle *fwnode,
+					     struct partition_affinity *parts,
+					     int nr_parts,
+					     int chained_irq,
+					     const struct irq_domain_ops *ops);
+struct irq_domain *partition_get_domain(struct partition_desc *dsc);
+#else
+static inline int partition_translate_id(struct partition_desc *desc,
+					 void *partition_id)
+{
+	return -EINVAL;
+}
+
+static inline
+struct partition_desc *partition_create_desc(struct fwnode_handle *fwnode,
+					     struct partition_affinity *parts,
+					     int nr_parts,
+					     int chained_irq,
+					     const struct irq_domain_ops *ops)
+{
+	return NULL;
+}
+
+static inline
+struct irq_domain *partition_get_domain(struct partition_desc *dsc)
+{
+	return NULL;
+}
+#endif
-- 
2.1.4

[PATCH 4/5] irqchip/gic-v3: Add support for partitioned PPIs

[Marc Zyngier]

Plug the partitioning layer into the GICv3 PPI code, parsing the
DT and building the partition affinities and providing the generic
code with partition data and callbacks.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 drivers/irqchip/Kconfig      |   1 +
 drivers/irqchip/irq-gic-v3.c | 176 ++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 175 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index ea1836b..6c17de7 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -27,6 +27,7 @@ config ARM_GIC_V3
 	select IRQ_DOMAIN
 	select MULTI_IRQ_HANDLER
 	select IRQ_DOMAIN_HIERARCHY
+	select PARTITION_PERCPU
 
 config ARM_GIC_V3_ITS
 	bool
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 5b7d3c2..f83e5f4 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -29,6 +29,7 @@
 
 #include <linux/irqchip.h>
 #include <linux/irqchip/arm-gic-v3.h>
+#include <linux/irqchip/irq-partition-percpu.h>
 
 #include <asm/cputype.h>
 #include <asm/exception.h>
@@ -44,6 +45,7 @@ struct redist_region {
 };
 
 struct gic_chip_data {
+	struct fwnode_handle	*fwnode;
 	void __iomem		*dist_base;
 	struct redist_region	*redist_regions;
 	struct rdists		rdists;
@@ -51,6 +53,7 @@ struct gic_chip_data {
 	u64			redist_stride;
 	u32			nr_redist_regions;
 	unsigned int		irq_nr;
+	struct partition_desc	*ppi_descs[16];
 };
 
 static struct gic_chip_data gic_data __read_mostly;
@@ -812,10 +815,62 @@ static void gic_irq_domain_free(struct irq_domain *domain, unsigned int virq,
 	}
 }
 
+static int gic_irq_domain_select(struct irq_domain *d,
+				 struct irq_fwspec *fwspec,
+				 enum irq_domain_bus_token bus_token)
+{
+	/* Not for us */
+        if (fwspec->fwnode != d->fwnode)
+		return 0;
+
+	/* If this is not DT, then we have a single domain */
+	if (!is_of_node(fwspec->fwnode))
+		return 1;
+
+	/*
+	 * If this is a PPI and we have a 4th (non-null) parameter,
+	 * then we need to match the partition domain.
+	 */
+	if (fwspec->param_count >= 4 &&
+	    fwspec->param[0] == 1 && fwspec->param[3] != 0)
+		return d == partition_get_domain(gic_data.ppi_descs[fwspec->param[1]]);
+
+	return d == gic_data.domain;
+}
+
 static const struct irq_domain_ops gic_irq_domain_ops = {
 	.translate = gic_irq_domain_translate,
 	.alloc = gic_irq_domain_alloc,
 	.free = gic_irq_domain_free,
+	.select = gic_irq_domain_select,
+};
+
+static int partition_domain_translate(struct irq_domain *d,
+				      struct irq_fwspec *fwspec,
+				      unsigned long *hwirq,
+				      unsigned int *type)
+{
+	struct device_node *np;
+	int ret;
+
+	np = of_find_node_by_phandle(fwspec->param[3]);
+	if (WARN_ON(!np))
+		return -EINVAL;
+
+	ret = partition_translate_id(gic_data.ppi_descs[fwspec->param[1]],
+				     of_node_to_fwnode(np));
+	if (ret < 0)
+		return ret;
+
+	*hwirq = ret;
+	*type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
+
+	return 0;
+}
+
+static const struct irq_domain_ops partition_domain_ops = {
+	.translate = partition_domain_translate,
+	.select = gic_irq_domain_select,
 };
 
 static void gicv3_enable_quirks(void)
@@ -843,6 +898,7 @@ static int __init gic_init_bases(void __iomem *dist_base,
 	if (static_key_true(&supports_deactivate))
 		pr_info("GIC: Using split EOI/Deactivate mode\n");
 
+	gic_data.fwnode = handle;
 	gic_data.dist_base = dist_base;
 	gic_data.redist_regions = rdist_regs;
 	gic_data.nr_redist_regions = nr_redist_regions;
@@ -901,6 +957,119 @@ static int __init gic_validate_dist_version(void __iomem *dist_base)
 	return 0;
 }
 
+static int get_cpu_number(struct device_node *dn)
+{
+	const __be32 *cell;
+	u64 hwid;
+	int i;
+
+	cell = of_get_property(dn, "reg", NULL);
+	if (!cell)
+		return -1;
+
+	hwid = of_read_number(cell, of_n_addr_cells(dn));
+
+	/*
+	 * Non affinity bits must be set to 0 in the DT
+	 */
+	if (hwid & ~MPIDR_HWID_BITMASK)
+		return -1;
+
+	for (i = 0; i < num_possible_cpus(); i++)
+		if (cpu_logical_map(i) == hwid)
+			return i;
+
+	return -1;
+}
+
+/* Create all possible partitions at boot time */
+static void gic_populate_ppi_partitions(struct device_node *gic_node)
+{
+	struct device_node *parts_node, *child_part;
+	int part_idx = 0, i;
+	int nr_parts;
+	struct partition_affinity *parts;
+
+	parts_node = of_find_node_by_name(gic_node, "ppi-partitions");
+	if (!parts_node)
+		return;
+
+	nr_parts = of_get_child_count(parts_node);
+
+	if (!nr_parts)
+		return;
+
+	parts = kzalloc(sizeof(*parts) * nr_parts, GFP_KERNEL);
+	if (WARN_ON(!parts))
+		return;
+
+	for_each_child_of_node(parts_node, child_part) {
+		struct partition_affinity *part;
+		int n;
+
+		part = &parts[part_idx];
+
+		part->partition_id = of_node_to_fwnode(child_part);
+
+		pr_info("GIC: PPI partition %s[%d] { ",
+			child_part->name, part_idx);
+
+		n = of_property_count_elems_of_size(child_part, "affinity",
+						    sizeof(u32));
+		WARN_ON(n <= 0);
+
+		for (i = 0; i < n; i++) {
+			int err, cpu;
+			u32 cpu_phandle;
+			struct device_node *cpu_node;
+
+			err = of_property_read_u32_index(child_part, "affinity",
+							 i, &cpu_phandle);
+			if (WARN_ON(err))
+				continue;
+
+			cpu_node = of_find_node_by_phandle(cpu_phandle);
+			if (WARN_ON(!cpu_node))
+				continue;
+
+			cpu = get_cpu_number(cpu_node);
+			if (WARN_ON(cpu == -1))
+				continue;
+
+			pr_cont("%s[%d] ", cpu_node->full_name, cpu);
+
+			cpumask_set_cpu(cpu, &part->mask);
+		}
+
+		pr_cont("}\n");
+		part_idx++;
+	}
+
+	for (i = 0; i < 16; i++) {
+		unsigned int irq;
+		struct partition_desc *desc;
+		struct irq_fwspec ppi_fwspec = {
+			.fwnode		= gic_data.fwnode,
+			.param_count	= 3,
+			.param		= {
+				[0]	= 1,
+				[1]	= i,
+				[2]	= IRQ_TYPE_NONE,
+			},
+		};
+
+		irq = irq_create_fwspec_mapping(&ppi_fwspec);
+		if (WARN_ON(!irq))
+			continue;
+		desc = partition_create_desc(gic_data.fwnode, parts, nr_parts,
+					     irq, &partition_domain_ops);
+		if (WARN_ON(!desc))
+			continue;
+
+		gic_data.ppi_descs[i] = desc;
+	}
+}
+
 static int __init gic_of_init(struct device_node *node, struct device_node *parent)
 {
 	void __iomem *dist_base;
@@ -952,8 +1121,11 @@ static int __init gic_of_init(struct device_node *node, struct device_node *pare
 
 	err = gic_init_bases(dist_base, rdist_regs, nr_redist_regions,
 			     redist_stride, &node->fwnode);
-	if (!err)
-		return 0;
+	if (err)
+		goto out_unmap_rdist;
+
+	gic_populate_ppi_partitions(node);
+	return 0;
 
 out_unmap_rdist:
 	for (i = 0; i < nr_redist_regions; i++)
-- 
2.1.4

[PATCH 5/5] DT: arm,gic-v3: Documment PPI partition support

[Marc Zyngier]

Add a decription of the PPI partitioning support.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 .../bindings/interrupt-controller/arm,gic-v3.txt   | 34 ++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
index 007a5b4..4c29cda 100644
--- a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
+++ b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
@@ -11,6 +11,8 @@ Main node required properties:
 - interrupt-controller : Identifies the node as an interrupt controller
 - #interrupt-cells : Specifies the number of cells needed to encode an
   interrupt source. Must be a single cell with a value of at least 3.
+  If the system requires describing PPI affinity, then the value must
+  be at least 4.
 
   The 1st cell is the interrupt type; 0 for SPI interrupts, 1 for PPI
   interrupts. Other values are reserved for future use.
@@ -24,7 +26,14 @@ Main node required properties:
 		1 = edge triggered
 		4 = level triggered
 
-  Cells 4 and beyond are reserved for future use and must have a value
+  The 4th cell is a phandle to a node describing a set of CPUs this
+  interrupt is affine to. The interrupt must be a PPI, and the node
+  pointed must be a subnode of the "ppi-partitions" subnode. For
+  interrupt types other than PPI or PPIs that are not partitionned,
+  this cell must be zero. See the "ppi-partitions" node description
+  below.
+
+  Cells 5 and beyond are reserved for future use and must have a value
   of 0 if present.
 
 - reg : Specifies base physical address(s) and size of the GIC
@@ -50,6 +59,11 @@ Optional
 
 Sub-nodes:
 
+PPI affinity can be expressed as a single "ppi-partitions" node,
+containing a set of sub-nodes, each with the following property:
+- affinity: Should be a list of phandles to CPU nodes (as described in
+Documentation/devicetree/bindings/arm/cpus.txt).
+
 GICv3 has one or more Interrupt Translation Services (ITS) that are
 used to route Message Signalled Interrupts (MSI) to the CPUs.
 
@@ -91,7 +105,7 @@ Examples:
 
 	gic: interrupt-controller@2c010000 {
 		compatible = "arm,gic-v3";
-		#interrupt-cells = <3>;
+		#interrupt-cells = <4>;
 		#address-cells = <2>;
 		#size-cells = <2>;
 		ranges;
@@ -119,4 +133,20 @@ Examples:
 			#msi-cells = <1>;
 			reg = <0x0 0x2c400000 0 0x200000>;
 		};
+
+		ppi-partitions {
+			part0: interrupt-partition-0 {
+				affinity = <&cpu0 &cpu2>;
+			};
+
+			part1: interrupt-partition-1 {
+				affinity = <&cpu1 &cpu3>;
+			};
+		};
+	};
+
+
+	device@0 {
+		reg = <0 0 0 4>;
+		interrupts = <1 1 4 &part0>;
 	};
-- 
2.1.4

Re: [PATCH 5/5] DT: arm,gic-v3: Documment PPI partition support

[Rob Herring]

On Mon, Apr 11, 2016 at 09:57:55AM +0100, Marc Zyngier wrote:
> Add a decription of the PPI partitioning support.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  .../bindings/interrupt-controller/arm,gic-v3.txt   | 34 ++++++++++++++++++++--
>  1 file changed, 32 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
> index 007a5b4..4c29cda 100644
> --- a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
> +++ b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
> @@ -11,6 +11,8 @@ Main node required properties:
>  - interrupt-controller : Identifies the node as an interrupt controller
>  - #interrupt-cells : Specifies the number of cells needed to encode an
>    interrupt source. Must be a single cell with a value of at least 3.
> +  If the system requires describing PPI affinity, then the value must
> +  be at least 4.

You're winning for cell count...

One alternative that would save adding a cell and keep it contained 
within would be just list the affinities in the GIC node in the form of 
'<PPI#> <count> <cpu phandles>':

ppi-affinity = <1 2 &cpu2 &cpu3>,
		<5 1 &cpu4>,
		...

This would be harder to parse though if you have a large number of PPIs 
with affinities.

That said, I've got no real issue with this as is.

Rob

Re: [PATCH 5/5] DT: arm,gic-v3: Documment PPI partition support

[Marc Zyngier]

Hi Rob,

On 12/04/16 17:29, Rob Herring wrote:
> On Mon, Apr 11, 2016 at 09:57:55AM +0100, Marc Zyngier wrote:
>> Add a decription of the PPI partitioning support.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  .../bindings/interrupt-controller/arm,gic-v3.txt   | 34 ++++++++++++++++++++--
>>  1 file changed, 32 insertions(+), 2 deletions(-)
>>
>> diff --git a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
>> index 007a5b4..4c29cda 100644
>> --- a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
>> +++ b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
>> @@ -11,6 +11,8 @@ Main node required properties:
>>  - interrupt-controller : Identifies the node as an interrupt controller
>>  - #interrupt-cells : Specifies the number of cells needed to encode an
>>    interrupt source. Must be a single cell with a value of at least 3.
>> +  If the system requires describing PPI affinity, then the value must
>> +  be at least 4.
> 
> You're winning for cell count...

Yeah, it feels like we aim at making people's life difficult...

> One alternative that would save adding a cell and keep it contained 
> within would be just list the affinities in the GIC node in the form of 
> '<PPI#> <count> <cpu phandles>':
> 
> ppi-affinity = <1 2 &cpu2 &cpu3>,
> 		<5 1 &cpu4>,
> 		...

But how would that work if you have two sets of CPUs (set-1=[cpu0,
cpu1]; set-2=[cpu2, cpu3]), and for the same PPI, device A is connected
to set-1 and device-B is connected to set-2?

You need a way to distinguish these two interrupts and so far, the only
way I've found is to reference the affinity in the interrupt specifier.

That being said, I'm definitely open to suggestions on how to describe
this in a better way.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 5/5] DT: arm,gic-v3: Documment PPI partition support

[Rob Herring]

On Tue, Apr 12, 2016 at 11:42 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> Hi Rob,
>
> On 12/04/16 17:29, Rob Herring wrote:
>> On Mon, Apr 11, 2016 at 09:57:55AM +0100, Marc Zyngier wrote:
>>> Add a decription of the PPI partitioning support.
>>>
>>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>>> ---
>>>  .../bindings/interrupt-controller/arm,gic-v3.txt   | 34 ++++++++++++++++++++--
>>>  1 file changed, 32 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
>>> index 007a5b4..4c29cda 100644
>>> --- a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
>>> +++ b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
>>> @@ -11,6 +11,8 @@ Main node required properties:
>>>  - interrupt-controller : Identifies the node as an interrupt controller
>>>  - #interrupt-cells : Specifies the number of cells needed to encode an
>>>    interrupt source. Must be a single cell with a value of at least 3.
>>> +  If the system requires describing PPI affinity, then the value must
>>> +  be at least 4.
>>
>> You're winning for cell count...
>
> Yeah, it feels like we aim at making people's life difficult...
>
>> One alternative that would save adding a cell and keep it contained
>> within would be just list the affinities in the GIC node in the form of
>> '<PPI#> <count> <cpu phandles>':
>>
>> ppi-affinity = <1 2 &cpu2 &cpu3>,
>>               <5 1 &cpu4>,
>>               ...
>
> But how would that work if you have two sets of CPUs (set-1=[cpu0,
> cpu1]; set-2=[cpu2, cpu3]), and for the same PPI, device A is connected
> to set-1 and device-B is connected to set-2?

Oh right. Need to take those h/w designers out back...

> You need a way to distinguish these two interrupts and so far, the only
> way I've found is to reference the affinity in the interrupt specifier.
>
> That being said, I'm definitely open to suggestions on how to describe
> this in a better way.

In that case, I think it looks fine.

Acked-by: Rob Herring <robh@kernel.org>

Rob

Re: [PATCH 0/5] Partitioning per-cpu interrupts

[Marc Zyngier]

On 11/04/16 09:57, Marc Zyngier wrote:
> We've unfortunately started seeing a situation where percpu interrupts
> are partitioned in the system: one arbitrary set of CPUs has an
> interrupt connected to a type of device, while another disjoint set of
> CPUs has the same interrupt connected to another type of device.
>     
> This makes it impossible to have a device driver requesting this
> interrupt using the current percpu-interrupt abstraction, as the same
> interrupt number is now potentially claimed by at least two drivers,
> and we forbid interrupt sharing on per-cpu interrupt.
> 
> A potential solution to this has been proposed by Will Deacon,
> expanding the handling in the core code:
> 
> http://lists.infradead.org/pipermail/linux-arm-kernel/2015-November/388800.html
> 
> followed by a counter-proposal from Thomas Gleixner, which Will tried
> to implement, but ran into issues where the probing code was running
> in preemptible context, making the percpu-ness of interrupts difficult
> to guarantee.
>     
> Another approach to this is to turn things upside down. Let's assume
> that our system describes all the possible partitions for a given
> interrupt, and give each of them a unique identifier. It is then
> possible to create a namespace where the affinity identifier itself is
> a form of interrupt number. At this point, it becomes easy to
> implement a set of partitions as a cascaded irqchip, each affinity
> identifier being the secondary HW irq, as outlined in the following
> example:
> 
> 	Aff-0: { cpu0 cpu3 }
> 	Aff-1: { cpu1 cpu2 }
> 	Aff-2: { cpu4 cpu5 cpu6 cpu7 }
> 
> Let's assume that HW interrupt 1 is partitioned over these 3
> affinities. When HW interrupt 1 fires on a given CPU, all it takes is
> to find out which affinity this CPU belongs to, which gives us a new
> HW interrupt number. Bingo. Of course, this only works as long as you
> don't have overlapping affinities (but if you do your system is broken
> anyway).
>     
> This allows us to keep a number of nice properties:
> 
> - Each partition results in a separate percpu-interrupt (with a
>   restricted affinity), which keeps drivers happy. This alone
>   garantees that we do not have to change the programming model for
>   per-cpu interrupts.
> 
> - Because the underlying interrupt is still per-cpu, the overhead of
>   the indirection can be kept pretty minimal.
> 
> - The core code can ignore most of that crap.
>     
> For that purpose, we implement a small library that deals with some of
> the boilerplate code, relying on platform-specific drivers to provide
> a description of the affinity sets and a set of callbacks. This also
> relies on a small change in the irqdomain layer, and now offers a way
> for the affinity of a percpu interrupt to be retrieved by a driver.
> 
> As an example, the GICv3 driver has been adapted to use this new
> feature. Patches on top of v4.6-r3, tested on an arm64 FVP model.

Any comment on this? The Rockchip dudes have confirmed that this solves
their problems (big-little system with PMUs using the same PPI).

I've also posted a proof of concept patch for the ARM PMU over there:

https://lkml.org/lkml/2016/4/25/227

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 0/5] Partitioning per-cpu interrupts

[Thomas Gleixner]

On Thu, 28 Apr 2016, Marc Zyngier wrote:
> Any comment on this? The Rockchip dudes have confirmed that this solves
> their problems (big-little system with PMUs using the same PPI).

It's in my backlog ....

[tip:irq/core] irqdomain: Allow domain matching on irq_fwspec

[tip-bot for Marc Zyngier]

Commit-ID:  651e8b54abdeeaa36f5f54ffa05c18707a3cc1d0
Gitweb:     http://git.kernel.org/tip/651e8b54abdeeaa36f5f54ffa05c18707a3cc1d0
Author:     Marc Zyngier <marc.zyngier@arm.com>
AuthorDate: Mon, 11 Apr 2016 09:57:51 +0100
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Mon, 2 May 2016 13:42:50 +0200

irqdomain: Allow domain matching on irq_fwspec

When iterating over the irq domain list, we try to match a domain
either by calling a match() function or by comparing a number
of fields passed as parameters.

Both approaches are a bit restrictive:
- match() is DT specific and only takes a device node
- the fallback case only deals with the fwnode_handle

It would be useful if we had a per-domain function that would
actually perform the matching check on the whole of the
irq_fwspec structure. This would allow for a domain to triage
matching attempts that need to extend beyond the fwnode.

Let's introduce irq_find_matching_fwspec(), which takes a full
blown irq_fwspec structure, and call into a select() function
implemented by the irqdomain. irq_find_matching_fwnode() is
made a wrapper around irq_find_matching_fwspec in order to
preserve compatibility.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: devicetree@vger.kernel.org
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Rob Herring <robh+dt@kernel.org>
Link: http://lkml.kernel.org/r/1460365075-7316-2-git-send-email-marc.zyngier@arm.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

---
 include/linux/irqdomain.h | 15 ++++++++++++++-
 kernel/irq/irqdomain.c    | 19 ++++++++++---------
 2 files changed, 24 insertions(+), 10 deletions(-)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index 736abd7..f1f36e0 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -96,6 +96,8 @@ enum irq_domain_bus_token {
 struct irq_domain_ops {
 	int (*match)(struct irq_domain *d, struct device_node *node,
 		     enum irq_domain_bus_token bus_token);
+	int (*select)(struct irq_domain *d, struct irq_fwspec *fwspec,
+		      enum irq_domain_bus_token bus_token);
 	int (*map)(struct irq_domain *d, unsigned int virq, irq_hw_number_t hw);
 	void (*unmap)(struct irq_domain *d, unsigned int virq);
 	int (*xlate)(struct irq_domain *d, struct device_node *node,
@@ -211,7 +213,7 @@ struct irq_domain *irq_domain_add_legacy(struct device_node *of_node,
 					 irq_hw_number_t first_hwirq,
 					 const struct irq_domain_ops *ops,
 					 void *host_data);
-extern struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
+extern struct irq_domain *irq_find_matching_fwspec(struct irq_fwspec *fwspec,
 						   enum irq_domain_bus_token bus_token);
 extern void irq_set_default_host(struct irq_domain *host);
 extern int irq_domain_alloc_descs(int virq, unsigned int nr_irqs,
@@ -227,6 +229,17 @@ static inline bool is_fwnode_irqchip(struct fwnode_handle *fwnode)
 	return fwnode && fwnode->type == FWNODE_IRQCHIP;
 }
 
+static inline
+struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
+					    enum irq_domain_bus_token bus_token)
+{
+	struct irq_fwspec fwspec = {
+		.fwnode = fwnode,
+	};
+
+	return irq_find_matching_fwspec(&fwspec, bus_token);
+}
+
 static inline struct irq_domain *irq_find_matching_host(struct device_node *node,
 							enum irq_domain_bus_token bus_token)
 {
diff --git a/kernel/irq/irqdomain.c b/kernel/irq/irqdomain.c
index 3a519a0..503c5b9 100644
--- a/kernel/irq/irqdomain.c
+++ b/kernel/irq/irqdomain.c
@@ -243,14 +243,15 @@ struct irq_domain *irq_domain_add_legacy(struct device_node *of_node,
 EXPORT_SYMBOL_GPL(irq_domain_add_legacy);
 
 /**
- * irq_find_matching_fwnode() - Locates a domain for a given fwnode
- * @fwnode: FW descriptor of the interrupt controller
+ * irq_find_matching_fwspec() - Locates a domain for a given fwspec
+ * @fwspec: FW specifier for an interrupt
  * @bus_token: domain-specific data
  */
-struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
+struct irq_domain *irq_find_matching_fwspec(struct irq_fwspec *fwspec,
 					    enum irq_domain_bus_token bus_token)
 {
 	struct irq_domain *h, *found = NULL;
+	struct fwnode_handle *fwnode = fwspec->fwnode;
 	int rc;
 
 	/* We might want to match the legacy controller last since
@@ -264,7 +265,9 @@ struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
 	 */
 	mutex_lock(&irq_domain_mutex);
 	list_for_each_entry(h, &irq_domain_list, link) {
-		if (h->ops->match)
+		if (h->ops->select && fwspec->param_count)
+			rc = h->ops->select(h, fwspec, bus_token);
+		else if (h->ops->match)
 			rc = h->ops->match(h, to_of_node(fwnode), bus_token);
 		else
 			rc = ((fwnode != NULL) && (h->fwnode == fwnode) &&
@@ -279,7 +282,7 @@ struct irq_domain *irq_find_matching_fwnode(struct fwnode_handle *fwnode,
 	mutex_unlock(&irq_domain_mutex);
 	return found;
 }
-EXPORT_SYMBOL_GPL(irq_find_matching_fwnode);
+EXPORT_SYMBOL_GPL(irq_find_matching_fwspec);
 
 /**
  * irq_set_default_host() - Set a "default" irq domain
@@ -574,11 +577,9 @@ unsigned int irq_create_fwspec_mapping(struct irq_fwspec *fwspec)
 	int virq;
 
 	if (fwspec->fwnode) {
-		domain = irq_find_matching_fwnode(fwspec->fwnode,
-						  DOMAIN_BUS_WIRED);
+		domain = irq_find_matching_fwspec(fwspec, DOMAIN_BUS_WIRED);
 		if (!domain)
-			domain = irq_find_matching_fwnode(fwspec->fwnode,
-							  DOMAIN_BUS_ANY);
+			domain = irq_find_matching_fwspec(fwspec, DOMAIN_BUS_ANY);
 	} else {
 		domain = irq_default_domain;
 	}

[tip:irq/core] genirq: Allow the affinity of a percpu interrupt to be set/retrieved

[tip-bot for Marc Zyngier]

Commit-ID:  222df54fd8b7641dcc81476f157806bb3144ee1d
Gitweb:     http://git.kernel.org/tip/222df54fd8b7641dcc81476f157806bb3144ee1d
Author:     Marc Zyngier <marc.zyngier@arm.com>
AuthorDate: Mon, 11 Apr 2016 09:57:52 +0100
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Mon, 2 May 2016 13:42:51 +0200

genirq: Allow the affinity of a percpu interrupt to be set/retrieved

In order to prepare the genirq layer for the concept of partitionned
percpu interrupts, let's allow an affinity to be associated with
such an interrupt. We introduce:

- irq_set_percpu_devid_partition: flag an interrupt as a percpu-devid
  interrupt, and associate it with an affinity
- irq_get_percpu_devid_partition: allow the affinity of that interrupt
  to be retrieved.

This will allow a driver to discover which CPUs the per-cpu interrupt
can actually fire on.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: devicetree@vger.kernel.org
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Rob Herring <robh+dt@kernel.org>
Link: http://lkml.kernel.org/r/1460365075-7316-3-git-send-email-marc.zyngier@arm.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

---
 include/linux/irq.h     |  4 ++++
 include/linux/irqdesc.h |  1 +
 kernel/irq/irqdesc.c    | 26 +++++++++++++++++++++++++-
 3 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/include/linux/irq.h b/include/linux/irq.h
index c4de623..4d758a7 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -530,6 +530,10 @@ static inline void irq_set_chip_and_handler(unsigned int irq, struct irq_chip *c
 }
 
 extern int irq_set_percpu_devid(unsigned int irq);
+extern int irq_set_percpu_devid_partition(unsigned int irq,
+					  const struct cpumask *affinity);
+extern int irq_get_percpu_devid_partition(unsigned int irq,
+					  struct cpumask *affinity);
 
 extern void
 __irq_set_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
diff --git a/include/linux/irqdesc.h b/include/linux/irqdesc.h
index dcca77c..b51beeb 100644
--- a/include/linux/irqdesc.h
+++ b/include/linux/irqdesc.h
@@ -66,6 +66,7 @@ struct irq_desc {
 	int			threads_handled_last;
 	raw_spinlock_t		lock;
 	struct cpumask		*percpu_enabled;
+	const struct cpumask	*percpu_affinity;
 #ifdef CONFIG_SMP
 	const struct cpumask	*affinity_hint;
 	struct irq_affinity_notify *affinity_notify;
diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c
index 0ccd028..8731e1c 100644
--- a/kernel/irq/irqdesc.c
+++ b/kernel/irq/irqdesc.c
@@ -595,7 +595,8 @@ void __irq_put_desc_unlock(struct irq_desc *desc, unsigned long flags, bool bus)
 		chip_bus_sync_unlock(desc);
 }
 
-int irq_set_percpu_devid(unsigned int irq)
+int irq_set_percpu_devid_partition(unsigned int irq,
+				   const struct cpumask *affinity)
 {
 	struct irq_desc *desc = irq_to_desc(irq);
 
@@ -610,10 +611,33 @@ int irq_set_percpu_devid(unsigned int irq)
 	if (!desc->percpu_enabled)
 		return -ENOMEM;
 
+	if (affinity)
+		desc->percpu_affinity = affinity;
+	else
+		desc->percpu_affinity = cpu_possible_mask;
+
 	irq_set_percpu_devid_flags(irq);
 	return 0;
 }
 
+int irq_set_percpu_devid(unsigned int irq)
+{
+	return irq_set_percpu_devid_partition(irq, NULL);
+}
+
+int irq_get_percpu_devid_partition(unsigned int irq, struct cpumask *affinity)
+{
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	if (!desc || !desc->percpu_enabled)
+		return -EINVAL;
+
+	if (affinity)
+		cpumask_copy(affinity, desc->percpu_affinity);
+
+	return 0;
+}
+
 void kstat_incr_irq_this_cpu(unsigned int irq)
 {
 	kstat_incr_irqs_this_cpu(irq_to_desc(irq));

[tip:irq/core] irqchip: Add per-cpu interrupt partitioning library

[tip-bot for Marc Zyngier]

Commit-ID:  9e2c986cb460bf97154f18e85aa833739a1e8dc7
Gitweb:     http://git.kernel.org/tip/9e2c986cb460bf97154f18e85aa833739a1e8dc7
Author:     Marc Zyngier <marc.zyngier@arm.com>
AuthorDate: Mon, 11 Apr 2016 09:57:53 +0100
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Mon, 2 May 2016 13:42:51 +0200

irqchip: Add per-cpu interrupt partitioning library

We've unfortunately started seeing a situation where percpu interrupts
are partitioned in the system: one arbitrary set of CPUs has an
interrupt connected to a type of device, while another disjoint
set of CPUs has the same interrupt connected to another type of device.

This makes it impossible to have a device driver requesting this interrupt
using the current percpu-interrupt abstraction, as the same interrupt number
is now potentially claimed by at least two drivers, and we forbid interrupt
sharing on per-cpu interrupt.

A solution to this is to turn things upside down. Let's assume that our
system describes all the possible partitions for a given interrupt, and
give each of them a unique identifier. It is then possible to create
a namespace where the affinity identifier itself is a form of interrupt
number. At this point, it becomes easy to implement a set of partitions
as a cascaded irqchip, each affinity identifier being the HW irq.

This allows us to keep a number of nice properties:
- Each partition results in a separate percpu-interrupt (with a restrictied
  affinity), which keeps drivers happy.
- Because the underlying interrupt is still per-cpu, the overhead of
  the indirection can be kept pretty minimal.
- The core code can ignore most of that crap.

For that purpose, we implement a small library that deals with some of
the boilerplate code, relying on platform-specific drivers to provide
a description of the affinity sets and a set of callbacks.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: devicetree@vger.kernel.org
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Rob Herring <robh+dt@kernel.org>
Link: http://lkml.kernel.org/r/1460365075-7316-4-git-send-email-marc.zyngier@arm.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

---
 drivers/irqchip/Kconfig                      |   3 +
 drivers/irqchip/Makefile                     |   1 +
 drivers/irqchip/irq-partition-percpu.c       | 256 +++++++++++++++++++++++++++
 include/linux/irqchip/irq-partition-percpu.h |  59 ++++++
 4 files changed, 319 insertions(+)

diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index 3e12479..ea1836b 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -244,3 +244,6 @@ config IRQ_MXS
 config MVEBU_ODMI
 	bool
 	select GENERIC_MSI_IRQ_DOMAIN
+
+config PARTITION_PERCPU
+	bool
diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
index b03cfcb..e354b00c 100644
--- a/drivers/irqchip/Makefile
+++ b/drivers/irqchip/Makefile
@@ -27,6 +27,7 @@ obj-$(CONFIG_REALVIEW_DT)		+= irq-gic-realview.o
 obj-$(CONFIG_ARM_GIC_V2M)		+= irq-gic-v2m.o
 obj-$(CONFIG_ARM_GIC_V3)		+= irq-gic-v3.o irq-gic-common.o
 obj-$(CONFIG_ARM_GIC_V3_ITS)		+= irq-gic-v3-its.o irq-gic-v3-its-pci-msi.o irq-gic-v3-its-platform-msi.o
+obj-$(CONFIG_PARTITION_PERCPU)		+= irq-partition-percpu.o
 obj-$(CONFIG_HISILICON_IRQ_MBIGEN)	+= irq-mbigen.o
 obj-$(CONFIG_ARM_NVIC)			+= irq-nvic.o
 obj-$(CONFIG_ARM_VIC)			+= irq-vic.o
diff --git a/drivers/irqchip/irq-partition-percpu.c b/drivers/irqchip/irq-partition-percpu.c
new file mode 100644
index 0000000..ccd72c2
--- /dev/null
+++ b/drivers/irqchip/irq-partition-percpu.c
@@ -0,0 +1,256 @@
+/*
+ * Copyright (C) 2016 ARM Limited, All Rights Reserved.
+ * Author: Marc Zyngier <marc.zyngier@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/bitops.h>
+#include <linux/interrupt.h>
+#include <linux/irqchip.h>
+#include <linux/irqchip/chained_irq.h>
+#include <linux/irqchip/irq-partition-percpu.h>
+#include <linux/irqdomain.h>
+#include <linux/seq_file.h>
+#include <linux/slab.h>
+
+struct partition_desc {
+	int				nr_parts;
+	struct partition_affinity	*parts;
+	struct irq_domain		*domain;
+	struct irq_desc			*chained_desc;
+	unsigned long			*bitmap;
+	struct irq_domain_ops		ops;
+};
+
+static bool partition_check_cpu(struct partition_desc *part,
+				unsigned int cpu, unsigned int hwirq)
+{
+	return cpumask_test_cpu(cpu, &part->parts[hwirq].mask);
+}
+
+static void partition_irq_mask(struct irq_data *d)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (partition_check_cpu(part, smp_processor_id(), d->hwirq) &&
+	    chip->irq_mask)
+		chip->irq_mask(data);
+}
+
+static void partition_irq_unmask(struct irq_data *d)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (partition_check_cpu(part, smp_processor_id(), d->hwirq) &&
+	    chip->irq_unmask)
+		chip->irq_unmask(data);
+}
+
+static int partition_irq_set_irqchip_state(struct irq_data *d,
+					   enum irqchip_irq_state which,
+					   bool val)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (partition_check_cpu(part, smp_processor_id(), d->hwirq) &&
+	    chip->irq_set_irqchip_state)
+		return chip->irq_set_irqchip_state(data, which, val);
+
+	return -EINVAL;
+}
+
+static int partition_irq_get_irqchip_state(struct irq_data *d,
+					   enum irqchip_irq_state which,
+					   bool *val)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (partition_check_cpu(part, smp_processor_id(), d->hwirq) &&
+	    chip->irq_get_irqchip_state)
+		return chip->irq_get_irqchip_state(data, which, val);
+
+	return -EINVAL;
+}
+
+static int partition_irq_set_type(struct irq_data *d, unsigned int type)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	if (chip->irq_set_type)
+		return chip->irq_set_type(data, type);
+
+	return -EINVAL;
+}
+
+static void partition_irq_print_chip(struct irq_data *d, struct seq_file *p)
+{
+	struct partition_desc *part = irq_data_get_irq_chip_data(d);
+	struct irq_chip *chip = irq_desc_get_chip(part->chained_desc);
+	struct irq_data *data = irq_desc_get_irq_data(part->chained_desc);
+
+	seq_printf(p, " %5s-%lu", chip->name, data->hwirq);
+}
+
+static struct irq_chip partition_irq_chip = {
+	.irq_mask		= partition_irq_mask,
+	.irq_unmask		= partition_irq_unmask,
+	.irq_set_type		= partition_irq_set_type,
+	.irq_get_irqchip_state	= partition_irq_get_irqchip_state,
+	.irq_set_irqchip_state	= partition_irq_set_irqchip_state,
+	.irq_print_chip		= partition_irq_print_chip,
+};
+
+static void partition_handle_irq(struct irq_desc *desc)
+{
+	struct partition_desc *part = irq_desc_get_handler_data(desc);
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+	int cpu = smp_processor_id();
+	int hwirq;
+
+	chained_irq_enter(chip, desc);
+
+	for_each_set_bit(hwirq, part->bitmap, part->nr_parts) {
+		if (partition_check_cpu(part, cpu, hwirq))
+			break;
+	}
+
+	if (unlikely(hwirq == part->nr_parts)) {
+		handle_bad_irq(desc);
+	} else {
+		unsigned int irq;
+		irq = irq_find_mapping(part->domain, hwirq);
+		generic_handle_irq(irq);
+	}
+
+	chained_irq_exit(chip, desc);
+}
+
+static int partition_domain_alloc(struct irq_domain *domain, unsigned int virq,
+				  unsigned int nr_irqs, void *arg)
+{
+	int ret;
+	irq_hw_number_t hwirq;
+	unsigned int type;
+	struct irq_fwspec *fwspec = arg;
+	struct partition_desc *part;
+
+	BUG_ON(nr_irqs != 1);
+	ret = domain->ops->translate(domain, fwspec, &hwirq, &type);
+	if (ret)
+		return ret;
+
+	part = domain->host_data;
+
+	set_bit(hwirq, part->bitmap);
+	irq_set_chained_handler_and_data(irq_desc_get_irq(part->chained_desc),
+					 partition_handle_irq, part);
+	irq_set_percpu_devid_partition(virq, &part->parts[hwirq].mask);
+	irq_domain_set_info(domain, virq, hwirq, &partition_irq_chip, part,
+			    handle_percpu_devid_irq, NULL, NULL);
+	irq_set_status_flags(virq, IRQ_NOAUTOEN);
+
+	return 0;
+}
+
+static void partition_domain_free(struct irq_domain *domain, unsigned int virq,
+				  unsigned int nr_irqs)
+{
+	struct irq_data *d;
+
+	BUG_ON(nr_irqs != 1);
+
+	d = irq_domain_get_irq_data(domain, virq);
+	irq_set_handler(virq, NULL);
+	irq_domain_reset_irq_data(d);
+}
+
+int partition_translate_id(struct partition_desc *desc, void *partition_id)
+{
+	struct partition_affinity *part = NULL;
+	int i;
+
+	for (i = 0; i < desc->nr_parts; i++) {
+		if (desc->parts[i].partition_id == partition_id) {
+			part = &desc->parts[i];
+			break;
+		}
+	}
+
+	if (WARN_ON(!part)) {
+		pr_err("Failed to find partition\n");
+		return -EINVAL;
+	}
+
+	return i;
+}
+
+struct partition_desc *partition_create_desc(struct fwnode_handle *fwnode,
+					     struct partition_affinity *parts,
+					     int nr_parts,
+					     int chained_irq,
+					     const struct irq_domain_ops *ops)
+{
+	struct partition_desc *desc;
+	struct irq_domain *d;
+
+	BUG_ON(!ops->select || !ops->translate);
+
+	desc = kzalloc(sizeof(*desc), GFP_KERNEL);
+	if (!desc)
+		return NULL;
+
+	desc->ops = *ops;
+	desc->ops.free = partition_domain_free;
+	desc->ops.alloc = partition_domain_alloc;
+
+	d = irq_domain_create_linear(fwnode, nr_parts, &desc->ops, desc);
+	if (!d)
+		goto out;
+	desc->domain = d;
+
+	desc->bitmap = kzalloc(sizeof(long) * BITS_TO_LONGS(nr_parts),
+			       GFP_KERNEL);
+	if (WARN_ON(!desc->bitmap))
+		goto out;
+
+	desc->chained_desc = irq_to_desc(chained_irq);
+	desc->nr_parts = nr_parts;
+	desc->parts = parts;
+
+	return desc;
+out:
+	if (d)
+		irq_domain_remove(d);
+	kfree(desc);
+
+	return NULL;
+}
+
+struct irq_domain *partition_get_domain(struct partition_desc *dsc)
+{
+	if (dsc)
+		return dsc->domain;
+
+	return NULL;
+}
diff --git a/include/linux/irqchip/irq-partition-percpu.h b/include/linux/irqchip/irq-partition-percpu.h
new file mode 100644
index 0000000..87433a5
--- /dev/null
+++ b/include/linux/irqchip/irq-partition-percpu.h
@@ -0,0 +1,59 @@
+/*
+ * Copyright (C) 2016 ARM Limited, All Rights Reserved.
+ * Author: Marc Zyngier <marc.zyngier@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/fwnode.h>
+#include <linux/cpumask.h>
+#include <linux/irqdomain.h>
+
+struct partition_affinity {
+	cpumask_t			mask;
+	void				*partition_id;
+};
+
+struct partition_desc;
+
+#ifdef CONFIG_PARTITION_PERCPU
+int partition_translate_id(struct partition_desc *desc, void *partition_id);
+struct partition_desc *partition_create_desc(struct fwnode_handle *fwnode,
+					     struct partition_affinity *parts,
+					     int nr_parts,
+					     int chained_irq,
+					     const struct irq_domain_ops *ops);
+struct irq_domain *partition_get_domain(struct partition_desc *dsc);
+#else
+static inline int partition_translate_id(struct partition_desc *desc,
+					 void *partition_id)
+{
+	return -EINVAL;
+}
+
+static inline
+struct partition_desc *partition_create_desc(struct fwnode_handle *fwnode,
+					     struct partition_affinity *parts,
+					     int nr_parts,
+					     int chained_irq,
+					     const struct irq_domain_ops *ops)
+{
+	return NULL;
+}
+
+static inline
+struct irq_domain *partition_get_domain(struct partition_desc *dsc)
+{
+	return NULL;
+}
+#endif

[tip:irq/core] irqchip/gic-v3: Add support for partitioned PPIs

[tip-bot for Marc Zyngier]

Commit-ID:  e3825ba1af3a27d7522c9f5f929f5a13b8b138ae
Gitweb:     http://git.kernel.org/tip/e3825ba1af3a27d7522c9f5f929f5a13b8b138ae
Author:     Marc Zyngier <marc.zyngier@arm.com>
AuthorDate: Mon, 11 Apr 2016 09:57:54 +0100
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Mon, 2 May 2016 13:42:51 +0200

irqchip/gic-v3: Add support for partitioned PPIs

Plug the partitioning layer into the GICv3 PPI code, parsing the
DT and building the partition affinities and providing the generic
code with partition data and callbacks.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: devicetree@vger.kernel.org
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Rob Herring <robh+dt@kernel.org>
Link: http://lkml.kernel.org/r/1460365075-7316-5-git-send-email-marc.zyngier@arm.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

---
 drivers/irqchip/Kconfig      |   1 +
 drivers/irqchip/irq-gic-v3.c | 176 ++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 175 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index ea1836b..6c17de7 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -27,6 +27,7 @@ config ARM_GIC_V3
 	select IRQ_DOMAIN
 	select MULTI_IRQ_HANDLER
 	select IRQ_DOMAIN_HIERARCHY
+	select PARTITION_PERCPU
 
 config ARM_GIC_V3_ITS
 	bool
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 5b7d3c2..f83e5f4 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -29,6 +29,7 @@
 
 #include <linux/irqchip.h>
 #include <linux/irqchip/arm-gic-v3.h>
+#include <linux/irqchip/irq-partition-percpu.h>
 
 #include <asm/cputype.h>
 #include <asm/exception.h>
@@ -44,6 +45,7 @@ struct redist_region {
 };
 
 struct gic_chip_data {
+	struct fwnode_handle	*fwnode;
 	void __iomem		*dist_base;
 	struct redist_region	*redist_regions;
 	struct rdists		rdists;
@@ -51,6 +53,7 @@ struct gic_chip_data {
 	u64			redist_stride;
 	u32			nr_redist_regions;
 	unsigned int		irq_nr;
+	struct partition_desc	*ppi_descs[16];
 };
 
 static struct gic_chip_data gic_data __read_mostly;
@@ -812,10 +815,62 @@ static void gic_irq_domain_free(struct irq_domain *domain, unsigned int virq,
 	}
 }
 
+static int gic_irq_domain_select(struct irq_domain *d,
+				 struct irq_fwspec *fwspec,
+				 enum irq_domain_bus_token bus_token)
+{
+	/* Not for us */
+        if (fwspec->fwnode != d->fwnode)
+		return 0;
+
+	/* If this is not DT, then we have a single domain */
+	if (!is_of_node(fwspec->fwnode))
+		return 1;
+
+	/*
+	 * If this is a PPI and we have a 4th (non-null) parameter,
+	 * then we need to match the partition domain.
+	 */
+	if (fwspec->param_count >= 4 &&
+	    fwspec->param[0] == 1 && fwspec->param[3] != 0)
+		return d == partition_get_domain(gic_data.ppi_descs[fwspec->param[1]]);
+
+	return d == gic_data.domain;
+}
+
 static const struct irq_domain_ops gic_irq_domain_ops = {
 	.translate = gic_irq_domain_translate,
 	.alloc = gic_irq_domain_alloc,
 	.free = gic_irq_domain_free,
+	.select = gic_irq_domain_select,
+};
+
+static int partition_domain_translate(struct irq_domain *d,
+				      struct irq_fwspec *fwspec,
+				      unsigned long *hwirq,
+				      unsigned int *type)
+{
+	struct device_node *np;
+	int ret;
+
+	np = of_find_node_by_phandle(fwspec->param[3]);
+	if (WARN_ON(!np))
+		return -EINVAL;
+
+	ret = partition_translate_id(gic_data.ppi_descs[fwspec->param[1]],
+				     of_node_to_fwnode(np));
+	if (ret < 0)
+		return ret;
+
+	*hwirq = ret;
+	*type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
+
+	return 0;
+}
+
+static const struct irq_domain_ops partition_domain_ops = {
+	.translate = partition_domain_translate,
+	.select = gic_irq_domain_select,
 };
 
 static void gicv3_enable_quirks(void)
@@ -843,6 +898,7 @@ static int __init gic_init_bases(void __iomem *dist_base,
 	if (static_key_true(&supports_deactivate))
 		pr_info("GIC: Using split EOI/Deactivate mode\n");
 
+	gic_data.fwnode = handle;
 	gic_data.dist_base = dist_base;
 	gic_data.redist_regions = rdist_regs;
 	gic_data.nr_redist_regions = nr_redist_regions;
@@ -901,6 +957,119 @@ static int __init gic_validate_dist_version(void __iomem *dist_base)
 	return 0;
 }
 
+static int get_cpu_number(struct device_node *dn)
+{
+	const __be32 *cell;
+	u64 hwid;
+	int i;
+
+	cell = of_get_property(dn, "reg", NULL);
+	if (!cell)
+		return -1;
+
+	hwid = of_read_number(cell, of_n_addr_cells(dn));
+
+	/*
+	 * Non affinity bits must be set to 0 in the DT
+	 */
+	if (hwid & ~MPIDR_HWID_BITMASK)
+		return -1;
+
+	for (i = 0; i < num_possible_cpus(); i++)
+		if (cpu_logical_map(i) == hwid)
+			return i;
+
+	return -1;
+}
+
+/* Create all possible partitions at boot time */
+static void gic_populate_ppi_partitions(struct device_node *gic_node)
+{
+	struct device_node *parts_node, *child_part;
+	int part_idx = 0, i;
+	int nr_parts;
+	struct partition_affinity *parts;
+
+	parts_node = of_find_node_by_name(gic_node, "ppi-partitions");
+	if (!parts_node)
+		return;
+
+	nr_parts = of_get_child_count(parts_node);
+
+	if (!nr_parts)
+		return;
+
+	parts = kzalloc(sizeof(*parts) * nr_parts, GFP_KERNEL);
+	if (WARN_ON(!parts))
+		return;
+
+	for_each_child_of_node(parts_node, child_part) {
+		struct partition_affinity *part;
+		int n;
+
+		part = &parts[part_idx];
+
+		part->partition_id = of_node_to_fwnode(child_part);
+
+		pr_info("GIC: PPI partition %s[%d] { ",
+			child_part->name, part_idx);
+
+		n = of_property_count_elems_of_size(child_part, "affinity",
+						    sizeof(u32));
+		WARN_ON(n <= 0);
+
+		for (i = 0; i < n; i++) {
+			int err, cpu;
+			u32 cpu_phandle;
+			struct device_node *cpu_node;
+
+			err = of_property_read_u32_index(child_part, "affinity",
+							 i, &cpu_phandle);
+			if (WARN_ON(err))
+				continue;
+
+			cpu_node = of_find_node_by_phandle(cpu_phandle);
+			if (WARN_ON(!cpu_node))
+				continue;
+
+			cpu = get_cpu_number(cpu_node);
+			if (WARN_ON(cpu == -1))
+				continue;
+
+			pr_cont("%s[%d] ", cpu_node->full_name, cpu);
+
+			cpumask_set_cpu(cpu, &part->mask);
+		}
+
+		pr_cont("}\n");
+		part_idx++;
+	}
+
+	for (i = 0; i < 16; i++) {
+		unsigned int irq;
+		struct partition_desc *desc;
+		struct irq_fwspec ppi_fwspec = {
+			.fwnode		= gic_data.fwnode,
+			.param_count	= 3,
+			.param		= {
+				[0]	= 1,
+				[1]	= i,
+				[2]	= IRQ_TYPE_NONE,
+			},
+		};
+
+		irq = irq_create_fwspec_mapping(&ppi_fwspec);
+		if (WARN_ON(!irq))
+			continue;
+		desc = partition_create_desc(gic_data.fwnode, parts, nr_parts,
+					     irq, &partition_domain_ops);
+		if (WARN_ON(!desc))
+			continue;
+
+		gic_data.ppi_descs[i] = desc;
+	}
+}
+
 static int __init gic_of_init(struct device_node *node, struct device_node *parent)
 {
 	void __iomem *dist_base;
@@ -952,8 +1121,11 @@ static int __init gic_of_init(struct device_node *node, struct device_node *pare
 
 	err = gic_init_bases(dist_base, rdist_regs, nr_redist_regions,
 			     redist_stride, &node->fwnode);
-	if (!err)
-		return 0;
+	if (err)
+		goto out_unmap_rdist;
+
+	gic_populate_ppi_partitions(node);
+	return 0;
 
 out_unmap_rdist:
 	for (i = 0; i < nr_redist_regions; i++)

[tip:irq/core] DT/arm,gic-v3: Documment PPI partition support

[tip-bot for Marc Zyngier]

Commit-ID:  287e9357abcc0ef079bf4e439e098a3bd6246a05
Gitweb:     http://git.kernel.org/tip/287e9357abcc0ef079bf4e439e098a3bd6246a05
Author:     Marc Zyngier <marc.zyngier@arm.com>
AuthorDate: Mon, 11 Apr 2016 09:57:55 +0100
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Mon, 2 May 2016 13:42:51 +0200

DT/arm,gic-v3: Documment PPI partition support

Add a decription of the PPI partitioning support.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Acked-by: Rob Herring <robh+dt@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: devicetree@vger.kernel.org
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Will Deacon <will.deacon@arm.com>
Link: http://lkml.kernel.org/r/1460365075-7316-6-git-send-email-marc.zyngier@arm.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

---
 .../bindings/interrupt-controller/arm,gic-v3.txt   | 34 ++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
index 007a5b4..4c29cda 100644
--- a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
+++ b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
@@ -11,6 +11,8 @@ Main node required properties:
 - interrupt-controller : Identifies the node as an interrupt controller
 - #interrupt-cells : Specifies the number of cells needed to encode an
   interrupt source. Must be a single cell with a value of at least 3.
+  If the system requires describing PPI affinity, then the value must
+  be at least 4.
 
   The 1st cell is the interrupt type; 0 for SPI interrupts, 1 for PPI
   interrupts. Other values are reserved for future use.
@@ -24,7 +26,14 @@ Main node required properties:
 		1 = edge triggered
 		4 = level triggered
 
-  Cells 4 and beyond are reserved for future use and must have a value
+  The 4th cell is a phandle to a node describing a set of CPUs this
+  interrupt is affine to. The interrupt must be a PPI, and the node
+  pointed must be a subnode of the "ppi-partitions" subnode. For
+  interrupt types other than PPI or PPIs that are not partitionned,
+  this cell must be zero. See the "ppi-partitions" node description
+  below.
+
+  Cells 5 and beyond are reserved for future use and must have a value
   of 0 if present.
 
 - reg : Specifies base physical address(s) and size of the GIC
@@ -50,6 +59,11 @@ Optional
 
 Sub-nodes:
 
+PPI affinity can be expressed as a single "ppi-partitions" node,
+containing a set of sub-nodes, each with the following property:
+- affinity: Should be a list of phandles to CPU nodes (as described in
+Documentation/devicetree/bindings/arm/cpus.txt).
+
 GICv3 has one or more Interrupt Translation Services (ITS) that are
 used to route Message Signalled Interrupts (MSI) to the CPUs.
 
@@ -91,7 +105,7 @@ Examples:
 
 	gic: interrupt-controller@2c010000 {
 		compatible = "arm,gic-v3";
-		#interrupt-cells = <3>;
+		#interrupt-cells = <4>;
 		#address-cells = <2>;
 		#size-cells = <2>;
 		ranges;
@@ -119,4 +133,20 @@ Examples:
 			#msi-cells = <1>;
 			reg = <0x0 0x2c400000 0 0x200000>;
 		};
+
+		ppi-partitions {
+			part0: interrupt-partition-0 {
+				affinity = <&cpu0 &cpu2>;
+			};
+
+			part1: interrupt-partition-1 {
+				affinity = <&cpu1 &cpu3>;
+			};
+		};
+	};
+
+
+	device@0 {
+		reg = <0 0 0 4>;
+		interrupts = <1 1 4 &part0>;
 	};

Re: [PATCH 2/5] genirq: Allow the affinity of a percpu interrupt to be set/retrieved

[Geert Uytterhoeven]

Hi Marc,

On Mon, Apr 11, 2016 at 10:57 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> In order to prepare the genirq layer for the concept of partitionned
> percpu interrupts, let's allow an affinity to be associated with
> such an interrupt. We introduce:
>
> - irq_set_percpu_devid_partition: flag an interrupt as a percpu-devid
>   interrupt, and associate it with an affinity
> - irq_get_percpu_devid_partition: allow the affinity of that interrupt
>   to be retrieved.
>
> This will allow a driver to discover which CPUs the per-cpu interrupt
> can actually fire on.
>
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

> --- a/include/linux/irqdesc.h
> +++ b/include/linux/irqdesc.h
> @@ -66,6 +66,7 @@ struct irq_desc {
>         int                     threads_handled_last;
>         raw_spinlock_t          lock;
>         struct cpumask          *percpu_enabled;
> +       const struct cpumask    *percpu_affinity;

Adding this field showed up on my bloat-o-meter radar...

Does it make sense to move it (and percpu_enabled) inside the "#ifdef
CONFIG_SMP" below, and rework the code to not need it on UP?

>  #ifdef CONFIG_SMP
>         const struct cpumask    *affinity_hint;
>         struct irq_affinity_notify *affinity_notify;

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH 2/5] genirq: Allow the affinity of a percpu interrupt to be set/retrieved

[Marc Zyngier]

Hi Geert,

On 19/05/16 12:08, Geert Uytterhoeven wrote:
> Hi Marc,
> 
> On Mon, Apr 11, 2016 at 10:57 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> In order to prepare the genirq layer for the concept of partitionned
>> percpu interrupts, let's allow an affinity to be associated with
>> such an interrupt. We introduce:
>>
>> - irq_set_percpu_devid_partition: flag an interrupt as a percpu-devid
>>   interrupt, and associate it with an affinity
>> - irq_get_percpu_devid_partition: allow the affinity of that interrupt
>>   to be retrieved.
>>
>> This will allow a driver to discover which CPUs the per-cpu interrupt
>> can actually fire on.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> 
>> --- a/include/linux/irqdesc.h
>> +++ b/include/linux/irqdesc.h
>> @@ -66,6 +66,7 @@ struct irq_desc {
>>         int                     threads_handled_last;
>>         raw_spinlock_t          lock;
>>         struct cpumask          *percpu_enabled;
>> +       const struct cpumask    *percpu_affinity;
> 
> Adding this field showed up on my bloat-o-meter radar...

By how much?

> Does it make sense to move it (and percpu_enabled) inside the "#ifdef
> CONFIG_SMP" below, and rework the code to not need it on UP?
> 
>>  #ifdef CONFIG_SMP
>>         const struct cpumask    *affinity_hint;
>>         struct irq_affinity_notify *affinity_notify;

I wonder if we couldn't actually unify affinity_hint and
percpu_affinity. I'll have a look.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 2/5] genirq: Allow the affinity of a percpu interrupt to be set/retrieved

[Geert Uytterhoeven]

Hi Marc,

On Thu, May 19, 2016 at 3:13 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On 19/05/16 12:08, Geert Uytterhoeven wrote:
>> On Mon, Apr 11, 2016 at 10:57 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>> In order to prepare the genirq layer for the concept of partitionned
>>> percpu interrupts, let's allow an affinity to be associated with
>>> such an interrupt. We introduce:
>>>
>>> - irq_set_percpu_devid_partition: flag an interrupt as a percpu-devid
>>>   interrupt, and associate it with an affinity
>>> - irq_get_percpu_devid_partition: allow the affinity of that interrupt
>>>   to be retrieved.
>>>
>>> This will allow a driver to discover which CPUs the per-cpu interrupt
>>> can actually fire on.
>>>
>>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>>
>>> --- a/include/linux/irqdesc.h
>>> +++ b/include/linux/irqdesc.h
>>> @@ -66,6 +66,7 @@ struct irq_desc {
>>>         int                     threads_handled_last;
>>>         raw_spinlock_t          lock;
>>>         struct cpumask          *percpu_enabled;
>>> +       const struct cpumask    *percpu_affinity;
>>
>> Adding this field showed up on my bloat-o-meter radar...
>
> By how much?

By NR_IRQS * 4 bytes, which ranges from 32 to 1024 bytes on m68k,
depending on the platform.

>> Does it make sense to move it (and percpu_enabled) inside the "#ifdef
>> CONFIG_SMP" below, and rework the code to not need it on UP?
>>
>>>  #ifdef CONFIG_SMP
>>>         const struct cpumask    *affinity_hint;
>>>         struct irq_affinity_notify *affinity_notify;
>
> I wonder if we couldn't actually unify affinity_hint and
> percpu_affinity. I'll have a look.

Thanks!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds