[PATCH 00/10] staging: fsl-mc: MC bus MSI support

[PATCH 00/10] staging: fsl-mc: MC bus MSI support

[J. German Rivera]

This patch series addresses the following item from the TODO list
for the MC bus driver to exit staging:

* Interrupt support. For meaningful driver support we need
  interrupts, and thus need message interrupt support by the bus
  driver.

MC Bus MSI Support Architecture
===============================
A new IRQ domain bus token is added for the FSL-MC bus.
An MSI IRQ domain is created for each top-level (root) data-path
resource container (DPRC), based on its msi-parent in the device
tree (which is the GIC-ITS). Child DPRCs inherit the MSI IRQ
domain form their parent DPRC.

MC Bus MSI Allocation
---------------------
Given the way in which the GIC-ITS works, we need to pre-allocate
a block of MSIs in the GIC-ITS for the IRQs of all the DPAA2 objects
in the same data-path resource container (DPRC) and for the IRQ of
the DPRC iself.

This is due to the fact that all the IRQs for DPAA2 objects in the
same DPRC (and the DPRC's own IRQ) must use the same "device Id" in
the GIC-ITS. Thus, all these IRQs must share the same ITT table in
the GIC-ITS, and therefore must be allocated in the GIC-ITS as
a block of MSIs for the same "device Id".

This is because all the DPAA2 objects in the same DPRC (and the
DPRC itself) use the DPRC's SMMU stream ID as their device Id for
the GIC-ITS.
The DPAA2 Management Complex (MC) firmware does not assign a separate
SMMU stream ID to each DPAA2 object. The MC only assigns SMMU stream
IDs to DPRCs. In MC terms, the stream ID assigned to a DPRC is known
as the DPRC's Isolation Context ID (ICID).

As a consequence of having to pre-allocate a block of MSIs in
the GIC-ITS, the object allocator of the MC bus driver needs to be
extended to provide IRQ allocation services to DPAA2 device drivers
and to the DPRC driver. For a given DPAA2 object, MSIs are allocated
from the corresponding DPRC's pool of pre-allocated MSIs. The MSI
for the DPRC itself is also allocated from this pool.

The following are the patches in this series:

Patch 1: Added domain bus token DOMAIN_BUS_FSL_MC_MSI
Patch 2: Added generic MSI support for FSL-MC devices
Patch 3: Added GICv3-ITS support for FSL-MC MSIs
Patch 4: Extended MC bus allocator to include IRQs
Patch 5: Changed DPRC built-in portal's mc_io to be atomic
Patch 6: Populate the IRQ pool for an MC bus instance
Patch 7: Set MSI domain for DPRC objects
Patch 8: Fixed bug in dprc_probe() error path
Patch 9: Added DPRC interrupt handler
Patch 10: Added MSI support to the MC bus driver

[PATCH 01/10] irqdomain: Added domain bus token DOMAIN_BUS_FSL_MC_MSI

[J. German Rivera]

Since an FSL-MC bus is a new bus type that is neither PCI nor
PLATFORM, we need a new domain bus token to disambiguate the
IRQ domain for FSL-MC MSIs.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 include/linux/irqdomain.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index d5e5c5b..c0cb5d1 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -73,6 +73,7 @@ enum irq_domain_bus_token {
 	DOMAIN_BUS_PCI_MSI,
 	DOMAIN_BUS_PLATFORM_MSI,
 	DOMAIN_BUS_NEXUS,
+	DOMAIN_BUS_FSL_MC_MSI,
 };
 
 /**
-- 
2.3.3

[PATCH 02/10] staging: fsl-mc: Added generic MSI support for FSL-MC devices

[J. German Rivera]

Created an MSI domain for the fsl-mc bus-- including functions
to create a domain, find a domain, alloc/free domain irqs, and
bus specific overrides for domain and irq_chip ops.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/Kconfig          |   1 +
 drivers/staging/fsl-mc/bus/Makefile         |   1 +
 drivers/staging/fsl-mc/bus/mc-msi.c         | 278 ++++++++++++++++++++++++++++
 drivers/staging/fsl-mc/include/mc-private.h |  17 ++
 drivers/staging/fsl-mc/include/mc.h         |  17 ++
 5 files changed, 314 insertions(+)
 create mode 100644 drivers/staging/fsl-mc/bus/mc-msi.c

diff --git a/drivers/staging/fsl-mc/bus/Kconfig b/drivers/staging/fsl-mc/bus/Kconfig
index 0d779d9..c498ac6 100644
--- a/drivers/staging/fsl-mc/bus/Kconfig
+++ b/drivers/staging/fsl-mc/bus/Kconfig
@@ -9,6 +9,7 @@
 config FSL_MC_BUS
 	tristate "Freescale Management Complex (MC) bus driver"
 	depends on OF && ARM64
+	select GENERIC_MSI_IRQ_DOMAIN
 	help
 	  Driver to enable the bus infrastructure for the Freescale
           QorIQ Management Complex (fsl-mc). The fsl-mc is a hardware
diff --git a/drivers/staging/fsl-mc/bus/Makefile b/drivers/staging/fsl-mc/bus/Makefile
index 25433a9..a5f2ba4 100644
--- a/drivers/staging/fsl-mc/bus/Makefile
+++ b/drivers/staging/fsl-mc/bus/Makefile
@@ -13,5 +13,6 @@ mc-bus-driver-objs := mc-bus.o \
 		      dpmng.o \
 		      dprc-driver.o \
 		      mc-allocator.o \
+		      mc-msi.o \
 		      dpmcp.o \
 		      dpbp.o
diff --git a/drivers/staging/fsl-mc/bus/mc-msi.c b/drivers/staging/fsl-mc/bus/mc-msi.c
new file mode 100644
index 0000000..dfe8016
--- /dev/null
+++ b/drivers/staging/fsl-mc/bus/mc-msi.c
@@ -0,0 +1,278 @@
+/*
+ * Freescale Management Complex (MC) bus driver MSI support
+ *
+ * Copyright (C) 2015 Freescale Semiconductor, Inc.
+ * Author: German Rivera <German.Rivera@freescale.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include "../include/mc-private.h"
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/irqchip/arm-gic-v3.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
+#include <linux/msi.h>
+#include "../include/mc-sys.h"
+#include "dprc-cmd.h"
+
+static void fsl_mc_msi_set_desc(msi_alloc_info_t *arg,
+				struct msi_desc *desc)
+{
+	arg->desc = desc;
+	arg->hwirq = (irq_hw_number_t)desc->msi_attrib.entry_nr;
+}
+
+static void fsl_mc_msi_update_dom_ops(struct msi_domain_info *info)
+{
+	struct msi_domain_ops *ops = info->ops;
+
+	if (WARN_ON(!ops))
+		return;
+
+	if (!ops->set_desc)
+		ops->set_desc = fsl_mc_msi_set_desc;
+}
+
+static void __fsl_mc_msi_write_msg(struct fsl_mc_device *mc_bus_dev,
+				   struct fsl_mc_device_irq *mc_dev_irq)
+{
+	int error;
+	struct fsl_mc_device *owner_mc_dev = mc_dev_irq->mc_dev;
+	struct msi_desc *msi_desc = mc_dev_irq->msi_desc;
+	struct dprc_irq_cfg irq_cfg;
+
+	/*
+	 * msi_desc->msg.address is 0x0 when this function is invoked in
+	 * the free_irq() code path. In this case, for the MC, we don't
+	 * really need to "unprogram" the MSI, so we just return.
+	 */
+	if (msi_desc->msg.address_lo == 0x0 && msi_desc->msg.address_hi == 0x0)
+		return;
+
+	if (WARN_ON(!owner_mc_dev))
+		return;
+
+	irq_cfg.paddr = ((u64)msi_desc->msg.address_hi << 32) |
+			msi_desc->msg.address_lo;
+	irq_cfg.val = msi_desc->msg.data;
+	irq_cfg.user_irq_id = msi_desc->irq;
+
+	/*
+	 * DPRCs and other objects use different commands to set up IRQs,
+	 * so we have to differentiate them here.
+	 */
+	if (owner_mc_dev == mc_bus_dev) {
+		/*
+		 * IRQ is for the mc_bus_dev's DPRC itself
+		 */
+		error = dprc_set_irq(mc_bus_dev->mc_io,
+				     MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
+				     mc_bus_dev->mc_handle,
+				     mc_dev_irq->dev_irq_index,
+				     &irq_cfg);
+		if (error < 0) {
+			dev_err(&owner_mc_dev->dev,
+				"dprc_set_irq() failed: %d\n", error);
+		}
+	} else {
+		error = dprc_set_obj_irq(mc_bus_dev->mc_io,
+					 MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
+					 mc_bus_dev->mc_handle,
+					 owner_mc_dev->obj_desc.type,
+					 owner_mc_dev->obj_desc.id,
+					 mc_dev_irq->dev_irq_index,
+					 &irq_cfg);
+		if (error < 0) {
+			dev_err(&owner_mc_dev->dev,
+				"dprc_obj_set_irq() failed: %d\n", error);
+		}
+	}
+}
+
+/*
+ * NOTE: This function is invoked with interrupts disabled
+ */
+static void fsl_mc_msi_write_msg(struct irq_data *irq_data,
+				 struct msi_msg *msg)
+{
+	struct msi_desc *msi_desc = irq_data_get_msi_desc(irq_data);
+	struct fsl_mc_device *mc_bus_dev = to_fsl_mc_device(msi_desc->dev);
+	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
+	struct fsl_mc_device_irq *mc_dev_irq =
+		&mc_bus->irq_resources[msi_desc->msi_attrib.entry_nr];
+
+	WARN_ON(mc_dev_irq->msi_desc != msi_desc);
+	msi_desc->msg = *msg;
+
+	/*
+	 * Program the MSI (paddr, value) pair in the device:
+	 */
+	__fsl_mc_msi_write_msg(mc_bus_dev, mc_dev_irq);
+}
+
+static void fsl_mc_msi_update_chip_ops(struct msi_domain_info *info)
+{
+	struct irq_chip *chip = info->chip;
+
+	if (WARN_ON((!chip)))
+		return;
+
+	if (!chip->irq_write_msi_msg)
+		chip->irq_write_msi_msg = fsl_mc_msi_write_msg;
+}
+
+/**
+ * fsl_mc_msi_create_irq_domain - Create a fsl-mc MSI interrupt domain
+ * @np:		Optional device-tree node of the interrupt controller
+ * @info:	MSI domain info
+ * @parent:	Parent irq domain
+ *
+ * Updates the domain and chip ops and creates a fsl-mc MSI
+ * interrupt domain.
+ *
+ * Returns:
+ * A domain pointer or NULL in case of failure.
+ */
+struct irq_domain *fsl_mc_msi_create_irq_domain(struct fwnode_handle *fwnode,
+						struct msi_domain_info *info,
+						struct irq_domain *parent)
+{
+	struct irq_domain *domain;
+
+	if (info->flags & MSI_FLAG_USE_DEF_DOM_OPS)
+		fsl_mc_msi_update_dom_ops(info);
+	if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS)
+		fsl_mc_msi_update_chip_ops(info);
+
+	domain = msi_create_irq_domain(fwnode, info, parent);
+	if (domain)
+		domain->bus_token = DOMAIN_BUS_FSL_MC_MSI;
+
+	return domain;
+}
+
+int fsl_mc_find_msi_domain(struct device_node *mc_of_node,
+			   struct irq_domain **mc_msi_domain)
+{
+	struct device_node *msi_parent_node;
+	struct irq_domain *msi_domain;
+
+	msi_parent_node = of_parse_phandle(mc_of_node, "msi-parent", 0);
+	if (!msi_parent_node) {
+		pr_err("msi-parent phandle missing for %s\n",
+		       mc_of_node->full_name);
+		return -ENOENT;
+	}
+
+	/*
+	 * Look up the fsl-mc MSI domain:
+	 */
+	msi_domain = irq_find_matching_host(msi_parent_node,
+					    DOMAIN_BUS_FSL_MC_MSI);
+	if (!msi_domain) {
+		pr_err("Unable to find fsl-mc MSI domain for %s\n",
+		       mc_of_node->full_name);
+
+		return -ENOENT;
+	}
+
+	*mc_msi_domain = msi_domain;
+	return 0;
+}
+
+static void fsl_mc_msi_free_descs(struct device *dev)
+{
+	struct msi_desc *desc, *tmp;
+
+	list_for_each_entry_safe(desc, tmp, dev_to_msi_list(dev), list) {
+		list_del(&desc->list);
+		free_msi_entry(desc);
+	}
+}
+
+static int fsl_mc_msi_alloc_descs(struct device *dev, unsigned int irq_count)
+
+{
+	unsigned int i;
+	int error;
+	struct msi_desc *msi_desc;
+
+	for (i = 0; i < irq_count; i++) {
+		msi_desc = alloc_msi_entry(dev);
+		if (!msi_desc) {
+			dev_err(dev, "Failed to allocate msi entry\n");
+			error = -ENOMEM;
+			goto cleanup_msi_descs;
+		}
+
+		msi_desc->msi_attrib.is_msix = 1;
+		msi_desc->msi_attrib.is_64 = 1;
+		msi_desc->msi_attrib.entry_nr = i;
+		msi_desc->nvec_used = 1;
+		INIT_LIST_HEAD(&msi_desc->list);
+		list_add_tail(&msi_desc->list, dev_to_msi_list(dev));
+	}
+
+	return 0;
+
+cleanup_msi_descs:
+	fsl_mc_msi_free_descs(dev);
+	return error;
+}
+
+int fsl_mc_msi_domain_alloc_irqs(struct device *dev,
+				 unsigned int irq_count)
+{
+	struct irq_domain *msi_domain;
+	int error;
+
+	if (WARN_ON(!list_empty(dev_to_msi_list(dev))))
+		return -EINVAL;
+
+	error = fsl_mc_msi_alloc_descs(dev, irq_count);
+	if (error < 0)
+		return error;
+
+	msi_domain = dev_get_msi_domain(dev);
+	if (WARN_ON(!msi_domain)) {
+		error = -EINVAL;
+		goto cleanup_msi_descs;
+	}
+
+	/*
+	 * NOTE: Calling this function will trigger the invocation of the
+	 * its_fsl_mc_msi_prepare() callback
+	 */
+	error = msi_domain_alloc_irqs(msi_domain, dev, irq_count);
+
+	if (error) {
+		dev_err(dev, "Failed to allocate IRQs\n");
+		goto cleanup_msi_descs;
+	}
+
+	return 0;
+
+cleanup_msi_descs:
+	fsl_mc_msi_free_descs(dev);
+	return error;
+}
+
+void fsl_mc_msi_domain_free_irqs(struct device *dev)
+{
+	struct irq_domain *msi_domain;
+
+	msi_domain = dev_get_msi_domain(dev);
+	if (WARN_ON(!msi_domain))
+		return;
+
+	msi_domain_free_irqs(msi_domain, dev);
+
+	if (WARN_ON(list_empty(dev_to_msi_list(dev))))
+		return;
+
+	fsl_mc_msi_free_descs(dev);
+}
diff --git a/drivers/staging/fsl-mc/include/mc-private.h b/drivers/staging/fsl-mc/include/mc-private.h
index 2c4cc79..3dabd003 100644
--- a/drivers/staging/fsl-mc/include/mc-private.h
+++ b/drivers/staging/fsl-mc/include/mc-private.h
@@ -26,6 +26,9 @@
 	 strcmp(_obj_type, "dpmcp") == 0 || \
 	 strcmp(_obj_type, "dpcon") == 0)
 
+struct irq_domain;
+struct msi_domain_info;
+
 /**
  * struct fsl_mc - Private data of a "fsl,qoriq-mc" platform device
  * @root_mc_bus_dev: MC object device representing the root DPRC
@@ -79,11 +82,13 @@ struct fsl_mc_resource_pool {
  * @resource_pools: array of resource pools (one pool per resource type)
  * for this MC bus. These resources represent allocatable entities
  * from the physical DPRC.
+ * @irq_resources: Pointer to array of IRQ objects for the IRQ pool
  * @scan_mutex: Serializes bus scanning
  */
 struct fsl_mc_bus {
 	struct fsl_mc_device mc_dev;
 	struct fsl_mc_resource_pool resource_pools[FSL_MC_NUM_POOL_TYPES];
+	struct fsl_mc_device_irq *irq_resources;
 	struct mutex scan_mutex;    /* serializes bus scanning */
 };
 
@@ -116,4 +121,16 @@ int __must_check fsl_mc_resource_allocate(struct fsl_mc_bus *mc_bus,
 
 void fsl_mc_resource_free(struct fsl_mc_resource *resource);
 
+struct irq_domain *fsl_mc_msi_create_irq_domain(struct fwnode_handle *fwnode,
+						struct msi_domain_info *info,
+						struct irq_domain *parent);
+
+int fsl_mc_find_msi_domain(struct device_node *mc_of_node,
+			   struct irq_domain **mc_msi_domain);
+
+int fsl_mc_msi_domain_alloc_irqs(struct device *dev,
+				 unsigned int irq_count);
+
+void fsl_mc_msi_domain_free_irqs(struct device *dev);
+
 #endif /* _FSL_MC_PRIVATE_H_ */
diff --git a/drivers/staging/fsl-mc/include/mc.h b/drivers/staging/fsl-mc/include/mc.h
index a933291..1c1d6ae 100644
--- a/drivers/staging/fsl-mc/include/mc.h
+++ b/drivers/staging/fsl-mc/include/mc.h
@@ -104,6 +104,23 @@ struct fsl_mc_resource {
 };
 
 /**
+ * struct fsl_mc_device_irq - MC object device message-based interrupt
+ * @msi_desc: pointer to MSI descriptor allocated by fsl_mc_msi_alloc_descs()
+ * @mc_dev: MC object device that owns this interrupt
+ * @dev_irq_index: device-relative IRQ index
+ * @resource: MC generic resource associated with the interrupt
+ */
+struct fsl_mc_device_irq {
+	struct msi_desc *msi_desc;
+	struct fsl_mc_device *mc_dev;
+	u8 dev_irq_index;
+	struct fsl_mc_resource resource;
+};
+
+#define to_fsl_mc_irq(_mc_resource) \
+	container_of(_mc_resource, struct fsl_mc_device_irq, resource)
+
+/**
  * Bit masks for a MC object device (struct fsl_mc_device) flags
  */
 #define FSL_MC_IS_DPRC	0x0001
-- 
2.3.3

[PATCH 03/10] staging: fsl-mc: Added GICv3-ITS support for FSL-MC MSIs

[J. German Rivera]

Added platform-specific MSI support layer for FSL-MC devices.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/Makefile                |   1 +
 .../staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c | 127 +++++++++++++++++++++
 drivers/staging/fsl-mc/include/mc-private.h        |   4 +
 3 files changed, 132 insertions(+)
 create mode 100644 drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c

diff --git a/drivers/staging/fsl-mc/bus/Makefile b/drivers/staging/fsl-mc/bus/Makefile
index a5f2ba4..e731517 100644
--- a/drivers/staging/fsl-mc/bus/Makefile
+++ b/drivers/staging/fsl-mc/bus/Makefile
@@ -14,5 +14,6 @@ mc-bus-driver-objs := mc-bus.o \
 		      dprc-driver.o \
 		      mc-allocator.o \
 		      mc-msi.o \
+		      irq-gic-v3-its-fsl-mc-msi.o \
 		      dpmcp.o \
 		      dpbp.o
diff --git a/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c b/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c
new file mode 100644
index 0000000..5319afa
--- /dev/null
+++ b/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c
@@ -0,0 +1,127 @@
+/*
+ * Freescale Management Complex (MC) bus driver MSI support
+ *
+ * Copyright (C) 2015 Freescale Semiconductor, Inc.
+ * Author: German Rivera <German.Rivera@freescale.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include "../include/mc-private.h"
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/irqchip/arm-gic-v3.h>
+#include <linux/irq.h>
+#include <linux/msi.h>
+#include <linux/of.h>
+#include <linux/of_irq.h>
+#include "../include/mc-sys.h"
+#include "dprc-cmd.h"
+
+static struct irq_chip its_msi_irq_chip = {
+	.name = "fsl-mc-bus-msi",
+	.irq_mask = irq_chip_mask_parent,
+	.irq_unmask = irq_chip_unmask_parent,
+	.irq_eoi = irq_chip_eoi_parent,
+	.irq_set_affinity = msi_domain_set_affinity
+};
+
+static int its_fsl_mc_msi_prepare(struct irq_domain *msi_domain,
+				  struct device *dev,
+				  int nvec, msi_alloc_info_t *info)
+{
+	u32 its_dev_id;
+	struct fsl_mc_device *mc_bus_dev = to_fsl_mc_device(dev);
+	struct msi_domain_info *msi_info;
+
+	if (WARN_ON(dev->bus != &fsl_mc_bus_type))
+		return -EINVAL;
+
+	if (WARN_ON(!(mc_bus_dev->flags & FSL_MC_IS_DPRC)))
+		return -EINVAL;
+
+	/*
+	 * Set the device Id to be passed to the GIC-ITS:
+	 *
+	 * NOTE: This device id corresponds to the IOMMU stream ID
+	 * associated with the DPRC object (ICID).
+	 */
+	its_dev_id = mc_bus_dev->icid;
+	info->scratchpad[0].ul = its_dev_id;
+	msi_info = msi_get_domain_info(msi_domain->parent);
+	return msi_info->ops->msi_prepare(msi_domain->parent, dev, nvec, info);
+}
+
+static struct msi_domain_ops its_fsl_mc_msi_ops = {
+	.msi_prepare = its_fsl_mc_msi_prepare,
+};
+
+static struct msi_domain_info its_fsl_mc_msi_domain_info = {
+	.flags	= (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
+	.ops	= &its_fsl_mc_msi_ops,
+	.chip	= &its_msi_irq_chip,
+};
+
+static const struct of_device_id its_device_id[] = {
+	{	.compatible	= "arm,gic-v3-its",	},
+	{},
+};
+
+int __init its_fsl_mc_msi_init(void)
+{
+	struct device_node *np;
+	struct irq_domain *parent;
+	struct irq_domain *mc_msi_domain;
+
+	for (np = of_find_matching_node(NULL, its_device_id); np;
+	     np = of_find_matching_node(np, its_device_id)) {
+		if (!of_property_read_bool(np, "msi-controller"))
+			continue;
+
+		parent = irq_find_matching_host(np, DOMAIN_BUS_NEXUS);
+		if (!parent || !msi_get_domain_info(parent)) {
+			pr_err("%s: unable to locate ITS domain\n",
+			       np->full_name);
+			continue;
+		}
+
+		mc_msi_domain =
+		    fsl_mc_msi_create_irq_domain(of_node_to_fwnode(np),
+						 &its_fsl_mc_msi_domain_info,
+						 parent);
+		if (!mc_msi_domain) {
+			pr_err("%s: unable to create fsl-mc domain\n",
+			       np->full_name);
+			continue;
+		}
+
+		WARN_ON(mc_msi_domain->host_data !=
+				&its_fsl_mc_msi_domain_info);
+
+		pr_info("fsl-mc MSI: %s domain created\n", np->full_name);
+	}
+
+	return 0;
+}
+
+void its_fsl_mc_msi_cleanup(void)
+{
+	struct device_node *np;
+
+	for (np = of_find_matching_node(NULL, its_device_id); np;
+	     np = of_find_matching_node(np, its_device_id)) {
+		struct irq_domain *mc_msi_domain =
+			irq_find_matching_host(np, DOMAIN_BUS_FSL_MC_MSI);
+
+		if (!of_property_read_bool(np, "msi-controller"))
+			continue;
+
+		mc_msi_domain =
+			irq_find_matching_host(np, DOMAIN_BUS_FSL_MC_MSI);
+		if (mc_msi_domain &&
+		    mc_msi_domain->host_data == &its_fsl_mc_msi_domain_info)
+			irq_domain_remove(mc_msi_domain);
+	}
+}
diff --git a/drivers/staging/fsl-mc/include/mc-private.h b/drivers/staging/fsl-mc/include/mc-private.h
index 3dabd003..f2504bc 100644
--- a/drivers/staging/fsl-mc/include/mc-private.h
+++ b/drivers/staging/fsl-mc/include/mc-private.h
@@ -133,4 +133,8 @@ int fsl_mc_msi_domain_alloc_irqs(struct device *dev,
 
 void fsl_mc_msi_domain_free_irqs(struct device *dev);
 
+int __init its_fsl_mc_msi_init(void);
+
+void its_fsl_mc_msi_cleanup(void);
+
 #endif /* _FSL_MC_PRIVATE_H_ */
-- 
2.3.3

[PATCH 04/10] staging: fsl-mc: Extended MC bus allocator to include IRQs

[J. German Rivera]

All the IRQs for DPAA2 objects in the same DPRC must use
the ICID of that DPRC, as their device Id in the GIC-ITS.
Thus, all these IRQs must share the same ITT table in the GIC.
As a result, a pool of IRQs with the same device Id must be
preallocated per DPRC (fsl-mc bus instance). So, the fsl-mc
bus object allocator is extended to also provide services
to allocate IRQs to DPAA2 devices, from their parent fsl-mc bus
IRQ pool.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/mc-allocator.c   | 199 ++++++++++++++++++++++++++++
 drivers/staging/fsl-mc/include/mc-private.h |  15 +++
 drivers/staging/fsl-mc/include/mc.h         |   9 ++
 3 files changed, 223 insertions(+)

diff --git a/drivers/staging/fsl-mc/bus/mc-allocator.c b/drivers/staging/fsl-mc/bus/mc-allocator.c
index 88d1857..6f1049a 100644
--- a/drivers/staging/fsl-mc/bus/mc-allocator.c
+++ b/drivers/staging/fsl-mc/bus/mc-allocator.c
@@ -15,6 +15,7 @@
 #include "../include/dpcon-cmd.h"
 #include "dpmcp-cmd.h"
 #include "dpmcp.h"
+#include <linux/msi.h>
 
 /**
  * fsl_mc_resource_pool_add_device - add allocatable device to a resource
@@ -160,6 +161,7 @@ static const char *const fsl_mc_pool_type_strings[] = {
 	[FSL_MC_POOL_DPMCP] = "dpmcp",
 	[FSL_MC_POOL_DPBP] = "dpbp",
 	[FSL_MC_POOL_DPCON] = "dpcon",
+	[FSL_MC_POOL_IRQ] = "irq",
 };
 
 static int __must_check object_type_to_pool_type(const char *object_type,
@@ -465,6 +467,203 @@ void fsl_mc_object_free(struct fsl_mc_device *mc_adev)
 }
 EXPORT_SYMBOL_GPL(fsl_mc_object_free);
 
+/*
+ * Initialize the interrupt pool associated with a MC bus.
+ * It allocates a block of IRQs from the GIC-ITS
+ */
+int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
+			     unsigned int irq_count)
+{
+	unsigned int i;
+	struct msi_desc *msi_desc;
+	struct fsl_mc_device_irq *irq_resources;
+	struct fsl_mc_device_irq *mc_dev_irq;
+	int error;
+	struct fsl_mc_device *mc_bus_dev = &mc_bus->mc_dev;
+	struct fsl_mc_resource_pool *res_pool =
+			&mc_bus->resource_pools[FSL_MC_POOL_IRQ];
+
+	if (WARN_ON(irq_count == 0 ||
+		    irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS))
+		return -EINVAL;
+
+	error = fsl_mc_msi_domain_alloc_irqs(&mc_bus_dev->dev, irq_count);
+	if (error < 0)
+		return error;
+
+	irq_resources = devm_kzalloc(&mc_bus_dev->dev,
+				     sizeof(*irq_resources) * irq_count,
+				     GFP_KERNEL);
+	if (!irq_resources) {
+		error = -ENOMEM;
+		goto cleanup_msi_irqs;
+	}
+
+	for (i = 0; i < irq_count; i++) {
+		mc_dev_irq = &irq_resources[i];
+
+		/*
+		 * NOTE: This mc_dev_irq's MSI addr/value pair will be set
+		 * by the fsl_mc_msi_write_msg() callback
+		 */
+		mc_dev_irq->resource.type = res_pool->type;
+		mc_dev_irq->resource.data = mc_dev_irq;
+		mc_dev_irq->resource.parent_pool = res_pool;
+		INIT_LIST_HEAD(&mc_dev_irq->resource.node);
+		list_add_tail(&mc_dev_irq->resource.node, &res_pool->free_list);
+	}
+
+	for_each_msi_entry(msi_desc, &mc_bus_dev->dev) {
+		mc_dev_irq = &irq_resources[msi_desc->msi_attrib.entry_nr];
+		mc_dev_irq->msi_desc = msi_desc;
+		mc_dev_irq->resource.id = msi_desc->irq;
+	}
+
+	res_pool->max_count = irq_count;
+	res_pool->free_count = irq_count;
+	mc_bus->irq_resources = irq_resources;
+	return 0;
+
+cleanup_msi_irqs:
+	fsl_mc_msi_domain_free_irqs(&mc_bus_dev->dev);
+	return error;
+}
+EXPORT_SYMBOL_GPL(fsl_mc_populate_irq_pool);
+
+/**
+ * Teardown the interrupt pool associated with an MC bus.
+ * It frees the IRQs that were allocated to the pool, back to the GIC-ITS.
+ */
+void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus)
+{
+	struct fsl_mc_device *mc_bus_dev = &mc_bus->mc_dev;
+	struct fsl_mc_resource_pool *res_pool =
+			&mc_bus->resource_pools[FSL_MC_POOL_IRQ];
+
+	if (WARN_ON(!mc_bus->irq_resources))
+		return;
+
+	if (WARN_ON(res_pool->max_count == 0))
+		return;
+
+	if (WARN_ON(res_pool->free_count != res_pool->max_count))
+		return;
+
+	INIT_LIST_HEAD(&res_pool->free_list);
+	res_pool->max_count = 0;
+	res_pool->free_count = 0;
+	mc_bus->irq_resources = NULL;
+	fsl_mc_msi_domain_free_irqs(&mc_bus_dev->dev);
+}
+EXPORT_SYMBOL_GPL(fsl_mc_cleanup_irq_pool);
+
+/**
+ * It allocates the IRQs required by a given MC object device. The
+ * IRQs are allocated from the interrupt pool associated with the
+ * MC bus that contains the device, if the device is not a DPRC device.
+ * Otherwise, the IRQs are allocated from the interrupt pool associated
+ * with the MC bus that represents the DPRC device itself.
+ */
+int __must_check fsl_mc_allocate_irqs(struct fsl_mc_device *mc_dev)
+{
+	int i;
+	int irq_count;
+	int res_allocated_count = 0;
+	int error = -EINVAL;
+	struct fsl_mc_device_irq **irqs = NULL;
+	struct fsl_mc_bus *mc_bus;
+	struct fsl_mc_resource_pool *res_pool;
+
+	if (WARN_ON(mc_dev->irqs))
+		return -EINVAL;
+
+	irq_count = mc_dev->obj_desc.irq_count;
+	if (WARN_ON(irq_count == 0))
+		return -EINVAL;
+
+	if (strcmp(mc_dev->obj_desc.type, "dprc") == 0)
+		mc_bus = to_fsl_mc_bus(mc_dev);
+	else
+		mc_bus = to_fsl_mc_bus(to_fsl_mc_device(mc_dev->dev.parent));
+
+	if (WARN_ON(!mc_bus->irq_resources))
+		return -EINVAL;
+
+	res_pool = &mc_bus->resource_pools[FSL_MC_POOL_IRQ];
+	if (res_pool->free_count < irq_count) {
+		dev_err(&mc_dev->dev,
+			"Not able to allocate %u irqs for device\n", irq_count);
+		return -ENOSPC;
+	}
+
+	irqs = devm_kzalloc(&mc_dev->dev, irq_count * sizeof(irqs[0]),
+			    GFP_KERNEL);
+	if (!irqs)
+		return -ENOMEM;
+
+	for (i = 0; i < irq_count; i++) {
+		struct fsl_mc_resource *resource;
+
+		error = fsl_mc_resource_allocate(mc_bus, FSL_MC_POOL_IRQ,
+						 &resource);
+		if (error < 0)
+			goto error_resource_alloc;
+
+		irqs[i] = to_fsl_mc_irq(resource);
+		res_allocated_count++;
+
+		WARN_ON(irqs[i]->mc_dev);
+		irqs[i]->mc_dev = mc_dev;
+		irqs[i]->dev_irq_index = i;
+	}
+
+	mc_dev->irqs = irqs;
+	return 0;
+
+error_resource_alloc:
+	for (i = 0; i < res_allocated_count; i++) {
+		irqs[i]->mc_dev = NULL;
+		fsl_mc_resource_free(&irqs[i]->resource);
+	}
+
+	return error;
+}
+EXPORT_SYMBOL_GPL(fsl_mc_allocate_irqs);
+
+/*
+ * It frees the IRQs that were allocated for a MC object device, by
+ * returning them to the corresponding interrupt pool.
+ */
+void fsl_mc_free_irqs(struct fsl_mc_device *mc_dev)
+{
+	int i;
+	int irq_count;
+	struct fsl_mc_bus *mc_bus;
+	struct fsl_mc_device_irq **irqs = mc_dev->irqs;
+
+	if (WARN_ON(!irqs))
+		return;
+
+	irq_count = mc_dev->obj_desc.irq_count;
+
+	if (strcmp(mc_dev->obj_desc.type, "dprc") == 0)
+		mc_bus = to_fsl_mc_bus(mc_dev);
+	else
+		mc_bus = to_fsl_mc_bus(to_fsl_mc_device(mc_dev->dev.parent));
+
+	if (WARN_ON(!mc_bus->irq_resources))
+		return;
+
+	for (i = 0; i < irq_count; i++) {
+		WARN_ON(!irqs[i]->mc_dev);
+		irqs[i]->mc_dev = NULL;
+		fsl_mc_resource_free(&irqs[i]->resource);
+	}
+
+	mc_dev->irqs = NULL;
+}
+EXPORT_SYMBOL_GPL(fsl_mc_free_irqs);
+
 /**
  * fsl_mc_allocator_probe - callback invoked when an allocatable device is
  * being added to the system
diff --git a/drivers/staging/fsl-mc/include/mc-private.h b/drivers/staging/fsl-mc/include/mc-private.h
index f2504bc..97295f0 100644
--- a/drivers/staging/fsl-mc/include/mc-private.h
+++ b/drivers/staging/fsl-mc/include/mc-private.h
@@ -30,6 +30,16 @@ struct irq_domain;
 struct msi_domain_info;
 
 /**
+ * Maximum number of total IRQs that can be pre-allocated for an MC bus'
+ * IRQ pool
+ */
+#define FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS	256
+
+struct device_node;
+struct irq_domain;
+struct msi_domain_info;
+
+/**
  * struct fsl_mc - Private data of a "fsl,qoriq-mc" platform device
  * @root_mc_bus_dev: MC object device representing the root DPRC
  * @num_translation_ranges: number of entries in addr_translation_ranges
@@ -137,4 +147,9 @@ int __init its_fsl_mc_msi_init(void);
 
 void its_fsl_mc_msi_cleanup(void);
 
+int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
+			     unsigned int irq_count);
+
+void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus);
+
 #endif /* _FSL_MC_PRIVATE_H_ */
diff --git a/drivers/staging/fsl-mc/include/mc.h b/drivers/staging/fsl-mc/include/mc.h
index 1c1d6ae..ac7c1ce 100644
--- a/drivers/staging/fsl-mc/include/mc.h
+++ b/drivers/staging/fsl-mc/include/mc.h
@@ -14,12 +14,14 @@
 #include <linux/device.h>
 #include <linux/mod_devicetable.h>
 #include <linux/list.h>
+#include <linux/interrupt.h>
 #include "../include/dprc.h"
 
 #define FSL_MC_VENDOR_FREESCALE	0x1957
 
 struct fsl_mc_device;
 struct fsl_mc_io;
+struct fsl_mc_bus;
 
 /**
  * struct fsl_mc_driver - MC object device driver object
@@ -75,6 +77,7 @@ enum fsl_mc_pool_type {
 	FSL_MC_POOL_DPMCP = 0x0,    /* corresponds to "dpmcp" in the MC */
 	FSL_MC_POOL_DPBP,	    /* corresponds to "dpbp" in the MC */
 	FSL_MC_POOL_DPCON,	    /* corresponds to "dpcon" in the MC */
+	FSL_MC_POOL_IRQ,
 
 	/*
 	 * NOTE: New resource pool types must be added before this entry
@@ -141,6 +144,7 @@ struct fsl_mc_device_irq {
  * NULL if none.
  * @obj_desc: MC description of the DPAA device
  * @regions: pointer to array of MMIO region entries
+ * @irqs: pointer to array of pointers to interrupts allocated to this device
  * @resource: generic resource associated with this MC object device, if any.
  *
  * Generic device object for MC object devices that are "attached" to a
@@ -172,6 +176,7 @@ struct fsl_mc_device {
 	struct fsl_mc_io *mc_io;
 	struct dprc_obj_desc obj_desc;
 	struct resource *regions;
+	struct fsl_mc_device_irq **irqs;
 	struct fsl_mc_resource *resource;
 };
 
@@ -215,6 +220,10 @@ int __must_check fsl_mc_object_allocate(struct fsl_mc_device *mc_dev,
 
 void fsl_mc_object_free(struct fsl_mc_device *mc_adev);
 
+int __must_check fsl_mc_allocate_irqs(struct fsl_mc_device *mc_dev);
+
+void fsl_mc_free_irqs(struct fsl_mc_device *mc_dev);
+
 extern struct bus_type fsl_mc_bus_type;
 
 #endif /* _FSL_MC_H_ */
-- 
2.3.3

[PATCH 05/10] staging: fsl-mc: Changed DPRC built-in portal's mc_io to be atomic

[J. German Rivera]

The DPRC built-in portal's mc_io is used to send commands to the MC
to program MSIs for MC objects. This is done by the
fsl_mc_msi_write_msg() callback, which is invoked by the generic MSI
layer with interrupts disabled. As a result, the mc_io used in
fsl_mc_msi_write_msg needs to be an atomic mc_io.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/dprc-driver.c | 4 +++-
 drivers/staging/fsl-mc/bus/mc-bus.c      | 3 ++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index a9ead0d..7bb30dd 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -396,7 +396,9 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 		error = fsl_create_mc_io(&mc_dev->dev,
 					 mc_dev->regions[0].start,
 					 region_size,
-					 NULL, 0, &mc_dev->mc_io);
+					 NULL,
+					 FSL_MC_IO_ATOMIC_CONTEXT_PORTAL,
+					 &mc_dev->mc_io);
 		if (error < 0)
 			return error;
 	}
diff --git a/drivers/staging/fsl-mc/bus/mc-bus.c b/drivers/staging/fsl-mc/bus/mc-bus.c
index 84db55b..d34f1af 100644
--- a/drivers/staging/fsl-mc/bus/mc-bus.c
+++ b/drivers/staging/fsl-mc/bus/mc-bus.c
@@ -702,7 +702,8 @@ static int fsl_mc_bus_probe(struct platform_device *pdev)
 	mc_portal_phys_addr = res.start;
 	mc_portal_size = resource_size(&res);
 	error = fsl_create_mc_io(&pdev->dev, mc_portal_phys_addr,
-				 mc_portal_size, NULL, 0, &mc_io);
+				 mc_portal_size, NULL,
+				 FSL_MC_IO_ATOMIC_CONTEXT_PORTAL, &mc_io);
 	if (error < 0)
 		return error;
 
-- 
2.3.3

[PATCH 06/10] staging: fsl-mc: Populate the IRQ pool for an MC bus instance

[J. German Rivera]

Scan the corresponding DPRC container to get total count
of IRQs needed by all its child DPAA2 objects. Then,
preallocate a set of MSI IRQs with the DPRC's ICID
(GIT-ITS device Id) to populate the the DPRC's IRQ pool.
Each child DPAA2 object in the DPRC and the DPRC object itself
will allocate their necessary MSI IRQs from the DPRC's IRQ pool,
in their driver probe function.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/dprc-driver.c    | 24 ++++++++++++++++++++++--
 drivers/staging/fsl-mc/include/mc-private.h |  3 ++-
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index 7bb30dd..3bcd161 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -241,6 +241,7 @@ static void dprc_cleanup_all_resource_pools(struct fsl_mc_device *mc_bus_dev)
  * dprc_scan_objects - Discover objects in a DPRC
  *
  * @mc_bus_dev: pointer to the fsl-mc device that represents a DPRC object
+ * @total_irq_count: total number of IRQs needed by objects in the DPRC.
  *
  * Detects objects added and removed from a DPRC and synchronizes the
  * state of the Linux bus driver, MC by adding and removing
@@ -254,11 +255,13 @@ static void dprc_cleanup_all_resource_pools(struct fsl_mc_device *mc_bus_dev)
  * populated before they can get allocation requests from probe callbacks
  * of the device drivers for the non-allocatable devices.
  */
-int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
+int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
+		      unsigned int *total_irq_count)
 {
 	int num_child_objects;
 	int dprc_get_obj_failures;
 	int error;
+	unsigned int irq_count = mc_bus_dev->obj_desc.irq_count;
 	struct dprc_obj_desc *child_obj_desc_array = NULL;
 
 	error = dprc_get_obj_count(mc_bus_dev->mc_io,
@@ -307,6 +310,7 @@ int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
 				continue;
 			}
 
+			irq_count += obj_desc->irq_count;
 			dev_dbg(&mc_bus_dev->dev,
 				"Discovered object: type %s, id %d\n",
 				obj_desc->type, obj_desc->id);
@@ -319,6 +323,7 @@ int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
 		}
 	}
 
+	*total_irq_count = irq_count;
 	dprc_remove_devices(mc_bus_dev, child_obj_desc_array,
 			    num_child_objects);
 
@@ -344,6 +349,7 @@ EXPORT_SYMBOL_GPL(dprc_scan_objects);
 int dprc_scan_container(struct fsl_mc_device *mc_bus_dev)
 {
 	int error;
+	unsigned int irq_count;
 	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
 
 	dprc_init_all_resource_pools(mc_bus_dev);
@@ -352,11 +358,25 @@ int dprc_scan_container(struct fsl_mc_device *mc_bus_dev)
 	 * Discover objects in the DPRC:
 	 */
 	mutex_lock(&mc_bus->scan_mutex);
-	error = dprc_scan_objects(mc_bus_dev);
+	error = dprc_scan_objects(mc_bus_dev, &irq_count);
 	mutex_unlock(&mc_bus->scan_mutex);
 	if (error < 0)
 		goto error;
 
+	if (dev_get_msi_domain(&mc_bus_dev->dev) && !mc_bus->irq_resources) {
+		if (irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS) {
+			dev_warn(&mc_bus_dev->dev,
+				 "IRQs needed (%u) exceed IRQs preallocated (%u)\n",
+				 irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+		}
+
+		error = fsl_mc_populate_irq_pool(
+				mc_bus,
+				FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+		if (error < 0)
+			goto error;
+	}
+
 	return 0;
 error:
 	dprc_cleanup_all_resource_pools(mc_bus_dev);
diff --git a/drivers/staging/fsl-mc/include/mc-private.h b/drivers/staging/fsl-mc/include/mc-private.h
index 97295f0..27834ea 100644
--- a/drivers/staging/fsl-mc/include/mc-private.h
+++ b/drivers/staging/fsl-mc/include/mc-private.h
@@ -114,7 +114,8 @@ void fsl_mc_device_remove(struct fsl_mc_device *mc_dev);
 
 int dprc_scan_container(struct fsl_mc_device *mc_bus_dev);
 
-int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev);
+int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
+		      unsigned int *total_irq_count);
 
 int __init dprc_driver_init(void);
 
-- 
2.3.3

[PATCH 07/10] staging: fsl-mc: set MSI domain for DPRC objects

[J. German Rivera]

THE MSI domain associated with a root DPRC object is
obtained form the device tree. Child DPRCs inherit
the parent DPRC MSI domain.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/dprc-driver.c | 39 ++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index 3bcd161..455379d 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -13,6 +13,7 @@
 #include "../include/mc-sys.h"
 #include <linux/module.h>
 #include <linux/slab.h>
+#include <linux/interrupt.h>
 #include "dprc-cmd.h"
 
 struct dprc_child_objs {
@@ -398,11 +399,16 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 {
 	int error;
 	size_t region_size;
+	struct device *parent_dev = mc_dev->dev.parent;
 	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+	bool msi_domain_set = false;
 
 	if (WARN_ON(strcmp(mc_dev->obj_desc.type, "dprc") != 0))
 		return -EINVAL;
 
+	if (WARN_ON(dev_get_msi_domain(&mc_dev->dev)))
+		return -EINVAL;
+
 	if (!mc_dev->mc_io) {
 		/*
 		 * This is a child DPRC:
@@ -421,6 +427,30 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 					 &mc_dev->mc_io);
 		if (error < 0)
 			return error;
+		/*
+		 * Inherit parent MSI domain:
+		 */
+		dev_set_msi_domain(&mc_dev->dev,
+				   dev_get_msi_domain(parent_dev));
+		msi_domain_set = true;
+	} else {
+		/*
+		 * This is a root DPRC
+		 */
+		struct irq_domain *mc_msi_domain;
+
+		if (WARN_ON(parent_dev->bus == &fsl_mc_bus_type))
+			return -EINVAL;
+
+		error = fsl_mc_find_msi_domain(parent_dev->of_node,
+					       &mc_msi_domain);
+		if (error < 0) {
+			dev_warn(&mc_dev->dev,
+				 "WARNING: MC bus without interrupt support\n");
+		} else {
+			dev_set_msi_domain(&mc_dev->dev, mc_msi_domain);
+			msi_domain_set = true;
+		}
 	}
 
 	error = dprc_open(mc_dev->mc_io, 0, mc_dev->obj_desc.id,
@@ -446,6 +476,9 @@ error_cleanup_open:
 	(void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
 
 error_cleanup_mc_io:
+	if (msi_domain_set)
+		dev_set_msi_domain(&mc_dev->dev, NULL);
+
 	fsl_destroy_mc_io(mc_dev->mc_io);
 	return error;
 }
@@ -463,6 +496,7 @@ error_cleanup_mc_io:
 static int dprc_remove(struct fsl_mc_device *mc_dev)
 {
 	int error;
+	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
 
 	if (WARN_ON(strcmp(mc_dev->obj_desc.type, "dprc") != 0))
 		return -EINVAL;
@@ -475,6 +509,11 @@ static int dprc_remove(struct fsl_mc_device *mc_dev)
 	if (error < 0)
 		dev_err(&mc_dev->dev, "dprc_close() failed: %d\n", error);
 
+	if (dev_get_msi_domain(&mc_dev->dev)) {
+		fsl_mc_cleanup_irq_pool(mc_bus);
+		dev_set_msi_domain(&mc_dev->dev, NULL);
+	}
+
 	dev_info(&mc_dev->dev, "DPRC device unbound from driver");
 	return 0;
 }
-- 
2.3.3

[PATCH 08/10] staging: fsl-mc: Fixed bug in dprc_probe() error path

[J. German Rivera]

Destroy mc_io in error path in dprc_probe() only if the mc_io was
created in this function. Minor refactoring in error labels.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/dprc-driver.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index 455379d..3eeafaa 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -401,6 +401,7 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 	size_t region_size;
 	struct device *parent_dev = mc_dev->dev.parent;
 	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+	bool mc_io_created = false;
 	bool msi_domain_set = false;
 
 	if (WARN_ON(strcmp(mc_dev->obj_desc.type, "dprc") != 0))
@@ -413,6 +414,9 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 		/*
 		 * This is a child DPRC:
 		 */
+		if (WARN_ON(parent_dev->bus != &fsl_mc_bus_type))
+			return -EINVAL;
+
 		if (WARN_ON(mc_dev->obj_desc.region_count == 0))
 			return -EINVAL;
 
@@ -427,6 +431,9 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 					 &mc_dev->mc_io);
 		if (error < 0)
 			return error;
+
+		mc_io_created = true;
+
 		/*
 		 * Inherit parent MSI domain:
 		 */
@@ -457,7 +464,7 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 			  &mc_dev->mc_handle);
 	if (error < 0) {
 		dev_err(&mc_dev->dev, "dprc_open() failed: %d\n", error);
-		goto error_cleanup_mc_io;
+		goto dprc_open_error;
 	}
 
 	mutex_init(&mc_bus->scan_mutex);
@@ -467,19 +474,23 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 	 */
 	error = dprc_scan_container(mc_dev);
 	if (error < 0)
-		goto error_cleanup_open;
+		goto dprc_scan_container_error;
+
 
 	dev_info(&mc_dev->dev, "DPRC device bound to driver");
 	return 0;
 
-error_cleanup_open:
+dprc_scan_container_error:
 	(void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
+dprc_open_error:
+	if (mc_io_created) {
+		fsl_destroy_mc_io(mc_dev->mc_io);
+		mc_dev->mc_io = NULL;
+	}
 
-error_cleanup_mc_io:
 	if (msi_domain_set)
 		dev_set_msi_domain(&mc_dev->dev, NULL);
 
-	fsl_destroy_mc_io(mc_dev->mc_io);
 	return error;
 }
 
-- 
2.3.3

[PATCH 09/10] staging: fsl-mc: Added DPRC interrupt handler

[J. German Rivera]

The interrupt handler for DPRC IRQs is added. DPRC IRQs are
generated for hot plug events related to DPAA2 objects in a given
DPRC. These events include, creating/destroying DPAA2 objects in
the DPRC, changing the "plugged" state of DPAA2 objects and moving
objects between DPRCs.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/dprc-driver.c | 247 +++++++++++++++++++++++++++++++
 1 file changed, 247 insertions(+)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index 3eeafaa..38ec2b2 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -14,6 +14,7 @@
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/interrupt.h>
+#include <linux/msi.h>
 #include "dprc-cmd.h"
 
 struct dprc_child_objs {
@@ -386,6 +387,230 @@ error:
 EXPORT_SYMBOL_GPL(dprc_scan_container);
 
 /**
+ * dprc_irq0_handler - Regular ISR for DPRC interrupt 0
+ *
+ * @irq: IRQ number of the interrupt being handled
+ * @arg: Pointer to device structure
+ */
+static irqreturn_t dprc_irq0_handler(int irq_num, void *arg)
+{
+	return IRQ_WAKE_THREAD;
+}
+
+/**
+ * dprc_irq0_handler_thread - Handler thread function for DPRC interrupt 0
+ *
+ * @irq: IRQ number of the interrupt being handled
+ * @arg: Pointer to device structure
+ */
+static irqreturn_t dprc_irq0_handler_thread(int irq_num, void *arg)
+{
+	int error;
+	u32 status;
+	struct device *dev = (struct device *)arg;
+	struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+	struct fsl_mc_io *mc_io = mc_dev->mc_io;
+	struct msi_desc *msi_desc = mc_dev->irqs[0]->msi_desc;
+
+	dev_dbg(dev, "DPRC IRQ %d triggered on CPU %u\n",
+		irq_num, smp_processor_id());
+
+	if (WARN_ON(!(mc_dev->flags & FSL_MC_IS_DPRC)))
+		return IRQ_HANDLED;
+
+	mutex_lock(&mc_bus->scan_mutex);
+	if (WARN_ON(!msi_desc || msi_desc->irq != (u32)irq_num))
+		goto out;
+
+	error = dprc_get_irq_status(mc_io, 0, mc_dev->mc_handle, 0,
+				    &status);
+	if (error < 0) {
+		dev_err(dev,
+			"dprc_get_irq_status() failed: %d\n", error);
+		goto out;
+	}
+
+	error = dprc_clear_irq_status(mc_io, 0, mc_dev->mc_handle, 0,
+				      status);
+	if (error < 0) {
+		dev_err(dev,
+			"dprc_clear_irq_status() failed: %d\n", error);
+		goto out;
+	}
+
+	if (status & (DPRC_IRQ_EVENT_OBJ_ADDED |
+		      DPRC_IRQ_EVENT_OBJ_REMOVED |
+		      DPRC_IRQ_EVENT_CONTAINER_DESTROYED |
+		      DPRC_IRQ_EVENT_OBJ_DESTROYED |
+		      DPRC_IRQ_EVENT_OBJ_CREATED)) {
+		unsigned int irq_count;
+
+		error = dprc_scan_objects(mc_dev, &irq_count);
+		if (error < 0) {
+			/*
+			 * If the error is -ENXIO, we ignore it, as it indicates
+			 * that the object scan was aborted, as we detected that
+			 * an object was removed from the DPRC in the MC, while
+			 * we were scanning the DPRC.
+			 */
+			if (error != -ENXIO) {
+				dev_err(dev, "dprc_scan_objects() failed: %d\n",
+					error);
+			}
+
+			goto out;
+		}
+
+		if (irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS) {
+			dev_warn(dev,
+				 "IRQs needed (%u) exceed IRQs preallocated (%u)\n",
+				 irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+		}
+	}
+
+out:
+	mutex_unlock(&mc_bus->scan_mutex);
+	return IRQ_HANDLED;
+}
+
+/*
+ * Disable and clear interrupt for a given DPRC object
+ */
+static int disable_dprc_irq(struct fsl_mc_device *mc_dev)
+{
+	int error;
+	struct fsl_mc_io *mc_io = mc_dev->mc_io;
+
+	WARN_ON(mc_dev->obj_desc.irq_count != 1);
+
+	/*
+	 * Disable generation of interrupt, while we configure it:
+	 */
+	error = dprc_set_irq_enable(mc_io, 0, mc_dev->mc_handle, 0, 0);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Disabling DPRC IRQ failed: dprc_set_irq_enable() failed: %d\n",
+			error);
+		return error;
+	}
+
+	/*
+	 * Disable all interrupt causes for the interrupt:
+	 */
+	error = dprc_set_irq_mask(mc_io, 0, mc_dev->mc_handle, 0, 0x0);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Disabling DPRC IRQ failed: dprc_set_irq_mask() failed: %d\n",
+			error);
+		return error;
+	}
+
+	/*
+	 * Clear any leftover interrupts:
+	 */
+	error = dprc_clear_irq_status(mc_io, 0, mc_dev->mc_handle, 0, ~0x0U);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Disabling DPRC IRQ failed: dprc_clear_irq_status() failed: %d\n",
+			error);
+		return error;
+	}
+
+	return 0;
+}
+
+static int register_dprc_irq_handler(struct fsl_mc_device *mc_dev)
+{
+	int error;
+	struct fsl_mc_device_irq *irq = mc_dev->irqs[0];
+
+	WARN_ON(mc_dev->obj_desc.irq_count != 1);
+
+	/*
+	 * NOTE: devm_request_threaded_irq() invokes the device-specific
+	 * function that programs the MSI physically in the device
+	 */
+	error = devm_request_threaded_irq(&mc_dev->dev,
+					  irq->msi_desc->irq,
+					  dprc_irq0_handler,
+					  dprc_irq0_handler_thread,
+					  IRQF_NO_SUSPEND | IRQF_ONESHOT,
+					  "FSL MC DPRC irq0",
+					  &mc_dev->dev);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"devm_request_threaded_irq() failed: %d\n",
+			error);
+		return error;
+	}
+
+	return 0;
+}
+
+static int enable_dprc_irq(struct fsl_mc_device *mc_dev)
+{
+	int error;
+
+	/*
+	 * Enable all interrupt causes for the interrupt:
+	 */
+	error = dprc_set_irq_mask(mc_dev->mc_io, 0, mc_dev->mc_handle, 0,
+				  ~0x0u);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Enabling DPRC IRQ failed: dprc_set_irq_mask() failed: %d\n",
+			error);
+
+		return error;
+	}
+
+	/*
+	 * Enable generation of the interrupt:
+	 */
+	error = dprc_set_irq_enable(mc_dev->mc_io, 0, mc_dev->mc_handle, 0, 1);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Enabling DPRC IRQ failed: dprc_set_irq_enable() failed: %d\n",
+			error);
+
+		return error;
+	}
+
+	return 0;
+}
+
+/*
+ * Setup interrupt for a given DPRC device
+ */
+static int dprc_setup_irq(struct fsl_mc_device *mc_dev)
+{
+	int error;
+
+	error = fsl_mc_allocate_irqs(mc_dev);
+	if (error < 0)
+		return error;
+
+	error = disable_dprc_irq(mc_dev);
+	if (error < 0)
+		goto error_free_irqs;
+
+	error = register_dprc_irq_handler(mc_dev);
+	if (error < 0)
+		goto error_free_irqs;
+
+	error = enable_dprc_irq(mc_dev);
+	if (error < 0)
+		goto error_free_irqs;
+
+	return 0;
+
+error_free_irqs:
+	fsl_mc_free_irqs(mc_dev);
+	return error;
+}
+
+/**
  * dprc_probe - callback invoked when a DPRC is being bound to this driver
  *
  * @mc_dev: Pointer to fsl-mc device representing a DPRC
@@ -476,10 +701,17 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 	if (error < 0)
 		goto dprc_scan_container_error;
 
+	/*
+	 * Configure interrupt for the DPRC object associated with this MC bus:
+	 */
+	error = dprc_setup_irq(mc_dev);
+	if (error < 0)
+		goto dprc_setup_irqs_error;
 
 	dev_info(&mc_dev->dev, "DPRC device bound to driver");
 	return 0;
 
+dprc_setup_irqs_error:
 dprc_scan_container_error:
 	(void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
 dprc_open_error:
@@ -494,6 +726,15 @@ dprc_open_error:
 	return error;
 }
 
+/*
+ * Tear down interrupt for a given DPRC object
+ */
+static void dprc_teardown_irq(struct fsl_mc_device *mc_dev)
+{
+	(void)disable_dprc_irq(mc_dev);
+	fsl_mc_free_irqs(mc_dev);
+}
+
 /**
  * dprc_remove - callback invoked when a DPRC is being unbound from this driver
  *
@@ -514,6 +755,12 @@ static int dprc_remove(struct fsl_mc_device *mc_dev)
 	if (WARN_ON(!mc_dev->mc_io))
 		return -EINVAL;
 
+	if (WARN_ON(!mc_bus->irq_resources))
+		return -EINVAL;
+
+	if (dev_get_msi_domain(&mc_dev->dev))
+		dprc_teardown_irq(mc_dev);
+
 	device_for_each_child(&mc_dev->dev, NULL, __fsl_mc_device_remove);
 	dprc_cleanup_all_resource_pools(mc_dev);
 	error = dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
-- 
2.3.3

[PATCH 10/10] staging: fsl-mc: Added MSI support to the MC bus driver

[J. German Rivera]

Initialize/Cleanup ITS-MSI support for the MC bus driver at driver
init/exit time. Associate an MSI domain with each DPAA2 child device.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
 drivers/staging/fsl-mc/bus/mc-bus.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/staging/fsl-mc/bus/mc-bus.c b/drivers/staging/fsl-mc/bus/mc-bus.c
index d34f1af..9317561 100644
--- a/drivers/staging/fsl-mc/bus/mc-bus.c
+++ b/drivers/staging/fsl-mc/bus/mc-bus.c
@@ -16,6 +16,8 @@
 #include <linux/ioport.h>
 #include <linux/slab.h>
 #include <linux/limits.h>
+#include <linux/bitops.h>
+#include <linux/msi.h>
 #include "../include/dpmng.h"
 #include "../include/mc-sys.h"
 #include "dprc-cmd.h"
@@ -472,6 +474,8 @@ int fsl_mc_device_add(struct dprc_obj_desc *obj_desc,
 		mc_dev->icid = parent_mc_dev->icid;
 		mc_dev->dma_mask = FSL_MC_DEFAULT_DMA_MASK;
 		mc_dev->dev.dma_mask = &mc_dev->dma_mask;
+		dev_set_msi_domain(&mc_dev->dev,
+				   dev_get_msi_domain(&parent_mc_dev->dev));
 	}
 
 	/*
@@ -833,8 +837,15 @@ static int __init fsl_mc_bus_driver_init(void)
 	if (error < 0)
 		goto error_cleanup_dprc_driver;
 
+	error = its_fsl_mc_msi_init();
+	if (error < 0)
+		goto error_cleanup_mc_allocator;
+
 	return 0;
 
+error_cleanup_mc_allocator:
+	fsl_mc_allocator_driver_exit();
+
 error_cleanup_dprc_driver:
 	dprc_driver_exit();
 
@@ -856,6 +867,7 @@ static void __exit fsl_mc_bus_driver_exit(void)
 	if (WARN_ON(!mc_dev_cache))
 		return;
 
+	its_fsl_mc_msi_cleanup();
 	fsl_mc_allocator_driver_exit();
 	dprc_driver_exit();
 	platform_driver_unregister(&fsl_mc_bus_driver);
-- 
2.3.3

Re: [PATCH 02/10] staging: fsl-mc: Added generic MSI support for FSL-MC devices

[Jiang Liu]

On 2015/10/26 23:49, J. German Rivera wrote:
> Created an MSI domain for the fsl-mc bus-- including functions
> to create a domain, find a domain, alloc/free domain irqs, and
> bus specific overrides for domain and irq_chip ops.
> 
> Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
> ---
>  drivers/staging/fsl-mc/bus/Kconfig          |   1 +
>  drivers/staging/fsl-mc/bus/Makefile         |   1 +
>  drivers/staging/fsl-mc/bus/mc-msi.c         | 278 ++++++++++++++++++++++++++++
>  drivers/staging/fsl-mc/include/mc-private.h |  17 ++
>  drivers/staging/fsl-mc/include/mc.h         |  17 ++
>  5 files changed, 314 insertions(+)
>  create mode 100644 drivers/staging/fsl-mc/bus/mc-msi.c
> 
<snit>

> +
> +static void fsl_mc_msi_free_descs(struct device *dev)
> +{
> +	struct msi_desc *desc, *tmp;
> +
> +	list_for_each_entry_safe(desc, tmp, dev_to_msi_list(dev), list) {
> +		list_del(&desc->list);
> +		free_msi_entry(desc);
> +	}
> +}
> +
> +static int fsl_mc_msi_alloc_descs(struct device *dev, unsigned int irq_count)
> +
> +{
> +	unsigned int i;
> +	int error;
> +	struct msi_desc *msi_desc;
> +
> +	for (i = 0; i < irq_count; i++) {
> +		msi_desc = alloc_msi_entry(dev);
> +		if (!msi_desc) {
> +			dev_err(dev, "Failed to allocate msi entry\n");
> +			error = -ENOMEM;
> +			goto cleanup_msi_descs;
> +		}
> +
> +		msi_desc->msi_attrib.is_msix = 1;
> +		msi_desc->msi_attrib.is_64 = 1;
> +		msi_desc->msi_attrib.entry_nr = i;

Hi Rivera,
	Field msi_desc->msi_attrib is for PCI MSI only, it would be better to
introduce a dedicated structure for FSL-MC, just like
struct platform_msi_desc.
Thanks,
Gerry

> +		msi_desc->nvec_used = 1;
> +		INIT_LIST_HEAD(&msi_desc->list);
> +		list_add_tail(&msi_desc->list, dev_to_msi_list(dev));
> +	}
> +
> +	return 0;
> +
> +cleanup_msi_descs:
> +	fsl_mc_msi_free_descs(dev);
> +	return error;
> +}
> +
> +int fsl_mc_msi_domain_alloc_irqs(struct device *dev,
> +				 unsigned int irq_count)
> +{
> +	struct irq_domain *msi_domain;
> +	int error;
> +
> +	if (WARN_ON(!list_empty(dev_to_msi_list(dev))))
> +		return -EINVAL;
> +
> +	error = fsl_mc_msi_alloc_descs(dev, irq_count);
> +	if (error < 0)
> +		return error;
> +
> +	msi_domain = dev_get_msi_domain(dev);
> +	if (WARN_ON(!msi_domain)) {
> +		error = -EINVAL;
> +		goto cleanup_msi_descs;
> +	}
> +
> +	/*
> +	 * NOTE: Calling this function will trigger the invocation of the
> +	 * its_fsl_mc_msi_prepare() callback
> +	 */
> +	error = msi_domain_alloc_irqs(msi_domain, dev, irq_count);
> +
> +	if (error) {
> +		dev_err(dev, "Failed to allocate IRQs\n");
> +		goto cleanup_msi_descs;
> +	}
> +
> +	return 0;
> +
> +cleanup_msi_descs:
> +	fsl_mc_msi_free_descs(dev);
> +	return error;
> +}
> +
<snit>

Re: [PATCH 08/10] staging: fsl-mc: Fixed bug in dprc_probe() error path

[Dan Carpenter]

On Mon, Oct 26, 2015 at 10:49:19AM -0500, J. German Rivera wrote:
> -error_cleanup_open:
> +dprc_scan_container_error:
>  	(void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);

The error label was better in the original code.  A "goto
error_cleanup_open", that tells you what the goto does.  "goto
dprc_scan_container_error" tells you nothing because you can see from
the line before that there was a container error.  No new information.
It's just like a function name tells you what a function does.  You
wouldn't name the function called_from_main().

regards,
dan carpenter