[PATCH v2 00/11] staging: fsl-mc: MC bus MSI support

[PATCH v2 00/11] staging: fsl-mc: MC bus MSI support

[J. German Rivera]

This patch series addresses the following item from the TODO list
for the MC bus driver to exit staging:

* Interrupt support. For meaningful driver support we need
  interrupts, and thus need message interrupt support by the bus
  driver.

MC Bus MSI Support Architecture
===============================
A new IRQ domain bus token is added for the FSL-MC bus.
An MSI IRQ domain is created for each top-level (root) data-path
resource container (DPRC), based on its msi-parent in the device
tree (which is the GIC-ITS). Child DPRCs inherit the MSI IRQ
domain form their parent DPRC.

MC Bus MSI Allocation
---------------------
Given the way in which the GIC-ITS works, we need to pre-allocate
a block of MSIs in the GIC-ITS for the IRQs of all the DPAA2 objects
in the same data-path resource container (DPRC) and for the IRQ of
the DPRC iself.

This is due to the fact that all the IRQs for DPAA2 objects in the
same DPRC (and the DPRC's own IRQ) must use the same "device Id" in
the GIC-ITS. Thus, all these IRQs must share the same ITT table in
the GIC-ITS, and therefore must be allocated in the GIC-ITS as
a block of MSIs for the same "device Id".

This is because all the DPAA2 objects in the same DPRC (and the
DPRC itself) use the DPRC's SMMU stream ID as their device Id for
the GIC-ITS.
The DPAA2 Management Complex (MC) firmware does not assign a separate
SMMU stream ID to each DPAA2 object. The MC only assigns SMMU stream
IDs to DPRCs. In MC terms, the stream ID assigned to a DPRC is known
as the DPRC's Isolation Context ID (ICID).

As a consequence of having to pre-allocate a block of MSIs in
the GIC-ITS, the object allocator of the MC bus driver needs to be
extended to provide IRQ allocation services to DPAA2 device drivers
and to the DPRC driver. For a given DPAA2 object, MSIs are allocated
from the corresponding DPRC's pool of pre-allocated MSIs. The MSI
for the DPRC itself is also allocated from this pool.

The following are the patches in this series:

Patch 1: Added domain bus token DOMAIN_BUS_FSL_MC_MSI
Patch 2: Added generic MSI support for FSL-MC devices
Patch 3: Added GICv3-ITS support for FSL-MC MSIs
Patch 4: Extended MC bus allocator to include IRQs
Patch 5: Changed DPRC built-in portal's mc_io to be atomic
Patch 6: Populate the IRQ pool for an MC bus instance
Patch 7: Set MSI domain for DPRC objects
Patch 8: Fixed bug in dprc_probe() error path
Patch 9: Added DPRC interrupt handler
Patch 10: Added MSI support to the MC bus driver

CHANGE HISTORY

Changes in v2:
- Addressed comment from Jiang Liu
  * Added a dedicated structure for FSL-MC in struct msi_desc
- Addressed comment from Dan Carpenter
  * Renamed goto error labels to indicate what the goto does

[PATCH v2 01/11] irqdomain: Added domain bus token DOMAIN_BUS_FSL_MC_MSI

[J. German Rivera]

Since an FSL-MC bus is a new bus type that is neither PCI nor
PLATFORM, we need a new domain bus token to disambiguate the
IRQ domain for FSL-MC MSIs.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 include/linux/irqdomain.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index d5e5c5b..c0cb5d1 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -73,6 +73,7 @@ enum irq_domain_bus_token {
 	DOMAIN_BUS_PCI_MSI,
 	DOMAIN_BUS_PLATFORM_MSI,
 	DOMAIN_BUS_NEXUS,
+	DOMAIN_BUS_FSL_MC_MSI,
 };

 /**
--
2.3.3

Re: [PATCH v2 01/11] irqdomain: Added domain bus token DOMAIN_BUS_FSL_MC_MSI

[Jiang Liu]

On 2015/10/31 3:43, J. German Rivera wrote:
> Since an FSL-MC bus is a new bus type that is neither PCI nor
> PLATFORM, we need a new domain bus token to disambiguate the
> IRQ domain for FSL-MC MSIs.
> 
> Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
> ---
> Changes in v2: none
> 
>  include/linux/irqdomain.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
> index d5e5c5b..c0cb5d1 100644
> --- a/include/linux/irqdomain.h
> +++ b/include/linux/irqdomain.h
> @@ -73,6 +73,7 @@ enum irq_domain_bus_token {
>  	DOMAIN_BUS_PCI_MSI,
>  	DOMAIN_BUS_PLATFORM_MSI,
>  	DOMAIN_BUS_NEXUS,
> +	DOMAIN_BUS_FSL_MC_MSI,
>  };

Reviewed-by: Jiang Liu <jiang.liu@linux.intel.com>

> 
>  /**
> --
> 2.3.3
> 

[PATCH v2 02/11] fsl-mc: msi: Added FSL-MC-specific member to the msi_desc's union

[J. German Rivera]

FSL-MC is a bus type different from PCI and platform, so it needs
its own member in the msi_desc's union.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2:
- Addressed comment from Jiang Liu
  * Added a dedicated structure for FSL-MC in struct msi_desc

 include/linux/msi.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/include/linux/msi.h b/include/linux/msi.h
index f71a25e..152e51a 100644
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -33,6 +33,14 @@ struct platform_msi_desc {
 };

 /**
+ * fsl_mc_msi_desc - FSL-MC device specific msi descriptor data
+ * @msi_index:		The index of the MSI descriptor
+ */
+struct fsl_mc_msi_desc {
+	u16				msi_index;
+};
+
+/**
  * struct msi_desc - Descriptor structure for MSI based interrupts
  * @list:	List head for management
  * @irq:	The base interrupt number
@@ -87,6 +95,7 @@ struct msi_desc {
 		 * tree wide cleanup.
 		 */
 		struct platform_msi_desc platform;
+		struct fsl_mc_msi_desc fsl_mc;
 	};
 };

--
2.3.3

Re: [PATCH v2 02/11] fsl-mc: msi: Added FSL-MC-specific member to the msi_desc's union

[Jiang Liu]

On 2015/10/31 3:43, J. German Rivera wrote:
> FSL-MC is a bus type different from PCI and platform, so it needs
> its own member in the msi_desc's union.
> 
> Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
> ---
> Changes in v2:
> - Addressed comment from Jiang Liu
>   * Added a dedicated structure for FSL-MC in struct msi_desc
> 
>  include/linux/msi.h | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/include/linux/msi.h b/include/linux/msi.h
> index f71a25e..152e51a 100644
> --- a/include/linux/msi.h
> +++ b/include/linux/msi.h
> @@ -33,6 +33,14 @@ struct platform_msi_desc {
>  };
> 
>  /**
> + * fsl_mc_msi_desc - FSL-MC device specific msi descriptor data
> + * @msi_index:		The index of the MSI descriptor
> + */
> +struct fsl_mc_msi_desc {
> +	u16				msi_index;
> +};
> +
> +/**
>   * struct msi_desc - Descriptor structure for MSI based interrupts
>   * @list:	List head for management
>   * @irq:	The base interrupt number
> @@ -87,6 +95,7 @@ struct msi_desc {
>  		 * tree wide cleanup.
>  		 */
>  		struct platform_msi_desc platform;
> +		struct fsl_mc_msi_desc fsl_mc;
>  	};
>  };

Reviewed-by: Jiang Liu <jiang.liu@linux.intel.com>

> 
> --
> 2.3.3
> 

[PATCH v2 03/11] staging: fsl-mc: Added generic MSI support for FSL-MC devices

[J. German Rivera]

Created an MSI domain for the fsl-mc bus-- including functions
to create a domain, find a domain, alloc/free domain irqs, and
bus specific overrides for domain and irq_chip ops.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 drivers/staging/fsl-mc/bus/Kconfig          |   1 +
 drivers/staging/fsl-mc/bus/Makefile         |   1 +
 drivers/staging/fsl-mc/bus/mc-msi.c         | 276 ++++++++++++++++++++++++++++
 drivers/staging/fsl-mc/include/mc-private.h |  17 ++
 drivers/staging/fsl-mc/include/mc.h         |  17 ++
 5 files changed, 312 insertions(+)
 create mode 100644 drivers/staging/fsl-mc/bus/mc-msi.c

diff --git a/drivers/staging/fsl-mc/bus/Kconfig b/drivers/staging/fsl-mc/bus/Kconfig
index 0d779d9..c498ac6 100644
--- a/drivers/staging/fsl-mc/bus/Kconfig
+++ b/drivers/staging/fsl-mc/bus/Kconfig
@@ -9,6 +9,7 @@
 config FSL_MC_BUS
 	tristate "Freescale Management Complex (MC) bus driver"
 	depends on OF && ARM64
+	select GENERIC_MSI_IRQ_DOMAIN
 	help
 	  Driver to enable the bus infrastructure for the Freescale
           QorIQ Management Complex (fsl-mc). The fsl-mc is a hardware
diff --git a/drivers/staging/fsl-mc/bus/Makefile b/drivers/staging/fsl-mc/bus/Makefile
index 25433a9..a5f2ba4 100644
--- a/drivers/staging/fsl-mc/bus/Makefile
+++ b/drivers/staging/fsl-mc/bus/Makefile
@@ -13,5 +13,6 @@ mc-bus-driver-objs := mc-bus.o \
 		      dpmng.o \
 		      dprc-driver.o \
 		      mc-allocator.o \
+		      mc-msi.o \
 		      dpmcp.o \
 		      dpbp.o
diff --git a/drivers/staging/fsl-mc/bus/mc-msi.c b/drivers/staging/fsl-mc/bus/mc-msi.c
new file mode 100644
index 0000000..81b53e3
--- /dev/null
+++ b/drivers/staging/fsl-mc/bus/mc-msi.c
@@ -0,0 +1,276 @@
+/*
+ * Freescale Management Complex (MC) bus driver MSI support
+ *
+ * Copyright (C) 2015 Freescale Semiconductor, Inc.
+ * Author: German Rivera <German.Rivera@freescale.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include "../include/mc-private.h"
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/irqchip/arm-gic-v3.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
+#include <linux/msi.h>
+#include "../include/mc-sys.h"
+#include "dprc-cmd.h"
+
+static void fsl_mc_msi_set_desc(msi_alloc_info_t *arg,
+				struct msi_desc *desc)
+{
+	arg->desc = desc;
+	arg->hwirq = (irq_hw_number_t)desc->fsl_mc.msi_index;
+}
+
+static void fsl_mc_msi_update_dom_ops(struct msi_domain_info *info)
+{
+	struct msi_domain_ops *ops = info->ops;
+
+	if (WARN_ON(!ops))
+		return;
+
+	if (!ops->set_desc)
+		ops->set_desc = fsl_mc_msi_set_desc;
+}
+
+static void __fsl_mc_msi_write_msg(struct fsl_mc_device *mc_bus_dev,
+				   struct fsl_mc_device_irq *mc_dev_irq)
+{
+	int error;
+	struct fsl_mc_device *owner_mc_dev = mc_dev_irq->mc_dev;
+	struct msi_desc *msi_desc = mc_dev_irq->msi_desc;
+	struct dprc_irq_cfg irq_cfg;
+
+	/*
+	 * msi_desc->msg.address is 0x0 when this function is invoked in
+	 * the free_irq() code path. In this case, for the MC, we don't
+	 * really need to "unprogram" the MSI, so we just return.
+	 */
+	if (msi_desc->msg.address_lo == 0x0 && msi_desc->msg.address_hi == 0x0)
+		return;
+
+	if (WARN_ON(!owner_mc_dev))
+		return;
+
+	irq_cfg.paddr = ((u64)msi_desc->msg.address_hi << 32) |
+			msi_desc->msg.address_lo;
+	irq_cfg.val = msi_desc->msg.data;
+	irq_cfg.user_irq_id = msi_desc->irq;
+
+	/*
+	 * DPRCs and other objects use different commands to set up IRQs,
+	 * so we have to differentiate them here.
+	 */
+	if (owner_mc_dev == mc_bus_dev) {
+		/*
+		 * IRQ is for the mc_bus_dev's DPRC itself
+		 */
+		error = dprc_set_irq(mc_bus_dev->mc_io,
+				     MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
+				     mc_bus_dev->mc_handle,
+				     mc_dev_irq->dev_irq_index,
+				     &irq_cfg);
+		if (error < 0) {
+			dev_err(&owner_mc_dev->dev,
+				"dprc_set_irq() failed: %d\n", error);
+		}
+	} else {
+		error = dprc_set_obj_irq(mc_bus_dev->mc_io,
+					 MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
+					 mc_bus_dev->mc_handle,
+					 owner_mc_dev->obj_desc.type,
+					 owner_mc_dev->obj_desc.id,
+					 mc_dev_irq->dev_irq_index,
+					 &irq_cfg);
+		if (error < 0) {
+			dev_err(&owner_mc_dev->dev,
+				"dprc_obj_set_irq() failed: %d\n", error);
+		}
+	}
+}
+
+/*
+ * NOTE: This function is invoked with interrupts disabled
+ */
+static void fsl_mc_msi_write_msg(struct irq_data *irq_data,
+				 struct msi_msg *msg)
+{
+	struct msi_desc *msi_desc = irq_data_get_msi_desc(irq_data);
+	struct fsl_mc_device *mc_bus_dev = to_fsl_mc_device(msi_desc->dev);
+	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
+	struct fsl_mc_device_irq *mc_dev_irq =
+		&mc_bus->irq_resources[msi_desc->fsl_mc.msi_index];
+
+	WARN_ON(mc_dev_irq->msi_desc != msi_desc);
+	msi_desc->msg = *msg;
+
+	/*
+	 * Program the MSI (paddr, value) pair in the device:
+	 */
+	__fsl_mc_msi_write_msg(mc_bus_dev, mc_dev_irq);
+}
+
+static void fsl_mc_msi_update_chip_ops(struct msi_domain_info *info)
+{
+	struct irq_chip *chip = info->chip;
+
+	if (WARN_ON((!chip)))
+		return;
+
+	if (!chip->irq_write_msi_msg)
+		chip->irq_write_msi_msg = fsl_mc_msi_write_msg;
+}
+
+/**
+ * fsl_mc_msi_create_irq_domain - Create a fsl-mc MSI interrupt domain
+ * @np:		Optional device-tree node of the interrupt controller
+ * @info:	MSI domain info
+ * @parent:	Parent irq domain
+ *
+ * Updates the domain and chip ops and creates a fsl-mc MSI
+ * interrupt domain.
+ *
+ * Returns:
+ * A domain pointer or NULL in case of failure.
+ */
+struct irq_domain *fsl_mc_msi_create_irq_domain(struct fwnode_handle *fwnode,
+						struct msi_domain_info *info,
+						struct irq_domain *parent)
+{
+	struct irq_domain *domain;
+
+	if (info->flags & MSI_FLAG_USE_DEF_DOM_OPS)
+		fsl_mc_msi_update_dom_ops(info);
+	if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS)
+		fsl_mc_msi_update_chip_ops(info);
+
+	domain = msi_create_irq_domain(fwnode, info, parent);
+	if (domain)
+		domain->bus_token = DOMAIN_BUS_FSL_MC_MSI;
+
+	return domain;
+}
+
+int fsl_mc_find_msi_domain(struct device_node *mc_of_node,
+			   struct irq_domain **mc_msi_domain)
+{
+	struct device_node *msi_parent_node;
+	struct irq_domain *msi_domain;
+
+	msi_parent_node = of_parse_phandle(mc_of_node, "msi-parent", 0);
+	if (!msi_parent_node) {
+		pr_err("msi-parent phandle missing for %s\n",
+		       mc_of_node->full_name);
+		return -ENOENT;
+	}
+
+	/*
+	 * Look up the fsl-mc MSI domain:
+	 */
+	msi_domain = irq_find_matching_host(msi_parent_node,
+					    DOMAIN_BUS_FSL_MC_MSI);
+	if (!msi_domain) {
+		pr_err("Unable to find fsl-mc MSI domain for %s\n",
+		       mc_of_node->full_name);
+
+		return -ENOENT;
+	}
+
+	*mc_msi_domain = msi_domain;
+	return 0;
+}
+
+static void fsl_mc_msi_free_descs(struct device *dev)
+{
+	struct msi_desc *desc, *tmp;
+
+	list_for_each_entry_safe(desc, tmp, dev_to_msi_list(dev), list) {
+		list_del(&desc->list);
+		free_msi_entry(desc);
+	}
+}
+
+static int fsl_mc_msi_alloc_descs(struct device *dev, unsigned int irq_count)
+
+{
+	unsigned int i;
+	int error;
+	struct msi_desc *msi_desc;
+
+	for (i = 0; i < irq_count; i++) {
+		msi_desc = alloc_msi_entry(dev);
+		if (!msi_desc) {
+			dev_err(dev, "Failed to allocate msi entry\n");
+			error = -ENOMEM;
+			goto cleanup_msi_descs;
+		}
+
+		msi_desc->fsl_mc.msi_index = i;
+		msi_desc->nvec_used = 1;
+		INIT_LIST_HEAD(&msi_desc->list);
+		list_add_tail(&msi_desc->list, dev_to_msi_list(dev));
+	}
+
+	return 0;
+
+cleanup_msi_descs:
+	fsl_mc_msi_free_descs(dev);
+	return error;
+}
+
+int fsl_mc_msi_domain_alloc_irqs(struct device *dev,
+				 unsigned int irq_count)
+{
+	struct irq_domain *msi_domain;
+	int error;
+
+	if (WARN_ON(!list_empty(dev_to_msi_list(dev))))
+		return -EINVAL;
+
+	error = fsl_mc_msi_alloc_descs(dev, irq_count);
+	if (error < 0)
+		return error;
+
+	msi_domain = dev_get_msi_domain(dev);
+	if (WARN_ON(!msi_domain)) {
+		error = -EINVAL;
+		goto cleanup_msi_descs;
+	}
+
+	/*
+	 * NOTE: Calling this function will trigger the invocation of the
+	 * its_fsl_mc_msi_prepare() callback
+	 */
+	error = msi_domain_alloc_irqs(msi_domain, dev, irq_count);
+
+	if (error) {
+		dev_err(dev, "Failed to allocate IRQs\n");
+		goto cleanup_msi_descs;
+	}
+
+	return 0;
+
+cleanup_msi_descs:
+	fsl_mc_msi_free_descs(dev);
+	return error;
+}
+
+void fsl_mc_msi_domain_free_irqs(struct device *dev)
+{
+	struct irq_domain *msi_domain;
+
+	msi_domain = dev_get_msi_domain(dev);
+	if (WARN_ON(!msi_domain))
+		return;
+
+	msi_domain_free_irqs(msi_domain, dev);
+
+	if (WARN_ON(list_empty(dev_to_msi_list(dev))))
+		return;
+
+	fsl_mc_msi_free_descs(dev);
+}
diff --git a/drivers/staging/fsl-mc/include/mc-private.h b/drivers/staging/fsl-mc/include/mc-private.h
index 2c4cc79..3dabd003 100644
--- a/drivers/staging/fsl-mc/include/mc-private.h
+++ b/drivers/staging/fsl-mc/include/mc-private.h
@@ -26,6 +26,9 @@
 	 strcmp(_obj_type, "dpmcp") == 0 || \
 	 strcmp(_obj_type, "dpcon") == 0)

+struct irq_domain;
+struct msi_domain_info;
+
 /**
  * struct fsl_mc - Private data of a "fsl,qoriq-mc" platform device
  * @root_mc_bus_dev: MC object device representing the root DPRC
@@ -79,11 +82,13 @@ struct fsl_mc_resource_pool {
  * @resource_pools: array of resource pools (one pool per resource type)
  * for this MC bus. These resources represent allocatable entities
  * from the physical DPRC.
+ * @irq_resources: Pointer to array of IRQ objects for the IRQ pool
  * @scan_mutex: Serializes bus scanning
  */
 struct fsl_mc_bus {
 	struct fsl_mc_device mc_dev;
 	struct fsl_mc_resource_pool resource_pools[FSL_MC_NUM_POOL_TYPES];
+	struct fsl_mc_device_irq *irq_resources;
 	struct mutex scan_mutex;    /* serializes bus scanning */
 };

@@ -116,4 +121,16 @@ int __must_check fsl_mc_resource_allocate(struct fsl_mc_bus *mc_bus,

 void fsl_mc_resource_free(struct fsl_mc_resource *resource);

+struct irq_domain *fsl_mc_msi_create_irq_domain(struct fwnode_handle *fwnode,
+						struct msi_domain_info *info,
+						struct irq_domain *parent);
+
+int fsl_mc_find_msi_domain(struct device_node *mc_of_node,
+			   struct irq_domain **mc_msi_domain);
+
+int fsl_mc_msi_domain_alloc_irqs(struct device *dev,
+				 unsigned int irq_count);
+
+void fsl_mc_msi_domain_free_irqs(struct device *dev);
+
 #endif /* _FSL_MC_PRIVATE_H_ */
diff --git a/drivers/staging/fsl-mc/include/mc.h b/drivers/staging/fsl-mc/include/mc.h
index a933291..1c1d6ae 100644
--- a/drivers/staging/fsl-mc/include/mc.h
+++ b/drivers/staging/fsl-mc/include/mc.h
@@ -104,6 +104,23 @@ struct fsl_mc_resource {
 };

 /**
+ * struct fsl_mc_device_irq - MC object device message-based interrupt
+ * @msi_desc: pointer to MSI descriptor allocated by fsl_mc_msi_alloc_descs()
+ * @mc_dev: MC object device that owns this interrupt
+ * @dev_irq_index: device-relative IRQ index
+ * @resource: MC generic resource associated with the interrupt
+ */
+struct fsl_mc_device_irq {
+	struct msi_desc *msi_desc;
+	struct fsl_mc_device *mc_dev;
+	u8 dev_irq_index;
+	struct fsl_mc_resource resource;
+};
+
+#define to_fsl_mc_irq(_mc_resource) \
+	container_of(_mc_resource, struct fsl_mc_device_irq, resource)
+
+/**
  * Bit masks for a MC object device (struct fsl_mc_device) flags
  */
 #define FSL_MC_IS_DPRC	0x0001
--
2.3.3

Re: [PATCH v2 03/11] staging: fsl-mc: Added generic MSI support for FSL-MC devices

[Marc Zyngier]

On 30/10/15 19:43, J. German Rivera wrote:
> Created an MSI domain for the fsl-mc bus-- including functions
> to create a domain, find a domain, alloc/free domain irqs, and
> bus specific overrides for domain and irq_chip ops.
> 
> Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
> ---
> Changes in v2: none
> 
>  drivers/staging/fsl-mc/bus/Kconfig          |   1 +
>  drivers/staging/fsl-mc/bus/Makefile         |   1 +
>  drivers/staging/fsl-mc/bus/mc-msi.c         | 276 ++++++++++++++++++++++++++++
>  drivers/staging/fsl-mc/include/mc-private.h |  17 ++
>  drivers/staging/fsl-mc/include/mc.h         |  17 ++
>  5 files changed, 312 insertions(+)
>  create mode 100644 drivers/staging/fsl-mc/bus/mc-msi.c
> 
> diff --git a/drivers/staging/fsl-mc/bus/Kconfig b/drivers/staging/fsl-mc/bus/Kconfig
> index 0d779d9..c498ac6 100644
> --- a/drivers/staging/fsl-mc/bus/Kconfig
> +++ b/drivers/staging/fsl-mc/bus/Kconfig
> @@ -9,6 +9,7 @@
>  config FSL_MC_BUS
>  	tristate "Freescale Management Complex (MC) bus driver"
>  	depends on OF && ARM64
> +	select GENERIC_MSI_IRQ_DOMAIN
>  	help
>  	  Driver to enable the bus infrastructure for the Freescale
>            QorIQ Management Complex (fsl-mc). The fsl-mc is a hardware
> diff --git a/drivers/staging/fsl-mc/bus/Makefile b/drivers/staging/fsl-mc/bus/Makefile
> index 25433a9..a5f2ba4 100644
> --- a/drivers/staging/fsl-mc/bus/Makefile
> +++ b/drivers/staging/fsl-mc/bus/Makefile
> @@ -13,5 +13,6 @@ mc-bus-driver-objs := mc-bus.o \
>  		      dpmng.o \
>  		      dprc-driver.o \
>  		      mc-allocator.o \
> +		      mc-msi.o \
>  		      dpmcp.o \
>  		      dpbp.o
> diff --git a/drivers/staging/fsl-mc/bus/mc-msi.c b/drivers/staging/fsl-mc/bus/mc-msi.c
> new file mode 100644
> index 0000000..81b53e3
> --- /dev/null
> +++ b/drivers/staging/fsl-mc/bus/mc-msi.c
> @@ -0,0 +1,276 @@
> +/*
> + * Freescale Management Complex (MC) bus driver MSI support
> + *
> + * Copyright (C) 2015 Freescale Semiconductor, Inc.
> + * Author: German Rivera <German.Rivera@freescale.com>
> + *
> + * This file is licensed under the terms of the GNU General Public
> + * License version 2. This program is licensed "as is" without any
> + * warranty of any kind, whether express or implied.
> + */
> +
> +#include "../include/mc-private.h"
> +#include <linux/of_device.h>
> +#include <linux/of_address.h>
> +#include <linux/irqchip/arm-gic-v3.h>
> +#include <linux/irq.h>
> +#include <linux/irqdomain.h>
> +#include <linux/msi.h>
> +#include "../include/mc-sys.h"
> +#include "dprc-cmd.h"
> +
> +static void fsl_mc_msi_set_desc(msi_alloc_info_t *arg,
> +				struct msi_desc *desc)
> +{
> +	arg->desc = desc;
> +	arg->hwirq = (irq_hw_number_t)desc->fsl_mc.msi_index;
> +}
> +
> +static void fsl_mc_msi_update_dom_ops(struct msi_domain_info *info)
> +{
> +	struct msi_domain_ops *ops = info->ops;
> +
> +	if (WARN_ON(!ops))
> +		return;
> +
> +	if (!ops->set_desc)
> +		ops->set_desc = fsl_mc_msi_set_desc;

When would that be overridden by the MSI controller?

> +}
> +
> +static void __fsl_mc_msi_write_msg(struct fsl_mc_device *mc_bus_dev,
> +				   struct fsl_mc_device_irq *mc_dev_irq)
> +{
> +	int error;
> +	struct fsl_mc_device *owner_mc_dev = mc_dev_irq->mc_dev;
> +	struct msi_desc *msi_desc = mc_dev_irq->msi_desc;
> +	struct dprc_irq_cfg irq_cfg;
> +
> +	/*
> +	 * msi_desc->msg.address is 0x0 when this function is invoked in
> +	 * the free_irq() code path. In this case, for the MC, we don't
> +	 * really need to "unprogram" the MSI, so we just return.
> +	 */
> +	if (msi_desc->msg.address_lo == 0x0 && msi_desc->msg.address_hi == 0x0)
> +		return;
> +
> +	if (WARN_ON(!owner_mc_dev))
> +		return;
> +
> +	irq_cfg.paddr = ((u64)msi_desc->msg.address_hi << 32) |
> +			msi_desc->msg.address_lo;

This should really be a phys_addr_t.

> +	irq_cfg.val = msi_desc->msg.data;
> +	irq_cfg.user_irq_id = msi_desc->irq;
> +
> +	/*
> +	 * DPRCs and other objects use different commands to set up IRQs,
> +	 * so we have to differentiate them here.
> +	 */
> +	if (owner_mc_dev == mc_bus_dev) {
> +		/*
> +		 * IRQ is for the mc_bus_dev's DPRC itself
> +		 */
> +		error = dprc_set_irq(mc_bus_dev->mc_io,
> +				     MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
> +				     mc_bus_dev->mc_handle,
> +				     mc_dev_irq->dev_irq_index,
> +				     &irq_cfg);
> +		if (error < 0) {
> +			dev_err(&owner_mc_dev->dev,
> +				"dprc_set_irq() failed: %d\n", error);
> +		}
> +	} else {
> +		error = dprc_set_obj_irq(mc_bus_dev->mc_io,
> +					 MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
> +					 mc_bus_dev->mc_handle,
> +					 owner_mc_dev->obj_desc.type,
> +					 owner_mc_dev->obj_desc.id,
> +					 mc_dev_irq->dev_irq_index,
> +					 &irq_cfg);
> +		if (error < 0) {
> +			dev_err(&owner_mc_dev->dev,
> +				"dprc_obj_set_irq() failed: %d\n", error);
> +		}
> +	}

It feels a bit odd that you have all of this under a single MSI
umbrella, and yet have to differentiate between them. Do they have a
different programming interface? Or is that because these dprc_set_*_irq
functions do some other stuff behind the scene (I'm too lazy to check...)?

> +}
> +
> +/*
> + * NOTE: This function is invoked with interrupts disabled
> + */
> +static void fsl_mc_msi_write_msg(struct irq_data *irq_data,
> +				 struct msi_msg *msg)
> +{
> +	struct msi_desc *msi_desc = irq_data_get_msi_desc(irq_data);
> +	struct fsl_mc_device *mc_bus_dev = to_fsl_mc_device(msi_desc->dev);
> +	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
> +	struct fsl_mc_device_irq *mc_dev_irq =
> +		&mc_bus->irq_resources[msi_desc->fsl_mc.msi_index];
> +
> +	WARN_ON(mc_dev_irq->msi_desc != msi_desc);
> +	msi_desc->msg = *msg;

I wonder why we don't do that in core code...

> +
> +	/*
> +	 * Program the MSI (paddr, value) pair in the device:
> +	 */
> +	__fsl_mc_msi_write_msg(mc_bus_dev, mc_dev_irq);
> +}
> +
> +static void fsl_mc_msi_update_chip_ops(struct msi_domain_info *info)
> +{
> +	struct irq_chip *chip = info->chip;
> +
> +	if (WARN_ON((!chip)))
> +		return;
> +
> +	if (!chip->irq_write_msi_msg)
> +		chip->irq_write_msi_msg = fsl_mc_msi_write_msg;

Same as above. Do you foresee any circumstances where you you wouldn't
use this function? My hunch is that you should warn if
chip->irq_write_msi_msg is provided by the caller.

> +}

[...]

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

RE: [PATCH v2 03/11] staging: fsl-mc: Added generic MSI support for FSL-MC devices

[Jose Rivera]

> -----Original Message-----
> From: Marc Zyngier [mailto:marc.zyngier@arm.com]
> Sent: Friday, November 06, 2015 11:51 AM
> To: Rivera Jose-B46482; gregkh@linuxfoundation.org; arnd@arndb.de;
> devel@driverdev.osuosl.org; linux-kernel@vger.kernel.org
> Cc: Yoder Stuart-B08248; katz Itai-RM05202; Pan Lijun-B44306; Li Yang-
> Leo-R58472; Wood Scott-B07421; agraf@suse.de; Hamciuc Bogdan-BHAMCIU1;
> Marginean Alexandru-R89243; Sharma Bhupesh-B45370; Erez Nir-RM30794;
> Schmitt Richard-B43082; dan.carpenter@oracle.com;
> jiang.liu@linux.intel.com
> Subject: Re: [PATCH v2 03/11] staging: fsl-mc: Added generic MSI support
> for FSL-MC devices
> 
> On 30/10/15 19:43, J. German Rivera wrote:
> > Created an MSI domain for the fsl-mc bus-- including functions to
> > create a domain, find a domain, alloc/free domain irqs, and bus
> > specific overrides for domain and irq_chip ops.
> >
> > Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
> > ---
> > Changes in v2: none
> >
> >  drivers/staging/fsl-mc/bus/Kconfig          |   1 +
> >  drivers/staging/fsl-mc/bus/Makefile         |   1 +
> >  drivers/staging/fsl-mc/bus/mc-msi.c         | 276
> ++++++++++++++++++++++++++++
> >  drivers/staging/fsl-mc/include/mc-private.h |  17 ++
> >  drivers/staging/fsl-mc/include/mc.h         |  17 ++
> >  5 files changed, 312 insertions(+)
> >  create mode 100644 drivers/staging/fsl-mc/bus/mc-msi.c
> >
> > diff --git a/drivers/staging/fsl-mc/bus/Kconfig
> > b/drivers/staging/fsl-mc/bus/Kconfig
> > index 0d779d9..c498ac6 100644
> > --- a/drivers/staging/fsl-mc/bus/Kconfig
> > +++ b/drivers/staging/fsl-mc/bus/Kconfig
> > @@ -9,6 +9,7 @@
> >  config FSL_MC_BUS
> >  	tristate "Freescale Management Complex (MC) bus driver"
> >  	depends on OF && ARM64
> > +	select GENERIC_MSI_IRQ_DOMAIN
> >  	help
> >  	  Driver to enable the bus infrastructure for the Freescale
> >            QorIQ Management Complex (fsl-mc). The fsl-mc is a hardware
> > diff --git a/drivers/staging/fsl-mc/bus/Makefile
> > b/drivers/staging/fsl-mc/bus/Makefile
> > index 25433a9..a5f2ba4 100644
> > --- a/drivers/staging/fsl-mc/bus/Makefile
> > +++ b/drivers/staging/fsl-mc/bus/Makefile
> > @@ -13,5 +13,6 @@ mc-bus-driver-objs := mc-bus.o \
> >  		      dpmng.o \
> >  		      dprc-driver.o \
> >  		      mc-allocator.o \
> > +		      mc-msi.o \
> >  		      dpmcp.o \
> >  		      dpbp.o
> > diff --git a/drivers/staging/fsl-mc/bus/mc-msi.c
> > b/drivers/staging/fsl-mc/bus/mc-msi.c
> > new file mode 100644
> > index 0000000..81b53e3
> > --- /dev/null
> > +++ b/drivers/staging/fsl-mc/bus/mc-msi.c
> > @@ -0,0 +1,276 @@
> > +/*
> > + * Freescale Management Complex (MC) bus driver MSI support
> > + *
> > + * Copyright (C) 2015 Freescale Semiconductor, Inc.
> > + * Author: German Rivera <German.Rivera@freescale.com>
> > + *
> > + * This file is licensed under the terms of the GNU General Public
> > + * License version 2. This program is licensed "as is" without any
> > + * warranty of any kind, whether express or implied.
> > + */
> > +
> > +#include "../include/mc-private.h"
> > +#include <linux/of_device.h>
> > +#include <linux/of_address.h>
> > +#include <linux/irqchip/arm-gic-v3.h> #include <linux/irq.h> #include
> > +<linux/irqdomain.h> #include <linux/msi.h> #include
> > +"../include/mc-sys.h"
> > +#include "dprc-cmd.h"
> > +
> > +static void fsl_mc_msi_set_desc(msi_alloc_info_t *arg,
> > +				struct msi_desc *desc)
> > +{
> > +	arg->desc = desc;
> > +	arg->hwirq = (irq_hw_number_t)desc->fsl_mc.msi_index;
> > +}
> > +
> > +static void fsl_mc_msi_update_dom_ops(struct msi_domain_info *info) {
> > +	struct msi_domain_ops *ops = info->ops;
> > +
> > +	if (WARN_ON(!ops))
> > +		return;
> > +
> > +	if (!ops->set_desc)
> > +		ops->set_desc = fsl_mc_msi_set_desc;
> 
> When would that be overridden by the MSI controller?
>
It should not be set by the MSI controller. As you suggested in another, I will add
A WARN_ON if set_desc is not NULL.
 
> > +}
> > +
> > +static void __fsl_mc_msi_write_msg(struct fsl_mc_device *mc_bus_dev,
> > +				   struct fsl_mc_device_irq *mc_dev_irq) {
> > +	int error;
> > +	struct fsl_mc_device *owner_mc_dev = mc_dev_irq->mc_dev;
> > +	struct msi_desc *msi_desc = mc_dev_irq->msi_desc;
> > +	struct dprc_irq_cfg irq_cfg;
> > +
> > +	/*
> > +	 * msi_desc->msg.address is 0x0 when this function is invoked in
> > +	 * the free_irq() code path. In this case, for the MC, we don't
> > +	 * really need to "unprogram" the MSI, so we just return.
> > +	 */
> > +	if (msi_desc->msg.address_lo == 0x0 && msi_desc->msg.address_hi ==
> 0x0)
> > +		return;
> > +
> > +	if (WARN_ON(!owner_mc_dev))
> > +		return;
> > +
> > +	irq_cfg.paddr = ((u64)msi_desc->msg.address_hi << 32) |
> > +			msi_desc->msg.address_lo;
> 
> This should really be a phys_addr_t.
> 
I'll change the type of paddr in the dprc_irq_cfg struct to be phys_addr_t instead of u64.

> > +	irq_cfg.val = msi_desc->msg.data;
> > +	irq_cfg.user_irq_id = msi_desc->irq;
> > +
> > +	/*
> > +	 * DPRCs and other objects use different commands to set up IRQs,
> > +	 * so we have to differentiate them here.
> > +	 */
> > +	if (owner_mc_dev == mc_bus_dev) {
> > +		/*
> > +		 * IRQ is for the mc_bus_dev's DPRC itself
> > +		 */
> > +		error = dprc_set_irq(mc_bus_dev->mc_io,
> > +				     MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
> > +				     mc_bus_dev->mc_handle,
> > +				     mc_dev_irq->dev_irq_index,
> > +				     &irq_cfg);
> > +		if (error < 0) {
> > +			dev_err(&owner_mc_dev->dev,
> > +				"dprc_set_irq() failed: %d\n", error);
> > +		}
> > +	} else {
> > +		error = dprc_set_obj_irq(mc_bus_dev->mc_io,
> > +					 MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
> > +					 mc_bus_dev->mc_handle,
> > +					 owner_mc_dev->obj_desc.type,
> > +					 owner_mc_dev->obj_desc.id,
> > +					 mc_dev_irq->dev_irq_index,
> > +					 &irq_cfg);
> > +		if (error < 0) {
> > +			dev_err(&owner_mc_dev->dev,
> > +				"dprc_obj_set_irq() failed: %d\n", error);
> > +		}
> > +	}
> 
> It feels a bit odd that you have all of this under a single MSI umbrella,
> and yet have to differentiate between them. Do they have a different
> programming interface? Or is that because these dprc_set_*_irq functions
> do some other stuff behind the scene (I'm too lazy to check...)?
> 
Due to MC firmware API limitations, dprc_set_obj_irq can only be used
to set IRQs for the DPRC's children not for the DPRC itself.

> > +}
> > +
> > +/*
> > + * NOTE: This function is invoked with interrupts disabled  */ static
> > +void fsl_mc_msi_write_msg(struct irq_data *irq_data,
> > +				 struct msi_msg *msg)
> > +{
> > +	struct msi_desc *msi_desc = irq_data_get_msi_desc(irq_data);
> > +	struct fsl_mc_device *mc_bus_dev = to_fsl_mc_device(msi_desc->dev);
> > +	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
> > +	struct fsl_mc_device_irq *mc_dev_irq =
> > +		&mc_bus->irq_resources[msi_desc->fsl_mc.msi_index];
> > +
> > +	WARN_ON(mc_dev_irq->msi_desc != msi_desc);
> > +	msi_desc->msg = *msg;
> 
> I wonder why we don't do that in core code...
> 
> > +
> > +	/*
> > +	 * Program the MSI (paddr, value) pair in the device:
> > +	 */
> > +	__fsl_mc_msi_write_msg(mc_bus_dev, mc_dev_irq); }
> > +
> > +static void fsl_mc_msi_update_chip_ops(struct msi_domain_info *info)
> > +{
> > +	struct irq_chip *chip = info->chip;
> > +
> > +	if (WARN_ON((!chip)))
> > +		return;
> > +
> > +	if (!chip->irq_write_msi_msg)
> > +		chip->irq_write_msi_msg = fsl_mc_msi_write_msg;
> 
> Same as above. Do you foresee any circumstances where you you wouldn't
> use this function? My hunch is that you should warn if
> chip->irq_write_msi_msg is provided by the caller.
> 
I agree with you. The caller should not set the irq_write_msi_msg function.
I'll add a WARN_ON to catch that.
> > +}
> 
> [...]
> 
> Thanks,
> 
> 	M.
> --
> Jazz is not dead. It just smells funny...

Re: [PATCH v2 03/11] staging: fsl-mc: Added generic MSI support for FSL-MC devices

[Marc Zyngier]

On 06/11/15 23:20, Jose Rivera wrote:

[...]

>>> +static void __fsl_mc_msi_write_msg(struct fsl_mc_device *mc_bus_dev,
>>> +				   struct fsl_mc_device_irq *mc_dev_irq) {
>>> +	int error;
>>> +	struct fsl_mc_device *owner_mc_dev = mc_dev_irq->mc_dev;
>>> +	struct msi_desc *msi_desc = mc_dev_irq->msi_desc;
>>> +	struct dprc_irq_cfg irq_cfg;

[...]

>>> +	/*
>>> +	 * DPRCs and other objects use different commands to set up IRQs,
>>> +	 * so we have to differentiate them here.
>>> +	 */
>>> +	if (owner_mc_dev == mc_bus_dev) {
>>> +		/*
>>> +		 * IRQ is for the mc_bus_dev's DPRC itself
>>> +		 */
>>> +		error = dprc_set_irq(mc_bus_dev->mc_io,
>>> +				     MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
>>> +				     mc_bus_dev->mc_handle,
>>> +				     mc_dev_irq->dev_irq_index,
>>> +				     &irq_cfg);
>>> +		if (error < 0) {
>>> +			dev_err(&owner_mc_dev->dev,
>>> +				"dprc_set_irq() failed: %d\n", error);
>>> +		}
>>> +	} else {
>>> +		error = dprc_set_obj_irq(mc_bus_dev->mc_io,
>>> +					 MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
>>> +					 mc_bus_dev->mc_handle,
>>> +					 owner_mc_dev->obj_desc.type,
>>> +					 owner_mc_dev->obj_desc.id,
>>> +					 mc_dev_irq->dev_irq_index,
>>> +					 &irq_cfg);
>>> +		if (error < 0) {
>>> +			dev_err(&owner_mc_dev->dev,
>>> +				"dprc_obj_set_irq() failed: %d\n", error);
>>> +		}
>>> +	}
>>
>> It feels a bit odd that you have all of this under a single MSI umbrella,
>> and yet have to differentiate between them. Do they have a different
>> programming interface? Or is that because these dprc_set_*_irq functions
>> do some other stuff behind the scene (I'm too lazy to check...)?
>>
> Due to MC firmware API limitations, dprc_set_obj_irq can only be used
> to set IRQs for the DPRC's children not for the DPRC itself.

Right. So this makes me wonder whether or not you have the right
approach here. The logic behind the bus type was that devices with a
common programming interface would share a bus type (the odd duck being
platform which is used to implement anything else).

Your description seems to suggest that only devices behind the DPRC
share that programming interface, and that the DPRC itself is the local
weirdo.  Should it be using the platform-msi subsystem instead? Or is it
just a matter of firmware oddity?

This is probably not a big deal, but it is worth keeping it in mind,
specially if that has visible consequences (in your device tree, for
example).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

RE: [PATCH v2 03/11] staging: fsl-mc: Added generic MSI support for FSL-MC devices

[Jose Rivera]

> -----Original Message-----
> From: Marc Zyngier [mailto:marc.zyngier@arm.com]
> Sent: Monday, November 09, 2015 4:46 AM
> To: Rivera Jose-B46482; gregkh@linuxfoundation.org; arnd@arndb.de;
> devel@driverdev.osuosl.org; linux-kernel@vger.kernel.org
> Cc: Yoder Stuart-B08248; katz Itai-RM05202; Pan Lijun-B44306; Li Yang-
> Leo-R58472; Wood Scott-B07421; agraf@suse.de; Hamciuc Bogdan-BHAMCIU1;
> Marginean Alexandru-R89243; Sharma Bhupesh-B45370; Erez Nir-RM30794;
> Schmitt Richard-B43082; dan.carpenter@oracle.com;
> jiang.liu@linux.intel.com
> Subject: Re: [PATCH v2 03/11] staging: fsl-mc: Added generic MSI support
> for FSL-MC devices
> 
> On 06/11/15 23:20, Jose Rivera wrote:
> 
> [...]
> 
> >>> +static void __fsl_mc_msi_write_msg(struct fsl_mc_device *mc_bus_dev,
> >>> +				   struct fsl_mc_device_irq *mc_dev_irq) {
> >>> +	int error;
> >>> +	struct fsl_mc_device *owner_mc_dev = mc_dev_irq->mc_dev;
> >>> +	struct msi_desc *msi_desc = mc_dev_irq->msi_desc;
> >>> +	struct dprc_irq_cfg irq_cfg;
> 
> [...]
> 
> >>> +	/*
> >>> +	 * DPRCs and other objects use different commands to set up IRQs,
> >>> +	 * so we have to differentiate them here.
> >>> +	 */
> >>> +	if (owner_mc_dev == mc_bus_dev) {
> >>> +		/*
> >>> +		 * IRQ is for the mc_bus_dev's DPRC itself
> >>> +		 */
> >>> +		error = dprc_set_irq(mc_bus_dev->mc_io,
> >>> +				     MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
> >>> +				     mc_bus_dev->mc_handle,
> >>> +				     mc_dev_irq->dev_irq_index,
> >>> +				     &irq_cfg);
> >>> +		if (error < 0) {
> >>> +			dev_err(&owner_mc_dev->dev,
> >>> +				"dprc_set_irq() failed: %d\n", error);
> >>> +		}
> >>> +	} else {
> >>> +		error = dprc_set_obj_irq(mc_bus_dev->mc_io,
> >>> +					 MC_CMD_FLAG_INTR_DIS | MC_CMD_FLAG_PRI,
> >>> +					 mc_bus_dev->mc_handle,
> >>> +					 owner_mc_dev->obj_desc.type,
> >>> +					 owner_mc_dev->obj_desc.id,
> >>> +					 mc_dev_irq->dev_irq_index,
> >>> +					 &irq_cfg);
> >>> +		if (error < 0) {
> >>> +			dev_err(&owner_mc_dev->dev,
> >>> +				"dprc_obj_set_irq() failed: %d\n", error);
> >>> +		}
> >>> +	}
> >>
> >> It feels a bit odd that you have all of this under a single MSI
> >> umbrella, and yet have to differentiate between them. Do they have a
> >> different programming interface? Or is that because these
> >> dprc_set_*_irq functions do some other stuff behind the scene (I'm too
> lazy to check...)?
> >>
> > Due to MC firmware API limitations, dprc_set_obj_irq can only be used
> > to set IRQs for the DPRC's children not for the DPRC itself.
> 
> Right. So this makes me wonder whether or not you have the right approach
> here. The logic behind the bus type was that devices with a common
> programming interface would share a bus type (the odd duck being platform
> which is used to implement anything else).
> 
> Your description seems to suggest that only devices behind the DPRC share
> that programming interface, and that the DPRC itself is the local weirdo.
> Should it be using the platform-msi subsystem instead? Or is it just a
> matter of firmware oddity?
> 
It's really a MC API oddity--  both the DPRC and the devices behind it share
the same MSI context (same ITT in the ITS) but they just require different
APIs to set the MSI addr/data.

> This is probably not a big deal, but it is worth keeping it in mind,
> specially if that has visible consequences (in your device tree, for
> example).
> 
> Thanks,
> 
> 	M.
> --
> Jazz is not dead. It just smells funny...

[PATCH v2 04/11] staging: fsl-mc: Added GICv3-ITS support for FSL-MC MSIs

[J. German Rivera]

Added platform-specific MSI support layer for FSL-MC devices.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 drivers/staging/fsl-mc/bus/Makefile                |   1 +
 .../staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c | 127 +++++++++++++++++++++
 drivers/staging/fsl-mc/include/mc-private.h        |   4 +
 3 files changed, 132 insertions(+)
 create mode 100644 drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c

diff --git a/drivers/staging/fsl-mc/bus/Makefile b/drivers/staging/fsl-mc/bus/Makefile
index a5f2ba4..e731517 100644
--- a/drivers/staging/fsl-mc/bus/Makefile
+++ b/drivers/staging/fsl-mc/bus/Makefile
@@ -14,5 +14,6 @@ mc-bus-driver-objs := mc-bus.o \
 		      dprc-driver.o \
 		      mc-allocator.o \
 		      mc-msi.o \
+		      irq-gic-v3-its-fsl-mc-msi.o \
 		      dpmcp.o \
 		      dpbp.o
diff --git a/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c b/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c
new file mode 100644
index 0000000..5319afa
--- /dev/null
+++ b/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c
@@ -0,0 +1,127 @@
+/*
+ * Freescale Management Complex (MC) bus driver MSI support
+ *
+ * Copyright (C) 2015 Freescale Semiconductor, Inc.
+ * Author: German Rivera <German.Rivera@freescale.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include "../include/mc-private.h"
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/irqchip/arm-gic-v3.h>
+#include <linux/irq.h>
+#include <linux/msi.h>
+#include <linux/of.h>
+#include <linux/of_irq.h>
+#include "../include/mc-sys.h"
+#include "dprc-cmd.h"
+
+static struct irq_chip its_msi_irq_chip = {
+	.name = "fsl-mc-bus-msi",
+	.irq_mask = irq_chip_mask_parent,
+	.irq_unmask = irq_chip_unmask_parent,
+	.irq_eoi = irq_chip_eoi_parent,
+	.irq_set_affinity = msi_domain_set_affinity
+};
+
+static int its_fsl_mc_msi_prepare(struct irq_domain *msi_domain,
+				  struct device *dev,
+				  int nvec, msi_alloc_info_t *info)
+{
+	u32 its_dev_id;
+	struct fsl_mc_device *mc_bus_dev = to_fsl_mc_device(dev);
+	struct msi_domain_info *msi_info;
+
+	if (WARN_ON(dev->bus != &fsl_mc_bus_type))
+		return -EINVAL;
+
+	if (WARN_ON(!(mc_bus_dev->flags & FSL_MC_IS_DPRC)))
+		return -EINVAL;
+
+	/*
+	 * Set the device Id to be passed to the GIC-ITS:
+	 *
+	 * NOTE: This device id corresponds to the IOMMU stream ID
+	 * associated with the DPRC object (ICID).
+	 */
+	its_dev_id = mc_bus_dev->icid;
+	info->scratchpad[0].ul = its_dev_id;
+	msi_info = msi_get_domain_info(msi_domain->parent);
+	return msi_info->ops->msi_prepare(msi_domain->parent, dev, nvec, info);
+}
+
+static struct msi_domain_ops its_fsl_mc_msi_ops = {
+	.msi_prepare = its_fsl_mc_msi_prepare,
+};
+
+static struct msi_domain_info its_fsl_mc_msi_domain_info = {
+	.flags	= (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
+	.ops	= &its_fsl_mc_msi_ops,
+	.chip	= &its_msi_irq_chip,
+};
+
+static const struct of_device_id its_device_id[] = {
+	{	.compatible	= "arm,gic-v3-its",	},
+	{},
+};
+
+int __init its_fsl_mc_msi_init(void)
+{
+	struct device_node *np;
+	struct irq_domain *parent;
+	struct irq_domain *mc_msi_domain;
+
+	for (np = of_find_matching_node(NULL, its_device_id); np;
+	     np = of_find_matching_node(np, its_device_id)) {
+		if (!of_property_read_bool(np, "msi-controller"))
+			continue;
+
+		parent = irq_find_matching_host(np, DOMAIN_BUS_NEXUS);
+		if (!parent || !msi_get_domain_info(parent)) {
+			pr_err("%s: unable to locate ITS domain\n",
+			       np->full_name);
+			continue;
+		}
+
+		mc_msi_domain =
+		    fsl_mc_msi_create_irq_domain(of_node_to_fwnode(np),
+						 &its_fsl_mc_msi_domain_info,
+						 parent);
+		if (!mc_msi_domain) {
+			pr_err("%s: unable to create fsl-mc domain\n",
+			       np->full_name);
+			continue;
+		}
+
+		WARN_ON(mc_msi_domain->host_data !=
+				&its_fsl_mc_msi_domain_info);
+
+		pr_info("fsl-mc MSI: %s domain created\n", np->full_name);
+	}
+
+	return 0;
+}
+
+void its_fsl_mc_msi_cleanup(void)
+{
+	struct device_node *np;
+
+	for (np = of_find_matching_node(NULL, its_device_id); np;
+	     np = of_find_matching_node(np, its_device_id)) {
+		struct irq_domain *mc_msi_domain =
+			irq_find_matching_host(np, DOMAIN_BUS_FSL_MC_MSI);
+
+		if (!of_property_read_bool(np, "msi-controller"))
+			continue;
+
+		mc_msi_domain =
+			irq_find_matching_host(np, DOMAIN_BUS_FSL_MC_MSI);
+		if (mc_msi_domain &&
+		    mc_msi_domain->host_data == &its_fsl_mc_msi_domain_info)
+			irq_domain_remove(mc_msi_domain);
+	}
+}
diff --git a/drivers/staging/fsl-mc/include/mc-private.h b/drivers/staging/fsl-mc/include/mc-private.h
index 3dabd003..f2504bc 100644
--- a/drivers/staging/fsl-mc/include/mc-private.h
+++ b/drivers/staging/fsl-mc/include/mc-private.h
@@ -133,4 +133,8 @@ int fsl_mc_msi_domain_alloc_irqs(struct device *dev,

 void fsl_mc_msi_domain_free_irqs(struct device *dev);

+int __init its_fsl_mc_msi_init(void);
+
+void its_fsl_mc_msi_cleanup(void);
+
 #endif /* _FSL_MC_PRIVATE_H_ */
--
2.3.3

[PATCH v2 05/11] staging: fsl-mc: Extended MC bus allocator to include IRQs

[J. German Rivera]

All the IRQs for DPAA2 objects in the same DPRC must use
the ICID of that DPRC, as their device Id in the GIC-ITS.
Thus, all these IRQs must share the same ITT table in the GIC.
As a result, a pool of IRQs with the same device Id must be
preallocated per DPRC (fsl-mc bus instance). So, the fsl-mc
bus object allocator is extended to also provide services
to allocate IRQs to DPAA2 devices, from their parent fsl-mc bus
IRQ pool.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 drivers/staging/fsl-mc/bus/mc-allocator.c   | 199 ++++++++++++++++++++++++++++
 drivers/staging/fsl-mc/include/mc-private.h |  15 +++
 drivers/staging/fsl-mc/include/mc.h         |   9 ++
 3 files changed, 223 insertions(+)

diff --git a/drivers/staging/fsl-mc/bus/mc-allocator.c b/drivers/staging/fsl-mc/bus/mc-allocator.c
index 88d1857..c5fa628 100644
--- a/drivers/staging/fsl-mc/bus/mc-allocator.c
+++ b/drivers/staging/fsl-mc/bus/mc-allocator.c
@@ -15,6 +15,7 @@
 #include "../include/dpcon-cmd.h"
 #include "dpmcp-cmd.h"
 #include "dpmcp.h"
+#include <linux/msi.h>

 /**
  * fsl_mc_resource_pool_add_device - add allocatable device to a resource
@@ -160,6 +161,7 @@ static const char *const fsl_mc_pool_type_strings[] = {
 	[FSL_MC_POOL_DPMCP] = "dpmcp",
 	[FSL_MC_POOL_DPBP] = "dpbp",
 	[FSL_MC_POOL_DPCON] = "dpcon",
+	[FSL_MC_POOL_IRQ] = "irq",
 };

 static int __must_check object_type_to_pool_type(const char *object_type,
@@ -465,6 +467,203 @@ void fsl_mc_object_free(struct fsl_mc_device *mc_adev)
 }
 EXPORT_SYMBOL_GPL(fsl_mc_object_free);

+/*
+ * Initialize the interrupt pool associated with a MC bus.
+ * It allocates a block of IRQs from the GIC-ITS
+ */
+int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
+			     unsigned int irq_count)
+{
+	unsigned int i;
+	struct msi_desc *msi_desc;
+	struct fsl_mc_device_irq *irq_resources;
+	struct fsl_mc_device_irq *mc_dev_irq;
+	int error;
+	struct fsl_mc_device *mc_bus_dev = &mc_bus->mc_dev;
+	struct fsl_mc_resource_pool *res_pool =
+			&mc_bus->resource_pools[FSL_MC_POOL_IRQ];
+
+	if (WARN_ON(irq_count == 0 ||
+		    irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS))
+		return -EINVAL;
+
+	error = fsl_mc_msi_domain_alloc_irqs(&mc_bus_dev->dev, irq_count);
+	if (error < 0)
+		return error;
+
+	irq_resources = devm_kzalloc(&mc_bus_dev->dev,
+				     sizeof(*irq_resources) * irq_count,
+				     GFP_KERNEL);
+	if (!irq_resources) {
+		error = -ENOMEM;
+		goto cleanup_msi_irqs;
+	}
+
+	for (i = 0; i < irq_count; i++) {
+		mc_dev_irq = &irq_resources[i];
+
+		/*
+		 * NOTE: This mc_dev_irq's MSI addr/value pair will be set
+		 * by the fsl_mc_msi_write_msg() callback
+		 */
+		mc_dev_irq->resource.type = res_pool->type;
+		mc_dev_irq->resource.data = mc_dev_irq;
+		mc_dev_irq->resource.parent_pool = res_pool;
+		INIT_LIST_HEAD(&mc_dev_irq->resource.node);
+		list_add_tail(&mc_dev_irq->resource.node, &res_pool->free_list);
+	}
+
+	for_each_msi_entry(msi_desc, &mc_bus_dev->dev) {
+		mc_dev_irq = &irq_resources[msi_desc->fsl_mc.msi_index];
+		mc_dev_irq->msi_desc = msi_desc;
+		mc_dev_irq->resource.id = msi_desc->irq;
+	}
+
+	res_pool->max_count = irq_count;
+	res_pool->free_count = irq_count;
+	mc_bus->irq_resources = irq_resources;
+	return 0;
+
+cleanup_msi_irqs:
+	fsl_mc_msi_domain_free_irqs(&mc_bus_dev->dev);
+	return error;
+}
+EXPORT_SYMBOL_GPL(fsl_mc_populate_irq_pool);
+
+/**
+ * Teardown the interrupt pool associated with an MC bus.
+ * It frees the IRQs that were allocated to the pool, back to the GIC-ITS.
+ */
+void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus)
+{
+	struct fsl_mc_device *mc_bus_dev = &mc_bus->mc_dev;
+	struct fsl_mc_resource_pool *res_pool =
+			&mc_bus->resource_pools[FSL_MC_POOL_IRQ];
+
+	if (WARN_ON(!mc_bus->irq_resources))
+		return;
+
+	if (WARN_ON(res_pool->max_count == 0))
+		return;
+
+	if (WARN_ON(res_pool->free_count != res_pool->max_count))
+		return;
+
+	INIT_LIST_HEAD(&res_pool->free_list);
+	res_pool->max_count = 0;
+	res_pool->free_count = 0;
+	mc_bus->irq_resources = NULL;
+	fsl_mc_msi_domain_free_irqs(&mc_bus_dev->dev);
+}
+EXPORT_SYMBOL_GPL(fsl_mc_cleanup_irq_pool);
+
+/**
+ * It allocates the IRQs required by a given MC object device. The
+ * IRQs are allocated from the interrupt pool associated with the
+ * MC bus that contains the device, if the device is not a DPRC device.
+ * Otherwise, the IRQs are allocated from the interrupt pool associated
+ * with the MC bus that represents the DPRC device itself.
+ */
+int __must_check fsl_mc_allocate_irqs(struct fsl_mc_device *mc_dev)
+{
+	int i;
+	int irq_count;
+	int res_allocated_count = 0;
+	int error = -EINVAL;
+	struct fsl_mc_device_irq **irqs = NULL;
+	struct fsl_mc_bus *mc_bus;
+	struct fsl_mc_resource_pool *res_pool;
+
+	if (WARN_ON(mc_dev->irqs))
+		return -EINVAL;
+
+	irq_count = mc_dev->obj_desc.irq_count;
+	if (WARN_ON(irq_count == 0))
+		return -EINVAL;
+
+	if (strcmp(mc_dev->obj_desc.type, "dprc") == 0)
+		mc_bus = to_fsl_mc_bus(mc_dev);
+	else
+		mc_bus = to_fsl_mc_bus(to_fsl_mc_device(mc_dev->dev.parent));
+
+	if (WARN_ON(!mc_bus->irq_resources))
+		return -EINVAL;
+
+	res_pool = &mc_bus->resource_pools[FSL_MC_POOL_IRQ];
+	if (res_pool->free_count < irq_count) {
+		dev_err(&mc_dev->dev,
+			"Not able to allocate %u irqs for device\n", irq_count);
+		return -ENOSPC;
+	}
+
+	irqs = devm_kzalloc(&mc_dev->dev, irq_count * sizeof(irqs[0]),
+			    GFP_KERNEL);
+	if (!irqs)
+		return -ENOMEM;
+
+	for (i = 0; i < irq_count; i++) {
+		struct fsl_mc_resource *resource;
+
+		error = fsl_mc_resource_allocate(mc_bus, FSL_MC_POOL_IRQ,
+						 &resource);
+		if (error < 0)
+			goto error_resource_alloc;
+
+		irqs[i] = to_fsl_mc_irq(resource);
+		res_allocated_count++;
+
+		WARN_ON(irqs[i]->mc_dev);
+		irqs[i]->mc_dev = mc_dev;
+		irqs[i]->dev_irq_index = i;
+	}
+
+	mc_dev->irqs = irqs;
+	return 0;
+
+error_resource_alloc:
+	for (i = 0; i < res_allocated_count; i++) {
+		irqs[i]->mc_dev = NULL;
+		fsl_mc_resource_free(&irqs[i]->resource);
+	}
+
+	return error;
+}
+EXPORT_SYMBOL_GPL(fsl_mc_allocate_irqs);
+
+/*
+ * It frees the IRQs that were allocated for a MC object device, by
+ * returning them to the corresponding interrupt pool.
+ */
+void fsl_mc_free_irqs(struct fsl_mc_device *mc_dev)
+{
+	int i;
+	int irq_count;
+	struct fsl_mc_bus *mc_bus;
+	struct fsl_mc_device_irq **irqs = mc_dev->irqs;
+
+	if (WARN_ON(!irqs))
+		return;
+
+	irq_count = mc_dev->obj_desc.irq_count;
+
+	if (strcmp(mc_dev->obj_desc.type, "dprc") == 0)
+		mc_bus = to_fsl_mc_bus(mc_dev);
+	else
+		mc_bus = to_fsl_mc_bus(to_fsl_mc_device(mc_dev->dev.parent));
+
+	if (WARN_ON(!mc_bus->irq_resources))
+		return;
+
+	for (i = 0; i < irq_count; i++) {
+		WARN_ON(!irqs[i]->mc_dev);
+		irqs[i]->mc_dev = NULL;
+		fsl_mc_resource_free(&irqs[i]->resource);
+	}
+
+	mc_dev->irqs = NULL;
+}
+EXPORT_SYMBOL_GPL(fsl_mc_free_irqs);
+
 /**
  * fsl_mc_allocator_probe - callback invoked when an allocatable device is
  * being added to the system
diff --git a/drivers/staging/fsl-mc/include/mc-private.h b/drivers/staging/fsl-mc/include/mc-private.h
index f2504bc..97295f0 100644
--- a/drivers/staging/fsl-mc/include/mc-private.h
+++ b/drivers/staging/fsl-mc/include/mc-private.h
@@ -30,6 +30,16 @@ struct irq_domain;
 struct msi_domain_info;

 /**
+ * Maximum number of total IRQs that can be pre-allocated for an MC bus'
+ * IRQ pool
+ */
+#define FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS	256
+
+struct device_node;
+struct irq_domain;
+struct msi_domain_info;
+
+/**
  * struct fsl_mc - Private data of a "fsl,qoriq-mc" platform device
  * @root_mc_bus_dev: MC object device representing the root DPRC
  * @num_translation_ranges: number of entries in addr_translation_ranges
@@ -137,4 +147,9 @@ int __init its_fsl_mc_msi_init(void);

 void its_fsl_mc_msi_cleanup(void);

+int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
+			     unsigned int irq_count);
+
+void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus);
+
 #endif /* _FSL_MC_PRIVATE_H_ */
diff --git a/drivers/staging/fsl-mc/include/mc.h b/drivers/staging/fsl-mc/include/mc.h
index 1c1d6ae..ac7c1ce 100644
--- a/drivers/staging/fsl-mc/include/mc.h
+++ b/drivers/staging/fsl-mc/include/mc.h
@@ -14,12 +14,14 @@
 #include <linux/device.h>
 #include <linux/mod_devicetable.h>
 #include <linux/list.h>
+#include <linux/interrupt.h>
 #include "../include/dprc.h"

 #define FSL_MC_VENDOR_FREESCALE	0x1957

 struct fsl_mc_device;
 struct fsl_mc_io;
+struct fsl_mc_bus;

 /**
  * struct fsl_mc_driver - MC object device driver object
@@ -75,6 +77,7 @@ enum fsl_mc_pool_type {
 	FSL_MC_POOL_DPMCP = 0x0,    /* corresponds to "dpmcp" in the MC */
 	FSL_MC_POOL_DPBP,	    /* corresponds to "dpbp" in the MC */
 	FSL_MC_POOL_DPCON,	    /* corresponds to "dpcon" in the MC */
+	FSL_MC_POOL_IRQ,

 	/*
 	 * NOTE: New resource pool types must be added before this entry
@@ -141,6 +144,7 @@ struct fsl_mc_device_irq {
  * NULL if none.
  * @obj_desc: MC description of the DPAA device
  * @regions: pointer to array of MMIO region entries
+ * @irqs: pointer to array of pointers to interrupts allocated to this device
  * @resource: generic resource associated with this MC object device, if any.
  *
  * Generic device object for MC object devices that are "attached" to a
@@ -172,6 +176,7 @@ struct fsl_mc_device {
 	struct fsl_mc_io *mc_io;
 	struct dprc_obj_desc obj_desc;
 	struct resource *regions;
+	struct fsl_mc_device_irq **irqs;
 	struct fsl_mc_resource *resource;
 };

@@ -215,6 +220,10 @@ int __must_check fsl_mc_object_allocate(struct fsl_mc_device *mc_dev,

 void fsl_mc_object_free(struct fsl_mc_device *mc_adev);

+int __must_check fsl_mc_allocate_irqs(struct fsl_mc_device *mc_dev);
+
+void fsl_mc_free_irqs(struct fsl_mc_device *mc_dev);
+
 extern struct bus_type fsl_mc_bus_type;

 #endif /* _FSL_MC_H_ */
--
2.3.3

[PATCH v2 06/11] staging: fsl-mc: Changed DPRC built-in portal's mc_io to be atomic

[J. German Rivera]

The DPRC built-in portal's mc_io is used to send commands to the MC
to program MSIs for MC objects. This is done by the
fsl_mc_msi_write_msg() callback, which is invoked by the generic MSI
layer with interrupts disabled. As a result, the mc_io used in
fsl_mc_msi_write_msg needs to be an atomic mc_io.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 drivers/staging/fsl-mc/bus/dprc-driver.c | 4 +++-
 drivers/staging/fsl-mc/bus/mc-bus.c      | 3 ++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index a9ead0d..7bb30dd 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -396,7 +396,9 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 		error = fsl_create_mc_io(&mc_dev->dev,
 					 mc_dev->regions[0].start,
 					 region_size,
-					 NULL, 0, &mc_dev->mc_io);
+					 NULL,
+					 FSL_MC_IO_ATOMIC_CONTEXT_PORTAL,
+					 &mc_dev->mc_io);
 		if (error < 0)
 			return error;
 	}
diff --git a/drivers/staging/fsl-mc/bus/mc-bus.c b/drivers/staging/fsl-mc/bus/mc-bus.c
index 84db55b..d34f1af 100644
--- a/drivers/staging/fsl-mc/bus/mc-bus.c
+++ b/drivers/staging/fsl-mc/bus/mc-bus.c
@@ -702,7 +702,8 @@ static int fsl_mc_bus_probe(struct platform_device *pdev)
 	mc_portal_phys_addr = res.start;
 	mc_portal_size = resource_size(&res);
 	error = fsl_create_mc_io(&pdev->dev, mc_portal_phys_addr,
-				 mc_portal_size, NULL, 0, &mc_io);
+				 mc_portal_size, NULL,
+				 FSL_MC_IO_ATOMIC_CONTEXT_PORTAL, &mc_io);
 	if (error < 0)
 		return error;

--
2.3.3

[PATCH v2 07/11] staging: fsl-mc: Populate the IRQ pool for an MC bus instance

[J. German Rivera]

Scan the corresponding DPRC container to get total count
of IRQs needed by all its child DPAA2 objects. Then,
preallocate a set of MSI IRQs with the DPRC's ICID
(GIT-ITS device Id) to populate the the DPRC's IRQ pool.
Each child DPAA2 object in the DPRC and the DPRC object itself
will allocate their necessary MSI IRQs from the DPRC's IRQ pool,
in their driver probe function.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 drivers/staging/fsl-mc/bus/dprc-driver.c    | 24 ++++++++++++++++++++++--
 drivers/staging/fsl-mc/include/mc-private.h |  3 ++-
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index 7bb30dd..3bcd161 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -241,6 +241,7 @@ static void dprc_cleanup_all_resource_pools(struct fsl_mc_device *mc_bus_dev)
  * dprc_scan_objects - Discover objects in a DPRC
  *
  * @mc_bus_dev: pointer to the fsl-mc device that represents a DPRC object
+ * @total_irq_count: total number of IRQs needed by objects in the DPRC.
  *
  * Detects objects added and removed from a DPRC and synchronizes the
  * state of the Linux bus driver, MC by adding and removing
@@ -254,11 +255,13 @@ static void dprc_cleanup_all_resource_pools(struct fsl_mc_device *mc_bus_dev)
  * populated before they can get allocation requests from probe callbacks
  * of the device drivers for the non-allocatable devices.
  */
-int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
+int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
+		      unsigned int *total_irq_count)
 {
 	int num_child_objects;
 	int dprc_get_obj_failures;
 	int error;
+	unsigned int irq_count = mc_bus_dev->obj_desc.irq_count;
 	struct dprc_obj_desc *child_obj_desc_array = NULL;

 	error = dprc_get_obj_count(mc_bus_dev->mc_io,
@@ -307,6 +310,7 @@ int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
 				continue;
 			}

+			irq_count += obj_desc->irq_count;
 			dev_dbg(&mc_bus_dev->dev,
 				"Discovered object: type %s, id %d\n",
 				obj_desc->type, obj_desc->id);
@@ -319,6 +323,7 @@ int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
 		}
 	}

+	*total_irq_count = irq_count;
 	dprc_remove_devices(mc_bus_dev, child_obj_desc_array,
 			    num_child_objects);

@@ -344,6 +349,7 @@ EXPORT_SYMBOL_GPL(dprc_scan_objects);
 int dprc_scan_container(struct fsl_mc_device *mc_bus_dev)
 {
 	int error;
+	unsigned int irq_count;
 	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);

 	dprc_init_all_resource_pools(mc_bus_dev);
@@ -352,11 +358,25 @@ int dprc_scan_container(struct fsl_mc_device *mc_bus_dev)
 	 * Discover objects in the DPRC:
 	 */
 	mutex_lock(&mc_bus->scan_mutex);
-	error = dprc_scan_objects(mc_bus_dev);
+	error = dprc_scan_objects(mc_bus_dev, &irq_count);
 	mutex_unlock(&mc_bus->scan_mutex);
 	if (error < 0)
 		goto error;

+	if (dev_get_msi_domain(&mc_bus_dev->dev) && !mc_bus->irq_resources) {
+		if (irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS) {
+			dev_warn(&mc_bus_dev->dev,
+				 "IRQs needed (%u) exceed IRQs preallocated (%u)\n",
+				 irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+		}
+
+		error = fsl_mc_populate_irq_pool(
+				mc_bus,
+				FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+		if (error < 0)
+			goto error;
+	}
+
 	return 0;
 error:
 	dprc_cleanup_all_resource_pools(mc_bus_dev);
diff --git a/drivers/staging/fsl-mc/include/mc-private.h b/drivers/staging/fsl-mc/include/mc-private.h
index 97295f0..27834ea 100644
--- a/drivers/staging/fsl-mc/include/mc-private.h
+++ b/drivers/staging/fsl-mc/include/mc-private.h
@@ -114,7 +114,8 @@ void fsl_mc_device_remove(struct fsl_mc_device *mc_dev);

 int dprc_scan_container(struct fsl_mc_device *mc_bus_dev);

-int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev);
+int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
+		      unsigned int *total_irq_count);

 int __init dprc_driver_init(void);

--
2.3.3

[PATCH v2 08/11] staging: fsl-mc: set MSI domain for DPRC objects

[J. German Rivera]

THE MSI domain associated with a root DPRC object is
obtained form the device tree. Child DPRCs inherit
the parent DPRC MSI domain.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 drivers/staging/fsl-mc/bus/dprc-driver.c | 39 ++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index 3bcd161..455379d 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -13,6 +13,7 @@
 #include "../include/mc-sys.h"
 #include <linux/module.h>
 #include <linux/slab.h>
+#include <linux/interrupt.h>
 #include "dprc-cmd.h"

 struct dprc_child_objs {
@@ -398,11 +399,16 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 {
 	int error;
 	size_t region_size;
+	struct device *parent_dev = mc_dev->dev.parent;
 	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+	bool msi_domain_set = false;

 	if (WARN_ON(strcmp(mc_dev->obj_desc.type, "dprc") != 0))
 		return -EINVAL;

+	if (WARN_ON(dev_get_msi_domain(&mc_dev->dev)))
+		return -EINVAL;
+
 	if (!mc_dev->mc_io) {
 		/*
 		 * This is a child DPRC:
@@ -421,6 +427,30 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 					 &mc_dev->mc_io);
 		if (error < 0)
 			return error;
+		/*
+		 * Inherit parent MSI domain:
+		 */
+		dev_set_msi_domain(&mc_dev->dev,
+				   dev_get_msi_domain(parent_dev));
+		msi_domain_set = true;
+	} else {
+		/*
+		 * This is a root DPRC
+		 */
+		struct irq_domain *mc_msi_domain;
+
+		if (WARN_ON(parent_dev->bus == &fsl_mc_bus_type))
+			return -EINVAL;
+
+		error = fsl_mc_find_msi_domain(parent_dev->of_node,
+					       &mc_msi_domain);
+		if (error < 0) {
+			dev_warn(&mc_dev->dev,
+				 "WARNING: MC bus without interrupt support\n");
+		} else {
+			dev_set_msi_domain(&mc_dev->dev, mc_msi_domain);
+			msi_domain_set = true;
+		}
 	}

 	error = dprc_open(mc_dev->mc_io, 0, mc_dev->obj_desc.id,
@@ -446,6 +476,9 @@ error_cleanup_open:
 	(void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);

 error_cleanup_mc_io:
+	if (msi_domain_set)
+		dev_set_msi_domain(&mc_dev->dev, NULL);
+
 	fsl_destroy_mc_io(mc_dev->mc_io);
 	return error;
 }
@@ -463,6 +496,7 @@ error_cleanup_mc_io:
 static int dprc_remove(struct fsl_mc_device *mc_dev)
 {
 	int error;
+	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);

 	if (WARN_ON(strcmp(mc_dev->obj_desc.type, "dprc") != 0))
 		return -EINVAL;
@@ -475,6 +509,11 @@ static int dprc_remove(struct fsl_mc_device *mc_dev)
 	if (error < 0)
 		dev_err(&mc_dev->dev, "dprc_close() failed: %d\n", error);

+	if (dev_get_msi_domain(&mc_dev->dev)) {
+		fsl_mc_cleanup_irq_pool(mc_bus);
+		dev_set_msi_domain(&mc_dev->dev, NULL);
+	}
+
 	dev_info(&mc_dev->dev, "DPRC device unbound from driver");
 	return 0;
 }
--
2.3.3

[PATCH v2 09/11] staging: fsl-mc: Fixed bug in dprc_probe() error path

[J. German Rivera]

Destroy mc_io in error path in dprc_probe() only if the mc_io was
created in this function.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2:
- Addressed comment from Dan Carpenter
  * Renamed goto error labels to indicate what the goto does

 drivers/staging/fsl-mc/bus/dprc-driver.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index 455379d..7f1ceb5 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -401,6 +401,7 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 	size_t region_size;
 	struct device *parent_dev = mc_dev->dev.parent;
 	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+	bool mc_io_created = false;
 	bool msi_domain_set = false;

 	if (WARN_ON(strcmp(mc_dev->obj_desc.type, "dprc") != 0))
@@ -413,6 +414,9 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 		/*
 		 * This is a child DPRC:
 		 */
+		if (WARN_ON(parent_dev->bus != &fsl_mc_bus_type))
+			return -EINVAL;
+
 		if (WARN_ON(mc_dev->obj_desc.region_count == 0))
 			return -EINVAL;

@@ -427,6 +431,9 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 					 &mc_dev->mc_io);
 		if (error < 0)
 			return error;
+
+		mc_io_created = true;
+
 		/*
 		 * Inherit parent MSI domain:
 		 */
@@ -457,7 +464,7 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 			  &mc_dev->mc_handle);
 	if (error < 0) {
 		dev_err(&mc_dev->dev, "dprc_open() failed: %d\n", error);
-		goto error_cleanup_mc_io;
+		goto error_cleanup_msi_domain;
 	}

 	mutex_init(&mc_bus->scan_mutex);
@@ -475,11 +482,15 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 error_cleanup_open:
 	(void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);

-error_cleanup_mc_io:
+error_cleanup_msi_domain:
 	if (msi_domain_set)
 		dev_set_msi_domain(&mc_dev->dev, NULL);

-	fsl_destroy_mc_io(mc_dev->mc_io);
+	if (mc_io_created) {
+		fsl_destroy_mc_io(mc_dev->mc_io);
+		mc_dev->mc_io = NULL;
+	}
+
 	return error;
 }

--
2.3.3

[PATCH v2 10/11] staging: fsl-mc: Added DPRC interrupt handler

[J. German Rivera]

The interrupt handler for DPRC IRQs is added. DPRC IRQs are
generated for hot plug events related to DPAA2 objects in a given
DPRC. These events include, creating/destroying DPAA2 objects in
the DPRC, changing the "plugged" state of DPAA2 objects and moving
objects between DPRCs.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 drivers/staging/fsl-mc/bus/dprc-driver.c | 247 +++++++++++++++++++++++++++++++
 1 file changed, 247 insertions(+)

diff --git a/drivers/staging/fsl-mc/bus/dprc-driver.c b/drivers/staging/fsl-mc/bus/dprc-driver.c
index 7f1ceb5..0bd2e63 100644
--- a/drivers/staging/fsl-mc/bus/dprc-driver.c
+++ b/drivers/staging/fsl-mc/bus/dprc-driver.c
@@ -14,6 +14,7 @@
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/interrupt.h>
+#include <linux/msi.h>
 #include "dprc-cmd.h"

 struct dprc_child_objs {
@@ -386,6 +387,230 @@ error:
 EXPORT_SYMBOL_GPL(dprc_scan_container);

 /**
+ * dprc_irq0_handler - Regular ISR for DPRC interrupt 0
+ *
+ * @irq: IRQ number of the interrupt being handled
+ * @arg: Pointer to device structure
+ */
+static irqreturn_t dprc_irq0_handler(int irq_num, void *arg)
+{
+	return IRQ_WAKE_THREAD;
+}
+
+/**
+ * dprc_irq0_handler_thread - Handler thread function for DPRC interrupt 0
+ *
+ * @irq: IRQ number of the interrupt being handled
+ * @arg: Pointer to device structure
+ */
+static irqreturn_t dprc_irq0_handler_thread(int irq_num, void *arg)
+{
+	int error;
+	u32 status;
+	struct device *dev = (struct device *)arg;
+	struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+	struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+	struct fsl_mc_io *mc_io = mc_dev->mc_io;
+	struct msi_desc *msi_desc = mc_dev->irqs[0]->msi_desc;
+
+	dev_dbg(dev, "DPRC IRQ %d triggered on CPU %u\n",
+		irq_num, smp_processor_id());
+
+	if (WARN_ON(!(mc_dev->flags & FSL_MC_IS_DPRC)))
+		return IRQ_HANDLED;
+
+	mutex_lock(&mc_bus->scan_mutex);
+	if (WARN_ON(!msi_desc || msi_desc->irq != (u32)irq_num))
+		goto out;
+
+	error = dprc_get_irq_status(mc_io, 0, mc_dev->mc_handle, 0,
+				    &status);
+	if (error < 0) {
+		dev_err(dev,
+			"dprc_get_irq_status() failed: %d\n", error);
+		goto out;
+	}
+
+	error = dprc_clear_irq_status(mc_io, 0, mc_dev->mc_handle, 0,
+				      status);
+	if (error < 0) {
+		dev_err(dev,
+			"dprc_clear_irq_status() failed: %d\n", error);
+		goto out;
+	}
+
+	if (status & (DPRC_IRQ_EVENT_OBJ_ADDED |
+		      DPRC_IRQ_EVENT_OBJ_REMOVED |
+		      DPRC_IRQ_EVENT_CONTAINER_DESTROYED |
+		      DPRC_IRQ_EVENT_OBJ_DESTROYED |
+		      DPRC_IRQ_EVENT_OBJ_CREATED)) {
+		unsigned int irq_count;
+
+		error = dprc_scan_objects(mc_dev, &irq_count);
+		if (error < 0) {
+			/*
+			 * If the error is -ENXIO, we ignore it, as it indicates
+			 * that the object scan was aborted, as we detected that
+			 * an object was removed from the DPRC in the MC, while
+			 * we were scanning the DPRC.
+			 */
+			if (error != -ENXIO) {
+				dev_err(dev, "dprc_scan_objects() failed: %d\n",
+					error);
+			}
+
+			goto out;
+		}
+
+		if (irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS) {
+			dev_warn(dev,
+				 "IRQs needed (%u) exceed IRQs preallocated (%u)\n",
+				 irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+		}
+	}
+
+out:
+	mutex_unlock(&mc_bus->scan_mutex);
+	return IRQ_HANDLED;
+}
+
+/*
+ * Disable and clear interrupt for a given DPRC object
+ */
+static int disable_dprc_irq(struct fsl_mc_device *mc_dev)
+{
+	int error;
+	struct fsl_mc_io *mc_io = mc_dev->mc_io;
+
+	WARN_ON(mc_dev->obj_desc.irq_count != 1);
+
+	/*
+	 * Disable generation of interrupt, while we configure it:
+	 */
+	error = dprc_set_irq_enable(mc_io, 0, mc_dev->mc_handle, 0, 0);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Disabling DPRC IRQ failed: dprc_set_irq_enable() failed: %d\n",
+			error);
+		return error;
+	}
+
+	/*
+	 * Disable all interrupt causes for the interrupt:
+	 */
+	error = dprc_set_irq_mask(mc_io, 0, mc_dev->mc_handle, 0, 0x0);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Disabling DPRC IRQ failed: dprc_set_irq_mask() failed: %d\n",
+			error);
+		return error;
+	}
+
+	/*
+	 * Clear any leftover interrupts:
+	 */
+	error = dprc_clear_irq_status(mc_io, 0, mc_dev->mc_handle, 0, ~0x0U);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Disabling DPRC IRQ failed: dprc_clear_irq_status() failed: %d\n",
+			error);
+		return error;
+	}
+
+	return 0;
+}
+
+static int register_dprc_irq_handler(struct fsl_mc_device *mc_dev)
+{
+	int error;
+	struct fsl_mc_device_irq *irq = mc_dev->irqs[0];
+
+	WARN_ON(mc_dev->obj_desc.irq_count != 1);
+
+	/*
+	 * NOTE: devm_request_threaded_irq() invokes the device-specific
+	 * function that programs the MSI physically in the device
+	 */
+	error = devm_request_threaded_irq(&mc_dev->dev,
+					  irq->msi_desc->irq,
+					  dprc_irq0_handler,
+					  dprc_irq0_handler_thread,
+					  IRQF_NO_SUSPEND | IRQF_ONESHOT,
+					  "FSL MC DPRC irq0",
+					  &mc_dev->dev);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"devm_request_threaded_irq() failed: %d\n",
+			error);
+		return error;
+	}
+
+	return 0;
+}
+
+static int enable_dprc_irq(struct fsl_mc_device *mc_dev)
+{
+	int error;
+
+	/*
+	 * Enable all interrupt causes for the interrupt:
+	 */
+	error = dprc_set_irq_mask(mc_dev->mc_io, 0, mc_dev->mc_handle, 0,
+				  ~0x0u);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Enabling DPRC IRQ failed: dprc_set_irq_mask() failed: %d\n",
+			error);
+
+		return error;
+	}
+
+	/*
+	 * Enable generation of the interrupt:
+	 */
+	error = dprc_set_irq_enable(mc_dev->mc_io, 0, mc_dev->mc_handle, 0, 1);
+	if (error < 0) {
+		dev_err(&mc_dev->dev,
+			"Enabling DPRC IRQ failed: dprc_set_irq_enable() failed: %d\n",
+			error);
+
+		return error;
+	}
+
+	return 0;
+}
+
+/*
+ * Setup interrupt for a given DPRC device
+ */
+static int dprc_setup_irq(struct fsl_mc_device *mc_dev)
+{
+	int error;
+
+	error = fsl_mc_allocate_irqs(mc_dev);
+	if (error < 0)
+		return error;
+
+	error = disable_dprc_irq(mc_dev);
+	if (error < 0)
+		goto error_free_irqs;
+
+	error = register_dprc_irq_handler(mc_dev);
+	if (error < 0)
+		goto error_free_irqs;
+
+	error = enable_dprc_irq(mc_dev);
+	if (error < 0)
+		goto error_free_irqs;
+
+	return 0;
+
+error_free_irqs:
+	fsl_mc_free_irqs(mc_dev);
+	return error;
+}
+
+/**
  * dprc_probe - callback invoked when a DPRC is being bound to this driver
  *
  * @mc_dev: Pointer to fsl-mc device representing a DPRC
@@ -476,6 +701,13 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
 	if (error < 0)
 		goto error_cleanup_open;

+	/*
+	 * Configure interrupt for the DPRC object associated with this MC bus:
+	 */
+	error = dprc_setup_irq(mc_dev);
+	if (error < 0)
+		goto error_cleanup_open;
+
 	dev_info(&mc_dev->dev, "DPRC device bound to driver");
 	return 0;

@@ -494,6 +726,15 @@ error_cleanup_msi_domain:
 	return error;
 }

+/*
+ * Tear down interrupt for a given DPRC object
+ */
+static void dprc_teardown_irq(struct fsl_mc_device *mc_dev)
+{
+	(void)disable_dprc_irq(mc_dev);
+	fsl_mc_free_irqs(mc_dev);
+}
+
 /**
  * dprc_remove - callback invoked when a DPRC is being unbound from this driver
  *
@@ -514,6 +755,12 @@ static int dprc_remove(struct fsl_mc_device *mc_dev)
 	if (WARN_ON(!mc_dev->mc_io))
 		return -EINVAL;

+	if (WARN_ON(!mc_bus->irq_resources))
+		return -EINVAL;
+
+	if (dev_get_msi_domain(&mc_dev->dev))
+		dprc_teardown_irq(mc_dev);
+
 	device_for_each_child(&mc_dev->dev, NULL, __fsl_mc_device_remove);
 	dprc_cleanup_all_resource_pools(mc_dev);
 	error = dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
--
2.3.3

[PATCH v2 11/11] staging: fsl-mc: Added MSI support to the MC bus driver

[J. German Rivera]

Initialize/Cleanup ITS-MSI support for the MC bus driver at driver
init/exit time. Associate an MSI domain with each DPAA2 child device.

Signed-off-by: J. German Rivera <German.Rivera@freescale.com>
---
Changes in v2: none

 drivers/staging/fsl-mc/bus/mc-bus.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/staging/fsl-mc/bus/mc-bus.c b/drivers/staging/fsl-mc/bus/mc-bus.c
index d34f1af..9317561 100644
--- a/drivers/staging/fsl-mc/bus/mc-bus.c
+++ b/drivers/staging/fsl-mc/bus/mc-bus.c
@@ -16,6 +16,8 @@
 #include <linux/ioport.h>
 #include <linux/slab.h>
 #include <linux/limits.h>
+#include <linux/bitops.h>
+#include <linux/msi.h>
 #include "../include/dpmng.h"
 #include "../include/mc-sys.h"
 #include "dprc-cmd.h"
@@ -472,6 +474,8 @@ int fsl_mc_device_add(struct dprc_obj_desc *obj_desc,
 		mc_dev->icid = parent_mc_dev->icid;
 		mc_dev->dma_mask = FSL_MC_DEFAULT_DMA_MASK;
 		mc_dev->dev.dma_mask = &mc_dev->dma_mask;
+		dev_set_msi_domain(&mc_dev->dev,
+				   dev_get_msi_domain(&parent_mc_dev->dev));
 	}

 	/*
@@ -833,8 +837,15 @@ static int __init fsl_mc_bus_driver_init(void)
 	if (error < 0)
 		goto error_cleanup_dprc_driver;

+	error = its_fsl_mc_msi_init();
+	if (error < 0)
+		goto error_cleanup_mc_allocator;
+
 	return 0;

+error_cleanup_mc_allocator:
+	fsl_mc_allocator_driver_exit();
+
 error_cleanup_dprc_driver:
 	dprc_driver_exit();

@@ -856,6 +867,7 @@ static void __exit fsl_mc_bus_driver_exit(void)
 	if (WARN_ON(!mc_dev_cache))
 		return;

+	its_fsl_mc_msi_cleanup();
 	fsl_mc_allocator_driver_exit();
 	dprc_driver_exit();
 	platform_driver_unregister(&fsl_mc_bus_driver);
--
2.3.3