[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
 QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
 Security Engine subsystems. All subsystems communicate with QMTM using
 messages which include information about the work to be performed and
 the location of associated data buffers.  

V2:
 * Adding COMPILE_TEST dependency for QMTM in Kconfig
 * Updated license banner as per review comments
 * Fixed alignment for macros
 * Changed EXPORT_SYMBOL to EXPORT_SYMBOL_GPL

V1:
 * inital version

Signed-off-by: Ravi Patel <rapatel@apm.com>
Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
---
Ravi Patel (4):
  Documentation: Add documentation for APM X-Gene SoC Queue
    Manager/Traffic Manager DTS binding
  misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic
    Manager
  arm64: boot: dts: Add DTS entries for APM X-Gene SoC Queue
    Manager/Traffic Manager
  misc: xgene: Add error handling for APM X-Gene SoC Queue
    Manager/Traffic Manager

 .../devicetree/bindings/misc/apm-xgene-qmtm.txt    |   51 ++
 MAINTAINERS                                        |    7 +
 arch/arm64/boot/dts/apm-storm.dtsi                 |   17 +
 drivers/misc/Kconfig                               |    1 +
 drivers/misc/Makefile                              |    1 +
 drivers/misc/xgene/Kconfig                         |    1 +
 drivers/misc/xgene/Makefile                        |    5 +
 drivers/misc/xgene/qmtm/Kconfig                    |    9 +
 drivers/misc/xgene/qmtm/Makefile                   |    7 +
 drivers/misc/xgene/qmtm/xgene_qmtm_error.c         |  283 ++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_main.c          |  765 ++++++++++++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_main.h          |  138 ++++
 drivers/misc/xgene/qmtm/xgene_qmtm_storm.c         |  370 ++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_storm.h         |  139 ++++
 include/misc/xgene/xgene_qmtm.h                    |  277 +++++++
 15 files changed, 2071 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
 create mode 100644 drivers/misc/xgene/Kconfig
 create mode 100644 drivers/misc/xgene/Makefile
 create mode 100644 drivers/misc/xgene/qmtm/Kconfig
 create mode 100644 drivers/misc/xgene/qmtm/Makefile
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_error.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_main.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_main.h
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_storm.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_storm.h
 create mode 100644 include/misc/xgene/xgene_qmtm.h

-- 
1.7.9.5

[PATCH V2 1/4] Documentation: Add documentation for APM X-Gene SoC Queue Manager/Traffic Manager DTS binding

[Ravi Patel]

This patch adds device tree binding documentation for APM X-Gene SoC
 Queue Manager/Traffic Manager.

Signed-off-by: Ravi Patel <rapatel@apm.com>
Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
---
 .../devicetree/bindings/misc/apm-xgene-qmtm.txt    |   51 ++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt

diff --git a/Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt b/Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
new file mode 100644
index 0000000..c3fcbd2
--- /dev/null
+++ b/Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
@@ -0,0 +1,51 @@
+* APM X-Gene SoC Queue Manager/Traffic Manager nodes
+
+QMTM nodes are defined to describe on-chip Queue Managers in APM X-Gene SoC.
+APM X-Gene SoC Ethernet, PktDMA (XOR Engine), and Security Engine subsystems
+communicate with a central Queue Manager using messages which include
+information about the work to be performed and the location of the associated
+data buffers. There are multiple instances of QMTM. Each QMTM instance has its
+own node. Its corresponding clock nodes are shown below.
+
+Required properties:
+- compatible		: Shall be "apm,xgene-qmtm-lite" for QMLite instance
+- reg			: First memory resource shall be the QMTM CSR memory
+			  resource.
+			  Second memory resource shall be the QMTM IO-Fabric
+			  memory resource.
+- slave-name		: Shall be "CPU_QMTM3" which is receiver for ingress
+			  work messages for the QMTM. Here receiver is CPU.
+- interrupts		: First interrupt resource shall be the QMTM Error
+			  interrupt.
+			  Remaining interrupt resources shall be the Ingress
+			  work message interrupt mapping for receiver,
+			  receiving work messages for the QMTM.
+- clocks		: Reference to the clock entry.
+
+Optional properties:
+- status		: Shall be "ok" if enabled or "disabled" if disabled.
+			  Default is "ok".
+
+Example:
+		qmlclk: qmlclk {
+			compatible = "apm,xgene-device-clock";
+			#clock-cells = <1>;
+			clock-names = "qmlclk";
+			status = "ok";
+			csr-offset = <0x0>;
+			csr-mask = <0x3>;
+			enable-offset = <0x8>;
+			enable-mask = <0x3>;
+		};
+
+		qmlite: qmtm at 17030000 {
+			compatible = "apm,xgene-qmtm-lite";
+			reg = <0x0 0x17030000 0x0 0x10000>,
+			      <0x0 0x10000000 0x0 0x400000>;
+			slave-name = "CPU_QMTM3";
+			interrupts = <0x0 0x40 0x4>,
+				     <0x0 0x3c 0x4>;
+			status = "ok";
+			#clock-cells = <1>;
+			clocks = <&qmlclk 0>;
+		};
-- 
1.7.9.5

[PATCH V2 2/4] misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

This patch adds APM X-Gene SoC Queue Manager/Traffic Manager base driver.
 QMTM is requried by Ethernet, PktDMA (XOR Engine) and Security Engine
 subsystems.

Signed-off-by: Ravi Patel <rapatel@apm.com>
Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
---
 MAINTAINERS                                |    7 +
 drivers/misc/Kconfig                       |    1 +
 drivers/misc/Makefile                      |    1 +
 drivers/misc/xgene/Kconfig                 |    1 +
 drivers/misc/xgene/Makefile                |    5 +
 drivers/misc/xgene/qmtm/Kconfig            |    9 +
 drivers/misc/xgene/qmtm/Makefile           |    7 +
 drivers/misc/xgene/qmtm/xgene_qmtm_main.c  |  761 ++++++++++++++++++++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_main.h  |  134 +++++
 drivers/misc/xgene/qmtm/xgene_qmtm_storm.c |  370 ++++++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_storm.h |  139 +++++
 include/misc/xgene/xgene_qmtm.h            |  277 ++++++++++
 12 files changed, 1712 insertions(+)
 create mode 100644 drivers/misc/xgene/Kconfig
 create mode 100644 drivers/misc/xgene/Makefile
 create mode 100644 drivers/misc/xgene/qmtm/Kconfig
 create mode 100644 drivers/misc/xgene/qmtm/Makefile
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_main.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_main.h
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_storm.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_storm.h
 create mode 100644 include/misc/xgene/xgene_qmtm.h

diff --git a/MAINTAINERS b/MAINTAINERS
index f216db8..920cae8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -665,6 +665,13 @@ S:	Maintained
 F:	drivers/net/appletalk/
 F:	net/appletalk/
 
+APPLIEDMICRO (APM) X-GENE SOC QUEUE MANAGER/TRAFFIC MANAGER (QMTM) DRIVER
+M:	Ravi Patel <rapatel@apm.com>
+M:	Keyur Chudgar <kchudgar@apm.com>
+S:	Maintained
+F:	drivers/misc/xgene/
+F:	include/misc/xgene/xgene_qmtm.h
+
 APTINA CAMERA SENSOR PLL
 M:	Laurent Pinchart <Laurent.pinchart@ideasonboard.com>
 L:	linux-media at vger.kernel.org
diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index a3e291d..b309553 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -525,4 +525,5 @@ source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
 source "drivers/misc/mic/Kconfig"
+source "drivers/misc/xgene/Kconfig"
 endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index f45473e..0fd3b1b 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -53,3 +53,4 @@ obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
 obj-y				+= mic/
+obj-$(CONFIG_ARCH_XGENE)	+= xgene/
diff --git a/drivers/misc/xgene/Kconfig b/drivers/misc/xgene/Kconfig
new file mode 100644
index 0000000..8f38568
--- /dev/null
+++ b/drivers/misc/xgene/Kconfig
@@ -0,0 +1 @@
+source "drivers/misc/xgene/qmtm/Kconfig"
diff --git a/drivers/misc/xgene/Makefile b/drivers/misc/xgene/Makefile
new file mode 100644
index 0000000..198c2e6
--- /dev/null
+++ b/drivers/misc/xgene/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for APM X-GENE misc drivers.
+#
+
+obj-$(CONFIG_XGENE_QMTM)	+= qmtm/
diff --git a/drivers/misc/xgene/qmtm/Kconfig b/drivers/misc/xgene/qmtm/Kconfig
new file mode 100644
index 0000000..8efda4d
--- /dev/null
+++ b/drivers/misc/xgene/qmtm/Kconfig
@@ -0,0 +1,9 @@
+config XGENE_QMTM
+	tristate "APM X-Gene Queue Manager/Traffic Manager driver"
+	depends on ARM64 || COMPILE_TEST
+	default y
+	help
+	  This option enables APM X-Gene Queue Manager Traffic Manager (QMTM)
+	  driver support.
+	  QMTM is required for Ethernet, PktDMA (XOR Engine) and Security
+	  Engine.
diff --git a/drivers/misc/xgene/qmtm/Makefile b/drivers/misc/xgene/qmtm/Makefile
new file mode 100644
index 0000000..68c2a86
--- /dev/null
+++ b/drivers/misc/xgene/qmtm/Makefile
@@ -0,0 +1,7 @@
+#
+# Makefile for APM X-GENE Queue Manager Traffic Manager driver.
+#
+
+obj-$(CONFIG_XGENE_QMTM) += xgene-qmtm.o
+
+xgene-qmtm-objs := xgene_qmtm_main.o xgene_qmtm_storm.o
diff --git a/drivers/misc/xgene/qmtm/xgene_qmtm_main.c b/drivers/misc/xgene/qmtm/xgene_qmtm_main.c
new file mode 100644
index 0000000..cda63e0
--- /dev/null
+++ b/drivers/misc/xgene/qmtm/xgene_qmtm_main.c
@@ -0,0 +1,761 @@
+/*
+ * AppliedMicro X-Gene SOC Queue Manager/Traffic Manager driver
+ *
+ * Copyright (c) 2013 Applied Micro Circuits Corporation.
+ * Author: Ravi Patel <rapatel@apm.com>
+ *         Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/module.h>
+#include <linux/io.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/slab.h>
+#include "xgene_qmtm_main.h"
+
+#define XGENE_QMTM_DRIVER_VER	"1.0"
+#define XGENE_QMTM_DRIVER_NAME	"xgene-qmtm"
+#define XGENE_QMTM_DRIVER_DESC	"APM X-Gene QMTM driver"
+
+/* CSR Address Macros */
+#define CSR_QM_CONFIG_ADDR	0x00000004
+#define  QM_ENABLE_WR(src)	(((u32)(src)<<31) & 0x80000000)
+
+#define CSR_PBM_ADDR		0x00000008
+#define  OVERWRITE_WR(src)	(((u32)(src)<<31) & 0x80000000)
+#define  SLVID_PBN_WR(src)	(((u32)(src)) & 0x000003ff)
+#define  SLAVE_ID_SHIFT		6
+
+#define CSR_PBM_BUF_WR_ADDR	0x0000000c
+#define CSR_PBM_BUF_RD_ADDR	0x00000010
+#define  PB_SIZE_WR(src)	(((u32)(src)<<31) & 0x80000000)
+#define  PREFETCH_BUF_EN_SET(dst, src) \
+	(((dst) & ~0x00200000) | (((u32)(src)<<21) & 0x00200000))
+#define  IS_FREE_POOL_SET(dst, src) \
+	(((dst) & ~0x00100000) | (((u32)(src)<<20) & 0x00100000))
+#define  TLVQ_SET(dst, src) \
+	(((dst) & ~0x00080000) | (((u32)(src)<<19) & 0x00080000))
+#define  CORRESPONDING_QNUM_SET(dst, src) \
+	(((dst) & ~0x0007fe00) | (((u32)(src)<<9) & 0x0007fe00))
+
+#define CSR_PBM_CTICK0_ADDR	0x00000018
+#define  MIN_COAL_TAP		0x0
+#define  MAX_COAL_TAP		0x7
+
+#define CSR_THRESHOLD0_SET1_ADDR	0x00000030
+#define CSR_THRESHOLD1_SET1_ADDR	0x00000034
+#define CSR_HYSTERESIS_ADDR		0x00000068
+#define CSR_QM_MBOX_NE_INT_MODE_ADDR	0x0000017c
+#define CSR_QMLITE_PBN_MAP_0_ADDR	0x00000228
+
+#define CSR_RECOMB_CTRL_0_ADDR		0x00000230
+#define  RECOMB_EN0_SET(dst, src) \
+	(((dst) & ~0x00000001) | (((u32)(src)) & 0x00000001))
+
+/* QMTM Diag CSR */
+#define QM_GLBL_DIAG_CSR_BASE_ADDR_OFFSET	0xd000
+#define QM_CFG_MEM_RAM_SHUTDOWN_ADDR		0x00000070
+#define QM_CFG_MEM_RAM_SHUTDOWN_DEFAULT		0xffffffff
+
+/* PBN macros */
+#define QMTM_MIN_PBN_ID	0
+#define QMTM_MAX_PBN_ID	31
+
+/* Common for Queue ID and PBN */
+#define RES_MASK(nr)	(1UL << ((nr) % 32))
+#define RES_WORD(nr)	((nr) / 32)
+
+void xgene_qmtm_wr32(struct xgene_qmtm *qmtm, u32 offset, u32 data)
+{
+	void *addr = (u8 *)qmtm->csr_vaddr + offset;
+	writel(data, addr);
+}
+
+void xgene_qmtm_rd32(struct xgene_qmtm *qmtm, u32 offset, u32 *data)
+{
+	void *addr = (u8 *)qmtm->csr_vaddr + offset;
+	*data = readl(addr);
+}
+
+/* Get available PBN or Queue Id */
+static int xgene_qmtm_get_resource_id(u32 *resource, u32 start, u32 end)
+{
+	u32 index;
+
+	for (index = start; index < end; index++) {
+		if ((resource[RES_WORD(index)] & RES_MASK(index)) == 0) {
+			resource[RES_WORD(index)] |= RES_MASK(index);
+			return index;
+		}
+	}
+
+	return -ENODEV;
+}
+
+/* Put used PBN or Queue Id */
+static inline void xgene_qmtm_put_resource_id(u32 *resource, u32 index)
+{
+	resource[RES_WORD(index)] &= ~(u32) RES_MASK(index);
+}
+
+static void xgene_qmtm_write_pbm(struct xgene_qmtm_qinfo *qinfo, u32 val)
+{
+	u32 pbm = SLVID_PBN_WR((qinfo->slave_id << SLAVE_ID_SHIFT) |
+				    qinfo->pbn) | OVERWRITE_WR(1);
+
+	xgene_qmtm_wr32(qinfo->qmtm, CSR_PBM_ADDR, pbm);
+
+	if (qinfo->qmtm_ip == QMTM0 || qinfo->qmtm_ip == QMTM2)
+		val |= PB_SIZE_WR(1);
+
+	xgene_qmtm_wr32(qinfo->qmtm, CSR_PBM_BUF_WR_ADDR, val);
+}
+
+static u32 xgene_qmtm_read_pbm(struct xgene_qmtm_qinfo *qinfo)
+{
+	u32 pbm = SLVID_PBN_WR((qinfo->slave_id << SLAVE_ID_SHIFT) |
+				    qinfo->pbn);
+
+	xgene_qmtm_wr32(qinfo->qmtm, CSR_PBM_ADDR, pbm);
+	xgene_qmtm_rd32(qinfo->qmtm, CSR_PBM_BUF_RD_ADDR, &pbm);
+
+	return pbm;
+}
+
+static void xgene_qmtm_set_pbm(struct xgene_qmtm_qinfo *qinfo)
+{
+	u16 is_fp = qinfo->qtype == QTYPE_FP ? 1 : 0;
+	u16 is_vq = qinfo->qtype == QTYPE_VQ ? 1 : 0;
+	u32 val = 0;
+
+	val = CORRESPONDING_QNUM_SET(val, qinfo->queue_id);
+	val = IS_FREE_POOL_SET(val, is_fp);
+	val = TLVQ_SET(val, is_vq);
+	val = PREFETCH_BUF_EN_SET(val, 1);
+	xgene_qmtm_write_pbm(qinfo, val);
+}
+
+static void xgene_qmtm_clr_pbm(struct xgene_qmtm_qinfo *qinfo)
+{
+	xgene_qmtm_write_pbm(qinfo, 0);
+}
+
+/**
+ * xgene_qmtm_set_qinfo - Create and configure a queue
+ * @sdev:	Slave context
+ * @qtype:	Queue type (P_QUEUE or FP_QUEUE)
+ * @qsize:	Queue size see xgene_qmtm_qsize
+ * @qaccess:	Queue access method see xgene_qmtm_qaccess
+ * @flags:	Queue Information flags
+ * @qpaddr:	If Desire Queue Physical Address to use
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to create and configure a queue.
+ *
+ * Return:	0 on Success or -1 on Failure
+ *		On Success, updates following in qinfo,
+ *		qmtm_ip - QMTM0, QMTM1, QMTM2 or QMTM3
+ *		qmtm - QMTM instance context
+ *		slave - Slave see xgene_slave
+ *		slave_id - Slave ID see xgene_qmtm_slave_id
+ *		pbn - Slave ID's prefetch buffer number
+ *		queue_id - Queue ID
+ *		qdesc - Queue descriptor
+ */
+int xgene_qmtm_set_qinfo(struct xgene_qmtm_qinfo *set)
+{
+	struct xgene_qmtm_sdev *sdev = set->sdev;
+	struct device *dev;
+	struct xgene_qmtm *qmtm;
+	struct xgene_qmtm_qinfo *qinfo;
+	u32 *queue_resource = NULL, *pbn_resource = NULL;
+	int rc;
+	u8 pbn = 0;
+	u16 queue_id = 0;
+
+	qmtm = sdev->qmtm;
+	dev = &qmtm->pdev->dev;
+
+	if (set->flags & XGENE_SLAVE_PB_CONFIGURE) {
+		u8 pbn_start, pbn_count;
+
+		if (set->qtype == QTYPE_FP) {
+			pbn_resource = &sdev->fq_pbn_pool;
+			pbn_start = sdev->fq_pbn_start & ~(u8) 0x20;
+			pbn_count = sdev->fq_pbn_count;
+		} else {
+			pbn_resource = &sdev->wq_pbn_pool;
+			pbn_start = sdev->wq_pbn_start;
+			pbn_count = sdev->wq_pbn_count;
+		}
+
+		rc = xgene_qmtm_get_resource_id(pbn_resource, pbn_start,
+						pbn_start + pbn_count);
+		if (rc < 0) {
+			dev_err(dev, "SETQ: slave %d out of PBN\n",
+				sdev->slave);
+			goto _ret_set_qinfo;
+		}
+
+		pbn = rc;
+	}
+
+	queue_resource = qmtm->queue_pool;
+	rc = xgene_qmtm_get_resource_id(queue_resource, 0, qmtm->max_queues);
+	if (rc < 0) {
+		dev_err(dev, "SETQ: QMTM %d out of Queue ID\n", sdev->qmtm_ip);
+		goto _put_pbn_resource;
+	}
+
+	queue_id = rc;
+	qinfo = kzalloc(sizeof(struct xgene_qmtm_qinfo), GFP_KERNEL);
+	if (qinfo == NULL) {
+		dev_err(dev, "SETQ: Unable to allocate qinfo\n");
+		rc = -ENOMEM;
+		goto _put_queue_resource;
+	}
+
+	qinfo->slave = sdev->slave;
+	qinfo->slave_id = sdev->slave_id;
+	qinfo->qmtm_ip = sdev->qmtm_ip;
+	qinfo->qtype = set->qtype;
+	qinfo->qsize = set->qsize;
+	qinfo->qaccess = set->qaccess;
+	qinfo->flags = set->flags;
+	qinfo->pbn = set->qtype == QTYPE_FP ? (pbn | 0x20) : pbn;
+	qinfo->queue_id = queue_id;
+	qinfo->qpaddr = set->qpaddr;
+	qinfo->qfabric = qmtm->fabric_vaddr + (queue_id << 6);
+	qinfo->sdev = sdev;
+	qinfo->qmtm = qmtm;
+	rc = qmtm->set_qstate(qinfo);
+	if (rc < 0) {
+		dev_err(dev, "SETQ: set_qstate error for %s Queue ID %d\n",
+			sdev->name, queue_id);
+		goto _del_qstate;
+	}
+
+	if (qinfo->qaccess == QACCESS_ALT)
+		qinfo->qdesc->command = qinfo->qfabric + 0x2C;
+
+	if (set->flags & XGENE_SLAVE_PB_CONFIGURE) {
+		xgene_qmtm_set_pbm(qinfo);
+
+		if (set->qaccess == QACCESS_ALT &&
+		    sdev->slave_id == QMTM_SLAVE_ID_CPU &&
+		    set->qtype == QTYPE_PQ) {
+			u32 data;
+
+			xgene_qmtm_rd32(qmtm, CSR_QM_MBOX_NE_INT_MODE_ADDR,
+					&data);
+			data |= (u32) (1 << (31 - pbn));
+			xgene_qmtm_wr32(qmtm, CSR_QM_MBOX_NE_INT_MODE_ADDR,
+					data);
+		}
+
+		if (set->qtype == QTYPE_PQ &&
+		    (sdev->slave_id == QMTM_SLAVE_ID_CPU ||
+		     sdev->slave_id == QMTM_SLAVE_ID_MSLIM))
+			qinfo->qdesc->irq = qmtm->dequeue_irq[pbn];
+	}
+
+	qmtm->qinfo[queue_id] = qinfo;
+	memcpy(set, qinfo, sizeof(struct xgene_qmtm_qinfo));
+	return rc;
+
+_del_qstate:
+	qmtm->clr_qstate(qinfo);
+	kfree(qinfo);
+
+_put_queue_resource:
+	xgene_qmtm_put_resource_id(queue_resource, queue_id);
+
+_put_pbn_resource:
+	if (set->flags & XGENE_SLAVE_PB_CONFIGURE)
+		xgene_qmtm_put_resource_id(pbn_resource, pbn);
+
+_ret_set_qinfo:
+	return rc;
+}
+EXPORT_SYMBOL_GPL(xgene_qmtm_set_qinfo);
+
+/**
+ * xgene_qmtm_clr_qinfo - Unconfigure and delete a queue
+ * @sdev:	Slave context
+ * @queue_id:	Queue ID
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to unconfigure and delete a queue.
+ */
+void xgene_qmtm_clr_qinfo(struct xgene_qmtm_qinfo *clr)
+{
+	struct xgene_qmtm_sdev *sdev = clr->sdev;
+	struct xgene_qmtm *qmtm;
+	struct xgene_qmtm_qinfo *qinfo;
+	u8 queue_id = clr->queue_id;
+
+	qmtm = sdev->qmtm;
+	qinfo = qmtm->qinfo[queue_id];
+
+	if (qinfo->flags & XGENE_SLAVE_PB_CONFIGURE) {
+		u32 *pbn_resource;
+		u8 qtype = qinfo->qtype;
+		u8 pbn = (qtype == QTYPE_FP) ?
+		    (qinfo->pbn & ~(u8) 0x20) : qinfo->pbn;
+
+		if (qinfo->qaccess == QACCESS_ALT &&
+		    qinfo->slave_id == QMTM_SLAVE_ID_CPU && qtype == QTYPE_PQ) {
+			u32 data;
+
+			xgene_qmtm_rd32(qmtm, CSR_QM_MBOX_NE_INT_MODE_ADDR,
+					&data);
+			data &= ~(u32) (1 << (31 - pbn));
+			xgene_qmtm_wr32(qmtm, CSR_QM_MBOX_NE_INT_MODE_ADDR,
+					data);
+		}
+
+		if (qinfo->qtype == QTYPE_FP)
+			pbn_resource = &sdev->fq_pbn_pool;
+		else
+			pbn_resource = &sdev->wq_pbn_pool;
+
+		xgene_qmtm_clr_pbm(qinfo);
+		xgene_qmtm_put_resource_id(pbn_resource, pbn);
+	}
+
+	qmtm->clr_qstate(qinfo);
+	kfree(qinfo);
+	xgene_qmtm_put_resource_id(qmtm->queue_pool, queue_id);
+	qmtm->qinfo[queue_id] = NULL;
+}
+EXPORT_SYMBOL_GPL(xgene_qmtm_clr_qinfo);
+
+/**
+ * xgene_qmtm_read_qstate - Get Queue State information for a queue
+ * @qmtm:	QMTM instance context
+ * @queue_id:	Queue ID
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to read queue state of a queue.
+ *
+ * Return:	Updates following in qinfo,
+ *		qstate - Current Queue State in QMTM
+ *		nummsgs - Number os messages in the Queue
+ *		pbm_state - Current prefetch buffer manager state
+ */
+void xgene_qmtm_read_qstate(struct xgene_qmtm_qinfo *qinfo)
+{
+	struct xgene_qmtm *qmtm = qinfo->qmtm;
+	u8 queue_id = qinfo->queue_id;
+
+	memcpy(qinfo, qmtm->qinfo[queue_id], sizeof(struct xgene_qmtm_qinfo));
+	qmtm->read_qstate(qinfo);
+
+	if (qinfo->flags & XGENE_SLAVE_PB_CONFIGURE)
+		qinfo->pbm_state = xgene_qmtm_read_pbm(qinfo);
+}
+EXPORT_SYMBOL_GPL(xgene_qmtm_read_qstate);
+
+/**
+ * xgene_qmtm_intr_coalesce - Set interrupt coalescing for ingrgess queue
+ * @qmtm:	QMTM instance context
+ * @pbn:	CPU's prefetch buffer number corresponding to the interrupt
+ * @tap:	Tap value to set
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to set interrupt for its ingress queue.
+ *
+ * Return:	0 on Success or -1 on Failure
+ */
+int xgene_qmtm_intr_coalesce(struct xgene_qmtm_qinfo *qinfo, u8 tap)
+{
+	u32 val, offset, mask, shift;
+	u8 pbn = qinfo->pbn;
+
+	if (tap < MIN_COAL_TAP || tap > MAX_COAL_TAP)
+		return -EINVAL;
+
+	if (pbn < QMTM_MIN_PBN_ID || pbn > QMTM_MAX_PBN_ID)
+		return -EINVAL;
+
+	offset = 4 * (pbn / 8);
+	shift = 4 * (7 - (pbn % 8));
+	mask = 7 << shift;
+
+	xgene_qmtm_rd32(qinfo->qmtm, CSR_PBM_CTICK0_ADDR + offset, &val);
+	val = (val & ~(u32) mask) | (((u32) tap << shift) & mask);
+	xgene_qmtm_wr32(qinfo->qmtm, CSR_PBM_CTICK0_ADDR + offset, val);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(xgene_qmtm_intr_coalesce);
+
+/**
+ * xgene_qmtm_fp_dealloc_msg - Fill a buffer in a free pool queue
+ * @qdesc:	Queue descriptor
+ * @msg:	QMTM free pool buffer message to fill in to queue
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to fill a buffer in its free pool queue.
+ */
+void xgene_qmtm_fp_dealloc_msg(struct xgene_qmtm_qdesc *qdesc,
+			       struct xgene_qmtm_msg16 *msg)
+{
+	u32 qtail = qdesc->qtail;
+	u32 count = qdesc->count;
+	u8 *tailptr = (u8 *)&qdesc->msg16[qtail];
+
+	memcpy(tailptr, msg, 16);
+
+	if (++qtail == count)
+		qtail = 0;
+
+	writel(1, qdesc->command);
+	qdesc->qtail = qtail;
+}
+EXPORT_SYMBOL_GPL(xgene_qmtm_fp_dealloc_msg);
+
+/**
+ * xgene_qmtm_enqueue_msg - Enqueue a work message in subsystem's work queue
+ * @qdesc:	Queue descriptor
+ * @msg:	X-Gene SOC subsystem's work message to enqueue in to queue
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to enqueue work message in its work queue.
+ */
+void xgene_qmtm_enqueue_msg(struct xgene_qmtm_qdesc *qdesc,
+			    struct xgene_qmtm_msg64 *msg)
+{
+	u32 qtail = qdesc->qtail;
+	u32 count = qdesc->count;
+	u8 *tailptr = (u8 *)&qdesc->msg32[qtail];
+
+	memcpy(tailptr, msg, 32);
+
+	if (++qtail == count)
+		qtail = 0;
+
+	if (!msg->msg32_1.msg16.NV) {
+		writel(1, qdesc->command);
+	} else {
+		memcpy(tailptr + 32, (u8 *) msg + 32, 32);
+
+		if (++qtail == count)
+			qtail = 0;
+
+		writel(2, qdesc->command);
+	}
+
+	qdesc->qtail = qtail;
+}
+EXPORT_SYMBOL_GPL(xgene_qmtm_enqueue_msg);
+
+/**
+ * xgene_qmtm_dequeue_msg - Dequeue a work message from QMTM instance
+ * @qdesc:	Queue descriptor
+ * @msg:	Dequeued work message from X-Gene SOC subsystem to CPU
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to dequeue work message from its ingress
+ * queue.
+ *
+ * Return:	0 on Success or -1 on Failure
+ */
+int xgene_qmtm_dequeue_msg(struct xgene_qmtm_qdesc *qdesc,
+			   struct xgene_qmtm_msg64 *msg)
+{
+	u32 qhead = qdesc->qhead;
+	u32 count = qdesc->count;
+	u32 *headptr = (u32 *)&qdesc->msg32[qhead];
+
+	if (headptr[EMPTY_SLOT_INDEX] == EMPTY_SLOT)
+		return -EAGAIN;
+
+	memcpy(msg, headptr, 32);
+	headptr[EMPTY_SLOT_INDEX] = EMPTY_SLOT;
+
+	if (++qhead == count)
+		qhead = 0;
+
+	if (!msg->msg32_1.msg16.NV) {
+		writel(0xFFFFFFFF, qdesc->command);
+	} else {
+		headptr += 8;
+
+		if (headptr[EMPTY_SLOT_INDEX] == EMPTY_SLOT)
+			return -EAGAIN;
+
+		memcpy((u8 *) msg + 32, headptr, 32);
+		headptr[EMPTY_SLOT_INDEX] = EMPTY_SLOT;
+
+		if (++qhead == count)
+			qhead = 0;
+
+		writel(0xFFFFFFFE, qdesc->command);
+	}
+
+	qdesc->qhead = qhead;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(xgene_qmtm_dequeue_msg);
+
+/**
+ * xgene_qmtm_get_sdev - Get slave context from slave name
+ * @name:	Slave name
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to get its slave context from its name.
+ *
+ * Return:	Slave context on Success or NULL on Failure
+ */
+struct xgene_qmtm_sdev *xgene_qmtm_get_sdev(char *name)
+{
+	return storm_qmtm_get_sdev(name);
+}
+EXPORT_SYMBOL_GPL(xgene_qmtm_get_sdev);
+
+static int xgene_qmtm_enable(struct xgene_qmtm *qmtm)
+{
+	struct xgene_qmtm_qinfo qinfo;
+	struct device *dev = &qmtm->pdev->dev;
+	int rc, mwait = 0;
+	u32 val;
+	u32 queue_id;
+
+	qmtm->clk = devm_clk_get(dev, NULL);
+	if (IS_ERR(qmtm->clk)) {
+		dev_err(dev, "can't get clock\n");
+		return PTR_ERR(qmtm->clk);
+	}
+
+	rc = clk_prepare_enable(qmtm->clk);
+	if (rc < 0) {
+		dev_err(dev, "clock prepare enable failed");
+		return rc;
+	}
+
+	xgene_qmtm_wr32(qmtm, QM_GLBL_DIAG_CSR_BASE_ADDR_OFFSET +
+			QM_CFG_MEM_RAM_SHUTDOWN_ADDR, 0);
+	do {
+		/* Wait for Memory to come out of shutdown */
+		usleep_range(1000, 2000);
+		xgene_qmtm_rd32(qmtm, QM_GLBL_DIAG_CSR_BASE_ADDR_OFFSET +
+				QM_CFG_MEM_RAM_SHUTDOWN_ADDR, &val);
+
+		if (mwait++ >= 1000) {
+			rc = -EIO;
+			dev_err(dev, "RAM not out of shutdown %d\n", rc);
+			clk_disable_unprepare(qmtm->clk);
+			return rc;
+		}
+	} while (val == QM_CFG_MEM_RAM_SHUTDOWN_DEFAULT);
+
+	switch (qmtm->qmtm_ip) {
+	case QMTM0:
+	case QMTM2:
+		xgene_qmtm_rd32(qmtm, CSR_RECOMB_CTRL_0_ADDR, &val);
+		val = RECOMB_EN0_SET(val, 1);
+		xgene_qmtm_wr32(qmtm, CSR_RECOMB_CTRL_0_ADDR, val);
+		break;
+	case QMTM3:
+		xgene_qmtm_wr32(qmtm, CSR_QMLITE_PBN_MAP_0_ADDR, 0x00000000);
+	}
+
+	/* program threshold set 1 and all hysteresis */
+	xgene_qmtm_wr32(qmtm, CSR_THRESHOLD0_SET1_ADDR, 100);
+	xgene_qmtm_wr32(qmtm, CSR_THRESHOLD1_SET1_ADDR, 200);
+	xgene_qmtm_wr32(qmtm, CSR_HYSTERESIS_ADDR, 0xFFFFFFFF);
+
+	/* Enable QPcore */
+	xgene_qmtm_wr32(qmtm, CSR_QM_CONFIG_ADDR, QM_ENABLE_WR(1));
+
+	/* Clear all HW queue state in case they were not de-activated */
+	memset(&qinfo, 0, sizeof(qinfo));
+	qinfo.qmtm = qmtm;
+
+	for (queue_id = 0; queue_id < qmtm->max_queues; queue_id++) {
+		qinfo.queue_id = queue_id;
+		qmtm->write_qstate(&qinfo);
+	}
+
+	return rc;
+}
+
+static int xgene_qmtm_disable(struct xgene_qmtm *qmtm)
+{
+	u32 queue_id;
+
+	for (queue_id = 0; queue_id < qmtm->max_queues; queue_id++) {
+		if (qmtm->qinfo[queue_id]) {
+			dev_err(&qmtm->pdev->dev,
+				"QMTM %d Queue ID %d Resource in use\n",
+				qmtm->qmtm_ip, queue_id);
+			return -EAGAIN;
+		}
+	}
+
+	/* Disable QPcore */
+	xgene_qmtm_wr32(qmtm, CSR_QM_CONFIG_ADDR, QM_ENABLE_WR(0));
+	clk_disable_unprepare(qmtm->clk);
+
+	return 0;
+}
+
+static struct xgene_qmtm *xgene_alloc_qmtm(struct platform_device *pdev)
+{
+	struct xgene_qmtm *qmtm;
+	int max_queues, malloc_size;
+
+	qmtm = devm_kzalloc(&pdev->dev, sizeof(struct xgene_qmtm), GFP_KERNEL);
+	if (qmtm == NULL) {
+		dev_err(&pdev->dev, "Unable to allocate QMTM context\n");
+		return NULL;
+	}
+
+	qmtm->pdev = pdev;
+	platform_set_drvdata(pdev, qmtm);
+	max_queues = QMTM_MAX_QUEUES;
+	malloc_size = max_queues * (sizeof(struct xgene_qmtm_info *));
+	qmtm->qinfo = devm_kzalloc(&pdev->dev, malloc_size, GFP_KERNEL);
+	if (qmtm->qinfo == NULL) {
+		dev_err(&pdev->dev, "Unable to allocate QMTM Queue context\n");
+		return NULL;
+	}
+
+	malloc_size = RES_WORD(max_queues + 31) * sizeof(u32);
+	qmtm->queue_pool = devm_kzalloc(&pdev->dev, malloc_size, GFP_KERNEL);
+	if (qmtm->queue_pool == NULL) {
+		dev_err(&pdev->dev, "Unable to allocate QMTM Queue Pool\n");
+		return NULL;
+	}
+
+	qmtm->max_queues = max_queues;
+
+	return qmtm;
+}
+
+static int xgene_get_qmtm(struct xgene_qmtm *qmtm)
+{
+	struct platform_device *pdev = qmtm->pdev;
+	struct resource *res;
+	struct xgene_qmtm_sdev *sdev;
+	const char *name;
+	int rc, inum = 1;
+	u16 pbn;
+
+	/* Retrieve QM CSR register address and size */
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+	if (!res) {
+		dev_err(&pdev->dev, "Failed to get QMTM CSR region\n");
+		return -ENODEV;
+	}
+
+	qmtm->csr_vaddr = devm_ioremap_resource(&pdev->dev, res);
+	if (IS_ERR(qmtm->csr_vaddr)) {
+		dev_err(&pdev->dev, "Invalid QMTM CSR region\n");
+		return PTR_ERR(qmtm->csr_vaddr);
+	}
+
+	/* Retrieve Primary Fabric address and size */
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
+	if (!res) {
+		dev_err(&pdev->dev, "Failed to get QMTM Fabric region\n");
+		return -ENODEV;
+	}
+
+	qmtm->fabric_vaddr = devm_ioremap_resource(&pdev->dev, res);
+	if (IS_ERR(qmtm->fabric_vaddr)) {
+		dev_err(&pdev->dev, "Invalid QMTM Fabric region\n");
+		return PTR_ERR(qmtm->fabric_vaddr);
+	}
+
+	rc = of_property_read_string(pdev->dev.of_node, "slave-name", &name);
+	if (rc < 0) {
+		dev_err(&pdev->dev, "Failed to get QMTM Ingress slave-name\n");
+		return rc;
+	}
+
+	sdev = xgene_qmtm_get_sdev((char *)name);
+	if (sdev == NULL) {
+		dev_err(&pdev->dev, "Ingress Slave error\n");
+		return -ENODEV;
+	}
+
+	qmtm->idev = sdev;
+	qmtm->qmtm_ip = sdev->qmtm_ip;
+
+	for (pbn = sdev->wq_pbn_start; pbn < (sdev->wq_pbn_start +
+					      sdev->wq_pbn_count);
+	     pbn++, inum++) {
+		int irq = platform_get_irq(pdev, inum);
+
+		if (irq < 0) {
+			dev_err(&pdev->dev, "Failed to map QMTM%d PBN %d IRQ\n",
+				qmtm->qmtm_ip, pbn);
+			continue;
+		}
+
+		qmtm->dequeue_irq[pbn] = irq;
+	}
+
+	return rc;
+}
+
+static int xgene_qmtm_probe(struct platform_device *pdev)
+{
+	struct xgene_qmtm *qmtm;
+	int rc;
+
+	qmtm = xgene_alloc_qmtm(pdev);
+	if (qmtm == NULL)
+		return -ENOMEM;
+
+	rc = xgene_get_qmtm(qmtm);
+	if (rc)
+		return rc;
+
+	return xgene_qmtm_enable(qmtm);
+}
+
+static int xgene_qmtm_remove(struct platform_device *pdev)
+{
+	struct xgene_qmtm *qmtm = platform_get_drvdata(pdev);
+	return xgene_qmtm_disable(qmtm);
+}
+
+static struct of_device_id xgene_qmtm_match[] = {
+	{.compatible = "apm,xgene-qmtm-xge0",},
+	{.compatible = "apm,xgene-qmtm-soc",},
+	{.compatible = "apm,xgene-qmtm-xge2",},
+	{.compatible = "apm,xgene-qmtm-lite",},
+	{},
+};
+
+MODULE_DEVICE_TABLE(of, xgene_qmtm_match);
+
+static struct platform_driver xgene_qmtm_driver = {
+	.driver = {
+		   .name = XGENE_QMTM_DRIVER_NAME,
+		   .owner = THIS_MODULE,
+		   .of_match_table = xgene_qmtm_match,
+		   },
+	.probe = xgene_qmtm_probe,
+	.remove = xgene_qmtm_remove,
+};
+
+module_platform_driver(xgene_qmtm_driver);
+
+MODULE_VERSION(XGENE_QMTM_DRIVER_VER);
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Ravi Patel <rapatel@apm.com>");
+MODULE_DESCRIPTION(XGENE_QMTM_DRIVER_DESC);
diff --git a/drivers/misc/xgene/qmtm/xgene_qmtm_main.h b/drivers/misc/xgene/qmtm/xgene_qmtm_main.h
new file mode 100644
index 0000000..d937462
--- /dev/null
+++ b/drivers/misc/xgene/qmtm/xgene_qmtm_main.h
@@ -0,0 +1,134 @@
+/*
+ * AppliedMicro X-Gene SOC Queue Manager/Traffic Manager driver
+ *
+ * Copyright (c) 2013 Applied Micro Circuits Corporation.
+ * Author: Ravi Patel <rapatel@apm.com>
+ *         Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __XGENE_QMTM_MAIN_H__
+#define __XGENE_QMTM_MAIN_H__
+
+#include <linux/of_platform.h>
+#include <misc/xgene/xgene_qmtm.h>
+
+/* QMTM Platform Information */
+enum xgene_qmtm_pinfo {
+	STORM_QMTM,
+	MAX_PLATFORM,
+};
+
+/* QMTM IP Blocks */
+enum xgene_qmtm_ip {
+	QMTM0,
+	QMTM1,
+	QMTM2,
+	QMTM3,
+	QMTM_MAX,
+};
+
+#define QMTM_MAX_QUEUES	1024
+#define QMTM_MAX_PBN	32
+
+struct xgene_qmtm {
+	void *csr_vaddr;
+	void *fabric_vaddr;
+	u16 qmtm_ip;		/* qmtm_ip, see xgene_qmtm_ip */
+	u16 error_irq;
+	u32 max_queues;
+	u16 dequeue_irq[QMTM_MAX_PBN];
+	char error_irq_s[16];
+	char error_queue_irq_s[16];
+	struct xgene_qmtm_sdev *idev;
+	u32 *queue_pool;
+	struct xgene_qmtm_qinfo *(*qinfo);
+	struct xgene_qmtm_qinfo *error_qinfo;
+	struct clk *clk;
+	struct platform_device *pdev;
+	void (*write_qstate) (struct xgene_qmtm_qinfo *qinfo);
+	void (*read_qstate) (struct xgene_qmtm_qinfo *qinfo);
+	int (*set_qstate) (struct xgene_qmtm_qinfo *qinfo);
+	void (*clr_qstate) (struct xgene_qmtm_qinfo *qinfo);
+};
+
+/* QMTM Slave */
+enum xgene_slave {
+	SLAVE_ETH0,
+	SLAVE_ETH1,
+	SLAVE_ETH2,
+	SLAVE_ETH3,
+	SLAVE_XGE0,
+	SLAVE_XGE1,
+	SLAVE_XGE2,
+	SLAVE_XGE3,
+	SLAVE_METH,
+	SLAVE_PKTDMA,
+	SLAVE_CTX_QMTM0,
+	SLAVE_CTX_QMTM1,
+	SLAVE_CTX_QMTM2,
+	SLAVE_SEC,
+	SLAVE_CLASS,
+	SLAVE_MSLIM_QMTM0,
+	SLAVE_MSLIM_QMTM1,
+	SLAVE_MSLIM_QMTM2,
+	SLAVE_MSLIM_QMTM3,
+	SLAVE_PMPRO,
+	SLAVE_SMPRO_QMTM0,
+	SLAVE_SMPRO_QMTM1,
+	SLAVE_SMPRO_QMTM2,
+	SLAVE_SMPRO_QMTM3,
+	SLAVE_CPU_QMTM0,
+	SLAVE_CPU_QMTM1,
+	SLAVE_CPU_QMTM2,
+	SLAVE_CPU_QMTM3,
+	SLAVE_MAX,
+};
+
+/* QMTM Slave IDs */
+enum xgene_qmtm_slave_id {
+	QMTM_SLAVE_ID_ETH0,
+	QMTM_SLAVE_ID_ETH1,
+	QMTM_SLAVE_ID_RES2,
+	QMTM_SLAVE_ID_PKTDMA,
+	QMTM_SLAVE_ID_CTX,
+	QMTM_SLAVE_ID_SEC,
+	QMTM_SLAVE_ID_CLASS,
+	QMTM_SLAVE_ID_MSLIM,
+	QMTM_SLAVE_ID_RES8,
+	QMTM_SLAVE_ID_RES9,
+	QMTM_SLAVE_ID_RESA,
+	QMTM_SLAVE_ID_RESB,
+	QMTM_SLAVE_ID_RESC,
+	QMTM_SLAVE_ID_PMPRO,
+	QMTM_SLAVE_ID_SMPRO,
+	QMTM_SLAVE_ID_CPU,
+	QMTM_SLAVE_ID_MAX,
+};
+
+/* QMTM Free Pool Queue modes */
+enum xgene_qmtm_fp_mode {
+	MSG_NO_CHANGE,
+	ROUND_ADDR,
+	REDUCE_LEN,
+	CHANGE_LEN,
+};
+
+#define VIRT_TO_PHYS(x)	virt_to_phys(x)
+#define PHYS_TO_VIRT(x)	phys_to_virt(x)
+
+/* QMTM CSR read/write routine */
+void xgene_qmtm_wr32(struct xgene_qmtm *qmtm, u32 offset, u32 data);
+void xgene_qmtm_rd32(struct xgene_qmtm *qmtm, u32 offset, u32 *data);
+
+struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name);
+
+#endif /* __XGENE_QMTM_MAIN_H__ */
diff --git a/drivers/misc/xgene/qmtm/xgene_qmtm_storm.c b/drivers/misc/xgene/qmtm/xgene_qmtm_storm.c
new file mode 100644
index 0000000..3ec40bd
--- /dev/null
+++ b/drivers/misc/xgene/qmtm/xgene_qmtm_storm.c
@@ -0,0 +1,370 @@
+/**
+ * AppliedMicro X-Gene SOC Queue Manager/Traffic Manager driver
+ *
+ * Copyright (c) 2013 Applied Micro Circuits Corporation.
+ * Author: Ravi Patel <rapatel@apm.com>
+ *         Keyur Chudgar <kchudgar@apm.com>
+ *         Fushen Chen <fchen@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/slab.h>
+#include "xgene_qmtm_main.h"
+#include "xgene_qmtm_storm.h"
+
+#define CSR_QSTATE_ADDR		0x0000006c
+#define QNUMBER_WR(src)		(((u32)(src)) & 0x000003ff)
+
+#define CSR_QSTATE_WR_0_ADDR	0x00000070
+#define CSR_QSTATE_WR_1_ADDR	0x00000074
+#define CSR_QSTATE_WR_2_ADDR	0x00000078
+#define CSR_QSTATE_WR_3_ADDR	0x0000007c
+#define CSR_QSTATE_WR_4_ADDR	0x00000080
+
+static struct xgene_qmtm_sdev storm_sdev[SLAVE_MAX] = {
+	[SLAVE_ETH0] = {
+			.name = "SGMII0",
+			.compatible = "apm,xgene-qmtm-soc",
+			.slave = SLAVE_ETH0,
+			.qmtm_ip = QMTM1,
+			.slave_id = QMTM_SLAVE_ID_ETH0,
+			.wq_pbn_start = 0x00,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x20,
+			.fq_pbn_count = 0x08,
+			},
+	[SLAVE_ETH1] = {
+			.name = "SGMII1",
+			.compatible = "apm,xgene-qmtm-soc",
+			.slave = SLAVE_ETH1,
+			.qmtm_ip = QMTM1,
+			.slave_id = QMTM_SLAVE_ID_ETH0,
+			.wq_pbn_start = 0x08,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x28,
+			.fq_pbn_count = 0x08,
+			},
+	[SLAVE_ETH2] = {
+			.name = "SGMII2",
+			.compatible = "apm,xgene-qmtm-soc",
+			.slave = SLAVE_ETH2,
+			.qmtm_ip = QMTM1,
+			.slave_id = QMTM_SLAVE_ID_ETH1,
+			.wq_pbn_start = 0x00,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x20,
+			.fq_pbn_count = 0x08,
+			},
+	[SLAVE_ETH3] = {
+			.name = "SGMII3",
+			.compatible = "apm,xgene-qmtm-soc",
+			.slave = SLAVE_ETH3,
+			.qmtm_ip = QMTM1,
+			.slave_id = QMTM_SLAVE_ID_ETH1,
+			.wq_pbn_start = 0x08,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x28,
+			.fq_pbn_count = 0x08,
+			},
+	[SLAVE_XGE0] = {
+			.name = "SXGMII0",
+			.compatible = "apm,xgene-qmtm-xge0",
+			.slave = SLAVE_XGE0,
+			.qmtm_ip = QMTM0,
+			.slave_id = QMTM_SLAVE_ID_ETH0,
+			.wq_pbn_start = 0x00,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x20,
+			.fq_pbn_count = 0x08,
+			},
+	[SLAVE_XGE1] = {
+			.name = "SXGMII1",
+			.compatible = "apm,xgene-qmtm-xge0",
+			.slave = SLAVE_XGE1,
+			.qmtm_ip = QMTM0,
+			.slave_id = QMTM_SLAVE_ID_ETH1,
+			.wq_pbn_start = 0x00,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x20,
+			.fq_pbn_count = 0x08,
+			},
+	[SLAVE_XGE2] = {
+			.name = "SXGMII2",
+			.compatible = "apm,xgene-qmtm-xge2",
+			.slave = SLAVE_XGE2,
+			.qmtm_ip = QMTM2,
+			.slave_id = QMTM_SLAVE_ID_ETH0,
+			.wq_pbn_start = 0x00,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x20,
+			.fq_pbn_count = 0x08,
+			},
+	[SLAVE_XGE3] = {
+			.name = "SXGMII3",
+			.compatible = "apm,xgene-qmtm-xge2",
+			.slave = SLAVE_XGE3,
+			.qmtm_ip = QMTM2,
+			.slave_id = QMTM_SLAVE_ID_ETH1,
+			.wq_pbn_start = 0x00,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x20,
+			.fq_pbn_count = 0x08,
+			},
+	[SLAVE_METH] = {
+			.name = "RGMII",
+			.compatible = "apm,xgene-qmtm-lite",
+			.slave = SLAVE_METH,
+			.qmtm_ip = QMTM3,
+			.slave_id = QMTM_SLAVE_ID_ETH0,
+			.wq_pbn_start = 0x00,
+			.wq_pbn_count = 0x04,
+			.fq_pbn_start = 0x20,
+			.fq_pbn_count = 0x04,
+			},
+	[SLAVE_PKTDMA] = {
+			  .name = "PKTDMA",
+			  .compatible = "apm,xgene-qmtm-soc",
+			  .slave = SLAVE_PKTDMA,
+			  .qmtm_ip = QMTM1,
+			  .slave_id = QMTM_SLAVE_ID_PKTDMA,
+			  .wq_pbn_start = 0x00,
+			  .wq_pbn_count = 0x04,
+			  .fq_pbn_start = 0x20,
+			  .fq_pbn_count = 0x08,
+			  },
+	[SLAVE_CPU_QMTM0] = {
+			     .name = "CPU_QMTM0",
+			     .compatible = "apm,xgene-qmtm-xge0",
+			     .slave = SLAVE_CPU_QMTM0,
+			     .qmtm_ip = QMTM0,
+			     .slave_id = QMTM_SLAVE_ID_CPU,
+			     .wq_pbn_start = 0x00,
+			     .wq_pbn_count = 0x10,
+			     .fq_pbn_start = 0x20,
+			     .fq_pbn_count = 0x10,
+			     },
+	[SLAVE_CPU_QMTM1] = {
+			     .name = "CPU_QMTM1",
+			     .compatible = "apm,xgene-qmtm-soc",
+			     .slave = SLAVE_CPU_QMTM1,
+			     .qmtm_ip = QMTM1,
+			     .slave_id = QMTM_SLAVE_ID_CPU,
+			     .wq_pbn_start = 0x00,
+			     .wq_pbn_count = 0x20,
+			     .fq_pbn_start = 0x20,
+			     .fq_pbn_count = 0x20,
+			     },
+	[SLAVE_CPU_QMTM2] = {
+			     .name = "CPU_QMTM2",
+			     .compatible = "apm,xgene-qmtm-xge2",
+			     .slave = SLAVE_CPU_QMTM2,
+			     .qmtm_ip = QMTM2,
+			     .slave_id = QMTM_SLAVE_ID_CPU,
+			     .wq_pbn_start = 0x10,
+			     .wq_pbn_count = 0x10,
+			     .fq_pbn_start = 0x30,
+			     .fq_pbn_count = 0x10,
+			     },
+	[SLAVE_CPU_QMTM3] = {
+			     .name = "CPU_QMTM3",
+			     .compatible = "apm,xgene-qmtm-lite",
+			     .slave = SLAVE_CPU_QMTM3,
+			     .qmtm_ip = QMTM3,
+			     .slave_id = QMTM_SLAVE_ID_CPU,
+			     .wq_pbn_start = 0x00,
+			     .wq_pbn_count = 0x01,
+			     .fq_pbn_start = 0x20,
+			     .fq_pbn_count = 0x01,
+			     },
+};
+
+static void storm_qmtm_write_qstate(struct xgene_qmtm_qinfo *qinfo)
+{
+	struct xgene_qmtm *qmtm = qinfo->qmtm;
+	u16 queue_id = qinfo->queue_id;
+	struct storm_qmtm_csr_qstate *csr_qstate =
+	    &((union storm_qmtm_qstate *)qinfo->qstate)->csr;
+
+	/* write queue number */
+	queue_id = QNUMBER_WR(queue_id);
+	xgene_qmtm_wr32(qmtm, CSR_QSTATE_ADDR, (u32) queue_id);
+
+	/* write queue state */
+	xgene_qmtm_wr32(qmtm, CSR_QSTATE_WR_0_ADDR, csr_qstate->w0);
+	xgene_qmtm_wr32(qmtm, CSR_QSTATE_WR_1_ADDR, csr_qstate->w1);
+	xgene_qmtm_wr32(qmtm, CSR_QSTATE_WR_2_ADDR, csr_qstate->w2);
+	xgene_qmtm_wr32(qmtm, CSR_QSTATE_WR_3_ADDR, csr_qstate->w3);
+	xgene_qmtm_wr32(qmtm, CSR_QSTATE_WR_4_ADDR, csr_qstate->w4);
+}
+
+static void storm_qmtm_read_qstate(struct xgene_qmtm_qinfo *qinfo)
+{
+	struct xgene_qmtm *qmtm = qinfo->qmtm;
+	struct storm_qmtm_csr_qstate *qfabric = (struct storm_qmtm_csr_qstate *)
+	    (qmtm->fabric_vaddr + (qinfo->queue_id << 6));
+	struct storm_qmtm_csr_qstate *csr_qstate =
+	    &((union storm_qmtm_qstate *)qinfo->qstate)->csr;
+	struct storm_qmtm_pq_fp_qstate *pq_fp =
+	    &((union storm_qmtm_qstate *)(qinfo->qstate))->pq;
+
+	/* read queue state */
+	csr_qstate->w0 = readl(&qfabric->w0);
+	csr_qstate->w1 = readl(&qfabric->w1);
+	csr_qstate->w2 = readl(&qfabric->w2);
+	csr_qstate->w3 = readl(&qfabric->w3);
+	csr_qstate->w4 = readl(&qfabric->w4);
+	qinfo->nummsgs = pq_fp->nummsg;
+}
+
+static int storm_qmtm_set_qstate(struct xgene_qmtm_qinfo *qinfo)
+{
+	int rc = 0;
+	struct storm_qmtm_pq_fp_qstate *pq_fp =
+	    &((union storm_qmtm_qstate *)(qinfo->qstate))->pq;
+	u32 qsize = 0;
+	u8 qtype = qinfo->qtype;
+
+	if (qtype != QTYPE_PQ && qtype != QTYPE_FP) {
+		pr_err("Queue type %d is invalid\n", qinfo->qtype);
+		rc = -EINVAL;
+		goto _ret_set_qstate;
+	}
+
+	pq_fp->cfgqtype = qinfo->qtype;
+
+	/* if its a free queue, ask QMTM to set len to 0 when dealloc */
+	if (qtype == QTYPE_FP)
+		pq_fp->fp_mode = CHANGE_LEN;
+
+	if (qinfo->slave >= SLAVE_XGE0 && qinfo->slave <= SLAVE_XGE3) {
+		pq_fp->cfgRecombBuf = 1;
+		pq_fp->cfgRecombBufTimeoutL = 0xf;
+		pq_fp->cfgRecombBufTimeoutH = 0x7;
+	}
+
+	pq_fp->cfgselthrsh = 1;
+
+	/*  Allow the queue to accept message with non-zero LErr */
+	pq_fp->cfgacceptlerr = 1;
+	pq_fp->qcoherent = 1;
+
+	switch (qinfo->qsize) {
+	case QSIZE_512B:
+		qsize = 512;
+		break;
+	case QSIZE_2KB:
+		qsize = 2 * 1024;
+		break;
+	case QSIZE_16KB:
+		qsize = 16 * 1024;
+		break;
+	case QSIZE_64KB:
+		qsize = 64 * 1024;
+		break;
+	case QSIZE_512KB:
+		qsize = 512 * 1024;
+		break;
+	default:
+		pr_err("Queue size %d is invalid\n", qinfo->qsize);
+		rc = -EINVAL;
+		goto _ret_set_qstate;
+	}
+
+	qinfo->qdesc = kzalloc(sizeof(struct xgene_qmtm_qdesc), GFP_KERNEL);
+
+	if (qinfo->qdesc == NULL) {
+		rc = -ENOMEM;
+		goto _ret_set_qstate;
+	}
+
+	qinfo->qdesc->count = (qtype == QTYPE_PQ) ? (qsize / 32) : (qsize / 16);
+
+	if (qinfo->flags & XGENE_SLAVE_Q_ADDR_ALLOC) {
+		qinfo->qdesc->qvaddr = kzalloc(qsize, GFP_KERNEL);
+		if (qinfo->qdesc->qvaddr == NULL) {
+			kfree(qinfo->qdesc);
+			rc = -ENOMEM;
+			goto _ret_set_qstate;
+		}
+
+		qinfo->qpaddr = (u64) VIRT_TO_PHYS(qinfo->qdesc->qvaddr);
+	} else {
+		qinfo->qdesc->qvaddr = PHYS_TO_VIRT(qinfo->qpaddr);
+		memset(qinfo->qdesc->qvaddr, 0, qsize);
+	}
+
+	if ((qtype == QTYPE_PQ) && (qinfo->slave_id == QMTM_SLAVE_ID_CPU ||
+				    qinfo->slave_id == QMTM_SLAVE_ID_MSLIM)) {
+		u32 i;
+
+		for (i = 0; i < qinfo->qdesc->count; i++) {
+			u32 *slot = (u32 *)&qinfo->qdesc->msg32[i];
+
+			slot[EMPTY_SLOT_INDEX] = EMPTY_SLOT;
+		}
+	}
+
+	pq_fp->cfgstartaddr = (u32) (qinfo->qpaddr >> 8);
+	pq_fp->cfgqsize = qinfo->qsize;
+	storm_qmtm_write_qstate(qinfo);
+
+_ret_set_qstate:
+	return rc;
+}
+
+static void storm_qmtm_clr_qstate(struct xgene_qmtm_qinfo *qinfo)
+{
+	memset(qinfo->qstate, 0, sizeof(union storm_qmtm_qstate));
+	storm_qmtm_write_qstate(qinfo);
+
+	if (qinfo->flags & XGENE_SLAVE_Q_ADDR_ALLOC && qinfo->qdesc->qvaddr) {
+		kfree(qinfo->qdesc->qvaddr);
+		qinfo->qdesc->qvaddr = NULL;
+	}
+
+	kfree(qinfo->qdesc);
+}
+
+struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name)
+{
+	struct xgene_qmtm *qmtm = NULL;
+	struct xgene_qmtm_sdev *sdev = NULL;
+	struct device_node *np = NULL;
+	struct platform_device *platdev;
+	u8 slave;
+
+	for (slave = 0; slave < SLAVE_MAX; slave++) {
+		sdev = &storm_sdev[slave];
+		if (sdev->name && strcmp(name, sdev->name) == 0) {
+			np = of_find_compatible_node(NULL, NULL,
+				sdev->compatible);
+			break;
+		}
+	}
+
+	if (np == NULL)
+		return NULL;
+
+	platdev = of_find_device_by_node(np);
+	qmtm = platform_get_drvdata(platdev);
+
+	if (!qmtm->write_qstate) {
+		qmtm->write_qstate = storm_qmtm_write_qstate;
+		qmtm->read_qstate = storm_qmtm_read_qstate;
+		qmtm->set_qstate = storm_qmtm_set_qstate;
+		qmtm->clr_qstate = storm_qmtm_clr_qstate;
+	}
+
+	sdev->qmtm = qmtm;
+	sdev->idev = qmtm->idev;
+
+	return sdev;
+}
diff --git a/drivers/misc/xgene/qmtm/xgene_qmtm_storm.h b/drivers/misc/xgene/qmtm/xgene_qmtm_storm.h
new file mode 100644
index 0000000..e6b0828
--- /dev/null
+++ b/drivers/misc/xgene/qmtm/xgene_qmtm_storm.h
@@ -0,0 +1,139 @@
+/*
+ * AppliedMicro X-Gene SOC Queue Manager/Traffic Manager driver
+ *
+ * Copyright (c) 2013 Applied Micro Circuits Corporation.
+ * Author: Ravi Patel <rapatel@apm.com>
+ *         Keyur Chudgar <kchudgar@apm.com>
+ *         Fushen Chen <fchen@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __XGENE_QMTM_STORM_H__
+#define __XGENE_QMTM_STORM_H__
+
+/* QMTM Queue State */
+struct storm_qmtm_csr_qstate {
+	u32 w0;
+	u32 w1;
+	u32 w2;
+	u32 w3;
+	u32 w4;
+	u32 res;
+} __packed;
+
+/*
+ * Physical or free pool queue state (pq or fp)
+ */
+struct storm_qmtm_pq_fp_qstate {
+	/* word 0 31:0 */
+	u32 resize_qid:10;
+	u32 resize_start:1;
+	u32 resize_done:1;
+	u32 cfgtmvqen:1;	/* enable pq to belong to vq */
+	u32 cfgtmvq:10;		/* parent vq */
+	u32 cfgsaben:1;		/* enable SAB broadcasting */
+	u32 cpu_notify:8;	/* cfgcpusab */
+
+	/* word 1 63:32 */
+	u32 cfgnotifyqne:1;	/* enable queue not empty interrupt */
+	u32 nummsg:16;
+	u32 headptr:15;
+
+	/* word 2 95:64 */
+	u32 cfgcrid:1;
+	u32 rid:3;
+	u32 qcoherent:1;
+	u64 cfgstartaddr:34;	/* 27/7 split */
+
+	/* word 3 127:96 */
+	u32 vc_chanctl:2;
+	u32 vc_chanid:2;
+	u32 slot_pending:6;
+	u32 stashing:1;
+	u32 reserved_0:1;
+	u32 cfgacceptlerr:1;
+	u32 fp_mode:3;		/* free pool mode */
+	u32 cfgqsize:3;		/* queue size, see xgene_qmtm_qsize */
+	u32 qstatelock:1;
+	u32 cfgRecombBuf:1;
+	u32 cfgRecombBufTimeoutL:4;	/* 4/3 split */
+
+	/* word 4 159:128 */
+	u32 cfgRecombBufTimeoutH:3;	/* 4/3 split */
+	u32 cfgselthrsh:3;	/* associated threshold set */
+	u32 resv2:13;
+	u32 cfgqtype:2;		/* queue type, refer xgene_qmtm_qtype */
+	u32 resv1:11;
+} __packed;
+
+struct storm_qmtm_vq_qstate {
+	/* word 0 31:0 */
+	u32 nummsg:18;
+	u32 cfgsaben:1;		/* enable SAB broadcasting */
+	u32 cfgnotifyqne:1;	/* enable queue not empty interrupt */
+	u32 cfgcrid:1;		/* critical rid config */
+	u32 cpu_notify:8;
+	u32 rid:3;		/* curr. region id of queue fill level */
+
+	/* word 1 63:32 */
+	u32 q7selarb:2;
+	u32 q7txallowed:1;
+	u32 q7reqvld:1;
+	u32 q7_sel:10;		/* b45:36 */
+	u32 q6selarb:2;
+	u32 q6txallowed:1;
+	u32 q6reqvld:1;
+	u32 q6_sel:10;		/* b59:50 */
+	u32 q5selarb:2;
+	u32 q5txallowed:1;
+	u32 q5reqvld:1;
+
+	/* word 2 95:64 */
+	u32 q5_sel:10;		/* b73:64 */
+	u32 q4selarb:2;		/* b75:74 */
+	u32 q4txallowed:1;	/* b76 */
+	u32 q4reqvld:1;
+	u32 q4_sel:10;		/* b87:78 */
+	u32 q3selarb:2;
+	u32 q3txallowed:1;
+	u32 q3reqvld:1;		/* b91 */
+	u32 q3_sel_4b:4;	/* b95:92    split 6/4 */
+
+	/* word 3 127:96 */
+	u32 q3_sel_6b:6;	/* b101:96   split 6/4 */
+	u32 q2selarb:2;
+	u32 q2txallowed:1;
+	u32 q2reqvld:1;
+	u32 q2_sel:10;		/* q2_sel_3b */
+	u32 q1selarb:2;
+	u32 q1txallowed:1;
+	u32 q1reqvld:1;
+	u32 q1_sel_8b:8;	/* b127:120  split 2/8 */
+
+	/* word 4 159:128 */
+	u32 q1_sel_2b:2;	/* b129:128  split 2/8 */
+	u32 q0selarb:2;
+	u32 q0txallowed:1;
+	u32 q0reqvld:1;
+	u32 q0_sel:10;
+	u32 cfgselthrsh:3;	/* associated threshold set */
+	u32 cfgqtype:2;		/* queue type, refer xgene_qmtm_qtype */
+	u32 resv1:11;
+};
+
+union storm_qmtm_qstate {
+	struct storm_qmtm_csr_qstate csr;
+	struct storm_qmtm_pq_fp_qstate pq;
+	struct storm_qmtm_pq_fp_qstate fp;
+	struct storm_qmtm_vq_qstate vq;
+} __packed;
+
+#endif /* __XGENE_QMTM_STORM_H__ */
diff --git a/include/misc/xgene/xgene_qmtm.h b/include/misc/xgene/xgene_qmtm.h
new file mode 100644
index 0000000..d7b0930
--- /dev/null
+++ b/include/misc/xgene/xgene_qmtm.h
@@ -0,0 +1,277 @@
+/*
+ * AppliedMicro X-Gene SOC Queue Manager/Traffic Manager driver
+ *
+ * Copyright (c) 2013 Applied Micro Circuits Corporation.
+ * Author: Ravi Patel <rapatel@apm.com>
+ *         Keyur Chudgar <kchudgar@apm.com>
+ *         Fushen Chen <fchen@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __XGENE_QMTM_H__
+#define __XGENE_QMTM_H__
+
+/* QMTM Queue types */
+enum xgene_qmtm_qtype {
+	QTYPE_DISABLED,		/* Queue Type is un-configured or disabled */
+	QTYPE_PQ,		/* Queue Type is Physical Work Queue */
+	QTYPE_FP,		/* Queue Type is Free Pool Queue */
+	QTYPE_VQ,		/* Queue Type is Virtual Queue */
+	QTYPE_MAX,
+};
+
+/* QMTM Queue possible sizes */
+enum xgene_qmtm_qsize {
+	QSIZE_512B,
+	QSIZE_2KB,
+	QSIZE_16KB,
+	QSIZE_64KB,
+	QSIZE_512KB,
+	QSIZE_MAX,
+};
+
+/* QMTM Queue Access Method */
+enum xgene_qmtm_qaccess {
+	QACCESS_ALT,		/* Alternate enq/deq */
+	QACCESS_QMI,		/* Access using QMI interface */
+	QACCESS_MBOX,		/* Access using mailboxes */
+	QACCESS_MAX,
+};
+
+/* QMTM Data Length encoded as per QM message format */
+enum xgene_qmtm_data_len {
+	DATA_LEN_256B = 0x0100,
+	DATA_LEN_1K = 0x0400,
+	DATA_LEN_2K = 0x0800,
+	DATA_LEN_4K = 0x1000,
+	DATA_LEN_16K = 0x4000,
+};
+
+enum xgene_qmtm_mask_len {
+	MASK_LEN_256B = (DATA_LEN_256B - 1),
+	MASK_LEN_1K = (DATA_LEN_1K - 1),
+	MASK_LEN_2K = (DATA_LEN_2K - 1),
+	MASK_LEN_4K = (DATA_LEN_4K - 1),
+	MASK_LEN_16K = (DATA_LEN_16K - 1),
+};
+
+/* QMTM Buffer Length encoded as per QM message format */
+enum xgene_qmtm_buf_len {
+	BUF_LEN_256B = 0x7000,
+	BUF_LEN_1K = 0x6000,
+	BUF_LEN_2K = 0x5000,
+	BUF_LEN_4K = 0x4000,
+	BUF_LEN_16K = 0x0000,
+};
+
+/* QMTM messaging structures */
+/* 16 byte QMTM message format */
+struct xgene_qmtm_msg16 {
+	u32 UserInfo;
+
+	u32 FPQNum:12;
+	u32 Rv2:2;
+	u32 ELErr:2;
+	u32 LEI:2;
+	u32 NV:1;
+	u32 LL:1;
+	u32 PB:1;
+	u32 HB:1;
+	u32 Rv:1;
+	u32 IN:1;
+	u32 RType:4;
+	u32 LErr:3;
+	u32 HL:1;
+
+	u64 DataAddr:42;	/* 32/10 split */
+	u32 Rv6:6;
+	u32 BufDataLen:15;
+	u32 C:1;
+} __packed;
+
+/* Upper 16 byte portion of 32 byte of QMTM message format */
+struct xgene_qmtm_msg_up16 {
+	u64 H0Info_msb:48;
+	u32 TotDataLengthLinkListLSBs:12;
+	u32 Rv1:1;
+	u32 DR:1;
+	u32 Rv0:1;
+	u32 HR:1;
+
+	u64 H0Info_lsb:48;
+	u32 H0Enq_Num:12;
+	u32 H0FPSel:4;
+} __packed;
+
+/* 8 byte portion of QMTM extended (64B) message format */
+struct xgene_qmtm_msg_ext8 {
+	u64 NxtDataAddr:42;
+	u32 Rv2:2;
+	u32 NxtFPQNum:4;
+	u32 NxtBufDataLength:15;
+	u32 Rv1:1;
+} __packed;
+
+/* 8 byte Link list portion of QMTM extended (64B) message format */
+struct xgene_qmtm_msg_ll8 {
+	u64 NxtDataPtr:42;
+	u32 Rv2:2;
+	u32 NxtFPQNum:4;
+	u8 NxtLinkListength;
+	u8 TotDataLengthLinkListMSBs;
+} __packed;
+
+/* This structure represents 32 byte QMTM message format */
+struct xgene_qmtm_msg32 {
+	struct xgene_qmtm_msg16 msg16;
+	struct xgene_qmtm_msg_up16 msgup16;
+} __packed;
+
+ /* 32 byte of QMTM extended (64B) message format */
+struct xgene_qmtm_msg_ext32 {
+	struct xgene_qmtm_msg_ext8 msg8_2;
+	struct xgene_qmtm_msg_ext8 msg8_1;
+	union {
+		struct xgene_qmtm_msg_ext8 msg8_4;
+		struct xgene_qmtm_msg_ll8 msg8_ll;
+	};
+	struct xgene_qmtm_msg_ext8 msg8_3;
+} __packed;
+
+/* 64 byte QMTM message format */
+struct xgene_qmtm_msg64 {
+	struct xgene_qmtm_msg32 msg32_1;
+	struct xgene_qmtm_msg_ext32 msg32_2;
+} __packed;
+
+/* Empty Slot Soft Signature */
+#define EMPTY_SLOT_INDEX	7
+#define EMPTY_SLOT		0x22222222
+
+/* Destination QM, 2 MSb in work queue, dstqid */
+#define QMTM_QUEUE_ID(qm, qid)	(((u16)(qm) << 10) | qid)
+
+/* QMTM Slave Device Information */
+struct xgene_qmtm_sdev {
+	u8 qmtm_ip;
+	u8 slave;
+	u8 wq_pbn_start;
+	u8 wq_pbn_count;
+	u8 fq_pbn_start;
+	u8 fq_pbn_count;
+	u16 slave_id;		/* slave id see xgene_qmtm_slave_id */
+	u32 wq_pbn_pool;	/* Bit mask indicates in use WQ PBN */
+	u32 fq_pbn_pool;	/* Bit mask indicates in use FP PBN */
+	char *name;
+	char *compatible;
+	struct xgene_qmtm *qmtm;
+	struct xgene_qmtm_sdev *idev;
+};
+
+/* QMTM Queue Information structure */
+/* Per queue descriptor */
+struct xgene_qmtm_qdesc {
+	u16 qhead;
+	u16 qtail;
+	u16 count;
+	u16 irq;
+	void *command;
+	union {
+		void *qvaddr;
+		struct xgene_qmtm_msg16 *msg16;
+		struct xgene_qmtm_msg32 *msg32;
+	};
+};
+
+/* Per queue state database */
+struct xgene_qmtm_qinfo {
+	u8 slave;
+	u8 qtype;
+	u8 qsize;
+	u8 qaccess;
+	u8 flags;
+	u8 qmtm_ip;
+	u8 slave_id;
+	u8 pbn;
+	u16 queue_id;
+	u16 nummsgs;
+	u32 pbm_state;
+	u64 qpaddr;
+	void *qfabric;
+	u32 qstate[6];
+	struct xgene_qmtm_qdesc *qdesc;
+	struct xgene_qmtm_sdev *sdev;
+	struct xgene_qmtm *qmtm;
+};
+
+/* QMTM Queue Information flags */
+#define XGENE_SLAVE_PB_CONFIGURE	0x01
+#define XGENE_SLAVE_Q_ADDR_ALLOC	0x02
+#define XGENE_SLAVE_DEFAULT_FLAGS	(XGENE_SLAVE_PB_CONFIGURE | \
+	XGENE_SLAVE_Q_ADDR_ALLOC)
+
+static inline u16 xgene_qmtm_encode_bufdatalen(u32 len)
+{
+	if (len <= DATA_LEN_256B)
+		return BUF_LEN_256B | (len & MASK_LEN_256B);
+	else if (len <= DATA_LEN_1K)
+		return BUF_LEN_1K | (len & MASK_LEN_1K);
+	else if (len <= DATA_LEN_2K)
+		return BUF_LEN_2K | (len & MASK_LEN_2K);
+	else if (len <= DATA_LEN_4K)
+		return BUF_LEN_4K | (len & MASK_LEN_4K);
+	else if (len < DATA_LEN_16K)
+		return BUF_LEN_16K | (len & MASK_LEN_16K);
+	else
+		return BUF_LEN_16K;
+}
+
+static inline u16 xgene_qmtm_encode_datalen(u32 len)
+{
+	return len & MASK_LEN_16K;
+}
+
+static inline u32 xgene_qmtm_decode_datalen(u16 bufdatalen)
+{
+	switch (bufdatalen & BUF_LEN_256B) {
+	case BUF_LEN_256B:
+		return bufdatalen & MASK_LEN_256B ? : DATA_LEN_256B;
+	case BUF_LEN_1K:
+		return bufdatalen & MASK_LEN_1K ? : DATA_LEN_1K;
+	case BUF_LEN_2K:
+		return bufdatalen & MASK_LEN_2K ? : DATA_LEN_2K;
+	case BUF_LEN_4K:
+		return bufdatalen & MASK_LEN_4K ? : DATA_LEN_4K;
+	default:
+		return bufdatalen & MASK_LEN_16K ? : DATA_LEN_16K;
+	};
+}
+
+struct xgene_qmtm_sdev *xgene_qmtm_get_sdev(char *name);
+
+int xgene_qmtm_set_qinfo(struct xgene_qmtm_qinfo *qinfo);
+
+void xgene_qmtm_clr_qinfo(struct xgene_qmtm_qinfo *qinfo);
+
+void xgene_qmtm_read_qstate(struct xgene_qmtm_qinfo *qinfo);
+
+void xgene_qmtm_fp_dealloc_msg(struct xgene_qmtm_qdesc *qdesc,
+			       struct xgene_qmtm_msg16 *msg);
+
+void xgene_qmtm_enqueue_msg(struct xgene_qmtm_qdesc *qdesc,
+			    struct xgene_qmtm_msg64 *msg);
+
+int xgene_qmtm_dequeue_msg(struct xgene_qmtm_qdesc *qdesc,
+			   struct xgene_qmtm_msg64 *msg);
+
+int xgene_qmtm_intr_coalesce(struct xgene_qmtm_qinfo *qinfo, u8 tap);
+
+#endif /* __XGENE_QMTM_H__ */
-- 
1.7.9.5

[PATCH V2 3/4] arm64: boot: dts: Add DTS entries for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

This patch adds APM X-Gene SoC Queue Manager/Traffic Manager DTS entries.

Signed-off-by: Ravi Patel <rapatel@apm.com>
Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
---
 arch/arm64/boot/dts/apm-storm.dtsi |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/arch/arm64/boot/dts/apm-storm.dtsi b/arch/arm64/boot/dts/apm-storm.dtsi
index d37d736..f64f946 100644
--- a/arch/arm64/boot/dts/apm-storm.dtsi
+++ b/arch/arm64/boot/dts/apm-storm.dtsi
@@ -152,6 +152,11 @@
 				reg = <0x0 0x1703C000 0x0 0x1000>;
 				reg-names = "csr-reg";
 				clock-output-names = "qmlclk";
+				status = "ok";
+				csr-offset = <0x0>;
+				csr-mask = <0x3>;
+				enable-offset = <0x8>;
+				enable-mask = <0x3>;
 			};
 
 			ethclk: ethclk {
@@ -187,5 +192,17 @@
 			interrupt-parent = <&gic>;
 			interrupts = <0x0 0x4c 0x4>;
 		};
+
+		qmlite: qmtm at 17030000 {
+			compatible = "apm,xgene-qmtm-lite";
+			reg = <0x0 0x17030000 0x0 0x10000>,
+			      <0x0 0x10000000 0x0 0x400000>;
+			slave-name = "CPU_QMTM3";
+			interrupts = <0x0 0x40 0x4>,
+				     <0x0 0x3c 0x4>;
+			status = "ok";
+			#clock-cells = <1>;
+			clocks = <&qmlclk 0>;
+		};
 	};
 };
-- 
1.7.9.5

[PATCH V2 4/4] misc: xgene: Add error handling for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

This patch adds support for error handling in APM X-Gene SoC Queue
  Manager/Traffic Manager.

Signed-off-by: Ravi Patel <rapatel@apm.com>
Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
---
 drivers/misc/xgene/qmtm/Makefile           |    2 +-
 drivers/misc/xgene/qmtm/xgene_qmtm_error.c |  283 ++++++++++++++++++++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_main.c  |    6 +-
 drivers/misc/xgene/qmtm/xgene_qmtm_main.h  |    4 +
 4 files changed, 293 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_error.c

diff --git a/drivers/misc/xgene/qmtm/Makefile b/drivers/misc/xgene/qmtm/Makefile
index 68c2a86..8448253 100644
--- a/drivers/misc/xgene/qmtm/Makefile
+++ b/drivers/misc/xgene/qmtm/Makefile
@@ -4,4 +4,4 @@
 
 obj-$(CONFIG_XGENE_QMTM) += xgene-qmtm.o
 
-xgene-qmtm-objs := xgene_qmtm_main.o xgene_qmtm_storm.o
+xgene-qmtm-objs := xgene_qmtm_main.o xgene_qmtm_storm.o xgene_qmtm_error.o
diff --git a/drivers/misc/xgene/qmtm/xgene_qmtm_error.c b/drivers/misc/xgene/qmtm/xgene_qmtm_error.c
new file mode 100644
index 0000000..5b4f4a9
--- /dev/null
+++ b/drivers/misc/xgene/qmtm/xgene_qmtm_error.c
@@ -0,0 +1,283 @@
+/*
+ * AppliedMicro X-Gene SOC Queue Manager/Traffic Manager driver
+ *
+ * Copyright (c) 2013 Applied Micro Circuits Corporation.
+ * Author: Ravi Patel <rapatel@apm.com>
+ *         Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/interrupt.h>
+#include "xgene_qmtm_main.h"
+
+#define QM_INTERRUPT_ADDR		0x00000124
+#define QM_INTERRUPTMASK_ADDR		0x00000128
+#define QUEUE_NOT_EMPTYMASK_MASK	0x80000000
+#define ACR_FIFO_CRITICALMASK_MASK	0x00000008
+#define QPCORE_ACR_ERRORMASK_MASK	0x00000004
+#define DEQ_AXI_ERRORMASK_MASK		0x00000002
+#define PBM_DEC_ERRORMASK_MASK		0x00000001
+
+#define CSR_PBM_ERRINF_ADDR		0x00000134
+#define  ACR_QID_RD(src)		(((src) & 0x00ffc000)>>14)
+#define  QID_RD(src)			(((src) & 0x000003ff))
+
+#define CSR_MSGRD_ERRINF_ADDR		0x00000138
+
+#define CSR_ERRQ_ADDR			0x00000218
+#define  UNEXPECTED_EN_SET(dst, src) \
+	(((dst) & ~0x80000000) | (((u32)(src)<<31) & 0x80000000))
+#define  UNEXPECTED_QID_SET(dst, src) \
+	(((dst) & ~0x03ff0000) | (((u32)(src)<<16) & 0x03ff0000))
+#define  EXPECTED_EN_SET(dst, src) \
+	(((dst) & ~0x00008000) | (((u32)(src)<<15) & 0x00008000))
+#define  EXPECTED_QID_SET(dst, src) \
+	(((dst) & ~0x000003ff) | (((u32)(src)) & 0x000003ff))
+
+/* QMTM Error Reporting */
+enum xgene_qmtm_lerr {
+	QMTM_NO_ERR,
+	QMTM_MSG_SIZE_ERR,
+	QMTM_HOP_COUNT_ERR,
+	QMTM_VQ_ENQ_ERR,
+	QMTM_DISABLEDQ_ENQ_ERR,
+	QMTM_Q_OVERFLOW_ERR,
+	QMTM_ENQUEUE_ERR,
+	QMTM_DEQUEUE_ERR,
+};
+
+/* Parse Error Message received on Error Queue */
+static void xgene_qmtm_error_msg(struct xgene_qmtm_qinfo *qinfo,
+				 struct xgene_qmtm_msg32 *msg)
+{
+	struct xgene_qmtm_msg16 *msg16 = &msg->msg16;
+	struct device *dev = &qinfo->qmtm->pdev->dev;
+	u16 queue_id = qinfo->queue_id;
+
+	dev_err(dev, "Error ELErr[%d] LErr[%d] for Qid[%d]\n",
+		msg16->ELErr, msg16->LErr, queue_id);
+
+	switch (msg16->LErr) {
+	case QMTM_MSG_SIZE_ERR:
+		dev_err(dev, "Msg Size Error for Enqueue on Queue %d\n",
+			queue_id);
+		break;
+	case QMTM_HOP_COUNT_ERR:
+		dev_err(dev, "Hop count error, hop count of 3 for Queue %d\n",
+			queue_id);
+		break;
+	case QMTM_VQ_ENQ_ERR:
+		dev_err(dev, "Enqueue on Virtual Queue %d\n", queue_id);
+		break;
+	case QMTM_DISABLEDQ_ENQ_ERR:
+		dev_err(dev, "Enqueue on disabled Queue %d\n", queue_id);
+		break;
+	case QMTM_Q_OVERFLOW_ERR:
+		dev_err(dev, "Queue %d overflow, message sent to Error Queue\n",
+			queue_id);
+		break;
+	case QMTM_ENQUEUE_ERR:
+		dev_err(dev, "Enqueue Queue\n");
+		break;
+	case QMTM_DEQUEUE_ERR:
+		dev_err(dev, "Dequeue Queue\n");
+		break;
+	default:
+		dev_err(dev, "Unknown Error\n");
+		break;
+	}
+}
+
+static void xgene_qmtm_error(struct xgene_qmtm *qmtm)
+{
+	struct device *dev = &qmtm->pdev->dev;
+	struct xgene_qmtm_qinfo qinfo;
+	u32 status;
+	u32 pbm_err;
+	u32 msgrd_err;
+
+	memset(&qinfo, 0, sizeof(qinfo));
+	qinfo.qmtm = qmtm;
+
+	xgene_qmtm_rd32(qmtm, QM_INTERRUPT_ADDR, &status);
+	dev_err(dev, "error interrupt status 0x%08X\n", status);
+
+	xgene_qmtm_rd32(qmtm, CSR_PBM_ERRINF_ADDR, &pbm_err);
+	dev_err(dev, "CSR PBM ERRINF (0x%X) value 0x%08X\n",
+		CSR_PBM_ERRINF_ADDR, pbm_err);
+
+	xgene_qmtm_rd32(qmtm, CSR_MSGRD_ERRINF_ADDR, &msgrd_err);
+	dev_err(dev, "CSR MSGRD ERRINF (0x%X) value 0x%08X\n",
+		CSR_MSGRD_ERRINF_ADDR, msgrd_err);
+
+	qinfo.queue_id = QID_RD(msgrd_err);
+	dev_err(dev, "DEQ QID %d\n", qinfo.queue_id);
+	xgene_qmtm_read_qstate(&qinfo);
+	print_hex_dump(KERN_ERR, "DEQSTATE ", DUMP_PREFIX_ADDRESS, 16, 4,
+		       qinfo.qstate, sizeof(qinfo.qstate), 1);
+
+	qinfo.queue_id = ACR_QID_RD(msgrd_err);
+	dev_err(dev, "ENQ QID %d\n", qinfo.queue_id);
+	xgene_qmtm_read_qstate(&qinfo);
+	print_hex_dump(KERN_INFO, "ENQSTATE ", DUMP_PREFIX_ADDRESS, 16, 4,
+		       qinfo.qstate, sizeof(qinfo.qstate), 1);
+
+	xgene_qmtm_wr32(qmtm, QM_INTERRUPT_ADDR, status);
+}
+
+static irqreturn_t xgene_qmtm_error_intr(int irq, void *qdev)
+{
+	xgene_qmtm_error((struct xgene_qmtm *)qdev);
+
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t xgene_qmtm_error_queue_intr(int irq, void *qdev)
+{
+	struct xgene_qmtm_msg64 msg;
+	struct xgene_qmtm_qinfo *qinfo = (struct xgene_qmtm_qinfo *)qdev;
+	struct xgene_qmtm *qmtm = qinfo->qmtm;
+	struct device *dev = &qmtm->pdev->dev;
+	u16 queue_id = qinfo->queue_id;
+	u8 qmtm_ip = qinfo->qmtm_ip;
+	int rc;
+
+	rc = xgene_qmtm_dequeue_msg(qinfo->qdesc, &msg);
+	if (rc < 0) {
+		/* Return if invalid interrupt */
+		dev_err(dev, "QMTM%d QID %d PBN %d IRQ %d spurious\n",
+			qmtm_ip, queue_id, qinfo->pbn, irq);
+		return IRQ_HANDLED;
+	}
+
+	xgene_qmtm_error(qmtm);
+	dev_err(dev, "QMTM%d Error: QID %d\n", qmtm_ip, queue_id);
+	print_hex_dump(KERN_INFO, "Err q MSG: ", DUMP_PREFIX_ADDRESS,
+		       16, 4, &msg, msg.msg32_1.msg16.NV ? 64 : 32, 1);
+	xgene_qmtm_error_msg(qinfo, &msg.msg32_1);
+
+	return IRQ_HANDLED;
+}
+
+int xgene_qmtm_enable_error(struct xgene_qmtm *qmtm)
+{
+	struct device *dev = &qmtm->pdev->dev;
+	struct xgene_qmtm_qinfo qinfo;
+	int rc = 0;
+	u32 val;
+	u16 irq = platform_get_irq(qmtm->pdev, 0);
+	u8 qmtm_ip = qmtm->qmtm_ip;
+
+	if (irq) {
+		u32 mask;
+
+		memset(qmtm->error_irq_s, 0, sizeof(qmtm->error_irq_s));
+		snprintf(qmtm->error_irq_s, sizeof(qmtm->error_irq_s),
+			 "%s_Err", qmtm->idev->name);
+
+		rc = devm_request_irq(dev, irq, xgene_qmtm_error_intr, 0,
+				      qmtm->error_irq_s, qmtm);
+		if (rc < 0) {
+			dev_err(dev, "request_irq %d failed for %s (%d)\n",
+				irq, qmtm->error_irq_s, rc);
+			return rc;
+		}
+
+		qmtm->error_irq = irq;
+
+		/* Enable QM hardware interrupts */
+		mask = ~(u32) (PBM_DEC_ERRORMASK_MASK
+			       | ACR_FIFO_CRITICALMASK_MASK
+			       | QUEUE_NOT_EMPTYMASK_MASK
+			       | DEQ_AXI_ERRORMASK_MASK
+			       | QPCORE_ACR_ERRORMASK_MASK);
+		xgene_qmtm_wr32(qmtm, QM_INTERRUPTMASK_ADDR, mask);
+	}
+
+	if (qmtm_ip == QMTM3)
+		return rc;
+
+	memset(&qinfo, 0, sizeof(qinfo));
+	qinfo.sdev = qmtm->idev;
+	qinfo.qaccess = QACCESS_ALT;
+	qinfo.qtype = QTYPE_PQ;
+	qinfo.qsize = QSIZE_2KB;
+	qinfo.flags = XGENE_SLAVE_DEFAULT_FLAGS;
+
+	/* create error queue */
+	rc = xgene_qmtm_set_qinfo(&qinfo);
+	if (rc < 0) {
+		dev_err(dev, "QMTM %d unable to configure error queue\n",
+			qmtm_ip);
+		return rc;
+	}
+
+	qmtm->error_qinfo = qmtm->qinfo[qinfo.queue_id];
+	memset(qmtm->error_queue_irq_s, 0, sizeof(qmtm->error_queue_irq_s));
+	snprintf(qmtm->error_queue_irq_s, sizeof(qmtm->error_queue_irq_s),
+		 "%s_ErQ", qmtm->idev->name);
+
+	rc = devm_request_irq(dev, qinfo.qdesc->irq,
+			      xgene_qmtm_error_queue_intr,
+			      0, qmtm->error_queue_irq_s, qmtm->error_qinfo);
+	if (rc < 0) {
+		dev_err(dev, "request_irq %d failed for %s (%d)\n",
+			qinfo.qdesc->irq, qmtm->error_queue_irq_s, rc);
+		xgene_qmtm_clr_qinfo(&qinfo);
+		qmtm->error_qinfo = NULL;
+		return rc;
+	}
+
+	val = 0;
+	val = UNEXPECTED_EN_SET(val, 1);
+	val = UNEXPECTED_QID_SET(val, qinfo.queue_id);
+	val = EXPECTED_EN_SET(val, 1);
+	val = EXPECTED_QID_SET(val, qinfo.queue_id);
+	xgene_qmtm_wr32(qmtm, CSR_ERRQ_ADDR, val);
+
+	return rc;
+}
+
+void xgene_qmtm_disable_error(struct xgene_qmtm *qmtm)
+{
+	struct xgene_qmtm_qinfo *error_qinfo = qmtm->error_qinfo;
+	struct device *dev = &qmtm->pdev->dev;
+
+	/* Free QMTM Error IRQ */
+	if (qmtm->error_irq) {
+		u32 mask;
+
+		/* Disable QM hardware interrupts */
+		mask = PBM_DEC_ERRORMASK_MASK
+		    | ACR_FIFO_CRITICALMASK_MASK
+		    | QUEUE_NOT_EMPTYMASK_MASK
+		    | DEQ_AXI_ERRORMASK_MASK | QPCORE_ACR_ERRORMASK_MASK;
+		xgene_qmtm_wr32(qmtm, QM_INTERRUPTMASK_ADDR, mask);
+		devm_free_irq(dev, qmtm->error_irq, qmtm);
+		qmtm->error_irq = 0;
+	}
+
+	if (error_qinfo) {
+		struct xgene_qmtm_qinfo qinfo;
+
+		/* Free QMTM Error Queue IRQ */
+		devm_free_irq(dev, error_qinfo->qdesc->irq, error_qinfo);
+
+		/* Delete error queue */
+		qinfo.sdev = error_qinfo->qmtm->idev;
+		qinfo.queue_id = error_qinfo->queue_id;
+		xgene_qmtm_clr_qinfo(&qinfo);
+		qmtm->error_qinfo = NULL;
+
+		/* Unassign error queue */
+		xgene_qmtm_wr32(qmtm, CSR_ERRQ_ADDR, 0);
+	}
+}
diff --git a/drivers/misc/xgene/qmtm/xgene_qmtm_main.c b/drivers/misc/xgene/qmtm/xgene_qmtm_main.c
index cda63e0..833ff82 100644
--- a/drivers/misc/xgene/qmtm/xgene_qmtm_main.c
+++ b/drivers/misc/xgene/qmtm/xgene_qmtm_main.c
@@ -588,13 +588,17 @@ static int xgene_qmtm_enable(struct xgene_qmtm *qmtm)
 		qmtm->write_qstate(&qinfo);
 	}
 
-	return rc;
+	/* Enable error reporting */
+	return xgene_qmtm_enable_error(qmtm);
 }
 
 static int xgene_qmtm_disable(struct xgene_qmtm *qmtm)
 {
 	u32 queue_id;
 
+	/* Disable error reporting */
+	xgene_qmtm_disable_error(qmtm);
+
 	for (queue_id = 0; queue_id < qmtm->max_queues; queue_id++) {
 		if (qmtm->qinfo[queue_id]) {
 			dev_err(&qmtm->pdev->dev,
diff --git a/drivers/misc/xgene/qmtm/xgene_qmtm_main.h b/drivers/misc/xgene/qmtm/xgene_qmtm_main.h
index d937462..2a60225 100644
--- a/drivers/misc/xgene/qmtm/xgene_qmtm_main.h
+++ b/drivers/misc/xgene/qmtm/xgene_qmtm_main.h
@@ -129,6 +129,10 @@ enum xgene_qmtm_fp_mode {
 void xgene_qmtm_wr32(struct xgene_qmtm *qmtm, u32 offset, u32 data);
 void xgene_qmtm_rd32(struct xgene_qmtm *qmtm, u32 offset, u32 *data);
 
+/* QMTM Error handling */
+int xgene_qmtm_enable_error(struct xgene_qmtm *qmtm);
+void xgene_qmtm_disable_error(struct xgene_qmtm *qmtm);
+
 struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name);
 
 #endif /* __XGENE_QMTM_MAIN_H__ */
-- 
1.7.9.5

[PATCH V2 1/4] Documentation: Add documentation for APM X-Gene SoC Queue Manager/Traffic Manager DTS binding

[Arnd Bergmann]

On Saturday 21 December 2013, Ravi Patel wrote:
>  create mode 100644 Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
> 
> diff --git a/Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt b/Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
> new file mode 100644
> index 0000000..c3fcbd2
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
> @@ -0,0 +1,51 @@
> +* APM X-Gene SoC Queue Manager/Traffic Manager nodes
> +
> +QMTM nodes are defined to describe on-chip Queue Managers in APM X-Gene SoC.
> +APM X-Gene SoC Ethernet, PktDMA (XOR Engine), and Security Engine subsystems
> +communicate with a central Queue Manager using messages which include
> +information about the work to be performed and the location of the associated
> +data buffers. There are multiple instances of QMTM. Each QMTM instance has its
> +own node. Its corresponding clock nodes are shown below.
> +
> +Required properties:
> +- compatible		: Shall be "apm,xgene-qmtm-lite" for QMLite instance
> +- reg			: First memory resource shall be the QMTM CSR memory
> +			  resource.
> +			  Second memory resource shall be the QMTM IO-Fabric
> +			  memory resource.
> +- slave-name		: Shall be "CPU_QMTM3" which is receiver for ingress
> +			  work messages for the QMTM. Here receiver is CPU.

What is that string used for? Who decides what strings are valid in the
future? Why does the device even need to know the name of the receiver,
shouldn't that have a separate node?

> +- interrupts		: First interrupt resource shall be the QMTM Error
> +			  interrupt.
> +			  Remaining interrupt resources shall be the Ingress
> +			  work message interrupt mapping for receiver,
> +			  receiving work messages for the QMTM.
> +- clocks		: Reference to the clock entry.
> +
> +Optional properties:
> +- status		: Shall be "ok" if enabled or "disabled" if disabled.
> +			  Default is "ok".
> +
> +Example:
> +		qmlclk: qmlclk {
> +			compatible = "apm,xgene-device-clock";
> +			#clock-cells = <1>;
> +			clock-names = "qmlclk";
> +			status = "ok";
> +			csr-offset = <0x0>;
> +			csr-mask = <0x3>;
> +			enable-offset = <0x8>;
> +			enable-mask = <0x3>;
> +		};

This looks like an invalid node, which makes a bad example. Why do you 
have a "clock-names" entry here but no "clocks"?

> +		qmlite: qmtm at 17030000 {
> +			compatible = "apm,xgene-qmtm-lite";
> +			reg = <0x0 0x17030000 0x0 0x10000>,
> +			      <0x0 0x10000000 0x0 0x400000>;
> +			slave-name = "CPU_QMTM3";
> +			interrupts = <0x0 0x40 0x4>,
> +				     <0x0 0x3c 0x4>;
> +			status = "ok";
> +			#clock-cells = <1>;
> +			clocks = <&qmlclk 0>;
> +		};

#clock-cells is neither a required nor an optional property for
"apm,xgene-qmtm-lite" according to the documentation above. Please
make the example match the specification.

	Arnd

[PATCH V2 2/4] misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Saturday 21 December 2013, Ravi Patel wrote:
> +/* CSR Address Macros */
> +#define CSR_QM_CONFIG_ADDR	0x00000004
> +#define  QM_ENABLE_WR(src)	(((u32)(src)<<31) & 0x80000000)
> +
> +#define CSR_PBM_ADDR		0x00000008
> +#define  OVERWRITE_WR(src)	(((u32)(src)<<31) & 0x80000000)
> +#define  SLVID_PBN_WR(src)	(((u32)(src)) & 0x000003ff)
> +#define  SLAVE_ID_SHIFT		6
> +
> +#define CSR_PBM_BUF_WR_ADDR	0x0000000c
> +#define CSR_PBM_BUF_RD_ADDR	0x00000010
> +#define  PB_SIZE_WR(src)	(((u32)(src)<<31) & 0x80000000)
> +#define  PREFETCH_BUF_EN_SET(dst, src) \
> +	(((dst) & ~0x00200000) | (((u32)(src)<<21) & 0x00200000))
> +#define  IS_FREE_POOL_SET(dst, src) \
> +	(((dst) & ~0x00100000) | (((u32)(src)<<20) & 0x00100000))
> +#define  TLVQ_SET(dst, src) \
> +	(((dst) & ~0x00080000) | (((u32)(src)<<19) & 0x00080000))
> +#define  CORRESPONDING_QNUM_SET(dst, src) \
> +	(((dst) & ~0x0007fe00) | (((u32)(src)<<9) & 0x0007fe00))

These macros are rather unreadable and unhelpful. Can't you just
open-code them in the users?

> +#define CSR_THRESHOLD0_SET1_ADDR	0x00000030
> +#define CSR_THRESHOLD1_SET1_ADDR	0x00000034
> +#define CSR_HYSTERESIS_ADDR		0x00000068
> +#define CSR_QM_MBOX_NE_INT_MODE_ADDR	0x0000017c
> +#define CSR_QMLITE_PBN_MAP_0_ADDR	0x00000228

What about all the other registers inbetween?

> +void xgene_qmtm_wr32(struct xgene_qmtm *qmtm, u32 offset, u32 data)
> +{
> +	void *addr = (u8 *)qmtm->csr_vaddr + offset;
> +	writel(data, addr);
> +}

Type mismatch: writel requires a "void __iomem*" pointer. Please check your
driver using "sparse" to find bugs like this.

> +/**
> + * xgene_qmtm_set_qinfo - Create and configure a queue
> + * @sdev:	Slave context
> + * @qtype:	Queue type (P_QUEUE or FP_QUEUE)
> + * @qsize:	Queue size see xgene_qmtm_qsize
> + * @qaccess:	Queue access method see xgene_qmtm_qaccess
> + * @flags:	Queue Information flags
> + * @qpaddr:	If Desire Queue Physical Address to use
> + *
> + * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
> + * and Security Engine subsystems to create and configure a queue.
> + *
> + * Return:	0 on Success or -1 on Failure

-1 on failure is not an appropriate return code. Please use standard
error numbers such as -EINVAL.

> +	rc = of_property_read_string(pdev->dev.of_node, "slave-name", &name);
> +	if (rc < 0) {
> +		dev_err(&pdev->dev, "Failed to get QMTM Ingress slave-name\n");
> +		return rc;
> +	}
> +
> +	sdev = xgene_qmtm_get_sdev((char *)name);
> +	if (sdev == NULL) {
> +		dev_err(&pdev->dev, "Ingress Slave error\n");
> +		return -ENODEV;
> +	}

Why are you referring to another device by a string? Shouldn't that be a phandle?

> +static struct of_device_id xgene_qmtm_match[] = {
> +	{.compatible = "apm,xgene-qmtm-xge0",},
> +	{.compatible = "apm,xgene-qmtm-soc",},
> +	{.compatible = "apm,xgene-qmtm-xge2",},
> +	{.compatible = "apm,xgene-qmtm-lite",},
> +	{},
> +};
> +
> +MODULE_DEVICE_TABLE(of, xgene_qmtm_match);

You don't seem to differentiate between the four different strings. Why can't
they just all be compatible to "apm,xgene-qmtm"?

> +
> +#define VIRT_TO_PHYS(x)	virt_to_phys(x)
> +#define PHYS_TO_VIRT(x)	phys_to_virt(x)

These don't belong here, use the standard functions. Also, it seems unlikely that a
device driver actually wants physical addresses. Are these used for DMA?

> +static struct xgene_qmtm_sdev storm_sdev[SLAVE_MAX] = {
> +	[SLAVE_ETH0] = {
> +			.name = "SGMII0",
> +			.compatible = "apm,xgene-qmtm-soc",
> +			.slave = SLAVE_ETH0,
> +			.qmtm_ip = QMTM1,
> +			.slave_id = QMTM_SLAVE_ID_ETH0,
> +			.wq_pbn_start = 0x00,
> +			.wq_pbn_count = 0x08,
> +			.fq_pbn_start = 0x20,
> +			.fq_pbn_count = 0x08,
> +			},

This table is backwards: the qmtm driver provides services to client drivers,
it should have zero knowledge about what the clients are.

Remove this array here and put the data into properties of the client
drivers.

> +	if (qinfo->flags & XGENE_SLAVE_Q_ADDR_ALLOC) {
> +		qinfo->qdesc->qvaddr = kzalloc(qsize, GFP_KERNEL);
> +		if (qinfo->qdesc->qvaddr == NULL) {
> +			kfree(qinfo->qdesc);
> +			rc = -ENOMEM;
> +			goto _ret_set_qstate;
> +		}
> +
> +		qinfo->qpaddr = (u64) VIRT_TO_PHYS(qinfo->qdesc->qvaddr);
> +	} else {
> +		qinfo->qdesc->qvaddr = PHYS_TO_VIRT(qinfo->qpaddr);
> +		memset(qinfo->qdesc->qvaddr, 0, qsize);
> +	}

This looks a lot like you are doing DMA. Please use the appropriate DMA
abstractions for this.

> +struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name)
> +{
> +	struct xgene_qmtm *qmtm = NULL;
> +	struct xgene_qmtm_sdev *sdev = NULL;
> +	struct device_node *np = NULL;
> +	struct platform_device *platdev;
> +	u8 slave;
> +
> +	for (slave = 0; slave < SLAVE_MAX; slave++) {
> +		sdev = &storm_sdev[slave];
> +		if (sdev->name && strcmp(name, sdev->name) == 0) {
> +			np = of_find_compatible_node(NULL, NULL,
> +				sdev->compatible);
> +			break;
> +		}
> +	}
> +
> +	if (np == NULL)
> +		return NULL;
> +
> +	platdev = of_find_device_by_node(np);
> +	qmtm = platform_get_drvdata(platdev);

Get rid of the of_find_compatible_node() here, this is another indication that
you are doing things backwards.

> +/* QMTM messaging structures */
> +/* 16 byte QMTM message format */
> +struct xgene_qmtm_msg16 {
> +	u32 UserInfo;
> +
> +	u32 FPQNum:12;
> +	u32 Rv2:2;
> +	u32 ELErr:2;
> +	u32 LEI:2;
> +	u32 NV:1;
> +	u32 LL:1;
> +	u32 PB:1;
> +	u32 HB:1;
> +	u32 Rv:1;
> +	u32 IN:1;
> +	u32 RType:4;
> +	u32 LErr:3;
> +	u32 HL:1;
> +
> +	u64 DataAddr:42;	/* 32/10 split */
> +	u32 Rv6:6;
> +	u32 BufDataLen:15;
> +	u32 C:1;
> +} __packed;

Bitfields are generally not endian-safe. Better use raw u32 values and
bit masks. Also don't use __packed in hardware descriptors, as it messes
up atomicity of accesses.

	Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Saturday 21 December 2013, Ravi Patel wrote:
> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
>  Security Engine subsystems. All subsystems communicate with QMTM using
>  messages which include information about the work to be performed and
>  the location of associated data buffers.  

Please describe here what the purpose of the qmtm is, as this is not
entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver
and why it is not possible to extend the dma slave API to describe qmtm
as a dma engine.

	Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Greg KH]

On Fri, Dec 20, 2013 at 06:57:27PM -0800, Ravi Patel wrote:
> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
>  Security Engine subsystems. All subsystems communicate with QMTM using
>  messages which include information about the work to be performed and
>  the location of associated data buffers.  

As you failed to respond or even acknowledge any of my review comments
before, I don't understand what you expect me to do with this patch dump
again.

Remember, kernel development is about trust.  You need me to trust that
you know what you are doing, and will be able to handle feedback and
respond to issues that come up over time.  Right now that trust is in
the negative zone, around "general annoyance" levels.  It's up to you to
change that or not, just like it is up to me to take your code, or not.

grumble,

greg k-h

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Sat, Dec 21, 2013 at 1:06 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Fri, Dec 20, 2013 at 06:57:27PM -0800, Ravi Patel wrote:
>> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
>>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
>>  Security Engine subsystems. All subsystems communicate with QMTM using
>>  messages which include information about the work to be performed and
>>  the location of associated data buffers.
>
> As you failed to respond or even acknowledge any of my review comments
> before, I don't understand what you expect me to do with this patch dump
> again.
>
> Remember, kernel development is about trust.  You need me to trust that
> you know what you are doing, and will be able to handle feedback and
> respond to issues that come up over time.  Right now that trust is in
> the negative zone, around "general annoyance" levels.  It's up to you to
> change that or not, just like it is up to me to take your code, or not.
>
> grumble,
>
> greg k-h

Greg,

I apologize for not responding to your comments before sending patch v2
The main intention of sending patch v2 was to remove APM confidential notice,
which was present in the first QMTM patch.
I wanted to send out patch v2 quickly, so that community can start reviewing
 the QMTM code.

I will go through all your comments and respond it.

pardon,

Ravi

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Loc Ho]

Hi Arnd,

On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Saturday 21 December 2013, Ravi Patel wrote:
>> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
>>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
>>  Security Engine subsystems. All subsystems communicate with QMTM using
>>  messages which include information about the work to be performed and
>>  the location of associated data buffers.
>
> Please describe here what the purpose of the qmtm is, as this is not
> entirely clear from the code.
>
> In particular, please describe how this differs from a dmaengine driver
> and why it is not possible to extend the dma slave API to describe qmtm
> as a dma engine.
>
[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA
engine driver which interfaces with this QM driver. We would have DMA
client interfaces with the X-Gene DMA driver (not available yet) via
DMA API which in turn interfaces with this QM driver via DMA API.
Won't this be kind of awkward? Also, the QM only manage messages (or
descriptors) which are 32-bytes or 64-bytes. It doesn't actually do
any data transfer of various sizes.

-Loc

[PATCH V2 2/4] misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Sat, Dec 21, 2013 at 12:04 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Saturday 21 December 2013, Ravi Patel wrote:
>> +/* CSR Address Macros */
>> +#define CSR_QM_CONFIG_ADDR   0x00000004
>> +#define  QM_ENABLE_WR(src)   (((u32)(src)<<31) & 0x80000000)
>> +
>> +#define CSR_PBM_ADDR         0x00000008
>> +#define  OVERWRITE_WR(src)   (((u32)(src)<<31) & 0x80000000)
>> +#define  SLVID_PBN_WR(src)   (((u32)(src)) & 0x000003ff)
>> +#define  SLAVE_ID_SHIFT              6
>> +
>> +#define CSR_PBM_BUF_WR_ADDR  0x0000000c
>> +#define CSR_PBM_BUF_RD_ADDR  0x00000010
>> +#define  PB_SIZE_WR(src)     (((u32)(src)<<31) & 0x80000000)
>> +#define  PREFETCH_BUF_EN_SET(dst, src) \
>> +     (((dst) & ~0x00200000) | (((u32)(src)<<21) & 0x00200000))
>> +#define  IS_FREE_POOL_SET(dst, src) \
>> +     (((dst) & ~0x00100000) | (((u32)(src)<<20) & 0x00100000))
>> +#define  TLVQ_SET(dst, src) \
>> +     (((dst) & ~0x00080000) | (((u32)(src)<<19) & 0x00080000))
>> +#define  CORRESPONDING_QNUM_SET(dst, src) \
>> +     (((dst) & ~0x0007fe00) | (((u32)(src)<<9) & 0x0007fe00))
>
> These macros are rather unreadable and unhelpful. Can't you just
> open-code them in the users?

Sure, will take of this in next patch.

>> +#define CSR_THRESHOLD0_SET1_ADDR     0x00000030
>> +#define CSR_THRESHOLD1_SET1_ADDR     0x00000034
>> +#define CSR_HYSTERESIS_ADDR          0x00000068
>> +#define CSR_QM_MBOX_NE_INT_MODE_ADDR 0x0000017c
>> +#define CSR_QMLITE_PBN_MAP_0_ADDR    0x00000228
>
> What about all the other registers in between?

All the other registers in between needs no change in values, default
values are fine.
These registers are not accessed and so we didn't defined them.
Let us know if you need them for a reason.

>> +void xgene_qmtm_wr32(struct xgene_qmtm *qmtm, u32 offset, u32 data)
>> +{
>> +     void *addr = (u8 *)qmtm->csr_vaddr + offset;
>> +     writel(data, addr);
>> +}
>
> Type mismatch: writel requires a "void __iomem*" pointer. Please check your
> driver using "sparse" to find bugs like this.

Sure, will take care of this in next patch

>> +/**
>> + * xgene_qmtm_set_qinfo - Create and configure a queue
>> + * @sdev:    Slave context
>> + * @qtype:   Queue type (P_QUEUE or FP_QUEUE)
>> + * @qsize:   Queue size see xgene_qmtm_qsize
>> + * @qaccess: Queue access method see xgene_qmtm_qaccess
>> + * @flags:   Queue Information flags
>> + * @qpaddr:  If Desire Queue Physical Address to use
>> + *
>> + * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
>> + * and Security Engine subsystems to create and configure a queue.
>> + *
>> + * Return:   0 on Success or -1 on Failure
>
> -1 on failure is not an appropriate return code. Please use standard
> error numbers such as -EINVAL.

Sure, will take care of this in next patch

>> +     rc = of_property_read_string(pdev->dev.of_node, "slave-name", &name);
>> +     if (rc < 0) {
>> +             dev_err(&pdev->dev, "Failed to get QMTM Ingress slave-name\n");
>> +             return rc;
>> +     }
>> +
>> +     sdev = xgene_qmtm_get_sdev((char *)name);
>> +     if (sdev == NULL) {
>> +             dev_err(&pdev->dev, "Ingress Slave error\n");
>> +             return -ENODEV;
>> +     }
>
> Why are you referring to another device by a string? Shouldn't that be a phandle?

Let me go through this and find the possibility.

>> +static struct of_device_id xgene_qmtm_match[] = {
>> +     {.compatible = "apm,xgene-qmtm-xge0",},
>> +     {.compatible = "apm,xgene-qmtm-soc",},
>> +     {.compatible = "apm,xgene-qmtm-xge2",},
>> +     {.compatible = "apm,xgene-qmtm-lite",},
>> +     {},
>> +};
>> +
>> +MODULE_DEVICE_TABLE(of, xgene_qmtm_match);
>
> You don't seem to differentiate between the four different strings. Why can't
> they just all be compatible to "apm,xgene-qmtm"?

All the QMTM manages resources for different subsystems.
So all the 4 QMTM are different, so each of them should have a unique identity.

>> +
>> +#define VIRT_TO_PHYS(x)      virt_to_phys(x)
>> +#define PHYS_TO_VIRT(x)      phys_to_virt(x)
>
> These don't belong here, use the standard functions. Also, it seems unlikely that a
> device driver actually wants physical addresses. Are these used for DMA?

The memory addresses passed in this macro/function are not used for DMA.
For creating a physical queue in the QMTM hardware, it needs the
physical address
of the queue kmalloc'ed by the driver,

>> +static struct xgene_qmtm_sdev storm_sdev[SLAVE_MAX] = {
>> +     [SLAVE_ETH0] = {
>> +                     .name = "SGMII0",
>> +                     .compatible = "apm,xgene-qmtm-soc",
>> +                     .slave = SLAVE_ETH0,
>> +                     .qmtm_ip = QMTM1,
>> +                     .slave_id = QMTM_SLAVE_ID_ETH0,
>> +                     .wq_pbn_start = 0x00,
>> +                     .wq_pbn_count = 0x08,
>> +                     .fq_pbn_start = 0x20,
>> +                     .fq_pbn_count = 0x08,
>> +                     },
>
> This table is backwards: the qmtm driver provides services to client drivers,
> it should have zero knowledge about what the clients are.
>
> Remove this array here and put the data into properties of the client
> drivers.

QMTM is managing the queue resources in its hardware for all the clients.
The clients meaning, Ethernet, PktDMA (XOR Engine) and Security Engine,
has zero knowledge of its qm resources.
So for that reason this array defining QMTM's client properties is
defined in QMTM driver

>> +     if (qinfo->flags & XGENE_SLAVE_Q_ADDR_ALLOC) {
>> +             qinfo->qdesc->qvaddr = kzalloc(qsize, GFP_KERNEL);
>> +             if (qinfo->qdesc->qvaddr == NULL) {
>> +                     kfree(qinfo->qdesc);
>> +                     rc = -ENOMEM;
>> +                     goto _ret_set_qstate;
>> +             }
>> +
>> +             qinfo->qpaddr = (u64) VIRT_TO_PHYS(qinfo->qdesc->qvaddr);
>> +     } else {
>> +             qinfo->qdesc->qvaddr = PHYS_TO_VIRT(qinfo->qpaddr);
>> +             memset(qinfo->qdesc->qvaddr, 0, qsize);
>> +     }
>
> This looks a lot like you are doing DMA. Please use the appropriate DMA
> abstractions for this.

Sure, will take are for the DMA abstraction in next revision.

>> +struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name)
>> +{
>> +     struct xgene_qmtm *qmtm = NULL;
>> +     struct xgene_qmtm_sdev *sdev = NULL;
>> +     struct device_node *np = NULL;
>> +     struct platform_device *platdev;
>> +     u8 slave;
>> +
>> +     for (slave = 0; slave < SLAVE_MAX; slave++) {
>> +             sdev = &storm_sdev[slave];
>> +             if (sdev->name && strcmp(name, sdev->name) == 0) {
>> +                     np = of_find_compatible_node(NULL, NULL,
>> +                             sdev->compatible);
>> +                     break;
>> +             }
>> +     }
>> +
>> +     if (np == NULL)
>> +             return NULL;
>> +
>> +     platdev = of_find_device_by_node(np);
>> +     qmtm = platform_get_drvdata(platdev);
>
> Get rid of the of_find_compatible_node() here, this is another indication that
> you are doing things backwards.

Since QMTM is managing resource for its client, we have defined the mapping
in the QMTM driver.
So whenever QMTM client like Ethernet driver probe is called, it will
pass on its ID/slave-name
to get its context which is managed by QMTM

>> +/* QMTM messaging structures */
>> +/* 16 byte QMTM message format */
>> +struct xgene_qmtm_msg16 {
>> +     u32 UserInfo;
>> +
>> +     u32 FPQNum:12;
>> +     u32 Rv2:2;
>> +     u32 ELErr:2;
>> +     u32 LEI:2;
>> +     u32 NV:1;
>> +     u32 LL:1;
>> +     u32 PB:1;
>> +     u32 HB:1;
>> +     u32 Rv:1;
>> +     u32 IN:1;
>> +     u32 RType:4;
>> +     u32 LErr:3;
>> +     u32 HL:1;
>> +
>> +     u64 DataAddr:42;        /* 32/10 split */
>> +     u32 Rv6:6;
>> +     u32 BufDataLen:15;
>> +     u32 C:1;
>> +} __packed;
>
> Bitfields are generally not endian-safe. Better use raw u32 values and
> bit masks. Also don't use __packed in hardware descriptors, as it messes
> up atomicity of accesses.

I explained in another email the reason for using bit fields.
Let me know if you want me to paste it here again.

Thanks,
Ravi

[PATCH V2 2/4] misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Saturday 21 December 2013 17:45:30 Ravi Patel wrote:
> On Sat, Dec 21, 2013 at 12:04 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Saturday 21 December 2013, Ravi Patel wrote:
> 
> >> +#define CSR_THRESHOLD0_SET1_ADDR     0x00000030
> >> +#define CSR_THRESHOLD1_SET1_ADDR     0x00000034
> >> +#define CSR_HYSTERESIS_ADDR          0x00000068
> >> +#define CSR_QM_MBOX_NE_INT_MODE_ADDR 0x0000017c
> >> +#define CSR_QMLITE_PBN_MAP_0_ADDR    0x00000228
> >
> > What about all the other registers in between?
> 
> All the other registers in between needs no change in values, default
> values are fine.
> These registers are not accessed and so we didn't defined them.
> Let us know if you need them for a reason.

Ok, this is fine. I was just making sure there isn't another driver
that talks to these registers.

> >> +     rc = of_property_read_string(pdev->dev.of_node, "slave-name", &name);
> >> +     if (rc < 0) {
> >> +             dev_err(&pdev->dev, "Failed to get QMTM Ingress slave-name\n");
> >> +             return rc;
> >> +     }
> >> +
> >> +     sdev = xgene_qmtm_get_sdev((char *)name);
> >> +     if (sdev == NULL) {
> >> +             dev_err(&pdev->dev, "Ingress Slave error\n");
> >> +             return -ENODEV;
> >> +     }
> >
> > Why are you referring to another device by a string? Shouldn't that be a phandle?
> 
> Let me go through this and find the possibility.

To clarify, I think the slave driver should look up a phandle to find
the qmtm device.

> >> +static struct of_device_id xgene_qmtm_match[] = {
> >> +     {.compatible = "apm,xgene-qmtm-xge0",},
> >> +     {.compatible = "apm,xgene-qmtm-soc",},
> >> +     {.compatible = "apm,xgene-qmtm-xge2",},
> >> +     {.compatible = "apm,xgene-qmtm-lite",},
> >> +     {},
> >> +};
> >> +
> >> +MODULE_DEVICE_TABLE(of, xgene_qmtm_match);
> >
> > You don't seem to differentiate between the four different strings. Why can't
> > they just all be compatible to "apm,xgene-qmtm"?
> 
> All the QMTM manages resources for different subsystems.
> So all the 4 QMTM are different, so each of them should have a unique identity.

Are you sure it's not just four instances of the same hardware block
connected to different slaves?

The "compatible" value should only identify differences in the register set.

> >> +#define VIRT_TO_PHYS(x)      virt_to_phys(x)
> >> +#define PHYS_TO_VIRT(x)      phys_to_virt(x)
> >
> > These don't belong here, use the standard functions. Also, it seems unlikely that a
> > device driver actually wants physical addresses. Are these used for DMA?
> 
> The memory addresses passed in this macro/function are not used for DMA.
> For creating a physical queue in the QMTM hardware, it needs the
> physical address
> of the queue kmalloc'ed by the driver,

Who is accessing the physical address then? Does this get passed to
some sort of hypervisor?

> >> +static struct xgene_qmtm_sdev storm_sdev[SLAVE_MAX] = {
> >> +     [SLAVE_ETH0] = {
> >> +                     .name = "SGMII0",
> >> +                     .compatible = "apm,xgene-qmtm-soc",
> >> +                     .slave = SLAVE_ETH0,
> >> +                     .qmtm_ip = QMTM1,
> >> +                     .slave_id = QMTM_SLAVE_ID_ETH0,
> >> +                     .wq_pbn_start = 0x00,
> >> +                     .wq_pbn_count = 0x08,
> >> +                     .fq_pbn_start = 0x20,
> >> +                     .fq_pbn_count = 0x08,
> >> +                     },
> >
> > This table is backwards: the qmtm driver provides services to client drivers,
> > it should have zero knowledge about what the clients are.
> >
> > Remove this array here and put the data into properties of the client
> > drivers.
> 
> QMTM is managing the queue resources in its hardware for all the clients.
> The clients meaning, Ethernet, PktDMA (XOR Engine) and Security Engine,
> has zero knowledge of its qm resources.
> So for that reason this array defining QMTM's client properties is
> defined in QMTM driver

No, that's the wrong way around. Please have a look at how other
subsystems (irq, dmaengine, clock, ...) manage the split between
knowledge.

> >> +struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name)
> >> +{
> >> +     struct xgene_qmtm *qmtm = NULL;
> >> +     struct xgene_qmtm_sdev *sdev = NULL;
> >> +     struct device_node *np = NULL;
> >> +     struct platform_device *platdev;
> >> +     u8 slave;
> >> +
> >> +     for (slave = 0; slave < SLAVE_MAX; slave++) {
> >> +             sdev = &storm_sdev[slave];
> >> +             if (sdev->name && strcmp(name, sdev->name) == 0) {
> >> +                     np = of_find_compatible_node(NULL, NULL,
> >> +                             sdev->compatible);
> >> +                     break;
> >> +             }
> >> +     }
> >> +
> >> +     if (np == NULL)
> >> +             return NULL;
> >> +
> >> +     platdev = of_find_device_by_node(np);
> >> +     qmtm = platform_get_drvdata(platdev);
> >
> > Get rid of the of_find_compatible_node() here, this is another indication that
> > you are doing things backwards.
> 
> Since QMTM is managing resource for its client, we have defined the mapping
> in the QMTM driver.
> So whenever QMTM client like Ethernet driver probe is called, it will
> pass on its ID/slave-name
> to get its context which is managed by QMTM

The client/slave driver should instead pass the slave ID and other
required information, essentially the stuff from the table above,
in an abstract form.

> >> +/* QMTM messaging structures */
> >> +/* 16 byte QMTM message format */
> >> +struct xgene_qmtm_msg16 {
> >> +     u32 UserInfo;
> >> +
> >> +     u32 FPQNum:12;
> >> +     u32 Rv2:2;
> >> +     u32 ELErr:2;
> >> +     u32 LEI:2;
> >> +     u32 NV:1;
> >> +     u32 LL:1;
> >> +     u32 PB:1;
> >> +     u32 HB:1;
> >> +     u32 Rv:1;
> >> +     u32 IN:1;
> >> +     u32 RType:4;
> >> +     u32 LErr:3;
> >> +     u32 HL:1;
> >> +
> >> +     u64 DataAddr:42;        /* 32/10 split */
> >> +     u32 Rv6:6;
> >> +     u32 BufDataLen:15;
> >> +     u32 C:1;
> >> +} __packed;
> >
> > Bitfields are generally not endian-safe. Better use raw u32 values and
> > bit masks. Also don't use __packed in hardware descriptors, as it messes
> > up atomicity of accesses.
> 
> I explained in another email the reason for using bit fields.
> Let me know if you want me to paste it here again.

No need for that. But driver code really needs to be written in an
endian-neutral way if the CPU supports both little-endian and
big-endian modes.

	Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
> On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Saturday 21 December 2013, Ravi Patel wrote:
> >> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
> >>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
> >>  Security Engine subsystems. All subsystems communicate with QMTM using
> >>  messages which include information about the work to be performed and
> >>  the location of associated data buffers.
> >
> > Please describe here what the purpose of the qmtm is, as this is not
> > entirely clear from the code.
> >
> > In particular, please describe how this differs from a dmaengine driver
> > and why it is not possible to extend the dma slave API to describe qmtm
> > as a dma engine.
> >
> [Loc Ho]
> If the QM driver implements the DMA API, what about the actual DMA
> engine driver which interfaces with this QM driver. We would have DMA
> client interfaces with the X-Gene DMA driver (not available yet) via
> DMA API which in turn interfaces with this QM driver via DMA API.
> Won't this be kind of awkward? Also, the QM only manage messages (or
> descriptors) which are 32-bytes or 64-bytes. It doesn't actually do
> any data transfer of various sizes.

Please describe here what the purpose of the qmtm is, as this is not
entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA
engine. If it's something completely different, you have to let
us know what it is so we can do a proper review rather than guessing.

Please provide a link to the data sheet if you are unable to explain.

	Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Sat, Dec 21, 2013 at 11:03 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
>> On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> > On Saturday 21 December 2013, Ravi Patel wrote:
>> >> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
>> >>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
>> >>  Security Engine subsystems. All subsystems communicate with QMTM using
>> >>  messages which include information about the work to be performed and
>> >>  the location of associated data buffers.
>> >
>> > Please describe here what the purpose of the qmtm is, as this is not
>> > entirely clear from the code.
>> >
>> > In particular, please describe how this differs from a dmaengine driver
>> > and why it is not possible to extend the dma slave API to describe qmtm
>> > as a dma engine.
>> >
>> [Loc Ho]
>> If the QM driver implements the DMA API, what about the actual DMA
>> engine driver which interfaces with this QM driver. We would have DMA
>> client interfaces with the X-Gene DMA driver (not available yet) via
>> DMA API which in turn interfaces with this QM driver via DMA API.
>> Won't this be kind of awkward? Also, the QM only manage messages (or
>> descriptors) which are 32-bytes or 64-bytes. It doesn't actually do
>> any data transfer of various sizes.
>
> Please describe here what the purpose of the qmtm is, as this is not
> entirely clear from the code or from your reply.
>
> Greg was guessing that it's a bus controller, my best guess is a DMA
> engine. If it's something completely different, you have to let
> us know what it is so we can do a proper review rather than guessing.
>
> Please provide a link to the data sheet if you are unable to explain.

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

For simplicity, I have shown only Ethernet.
PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Greg KH]

On Sat, Jan 04, 2014 at 03:59:46PM -0800, Ravi Patel wrote:
> On Sat, Dec 21, 2013 at 11:03 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
> >> On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> >> > On Saturday 21 December 2013, Ravi Patel wrote:
> >> >> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
> >> >>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
> >> >>  Security Engine subsystems. All subsystems communicate with QMTM using
> >> >>  messages which include information about the work to be performed and
> >> >>  the location of associated data buffers.
> >> >
> >> > Please describe here what the purpose of the qmtm is, as this is not
> >> > entirely clear from the code.
> >> >
> >> > In particular, please describe how this differs from a dmaengine driver
> >> > and why it is not possible to extend the dma slave API to describe qmtm
> >> > as a dma engine.
> >> >
> >> [Loc Ho]
> >> If the QM driver implements the DMA API, what about the actual DMA
> >> engine driver which interfaces with this QM driver. We would have DMA
> >> client interfaces with the X-Gene DMA driver (not available yet) via
> >> DMA API which in turn interfaces with this QM driver via DMA API.
> >> Won't this be kind of awkward? Also, the QM only manage messages (or
> >> descriptors) which are 32-bytes or 64-bytes. It doesn't actually do
> >> any data transfer of various sizes.
> >
> > Please describe here what the purpose of the qmtm is, as this is not
> > entirely clear from the code or from your reply.
> >
> > Greg was guessing that it's a bus controller, my best guess is a DMA
> > engine. If it's something completely different, you have to let
> > us know what it is so we can do a proper review rather than guessing.
> >
> > Please provide a link to the data sheet if you are unable to explain.
> 
> Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
> subsystem.
> 
> https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

There is nothing at this link :(

> PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.

How does a "security" subsystem have anything to do with ethernet?

confused,

greg k-h

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Sat, Jan 4, 2014 at 7:38 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Sat, Jan 04, 2014 at 03:59:46PM -0800, Ravi Patel wrote:
>> On Sat, Dec 21, 2013 at 11:03 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> > On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
>> >> On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> >> > On Saturday 21 December 2013, Ravi Patel wrote:
>> >> >> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
>> >> >>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
>> >> >>  Security Engine subsystems. All subsystems communicate with QMTM using
>> >> >>  messages which include information about the work to be performed and
>> >> >>  the location of associated data buffers.
>> >> >
>> >> > Please describe here what the purpose of the qmtm is, as this is not
>> >> > entirely clear from the code.
>> >> >
>> >> > In particular, please describe how this differs from a dmaengine driver
>> >> > and why it is not possible to extend the dma slave API to describe qmtm
>> >> > as a dma engine.
>> >> >
>> >> [Loc Ho]
>> >> If the QM driver implements the DMA API, what about the actual DMA
>> >> engine driver which interfaces with this QM driver. We would have DMA
>> >> client interfaces with the X-Gene DMA driver (not available yet) via
>> >> DMA API which in turn interfaces with this QM driver via DMA API.
>> >> Won't this be kind of awkward? Also, the QM only manage messages (or
>> >> descriptors) which are 32-bytes or 64-bytes. It doesn't actually do
>> >> any data transfer of various sizes.
>> >
>> > Please describe here what the purpose of the qmtm is, as this is not
>> > entirely clear from the code or from your reply.
>> >
>> > Greg was guessing that it's a bus controller, my best guess is a DMA
>> > engine. If it's something completely different, you have to let
>> > us know what it is so we can do a proper review rather than guessing.
>> >
>> > Please provide a link to the data sheet if you are unable to explain.
>>
>> Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
>> subsystem.
>>
>> https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing
>
> There is nothing at this link :(

The URL is pointing to text document.
I created another URL to the same document in pdf format.
https://docs.google.com/file/d/0B28TgQZ3JLoRbXJiUXJjSUNxeDA/edit?usp=sharing&pli=1


>> PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.
>
> How does a "security" subsystem have anything to do with ethernet?
>
> confused,

PktDMA and Security are completely independent of Ethernet.
Security Subsystem (SEC) is a CPU offload packet processing engine for security
 applications.
PktDMA subsystem is a general purpose data transfer engine which moves data
 from one location in memory to another.
PktDMA can perform certain data manipulation functions like XOR as
data is passing
 through the engine.
Both PktDMA and Security engine has their own QMTM work message format which
 will be prepared by their driver

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Loc Ho]

Hi,

>> >> >> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
>> >> >>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
>> >> >>  Security Engine subsystems. All subsystems communicate with QMTM using
>> >> >>  messages which include information about the work to be performed and
>> >> >>  the location of associated data buffers.
>> >> >
>> >> > Please describe here what the purpose of the qmtm is, as this is not
>> >> > entirely clear from the code.
>> >> >
>> >> > In particular, please describe how this differs from a dmaengine driver
>> >> > and why it is not possible to extend the dma slave API to describe qmtm
>> >> > as a dma engine.
>> >> >
>> >> [Loc Ho]
>> >> If the QM driver implements the DMA API, what about the actual DMA
>> >> engine driver which interfaces with this QM driver. We would have DMA
>> >> client interfaces with the X-Gene DMA driver (not available yet) via
>> >> DMA API which in turn interfaces with this QM driver via DMA API.
>> >> Won't this be kind of awkward? Also, the QM only manage messages (or
>> >> descriptors) which are 32-bytes or 64-bytes. It doesn't actually do
>> >> any data transfer of various sizes.
>> >
>> > Please describe here what the purpose of the qmtm is, as this is not
>> > entirely clear from the code or from your reply.
>> >
>> > Greg was guessing that it's a bus controller, my best guess is a DMA
>> > engine. If it's something completely different, you have to let
>> > us know what it is so we can do a proper review rather than guessing.
>> >
>> > Please provide a link to the data sheet if you are unable to explain.
>>
>> Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
>> subsystem.
>>
>> https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing
>
> There is nothing at this link :(

I tried click on the txt file link from web site
http://www.spinics.net/lists/netdev/msg264628.html and don't see any
problem with accessing the txt file.

-Loc

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Greg KH]

On Sat, Jan 04, 2014 at 09:39:08PM -0800, Loc Ho wrote:
> Hi,
> 
> >> >> >> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
> >> >> >>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
> >> >> >>  Security Engine subsystems. All subsystems communicate with QMTM using
> >> >> >>  messages which include information about the work to be performed and
> >> >> >>  the location of associated data buffers.
> >> >> >
> >> >> > Please describe here what the purpose of the qmtm is, as this is not
> >> >> > entirely clear from the code.
> >> >> >
> >> >> > In particular, please describe how this differs from a dmaengine driver
> >> >> > and why it is not possible to extend the dma slave API to describe qmtm
> >> >> > as a dma engine.
> >> >> >
> >> >> [Loc Ho]
> >> >> If the QM driver implements the DMA API, what about the actual DMA
> >> >> engine driver which interfaces with this QM driver. We would have DMA
> >> >> client interfaces with the X-Gene DMA driver (not available yet) via
> >> >> DMA API which in turn interfaces with this QM driver via DMA API.
> >> >> Won't this be kind of awkward? Also, the QM only manage messages (or
> >> >> descriptors) which are 32-bytes or 64-bytes. It doesn't actually do
> >> >> any data transfer of various sizes.
> >> >
> >> > Please describe here what the purpose of the qmtm is, as this is not
> >> > entirely clear from the code or from your reply.
> >> >
> >> > Greg was guessing that it's a bus controller, my best guess is a DMA
> >> > engine. If it's something completely different, you have to let
> >> > us know what it is so we can do a proper review rather than guessing.
> >> >
> >> > Please provide a link to the data sheet if you are unable to explain.
> >>
> >> Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
> >> subsystem.
> >>
> >> https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing
> >
> > There is nothing at this link :(
> 
> I tried click on the txt file link from web site
> http://www.spinics.net/lists/netdev/msg264628.html and don't see any
> problem with accessing the txt file.

Now it's working :)

Anyway, as this is a .txt file, why wasn't it just sent in email, and
added to the patch series as a documentation file?

thanks,

greg k-h

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Sunday 05 January 2014, Ravi Patel wrote:
> On Sat, Dec 21, 2013 at 11:03 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
> >> On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> >
> > Please describe here what the purpose of the qmtm is, as this is not
> > entirely clear from the code or from your reply.
> >
> > Greg was guessing that it's a bus controller, my best guess is a DMA
> > engine. If it's something completely different, you have to let
> > us know what it is so we can do a proper review rather than guessing.
> >
> > Please provide a link to the data sheet if you are unable to explain.
> 
> Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
> subsystem.
> 
> https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing
> 
> For simplicity, I have shown only Ethernet.
> PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.
> 

Thanks, that helps a lot. Please add this file to an appropriate place
in the Documentation directory in the next version of your patches.

There is still one central aspect that remains unclear to me, which is
what the QMTM is actually good for, as opposed to how it gets used from
the OS. In the text description, it sounds like the ethernet is the DMA
master and performs DMA all by itself but from that it's not clear why
a message to and from the QMTM is needed. From your drawing on the other
hand, it seems like the QMTM is really the DMA master and performs the
DMA on behalf of the ethernet device, which isn't connected to the
coherent interface itself. If this is correct, it seems that QMTM is more
like a DMA engine after all that should use the existing slave API to
provide services to slave drivers.

	Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Sun, Jan 5, 2014 at 10:11 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Sunday 05 January 2014, Ravi Patel wrote:
>> On Sat, Dec 21, 2013 at 11:03 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> > On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
>> >> On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> >
>> > Please describe here what the purpose of the qmtm is, as this is not
>> > entirely clear from the code or from your reply.
>> >
>> > Greg was guessing that it's a bus controller, my best guess is a DMA
>> > engine. If it's something completely different, you have to let
>> > us know what it is so we can do a proper review rather than guessing.
>> >
>> > Please provide a link to the data sheet if you are unable to explain.
>>
>> Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
>> subsystem.
>>
>> https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing
>>
>> For simplicity, I have shown only Ethernet.
>> PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.
>
> Thanks, that helps a lot. Please add this file to an appropriate place
> in the Documentation directory in the next version of your patches.
>
> There is still one central aspect that remains unclear to me, which is
> what the QMTM is actually good for, as opposed to how it gets used from
> the OS.

The document shows the run-time flow of messages between CPU, QMTM and Ethernet.
QMTM is a centralized resource manager/driver which exports APIs to
1. Initialize & allocate queue & pbn to the Ethernet, PktDMA and
Security subsystem.
2. Read queue state so that subsystems driver knows how much more work
it can offload
    to its subsystem.
3. Apply QoS for subsystems on their queues.

> In the text description, it sounds like the ethernet is the DMA
> master and performs DMA all by itself but from that it's not clear why
> a message to and from the QMTM is needed. From your drawing on the other
> hand, it seems like the QMTM is really the DMA master and performs the
> DMA on behalf of the ethernet device, which isn't connected to the
> coherent interface itself. If this is correct, it seems that QMTM is more
> like a DMA engine after all that should use the existing slave API to
> provide services to slave drivers.

Each subsystem defines their own format of work message.
A message (or queue descriptor) has attribute fields (QMTM specific)
which is common
 for all subsystem work messages.
The remaining fields of a message are specific to subsystem.
QMTM device doesn't have any knowledge of these subsystem specific
fields and the data
operation which subsystem is going to perform using these fields.
e.g.
1. Ethernet work message includes data address & length which is used
by Ethernet
DMA engine for copying the data to/from its internal FIFO
2. PktDMA work message includes multiple data addresses & lengths for
doing scatter/gather,
XOR operations and result data address/es to give back result to the
CPU (driver)
3. Security work message includes data address & length for doing
encryption or decryption,
the type of encryption or decryption and result data address to give
back result to the CPU (driver)

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Sun, Jan 5, 2014 at 10:01 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Sat, Jan 04, 2014 at 09:39:08PM -0800, Loc Ho wrote:
>> Hi,
>>
>> >> >> >> This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
>> >> >> >>  QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and
>> >> >> >>  Security Engine subsystems. All subsystems communicate with QMTM using
>> >> >> >>  messages which include information about the work to be performed and
>> >> >> >>  the location of associated data buffers.
>> >> >> >
>> >> >> > Please describe here what the purpose of the qmtm is, as this is not
>> >> >> > entirely clear from the code.
>> >> >> >
>> >> >> > In particular, please describe how this differs from a dmaengine driver
>> >> >> > and why it is not possible to extend the dma slave API to describe qmtm
>> >> >> > as a dma engine.
>> >> >> >
>> >> >> [Loc Ho]
>> >> >> If the QM driver implements the DMA API, what about the actual DMA
>> >> >> engine driver which interfaces with this QM driver. We would have DMA
>> >> >> client interfaces with the X-Gene DMA driver (not available yet) via
>> >> >> DMA API which in turn interfaces with this QM driver via DMA API.
>> >> >> Won't this be kind of awkward? Also, the QM only manage messages (or
>> >> >> descriptors) which are 32-bytes or 64-bytes. It doesn't actually do
>> >> >> any data transfer of various sizes.
>> >> >
>> >> > Please describe here what the purpose of the qmtm is, as this is not
>> >> > entirely clear from the code or from your reply.
>> >> >
>> >> > Greg was guessing that it's a bus controller, my best guess is a DMA
>> >> > engine. If it's something completely different, you have to let
>> >> > us know what it is so we can do a proper review rather than guessing.
>> >> >
>> >> > Please provide a link to the data sheet if you are unable to explain.
>> >>
>> >> Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
>> >> subsystem.
>> >>
>> >> https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing
>> >
>> > There is nothing at this link :(
>>
>> I tried click on the txt file link from web site
>> http://www.spinics.net/lists/netdev/msg264628.html and don't see any
>> problem with accessing the txt file.
>
> Now it's working :)
>
> Anyway, as this is a .txt file, why wasn't it just sent in email, and
> added to the patch series as a documentation file?

Sure I will include the text documentation to an appropriate place in the
Documentation directory in my next version of QMTM patch.

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Sun, Jan 5, 2014 at 12:48 PM, Ravi Patel <rapatel@apm.com> wrote:
> On Sun, Jan 5, 2014 at 10:11 AM, Arnd Bergmann <arnd@arndb.de> wrote:
>> On Sunday 05 January 2014, Ravi Patel wrote:
>>> On Sat, Dec 21, 2013 at 11:03 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>>> >
>>> > Please describe here what the purpose of the qmtm is, as this is not
>>> > entirely clear from the code or from your reply.
>>> >
>>> > Greg was guessing that it's a bus controller, my best guess is a DMA
>>> > engine. If it's something completely different, you have to let
>>> > us know what it is so we can do a proper review rather than guessing.
>>> >
>>> > Please provide a link to the data sheet if you are unable to explain.
>>>
>>> Here is URL to a text document explaining role of QMTM device with CPU, Ethernet
>>> subsystem.
>>>
>>> https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing
>>>
>>> For simplicity, I have shown only Ethernet.
>>> PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.
>>
>> Thanks, that helps a lot. Please add this file to an appropriate place
>> in the Documentation directory in the next version of your patches.
>>
>> There is still one central aspect that remains unclear to me, which is
>> what the QMTM is actually good for, as opposed to how it gets used from
>> the OS.
>
> The document shows the run-time flow of messages between CPU, QMTM and Ethernet.
> QMTM is a centralized resource manager/driver which exports APIs to
> 1. Initialize & allocate queue & pbn to the Ethernet, PktDMA and
> Security subsystem.
> 2. Read queue state so that subsystems driver knows how much more work
> it can offload
>     to its subsystem.
> 3. Apply QoS for subsystems on their queues.
>
>> In the text description, it sounds like the ethernet is the DMA
>> master and performs DMA all by itself but from that it's not clear why
>> a message to and from the QMTM is needed. From your drawing on the other
>> hand, it seems like the QMTM is really the DMA master and performs the
>> DMA on behalf of the ethernet device, which isn't connected to the
>> coherent interface itself. If this is correct, it seems that QMTM is more
>> like a DMA engine after all that should use the existing slave API to
>> provide services to slave drivers.
>
> Each subsystem defines their own format of work message.
> A message (or queue descriptor) has attribute fields (QMTM specific)
> which is common
>  for all subsystem work messages.
> The remaining fields of a message are specific to subsystem.
> QMTM device doesn't have any knowledge of these subsystem specific
> fields and the data
> operation which subsystem is going to perform using these fields.
> e.g.
> 1. Ethernet work message includes data address & length which is used
> by Ethernet
> DMA engine for copying the data to/from its internal FIFO
> 2. PktDMA work message includes multiple data addresses & lengths for
> doing scatter/gather,
> XOR operations and result data address/es to give back result to the
> CPU (driver)
> 3. Security work message includes data address & length for doing
> encryption or decryption,
> the type of encryption or decryption and result data address to give
> back result to the CPU (driver)

Do you want any further clarification or document related to QMTM.
We want to make sure everyone is on same page, understand and
conclude upon that QMTM is a device and and not a bus or a dma
engine.

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Friday 10 January 2014, Ravi Patel wrote:

> Do you want any further clarification or document related to QMTM.
> We want to make sure everyone is on same page, understand and
> conclude upon that QMTM is a device and and not a bus or a dma
> engine.

I have a much better understanding now, but there are still a few open
questions from my side. Let me try to explain in my own words what I
think is the relevant information (part of this is still guessing).
It took me a while to figure out what it does from your description,
and then some more time to see what it's actually good for (as
opposed to adding complexity).

Please confirm or correct the individual statements in this
description:

The QMTM serves as a relay for short (a few bytes) messages between
the OS software and various slave hardware blocks on the SoC.
The messages are typically but not always DMA descriptors used by the
slave device used for starting bus master transactions by the slave,
or for notifying sofware about the competion of a DMA transaction.

The message format is specific to the slave device and the QMTM
only understands the common header of the message.

OS software sees the messages in cache-coherent memory and does
not require any cache flushes or MMIO access for inbound messages
and only a single posted MMIO write for outbound messages.

The queues are likely designed to be per-thread and don't require
software-side locking.

For outbound messages, the QMTM is the bus master of a device-to-device
DMA transaction that gets started once a message is queued and the
device has signaled that it is ready for receiving it. The QMTM needs
to know the bus address of the device as well as a slave ID for
the signal pin.
For inbound messages, the QMTM slave initiates a busmaster transaction
and needs to know the bus address of its QMTM port, while the QMTM
needs to know only the slave ID that is associated with the queue.

In addition to those hardware properties, the QMTM driver needs to
set up a memory buffer for the message queue as seen by the CPU,
and needs tell the QMTM the location as well as some other
properties such as the message length.

For inbound messages, the QMTM serves a similar purpose as an MSI
controller, ensuring that inbound DMA data has arrived in RAM
before an interrupt is delivered to the CPU and thereby avoiding
the need for an expensive MMIO read to serialize the DMA.

The resources managed by the QMTM are both SoC-global (e.g. bus
bandwidth) and slave specific (e.g. ethernet bandwith or buffer space).
Global resource management is performed to prevent one slave
device from monopolizing the system or preventing other slaves
from making forward progress.
Examples for local resource management (I had to think about this 
a long time, but probably some of these are wrong) would be
* balancing between multiple non-busmaster devices connected to
  a dma-engine
* distributing incoming ethernet data to the available CPUs based on
  a flow classifier in the MAC, e.g. by IOV MAC address, VLAN tag
  or even individual TCP connection depending on the NIC's capabilities.
* 802.1p flow control for incoming ethernet data based on the amount
  of data queued up between the MAC and the driver
* interrupt mitigation for both inbound data and outbound completion,
  by delaying the IRQ to the OS until multiple messages have arrived
  or a queue specific amount of time has passed.
* controlling the amount of outbound buffer space per flow to minimize
  buffer-bloat between an ethernet driver and the NIC hardware.
* reordering data from outbound flows based on priority.

This is basically my current interpretation, I hope I got at least
some of it right this time ;-)

	Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Sun, Jan 12, 2014 at 1:19 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Friday 10 January 2014, Ravi Patel wrote:
>
>> Do you want any further clarification or document related to QMTM.
>> We want to make sure everyone is on same page, understand and
>> conclude upon that QMTM is a device and and not a bus or a dma
>> engine.
>
> I have a much better understanding now, but there are still a few open
> questions from my side. Let me try to explain in my own words what I
> think is the relevant information (part of this is still guessing).
> It took me a while to figure out what it does from your description,
> and then some more time to see what it's actually good for (as
> opposed to adding complexity).
>
> Please confirm or correct the individual statements in this
> description:
>
> The QMTM serves as a relay for short (a few bytes) messages between
> the OS software and various slave hardware blocks on the SoC.
> The messages are typically but not always DMA descriptors used by the
> slave device used for starting bus master transactions by the slave,
> or for notifying sofware about the competion of a DMA transaction.
>
> The message format is specific to the slave device and the QMTM
> only understands the common header of the message.
>
> OS software sees the messages in cache-coherent memory and does
> not require any cache flushes or MMIO access for inbound messages
> and only a single posted MMIO write for outbound messages.
>
> The queues are likely designed to be per-thread and don't require
> software-side locking.
>
> For outbound messages, the QMTM is the bus master of a device-to-device
> DMA transaction that gets started once a message is queued and the
> device has signaled that it is ready for receiving it. The QMTM needs
> to know the bus address of the device as well as a slave ID for
> the signal pin.
> For inbound messages, the QMTM slave initiates a busmaster transaction
> and needs to know the bus address of its QMTM port, while the QMTM
> needs to know only the slave ID that is associated with the queue.
>
> In addition to those hardware properties, the QMTM driver needs to
> set up a memory buffer for the message queue as seen by the CPU,
> and needs tell the QMTM the location as well as some other
> properties such as the message length.

Your description is correct upto this point.

> For inbound messages, the QMTM serves a similar purpose as an MSI
> controller, ensuring that inbound DMA data has arrived in RAM
> before an interrupt is delivered to the CPU and thereby avoiding
> the need for an expensive MMIO read to serialize the DMA.

For inbound messages, slave device generates message on a completion
of a inbound DMA operation or any relevant operation targeted to the
CPU. The QMTM's role is to just trigger an interrupt to CPU when there
is a new message arrived from a slave device. QMTM doesn't know what
the message was for. It is upto the upper layer drivers to decide how
to process this message.

> The resources managed by the QMTM are both SoC-global (e.g. bus
> bandwidth) and slave specific (e.g. ethernet bandwith or buffer space).
> Global resource management is performed to prevent one slave
> device from monopolizing the system or preventing other slaves
> from making forward progress.
> Examples for local resource management (I had to think about this
> a long time, but probably some of these are wrong) would be
> * balancing between multiple non-busmaster devices connected to
>   a dma-engine
> * distributing incoming ethernet data to the available CPUs based on
>   a flow classifier in the MAC, e.g. by IOV MAC address, VLAN tag
>   or even individual TCP connection depending on the NIC's capabilities.
> * 802.1p flow control for incoming ethernet data based on the amount
>   of data queued up between the MAC and the driver
> * interrupt mitigation for both inbound data and outbound completion,
>   by delaying the IRQ to the OS until multiple messages have arrived
>   or a queue specific amount of time has passed.
> * controlling the amount of outbound buffer space per flow to minimize
>   buffer-bloat between an ethernet driver and the NIC hardware.
> * reordering data from outbound flows based on priority.
>
> This is basically my current interpretation, I hope I got at least
> some of it right this time ;-)

You have got them right. Although we have taken Ethernet examples here,
most of the local resource management apply to other slave devices also.

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Monday 13 January 2014, Ravi Patel wrote:
> > For inbound messages, the QMTM serves a similar purpose as an MSI
> > controller, ensuring that inbound DMA data has arrived in RAM
> > before an interrupt is delivered to the CPU and thereby avoiding
> > the need for an expensive MMIO read to serialize the DMA.
> 
> For inbound messages, slave device generates message on a completion
> of a inbound DMA operation or any relevant operation targeted to the
> CPU. The QMTM's role is to just trigger an interrupt to CPU when there
> is a new message arrived from a slave device. QMTM doesn't know what
> the message was for. It is upto the upper layer drivers to decide how
> to process this message.

That doesn't seem to contradict what I wrote above. The DMA ordering
would be an implicit side-effect of the message generated by the
slave device if the QMTM is on the same bus as the external memory
controller and the message has the "strict ordering" bit set on the
bus transaction.

	Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Monday 13 January 2014, Ravi Patel wrote:
> > Examples for local resource management (I had to think about this
> > a long time, but probably some of these are wrong) would be
> > * balancing between multiple non-busmaster devices connected to
> >   a dma-engine
> > * distributing incoming ethernet data to the available CPUs based on
> >   a flow classifier in the MAC, e.g. by IOV MAC address, VLAN tag
> >   or even individual TCP connection depending on the NIC's capabilities.
> > * 802.1p flow control for incoming ethernet data based on the amount
> >   of data queued up between the MAC and the driver
> > * interrupt mitigation for both inbound data and outbound completion,
> >   by delaying the IRQ to the OS until multiple messages have arrived
> >   or a queue specific amount of time has passed.
> > * controlling the amount of outbound buffer space per flow to minimize
> >   buffer-bloat between an ethernet driver and the NIC hardware.
> > * reordering data from outbound flows based on priority.
> >
> > This is basically my current interpretation, I hope I got at least
> > some of it right this time ;-)
> 
> You have got them right. Although we have taken Ethernet examples here,
> most of the local resource management apply to other slave devices also.

I'm very suprised I got all those right, it seems it's a quite sophisticated
piece of hardware then. I guess most other slave devices only use a subset
of the capabilities that ethernet wants.

Now that I have a better understanding of what the driver is good for and
how it's used, let's have a look at how we can make it fit into the Linux
driver APIs and the DT bindings. We don't have anything exactly like this
yet, but I think the "mailbox" framework is a close enough match that we
can fit it in there, with some extensions. This framework is still in the
process of being created (so far there is only a TI OMAP specific driver,
and one for pl320), and I've not seen any mailboxes that support IRQ
mitigation or multiple software interfaces per hardware mailbox, but those
should be easy enough to add.

For the DT binding, I would suggest using something along the lines of
what we have for clocks, pinctrl and dmaengine. OMAP doesn't use this
(yet), but now would be a good time to standardize it. The QMTM node
should define a "#mailbox-cells" property that indicates how many
32-bit cells a qmtm needs to describe the connection between the
controller and the slave. My best guess is that this would be hardcoded
to <3>, using two cells for a 64-bit FIFO bus address, and a 32-bit cell
for the slave-id signal number. All other parameters that you have in
the big table in the qmtm driver at the moment can then get moved into
the slave drivers, as they are constant per type of slave. This will
simplify the QMTM driver.

In the slave, you should have a "mailboxes" property with a phandle
to the qmtm followed by the three cells to identify the actual
queue. If it's possible that a device uses more than one rx and
one tx queue, we also need a "mailbox-names" property to identify
the individual queues.

For the in-kernel interfaces, we should probably start a conversation
with the owners of the mailbox drivers to get a common API, for now
I'd suggest you just leave it as it is, and only adapt for the new
binding.

	Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Ravi Patel]

On Tue, Jan 14, 2014 at 7:15 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> Now that I have a better understanding of what the driver is good for and
> how it's used, let's have a look at how we can make it fit into the Linux
> driver APIs and the DT bindings. We don't have anything exactly like this
> yet, but I think the "mailbox" framework is a close enough match that we
> can fit it in there, with some extensions. This framework is still in the
> process of being created (so far there is only a TI OMAP specific driver,
> and one for pl320), and I've not seen any mailboxes that support IRQ
> mitigation or multiple software interfaces per hardware mailbox, but those
> should be easy enough to add.

OK. We will put Queue Manager driver under drivers/mailbox directory along
with TI OMAP and pl320 drivers.

> For the DT binding, I would suggest using something along the lines of
> what we have for clocks, pinctrl and dmaengine. OMAP doesn't use this
> (yet), but now would be a good time to standardize it. The QMTM node
> should define a "#mailbox-cells" property that indicates how many
> 32-bit cells a qmtm needs to describe the connection between the
> controller and the slave. My best guess is that this would be hardcoded
> to <3>, using two cells for a 64-bit FIFO bus address, and a 32-bit cell
> for the slave-id signal number. All other parameters that you have in
> the big table in the qmtm driver at the moment can then get moved into
> the slave drivers, as they are constant per type of slave. This will
> simplify the QMTM driver.
>
> In the slave, you should have a "mailboxes" property with a phandle
> to the qmtm followed by the three cells to identify the actual
> queue. If it's possible that a device uses more than one rx and
> one tx queue, we also need a "mailbox-names" property to identify
> the individual queues.

We explored on DT bindings suggestion given by you. We have come
up with a sample DT binding for how it will look like. Herewith we have
provided the same. Would you please review and give us your
comments before we change our driver and DTS file to accomodate it?

Sample DTS node for QM:
                qmlite: qmtm at 17030000 {
                        compatible = "apm,xgene-qmtm-lite";
                        reg = <0x0 0x17030000 0x0 0x10000>,
                              <0x0 0x10000000 0x0 0x400000>;
                        interrupts = <0x0 0x40 0x4>,
                                     <0x0 0x3c 0x4>;
                        status = "ok";
                        #clock-cells = <1>;
                        clocks = <&qmlclk 0>;
                        #mailbox-cells = <3>;
                };

Sample DTS node for Ethernet:
                menet: ethernet at 17020000 {
                        compatible = "apm,xgene-enet";
                        status = "disabled";
                        reg = <0x0 0x17020000 0x0 0x30>,
                              <0x0 0x17020000 0x0 0x10000>,
                              <0x0 0x17020000 0x0 0x20>;
                        mailboxes = <&qmlite 0x0 0x1000002c 0x0000>,
                                            <&qmlite 0x0 0x10000052 0x0020>,
                                            <&qmlite 0x0 0x10000060 0x0f00>
                        mailbox-names = "mb-tx", "mb-fp", "mb-rx";
                        interrupts = <0x0 0x38 0x4>,
                                     <0x0 0x39 0x4>,
                                     <0x0 0x3a 0x4>;
                        #clock-cells = <1>;
                        clocks = <&eth8clk 0>;
                        local-mac-address = <0x0 0x11 0x3a 0x8a 0x5a 0x78>;
                        max-frame-size = <0x233a>;
                        phyid = <3>;
                        phy-mode = "rgmii";
                };

The mailbox node in DTS has following format:
mailbox = <&parent 'higher 32 bit bus address' 'lower 32 bit bus
address' 'signal id'>
Ethernet driver will call following function in platform_probe:
 mailbox_get(dev, "mb-tx");
mailbox_get API will return the the context of allocated and configured mailbox.
For now, mailbox_get API will be implemented in xgene QMTM driver.
Eventually when mailbox framework in Linux will be standardized, we
will use it instead.

> For the in-kernel interfaces, we should probably start a conversation
> with the owners of the mailbox drivers to get a common API, for now
> I'd suggest you just leave it as it is, and only adapt for the new
> binding.

Sure. For now we will put our driver mostly as is in the
drivers/mailbox. Can you please help us identify the owners of the
mailbox drivers? As you suggested, we can start conversation with them
to define common in kernel APIs.

Ravi

On Tue, Jan 14, 2014 at 7:15 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Monday 13 January 2014, Ravi Patel wrote:
>> > Examples for local resource management (I had to think about this
>> > a long time, but probably some of these are wrong) would be
>> > * balancing between multiple non-busmaster devices connected to
>> >   a dma-engine
>> > * distributing incoming ethernet data to the available CPUs based on
>> >   a flow classifier in the MAC, e.g. by IOV MAC address, VLAN tag
>> >   or even individual TCP connection depending on the NIC's capabilities.
>> > * 802.1p flow control for incoming ethernet data based on the amount
>> >   of data queued up between the MAC and the driver
>> > * interrupt mitigation for both inbound data and outbound completion,
>> >   by delaying the IRQ to the OS until multiple messages have arrived
>> >   or a queue specific amount of time has passed.
>> > * controlling the amount of outbound buffer space per flow to minimize
>> >   buffer-bloat between an ethernet driver and the NIC hardware.
>> > * reordering data from outbound flows based on priority.
>> >
>> > This is basically my current interpretation, I hope I got at least
>> > some of it right this time ;-)
>>
>> You have got them right. Although we have taken Ethernet examples here,
>> most of the local resource management apply to other slave devices also.
>
> I'm very suprised I got all those right, it seems it's a quite sophisticated
> piece of hardware then. I guess most other slave devices only use a subset
> of the capabilities that ethernet wants.
>
> Now that I have a better understanding of what the driver is good for and
> how it's used, let's have a look at how we can make it fit into the Linux
> driver APIs and the DT bindings. We don't have anything exactly like this
> yet, but I think the "mailbox" framework is a close enough match that we
> can fit it in there, with some extensions. This framework is still in the
> process of being created (so far there is only a TI OMAP specific driver,
> and one for pl320), and I've not seen any mailboxes that support IRQ
> mitigation or multiple software interfaces per hardware mailbox, but those
> should be easy enough to add.
>
> For the DT binding, I would suggest using something along the lines of
> what we have for clocks, pinctrl and dmaengine. OMAP doesn't use this
> (yet), but now would be a good time to standardize it. The QMTM node
> should define a "#mailbox-cells" property that indicates how many
> 32-bit cells a qmtm needs to describe the connection between the
> controller and the slave. My best guess is that this would be hardcoded
> to <3>, using two cells for a 64-bit FIFO bus address, and a 32-bit cell
> for the slave-id signal number. All other parameters that you have in
> the big table in the qmtm driver at the moment can then get moved into
> the slave drivers, as they are constant per type of slave. This will
> simplify the QMTM driver.
>
> In the slave, you should have a "mailboxes" property with a phandle
> to the qmtm followed by the three cells to identify the actual
> queue. If it's possible that a device uses more than one rx and
> one tx queue, we also need a "mailbox-names" property to identify
> the individual queues.
>
> For the in-kernel interfaces, we should probably start a conversation
> with the owners of the mailbox drivers to get a common API, for now
> I'd suggest you just leave it as it is, and only adapt for the new
> binding.
>
>         Arnd

[PATCH V2 0/4] misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

[Arnd Bergmann]

On Tuesday 28 January 2014, Ravi Patel wrote:
> On Tue, Jan 14, 2014 at 7:15 AM, Arnd Bergmann <arnd@arndb.de> wrote:
-
> > For the DT binding, I would suggest using something along the lines of
> > what we have for clocks, pinctrl and dmaengine. OMAP doesn't use this
> > (yet), but now would be a good time to standardize it. The QMTM node
> > should define a "#mailbox-cells" property that indicates how many
> > 32-bit cells a qmtm needs to describe the connection between the
> > controller and the slave. My best guess is that this would be hardcoded
> > to <3>, using two cells for a 64-bit FIFO bus address, and a 32-bit cell
> > for the slave-id signal number. All other parameters that you have in
> > the big table in the qmtm driver at the moment can then get moved into
> > the slave drivers, as they are constant per type of slave. This will
> > simplify the QMTM driver.
> >
> > In the slave, you should have a "mailboxes" property with a phandle
> > to the qmtm followed by the three cells to identify the actual
> > queue. If it's possible that a device uses more than one rx and
> > one tx queue, we also need a "mailbox-names" property to identify
> > the individual queues.
> 
> We explored on DT bindings suggestion given by you. We have come
> up with a sample DT binding for how it will look like. Herewith we have
> provided the same. Would you please review and give us your
> comments before we change our driver and DTS file to accomodate it?
> 
> Sample DTS node for QM:
>                 qmlite: qmtm at 17030000 {
>                         compatible = "apm,xgene-qmtm-lite";

I would use 'mailbox at 17030000' as the node name, as the name part
is supposed to be descriptive of the function rather than the implemention.

>                         reg = <0x0 0x17030000 0x0 0x10000>,
>                               <0x0 0x10000000 0x0 0x400000>;
>                         interrupts = <0x0 0x40 0x4>,
>                                      <0x0 0x3c 0x4>;
>                         status = "ok";
>                         #clock-cells = <1>;
>                         clocks = <&qmlclk 0>;
>                         #mailbox-cells = <3>;
>                 };

The #clock-cells seems misplaced here, unless this is also a clock
provider, which I don't think it is.

> 
> Sample DTS node for Ethernet:
>                 menet: ethernet at 17020000 {
>                         compatible = "apm,xgene-enet";
>                         status = "disabled";
>                         reg = <0x0 0x17020000 0x0 0x30>,
>                               <0x0 0x17020000 0x0 0x10000>,
>                               <0x0 0x17020000 0x0 0x20>;

Unrelated, but it seems strange to have three register sets of different
sizes at the same offset.

>                         mailboxes = <&qmlite 0x0 0x1000002c 0x0000>,
>                                             <&qmlite 0x0 0x10000052 0x0020>,
>                                             <&qmlite 0x0 0x10000060 0x0f00>
>                         mailbox-names = "mb-tx", "mb-fp", "mb-rx";

I would leave out the "mb-" part of the strings and just document them
as "tx", "rx" and "fp".

>                         interrupts = <0x0 0x38 0x4>,
>                                      <0x0 0x39 0x4>,
>                                      <0x0 0x3a 0x4>;
>                         #clock-cells = <1>;

Same comment about #clock-cells here.

>                         clocks = <&eth8clk 0>;
>                         local-mac-address = <0x0 0x11 0x3a 0x8a 0x5a 0x78>;
>                         max-frame-size = <0x233a>;
>                         phyid = <3>;
>                         phy-mode = "rgmii";
>                 };
> 
> The mailbox node in DTS has following format:
> mailbox = <&parent 'higher 32 bit bus address' 'lower 32 bit bus
> address' 'signal id'>

sounds good.

> Ethernet driver will call following function in platform_probe:
>  mailbox_get(dev, "mb-tx");
> mailbox_get API will return the the context of allocated and configured mailbox.
> For now, mailbox_get API will be implemented in xgene QMTM driver.
> Eventually when mailbox framework in Linux will be standardized, we
> will use it instead.

Ok.

> > For the in-kernel interfaces, we should probably start a conversation
> > with the owners of the mailbox drivers to get a common API, for now
> > I'd suggest you just leave it as it is, and only adapt for the new
> > binding.
> 
> Sure. For now we will put our driver mostly as is in the
> drivers/mailbox. Can you please help us identify the owners of the
> mailbox drivers? As you suggested, we can start conversation with them
> to define common in kernel APIs.
 
Please talk to "Suman Anna" <s-anna@ti.com> for the TI part and Rob
Herring <robh@kernel.org> for pl320. The pl320 driver was written
by Mark Langsdorf for Calxeda, but I don't have an updated email
address for him and assume that the calxeda address is no longer
functional.

	Arnd