[PATCH v2 0/3] mailbox: Add support for Hi3660 mailbox

[PATCH v2 0/3] mailbox: Add support for Hi3660 mailbox

[Kaihua Zhong]

From: Leo Yan <leo.yan@linaro.org>

Hi3660 mailbox controller is used to send message within multiple
processors, MCU, HIFI, etc. This patch series is to implement an
initial version for Hi3660 mailbox driver with "automatic
acknowledge" mode.

The patch set have been verified with Hi3660 stub clock driver, so
we can send message to MCU to execute CPU frequency scaling. This is
tested on 96boards Hikey960.

Changes from v1:
* Refactored structure definition according to Jassi's suggestion;
* Refactored and simplized mailbox driver with "automatic ack" mode;
* Refined commit logs to give background info for driver;
* Added document for DT binding;
* Added cover letter to track the changelog.


Kaihua Zhong (2):
  mailbox: Add support for Hi3660 mailbox
  dts: arm64: Add mailbox binding for hi3660

Leo Yan (1):
  dt-bindings: mailbox: Introduce Hi3660 controller binding

 .../bindings/mailbox/hisilicon,hi3660-mailbox.txt  |  52 ++++
 arch/arm64/boot/dts/hisilicon/hi3660.dtsi          |   8 +
 drivers/mailbox/Kconfig                            |   8 +
 drivers/mailbox/Makefile                           |   2 +
 drivers/mailbox/hi3660-mailbox.c                   | 331 +++++++++++++++++++++
 5 files changed, 401 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
 create mode 100644 drivers/mailbox/hi3660-mailbox.c

-- 
1.9.1

[PATCH v2 1/3] dt-bindings: mailbox: Introduce Hi3660 controller binding

[Kaihua Zhong]

From: Leo Yan <leo.yan@linaro.org>

Introduce a binding for the Hi3660 mailbox controller, the mailbox is
used within application processor (AP), communication processor (CP),
HIFI and MCU, etc.

Cc: John Stultz <john.stultz@linaro.org>
Cc: Guodong Xu <guodong.xu@linaro.org>
Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
Cc: Niranjan Yadla <nyadla@cadence.com>
Cc: Raj Pawate <pawateb@cadence.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 .../bindings/mailbox/hisilicon,hi3660-mailbox.txt  | 52 ++++++++++++++++++++++
 1 file changed, 52 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt

diff --git a/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt b/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
new file mode 100644
index 0000000..8a8d7e1
--- /dev/null
+++ b/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
@@ -0,0 +1,52 @@
+Hisilicon Hi3660 Mailbox Driver
+
+Hisilicon Hi3660 mailbox controller supports up to 32 channels.  Messages
+are passed between processors, including application & communication
+processors, MCU, HIFI, etc.  Each channel is unidirectional and accessed
+by using MMIO registers; it supports maximum to 8 words message.
+
+Controller
+----------
+
+Required properties:
+- compatible:		: Shall be "hisilicon,hi3660-mbox"
+- reg:			: Offset and length of the device's register set
+- #mbox-cells:		: Must be 3
+			  <&phandle channel dst_irq ack_irq>
+			    phandle	: Label name of controller
+			    channel	: Channel number
+			    dst_irq	: Remote interrupt vector
+			    ack_irq	: Local interrupt vector
+
+- interrupts:		: Contains the two IRQ lines for mailbox.
+
+Example:
+
+mailbox: mailbox@e896b000 {
+	compatible = "hisilicon,hi3660-mbox";
+	reg = <0x0 0xe896b000 0x0 0x1000>;
+	interrupts = <0x0 0xc0 0x4>,
+		     <0x0 0xc1 0x4>;
+	#mbox-cells = <3>;
+};
+
+Client
+------
+
+Required properties:
+- compatible		: See the client docs
+- mboxes		: Standard property to specify a Mailbox (See ./mailbox.txt)
+			  Cells must match 'mbox-cells' (See Controller docs above)
+
+Optional properties
+- mbox-names		: Name given to channels seen in the 'mboxes' property.
+
+Example:
+
+stub_clock: stub_clock {
+	compatible = "hisilicon,hi3660-stub-clk";
+	reg = <0x0 0xe896b500 0x0 0x0100>;
+	#clock-cells = <1>;
+	mbox-names = "mbox-tx";
+	mboxes = <&mailbox 13 3 0>;
+};
-- 
1.9.1

[PATCH v2 2/3] mailbox: Add support for Hi3660 mailbox

[Kaihua Zhong]

Hi3660 mailbox controller is used to send message within multiple
processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
channel can only be used for single transferring direction.  Once the
channel is enabled, it needs to specify the destination interrupt and
acknowledge interrupt, these two interrupt vectors are used to create
the connection between the mailbox and interrupt controllers.

The application processor (or from point of view of kernel) is not the
only one master which can launch the data transferring, other
processors or MCU/DSP also can kick off the data transferring.  So this
driver implements a locking mechanism to support exclusive accessing.

The data transferring supports two modes, one is named as "automatic
acknowledge" mode so after send message the kernel doesn't need to wait
for acknowledge from remote and directly return; there have another mode
is to rely on handling interrupt for acknowledge.

This commit is for initial version driver, which only supports
"automatic acknowledge" mode to support CPU clock, which is the only
one consumer to use mailbox and has been verified.  Later may enhance
this driver for interrupt mode (e.g. for supporting HIFI).

Cc: John Stultz <john.stultz@linaro.org>
Cc: Guodong Xu <guodong.xu@linaro.org>
Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
Cc: Niranjan Yadla <nyadla@cadence.com>
Cc: Raj Pawate <pawateb@cadence.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
Signed-off-by: Ruyi Wang <wangruyi@huawei.com>
Signed-off-by: Kaihua Zhong <zhongkaihua@huawei.com>
Signed-off-by: Kevin Wang <kevin.wangtao@hisilicon.com>
---
 drivers/mailbox/Kconfig          |   8 +
 drivers/mailbox/Makefile         |   2 +
 drivers/mailbox/hi3660-mailbox.c | 331 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 341 insertions(+)
 create mode 100644 drivers/mailbox/hi3660-mailbox.c

diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
index c5731e5..4b5d6e9 100644
--- a/drivers/mailbox/Kconfig
+++ b/drivers/mailbox/Kconfig
@@ -108,6 +108,14 @@ config TI_MESSAGE_MANAGER
 	  multiple processors within the SoC. Select this driver if your
 	  platform has support for the hardware block.
 
+config HI3660_MBOX
+	tristate "Hi3660 Mailbox"
+	depends on ARCH_HISI && OF
+	help
+	  An implementation of the hi3660 mailbox. It is used to send message
+	  between application processors and other processors/MCU/DSP. Select
+	  Y here if you want to use Hi3660 mailbox controller.
+
 config HI6220_MBOX
 	tristate "Hi6220 Mailbox"
 	depends on ARCH_HISI
diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
index d54e412..7d1bd51 100644
--- a/drivers/mailbox/Makefile
+++ b/drivers/mailbox/Makefile
@@ -26,6 +26,8 @@ obj-$(CONFIG_TI_MESSAGE_MANAGER) += ti-msgmgr.o
 
 obj-$(CONFIG_XGENE_SLIMPRO_MBOX) += mailbox-xgene-slimpro.o
 
+obj-$(CONFIG_HI3660_MBOX)	+= hi3660-mailbox.o
+
 obj-$(CONFIG_HI6220_MBOX)	+= hi6220-mailbox.o
 
 obj-$(CONFIG_BCM_PDC_MBOX)	+= bcm-pdc-mailbox.o
diff --git a/drivers/mailbox/hi3660-mailbox.c b/drivers/mailbox/hi3660-mailbox.c
new file mode 100644
index 0000000..67df8f8
--- /dev/null
+++ b/drivers/mailbox/hi3660-mailbox.c
@@ -0,0 +1,331 @@
+/*
+ * Hisilicon's Hi3660 mailbox controller driver
+ *
+ * Copyright (c) 2017 Hisilicon Limited.
+ * Copyright (c) 2017 Linaro Limited.
+ *
+ * Author: Leo Yan <leo.yan@linaro.org>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/bitops.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/mailbox_controller.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+
+#include "mailbox.h"
+
+#define MBOX_CHAN_MAX			32
+
+#define MBOX_RX				(0x0)
+#define MBOX_TX				(0x1)
+
+#define MBOX_BASE(mbox, ch)		((mbox)->base + ((ch) * 0x40))
+#define MBOX_SRC_REG			(0x00)
+#define MBOX_DST_REG			(0x04)
+#define MBOX_DCLR_REG			(0x08)
+#define MBOX_DSTAT_REG			(0x0c)
+#define MBOX_MODE_REG			(0x10)
+#define MBOX_IMASK_REG			(0x14)
+#define MBOX_ICLR_REG			(0x18)
+#define MBOX_SEND_REG			(0x1c)
+#define MBOX_DATA_REG			(0x20)
+
+#define MBOX_IPC_LOCK_REG		(0xa00)
+#define MBOX_IPC_UNLOCK			(0x1acce551)
+
+#define MBOX_AUTOMATIC_ACK		(1)
+
+#define MBOX_STATE_IDLE			BIT(4)
+#define MBOX_STATE_ACK			BIT(7)
+
+#define MBOX_MSG_LEN			8
+
+/**
+ * Hi3660 mailbox channel device data
+ *
+ * A channel can be used for TX or RX, it can trigger remote
+ * processor interrupt to notify remote processor and can receive
+ * interrupt if has incoming message.
+ *
+ * @dst_irq:	Interrupt vector for remote processor
+ * @ack_irq:	Interrupt vector for local processor
+ */
+struct hi3660_mbox_dev {
+	unsigned int dst_irq;
+	unsigned int ack_irq;
+};
+
+/**
+ * Hi3660 mailbox controller data
+ *
+ * Mailbox controller includes 32 channels and can allocate
+ * channel for message transferring.
+ *
+ * @dev:	Device to which it is attached
+ * @base:	Base address of the register mapping region
+ * @chan:	Representation of channels in mailbox controller
+ * @mdev:	Representation of channel device data
+ * @controller:	Representation of a communication channel controller
+ */
+struct hi3660_mbox {
+	struct device *dev;
+	void __iomem *base;
+	struct mbox_chan chan[MBOX_CHAN_MAX];
+	struct hi3660_mbox_dev mdev[MBOX_CHAN_MAX];
+	struct mbox_controller controller;
+};
+
+static inline struct hi3660_mbox *to_hi3660_mbox(struct mbox_chan *chan)
+{
+	return container_of(chan->mbox, struct hi3660_mbox, controller);
+}
+
+static int hi3660_mbox_check_state(struct mbox_chan *chan)
+{
+	unsigned long ch = (unsigned long)chan->con_priv;
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
+	void __iomem *base = MBOX_BASE(mbox, ch);
+	unsigned long val;
+	unsigned int state, ret;
+
+	/* Mailbox is idle so directly bail out */
+	state = readl_relaxed(base + MBOX_MODE_REG);
+	if (state & MBOX_STATE_IDLE)
+		return 0;
+
+	/* Wait for acknowledge from remote */
+	ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
+			val, (val & MBOX_STATE_ACK), 1000, 300000);
+	if (ret) {
+		dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
+		return ret;
+	}
+
+	/* Ensure channel is released */
+	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
+	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
+	__asm__ volatile ("sev");
+	return 0;
+}
+
+static int hi3660_mbox_unlock(struct mbox_chan *chan)
+{
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	unsigned int val, retry = 3;
+
+	do {
+		writel_relaxed(MBOX_IPC_UNLOCK, mbox->base + MBOX_IPC_LOCK_REG);
+
+		val = readl_relaxed(mbox->base + MBOX_IPC_LOCK_REG);
+		if (!val)
+			break;
+
+		udelay(10);
+	} while (retry--);
+
+	return (!val) ? 0 : -ETIMEDOUT;
+}
+
+static int hi3660_mbox_acquire_channel(struct mbox_chan *chan)
+{
+	unsigned long ch = (unsigned long)chan->con_priv;
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
+	void __iomem *base = MBOX_BASE(mbox, ch);
+	unsigned int val = 0, retry = 10;
+
+	/*
+	 * Hardware locking for exclusive accessing within CPUs
+	 * without exclusive monitor mechanism.
+	 */
+	do {
+		val = readl_relaxed(base + MBOX_MODE_REG);
+		if (!(val & MBOX_STATE_IDLE)) {
+			__asm__ volatile ("wfe");
+			continue;
+		}
+
+		/* Check if channel has be acquired */
+		writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
+		val = readl_relaxed(base + MBOX_SRC_REG) & BIT(mdev->ack_irq);
+		if (val)
+			break;
+
+	} while (retry--);
+
+	return (val) ? 0 : -ETIMEDOUT;
+}
+
+static int hi3660_mbox_send(struct mbox_chan *chan, u32 *msg)
+{
+	unsigned long ch = (unsigned long)chan->con_priv;
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
+	void __iomem *base = MBOX_BASE(mbox, ch);
+	unsigned int i;
+
+	/* Clear mask for destination interrupt */
+	writel_relaxed(~BIT(mdev->dst_irq), base + MBOX_IMASK_REG);
+
+	/* Config destination for interrupt vector */
+	writel_relaxed(BIT(mdev->dst_irq), base + MBOX_DST_REG);
+
+	/* Automatic acknowledge mode */
+	writel_relaxed(MBOX_AUTOMATIC_ACK, base + MBOX_MODE_REG);
+
+	/* Fill message data */
+	for (i = 0; i < MBOX_MSG_LEN; i++)
+		writel_relaxed(msg[i], base + MBOX_DATA_REG + i * 4);
+
+	/* Trigger data transferring */
+	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SEND_REG);
+	return 0;
+}
+
+static int hi3660_mbox_send_data(struct mbox_chan *chan, void *msg)
+{
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	int err;
+
+	err = hi3660_mbox_check_state(chan);
+	if (err) {
+		dev_err(mbox->dev, "checking state failed\n");
+		return err;
+	}
+
+	err = hi3660_mbox_unlock(chan);
+	if (err) {
+		dev_err(mbox->dev, "unlocking mailbox failed\n");
+		return err;
+	}
+
+	err = hi3660_mbox_acquire_channel(chan);
+	if (err) {
+		dev_err(mbox->dev, "acquiring channel failed\n");
+		return err;
+	}
+
+	return hi3660_mbox_send(chan, msg);
+}
+
+static struct mbox_chan_ops hi3660_mbox_ops = {
+	.send_data    = hi3660_mbox_send_data,
+};
+
+static struct mbox_chan *hi3660_mbox_xlate(struct mbox_controller *controller,
+					   const struct of_phandle_args *spec)
+{
+	struct hi3660_mbox *mbox = dev_get_drvdata(controller->dev);
+	struct hi3660_mbox_dev *mdev;
+	unsigned int ch = spec->args[0];
+
+	if (ch >= MBOX_CHAN_MAX) {
+		dev_err(mbox->dev, "Invalid channel idx %d\n", ch);
+		return ERR_PTR(-EINVAL);
+	}
+
+	mdev = &mbox->mdev[ch];
+	mdev->dst_irq = spec->args[1];
+	mdev->ack_irq = spec->args[2];
+
+	return &mbox->chan[ch];
+}
+
+static const struct of_device_id hi3660_mbox_of_match[] = {
+	{ .compatible = "hisilicon,hi3660-mbox", },
+	{},
+};
+
+MODULE_DEVICE_TABLE(of, hi3660_mbox_of_match);
+
+static int hi3660_mbox_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct hi3660_mbox *mbox;
+	struct mbox_chan *chan;
+	struct resource *res;
+	unsigned long ch;
+	int err;
+
+	mbox = devm_kzalloc(dev, sizeof(*mbox), GFP_KERNEL);
+	if (!mbox)
+		return -ENOMEM;
+
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+	mbox->base = devm_ioremap_resource(dev, res);
+	if (IS_ERR(mbox->base))
+		return PTR_ERR(mbox->base);
+
+	mbox->dev = dev;
+	mbox->controller.dev = dev;
+	mbox->controller.chans = mbox->chan;
+	mbox->controller.num_chans = MBOX_CHAN_MAX;
+	mbox->controller.ops = &hi3660_mbox_ops;
+	mbox->controller.of_xlate = hi3660_mbox_xlate;
+
+	/* Initialize mailbox channel data */
+	chan = mbox->chan;
+	for (ch = 0; ch < MBOX_CHAN_MAX; ch++)
+		chan[ch].con_priv = (void *)ch;
+
+	err = mbox_controller_register(&mbox->controller);
+	if (err) {
+		dev_err(dev, "Failed to register mailbox %d\n", err);
+		return err;
+	}
+
+	platform_set_drvdata(pdev, mbox);
+	dev_info(dev, "Mailbox enabled\n");
+	return 0;
+}
+
+static int hi3660_mbox_remove(struct platform_device *pdev)
+{
+	struct hi3660_mbox *mbox = platform_get_drvdata(pdev);
+
+	mbox_controller_unregister(&mbox->controller);
+	return 0;
+}
+
+static struct platform_driver hi3660_mbox_driver = {
+	.probe  = hi3660_mbox_probe,
+	.remove = hi3660_mbox_remove,
+	.driver = {
+		.name = "hi3660-mbox",
+		.of_match_table = hi3660_mbox_of_match,
+	},
+};
+
+static int __init hi3660_mbox_init(void)
+{
+	return platform_driver_register(&hi3660_mbox_driver);
+}
+core_initcall(hi3660_mbox_init);
+
+static void __exit hi3660_mbox_exit(void)
+{
+	platform_driver_unregister(&hi3660_mbox_driver);
+}
+module_exit(hi3660_mbox_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Hisilicon Hi3660 Mailbox Controller");
+MODULE_AUTHOR("Leo Yan <leo.yan@linaro.org>");
-- 
1.9.1

[PATCH v2 3/3] dts: arm64: Add mailbox binding for hi3660

[Kaihua Zhong]

Add DT binding for mailbox driver.

Cc: John Stultz <john.stultz@linaro.org>
Cc: Guodong Xu <guodong.xu@linaro.org>
Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
Cc: Niranjan Yadla <nyadla@cadence.com>
Cc: Raj Pawate <pawateb@cadence.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
Signed-off-by: Ruyi Wang <wangruyi@huawei.com>
Signed-off-by: Kaihua Zhong <zhongkaihua@huawei.com>
---
 arch/arm64/boot/dts/hisilicon/hi3660.dtsi | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/arm64/boot/dts/hisilicon/hi3660.dtsi b/arch/arm64/boot/dts/hisilicon/hi3660.dtsi
index 610990f..451b6bf 100644
--- a/arch/arm64/boot/dts/hisilicon/hi3660.dtsi
+++ b/arch/arm64/boot/dts/hisilicon/hi3660.dtsi
@@ -285,6 +285,14 @@
 			#reset-cells = <2>;
 		};
 
+		mailbox: mailbox@e896b000 {
+			compatible = "hisilicon,hi3660-mbox";
+			reg = <0x0 0xe896b000 0x0 0x1000>;
+			interrupts = <GIC_SPI 192 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 193 IRQ_TYPE_LEVEL_HIGH>;
+			#mbox-cells = <3>;
+		};
+
 		dual_timer0: timer@fff14000 {
 			compatible = "arm,sp804", "arm,primecell";
 			reg = <0x0 0xfff14000 0x0 0x1000>;
-- 
1.9.1

Re: [PATCH v2 2/3] mailbox: Add support for Hi3660 mailbox

[Julien Thierry]

Hi Kaihua,

On 27/10/17 07:15, Kaihua Zhong wrote:
> Hi3660 mailbox controller is used to send message within multiple
> processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
> channel can only be used for single transferring direction.  Once the
> channel is enabled, it needs to specify the destination interrupt and
> acknowledge interrupt, these two interrupt vectors are used to create
> the connection between the mailbox and interrupt controllers.
> 
> The application processor (or from point of view of kernel) is not the
> only one master which can launch the data transferring, other
> processors or MCU/DSP also can kick off the data transferring.  So this
> driver implements a locking mechanism to support exclusive accessing.
> 
> The data transferring supports two modes, one is named as "automatic
> acknowledge" mode so after send message the kernel doesn't need to wait
> for acknowledge from remote and directly return; there have another mode
> is to rely on handling interrupt for acknowledge.
> 
> This commit is for initial version driver, which only supports
> "automatic acknowledge" mode to support CPU clock, which is the only
> one consumer to use mailbox and has been verified.  Later may enhance
> this driver for interrupt mode (e.g. for supporting HIFI).
> 
> Cc: John Stultz <john.stultz@linaro.org>
> Cc: Guodong Xu <guodong.xu@linaro.org>
> Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
> Cc: Niranjan Yadla <nyadla@cadence.com>
> Cc: Raj Pawate <pawateb@cadence.com>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>
> Signed-off-by: Ruyi Wang <wangruyi@huawei.com>
> Signed-off-by: Kaihua Zhong <zhongkaihua@huawei.com>
> Signed-off-by: Kevin Wang <kevin.wangtao@hisilicon.com>
> ---
>   drivers/mailbox/Kconfig          |   8 +
>   drivers/mailbox/Makefile         |   2 +
>   drivers/mailbox/hi3660-mailbox.c | 331 +++++++++++++++++++++++++++++++++++++++
>   3 files changed, 341 insertions(+)
>   create mode 100644 drivers/mailbox/hi3660-mailbox.c
> 
> diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
> index c5731e5..4b5d6e9 100644
> --- a/drivers/mailbox/Kconfig
> +++ b/drivers/mailbox/Kconfig
> @@ -108,6 +108,14 @@ config TI_MESSAGE_MANAGER
>   	  multiple processors within the SoC. Select this driver if your
>   	  platform has support for the hardware block.
>   
> +config HI3660_MBOX
> +	tristate "Hi3660 Mailbox"
> +	depends on ARCH_HISI && OF
> +	help
> +	  An implementation of the hi3660 mailbox. It is used to send message
> +	  between application processors and other processors/MCU/DSP. Select
> +	  Y here if you want to use Hi3660 mailbox controller.
> +
>   config HI6220_MBOX
>   	tristate "Hi6220 Mailbox"
>   	depends on ARCH_HISI
> diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
> index d54e412..7d1bd51 100644
> --- a/drivers/mailbox/Makefile
> +++ b/drivers/mailbox/Makefile
> @@ -26,6 +26,8 @@ obj-$(CONFIG_TI_MESSAGE_MANAGER) += ti-msgmgr.o
>   
>   obj-$(CONFIG_XGENE_SLIMPRO_MBOX) += mailbox-xgene-slimpro.o
>   
> +obj-$(CONFIG_HI3660_MBOX)	+= hi3660-mailbox.o
> +
>   obj-$(CONFIG_HI6220_MBOX)	+= hi6220-mailbox.o
>   
>   obj-$(CONFIG_BCM_PDC_MBOX)	+= bcm-pdc-mailbox.o
> diff --git a/drivers/mailbox/hi3660-mailbox.c b/drivers/mailbox/hi3660-mailbox.c
> new file mode 100644
> index 0000000..67df8f8
> --- /dev/null
> +++ b/drivers/mailbox/hi3660-mailbox.c
> @@ -0,0 +1,331 @@
> +/*
> + * Hisilicon's Hi3660 mailbox controller driver
> + *
> + * Copyright (c) 2017 Hisilicon Limited.
> + * Copyright (c) 2017 Linaro Limited.
> + *
> + * Author: Leo Yan <leo.yan@linaro.org>
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, version 2 of the License.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + */
> +
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/err.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iopoll.h>
> +#include <linux/mailbox_controller.h>
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <linux/slab.h>
> +
> +#include "mailbox.h"
> +
> +#define MBOX_CHAN_MAX			32
> +
> +#define MBOX_RX				(0x0)
> +#define MBOX_TX				(0x1)
> +
> +#define MBOX_BASE(mbox, ch)		((mbox)->base + ((ch) * 0x40))
> +#define MBOX_SRC_REG			(0x00)
> +#define MBOX_DST_REG			(0x04)
> +#define MBOX_DCLR_REG			(0x08)
> +#define MBOX_DSTAT_REG			(0x0c)
> +#define MBOX_MODE_REG			(0x10)
> +#define MBOX_IMASK_REG			(0x14)
> +#define MBOX_ICLR_REG			(0x18)
> +#define MBOX_SEND_REG			(0x1c)
> +#define MBOX_DATA_REG			(0x20)
> +
> +#define MBOX_IPC_LOCK_REG		(0xa00)
> +#define MBOX_IPC_UNLOCK			(0x1acce551)
> +
> +#define MBOX_AUTOMATIC_ACK		(1)
> +
> +#define MBOX_STATE_IDLE			BIT(4)
> +#define MBOX_STATE_ACK			BIT(7)
> +
> +#define MBOX_MSG_LEN			8
> +
> +/**
> + * Hi3660 mailbox channel device data
> + *
> + * A channel can be used for TX or RX, it can trigger remote
> + * processor interrupt to notify remote processor and can receive
> + * interrupt if has incoming message.
> + *
> + * @dst_irq:	Interrupt vector for remote processor
> + * @ack_irq:	Interrupt vector for local processor
> + */
> +struct hi3660_mbox_dev {
> +	unsigned int dst_irq;
> +	unsigned int ack_irq;
> +};
> +
> +/**
> + * Hi3660 mailbox controller data
> + *
> + * Mailbox controller includes 32 channels and can allocate
> + * channel for message transferring.
> + *
> + * @dev:	Device to which it is attached
> + * @base:	Base address of the register mapping region
> + * @chan:	Representation of channels in mailbox controller
> + * @mdev:	Representation of channel device data
> + * @controller:	Representation of a communication channel controller
> + */
> +struct hi3660_mbox {
> +	struct device *dev;
> +	void __iomem *base;
> +	struct mbox_chan chan[MBOX_CHAN_MAX];
> +	struct hi3660_mbox_dev mdev[MBOX_CHAN_MAX];
> +	struct mbox_controller controller;
> +};
> +
> +static inline struct hi3660_mbox *to_hi3660_mbox(struct mbox_chan *chan)
> +{
> +	return container_of(chan->mbox, struct hi3660_mbox, controller);
> +}
> +
> +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned long val;
> +	unsigned int state, ret;
> +
> +	/* Mailbox is idle so directly bail out */
> +	state = readl_relaxed(base + MBOX_MODE_REG);
> +	if (state & MBOX_STATE_IDLE)
> +		return 0;
> +
> +	/* Wait for acknowledge from remote */
> +	ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
> +			val, (val & MBOX_STATE_ACK), 1000, 300000);
> +	if (ret) {
> +		dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
> +		return ret;
> +	}
> +
> +	/* Ensure channel is released */
> +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> +	__asm__ volatile ("sev");

There is an existing sev() macro for this.

> +	return 0;
> +}
> +
> +static int hi3660_mbox_unlock(struct mbox_chan *chan)
> +{
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	unsigned int val, retry = 3;
> +
> +	do {
> +		writel_relaxed(MBOX_IPC_UNLOCK, mbox->base + MBOX_IPC_LOCK_REG);
> +
> +		val = readl_relaxed(mbox->base + MBOX_IPC_LOCK_REG);
> +		if (!val)
> +			break;
> +
> +		udelay(10);
> +	} while (retry--);
> +
> +	return (!val) ? 0 : -ETIMEDOUT;
> +}
> +
> +static int hi3660_mbox_acquire_channel(struct mbox_chan *chan)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned int val = 0, retry = 10;
> +
> +	/*
> +	 * Hardware locking for exclusive accessing within CPUs
> +	 * without exclusive monitor mechanism.
> +	 */
> +	do {
> +		val = readl_relaxed(base + MBOX_MODE_REG);
> +		if (!(val & MBOX_STATE_IDLE)) {
> +			__asm__ volatile ("wfe");
> +			continue;

So this is going to "wfe" when retry == 0 and the continue statement 
will take us out of the loop?

Also, there is a wfe() macro for arm and arm64 which could be used here.

> +		}
> +
> +		/* Check if channel has be acquired */
> +		writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> +		val = readl_relaxed(base + MBOX_SRC_REG) & BIT(mdev->ack_irq);
> +		if (val)
> +			break;
> +
> +	} while (retry--);
> +
> +	return (val) ? 0 : -ETIMEDOUT;

If timeout occurs while waiting for the MBOX to be idle, val will hold 
the last value from MBOX_MODE_REG, and if this can be different than 0, 
this will hide the fact it timed out.

Maybe the following would be more appropriate:

return (retry >= 0) ? 0 : -ETIMEDOUT;

Also, your do {} while loops for the timeouts falsely give the 
impression we do "retry" attempts when we actually do "retry + 1" 
attempts (but I guess it doesn't make a big difference from the 
functional point of view).

> +}
> +
> +static int hi3660_mbox_send(struct mbox_chan *chan, u32 *msg)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned int i;
> +
> +	/* Clear mask for destination interrupt */
> +	writel_relaxed(~BIT(mdev->dst_irq), base + MBOX_IMASK_REG);
> +
> +	/* Config destination for interrupt vector */
> +	writel_relaxed(BIT(mdev->dst_irq), base + MBOX_DST_REG);
> +
> +	/* Automatic acknowledge mode */
> +	writel_relaxed(MBOX_AUTOMATIC_ACK, base + MBOX_MODE_REG);
> +
> +	/* Fill message data */
> +	for (i = 0; i < MBOX_MSG_LEN; i++)
> +		writel_relaxed(msg[i], base + MBOX_DATA_REG + i * 4);
> +
> +	/* Trigger data transferring */
> +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SEND_REG);
> +	return 0;
> +}
> +
> +static int hi3660_mbox_send_data(struct mbox_chan *chan, void *msg)
> +{
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	int err;
> +
> +	err = hi3660_mbox_check_state(chan);
> +	if (err) {
> +		dev_err(mbox->dev, "checking state failed\n");
> +		return err;
> +	}
> +
> +	err = hi3660_mbox_unlock(chan);
> +	if (err) {
> +		dev_err(mbox->dev, "unlocking mailbox failed\n");
> +		return err;
> +	}
> +
> +	err = hi3660_mbox_acquire_channel(chan);
> +	if (err) {
> +		dev_err(mbox->dev, "acquiring channel failed\n");
> +		return err;
> +	}
> +
> +	return hi3660_mbox_send(chan, msg);
> +}
> +
> +static struct mbox_chan_ops hi3660_mbox_ops = {
> +	.send_data    = hi3660_mbox_send_data,
> +};
> +
> +static struct mbox_chan *hi3660_mbox_xlate(struct mbox_controller *controller,
> +					   const struct of_phandle_args *spec)
> +{
> +	struct hi3660_mbox *mbox = dev_get_drvdata(controller->dev);

nit:
In to_hi3660_mbox, you use the fact that the mbox_controller structure 
is part of hi3660_mbox and use container of to retrieve hi3660_mbox.

I think it would be nice to be consistent about this, either use 
container_of or dev drvdata, but not both.

Cheers,

-- 
Julien Thierry

Re: [PATCH v2 2/3] mailbox: Add support for Hi3660 mailbox

[Mark Rutland]

On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> Hi3660 mailbox controller is used to send message within multiple
> processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
> channel can only be used for single transferring direction.  Once the
> channel is enabled, it needs to specify the destination interrupt and
> acknowledge interrupt, these two interrupt vectors are used to create
> the connection between the mailbox and interrupt controllers.
> 
> The application processor (or from point of view of kernel) is not the
> only one master which can launch the data transferring, other
> processors or MCU/DSP also can kick off the data transferring.  So this
> driver implements a locking mechanism to support exclusive accessing.

... and that locking mechanism is what precisely?

Where is the protocol defined?

> +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned long val;
> +	unsigned int state, ret;
> +
> +	/* Mailbox is idle so directly bail out */
> +	state = readl_relaxed(base + MBOX_MODE_REG);
> +	if (state & MBOX_STATE_IDLE)
> +		return 0;
> +
> +	/* Wait for acknowledge from remote */
> +	ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
> +			val, (val & MBOX_STATE_ACK), 1000, 300000);
> +	if (ret) {
> +		dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
> +		return ret;
> +	}
> +
> +	/* Ensure channel is released */
> +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> +	__asm__ volatile ("sev");
> +	return 0;
> +}

Drivers really shouldn't be using SEV directly (even if via the sev() macro)...

This SEV isn't ordered w.r.t. anything, and it's unclear what ordering you
need, so this simply does not work.

> +static int hi3660_mbox_acquire_channel(struct mbox_chan *chan)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned int val = 0, retry = 10;
> +
> +	/*
> +	 * Hardware locking for exclusive accessing within CPUs
> +	 * without exclusive monitor mechanism.
> +	 */
> +	do {
> +		val = readl_relaxed(base + MBOX_MODE_REG);
> +		if (!(val & MBOX_STATE_IDLE)) {
> +			__asm__ volatile ("wfe");

... and likewise for WFE / wfe().

> +			continue;
> +		}
> +
> +		/* Check if channel has be acquired */
> +		writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> +		val = readl_relaxed(base + MBOX_SRC_REG) & BIT(mdev->ack_irq);
> +		if (val)
> +			break;
> +
> +	} while (retry--);
> +
> +	return (val) ? 0 : -ETIMEDOUT;
> +}

Please elaborate on how this synchronisation mechanism is expected to work, and
why it is necessary in the first place.

Thanks,
Mark.

Re: [PATCH v2 1/3] dt-bindings: mailbox: Introduce Hi3660 controller binding

[Rob Herring]

On Fri, Oct 27, 2017 at 02:15:02PM +0800, Kaihua Zhong wrote:
> From: Leo Yan <leo.yan@linaro.org>
> 
> Introduce a binding for the Hi3660 mailbox controller, the mailbox is
> used within application processor (AP), communication processor (CP),
> HIFI and MCU, etc.
> 
> Cc: John Stultz <john.stultz@linaro.org>
> Cc: Guodong Xu <guodong.xu@linaro.org>
> Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
> Cc: Niranjan Yadla <nyadla@cadence.com>
> Cc: Raj Pawate <pawateb@cadence.com>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>
> ---
>  .../bindings/mailbox/hisilicon,hi3660-mailbox.txt  | 52 ++++++++++++++++++++++
>  1 file changed, 52 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
> 
> diff --git a/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt b/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
> new file mode 100644
> index 0000000..8a8d7e1
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
> @@ -0,0 +1,52 @@
> +Hisilicon Hi3660 Mailbox Driver

Bindings are for h/w, not drivers.

> +
> +Hisilicon Hi3660 mailbox controller supports up to 32 channels.  Messages
> +are passed between processors, including application & communication
> +processors, MCU, HIFI, etc.  Each channel is unidirectional and accessed
> +by using MMIO registers; it supports maximum to 8 words message.
> +
> +Controller
> +----------
> +
> +Required properties:
> +- compatible:		: Shall be "hisilicon,hi3660-mbox"
> +- reg:			: Offset and length of the device's register set
> +- #mbox-cells:		: Must be 3
> +			  <&phandle channel dst_irq ack_irq>
> +			    phandle	: Label name of controller
> +			    channel	: Channel number
> +			    dst_irq	: Remote interrupt vector
> +			    ack_irq	: Local interrupt vector
> +
> +- interrupts:		: Contains the two IRQ lines for mailbox.
> +
> +Example:
> +
> +mailbox: mailbox@e896b000 {
> +	compatible = "hisilicon,hi3660-mbox";
> +	reg = <0x0 0xe896b000 0x0 0x1000>;
> +	interrupts = <0x0 0xc0 0x4>,
> +		     <0x0 0xc1 0x4>;
> +	#mbox-cells = <3>;
> +};
> +
> +Client
> +------
> +
> +Required properties:
> +- compatible		: See the client docs
> +- mboxes		: Standard property to specify a Mailbox (See ./mailbox.txt)
> +			  Cells must match 'mbox-cells' (See Controller docs above)
> +
> +Optional properties
> +- mbox-names		: Name given to channels seen in the 'mboxes' property.
> +
> +Example:
> +
> +stub_clock: stub_clock {

clock@e896b500

> +	compatible = "hisilicon,hi3660-stub-clk";
> +	reg = <0x0 0xe896b500 0x0 0x0100>;
> +	#clock-cells = <1>;
> +	mbox-names = "mbox-tx";

Don't need -names when there is only one. Plus "mbox-" part is 
redundant.

> +	mboxes = <&mailbox 13 3 0>;
> +};
> -- 
> 1.9.1
> 

Re: [PATCH v2 1/3] dt-bindings: mailbox: Introduce Hi3660 controller binding

[Leo Yan]

On Fri, Oct 27, 2017 at 09:38:44AM -0500, Rob Herring wrote:
> On Fri, Oct 27, 2017 at 02:15:02PM +0800, Kaihua Zhong wrote:
> > From: Leo Yan <leo.yan@linaro.org>
> > 
> > Introduce a binding for the Hi3660 mailbox controller, the mailbox is
> > used within application processor (AP), communication processor (CP),
> > HIFI and MCU, etc.
> > 
> > Cc: John Stultz <john.stultz@linaro.org>
> > Cc: Guodong Xu <guodong.xu@linaro.org>
> > Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
> > Cc: Niranjan Yadla <nyadla@cadence.com>
> > Cc: Raj Pawate <pawateb@cadence.com>
> > Signed-off-by: Leo Yan <leo.yan@linaro.org>
> > ---
> >  .../bindings/mailbox/hisilicon,hi3660-mailbox.txt  | 52 ++++++++++++++++++++++
> >  1 file changed, 52 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
> > 
> > diff --git a/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt b/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
> > new file mode 100644
> > index 0000000..8a8d7e1
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/mailbox/hisilicon,hi3660-mailbox.txt
> > @@ -0,0 +1,52 @@
> > +Hisilicon Hi3660 Mailbox Driver
> 
> Bindings are for h/w, not drivers.

Thanks for reviewing, Rob.

Will fix for this and below comments.

> > +
> > +Hisilicon Hi3660 mailbox controller supports up to 32 channels.  Messages
> > +are passed between processors, including application & communication
> > +processors, MCU, HIFI, etc.  Each channel is unidirectional and accessed
> > +by using MMIO registers; it supports maximum to 8 words message.
> > +
> > +Controller
> > +----------
> > +
> > +Required properties:
> > +- compatible:		: Shall be "hisilicon,hi3660-mbox"
> > +- reg:			: Offset and length of the device's register set
> > +- #mbox-cells:		: Must be 3
> > +			  <&phandle channel dst_irq ack_irq>
> > +			    phandle	: Label name of controller
> > +			    channel	: Channel number
> > +			    dst_irq	: Remote interrupt vector
> > +			    ack_irq	: Local interrupt vector
> > +
> > +- interrupts:		: Contains the two IRQ lines for mailbox.
> > +
> > +Example:
> > +
> > +mailbox: mailbox@e896b000 {
> > +	compatible = "hisilicon,hi3660-mbox";
> > +	reg = <0x0 0xe896b000 0x0 0x1000>;
> > +	interrupts = <0x0 0xc0 0x4>,
> > +		     <0x0 0xc1 0x4>;
> > +	#mbox-cells = <3>;
> > +};
> > +
> > +Client
> > +------
> > +
> > +Required properties:
> > +- compatible		: See the client docs
> > +- mboxes		: Standard property to specify a Mailbox (See ./mailbox.txt)
> > +			  Cells must match 'mbox-cells' (See Controller docs above)
> > +
> > +Optional properties
> > +- mbox-names		: Name given to channels seen in the 'mboxes' property.
> > +
> > +Example:
> > +
> > +stub_clock: stub_clock {
> 
> clock@e896b500
> 
> > +	compatible = "hisilicon,hi3660-stub-clk";
> > +	reg = <0x0 0xe896b500 0x0 0x0100>;
> > +	#clock-cells = <1>;
> > +	mbox-names = "mbox-tx";
> 
> Don't need -names when there is only one. Plus "mbox-" part is 
> redundant.
> 
> > +	mboxes = <&mailbox 13 3 0>;
> > +};
> > -- 
> > 1.9.1
> > 

Re: [PATCH v2 2/3] mailbox: Add support for Hi3660 mailbox

[Leo Yan]

Hi Mark,

On Fri, Oct 27, 2017 at 11:46:00AM +0100, Mark Rutland wrote:
> On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> > Hi3660 mailbox controller is used to send message within multiple
> > processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
> > channel can only be used for single transferring direction.  Once the
> > channel is enabled, it needs to specify the destination interrupt and
> > acknowledge interrupt, these two interrupt vectors are used to create
> > the connection between the mailbox and interrupt controllers.
> > 
> > The application processor (or from point of view of kernel) is not the
> > only one master which can launch the data transferring, other
> > processors or MCU/DSP also can kick off the data transferring.  So this
> > driver implements a locking mechanism to support exclusive accessing.
> 
> ... and that locking mechanism is what precisely?
> 
> Where is the protocol defined?
> 
> > +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> > +{
> > +	unsigned long ch = (unsigned long)chan->con_priv;
> > +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> > +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> > +	void __iomem *base = MBOX_BASE(mbox, ch);
> > +	unsigned long val;
> > +	unsigned int state, ret;
> > +
> > +	/* Mailbox is idle so directly bail out */
> > +	state = readl_relaxed(base + MBOX_MODE_REG);
> > +	if (state & MBOX_STATE_IDLE)
> > +		return 0;
> > +
> > +	/* Wait for acknowledge from remote */
> > +	ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
> > +			val, (val & MBOX_STATE_ACK), 1000, 300000);
> > +	if (ret) {
> > +		dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
> > +		return ret;
> > +	}
> > +
> > +	/* Ensure channel is released */
> > +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> > +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> > +	__asm__ volatile ("sev");
> > +	return 0;
> > +}
> 
> Drivers really shouldn't be using SEV directly (even if via the sev() macro)...
> 
> This SEV isn't ordered w.r.t. anything, and it's unclear what ordering you
> need, so this simply does not work.

I will leave your questions for Hisilicon colleagues, essentially
your questions are related with mailbox mechanism.

But I'd like to firstly get clear your question for "This SEV isn't
ordered w.r.t. anything". From my understanding, ARMv8 architecture
natually adds DMB before SEV so all previous register writing
opreations should be ensured to endpoint before SEV?

[...]

Thanks,
Leo Yan

Re: [PATCH v2 2/3] mailbox: Add support for Hi3660 mailbox

[Mark Rutland]

Hi,

On Mon, Oct 30, 2017 at 12:45:06PM +0800, Leo Yan wrote:
> On Fri, Oct 27, 2017 at 11:46:00AM +0100, Mark Rutland wrote:
> > On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> > > +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> > > +{

> > > +	/* Ensure channel is released */
> > > +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> > > +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> > > +	__asm__ volatile ("sev");
> > > +	return 0;
> > > +}
> > 
> > Drivers really shouldn't be using SEV directly (even if via the
> > sev() macro)...
> > 
> > This SEV isn't ordered w.r.t. anything, and it's unclear what
> > ordering you need, so this simply does not work.
> 
> I will leave your questions for Hisilicon colleagues, essentially your
> questions are related with mailbox mechanism.
> 
> But I'd like to firstly get clear your question for "This SEV isn't
> ordered w.r.t. anything". From my understanding, ARMv8 architecture
> natually adds DMB before SEV so all previous register writing
> opreations should be ensured to endpoint before SEV?

This is not the case; SEV does not add any implicit memory barrier, and
is not ordered w.r.t. memory accesses.

See ARM DDI 0487B.b, page D1-1905, "The Send Event instructions":

    The PE is not required to guarantee the ordering of this event with
    respect to the completion of memory accesses by instructions before
    the SEV instruction. Therefore, ARM recommends that software
    includes a DSB instruction before any SEV instruction.

Note that a DMB is not sufficient, as SEV is not a memory access.

Thanks,
Mark.

Re: [PATCH v2 2/3] mailbox: Add support for Hi3660 mailbox

[Leo Yan]

Hi Mark,

On Mon, Oct 30, 2017 at 10:19:40AM +0000, Mark Rutland wrote:
> Hi,
> 
> On Mon, Oct 30, 2017 at 12:45:06PM +0800, Leo Yan wrote:
> > On Fri, Oct 27, 2017 at 11:46:00AM +0100, Mark Rutland wrote:
> > > On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> > > > +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> > > > +{
> 
> > > > +	/* Ensure channel is released */
> > > > +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> > > > +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> > > > +	__asm__ volatile ("sev");
> > > > +	return 0;
> > > > +}
> > > 
> > > Drivers really shouldn't be using SEV directly (even if via the
> > > sev() macro)...
> > > 
> > > This SEV isn't ordered w.r.t. anything, and it's unclear what
> > > ordering you need, so this simply does not work.
> > 
> > I will leave your questions for Hisilicon colleagues, essentially your
> > questions are related with mailbox mechanism.
> > 
> > But I'd like to firstly get clear your question for "This SEV isn't
> > ordered w.r.t. anything". From my understanding, ARMv8 architecture
> > natually adds DMB before SEV so all previous register writing
> > opreations should be ensured to endpoint before SEV?
> 
> This is not the case; SEV does not add any implicit memory barrier, and
> is not ordered w.r.t. memory accesses.
> 
> See ARM DDI 0487B.b, page D1-1905, "The Send Event instructions":
> 
>     The PE is not required to guarantee the ordering of this event with
>     respect to the completion of memory accesses by instructions before
>     the SEV instruction. Therefore, ARM recommends that software
>     includes a DSB instruction before any SEV instruction.

My fault and thanks for explanation.

> Note that a DMB is not sufficient, as SEV is not a memory access.

Understood now, so below code should be safe?

wmb();  -> dsb(st);
sev();

Thanks,
Leo Yan

Re: [PATCH v2 2/3] mailbox: Add support for Hi3660 mailbox

[Mark Rutland]

On Mon, Oct 30, 2017 at 07:13:13PM +0800, Leo Yan wrote:
> Hi Mark,
> 
> On Mon, Oct 30, 2017 at 10:19:40AM +0000, Mark Rutland wrote:
> > Hi,
> > 
> > On Mon, Oct 30, 2017 at 12:45:06PM +0800, Leo Yan wrote:
> > > On Fri, Oct 27, 2017 at 11:46:00AM +0100, Mark Rutland wrote:
> > > > On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> > > > > +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> > > > > +{
> > 
> > > > > +	/* Ensure channel is released */
> > > > > +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> > > > > +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> > > > > +	__asm__ volatile ("sev");
> > > > > +	return 0;
> > > > > +}
> > > > 
> > > > Drivers really shouldn't be using SEV directly (even if via the
> > > > sev() macro)...
> > > > 
> > > > This SEV isn't ordered w.r.t. anything, and it's unclear what
> > > > ordering you need, so this simply does not work.
> > > 
> > > I will leave your questions for Hisilicon colleagues, essentially your
> > > questions are related with mailbox mechanism.
> > > 
> > > But I'd like to firstly get clear your question for "This SEV isn't
> > > ordered w.r.t. anything". From my understanding, ARMv8 architecture
> > > natually adds DMB before SEV so all previous register writing
> > > opreations should be ensured to endpoint before SEV?
> > 
> > This is not the case; SEV does not add any implicit memory barrier, and
> > is not ordered w.r.t. memory accesses.
> > 
> > See ARM DDI 0487B.b, page D1-1905, "The Send Event instructions":
> > 
> >     The PE is not required to guarantee the ordering of this event with
> >     respect to the completion of memory accesses by instructions before
> >     the SEV instruction. Therefore, ARM recommends that software
> >     includes a DSB instruction before any SEV instruction.
> 
> My fault and thanks for explanation.
> 
> > Note that a DMB is not sufficient, as SEV is not a memory access.
> 
> Understood now, so below code should be safe?
> 
> wmb();  -> dsb(st);
> sev();

Whether that is safe depends on what you are trying to ensure is
ordered, and what the other side (with the WFE) is doing.

For example, my understanding is that in general, WFE is also not
ordered w.r.t.  memory accesses.

This is a very subtle part of the architecture, and I'm very much not
keen on using WFE and SEV outside of architecture code implementing
locking primitives.

Thanks,
Mark.

Re: [PATCH v2 2/3] mailbox: Add support for Hi3660 mailbox

[Zhong Kaihua]

Hi, Julien,

On 2017/10/27 18:40, Julien Thierry wrote:
> Hi Kaihua,
> 
> On 27/10/17 07:15, Kaihua Zhong wrote:
>> Hi3660 mailbox controller is used to send message within multiple
>> processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
>> channel can only be used for single transferring direction.  Once the
>> channel is enabled, it needs to specify the destination interrupt and
>> acknowledge interrupt, these two interrupt vectors are used to create
>> the connection between the mailbox and interrupt controllers.
>>
>> The application processor (or from point of view of kernel) is not the
>> only one master which can launch the data transferring, other
>> processors or MCU/DSP also can kick off the data transferring.  So this
>> driver implements a locking mechanism to support exclusive accessing.
>>
>> The data transferring supports two modes, one is named as "automatic
>> acknowledge" mode so after send message the kernel doesn't need to wait
>> for acknowledge from remote and directly return; there have another mode
>> is to rely on handling interrupt for acknowledge.
>>
>> This commit is for initial version driver, which only supports
>> "automatic acknowledge" mode to support CPU clock, which is the only
>> one consumer to use mailbox and has been verified.  Later may enhance
>> this driver for interrupt mode (e.g. for supporting HIFI).
>>
>> Cc: John Stultz <john.stultz@linaro.org>
>> Cc: Guodong Xu <guodong.xu@linaro.org>
>> Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
>> Cc: Niranjan Yadla <nyadla@cadence.com>
>> Cc: Raj Pawate <pawateb@cadence.com>
>> Signed-off-by: Leo Yan <leo.yan@linaro.org>
>> Signed-off-by: Ruyi Wang <wangruyi@huawei.com>
>> Signed-off-by: Kaihua Zhong <zhongkaihua@huawei.com>
>> Signed-off-by: Kevin Wang <kevin.wangtao@hisilicon.com>
>> ---
>>   drivers/mailbox/Kconfig          |   8 +
>>   drivers/mailbox/Makefile         |   2 +
>>   drivers/mailbox/hi3660-mailbox.c | 331 +++++++++++++++++++++++++++++++++++++++
>>   3 files changed, 341 insertions(+)
>>   create mode 100644 drivers/mailbox/hi3660-mailbox.c
>>
>> diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
>> index c5731e5..4b5d6e9 100644
>> --- a/drivers/mailbox/Kconfig
>> +++ b/drivers/mailbox/Kconfig
>> @@ -108,6 +108,14 @@ config TI_MESSAGE_MANAGER
>>         multiple processors within the SoC. Select this driver if your
>>         platform has support for the hardware block.
>>   +config HI3660_MBOX
>> +    tristate "Hi3660 Mailbox"
>> +    depends on ARCH_HISI && OF
>> +    help
>> +      An implementation of the hi3660 mailbox. It is used to send message
>> +      between application processors and other processors/MCU/DSP. Select
>> +      Y here if you want to use Hi3660 mailbox controller.
>> +
>>   config HI6220_MBOX
>>       tristate "Hi6220 Mailbox"
>>       depends on ARCH_HISI
>> diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
>> index d54e412..7d1bd51 100644
>> --- a/drivers/mailbox/Makefile
>> +++ b/drivers/mailbox/Makefile
>> @@ -26,6 +26,8 @@ obj-$(CONFIG_TI_MESSAGE_MANAGER) += ti-msgmgr.o
>>     obj-$(CONFIG_XGENE_SLIMPRO_MBOX) += mailbox-xgene-slimpro.o
>>   +obj-$(CONFIG_HI3660_MBOX)    += hi3660-mailbox.o
>> +
>>   obj-$(CONFIG_HI6220_MBOX)    += hi6220-mailbox.o
>>     obj-$(CONFIG_BCM_PDC_MBOX)    += bcm-pdc-mailbox.o
>> diff --git a/drivers/mailbox/hi3660-mailbox.c b/drivers/mailbox/hi3660-mailbox.c
>> new file mode 100644
>> index 0000000..67df8f8
>> --- /dev/null
>> +++ b/drivers/mailbox/hi3660-mailbox.c
>> @@ -0,0 +1,331 @@
>> +/*
>> + * Hisilicon's Hi3660 mailbox controller driver
>> + *
>> + * Copyright (c) 2017 Hisilicon Limited.
>> + * Copyright (c) 2017 Linaro Limited.
>> + *
>> + * Author: Leo Yan <leo.yan@linaro.org>
>> + *
>> + * This program is free software: you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation, version 2 of the License.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + */
>> +
>> +#include <linux/bitops.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/err.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/io.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/mailbox_controller.h>
>> +#include <linux/module.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/slab.h>
>> +
>> +#include "mailbox.h"
>> +
>> +#define MBOX_CHAN_MAX            32
>> +
>> +#define MBOX_RX                (0x0)
>> +#define MBOX_TX                (0x1)
>> +
>> +#define MBOX_BASE(mbox, ch)        ((mbox)->base + ((ch) * 0x40))
>> +#define MBOX_SRC_REG            (0x00)
>> +#define MBOX_DST_REG            (0x04)
>> +#define MBOX_DCLR_REG            (0x08)
>> +#define MBOX_DSTAT_REG            (0x0c)
>> +#define MBOX_MODE_REG            (0x10)
>> +#define MBOX_IMASK_REG            (0x14)
>> +#define MBOX_ICLR_REG            (0x18)
>> +#define MBOX_SEND_REG            (0x1c)
>> +#define MBOX_DATA_REG            (0x20)
>> +
>> +#define MBOX_IPC_LOCK_REG        (0xa00)
>> +#define MBOX_IPC_UNLOCK            (0x1acce551)
>> +
>> +#define MBOX_AUTOMATIC_ACK        (1)
>> +
>> +#define MBOX_STATE_IDLE            BIT(4)
>> +#define MBOX_STATE_ACK            BIT(7)
>> +
>> +#define MBOX_MSG_LEN            8
>> +
>> +/**
>> + * Hi3660 mailbox channel device data
>> + *
>> + * A channel can be used for TX or RX, it can trigger remote
>> + * processor interrupt to notify remote processor and can receive
>> + * interrupt if has incoming message.
>> + *
>> + * @dst_irq:    Interrupt vector for remote processor
>> + * @ack_irq:    Interrupt vector for local processor
>> + */
>> +struct hi3660_mbox_dev {
>> +    unsigned int dst_irq;
>> +    unsigned int ack_irq;
>> +};
>> +
>> +/**
>> + * Hi3660 mailbox controller data
>> + *
>> + * Mailbox controller includes 32 channels and can allocate
>> + * channel for message transferring.
>> + *
>> + * @dev:    Device to which it is attached
>> + * @base:    Base address of the register mapping region
>> + * @chan:    Representation of channels in mailbox controller
>> + * @mdev:    Representation of channel device data
>> + * @controller:    Representation of a communication channel controller
>> + */
>> +struct hi3660_mbox {
>> +    struct device *dev;
>> +    void __iomem *base;
>> +    struct mbox_chan chan[MBOX_CHAN_MAX];
>> +    struct hi3660_mbox_dev mdev[MBOX_CHAN_MAX];
>> +    struct mbox_controller controller;
>> +};
>> +
>> +static inline struct hi3660_mbox *to_hi3660_mbox(struct mbox_chan *chan)
>> +{
>> +    return container_of(chan->mbox, struct hi3660_mbox, controller);
>> +}
>> +
>> +static int hi3660_mbox_check_state(struct mbox_chan *chan)
>> +{
>> +    unsigned long ch = (unsigned long)chan->con_priv;
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
>> +    void __iomem *base = MBOX_BASE(mbox, ch);
>> +    unsigned long val;
>> +    unsigned int state, ret;
>> +
>> +    /* Mailbox is idle so directly bail out */
>> +    state = readl_relaxed(base + MBOX_MODE_REG);
>> +    if (state & MBOX_STATE_IDLE)
>> +        return 0;
>> +
>> +    /* Wait for acknowledge from remote */
>> +    ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
>> +            val, (val & MBOX_STATE_ACK), 1000, 300000);
>> +    if (ret) {
>> +        dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
>> +        return ret;
>> +    }
>> +
>> +    /* Ensure channel is released */
>> +    writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
>> +    writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
>> +    __asm__ volatile ("sev");
> 
> There is an existing sev() macro for this.
> 
Macro will be used in next patch，if this function is not redesigned.
Mark and Leo is discussing memory barrier protection issues.

>> +    return 0;
>> +}
>> +
>> +static int hi3660_mbox_unlock(struct mbox_chan *chan)
>> +{
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    unsigned int val, retry = 3;
>> +
>> +    do {
>> +        writel_relaxed(MBOX_IPC_UNLOCK, mbox->base + MBOX_IPC_LOCK_REG);
>> +
>> +        val = readl_relaxed(mbox->base + MBOX_IPC_LOCK_REG);
>> +        if (!val)
>> +            break;
>> +
>> +        udelay(10);
>> +    } while (retry--);
>> +
>> +    return (!val) ? 0 : -ETIMEDOUT;
>> +}
>> +
>> +static int hi3660_mbox_acquire_channel(struct mbox_chan *chan)
>> +{
>> +    unsigned long ch = (unsigned long)chan->con_priv;
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
>> +    void __iomem *base = MBOX_BASE(mbox, ch);
>> +    unsigned int val = 0, retry = 10;
>> +
>> +    /*
>> +     * Hardware locking for exclusive accessing within CPUs
>> +     * without exclusive monitor mechanism.
>> +     */
>> +    do {
>> +        val = readl_relaxed(base + MBOX_MODE_REG);
>> +        if (!(val & MBOX_STATE_IDLE)) {
>> +            __asm__ volatile ("wfe");
>> +            continue;
> 
> So this is going to "wfe" when retry == 0 and the continue statement will take us out of the loop?
> 
> Also, there is a wfe() macro for arm and arm64 which could be used here.
> 
>> +        }
>> +
>> +        /* Check if channel has be acquired */
>> +        writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
>> +        val = readl_relaxed(base + MBOX_SRC_REG) & BIT(mdev->ack_irq);
>> +        if (val)
>> +            break;
>> +
>> +    } while (retry--);
>> +
>> +    return (val) ? 0 : -ETIMEDOUT;
> 
> If timeout occurs while waiting for the MBOX to be idle, val will hold the last value from MBOX_MODE_REG, and if this can be different than 0, this will hide the fact it timed out.
> 
> Maybe the following would be more appropriate:
> 
> return (retry >= 0) ? 0 : -ETIMEDOUT;
> 
> Also, your do {} while loops for the timeouts falsely give the impression we do "retry" attempts when we actually do "retry + 1" attempts (but I guess it doesn't make a big difference from the functional point of view).
> 

Agreed. The current logic of code is confusing as val is used twice.

>> +}
>> +
>> +static int hi3660_mbox_send(struct mbox_chan *chan, u32 *msg)
>> +{
>> +    unsigned long ch = (unsigned long)chan->con_priv;
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
>> +    void __iomem *base = MBOX_BASE(mbox, ch);
>> +    unsigned int i;
>> +
>> +    /* Clear mask for destination interrupt */
>> +    writel_relaxed(~BIT(mdev->dst_irq), base + MBOX_IMASK_REG);
>> +
>> +    /* Config destination for interrupt vector */
>> +    writel_relaxed(BIT(mdev->dst_irq), base + MBOX_DST_REG);
>> +
>> +    /* Automatic acknowledge mode */
>> +    writel_relaxed(MBOX_AUTOMATIC_ACK, base + MBOX_MODE_REG);
>> +
>> +    /* Fill message data */
>> +    for (i = 0; i < MBOX_MSG_LEN; i++)
>> +        writel_relaxed(msg[i], base + MBOX_DATA_REG + i * 4);
>> +
>> +    /* Trigger data transferring */
>> +    writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SEND_REG);
>> +    return 0;
>> +}
>> +
>> +static int hi3660_mbox_send_data(struct mbox_chan *chan, void *msg)
>> +{
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    int err;
>> +
>> +    err = hi3660_mbox_check_state(chan);
>> +    if (err) {
>> +        dev_err(mbox->dev, "checking state failed\n");
>> +        return err;
>> +    }
>> +
>> +    err = hi3660_mbox_unlock(chan);
>> +    if (err) {
>> +        dev_err(mbox->dev, "unlocking mailbox failed\n");
>> +        return err;
>> +    }
>> +
>> +    err = hi3660_mbox_acquire_channel(chan);
>> +    if (err) {
>> +        dev_err(mbox->dev, "acquiring channel failed\n");
>> +        return err;
>> +    }
>> +
>> +    return hi3660_mbox_send(chan, msg);
>> +}
>> +
>> +static struct mbox_chan_ops hi3660_mbox_ops = {
>> +    .send_data    = hi3660_mbox_send_data,
>> +};
>> +
>> +static struct mbox_chan *hi3660_mbox_xlate(struct mbox_controller *controller,
>> +                       const struct of_phandle_args *spec)
>> +{
>> +    struct hi3660_mbox *mbox = dev_get_drvdata(controller->dev);
> 
> nit:
> In to_hi3660_mbox, you use the fact that the mbox_controller structure is part of hi3660_mbox and use container of to retrieve hi3660_mbox.
> 
> I think it would be nice to be consistent about this, either use container_of or dev drvdata, but not both.
> 
> Cheers,
> 
I will discuss with Leo.

Thanks for your comments.

Best Regards,