[PATCH v2 00/10] serial: Add Tegra Combined UART driver

[PATCH v2 00/10] serial: Add Tegra Combined UART driver

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

Hi everyone,

this is a reworked version of Mikko's earlier proposal[0]. I've reworked
the TCU driver itself so that it relies less on global variables as well
as added a Kconfig option to allow the console support to be selected. I
also fixed a couple of issues that manifested themselves as I was moving
towards the IRQ driven mode (TCU was passing a pointer to a local
variable which was getting stored in the mailbox's ring buffer and the
data pointed at was becoming stale by the time the mailbox got around to
dequeue it).

The biggest bulk of the changes is in the mailbox driver. This series
addresses all of Jassi's comments from back at the time. One notable
additional change is that shared mailboxes are now interrupt driven,
which removes the need for polling mode.

Unfortunately there is still an issue because the TCU uses the mailbox
in atomic context for both TTY and console modes, so we get a sleeping-
while-atomic BUG when using mbox_client->tx_block = true in order to
rate-limit mbox_send_message(). In order to work around this, I added a
mechanism to mbox_send_message() that will allow blocking from atomic
context if the mailbox controller implements the new ->flush() callback.
For Tegra HSP shared mailboxes this is done by spinning on the shared
mailbox register until the receiver has marked the mailbox as empty. I
have been running this locally for a couple of days now and it works
perfectly.

Furthermore this series incorporates Mikko's work in progress on
splitting up the mailbox controllers and allowing multiple controllers
to match on the same device tree node during mbox_request_channel().

Last but not least there are no build-time dependencies between the
mailbox and serial drivers, so I think the easiest way to merge this is
if Jassi picks up patches 1-5, Greg takes patches 6 & 7 and I pick up
patches 8-10 into the Tegra tree.

Changes in v2:
- address all of Pekka's comments regarding shared interrupts, registers
  that don't exist on Tegra186 and shared mailbox directionality
- add a patch to enable the TCU in the 64-bit ARM default configuration

Thanks,
Thierry

[0]: https://lore.kernel.org/patchwork/project/lkml/list/?series=357641

Mikko Perttunen (5):
  mailbox: Allow multiple controllers per device
  dt-bindings: tegra186-hsp: Add shared mailboxes
  dt-bindings: serial: Add bindings for nvidia,tegra194-tcu
  arm64: tegra: Add nodes for TCU on Tegra194
  arm64: tegra: Mark TCU as primary serial port on Tegra194 P2888

Thierry Reding (5):
  mailbox: Support blocking transfers in atomic context
  mailbox: tegra-hsp: Add support for shared mailboxes
  mailbox: tegra-hsp: Add suspend/resume support
  serial: Add Tegra Combined UART driver
  arm64: defconfig: Enable Tegra TCU

 .../bindings/mailbox/nvidia,tegra186-hsp.txt  |  30 +-
 .../bindings/serial/nvidia,tegra194-tcu.txt   |  35 ++
 .../arm64/boot/dts/nvidia/tegra194-p2888.dtsi |   2 +-
 arch/arm64/boot/dts/nvidia/tegra194.dtsi      |  38 +-
 arch/arm64/configs/defconfig                  |   1 +
 drivers/mailbox/mailbox.c                     |  11 +-
 drivers/mailbox/tegra-hsp.c                   | 517 +++++++++++++++---
 drivers/tty/serial/Kconfig                    |  22 +
 drivers/tty/serial/Makefile                   |   1 +
 drivers/tty/serial/tegra-tcu.c                | 299 ++++++++++
 include/dt-bindings/mailbox/tegra186-hsp.h    |  11 +
 include/linux/mailbox_controller.h            |   4 +
 include/uapi/linux/serial_core.h              |   3 +
 13 files changed, 903 insertions(+), 71 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
 create mode 100644 drivers/tty/serial/tegra-tcu.c

-- 
2.19.1

[PATCH v2 00/10] serial: Add Tegra Combined UART driver

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

Hi everyone,

this is a reworked version of Mikko's earlier proposal[0]. I've reworked
the TCU driver itself so that it relies less on global variables as well
as added a Kconfig option to allow the console support to be selected. I
also fixed a couple of issues that manifested themselves as I was moving
towards the IRQ driven mode (TCU was passing a pointer to a local
variable which was getting stored in the mailbox's ring buffer and the
data pointed at was becoming stale by the time the mailbox got around to
dequeue it).

The biggest bulk of the changes is in the mailbox driver. This series
addresses all of Jassi's comments from back at the time. One notable
additional change is that shared mailboxes are now interrupt driven,
which removes the need for polling mode.

Unfortunately there is still an issue because the TCU uses the mailbox
in atomic context for both TTY and console modes, so we get a sleeping-
while-atomic BUG when using mbox_client->tx_block = true in order to
rate-limit mbox_send_message(). In order to work around this, I added a
mechanism to mbox_send_message() that will allow blocking from atomic
context if the mailbox controller implements the new ->flush() callback.
For Tegra HSP shared mailboxes this is done by spinning on the shared
mailbox register until the receiver has marked the mailbox as empty. I
have been running this locally for a couple of days now and it works
perfectly.

Furthermore this series incorporates Mikko's work in progress on
splitting up the mailbox controllers and allowing multiple controllers
to match on the same device tree node during mbox_request_channel().

Last but not least there are no build-time dependencies between the
mailbox and serial drivers, so I think the easiest way to merge this is
if Jassi picks up patches 1-5, Greg takes patches 6 & 7 and I pick up
patches 8-10 into the Tegra tree.

Changes in v2:
- address all of Pekka's comments regarding shared interrupts, registers
  that don't exist on Tegra186 and shared mailbox directionality
- add a patch to enable the TCU in the 64-bit ARM default configuration

Thanks,
Thierry

[0]: https://lore.kernel.org/patchwork/project/lkml/list/?series=357641

Mikko Perttunen (5):
  mailbox: Allow multiple controllers per device
  dt-bindings: tegra186-hsp: Add shared mailboxes
  dt-bindings: serial: Add bindings for nvidia,tegra194-tcu
  arm64: tegra: Add nodes for TCU on Tegra194
  arm64: tegra: Mark TCU as primary serial port on Tegra194 P2888

Thierry Reding (5):
  mailbox: Support blocking transfers in atomic context
  mailbox: tegra-hsp: Add support for shared mailboxes
  mailbox: tegra-hsp: Add suspend/resume support
  serial: Add Tegra Combined UART driver
  arm64: defconfig: Enable Tegra TCU

 .../bindings/mailbox/nvidia,tegra186-hsp.txt  |  30 +-
 .../bindings/serial/nvidia,tegra194-tcu.txt   |  35 ++
 .../arm64/boot/dts/nvidia/tegra194-p2888.dtsi |   2 +-
 arch/arm64/boot/dts/nvidia/tegra194.dtsi      |  38 +-
 arch/arm64/configs/defconfig                  |   1 +
 drivers/mailbox/mailbox.c                     |  11 +-
 drivers/mailbox/tegra-hsp.c                   | 517 +++++++++++++++---
 drivers/tty/serial/Kconfig                    |  22 +
 drivers/tty/serial/Makefile                   |   1 +
 drivers/tty/serial/tegra-tcu.c                | 299 ++++++++++
 include/dt-bindings/mailbox/tegra186-hsp.h    |  11 +
 include/linux/mailbox_controller.h            |   4 +
 include/uapi/linux/serial_core.h              |   3 +
 13 files changed, 903 insertions(+), 71 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
 create mode 100644 drivers/tty/serial/tegra-tcu.c

-- 
2.19.1

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

The mailbox framework supports blocking transfers via completions for
clients that can sleep. In order to support blocking transfers in cases
where the transmission is not permitted to sleep, add a new ->flush()
callback that controller drivers can implement to busy loop until the
transmission has been completed. This will automatically be called when
available and interrupts are disabled for clients that request blocking
transfers.

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 drivers/mailbox/mailbox.c          | 8 ++++++++
 include/linux/mailbox_controller.h | 4 ++++
 2 files changed, 12 insertions(+)

diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
index 674b35f402f5..0eaf21259874 100644
--- a/drivers/mailbox/mailbox.c
+++ b/drivers/mailbox/mailbox.c
@@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
 		unsigned long wait;
 		int ret;
 
+		if (irqs_disabled() && chan->mbox->ops->flush) {
+			ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
+			if (ret < 0)
+				tx_tick(chan, ret);
+
+			return ret;
+		}
+
 		if (!chan->cl->tx_tout) /* wait forever */
 			wait = msecs_to_jiffies(3600000);
 		else
diff --git a/include/linux/mailbox_controller.h b/include/linux/mailbox_controller.h
index 74deadb42d76..2a07d93f781a 100644
--- a/include/linux/mailbox_controller.h
+++ b/include/linux/mailbox_controller.h
@@ -24,6 +24,9 @@ struct mbox_chan;
  *		transmission of data is reported by the controller via
  *		mbox_chan_txdone (if it has some TX ACK irq). It must not
  *		sleep.
+ * @flush:	Called when a client requests transmissions to be blocking but
+ *		the context doesn't allow sleeping. Typically the controller
+ *		will implement a busy loop waiting for the data to flush out.
  * @startup:	Called when a client requests the chan. The controller
  *		could ask clients for additional parameters of communication
  *		to be provided via client's chan_data. This call may
@@ -46,6 +49,7 @@ struct mbox_chan;
  */
 struct mbox_chan_ops {
 	int (*send_data)(struct mbox_chan *chan, void *data);
+	int (*flush)(struct mbox_chan *chan, unsigned long timeout);
 	int (*startup)(struct mbox_chan *chan);
 	void (*shutdown)(struct mbox_chan *chan);
 	bool (*last_tx_done)(struct mbox_chan *chan);
-- 
2.19.1

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

The mailbox framework supports blocking transfers via completions for
clients that can sleep. In order to support blocking transfers in cases
where the transmission is not permitted to sleep, add a new ->flush()
callback that controller drivers can implement to busy loop until the
transmission has been completed. This will automatically be called when
available and interrupts are disabled for clients that request blocking
transfers.

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 drivers/mailbox/mailbox.c          | 8 ++++++++
 include/linux/mailbox_controller.h | 4 ++++
 2 files changed, 12 insertions(+)

diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
index 674b35f402f5..0eaf21259874 100644
--- a/drivers/mailbox/mailbox.c
+++ b/drivers/mailbox/mailbox.c
@@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
 		unsigned long wait;
 		int ret;
 
+		if (irqs_disabled() && chan->mbox->ops->flush) {
+			ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
+			if (ret < 0)
+				tx_tick(chan, ret);
+
+			return ret;
+		}
+
 		if (!chan->cl->tx_tout) /* wait forever */
 			wait = msecs_to_jiffies(3600000);
 		else
diff --git a/include/linux/mailbox_controller.h b/include/linux/mailbox_controller.h
index 74deadb42d76..2a07d93f781a 100644
--- a/include/linux/mailbox_controller.h
+++ b/include/linux/mailbox_controller.h
@@ -24,6 +24,9 @@ struct mbox_chan;
  *		transmission of data is reported by the controller via
  *		mbox_chan_txdone (if it has some TX ACK irq). It must not
  *		sleep.
+ * @flush:	Called when a client requests transmissions to be blocking but
+ *		the context doesn't allow sleeping. Typically the controller
+ *		will implement a busy loop waiting for the data to flush out.
  * @startup:	Called when a client requests the chan. The controller
  *		could ask clients for additional parameters of communication
  *		to be provided via client's chan_data. This call may
@@ -46,6 +49,7 @@ struct mbox_chan;
  */
 struct mbox_chan_ops {
 	int (*send_data)(struct mbox_chan *chan, void *data);
+	int (*flush)(struct mbox_chan *chan, unsigned long timeout);
 	int (*startup)(struct mbox_chan *chan);
 	void (*shutdown)(struct mbox_chan *chan);
 	bool (*last_tx_done)(struct mbox_chan *chan);
-- 
2.19.1

[PATCH v2 02/10] mailbox: Allow multiple controllers per device

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

Look through the whole controller list when mapping device tree
phandles to controllers instead of stopping at the first one.

Each controller is intended to only contain one kind of mailbox,
but some devices (like Tegra HSP) implement multiple kinds and use
the same device tree node for all of them. As such, we need to allow
multiple mbox_controllers per device tree node.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 drivers/mailbox/mailbox.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
index 0eaf21259874..f34820bcf33c 100644
--- a/drivers/mailbox/mailbox.c
+++ b/drivers/mailbox/mailbox.c
@@ -335,7 +335,8 @@ struct mbox_chan *mbox_request_channel(struct mbox_client *cl, int index)
 	list_for_each_entry(mbox, &mbox_cons, node)
 		if (mbox->dev->of_node == spec.np) {
 			chan = mbox->of_xlate(mbox, &spec);
-			break;
+			if (!IS_ERR(chan))
+				break;
 		}
 
 	of_node_put(spec.np);
-- 
2.19.1

[PATCH v2 02/10] mailbox: Allow multiple controllers per device

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

Look through the whole controller list when mapping device tree
phandles to controllers instead of stopping at the first one.

Each controller is intended to only contain one kind of mailbox,
but some devices (like Tegra HSP) implement multiple kinds and use
the same device tree node for all of them. As such, we need to allow
multiple mbox_controllers per device tree node.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 drivers/mailbox/mailbox.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
index 0eaf21259874..f34820bcf33c 100644
--- a/drivers/mailbox/mailbox.c
+++ b/drivers/mailbox/mailbox.c
@@ -335,7 +335,8 @@ struct mbox_chan *mbox_request_channel(struct mbox_client *cl, int index)
 	list_for_each_entry(mbox, &mbox_cons, node)
 		if (mbox->dev->of_node == spec.np) {
 			chan = mbox->of_xlate(mbox, &spec);
-			break;
+			if (!IS_ERR(chan))
+				break;
 		}
 
 	of_node_put(spec.np);
-- 
2.19.1

[PATCH v2 03/10] dt-bindings: tegra186-hsp: Add shared mailboxes

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

Shared mailboxes are a mechanism to transport data from one processor in
the system to another. They are bidirectional links with both a producer
and a consumer. Interrupts are used to let the consumer know when data
was written to the mailbox by the producer, and to let the producer know
when the consumer has read the data from the mailbox. These interrupts
are mapped to one or more "shared interrupts". Typically each processor
in the system owns one of these shared interrupts.

Add documentation to the device tree bindings about how clients can use
mailbox specifiers to request a specific shared mailbox and select which
direction they drive. Also document how to specify the shared interrupts
in addition to the existing doorbell interrupt.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Acked-by: Jon Hunter <jonathanh@nvidia.com>
Reviewed-by: Rob Herring <robh@kernel.org>
Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
Changes in v2:
- describe in more detail how the mailbox specifiers are constructed
- encode mailbox direction in the device tree mailbox specifier

Rob,

you had already reviewed this and I've kept your tag for now, even
though there have been slight changes in this. If you don't mind taking
another look to verify that the changes I've made are still okay.

Thanks,
Thierry

 .../bindings/mailbox/nvidia,tegra186-hsp.txt  | 30 +++++++++++++++----
 include/dt-bindings/mailbox/tegra186-hsp.h    | 11 +++++++
 2 files changed, 36 insertions(+), 5 deletions(-)

diff --git a/Documentation/devicetree/bindings/mailbox/nvidia,tegra186-hsp.txt b/Documentation/devicetree/bindings/mailbox/nvidia,tegra186-hsp.txt
index b99d25fc2f26..ff3eafc5a882 100644
--- a/Documentation/devicetree/bindings/mailbox/nvidia,tegra186-hsp.txt
+++ b/Documentation/devicetree/bindings/mailbox/nvidia,tegra186-hsp.txt
@@ -15,12 +15,15 @@ Required properties:
     Array of strings.
     one of:
     - "nvidia,tegra186-hsp"
+    - "nvidia,tegra194-hsp", "nvidia,tegra186-hsp"
 - reg : Offset and length of the register set for the device.
 - interrupt-names
     Array of strings.
     Contains a list of names for the interrupts described by the interrupt
     property. May contain the following entries, in any order:
     - "doorbell"
+    - "sharedN", where 'N' is a number from zero up to the number of
+      external interrupts supported by the HSP instance minus one.
     Users of this binding MUST look up entries in the interrupt property
     by name, using this interrupt-names property to do so.
 - interrupts
@@ -29,12 +32,29 @@ Required properties:
     in a matching order.
 - #mbox-cells : Should be 2.
 
-The mbox specifier of the "mboxes" property in the client node should
-contain two data. The first one should be the HSP type and the second
-one should be the ID that the client is going to use. Those information
-can be found in the following file.
+The mbox specifier of the "mboxes" property in the client node should contain
+two cells. The first cell determines the HSP type and the second cell is used
+to identify the mailbox that the client is going to use.
 
-- <dt-bindings/mailbox/tegra186-hsp.h>.
+For doorbells, the second cell specifies the index of the doorbell to use.
+
+For shared mailboxes, the second cell is composed of two fields:
+- bits 31..24:
+    A bit mask of flags that further specify how the shared mailbox will be
+    used. Valid flags are:
+    - bit 31:
+        Defines the direction of the mailbox. If set, the mailbox will be used
+        as a producer (i.e. used to send data). If cleared, the mailbox is the
+        consumer of data sent by a producer.
+
+- bits 23.. 0:
+    The index of the shared mailbox to use. The number of available mailboxes
+    may vary by instance of the HSP block and SoC generation.
+
+The following file contains definitions that can be used to construct mailbox
+specifiers:
+
+    <dt-bindings/mailbox/tegra186-hsp.h>
 
 Example:
 
diff --git a/include/dt-bindings/mailbox/tegra186-hsp.h b/include/dt-bindings/mailbox/tegra186-hsp.h
index bcab5b7ca785..3bdec7a84d35 100644
--- a/include/dt-bindings/mailbox/tegra186-hsp.h
+++ b/include/dt-bindings/mailbox/tegra186-hsp.h
@@ -22,4 +22,15 @@
 #define TEGRA_HSP_DB_MASTER_CCPLEX 17
 #define TEGRA_HSP_DB_MASTER_BPMP 19
 
+/*
+ * Shared mailboxes are unidirectional, so the direction needs to be specified
+ * in the device tree.
+ */
+#define TEGRA_HSP_SM_MASK 0x00ffffff
+#define TEGRA_HSP_SM_FLAG_RX (0 << 31)
+#define TEGRA_HSP_SM_FLAG_TX (1 << 31)
+
+#define TEGRA_HSP_SM_RX(x) (TEGRA_HSP_SM_FLAG_RX | ((x) & TEGRA_HSP_SM_MASK))
+#define TEGRA_HSP_SM_TX(x) (TEGRA_HSP_SM_FLAG_TX | ((x) & TEGRA_HSP_SM_MASK))
+
 #endif
-- 
2.19.1

[PATCH v2 03/10] dt-bindings: tegra186-hsp: Add shared mailboxes

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

Shared mailboxes are a mechanism to transport data from one processor in
the system to another. They are bidirectional links with both a producer
and a consumer. Interrupts are used to let the consumer know when data
was written to the mailbox by the producer, and to let the producer know
when the consumer has read the data from the mailbox. These interrupts
are mapped to one or more "shared interrupts". Typically each processor
in the system owns one of these shared interrupts.

Add documentation to the device tree bindings about how clients can use
mailbox specifiers to request a specific shared mailbox and select which
direction they drive. Also document how to specify the shared interrupts
in addition to the existing doorbell interrupt.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Acked-by: Jon Hunter <jonathanh@nvidia.com>
Reviewed-by: Rob Herring <robh@kernel.org>
Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
Changes in v2:
- describe in more detail how the mailbox specifiers are constructed
- encode mailbox direction in the device tree mailbox specifier

Rob,

you had already reviewed this and I've kept your tag for now, even
though there have been slight changes in this. If you don't mind taking
another look to verify that the changes I've made are still okay.

Thanks,
Thierry

 .../bindings/mailbox/nvidia,tegra186-hsp.txt  | 30 +++++++++++++++----
 include/dt-bindings/mailbox/tegra186-hsp.h    | 11 +++++++
 2 files changed, 36 insertions(+), 5 deletions(-)

diff --git a/Documentation/devicetree/bindings/mailbox/nvidia,tegra186-hsp.txt b/Documentation/devicetree/bindings/mailbox/nvidia,tegra186-hsp.txt
index b99d25fc2f26..ff3eafc5a882 100644
--- a/Documentation/devicetree/bindings/mailbox/nvidia,tegra186-hsp.txt
+++ b/Documentation/devicetree/bindings/mailbox/nvidia,tegra186-hsp.txt
@@ -15,12 +15,15 @@ Required properties:
     Array of strings.
     one of:
     - "nvidia,tegra186-hsp"
+    - "nvidia,tegra194-hsp", "nvidia,tegra186-hsp"
 - reg : Offset and length of the register set for the device.
 - interrupt-names
     Array of strings.
     Contains a list of names for the interrupts described by the interrupt
     property. May contain the following entries, in any order:
     - "doorbell"
+    - "sharedN", where 'N' is a number from zero up to the number of
+      external interrupts supported by the HSP instance minus one.
     Users of this binding MUST look up entries in the interrupt property
     by name, using this interrupt-names property to do so.
 - interrupts
@@ -29,12 +32,29 @@ Required properties:
     in a matching order.
 - #mbox-cells : Should be 2.
 
-The mbox specifier of the "mboxes" property in the client node should
-contain two data. The first one should be the HSP type and the second
-one should be the ID that the client is going to use. Those information
-can be found in the following file.
+The mbox specifier of the "mboxes" property in the client node should contain
+two cells. The first cell determines the HSP type and the second cell is used
+to identify the mailbox that the client is going to use.
 
-- <dt-bindings/mailbox/tegra186-hsp.h>.
+For doorbells, the second cell specifies the index of the doorbell to use.
+
+For shared mailboxes, the second cell is composed of two fields:
+- bits 31..24:
+    A bit mask of flags that further specify how the shared mailbox will be
+    used. Valid flags are:
+    - bit 31:
+        Defines the direction of the mailbox. If set, the mailbox will be used
+        as a producer (i.e. used to send data). If cleared, the mailbox is the
+        consumer of data sent by a producer.
+
+- bits 23.. 0:
+    The index of the shared mailbox to use. The number of available mailboxes
+    may vary by instance of the HSP block and SoC generation.
+
+The following file contains definitions that can be used to construct mailbox
+specifiers:
+
+    <dt-bindings/mailbox/tegra186-hsp.h>
 
 Example:
 
diff --git a/include/dt-bindings/mailbox/tegra186-hsp.h b/include/dt-bindings/mailbox/tegra186-hsp.h
index bcab5b7ca785..3bdec7a84d35 100644
--- a/include/dt-bindings/mailbox/tegra186-hsp.h
+++ b/include/dt-bindings/mailbox/tegra186-hsp.h
@@ -22,4 +22,15 @@
 #define TEGRA_HSP_DB_MASTER_CCPLEX 17
 #define TEGRA_HSP_DB_MASTER_BPMP 19
 
+/*
+ * Shared mailboxes are unidirectional, so the direction needs to be specified
+ * in the device tree.
+ */
+#define TEGRA_HSP_SM_MASK 0x00ffffff
+#define TEGRA_HSP_SM_FLAG_RX (0 << 31)
+#define TEGRA_HSP_SM_FLAG_TX (1 << 31)
+
+#define TEGRA_HSP_SM_RX(x) (TEGRA_HSP_SM_FLAG_RX | ((x) & TEGRA_HSP_SM_MASK))
+#define TEGRA_HSP_SM_TX(x) (TEGRA_HSP_SM_FLAG_TX | ((x) & TEGRA_HSP_SM_MASK))
+
 #endif
-- 
2.19.1

[PATCH v2 04/10] mailbox: tegra-hsp: Add support for shared mailboxes

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

The Tegra HSP block supports 'shared mailboxes' that are simple 32-bit
registers consisting of a FULL bit in MSB position and 31 bits of data.
The hardware can be configured to trigger interrupts when a mailbox
is empty or full. Add support for these shared mailboxes to the HSP
driver.

The initial use for the mailboxes is the Tegra Combined UART. For this
purpose, we use interrupts to receive data, and spinning to wait for
the transmit mailbox to be emptied to minimize unnecessary overhead.

Based on work by Mikko Perttunen <mperttunen@nvidia.com>.

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
Changes in v2:
- do not write per-mailbox interrupt enable registers on Tegra186
- merge handlers for empty and full interrupts
- track direction of shared mailboxes

 drivers/mailbox/tegra-hsp.c | 498 +++++++++++++++++++++++++++++++-----
 1 file changed, 437 insertions(+), 61 deletions(-)

diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
index 0cde356c11ab..0100a974149b 100644
--- a/drivers/mailbox/tegra-hsp.c
+++ b/drivers/mailbox/tegra-hsp.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.
+ * Copyright (c) 2016-2018, NVIDIA CORPORATION.  All rights reserved.
  *
  * This program is free software; you can redistribute it and/or modify it
  * under the terms and conditions of the GNU General Public License,
@@ -11,6 +11,7 @@
  * more details.
  */
 
+#include <linux/delay.h>
 #include <linux/interrupt.h>
 #include <linux/io.h>
 #include <linux/mailbox_controller.h>
@@ -21,6 +22,17 @@
 
 #include <dt-bindings/mailbox/tegra186-hsp.h>
 
+#include "mailbox.h"
+
+#define HSP_INT_IE(x)		(0x100 + ((x) * 4))
+#define HSP_INT_IV		0x300
+#define HSP_INT_IR		0x304
+
+#define HSP_INT_EMPTY_SHIFT	0
+#define HSP_INT_EMPTY_MASK	0xff
+#define HSP_INT_FULL_SHIFT	8
+#define HSP_INT_FULL_MASK	0xff
+
 #define HSP_INT_DIMENSIONING	0x380
 #define HSP_nSM_SHIFT		0
 #define HSP_nSS_SHIFT		4
@@ -34,6 +46,11 @@
 #define HSP_DB_RAW	0x8
 #define HSP_DB_PENDING	0xc
 
+#define HSP_SM_SHRD_MBOX	0x0
+#define HSP_SM_SHRD_MBOX_FULL	BIT(31)
+#define HSP_SM_SHRD_MBOX_FULL_INT_IE	0x04
+#define HSP_SM_SHRD_MBOX_EMPTY_INT_IE	0x08
+
 #define HSP_DB_CCPLEX		1
 #define HSP_DB_BPMP		3
 #define HSP_DB_MAX		7
@@ -55,6 +72,12 @@ struct tegra_hsp_doorbell {
 	unsigned int index;
 };
 
+struct tegra_hsp_mailbox {
+	struct tegra_hsp_channel channel;
+	unsigned int index;
+	bool producer;
+};
+
 struct tegra_hsp_db_map {
 	const char *name;
 	unsigned int master;
@@ -63,13 +86,18 @@ struct tegra_hsp_db_map {
 
 struct tegra_hsp_soc {
 	const struct tegra_hsp_db_map *map;
+	bool has_per_mb_ie;
 };
 
 struct tegra_hsp {
+	struct device *dev;
 	const struct tegra_hsp_soc *soc;
-	struct mbox_controller mbox;
+	struct mbox_controller mbox_db;
+	struct mbox_controller mbox_sm;
 	void __iomem *regs;
-	unsigned int irq;
+	unsigned int doorbell_irq;
+	unsigned int *shared_irqs;
+	unsigned int shared_irq;
 	unsigned int num_sm;
 	unsigned int num_as;
 	unsigned int num_ss;
@@ -78,13 +106,10 @@ struct tegra_hsp {
 	spinlock_t lock;
 
 	struct list_head doorbells;
-};
+	struct tegra_hsp_mailbox *mailboxes;
 
-static inline struct tegra_hsp *
-to_tegra_hsp(struct mbox_controller *mbox)
-{
-	return container_of(mbox, struct tegra_hsp, mbox);
-}
+	unsigned long mask;
+};
 
 static inline u32 tegra_hsp_readl(struct tegra_hsp *hsp, unsigned int offset)
 {
@@ -158,7 +183,7 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
 
 	spin_lock(&hsp->lock);
 
-	for_each_set_bit(master, &value, hsp->mbox.num_chans) {
+	for_each_set_bit(master, &value, hsp->mbox_db.num_chans) {
 		struct tegra_hsp_doorbell *db;
 
 		db = __tegra_hsp_doorbell_get(hsp, master);
@@ -182,6 +207,71 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
+static irqreturn_t tegra_hsp_shared_irq(int irq, void *data)
+{
+	struct tegra_hsp *hsp = data;
+	unsigned long bit, mask;
+	u32 status, value;
+	void *msg;
+
+	status = tegra_hsp_readl(hsp, HSP_INT_IR) & hsp->mask;
+
+	/* process EMPTY interrupts first */
+	mask = (status >> HSP_INT_EMPTY_SHIFT) & HSP_INT_EMPTY_MASK;
+
+	for_each_set_bit(bit, &mask, hsp->num_sm) {
+		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
+
+		if (mb->producer) {
+			/*
+			 * Disable EMPTY interrupts until data is sent with
+			 * the next message. These interrupts are level-
+			 * triggered, so if we kept them enabled they would
+			 * constantly trigger until we next write data into
+			 * the message.
+			 */
+			spin_lock(&hsp->lock);
+
+			hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+			tegra_hsp_writel(hsp, hsp->mask,
+					 HSP_INT_IE(hsp->shared_irq));
+
+			spin_unlock(&hsp->lock);
+
+			mbox_chan_txdone(mb->channel.chan, 0);
+		}
+	}
+
+	/* process FULL interrupts */
+	mask = (status >> HSP_INT_FULL_SHIFT) & HSP_INT_FULL_MASK;
+
+	for_each_set_bit(bit, &mask, hsp->num_sm) {
+		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
+
+		if (!mb->producer) {
+			value = tegra_hsp_channel_readl(&mb->channel,
+							HSP_SM_SHRD_MBOX);
+			value &= ~HSP_SM_SHRD_MBOX_FULL;
+			msg = (void *)(unsigned long)value;
+			mbox_chan_received_data(mb->channel.chan, msg);
+
+			/*
+			 * Need to clear all bits here since some producers,
+			 * such as TCU, depend on fields in the register
+			 * getting cleared by the consumer.
+			 *
+			 * The mailbox API doesn't give the consumers a way
+			 * of doing that explicitly, so we have to make sure
+			 * we cover all possible cases.
+			 */
+			tegra_hsp_channel_writel(&mb->channel, 0x0,
+						 HSP_SM_SHRD_MBOX);
+		}
+	}
+
+	return IRQ_HANDLED;
+}
+
 static struct tegra_hsp_channel *
 tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
 			  unsigned int master, unsigned int index)
@@ -194,7 +284,7 @@ tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
 	if (!db)
 		return ERR_PTR(-ENOMEM);
 
-	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) << 16;
+	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) * SZ_64K;
 	offset += index * 0x100;
 
 	db->channel.regs = hsp->regs + offset;
@@ -235,8 +325,8 @@ static int tegra_hsp_doorbell_startup(struct mbox_chan *chan)
 	unsigned long flags;
 	u32 value;
 
-	if (db->master >= hsp->mbox.num_chans) {
-		dev_err(hsp->mbox.dev,
+	if (db->master >= chan->mbox->num_chans) {
+		dev_err(chan->mbox->dev,
 			"invalid master ID %u for HSP channel\n",
 			db->master);
 		return -EINVAL;
@@ -281,46 +371,168 @@ static void tegra_hsp_doorbell_shutdown(struct mbox_chan *chan)
 	spin_unlock_irqrestore(&hsp->lock, flags);
 }
 
-static const struct mbox_chan_ops tegra_hsp_doorbell_ops = {
+static const struct mbox_chan_ops tegra_hsp_db_ops = {
 	.send_data = tegra_hsp_doorbell_send_data,
 	.startup = tegra_hsp_doorbell_startup,
 	.shutdown = tegra_hsp_doorbell_shutdown,
 };
 
-static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
+static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
+{
+	struct tegra_hsp_mailbox *mb = chan->con_priv;
+	struct tegra_hsp *hsp = mb->channel.hsp;
+	unsigned long flags;
+	u32 value;
+
+	WARN_ON(!mb->producer);
+
+	/* copy data and mark mailbox full */
+	value = (u32)(unsigned long)data;
+	value |= HSP_SM_SHRD_MBOX_FULL;
+
+	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
+
+	if (!irqs_disabled()) {
+		/* enable EMPTY interrupt for the shared mailbox */
+		spin_lock_irqsave(&hsp->lock, flags);
+
+		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
+
+		spin_unlock_irqrestore(&hsp->lock, flags);
+	}
+
+	return 0;
+}
+
+static int tegra_hsp_mailbox_flush(struct mbox_chan *chan,
+				   unsigned long timeout)
+{
+	struct tegra_hsp_mailbox *mb = chan->con_priv;
+	struct tegra_hsp_channel *ch = &mb->channel;
+	u32 value;
+
+	timeout = jiffies + msecs_to_jiffies(timeout);
+
+	while (time_before(jiffies, timeout)) {
+		value = tegra_hsp_channel_readl(ch, HSP_SM_SHRD_MBOX);
+		if ((value & HSP_SM_SHRD_MBOX_FULL) == 0) {
+			mbox_chan_txdone(chan, 0);
+			return 0;
+		}
+
+		udelay(1);
+	}
+
+	return -ETIME;
+}
+
+static int tegra_hsp_mailbox_startup(struct mbox_chan *chan)
+{
+	struct tegra_hsp_mailbox *mb = chan->con_priv;
+	struct tegra_hsp_channel *ch = &mb->channel;
+	struct tegra_hsp *hsp = mb->channel.hsp;
+	unsigned long flags;
+
+	chan->txdone_method = TXDONE_BY_IRQ;
+
+	/*
+	 * Shared mailboxes start out as consumers by default. FULL and EMPTY
+	 * interrupts are coalesced at the same shared interrupt.
+	 *
+	 * Keep EMPTY interrupts disabled at startup and only enable them when
+	 * the mailbox is actually full. This is required because the FULL and
+	 * EMPTY interrupts are level-triggered, so keeping EMPTY interrupts
+	 * enabled all the time would cause an interrupt storm while mailboxes
+	 * are idle.
+	 */
+
+	spin_lock_irqsave(&hsp->lock, flags);
+
+	if (mb->producer)
+		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+	else
+		hsp->mask |= BIT(HSP_INT_FULL_SHIFT + mb->index);
+
+	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
+
+	spin_unlock_irqrestore(&hsp->lock, flags);
+
+	if (hsp->soc->has_per_mb_ie) {
+		if (mb->producer)
+			tegra_hsp_channel_writel(ch, 0x0,
+						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
+		else
+			tegra_hsp_channel_writel(ch, 0x1,
+						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
+	}
+
+	return 0;
+}
+
+static void tegra_hsp_mailbox_shutdown(struct mbox_chan *chan)
+{
+	struct tegra_hsp_mailbox *mb = chan->con_priv;
+	struct tegra_hsp_channel *ch = &mb->channel;
+	struct tegra_hsp *hsp = mb->channel.hsp;
+	unsigned long flags;
+
+	if (hsp->soc->has_per_mb_ie) {
+		if (mb->producer)
+			tegra_hsp_channel_writel(ch, 0x0,
+						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
+		else
+			tegra_hsp_channel_writel(ch, 0x0,
+						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
+	}
+
+	spin_lock_irqsave(&hsp->lock, flags);
+
+	if (mb->producer)
+		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+	else
+		hsp->mask &= ~BIT(HSP_INT_FULL_SHIFT + mb->index);
+
+	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
+
+	spin_unlock_irqrestore(&hsp->lock, flags);
+}
+
+static const struct mbox_chan_ops tegra_hsp_sm_ops = {
+	.send_data = tegra_hsp_mailbox_send_data,
+	.flush = tegra_hsp_mailbox_flush,
+	.startup = tegra_hsp_mailbox_startup,
+	.shutdown = tegra_hsp_mailbox_shutdown,
+};
+
+static struct mbox_chan *tegra_hsp_db_xlate(struct mbox_controller *mbox,
 					    const struct of_phandle_args *args)
 {
+	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_db);
+	unsigned int type = args->args[0], master = args->args[1];
 	struct tegra_hsp_channel *channel = ERR_PTR(-ENODEV);
-	struct tegra_hsp *hsp = to_tegra_hsp(mbox);
-	unsigned int type = args->args[0];
-	unsigned int master = args->args[1];
 	struct tegra_hsp_doorbell *db;
 	struct mbox_chan *chan;
 	unsigned long flags;
 	unsigned int i;
 
-	switch (type) {
-	case TEGRA_HSP_MBOX_TYPE_DB:
-		db = tegra_hsp_doorbell_get(hsp, master);
-		if (db)
-			channel = &db->channel;
+	if (type != TEGRA_HSP_MBOX_TYPE_DB || !hsp->doorbell_irq)
+		return ERR_PTR(-ENODEV);
 
-		break;
-
-	default:
-		break;
-	}
+	db = tegra_hsp_doorbell_get(hsp, master);
+	if (db)
+		channel = &db->channel;
 
 	if (IS_ERR(channel))
 		return ERR_CAST(channel);
 
 	spin_lock_irqsave(&hsp->lock, flags);
 
-	for (i = 0; i < hsp->mbox.num_chans; i++) {
-		chan = &hsp->mbox.chans[i];
+	for (i = 0; i < mbox->num_chans; i++) {
+		chan = &mbox->chans[i];
 		if (!chan->con_priv) {
-			chan->con_priv = channel;
 			channel->chan = chan;
+			chan->con_priv = db;
 			break;
 		}
 
@@ -332,6 +544,29 @@ static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
 	return chan ?: ERR_PTR(-EBUSY);
 }
 
+static struct mbox_chan *tegra_hsp_sm_xlate(struct mbox_controller *mbox,
+					    const struct of_phandle_args *args)
+{
+	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_sm);
+	unsigned int type = args->args[0], index;
+	struct tegra_hsp_mailbox *mb;
+
+	index = args->args[1] & TEGRA_HSP_SM_MASK;
+
+	if (type != TEGRA_HSP_MBOX_TYPE_SM || !hsp->shared_irqs ||
+	    index >= hsp->num_sm)
+		return ERR_PTR(-ENODEV);
+
+	mb = &hsp->mailboxes[index];
+
+	if ((args->args[1] & TEGRA_HSP_SM_FLAG_TX) == 0)
+		mb->producer = false;
+	else
+		mb->producer = true;
+
+	return mb->channel.chan;
+}
+
 static void tegra_hsp_remove_doorbells(struct tegra_hsp *hsp)
 {
 	struct tegra_hsp_doorbell *db, *tmp;
@@ -364,10 +599,65 @@ static int tegra_hsp_add_doorbells(struct tegra_hsp *hsp)
 	return 0;
 }
 
+static int tegra_hsp_add_mailboxes(struct tegra_hsp *hsp, struct device *dev)
+{
+	int i;
+
+	hsp->mailboxes = devm_kcalloc(dev, hsp->num_sm, sizeof(*hsp->mailboxes),
+				      GFP_KERNEL);
+	if (!hsp->mailboxes)
+		return -ENOMEM;
+
+	for (i = 0; i < hsp->num_sm; i++) {
+		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
+
+		mb->index = i;
+
+		mb->channel.hsp = hsp;
+		mb->channel.regs = hsp->regs + SZ_64K + i * SZ_32K;
+		mb->channel.chan = &hsp->mbox_sm.chans[i];
+		mb->channel.chan->con_priv = mb;
+	}
+
+	return 0;
+}
+
+static int tegra_hsp_request_shared_irqs(struct tegra_hsp *hsp)
+{
+	unsigned int i, irq = 0;
+	int err;
+
+	for (i = 0; i < hsp->num_si; i++) {
+		if (hsp->shared_irq == 0 && hsp->shared_irqs[i] > 0) {
+			irq = hsp->shared_irqs[i];
+			hsp->shared_irq = i;
+			break;
+		}
+	}
+
+	if (irq > 0) {
+		err = devm_request_irq(hsp->dev, irq, tegra_hsp_shared_irq, 0,
+				       dev_name(hsp->dev), hsp);
+		if (err < 0) {
+			dev_err(hsp->dev, "failed to request interrupt: %d\n",
+				err);
+			return err;
+		}
+
+		/* disable all interrupts */
+		tegra_hsp_writel(hsp, 0, HSP_INT_IE(hsp->shared_irq));
+
+		dev_dbg(hsp->dev, "interrupt requested: %u\n", irq);
+	}
+
+	return 0;
+}
+
 static int tegra_hsp_probe(struct platform_device *pdev)
 {
 	struct tegra_hsp *hsp;
 	struct resource *res;
+	unsigned int i;
 	u32 value;
 	int err;
 
@@ -375,6 +665,7 @@ static int tegra_hsp_probe(struct platform_device *pdev)
 	if (!hsp)
 		return -ENOMEM;
 
+	hsp->dev = &pdev->dev;
 	hsp->soc = of_device_get_match_data(&pdev->dev);
 	INIT_LIST_HEAD(&hsp->doorbells);
 	spin_lock_init(&hsp->lock);
@@ -392,58 +683,136 @@ static int tegra_hsp_probe(struct platform_device *pdev)
 	hsp->num_si = (value >> HSP_nSI_SHIFT) & HSP_nINT_MASK;
 
 	err = platform_get_irq_byname(pdev, "doorbell");
-	if (err < 0) {
-		dev_err(&pdev->dev, "failed to get doorbell IRQ: %d\n", err);
-		return err;
+	if (err >= 0)
+		hsp->doorbell_irq = err;
+
+	if (hsp->num_si > 0) {
+		unsigned int count = 0;
+
+		hsp->shared_irqs = devm_kcalloc(&pdev->dev, hsp->num_si,
+						sizeof(*hsp->shared_irqs),
+						GFP_KERNEL);
+		if (!hsp->shared_irqs)
+			return -ENOMEM;
+
+		for (i = 0; i < hsp->num_si; i++) {
+			char *name;
+
+			name = kasprintf(GFP_KERNEL, "shared%u", i);
+			if (!name)
+				return -ENOMEM;
+
+			err = platform_get_irq_byname(pdev, name);
+			if (err >= 0) {
+				hsp->shared_irqs[i] = err;
+				count++;
+			}
+
+			kfree(name);
+		}
+
+		if (count == 0) {
+			devm_kfree(&pdev->dev, hsp->shared_irqs);
+			hsp->shared_irqs = NULL;
+		}
+	}
+
+	/* setup the doorbell controller */
+	hsp->mbox_db.of_xlate = tegra_hsp_db_xlate;
+	hsp->mbox_db.num_chans = 32;
+	hsp->mbox_db.dev = &pdev->dev;
+	hsp->mbox_db.ops = &tegra_hsp_db_ops;
+
+	hsp->mbox_db.chans = devm_kcalloc(&pdev->dev, hsp->mbox_db.num_chans,
+					  sizeof(*hsp->mbox_db.chans),
+					  GFP_KERNEL);
+	if (!hsp->mbox_db.chans)
+		return -ENOMEM;
+
+	if (hsp->doorbell_irq) {
+		err = tegra_hsp_add_doorbells(hsp);
+		if (err < 0) {
+			dev_err(&pdev->dev, "failed to add doorbells: %d\n",
+			        err);
+			return err;
+		}
 	}
 
-	hsp->irq = err;
+	err = mbox_controller_register(&hsp->mbox_db);
+	if (err < 0) {
+		dev_err(&pdev->dev, "failed to register doorbell mailbox: %d\n", err);
+		goto remove_doorbells;
+	}
 
-	hsp->mbox.of_xlate = of_tegra_hsp_xlate;
-	hsp->mbox.num_chans = 32;
-	hsp->mbox.dev = &pdev->dev;
-	hsp->mbox.txdone_irq = false;
-	hsp->mbox.txdone_poll = false;
-	hsp->mbox.ops = &tegra_hsp_doorbell_ops;
+	/* setup the shared mailbox controller */
+	hsp->mbox_sm.of_xlate = tegra_hsp_sm_xlate;
+	hsp->mbox_sm.num_chans = hsp->num_sm;
+	hsp->mbox_sm.dev = &pdev->dev;
+	hsp->mbox_sm.ops = &tegra_hsp_sm_ops;
 
-	hsp->mbox.chans = devm_kcalloc(&pdev->dev, hsp->mbox.num_chans,
-					sizeof(*hsp->mbox.chans),
-					GFP_KERNEL);
-	if (!hsp->mbox.chans)
+	hsp->mbox_sm.chans = devm_kcalloc(&pdev->dev, hsp->mbox_sm.num_chans,
+					  sizeof(*hsp->mbox_sm.chans),
+					  GFP_KERNEL);
+	if (!hsp->mbox_sm.chans)
 		return -ENOMEM;
 
-	err = tegra_hsp_add_doorbells(hsp);
+	if (hsp->shared_irqs) {
+		err = tegra_hsp_add_mailboxes(hsp, &pdev->dev);
+		if (err < 0) {
+			dev_err(&pdev->dev, "failed to add mailboxes: %d\n",
+			        err);
+			goto unregister_mbox_db;
+		}
+	}
+
+	err = mbox_controller_register(&hsp->mbox_sm);
 	if (err < 0) {
-		dev_err(&pdev->dev, "failed to add doorbells: %d\n", err);
-		return err;
+		dev_err(&pdev->dev, "failed to register shared mailbox: %d\n", err);
+		goto unregister_mbox_db;
 	}
 
 	platform_set_drvdata(pdev, hsp);
 
-	err = mbox_controller_register(&hsp->mbox);
-	if (err) {
-		dev_err(&pdev->dev, "failed to register mailbox: %d\n", err);
-		tegra_hsp_remove_doorbells(hsp);
-		return err;
+	if (hsp->doorbell_irq) {
+		err = devm_request_irq(&pdev->dev, hsp->doorbell_irq,
+				       tegra_hsp_doorbell_irq, IRQF_NO_SUSPEND,
+				       dev_name(&pdev->dev), hsp);
+		if (err < 0) {
+			dev_err(&pdev->dev,
+			        "failed to request doorbell IRQ#%u: %d\n",
+				hsp->doorbell_irq, err);
+			goto unregister_mbox_sm;
+		}
 	}
 
-	err = devm_request_irq(&pdev->dev, hsp->irq, tegra_hsp_doorbell_irq,
-			       IRQF_NO_SUSPEND, dev_name(&pdev->dev), hsp);
-	if (err < 0) {
-		dev_err(&pdev->dev, "failed to request IRQ#%u: %d\n",
-			hsp->irq, err);
-		return err;
+	if (hsp->shared_irqs) {
+		err = tegra_hsp_request_shared_irqs(hsp);
+		if (err < 0)
+			goto unregister_mbox_sm;
 	}
 
 	return 0;
+
+unregister_mbox_sm:
+	mbox_controller_unregister(&hsp->mbox_sm);
+unregister_mbox_db:
+	mbox_controller_unregister(&hsp->mbox_db);
+remove_doorbells:
+	if (hsp->doorbell_irq)
+		tegra_hsp_remove_doorbells(hsp);
+
+	return err;
 }
 
 static int tegra_hsp_remove(struct platform_device *pdev)
 {
 	struct tegra_hsp *hsp = platform_get_drvdata(pdev);
 
-	mbox_controller_unregister(&hsp->mbox);
-	tegra_hsp_remove_doorbells(hsp);
+	mbox_controller_unregister(&hsp->mbox_sm);
+	mbox_controller_unregister(&hsp->mbox_db);
+
+	if (hsp->doorbell_irq)
+		tegra_hsp_remove_doorbells(hsp);
 
 	return 0;
 }
@@ -456,10 +825,17 @@ static const struct tegra_hsp_db_map tegra186_hsp_db_map[] = {
 
 static const struct tegra_hsp_soc tegra186_hsp_soc = {
 	.map = tegra186_hsp_db_map,
+	.has_per_mb_ie = false,
+};
+
+static const struct tegra_hsp_soc tegra194_hsp_soc = {
+	.map = tegra186_hsp_db_map,
+	.has_per_mb_ie = true,
 };
 
 static const struct of_device_id tegra_hsp_match[] = {
 	{ .compatible = "nvidia,tegra186-hsp", .data = &tegra186_hsp_soc },
+	{ .compatible = "nvidia,tegra194-hsp", .data = &tegra194_hsp_soc },
 	{ }
 };
 
-- 
2.19.1

[PATCH v2 04/10] mailbox: tegra-hsp: Add support for shared mailboxes

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

The Tegra HSP block supports 'shared mailboxes' that are simple 32-bit
registers consisting of a FULL bit in MSB position and 31 bits of data.
The hardware can be configured to trigger interrupts when a mailbox
is empty or full. Add support for these shared mailboxes to the HSP
driver.

The initial use for the mailboxes is the Tegra Combined UART. For this
purpose, we use interrupts to receive data, and spinning to wait for
the transmit mailbox to be emptied to minimize unnecessary overhead.

Based on work by Mikko Perttunen <mperttunen@nvidia.com>.

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
Changes in v2:
- do not write per-mailbox interrupt enable registers on Tegra186
- merge handlers for empty and full interrupts
- track direction of shared mailboxes

 drivers/mailbox/tegra-hsp.c | 498 +++++++++++++++++++++++++++++++-----
 1 file changed, 437 insertions(+), 61 deletions(-)

diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
index 0cde356c11ab..0100a974149b 100644
--- a/drivers/mailbox/tegra-hsp.c
+++ b/drivers/mailbox/tegra-hsp.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.
+ * Copyright (c) 2016-2018, NVIDIA CORPORATION.  All rights reserved.
  *
  * This program is free software; you can redistribute it and/or modify it
  * under the terms and conditions of the GNU General Public License,
@@ -11,6 +11,7 @@
  * more details.
  */
 
+#include <linux/delay.h>
 #include <linux/interrupt.h>
 #include <linux/io.h>
 #include <linux/mailbox_controller.h>
@@ -21,6 +22,17 @@
 
 #include <dt-bindings/mailbox/tegra186-hsp.h>
 
+#include "mailbox.h"
+
+#define HSP_INT_IE(x)		(0x100 + ((x) * 4))
+#define HSP_INT_IV		0x300
+#define HSP_INT_IR		0x304
+
+#define HSP_INT_EMPTY_SHIFT	0
+#define HSP_INT_EMPTY_MASK	0xff
+#define HSP_INT_FULL_SHIFT	8
+#define HSP_INT_FULL_MASK	0xff
+
 #define HSP_INT_DIMENSIONING	0x380
 #define HSP_nSM_SHIFT		0
 #define HSP_nSS_SHIFT		4
@@ -34,6 +46,11 @@
 #define HSP_DB_RAW	0x8
 #define HSP_DB_PENDING	0xc
 
+#define HSP_SM_SHRD_MBOX	0x0
+#define HSP_SM_SHRD_MBOX_FULL	BIT(31)
+#define HSP_SM_SHRD_MBOX_FULL_INT_IE	0x04
+#define HSP_SM_SHRD_MBOX_EMPTY_INT_IE	0x08
+
 #define HSP_DB_CCPLEX		1
 #define HSP_DB_BPMP		3
 #define HSP_DB_MAX		7
@@ -55,6 +72,12 @@ struct tegra_hsp_doorbell {
 	unsigned int index;
 };
 
+struct tegra_hsp_mailbox {
+	struct tegra_hsp_channel channel;
+	unsigned int index;
+	bool producer;
+};
+
 struct tegra_hsp_db_map {
 	const char *name;
 	unsigned int master;
@@ -63,13 +86,18 @@ struct tegra_hsp_db_map {
 
 struct tegra_hsp_soc {
 	const struct tegra_hsp_db_map *map;
+	bool has_per_mb_ie;
 };
 
 struct tegra_hsp {
+	struct device *dev;
 	const struct tegra_hsp_soc *soc;
-	struct mbox_controller mbox;
+	struct mbox_controller mbox_db;
+	struct mbox_controller mbox_sm;
 	void __iomem *regs;
-	unsigned int irq;
+	unsigned int doorbell_irq;
+	unsigned int *shared_irqs;
+	unsigned int shared_irq;
 	unsigned int num_sm;
 	unsigned int num_as;
 	unsigned int num_ss;
@@ -78,13 +106,10 @@ struct tegra_hsp {
 	spinlock_t lock;
 
 	struct list_head doorbells;
-};
+	struct tegra_hsp_mailbox *mailboxes;
 
-static inline struct tegra_hsp *
-to_tegra_hsp(struct mbox_controller *mbox)
-{
-	return container_of(mbox, struct tegra_hsp, mbox);
-}
+	unsigned long mask;
+};
 
 static inline u32 tegra_hsp_readl(struct tegra_hsp *hsp, unsigned int offset)
 {
@@ -158,7 +183,7 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
 
 	spin_lock(&hsp->lock);
 
-	for_each_set_bit(master, &value, hsp->mbox.num_chans) {
+	for_each_set_bit(master, &value, hsp->mbox_db.num_chans) {
 		struct tegra_hsp_doorbell *db;
 
 		db = __tegra_hsp_doorbell_get(hsp, master);
@@ -182,6 +207,71 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
+static irqreturn_t tegra_hsp_shared_irq(int irq, void *data)
+{
+	struct tegra_hsp *hsp = data;
+	unsigned long bit, mask;
+	u32 status, value;
+	void *msg;
+
+	status = tegra_hsp_readl(hsp, HSP_INT_IR) & hsp->mask;
+
+	/* process EMPTY interrupts first */
+	mask = (status >> HSP_INT_EMPTY_SHIFT) & HSP_INT_EMPTY_MASK;
+
+	for_each_set_bit(bit, &mask, hsp->num_sm) {
+		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
+
+		if (mb->producer) {
+			/*
+			 * Disable EMPTY interrupts until data is sent with
+			 * the next message. These interrupts are level-
+			 * triggered, so if we kept them enabled they would
+			 * constantly trigger until we next write data into
+			 * the message.
+			 */
+			spin_lock(&hsp->lock);
+
+			hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+			tegra_hsp_writel(hsp, hsp->mask,
+					 HSP_INT_IE(hsp->shared_irq));
+
+			spin_unlock(&hsp->lock);
+
+			mbox_chan_txdone(mb->channel.chan, 0);
+		}
+	}
+
+	/* process FULL interrupts */
+	mask = (status >> HSP_INT_FULL_SHIFT) & HSP_INT_FULL_MASK;
+
+	for_each_set_bit(bit, &mask, hsp->num_sm) {
+		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
+
+		if (!mb->producer) {
+			value = tegra_hsp_channel_readl(&mb->channel,
+							HSP_SM_SHRD_MBOX);
+			value &= ~HSP_SM_SHRD_MBOX_FULL;
+			msg = (void *)(unsigned long)value;
+			mbox_chan_received_data(mb->channel.chan, msg);
+
+			/*
+			 * Need to clear all bits here since some producers,
+			 * such as TCU, depend on fields in the register
+			 * getting cleared by the consumer.
+			 *
+			 * The mailbox API doesn't give the consumers a way
+			 * of doing that explicitly, so we have to make sure
+			 * we cover all possible cases.
+			 */
+			tegra_hsp_channel_writel(&mb->channel, 0x0,
+						 HSP_SM_SHRD_MBOX);
+		}
+	}
+
+	return IRQ_HANDLED;
+}
+
 static struct tegra_hsp_channel *
 tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
 			  unsigned int master, unsigned int index)
@@ -194,7 +284,7 @@ tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
 	if (!db)
 		return ERR_PTR(-ENOMEM);
 
-	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) << 16;
+	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) * SZ_64K;
 	offset += index * 0x100;
 
 	db->channel.regs = hsp->regs + offset;
@@ -235,8 +325,8 @@ static int tegra_hsp_doorbell_startup(struct mbox_chan *chan)
 	unsigned long flags;
 	u32 value;
 
-	if (db->master >= hsp->mbox.num_chans) {
-		dev_err(hsp->mbox.dev,
+	if (db->master >= chan->mbox->num_chans) {
+		dev_err(chan->mbox->dev,
 			"invalid master ID %u for HSP channel\n",
 			db->master);
 		return -EINVAL;
@@ -281,46 +371,168 @@ static void tegra_hsp_doorbell_shutdown(struct mbox_chan *chan)
 	spin_unlock_irqrestore(&hsp->lock, flags);
 }
 
-static const struct mbox_chan_ops tegra_hsp_doorbell_ops = {
+static const struct mbox_chan_ops tegra_hsp_db_ops = {
 	.send_data = tegra_hsp_doorbell_send_data,
 	.startup = tegra_hsp_doorbell_startup,
 	.shutdown = tegra_hsp_doorbell_shutdown,
 };
 
-static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
+static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
+{
+	struct tegra_hsp_mailbox *mb = chan->con_priv;
+	struct tegra_hsp *hsp = mb->channel.hsp;
+	unsigned long flags;
+	u32 value;
+
+	WARN_ON(!mb->producer);
+
+	/* copy data and mark mailbox full */
+	value = (u32)(unsigned long)data;
+	value |= HSP_SM_SHRD_MBOX_FULL;
+
+	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
+
+	if (!irqs_disabled()) {
+		/* enable EMPTY interrupt for the shared mailbox */
+		spin_lock_irqsave(&hsp->lock, flags);
+
+		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
+
+		spin_unlock_irqrestore(&hsp->lock, flags);
+	}
+
+	return 0;
+}
+
+static int tegra_hsp_mailbox_flush(struct mbox_chan *chan,
+				   unsigned long timeout)
+{
+	struct tegra_hsp_mailbox *mb = chan->con_priv;
+	struct tegra_hsp_channel *ch = &mb->channel;
+	u32 value;
+
+	timeout = jiffies + msecs_to_jiffies(timeout);
+
+	while (time_before(jiffies, timeout)) {
+		value = tegra_hsp_channel_readl(ch, HSP_SM_SHRD_MBOX);
+		if ((value & HSP_SM_SHRD_MBOX_FULL) == 0) {
+			mbox_chan_txdone(chan, 0);
+			return 0;
+		}
+
+		udelay(1);
+	}
+
+	return -ETIME;
+}
+
+static int tegra_hsp_mailbox_startup(struct mbox_chan *chan)
+{
+	struct tegra_hsp_mailbox *mb = chan->con_priv;
+	struct tegra_hsp_channel *ch = &mb->channel;
+	struct tegra_hsp *hsp = mb->channel.hsp;
+	unsigned long flags;
+
+	chan->txdone_method = TXDONE_BY_IRQ;
+
+	/*
+	 * Shared mailboxes start out as consumers by default. FULL and EMPTY
+	 * interrupts are coalesced at the same shared interrupt.
+	 *
+	 * Keep EMPTY interrupts disabled at startup and only enable them when
+	 * the mailbox is actually full. This is required because the FULL and
+	 * EMPTY interrupts are level-triggered, so keeping EMPTY interrupts
+	 * enabled all the time would cause an interrupt storm while mailboxes
+	 * are idle.
+	 */
+
+	spin_lock_irqsave(&hsp->lock, flags);
+
+	if (mb->producer)
+		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+	else
+		hsp->mask |= BIT(HSP_INT_FULL_SHIFT + mb->index);
+
+	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
+
+	spin_unlock_irqrestore(&hsp->lock, flags);
+
+	if (hsp->soc->has_per_mb_ie) {
+		if (mb->producer)
+			tegra_hsp_channel_writel(ch, 0x0,
+						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
+		else
+			tegra_hsp_channel_writel(ch, 0x1,
+						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
+	}
+
+	return 0;
+}
+
+static void tegra_hsp_mailbox_shutdown(struct mbox_chan *chan)
+{
+	struct tegra_hsp_mailbox *mb = chan->con_priv;
+	struct tegra_hsp_channel *ch = &mb->channel;
+	struct tegra_hsp *hsp = mb->channel.hsp;
+	unsigned long flags;
+
+	if (hsp->soc->has_per_mb_ie) {
+		if (mb->producer)
+			tegra_hsp_channel_writel(ch, 0x0,
+						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
+		else
+			tegra_hsp_channel_writel(ch, 0x0,
+						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
+	}
+
+	spin_lock_irqsave(&hsp->lock, flags);
+
+	if (mb->producer)
+		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+	else
+		hsp->mask &= ~BIT(HSP_INT_FULL_SHIFT + mb->index);
+
+	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
+
+	spin_unlock_irqrestore(&hsp->lock, flags);
+}
+
+static const struct mbox_chan_ops tegra_hsp_sm_ops = {
+	.send_data = tegra_hsp_mailbox_send_data,
+	.flush = tegra_hsp_mailbox_flush,
+	.startup = tegra_hsp_mailbox_startup,
+	.shutdown = tegra_hsp_mailbox_shutdown,
+};
+
+static struct mbox_chan *tegra_hsp_db_xlate(struct mbox_controller *mbox,
 					    const struct of_phandle_args *args)
 {
+	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_db);
+	unsigned int type = args->args[0], master = args->args[1];
 	struct tegra_hsp_channel *channel = ERR_PTR(-ENODEV);
-	struct tegra_hsp *hsp = to_tegra_hsp(mbox);
-	unsigned int type = args->args[0];
-	unsigned int master = args->args[1];
 	struct tegra_hsp_doorbell *db;
 	struct mbox_chan *chan;
 	unsigned long flags;
 	unsigned int i;
 
-	switch (type) {
-	case TEGRA_HSP_MBOX_TYPE_DB:
-		db = tegra_hsp_doorbell_get(hsp, master);
-		if (db)
-			channel = &db->channel;
+	if (type != TEGRA_HSP_MBOX_TYPE_DB || !hsp->doorbell_irq)
+		return ERR_PTR(-ENODEV);
 
-		break;
-
-	default:
-		break;
-	}
+	db = tegra_hsp_doorbell_get(hsp, master);
+	if (db)
+		channel = &db->channel;
 
 	if (IS_ERR(channel))
 		return ERR_CAST(channel);
 
 	spin_lock_irqsave(&hsp->lock, flags);
 
-	for (i = 0; i < hsp->mbox.num_chans; i++) {
-		chan = &hsp->mbox.chans[i];
+	for (i = 0; i < mbox->num_chans; i++) {
+		chan = &mbox->chans[i];
 		if (!chan->con_priv) {
-			chan->con_priv = channel;
 			channel->chan = chan;
+			chan->con_priv = db;
 			break;
 		}
 
@@ -332,6 +544,29 @@ static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
 	return chan ?: ERR_PTR(-EBUSY);
 }
 
+static struct mbox_chan *tegra_hsp_sm_xlate(struct mbox_controller *mbox,
+					    const struct of_phandle_args *args)
+{
+	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_sm);
+	unsigned int type = args->args[0], index;
+	struct tegra_hsp_mailbox *mb;
+
+	index = args->args[1] & TEGRA_HSP_SM_MASK;
+
+	if (type != TEGRA_HSP_MBOX_TYPE_SM || !hsp->shared_irqs ||
+	    index >= hsp->num_sm)
+		return ERR_PTR(-ENODEV);
+
+	mb = &hsp->mailboxes[index];
+
+	if ((args->args[1] & TEGRA_HSP_SM_FLAG_TX) == 0)
+		mb->producer = false;
+	else
+		mb->producer = true;
+
+	return mb->channel.chan;
+}
+
 static void tegra_hsp_remove_doorbells(struct tegra_hsp *hsp)
 {
 	struct tegra_hsp_doorbell *db, *tmp;
@@ -364,10 +599,65 @@ static int tegra_hsp_add_doorbells(struct tegra_hsp *hsp)
 	return 0;
 }
 
+static int tegra_hsp_add_mailboxes(struct tegra_hsp *hsp, struct device *dev)
+{
+	int i;
+
+	hsp->mailboxes = devm_kcalloc(dev, hsp->num_sm, sizeof(*hsp->mailboxes),
+				      GFP_KERNEL);
+	if (!hsp->mailboxes)
+		return -ENOMEM;
+
+	for (i = 0; i < hsp->num_sm; i++) {
+		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
+
+		mb->index = i;
+
+		mb->channel.hsp = hsp;
+		mb->channel.regs = hsp->regs + SZ_64K + i * SZ_32K;
+		mb->channel.chan = &hsp->mbox_sm.chans[i];
+		mb->channel.chan->con_priv = mb;
+	}
+
+	return 0;
+}
+
+static int tegra_hsp_request_shared_irqs(struct tegra_hsp *hsp)
+{
+	unsigned int i, irq = 0;
+	int err;
+
+	for (i = 0; i < hsp->num_si; i++) {
+		if (hsp->shared_irq == 0 && hsp->shared_irqs[i] > 0) {
+			irq = hsp->shared_irqs[i];
+			hsp->shared_irq = i;
+			break;
+		}
+	}
+
+	if (irq > 0) {
+		err = devm_request_irq(hsp->dev, irq, tegra_hsp_shared_irq, 0,
+				       dev_name(hsp->dev), hsp);
+		if (err < 0) {
+			dev_err(hsp->dev, "failed to request interrupt: %d\n",
+				err);
+			return err;
+		}
+
+		/* disable all interrupts */
+		tegra_hsp_writel(hsp, 0, HSP_INT_IE(hsp->shared_irq));
+
+		dev_dbg(hsp->dev, "interrupt requested: %u\n", irq);
+	}
+
+	return 0;
+}
+
 static int tegra_hsp_probe(struct platform_device *pdev)
 {
 	struct tegra_hsp *hsp;
 	struct resource *res;
+	unsigned int i;
 	u32 value;
 	int err;
 
@@ -375,6 +665,7 @@ static int tegra_hsp_probe(struct platform_device *pdev)
 	if (!hsp)
 		return -ENOMEM;
 
+	hsp->dev = &pdev->dev;
 	hsp->soc = of_device_get_match_data(&pdev->dev);
 	INIT_LIST_HEAD(&hsp->doorbells);
 	spin_lock_init(&hsp->lock);
@@ -392,58 +683,136 @@ static int tegra_hsp_probe(struct platform_device *pdev)
 	hsp->num_si = (value >> HSP_nSI_SHIFT) & HSP_nINT_MASK;
 
 	err = platform_get_irq_byname(pdev, "doorbell");
-	if (err < 0) {
-		dev_err(&pdev->dev, "failed to get doorbell IRQ: %d\n", err);
-		return err;
+	if (err >= 0)
+		hsp->doorbell_irq = err;
+
+	if (hsp->num_si > 0) {
+		unsigned int count = 0;
+
+		hsp->shared_irqs = devm_kcalloc(&pdev->dev, hsp->num_si,
+						sizeof(*hsp->shared_irqs),
+						GFP_KERNEL);
+		if (!hsp->shared_irqs)
+			return -ENOMEM;
+
+		for (i = 0; i < hsp->num_si; i++) {
+			char *name;
+
+			name = kasprintf(GFP_KERNEL, "shared%u", i);
+			if (!name)
+				return -ENOMEM;
+
+			err = platform_get_irq_byname(pdev, name);
+			if (err >= 0) {
+				hsp->shared_irqs[i] = err;
+				count++;
+			}
+
+			kfree(name);
+		}
+
+		if (count == 0) {
+			devm_kfree(&pdev->dev, hsp->shared_irqs);
+			hsp->shared_irqs = NULL;
+		}
+	}
+
+	/* setup the doorbell controller */
+	hsp->mbox_db.of_xlate = tegra_hsp_db_xlate;
+	hsp->mbox_db.num_chans = 32;
+	hsp->mbox_db.dev = &pdev->dev;
+	hsp->mbox_db.ops = &tegra_hsp_db_ops;
+
+	hsp->mbox_db.chans = devm_kcalloc(&pdev->dev, hsp->mbox_db.num_chans,
+					  sizeof(*hsp->mbox_db.chans),
+					  GFP_KERNEL);
+	if (!hsp->mbox_db.chans)
+		return -ENOMEM;
+
+	if (hsp->doorbell_irq) {
+		err = tegra_hsp_add_doorbells(hsp);
+		if (err < 0) {
+			dev_err(&pdev->dev, "failed to add doorbells: %d\n",
+			        err);
+			return err;
+		}
 	}
 
-	hsp->irq = err;
+	err = mbox_controller_register(&hsp->mbox_db);
+	if (err < 0) {
+		dev_err(&pdev->dev, "failed to register doorbell mailbox: %d\n", err);
+		goto remove_doorbells;
+	}
 
-	hsp->mbox.of_xlate = of_tegra_hsp_xlate;
-	hsp->mbox.num_chans = 32;
-	hsp->mbox.dev = &pdev->dev;
-	hsp->mbox.txdone_irq = false;
-	hsp->mbox.txdone_poll = false;
-	hsp->mbox.ops = &tegra_hsp_doorbell_ops;
+	/* setup the shared mailbox controller */
+	hsp->mbox_sm.of_xlate = tegra_hsp_sm_xlate;
+	hsp->mbox_sm.num_chans = hsp->num_sm;
+	hsp->mbox_sm.dev = &pdev->dev;
+	hsp->mbox_sm.ops = &tegra_hsp_sm_ops;
 
-	hsp->mbox.chans = devm_kcalloc(&pdev->dev, hsp->mbox.num_chans,
-					sizeof(*hsp->mbox.chans),
-					GFP_KERNEL);
-	if (!hsp->mbox.chans)
+	hsp->mbox_sm.chans = devm_kcalloc(&pdev->dev, hsp->mbox_sm.num_chans,
+					  sizeof(*hsp->mbox_sm.chans),
+					  GFP_KERNEL);
+	if (!hsp->mbox_sm.chans)
 		return -ENOMEM;
 
-	err = tegra_hsp_add_doorbells(hsp);
+	if (hsp->shared_irqs) {
+		err = tegra_hsp_add_mailboxes(hsp, &pdev->dev);
+		if (err < 0) {
+			dev_err(&pdev->dev, "failed to add mailboxes: %d\n",
+			        err);
+			goto unregister_mbox_db;
+		}
+	}
+
+	err = mbox_controller_register(&hsp->mbox_sm);
 	if (err < 0) {
-		dev_err(&pdev->dev, "failed to add doorbells: %d\n", err);
-		return err;
+		dev_err(&pdev->dev, "failed to register shared mailbox: %d\n", err);
+		goto unregister_mbox_db;
 	}
 
 	platform_set_drvdata(pdev, hsp);
 
-	err = mbox_controller_register(&hsp->mbox);
-	if (err) {
-		dev_err(&pdev->dev, "failed to register mailbox: %d\n", err);
-		tegra_hsp_remove_doorbells(hsp);
-		return err;
+	if (hsp->doorbell_irq) {
+		err = devm_request_irq(&pdev->dev, hsp->doorbell_irq,
+				       tegra_hsp_doorbell_irq, IRQF_NO_SUSPEND,
+				       dev_name(&pdev->dev), hsp);
+		if (err < 0) {
+			dev_err(&pdev->dev,
+			        "failed to request doorbell IRQ#%u: %d\n",
+				hsp->doorbell_irq, err);
+			goto unregister_mbox_sm;
+		}
 	}
 
-	err = devm_request_irq(&pdev->dev, hsp->irq, tegra_hsp_doorbell_irq,
-			       IRQF_NO_SUSPEND, dev_name(&pdev->dev), hsp);
-	if (err < 0) {
-		dev_err(&pdev->dev, "failed to request IRQ#%u: %d\n",
-			hsp->irq, err);
-		return err;
+	if (hsp->shared_irqs) {
+		err = tegra_hsp_request_shared_irqs(hsp);
+		if (err < 0)
+			goto unregister_mbox_sm;
 	}
 
 	return 0;
+
+unregister_mbox_sm:
+	mbox_controller_unregister(&hsp->mbox_sm);
+unregister_mbox_db:
+	mbox_controller_unregister(&hsp->mbox_db);
+remove_doorbells:
+	if (hsp->doorbell_irq)
+		tegra_hsp_remove_doorbells(hsp);
+
+	return err;
 }
 
 static int tegra_hsp_remove(struct platform_device *pdev)
 {
 	struct tegra_hsp *hsp = platform_get_drvdata(pdev);
 
-	mbox_controller_unregister(&hsp->mbox);
-	tegra_hsp_remove_doorbells(hsp);
+	mbox_controller_unregister(&hsp->mbox_sm);
+	mbox_controller_unregister(&hsp->mbox_db);
+
+	if (hsp->doorbell_irq)
+		tegra_hsp_remove_doorbells(hsp);
 
 	return 0;
 }
@@ -456,10 +825,17 @@ static const struct tegra_hsp_db_map tegra186_hsp_db_map[] = {
 
 static const struct tegra_hsp_soc tegra186_hsp_soc = {
 	.map = tegra186_hsp_db_map,
+	.has_per_mb_ie = false,
+};
+
+static const struct tegra_hsp_soc tegra194_hsp_soc = {
+	.map = tegra186_hsp_db_map,
+	.has_per_mb_ie = true,
 };
 
 static const struct of_device_id tegra_hsp_match[] = {
 	{ .compatible = "nvidia,tegra186-hsp", .data = &tegra186_hsp_soc },
+	{ .compatible = "nvidia,tegra194-hsp", .data = &tegra194_hsp_soc },
 	{ }
 };
 
-- 
2.19.1

[PATCH v2 05/10] mailbox: tegra-hsp: Add suspend/resume support

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

Upon resuming from a system sleep state, the interrupts for all active
shared mailboxes need to be reenabled, otherwise they will not work.

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 drivers/mailbox/tegra-hsp.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
index 0100a974149b..1259abf3542f 100644
--- a/drivers/mailbox/tegra-hsp.c
+++ b/drivers/mailbox/tegra-hsp.c
@@ -18,6 +18,7 @@
 #include <linux/of.h>
 #include <linux/of_device.h>
 #include <linux/platform_device.h>
+#include <linux/pm.h>
 #include <linux/slab.h>
 
 #include <dt-bindings/mailbox/tegra186-hsp.h>
@@ -817,6 +818,23 @@ static int tegra_hsp_remove(struct platform_device *pdev)
 	return 0;
 }
 
+static int tegra_hsp_resume(struct device *dev)
+{
+	struct tegra_hsp *hsp = dev_get_drvdata(dev);
+	unsigned int i;
+
+	for (i = 0; i < hsp->num_sm; i++) {
+		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
+
+		if (mb->channel.chan->cl)
+			tegra_hsp_mailbox_startup(mb->channel.chan);
+	}
+
+	return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(tegra_hsp_pm_ops, NULL, tegra_hsp_resume);
+
 static const struct tegra_hsp_db_map tegra186_hsp_db_map[] = {
 	{ "ccplex", TEGRA_HSP_DB_MASTER_CCPLEX, HSP_DB_CCPLEX, },
 	{ "bpmp",   TEGRA_HSP_DB_MASTER_BPMP,   HSP_DB_BPMP,   },
@@ -843,6 +861,7 @@ static struct platform_driver tegra_hsp_driver = {
 	.driver = {
 		.name = "tegra-hsp",
 		.of_match_table = tegra_hsp_match,
+		.pm = &tegra_hsp_pm_ops,
 	},
 	.probe = tegra_hsp_probe,
 	.remove = tegra_hsp_remove,
-- 
2.19.1

[PATCH v2 05/10] mailbox: tegra-hsp: Add suspend/resume support

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

Upon resuming from a system sleep state, the interrupts for all active
shared mailboxes need to be reenabled, otherwise they will not work.

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 drivers/mailbox/tegra-hsp.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
index 0100a974149b..1259abf3542f 100644
--- a/drivers/mailbox/tegra-hsp.c
+++ b/drivers/mailbox/tegra-hsp.c
@@ -18,6 +18,7 @@
 #include <linux/of.h>
 #include <linux/of_device.h>
 #include <linux/platform_device.h>
+#include <linux/pm.h>
 #include <linux/slab.h>
 
 #include <dt-bindings/mailbox/tegra186-hsp.h>
@@ -817,6 +818,23 @@ static int tegra_hsp_remove(struct platform_device *pdev)
 	return 0;
 }
 
+static int tegra_hsp_resume(struct device *dev)
+{
+	struct tegra_hsp *hsp = dev_get_drvdata(dev);
+	unsigned int i;
+
+	for (i = 0; i < hsp->num_sm; i++) {
+		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
+
+		if (mb->channel.chan->cl)
+			tegra_hsp_mailbox_startup(mb->channel.chan);
+	}
+
+	return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(tegra_hsp_pm_ops, NULL, tegra_hsp_resume);
+
 static const struct tegra_hsp_db_map tegra186_hsp_db_map[] = {
 	{ "ccplex", TEGRA_HSP_DB_MASTER_CCPLEX, HSP_DB_CCPLEX, },
 	{ "bpmp",   TEGRA_HSP_DB_MASTER_BPMP,   HSP_DB_BPMP,   },
@@ -843,6 +861,7 @@ static struct platform_driver tegra_hsp_driver = {
 	.driver = {
 		.name = "tegra-hsp",
 		.of_match_table = tegra_hsp_match,
+		.pm = &tegra_hsp_pm_ops,
 	},
 	.probe = tegra_hsp_probe,
 	.remove = tegra_hsp_remove,
-- 
2.19.1

[PATCH v2 06/10] dt-bindings: serial: Add bindings for nvidia, tegra194-tcu

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

Add bindings for the Tegra Combined UART device used to talk to the
UART console on Tegra194 systems.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Reviewed-by: Rob Herring <robh@kernel.org>
Acked-by: Jon Hunter <jonathanh@nvidia.com>
Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 .../bindings/serial/nvidia,tegra194-tcu.txt   | 35 +++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt

diff --git a/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
new file mode 100644
index 000000000000..085a8591accd
--- /dev/null
+++ b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
@@ -0,0 +1,35 @@
+NVIDIA Tegra Combined UART (TCU)
+
+The TCU is a system for sharing a hardware UART instance among multiple
+systems within the Tegra SoC. It is implemented through a mailbox-
+based protocol where each "virtual UART" has a pair of mailboxes, one
+for transmitting and one for receiving, that is used to communicate
+with the hardware implementing the TCU.
+
+Required properties:
+- name : Should be tcu
+- compatible
+    Array of strings
+    One of:
+    - "nvidia,tegra194-tcu"
+- mbox-names:
+    "rx" - Mailbox for receiving data from hardware UART
+    "tx" - Mailbox for transmitting data to hardware UART
+- mboxes: Mailboxes corresponding to the mbox-names.
+
+This node is a mailbox consumer. See the following files for details of
+the mailbox subsystem, and the specifiers implemented by the relevant
+provider(s):
+
+- .../mailbox/mailbox.txt
+- .../mailbox/nvidia,tegra186-hsp.txt
+
+Example bindings:
+-----------------
+
+tcu: tcu {
+	compatible = "nvidia,tegra194-tcu";
+	mboxes = <&hsp_top0 TEGRA_HSP_MBOX_TYPE_SM 0>,
+	         <&hsp_aon TEGRA_HSP_MBOX_TYPE_SM 1>;
+	mbox-names = "rx", "tx";
+};
-- 
2.19.1

[PATCH v2 06/10] dt-bindings: serial: Add bindings for nvidia, tegra194-tcu

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

Add bindings for the Tegra Combined UART device used to talk to the
UART console on Tegra194 systems.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Reviewed-by: Rob Herring <robh@kernel.org>
Acked-by: Jon Hunter <jonathanh@nvidia.com>
Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 .../bindings/serial/nvidia,tegra194-tcu.txt   | 35 +++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt

diff --git a/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
new file mode 100644
index 000000000000..085a8591accd
--- /dev/null
+++ b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
@@ -0,0 +1,35 @@
+NVIDIA Tegra Combined UART (TCU)
+
+The TCU is a system for sharing a hardware UART instance among multiple
+systems within the Tegra SoC. It is implemented through a mailbox-
+based protocol where each "virtual UART" has a pair of mailboxes, one
+for transmitting and one for receiving, that is used to communicate
+with the hardware implementing the TCU.
+
+Required properties:
+- name : Should be tcu
+- compatible
+    Array of strings
+    One of:
+    - "nvidia,tegra194-tcu"
+- mbox-names:
+    "rx" - Mailbox for receiving data from hardware UART
+    "tx" - Mailbox for transmitting data to hardware UART
+- mboxes: Mailboxes corresponding to the mbox-names.
+
+This node is a mailbox consumer. See the following files for details of
+the mailbox subsystem, and the specifiers implemented by the relevant
+provider(s):
+
+- .../mailbox/mailbox.txt
+- .../mailbox/nvidia,tegra186-hsp.txt
+
+Example bindings:
+-----------------
+
+tcu: tcu {
+	compatible = "nvidia,tegra194-tcu";
+	mboxes = <&hsp_top0 TEGRA_HSP_MBOX_TYPE_SM 0>,
+	         <&hsp_aon TEGRA_HSP_MBOX_TYPE_SM 1>;
+	mbox-names = "rx", "tx";
+};
-- 
2.19.1

[PATCH v2 07/10] serial: Add Tegra Combined UART driver

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

The Tegra Combined UART (TCU) is a mailbox-based mechanism that allows
multiplexing multiple "virtual UARTs" into a single hardware serial
port. The TCU is the primary serial port on Tegra194 devices.

Add a TCU driver utilizing the mailbox framework, as the used mailboxes
are part of Tegra HSP blocks that are already controlled by the Tegra
HSP mailbox driver.

Based on work by  Mikko Perttunen <mperttunen@nvidia.com>.

Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 drivers/tty/serial/Kconfig       |  22 +++
 drivers/tty/serial/Makefile      |   1 +
 drivers/tty/serial/tegra-tcu.c   | 299 +++++++++++++++++++++++++++++++
 include/uapi/linux/serial_core.h |   3 +
 4 files changed, 325 insertions(+)
 create mode 100644 drivers/tty/serial/tegra-tcu.c

diff --git a/drivers/tty/serial/Kconfig b/drivers/tty/serial/Kconfig
index 32886c304641..785306388aa4 100644
--- a/drivers/tty/serial/Kconfig
+++ b/drivers/tty/serial/Kconfig
@@ -323,6 +323,28 @@ config SERIAL_TEGRA
 	  are enabled). This driver uses the APB DMA to achieve higher baudrate
 	  and better performance.
 
+config SERIAL_TEGRA_TCU
+	tristate "NVIDIA Tegra Combined UART"
+	depends on ARCH_TEGRA && TEGRA_HSP_MBOX
+	select SERIAL_CORE
+	help
+	  Support for the mailbox-based TCU (Tegra Combined UART) serial port.
+	  TCU is a virtual serial port that allows multiplexing multiple data
+	  streams into a single hardware serial port.
+
+config SERIAL_TEGRA_TCU_CONSOLE
+	bool "Support for console on a Tegra TCU serial port"
+	depends on SERIAL_TEGRA_TCU=y
+	select SERIAL_CORE_CONSOLE
+	default y
+	---help---
+	  If you say Y here, it will be possible to use a the Tegra TCU as the
+	  system console (the system console is the device which receives all
+	  kernel messages and warnings and which allows logins in single user
+	  mode).
+
+	  If unsure, say Y.
+
 config SERIAL_MAX3100
 	tristate "MAX3100 support"
 	depends on SPI
diff --git a/drivers/tty/serial/Makefile b/drivers/tty/serial/Makefile
index daac675612df..4ad82231ff8a 100644
--- a/drivers/tty/serial/Makefile
+++ b/drivers/tty/serial/Makefile
@@ -76,6 +76,7 @@ obj-$(CONFIG_SERIAL_LANTIQ)	+= lantiq.o
 obj-$(CONFIG_SERIAL_XILINX_PS_UART) += xilinx_uartps.o
 obj-$(CONFIG_SERIAL_SIRFSOC) += sirfsoc_uart.o
 obj-$(CONFIG_SERIAL_TEGRA) += serial-tegra.o
+obj-$(CONFIG_SERIAL_TEGRA_TCU) += tegra-tcu.o
 obj-$(CONFIG_SERIAL_AR933X)   += ar933x_uart.o
 obj-$(CONFIG_SERIAL_EFM32_UART) += efm32-uart.o
 obj-$(CONFIG_SERIAL_ARC)	+= arc_uart.o
diff --git a/drivers/tty/serial/tegra-tcu.c b/drivers/tty/serial/tegra-tcu.c
new file mode 100644
index 000000000000..1d360cd03b18
--- /dev/null
+++ b/drivers/tty/serial/tegra-tcu.c
@@ -0,0 +1,299 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+ */
+
+#include <linux/console.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/serial.h>
+#include <linux/serial_core.h>
+#include <linux/slab.h>
+#include <linux/tty.h>
+#include <linux/tty_flip.h>
+
+#define TCU_MBOX_BYTE(i, x)			((x) << (i * 8))
+#define TCU_MBOX_BYTE_V(x, i)			(((x) >> (i * 8)) & 0xff)
+#define TCU_MBOX_NUM_BYTES(x)			((x) << 24)
+#define TCU_MBOX_NUM_BYTES_V(x)			(((x) >> 24) & 0x3)
+
+struct tegra_tcu {
+	struct uart_driver driver;
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	struct console console;
+#endif
+	struct uart_port port;
+
+	struct mbox_client tx_client, rx_client;
+	struct mbox_chan *tx, *rx;
+};
+
+static unsigned int tegra_tcu_uart_tx_empty(struct uart_port *port)
+{
+	return TIOCSER_TEMT;
+}
+
+static void tegra_tcu_uart_set_mctrl(struct uart_port *port, unsigned int mctrl)
+{
+}
+
+static unsigned int tegra_tcu_uart_get_mctrl(struct uart_port *port)
+{
+	return 0;
+}
+
+static void tegra_tcu_uart_stop_tx(struct uart_port *port)
+{
+}
+
+static void tegra_tcu_write_one(struct tegra_tcu *tcu, u32 value,
+			       unsigned int count)
+{
+	void *msg;
+
+	value |= TCU_MBOX_NUM_BYTES(count);
+	msg = (void *)(unsigned long)value;
+	mbox_send_message(tcu->tx, msg);
+}
+
+static void tegra_tcu_write(struct tegra_tcu *tcu, const char *s,
+			    unsigned int count)
+{
+	unsigned int written = 0, i = 0;
+	bool insert_nl = false;
+	u32 value = 0;
+
+	while (i < count) {
+		if (insert_nl) {
+			value |= TCU_MBOX_BYTE(written++, '\n');
+			insert_nl = false;
+			i++;
+		} else if (s[i] == '\n') {
+			value |= TCU_MBOX_BYTE(written++, '\r');
+			insert_nl = true;
+		} else {
+			value |= TCU_MBOX_BYTE(written++, s[i++]);
+		}
+
+		if (written == 3) {
+			tegra_tcu_write_one(tcu, value, 3);
+			value = written = 0;
+		}
+	}
+
+	if (written)
+		tegra_tcu_write_one(tcu, value, written);
+}
+
+static void tegra_tcu_uart_start_tx(struct uart_port *port)
+{
+	struct tegra_tcu *tcu = port->private_data;
+	struct circ_buf *xmit = &port->state->xmit;
+	unsigned long count;
+
+	for (;;) {
+		count = CIRC_CNT_TO_END(xmit->head, xmit->tail, UART_XMIT_SIZE);
+		if (!count)
+			break;
+
+		tegra_tcu_write(tcu, &xmit->buf[xmit->tail], count);
+		xmit->tail = (xmit->tail + count) & (UART_XMIT_SIZE - 1);
+	}
+
+	uart_write_wakeup(port);
+}
+
+static void tegra_tcu_uart_stop_rx(struct uart_port *port)
+{
+}
+
+static void tegra_tcu_uart_break_ctl(struct uart_port *port, int ctl)
+{
+}
+
+static int tegra_tcu_uart_startup(struct uart_port *port)
+{
+	return 0;
+}
+
+static void tegra_tcu_uart_shutdown(struct uart_port *port)
+{
+}
+
+static void tegra_tcu_uart_set_termios(struct uart_port *port,
+				       struct ktermios *new,
+				       struct ktermios *old)
+{
+}
+
+static const struct uart_ops tegra_tcu_uart_ops = {
+	.tx_empty = tegra_tcu_uart_tx_empty,
+	.set_mctrl = tegra_tcu_uart_set_mctrl,
+	.get_mctrl = tegra_tcu_uart_get_mctrl,
+	.stop_tx = tegra_tcu_uart_stop_tx,
+	.start_tx = tegra_tcu_uart_start_tx,
+	.stop_rx = tegra_tcu_uart_stop_rx,
+	.break_ctl = tegra_tcu_uart_break_ctl,
+	.startup = tegra_tcu_uart_startup,
+	.shutdown = tegra_tcu_uart_shutdown,
+	.set_termios = tegra_tcu_uart_set_termios,
+};
+
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+static void tegra_tcu_console_write(struct console *cons, const char *s,
+				    unsigned int count)
+{
+	struct tegra_tcu *tcu = container_of(cons, struct tegra_tcu, console);
+
+	tegra_tcu_write(tcu, s, count);
+}
+
+static int tegra_tcu_console_setup(struct console *cons, char *options)
+{
+	return 0;
+}
+#endif
+
+static void tegra_tcu_receive(struct mbox_client *cl, void *msg)
+{
+	struct tegra_tcu *tcu = container_of(cl, struct tegra_tcu, rx_client);
+	struct tty_port *port = &tcu->port.state->port;
+	u32 value = (u32)(unsigned long)msg;
+	unsigned int num_bytes, i;
+
+	num_bytes = TCU_MBOX_NUM_BYTES_V(value);
+
+	for (i = 0; i < num_bytes; i++)
+		tty_insert_flip_char(port, TCU_MBOX_BYTE_V(value, i),
+				     TTY_NORMAL);
+
+	tty_flip_buffer_push(port);
+}
+
+static int tegra_tcu_probe(struct platform_device *pdev)
+{
+	struct uart_port *port;
+	struct tegra_tcu *tcu;
+	int err;
+
+	tcu = devm_kzalloc(&pdev->dev, sizeof(*tcu), GFP_KERNEL);
+	if (!tcu)
+		return -ENOMEM;
+
+	tcu->tx_client.dev = &pdev->dev;
+	tcu->tx_client.tx_block = true;
+	tcu->tx_client.tx_tout = 10000;
+	tcu->rx_client.dev = &pdev->dev;
+	tcu->rx_client.rx_callback = tegra_tcu_receive;
+
+	tcu->tx = mbox_request_channel_byname(&tcu->tx_client, "tx");
+	if (IS_ERR(tcu->tx)) {
+		err = PTR_ERR(tcu->tx);
+		dev_err(&pdev->dev, "failed to get tx mailbox: %d\n", err);
+		return err;
+	}
+
+	tcu->rx = mbox_request_channel_byname(&tcu->rx_client, "rx");
+	if (IS_ERR(tcu->rx)) {
+		err = PTR_ERR(tcu->rx);
+		dev_err(&pdev->dev, "failed to get rx mailbox: %d\n", err);
+		goto free_tx;
+	}
+
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	/* setup the console */
+	strcpy(tcu->console.name, "ttyTCU");
+	tcu->console.device = uart_console_device;
+	tcu->console.flags = CON_PRINTBUFFER | CON_ANYTIME;
+	tcu->console.index = -1;
+	tcu->console.write = tegra_tcu_console_write;
+	tcu->console.setup = tegra_tcu_console_setup;
+	tcu->console.data = &tcu->driver;
+#endif
+
+	/* setup the driver */
+	tcu->driver.owner = THIS_MODULE;
+	tcu->driver.driver_name = "tegra-tcu";
+	tcu->driver.dev_name = "ttyTCU";
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	tcu->driver.cons = &tcu->console;
+#endif
+	tcu->driver.nr = 1;
+
+	err = uart_register_driver(&tcu->driver);
+	if (err) {
+		dev_err(&pdev->dev, "failed to register UART driver: %d\n",
+			err);
+		goto free_rx;
+	}
+
+	/* setup the port */
+	port = &tcu->port;
+	spin_lock_init(&port->lock);
+	port->dev = &pdev->dev;
+	port->type = PORT_TEGRA_TCU;
+	port->ops = &tegra_tcu_uart_ops;
+	port->fifosize = 1;
+	port->iotype = UPIO_MEM;
+	port->flags = UPF_BOOT_AUTOCONF;
+	port->private_data = tcu;
+
+	err = uart_add_one_port(&tcu->driver, port);
+	if (err) {
+		dev_err(&pdev->dev, "failed to add UART port: %d\n", err);
+		goto unregister_uart;
+	}
+
+	platform_set_drvdata(pdev, tcu);
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	register_console(&tcu->console);
+#endif
+
+	return 0;
+
+unregister_uart:
+	uart_unregister_driver(&tcu->driver);
+free_rx:
+	mbox_free_channel(tcu->rx);
+free_tx:
+	mbox_free_channel(tcu->tx);
+
+	return err;
+}
+
+static int tegra_tcu_remove(struct platform_device *pdev)
+{
+	struct tegra_tcu *tcu = platform_get_drvdata(pdev);
+
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	unregister_console(&tcu->console);
+#endif
+	uart_remove_one_port(&tcu->driver, &tcu->port);
+	uart_unregister_driver(&tcu->driver);
+	mbox_free_channel(tcu->rx);
+	mbox_free_channel(tcu->tx);
+
+	return 0;
+}
+
+static const struct of_device_id tegra_tcu_match[] = {
+	{ .compatible = "nvidia,tegra194-tcu" },
+	{ }
+};
+
+static struct platform_driver tegra_tcu_driver = {
+	.driver = {
+		.name = "tegra-tcu",
+		.of_match_table = tegra_tcu_match,
+	},
+	.probe = tegra_tcu_probe,
+	.remove = tegra_tcu_remove,
+};
+module_platform_driver(tegra_tcu_driver);
+
+MODULE_AUTHOR("Mikko Perttunen <mperttunen@nvidia.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("NVIDIA Tegra Combined UART driver");
diff --git a/include/uapi/linux/serial_core.h b/include/uapi/linux/serial_core.h
index dce5f9dae121..69883c32cb98 100644
--- a/include/uapi/linux/serial_core.h
+++ b/include/uapi/linux/serial_core.h
@@ -79,6 +79,9 @@
 /* Nuvoton UART */
 #define PORT_NPCM	40
 
+/* NVIDIA Tegra Combined UART */
+#define PORT_TEGRA_TCU	41
+
 /* Intel EG20 */
 #define PORT_PCH_8LINE	44
 #define PORT_PCH_2LINE	45
-- 
2.19.1

[PATCH v2 07/10] serial: Add Tegra Combined UART driver

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

The Tegra Combined UART (TCU) is a mailbox-based mechanism that allows
multiplexing multiple "virtual UARTs" into a single hardware serial
port. The TCU is the primary serial port on Tegra194 devices.

Add a TCU driver utilizing the mailbox framework, as the used mailboxes
are part of Tegra HSP blocks that are already controlled by the Tegra
HSP mailbox driver.

Based on work by  Mikko Perttunen <mperttunen@nvidia.com>.

Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 drivers/tty/serial/Kconfig       |  22 +++
 drivers/tty/serial/Makefile      |   1 +
 drivers/tty/serial/tegra-tcu.c   | 299 +++++++++++++++++++++++++++++++
 include/uapi/linux/serial_core.h |   3 +
 4 files changed, 325 insertions(+)
 create mode 100644 drivers/tty/serial/tegra-tcu.c

diff --git a/drivers/tty/serial/Kconfig b/drivers/tty/serial/Kconfig
index 32886c304641..785306388aa4 100644
--- a/drivers/tty/serial/Kconfig
+++ b/drivers/tty/serial/Kconfig
@@ -323,6 +323,28 @@ config SERIAL_TEGRA
 	  are enabled). This driver uses the APB DMA to achieve higher baudrate
 	  and better performance.
 
+config SERIAL_TEGRA_TCU
+	tristate "NVIDIA Tegra Combined UART"
+	depends on ARCH_TEGRA && TEGRA_HSP_MBOX
+	select SERIAL_CORE
+	help
+	  Support for the mailbox-based TCU (Tegra Combined UART) serial port.
+	  TCU is a virtual serial port that allows multiplexing multiple data
+	  streams into a single hardware serial port.
+
+config SERIAL_TEGRA_TCU_CONSOLE
+	bool "Support for console on a Tegra TCU serial port"
+	depends on SERIAL_TEGRA_TCU=y
+	select SERIAL_CORE_CONSOLE
+	default y
+	---help---
+	  If you say Y here, it will be possible to use a the Tegra TCU as the
+	  system console (the system console is the device which receives all
+	  kernel messages and warnings and which allows logins in single user
+	  mode).
+
+	  If unsure, say Y.
+
 config SERIAL_MAX3100
 	tristate "MAX3100 support"
 	depends on SPI
diff --git a/drivers/tty/serial/Makefile b/drivers/tty/serial/Makefile
index daac675612df..4ad82231ff8a 100644
--- a/drivers/tty/serial/Makefile
+++ b/drivers/tty/serial/Makefile
@@ -76,6 +76,7 @@ obj-$(CONFIG_SERIAL_LANTIQ)	+= lantiq.o
 obj-$(CONFIG_SERIAL_XILINX_PS_UART) += xilinx_uartps.o
 obj-$(CONFIG_SERIAL_SIRFSOC) += sirfsoc_uart.o
 obj-$(CONFIG_SERIAL_TEGRA) += serial-tegra.o
+obj-$(CONFIG_SERIAL_TEGRA_TCU) += tegra-tcu.o
 obj-$(CONFIG_SERIAL_AR933X)   += ar933x_uart.o
 obj-$(CONFIG_SERIAL_EFM32_UART) += efm32-uart.o
 obj-$(CONFIG_SERIAL_ARC)	+= arc_uart.o
diff --git a/drivers/tty/serial/tegra-tcu.c b/drivers/tty/serial/tegra-tcu.c
new file mode 100644
index 000000000000..1d360cd03b18
--- /dev/null
+++ b/drivers/tty/serial/tegra-tcu.c
@@ -0,0 +1,299 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+ */
+
+#include <linux/console.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/serial.h>
+#include <linux/serial_core.h>
+#include <linux/slab.h>
+#include <linux/tty.h>
+#include <linux/tty_flip.h>
+
+#define TCU_MBOX_BYTE(i, x)			((x) << (i * 8))
+#define TCU_MBOX_BYTE_V(x, i)			(((x) >> (i * 8)) & 0xff)
+#define TCU_MBOX_NUM_BYTES(x)			((x) << 24)
+#define TCU_MBOX_NUM_BYTES_V(x)			(((x) >> 24) & 0x3)
+
+struct tegra_tcu {
+	struct uart_driver driver;
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	struct console console;
+#endif
+	struct uart_port port;
+
+	struct mbox_client tx_client, rx_client;
+	struct mbox_chan *tx, *rx;
+};
+
+static unsigned int tegra_tcu_uart_tx_empty(struct uart_port *port)
+{
+	return TIOCSER_TEMT;
+}
+
+static void tegra_tcu_uart_set_mctrl(struct uart_port *port, unsigned int mctrl)
+{
+}
+
+static unsigned int tegra_tcu_uart_get_mctrl(struct uart_port *port)
+{
+	return 0;
+}
+
+static void tegra_tcu_uart_stop_tx(struct uart_port *port)
+{
+}
+
+static void tegra_tcu_write_one(struct tegra_tcu *tcu, u32 value,
+			       unsigned int count)
+{
+	void *msg;
+
+	value |= TCU_MBOX_NUM_BYTES(count);
+	msg = (void *)(unsigned long)value;
+	mbox_send_message(tcu->tx, msg);
+}
+
+static void tegra_tcu_write(struct tegra_tcu *tcu, const char *s,
+			    unsigned int count)
+{
+	unsigned int written = 0, i = 0;
+	bool insert_nl = false;
+	u32 value = 0;
+
+	while (i < count) {
+		if (insert_nl) {
+			value |= TCU_MBOX_BYTE(written++, '\n');
+			insert_nl = false;
+			i++;
+		} else if (s[i] == '\n') {
+			value |= TCU_MBOX_BYTE(written++, '\r');
+			insert_nl = true;
+		} else {
+			value |= TCU_MBOX_BYTE(written++, s[i++]);
+		}
+
+		if (written == 3) {
+			tegra_tcu_write_one(tcu, value, 3);
+			value = written = 0;
+		}
+	}
+
+	if (written)
+		tegra_tcu_write_one(tcu, value, written);
+}
+
+static void tegra_tcu_uart_start_tx(struct uart_port *port)
+{
+	struct tegra_tcu *tcu = port->private_data;
+	struct circ_buf *xmit = &port->state->xmit;
+	unsigned long count;
+
+	for (;;) {
+		count = CIRC_CNT_TO_END(xmit->head, xmit->tail, UART_XMIT_SIZE);
+		if (!count)
+			break;
+
+		tegra_tcu_write(tcu, &xmit->buf[xmit->tail], count);
+		xmit->tail = (xmit->tail + count) & (UART_XMIT_SIZE - 1);
+	}
+
+	uart_write_wakeup(port);
+}
+
+static void tegra_tcu_uart_stop_rx(struct uart_port *port)
+{
+}
+
+static void tegra_tcu_uart_break_ctl(struct uart_port *port, int ctl)
+{
+}
+
+static int tegra_tcu_uart_startup(struct uart_port *port)
+{
+	return 0;
+}
+
+static void tegra_tcu_uart_shutdown(struct uart_port *port)
+{
+}
+
+static void tegra_tcu_uart_set_termios(struct uart_port *port,
+				       struct ktermios *new,
+				       struct ktermios *old)
+{
+}
+
+static const struct uart_ops tegra_tcu_uart_ops = {
+	.tx_empty = tegra_tcu_uart_tx_empty,
+	.set_mctrl = tegra_tcu_uart_set_mctrl,
+	.get_mctrl = tegra_tcu_uart_get_mctrl,
+	.stop_tx = tegra_tcu_uart_stop_tx,
+	.start_tx = tegra_tcu_uart_start_tx,
+	.stop_rx = tegra_tcu_uart_stop_rx,
+	.break_ctl = tegra_tcu_uart_break_ctl,
+	.startup = tegra_tcu_uart_startup,
+	.shutdown = tegra_tcu_uart_shutdown,
+	.set_termios = tegra_tcu_uart_set_termios,
+};
+
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+static void tegra_tcu_console_write(struct console *cons, const char *s,
+				    unsigned int count)
+{
+	struct tegra_tcu *tcu = container_of(cons, struct tegra_tcu, console);
+
+	tegra_tcu_write(tcu, s, count);
+}
+
+static int tegra_tcu_console_setup(struct console *cons, char *options)
+{
+	return 0;
+}
+#endif
+
+static void tegra_tcu_receive(struct mbox_client *cl, void *msg)
+{
+	struct tegra_tcu *tcu = container_of(cl, struct tegra_tcu, rx_client);
+	struct tty_port *port = &tcu->port.state->port;
+	u32 value = (u32)(unsigned long)msg;
+	unsigned int num_bytes, i;
+
+	num_bytes = TCU_MBOX_NUM_BYTES_V(value);
+
+	for (i = 0; i < num_bytes; i++)
+		tty_insert_flip_char(port, TCU_MBOX_BYTE_V(value, i),
+				     TTY_NORMAL);
+
+	tty_flip_buffer_push(port);
+}
+
+static int tegra_tcu_probe(struct platform_device *pdev)
+{
+	struct uart_port *port;
+	struct tegra_tcu *tcu;
+	int err;
+
+	tcu = devm_kzalloc(&pdev->dev, sizeof(*tcu), GFP_KERNEL);
+	if (!tcu)
+		return -ENOMEM;
+
+	tcu->tx_client.dev = &pdev->dev;
+	tcu->tx_client.tx_block = true;
+	tcu->tx_client.tx_tout = 10000;
+	tcu->rx_client.dev = &pdev->dev;
+	tcu->rx_client.rx_callback = tegra_tcu_receive;
+
+	tcu->tx = mbox_request_channel_byname(&tcu->tx_client, "tx");
+	if (IS_ERR(tcu->tx)) {
+		err = PTR_ERR(tcu->tx);
+		dev_err(&pdev->dev, "failed to get tx mailbox: %d\n", err);
+		return err;
+	}
+
+	tcu->rx = mbox_request_channel_byname(&tcu->rx_client, "rx");
+	if (IS_ERR(tcu->rx)) {
+		err = PTR_ERR(tcu->rx);
+		dev_err(&pdev->dev, "failed to get rx mailbox: %d\n", err);
+		goto free_tx;
+	}
+
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	/* setup the console */
+	strcpy(tcu->console.name, "ttyTCU");
+	tcu->console.device = uart_console_device;
+	tcu->console.flags = CON_PRINTBUFFER | CON_ANYTIME;
+	tcu->console.index = -1;
+	tcu->console.write = tegra_tcu_console_write;
+	tcu->console.setup = tegra_tcu_console_setup;
+	tcu->console.data = &tcu->driver;
+#endif
+
+	/* setup the driver */
+	tcu->driver.owner = THIS_MODULE;
+	tcu->driver.driver_name = "tegra-tcu";
+	tcu->driver.dev_name = "ttyTCU";
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	tcu->driver.cons = &tcu->console;
+#endif
+	tcu->driver.nr = 1;
+
+	err = uart_register_driver(&tcu->driver);
+	if (err) {
+		dev_err(&pdev->dev, "failed to register UART driver: %d\n",
+			err);
+		goto free_rx;
+	}
+
+	/* setup the port */
+	port = &tcu->port;
+	spin_lock_init(&port->lock);
+	port->dev = &pdev->dev;
+	port->type = PORT_TEGRA_TCU;
+	port->ops = &tegra_tcu_uart_ops;
+	port->fifosize = 1;
+	port->iotype = UPIO_MEM;
+	port->flags = UPF_BOOT_AUTOCONF;
+	port->private_data = tcu;
+
+	err = uart_add_one_port(&tcu->driver, port);
+	if (err) {
+		dev_err(&pdev->dev, "failed to add UART port: %d\n", err);
+		goto unregister_uart;
+	}
+
+	platform_set_drvdata(pdev, tcu);
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	register_console(&tcu->console);
+#endif
+
+	return 0;
+
+unregister_uart:
+	uart_unregister_driver(&tcu->driver);
+free_rx:
+	mbox_free_channel(tcu->rx);
+free_tx:
+	mbox_free_channel(tcu->tx);
+
+	return err;
+}
+
+static int tegra_tcu_remove(struct platform_device *pdev)
+{
+	struct tegra_tcu *tcu = platform_get_drvdata(pdev);
+
+#if IS_ENABLED(CONFIG_SERIAL_TEGRA_TCU_CONSOLE)
+	unregister_console(&tcu->console);
+#endif
+	uart_remove_one_port(&tcu->driver, &tcu->port);
+	uart_unregister_driver(&tcu->driver);
+	mbox_free_channel(tcu->rx);
+	mbox_free_channel(tcu->tx);
+
+	return 0;
+}
+
+static const struct of_device_id tegra_tcu_match[] = {
+	{ .compatible = "nvidia,tegra194-tcu" },
+	{ }
+};
+
+static struct platform_driver tegra_tcu_driver = {
+	.driver = {
+		.name = "tegra-tcu",
+		.of_match_table = tegra_tcu_match,
+	},
+	.probe = tegra_tcu_probe,
+	.remove = tegra_tcu_remove,
+};
+module_platform_driver(tegra_tcu_driver);
+
+MODULE_AUTHOR("Mikko Perttunen <mperttunen@nvidia.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("NVIDIA Tegra Combined UART driver");
diff --git a/include/uapi/linux/serial_core.h b/include/uapi/linux/serial_core.h
index dce5f9dae121..69883c32cb98 100644
--- a/include/uapi/linux/serial_core.h
+++ b/include/uapi/linux/serial_core.h
@@ -79,6 +79,9 @@
 /* Nuvoton UART */
 #define PORT_NPCM	40
 
+/* NVIDIA Tegra Combined UART */
+#define PORT_TEGRA_TCU	41
+
 /* Intel EG20 */
 #define PORT_PCH_8LINE	44
 #define PORT_PCH_2LINE	45
-- 
2.19.1

[PATCH v2 08/10] arm64: tegra: Add nodes for TCU on Tegra194

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

Add nodes required for communication through the Tegra Combined UART.
This includes the AON HSP instance, addition of shared interrupts
for the TOP0 HSP instance, and finally the TCU node itself. Also
mark the HSP instances as compatible to tegra194-hsp, as the hardware
is not identical but is compatible to tegra186-hsp.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Acked-by: Jon Hunter <jonathanh@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
Changes in v2:
- encode direction of mailboxes in device tree mailbox specifier

 arch/arm64/boot/dts/nvidia/tegra194.dtsi | 38 ++++++++++++++++++++++--
 1 file changed, 35 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/boot/dts/nvidia/tegra194.dtsi b/arch/arm64/boot/dts/nvidia/tegra194.dtsi
index c2091bb16546..4451532a2b4c 100644
--- a/arch/arm64/boot/dts/nvidia/tegra194.dtsi
+++ b/arch/arm64/boot/dts/nvidia/tegra194.dtsi
@@ -340,10 +340,35 @@
 		};
 
 		hsp_top0: hsp@3c00000 {
-			compatible = "nvidia,tegra186-hsp";
+			compatible = "nvidia,tegra194-hsp", "nvidia,tegra186-hsp";
 			reg = <0x03c00000 0xa0000>;
-			interrupts = <GIC_SPI 176 IRQ_TYPE_LEVEL_HIGH>;
-			interrupt-names = "doorbell";
+			interrupts = <GIC_SPI 176 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 124 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 125 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 126 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH>;
+			interrupt-names = "doorbell", "shared0", "shared1", "shared2",
+			                  "shared3", "shared4", "shared5", "shared6",
+			                  "shared7";
+			#mbox-cells = <2>;
+		};
+
+		hsp_aon: hsp@c150000 {
+			compatible = "nvidia,tegra194-hsp", "nvidia,tegra186-hsp";
+			reg = <0x0c150000 0xa0000>;
+			interrupts = <GIC_SPI 133 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 135 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 136 IRQ_TYPE_LEVEL_HIGH>;
+			/*
+			 * Shared interrupt 0 is routed only to AON/SPE, so
+			 * we only have 4 shared interrupts for the CCPLEX.
+			 */
+			interrupt-names = "shared1", "shared2", "shared3", "shared4";
 			#mbox-cells = <2>;
 		};
 
@@ -531,6 +556,13 @@
 		method = "smc";
 	};
 
+	tcu: tcu {
+		compatible = "nvidia,tegra194-tcu";
+		mboxes = <&hsp_top0 TEGRA_HSP_MBOX_TYPE_SM TEGRA_HSP_SM_RX(0)>,
+		         <&hsp_aon TEGRA_HSP_MBOX_TYPE_SM TEGRA_HSP_SM_TX(1)>;
+		mbox-names = "rx", "tx";
+	};
+
 	timer {
 		compatible = "arm,armv8-timer";
 		interrupts = <GIC_PPI 13
-- 
2.19.1

[PATCH v2 08/10] arm64: tegra: Add nodes for TCU on Tegra194

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

Add nodes required for communication through the Tegra Combined UART.
This includes the AON HSP instance, addition of shared interrupts
for the TOP0 HSP instance, and finally the TCU node itself. Also
mark the HSP instances as compatible to tegra194-hsp, as the hardware
is not identical but is compatible to tegra186-hsp.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Acked-by: Jon Hunter <jonathanh@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
Changes in v2:
- encode direction of mailboxes in device tree mailbox specifier

 arch/arm64/boot/dts/nvidia/tegra194.dtsi | 38 ++++++++++++++++++++++--
 1 file changed, 35 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/boot/dts/nvidia/tegra194.dtsi b/arch/arm64/boot/dts/nvidia/tegra194.dtsi
index c2091bb16546..4451532a2b4c 100644
--- a/arch/arm64/boot/dts/nvidia/tegra194.dtsi
+++ b/arch/arm64/boot/dts/nvidia/tegra194.dtsi
@@ -340,10 +340,35 @@
 		};
 
 		hsp_top0: hsp at 3c00000 {
-			compatible = "nvidia,tegra186-hsp";
+			compatible = "nvidia,tegra194-hsp", "nvidia,tegra186-hsp";
 			reg = <0x03c00000 0xa0000>;
-			interrupts = <GIC_SPI 176 IRQ_TYPE_LEVEL_HIGH>;
-			interrupt-names = "doorbell";
+			interrupts = <GIC_SPI 176 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 124 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 125 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 126 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH>;
+			interrupt-names = "doorbell", "shared0", "shared1", "shared2",
+			                  "shared3", "shared4", "shared5", "shared6",
+			                  "shared7";
+			#mbox-cells = <2>;
+		};
+
+		hsp_aon: hsp at c150000 {
+			compatible = "nvidia,tegra194-hsp", "nvidia,tegra186-hsp";
+			reg = <0x0c150000 0xa0000>;
+			interrupts = <GIC_SPI 133 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 135 IRQ_TYPE_LEVEL_HIGH>,
+			             <GIC_SPI 136 IRQ_TYPE_LEVEL_HIGH>;
+			/*
+			 * Shared interrupt 0 is routed only to AON/SPE, so
+			 * we only have 4 shared interrupts for the CCPLEX.
+			 */
+			interrupt-names = "shared1", "shared2", "shared3", "shared4";
 			#mbox-cells = <2>;
 		};
 
@@ -531,6 +556,13 @@
 		method = "smc";
 	};
 
+	tcu: tcu {
+		compatible = "nvidia,tegra194-tcu";
+		mboxes = <&hsp_top0 TEGRA_HSP_MBOX_TYPE_SM TEGRA_HSP_SM_RX(0)>,
+		         <&hsp_aon TEGRA_HSP_MBOX_TYPE_SM TEGRA_HSP_SM_TX(1)>;
+		mbox-names = "rx", "tx";
+	};
+
 	timer {
 		compatible = "arm,armv8-timer";
 		interrupts = <GIC_PPI 13
-- 
2.19.1

[PATCH v2 09/10] arm64: tegra: Mark TCU as primary serial port on Tegra194 P2888

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

The Tegra Combined UART is the proper primary serial port on P2888,
so use it.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Acked-by: Jon Hunter <jonathanh@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi b/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi
index 57d3f00464ce..fcbe2a88f8db 100644
--- a/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi
+++ b/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi
@@ -10,7 +10,7 @@
 	aliases {
 		sdhci0 = "/cbb/sdhci@3460000";
 		sdhci1 = "/cbb/sdhci@3400000";
-		serial0 = &uartb;
+		serial0 = &tcu;
 		i2c0 = "/bpmp/i2c";
 		i2c1 = "/cbb/i2c@3160000";
 		i2c2 = "/cbb/i2c@c240000";
-- 
2.19.1

[PATCH v2 09/10] arm64: tegra: Mark TCU as primary serial port on Tegra194 P2888

[Thierry Reding]

From: Mikko Perttunen <mperttunen@nvidia.com>

The Tegra Combined UART is the proper primary serial port on P2888,
so use it.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
Acked-by: Jon Hunter <jonathanh@nvidia.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi b/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi
index 57d3f00464ce..fcbe2a88f8db 100644
--- a/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi
+++ b/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi
@@ -10,7 +10,7 @@
 	aliases {
 		sdhci0 = "/cbb/sdhci at 3460000";
 		sdhci1 = "/cbb/sdhci at 3400000";
-		serial0 = &uartb;
+		serial0 = &tcu;
 		i2c0 = "/bpmp/i2c";
 		i2c1 = "/cbb/i2c at 3160000";
 		i2c2 = "/cbb/i2c at c240000";
-- 
2.19.1

[PATCH v2 10/10] arm64: defconfig: Enable Tegra TCU

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

The Tegra Combined UART is used on some Tegra194 devices as a way of
multiplexing output from multiple producers onto a single physical UART.
Enable this by default so that it can be used as the default console to
write kernel messages to.

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 arch/arm64/configs/defconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index c0a5275210c7..fdb9d60905ac 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -300,6 +300,7 @@ CONFIG_SERIAL_MESON_CONSOLE=y
 CONFIG_SERIAL_SAMSUNG=y
 CONFIG_SERIAL_SAMSUNG_CONSOLE=y
 CONFIG_SERIAL_TEGRA=y
+CONFIG_SERIAL_TEGRA_TCU=y
 CONFIG_SERIAL_SH_SCI=y
 CONFIG_SERIAL_MSM=y
 CONFIG_SERIAL_MSM_CONSOLE=y
-- 
2.19.1

[PATCH v2 10/10] arm64: defconfig: Enable Tegra TCU

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

The Tegra Combined UART is used on some Tegra194 devices as a way of
multiplexing output from multiple producers onto a single physical UART.
Enable this by default so that it can be used as the default console to
write kernel messages to.

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
 arch/arm64/configs/defconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index c0a5275210c7..fdb9d60905ac 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -300,6 +300,7 @@ CONFIG_SERIAL_MESON_CONSOLE=y
 CONFIG_SERIAL_SAMSUNG=y
 CONFIG_SERIAL_SAMSUNG_CONSOLE=y
 CONFIG_SERIAL_TEGRA=y
+CONFIG_SERIAL_TEGRA_TCU=y
 CONFIG_SERIAL_SH_SCI=y
 CONFIG_SERIAL_MSM=y
 CONFIG_SERIAL_MSM_CONSOLE=y
-- 
2.19.1

Re: [PATCH v2 06/10] dt-bindings: serial: Add bindings for nvidia,tegra194-tcu

[Jon Hunter]

On 12/11/2018 15:18, Thierry Reding wrote:
> From: Mikko Perttunen <mperttunen@nvidia.com>
> 
> Add bindings for the Tegra Combined UART device used to talk to the
> UART console on Tegra194 systems.
> 
> Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
> Reviewed-by: Rob Herring <robh@kernel.org>
> Acked-by: Jon Hunter <jonathanh@nvidia.com>
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
>  .../bindings/serial/nvidia,tegra194-tcu.txt   | 35 +++++++++++++++++++
>  1 file changed, 35 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> 
> diff --git a/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> new file mode 100644
> index 000000000000..085a8591accd
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> @@ -0,0 +1,35 @@
> +NVIDIA Tegra Combined UART (TCU)
> +
> +The TCU is a system for sharing a hardware UART instance among multiple
> +systems within the Tegra SoC. It is implemented through a mailbox-
> +based protocol where each "virtual UART" has a pair of mailboxes, one
> +for transmitting and one for receiving, that is used to communicate
> +with the hardware implementing the TCU.
> +
> +Required properties:
> +- name : Should be tcu
> +- compatible
> +    Array of strings
> +    One of:
> +    - "nvidia,tegra194-tcu"
> +- mbox-names:
> +    "rx" - Mailbox for receiving data from hardware UART
> +    "tx" - Mailbox for transmitting data to hardware UART
> +- mboxes: Mailboxes corresponding to the mbox-names.

Looks like there is some trailing white-space in the above line and git
warns after applying this patch.

Cheers
Jon

-- 
nvpublic

[PATCH v2 06/10] dt-bindings: serial: Add bindings for nvidia,tegra194-tcu

[Jon Hunter]

On 12/11/2018 15:18, Thierry Reding wrote:
> From: Mikko Perttunen <mperttunen@nvidia.com>
> 
> Add bindings for the Tegra Combined UART device used to talk to the
> UART console on Tegra194 systems.
> 
> Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
> Reviewed-by: Rob Herring <robh@kernel.org>
> Acked-by: Jon Hunter <jonathanh@nvidia.com>
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
>  .../bindings/serial/nvidia,tegra194-tcu.txt   | 35 +++++++++++++++++++
>  1 file changed, 35 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> 
> diff --git a/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> new file mode 100644
> index 000000000000..085a8591accd
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> @@ -0,0 +1,35 @@
> +NVIDIA Tegra Combined UART (TCU)
> +
> +The TCU is a system for sharing a hardware UART instance among multiple
> +systems within the Tegra SoC. It is implemented through a mailbox-
> +based protocol where each "virtual UART" has a pair of mailboxes, one
> +for transmitting and one for receiving, that is used to communicate
> +with the hardware implementing the TCU.
> +
> +Required properties:
> +- name : Should be tcu
> +- compatible
> +    Array of strings
> +    One of:
> +    - "nvidia,tegra194-tcu"
> +- mbox-names:
> +    "rx" - Mailbox for receiving data from hardware UART
> +    "tx" - Mailbox for transmitting data to hardware UART
> +- mboxes: Mailboxes corresponding to the mbox-names.

Looks like there is some trailing white-space in the above line and git
warns after applying this patch.

Cheers
Jon

-- 
nvpublic

Re: [PATCH v2 06/10] dt-bindings: serial: Add bindings for nvidia,tegra194-tcu

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 2155 bytes --]

On Tue, Nov 13, 2018 at 09:39:08AM +0000, Jon Hunter wrote:
> 
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Mikko Perttunen <mperttunen@nvidia.com>
> > 
> > Add bindings for the Tegra Combined UART device used to talk to the
> > UART console on Tegra194 systems.
> > 
> > Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
> > Reviewed-by: Rob Herring <robh@kernel.org>
> > Acked-by: Jon Hunter <jonathanh@nvidia.com>
> > Acked-by: Thierry Reding <treding@nvidia.com>
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  .../bindings/serial/nvidia,tegra194-tcu.txt   | 35 +++++++++++++++++++
> >  1 file changed, 35 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> > 
> > diff --git a/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> > new file mode 100644
> > index 000000000000..085a8591accd
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> > @@ -0,0 +1,35 @@
> > +NVIDIA Tegra Combined UART (TCU)
> > +
> > +The TCU is a system for sharing a hardware UART instance among multiple
> > +systems within the Tegra SoC. It is implemented through a mailbox-
> > +based protocol where each "virtual UART" has a pair of mailboxes, one
> > +for transmitting and one for receiving, that is used to communicate
> > +with the hardware implementing the TCU.
> > +
> > +Required properties:
> > +- name : Should be tcu
> > +- compatible
> > +    Array of strings
> > +    One of:
> > +    - "nvidia,tegra194-tcu"
> > +- mbox-names:
> > +    "rx" - Mailbox for receiving data from hardware UART
> > +    "tx" - Mailbox for transmitting data to hardware UART
> > +- mboxes: Mailboxes corresponding to the mbox-names.
> 
> Looks like there is some trailing white-space in the above line and git
> warns after applying this patch.

I'm not seeing it. I remember seeing that trailing space in an earlier
version of this patch and fixing it up. Did you perhaps end up applying
the wrong version?

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 06/10] dt-bindings: serial: Add bindings for nvidia,tegra194-tcu

[Thierry Reding]

On Tue, Nov 13, 2018 at 09:39:08AM +0000, Jon Hunter wrote:
> 
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Mikko Perttunen <mperttunen@nvidia.com>
> > 
> > Add bindings for the Tegra Combined UART device used to talk to the
> > UART console on Tegra194 systems.
> > 
> > Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
> > Reviewed-by: Rob Herring <robh@kernel.org>
> > Acked-by: Jon Hunter <jonathanh@nvidia.com>
> > Acked-by: Thierry Reding <treding@nvidia.com>
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  .../bindings/serial/nvidia,tegra194-tcu.txt   | 35 +++++++++++++++++++
> >  1 file changed, 35 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> > 
> > diff --git a/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> > new file mode 100644
> > index 000000000000..085a8591accd
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
> > @@ -0,0 +1,35 @@
> > +NVIDIA Tegra Combined UART (TCU)
> > +
> > +The TCU is a system for sharing a hardware UART instance among multiple
> > +systems within the Tegra SoC. It is implemented through a mailbox-
> > +based protocol where each "virtual UART" has a pair of mailboxes, one
> > +for transmitting and one for receiving, that is used to communicate
> > +with the hardware implementing the TCU.
> > +
> > +Required properties:
> > +- name : Should be tcu
> > +- compatible
> > +    Array of strings
> > +    One of:
> > +    - "nvidia,tegra194-tcu"
> > +- mbox-names:
> > +    "rx" - Mailbox for receiving data from hardware UART
> > +    "tx" - Mailbox for transmitting data to hardware UART
> > +- mboxes: Mailboxes corresponding to the mbox-names.
> 
> Looks like there is some trailing white-space in the above line and git
> warns after applying this patch.

I'm not seeing it. I remember seeing that trailing space in an earlier
version of this patch and fixing it up. Did you perhaps end up applying
the wrong version?

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181113/e776e2f8/attachment.sig>

Re: [PATCH v2 06/10] dt-bindings: serial: Add bindings for nvidia,tegra194-tcu

[Jon Hunter]

On 13/11/2018 10:03, Thierry Reding wrote:
> On Tue, Nov 13, 2018 at 09:39:08AM +0000, Jon Hunter wrote:
>>
>> On 12/11/2018 15:18, Thierry Reding wrote:
>>> From: Mikko Perttunen <mperttunen@nvidia.com>
>>>
>>> Add bindings for the Tegra Combined UART device used to talk to the
>>> UART console on Tegra194 systems.
>>>
>>> Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
>>> Reviewed-by: Rob Herring <robh@kernel.org>
>>> Acked-by: Jon Hunter <jonathanh@nvidia.com>
>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>> Signed-off-by: Thierry Reding <treding@nvidia.com>
>>> ---
>>>  .../bindings/serial/nvidia,tegra194-tcu.txt   | 35 +++++++++++++++++++
>>>  1 file changed, 35 insertions(+)
>>>  create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
>>>
>>> diff --git a/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
>>> new file mode 100644
>>> index 000000000000..085a8591accd
>>> --- /dev/null
>>> +++ b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
>>> @@ -0,0 +1,35 @@
>>> +NVIDIA Tegra Combined UART (TCU)
>>> +
>>> +The TCU is a system for sharing a hardware UART instance among multiple
>>> +systems within the Tegra SoC. It is implemented through a mailbox-
>>> +based protocol where each "virtual UART" has a pair of mailboxes, one
>>> +for transmitting and one for receiving, that is used to communicate
>>> +with the hardware implementing the TCU.
>>> +
>>> +Required properties:
>>> +- name : Should be tcu
>>> +- compatible
>>> +    Array of strings
>>> +    One of:
>>> +    - "nvidia,tegra194-tcu"
>>> +- mbox-names:
>>> +    "rx" - Mailbox for receiving data from hardware UART
>>> +    "tx" - Mailbox for transmitting data to hardware UART
>>> +- mboxes: Mailboxes corresponding to the mbox-names.
>>
>> Looks like there is some trailing white-space in the above line and git
>> warns after applying this patch.
> 
> I'm not seeing it. I remember seeing that trailing space in an earlier
> version of this patch and fixing it up. Did you perhaps end up applying
> the wrong version?

Yes you are right. This version does not have it. Seems that thunderbird
does not correctly overwrite the previous mbox but once I deleted and
saved again, I no longer see this.

Sorry for the noise. Jon

-- 
nvpublic

[PATCH v2 06/10] dt-bindings: serial: Add bindings for nvidia,tegra194-tcu

[Jon Hunter]

On 13/11/2018 10:03, Thierry Reding wrote:
> On Tue, Nov 13, 2018 at 09:39:08AM +0000, Jon Hunter wrote:
>>
>> On 12/11/2018 15:18, Thierry Reding wrote:
>>> From: Mikko Perttunen <mperttunen@nvidia.com>
>>>
>>> Add bindings for the Tegra Combined UART device used to talk to the
>>> UART console on Tegra194 systems.
>>>
>>> Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
>>> Reviewed-by: Rob Herring <robh@kernel.org>
>>> Acked-by: Jon Hunter <jonathanh@nvidia.com>
>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>> Signed-off-by: Thierry Reding <treding@nvidia.com>
>>> ---
>>>  .../bindings/serial/nvidia,tegra194-tcu.txt   | 35 +++++++++++++++++++
>>>  1 file changed, 35 insertions(+)
>>>  create mode 100644 Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
>>>
>>> diff --git a/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
>>> new file mode 100644
>>> index 000000000000..085a8591accd
>>> --- /dev/null
>>> +++ b/Documentation/devicetree/bindings/serial/nvidia,tegra194-tcu.txt
>>> @@ -0,0 +1,35 @@
>>> +NVIDIA Tegra Combined UART (TCU)
>>> +
>>> +The TCU is a system for sharing a hardware UART instance among multiple
>>> +systems within the Tegra SoC. It is implemented through a mailbox-
>>> +based protocol where each "virtual UART" has a pair of mailboxes, one
>>> +for transmitting and one for receiving, that is used to communicate
>>> +with the hardware implementing the TCU.
>>> +
>>> +Required properties:
>>> +- name : Should be tcu
>>> +- compatible
>>> +    Array of strings
>>> +    One of:
>>> +    - "nvidia,tegra194-tcu"
>>> +- mbox-names:
>>> +    "rx" - Mailbox for receiving data from hardware UART
>>> +    "tx" - Mailbox for transmitting data to hardware UART
>>> +- mboxes: Mailboxes corresponding to the mbox-names.
>>
>> Looks like there is some trailing white-space in the above line and git
>> warns after applying this patch.
> 
> I'm not seeing it. I remember seeing that trailing space in an earlier
> version of this patch and fixing it up. Did you perhaps end up applying
> the wrong version?

Yes you are right. This version does not have it. Seems that thunderbird
does not correctly overwrite the previous mbox but once I deleted and
saved again, I no longer see this.

Sorry for the noise. Jon

-- 
nvpublic

Re: [PATCH v2 04/10] mailbox: tegra-hsp: Add support for shared mailboxes

[Jon Hunter]

On 12/11/2018 15:18, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> The Tegra HSP block supports 'shared mailboxes' that are simple 32-bit
> registers consisting of a FULL bit in MSB position and 31 bits of data.
> The hardware can be configured to trigger interrupts when a mailbox
> is empty or full. Add support for these shared mailboxes to the HSP
> driver.
> 
> The initial use for the mailboxes is the Tegra Combined UART. For this
> purpose, we use interrupts to receive data, and spinning to wait for
> the transmit mailbox to be emptied to minimize unnecessary overhead.
> 
> Based on work by Mikko Perttunen <mperttunen@nvidia.com>.
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
> Changes in v2:
> - do not write per-mailbox interrupt enable registers on Tegra186
> - merge handlers for empty and full interrupts
> - track direction of shared mailboxes
> 
>  drivers/mailbox/tegra-hsp.c | 498 +++++++++++++++++++++++++++++++-----
>  1 file changed, 437 insertions(+), 61 deletions(-)
> 
> diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
> index 0cde356c11ab..0100a974149b 100644
> --- a/drivers/mailbox/tegra-hsp.c
> +++ b/drivers/mailbox/tegra-hsp.c
> @@ -1,5 +1,5 @@
>  /*
> - * Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.
> + * Copyright (c) 2016-2018, NVIDIA CORPORATION.  All rights reserved.
>   *
>   * This program is free software; you can redistribute it and/or modify it
>   * under the terms and conditions of the GNU General Public License,
> @@ -11,6 +11,7 @@
>   * more details.
>   */
>  
> +#include <linux/delay.h>
>  #include <linux/interrupt.h>
>  #include <linux/io.h>
>  #include <linux/mailbox_controller.h>
> @@ -21,6 +22,17 @@
>  
>  #include <dt-bindings/mailbox/tegra186-hsp.h>
>  
> +#include "mailbox.h"
> +
> +#define HSP_INT_IE(x)		(0x100 + ((x) * 4))
> +#define HSP_INT_IV		0x300
> +#define HSP_INT_IR		0x304
> +
> +#define HSP_INT_EMPTY_SHIFT	0
> +#define HSP_INT_EMPTY_MASK	0xff
> +#define HSP_INT_FULL_SHIFT	8
> +#define HSP_INT_FULL_MASK	0xff
> +
>  #define HSP_INT_DIMENSIONING	0x380
>  #define HSP_nSM_SHIFT		0
>  #define HSP_nSS_SHIFT		4
> @@ -34,6 +46,11 @@
>  #define HSP_DB_RAW	0x8
>  #define HSP_DB_PENDING	0xc
>  
> +#define HSP_SM_SHRD_MBOX	0x0
> +#define HSP_SM_SHRD_MBOX_FULL	BIT(31)
> +#define HSP_SM_SHRD_MBOX_FULL_INT_IE	0x04
> +#define HSP_SM_SHRD_MBOX_EMPTY_INT_IE	0x08
> +
>  #define HSP_DB_CCPLEX		1
>  #define HSP_DB_BPMP		3
>  #define HSP_DB_MAX		7
> @@ -55,6 +72,12 @@ struct tegra_hsp_doorbell {
>  	unsigned int index;
>  };
>  
> +struct tegra_hsp_mailbox {
> +	struct tegra_hsp_channel channel;
> +	unsigned int index;
> +	bool producer;
> +};
> +
>  struct tegra_hsp_db_map {
>  	const char *name;
>  	unsigned int master;
> @@ -63,13 +86,18 @@ struct tegra_hsp_db_map {
>  
>  struct tegra_hsp_soc {
>  	const struct tegra_hsp_db_map *map;
> +	bool has_per_mb_ie;
>  };
>  
>  struct tegra_hsp {
> +	struct device *dev;
>  	const struct tegra_hsp_soc *soc;
> -	struct mbox_controller mbox;
> +	struct mbox_controller mbox_db;
> +	struct mbox_controller mbox_sm;
>  	void __iomem *regs;
> -	unsigned int irq;
> +	unsigned int doorbell_irq;
> +	unsigned int *shared_irqs;
> +	unsigned int shared_irq;
>  	unsigned int num_sm;
>  	unsigned int num_as;
>  	unsigned int num_ss;
> @@ -78,13 +106,10 @@ struct tegra_hsp {
>  	spinlock_t lock;
>  
>  	struct list_head doorbells;
> -};
> +	struct tegra_hsp_mailbox *mailboxes;
>  
> -static inline struct tegra_hsp *
> -to_tegra_hsp(struct mbox_controller *mbox)
> -{
> -	return container_of(mbox, struct tegra_hsp, mbox);
> -}
> +	unsigned long mask;
> +};
>  
>  static inline u32 tegra_hsp_readl(struct tegra_hsp *hsp, unsigned int offset)
>  {
> @@ -158,7 +183,7 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
>  
>  	spin_lock(&hsp->lock);
>  
> -	for_each_set_bit(master, &value, hsp->mbox.num_chans) {
> +	for_each_set_bit(master, &value, hsp->mbox_db.num_chans) {
>  		struct tegra_hsp_doorbell *db;
>  
>  		db = __tegra_hsp_doorbell_get(hsp, master);
> @@ -182,6 +207,71 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
>  	return IRQ_HANDLED;
>  }
>  
> +static irqreturn_t tegra_hsp_shared_irq(int irq, void *data)
> +{
> +	struct tegra_hsp *hsp = data;
> +	unsigned long bit, mask;
> +	u32 status, value;
> +	void *msg;
> +
> +	status = tegra_hsp_readl(hsp, HSP_INT_IR) & hsp->mask;
> +
> +	/* process EMPTY interrupts first */
> +	mask = (status >> HSP_INT_EMPTY_SHIFT) & HSP_INT_EMPTY_MASK;
> +
> +	for_each_set_bit(bit, &mask, hsp->num_sm) {
> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
> +
> +		if (mb->producer) {
> +			/*
> +			 * Disable EMPTY interrupts until data is sent with
> +			 * the next message. These interrupts are level-
> +			 * triggered, so if we kept them enabled they would
> +			 * constantly trigger until we next write data into
> +			 * the message.
> +			 */
> +			spin_lock(&hsp->lock);
> +
> +			hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> +			tegra_hsp_writel(hsp, hsp->mask,
> +					 HSP_INT_IE(hsp->shared_irq));
> +
> +			spin_unlock(&hsp->lock);
> +
> +			mbox_chan_txdone(mb->channel.chan, 0);
> +		}
> +	}
> +
> +	/* process FULL interrupts */
> +	mask = (status >> HSP_INT_FULL_SHIFT) & HSP_INT_FULL_MASK;
> +
> +	for_each_set_bit(bit, &mask, hsp->num_sm) {
> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
> +
> +		if (!mb->producer) {
> +			value = tegra_hsp_channel_readl(&mb->channel,
> +							HSP_SM_SHRD_MBOX);
> +			value &= ~HSP_SM_SHRD_MBOX_FULL;
> +			msg = (void *)(unsigned long)value;
> +			mbox_chan_received_data(mb->channel.chan, msg);
> +
> +			/*
> +			 * Need to clear all bits here since some producers,
> +			 * such as TCU, depend on fields in the register
> +			 * getting cleared by the consumer.
> +			 *
> +			 * The mailbox API doesn't give the consumers a way
> +			 * of doing that explicitly, so we have to make sure
> +			 * we cover all possible cases.
> +			 */
> +			tegra_hsp_channel_writel(&mb->channel, 0x0,
> +						 HSP_SM_SHRD_MBOX);
> +		}
> +	}
> +
> +	return IRQ_HANDLED;
> +}
> +
>  static struct tegra_hsp_channel *
>  tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
>  			  unsigned int master, unsigned int index)
> @@ -194,7 +284,7 @@ tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
>  	if (!db)
>  		return ERR_PTR(-ENOMEM);
>  
> -	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) << 16;
> +	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) * SZ_64K;
>  	offset += index * 0x100;
>  
>  	db->channel.regs = hsp->regs + offset;
> @@ -235,8 +325,8 @@ static int tegra_hsp_doorbell_startup(struct mbox_chan *chan)
>  	unsigned long flags;
>  	u32 value;
>  
> -	if (db->master >= hsp->mbox.num_chans) {
> -		dev_err(hsp->mbox.dev,
> +	if (db->master >= chan->mbox->num_chans) {
> +		dev_err(chan->mbox->dev,
>  			"invalid master ID %u for HSP channel\n",
>  			db->master);
>  		return -EINVAL;
> @@ -281,46 +371,168 @@ static void tegra_hsp_doorbell_shutdown(struct mbox_chan *chan)
>  	spin_unlock_irqrestore(&hsp->lock, flags);
>  }
>  
> -static const struct mbox_chan_ops tegra_hsp_doorbell_ops = {
> +static const struct mbox_chan_ops tegra_hsp_db_ops = {
>  	.send_data = tegra_hsp_doorbell_send_data,
>  	.startup = tegra_hsp_doorbell_startup,
>  	.shutdown = tegra_hsp_doorbell_shutdown,
>  };
>  
> -static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
> +static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
> +{
> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> +	struct tegra_hsp *hsp = mb->channel.hsp;
> +	unsigned long flags;
> +	u32 value;
> +
> +	WARN_ON(!mb->producer);

Should we return here?

> +
> +	/* copy data and mark mailbox full */
> +	value = (u32)(unsigned long)data;
> +	value |= HSP_SM_SHRD_MBOX_FULL;
> +
> +	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
> +
> +	if (!irqs_disabled()) {
> +		/* enable EMPTY interrupt for the shared mailbox */
> +		spin_lock_irqsave(&hsp->lock, flags);
> +
> +		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> +		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> +
> +		spin_unlock_irqrestore(&hsp->lock, flags);
> +	}
> +
> +	return 0;
> +}
> +
> +static int tegra_hsp_mailbox_flush(struct mbox_chan *chan,
> +				   unsigned long timeout)
> +{
> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> +	struct tegra_hsp_channel *ch = &mb->channel;
> +	u32 value;
> +
> +	timeout = jiffies + msecs_to_jiffies(timeout);
> +
> +	while (time_before(jiffies, timeout)) {
> +		value = tegra_hsp_channel_readl(ch, HSP_SM_SHRD_MBOX);
> +		if ((value & HSP_SM_SHRD_MBOX_FULL) == 0) {
> +			mbox_chan_txdone(chan, 0);
> +			return 0;
> +		}
> +
> +		udelay(1);
> +	}
> +
> +	return -ETIME;
> +}
> +
> +static int tegra_hsp_mailbox_startup(struct mbox_chan *chan)
> +{
> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> +	struct tegra_hsp_channel *ch = &mb->channel;
> +	struct tegra_hsp *hsp = mb->channel.hsp;
> +	unsigned long flags;
> +
> +	chan->txdone_method = TXDONE_BY_IRQ;
> +
> +	/*
> +	 * Shared mailboxes start out as consumers by default. FULL and EMPTY
> +	 * interrupts are coalesced at the same shared interrupt.
> +	 *
> +	 * Keep EMPTY interrupts disabled at startup and only enable them when
> +	 * the mailbox is actually full. This is required because the FULL and
> +	 * EMPTY interrupts are level-triggered, so keeping EMPTY interrupts
> +	 * enabled all the time would cause an interrupt storm while mailboxes
> +	 * are idle.
> +	 */
> +
> +	spin_lock_irqsave(&hsp->lock, flags);
> +
> +	if (mb->producer)
> +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> +	else
> +		hsp->mask |= BIT(HSP_INT_FULL_SHIFT + mb->index);
> +
> +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> +
> +	spin_unlock_irqrestore(&hsp->lock, flags);
> +
> +	if (hsp->soc->has_per_mb_ie) {
> +		if (mb->producer)
> +			tegra_hsp_channel_writel(ch, 0x0,
> +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
> +		else
> +			tegra_hsp_channel_writel(ch, 0x1,
> +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
> +	}
> +
> +	return 0;
> +}
> +
> +static void tegra_hsp_mailbox_shutdown(struct mbox_chan *chan)
> +{
> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> +	struct tegra_hsp_channel *ch = &mb->channel;
> +	struct tegra_hsp *hsp = mb->channel.hsp;
> +	unsigned long flags;
> +
> +	if (hsp->soc->has_per_mb_ie) {
> +		if (mb->producer)
> +			tegra_hsp_channel_writel(ch, 0x0,
> +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
> +		else
> +			tegra_hsp_channel_writel(ch, 0x0,
> +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
> +	}
> +
> +	spin_lock_irqsave(&hsp->lock, flags);
> +
> +	if (mb->producer)
> +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> +	else
> +		hsp->mask &= ~BIT(HSP_INT_FULL_SHIFT + mb->index);
> +
> +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> +
> +	spin_unlock_irqrestore(&hsp->lock, flags);
> +}
> +
> +static const struct mbox_chan_ops tegra_hsp_sm_ops = {
> +	.send_data = tegra_hsp_mailbox_send_data,
> +	.flush = tegra_hsp_mailbox_flush,
> +	.startup = tegra_hsp_mailbox_startup,
> +	.shutdown = tegra_hsp_mailbox_shutdown,
> +};
> +
> +static struct mbox_chan *tegra_hsp_db_xlate(struct mbox_controller *mbox,
>  					    const struct of_phandle_args *args)
>  {
> +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_db);
> +	unsigned int type = args->args[0], master = args->args[1];
>  	struct tegra_hsp_channel *channel = ERR_PTR(-ENODEV);
> -	struct tegra_hsp *hsp = to_tegra_hsp(mbox);
> -	unsigned int type = args->args[0];
> -	unsigned int master = args->args[1];
>  	struct tegra_hsp_doorbell *db;
>  	struct mbox_chan *chan;
>  	unsigned long flags;
>  	unsigned int i;
>  
> -	switch (type) {
> -	case TEGRA_HSP_MBOX_TYPE_DB:
> -		db = tegra_hsp_doorbell_get(hsp, master);
> -		if (db)
> -			channel = &db->channel;
> +	if (type != TEGRA_HSP_MBOX_TYPE_DB || !hsp->doorbell_irq)
> +		return ERR_PTR(-ENODEV);
>  
> -		break;
> -
> -	default:
> -		break;
> -	}
> +	db = tegra_hsp_doorbell_get(hsp, master);
> +	if (db)
> +		channel = &db->channel;
>  
>  	if (IS_ERR(channel))
>  		return ERR_CAST(channel);
>  
>  	spin_lock_irqsave(&hsp->lock, flags);
>  
> -	for (i = 0; i < hsp->mbox.num_chans; i++) {
> -		chan = &hsp->mbox.chans[i];
> +	for (i = 0; i < mbox->num_chans; i++) {
> +		chan = &mbox->chans[i];
>  		if (!chan->con_priv) {
> -			chan->con_priv = channel;
>  			channel->chan = chan;
> +			chan->con_priv = db;
>  			break;
>  		}
>  
> @@ -332,6 +544,29 @@ static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
>  	return chan ?: ERR_PTR(-EBUSY);
>  }
>  
> +static struct mbox_chan *tegra_hsp_sm_xlate(struct mbox_controller *mbox,
> +					    const struct of_phandle_args *args)
> +{
> +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_sm);
> +	unsigned int type = args->args[0], index;
> +	struct tegra_hsp_mailbox *mb;
> +
> +	index = args->args[1] & TEGRA_HSP_SM_MASK;
> +
> +	if (type != TEGRA_HSP_MBOX_TYPE_SM || !hsp->shared_irqs ||
> +	    index >= hsp->num_sm)
> +		return ERR_PTR(-ENODEV);
> +
> +	mb = &hsp->mailboxes[index];
> +
> +	if ((args->args[1] & TEGRA_HSP_SM_FLAG_TX) == 0)
> +		mb->producer = false;
> +	else
> +		mb->producer = true;
> +
> +	return mb->channel.chan;
> +}
> +
>  static void tegra_hsp_remove_doorbells(struct tegra_hsp *hsp)
>  {
>  	struct tegra_hsp_doorbell *db, *tmp;
> @@ -364,10 +599,65 @@ static int tegra_hsp_add_doorbells(struct tegra_hsp *hsp)
>  	return 0;
>  }
>  
> +static int tegra_hsp_add_mailboxes(struct tegra_hsp *hsp, struct device *dev)
> +{
> +	int i;
> +
> +	hsp->mailboxes = devm_kcalloc(dev, hsp->num_sm, sizeof(*hsp->mailboxes),
> +				      GFP_KERNEL);
> +	if (!hsp->mailboxes)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < hsp->num_sm; i++) {
> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
> +
> +		mb->index = i;
> +
> +		mb->channel.hsp = hsp;
> +		mb->channel.regs = hsp->regs + SZ_64K + i * SZ_32K;
> +		mb->channel.chan = &hsp->mbox_sm.chans[i];
> +		mb->channel.chan->con_priv = mb;
> +	}
> +
> +	return 0;
> +}
> +
> +static int tegra_hsp_request_shared_irqs(struct tegra_hsp *hsp)
> +{
> +	unsigned int i, irq = 0;
> +	int err;
> +
> +	for (i = 0; i < hsp->num_si; i++) {
> +		if (hsp->shared_irq == 0 && hsp->shared_irqs[i] > 0) {
> +			irq = hsp->shared_irqs[i];
> +			hsp->shared_irq = i;
> +			break;
> +		}
> +	}
> +
> +	if (irq > 0) {
> +		err = devm_request_irq(hsp->dev, irq, tegra_hsp_shared_irq, 0,
> +				       dev_name(hsp->dev), hsp);
> +		if (err < 0) {
> +			dev_err(hsp->dev, "failed to request interrupt: %d\n",
> +				err);
> +			return err;
> +		}

Are we suppose to loop through all the shared interrupts and find one
that is available? Looking at the above it seems that we will try to use
the first shared interrupt we have a valid mapping for, regardless of if
it is available/in-use.

Otherwise I am not sure why it is necessary to stored all the shared
irqs, because AFAICT we only use the first we have a valid mapping for.
Maybe I am missing something ...


> +
> +		/* disable all interrupts */
> +		tegra_hsp_writel(hsp, 0, HSP_INT_IE(hsp->shared_irq)> +
> +		dev_dbg(hsp->dev, "interrupt requested: %u\n", irq);
> +	}
> +
> +	return 0;
> +}
> +
>  static int tegra_hsp_probe(struct platform_device *pdev)
>  {
>  	struct tegra_hsp *hsp;
>  	struct resource *res;
> +	unsigned int i;
>  	u32 value;
>  	int err;
>  
> @@ -375,6 +665,7 @@ static int tegra_hsp_probe(struct platform_device *pdev)
>  	if (!hsp)
>  		return -ENOMEM;
>  
> +	hsp->dev = &pdev->dev;
>  	hsp->soc = of_device_get_match_data(&pdev->dev);
>  	INIT_LIST_HEAD(&hsp->doorbells);
>  	spin_lock_init(&hsp->lock);
> @@ -392,58 +683,136 @@ static int tegra_hsp_probe(struct platform_device *pdev)
>  	hsp->num_si = (value >> HSP_nSI_SHIFT) & HSP_nINT_MASK;
>  
>  	err = platform_get_irq_byname(pdev, "doorbell");
> -	if (err < 0) {
> -		dev_err(&pdev->dev, "failed to get doorbell IRQ: %d\n", err);
> -		return err;
> +	if (err >= 0)
> +		hsp->doorbell_irq = err;
> +
> +	if (hsp->num_si > 0) {
> +		unsigned int count = 0;
> +
> +		hsp->shared_irqs = devm_kcalloc(&pdev->dev, hsp->num_si,
> +						sizeof(*hsp->shared_irqs),
> +						GFP_KERNEL);
> +		if (!hsp->shared_irqs)
> +			return -ENOMEM;
> +
> +		for (i = 0; i < hsp->num_si; i++) {
> +			char *name;
> +
> +			name = kasprintf(GFP_KERNEL, "shared%u", i);
> +			if (!name)
> +				return -ENOMEM;
> +
> +			err = platform_get_irq_byname(pdev, name);
> +			if (err >= 0) {
> +				hsp->shared_irqs[i] = err;
> +				count++;
> +			}
> +
> +			kfree(name);
> +		}
> +
> +		if (count == 0) {
> +			devm_kfree(&pdev->dev, hsp->shared_irqs);
> +			hsp->shared_irqs = NULL;
> +		}
> +	}
> +
> +	/* setup the doorbell controller */
> +	hsp->mbox_db.of_xlate = tegra_hsp_db_xlate;
> +	hsp->mbox_db.num_chans = 32;
> +	hsp->mbox_db.dev = &pdev->dev;
> +	hsp->mbox_db.ops = &tegra_hsp_db_ops;
> +
> +	hsp->mbox_db.chans = devm_kcalloc(&pdev->dev, hsp->mbox_db.num_chans,
> +					  sizeof(*hsp->mbox_db.chans),
> +					  GFP_KERNEL);
> +	if (!hsp->mbox_db.chans)
> +		return -ENOMEM;
> +
> +	if (hsp->doorbell_irq) {
> +		err = tegra_hsp_add_doorbells(hsp);
> +		if (err < 0) {
> +			dev_err(&pdev->dev, "failed to add doorbells: %d\n",
> +			        err);
> +			return err;
> +		}
>  	}
>  
> -	hsp->irq = err;
> +	err = mbox_controller_register(&hsp->mbox_db);
> +	if (err < 0) {
> +		dev_err(&pdev->dev, "failed to register doorbell mailbox: %d\n", err);
> +		goto remove_doorbells;
> +	}
>  
> -	hsp->mbox.of_xlate = of_tegra_hsp_xlate;
> -	hsp->mbox.num_chans = 32;
> -	hsp->mbox.dev = &pdev->dev;
> -	hsp->mbox.txdone_irq = false;
> -	hsp->mbox.txdone_poll = false;
> -	hsp->mbox.ops = &tegra_hsp_doorbell_ops;
> +	/* setup the shared mailbox controller */
> +	hsp->mbox_sm.of_xlate = tegra_hsp_sm_xlate;
> +	hsp->mbox_sm.num_chans = hsp->num_sm;
> +	hsp->mbox_sm.dev = &pdev->dev;
> +	hsp->mbox_sm.ops = &tegra_hsp_sm_ops;
>  
> -	hsp->mbox.chans = devm_kcalloc(&pdev->dev, hsp->mbox.num_chans,
> -					sizeof(*hsp->mbox.chans),
> -					GFP_KERNEL);
> -	if (!hsp->mbox.chans)
> +	hsp->mbox_sm.chans = devm_kcalloc(&pdev->dev, hsp->mbox_sm.num_chans,
> +					  sizeof(*hsp->mbox_sm.chans),
> +					  GFP_KERNEL);
> +	if (!hsp->mbox_sm.chans)
>  		return -ENOMEM;
>  
> -	err = tegra_hsp_add_doorbells(hsp);
> +	if (hsp->shared_irqs) {
> +		err = tegra_hsp_add_mailboxes(hsp, &pdev->dev);
> +		if (err < 0) {
> +			dev_err(&pdev->dev, "failed to add mailboxes: %d\n",
> +			        err);
> +			goto unregister_mbox_db;
> +		}
> +	}
> +
> +	err = mbox_controller_register(&hsp->mbox_sm);
>  	if (err < 0) {
> -		dev_err(&pdev->dev, "failed to add doorbells: %d\n", err);
> -		return err;
> +		dev_err(&pdev->dev, "failed to register shared mailbox: %d\n", err);
> +		goto unregister_mbox_db;
>  	}
>  
>  	platform_set_drvdata(pdev, hsp);
>  
> -	err = mbox_controller_register(&hsp->mbox);
> -	if (err) {
> -		dev_err(&pdev->dev, "failed to register mailbox: %d\n", err);
> -		tegra_hsp_remove_doorbells(hsp);
> -		return err;
> +	if (hsp->doorbell_irq) {
> +		err = devm_request_irq(&pdev->dev, hsp->doorbell_irq,
> +				       tegra_hsp_doorbell_irq, IRQF_NO_SUSPEND,
> +				       dev_name(&pdev->dev), hsp);
> +		if (err < 0) {
> +			dev_err(&pdev->dev,
> +			        "failed to request doorbell IRQ#%u: %d\n",
> +				hsp->doorbell_irq, err);
> +			goto unregister_mbox_sm;
> +		}
>  	}
>  
> -	err = devm_request_irq(&pdev->dev, hsp->irq, tegra_hsp_doorbell_irq,
> -			       IRQF_NO_SUSPEND, dev_name(&pdev->dev), hsp);
> -	if (err < 0) {
> -		dev_err(&pdev->dev, "failed to request IRQ#%u: %d\n",
> -			hsp->irq, err);
> -		return err;
> +	if (hsp->shared_irqs) {
> +		err = tegra_hsp_request_shared_irqs(hsp);
> +		if (err < 0)
> +			goto unregister_mbox_sm;

Any reason why we don't request the doorbell and shared irqs earlier?
Given we use devm seems it could simplify the clean and avoid
unregistering the mailbox.

Cheers
Jon

-- 
nvpublic

[PATCH v2 04/10] mailbox: tegra-hsp: Add support for shared mailboxes

[Jon Hunter]

On 12/11/2018 15:18, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> The Tegra HSP block supports 'shared mailboxes' that are simple 32-bit
> registers consisting of a FULL bit in MSB position and 31 bits of data.
> The hardware can be configured to trigger interrupts when a mailbox
> is empty or full. Add support for these shared mailboxes to the HSP
> driver.
> 
> The initial use for the mailboxes is the Tegra Combined UART. For this
> purpose, we use interrupts to receive data, and spinning to wait for
> the transmit mailbox to be emptied to minimize unnecessary overhead.
> 
> Based on work by Mikko Perttunen <mperttunen@nvidia.com>.
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
> Changes in v2:
> - do not write per-mailbox interrupt enable registers on Tegra186
> - merge handlers for empty and full interrupts
> - track direction of shared mailboxes
> 
>  drivers/mailbox/tegra-hsp.c | 498 +++++++++++++++++++++++++++++++-----
>  1 file changed, 437 insertions(+), 61 deletions(-)
> 
> diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
> index 0cde356c11ab..0100a974149b 100644
> --- a/drivers/mailbox/tegra-hsp.c
> +++ b/drivers/mailbox/tegra-hsp.c
> @@ -1,5 +1,5 @@
>  /*
> - * Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.
> + * Copyright (c) 2016-2018, NVIDIA CORPORATION.  All rights reserved.
>   *
>   * This program is free software; you can redistribute it and/or modify it
>   * under the terms and conditions of the GNU General Public License,
> @@ -11,6 +11,7 @@
>   * more details.
>   */
>  
> +#include <linux/delay.h>
>  #include <linux/interrupt.h>
>  #include <linux/io.h>
>  #include <linux/mailbox_controller.h>
> @@ -21,6 +22,17 @@
>  
>  #include <dt-bindings/mailbox/tegra186-hsp.h>
>  
> +#include "mailbox.h"
> +
> +#define HSP_INT_IE(x)		(0x100 + ((x) * 4))
> +#define HSP_INT_IV		0x300
> +#define HSP_INT_IR		0x304
> +
> +#define HSP_INT_EMPTY_SHIFT	0
> +#define HSP_INT_EMPTY_MASK	0xff
> +#define HSP_INT_FULL_SHIFT	8
> +#define HSP_INT_FULL_MASK	0xff
> +
>  #define HSP_INT_DIMENSIONING	0x380
>  #define HSP_nSM_SHIFT		0
>  #define HSP_nSS_SHIFT		4
> @@ -34,6 +46,11 @@
>  #define HSP_DB_RAW	0x8
>  #define HSP_DB_PENDING	0xc
>  
> +#define HSP_SM_SHRD_MBOX	0x0
> +#define HSP_SM_SHRD_MBOX_FULL	BIT(31)
> +#define HSP_SM_SHRD_MBOX_FULL_INT_IE	0x04
> +#define HSP_SM_SHRD_MBOX_EMPTY_INT_IE	0x08
> +
>  #define HSP_DB_CCPLEX		1
>  #define HSP_DB_BPMP		3
>  #define HSP_DB_MAX		7
> @@ -55,6 +72,12 @@ struct tegra_hsp_doorbell {
>  	unsigned int index;
>  };
>  
> +struct tegra_hsp_mailbox {
> +	struct tegra_hsp_channel channel;
> +	unsigned int index;
> +	bool producer;
> +};
> +
>  struct tegra_hsp_db_map {
>  	const char *name;
>  	unsigned int master;
> @@ -63,13 +86,18 @@ struct tegra_hsp_db_map {
>  
>  struct tegra_hsp_soc {
>  	const struct tegra_hsp_db_map *map;
> +	bool has_per_mb_ie;
>  };
>  
>  struct tegra_hsp {
> +	struct device *dev;
>  	const struct tegra_hsp_soc *soc;
> -	struct mbox_controller mbox;
> +	struct mbox_controller mbox_db;
> +	struct mbox_controller mbox_sm;
>  	void __iomem *regs;
> -	unsigned int irq;
> +	unsigned int doorbell_irq;
> +	unsigned int *shared_irqs;
> +	unsigned int shared_irq;
>  	unsigned int num_sm;
>  	unsigned int num_as;
>  	unsigned int num_ss;
> @@ -78,13 +106,10 @@ struct tegra_hsp {
>  	spinlock_t lock;
>  
>  	struct list_head doorbells;
> -};
> +	struct tegra_hsp_mailbox *mailboxes;
>  
> -static inline struct tegra_hsp *
> -to_tegra_hsp(struct mbox_controller *mbox)
> -{
> -	return container_of(mbox, struct tegra_hsp, mbox);
> -}
> +	unsigned long mask;
> +};
>  
>  static inline u32 tegra_hsp_readl(struct tegra_hsp *hsp, unsigned int offset)
>  {
> @@ -158,7 +183,7 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
>  
>  	spin_lock(&hsp->lock);
>  
> -	for_each_set_bit(master, &value, hsp->mbox.num_chans) {
> +	for_each_set_bit(master, &value, hsp->mbox_db.num_chans) {
>  		struct tegra_hsp_doorbell *db;
>  
>  		db = __tegra_hsp_doorbell_get(hsp, master);
> @@ -182,6 +207,71 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
>  	return IRQ_HANDLED;
>  }
>  
> +static irqreturn_t tegra_hsp_shared_irq(int irq, void *data)
> +{
> +	struct tegra_hsp *hsp = data;
> +	unsigned long bit, mask;
> +	u32 status, value;
> +	void *msg;
> +
> +	status = tegra_hsp_readl(hsp, HSP_INT_IR) & hsp->mask;
> +
> +	/* process EMPTY interrupts first */
> +	mask = (status >> HSP_INT_EMPTY_SHIFT) & HSP_INT_EMPTY_MASK;
> +
> +	for_each_set_bit(bit, &mask, hsp->num_sm) {
> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
> +
> +		if (mb->producer) {
> +			/*
> +			 * Disable EMPTY interrupts until data is sent with
> +			 * the next message. These interrupts are level-
> +			 * triggered, so if we kept them enabled they would
> +			 * constantly trigger until we next write data into
> +			 * the message.
> +			 */
> +			spin_lock(&hsp->lock);
> +
> +			hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> +			tegra_hsp_writel(hsp, hsp->mask,
> +					 HSP_INT_IE(hsp->shared_irq));
> +
> +			spin_unlock(&hsp->lock);
> +
> +			mbox_chan_txdone(mb->channel.chan, 0);
> +		}
> +	}
> +
> +	/* process FULL interrupts */
> +	mask = (status >> HSP_INT_FULL_SHIFT) & HSP_INT_FULL_MASK;
> +
> +	for_each_set_bit(bit, &mask, hsp->num_sm) {
> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
> +
> +		if (!mb->producer) {
> +			value = tegra_hsp_channel_readl(&mb->channel,
> +							HSP_SM_SHRD_MBOX);
> +			value &= ~HSP_SM_SHRD_MBOX_FULL;
> +			msg = (void *)(unsigned long)value;
> +			mbox_chan_received_data(mb->channel.chan, msg);
> +
> +			/*
> +			 * Need to clear all bits here since some producers,
> +			 * such as TCU, depend on fields in the register
> +			 * getting cleared by the consumer.
> +			 *
> +			 * The mailbox API doesn't give the consumers a way
> +			 * of doing that explicitly, so we have to make sure
> +			 * we cover all possible cases.
> +			 */
> +			tegra_hsp_channel_writel(&mb->channel, 0x0,
> +						 HSP_SM_SHRD_MBOX);
> +		}
> +	}
> +
> +	return IRQ_HANDLED;
> +}
> +
>  static struct tegra_hsp_channel *
>  tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
>  			  unsigned int master, unsigned int index)
> @@ -194,7 +284,7 @@ tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
>  	if (!db)
>  		return ERR_PTR(-ENOMEM);
>  
> -	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) << 16;
> +	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) * SZ_64K;
>  	offset += index * 0x100;
>  
>  	db->channel.regs = hsp->regs + offset;
> @@ -235,8 +325,8 @@ static int tegra_hsp_doorbell_startup(struct mbox_chan *chan)
>  	unsigned long flags;
>  	u32 value;
>  
> -	if (db->master >= hsp->mbox.num_chans) {
> -		dev_err(hsp->mbox.dev,
> +	if (db->master >= chan->mbox->num_chans) {
> +		dev_err(chan->mbox->dev,
>  			"invalid master ID %u for HSP channel\n",
>  			db->master);
>  		return -EINVAL;
> @@ -281,46 +371,168 @@ static void tegra_hsp_doorbell_shutdown(struct mbox_chan *chan)
>  	spin_unlock_irqrestore(&hsp->lock, flags);
>  }
>  
> -static const struct mbox_chan_ops tegra_hsp_doorbell_ops = {
> +static const struct mbox_chan_ops tegra_hsp_db_ops = {
>  	.send_data = tegra_hsp_doorbell_send_data,
>  	.startup = tegra_hsp_doorbell_startup,
>  	.shutdown = tegra_hsp_doorbell_shutdown,
>  };
>  
> -static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
> +static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
> +{
> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> +	struct tegra_hsp *hsp = mb->channel.hsp;
> +	unsigned long flags;
> +	u32 value;
> +
> +	WARN_ON(!mb->producer);

Should we return here?

> +
> +	/* copy data and mark mailbox full */
> +	value = (u32)(unsigned long)data;
> +	value |= HSP_SM_SHRD_MBOX_FULL;
> +
> +	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
> +
> +	if (!irqs_disabled()) {
> +		/* enable EMPTY interrupt for the shared mailbox */
> +		spin_lock_irqsave(&hsp->lock, flags);
> +
> +		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> +		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> +
> +		spin_unlock_irqrestore(&hsp->lock, flags);
> +	}
> +
> +	return 0;
> +}
> +
> +static int tegra_hsp_mailbox_flush(struct mbox_chan *chan,
> +				   unsigned long timeout)
> +{
> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> +	struct tegra_hsp_channel *ch = &mb->channel;
> +	u32 value;
> +
> +	timeout = jiffies + msecs_to_jiffies(timeout);
> +
> +	while (time_before(jiffies, timeout)) {
> +		value = tegra_hsp_channel_readl(ch, HSP_SM_SHRD_MBOX);
> +		if ((value & HSP_SM_SHRD_MBOX_FULL) == 0) {
> +			mbox_chan_txdone(chan, 0);
> +			return 0;
> +		}
> +
> +		udelay(1);
> +	}
> +
> +	return -ETIME;
> +}
> +
> +static int tegra_hsp_mailbox_startup(struct mbox_chan *chan)
> +{
> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> +	struct tegra_hsp_channel *ch = &mb->channel;
> +	struct tegra_hsp *hsp = mb->channel.hsp;
> +	unsigned long flags;
> +
> +	chan->txdone_method = TXDONE_BY_IRQ;
> +
> +	/*
> +	 * Shared mailboxes start out as consumers by default. FULL and EMPTY
> +	 * interrupts are coalesced at the same shared interrupt.
> +	 *
> +	 * Keep EMPTY interrupts disabled at startup and only enable them when
> +	 * the mailbox is actually full. This is required because the FULL and
> +	 * EMPTY interrupts are level-triggered, so keeping EMPTY interrupts
> +	 * enabled all the time would cause an interrupt storm while mailboxes
> +	 * are idle.
> +	 */
> +
> +	spin_lock_irqsave(&hsp->lock, flags);
> +
> +	if (mb->producer)
> +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> +	else
> +		hsp->mask |= BIT(HSP_INT_FULL_SHIFT + mb->index);
> +
> +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> +
> +	spin_unlock_irqrestore(&hsp->lock, flags);
> +
> +	if (hsp->soc->has_per_mb_ie) {
> +		if (mb->producer)
> +			tegra_hsp_channel_writel(ch, 0x0,
> +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
> +		else
> +			tegra_hsp_channel_writel(ch, 0x1,
> +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
> +	}
> +
> +	return 0;
> +}
> +
> +static void tegra_hsp_mailbox_shutdown(struct mbox_chan *chan)
> +{
> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> +	struct tegra_hsp_channel *ch = &mb->channel;
> +	struct tegra_hsp *hsp = mb->channel.hsp;
> +	unsigned long flags;
> +
> +	if (hsp->soc->has_per_mb_ie) {
> +		if (mb->producer)
> +			tegra_hsp_channel_writel(ch, 0x0,
> +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
> +		else
> +			tegra_hsp_channel_writel(ch, 0x0,
> +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
> +	}
> +
> +	spin_lock_irqsave(&hsp->lock, flags);
> +
> +	if (mb->producer)
> +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> +	else
> +		hsp->mask &= ~BIT(HSP_INT_FULL_SHIFT + mb->index);
> +
> +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> +
> +	spin_unlock_irqrestore(&hsp->lock, flags);
> +}
> +
> +static const struct mbox_chan_ops tegra_hsp_sm_ops = {
> +	.send_data = tegra_hsp_mailbox_send_data,
> +	.flush = tegra_hsp_mailbox_flush,
> +	.startup = tegra_hsp_mailbox_startup,
> +	.shutdown = tegra_hsp_mailbox_shutdown,
> +};
> +
> +static struct mbox_chan *tegra_hsp_db_xlate(struct mbox_controller *mbox,
>  					    const struct of_phandle_args *args)
>  {
> +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_db);
> +	unsigned int type = args->args[0], master = args->args[1];
>  	struct tegra_hsp_channel *channel = ERR_PTR(-ENODEV);
> -	struct tegra_hsp *hsp = to_tegra_hsp(mbox);
> -	unsigned int type = args->args[0];
> -	unsigned int master = args->args[1];
>  	struct tegra_hsp_doorbell *db;
>  	struct mbox_chan *chan;
>  	unsigned long flags;
>  	unsigned int i;
>  
> -	switch (type) {
> -	case TEGRA_HSP_MBOX_TYPE_DB:
> -		db = tegra_hsp_doorbell_get(hsp, master);
> -		if (db)
> -			channel = &db->channel;
> +	if (type != TEGRA_HSP_MBOX_TYPE_DB || !hsp->doorbell_irq)
> +		return ERR_PTR(-ENODEV);
>  
> -		break;
> -
> -	default:
> -		break;
> -	}
> +	db = tegra_hsp_doorbell_get(hsp, master);
> +	if (db)
> +		channel = &db->channel;
>  
>  	if (IS_ERR(channel))
>  		return ERR_CAST(channel);
>  
>  	spin_lock_irqsave(&hsp->lock, flags);
>  
> -	for (i = 0; i < hsp->mbox.num_chans; i++) {
> -		chan = &hsp->mbox.chans[i];
> +	for (i = 0; i < mbox->num_chans; i++) {
> +		chan = &mbox->chans[i];
>  		if (!chan->con_priv) {
> -			chan->con_priv = channel;
>  			channel->chan = chan;
> +			chan->con_priv = db;
>  			break;
>  		}
>  
> @@ -332,6 +544,29 @@ static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
>  	return chan ?: ERR_PTR(-EBUSY);
>  }
>  
> +static struct mbox_chan *tegra_hsp_sm_xlate(struct mbox_controller *mbox,
> +					    const struct of_phandle_args *args)
> +{
> +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_sm);
> +	unsigned int type = args->args[0], index;
> +	struct tegra_hsp_mailbox *mb;
> +
> +	index = args->args[1] & TEGRA_HSP_SM_MASK;
> +
> +	if (type != TEGRA_HSP_MBOX_TYPE_SM || !hsp->shared_irqs ||
> +	    index >= hsp->num_sm)
> +		return ERR_PTR(-ENODEV);
> +
> +	mb = &hsp->mailboxes[index];
> +
> +	if ((args->args[1] & TEGRA_HSP_SM_FLAG_TX) == 0)
> +		mb->producer = false;
> +	else
> +		mb->producer = true;
> +
> +	return mb->channel.chan;
> +}
> +
>  static void tegra_hsp_remove_doorbells(struct tegra_hsp *hsp)
>  {
>  	struct tegra_hsp_doorbell *db, *tmp;
> @@ -364,10 +599,65 @@ static int tegra_hsp_add_doorbells(struct tegra_hsp *hsp)
>  	return 0;
>  }
>  
> +static int tegra_hsp_add_mailboxes(struct tegra_hsp *hsp, struct device *dev)
> +{
> +	int i;
> +
> +	hsp->mailboxes = devm_kcalloc(dev, hsp->num_sm, sizeof(*hsp->mailboxes),
> +				      GFP_KERNEL);
> +	if (!hsp->mailboxes)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < hsp->num_sm; i++) {
> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
> +
> +		mb->index = i;
> +
> +		mb->channel.hsp = hsp;
> +		mb->channel.regs = hsp->regs + SZ_64K + i * SZ_32K;
> +		mb->channel.chan = &hsp->mbox_sm.chans[i];
> +		mb->channel.chan->con_priv = mb;
> +	}
> +
> +	return 0;
> +}
> +
> +static int tegra_hsp_request_shared_irqs(struct tegra_hsp *hsp)
> +{
> +	unsigned int i, irq = 0;
> +	int err;
> +
> +	for (i = 0; i < hsp->num_si; i++) {
> +		if (hsp->shared_irq == 0 && hsp->shared_irqs[i] > 0) {
> +			irq = hsp->shared_irqs[i];
> +			hsp->shared_irq = i;
> +			break;
> +		}
> +	}
> +
> +	if (irq > 0) {
> +		err = devm_request_irq(hsp->dev, irq, tegra_hsp_shared_irq, 0,
> +				       dev_name(hsp->dev), hsp);
> +		if (err < 0) {
> +			dev_err(hsp->dev, "failed to request interrupt: %d\n",
> +				err);
> +			return err;
> +		}

Are we suppose to loop through all the shared interrupts and find one
that is available? Looking at the above it seems that we will try to use
the first shared interrupt we have a valid mapping for, regardless of if
it is available/in-use.

Otherwise I am not sure why it is necessary to stored all the shared
irqs, because AFAICT we only use the first we have a valid mapping for.
Maybe I am missing something ...


> +
> +		/* disable all interrupts */
> +		tegra_hsp_writel(hsp, 0, HSP_INT_IE(hsp->shared_irq)> +
> +		dev_dbg(hsp->dev, "interrupt requested: %u\n", irq);
> +	}
> +
> +	return 0;
> +}
> +
>  static int tegra_hsp_probe(struct platform_device *pdev)
>  {
>  	struct tegra_hsp *hsp;
>  	struct resource *res;
> +	unsigned int i;
>  	u32 value;
>  	int err;
>  
> @@ -375,6 +665,7 @@ static int tegra_hsp_probe(struct platform_device *pdev)
>  	if (!hsp)
>  		return -ENOMEM;
>  
> +	hsp->dev = &pdev->dev;
>  	hsp->soc = of_device_get_match_data(&pdev->dev);
>  	INIT_LIST_HEAD(&hsp->doorbells);
>  	spin_lock_init(&hsp->lock);
> @@ -392,58 +683,136 @@ static int tegra_hsp_probe(struct platform_device *pdev)
>  	hsp->num_si = (value >> HSP_nSI_SHIFT) & HSP_nINT_MASK;
>  
>  	err = platform_get_irq_byname(pdev, "doorbell");
> -	if (err < 0) {
> -		dev_err(&pdev->dev, "failed to get doorbell IRQ: %d\n", err);
> -		return err;
> +	if (err >= 0)
> +		hsp->doorbell_irq = err;
> +
> +	if (hsp->num_si > 0) {
> +		unsigned int count = 0;
> +
> +		hsp->shared_irqs = devm_kcalloc(&pdev->dev, hsp->num_si,
> +						sizeof(*hsp->shared_irqs),
> +						GFP_KERNEL);
> +		if (!hsp->shared_irqs)
> +			return -ENOMEM;
> +
> +		for (i = 0; i < hsp->num_si; i++) {
> +			char *name;
> +
> +			name = kasprintf(GFP_KERNEL, "shared%u", i);
> +			if (!name)
> +				return -ENOMEM;
> +
> +			err = platform_get_irq_byname(pdev, name);
> +			if (err >= 0) {
> +				hsp->shared_irqs[i] = err;
> +				count++;
> +			}
> +
> +			kfree(name);
> +		}
> +
> +		if (count == 0) {
> +			devm_kfree(&pdev->dev, hsp->shared_irqs);
> +			hsp->shared_irqs = NULL;
> +		}
> +	}
> +
> +	/* setup the doorbell controller */
> +	hsp->mbox_db.of_xlate = tegra_hsp_db_xlate;
> +	hsp->mbox_db.num_chans = 32;
> +	hsp->mbox_db.dev = &pdev->dev;
> +	hsp->mbox_db.ops = &tegra_hsp_db_ops;
> +
> +	hsp->mbox_db.chans = devm_kcalloc(&pdev->dev, hsp->mbox_db.num_chans,
> +					  sizeof(*hsp->mbox_db.chans),
> +					  GFP_KERNEL);
> +	if (!hsp->mbox_db.chans)
> +		return -ENOMEM;
> +
> +	if (hsp->doorbell_irq) {
> +		err = tegra_hsp_add_doorbells(hsp);
> +		if (err < 0) {
> +			dev_err(&pdev->dev, "failed to add doorbells: %d\n",
> +			        err);
> +			return err;
> +		}
>  	}
>  
> -	hsp->irq = err;
> +	err = mbox_controller_register(&hsp->mbox_db);
> +	if (err < 0) {
> +		dev_err(&pdev->dev, "failed to register doorbell mailbox: %d\n", err);
> +		goto remove_doorbells;
> +	}
>  
> -	hsp->mbox.of_xlate = of_tegra_hsp_xlate;
> -	hsp->mbox.num_chans = 32;
> -	hsp->mbox.dev = &pdev->dev;
> -	hsp->mbox.txdone_irq = false;
> -	hsp->mbox.txdone_poll = false;
> -	hsp->mbox.ops = &tegra_hsp_doorbell_ops;
> +	/* setup the shared mailbox controller */
> +	hsp->mbox_sm.of_xlate = tegra_hsp_sm_xlate;
> +	hsp->mbox_sm.num_chans = hsp->num_sm;
> +	hsp->mbox_sm.dev = &pdev->dev;
> +	hsp->mbox_sm.ops = &tegra_hsp_sm_ops;
>  
> -	hsp->mbox.chans = devm_kcalloc(&pdev->dev, hsp->mbox.num_chans,
> -					sizeof(*hsp->mbox.chans),
> -					GFP_KERNEL);
> -	if (!hsp->mbox.chans)
> +	hsp->mbox_sm.chans = devm_kcalloc(&pdev->dev, hsp->mbox_sm.num_chans,
> +					  sizeof(*hsp->mbox_sm.chans),
> +					  GFP_KERNEL);
> +	if (!hsp->mbox_sm.chans)
>  		return -ENOMEM;
>  
> -	err = tegra_hsp_add_doorbells(hsp);
> +	if (hsp->shared_irqs) {
> +		err = tegra_hsp_add_mailboxes(hsp, &pdev->dev);
> +		if (err < 0) {
> +			dev_err(&pdev->dev, "failed to add mailboxes: %d\n",
> +			        err);
> +			goto unregister_mbox_db;
> +		}
> +	}
> +
> +	err = mbox_controller_register(&hsp->mbox_sm);
>  	if (err < 0) {
> -		dev_err(&pdev->dev, "failed to add doorbells: %d\n", err);
> -		return err;
> +		dev_err(&pdev->dev, "failed to register shared mailbox: %d\n", err);
> +		goto unregister_mbox_db;
>  	}
>  
>  	platform_set_drvdata(pdev, hsp);
>  
> -	err = mbox_controller_register(&hsp->mbox);
> -	if (err) {
> -		dev_err(&pdev->dev, "failed to register mailbox: %d\n", err);
> -		tegra_hsp_remove_doorbells(hsp);
> -		return err;
> +	if (hsp->doorbell_irq) {
> +		err = devm_request_irq(&pdev->dev, hsp->doorbell_irq,
> +				       tegra_hsp_doorbell_irq, IRQF_NO_SUSPEND,
> +				       dev_name(&pdev->dev), hsp);
> +		if (err < 0) {
> +			dev_err(&pdev->dev,
> +			        "failed to request doorbell IRQ#%u: %d\n",
> +				hsp->doorbell_irq, err);
> +			goto unregister_mbox_sm;
> +		}
>  	}
>  
> -	err = devm_request_irq(&pdev->dev, hsp->irq, tegra_hsp_doorbell_irq,
> -			       IRQF_NO_SUSPEND, dev_name(&pdev->dev), hsp);
> -	if (err < 0) {
> -		dev_err(&pdev->dev, "failed to request IRQ#%u: %d\n",
> -			hsp->irq, err);
> -		return err;
> +	if (hsp->shared_irqs) {
> +		err = tegra_hsp_request_shared_irqs(hsp);
> +		if (err < 0)
> +			goto unregister_mbox_sm;

Any reason why we don't request the doorbell and shared irqs earlier?
Given we use devm seems it could simplify the clean and avoid
unregistering the mailbox.

Cheers
Jon

-- 
nvpublic

Re: [PATCH v2 05/10] mailbox: tegra-hsp: Add suspend/resume support

[Jon Hunter]

On 12/11/2018 15:18, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> Upon resuming from a system sleep state, the interrupts for all active
> shared mailboxes need to be reenabled, otherwise they will not work.
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
>  drivers/mailbox/tegra-hsp.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
> index 0100a974149b..1259abf3542f 100644
> --- a/drivers/mailbox/tegra-hsp.c
> +++ b/drivers/mailbox/tegra-hsp.c
> @@ -18,6 +18,7 @@
>  #include <linux/of.h>
>  #include <linux/of_device.h>
>  #include <linux/platform_device.h>
> +#include <linux/pm.h>
>  #include <linux/slab.h>
>  
>  #include <dt-bindings/mailbox/tegra186-hsp.h>
> @@ -817,6 +818,23 @@ static int tegra_hsp_remove(struct platform_device *pdev)
>  	return 0;
>  }
>  
> +static int tegra_hsp_resume(struct device *dev)
> +{
> +	struct tegra_hsp *hsp = dev_get_drvdata(dev);
> +	unsigned int i;
> +
> +	for (i = 0; i < hsp->num_sm; i++) {
> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
> +
> +		if (mb->channel.chan->cl)
> +			tegra_hsp_mailbox_startup(mb->channel.chan);
> +	}
> +
> +	return 0;
> +}
> +
> +static SIMPLE_DEV_PM_OPS(tegra_hsp_pm_ops, NULL, tegra_hsp_resume);

Is it worth disabling interrupts on suspend to avoid any in-flight
interrupt triggering a bad access on entering suspend? I assume that the
context of the mailbox registers get cleared/lost at some point and so I
was not sure if there is a time where they have a bad state or are
inaccessible?

Cheers
Jon

-- 
nvpublic

[PATCH v2 05/10] mailbox: tegra-hsp: Add suspend/resume support

[Jon Hunter]

On 12/11/2018 15:18, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> Upon resuming from a system sleep state, the interrupts for all active
> shared mailboxes need to be reenabled, otherwise they will not work.
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
>  drivers/mailbox/tegra-hsp.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
> index 0100a974149b..1259abf3542f 100644
> --- a/drivers/mailbox/tegra-hsp.c
> +++ b/drivers/mailbox/tegra-hsp.c
> @@ -18,6 +18,7 @@
>  #include <linux/of.h>
>  #include <linux/of_device.h>
>  #include <linux/platform_device.h>
> +#include <linux/pm.h>
>  #include <linux/slab.h>
>  
>  #include <dt-bindings/mailbox/tegra186-hsp.h>
> @@ -817,6 +818,23 @@ static int tegra_hsp_remove(struct platform_device *pdev)
>  	return 0;
>  }
>  
> +static int tegra_hsp_resume(struct device *dev)
> +{
> +	struct tegra_hsp *hsp = dev_get_drvdata(dev);
> +	unsigned int i;
> +
> +	for (i = 0; i < hsp->num_sm; i++) {
> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
> +
> +		if (mb->channel.chan->cl)
> +			tegra_hsp_mailbox_startup(mb->channel.chan);
> +	}
> +
> +	return 0;
> +}
> +
> +static SIMPLE_DEV_PM_OPS(tegra_hsp_pm_ops, NULL, tegra_hsp_resume);

Is it worth disabling interrupts on suspend to avoid any in-flight
interrupt triggering a bad access on entering suspend? I assume that the
context of the mailbox registers get cleared/lost at some point and so I
was not sure if there is a time where they have a bad state or are
inaccessible?

Cheers
Jon

-- 
nvpublic

Re: [PATCH v2 04/10] mailbox: tegra-hsp: Add support for shared mailboxes

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 22776 bytes --]

On Tue, Nov 13, 2018 at 11:09:22AM +0000, Jon Hunter wrote:
> 
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > The Tegra HSP block supports 'shared mailboxes' that are simple 32-bit
> > registers consisting of a FULL bit in MSB position and 31 bits of data.
> > The hardware can be configured to trigger interrupts when a mailbox
> > is empty or full. Add support for these shared mailboxes to the HSP
> > driver.
> > 
> > The initial use for the mailboxes is the Tegra Combined UART. For this
> > purpose, we use interrupts to receive data, and spinning to wait for
> > the transmit mailbox to be emptied to minimize unnecessary overhead.
> > 
> > Based on work by Mikko Perttunen <mperttunen@nvidia.com>.
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> > Changes in v2:
> > - do not write per-mailbox interrupt enable registers on Tegra186
> > - merge handlers for empty and full interrupts
> > - track direction of shared mailboxes
> > 
> >  drivers/mailbox/tegra-hsp.c | 498 +++++++++++++++++++++++++++++++-----
> >  1 file changed, 437 insertions(+), 61 deletions(-)
> > 
> > diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
> > index 0cde356c11ab..0100a974149b 100644
> > --- a/drivers/mailbox/tegra-hsp.c
> > +++ b/drivers/mailbox/tegra-hsp.c
> > @@ -1,5 +1,5 @@
> >  /*
> > - * Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.
> > + * Copyright (c) 2016-2018, NVIDIA CORPORATION.  All rights reserved.
> >   *
> >   * This program is free software; you can redistribute it and/or modify it
> >   * under the terms and conditions of the GNU General Public License,
> > @@ -11,6 +11,7 @@
> >   * more details.
> >   */
> >  
> > +#include <linux/delay.h>
> >  #include <linux/interrupt.h>
> >  #include <linux/io.h>
> >  #include <linux/mailbox_controller.h>
> > @@ -21,6 +22,17 @@
> >  
> >  #include <dt-bindings/mailbox/tegra186-hsp.h>
> >  
> > +#include "mailbox.h"
> > +
> > +#define HSP_INT_IE(x)		(0x100 + ((x) * 4))
> > +#define HSP_INT_IV		0x300
> > +#define HSP_INT_IR		0x304
> > +
> > +#define HSP_INT_EMPTY_SHIFT	0
> > +#define HSP_INT_EMPTY_MASK	0xff
> > +#define HSP_INT_FULL_SHIFT	8
> > +#define HSP_INT_FULL_MASK	0xff
> > +
> >  #define HSP_INT_DIMENSIONING	0x380
> >  #define HSP_nSM_SHIFT		0
> >  #define HSP_nSS_SHIFT		4
> > @@ -34,6 +46,11 @@
> >  #define HSP_DB_RAW	0x8
> >  #define HSP_DB_PENDING	0xc
> >  
> > +#define HSP_SM_SHRD_MBOX	0x0
> > +#define HSP_SM_SHRD_MBOX_FULL	BIT(31)
> > +#define HSP_SM_SHRD_MBOX_FULL_INT_IE	0x04
> > +#define HSP_SM_SHRD_MBOX_EMPTY_INT_IE	0x08
> > +
> >  #define HSP_DB_CCPLEX		1
> >  #define HSP_DB_BPMP		3
> >  #define HSP_DB_MAX		7
> > @@ -55,6 +72,12 @@ struct tegra_hsp_doorbell {
> >  	unsigned int index;
> >  };
> >  
> > +struct tegra_hsp_mailbox {
> > +	struct tegra_hsp_channel channel;
> > +	unsigned int index;
> > +	bool producer;
> > +};
> > +
> >  struct tegra_hsp_db_map {
> >  	const char *name;
> >  	unsigned int master;
> > @@ -63,13 +86,18 @@ struct tegra_hsp_db_map {
> >  
> >  struct tegra_hsp_soc {
> >  	const struct tegra_hsp_db_map *map;
> > +	bool has_per_mb_ie;
> >  };
> >  
> >  struct tegra_hsp {
> > +	struct device *dev;
> >  	const struct tegra_hsp_soc *soc;
> > -	struct mbox_controller mbox;
> > +	struct mbox_controller mbox_db;
> > +	struct mbox_controller mbox_sm;
> >  	void __iomem *regs;
> > -	unsigned int irq;
> > +	unsigned int doorbell_irq;
> > +	unsigned int *shared_irqs;
> > +	unsigned int shared_irq;
> >  	unsigned int num_sm;
> >  	unsigned int num_as;
> >  	unsigned int num_ss;
> > @@ -78,13 +106,10 @@ struct tegra_hsp {
> >  	spinlock_t lock;
> >  
> >  	struct list_head doorbells;
> > -};
> > +	struct tegra_hsp_mailbox *mailboxes;
> >  
> > -static inline struct tegra_hsp *
> > -to_tegra_hsp(struct mbox_controller *mbox)
> > -{
> > -	return container_of(mbox, struct tegra_hsp, mbox);
> > -}
> > +	unsigned long mask;
> > +};
> >  
> >  static inline u32 tegra_hsp_readl(struct tegra_hsp *hsp, unsigned int offset)
> >  {
> > @@ -158,7 +183,7 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
> >  
> >  	spin_lock(&hsp->lock);
> >  
> > -	for_each_set_bit(master, &value, hsp->mbox.num_chans) {
> > +	for_each_set_bit(master, &value, hsp->mbox_db.num_chans) {
> >  		struct tegra_hsp_doorbell *db;
> >  
> >  		db = __tegra_hsp_doorbell_get(hsp, master);
> > @@ -182,6 +207,71 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
> >  	return IRQ_HANDLED;
> >  }
> >  
> > +static irqreturn_t tegra_hsp_shared_irq(int irq, void *data)
> > +{
> > +	struct tegra_hsp *hsp = data;
> > +	unsigned long bit, mask;
> > +	u32 status, value;
> > +	void *msg;
> > +
> > +	status = tegra_hsp_readl(hsp, HSP_INT_IR) & hsp->mask;
> > +
> > +	/* process EMPTY interrupts first */
> > +	mask = (status >> HSP_INT_EMPTY_SHIFT) & HSP_INT_EMPTY_MASK;
> > +
> > +	for_each_set_bit(bit, &mask, hsp->num_sm) {
> > +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
> > +
> > +		if (mb->producer) {
> > +			/*
> > +			 * Disable EMPTY interrupts until data is sent with
> > +			 * the next message. These interrupts are level-
> > +			 * triggered, so if we kept them enabled they would
> > +			 * constantly trigger until we next write data into
> > +			 * the message.
> > +			 */
> > +			spin_lock(&hsp->lock);
> > +
> > +			hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> > +			tegra_hsp_writel(hsp, hsp->mask,
> > +					 HSP_INT_IE(hsp->shared_irq));
> > +
> > +			spin_unlock(&hsp->lock);
> > +
> > +			mbox_chan_txdone(mb->channel.chan, 0);
> > +		}
> > +	}
> > +
> > +	/* process FULL interrupts */
> > +	mask = (status >> HSP_INT_FULL_SHIFT) & HSP_INT_FULL_MASK;
> > +
> > +	for_each_set_bit(bit, &mask, hsp->num_sm) {
> > +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
> > +
> > +		if (!mb->producer) {
> > +			value = tegra_hsp_channel_readl(&mb->channel,
> > +							HSP_SM_SHRD_MBOX);
> > +			value &= ~HSP_SM_SHRD_MBOX_FULL;
> > +			msg = (void *)(unsigned long)value;
> > +			mbox_chan_received_data(mb->channel.chan, msg);
> > +
> > +			/*
> > +			 * Need to clear all bits here since some producers,
> > +			 * such as TCU, depend on fields in the register
> > +			 * getting cleared by the consumer.
> > +			 *
> > +			 * The mailbox API doesn't give the consumers a way
> > +			 * of doing that explicitly, so we have to make sure
> > +			 * we cover all possible cases.
> > +			 */
> > +			tegra_hsp_channel_writel(&mb->channel, 0x0,
> > +						 HSP_SM_SHRD_MBOX);
> > +		}
> > +	}
> > +
> > +	return IRQ_HANDLED;
> > +}
> > +
> >  static struct tegra_hsp_channel *
> >  tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
> >  			  unsigned int master, unsigned int index)
> > @@ -194,7 +284,7 @@ tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
> >  	if (!db)
> >  		return ERR_PTR(-ENOMEM);
> >  
> > -	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) << 16;
> > +	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) * SZ_64K;
> >  	offset += index * 0x100;
> >  
> >  	db->channel.regs = hsp->regs + offset;
> > @@ -235,8 +325,8 @@ static int tegra_hsp_doorbell_startup(struct mbox_chan *chan)
> >  	unsigned long flags;
> >  	u32 value;
> >  
> > -	if (db->master >= hsp->mbox.num_chans) {
> > -		dev_err(hsp->mbox.dev,
> > +	if (db->master >= chan->mbox->num_chans) {
> > +		dev_err(chan->mbox->dev,
> >  			"invalid master ID %u for HSP channel\n",
> >  			db->master);
> >  		return -EINVAL;
> > @@ -281,46 +371,168 @@ static void tegra_hsp_doorbell_shutdown(struct mbox_chan *chan)
> >  	spin_unlock_irqrestore(&hsp->lock, flags);
> >  }
> >  
> > -static const struct mbox_chan_ops tegra_hsp_doorbell_ops = {
> > +static const struct mbox_chan_ops tegra_hsp_db_ops = {
> >  	.send_data = tegra_hsp_doorbell_send_data,
> >  	.startup = tegra_hsp_doorbell_startup,
> >  	.shutdown = tegra_hsp_doorbell_shutdown,
> >  };
> >  
> > -static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
> > +static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
> > +{
> > +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> > +	struct tegra_hsp *hsp = mb->channel.hsp;
> > +	unsigned long flags;
> > +	u32 value;
> > +
> > +	WARN_ON(!mb->producer);
> 
> Should we return here?

Yeah, that's a good idea. I made this return -EPERM for lack of a better
error code.

> > +
> > +	/* copy data and mark mailbox full */
> > +	value = (u32)(unsigned long)data;
> > +	value |= HSP_SM_SHRD_MBOX_FULL;
> > +
> > +	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
> > +
> > +	if (!irqs_disabled()) {
> > +		/* enable EMPTY interrupt for the shared mailbox */
> > +		spin_lock_irqsave(&hsp->lock, flags);
> > +
> > +		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> > +		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> > +
> > +		spin_unlock_irqrestore(&hsp->lock, flags);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int tegra_hsp_mailbox_flush(struct mbox_chan *chan,
> > +				   unsigned long timeout)
> > +{
> > +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> > +	struct tegra_hsp_channel *ch = &mb->channel;
> > +	u32 value;
> > +
> > +	timeout = jiffies + msecs_to_jiffies(timeout);
> > +
> > +	while (time_before(jiffies, timeout)) {
> > +		value = tegra_hsp_channel_readl(ch, HSP_SM_SHRD_MBOX);
> > +		if ((value & HSP_SM_SHRD_MBOX_FULL) == 0) {
> > +			mbox_chan_txdone(chan, 0);
> > +			return 0;
> > +		}
> > +
> > +		udelay(1);
> > +	}
> > +
> > +	return -ETIME;
> > +}
> > +
> > +static int tegra_hsp_mailbox_startup(struct mbox_chan *chan)
> > +{
> > +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> > +	struct tegra_hsp_channel *ch = &mb->channel;
> > +	struct tegra_hsp *hsp = mb->channel.hsp;
> > +	unsigned long flags;
> > +
> > +	chan->txdone_method = TXDONE_BY_IRQ;
> > +
> > +	/*
> > +	 * Shared mailboxes start out as consumers by default. FULL and EMPTY
> > +	 * interrupts are coalesced at the same shared interrupt.
> > +	 *
> > +	 * Keep EMPTY interrupts disabled at startup and only enable them when
> > +	 * the mailbox is actually full. This is required because the FULL and
> > +	 * EMPTY interrupts are level-triggered, so keeping EMPTY interrupts
> > +	 * enabled all the time would cause an interrupt storm while mailboxes
> > +	 * are idle.
> > +	 */
> > +
> > +	spin_lock_irqsave(&hsp->lock, flags);
> > +
> > +	if (mb->producer)
> > +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> > +	else
> > +		hsp->mask |= BIT(HSP_INT_FULL_SHIFT + mb->index);
> > +
> > +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> > +
> > +	spin_unlock_irqrestore(&hsp->lock, flags);
> > +
> > +	if (hsp->soc->has_per_mb_ie) {
> > +		if (mb->producer)
> > +			tegra_hsp_channel_writel(ch, 0x0,
> > +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
> > +		else
> > +			tegra_hsp_channel_writel(ch, 0x1,
> > +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static void tegra_hsp_mailbox_shutdown(struct mbox_chan *chan)
> > +{
> > +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> > +	struct tegra_hsp_channel *ch = &mb->channel;
> > +	struct tegra_hsp *hsp = mb->channel.hsp;
> > +	unsigned long flags;
> > +
> > +	if (hsp->soc->has_per_mb_ie) {
> > +		if (mb->producer)
> > +			tegra_hsp_channel_writel(ch, 0x0,
> > +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
> > +		else
> > +			tegra_hsp_channel_writel(ch, 0x0,
> > +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
> > +	}
> > +
> > +	spin_lock_irqsave(&hsp->lock, flags);
> > +
> > +	if (mb->producer)
> > +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> > +	else
> > +		hsp->mask &= ~BIT(HSP_INT_FULL_SHIFT + mb->index);
> > +
> > +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> > +
> > +	spin_unlock_irqrestore(&hsp->lock, flags);
> > +}
> > +
> > +static const struct mbox_chan_ops tegra_hsp_sm_ops = {
> > +	.send_data = tegra_hsp_mailbox_send_data,
> > +	.flush = tegra_hsp_mailbox_flush,
> > +	.startup = tegra_hsp_mailbox_startup,
> > +	.shutdown = tegra_hsp_mailbox_shutdown,
> > +};
> > +
> > +static struct mbox_chan *tegra_hsp_db_xlate(struct mbox_controller *mbox,
> >  					    const struct of_phandle_args *args)
> >  {
> > +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_db);
> > +	unsigned int type = args->args[0], master = args->args[1];
> >  	struct tegra_hsp_channel *channel = ERR_PTR(-ENODEV);
> > -	struct tegra_hsp *hsp = to_tegra_hsp(mbox);
> > -	unsigned int type = args->args[0];
> > -	unsigned int master = args->args[1];
> >  	struct tegra_hsp_doorbell *db;
> >  	struct mbox_chan *chan;
> >  	unsigned long flags;
> >  	unsigned int i;
> >  
> > -	switch (type) {
> > -	case TEGRA_HSP_MBOX_TYPE_DB:
> > -		db = tegra_hsp_doorbell_get(hsp, master);
> > -		if (db)
> > -			channel = &db->channel;
> > +	if (type != TEGRA_HSP_MBOX_TYPE_DB || !hsp->doorbell_irq)
> > +		return ERR_PTR(-ENODEV);
> >  
> > -		break;
> > -
> > -	default:
> > -		break;
> > -	}
> > +	db = tegra_hsp_doorbell_get(hsp, master);
> > +	if (db)
> > +		channel = &db->channel;
> >  
> >  	if (IS_ERR(channel))
> >  		return ERR_CAST(channel);
> >  
> >  	spin_lock_irqsave(&hsp->lock, flags);
> >  
> > -	for (i = 0; i < hsp->mbox.num_chans; i++) {
> > -		chan = &hsp->mbox.chans[i];
> > +	for (i = 0; i < mbox->num_chans; i++) {
> > +		chan = &mbox->chans[i];
> >  		if (!chan->con_priv) {
> > -			chan->con_priv = channel;
> >  			channel->chan = chan;
> > +			chan->con_priv = db;
> >  			break;
> >  		}
> >  
> > @@ -332,6 +544,29 @@ static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
> >  	return chan ?: ERR_PTR(-EBUSY);
> >  }
> >  
> > +static struct mbox_chan *tegra_hsp_sm_xlate(struct mbox_controller *mbox,
> > +					    const struct of_phandle_args *args)
> > +{
> > +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_sm);
> > +	unsigned int type = args->args[0], index;
> > +	struct tegra_hsp_mailbox *mb;
> > +
> > +	index = args->args[1] & TEGRA_HSP_SM_MASK;
> > +
> > +	if (type != TEGRA_HSP_MBOX_TYPE_SM || !hsp->shared_irqs ||
> > +	    index >= hsp->num_sm)
> > +		return ERR_PTR(-ENODEV);
> > +
> > +	mb = &hsp->mailboxes[index];
> > +
> > +	if ((args->args[1] & TEGRA_HSP_SM_FLAG_TX) == 0)
> > +		mb->producer = false;
> > +	else
> > +		mb->producer = true;
> > +
> > +	return mb->channel.chan;
> > +}
> > +
> >  static void tegra_hsp_remove_doorbells(struct tegra_hsp *hsp)
> >  {
> >  	struct tegra_hsp_doorbell *db, *tmp;
> > @@ -364,10 +599,65 @@ static int tegra_hsp_add_doorbells(struct tegra_hsp *hsp)
> >  	return 0;
> >  }
> >  
> > +static int tegra_hsp_add_mailboxes(struct tegra_hsp *hsp, struct device *dev)
> > +{
> > +	int i;
> > +
> > +	hsp->mailboxes = devm_kcalloc(dev, hsp->num_sm, sizeof(*hsp->mailboxes),
> > +				      GFP_KERNEL);
> > +	if (!hsp->mailboxes)
> > +		return -ENOMEM;
> > +
> > +	for (i = 0; i < hsp->num_sm; i++) {
> > +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
> > +
> > +		mb->index = i;
> > +
> > +		mb->channel.hsp = hsp;
> > +		mb->channel.regs = hsp->regs + SZ_64K + i * SZ_32K;
> > +		mb->channel.chan = &hsp->mbox_sm.chans[i];
> > +		mb->channel.chan->con_priv = mb;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int tegra_hsp_request_shared_irqs(struct tegra_hsp *hsp)
> > +{
> > +	unsigned int i, irq = 0;
> > +	int err;
> > +
> > +	for (i = 0; i < hsp->num_si; i++) {
> > +		if (hsp->shared_irq == 0 && hsp->shared_irqs[i] > 0) {
> > +			irq = hsp->shared_irqs[i];
> > +			hsp->shared_irq = i;
> > +			break;
> > +		}
> > +	}
> > +
> > +	if (irq > 0) {
> > +		err = devm_request_irq(hsp->dev, irq, tegra_hsp_shared_irq, 0,
> > +				       dev_name(hsp->dev), hsp);
> > +		if (err < 0) {
> > +			dev_err(hsp->dev, "failed to request interrupt: %d\n",
> > +				err);
> > +			return err;
> > +		}
> 
> Are we suppose to loop through all the shared interrupts and find one
> that is available? Looking at the above it seems that we will try to use
> the first shared interrupt we have a valid mapping for, regardless of if
> it is available/in-use.
> 
> Otherwise I am not sure why it is necessary to stored all the shared
> irqs, because AFAICT we only use the first we have a valid mapping for.
> Maybe I am missing something ...

Yeah, that's a good idea. In practice I don't think this matters at all
because there just isn't another user of these interrupts, but it might
be more explicit and self-explanatory if we try all of the interrupts
in turn until we can request one.

> > +
> > +		/* disable all interrupts */
> > +		tegra_hsp_writel(hsp, 0, HSP_INT_IE(hsp->shared_irq)> +
> > +		dev_dbg(hsp->dev, "interrupt requested: %u\n", irq);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> >  static int tegra_hsp_probe(struct platform_device *pdev)
> >  {
> >  	struct tegra_hsp *hsp;
> >  	struct resource *res;
> > +	unsigned int i;
> >  	u32 value;
> >  	int err;
> >  
> > @@ -375,6 +665,7 @@ static int tegra_hsp_probe(struct platform_device *pdev)
> >  	if (!hsp)
> >  		return -ENOMEM;
> >  
> > +	hsp->dev = &pdev->dev;
> >  	hsp->soc = of_device_get_match_data(&pdev->dev);
> >  	INIT_LIST_HEAD(&hsp->doorbells);
> >  	spin_lock_init(&hsp->lock);
> > @@ -392,58 +683,136 @@ static int tegra_hsp_probe(struct platform_device *pdev)
> >  	hsp->num_si = (value >> HSP_nSI_SHIFT) & HSP_nINT_MASK;
> >  
> >  	err = platform_get_irq_byname(pdev, "doorbell");
> > -	if (err < 0) {
> > -		dev_err(&pdev->dev, "failed to get doorbell IRQ: %d\n", err);
> > -		return err;
> > +	if (err >= 0)
> > +		hsp->doorbell_irq = err;
> > +
> > +	if (hsp->num_si > 0) {
> > +		unsigned int count = 0;
> > +
> > +		hsp->shared_irqs = devm_kcalloc(&pdev->dev, hsp->num_si,
> > +						sizeof(*hsp->shared_irqs),
> > +						GFP_KERNEL);
> > +		if (!hsp->shared_irqs)
> > +			return -ENOMEM;
> > +
> > +		for (i = 0; i < hsp->num_si; i++) {
> > +			char *name;
> > +
> > +			name = kasprintf(GFP_KERNEL, "shared%u", i);
> > +			if (!name)
> > +				return -ENOMEM;
> > +
> > +			err = platform_get_irq_byname(pdev, name);
> > +			if (err >= 0) {
> > +				hsp->shared_irqs[i] = err;
> > +				count++;
> > +			}
> > +
> > +			kfree(name);
> > +		}
> > +
> > +		if (count == 0) {
> > +			devm_kfree(&pdev->dev, hsp->shared_irqs);
> > +			hsp->shared_irqs = NULL;
> > +		}
> > +	}
> > +
> > +	/* setup the doorbell controller */
> > +	hsp->mbox_db.of_xlate = tegra_hsp_db_xlate;
> > +	hsp->mbox_db.num_chans = 32;
> > +	hsp->mbox_db.dev = &pdev->dev;
> > +	hsp->mbox_db.ops = &tegra_hsp_db_ops;
> > +
> > +	hsp->mbox_db.chans = devm_kcalloc(&pdev->dev, hsp->mbox_db.num_chans,
> > +					  sizeof(*hsp->mbox_db.chans),
> > +					  GFP_KERNEL);
> > +	if (!hsp->mbox_db.chans)
> > +		return -ENOMEM;
> > +
> > +	if (hsp->doorbell_irq) {
> > +		err = tegra_hsp_add_doorbells(hsp);
> > +		if (err < 0) {
> > +			dev_err(&pdev->dev, "failed to add doorbells: %d\n",
> > +			        err);
> > +			return err;
> > +		}
> >  	}
> >  
> > -	hsp->irq = err;
> > +	err = mbox_controller_register(&hsp->mbox_db);
> > +	if (err < 0) {
> > +		dev_err(&pdev->dev, "failed to register doorbell mailbox: %d\n", err);
> > +		goto remove_doorbells;
> > +	}
> >  
> > -	hsp->mbox.of_xlate = of_tegra_hsp_xlate;
> > -	hsp->mbox.num_chans = 32;
> > -	hsp->mbox.dev = &pdev->dev;
> > -	hsp->mbox.txdone_irq = false;
> > -	hsp->mbox.txdone_poll = false;
> > -	hsp->mbox.ops = &tegra_hsp_doorbell_ops;
> > +	/* setup the shared mailbox controller */
> > +	hsp->mbox_sm.of_xlate = tegra_hsp_sm_xlate;
> > +	hsp->mbox_sm.num_chans = hsp->num_sm;
> > +	hsp->mbox_sm.dev = &pdev->dev;
> > +	hsp->mbox_sm.ops = &tegra_hsp_sm_ops;
> >  
> > -	hsp->mbox.chans = devm_kcalloc(&pdev->dev, hsp->mbox.num_chans,
> > -					sizeof(*hsp->mbox.chans),
> > -					GFP_KERNEL);
> > -	if (!hsp->mbox.chans)
> > +	hsp->mbox_sm.chans = devm_kcalloc(&pdev->dev, hsp->mbox_sm.num_chans,
> > +					  sizeof(*hsp->mbox_sm.chans),
> > +					  GFP_KERNEL);
> > +	if (!hsp->mbox_sm.chans)
> >  		return -ENOMEM;
> >  
> > -	err = tegra_hsp_add_doorbells(hsp);
> > +	if (hsp->shared_irqs) {
> > +		err = tegra_hsp_add_mailboxes(hsp, &pdev->dev);
> > +		if (err < 0) {
> > +			dev_err(&pdev->dev, "failed to add mailboxes: %d\n",
> > +			        err);
> > +			goto unregister_mbox_db;
> > +		}
> > +	}
> > +
> > +	err = mbox_controller_register(&hsp->mbox_sm);
> >  	if (err < 0) {
> > -		dev_err(&pdev->dev, "failed to add doorbells: %d\n", err);
> > -		return err;
> > +		dev_err(&pdev->dev, "failed to register shared mailbox: %d\n", err);
> > +		goto unregister_mbox_db;
> >  	}
> >  
> >  	platform_set_drvdata(pdev, hsp);
> >  
> > -	err = mbox_controller_register(&hsp->mbox);
> > -	if (err) {
> > -		dev_err(&pdev->dev, "failed to register mailbox: %d\n", err);
> > -		tegra_hsp_remove_doorbells(hsp);
> > -		return err;
> > +	if (hsp->doorbell_irq) {
> > +		err = devm_request_irq(&pdev->dev, hsp->doorbell_irq,
> > +				       tegra_hsp_doorbell_irq, IRQF_NO_SUSPEND,
> > +				       dev_name(&pdev->dev), hsp);
> > +		if (err < 0) {
> > +			dev_err(&pdev->dev,
> > +			        "failed to request doorbell IRQ#%u: %d\n",
> > +				hsp->doorbell_irq, err);
> > +			goto unregister_mbox_sm;
> > +		}
> >  	}
> >  
> > -	err = devm_request_irq(&pdev->dev, hsp->irq, tegra_hsp_doorbell_irq,
> > -			       IRQF_NO_SUSPEND, dev_name(&pdev->dev), hsp);
> > -	if (err < 0) {
> > -		dev_err(&pdev->dev, "failed to request IRQ#%u: %d\n",
> > -			hsp->irq, err);
> > -		return err;
> > +	if (hsp->shared_irqs) {
> > +		err = tegra_hsp_request_shared_irqs(hsp);
> > +		if (err < 0)
> > +			goto unregister_mbox_sm;
> 
> Any reason why we don't request the doorbell and shared irqs earlier?
> Given we use devm seems it could simplify the clean and avoid
> unregistering the mailbox.

The interrupt handlers need to access the fields that are set up in the
above sequence, so if we request them earlier they may run without this
data being initialized and crash. Now, since we disable the shared
interrupts that shouldn't happen, but better to be safe than sorry.

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 04/10] mailbox: tegra-hsp: Add support for shared mailboxes

[Thierry Reding]

On Tue, Nov 13, 2018 at 11:09:22AM +0000, Jon Hunter wrote:
> 
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > The Tegra HSP block supports 'shared mailboxes' that are simple 32-bit
> > registers consisting of a FULL bit in MSB position and 31 bits of data.
> > The hardware can be configured to trigger interrupts when a mailbox
> > is empty or full. Add support for these shared mailboxes to the HSP
> > driver.
> > 
> > The initial use for the mailboxes is the Tegra Combined UART. For this
> > purpose, we use interrupts to receive data, and spinning to wait for
> > the transmit mailbox to be emptied to minimize unnecessary overhead.
> > 
> > Based on work by Mikko Perttunen <mperttunen@nvidia.com>.
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> > Changes in v2:
> > - do not write per-mailbox interrupt enable registers on Tegra186
> > - merge handlers for empty and full interrupts
> > - track direction of shared mailboxes
> > 
> >  drivers/mailbox/tegra-hsp.c | 498 +++++++++++++++++++++++++++++++-----
> >  1 file changed, 437 insertions(+), 61 deletions(-)
> > 
> > diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
> > index 0cde356c11ab..0100a974149b 100644
> > --- a/drivers/mailbox/tegra-hsp.c
> > +++ b/drivers/mailbox/tegra-hsp.c
> > @@ -1,5 +1,5 @@
> >  /*
> > - * Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.
> > + * Copyright (c) 2016-2018, NVIDIA CORPORATION.  All rights reserved.
> >   *
> >   * This program is free software; you can redistribute it and/or modify it
> >   * under the terms and conditions of the GNU General Public License,
> > @@ -11,6 +11,7 @@
> >   * more details.
> >   */
> >  
> > +#include <linux/delay.h>
> >  #include <linux/interrupt.h>
> >  #include <linux/io.h>
> >  #include <linux/mailbox_controller.h>
> > @@ -21,6 +22,17 @@
> >  
> >  #include <dt-bindings/mailbox/tegra186-hsp.h>
> >  
> > +#include "mailbox.h"
> > +
> > +#define HSP_INT_IE(x)		(0x100 + ((x) * 4))
> > +#define HSP_INT_IV		0x300
> > +#define HSP_INT_IR		0x304
> > +
> > +#define HSP_INT_EMPTY_SHIFT	0
> > +#define HSP_INT_EMPTY_MASK	0xff
> > +#define HSP_INT_FULL_SHIFT	8
> > +#define HSP_INT_FULL_MASK	0xff
> > +
> >  #define HSP_INT_DIMENSIONING	0x380
> >  #define HSP_nSM_SHIFT		0
> >  #define HSP_nSS_SHIFT		4
> > @@ -34,6 +46,11 @@
> >  #define HSP_DB_RAW	0x8
> >  #define HSP_DB_PENDING	0xc
> >  
> > +#define HSP_SM_SHRD_MBOX	0x0
> > +#define HSP_SM_SHRD_MBOX_FULL	BIT(31)
> > +#define HSP_SM_SHRD_MBOX_FULL_INT_IE	0x04
> > +#define HSP_SM_SHRD_MBOX_EMPTY_INT_IE	0x08
> > +
> >  #define HSP_DB_CCPLEX		1
> >  #define HSP_DB_BPMP		3
> >  #define HSP_DB_MAX		7
> > @@ -55,6 +72,12 @@ struct tegra_hsp_doorbell {
> >  	unsigned int index;
> >  };
> >  
> > +struct tegra_hsp_mailbox {
> > +	struct tegra_hsp_channel channel;
> > +	unsigned int index;
> > +	bool producer;
> > +};
> > +
> >  struct tegra_hsp_db_map {
> >  	const char *name;
> >  	unsigned int master;
> > @@ -63,13 +86,18 @@ struct tegra_hsp_db_map {
> >  
> >  struct tegra_hsp_soc {
> >  	const struct tegra_hsp_db_map *map;
> > +	bool has_per_mb_ie;
> >  };
> >  
> >  struct tegra_hsp {
> > +	struct device *dev;
> >  	const struct tegra_hsp_soc *soc;
> > -	struct mbox_controller mbox;
> > +	struct mbox_controller mbox_db;
> > +	struct mbox_controller mbox_sm;
> >  	void __iomem *regs;
> > -	unsigned int irq;
> > +	unsigned int doorbell_irq;
> > +	unsigned int *shared_irqs;
> > +	unsigned int shared_irq;
> >  	unsigned int num_sm;
> >  	unsigned int num_as;
> >  	unsigned int num_ss;
> > @@ -78,13 +106,10 @@ struct tegra_hsp {
> >  	spinlock_t lock;
> >  
> >  	struct list_head doorbells;
> > -};
> > +	struct tegra_hsp_mailbox *mailboxes;
> >  
> > -static inline struct tegra_hsp *
> > -to_tegra_hsp(struct mbox_controller *mbox)
> > -{
> > -	return container_of(mbox, struct tegra_hsp, mbox);
> > -}
> > +	unsigned long mask;
> > +};
> >  
> >  static inline u32 tegra_hsp_readl(struct tegra_hsp *hsp, unsigned int offset)
> >  {
> > @@ -158,7 +183,7 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
> >  
> >  	spin_lock(&hsp->lock);
> >  
> > -	for_each_set_bit(master, &value, hsp->mbox.num_chans) {
> > +	for_each_set_bit(master, &value, hsp->mbox_db.num_chans) {
> >  		struct tegra_hsp_doorbell *db;
> >  
> >  		db = __tegra_hsp_doorbell_get(hsp, master);
> > @@ -182,6 +207,71 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
> >  	return IRQ_HANDLED;
> >  }
> >  
> > +static irqreturn_t tegra_hsp_shared_irq(int irq, void *data)
> > +{
> > +	struct tegra_hsp *hsp = data;
> > +	unsigned long bit, mask;
> > +	u32 status, value;
> > +	void *msg;
> > +
> > +	status = tegra_hsp_readl(hsp, HSP_INT_IR) & hsp->mask;
> > +
> > +	/* process EMPTY interrupts first */
> > +	mask = (status >> HSP_INT_EMPTY_SHIFT) & HSP_INT_EMPTY_MASK;
> > +
> > +	for_each_set_bit(bit, &mask, hsp->num_sm) {
> > +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
> > +
> > +		if (mb->producer) {
> > +			/*
> > +			 * Disable EMPTY interrupts until data is sent with
> > +			 * the next message. These interrupts are level-
> > +			 * triggered, so if we kept them enabled they would
> > +			 * constantly trigger until we next write data into
> > +			 * the message.
> > +			 */
> > +			spin_lock(&hsp->lock);
> > +
> > +			hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> > +			tegra_hsp_writel(hsp, hsp->mask,
> > +					 HSP_INT_IE(hsp->shared_irq));
> > +
> > +			spin_unlock(&hsp->lock);
> > +
> > +			mbox_chan_txdone(mb->channel.chan, 0);
> > +		}
> > +	}
> > +
> > +	/* process FULL interrupts */
> > +	mask = (status >> HSP_INT_FULL_SHIFT) & HSP_INT_FULL_MASK;
> > +
> > +	for_each_set_bit(bit, &mask, hsp->num_sm) {
> > +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
> > +
> > +		if (!mb->producer) {
> > +			value = tegra_hsp_channel_readl(&mb->channel,
> > +							HSP_SM_SHRD_MBOX);
> > +			value &= ~HSP_SM_SHRD_MBOX_FULL;
> > +			msg = (void *)(unsigned long)value;
> > +			mbox_chan_received_data(mb->channel.chan, msg);
> > +
> > +			/*
> > +			 * Need to clear all bits here since some producers,
> > +			 * such as TCU, depend on fields in the register
> > +			 * getting cleared by the consumer.
> > +			 *
> > +			 * The mailbox API doesn't give the consumers a way
> > +			 * of doing that explicitly, so we have to make sure
> > +			 * we cover all possible cases.
> > +			 */
> > +			tegra_hsp_channel_writel(&mb->channel, 0x0,
> > +						 HSP_SM_SHRD_MBOX);
> > +		}
> > +	}
> > +
> > +	return IRQ_HANDLED;
> > +}
> > +
> >  static struct tegra_hsp_channel *
> >  tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
> >  			  unsigned int master, unsigned int index)
> > @@ -194,7 +284,7 @@ tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
> >  	if (!db)
> >  		return ERR_PTR(-ENOMEM);
> >  
> > -	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) << 16;
> > +	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) * SZ_64K;
> >  	offset += index * 0x100;
> >  
> >  	db->channel.regs = hsp->regs + offset;
> > @@ -235,8 +325,8 @@ static int tegra_hsp_doorbell_startup(struct mbox_chan *chan)
> >  	unsigned long flags;
> >  	u32 value;
> >  
> > -	if (db->master >= hsp->mbox.num_chans) {
> > -		dev_err(hsp->mbox.dev,
> > +	if (db->master >= chan->mbox->num_chans) {
> > +		dev_err(chan->mbox->dev,
> >  			"invalid master ID %u for HSP channel\n",
> >  			db->master);
> >  		return -EINVAL;
> > @@ -281,46 +371,168 @@ static void tegra_hsp_doorbell_shutdown(struct mbox_chan *chan)
> >  	spin_unlock_irqrestore(&hsp->lock, flags);
> >  }
> >  
> > -static const struct mbox_chan_ops tegra_hsp_doorbell_ops = {
> > +static const struct mbox_chan_ops tegra_hsp_db_ops = {
> >  	.send_data = tegra_hsp_doorbell_send_data,
> >  	.startup = tegra_hsp_doorbell_startup,
> >  	.shutdown = tegra_hsp_doorbell_shutdown,
> >  };
> >  
> > -static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
> > +static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
> > +{
> > +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> > +	struct tegra_hsp *hsp = mb->channel.hsp;
> > +	unsigned long flags;
> > +	u32 value;
> > +
> > +	WARN_ON(!mb->producer);
> 
> Should we return here?

Yeah, that's a good idea. I made this return -EPERM for lack of a better
error code.

> > +
> > +	/* copy data and mark mailbox full */
> > +	value = (u32)(unsigned long)data;
> > +	value |= HSP_SM_SHRD_MBOX_FULL;
> > +
> > +	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
> > +
> > +	if (!irqs_disabled()) {
> > +		/* enable EMPTY interrupt for the shared mailbox */
> > +		spin_lock_irqsave(&hsp->lock, flags);
> > +
> > +		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> > +		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> > +
> > +		spin_unlock_irqrestore(&hsp->lock, flags);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int tegra_hsp_mailbox_flush(struct mbox_chan *chan,
> > +				   unsigned long timeout)
> > +{
> > +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> > +	struct tegra_hsp_channel *ch = &mb->channel;
> > +	u32 value;
> > +
> > +	timeout = jiffies + msecs_to_jiffies(timeout);
> > +
> > +	while (time_before(jiffies, timeout)) {
> > +		value = tegra_hsp_channel_readl(ch, HSP_SM_SHRD_MBOX);
> > +		if ((value & HSP_SM_SHRD_MBOX_FULL) == 0) {
> > +			mbox_chan_txdone(chan, 0);
> > +			return 0;
> > +		}
> > +
> > +		udelay(1);
> > +	}
> > +
> > +	return -ETIME;
> > +}
> > +
> > +static int tegra_hsp_mailbox_startup(struct mbox_chan *chan)
> > +{
> > +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> > +	struct tegra_hsp_channel *ch = &mb->channel;
> > +	struct tegra_hsp *hsp = mb->channel.hsp;
> > +	unsigned long flags;
> > +
> > +	chan->txdone_method = TXDONE_BY_IRQ;
> > +
> > +	/*
> > +	 * Shared mailboxes start out as consumers by default. FULL and EMPTY
> > +	 * interrupts are coalesced at the same shared interrupt.
> > +	 *
> > +	 * Keep EMPTY interrupts disabled at startup and only enable them when
> > +	 * the mailbox is actually full. This is required because the FULL and
> > +	 * EMPTY interrupts are level-triggered, so keeping EMPTY interrupts
> > +	 * enabled all the time would cause an interrupt storm while mailboxes
> > +	 * are idle.
> > +	 */
> > +
> > +	spin_lock_irqsave(&hsp->lock, flags);
> > +
> > +	if (mb->producer)
> > +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> > +	else
> > +		hsp->mask |= BIT(HSP_INT_FULL_SHIFT + mb->index);
> > +
> > +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> > +
> > +	spin_unlock_irqrestore(&hsp->lock, flags);
> > +
> > +	if (hsp->soc->has_per_mb_ie) {
> > +		if (mb->producer)
> > +			tegra_hsp_channel_writel(ch, 0x0,
> > +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
> > +		else
> > +			tegra_hsp_channel_writel(ch, 0x1,
> > +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static void tegra_hsp_mailbox_shutdown(struct mbox_chan *chan)
> > +{
> > +	struct tegra_hsp_mailbox *mb = chan->con_priv;
> > +	struct tegra_hsp_channel *ch = &mb->channel;
> > +	struct tegra_hsp *hsp = mb->channel.hsp;
> > +	unsigned long flags;
> > +
> > +	if (hsp->soc->has_per_mb_ie) {
> > +		if (mb->producer)
> > +			tegra_hsp_channel_writel(ch, 0x0,
> > +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
> > +		else
> > +			tegra_hsp_channel_writel(ch, 0x0,
> > +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
> > +	}
> > +
> > +	spin_lock_irqsave(&hsp->lock, flags);
> > +
> > +	if (mb->producer)
> > +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
> > +	else
> > +		hsp->mask &= ~BIT(HSP_INT_FULL_SHIFT + mb->index);
> > +
> > +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
> > +
> > +	spin_unlock_irqrestore(&hsp->lock, flags);
> > +}
> > +
> > +static const struct mbox_chan_ops tegra_hsp_sm_ops = {
> > +	.send_data = tegra_hsp_mailbox_send_data,
> > +	.flush = tegra_hsp_mailbox_flush,
> > +	.startup = tegra_hsp_mailbox_startup,
> > +	.shutdown = tegra_hsp_mailbox_shutdown,
> > +};
> > +
> > +static struct mbox_chan *tegra_hsp_db_xlate(struct mbox_controller *mbox,
> >  					    const struct of_phandle_args *args)
> >  {
> > +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_db);
> > +	unsigned int type = args->args[0], master = args->args[1];
> >  	struct tegra_hsp_channel *channel = ERR_PTR(-ENODEV);
> > -	struct tegra_hsp *hsp = to_tegra_hsp(mbox);
> > -	unsigned int type = args->args[0];
> > -	unsigned int master = args->args[1];
> >  	struct tegra_hsp_doorbell *db;
> >  	struct mbox_chan *chan;
> >  	unsigned long flags;
> >  	unsigned int i;
> >  
> > -	switch (type) {
> > -	case TEGRA_HSP_MBOX_TYPE_DB:
> > -		db = tegra_hsp_doorbell_get(hsp, master);
> > -		if (db)
> > -			channel = &db->channel;
> > +	if (type != TEGRA_HSP_MBOX_TYPE_DB || !hsp->doorbell_irq)
> > +		return ERR_PTR(-ENODEV);
> >  
> > -		break;
> > -
> > -	default:
> > -		break;
> > -	}
> > +	db = tegra_hsp_doorbell_get(hsp, master);
> > +	if (db)
> > +		channel = &db->channel;
> >  
> >  	if (IS_ERR(channel))
> >  		return ERR_CAST(channel);
> >  
> >  	spin_lock_irqsave(&hsp->lock, flags);
> >  
> > -	for (i = 0; i < hsp->mbox.num_chans; i++) {
> > -		chan = &hsp->mbox.chans[i];
> > +	for (i = 0; i < mbox->num_chans; i++) {
> > +		chan = &mbox->chans[i];
> >  		if (!chan->con_priv) {
> > -			chan->con_priv = channel;
> >  			channel->chan = chan;
> > +			chan->con_priv = db;
> >  			break;
> >  		}
> >  
> > @@ -332,6 +544,29 @@ static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
> >  	return chan ?: ERR_PTR(-EBUSY);
> >  }
> >  
> > +static struct mbox_chan *tegra_hsp_sm_xlate(struct mbox_controller *mbox,
> > +					    const struct of_phandle_args *args)
> > +{
> > +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_sm);
> > +	unsigned int type = args->args[0], index;
> > +	struct tegra_hsp_mailbox *mb;
> > +
> > +	index = args->args[1] & TEGRA_HSP_SM_MASK;
> > +
> > +	if (type != TEGRA_HSP_MBOX_TYPE_SM || !hsp->shared_irqs ||
> > +	    index >= hsp->num_sm)
> > +		return ERR_PTR(-ENODEV);
> > +
> > +	mb = &hsp->mailboxes[index];
> > +
> > +	if ((args->args[1] & TEGRA_HSP_SM_FLAG_TX) == 0)
> > +		mb->producer = false;
> > +	else
> > +		mb->producer = true;
> > +
> > +	return mb->channel.chan;
> > +}
> > +
> >  static void tegra_hsp_remove_doorbells(struct tegra_hsp *hsp)
> >  {
> >  	struct tegra_hsp_doorbell *db, *tmp;
> > @@ -364,10 +599,65 @@ static int tegra_hsp_add_doorbells(struct tegra_hsp *hsp)
> >  	return 0;
> >  }
> >  
> > +static int tegra_hsp_add_mailboxes(struct tegra_hsp *hsp, struct device *dev)
> > +{
> > +	int i;
> > +
> > +	hsp->mailboxes = devm_kcalloc(dev, hsp->num_sm, sizeof(*hsp->mailboxes),
> > +				      GFP_KERNEL);
> > +	if (!hsp->mailboxes)
> > +		return -ENOMEM;
> > +
> > +	for (i = 0; i < hsp->num_sm; i++) {
> > +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
> > +
> > +		mb->index = i;
> > +
> > +		mb->channel.hsp = hsp;
> > +		mb->channel.regs = hsp->regs + SZ_64K + i * SZ_32K;
> > +		mb->channel.chan = &hsp->mbox_sm.chans[i];
> > +		mb->channel.chan->con_priv = mb;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int tegra_hsp_request_shared_irqs(struct tegra_hsp *hsp)
> > +{
> > +	unsigned int i, irq = 0;
> > +	int err;
> > +
> > +	for (i = 0; i < hsp->num_si; i++) {
> > +		if (hsp->shared_irq == 0 && hsp->shared_irqs[i] > 0) {
> > +			irq = hsp->shared_irqs[i];
> > +			hsp->shared_irq = i;
> > +			break;
> > +		}
> > +	}
> > +
> > +	if (irq > 0) {
> > +		err = devm_request_irq(hsp->dev, irq, tegra_hsp_shared_irq, 0,
> > +				       dev_name(hsp->dev), hsp);
> > +		if (err < 0) {
> > +			dev_err(hsp->dev, "failed to request interrupt: %d\n",
> > +				err);
> > +			return err;
> > +		}
> 
> Are we suppose to loop through all the shared interrupts and find one
> that is available? Looking at the above it seems that we will try to use
> the first shared interrupt we have a valid mapping for, regardless of if
> it is available/in-use.
> 
> Otherwise I am not sure why it is necessary to stored all the shared
> irqs, because AFAICT we only use the first we have a valid mapping for.
> Maybe I am missing something ...

Yeah, that's a good idea. In practice I don't think this matters at all
because there just isn't another user of these interrupts, but it might
be more explicit and self-explanatory if we try all of the interrupts
in turn until we can request one.

> > +
> > +		/* disable all interrupts */
> > +		tegra_hsp_writel(hsp, 0, HSP_INT_IE(hsp->shared_irq)> +
> > +		dev_dbg(hsp->dev, "interrupt requested: %u\n", irq);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> >  static int tegra_hsp_probe(struct platform_device *pdev)
> >  {
> >  	struct tegra_hsp *hsp;
> >  	struct resource *res;
> > +	unsigned int i;
> >  	u32 value;
> >  	int err;
> >  
> > @@ -375,6 +665,7 @@ static int tegra_hsp_probe(struct platform_device *pdev)
> >  	if (!hsp)
> >  		return -ENOMEM;
> >  
> > +	hsp->dev = &pdev->dev;
> >  	hsp->soc = of_device_get_match_data(&pdev->dev);
> >  	INIT_LIST_HEAD(&hsp->doorbells);
> >  	spin_lock_init(&hsp->lock);
> > @@ -392,58 +683,136 @@ static int tegra_hsp_probe(struct platform_device *pdev)
> >  	hsp->num_si = (value >> HSP_nSI_SHIFT) & HSP_nINT_MASK;
> >  
> >  	err = platform_get_irq_byname(pdev, "doorbell");
> > -	if (err < 0) {
> > -		dev_err(&pdev->dev, "failed to get doorbell IRQ: %d\n", err);
> > -		return err;
> > +	if (err >= 0)
> > +		hsp->doorbell_irq = err;
> > +
> > +	if (hsp->num_si > 0) {
> > +		unsigned int count = 0;
> > +
> > +		hsp->shared_irqs = devm_kcalloc(&pdev->dev, hsp->num_si,
> > +						sizeof(*hsp->shared_irqs),
> > +						GFP_KERNEL);
> > +		if (!hsp->shared_irqs)
> > +			return -ENOMEM;
> > +
> > +		for (i = 0; i < hsp->num_si; i++) {
> > +			char *name;
> > +
> > +			name = kasprintf(GFP_KERNEL, "shared%u", i);
> > +			if (!name)
> > +				return -ENOMEM;
> > +
> > +			err = platform_get_irq_byname(pdev, name);
> > +			if (err >= 0) {
> > +				hsp->shared_irqs[i] = err;
> > +				count++;
> > +			}
> > +
> > +			kfree(name);
> > +		}
> > +
> > +		if (count == 0) {
> > +			devm_kfree(&pdev->dev, hsp->shared_irqs);
> > +			hsp->shared_irqs = NULL;
> > +		}
> > +	}
> > +
> > +	/* setup the doorbell controller */
> > +	hsp->mbox_db.of_xlate = tegra_hsp_db_xlate;
> > +	hsp->mbox_db.num_chans = 32;
> > +	hsp->mbox_db.dev = &pdev->dev;
> > +	hsp->mbox_db.ops = &tegra_hsp_db_ops;
> > +
> > +	hsp->mbox_db.chans = devm_kcalloc(&pdev->dev, hsp->mbox_db.num_chans,
> > +					  sizeof(*hsp->mbox_db.chans),
> > +					  GFP_KERNEL);
> > +	if (!hsp->mbox_db.chans)
> > +		return -ENOMEM;
> > +
> > +	if (hsp->doorbell_irq) {
> > +		err = tegra_hsp_add_doorbells(hsp);
> > +		if (err < 0) {
> > +			dev_err(&pdev->dev, "failed to add doorbells: %d\n",
> > +			        err);
> > +			return err;
> > +		}
> >  	}
> >  
> > -	hsp->irq = err;
> > +	err = mbox_controller_register(&hsp->mbox_db);
> > +	if (err < 0) {
> > +		dev_err(&pdev->dev, "failed to register doorbell mailbox: %d\n", err);
> > +		goto remove_doorbells;
> > +	}
> >  
> > -	hsp->mbox.of_xlate = of_tegra_hsp_xlate;
> > -	hsp->mbox.num_chans = 32;
> > -	hsp->mbox.dev = &pdev->dev;
> > -	hsp->mbox.txdone_irq = false;
> > -	hsp->mbox.txdone_poll = false;
> > -	hsp->mbox.ops = &tegra_hsp_doorbell_ops;
> > +	/* setup the shared mailbox controller */
> > +	hsp->mbox_sm.of_xlate = tegra_hsp_sm_xlate;
> > +	hsp->mbox_sm.num_chans = hsp->num_sm;
> > +	hsp->mbox_sm.dev = &pdev->dev;
> > +	hsp->mbox_sm.ops = &tegra_hsp_sm_ops;
> >  
> > -	hsp->mbox.chans = devm_kcalloc(&pdev->dev, hsp->mbox.num_chans,
> > -					sizeof(*hsp->mbox.chans),
> > -					GFP_KERNEL);
> > -	if (!hsp->mbox.chans)
> > +	hsp->mbox_sm.chans = devm_kcalloc(&pdev->dev, hsp->mbox_sm.num_chans,
> > +					  sizeof(*hsp->mbox_sm.chans),
> > +					  GFP_KERNEL);
> > +	if (!hsp->mbox_sm.chans)
> >  		return -ENOMEM;
> >  
> > -	err = tegra_hsp_add_doorbells(hsp);
> > +	if (hsp->shared_irqs) {
> > +		err = tegra_hsp_add_mailboxes(hsp, &pdev->dev);
> > +		if (err < 0) {
> > +			dev_err(&pdev->dev, "failed to add mailboxes: %d\n",
> > +			        err);
> > +			goto unregister_mbox_db;
> > +		}
> > +	}
> > +
> > +	err = mbox_controller_register(&hsp->mbox_sm);
> >  	if (err < 0) {
> > -		dev_err(&pdev->dev, "failed to add doorbells: %d\n", err);
> > -		return err;
> > +		dev_err(&pdev->dev, "failed to register shared mailbox: %d\n", err);
> > +		goto unregister_mbox_db;
> >  	}
> >  
> >  	platform_set_drvdata(pdev, hsp);
> >  
> > -	err = mbox_controller_register(&hsp->mbox);
> > -	if (err) {
> > -		dev_err(&pdev->dev, "failed to register mailbox: %d\n", err);
> > -		tegra_hsp_remove_doorbells(hsp);
> > -		return err;
> > +	if (hsp->doorbell_irq) {
> > +		err = devm_request_irq(&pdev->dev, hsp->doorbell_irq,
> > +				       tegra_hsp_doorbell_irq, IRQF_NO_SUSPEND,
> > +				       dev_name(&pdev->dev), hsp);
> > +		if (err < 0) {
> > +			dev_err(&pdev->dev,
> > +			        "failed to request doorbell IRQ#%u: %d\n",
> > +				hsp->doorbell_irq, err);
> > +			goto unregister_mbox_sm;
> > +		}
> >  	}
> >  
> > -	err = devm_request_irq(&pdev->dev, hsp->irq, tegra_hsp_doorbell_irq,
> > -			       IRQF_NO_SUSPEND, dev_name(&pdev->dev), hsp);
> > -	if (err < 0) {
> > -		dev_err(&pdev->dev, "failed to request IRQ#%u: %d\n",
> > -			hsp->irq, err);
> > -		return err;
> > +	if (hsp->shared_irqs) {
> > +		err = tegra_hsp_request_shared_irqs(hsp);
> > +		if (err < 0)
> > +			goto unregister_mbox_sm;
> 
> Any reason why we don't request the doorbell and shared irqs earlier?
> Given we use devm seems it could simplify the clean and avoid
> unregistering the mailbox.

The interrupt handlers need to access the fields that are set up in the
above sequence, so if we request them earlier they may run without this
data being initialized and crash. Now, since we disable the shared
interrupts that shouldn't happen, but better to be safe than sorry.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181113/8031dc3f/attachment-0001.sig>

Re: [PATCH v2 04/10] mailbox: tegra-hsp: Add support for shared mailboxes

[Jon Hunter]

On 13/11/2018 13:09, Thierry Reding wrote:
> On Tue, Nov 13, 2018 at 11:09:22AM +0000, Jon Hunter wrote:
>>
>> On 12/11/2018 15:18, Thierry Reding wrote:
>>> From: Thierry Reding <treding@nvidia.com>
>>>
>>> The Tegra HSP block supports 'shared mailboxes' that are simple 32-bit
>>> registers consisting of a FULL bit in MSB position and 31 bits of data.
>>> The hardware can be configured to trigger interrupts when a mailbox
>>> is empty or full. Add support for these shared mailboxes to the HSP
>>> driver.
>>>
>>> The initial use for the mailboxes is the Tegra Combined UART. For this
>>> purpose, we use interrupts to receive data, and spinning to wait for
>>> the transmit mailbox to be emptied to minimize unnecessary overhead.
>>>
>>> Based on work by Mikko Perttunen <mperttunen@nvidia.com>.
>>>
>>> Signed-off-by: Thierry Reding <treding@nvidia.com>
>>> ---
>>> Changes in v2:
>>> - do not write per-mailbox interrupt enable registers on Tegra186
>>> - merge handlers for empty and full interrupts
>>> - track direction of shared mailboxes
>>>
>>>  drivers/mailbox/tegra-hsp.c | 498 +++++++++++++++++++++++++++++++-----
>>>  1 file changed, 437 insertions(+), 61 deletions(-)
>>>
>>> diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
>>> index 0cde356c11ab..0100a974149b 100644
>>> --- a/drivers/mailbox/tegra-hsp.c
>>> +++ b/drivers/mailbox/tegra-hsp.c
>>> @@ -1,5 +1,5 @@
>>>  /*
>>> - * Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.
>>> + * Copyright (c) 2016-2018, NVIDIA CORPORATION.  All rights reserved.
>>>   *
>>>   * This program is free software; you can redistribute it and/or modify it
>>>   * under the terms and conditions of the GNU General Public License,
>>> @@ -11,6 +11,7 @@
>>>   * more details.
>>>   */
>>>  
>>> +#include <linux/delay.h>
>>>  #include <linux/interrupt.h>
>>>  #include <linux/io.h>
>>>  #include <linux/mailbox_controller.h>
>>> @@ -21,6 +22,17 @@
>>>  
>>>  #include <dt-bindings/mailbox/tegra186-hsp.h>
>>>  
>>> +#include "mailbox.h"
>>> +
>>> +#define HSP_INT_IE(x)		(0x100 + ((x) * 4))
>>> +#define HSP_INT_IV		0x300
>>> +#define HSP_INT_IR		0x304
>>> +
>>> +#define HSP_INT_EMPTY_SHIFT	0
>>> +#define HSP_INT_EMPTY_MASK	0xff
>>> +#define HSP_INT_FULL_SHIFT	8
>>> +#define HSP_INT_FULL_MASK	0xff
>>> +
>>>  #define HSP_INT_DIMENSIONING	0x380
>>>  #define HSP_nSM_SHIFT		0
>>>  #define HSP_nSS_SHIFT		4
>>> @@ -34,6 +46,11 @@
>>>  #define HSP_DB_RAW	0x8
>>>  #define HSP_DB_PENDING	0xc
>>>  
>>> +#define HSP_SM_SHRD_MBOX	0x0
>>> +#define HSP_SM_SHRD_MBOX_FULL	BIT(31)
>>> +#define HSP_SM_SHRD_MBOX_FULL_INT_IE	0x04
>>> +#define HSP_SM_SHRD_MBOX_EMPTY_INT_IE	0x08
>>> +
>>>  #define HSP_DB_CCPLEX		1
>>>  #define HSP_DB_BPMP		3
>>>  #define HSP_DB_MAX		7
>>> @@ -55,6 +72,12 @@ struct tegra_hsp_doorbell {
>>>  	unsigned int index;
>>>  };
>>>  
>>> +struct tegra_hsp_mailbox {
>>> +	struct tegra_hsp_channel channel;
>>> +	unsigned int index;
>>> +	bool producer;
>>> +};
>>> +
>>>  struct tegra_hsp_db_map {
>>>  	const char *name;
>>>  	unsigned int master;
>>> @@ -63,13 +86,18 @@ struct tegra_hsp_db_map {
>>>  
>>>  struct tegra_hsp_soc {
>>>  	const struct tegra_hsp_db_map *map;
>>> +	bool has_per_mb_ie;
>>>  };
>>>  
>>>  struct tegra_hsp {
>>> +	struct device *dev;
>>>  	const struct tegra_hsp_soc *soc;
>>> -	struct mbox_controller mbox;
>>> +	struct mbox_controller mbox_db;
>>> +	struct mbox_controller mbox_sm;
>>>  	void __iomem *regs;
>>> -	unsigned int irq;
>>> +	unsigned int doorbell_irq;
>>> +	unsigned int *shared_irqs;
>>> +	unsigned int shared_irq;
>>>  	unsigned int num_sm;
>>>  	unsigned int num_as;
>>>  	unsigned int num_ss;
>>> @@ -78,13 +106,10 @@ struct tegra_hsp {
>>>  	spinlock_t lock;
>>>  
>>>  	struct list_head doorbells;
>>> -};
>>> +	struct tegra_hsp_mailbox *mailboxes;
>>>  
>>> -static inline struct tegra_hsp *
>>> -to_tegra_hsp(struct mbox_controller *mbox)
>>> -{
>>> -	return container_of(mbox, struct tegra_hsp, mbox);
>>> -}
>>> +	unsigned long mask;
>>> +};
>>>  
>>>  static inline u32 tegra_hsp_readl(struct tegra_hsp *hsp, unsigned int offset)
>>>  {
>>> @@ -158,7 +183,7 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
>>>  
>>>  	spin_lock(&hsp->lock);
>>>  
>>> -	for_each_set_bit(master, &value, hsp->mbox.num_chans) {
>>> +	for_each_set_bit(master, &value, hsp->mbox_db.num_chans) {
>>>  		struct tegra_hsp_doorbell *db;
>>>  
>>>  		db = __tegra_hsp_doorbell_get(hsp, master);
>>> @@ -182,6 +207,71 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
>>>  	return IRQ_HANDLED;
>>>  }
>>>  
>>> +static irqreturn_t tegra_hsp_shared_irq(int irq, void *data)
>>> +{
>>> +	struct tegra_hsp *hsp = data;
>>> +	unsigned long bit, mask;
>>> +	u32 status, value;
>>> +	void *msg;
>>> +
>>> +	status = tegra_hsp_readl(hsp, HSP_INT_IR) & hsp->mask;
>>> +
>>> +	/* process EMPTY interrupts first */
>>> +	mask = (status >> HSP_INT_EMPTY_SHIFT) & HSP_INT_EMPTY_MASK;
>>> +
>>> +	for_each_set_bit(bit, &mask, hsp->num_sm) {
>>> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
>>> +
>>> +		if (mb->producer) {
>>> +			/*
>>> +			 * Disable EMPTY interrupts until data is sent with
>>> +			 * the next message. These interrupts are level-
>>> +			 * triggered, so if we kept them enabled they would
>>> +			 * constantly trigger until we next write data into
>>> +			 * the message.
>>> +			 */
>>> +			spin_lock(&hsp->lock);
>>> +
>>> +			hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
>>> +			tegra_hsp_writel(hsp, hsp->mask,
>>> +					 HSP_INT_IE(hsp->shared_irq));
>>> +
>>> +			spin_unlock(&hsp->lock);
>>> +
>>> +			mbox_chan_txdone(mb->channel.chan, 0);
>>> +		}
>>> +	}
>>> +
>>> +	/* process FULL interrupts */
>>> +	mask = (status >> HSP_INT_FULL_SHIFT) & HSP_INT_FULL_MASK;
>>> +
>>> +	for_each_set_bit(bit, &mask, hsp->num_sm) {
>>> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
>>> +
>>> +		if (!mb->producer) {
>>> +			value = tegra_hsp_channel_readl(&mb->channel,
>>> +							HSP_SM_SHRD_MBOX);
>>> +			value &= ~HSP_SM_SHRD_MBOX_FULL;
>>> +			msg = (void *)(unsigned long)value;
>>> +			mbox_chan_received_data(mb->channel.chan, msg);
>>> +
>>> +			/*
>>> +			 * Need to clear all bits here since some producers,
>>> +			 * such as TCU, depend on fields in the register
>>> +			 * getting cleared by the consumer.
>>> +			 *
>>> +			 * The mailbox API doesn't give the consumers a way
>>> +			 * of doing that explicitly, so we have to make sure
>>> +			 * we cover all possible cases.
>>> +			 */
>>> +			tegra_hsp_channel_writel(&mb->channel, 0x0,
>>> +						 HSP_SM_SHRD_MBOX);
>>> +		}
>>> +	}
>>> +
>>> +	return IRQ_HANDLED;
>>> +}
>>> +
>>>  static struct tegra_hsp_channel *
>>>  tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
>>>  			  unsigned int master, unsigned int index)
>>> @@ -194,7 +284,7 @@ tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
>>>  	if (!db)
>>>  		return ERR_PTR(-ENOMEM);
>>>  
>>> -	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) << 16;
>>> +	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) * SZ_64K;
>>>  	offset += index * 0x100;
>>>  
>>>  	db->channel.regs = hsp->regs + offset;
>>> @@ -235,8 +325,8 @@ static int tegra_hsp_doorbell_startup(struct mbox_chan *chan)
>>>  	unsigned long flags;
>>>  	u32 value;
>>>  
>>> -	if (db->master >= hsp->mbox.num_chans) {
>>> -		dev_err(hsp->mbox.dev,
>>> +	if (db->master >= chan->mbox->num_chans) {
>>> +		dev_err(chan->mbox->dev,
>>>  			"invalid master ID %u for HSP channel\n",
>>>  			db->master);
>>>  		return -EINVAL;
>>> @@ -281,46 +371,168 @@ static void tegra_hsp_doorbell_shutdown(struct mbox_chan *chan)
>>>  	spin_unlock_irqrestore(&hsp->lock, flags);
>>>  }
>>>  
>>> -static const struct mbox_chan_ops tegra_hsp_doorbell_ops = {
>>> +static const struct mbox_chan_ops tegra_hsp_db_ops = {
>>>  	.send_data = tegra_hsp_doorbell_send_data,
>>>  	.startup = tegra_hsp_doorbell_startup,
>>>  	.shutdown = tegra_hsp_doorbell_shutdown,
>>>  };
>>>  
>>> -static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
>>> +static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
>>> +{
>>> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
>>> +	struct tegra_hsp *hsp = mb->channel.hsp;
>>> +	unsigned long flags;
>>> +	u32 value;
>>> +
>>> +	WARN_ON(!mb->producer);
>>
>> Should we return here?
> 
> Yeah, that's a good idea. I made this return -EPERM for lack of a better
> error code.
> 
>>> +
>>> +	/* copy data and mark mailbox full */
>>> +	value = (u32)(unsigned long)data;
>>> +	value |= HSP_SM_SHRD_MBOX_FULL;
>>> +
>>> +	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
>>> +
>>> +	if (!irqs_disabled()) {
>>> +		/* enable EMPTY interrupt for the shared mailbox */
>>> +		spin_lock_irqsave(&hsp->lock, flags);
>>> +
>>> +		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
>>> +		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
>>> +
>>> +		spin_unlock_irqrestore(&hsp->lock, flags);
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int tegra_hsp_mailbox_flush(struct mbox_chan *chan,
>>> +				   unsigned long timeout)
>>> +{
>>> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
>>> +	struct tegra_hsp_channel *ch = &mb->channel;
>>> +	u32 value;
>>> +
>>> +	timeout = jiffies + msecs_to_jiffies(timeout);
>>> +
>>> +	while (time_before(jiffies, timeout)) {
>>> +		value = tegra_hsp_channel_readl(ch, HSP_SM_SHRD_MBOX);
>>> +		if ((value & HSP_SM_SHRD_MBOX_FULL) == 0) {
>>> +			mbox_chan_txdone(chan, 0);
>>> +			return 0;
>>> +		}
>>> +
>>> +		udelay(1);
>>> +	}
>>> +
>>> +	return -ETIME;
>>> +}
>>> +
>>> +static int tegra_hsp_mailbox_startup(struct mbox_chan *chan)
>>> +{
>>> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
>>> +	struct tegra_hsp_channel *ch = &mb->channel;
>>> +	struct tegra_hsp *hsp = mb->channel.hsp;
>>> +	unsigned long flags;
>>> +
>>> +	chan->txdone_method = TXDONE_BY_IRQ;
>>> +
>>> +	/*
>>> +	 * Shared mailboxes start out as consumers by default. FULL and EMPTY
>>> +	 * interrupts are coalesced at the same shared interrupt.
>>> +	 *
>>> +	 * Keep EMPTY interrupts disabled at startup and only enable them when
>>> +	 * the mailbox is actually full. This is required because the FULL and
>>> +	 * EMPTY interrupts are level-triggered, so keeping EMPTY interrupts
>>> +	 * enabled all the time would cause an interrupt storm while mailboxes
>>> +	 * are idle.
>>> +	 */
>>> +
>>> +	spin_lock_irqsave(&hsp->lock, flags);
>>> +
>>> +	if (mb->producer)
>>> +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
>>> +	else
>>> +		hsp->mask |= BIT(HSP_INT_FULL_SHIFT + mb->index);
>>> +
>>> +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
>>> +
>>> +	spin_unlock_irqrestore(&hsp->lock, flags);
>>> +
>>> +	if (hsp->soc->has_per_mb_ie) {
>>> +		if (mb->producer)
>>> +			tegra_hsp_channel_writel(ch, 0x0,
>>> +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
>>> +		else
>>> +			tegra_hsp_channel_writel(ch, 0x1,
>>> +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static void tegra_hsp_mailbox_shutdown(struct mbox_chan *chan)
>>> +{
>>> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
>>> +	struct tegra_hsp_channel *ch = &mb->channel;
>>> +	struct tegra_hsp *hsp = mb->channel.hsp;
>>> +	unsigned long flags;
>>> +
>>> +	if (hsp->soc->has_per_mb_ie) {
>>> +		if (mb->producer)
>>> +			tegra_hsp_channel_writel(ch, 0x0,
>>> +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
>>> +		else
>>> +			tegra_hsp_channel_writel(ch, 0x0,
>>> +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
>>> +	}
>>> +
>>> +	spin_lock_irqsave(&hsp->lock, flags);
>>> +
>>> +	if (mb->producer)
>>> +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
>>> +	else
>>> +		hsp->mask &= ~BIT(HSP_INT_FULL_SHIFT + mb->index);
>>> +
>>> +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
>>> +
>>> +	spin_unlock_irqrestore(&hsp->lock, flags);
>>> +}
>>> +
>>> +static const struct mbox_chan_ops tegra_hsp_sm_ops = {
>>> +	.send_data = tegra_hsp_mailbox_send_data,
>>> +	.flush = tegra_hsp_mailbox_flush,
>>> +	.startup = tegra_hsp_mailbox_startup,
>>> +	.shutdown = tegra_hsp_mailbox_shutdown,
>>> +};
>>> +
>>> +static struct mbox_chan *tegra_hsp_db_xlate(struct mbox_controller *mbox,
>>>  					    const struct of_phandle_args *args)
>>>  {
>>> +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_db);
>>> +	unsigned int type = args->args[0], master = args->args[1];
>>>  	struct tegra_hsp_channel *channel = ERR_PTR(-ENODEV);
>>> -	struct tegra_hsp *hsp = to_tegra_hsp(mbox);
>>> -	unsigned int type = args->args[0];
>>> -	unsigned int master = args->args[1];
>>>  	struct tegra_hsp_doorbell *db;
>>>  	struct mbox_chan *chan;
>>>  	unsigned long flags;
>>>  	unsigned int i;
>>>  
>>> -	switch (type) {
>>> -	case TEGRA_HSP_MBOX_TYPE_DB:
>>> -		db = tegra_hsp_doorbell_get(hsp, master);
>>> -		if (db)
>>> -			channel = &db->channel;
>>> +	if (type != TEGRA_HSP_MBOX_TYPE_DB || !hsp->doorbell_irq)
>>> +		return ERR_PTR(-ENODEV);
>>>  
>>> -		break;
>>> -
>>> -	default:
>>> -		break;
>>> -	}
>>> +	db = tegra_hsp_doorbell_get(hsp, master);
>>> +	if (db)
>>> +		channel = &db->channel;
>>>  
>>>  	if (IS_ERR(channel))
>>>  		return ERR_CAST(channel);
>>>  
>>>  	spin_lock_irqsave(&hsp->lock, flags);
>>>  
>>> -	for (i = 0; i < hsp->mbox.num_chans; i++) {
>>> -		chan = &hsp->mbox.chans[i];
>>> +	for (i = 0; i < mbox->num_chans; i++) {
>>> +		chan = &mbox->chans[i];
>>>  		if (!chan->con_priv) {
>>> -			chan->con_priv = channel;
>>>  			channel->chan = chan;
>>> +			chan->con_priv = db;
>>>  			break;
>>>  		}
>>>  
>>> @@ -332,6 +544,29 @@ static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
>>>  	return chan ?: ERR_PTR(-EBUSY);
>>>  }
>>>  
>>> +static struct mbox_chan *tegra_hsp_sm_xlate(struct mbox_controller *mbox,
>>> +					    const struct of_phandle_args *args)
>>> +{
>>> +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_sm);
>>> +	unsigned int type = args->args[0], index;
>>> +	struct tegra_hsp_mailbox *mb;
>>> +
>>> +	index = args->args[1] & TEGRA_HSP_SM_MASK;
>>> +
>>> +	if (type != TEGRA_HSP_MBOX_TYPE_SM || !hsp->shared_irqs ||
>>> +	    index >= hsp->num_sm)
>>> +		return ERR_PTR(-ENODEV);
>>> +
>>> +	mb = &hsp->mailboxes[index];
>>> +
>>> +	if ((args->args[1] & TEGRA_HSP_SM_FLAG_TX) == 0)
>>> +		mb->producer = false;
>>> +	else
>>> +		mb->producer = true;
>>> +
>>> +	return mb->channel.chan;
>>> +}
>>> +
>>>  static void tegra_hsp_remove_doorbells(struct tegra_hsp *hsp)
>>>  {
>>>  	struct tegra_hsp_doorbell *db, *tmp;
>>> @@ -364,10 +599,65 @@ static int tegra_hsp_add_doorbells(struct tegra_hsp *hsp)
>>>  	return 0;
>>>  }
>>>  
>>> +static int tegra_hsp_add_mailboxes(struct tegra_hsp *hsp, struct device *dev)
>>> +{
>>> +	int i;
>>> +
>>> +	hsp->mailboxes = devm_kcalloc(dev, hsp->num_sm, sizeof(*hsp->mailboxes),
>>> +				      GFP_KERNEL);
>>> +	if (!hsp->mailboxes)
>>> +		return -ENOMEM;
>>> +
>>> +	for (i = 0; i < hsp->num_sm; i++) {
>>> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
>>> +
>>> +		mb->index = i;
>>> +
>>> +		mb->channel.hsp = hsp;
>>> +		mb->channel.regs = hsp->regs + SZ_64K + i * SZ_32K;
>>> +		mb->channel.chan = &hsp->mbox_sm.chans[i];
>>> +		mb->channel.chan->con_priv = mb;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int tegra_hsp_request_shared_irqs(struct tegra_hsp *hsp)
>>> +{
>>> +	unsigned int i, irq = 0;
>>> +	int err;
>>> +
>>> +	for (i = 0; i < hsp->num_si; i++) {
>>> +		if (hsp->shared_irq == 0 && hsp->shared_irqs[i] > 0) {
>>> +			irq = hsp->shared_irqs[i];
>>> +			hsp->shared_irq = i;
>>> +			break;
>>> +		}
>>> +	}
>>> +
>>> +	if (irq > 0) {
>>> +		err = devm_request_irq(hsp->dev, irq, tegra_hsp_shared_irq, 0,
>>> +				       dev_name(hsp->dev), hsp);
>>> +		if (err < 0) {
>>> +			dev_err(hsp->dev, "failed to request interrupt: %d\n",
>>> +				err);
>>> +			return err;
>>> +		}
>>
>> Are we suppose to loop through all the shared interrupts and find one
>> that is available? Looking at the above it seems that we will try to use
>> the first shared interrupt we have a valid mapping for, regardless of if
>> it is available/in-use.
>>
>> Otherwise I am not sure why it is necessary to stored all the shared
>> irqs, because AFAICT we only use the first we have a valid mapping for.
>> Maybe I am missing something ...
> 
> Yeah, that's a good idea. In practice I don't think this matters at all
> because there just isn't another user of these interrupts, but it might
> be more explicit and self-explanatory if we try all of the interrupts
> in turn until we can request one.

By the way, any reason why we could not put the call to
platform_get_irq_byname() in the above function? May simplify the code a
bit to have a single loop.

Cheers
Jon

-- 
nvpublic

[PATCH v2 04/10] mailbox: tegra-hsp: Add support for shared mailboxes

[Jon Hunter]

On 13/11/2018 13:09, Thierry Reding wrote:
> On Tue, Nov 13, 2018 at 11:09:22AM +0000, Jon Hunter wrote:
>>
>> On 12/11/2018 15:18, Thierry Reding wrote:
>>> From: Thierry Reding <treding@nvidia.com>
>>>
>>> The Tegra HSP block supports 'shared mailboxes' that are simple 32-bit
>>> registers consisting of a FULL bit in MSB position and 31 bits of data.
>>> The hardware can be configured to trigger interrupts when a mailbox
>>> is empty or full. Add support for these shared mailboxes to the HSP
>>> driver.
>>>
>>> The initial use for the mailboxes is the Tegra Combined UART. For this
>>> purpose, we use interrupts to receive data, and spinning to wait for
>>> the transmit mailbox to be emptied to minimize unnecessary overhead.
>>>
>>> Based on work by Mikko Perttunen <mperttunen@nvidia.com>.
>>>
>>> Signed-off-by: Thierry Reding <treding@nvidia.com>
>>> ---
>>> Changes in v2:
>>> - do not write per-mailbox interrupt enable registers on Tegra186
>>> - merge handlers for empty and full interrupts
>>> - track direction of shared mailboxes
>>>
>>>  drivers/mailbox/tegra-hsp.c | 498 +++++++++++++++++++++++++++++++-----
>>>  1 file changed, 437 insertions(+), 61 deletions(-)
>>>
>>> diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
>>> index 0cde356c11ab..0100a974149b 100644
>>> --- a/drivers/mailbox/tegra-hsp.c
>>> +++ b/drivers/mailbox/tegra-hsp.c
>>> @@ -1,5 +1,5 @@
>>>  /*
>>> - * Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.
>>> + * Copyright (c) 2016-2018, NVIDIA CORPORATION.  All rights reserved.
>>>   *
>>>   * This program is free software; you can redistribute it and/or modify it
>>>   * under the terms and conditions of the GNU General Public License,
>>> @@ -11,6 +11,7 @@
>>>   * more details.
>>>   */
>>>  
>>> +#include <linux/delay.h>
>>>  #include <linux/interrupt.h>
>>>  #include <linux/io.h>
>>>  #include <linux/mailbox_controller.h>
>>> @@ -21,6 +22,17 @@
>>>  
>>>  #include <dt-bindings/mailbox/tegra186-hsp.h>
>>>  
>>> +#include "mailbox.h"
>>> +
>>> +#define HSP_INT_IE(x)		(0x100 + ((x) * 4))
>>> +#define HSP_INT_IV		0x300
>>> +#define HSP_INT_IR		0x304
>>> +
>>> +#define HSP_INT_EMPTY_SHIFT	0
>>> +#define HSP_INT_EMPTY_MASK	0xff
>>> +#define HSP_INT_FULL_SHIFT	8
>>> +#define HSP_INT_FULL_MASK	0xff
>>> +
>>>  #define HSP_INT_DIMENSIONING	0x380
>>>  #define HSP_nSM_SHIFT		0
>>>  #define HSP_nSS_SHIFT		4
>>> @@ -34,6 +46,11 @@
>>>  #define HSP_DB_RAW	0x8
>>>  #define HSP_DB_PENDING	0xc
>>>  
>>> +#define HSP_SM_SHRD_MBOX	0x0
>>> +#define HSP_SM_SHRD_MBOX_FULL	BIT(31)
>>> +#define HSP_SM_SHRD_MBOX_FULL_INT_IE	0x04
>>> +#define HSP_SM_SHRD_MBOX_EMPTY_INT_IE	0x08
>>> +
>>>  #define HSP_DB_CCPLEX		1
>>>  #define HSP_DB_BPMP		3
>>>  #define HSP_DB_MAX		7
>>> @@ -55,6 +72,12 @@ struct tegra_hsp_doorbell {
>>>  	unsigned int index;
>>>  };
>>>  
>>> +struct tegra_hsp_mailbox {
>>> +	struct tegra_hsp_channel channel;
>>> +	unsigned int index;
>>> +	bool producer;
>>> +};
>>> +
>>>  struct tegra_hsp_db_map {
>>>  	const char *name;
>>>  	unsigned int master;
>>> @@ -63,13 +86,18 @@ struct tegra_hsp_db_map {
>>>  
>>>  struct tegra_hsp_soc {
>>>  	const struct tegra_hsp_db_map *map;
>>> +	bool has_per_mb_ie;
>>>  };
>>>  
>>>  struct tegra_hsp {
>>> +	struct device *dev;
>>>  	const struct tegra_hsp_soc *soc;
>>> -	struct mbox_controller mbox;
>>> +	struct mbox_controller mbox_db;
>>> +	struct mbox_controller mbox_sm;
>>>  	void __iomem *regs;
>>> -	unsigned int irq;
>>> +	unsigned int doorbell_irq;
>>> +	unsigned int *shared_irqs;
>>> +	unsigned int shared_irq;
>>>  	unsigned int num_sm;
>>>  	unsigned int num_as;
>>>  	unsigned int num_ss;
>>> @@ -78,13 +106,10 @@ struct tegra_hsp {
>>>  	spinlock_t lock;
>>>  
>>>  	struct list_head doorbells;
>>> -};
>>> +	struct tegra_hsp_mailbox *mailboxes;
>>>  
>>> -static inline struct tegra_hsp *
>>> -to_tegra_hsp(struct mbox_controller *mbox)
>>> -{
>>> -	return container_of(mbox, struct tegra_hsp, mbox);
>>> -}
>>> +	unsigned long mask;
>>> +};
>>>  
>>>  static inline u32 tegra_hsp_readl(struct tegra_hsp *hsp, unsigned int offset)
>>>  {
>>> @@ -158,7 +183,7 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
>>>  
>>>  	spin_lock(&hsp->lock);
>>>  
>>> -	for_each_set_bit(master, &value, hsp->mbox.num_chans) {
>>> +	for_each_set_bit(master, &value, hsp->mbox_db.num_chans) {
>>>  		struct tegra_hsp_doorbell *db;
>>>  
>>>  		db = __tegra_hsp_doorbell_get(hsp, master);
>>> @@ -182,6 +207,71 @@ static irqreturn_t tegra_hsp_doorbell_irq(int irq, void *data)
>>>  	return IRQ_HANDLED;
>>>  }
>>>  
>>> +static irqreturn_t tegra_hsp_shared_irq(int irq, void *data)
>>> +{
>>> +	struct tegra_hsp *hsp = data;
>>> +	unsigned long bit, mask;
>>> +	u32 status, value;
>>> +	void *msg;
>>> +
>>> +	status = tegra_hsp_readl(hsp, HSP_INT_IR) & hsp->mask;
>>> +
>>> +	/* process EMPTY interrupts first */
>>> +	mask = (status >> HSP_INT_EMPTY_SHIFT) & HSP_INT_EMPTY_MASK;
>>> +
>>> +	for_each_set_bit(bit, &mask, hsp->num_sm) {
>>> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
>>> +
>>> +		if (mb->producer) {
>>> +			/*
>>> +			 * Disable EMPTY interrupts until data is sent with
>>> +			 * the next message. These interrupts are level-
>>> +			 * triggered, so if we kept them enabled they would
>>> +			 * constantly trigger until we next write data into
>>> +			 * the message.
>>> +			 */
>>> +			spin_lock(&hsp->lock);
>>> +
>>> +			hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
>>> +			tegra_hsp_writel(hsp, hsp->mask,
>>> +					 HSP_INT_IE(hsp->shared_irq));
>>> +
>>> +			spin_unlock(&hsp->lock);
>>> +
>>> +			mbox_chan_txdone(mb->channel.chan, 0);
>>> +		}
>>> +	}
>>> +
>>> +	/* process FULL interrupts */
>>> +	mask = (status >> HSP_INT_FULL_SHIFT) & HSP_INT_FULL_MASK;
>>> +
>>> +	for_each_set_bit(bit, &mask, hsp->num_sm) {
>>> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[bit];
>>> +
>>> +		if (!mb->producer) {
>>> +			value = tegra_hsp_channel_readl(&mb->channel,
>>> +							HSP_SM_SHRD_MBOX);
>>> +			value &= ~HSP_SM_SHRD_MBOX_FULL;
>>> +			msg = (void *)(unsigned long)value;
>>> +			mbox_chan_received_data(mb->channel.chan, msg);
>>> +
>>> +			/*
>>> +			 * Need to clear all bits here since some producers,
>>> +			 * such as TCU, depend on fields in the register
>>> +			 * getting cleared by the consumer.
>>> +			 *
>>> +			 * The mailbox API doesn't give the consumers a way
>>> +			 * of doing that explicitly, so we have to make sure
>>> +			 * we cover all possible cases.
>>> +			 */
>>> +			tegra_hsp_channel_writel(&mb->channel, 0x0,
>>> +						 HSP_SM_SHRD_MBOX);
>>> +		}
>>> +	}
>>> +
>>> +	return IRQ_HANDLED;
>>> +}
>>> +
>>>  static struct tegra_hsp_channel *
>>>  tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
>>>  			  unsigned int master, unsigned int index)
>>> @@ -194,7 +284,7 @@ tegra_hsp_doorbell_create(struct tegra_hsp *hsp, const char *name,
>>>  	if (!db)
>>>  		return ERR_PTR(-ENOMEM);
>>>  
>>> -	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) << 16;
>>> +	offset = (1 + (hsp->num_sm / 2) + hsp->num_ss + hsp->num_as) * SZ_64K;
>>>  	offset += index * 0x100;
>>>  
>>>  	db->channel.regs = hsp->regs + offset;
>>> @@ -235,8 +325,8 @@ static int tegra_hsp_doorbell_startup(struct mbox_chan *chan)
>>>  	unsigned long flags;
>>>  	u32 value;
>>>  
>>> -	if (db->master >= hsp->mbox.num_chans) {
>>> -		dev_err(hsp->mbox.dev,
>>> +	if (db->master >= chan->mbox->num_chans) {
>>> +		dev_err(chan->mbox->dev,
>>>  			"invalid master ID %u for HSP channel\n",
>>>  			db->master);
>>>  		return -EINVAL;
>>> @@ -281,46 +371,168 @@ static void tegra_hsp_doorbell_shutdown(struct mbox_chan *chan)
>>>  	spin_unlock_irqrestore(&hsp->lock, flags);
>>>  }
>>>  
>>> -static const struct mbox_chan_ops tegra_hsp_doorbell_ops = {
>>> +static const struct mbox_chan_ops tegra_hsp_db_ops = {
>>>  	.send_data = tegra_hsp_doorbell_send_data,
>>>  	.startup = tegra_hsp_doorbell_startup,
>>>  	.shutdown = tegra_hsp_doorbell_shutdown,
>>>  };
>>>  
>>> -static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
>>> +static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
>>> +{
>>> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
>>> +	struct tegra_hsp *hsp = mb->channel.hsp;
>>> +	unsigned long flags;
>>> +	u32 value;
>>> +
>>> +	WARN_ON(!mb->producer);
>>
>> Should we return here?
> 
> Yeah, that's a good idea. I made this return -EPERM for lack of a better
> error code.
> 
>>> +
>>> +	/* copy data and mark mailbox full */
>>> +	value = (u32)(unsigned long)data;
>>> +	value |= HSP_SM_SHRD_MBOX_FULL;
>>> +
>>> +	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
>>> +
>>> +	if (!irqs_disabled()) {
>>> +		/* enable EMPTY interrupt for the shared mailbox */
>>> +		spin_lock_irqsave(&hsp->lock, flags);
>>> +
>>> +		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
>>> +		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
>>> +
>>> +		spin_unlock_irqrestore(&hsp->lock, flags);
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int tegra_hsp_mailbox_flush(struct mbox_chan *chan,
>>> +				   unsigned long timeout)
>>> +{
>>> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
>>> +	struct tegra_hsp_channel *ch = &mb->channel;
>>> +	u32 value;
>>> +
>>> +	timeout = jiffies + msecs_to_jiffies(timeout);
>>> +
>>> +	while (time_before(jiffies, timeout)) {
>>> +		value = tegra_hsp_channel_readl(ch, HSP_SM_SHRD_MBOX);
>>> +		if ((value & HSP_SM_SHRD_MBOX_FULL) == 0) {
>>> +			mbox_chan_txdone(chan, 0);
>>> +			return 0;
>>> +		}
>>> +
>>> +		udelay(1);
>>> +	}
>>> +
>>> +	return -ETIME;
>>> +}
>>> +
>>> +static int tegra_hsp_mailbox_startup(struct mbox_chan *chan)
>>> +{
>>> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
>>> +	struct tegra_hsp_channel *ch = &mb->channel;
>>> +	struct tegra_hsp *hsp = mb->channel.hsp;
>>> +	unsigned long flags;
>>> +
>>> +	chan->txdone_method = TXDONE_BY_IRQ;
>>> +
>>> +	/*
>>> +	 * Shared mailboxes start out as consumers by default. FULL and EMPTY
>>> +	 * interrupts are coalesced at the same shared interrupt.
>>> +	 *
>>> +	 * Keep EMPTY interrupts disabled at startup and only enable them when
>>> +	 * the mailbox is actually full. This is required because the FULL and
>>> +	 * EMPTY interrupts are level-triggered, so keeping EMPTY interrupts
>>> +	 * enabled all the time would cause an interrupt storm while mailboxes
>>> +	 * are idle.
>>> +	 */
>>> +
>>> +	spin_lock_irqsave(&hsp->lock, flags);
>>> +
>>> +	if (mb->producer)
>>> +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
>>> +	else
>>> +		hsp->mask |= BIT(HSP_INT_FULL_SHIFT + mb->index);
>>> +
>>> +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
>>> +
>>> +	spin_unlock_irqrestore(&hsp->lock, flags);
>>> +
>>> +	if (hsp->soc->has_per_mb_ie) {
>>> +		if (mb->producer)
>>> +			tegra_hsp_channel_writel(ch, 0x0,
>>> +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
>>> +		else
>>> +			tegra_hsp_channel_writel(ch, 0x1,
>>> +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static void tegra_hsp_mailbox_shutdown(struct mbox_chan *chan)
>>> +{
>>> +	struct tegra_hsp_mailbox *mb = chan->con_priv;
>>> +	struct tegra_hsp_channel *ch = &mb->channel;
>>> +	struct tegra_hsp *hsp = mb->channel.hsp;
>>> +	unsigned long flags;
>>> +
>>> +	if (hsp->soc->has_per_mb_ie) {
>>> +		if (mb->producer)
>>> +			tegra_hsp_channel_writel(ch, 0x0,
>>> +						 HSP_SM_SHRD_MBOX_EMPTY_INT_IE);
>>> +		else
>>> +			tegra_hsp_channel_writel(ch, 0x0,
>>> +						 HSP_SM_SHRD_MBOX_FULL_INT_IE);
>>> +	}
>>> +
>>> +	spin_lock_irqsave(&hsp->lock, flags);
>>> +
>>> +	if (mb->producer)
>>> +		hsp->mask &= ~BIT(HSP_INT_EMPTY_SHIFT + mb->index);
>>> +	else
>>> +		hsp->mask &= ~BIT(HSP_INT_FULL_SHIFT + mb->index);
>>> +
>>> +	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
>>> +
>>> +	spin_unlock_irqrestore(&hsp->lock, flags);
>>> +}
>>> +
>>> +static const struct mbox_chan_ops tegra_hsp_sm_ops = {
>>> +	.send_data = tegra_hsp_mailbox_send_data,
>>> +	.flush = tegra_hsp_mailbox_flush,
>>> +	.startup = tegra_hsp_mailbox_startup,
>>> +	.shutdown = tegra_hsp_mailbox_shutdown,
>>> +};
>>> +
>>> +static struct mbox_chan *tegra_hsp_db_xlate(struct mbox_controller *mbox,
>>>  					    const struct of_phandle_args *args)
>>>  {
>>> +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_db);
>>> +	unsigned int type = args->args[0], master = args->args[1];
>>>  	struct tegra_hsp_channel *channel = ERR_PTR(-ENODEV);
>>> -	struct tegra_hsp *hsp = to_tegra_hsp(mbox);
>>> -	unsigned int type = args->args[0];
>>> -	unsigned int master = args->args[1];
>>>  	struct tegra_hsp_doorbell *db;
>>>  	struct mbox_chan *chan;
>>>  	unsigned long flags;
>>>  	unsigned int i;
>>>  
>>> -	switch (type) {
>>> -	case TEGRA_HSP_MBOX_TYPE_DB:
>>> -		db = tegra_hsp_doorbell_get(hsp, master);
>>> -		if (db)
>>> -			channel = &db->channel;
>>> +	if (type != TEGRA_HSP_MBOX_TYPE_DB || !hsp->doorbell_irq)
>>> +		return ERR_PTR(-ENODEV);
>>>  
>>> -		break;
>>> -
>>> -	default:
>>> -		break;
>>> -	}
>>> +	db = tegra_hsp_doorbell_get(hsp, master);
>>> +	if (db)
>>> +		channel = &db->channel;
>>>  
>>>  	if (IS_ERR(channel))
>>>  		return ERR_CAST(channel);
>>>  
>>>  	spin_lock_irqsave(&hsp->lock, flags);
>>>  
>>> -	for (i = 0; i < hsp->mbox.num_chans; i++) {
>>> -		chan = &hsp->mbox.chans[i];
>>> +	for (i = 0; i < mbox->num_chans; i++) {
>>> +		chan = &mbox->chans[i];
>>>  		if (!chan->con_priv) {
>>> -			chan->con_priv = channel;
>>>  			channel->chan = chan;
>>> +			chan->con_priv = db;
>>>  			break;
>>>  		}
>>>  
>>> @@ -332,6 +544,29 @@ static struct mbox_chan *of_tegra_hsp_xlate(struct mbox_controller *mbox,
>>>  	return chan ?: ERR_PTR(-EBUSY);
>>>  }
>>>  
>>> +static struct mbox_chan *tegra_hsp_sm_xlate(struct mbox_controller *mbox,
>>> +					    const struct of_phandle_args *args)
>>> +{
>>> +	struct tegra_hsp *hsp = container_of(mbox, struct tegra_hsp, mbox_sm);
>>> +	unsigned int type = args->args[0], index;
>>> +	struct tegra_hsp_mailbox *mb;
>>> +
>>> +	index = args->args[1] & TEGRA_HSP_SM_MASK;
>>> +
>>> +	if (type != TEGRA_HSP_MBOX_TYPE_SM || !hsp->shared_irqs ||
>>> +	    index >= hsp->num_sm)
>>> +		return ERR_PTR(-ENODEV);
>>> +
>>> +	mb = &hsp->mailboxes[index];
>>> +
>>> +	if ((args->args[1] & TEGRA_HSP_SM_FLAG_TX) == 0)
>>> +		mb->producer = false;
>>> +	else
>>> +		mb->producer = true;
>>> +
>>> +	return mb->channel.chan;
>>> +}
>>> +
>>>  static void tegra_hsp_remove_doorbells(struct tegra_hsp *hsp)
>>>  {
>>>  	struct tegra_hsp_doorbell *db, *tmp;
>>> @@ -364,10 +599,65 @@ static int tegra_hsp_add_doorbells(struct tegra_hsp *hsp)
>>>  	return 0;
>>>  }
>>>  
>>> +static int tegra_hsp_add_mailboxes(struct tegra_hsp *hsp, struct device *dev)
>>> +{
>>> +	int i;
>>> +
>>> +	hsp->mailboxes = devm_kcalloc(dev, hsp->num_sm, sizeof(*hsp->mailboxes),
>>> +				      GFP_KERNEL);
>>> +	if (!hsp->mailboxes)
>>> +		return -ENOMEM;
>>> +
>>> +	for (i = 0; i < hsp->num_sm; i++) {
>>> +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
>>> +
>>> +		mb->index = i;
>>> +
>>> +		mb->channel.hsp = hsp;
>>> +		mb->channel.regs = hsp->regs + SZ_64K + i * SZ_32K;
>>> +		mb->channel.chan = &hsp->mbox_sm.chans[i];
>>> +		mb->channel.chan->con_priv = mb;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int tegra_hsp_request_shared_irqs(struct tegra_hsp *hsp)
>>> +{
>>> +	unsigned int i, irq = 0;
>>> +	int err;
>>> +
>>> +	for (i = 0; i < hsp->num_si; i++) {
>>> +		if (hsp->shared_irq == 0 && hsp->shared_irqs[i] > 0) {
>>> +			irq = hsp->shared_irqs[i];
>>> +			hsp->shared_irq = i;
>>> +			break;
>>> +		}
>>> +	}
>>> +
>>> +	if (irq > 0) {
>>> +		err = devm_request_irq(hsp->dev, irq, tegra_hsp_shared_irq, 0,
>>> +				       dev_name(hsp->dev), hsp);
>>> +		if (err < 0) {
>>> +			dev_err(hsp->dev, "failed to request interrupt: %d\n",
>>> +				err);
>>> +			return err;
>>> +		}
>>
>> Are we suppose to loop through all the shared interrupts and find one
>> that is available? Looking at the above it seems that we will try to use
>> the first shared interrupt we have a valid mapping for, regardless of if
>> it is available/in-use.
>>
>> Otherwise I am not sure why it is necessary to stored all the shared
>> irqs, because AFAICT we only use the first we have a valid mapping for.
>> Maybe I am missing something ...
> 
> Yeah, that's a good idea. In practice I don't think this matters at all
> because there just isn't another user of these interrupts, but it might
> be more explicit and self-explanatory if we try all of the interrupts
> in turn until we can request one.

By the way, any reason why we could not put the call to
platform_get_irq_byname() in the above function? May simplify the code a
bit to have a single loop.

Cheers
Jon

-- 
nvpublic

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> From: Thierry Reding <treding@nvidia.com>
>
> The mailbox framework supports blocking transfers via completions for
> clients that can sleep. In order to support blocking transfers in cases
> where the transmission is not permitted to sleep, add a new ->flush()
> callback that controller drivers can implement to busy loop until the
> transmission has been completed. This will automatically be called when
> available and interrupts are disabled for clients that request blocking
> transfers.
>
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
>  drivers/mailbox/mailbox.c          | 8 ++++++++
>  include/linux/mailbox_controller.h | 4 ++++
>  2 files changed, 12 insertions(+)
>
> diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> index 674b35f402f5..0eaf21259874 100644
> --- a/drivers/mailbox/mailbox.c
> +++ b/drivers/mailbox/mailbox.c
> @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
>                 unsigned long wait;
>                 int ret;
>
> +               if (irqs_disabled() && chan->mbox->ops->flush) {
> +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> +                       if (ret < 0)
> +                               tx_tick(chan, ret);
> +
> +                       return ret;
> +               }
> +
This is hacky. I think we can do without busy waiting in atomic
context. You could queue locally while in atomic context and then
transfer in blocking mode. I don't think we should worry about the
'throughput' as there already is no h/w rate control even with
busy-waiting.

Thanks.

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> From: Thierry Reding <treding@nvidia.com>
>
> The mailbox framework supports blocking transfers via completions for
> clients that can sleep. In order to support blocking transfers in cases
> where the transmission is not permitted to sleep, add a new ->flush()
> callback that controller drivers can implement to busy loop until the
> transmission has been completed. This will automatically be called when
> available and interrupts are disabled for clients that request blocking
> transfers.
>
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
>  drivers/mailbox/mailbox.c          | 8 ++++++++
>  include/linux/mailbox_controller.h | 4 ++++
>  2 files changed, 12 insertions(+)
>
> diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> index 674b35f402f5..0eaf21259874 100644
> --- a/drivers/mailbox/mailbox.c
> +++ b/drivers/mailbox/mailbox.c
> @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
>                 unsigned long wait;
>                 int ret;
>
> +               if (irqs_disabled() && chan->mbox->ops->flush) {
> +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> +                       if (ret < 0)
> +                               tx_tick(chan, ret);
> +
> +                       return ret;
> +               }
> +
This is hacky. I think we can do without busy waiting in atomic
context. You could queue locally while in atomic context and then
transfer in blocking mode. I don't think we should worry about the
'throughput' as there already is no h/w rate control even with
busy-waiting.

Thanks.

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 2863 bytes --]

On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > From: Thierry Reding <treding@nvidia.com>
> >
> > The mailbox framework supports blocking transfers via completions for
> > clients that can sleep. In order to support blocking transfers in cases
> > where the transmission is not permitted to sleep, add a new ->flush()
> > callback that controller drivers can implement to busy loop until the
> > transmission has been completed. This will automatically be called when
> > available and interrupts are disabled for clients that request blocking
> > transfers.
> >
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  drivers/mailbox/mailbox.c          | 8 ++++++++
> >  include/linux/mailbox_controller.h | 4 ++++
> >  2 files changed, 12 insertions(+)
> >
> > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > index 674b35f402f5..0eaf21259874 100644
> > --- a/drivers/mailbox/mailbox.c
> > +++ b/drivers/mailbox/mailbox.c
> > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> >                 unsigned long wait;
> >                 int ret;
> >
> > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > +                       if (ret < 0)
> > +                               tx_tick(chan, ret);
> > +
> > +                       return ret;
> > +               }
> > +
> This is hacky. I think we can do without busy waiting in atomic
> context. You could queue locally while in atomic context and then
> transfer in blocking mode. I don't think we should worry about the
> 'throughput' as there already is no h/w rate control even with
> busy-waiting.

I actually tried to do that before I added this flushing mechanism. The
problem is, like you said, one of rate control. As mentioned in the
cover letter, the shared mailboxes implemented in tegra-hsp are used as
RX and TX channels for the TCU, which is like a virtual UART. The TTY
driver included as part of this series will use one of the mailboxes to
transmit data that is written to the console. The problem is that if
these transmissions are not rate-limited on the TTY driver side, the
console will just keep writing data and eventually overflow the buffer
that we have in the mailbox subsystem.

The problem is that data comes in at a much higher rate than what we can
output. This is especially true at boot when the TCU console takes over
and the whole log buffer is dumped on it.

So the only way to rate-limit is to either make mbox_send_message()
block, but that can only be done in non-atomic context. The console,
however, will always run in atomic context, so the only way to do rate-
limiting is by busy looping.

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > From: Thierry Reding <treding@nvidia.com>
> >
> > The mailbox framework supports blocking transfers via completions for
> > clients that can sleep. In order to support blocking transfers in cases
> > where the transmission is not permitted to sleep, add a new ->flush()
> > callback that controller drivers can implement to busy loop until the
> > transmission has been completed. This will automatically be called when
> > available and interrupts are disabled for clients that request blocking
> > transfers.
> >
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  drivers/mailbox/mailbox.c          | 8 ++++++++
> >  include/linux/mailbox_controller.h | 4 ++++
> >  2 files changed, 12 insertions(+)
> >
> > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > index 674b35f402f5..0eaf21259874 100644
> > --- a/drivers/mailbox/mailbox.c
> > +++ b/drivers/mailbox/mailbox.c
> > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> >                 unsigned long wait;
> >                 int ret;
> >
> > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > +                       if (ret < 0)
> > +                               tx_tick(chan, ret);
> > +
> > +                       return ret;
> > +               }
> > +
> This is hacky. I think we can do without busy waiting in atomic
> context. You could queue locally while in atomic context and then
> transfer in blocking mode. I don't think we should worry about the
> 'throughput' as there already is no h/w rate control even with
> busy-waiting.

I actually tried to do that before I added this flushing mechanism. The
problem is, like you said, one of rate control. As mentioned in the
cover letter, the shared mailboxes implemented in tegra-hsp are used as
RX and TX channels for the TCU, which is like a virtual UART. The TTY
driver included as part of this series will use one of the mailboxes to
transmit data that is written to the console. The problem is that if
these transmissions are not rate-limited on the TTY driver side, the
console will just keep writing data and eventually overflow the buffer
that we have in the mailbox subsystem.

The problem is that data comes in at a much higher rate than what we can
output. This is especially true at boot when the TCU console takes over
and the whole log buffer is dumped on it.

So the only way to rate-limit is to either make mbox_send_message()
block, but that can only be done in non-atomic context. The console,
however, will always run in atomic context, so the only way to do rate-
limiting is by busy looping.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181120/20d316ab/attachment-0001.sig>

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 4139 bytes --]

On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > >
> > > From: Thierry Reding <treding@nvidia.com>
> > >
> > > The mailbox framework supports blocking transfers via completions for
> > > clients that can sleep. In order to support blocking transfers in cases
> > > where the transmission is not permitted to sleep, add a new ->flush()
> > > callback that controller drivers can implement to busy loop until the
> > > transmission has been completed. This will automatically be called when
> > > available and interrupts are disabled for clients that request blocking
> > > transfers.
> > >
> > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > ---
> > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > >  include/linux/mailbox_controller.h | 4 ++++
> > >  2 files changed, 12 insertions(+)
> > >
> > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > index 674b35f402f5..0eaf21259874 100644
> > > --- a/drivers/mailbox/mailbox.c
> > > +++ b/drivers/mailbox/mailbox.c
> > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > >                 unsigned long wait;
> > >                 int ret;
> > >
> > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > +                       if (ret < 0)
> > > +                               tx_tick(chan, ret);
> > > +
> > > +                       return ret;
> > > +               }
> > > +
> > This is hacky. I think we can do without busy waiting in atomic
> > context. You could queue locally while in atomic context and then
> > transfer in blocking mode. I don't think we should worry about the
> > 'throughput' as there already is no h/w rate control even with
> > busy-waiting.
> 
> I actually tried to do that before I added this flushing mechanism. The
> problem is, like you said, one of rate control. As mentioned in the
> cover letter, the shared mailboxes implemented in tegra-hsp are used as
> RX and TX channels for the TCU, which is like a virtual UART. The TTY
> driver included as part of this series will use one of the mailboxes to
> transmit data that is written to the console. The problem is that if
> these transmissions are not rate-limited on the TTY driver side, the
> console will just keep writing data and eventually overflow the buffer
> that we have in the mailbox subsystem.
> 
> The problem is that data comes in at a much higher rate than what we can
> output. This is especially true at boot when the TCU console takes over
> and the whole log buffer is dumped on it.
> 
> So the only way to rate-limit is to either make mbox_send_message()
> block, but that can only be done in non-atomic context. The console,
> however, will always run in atomic context, so the only way to do rate-
> limiting is by busy looping.

What I also tried before was to implement busy looping within the
->send_data() callback of the driver so that we didn't have to put this
into the core. Unfortunately, however, the ->send_data() callback is
called under chan->lock, which means that from mbox_send_message() we
don't have a way to mark the transfer as done. In order to do that we'd
have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
that in turn also attempts to take the chan->lock, which would cause a
deadlock.

The explicit flushing is the best alternative that I could come up with.
I think it's not all that hacky, because it's very explicit about what's
going on and it has the nice side-effect that it will allow the mailbox
to work in interrupt driven mode if possible and only resorting to the
busy loop in atomic context.

At this point I think I have explored all other options and I frankly
can't find a more proper way to achieve what we need here. Perhaps you
can think of additional ways to accomplish this?

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > >
> > > From: Thierry Reding <treding@nvidia.com>
> > >
> > > The mailbox framework supports blocking transfers via completions for
> > > clients that can sleep. In order to support blocking transfers in cases
> > > where the transmission is not permitted to sleep, add a new ->flush()
> > > callback that controller drivers can implement to busy loop until the
> > > transmission has been completed. This will automatically be called when
> > > available and interrupts are disabled for clients that request blocking
> > > transfers.
> > >
> > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > ---
> > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > >  include/linux/mailbox_controller.h | 4 ++++
> > >  2 files changed, 12 insertions(+)
> > >
> > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > index 674b35f402f5..0eaf21259874 100644
> > > --- a/drivers/mailbox/mailbox.c
> > > +++ b/drivers/mailbox/mailbox.c
> > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > >                 unsigned long wait;
> > >                 int ret;
> > >
> > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > +                       if (ret < 0)
> > > +                               tx_tick(chan, ret);
> > > +
> > > +                       return ret;
> > > +               }
> > > +
> > This is hacky. I think we can do without busy waiting in atomic
> > context. You could queue locally while in atomic context and then
> > transfer in blocking mode. I don't think we should worry about the
> > 'throughput' as there already is no h/w rate control even with
> > busy-waiting.
> 
> I actually tried to do that before I added this flushing mechanism. The
> problem is, like you said, one of rate control. As mentioned in the
> cover letter, the shared mailboxes implemented in tegra-hsp are used as
> RX and TX channels for the TCU, which is like a virtual UART. The TTY
> driver included as part of this series will use one of the mailboxes to
> transmit data that is written to the console. The problem is that if
> these transmissions are not rate-limited on the TTY driver side, the
> console will just keep writing data and eventually overflow the buffer
> that we have in the mailbox subsystem.
> 
> The problem is that data comes in at a much higher rate than what we can
> output. This is especially true at boot when the TCU console takes over
> and the whole log buffer is dumped on it.
> 
> So the only way to rate-limit is to either make mbox_send_message()
> block, but that can only be done in non-atomic context. The console,
> however, will always run in atomic context, so the only way to do rate-
> limiting is by busy looping.

What I also tried before was to implement busy looping within the
->send_data() callback of the driver so that we didn't have to put this
into the core. Unfortunately, however, the ->send_data() callback is
called under chan->lock, which means that from mbox_send_message() we
don't have a way to mark the transfer as done. In order to do that we'd
have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
that in turn also attempts to take the chan->lock, which would cause a
deadlock.

The explicit flushing is the best alternative that I could come up with.
I think it's not all that hacky, because it's very explicit about what's
going on and it has the nice side-effect that it will allow the mailbox
to work in interrupt driven mode if possible and only resorting to the
busy loop in atomic context.

At this point I think I have explored all other options and I frankly
can't find a more proper way to achieve what we need here. Perhaps you
can think of additional ways to accomplish this?

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181121/8a844def/attachment.sig>

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Wed, Nov 21, 2018 at 8:27 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> > On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > >
> > > > From: Thierry Reding <treding@nvidia.com>
> > > >
> > > > The mailbox framework supports blocking transfers via completions for
> > > > clients that can sleep. In order to support blocking transfers in cases
> > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > callback that controller drivers can implement to busy loop until the
> > > > transmission has been completed. This will automatically be called when
> > > > available and interrupts are disabled for clients that request blocking
> > > > transfers.
> > > >
> > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > ---
> > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > >  include/linux/mailbox_controller.h | 4 ++++
> > > >  2 files changed, 12 insertions(+)
> > > >
> > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > index 674b35f402f5..0eaf21259874 100644
> > > > --- a/drivers/mailbox/mailbox.c
> > > > +++ b/drivers/mailbox/mailbox.c
> > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > >                 unsigned long wait;
> > > >                 int ret;
> > > >
> > > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > +                       if (ret < 0)
> > > > +                               tx_tick(chan, ret);
> > > > +
> > > > +                       return ret;
> > > > +               }
> > > > +
> > > This is hacky. I think we can do without busy waiting in atomic
> > > context. You could queue locally while in atomic context and then
> > > transfer in blocking mode. I don't think we should worry about the
> > > 'throughput' as there already is no h/w rate control even with
> > > busy-waiting.
> >
> > I actually tried to do that before I added this flushing mechanism. The
> > problem is, like you said, one of rate control. As mentioned in the
> > cover letter, the shared mailboxes implemented in tegra-hsp are used as
> > RX and TX channels for the TCU, which is like a virtual UART. The TTY
> > driver included as part of this series will use one of the mailboxes to
> > transmit data that is written to the console. The problem is that if
> > these transmissions are not rate-limited on the TTY driver side, the
> > console will just keep writing data and eventually overflow the buffer
> > that we have in the mailbox subsystem.
> >
> > The problem is that data comes in at a much higher rate than what we can
> > output. This is especially true at boot when the TCU console takes over
> > and the whole log buffer is dumped on it.
> >
> > So the only way to rate-limit is to either make mbox_send_message()
> > block, but that can only be done in non-atomic context. The console,
> > however, will always run in atomic context, so the only way to do rate-
> > limiting is by busy looping.
>
> What I also tried before was to implement busy looping within the
> ->send_data() callback of the driver so that we didn't have to put this
> into the core. Unfortunately, however, the ->send_data() callback is
> called under chan->lock, which means that from mbox_send_message() we
> don't have a way to mark the transfer as done. In order to do that we'd
> have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
> that in turn also attempts to take the chan->lock, which would cause a
> deadlock.
>
> The explicit flushing is the best alternative that I could come up with.
> I think it's not all that hacky, because it's very explicit about what's
> going on and it has the nice side-effect that it will allow the mailbox
> to work in interrupt driven mode if possible and only resorting to the
> busy loop in atomic context.
>
> At this point I think I have explored all other options and I frankly
> can't find a more proper way to achieve what we need here. Perhaps you
> can think of additional ways to accomplish this?
>
Well, I would have a local ring buffer (array) of enough size to hold
the characters and then have a task consuming data from that ring
buffer by transmitting over mailbox.

-jassi

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Wed, Nov 21, 2018 at 8:27 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> > On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > >
> > > > From: Thierry Reding <treding@nvidia.com>
> > > >
> > > > The mailbox framework supports blocking transfers via completions for
> > > > clients that can sleep. In order to support blocking transfers in cases
> > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > callback that controller drivers can implement to busy loop until the
> > > > transmission has been completed. This will automatically be called when
> > > > available and interrupts are disabled for clients that request blocking
> > > > transfers.
> > > >
> > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > ---
> > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > >  include/linux/mailbox_controller.h | 4 ++++
> > > >  2 files changed, 12 insertions(+)
> > > >
> > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > index 674b35f402f5..0eaf21259874 100644
> > > > --- a/drivers/mailbox/mailbox.c
> > > > +++ b/drivers/mailbox/mailbox.c
> > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > >                 unsigned long wait;
> > > >                 int ret;
> > > >
> > > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > +                       if (ret < 0)
> > > > +                               tx_tick(chan, ret);
> > > > +
> > > > +                       return ret;
> > > > +               }
> > > > +
> > > This is hacky. I think we can do without busy waiting in atomic
> > > context. You could queue locally while in atomic context and then
> > > transfer in blocking mode. I don't think we should worry about the
> > > 'throughput' as there already is no h/w rate control even with
> > > busy-waiting.
> >
> > I actually tried to do that before I added this flushing mechanism. The
> > problem is, like you said, one of rate control. As mentioned in the
> > cover letter, the shared mailboxes implemented in tegra-hsp are used as
> > RX and TX channels for the TCU, which is like a virtual UART. The TTY
> > driver included as part of this series will use one of the mailboxes to
> > transmit data that is written to the console. The problem is that if
> > these transmissions are not rate-limited on the TTY driver side, the
> > console will just keep writing data and eventually overflow the buffer
> > that we have in the mailbox subsystem.
> >
> > The problem is that data comes in at a much higher rate than what we can
> > output. This is especially true at boot when the TCU console takes over
> > and the whole log buffer is dumped on it.
> >
> > So the only way to rate-limit is to either make mbox_send_message()
> > block, but that can only be done in non-atomic context. The console,
> > however, will always run in atomic context, so the only way to do rate-
> > limiting is by busy looping.
>
> What I also tried before was to implement busy looping within the
> ->send_data() callback of the driver so that we didn't have to put this
> into the core. Unfortunately, however, the ->send_data() callback is
> called under chan->lock, which means that from mbox_send_message() we
> don't have a way to mark the transfer as done. In order to do that we'd
> have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
> that in turn also attempts to take the chan->lock, which would cause a
> deadlock.
>
> The explicit flushing is the best alternative that I could come up with.
> I think it's not all that hacky, because it's very explicit about what's
> going on and it has the nice side-effect that it will allow the mailbox
> to work in interrupt driven mode if possible and only resorting to the
> busy loop in atomic context.
>
> At this point I think I have explored all other options and I frankly
> can't find a more proper way to achieve what we need here. Perhaps you
> can think of additional ways to accomplish this?
>
Well, I would have a local ring buffer (array) of enough size to hold
the characters and then have a task consuming data from that ring
buffer by transmitting over mailbox.

-jassi

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 5884 bytes --]

On Wed, Nov 21, 2018 at 08:18:22PM -0600, Jassi Brar wrote:
> On Wed, Nov 21, 2018 at 8:27 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> > > On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > > > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > > >
> > > > > From: Thierry Reding <treding@nvidia.com>
> > > > >
> > > > > The mailbox framework supports blocking transfers via completions for
> > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > callback that controller drivers can implement to busy loop until the
> > > > > transmission has been completed. This will automatically be called when
> > > > > available and interrupts are disabled for clients that request blocking
> > > > > transfers.
> > > > >
> > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > ---
> > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > >  2 files changed, 12 insertions(+)
> > > > >
> > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > --- a/drivers/mailbox/mailbox.c
> > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > >                 unsigned long wait;
> > > > >                 int ret;
> > > > >
> > > > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > +                       if (ret < 0)
> > > > > +                               tx_tick(chan, ret);
> > > > > +
> > > > > +                       return ret;
> > > > > +               }
> > > > > +
> > > > This is hacky. I think we can do without busy waiting in atomic
> > > > context. You could queue locally while in atomic context and then
> > > > transfer in blocking mode. I don't think we should worry about the
> > > > 'throughput' as there already is no h/w rate control even with
> > > > busy-waiting.
> > >
> > > I actually tried to do that before I added this flushing mechanism. The
> > > problem is, like you said, one of rate control. As mentioned in the
> > > cover letter, the shared mailboxes implemented in tegra-hsp are used as
> > > RX and TX channels for the TCU, which is like a virtual UART. The TTY
> > > driver included as part of this series will use one of the mailboxes to
> > > transmit data that is written to the console. The problem is that if
> > > these transmissions are not rate-limited on the TTY driver side, the
> > > console will just keep writing data and eventually overflow the buffer
> > > that we have in the mailbox subsystem.
> > >
> > > The problem is that data comes in at a much higher rate than what we can
> > > output. This is especially true at boot when the TCU console takes over
> > > and the whole log buffer is dumped on it.
> > >
> > > So the only way to rate-limit is to either make mbox_send_message()
> > > block, but that can only be done in non-atomic context. The console,
> > > however, will always run in atomic context, so the only way to do rate-
> > > limiting is by busy looping.
> >
> > What I also tried before was to implement busy looping within the
> > ->send_data() callback of the driver so that we didn't have to put this
> > into the core. Unfortunately, however, the ->send_data() callback is
> > called under chan->lock, which means that from mbox_send_message() we
> > don't have a way to mark the transfer as done. In order to do that we'd
> > have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
> > that in turn also attempts to take the chan->lock, which would cause a
> > deadlock.
> >
> > The explicit flushing is the best alternative that I could come up with.
> > I think it's not all that hacky, because it's very explicit about what's
> > going on and it has the nice side-effect that it will allow the mailbox
> > to work in interrupt driven mode if possible and only resorting to the
> > busy loop in atomic context.
> >
> > At this point I think I have explored all other options and I frankly
> > can't find a more proper way to achieve what we need here. Perhaps you
> > can think of additional ways to accomplish this?
> >
> Well, I would have a local ring buffer (array) of enough size to hold
> the characters and then have a task consuming data from that ring
> buffer by transmitting over mailbox.

There's already such a ringbuffer in the printk code. To implement what
you suggest would effectively be creating a copy of that buffer because
we'd be allocating the buffer and the console code would just dump each
and every character in the logbuf into that ring buffer without rate-
limitation.

To make matters worse, the ringbuffer would be empty most of the time
after the initial dump of the logbuf, so we'd be wasting all that buffer
space.

It just seems to me like we should be keeping the TCU driver as close as
possible to other UART drivers which also busy loop in order to rate-
limit what the console can write. Given the current mailbox framework it
is not possible to do that (in interrupt context), so an extension seems
like the most sensible option.

Perhaps you'd be less concerned about such a change if it was perhaps
more explicit? Just throwing ideas around, I think something that could
also work is if we explicitly add a mbox_flush() function that would
basically be calling ->flush(). That way users of the mailbox can make
their requirement very explicit. I haven't actually tested that, but I
think it would work. Does that sound more acceptable to you?

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

On Wed, Nov 21, 2018 at 08:18:22PM -0600, Jassi Brar wrote:
> On Wed, Nov 21, 2018 at 8:27 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> > > On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > > > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > > >
> > > > > From: Thierry Reding <treding@nvidia.com>
> > > > >
> > > > > The mailbox framework supports blocking transfers via completions for
> > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > callback that controller drivers can implement to busy loop until the
> > > > > transmission has been completed. This will automatically be called when
> > > > > available and interrupts are disabled for clients that request blocking
> > > > > transfers.
> > > > >
> > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > ---
> > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > >  2 files changed, 12 insertions(+)
> > > > >
> > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > --- a/drivers/mailbox/mailbox.c
> > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > >                 unsigned long wait;
> > > > >                 int ret;
> > > > >
> > > > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > +                       if (ret < 0)
> > > > > +                               tx_tick(chan, ret);
> > > > > +
> > > > > +                       return ret;
> > > > > +               }
> > > > > +
> > > > This is hacky. I think we can do without busy waiting in atomic
> > > > context. You could queue locally while in atomic context and then
> > > > transfer in blocking mode. I don't think we should worry about the
> > > > 'throughput' as there already is no h/w rate control even with
> > > > busy-waiting.
> > >
> > > I actually tried to do that before I added this flushing mechanism. The
> > > problem is, like you said, one of rate control. As mentioned in the
> > > cover letter, the shared mailboxes implemented in tegra-hsp are used as
> > > RX and TX channels for the TCU, which is like a virtual UART. The TTY
> > > driver included as part of this series will use one of the mailboxes to
> > > transmit data that is written to the console. The problem is that if
> > > these transmissions are not rate-limited on the TTY driver side, the
> > > console will just keep writing data and eventually overflow the buffer
> > > that we have in the mailbox subsystem.
> > >
> > > The problem is that data comes in at a much higher rate than what we can
> > > output. This is especially true at boot when the TCU console takes over
> > > and the whole log buffer is dumped on it.
> > >
> > > So the only way to rate-limit is to either make mbox_send_message()
> > > block, but that can only be done in non-atomic context. The console,
> > > however, will always run in atomic context, so the only way to do rate-
> > > limiting is by busy looping.
> >
> > What I also tried before was to implement busy looping within the
> > ->send_data() callback of the driver so that we didn't have to put this
> > into the core. Unfortunately, however, the ->send_data() callback is
> > called under chan->lock, which means that from mbox_send_message() we
> > don't have a way to mark the transfer as done. In order to do that we'd
> > have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
> > that in turn also attempts to take the chan->lock, which would cause a
> > deadlock.
> >
> > The explicit flushing is the best alternative that I could come up with.
> > I think it's not all that hacky, because it's very explicit about what's
> > going on and it has the nice side-effect that it will allow the mailbox
> > to work in interrupt driven mode if possible and only resorting to the
> > busy loop in atomic context.
> >
> > At this point I think I have explored all other options and I frankly
> > can't find a more proper way to achieve what we need here. Perhaps you
> > can think of additional ways to accomplish this?
> >
> Well, I would have a local ring buffer (array) of enough size to hold
> the characters and then have a task consuming data from that ring
> buffer by transmitting over mailbox.

There's already such a ringbuffer in the printk code. To implement what
you suggest would effectively be creating a copy of that buffer because
we'd be allocating the buffer and the console code would just dump each
and every character in the logbuf into that ring buffer without rate-
limitation.

To make matters worse, the ringbuffer would be empty most of the time
after the initial dump of the logbuf, so we'd be wasting all that buffer
space.

It just seems to me like we should be keeping the TCU driver as close as
possible to other UART drivers which also busy loop in order to rate-
limit what the console can write. Given the current mailbox framework it
is not possible to do that (in interrupt context), so an extension seems
like the most sensible option.

Perhaps you'd be less concerned about such a change if it was perhaps
more explicit? Just throwing ideas around, I think something that could
also work is if we explicitly add a mbox_flush() function that would
basically be calling ->flush(). That way users of the mailbox can make
their requirement very explicit. I haven't actually tested that, but I
think it would work. Does that sound more acceptable to you?

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181122/d3b3ee6e/attachment.sig>

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

Hi Thierry,

On Thu, Nov 22, 2018 at 2:47 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Wed, Nov 21, 2018 at 08:18:22PM -0600, Jassi Brar wrote:
> > On Wed, Nov 21, 2018 at 8:27 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > >
> > > On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> > > > On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > > > > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > > > >
> > > > > > From: Thierry Reding <treding@nvidia.com>
> > > > > >
> > > > > > The mailbox framework supports blocking transfers via completions for
> > > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > > callback that controller drivers can implement to busy loop until the
> > > > > > transmission has been completed. This will automatically be called when
> > > > > > available and interrupts are disabled for clients that request blocking
> > > > > > transfers.
> > > > > >
> > > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > > ---
> > > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > > >  2 files changed, 12 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > > --- a/drivers/mailbox/mailbox.c
> > > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > > >                 unsigned long wait;
> > > > > >                 int ret;
> > > > > >
> > > > > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > > +                       if (ret < 0)
> > > > > > +                               tx_tick(chan, ret);
> > > > > > +
> > > > > > +                       return ret;
> > > > > > +               }
> > > > > > +
> > > > > This is hacky. I think we can do without busy waiting in atomic
> > > > > context. You could queue locally while in atomic context and then
> > > > > transfer in blocking mode. I don't think we should worry about the
> > > > > 'throughput' as there already is no h/w rate control even with
> > > > > busy-waiting.
> > > >
> > > > I actually tried to do that before I added this flushing mechanism. The
> > > > problem is, like you said, one of rate control. As mentioned in the
> > > > cover letter, the shared mailboxes implemented in tegra-hsp are used as
> > > > RX and TX channels for the TCU, which is like a virtual UART. The TTY
> > > > driver included as part of this series will use one of the mailboxes to
> > > > transmit data that is written to the console. The problem is that if
> > > > these transmissions are not rate-limited on the TTY driver side, the
> > > > console will just keep writing data and eventually overflow the buffer
> > > > that we have in the mailbox subsystem.
> > > >
> > > > The problem is that data comes in at a much higher rate than what we can
> > > > output. This is especially true at boot when the TCU console takes over
> > > > and the whole log buffer is dumped on it.
> > > >
> > > > So the only way to rate-limit is to either make mbox_send_message()
> > > > block, but that can only be done in non-atomic context. The console,
> > > > however, will always run in atomic context, so the only way to do rate-
> > > > limiting is by busy looping.
> > >
> > > What I also tried before was to implement busy looping within the
> > > ->send_data() callback of the driver so that we didn't have to put this
> > > into the core. Unfortunately, however, the ->send_data() callback is
> > > called under chan->lock, which means that from mbox_send_message() we
> > > don't have a way to mark the transfer as done. In order to do that we'd
> > > have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
> > > that in turn also attempts to take the chan->lock, which would cause a
> > > deadlock.
> > >
> > > The explicit flushing is the best alternative that I could come up with.
> > > I think it's not all that hacky, because it's very explicit about what's
> > > going on and it has the nice side-effect that it will allow the mailbox
> > > to work in interrupt driven mode if possible and only resorting to the
> > > busy loop in atomic context.
> > >
> > > At this point I think I have explored all other options and I frankly
> > > can't find a more proper way to achieve what we need here. Perhaps you
> > > can think of additional ways to accomplish this?
> > >
> > Well, I would have a local ring buffer (array) of enough size to hold
> > the characters and then have a task consuming data from that ring
> > buffer by transmitting over mailbox.
>
> There's already such a ringbuffer in the printk code. To implement what
> you suggest would effectively be creating a copy of that buffer because
> we'd be allocating the buffer and the console code would just dump each
> and every character in the logbuf into that ring buffer without rate-
> limitation.
>
Well, the console assumes there exists an atomic path to put character
on the bus, But because there isn't in case of tcu, we have to emulate
that. Frankly I prefer the one-off driver jump some hoops, rather than
implement exceptions in the api.
BTW, there is already no rate-limitation because its all virtual -
data is consumed as fast as possible.

> To make matters worse, the ringbuffer would be empty most of the time
> after the initial dump of the logbuf, so we'd be wasting all that buffer
> space.
>
The idea is console and uart-ops both feed into this buffer and the
only consumer thread runs the mailbox.

> It just seems to me like we should be keeping the TCU driver as close as
> possible to other UART drivers which also busy loop in order to rate-
> limit what the console can write. Given the current mailbox framework it
> is not possible to do that (in interrupt context), so an extension seems
> like the most sensible option.
>
> Perhaps you'd be less concerned about such a change if it was perhaps
> more explicit? Just throwing ideas around, I think something that could
> also work is if we explicitly add a mbox_flush() function that would
> basically be calling ->flush(). That way users of the mailbox can make
> their requirement very explicit. I haven't actually tested that, but I
> think it would work. Does that sound more acceptable to you?
>
I am happy to see features and bugfixes added to the api. What I am
not eager about is supporting less than 100% legit and very platform
specific usecases, especially when there is a work around.

Thanks.

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

Hi Thierry,

On Thu, Nov 22, 2018 at 2:47 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Wed, Nov 21, 2018 at 08:18:22PM -0600, Jassi Brar wrote:
> > On Wed, Nov 21, 2018 at 8:27 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > >
> > > On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> > > > On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > > > > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > > > >
> > > > > > From: Thierry Reding <treding@nvidia.com>
> > > > > >
> > > > > > The mailbox framework supports blocking transfers via completions for
> > > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > > callback that controller drivers can implement to busy loop until the
> > > > > > transmission has been completed. This will automatically be called when
> > > > > > available and interrupts are disabled for clients that request blocking
> > > > > > transfers.
> > > > > >
> > > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > > ---
> > > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > > >  2 files changed, 12 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > > --- a/drivers/mailbox/mailbox.c
> > > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > > >                 unsigned long wait;
> > > > > >                 int ret;
> > > > > >
> > > > > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > > +                       if (ret < 0)
> > > > > > +                               tx_tick(chan, ret);
> > > > > > +
> > > > > > +                       return ret;
> > > > > > +               }
> > > > > > +
> > > > > This is hacky. I think we can do without busy waiting in atomic
> > > > > context. You could queue locally while in atomic context and then
> > > > > transfer in blocking mode. I don't think we should worry about the
> > > > > 'throughput' as there already is no h/w rate control even with
> > > > > busy-waiting.
> > > >
> > > > I actually tried to do that before I added this flushing mechanism. The
> > > > problem is, like you said, one of rate control. As mentioned in the
> > > > cover letter, the shared mailboxes implemented in tegra-hsp are used as
> > > > RX and TX channels for the TCU, which is like a virtual UART. The TTY
> > > > driver included as part of this series will use one of the mailboxes to
> > > > transmit data that is written to the console. The problem is that if
> > > > these transmissions are not rate-limited on the TTY driver side, the
> > > > console will just keep writing data and eventually overflow the buffer
> > > > that we have in the mailbox subsystem.
> > > >
> > > > The problem is that data comes in at a much higher rate than what we can
> > > > output. This is especially true at boot when the TCU console takes over
> > > > and the whole log buffer is dumped on it.
> > > >
> > > > So the only way to rate-limit is to either make mbox_send_message()
> > > > block, but that can only be done in non-atomic context. The console,
> > > > however, will always run in atomic context, so the only way to do rate-
> > > > limiting is by busy looping.
> > >
> > > What I also tried before was to implement busy looping within the
> > > ->send_data() callback of the driver so that we didn't have to put this
> > > into the core. Unfortunately, however, the ->send_data() callback is
> > > called under chan->lock, which means that from mbox_send_message() we
> > > don't have a way to mark the transfer as done. In order to do that we'd
> > > have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
> > > that in turn also attempts to take the chan->lock, which would cause a
> > > deadlock.
> > >
> > > The explicit flushing is the best alternative that I could come up with.
> > > I think it's not all that hacky, because it's very explicit about what's
> > > going on and it has the nice side-effect that it will allow the mailbox
> > > to work in interrupt driven mode if possible and only resorting to the
> > > busy loop in atomic context.
> > >
> > > At this point I think I have explored all other options and I frankly
> > > can't find a more proper way to achieve what we need here. Perhaps you
> > > can think of additional ways to accomplish this?
> > >
> > Well, I would have a local ring buffer (array) of enough size to hold
> > the characters and then have a task consuming data from that ring
> > buffer by transmitting over mailbox.
>
> There's already such a ringbuffer in the printk code. To implement what
> you suggest would effectively be creating a copy of that buffer because
> we'd be allocating the buffer and the console code would just dump each
> and every character in the logbuf into that ring buffer without rate-
> limitation.
>
Well, the console assumes there exists an atomic path to put character
on the bus, But because there isn't in case of tcu, we have to emulate
that. Frankly I prefer the one-off driver jump some hoops, rather than
implement exceptions in the api.
BTW, there is already no rate-limitation because its all virtual -
data is consumed as fast as possible.

> To make matters worse, the ringbuffer would be empty most of the time
> after the initial dump of the logbuf, so we'd be wasting all that buffer
> space.
>
The idea is console and uart-ops both feed into this buffer and the
only consumer thread runs the mailbox.

> It just seems to me like we should be keeping the TCU driver as close as
> possible to other UART drivers which also busy loop in order to rate-
> limit what the console can write. Given the current mailbox framework it
> is not possible to do that (in interrupt context), so an extension seems
> like the most sensible option.
>
> Perhaps you'd be less concerned about such a change if it was perhaps
> more explicit? Just throwing ideas around, I think something that could
> also work is if we explicitly add a mbox_flush() function that would
> basically be calling ->flush(). That way users of the mailbox can make
> their requirement very explicit. I haven't actually tested that, but I
> think it would work. Does that sound more acceptable to you?
>
I am happy to see features and bugfixes added to the api. What I am
not eager about is supporting less than 100% legit and very platform
specific usecases, especially when there is a work around.

Thanks.

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 9640 bytes --]

On Thu, Nov 22, 2018 at 10:07:09AM -0600, Jassi Brar wrote:
> Hi Thierry,
> 
> On Thu, Nov 22, 2018 at 2:47 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > On Wed, Nov 21, 2018 at 08:18:22PM -0600, Jassi Brar wrote:
> > > On Wed, Nov 21, 2018 at 8:27 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > >
> > > > On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> > > > > On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > > > > > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > > > > >
> > > > > > > From: Thierry Reding <treding@nvidia.com>
> > > > > > >
> > > > > > > The mailbox framework supports blocking transfers via completions for
> > > > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > > > callback that controller drivers can implement to busy loop until the
> > > > > > > transmission has been completed. This will automatically be called when
> > > > > > > available and interrupts are disabled for clients that request blocking
> > > > > > > transfers.
> > > > > > >
> > > > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > > > ---
> > > > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > > > >  2 files changed, 12 insertions(+)
> > > > > > >
> > > > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > > > --- a/drivers/mailbox/mailbox.c
> > > > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > > > >                 unsigned long wait;
> > > > > > >                 int ret;
> > > > > > >
> > > > > > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > > > +                       if (ret < 0)
> > > > > > > +                               tx_tick(chan, ret);
> > > > > > > +
> > > > > > > +                       return ret;
> > > > > > > +               }
> > > > > > > +
> > > > > > This is hacky. I think we can do without busy waiting in atomic
> > > > > > context. You could queue locally while in atomic context and then
> > > > > > transfer in blocking mode. I don't think we should worry about the
> > > > > > 'throughput' as there already is no h/w rate control even with
> > > > > > busy-waiting.
> > > > >
> > > > > I actually tried to do that before I added this flushing mechanism. The
> > > > > problem is, like you said, one of rate control. As mentioned in the
> > > > > cover letter, the shared mailboxes implemented in tegra-hsp are used as
> > > > > RX and TX channels for the TCU, which is like a virtual UART. The TTY
> > > > > driver included as part of this series will use one of the mailboxes to
> > > > > transmit data that is written to the console. The problem is that if
> > > > > these transmissions are not rate-limited on the TTY driver side, the
> > > > > console will just keep writing data and eventually overflow the buffer
> > > > > that we have in the mailbox subsystem.
> > > > >
> > > > > The problem is that data comes in at a much higher rate than what we can
> > > > > output. This is especially true at boot when the TCU console takes over
> > > > > and the whole log buffer is dumped on it.
> > > > >
> > > > > So the only way to rate-limit is to either make mbox_send_message()
> > > > > block, but that can only be done in non-atomic context. The console,
> > > > > however, will always run in atomic context, so the only way to do rate-
> > > > > limiting is by busy looping.
> > > >
> > > > What I also tried before was to implement busy looping within the
> > > > ->send_data() callback of the driver so that we didn't have to put this
> > > > into the core. Unfortunately, however, the ->send_data() callback is
> > > > called under chan->lock, which means that from mbox_send_message() we
> > > > don't have a way to mark the transfer as done. In order to do that we'd
> > > > have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
> > > > that in turn also attempts to take the chan->lock, which would cause a
> > > > deadlock.
> > > >
> > > > The explicit flushing is the best alternative that I could come up with.
> > > > I think it's not all that hacky, because it's very explicit about what's
> > > > going on and it has the nice side-effect that it will allow the mailbox
> > > > to work in interrupt driven mode if possible and only resorting to the
> > > > busy loop in atomic context.
> > > >
> > > > At this point I think I have explored all other options and I frankly
> > > > can't find a more proper way to achieve what we need here. Perhaps you
> > > > can think of additional ways to accomplish this?
> > > >
> > > Well, I would have a local ring buffer (array) of enough size to hold
> > > the characters and then have a task consuming data from that ring
> > > buffer by transmitting over mailbox.
> >
> > There's already such a ringbuffer in the printk code. To implement what
> > you suggest would effectively be creating a copy of that buffer because
> > we'd be allocating the buffer and the console code would just dump each
> > and every character in the logbuf into that ring buffer without rate-
> > limitation.
> >
> Well, the console assumes there exists an atomic path to put character
> on the bus, But because there isn't in case of tcu, we have to emulate
> that. Frankly I prefer the one-off driver jump some hoops, rather than
> implement exceptions in the api.

I wouldn't have any objections to that if the hoops were reasonable
ones. What you're asking me to do is basically implement a second copy
of the logbuf. I don't call that a reasonable hoop to jump through. It
is also not guaranteed to work properly because we can always end up
with a situation where we produce more data than we can consume. Also,
by providing this additional buffer we make things worse because the
standard mechanisms of the logbuf are side-stepped. Typically the logbuf
code will warn if it overflows. In our case we provide a buffer that the
console can dump into, so instead of the logbuf overflowing and warning
about it, we'd now overflow the mailbox buffer and we'd have to add
extra code to warn about overflows.

The console really only works because it assumes that the output driver
will stall and thereby rate-limit.

> BTW, there is already no rate-limitation because its all virtual -
> data is consumed as fast as possible.

No, that's not true. On the receiving end of the TX mailbox is a micro-
processor that reads out the mailbox data. Once it has read the data it
needs to clear the FULL bit so that the TCU driver can write more data.
Even if the microprocessor on the receiving end did buffering (there is
no indication that it does) it would eventually run out of buffer space
and have to stall until it's clocked all the characters out of the
physical UART. At that point no amount of buffering is going to save us
and the only option is to stall, which, again, can currently only be
done from the mailbox, because it is the only one that knows when it is
busy. But it's also the one place where we can't because the framework
doesn't allow it.

> > To make matters worse, the ringbuffer would be empty most of the time
> > after the initial dump of the logbuf, so we'd be wasting all that buffer
> > space.
> >
> The idea is console and uart-ops both feed into this buffer and the
> only consumer thread runs the mailbox.
> 
> > It just seems to me like we should be keeping the TCU driver as close as
> > possible to other UART drivers which also busy loop in order to rate-
> > limit what the console can write. Given the current mailbox framework it
> > is not possible to do that (in interrupt context), so an extension seems
> > like the most sensible option.
> >
> > Perhaps you'd be less concerned about such a change if it was perhaps
> > more explicit? Just throwing ideas around, I think something that could
> > also work is if we explicitly add a mbox_flush() function that would
> > basically be calling ->flush(). That way users of the mailbox can make
> > their requirement very explicit. I haven't actually tested that, but I
> > think it would work. Does that sound more acceptable to you?
> >
> I am happy to see features and bugfixes added to the api. What I am
> not eager about is supporting less than 100% legit and very platform
> specific usecases, especially when there is a work around.

Look, I'd be willing to push this all into the driver, but the framework
doesn't allow me to do that. If it was possible to run the state machine
outside of mbox_send_message() then I'd be able to just move the busy
loop or flush into the driver. But the locking is such that I can't do
that because it will cause a deadlock.

The ringbuffer workaround would be very brittle and if at all only work
by accident, not to mention that it would require duplicating much of
the logbuf logic.

Also I don't consider usage in atomic context a very platform specific
use-case, and I don't understand why I should be required to resort to
some unreliable workaround rather than find a proper fix that guarantees
proper operation.

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

On Thu, Nov 22, 2018 at 10:07:09AM -0600, Jassi Brar wrote:
> Hi Thierry,
> 
> On Thu, Nov 22, 2018 at 2:47 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > On Wed, Nov 21, 2018 at 08:18:22PM -0600, Jassi Brar wrote:
> > > On Wed, Nov 21, 2018 at 8:27 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > >
> > > > On Tue, Nov 20, 2018 at 04:29:07PM +0100, Thierry Reding wrote:
> > > > > On Sat, Nov 17, 2018 at 11:27:17AM -0600, Jassi Brar wrote:
> > > > > > On Mon, Nov 12, 2018 at 9:18 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > > > > >
> > > > > > > From: Thierry Reding <treding@nvidia.com>
> > > > > > >
> > > > > > > The mailbox framework supports blocking transfers via completions for
> > > > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > > > callback that controller drivers can implement to busy loop until the
> > > > > > > transmission has been completed. This will automatically be called when
> > > > > > > available and interrupts are disabled for clients that request blocking
> > > > > > > transfers.
> > > > > > >
> > > > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > > > ---
> > > > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > > > >  2 files changed, 12 insertions(+)
> > > > > > >
> > > > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > > > --- a/drivers/mailbox/mailbox.c
> > > > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > > > >                 unsigned long wait;
> > > > > > >                 int ret;
> > > > > > >
> > > > > > > +               if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > > > +                       ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > > > +                       if (ret < 0)
> > > > > > > +                               tx_tick(chan, ret);
> > > > > > > +
> > > > > > > +                       return ret;
> > > > > > > +               }
> > > > > > > +
> > > > > > This is hacky. I think we can do without busy waiting in atomic
> > > > > > context. You could queue locally while in atomic context and then
> > > > > > transfer in blocking mode. I don't think we should worry about the
> > > > > > 'throughput' as there already is no h/w rate control even with
> > > > > > busy-waiting.
> > > > >
> > > > > I actually tried to do that before I added this flushing mechanism. The
> > > > > problem is, like you said, one of rate control. As mentioned in the
> > > > > cover letter, the shared mailboxes implemented in tegra-hsp are used as
> > > > > RX and TX channels for the TCU, which is like a virtual UART. The TTY
> > > > > driver included as part of this series will use one of the mailboxes to
> > > > > transmit data that is written to the console. The problem is that if
> > > > > these transmissions are not rate-limited on the TTY driver side, the
> > > > > console will just keep writing data and eventually overflow the buffer
> > > > > that we have in the mailbox subsystem.
> > > > >
> > > > > The problem is that data comes in at a much higher rate than what we can
> > > > > output. This is especially true at boot when the TCU console takes over
> > > > > and the whole log buffer is dumped on it.
> > > > >
> > > > > So the only way to rate-limit is to either make mbox_send_message()
> > > > > block, but that can only be done in non-atomic context. The console,
> > > > > however, will always run in atomic context, so the only way to do rate-
> > > > > limiting is by busy looping.
> > > >
> > > > What I also tried before was to implement busy looping within the
> > > > ->send_data() callback of the driver so that we didn't have to put this
> > > > into the core. Unfortunately, however, the ->send_data() callback is
> > > > called under chan->lock, which means that from mbox_send_message() we
> > > > don't have a way to mark the transfer as done. In order to do that we'd
> > > > have to call mbox_chan_txdone(), but that ends up calling tx_tick() and
> > > > that in turn also attempts to take the chan->lock, which would cause a
> > > > deadlock.
> > > >
> > > > The explicit flushing is the best alternative that I could come up with.
> > > > I think it's not all that hacky, because it's very explicit about what's
> > > > going on and it has the nice side-effect that it will allow the mailbox
> > > > to work in interrupt driven mode if possible and only resorting to the
> > > > busy loop in atomic context.
> > > >
> > > > At this point I think I have explored all other options and I frankly
> > > > can't find a more proper way to achieve what we need here. Perhaps you
> > > > can think of additional ways to accomplish this?
> > > >
> > > Well, I would have a local ring buffer (array) of enough size to hold
> > > the characters and then have a task consuming data from that ring
> > > buffer by transmitting over mailbox.
> >
> > There's already such a ringbuffer in the printk code. To implement what
> > you suggest would effectively be creating a copy of that buffer because
> > we'd be allocating the buffer and the console code would just dump each
> > and every character in the logbuf into that ring buffer without rate-
> > limitation.
> >
> Well, the console assumes there exists an atomic path to put character
> on the bus, But because there isn't in case of tcu, we have to emulate
> that. Frankly I prefer the one-off driver jump some hoops, rather than
> implement exceptions in the api.

I wouldn't have any objections to that if the hoops were reasonable
ones. What you're asking me to do is basically implement a second copy
of the logbuf. I don't call that a reasonable hoop to jump through. It
is also not guaranteed to work properly because we can always end up
with a situation where we produce more data than we can consume. Also,
by providing this additional buffer we make things worse because the
standard mechanisms of the logbuf are side-stepped. Typically the logbuf
code will warn if it overflows. In our case we provide a buffer that the
console can dump into, so instead of the logbuf overflowing and warning
about it, we'd now overflow the mailbox buffer and we'd have to add
extra code to warn about overflows.

The console really only works because it assumes that the output driver
will stall and thereby rate-limit.

> BTW, there is already no rate-limitation because its all virtual -
> data is consumed as fast as possible.

No, that's not true. On the receiving end of the TX mailbox is a micro-
processor that reads out the mailbox data. Once it has read the data it
needs to clear the FULL bit so that the TCU driver can write more data.
Even if the microprocessor on the receiving end did buffering (there is
no indication that it does) it would eventually run out of buffer space
and have to stall until it's clocked all the characters out of the
physical UART. At that point no amount of buffering is going to save us
and the only option is to stall, which, again, can currently only be
done from the mailbox, because it is the only one that knows when it is
busy. But it's also the one place where we can't because the framework
doesn't allow it.

> > To make matters worse, the ringbuffer would be empty most of the time
> > after the initial dump of the logbuf, so we'd be wasting all that buffer
> > space.
> >
> The idea is console and uart-ops both feed into this buffer and the
> only consumer thread runs the mailbox.
> 
> > It just seems to me like we should be keeping the TCU driver as close as
> > possible to other UART drivers which also busy loop in order to rate-
> > limit what the console can write. Given the current mailbox framework it
> > is not possible to do that (in interrupt context), so an extension seems
> > like the most sensible option.
> >
> > Perhaps you'd be less concerned about such a change if it was perhaps
> > more explicit? Just throwing ideas around, I think something that could
> > also work is if we explicitly add a mbox_flush() function that would
> > basically be calling ->flush(). That way users of the mailbox can make
> > their requirement very explicit. I haven't actually tested that, but I
> > think it would work. Does that sound more acceptable to you?
> >
> I am happy to see features and bugfixes added to the api. What I am
> not eager about is supporting less than 100% legit and very platform
> specific usecases, especially when there is a work around.

Look, I'd be willing to push this all into the driver, but the framework
doesn't allow me to do that. If it was possible to run the state machine
outside of mbox_send_message() then I'd be able to just move the busy
loop or flush into the driver. But the locking is such that I can't do
that because it will cause a deadlock.

The ringbuffer workaround would be very brittle and if at all only work
by accident, not to mention that it would require duplicating much of
the logbuf logic.

Also I don't consider usage in atomic context a very platform specific
use-case, and I don't understand why I should be required to resort to
some unreliable workaround rather than find a proper fix that guarantees
proper operation.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181122/72a36a13/attachment.sig>

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 4941 bytes --]

On Thu, Nov 22, 2018 at 09:47:12AM +0100, Thierry Reding wrote:
[...]
> Perhaps you'd be less concerned about such a change if it was perhaps
> more explicit? Just throwing ideas around, I think something that could
> also work is if we explicitly add a mbox_flush() function that would
> basically be calling ->flush(). That way users of the mailbox can make
> their requirement very explicit. I haven't actually tested that, but I
> think it would work. Does that sound more acceptable to you?

I tried implementing the explicit flushing on top of this series and it
would look roughly like the below. What do you think?

Thierry

--->8---
diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
index 3e7e2c4358aa..fbdcc82a61ae 100644
--- a/drivers/mailbox/mailbox.c
+++ b/drivers/mailbox/mailbox.c
@@ -267,14 +267,6 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
 		unsigned long wait;
 		int ret;
 
-		if (irqs_disabled() && chan->mbox->ops->flush) {
-			ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
-			if (ret < 0)
-				tx_tick(chan, ret);
-
-			return ret;
-		}
-
 		if (!chan->cl->tx_tout) /* wait forever */
 			wait = msecs_to_jiffies(3600000);
 		else
@@ -291,6 +283,34 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
 }
 EXPORT_SYMBOL_GPL(mbox_send_message);
 
+/**
+ * mbox_flush - flush a mailbox channel
+ * @chan: mailbox channel to flush
+ * @timeout: time, in milliseconds, to allow the flush operation to succeed
+ *
+ * Mailbox controllers that need to work in atomic context can implement the
+ * ->flush() callback to busy loop until a transmission has been completed.
+ * The implementation must call mbox_chan_txdone() upon success. Clients can
+ * call the mbox_flush() function at any time after mbox_send_message() to
+ * flush the transmission. After the function returns success, the mailbox
+ * transmission is guaranteed to have completed.
+ *
+ * Returns: 0 on success or a negative error code on failure.
+ */
+int mbox_flush(struct mbox_chan *chan, unsigned long timeout)
+{
+	int ret;
+
+	if (!chan->mbox->ops->flush)
+		return -ENOTSUPP;
+
+	ret = chan->mbox->ops->flush(chan, timeout);
+	if (ret < 0)
+		tx_tick(chan, ret);
+
+	return ret;
+}
+
 /**
  * mbox_request_channel - Request a mailbox channel.
  * @cl: Identity of the client requesting the channel.
diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
index f99c406cb3cc..e443f6a2ec4b 100644
--- a/drivers/mailbox/tegra-hsp.c
+++ b/drivers/mailbox/tegra-hsp.c
@@ -387,15 +387,13 @@ static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
 
 	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
 
-	if (!irqs_disabled()) {
-		/* enable EMPTY interrupt for the shared mailbox */
-		spin_lock_irqsave(&hsp->lock, flags);
+	/* enable EMPTY interrupt for the shared mailbox */
+	spin_lock_irqsave(&hsp->lock, flags);
 
-		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
-		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
+	hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
 
-		spin_unlock_irqrestore(&hsp->lock, flags);
-	}
+	spin_unlock_irqrestore(&hsp->lock, flags);
 
 	return 0;
 }
diff --git a/drivers/tty/serial/tegra-tcu.c b/drivers/tty/serial/tegra-tcu.c
index 1d360cd03b18..59eaa13e169e 100644
--- a/drivers/tty/serial/tegra-tcu.c
+++ b/drivers/tty/serial/tegra-tcu.c
@@ -57,6 +57,7 @@ static void tegra_tcu_write_one(struct tegra_tcu *tcu, u32 value,
 	value |= TCU_MBOX_NUM_BYTES(count);
 	msg = (void *)(unsigned long)value;
 	mbox_send_message(tcu->tx, msg);
+	mbox_flush(tcu->tx, 1000);
 }
 
 static void tegra_tcu_write(struct tegra_tcu *tcu, const char *s,
@@ -184,9 +185,6 @@ static int tegra_tcu_probe(struct platform_device *pdev)
 		return -ENOMEM;
 
 	tcu->tx_client.dev = &pdev->dev;
-	tcu->tx_client.tx_block = true;
-	tcu->tx_client.tx_tout = 10000;
-	tcu->rx_client.dev = &pdev->dev;
 	tcu->rx_client.rx_callback = tegra_tcu_receive;
 
 	tcu->tx = mbox_request_channel_byname(&tcu->tx_client, "tx");
diff --git a/include/linux/mailbox_client.h b/include/linux/mailbox_client.h
index 44348710953f..faa7da3c9c8b 100644
--- a/include/linux/mailbox_client.h
+++ b/include/linux/mailbox_client.h
@@ -44,6 +44,7 @@ struct mbox_chan *mbox_request_channel_byname(struct mbox_client *cl,
 					      const char *name);
 struct mbox_chan *mbox_request_channel(struct mbox_client *cl, int index);
 int mbox_send_message(struct mbox_chan *chan, void *mssg);
+int mbox_flush(struct mbox_chan *chan, unsigned long timeout);
 void mbox_client_txdone(struct mbox_chan *chan, int r); /* atomic */
 bool mbox_client_peek_data(struct mbox_chan *chan); /* atomic */
 void mbox_free_channel(struct mbox_chan *chan); /* may sleep */

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

On Thu, Nov 22, 2018 at 09:47:12AM +0100, Thierry Reding wrote:
[...]
> Perhaps you'd be less concerned about such a change if it was perhaps
> more explicit? Just throwing ideas around, I think something that could
> also work is if we explicitly add a mbox_flush() function that would
> basically be calling ->flush(). That way users of the mailbox can make
> their requirement very explicit. I haven't actually tested that, but I
> think it would work. Does that sound more acceptable to you?

I tried implementing the explicit flushing on top of this series and it
would look roughly like the below. What do you think?

Thierry

--->8---
diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
index 3e7e2c4358aa..fbdcc82a61ae 100644
--- a/drivers/mailbox/mailbox.c
+++ b/drivers/mailbox/mailbox.c
@@ -267,14 +267,6 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
 		unsigned long wait;
 		int ret;
 
-		if (irqs_disabled() && chan->mbox->ops->flush) {
-			ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
-			if (ret < 0)
-				tx_tick(chan, ret);
-
-			return ret;
-		}
-
 		if (!chan->cl->tx_tout) /* wait forever */
 			wait = msecs_to_jiffies(3600000);
 		else
@@ -291,6 +283,34 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
 }
 EXPORT_SYMBOL_GPL(mbox_send_message);
 
+/**
+ * mbox_flush - flush a mailbox channel
+ * @chan: mailbox channel to flush
+ * @timeout: time, in milliseconds, to allow the flush operation to succeed
+ *
+ * Mailbox controllers that need to work in atomic context can implement the
+ * ->flush() callback to busy loop until a transmission has been completed.
+ * The implementation must call mbox_chan_txdone() upon success. Clients can
+ * call the mbox_flush() function at any time after mbox_send_message() to
+ * flush the transmission. After the function returns success, the mailbox
+ * transmission is guaranteed to have completed.
+ *
+ * Returns: 0 on success or a negative error code on failure.
+ */
+int mbox_flush(struct mbox_chan *chan, unsigned long timeout)
+{
+	int ret;
+
+	if (!chan->mbox->ops->flush)
+		return -ENOTSUPP;
+
+	ret = chan->mbox->ops->flush(chan, timeout);
+	if (ret < 0)
+		tx_tick(chan, ret);
+
+	return ret;
+}
+
 /**
  * mbox_request_channel - Request a mailbox channel.
  * @cl: Identity of the client requesting the channel.
diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
index f99c406cb3cc..e443f6a2ec4b 100644
--- a/drivers/mailbox/tegra-hsp.c
+++ b/drivers/mailbox/tegra-hsp.c
@@ -387,15 +387,13 @@ static int tegra_hsp_mailbox_send_data(struct mbox_chan *chan, void *data)
 
 	tegra_hsp_channel_writel(&mb->channel, value, HSP_SM_SHRD_MBOX);
 
-	if (!irqs_disabled()) {
-		/* enable EMPTY interrupt for the shared mailbox */
-		spin_lock_irqsave(&hsp->lock, flags);
+	/* enable EMPTY interrupt for the shared mailbox */
+	spin_lock_irqsave(&hsp->lock, flags);
 
-		hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
-		tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
+	hsp->mask |= BIT(HSP_INT_EMPTY_SHIFT + mb->index);
+	tegra_hsp_writel(hsp, hsp->mask, HSP_INT_IE(hsp->shared_irq));
 
-		spin_unlock_irqrestore(&hsp->lock, flags);
-	}
+	spin_unlock_irqrestore(&hsp->lock, flags);
 
 	return 0;
 }
diff --git a/drivers/tty/serial/tegra-tcu.c b/drivers/tty/serial/tegra-tcu.c
index 1d360cd03b18..59eaa13e169e 100644
--- a/drivers/tty/serial/tegra-tcu.c
+++ b/drivers/tty/serial/tegra-tcu.c
@@ -57,6 +57,7 @@ static void tegra_tcu_write_one(struct tegra_tcu *tcu, u32 value,
 	value |= TCU_MBOX_NUM_BYTES(count);
 	msg = (void *)(unsigned long)value;
 	mbox_send_message(tcu->tx, msg);
+	mbox_flush(tcu->tx, 1000);
 }
 
 static void tegra_tcu_write(struct tegra_tcu *tcu, const char *s,
@@ -184,9 +185,6 @@ static int tegra_tcu_probe(struct platform_device *pdev)
 		return -ENOMEM;
 
 	tcu->tx_client.dev = &pdev->dev;
-	tcu->tx_client.tx_block = true;
-	tcu->tx_client.tx_tout = 10000;
-	tcu->rx_client.dev = &pdev->dev;
 	tcu->rx_client.rx_callback = tegra_tcu_receive;
 
 	tcu->tx = mbox_request_channel_byname(&tcu->tx_client, "tx");
diff --git a/include/linux/mailbox_client.h b/include/linux/mailbox_client.h
index 44348710953f..faa7da3c9c8b 100644
--- a/include/linux/mailbox_client.h
+++ b/include/linux/mailbox_client.h
@@ -44,6 +44,7 @@ struct mbox_chan *mbox_request_channel_byname(struct mbox_client *cl,
 					      const char *name);
 struct mbox_chan *mbox_request_channel(struct mbox_client *cl, int index);
 int mbox_send_message(struct mbox_chan *chan, void *mssg);
+int mbox_flush(struct mbox_chan *chan, unsigned long timeout);
 void mbox_client_txdone(struct mbox_chan *chan, int r); /* atomic */
 bool mbox_client_peek_data(struct mbox_chan *chan); /* atomic */
 void mbox_free_channel(struct mbox_chan *chan); /* may sleep */
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181123/a7155856/attachment-0001.sig>

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 1241 bytes --]

On Fri, Nov 23, 2018 at 12:17:00PM +0100, Thierry Reding wrote:
> On Thu, Nov 22, 2018 at 09:47:12AM +0100, Thierry Reding wrote:
> [...]
> > Perhaps you'd be less concerned about such a change if it was perhaps
> > more explicit? Just throwing ideas around, I think something that could
> > also work is if we explicitly add a mbox_flush() function that would
> > basically be calling ->flush(). That way users of the mailbox can make
> > their requirement very explicit. I haven't actually tested that, but I
> > think it would work. Does that sound more acceptable to you?
> 
> I tried implementing the explicit flushing on top of this series and it
> would look roughly like the below. What do you think?
> 
> Thierry
> 
> --->8---
> diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
[...]
> @@ -184,9 +185,6 @@ static int tegra_tcu_probe(struct platform_device *pdev)
>  		return -ENOMEM;
>  
>  	tcu->tx_client.dev = &pdev->dev;
> -	tcu->tx_client.tx_block = true;
> -	tcu->tx_client.tx_tout = 10000;
> -	tcu->rx_client.dev = &pdev->dev;

Somehow this line ended up being removed in the diff, but it's actually
required. Only tx_block and tx_tout should be removed in this hunk.

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

On Fri, Nov 23, 2018 at 12:17:00PM +0100, Thierry Reding wrote:
> On Thu, Nov 22, 2018 at 09:47:12AM +0100, Thierry Reding wrote:
> [...]
> > Perhaps you'd be less concerned about such a change if it was perhaps
> > more explicit? Just throwing ideas around, I think something that could
> > also work is if we explicitly add a mbox_flush() function that would
> > basically be calling ->flush(). That way users of the mailbox can make
> > their requirement very explicit. I haven't actually tested that, but I
> > think it would work. Does that sound more acceptable to you?
> 
> I tried implementing the explicit flushing on top of this series and it
> would look roughly like the below. What do you think?
> 
> Thierry
> 
> --->8---
> diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
[...]
> @@ -184,9 +185,6 @@ static int tegra_tcu_probe(struct platform_device *pdev)
>  		return -ENOMEM;
>  
>  	tcu->tx_client.dev = &pdev->dev;
> -	tcu->tx_client.tx_block = true;
> -	tcu->tx_client.tx_tout = 10000;
> -	tcu->rx_client.dev = &pdev->dev;

Somehow this line ended up being removed in the diff, but it's actually
required. Only tx_block and tx_tout should be removed in this hunk.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181123/26eb64d0/attachment.sig>

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jon Hunter]

On 12/11/2018 15:18, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> The mailbox framework supports blocking transfers via completions for
> clients that can sleep. In order to support blocking transfers in cases
> where the transmission is not permitted to sleep, add a new ->flush()
> callback that controller drivers can implement to busy loop until the
> transmission has been completed. This will automatically be called when
> available and interrupts are disabled for clients that request blocking
> transfers.
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
>  drivers/mailbox/mailbox.c          | 8 ++++++++
>  include/linux/mailbox_controller.h | 4 ++++
>  2 files changed, 12 insertions(+)
> 
> diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> index 674b35f402f5..0eaf21259874 100644
> --- a/drivers/mailbox/mailbox.c
> +++ b/drivers/mailbox/mailbox.c
> @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
>  		unsigned long wait;
>  		int ret;
>  
> +		if (irqs_disabled() && chan->mbox->ops->flush) {
> +			ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> +			if (ret < 0)
> +				tx_tick(chan, ret);
> +
> +			return ret;
> +		}

It seems to me that if mbox_send_message() is called from an atomic
context AND tx_block is true, then if 'flush' is not populated this
should be an error condition as we do not wish to call
wait_for_completion from an atomic context.

I understand that there is some debate about adding such flush support,
but irrespective of the above change, it seems to me that if the
mbox_send_message() can be called from an atomic context (which it
appears to), then it should be detecting if someone is trying to do so
with 'tx_block' set as this should be an error.

Furthermore, if the underlying mailbox driver can support sending a
message from an atomic context and busy wait until it is done, surely
the mailbox framework should provide a means to support this?

Now it could be possible for the underlying mailbox driver to detect we
are in an atomic context and if the 'tx_block' is set do the right thing
by busy waiting until the message is sent. However, the problem with
that is, that for the mbox_send_message() to ensure the right thing is
done, it needs to check that 'tx_done' is set in the case of a blocking
transfer in an atomic context. At that point you may as well add the
flush operator as I think it is more implicit/clear.

Cheers
Jon

-- 
nvpublic

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jon Hunter]

On 12/11/2018 15:18, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> The mailbox framework supports blocking transfers via completions for
> clients that can sleep. In order to support blocking transfers in cases
> where the transmission is not permitted to sleep, add a new ->flush()
> callback that controller drivers can implement to busy loop until the
> transmission has been completed. This will automatically be called when
> available and interrupts are disabled for clients that request blocking
> transfers.
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
>  drivers/mailbox/mailbox.c          | 8 ++++++++
>  include/linux/mailbox_controller.h | 4 ++++
>  2 files changed, 12 insertions(+)
> 
> diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> index 674b35f402f5..0eaf21259874 100644
> --- a/drivers/mailbox/mailbox.c
> +++ b/drivers/mailbox/mailbox.c
> @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
>  		unsigned long wait;
>  		int ret;
>  
> +		if (irqs_disabled() && chan->mbox->ops->flush) {
> +			ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> +			if (ret < 0)
> +				tx_tick(chan, ret);
> +
> +			return ret;
> +		}

It seems to me that if mbox_send_message() is called from an atomic
context AND tx_block is true, then if 'flush' is not populated this
should be an error condition as we do not wish to call
wait_for_completion from an atomic context.

I understand that there is some debate about adding such flush support,
but irrespective of the above change, it seems to me that if the
mbox_send_message() can be called from an atomic context (which it
appears to), then it should be detecting if someone is trying to do so
with 'tx_block' set as this should be an error.

Furthermore, if the underlying mailbox driver can support sending a
message from an atomic context and busy wait until it is done, surely
the mailbox framework should provide a means to support this?

Now it could be possible for the underlying mailbox driver to detect we
are in an atomic context and if the 'tx_block' is set do the right thing
by busy waiting until the message is sent. However, the problem with
that is, that for the mbox_send_message() to ensure the right thing is
done, it needs to check that 'tx_done' is set in the case of a blocking
transfer in an atomic context. At that point you may as well add the
flush operator as I think it is more implicit/clear.

Cheers
Jon

-- 
nvpublic

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 3909 bytes --]

On Wed, Nov 28, 2018 at 09:43:29AM +0000, Jon Hunter wrote:
> 
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > The mailbox framework supports blocking transfers via completions for
> > clients that can sleep. In order to support blocking transfers in cases
> > where the transmission is not permitted to sleep, add a new ->flush()
> > callback that controller drivers can implement to busy loop until the
> > transmission has been completed. This will automatically be called when
> > available and interrupts are disabled for clients that request blocking
> > transfers.
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  drivers/mailbox/mailbox.c          | 8 ++++++++
> >  include/linux/mailbox_controller.h | 4 ++++
> >  2 files changed, 12 insertions(+)
> > 
> > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > index 674b35f402f5..0eaf21259874 100644
> > --- a/drivers/mailbox/mailbox.c
> > +++ b/drivers/mailbox/mailbox.c
> > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> >  		unsigned long wait;
> >  		int ret;
> >  
> > +		if (irqs_disabled() && chan->mbox->ops->flush) {
> > +			ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > +			if (ret < 0)
> > +				tx_tick(chan, ret);
> > +
> > +			return ret;
> > +		}
> 
> It seems to me that if mbox_send_message() is called from an atomic
> context AND tx_block is true, then if 'flush' is not populated this
> should be an error condition as we do not wish to call
> wait_for_completion from an atomic context.
> 
> I understand that there is some debate about adding such flush support,
> but irrespective of the above change, it seems to me that if the
> mbox_send_message() can be called from an atomic context (which it
> appears to), then it should be detecting if someone is trying to do so
> with 'tx_block' set as this should be an error.
> 
> Furthermore, if the underlying mailbox driver can support sending a
> message from an atomic context and busy wait until it is done, surely
> the mailbox framework should provide a means to support this?
> 
> Now it could be possible for the underlying mailbox driver to detect we
> are in an atomic context and if the 'tx_block' is set do the right thing
> by busy waiting until the message is sent. However, the problem with
> that is, that for the mbox_send_message() to ensure the right thing is
> done, it needs to check that 'tx_done' is set in the case of a blocking
> transfer in an atomic context. At that point you may as well add the
> flush operator as I think it is more implicit/clear.

Heh, interesting timing, I just sent out v3 of this series with a
slightly different solution. Basically what v3 implements is explicit
flushing of the mailbox via a new function called mbox_flush().

I originally liked the idea of "hiding" the flush operation in the
mbox_send_message() function, but the more I look at the new explicit
flush solution, the more I prefer it. One reason why I prefer it is
because it no longer has the slightly odd "irqs_disabled()" check that
always made me a bit uneasy. The other advantage of the explicit flush
is that it becomes completely opt-in for both mailbox providers and
consumers. Furthermore the consumer should always know if it can be
called in atomic context or not, so none of these code paths should
have to be "clever" and check at runtime whether or not interrupts are
enabled. If the consumer knows that it will be called in atomic context
it can just unconditionally flush the mailbox.

Also, it nicely addresses the concern about erroring out if the provider
doesn't support flushing. The new mbox_flush() API returns -ENOTSUPP if
called on a mailbox channel provided by a driver that doesn't implement
->flush().

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

On Wed, Nov 28, 2018 at 09:43:29AM +0000, Jon Hunter wrote:
> 
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > The mailbox framework supports blocking transfers via completions for
> > clients that can sleep. In order to support blocking transfers in cases
> > where the transmission is not permitted to sleep, add a new ->flush()
> > callback that controller drivers can implement to busy loop until the
> > transmission has been completed. This will automatically be called when
> > available and interrupts are disabled for clients that request blocking
> > transfers.
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  drivers/mailbox/mailbox.c          | 8 ++++++++
> >  include/linux/mailbox_controller.h | 4 ++++
> >  2 files changed, 12 insertions(+)
> > 
> > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > index 674b35f402f5..0eaf21259874 100644
> > --- a/drivers/mailbox/mailbox.c
> > +++ b/drivers/mailbox/mailbox.c
> > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> >  		unsigned long wait;
> >  		int ret;
> >  
> > +		if (irqs_disabled() && chan->mbox->ops->flush) {
> > +			ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > +			if (ret < 0)
> > +				tx_tick(chan, ret);
> > +
> > +			return ret;
> > +		}
> 
> It seems to me that if mbox_send_message() is called from an atomic
> context AND tx_block is true, then if 'flush' is not populated this
> should be an error condition as we do not wish to call
> wait_for_completion from an atomic context.
> 
> I understand that there is some debate about adding such flush support,
> but irrespective of the above change, it seems to me that if the
> mbox_send_message() can be called from an atomic context (which it
> appears to), then it should be detecting if someone is trying to do so
> with 'tx_block' set as this should be an error.
> 
> Furthermore, if the underlying mailbox driver can support sending a
> message from an atomic context and busy wait until it is done, surely
> the mailbox framework should provide a means to support this?
> 
> Now it could be possible for the underlying mailbox driver to detect we
> are in an atomic context and if the 'tx_block' is set do the right thing
> by busy waiting until the message is sent. However, the problem with
> that is, that for the mbox_send_message() to ensure the right thing is
> done, it needs to check that 'tx_done' is set in the case of a blocking
> transfer in an atomic context. At that point you may as well add the
> flush operator as I think it is more implicit/clear.

Heh, interesting timing, I just sent out v3 of this series with a
slightly different solution. Basically what v3 implements is explicit
flushing of the mailbox via a new function called mbox_flush().

I originally liked the idea of "hiding" the flush operation in the
mbox_send_message() function, but the more I look at the new explicit
flush solution, the more I prefer it. One reason why I prefer it is
because it no longer has the slightly odd "irqs_disabled()" check that
always made me a bit uneasy. The other advantage of the explicit flush
is that it becomes completely opt-in for both mailbox providers and
consumers. Furthermore the consumer should always know if it can be
called in atomic context or not, so none of these code paths should
have to be "clever" and check at runtime whether or not interrupts are
enabled. If the consumer knows that it will be called in atomic context
it can just unconditionally flush the mailbox.

Also, it nicely addresses the concern about erroring out if the provider
doesn't support flushing. The new mbox_flush() API returns -ENOTSUPP if
called on a mailbox channel provided by a driver that doesn't implement
->flush().

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20181128/43125f83/attachment.sig>

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
>
>
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> >
> > The mailbox framework supports blocking transfers via completions for
> > clients that can sleep. In order to support blocking transfers in cases
> > where the transmission is not permitted to sleep, add a new ->flush()
> > callback that controller drivers can implement to busy loop until the
> > transmission has been completed. This will automatically be called when
> > available and interrupts are disabled for clients that request blocking
> > transfers.
> >
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  drivers/mailbox/mailbox.c          | 8 ++++++++
> >  include/linux/mailbox_controller.h | 4 ++++
> >  2 files changed, 12 insertions(+)
> >
> > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > index 674b35f402f5..0eaf21259874 100644
> > --- a/drivers/mailbox/mailbox.c
> > +++ b/drivers/mailbox/mailbox.c
> > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> >               unsigned long wait;
> >               int ret;
> >
> > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > +                     if (ret < 0)
> > +                             tx_tick(chan, ret);
> > +
> > +                     return ret;
> > +             }
>
> It seems to me that if mbox_send_message() is called from an atomic
> context AND tx_block is true, then if 'flush' is not populated this
> should be an error condition as we do not wish to call
> wait_for_completion from an atomic context.
>
> I understand that there is some debate about adding such flush support,
> but irrespective of the above change, it seems to me that if the
> mbox_send_message() can be called from an atomic context (which it
> appears to), then it should be detecting if someone is trying to do so
> with 'tx_block' set as this should be an error.
>
Layers within kernel space have to trust each other. A badly written
client can break the consumer in so many ways, we can not catch every
possibility.

> Furthermore, if the underlying mailbox driver can support sending a
> message from an atomic context and busy wait until it is done, surely
> the mailbox framework should provide a means to support this?
>
Being able to put the message on bus in atomic context is a feature -
which we do support. But busy-waiting in a loop is not a feature, and
we don't want to encourage that.

Cheers!

[PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
>
>
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> >
> > The mailbox framework supports blocking transfers via completions for
> > clients that can sleep. In order to support blocking transfers in cases
> > where the transmission is not permitted to sleep, add a new ->flush()
> > callback that controller drivers can implement to busy loop until the
> > transmission has been completed. This will automatically be called when
> > available and interrupts are disabled for clients that request blocking
> > transfers.
> >
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  drivers/mailbox/mailbox.c          | 8 ++++++++
> >  include/linux/mailbox_controller.h | 4 ++++
> >  2 files changed, 12 insertions(+)
> >
> > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > index 674b35f402f5..0eaf21259874 100644
> > --- a/drivers/mailbox/mailbox.c
> > +++ b/drivers/mailbox/mailbox.c
> > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> >               unsigned long wait;
> >               int ret;
> >
> > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > +                     if (ret < 0)
> > +                             tx_tick(chan, ret);
> > +
> > +                     return ret;
> > +             }
>
> It seems to me that if mbox_send_message() is called from an atomic
> context AND tx_block is true, then if 'flush' is not populated this
> should be an error condition as we do not wish to call
> wait_for_completion from an atomic context.
>
> I understand that there is some debate about adding such flush support,
> but irrespective of the above change, it seems to me that if the
> mbox_send_message() can be called from an atomic context (which it
> appears to), then it should be detecting if someone is trying to do so
> with 'tx_block' set as this should be an error.
>
Layers within kernel space have to trust each other. A badly written
client can break the consumer in so many ways, we can not catch every
possibility.

> Furthermore, if the underlying mailbox driver can support sending a
> message from an atomic context and busy wait until it is done, surely
> the mailbox framework should provide a means to support this?
>
Being able to put the message on bus in atomic context is a feature -
which we do support. But busy-waiting in a loop is not a feature, and
we don't want to encourage that.

Cheers!

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 5944 bytes --]

On Wed, Nov 28, 2018 at 11:23:36PM -0600, Jassi Brar wrote:
> On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
> >
> >
> > On 12/11/2018 15:18, Thierry Reding wrote:
> > > From: Thierry Reding <treding@nvidia.com>
> > >
> > > The mailbox framework supports blocking transfers via completions for
> > > clients that can sleep. In order to support blocking transfers in cases
> > > where the transmission is not permitted to sleep, add a new ->flush()
> > > callback that controller drivers can implement to busy loop until the
> > > transmission has been completed. This will automatically be called when
> > > available and interrupts are disabled for clients that request blocking
> > > transfers.
> > >
> > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > ---
> > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > >  include/linux/mailbox_controller.h | 4 ++++
> > >  2 files changed, 12 insertions(+)
> > >
> > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > index 674b35f402f5..0eaf21259874 100644
> > > --- a/drivers/mailbox/mailbox.c
> > > +++ b/drivers/mailbox/mailbox.c
> > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > >               unsigned long wait;
> > >               int ret;
> > >
> > > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > +                     if (ret < 0)
> > > +                             tx_tick(chan, ret);
> > > +
> > > +                     return ret;
> > > +             }
> >
> > It seems to me that if mbox_send_message() is called from an atomic
> > context AND tx_block is true, then if 'flush' is not populated this
> > should be an error condition as we do not wish to call
> > wait_for_completion from an atomic context.
> >
> > I understand that there is some debate about adding such flush support,
> > but irrespective of the above change, it seems to me that if the
> > mbox_send_message() can be called from an atomic context (which it
> > appears to), then it should be detecting if someone is trying to do so
> > with 'tx_block' set as this should be an error.
> >
> Layers within kernel space have to trust each other. A badly written
> client can break the consumer in so many ways, we can not catch every
> possibility.
> 
> > Furthermore, if the underlying mailbox driver can support sending a
> > message from an atomic context and busy wait until it is done, surely
> > the mailbox framework should provide a means to support this?
> >
> Being able to put the message on bus in atomic context is a feature -
> which we do support. But busy-waiting in a loop is not a feature, and
> we don't want to encourage that.

I agree that in generally busy-waiting is a bad idea and shouldn't be
encouraged. However, I also think that an exception proves the rule. If
you look at the console drivers in drivers/tty/serial, all of them will
busy loop prior to or after sending a character. This is pretty much
part of the API and as such busy-looping is very much a feature.

The reason why this is done is because on one hand we have an atomic
context and on the other hand we want to make sure that all characters
actually make it to the console when we print them.

As an example how this can break, I've taken your suggestion to
implement a producer/consumer mode in the TCU driver where the console
write function will just stash characters into a circular buffer and a
work queue will then use mbox_send_message() to drain the circular
buffer. While this does work on the surface, I was able to concern both
of the issues that I was concerned about: 1) it is easy to overflow the
circular buffer by just dumping enough data at once to the console and
2) when a crash happens, everything in the kernel stops, including the
consumer workqueue that is supposed to drain the circular buffer and
flush messages to the TCU. The result is that, in my particular case,
the boot log will just stop at a random place in the middle of messages
from much earlier in the boot because the TCU hasn't caught up yet and
there's a lot of characters still in the circular buffer.

Now, 1) can be mitigated by increasing the circular buffer size. A value
that seems to give reliably good results in 2 << CONFIG_LOG_BUF_SHIFT. I
thought that I could also mitigate 2) by busy looping in the TCU driver,
but it turns out that that doesn't work. The reason is that since we are
in atomic context with all interrupts disabled, the mailbox won't ever
consume any new characters, so the read pointer in the circular buffer
won't increment, leaving me with no condition upon which to loop that
would work.

Given that 2) can't be solved makes this implementation of the TCU
completely useless as a console.

All of that said, have you had a chance to look at my other proposed
solution for this? I sent it out yesterday, but you can also find the
series here:

	http://patchwork.ozlabs.org/project/linux-tegra/list/?series=78477

The difference to the earlier versions is that the flushing is now
explicit. I think this combines the best of both worlds. On one hand it
makes the mechanism completely opt-in, so nothing gets hidden in the
regular functions. On the other hand, it allows clients to make use of
this functionality very explicitly. A client that doesn't call the
mbox_flush() function will just not busy-loop. But clients that need to
make sure messages are flushed in atomic contexts can now do that. Does
that sound like a more acceptable solution to you? We could even go and
add documentation to mbox_flush() that it should only be used under
special circumstances.

If you still think that's a bad idea, do you have any other suggestions
on how to move forward?

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Thu, Nov 29, 2018 at 9:23 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Wed, Nov 28, 2018 at 11:23:36PM -0600, Jassi Brar wrote:
> > On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
> > >
> > >
> > > On 12/11/2018 15:18, Thierry Reding wrote:
> > > > From: Thierry Reding <treding@nvidia.com>
> > > >
> > > > The mailbox framework supports blocking transfers via completions for
> > > > clients that can sleep. In order to support blocking transfers in cases
> > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > callback that controller drivers can implement to busy loop until the
> > > > transmission has been completed. This will automatically be called when
> > > > available and interrupts are disabled for clients that request blocking
> > > > transfers.
> > > >
> > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > ---
> > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > >  include/linux/mailbox_controller.h | 4 ++++
> > > >  2 files changed, 12 insertions(+)
> > > >
> > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > index 674b35f402f5..0eaf21259874 100644
> > > > --- a/drivers/mailbox/mailbox.c
> > > > +++ b/drivers/mailbox/mailbox.c
> > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > >               unsigned long wait;
> > > >               int ret;
> > > >
> > > > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > +                     if (ret < 0)
> > > > +                             tx_tick(chan, ret);
> > > > +
> > > > +                     return ret;
> > > > +             }
> > >
> > > It seems to me that if mbox_send_message() is called from an atomic
> > > context AND tx_block is true, then if 'flush' is not populated this
> > > should be an error condition as we do not wish to call
> > > wait_for_completion from an atomic context.
> > >
> > > I understand that there is some debate about adding such flush support,
> > > but irrespective of the above change, it seems to me that if the
> > > mbox_send_message() can be called from an atomic context (which it
> > > appears to), then it should be detecting if someone is trying to do so
> > > with 'tx_block' set as this should be an error.
> > >
> > Layers within kernel space have to trust each other. A badly written
> > client can break the consumer in so many ways, we can not catch every
> > possibility.
> >
> > > Furthermore, if the underlying mailbox driver can support sending a
> > > message from an atomic context and busy wait until it is done, surely
> > > the mailbox framework should provide a means to support this?
> > >
> > Being able to put the message on bus in atomic context is a feature -
> > which we do support. But busy-waiting in a loop is not a feature, and
> > we don't want to encourage that.
>
> I agree that in generally busy-waiting is a bad idea and shouldn't be
> encouraged. However, I also think that an exception proves the rule. If
> you look at the console drivers in drivers/tty/serial, all of them will
> busy loop prior to or after sending a character. This is pretty much
> part of the API and as such busy-looping is very much a feature.
>
> The reason why this is done is because on one hand we have an atomic
> context and on the other hand we want to make sure that all characters
> actually make it to the console when we print them.
>
> As an example how this can break, I've taken your suggestion to
> implement a producer/consumer mode in the TCU driver where the console
> write function will just stash characters into a circular buffer and a
> work queue will then use mbox_send_message() to drain the circular
> buffer. While this does work on the surface, I was able to concern both
> of the issues that I was concerned about: 1) it is easy to overflow the
> circular buffer by just dumping enough data at once to the console and
> 2) when a crash happens, everything in the kernel stops, including the
> consumer workqueue that is supposed to drain the circular buffer and
> flush messages to the TCU. The result is that, in my particular case,
> the boot log will just stop at a random place in the middle of messages
> from much earlier in the boot because the TCU hasn't caught up yet and
> there's a lot of characters still in the circular buffer.
>
> Now, 1) can be mitigated by increasing the circular buffer size. A value
> that seems to give reliably good results in 2 << CONFIG_LOG_BUF_SHIFT.
>
Yes please.

> I thought that I could also mitigate 2) by busy looping in the TCU driver,
> but it turns out that that doesn't work. The reason is that since we are
> in atomic context with all interrupts disabled, the mailbox won't ever
> consume any new characters, so the read pointer in the circular buffer
> won't increment, leaving me with no condition upon which to loop that
> would work.
>
So you want to be able to rely on an emulated console (running on a
totally different subsystem) to dump development-phase early-boot
logs? At the cost of legitimizing busy looping in atomic context - one
random driver messing up the api for ever. Maybe you could have the
ring buffer in some shmem and only pass the number of valid characters
in it, to the remote?

>
>         http://patchwork.ozlabs.org/project/linux-tegra/list/?series=78477
>
> The difference to the earlier versions is that the flushing is now
> explicit. I think this combines the best of both worlds. On one hand it
> makes the mechanism completely opt-in, so nothing gets hidden in the
> regular functions. On the other hand, it allows clients to make use of
> this functionality very explicitly. A client that doesn't call the
> mbox_flush() function will just not busy-loop. But clients that need to
> make sure messages are flushed in atomic contexts can now do that. Does
> that sound like a more acceptable solution to you? We could even go and
> add documentation to mbox_flush() that it should only be used under
> special circumstances.
>
> If you still think that's a bad idea, do you have any other suggestions
> on how to move forward?
>
It would help if other maintainers chime in if a subsystem should
support busy-wait in atomic context for a one-off driver.

Regards.

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Thu, Nov 29, 2018 at 9:23 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Wed, Nov 28, 2018 at 11:23:36PM -0600, Jassi Brar wrote:
> > On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
> > >
> > >
> > > On 12/11/2018 15:18, Thierry Reding wrote:
> > > > From: Thierry Reding <treding@nvidia.com>
> > > >
> > > > The mailbox framework supports blocking transfers via completions for
> > > > clients that can sleep. In order to support blocking transfers in cases
> > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > callback that controller drivers can implement to busy loop until the
> > > > transmission has been completed. This will automatically be called when
> > > > available and interrupts are disabled for clients that request blocking
> > > > transfers.
> > > >
> > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > ---
> > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > >  include/linux/mailbox_controller.h | 4 ++++
> > > >  2 files changed, 12 insertions(+)
> > > >
> > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > index 674b35f402f5..0eaf21259874 100644
> > > > --- a/drivers/mailbox/mailbox.c
> > > > +++ b/drivers/mailbox/mailbox.c
> > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > >               unsigned long wait;
> > > >               int ret;
> > > >
> > > > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > +                     if (ret < 0)
> > > > +                             tx_tick(chan, ret);
> > > > +
> > > > +                     return ret;
> > > > +             }
> > >
> > > It seems to me that if mbox_send_message() is called from an atomic
> > > context AND tx_block is true, then if 'flush' is not populated this
> > > should be an error condition as we do not wish to call
> > > wait_for_completion from an atomic context.
> > >
> > > I understand that there is some debate about adding such flush support,
> > > but irrespective of the above change, it seems to me that if the
> > > mbox_send_message() can be called from an atomic context (which it
> > > appears to), then it should be detecting if someone is trying to do so
> > > with 'tx_block' set as this should be an error.
> > >
> > Layers within kernel space have to trust each other. A badly written
> > client can break the consumer in so many ways, we can not catch every
> > possibility.
> >
> > > Furthermore, if the underlying mailbox driver can support sending a
> > > message from an atomic context and busy wait until it is done, surely
> > > the mailbox framework should provide a means to support this?
> > >
> > Being able to put the message on bus in atomic context is a feature -
> > which we do support. But busy-waiting in a loop is not a feature, and
> > we don't want to encourage that.
>
> I agree that in generally busy-waiting is a bad idea and shouldn't be
> encouraged. However, I also think that an exception proves the rule. If
> you look at the console drivers in drivers/tty/serial, all of them will
> busy loop prior to or after sending a character. This is pretty much
> part of the API and as such busy-looping is very much a feature.
>
> The reason why this is done is because on one hand we have an atomic
> context and on the other hand we want to make sure that all characters
> actually make it to the console when we print them.
>
> As an example how this can break, I've taken your suggestion to
> implement a producer/consumer mode in the TCU driver where the console
> write function will just stash characters into a circular buffer and a
> work queue will then use mbox_send_message() to drain the circular
> buffer. While this does work on the surface, I was able to concern both
> of the issues that I was concerned about: 1) it is easy to overflow the
> circular buffer by just dumping enough data at once to the console and
> 2) when a crash happens, everything in the kernel stops, including the
> consumer workqueue that is supposed to drain the circular buffer and
> flush messages to the TCU. The result is that, in my particular case,
> the boot log will just stop at a random place in the middle of messages
> from much earlier in the boot because the TCU hasn't caught up yet and
> there's a lot of characters still in the circular buffer.
>
> Now, 1) can be mitigated by increasing the circular buffer size. A value
> that seems to give reliably good results in 2 << CONFIG_LOG_BUF_SHIFT.
>
Yes please.

> I thought that I could also mitigate 2) by busy looping in the TCU driver,
> but it turns out that that doesn't work. The reason is that since we are
> in atomic context with all interrupts disabled, the mailbox won't ever
> consume any new characters, so the read pointer in the circular buffer
> won't increment, leaving me with no condition upon which to loop that
> would work.
>
So you want to be able to rely on an emulated console (running on a
totally different subsystem) to dump development-phase early-boot
logs? At the cost of legitimizing busy looping in atomic context - one
random driver messing up the api for ever. Maybe you could have the
ring buffer in some shmem and only pass the number of valid characters
in it, to the remote?

>
>         http://patchwork.ozlabs.org/project/linux-tegra/list/?series=78477
>
> The difference to the earlier versions is that the flushing is now
> explicit. I think this combines the best of both worlds. On one hand it
> makes the mechanism completely opt-in, so nothing gets hidden in the
> regular functions. On the other hand, it allows clients to make use of
> this functionality very explicitly. A client that doesn't call the
> mbox_flush() function will just not busy-loop. But clients that need to
> make sure messages are flushed in atomic contexts can now do that. Does
> that sound like a more acceptable solution to you? We could even go and
> add documentation to mbox_flush() that it should only be used under
> special circumstances.
>
> If you still think that's a bad idea, do you have any other suggestions
> on how to move forward?
>
It would help if other maintainers chime in if a subsystem should
support busy-wait in atomic context for a one-off driver.

Regards.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Mikko Perttunen]

On 07/12/2018 11.26, Jassi Brar wrote:
>> I thought that I could also mitigate 2) by busy looping in the TCU driver,
>> but it turns out that that doesn't work. The reason is that since we are
>> in atomic context with all interrupts disabled, the mailbox won't ever
>> consume any new characters, so the read pointer in the circular buffer
>> won't increment, leaving me with no condition upon which to loop that
>> would work.
>>
> So you want to be able to rely on an emulated console (running on a
> totally different subsystem) to dump development-phase early-boot
> logs? At the cost of legitimizing busy looping in atomic context - one
> random driver messing up the api for ever. Maybe you could have the
> ring buffer in some shmem and only pass the number of valid characters
> in it, to the remote?
> 

I would like to note that this is the one and only console interface 
that exists on these systems, for both development phase and production. 
Other UARTs are not externally accessible on the systems, or they are 
comparatively difficult to access as they aren't intended to be used as 
consoles in the system design. The combination of hardware and firmware 
implementing the console is black box from software's point of view 
(similarly to any other UART HW). The interface has also been fixed at 
an early system design phase, as there are many operating systems 
running on these boards, each with their own drivers.

Cheers,
Mikko

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Mikko Perttunen]

On 07/12/2018 11.26, Jassi Brar wrote:
>> I thought that I could also mitigate 2) by busy looping in the TCU driver,
>> but it turns out that that doesn't work. The reason is that since we are
>> in atomic context with all interrupts disabled, the mailbox won't ever
>> consume any new characters, so the read pointer in the circular buffer
>> won't increment, leaving me with no condition upon which to loop that
>> would work.
>>
> So you want to be able to rely on an emulated console (running on a
> totally different subsystem) to dump development-phase early-boot
> logs? At the cost of legitimizing busy looping in atomic context - one
> random driver messing up the api for ever. Maybe you could have the
> ring buffer in some shmem and only pass the number of valid characters
> in it, to the remote?
> 

I would like to note that this is the one and only console interface 
that exists on these systems, for both development phase and production. 
Other UARTs are not externally accessible on the systems, or they are 
comparatively difficult to access as they aren't intended to be used as 
consoles in the system design. The combination of hardware and firmware 
implementing the console is black box from software's point of view 
(similarly to any other UART HW). The interface has also been fixed at 
an early system design phase, as there are many operating systems 
running on these boards, each with their own drivers.

Cheers,
Mikko

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 10848 bytes --]

On Thu, Dec 06, 2018 at 11:56:25PM -0600, Jassi Brar wrote:
> On Thu, Nov 29, 2018 at 9:23 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > On Wed, Nov 28, 2018 at 11:23:36PM -0600, Jassi Brar wrote:
> > > On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
> > > >
> > > >
> > > > On 12/11/2018 15:18, Thierry Reding wrote:
> > > > > From: Thierry Reding <treding@nvidia.com>
> > > > >
> > > > > The mailbox framework supports blocking transfers via completions for
> > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > callback that controller drivers can implement to busy loop until the
> > > > > transmission has been completed. This will automatically be called when
> > > > > available and interrupts are disabled for clients that request blocking
> > > > > transfers.
> > > > >
> > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > ---
> > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > >  2 files changed, 12 insertions(+)
> > > > >
> > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > --- a/drivers/mailbox/mailbox.c
> > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > >               unsigned long wait;
> > > > >               int ret;
> > > > >
> > > > > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > +                     if (ret < 0)
> > > > > +                             tx_tick(chan, ret);
> > > > > +
> > > > > +                     return ret;
> > > > > +             }
> > > >
> > > > It seems to me that if mbox_send_message() is called from an atomic
> > > > context AND tx_block is true, then if 'flush' is not populated this
> > > > should be an error condition as we do not wish to call
> > > > wait_for_completion from an atomic context.
> > > >
> > > > I understand that there is some debate about adding such flush support,
> > > > but irrespective of the above change, it seems to me that if the
> > > > mbox_send_message() can be called from an atomic context (which it
> > > > appears to), then it should be detecting if someone is trying to do so
> > > > with 'tx_block' set as this should be an error.
> > > >
> > > Layers within kernel space have to trust each other. A badly written
> > > client can break the consumer in so many ways, we can not catch every
> > > possibility.
> > >
> > > > Furthermore, if the underlying mailbox driver can support sending a
> > > > message from an atomic context and busy wait until it is done, surely
> > > > the mailbox framework should provide a means to support this?
> > > >
> > > Being able to put the message on bus in atomic context is a feature -
> > > which we do support. But busy-waiting in a loop is not a feature, and
> > > we don't want to encourage that.
> >
> > I agree that in generally busy-waiting is a bad idea and shouldn't be
> > encouraged. However, I also think that an exception proves the rule. If
> > you look at the console drivers in drivers/tty/serial, all of them will
> > busy loop prior to or after sending a character. This is pretty much
> > part of the API and as such busy-looping is very much a feature.
> >
> > The reason why this is done is because on one hand we have an atomic
> > context and on the other hand we want to make sure that all characters
> > actually make it to the console when we print them.
> >
> > As an example how this can break, I've taken your suggestion to
> > implement a producer/consumer mode in the TCU driver where the console
> > write function will just stash characters into a circular buffer and a
> > work queue will then use mbox_send_message() to drain the circular
> > buffer. While this does work on the surface, I was able to concern both
> > of the issues that I was concerned about: 1) it is easy to overflow the
> > circular buffer by just dumping enough data at once to the console and
> > 2) when a crash happens, everything in the kernel stops, including the
> > consumer workqueue that is supposed to drain the circular buffer and
> > flush messages to the TCU. The result is that, in my particular case,
> > the boot log will just stop at a random place in the middle of messages
> > from much earlier in the boot because the TCU hasn't caught up yet and
> > there's a lot of characters still in the circular buffer.
> >
> > Now, 1) can be mitigated by increasing the circular buffer size. A value
> > that seems to give reliably good results in 2 << CONFIG_LOG_BUF_SHIFT.
> >
> Yes please.

As I explained elsewhere, I actually went and implemented this. But
given the nature of buffering, this ends up making the TCU completely
useless as a console because in case of a crash, the system will stop
working with a large number of characters still stuck in the buffer.
And that's especially bad in case of a crash because those last
characters that get stuck in the buffer are the most relevant ones
because they contain the stack dump.

> > I thought that I could also mitigate 2) by busy looping in the TCU driver,
> > but it turns out that that doesn't work. The reason is that since we are
> > in atomic context with all interrupts disabled, the mailbox won't ever
> > consume any new characters, so the read pointer in the circular buffer
> > won't increment, leaving me with no condition upon which to loop that
> > would work.
> >
> So you want to be able to rely on an emulated console (running on a
> totally different subsystem) to dump development-phase early-boot
> logs? At the cost of legitimizing busy looping in atomic context - one
> random driver messing up the api for ever. Maybe you could have the
> ring buffer in some shmem and only pass the number of valid characters
> in it, to the remote?

First of all, this is not about development-phase early-boot messages.
We're talking about the system console here. This is what everyone will
want to be using when developing on this device. Sure at some point you
may end up with a system that works and you can have the console on the
network or an attached display or whatever, but even then you may still
want to attach to the console if you ever run into issues where the
system doesn't come up.

Secondly, I don't understand why you think this is an emulated console.
The way that this works is that there's a microprocessor in the system
that interacts with other microprocessors via shared mailboxes to
receive log messages. That microprocessor collects these log messages
and outputs them on a physical UART via multiplexed streams. The host
system can connect to that physical UART and demultiplex these streams
to get the individual log messages from each of the components in the
system.

Lastly, I don't understand why you think we're messing up the API here.
The proposal in this series doesn't even change any of the API, but it
makes it aware of the state of interrupts internally so that it can do
the right thing depending on the call stack. The other proposal, in v3,
is more explicit in that it adds new API to flush the mailbox. The new
API is completely optional and I even offered to document it as being
discouraged because it involves busy looping. At the same time it does
solve a real problem and it doesn't impact any existing mailbox drivers
nor any of their users (because it is completely opt-in).

While I can somewhat relate to your reluctance to extend the API, I do
not see a way around it. Sure, you could decide that this is something
that Linux just won't support, but that would be a first. I'm not aware
of any cases where a concious decision was ever made not to support a
feature because it didn't fit into any existing frameworks. Typically
we deal with this by improving things so that we can support the
additional use-cases. It's really one of the strong points of Linux.

I'm open to any suggestions on how this could be done better. You had
already suggested the ring buffer and I did go and implement it, but it
only confirmed the concerns that I've had with it all along. I realize
that at this point I may be blind to other obvious solutions, but I've
done my best to come up with alternatives and none of them work. If you
have any other ideas, please do share them and I will investigate.

> >         http://patchwork.ozlabs.org/project/linux-tegra/list/?series=78477
> >
> > The difference to the earlier versions is that the flushing is now
> > explicit. I think this combines the best of both worlds. On one hand it
> > makes the mechanism completely opt-in, so nothing gets hidden in the
> > regular functions. On the other hand, it allows clients to make use of
> > this functionality very explicitly. A client that doesn't call the
> > mbox_flush() function will just not busy-loop. But clients that need to
> > make sure messages are flushed in atomic contexts can now do that. Does
> > that sound like a more acceptable solution to you? We could even go and
> > add documentation to mbox_flush() that it should only be used under
> > special circumstances.
> >
> > If you still think that's a bad idea, do you have any other suggestions
> > on how to move forward?
> >
> It would help if other maintainers chime in if a subsystem should
> support busy-wait in atomic context for a one-off driver.

Just as an additional note: it's not the driver that actually requires
the busy looping, it's the use-case (i.e. the consumer). The driver is
working perfectly fine with interrupts enabled, it's just that in the
particular use-case of the console that we have no way of detecting if
or when an interrupt occurred.

Greg,

any ideas on how we can move forward here? For reasons given elsewhere
in this thread I understand that there is no way to make the console
code run in non-atomic context. Have you ever run into a case where the
console driver couldn't busy-loop? Were there any solutions to this?

I've looked through quite a few drivers and they all end up with a busy
loop, waiting for the transmission FIFO to become empty. There are also
a few implementations for hypervisors that call back into some firmware
in order to send the characters, but I expect those to do the busy
looping in the firmware.

Perhaps the most prominent case that I came across and that is quite
similar to this discussion is netconsole. There's a lot of code in the
network layer that exists only to allow using a network device for
outputting a console. I don't think the changes to the mailbox
framework are anywhere near as complicated.

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Greg KH]

On Fri, Dec 07, 2018 at 12:32:45PM +0100, Thierry Reding wrote:
> Greg,
> 
> any ideas on how we can move forward here? For reasons given elsewhere
> in this thread I understand that there is no way to make the console
> code run in non-atomic context. Have you ever run into a case where the
> console driver couldn't busy-loop? Were there any solutions to this?

I don't know of any such cases, hardware sucks at times, and we just
have to deal with getting it to work by doing stuff like this :(

> I've looked through quite a few drivers and they all end up with a busy
> loop, waiting for the transmission FIFO to become empty. There are also
> a few implementations for hypervisors that call back into some firmware
> in order to send the characters, but I expect those to do the busy
> looping in the firmware.

busy loops are ok here, as you point out.

> Perhaps the most prominent case that I came across and that is quite
> similar to this discussion is netconsole. There's a lot of code in the
> network layer that exists only to allow using a network device for
> outputting a console. I don't think the changes to the mailbox
> framework are anywhere near as complicated.

Neither do I, I have no objection to your changes at all.

thanks,

greg k-h

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Greg KH]

On Fri, Dec 07, 2018 at 12:32:45PM +0100, Thierry Reding wrote:
> Greg,
> 
> any ideas on how we can move forward here? For reasons given elsewhere
> in this thread I understand that there is no way to make the console
> code run in non-atomic context. Have you ever run into a case where the
> console driver couldn't busy-loop? Were there any solutions to this?

I don't know of any such cases, hardware sucks at times, and we just
have to deal with getting it to work by doing stuff like this :(

> I've looked through quite a few drivers and they all end up with a busy
> loop, waiting for the transmission FIFO to become empty. There are also
> a few implementations for hypervisors that call back into some firmware
> in order to send the characters, but I expect those to do the busy
> looping in the firmware.

busy loops are ok here, as you point out.

> Perhaps the most prominent case that I came across and that is quite
> similar to this discussion is netconsole. There's a lot of code in the
> network layer that exists only to allow using a network device for
> outputting a console. I don't think the changes to the mailbox
> framework are anywhere near as complicated.

Neither do I, I have no objection to your changes at all.

thanks,

greg k-h

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Fri, Dec 7, 2018 at 11:50 AM Mikko Perttunen <cyndis@kapsi.fi> wrote:
>
> On 07/12/2018 11.26, Jassi Brar wrote:
> >> I thought that I could also mitigate 2) by busy looping in the TCU driver,
> >> but it turns out that that doesn't work. The reason is that since we are
> >> in atomic context with all interrupts disabled, the mailbox won't ever
> >> consume any new characters, so the read pointer in the circular buffer
> >> won't increment, leaving me with no condition upon which to loop that
> >> would work.
> >>
> > So you want to be able to rely on an emulated console (running on a
> > totally different subsystem) to dump development-phase early-boot
> > logs? At the cost of legitimizing busy looping in atomic context - one
> > random driver messing up the api for ever. Maybe you could have the
> > ring buffer in some shmem and only pass the number of valid characters
> > in it, to the remote?
> >
>
> I would like to note that this is the one and only console interface
> that exists on these systems, for both development phase and production.
> Other UARTs are not externally accessible on the systems, or they are
> comparatively difficult to access as they aren't intended to be used as
> consoles in the system design.

> The combination of hardware and firmware
> implementing the console is black box from software's point of view
> (similarly to any other UART HW). The interface has also been fixed at
> an early system design phase, as there are many operating systems
> running on these boards, each with their own drivers.
>
That is the saddest part - someone, who writes test cases for the h/w
team and with almost no knowledge of how OSes work, decides how the
firmware is going to work and calls it done. Then the Linux is left to
deal with the mess as we "can't do anything about it".

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Fri, Dec 7, 2018 at 11:50 AM Mikko Perttunen <cyndis@kapsi.fi> wrote:
>
> On 07/12/2018 11.26, Jassi Brar wrote:
> >> I thought that I could also mitigate 2) by busy looping in the TCU driver,
> >> but it turns out that that doesn't work. The reason is that since we are
> >> in atomic context with all interrupts disabled, the mailbox won't ever
> >> consume any new characters, so the read pointer in the circular buffer
> >> won't increment, leaving me with no condition upon which to loop that
> >> would work.
> >>
> > So you want to be able to rely on an emulated console (running on a
> > totally different subsystem) to dump development-phase early-boot
> > logs? At the cost of legitimizing busy looping in atomic context - one
> > random driver messing up the api for ever. Maybe you could have the
> > ring buffer in some shmem and only pass the number of valid characters
> > in it, to the remote?
> >
>
> I would like to note that this is the one and only console interface
> that exists on these systems, for both development phase and production.
> Other UARTs are not externally accessible on the systems, or they are
> comparatively difficult to access as they aren't intended to be used as
> consoles in the system design.

> The combination of hardware and firmware
> implementing the console is black box from software's point of view
> (similarly to any other UART HW). The interface has also been fixed at
> an early system design phase, as there are many operating systems
> running on these boards, each with their own drivers.
>
That is the saddest part - someone, who writes test cases for the h/w
team and with almost no knowledge of how OSes work, decides how the
firmware is going to work and calls it done. Then the Linux is left to
deal with the mess as we "can't do anything about it".

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Fri, Dec 7, 2018 at 5:02 PM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Thu, Dec 06, 2018 at 11:56:25PM -0600, Jassi Brar wrote:
> > On Thu, Nov 29, 2018 at 9:23 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > >
> > > On Wed, Nov 28, 2018 at 11:23:36PM -0600, Jassi Brar wrote:
> > > > On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
> > > > >
> > > > >
> > > > > On 12/11/2018 15:18, Thierry Reding wrote:
> > > > > > From: Thierry Reding <treding@nvidia.com>
> > > > > >
> > > > > > The mailbox framework supports blocking transfers via completions for
> > > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > > callback that controller drivers can implement to busy loop until the
> > > > > > transmission has been completed. This will automatically be called when
> > > > > > available and interrupts are disabled for clients that request blocking
> > > > > > transfers.
> > > > > >
> > > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > > ---
> > > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > > >  2 files changed, 12 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > > --- a/drivers/mailbox/mailbox.c
> > > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > > >               unsigned long wait;
> > > > > >               int ret;
> > > > > >
> > > > > > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > > +                     if (ret < 0)
> > > > > > +                             tx_tick(chan, ret);
> > > > > > +
> > > > > > +                     return ret;
> > > > > > +             }
> > > > >
> > > > > It seems to me that if mbox_send_message() is called from an atomic
> > > > > context AND tx_block is true, then if 'flush' is not populated this
> > > > > should be an error condition as we do not wish to call
> > > > > wait_for_completion from an atomic context.
> > > > >
> > > > > I understand that there is some debate about adding such flush support,
> > > > > but irrespective of the above change, it seems to me that if the
> > > > > mbox_send_message() can be called from an atomic context (which it
> > > > > appears to), then it should be detecting if someone is trying to do so
> > > > > with 'tx_block' set as this should be an error.
> > > > >
> > > > Layers within kernel space have to trust each other. A badly written
> > > > client can break the consumer in so many ways, we can not catch every
> > > > possibility.
> > > >
> > > > > Furthermore, if the underlying mailbox driver can support sending a
> > > > > message from an atomic context and busy wait until it is done, surely
> > > > > the mailbox framework should provide a means to support this?
> > > > >
> > > > Being able to put the message on bus in atomic context is a feature -
> > > > which we do support. But busy-waiting in a loop is not a feature, and
> > > > we don't want to encourage that.
> > >
> > > I agree that in generally busy-waiting is a bad idea and shouldn't be
> > > encouraged. However, I also think that an exception proves the rule. If
> > > you look at the console drivers in drivers/tty/serial, all of them will
> > > busy loop prior to or after sending a character. This is pretty much
> > > part of the API and as such busy-looping is very much a feature.
> > >
> > > The reason why this is done is because on one hand we have an atomic
> > > context and on the other hand we want to make sure that all characters
> > > actually make it to the console when we print them.
> > >
> > > As an example how this can break, I've taken your suggestion to
> > > implement a producer/consumer mode in the TCU driver where the console
> > > write function will just stash characters into a circular buffer and a
> > > work queue will then use mbox_send_message() to drain the circular
> > > buffer. While this does work on the surface, I was able to concern both
> > > of the issues that I was concerned about: 1) it is easy to overflow the
> > > circular buffer by just dumping enough data at once to the console and
> > > 2) when a crash happens, everything in the kernel stops, including the
> > > consumer workqueue that is supposed to drain the circular buffer and
> > > flush messages to the TCU. The result is that, in my particular case,
> > > the boot log will just stop at a random place in the middle of messages
> > > from much earlier in the boot because the TCU hasn't caught up yet and
> > > there's a lot of characters still in the circular buffer.
> > >
> > > Now, 1) can be mitigated by increasing the circular buffer size. A value
> > > that seems to give reliably good results in 2 << CONFIG_LOG_BUF_SHIFT.
> > >
> > Yes please.
>
> As I explained elsewhere, I actually went and implemented this. But
> given the nature of buffering, this ends up making the TCU completely
> useless as a console because in case of a crash, the system will stop
> working with a large number of characters still stuck in the buffer.
> And that's especially bad in case of a crash because those last
> characters that get stuck in the buffer are the most relevant ones
> because they contain the stack dump.
>
I don't question the utility of TCU. I just wonder if mailbox api
should provide a backdoor for atomic busy-wait in order to support a
sub-optimal hw+fw design.

> > > I thought that I could also mitigate 2) by busy looping in the TCU driver,
> > > but it turns out that that doesn't work. The reason is that since we are
> > > in atomic context with all interrupts disabled, the mailbox won't ever
> > > consume any new characters, so the read pointer in the circular buffer
> > > won't increment, leaving me with no condition upon which to loop that
> > > would work.
> > >
> > So you want to be able to rely on an emulated console (running on a
> > totally different subsystem) to dump development-phase early-boot
> > logs? At the cost of legitimizing busy looping in atomic context - one
> > random driver messing up the api for ever. Maybe you could have the
> > ring buffer in some shmem and only pass the number of valid characters
> > in it, to the remote?
>
> First of all, this is not about development-phase early-boot messages.
> We're talking about the system console here. This is what everyone will
> want to be using when developing on this device. Sure at some point you
> may end up with a system that works and you can have the console on the
> network or an attached display or whatever, but even then you may still
> want to attach to the console if you ever run into issues where the
> system doesn't come up.
>
> Secondly, I don't understand why you think this is an emulated console.
>
It is emulated/virtual because Linux doesn't put characters on a
physical bus. The data is simply handed forward to a remote entity.

> Lastly, I don't understand why you think we're messing up the API here.
> The proposal in this series doesn't even change any of the API, but it
> makes it aware of the state of interrupts internally so that it can do
> the right thing depending on the call stack. The other proposal, in v3,
> is more explicit in that it adds new API to flush the mailbox. The new
> API is completely optional and I even offered to document it as being
> discouraged because it involves busy looping. At the same time it does
> solve a real problem and it doesn't impact any existing mailbox drivers
> nor any of their users (because it is completely opt-in).
>
The 'flush' api is going to have no use other than implement
busy-waits. I am afraid people are simply going to take it easy and
copy the busy-waits from the test/verification codes.
"discouraging" seldom works because the developer comes with the
failproof excuse "the h/w doesn't provide any other way". Frankly I
would like to push back on such designs.

 Anyways, let us add the new 'flush' api.

Thanks.

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Fri, Dec 7, 2018 at 5:02 PM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Thu, Dec 06, 2018 at 11:56:25PM -0600, Jassi Brar wrote:
> > On Thu, Nov 29, 2018 at 9:23 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > >
> > > On Wed, Nov 28, 2018 at 11:23:36PM -0600, Jassi Brar wrote:
> > > > On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
> > > > >
> > > > >
> > > > > On 12/11/2018 15:18, Thierry Reding wrote:
> > > > > > From: Thierry Reding <treding@nvidia.com>
> > > > > >
> > > > > > The mailbox framework supports blocking transfers via completions for
> > > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > > callback that controller drivers can implement to busy loop until the
> > > > > > transmission has been completed. This will automatically be called when
> > > > > > available and interrupts are disabled for clients that request blocking
> > > > > > transfers.
> > > > > >
> > > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > > ---
> > > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > > >  2 files changed, 12 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > > --- a/drivers/mailbox/mailbox.c
> > > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > > >               unsigned long wait;
> > > > > >               int ret;
> > > > > >
> > > > > > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > > +                     if (ret < 0)
> > > > > > +                             tx_tick(chan, ret);
> > > > > > +
> > > > > > +                     return ret;
> > > > > > +             }
> > > > >
> > > > > It seems to me that if mbox_send_message() is called from an atomic
> > > > > context AND tx_block is true, then if 'flush' is not populated this
> > > > > should be an error condition as we do not wish to call
> > > > > wait_for_completion from an atomic context.
> > > > >
> > > > > I understand that there is some debate about adding such flush support,
> > > > > but irrespective of the above change, it seems to me that if the
> > > > > mbox_send_message() can be called from an atomic context (which it
> > > > > appears to), then it should be detecting if someone is trying to do so
> > > > > with 'tx_block' set as this should be an error.
> > > > >
> > > > Layers within kernel space have to trust each other. A badly written
> > > > client can break the consumer in so many ways, we can not catch every
> > > > possibility.
> > > >
> > > > > Furthermore, if the underlying mailbox driver can support sending a
> > > > > message from an atomic context and busy wait until it is done, surely
> > > > > the mailbox framework should provide a means to support this?
> > > > >
> > > > Being able to put the message on bus in atomic context is a feature -
> > > > which we do support. But busy-waiting in a loop is not a feature, and
> > > > we don't want to encourage that.
> > >
> > > I agree that in generally busy-waiting is a bad idea and shouldn't be
> > > encouraged. However, I also think that an exception proves the rule. If
> > > you look at the console drivers in drivers/tty/serial, all of them will
> > > busy loop prior to or after sending a character. This is pretty much
> > > part of the API and as such busy-looping is very much a feature.
> > >
> > > The reason why this is done is because on one hand we have an atomic
> > > context and on the other hand we want to make sure that all characters
> > > actually make it to the console when we print them.
> > >
> > > As an example how this can break, I've taken your suggestion to
> > > implement a producer/consumer mode in the TCU driver where the console
> > > write function will just stash characters into a circular buffer and a
> > > work queue will then use mbox_send_message() to drain the circular
> > > buffer. While this does work on the surface, I was able to concern both
> > > of the issues that I was concerned about: 1) it is easy to overflow the
> > > circular buffer by just dumping enough data at once to the console and
> > > 2) when a crash happens, everything in the kernel stops, including the
> > > consumer workqueue that is supposed to drain the circular buffer and
> > > flush messages to the TCU. The result is that, in my particular case,
> > > the boot log will just stop at a random place in the middle of messages
> > > from much earlier in the boot because the TCU hasn't caught up yet and
> > > there's a lot of characters still in the circular buffer.
> > >
> > > Now, 1) can be mitigated by increasing the circular buffer size. A value
> > > that seems to give reliably good results in 2 << CONFIG_LOG_BUF_SHIFT.
> > >
> > Yes please.
>
> As I explained elsewhere, I actually went and implemented this. But
> given the nature of buffering, this ends up making the TCU completely
> useless as a console because in case of a crash, the system will stop
> working with a large number of characters still stuck in the buffer.
> And that's especially bad in case of a crash because those last
> characters that get stuck in the buffer are the most relevant ones
> because they contain the stack dump.
>
I don't question the utility of TCU. I just wonder if mailbox api
should provide a backdoor for atomic busy-wait in order to support a
sub-optimal hw+fw design.

> > > I thought that I could also mitigate 2) by busy looping in the TCU driver,
> > > but it turns out that that doesn't work. The reason is that since we are
> > > in atomic context with all interrupts disabled, the mailbox won't ever
> > > consume any new characters, so the read pointer in the circular buffer
> > > won't increment, leaving me with no condition upon which to loop that
> > > would work.
> > >
> > So you want to be able to rely on an emulated console (running on a
> > totally different subsystem) to dump development-phase early-boot
> > logs? At the cost of legitimizing busy looping in atomic context - one
> > random driver messing up the api for ever. Maybe you could have the
> > ring buffer in some shmem and only pass the number of valid characters
> > in it, to the remote?
>
> First of all, this is not about development-phase early-boot messages.
> We're talking about the system console here. This is what everyone will
> want to be using when developing on this device. Sure at some point you
> may end up with a system that works and you can have the console on the
> network or an attached display or whatever, but even then you may still
> want to attach to the console if you ever run into issues where the
> system doesn't come up.
>
> Secondly, I don't understand why you think this is an emulated console.
>
It is emulated/virtual because Linux doesn't put characters on a
physical bus. The data is simply handed forward to a remote entity.

> Lastly, I don't understand why you think we're messing up the API here.
> The proposal in this series doesn't even change any of the API, but it
> makes it aware of the state of interrupts internally so that it can do
> the right thing depending on the call stack. The other proposal, in v3,
> is more explicit in that it adds new API to flush the mailbox. The new
> API is completely optional and I even offered to document it as being
> discouraged because it involves busy looping. At the same time it does
> solve a real problem and it doesn't impact any existing mailbox drivers
> nor any of their users (because it is completely opt-in).
>
The 'flush' api is going to have no use other than implement
busy-waits. I am afraid people are simply going to take it easy and
copy the busy-waits from the test/verification codes.
"discouraging" seldom works because the developer comes with the
failproof excuse "the h/w doesn't provide any other way". Frankly I
would like to push back on such designs.

 Anyways, let us add the new 'flush' api.

Thanks.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Greg KH]

On Sat, Dec 08, 2018 at 11:21:41AM +0530, Jassi Brar wrote:
> On Fri, Dec 7, 2018 at 11:50 AM Mikko Perttunen <cyndis@kapsi.fi> wrote:
> >
> > On 07/12/2018 11.26, Jassi Brar wrote:
> > >> I thought that I could also mitigate 2) by busy looping in the TCU driver,
> > >> but it turns out that that doesn't work. The reason is that since we are
> > >> in atomic context with all interrupts disabled, the mailbox won't ever
> > >> consume any new characters, so the read pointer in the circular buffer
> > >> won't increment, leaving me with no condition upon which to loop that
> > >> would work.
> > >>
> > > So you want to be able to rely on an emulated console (running on a
> > > totally different subsystem) to dump development-phase early-boot
> > > logs? At the cost of legitimizing busy looping in atomic context - one
> > > random driver messing up the api for ever. Maybe you could have the
> > > ring buffer in some shmem and only pass the number of valid characters
> > > in it, to the remote?
> > >
> >
> > I would like to note that this is the one and only console interface
> > that exists on these systems, for both development phase and production.
> > Other UARTs are not externally accessible on the systems, or they are
> > comparatively difficult to access as they aren't intended to be used as
> > consoles in the system design.
> 
> > The combination of hardware and firmware
> > implementing the console is black box from software's point of view
> > (similarly to any other UART HW). The interface has also been fixed at
> > an early system design phase, as there are many operating systems
> > running on these boards, each with their own drivers.
> >
> That is the saddest part - someone, who writes test cases for the h/w
> team and with almost no knowledge of how OSes work, decides how the
> firmware is going to work and calls it done. Then the Linux is left to
> deal with the mess as we "can't do anything about it".

That is totally normal, and is how hardware has been almost _always_
been designed and implemented.  Nothing new here at all, it's just the
life us kernel developers get used to very quickly as it is our job to
make that hardware work and appear to userspace as something sane and
universal.

Sorry,

greg k-h

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Greg KH]

On Sat, Dec 08, 2018 at 11:21:41AM +0530, Jassi Brar wrote:
> On Fri, Dec 7, 2018 at 11:50 AM Mikko Perttunen <cyndis@kapsi.fi> wrote:
> >
> > On 07/12/2018 11.26, Jassi Brar wrote:
> > >> I thought that I could also mitigate 2) by busy looping in the TCU driver,
> > >> but it turns out that that doesn't work. The reason is that since we are
> > >> in atomic context with all interrupts disabled, the mailbox won't ever
> > >> consume any new characters, so the read pointer in the circular buffer
> > >> won't increment, leaving me with no condition upon which to loop that
> > >> would work.
> > >>
> > > So you want to be able to rely on an emulated console (running on a
> > > totally different subsystem) to dump development-phase early-boot
> > > logs? At the cost of legitimizing busy looping in atomic context - one
> > > random driver messing up the api for ever. Maybe you could have the
> > > ring buffer in some shmem and only pass the number of valid characters
> > > in it, to the remote?
> > >
> >
> > I would like to note that this is the one and only console interface
> > that exists on these systems, for both development phase and production.
> > Other UARTs are not externally accessible on the systems, or they are
> > comparatively difficult to access as they aren't intended to be used as
> > consoles in the system design.
> 
> > The combination of hardware and firmware
> > implementing the console is black box from software's point of view
> > (similarly to any other UART HW). The interface has also been fixed at
> > an early system design phase, as there are many operating systems
> > running on these boards, each with their own drivers.
> >
> That is the saddest part - someone, who writes test cases for the h/w
> team and with almost no knowledge of how OSes work, decides how the
> firmware is going to work and calls it done. Then the Linux is left to
> deal with the mess as we "can't do anything about it".

That is totally normal, and is how hardware has been almost _always_
been designed and implemented.  Nothing new here at all, it's just the
life us kernel developers get used to very quickly as it is our job to
make that hardware work and appear to userspace as something sane and
universal.

Sorry,

greg k-h

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Sat, Dec 8, 2018 at 2:50 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Sat, Dec 08, 2018 at 11:21:41AM +0530, Jassi Brar wrote:
> > On Fri, Dec 7, 2018 at 11:50 AM Mikko Perttunen <cyndis@kapsi.fi> wrote:
> > >
> > > On 07/12/2018 11.26, Jassi Brar wrote:
> > > >> I thought that I could also mitigate 2) by busy looping in the TCU driver,
> > > >> but it turns out that that doesn't work. The reason is that since we are
> > > >> in atomic context with all interrupts disabled, the mailbox won't ever
> > > >> consume any new characters, so the read pointer in the circular buffer
> > > >> won't increment, leaving me with no condition upon which to loop that
> > > >> would work.
> > > >>
> > > > So you want to be able to rely on an emulated console (running on a
> > > > totally different subsystem) to dump development-phase early-boot
> > > > logs? At the cost of legitimizing busy looping in atomic context - one
> > > > random driver messing up the api for ever. Maybe you could have the
> > > > ring buffer in some shmem and only pass the number of valid characters
> > > > in it, to the remote?
> > > >
> > >
> > > I would like to note that this is the one and only console interface
> > > that exists on these systems, for both development phase and production.
> > > Other UARTs are not externally accessible on the systems, or they are
> > > comparatively difficult to access as they aren't intended to be used as
> > > consoles in the system design.
> >
> > > The combination of hardware and firmware
> > > implementing the console is black box from software's point of view
> > > (similarly to any other UART HW). The interface has also been fixed at
> > > an early system design phase, as there are many operating systems
> > > running on these boards, each with their own drivers.
> > >
> > That is the saddest part - someone, who writes test cases for the h/w
> > team and with almost no knowledge of how OSes work, decides how the
> > firmware is going to work and calls it done. Then the Linux is left to
> > deal with the mess as we "can't do anything about it".
>
> That is totally normal, and is how hardware has been almost _always_
> been designed and implemented.  Nothing new here at all, it's just the
> life us kernel developers get used to very quickly as it is our job to
> make that hardware work and appear to userspace as something sane and
> universal.
>
Hardware yes, we can't really change much after the fact.
In this case the issue arises from the firmware - TCU's mailbox
protocol implementation. Which usually is just another image loaded
onto the remote master during cold-boot, and should actually be not
that hard to fix/change. Even if other OSes (really?) have adapted, a
pushback right now will help fix atleast the next version, otherwise
the api designer will never know what s/he is doing wrong.

Anyways, thanks for chiming in. I will pull the patchset.

Thanks.

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Sat, Dec 8, 2018 at 2:50 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Sat, Dec 08, 2018 at 11:21:41AM +0530, Jassi Brar wrote:
> > On Fri, Dec 7, 2018 at 11:50 AM Mikko Perttunen <cyndis@kapsi.fi> wrote:
> > >
> > > On 07/12/2018 11.26, Jassi Brar wrote:
> > > >> I thought that I could also mitigate 2) by busy looping in the TCU driver,
> > > >> but it turns out that that doesn't work. The reason is that since we are
> > > >> in atomic context with all interrupts disabled, the mailbox won't ever
> > > >> consume any new characters, so the read pointer in the circular buffer
> > > >> won't increment, leaving me with no condition upon which to loop that
> > > >> would work.
> > > >>
> > > > So you want to be able to rely on an emulated console (running on a
> > > > totally different subsystem) to dump development-phase early-boot
> > > > logs? At the cost of legitimizing busy looping in atomic context - one
> > > > random driver messing up the api for ever. Maybe you could have the
> > > > ring buffer in some shmem and only pass the number of valid characters
> > > > in it, to the remote?
> > > >
> > >
> > > I would like to note that this is the one and only console interface
> > > that exists on these systems, for both development phase and production.
> > > Other UARTs are not externally accessible on the systems, or they are
> > > comparatively difficult to access as they aren't intended to be used as
> > > consoles in the system design.
> >
> > > The combination of hardware and firmware
> > > implementing the console is black box from software's point of view
> > > (similarly to any other UART HW). The interface has also been fixed at
> > > an early system design phase, as there are many operating systems
> > > running on these boards, each with their own drivers.
> > >
> > That is the saddest part - someone, who writes test cases for the h/w
> > team and with almost no knowledge of how OSes work, decides how the
> > firmware is going to work and calls it done. Then the Linux is left to
> > deal with the mess as we "can't do anything about it".
>
> That is totally normal, and is how hardware has been almost _always_
> been designed and implemented.  Nothing new here at all, it's just the
> life us kernel developers get used to very quickly as it is our job to
> make that hardware work and appear to userspace as something sane and
> universal.
>
Hardware yes, we can't really change much after the fact.
In this case the issue arises from the firmware - TCU's mailbox
protocol implementation. Which usually is just another image loaded
onto the remote master during cold-boot, and should actually be not
that hard to fix/change. Even if other OSes (really?) have adapted, a
pushback right now will help fix atleast the next version, otherwise
the api designer will never know what s/he is doing wrong.

Anyways, thanks for chiming in. I will pull the patchset.

Thanks.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 10309 bytes --]

On Sat, Dec 08, 2018 at 11:39:05AM +0530, Jassi Brar wrote:
> On Fri, Dec 7, 2018 at 5:02 PM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > On Thu, Dec 06, 2018 at 11:56:25PM -0600, Jassi Brar wrote:
> > > On Thu, Nov 29, 2018 at 9:23 AM Thierry Reding <thierry.reding@gmail.com> wrote:
> > > >
> > > > On Wed, Nov 28, 2018 at 11:23:36PM -0600, Jassi Brar wrote:
> > > > > On Wed, Nov 28, 2018 at 3:43 AM Jon Hunter <jonathanh@nvidia.com> wrote:
> > > > > >
> > > > > >
> > > > > > On 12/11/2018 15:18, Thierry Reding wrote:
> > > > > > > From: Thierry Reding <treding@nvidia.com>
> > > > > > >
> > > > > > > The mailbox framework supports blocking transfers via completions for
> > > > > > > clients that can sleep. In order to support blocking transfers in cases
> > > > > > > where the transmission is not permitted to sleep, add a new ->flush()
> > > > > > > callback that controller drivers can implement to busy loop until the
> > > > > > > transmission has been completed. This will automatically be called when
> > > > > > > available and interrupts are disabled for clients that request blocking
> > > > > > > transfers.
> > > > > > >
> > > > > > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > > > > > ---
> > > > > > >  drivers/mailbox/mailbox.c          | 8 ++++++++
> > > > > > >  include/linux/mailbox_controller.h | 4 ++++
> > > > > > >  2 files changed, 12 insertions(+)
> > > > > > >
> > > > > > > diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> > > > > > > index 674b35f402f5..0eaf21259874 100644
> > > > > > > --- a/drivers/mailbox/mailbox.c
> > > > > > > +++ b/drivers/mailbox/mailbox.c
> > > > > > > @@ -267,6 +267,14 @@ int mbox_send_message(struct mbox_chan *chan, void *mssg)
> > > > > > >               unsigned long wait;
> > > > > > >               int ret;
> > > > > > >
> > > > > > > +             if (irqs_disabled() && chan->mbox->ops->flush) {
> > > > > > > +                     ret = chan->mbox->ops->flush(chan, chan->cl->tx_tout);
> > > > > > > +                     if (ret < 0)
> > > > > > > +                             tx_tick(chan, ret);
> > > > > > > +
> > > > > > > +                     return ret;
> > > > > > > +             }
> > > > > >
> > > > > > It seems to me that if mbox_send_message() is called from an atomic
> > > > > > context AND tx_block is true, then if 'flush' is not populated this
> > > > > > should be an error condition as we do not wish to call
> > > > > > wait_for_completion from an atomic context.
> > > > > >
> > > > > > I understand that there is some debate about adding such flush support,
> > > > > > but irrespective of the above change, it seems to me that if the
> > > > > > mbox_send_message() can be called from an atomic context (which it
> > > > > > appears to), then it should be detecting if someone is trying to do so
> > > > > > with 'tx_block' set as this should be an error.
> > > > > >
> > > > > Layers within kernel space have to trust each other. A badly written
> > > > > client can break the consumer in so many ways, we can not catch every
> > > > > possibility.
> > > > >
> > > > > > Furthermore, if the underlying mailbox driver can support sending a
> > > > > > message from an atomic context and busy wait until it is done, surely
> > > > > > the mailbox framework should provide a means to support this?
> > > > > >
> > > > > Being able to put the message on bus in atomic context is a feature -
> > > > > which we do support. But busy-waiting in a loop is not a feature, and
> > > > > we don't want to encourage that.
> > > >
> > > > I agree that in generally busy-waiting is a bad idea and shouldn't be
> > > > encouraged. However, I also think that an exception proves the rule. If
> > > > you look at the console drivers in drivers/tty/serial, all of them will
> > > > busy loop prior to or after sending a character. This is pretty much
> > > > part of the API and as such busy-looping is very much a feature.
> > > >
> > > > The reason why this is done is because on one hand we have an atomic
> > > > context and on the other hand we want to make sure that all characters
> > > > actually make it to the console when we print them.
> > > >
> > > > As an example how this can break, I've taken your suggestion to
> > > > implement a producer/consumer mode in the TCU driver where the console
> > > > write function will just stash characters into a circular buffer and a
> > > > work queue will then use mbox_send_message() to drain the circular
> > > > buffer. While this does work on the surface, I was able to concern both
> > > > of the issues that I was concerned about: 1) it is easy to overflow the
> > > > circular buffer by just dumping enough data at once to the console and
> > > > 2) when a crash happens, everything in the kernel stops, including the
> > > > consumer workqueue that is supposed to drain the circular buffer and
> > > > flush messages to the TCU. The result is that, in my particular case,
> > > > the boot log will just stop at a random place in the middle of messages
> > > > from much earlier in the boot because the TCU hasn't caught up yet and
> > > > there's a lot of characters still in the circular buffer.
> > > >
> > > > Now, 1) can be mitigated by increasing the circular buffer size. A value
> > > > that seems to give reliably good results in 2 << CONFIG_LOG_BUF_SHIFT.
> > > >
> > > Yes please.
> >
> > As I explained elsewhere, I actually went and implemented this. But
> > given the nature of buffering, this ends up making the TCU completely
> > useless as a console because in case of a crash, the system will stop
> > working with a large number of characters still stuck in the buffer.
> > And that's especially bad in case of a crash because those last
> > characters that get stuck in the buffer are the most relevant ones
> > because they contain the stack dump.
> >
> I don't question the utility of TCU. I just wonder if mailbox api
> should provide a backdoor for atomic busy-wait in order to support a
> sub-optimal hw+fw design.
> 
> > > > I thought that I could also mitigate 2) by busy looping in the TCU driver,
> > > > but it turns out that that doesn't work. The reason is that since we are
> > > > in atomic context with all interrupts disabled, the mailbox won't ever
> > > > consume any new characters, so the read pointer in the circular buffer
> > > > won't increment, leaving me with no condition upon which to loop that
> > > > would work.
> > > >
> > > So you want to be able to rely on an emulated console (running on a
> > > totally different subsystem) to dump development-phase early-boot
> > > logs? At the cost of legitimizing busy looping in atomic context - one
> > > random driver messing up the api for ever. Maybe you could have the
> > > ring buffer in some shmem and only pass the number of valid characters
> > > in it, to the remote?
> >
> > First of all, this is not about development-phase early-boot messages.
> > We're talking about the system console here. This is what everyone will
> > want to be using when developing on this device. Sure at some point you
> > may end up with a system that works and you can have the console on the
> > network or an attached display or whatever, but even then you may still
> > want to attach to the console if you ever run into issues where the
> > system doesn't come up.
> >
> > Secondly, I don't understand why you think this is an emulated console.
> >
> It is emulated/virtual because Linux doesn't put characters on a
> physical bus. The data is simply handed forward to a remote entity.

Okay, I understand. However, from a kernel point of view there's no
difference between the TCU and any physical UART. Once the data is
handed to the TCU TX mailbox, it's out of control of the TCU driver.
Whether there's a real UART behind that or some remote processor that
reads the data isn't really relevant.

> > Lastly, I don't understand why you think we're messing up the API here.
> > The proposal in this series doesn't even change any of the API, but it
> > makes it aware of the state of interrupts internally so that it can do
> > the right thing depending on the call stack. The other proposal, in v3,
> > is more explicit in that it adds new API to flush the mailbox. The new
> > API is completely optional and I even offered to document it as being
> > discouraged because it involves busy looping. At the same time it does
> > solve a real problem and it doesn't impact any existing mailbox drivers
> > nor any of their users (because it is completely opt-in).
> >
> The 'flush' api is going to have no use other than implement
> busy-waits. I am afraid people are simply going to take it easy and
> copy the busy-waits from the test/verification codes.
> "discouraging" seldom works because the developer comes with the
> failproof excuse "the h/w doesn't provide any other way". Frankly I
> would like to push back on such designs.

I certainly approve of that stance.

However, I'd like to reiterate that the TCU design does in fact support
"another way". If you look at the mailbox driver implementation (in the
drivers/mailbox/tegra-hsp.c driver), you'll see that it does support an
interrupt driven mode. After you had reviewed Mikko's earliest proposal
that was indeed implementing busy looping exclusively we both set out
to properly implement interrupt support. This took quite a while to do
because of some gaps in the documentation, but we eventually got it to
work properly. And then it was discovered that it was all a waste of
effort because the console driver couldn't make use of it anyway. Well,
I should say "waste of effort" because I'm still happy that the proper
implementation exists and we can use it under the right circumstances.

So, at least in this particular case, it is not the hardware or firmware
design that's flawed or was taking any shortcuts. It's really just the
particular use-case of the console that doesn't allow us to make use of
the right implementation, which is why we have to resort to the inferior
method of busy-looping.

>  Anyways, let us add the new 'flush' api.

Great, thanks!

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 05/10] mailbox: tegra-hsp: Add suspend/resume support

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 2233 bytes --]

On Tue, Nov 13, 2018 at 11:17:58AM +0000, Jon Hunter wrote:
> 
> On 12/11/2018 15:18, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > Upon resuming from a system sleep state, the interrupts for all active
> > shared mailboxes need to be reenabled, otherwise they will not work.
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> >  drivers/mailbox/tegra-hsp.c | 19 +++++++++++++++++++
> >  1 file changed, 19 insertions(+)
> > 
> > diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
> > index 0100a974149b..1259abf3542f 100644
> > --- a/drivers/mailbox/tegra-hsp.c
> > +++ b/drivers/mailbox/tegra-hsp.c
> > @@ -18,6 +18,7 @@
> >  #include <linux/of.h>
> >  #include <linux/of_device.h>
> >  #include <linux/platform_device.h>
> > +#include <linux/pm.h>
> >  #include <linux/slab.h>
> >  
> >  #include <dt-bindings/mailbox/tegra186-hsp.h>
> > @@ -817,6 +818,23 @@ static int tegra_hsp_remove(struct platform_device *pdev)
> >  	return 0;
> >  }
> >  
> > +static int tegra_hsp_resume(struct device *dev)
> > +{
> > +	struct tegra_hsp *hsp = dev_get_drvdata(dev);
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < hsp->num_sm; i++) {
> > +		struct tegra_hsp_mailbox *mb = &hsp->mailboxes[i];
> > +
> > +		if (mb->channel.chan->cl)
> > +			tegra_hsp_mailbox_startup(mb->channel.chan);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static SIMPLE_DEV_PM_OPS(tegra_hsp_pm_ops, NULL, tegra_hsp_resume);
> 
> Is it worth disabling interrupts on suspend to avoid any in-flight
> interrupt triggering a bad access on entering suspend? I assume that the
> context of the mailbox registers get cleared/lost at some point and so I
> was not sure if there is a time where they have a bad state or are
> inaccessible?

Theoretically I think we'd have to do that. However, since we end up
having to busy-loop for any current use-cases (i.e. console) anyway,
we'll never end up in a situation where an interrupt could occur at
this point. The console will long have been suspended when we reach
this point. I'll take a look if a use-case can be constructed where
such an in-flight interrupt could be produced.

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Mon, Dec 10, 2018 at 3:22 PM Thierry Reding <thierry.reding@gmail.com> wrote:
> On Sat, Dec 08, 2018 at 11:39:05AM +0530, Jassi Brar wrote:
> > On Fri, Dec 7, 2018 at 5:02 PM Thierry Reding <thierry.reding@gmail.com> wrote:

> > >
> > >
> > > Secondly, I don't understand why you think this is an emulated console.
> > >
> > It is emulated/virtual because Linux doesn't put characters on a
> > physical bus. The data is simply handed forward to a remote entity.
>
> Okay, I understand. However, from a kernel point of view there's no
> difference between the TCU and any physical UART. Once the data is
> handed to the TCU TX mailbox, it's out of control of the TCU driver.
> Whether there's a real UART behind that or some remote processor that
> reads the data isn't really relevant.
>
That way there can be no virtual/emulated device ?

> >
> > The 'flush' api is going to have no use other than implement
> > busy-waits. I am afraid people are simply going to take it easy and
> > copy the busy-waits from the test/verification codes.
> > "discouraging" seldom works because the developer comes with the
> > failproof excuse "the h/w doesn't provide any other way". Frankly I
> > would like to push back on such designs.
>
> I certainly approve of that stance.
>
> However, I'd like to reiterate that the TCU design does in fact support
> "another way". If you look at the mailbox driver implementation (in the
> drivers/mailbox/tegra-hsp.c driver), you'll see that it does support an
> interrupt driven mode. After you had reviewed Mikko's earliest proposal
> that was indeed implementing busy looping exclusively we both set out
> to properly implement interrupt support. This took quite a while to do
> because of some gaps in the documentation, but we eventually got it to
> work properly. And then it was discovered that it was all a waste of
> effort because the console driver couldn't make use of it anyway. Well,
> I should say "waste of effort" because I'm still happy that the proper
> implementation exists and we can use it under the right circumstances.
>
> So, at least in this particular case, it is not the hardware or firmware
> design that's flawed or was taking any shortcuts. It's really just the
> particular use-case of the console that doesn't allow us to make use of
> the right implementation, which is why we have to resort to the inferior
> method of busy-looping.
>
I am not complaining about the hardware, but the firmware.
It is essential we dump logs during early boot but the platform(Linux)
doesn't have access to serial port. All the firmware allows is 24bits
per transfer!! We could do better.
A smarter option could have been Linux simply placing characters in
the shmem ring-buffer, while the remote consuming (and then poisoning)
the ring buffer upon 'hint' from Linux.

-Jassi

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Mon, Dec 10, 2018 at 3:22 PM Thierry Reding <thierry.reding@gmail.com> wrote:
> On Sat, Dec 08, 2018 at 11:39:05AM +0530, Jassi Brar wrote:
> > On Fri, Dec 7, 2018 at 5:02 PM Thierry Reding <thierry.reding@gmail.com> wrote:

> > >
> > >
> > > Secondly, I don't understand why you think this is an emulated console.
> > >
> > It is emulated/virtual because Linux doesn't put characters on a
> > physical bus. The data is simply handed forward to a remote entity.
>
> Okay, I understand. However, from a kernel point of view there's no
> difference between the TCU and any physical UART. Once the data is
> handed to the TCU TX mailbox, it's out of control of the TCU driver.
> Whether there's a real UART behind that or some remote processor that
> reads the data isn't really relevant.
>
That way there can be no virtual/emulated device ?

> >
> > The 'flush' api is going to have no use other than implement
> > busy-waits. I am afraid people are simply going to take it easy and
> > copy the busy-waits from the test/verification codes.
> > "discouraging" seldom works because the developer comes with the
> > failproof excuse "the h/w doesn't provide any other way". Frankly I
> > would like to push back on such designs.
>
> I certainly approve of that stance.
>
> However, I'd like to reiterate that the TCU design does in fact support
> "another way". If you look at the mailbox driver implementation (in the
> drivers/mailbox/tegra-hsp.c driver), you'll see that it does support an
> interrupt driven mode. After you had reviewed Mikko's earliest proposal
> that was indeed implementing busy looping exclusively we both set out
> to properly implement interrupt support. This took quite a while to do
> because of some gaps in the documentation, but we eventually got it to
> work properly. And then it was discovered that it was all a waste of
> effort because the console driver couldn't make use of it anyway. Well,
> I should say "waste of effort" because I'm still happy that the proper
> implementation exists and we can use it under the right circumstances.
>
> So, at least in this particular case, it is not the hardware or firmware
> design that's flawed or was taking any shortcuts. It's really just the
> particular use-case of the console that doesn't allow us to make use of
> the right implementation, which is why we have to resort to the inferior
> method of busy-looping.
>
I am not complaining about the hardware, but the firmware.
It is essential we dump logs during early boot but the platform(Linux)
doesn't have access to serial port. All the firmware allows is 24bits
per transfer!! We could do better.
A smarter option could have been Linux simply placing characters in
the shmem ring-buffer, while the remote consuming (and then poisoning)
the ring buffer upon 'hint' from Linux.

-Jassi

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 3945 bytes --]

On Tue, Dec 11, 2018 at 02:00:48AM +0530, Jassi Brar wrote:
> On Mon, Dec 10, 2018 at 3:22 PM Thierry Reding <thierry.reding@gmail.com> wrote:
> > On Sat, Dec 08, 2018 at 11:39:05AM +0530, Jassi Brar wrote:
> > > On Fri, Dec 7, 2018 at 5:02 PM Thierry Reding <thierry.reding@gmail.com> wrote:
> 
> > > >
> > > >
> > > > Secondly, I don't understand why you think this is an emulated console.
> > > >
> > > It is emulated/virtual because Linux doesn't put characters on a
> > > physical bus. The data is simply handed forward to a remote entity.
> >
> > Okay, I understand. However, from a kernel point of view there's no
> > difference between the TCU and any physical UART. Once the data is
> > handed to the TCU TX mailbox, it's out of control of the TCU driver.
> > Whether there's a real UART behind that or some remote processor that
> > reads the data isn't really relevant.
> >
> That way there can be no virtual/emulated device ?

Virtual/emulated I interpret as purely implemented in software. That is
within the Linux kernel. Once you leave the domain of the kernel it's
no longer under the complete control of the kernel. So where exactly it
goes doesn't matter, it's pretty much the same as a hardware device from
the kernel's perspective.

But I guess you could define this differently.

> > >
> > > The 'flush' api is going to have no use other than implement
> > > busy-waits. I am afraid people are simply going to take it easy and
> > > copy the busy-waits from the test/verification codes.
> > > "discouraging" seldom works because the developer comes with the
> > > failproof excuse "the h/w doesn't provide any other way". Frankly I
> > > would like to push back on such designs.
> >
> > I certainly approve of that stance.
> >
> > However, I'd like to reiterate that the TCU design does in fact support
> > "another way". If you look at the mailbox driver implementation (in the
> > drivers/mailbox/tegra-hsp.c driver), you'll see that it does support an
> > interrupt driven mode. After you had reviewed Mikko's earliest proposal
> > that was indeed implementing busy looping exclusively we both set out
> > to properly implement interrupt support. This took quite a while to do
> > because of some gaps in the documentation, but we eventually got it to
> > work properly. And then it was discovered that it was all a waste of
> > effort because the console driver couldn't make use of it anyway. Well,
> > I should say "waste of effort" because I'm still happy that the proper
> > implementation exists and we can use it under the right circumstances.
> >
> > So, at least in this particular case, it is not the hardware or firmware
> > design that's flawed or was taking any shortcuts. It's really just the
> > particular use-case of the console that doesn't allow us to make use of
> > the right implementation, which is why we have to resort to the inferior
> > method of busy-looping.
> >
> I am not complaining about the hardware, but the firmware.
> It is essential we dump logs during early boot but the platform(Linux)
> doesn't have access to serial port. All the firmware allows is 24bits
> per transfer!! We could do better.

Hardware UARTs don't usually have much more FIFO space than that either.

> A smarter option could have been Linux simply placing characters in
> the shmem ring-buffer, while the remote consuming (and then poisoning)
> the ring buffer upon 'hint' from Linux.

I don't think that would've been much smarter, especially not in this
case. As we discussed earlier, no matter how large you make the ring-
buffer you can always run into situations where you overflow it. The
ring-buffer implementation is also a lot more complicated and error-
prone. Plus there is the fact that in this particular case we actually
don't want buffering because the buffer may hide important information
in case of a crash.

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Tue, Dec 11, 2018 at 2:15 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Tue, Dec 11, 2018 at 02:00:48AM +0530, Jassi Brar wrote:
> > On Mon, Dec 10, 2018 at 3:22 PM Thierry Reding <thierry.reding@gmail.com> wrote:

> > > >
> > > > The 'flush' api is going to have no use other than implement
> > > > busy-waits. I am afraid people are simply going to take it easy and
> > > > copy the busy-waits from the test/verification codes.
> > > > "discouraging" seldom works because the developer comes with the
> > > > failproof excuse "the h/w doesn't provide any other way". Frankly I
> > > > would like to push back on such designs.
> > >
> > > I certainly approve of that stance.
> > >
> > > However, I'd like to reiterate that the TCU design does in fact support
> > > "another way". If you look at the mailbox driver implementation (in the
> > > drivers/mailbox/tegra-hsp.c driver), you'll see that it does support an
> > > interrupt driven mode. After you had reviewed Mikko's earliest proposal
> > > that was indeed implementing busy looping exclusively we both set out
> > > to properly implement interrupt support. This took quite a while to do
> > > because of some gaps in the documentation, but we eventually got it to
> > > work properly. And then it was discovered that it was all a waste of
> > > effort because the console driver couldn't make use of it anyway. Well,
> > > I should say "waste of effort" because I'm still happy that the proper
> > > implementation exists and we can use it under the right circumstances.
> > >
> > > So, at least in this particular case, it is not the hardware or firmware
> > > design that's flawed or was taking any shortcuts. It's really just the
> > > particular use-case of the console that doesn't allow us to make use of
> > > the right implementation, which is why we have to resort to the inferior
> > > method of busy-looping.
> > >
> > I am not complaining about the hardware, but the firmware.
> > It is essential we dump logs during early boot but the platform(Linux)
> > doesn't have access to serial port. All the firmware allows is 24bits
> > per transfer!! We could do better.
>
> Hardware UARTs don't usually have much more FIFO space than that either.
>
> > A smarter option could have been Linux simply placing characters in
> > the shmem ring-buffer, while the remote consuming (and then poisoning)
> > the ring buffer upon 'hint' from Linux.
>
> I don't think that would've been much smarter, especially not in this
> case. As we discussed earlier, no matter how large you make the ring-
> buffer you can always run into situations where you overflow it. The
> ring-buffer implementation is also a lot more complicated and error-
> prone.
>
Please think about it again.
The ring buffer becomes the effective h/w fifo. And you don't have to
wait at all for the mailbox register to clear .... you could simply
overwrite it when Linux puts some data in the ring-buffer (the data
written will just be a 'hint' command for remote to go look into the
ring-buffer for new data).

> Plus there is the fact that in this particular case we actually
> don't want buffering because the buffer may hide important information
> in case of a crash.
>
Even if the Linux crashes, whatever data is placed in the ring-buffer
will eventually be printed by the still-alive remote.

Anyways once we have the flush api, I don't really care how broken your f/w is.

Re: [PATCH v2 01/10] mailbox: Support blocking transfers in atomic context

[Jassi Brar]

On Tue, Dec 11, 2018 at 2:15 AM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Tue, Dec 11, 2018 at 02:00:48AM +0530, Jassi Brar wrote:
> > On Mon, Dec 10, 2018 at 3:22 PM Thierry Reding <thierry.reding@gmail.com> wrote:

> > > >
> > > > The 'flush' api is going to have no use other than implement
> > > > busy-waits. I am afraid people are simply going to take it easy and
> > > > copy the busy-waits from the test/verification codes.
> > > > "discouraging" seldom works because the developer comes with the
> > > > failproof excuse "the h/w doesn't provide any other way". Frankly I
> > > > would like to push back on such designs.
> > >
> > > I certainly approve of that stance.
> > >
> > > However, I'd like to reiterate that the TCU design does in fact support
> > > "another way". If you look at the mailbox driver implementation (in the
> > > drivers/mailbox/tegra-hsp.c driver), you'll see that it does support an
> > > interrupt driven mode. After you had reviewed Mikko's earliest proposal
> > > that was indeed implementing busy looping exclusively we both set out
> > > to properly implement interrupt support. This took quite a while to do
> > > because of some gaps in the documentation, but we eventually got it to
> > > work properly. And then it was discovered that it was all a waste of
> > > effort because the console driver couldn't make use of it anyway. Well,
> > > I should say "waste of effort" because I'm still happy that the proper
> > > implementation exists and we can use it under the right circumstances.
> > >
> > > So, at least in this particular case, it is not the hardware or firmware
> > > design that's flawed or was taking any shortcuts. It's really just the
> > > particular use-case of the console that doesn't allow us to make use of
> > > the right implementation, which is why we have to resort to the inferior
> > > method of busy-looping.
> > >
> > I am not complaining about the hardware, but the firmware.
> > It is essential we dump logs during early boot but the platform(Linux)
> > doesn't have access to serial port. All the firmware allows is 24bits
> > per transfer!! We could do better.
>
> Hardware UARTs don't usually have much more FIFO space than that either.
>
> > A smarter option could have been Linux simply placing characters in
> > the shmem ring-buffer, while the remote consuming (and then poisoning)
> > the ring buffer upon 'hint' from Linux.
>
> I don't think that would've been much smarter, especially not in this
> case. As we discussed earlier, no matter how large you make the ring-
> buffer you can always run into situations where you overflow it. The
> ring-buffer implementation is also a lot more complicated and error-
> prone.
>
Please think about it again.
The ring buffer becomes the effective h/w fifo. And you don't have to
wait at all for the mailbox register to clear .... you could simply
overwrite it when Linux puts some data in the ring-buffer (the data
written will just be a 'hint' command for remote to go look into the
ring-buffer for new data).

> Plus there is the fact that in this particular case we actually
> don't want buffering because the buffer may hide important information
> in case of a crash.
>
Even if the Linux crashes, whatever data is placed in the ring-buffer
will eventually be printed by the still-alive remote.

Anyways once we have the flush api, I don't really care how broken your f/w is.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel