All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v4 00/16] Introduce SCMI VirtIO transport
@ 2021-06-11 16:59 ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Hi all,

I'm posting this V4 series starting from the work done up to V3 by
OpenSynergy.

I kept original autorship and mailing list distribution unchanged, even
though I'm not really sure if I'm allowed to post to all the original MLs.

The main aim of this rework is to simplify where possible the SCMI VirtIO
support added in V3 by adding upfront and then using some new mechanisms in
the SCMI Core Transport layer.

Indeed patches 01,02,03,04,05 add such additional mechanisms to the SCMI
core; these were previously posted separately in a distinct patchset, I
will retire that series and continue posting those here to ease testing.

In terms of rework I dropped original V3 patches 02,08,12 as no more needed
while keeping for now the rework distinct marking such patches as
[RFC][REWORK] in order to easily spot changes.
(they will be squashed into related original V3 patches next)

DT bindings patch has been ported on top of freshly YAML converted arm,scmi
bindings.

Patch 16 adds support for polling mode to virtio-scmi.

Known/open issues in V4:

 - possible violation by virtio-scmi transport of the order constraints on
   correlated response/delayed_response messages: this will be addressed
   next with changes in the SCMI core.

 - probing sequence simplification (if possible)

As such this series is anyway still work in progress, needing also some
general cleanup.

Moreover SCMI core reworks about atomicity and polling, posted as a
distinct series, is not included here either (even though tested against)
and it will kept separate instead for now, to avoid mixing too much stuff
together since not strictly needed for this series.

This V4 series is based on top of next20210610 in order to include all the
recent DT YAML-conversion changes and all the fixes queued as of today in
sudeep/for-next/scmi.

The series has been tested using an emulated fake SCMI device and also a
proper SCP-fw stack running through QEMU vhost-users, with the SCMI stack
compiled, in both cases, as builtin and as s loadable module, running tests
against mocked SCMI Sensors using HWMON and IIO interfaces to check the
functionality of notifications and sync/async commands.

Any feedback/testing is welcome :D

Thanks,
Cristian
---
V3 --> V4:
 - dropped V3 patches 02,08,12
 - using new delegated xfers support and monotonically increasing tokens
 - ported SCMI virtio transport DT bindings to YAML format
 - added virtio-scmi polling support

Cristian Marussi (8):
  firmware: arm_scmi: Fix max pending messages boundary check
  firmware: arm_scmi: Add support for type handling in common functions
  firmware: arm_scmi: Add transport optional init/exit support
  firmware: arm_scmi: Introduce monotonically increasing tokens
  firmware: arm_scmi: Introduce delegated xfers support
  [RFC][REWORK] firmware: arm_scmi: Add op to override max message #
  [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers
  firmware: arm_scmi: Add polling mode to virtio transport

Igor Skalkin (4):
  firmware: arm_scmi, smccc, mailbox: Make shmem based transports
    optional
  firmware: arm_scmi: Add op to override max message #
  dt-bindings: arm: Add virtio transport for SCMI
  firmware: arm_scmi: Add virtio transport

Peter Hilber (4):
  firmware: arm_scmi: Add optional link_supplier() transport op
  firmware: arm_scmi: Add per-device transport private info
  firmware: arm_scmi: Add is_scmi_protocol_device()
  firmware: arm_scmi: Add message passing abstractions for transports

 .../bindings/firmware/arm,scmi.yaml           |   8 +-
 MAINTAINERS                                   |   1 +
 drivers/firmware/Kconfig                      |  26 +-
 drivers/firmware/arm_scmi/Makefile            |   4 +-
 drivers/firmware/arm_scmi/bus.c               |   5 +
 drivers/firmware/arm_scmi/common.h            |  91 ++-
 drivers/firmware/arm_scmi/driver.c            | 520 +++++++++++++++--
 drivers/firmware/arm_scmi/msg.c               | 148 +++++
 drivers/firmware/arm_scmi/virtio.c            | 541 ++++++++++++++++++
 drivers/firmware/smccc/Kconfig                |   1 +
 drivers/mailbox/Kconfig                       |   1 +
 include/uapi/linux/virtio_ids.h               |   1 +
 include/uapi/linux/virtio_scmi.h              |  25 +
 13 files changed, 1326 insertions(+), 46 deletions(-)
 create mode 100644 drivers/firmware/arm_scmi/msg.c
 create mode 100644 drivers/firmware/arm_scmi/virtio.c
 create mode 100644 include/uapi/linux/virtio_scmi.h

-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 00/16] Introduce SCMI VirtIO transport
@ 2021-06-11 16:59 ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Hi all,

I'm posting this V4 series starting from the work done up to V3 by
OpenSynergy.

I kept original autorship and mailing list distribution unchanged, even
though I'm not really sure if I'm allowed to post to all the original MLs.

The main aim of this rework is to simplify where possible the SCMI VirtIO
support added in V3 by adding upfront and then using some new mechanisms in
the SCMI Core Transport layer.

Indeed patches 01,02,03,04,05 add such additional mechanisms to the SCMI
core; these were previously posted separately in a distinct patchset, I
will retire that series and continue posting those here to ease testing.

In terms of rework I dropped original V3 patches 02,08,12 as no more needed
while keeping for now the rework distinct marking such patches as
[RFC][REWORK] in order to easily spot changes.
(they will be squashed into related original V3 patches next)

DT bindings patch has been ported on top of freshly YAML converted arm,scmi
bindings.

Patch 16 adds support for polling mode to virtio-scmi.

Known/open issues in V4:

 - possible violation by virtio-scmi transport of the order constraints on
   correlated response/delayed_response messages: this will be addressed
   next with changes in the SCMI core.

 - probing sequence simplification (if possible)

As such this series is anyway still work in progress, needing also some
general cleanup.

Moreover SCMI core reworks about atomicity and polling, posted as a
distinct series, is not included here either (even though tested against)
and it will kept separate instead for now, to avoid mixing too much stuff
together since not strictly needed for this series.

This V4 series is based on top of next20210610 in order to include all the
recent DT YAML-conversion changes and all the fixes queued as of today in
sudeep/for-next/scmi.

The series has been tested using an emulated fake SCMI device and also a
proper SCP-fw stack running through QEMU vhost-users, with the SCMI stack
compiled, in both cases, as builtin and as s loadable module, running tests
against mocked SCMI Sensors using HWMON and IIO interfaces to check the
functionality of notifications and sync/async commands.

Any feedback/testing is welcome :D

Thanks,
Cristian
---
V3 --> V4:
 - dropped V3 patches 02,08,12
 - using new delegated xfers support and monotonically increasing tokens
 - ported SCMI virtio transport DT bindings to YAML format
 - added virtio-scmi polling support

Cristian Marussi (8):
  firmware: arm_scmi: Fix max pending messages boundary check
  firmware: arm_scmi: Add support for type handling in common functions
  firmware: arm_scmi: Add transport optional init/exit support
  firmware: arm_scmi: Introduce monotonically increasing tokens
  firmware: arm_scmi: Introduce delegated xfers support
  [RFC][REWORK] firmware: arm_scmi: Add op to override max message #
  [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers
  firmware: arm_scmi: Add polling mode to virtio transport

Igor Skalkin (4):
  firmware: arm_scmi, smccc, mailbox: Make shmem based transports
    optional
  firmware: arm_scmi: Add op to override max message #
  dt-bindings: arm: Add virtio transport for SCMI
  firmware: arm_scmi: Add virtio transport

Peter Hilber (4):
  firmware: arm_scmi: Add optional link_supplier() transport op
  firmware: arm_scmi: Add per-device transport private info
  firmware: arm_scmi: Add is_scmi_protocol_device()
  firmware: arm_scmi: Add message passing abstractions for transports

 .../bindings/firmware/arm,scmi.yaml           |   8 +-
 MAINTAINERS                                   |   1 +
 drivers/firmware/Kconfig                      |  26 +-
 drivers/firmware/arm_scmi/Makefile            |   4 +-
 drivers/firmware/arm_scmi/bus.c               |   5 +
 drivers/firmware/arm_scmi/common.h            |  91 ++-
 drivers/firmware/arm_scmi/driver.c            | 520 +++++++++++++++--
 drivers/firmware/arm_scmi/msg.c               | 148 +++++
 drivers/firmware/arm_scmi/virtio.c            | 541 ++++++++++++++++++
 drivers/firmware/smccc/Kconfig                |   1 +
 drivers/mailbox/Kconfig                       |   1 +
 include/uapi/linux/virtio_ids.h               |   1 +
 include/uapi/linux/virtio_scmi.h              |  25 +
 13 files changed, 1326 insertions(+), 46 deletions(-)
 create mode 100644 drivers/firmware/arm_scmi/msg.c
 create mode 100644 drivers/firmware/arm_scmi/virtio.c
 create mode 100644 include/uapi/linux/virtio_scmi.h

-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 01/16] firmware: arm_scmi: Fix max pending messages boundary check
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

SCMI message headers carry a sequence number and such field is sized to
allow for MSG_TOKEN_MAX distinct numbers; moreover zero is not really an
acceptable maximum number of pending in-flight messages.

Fix accordignly the checks performed on the value exported by transports
in scmi_desc.max_msg.

Reported-by: Vincent Guittot <vincent.guittot@linaro.org>
Fixes: aa4f886f3893 ("firmware: arm_scmi: add basic driver infrastructure for SCMI")
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/driver.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 66e5e694be7d..6713b259f1e6 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1025,8 +1025,9 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 	const struct scmi_desc *desc = sinfo->desc;
 
 	/* Pre-allocated messages, no more than what hdr.seq can support */
-	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
-		dev_err(dev, "Maximum message of %d exceeds supported %ld\n",
+	if (WARN_ON(!desc->max_msg || desc->max_msg > MSG_TOKEN_MAX)) {
+		dev_err(dev,
+			"Invalid max_msg %d. Maximum messages supported %ld.\n",
 			desc->max_msg, MSG_TOKEN_MAX);
 		return -EINVAL;
 	}
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 01/16] firmware: arm_scmi: Fix max pending messages boundary check
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

SCMI message headers carry a sequence number and such field is sized to
allow for MSG_TOKEN_MAX distinct numbers; moreover zero is not really an
acceptable maximum number of pending in-flight messages.

Fix accordignly the checks performed on the value exported by transports
in scmi_desc.max_msg.

Reported-by: Vincent Guittot <vincent.guittot@linaro.org>
Fixes: aa4f886f3893 ("firmware: arm_scmi: add basic driver infrastructure for SCMI")
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/driver.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 66e5e694be7d..6713b259f1e6 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1025,8 +1025,9 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 	const struct scmi_desc *desc = sinfo->desc;
 
 	/* Pre-allocated messages, no more than what hdr.seq can support */
-	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
-		dev_err(dev, "Maximum message of %d exceeds supported %ld\n",
+	if (WARN_ON(!desc->max_msg || desc->max_msg > MSG_TOKEN_MAX)) {
+		dev_err(dev,
+			"Invalid max_msg %d. Maximum messages supported %ld.\n",
 			desc->max_msg, MSG_TOKEN_MAX);
 		return -EINVAL;
 	}
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 02/16] firmware: arm_scmi: Add support for type handling in common functions
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Add SCMI type handling to pack/unpack_scmi_header common helper functions.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
Needed later in the series to support delegated xfers
---
 drivers/firmware/arm_scmi/common.h | 6 +++++-
 drivers/firmware/arm_scmi/driver.c | 1 +
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 8685619d38f9..7c2b9fd7e929 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -70,6 +70,7 @@ struct scmi_msg_resp_prot_version {
  *
  * @id: The identifier of the message being sent
  * @protocol_id: The identifier of the protocol used to send @id message
+ * @type: The SCMI type for this message
  * @seq: The token to identify the message. When a message returns, the
  *	platform returns the whole message header unmodified including the
  *	token
@@ -80,6 +81,7 @@ struct scmi_msg_resp_prot_version {
 struct scmi_msg_hdr {
 	u8 id;
 	u8 protocol_id;
+	u8 type;
 	u16 seq;
 	u32 status;
 	bool poll_completion;
@@ -89,13 +91,14 @@ struct scmi_msg_hdr {
  * pack_scmi_header() - packs and returns 32-bit header
  *
  * @hdr: pointer to header containing all the information on message id,
- *	protocol id and sequence id.
+ *	protocol id, sequence id and type.
  *
  * Return: 32-bit packed message header to be sent to the platform.
  */
 static inline u32 pack_scmi_header(struct scmi_msg_hdr *hdr)
 {
 	return FIELD_PREP(MSG_ID_MASK, hdr->id) |
+		FIELD_PREP(MSG_TYPE_MASK, hdr->type) |
 		FIELD_PREP(MSG_TOKEN_ID_MASK, hdr->seq) |
 		FIELD_PREP(MSG_PROTOCOL_ID_MASK, hdr->protocol_id);
 }
@@ -110,6 +113,7 @@ static inline void unpack_scmi_header(u32 msg_hdr, struct scmi_msg_hdr *hdr)
 {
 	hdr->id = MSG_XTRACT_ID(msg_hdr);
 	hdr->protocol_id = MSG_XTRACT_PROT_ID(msg_hdr);
+	hdr->type = MSG_XTRACT_TYPE(msg_hdr);
 }
 
 /**
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 6713b259f1e6..f15d75af87ea 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -565,6 +565,7 @@ static int xfer_get_init(const struct scmi_protocol_handle *ph,
 
 	xfer->tx.len = tx_size;
 	xfer->rx.len = rx_size ? : info->desc->max_msg_size;
+	xfer->hdr.type = MSG_TYPE_COMMAND;
 	xfer->hdr.id = msg_id;
 	xfer->hdr.poll_completion = false;
 
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 02/16] firmware: arm_scmi: Add support for type handling in common functions
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Add SCMI type handling to pack/unpack_scmi_header common helper functions.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
Needed later in the series to support delegated xfers
---
 drivers/firmware/arm_scmi/common.h | 6 +++++-
 drivers/firmware/arm_scmi/driver.c | 1 +
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 8685619d38f9..7c2b9fd7e929 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -70,6 +70,7 @@ struct scmi_msg_resp_prot_version {
  *
  * @id: The identifier of the message being sent
  * @protocol_id: The identifier of the protocol used to send @id message
+ * @type: The SCMI type for this message
  * @seq: The token to identify the message. When a message returns, the
  *	platform returns the whole message header unmodified including the
  *	token
@@ -80,6 +81,7 @@ struct scmi_msg_resp_prot_version {
 struct scmi_msg_hdr {
 	u8 id;
 	u8 protocol_id;
+	u8 type;
 	u16 seq;
 	u32 status;
 	bool poll_completion;
@@ -89,13 +91,14 @@ struct scmi_msg_hdr {
  * pack_scmi_header() - packs and returns 32-bit header
  *
  * @hdr: pointer to header containing all the information on message id,
- *	protocol id and sequence id.
+ *	protocol id, sequence id and type.
  *
  * Return: 32-bit packed message header to be sent to the platform.
  */
 static inline u32 pack_scmi_header(struct scmi_msg_hdr *hdr)
 {
 	return FIELD_PREP(MSG_ID_MASK, hdr->id) |
+		FIELD_PREP(MSG_TYPE_MASK, hdr->type) |
 		FIELD_PREP(MSG_TOKEN_ID_MASK, hdr->seq) |
 		FIELD_PREP(MSG_PROTOCOL_ID_MASK, hdr->protocol_id);
 }
@@ -110,6 +113,7 @@ static inline void unpack_scmi_header(u32 msg_hdr, struct scmi_msg_hdr *hdr)
 {
 	hdr->id = MSG_XTRACT_ID(msg_hdr);
 	hdr->protocol_id = MSG_XTRACT_PROT_ID(msg_hdr);
+	hdr->type = MSG_XTRACT_TYPE(msg_hdr);
 }
 
 /**
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 6713b259f1e6..f15d75af87ea 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -565,6 +565,7 @@ static int xfer_get_init(const struct scmi_protocol_handle *ph,
 
 	xfer->tx.len = tx_size;
 	xfer->rx.len = rx_size ? : info->desc->max_msg_size;
+	xfer->hdr.type = MSG_TYPE_COMMAND;
 	xfer->hdr.id = msg_id;
 	xfer->hdr.poll_completion = false;
 
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 03/16] firmware: arm_scmi: Add transport optional init/exit support
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Some SCMI transport could need to perform some transport specific setup
before they can be used by the SCMI core transport layer: typically this
early setup consists in registering with some other kernel subsystem.

Add the optional capability for a transport to provide a couple of .init
and .exit functions that are assured to be called early during the SCMI
core initialization phase, well before the SCMI core probing step.

[ Peter: Adapted RFC patch by Cristian for submission to upstream. ]
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
[ Cristian: Fixed scmi_transports_exit point of invocation ]
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |  8 ++++
 drivers/firmware/arm_scmi/driver.c | 59 ++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 7c2b9fd7e929..6bb734e0e3ac 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -321,6 +321,12 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
 /**
  * struct scmi_desc - Description of SoC integration
  *
+ * @init: An optional function that a transport can provide to initialize some
+ *	  transport-specific setup during SCMI core initialization, so ahead of
+ *	  SCMI core probing.
+ * @exit: An optional function that a transport can provide to de-initialize
+ *	  some transport-specific setup during SCMI core de-initialization, so
+ *	  after SCMI core removal.
  * @ops: Pointer to the transport specific ops structure
  * @max_rx_timeout_ms: Timeout for communication with SoC (in Milliseconds)
  * @max_msg: Maximum number of messages that can be pending
@@ -328,6 +334,8 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
  * @max_msg_size: Maximum size of data per message that can be handled.
  */
 struct scmi_desc {
+	int (*init)(void);
+	void (*exit)(void);
 	const struct scmi_transport_ops *ops;
 	int max_rx_timeout_ms;
 	int max_msg;
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index f15d75af87ea..20f8f0581f3a 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1594,10 +1594,67 @@ static struct platform_driver scmi_driver = {
 	.remove = scmi_remove,
 };
 
+/**
+ * __scmi_transports_setup  - Common helper to call transport-specific
+ * .init/.exit code if provided.
+ *
+ * @init: A flag to distinguish between init and exit.
+ *
+ * Note that, if provided, we invoke .init/.exit functions for all the
+ * transports currently compiled in.
+ *
+ * Return: 0 on Success.
+ */
+static inline int __scmi_transports_setup(bool init)
+{
+	int ret = 0;
+	const struct of_device_id *trans;
+
+	for (trans = scmi_of_match; trans->data; trans++) {
+		const struct scmi_desc *tdesc = trans->data;
+
+		if ((init && !tdesc->init) || (!init && !tdesc->exit))
+			continue;
+
+		pr_debug("SCMI %sInitializing %s transport\n",
+			 init ? "" : "De-", trans->compatible);
+
+		if (init)
+			ret = tdesc->init();
+		else
+			tdesc->exit();
+
+		if (ret) {
+			pr_err("SCMI transport %s FAILED initialization!\n",
+			       trans->compatible);
+			break;
+		}
+	}
+
+	return ret;
+}
+
+static int __init scmi_transports_init(void)
+{
+	return __scmi_transports_setup(true);
+}
+
+static void __exit scmi_transports_exit(void)
+{
+	__scmi_transports_setup(false);
+}
+
 static int __init scmi_driver_init(void)
 {
+	int ret;
+
 	scmi_bus_init();
 
+	/* Initialize any compiled-in transport which provided an init/exit */
+	ret = scmi_transports_init();
+	if (ret)
+		return ret;
+
 	scmi_base_register();
 
 	scmi_clock_register();
@@ -1626,6 +1683,8 @@ static void __exit scmi_driver_exit(void)
 
 	scmi_bus_exit();
 
+	scmi_transports_exit();
+
 	platform_driver_unregister(&scmi_driver);
 }
 module_exit(scmi_driver_exit);
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 03/16] firmware: arm_scmi: Add transport optional init/exit support
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Some SCMI transport could need to perform some transport specific setup
before they can be used by the SCMI core transport layer: typically this
early setup consists in registering with some other kernel subsystem.

Add the optional capability for a transport to provide a couple of .init
and .exit functions that are assured to be called early during the SCMI
core initialization phase, well before the SCMI core probing step.

[ Peter: Adapted RFC patch by Cristian for submission to upstream. ]
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
[ Cristian: Fixed scmi_transports_exit point of invocation ]
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |  8 ++++
 drivers/firmware/arm_scmi/driver.c | 59 ++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 7c2b9fd7e929..6bb734e0e3ac 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -321,6 +321,12 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
 /**
  * struct scmi_desc - Description of SoC integration
  *
+ * @init: An optional function that a transport can provide to initialize some
+ *	  transport-specific setup during SCMI core initialization, so ahead of
+ *	  SCMI core probing.
+ * @exit: An optional function that a transport can provide to de-initialize
+ *	  some transport-specific setup during SCMI core de-initialization, so
+ *	  after SCMI core removal.
  * @ops: Pointer to the transport specific ops structure
  * @max_rx_timeout_ms: Timeout for communication with SoC (in Milliseconds)
  * @max_msg: Maximum number of messages that can be pending
@@ -328,6 +334,8 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
  * @max_msg_size: Maximum size of data per message that can be handled.
  */
 struct scmi_desc {
+	int (*init)(void);
+	void (*exit)(void);
 	const struct scmi_transport_ops *ops;
 	int max_rx_timeout_ms;
 	int max_msg;
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index f15d75af87ea..20f8f0581f3a 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1594,10 +1594,67 @@ static struct platform_driver scmi_driver = {
 	.remove = scmi_remove,
 };
 
+/**
+ * __scmi_transports_setup  - Common helper to call transport-specific
+ * .init/.exit code if provided.
+ *
+ * @init: A flag to distinguish between init and exit.
+ *
+ * Note that, if provided, we invoke .init/.exit functions for all the
+ * transports currently compiled in.
+ *
+ * Return: 0 on Success.
+ */
+static inline int __scmi_transports_setup(bool init)
+{
+	int ret = 0;
+	const struct of_device_id *trans;
+
+	for (trans = scmi_of_match; trans->data; trans++) {
+		const struct scmi_desc *tdesc = trans->data;
+
+		if ((init && !tdesc->init) || (!init && !tdesc->exit))
+			continue;
+
+		pr_debug("SCMI %sInitializing %s transport\n",
+			 init ? "" : "De-", trans->compatible);
+
+		if (init)
+			ret = tdesc->init();
+		else
+			tdesc->exit();
+
+		if (ret) {
+			pr_err("SCMI transport %s FAILED initialization!\n",
+			       trans->compatible);
+			break;
+		}
+	}
+
+	return ret;
+}
+
+static int __init scmi_transports_init(void)
+{
+	return __scmi_transports_setup(true);
+}
+
+static void __exit scmi_transports_exit(void)
+{
+	__scmi_transports_setup(false);
+}
+
 static int __init scmi_driver_init(void)
 {
+	int ret;
+
 	scmi_bus_init();
 
+	/* Initialize any compiled-in transport which provided an init/exit */
+	ret = scmi_transports_init();
+	if (ret)
+		return ret;
+
 	scmi_base_register();
 
 	scmi_clock_register();
@@ -1626,6 +1683,8 @@ static void __exit scmi_driver_exit(void)
 
 	scmi_bus_exit();
 
+	scmi_transports_exit();
+
 	platform_driver_unregister(&scmi_driver);
 }
 module_exit(scmi_driver_exit);
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Tokens are sequence numbers embedded in the each SCMI message header: they
are used to correlate commands with responses (and delayed responses), but
their usage and policy of selection is entirely up to the caller (usually
the OSPM agent), while they are completely opaque to the callee (SCMI
server platform) which merely copies them back from the command into the
response message header.
This also means that the platform does not, can not and should not enforce
any kind of policy on received messages depending on the contained sequence
number: platform can perfectly handle concurrent requests carrying the same
identifiying token if that should happen.

Moreover the platform is not required to produce in-order responses to
agent requests, the only constraint in these regards is that in case of
an asynchronous message the delayed response must be sent after the
immediate response for the synchronous part of the command transaction.

Currenly the SCMI stack of the OSPM agent selects a token for the egressing
commands picking the lowest possible number which is not already in use by
an existing in-flight transaction, which means, in other words, that we
immediately reuse any token after its transaction has completed or it has
timed out: this policy indeed does simplify management and lookup of tokens
and associated xfers.

Under the above assumptions and constraints, since there is really no state
shared between the agent and the platform to let the platform know when a
token and its associated message has timed out, the current policy of early
reuse of tokens can easily lead to the situation in which a spurious or
late received response (or delayed_response), related to an old stale and
timed out transaction, can be wrongly associated to a newer valid in-flight
xfer that just happens to have reused the same token.

This misbehaviour on such ghost responses is more easily exposed on those
transports that naturally have an higher level of parallelism in processing
multiple concurrent in-flight messages.

This commit introduces a new policy of selection of tokens for the OSPM
agent: each new transfer now gets the next available and monotonically
increasing token, until tokens are exhausted and the counter rolls over.

Such new policy mitigates the above issues with ghost responses since the
tokens are now reused as late as possible (when they roll back ideally)
and so it is much easier to identify such ghost responses to stale timed
out transactions: this also helps in simplifying the specific transports
implementation since stale transport messages can be easily identified
and discarded early on in the rx path without the need to cross check
their actual state with the core transport layer.
This mitigation is even more effective when, as is usually the case, the
maximum number of pending messages is capped by the platform to a much
lower number than the whole possible range of tokens values (2^10).

This internal policy change in the core SCMI transport layer is fully
transparent to the specific transports so it has not and should not have
any impact on the transports implementation.

The empirically observed cost of such new procedure of token selection
amounts in the best case to ~10us out of an observed full transaction cost
of 3ms for the completion of a synchronous sensor reading command on a
platform supporting commands completion interrupts.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |  23 +++
 drivers/firmware/arm_scmi/driver.c | 243 +++++++++++++++++++++++++----
 2 files changed, 233 insertions(+), 33 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 6bb734e0e3ac..e64c5ca9ee7c 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -14,7 +14,10 @@
 #include <linux/device.h>
 #include <linux/errno.h>
 #include <linux/kernel.h>
+#include <linux/hashtable.h>
+#include <linux/list.h>
 #include <linux/module.h>
+#include <linux/refcount.h>
 #include <linux/scmi_protocol.h>
 #include <linux/types.h>
 
@@ -127,6 +130,21 @@ struct scmi_msg {
 	size_t len;
 };
 
+/**
+ * An helper macro to lookup an xfer from the @pending_xfers hashtable
+ * using the message sequence number token as a key.
+ */
+#define XFER_FIND(__ht, __k)					\
+({								\
+	typeof(__k) k_ = __k;					\
+	struct scmi_xfer *xfer_ = NULL;				\
+								\
+	hash_for_each_possible((__ht), xfer_, node, k_)		\
+		if (xfer_->hdr.seq == k_)			\
+			break;					\
+	 xfer_;							\
+})
+
 /**
  * struct scmi_xfer - Structure representing a message flow
  *
@@ -138,6 +156,9 @@ struct scmi_msg {
  *	buffer for the rx path as we use for the tx path.
  * @done: command message transmit completion event
  * @async_done: pointer to delayed response message received event completion
+ * @users: A refcount to track the active users for this xfer
+ * @node: An hlist_node reference used to store this xfer, alternatively, on
+ *	  the free list @free_xfers or in the @pending_xfers hashtable
  */
 struct scmi_xfer {
 	int transfer_id;
@@ -146,6 +167,8 @@ struct scmi_xfer {
 	struct scmi_msg rx;
 	struct completion done;
 	struct completion *async_done;
+	refcount_t users;
+	struct hlist_node node;
 };
 
 struct scmi_xfer_ops;
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 20f8f0581f3a..f0b20ddb24f4 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -21,6 +21,7 @@
 #include <linux/io.h>
 #include <linux/kernel.h>
 #include <linux/ktime.h>
+#include <linux/hashtable.h>
 #include <linux/list.h>
 #include <linux/module.h>
 #include <linux/of_address.h>
@@ -65,19 +66,29 @@ struct scmi_requested_dev {
 	struct list_head node;
 };
 
+#define SCMI_PENDING_XFERS_HT_ORDER_SZ	9
+
 /**
  * struct scmi_xfers_info - Structure to manage transfer information
  *
- * @xfer_block: Preallocated Message array
  * @xfer_alloc_table: Bitmap table for allocated messages.
  *	Index of this bitmap table is also used for message
  *	sequence identifier.
  * @xfer_lock: Protection for message allocation
+ * @last_token: A counter to use as base to generate for monotonically
+ *		increasing tokens.
+ * @free_xfers: A free list for available to use xfers. It is initialized with
+ *		a number of xfers equal to the maximum allowed in-flight
+ *		messages.
+ * @pending_xfers: An hashtable, indexed by msg_hdr.seq, used to keep all the
+ *		   currently in-flight messages.
  */
 struct scmi_xfers_info {
-	struct scmi_xfer *xfer_block;
 	unsigned long *xfer_alloc_table;
 	spinlock_t xfer_lock;
+	atomic_t last_token;
+	struct hlist_head free_xfers;
+	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
 };
 
 /**
@@ -203,6 +214,117 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
 	return info->notify_priv;
 }
 
+/**
+ * scmi_xfer_token_set  - Reserve and set new token for the xfer at hand
+ *
+ * @minfo: Pointer to Tx/Rx Message management info based on channel type
+ * @xfer: The xfer to act upon
+ *
+ * Pick the next unused monotonically increasing token and set it into
+ * xfer->hdr.seq: picking a monotonically increasing value avoids immediate
+ * reuse of freshly completed or timed-out xfers, thus mitigating the risk
+ * of incorrect association of a late and expired xfer with a live in-flight
+ * transaction, both happening to re-use the same token identifier.
+ *
+ * Since platform is NOT required to answer our request in-order we should
+ * account for a few rare but possible scenarios:
+ *
+ *  - exactly 'next_token' may be NOT available so pick xfer_id >= next_token
+ *    using find_next_zero_bit() starting from candidate next_token bit
+ *
+ *  - all tokens ahead upto (MSG_TOKEN_ID_MASK - 1) are used in-flight but we
+ *    are plenty of free tokens at start, so try a second pass using
+ *    find_next_zero_bit() and starting from 0.
+ *
+ *  X = used in-flight
+ *
+ * Normal
+ * ------
+ *
+ *		|- xfer_id picked
+ *   -----------+----------------------------------------------------------
+ *   | | |X|X|X| | | | | | ... ... ... ... ... ... ... ... ... ... ...|X|X|
+ *   ----------------------------------------------------------------------
+ *		^
+ *		|- next_token
+ *
+ * Out-of-order pending at start
+ * -----------------------------
+ *
+ *	  |- xfer_id picked, last_token fixed
+ *   -----+----------------------------------------------------------------
+ *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... ... ...|X| |
+ *   ----------------------------------------------------------------------
+ *    ^
+ *    |- next_token
+ *
+ *
+ * Out-of-order pending at end
+ * ---------------------------
+ *
+ *	  |- xfer_id picked, last_token fixed
+ *   -----+----------------------------------------------------------------
+ *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... |X|X|X||X|X|
+ *   ----------------------------------------------------------------------
+ *								^
+ *								|- next_token
+ *
+ * Context: Assumes to be called with @xfer_lock already acquired.
+ *
+ * Return: 0 on Success or error
+ */
+static int scmi_xfer_token_set(struct scmi_xfers_info *minfo,
+			       struct scmi_xfer *xfer)
+{
+	unsigned long xfer_id, next_token;
+
+	/* Pick a candidate monotonic token in range [0, MSG_TOKEN_MAX - 1] */
+	next_token = (atomic_inc_return(&minfo->last_token) &
+		      (MSG_TOKEN_MAX - 1));
+
+	/* Pick the next available xfer_id >= next_token */
+	xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
+				     MSG_TOKEN_MAX, next_token);
+	if (xfer_id == MSG_TOKEN_MAX) {
+		/*
+		 * After heavily out-of-order responses, there are no free
+		 * tokens ahead, but only at start of xfer_alloc_table so
+		 * try again from the beginning.
+		 */
+		xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
+					     MSG_TOKEN_MAX, 0);
+		/*
+		 * Something is wrong if we got here since there can be a
+		 * maximum number of (MSG_TOKEN_MAX - 1) in-flight messages
+		 * but we have not found any free token [0, MSG_TOKEN_MAX - 1].
+		 */
+		if (WARN_ON_ONCE(xfer_id == MSG_TOKEN_MAX))
+			return -ENOMEM;
+	}
+
+	/* Update +/- last_token accordingly if we skipped some hole */
+	if (xfer_id != next_token)
+		atomic_add((int)(xfer_id - next_token), &minfo->last_token);
+
+	/* Set in-flight */
+	set_bit(xfer_id, minfo->xfer_alloc_table);
+	xfer->hdr.seq = (u16)xfer_id;
+
+	return 0;
+}
+
+/**
+ * scmi_xfer_token_clear  - Release the token
+ *
+ * @minfo: Pointer to Tx/Rx Message management info based on channel type
+ * @xfer: The xfer to act upon
+ */
+static inline void scmi_xfer_token_clear(struct scmi_xfers_info *minfo,
+					 struct scmi_xfer *xfer)
+{
+	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
+}
+
 /**
  * scmi_xfer_get() - Allocate one message
  *
@@ -212,36 +334,49 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
  * Helper function which is used by various message functions that are
  * exposed to clients of this driver for allocating a message traffic event.
  *
- * This function can sleep depending on pending requests already in the system
- * for the SCMI entity. Further, this also holds a spinlock to maintain
- * integrity of internal data structures.
+ * Picks an xfer from the free list @free_xfers (if any available), sets a
+ * monotonically increasing token and stores the inflight xfer into the
+ * @pending_xfers hashtable for later retrieval.
+ *
+ * The successfully initialized xfer is refcounted.
+ *
+ * Context: Holds @xfer_lock while manipulating @xfer_alloc_table and
+ *	    @free_xfers.
  *
  * Return: 0 if all went fine, else corresponding error.
  */
 static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
 				       struct scmi_xfers_info *minfo)
 {
-	u16 xfer_id;
+	int ret;
+	unsigned long flags;
 	struct scmi_xfer *xfer;
-	unsigned long flags, bit_pos;
-	struct scmi_info *info = handle_to_scmi_info(handle);
 
-	/* Keep the locked section as small as possible */
 	spin_lock_irqsave(&minfo->xfer_lock, flags);
-	bit_pos = find_first_zero_bit(minfo->xfer_alloc_table,
-				      info->desc->max_msg);
-	if (bit_pos == info->desc->max_msg) {
+	if (hlist_empty(&minfo->free_xfers)) {
 		spin_unlock_irqrestore(&minfo->xfer_lock, flags);
 		return ERR_PTR(-ENOMEM);
 	}
-	set_bit(bit_pos, minfo->xfer_alloc_table);
-	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
 
-	xfer_id = bit_pos;
+	/* grab an xfer from the free_list */
+	xfer = hlist_entry(minfo->free_xfers.first, struct scmi_xfer, node);
+	hlist_del_init(&xfer->node);
 
-	xfer = &minfo->xfer_block[xfer_id];
-	xfer->hdr.seq = xfer_id;
-	xfer->transfer_id = atomic_inc_return(&transfer_last_id);
+	/* Pick and set monotonic token */
+	ret = scmi_xfer_token_set(minfo, xfer);
+	if (!ret) {
+		hash_add(minfo->pending_xfers, &xfer->node, xfer->hdr.seq);
+	} else {
+		dev_err(handle->dev, "Failed to get monotonic token %d\n", ret);
+		hlist_add_head(&xfer->node, &minfo->free_xfers);
+		xfer = ERR_PTR(ret);
+	}
+	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
+
+	if (!IS_ERR(xfer)) {
+		refcount_set(&xfer->users, 1);
+		xfer->transfer_id = atomic_inc_return(&transfer_last_id);
+	}
 
 	return xfer;
 }
@@ -252,6 +387,9 @@ static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
  * @minfo: Pointer to Tx/Rx Message management info based on channel type
  * @xfer: message that was reserved by scmi_xfer_get
  *
+ * After refcount check, possibly release an xfer, clearing the token slot,
+ * removing xfer from @pending_xfers and putting it back into free_xfers.
+ *
  * This holds a spinlock to maintain integrity of internal data structures.
  */
 static void
@@ -259,16 +397,41 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
 {
 	unsigned long flags;
 
-	/*
-	 * Keep the locked section as small as possible
-	 * NOTE: we might escape with smp_mb and no lock here..
-	 * but just be conservative and symmetric.
-	 */
 	spin_lock_irqsave(&minfo->xfer_lock, flags);
-	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
+	if (refcount_dec_and_test(&xfer->users)) {
+		scmi_xfer_token_clear(minfo, xfer);
+		hash_del(&xfer->node);
+		hlist_add_head(&xfer->node, &minfo->free_xfers);
+	}
 	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
 }
 
+/**
+ * scmi_xfer_lookup_unlocked  -  Helper to lookup an xfer_id
+ *
+ * @minfo: Pointer to Tx/Rx Message management info based on channel type
+ * @xfer_id: Token ID to lookup in @pending_xfers
+ *
+ * Refcounting is untouched.
+ *
+ * Context: Assumes to be called with @xfer_lock already acquired.
+ *
+ * Return: A valid xfer on Success or error otherwise
+ */
+static struct scmi_xfer *
+scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
+{
+	struct scmi_xfer *xfer = NULL;
+
+	if (xfer_id >= MSG_TOKEN_MAX)
+		return ERR_PTR(-EINVAL);
+
+	if (test_bit(xfer_id, minfo->xfer_alloc_table))
+		xfer = XFER_FIND(minfo->pending_xfers, xfer_id);
+
+	return xfer ?: ERR_PTR(-EINVAL);
+}
+
 static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 {
 	struct scmi_xfer *xfer;
@@ -305,19 +468,22 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 static void scmi_handle_response(struct scmi_chan_info *cinfo,
 				 u16 xfer_id, u8 msg_type)
 {
+	unsigned long flags;
 	struct scmi_xfer *xfer;
 	struct device *dev = cinfo->dev;
 	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
 	struct scmi_xfers_info *minfo = &info->tx_minfo;
 
 	/* Are we even expecting this? */
-	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
+	spin_lock_irqsave(&minfo->xfer_lock, flags);
+	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
+	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
+	if (IS_ERR(xfer)) {
 		dev_err(dev, "message for %d is not expected!\n", xfer_id);
 		info->desc->ops->clear_channel(cinfo);
 		return;
 	}
 
-	xfer = &minfo->xfer_block[xfer_id];
 	/*
 	 * Even if a response was indeed expected on this slot at this point,
 	 * a buggy platform could wrongly reply feeding us an unexpected
@@ -1033,18 +1199,25 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 		return -EINVAL;
 	}
 
-	info->xfer_block = devm_kcalloc(dev, desc->max_msg,
-					sizeof(*info->xfer_block), GFP_KERNEL);
-	if (!info->xfer_block)
-		return -ENOMEM;
+	hash_init(info->pending_xfers);
 
-	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(desc->max_msg),
+	/* Allocate a bitmask sized to hold MSG_TOKEN_MAX tokens */
+	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(MSG_TOKEN_MAX),
 					      sizeof(long), GFP_KERNEL);
 	if (!info->xfer_alloc_table)
 		return -ENOMEM;
 
-	/* Pre-initialize the buffer pointer to pre-allocated buffers */
-	for (i = 0, xfer = info->xfer_block; i < desc->max_msg; i++, xfer++) {
+	/*
+	 * Preallocate a number of xfers equal to max inflight messages,
+	 * pre-initialize the buffer pointer to pre-allocated buffers and
+	 * attach all of them to the free list
+	 */
+	INIT_HLIST_HEAD(&info->free_xfers);
+	for (i = 0; i < desc->max_msg; i++) {
+		xfer = devm_kzalloc(dev, sizeof(*xfer), GFP_KERNEL);
+		if (!xfer)
+			return -ENOMEM;
+
 		xfer->rx.buf = devm_kcalloc(dev, sizeof(u8), desc->max_msg_size,
 					    GFP_KERNEL);
 		if (!xfer->rx.buf)
@@ -1052,8 +1225,12 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 
 		xfer->tx.buf = xfer->rx.buf;
 		init_completion(&xfer->done);
+
+		/* Add initialized xfer to the free list */
+		hlist_add_head(&xfer->node, &info->free_xfers);
 	}
 
+	atomic_set(&info->last_token, -1);
 	spin_lock_init(&info->xfer_lock);
 
 	return 0;
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Tokens are sequence numbers embedded in the each SCMI message header: they
are used to correlate commands with responses (and delayed responses), but
their usage and policy of selection is entirely up to the caller (usually
the OSPM agent), while they are completely opaque to the callee (SCMI
server platform) which merely copies them back from the command into the
response message header.
This also means that the platform does not, can not and should not enforce
any kind of policy on received messages depending on the contained sequence
number: platform can perfectly handle concurrent requests carrying the same
identifiying token if that should happen.

Moreover the platform is not required to produce in-order responses to
agent requests, the only constraint in these regards is that in case of
an asynchronous message the delayed response must be sent after the
immediate response for the synchronous part of the command transaction.

Currenly the SCMI stack of the OSPM agent selects a token for the egressing
commands picking the lowest possible number which is not already in use by
an existing in-flight transaction, which means, in other words, that we
immediately reuse any token after its transaction has completed or it has
timed out: this policy indeed does simplify management and lookup of tokens
and associated xfers.

Under the above assumptions and constraints, since there is really no state
shared between the agent and the platform to let the platform know when a
token and its associated message has timed out, the current policy of early
reuse of tokens can easily lead to the situation in which a spurious or
late received response (or delayed_response), related to an old stale and
timed out transaction, can be wrongly associated to a newer valid in-flight
xfer that just happens to have reused the same token.

This misbehaviour on such ghost responses is more easily exposed on those
transports that naturally have an higher level of parallelism in processing
multiple concurrent in-flight messages.

This commit introduces a new policy of selection of tokens for the OSPM
agent: each new transfer now gets the next available and monotonically
increasing token, until tokens are exhausted and the counter rolls over.

Such new policy mitigates the above issues with ghost responses since the
tokens are now reused as late as possible (when they roll back ideally)
and so it is much easier to identify such ghost responses to stale timed
out transactions: this also helps in simplifying the specific transports
implementation since stale transport messages can be easily identified
and discarded early on in the rx path without the need to cross check
their actual state with the core transport layer.
This mitigation is even more effective when, as is usually the case, the
maximum number of pending messages is capped by the platform to a much
lower number than the whole possible range of tokens values (2^10).

This internal policy change in the core SCMI transport layer is fully
transparent to the specific transports so it has not and should not have
any impact on the transports implementation.

The empirically observed cost of such new procedure of token selection
amounts in the best case to ~10us out of an observed full transaction cost
of 3ms for the completion of a synchronous sensor reading command on a
platform supporting commands completion interrupts.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |  23 +++
 drivers/firmware/arm_scmi/driver.c | 243 +++++++++++++++++++++++++----
 2 files changed, 233 insertions(+), 33 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 6bb734e0e3ac..e64c5ca9ee7c 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -14,7 +14,10 @@
 #include <linux/device.h>
 #include <linux/errno.h>
 #include <linux/kernel.h>
+#include <linux/hashtable.h>
+#include <linux/list.h>
 #include <linux/module.h>
+#include <linux/refcount.h>
 #include <linux/scmi_protocol.h>
 #include <linux/types.h>
 
@@ -127,6 +130,21 @@ struct scmi_msg {
 	size_t len;
 };
 
+/**
+ * An helper macro to lookup an xfer from the @pending_xfers hashtable
+ * using the message sequence number token as a key.
+ */
+#define XFER_FIND(__ht, __k)					\
+({								\
+	typeof(__k) k_ = __k;					\
+	struct scmi_xfer *xfer_ = NULL;				\
+								\
+	hash_for_each_possible((__ht), xfer_, node, k_)		\
+		if (xfer_->hdr.seq == k_)			\
+			break;					\
+	 xfer_;							\
+})
+
 /**
  * struct scmi_xfer - Structure representing a message flow
  *
@@ -138,6 +156,9 @@ struct scmi_msg {
  *	buffer for the rx path as we use for the tx path.
  * @done: command message transmit completion event
  * @async_done: pointer to delayed response message received event completion
+ * @users: A refcount to track the active users for this xfer
+ * @node: An hlist_node reference used to store this xfer, alternatively, on
+ *	  the free list @free_xfers or in the @pending_xfers hashtable
  */
 struct scmi_xfer {
 	int transfer_id;
@@ -146,6 +167,8 @@ struct scmi_xfer {
 	struct scmi_msg rx;
 	struct completion done;
 	struct completion *async_done;
+	refcount_t users;
+	struct hlist_node node;
 };
 
 struct scmi_xfer_ops;
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 20f8f0581f3a..f0b20ddb24f4 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -21,6 +21,7 @@
 #include <linux/io.h>
 #include <linux/kernel.h>
 #include <linux/ktime.h>
+#include <linux/hashtable.h>
 #include <linux/list.h>
 #include <linux/module.h>
 #include <linux/of_address.h>
@@ -65,19 +66,29 @@ struct scmi_requested_dev {
 	struct list_head node;
 };
 
+#define SCMI_PENDING_XFERS_HT_ORDER_SZ	9
+
 /**
  * struct scmi_xfers_info - Structure to manage transfer information
  *
- * @xfer_block: Preallocated Message array
  * @xfer_alloc_table: Bitmap table for allocated messages.
  *	Index of this bitmap table is also used for message
  *	sequence identifier.
  * @xfer_lock: Protection for message allocation
+ * @last_token: A counter to use as base to generate for monotonically
+ *		increasing tokens.
+ * @free_xfers: A free list for available to use xfers. It is initialized with
+ *		a number of xfers equal to the maximum allowed in-flight
+ *		messages.
+ * @pending_xfers: An hashtable, indexed by msg_hdr.seq, used to keep all the
+ *		   currently in-flight messages.
  */
 struct scmi_xfers_info {
-	struct scmi_xfer *xfer_block;
 	unsigned long *xfer_alloc_table;
 	spinlock_t xfer_lock;
+	atomic_t last_token;
+	struct hlist_head free_xfers;
+	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
 };
 
 /**
@@ -203,6 +214,117 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
 	return info->notify_priv;
 }
 
+/**
+ * scmi_xfer_token_set  - Reserve and set new token for the xfer at hand
+ *
+ * @minfo: Pointer to Tx/Rx Message management info based on channel type
+ * @xfer: The xfer to act upon
+ *
+ * Pick the next unused monotonically increasing token and set it into
+ * xfer->hdr.seq: picking a monotonically increasing value avoids immediate
+ * reuse of freshly completed or timed-out xfers, thus mitigating the risk
+ * of incorrect association of a late and expired xfer with a live in-flight
+ * transaction, both happening to re-use the same token identifier.
+ *
+ * Since platform is NOT required to answer our request in-order we should
+ * account for a few rare but possible scenarios:
+ *
+ *  - exactly 'next_token' may be NOT available so pick xfer_id >= next_token
+ *    using find_next_zero_bit() starting from candidate next_token bit
+ *
+ *  - all tokens ahead upto (MSG_TOKEN_ID_MASK - 1) are used in-flight but we
+ *    are plenty of free tokens at start, so try a second pass using
+ *    find_next_zero_bit() and starting from 0.
+ *
+ *  X = used in-flight
+ *
+ * Normal
+ * ------
+ *
+ *		|- xfer_id picked
+ *   -----------+----------------------------------------------------------
+ *   | | |X|X|X| | | | | | ... ... ... ... ... ... ... ... ... ... ...|X|X|
+ *   ----------------------------------------------------------------------
+ *		^
+ *		|- next_token
+ *
+ * Out-of-order pending at start
+ * -----------------------------
+ *
+ *	  |- xfer_id picked, last_token fixed
+ *   -----+----------------------------------------------------------------
+ *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... ... ...|X| |
+ *   ----------------------------------------------------------------------
+ *    ^
+ *    |- next_token
+ *
+ *
+ * Out-of-order pending at end
+ * ---------------------------
+ *
+ *	  |- xfer_id picked, last_token fixed
+ *   -----+----------------------------------------------------------------
+ *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... |X|X|X||X|X|
+ *   ----------------------------------------------------------------------
+ *								^
+ *								|- next_token
+ *
+ * Context: Assumes to be called with @xfer_lock already acquired.
+ *
+ * Return: 0 on Success or error
+ */
+static int scmi_xfer_token_set(struct scmi_xfers_info *minfo,
+			       struct scmi_xfer *xfer)
+{
+	unsigned long xfer_id, next_token;
+
+	/* Pick a candidate monotonic token in range [0, MSG_TOKEN_MAX - 1] */
+	next_token = (atomic_inc_return(&minfo->last_token) &
+		      (MSG_TOKEN_MAX - 1));
+
+	/* Pick the next available xfer_id >= next_token */
+	xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
+				     MSG_TOKEN_MAX, next_token);
+	if (xfer_id == MSG_TOKEN_MAX) {
+		/*
+		 * After heavily out-of-order responses, there are no free
+		 * tokens ahead, but only at start of xfer_alloc_table so
+		 * try again from the beginning.
+		 */
+		xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
+					     MSG_TOKEN_MAX, 0);
+		/*
+		 * Something is wrong if we got here since there can be a
+		 * maximum number of (MSG_TOKEN_MAX - 1) in-flight messages
+		 * but we have not found any free token [0, MSG_TOKEN_MAX - 1].
+		 */
+		if (WARN_ON_ONCE(xfer_id == MSG_TOKEN_MAX))
+			return -ENOMEM;
+	}
+
+	/* Update +/- last_token accordingly if we skipped some hole */
+	if (xfer_id != next_token)
+		atomic_add((int)(xfer_id - next_token), &minfo->last_token);
+
+	/* Set in-flight */
+	set_bit(xfer_id, minfo->xfer_alloc_table);
+	xfer->hdr.seq = (u16)xfer_id;
+
+	return 0;
+}
+
+/**
+ * scmi_xfer_token_clear  - Release the token
+ *
+ * @minfo: Pointer to Tx/Rx Message management info based on channel type
+ * @xfer: The xfer to act upon
+ */
+static inline void scmi_xfer_token_clear(struct scmi_xfers_info *minfo,
+					 struct scmi_xfer *xfer)
+{
+	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
+}
+
 /**
  * scmi_xfer_get() - Allocate one message
  *
@@ -212,36 +334,49 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
  * Helper function which is used by various message functions that are
  * exposed to clients of this driver for allocating a message traffic event.
  *
- * This function can sleep depending on pending requests already in the system
- * for the SCMI entity. Further, this also holds a spinlock to maintain
- * integrity of internal data structures.
+ * Picks an xfer from the free list @free_xfers (if any available), sets a
+ * monotonically increasing token and stores the inflight xfer into the
+ * @pending_xfers hashtable for later retrieval.
+ *
+ * The successfully initialized xfer is refcounted.
+ *
+ * Context: Holds @xfer_lock while manipulating @xfer_alloc_table and
+ *	    @free_xfers.
  *
  * Return: 0 if all went fine, else corresponding error.
  */
 static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
 				       struct scmi_xfers_info *minfo)
 {
-	u16 xfer_id;
+	int ret;
+	unsigned long flags;
 	struct scmi_xfer *xfer;
-	unsigned long flags, bit_pos;
-	struct scmi_info *info = handle_to_scmi_info(handle);
 
-	/* Keep the locked section as small as possible */
 	spin_lock_irqsave(&minfo->xfer_lock, flags);
-	bit_pos = find_first_zero_bit(minfo->xfer_alloc_table,
-				      info->desc->max_msg);
-	if (bit_pos == info->desc->max_msg) {
+	if (hlist_empty(&minfo->free_xfers)) {
 		spin_unlock_irqrestore(&minfo->xfer_lock, flags);
 		return ERR_PTR(-ENOMEM);
 	}
-	set_bit(bit_pos, minfo->xfer_alloc_table);
-	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
 
-	xfer_id = bit_pos;
+	/* grab an xfer from the free_list */
+	xfer = hlist_entry(minfo->free_xfers.first, struct scmi_xfer, node);
+	hlist_del_init(&xfer->node);
 
-	xfer = &minfo->xfer_block[xfer_id];
-	xfer->hdr.seq = xfer_id;
-	xfer->transfer_id = atomic_inc_return(&transfer_last_id);
+	/* Pick and set monotonic token */
+	ret = scmi_xfer_token_set(minfo, xfer);
+	if (!ret) {
+		hash_add(minfo->pending_xfers, &xfer->node, xfer->hdr.seq);
+	} else {
+		dev_err(handle->dev, "Failed to get monotonic token %d\n", ret);
+		hlist_add_head(&xfer->node, &minfo->free_xfers);
+		xfer = ERR_PTR(ret);
+	}
+	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
+
+	if (!IS_ERR(xfer)) {
+		refcount_set(&xfer->users, 1);
+		xfer->transfer_id = atomic_inc_return(&transfer_last_id);
+	}
 
 	return xfer;
 }
@@ -252,6 +387,9 @@ static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
  * @minfo: Pointer to Tx/Rx Message management info based on channel type
  * @xfer: message that was reserved by scmi_xfer_get
  *
+ * After refcount check, possibly release an xfer, clearing the token slot,
+ * removing xfer from @pending_xfers and putting it back into free_xfers.
+ *
  * This holds a spinlock to maintain integrity of internal data structures.
  */
 static void
@@ -259,16 +397,41 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
 {
 	unsigned long flags;
 
-	/*
-	 * Keep the locked section as small as possible
-	 * NOTE: we might escape with smp_mb and no lock here..
-	 * but just be conservative and symmetric.
-	 */
 	spin_lock_irqsave(&minfo->xfer_lock, flags);
-	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
+	if (refcount_dec_and_test(&xfer->users)) {
+		scmi_xfer_token_clear(minfo, xfer);
+		hash_del(&xfer->node);
+		hlist_add_head(&xfer->node, &minfo->free_xfers);
+	}
 	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
 }
 
+/**
+ * scmi_xfer_lookup_unlocked  -  Helper to lookup an xfer_id
+ *
+ * @minfo: Pointer to Tx/Rx Message management info based on channel type
+ * @xfer_id: Token ID to lookup in @pending_xfers
+ *
+ * Refcounting is untouched.
+ *
+ * Context: Assumes to be called with @xfer_lock already acquired.
+ *
+ * Return: A valid xfer on Success or error otherwise
+ */
+static struct scmi_xfer *
+scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
+{
+	struct scmi_xfer *xfer = NULL;
+
+	if (xfer_id >= MSG_TOKEN_MAX)
+		return ERR_PTR(-EINVAL);
+
+	if (test_bit(xfer_id, minfo->xfer_alloc_table))
+		xfer = XFER_FIND(minfo->pending_xfers, xfer_id);
+
+	return xfer ?: ERR_PTR(-EINVAL);
+}
+
 static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 {
 	struct scmi_xfer *xfer;
@@ -305,19 +468,22 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 static void scmi_handle_response(struct scmi_chan_info *cinfo,
 				 u16 xfer_id, u8 msg_type)
 {
+	unsigned long flags;
 	struct scmi_xfer *xfer;
 	struct device *dev = cinfo->dev;
 	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
 	struct scmi_xfers_info *minfo = &info->tx_minfo;
 
 	/* Are we even expecting this? */
-	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
+	spin_lock_irqsave(&minfo->xfer_lock, flags);
+	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
+	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
+	if (IS_ERR(xfer)) {
 		dev_err(dev, "message for %d is not expected!\n", xfer_id);
 		info->desc->ops->clear_channel(cinfo);
 		return;
 	}
 
-	xfer = &minfo->xfer_block[xfer_id];
 	/*
 	 * Even if a response was indeed expected on this slot at this point,
 	 * a buggy platform could wrongly reply feeding us an unexpected
@@ -1033,18 +1199,25 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 		return -EINVAL;
 	}
 
-	info->xfer_block = devm_kcalloc(dev, desc->max_msg,
-					sizeof(*info->xfer_block), GFP_KERNEL);
-	if (!info->xfer_block)
-		return -ENOMEM;
+	hash_init(info->pending_xfers);
 
-	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(desc->max_msg),
+	/* Allocate a bitmask sized to hold MSG_TOKEN_MAX tokens */
+	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(MSG_TOKEN_MAX),
 					      sizeof(long), GFP_KERNEL);
 	if (!info->xfer_alloc_table)
 		return -ENOMEM;
 
-	/* Pre-initialize the buffer pointer to pre-allocated buffers */
-	for (i = 0, xfer = info->xfer_block; i < desc->max_msg; i++, xfer++) {
+	/*
+	 * Preallocate a number of xfers equal to max inflight messages,
+	 * pre-initialize the buffer pointer to pre-allocated buffers and
+	 * attach all of them to the free list
+	 */
+	INIT_HLIST_HEAD(&info->free_xfers);
+	for (i = 0; i < desc->max_msg; i++) {
+		xfer = devm_kzalloc(dev, sizeof(*xfer), GFP_KERNEL);
+		if (!xfer)
+			return -ENOMEM;
+
 		xfer->rx.buf = devm_kcalloc(dev, sizeof(u8), desc->max_msg_size,
 					    GFP_KERNEL);
 		if (!xfer->rx.buf)
@@ -1052,8 +1225,12 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 
 		xfer->tx.buf = xfer->rx.buf;
 		init_completion(&xfer->done);
+
+		/* Add initialized xfer to the free list */
+		hlist_add_head(&xfer->node, &info->free_xfers);
 	}
 
+	atomic_set(&info->last_token, -1);
 	spin_lock_init(&info->xfer_lock);
 
 	return 0;
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 05/16] firmware: arm_scmi: Introduce delegated xfers support
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Introduce optional support for delegated xfers allocation.

An SCMI transport can optionally declare to support delegated xfers and
then use a few helper functions exposed by the core SCMI transport layer to
query the core for existing in-flight transfers matching a provided message
header or alternatively and transparently obtain a brand new xfer to handle
a freshly received notification message.
In both cases the obtained xfer is uniquely mapped into a specific xfer
through the means of the message header acting as key.

In this way such a transport can properly store its own transport specific
payload into the xfer uniquely associated to the message header before
even calling into the core scmi_rx_callback() in the usual way, so that
the transport specific message envelope structures can be freed early
and there is no more need to keep track of their status till the core
fully processes the xfer to completion or times out.

The scmi_rx_callbak() does not need to be modified to carry additional
transport-specific ancillary data related to such message envelopes since
an unique natural association is established between the xfer and the
related message header.

Existing transports that do not need anything of the above will continue
to work as before without any change.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |  14 +++
 drivers/firmware/arm_scmi/driver.c | 132 ++++++++++++++++++++++++++++-
 2 files changed, 143 insertions(+), 3 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index e64c5ca9ee7c..0edc04bc434c 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -80,6 +80,7 @@ struct scmi_msg_resp_prot_version {
  * @status: Status of the transfer once it's complete
  * @poll_completion: Indicate if the transfer needs to be polled for
  *	completion or interrupt mode is used
+ * @saved_hdr: A copy of the original msg_hdr
  */
 struct scmi_msg_hdr {
 	u8 id;
@@ -88,6 +89,7 @@ struct scmi_msg_hdr {
 	u16 seq;
 	u32 status;
 	bool poll_completion;
+	u32 saved_hdr;
 };
 
 /**
@@ -154,6 +156,9 @@ struct scmi_msg {
  * @rx: Receive message, the buffer should be pre-allocated to store
  *	message. If request-ACK protocol is used, we can reuse the same
  *	buffer for the rx path as we use for the tx path.
+ * @rx_raw_len: A field which can be optionally used by a specific transport
+ *		to save transport specific message length
+ *		It is not used by the SCMI transport core
  * @done: command message transmit completion event
  * @async_done: pointer to delayed response message received event completion
  * @users: A refcount to track the active users for this xfer
@@ -165,6 +170,7 @@ struct scmi_xfer {
 	struct scmi_msg_hdr hdr;
 	struct scmi_msg tx;
 	struct scmi_msg rx;
+	size_t rx_raw_len;
 	struct completion done;
 	struct completion *async_done;
 	refcount_t users;
@@ -355,6 +361,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
  * @max_msg: Maximum number of messages that can be pending
  *	simultaneously in the system
  * @max_msg_size: Maximum size of data per message that can be handled.
+ * @support_xfers_delegation: A flag to indicate if the described transport
+ *			      will handle delegated xfers, so the core can
+ *			      derive proper related assumptions.
  */
 struct scmi_desc {
 	int (*init)(void);
@@ -363,6 +372,7 @@ struct scmi_desc {
 	int max_rx_timeout_ms;
 	int max_msg;
 	int max_msg_size;
+	bool support_xfers_delegation;
 };
 
 extern const struct scmi_desc scmi_mailbox_desc;
@@ -391,4 +401,8 @@ void scmi_notification_instance_data_set(const struct scmi_handle *handle,
 					 void *priv);
 void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
 
+int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
+			  struct scmi_xfer **xfer);
+void scmi_transfer_release(struct scmi_chan_info *cinfo,
+			   struct scmi_xfer *xfer);
 #endif /* _SCMI_COMMON_H */
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index f0b20ddb24f4..371d3804cd79 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -432,6 +432,124 @@ scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
 	return xfer ?: ERR_PTR(-EINVAL);
 }
 
+/**
+ * scmi_xfer_acquire  -  Helper to lookup and acquire an xfer
+ *
+ * @minfo: Pointer to Tx/Rx Message management info based on channel type
+ * @xfer_id: Token ID to lookup in @pending_xfers
+ *
+ * When a valid xfer is found for the provided @xfer_id, reference counting is
+ * properly updated.
+ *
+ * Return: A valid @xfer on Success or error otherwise.
+ */
+static struct scmi_xfer *
+scmi_xfer_acquire(struct scmi_xfers_info *minfo, u16 xfer_id)
+{
+	unsigned long flags;
+	struct scmi_xfer *xfer;
+
+	spin_lock_irqsave(&minfo->xfer_lock, flags);
+	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
+	if (!IS_ERR(xfer))
+		refcount_inc(&xfer->users);
+	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
+
+	return xfer;
+}
+
+/**
+ * scmi_transfer_acquire  -  Lookup for an existing xfer or freshly allocate a
+ * new one depending on the type of the message
+ *
+ * @cinfo: A reference to the channel descriptor.
+ * @msg_hdr: A pointer to the message header to lookup.
+ * @xfer: A reference to the pre-existent or freshly allocated xfer
+ *	  associated with the provided *msg_hdr.
+ *
+ * This function can be used by transports supporting delegated xfers to obtain
+ * a valid @xfer associated with the provided @msg_hdr param.
+ *
+ * The nature of the resulting @xfer depends on the type of message specified in
+ * @msg_hdr:
+ *  - for responses and delayed responses a pre-existent/pre-allocated in-flight
+ *    xfer descriptor will be returned (properly refcounted)
+ *  - for notifications a brand new xfer will be allocated; in this case the
+ *    provided message header sequence number will also be mangled to match
+ *    the token in the freshly allocated xfer: this is needed to establish a
+ *    link between the picked xfer and the msg_hdr that will be subsequently
+ *    passed back via usual scmi_rx_callback().
+ *
+ * Return: 0 if a valid xfer is returned in @xfer, error otherwise.
+ */
+int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
+			  struct scmi_xfer **xfer)
+{
+	u8 msg_type;
+	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
+
+	if (!xfer || !msg_hdr || !info->desc->support_xfers_delegation)
+		return -EINVAL;
+
+	msg_type = MSG_XTRACT_TYPE(*msg_hdr);
+	switch (msg_type) {
+	case MSG_TYPE_COMMAND:
+	case MSG_TYPE_DELAYED_RESP:
+		/* Grab an existing xfer for xfer_id */
+		*xfer = scmi_xfer_acquire(&info->tx_minfo,
+					  MSG_XTRACT_TOKEN(*msg_hdr));
+		break;
+	case MSG_TYPE_NOTIFICATION:
+		/* Get a brand new RX xfer */
+		*xfer = scmi_xfer_get(cinfo->handle, &info->rx_minfo);
+		if (!IS_ERR(*xfer)) {
+			/* Save original msg_hdr and fix sequence number */
+			(*xfer)->hdr.saved_hdr = *msg_hdr;
+			*msg_hdr &= ~MSG_TOKEN_ID_MASK;
+			*msg_hdr |= FIELD_PREP(MSG_TOKEN_ID_MASK,
+					       (*xfer)->hdr.seq);
+		}
+		break;
+	default:
+		*xfer = ERR_PTR(-EINVAL);
+		break;
+	}
+
+	if (IS_ERR(*xfer)) {
+		dev_err(cinfo->dev,
+			"Failed to acquire a valid xfer for hdr:0x%X\n",
+			*msg_hdr);
+		return PTR_ERR(*xfer);
+	}
+
+	/* Fix xfer->hdr.type with actual msg_hdr carried type */
+	unpack_scmi_header(*msg_hdr, &((*xfer)->hdr));
+
+	return 0;
+}
+
+/**
+ * scmi_transfer_release  - Release an previously acquired xfer
+ *
+ * @cinfo: A reference to the channel descriptor.
+ * @xfer: A reference to the xfer to release.
+ */
+void scmi_transfer_release(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer)
+{
+	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
+	struct scmi_xfers_info *minfo;
+
+	if (!xfer || !info->desc->support_xfers_delegation)
+		return;
+
+	if (xfer->hdr.type == MSG_TYPE_NOTIFICATION)
+		minfo = &info->rx_minfo;
+	else
+		minfo = &info->tx_minfo;
+
+	__scmi_xfer_put(minfo, xfer);
+}
+
 static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 {
 	struct scmi_xfer *xfer;
@@ -441,7 +559,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 	ktime_t ts;
 
 	ts = ktime_get_boottime();
-	xfer = scmi_xfer_get(cinfo->handle, minfo);
+
+	if (!info->desc->support_xfers_delegation)
+		xfer = scmi_xfer_get(cinfo->handle, minfo);
+	else
+		xfer = scmi_xfer_acquire(minfo, MSG_XTRACT_TOKEN(msg_hdr));
 	if (IS_ERR(xfer)) {
 		dev_err(dev, "failed to get free message slot (%ld)\n",
 			PTR_ERR(xfer));
@@ -449,8 +571,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 		return;
 	}
 
-	unpack_scmi_header(msg_hdr, &xfer->hdr);
 	scmi_dump_header_dbg(dev, &xfer->hdr);
+
+	if (!info->desc->support_xfers_delegation)
+		unpack_scmi_header(msg_hdr, &xfer->hdr);
+
 	info->desc->ops->fetch_notification(cinfo, info->desc->max_msg_size,
 					    xfer);
 	scmi_notify(cinfo->handle, xfer->hdr.protocol_id,
@@ -496,7 +621,8 @@ static void scmi_handle_response(struct scmi_chan_info *cinfo,
 			xfer_id);
 		info->desc->ops->clear_channel(cinfo);
 		/* It was unexpected, so nobody will clear the xfer if not us */
-		__scmi_xfer_put(minfo, xfer);
+		if (!info->desc->support_xfers_delegation) //XXX ??? Really
+			__scmi_xfer_put(minfo, xfer);
 		return;
 	}
 
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 05/16] firmware: arm_scmi: Introduce delegated xfers support
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Introduce optional support for delegated xfers allocation.

An SCMI transport can optionally declare to support delegated xfers and
then use a few helper functions exposed by the core SCMI transport layer to
query the core for existing in-flight transfers matching a provided message
header or alternatively and transparently obtain a brand new xfer to handle
a freshly received notification message.
In both cases the obtained xfer is uniquely mapped into a specific xfer
through the means of the message header acting as key.

In this way such a transport can properly store its own transport specific
payload into the xfer uniquely associated to the message header before
even calling into the core scmi_rx_callback() in the usual way, so that
the transport specific message envelope structures can be freed early
and there is no more need to keep track of their status till the core
fully processes the xfer to completion or times out.

The scmi_rx_callbak() does not need to be modified to carry additional
transport-specific ancillary data related to such message envelopes since
an unique natural association is established between the xfer and the
related message header.

Existing transports that do not need anything of the above will continue
to work as before without any change.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |  14 +++
 drivers/firmware/arm_scmi/driver.c | 132 ++++++++++++++++++++++++++++-
 2 files changed, 143 insertions(+), 3 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index e64c5ca9ee7c..0edc04bc434c 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -80,6 +80,7 @@ struct scmi_msg_resp_prot_version {
  * @status: Status of the transfer once it's complete
  * @poll_completion: Indicate if the transfer needs to be polled for
  *	completion or interrupt mode is used
+ * @saved_hdr: A copy of the original msg_hdr
  */
 struct scmi_msg_hdr {
 	u8 id;
@@ -88,6 +89,7 @@ struct scmi_msg_hdr {
 	u16 seq;
 	u32 status;
 	bool poll_completion;
+	u32 saved_hdr;
 };
 
 /**
@@ -154,6 +156,9 @@ struct scmi_msg {
  * @rx: Receive message, the buffer should be pre-allocated to store
  *	message. If request-ACK protocol is used, we can reuse the same
  *	buffer for the rx path as we use for the tx path.
+ * @rx_raw_len: A field which can be optionally used by a specific transport
+ *		to save transport specific message length
+ *		It is not used by the SCMI transport core
  * @done: command message transmit completion event
  * @async_done: pointer to delayed response message received event completion
  * @users: A refcount to track the active users for this xfer
@@ -165,6 +170,7 @@ struct scmi_xfer {
 	struct scmi_msg_hdr hdr;
 	struct scmi_msg tx;
 	struct scmi_msg rx;
+	size_t rx_raw_len;
 	struct completion done;
 	struct completion *async_done;
 	refcount_t users;
@@ -355,6 +361,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
  * @max_msg: Maximum number of messages that can be pending
  *	simultaneously in the system
  * @max_msg_size: Maximum size of data per message that can be handled.
+ * @support_xfers_delegation: A flag to indicate if the described transport
+ *			      will handle delegated xfers, so the core can
+ *			      derive proper related assumptions.
  */
 struct scmi_desc {
 	int (*init)(void);
@@ -363,6 +372,7 @@ struct scmi_desc {
 	int max_rx_timeout_ms;
 	int max_msg;
 	int max_msg_size;
+	bool support_xfers_delegation;
 };
 
 extern const struct scmi_desc scmi_mailbox_desc;
@@ -391,4 +401,8 @@ void scmi_notification_instance_data_set(const struct scmi_handle *handle,
 					 void *priv);
 void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
 
+int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
+			  struct scmi_xfer **xfer);
+void scmi_transfer_release(struct scmi_chan_info *cinfo,
+			   struct scmi_xfer *xfer);
 #endif /* _SCMI_COMMON_H */
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index f0b20ddb24f4..371d3804cd79 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -432,6 +432,124 @@ scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
 	return xfer ?: ERR_PTR(-EINVAL);
 }
 
+/**
+ * scmi_xfer_acquire  -  Helper to lookup and acquire an xfer
+ *
+ * @minfo: Pointer to Tx/Rx Message management info based on channel type
+ * @xfer_id: Token ID to lookup in @pending_xfers
+ *
+ * When a valid xfer is found for the provided @xfer_id, reference counting is
+ * properly updated.
+ *
+ * Return: A valid @xfer on Success or error otherwise.
+ */
+static struct scmi_xfer *
+scmi_xfer_acquire(struct scmi_xfers_info *minfo, u16 xfer_id)
+{
+	unsigned long flags;
+	struct scmi_xfer *xfer;
+
+	spin_lock_irqsave(&minfo->xfer_lock, flags);
+	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
+	if (!IS_ERR(xfer))
+		refcount_inc(&xfer->users);
+	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
+
+	return xfer;
+}
+
+/**
+ * scmi_transfer_acquire  -  Lookup for an existing xfer or freshly allocate a
+ * new one depending on the type of the message
+ *
+ * @cinfo: A reference to the channel descriptor.
+ * @msg_hdr: A pointer to the message header to lookup.
+ * @xfer: A reference to the pre-existent or freshly allocated xfer
+ *	  associated with the provided *msg_hdr.
+ *
+ * This function can be used by transports supporting delegated xfers to obtain
+ * a valid @xfer associated with the provided @msg_hdr param.
+ *
+ * The nature of the resulting @xfer depends on the type of message specified in
+ * @msg_hdr:
+ *  - for responses and delayed responses a pre-existent/pre-allocated in-flight
+ *    xfer descriptor will be returned (properly refcounted)
+ *  - for notifications a brand new xfer will be allocated; in this case the
+ *    provided message header sequence number will also be mangled to match
+ *    the token in the freshly allocated xfer: this is needed to establish a
+ *    link between the picked xfer and the msg_hdr that will be subsequently
+ *    passed back via usual scmi_rx_callback().
+ *
+ * Return: 0 if a valid xfer is returned in @xfer, error otherwise.
+ */
+int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
+			  struct scmi_xfer **xfer)
+{
+	u8 msg_type;
+	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
+
+	if (!xfer || !msg_hdr || !info->desc->support_xfers_delegation)
+		return -EINVAL;
+
+	msg_type = MSG_XTRACT_TYPE(*msg_hdr);
+	switch (msg_type) {
+	case MSG_TYPE_COMMAND:
+	case MSG_TYPE_DELAYED_RESP:
+		/* Grab an existing xfer for xfer_id */
+		*xfer = scmi_xfer_acquire(&info->tx_minfo,
+					  MSG_XTRACT_TOKEN(*msg_hdr));
+		break;
+	case MSG_TYPE_NOTIFICATION:
+		/* Get a brand new RX xfer */
+		*xfer = scmi_xfer_get(cinfo->handle, &info->rx_minfo);
+		if (!IS_ERR(*xfer)) {
+			/* Save original msg_hdr and fix sequence number */
+			(*xfer)->hdr.saved_hdr = *msg_hdr;
+			*msg_hdr &= ~MSG_TOKEN_ID_MASK;
+			*msg_hdr |= FIELD_PREP(MSG_TOKEN_ID_MASK,
+					       (*xfer)->hdr.seq);
+		}
+		break;
+	default:
+		*xfer = ERR_PTR(-EINVAL);
+		break;
+	}
+
+	if (IS_ERR(*xfer)) {
+		dev_err(cinfo->dev,
+			"Failed to acquire a valid xfer for hdr:0x%X\n",
+			*msg_hdr);
+		return PTR_ERR(*xfer);
+	}
+
+	/* Fix xfer->hdr.type with actual msg_hdr carried type */
+	unpack_scmi_header(*msg_hdr, &((*xfer)->hdr));
+
+	return 0;
+}
+
+/**
+ * scmi_transfer_release  - Release an previously acquired xfer
+ *
+ * @cinfo: A reference to the channel descriptor.
+ * @xfer: A reference to the xfer to release.
+ */
+void scmi_transfer_release(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer)
+{
+	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
+	struct scmi_xfers_info *minfo;
+
+	if (!xfer || !info->desc->support_xfers_delegation)
+		return;
+
+	if (xfer->hdr.type == MSG_TYPE_NOTIFICATION)
+		minfo = &info->rx_minfo;
+	else
+		minfo = &info->tx_minfo;
+
+	__scmi_xfer_put(minfo, xfer);
+}
+
 static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 {
 	struct scmi_xfer *xfer;
@@ -441,7 +559,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 	ktime_t ts;
 
 	ts = ktime_get_boottime();
-	xfer = scmi_xfer_get(cinfo->handle, minfo);
+
+	if (!info->desc->support_xfers_delegation)
+		xfer = scmi_xfer_get(cinfo->handle, minfo);
+	else
+		xfer = scmi_xfer_acquire(minfo, MSG_XTRACT_TOKEN(msg_hdr));
 	if (IS_ERR(xfer)) {
 		dev_err(dev, "failed to get free message slot (%ld)\n",
 			PTR_ERR(xfer));
@@ -449,8 +571,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 		return;
 	}
 
-	unpack_scmi_header(msg_hdr, &xfer->hdr);
 	scmi_dump_header_dbg(dev, &xfer->hdr);
+
+	if (!info->desc->support_xfers_delegation)
+		unpack_scmi_header(msg_hdr, &xfer->hdr);
+
 	info->desc->ops->fetch_notification(cinfo, info->desc->max_msg_size,
 					    xfer);
 	scmi_notify(cinfo->handle, xfer->hdr.protocol_id,
@@ -496,7 +621,8 @@ static void scmi_handle_response(struct scmi_chan_info *cinfo,
 			xfer_id);
 		info->desc->ops->clear_channel(cinfo);
 		/* It was unexpected, so nobody will clear the xfer if not us */
-		__scmi_xfer_put(minfo, xfer);
+		if (!info->desc->support_xfers_delegation) //XXX ??? Really
+			__scmi_xfer_put(minfo, xfer);
 		return;
 	}
 
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 06/16] firmware: arm_scmi, smccc, mailbox: Make shmem based transports optional
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Igor Skalkin <igor.skalkin@opensynergy.com>

Upon adding the virtio transport in this patch series, SCMI will also
work without shared memory based transports. Also, the mailbox transport
may not be needed if the smc transport is used.

- Compile shmem.c only if a shmem based transport is available.

- Remove hard dependency of SCMI on mailbox.

[ Peter: Adapted patch for submission to upstream. ]

Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/Kconfig           | 8 +++++++-
 drivers/firmware/arm_scmi/Makefile | 2 +-
 drivers/firmware/arm_scmi/common.h | 2 ++
 drivers/firmware/smccc/Kconfig     | 1 +
 drivers/mailbox/Kconfig            | 1 +
 5 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
index 1db738d5b301..358f895847b5 100644
--- a/drivers/firmware/Kconfig
+++ b/drivers/firmware/Kconfig
@@ -9,7 +9,7 @@ menu "Firmware Drivers"
 config ARM_SCMI_PROTOCOL
 	tristate "ARM System Control and Management Interface (SCMI) Message Protocol"
 	depends on ARM || ARM64 || COMPILE_TEST
-	depends on MAILBOX || HAVE_ARM_SMCCC_DISCOVERY
+	depends on ARM_SCMI_HAVE_SHMEM
 	help
 	  ARM System Control and Management Interface (SCMI) protocol is a
 	  set of operating system-independent software interfaces that are
@@ -27,6 +27,12 @@ config ARM_SCMI_PROTOCOL
 	  This protocol library provides interface for all the client drivers
 	  making use of the features offered by the SCMI.
 
+config ARM_SCMI_HAVE_SHMEM
+	bool
+	help
+	  This declares whether a shared memory based transport for SCMI is
+	  available.
+
 config ARM_SCMI_POWER_DOMAIN
 	tristate "SCMI power domain driver"
 	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
index 6a2ef63306d6..5a2d4c32e0ae 100644
--- a/drivers/firmware/arm_scmi/Makefile
+++ b/drivers/firmware/arm_scmi/Makefile
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 scmi-bus-y = bus.o
 scmi-driver-y = driver.o notify.o
-scmi-transport-y = shmem.o
+scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
 scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
 scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
 scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 0edc04bc434c..4666777019fa 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -375,7 +375,9 @@ struct scmi_desc {
 	bool support_xfers_delegation;
 };
 
+#ifdef CONFIG_MAILBOX
 extern const struct scmi_desc scmi_mailbox_desc;
+#endif
 #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
 extern const struct scmi_desc scmi_smc_desc;
 #endif
diff --git a/drivers/firmware/smccc/Kconfig b/drivers/firmware/smccc/Kconfig
index 15e7466179a6..69c4d6cabf62 100644
--- a/drivers/firmware/smccc/Kconfig
+++ b/drivers/firmware/smccc/Kconfig
@@ -9,6 +9,7 @@ config HAVE_ARM_SMCCC_DISCOVERY
 	bool
 	depends on ARM_PSCI_FW
 	default y
+	select ARM_SCMI_HAVE_SHMEM
 	help
 	 SMCCC v1.0 lacked discoverability and hence PSCI v1.0 was updated
 	 to add SMCCC discovery mechanism though the PSCI firmware
diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
index 68de2c6af727..fc02c38c0739 100644
--- a/drivers/mailbox/Kconfig
+++ b/drivers/mailbox/Kconfig
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 menuconfig MAILBOX
 	bool "Mailbox Hardware Support"
+	select ARM_SCMI_HAVE_SHMEM
 	help
 	  Mailbox is a framework to control hardware communication between
 	  on-chip processors through queued messages and interrupt driven
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 06/16] firmware: arm_scmi, smccc, mailbox: Make shmem based transports optional
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Igor Skalkin <igor.skalkin@opensynergy.com>

Upon adding the virtio transport in this patch series, SCMI will also
work without shared memory based transports. Also, the mailbox transport
may not be needed if the smc transport is used.

- Compile shmem.c only if a shmem based transport is available.

- Remove hard dependency of SCMI on mailbox.

[ Peter: Adapted patch for submission to upstream. ]

Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/Kconfig           | 8 +++++++-
 drivers/firmware/arm_scmi/Makefile | 2 +-
 drivers/firmware/arm_scmi/common.h | 2 ++
 drivers/firmware/smccc/Kconfig     | 1 +
 drivers/mailbox/Kconfig            | 1 +
 5 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
index 1db738d5b301..358f895847b5 100644
--- a/drivers/firmware/Kconfig
+++ b/drivers/firmware/Kconfig
@@ -9,7 +9,7 @@ menu "Firmware Drivers"
 config ARM_SCMI_PROTOCOL
 	tristate "ARM System Control and Management Interface (SCMI) Message Protocol"
 	depends on ARM || ARM64 || COMPILE_TEST
-	depends on MAILBOX || HAVE_ARM_SMCCC_DISCOVERY
+	depends on ARM_SCMI_HAVE_SHMEM
 	help
 	  ARM System Control and Management Interface (SCMI) protocol is a
 	  set of operating system-independent software interfaces that are
@@ -27,6 +27,12 @@ config ARM_SCMI_PROTOCOL
 	  This protocol library provides interface for all the client drivers
 	  making use of the features offered by the SCMI.
 
+config ARM_SCMI_HAVE_SHMEM
+	bool
+	help
+	  This declares whether a shared memory based transport for SCMI is
+	  available.
+
 config ARM_SCMI_POWER_DOMAIN
 	tristate "SCMI power domain driver"
 	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
index 6a2ef63306d6..5a2d4c32e0ae 100644
--- a/drivers/firmware/arm_scmi/Makefile
+++ b/drivers/firmware/arm_scmi/Makefile
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 scmi-bus-y = bus.o
 scmi-driver-y = driver.o notify.o
-scmi-transport-y = shmem.o
+scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
 scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
 scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
 scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 0edc04bc434c..4666777019fa 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -375,7 +375,9 @@ struct scmi_desc {
 	bool support_xfers_delegation;
 };
 
+#ifdef CONFIG_MAILBOX
 extern const struct scmi_desc scmi_mailbox_desc;
+#endif
 #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
 extern const struct scmi_desc scmi_smc_desc;
 #endif
diff --git a/drivers/firmware/smccc/Kconfig b/drivers/firmware/smccc/Kconfig
index 15e7466179a6..69c4d6cabf62 100644
--- a/drivers/firmware/smccc/Kconfig
+++ b/drivers/firmware/smccc/Kconfig
@@ -9,6 +9,7 @@ config HAVE_ARM_SMCCC_DISCOVERY
 	bool
 	depends on ARM_PSCI_FW
 	default y
+	select ARM_SCMI_HAVE_SHMEM
 	help
 	 SMCCC v1.0 lacked discoverability and hence PSCI v1.0 was updated
 	 to add SMCCC discovery mechanism though the PSCI firmware
diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
index 68de2c6af727..fc02c38c0739 100644
--- a/drivers/mailbox/Kconfig
+++ b/drivers/mailbox/Kconfig
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 menuconfig MAILBOX
 	bool "Mailbox Hardware Support"
+	select ARM_SCMI_HAVE_SHMEM
 	help
 	  Mailbox is a framework to control hardware communication between
 	  on-chip processors through queued messages and interrupt driven
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 07/16] firmware: arm_scmi: Add op to override max message #
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Igor Skalkin <igor.skalkin@opensynergy.com>

The number of simultaneously pending messages that the upcoming
scmi-virtio transport can support depends on the virtio device (SCMI
platform) and can differ for each channel. (The scmi-virtio transport
does only have one tx and at most 1 rx channel.)

Add an optional transport op so that scmi-virtio can report the actual
max message # for each channel type. Respect these new limits.

Reflect that the limit in struct scmi_desc is now only a default any
more.

[ Peter: Adapted patch for submission to upstream. ]

Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/arm_scmi/common.h |  9 ++++++--
 drivers/firmware/arm_scmi/driver.c | 34 ++++++++++++++++++++++++------
 2 files changed, 34 insertions(+), 9 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 4666777019fa..253a218fe16a 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -319,6 +319,9 @@ struct scmi_chan_info {
  * @chan_available: Callback to check if channel is available or not
  * @chan_setup: Callback to allocate and setup a channel
  * @chan_free: Callback to free a channel
+ * @get_max_msg: Optional callback to provide max_msg dynamically
+ * 	@return: Maximum number of messages for the channel type (tx or rx)
+ * 		 that can be pending simultaneously in the system
  * @send_message: Callback to send a message
  * @mark_txdone: Callback to mark tx as done
  * @fetch_response: Callback to fetch response
@@ -331,6 +334,7 @@ struct scmi_transport_ops {
 	int (*chan_setup)(struct scmi_chan_info *cinfo, struct device *dev,
 			  bool tx);
 	int (*chan_free)(int id, void *p, void *data);
+	unsigned int (*get_max_msg)(bool tx, struct scmi_chan_info *base_cinfo);
 	int (*send_message)(struct scmi_chan_info *cinfo,
 			    struct scmi_xfer *xfer);
 	void (*mark_txdone)(struct scmi_chan_info *cinfo, int ret);
@@ -358,8 +362,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
  *	  after SCMI core removal.
  * @ops: Pointer to the transport specific ops structure
  * @max_rx_timeout_ms: Timeout for communication with SoC (in Milliseconds)
- * @max_msg: Maximum number of messages that can be pending
- *	simultaneously in the system
+ * @max_msg: Maximum number of messages for a channel type (tx or rx) that can
+ *	be pending simultaneously in the system. May be overridden by the
+ *	get_max_msg op.
  * @max_msg_size: Maximum size of data per message that can be handled.
  * @support_xfers_delegation: A flag to indicate if the described transport
  *			      will handle delegated xfers, so the core can
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 371d3804cd79..fdd711d85498 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -75,6 +75,7 @@ struct scmi_requested_dev {
  *	Index of this bitmap table is also used for message
  *	sequence identifier.
  * @xfer_lock: Protection for message allocation
+ * @max_msg: Maximum number of messages that can be pending
  * @last_token: A counter to use as base to generate for monotonically
  *		increasing tokens.
  * @free_xfers: A free list for available to use xfers. It is initialized with
@@ -86,6 +87,7 @@ struct scmi_requested_dev {
 struct scmi_xfers_info {
 	unsigned long *xfer_alloc_table;
 	spinlock_t xfer_lock;
+	int max_msg;
 	atomic_t last_token;
 	struct hlist_head free_xfers;
 	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
@@ -1310,18 +1312,25 @@ int scmi_handle_put(const struct scmi_handle *handle)
 }
 
 static int __scmi_xfer_info_init(struct scmi_info *sinfo,
-				 struct scmi_xfers_info *info)
+				 struct scmi_xfers_info *info,
+				 bool tx,
+				 struct scmi_chan_info *base_cinfo)
 {
 	int i;
 	struct scmi_xfer *xfer;
 	struct device *dev = sinfo->dev;
 	const struct scmi_desc *desc = sinfo->desc;
 
+	info->max_msg = desc->max_msg;
+
+	if (desc->ops->get_max_msg)
+		info->max_msg =	desc->ops->get_max_msg(tx, base_cinfo);
+
 	/* Pre-allocated messages, no more than what hdr.seq can support */
-	if (WARN_ON(!desc->max_msg || desc->max_msg > MSG_TOKEN_MAX)) {
+	if (WARN_ON(!info->max_msg || info->max_msg > MSG_TOKEN_MAX)) {
 		dev_err(dev,
 			"Invalid max_msg %d. Maximum messages supported %ld.\n",
-			desc->max_msg, MSG_TOKEN_MAX);
+			info->max_msg, MSG_TOKEN_MAX);
 		return -EINVAL;
 	}
 
@@ -1339,7 +1348,7 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 	 * attach all of them to the free list
 	 */
 	INIT_HLIST_HEAD(&info->free_xfers);
-	for (i = 0; i < desc->max_msg; i++) {
+	for (i = 0; i < info->max_msg; i++) {
 		xfer = devm_kzalloc(dev, sizeof(*xfer), GFP_KERNEL);
 		if (!xfer)
 			return -ENOMEM;
@@ -1364,10 +1373,21 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 
 static int scmi_xfer_info_init(struct scmi_info *sinfo)
 {
-	int ret = __scmi_xfer_info_init(sinfo, &sinfo->tx_minfo);
+	int ret;
+	struct scmi_chan_info *base_tx_cinfo;
+	struct scmi_chan_info *base_rx_cinfo;
+
+	base_tx_cinfo = idr_find(&sinfo->tx_idr, SCMI_PROTOCOL_BASE);
+	if (unlikely(!base_tx_cinfo))
+		return -EINVAL;
+
+	ret = __scmi_xfer_info_init(sinfo, &sinfo->tx_minfo, true,
+				    base_tx_cinfo);
 
-	if (!ret && idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE))
-		ret = __scmi_xfer_info_init(sinfo, &sinfo->rx_minfo);
+	base_rx_cinfo = idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE);
+	if (!ret && base_rx_cinfo)
+		ret = __scmi_xfer_info_init(sinfo, &sinfo->rx_minfo, false,
+					    base_rx_cinfo);
 
 	return ret;
 }
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 07/16] firmware: arm_scmi: Add op to override max message #
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Igor Skalkin <igor.skalkin@opensynergy.com>

The number of simultaneously pending messages that the upcoming
scmi-virtio transport can support depends on the virtio device (SCMI
platform) and can differ for each channel. (The scmi-virtio transport
does only have one tx and at most 1 rx channel.)

Add an optional transport op so that scmi-virtio can report the actual
max message # for each channel type. Respect these new limits.

Reflect that the limit in struct scmi_desc is now only a default any
more.

[ Peter: Adapted patch for submission to upstream. ]

Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/arm_scmi/common.h |  9 ++++++--
 drivers/firmware/arm_scmi/driver.c | 34 ++++++++++++++++++++++++------
 2 files changed, 34 insertions(+), 9 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 4666777019fa..253a218fe16a 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -319,6 +319,9 @@ struct scmi_chan_info {
  * @chan_available: Callback to check if channel is available or not
  * @chan_setup: Callback to allocate and setup a channel
  * @chan_free: Callback to free a channel
+ * @get_max_msg: Optional callback to provide max_msg dynamically
+ * 	@return: Maximum number of messages for the channel type (tx or rx)
+ * 		 that can be pending simultaneously in the system
  * @send_message: Callback to send a message
  * @mark_txdone: Callback to mark tx as done
  * @fetch_response: Callback to fetch response
@@ -331,6 +334,7 @@ struct scmi_transport_ops {
 	int (*chan_setup)(struct scmi_chan_info *cinfo, struct device *dev,
 			  bool tx);
 	int (*chan_free)(int id, void *p, void *data);
+	unsigned int (*get_max_msg)(bool tx, struct scmi_chan_info *base_cinfo);
 	int (*send_message)(struct scmi_chan_info *cinfo,
 			    struct scmi_xfer *xfer);
 	void (*mark_txdone)(struct scmi_chan_info *cinfo, int ret);
@@ -358,8 +362,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
  *	  after SCMI core removal.
  * @ops: Pointer to the transport specific ops structure
  * @max_rx_timeout_ms: Timeout for communication with SoC (in Milliseconds)
- * @max_msg: Maximum number of messages that can be pending
- *	simultaneously in the system
+ * @max_msg: Maximum number of messages for a channel type (tx or rx) that can
+ *	be pending simultaneously in the system. May be overridden by the
+ *	get_max_msg op.
  * @max_msg_size: Maximum size of data per message that can be handled.
  * @support_xfers_delegation: A flag to indicate if the described transport
  *			      will handle delegated xfers, so the core can
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 371d3804cd79..fdd711d85498 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -75,6 +75,7 @@ struct scmi_requested_dev {
  *	Index of this bitmap table is also used for message
  *	sequence identifier.
  * @xfer_lock: Protection for message allocation
+ * @max_msg: Maximum number of messages that can be pending
  * @last_token: A counter to use as base to generate for monotonically
  *		increasing tokens.
  * @free_xfers: A free list for available to use xfers. It is initialized with
@@ -86,6 +87,7 @@ struct scmi_requested_dev {
 struct scmi_xfers_info {
 	unsigned long *xfer_alloc_table;
 	spinlock_t xfer_lock;
+	int max_msg;
 	atomic_t last_token;
 	struct hlist_head free_xfers;
 	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
@@ -1310,18 +1312,25 @@ int scmi_handle_put(const struct scmi_handle *handle)
 }
 
 static int __scmi_xfer_info_init(struct scmi_info *sinfo,
-				 struct scmi_xfers_info *info)
+				 struct scmi_xfers_info *info,
+				 bool tx,
+				 struct scmi_chan_info *base_cinfo)
 {
 	int i;
 	struct scmi_xfer *xfer;
 	struct device *dev = sinfo->dev;
 	const struct scmi_desc *desc = sinfo->desc;
 
+	info->max_msg = desc->max_msg;
+
+	if (desc->ops->get_max_msg)
+		info->max_msg =	desc->ops->get_max_msg(tx, base_cinfo);
+
 	/* Pre-allocated messages, no more than what hdr.seq can support */
-	if (WARN_ON(!desc->max_msg || desc->max_msg > MSG_TOKEN_MAX)) {
+	if (WARN_ON(!info->max_msg || info->max_msg > MSG_TOKEN_MAX)) {
 		dev_err(dev,
 			"Invalid max_msg %d. Maximum messages supported %ld.\n",
-			desc->max_msg, MSG_TOKEN_MAX);
+			info->max_msg, MSG_TOKEN_MAX);
 		return -EINVAL;
 	}
 
@@ -1339,7 +1348,7 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 	 * attach all of them to the free list
 	 */
 	INIT_HLIST_HEAD(&info->free_xfers);
-	for (i = 0; i < desc->max_msg; i++) {
+	for (i = 0; i < info->max_msg; i++) {
 		xfer = devm_kzalloc(dev, sizeof(*xfer), GFP_KERNEL);
 		if (!xfer)
 			return -ENOMEM;
@@ -1364,10 +1373,21 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 
 static int scmi_xfer_info_init(struct scmi_info *sinfo)
 {
-	int ret = __scmi_xfer_info_init(sinfo, &sinfo->tx_minfo);
+	int ret;
+	struct scmi_chan_info *base_tx_cinfo;
+	struct scmi_chan_info *base_rx_cinfo;
+
+	base_tx_cinfo = idr_find(&sinfo->tx_idr, SCMI_PROTOCOL_BASE);
+	if (unlikely(!base_tx_cinfo))
+		return -EINVAL;
+
+	ret = __scmi_xfer_info_init(sinfo, &sinfo->tx_minfo, true,
+				    base_tx_cinfo);
 
-	if (!ret && idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE))
-		ret = __scmi_xfer_info_init(sinfo, &sinfo->rx_minfo);
+	base_rx_cinfo = idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE);
+	if (!ret && base_rx_cinfo)
+		ret = __scmi_xfer_info_init(sinfo, &sinfo->rx_minfo, false,
+					    base_rx_cinfo);
 
 	return ret;
 }
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 08/16] [RFC][REWORK] firmware: arm_scmi: Add op to override max message #
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Refactoring core scmi_xfer_info_init routines to account for transport
specific maximum number of pending messages.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |  6 ++--
 drivers/firmware/arm_scmi/driver.c | 52 ++++++++++++++++++------------
 2 files changed, 35 insertions(+), 23 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 253a218fe16a..6b8d388fcf74 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -320,8 +320,8 @@ struct scmi_chan_info {
  * @chan_setup: Callback to allocate and setup a channel
  * @chan_free: Callback to free a channel
  * @get_max_msg: Optional callback to provide max_msg dynamically
- * 	@return: Maximum number of messages for the channel type (tx or rx)
- * 		 that can be pending simultaneously in the system
+ *		 Returns the maximum number of messages for the channel type
+ *		 (tx or rx) that can be pending simultaneously in the system
  * @send_message: Callback to send a message
  * @mark_txdone: Callback to mark tx as done
  * @fetch_response: Callback to fetch response
@@ -334,7 +334,7 @@ struct scmi_transport_ops {
 	int (*chan_setup)(struct scmi_chan_info *cinfo, struct device *dev,
 			  bool tx);
 	int (*chan_free)(int id, void *p, void *data);
-	unsigned int (*get_max_msg)(bool tx, struct scmi_chan_info *base_cinfo);
+	unsigned int (*get_max_msg)(struct scmi_chan_info *base_cinfo);
 	int (*send_message)(struct scmi_chan_info *cinfo,
 			    struct scmi_xfer *xfer);
 	void (*mark_txdone)(struct scmi_chan_info *cinfo, int ret);
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index fdd711d85498..041a9e008cbf 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1312,20 +1312,13 @@ int scmi_handle_put(const struct scmi_handle *handle)
 }
 
 static int __scmi_xfer_info_init(struct scmi_info *sinfo,
-				 struct scmi_xfers_info *info,
-				 bool tx,
-				 struct scmi_chan_info *base_cinfo)
+				 struct scmi_xfers_info *info)
 {
 	int i;
 	struct scmi_xfer *xfer;
 	struct device *dev = sinfo->dev;
 	const struct scmi_desc *desc = sinfo->desc;
 
-	info->max_msg = desc->max_msg;
-
-	if (desc->ops->get_max_msg)
-		info->max_msg =	desc->ops->get_max_msg(tx, base_cinfo);
-
 	/* Pre-allocated messages, no more than what hdr.seq can support */
 	if (WARN_ON(!info->max_msg || info->max_msg > MSG_TOKEN_MAX)) {
 		dev_err(dev,
@@ -1371,23 +1364,42 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 	return 0;
 }
 
+static int scmi_channels_max_msg_configure(struct scmi_info *sinfo)
+{
+	const struct scmi_desc *desc = sinfo->desc;
+
+	if (!desc->ops->get_max_msg) {
+		sinfo->tx_minfo.max_msg = desc->max_msg;
+		sinfo->rx_minfo.max_msg = desc->max_msg;
+	} else {
+		struct scmi_chan_info *base_cinfo;
+
+		base_cinfo = idr_find(&sinfo->tx_idr, SCMI_PROTOCOL_BASE);
+		if (!base_cinfo)
+			return -EINVAL;
+
+		sinfo->tx_minfo.max_msg = desc->ops->get_max_msg(base_cinfo);
+
+		base_cinfo = idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE);
+		if (base_cinfo)
+			sinfo->rx_minfo.max_msg =
+				desc->ops->get_max_msg(base_cinfo);
+	}
+
+	return 0;
+}
+
 static int scmi_xfer_info_init(struct scmi_info *sinfo)
 {
 	int ret;
-	struct scmi_chan_info *base_tx_cinfo;
-	struct scmi_chan_info *base_rx_cinfo;
 
-	base_tx_cinfo = idr_find(&sinfo->tx_idr, SCMI_PROTOCOL_BASE);
-	if (unlikely(!base_tx_cinfo))
-		return -EINVAL;
-
-	ret = __scmi_xfer_info_init(sinfo, &sinfo->tx_minfo, true,
-				    base_tx_cinfo);
+	ret = scmi_channels_max_msg_configure(sinfo);
+	if (ret)
+		return ret;
 
-	base_rx_cinfo = idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE);
-	if (!ret && base_rx_cinfo)
-		ret = __scmi_xfer_info_init(sinfo, &sinfo->rx_minfo, false,
-					    base_rx_cinfo);
+	ret = __scmi_xfer_info_init(sinfo, &sinfo->tx_minfo);
+	if (!ret && idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE))
+		ret = __scmi_xfer_info_init(sinfo, &sinfo->rx_minfo);
 
 	return ret;
 }
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 08/16] [RFC][REWORK] firmware: arm_scmi: Add op to override max message #
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Refactoring core scmi_xfer_info_init routines to account for transport
specific maximum number of pending messages.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |  6 ++--
 drivers/firmware/arm_scmi/driver.c | 52 ++++++++++++++++++------------
 2 files changed, 35 insertions(+), 23 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 253a218fe16a..6b8d388fcf74 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -320,8 +320,8 @@ struct scmi_chan_info {
  * @chan_setup: Callback to allocate and setup a channel
  * @chan_free: Callback to free a channel
  * @get_max_msg: Optional callback to provide max_msg dynamically
- * 	@return: Maximum number of messages for the channel type (tx or rx)
- * 		 that can be pending simultaneously in the system
+ *		 Returns the maximum number of messages for the channel type
+ *		 (tx or rx) that can be pending simultaneously in the system
  * @send_message: Callback to send a message
  * @mark_txdone: Callback to mark tx as done
  * @fetch_response: Callback to fetch response
@@ -334,7 +334,7 @@ struct scmi_transport_ops {
 	int (*chan_setup)(struct scmi_chan_info *cinfo, struct device *dev,
 			  bool tx);
 	int (*chan_free)(int id, void *p, void *data);
-	unsigned int (*get_max_msg)(bool tx, struct scmi_chan_info *base_cinfo);
+	unsigned int (*get_max_msg)(struct scmi_chan_info *base_cinfo);
 	int (*send_message)(struct scmi_chan_info *cinfo,
 			    struct scmi_xfer *xfer);
 	void (*mark_txdone)(struct scmi_chan_info *cinfo, int ret);
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index fdd711d85498..041a9e008cbf 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1312,20 +1312,13 @@ int scmi_handle_put(const struct scmi_handle *handle)
 }
 
 static int __scmi_xfer_info_init(struct scmi_info *sinfo,
-				 struct scmi_xfers_info *info,
-				 bool tx,
-				 struct scmi_chan_info *base_cinfo)
+				 struct scmi_xfers_info *info)
 {
 	int i;
 	struct scmi_xfer *xfer;
 	struct device *dev = sinfo->dev;
 	const struct scmi_desc *desc = sinfo->desc;
 
-	info->max_msg = desc->max_msg;
-
-	if (desc->ops->get_max_msg)
-		info->max_msg =	desc->ops->get_max_msg(tx, base_cinfo);
-
 	/* Pre-allocated messages, no more than what hdr.seq can support */
 	if (WARN_ON(!info->max_msg || info->max_msg > MSG_TOKEN_MAX)) {
 		dev_err(dev,
@@ -1371,23 +1364,42 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
 	return 0;
 }
 
+static int scmi_channels_max_msg_configure(struct scmi_info *sinfo)
+{
+	const struct scmi_desc *desc = sinfo->desc;
+
+	if (!desc->ops->get_max_msg) {
+		sinfo->tx_minfo.max_msg = desc->max_msg;
+		sinfo->rx_minfo.max_msg = desc->max_msg;
+	} else {
+		struct scmi_chan_info *base_cinfo;
+
+		base_cinfo = idr_find(&sinfo->tx_idr, SCMI_PROTOCOL_BASE);
+		if (!base_cinfo)
+			return -EINVAL;
+
+		sinfo->tx_minfo.max_msg = desc->ops->get_max_msg(base_cinfo);
+
+		base_cinfo = idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE);
+		if (base_cinfo)
+			sinfo->rx_minfo.max_msg =
+				desc->ops->get_max_msg(base_cinfo);
+	}
+
+	return 0;
+}
+
 static int scmi_xfer_info_init(struct scmi_info *sinfo)
 {
 	int ret;
-	struct scmi_chan_info *base_tx_cinfo;
-	struct scmi_chan_info *base_rx_cinfo;
 
-	base_tx_cinfo = idr_find(&sinfo->tx_idr, SCMI_PROTOCOL_BASE);
-	if (unlikely(!base_tx_cinfo))
-		return -EINVAL;
-
-	ret = __scmi_xfer_info_init(sinfo, &sinfo->tx_minfo, true,
-				    base_tx_cinfo);
+	ret = scmi_channels_max_msg_configure(sinfo);
+	if (ret)
+		return ret;
 
-	base_rx_cinfo = idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE);
-	if (!ret && base_rx_cinfo)
-		ret = __scmi_xfer_info_init(sinfo, &sinfo->rx_minfo, false,
-					    base_rx_cinfo);
+	ret = __scmi_xfer_info_init(sinfo, &sinfo->tx_minfo);
+	if (!ret && idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE))
+		ret = __scmi_xfer_info_init(sinfo, &sinfo->rx_minfo);
 
 	return ret;
 }
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 09/16] firmware: arm_scmi: Add optional link_supplier() transport op
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Peter Hilber <peter.hilber@opensynergy.com>

For the scmi-virtio transport, it might not be possible to refer to the
proper virtio device at device tree build time. Therefore, add an op
which will allow scmi-virtio to dynamically link to the proper virtio
device during probe.

Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/arm_scmi/common.h | 2 ++
 drivers/firmware/arm_scmi/driver.c | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 6b8d388fcf74..a31194b3000a 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -316,6 +316,7 @@ struct scmi_chan_info {
 /**
  * struct scmi_transport_ops - Structure representing a SCMI transport ops
  *
+ * @link_supplier: Optional callback to add link to a supplier device
  * @chan_available: Callback to check if channel is available or not
  * @chan_setup: Callback to allocate and setup a channel
  * @chan_free: Callback to free a channel
@@ -330,6 +331,7 @@ struct scmi_chan_info {
  * @poll_done: Callback to poll transfer status
  */
 struct scmi_transport_ops {
+	int (*link_supplier)(struct device *dev);
 	bool (*chan_available)(struct device *dev, int idx);
 	int (*chan_setup)(struct scmi_chan_info *cinfo, struct device *dev,
 			  bool tx);
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 041a9e008cbf..69fa97250d98 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1757,6 +1757,12 @@ static int scmi_probe(struct platform_device *pdev)
 	handle->devm_protocol_get = scmi_devm_protocol_get;
 	handle->devm_protocol_put = scmi_devm_protocol_put;
 
+	if (desc->ops->link_supplier) {
+		ret = desc->ops->link_supplier(dev);
+		if (ret)
+			return ret;
+	}
+
 	ret = scmi_txrx_setup(info, dev, SCMI_PROTOCOL_BASE);
 	if (ret)
 		return ret;
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 09/16] firmware: arm_scmi: Add optional link_supplier() transport op
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Peter Hilber <peter.hilber@opensynergy.com>

For the scmi-virtio transport, it might not be possible to refer to the
proper virtio device at device tree build time. Therefore, add an op
which will allow scmi-virtio to dynamically link to the proper virtio
device during probe.

Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/arm_scmi/common.h | 2 ++
 drivers/firmware/arm_scmi/driver.c | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 6b8d388fcf74..a31194b3000a 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -316,6 +316,7 @@ struct scmi_chan_info {
 /**
  * struct scmi_transport_ops - Structure representing a SCMI transport ops
  *
+ * @link_supplier: Optional callback to add link to a supplier device
  * @chan_available: Callback to check if channel is available or not
  * @chan_setup: Callback to allocate and setup a channel
  * @chan_free: Callback to free a channel
@@ -330,6 +331,7 @@ struct scmi_chan_info {
  * @poll_done: Callback to poll transfer status
  */
 struct scmi_transport_ops {
+	int (*link_supplier)(struct device *dev);
 	bool (*chan_available)(struct device *dev, int idx);
 	int (*chan_setup)(struct scmi_chan_info *cinfo, struct device *dev,
 			  bool tx);
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 041a9e008cbf..69fa97250d98 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1757,6 +1757,12 @@ static int scmi_probe(struct platform_device *pdev)
 	handle->devm_protocol_get = scmi_devm_protocol_get;
 	handle->devm_protocol_put = scmi_devm_protocol_put;
 
+	if (desc->ops->link_supplier) {
+		ret = desc->ops->link_supplier(dev);
+		if (ret)
+			return ret;
+	}
+
 	ret = scmi_txrx_setup(info, dev, SCMI_PROTOCOL_BASE);
 	if (ret)
 		return ret;
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 10/16] firmware: arm_scmi: Add per-device transport private info
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Peter Hilber <peter.hilber@opensynergy.com>

The scmi-virtio transport will link a supplier device to the arm-scmi
device in the link_supplier() op. The transport should then save a
pointer to the linked device.

To enable this, add a transport private info to the scmi_info. (The
scmi_info is already reachable through the arm-scmi device driver_data.)

Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/arm_scmi/common.h |  2 ++
 drivers/firmware/arm_scmi/driver.c | 35 ++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index a31194b3000a..7219970fc5eb 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -389,6 +389,8 @@ extern const struct scmi_desc scmi_mailbox_desc;
 extern const struct scmi_desc scmi_smc_desc;
 #endif
 
+int scmi_set_transport_info(struct device *dev, void *transport_info);
+void *scmi_get_transport_info(struct device *dev);
 void scmi_rx_callback(struct scmi_chan_info *cinfo, u32 msg_hdr);
 void scmi_free_channel(struct scmi_chan_info *cinfo, struct idr *idr, int id);
 
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 69fa97250d98..bb2de15554a9 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -138,6 +138,7 @@ struct scmi_protocol_instance {
  * @active_protocols: IDR storing device_nodes for protocols actually defined
  *		      in the DT and confirmed as implemented by fw.
  * @notify_priv: Pointer to private data structure specific to notifications.
+ * @transport_info: Transport private info
  * @node: List head
  * @users: Number of users of this instance
  */
@@ -156,6 +157,7 @@ struct scmi_info {
 	u8 *protocols_imp;
 	struct idr active_protocols;
 	void *notify_priv;
+	void *transport_info;
 	struct list_head node;
 	int users;
 };
@@ -679,6 +681,39 @@ void scmi_rx_callback(struct scmi_chan_info *cinfo, u32 msg_hdr)
 	}
 }
 
+/**
+ * scmi_set_transport_info() - Set transport private info
+ *
+ * @dev: SCMI instance device
+ * @transport_info: transport private info
+ *
+ * Return: 0 on success, otherwise error.
+ */
+int scmi_set_transport_info(struct device *dev, void *transport_info)
+{
+	struct scmi_info *info = dev_get_drvdata(dev);
+
+	if (!info)
+		return -EBADR;
+
+	info->transport_info = transport_info;
+	return 0;
+}
+
+/**
+ * scmi_get_transport_info() - Get transport private info
+ *
+ * @dev: SCMI instance device
+ *
+ * Return: transport private info on success, otherwise NULL.
+ */
+void *scmi_get_transport_info(struct device *dev)
+{
+	struct scmi_info *info = dev_get_drvdata(dev);
+
+	return info ? info->transport_info : NULL;
+}
+
 /**
  * xfer_put() - Release a transmit message
  *
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 10/16] firmware: arm_scmi: Add per-device transport private info
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Peter Hilber <peter.hilber@opensynergy.com>

The scmi-virtio transport will link a supplier device to the arm-scmi
device in the link_supplier() op. The transport should then save a
pointer to the linked device.

To enable this, add a transport private info to the scmi_info. (The
scmi_info is already reachable through the arm-scmi device driver_data.)

Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/arm_scmi/common.h |  2 ++
 drivers/firmware/arm_scmi/driver.c | 35 ++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index a31194b3000a..7219970fc5eb 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -389,6 +389,8 @@ extern const struct scmi_desc scmi_mailbox_desc;
 extern const struct scmi_desc scmi_smc_desc;
 #endif
 
+int scmi_set_transport_info(struct device *dev, void *transport_info);
+void *scmi_get_transport_info(struct device *dev);
 void scmi_rx_callback(struct scmi_chan_info *cinfo, u32 msg_hdr);
 void scmi_free_channel(struct scmi_chan_info *cinfo, struct idr *idr, int id);
 
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 69fa97250d98..bb2de15554a9 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -138,6 +138,7 @@ struct scmi_protocol_instance {
  * @active_protocols: IDR storing device_nodes for protocols actually defined
  *		      in the DT and confirmed as implemented by fw.
  * @notify_priv: Pointer to private data structure specific to notifications.
+ * @transport_info: Transport private info
  * @node: List head
  * @users: Number of users of this instance
  */
@@ -156,6 +157,7 @@ struct scmi_info {
 	u8 *protocols_imp;
 	struct idr active_protocols;
 	void *notify_priv;
+	void *transport_info;
 	struct list_head node;
 	int users;
 };
@@ -679,6 +681,39 @@ void scmi_rx_callback(struct scmi_chan_info *cinfo, u32 msg_hdr)
 	}
 }
 
+/**
+ * scmi_set_transport_info() - Set transport private info
+ *
+ * @dev: SCMI instance device
+ * @transport_info: transport private info
+ *
+ * Return: 0 on success, otherwise error.
+ */
+int scmi_set_transport_info(struct device *dev, void *transport_info)
+{
+	struct scmi_info *info = dev_get_drvdata(dev);
+
+	if (!info)
+		return -EBADR;
+
+	info->transport_info = transport_info;
+	return 0;
+}
+
+/**
+ * scmi_get_transport_info() - Get transport private info
+ *
+ * @dev: SCMI instance device
+ *
+ * Return: transport private info on success, otherwise NULL.
+ */
+void *scmi_get_transport_info(struct device *dev)
+{
+	struct scmi_info *info = dev_get_drvdata(dev);
+
+	return info ? info->transport_info : NULL;
+}
+
 /**
  * xfer_put() - Release a transmit message
  *
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 11/16] firmware: arm_scmi: Add is_scmi_protocol_device()
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Peter Hilber <peter.hilber@opensynergy.com>

The scmi-virtio transport driver will need to distinguish SCMI protocol
devices from the SCMI instance device in the chan_setup() and
chan_free() ops. Add this internal helper to be able to distinguish the
two.

Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/arm_scmi/bus.c    | 5 +++++
 drivers/firmware/arm_scmi/common.h | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c
index 784cf0027da3..06148e972d1a 100644
--- a/drivers/firmware/arm_scmi/bus.c
+++ b/drivers/firmware/arm_scmi/bus.c
@@ -134,6 +134,11 @@ static struct bus_type scmi_bus_type = {
 	.remove = scmi_dev_remove,
 };
 
+bool is_scmi_protocol_device(struct device *dev)
+{
+	return dev->bus == &scmi_bus_type;
+}
+
 int scmi_driver_register(struct scmi_driver *driver, struct module *owner,
 			 const char *mod_name)
 {
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 7219970fc5eb..b783ae058c8a 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -264,6 +264,8 @@ struct scmi_protocol {
 	const struct scmi_protocol_events	*events;
 };
 
+bool is_scmi_protocol_device(struct device *dev);
+
 int __init scmi_bus_init(void);
 void __exit scmi_bus_exit(void);
 
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 11/16] firmware: arm_scmi: Add is_scmi_protocol_device()
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Peter Hilber <peter.hilber@opensynergy.com>

The scmi-virtio transport driver will need to distinguish SCMI protocol
devices from the SCMI instance device in the chan_setup() and
chan_free() ops. Add this internal helper to be able to distinguish the
two.

Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/arm_scmi/bus.c    | 5 +++++
 drivers/firmware/arm_scmi/common.h | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c
index 784cf0027da3..06148e972d1a 100644
--- a/drivers/firmware/arm_scmi/bus.c
+++ b/drivers/firmware/arm_scmi/bus.c
@@ -134,6 +134,11 @@ static struct bus_type scmi_bus_type = {
 	.remove = scmi_dev_remove,
 };
 
+bool is_scmi_protocol_device(struct device *dev)
+{
+	return dev->bus == &scmi_bus_type;
+}
+
 int scmi_driver_register(struct scmi_driver *driver, struct module *owner,
 			 const char *mod_name)
 {
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 7219970fc5eb..b783ae058c8a 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -264,6 +264,8 @@ struct scmi_protocol {
 	const struct scmi_protocol_events	*events;
 };
 
+bool is_scmi_protocol_device(struct device *dev);
+
 int __init scmi_bus_init(void);
 void __exit scmi_bus_exit(void);
 
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 12/16] firmware: arm_scmi: Add message passing abstractions for transports
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Peter Hilber <peter.hilber@opensynergy.com>

Add abstractions for future transports using message passing, such as
virtio. Derive the abstractions from the shared memory abstractions.

Abstract the transport SDU through the opaque struct scmi_msg_payld.
Also enable the transport to determine all other required information
about the transport SDU.

Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/Kconfig           |   8 +-
 drivers/firmware/arm_scmi/Makefile |   1 +
 drivers/firmware/arm_scmi/common.h |  15 ++++
 drivers/firmware/arm_scmi/msg.c    | 113 +++++++++++++++++++++++++++++
 4 files changed, 136 insertions(+), 1 deletion(-)
 create mode 100644 drivers/firmware/arm_scmi/msg.c

diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
index 358f895847b5..e74d25065d07 100644
--- a/drivers/firmware/Kconfig
+++ b/drivers/firmware/Kconfig
@@ -9,7 +9,7 @@ menu "Firmware Drivers"
 config ARM_SCMI_PROTOCOL
 	tristate "ARM System Control and Management Interface (SCMI) Message Protocol"
 	depends on ARM || ARM64 || COMPILE_TEST
-	depends on ARM_SCMI_HAVE_SHMEM
+	depends on ARM_SCMI_HAVE_SHMEM || ARM_SCMI_HAVE_MSG
 	help
 	  ARM System Control and Management Interface (SCMI) protocol is a
 	  set of operating system-independent software interfaces that are
@@ -33,6 +33,12 @@ config ARM_SCMI_HAVE_SHMEM
 	  This declares whether a shared memory based transport for SCMI is
 	  available.
 
+config ARM_SCMI_HAVE_MSG
+	bool
+	help
+	  This declares whether a message passing based transport for SCMI is
+	  available.
+
 config ARM_SCMI_POWER_DOMAIN
 	tristate "SCMI power domain driver"
 	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
index 5a2d4c32e0ae..f6b4acb8abdb 100644
--- a/drivers/firmware/arm_scmi/Makefile
+++ b/drivers/firmware/arm_scmi/Makefile
@@ -4,6 +4,7 @@ scmi-driver-y = driver.o notify.o
 scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
 scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
 scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
+scmi-transport-$(CONFIG_ARM_SCMI_HAVE_MSG) += msg.o
 scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
 scmi-module-objs := $(scmi-bus-y) $(scmi-driver-y) $(scmi-protocols-y) \
 		    $(scmi-transport-y)
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index b783ae058c8a..fa4075336580 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -410,6 +410,21 @@ void shmem_clear_channel(struct scmi_shared_mem __iomem *shmem);
 bool shmem_poll_done(struct scmi_shared_mem __iomem *shmem,
 		     struct scmi_xfer *xfer);
 
+/* declarations for message passing transports */
+struct scmi_msg_payld;
+
+/** Maximum overhead of message w.r.t. struct scmi_desc.max_msg_size */
+#define SCMI_MSG_MAX_PROT_OVERHEAD (2 * sizeof(__le32))
+
+size_t msg_response_size(struct scmi_xfer *xfer);
+size_t msg_command_size(struct scmi_xfer *xfer);
+void msg_tx_prepare(struct scmi_msg_payld *msg, struct scmi_xfer *xfer);
+u32 msg_read_header(struct scmi_msg_payld *msg);
+void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
+			struct scmi_xfer *xfer);
+void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
+			    size_t max_len, struct scmi_xfer *xfer);
+
 void scmi_notification_instance_data_set(const struct scmi_handle *handle,
 					 void *priv);
 void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
diff --git a/drivers/firmware/arm_scmi/msg.c b/drivers/firmware/arm_scmi/msg.c
new file mode 100644
index 000000000000..8a2d3303d281
--- /dev/null
+++ b/drivers/firmware/arm_scmi/msg.c
@@ -0,0 +1,113 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * For transports using message passing.
+ *
+ * Derived from shm.c.
+ *
+ * Copyright (C) 2019 ARM Ltd.
+ * Copyright (C) 2020 OpenSynergy GmbH
+ */
+
+#include <linux/io.h>
+#include <linux/processor.h>
+#include <linux/types.h>
+
+#include "common.h"
+
+/*
+ * struct scmi_msg_payld - Transport SDU layout
+ *
+ * The SCMI specification requires all parameters, message headers, return
+ * arguments or any protocol data to be expressed in little endian format only.
+ */
+struct scmi_msg_payld {
+	__le32 msg_header;
+	__le32 msg_payload[];
+};
+
+/**
+ * msg_command_size() - Actual size of transport SDU for command.
+ *
+ * @xfer: message which core has prepared for sending
+ *
+ * Return: transport SDU size.
+ */
+size_t msg_command_size(struct scmi_xfer *xfer)
+{
+	return sizeof(struct scmi_msg_payld) + xfer->tx.len;
+}
+
+/**
+ * msg_response_size() - Maximum size of transport SDU for response.
+ *
+ * @xfer: message which core has prepared for sending
+ *
+ * Return: transport SDU size.
+ */
+size_t msg_response_size(struct scmi_xfer *xfer)
+{
+	return sizeof(struct scmi_msg_payld) + sizeof(__le32) + xfer->rx.len;
+}
+
+/**
+ * msg_tx_prepare() - Set up transport SDU for command.
+ *
+ * @msg: transport SDU for command
+ * @xfer: message which is being sent
+ */
+void msg_tx_prepare(struct scmi_msg_payld *msg, struct scmi_xfer *xfer)
+{
+	msg->msg_header = cpu_to_le32(pack_scmi_header(&xfer->hdr));
+	if (xfer->tx.buf)
+		memcpy(msg->msg_payload, xfer->tx.buf, xfer->tx.len);
+}
+
+/**
+ * msg_read_header() - Read SCMI header from transport SDU.
+ *
+ * @msg: transport SDU
+ *
+ * Return: SCMI header
+ */
+u32 msg_read_header(struct scmi_msg_payld *msg)
+{
+	return le32_to_cpu(msg->msg_header);
+}
+
+/**
+ * msg_fetch_response() - Fetch response SCMI payload from transport SDU.
+ *
+ * @msg: transport SDU with response
+ * @len: transport SDU size
+ * @xfer: message being responded to
+ */
+void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
+			struct scmi_xfer *xfer)
+{
+	size_t prefix_len = sizeof(*msg) + sizeof(msg->msg_payload[0]);
+
+	xfer->hdr.status = le32_to_cpu(msg->msg_payload[0]);
+	xfer->rx.len = min_t(size_t, xfer->rx.len,
+			     len >= prefix_len ? len - prefix_len : 0);
+
+	/* Take a copy to the rx buffer.. */
+	memcpy(xfer->rx.buf, &msg->msg_payload[1], xfer->rx.len);
+}
+
+/**
+ * msg_fetch_notification() - Fetch notification payload from transport SDU.
+ *
+ * @msg: transport SDU with notification
+ * @len: transport SDU size
+ * @max_len: maximum SCMI payload size to fetch
+ * @xfer: notification message
+ */
+void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
+			    size_t max_len, struct scmi_xfer *xfer)
+{
+	xfer->rx.len = min_t(size_t, max_len,
+			     len >= sizeof(*msg) ? len - sizeof(*msg) : 0);
+
+	/* Take a copy to the rx buffer.. */
+	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx.len);
+}
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 12/16] firmware: arm_scmi: Add message passing abstractions for transports
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Peter Hilber <peter.hilber@opensynergy.com>

Add abstractions for future transports using message passing, such as
virtio. Derive the abstractions from the shared memory abstractions.

Abstract the transport SDU through the opaque struct scmi_msg_payld.
Also enable the transport to determine all other required information
about the transport SDU.

Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 drivers/firmware/Kconfig           |   8 +-
 drivers/firmware/arm_scmi/Makefile |   1 +
 drivers/firmware/arm_scmi/common.h |  15 ++++
 drivers/firmware/arm_scmi/msg.c    | 113 +++++++++++++++++++++++++++++
 4 files changed, 136 insertions(+), 1 deletion(-)
 create mode 100644 drivers/firmware/arm_scmi/msg.c

diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
index 358f895847b5..e74d25065d07 100644
--- a/drivers/firmware/Kconfig
+++ b/drivers/firmware/Kconfig
@@ -9,7 +9,7 @@ menu "Firmware Drivers"
 config ARM_SCMI_PROTOCOL
 	tristate "ARM System Control and Management Interface (SCMI) Message Protocol"
 	depends on ARM || ARM64 || COMPILE_TEST
-	depends on ARM_SCMI_HAVE_SHMEM
+	depends on ARM_SCMI_HAVE_SHMEM || ARM_SCMI_HAVE_MSG
 	help
 	  ARM System Control and Management Interface (SCMI) protocol is a
 	  set of operating system-independent software interfaces that are
@@ -33,6 +33,12 @@ config ARM_SCMI_HAVE_SHMEM
 	  This declares whether a shared memory based transport for SCMI is
 	  available.
 
+config ARM_SCMI_HAVE_MSG
+	bool
+	help
+	  This declares whether a message passing based transport for SCMI is
+	  available.
+
 config ARM_SCMI_POWER_DOMAIN
 	tristate "SCMI power domain driver"
 	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
index 5a2d4c32e0ae..f6b4acb8abdb 100644
--- a/drivers/firmware/arm_scmi/Makefile
+++ b/drivers/firmware/arm_scmi/Makefile
@@ -4,6 +4,7 @@ scmi-driver-y = driver.o notify.o
 scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
 scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
 scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
+scmi-transport-$(CONFIG_ARM_SCMI_HAVE_MSG) += msg.o
 scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
 scmi-module-objs := $(scmi-bus-y) $(scmi-driver-y) $(scmi-protocols-y) \
 		    $(scmi-transport-y)
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index b783ae058c8a..fa4075336580 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -410,6 +410,21 @@ void shmem_clear_channel(struct scmi_shared_mem __iomem *shmem);
 bool shmem_poll_done(struct scmi_shared_mem __iomem *shmem,
 		     struct scmi_xfer *xfer);
 
+/* declarations for message passing transports */
+struct scmi_msg_payld;
+
+/** Maximum overhead of message w.r.t. struct scmi_desc.max_msg_size */
+#define SCMI_MSG_MAX_PROT_OVERHEAD (2 * sizeof(__le32))
+
+size_t msg_response_size(struct scmi_xfer *xfer);
+size_t msg_command_size(struct scmi_xfer *xfer);
+void msg_tx_prepare(struct scmi_msg_payld *msg, struct scmi_xfer *xfer);
+u32 msg_read_header(struct scmi_msg_payld *msg);
+void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
+			struct scmi_xfer *xfer);
+void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
+			    size_t max_len, struct scmi_xfer *xfer);
+
 void scmi_notification_instance_data_set(const struct scmi_handle *handle,
 					 void *priv);
 void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
diff --git a/drivers/firmware/arm_scmi/msg.c b/drivers/firmware/arm_scmi/msg.c
new file mode 100644
index 000000000000..8a2d3303d281
--- /dev/null
+++ b/drivers/firmware/arm_scmi/msg.c
@@ -0,0 +1,113 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * For transports using message passing.
+ *
+ * Derived from shm.c.
+ *
+ * Copyright (C) 2019 ARM Ltd.
+ * Copyright (C) 2020 OpenSynergy GmbH
+ */
+
+#include <linux/io.h>
+#include <linux/processor.h>
+#include <linux/types.h>
+
+#include "common.h"
+
+/*
+ * struct scmi_msg_payld - Transport SDU layout
+ *
+ * The SCMI specification requires all parameters, message headers, return
+ * arguments or any protocol data to be expressed in little endian format only.
+ */
+struct scmi_msg_payld {
+	__le32 msg_header;
+	__le32 msg_payload[];
+};
+
+/**
+ * msg_command_size() - Actual size of transport SDU for command.
+ *
+ * @xfer: message which core has prepared for sending
+ *
+ * Return: transport SDU size.
+ */
+size_t msg_command_size(struct scmi_xfer *xfer)
+{
+	return sizeof(struct scmi_msg_payld) + xfer->tx.len;
+}
+
+/**
+ * msg_response_size() - Maximum size of transport SDU for response.
+ *
+ * @xfer: message which core has prepared for sending
+ *
+ * Return: transport SDU size.
+ */
+size_t msg_response_size(struct scmi_xfer *xfer)
+{
+	return sizeof(struct scmi_msg_payld) + sizeof(__le32) + xfer->rx.len;
+}
+
+/**
+ * msg_tx_prepare() - Set up transport SDU for command.
+ *
+ * @msg: transport SDU for command
+ * @xfer: message which is being sent
+ */
+void msg_tx_prepare(struct scmi_msg_payld *msg, struct scmi_xfer *xfer)
+{
+	msg->msg_header = cpu_to_le32(pack_scmi_header(&xfer->hdr));
+	if (xfer->tx.buf)
+		memcpy(msg->msg_payload, xfer->tx.buf, xfer->tx.len);
+}
+
+/**
+ * msg_read_header() - Read SCMI header from transport SDU.
+ *
+ * @msg: transport SDU
+ *
+ * Return: SCMI header
+ */
+u32 msg_read_header(struct scmi_msg_payld *msg)
+{
+	return le32_to_cpu(msg->msg_header);
+}
+
+/**
+ * msg_fetch_response() - Fetch response SCMI payload from transport SDU.
+ *
+ * @msg: transport SDU with response
+ * @len: transport SDU size
+ * @xfer: message being responded to
+ */
+void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
+			struct scmi_xfer *xfer)
+{
+	size_t prefix_len = sizeof(*msg) + sizeof(msg->msg_payload[0]);
+
+	xfer->hdr.status = le32_to_cpu(msg->msg_payload[0]);
+	xfer->rx.len = min_t(size_t, xfer->rx.len,
+			     len >= prefix_len ? len - prefix_len : 0);
+
+	/* Take a copy to the rx buffer.. */
+	memcpy(xfer->rx.buf, &msg->msg_payload[1], xfer->rx.len);
+}
+
+/**
+ * msg_fetch_notification() - Fetch notification payload from transport SDU.
+ *
+ * @msg: transport SDU with notification
+ * @len: transport SDU size
+ * @max_len: maximum SCMI payload size to fetch
+ * @xfer: notification message
+ */
+void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
+			    size_t max_len, struct scmi_xfer *xfer)
+{
+	xfer->rx.len = min_t(size_t, max_len,
+			     len >= sizeof(*msg) ? len - sizeof(*msg) : 0);
+
+	/* Take a copy to the rx buffer.. */
+	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx.len);
+}
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy, Rob Herring, devicetree

From: Igor Skalkin <igor.skalkin@opensynergy.com>

Document the properties for arm,scmi-virtio compatible nodes.
The backing virtio SCMI device is described in patch [1].

While doing that, make shmem property required only for pre-existing
mailbox and smc transports, since virtio-scmi does not need it.

[1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html

CC: Rob Herring <robh+dt@kernel.org>
CC: devicetree@vger.kernel.org
Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
[ Peter: Adapted patch for submission to upstream. ]
Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
[ Cristian: converted to yaml format, moved shmen required property. ]
Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
v3 --> V4
- convertd to YAML
- make shmem required only for pre-existing mailbox and smc transport
- updated VirtIO specification patch message reference
- dropped virtio-mmio SCMI device example since really not pertinent to
  virtio-scmi dt bindings transport: it is not even referenced in SCMI
  virtio DT node since they are enumerated by VirtIO subsystem and there
  could be PCI based SCMI devices anyway.
---
 Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
index cebf6ffe70d5..5c4c6782e052 100644
--- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
+++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
@@ -34,6 +34,10 @@ properties:
       - description: SCMI compliant firmware with ARM SMC/HVC transport
         items:
           - const: arm,scmi-smc
+      - description: SCMI compliant firmware with SCMI Virtio transport.
+                     The virtio transport only supports a single device.
+        items:
+          - const: arm,scmi-virtio
 
   interrupts:
     description:
@@ -172,6 +176,7 @@ patternProperties:
       Each sub-node represents a protocol supported. If the platform
       supports a dedicated communication channel for a particular protocol,
       then the corresponding transport properties must be present.
+      The virtio transport does not support a dedicated communication channel.
 
     properties:
       reg:
@@ -195,7 +200,6 @@ patternProperties:
 
 required:
   - compatible
-  - shmem
 
 if:
   properties:
@@ -209,6 +213,7 @@ then:
 
   required:
     - mboxes
+    - shmem
 
 else:
   if:
@@ -219,6 +224,7 @@ else:
   then:
     required:
       - arm,smc-id
+      - shmem
 
 examples:
   - |
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy, Rob Herring, devicetree

From: Igor Skalkin <igor.skalkin@opensynergy.com>

Document the properties for arm,scmi-virtio compatible nodes.
The backing virtio SCMI device is described in patch [1].

While doing that, make shmem property required only for pre-existing
mailbox and smc transports, since virtio-scmi does not need it.

[1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html

CC: Rob Herring <robh+dt@kernel.org>
CC: devicetree@vger.kernel.org
Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
[ Peter: Adapted patch for submission to upstream. ]
Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
[ Cristian: converted to yaml format, moved shmen required property. ]
Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
v3 --> V4
- convertd to YAML
- make shmem required only for pre-existing mailbox and smc transport
- updated VirtIO specification patch message reference
- dropped virtio-mmio SCMI device example since really not pertinent to
  virtio-scmi dt bindings transport: it is not even referenced in SCMI
  virtio DT node since they are enumerated by VirtIO subsystem and there
  could be PCI based SCMI devices anyway.
---
 Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
index cebf6ffe70d5..5c4c6782e052 100644
--- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
+++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
@@ -34,6 +34,10 @@ properties:
       - description: SCMI compliant firmware with ARM SMC/HVC transport
         items:
           - const: arm,scmi-smc
+      - description: SCMI compliant firmware with SCMI Virtio transport.
+                     The virtio transport only supports a single device.
+        items:
+          - const: arm,scmi-virtio
 
   interrupts:
     description:
@@ -172,6 +176,7 @@ patternProperties:
       Each sub-node represents a protocol supported. If the platform
       supports a dedicated communication channel for a particular protocol,
       then the corresponding transport properties must be present.
+      The virtio transport does not support a dedicated communication channel.
 
     properties:
       reg:
@@ -195,7 +200,6 @@ patternProperties:
 
 required:
   - compatible
-  - shmem
 
 if:
   properties:
@@ -209,6 +213,7 @@ then:
 
   required:
     - mboxes
+    - shmem
 
 else:
   if:
@@ -219,6 +224,7 @@ else:
   then:
     required:
       - arm,smc-id
+      - shmem
 
 examples:
   - |
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Igor Skalkin <igor.skalkin@opensynergy.com>

This transport enables accessing an SCMI platform as a virtio device.

Implement an SCMI virtio driver according to the virtio SCMI device spec
[1]. Virtio device id 32 has been reserved for the SCMI device [2].

The virtio transport has one Tx channel (virtio cmdq, A2P channel) and
at most one Rx channel (virtio eventq, P2A channel).

The following feature bit defined in [1] is not implemented:
VIRTIO_SCMI_F_SHARED_MEMORY.

After the preparatory patches, this implements the virtio transport as
paraphrased:

Only support a single arm-scmi device (which is consistent with the SCMI
spec). scmi-virtio init is called from arm-scmi module init. During the
arm-scmi probing, link to the first probed scmi-virtio device. Defer
arm-scmi probing if no scmi-virtio device is bound yet.

For simplicity, restrict the number of messages which can be pending
simultaneously according to the virtqueue capacity. (The virtqueue sizes
are negotiated with the virtio device.)

As soon as Rx channel message buffers are allocated or have been read
out by the arm-scmi driver, feed them to the virtio device.

Since some virtio devices may not have the short response time exhibited
by SCMI platforms using other transports, set a generous response
timeout.

Limitations:

- Polling is not supported.

- The timeout for delayed responses has not been adjusted.

[1] https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-scmi.tex
[2] https://www.oasis-open.org/committees/ballot.php?id=3496

[ Peter: Adapted patch for submission to upstream. ]

Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 MAINTAINERS                        |   1 +
 drivers/firmware/Kconfig           |  12 +
 drivers/firmware/arm_scmi/Makefile |   1 +
 drivers/firmware/arm_scmi/common.h |   3 +
 drivers/firmware/arm_scmi/driver.c |   3 +
 drivers/firmware/arm_scmi/virtio.c | 523 +++++++++++++++++++++++++++++
 include/uapi/linux/virtio_ids.h    |   1 +
 include/uapi/linux/virtio_scmi.h   |  25 ++
 8 files changed, 569 insertions(+)
 create mode 100644 drivers/firmware/arm_scmi/virtio.c
 create mode 100644 include/uapi/linux/virtio_scmi.h

diff --git a/MAINTAINERS b/MAINTAINERS
index f408cf2d2781..e1b27ed11060 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17878,6 +17878,7 @@ F:	drivers/regulator/scmi-regulator.c
 F:	drivers/reset/reset-scmi.c
 F:	include/linux/sc[mp]i_protocol.h
 F:	include/trace/events/scmi.h
+F:	include/uapi/linux/virtio_scmi.h
 
 SYSTEM RESET/SHUTDOWN DRIVERS
 M:	Sebastian Reichel <sre@kernel.org>
diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
index e74d25065d07..2c37d5af66ad 100644
--- a/drivers/firmware/Kconfig
+++ b/drivers/firmware/Kconfig
@@ -39,6 +39,18 @@ config ARM_SCMI_HAVE_MSG
 	  This declares whether a message passing based transport for SCMI is
 	  available.
 
+# This config option includes the dependencies of ARM_SCMI_PROTOCOL so that
+# this config doesn't show up when SCMI wouldn't be available.
+config VIRTIO_SCMI
+	bool "Virtio transport for SCMI"
+	select ARM_SCMI_HAVE_MSG
+	depends on VIRTIO && (ARM || ARM64 || COMPILE_TEST)
+	help
+	  This enables the virtio based transport for SCMI.
+
+	  If you want to use the ARM SCMI protocol between the virtio guest and
+	  a host providing a virtio SCMI device, answer Y.
+
 config ARM_SCMI_POWER_DOMAIN
 	tristate "SCMI power domain driver"
 	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
index f6b4acb8abdb..db1787606fb2 100644
--- a/drivers/firmware/arm_scmi/Makefile
+++ b/drivers/firmware/arm_scmi/Makefile
@@ -5,6 +5,7 @@ scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
 scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
 scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
 scmi-transport-$(CONFIG_ARM_SCMI_HAVE_MSG) += msg.o
+scmi-transport-$(CONFIG_VIRTIO_SCMI) += virtio.o
 scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
 scmi-module-objs := $(scmi-bus-y) $(scmi-driver-y) $(scmi-protocols-y) \
 		    $(scmi-transport-y)
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index fa4075336580..c143a449d278 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -390,6 +390,9 @@ extern const struct scmi_desc scmi_mailbox_desc;
 #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
 extern const struct scmi_desc scmi_smc_desc;
 #endif
+#ifdef CONFIG_VIRTIO_SCMI
+extern const struct scmi_desc scmi_virtio_desc;
+#endif
 
 int scmi_set_transport_info(struct device *dev, void *transport_info);
 void *scmi_get_transport_info(struct device *dev);
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index bb2de15554a9..ccc7dd49261e 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1954,6 +1954,9 @@ static const struct of_device_id scmi_of_match[] = {
 #endif
 #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
 	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
+#endif
+#ifdef CONFIG_VIRTIO_SCMI
+	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
 #endif
 	{ /* Sentinel */ },
 };
diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
new file mode 100644
index 000000000000..20972adf6dc7
--- /dev/null
+++ b/drivers/firmware/arm_scmi/virtio.c
@@ -0,0 +1,523 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Virtio Transport driver for Arm System Control and Management Interface
+ * (SCMI).
+ *
+ * Copyright (C) 2020 OpenSynergy.
+ */
+
+/**
+ * DOC: Theory of Operation
+ *
+ * The scmi-virtio transport implements a driver for the virtio SCMI device.
+ *
+ * There is one Tx channel (virtio cmdq, A2P channel) and at most one Rx
+ * channel (virtio eventq, P2A channel). Each channel is implemented through a
+ * virtqueue. Access to each virtqueue is protected by spinlocks.
+ */
+
+#include <linux/errno.h>
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/virtio.h>
+#include <linux/virtio_config.h>
+#include <uapi/linux/virtio_ids.h>
+#include <uapi/linux/virtio_scmi.h>
+
+#include "common.h"
+
+#define VIRTIO_SCMI_MAX_MSG_SIZE 128 /* Value may be increased. */
+#define VIRTIO_SCMI_MAX_PDU_SIZE \
+	(VIRTIO_SCMI_MAX_MSG_SIZE + SCMI_MSG_MAX_PROT_OVERHEAD)
+#define DESCRIPTORS_PER_TX_MSG 2
+
+/**
+ * struct scmi_vio_channel - Transport channel information
+ *
+ * @lock: Protects access to all members except ready.
+ * @ready_lock: Protects access to ready. If required, it must be taken before
+ *              lock.
+ * @vqueue: Associated virtqueue
+ * @cinfo: SCMI Tx or Rx channel
+ * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
+ * @is_rx: Whether channel is an Rx channel
+ * @ready: Whether transport user is ready to hear about channel
+ */
+struct scmi_vio_channel {
+	spinlock_t lock;
+	spinlock_t ready_lock;
+	struct virtqueue *vqueue;
+	struct scmi_chan_info *cinfo;
+	struct list_head free_list;
+	u8 is_rx;
+	u8 ready;
+};
+
+/**
+ * struct scmi_vio_msg - Transport PDU information
+ *
+ * @request: SDU used for commands
+ * @input: SDU used for (delayed) responses and notifications
+ * @list: List which scmi_vio_msg may be part of
+ * @rx_len: Input SDU size in bytes, once input has been received
+ */
+struct scmi_vio_msg {
+	struct scmi_msg_payld *request;
+	struct scmi_msg_payld *input;
+	struct list_head list;
+	unsigned int rx_len;
+};
+
+static bool scmi_vio_have_vq_rx(struct virtio_device *vdev)
+{
+	return virtio_has_feature(vdev, VIRTIO_SCMI_F_P2A_CHANNELS);
+}
+
+static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
+			       struct scmi_vio_msg *msg)
+{
+	struct scatterlist sg_in;
+	int rc;
+	unsigned long flags;
+
+	sg_init_one(&sg_in, msg->input, VIRTIO_SCMI_MAX_PDU_SIZE);
+
+	spin_lock_irqsave(&vioch->lock, flags);
+
+	rc = virtqueue_add_inbuf(vioch->vqueue, &sg_in, 1, msg, GFP_ATOMIC);
+	if (rc)
+		dev_err_once(vioch->cinfo->dev,
+			     "%s() failed to add to virtqueue (%d)\n", __func__,
+			     rc);
+	else
+		virtqueue_kick(vioch->vqueue);
+
+	spin_unlock_irqrestore(&vioch->lock, flags);
+
+	return rc;
+}
+
+static void scmi_vio_complete_cb(struct virtqueue *vqueue)
+{
+	unsigned long ready_flags;
+	unsigned long flags;
+	unsigned int length;
+	struct scmi_vio_channel *vioch;
+	struct scmi_vio_msg *msg;
+	bool cb_enabled = true;
+
+	if (WARN_ON_ONCE(!vqueue->vdev->priv))
+		return;
+	vioch = &((struct scmi_vio_channel *)vqueue->vdev->priv)[vqueue->index];
+
+	for (;;) {
+		spin_lock_irqsave(&vioch->ready_lock, ready_flags);
+
+		if (!vioch->ready) {
+			if (!cb_enabled)
+				(void)virtqueue_enable_cb(vqueue);
+			goto unlock_ready_out;
+		}
+
+		spin_lock_irqsave(&vioch->lock, flags);
+		if (cb_enabled) {
+			virtqueue_disable_cb(vqueue);
+			cb_enabled = false;
+		}
+		msg = virtqueue_get_buf(vqueue, &length);
+		if (!msg) {
+			if (virtqueue_enable_cb(vqueue))
+				goto unlock_out;
+			else
+				cb_enabled = true;
+		}
+		spin_unlock_irqrestore(&vioch->lock, flags);
+
+		if (msg) {
+			msg->rx_len = length;
+
+			/*
+			 * Hold the ready_lock during the callback to avoid
+			 * races when the arm-scmi driver is unbinding while
+			 * the virtio device is not quiesced yet.
+			 */
+			scmi_rx_callback(vioch->cinfo,
+					 msg_read_header(msg->input), msg);
+		}
+		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
+	}
+
+unlock_out:
+	spin_unlock_irqrestore(&vioch->lock, flags);
+unlock_ready_out:
+	spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
+}
+
+static const char *const scmi_vio_vqueue_names[] = { "tx", "rx" };
+
+static vq_callback_t *scmi_vio_complete_callbacks[] = {
+	scmi_vio_complete_cb,
+	scmi_vio_complete_cb
+};
+
+static unsigned int virtio_get_max_msg(bool tx,
+				       struct scmi_chan_info *base_cinfo)
+{
+	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
+	unsigned int ret;
+
+	ret = virtqueue_get_vring_size(vioch->vqueue);
+
+	/* Tx messages need multiple descriptors. */
+	if (tx)
+		ret /= DESCRIPTORS_PER_TX_MSG;
+
+	if (ret > MSG_TOKEN_MAX) {
+		dev_info_once(
+			base_cinfo->dev,
+			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
+			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
+		ret = MSG_TOKEN_MAX;
+	}
+
+	return ret;
+}
+
+static int scmi_vio_match_any_dev(struct device *dev, const void *data)
+{
+	return 1;
+}
+
+static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
+
+static int virtio_link_supplier(struct device *dev)
+{
+	struct device *vdev = driver_find_device(
+		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
+
+	if (!vdev) {
+		dev_notice_once(
+			dev,
+			"Deferring probe after not finding a bound scmi-virtio device\n");
+		return -EPROBE_DEFER;
+	}
+
+	/* Add device link for remove order and sysfs link. */
+	if (!device_link_add(dev, vdev, DL_FLAG_AUTOREMOVE_CONSUMER)) {
+		put_device(vdev);
+		dev_err(dev, "Adding link to supplier virtio device failed\n");
+		return -ECANCELED;
+	}
+
+	put_device(vdev);
+	return scmi_set_transport_info(dev, dev_to_virtio(vdev));
+}
+
+static bool virtio_chan_available(struct device *dev, int idx)
+{
+	struct virtio_device *vdev;
+
+	/* scmi-virtio doesn't support per-protocol channels */
+	if (is_scmi_protocol_device(dev))
+		return false;
+
+	vdev = scmi_get_transport_info(dev);
+	if (!vdev)
+		return false;
+
+	switch (idx) {
+	case VIRTIO_SCMI_VQ_TX:
+		return true;
+	case VIRTIO_SCMI_VQ_RX:
+		return scmi_vio_have_vq_rx(vdev);
+	default:
+		return false;
+	}
+}
+
+static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
+			     bool tx)
+{
+	unsigned long flags;
+	struct virtio_device *vdev;
+	struct scmi_vio_channel *vioch;
+	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
+	int max_msg;
+	int i;
+
+	if (!virtio_chan_available(dev, index))
+		return -ENODEV;
+
+	vdev = scmi_get_transport_info(dev);
+	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
+
+	spin_lock_irqsave(&vioch->lock, flags);
+	cinfo->transport_info = vioch;
+	vioch->cinfo = cinfo;
+	spin_unlock_irqrestore(&vioch->lock, flags);
+
+	max_msg = virtio_get_max_msg(tx, cinfo);
+
+	for (i = 0; i < max_msg; i++) {
+		struct scmi_vio_msg *msg;
+
+		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
+		if (!msg)
+			return -ENOMEM;
+
+		if (tx) {
+			msg->request = devm_kzalloc(cinfo->dev,
+						    VIRTIO_SCMI_MAX_PDU_SIZE,
+						    GFP_KERNEL);
+			if (!msg->request)
+				return -ENOMEM;
+		}
+
+		msg->input = devm_kzalloc(cinfo->dev, VIRTIO_SCMI_MAX_PDU_SIZE,
+					  GFP_KERNEL);
+		if (!msg->input)
+			return -ENOMEM;
+
+		if (tx) {
+			spin_lock_irqsave(&vioch->lock, flags);
+			list_add_tail(&msg->list, &vioch->free_list);
+			spin_unlock_irqrestore(&vioch->lock, flags);
+		} else {
+			scmi_vio_feed_vq_rx(vioch, msg);
+		}
+	}
+
+	spin_lock_irqsave(&vioch->ready_lock, flags);
+	vioch->ready = true;
+	spin_unlock_irqrestore(&vioch->ready_lock, flags);
+
+	return 0;
+}
+
+static int virtio_chan_free(int id, void *p, void *data)
+{
+	unsigned long flags;
+	struct scmi_chan_info *cinfo = p;
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+
+	spin_lock_irqsave(&vioch->ready_lock, flags);
+	vioch->ready = false;
+	spin_unlock_irqrestore(&vioch->ready_lock, flags);
+
+	scmi_free_channel(cinfo, data, id);
+	return 0;
+}
+
+static int virtio_send_message(struct scmi_chan_info *cinfo,
+			       struct scmi_xfer *xfer)
+{
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+	struct scatterlist sg_out;
+	struct scatterlist sg_in;
+	struct scatterlist *sgs[DESCRIPTORS_PER_TX_MSG] = { &sg_out, &sg_in };
+	unsigned long flags;
+	int rc;
+	struct scmi_vio_msg *msg;
+
+	/*
+	 * TODO: For now, we don't support polling. But it should not be
+	 * difficult to add support.
+	 */
+	if (xfer->hdr.poll_completion)
+		return -EINVAL;
+
+	spin_lock_irqsave(&vioch->lock, flags);
+
+	if (list_empty(&vioch->free_list)) {
+		spin_unlock_irqrestore(&vioch->lock, flags);
+		return -EBUSY;
+	}
+
+	msg = list_first_entry(&vioch->free_list, typeof(*msg), list);
+	list_del(&msg->list);
+
+	msg_tx_prepare(msg->request, xfer);
+
+	sg_init_one(&sg_out, msg->request, msg_command_size(xfer));
+	sg_init_one(&sg_in, msg->input, msg_response_size(xfer));
+
+	rc = virtqueue_add_sgs(vioch->vqueue, sgs, 1, 1, msg, GFP_ATOMIC);
+	if (rc) {
+		list_add(&msg->list, &vioch->free_list);
+		dev_err_once(vioch->cinfo->dev,
+			     "%s() failed to add to virtqueue (%d)\n", __func__,
+			     rc);
+	} else {
+		virtqueue_kick(vioch->vqueue);
+	}
+
+	spin_unlock_irqrestore(&vioch->lock, flags);
+
+	return rc;
+}
+
+static void virtio_fetch_response(struct scmi_chan_info *cinfo,
+				  struct scmi_xfer *xfer, void *msg_handle)
+{
+	struct scmi_vio_msg *msg = msg_handle;
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+
+	if (!msg) {
+		dev_dbg_once(&vioch->vqueue->vdev->dev,
+			     "Ignoring %s() call with NULL msg_handle\n",
+			     __func__);
+		return;
+	}
+
+	msg_fetch_response(msg->input, msg->rx_len, xfer);
+}
+
+static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
+				      size_t max_len, struct scmi_xfer *xfer,
+				      void *msg_handle)
+{
+	struct scmi_vio_msg *msg = msg_handle;
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+
+	if (!msg) {
+		dev_dbg_once(&vioch->vqueue->vdev->dev,
+			     "Ignoring %s() call with NULL msg_handle\n",
+			     __func__);
+		return;
+	}
+
+	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
+}
+
+static void dummy_clear_channel(struct scmi_chan_info *cinfo)
+{
+}
+
+static bool dummy_poll_done(struct scmi_chan_info *cinfo,
+			    struct scmi_xfer *xfer)
+{
+	return false;
+}
+
+static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
+{
+	unsigned long flags;
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+	struct scmi_vio_msg *msg = msg_handle;
+
+	if (!msg) {
+		dev_dbg_once(&vioch->vqueue->vdev->dev,
+			     "Ignoring %s() call with NULL msg_handle\n",
+			     __func__);
+		return;
+	}
+
+	if (vioch->is_rx) {
+		scmi_vio_feed_vq_rx(vioch, msg);
+	} else {
+		spin_lock_irqsave(&vioch->lock, flags);
+		list_add(&msg->list, &vioch->free_list);
+		spin_unlock_irqrestore(&vioch->lock, flags);
+	}
+}
+
+static const struct scmi_transport_ops scmi_virtio_ops = {
+	.link_supplier = virtio_link_supplier,
+	.chan_available = virtio_chan_available,
+	.chan_setup = virtio_chan_setup,
+	.chan_free = virtio_chan_free,
+	.get_max_msg = virtio_get_max_msg,
+	.send_message = virtio_send_message,
+	.fetch_response = virtio_fetch_response,
+	.fetch_notification = virtio_fetch_notification,
+	.clear_channel = dummy_clear_channel,
+	.poll_done = dummy_poll_done,
+	.drop_message = virtio_drop_message,
+};
+
+static int scmi_vio_probe(struct virtio_device *vdev)
+{
+	struct device *dev = &vdev->dev;
+	struct scmi_vio_channel *channels;
+	bool have_vq_rx;
+	int vq_cnt;
+	int i;
+	int ret;
+	struct virtqueue *vqs[VIRTIO_SCMI_VQ_MAX_CNT];
+
+	have_vq_rx = scmi_vio_have_vq_rx(vdev);
+	vq_cnt = have_vq_rx ? VIRTIO_SCMI_VQ_MAX_CNT : 1;
+
+	channels = devm_kcalloc(dev, vq_cnt, sizeof(*channels), GFP_KERNEL);
+	if (!channels)
+		return -ENOMEM;
+
+	if (have_vq_rx)
+		channels[VIRTIO_SCMI_VQ_RX].is_rx = true;
+
+	ret = virtio_find_vqs(vdev, vq_cnt, vqs, scmi_vio_complete_callbacks,
+			      scmi_vio_vqueue_names, NULL);
+	if (ret) {
+		dev_err(dev, "Failed to get %d virtqueue(s)\n", vq_cnt);
+		return ret;
+	}
+	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);
+
+	for (i = 0; i < vq_cnt; i++) {
+		spin_lock_init(&channels[i].lock);
+		spin_lock_init(&channels[i].ready_lock);
+		INIT_LIST_HEAD(&channels[i].free_list);
+		channels[i].vqueue = vqs[i];
+	}
+
+	vdev->priv = channels;
+
+	return 0;
+}
+
+static void scmi_vio_remove(struct virtio_device *vdev)
+{
+	vdev->config->reset(vdev);
+	vdev->config->del_vqs(vdev);
+}
+
+static unsigned int features[] = {
+	VIRTIO_SCMI_F_P2A_CHANNELS,
+};
+
+static const struct virtio_device_id id_table[] = {
+	{ VIRTIO_ID_SCMI, VIRTIO_DEV_ANY_ID },
+	{ 0 }
+};
+
+static struct virtio_driver virtio_scmi_driver = {
+	.driver.name = "scmi-virtio",
+	.driver.owner = THIS_MODULE,
+	.feature_table = features,
+	.feature_table_size = ARRAY_SIZE(features),
+	.id_table = id_table,
+	.probe = scmi_vio_probe,
+	.remove = scmi_vio_remove,
+};
+
+static int __init virtio_scmi_init(void)
+{
+	return register_virtio_driver(&virtio_scmi_driver);
+}
+
+static void __exit virtio_scmi_exit(void)
+{
+	unregister_virtio_driver(&virtio_scmi_driver);
+}
+
+const struct scmi_desc scmi_virtio_desc = {
+	.init = virtio_scmi_init,
+	.exit = virtio_scmi_exit,
+	.ops = &scmi_virtio_ops,
+	.max_rx_timeout_ms = 60000, /* for non-realtime virtio devices */
+	.max_msg = 0, /* overridden by virtio_get_max_msg() */
+	.max_msg_size = VIRTIO_SCMI_MAX_MSG_SIZE,
+};
diff --git a/include/uapi/linux/virtio_ids.h b/include/uapi/linux/virtio_ids.h
index 4fe842c3a3a9..ead8178efffa 100644
--- a/include/uapi/linux/virtio_ids.h
+++ b/include/uapi/linux/virtio_ids.h
@@ -55,6 +55,7 @@
 #define VIRTIO_ID_FS			26 /* virtio filesystem */
 #define VIRTIO_ID_PMEM			27 /* virtio pmem */
 #define VIRTIO_ID_MAC80211_HWSIM	29 /* virtio mac80211-hwsim */
+#define VIRTIO_ID_SCMI			32 /* virtio SCMI */
 #define VIRTIO_ID_BT			40 /* virtio bluetooth */
 
 #endif /* _LINUX_VIRTIO_IDS_H */
diff --git a/include/uapi/linux/virtio_scmi.h b/include/uapi/linux/virtio_scmi.h
new file mode 100644
index 000000000000..732b01504c35
--- /dev/null
+++ b/include/uapi/linux/virtio_scmi.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * Copyright (C) 2020 OpenSynergy GmbH
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_SCMI_H
+#define _UAPI_LINUX_VIRTIO_SCMI_H
+
+#include <linux/virtio_types.h>
+
+/* Feature bits */
+
+/* Device implements some SCMI notifications, or delayed responses. */
+#define VIRTIO_SCMI_F_P2A_CHANNELS 0
+
+/* Device implements any SCMI statistics shared memory region */
+#define VIRTIO_SCMI_F_SHARED_MEMORY 1
+
+/* Virtqueues */
+
+#define VIRTIO_SCMI_VQ_TX 0 /* cmdq */
+#define VIRTIO_SCMI_VQ_RX 1 /* eventq */
+#define VIRTIO_SCMI_VQ_MAX_CNT 2
+
+#endif /* _UAPI_LINUX_VIRTIO_SCMI_H */
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

From: Igor Skalkin <igor.skalkin@opensynergy.com>

This transport enables accessing an SCMI platform as a virtio device.

Implement an SCMI virtio driver according to the virtio SCMI device spec
[1]. Virtio device id 32 has been reserved for the SCMI device [2].

The virtio transport has one Tx channel (virtio cmdq, A2P channel) and
at most one Rx channel (virtio eventq, P2A channel).

The following feature bit defined in [1] is not implemented:
VIRTIO_SCMI_F_SHARED_MEMORY.

After the preparatory patches, this implements the virtio transport as
paraphrased:

Only support a single arm-scmi device (which is consistent with the SCMI
spec). scmi-virtio init is called from arm-scmi module init. During the
arm-scmi probing, link to the first probed scmi-virtio device. Defer
arm-scmi probing if no scmi-virtio device is bound yet.

For simplicity, restrict the number of messages which can be pending
simultaneously according to the virtqueue capacity. (The virtqueue sizes
are negotiated with the virtio device.)

As soon as Rx channel message buffers are allocated or have been read
out by the arm-scmi driver, feed them to the virtio device.

Since some virtio devices may not have the short response time exhibited
by SCMI platforms using other transports, set a generous response
timeout.

Limitations:

- Polling is not supported.

- The timeout for delayed responses has not been adjusted.

[1] https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-scmi.tex
[2] https://www.oasis-open.org/committees/ballot.php?id=3496

[ Peter: Adapted patch for submission to upstream. ]

Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
---
 MAINTAINERS                        |   1 +
 drivers/firmware/Kconfig           |  12 +
 drivers/firmware/arm_scmi/Makefile |   1 +
 drivers/firmware/arm_scmi/common.h |   3 +
 drivers/firmware/arm_scmi/driver.c |   3 +
 drivers/firmware/arm_scmi/virtio.c | 523 +++++++++++++++++++++++++++++
 include/uapi/linux/virtio_ids.h    |   1 +
 include/uapi/linux/virtio_scmi.h   |  25 ++
 8 files changed, 569 insertions(+)
 create mode 100644 drivers/firmware/arm_scmi/virtio.c
 create mode 100644 include/uapi/linux/virtio_scmi.h

diff --git a/MAINTAINERS b/MAINTAINERS
index f408cf2d2781..e1b27ed11060 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17878,6 +17878,7 @@ F:	drivers/regulator/scmi-regulator.c
 F:	drivers/reset/reset-scmi.c
 F:	include/linux/sc[mp]i_protocol.h
 F:	include/trace/events/scmi.h
+F:	include/uapi/linux/virtio_scmi.h
 
 SYSTEM RESET/SHUTDOWN DRIVERS
 M:	Sebastian Reichel <sre@kernel.org>
diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
index e74d25065d07..2c37d5af66ad 100644
--- a/drivers/firmware/Kconfig
+++ b/drivers/firmware/Kconfig
@@ -39,6 +39,18 @@ config ARM_SCMI_HAVE_MSG
 	  This declares whether a message passing based transport for SCMI is
 	  available.
 
+# This config option includes the dependencies of ARM_SCMI_PROTOCOL so that
+# this config doesn't show up when SCMI wouldn't be available.
+config VIRTIO_SCMI
+	bool "Virtio transport for SCMI"
+	select ARM_SCMI_HAVE_MSG
+	depends on VIRTIO && (ARM || ARM64 || COMPILE_TEST)
+	help
+	  This enables the virtio based transport for SCMI.
+
+	  If you want to use the ARM SCMI protocol between the virtio guest and
+	  a host providing a virtio SCMI device, answer Y.
+
 config ARM_SCMI_POWER_DOMAIN
 	tristate "SCMI power domain driver"
 	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
index f6b4acb8abdb..db1787606fb2 100644
--- a/drivers/firmware/arm_scmi/Makefile
+++ b/drivers/firmware/arm_scmi/Makefile
@@ -5,6 +5,7 @@ scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
 scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
 scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
 scmi-transport-$(CONFIG_ARM_SCMI_HAVE_MSG) += msg.o
+scmi-transport-$(CONFIG_VIRTIO_SCMI) += virtio.o
 scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
 scmi-module-objs := $(scmi-bus-y) $(scmi-driver-y) $(scmi-protocols-y) \
 		    $(scmi-transport-y)
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index fa4075336580..c143a449d278 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -390,6 +390,9 @@ extern const struct scmi_desc scmi_mailbox_desc;
 #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
 extern const struct scmi_desc scmi_smc_desc;
 #endif
+#ifdef CONFIG_VIRTIO_SCMI
+extern const struct scmi_desc scmi_virtio_desc;
+#endif
 
 int scmi_set_transport_info(struct device *dev, void *transport_info);
 void *scmi_get_transport_info(struct device *dev);
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index bb2de15554a9..ccc7dd49261e 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -1954,6 +1954,9 @@ static const struct of_device_id scmi_of_match[] = {
 #endif
 #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
 	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
+#endif
+#ifdef CONFIG_VIRTIO_SCMI
+	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
 #endif
 	{ /* Sentinel */ },
 };
diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
new file mode 100644
index 000000000000..20972adf6dc7
--- /dev/null
+++ b/drivers/firmware/arm_scmi/virtio.c
@@ -0,0 +1,523 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Virtio Transport driver for Arm System Control and Management Interface
+ * (SCMI).
+ *
+ * Copyright (C) 2020 OpenSynergy.
+ */
+
+/**
+ * DOC: Theory of Operation
+ *
+ * The scmi-virtio transport implements a driver for the virtio SCMI device.
+ *
+ * There is one Tx channel (virtio cmdq, A2P channel) and at most one Rx
+ * channel (virtio eventq, P2A channel). Each channel is implemented through a
+ * virtqueue. Access to each virtqueue is protected by spinlocks.
+ */
+
+#include <linux/errno.h>
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/virtio.h>
+#include <linux/virtio_config.h>
+#include <uapi/linux/virtio_ids.h>
+#include <uapi/linux/virtio_scmi.h>
+
+#include "common.h"
+
+#define VIRTIO_SCMI_MAX_MSG_SIZE 128 /* Value may be increased. */
+#define VIRTIO_SCMI_MAX_PDU_SIZE \
+	(VIRTIO_SCMI_MAX_MSG_SIZE + SCMI_MSG_MAX_PROT_OVERHEAD)
+#define DESCRIPTORS_PER_TX_MSG 2
+
+/**
+ * struct scmi_vio_channel - Transport channel information
+ *
+ * @lock: Protects access to all members except ready.
+ * @ready_lock: Protects access to ready. If required, it must be taken before
+ *              lock.
+ * @vqueue: Associated virtqueue
+ * @cinfo: SCMI Tx or Rx channel
+ * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
+ * @is_rx: Whether channel is an Rx channel
+ * @ready: Whether transport user is ready to hear about channel
+ */
+struct scmi_vio_channel {
+	spinlock_t lock;
+	spinlock_t ready_lock;
+	struct virtqueue *vqueue;
+	struct scmi_chan_info *cinfo;
+	struct list_head free_list;
+	u8 is_rx;
+	u8 ready;
+};
+
+/**
+ * struct scmi_vio_msg - Transport PDU information
+ *
+ * @request: SDU used for commands
+ * @input: SDU used for (delayed) responses and notifications
+ * @list: List which scmi_vio_msg may be part of
+ * @rx_len: Input SDU size in bytes, once input has been received
+ */
+struct scmi_vio_msg {
+	struct scmi_msg_payld *request;
+	struct scmi_msg_payld *input;
+	struct list_head list;
+	unsigned int rx_len;
+};
+
+static bool scmi_vio_have_vq_rx(struct virtio_device *vdev)
+{
+	return virtio_has_feature(vdev, VIRTIO_SCMI_F_P2A_CHANNELS);
+}
+
+static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
+			       struct scmi_vio_msg *msg)
+{
+	struct scatterlist sg_in;
+	int rc;
+	unsigned long flags;
+
+	sg_init_one(&sg_in, msg->input, VIRTIO_SCMI_MAX_PDU_SIZE);
+
+	spin_lock_irqsave(&vioch->lock, flags);
+
+	rc = virtqueue_add_inbuf(vioch->vqueue, &sg_in, 1, msg, GFP_ATOMIC);
+	if (rc)
+		dev_err_once(vioch->cinfo->dev,
+			     "%s() failed to add to virtqueue (%d)\n", __func__,
+			     rc);
+	else
+		virtqueue_kick(vioch->vqueue);
+
+	spin_unlock_irqrestore(&vioch->lock, flags);
+
+	return rc;
+}
+
+static void scmi_vio_complete_cb(struct virtqueue *vqueue)
+{
+	unsigned long ready_flags;
+	unsigned long flags;
+	unsigned int length;
+	struct scmi_vio_channel *vioch;
+	struct scmi_vio_msg *msg;
+	bool cb_enabled = true;
+
+	if (WARN_ON_ONCE(!vqueue->vdev->priv))
+		return;
+	vioch = &((struct scmi_vio_channel *)vqueue->vdev->priv)[vqueue->index];
+
+	for (;;) {
+		spin_lock_irqsave(&vioch->ready_lock, ready_flags);
+
+		if (!vioch->ready) {
+			if (!cb_enabled)
+				(void)virtqueue_enable_cb(vqueue);
+			goto unlock_ready_out;
+		}
+
+		spin_lock_irqsave(&vioch->lock, flags);
+		if (cb_enabled) {
+			virtqueue_disable_cb(vqueue);
+			cb_enabled = false;
+		}
+		msg = virtqueue_get_buf(vqueue, &length);
+		if (!msg) {
+			if (virtqueue_enable_cb(vqueue))
+				goto unlock_out;
+			else
+				cb_enabled = true;
+		}
+		spin_unlock_irqrestore(&vioch->lock, flags);
+
+		if (msg) {
+			msg->rx_len = length;
+
+			/*
+			 * Hold the ready_lock during the callback to avoid
+			 * races when the arm-scmi driver is unbinding while
+			 * the virtio device is not quiesced yet.
+			 */
+			scmi_rx_callback(vioch->cinfo,
+					 msg_read_header(msg->input), msg);
+		}
+		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
+	}
+
+unlock_out:
+	spin_unlock_irqrestore(&vioch->lock, flags);
+unlock_ready_out:
+	spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
+}
+
+static const char *const scmi_vio_vqueue_names[] = { "tx", "rx" };
+
+static vq_callback_t *scmi_vio_complete_callbacks[] = {
+	scmi_vio_complete_cb,
+	scmi_vio_complete_cb
+};
+
+static unsigned int virtio_get_max_msg(bool tx,
+				       struct scmi_chan_info *base_cinfo)
+{
+	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
+	unsigned int ret;
+
+	ret = virtqueue_get_vring_size(vioch->vqueue);
+
+	/* Tx messages need multiple descriptors. */
+	if (tx)
+		ret /= DESCRIPTORS_PER_TX_MSG;
+
+	if (ret > MSG_TOKEN_MAX) {
+		dev_info_once(
+			base_cinfo->dev,
+			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
+			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
+		ret = MSG_TOKEN_MAX;
+	}
+
+	return ret;
+}
+
+static int scmi_vio_match_any_dev(struct device *dev, const void *data)
+{
+	return 1;
+}
+
+static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
+
+static int virtio_link_supplier(struct device *dev)
+{
+	struct device *vdev = driver_find_device(
+		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
+
+	if (!vdev) {
+		dev_notice_once(
+			dev,
+			"Deferring probe after not finding a bound scmi-virtio device\n");
+		return -EPROBE_DEFER;
+	}
+
+	/* Add device link for remove order and sysfs link. */
+	if (!device_link_add(dev, vdev, DL_FLAG_AUTOREMOVE_CONSUMER)) {
+		put_device(vdev);
+		dev_err(dev, "Adding link to supplier virtio device failed\n");
+		return -ECANCELED;
+	}
+
+	put_device(vdev);
+	return scmi_set_transport_info(dev, dev_to_virtio(vdev));
+}
+
+static bool virtio_chan_available(struct device *dev, int idx)
+{
+	struct virtio_device *vdev;
+
+	/* scmi-virtio doesn't support per-protocol channels */
+	if (is_scmi_protocol_device(dev))
+		return false;
+
+	vdev = scmi_get_transport_info(dev);
+	if (!vdev)
+		return false;
+
+	switch (idx) {
+	case VIRTIO_SCMI_VQ_TX:
+		return true;
+	case VIRTIO_SCMI_VQ_RX:
+		return scmi_vio_have_vq_rx(vdev);
+	default:
+		return false;
+	}
+}
+
+static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
+			     bool tx)
+{
+	unsigned long flags;
+	struct virtio_device *vdev;
+	struct scmi_vio_channel *vioch;
+	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
+	int max_msg;
+	int i;
+
+	if (!virtio_chan_available(dev, index))
+		return -ENODEV;
+
+	vdev = scmi_get_transport_info(dev);
+	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
+
+	spin_lock_irqsave(&vioch->lock, flags);
+	cinfo->transport_info = vioch;
+	vioch->cinfo = cinfo;
+	spin_unlock_irqrestore(&vioch->lock, flags);
+
+	max_msg = virtio_get_max_msg(tx, cinfo);
+
+	for (i = 0; i < max_msg; i++) {
+		struct scmi_vio_msg *msg;
+
+		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
+		if (!msg)
+			return -ENOMEM;
+
+		if (tx) {
+			msg->request = devm_kzalloc(cinfo->dev,
+						    VIRTIO_SCMI_MAX_PDU_SIZE,
+						    GFP_KERNEL);
+			if (!msg->request)
+				return -ENOMEM;
+		}
+
+		msg->input = devm_kzalloc(cinfo->dev, VIRTIO_SCMI_MAX_PDU_SIZE,
+					  GFP_KERNEL);
+		if (!msg->input)
+			return -ENOMEM;
+
+		if (tx) {
+			spin_lock_irqsave(&vioch->lock, flags);
+			list_add_tail(&msg->list, &vioch->free_list);
+			spin_unlock_irqrestore(&vioch->lock, flags);
+		} else {
+			scmi_vio_feed_vq_rx(vioch, msg);
+		}
+	}
+
+	spin_lock_irqsave(&vioch->ready_lock, flags);
+	vioch->ready = true;
+	spin_unlock_irqrestore(&vioch->ready_lock, flags);
+
+	return 0;
+}
+
+static int virtio_chan_free(int id, void *p, void *data)
+{
+	unsigned long flags;
+	struct scmi_chan_info *cinfo = p;
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+
+	spin_lock_irqsave(&vioch->ready_lock, flags);
+	vioch->ready = false;
+	spin_unlock_irqrestore(&vioch->ready_lock, flags);
+
+	scmi_free_channel(cinfo, data, id);
+	return 0;
+}
+
+static int virtio_send_message(struct scmi_chan_info *cinfo,
+			       struct scmi_xfer *xfer)
+{
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+	struct scatterlist sg_out;
+	struct scatterlist sg_in;
+	struct scatterlist *sgs[DESCRIPTORS_PER_TX_MSG] = { &sg_out, &sg_in };
+	unsigned long flags;
+	int rc;
+	struct scmi_vio_msg *msg;
+
+	/*
+	 * TODO: For now, we don't support polling. But it should not be
+	 * difficult to add support.
+	 */
+	if (xfer->hdr.poll_completion)
+		return -EINVAL;
+
+	spin_lock_irqsave(&vioch->lock, flags);
+
+	if (list_empty(&vioch->free_list)) {
+		spin_unlock_irqrestore(&vioch->lock, flags);
+		return -EBUSY;
+	}
+
+	msg = list_first_entry(&vioch->free_list, typeof(*msg), list);
+	list_del(&msg->list);
+
+	msg_tx_prepare(msg->request, xfer);
+
+	sg_init_one(&sg_out, msg->request, msg_command_size(xfer));
+	sg_init_one(&sg_in, msg->input, msg_response_size(xfer));
+
+	rc = virtqueue_add_sgs(vioch->vqueue, sgs, 1, 1, msg, GFP_ATOMIC);
+	if (rc) {
+		list_add(&msg->list, &vioch->free_list);
+		dev_err_once(vioch->cinfo->dev,
+			     "%s() failed to add to virtqueue (%d)\n", __func__,
+			     rc);
+	} else {
+		virtqueue_kick(vioch->vqueue);
+	}
+
+	spin_unlock_irqrestore(&vioch->lock, flags);
+
+	return rc;
+}
+
+static void virtio_fetch_response(struct scmi_chan_info *cinfo,
+				  struct scmi_xfer *xfer, void *msg_handle)
+{
+	struct scmi_vio_msg *msg = msg_handle;
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+
+	if (!msg) {
+		dev_dbg_once(&vioch->vqueue->vdev->dev,
+			     "Ignoring %s() call with NULL msg_handle\n",
+			     __func__);
+		return;
+	}
+
+	msg_fetch_response(msg->input, msg->rx_len, xfer);
+}
+
+static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
+				      size_t max_len, struct scmi_xfer *xfer,
+				      void *msg_handle)
+{
+	struct scmi_vio_msg *msg = msg_handle;
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+
+	if (!msg) {
+		dev_dbg_once(&vioch->vqueue->vdev->dev,
+			     "Ignoring %s() call with NULL msg_handle\n",
+			     __func__);
+		return;
+	}
+
+	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
+}
+
+static void dummy_clear_channel(struct scmi_chan_info *cinfo)
+{
+}
+
+static bool dummy_poll_done(struct scmi_chan_info *cinfo,
+			    struct scmi_xfer *xfer)
+{
+	return false;
+}
+
+static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
+{
+	unsigned long flags;
+	struct scmi_vio_channel *vioch = cinfo->transport_info;
+	struct scmi_vio_msg *msg = msg_handle;
+
+	if (!msg) {
+		dev_dbg_once(&vioch->vqueue->vdev->dev,
+			     "Ignoring %s() call with NULL msg_handle\n",
+			     __func__);
+		return;
+	}
+
+	if (vioch->is_rx) {
+		scmi_vio_feed_vq_rx(vioch, msg);
+	} else {
+		spin_lock_irqsave(&vioch->lock, flags);
+		list_add(&msg->list, &vioch->free_list);
+		spin_unlock_irqrestore(&vioch->lock, flags);
+	}
+}
+
+static const struct scmi_transport_ops scmi_virtio_ops = {
+	.link_supplier = virtio_link_supplier,
+	.chan_available = virtio_chan_available,
+	.chan_setup = virtio_chan_setup,
+	.chan_free = virtio_chan_free,
+	.get_max_msg = virtio_get_max_msg,
+	.send_message = virtio_send_message,
+	.fetch_response = virtio_fetch_response,
+	.fetch_notification = virtio_fetch_notification,
+	.clear_channel = dummy_clear_channel,
+	.poll_done = dummy_poll_done,
+	.drop_message = virtio_drop_message,
+};
+
+static int scmi_vio_probe(struct virtio_device *vdev)
+{
+	struct device *dev = &vdev->dev;
+	struct scmi_vio_channel *channels;
+	bool have_vq_rx;
+	int vq_cnt;
+	int i;
+	int ret;
+	struct virtqueue *vqs[VIRTIO_SCMI_VQ_MAX_CNT];
+
+	have_vq_rx = scmi_vio_have_vq_rx(vdev);
+	vq_cnt = have_vq_rx ? VIRTIO_SCMI_VQ_MAX_CNT : 1;
+
+	channels = devm_kcalloc(dev, vq_cnt, sizeof(*channels), GFP_KERNEL);
+	if (!channels)
+		return -ENOMEM;
+
+	if (have_vq_rx)
+		channels[VIRTIO_SCMI_VQ_RX].is_rx = true;
+
+	ret = virtio_find_vqs(vdev, vq_cnt, vqs, scmi_vio_complete_callbacks,
+			      scmi_vio_vqueue_names, NULL);
+	if (ret) {
+		dev_err(dev, "Failed to get %d virtqueue(s)\n", vq_cnt);
+		return ret;
+	}
+	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);
+
+	for (i = 0; i < vq_cnt; i++) {
+		spin_lock_init(&channels[i].lock);
+		spin_lock_init(&channels[i].ready_lock);
+		INIT_LIST_HEAD(&channels[i].free_list);
+		channels[i].vqueue = vqs[i];
+	}
+
+	vdev->priv = channels;
+
+	return 0;
+}
+
+static void scmi_vio_remove(struct virtio_device *vdev)
+{
+	vdev->config->reset(vdev);
+	vdev->config->del_vqs(vdev);
+}
+
+static unsigned int features[] = {
+	VIRTIO_SCMI_F_P2A_CHANNELS,
+};
+
+static const struct virtio_device_id id_table[] = {
+	{ VIRTIO_ID_SCMI, VIRTIO_DEV_ANY_ID },
+	{ 0 }
+};
+
+static struct virtio_driver virtio_scmi_driver = {
+	.driver.name = "scmi-virtio",
+	.driver.owner = THIS_MODULE,
+	.feature_table = features,
+	.feature_table_size = ARRAY_SIZE(features),
+	.id_table = id_table,
+	.probe = scmi_vio_probe,
+	.remove = scmi_vio_remove,
+};
+
+static int __init virtio_scmi_init(void)
+{
+	return register_virtio_driver(&virtio_scmi_driver);
+}
+
+static void __exit virtio_scmi_exit(void)
+{
+	unregister_virtio_driver(&virtio_scmi_driver);
+}
+
+const struct scmi_desc scmi_virtio_desc = {
+	.init = virtio_scmi_init,
+	.exit = virtio_scmi_exit,
+	.ops = &scmi_virtio_ops,
+	.max_rx_timeout_ms = 60000, /* for non-realtime virtio devices */
+	.max_msg = 0, /* overridden by virtio_get_max_msg() */
+	.max_msg_size = VIRTIO_SCMI_MAX_MSG_SIZE,
+};
diff --git a/include/uapi/linux/virtio_ids.h b/include/uapi/linux/virtio_ids.h
index 4fe842c3a3a9..ead8178efffa 100644
--- a/include/uapi/linux/virtio_ids.h
+++ b/include/uapi/linux/virtio_ids.h
@@ -55,6 +55,7 @@
 #define VIRTIO_ID_FS			26 /* virtio filesystem */
 #define VIRTIO_ID_PMEM			27 /* virtio pmem */
 #define VIRTIO_ID_MAC80211_HWSIM	29 /* virtio mac80211-hwsim */
+#define VIRTIO_ID_SCMI			32 /* virtio SCMI */
 #define VIRTIO_ID_BT			40 /* virtio bluetooth */
 
 #endif /* _LINUX_VIRTIO_IDS_H */
diff --git a/include/uapi/linux/virtio_scmi.h b/include/uapi/linux/virtio_scmi.h
new file mode 100644
index 000000000000..732b01504c35
--- /dev/null
+++ b/include/uapi/linux/virtio_scmi.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * Copyright (C) 2020 OpenSynergy GmbH
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_SCMI_H
+#define _UAPI_LINUX_VIRTIO_SCMI_H
+
+#include <linux/virtio_types.h>
+
+/* Feature bits */
+
+/* Device implements some SCMI notifications, or delayed responses. */
+#define VIRTIO_SCMI_F_P2A_CHANNELS 0
+
+/* Device implements any SCMI statistics shared memory region */
+#define VIRTIO_SCMI_F_SHARED_MEMORY 1
+
+/* Virtqueues */
+
+#define VIRTIO_SCMI_VQ_TX 0 /* cmdq */
+#define VIRTIO_SCMI_VQ_RX 1 /* eventq */
+#define VIRTIO_SCMI_VQ_MAX_CNT 2
+
+#endif /* _UAPI_LINUX_VIRTIO_SCMI_H */
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 15/16] [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Draft changes to virtio-scmi to use new support for core delegated xfers
in an attempt to simplify the interactions between virtio-scmi transport
and the SCMI core transport layer.

TODO:
 - Polling is still not supported.
 - Probe/remove sequence still to be reviewed.
 - Concurrent or inverted reception of related responses and delayed
   responses is still not addressed.
   (it will be addressed in the SCMI core anyway most probably)

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |   5 +
 drivers/firmware/arm_scmi/msg.c    |  35 +++++
 drivers/firmware/arm_scmi/virtio.c | 212 +++++++++++++++--------------
 3 files changed, 152 insertions(+), 100 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index c143a449d278..22e5532fc698 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -428,6 +428,11 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
 void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
 			    size_t max_len, struct scmi_xfer *xfer);
 
+void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
+			   size_t max_len, struct scmi_xfer *xfer);
+void msg_fetch_raw_response(struct scmi_xfer *xfer);
+void msg_fetch_raw_notification(struct scmi_xfer *xfer);
+
 void scmi_notification_instance_data_set(const struct scmi_handle *handle,
 					 void *priv);
 void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
diff --git a/drivers/firmware/arm_scmi/msg.c b/drivers/firmware/arm_scmi/msg.c
index 8a2d3303d281..3ed3ad0961ae 100644
--- a/drivers/firmware/arm_scmi/msg.c
+++ b/drivers/firmware/arm_scmi/msg.c
@@ -74,6 +74,17 @@ u32 msg_read_header(struct scmi_msg_payld *msg)
 	return le32_to_cpu(msg->msg_header);
 }
 
+void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
+			   size_t max_len, struct scmi_xfer *xfer)
+{
+	xfer->rx_raw_len = min_t(size_t, max_len,
+				 msg_len >= sizeof(*msg) ?
+				 msg_len - sizeof(*msg) : 0);
+
+	/* Take a copy to the rx buffer.. */
+	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx_raw_len);
+}
+
 /**
  * msg_fetch_response() - Fetch response SCMI payload from transport SDU.
  *
@@ -94,6 +105,25 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
 	memcpy(xfer->rx.buf, &msg->msg_payload[1], xfer->rx.len);
 }
 
+void msg_fetch_raw_response(struct scmi_xfer *xfer)
+{
+	__le32 *msg_payload = xfer->rx.buf;
+
+	if (xfer->rx_raw_len < sizeof(xfer->hdr.status)) {
+		xfer->rx.len = 0;
+		return;
+	}
+
+	xfer->hdr.status = le32_to_cpu(msg_payload[0]);
+	/*
+	 * rx.len has been already pre-calculated by fetch_raw
+	 * for the whole payload including status, so shrink it
+	 */
+	xfer->rx.len = xfer->rx_raw_len - sizeof(xfer->hdr.status);
+	/* Carveout status 4-byte field */
+	memmove(xfer->rx.buf, &msg_payload[1], xfer->rx.len);
+}
+
 /**
  * msg_fetch_notification() - Fetch notification payload from transport SDU.
  *
@@ -111,3 +141,8 @@ void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
 	/* Take a copy to the rx buffer.. */
 	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx.len);
 }
+
+void msg_fetch_raw_notification(struct scmi_xfer *xfer)
+{
+	xfer->rx.len = xfer->rx_raw_len;
+}
diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
index 20972adf6dc7..4412bc590ca7 100644
--- a/drivers/firmware/arm_scmi/virtio.c
+++ b/drivers/firmware/arm_scmi/virtio.c
@@ -37,23 +37,24 @@
 /**
  * struct scmi_vio_channel - Transport channel information
  *
- * @lock: Protects access to all members except ready.
- * @ready_lock: Protects access to ready. If required, it must be taken before
- *              lock.
  * @vqueue: Associated virtqueue
  * @cinfo: SCMI Tx or Rx channel
  * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
  * @is_rx: Whether channel is an Rx channel
  * @ready: Whether transport user is ready to hear about channel
+ * @lock: Protects access to all members except ready.
+ * @ready_lock: Protects access to ready. If required, it must be taken before
+ *              lock.
  */
 struct scmi_vio_channel {
-	spinlock_t lock;
-	spinlock_t ready_lock;
 	struct virtqueue *vqueue;
 	struct scmi_chan_info *cinfo;
 	struct list_head free_list;
-	u8 is_rx;
-	u8 ready;
+	bool is_rx;
+	bool ready;
+	unsigned int max_msg;
+	spinlock_t lock;
+	spinlock_t ready_lock;
 };
 
 /**
@@ -100,6 +101,73 @@ static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
 	return rc;
 }
 
+static void scmi_finalize_message(struct scmi_vio_channel *vioch,
+				  struct scmi_vio_msg *msg)
+{
+	unsigned long flags;
+
+	if (vioch->is_rx) {
+		scmi_vio_feed_vq_rx(vioch, msg);
+	} else {
+		spin_lock_irqsave(&vioch->lock, flags);
+		list_add(&msg->list, &vioch->free_list);
+		spin_unlock_irqrestore(&vioch->lock, flags);
+	}
+}
+
+static void scmi_process_vqueue_input(struct scmi_vio_channel *vioch,
+				      struct scmi_vio_msg *msg)
+{
+	u32 msg_hdr;
+	int ret;
+	struct scmi_xfer *xfer = NULL;
+
+	msg_hdr = msg_read_header(msg->input);
+	/*
+	 * Acquire from the core transport layer a currently valid xfer
+	 * descriptor associated to the received msg_hdr: this could be a
+	 * previously allocated xfer for responses and delayed responses to
+	 * in-flight commands, or a freshly allocated new xfer for a just
+	 * received notification.
+	 *
+	 * In case of responses and delayed_responses the acquired xfer, at
+	 * the time scmi_transfer_acquire() succcessfully returns is guaranteed
+	 * to be still associated with a valid (not timed-out nor stale)
+	 * descriptor and proper refcounting is kept in the core along this xfer
+	 * so that should the core time out the xfer concurrently to this receive
+	 * path the xfer will be properly deallocated only once the last user is
+	 * done with it. (and this code path will terminate normally even though
+	 * all the processing related to the timed out xfer will be discarded).
+	 */
+	ret = scmi_transfer_acquire(vioch->cinfo, &msg_hdr, &xfer);
+	if (ret) {
+		dev_err(vioch->cinfo->dev,
+			"Cannot find matching xfer for hdr:0x%X\n", msg_hdr);
+		scmi_finalize_message(vioch, msg);
+		return;
+	}
+
+	dev_dbg(vioch->cinfo->dev,
+		"VQUEUE[%d] - INPUT MSG_RX_LEN:%d - HDR:0x%X  TYPE:%d  XFER_ID:%d  XFER:%px\n",
+		vioch->vqueue->index, msg->rx_len, msg_hdr, xfer->hdr.type,
+		xfer->hdr.seq, xfer);
+
+	msg_fetch_raw_payload(msg->input, msg->rx_len,
+			      scmi_virtio_desc.max_msg_size, xfer);
+
+	/* Drop processed virtio message anyway */
+	scmi_finalize_message(vioch, msg);
+
+	if (vioch->is_rx || !xfer->hdr.poll_completion)
+		scmi_rx_callback(vioch->cinfo, msg_hdr);
+	else
+		dev_warn(vioch->cinfo->dev,
+			 "Polling mode NOT supported. Dropped hdr:0X%X\n",
+			 msg_hdr);
+
+	scmi_transfer_release(vioch->cinfo, xfer);
+}
+
 static void scmi_vio_complete_cb(struct virtqueue *vqueue)
 {
 	unsigned long ready_flags;
@@ -138,15 +206,9 @@ static void scmi_vio_complete_cb(struct virtqueue *vqueue)
 
 		if (msg) {
 			msg->rx_len = length;
-
-			/*
-			 * Hold the ready_lock during the callback to avoid
-			 * races when the arm-scmi driver is unbinding while
-			 * the virtio device is not quiesced yet.
-			 */
-			scmi_rx_callback(vioch->cinfo,
-					 msg_read_header(msg->input), msg);
+			scmi_process_vqueue_input(vioch, msg);
 		}
+
 		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
 	}
 
@@ -163,27 +225,11 @@ static vq_callback_t *scmi_vio_complete_callbacks[] = {
 	scmi_vio_complete_cb
 };
 
-static unsigned int virtio_get_max_msg(bool tx,
-				       struct scmi_chan_info *base_cinfo)
+static unsigned int virtio_get_max_msg(struct scmi_chan_info *base_cinfo)
 {
 	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
-	unsigned int ret;
 
-	ret = virtqueue_get_vring_size(vioch->vqueue);
-
-	/* Tx messages need multiple descriptors. */
-	if (tx)
-		ret /= DESCRIPTORS_PER_TX_MSG;
-
-	if (ret > MSG_TOKEN_MAX) {
-		dev_info_once(
-			base_cinfo->dev,
-			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
-			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
-		ret = MSG_TOKEN_MAX;
-	}
-
-	return ret;
+	return vioch->max_msg;
 }
 
 static int scmi_vio_match_any_dev(struct device *dev, const void *data)
@@ -195,13 +241,14 @@ static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
 
 static int virtio_link_supplier(struct device *dev)
 {
-	struct device *vdev = driver_find_device(
-		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
+	struct device *vdev;
+
+	vdev = driver_find_device(&virtio_scmi_driver.driver,
+				  NULL, NULL, scmi_vio_match_any_dev);
 
 	if (!vdev) {
-		dev_notice_once(
-			dev,
-			"Deferring probe after not finding a bound scmi-virtio device\n");
+		dev_notice_once(dev,
+				"Deferring probe after not finding a bound scmi-virtio device\n");
 		return -EPROBE_DEFER;
 	}
 
@@ -245,12 +292,8 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
 	struct virtio_device *vdev;
 	struct scmi_vio_channel *vioch;
 	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
-	int max_msg;
 	int i;
 
-	if (!virtio_chan_available(dev, index))
-		return -ENODEV;
-
 	vdev = scmi_get_transport_info(dev);
 	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
 
@@ -259,9 +302,7 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
 	vioch->cinfo = cinfo;
 	spin_unlock_irqrestore(&vioch->lock, flags);
 
-	max_msg = virtio_get_max_msg(tx, cinfo);
-
-	for (i = 0; i < max_msg; i++) {
+	for (i = 0; i < vioch->max_msg; i++) {
 		struct scmi_vio_msg *msg;
 
 		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
@@ -322,13 +363,6 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
 	int rc;
 	struct scmi_vio_msg *msg;
 
-	/*
-	 * TODO: For now, we don't support polling. But it should not be
-	 * difficult to add support.
-	 */
-	if (xfer->hdr.poll_completion)
-		return -EINVAL;
-
 	spin_lock_irqsave(&vioch->lock, flags);
 
 	if (list_empty(&vioch->free_list)) {
@@ -351,6 +385,11 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
 			     "%s() failed to add to virtqueue (%d)\n", __func__,
 			     rc);
 	} else {
+		dev_dbg(vioch->cinfo->dev,
+			"VQUEUE[%d] - REQUEST - PROTO:0x%X  ID:0x%X  XFER_ID:%d  XFER:%px  RX_LEN:%zd\n",
+		 vioch->vqueue->index, xfer->hdr.protocol_id,
+		 xfer->hdr.id, xfer->hdr.seq, xfer, xfer->rx.len);
+
 		virtqueue_kick(vioch->vqueue);
 	}
 
@@ -360,36 +399,15 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
 }
 
 static void virtio_fetch_response(struct scmi_chan_info *cinfo,
-				  struct scmi_xfer *xfer, void *msg_handle)
+				  struct scmi_xfer *xfer)
 {
-	struct scmi_vio_msg *msg = msg_handle;
-	struct scmi_vio_channel *vioch = cinfo->transport_info;
-
-	if (!msg) {
-		dev_dbg_once(&vioch->vqueue->vdev->dev,
-			     "Ignoring %s() call with NULL msg_handle\n",
-			     __func__);
-		return;
-	}
-
-	msg_fetch_response(msg->input, msg->rx_len, xfer);
+	msg_fetch_raw_response(xfer);
 }
 
 static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
-				      size_t max_len, struct scmi_xfer *xfer,
-				      void *msg_handle)
+				      size_t max_len, struct scmi_xfer *xfer)
 {
-	struct scmi_vio_msg *msg = msg_handle;
-	struct scmi_vio_channel *vioch = cinfo->transport_info;
-
-	if (!msg) {
-		dev_dbg_once(&vioch->vqueue->vdev->dev,
-			     "Ignoring %s() call with NULL msg_handle\n",
-			     __func__);
-		return;
-	}
-
-	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
+	msg_fetch_raw_notification(xfer);
 }
 
 static void dummy_clear_channel(struct scmi_chan_info *cinfo)
@@ -402,28 +420,6 @@ static bool dummy_poll_done(struct scmi_chan_info *cinfo,
 	return false;
 }
 
-static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
-{
-	unsigned long flags;
-	struct scmi_vio_channel *vioch = cinfo->transport_info;
-	struct scmi_vio_msg *msg = msg_handle;
-
-	if (!msg) {
-		dev_dbg_once(&vioch->vqueue->vdev->dev,
-			     "Ignoring %s() call with NULL msg_handle\n",
-			     __func__);
-		return;
-	}
-
-	if (vioch->is_rx) {
-		scmi_vio_feed_vq_rx(vioch, msg);
-	} else {
-		spin_lock_irqsave(&vioch->lock, flags);
-		list_add(&msg->list, &vioch->free_list);
-		spin_unlock_irqrestore(&vioch->lock, flags);
-	}
-}
-
 static const struct scmi_transport_ops scmi_virtio_ops = {
 	.link_supplier = virtio_link_supplier,
 	.chan_available = virtio_chan_available,
@@ -435,7 +431,6 @@ static const struct scmi_transport_ops scmi_virtio_ops = {
 	.fetch_notification = virtio_fetch_notification,
 	.clear_channel = dummy_clear_channel,
 	.poll_done = dummy_poll_done,
-	.drop_message = virtio_drop_message,
 };
 
 static int scmi_vio_probe(struct virtio_device *vdev)
@@ -467,10 +462,26 @@ static int scmi_vio_probe(struct virtio_device *vdev)
 	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);
 
 	for (i = 0; i < vq_cnt; i++) {
+		unsigned int sz;
+
 		spin_lock_init(&channels[i].lock);
 		spin_lock_init(&channels[i].ready_lock);
 		INIT_LIST_HEAD(&channels[i].free_list);
 		channels[i].vqueue = vqs[i];
+
+		sz = virtqueue_get_vring_size(channels[i].vqueue);
+		/* Tx messages need multiple descriptors. */
+		if (!channels[i].is_rx)
+			sz /= DESCRIPTORS_PER_TX_MSG;
+
+		if (sz > MSG_TOKEN_MAX) {
+			dev_info_once(dev,
+				      "%s virtqueue could hold %d messages. Only %ld allowed to be pending.\n",
+				      channels[i].is_rx ? "rx" : "tx",
+				      sz, MSG_TOKEN_MAX);
+			sz = MSG_TOKEN_MAX;
+		}
+		channels[i].max_msg = sz;
 	}
 
 	vdev->priv = channels;
@@ -520,4 +531,5 @@ const struct scmi_desc scmi_virtio_desc = {
 	.max_rx_timeout_ms = 60000, /* for non-realtime virtio devices */
 	.max_msg = 0, /* overridden by virtio_get_max_msg() */
 	.max_msg_size = VIRTIO_SCMI_MAX_MSG_SIZE,
+	.support_xfers_delegation = true,
 };
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 15/16] [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Draft changes to virtio-scmi to use new support for core delegated xfers
in an attempt to simplify the interactions between virtio-scmi transport
and the SCMI core transport layer.

TODO:
 - Polling is still not supported.
 - Probe/remove sequence still to be reviewed.
 - Concurrent or inverted reception of related responses and delayed
   responses is still not addressed.
   (it will be addressed in the SCMI core anyway most probably)

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/common.h |   5 +
 drivers/firmware/arm_scmi/msg.c    |  35 +++++
 drivers/firmware/arm_scmi/virtio.c | 212 +++++++++++++++--------------
 3 files changed, 152 insertions(+), 100 deletions(-)

diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index c143a449d278..22e5532fc698 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -428,6 +428,11 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
 void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
 			    size_t max_len, struct scmi_xfer *xfer);
 
+void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
+			   size_t max_len, struct scmi_xfer *xfer);
+void msg_fetch_raw_response(struct scmi_xfer *xfer);
+void msg_fetch_raw_notification(struct scmi_xfer *xfer);
+
 void scmi_notification_instance_data_set(const struct scmi_handle *handle,
 					 void *priv);
 void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
diff --git a/drivers/firmware/arm_scmi/msg.c b/drivers/firmware/arm_scmi/msg.c
index 8a2d3303d281..3ed3ad0961ae 100644
--- a/drivers/firmware/arm_scmi/msg.c
+++ b/drivers/firmware/arm_scmi/msg.c
@@ -74,6 +74,17 @@ u32 msg_read_header(struct scmi_msg_payld *msg)
 	return le32_to_cpu(msg->msg_header);
 }
 
+void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
+			   size_t max_len, struct scmi_xfer *xfer)
+{
+	xfer->rx_raw_len = min_t(size_t, max_len,
+				 msg_len >= sizeof(*msg) ?
+				 msg_len - sizeof(*msg) : 0);
+
+	/* Take a copy to the rx buffer.. */
+	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx_raw_len);
+}
+
 /**
  * msg_fetch_response() - Fetch response SCMI payload from transport SDU.
  *
@@ -94,6 +105,25 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
 	memcpy(xfer->rx.buf, &msg->msg_payload[1], xfer->rx.len);
 }
 
+void msg_fetch_raw_response(struct scmi_xfer *xfer)
+{
+	__le32 *msg_payload = xfer->rx.buf;
+
+	if (xfer->rx_raw_len < sizeof(xfer->hdr.status)) {
+		xfer->rx.len = 0;
+		return;
+	}
+
+	xfer->hdr.status = le32_to_cpu(msg_payload[0]);
+	/*
+	 * rx.len has been already pre-calculated by fetch_raw
+	 * for the whole payload including status, so shrink it
+	 */
+	xfer->rx.len = xfer->rx_raw_len - sizeof(xfer->hdr.status);
+	/* Carveout status 4-byte field */
+	memmove(xfer->rx.buf, &msg_payload[1], xfer->rx.len);
+}
+
 /**
  * msg_fetch_notification() - Fetch notification payload from transport SDU.
  *
@@ -111,3 +141,8 @@ void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
 	/* Take a copy to the rx buffer.. */
 	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx.len);
 }
+
+void msg_fetch_raw_notification(struct scmi_xfer *xfer)
+{
+	xfer->rx.len = xfer->rx_raw_len;
+}
diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
index 20972adf6dc7..4412bc590ca7 100644
--- a/drivers/firmware/arm_scmi/virtio.c
+++ b/drivers/firmware/arm_scmi/virtio.c
@@ -37,23 +37,24 @@
 /**
  * struct scmi_vio_channel - Transport channel information
  *
- * @lock: Protects access to all members except ready.
- * @ready_lock: Protects access to ready. If required, it must be taken before
- *              lock.
  * @vqueue: Associated virtqueue
  * @cinfo: SCMI Tx or Rx channel
  * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
  * @is_rx: Whether channel is an Rx channel
  * @ready: Whether transport user is ready to hear about channel
+ * @lock: Protects access to all members except ready.
+ * @ready_lock: Protects access to ready. If required, it must be taken before
+ *              lock.
  */
 struct scmi_vio_channel {
-	spinlock_t lock;
-	spinlock_t ready_lock;
 	struct virtqueue *vqueue;
 	struct scmi_chan_info *cinfo;
 	struct list_head free_list;
-	u8 is_rx;
-	u8 ready;
+	bool is_rx;
+	bool ready;
+	unsigned int max_msg;
+	spinlock_t lock;
+	spinlock_t ready_lock;
 };
 
 /**
@@ -100,6 +101,73 @@ static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
 	return rc;
 }
 
+static void scmi_finalize_message(struct scmi_vio_channel *vioch,
+				  struct scmi_vio_msg *msg)
+{
+	unsigned long flags;
+
+	if (vioch->is_rx) {
+		scmi_vio_feed_vq_rx(vioch, msg);
+	} else {
+		spin_lock_irqsave(&vioch->lock, flags);
+		list_add(&msg->list, &vioch->free_list);
+		spin_unlock_irqrestore(&vioch->lock, flags);
+	}
+}
+
+static void scmi_process_vqueue_input(struct scmi_vio_channel *vioch,
+				      struct scmi_vio_msg *msg)
+{
+	u32 msg_hdr;
+	int ret;
+	struct scmi_xfer *xfer = NULL;
+
+	msg_hdr = msg_read_header(msg->input);
+	/*
+	 * Acquire from the core transport layer a currently valid xfer
+	 * descriptor associated to the received msg_hdr: this could be a
+	 * previously allocated xfer for responses and delayed responses to
+	 * in-flight commands, or a freshly allocated new xfer for a just
+	 * received notification.
+	 *
+	 * In case of responses and delayed_responses the acquired xfer, at
+	 * the time scmi_transfer_acquire() succcessfully returns is guaranteed
+	 * to be still associated with a valid (not timed-out nor stale)
+	 * descriptor and proper refcounting is kept in the core along this xfer
+	 * so that should the core time out the xfer concurrently to this receive
+	 * path the xfer will be properly deallocated only once the last user is
+	 * done with it. (and this code path will terminate normally even though
+	 * all the processing related to the timed out xfer will be discarded).
+	 */
+	ret = scmi_transfer_acquire(vioch->cinfo, &msg_hdr, &xfer);
+	if (ret) {
+		dev_err(vioch->cinfo->dev,
+			"Cannot find matching xfer for hdr:0x%X\n", msg_hdr);
+		scmi_finalize_message(vioch, msg);
+		return;
+	}
+
+	dev_dbg(vioch->cinfo->dev,
+		"VQUEUE[%d] - INPUT MSG_RX_LEN:%d - HDR:0x%X  TYPE:%d  XFER_ID:%d  XFER:%px\n",
+		vioch->vqueue->index, msg->rx_len, msg_hdr, xfer->hdr.type,
+		xfer->hdr.seq, xfer);
+
+	msg_fetch_raw_payload(msg->input, msg->rx_len,
+			      scmi_virtio_desc.max_msg_size, xfer);
+
+	/* Drop processed virtio message anyway */
+	scmi_finalize_message(vioch, msg);
+
+	if (vioch->is_rx || !xfer->hdr.poll_completion)
+		scmi_rx_callback(vioch->cinfo, msg_hdr);
+	else
+		dev_warn(vioch->cinfo->dev,
+			 "Polling mode NOT supported. Dropped hdr:0X%X\n",
+			 msg_hdr);
+
+	scmi_transfer_release(vioch->cinfo, xfer);
+}
+
 static void scmi_vio_complete_cb(struct virtqueue *vqueue)
 {
 	unsigned long ready_flags;
@@ -138,15 +206,9 @@ static void scmi_vio_complete_cb(struct virtqueue *vqueue)
 
 		if (msg) {
 			msg->rx_len = length;
-
-			/*
-			 * Hold the ready_lock during the callback to avoid
-			 * races when the arm-scmi driver is unbinding while
-			 * the virtio device is not quiesced yet.
-			 */
-			scmi_rx_callback(vioch->cinfo,
-					 msg_read_header(msg->input), msg);
+			scmi_process_vqueue_input(vioch, msg);
 		}
+
 		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
 	}
 
@@ -163,27 +225,11 @@ static vq_callback_t *scmi_vio_complete_callbacks[] = {
 	scmi_vio_complete_cb
 };
 
-static unsigned int virtio_get_max_msg(bool tx,
-				       struct scmi_chan_info *base_cinfo)
+static unsigned int virtio_get_max_msg(struct scmi_chan_info *base_cinfo)
 {
 	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
-	unsigned int ret;
 
-	ret = virtqueue_get_vring_size(vioch->vqueue);
-
-	/* Tx messages need multiple descriptors. */
-	if (tx)
-		ret /= DESCRIPTORS_PER_TX_MSG;
-
-	if (ret > MSG_TOKEN_MAX) {
-		dev_info_once(
-			base_cinfo->dev,
-			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
-			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
-		ret = MSG_TOKEN_MAX;
-	}
-
-	return ret;
+	return vioch->max_msg;
 }
 
 static int scmi_vio_match_any_dev(struct device *dev, const void *data)
@@ -195,13 +241,14 @@ static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
 
 static int virtio_link_supplier(struct device *dev)
 {
-	struct device *vdev = driver_find_device(
-		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
+	struct device *vdev;
+
+	vdev = driver_find_device(&virtio_scmi_driver.driver,
+				  NULL, NULL, scmi_vio_match_any_dev);
 
 	if (!vdev) {
-		dev_notice_once(
-			dev,
-			"Deferring probe after not finding a bound scmi-virtio device\n");
+		dev_notice_once(dev,
+				"Deferring probe after not finding a bound scmi-virtio device\n");
 		return -EPROBE_DEFER;
 	}
 
@@ -245,12 +292,8 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
 	struct virtio_device *vdev;
 	struct scmi_vio_channel *vioch;
 	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
-	int max_msg;
 	int i;
 
-	if (!virtio_chan_available(dev, index))
-		return -ENODEV;
-
 	vdev = scmi_get_transport_info(dev);
 	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
 
@@ -259,9 +302,7 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
 	vioch->cinfo = cinfo;
 	spin_unlock_irqrestore(&vioch->lock, flags);
 
-	max_msg = virtio_get_max_msg(tx, cinfo);
-
-	for (i = 0; i < max_msg; i++) {
+	for (i = 0; i < vioch->max_msg; i++) {
 		struct scmi_vio_msg *msg;
 
 		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
@@ -322,13 +363,6 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
 	int rc;
 	struct scmi_vio_msg *msg;
 
-	/*
-	 * TODO: For now, we don't support polling. But it should not be
-	 * difficult to add support.
-	 */
-	if (xfer->hdr.poll_completion)
-		return -EINVAL;
-
 	spin_lock_irqsave(&vioch->lock, flags);
 
 	if (list_empty(&vioch->free_list)) {
@@ -351,6 +385,11 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
 			     "%s() failed to add to virtqueue (%d)\n", __func__,
 			     rc);
 	} else {
+		dev_dbg(vioch->cinfo->dev,
+			"VQUEUE[%d] - REQUEST - PROTO:0x%X  ID:0x%X  XFER_ID:%d  XFER:%px  RX_LEN:%zd\n",
+		 vioch->vqueue->index, xfer->hdr.protocol_id,
+		 xfer->hdr.id, xfer->hdr.seq, xfer, xfer->rx.len);
+
 		virtqueue_kick(vioch->vqueue);
 	}
 
@@ -360,36 +399,15 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
 }
 
 static void virtio_fetch_response(struct scmi_chan_info *cinfo,
-				  struct scmi_xfer *xfer, void *msg_handle)
+				  struct scmi_xfer *xfer)
 {
-	struct scmi_vio_msg *msg = msg_handle;
-	struct scmi_vio_channel *vioch = cinfo->transport_info;
-
-	if (!msg) {
-		dev_dbg_once(&vioch->vqueue->vdev->dev,
-			     "Ignoring %s() call with NULL msg_handle\n",
-			     __func__);
-		return;
-	}
-
-	msg_fetch_response(msg->input, msg->rx_len, xfer);
+	msg_fetch_raw_response(xfer);
 }
 
 static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
-				      size_t max_len, struct scmi_xfer *xfer,
-				      void *msg_handle)
+				      size_t max_len, struct scmi_xfer *xfer)
 {
-	struct scmi_vio_msg *msg = msg_handle;
-	struct scmi_vio_channel *vioch = cinfo->transport_info;
-
-	if (!msg) {
-		dev_dbg_once(&vioch->vqueue->vdev->dev,
-			     "Ignoring %s() call with NULL msg_handle\n",
-			     __func__);
-		return;
-	}
-
-	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
+	msg_fetch_raw_notification(xfer);
 }
 
 static void dummy_clear_channel(struct scmi_chan_info *cinfo)
@@ -402,28 +420,6 @@ static bool dummy_poll_done(struct scmi_chan_info *cinfo,
 	return false;
 }
 
-static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
-{
-	unsigned long flags;
-	struct scmi_vio_channel *vioch = cinfo->transport_info;
-	struct scmi_vio_msg *msg = msg_handle;
-
-	if (!msg) {
-		dev_dbg_once(&vioch->vqueue->vdev->dev,
-			     "Ignoring %s() call with NULL msg_handle\n",
-			     __func__);
-		return;
-	}
-
-	if (vioch->is_rx) {
-		scmi_vio_feed_vq_rx(vioch, msg);
-	} else {
-		spin_lock_irqsave(&vioch->lock, flags);
-		list_add(&msg->list, &vioch->free_list);
-		spin_unlock_irqrestore(&vioch->lock, flags);
-	}
-}
-
 static const struct scmi_transport_ops scmi_virtio_ops = {
 	.link_supplier = virtio_link_supplier,
 	.chan_available = virtio_chan_available,
@@ -435,7 +431,6 @@ static const struct scmi_transport_ops scmi_virtio_ops = {
 	.fetch_notification = virtio_fetch_notification,
 	.clear_channel = dummy_clear_channel,
 	.poll_done = dummy_poll_done,
-	.drop_message = virtio_drop_message,
 };
 
 static int scmi_vio_probe(struct virtio_device *vdev)
@@ -467,10 +462,26 @@ static int scmi_vio_probe(struct virtio_device *vdev)
 	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);
 
 	for (i = 0; i < vq_cnt; i++) {
+		unsigned int sz;
+
 		spin_lock_init(&channels[i].lock);
 		spin_lock_init(&channels[i].ready_lock);
 		INIT_LIST_HEAD(&channels[i].free_list);
 		channels[i].vqueue = vqs[i];
+
+		sz = virtqueue_get_vring_size(channels[i].vqueue);
+		/* Tx messages need multiple descriptors. */
+		if (!channels[i].is_rx)
+			sz /= DESCRIPTORS_PER_TX_MSG;
+
+		if (sz > MSG_TOKEN_MAX) {
+			dev_info_once(dev,
+				      "%s virtqueue could hold %d messages. Only %ld allowed to be pending.\n",
+				      channels[i].is_rx ? "rx" : "tx",
+				      sz, MSG_TOKEN_MAX);
+			sz = MSG_TOKEN_MAX;
+		}
+		channels[i].max_msg = sz;
 	}
 
 	vdev->priv = channels;
@@ -520,4 +531,5 @@ const struct scmi_desc scmi_virtio_desc = {
 	.max_rx_timeout_ms = 60000, /* for non-realtime virtio devices */
 	.max_msg = 0, /* overridden by virtio_get_max_msg() */
 	.max_msg_size = VIRTIO_SCMI_MAX_MSG_SIZE,
+	.support_xfers_delegation = true,
 };
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 16/16] firmware: arm_scmi: Add polling mode to virtio transport
  2021-06-11 16:59 ` Cristian Marussi
@ 2021-06-11 16:59   ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Enable polling mode support to virtio transport.

Upon reception of a synchronous command response for a transfer that has
the hdr.poll_completion flag set, virtio transport now simply completes the
same completion which .poll_done() is spinning on.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
Note that as of now in order to test this you have to forcibly
enable hdr.poll_completion in the core and increase SCMI_MAX_POLL_TO_NS.

These workarounds won't be needed anymore once addressed by a distinct
series currently under review.
---
 drivers/firmware/arm_scmi/virtio.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
index 4412bc590ca7..f77c7f288d4c 100644
--- a/drivers/firmware/arm_scmi/virtio.c
+++ b/drivers/firmware/arm_scmi/virtio.c
@@ -158,12 +158,12 @@ static void scmi_process_vqueue_input(struct scmi_vio_channel *vioch,
 	/* Drop processed virtio message anyway */
 	scmi_finalize_message(vioch, msg);
 
+	/* Deliver DRESP, NOTIF and non-polled RESP */
 	if (vioch->is_rx || !xfer->hdr.poll_completion)
 		scmi_rx_callback(vioch->cinfo, msg_hdr);
 	else
-		dev_warn(vioch->cinfo->dev,
-			 "Polling mode NOT supported. Dropped hdr:0X%X\n",
-			 msg_hdr);
+		/* poll_done() is busy-waiting on this */
+		complete(&xfer->done);
 
 	scmi_transfer_release(vioch->cinfo, xfer);
 }
@@ -414,10 +414,16 @@ static void dummy_clear_channel(struct scmi_chan_info *cinfo)
 {
 }
 
-static bool dummy_poll_done(struct scmi_chan_info *cinfo,
-			    struct scmi_xfer *xfer)
+static bool virtio_poll_done(struct scmi_chan_info *cinfo,
+			     struct scmi_xfer *xfer)
 {
-	return false;
+	/*
+	 * In polling mode SCMI core does not use xfer->done completion,
+	 * so we can busy-wait on this same completion without adding
+	 * a new flag: this is completed properly upon msg reception in
+	 * scmi_process_vqueue_input().
+	 */
+	return try_wait_for_completion(&xfer->done);
 }
 
 static const struct scmi_transport_ops scmi_virtio_ops = {
@@ -430,7 +436,7 @@ static const struct scmi_transport_ops scmi_virtio_ops = {
 	.fetch_response = virtio_fetch_response,
 	.fetch_notification = virtio_fetch_notification,
 	.clear_channel = dummy_clear_channel,
-	.poll_done = dummy_poll_done,
+	.poll_done = virtio_poll_done,
 };
 
 static int scmi_vio_probe(struct virtio_device *vdev)
-- 
2.17.1


^ permalink raw reply	[flat|nested] 90+ messages in thread

* [PATCH v4 16/16] firmware: arm_scmi: Add polling mode to virtio transport
@ 2021-06-11 16:59   ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-11 16:59 UTC (permalink / raw)
  To: linux-kernel, linux-arm-kernel, virtualization, virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	cristian.marussi, igor.skalkin, peter.hilber, alex.bennee,
	jean-philippe, mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Enable polling mode support to virtio transport.

Upon reception of a synchronous command response for a transfer that has
the hdr.poll_completion flag set, virtio transport now simply completes the
same completion which .poll_done() is spinning on.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
Note that as of now in order to test this you have to forcibly
enable hdr.poll_completion in the core and increase SCMI_MAX_POLL_TO_NS.

These workarounds won't be needed anymore once addressed by a distinct
series currently under review.
---
 drivers/firmware/arm_scmi/virtio.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
index 4412bc590ca7..f77c7f288d4c 100644
--- a/drivers/firmware/arm_scmi/virtio.c
+++ b/drivers/firmware/arm_scmi/virtio.c
@@ -158,12 +158,12 @@ static void scmi_process_vqueue_input(struct scmi_vio_channel *vioch,
 	/* Drop processed virtio message anyway */
 	scmi_finalize_message(vioch, msg);
 
+	/* Deliver DRESP, NOTIF and non-polled RESP */
 	if (vioch->is_rx || !xfer->hdr.poll_completion)
 		scmi_rx_callback(vioch->cinfo, msg_hdr);
 	else
-		dev_warn(vioch->cinfo->dev,
-			 "Polling mode NOT supported. Dropped hdr:0X%X\n",
-			 msg_hdr);
+		/* poll_done() is busy-waiting on this */
+		complete(&xfer->done);
 
 	scmi_transfer_release(vioch->cinfo, xfer);
 }
@@ -414,10 +414,16 @@ static void dummy_clear_channel(struct scmi_chan_info *cinfo)
 {
 }
 
-static bool dummy_poll_done(struct scmi_chan_info *cinfo,
-			    struct scmi_xfer *xfer)
+static bool virtio_poll_done(struct scmi_chan_info *cinfo,
+			     struct scmi_xfer *xfer)
 {
-	return false;
+	/*
+	 * In polling mode SCMI core does not use xfer->done completion,
+	 * so we can busy-wait on this same completion without adding
+	 * a new flag: this is completed properly upon msg reception in
+	 * scmi_process_vqueue_input().
+	 */
+	return try_wait_for_completion(&xfer->done);
 }
 
 static const struct scmi_transport_ops scmi_virtio_ops = {
@@ -430,7 +436,7 @@ static const struct scmi_transport_ops scmi_virtio_ops = {
 	.fetch_response = virtio_fetch_response,
 	.fetch_notification = virtio_fetch_notification,
 	.clear_channel = dummy_clear_channel,
-	.poll_done = dummy_poll_done,
+	.poll_done = virtio_poll_done,
 };
 
 static int scmi_vio_probe(struct virtio_device *vdev)
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 00/16] Introduce SCMI VirtIO transport
  2021-06-11 16:59 ` Cristian Marussi
  (?)
@ 2021-06-14 11:43   ` Christoph Hellwig
  -1 siblings, 0 replies; 90+ messages in thread
From: Christoph Hellwig @ 2021-06-14 11:43 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, peter.hilber, alex.bennee, jean-philippe,
	mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

On Fri, Jun 11, 2021 at 05:59:21PM +0100, Cristian Marussi wrote:
> Hi all,
> 
> I'm posting this V4 series starting from the work done up to V3 by
> OpenSynergy.

Who is 'OpenSynergy'?

> The main aim of this rework is to simplify where possible the SCMI VirtIO
> support added in V3 by adding upfront and then using some new mechanisms in
> the SCMI Core Transport layer.

And what is 'SCMI', and why would anyone want a new virtio transport?

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 00/16] Introduce SCMI VirtIO transport
@ 2021-06-14 11:43   ` Christoph Hellwig
  0 siblings, 0 replies; 90+ messages in thread
From: Christoph Hellwig @ 2021-06-14 11:43 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: virtio-dev, mikhail.golubev, f.fainelli, vincent.guittot,
	igor.skalkin, jean-philippe, Jonathan.Cameron, linux-kernel,
	virtualization, Vasyl.Vavrychuk, peter.hilber, james.quinlan,
	sudeep.holla, souvik.chakravarty, etienne.carriere,
	linux-arm-kernel, Andriy.Tryshnivskyy

On Fri, Jun 11, 2021 at 05:59:21PM +0100, Cristian Marussi wrote:
> Hi all,
> 
> I'm posting this V4 series starting from the work done up to V3 by
> OpenSynergy.

Who is 'OpenSynergy'?

> The main aim of this rework is to simplify where possible the SCMI VirtIO
> support added in V3 by adding upfront and then using some new mechanisms in
> the SCMI Core Transport layer.

And what is 'SCMI', and why would anyone want a new virtio transport?
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 00/16] Introduce SCMI VirtIO transport
@ 2021-06-14 11:43   ` Christoph Hellwig
  0 siblings, 0 replies; 90+ messages in thread
From: Christoph Hellwig @ 2021-06-14 11:43 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, peter.hilber, alex.bennee, jean-philippe,
	mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

On Fri, Jun 11, 2021 at 05:59:21PM +0100, Cristian Marussi wrote:
> Hi all,
> 
> I'm posting this V4 series starting from the work done up to V3 by
> OpenSynergy.

Who is 'OpenSynergy'?

> The main aim of this rework is to simplify where possible the SCMI VirtIO
> support added in V3 by adding upfront and then using some new mechanisms in
> the SCMI Core Transport layer.

And what is 'SCMI', and why would anyone want a new virtio transport?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 03/16] firmware: arm_scmi: Add transport optional init/exit support
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-06-14 13:29     ` Jonathan Cameron
  -1 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 13:29 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:24 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> Some SCMI transport could need to perform some transport specific setup
> before they can be used by the SCMI core transport layer: typically this
> early setup consists in registering with some other kernel subsystem.
> 
> Add the optional capability for a transport to provide a couple of .init
> and .exit functions that are assured to be called early during the SCMI
> core initialization phase, well before the SCMI core probing step.
> 
> [ Peter: Adapted RFC patch by Cristian for submission to upstream. ]
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> [ Cristian: Fixed scmi_transports_exit point of invocation ]
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>

Drive by comment inline.  Feel free to ignore ;)

Jonathan

> ---
>  drivers/firmware/arm_scmi/common.h |  8 ++++
>  drivers/firmware/arm_scmi/driver.c | 59 ++++++++++++++++++++++++++++++
>  2 files changed, 67 insertions(+)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index 7c2b9fd7e929..6bb734e0e3ac 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -321,6 +321,12 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
>  /**
>   * struct scmi_desc - Description of SoC integration
>   *
> + * @init: An optional function that a transport can provide to initialize some
> + *	  transport-specific setup during SCMI core initialization, so ahead of
> + *	  SCMI core probing.
> + * @exit: An optional function that a transport can provide to de-initialize
> + *	  some transport-specific setup during SCMI core de-initialization, so
> + *	  after SCMI core removal.
>   * @ops: Pointer to the transport specific ops structure
>   * @max_rx_timeout_ms: Timeout for communication with SoC (in Milliseconds)
>   * @max_msg: Maximum number of messages that can be pending
> @@ -328,6 +334,8 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
>   * @max_msg_size: Maximum size of data per message that can be handled.
>   */
>  struct scmi_desc {
> +	int (*init)(void);
> +	void (*exit)(void);
>  	const struct scmi_transport_ops *ops;
>  	int max_rx_timeout_ms;
>  	int max_msg;
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index f15d75af87ea..20f8f0581f3a 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -1594,10 +1594,67 @@ static struct platform_driver scmi_driver = {
>  	.remove = scmi_remove,
>  };
>  
> +/**
> + * __scmi_transports_setup  - Common helper to call transport-specific
> + * .init/.exit code if provided.
> + *
> + * @init: A flag to distinguish between init and exit.
> + *
> + * Note that, if provided, we invoke .init/.exit functions for all the
> + * transports currently compiled in.
> + *
> + * Return: 0 on Success.
> + */
> +static inline int __scmi_transports_setup(bool init)
> +{
> +	int ret = 0;
> +	const struct of_device_id *trans;
> +
> +	for (trans = scmi_of_match; trans->data; trans++) {
> +		const struct scmi_desc *tdesc = trans->data;
> +
> +		if ((init && !tdesc->init) || (!init && !tdesc->exit))
> +			continue;
> +
> +		pr_debug("SCMI %sInitializing %s transport\n",
> +			 init ? "" : "De-", trans->compatible);

Clever formatting can makes grepping for messages harder.

Perhaps
		if (init)
			pr_debug("SCMI Initializing %s transport\n",
				 trans->compatible);
		else
			pr_debug("SCMI Deinitializing %s transport\n",
				 trans->compatible);

would be nicer even though it burns some lines. Also avoids somewhat
ugly capitalization : De-Initializing xxx transport.

You could combine it with the convenient if(init) below

> +
> +		if (init)
> +			ret = tdesc->init();
> +		else
> +			tdesc->exit();
> +
> +		if (ret) {
> +			pr_err("SCMI transport %s FAILED initialization!\n",
> +			       trans->compatible);
> +			break;
> +		}
> +	}
> +
> +	return ret;
> +}
> +
> +static int __init scmi_transports_init(void)
> +{
> +	return __scmi_transports_setup(true);
> +}
> +
> +static void __exit scmi_transports_exit(void)
> +{
> +	__scmi_transports_setup(false);
> +}
> +
>  static int __init scmi_driver_init(void)
>  {
> +	int ret;
> +
>  	scmi_bus_init();
>  
> +	/* Initialize any compiled-in transport which provided an init/exit */
> +	ret = scmi_transports_init();
> +	if (ret)
> +		return ret;
> +
>  	scmi_base_register();
>  
>  	scmi_clock_register();
> @@ -1626,6 +1683,8 @@ static void __exit scmi_driver_exit(void)
>  
>  	scmi_bus_exit();
>  
> +	scmi_transports_exit();
> +
>  	platform_driver_unregister(&scmi_driver);
>  }
>  module_exit(scmi_driver_exit);


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 03/16] firmware: arm_scmi: Add transport optional init/exit support
@ 2021-06-14 13:29     ` Jonathan Cameron
  0 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 13:29 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:24 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> Some SCMI transport could need to perform some transport specific setup
> before they can be used by the SCMI core transport layer: typically this
> early setup consists in registering with some other kernel subsystem.
> 
> Add the optional capability for a transport to provide a couple of .init
> and .exit functions that are assured to be called early during the SCMI
> core initialization phase, well before the SCMI core probing step.
> 
> [ Peter: Adapted RFC patch by Cristian for submission to upstream. ]
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> [ Cristian: Fixed scmi_transports_exit point of invocation ]
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>

Drive by comment inline.  Feel free to ignore ;)

Jonathan

> ---
>  drivers/firmware/arm_scmi/common.h |  8 ++++
>  drivers/firmware/arm_scmi/driver.c | 59 ++++++++++++++++++++++++++++++
>  2 files changed, 67 insertions(+)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index 7c2b9fd7e929..6bb734e0e3ac 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -321,6 +321,12 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
>  /**
>   * struct scmi_desc - Description of SoC integration
>   *
> + * @init: An optional function that a transport can provide to initialize some
> + *	  transport-specific setup during SCMI core initialization, so ahead of
> + *	  SCMI core probing.
> + * @exit: An optional function that a transport can provide to de-initialize
> + *	  some transport-specific setup during SCMI core de-initialization, so
> + *	  after SCMI core removal.
>   * @ops: Pointer to the transport specific ops structure
>   * @max_rx_timeout_ms: Timeout for communication with SoC (in Milliseconds)
>   * @max_msg: Maximum number of messages that can be pending
> @@ -328,6 +334,8 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
>   * @max_msg_size: Maximum size of data per message that can be handled.
>   */
>  struct scmi_desc {
> +	int (*init)(void);
> +	void (*exit)(void);
>  	const struct scmi_transport_ops *ops;
>  	int max_rx_timeout_ms;
>  	int max_msg;
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index f15d75af87ea..20f8f0581f3a 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -1594,10 +1594,67 @@ static struct platform_driver scmi_driver = {
>  	.remove = scmi_remove,
>  };
>  
> +/**
> + * __scmi_transports_setup  - Common helper to call transport-specific
> + * .init/.exit code if provided.
> + *
> + * @init: A flag to distinguish between init and exit.
> + *
> + * Note that, if provided, we invoke .init/.exit functions for all the
> + * transports currently compiled in.
> + *
> + * Return: 0 on Success.
> + */
> +static inline int __scmi_transports_setup(bool init)
> +{
> +	int ret = 0;
> +	const struct of_device_id *trans;
> +
> +	for (trans = scmi_of_match; trans->data; trans++) {
> +		const struct scmi_desc *tdesc = trans->data;
> +
> +		if ((init && !tdesc->init) || (!init && !tdesc->exit))
> +			continue;
> +
> +		pr_debug("SCMI %sInitializing %s transport\n",
> +			 init ? "" : "De-", trans->compatible);

Clever formatting can makes grepping for messages harder.

Perhaps
		if (init)
			pr_debug("SCMI Initializing %s transport\n",
				 trans->compatible);
		else
			pr_debug("SCMI Deinitializing %s transport\n",
				 trans->compatible);

would be nicer even though it burns some lines. Also avoids somewhat
ugly capitalization : De-Initializing xxx transport.

You could combine it with the convenient if(init) below

> +
> +		if (init)
> +			ret = tdesc->init();
> +		else
> +			tdesc->exit();
> +
> +		if (ret) {
> +			pr_err("SCMI transport %s FAILED initialization!\n",
> +			       trans->compatible);
> +			break;
> +		}
> +	}
> +
> +	return ret;
> +}
> +
> +static int __init scmi_transports_init(void)
> +{
> +	return __scmi_transports_setup(true);
> +}
> +
> +static void __exit scmi_transports_exit(void)
> +{
> +	__scmi_transports_setup(false);
> +}
> +
>  static int __init scmi_driver_init(void)
>  {
> +	int ret;
> +
>  	scmi_bus_init();
>  
> +	/* Initialize any compiled-in transport which provided an init/exit */
> +	ret = scmi_transports_init();
> +	if (ret)
> +		return ret;
> +
>  	scmi_base_register();
>  
>  	scmi_clock_register();
> @@ -1626,6 +1683,8 @@ static void __exit scmi_driver_exit(void)
>  
>  	scmi_bus_exit();
>  
> +	scmi_transports_exit();
> +
>  	platform_driver_unregister(&scmi_driver);
>  }
>  module_exit(scmi_driver_exit);


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-06-14 13:53     ` Jonathan Cameron
  -1 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 13:53 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:25 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> Tokens are sequence numbers embedded in the each SCMI message header: they
> are used to correlate commands with responses (and delayed responses), but
> their usage and policy of selection is entirely up to the caller (usually
> the OSPM agent), while they are completely opaque to the callee (SCMI
> server platform) which merely copies them back from the command into the
> response message header.
> This also means that the platform does not, can not and should not enforce
> any kind of policy on received messages depending on the contained sequence
> number: platform can perfectly handle concurrent requests carrying the same
> identifiying token if that should happen.
> 
> Moreover the platform is not required to produce in-order responses to
> agent requests, the only constraint in these regards is that in case of
> an asynchronous message the delayed response must be sent after the
> immediate response for the synchronous part of the command transaction.
> 
> Currenly the SCMI stack of the OSPM agent selects a token for the egressing
> commands picking the lowest possible number which is not already in use by
> an existing in-flight transaction, which means, in other words, that we
> immediately reuse any token after its transaction has completed or it has
> timed out: this policy indeed does simplify management and lookup of tokens
> and associated xfers.
> 
> Under the above assumptions and constraints, since there is really no state
> shared between the agent and the platform to let the platform know when a
> token and its associated message has timed out, the current policy of early
> reuse of tokens can easily lead to the situation in which a spurious or
> late received response (or delayed_response), related to an old stale and
> timed out transaction, can be wrongly associated to a newer valid in-flight
> xfer that just happens to have reused the same token.
> 
> This misbehaviour on such ghost responses is more easily exposed on those
> transports that naturally have an higher level of parallelism in processing
> multiple concurrent in-flight messages.
> 
> This commit introduces a new policy of selection of tokens for the OSPM
> agent: each new transfer now gets the next available and monotonically
> increasing token, until tokens are exhausted and the counter rolls over.
> 
> Such new policy mitigates the above issues with ghost responses since the
> tokens are now reused as late as possible (when they roll back ideally)
> and so it is much easier to identify such ghost responses to stale timed
> out transactions: this also helps in simplifying the specific transports
> implementation since stale transport messages can be easily identified
> and discarded early on in the rx path without the need to cross check
> their actual state with the core transport layer.
> This mitigation is even more effective when, as is usually the case, the
> maximum number of pending messages is capped by the platform to a much
> lower number than the whole possible range of tokens values (2^10).
> 
> This internal policy change in the core SCMI transport layer is fully
> transparent to the specific transports so it has not and should not have
> any impact on the transports implementation.
> 
> The empirically observed cost of such new procedure of token selection
> amounts in the best case to ~10us out of an observed full transaction cost
> of 3ms for the completion of a synchronous sensor reading command on a
> platform supporting commands completion interrupts.

Hi Cristian,

Just curious... How badly did a cyclic IDR perform for this usecase?
Feature wise it seems suitable, but perhaps to heavy weight for this
rather constrained case where you can assume the number of IDs in
use at a time is rather small.

Also, I've not looked closely at the code so there may be other relevant
constraint or subtlety I'm missing.

Jonathan

> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/common.h |  23 +++
>  drivers/firmware/arm_scmi/driver.c | 243 +++++++++++++++++++++++++----
>  2 files changed, 233 insertions(+), 33 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index 6bb734e0e3ac..e64c5ca9ee7c 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -14,7 +14,10 @@
>  #include <linux/device.h>
>  #include <linux/errno.h>
>  #include <linux/kernel.h>
> +#include <linux/hashtable.h>
> +#include <linux/list.h>
>  #include <linux/module.h>
> +#include <linux/refcount.h>
>  #include <linux/scmi_protocol.h>
>  #include <linux/types.h>
>  
> @@ -127,6 +130,21 @@ struct scmi_msg {
>  	size_t len;
>  };
>  
> +/**
> + * An helper macro to lookup an xfer from the @pending_xfers hashtable
> + * using the message sequence number token as a key.
> + */
> +#define XFER_FIND(__ht, __k)					\
> +({								\
> +	typeof(__k) k_ = __k;					\
> +	struct scmi_xfer *xfer_ = NULL;				\
> +								\
> +	hash_for_each_possible((__ht), xfer_, node, k_)		\
> +		if (xfer_->hdr.seq == k_)			\
> +			break;					\
> +	 xfer_;							\
> +})
> +
>  /**
>   * struct scmi_xfer - Structure representing a message flow
>   *
> @@ -138,6 +156,9 @@ struct scmi_msg {
>   *	buffer for the rx path as we use for the tx path.
>   * @done: command message transmit completion event
>   * @async_done: pointer to delayed response message received event completion
> + * @users: A refcount to track the active users for this xfer
> + * @node: An hlist_node reference used to store this xfer, alternatively, on
> + *	  the free list @free_xfers or in the @pending_xfers hashtable
>   */
>  struct scmi_xfer {
>  	int transfer_id;
> @@ -146,6 +167,8 @@ struct scmi_xfer {
>  	struct scmi_msg rx;
>  	struct completion done;
>  	struct completion *async_done;
> +	refcount_t users;
> +	struct hlist_node node;
>  };
>  
>  struct scmi_xfer_ops;
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 20f8f0581f3a..f0b20ddb24f4 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -21,6 +21,7 @@
>  #include <linux/io.h>
>  #include <linux/kernel.h>
>  #include <linux/ktime.h>
> +#include <linux/hashtable.h>
>  #include <linux/list.h>
>  #include <linux/module.h>
>  #include <linux/of_address.h>
> @@ -65,19 +66,29 @@ struct scmi_requested_dev {
>  	struct list_head node;
>  };
>  
> +#define SCMI_PENDING_XFERS_HT_ORDER_SZ	9
> +
>  /**
>   * struct scmi_xfers_info - Structure to manage transfer information
>   *
> - * @xfer_block: Preallocated Message array
>   * @xfer_alloc_table: Bitmap table for allocated messages.
>   *	Index of this bitmap table is also used for message
>   *	sequence identifier.
>   * @xfer_lock: Protection for message allocation
> + * @last_token: A counter to use as base to generate for monotonically
> + *		increasing tokens.
> + * @free_xfers: A free list for available to use xfers. It is initialized with
> + *		a number of xfers equal to the maximum allowed in-flight
> + *		messages.
> + * @pending_xfers: An hashtable, indexed by msg_hdr.seq, used to keep all the
> + *		   currently in-flight messages.
>   */
>  struct scmi_xfers_info {
> -	struct scmi_xfer *xfer_block;
>  	unsigned long *xfer_alloc_table;
>  	spinlock_t xfer_lock;
> +	atomic_t last_token;
> +	struct hlist_head free_xfers;
> +	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
>  };
>  
>  /**
> @@ -203,6 +214,117 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
>  	return info->notify_priv;
>  }
>  
> +/**
> + * scmi_xfer_token_set  - Reserve and set new token for the xfer at hand
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer: The xfer to act upon
> + *
> + * Pick the next unused monotonically increasing token and set it into
> + * xfer->hdr.seq: picking a monotonically increasing value avoids immediate
> + * reuse of freshly completed or timed-out xfers, thus mitigating the risk
> + * of incorrect association of a late and expired xfer with a live in-flight
> + * transaction, both happening to re-use the same token identifier.
> + *
> + * Since platform is NOT required to answer our request in-order we should
> + * account for a few rare but possible scenarios:
> + *
> + *  - exactly 'next_token' may be NOT available so pick xfer_id >= next_token
> + *    using find_next_zero_bit() starting from candidate next_token bit
> + *
> + *  - all tokens ahead upto (MSG_TOKEN_ID_MASK - 1) are used in-flight but we
> + *    are plenty of free tokens at start, so try a second pass using
> + *    find_next_zero_bit() and starting from 0.
> + *
> + *  X = used in-flight
> + *
> + * Normal
> + * ------
> + *
> + *		|- xfer_id picked
> + *   -----------+----------------------------------------------------------
> + *   | | |X|X|X| | | | | | ... ... ... ... ... ... ... ... ... ... ...|X|X|
> + *   ----------------------------------------------------------------------
> + *		^
> + *		|- next_token
> + *
> + * Out-of-order pending at start
> + * -----------------------------
> + *
> + *	  |- xfer_id picked, last_token fixed
> + *   -----+----------------------------------------------------------------
> + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... ... ...|X| |
> + *   ----------------------------------------------------------------------
> + *    ^
> + *    |- next_token
> + *
> + *
> + * Out-of-order pending at end
> + * ---------------------------
> + *
> + *	  |- xfer_id picked, last_token fixed
> + *   -----+----------------------------------------------------------------
> + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... |X|X|X||X|X|
> + *   ----------------------------------------------------------------------
> + *								^
> + *								|- next_token
> + *
> + * Context: Assumes to be called with @xfer_lock already acquired.
> + *
> + * Return: 0 on Success or error
> + */
> +static int scmi_xfer_token_set(struct scmi_xfers_info *minfo,
> +			       struct scmi_xfer *xfer)
> +{
> +	unsigned long xfer_id, next_token;
> +
> +	/* Pick a candidate monotonic token in range [0, MSG_TOKEN_MAX - 1] */
> +	next_token = (atomic_inc_return(&minfo->last_token) &
> +		      (MSG_TOKEN_MAX - 1));
> +
> +	/* Pick the next available xfer_id >= next_token */
> +	xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> +				     MSG_TOKEN_MAX, next_token);
> +	if (xfer_id == MSG_TOKEN_MAX) {
> +		/*
> +		 * After heavily out-of-order responses, there are no free
> +		 * tokens ahead, but only at start of xfer_alloc_table so
> +		 * try again from the beginning.
> +		 */
> +		xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> +					     MSG_TOKEN_MAX, 0);
> +		/*
> +		 * Something is wrong if we got here since there can be a
> +		 * maximum number of (MSG_TOKEN_MAX - 1) in-flight messages
> +		 * but we have not found any free token [0, MSG_TOKEN_MAX - 1].
> +		 */
> +		if (WARN_ON_ONCE(xfer_id == MSG_TOKEN_MAX))
> +			return -ENOMEM;
> +	}
> +
> +	/* Update +/- last_token accordingly if we skipped some hole */
> +	if (xfer_id != next_token)
> +		atomic_add((int)(xfer_id - next_token), &minfo->last_token);
> +
> +	/* Set in-flight */
> +	set_bit(xfer_id, minfo->xfer_alloc_table);
> +	xfer->hdr.seq = (u16)xfer_id;
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_xfer_token_clear  - Release the token
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer: The xfer to act upon
> + */
> +static inline void scmi_xfer_token_clear(struct scmi_xfers_info *minfo,
> +					 struct scmi_xfer *xfer)
> +{
> +	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> +}
> +
>  /**
>   * scmi_xfer_get() - Allocate one message
>   *
> @@ -212,36 +334,49 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
>   * Helper function which is used by various message functions that are
>   * exposed to clients of this driver for allocating a message traffic event.
>   *
> - * This function can sleep depending on pending requests already in the system
> - * for the SCMI entity. Further, this also holds a spinlock to maintain
> - * integrity of internal data structures.
> + * Picks an xfer from the free list @free_xfers (if any available), sets a
> + * monotonically increasing token and stores the inflight xfer into the
> + * @pending_xfers hashtable for later retrieval.
> + *
> + * The successfully initialized xfer is refcounted.
> + *
> + * Context: Holds @xfer_lock while manipulating @xfer_alloc_table and
> + *	    @free_xfers.
>   *
>   * Return: 0 if all went fine, else corresponding error.
>   */
>  static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
>  				       struct scmi_xfers_info *minfo)
>  {
> -	u16 xfer_id;
> +	int ret;
> +	unsigned long flags;
>  	struct scmi_xfer *xfer;
> -	unsigned long flags, bit_pos;
> -	struct scmi_info *info = handle_to_scmi_info(handle);
>  
> -	/* Keep the locked section as small as possible */
>  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> -	bit_pos = find_first_zero_bit(minfo->xfer_alloc_table,
> -				      info->desc->max_msg);
> -	if (bit_pos == info->desc->max_msg) {
> +	if (hlist_empty(&minfo->free_xfers)) {
>  		spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  		return ERR_PTR(-ENOMEM);
>  	}
> -	set_bit(bit_pos, minfo->xfer_alloc_table);
> -	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  
> -	xfer_id = bit_pos;
> +	/* grab an xfer from the free_list */
> +	xfer = hlist_entry(minfo->free_xfers.first, struct scmi_xfer, node);
> +	hlist_del_init(&xfer->node);
>  
> -	xfer = &minfo->xfer_block[xfer_id];
> -	xfer->hdr.seq = xfer_id;
> -	xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> +	/* Pick and set monotonic token */
> +	ret = scmi_xfer_token_set(minfo, xfer);
> +	if (!ret) {
> +		hash_add(minfo->pending_xfers, &xfer->node, xfer->hdr.seq);
> +	} else {
> +		dev_err(handle->dev, "Failed to get monotonic token %d\n", ret);
> +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> +		xfer = ERR_PTR(ret);
> +	}
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +
> +	if (!IS_ERR(xfer)) {
> +		refcount_set(&xfer->users, 1);
> +		xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> +	}
>  
>  	return xfer;
>  }
> @@ -252,6 +387,9 @@ static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
>   * @minfo: Pointer to Tx/Rx Message management info based on channel type
>   * @xfer: message that was reserved by scmi_xfer_get
>   *
> + * After refcount check, possibly release an xfer, clearing the token slot,
> + * removing xfer from @pending_xfers and putting it back into free_xfers.
> + *
>   * This holds a spinlock to maintain integrity of internal data structures.
>   */
>  static void
> @@ -259,16 +397,41 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>  {
>  	unsigned long flags;
>  
> -	/*
> -	 * Keep the locked section as small as possible
> -	 * NOTE: we might escape with smp_mb and no lock here..
> -	 * but just be conservative and symmetric.
> -	 */
>  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> -	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> +	if (refcount_dec_and_test(&xfer->users)) {
> +		scmi_xfer_token_clear(minfo, xfer);
> +		hash_del(&xfer->node);
> +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> +	}
>  	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  }
>  
> +/**
> + * scmi_xfer_lookup_unlocked  -  Helper to lookup an xfer_id
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer_id: Token ID to lookup in @pending_xfers
> + *
> + * Refcounting is untouched.
> + *
> + * Context: Assumes to be called with @xfer_lock already acquired.
> + *
> + * Return: A valid xfer on Success or error otherwise
> + */
> +static struct scmi_xfer *
> +scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
> +{
> +	struct scmi_xfer *xfer = NULL;
> +
> +	if (xfer_id >= MSG_TOKEN_MAX)
> +		return ERR_PTR(-EINVAL);
> +
> +	if (test_bit(xfer_id, minfo->xfer_alloc_table))
> +		xfer = XFER_FIND(minfo->pending_xfers, xfer_id);
> +
> +	return xfer ?: ERR_PTR(-EINVAL);
> +}
> +
>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  {
>  	struct scmi_xfer *xfer;
> @@ -305,19 +468,22 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  static void scmi_handle_response(struct scmi_chan_info *cinfo,
>  				 u16 xfer_id, u8 msg_type)
>  {
> +	unsigned long flags;
>  	struct scmi_xfer *xfer;
>  	struct device *dev = cinfo->dev;
>  	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>  	struct scmi_xfers_info *minfo = &info->tx_minfo;
>  
>  	/* Are we even expecting this? */
> -	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
> +	spin_lock_irqsave(&minfo->xfer_lock, flags);
> +	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +	if (IS_ERR(xfer)) {
>  		dev_err(dev, "message for %d is not expected!\n", xfer_id);
>  		info->desc->ops->clear_channel(cinfo);
>  		return;
>  	}
>  
> -	xfer = &minfo->xfer_block[xfer_id];
>  	/*
>  	 * Even if a response was indeed expected on this slot at this point,
>  	 * a buggy platform could wrongly reply feeding us an unexpected
> @@ -1033,18 +1199,25 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  		return -EINVAL;
>  	}
>  
> -	info->xfer_block = devm_kcalloc(dev, desc->max_msg,
> -					sizeof(*info->xfer_block), GFP_KERNEL);
> -	if (!info->xfer_block)
> -		return -ENOMEM;
> +	hash_init(info->pending_xfers);
>  
> -	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(desc->max_msg),
> +	/* Allocate a bitmask sized to hold MSG_TOKEN_MAX tokens */
> +	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(MSG_TOKEN_MAX),
>  					      sizeof(long), GFP_KERNEL);
>  	if (!info->xfer_alloc_table)
>  		return -ENOMEM;
>  
> -	/* Pre-initialize the buffer pointer to pre-allocated buffers */
> -	for (i = 0, xfer = info->xfer_block; i < desc->max_msg; i++, xfer++) {
> +	/*
> +	 * Preallocate a number of xfers equal to max inflight messages,
> +	 * pre-initialize the buffer pointer to pre-allocated buffers and
> +	 * attach all of them to the free list
> +	 */
> +	INIT_HLIST_HEAD(&info->free_xfers);
> +	for (i = 0; i < desc->max_msg; i++) {
> +		xfer = devm_kzalloc(dev, sizeof(*xfer), GFP_KERNEL);
> +		if (!xfer)
> +			return -ENOMEM;
> +
>  		xfer->rx.buf = devm_kcalloc(dev, sizeof(u8), desc->max_msg_size,
>  					    GFP_KERNEL);
>  		if (!xfer->rx.buf)
> @@ -1052,8 +1225,12 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  
>  		xfer->tx.buf = xfer->rx.buf;
>  		init_completion(&xfer->done);
> +
> +		/* Add initialized xfer to the free list */
> +		hlist_add_head(&xfer->node, &info->free_xfers);
>  	}
>  
> +	atomic_set(&info->last_token, -1);
>  	spin_lock_init(&info->xfer_lock);
>  
>  	return 0;


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
@ 2021-06-14 13:53     ` Jonathan Cameron
  0 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 13:53 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:25 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> Tokens are sequence numbers embedded in the each SCMI message header: they
> are used to correlate commands with responses (and delayed responses), but
> their usage and policy of selection is entirely up to the caller (usually
> the OSPM agent), while they are completely opaque to the callee (SCMI
> server platform) which merely copies them back from the command into the
> response message header.
> This also means that the platform does not, can not and should not enforce
> any kind of policy on received messages depending on the contained sequence
> number: platform can perfectly handle concurrent requests carrying the same
> identifiying token if that should happen.
> 
> Moreover the platform is not required to produce in-order responses to
> agent requests, the only constraint in these regards is that in case of
> an asynchronous message the delayed response must be sent after the
> immediate response for the synchronous part of the command transaction.
> 
> Currenly the SCMI stack of the OSPM agent selects a token for the egressing
> commands picking the lowest possible number which is not already in use by
> an existing in-flight transaction, which means, in other words, that we
> immediately reuse any token after its transaction has completed or it has
> timed out: this policy indeed does simplify management and lookup of tokens
> and associated xfers.
> 
> Under the above assumptions and constraints, since there is really no state
> shared between the agent and the platform to let the platform know when a
> token and its associated message has timed out, the current policy of early
> reuse of tokens can easily lead to the situation in which a spurious or
> late received response (or delayed_response), related to an old stale and
> timed out transaction, can be wrongly associated to a newer valid in-flight
> xfer that just happens to have reused the same token.
> 
> This misbehaviour on such ghost responses is more easily exposed on those
> transports that naturally have an higher level of parallelism in processing
> multiple concurrent in-flight messages.
> 
> This commit introduces a new policy of selection of tokens for the OSPM
> agent: each new transfer now gets the next available and monotonically
> increasing token, until tokens are exhausted and the counter rolls over.
> 
> Such new policy mitigates the above issues with ghost responses since the
> tokens are now reused as late as possible (when they roll back ideally)
> and so it is much easier to identify such ghost responses to stale timed
> out transactions: this also helps in simplifying the specific transports
> implementation since stale transport messages can be easily identified
> and discarded early on in the rx path without the need to cross check
> their actual state with the core transport layer.
> This mitigation is even more effective when, as is usually the case, the
> maximum number of pending messages is capped by the platform to a much
> lower number than the whole possible range of tokens values (2^10).
> 
> This internal policy change in the core SCMI transport layer is fully
> transparent to the specific transports so it has not and should not have
> any impact on the transports implementation.
> 
> The empirically observed cost of such new procedure of token selection
> amounts in the best case to ~10us out of an observed full transaction cost
> of 3ms for the completion of a synchronous sensor reading command on a
> platform supporting commands completion interrupts.

Hi Cristian,

Just curious... How badly did a cyclic IDR perform for this usecase?
Feature wise it seems suitable, but perhaps to heavy weight for this
rather constrained case where you can assume the number of IDs in
use at a time is rather small.

Also, I've not looked closely at the code so there may be other relevant
constraint or subtlety I'm missing.

Jonathan

> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/common.h |  23 +++
>  drivers/firmware/arm_scmi/driver.c | 243 +++++++++++++++++++++++++----
>  2 files changed, 233 insertions(+), 33 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index 6bb734e0e3ac..e64c5ca9ee7c 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -14,7 +14,10 @@
>  #include <linux/device.h>
>  #include <linux/errno.h>
>  #include <linux/kernel.h>
> +#include <linux/hashtable.h>
> +#include <linux/list.h>
>  #include <linux/module.h>
> +#include <linux/refcount.h>
>  #include <linux/scmi_protocol.h>
>  #include <linux/types.h>
>  
> @@ -127,6 +130,21 @@ struct scmi_msg {
>  	size_t len;
>  };
>  
> +/**
> + * An helper macro to lookup an xfer from the @pending_xfers hashtable
> + * using the message sequence number token as a key.
> + */
> +#define XFER_FIND(__ht, __k)					\
> +({								\
> +	typeof(__k) k_ = __k;					\
> +	struct scmi_xfer *xfer_ = NULL;				\
> +								\
> +	hash_for_each_possible((__ht), xfer_, node, k_)		\
> +		if (xfer_->hdr.seq == k_)			\
> +			break;					\
> +	 xfer_;							\
> +})
> +
>  /**
>   * struct scmi_xfer - Structure representing a message flow
>   *
> @@ -138,6 +156,9 @@ struct scmi_msg {
>   *	buffer for the rx path as we use for the tx path.
>   * @done: command message transmit completion event
>   * @async_done: pointer to delayed response message received event completion
> + * @users: A refcount to track the active users for this xfer
> + * @node: An hlist_node reference used to store this xfer, alternatively, on
> + *	  the free list @free_xfers or in the @pending_xfers hashtable
>   */
>  struct scmi_xfer {
>  	int transfer_id;
> @@ -146,6 +167,8 @@ struct scmi_xfer {
>  	struct scmi_msg rx;
>  	struct completion done;
>  	struct completion *async_done;
> +	refcount_t users;
> +	struct hlist_node node;
>  };
>  
>  struct scmi_xfer_ops;
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 20f8f0581f3a..f0b20ddb24f4 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -21,6 +21,7 @@
>  #include <linux/io.h>
>  #include <linux/kernel.h>
>  #include <linux/ktime.h>
> +#include <linux/hashtable.h>
>  #include <linux/list.h>
>  #include <linux/module.h>
>  #include <linux/of_address.h>
> @@ -65,19 +66,29 @@ struct scmi_requested_dev {
>  	struct list_head node;
>  };
>  
> +#define SCMI_PENDING_XFERS_HT_ORDER_SZ	9
> +
>  /**
>   * struct scmi_xfers_info - Structure to manage transfer information
>   *
> - * @xfer_block: Preallocated Message array
>   * @xfer_alloc_table: Bitmap table for allocated messages.
>   *	Index of this bitmap table is also used for message
>   *	sequence identifier.
>   * @xfer_lock: Protection for message allocation
> + * @last_token: A counter to use as base to generate for monotonically
> + *		increasing tokens.
> + * @free_xfers: A free list for available to use xfers. It is initialized with
> + *		a number of xfers equal to the maximum allowed in-flight
> + *		messages.
> + * @pending_xfers: An hashtable, indexed by msg_hdr.seq, used to keep all the
> + *		   currently in-flight messages.
>   */
>  struct scmi_xfers_info {
> -	struct scmi_xfer *xfer_block;
>  	unsigned long *xfer_alloc_table;
>  	spinlock_t xfer_lock;
> +	atomic_t last_token;
> +	struct hlist_head free_xfers;
> +	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
>  };
>  
>  /**
> @@ -203,6 +214,117 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
>  	return info->notify_priv;
>  }
>  
> +/**
> + * scmi_xfer_token_set  - Reserve and set new token for the xfer at hand
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer: The xfer to act upon
> + *
> + * Pick the next unused monotonically increasing token and set it into
> + * xfer->hdr.seq: picking a monotonically increasing value avoids immediate
> + * reuse of freshly completed or timed-out xfers, thus mitigating the risk
> + * of incorrect association of a late and expired xfer with a live in-flight
> + * transaction, both happening to re-use the same token identifier.
> + *
> + * Since platform is NOT required to answer our request in-order we should
> + * account for a few rare but possible scenarios:
> + *
> + *  - exactly 'next_token' may be NOT available so pick xfer_id >= next_token
> + *    using find_next_zero_bit() starting from candidate next_token bit
> + *
> + *  - all tokens ahead upto (MSG_TOKEN_ID_MASK - 1) are used in-flight but we
> + *    are plenty of free tokens at start, so try a second pass using
> + *    find_next_zero_bit() and starting from 0.
> + *
> + *  X = used in-flight
> + *
> + * Normal
> + * ------
> + *
> + *		|- xfer_id picked
> + *   -----------+----------------------------------------------------------
> + *   | | |X|X|X| | | | | | ... ... ... ... ... ... ... ... ... ... ...|X|X|
> + *   ----------------------------------------------------------------------
> + *		^
> + *		|- next_token
> + *
> + * Out-of-order pending at start
> + * -----------------------------
> + *
> + *	  |- xfer_id picked, last_token fixed
> + *   -----+----------------------------------------------------------------
> + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... ... ...|X| |
> + *   ----------------------------------------------------------------------
> + *    ^
> + *    |- next_token
> + *
> + *
> + * Out-of-order pending at end
> + * ---------------------------
> + *
> + *	  |- xfer_id picked, last_token fixed
> + *   -----+----------------------------------------------------------------
> + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... |X|X|X||X|X|
> + *   ----------------------------------------------------------------------
> + *								^
> + *								|- next_token
> + *
> + * Context: Assumes to be called with @xfer_lock already acquired.
> + *
> + * Return: 0 on Success or error
> + */
> +static int scmi_xfer_token_set(struct scmi_xfers_info *minfo,
> +			       struct scmi_xfer *xfer)
> +{
> +	unsigned long xfer_id, next_token;
> +
> +	/* Pick a candidate monotonic token in range [0, MSG_TOKEN_MAX - 1] */
> +	next_token = (atomic_inc_return(&minfo->last_token) &
> +		      (MSG_TOKEN_MAX - 1));
> +
> +	/* Pick the next available xfer_id >= next_token */
> +	xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> +				     MSG_TOKEN_MAX, next_token);
> +	if (xfer_id == MSG_TOKEN_MAX) {
> +		/*
> +		 * After heavily out-of-order responses, there are no free
> +		 * tokens ahead, but only at start of xfer_alloc_table so
> +		 * try again from the beginning.
> +		 */
> +		xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> +					     MSG_TOKEN_MAX, 0);
> +		/*
> +		 * Something is wrong if we got here since there can be a
> +		 * maximum number of (MSG_TOKEN_MAX - 1) in-flight messages
> +		 * but we have not found any free token [0, MSG_TOKEN_MAX - 1].
> +		 */
> +		if (WARN_ON_ONCE(xfer_id == MSG_TOKEN_MAX))
> +			return -ENOMEM;
> +	}
> +
> +	/* Update +/- last_token accordingly if we skipped some hole */
> +	if (xfer_id != next_token)
> +		atomic_add((int)(xfer_id - next_token), &minfo->last_token);
> +
> +	/* Set in-flight */
> +	set_bit(xfer_id, minfo->xfer_alloc_table);
> +	xfer->hdr.seq = (u16)xfer_id;
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_xfer_token_clear  - Release the token
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer: The xfer to act upon
> + */
> +static inline void scmi_xfer_token_clear(struct scmi_xfers_info *minfo,
> +					 struct scmi_xfer *xfer)
> +{
> +	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> +}
> +
>  /**
>   * scmi_xfer_get() - Allocate one message
>   *
> @@ -212,36 +334,49 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
>   * Helper function which is used by various message functions that are
>   * exposed to clients of this driver for allocating a message traffic event.
>   *
> - * This function can sleep depending on pending requests already in the system
> - * for the SCMI entity. Further, this also holds a spinlock to maintain
> - * integrity of internal data structures.
> + * Picks an xfer from the free list @free_xfers (if any available), sets a
> + * monotonically increasing token and stores the inflight xfer into the
> + * @pending_xfers hashtable for later retrieval.
> + *
> + * The successfully initialized xfer is refcounted.
> + *
> + * Context: Holds @xfer_lock while manipulating @xfer_alloc_table and
> + *	    @free_xfers.
>   *
>   * Return: 0 if all went fine, else corresponding error.
>   */
>  static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
>  				       struct scmi_xfers_info *minfo)
>  {
> -	u16 xfer_id;
> +	int ret;
> +	unsigned long flags;
>  	struct scmi_xfer *xfer;
> -	unsigned long flags, bit_pos;
> -	struct scmi_info *info = handle_to_scmi_info(handle);
>  
> -	/* Keep the locked section as small as possible */
>  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> -	bit_pos = find_first_zero_bit(minfo->xfer_alloc_table,
> -				      info->desc->max_msg);
> -	if (bit_pos == info->desc->max_msg) {
> +	if (hlist_empty(&minfo->free_xfers)) {
>  		spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  		return ERR_PTR(-ENOMEM);
>  	}
> -	set_bit(bit_pos, minfo->xfer_alloc_table);
> -	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  
> -	xfer_id = bit_pos;
> +	/* grab an xfer from the free_list */
> +	xfer = hlist_entry(minfo->free_xfers.first, struct scmi_xfer, node);
> +	hlist_del_init(&xfer->node);
>  
> -	xfer = &minfo->xfer_block[xfer_id];
> -	xfer->hdr.seq = xfer_id;
> -	xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> +	/* Pick and set monotonic token */
> +	ret = scmi_xfer_token_set(minfo, xfer);
> +	if (!ret) {
> +		hash_add(minfo->pending_xfers, &xfer->node, xfer->hdr.seq);
> +	} else {
> +		dev_err(handle->dev, "Failed to get monotonic token %d\n", ret);
> +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> +		xfer = ERR_PTR(ret);
> +	}
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +
> +	if (!IS_ERR(xfer)) {
> +		refcount_set(&xfer->users, 1);
> +		xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> +	}
>  
>  	return xfer;
>  }
> @@ -252,6 +387,9 @@ static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
>   * @minfo: Pointer to Tx/Rx Message management info based on channel type
>   * @xfer: message that was reserved by scmi_xfer_get
>   *
> + * After refcount check, possibly release an xfer, clearing the token slot,
> + * removing xfer from @pending_xfers and putting it back into free_xfers.
> + *
>   * This holds a spinlock to maintain integrity of internal data structures.
>   */
>  static void
> @@ -259,16 +397,41 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>  {
>  	unsigned long flags;
>  
> -	/*
> -	 * Keep the locked section as small as possible
> -	 * NOTE: we might escape with smp_mb and no lock here..
> -	 * but just be conservative and symmetric.
> -	 */
>  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> -	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> +	if (refcount_dec_and_test(&xfer->users)) {
> +		scmi_xfer_token_clear(minfo, xfer);
> +		hash_del(&xfer->node);
> +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> +	}
>  	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  }
>  
> +/**
> + * scmi_xfer_lookup_unlocked  -  Helper to lookup an xfer_id
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer_id: Token ID to lookup in @pending_xfers
> + *
> + * Refcounting is untouched.
> + *
> + * Context: Assumes to be called with @xfer_lock already acquired.
> + *
> + * Return: A valid xfer on Success or error otherwise
> + */
> +static struct scmi_xfer *
> +scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
> +{
> +	struct scmi_xfer *xfer = NULL;
> +
> +	if (xfer_id >= MSG_TOKEN_MAX)
> +		return ERR_PTR(-EINVAL);
> +
> +	if (test_bit(xfer_id, minfo->xfer_alloc_table))
> +		xfer = XFER_FIND(minfo->pending_xfers, xfer_id);
> +
> +	return xfer ?: ERR_PTR(-EINVAL);
> +}
> +
>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  {
>  	struct scmi_xfer *xfer;
> @@ -305,19 +468,22 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  static void scmi_handle_response(struct scmi_chan_info *cinfo,
>  				 u16 xfer_id, u8 msg_type)
>  {
> +	unsigned long flags;
>  	struct scmi_xfer *xfer;
>  	struct device *dev = cinfo->dev;
>  	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>  	struct scmi_xfers_info *minfo = &info->tx_minfo;
>  
>  	/* Are we even expecting this? */
> -	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
> +	spin_lock_irqsave(&minfo->xfer_lock, flags);
> +	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +	if (IS_ERR(xfer)) {
>  		dev_err(dev, "message for %d is not expected!\n", xfer_id);
>  		info->desc->ops->clear_channel(cinfo);
>  		return;
>  	}
>  
> -	xfer = &minfo->xfer_block[xfer_id];
>  	/*
>  	 * Even if a response was indeed expected on this slot at this point,
>  	 * a buggy platform could wrongly reply feeding us an unexpected
> @@ -1033,18 +1199,25 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  		return -EINVAL;
>  	}
>  
> -	info->xfer_block = devm_kcalloc(dev, desc->max_msg,
> -					sizeof(*info->xfer_block), GFP_KERNEL);
> -	if (!info->xfer_block)
> -		return -ENOMEM;
> +	hash_init(info->pending_xfers);
>  
> -	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(desc->max_msg),
> +	/* Allocate a bitmask sized to hold MSG_TOKEN_MAX tokens */
> +	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(MSG_TOKEN_MAX),
>  					      sizeof(long), GFP_KERNEL);
>  	if (!info->xfer_alloc_table)
>  		return -ENOMEM;
>  
> -	/* Pre-initialize the buffer pointer to pre-allocated buffers */
> -	for (i = 0, xfer = info->xfer_block; i < desc->max_msg; i++, xfer++) {
> +	/*
> +	 * Preallocate a number of xfers equal to max inflight messages,
> +	 * pre-initialize the buffer pointer to pre-allocated buffers and
> +	 * attach all of them to the free list
> +	 */
> +	INIT_HLIST_HEAD(&info->free_xfers);
> +	for (i = 0; i < desc->max_msg; i++) {
> +		xfer = devm_kzalloc(dev, sizeof(*xfer), GFP_KERNEL);
> +		if (!xfer)
> +			return -ENOMEM;
> +
>  		xfer->rx.buf = devm_kcalloc(dev, sizeof(u8), desc->max_msg_size,
>  					    GFP_KERNEL);
>  		if (!xfer->rx.buf)
> @@ -1052,8 +1225,12 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  
>  		xfer->tx.buf = xfer->rx.buf;
>  		init_completion(&xfer->done);
> +
> +		/* Add initialized xfer to the free list */
> +		hlist_add_head(&xfer->node, &info->free_xfers);
>  	}
>  
> +	atomic_set(&info->last_token, -1);
>  	spin_lock_init(&info->xfer_lock);
>  
>  	return 0;


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 00/16] Introduce SCMI VirtIO transport
  2021-06-14 11:43   ` Christoph Hellwig
@ 2021-06-14 14:03     ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-14 14:03 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, peter.hilber, alex.bennee, jean-philippe,
	mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Hi Christoph,

On Mon, Jun 14, 2021 at 12:43:52PM +0100, Christoph Hellwig wrote:
> On Fri, Jun 11, 2021 at 05:59:21PM +0100, Cristian Marussi wrote:
> > Hi all,
> > 
> > I'm posting this V4 series starting from the work done up to V3 by
> > OpenSynergy.
> 
> Who is 'OpenSynergy'?
> 
> > The main aim of this rework is to simplify where possible the SCMI VirtIO
> > support added in V3 by adding upfront and then using some new mechanisms in
> > the SCMI Core Transport layer.
> 
> And what is 'SCMI', and why would anyone want a new virtio transport?

I'll start answering this last question first : SCMI stands for System Control
and Management Interface whose latest specification can be found at:

https://developer.arm.com/documentation/den0056/latest

The spec aims to provide a common way to handle power & performance related
needs by standardizing a set of protocols (clocks, power domains, sensors,
voltages, resets, etc..) to enable an SCMI agent (Kernel) to talk to an
external System Control Processor entity which acts as an SCMI Platform and
satisfies (or denies) such requests in a centralized manner for the Kernel
and any other SCMI agent present on system.

Such SCMI stack can be indeed deployed in a variety of way depending on
where the SCP running the SCMI Platofrm if physically situated: an
external microntroller ? part of the EL-3 Platform firmware ? some
functionality embedded in an Hypervisor ? a guest acting as an SCP ?

Support for SCMI is already in mainline as of today under:

	drivers/firmware/arm_scmi

But the currently existing transport mechanisms through which the SCMI agent
and the platform talks are based on mailboxes or SMCs.

In case the SCMI Stack is deployed in a virtualized environment we need
some sort of SCMI transport that runs on VirtIO to establish comms
between the VMs.

OpenSynergy is an ARM partner who has deployed a virtualized SCMI stack
in their own product and developed this series up to V3, taking care
also the proposed related VirtIO spec changes:

https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html

Such proposed VirtIO changes expose a new VirtiIO SCMI Device that represents
the SCMI platform running the SCMI Server fw which answers the SCMI Kernel
Agent requests.

Similar approaches with virtualized SCMI stacks are being developed by
Linaro and other partners as far as I know.

I picked up this series from V4 since it was apparent that some changes were
needed in the core SCMI stack to better integrate this new VirtIO transport
that OpenSynergy developed up to V3.

Hope to have been clear and concise (not really :D) enough.

Thanks,
Cristian

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 00/16] Introduce SCMI VirtIO transport
@ 2021-06-14 14:03     ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-14 14:03 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, peter.hilber, alex.bennee, jean-philippe,
	mikhail.golubev, anton.yakovlev, Vasyl.Vavrychuk,
	Andriy.Tryshnivskyy

Hi Christoph,

On Mon, Jun 14, 2021 at 12:43:52PM +0100, Christoph Hellwig wrote:
> On Fri, Jun 11, 2021 at 05:59:21PM +0100, Cristian Marussi wrote:
> > Hi all,
> > 
> > I'm posting this V4 series starting from the work done up to V3 by
> > OpenSynergy.
> 
> Who is 'OpenSynergy'?
> 
> > The main aim of this rework is to simplify where possible the SCMI VirtIO
> > support added in V3 by adding upfront and then using some new mechanisms in
> > the SCMI Core Transport layer.
> 
> And what is 'SCMI', and why would anyone want a new virtio transport?

I'll start answering this last question first : SCMI stands for System Control
and Management Interface whose latest specification can be found at:

https://developer.arm.com/documentation/den0056/latest

The spec aims to provide a common way to handle power & performance related
needs by standardizing a set of protocols (clocks, power domains, sensors,
voltages, resets, etc..) to enable an SCMI agent (Kernel) to talk to an
external System Control Processor entity which acts as an SCMI Platform and
satisfies (or denies) such requests in a centralized manner for the Kernel
and any other SCMI agent present on system.

Such SCMI stack can be indeed deployed in a variety of way depending on
where the SCP running the SCMI Platofrm if physically situated: an
external microntroller ? part of the EL-3 Platform firmware ? some
functionality embedded in an Hypervisor ? a guest acting as an SCP ?

Support for SCMI is already in mainline as of today under:

	drivers/firmware/arm_scmi

But the currently existing transport mechanisms through which the SCMI agent
and the platform talks are based on mailboxes or SMCs.

In case the SCMI Stack is deployed in a virtualized environment we need
some sort of SCMI transport that runs on VirtIO to establish comms
between the VMs.

OpenSynergy is an ARM partner who has deployed a virtualized SCMI stack
in their own product and developed this series up to V3, taking care
also the proposed related VirtIO spec changes:

https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html

Such proposed VirtIO changes expose a new VirtiIO SCMI Device that represents
the SCMI platform running the SCMI Server fw which answers the SCMI Kernel
Agent requests.

Similar approaches with virtualized SCMI stacks are being developed by
Linaro and other partners as far as I know.

I picked up this series from V4 since it was apparent that some changes were
needed in the core SCMI stack to better integrate this new VirtIO transport
that OpenSynergy developed up to V3.

Hope to have been clear and concise (not really :D) enough.

Thanks,
Cristian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 07/16] firmware: arm_scmi: Add op to override max message #
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-06-14 14:04     ` Jonathan Cameron
  -1 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 14:04 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:28 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> The number of simultaneously pending messages that the upcoming
> scmi-virtio transport can support depends on the virtio device (SCMI
> platform) and can differ for each channel. (The scmi-virtio transport
> does only have one tx and at most 1 rx channel.)
> 
> Add an optional transport op so that scmi-virtio can report the actual
> max message # for each channel type. Respect these new limits.
> 
> Reflect that the limit in struct scmi_desc is now only a default any
> more.
> 
> [ Peter: Adapted patch for submission to upstream. ]
> 
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
Cristian,

Give you are sending this on to the list, should have your
Signed-off-by as well.

Jonathan

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 07/16] firmware: arm_scmi: Add op to override max message #
@ 2021-06-14 14:04     ` Jonathan Cameron
  0 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 14:04 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:28 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> The number of simultaneously pending messages that the upcoming
> scmi-virtio transport can support depends on the virtio device (SCMI
> platform) and can differ for each channel. (The scmi-virtio transport
> does only have one tx and at most 1 rx channel.)
> 
> Add an optional transport op so that scmi-virtio can report the actual
> max message # for each channel type. Respect these new limits.
> 
> Reflect that the limit in struct scmi_desc is now only a default any
> more.
> 
> [ Peter: Adapted patch for submission to upstream. ]
> 
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
Cristian,

Give you are sending this on to the list, should have your
Signed-off-by as well.

Jonathan

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 12/16] firmware: arm_scmi: Add message passing abstractions for transports
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-06-14 14:10     ` Jonathan Cameron
  -1 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 14:10 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:33 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> From: Peter Hilber <peter.hilber@opensynergy.com>
> 
> Add abstractions for future transports using message passing, such as
> virtio. Derive the abstractions from the shared memory abstractions.
> 
> Abstract the transport SDU through the opaque struct scmi_msg_payld.
> Also enable the transport to determine all other required information
> about the transport SDU.
> 
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> ---
...
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index b783ae058c8a..fa4075336580 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -410,6 +410,21 @@ void shmem_clear_channel(struct scmi_shared_mem __iomem *shmem);
>  bool shmem_poll_done(struct scmi_shared_mem __iomem *shmem,
>  		     struct scmi_xfer *xfer);
>  
> +/* declarations for message passing transports */
> +struct scmi_msg_payld;
> +
> +/** Maximum overhead of message w.r.t. struct scmi_desc.max_msg_size */

Doesn't look to be kernel-doc..

> +#define SCMI_MSG_MAX_PROT_OVERHEAD (2 * sizeof(__le32))

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 12/16] firmware: arm_scmi: Add message passing abstractions for transports
@ 2021-06-14 14:10     ` Jonathan Cameron
  0 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 14:10 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:33 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> From: Peter Hilber <peter.hilber@opensynergy.com>
> 
> Add abstractions for future transports using message passing, such as
> virtio. Derive the abstractions from the shared memory abstractions.
> 
> Abstract the transport SDU through the opaque struct scmi_msg_payld.
> Also enable the transport to determine all other required information
> about the transport SDU.
> 
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> ---
...
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index b783ae058c8a..fa4075336580 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -410,6 +410,21 @@ void shmem_clear_channel(struct scmi_shared_mem __iomem *shmem);
>  bool shmem_poll_done(struct scmi_shared_mem __iomem *shmem,
>  		     struct scmi_xfer *xfer);
>  
> +/* declarations for message passing transports */
> +struct scmi_msg_payld;
> +
> +/** Maximum overhead of message w.r.t. struct scmi_desc.max_msg_size */

Doesn't look to be kernel-doc..

> +#define SCMI_MSG_MAX_PROT_OVERHEAD (2 * sizeof(__le32))

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-06-14 14:23     ` Jonathan Cameron
  -1 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 14:23 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:35 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> This transport enables accessing an SCMI platform as a virtio device.
> 
> Implement an SCMI virtio driver according to the virtio SCMI device spec
> [1]. Virtio device id 32 has been reserved for the SCMI device [2].
> 
> The virtio transport has one Tx channel (virtio cmdq, A2P channel) and
> at most one Rx channel (virtio eventq, P2A channel).
> 
> The following feature bit defined in [1] is not implemented:
> VIRTIO_SCMI_F_SHARED_MEMORY.
> 
> After the preparatory patches, this implements the virtio transport as
> paraphrased:
> 
> Only support a single arm-scmi device (which is consistent with the SCMI
> spec). scmi-virtio init is called from arm-scmi module init. During the
> arm-scmi probing, link to the first probed scmi-virtio device. Defer
> arm-scmi probing if no scmi-virtio device is bound yet.
> 
> For simplicity, restrict the number of messages which can be pending
> simultaneously according to the virtqueue capacity. (The virtqueue sizes
> are negotiated with the virtio device.)
> 
> As soon as Rx channel message buffers are allocated or have been read
> out by the arm-scmi driver, feed them to the virtio device.
> 
> Since some virtio devices may not have the short response time exhibited
> by SCMI platforms using other transports, set a generous response
> timeout.
> 
> Limitations:
> 
> - Polling is not supported.
> 
> - The timeout for delayed responses has not been adjusted.
> 
> [1] https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-scmi.tex
> [2] https://www.oasis-open.org/committees/ballot.php?id=3496
> 
> [ Peter: Adapted patch for submission to upstream. ]
> 
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>

A few drive by comments inline.

J
> ---
>  MAINTAINERS                        |   1 +
>  drivers/firmware/Kconfig           |  12 +
>  drivers/firmware/arm_scmi/Makefile |   1 +
>  drivers/firmware/arm_scmi/common.h |   3 +
>  drivers/firmware/arm_scmi/driver.c |   3 +
>  drivers/firmware/arm_scmi/virtio.c | 523 +++++++++++++++++++++++++++++
>  include/uapi/linux/virtio_ids.h    |   1 +
>  include/uapi/linux/virtio_scmi.h   |  25 ++
>  8 files changed, 569 insertions(+)
>  create mode 100644 drivers/firmware/arm_scmi/virtio.c
>  create mode 100644 include/uapi/linux/virtio_scmi.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index f408cf2d2781..e1b27ed11060 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -17878,6 +17878,7 @@ F:	drivers/regulator/scmi-regulator.c
>  F:	drivers/reset/reset-scmi.c
>  F:	include/linux/sc[mp]i_protocol.h
>  F:	include/trace/events/scmi.h
> +F:	include/uapi/linux/virtio_scmi.h
>  
>  SYSTEM RESET/SHUTDOWN DRIVERS
>  M:	Sebastian Reichel <sre@kernel.org>
> diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
> index e74d25065d07..2c37d5af66ad 100644
> --- a/drivers/firmware/Kconfig
> +++ b/drivers/firmware/Kconfig
> @@ -39,6 +39,18 @@ config ARM_SCMI_HAVE_MSG
>  	  This declares whether a message passing based transport for SCMI is
>  	  available.
>  
> +# This config option includes the dependencies of ARM_SCMI_PROTOCOL so that
> +# this config doesn't show up when SCMI wouldn't be available.
> +config VIRTIO_SCMI
> +	bool "Virtio transport for SCMI"
> +	select ARM_SCMI_HAVE_MSG
> +	depends on VIRTIO && (ARM || ARM64 || COMPILE_TEST)
> +	help
> +	  This enables the virtio based transport for SCMI.
> +
> +	  If you want to use the ARM SCMI protocol between the virtio guest and
> +	  a host providing a virtio SCMI device, answer Y.
> +
>  config ARM_SCMI_POWER_DOMAIN
>  	tristate "SCMI power domain driver"
>  	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
> diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
> index f6b4acb8abdb..db1787606fb2 100644
> --- a/drivers/firmware/arm_scmi/Makefile
> +++ b/drivers/firmware/arm_scmi/Makefile
> @@ -5,6 +5,7 @@ scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
>  scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
>  scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
>  scmi-transport-$(CONFIG_ARM_SCMI_HAVE_MSG) += msg.o
> +scmi-transport-$(CONFIG_VIRTIO_SCMI) += virtio.o
>  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
>  scmi-module-objs := $(scmi-bus-y) $(scmi-driver-y) $(scmi-protocols-y) \
>  		    $(scmi-transport-y)
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index fa4075336580..c143a449d278 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -390,6 +390,9 @@ extern const struct scmi_desc scmi_mailbox_desc;
>  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
>  extern const struct scmi_desc scmi_smc_desc;
>  #endif
> +#ifdef CONFIG_VIRTIO_SCMI
> +extern const struct scmi_desc scmi_virtio_desc;
> +#endif
>  
>  int scmi_set_transport_info(struct device *dev, void *transport_info);
>  void *scmi_get_transport_info(struct device *dev);
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index bb2de15554a9..ccc7dd49261e 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -1954,6 +1954,9 @@ static const struct of_device_id scmi_of_match[] = {
>  #endif
>  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
>  	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
> +#endif
> +#ifdef CONFIG_VIRTIO_SCMI
> +	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
>  #endif
>  	{ /* Sentinel */ },
>  };
> diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
> new file mode 100644
> index 000000000000..20972adf6dc7
> --- /dev/null
> +++ b/drivers/firmware/arm_scmi/virtio.c
> @@ -0,0 +1,523 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Virtio Transport driver for Arm System Control and Management Interface
> + * (SCMI).
> + *
> + * Copyright (C) 2020 OpenSynergy.
> + */
> +
> +/**
> + * DOC: Theory of Operation
> + *
> + * The scmi-virtio transport implements a driver for the virtio SCMI device.
> + *
> + * There is one Tx channel (virtio cmdq, A2P channel) and at most one Rx
> + * channel (virtio eventq, P2A channel). Each channel is implemented through a
> + * virtqueue. Access to each virtqueue is protected by spinlocks.
> + */
> +
> +#include <linux/errno.h>

> +#include <linux/of.h>
> +#include <linux/of_platform.h>
Why either of these?

perhaps
mod_devicetable.h is appropriate.

> +#include <linux/platform_device.h>
> +#include <linux/module.h>
> +#include <linux/slab.h>
> +#include <linux/virtio.h>
> +#include <linux/virtio_config.h>
> +#include <uapi/linux/virtio_ids.h>
> +#include <uapi/linux/virtio_scmi.h>
> +
> +#include "common.h"
> +
> +#define VIRTIO_SCMI_MAX_MSG_SIZE 128 /* Value may be increased. */
> +#define VIRTIO_SCMI_MAX_PDU_SIZE \
> +	(VIRTIO_SCMI_MAX_MSG_SIZE + SCMI_MSG_MAX_PROT_OVERHEAD)
> +#define DESCRIPTORS_PER_TX_MSG 2
> +
> +/**
> + * struct scmi_vio_channel - Transport channel information
> + *
> + * @lock: Protects access to all members except ready.
> + * @ready_lock: Protects access to ready. If required, it must be taken before
> + *              lock.
> + * @vqueue: Associated virtqueue
> + * @cinfo: SCMI Tx or Rx channel
> + * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
> + * @is_rx: Whether channel is an Rx channel
> + * @ready: Whether transport user is ready to hear about channel
> + */
> +struct scmi_vio_channel {
> +	spinlock_t lock;
> +	spinlock_t ready_lock;
> +	struct virtqueue *vqueue;
> +	struct scmi_chan_info *cinfo;
> +	struct list_head free_list;
> +	u8 is_rx;
> +	u8 ready;
> +};
> +
> +/**
> + * struct scmi_vio_msg - Transport PDU information
> + *
> + * @request: SDU used for commands
> + * @input: SDU used for (delayed) responses and notifications
> + * @list: List which scmi_vio_msg may be part of
> + * @rx_len: Input SDU size in bytes, once input has been received
> + */
> +struct scmi_vio_msg {
> +	struct scmi_msg_payld *request;
> +	struct scmi_msg_payld *input;
> +	struct list_head list;
> +	unsigned int rx_len;
> +};
> +
> +static bool scmi_vio_have_vq_rx(struct virtio_device *vdev)
> +{
> +	return virtio_has_feature(vdev, VIRTIO_SCMI_F_P2A_CHANNELS);
> +}
> +
> +static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
> +			       struct scmi_vio_msg *msg)
> +{
> +	struct scatterlist sg_in;
> +	int rc;
> +	unsigned long flags;
> +
> +	sg_init_one(&sg_in, msg->input, VIRTIO_SCMI_MAX_PDU_SIZE);
> +
> +	spin_lock_irqsave(&vioch->lock, flags);
> +
> +	rc = virtqueue_add_inbuf(vioch->vqueue, &sg_in, 1, msg, GFP_ATOMIC);
> +	if (rc)
> +		dev_err_once(vioch->cinfo->dev,
> +			     "%s() failed to add to virtqueue (%d)\n", __func__,
> +			     rc);
> +	else
> +		virtqueue_kick(vioch->vqueue);
> +
> +	spin_unlock_irqrestore(&vioch->lock, flags);
> +
> +	return rc;
> +}
> +
> +static void scmi_vio_complete_cb(struct virtqueue *vqueue)
> +{
> +	unsigned long ready_flags;
> +	unsigned long flags;
> +	unsigned int length;
> +	struct scmi_vio_channel *vioch;
> +	struct scmi_vio_msg *msg;
> +	bool cb_enabled = true;
> +
> +	if (WARN_ON_ONCE(!vqueue->vdev->priv))
> +		return;
> +	vioch = &((struct scmi_vio_channel *)vqueue->vdev->priv)[vqueue->index];
> +
> +	for (;;) {
> +		spin_lock_irqsave(&vioch->ready_lock, ready_flags);
> +
> +		if (!vioch->ready) {
> +			if (!cb_enabled)
> +				(void)virtqueue_enable_cb(vqueue);
> +			goto unlock_ready_out;
> +		}
> +
> +		spin_lock_irqsave(&vioch->lock, flags);
> +		if (cb_enabled) {
> +			virtqueue_disable_cb(vqueue);
> +			cb_enabled = false;
> +		}
> +		msg = virtqueue_get_buf(vqueue, &length);
> +		if (!msg) {
> +			if (virtqueue_enable_cb(vqueue))
> +				goto unlock_out;
> +			else

Drop the else as it doesn't add readability as far as I can see.

> +				cb_enabled = true;
> +		}
> +		spin_unlock_irqrestore(&vioch->lock, flags);
> +
> +		if (msg) {
> +			msg->rx_len = length;
> +
> +			/*
> +			 * Hold the ready_lock during the callback to avoid
> +			 * races when the arm-scmi driver is unbinding while
> +			 * the virtio device is not quiesced yet.
> +			 */
> +			scmi_rx_callback(vioch->cinfo,
> +					 msg_read_header(msg->input), msg);
> +		}
> +		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> +	}
> +
> +unlock_out:
> +	spin_unlock_irqrestore(&vioch->lock, flags);
> +unlock_ready_out:
> +	spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> +}
> +
> +static const char *const scmi_vio_vqueue_names[] = { "tx", "rx" };
> +
> +static vq_callback_t *scmi_vio_complete_callbacks[] = {
> +	scmi_vio_complete_cb,
> +	scmi_vio_complete_cb
> +};
> +
> +static unsigned int virtio_get_max_msg(bool tx,
> +				       struct scmi_chan_info *base_cinfo)
> +{
> +	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
> +	unsigned int ret;
> +
> +	ret = virtqueue_get_vring_size(vioch->vqueue);
> +
> +	/* Tx messages need multiple descriptors. */
> +	if (tx)
> +		ret /= DESCRIPTORS_PER_TX_MSG;
> +
> +	if (ret > MSG_TOKEN_MAX) {
> +		dev_info_once(
> +			base_cinfo->dev,
> +			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
> +			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
> +		ret = MSG_TOKEN_MAX;
> +	}
> +
> +	return ret;
> +}
> +
> +static int scmi_vio_match_any_dev(struct device *dev, const void *data)
> +{
> +	return 1;
> +}
> +
> +static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
> +
> +static int virtio_link_supplier(struct device *dev)
> +{
> +	struct device *vdev = driver_find_device(
> +		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
> +
> +	if (!vdev) {
> +		dev_notice_once(
> +			dev,
> +			"Deferring probe after not finding a bound scmi-virtio device\n");
> +		return -EPROBE_DEFER;
> +	}
> +
> +	/* Add device link for remove order and sysfs link. */
> +	if (!device_link_add(dev, vdev, DL_FLAG_AUTOREMOVE_CONSUMER)) {
> +		put_device(vdev);
> +		dev_err(dev, "Adding link to supplier virtio device failed\n");
> +		return -ECANCELED;
> +	}
> +
> +	put_device(vdev);
> +	return scmi_set_transport_info(dev, dev_to_virtio(vdev));
> +}
> +
> +static bool virtio_chan_available(struct device *dev, int idx)
> +{
> +	struct virtio_device *vdev;
> +
> +	/* scmi-virtio doesn't support per-protocol channels */
> +	if (is_scmi_protocol_device(dev))
> +		return false;
> +
> +	vdev = scmi_get_transport_info(dev);
> +	if (!vdev)
> +		return false;
> +
> +	switch (idx) {
> +	case VIRTIO_SCMI_VQ_TX:
> +		return true;
> +	case VIRTIO_SCMI_VQ_RX:
> +		return scmi_vio_have_vq_rx(vdev);
> +	default:
> +		return false;
> +	}
> +}
> +
> +static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> +			     bool tx)
> +{
> +	unsigned long flags;
> +	struct virtio_device *vdev;
> +	struct scmi_vio_channel *vioch;
> +	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
> +	int max_msg;
> +	int i;
> +
> +	if (!virtio_chan_available(dev, index))
> +		return -ENODEV;
> +
> +	vdev = scmi_get_transport_info(dev);
> +	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
> +
> +	spin_lock_irqsave(&vioch->lock, flags);
> +	cinfo->transport_info = vioch;
> +	vioch->cinfo = cinfo;
> +	spin_unlock_irqrestore(&vioch->lock, flags);
> +
> +	max_msg = virtio_get_max_msg(tx, cinfo);
> +
> +	for (i = 0; i < max_msg; i++) {
> +		struct scmi_vio_msg *msg;
> +
> +		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
> +		if (!msg)
> +			return -ENOMEM;
> +
> +		if (tx) {
> +			msg->request = devm_kzalloc(cinfo->dev,
> +						    VIRTIO_SCMI_MAX_PDU_SIZE,
> +						    GFP_KERNEL);
> +			if (!msg->request)
> +				return -ENOMEM;
> +		}
> +
> +		msg->input = devm_kzalloc(cinfo->dev, VIRTIO_SCMI_MAX_PDU_SIZE,
> +					  GFP_KERNEL);
> +		if (!msg->input)
> +			return -ENOMEM;
> +
> +		if (tx) {
> +			spin_lock_irqsave(&vioch->lock, flags);
> +			list_add_tail(&msg->list, &vioch->free_list);
> +			spin_unlock_irqrestore(&vioch->lock, flags);
> +		} else {
> +			scmi_vio_feed_vq_rx(vioch, msg);
> +		}
> +	}
> +
> +	spin_lock_irqsave(&vioch->ready_lock, flags);
> +	vioch->ready = true;
> +	spin_unlock_irqrestore(&vioch->ready_lock, flags);
> +
> +	return 0;
> +}
> +
> +static int virtio_chan_free(int id, void *p, void *data)
> +{
> +	unsigned long flags;
> +	struct scmi_chan_info *cinfo = p;
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +
> +	spin_lock_irqsave(&vioch->ready_lock, flags);
> +	vioch->ready = false;
> +	spin_unlock_irqrestore(&vioch->ready_lock, flags);
> +
> +	scmi_free_channel(cinfo, data, id);

Blank line here would be nice + consistent.

> +	return 0;
> +}
> +
> +static int virtio_send_message(struct scmi_chan_info *cinfo,
> +			       struct scmi_xfer *xfer)
> +{
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +	struct scatterlist sg_out;
> +	struct scatterlist sg_in;
> +	struct scatterlist *sgs[DESCRIPTORS_PER_TX_MSG] = { &sg_out, &sg_in };
> +	unsigned long flags;
> +	int rc;
> +	struct scmi_vio_msg *msg;
> +
> +	/*
> +	 * TODO: For now, we don't support polling. But it should not be
> +	 * difficult to add support.
> +	 */
> +	if (xfer->hdr.poll_completion)
> +		return -EINVAL;
> +
> +	spin_lock_irqsave(&vioch->lock, flags);
> +
> +	if (list_empty(&vioch->free_list)) {
> +		spin_unlock_irqrestore(&vioch->lock, flags);
> +		return -EBUSY;
> +	}
> +
> +	msg = list_first_entry(&vioch->free_list, typeof(*msg), list);
> +	list_del(&msg->list);
> +
> +	msg_tx_prepare(msg->request, xfer);
> +
> +	sg_init_one(&sg_out, msg->request, msg_command_size(xfer));
> +	sg_init_one(&sg_in, msg->input, msg_response_size(xfer));
> +
> +	rc = virtqueue_add_sgs(vioch->vqueue, sgs, 1, 1, msg, GFP_ATOMIC);
> +	if (rc) {
> +		list_add(&msg->list, &vioch->free_list);
> +		dev_err_once(vioch->cinfo->dev,
> +			     "%s() failed to add to virtqueue (%d)\n", __func__,
> +			     rc);
> +	} else {
> +		virtqueue_kick(vioch->vqueue);
> +	}
> +
> +	spin_unlock_irqrestore(&vioch->lock, flags);
> +
> +	return rc;
> +}
> +
> +static void virtio_fetch_response(struct scmi_chan_info *cinfo,
> +				  struct scmi_xfer *xfer, void *msg_handle)
> +{
> +	struct scmi_vio_msg *msg = msg_handle;
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +
> +	if (!msg) {
> +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> +			     "Ignoring %s() call with NULL msg_handle\n",
> +			     __func__);
> +		return;
> +	}
> +
> +	msg_fetch_response(msg->input, msg->rx_len, xfer);
> +}
> +
> +static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
> +				      size_t max_len, struct scmi_xfer *xfer,
> +				      void *msg_handle)
> +{
> +	struct scmi_vio_msg *msg = msg_handle;
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +
> +	if (!msg) {
> +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> +			     "Ignoring %s() call with NULL msg_handle\n",
> +			     __func__);
> +		return;
> +	}
> +
> +	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
> +}
> +
> +static void dummy_clear_channel(struct scmi_chan_info *cinfo)
> +{
> +}
> +
> +static bool dummy_poll_done(struct scmi_chan_info *cinfo,
> +			    struct scmi_xfer *xfer)
> +{
> +	return false;
> +}
> +
> +static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
> +{
> +	unsigned long flags;
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +	struct scmi_vio_msg *msg = msg_handle;
> +
> +	if (!msg) {
> +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> +			     "Ignoring %s() call with NULL msg_handle\n",
> +			     __func__);
> +		return;
> +	}
> +
> +	if (vioch->is_rx) {
> +		scmi_vio_feed_vq_rx(vioch, msg);
> +	} else {
> +		spin_lock_irqsave(&vioch->lock, flags);
> +		list_add(&msg->list, &vioch->free_list);
> +		spin_unlock_irqrestore(&vioch->lock, flags);
> +	}
> +}
> +
> +static const struct scmi_transport_ops scmi_virtio_ops = {
> +	.link_supplier = virtio_link_supplier,
> +	.chan_available = virtio_chan_available,
> +	.chan_setup = virtio_chan_setup,
> +	.chan_free = virtio_chan_free,
> +	.get_max_msg = virtio_get_max_msg,
> +	.send_message = virtio_send_message,
> +	.fetch_response = virtio_fetch_response,
> +	.fetch_notification = virtio_fetch_notification,
> +	.clear_channel = dummy_clear_channel,
> +	.poll_done = dummy_poll_done,
> +	.drop_message = virtio_drop_message,
> +};
> +
> +static int scmi_vio_probe(struct virtio_device *vdev)
> +{
> +	struct device *dev = &vdev->dev;
> +	struct scmi_vio_channel *channels;
> +	bool have_vq_rx;
> +	int vq_cnt;
> +	int i;
> +	int ret;
> +	struct virtqueue *vqs[VIRTIO_SCMI_VQ_MAX_CNT];
> +
> +	have_vq_rx = scmi_vio_have_vq_rx(vdev);
> +	vq_cnt = have_vq_rx ? VIRTIO_SCMI_VQ_MAX_CNT : 1;
> +
> +	channels = devm_kcalloc(dev, vq_cnt, sizeof(*channels), GFP_KERNEL);
> +	if (!channels)
> +		return -ENOMEM;
> +
> +	if (have_vq_rx)
> +		channels[VIRTIO_SCMI_VQ_RX].is_rx = true;
> +
> +	ret = virtio_find_vqs(vdev, vq_cnt, vqs, scmi_vio_complete_callbacks,
> +			      scmi_vio_vqueue_names, NULL);
> +	if (ret) {
> +		dev_err(dev, "Failed to get %d virtqueue(s)\n", vq_cnt);
> +		return ret;
> +	}
> +	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);

Rather noisy given I assume there are plenty of other ways to se this
succeeded (i.e. probe succeeded)

> +
> +	for (i = 0; i < vq_cnt; i++) {
> +		spin_lock_init(&channels[i].lock);
> +		spin_lock_init(&channels[i].ready_lock);
> +		INIT_LIST_HEAD(&channels[i].free_list);
> +		channels[i].vqueue = vqs[i];
> +	}
> +
> +	vdev->priv = channels;
> +
> +	return 0;
> +}
> +
> +static void scmi_vio_remove(struct virtio_device *vdev)
> +{
> +	vdev->config->reset(vdev);
> +	vdev->config->del_vqs(vdev);
> +}
> +
> +static unsigned int features[] = {
> +	VIRTIO_SCMI_F_P2A_CHANNELS,
> +};
> +
> +static const struct virtio_device_id id_table[] = {
> +	{ VIRTIO_ID_SCMI, VIRTIO_DEV_ANY_ID },
> +	{ 0 }
> +};
> +
> +static struct virtio_driver virtio_scmi_driver = {
> +	.driver.name = "scmi-virtio",
> +	.driver.owner = THIS_MODULE,
> +	.feature_table = features,
> +	.feature_table_size = ARRAY_SIZE(features),
> +	.id_table = id_table,
> +	.probe = scmi_vio_probe,
> +	.remove = scmi_vio_remove,
> +};
> +
> +static int __init virtio_scmi_init(void)
> +{
> +	return register_virtio_driver(&virtio_scmi_driver);
> +}
> +
> +static void __exit virtio_scmi_exit(void)
> +{
> +	unregister_virtio_driver(&virtio_scmi_driver);
> +}

module_virtio_driver() ?

> +
> +const struct scmi_desc scmi_virtio_desc = {
> +	.init = virtio_scmi_init,
> +	.exit = virtio_scmi_exit,
> +	.ops = &scmi_virtio_ops,
> +	.max_rx_timeout_ms = 60000, /* for non-realtime virtio devices */
> +	.max_msg = 0, /* overridden by virtio_get_max_msg() */
> +	.max_msg_size = VIRTIO_SCMI_MAX_MSG_SIZE,
> +};
> diff --git a/include/uapi/linux/virtio_ids.h b/include/uapi/linux/virtio_ids.h
> index 4fe842c3a3a9..ead8178efffa 100644
> --- a/include/uapi/linux/virtio_ids.h
> +++ b/include/uapi/linux/virtio_ids.h
> @@ -55,6 +55,7 @@
>  #define VIRTIO_ID_FS			26 /* virtio filesystem */
>  #define VIRTIO_ID_PMEM			27 /* virtio pmem */
>  #define VIRTIO_ID_MAC80211_HWSIM	29 /* virtio mac80211-hwsim */
> +#define VIRTIO_ID_SCMI			32 /* virtio SCMI */
>  #define VIRTIO_ID_BT			40 /* virtio bluetooth */
>  
>  #endif /* _LINUX_VIRTIO_IDS_H */
> diff --git a/include/uapi/linux/virtio_scmi.h b/include/uapi/linux/virtio_scmi.h
> new file mode 100644
> index 000000000000..732b01504c35
> --- /dev/null
> +++ b/include/uapi/linux/virtio_scmi.h
> @@ -0,0 +1,25 @@
> +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
> +/*
> + * Copyright (C) 2020 OpenSynergy GmbH
> + */
> +
> +#ifndef _UAPI_LINUX_VIRTIO_SCMI_H
> +#define _UAPI_LINUX_VIRTIO_SCMI_H
> +
> +#include <linux/virtio_types.h>
> +
> +/* Feature bits */

Drop comment for things that aren't here?

> +
> +/* Device implements some SCMI notifications, or delayed responses. */
> +#define VIRTIO_SCMI_F_P2A_CHANNELS 0
> +
> +/* Device implements any SCMI statistics shared memory region */
> +#define VIRTIO_SCMI_F_SHARED_MEMORY 1
> +
> +/* Virtqueues */
> +
> +#define VIRTIO_SCMI_VQ_TX 0 /* cmdq */
> +#define VIRTIO_SCMI_VQ_RX 1 /* eventq */
> +#define VIRTIO_SCMI_VQ_MAX_CNT 2
> +
> +#endif /* _UAPI_LINUX_VIRTIO_SCMI_H */


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
@ 2021-06-14 14:23     ` Jonathan Cameron
  0 siblings, 0 replies; 90+ messages in thread
From: Jonathan Cameron @ 2021-06-14 14:23 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Fri, 11 Jun 2021 17:59:35 +0100
Cristian Marussi <cristian.marussi@arm.com> wrote:

> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> This transport enables accessing an SCMI platform as a virtio device.
> 
> Implement an SCMI virtio driver according to the virtio SCMI device spec
> [1]. Virtio device id 32 has been reserved for the SCMI device [2].
> 
> The virtio transport has one Tx channel (virtio cmdq, A2P channel) and
> at most one Rx channel (virtio eventq, P2A channel).
> 
> The following feature bit defined in [1] is not implemented:
> VIRTIO_SCMI_F_SHARED_MEMORY.
> 
> After the preparatory patches, this implements the virtio transport as
> paraphrased:
> 
> Only support a single arm-scmi device (which is consistent with the SCMI
> spec). scmi-virtio init is called from arm-scmi module init. During the
> arm-scmi probing, link to the first probed scmi-virtio device. Defer
> arm-scmi probing if no scmi-virtio device is bound yet.
> 
> For simplicity, restrict the number of messages which can be pending
> simultaneously according to the virtqueue capacity. (The virtqueue sizes
> are negotiated with the virtio device.)
> 
> As soon as Rx channel message buffers are allocated or have been read
> out by the arm-scmi driver, feed them to the virtio device.
> 
> Since some virtio devices may not have the short response time exhibited
> by SCMI platforms using other transports, set a generous response
> timeout.
> 
> Limitations:
> 
> - Polling is not supported.
> 
> - The timeout for delayed responses has not been adjusted.
> 
> [1] https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-scmi.tex
> [2] https://www.oasis-open.org/committees/ballot.php?id=3496
> 
> [ Peter: Adapted patch for submission to upstream. ]
> 
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>

A few drive by comments inline.

J
> ---
>  MAINTAINERS                        |   1 +
>  drivers/firmware/Kconfig           |  12 +
>  drivers/firmware/arm_scmi/Makefile |   1 +
>  drivers/firmware/arm_scmi/common.h |   3 +
>  drivers/firmware/arm_scmi/driver.c |   3 +
>  drivers/firmware/arm_scmi/virtio.c | 523 +++++++++++++++++++++++++++++
>  include/uapi/linux/virtio_ids.h    |   1 +
>  include/uapi/linux/virtio_scmi.h   |  25 ++
>  8 files changed, 569 insertions(+)
>  create mode 100644 drivers/firmware/arm_scmi/virtio.c
>  create mode 100644 include/uapi/linux/virtio_scmi.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index f408cf2d2781..e1b27ed11060 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -17878,6 +17878,7 @@ F:	drivers/regulator/scmi-regulator.c
>  F:	drivers/reset/reset-scmi.c
>  F:	include/linux/sc[mp]i_protocol.h
>  F:	include/trace/events/scmi.h
> +F:	include/uapi/linux/virtio_scmi.h
>  
>  SYSTEM RESET/SHUTDOWN DRIVERS
>  M:	Sebastian Reichel <sre@kernel.org>
> diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
> index e74d25065d07..2c37d5af66ad 100644
> --- a/drivers/firmware/Kconfig
> +++ b/drivers/firmware/Kconfig
> @@ -39,6 +39,18 @@ config ARM_SCMI_HAVE_MSG
>  	  This declares whether a message passing based transport for SCMI is
>  	  available.
>  
> +# This config option includes the dependencies of ARM_SCMI_PROTOCOL so that
> +# this config doesn't show up when SCMI wouldn't be available.
> +config VIRTIO_SCMI
> +	bool "Virtio transport for SCMI"
> +	select ARM_SCMI_HAVE_MSG
> +	depends on VIRTIO && (ARM || ARM64 || COMPILE_TEST)
> +	help
> +	  This enables the virtio based transport for SCMI.
> +
> +	  If you want to use the ARM SCMI protocol between the virtio guest and
> +	  a host providing a virtio SCMI device, answer Y.
> +
>  config ARM_SCMI_POWER_DOMAIN
>  	tristate "SCMI power domain driver"
>  	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
> diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
> index f6b4acb8abdb..db1787606fb2 100644
> --- a/drivers/firmware/arm_scmi/Makefile
> +++ b/drivers/firmware/arm_scmi/Makefile
> @@ -5,6 +5,7 @@ scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
>  scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
>  scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
>  scmi-transport-$(CONFIG_ARM_SCMI_HAVE_MSG) += msg.o
> +scmi-transport-$(CONFIG_VIRTIO_SCMI) += virtio.o
>  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
>  scmi-module-objs := $(scmi-bus-y) $(scmi-driver-y) $(scmi-protocols-y) \
>  		    $(scmi-transport-y)
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index fa4075336580..c143a449d278 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -390,6 +390,9 @@ extern const struct scmi_desc scmi_mailbox_desc;
>  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
>  extern const struct scmi_desc scmi_smc_desc;
>  #endif
> +#ifdef CONFIG_VIRTIO_SCMI
> +extern const struct scmi_desc scmi_virtio_desc;
> +#endif
>  
>  int scmi_set_transport_info(struct device *dev, void *transport_info);
>  void *scmi_get_transport_info(struct device *dev);
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index bb2de15554a9..ccc7dd49261e 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -1954,6 +1954,9 @@ static const struct of_device_id scmi_of_match[] = {
>  #endif
>  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
>  	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
> +#endif
> +#ifdef CONFIG_VIRTIO_SCMI
> +	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
>  #endif
>  	{ /* Sentinel */ },
>  };
> diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
> new file mode 100644
> index 000000000000..20972adf6dc7
> --- /dev/null
> +++ b/drivers/firmware/arm_scmi/virtio.c
> @@ -0,0 +1,523 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Virtio Transport driver for Arm System Control and Management Interface
> + * (SCMI).
> + *
> + * Copyright (C) 2020 OpenSynergy.
> + */
> +
> +/**
> + * DOC: Theory of Operation
> + *
> + * The scmi-virtio transport implements a driver for the virtio SCMI device.
> + *
> + * There is one Tx channel (virtio cmdq, A2P channel) and at most one Rx
> + * channel (virtio eventq, P2A channel). Each channel is implemented through a
> + * virtqueue. Access to each virtqueue is protected by spinlocks.
> + */
> +
> +#include <linux/errno.h>

> +#include <linux/of.h>
> +#include <linux/of_platform.h>
Why either of these?

perhaps
mod_devicetable.h is appropriate.

> +#include <linux/platform_device.h>
> +#include <linux/module.h>
> +#include <linux/slab.h>
> +#include <linux/virtio.h>
> +#include <linux/virtio_config.h>
> +#include <uapi/linux/virtio_ids.h>
> +#include <uapi/linux/virtio_scmi.h>
> +
> +#include "common.h"
> +
> +#define VIRTIO_SCMI_MAX_MSG_SIZE 128 /* Value may be increased. */
> +#define VIRTIO_SCMI_MAX_PDU_SIZE \
> +	(VIRTIO_SCMI_MAX_MSG_SIZE + SCMI_MSG_MAX_PROT_OVERHEAD)
> +#define DESCRIPTORS_PER_TX_MSG 2
> +
> +/**
> + * struct scmi_vio_channel - Transport channel information
> + *
> + * @lock: Protects access to all members except ready.
> + * @ready_lock: Protects access to ready. If required, it must be taken before
> + *              lock.
> + * @vqueue: Associated virtqueue
> + * @cinfo: SCMI Tx or Rx channel
> + * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
> + * @is_rx: Whether channel is an Rx channel
> + * @ready: Whether transport user is ready to hear about channel
> + */
> +struct scmi_vio_channel {
> +	spinlock_t lock;
> +	spinlock_t ready_lock;
> +	struct virtqueue *vqueue;
> +	struct scmi_chan_info *cinfo;
> +	struct list_head free_list;
> +	u8 is_rx;
> +	u8 ready;
> +};
> +
> +/**
> + * struct scmi_vio_msg - Transport PDU information
> + *
> + * @request: SDU used for commands
> + * @input: SDU used for (delayed) responses and notifications
> + * @list: List which scmi_vio_msg may be part of
> + * @rx_len: Input SDU size in bytes, once input has been received
> + */
> +struct scmi_vio_msg {
> +	struct scmi_msg_payld *request;
> +	struct scmi_msg_payld *input;
> +	struct list_head list;
> +	unsigned int rx_len;
> +};
> +
> +static bool scmi_vio_have_vq_rx(struct virtio_device *vdev)
> +{
> +	return virtio_has_feature(vdev, VIRTIO_SCMI_F_P2A_CHANNELS);
> +}
> +
> +static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
> +			       struct scmi_vio_msg *msg)
> +{
> +	struct scatterlist sg_in;
> +	int rc;
> +	unsigned long flags;
> +
> +	sg_init_one(&sg_in, msg->input, VIRTIO_SCMI_MAX_PDU_SIZE);
> +
> +	spin_lock_irqsave(&vioch->lock, flags);
> +
> +	rc = virtqueue_add_inbuf(vioch->vqueue, &sg_in, 1, msg, GFP_ATOMIC);
> +	if (rc)
> +		dev_err_once(vioch->cinfo->dev,
> +			     "%s() failed to add to virtqueue (%d)\n", __func__,
> +			     rc);
> +	else
> +		virtqueue_kick(vioch->vqueue);
> +
> +	spin_unlock_irqrestore(&vioch->lock, flags);
> +
> +	return rc;
> +}
> +
> +static void scmi_vio_complete_cb(struct virtqueue *vqueue)
> +{
> +	unsigned long ready_flags;
> +	unsigned long flags;
> +	unsigned int length;
> +	struct scmi_vio_channel *vioch;
> +	struct scmi_vio_msg *msg;
> +	bool cb_enabled = true;
> +
> +	if (WARN_ON_ONCE(!vqueue->vdev->priv))
> +		return;
> +	vioch = &((struct scmi_vio_channel *)vqueue->vdev->priv)[vqueue->index];
> +
> +	for (;;) {
> +		spin_lock_irqsave(&vioch->ready_lock, ready_flags);
> +
> +		if (!vioch->ready) {
> +			if (!cb_enabled)
> +				(void)virtqueue_enable_cb(vqueue);
> +			goto unlock_ready_out;
> +		}
> +
> +		spin_lock_irqsave(&vioch->lock, flags);
> +		if (cb_enabled) {
> +			virtqueue_disable_cb(vqueue);
> +			cb_enabled = false;
> +		}
> +		msg = virtqueue_get_buf(vqueue, &length);
> +		if (!msg) {
> +			if (virtqueue_enable_cb(vqueue))
> +				goto unlock_out;
> +			else

Drop the else as it doesn't add readability as far as I can see.

> +				cb_enabled = true;
> +		}
> +		spin_unlock_irqrestore(&vioch->lock, flags);
> +
> +		if (msg) {
> +			msg->rx_len = length;
> +
> +			/*
> +			 * Hold the ready_lock during the callback to avoid
> +			 * races when the arm-scmi driver is unbinding while
> +			 * the virtio device is not quiesced yet.
> +			 */
> +			scmi_rx_callback(vioch->cinfo,
> +					 msg_read_header(msg->input), msg);
> +		}
> +		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> +	}
> +
> +unlock_out:
> +	spin_unlock_irqrestore(&vioch->lock, flags);
> +unlock_ready_out:
> +	spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> +}
> +
> +static const char *const scmi_vio_vqueue_names[] = { "tx", "rx" };
> +
> +static vq_callback_t *scmi_vio_complete_callbacks[] = {
> +	scmi_vio_complete_cb,
> +	scmi_vio_complete_cb
> +};
> +
> +static unsigned int virtio_get_max_msg(bool tx,
> +				       struct scmi_chan_info *base_cinfo)
> +{
> +	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
> +	unsigned int ret;
> +
> +	ret = virtqueue_get_vring_size(vioch->vqueue);
> +
> +	/* Tx messages need multiple descriptors. */
> +	if (tx)
> +		ret /= DESCRIPTORS_PER_TX_MSG;
> +
> +	if (ret > MSG_TOKEN_MAX) {
> +		dev_info_once(
> +			base_cinfo->dev,
> +			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
> +			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
> +		ret = MSG_TOKEN_MAX;
> +	}
> +
> +	return ret;
> +}
> +
> +static int scmi_vio_match_any_dev(struct device *dev, const void *data)
> +{
> +	return 1;
> +}
> +
> +static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
> +
> +static int virtio_link_supplier(struct device *dev)
> +{
> +	struct device *vdev = driver_find_device(
> +		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
> +
> +	if (!vdev) {
> +		dev_notice_once(
> +			dev,
> +			"Deferring probe after not finding a bound scmi-virtio device\n");
> +		return -EPROBE_DEFER;
> +	}
> +
> +	/* Add device link for remove order and sysfs link. */
> +	if (!device_link_add(dev, vdev, DL_FLAG_AUTOREMOVE_CONSUMER)) {
> +		put_device(vdev);
> +		dev_err(dev, "Adding link to supplier virtio device failed\n");
> +		return -ECANCELED;
> +	}
> +
> +	put_device(vdev);
> +	return scmi_set_transport_info(dev, dev_to_virtio(vdev));
> +}
> +
> +static bool virtio_chan_available(struct device *dev, int idx)
> +{
> +	struct virtio_device *vdev;
> +
> +	/* scmi-virtio doesn't support per-protocol channels */
> +	if (is_scmi_protocol_device(dev))
> +		return false;
> +
> +	vdev = scmi_get_transport_info(dev);
> +	if (!vdev)
> +		return false;
> +
> +	switch (idx) {
> +	case VIRTIO_SCMI_VQ_TX:
> +		return true;
> +	case VIRTIO_SCMI_VQ_RX:
> +		return scmi_vio_have_vq_rx(vdev);
> +	default:
> +		return false;
> +	}
> +}
> +
> +static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> +			     bool tx)
> +{
> +	unsigned long flags;
> +	struct virtio_device *vdev;
> +	struct scmi_vio_channel *vioch;
> +	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
> +	int max_msg;
> +	int i;
> +
> +	if (!virtio_chan_available(dev, index))
> +		return -ENODEV;
> +
> +	vdev = scmi_get_transport_info(dev);
> +	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
> +
> +	spin_lock_irqsave(&vioch->lock, flags);
> +	cinfo->transport_info = vioch;
> +	vioch->cinfo = cinfo;
> +	spin_unlock_irqrestore(&vioch->lock, flags);
> +
> +	max_msg = virtio_get_max_msg(tx, cinfo);
> +
> +	for (i = 0; i < max_msg; i++) {
> +		struct scmi_vio_msg *msg;
> +
> +		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
> +		if (!msg)
> +			return -ENOMEM;
> +
> +		if (tx) {
> +			msg->request = devm_kzalloc(cinfo->dev,
> +						    VIRTIO_SCMI_MAX_PDU_SIZE,
> +						    GFP_KERNEL);
> +			if (!msg->request)
> +				return -ENOMEM;
> +		}
> +
> +		msg->input = devm_kzalloc(cinfo->dev, VIRTIO_SCMI_MAX_PDU_SIZE,
> +					  GFP_KERNEL);
> +		if (!msg->input)
> +			return -ENOMEM;
> +
> +		if (tx) {
> +			spin_lock_irqsave(&vioch->lock, flags);
> +			list_add_tail(&msg->list, &vioch->free_list);
> +			spin_unlock_irqrestore(&vioch->lock, flags);
> +		} else {
> +			scmi_vio_feed_vq_rx(vioch, msg);
> +		}
> +	}
> +
> +	spin_lock_irqsave(&vioch->ready_lock, flags);
> +	vioch->ready = true;
> +	spin_unlock_irqrestore(&vioch->ready_lock, flags);
> +
> +	return 0;
> +}
> +
> +static int virtio_chan_free(int id, void *p, void *data)
> +{
> +	unsigned long flags;
> +	struct scmi_chan_info *cinfo = p;
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +
> +	spin_lock_irqsave(&vioch->ready_lock, flags);
> +	vioch->ready = false;
> +	spin_unlock_irqrestore(&vioch->ready_lock, flags);
> +
> +	scmi_free_channel(cinfo, data, id);

Blank line here would be nice + consistent.

> +	return 0;
> +}
> +
> +static int virtio_send_message(struct scmi_chan_info *cinfo,
> +			       struct scmi_xfer *xfer)
> +{
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +	struct scatterlist sg_out;
> +	struct scatterlist sg_in;
> +	struct scatterlist *sgs[DESCRIPTORS_PER_TX_MSG] = { &sg_out, &sg_in };
> +	unsigned long flags;
> +	int rc;
> +	struct scmi_vio_msg *msg;
> +
> +	/*
> +	 * TODO: For now, we don't support polling. But it should not be
> +	 * difficult to add support.
> +	 */
> +	if (xfer->hdr.poll_completion)
> +		return -EINVAL;
> +
> +	spin_lock_irqsave(&vioch->lock, flags);
> +
> +	if (list_empty(&vioch->free_list)) {
> +		spin_unlock_irqrestore(&vioch->lock, flags);
> +		return -EBUSY;
> +	}
> +
> +	msg = list_first_entry(&vioch->free_list, typeof(*msg), list);
> +	list_del(&msg->list);
> +
> +	msg_tx_prepare(msg->request, xfer);
> +
> +	sg_init_one(&sg_out, msg->request, msg_command_size(xfer));
> +	sg_init_one(&sg_in, msg->input, msg_response_size(xfer));
> +
> +	rc = virtqueue_add_sgs(vioch->vqueue, sgs, 1, 1, msg, GFP_ATOMIC);
> +	if (rc) {
> +		list_add(&msg->list, &vioch->free_list);
> +		dev_err_once(vioch->cinfo->dev,
> +			     "%s() failed to add to virtqueue (%d)\n", __func__,
> +			     rc);
> +	} else {
> +		virtqueue_kick(vioch->vqueue);
> +	}
> +
> +	spin_unlock_irqrestore(&vioch->lock, flags);
> +
> +	return rc;
> +}
> +
> +static void virtio_fetch_response(struct scmi_chan_info *cinfo,
> +				  struct scmi_xfer *xfer, void *msg_handle)
> +{
> +	struct scmi_vio_msg *msg = msg_handle;
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +
> +	if (!msg) {
> +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> +			     "Ignoring %s() call with NULL msg_handle\n",
> +			     __func__);
> +		return;
> +	}
> +
> +	msg_fetch_response(msg->input, msg->rx_len, xfer);
> +}
> +
> +static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
> +				      size_t max_len, struct scmi_xfer *xfer,
> +				      void *msg_handle)
> +{
> +	struct scmi_vio_msg *msg = msg_handle;
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +
> +	if (!msg) {
> +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> +			     "Ignoring %s() call with NULL msg_handle\n",
> +			     __func__);
> +		return;
> +	}
> +
> +	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
> +}
> +
> +static void dummy_clear_channel(struct scmi_chan_info *cinfo)
> +{
> +}
> +
> +static bool dummy_poll_done(struct scmi_chan_info *cinfo,
> +			    struct scmi_xfer *xfer)
> +{
> +	return false;
> +}
> +
> +static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
> +{
> +	unsigned long flags;
> +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> +	struct scmi_vio_msg *msg = msg_handle;
> +
> +	if (!msg) {
> +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> +			     "Ignoring %s() call with NULL msg_handle\n",
> +			     __func__);
> +		return;
> +	}
> +
> +	if (vioch->is_rx) {
> +		scmi_vio_feed_vq_rx(vioch, msg);
> +	} else {
> +		spin_lock_irqsave(&vioch->lock, flags);
> +		list_add(&msg->list, &vioch->free_list);
> +		spin_unlock_irqrestore(&vioch->lock, flags);
> +	}
> +}
> +
> +static const struct scmi_transport_ops scmi_virtio_ops = {
> +	.link_supplier = virtio_link_supplier,
> +	.chan_available = virtio_chan_available,
> +	.chan_setup = virtio_chan_setup,
> +	.chan_free = virtio_chan_free,
> +	.get_max_msg = virtio_get_max_msg,
> +	.send_message = virtio_send_message,
> +	.fetch_response = virtio_fetch_response,
> +	.fetch_notification = virtio_fetch_notification,
> +	.clear_channel = dummy_clear_channel,
> +	.poll_done = dummy_poll_done,
> +	.drop_message = virtio_drop_message,
> +};
> +
> +static int scmi_vio_probe(struct virtio_device *vdev)
> +{
> +	struct device *dev = &vdev->dev;
> +	struct scmi_vio_channel *channels;
> +	bool have_vq_rx;
> +	int vq_cnt;
> +	int i;
> +	int ret;
> +	struct virtqueue *vqs[VIRTIO_SCMI_VQ_MAX_CNT];
> +
> +	have_vq_rx = scmi_vio_have_vq_rx(vdev);
> +	vq_cnt = have_vq_rx ? VIRTIO_SCMI_VQ_MAX_CNT : 1;
> +
> +	channels = devm_kcalloc(dev, vq_cnt, sizeof(*channels), GFP_KERNEL);
> +	if (!channels)
> +		return -ENOMEM;
> +
> +	if (have_vq_rx)
> +		channels[VIRTIO_SCMI_VQ_RX].is_rx = true;
> +
> +	ret = virtio_find_vqs(vdev, vq_cnt, vqs, scmi_vio_complete_callbacks,
> +			      scmi_vio_vqueue_names, NULL);
> +	if (ret) {
> +		dev_err(dev, "Failed to get %d virtqueue(s)\n", vq_cnt);
> +		return ret;
> +	}
> +	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);

Rather noisy given I assume there are plenty of other ways to se this
succeeded (i.e. probe succeeded)

> +
> +	for (i = 0; i < vq_cnt; i++) {
> +		spin_lock_init(&channels[i].lock);
> +		spin_lock_init(&channels[i].ready_lock);
> +		INIT_LIST_HEAD(&channels[i].free_list);
> +		channels[i].vqueue = vqs[i];
> +	}
> +
> +	vdev->priv = channels;
> +
> +	return 0;
> +}
> +
> +static void scmi_vio_remove(struct virtio_device *vdev)
> +{
> +	vdev->config->reset(vdev);
> +	vdev->config->del_vqs(vdev);
> +}
> +
> +static unsigned int features[] = {
> +	VIRTIO_SCMI_F_P2A_CHANNELS,
> +};
> +
> +static const struct virtio_device_id id_table[] = {
> +	{ VIRTIO_ID_SCMI, VIRTIO_DEV_ANY_ID },
> +	{ 0 }
> +};
> +
> +static struct virtio_driver virtio_scmi_driver = {
> +	.driver.name = "scmi-virtio",
> +	.driver.owner = THIS_MODULE,
> +	.feature_table = features,
> +	.feature_table_size = ARRAY_SIZE(features),
> +	.id_table = id_table,
> +	.probe = scmi_vio_probe,
> +	.remove = scmi_vio_remove,
> +};
> +
> +static int __init virtio_scmi_init(void)
> +{
> +	return register_virtio_driver(&virtio_scmi_driver);
> +}
> +
> +static void __exit virtio_scmi_exit(void)
> +{
> +	unregister_virtio_driver(&virtio_scmi_driver);
> +}

module_virtio_driver() ?

> +
> +const struct scmi_desc scmi_virtio_desc = {
> +	.init = virtio_scmi_init,
> +	.exit = virtio_scmi_exit,
> +	.ops = &scmi_virtio_ops,
> +	.max_rx_timeout_ms = 60000, /* for non-realtime virtio devices */
> +	.max_msg = 0, /* overridden by virtio_get_max_msg() */
> +	.max_msg_size = VIRTIO_SCMI_MAX_MSG_SIZE,
> +};
> diff --git a/include/uapi/linux/virtio_ids.h b/include/uapi/linux/virtio_ids.h
> index 4fe842c3a3a9..ead8178efffa 100644
> --- a/include/uapi/linux/virtio_ids.h
> +++ b/include/uapi/linux/virtio_ids.h
> @@ -55,6 +55,7 @@
>  #define VIRTIO_ID_FS			26 /* virtio filesystem */
>  #define VIRTIO_ID_PMEM			27 /* virtio pmem */
>  #define VIRTIO_ID_MAC80211_HWSIM	29 /* virtio mac80211-hwsim */
> +#define VIRTIO_ID_SCMI			32 /* virtio SCMI */
>  #define VIRTIO_ID_BT			40 /* virtio bluetooth */
>  
>  #endif /* _LINUX_VIRTIO_IDS_H */
> diff --git a/include/uapi/linux/virtio_scmi.h b/include/uapi/linux/virtio_scmi.h
> new file mode 100644
> index 000000000000..732b01504c35
> --- /dev/null
> +++ b/include/uapi/linux/virtio_scmi.h
> @@ -0,0 +1,25 @@
> +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
> +/*
> + * Copyright (C) 2020 OpenSynergy GmbH
> + */
> +
> +#ifndef _UAPI_LINUX_VIRTIO_SCMI_H
> +#define _UAPI_LINUX_VIRTIO_SCMI_H
> +
> +#include <linux/virtio_types.h>
> +
> +/* Feature bits */

Drop comment for things that aren't here?

> +
> +/* Device implements some SCMI notifications, or delayed responses. */
> +#define VIRTIO_SCMI_F_P2A_CHANNELS 0
> +
> +/* Device implements any SCMI statistics shared memory region */
> +#define VIRTIO_SCMI_F_SHARED_MEMORY 1
> +
> +/* Virtqueues */
> +
> +#define VIRTIO_SCMI_VQ_TX 0 /* cmdq */
> +#define VIRTIO_SCMI_VQ_RX 1 /* eventq */
> +#define VIRTIO_SCMI_VQ_MAX_CNT 2
> +
> +#endif /* _UAPI_LINUX_VIRTIO_SCMI_H */


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 03/16] firmware: arm_scmi: Add transport optional init/exit support
  2021-06-14 13:29     ` Jonathan Cameron
@ 2021-06-16  9:04       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16  9:04 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi Jonathan,

thanks for having a look at this series.

On Mon, Jun 14, 2021 at 02:29:56PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:24 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > Some SCMI transport could need to perform some transport specific setup
> > before they can be used by the SCMI core transport layer: typically this
> > early setup consists in registering with some other kernel subsystem.
> > 
> > Add the optional capability for a transport to provide a couple of .init
> > and .exit functions that are assured to be called early during the SCMI
> > core initialization phase, well before the SCMI core probing step.
> > 
> > [ Peter: Adapted RFC patch by Cristian for submission to upstream. ]
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> > [ Cristian: Fixed scmi_transports_exit point of invocation ]
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> 
> Drive by comment inline.  Feel free to ignore ;)
> 
> Jonathan
> 
> > ---
> >  drivers/firmware/arm_scmi/common.h |  8 ++++
> >  drivers/firmware/arm_scmi/driver.c | 59 ++++++++++++++++++++++++++++++
> >  2 files changed, 67 insertions(+)
> > 
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index 7c2b9fd7e929..6bb734e0e3ac 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -321,6 +321,12 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
> >  /**
> >   * struct scmi_desc - Description of SoC integration
> >   *
> > + * @init: An optional function that a transport can provide to initialize some
> > + *	  transport-specific setup during SCMI core initialization, so ahead of
> > + *	  SCMI core probing.
> > + * @exit: An optional function that a transport can provide to de-initialize
> > + *	  some transport-specific setup during SCMI core de-initialization, so
> > + *	  after SCMI core removal.
> >   * @ops: Pointer to the transport specific ops structure
> >   * @max_rx_timeout_ms: Timeout for communication with SoC (in Milliseconds)
> >   * @max_msg: Maximum number of messages that can be pending
> > @@ -328,6 +334,8 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
> >   * @max_msg_size: Maximum size of data per message that can be handled.
> >   */
> >  struct scmi_desc {
> > +	int (*init)(void);
> > +	void (*exit)(void);
> >  	const struct scmi_transport_ops *ops;
> >  	int max_rx_timeout_ms;
> >  	int max_msg;
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index f15d75af87ea..20f8f0581f3a 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -1594,10 +1594,67 @@ static struct platform_driver scmi_driver = {
> >  	.remove = scmi_remove,
> >  };
> >  
> > +/**
> > + * __scmi_transports_setup  - Common helper to call transport-specific
> > + * .init/.exit code if provided.
> > + *
> > + * @init: A flag to distinguish between init and exit.
> > + *
> > + * Note that, if provided, we invoke .init/.exit functions for all the
> > + * transports currently compiled in.
> > + *
> > + * Return: 0 on Success.
> > + */
> > +static inline int __scmi_transports_setup(bool init)
> > +{
> > +	int ret = 0;
> > +	const struct of_device_id *trans;
> > +
> > +	for (trans = scmi_of_match; trans->data; trans++) {
> > +		const struct scmi_desc *tdesc = trans->data;
> > +
> > +		if ((init && !tdesc->init) || (!init && !tdesc->exit))
> > +			continue;
> > +
> > +		pr_debug("SCMI %sInitializing %s transport\n",
> > +			 init ? "" : "De-", trans->compatible);
> 
> Clever formatting can makes grepping for messages harder.
> 
> Perhaps
> 		if (init)
> 			pr_debug("SCMI Initializing %s transport\n",
> 				 trans->compatible);
> 		else
> 			pr_debug("SCMI Deinitializing %s transport\n",
> 				 trans->compatible);
> 
> would be nicer even though it burns some lines. Also avoids somewhat
> ugly capitalization : De-Initializing xxx transport.
> 
> You could combine it with the convenient if(init) below
> 

Right, I'll rework to avoid issues grepping.

Thanks,
Cristian


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 03/16] firmware: arm_scmi: Add transport optional init/exit support
@ 2021-06-16  9:04       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16  9:04 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi Jonathan,

thanks for having a look at this series.

On Mon, Jun 14, 2021 at 02:29:56PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:24 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > Some SCMI transport could need to perform some transport specific setup
> > before they can be used by the SCMI core transport layer: typically this
> > early setup consists in registering with some other kernel subsystem.
> > 
> > Add the optional capability for a transport to provide a couple of .init
> > and .exit functions that are assured to be called early during the SCMI
> > core initialization phase, well before the SCMI core probing step.
> > 
> > [ Peter: Adapted RFC patch by Cristian for submission to upstream. ]
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> > [ Cristian: Fixed scmi_transports_exit point of invocation ]
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> 
> Drive by comment inline.  Feel free to ignore ;)
> 
> Jonathan
> 
> > ---
> >  drivers/firmware/arm_scmi/common.h |  8 ++++
> >  drivers/firmware/arm_scmi/driver.c | 59 ++++++++++++++++++++++++++++++
> >  2 files changed, 67 insertions(+)
> > 
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index 7c2b9fd7e929..6bb734e0e3ac 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -321,6 +321,12 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
> >  /**
> >   * struct scmi_desc - Description of SoC integration
> >   *
> > + * @init: An optional function that a transport can provide to initialize some
> > + *	  transport-specific setup during SCMI core initialization, so ahead of
> > + *	  SCMI core probing.
> > + * @exit: An optional function that a transport can provide to de-initialize
> > + *	  some transport-specific setup during SCMI core de-initialization, so
> > + *	  after SCMI core removal.
> >   * @ops: Pointer to the transport specific ops structure
> >   * @max_rx_timeout_ms: Timeout for communication with SoC (in Milliseconds)
> >   * @max_msg: Maximum number of messages that can be pending
> > @@ -328,6 +334,8 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
> >   * @max_msg_size: Maximum size of data per message that can be handled.
> >   */
> >  struct scmi_desc {
> > +	int (*init)(void);
> > +	void (*exit)(void);
> >  	const struct scmi_transport_ops *ops;
> >  	int max_rx_timeout_ms;
> >  	int max_msg;
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index f15d75af87ea..20f8f0581f3a 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -1594,10 +1594,67 @@ static struct platform_driver scmi_driver = {
> >  	.remove = scmi_remove,
> >  };
> >  
> > +/**
> > + * __scmi_transports_setup  - Common helper to call transport-specific
> > + * .init/.exit code if provided.
> > + *
> > + * @init: A flag to distinguish between init and exit.
> > + *
> > + * Note that, if provided, we invoke .init/.exit functions for all the
> > + * transports currently compiled in.
> > + *
> > + * Return: 0 on Success.
> > + */
> > +static inline int __scmi_transports_setup(bool init)
> > +{
> > +	int ret = 0;
> > +	const struct of_device_id *trans;
> > +
> > +	for (trans = scmi_of_match; trans->data; trans++) {
> > +		const struct scmi_desc *tdesc = trans->data;
> > +
> > +		if ((init && !tdesc->init) || (!init && !tdesc->exit))
> > +			continue;
> > +
> > +		pr_debug("SCMI %sInitializing %s transport\n",
> > +			 init ? "" : "De-", trans->compatible);
> 
> Clever formatting can makes grepping for messages harder.
> 
> Perhaps
> 		if (init)
> 			pr_debug("SCMI Initializing %s transport\n",
> 				 trans->compatible);
> 		else
> 			pr_debug("SCMI Deinitializing %s transport\n",
> 				 trans->compatible);
> 
> would be nicer even though it burns some lines. Also avoids somewhat
> ugly capitalization : De-Initializing xxx transport.
> 
> You could combine it with the convenient if(init) below
> 

Right, I'll rework to avoid issues grepping.

Thanks,
Cristian


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
  2021-06-14 13:53     ` Jonathan Cameron
@ 2021-06-16  9:11       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16  9:11 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Mon, Jun 14, 2021 at 02:53:01PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:25 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > Tokens are sequence numbers embedded in the each SCMI message header: they
> > are used to correlate commands with responses (and delayed responses), but
> > their usage and policy of selection is entirely up to the caller (usually
> > the OSPM agent), while they are completely opaque to the callee (SCMI
> > server platform) which merely copies them back from the command into the
> > response message header.
> > This also means that the platform does not, can not and should not enforce
> > any kind of policy on received messages depending on the contained sequence
> > number: platform can perfectly handle concurrent requests carrying the same
> > identifiying token if that should happen.
> > 
> > Moreover the platform is not required to produce in-order responses to
> > agent requests, the only constraint in these regards is that in case of
> > an asynchronous message the delayed response must be sent after the
> > immediate response for the synchronous part of the command transaction.
> > 
> > Currenly the SCMI stack of the OSPM agent selects a token for the egressing
> > commands picking the lowest possible number which is not already in use by
> > an existing in-flight transaction, which means, in other words, that we
> > immediately reuse any token after its transaction has completed or it has
> > timed out: this policy indeed does simplify management and lookup of tokens
> > and associated xfers.
> > 
> > Under the above assumptions and constraints, since there is really no state
> > shared between the agent and the platform to let the platform know when a
> > token and its associated message has timed out, the current policy of early
> > reuse of tokens can easily lead to the situation in which a spurious or
> > late received response (or delayed_response), related to an old stale and
> > timed out transaction, can be wrongly associated to a newer valid in-flight
> > xfer that just happens to have reused the same token.
> > 
> > This misbehaviour on such ghost responses is more easily exposed on those
> > transports that naturally have an higher level of parallelism in processing
> > multiple concurrent in-flight messages.
> > 
> > This commit introduces a new policy of selection of tokens for the OSPM
> > agent: each new transfer now gets the next available and monotonically
> > increasing token, until tokens are exhausted and the counter rolls over.
> > 
> > Such new policy mitigates the above issues with ghost responses since the
> > tokens are now reused as late as possible (when they roll back ideally)
> > and so it is much easier to identify such ghost responses to stale timed
> > out transactions: this also helps in simplifying the specific transports
> > implementation since stale transport messages can be easily identified
> > and discarded early on in the rx path without the need to cross check
> > their actual state with the core transport layer.
> > This mitigation is even more effective when, as is usually the case, the
> > maximum number of pending messages is capped by the platform to a much
> > lower number than the whole possible range of tokens values (2^10).
> > 
> > This internal policy change in the core SCMI transport layer is fully
> > transparent to the specific transports so it has not and should not have
> > any impact on the transports implementation.
> > 
> > The empirically observed cost of such new procedure of token selection
> > amounts in the best case to ~10us out of an observed full transaction cost
> > of 3ms for the completion of a synchronous sensor reading command on a
> > platform supporting commands completion interrupts.
> 
> Hi Cristian,
> 
> Just curious... How badly did a cyclic IDR perform for this usecase?
> Feature wise it seems suitable, but perhaps to heavy weight for this
> rather constrained case where you can assume the number of IDs in
> use at a time is rather small.
> 
> Also, I've not looked closely at the code so there may be other relevant
> constraint or subtlety I'm missing.
> 

I'll give it a go to implement it with a cyclic IDR and see how much it
improves the performance; I'm not really strong towards an
implementation than another, the only real requirement here was to use
the full set of 1024 possible tokens before reusing them, even if in
reality the set of possibly pending transactions is far less than
1024 (max_msg <<< 1024) and the set of probably pending ones is even
smaller.

Thanks for the hint about the IDRs I'll see how it pans out in term of
complexity and perf before re-posting.

Cristian

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
@ 2021-06-16  9:11       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16  9:11 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Mon, Jun 14, 2021 at 02:53:01PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:25 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > Tokens are sequence numbers embedded in the each SCMI message header: they
> > are used to correlate commands with responses (and delayed responses), but
> > their usage and policy of selection is entirely up to the caller (usually
> > the OSPM agent), while they are completely opaque to the callee (SCMI
> > server platform) which merely copies them back from the command into the
> > response message header.
> > This also means that the platform does not, can not and should not enforce
> > any kind of policy on received messages depending on the contained sequence
> > number: platform can perfectly handle concurrent requests carrying the same
> > identifiying token if that should happen.
> > 
> > Moreover the platform is not required to produce in-order responses to
> > agent requests, the only constraint in these regards is that in case of
> > an asynchronous message the delayed response must be sent after the
> > immediate response for the synchronous part of the command transaction.
> > 
> > Currenly the SCMI stack of the OSPM agent selects a token for the egressing
> > commands picking the lowest possible number which is not already in use by
> > an existing in-flight transaction, which means, in other words, that we
> > immediately reuse any token after its transaction has completed or it has
> > timed out: this policy indeed does simplify management and lookup of tokens
> > and associated xfers.
> > 
> > Under the above assumptions and constraints, since there is really no state
> > shared between the agent and the platform to let the platform know when a
> > token and its associated message has timed out, the current policy of early
> > reuse of tokens can easily lead to the situation in which a spurious or
> > late received response (or delayed_response), related to an old stale and
> > timed out transaction, can be wrongly associated to a newer valid in-flight
> > xfer that just happens to have reused the same token.
> > 
> > This misbehaviour on such ghost responses is more easily exposed on those
> > transports that naturally have an higher level of parallelism in processing
> > multiple concurrent in-flight messages.
> > 
> > This commit introduces a new policy of selection of tokens for the OSPM
> > agent: each new transfer now gets the next available and monotonically
> > increasing token, until tokens are exhausted and the counter rolls over.
> > 
> > Such new policy mitigates the above issues with ghost responses since the
> > tokens are now reused as late as possible (when they roll back ideally)
> > and so it is much easier to identify such ghost responses to stale timed
> > out transactions: this also helps in simplifying the specific transports
> > implementation since stale transport messages can be easily identified
> > and discarded early on in the rx path without the need to cross check
> > their actual state with the core transport layer.
> > This mitigation is even more effective when, as is usually the case, the
> > maximum number of pending messages is capped by the platform to a much
> > lower number than the whole possible range of tokens values (2^10).
> > 
> > This internal policy change in the core SCMI transport layer is fully
> > transparent to the specific transports so it has not and should not have
> > any impact on the transports implementation.
> > 
> > The empirically observed cost of such new procedure of token selection
> > amounts in the best case to ~10us out of an observed full transaction cost
> > of 3ms for the completion of a synchronous sensor reading command on a
> > platform supporting commands completion interrupts.
> 
> Hi Cristian,
> 
> Just curious... How badly did a cyclic IDR perform for this usecase?
> Feature wise it seems suitable, but perhaps to heavy weight for this
> rather constrained case where you can assume the number of IDs in
> use at a time is rather small.
> 
> Also, I've not looked closely at the code so there may be other relevant
> constraint or subtlety I'm missing.
> 

I'll give it a go to implement it with a cyclic IDR and see how much it
improves the performance; I'm not really strong towards an
implementation than another, the only real requirement here was to use
the full set of 1024 possible tokens before reusing them, even if in
reality the set of possibly pending transactions is far less than
1024 (max_msg <<< 1024) and the set of probably pending ones is even
smaller.

Thanks for the hint about the IDRs I'll see how it pans out in term of
complexity and perf before re-posting.

Cristian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 07/16] firmware: arm_scmi: Add op to override max message #
  2021-06-14 14:04     ` Jonathan Cameron
@ 2021-06-16  9:13       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16  9:13 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Mon, Jun 14, 2021 at 03:04:15PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:28 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > From: Igor Skalkin <igor.skalkin@opensynergy.com>
> > 
> > The number of simultaneously pending messages that the upcoming
> > scmi-virtio transport can support depends on the virtio device (SCMI
> > platform) and can differ for each channel. (The scmi-virtio transport
> > does only have one tx and at most 1 rx channel.)
> > 
> > Add an optional transport op so that scmi-virtio can report the actual
> > max message # for each channel type. Respect these new limits.
> > 
> > Reflect that the limit in struct scmi_desc is now only a default any
> > more.
> > 
> > [ Peter: Adapted patch for submission to upstream. ]
> > 
> > Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> > Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> Cristian,
> 
> Give you are sending this on to the list, should have your
> Signed-off-by as well.
> 

Right, sorry for this, the series is still sort of WIP so I have not
properly cleanup and signed these if I have not touched it.
My bad, I'll fix.

Thanks,
Cristian

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 07/16] firmware: arm_scmi: Add op to override max message #
@ 2021-06-16  9:13       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16  9:13 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Mon, Jun 14, 2021 at 03:04:15PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:28 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > From: Igor Skalkin <igor.skalkin@opensynergy.com>
> > 
> > The number of simultaneously pending messages that the upcoming
> > scmi-virtio transport can support depends on the virtio device (SCMI
> > platform) and can differ for each channel. (The scmi-virtio transport
> > does only have one tx and at most 1 rx channel.)
> > 
> > Add an optional transport op so that scmi-virtio can report the actual
> > max message # for each channel type. Respect these new limits.
> > 
> > Reflect that the limit in struct scmi_desc is now only a default any
> > more.
> > 
> > [ Peter: Adapted patch for submission to upstream. ]
> > 
> > Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> > Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> Cristian,
> 
> Give you are sending this on to the list, should have your
> Signed-off-by as well.
> 

Right, sorry for this, the series is still sort of WIP so I have not
properly cleanup and signed these if I have not touched it.
My bad, I'll fix.

Thanks,
Cristian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 12/16] firmware: arm_scmi: Add message passing abstractions for transports
  2021-06-14 14:10     ` Jonathan Cameron
@ 2021-06-16  9:14       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16  9:14 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Mon, Jun 14, 2021 at 03:10:36PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:33 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > From: Peter Hilber <peter.hilber@opensynergy.com>
> > 
> > Add abstractions for future transports using message passing, such as
> > virtio. Derive the abstractions from the shared memory abstractions.
> > 
> > Abstract the transport SDU through the opaque struct scmi_msg_payld.
> > Also enable the transport to determine all other required information
> > about the transport SDU.
> > 
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> > ---
> ...
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index b783ae058c8a..fa4075336580 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -410,6 +410,21 @@ void shmem_clear_channel(struct scmi_shared_mem __iomem *shmem);
> >  bool shmem_poll_done(struct scmi_shared_mem __iomem *shmem,
> >  		     struct scmi_xfer *xfer);
> >  
> > +/* declarations for message passing transports */
> > +struct scmi_msg_payld;
> > +
> > +/** Maximum overhead of message w.r.t. struct scmi_desc.max_msg_size */
> 
> Doesn't look to be kernel-doc..
> 
Right, I'll fix.

Cristian

> > +#define SCMI_MSG_MAX_PROT_OVERHEAD (2 * sizeof(__le32))

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 12/16] firmware: arm_scmi: Add message passing abstractions for transports
@ 2021-06-16  9:14       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16  9:14 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On Mon, Jun 14, 2021 at 03:10:36PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:33 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > From: Peter Hilber <peter.hilber@opensynergy.com>
> > 
> > Add abstractions for future transports using message passing, such as
> > virtio. Derive the abstractions from the shared memory abstractions.
> > 
> > Abstract the transport SDU through the opaque struct scmi_msg_payld.
> > Also enable the transport to determine all other required information
> > about the transport SDU.
> > 
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> > ---
> ...
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index b783ae058c8a..fa4075336580 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -410,6 +410,21 @@ void shmem_clear_channel(struct scmi_shared_mem __iomem *shmem);
> >  bool shmem_poll_done(struct scmi_shared_mem __iomem *shmem,
> >  		     struct scmi_xfer *xfer);
> >  
> > +/* declarations for message passing transports */
> > +struct scmi_msg_payld;
> > +
> > +/** Maximum overhead of message w.r.t. struct scmi_desc.max_msg_size */
> 
> Doesn't look to be kernel-doc..
> 
Right, I'll fix.

Cristian

> > +#define SCMI_MSG_MAX_PROT_OVERHEAD (2 * sizeof(__le32))

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
  2021-06-14 14:23     ` Jonathan Cameron
@ 2021-06-16 10:18       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16 10:18 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Mon, Jun 14, 2021 at 03:23:15PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:35 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > From: Igor Skalkin <igor.skalkin@opensynergy.com>
> > 
> > This transport enables accessing an SCMI platform as a virtio device.
> > 
> > Implement an SCMI virtio driver according to the virtio SCMI device spec
> > [1]. Virtio device id 32 has been reserved for the SCMI device [2].
> > 
> > The virtio transport has one Tx channel (virtio cmdq, A2P channel) and
> > at most one Rx channel (virtio eventq, P2A channel).
> > 
> > The following feature bit defined in [1] is not implemented:
> > VIRTIO_SCMI_F_SHARED_MEMORY.
> > 
> > After the preparatory patches, this implements the virtio transport as
> > paraphrased:
> > 
> > Only support a single arm-scmi device (which is consistent with the SCMI
> > spec). scmi-virtio init is called from arm-scmi module init. During the
> > arm-scmi probing, link to the first probed scmi-virtio device. Defer
> > arm-scmi probing if no scmi-virtio device is bound yet.
> > 
> > For simplicity, restrict the number of messages which can be pending
> > simultaneously according to the virtqueue capacity. (The virtqueue sizes
> > are negotiated with the virtio device.)
> > 
> > As soon as Rx channel message buffers are allocated or have been read
> > out by the arm-scmi driver, feed them to the virtio device.
> > 
> > Since some virtio devices may not have the short response time exhibited
> > by SCMI platforms using other transports, set a generous response
> > timeout.
> > 
> > Limitations:
> > 
> > - Polling is not supported.
> > 
> > - The timeout for delayed responses has not been adjusted.
> > 
> > [1] https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-scmi.tex
> > [2] https://www.oasis-open.org/committees/ballot.php?id=3496
> > 
> > [ Peter: Adapted patch for submission to upstream. ]
> > 
> > Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> > Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> 
> A few drive by comments inline.
> 
> J
> > ---
> >  MAINTAINERS                        |   1 +
> >  drivers/firmware/Kconfig           |  12 +
> >  drivers/firmware/arm_scmi/Makefile |   1 +
> >  drivers/firmware/arm_scmi/common.h |   3 +
> >  drivers/firmware/arm_scmi/driver.c |   3 +
> >  drivers/firmware/arm_scmi/virtio.c | 523 +++++++++++++++++++++++++++++
> >  include/uapi/linux/virtio_ids.h    |   1 +
> >  include/uapi/linux/virtio_scmi.h   |  25 ++
> >  8 files changed, 569 insertions(+)
> >  create mode 100644 drivers/firmware/arm_scmi/virtio.c
> >  create mode 100644 include/uapi/linux/virtio_scmi.h
> > 
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index f408cf2d2781..e1b27ed11060 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -17878,6 +17878,7 @@ F:	drivers/regulator/scmi-regulator.c
> >  F:	drivers/reset/reset-scmi.c
> >  F:	include/linux/sc[mp]i_protocol.h
> >  F:	include/trace/events/scmi.h
> > +F:	include/uapi/linux/virtio_scmi.h
> >  
> >  SYSTEM RESET/SHUTDOWN DRIVERS
> >  M:	Sebastian Reichel <sre@kernel.org>
> > diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
> > index e74d25065d07..2c37d5af66ad 100644
> > --- a/drivers/firmware/Kconfig
> > +++ b/drivers/firmware/Kconfig
> > @@ -39,6 +39,18 @@ config ARM_SCMI_HAVE_MSG
> >  	  This declares whether a message passing based transport for SCMI is
> >  	  available.
> >  
> > +# This config option includes the dependencies of ARM_SCMI_PROTOCOL so that
> > +# this config doesn't show up when SCMI wouldn't be available.
> > +config VIRTIO_SCMI
> > +	bool "Virtio transport for SCMI"
> > +	select ARM_SCMI_HAVE_MSG
> > +	depends on VIRTIO && (ARM || ARM64 || COMPILE_TEST)
> > +	help
> > +	  This enables the virtio based transport for SCMI.
> > +
> > +	  If you want to use the ARM SCMI protocol between the virtio guest and
> > +	  a host providing a virtio SCMI device, answer Y.
> > +
> >  config ARM_SCMI_POWER_DOMAIN
> >  	tristate "SCMI power domain driver"
> >  	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
> > diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
> > index f6b4acb8abdb..db1787606fb2 100644
> > --- a/drivers/firmware/arm_scmi/Makefile
> > +++ b/drivers/firmware/arm_scmi/Makefile
> > @@ -5,6 +5,7 @@ scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
> >  scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
> >  scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
> >  scmi-transport-$(CONFIG_ARM_SCMI_HAVE_MSG) += msg.o
> > +scmi-transport-$(CONFIG_VIRTIO_SCMI) += virtio.o
> >  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
> >  scmi-module-objs := $(scmi-bus-y) $(scmi-driver-y) $(scmi-protocols-y) \
> >  		    $(scmi-transport-y)
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index fa4075336580..c143a449d278 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -390,6 +390,9 @@ extern const struct scmi_desc scmi_mailbox_desc;
> >  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
> >  extern const struct scmi_desc scmi_smc_desc;
> >  #endif
> > +#ifdef CONFIG_VIRTIO_SCMI
> > +extern const struct scmi_desc scmi_virtio_desc;
> > +#endif
> >  
> >  int scmi_set_transport_info(struct device *dev, void *transport_info);
> >  void *scmi_get_transport_info(struct device *dev);
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index bb2de15554a9..ccc7dd49261e 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -1954,6 +1954,9 @@ static const struct of_device_id scmi_of_match[] = {
> >  #endif
> >  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
> >  	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
> > +#endif
> > +#ifdef CONFIG_VIRTIO_SCMI
> > +	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
> >  #endif
> >  	{ /* Sentinel */ },
> >  };
> > diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
> > new file mode 100644
> > index 000000000000..20972adf6dc7
> > --- /dev/null
> > +++ b/drivers/firmware/arm_scmi/virtio.c
> > @@ -0,0 +1,523 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Virtio Transport driver for Arm System Control and Management Interface
> > + * (SCMI).
> > + *
> > + * Copyright (C) 2020 OpenSynergy.
> > + */
> > +
> > +/**
> > + * DOC: Theory of Operation
> > + *
> > + * The scmi-virtio transport implements a driver for the virtio SCMI device.
> > + *
> > + * There is one Tx channel (virtio cmdq, A2P channel) and at most one Rx
> > + * channel (virtio eventq, P2A channel). Each channel is implemented through a
> > + * virtqueue. Access to each virtqueue is protected by spinlocks.
> > + */
> > +
> > +#include <linux/errno.h>
> 
> > +#include <linux/of.h>
> > +#include <linux/of_platform.h>
> Why either of these?
> 
> perhaps
> mod_devicetable.h is appropriate.
> 

These are clearly leftovers from a previous version where there were
some DT configs, now dropped since not acceptable nor needed.

> > +#include <linux/platform_device.h>
> > +#include <linux/module.h>
> > +#include <linux/slab.h>
> > +#include <linux/virtio.h>
> > +#include <linux/virtio_config.h>
> > +#include <uapi/linux/virtio_ids.h>
> > +#include <uapi/linux/virtio_scmi.h>
> > +
> > +#include "common.h"
> > +
> > +#define VIRTIO_SCMI_MAX_MSG_SIZE 128 /* Value may be increased. */
> > +#define VIRTIO_SCMI_MAX_PDU_SIZE \
> > +	(VIRTIO_SCMI_MAX_MSG_SIZE + SCMI_MSG_MAX_PROT_OVERHEAD)
> > +#define DESCRIPTORS_PER_TX_MSG 2
> > +
> > +/**
> > + * struct scmi_vio_channel - Transport channel information
> > + *
> > + * @lock: Protects access to all members except ready.
> > + * @ready_lock: Protects access to ready. If required, it must be taken before
> > + *              lock.
> > + * @vqueue: Associated virtqueue
> > + * @cinfo: SCMI Tx or Rx channel
> > + * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
> > + * @is_rx: Whether channel is an Rx channel
> > + * @ready: Whether transport user is ready to hear about channel
> > + */
> > +struct scmi_vio_channel {
> > +	spinlock_t lock;
> > +	spinlock_t ready_lock;
> > +	struct virtqueue *vqueue;
> > +	struct scmi_chan_info *cinfo;
> > +	struct list_head free_list;
> > +	u8 is_rx;
> > +	u8 ready;
> > +};
> > +
> > +/**
> > + * struct scmi_vio_msg - Transport PDU information
> > + *
> > + * @request: SDU used for commands
> > + * @input: SDU used for (delayed) responses and notifications
> > + * @list: List which scmi_vio_msg may be part of
> > + * @rx_len: Input SDU size in bytes, once input has been received
> > + */
> > +struct scmi_vio_msg {
> > +	struct scmi_msg_payld *request;
> > +	struct scmi_msg_payld *input;
> > +	struct list_head list;
> > +	unsigned int rx_len;
> > +};
> > +
> > +static bool scmi_vio_have_vq_rx(struct virtio_device *vdev)
> > +{
> > +	return virtio_has_feature(vdev, VIRTIO_SCMI_F_P2A_CHANNELS);
> > +}
> > +
> > +static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
> > +			       struct scmi_vio_msg *msg)
> > +{
> > +	struct scatterlist sg_in;
> > +	int rc;
> > +	unsigned long flags;
> > +
> > +	sg_init_one(&sg_in, msg->input, VIRTIO_SCMI_MAX_PDU_SIZE);
> > +
> > +	spin_lock_irqsave(&vioch->lock, flags);
> > +
> > +	rc = virtqueue_add_inbuf(vioch->vqueue, &sg_in, 1, msg, GFP_ATOMIC);
> > +	if (rc)
> > +		dev_err_once(vioch->cinfo->dev,
> > +			     "%s() failed to add to virtqueue (%d)\n", __func__,
> > +			     rc);
> > +	else
> > +		virtqueue_kick(vioch->vqueue);
> > +
> > +	spin_unlock_irqrestore(&vioch->lock, flags);
> > +
> > +	return rc;
> > +}
> > +
> > +static void scmi_vio_complete_cb(struct virtqueue *vqueue)
> > +{
> > +	unsigned long ready_flags;
> > +	unsigned long flags;
> > +	unsigned int length;
> > +	struct scmi_vio_channel *vioch;
> > +	struct scmi_vio_msg *msg;
> > +	bool cb_enabled = true;
> > +
> > +	if (WARN_ON_ONCE(!vqueue->vdev->priv))
> > +		return;
> > +	vioch = &((struct scmi_vio_channel *)vqueue->vdev->priv)[vqueue->index];
> > +
> > +	for (;;) {
> > +		spin_lock_irqsave(&vioch->ready_lock, ready_flags);
> > +
> > +		if (!vioch->ready) {
> > +			if (!cb_enabled)
> > +				(void)virtqueue_enable_cb(vqueue);
> > +			goto unlock_ready_out;
> > +		}
> > +
> > +		spin_lock_irqsave(&vioch->lock, flags);
> > +		if (cb_enabled) {
> > +			virtqueue_disable_cb(vqueue);
> > +			cb_enabled = false;
> > +		}
> > +		msg = virtqueue_get_buf(vqueue, &length);
> > +		if (!msg) {
> > +			if (virtqueue_enable_cb(vqueue))
> > +				goto unlock_out;
> > +			else
> 
> Drop the else as it doesn't add readability as far as I can see.
> 

Sure.

> > +				cb_enabled = true;
> > +		}
> > +		spin_unlock_irqrestore(&vioch->lock, flags);
> > +
> > +		if (msg) {
> > +			msg->rx_len = length;
> > +
> > +			/*
> > +			 * Hold the ready_lock during the callback to avoid
> > +			 * races when the arm-scmi driver is unbinding while
> > +			 * the virtio device is not quiesced yet.
> > +			 */
> > +			scmi_rx_callback(vioch->cinfo,
> > +					 msg_read_header(msg->input), msg);
> > +		}
> > +		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> > +	}
> > +
> > +unlock_out:
> > +	spin_unlock_irqrestore(&vioch->lock, flags);
> > +unlock_ready_out:
> > +	spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> > +}
> > +
> > +static const char *const scmi_vio_vqueue_names[] = { "tx", "rx" };
> > +
> > +static vq_callback_t *scmi_vio_complete_callbacks[] = {
> > +	scmi_vio_complete_cb,
> > +	scmi_vio_complete_cb
> > +};
> > +
> > +static unsigned int virtio_get_max_msg(bool tx,
> > +				       struct scmi_chan_info *base_cinfo)
> > +{
> > +	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
> > +	unsigned int ret;
> > +
> > +	ret = virtqueue_get_vring_size(vioch->vqueue);
> > +
> > +	/* Tx messages need multiple descriptors. */
> > +	if (tx)
> > +		ret /= DESCRIPTORS_PER_TX_MSG;
> > +
> > +	if (ret > MSG_TOKEN_MAX) {
> > +		dev_info_once(
> > +			base_cinfo->dev,
> > +			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
> > +			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
> > +		ret = MSG_TOKEN_MAX;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int scmi_vio_match_any_dev(struct device *dev, const void *data)
> > +{
> > +	return 1;
> > +}
> > +
> > +static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
> > +
> > +static int virtio_link_supplier(struct device *dev)
> > +{
> > +	struct device *vdev = driver_find_device(
> > +		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
> > +
> > +	if (!vdev) {
> > +		dev_notice_once(
> > +			dev,
> > +			"Deferring probe after not finding a bound scmi-virtio device\n");
> > +		return -EPROBE_DEFER;
> > +	}
> > +
> > +	/* Add device link for remove order and sysfs link. */
> > +	if (!device_link_add(dev, vdev, DL_FLAG_AUTOREMOVE_CONSUMER)) {
> > +		put_device(vdev);
> > +		dev_err(dev, "Adding link to supplier virtio device failed\n");
> > +		return -ECANCELED;
> > +	}
> > +
> > +	put_device(vdev);
> > +	return scmi_set_transport_info(dev, dev_to_virtio(vdev));
> > +}
> > +
> > +static bool virtio_chan_available(struct device *dev, int idx)
> > +{
> > +	struct virtio_device *vdev;
> > +
> > +	/* scmi-virtio doesn't support per-protocol channels */
> > +	if (is_scmi_protocol_device(dev))
> > +		return false;
> > +
> > +	vdev = scmi_get_transport_info(dev);
> > +	if (!vdev)
> > +		return false;
> > +
> > +	switch (idx) {
> > +	case VIRTIO_SCMI_VQ_TX:
> > +		return true;
> > +	case VIRTIO_SCMI_VQ_RX:
> > +		return scmi_vio_have_vq_rx(vdev);
> > +	default:
> > +		return false;
> > +	}
> > +}
> > +
> > +static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> > +			     bool tx)
> > +{
> > +	unsigned long flags;
> > +	struct virtio_device *vdev;
> > +	struct scmi_vio_channel *vioch;
> > +	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
> > +	int max_msg;
> > +	int i;
> > +
> > +	if (!virtio_chan_available(dev, index))
> > +		return -ENODEV;
> > +
> > +	vdev = scmi_get_transport_info(dev);
> > +	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
> > +
> > +	spin_lock_irqsave(&vioch->lock, flags);
> > +	cinfo->transport_info = vioch;
> > +	vioch->cinfo = cinfo;
> > +	spin_unlock_irqrestore(&vioch->lock, flags);
> > +
> > +	max_msg = virtio_get_max_msg(tx, cinfo);
> > +
> > +	for (i = 0; i < max_msg; i++) {
> > +		struct scmi_vio_msg *msg;
> > +
> > +		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
> > +		if (!msg)
> > +			return -ENOMEM;
> > +
> > +		if (tx) {
> > +			msg->request = devm_kzalloc(cinfo->dev,
> > +						    VIRTIO_SCMI_MAX_PDU_SIZE,
> > +						    GFP_KERNEL);
> > +			if (!msg->request)
> > +				return -ENOMEM;
> > +		}
> > +
> > +		msg->input = devm_kzalloc(cinfo->dev, VIRTIO_SCMI_MAX_PDU_SIZE,
> > +					  GFP_KERNEL);
> > +		if (!msg->input)
> > +			return -ENOMEM;
> > +
> > +		if (tx) {
> > +			spin_lock_irqsave(&vioch->lock, flags);
> > +			list_add_tail(&msg->list, &vioch->free_list);
> > +			spin_unlock_irqrestore(&vioch->lock, flags);
> > +		} else {
> > +			scmi_vio_feed_vq_rx(vioch, msg);
> > +		}
> > +	}
> > +
> > +	spin_lock_irqsave(&vioch->ready_lock, flags);
> > +	vioch->ready = true;
> > +	spin_unlock_irqrestore(&vioch->ready_lock, flags);
> > +
> > +	return 0;
> > +}
> > +
> > +static int virtio_chan_free(int id, void *p, void *data)
> > +{
> > +	unsigned long flags;
> > +	struct scmi_chan_info *cinfo = p;
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +
> > +	spin_lock_irqsave(&vioch->ready_lock, flags);
> > +	vioch->ready = false;
> > +	spin_unlock_irqrestore(&vioch->ready_lock, flags);
> > +
> > +	scmi_free_channel(cinfo, data, id);
> 
> Blank line here would be nice + consistent.
> 

Right, I'll do.

> > +	return 0;
> > +}
> > +
> > +static int virtio_send_message(struct scmi_chan_info *cinfo,
> > +			       struct scmi_xfer *xfer)
> > +{
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +	struct scatterlist sg_out;
> > +	struct scatterlist sg_in;
> > +	struct scatterlist *sgs[DESCRIPTORS_PER_TX_MSG] = { &sg_out, &sg_in };
> > +	unsigned long flags;
> > +	int rc;
> > +	struct scmi_vio_msg *msg;
> > +
> > +	/*
> > +	 * TODO: For now, we don't support polling. But it should not be
> > +	 * difficult to add support.
> > +	 */
> > +	if (xfer->hdr.poll_completion)
> > +		return -EINVAL;
> > +
> > +	spin_lock_irqsave(&vioch->lock, flags);
> > +
> > +	if (list_empty(&vioch->free_list)) {
> > +		spin_unlock_irqrestore(&vioch->lock, flags);
> > +		return -EBUSY;
> > +	}
> > +
> > +	msg = list_first_entry(&vioch->free_list, typeof(*msg), list);
> > +	list_del(&msg->list);
> > +
> > +	msg_tx_prepare(msg->request, xfer);
> > +
> > +	sg_init_one(&sg_out, msg->request, msg_command_size(xfer));
> > +	sg_init_one(&sg_in, msg->input, msg_response_size(xfer));
> > +
> > +	rc = virtqueue_add_sgs(vioch->vqueue, sgs, 1, 1, msg, GFP_ATOMIC);
> > +	if (rc) {
> > +		list_add(&msg->list, &vioch->free_list);
> > +		dev_err_once(vioch->cinfo->dev,
> > +			     "%s() failed to add to virtqueue (%d)\n", __func__,
> > +			     rc);
> > +	} else {
> > +		virtqueue_kick(vioch->vqueue);
> > +	}
> > +
> > +	spin_unlock_irqrestore(&vioch->lock, flags);
> > +
> > +	return rc;
> > +}
> > +
> > +static void virtio_fetch_response(struct scmi_chan_info *cinfo,
> > +				  struct scmi_xfer *xfer, void *msg_handle)
> > +{
> > +	struct scmi_vio_msg *msg = msg_handle;
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +
> > +	if (!msg) {
> > +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> > +			     "Ignoring %s() call with NULL msg_handle\n",
> > +			     __func__);
> > +		return;
> > +	}
> > +
> > +	msg_fetch_response(msg->input, msg->rx_len, xfer);
> > +}
> > +
> > +static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
> > +				      size_t max_len, struct scmi_xfer *xfer,
> > +				      void *msg_handle)
> > +{
> > +	struct scmi_vio_msg *msg = msg_handle;
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +
> > +	if (!msg) {
> > +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> > +			     "Ignoring %s() call with NULL msg_handle\n",
> > +			     __func__);
> > +		return;
> > +	}
> > +
> > +	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
> > +}
> > +
> > +static void dummy_clear_channel(struct scmi_chan_info *cinfo)
> > +{
> > +}
> > +
> > +static bool dummy_poll_done(struct scmi_chan_info *cinfo,
> > +			    struct scmi_xfer *xfer)
> > +{
> > +	return false;
> > +}
> > +
> > +static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
> > +{
> > +	unsigned long flags;
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +	struct scmi_vio_msg *msg = msg_handle;
> > +
> > +	if (!msg) {
> > +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> > +			     "Ignoring %s() call with NULL msg_handle\n",
> > +			     __func__);
> > +		return;
> > +	}
> > +
> > +	if (vioch->is_rx) {
> > +		scmi_vio_feed_vq_rx(vioch, msg);
> > +	} else {
> > +		spin_lock_irqsave(&vioch->lock, flags);
> > +		list_add(&msg->list, &vioch->free_list);
> > +		spin_unlock_irqrestore(&vioch->lock, flags);
> > +	}
> > +}
> > +
> > +static const struct scmi_transport_ops scmi_virtio_ops = {
> > +	.link_supplier = virtio_link_supplier,
> > +	.chan_available = virtio_chan_available,
> > +	.chan_setup = virtio_chan_setup,
> > +	.chan_free = virtio_chan_free,
> > +	.get_max_msg = virtio_get_max_msg,
> > +	.send_message = virtio_send_message,
> > +	.fetch_response = virtio_fetch_response,
> > +	.fetch_notification = virtio_fetch_notification,
> > +	.clear_channel = dummy_clear_channel,
> > +	.poll_done = dummy_poll_done,
> > +	.drop_message = virtio_drop_message,
> > +};
> > +
> > +static int scmi_vio_probe(struct virtio_device *vdev)
> > +{
> > +	struct device *dev = &vdev->dev;
> > +	struct scmi_vio_channel *channels;
> > +	bool have_vq_rx;
> > +	int vq_cnt;
> > +	int i;
> > +	int ret;
> > +	struct virtqueue *vqs[VIRTIO_SCMI_VQ_MAX_CNT];
> > +
> > +	have_vq_rx = scmi_vio_have_vq_rx(vdev);
> > +	vq_cnt = have_vq_rx ? VIRTIO_SCMI_VQ_MAX_CNT : 1;
> > +
> > +	channels = devm_kcalloc(dev, vq_cnt, sizeof(*channels), GFP_KERNEL);
> > +	if (!channels)
> > +		return -ENOMEM;
> > +
> > +	if (have_vq_rx)
> > +		channels[VIRTIO_SCMI_VQ_RX].is_rx = true;
> > +
> > +	ret = virtio_find_vqs(vdev, vq_cnt, vqs, scmi_vio_complete_callbacks,
> > +			      scmi_vio_vqueue_names, NULL);
> > +	if (ret) {
> > +		dev_err(dev, "Failed to get %d virtqueue(s)\n", vq_cnt);
> > +		return ret;
> > +	}
> > +	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);
> 
> Rather noisy given I assume there are plenty of other ways to se this
> succeeded (i.e. probe succeeded)
> 

Yes, indeed, the only additional info that was meant to be carried here
I suppose is that when you find 2 virtqueues (max possible) you know
(deduct really) that you have the P2A channel, so also notifications and
delayed responses are supported.

I'll review in general the info msgs and drop to debug when not neeeded.
(or explicit the above I-have-notif_dresp information.

> > +
> > +	for (i = 0; i < vq_cnt; i++) {
> > +		spin_lock_init(&channels[i].lock);
> > +		spin_lock_init(&channels[i].ready_lock);
> > +		INIT_LIST_HEAD(&channels[i].free_list);
> > +		channels[i].vqueue = vqs[i];
> > +	}
> > +
> > +	vdev->priv = channels;
> > +
> > +	return 0;
> > +}
> > +
> > +static void scmi_vio_remove(struct virtio_device *vdev)
> > +{
> > +	vdev->config->reset(vdev);
> > +	vdev->config->del_vqs(vdev);
> > +}
> > +
> > +static unsigned int features[] = {
> > +	VIRTIO_SCMI_F_P2A_CHANNELS,
> > +};
> > +
> > +static const struct virtio_device_id id_table[] = {
> > +	{ VIRTIO_ID_SCMI, VIRTIO_DEV_ANY_ID },
> > +	{ 0 }
> > +};
> > +
> > +static struct virtio_driver virtio_scmi_driver = {
> > +	.driver.name = "scmi-virtio",
> > +	.driver.owner = THIS_MODULE,
> > +	.feature_table = features,
> > +	.feature_table_size = ARRAY_SIZE(features),
> > +	.id_table = id_table,
> > +	.probe = scmi_vio_probe,
> > +	.remove = scmi_vio_remove,
> > +};
> > +
> > +static int __init virtio_scmi_init(void)
> > +{
> > +	return register_virtio_driver(&virtio_scmi_driver);
> > +}
> > +
> > +static void __exit virtio_scmi_exit(void)
> > +{
> > +	unregister_virtio_driver(&virtio_scmi_driver);
> > +}
> 
> module_virtio_driver() ?
> 

No that's not possible at the moment, and indeed it is the whole
reason I introduced early in the series the transport specific
.init()/.exit() optional helpers.

The issue is that currently SCMI transports are in fact part of the SCMI
stack compilation unit; if you enable some transports those get compiled
in the main SCMI module, and if you then (hopefully) define a DT entry
describing an SCMI instance using a specific transport (setting compatible
"arm,scmi-virtio" in this case) such SCMI stack will use that transport and
its ops: problem is the SCMI main module can be compiled as loadable
module on its own, including all transports configured and all base SCMI
protos, so if you use here the standard module_virtio_driver() it won't
link in case you built the SCMI core as module and complains about

> multiple definition of `init_module'; 
> multiple definition of `cleanup_module';

The above .init()/.exit() instead are registered with the core as
optional transport helpers and invoked when the SCMI core is
initialized, so also well before the SCMI core is probed.

Making it a full fledged module instead (and so then using normal
module_virtio_driver() macros) that registers with the SCMI core somehow
after its probed by its own susbsystem (like virtIO), seemed to
complicate a lot the interactions between the core and the transports
(even though I'd like it much more as an approach) so I have not gone
through this road still....but I cannot exclude I'll come back to this
kind of approach when I'll try to simplify a bit the probing sequence of
this SCMI VirtIO driver...it's just that for now seemed easier to add
these optional transport helpers to let transport provide their early setup
stuff (if needed), like registering with another subsystem.
(upcoming SCMI OPTEE transport posted in a different series by Etienne
has the same needs)

Thanks,
Cristian


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
@ 2021-06-16 10:18       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-06-16 10:18 UTC (permalink / raw)
  To: Jonathan Cameron
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, f.fainelli, etienne.carriere,
	vincent.guittot, souvik.chakravarty, igor.skalkin, peter.hilber,
	alex.bennee, jean-philippe, mikhail.golubev, anton.yakovlev,
	Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Mon, Jun 14, 2021 at 03:23:15PM +0100, Jonathan Cameron wrote:
> On Fri, 11 Jun 2021 17:59:35 +0100
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
> > From: Igor Skalkin <igor.skalkin@opensynergy.com>
> > 
> > This transport enables accessing an SCMI platform as a virtio device.
> > 
> > Implement an SCMI virtio driver according to the virtio SCMI device spec
> > [1]. Virtio device id 32 has been reserved for the SCMI device [2].
> > 
> > The virtio transport has one Tx channel (virtio cmdq, A2P channel) and
> > at most one Rx channel (virtio eventq, P2A channel).
> > 
> > The following feature bit defined in [1] is not implemented:
> > VIRTIO_SCMI_F_SHARED_MEMORY.
> > 
> > After the preparatory patches, this implements the virtio transport as
> > paraphrased:
> > 
> > Only support a single arm-scmi device (which is consistent with the SCMI
> > spec). scmi-virtio init is called from arm-scmi module init. During the
> > arm-scmi probing, link to the first probed scmi-virtio device. Defer
> > arm-scmi probing if no scmi-virtio device is bound yet.
> > 
> > For simplicity, restrict the number of messages which can be pending
> > simultaneously according to the virtqueue capacity. (The virtqueue sizes
> > are negotiated with the virtio device.)
> > 
> > As soon as Rx channel message buffers are allocated or have been read
> > out by the arm-scmi driver, feed them to the virtio device.
> > 
> > Since some virtio devices may not have the short response time exhibited
> > by SCMI platforms using other transports, set a generous response
> > timeout.
> > 
> > Limitations:
> > 
> > - Polling is not supported.
> > 
> > - The timeout for delayed responses has not been adjusted.
> > 
> > [1] https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-scmi.tex
> > [2] https://www.oasis-open.org/committees/ballot.php?id=3496
> > 
> > [ Peter: Adapted patch for submission to upstream. ]
> > 
> > Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> > Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> 
> A few drive by comments inline.
> 
> J
> > ---
> >  MAINTAINERS                        |   1 +
> >  drivers/firmware/Kconfig           |  12 +
> >  drivers/firmware/arm_scmi/Makefile |   1 +
> >  drivers/firmware/arm_scmi/common.h |   3 +
> >  drivers/firmware/arm_scmi/driver.c |   3 +
> >  drivers/firmware/arm_scmi/virtio.c | 523 +++++++++++++++++++++++++++++
> >  include/uapi/linux/virtio_ids.h    |   1 +
> >  include/uapi/linux/virtio_scmi.h   |  25 ++
> >  8 files changed, 569 insertions(+)
> >  create mode 100644 drivers/firmware/arm_scmi/virtio.c
> >  create mode 100644 include/uapi/linux/virtio_scmi.h
> > 
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index f408cf2d2781..e1b27ed11060 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -17878,6 +17878,7 @@ F:	drivers/regulator/scmi-regulator.c
> >  F:	drivers/reset/reset-scmi.c
> >  F:	include/linux/sc[mp]i_protocol.h
> >  F:	include/trace/events/scmi.h
> > +F:	include/uapi/linux/virtio_scmi.h
> >  
> >  SYSTEM RESET/SHUTDOWN DRIVERS
> >  M:	Sebastian Reichel <sre@kernel.org>
> > diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
> > index e74d25065d07..2c37d5af66ad 100644
> > --- a/drivers/firmware/Kconfig
> > +++ b/drivers/firmware/Kconfig
> > @@ -39,6 +39,18 @@ config ARM_SCMI_HAVE_MSG
> >  	  This declares whether a message passing based transport for SCMI is
> >  	  available.
> >  
> > +# This config option includes the dependencies of ARM_SCMI_PROTOCOL so that
> > +# this config doesn't show up when SCMI wouldn't be available.
> > +config VIRTIO_SCMI
> > +	bool "Virtio transport for SCMI"
> > +	select ARM_SCMI_HAVE_MSG
> > +	depends on VIRTIO && (ARM || ARM64 || COMPILE_TEST)
> > +	help
> > +	  This enables the virtio based transport for SCMI.
> > +
> > +	  If you want to use the ARM SCMI protocol between the virtio guest and
> > +	  a host providing a virtio SCMI device, answer Y.
> > +
> >  config ARM_SCMI_POWER_DOMAIN
> >  	tristate "SCMI power domain driver"
> >  	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
> > diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
> > index f6b4acb8abdb..db1787606fb2 100644
> > --- a/drivers/firmware/arm_scmi/Makefile
> > +++ b/drivers/firmware/arm_scmi/Makefile
> > @@ -5,6 +5,7 @@ scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
> >  scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
> >  scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
> >  scmi-transport-$(CONFIG_ARM_SCMI_HAVE_MSG) += msg.o
> > +scmi-transport-$(CONFIG_VIRTIO_SCMI) += virtio.o
> >  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
> >  scmi-module-objs := $(scmi-bus-y) $(scmi-driver-y) $(scmi-protocols-y) \
> >  		    $(scmi-transport-y)
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index fa4075336580..c143a449d278 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -390,6 +390,9 @@ extern const struct scmi_desc scmi_mailbox_desc;
> >  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
> >  extern const struct scmi_desc scmi_smc_desc;
> >  #endif
> > +#ifdef CONFIG_VIRTIO_SCMI
> > +extern const struct scmi_desc scmi_virtio_desc;
> > +#endif
> >  
> >  int scmi_set_transport_info(struct device *dev, void *transport_info);
> >  void *scmi_get_transport_info(struct device *dev);
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index bb2de15554a9..ccc7dd49261e 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -1954,6 +1954,9 @@ static const struct of_device_id scmi_of_match[] = {
> >  #endif
> >  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
> >  	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
> > +#endif
> > +#ifdef CONFIG_VIRTIO_SCMI
> > +	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
> >  #endif
> >  	{ /* Sentinel */ },
> >  };
> > diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
> > new file mode 100644
> > index 000000000000..20972adf6dc7
> > --- /dev/null
> > +++ b/drivers/firmware/arm_scmi/virtio.c
> > @@ -0,0 +1,523 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Virtio Transport driver for Arm System Control and Management Interface
> > + * (SCMI).
> > + *
> > + * Copyright (C) 2020 OpenSynergy.
> > + */
> > +
> > +/**
> > + * DOC: Theory of Operation
> > + *
> > + * The scmi-virtio transport implements a driver for the virtio SCMI device.
> > + *
> > + * There is one Tx channel (virtio cmdq, A2P channel) and at most one Rx
> > + * channel (virtio eventq, P2A channel). Each channel is implemented through a
> > + * virtqueue. Access to each virtqueue is protected by spinlocks.
> > + */
> > +
> > +#include <linux/errno.h>
> 
> > +#include <linux/of.h>
> > +#include <linux/of_platform.h>
> Why either of these?
> 
> perhaps
> mod_devicetable.h is appropriate.
> 

These are clearly leftovers from a previous version where there were
some DT configs, now dropped since not acceptable nor needed.

> > +#include <linux/platform_device.h>
> > +#include <linux/module.h>
> > +#include <linux/slab.h>
> > +#include <linux/virtio.h>
> > +#include <linux/virtio_config.h>
> > +#include <uapi/linux/virtio_ids.h>
> > +#include <uapi/linux/virtio_scmi.h>
> > +
> > +#include "common.h"
> > +
> > +#define VIRTIO_SCMI_MAX_MSG_SIZE 128 /* Value may be increased. */
> > +#define VIRTIO_SCMI_MAX_PDU_SIZE \
> > +	(VIRTIO_SCMI_MAX_MSG_SIZE + SCMI_MSG_MAX_PROT_OVERHEAD)
> > +#define DESCRIPTORS_PER_TX_MSG 2
> > +
> > +/**
> > + * struct scmi_vio_channel - Transport channel information
> > + *
> > + * @lock: Protects access to all members except ready.
> > + * @ready_lock: Protects access to ready. If required, it must be taken before
> > + *              lock.
> > + * @vqueue: Associated virtqueue
> > + * @cinfo: SCMI Tx or Rx channel
> > + * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
> > + * @is_rx: Whether channel is an Rx channel
> > + * @ready: Whether transport user is ready to hear about channel
> > + */
> > +struct scmi_vio_channel {
> > +	spinlock_t lock;
> > +	spinlock_t ready_lock;
> > +	struct virtqueue *vqueue;
> > +	struct scmi_chan_info *cinfo;
> > +	struct list_head free_list;
> > +	u8 is_rx;
> > +	u8 ready;
> > +};
> > +
> > +/**
> > + * struct scmi_vio_msg - Transport PDU information
> > + *
> > + * @request: SDU used for commands
> > + * @input: SDU used for (delayed) responses and notifications
> > + * @list: List which scmi_vio_msg may be part of
> > + * @rx_len: Input SDU size in bytes, once input has been received
> > + */
> > +struct scmi_vio_msg {
> > +	struct scmi_msg_payld *request;
> > +	struct scmi_msg_payld *input;
> > +	struct list_head list;
> > +	unsigned int rx_len;
> > +};
> > +
> > +static bool scmi_vio_have_vq_rx(struct virtio_device *vdev)
> > +{
> > +	return virtio_has_feature(vdev, VIRTIO_SCMI_F_P2A_CHANNELS);
> > +}
> > +
> > +static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
> > +			       struct scmi_vio_msg *msg)
> > +{
> > +	struct scatterlist sg_in;
> > +	int rc;
> > +	unsigned long flags;
> > +
> > +	sg_init_one(&sg_in, msg->input, VIRTIO_SCMI_MAX_PDU_SIZE);
> > +
> > +	spin_lock_irqsave(&vioch->lock, flags);
> > +
> > +	rc = virtqueue_add_inbuf(vioch->vqueue, &sg_in, 1, msg, GFP_ATOMIC);
> > +	if (rc)
> > +		dev_err_once(vioch->cinfo->dev,
> > +			     "%s() failed to add to virtqueue (%d)\n", __func__,
> > +			     rc);
> > +	else
> > +		virtqueue_kick(vioch->vqueue);
> > +
> > +	spin_unlock_irqrestore(&vioch->lock, flags);
> > +
> > +	return rc;
> > +}
> > +
> > +static void scmi_vio_complete_cb(struct virtqueue *vqueue)
> > +{
> > +	unsigned long ready_flags;
> > +	unsigned long flags;
> > +	unsigned int length;
> > +	struct scmi_vio_channel *vioch;
> > +	struct scmi_vio_msg *msg;
> > +	bool cb_enabled = true;
> > +
> > +	if (WARN_ON_ONCE(!vqueue->vdev->priv))
> > +		return;
> > +	vioch = &((struct scmi_vio_channel *)vqueue->vdev->priv)[vqueue->index];
> > +
> > +	for (;;) {
> > +		spin_lock_irqsave(&vioch->ready_lock, ready_flags);
> > +
> > +		if (!vioch->ready) {
> > +			if (!cb_enabled)
> > +				(void)virtqueue_enable_cb(vqueue);
> > +			goto unlock_ready_out;
> > +		}
> > +
> > +		spin_lock_irqsave(&vioch->lock, flags);
> > +		if (cb_enabled) {
> > +			virtqueue_disable_cb(vqueue);
> > +			cb_enabled = false;
> > +		}
> > +		msg = virtqueue_get_buf(vqueue, &length);
> > +		if (!msg) {
> > +			if (virtqueue_enable_cb(vqueue))
> > +				goto unlock_out;
> > +			else
> 
> Drop the else as it doesn't add readability as far as I can see.
> 

Sure.

> > +				cb_enabled = true;
> > +		}
> > +		spin_unlock_irqrestore(&vioch->lock, flags);
> > +
> > +		if (msg) {
> > +			msg->rx_len = length;
> > +
> > +			/*
> > +			 * Hold the ready_lock during the callback to avoid
> > +			 * races when the arm-scmi driver is unbinding while
> > +			 * the virtio device is not quiesced yet.
> > +			 */
> > +			scmi_rx_callback(vioch->cinfo,
> > +					 msg_read_header(msg->input), msg);
> > +		}
> > +		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> > +	}
> > +
> > +unlock_out:
> > +	spin_unlock_irqrestore(&vioch->lock, flags);
> > +unlock_ready_out:
> > +	spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> > +}
> > +
> > +static const char *const scmi_vio_vqueue_names[] = { "tx", "rx" };
> > +
> > +static vq_callback_t *scmi_vio_complete_callbacks[] = {
> > +	scmi_vio_complete_cb,
> > +	scmi_vio_complete_cb
> > +};
> > +
> > +static unsigned int virtio_get_max_msg(bool tx,
> > +				       struct scmi_chan_info *base_cinfo)
> > +{
> > +	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
> > +	unsigned int ret;
> > +
> > +	ret = virtqueue_get_vring_size(vioch->vqueue);
> > +
> > +	/* Tx messages need multiple descriptors. */
> > +	if (tx)
> > +		ret /= DESCRIPTORS_PER_TX_MSG;
> > +
> > +	if (ret > MSG_TOKEN_MAX) {
> > +		dev_info_once(
> > +			base_cinfo->dev,
> > +			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
> > +			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
> > +		ret = MSG_TOKEN_MAX;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int scmi_vio_match_any_dev(struct device *dev, const void *data)
> > +{
> > +	return 1;
> > +}
> > +
> > +static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
> > +
> > +static int virtio_link_supplier(struct device *dev)
> > +{
> > +	struct device *vdev = driver_find_device(
> > +		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
> > +
> > +	if (!vdev) {
> > +		dev_notice_once(
> > +			dev,
> > +			"Deferring probe after not finding a bound scmi-virtio device\n");
> > +		return -EPROBE_DEFER;
> > +	}
> > +
> > +	/* Add device link for remove order and sysfs link. */
> > +	if (!device_link_add(dev, vdev, DL_FLAG_AUTOREMOVE_CONSUMER)) {
> > +		put_device(vdev);
> > +		dev_err(dev, "Adding link to supplier virtio device failed\n");
> > +		return -ECANCELED;
> > +	}
> > +
> > +	put_device(vdev);
> > +	return scmi_set_transport_info(dev, dev_to_virtio(vdev));
> > +}
> > +
> > +static bool virtio_chan_available(struct device *dev, int idx)
> > +{
> > +	struct virtio_device *vdev;
> > +
> > +	/* scmi-virtio doesn't support per-protocol channels */
> > +	if (is_scmi_protocol_device(dev))
> > +		return false;
> > +
> > +	vdev = scmi_get_transport_info(dev);
> > +	if (!vdev)
> > +		return false;
> > +
> > +	switch (idx) {
> > +	case VIRTIO_SCMI_VQ_TX:
> > +		return true;
> > +	case VIRTIO_SCMI_VQ_RX:
> > +		return scmi_vio_have_vq_rx(vdev);
> > +	default:
> > +		return false;
> > +	}
> > +}
> > +
> > +static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> > +			     bool tx)
> > +{
> > +	unsigned long flags;
> > +	struct virtio_device *vdev;
> > +	struct scmi_vio_channel *vioch;
> > +	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
> > +	int max_msg;
> > +	int i;
> > +
> > +	if (!virtio_chan_available(dev, index))
> > +		return -ENODEV;
> > +
> > +	vdev = scmi_get_transport_info(dev);
> > +	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
> > +
> > +	spin_lock_irqsave(&vioch->lock, flags);
> > +	cinfo->transport_info = vioch;
> > +	vioch->cinfo = cinfo;
> > +	spin_unlock_irqrestore(&vioch->lock, flags);
> > +
> > +	max_msg = virtio_get_max_msg(tx, cinfo);
> > +
> > +	for (i = 0; i < max_msg; i++) {
> > +		struct scmi_vio_msg *msg;
> > +
> > +		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
> > +		if (!msg)
> > +			return -ENOMEM;
> > +
> > +		if (tx) {
> > +			msg->request = devm_kzalloc(cinfo->dev,
> > +						    VIRTIO_SCMI_MAX_PDU_SIZE,
> > +						    GFP_KERNEL);
> > +			if (!msg->request)
> > +				return -ENOMEM;
> > +		}
> > +
> > +		msg->input = devm_kzalloc(cinfo->dev, VIRTIO_SCMI_MAX_PDU_SIZE,
> > +					  GFP_KERNEL);
> > +		if (!msg->input)
> > +			return -ENOMEM;
> > +
> > +		if (tx) {
> > +			spin_lock_irqsave(&vioch->lock, flags);
> > +			list_add_tail(&msg->list, &vioch->free_list);
> > +			spin_unlock_irqrestore(&vioch->lock, flags);
> > +		} else {
> > +			scmi_vio_feed_vq_rx(vioch, msg);
> > +		}
> > +	}
> > +
> > +	spin_lock_irqsave(&vioch->ready_lock, flags);
> > +	vioch->ready = true;
> > +	spin_unlock_irqrestore(&vioch->ready_lock, flags);
> > +
> > +	return 0;
> > +}
> > +
> > +static int virtio_chan_free(int id, void *p, void *data)
> > +{
> > +	unsigned long flags;
> > +	struct scmi_chan_info *cinfo = p;
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +
> > +	spin_lock_irqsave(&vioch->ready_lock, flags);
> > +	vioch->ready = false;
> > +	spin_unlock_irqrestore(&vioch->ready_lock, flags);
> > +
> > +	scmi_free_channel(cinfo, data, id);
> 
> Blank line here would be nice + consistent.
> 

Right, I'll do.

> > +	return 0;
> > +}
> > +
> > +static int virtio_send_message(struct scmi_chan_info *cinfo,
> > +			       struct scmi_xfer *xfer)
> > +{
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +	struct scatterlist sg_out;
> > +	struct scatterlist sg_in;
> > +	struct scatterlist *sgs[DESCRIPTORS_PER_TX_MSG] = { &sg_out, &sg_in };
> > +	unsigned long flags;
> > +	int rc;
> > +	struct scmi_vio_msg *msg;
> > +
> > +	/*
> > +	 * TODO: For now, we don't support polling. But it should not be
> > +	 * difficult to add support.
> > +	 */
> > +	if (xfer->hdr.poll_completion)
> > +		return -EINVAL;
> > +
> > +	spin_lock_irqsave(&vioch->lock, flags);
> > +
> > +	if (list_empty(&vioch->free_list)) {
> > +		spin_unlock_irqrestore(&vioch->lock, flags);
> > +		return -EBUSY;
> > +	}
> > +
> > +	msg = list_first_entry(&vioch->free_list, typeof(*msg), list);
> > +	list_del(&msg->list);
> > +
> > +	msg_tx_prepare(msg->request, xfer);
> > +
> > +	sg_init_one(&sg_out, msg->request, msg_command_size(xfer));
> > +	sg_init_one(&sg_in, msg->input, msg_response_size(xfer));
> > +
> > +	rc = virtqueue_add_sgs(vioch->vqueue, sgs, 1, 1, msg, GFP_ATOMIC);
> > +	if (rc) {
> > +		list_add(&msg->list, &vioch->free_list);
> > +		dev_err_once(vioch->cinfo->dev,
> > +			     "%s() failed to add to virtqueue (%d)\n", __func__,
> > +			     rc);
> > +	} else {
> > +		virtqueue_kick(vioch->vqueue);
> > +	}
> > +
> > +	spin_unlock_irqrestore(&vioch->lock, flags);
> > +
> > +	return rc;
> > +}
> > +
> > +static void virtio_fetch_response(struct scmi_chan_info *cinfo,
> > +				  struct scmi_xfer *xfer, void *msg_handle)
> > +{
> > +	struct scmi_vio_msg *msg = msg_handle;
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +
> > +	if (!msg) {
> > +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> > +			     "Ignoring %s() call with NULL msg_handle\n",
> > +			     __func__);
> > +		return;
> > +	}
> > +
> > +	msg_fetch_response(msg->input, msg->rx_len, xfer);
> > +}
> > +
> > +static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
> > +				      size_t max_len, struct scmi_xfer *xfer,
> > +				      void *msg_handle)
> > +{
> > +	struct scmi_vio_msg *msg = msg_handle;
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +
> > +	if (!msg) {
> > +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> > +			     "Ignoring %s() call with NULL msg_handle\n",
> > +			     __func__);
> > +		return;
> > +	}
> > +
> > +	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
> > +}
> > +
> > +static void dummy_clear_channel(struct scmi_chan_info *cinfo)
> > +{
> > +}
> > +
> > +static bool dummy_poll_done(struct scmi_chan_info *cinfo,
> > +			    struct scmi_xfer *xfer)
> > +{
> > +	return false;
> > +}
> > +
> > +static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
> > +{
> > +	unsigned long flags;
> > +	struct scmi_vio_channel *vioch = cinfo->transport_info;
> > +	struct scmi_vio_msg *msg = msg_handle;
> > +
> > +	if (!msg) {
> > +		dev_dbg_once(&vioch->vqueue->vdev->dev,
> > +			     "Ignoring %s() call with NULL msg_handle\n",
> > +			     __func__);
> > +		return;
> > +	}
> > +
> > +	if (vioch->is_rx) {
> > +		scmi_vio_feed_vq_rx(vioch, msg);
> > +	} else {
> > +		spin_lock_irqsave(&vioch->lock, flags);
> > +		list_add(&msg->list, &vioch->free_list);
> > +		spin_unlock_irqrestore(&vioch->lock, flags);
> > +	}
> > +}
> > +
> > +static const struct scmi_transport_ops scmi_virtio_ops = {
> > +	.link_supplier = virtio_link_supplier,
> > +	.chan_available = virtio_chan_available,
> > +	.chan_setup = virtio_chan_setup,
> > +	.chan_free = virtio_chan_free,
> > +	.get_max_msg = virtio_get_max_msg,
> > +	.send_message = virtio_send_message,
> > +	.fetch_response = virtio_fetch_response,
> > +	.fetch_notification = virtio_fetch_notification,
> > +	.clear_channel = dummy_clear_channel,
> > +	.poll_done = dummy_poll_done,
> > +	.drop_message = virtio_drop_message,
> > +};
> > +
> > +static int scmi_vio_probe(struct virtio_device *vdev)
> > +{
> > +	struct device *dev = &vdev->dev;
> > +	struct scmi_vio_channel *channels;
> > +	bool have_vq_rx;
> > +	int vq_cnt;
> > +	int i;
> > +	int ret;
> > +	struct virtqueue *vqs[VIRTIO_SCMI_VQ_MAX_CNT];
> > +
> > +	have_vq_rx = scmi_vio_have_vq_rx(vdev);
> > +	vq_cnt = have_vq_rx ? VIRTIO_SCMI_VQ_MAX_CNT : 1;
> > +
> > +	channels = devm_kcalloc(dev, vq_cnt, sizeof(*channels), GFP_KERNEL);
> > +	if (!channels)
> > +		return -ENOMEM;
> > +
> > +	if (have_vq_rx)
> > +		channels[VIRTIO_SCMI_VQ_RX].is_rx = true;
> > +
> > +	ret = virtio_find_vqs(vdev, vq_cnt, vqs, scmi_vio_complete_callbacks,
> > +			      scmi_vio_vqueue_names, NULL);
> > +	if (ret) {
> > +		dev_err(dev, "Failed to get %d virtqueue(s)\n", vq_cnt);
> > +		return ret;
> > +	}
> > +	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);
> 
> Rather noisy given I assume there are plenty of other ways to se this
> succeeded (i.e. probe succeeded)
> 

Yes, indeed, the only additional info that was meant to be carried here
I suppose is that when you find 2 virtqueues (max possible) you know
(deduct really) that you have the P2A channel, so also notifications and
delayed responses are supported.

I'll review in general the info msgs and drop to debug when not neeeded.
(or explicit the above I-have-notif_dresp information.

> > +
> > +	for (i = 0; i < vq_cnt; i++) {
> > +		spin_lock_init(&channels[i].lock);
> > +		spin_lock_init(&channels[i].ready_lock);
> > +		INIT_LIST_HEAD(&channels[i].free_list);
> > +		channels[i].vqueue = vqs[i];
> > +	}
> > +
> > +	vdev->priv = channels;
> > +
> > +	return 0;
> > +}
> > +
> > +static void scmi_vio_remove(struct virtio_device *vdev)
> > +{
> > +	vdev->config->reset(vdev);
> > +	vdev->config->del_vqs(vdev);
> > +}
> > +
> > +static unsigned int features[] = {
> > +	VIRTIO_SCMI_F_P2A_CHANNELS,
> > +};
> > +
> > +static const struct virtio_device_id id_table[] = {
> > +	{ VIRTIO_ID_SCMI, VIRTIO_DEV_ANY_ID },
> > +	{ 0 }
> > +};
> > +
> > +static struct virtio_driver virtio_scmi_driver = {
> > +	.driver.name = "scmi-virtio",
> > +	.driver.owner = THIS_MODULE,
> > +	.feature_table = features,
> > +	.feature_table_size = ARRAY_SIZE(features),
> > +	.id_table = id_table,
> > +	.probe = scmi_vio_probe,
> > +	.remove = scmi_vio_remove,
> > +};
> > +
> > +static int __init virtio_scmi_init(void)
> > +{
> > +	return register_virtio_driver(&virtio_scmi_driver);
> > +}
> > +
> > +static void __exit virtio_scmi_exit(void)
> > +{
> > +	unregister_virtio_driver(&virtio_scmi_driver);
> > +}
> 
> module_virtio_driver() ?
> 

No that's not possible at the moment, and indeed it is the whole
reason I introduced early in the series the transport specific
.init()/.exit() optional helpers.

The issue is that currently SCMI transports are in fact part of the SCMI
stack compilation unit; if you enable some transports those get compiled
in the main SCMI module, and if you then (hopefully) define a DT entry
describing an SCMI instance using a specific transport (setting compatible
"arm,scmi-virtio" in this case) such SCMI stack will use that transport and
its ops: problem is the SCMI main module can be compiled as loadable
module on its own, including all transports configured and all base SCMI
protos, so if you use here the standard module_virtio_driver() it won't
link in case you built the SCMI core as module and complains about

> multiple definition of `init_module'; 
> multiple definition of `cleanup_module';

The above .init()/.exit() instead are registered with the core as
optional transport helpers and invoked when the SCMI core is
initialized, so also well before the SCMI core is probed.

Making it a full fledged module instead (and so then using normal
module_virtio_driver() macros) that registers with the SCMI core somehow
after its probed by its own susbsystem (like virtIO), seemed to
complicate a lot the interactions between the core and the transports
(even though I'd like it much more as an approach) so I have not gone
through this road still....but I cannot exclude I'll come back to this
kind of approach when I'll try to simplify a bit the probing sequence of
this SCMI VirtIO driver...it's just that for now seemed easier to add
these optional transport helpers to let transport provide their early setup
stuff (if needed), like registering with another subsystem.
(upcoming SCMI OPTEE transport posted in a different series by Etienne
has the same needs)

Thanks,
Cristian


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
  2021-06-11 16:59   ` Cristian Marussi
  (?)
@ 2021-06-24 19:22     ` Rob Herring
  -1 siblings, 0 replies; 90+ messages in thread
From: Rob Herring @ 2021-06-24 19:22 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: jean-philippe, Rob Herring, vincent.guittot, alex.bennee,
	virtio-dev, virtualization, Jonathan.Cameron, f.fainelli,
	igor.skalkin, Vasyl.Vavrychuk, linux-kernel, sudeep.holla,
	peter.hilber, Andriy.Tryshnivskyy, anton.yakovlev,
	souvik.chakravarty, mikhail.golubev, linux-arm-kernel,
	devicetree, james.quinlan, etienne.carriere

On Fri, 11 Jun 2021 17:59:34 +0100, Cristian Marussi wrote:
> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> Document the properties for arm,scmi-virtio compatible nodes.
> The backing virtio SCMI device is described in patch [1].
> 
> While doing that, make shmem property required only for pre-existing
> mailbox and smc transports, since virtio-scmi does not need it.
> 
> [1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html
> 
> CC: Rob Herring <robh+dt@kernel.org>
> CC: devicetree@vger.kernel.org
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> [ Peter: Adapted patch for submission to upstream. ]
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> [ Cristian: converted to yaml format, moved shmen required property. ]
> Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
> v3 --> V4
> - convertd to YAML
> - make shmem required only for pre-existing mailbox and smc transport
> - updated VirtIO specification patch message reference
> - dropped virtio-mmio SCMI device example since really not pertinent to
>   virtio-scmi dt bindings transport: it is not even referenced in SCMI
>   virtio DT node since they are enumerated by VirtIO subsystem and there
>   could be PCI based SCMI devices anyway.
> ---
>  Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 

Reviewed-by: Rob Herring <robh@kernel.org>

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
@ 2021-06-24 19:22     ` Rob Herring
  0 siblings, 0 replies; 90+ messages in thread
From: Rob Herring @ 2021-06-24 19:22 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: jean-philippe, mikhail.golubev, f.fainelli, vincent.guittot,
	igor.skalkin, virtio-dev, devicetree, sudeep.holla, linux-kernel,
	virtualization, Vasyl.Vavrychuk, Rob Herring, peter.hilber,
	Andriy.Tryshnivskyy, james.quinlan, Jonathan.Cameron,
	souvik.chakravarty, etienne.carriere, linux-arm-kernel

On Fri, 11 Jun 2021 17:59:34 +0100, Cristian Marussi wrote:
> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> Document the properties for arm,scmi-virtio compatible nodes.
> The backing virtio SCMI device is described in patch [1].
> 
> While doing that, make shmem property required only for pre-existing
> mailbox and smc transports, since virtio-scmi does not need it.
> 
> [1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html
> 
> CC: Rob Herring <robh+dt@kernel.org>
> CC: devicetree@vger.kernel.org
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> [ Peter: Adapted patch for submission to upstream. ]
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> [ Cristian: converted to yaml format, moved shmen required property. ]
> Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
> v3 --> V4
> - convertd to YAML
> - make shmem required only for pre-existing mailbox and smc transport
> - updated VirtIO specification patch message reference
> - dropped virtio-mmio SCMI device example since really not pertinent to
>   virtio-scmi dt bindings transport: it is not even referenced in SCMI
>   virtio DT node since they are enumerated by VirtIO subsystem and there
>   could be PCI based SCMI devices anyway.
> ---
>  Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 

Reviewed-by: Rob Herring <robh@kernel.org>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
@ 2021-06-24 19:22     ` Rob Herring
  0 siblings, 0 replies; 90+ messages in thread
From: Rob Herring @ 2021-06-24 19:22 UTC (permalink / raw)
  To: Cristian Marussi
  Cc: jean-philippe, Rob Herring, vincent.guittot, alex.bennee,
	virtio-dev, virtualization, Jonathan.Cameron, f.fainelli,
	igor.skalkin, Vasyl.Vavrychuk, linux-kernel, sudeep.holla,
	peter.hilber, Andriy.Tryshnivskyy, anton.yakovlev,
	souvik.chakravarty, mikhail.golubev, linux-arm-kernel,
	devicetree, james.quinlan, etienne.carriere

On Fri, 11 Jun 2021 17:59:34 +0100, Cristian Marussi wrote:
> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> Document the properties for arm,scmi-virtio compatible nodes.
> The backing virtio SCMI device is described in patch [1].
> 
> While doing that, make shmem property required only for pre-existing
> mailbox and smc transports, since virtio-scmi does not need it.
> 
> [1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html
> 
> CC: Rob Herring <robh+dt@kernel.org>
> CC: devicetree@vger.kernel.org
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> [ Peter: Adapted patch for submission to upstream. ]
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> [ Cristian: converted to yaml format, moved shmen required property. ]
> Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
> v3 --> V4
> - convertd to YAML
> - make shmem required only for pre-existing mailbox and smc transport
> - updated VirtIO specification patch message reference
> - dropped virtio-mmio SCMI device example since really not pertinent to
>   virtio-scmi dt bindings transport: it is not even referenced in SCMI
>   virtio DT node since they are enumerated by VirtIO subsystem and there
>   could be PCI based SCMI devices anyway.
> ---
>  Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 

Reviewed-by: Rob Herring <robh@kernel.org>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 01/16] firmware: arm_scmi: Fix max pending messages boundary check
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-07-01  8:42     ` Peter Hilber
  -1 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:42 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi Cristian,

please find some remarks to the patch series in this email and the 
following.

On 11.06.21 18:59, Cristian Marussi wrote:
> SCMI message headers carry a sequence number and such field is sized to
> allow for MSG_TOKEN_MAX distinct numbers; moreover zero is not really an
> acceptable maximum number of pending in-flight messages.
> 
> Fix accordignly the checks performed on the value exported by transports
> in scmi_desc.max_msg.
> 
> Reported-by: Vincent Guittot <vincent.guittot@linaro.org>
> Fixes: aa4f886f3893 ("firmware: arm_scmi: add basic driver infrastructure for SCMI")
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/driver.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 66e5e694be7d..6713b259f1e6 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -1025,8 +1025,9 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  	const struct scmi_desc *desc = sinfo->desc;
>  
>  	/* Pre-allocated messages, no more than what hdr.seq can support */
> -	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
> -		dev_err(dev, "Maximum message of %d exceeds supported %ld\n",
> +	if (WARN_ON(!desc->max_msg || desc->max_msg > MSG_TOKEN_MAX)) {
> +		dev_err(dev,
> +			"Invalid max_msg %d. Maximum messages supported %ld.\n",

%ld -> %lu

>  			desc->max_msg, MSG_TOKEN_MAX);
>  		return -EINVAL;
>  	}
>

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 01/16] firmware: arm_scmi: Fix max pending messages boundary check
@ 2021-07-01  8:42     ` Peter Hilber
  0 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:42 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi Cristian,

please find some remarks to the patch series in this email and the 
following.

On 11.06.21 18:59, Cristian Marussi wrote:
> SCMI message headers carry a sequence number and such field is sized to
> allow for MSG_TOKEN_MAX distinct numbers; moreover zero is not really an
> acceptable maximum number of pending in-flight messages.
> 
> Fix accordignly the checks performed on the value exported by transports
> in scmi_desc.max_msg.
> 
> Reported-by: Vincent Guittot <vincent.guittot@linaro.org>
> Fixes: aa4f886f3893 ("firmware: arm_scmi: add basic driver infrastructure for SCMI")
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/driver.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 66e5e694be7d..6713b259f1e6 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -1025,8 +1025,9 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  	const struct scmi_desc *desc = sinfo->desc;
>  
>  	/* Pre-allocated messages, no more than what hdr.seq can support */
> -	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
> -		dev_err(dev, "Maximum message of %d exceeds supported %ld\n",
> +	if (WARN_ON(!desc->max_msg || desc->max_msg > MSG_TOKEN_MAX)) {
> +		dev_err(dev,
> +			"Invalid max_msg %d. Maximum messages supported %ld.\n",

%ld -> %lu

>  			desc->max_msg, MSG_TOKEN_MAX);
>  		return -EINVAL;
>  	}
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-07-01  8:42     ` Peter Hilber
  -1 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:42 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

I find `monotonically increasing tokens' misleading, since the token may 
wrap around quite often during normal usage. How about `modulo 
incrementing'?

On 11.06.21 18:59, Cristian Marussi wrote:
> Tokens are sequence numbers embedded in the each SCMI message header: they
> are used to correlate commands with responses (and delayed responses), but
> their usage and policy of selection is entirely up to the caller (usually
> the OSPM agent), while they are completely opaque to the callee (SCMI
> server platform) which merely copies them back from the command into the
> response message header.
> This also means that the platform does not, can not and should not enforce
> any kind of policy on received messages depending on the contained sequence
> number: platform can perfectly handle concurrent requests carrying the same
> identifiying token if that should happen.
> 
> Moreover the platform is not required to produce in-order responses to
> agent requests, the only constraint in these regards is that in case of
> an asynchronous message the delayed response must be sent after the
> immediate response for the synchronous part of the command transaction.
> 
> Currenly the SCMI stack of the OSPM agent selects a token for the egressing
> commands picking the lowest possible number which is not already in use by
> an existing in-flight transaction, which means, in other words, that we
> immediately reuse any token after its transaction has completed or it has
> timed out: this policy indeed does simplify management and lookup of tokens
> and associated xfers.
> 
> Under the above assumptions and constraints, since there is really no state
> shared between the agent and the platform to let the platform know when a
> token and its associated message has timed out, the current policy of early
> reuse of tokens can easily lead to the situation in which a spurious or
> late received response (or delayed_response), related to an old stale and
> timed out transaction, can be wrongly associated to a newer valid in-flight
> xfer that just happens to have reused the same token.
> 
> This misbehaviour on such ghost responses is more easily exposed on those
> transports that naturally have an higher level of parallelism in processing
> multiple concurrent in-flight messages.
> 
> This commit introduces a new policy of selection of tokens for the OSPM
> agent: each new transfer now gets the next available and monotonically
> increasing token, until tokens are exhausted and the counter rolls over.
> 
> Such new policy mitigates the above issues with ghost responses since the
> tokens are now reused as late as possible (when they roll back ideally)
> and so it is much easier to identify such ghost responses to stale timed
> out transactions: this also helps in simplifying the specific transports
> implementation since stale transport messages can be easily identified
> and discarded early on in the rx path without the need to cross check
> their actual state with the core transport layer.
> This mitigation is even more effective when, as is usually the case, the
> maximum number of pending messages is capped by the platform to a much
> lower number than the whole possible range of tokens values (2^10).
> 
> This internal policy change in the core SCMI transport layer is fully
> transparent to the specific transports so it has not and should not have
> any impact on the transports implementation.
> 
> The empirically observed cost of such new procedure of token selection
> amounts in the best case to ~10us out of an observed full transaction cost
> of 3ms for the completion of a synchronous sensor reading command on a
> platform supporting commands completion interrupts.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/common.h |  23 +++
>  drivers/firmware/arm_scmi/driver.c | 243 +++++++++++++++++++++++++----
>  2 files changed, 233 insertions(+), 33 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index 6bb734e0e3ac..e64c5ca9ee7c 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -14,7 +14,10 @@
>  #include <linux/device.h>
>  #include <linux/errno.h>
>  #include <linux/kernel.h>
> +#include <linux/hashtable.h>
> +#include <linux/list.h>
>  #include <linux/module.h>
> +#include <linux/refcount.h>
>  #include <linux/scmi_protocol.h>
>  #include <linux/types.h>
>  
> @@ -127,6 +130,21 @@ struct scmi_msg {
>  	size_t len;
>  };
>  
> +/**
> + * An helper macro to lookup an xfer from the @pending_xfers hashtable
> + * using the message sequence number token as a key.
> + */
> +#define XFER_FIND(__ht, __k)					\
> +({								\
> +	typeof(__k) k_ = __k;					\
> +	struct scmi_xfer *xfer_ = NULL;				\
> +								\
> +	hash_for_each_possible((__ht), xfer_, node, k_)		\
> +		if (xfer_->hdr.seq == k_)			\
> +			break;					\
> +	 xfer_;							\

There is an extra space before the return value.

> +})
> +
>  /**
>   * struct scmi_xfer - Structure representing a message flow
>   *
> @@ -138,6 +156,9 @@ struct scmi_msg {
>   *	buffer for the rx path as we use for the tx path.
>   * @done: command message transmit completion event
>   * @async_done: pointer to delayed response message received event completion
> + * @users: A refcount to track the active users for this xfer
> + * @node: An hlist_node reference used to store this xfer, alternatively, on
> + *	  the free list @free_xfers or in the @pending_xfers hashtable
>   */
>  struct scmi_xfer {
>  	int transfer_id;
> @@ -146,6 +167,8 @@ struct scmi_xfer {
>  	struct scmi_msg rx;
>  	struct completion done;
>  	struct completion *async_done;
> +	refcount_t users;
> +	struct hlist_node node;
>  };
>  
>  struct scmi_xfer_ops;
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 20f8f0581f3a..f0b20ddb24f4 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -21,6 +21,7 @@
>  #include <linux/io.h>
>  #include <linux/kernel.h>
>  #include <linux/ktime.h>
> +#include <linux/hashtable.h>
>  #include <linux/list.h>
>  #include <linux/module.h>
>  #include <linux/of_address.h>
> @@ -65,19 +66,29 @@ struct scmi_requested_dev {
>  	struct list_head node;
>  };
>  
> +#define SCMI_PENDING_XFERS_HT_ORDER_SZ	9

Micro-optimization note: This increases struct scmi_info size to > 8 kiB 
(on 64-bit architectures). A lower size of the hash table would be 
enough for some transports.

> +
>  /**
>   * struct scmi_xfers_info - Structure to manage transfer information
>   *
> - * @xfer_block: Preallocated Message array
>   * @xfer_alloc_table: Bitmap table for allocated messages.
>   *	Index of this bitmap table is also used for message
>   *	sequence identifier.
>   * @xfer_lock: Protection for message allocation
> + * @last_token: A counter to use as base to generate for monotonically
> + *		increasing tokens.
> + * @free_xfers: A free list for available to use xfers. It is initialized with
> + *		a number of xfers equal to the maximum allowed in-flight
> + *		messages.
> + * @pending_xfers: An hashtable, indexed by msg_hdr.seq, used to keep all the
> + *		   currently in-flight messages.
>   */
>  struct scmi_xfers_info {
> -	struct scmi_xfer *xfer_block;
>  	unsigned long *xfer_alloc_table;
>  	spinlock_t xfer_lock;
> +	atomic_t last_token;
> +	struct hlist_head free_xfers;
> +	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
>  };
>  
>  /**
> @@ -203,6 +214,117 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
>  	return info->notify_priv;
>  }
>  
> +/**
> + * scmi_xfer_token_set  - Reserve and set new token for the xfer at hand
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer: The xfer to act upon
> + *
> + * Pick the next unused monotonically increasing token and set it into
> + * xfer->hdr.seq: picking a monotonically increasing value avoids immediate
> + * reuse of freshly completed or timed-out xfers, thus mitigating the risk
> + * of incorrect association of a late and expired xfer with a live in-flight
> + * transaction, both happening to re-use the same token identifier.
> + *
> + * Since platform is NOT required to answer our request in-order we should
> + * account for a few rare but possible scenarios:
> + *
> + *  - exactly 'next_token' may be NOT available so pick xfer_id >= next_token
> + *    using find_next_zero_bit() starting from candidate next_token bit
> + *
> + *  - all tokens ahead upto (MSG_TOKEN_ID_MASK - 1) are used in-flight but we
> + *    are plenty of free tokens at start, so try a second pass using
> + *    find_next_zero_bit() and starting from 0.
> + *
> + *  X = used in-flight
> + *
> + * Normal
> + * ------
> + *
> + *		|- xfer_id picked
> + *   -----------+----------------------------------------------------------
> + *   | | |X|X|X| | | | | | ... ... ... ... ... ... ... ... ... ... ...|X|X|
> + *   ----------------------------------------------------------------------
> + *		^
> + *		|- next_token
> + *
> + * Out-of-order pending at start
> + * -----------------------------
> + *
> + *	  |- xfer_id picked, last_token fixed
> + *   -----+----------------------------------------------------------------
> + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... ... ...|X| |
> + *   ----------------------------------------------------------------------
> + *    ^
> + *    |- next_token
> + *
> + *
> + * Out-of-order pending at end
> + * ---------------------------
> + *
> + *	  |- xfer_id picked, last_token fixed
> + *   -----+----------------------------------------------------------------
> + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... |X|X|X||X|X|
> + *   ----------------------------------------------------------------------
> + *								^
> + *								|- next_token
> + *
> + * Context: Assumes to be called with @xfer_lock already acquired.
> + *
> + * Return: 0 on Success or error
> + */
> +static int scmi_xfer_token_set(struct scmi_xfers_info *minfo,
> +			       struct scmi_xfer *xfer)
> +{
> +	unsigned long xfer_id, next_token;
> +
> +	/* Pick a candidate monotonic token in range [0, MSG_TOKEN_MAX - 1] */
> +	next_token = (atomic_inc_return(&minfo->last_token) &
> +		      (MSG_TOKEN_MAX - 1));
> +
> +	/* Pick the next available xfer_id >= next_token */
> +	xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> +				     MSG_TOKEN_MAX, next_token);
> +	if (xfer_id == MSG_TOKEN_MAX) {
> +		/*
> +		 * After heavily out-of-order responses, there are no free
> +		 * tokens ahead, but only at start of xfer_alloc_table so
> +		 * try again from the beginning.
> +		 */
> +		xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> +					     MSG_TOKEN_MAX, 0);
> +		/*
> +		 * Something is wrong if we got here since there can be a
> +		 * maximum number of (MSG_TOKEN_MAX - 1) in-flight messages
> +		 * but we have not found any free token [0, MSG_TOKEN_MAX - 1].
> +		 */
> +		if (WARN_ON_ONCE(xfer_id == MSG_TOKEN_MAX))
> +			return -ENOMEM;
> +	}
> +
> +	/* Update +/- last_token accordingly if we skipped some hole */
> +	if (xfer_id != next_token)
> +		atomic_add((int)(xfer_id - next_token), &minfo->last_token);
> +
> +	/* Set in-flight */
> +	set_bit(xfer_id, minfo->xfer_alloc_table);
> +	xfer->hdr.seq = (u16)xfer_id;
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_xfer_token_clear  - Release the token
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer: The xfer to act upon
> + */
> +static inline void scmi_xfer_token_clear(struct scmi_xfers_info *minfo,
> +					 struct scmi_xfer *xfer)
> +{
> +	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> +}
> +
>  /**
>   * scmi_xfer_get() - Allocate one message
>   *
> @@ -212,36 +334,49 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
>   * Helper function which is used by various message functions that are
>   * exposed to clients of this driver for allocating a message traffic event.
>   *
> - * This function can sleep depending on pending requests already in the system
> - * for the SCMI entity. Further, this also holds a spinlock to maintain
> - * integrity of internal data structures.
> + * Picks an xfer from the free list @free_xfers (if any available), sets a
> + * monotonically increasing token and stores the inflight xfer into the
> + * @pending_xfers hashtable for later retrieval.
> + *
> + * The successfully initialized xfer is refcounted.
> + *
> + * Context: Holds @xfer_lock while manipulating @xfer_alloc_table and
> + *	    @free_xfers.
>   *
>   * Return: 0 if all went fine, else corresponding error.
>   */
>  static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
>  				       struct scmi_xfers_info *minfo)
>  {
> -	u16 xfer_id;
> +	int ret;
> +	unsigned long flags;
>  	struct scmi_xfer *xfer;
> -	unsigned long flags, bit_pos;
> -	struct scmi_info *info = handle_to_scmi_info(handle);
>  
> -	/* Keep the locked section as small as possible */
>  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> -	bit_pos = find_first_zero_bit(minfo->xfer_alloc_table,
> -				      info->desc->max_msg);
> -	if (bit_pos == info->desc->max_msg) {
> +	if (hlist_empty(&minfo->free_xfers)) {
>  		spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  		return ERR_PTR(-ENOMEM);
>  	}
> -	set_bit(bit_pos, minfo->xfer_alloc_table);
> -	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  
> -	xfer_id = bit_pos;
> +	/* grab an xfer from the free_list */
> +	xfer = hlist_entry(minfo->free_xfers.first, struct scmi_xfer, node);
> +	hlist_del_init(&xfer->node);
>  
> -	xfer = &minfo->xfer_block[xfer_id];
> -	xfer->hdr.seq = xfer_id;
> -	xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> +	/* Pick and set monotonic token */
> +	ret = scmi_xfer_token_set(minfo, xfer);
> +	if (!ret) {
> +		hash_add(minfo->pending_xfers, &xfer->node, xfer->hdr.seq);
> +	} else {
> +		dev_err(handle->dev, "Failed to get monotonic token %d\n", ret);
> +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> +		xfer = ERR_PTR(ret);
> +	}
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +
> +	if (!IS_ERR(xfer)) {
> +		refcount_set(&xfer->users, 1);

Maybe it would be better to do this inside the lock, so that there is no 
(unlikely) race with refcount_inc() in scmi_xfer_acquire().

> +		xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> +	}
>  
>  	return xfer;
>  }
> @@ -252,6 +387,9 @@ static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
>   * @minfo: Pointer to Tx/Rx Message management info based on channel type
>   * @xfer: message that was reserved by scmi_xfer_get
>   *
> + * After refcount check, possibly release an xfer, clearing the token slot,
> + * removing xfer from @pending_xfers and putting it back into free_xfers.
> + *
>   * This holds a spinlock to maintain integrity of internal data structures.
>   */
>  static void
> @@ -259,16 +397,41 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>  {
>  	unsigned long flags;
>  
> -	/*
> -	 * Keep the locked section as small as possible
> -	 * NOTE: we might escape with smp_mb and no lock here..
> -	 * but just be conservative and symmetric.
> -	 */
>  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> -	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> +	if (refcount_dec_and_test(&xfer->users)) {
> +		scmi_xfer_token_clear(minfo, xfer);
> +		hash_del(&xfer->node);
> +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> +	}
>  	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  }
>  
> +/**
> + * scmi_xfer_lookup_unlocked  -  Helper to lookup an xfer_id
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer_id: Token ID to lookup in @pending_xfers
> + *
> + * Refcounting is untouched.
> + *
> + * Context: Assumes to be called with @xfer_lock already acquired.
> + *
> + * Return: A valid xfer on Success or error otherwise
> + */
> +static struct scmi_xfer *
> +scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
> +{
> +	struct scmi_xfer *xfer = NULL;
> +
> +	if (xfer_id >= MSG_TOKEN_MAX)
> +		return ERR_PTR(-EINVAL);
> +
> +	if (test_bit(xfer_id, minfo->xfer_alloc_table))
> +		xfer = XFER_FIND(minfo->pending_xfers, xfer_id);
> +
> +	return xfer ?: ERR_PTR(-EINVAL);
> +}
> +
>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  {
>  	struct scmi_xfer *xfer;
> @@ -305,19 +468,22 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  static void scmi_handle_response(struct scmi_chan_info *cinfo,
>  				 u16 xfer_id, u8 msg_type)
>  {
> +	unsigned long flags;
>  	struct scmi_xfer *xfer;
>  	struct device *dev = cinfo->dev;
>  	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>  	struct scmi_xfers_info *minfo = &info->tx_minfo;
>  
>  	/* Are we even expecting this? */
> -	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
> +	spin_lock_irqsave(&minfo->xfer_lock, flags);
> +	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +	if (IS_ERR(xfer)) {
>  		dev_err(dev, "message for %d is not expected!\n", xfer_id);
>  		info->desc->ops->clear_channel(cinfo);
>  		return;
>  	}
>  
> -	xfer = &minfo->xfer_block[xfer_id];
>  	/*
>  	 * Even if a response was indeed expected on this slot at this point,
>  	 * a buggy platform could wrongly reply feeding us an unexpected
> @@ -1033,18 +1199,25 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  		return -EINVAL;
>  	}
>  
> -	info->xfer_block = devm_kcalloc(dev, desc->max_msg,
> -					sizeof(*info->xfer_block), GFP_KERNEL);
> -	if (!info->xfer_block)
> -		return -ENOMEM;
> +	hash_init(info->pending_xfers);
>  
> -	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(desc->max_msg),
> +	/* Allocate a bitmask sized to hold MSG_TOKEN_MAX tokens */
> +	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(MSG_TOKEN_MAX),
>  					      sizeof(long), GFP_KERNEL);
>  	if (!info->xfer_alloc_table)
>  		return -ENOMEM;
>  
> -	/* Pre-initialize the buffer pointer to pre-allocated buffers */
> -	for (i = 0, xfer = info->xfer_block; i < desc->max_msg; i++, xfer++) {
> +	/*
> +	 * Preallocate a number of xfers equal to max inflight messages,
> +	 * pre-initialize the buffer pointer to pre-allocated buffers and
> +	 * attach all of them to the free list
> +	 */
> +	INIT_HLIST_HEAD(&info->free_xfers);
> +	for (i = 0; i < desc->max_msg; i++) {
> +		xfer = devm_kzalloc(dev, sizeof(*xfer), GFP_KERNEL);
> +		if (!xfer)
> +			return -ENOMEM;
> +
>  		xfer->rx.buf = devm_kcalloc(dev, sizeof(u8), desc->max_msg_size,
>  					    GFP_KERNEL);
>  		if (!xfer->rx.buf)
> @@ -1052,8 +1225,12 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  
>  		xfer->tx.buf = xfer->rx.buf;
>  		init_completion(&xfer->done);
> +
> +		/* Add initialized xfer to the free list */
> +		hlist_add_head(&xfer->node, &info->free_xfers);
>  	}
>  
> +	atomic_set(&info->last_token, -1);
>  	spin_lock_init(&info->xfer_lock);
>  
>  	return 0;
> 




^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
@ 2021-07-01  8:42     ` Peter Hilber
  0 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:42 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

I find `monotonically increasing tokens' misleading, since the token may 
wrap around quite often during normal usage. How about `modulo 
incrementing'?

On 11.06.21 18:59, Cristian Marussi wrote:
> Tokens are sequence numbers embedded in the each SCMI message header: they
> are used to correlate commands with responses (and delayed responses), but
> their usage and policy of selection is entirely up to the caller (usually
> the OSPM agent), while they are completely opaque to the callee (SCMI
> server platform) which merely copies them back from the command into the
> response message header.
> This also means that the platform does not, can not and should not enforce
> any kind of policy on received messages depending on the contained sequence
> number: platform can perfectly handle concurrent requests carrying the same
> identifiying token if that should happen.
> 
> Moreover the platform is not required to produce in-order responses to
> agent requests, the only constraint in these regards is that in case of
> an asynchronous message the delayed response must be sent after the
> immediate response for the synchronous part of the command transaction.
> 
> Currenly the SCMI stack of the OSPM agent selects a token for the egressing
> commands picking the lowest possible number which is not already in use by
> an existing in-flight transaction, which means, in other words, that we
> immediately reuse any token after its transaction has completed or it has
> timed out: this policy indeed does simplify management and lookup of tokens
> and associated xfers.
> 
> Under the above assumptions and constraints, since there is really no state
> shared between the agent and the platform to let the platform know when a
> token and its associated message has timed out, the current policy of early
> reuse of tokens can easily lead to the situation in which a spurious or
> late received response (or delayed_response), related to an old stale and
> timed out transaction, can be wrongly associated to a newer valid in-flight
> xfer that just happens to have reused the same token.
> 
> This misbehaviour on such ghost responses is more easily exposed on those
> transports that naturally have an higher level of parallelism in processing
> multiple concurrent in-flight messages.
> 
> This commit introduces a new policy of selection of tokens for the OSPM
> agent: each new transfer now gets the next available and monotonically
> increasing token, until tokens are exhausted and the counter rolls over.
> 
> Such new policy mitigates the above issues with ghost responses since the
> tokens are now reused as late as possible (when they roll back ideally)
> and so it is much easier to identify such ghost responses to stale timed
> out transactions: this also helps in simplifying the specific transports
> implementation since stale transport messages can be easily identified
> and discarded early on in the rx path without the need to cross check
> their actual state with the core transport layer.
> This mitigation is even more effective when, as is usually the case, the
> maximum number of pending messages is capped by the platform to a much
> lower number than the whole possible range of tokens values (2^10).
> 
> This internal policy change in the core SCMI transport layer is fully
> transparent to the specific transports so it has not and should not have
> any impact on the transports implementation.
> 
> The empirically observed cost of such new procedure of token selection
> amounts in the best case to ~10us out of an observed full transaction cost
> of 3ms for the completion of a synchronous sensor reading command on a
> platform supporting commands completion interrupts.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/common.h |  23 +++
>  drivers/firmware/arm_scmi/driver.c | 243 +++++++++++++++++++++++++----
>  2 files changed, 233 insertions(+), 33 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index 6bb734e0e3ac..e64c5ca9ee7c 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -14,7 +14,10 @@
>  #include <linux/device.h>
>  #include <linux/errno.h>
>  #include <linux/kernel.h>
> +#include <linux/hashtable.h>
> +#include <linux/list.h>
>  #include <linux/module.h>
> +#include <linux/refcount.h>
>  #include <linux/scmi_protocol.h>
>  #include <linux/types.h>
>  
> @@ -127,6 +130,21 @@ struct scmi_msg {
>  	size_t len;
>  };
>  
> +/**
> + * An helper macro to lookup an xfer from the @pending_xfers hashtable
> + * using the message sequence number token as a key.
> + */
> +#define XFER_FIND(__ht, __k)					\
> +({								\
> +	typeof(__k) k_ = __k;					\
> +	struct scmi_xfer *xfer_ = NULL;				\
> +								\
> +	hash_for_each_possible((__ht), xfer_, node, k_)		\
> +		if (xfer_->hdr.seq == k_)			\
> +			break;					\
> +	 xfer_;							\

There is an extra space before the return value.

> +})
> +
>  /**
>   * struct scmi_xfer - Structure representing a message flow
>   *
> @@ -138,6 +156,9 @@ struct scmi_msg {
>   *	buffer for the rx path as we use for the tx path.
>   * @done: command message transmit completion event
>   * @async_done: pointer to delayed response message received event completion
> + * @users: A refcount to track the active users for this xfer
> + * @node: An hlist_node reference used to store this xfer, alternatively, on
> + *	  the free list @free_xfers or in the @pending_xfers hashtable
>   */
>  struct scmi_xfer {
>  	int transfer_id;
> @@ -146,6 +167,8 @@ struct scmi_xfer {
>  	struct scmi_msg rx;
>  	struct completion done;
>  	struct completion *async_done;
> +	refcount_t users;
> +	struct hlist_node node;
>  };
>  
>  struct scmi_xfer_ops;
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 20f8f0581f3a..f0b20ddb24f4 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -21,6 +21,7 @@
>  #include <linux/io.h>
>  #include <linux/kernel.h>
>  #include <linux/ktime.h>
> +#include <linux/hashtable.h>
>  #include <linux/list.h>
>  #include <linux/module.h>
>  #include <linux/of_address.h>
> @@ -65,19 +66,29 @@ struct scmi_requested_dev {
>  	struct list_head node;
>  };
>  
> +#define SCMI_PENDING_XFERS_HT_ORDER_SZ	9

Micro-optimization note: This increases struct scmi_info size to > 8 kiB 
(on 64-bit architectures). A lower size of the hash table would be 
enough for some transports.

> +
>  /**
>   * struct scmi_xfers_info - Structure to manage transfer information
>   *
> - * @xfer_block: Preallocated Message array
>   * @xfer_alloc_table: Bitmap table for allocated messages.
>   *	Index of this bitmap table is also used for message
>   *	sequence identifier.
>   * @xfer_lock: Protection for message allocation
> + * @last_token: A counter to use as base to generate for monotonically
> + *		increasing tokens.
> + * @free_xfers: A free list for available to use xfers. It is initialized with
> + *		a number of xfers equal to the maximum allowed in-flight
> + *		messages.
> + * @pending_xfers: An hashtable, indexed by msg_hdr.seq, used to keep all the
> + *		   currently in-flight messages.
>   */
>  struct scmi_xfers_info {
> -	struct scmi_xfer *xfer_block;
>  	unsigned long *xfer_alloc_table;
>  	spinlock_t xfer_lock;
> +	atomic_t last_token;
> +	struct hlist_head free_xfers;
> +	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
>  };
>  
>  /**
> @@ -203,6 +214,117 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
>  	return info->notify_priv;
>  }
>  
> +/**
> + * scmi_xfer_token_set  - Reserve and set new token for the xfer at hand
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer: The xfer to act upon
> + *
> + * Pick the next unused monotonically increasing token and set it into
> + * xfer->hdr.seq: picking a monotonically increasing value avoids immediate
> + * reuse of freshly completed or timed-out xfers, thus mitigating the risk
> + * of incorrect association of a late and expired xfer with a live in-flight
> + * transaction, both happening to re-use the same token identifier.
> + *
> + * Since platform is NOT required to answer our request in-order we should
> + * account for a few rare but possible scenarios:
> + *
> + *  - exactly 'next_token' may be NOT available so pick xfer_id >= next_token
> + *    using find_next_zero_bit() starting from candidate next_token bit
> + *
> + *  - all tokens ahead upto (MSG_TOKEN_ID_MASK - 1) are used in-flight but we
> + *    are plenty of free tokens at start, so try a second pass using
> + *    find_next_zero_bit() and starting from 0.
> + *
> + *  X = used in-flight
> + *
> + * Normal
> + * ------
> + *
> + *		|- xfer_id picked
> + *   -----------+----------------------------------------------------------
> + *   | | |X|X|X| | | | | | ... ... ... ... ... ... ... ... ... ... ...|X|X|
> + *   ----------------------------------------------------------------------
> + *		^
> + *		|- next_token
> + *
> + * Out-of-order pending at start
> + * -----------------------------
> + *
> + *	  |- xfer_id picked, last_token fixed
> + *   -----+----------------------------------------------------------------
> + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... ... ...|X| |
> + *   ----------------------------------------------------------------------
> + *    ^
> + *    |- next_token
> + *
> + *
> + * Out-of-order pending at end
> + * ---------------------------
> + *
> + *	  |- xfer_id picked, last_token fixed
> + *   -----+----------------------------------------------------------------
> + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... |X|X|X||X|X|
> + *   ----------------------------------------------------------------------
> + *								^
> + *								|- next_token
> + *
> + * Context: Assumes to be called with @xfer_lock already acquired.
> + *
> + * Return: 0 on Success or error
> + */
> +static int scmi_xfer_token_set(struct scmi_xfers_info *minfo,
> +			       struct scmi_xfer *xfer)
> +{
> +	unsigned long xfer_id, next_token;
> +
> +	/* Pick a candidate monotonic token in range [0, MSG_TOKEN_MAX - 1] */
> +	next_token = (atomic_inc_return(&minfo->last_token) &
> +		      (MSG_TOKEN_MAX - 1));
> +
> +	/* Pick the next available xfer_id >= next_token */
> +	xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> +				     MSG_TOKEN_MAX, next_token);
> +	if (xfer_id == MSG_TOKEN_MAX) {
> +		/*
> +		 * After heavily out-of-order responses, there are no free
> +		 * tokens ahead, but only at start of xfer_alloc_table so
> +		 * try again from the beginning.
> +		 */
> +		xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> +					     MSG_TOKEN_MAX, 0);
> +		/*
> +		 * Something is wrong if we got here since there can be a
> +		 * maximum number of (MSG_TOKEN_MAX - 1) in-flight messages
> +		 * but we have not found any free token [0, MSG_TOKEN_MAX - 1].
> +		 */
> +		if (WARN_ON_ONCE(xfer_id == MSG_TOKEN_MAX))
> +			return -ENOMEM;
> +	}
> +
> +	/* Update +/- last_token accordingly if we skipped some hole */
> +	if (xfer_id != next_token)
> +		atomic_add((int)(xfer_id - next_token), &minfo->last_token);
> +
> +	/* Set in-flight */
> +	set_bit(xfer_id, minfo->xfer_alloc_table);
> +	xfer->hdr.seq = (u16)xfer_id;
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_xfer_token_clear  - Release the token
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer: The xfer to act upon
> + */
> +static inline void scmi_xfer_token_clear(struct scmi_xfers_info *minfo,
> +					 struct scmi_xfer *xfer)
> +{
> +	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> +}
> +
>  /**
>   * scmi_xfer_get() - Allocate one message
>   *
> @@ -212,36 +334,49 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
>   * Helper function which is used by various message functions that are
>   * exposed to clients of this driver for allocating a message traffic event.
>   *
> - * This function can sleep depending on pending requests already in the system
> - * for the SCMI entity. Further, this also holds a spinlock to maintain
> - * integrity of internal data structures.
> + * Picks an xfer from the free list @free_xfers (if any available), sets a
> + * monotonically increasing token and stores the inflight xfer into the
> + * @pending_xfers hashtable for later retrieval.
> + *
> + * The successfully initialized xfer is refcounted.
> + *
> + * Context: Holds @xfer_lock while manipulating @xfer_alloc_table and
> + *	    @free_xfers.
>   *
>   * Return: 0 if all went fine, else corresponding error.
>   */
>  static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
>  				       struct scmi_xfers_info *minfo)
>  {
> -	u16 xfer_id;
> +	int ret;
> +	unsigned long flags;
>  	struct scmi_xfer *xfer;
> -	unsigned long flags, bit_pos;
> -	struct scmi_info *info = handle_to_scmi_info(handle);
>  
> -	/* Keep the locked section as small as possible */
>  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> -	bit_pos = find_first_zero_bit(minfo->xfer_alloc_table,
> -				      info->desc->max_msg);
> -	if (bit_pos == info->desc->max_msg) {
> +	if (hlist_empty(&minfo->free_xfers)) {
>  		spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  		return ERR_PTR(-ENOMEM);
>  	}
> -	set_bit(bit_pos, minfo->xfer_alloc_table);
> -	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  
> -	xfer_id = bit_pos;
> +	/* grab an xfer from the free_list */
> +	xfer = hlist_entry(minfo->free_xfers.first, struct scmi_xfer, node);
> +	hlist_del_init(&xfer->node);
>  
> -	xfer = &minfo->xfer_block[xfer_id];
> -	xfer->hdr.seq = xfer_id;
> -	xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> +	/* Pick and set monotonic token */
> +	ret = scmi_xfer_token_set(minfo, xfer);
> +	if (!ret) {
> +		hash_add(minfo->pending_xfers, &xfer->node, xfer->hdr.seq);
> +	} else {
> +		dev_err(handle->dev, "Failed to get monotonic token %d\n", ret);
> +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> +		xfer = ERR_PTR(ret);
> +	}
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +
> +	if (!IS_ERR(xfer)) {
> +		refcount_set(&xfer->users, 1);

Maybe it would be better to do this inside the lock, so that there is no 
(unlikely) race with refcount_inc() in scmi_xfer_acquire().

> +		xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> +	}
>  
>  	return xfer;
>  }
> @@ -252,6 +387,9 @@ static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
>   * @minfo: Pointer to Tx/Rx Message management info based on channel type
>   * @xfer: message that was reserved by scmi_xfer_get
>   *
> + * After refcount check, possibly release an xfer, clearing the token slot,
> + * removing xfer from @pending_xfers and putting it back into free_xfers.
> + *
>   * This holds a spinlock to maintain integrity of internal data structures.
>   */
>  static void
> @@ -259,16 +397,41 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>  {
>  	unsigned long flags;
>  
> -	/*
> -	 * Keep the locked section as small as possible
> -	 * NOTE: we might escape with smp_mb and no lock here..
> -	 * but just be conservative and symmetric.
> -	 */
>  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> -	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> +	if (refcount_dec_and_test(&xfer->users)) {
> +		scmi_xfer_token_clear(minfo, xfer);
> +		hash_del(&xfer->node);
> +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> +	}
>  	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  }
>  
> +/**
> + * scmi_xfer_lookup_unlocked  -  Helper to lookup an xfer_id
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer_id: Token ID to lookup in @pending_xfers
> + *
> + * Refcounting is untouched.
> + *
> + * Context: Assumes to be called with @xfer_lock already acquired.
> + *
> + * Return: A valid xfer on Success or error otherwise
> + */
> +static struct scmi_xfer *
> +scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
> +{
> +	struct scmi_xfer *xfer = NULL;
> +
> +	if (xfer_id >= MSG_TOKEN_MAX)
> +		return ERR_PTR(-EINVAL);
> +
> +	if (test_bit(xfer_id, minfo->xfer_alloc_table))
> +		xfer = XFER_FIND(minfo->pending_xfers, xfer_id);
> +
> +	return xfer ?: ERR_PTR(-EINVAL);
> +}
> +
>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  {
>  	struct scmi_xfer *xfer;
> @@ -305,19 +468,22 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  static void scmi_handle_response(struct scmi_chan_info *cinfo,
>  				 u16 xfer_id, u8 msg_type)
>  {
> +	unsigned long flags;
>  	struct scmi_xfer *xfer;
>  	struct device *dev = cinfo->dev;
>  	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>  	struct scmi_xfers_info *minfo = &info->tx_minfo;
>  
>  	/* Are we even expecting this? */
> -	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
> +	spin_lock_irqsave(&minfo->xfer_lock, flags);
> +	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +	if (IS_ERR(xfer)) {
>  		dev_err(dev, "message for %d is not expected!\n", xfer_id);
>  		info->desc->ops->clear_channel(cinfo);
>  		return;
>  	}
>  
> -	xfer = &minfo->xfer_block[xfer_id];
>  	/*
>  	 * Even if a response was indeed expected on this slot at this point,
>  	 * a buggy platform could wrongly reply feeding us an unexpected
> @@ -1033,18 +1199,25 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  		return -EINVAL;
>  	}
>  
> -	info->xfer_block = devm_kcalloc(dev, desc->max_msg,
> -					sizeof(*info->xfer_block), GFP_KERNEL);
> -	if (!info->xfer_block)
> -		return -ENOMEM;
> +	hash_init(info->pending_xfers);
>  
> -	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(desc->max_msg),
> +	/* Allocate a bitmask sized to hold MSG_TOKEN_MAX tokens */
> +	info->xfer_alloc_table = devm_kcalloc(dev, BITS_TO_LONGS(MSG_TOKEN_MAX),
>  					      sizeof(long), GFP_KERNEL);
>  	if (!info->xfer_alloc_table)
>  		return -ENOMEM;
>  
> -	/* Pre-initialize the buffer pointer to pre-allocated buffers */
> -	for (i = 0, xfer = info->xfer_block; i < desc->max_msg; i++, xfer++) {
> +	/*
> +	 * Preallocate a number of xfers equal to max inflight messages,
> +	 * pre-initialize the buffer pointer to pre-allocated buffers and
> +	 * attach all of them to the free list
> +	 */
> +	INIT_HLIST_HEAD(&info->free_xfers);
> +	for (i = 0; i < desc->max_msg; i++) {
> +		xfer = devm_kzalloc(dev, sizeof(*xfer), GFP_KERNEL);
> +		if (!xfer)
> +			return -ENOMEM;
> +
>  		xfer->rx.buf = devm_kcalloc(dev, sizeof(u8), desc->max_msg_size,
>  					    GFP_KERNEL);
>  		if (!xfer->rx.buf)
> @@ -1052,8 +1225,12 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
>  
>  		xfer->tx.buf = xfer->rx.buf;
>  		init_completion(&xfer->done);
> +
> +		/* Add initialized xfer to the free list */
> +		hlist_add_head(&xfer->node, &info->free_xfers);
>  	}
>  
> +	atomic_set(&info->last_token, -1);
>  	spin_lock_init(&info->xfer_lock);
>  
>  	return 0;
> 




_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 05/16] firmware: arm_scmi: Introduce delegated xfers support
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-07-01  8:42     ` Peter Hilber
  -1 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:42 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On 11.06.21 18:59, Cristian Marussi wrote:
> Introduce optional support for delegated xfers allocation.
> 
> An SCMI transport can optionally declare to support delegated xfers and
> then use a few helper functions exposed by the core SCMI transport layer to
> query the core for existing in-flight transfers matching a provided message
> header or alternatively and transparently obtain a brand new xfer to handle
> a freshly received notification message.
> In both cases the obtained xfer is uniquely mapped into a specific xfer
> through the means of the message header acting as key.
> 
> In this way such a transport can properly store its own transport specific
> payload into the xfer uniquely associated to the message header before
> even calling into the core scmi_rx_callback() in the usual way, so that
> the transport specific message envelope structures can be freed early
> and there is no more need to keep track of their status till the core
> fully processes the xfer to completion or times out.
> 
> The scmi_rx_callbak() does not need to be modified to carry additional
> transport-specific ancillary data related to such message envelopes since
> an unique natural association is established between the xfer and the
> related message header.
> 
> Existing transports that do not need anything of the above will continue
> to work as before without any change.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/common.h |  14 +++
>  drivers/firmware/arm_scmi/driver.c | 132 ++++++++++++++++++++++++++++-
>  2 files changed, 143 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index e64c5ca9ee7c..0edc04bc434c 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -80,6 +80,7 @@ struct scmi_msg_resp_prot_version {
>   * @status: Status of the transfer once it's complete
>   * @poll_completion: Indicate if the transfer needs to be polled for
>   *	completion or interrupt mode is used
> + * @saved_hdr: A copy of the original msg_hdr
>   */
>  struct scmi_msg_hdr {
>  	u8 id;
> @@ -88,6 +89,7 @@ struct scmi_msg_hdr {
>  	u16 seq;
>  	u32 status;
>  	bool poll_completion;
> +	u32 saved_hdr;
>  };
>  
>  /**
> @@ -154,6 +156,9 @@ struct scmi_msg {
>   * @rx: Receive message, the buffer should be pre-allocated to store
>   *	message. If request-ACK protocol is used, we can reuse the same
>   *	buffer for the rx path as we use for the tx path.
> + * @rx_raw_len: A field which can be optionally used by a specific transport
> + *		to save transport specific message length
> + *		It is not used by the SCMI transport core
>   * @done: command message transmit completion event
>   * @async_done: pointer to delayed response message received event completion
>   * @users: A refcount to track the active users for this xfer
> @@ -165,6 +170,7 @@ struct scmi_xfer {
>  	struct scmi_msg_hdr hdr;
>  	struct scmi_msg tx;
>  	struct scmi_msg rx;
> +	size_t rx_raw_len;
>  	struct completion done;
>  	struct completion *async_done;
>  	refcount_t users;
> @@ -355,6 +361,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
>   * @max_msg: Maximum number of messages that can be pending
>   *	simultaneously in the system
>   * @max_msg_size: Maximum size of data per message that can be handled.
> + * @support_xfers_delegation: A flag to indicate if the described transport
> + *			      will handle delegated xfers, so the core can
> + *			      derive proper related assumptions.
>   */
>  struct scmi_desc {
>  	int (*init)(void);
> @@ -363,6 +372,7 @@ struct scmi_desc {
>  	int max_rx_timeout_ms;
>  	int max_msg;
>  	int max_msg_size;
> +	bool support_xfers_delegation;
>  };
>  
>  extern const struct scmi_desc scmi_mailbox_desc;
> @@ -391,4 +401,8 @@ void scmi_notification_instance_data_set(const struct scmi_handle *handle,
>  					 void *priv);
>  void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
>  
> +int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
> +			  struct scmi_xfer **xfer);
> +void scmi_transfer_release(struct scmi_chan_info *cinfo,
> +			   struct scmi_xfer *xfer);
>  #endif /* _SCMI_COMMON_H */
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index f0b20ddb24f4..371d3804cd79 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -432,6 +432,124 @@ scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
>  	return xfer ?: ERR_PTR(-EINVAL);
>  }
>  
> +/**
> + * scmi_xfer_acquire  -  Helper to lookup and acquire an xfer
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer_id: Token ID to lookup in @pending_xfers
> + *
> + * When a valid xfer is found for the provided @xfer_id, reference counting is
> + * properly updated.
> + *
> + * Return: A valid @xfer on Success or error otherwise.
> + */
> +static struct scmi_xfer *
> +scmi_xfer_acquire(struct scmi_xfers_info *minfo, u16 xfer_id)
> +{
> +	unsigned long flags;
> +	struct scmi_xfer *xfer;
> +
> +	spin_lock_irqsave(&minfo->xfer_lock, flags);
> +	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
> +	if (!IS_ERR(xfer))
> +		refcount_inc(&xfer->users);
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +
> +	return xfer;
> +}
> +
> +/**
> + * scmi_transfer_acquire  -  Lookup for an existing xfer or freshly allocate a
> + * new one depending on the type of the message
> + *
> + * @cinfo: A reference to the channel descriptor.
> + * @msg_hdr: A pointer to the message header to lookup.
> + * @xfer: A reference to the pre-existent or freshly allocated xfer
> + *	  associated with the provided *msg_hdr.
> + *
> + * This function can be used by transports supporting delegated xfers to obtain
> + * a valid @xfer associated with the provided @msg_hdr param.
> + *
> + * The nature of the resulting @xfer depends on the type of message specified in
> + * @msg_hdr:
> + *  - for responses and delayed responses a pre-existent/pre-allocated in-flight
> + *    xfer descriptor will be returned (properly refcounted)
> + *  - for notifications a brand new xfer will be allocated; in this case the
> + *    provided message header sequence number will also be mangled to match
> + *    the token in the freshly allocated xfer: this is needed to establish a
> + *    link between the picked xfer and the msg_hdr that will be subsequently
> + *    passed back via usual scmi_rx_callback().
> + *
> + * Return: 0 if a valid xfer is returned in @xfer, error otherwise.
> + */
> +int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
> +			  struct scmi_xfer **xfer)
> +{
> +	u8 msg_type;
> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> +
> +	if (!xfer || !msg_hdr || !info->desc->support_xfers_delegation)
> +		return -EINVAL;
> +
> +	msg_type = MSG_XTRACT_TYPE(*msg_hdr);
> +	switch (msg_type) {
> +	case MSG_TYPE_COMMAND:
> +	case MSG_TYPE_DELAYED_RESP:
> +		/* Grab an existing xfer for xfer_id */
> +		*xfer = scmi_xfer_acquire(&info->tx_minfo,
> +					  MSG_XTRACT_TOKEN(*msg_hdr));
> +		break;
> +	case MSG_TYPE_NOTIFICATION:
> +		/* Get a brand new RX xfer */
> +		*xfer = scmi_xfer_get(cinfo->handle, &info->rx_minfo);
> +		if (!IS_ERR(*xfer)) {
> +			/* Save original msg_hdr and fix sequence number */
> +			(*xfer)->hdr.saved_hdr = *msg_hdr;

The saved header isn't used anywhere.

> +			*msg_hdr &= ~MSG_TOKEN_ID_MASK;
> +			*msg_hdr |= FIELD_PREP(MSG_TOKEN_ID_MASK,
> +					       (*xfer)->hdr.seq);

This will invalidate the token set by the platform in 
scmi_dump_header_dbg(). Maybe it would have been more elegant to 
introduce a dedicated hash table key field?

> +		}
> +		break;
> +	default:
> +		*xfer = ERR_PTR(-EINVAL);
> +		break;
> +	}
> +
> +	if (IS_ERR(*xfer)) {
> +		dev_err(cinfo->dev,
> +			"Failed to acquire a valid xfer for hdr:0x%X\n",
> +			*msg_hdr);
> +		return PTR_ERR(*xfer);
> +	}
> +
> +	/* Fix xfer->hdr.type with actual msg_hdr carried type */
> +	unpack_scmi_header(*msg_hdr, &((*xfer)->hdr));
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_transfer_release  - Release an previously acquired xfer
> + *
> + * @cinfo: A reference to the channel descriptor.
> + * @xfer: A reference to the xfer to release.
> + */
> +void scmi_transfer_release(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer)
> +{
> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> +	struct scmi_xfers_info *minfo;
> +
> +	if (!xfer || !info->desc->support_xfers_delegation)
> +		return;
> +
> +	if (xfer->hdr.type == MSG_TYPE_NOTIFICATION)
> +		minfo = &info->rx_minfo;
> +	else
> +		minfo = &info->tx_minfo;
> +
> +	__scmi_xfer_put(minfo, xfer);
> +}
> +
>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  {
>  	struct scmi_xfer *xfer;
> @@ -441,7 +559,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  	ktime_t ts;
>  
>  	ts = ktime_get_boottime();
> -	xfer = scmi_xfer_get(cinfo->handle, minfo);
> +
> +	if (!info->desc->support_xfers_delegation)
> +		xfer = scmi_xfer_get(cinfo->handle, minfo);
> +	else
> +		xfer = scmi_xfer_acquire(minfo, MSG_XTRACT_TOKEN(msg_hdr));
>  	if (IS_ERR(xfer)) {
>  		dev_err(dev, "failed to get free message slot (%ld)\n",
>  			PTR_ERR(xfer));
> @@ -449,8 +571,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  		return;
>  	}
>  
> -	unpack_scmi_header(msg_hdr, &xfer->hdr);
>  	scmi_dump_header_dbg(dev, &xfer->hdr);
> +
> +	if (!info->desc->support_xfers_delegation)
> +		unpack_scmi_header(msg_hdr, &xfer->hdr);
> +

Why dump the header before unpacking?

>  	info->desc->ops->fetch_notification(cinfo, info->desc->max_msg_size,
>  					    xfer);
>  	scmi_notify(cinfo->handle, xfer->hdr.protocol_id,
> @@ -496,7 +621,8 @@ static void scmi_handle_response(struct scmi_chan_info *cinfo,
>  			xfer_id);
>  		info->desc->ops->clear_channel(cinfo);
>  		/* It was unexpected, so nobody will clear the xfer if not us */
> -		__scmi_xfer_put(minfo, xfer);
> +		if (!info->desc->support_xfers_delegation) //XXX ??? Really
> +			__scmi_xfer_put(minfo, xfer);
>  		return;
>  	}
>  
> 




^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 05/16] firmware: arm_scmi: Introduce delegated xfers support
@ 2021-07-01  8:42     ` Peter Hilber
  0 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:42 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On 11.06.21 18:59, Cristian Marussi wrote:
> Introduce optional support for delegated xfers allocation.
> 
> An SCMI transport can optionally declare to support delegated xfers and
> then use a few helper functions exposed by the core SCMI transport layer to
> query the core for existing in-flight transfers matching a provided message
> header or alternatively and transparently obtain a brand new xfer to handle
> a freshly received notification message.
> In both cases the obtained xfer is uniquely mapped into a specific xfer
> through the means of the message header acting as key.
> 
> In this way such a transport can properly store its own transport specific
> payload into the xfer uniquely associated to the message header before
> even calling into the core scmi_rx_callback() in the usual way, so that
> the transport specific message envelope structures can be freed early
> and there is no more need to keep track of their status till the core
> fully processes the xfer to completion or times out.
> 
> The scmi_rx_callbak() does not need to be modified to carry additional
> transport-specific ancillary data related to such message envelopes since
> an unique natural association is established between the xfer and the
> related message header.
> 
> Existing transports that do not need anything of the above will continue
> to work as before without any change.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/common.h |  14 +++
>  drivers/firmware/arm_scmi/driver.c | 132 ++++++++++++++++++++++++++++-
>  2 files changed, 143 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index e64c5ca9ee7c..0edc04bc434c 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -80,6 +80,7 @@ struct scmi_msg_resp_prot_version {
>   * @status: Status of the transfer once it's complete
>   * @poll_completion: Indicate if the transfer needs to be polled for
>   *	completion or interrupt mode is used
> + * @saved_hdr: A copy of the original msg_hdr
>   */
>  struct scmi_msg_hdr {
>  	u8 id;
> @@ -88,6 +89,7 @@ struct scmi_msg_hdr {
>  	u16 seq;
>  	u32 status;
>  	bool poll_completion;
> +	u32 saved_hdr;
>  };
>  
>  /**
> @@ -154,6 +156,9 @@ struct scmi_msg {
>   * @rx: Receive message, the buffer should be pre-allocated to store
>   *	message. If request-ACK protocol is used, we can reuse the same
>   *	buffer for the rx path as we use for the tx path.
> + * @rx_raw_len: A field which can be optionally used by a specific transport
> + *		to save transport specific message length
> + *		It is not used by the SCMI transport core
>   * @done: command message transmit completion event
>   * @async_done: pointer to delayed response message received event completion
>   * @users: A refcount to track the active users for this xfer
> @@ -165,6 +170,7 @@ struct scmi_xfer {
>  	struct scmi_msg_hdr hdr;
>  	struct scmi_msg tx;
>  	struct scmi_msg rx;
> +	size_t rx_raw_len;
>  	struct completion done;
>  	struct completion *async_done;
>  	refcount_t users;
> @@ -355,6 +361,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
>   * @max_msg: Maximum number of messages that can be pending
>   *	simultaneously in the system
>   * @max_msg_size: Maximum size of data per message that can be handled.
> + * @support_xfers_delegation: A flag to indicate if the described transport
> + *			      will handle delegated xfers, so the core can
> + *			      derive proper related assumptions.
>   */
>  struct scmi_desc {
>  	int (*init)(void);
> @@ -363,6 +372,7 @@ struct scmi_desc {
>  	int max_rx_timeout_ms;
>  	int max_msg;
>  	int max_msg_size;
> +	bool support_xfers_delegation;
>  };
>  
>  extern const struct scmi_desc scmi_mailbox_desc;
> @@ -391,4 +401,8 @@ void scmi_notification_instance_data_set(const struct scmi_handle *handle,
>  					 void *priv);
>  void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
>  
> +int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
> +			  struct scmi_xfer **xfer);
> +void scmi_transfer_release(struct scmi_chan_info *cinfo,
> +			   struct scmi_xfer *xfer);
>  #endif /* _SCMI_COMMON_H */
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index f0b20ddb24f4..371d3804cd79 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -432,6 +432,124 @@ scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
>  	return xfer ?: ERR_PTR(-EINVAL);
>  }
>  
> +/**
> + * scmi_xfer_acquire  -  Helper to lookup and acquire an xfer
> + *
> + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> + * @xfer_id: Token ID to lookup in @pending_xfers
> + *
> + * When a valid xfer is found for the provided @xfer_id, reference counting is
> + * properly updated.
> + *
> + * Return: A valid @xfer on Success or error otherwise.
> + */
> +static struct scmi_xfer *
> +scmi_xfer_acquire(struct scmi_xfers_info *minfo, u16 xfer_id)
> +{
> +	unsigned long flags;
> +	struct scmi_xfer *xfer;
> +
> +	spin_lock_irqsave(&minfo->xfer_lock, flags);
> +	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
> +	if (!IS_ERR(xfer))
> +		refcount_inc(&xfer->users);
> +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> +
> +	return xfer;
> +}
> +
> +/**
> + * scmi_transfer_acquire  -  Lookup for an existing xfer or freshly allocate a
> + * new one depending on the type of the message
> + *
> + * @cinfo: A reference to the channel descriptor.
> + * @msg_hdr: A pointer to the message header to lookup.
> + * @xfer: A reference to the pre-existent or freshly allocated xfer
> + *	  associated with the provided *msg_hdr.
> + *
> + * This function can be used by transports supporting delegated xfers to obtain
> + * a valid @xfer associated with the provided @msg_hdr param.
> + *
> + * The nature of the resulting @xfer depends on the type of message specified in
> + * @msg_hdr:
> + *  - for responses and delayed responses a pre-existent/pre-allocated in-flight
> + *    xfer descriptor will be returned (properly refcounted)
> + *  - for notifications a brand new xfer will be allocated; in this case the
> + *    provided message header sequence number will also be mangled to match
> + *    the token in the freshly allocated xfer: this is needed to establish a
> + *    link between the picked xfer and the msg_hdr that will be subsequently
> + *    passed back via usual scmi_rx_callback().
> + *
> + * Return: 0 if a valid xfer is returned in @xfer, error otherwise.
> + */
> +int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
> +			  struct scmi_xfer **xfer)
> +{
> +	u8 msg_type;
> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> +
> +	if (!xfer || !msg_hdr || !info->desc->support_xfers_delegation)
> +		return -EINVAL;
> +
> +	msg_type = MSG_XTRACT_TYPE(*msg_hdr);
> +	switch (msg_type) {
> +	case MSG_TYPE_COMMAND:
> +	case MSG_TYPE_DELAYED_RESP:
> +		/* Grab an existing xfer for xfer_id */
> +		*xfer = scmi_xfer_acquire(&info->tx_minfo,
> +					  MSG_XTRACT_TOKEN(*msg_hdr));
> +		break;
> +	case MSG_TYPE_NOTIFICATION:
> +		/* Get a brand new RX xfer */
> +		*xfer = scmi_xfer_get(cinfo->handle, &info->rx_minfo);
> +		if (!IS_ERR(*xfer)) {
> +			/* Save original msg_hdr and fix sequence number */
> +			(*xfer)->hdr.saved_hdr = *msg_hdr;

The saved header isn't used anywhere.

> +			*msg_hdr &= ~MSG_TOKEN_ID_MASK;
> +			*msg_hdr |= FIELD_PREP(MSG_TOKEN_ID_MASK,
> +					       (*xfer)->hdr.seq);

This will invalidate the token set by the platform in 
scmi_dump_header_dbg(). Maybe it would have been more elegant to 
introduce a dedicated hash table key field?

> +		}
> +		break;
> +	default:
> +		*xfer = ERR_PTR(-EINVAL);
> +		break;
> +	}
> +
> +	if (IS_ERR(*xfer)) {
> +		dev_err(cinfo->dev,
> +			"Failed to acquire a valid xfer for hdr:0x%X\n",
> +			*msg_hdr);
> +		return PTR_ERR(*xfer);
> +	}
> +
> +	/* Fix xfer->hdr.type with actual msg_hdr carried type */
> +	unpack_scmi_header(*msg_hdr, &((*xfer)->hdr));
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_transfer_release  - Release an previously acquired xfer
> + *
> + * @cinfo: A reference to the channel descriptor.
> + * @xfer: A reference to the xfer to release.
> + */
> +void scmi_transfer_release(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer)
> +{
> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> +	struct scmi_xfers_info *minfo;
> +
> +	if (!xfer || !info->desc->support_xfers_delegation)
> +		return;
> +
> +	if (xfer->hdr.type == MSG_TYPE_NOTIFICATION)
> +		minfo = &info->rx_minfo;
> +	else
> +		minfo = &info->tx_minfo;
> +
> +	__scmi_xfer_put(minfo, xfer);
> +}
> +
>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  {
>  	struct scmi_xfer *xfer;
> @@ -441,7 +559,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  	ktime_t ts;
>  
>  	ts = ktime_get_boottime();
> -	xfer = scmi_xfer_get(cinfo->handle, minfo);
> +
> +	if (!info->desc->support_xfers_delegation)
> +		xfer = scmi_xfer_get(cinfo->handle, minfo);
> +	else
> +		xfer = scmi_xfer_acquire(minfo, MSG_XTRACT_TOKEN(msg_hdr));
>  	if (IS_ERR(xfer)) {
>  		dev_err(dev, "failed to get free message slot (%ld)\n",
>  			PTR_ERR(xfer));
> @@ -449,8 +571,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  		return;
>  	}
>  
> -	unpack_scmi_header(msg_hdr, &xfer->hdr);
>  	scmi_dump_header_dbg(dev, &xfer->hdr);
> +
> +	if (!info->desc->support_xfers_delegation)
> +		unpack_scmi_header(msg_hdr, &xfer->hdr);
> +

Why dump the header before unpacking?

>  	info->desc->ops->fetch_notification(cinfo, info->desc->max_msg_size,
>  					    xfer);
>  	scmi_notify(cinfo->handle, xfer->hdr.protocol_id,
> @@ -496,7 +621,8 @@ static void scmi_handle_response(struct scmi_chan_info *cinfo,
>  			xfer_id);
>  		info->desc->ops->clear_channel(cinfo);
>  		/* It was unexpected, so nobody will clear the xfer if not us */
> -		__scmi_xfer_put(minfo, xfer);
> +		if (!info->desc->support_xfers_delegation) //XXX ??? Really
> +			__scmi_xfer_put(minfo, xfer);
>  		return;
>  	}
>  
> 




_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 06/16] firmware: arm_scmi, smccc, mailbox: Make shmem based transports optional
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-07-01  8:42     ` Peter Hilber
  -1 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:42 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On 11.06.21 18:59, Cristian Marussi wrote:
> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> Upon adding the virtio transport in this patch series, SCMI will also
> work without shared memory based transports. Also, the mailbox transport
> may not be needed if the smc transport is used.
> 
> - Compile shmem.c only if a shmem based transport is available.
> 
> - Remove hard dependency of SCMI on mailbox.

The hard dependency has now already been removed with

   c05b07963e96 ("firmware: arm_scmi: Add SMCCC discovery dependency in")

> 
> [ Peter: Adapted patch for submission to upstream. ]
> 
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> ---
>  drivers/firmware/Kconfig           | 8 +++++++-
>  drivers/firmware/arm_scmi/Makefile | 2 +-
>  drivers/firmware/arm_scmi/common.h | 2 ++
>  drivers/firmware/smccc/Kconfig     | 1 +
>  drivers/mailbox/Kconfig            | 1 +
>  5 files changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
> index 1db738d5b301..358f895847b5 100644
> --- a/drivers/firmware/Kconfig
> +++ b/drivers/firmware/Kconfig
> @@ -9,7 +9,7 @@ menu "Firmware Drivers"
>  config ARM_SCMI_PROTOCOL
>  	tristate "ARM System Control and Management Interface (SCMI) Message Protocol"
>  	depends on ARM || ARM64 || COMPILE_TEST
> -	depends on MAILBOX || HAVE_ARM_SMCCC_DISCOVERY
> +	depends on ARM_SCMI_HAVE_SHMEM
>  	help
>  	  ARM System Control and Management Interface (SCMI) protocol is a
>  	  set of operating system-independent software interfaces that are
> @@ -27,6 +27,12 @@ config ARM_SCMI_PROTOCOL
>  	  This protocol library provides interface for all the client drivers
>  	  making use of the features offered by the SCMI.
>  
> +config ARM_SCMI_HAVE_SHMEM
> +	bool
> +	help
> +	  This declares whether a shared memory based transport for SCMI is
> +	  available.
> +
>  config ARM_SCMI_POWER_DOMAIN
>  	tristate "SCMI power domain driver"
>  	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
> diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
> index 6a2ef63306d6..5a2d4c32e0ae 100644
> --- a/drivers/firmware/arm_scmi/Makefile
> +++ b/drivers/firmware/arm_scmi/Makefile
> @@ -1,7 +1,7 @@
>  # SPDX-License-Identifier: GPL-2.0-only
>  scmi-bus-y = bus.o
>  scmi-driver-y = driver.o notify.o
> -scmi-transport-y = shmem.o
> +scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
>  scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
>  scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
>  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index 0edc04bc434c..4666777019fa 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -375,7 +375,9 @@ struct scmi_desc {
>  	bool support_xfers_delegation;
>  };
>  
> +#ifdef CONFIG_MAILBOX
>  extern const struct scmi_desc scmi_mailbox_desc;
> +#endif
>  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
>  extern const struct scmi_desc scmi_smc_desc;
>  #endif
> diff --git a/drivers/firmware/smccc/Kconfig b/drivers/firmware/smccc/Kconfig
> index 15e7466179a6..69c4d6cabf62 100644
> --- a/drivers/firmware/smccc/Kconfig
> +++ b/drivers/firmware/smccc/Kconfig
> @@ -9,6 +9,7 @@ config HAVE_ARM_SMCCC_DISCOVERY
>  	bool
>  	depends on ARM_PSCI_FW
>  	default y
> +	select ARM_SCMI_HAVE_SHMEM
>  	help
>  	 SMCCC v1.0 lacked discoverability and hence PSCI v1.0 was updated
>  	 to add SMCCC discovery mechanism though the PSCI firmware
> diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
> index 68de2c6af727..fc02c38c0739 100644
> --- a/drivers/mailbox/Kconfig
> +++ b/drivers/mailbox/Kconfig
> @@ -1,6 +1,7 @@
>  # SPDX-License-Identifier: GPL-2.0-only
>  menuconfig MAILBOX
>  	bool "Mailbox Hardware Support"
> +	select ARM_SCMI_HAVE_SHMEM
>  	help
>  	  Mailbox is a framework to control hardware communication between
>  	  on-chip processors through queued messages and interrupt driven
> 




^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 06/16] firmware: arm_scmi, smccc, mailbox: Make shmem based transports optional
@ 2021-07-01  8:42     ` Peter Hilber
  0 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:42 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On 11.06.21 18:59, Cristian Marussi wrote:
> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> Upon adding the virtio transport in this patch series, SCMI will also
> work without shared memory based transports. Also, the mailbox transport
> may not be needed if the smc transport is used.
> 
> - Compile shmem.c only if a shmem based transport is available.
> 
> - Remove hard dependency of SCMI on mailbox.

The hard dependency has now already been removed with

   c05b07963e96 ("firmware: arm_scmi: Add SMCCC discovery dependency in")

> 
> [ Peter: Adapted patch for submission to upstream. ]
> 
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> ---
>  drivers/firmware/Kconfig           | 8 +++++++-
>  drivers/firmware/arm_scmi/Makefile | 2 +-
>  drivers/firmware/arm_scmi/common.h | 2 ++
>  drivers/firmware/smccc/Kconfig     | 1 +
>  drivers/mailbox/Kconfig            | 1 +
>  5 files changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig
> index 1db738d5b301..358f895847b5 100644
> --- a/drivers/firmware/Kconfig
> +++ b/drivers/firmware/Kconfig
> @@ -9,7 +9,7 @@ menu "Firmware Drivers"
>  config ARM_SCMI_PROTOCOL
>  	tristate "ARM System Control and Management Interface (SCMI) Message Protocol"
>  	depends on ARM || ARM64 || COMPILE_TEST
> -	depends on MAILBOX || HAVE_ARM_SMCCC_DISCOVERY
> +	depends on ARM_SCMI_HAVE_SHMEM
>  	help
>  	  ARM System Control and Management Interface (SCMI) protocol is a
>  	  set of operating system-independent software interfaces that are
> @@ -27,6 +27,12 @@ config ARM_SCMI_PROTOCOL
>  	  This protocol library provides interface for all the client drivers
>  	  making use of the features offered by the SCMI.
>  
> +config ARM_SCMI_HAVE_SHMEM
> +	bool
> +	help
> +	  This declares whether a shared memory based transport for SCMI is
> +	  available.
> +
>  config ARM_SCMI_POWER_DOMAIN
>  	tristate "SCMI power domain driver"
>  	depends on ARM_SCMI_PROTOCOL || (COMPILE_TEST && OF)
> diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
> index 6a2ef63306d6..5a2d4c32e0ae 100644
> --- a/drivers/firmware/arm_scmi/Makefile
> +++ b/drivers/firmware/arm_scmi/Makefile
> @@ -1,7 +1,7 @@
>  # SPDX-License-Identifier: GPL-2.0-only
>  scmi-bus-y = bus.o
>  scmi-driver-y = driver.o notify.o
> -scmi-transport-y = shmem.o
> +scmi-transport-$(CONFIG_ARM_SCMI_HAVE_SHMEM) = shmem.o
>  scmi-transport-$(CONFIG_MAILBOX) += mailbox.o
>  scmi-transport-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smc.o
>  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o system.o voltage.o
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index 0edc04bc434c..4666777019fa 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -375,7 +375,9 @@ struct scmi_desc {
>  	bool support_xfers_delegation;
>  };
>  
> +#ifdef CONFIG_MAILBOX
>  extern const struct scmi_desc scmi_mailbox_desc;
> +#endif
>  #ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY
>  extern const struct scmi_desc scmi_smc_desc;
>  #endif
> diff --git a/drivers/firmware/smccc/Kconfig b/drivers/firmware/smccc/Kconfig
> index 15e7466179a6..69c4d6cabf62 100644
> --- a/drivers/firmware/smccc/Kconfig
> +++ b/drivers/firmware/smccc/Kconfig
> @@ -9,6 +9,7 @@ config HAVE_ARM_SMCCC_DISCOVERY
>  	bool
>  	depends on ARM_PSCI_FW
>  	default y
> +	select ARM_SCMI_HAVE_SHMEM
>  	help
>  	 SMCCC v1.0 lacked discoverability and hence PSCI v1.0 was updated
>  	 to add SMCCC discovery mechanism though the PSCI firmware
> diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
> index 68de2c6af727..fc02c38c0739 100644
> --- a/drivers/mailbox/Kconfig
> +++ b/drivers/mailbox/Kconfig
> @@ -1,6 +1,7 @@
>  # SPDX-License-Identifier: GPL-2.0-only
>  menuconfig MAILBOX
>  	bool "Mailbox Hardware Support"
> +	select ARM_SCMI_HAVE_SHMEM
>  	help
>  	  Mailbox is a framework to control hardware communication between
>  	  on-chip processors through queued messages and interrupt driven
> 




_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-07-01  8:43     ` Peter Hilber
  -1 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:43 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy,
	Rob Herring, devicetree

On 11.06.21 18:59, Cristian Marussi wrote:
> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> Document the properties for arm,scmi-virtio compatible nodes.
> The backing virtio SCMI device is described in patch [1].
> 
> While doing that, make shmem property required only for pre-existing
> mailbox and smc transports, since virtio-scmi does not need it.
> 
> [1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html
> 
> CC: Rob Herring <robh+dt@kernel.org>
> CC: devicetree@vger.kernel.org
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> [ Peter: Adapted patch for submission to upstream. ]
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> [ Cristian: converted to yaml format, moved shmen required property. ]
> Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
> v3 --> V4
> - convertd to YAML
> - make shmem required only for pre-existing mailbox and smc transport
> - updated VirtIO specification patch message reference
> - dropped virtio-mmio SCMI device example since really not pertinent to
>    virtio-scmi dt bindings transport: it is not even referenced in SCMI
>    virtio DT node since they are enumerated by VirtIO subsystem and there
>    could be PCI based SCMI devices anyway.
> ---
>   Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
>   1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> index cebf6ffe70d5..5c4c6782e052 100644
> --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> @@ -34,6 +34,10 @@ properties:
>         - description: SCMI compliant firmware with ARM SMC/HVC transport
>           items:
>             - const: arm,scmi-smc
> +      - description: SCMI compliant firmware with SCMI Virtio transport.
> +                     The virtio transport only supports a single device.
> +        items:
> +          - const: arm,scmi-virtio
>   
>     interrupts:
>       description:
> @@ -172,6 +176,7 @@ patternProperties:
>         Each sub-node represents a protocol supported. If the platform
>         supports a dedicated communication channel for a particular protocol,
>         then the corresponding transport properties must be present.
> +      The virtio transport does not support a dedicated communication channel.
>   
>       properties:
>         reg:
> @@ -195,7 +200,6 @@ patternProperties:
>   
>   required:
>     - compatible
> -  - shmem
>   
>   if:
>     properties:
> @@ -209,6 +213,7 @@ then:
>   
>     required:
>       - mboxes
> +    - shmem
>   
>   else:
>     if:
> @@ -219,6 +224,7 @@ else:
>     then:
>       required:
>         - arm,smc-id
> +      - shmem
>   
>   examples:
>     - |
> 

Maybe a minimal example for arm,scmi-virtio could be added, such as below:

diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml 
b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
index 5c4c6782e052..576faf970c1b 100644
--- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
+++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
@@ -344,4 +344,19 @@ examples:
          };
      };

+  - |
+    firmware {
+        scmi {
+            compatible = "arm,scmi-virtio";
+
+            #address-cells = <1>;
+            #size-cells = <0>;
+
+            scmi_devpd2: protocol@11 {
+                reg = <0x11>;
+                #power-domain-cells = <1>;
+            };
+        };
+    };
+
  ...

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
@ 2021-07-01  8:43     ` Peter Hilber
  0 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:43 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy,
	Rob Herring, devicetree

On 11.06.21 18:59, Cristian Marussi wrote:
> From: Igor Skalkin <igor.skalkin@opensynergy.com>
> 
> Document the properties for arm,scmi-virtio compatible nodes.
> The backing virtio SCMI device is described in patch [1].
> 
> While doing that, make shmem property required only for pre-existing
> mailbox and smc transports, since virtio-scmi does not need it.
> 
> [1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html
> 
> CC: Rob Herring <robh+dt@kernel.org>
> CC: devicetree@vger.kernel.org
> Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> [ Peter: Adapted patch for submission to upstream. ]
> Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> [ Cristian: converted to yaml format, moved shmen required property. ]
> Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
> v3 --> V4
> - convertd to YAML
> - make shmem required only for pre-existing mailbox and smc transport
> - updated VirtIO specification patch message reference
> - dropped virtio-mmio SCMI device example since really not pertinent to
>    virtio-scmi dt bindings transport: it is not even referenced in SCMI
>    virtio DT node since they are enumerated by VirtIO subsystem and there
>    could be PCI based SCMI devices anyway.
> ---
>   Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
>   1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> index cebf6ffe70d5..5c4c6782e052 100644
> --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> @@ -34,6 +34,10 @@ properties:
>         - description: SCMI compliant firmware with ARM SMC/HVC transport
>           items:
>             - const: arm,scmi-smc
> +      - description: SCMI compliant firmware with SCMI Virtio transport.
> +                     The virtio transport only supports a single device.
> +        items:
> +          - const: arm,scmi-virtio
>   
>     interrupts:
>       description:
> @@ -172,6 +176,7 @@ patternProperties:
>         Each sub-node represents a protocol supported. If the platform
>         supports a dedicated communication channel for a particular protocol,
>         then the corresponding transport properties must be present.
> +      The virtio transport does not support a dedicated communication channel.
>   
>       properties:
>         reg:
> @@ -195,7 +200,6 @@ patternProperties:
>   
>   required:
>     - compatible
> -  - shmem
>   
>   if:
>     properties:
> @@ -209,6 +213,7 @@ then:
>   
>     required:
>       - mboxes
> +    - shmem
>   
>   else:
>     if:
> @@ -219,6 +224,7 @@ else:
>     then:
>       required:
>         - arm,smc-id
> +      - shmem
>   
>   examples:
>     - |
> 

Maybe a minimal example for arm,scmi-virtio could be added, such as below:

diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml 
b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
index 5c4c6782e052..576faf970c1b 100644
--- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
+++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
@@ -344,4 +344,19 @@ examples:
          };
      };

+  - |
+    firmware {
+        scmi {
+            compatible = "arm,scmi-virtio";
+
+            #address-cells = <1>;
+            #size-cells = <0>;
+
+            scmi_devpd2: protocol@11 {
+                reg = <0x11>;
+                #power-domain-cells = <1>;
+            };
+        };
+    };
+
  ...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-07-01  8:43     ` Peter Hilber
  -1 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:43 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On 11.06.21 18:59, Cristian Marussi wrote:

<snip>

> +static struct virtio_driver virtio_scmi_driver = {
> +	.driver.name = "scmi-virtio",
> +	.driver.owner = THIS_MODULE,
> +	.feature_table = features,
> +	.feature_table_size = ARRAY_SIZE(features),
> +	.id_table = id_table,
> +	.probe = scmi_vio_probe,
> +	.remove = scmi_vio_remove,
> +};
> +

It might be good to also check for the VIRTIO_F_VERSION_1 feature bit in 
the optional .validate op (not yet implemented above), as some other 
devices do (quoting virtio-snd in the following):

> /**
>  * virtsnd_validate() - Validate if the device can be started.
>  * @vdev: VirtIO parent device.
>  *
>  * Context: Any context.
>  * Return: 0 on success, -EINVAL on failure.
>  */
> static int virtsnd_validate(struct virtio_device *vdev)
> {

<snip>

> 
> 	if (!virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
> 		dev_err(&vdev->dev,
> 			"device does not comply with spec version 1.x\n");
> 		return -EINVAL;
> 	}
> 

<snip>

> 
> static struct virtio_driver virtsnd_driver = {
> 	.driver.name = KBUILD_MODNAME,
> 	.driver.owner = THIS_MODULE,
> 	.id_table = id_table,
> 	.validate = virtsnd_validate,

(end of virtio-snd quote)

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
@ 2021-07-01  8:43     ` Peter Hilber
  0 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:43 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On 11.06.21 18:59, Cristian Marussi wrote:

<snip>

> +static struct virtio_driver virtio_scmi_driver = {
> +	.driver.name = "scmi-virtio",
> +	.driver.owner = THIS_MODULE,
> +	.feature_table = features,
> +	.feature_table_size = ARRAY_SIZE(features),
> +	.id_table = id_table,
> +	.probe = scmi_vio_probe,
> +	.remove = scmi_vio_remove,
> +};
> +

It might be good to also check for the VIRTIO_F_VERSION_1 feature bit in 
the optional .validate op (not yet implemented above), as some other 
devices do (quoting virtio-snd in the following):

> /**
>  * virtsnd_validate() - Validate if the device can be started.
>  * @vdev: VirtIO parent device.
>  *
>  * Context: Any context.
>  * Return: 0 on success, -EINVAL on failure.
>  */
> static int virtsnd_validate(struct virtio_device *vdev)
> {

<snip>

> 
> 	if (!virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
> 		dev_err(&vdev->dev,
> 			"device does not comply with spec version 1.x\n");
> 		return -EINVAL;
> 	}
> 

<snip>

> 
> static struct virtio_driver virtsnd_driver = {
> 	.driver.name = KBUILD_MODNAME,
> 	.driver.owner = THIS_MODULE,
> 	.id_table = id_table,
> 	.validate = virtsnd_validate,

(end of virtio-snd quote)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 15/16] [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers
  2021-06-11 16:59   ` Cristian Marussi
@ 2021-07-01  8:43     ` Peter Hilber
  -1 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:43 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On 11.06.21 18:59, Cristian Marussi wrote:
> Draft changes to virtio-scmi to use new support for core delegated xfers
> in an attempt to simplify the interactions between virtio-scmi transport
> and the SCMI core transport layer.
> 

These changes seem to make xfers delegation mandatory for 
message-passing transports, so that might be documented.

> TODO:
>   - Polling is still not supported.
>   - Probe/remove sequence still to be reviewed.
>   - Concurrent or inverted reception of related responses and delayed
>     responses is still not addressed.
>     (it will be addressed in the SCMI core anyway most probably)
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>   drivers/firmware/arm_scmi/common.h |   5 +
>   drivers/firmware/arm_scmi/msg.c    |  35 +++++
>   drivers/firmware/arm_scmi/virtio.c | 212 +++++++++++++++--------------
>   3 files changed, 152 insertions(+), 100 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index c143a449d278..22e5532fc698 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -428,6 +428,11 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
>   void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
>   			    size_t max_len, struct scmi_xfer *xfer);
>   
> +void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
> +			   size_t max_len, struct scmi_xfer *xfer);
> +void msg_fetch_raw_response(struct scmi_xfer *xfer);
> +void msg_fetch_raw_notification(struct scmi_xfer *xfer);
> +
>   void scmi_notification_instance_data_set(const struct scmi_handle *handle,
>   					 void *priv);
>   void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
> diff --git a/drivers/firmware/arm_scmi/msg.c b/drivers/firmware/arm_scmi/msg.c
> index 8a2d3303d281..3ed3ad0961ae 100644
> --- a/drivers/firmware/arm_scmi/msg.c
> +++ b/drivers/firmware/arm_scmi/msg.c
> @@ -74,6 +74,17 @@ u32 msg_read_header(struct scmi_msg_payld *msg)
>   	return le32_to_cpu(msg->msg_header);
>   }
>   
> +void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
> +			   size_t max_len, struct scmi_xfer *xfer)
> +{
> +	xfer->rx_raw_len = min_t(size_t, max_len,
> +				 msg_len >= sizeof(*msg) ?
> +				 msg_len - sizeof(*msg) : 0);
> +
> +	/* Take a copy to the rx buffer.. */
> +	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx_raw_len);

In the usage throughout arm-scmi so far, scmi_desc.max_msg_size seems to 
consistently refer to the payload excluding the return status. 
scmi_desc.max_msg_size is the actual max_len parameter to this function.

So the logic here would reduce the maximum payload length for responses 
to (scmi_desc.max_msg_size - sizeof(return status)).

> +}
> +
>   /**
>    * msg_fetch_response() - Fetch response SCMI payload from transport SDU.
>    *
> @@ -94,6 +105,25 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
>   	memcpy(xfer->rx.buf, &msg->msg_payload[1], xfer->rx.len);
>   }
>   
> +void msg_fetch_raw_response(struct scmi_xfer *xfer)
> +{
> +	__le32 *msg_payload = xfer->rx.buf;
> +
> +	if (xfer->rx_raw_len < sizeof(xfer->hdr.status)) {
> +		xfer->rx.len = 0;
> +		return;
> +	}
> +
> +	xfer->hdr.status = le32_to_cpu(msg_payload[0]);
> +	/*
> +	 * rx.len has been already pre-calculated by fetch_raw
> +	 * for the whole payload including status, so shrink it
> +	 */
> +	xfer->rx.len = xfer->rx_raw_len - sizeof(xfer->hdr.status);
> +	/* Carveout status 4-byte field */
> +	memmove(xfer->rx.buf, &msg_payload[1], xfer->rx.len);

Wouldn't it be feasible to align properly in msg_fetch_raw_payload() 
already? That could also get rid of .rx_raw_len.

> +}
> +
>   /**
>    * msg_fetch_notification() - Fetch notification payload from transport SDU.
>    *
> @@ -111,3 +141,8 @@ void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
>   	/* Take a copy to the rx buffer.. */
>   	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx.len);
>   }
> +
> +void msg_fetch_raw_notification(struct scmi_xfer *xfer)
> +{
> +	xfer->rx.len = xfer->rx_raw_len;
> +}
> diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
> index 20972adf6dc7..4412bc590ca7 100644
> --- a/drivers/firmware/arm_scmi/virtio.c
> +++ b/drivers/firmware/arm_scmi/virtio.c
> @@ -37,23 +37,24 @@
>   /**
>    * struct scmi_vio_channel - Transport channel information
>    *
> - * @lock: Protects access to all members except ready.
> - * @ready_lock: Protects access to ready. If required, it must be taken before
> - *              lock.
>    * @vqueue: Associated virtqueue
>    * @cinfo: SCMI Tx or Rx channel
>    * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
>    * @is_rx: Whether channel is an Rx channel
>    * @ready: Whether transport user is ready to hear about channel
> + * @lock: Protects access to all members except ready.
> + * @ready_lock: Protects access to ready. If required, it must be taken before
> + *              lock.
>    */
>   struct scmi_vio_channel {
> -	spinlock_t lock;
> -	spinlock_t ready_lock;
>   	struct virtqueue *vqueue;
>   	struct scmi_chan_info *cinfo;
>   	struct list_head free_list;
> -	u8 is_rx;
> -	u8 ready;
> +	bool is_rx;
> +	bool ready;
> +	unsigned int max_msg;
> +	spinlock_t lock;
> +	spinlock_t ready_lock;
>   };
>   
>   /**
> @@ -100,6 +101,73 @@ static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
>   	return rc;
>   }
>   
> +static void scmi_finalize_message(struct scmi_vio_channel *vioch,
> +				  struct scmi_vio_msg *msg)
> +{
> +	unsigned long flags;
> +
> +	if (vioch->is_rx) {
> +		scmi_vio_feed_vq_rx(vioch, msg);
> +	} else {
> +		spin_lock_irqsave(&vioch->lock, flags);
> +		list_add(&msg->list, &vioch->free_list);
> +		spin_unlock_irqrestore(&vioch->lock, flags);
> +	}
> +}
> +
> +static void scmi_process_vqueue_input(struct scmi_vio_channel *vioch,
> +				      struct scmi_vio_msg *msg)
> +{
> +	u32 msg_hdr;
> +	int ret;
> +	struct scmi_xfer *xfer = NULL;
> +
> +	msg_hdr = msg_read_header(msg->input);
> +	/*
> +	 * Acquire from the core transport layer a currently valid xfer
> +	 * descriptor associated to the received msg_hdr: this could be a
> +	 * previously allocated xfer for responses and delayed responses to
> +	 * in-flight commands, or a freshly allocated new xfer for a just
> +	 * received notification.
> +	 *
> +	 * In case of responses and delayed_responses the acquired xfer, at
> +	 * the time scmi_transfer_acquire() succcessfully returns is guaranteed
> +	 * to be still associated with a valid (not timed-out nor stale)
> +	 * descriptor and proper refcounting is kept in the core along this xfer
> +	 * so that should the core time out the xfer concurrently to this receive
> +	 * path the xfer will be properly deallocated only once the last user is
> +	 * done with it. (and this code path will terminate normally even though
> +	 * all the processing related to the timed out xfer will be discarded).
> +	 */

This comment would better fit to scmi_transfer_acquire().

> +	ret = scmi_transfer_acquire(vioch->cinfo, &msg_hdr, &xfer);
> +	if (ret) {
> +		dev_err(vioch->cinfo->dev,
> +			"Cannot find matching xfer for hdr:0x%X\n", msg_hdr);
> +		scmi_finalize_message(vioch, msg);
> +		return;
> +	}
> +
> +	dev_dbg(vioch->cinfo->dev,
> +		"VQUEUE[%d] - INPUT MSG_RX_LEN:%d - HDR:0x%X  TYPE:%d  XFER_ID:%d  XFER:%px\n",
> +		vioch->vqueue->index, msg->rx_len, msg_hdr, xfer->hdr.type,
> +		xfer->hdr.seq, xfer);
> +
> +	msg_fetch_raw_payload(msg->input, msg->rx_len,
> +			      scmi_virtio_desc.max_msg_size, xfer); > +
> +	/* Drop processed virtio message anyway */
> +	scmi_finalize_message(vioch, msg);
> +
> +	if (vioch->is_rx || !xfer->hdr.poll_completion)
> +		scmi_rx_callback(vioch->cinfo, msg_hdr);
> +	else
> +		dev_warn(vioch->cinfo->dev,
> +			 "Polling mode NOT supported. Dropped hdr:0X%X\n",
> +			 msg_hdr);
> +
> +	scmi_transfer_release(vioch->cinfo, xfer);
> +}
> +
>   static void scmi_vio_complete_cb(struct virtqueue *vqueue)
>   {
>   	unsigned long ready_flags;
> @@ -138,15 +206,9 @@ static void scmi_vio_complete_cb(struct virtqueue *vqueue)
>   
>   		if (msg) {
>   			msg->rx_len = length;
> -
> -			/*
> -			 * Hold the ready_lock during the callback to avoid
> -			 * races when the arm-scmi driver is unbinding while
> -			 * the virtio device is not quiesced yet.
> -			 */
> -			scmi_rx_callback(vioch->cinfo,
> -					 msg_read_header(msg->input), msg);
> +			scmi_process_vqueue_input(vioch, msg);
>   		}
> +
>   		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
>   	}
>   
> @@ -163,27 +225,11 @@ static vq_callback_t *scmi_vio_complete_callbacks[] = {
>   	scmi_vio_complete_cb
>   };
>   
> -static unsigned int virtio_get_max_msg(bool tx,
> -				       struct scmi_chan_info *base_cinfo)
> +static unsigned int virtio_get_max_msg(struct scmi_chan_info *base_cinfo)
>   {
>   	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
> -	unsigned int ret;
>   
> -	ret = virtqueue_get_vring_size(vioch->vqueue);
> -
> -	/* Tx messages need multiple descriptors. */
> -	if (tx)
> -		ret /= DESCRIPTORS_PER_TX_MSG;
> -
> -	if (ret > MSG_TOKEN_MAX) {
> -		dev_info_once(
> -			base_cinfo->dev,
> -			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
> -			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
> -		ret = MSG_TOKEN_MAX;
> -	}
> -
> -	return ret;
> +	return vioch->max_msg;
>   }
>   
>   static int scmi_vio_match_any_dev(struct device *dev, const void *data)
> @@ -195,13 +241,14 @@ static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
>   
>   static int virtio_link_supplier(struct device *dev)
>   {
> -	struct device *vdev = driver_find_device(
> -		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
> +	struct device *vdev;
> +
> +	vdev = driver_find_device(&virtio_scmi_driver.driver,
> +				  NULL, NULL, scmi_vio_match_any_dev);
>   
>   	if (!vdev) {
> -		dev_notice_once(
> -			dev,
> -			"Deferring probe after not finding a bound scmi-virtio device\n");
> +		dev_notice_once(dev,
> +				"Deferring probe after not finding a bound scmi-virtio device\n");
>   		return -EPROBE_DEFER;
>   	}
>   
> @@ -245,12 +292,8 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>   	struct virtio_device *vdev;
>   	struct scmi_vio_channel *vioch;
>   	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
> -	int max_msg;
>   	int i;
>   
> -	if (!virtio_chan_available(dev, index))
> -		return -ENODEV;
> -
>   	vdev = scmi_get_transport_info(dev);
>   	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
>   
> @@ -259,9 +302,7 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>   	vioch->cinfo = cinfo;
>   	spin_unlock_irqrestore(&vioch->lock, flags);
>   
> -	max_msg = virtio_get_max_msg(tx, cinfo);
> -
> -	for (i = 0; i < max_msg; i++) {
> +	for (i = 0; i < vioch->max_msg; i++) {
>   		struct scmi_vio_msg *msg;
>   
>   		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
> @@ -322,13 +363,6 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
>   	int rc;
>   	struct scmi_vio_msg *msg;
>   
> -	/*
> -	 * TODO: For now, we don't support polling. But it should not be
> -	 * difficult to add support.
> -	 */
> -	if (xfer->hdr.poll_completion)
> -		return -EINVAL;
> -
>   	spin_lock_irqsave(&vioch->lock, flags);
>   
>   	if (list_empty(&vioch->free_list)) {
> @@ -351,6 +385,11 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
>   			     "%s() failed to add to virtqueue (%d)\n", __func__,
>   			     rc);
>   	} else {
> +		dev_dbg(vioch->cinfo->dev,
> +			"VQUEUE[%d] - REQUEST - PROTO:0x%X  ID:0x%X  XFER_ID:%d  XFER:%px  RX_LEN:%zd\n",
> +		 vioch->vqueue->index, xfer->hdr.protocol_id,
> +		 xfer->hdr.id, xfer->hdr.seq, xfer, xfer->rx.len);

Indentation appears to be inconsistent.

> +
>   		virtqueue_kick(vioch->vqueue);
>   	}
>   
> @@ -360,36 +399,15 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
>   }
>   
>   static void virtio_fetch_response(struct scmi_chan_info *cinfo,
> -				  struct scmi_xfer *xfer, void *msg_handle)
> +				  struct scmi_xfer *xfer)
>   {
> -	struct scmi_vio_msg *msg = msg_handle;
> -	struct scmi_vio_channel *vioch = cinfo->transport_info;
> -
> -	if (!msg) {
> -		dev_dbg_once(&vioch->vqueue->vdev->dev,
> -			     "Ignoring %s() call with NULL msg_handle\n",
> -			     __func__);
> -		return;
> -	}
> -
> -	msg_fetch_response(msg->input, msg->rx_len, xfer);
> +	msg_fetch_raw_response(xfer);
>   }
>   
>   static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
> -				      size_t max_len, struct scmi_xfer *xfer,
> -				      void *msg_handle)
> +				      size_t max_len, struct scmi_xfer *xfer)
>   {
> -	struct scmi_vio_msg *msg = msg_handle;
> -	struct scmi_vio_channel *vioch = cinfo->transport_info;
> -
> -	if (!msg) {
> -		dev_dbg_once(&vioch->vqueue->vdev->dev,
> -			     "Ignoring %s() call with NULL msg_handle\n",
> -			     __func__);
> -		return;
> -	}
> -
> -	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
> +	msg_fetch_raw_notification(xfer);
>   }
>   
>   static void dummy_clear_channel(struct scmi_chan_info *cinfo)
> @@ -402,28 +420,6 @@ static bool dummy_poll_done(struct scmi_chan_info *cinfo,
>   	return false;
>   }
>   
> -static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
> -{
> -	unsigned long flags;
> -	struct scmi_vio_channel *vioch = cinfo->transport_info;
> -	struct scmi_vio_msg *msg = msg_handle;
> -
> -	if (!msg) {
> -		dev_dbg_once(&vioch->vqueue->vdev->dev,
> -			     "Ignoring %s() call with NULL msg_handle\n",
> -			     __func__);
> -		return;
> -	}
> -
> -	if (vioch->is_rx) {
> -		scmi_vio_feed_vq_rx(vioch, msg);
> -	} else {
> -		spin_lock_irqsave(&vioch->lock, flags);
> -		list_add(&msg->list, &vioch->free_list);
> -		spin_unlock_irqrestore(&vioch->lock, flags);
> -	}
> -}
> -
>   static const struct scmi_transport_ops scmi_virtio_ops = {
>   	.link_supplier = virtio_link_supplier,
>   	.chan_available = virtio_chan_available,
> @@ -435,7 +431,6 @@ static const struct scmi_transport_ops scmi_virtio_ops = {
>   	.fetch_notification = virtio_fetch_notification,
>   	.clear_channel = dummy_clear_channel,
>   	.poll_done = dummy_poll_done,
> -	.drop_message = virtio_drop_message,
>   };
>   
>   static int scmi_vio_probe(struct virtio_device *vdev)
> @@ -467,10 +462,26 @@ static int scmi_vio_probe(struct virtio_device *vdev)
>   	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);
>   
>   	for (i = 0; i < vq_cnt; i++) {
> +		unsigned int sz;
> +
>   		spin_lock_init(&channels[i].lock);
>   		spin_lock_init(&channels[i].ready_lock);
>   		INIT_LIST_HEAD(&channels[i].free_list);
>   		channels[i].vqueue = vqs[i];
> +
> +		sz = virtqueue_get_vring_size(channels[i].vqueue);
> +		/* Tx messages need multiple descriptors. */
> +		if (!channels[i].is_rx)
> +			sz /= DESCRIPTORS_PER_TX_MSG;
> +
> +		if (sz > MSG_TOKEN_MAX) {
> +			dev_info_once(dev,
> +				      "%s virtqueue could hold %d messages. Only %ld allowed to be pending.\n",
> +				      channels[i].is_rx ? "rx" : "tx",
> +				      sz, MSG_TOKEN_MAX);
> +			sz = MSG_TOKEN_MAX;
> +		}
> +		channels[i].max_msg = sz;
>   	}
>   
>   	vdev->priv = channels;
> @@ -520,4 +531,5 @@ const struct scmi_desc scmi_virtio_desc = {
>   	.max_rx_timeout_ms = 60000, /* for non-realtime virtio devices */
>   	.max_msg = 0, /* overridden by virtio_get_max_msg() */
>   	.max_msg_size = VIRTIO_SCMI_MAX_MSG_SIZE,
> +	.support_xfers_delegation = true,
>   };
> 




^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 15/16] [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers
@ 2021-07-01  8:43     ` Peter Hilber
  0 siblings, 0 replies; 90+ messages in thread
From: Peter Hilber @ 2021-07-01  8:43 UTC (permalink / raw)
  To: Cristian Marussi, linux-kernel, linux-arm-kernel, virtualization,
	virtio-dev
  Cc: sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

On 11.06.21 18:59, Cristian Marussi wrote:
> Draft changes to virtio-scmi to use new support for core delegated xfers
> in an attempt to simplify the interactions between virtio-scmi transport
> and the SCMI core transport layer.
> 

These changes seem to make xfers delegation mandatory for 
message-passing transports, so that might be documented.

> TODO:
>   - Polling is still not supported.
>   - Probe/remove sequence still to be reviewed.
>   - Concurrent or inverted reception of related responses and delayed
>     responses is still not addressed.
>     (it will be addressed in the SCMI core anyway most probably)
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>   drivers/firmware/arm_scmi/common.h |   5 +
>   drivers/firmware/arm_scmi/msg.c    |  35 +++++
>   drivers/firmware/arm_scmi/virtio.c | 212 +++++++++++++++--------------
>   3 files changed, 152 insertions(+), 100 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index c143a449d278..22e5532fc698 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -428,6 +428,11 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
>   void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
>   			    size_t max_len, struct scmi_xfer *xfer);
>   
> +void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
> +			   size_t max_len, struct scmi_xfer *xfer);
> +void msg_fetch_raw_response(struct scmi_xfer *xfer);
> +void msg_fetch_raw_notification(struct scmi_xfer *xfer);
> +
>   void scmi_notification_instance_data_set(const struct scmi_handle *handle,
>   					 void *priv);
>   void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
> diff --git a/drivers/firmware/arm_scmi/msg.c b/drivers/firmware/arm_scmi/msg.c
> index 8a2d3303d281..3ed3ad0961ae 100644
> --- a/drivers/firmware/arm_scmi/msg.c
> +++ b/drivers/firmware/arm_scmi/msg.c
> @@ -74,6 +74,17 @@ u32 msg_read_header(struct scmi_msg_payld *msg)
>   	return le32_to_cpu(msg->msg_header);
>   }
>   
> +void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
> +			   size_t max_len, struct scmi_xfer *xfer)
> +{
> +	xfer->rx_raw_len = min_t(size_t, max_len,
> +				 msg_len >= sizeof(*msg) ?
> +				 msg_len - sizeof(*msg) : 0);
> +
> +	/* Take a copy to the rx buffer.. */
> +	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx_raw_len);

In the usage throughout arm-scmi so far, scmi_desc.max_msg_size seems to 
consistently refer to the payload excluding the return status. 
scmi_desc.max_msg_size is the actual max_len parameter to this function.

So the logic here would reduce the maximum payload length for responses 
to (scmi_desc.max_msg_size - sizeof(return status)).

> +}
> +
>   /**
>    * msg_fetch_response() - Fetch response SCMI payload from transport SDU.
>    *
> @@ -94,6 +105,25 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
>   	memcpy(xfer->rx.buf, &msg->msg_payload[1], xfer->rx.len);
>   }
>   
> +void msg_fetch_raw_response(struct scmi_xfer *xfer)
> +{
> +	__le32 *msg_payload = xfer->rx.buf;
> +
> +	if (xfer->rx_raw_len < sizeof(xfer->hdr.status)) {
> +		xfer->rx.len = 0;
> +		return;
> +	}
> +
> +	xfer->hdr.status = le32_to_cpu(msg_payload[0]);
> +	/*
> +	 * rx.len has been already pre-calculated by fetch_raw
> +	 * for the whole payload including status, so shrink it
> +	 */
> +	xfer->rx.len = xfer->rx_raw_len - sizeof(xfer->hdr.status);
> +	/* Carveout status 4-byte field */
> +	memmove(xfer->rx.buf, &msg_payload[1], xfer->rx.len);

Wouldn't it be feasible to align properly in msg_fetch_raw_payload() 
already? That could also get rid of .rx_raw_len.

> +}
> +
>   /**
>    * msg_fetch_notification() - Fetch notification payload from transport SDU.
>    *
> @@ -111,3 +141,8 @@ void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
>   	/* Take a copy to the rx buffer.. */
>   	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx.len);
>   }
> +
> +void msg_fetch_raw_notification(struct scmi_xfer *xfer)
> +{
> +	xfer->rx.len = xfer->rx_raw_len;
> +}
> diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
> index 20972adf6dc7..4412bc590ca7 100644
> --- a/drivers/firmware/arm_scmi/virtio.c
> +++ b/drivers/firmware/arm_scmi/virtio.c
> @@ -37,23 +37,24 @@
>   /**
>    * struct scmi_vio_channel - Transport channel information
>    *
> - * @lock: Protects access to all members except ready.
> - * @ready_lock: Protects access to ready. If required, it must be taken before
> - *              lock.
>    * @vqueue: Associated virtqueue
>    * @cinfo: SCMI Tx or Rx channel
>    * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
>    * @is_rx: Whether channel is an Rx channel
>    * @ready: Whether transport user is ready to hear about channel
> + * @lock: Protects access to all members except ready.
> + * @ready_lock: Protects access to ready. If required, it must be taken before
> + *              lock.
>    */
>   struct scmi_vio_channel {
> -	spinlock_t lock;
> -	spinlock_t ready_lock;
>   	struct virtqueue *vqueue;
>   	struct scmi_chan_info *cinfo;
>   	struct list_head free_list;
> -	u8 is_rx;
> -	u8 ready;
> +	bool is_rx;
> +	bool ready;
> +	unsigned int max_msg;
> +	spinlock_t lock;
> +	spinlock_t ready_lock;
>   };
>   
>   /**
> @@ -100,6 +101,73 @@ static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
>   	return rc;
>   }
>   
> +static void scmi_finalize_message(struct scmi_vio_channel *vioch,
> +				  struct scmi_vio_msg *msg)
> +{
> +	unsigned long flags;
> +
> +	if (vioch->is_rx) {
> +		scmi_vio_feed_vq_rx(vioch, msg);
> +	} else {
> +		spin_lock_irqsave(&vioch->lock, flags);
> +		list_add(&msg->list, &vioch->free_list);
> +		spin_unlock_irqrestore(&vioch->lock, flags);
> +	}
> +}
> +
> +static void scmi_process_vqueue_input(struct scmi_vio_channel *vioch,
> +				      struct scmi_vio_msg *msg)
> +{
> +	u32 msg_hdr;
> +	int ret;
> +	struct scmi_xfer *xfer = NULL;
> +
> +	msg_hdr = msg_read_header(msg->input);
> +	/*
> +	 * Acquire from the core transport layer a currently valid xfer
> +	 * descriptor associated to the received msg_hdr: this could be a
> +	 * previously allocated xfer for responses and delayed responses to
> +	 * in-flight commands, or a freshly allocated new xfer for a just
> +	 * received notification.
> +	 *
> +	 * In case of responses and delayed_responses the acquired xfer, at
> +	 * the time scmi_transfer_acquire() succcessfully returns is guaranteed
> +	 * to be still associated with a valid (not timed-out nor stale)
> +	 * descriptor and proper refcounting is kept in the core along this xfer
> +	 * so that should the core time out the xfer concurrently to this receive
> +	 * path the xfer will be properly deallocated only once the last user is
> +	 * done with it. (and this code path will terminate normally even though
> +	 * all the processing related to the timed out xfer will be discarded).
> +	 */

This comment would better fit to scmi_transfer_acquire().

> +	ret = scmi_transfer_acquire(vioch->cinfo, &msg_hdr, &xfer);
> +	if (ret) {
> +		dev_err(vioch->cinfo->dev,
> +			"Cannot find matching xfer for hdr:0x%X\n", msg_hdr);
> +		scmi_finalize_message(vioch, msg);
> +		return;
> +	}
> +
> +	dev_dbg(vioch->cinfo->dev,
> +		"VQUEUE[%d] - INPUT MSG_RX_LEN:%d - HDR:0x%X  TYPE:%d  XFER_ID:%d  XFER:%px\n",
> +		vioch->vqueue->index, msg->rx_len, msg_hdr, xfer->hdr.type,
> +		xfer->hdr.seq, xfer);
> +
> +	msg_fetch_raw_payload(msg->input, msg->rx_len,
> +			      scmi_virtio_desc.max_msg_size, xfer); > +
> +	/* Drop processed virtio message anyway */
> +	scmi_finalize_message(vioch, msg);
> +
> +	if (vioch->is_rx || !xfer->hdr.poll_completion)
> +		scmi_rx_callback(vioch->cinfo, msg_hdr);
> +	else
> +		dev_warn(vioch->cinfo->dev,
> +			 "Polling mode NOT supported. Dropped hdr:0X%X\n",
> +			 msg_hdr);
> +
> +	scmi_transfer_release(vioch->cinfo, xfer);
> +}
> +
>   static void scmi_vio_complete_cb(struct virtqueue *vqueue)
>   {
>   	unsigned long ready_flags;
> @@ -138,15 +206,9 @@ static void scmi_vio_complete_cb(struct virtqueue *vqueue)
>   
>   		if (msg) {
>   			msg->rx_len = length;
> -
> -			/*
> -			 * Hold the ready_lock during the callback to avoid
> -			 * races when the arm-scmi driver is unbinding while
> -			 * the virtio device is not quiesced yet.
> -			 */
> -			scmi_rx_callback(vioch->cinfo,
> -					 msg_read_header(msg->input), msg);
> +			scmi_process_vqueue_input(vioch, msg);
>   		}
> +
>   		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
>   	}
>   
> @@ -163,27 +225,11 @@ static vq_callback_t *scmi_vio_complete_callbacks[] = {
>   	scmi_vio_complete_cb
>   };
>   
> -static unsigned int virtio_get_max_msg(bool tx,
> -				       struct scmi_chan_info *base_cinfo)
> +static unsigned int virtio_get_max_msg(struct scmi_chan_info *base_cinfo)
>   {
>   	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
> -	unsigned int ret;
>   
> -	ret = virtqueue_get_vring_size(vioch->vqueue);
> -
> -	/* Tx messages need multiple descriptors. */
> -	if (tx)
> -		ret /= DESCRIPTORS_PER_TX_MSG;
> -
> -	if (ret > MSG_TOKEN_MAX) {
> -		dev_info_once(
> -			base_cinfo->dev,
> -			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
> -			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
> -		ret = MSG_TOKEN_MAX;
> -	}
> -
> -	return ret;
> +	return vioch->max_msg;
>   }
>   
>   static int scmi_vio_match_any_dev(struct device *dev, const void *data)
> @@ -195,13 +241,14 @@ static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
>   
>   static int virtio_link_supplier(struct device *dev)
>   {
> -	struct device *vdev = driver_find_device(
> -		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
> +	struct device *vdev;
> +
> +	vdev = driver_find_device(&virtio_scmi_driver.driver,
> +				  NULL, NULL, scmi_vio_match_any_dev);
>   
>   	if (!vdev) {
> -		dev_notice_once(
> -			dev,
> -			"Deferring probe after not finding a bound scmi-virtio device\n");
> +		dev_notice_once(dev,
> +				"Deferring probe after not finding a bound scmi-virtio device\n");
>   		return -EPROBE_DEFER;
>   	}
>   
> @@ -245,12 +292,8 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>   	struct virtio_device *vdev;
>   	struct scmi_vio_channel *vioch;
>   	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
> -	int max_msg;
>   	int i;
>   
> -	if (!virtio_chan_available(dev, index))
> -		return -ENODEV;
> -
>   	vdev = scmi_get_transport_info(dev);
>   	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
>   
> @@ -259,9 +302,7 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>   	vioch->cinfo = cinfo;
>   	spin_unlock_irqrestore(&vioch->lock, flags);
>   
> -	max_msg = virtio_get_max_msg(tx, cinfo);
> -
> -	for (i = 0; i < max_msg; i++) {
> +	for (i = 0; i < vioch->max_msg; i++) {
>   		struct scmi_vio_msg *msg;
>   
>   		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
> @@ -322,13 +363,6 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
>   	int rc;
>   	struct scmi_vio_msg *msg;
>   
> -	/*
> -	 * TODO: For now, we don't support polling. But it should not be
> -	 * difficult to add support.
> -	 */
> -	if (xfer->hdr.poll_completion)
> -		return -EINVAL;
> -
>   	spin_lock_irqsave(&vioch->lock, flags);
>   
>   	if (list_empty(&vioch->free_list)) {
> @@ -351,6 +385,11 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
>   			     "%s() failed to add to virtqueue (%d)\n", __func__,
>   			     rc);
>   	} else {
> +		dev_dbg(vioch->cinfo->dev,
> +			"VQUEUE[%d] - REQUEST - PROTO:0x%X  ID:0x%X  XFER_ID:%d  XFER:%px  RX_LEN:%zd\n",
> +		 vioch->vqueue->index, xfer->hdr.protocol_id,
> +		 xfer->hdr.id, xfer->hdr.seq, xfer, xfer->rx.len);

Indentation appears to be inconsistent.

> +
>   		virtqueue_kick(vioch->vqueue);
>   	}
>   
> @@ -360,36 +399,15 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
>   }
>   
>   static void virtio_fetch_response(struct scmi_chan_info *cinfo,
> -				  struct scmi_xfer *xfer, void *msg_handle)
> +				  struct scmi_xfer *xfer)
>   {
> -	struct scmi_vio_msg *msg = msg_handle;
> -	struct scmi_vio_channel *vioch = cinfo->transport_info;
> -
> -	if (!msg) {
> -		dev_dbg_once(&vioch->vqueue->vdev->dev,
> -			     "Ignoring %s() call with NULL msg_handle\n",
> -			     __func__);
> -		return;
> -	}
> -
> -	msg_fetch_response(msg->input, msg->rx_len, xfer);
> +	msg_fetch_raw_response(xfer);
>   }
>   
>   static void virtio_fetch_notification(struct scmi_chan_info *cinfo,
> -				      size_t max_len, struct scmi_xfer *xfer,
> -				      void *msg_handle)
> +				      size_t max_len, struct scmi_xfer *xfer)
>   {
> -	struct scmi_vio_msg *msg = msg_handle;
> -	struct scmi_vio_channel *vioch = cinfo->transport_info;
> -
> -	if (!msg) {
> -		dev_dbg_once(&vioch->vqueue->vdev->dev,
> -			     "Ignoring %s() call with NULL msg_handle\n",
> -			     __func__);
> -		return;
> -	}
> -
> -	msg_fetch_notification(msg->input, msg->rx_len, max_len, xfer);
> +	msg_fetch_raw_notification(xfer);
>   }
>   
>   static void dummy_clear_channel(struct scmi_chan_info *cinfo)
> @@ -402,28 +420,6 @@ static bool dummy_poll_done(struct scmi_chan_info *cinfo,
>   	return false;
>   }
>   
> -static void virtio_drop_message(struct scmi_chan_info *cinfo, void *msg_handle)
> -{
> -	unsigned long flags;
> -	struct scmi_vio_channel *vioch = cinfo->transport_info;
> -	struct scmi_vio_msg *msg = msg_handle;
> -
> -	if (!msg) {
> -		dev_dbg_once(&vioch->vqueue->vdev->dev,
> -			     "Ignoring %s() call with NULL msg_handle\n",
> -			     __func__);
> -		return;
> -	}
> -
> -	if (vioch->is_rx) {
> -		scmi_vio_feed_vq_rx(vioch, msg);
> -	} else {
> -		spin_lock_irqsave(&vioch->lock, flags);
> -		list_add(&msg->list, &vioch->free_list);
> -		spin_unlock_irqrestore(&vioch->lock, flags);
> -	}
> -}
> -
>   static const struct scmi_transport_ops scmi_virtio_ops = {
>   	.link_supplier = virtio_link_supplier,
>   	.chan_available = virtio_chan_available,
> @@ -435,7 +431,6 @@ static const struct scmi_transport_ops scmi_virtio_ops = {
>   	.fetch_notification = virtio_fetch_notification,
>   	.clear_channel = dummy_clear_channel,
>   	.poll_done = dummy_poll_done,
> -	.drop_message = virtio_drop_message,
>   };
>   
>   static int scmi_vio_probe(struct virtio_device *vdev)
> @@ -467,10 +462,26 @@ static int scmi_vio_probe(struct virtio_device *vdev)
>   	dev_info(dev, "Found %d virtqueue(s)\n", vq_cnt);
>   
>   	for (i = 0; i < vq_cnt; i++) {
> +		unsigned int sz;
> +
>   		spin_lock_init(&channels[i].lock);
>   		spin_lock_init(&channels[i].ready_lock);
>   		INIT_LIST_HEAD(&channels[i].free_list);
>   		channels[i].vqueue = vqs[i];
> +
> +		sz = virtqueue_get_vring_size(channels[i].vqueue);
> +		/* Tx messages need multiple descriptors. */
> +		if (!channels[i].is_rx)
> +			sz /= DESCRIPTORS_PER_TX_MSG;
> +
> +		if (sz > MSG_TOKEN_MAX) {
> +			dev_info_once(dev,
> +				      "%s virtqueue could hold %d messages. Only %ld allowed to be pending.\n",
> +				      channels[i].is_rx ? "rx" : "tx",
> +				      sz, MSG_TOKEN_MAX);
> +			sz = MSG_TOKEN_MAX;
> +		}
> +		channels[i].max_msg = sz;
>   	}
>   
>   	vdev->priv = channels;
> @@ -520,4 +531,5 @@ const struct scmi_desc scmi_virtio_desc = {
>   	.max_rx_timeout_ms = 60000, /* for non-realtime virtio devices */
>   	.max_msg = 0, /* overridden by virtio_get_max_msg() */
>   	.max_msg_size = VIRTIO_SCMI_MAX_MSG_SIZE,
> +	.support_xfers_delegation = true,
>   };
> 




_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 01/16] firmware: arm_scmi: Fix max pending messages boundary check
  2021-07-01  8:42     ` Peter Hilber
@ 2021-07-01 10:04       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:04 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi Peter,

On Thu, Jul 01, 2021 at 10:42:40AM +0200, Peter Hilber wrote:
> Hi Cristian,
> 
> please find some remarks to the patch series in this email and the
> following.
> 

Thanks for your comments, very much appreciated. I'll reply inline.

Just to let you know, I have ready a V5 series where, beside some
general cleanup and further simplification, I addressed in the SCMI core
the issue that you pointed out about the possible concurrent and out-of-order
response/delayed_response delivery by the transport.

I've refrained from posting that on the list still, due to the merge window
being open. I'll post most probably next week. (still have to see if I
can also simplify probing sequence in V5...which is the last point in my
list)

> On 11.06.21 18:59, Cristian Marussi wrote:
> > SCMI message headers carry a sequence number and such field is sized to
> > allow for MSG_TOKEN_MAX distinct numbers; moreover zero is not really an
> > acceptable maximum number of pending in-flight messages.
> > 
> > Fix accordignly the checks performed on the value exported by transports
> > in scmi_desc.max_msg.
> > 
> > Reported-by: Vincent Guittot <vincent.guittot@linaro.org>
> > Fixes: aa4f886f3893 ("firmware: arm_scmi: add basic driver infrastructure for SCMI")
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> >  drivers/firmware/arm_scmi/driver.c | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index 66e5e694be7d..6713b259f1e6 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -1025,8 +1025,9 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
> >  	const struct scmi_desc *desc = sinfo->desc;
> >  	/* Pre-allocated messages, no more than what hdr.seq can support */
> > -	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
> > -		dev_err(dev, "Maximum message of %d exceeds supported %ld\n",
> > +	if (WARN_ON(!desc->max_msg || desc->max_msg > MSG_TOKEN_MAX)) {
> > +		dev_err(dev,
> > +			"Invalid max_msg %d. Maximum messages supported %ld.\n",
> 
> %ld -> %lu
> 

Right, I'll fix.

> >  			desc->max_msg, MSG_TOKEN_MAX);
> >  		return -EINVAL;
> >  	}
> > 

Thanks,
Cristian


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 01/16] firmware: arm_scmi: Fix max pending messages boundary check
@ 2021-07-01 10:04       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:04 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi Peter,

On Thu, Jul 01, 2021 at 10:42:40AM +0200, Peter Hilber wrote:
> Hi Cristian,
> 
> please find some remarks to the patch series in this email and the
> following.
> 

Thanks for your comments, very much appreciated. I'll reply inline.

Just to let you know, I have ready a V5 series where, beside some
general cleanup and further simplification, I addressed in the SCMI core
the issue that you pointed out about the possible concurrent and out-of-order
response/delayed_response delivery by the transport.

I've refrained from posting that on the list still, due to the merge window
being open. I'll post most probably next week. (still have to see if I
can also simplify probing sequence in V5...which is the last point in my
list)

> On 11.06.21 18:59, Cristian Marussi wrote:
> > SCMI message headers carry a sequence number and such field is sized to
> > allow for MSG_TOKEN_MAX distinct numbers; moreover zero is not really an
> > acceptable maximum number of pending in-flight messages.
> > 
> > Fix accordignly the checks performed on the value exported by transports
> > in scmi_desc.max_msg.
> > 
> > Reported-by: Vincent Guittot <vincent.guittot@linaro.org>
> > Fixes: aa4f886f3893 ("firmware: arm_scmi: add basic driver infrastructure for SCMI")
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> >  drivers/firmware/arm_scmi/driver.c | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index 66e5e694be7d..6713b259f1e6 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -1025,8 +1025,9 @@ static int __scmi_xfer_info_init(struct scmi_info *sinfo,
> >  	const struct scmi_desc *desc = sinfo->desc;
> >  	/* Pre-allocated messages, no more than what hdr.seq can support */
> > -	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
> > -		dev_err(dev, "Maximum message of %d exceeds supported %ld\n",
> > +	if (WARN_ON(!desc->max_msg || desc->max_msg > MSG_TOKEN_MAX)) {
> > +		dev_err(dev,
> > +			"Invalid max_msg %d. Maximum messages supported %ld.\n",
> 
> %ld -> %lu
> 

Right, I'll fix.

> >  			desc->max_msg, MSG_TOKEN_MAX);
> >  		return -EINVAL;
> >  	}
> > 

Thanks,
Cristian


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
  2021-07-01  8:42     ` Peter Hilber
@ 2021-07-01 10:16       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:16 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Thu, Jul 01, 2021 at 10:42:45AM +0200, Peter Hilber wrote:
> I find `monotonically increasing tokens' misleading, since the token may
> wrap around quite often during normal usage. How about `modulo
> incrementing'?
> 

Ok, I'll have a though about a better naming.

> On 11.06.21 18:59, Cristian Marussi wrote:
> > Tokens are sequence numbers embedded in the each SCMI message header: they
> > are used to correlate commands with responses (and delayed responses), but
> > their usage and policy of selection is entirely up to the caller (usually
> > the OSPM agent), while they are completely opaque to the callee (SCMI
> > server platform) which merely copies them back from the command into the
> > response message header.
> > This also means that the platform does not, can not and should not enforce
> > any kind of policy on received messages depending on the contained sequence
> > number: platform can perfectly handle concurrent requests carrying the same
> > identifiying token if that should happen.
> > 
> > Moreover the platform is not required to produce in-order responses to
> > agent requests, the only constraint in these regards is that in case of
> > an asynchronous message the delayed response must be sent after the
> > immediate response for the synchronous part of the command transaction.
> > 
> > Currenly the SCMI stack of the OSPM agent selects a token for the egressing
> > commands picking the lowest possible number which is not already in use by
> > an existing in-flight transaction, which means, in other words, that we
> > immediately reuse any token after its transaction has completed or it has
> > timed out: this policy indeed does simplify management and lookup of tokens
> > and associated xfers.
> > 
> > Under the above assumptions and constraints, since there is really no state
> > shared between the agent and the platform to let the platform know when a
> > token and its associated message has timed out, the current policy of early
> > reuse of tokens can easily lead to the situation in which a spurious or
> > late received response (or delayed_response), related to an old stale and
> > timed out transaction, can be wrongly associated to a newer valid in-flight
> > xfer that just happens to have reused the same token.
> > 
> > This misbehaviour on such ghost responses is more easily exposed on those
> > transports that naturally have an higher level of parallelism in processing
> > multiple concurrent in-flight messages.
> > 
> > This commit introduces a new policy of selection of tokens for the OSPM
> > agent: each new transfer now gets the next available and monotonically
> > increasing token, until tokens are exhausted and the counter rolls over.
> > 
> > Such new policy mitigates the above issues with ghost responses since the
> > tokens are now reused as late as possible (when they roll back ideally)
> > and so it is much easier to identify such ghost responses to stale timed
> > out transactions: this also helps in simplifying the specific transports
> > implementation since stale transport messages can be easily identified
> > and discarded early on in the rx path without the need to cross check
> > their actual state with the core transport layer.
> > This mitigation is even more effective when, as is usually the case, the
> > maximum number of pending messages is capped by the platform to a much
> > lower number than the whole possible range of tokens values (2^10).
> > 
> > This internal policy change in the core SCMI transport layer is fully
> > transparent to the specific transports so it has not and should not have
> > any impact on the transports implementation.
> > 
> > The empirically observed cost of such new procedure of token selection
> > amounts in the best case to ~10us out of an observed full transaction cost
> > of 3ms for the completion of a synchronous sensor reading command on a
> > platform supporting commands completion interrupts.
> > 
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> >  drivers/firmware/arm_scmi/common.h |  23 +++
> >  drivers/firmware/arm_scmi/driver.c | 243 +++++++++++++++++++++++++----
> >  2 files changed, 233 insertions(+), 33 deletions(-)
> > 
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index 6bb734e0e3ac..e64c5ca9ee7c 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -14,7 +14,10 @@
> >  #include <linux/device.h>
> >  #include <linux/errno.h>
> >  #include <linux/kernel.h>
> > +#include <linux/hashtable.h>
> > +#include <linux/list.h>
> >  #include <linux/module.h>
> > +#include <linux/refcount.h>
> >  #include <linux/scmi_protocol.h>
> >  #include <linux/types.h>
> > @@ -127,6 +130,21 @@ struct scmi_msg {
> >  	size_t len;
> >  };
> > +/**
> > + * An helper macro to lookup an xfer from the @pending_xfers hashtable
> > + * using the message sequence number token as a key.
> > + */
> > +#define XFER_FIND(__ht, __k)					\
> > +({								\
> > +	typeof(__k) k_ = __k;					\
> > +	struct scmi_xfer *xfer_ = NULL;				\
> > +								\
> > +	hash_for_each_possible((__ht), xfer_, node, k_)		\
> > +		if (xfer_->hdr.seq == k_)			\
> > +			break;					\
> > +	 xfer_;							\
> 
> There is an extra space before the return value.
> 

Ah right.

> > +})
> > +
> >  /**
> >   * struct scmi_xfer - Structure representing a message flow
> >   *
> > @@ -138,6 +156,9 @@ struct scmi_msg {
> >   *	buffer for the rx path as we use for the tx path.
> >   * @done: command message transmit completion event
> >   * @async_done: pointer to delayed response message received event completion
> > + * @users: A refcount to track the active users for this xfer
> > + * @node: An hlist_node reference used to store this xfer, alternatively, on
> > + *	  the free list @free_xfers or in the @pending_xfers hashtable
> >   */
> >  struct scmi_xfer {
> >  	int transfer_id;
> > @@ -146,6 +167,8 @@ struct scmi_xfer {
> >  	struct scmi_msg rx;
> >  	struct completion done;
> >  	struct completion *async_done;
> > +	refcount_t users;
> > +	struct hlist_node node;
> >  };
> >  struct scmi_xfer_ops;
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index 20f8f0581f3a..f0b20ddb24f4 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -21,6 +21,7 @@
> >  #include <linux/io.h>
> >  #include <linux/kernel.h>
> >  #include <linux/ktime.h>
> > +#include <linux/hashtable.h>
> >  #include <linux/list.h>
> >  #include <linux/module.h>
> >  #include <linux/of_address.h>
> > @@ -65,19 +66,29 @@ struct scmi_requested_dev {
> >  	struct list_head node;
> >  };
> > +#define SCMI_PENDING_XFERS_HT_ORDER_SZ	9
> 
> Micro-optimization note: This increases struct scmi_info size to > 8 kiB (on
> 64-bit architectures). A lower size of the hash table would be enough for
> some transports.
> 

Indeed, I did not like this fixed ht, and I went for it for simplicity
at first: I would have liked to dynamically size the HT based on the real
number of pending in-flight messages declared by the transport with
max_msg (since it should be good enough to approximate an HT with the
minimum size needed to also minimize collision.) Unfortunately kernel HT
support above works only with statically sized HT, you cannot just
allocate dynamically the underlying array, all supporting macros will
break. (unless I'm missing something)

Unless I'll find a solution for this dynamic sizing, I think that sizing
it down as you proposed will be a good idea.

> > +
> >  /**
> >   * struct scmi_xfers_info - Structure to manage transfer information
> >   *
> > - * @xfer_block: Preallocated Message array
> >   * @xfer_alloc_table: Bitmap table for allocated messages.
> >   *	Index of this bitmap table is also used for message
> >   *	sequence identifier.
> >   * @xfer_lock: Protection for message allocation
> > + * @last_token: A counter to use as base to generate for monotonically
> > + *		increasing tokens.
> > + * @free_xfers: A free list for available to use xfers. It is initialized with
> > + *		a number of xfers equal to the maximum allowed in-flight
> > + *		messages.
> > + * @pending_xfers: An hashtable, indexed by msg_hdr.seq, used to keep all the
> > + *		   currently in-flight messages.
> >   */
> >  struct scmi_xfers_info {
> > -	struct scmi_xfer *xfer_block;
> >  	unsigned long *xfer_alloc_table;
> >  	spinlock_t xfer_lock;
> > +	atomic_t last_token;
> > +	struct hlist_head free_xfers;
> > +	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
> >  };
> >  /**
> > @@ -203,6 +214,117 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
> >  	return info->notify_priv;
> >  }
> > +/**
> > + * scmi_xfer_token_set  - Reserve and set new token for the xfer at hand
> > + *
> > + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> > + * @xfer: The xfer to act upon
> > + *
> > + * Pick the next unused monotonically increasing token and set it into
> > + * xfer->hdr.seq: picking a monotonically increasing value avoids immediate
> > + * reuse of freshly completed or timed-out xfers, thus mitigating the risk
> > + * of incorrect association of a late and expired xfer with a live in-flight
> > + * transaction, both happening to re-use the same token identifier.
> > + *
> > + * Since platform is NOT required to answer our request in-order we should
> > + * account for a few rare but possible scenarios:
> > + *
> > + *  - exactly 'next_token' may be NOT available so pick xfer_id >= next_token
> > + *    using find_next_zero_bit() starting from candidate next_token bit
> > + *
> > + *  - all tokens ahead upto (MSG_TOKEN_ID_MASK - 1) are used in-flight but we
> > + *    are plenty of free tokens at start, so try a second pass using
> > + *    find_next_zero_bit() and starting from 0.
> > + *
> > + *  X = used in-flight
> > + *
> > + * Normal
> > + * ------
> > + *
> > + *		|- xfer_id picked
> > + *   -----------+----------------------------------------------------------
> > + *   | | |X|X|X| | | | | | ... ... ... ... ... ... ... ... ... ... ...|X|X|
> > + *   ----------------------------------------------------------------------
> > + *		^
> > + *		|- next_token
> > + *
> > + * Out-of-order pending at start
> > + * -----------------------------
> > + *
> > + *	  |- xfer_id picked, last_token fixed
> > + *   -----+----------------------------------------------------------------
> > + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... ... ...|X| |
> > + *   ----------------------------------------------------------------------
> > + *    ^
> > + *    |- next_token
> > + *
> > + *
> > + * Out-of-order pending at end
> > + * ---------------------------
> > + *
> > + *	  |- xfer_id picked, last_token fixed
> > + *   -----+----------------------------------------------------------------
> > + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... |X|X|X||X|X|
> > + *   ----------------------------------------------------------------------
> > + *								^
> > + *								|- next_token
> > + *
> > + * Context: Assumes to be called with @xfer_lock already acquired.
> > + *
> > + * Return: 0 on Success or error
> > + */
> > +static int scmi_xfer_token_set(struct scmi_xfers_info *minfo,
> > +			       struct scmi_xfer *xfer)
> > +{
> > +	unsigned long xfer_id, next_token;
> > +
> > +	/* Pick a candidate monotonic token in range [0, MSG_TOKEN_MAX - 1] */
> > +	next_token = (atomic_inc_return(&minfo->last_token) &
> > +		      (MSG_TOKEN_MAX - 1));
> > +
> > +	/* Pick the next available xfer_id >= next_token */
> > +	xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> > +				     MSG_TOKEN_MAX, next_token);
> > +	if (xfer_id == MSG_TOKEN_MAX) {
> > +		/*
> > +		 * After heavily out-of-order responses, there are no free
> > +		 * tokens ahead, but only at start of xfer_alloc_table so
> > +		 * try again from the beginning.
> > +		 */
> > +		xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> > +					     MSG_TOKEN_MAX, 0);
> > +		/*
> > +		 * Something is wrong if we got here since there can be a
> > +		 * maximum number of (MSG_TOKEN_MAX - 1) in-flight messages
> > +		 * but we have not found any free token [0, MSG_TOKEN_MAX - 1].
> > +		 */
> > +		if (WARN_ON_ONCE(xfer_id == MSG_TOKEN_MAX))
> > +			return -ENOMEM;
> > +	}
> > +
> > +	/* Update +/- last_token accordingly if we skipped some hole */
> > +	if (xfer_id != next_token)
> > +		atomic_add((int)(xfer_id - next_token), &minfo->last_token);
> > +
> > +	/* Set in-flight */
> > +	set_bit(xfer_id, minfo->xfer_alloc_table);
> > +	xfer->hdr.seq = (u16)xfer_id;
> > +
> > +	return 0;
> > +}
> > +
> > +/**
> > + * scmi_xfer_token_clear  - Release the token
> > + *
> > + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> > + * @xfer: The xfer to act upon
> > + */
> > +static inline void scmi_xfer_token_clear(struct scmi_xfers_info *minfo,
> > +					 struct scmi_xfer *xfer)
> > +{
> > +	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> > +}
> > +
> >  /**
> >   * scmi_xfer_get() - Allocate one message
> >   *
> > @@ -212,36 +334,49 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
> >   * Helper function which is used by various message functions that are
> >   * exposed to clients of this driver for allocating a message traffic event.
> >   *
> > - * This function can sleep depending on pending requests already in the system
> > - * for the SCMI entity. Further, this also holds a spinlock to maintain
> > - * integrity of internal data structures.
> > + * Picks an xfer from the free list @free_xfers (if any available), sets a
> > + * monotonically increasing token and stores the inflight xfer into the
> > + * @pending_xfers hashtable for later retrieval.
> > + *
> > + * The successfully initialized xfer is refcounted.
> > + *
> > + * Context: Holds @xfer_lock while manipulating @xfer_alloc_table and
> > + *	    @free_xfers.
> >   *
> >   * Return: 0 if all went fine, else corresponding error.
> >   */
> >  static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
> >  				       struct scmi_xfers_info *minfo)
> >  {
> > -	u16 xfer_id;
> > +	int ret;
> > +	unsigned long flags;
> >  	struct scmi_xfer *xfer;
> > -	unsigned long flags, bit_pos;
> > -	struct scmi_info *info = handle_to_scmi_info(handle);
> > -	/* Keep the locked section as small as possible */
> >  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> > -	bit_pos = find_first_zero_bit(minfo->xfer_alloc_table,
> > -				      info->desc->max_msg);
> > -	if (bit_pos == info->desc->max_msg) {
> > +	if (hlist_empty(&minfo->free_xfers)) {
> >  		spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> >  		return ERR_PTR(-ENOMEM);
> >  	}
> > -	set_bit(bit_pos, minfo->xfer_alloc_table);
> > -	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> > -	xfer_id = bit_pos;
> > +	/* grab an xfer from the free_list */
> > +	xfer = hlist_entry(minfo->free_xfers.first, struct scmi_xfer, node);
> > +	hlist_del_init(&xfer->node);
> > -	xfer = &minfo->xfer_block[xfer_id];
> > -	xfer->hdr.seq = xfer_id;
> > -	xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> > +	/* Pick and set monotonic token */
> > +	ret = scmi_xfer_token_set(minfo, xfer);
> > +	if (!ret) {
> > +		hash_add(minfo->pending_xfers, &xfer->node, xfer->hdr.seq);
> > +	} else {
> > +		dev_err(handle->dev, "Failed to get monotonic token %d\n", ret);
> > +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> > +		xfer = ERR_PTR(ret);
> > +	}
> > +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> > +
> > +	if (!IS_ERR(xfer)) {
> > +		refcount_set(&xfer->users, 1);
> 
> Maybe it would be better to do this inside the lock, so that there is no
> (unlikely) race with refcount_inc() in scmi_xfer_acquire().
> 

I'll check on this.

Thanks,
Cristian


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens
@ 2021-07-01 10:16       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:16 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Thu, Jul 01, 2021 at 10:42:45AM +0200, Peter Hilber wrote:
> I find `monotonically increasing tokens' misleading, since the token may
> wrap around quite often during normal usage. How about `modulo
> incrementing'?
> 

Ok, I'll have a though about a better naming.

> On 11.06.21 18:59, Cristian Marussi wrote:
> > Tokens are sequence numbers embedded in the each SCMI message header: they
> > are used to correlate commands with responses (and delayed responses), but
> > their usage and policy of selection is entirely up to the caller (usually
> > the OSPM agent), while they are completely opaque to the callee (SCMI
> > server platform) which merely copies them back from the command into the
> > response message header.
> > This also means that the platform does not, can not and should not enforce
> > any kind of policy on received messages depending on the contained sequence
> > number: platform can perfectly handle concurrent requests carrying the same
> > identifiying token if that should happen.
> > 
> > Moreover the platform is not required to produce in-order responses to
> > agent requests, the only constraint in these regards is that in case of
> > an asynchronous message the delayed response must be sent after the
> > immediate response for the synchronous part of the command transaction.
> > 
> > Currenly the SCMI stack of the OSPM agent selects a token for the egressing
> > commands picking the lowest possible number which is not already in use by
> > an existing in-flight transaction, which means, in other words, that we
> > immediately reuse any token after its transaction has completed or it has
> > timed out: this policy indeed does simplify management and lookup of tokens
> > and associated xfers.
> > 
> > Under the above assumptions and constraints, since there is really no state
> > shared between the agent and the platform to let the platform know when a
> > token and its associated message has timed out, the current policy of early
> > reuse of tokens can easily lead to the situation in which a spurious or
> > late received response (or delayed_response), related to an old stale and
> > timed out transaction, can be wrongly associated to a newer valid in-flight
> > xfer that just happens to have reused the same token.
> > 
> > This misbehaviour on such ghost responses is more easily exposed on those
> > transports that naturally have an higher level of parallelism in processing
> > multiple concurrent in-flight messages.
> > 
> > This commit introduces a new policy of selection of tokens for the OSPM
> > agent: each new transfer now gets the next available and monotonically
> > increasing token, until tokens are exhausted and the counter rolls over.
> > 
> > Such new policy mitigates the above issues with ghost responses since the
> > tokens are now reused as late as possible (when they roll back ideally)
> > and so it is much easier to identify such ghost responses to stale timed
> > out transactions: this also helps in simplifying the specific transports
> > implementation since stale transport messages can be easily identified
> > and discarded early on in the rx path without the need to cross check
> > their actual state with the core transport layer.
> > This mitigation is even more effective when, as is usually the case, the
> > maximum number of pending messages is capped by the platform to a much
> > lower number than the whole possible range of tokens values (2^10).
> > 
> > This internal policy change in the core SCMI transport layer is fully
> > transparent to the specific transports so it has not and should not have
> > any impact on the transports implementation.
> > 
> > The empirically observed cost of such new procedure of token selection
> > amounts in the best case to ~10us out of an observed full transaction cost
> > of 3ms for the completion of a synchronous sensor reading command on a
> > platform supporting commands completion interrupts.
> > 
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> >  drivers/firmware/arm_scmi/common.h |  23 +++
> >  drivers/firmware/arm_scmi/driver.c | 243 +++++++++++++++++++++++++----
> >  2 files changed, 233 insertions(+), 33 deletions(-)
> > 
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index 6bb734e0e3ac..e64c5ca9ee7c 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -14,7 +14,10 @@
> >  #include <linux/device.h>
> >  #include <linux/errno.h>
> >  #include <linux/kernel.h>
> > +#include <linux/hashtable.h>
> > +#include <linux/list.h>
> >  #include <linux/module.h>
> > +#include <linux/refcount.h>
> >  #include <linux/scmi_protocol.h>
> >  #include <linux/types.h>
> > @@ -127,6 +130,21 @@ struct scmi_msg {
> >  	size_t len;
> >  };
> > +/**
> > + * An helper macro to lookup an xfer from the @pending_xfers hashtable
> > + * using the message sequence number token as a key.
> > + */
> > +#define XFER_FIND(__ht, __k)					\
> > +({								\
> > +	typeof(__k) k_ = __k;					\
> > +	struct scmi_xfer *xfer_ = NULL;				\
> > +								\
> > +	hash_for_each_possible((__ht), xfer_, node, k_)		\
> > +		if (xfer_->hdr.seq == k_)			\
> > +			break;					\
> > +	 xfer_;							\
> 
> There is an extra space before the return value.
> 

Ah right.

> > +})
> > +
> >  /**
> >   * struct scmi_xfer - Structure representing a message flow
> >   *
> > @@ -138,6 +156,9 @@ struct scmi_msg {
> >   *	buffer for the rx path as we use for the tx path.
> >   * @done: command message transmit completion event
> >   * @async_done: pointer to delayed response message received event completion
> > + * @users: A refcount to track the active users for this xfer
> > + * @node: An hlist_node reference used to store this xfer, alternatively, on
> > + *	  the free list @free_xfers or in the @pending_xfers hashtable
> >   */
> >  struct scmi_xfer {
> >  	int transfer_id;
> > @@ -146,6 +167,8 @@ struct scmi_xfer {
> >  	struct scmi_msg rx;
> >  	struct completion done;
> >  	struct completion *async_done;
> > +	refcount_t users;
> > +	struct hlist_node node;
> >  };
> >  struct scmi_xfer_ops;
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index 20f8f0581f3a..f0b20ddb24f4 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -21,6 +21,7 @@
> >  #include <linux/io.h>
> >  #include <linux/kernel.h>
> >  #include <linux/ktime.h>
> > +#include <linux/hashtable.h>
> >  #include <linux/list.h>
> >  #include <linux/module.h>
> >  #include <linux/of_address.h>
> > @@ -65,19 +66,29 @@ struct scmi_requested_dev {
> >  	struct list_head node;
> >  };
> > +#define SCMI_PENDING_XFERS_HT_ORDER_SZ	9
> 
> Micro-optimization note: This increases struct scmi_info size to > 8 kiB (on
> 64-bit architectures). A lower size of the hash table would be enough for
> some transports.
> 

Indeed, I did not like this fixed ht, and I went for it for simplicity
at first: I would have liked to dynamically size the HT based on the real
number of pending in-flight messages declared by the transport with
max_msg (since it should be good enough to approximate an HT with the
minimum size needed to also minimize collision.) Unfortunately kernel HT
support above works only with statically sized HT, you cannot just
allocate dynamically the underlying array, all supporting macros will
break. (unless I'm missing something)

Unless I'll find a solution for this dynamic sizing, I think that sizing
it down as you proposed will be a good idea.

> > +
> >  /**
> >   * struct scmi_xfers_info - Structure to manage transfer information
> >   *
> > - * @xfer_block: Preallocated Message array
> >   * @xfer_alloc_table: Bitmap table for allocated messages.
> >   *	Index of this bitmap table is also used for message
> >   *	sequence identifier.
> >   * @xfer_lock: Protection for message allocation
> > + * @last_token: A counter to use as base to generate for monotonically
> > + *		increasing tokens.
> > + * @free_xfers: A free list for available to use xfers. It is initialized with
> > + *		a number of xfers equal to the maximum allowed in-flight
> > + *		messages.
> > + * @pending_xfers: An hashtable, indexed by msg_hdr.seq, used to keep all the
> > + *		   currently in-flight messages.
> >   */
> >  struct scmi_xfers_info {
> > -	struct scmi_xfer *xfer_block;
> >  	unsigned long *xfer_alloc_table;
> >  	spinlock_t xfer_lock;
> > +	atomic_t last_token;
> > +	struct hlist_head free_xfers;
> > +	DECLARE_HASHTABLE(pending_xfers, SCMI_PENDING_XFERS_HT_ORDER_SZ);
> >  };
> >  /**
> > @@ -203,6 +214,117 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
> >  	return info->notify_priv;
> >  }
> > +/**
> > + * scmi_xfer_token_set  - Reserve and set new token for the xfer at hand
> > + *
> > + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> > + * @xfer: The xfer to act upon
> > + *
> > + * Pick the next unused monotonically increasing token and set it into
> > + * xfer->hdr.seq: picking a monotonically increasing value avoids immediate
> > + * reuse of freshly completed or timed-out xfers, thus mitigating the risk
> > + * of incorrect association of a late and expired xfer with a live in-flight
> > + * transaction, both happening to re-use the same token identifier.
> > + *
> > + * Since platform is NOT required to answer our request in-order we should
> > + * account for a few rare but possible scenarios:
> > + *
> > + *  - exactly 'next_token' may be NOT available so pick xfer_id >= next_token
> > + *    using find_next_zero_bit() starting from candidate next_token bit
> > + *
> > + *  - all tokens ahead upto (MSG_TOKEN_ID_MASK - 1) are used in-flight but we
> > + *    are plenty of free tokens at start, so try a second pass using
> > + *    find_next_zero_bit() and starting from 0.
> > + *
> > + *  X = used in-flight
> > + *
> > + * Normal
> > + * ------
> > + *
> > + *		|- xfer_id picked
> > + *   -----------+----------------------------------------------------------
> > + *   | | |X|X|X| | | | | | ... ... ... ... ... ... ... ... ... ... ...|X|X|
> > + *   ----------------------------------------------------------------------
> > + *		^
> > + *		|- next_token
> > + *
> > + * Out-of-order pending at start
> > + * -----------------------------
> > + *
> > + *	  |- xfer_id picked, last_token fixed
> > + *   -----+----------------------------------------------------------------
> > + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... ... ...|X| |
> > + *   ----------------------------------------------------------------------
> > + *    ^
> > + *    |- next_token
> > + *
> > + *
> > + * Out-of-order pending at end
> > + * ---------------------------
> > + *
> > + *	  |- xfer_id picked, last_token fixed
> > + *   -----+----------------------------------------------------------------
> > + *   |X|X| | | | |X|X| ... ... ... ... ... ... ... ... ... ... |X|X|X||X|X|
> > + *   ----------------------------------------------------------------------
> > + *								^
> > + *								|- next_token
> > + *
> > + * Context: Assumes to be called with @xfer_lock already acquired.
> > + *
> > + * Return: 0 on Success or error
> > + */
> > +static int scmi_xfer_token_set(struct scmi_xfers_info *minfo,
> > +			       struct scmi_xfer *xfer)
> > +{
> > +	unsigned long xfer_id, next_token;
> > +
> > +	/* Pick a candidate monotonic token in range [0, MSG_TOKEN_MAX - 1] */
> > +	next_token = (atomic_inc_return(&minfo->last_token) &
> > +		      (MSG_TOKEN_MAX - 1));
> > +
> > +	/* Pick the next available xfer_id >= next_token */
> > +	xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> > +				     MSG_TOKEN_MAX, next_token);
> > +	if (xfer_id == MSG_TOKEN_MAX) {
> > +		/*
> > +		 * After heavily out-of-order responses, there are no free
> > +		 * tokens ahead, but only at start of xfer_alloc_table so
> > +		 * try again from the beginning.
> > +		 */
> > +		xfer_id = find_next_zero_bit(minfo->xfer_alloc_table,
> > +					     MSG_TOKEN_MAX, 0);
> > +		/*
> > +		 * Something is wrong if we got here since there can be a
> > +		 * maximum number of (MSG_TOKEN_MAX - 1) in-flight messages
> > +		 * but we have not found any free token [0, MSG_TOKEN_MAX - 1].
> > +		 */
> > +		if (WARN_ON_ONCE(xfer_id == MSG_TOKEN_MAX))
> > +			return -ENOMEM;
> > +	}
> > +
> > +	/* Update +/- last_token accordingly if we skipped some hole */
> > +	if (xfer_id != next_token)
> > +		atomic_add((int)(xfer_id - next_token), &minfo->last_token);
> > +
> > +	/* Set in-flight */
> > +	set_bit(xfer_id, minfo->xfer_alloc_table);
> > +	xfer->hdr.seq = (u16)xfer_id;
> > +
> > +	return 0;
> > +}
> > +
> > +/**
> > + * scmi_xfer_token_clear  - Release the token
> > + *
> > + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> > + * @xfer: The xfer to act upon
> > + */
> > +static inline void scmi_xfer_token_clear(struct scmi_xfers_info *minfo,
> > +					 struct scmi_xfer *xfer)
> > +{
> > +	clear_bit(xfer->hdr.seq, minfo->xfer_alloc_table);
> > +}
> > +
> >  /**
> >   * scmi_xfer_get() - Allocate one message
> >   *
> > @@ -212,36 +334,49 @@ void *scmi_notification_instance_data_get(const struct scmi_handle *handle)
> >   * Helper function which is used by various message functions that are
> >   * exposed to clients of this driver for allocating a message traffic event.
> >   *
> > - * This function can sleep depending on pending requests already in the system
> > - * for the SCMI entity. Further, this also holds a spinlock to maintain
> > - * integrity of internal data structures.
> > + * Picks an xfer from the free list @free_xfers (if any available), sets a
> > + * monotonically increasing token and stores the inflight xfer into the
> > + * @pending_xfers hashtable for later retrieval.
> > + *
> > + * The successfully initialized xfer is refcounted.
> > + *
> > + * Context: Holds @xfer_lock while manipulating @xfer_alloc_table and
> > + *	    @free_xfers.
> >   *
> >   * Return: 0 if all went fine, else corresponding error.
> >   */
> >  static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle,
> >  				       struct scmi_xfers_info *minfo)
> >  {
> > -	u16 xfer_id;
> > +	int ret;
> > +	unsigned long flags;
> >  	struct scmi_xfer *xfer;
> > -	unsigned long flags, bit_pos;
> > -	struct scmi_info *info = handle_to_scmi_info(handle);
> > -	/* Keep the locked section as small as possible */
> >  	spin_lock_irqsave(&minfo->xfer_lock, flags);
> > -	bit_pos = find_first_zero_bit(minfo->xfer_alloc_table,
> > -				      info->desc->max_msg);
> > -	if (bit_pos == info->desc->max_msg) {
> > +	if (hlist_empty(&minfo->free_xfers)) {
> >  		spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> >  		return ERR_PTR(-ENOMEM);
> >  	}
> > -	set_bit(bit_pos, minfo->xfer_alloc_table);
> > -	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> > -	xfer_id = bit_pos;
> > +	/* grab an xfer from the free_list */
> > +	xfer = hlist_entry(minfo->free_xfers.first, struct scmi_xfer, node);
> > +	hlist_del_init(&xfer->node);
> > -	xfer = &minfo->xfer_block[xfer_id];
> > -	xfer->hdr.seq = xfer_id;
> > -	xfer->transfer_id = atomic_inc_return(&transfer_last_id);
> > +	/* Pick and set monotonic token */
> > +	ret = scmi_xfer_token_set(minfo, xfer);
> > +	if (!ret) {
> > +		hash_add(minfo->pending_xfers, &xfer->node, xfer->hdr.seq);
> > +	} else {
> > +		dev_err(handle->dev, "Failed to get monotonic token %d\n", ret);
> > +		hlist_add_head(&xfer->node, &minfo->free_xfers);
> > +		xfer = ERR_PTR(ret);
> > +	}
> > +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> > +
> > +	if (!IS_ERR(xfer)) {
> > +		refcount_set(&xfer->users, 1);
> 
> Maybe it would be better to do this inside the lock, so that there is no
> (unlikely) race with refcount_inc() in scmi_xfer_acquire().
> 

I'll check on this.

Thanks,
Cristian


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 05/16] firmware: arm_scmi: Introduce delegated xfers support
  2021-07-01  8:42     ` Peter Hilber
@ 2021-07-01 10:24       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:24 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Thu, Jul 01, 2021 at 10:42:51AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> > Introduce optional support for delegated xfers allocation.
> > 
> > An SCMI transport can optionally declare to support delegated xfers and
> > then use a few helper functions exposed by the core SCMI transport layer to
> > query the core for existing in-flight transfers matching a provided message
> > header or alternatively and transparently obtain a brand new xfer to handle
> > a freshly received notification message.
> > In both cases the obtained xfer is uniquely mapped into a specific xfer
> > through the means of the message header acting as key.
> > 
> > In this way such a transport can properly store its own transport specific
> > payload into the xfer uniquely associated to the message header before
> > even calling into the core scmi_rx_callback() in the usual way, so that
> > the transport specific message envelope structures can be freed early
> > and there is no more need to keep track of their status till the core
> > fully processes the xfer to completion or times out.
> > 
> > The scmi_rx_callbak() does not need to be modified to carry additional
> > transport-specific ancillary data related to such message envelopes since
> > an unique natural association is established between the xfer and the
> > related message header.
> > 
> > Existing transports that do not need anything of the above will continue
> > to work as before without any change.
> > 
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> >  drivers/firmware/arm_scmi/common.h |  14 +++
> >  drivers/firmware/arm_scmi/driver.c | 132 ++++++++++++++++++++++++++++-
> >  2 files changed, 143 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index e64c5ca9ee7c..0edc04bc434c 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -80,6 +80,7 @@ struct scmi_msg_resp_prot_version {
> >   * @status: Status of the transfer once it's complete
> >   * @poll_completion: Indicate if the transfer needs to be polled for
> >   *	completion or interrupt mode is used
> > + * @saved_hdr: A copy of the original msg_hdr
> >   */
> >  struct scmi_msg_hdr {
> >  	u8 id;
> > @@ -88,6 +89,7 @@ struct scmi_msg_hdr {
> >  	u16 seq;
> >  	u32 status;
> >  	bool poll_completion;
> > +	u32 saved_hdr;
> >  };
> >  /**
> > @@ -154,6 +156,9 @@ struct scmi_msg {
> >   * @rx: Receive message, the buffer should be pre-allocated to store
> >   *	message. If request-ACK protocol is used, we can reuse the same
> >   *	buffer for the rx path as we use for the tx path.
> > + * @rx_raw_len: A field which can be optionally used by a specific transport
> > + *		to save transport specific message length
> > + *		It is not used by the SCMI transport core
> >   * @done: command message transmit completion event
> >   * @async_done: pointer to delayed response message received event completion
> >   * @users: A refcount to track the active users for this xfer
> > @@ -165,6 +170,7 @@ struct scmi_xfer {
> >  	struct scmi_msg_hdr hdr;
> >  	struct scmi_msg tx;
> >  	struct scmi_msg rx;
> > +	size_t rx_raw_len;
> >  	struct completion done;
> >  	struct completion *async_done;
> >  	refcount_t users;
> > @@ -355,6 +361,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
> >   * @max_msg: Maximum number of messages that can be pending
> >   *	simultaneously in the system
> >   * @max_msg_size: Maximum size of data per message that can be handled.
> > + * @support_xfers_delegation: A flag to indicate if the described transport
> > + *			      will handle delegated xfers, so the core can
> > + *			      derive proper related assumptions.
> >   */
> >  struct scmi_desc {
> >  	int (*init)(void);
> > @@ -363,6 +372,7 @@ struct scmi_desc {
> >  	int max_rx_timeout_ms;
> >  	int max_msg;
> >  	int max_msg_size;
> > +	bool support_xfers_delegation;
> >  };
> >  extern const struct scmi_desc scmi_mailbox_desc;
> > @@ -391,4 +401,8 @@ void scmi_notification_instance_data_set(const struct scmi_handle *handle,
> >  					 void *priv);
> >  void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
> > +int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
> > +			  struct scmi_xfer **xfer);
> > +void scmi_transfer_release(struct scmi_chan_info *cinfo,
> > +			   struct scmi_xfer *xfer);
> >  #endif /* _SCMI_COMMON_H */
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index f0b20ddb24f4..371d3804cd79 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -432,6 +432,124 @@ scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
> >  	return xfer ?: ERR_PTR(-EINVAL);
> >  }
> > +/**
> > + * scmi_xfer_acquire  -  Helper to lookup and acquire an xfer
> > + *
> > + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> > + * @xfer_id: Token ID to lookup in @pending_xfers
> > + *
> > + * When a valid xfer is found for the provided @xfer_id, reference counting is
> > + * properly updated.
> > + *
> > + * Return: A valid @xfer on Success or error otherwise.
> > + */
> > +static struct scmi_xfer *
> > +scmi_xfer_acquire(struct scmi_xfers_info *minfo, u16 xfer_id)
> > +{
> > +	unsigned long flags;
> > +	struct scmi_xfer *xfer;
> > +
> > +	spin_lock_irqsave(&minfo->xfer_lock, flags);
> > +	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
> > +	if (!IS_ERR(xfer))
> > +		refcount_inc(&xfer->users);
> > +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> > +
> > +	return xfer;
> > +}
> > +
> > +/**
> > + * scmi_transfer_acquire  -  Lookup for an existing xfer or freshly allocate a
> > + * new one depending on the type of the message
> > + *
> > + * @cinfo: A reference to the channel descriptor.
> > + * @msg_hdr: A pointer to the message header to lookup.
> > + * @xfer: A reference to the pre-existent or freshly allocated xfer
> > + *	  associated with the provided *msg_hdr.
> > + *
> > + * This function can be used by transports supporting delegated xfers to obtain
> > + * a valid @xfer associated with the provided @msg_hdr param.
> > + *
> > + * The nature of the resulting @xfer depends on the type of message specified in
> > + * @msg_hdr:
> > + *  - for responses and delayed responses a pre-existent/pre-allocated in-flight
> > + *    xfer descriptor will be returned (properly refcounted)
> > + *  - for notifications a brand new xfer will be allocated; in this case the
> > + *    provided message header sequence number will also be mangled to match
> > + *    the token in the freshly allocated xfer: this is needed to establish a
> > + *    link between the picked xfer and the msg_hdr that will be subsequently
> > + *    passed back via usual scmi_rx_callback().
> > + *
> > + * Return: 0 if a valid xfer is returned in @xfer, error otherwise.
> > + */
> > +int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
> > +			  struct scmi_xfer **xfer)
> > +{
> > +	u8 msg_type;
> > +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> > +
> > +	if (!xfer || !msg_hdr || !info->desc->support_xfers_delegation)
> > +		return -EINVAL;
> > +
> > +	msg_type = MSG_XTRACT_TYPE(*msg_hdr);
> > +	switch (msg_type) {
> > +	case MSG_TYPE_COMMAND:
> > +	case MSG_TYPE_DELAYED_RESP:
> > +		/* Grab an existing xfer for xfer_id */
> > +		*xfer = scmi_xfer_acquire(&info->tx_minfo,
> > +					  MSG_XTRACT_TOKEN(*msg_hdr));
> > +		break;
> > +	case MSG_TYPE_NOTIFICATION:
> > +		/* Get a brand new RX xfer */
> > +		*xfer = scmi_xfer_get(cinfo->handle, &info->rx_minfo);
> > +		if (!IS_ERR(*xfer)) {
> > +			/* Save original msg_hdr and fix sequence number */
> > +			(*xfer)->hdr.saved_hdr = *msg_hdr;
> 
> The saved header isn't used anywhere.
> 

Yes, in fact it's just stored just in case in the future for some reason
the spec would change somehow to require that the original msg sequence
number of a notification has to be used.

> > +			*msg_hdr &= ~MSG_TOKEN_ID_MASK;
> > +			*msg_hdr |= FIELD_PREP(MSG_TOKEN_ID_MASK,
> > +					       (*xfer)->hdr.seq);
> 
> This will invalidate the token set by the platform in
> scmi_dump_header_dbg(). Maybe it would have been more elegant to introduce a
> dedicated hash table key field?
> 

Not sure to have understood this thing about invalidation.

Anyway, I removed scmi_dump_header_dbg( ) in V5 since we have traces since a
while and those have anyway a dedicated xfer->transfer_id.

> > +		}
> > +		break;
> > +	default:
> > +		*xfer = ERR_PTR(-EINVAL);
> > +		break;
> > +	}
> > +
> > +	if (IS_ERR(*xfer)) {
> > +		dev_err(cinfo->dev,
> > +			"Failed to acquire a valid xfer for hdr:0x%X\n",
> > +			*msg_hdr);
> > +		return PTR_ERR(*xfer);
> > +	}
> > +
> > +	/* Fix xfer->hdr.type with actual msg_hdr carried type */
> > +	unpack_scmi_header(*msg_hdr, &((*xfer)->hdr));
> > +
> > +	return 0;
> > +}
> > +
> > +/**
> > + * scmi_transfer_release  - Release an previously acquired xfer
> > + *
> > + * @cinfo: A reference to the channel descriptor.
> > + * @xfer: A reference to the xfer to release.
> > + */
> > +void scmi_transfer_release(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer)
> > +{
> > +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> > +	struct scmi_xfers_info *minfo;
> > +
> > +	if (!xfer || !info->desc->support_xfers_delegation)
> > +		return;
> > +
> > +	if (xfer->hdr.type == MSG_TYPE_NOTIFICATION)
> > +		minfo = &info->rx_minfo;
> > +	else
> > +		minfo = &info->tx_minfo;
> > +
> > +	__scmi_xfer_put(minfo, xfer);
> > +}
> > +
> >  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
> >  {
> >  	struct scmi_xfer *xfer;
> > @@ -441,7 +559,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
> >  	ktime_t ts;
> >  	ts = ktime_get_boottime();
> > -	xfer = scmi_xfer_get(cinfo->handle, minfo);
> > +
> > +	if (!info->desc->support_xfers_delegation)
> > +		xfer = scmi_xfer_get(cinfo->handle, minfo);
> > +	else
> > +		xfer = scmi_xfer_acquire(minfo, MSG_XTRACT_TOKEN(msg_hdr));
> >  	if (IS_ERR(xfer)) {
> >  		dev_err(dev, "failed to get free message slot (%ld)\n",
> >  			PTR_ERR(xfer));
> > @@ -449,8 +571,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
> >  		return;
> >  	}
> > -	unpack_scmi_header(msg_hdr, &xfer->hdr);
> >  	scmi_dump_header_dbg(dev, &xfer->hdr);
> > +
> > +	if (!info->desc->support_xfers_delegation)
> > +		unpack_scmi_header(msg_hdr, &xfer->hdr);
> > +
> 
> Why dump the header before unpacking?
> 

Right, but as I said above I drop scmi_dump_header_dbg() in V5.

Thanks,
Cristian


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 05/16] firmware: arm_scmi: Introduce delegated xfers support
@ 2021-07-01 10:24       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:24 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Thu, Jul 01, 2021 at 10:42:51AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> > Introduce optional support for delegated xfers allocation.
> > 
> > An SCMI transport can optionally declare to support delegated xfers and
> > then use a few helper functions exposed by the core SCMI transport layer to
> > query the core for existing in-flight transfers matching a provided message
> > header or alternatively and transparently obtain a brand new xfer to handle
> > a freshly received notification message.
> > In both cases the obtained xfer is uniquely mapped into a specific xfer
> > through the means of the message header acting as key.
> > 
> > In this way such a transport can properly store its own transport specific
> > payload into the xfer uniquely associated to the message header before
> > even calling into the core scmi_rx_callback() in the usual way, so that
> > the transport specific message envelope structures can be freed early
> > and there is no more need to keep track of their status till the core
> > fully processes the xfer to completion or times out.
> > 
> > The scmi_rx_callbak() does not need to be modified to carry additional
> > transport-specific ancillary data related to such message envelopes since
> > an unique natural association is established between the xfer and the
> > related message header.
> > 
> > Existing transports that do not need anything of the above will continue
> > to work as before without any change.
> > 
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> >  drivers/firmware/arm_scmi/common.h |  14 +++
> >  drivers/firmware/arm_scmi/driver.c | 132 ++++++++++++++++++++++++++++-
> >  2 files changed, 143 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index e64c5ca9ee7c..0edc04bc434c 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -80,6 +80,7 @@ struct scmi_msg_resp_prot_version {
> >   * @status: Status of the transfer once it's complete
> >   * @poll_completion: Indicate if the transfer needs to be polled for
> >   *	completion or interrupt mode is used
> > + * @saved_hdr: A copy of the original msg_hdr
> >   */
> >  struct scmi_msg_hdr {
> >  	u8 id;
> > @@ -88,6 +89,7 @@ struct scmi_msg_hdr {
> >  	u16 seq;
> >  	u32 status;
> >  	bool poll_completion;
> > +	u32 saved_hdr;
> >  };
> >  /**
> > @@ -154,6 +156,9 @@ struct scmi_msg {
> >   * @rx: Receive message, the buffer should be pre-allocated to store
> >   *	message. If request-ACK protocol is used, we can reuse the same
> >   *	buffer for the rx path as we use for the tx path.
> > + * @rx_raw_len: A field which can be optionally used by a specific transport
> > + *		to save transport specific message length
> > + *		It is not used by the SCMI transport core
> >   * @done: command message transmit completion event
> >   * @async_done: pointer to delayed response message received event completion
> >   * @users: A refcount to track the active users for this xfer
> > @@ -165,6 +170,7 @@ struct scmi_xfer {
> >  	struct scmi_msg_hdr hdr;
> >  	struct scmi_msg tx;
> >  	struct scmi_msg rx;
> > +	size_t rx_raw_len;
> >  	struct completion done;
> >  	struct completion *async_done;
> >  	refcount_t users;
> > @@ -355,6 +361,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
> >   * @max_msg: Maximum number of messages that can be pending
> >   *	simultaneously in the system
> >   * @max_msg_size: Maximum size of data per message that can be handled.
> > + * @support_xfers_delegation: A flag to indicate if the described transport
> > + *			      will handle delegated xfers, so the core can
> > + *			      derive proper related assumptions.
> >   */
> >  struct scmi_desc {
> >  	int (*init)(void);
> > @@ -363,6 +372,7 @@ struct scmi_desc {
> >  	int max_rx_timeout_ms;
> >  	int max_msg;
> >  	int max_msg_size;
> > +	bool support_xfers_delegation;
> >  };
> >  extern const struct scmi_desc scmi_mailbox_desc;
> > @@ -391,4 +401,8 @@ void scmi_notification_instance_data_set(const struct scmi_handle *handle,
> >  					 void *priv);
> >  void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
> > +int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
> > +			  struct scmi_xfer **xfer);
> > +void scmi_transfer_release(struct scmi_chan_info *cinfo,
> > +			   struct scmi_xfer *xfer);
> >  #endif /* _SCMI_COMMON_H */
> > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > index f0b20ddb24f4..371d3804cd79 100644
> > --- a/drivers/firmware/arm_scmi/driver.c
> > +++ b/drivers/firmware/arm_scmi/driver.c
> > @@ -432,6 +432,124 @@ scmi_xfer_lookup_unlocked(struct scmi_xfers_info *minfo, u16 xfer_id)
> >  	return xfer ?: ERR_PTR(-EINVAL);
> >  }
> > +/**
> > + * scmi_xfer_acquire  -  Helper to lookup and acquire an xfer
> > + *
> > + * @minfo: Pointer to Tx/Rx Message management info based on channel type
> > + * @xfer_id: Token ID to lookup in @pending_xfers
> > + *
> > + * When a valid xfer is found for the provided @xfer_id, reference counting is
> > + * properly updated.
> > + *
> > + * Return: A valid @xfer on Success or error otherwise.
> > + */
> > +static struct scmi_xfer *
> > +scmi_xfer_acquire(struct scmi_xfers_info *minfo, u16 xfer_id)
> > +{
> > +	unsigned long flags;
> > +	struct scmi_xfer *xfer;
> > +
> > +	spin_lock_irqsave(&minfo->xfer_lock, flags);
> > +	xfer = scmi_xfer_lookup_unlocked(minfo, xfer_id);
> > +	if (!IS_ERR(xfer))
> > +		refcount_inc(&xfer->users);
> > +	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
> > +
> > +	return xfer;
> > +}
> > +
> > +/**
> > + * scmi_transfer_acquire  -  Lookup for an existing xfer or freshly allocate a
> > + * new one depending on the type of the message
> > + *
> > + * @cinfo: A reference to the channel descriptor.
> > + * @msg_hdr: A pointer to the message header to lookup.
> > + * @xfer: A reference to the pre-existent or freshly allocated xfer
> > + *	  associated with the provided *msg_hdr.
> > + *
> > + * This function can be used by transports supporting delegated xfers to obtain
> > + * a valid @xfer associated with the provided @msg_hdr param.
> > + *
> > + * The nature of the resulting @xfer depends on the type of message specified in
> > + * @msg_hdr:
> > + *  - for responses and delayed responses a pre-existent/pre-allocated in-flight
> > + *    xfer descriptor will be returned (properly refcounted)
> > + *  - for notifications a brand new xfer will be allocated; in this case the
> > + *    provided message header sequence number will also be mangled to match
> > + *    the token in the freshly allocated xfer: this is needed to establish a
> > + *    link between the picked xfer and the msg_hdr that will be subsequently
> > + *    passed back via usual scmi_rx_callback().
> > + *
> > + * Return: 0 if a valid xfer is returned in @xfer, error otherwise.
> > + */
> > +int scmi_transfer_acquire(struct scmi_chan_info *cinfo, u32 *msg_hdr,
> > +			  struct scmi_xfer **xfer)
> > +{
> > +	u8 msg_type;
> > +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> > +
> > +	if (!xfer || !msg_hdr || !info->desc->support_xfers_delegation)
> > +		return -EINVAL;
> > +
> > +	msg_type = MSG_XTRACT_TYPE(*msg_hdr);
> > +	switch (msg_type) {
> > +	case MSG_TYPE_COMMAND:
> > +	case MSG_TYPE_DELAYED_RESP:
> > +		/* Grab an existing xfer for xfer_id */
> > +		*xfer = scmi_xfer_acquire(&info->tx_minfo,
> > +					  MSG_XTRACT_TOKEN(*msg_hdr));
> > +		break;
> > +	case MSG_TYPE_NOTIFICATION:
> > +		/* Get a brand new RX xfer */
> > +		*xfer = scmi_xfer_get(cinfo->handle, &info->rx_minfo);
> > +		if (!IS_ERR(*xfer)) {
> > +			/* Save original msg_hdr and fix sequence number */
> > +			(*xfer)->hdr.saved_hdr = *msg_hdr;
> 
> The saved header isn't used anywhere.
> 

Yes, in fact it's just stored just in case in the future for some reason
the spec would change somehow to require that the original msg sequence
number of a notification has to be used.

> > +			*msg_hdr &= ~MSG_TOKEN_ID_MASK;
> > +			*msg_hdr |= FIELD_PREP(MSG_TOKEN_ID_MASK,
> > +					       (*xfer)->hdr.seq);
> 
> This will invalidate the token set by the platform in
> scmi_dump_header_dbg(). Maybe it would have been more elegant to introduce a
> dedicated hash table key field?
> 

Not sure to have understood this thing about invalidation.

Anyway, I removed scmi_dump_header_dbg( ) in V5 since we have traces since a
while and those have anyway a dedicated xfer->transfer_id.

> > +		}
> > +		break;
> > +	default:
> > +		*xfer = ERR_PTR(-EINVAL);
> > +		break;
> > +	}
> > +
> > +	if (IS_ERR(*xfer)) {
> > +		dev_err(cinfo->dev,
> > +			"Failed to acquire a valid xfer for hdr:0x%X\n",
> > +			*msg_hdr);
> > +		return PTR_ERR(*xfer);
> > +	}
> > +
> > +	/* Fix xfer->hdr.type with actual msg_hdr carried type */
> > +	unpack_scmi_header(*msg_hdr, &((*xfer)->hdr));
> > +
> > +	return 0;
> > +}
> > +
> > +/**
> > + * scmi_transfer_release  - Release an previously acquired xfer
> > + *
> > + * @cinfo: A reference to the channel descriptor.
> > + * @xfer: A reference to the xfer to release.
> > + */
> > +void scmi_transfer_release(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer)
> > +{
> > +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> > +	struct scmi_xfers_info *minfo;
> > +
> > +	if (!xfer || !info->desc->support_xfers_delegation)
> > +		return;
> > +
> > +	if (xfer->hdr.type == MSG_TYPE_NOTIFICATION)
> > +		minfo = &info->rx_minfo;
> > +	else
> > +		minfo = &info->tx_minfo;
> > +
> > +	__scmi_xfer_put(minfo, xfer);
> > +}
> > +
> >  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
> >  {
> >  	struct scmi_xfer *xfer;
> > @@ -441,7 +559,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
> >  	ktime_t ts;
> >  	ts = ktime_get_boottime();
> > -	xfer = scmi_xfer_get(cinfo->handle, minfo);
> > +
> > +	if (!info->desc->support_xfers_delegation)
> > +		xfer = scmi_xfer_get(cinfo->handle, minfo);
> > +	else
> > +		xfer = scmi_xfer_acquire(minfo, MSG_XTRACT_TOKEN(msg_hdr));
> >  	if (IS_ERR(xfer)) {
> >  		dev_err(dev, "failed to get free message slot (%ld)\n",
> >  			PTR_ERR(xfer));
> > @@ -449,8 +571,11 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
> >  		return;
> >  	}
> > -	unpack_scmi_header(msg_hdr, &xfer->hdr);
> >  	scmi_dump_header_dbg(dev, &xfer->hdr);
> > +
> > +	if (!info->desc->support_xfers_delegation)
> > +		unpack_scmi_header(msg_hdr, &xfer->hdr);
> > +
> 
> Why dump the header before unpacking?
> 

Right, but as I said above I drop scmi_dump_header_dbg() in V5.

Thanks,
Cristian


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 06/16] firmware: arm_scmi, smccc, mailbox: Make shmem based transports optional
  2021-07-01  8:42     ` Peter Hilber
@ 2021-07-01 10:27       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:27 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Thu, Jul 01, 2021 at 10:42:56AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> > From: Igor Skalkin <igor.skalkin@opensynergy.com>
> > 
> > Upon adding the virtio transport in this patch series, SCMI will also
> > work without shared memory based transports. Also, the mailbox transport
> > may not be needed if the smc transport is used.
> > 
> > - Compile shmem.c only if a shmem based transport is available.
> > 
> > - Remove hard dependency of SCMI on mailbox.
> 
> The hard dependency has now already been removed with
> 
>   c05b07963e96 ("firmware: arm_scmi: Add SMCCC discovery dependency in")
> 

I deeply reviewed transport Kconfig approach in V5 in general, making
all transport configurable (y/n), so this patch will be adapted to the new
Kconfig layout annd basically just provide the HAVE_SHMEM feature.

Thanks,
Cristian

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 06/16] firmware: arm_scmi, smccc, mailbox: Make shmem based transports optional
@ 2021-07-01 10:27       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:27 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Thu, Jul 01, 2021 at 10:42:56AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> > From: Igor Skalkin <igor.skalkin@opensynergy.com>
> > 
> > Upon adding the virtio transport in this patch series, SCMI will also
> > work without shared memory based transports. Also, the mailbox transport
> > may not be needed if the smc transport is used.
> > 
> > - Compile shmem.c only if a shmem based transport is available.
> > 
> > - Remove hard dependency of SCMI on mailbox.
> 
> The hard dependency has now already been removed with
> 
>   c05b07963e96 ("firmware: arm_scmi: Add SMCCC discovery dependency in")
> 

I deeply reviewed transport Kconfig approach in V5 in general, making
all transport configurable (y/n), so this patch will be adapted to the new
Kconfig layout annd basically just provide the HAVE_SHMEM feature.

Thanks,
Cristian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
  2021-07-01  8:43     ` Peter Hilber
@ 2021-07-01 10:31       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:31 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy,
	Rob Herring, devicetree

Hi,

On Thu, Jul 01, 2021 at 10:43:02AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> > From: Igor Skalkin <igor.skalkin@opensynergy.com>
> > 
> > Document the properties for arm,scmi-virtio compatible nodes.
> > The backing virtio SCMI device is described in patch [1].
> > 
> > While doing that, make shmem property required only for pre-existing
> > mailbox and smc transports, since virtio-scmi does not need it.
> > 
> > [1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html
> > 
> > CC: Rob Herring <robh+dt@kernel.org>
> > CC: devicetree@vger.kernel.org
> > Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> > [ Peter: Adapted patch for submission to upstream. ]
> > Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> > [ Cristian: converted to yaml format, moved shmen required property. ]
> > Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> > v3 --> V4
> > - convertd to YAML
> > - make shmem required only for pre-existing mailbox and smc transport
> > - updated VirtIO specification patch message reference
> > - dropped virtio-mmio SCMI device example since really not pertinent to
> >    virtio-scmi dt bindings transport: it is not even referenced in SCMI
> >    virtio DT node since they are enumerated by VirtIO subsystem and there
> >    could be PCI based SCMI devices anyway.
> > ---
> >   Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
> >   1 file changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > index cebf6ffe70d5..5c4c6782e052 100644
> > --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > @@ -34,6 +34,10 @@ properties:
> >         - description: SCMI compliant firmware with ARM SMC/HVC transport
> >           items:
> >             - const: arm,scmi-smc
> > +      - description: SCMI compliant firmware with SCMI Virtio transport.
> > +                     The virtio transport only supports a single device.
> > +        items:
> > +          - const: arm,scmi-virtio
> >     interrupts:
> >       description:
> > @@ -172,6 +176,7 @@ patternProperties:
> >         Each sub-node represents a protocol supported. If the platform
> >         supports a dedicated communication channel for a particular protocol,
> >         then the corresponding transport properties must be present.
> > +      The virtio transport does not support a dedicated communication channel.
> >       properties:
> >         reg:
> > @@ -195,7 +200,6 @@ patternProperties:
> >   required:
> >     - compatible
> > -  - shmem
> >   if:
> >     properties:
> > @@ -209,6 +213,7 @@ then:
> >     required:
> >       - mboxes
> > +    - shmem
> >   else:
> >     if:
> > @@ -219,6 +224,7 @@ else:
> >     then:
> >       required:
> >         - arm,smc-id
> > +      - shmem
> >   examples:
> >     - |
> > 
> 
> Maybe a minimal example for arm,scmi-virtio could be added, such as below:
> 
> diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> index 5c4c6782e052..576faf970c1b 100644
> --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> @@ -344,4 +344,19 @@ examples:
>          };
>      };
> 
> +  - |
> +    firmware {
> +        scmi {
> +            compatible = "arm,scmi-virtio";
> +
> +            #address-cells = <1>;
> +            #size-cells = <0>;
> +
> +            scmi_devpd2: protocol@11 {
> +                reg = <0x11>;
> +                #power-domain-cells = <1>;
> +            };
> +        };
> +    };
> +

Not sure about this, I explicitly removed the arm,scmi-virtio example
because with the current bindings it won't really exemplify anything
really specific to the virtio transport if not the usage of the compatible
string, and moreover the mmio case is just a case, there could be PCI
based virtio SCMI device that would have even less to show here as an
example.

Thanks,
Cristian

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI
@ 2021-07-01 10:31       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:31 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy,
	Rob Herring, devicetree

Hi,

On Thu, Jul 01, 2021 at 10:43:02AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> > From: Igor Skalkin <igor.skalkin@opensynergy.com>
> > 
> > Document the properties for arm,scmi-virtio compatible nodes.
> > The backing virtio SCMI device is described in patch [1].
> > 
> > While doing that, make shmem property required only for pre-existing
> > mailbox and smc transports, since virtio-scmi does not need it.
> > 
> > [1] https://lists.oasis-open.org/archives/virtio-comment/202102/msg00018.html
> > 
> > CC: Rob Herring <robh+dt@kernel.org>
> > CC: devicetree@vger.kernel.org
> > Signed-off-by: Igor Skalkin <igor.skalkin@opensynergy.com>
> > [ Peter: Adapted patch for submission to upstream. ]
> > Co-developed-by: Peter Hilber <peter.hilber@opensynergy.com>
> > Signed-off-by: Peter Hilber <peter.hilber@opensynergy.com>
> > [ Cristian: converted to yaml format, moved shmen required property. ]
> > Co-developed-by: Cristian Marussi <cristian.marussi@arm.com>
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> > v3 --> V4
> > - convertd to YAML
> > - make shmem required only for pre-existing mailbox and smc transport
> > - updated VirtIO specification patch message reference
> > - dropped virtio-mmio SCMI device example since really not pertinent to
> >    virtio-scmi dt bindings transport: it is not even referenced in SCMI
> >    virtio DT node since they are enumerated by VirtIO subsystem and there
> >    could be PCI based SCMI devices anyway.
> > ---
> >   Documentation/devicetree/bindings/firmware/arm,scmi.yaml | 8 +++++++-
> >   1 file changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > index cebf6ffe70d5..5c4c6782e052 100644
> > --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > @@ -34,6 +34,10 @@ properties:
> >         - description: SCMI compliant firmware with ARM SMC/HVC transport
> >           items:
> >             - const: arm,scmi-smc
> > +      - description: SCMI compliant firmware with SCMI Virtio transport.
> > +                     The virtio transport only supports a single device.
> > +        items:
> > +          - const: arm,scmi-virtio
> >     interrupts:
> >       description:
> > @@ -172,6 +176,7 @@ patternProperties:
> >         Each sub-node represents a protocol supported. If the platform
> >         supports a dedicated communication channel for a particular protocol,
> >         then the corresponding transport properties must be present.
> > +      The virtio transport does not support a dedicated communication channel.
> >       properties:
> >         reg:
> > @@ -195,7 +200,6 @@ patternProperties:
> >   required:
> >     - compatible
> > -  - shmem
> >   if:
> >     properties:
> > @@ -209,6 +213,7 @@ then:
> >     required:
> >       - mboxes
> > +    - shmem
> >   else:
> >     if:
> > @@ -219,6 +224,7 @@ else:
> >     then:
> >       required:
> >         - arm,smc-id
> > +      - shmem
> >   examples:
> >     - |
> > 
> 
> Maybe a minimal example for arm,scmi-virtio could be added, such as below:
> 
> diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> index 5c4c6782e052..576faf970c1b 100644
> --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> @@ -344,4 +344,19 @@ examples:
>          };
>      };
> 
> +  - |
> +    firmware {
> +        scmi {
> +            compatible = "arm,scmi-virtio";
> +
> +            #address-cells = <1>;
> +            #size-cells = <0>;
> +
> +            scmi_devpd2: protocol@11 {
> +                reg = <0x11>;
> +                #power-domain-cells = <1>;
> +            };
> +        };
> +    };
> +

Not sure about this, I explicitly removed the arm,scmi-virtio example
because with the current bindings it won't really exemplify anything
really specific to the virtio transport if not the usage of the compatible
string, and moreover the mmio case is just a case, there could be PCI
based virtio SCMI device that would have even less to show here as an
example.

Thanks,
Cristian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
  2021-07-01  8:43     ` Peter Hilber
@ 2021-07-01 10:34       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:34 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Thu, Jul 01, 2021 at 10:43:07AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> 
> <snip>
> 
> > +static struct virtio_driver virtio_scmi_driver = {
> > +	.driver.name = "scmi-virtio",
> > +	.driver.owner = THIS_MODULE,
> > +	.feature_table = features,
> > +	.feature_table_size = ARRAY_SIZE(features),
> > +	.id_table = id_table,
> > +	.probe = scmi_vio_probe,
> > +	.remove = scmi_vio_remove,
> > +};
> > +
> 
> It might be good to also check for the VIRTIO_F_VERSION_1 feature bit in the
> optional .validate op (not yet implemented above), as some other devices do
> (quoting virtio-snd in the following):
> 

Ah, thanks for pointing that out, I'll do because indeed in my hackish
emulation test-setup I forgot to emulate that bit and everything fall
flat.

> > /**
> >  * virtsnd_validate() - Validate if the device can be started.
> >  * @vdev: VirtIO parent device.
> >  *
> >  * Context: Any context.
> >  * Return: 0 on success, -EINVAL on failure.
> >  */
> > static int virtsnd_validate(struct virtio_device *vdev)
> > {
> 
> <snip>
> 
> > 
> > 	if (!virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
> > 		dev_err(&vdev->dev,
> > 			"device does not comply with spec version 1.x\n");
> > 		return -EINVAL;
> > 	}
> > 
> 
> <snip>
> 
> > 
> > static struct virtio_driver virtsnd_driver = {
> > 	.driver.name = KBUILD_MODNAME,
> > 	.driver.owner = THIS_MODULE,
> > 	.id_table = id_table,
> > 	.validate = virtsnd_validate,
> 
> (end of virtio-snd quote)

Thanks,
Cristian


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport
@ 2021-07-01 10:34       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 10:34 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi,

On Thu, Jul 01, 2021 at 10:43:07AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> 
> <snip>
> 
> > +static struct virtio_driver virtio_scmi_driver = {
> > +	.driver.name = "scmi-virtio",
> > +	.driver.owner = THIS_MODULE,
> > +	.feature_table = features,
> > +	.feature_table_size = ARRAY_SIZE(features),
> > +	.id_table = id_table,
> > +	.probe = scmi_vio_probe,
> > +	.remove = scmi_vio_remove,
> > +};
> > +
> 
> It might be good to also check for the VIRTIO_F_VERSION_1 feature bit in the
> optional .validate op (not yet implemented above), as some other devices do
> (quoting virtio-snd in the following):
> 

Ah, thanks for pointing that out, I'll do because indeed in my hackish
emulation test-setup I forgot to emulate that bit and everything fall
flat.

> > /**
> >  * virtsnd_validate() - Validate if the device can be started.
> >  * @vdev: VirtIO parent device.
> >  *
> >  * Context: Any context.
> >  * Return: 0 on success, -EINVAL on failure.
> >  */
> > static int virtsnd_validate(struct virtio_device *vdev)
> > {
> 
> <snip>
> 
> > 
> > 	if (!virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
> > 		dev_err(&vdev->dev,
> > 			"device does not comply with spec version 1.x\n");
> > 		return -EINVAL;
> > 	}
> > 
> 
> <snip>
> 
> > 
> > static struct virtio_driver virtsnd_driver = {
> > 	.driver.name = KBUILD_MODNAME,
> > 	.driver.owner = THIS_MODULE,
> > 	.id_table = id_table,
> > 	.validate = virtsnd_validate,
> 
> (end of virtio-snd quote)

Thanks,
Cristian


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 15/16] [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers
  2021-07-01  8:43     ` Peter Hilber
@ 2021-07-01 11:26       ` Cristian Marussi
  -1 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 11:26 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi Peter,

I agree with your remarks down below about raw_payload helpers, and
especially I did not like the need for the memmove and the awkard
rx_raw_len that I introduced, but in order to fix that I should have
changed deeply the way the core uses rx.buf since I cannot just use
the notion of the msg type here in the transport to discriminate between
command and notification. (because we still do not want to peek inside
the message structure from within this virtio transport)

But then I realized that I could have get rid of all of this raw_payload
machinery and simplify further, because now I have also management of
concurrent and out of order responses/delayed_responses and the xfer
obtained here is assured to be valid and sane.

So basically in V5 I get rid of these changes of mine about raw_payloads
and instead go like this in scmi_process_vqueue_input():

	ret = scmi_transfer_acquire(vioch->cinfo, &msg_hdr, &xfer);
	if (ret)
		scmi_finalize_message(vioch, msg)

	xfer->priv = msg;

	scmi_transfer_release(vioch->cinfo, xfer);

	if (vioch->is_rx || !xfer->hdr.poll_completion)
		scmi_rx_callback(vioch->cinfo, msg_hdr);

	scmi_finalize_message(vioch, msg);

and then let your original fetch_response/notification proceed as usual
picking the msg from xfer->priv instead.

Because, anyway, the xfer I've got is ensured to be valid and the new V5
serialization/ot-of-order logic in the core takes care of the case of
concurrent time-outs or out of order DRESPs reception (maybe this will be
more apparent looking at the code in V5...)

As a side consideration, I also dropped support for polling mode as a whole
in V5, since it seemed to not make much sense for virtio transport, where
there's always some sort of virtio core interrupt that triggers the complete
callbacks and this logic. (while polling support is meant in the core to
address the case in which a transport has no completion IRQ so it
cannnot ever call into scmi_rx_callback(), and such useless support here
would have stopped me from this further simplification.)

On Thu, Jul 01, 2021 at 10:43:12AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> > Draft changes to virtio-scmi to use new support for core delegated xfers
> > in an attempt to simplify the interactions between virtio-scmi transport
> > and the SCMI core transport layer.
> > 
> 
> These changes seem to make xfers delegation mandatory for message-passing
> transports, so that might be documented.

Not sure about this, since even it is true for virtio but it does not seem
to be necessarily true for any future message-passing based transport, imagining
as an example a message based transport with a queue of just one in flight message,
so with no state to keep (in V5 with raw_payload helpers dropped...), or a transport
that do use message-passing but it keeps its own states in a different way.
Anyway, I'll double check if these assumptions of mine hold true.

> 
> > TODO:
> >   - Polling is still not supported.
> >   - Probe/remove sequence still to be reviewed.
> >   - Concurrent or inverted reception of related responses and delayed
> >     responses is still not addressed.
> >     (it will be addressed in the SCMI core anyway most probably)
> > 
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> >   drivers/firmware/arm_scmi/common.h |   5 +
> >   drivers/firmware/arm_scmi/msg.c    |  35 +++++
> >   drivers/firmware/arm_scmi/virtio.c | 212 +++++++++++++++--------------
> >   3 files changed, 152 insertions(+), 100 deletions(-)
> > 
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index c143a449d278..22e5532fc698 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -428,6 +428,11 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
> >   void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
> >   			    size_t max_len, struct scmi_xfer *xfer);
> > +void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
> > +			   size_t max_len, struct scmi_xfer *xfer);
> > +void msg_fetch_raw_response(struct scmi_xfer *xfer);
> > +void msg_fetch_raw_notification(struct scmi_xfer *xfer);
> > +
> >   void scmi_notification_instance_data_set(const struct scmi_handle *handle,
> >   					 void *priv);
> >   void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
> > diff --git a/drivers/firmware/arm_scmi/msg.c b/drivers/firmware/arm_scmi/msg.c
> > index 8a2d3303d281..3ed3ad0961ae 100644
> > --- a/drivers/firmware/arm_scmi/msg.c
> > +++ b/drivers/firmware/arm_scmi/msg.c
> > @@ -74,6 +74,17 @@ u32 msg_read_header(struct scmi_msg_payld *msg)
> >   	return le32_to_cpu(msg->msg_header);
> >   }
> > +void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
> > +			   size_t max_len, struct scmi_xfer *xfer)
> > +{
> > +	xfer->rx_raw_len = min_t(size_t, max_len,
> > +				 msg_len >= sizeof(*msg) ?
> > +				 msg_len - sizeof(*msg) : 0);
> > +
> > +	/* Take a copy to the rx buffer.. */
> > +	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx_raw_len);
> 
> In the usage throughout arm-scmi so far, scmi_desc.max_msg_size seems to
> consistently refer to the payload excluding the return status.
> scmi_desc.max_msg_size is the actual max_len parameter to this function.
> 
> So the logic here would reduce the maximum payload length for responses to
> (scmi_desc.max_msg_size - sizeof(return status)).
> 

Ditto.

> > +}
> > +
> >   /**
> >    * msg_fetch_response() - Fetch response SCMI payload from transport SDU.
> >    *
> > @@ -94,6 +105,25 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
> >   	memcpy(xfer->rx.buf, &msg->msg_payload[1], xfer->rx.len);
> >   }
> > +void msg_fetch_raw_response(struct scmi_xfer *xfer)
> > +{
> > +	__le32 *msg_payload = xfer->rx.buf;
> > +
> > +	if (xfer->rx_raw_len < sizeof(xfer->hdr.status)) {
> > +		xfer->rx.len = 0;
> > +		return;
> > +	}
> > +
> > +	xfer->hdr.status = le32_to_cpu(msg_payload[0]);
> > +	/*
> > +	 * rx.len has been already pre-calculated by fetch_raw
> > +	 * for the whole payload including status, so shrink it
> > +	 */
> > +	xfer->rx.len = xfer->rx_raw_len - sizeof(xfer->hdr.status);
> > +	/* Carveout status 4-byte field */
> > +	memmove(xfer->rx.buf, &msg_payload[1], xfer->rx.len);
> 
> Wouldn't it be feasible to align properly in msg_fetch_raw_payload()
> already? That could also get rid of .rx_raw_len.
> 

Ditto.

> > +}
> > +
> >   /**
> >    * msg_fetch_notification() - Fetch notification payload from transport SDU.
> >    *
> > @@ -111,3 +141,8 @@ void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
> >   	/* Take a copy to the rx buffer.. */
> >   	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx.len);
> >   }
> > +
> > +void msg_fetch_raw_notification(struct scmi_xfer *xfer)
> > +{
> > +	xfer->rx.len = xfer->rx_raw_len;
> > +}
> > diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
> > index 20972adf6dc7..4412bc590ca7 100644
> > --- a/drivers/firmware/arm_scmi/virtio.c
> > +++ b/drivers/firmware/arm_scmi/virtio.c
> > @@ -37,23 +37,24 @@
> >   /**
> >    * struct scmi_vio_channel - Transport channel information
> >    *
> > - * @lock: Protects access to all members except ready.
> > - * @ready_lock: Protects access to ready. If required, it must be taken before
> > - *              lock.
> >    * @vqueue: Associated virtqueue
> >    * @cinfo: SCMI Tx or Rx channel
> >    * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
> >    * @is_rx: Whether channel is an Rx channel
> >    * @ready: Whether transport user is ready to hear about channel
> > + * @lock: Protects access to all members except ready.
> > + * @ready_lock: Protects access to ready. If required, it must be taken before
> > + *              lock.
> >    */
> >   struct scmi_vio_channel {
> > -	spinlock_t lock;
> > -	spinlock_t ready_lock;
> >   	struct virtqueue *vqueue;
> >   	struct scmi_chan_info *cinfo;
> >   	struct list_head free_list;
> > -	u8 is_rx;
> > -	u8 ready;
> > +	bool is_rx;
> > +	bool ready;
> > +	unsigned int max_msg;
> > +	spinlock_t lock;
> > +	spinlock_t ready_lock;
> >   };
> >   /**
> > @@ -100,6 +101,73 @@ static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
> >   	return rc;
> >   }
> > +static void scmi_finalize_message(struct scmi_vio_channel *vioch,
> > +				  struct scmi_vio_msg *msg)
> > +{
> > +	unsigned long flags;
> > +
> > +	if (vioch->is_rx) {
> > +		scmi_vio_feed_vq_rx(vioch, msg);
> > +	} else {
> > +		spin_lock_irqsave(&vioch->lock, flags);
> > +		list_add(&msg->list, &vioch->free_list);
> > +		spin_unlock_irqrestore(&vioch->lock, flags);
> > +	}
> > +}
> > +
> > +static void scmi_process_vqueue_input(struct scmi_vio_channel *vioch,
> > +				      struct scmi_vio_msg *msg)
> > +{
> > +	u32 msg_hdr;
> > +	int ret;
> > +	struct scmi_xfer *xfer = NULL;
> > +
> > +	msg_hdr = msg_read_header(msg->input);
> > +	/*
> > +	 * Acquire from the core transport layer a currently valid xfer
> > +	 * descriptor associated to the received msg_hdr: this could be a
> > +	 * previously allocated xfer for responses and delayed responses to
> > +	 * in-flight commands, or a freshly allocated new xfer for a just
> > +	 * received notification.
> > +	 *
> > +	 * In case of responses and delayed_responses the acquired xfer, at
> > +	 * the time scmi_transfer_acquire() succcessfully returns is guaranteed
> > +	 * to be still associated with a valid (not timed-out nor stale)
> > +	 * descriptor and proper refcounting is kept in the core along this xfer
> > +	 * so that should the core time out the xfer concurrently to this receive
> > +	 * path the xfer will be properly deallocated only once the last user is
> > +	 * done with it. (and this code path will terminate normally even though
> > +	 * all the processing related to the timed out xfer will be discarded).
> > +	 */
> 
> This comment would better fit to scmi_transfer_acquire().
> 

Yes indeed I strip down this comment in V5.

> > +	ret = scmi_transfer_acquire(vioch->cinfo, &msg_hdr, &xfer);
> > +	if (ret) {
> > +		dev_err(vioch->cinfo->dev,
> > +			"Cannot find matching xfer for hdr:0x%X\n", msg_hdr);
> > +		scmi_finalize_message(vioch, msg);
> > +		return;
> > +	}
> > +
> > +	dev_dbg(vioch->cinfo->dev,
> > +		"VQUEUE[%d] - INPUT MSG_RX_LEN:%d - HDR:0x%X  TYPE:%d  XFER_ID:%d  XFER:%px\n",
> > +		vioch->vqueue->index, msg->rx_len, msg_hdr, xfer->hdr.type,
> > +		xfer->hdr.seq, xfer);
> > +
> > +	msg_fetch_raw_payload(msg->input, msg->rx_len,
> > +			      scmi_virtio_desc.max_msg_size, xfer); > +
> > +	/* Drop processed virtio message anyway */
> > +	scmi_finalize_message(vioch, msg);
> > +
> > +	if (vioch->is_rx || !xfer->hdr.poll_completion)
> > +		scmi_rx_callback(vioch->cinfo, msg_hdr);
> > +	else
> > +		dev_warn(vioch->cinfo->dev,
> > +			 "Polling mode NOT supported. Dropped hdr:0X%X\n",
> > +			 msg_hdr);
> > +
> > +	scmi_transfer_release(vioch->cinfo, xfer);
> > +}
> > +
> >   static void scmi_vio_complete_cb(struct virtqueue *vqueue)
> >   {
> >   	unsigned long ready_flags;
> > @@ -138,15 +206,9 @@ static void scmi_vio_complete_cb(struct virtqueue *vqueue)
> >   		if (msg) {
> >   			msg->rx_len = length;
> > -
> > -			/*
> > -			 * Hold the ready_lock during the callback to avoid
> > -			 * races when the arm-scmi driver is unbinding while
> > -			 * the virtio device is not quiesced yet.
> > -			 */
> > -			scmi_rx_callback(vioch->cinfo,
> > -					 msg_read_header(msg->input), msg);
> > +			scmi_process_vqueue_input(vioch, msg);
> >   		}
> > +
> >   		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> >   	}
> > @@ -163,27 +225,11 @@ static vq_callback_t *scmi_vio_complete_callbacks[] = {
> >   	scmi_vio_complete_cb
> >   };
> > -static unsigned int virtio_get_max_msg(bool tx,
> > -				       struct scmi_chan_info *base_cinfo)
> > +static unsigned int virtio_get_max_msg(struct scmi_chan_info *base_cinfo)
> >   {
> >   	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
> > -	unsigned int ret;
> > -	ret = virtqueue_get_vring_size(vioch->vqueue);
> > -
> > -	/* Tx messages need multiple descriptors. */
> > -	if (tx)
> > -		ret /= DESCRIPTORS_PER_TX_MSG;
> > -
> > -	if (ret > MSG_TOKEN_MAX) {
> > -		dev_info_once(
> > -			base_cinfo->dev,
> > -			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
> > -			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
> > -		ret = MSG_TOKEN_MAX;
> > -	}
> > -
> > -	return ret;
> > +	return vioch->max_msg;
> >   }
> >   static int scmi_vio_match_any_dev(struct device *dev, const void *data)
> > @@ -195,13 +241,14 @@ static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
> >   static int virtio_link_supplier(struct device *dev)
> >   {
> > -	struct device *vdev = driver_find_device(
> > -		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
> > +	struct device *vdev;
> > +
> > +	vdev = driver_find_device(&virtio_scmi_driver.driver,
> > +				  NULL, NULL, scmi_vio_match_any_dev);
> >   	if (!vdev) {
> > -		dev_notice_once(
> > -			dev,
> > -			"Deferring probe after not finding a bound scmi-virtio device\n");
> > +		dev_notice_once(dev,
> > +				"Deferring probe after not finding a bound scmi-virtio device\n");
> >   		return -EPROBE_DEFER;
> >   	}
> > @@ -245,12 +292,8 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> >   	struct virtio_device *vdev;
> >   	struct scmi_vio_channel *vioch;
> >   	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
> > -	int max_msg;
> >   	int i;
> > -	if (!virtio_chan_available(dev, index))
> > -		return -ENODEV;
> > -
> >   	vdev = scmi_get_transport_info(dev);
> >   	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
> > @@ -259,9 +302,7 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> >   	vioch->cinfo = cinfo;
> >   	spin_unlock_irqrestore(&vioch->lock, flags);
> > -	max_msg = virtio_get_max_msg(tx, cinfo);
> > -
> > -	for (i = 0; i < max_msg; i++) {
> > +	for (i = 0; i < vioch->max_msg; i++) {
> >   		struct scmi_vio_msg *msg;
> >   		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
> > @@ -322,13 +363,6 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
> >   	int rc;
> >   	struct scmi_vio_msg *msg;
> > -	/*
> > -	 * TODO: For now, we don't support polling. But it should not be
> > -	 * difficult to add support.
> > -	 */
> > -	if (xfer->hdr.poll_completion)
> > -		return -EINVAL;
> > -
> >   	spin_lock_irqsave(&vioch->lock, flags);
> >   	if (list_empty(&vioch->free_list)) {
> > @@ -351,6 +385,11 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
> >   			     "%s() failed to add to virtqueue (%d)\n", __func__,
> >   			     rc);
> >   	} else {
> > +		dev_dbg(vioch->cinfo->dev,
> > +			"VQUEUE[%d] - REQUEST - PROTO:0x%X  ID:0x%X  XFER_ID:%d  XFER:%px  RX_LEN:%zd\n",
> > +		 vioch->vqueue->index, xfer->hdr.protocol_id,
> > +		 xfer->hdr.id, xfer->hdr.seq, xfer, xfer->rx.len);
> 
> Indentation appears to be inconsistent.
> 

Right.


Thanks,
Cristian


^ permalink raw reply	[flat|nested] 90+ messages in thread

* Re: [PATCH v4 15/16] [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers
@ 2021-07-01 11:26       ` Cristian Marussi
  0 siblings, 0 replies; 90+ messages in thread
From: Cristian Marussi @ 2021-07-01 11:26 UTC (permalink / raw)
  To: Peter Hilber
  Cc: linux-kernel, linux-arm-kernel, virtualization, virtio-dev,
	sudeep.holla, james.quinlan, Jonathan.Cameron, f.fainelli,
	etienne.carriere, vincent.guittot, souvik.chakravarty,
	igor.skalkin, alex.bennee, jean-philippe, mikhail.golubev,
	anton.yakovlev, Vasyl.Vavrychuk, Andriy.Tryshnivskyy

Hi Peter,

I agree with your remarks down below about raw_payload helpers, and
especially I did not like the need for the memmove and the awkard
rx_raw_len that I introduced, but in order to fix that I should have
changed deeply the way the core uses rx.buf since I cannot just use
the notion of the msg type here in the transport to discriminate between
command and notification. (because we still do not want to peek inside
the message structure from within this virtio transport)

But then I realized that I could have get rid of all of this raw_payload
machinery and simplify further, because now I have also management of
concurrent and out of order responses/delayed_responses and the xfer
obtained here is assured to be valid and sane.

So basically in V5 I get rid of these changes of mine about raw_payloads
and instead go like this in scmi_process_vqueue_input():

	ret = scmi_transfer_acquire(vioch->cinfo, &msg_hdr, &xfer);
	if (ret)
		scmi_finalize_message(vioch, msg)

	xfer->priv = msg;

	scmi_transfer_release(vioch->cinfo, xfer);

	if (vioch->is_rx || !xfer->hdr.poll_completion)
		scmi_rx_callback(vioch->cinfo, msg_hdr);

	scmi_finalize_message(vioch, msg);

and then let your original fetch_response/notification proceed as usual
picking the msg from xfer->priv instead.

Because, anyway, the xfer I've got is ensured to be valid and the new V5
serialization/ot-of-order logic in the core takes care of the case of
concurrent time-outs or out of order DRESPs reception (maybe this will be
more apparent looking at the code in V5...)

As a side consideration, I also dropped support for polling mode as a whole
in V5, since it seemed to not make much sense for virtio transport, where
there's always some sort of virtio core interrupt that triggers the complete
callbacks and this logic. (while polling support is meant in the core to
address the case in which a transport has no completion IRQ so it
cannnot ever call into scmi_rx_callback(), and such useless support here
would have stopped me from this further simplification.)

On Thu, Jul 01, 2021 at 10:43:12AM +0200, Peter Hilber wrote:
> On 11.06.21 18:59, Cristian Marussi wrote:
> > Draft changes to virtio-scmi to use new support for core delegated xfers
> > in an attempt to simplify the interactions between virtio-scmi transport
> > and the SCMI core transport layer.
> > 
> 
> These changes seem to make xfers delegation mandatory for message-passing
> transports, so that might be documented.

Not sure about this, since even it is true for virtio but it does not seem
to be necessarily true for any future message-passing based transport, imagining
as an example a message based transport with a queue of just one in flight message,
so with no state to keep (in V5 with raw_payload helpers dropped...), or a transport
that do use message-passing but it keeps its own states in a different way.
Anyway, I'll double check if these assumptions of mine hold true.

> 
> > TODO:
> >   - Polling is still not supported.
> >   - Probe/remove sequence still to be reviewed.
> >   - Concurrent or inverted reception of related responses and delayed
> >     responses is still not addressed.
> >     (it will be addressed in the SCMI core anyway most probably)
> > 
> > Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> > ---
> >   drivers/firmware/arm_scmi/common.h |   5 +
> >   drivers/firmware/arm_scmi/msg.c    |  35 +++++
> >   drivers/firmware/arm_scmi/virtio.c | 212 +++++++++++++++--------------
> >   3 files changed, 152 insertions(+), 100 deletions(-)
> > 
> > diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> > index c143a449d278..22e5532fc698 100644
> > --- a/drivers/firmware/arm_scmi/common.h
> > +++ b/drivers/firmware/arm_scmi/common.h
> > @@ -428,6 +428,11 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
> >   void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
> >   			    size_t max_len, struct scmi_xfer *xfer);
> > +void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
> > +			   size_t max_len, struct scmi_xfer *xfer);
> > +void msg_fetch_raw_response(struct scmi_xfer *xfer);
> > +void msg_fetch_raw_notification(struct scmi_xfer *xfer);
> > +
> >   void scmi_notification_instance_data_set(const struct scmi_handle *handle,
> >   					 void *priv);
> >   void *scmi_notification_instance_data_get(const struct scmi_handle *handle);
> > diff --git a/drivers/firmware/arm_scmi/msg.c b/drivers/firmware/arm_scmi/msg.c
> > index 8a2d3303d281..3ed3ad0961ae 100644
> > --- a/drivers/firmware/arm_scmi/msg.c
> > +++ b/drivers/firmware/arm_scmi/msg.c
> > @@ -74,6 +74,17 @@ u32 msg_read_header(struct scmi_msg_payld *msg)
> >   	return le32_to_cpu(msg->msg_header);
> >   }
> > +void msg_fetch_raw_payload(struct scmi_msg_payld *msg, size_t msg_len,
> > +			   size_t max_len, struct scmi_xfer *xfer)
> > +{
> > +	xfer->rx_raw_len = min_t(size_t, max_len,
> > +				 msg_len >= sizeof(*msg) ?
> > +				 msg_len - sizeof(*msg) : 0);
> > +
> > +	/* Take a copy to the rx buffer.. */
> > +	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx_raw_len);
> 
> In the usage throughout arm-scmi so far, scmi_desc.max_msg_size seems to
> consistently refer to the payload excluding the return status.
> scmi_desc.max_msg_size is the actual max_len parameter to this function.
> 
> So the logic here would reduce the maximum payload length for responses to
> (scmi_desc.max_msg_size - sizeof(return status)).
> 

Ditto.

> > +}
> > +
> >   /**
> >    * msg_fetch_response() - Fetch response SCMI payload from transport SDU.
> >    *
> > @@ -94,6 +105,25 @@ void msg_fetch_response(struct scmi_msg_payld *msg, size_t len,
> >   	memcpy(xfer->rx.buf, &msg->msg_payload[1], xfer->rx.len);
> >   }
> > +void msg_fetch_raw_response(struct scmi_xfer *xfer)
> > +{
> > +	__le32 *msg_payload = xfer->rx.buf;
> > +
> > +	if (xfer->rx_raw_len < sizeof(xfer->hdr.status)) {
> > +		xfer->rx.len = 0;
> > +		return;
> > +	}
> > +
> > +	xfer->hdr.status = le32_to_cpu(msg_payload[0]);
> > +	/*
> > +	 * rx.len has been already pre-calculated by fetch_raw
> > +	 * for the whole payload including status, so shrink it
> > +	 */
> > +	xfer->rx.len = xfer->rx_raw_len - sizeof(xfer->hdr.status);
> > +	/* Carveout status 4-byte field */
> > +	memmove(xfer->rx.buf, &msg_payload[1], xfer->rx.len);
> 
> Wouldn't it be feasible to align properly in msg_fetch_raw_payload()
> already? That could also get rid of .rx_raw_len.
> 

Ditto.

> > +}
> > +
> >   /**
> >    * msg_fetch_notification() - Fetch notification payload from transport SDU.
> >    *
> > @@ -111,3 +141,8 @@ void msg_fetch_notification(struct scmi_msg_payld *msg, size_t len,
> >   	/* Take a copy to the rx buffer.. */
> >   	memcpy(xfer->rx.buf, msg->msg_payload, xfer->rx.len);
> >   }
> > +
> > +void msg_fetch_raw_notification(struct scmi_xfer *xfer)
> > +{
> > +	xfer->rx.len = xfer->rx_raw_len;
> > +}
> > diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
> > index 20972adf6dc7..4412bc590ca7 100644
> > --- a/drivers/firmware/arm_scmi/virtio.c
> > +++ b/drivers/firmware/arm_scmi/virtio.c
> > @@ -37,23 +37,24 @@
> >   /**
> >    * struct scmi_vio_channel - Transport channel information
> >    *
> > - * @lock: Protects access to all members except ready.
> > - * @ready_lock: Protects access to ready. If required, it must be taken before
> > - *              lock.
> >    * @vqueue: Associated virtqueue
> >    * @cinfo: SCMI Tx or Rx channel
> >    * @free_list: List of unused scmi_vio_msg, maintained for Tx channels only
> >    * @is_rx: Whether channel is an Rx channel
> >    * @ready: Whether transport user is ready to hear about channel
> > + * @lock: Protects access to all members except ready.
> > + * @ready_lock: Protects access to ready. If required, it must be taken before
> > + *              lock.
> >    */
> >   struct scmi_vio_channel {
> > -	spinlock_t lock;
> > -	spinlock_t ready_lock;
> >   	struct virtqueue *vqueue;
> >   	struct scmi_chan_info *cinfo;
> >   	struct list_head free_list;
> > -	u8 is_rx;
> > -	u8 ready;
> > +	bool is_rx;
> > +	bool ready;
> > +	unsigned int max_msg;
> > +	spinlock_t lock;
> > +	spinlock_t ready_lock;
> >   };
> >   /**
> > @@ -100,6 +101,73 @@ static int scmi_vio_feed_vq_rx(struct scmi_vio_channel *vioch,
> >   	return rc;
> >   }
> > +static void scmi_finalize_message(struct scmi_vio_channel *vioch,
> > +				  struct scmi_vio_msg *msg)
> > +{
> > +	unsigned long flags;
> > +
> > +	if (vioch->is_rx) {
> > +		scmi_vio_feed_vq_rx(vioch, msg);
> > +	} else {
> > +		spin_lock_irqsave(&vioch->lock, flags);
> > +		list_add(&msg->list, &vioch->free_list);
> > +		spin_unlock_irqrestore(&vioch->lock, flags);
> > +	}
> > +}
> > +
> > +static void scmi_process_vqueue_input(struct scmi_vio_channel *vioch,
> > +				      struct scmi_vio_msg *msg)
> > +{
> > +	u32 msg_hdr;
> > +	int ret;
> > +	struct scmi_xfer *xfer = NULL;
> > +
> > +	msg_hdr = msg_read_header(msg->input);
> > +	/*
> > +	 * Acquire from the core transport layer a currently valid xfer
> > +	 * descriptor associated to the received msg_hdr: this could be a
> > +	 * previously allocated xfer for responses and delayed responses to
> > +	 * in-flight commands, or a freshly allocated new xfer for a just
> > +	 * received notification.
> > +	 *
> > +	 * In case of responses and delayed_responses the acquired xfer, at
> > +	 * the time scmi_transfer_acquire() succcessfully returns is guaranteed
> > +	 * to be still associated with a valid (not timed-out nor stale)
> > +	 * descriptor and proper refcounting is kept in the core along this xfer
> > +	 * so that should the core time out the xfer concurrently to this receive
> > +	 * path the xfer will be properly deallocated only once the last user is
> > +	 * done with it. (and this code path will terminate normally even though
> > +	 * all the processing related to the timed out xfer will be discarded).
> > +	 */
> 
> This comment would better fit to scmi_transfer_acquire().
> 

Yes indeed I strip down this comment in V5.

> > +	ret = scmi_transfer_acquire(vioch->cinfo, &msg_hdr, &xfer);
> > +	if (ret) {
> > +		dev_err(vioch->cinfo->dev,
> > +			"Cannot find matching xfer for hdr:0x%X\n", msg_hdr);
> > +		scmi_finalize_message(vioch, msg);
> > +		return;
> > +	}
> > +
> > +	dev_dbg(vioch->cinfo->dev,
> > +		"VQUEUE[%d] - INPUT MSG_RX_LEN:%d - HDR:0x%X  TYPE:%d  XFER_ID:%d  XFER:%px\n",
> > +		vioch->vqueue->index, msg->rx_len, msg_hdr, xfer->hdr.type,
> > +		xfer->hdr.seq, xfer);
> > +
> > +	msg_fetch_raw_payload(msg->input, msg->rx_len,
> > +			      scmi_virtio_desc.max_msg_size, xfer); > +
> > +	/* Drop processed virtio message anyway */
> > +	scmi_finalize_message(vioch, msg);
> > +
> > +	if (vioch->is_rx || !xfer->hdr.poll_completion)
> > +		scmi_rx_callback(vioch->cinfo, msg_hdr);
> > +	else
> > +		dev_warn(vioch->cinfo->dev,
> > +			 "Polling mode NOT supported. Dropped hdr:0X%X\n",
> > +			 msg_hdr);
> > +
> > +	scmi_transfer_release(vioch->cinfo, xfer);
> > +}
> > +
> >   static void scmi_vio_complete_cb(struct virtqueue *vqueue)
> >   {
> >   	unsigned long ready_flags;
> > @@ -138,15 +206,9 @@ static void scmi_vio_complete_cb(struct virtqueue *vqueue)
> >   		if (msg) {
> >   			msg->rx_len = length;
> > -
> > -			/*
> > -			 * Hold the ready_lock during the callback to avoid
> > -			 * races when the arm-scmi driver is unbinding while
> > -			 * the virtio device is not quiesced yet.
> > -			 */
> > -			scmi_rx_callback(vioch->cinfo,
> > -					 msg_read_header(msg->input), msg);
> > +			scmi_process_vqueue_input(vioch, msg);
> >   		}
> > +
> >   		spin_unlock_irqrestore(&vioch->ready_lock, ready_flags);
> >   	}
> > @@ -163,27 +225,11 @@ static vq_callback_t *scmi_vio_complete_callbacks[] = {
> >   	scmi_vio_complete_cb
> >   };
> > -static unsigned int virtio_get_max_msg(bool tx,
> > -				       struct scmi_chan_info *base_cinfo)
> > +static unsigned int virtio_get_max_msg(struct scmi_chan_info *base_cinfo)
> >   {
> >   	struct scmi_vio_channel *vioch = base_cinfo->transport_info;
> > -	unsigned int ret;
> > -	ret = virtqueue_get_vring_size(vioch->vqueue);
> > -
> > -	/* Tx messages need multiple descriptors. */
> > -	if (tx)
> > -		ret /= DESCRIPTORS_PER_TX_MSG;
> > -
> > -	if (ret > MSG_TOKEN_MAX) {
> > -		dev_info_once(
> > -			base_cinfo->dev,
> > -			"Only %ld messages can be pending simultaneously, while the %s virtqueue could hold %d\n",
> > -			MSG_TOKEN_MAX, tx ? "tx" : "rx", ret);
> > -		ret = MSG_TOKEN_MAX;
> > -	}
> > -
> > -	return ret;
> > +	return vioch->max_msg;
> >   }
> >   static int scmi_vio_match_any_dev(struct device *dev, const void *data)
> > @@ -195,13 +241,14 @@ static struct virtio_driver virtio_scmi_driver; /* Forward declaration */
> >   static int virtio_link_supplier(struct device *dev)
> >   {
> > -	struct device *vdev = driver_find_device(
> > -		&virtio_scmi_driver.driver, NULL, NULL, scmi_vio_match_any_dev);
> > +	struct device *vdev;
> > +
> > +	vdev = driver_find_device(&virtio_scmi_driver.driver,
> > +				  NULL, NULL, scmi_vio_match_any_dev);
> >   	if (!vdev) {
> > -		dev_notice_once(
> > -			dev,
> > -			"Deferring probe after not finding a bound scmi-virtio device\n");
> > +		dev_notice_once(dev,
> > +				"Deferring probe after not finding a bound scmi-virtio device\n");
> >   		return -EPROBE_DEFER;
> >   	}
> > @@ -245,12 +292,8 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> >   	struct virtio_device *vdev;
> >   	struct scmi_vio_channel *vioch;
> >   	int index = tx ? VIRTIO_SCMI_VQ_TX : VIRTIO_SCMI_VQ_RX;
> > -	int max_msg;
> >   	int i;
> > -	if (!virtio_chan_available(dev, index))
> > -		return -ENODEV;
> > -
> >   	vdev = scmi_get_transport_info(dev);
> >   	vioch = &((struct scmi_vio_channel *)vdev->priv)[index];
> > @@ -259,9 +302,7 @@ static int virtio_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> >   	vioch->cinfo = cinfo;
> >   	spin_unlock_irqrestore(&vioch->lock, flags);
> > -	max_msg = virtio_get_max_msg(tx, cinfo);
> > -
> > -	for (i = 0; i < max_msg; i++) {
> > +	for (i = 0; i < vioch->max_msg; i++) {
> >   		struct scmi_vio_msg *msg;
> >   		msg = devm_kzalloc(cinfo->dev, sizeof(*msg), GFP_KERNEL);
> > @@ -322,13 +363,6 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
> >   	int rc;
> >   	struct scmi_vio_msg *msg;
> > -	/*
> > -	 * TODO: For now, we don't support polling. But it should not be
> > -	 * difficult to add support.
> > -	 */
> > -	if (xfer->hdr.poll_completion)
> > -		return -EINVAL;
> > -
> >   	spin_lock_irqsave(&vioch->lock, flags);
> >   	if (list_empty(&vioch->free_list)) {
> > @@ -351,6 +385,11 @@ static int virtio_send_message(struct scmi_chan_info *cinfo,
> >   			     "%s() failed to add to virtqueue (%d)\n", __func__,
> >   			     rc);
> >   	} else {
> > +		dev_dbg(vioch->cinfo->dev,
> > +			"VQUEUE[%d] - REQUEST - PROTO:0x%X  ID:0x%X  XFER_ID:%d  XFER:%px  RX_LEN:%zd\n",
> > +		 vioch->vqueue->index, xfer->hdr.protocol_id,
> > +		 xfer->hdr.id, xfer->hdr.seq, xfer, xfer->rx.len);
> 
> Indentation appears to be inconsistent.
> 

Right.


Thanks,
Cristian


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 90+ messages in thread

end of thread, other threads:[~2021-07-01 11:29 UTC | newest]

Thread overview: 90+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-11 16:59 [PATCH v4 00/16] Introduce SCMI VirtIO transport Cristian Marussi
2021-06-11 16:59 ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 01/16] firmware: arm_scmi: Fix max pending messages boundary check Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-07-01  8:42   ` Peter Hilber
2021-07-01  8:42     ` Peter Hilber
2021-07-01 10:04     ` Cristian Marussi
2021-07-01 10:04       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 02/16] firmware: arm_scmi: Add support for type handling in common functions Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 03/16] firmware: arm_scmi: Add transport optional init/exit support Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-14 13:29   ` Jonathan Cameron
2021-06-14 13:29     ` Jonathan Cameron
2021-06-16  9:04     ` Cristian Marussi
2021-06-16  9:04       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-14 13:53   ` Jonathan Cameron
2021-06-14 13:53     ` Jonathan Cameron
2021-06-16  9:11     ` Cristian Marussi
2021-06-16  9:11       ` Cristian Marussi
2021-07-01  8:42   ` Peter Hilber
2021-07-01  8:42     ` Peter Hilber
2021-07-01 10:16     ` Cristian Marussi
2021-07-01 10:16       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 05/16] firmware: arm_scmi: Introduce delegated xfers support Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-07-01  8:42   ` Peter Hilber
2021-07-01  8:42     ` Peter Hilber
2021-07-01 10:24     ` Cristian Marussi
2021-07-01 10:24       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 06/16] firmware: arm_scmi, smccc, mailbox: Make shmem based transports optional Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-07-01  8:42   ` Peter Hilber
2021-07-01  8:42     ` Peter Hilber
2021-07-01 10:27     ` Cristian Marussi
2021-07-01 10:27       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 07/16] firmware: arm_scmi: Add op to override max message # Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-14 14:04   ` Jonathan Cameron
2021-06-14 14:04     ` Jonathan Cameron
2021-06-16  9:13     ` Cristian Marussi
2021-06-16  9:13       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 08/16] [RFC][REWORK] " Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 09/16] firmware: arm_scmi: Add optional link_supplier() transport op Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 10/16] firmware: arm_scmi: Add per-device transport private info Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 11/16] firmware: arm_scmi: Add is_scmi_protocol_device() Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 12/16] firmware: arm_scmi: Add message passing abstractions for transports Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-14 14:10   ` Jonathan Cameron
2021-06-14 14:10     ` Jonathan Cameron
2021-06-16  9:14     ` Cristian Marussi
2021-06-16  9:14       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 13/16] dt-bindings: arm: Add virtio transport for SCMI Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-24 19:22   ` Rob Herring
2021-06-24 19:22     ` Rob Herring
2021-06-24 19:22     ` Rob Herring
2021-07-01  8:43   ` Peter Hilber
2021-07-01  8:43     ` Peter Hilber
2021-07-01 10:31     ` Cristian Marussi
2021-07-01 10:31       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 14/16] firmware: arm_scmi: Add virtio transport Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-14 14:23   ` Jonathan Cameron
2021-06-14 14:23     ` Jonathan Cameron
2021-06-16 10:18     ` Cristian Marussi
2021-06-16 10:18       ` Cristian Marussi
2021-07-01  8:43   ` Peter Hilber
2021-07-01  8:43     ` Peter Hilber
2021-07-01 10:34     ` Cristian Marussi
2021-07-01 10:34       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 15/16] [RFC][REWORK] firmware: arm_scmi: make virtio-scmi use delegated xfers Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-07-01  8:43   ` Peter Hilber
2021-07-01  8:43     ` Peter Hilber
2021-07-01 11:26     ` Cristian Marussi
2021-07-01 11:26       ` Cristian Marussi
2021-06-11 16:59 ` [PATCH v4 16/16] firmware: arm_scmi: Add polling mode to virtio transport Cristian Marussi
2021-06-11 16:59   ` Cristian Marussi
2021-06-14 11:43 ` [PATCH v4 00/16] Introduce SCMI VirtIO transport Christoph Hellwig
2021-06-14 11:43   ` Christoph Hellwig
2021-06-14 11:43   ` Christoph Hellwig
2021-06-14 14:03   ` Cristian Marussi
2021-06-14 14:03     ` Cristian Marussi

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.