[RFC PATCH 00/11] SCMI Notifications Support

[RFC PATCH 00/11] SCMI Notifications Support

[Cristian Marussi]

Hi all,

this series wants to introduce SCMI Notification Support, built on top of
the standard Kernel notification chain subsystem.

At initialization time each SCMI Protocol takes care to register with the
new SCMI notification core the set of its own events which it intends to
support.

Using a possibly proposed API in scmi_protocol.h (not finalized though,
NO EXPORTs_) a Kernel user can register its own notifier_t callback
(so via a notifier_block as usual) against any registered event as
identified by the tuple:

		(proto_id, event_id, src_id)

where src_id represents a generic source identifier which is protocol
dependent like domain_id, performance_id, sensor_id and so forth.
(users can anyway do NOT provide any src_id, and subscribe instead to ALL
 the existing (if any) src_id sources for that proto_id/evt_id combination)

Each of the above tuple-specified event will be served on its own dedicated
blocking notification chain; given the great number of possible events, and
the extensibility of the SCMI Protocol itself, all the underlying machinery
of notifications chains it is dynamically created and destroyed at run-time
on-demand, depending on the number of effective registered users: no users
no allocations at all.

Upon a notification delivery all the users' registered notifier_t callbacks
will be in turn invoked and fed with the event_id as @action param and a
generated custom per-event report struct as @data param.
Each event report carries also a timestamp, gathered when the notification
message had first entered the richOS world in the SCMI rx ISR.

The final step of notification delivery via users' callback invocation is
instead delegated to a pool of deferred workers (Kernel cmwq): each
SCMI protocol has its own dedicated worker and queue to push events from
the rx ISR to the worker.

Additionally, since the original Kernel notification chain mechanism does
not stop users' registered callbacks from interacting with the notification
delivery itself (like cutting the chain with a NOTIFY_STOP or mangling the
*data report struct along the way), it was thought that such behaviour was
generally undesirable for a notification delivery service, and a possible
'anti-tampering' solution is proposed in patch [02/11]

("firmware: arm_scmi: Add notifications anti-tampering")

but maybe this attempt is not worth the cost of the additional complication
or simply deemed not needed (being anyway in Kernel land), in such a case
the above anti-tampering commit can simply be dropped from the series.

The series is marked as RFC mainly because:

- the API as said is tentative and not EXPORTed; currently consisting of a
  per-protocol interface like:
 	 scmi_register_<proto>_event_notifier(evt_id, *src_id, *nb)

  but it could be simplified to one single simpler generic one like:

 	 scmi_register_event_notifier(proto_id, evt_id, *src_id, *nb)

  It's open for discussion.

- no Event priorization has been considered: each protocol has its own
  queue and deferred worker instance, so as to avoid that one protocol
  flooding can overrun a single queue and influence other protocols'
  notifications' delivery.
  But that's it, all the workers are unbound, low_pri cmwq workers.

  Should we enforce some sort of built-in prio amongst the events ?
  Should this priority instead be compile time configurable ?

  Again, open for discussion.

- no configuration is possible: it can be imagined that on a real platform
  events' priority (if any) and events queues' depth could be something
  somehow compile-time configurable, but this is not addressed by this
  series at all.

Based on scmi-next [1], on top of:

commit 257d0e20ec4f ("include: trace: Add SCMI header with trace events")

This series has been tested on JUNO with an experimental firmware only
supporting Perf Notifications.

Any thoughts ?

Thanks

Cristian
----

[1] git://git.kernel.org/pub/scm/linux/kernel/git/sudeep.holla/linux.git

Cristian Marussi (8):
  firmware: arm_scmi: Add core notifications support
  firmware: arm_scmi: Add notifications anti-tampering
  firmware: arm_scmi: Enable core notifications
  firmware: arm_scmi: Add Power notifications support
  firmware: arm_scmi: Add Perf notifications support
  firmware: arm_scmi: Add Sensor notifications support
  firmware: arm_scmi: Add Reset notifications support
  firmware: arm_scmi: Add Base notifications support

Sudeep Holla (3):
  firmware: arm_scmi: Add receive buffer support for notifications
  firmware: arm_scmi: Update protocol commands and notification list
  firmware: arm_scmi: Add support for notifications message processing

 drivers/firmware/arm_scmi/Makefile  |    2 +-
 drivers/firmware/arm_scmi/base.c    |  132 ++++
 drivers/firmware/arm_scmi/bus.c     |    3 +
 drivers/firmware/arm_scmi/common.h  |    4 +
 drivers/firmware/arm_scmi/driver.c  |  121 +++-
 drivers/firmware/arm_scmi/notify.c  | 1047 +++++++++++++++++++++++++++
 drivers/firmware/arm_scmi/notify.h  |   79 ++
 drivers/firmware/arm_scmi/perf.c    |  167 ++++-
 drivers/firmware/arm_scmi/power.c   |  161 +++-
 drivers/firmware/arm_scmi/reset.c   |  126 +++-
 drivers/firmware/arm_scmi/sensors.c |  105 ++-
 include/linux/scmi_protocol.h       |   82 +++
 12 files changed, 1991 insertions(+), 38 deletions(-)
 create mode 100644 drivers/firmware/arm_scmi/notify.c
 create mode 100644 drivers/firmware/arm_scmi/notify.h

-- 
2.17.1

[RFC PATCH 01/11] firmware: arm_scmi: Add receive buffer support for notifications

[Cristian Marussi]

From: Sudeep Holla <sudeep.holla@arm.com>

With all the plumbing in place, let's just add the separate dedicated
receive buffers to handle notifications that can arrive asynchronously
from the platform firmware to OS.

Also add check to see if the platform supports any receive channels
before allocating the receive buffers.

Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
---
 drivers/firmware/arm_scmi/driver.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 2c96f6b5a7d8..9611e8037d77 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -123,6 +123,7 @@ struct scmi_chan_info {
  * @version: SCMI revision information containing protocol version,
  *	implementation version and (sub-)vendor identification.
  * @tx_minfo: Universal Transmit Message management info
+ * @rx_minfo: Universal Receive Message management info
  * @tx_idr: IDR object to map protocol id to Tx channel info pointer
  * @rx_idr: IDR object to map protocol id to Rx channel info pointer
  * @protocols_imp: List of protocols implemented, currently maximum of
@@ -136,6 +137,7 @@ struct scmi_info {
 	struct scmi_revision_info version;
 	struct scmi_handle handle;
 	struct scmi_xfers_info tx_minfo;
+	struct scmi_xfers_info rx_minfo;
 	struct idr tx_idr;
 	struct idr rx_idr;
 	u8 *protocols_imp;
@@ -690,13 +692,13 @@ int scmi_handle_put(const struct scmi_handle *handle)
 	return 0;
 }
 
-static int scmi_xfer_info_init(struct scmi_info *sinfo)
+static int __scmi_xfer_info_init(struct scmi_info *sinfo, bool tx)
 {
 	int i;
 	struct scmi_xfer *xfer;
 	struct device *dev = sinfo->dev;
 	const struct scmi_desc *desc = sinfo->desc;
-	struct scmi_xfers_info *info = &sinfo->tx_minfo;
+	struct scmi_xfers_info *info = tx ? &sinfo->tx_minfo : &sinfo->rx_minfo;
 
 	/* Pre-allocated messages, no more than what hdr.seq can support */
 	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
@@ -731,6 +733,16 @@ static int scmi_xfer_info_init(struct scmi_info *sinfo)
 	return 0;
 }
 
+static int scmi_xfer_info_init(struct scmi_info *sinfo)
+{
+	int ret = __scmi_xfer_info_init(sinfo, true);
+
+	if (!ret && idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE))
+		ret = __scmi_xfer_info_init(sinfo, false);
+
+	return ret;
+}
+
 static int scmi_mailbox_check(struct device_node *np, int idx)
 {
 	return of_parse_phandle_with_args(np, "mboxes", "#mbox-cells",
@@ -908,10 +920,6 @@ static int scmi_probe(struct platform_device *pdev)
 	info->desc = desc;
 	INIT_LIST_HEAD(&info->node);
 
-	ret = scmi_xfer_info_init(info);
-	if (ret)
-		return ret;
-
 	platform_set_drvdata(pdev, info);
 	idr_init(&info->tx_idr);
 	idr_init(&info->rx_idr);
@@ -924,6 +932,10 @@ static int scmi_probe(struct platform_device *pdev)
 	if (ret)
 		return ret;
 
+	ret = scmi_xfer_info_init(info);
+	if (ret)
+		return ret;
+
 	ret = scmi_base_protocol_init(handle);
 	if (ret) {
 		dev_err(dev, "unable to communicate with SCMI(%d)\n", ret);
-- 
2.17.1

Re: [RFC PATCH 01/11] firmware: arm_scmi: Add receive buffer support for notifications

[Jonathan Cameron]

On Mon, 20 Jan 2020 12:23:23 +0000
Cristian Marussi <cristian.marussi@arm.com> wrote:

> From: Sudeep Holla <sudeep.holla@arm.com>
> 
> With all the plumbing in place, let's just add the separate dedicated
> receive buffers to handle notifications that can arrive asynchronously
> from the platform firmware to OS.
> 
> Also add check to see if the platform supports any receive channels
> before allocating the receive buffers.

Perhaps hand hold the reader a tiny bit more by saying that we need
to move the initialization later so that we can know *if* the receive
channels are supported.  Took me a moment to figure out why you did that ;)

One minor suggestion inline.

> 
> Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
> ---
>  drivers/firmware/arm_scmi/driver.c | 24 ++++++++++++++++++------
>  1 file changed, 18 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 2c96f6b5a7d8..9611e8037d77 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -123,6 +123,7 @@ struct scmi_chan_info {
>   * @version: SCMI revision information containing protocol version,
>   *	implementation version and (sub-)vendor identification.
>   * @tx_minfo: Universal Transmit Message management info
> + * @rx_minfo: Universal Receive Message management info
>   * @tx_idr: IDR object to map protocol id to Tx channel info pointer
>   * @rx_idr: IDR object to map protocol id to Rx channel info pointer
>   * @protocols_imp: List of protocols implemented, currently maximum of
> @@ -136,6 +137,7 @@ struct scmi_info {
>  	struct scmi_revision_info version;
>  	struct scmi_handle handle;
>  	struct scmi_xfers_info tx_minfo;
> +	struct scmi_xfers_info rx_minfo;
>  	struct idr tx_idr;
>  	struct idr rx_idr;
>  	u8 *protocols_imp;
> @@ -690,13 +692,13 @@ int scmi_handle_put(const struct scmi_handle *handle)
>  	return 0;
>  }
>  
> -static int scmi_xfer_info_init(struct scmi_info *sinfo)
> +static int __scmi_xfer_info_init(struct scmi_info *sinfo, bool tx)
>  {
>  	int i;
>  	struct scmi_xfer *xfer;
>  	struct device *dev = sinfo->dev;
>  	const struct scmi_desc *desc = sinfo->desc;
> -	struct scmi_xfers_info *info = &sinfo->tx_minfo;
> +	struct scmi_xfers_info *info = tx ? &sinfo->tx_minfo : &sinfo->rx_minfo;

Perhaps cleaner to just pass in the relevant info structure rather than a boolean
to pick it.  Saves people having to check if the boolean is saying it's
tx or rx when reading the call sites.

>  
>  	/* Pre-allocated messages, no more than what hdr.seq can support */
>  	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
> @@ -731,6 +733,16 @@ static int scmi_xfer_info_init(struct scmi_info *sinfo)
>  	return 0;
>  }
>  
> +static int scmi_xfer_info_init(struct scmi_info *sinfo)
> +{
> +	int ret = __scmi_xfer_info_init(sinfo, true);
> +
> +	if (!ret && idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE))
> +		ret = __scmi_xfer_info_init(sinfo, false);
> +
> +	return ret;
> +}
> +
>  static int scmi_mailbox_check(struct device_node *np, int idx)
>  {
>  	return of_parse_phandle_with_args(np, "mboxes", "#mbox-cells",
> @@ -908,10 +920,6 @@ static int scmi_probe(struct platform_device *pdev)
>  	info->desc = desc;
>  	INIT_LIST_HEAD(&info->node);
>  
> -	ret = scmi_xfer_info_init(info);
> -	if (ret)
> -		return ret;
> -
>  	platform_set_drvdata(pdev, info);
>  	idr_init(&info->tx_idr);
>  	idr_init(&info->rx_idr);
> @@ -924,6 +932,10 @@ static int scmi_probe(struct platform_device *pdev)
>  	if (ret)
>  		return ret;
>  
> +	ret = scmi_xfer_info_init(info);
> +	if (ret)
> +		return ret;
> +
>  	ret = scmi_base_protocol_init(handle);
>  	if (ret) {
>  		dev_err(dev, "unable to communicate with SCMI(%d)\n", ret);

Re: [RFC PATCH 01/11] firmware: arm_scmi: Add receive buffer support for notifications

[Cristian Marussi]

Hi

On 27/01/2020 17:07, Jonathan Cameron wrote:
> On Mon, 20 Jan 2020 12:23:23 +0000
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
>> From: Sudeep Holla <sudeep.holla@arm.com>
>>
>> With all the plumbing in place, let's just add the separate dedicated
>> receive buffers to handle notifications that can arrive asynchronously
>> from the platform firmware to OS.
>>
>> Also add check to see if the platform supports any receive channels
>> before allocating the receive buffers.
> 
> Perhaps hand hold the reader a tiny bit more by saying that we need
> to move the initialization later so that we can know *if* the receive
> channels are supported.  Took me a moment to figure out why you did that ;)
> 

Addressed in v2.

> One minor suggestion inline.
> 
>>
>> Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
>> ---
>>  drivers/firmware/arm_scmi/driver.c | 24 ++++++++++++++++++------
>>  1 file changed, 18 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
>> index 2c96f6b5a7d8..9611e8037d77 100644
>> --- a/drivers/firmware/arm_scmi/driver.c
>> +++ b/drivers/firmware/arm_scmi/driver.c
>> @@ -123,6 +123,7 @@ struct scmi_chan_info {
>>   * @version: SCMI revision information containing protocol version,
>>   *	implementation version and (sub-)vendor identification.
>>   * @tx_minfo: Universal Transmit Message management info
>> + * @rx_minfo: Universal Receive Message management info
>>   * @tx_idr: IDR object to map protocol id to Tx channel info pointer
>>   * @rx_idr: IDR object to map protocol id to Rx channel info pointer
>>   * @protocols_imp: List of protocols implemented, currently maximum of
>> @@ -136,6 +137,7 @@ struct scmi_info {
>>  	struct scmi_revision_info version;
>>  	struct scmi_handle handle;
>>  	struct scmi_xfers_info tx_minfo;
>> +	struct scmi_xfers_info rx_minfo;
>>  	struct idr tx_idr;
>>  	struct idr rx_idr;
>>  	u8 *protocols_imp;
>> @@ -690,13 +692,13 @@ int scmi_handle_put(const struct scmi_handle *handle)
>>  	return 0;
>>  }
>>  
>> -static int scmi_xfer_info_init(struct scmi_info *sinfo)
>> +static int __scmi_xfer_info_init(struct scmi_info *sinfo, bool tx)
>>  {
>>  	int i;
>>  	struct scmi_xfer *xfer;
>>  	struct device *dev = sinfo->dev;
>>  	const struct scmi_desc *desc = sinfo->desc;
>> -	struct scmi_xfers_info *info = &sinfo->tx_minfo;
>> +	struct scmi_xfers_info *info = tx ? &sinfo->tx_minfo : &sinfo->rx_minfo;
> 
> Perhaps cleaner to just pass in the relevant info structure rather than a boolean
> to pick it.  Saves people having to check if the boolean is saying it's
> tx or rx when reading the call sites.
> 

Done in the upcoming v2.

Regards

Cristian

>>  
>>  	/* Pre-allocated messages, no more than what hdr.seq can support */
>>  	if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) {
>> @@ -731,6 +733,16 @@ static int scmi_xfer_info_init(struct scmi_info *sinfo)
>>  	return 0;
>>  }
>>  
>> +static int scmi_xfer_info_init(struct scmi_info *sinfo)
>> +{
>> +	int ret = __scmi_xfer_info_init(sinfo, true);
>> +
>> +	if (!ret && idr_find(&sinfo->rx_idr, SCMI_PROTOCOL_BASE))
>> +		ret = __scmi_xfer_info_init(sinfo, false);
>> +
>> +	return ret;
>> +}
>> +
>>  static int scmi_mailbox_check(struct device_node *np, int idx)
>>  {
>>  	return of_parse_phandle_with_args(np, "mboxes", "#mbox-cells",
>> @@ -908,10 +920,6 @@ static int scmi_probe(struct platform_device *pdev)
>>  	info->desc = desc;
>>  	INIT_LIST_HEAD(&info->node);
>>  
>> -	ret = scmi_xfer_info_init(info);
>> -	if (ret)
>> -		return ret;
>> -
>>  	platform_set_drvdata(pdev, info);
>>  	idr_init(&info->tx_idr);
>>  	idr_init(&info->rx_idr);
>> @@ -924,6 +932,10 @@ static int scmi_probe(struct platform_device *pdev)
>>  	if (ret)
>>  		return ret;
>>  
>> +	ret = scmi_xfer_info_init(info);
>> +	if (ret)
>> +		return ret;
>> +
>>  	ret = scmi_base_protocol_init(handle);
>>  	if (ret) {
>>  		dev_err(dev, "unable to communicate with SCMI(%d)\n", ret);
> 
> 

[RFC PATCH 02/11] firmware: arm_scmi: Update protocol commands and notification list

[Cristian Marussi]

From: Sudeep Holla <sudeep.holla@arm.com>

Add commands' enumerations and messages definitions for all existing
notify-enable commands across all protocols.

Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/base.c    | 7 +++++++
 drivers/firmware/arm_scmi/perf.c    | 5 +++++
 drivers/firmware/arm_scmi/power.c   | 6 ++++++
 drivers/firmware/arm_scmi/sensors.c | 4 ++++
 4 files changed, 22 insertions(+)

diff --git a/drivers/firmware/arm_scmi/base.c b/drivers/firmware/arm_scmi/base.c
index f804e8af6521..ce7d9203e41b 100644
--- a/drivers/firmware/arm_scmi/base.c
+++ b/drivers/firmware/arm_scmi/base.c
@@ -14,6 +14,13 @@ enum scmi_base_protocol_cmd {
 	BASE_DISCOVER_LIST_PROTOCOLS = 0x6,
 	BASE_DISCOVER_AGENT = 0x7,
 	BASE_NOTIFY_ERRORS = 0x8,
+	BASE_SET_DEVICE_PERMISSIONS = 0x9,
+	BASE_SET_PROTOCOL_PERMISSIONS = 0xa,
+	BASE_RESET_AGENT_CONFIGURATION = 0xb,
+};
+
+enum scmi_base_protocol_notify {
+	BASE_ERROR_EVENT = 0x0,
 };
 
 struct scmi_msg_resp_base_attributes {
diff --git a/drivers/firmware/arm_scmi/perf.c b/drivers/firmware/arm_scmi/perf.c
index ec81e6f7e7a4..88509ec637d0 100644
--- a/drivers/firmware/arm_scmi/perf.c
+++ b/drivers/firmware/arm_scmi/perf.c
@@ -27,6 +27,11 @@ enum scmi_performance_protocol_cmd {
 	PERF_DESCRIBE_FASTCHANNEL = 0xb,
 };
 
+enum scmi_performance_protocol_notify {
+	PERFORMANCE_LIMITS_CHANGED = 0x0,
+	PERFORMANCE_LEVEL_CHANGED = 0x1,
+};
+
 struct scmi_opp {
 	u32 perf;
 	u32 power;
diff --git a/drivers/firmware/arm_scmi/power.c b/drivers/firmware/arm_scmi/power.c
index 214886ce84f1..cf7f0312381b 100644
--- a/drivers/firmware/arm_scmi/power.c
+++ b/drivers/firmware/arm_scmi/power.c
@@ -12,6 +12,12 @@ enum scmi_power_protocol_cmd {
 	POWER_STATE_SET = 0x4,
 	POWER_STATE_GET = 0x5,
 	POWER_STATE_NOTIFY = 0x6,
+	POWER_STATE_CHANGE_REQUESTED_NOTIFY = 0x7,
+};
+
+enum scmi_power_protocol_notify {
+	POWER_STATE_CHANGED = 0x0,
+	POWER_STATE_CHANGE_REQUESTED = 0x1,
 };
 
 struct scmi_msg_resp_power_attributes {
diff --git a/drivers/firmware/arm_scmi/sensors.c b/drivers/firmware/arm_scmi/sensors.c
index eba61b9c1f53..db1b1ab303da 100644
--- a/drivers/firmware/arm_scmi/sensors.c
+++ b/drivers/firmware/arm_scmi/sensors.c
@@ -14,6 +14,10 @@ enum scmi_sensor_protocol_cmd {
 	SENSOR_READING_GET = 0x6,
 };
 
+enum scmi_sensor_protocol_notify {
+	SENSOR_TRIP_POINT_EVENT = 0x0,
+};
+
 struct scmi_msg_resp_sensor_attributes {
 	__le16 num_sensors;
 	u8 max_requests;
-- 
2.17.1

[RFC PATCH 03/11] firmware: arm_scmi: Add support for notifications message processing

[Cristian Marussi]

From: Sudeep Holla <sudeep.holla@arm.com>

Add the mechanisms to distinguish notifications from delayed responses and
to properly fetch notification messages upon reception: notifications
processing does not continue further after the fetch phase.

Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/driver.c | 92 +++++++++++++++++++++---------
 1 file changed, 65 insertions(+), 27 deletions(-)

diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 9611e8037d77..28ed1f0cb417 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -212,6 +212,15 @@ static void scmi_fetch_response(struct scmi_xfer *xfer,
 	memcpy_fromio(xfer->rx.buf, mem->msg_payload + 4, xfer->rx.len);
 }
 
+static void scmi_fetch_notification(struct scmi_xfer *xfer, size_t max_len,
+				    struct scmi_shared_mem __iomem *mem)
+{
+	/* Skip only length of header in payload area i.e 4 bytes */
+	xfer->rx.len = min_t(size_t, max_len, ioread32(&mem->length) - 4);
+
+	memcpy_fromio(xfer->rx.buf, mem->msg_payload, xfer->rx.len);
+}
+
 /**
  * pack_scmi_header() - packs and returns 32-bit header
  *
@@ -339,6 +348,58 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
 	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
 }
 
+static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
+{
+	struct scmi_xfer *xfer;
+	struct device *dev = cinfo->dev;
+	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
+	struct scmi_xfers_info *minfo = &info->rx_minfo;
+	struct scmi_shared_mem __iomem *mem = cinfo->payload;
+
+	xfer = scmi_xfer_get(cinfo->handle, minfo);
+	if (IS_ERR(xfer)) {
+		dev_err(dev, "failed to get free message slot (%ld)\n",
+			PTR_ERR(xfer));
+		iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE,
+			  &mem->channel_status);
+		return;
+	}
+
+	unpack_scmi_header(msg_hdr, &xfer->hdr);
+	scmi_dump_header_dbg(dev, &xfer->hdr);
+	scmi_fetch_notification(xfer, info->desc->max_msg_size, mem);
+	__scmi_xfer_put(minfo, xfer);
+
+	iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE, &mem->channel_status);
+}
+
+static void scmi_handle_xfer_delayed_resp(struct scmi_chan_info *cinfo,
+					  u16 xfer_id, bool delayed_resp)
+{
+	struct scmi_xfer *xfer;
+	struct device *dev = cinfo->dev;
+	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
+	struct scmi_xfers_info *minfo = &info->tx_minfo;
+	struct scmi_shared_mem __iomem *mem = cinfo->payload;
+
+	/* Are we even expecting this? */
+	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
+		dev_err(dev, "message for %d is not expected!\n", xfer_id);
+		return;
+	}
+
+	xfer = &minfo->xfer_block[xfer_id];
+
+	scmi_dump_header_dbg(dev, &xfer->hdr);
+
+	scmi_fetch_response(xfer, mem);
+
+	if (delayed_resp)
+		complete(xfer->async_done);
+	else
+		complete(&xfer->done);
+}
+
 /**
  * scmi_rx_callback() - mailbox client callback for receive messages
  *
@@ -355,41 +416,18 @@ static void scmi_rx_callback(struct mbox_client *cl, void *m)
 {
 	u8 msg_type;
 	u32 msg_hdr;
-	u16 xfer_id;
-	struct scmi_xfer *xfer;
 	struct scmi_chan_info *cinfo = client_to_scmi_chan_info(cl);
-	struct device *dev = cinfo->dev;
-	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
-	struct scmi_xfers_info *minfo = &info->tx_minfo;
 	struct scmi_shared_mem __iomem *mem = cinfo->payload;
 
 	msg_hdr = ioread32(&mem->msg_header);
 	msg_type = MSG_XTRACT_TYPE(msg_hdr);
-	xfer_id = MSG_XTRACT_TOKEN(msg_hdr);
 
 	if (msg_type == MSG_TYPE_NOTIFICATION)
-		return; /* Notifications not yet supported */
-
-	/* Are we even expecting this? */
-	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
-		dev_err(dev, "message for %d is not expected!\n", xfer_id);
-		return;
-	}
-
-	xfer = &minfo->xfer_block[xfer_id];
-
-	scmi_dump_header_dbg(dev, &xfer->hdr);
-
-	scmi_fetch_response(xfer, mem);
-
-	trace_scmi_rx_done(xfer->transfer_id, xfer->hdr.id,
-			   xfer->hdr.protocol_id, xfer->hdr.seq,
-			   msg_type);
-
-	if (msg_type == MSG_TYPE_DELAYED_RESP)
-		complete(xfer->async_done);
+		scmi_handle_notification(cinfo, msg_hdr);
 	else
-		complete(&xfer->done);
+		scmi_handle_xfer_delayed_resp(cinfo, MSG_XTRACT_TOKEN(msg_hdr),
+					      msg_type);
+
 }
 
 /**
-- 
2.17.1

Re: [RFC PATCH 03/11] firmware: arm_scmi: Add support for notifications message processing

[Cristian Marussi]

Hi Jim

On 22/01/2020 23:27, Jim Quinlan wrote:
> Hi,
> 
> I tried "git am" on an mbox file  from these commits and get stuck on
> 
>     Applying: firmware: arm_scmi: Add receive buffer support for notifications
>     Applying: firmware: arm_scmi: Update protocol commands and notification list
>     Applying: firmware: arm_scmi: Add support for notifications message processing
>     error: patch failed: drivers/firmware/arm_scmi/driver.c:355
>     error: drivers/firmware/arm_scmi/driver.c: patch does not apply
> 
> 
> Could you please apply this email patchset and let me know if it works for you?  I am doing this onto
> 
>     257d0e20ec4f include: trace: Add SCMI header with trace events     
> 
> 

Sorry... my fault .. it is on top of the following commit on that same branch in fact:

729d3530a504 drivers: firmware: scmi: Extend SCMI transport layer by trace events

10:51 $ git am patch_scmi_notif/ext_V1/final/00*                          
Applying: firmware: arm_scmi: Add receive buffer support for notifications
Applying: firmware: arm_scmi: Update protocol commands and notification list         
Applying: firmware: arm_scmi: Add support for notifications message processing
Applying: firmware: arm_scmi: Add core notifications support                    
Applying: firmware: arm_scmi: Add notifications anti-tampering                                 
Applying: firmware: arm_scmi: Enable core notifications                  
Applying: firmware: arm_scmi: Add Power notifications support             
Applying: firmware: arm_scmi: Add Perf notifications support                
Applying: firmware: arm_scmi: Add Sensor notifications support             
Applying: firmware: arm_scmi: Add Reset notifications support                  
Applying: firmware: arm_scmi: Add Base notifications support            

I'll follow up to my cover to warn about this.

Thanks for trying it out.

Regards

Cristian

> as you directed.
> 
> Thanks,
> Jim
> 
> 
> 
> On 1/20/20 7:23 AM, Cristian Marussi wrote:
>> From: Sudeep Holla <sudeep.holla@arm.com>
>>
>> Add the mechanisms to distinguish notifications from delayed responses and
>> to properly fetch notification messages upon reception: notifications
>> processing does not continue further after the fetch phase.
>>
>> Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
>> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
>> ---
>>  drivers/firmware/arm_scmi/driver.c | 92 +++++++++++++++++++++---------
>>  1 file changed, 65 insertions(+), 27 deletions(-)
>>
>> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
>> index 9611e8037d77..28ed1f0cb417 100644
>> --- a/drivers/firmware/arm_scmi/driver.c
>> +++ b/drivers/firmware/arm_scmi/driver.c
>> @@ -212,6 +212,15 @@ static void scmi_fetch_response(struct scmi_xfer *xfer,
>>  	memcpy_fromio(xfer->rx.buf, mem->msg_payload + 4, xfer->rx.len);
>>  }
>>  
>> +static void scmi_fetch_notification(struct scmi_xfer *xfer, size_t max_len,
>> +				    struct scmi_shared_mem __iomem *mem)
>> +{
>> +	/* Skip only length of header in payload area i.e 4 bytes */
>> +	xfer->rx.len = min_t(size_t, max_len, ioread32(&mem->length) - 4);
>> +
>> +	memcpy_fromio(xfer->rx.buf, mem->msg_payload, xfer->rx.len);
>> +}
>> +
>>  /**
>>   * pack_scmi_header() - packs and returns 32-bit header
>>   *
>> @@ -339,6 +348,58 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>>  	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>>  }
>>  
>> +static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>> +{
>> +	struct scmi_xfer *xfer;
>> +	struct device *dev = cinfo->dev;
>> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>> +	struct scmi_xfers_info *minfo = &info->rx_minfo;
>> +	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>> +
>> +	xfer = scmi_xfer_get(cinfo->handle, minfo);
>> +	if (IS_ERR(xfer)) {
>> +		dev_err(dev, "failed to get free message slot (%ld)\n",
>> +			PTR_ERR(xfer));
>> +		iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE,
>> +			  &mem->channel_status);
>> +		return;
>> +	}
>> +
>> +	unpack_scmi_header(msg_hdr, &xfer->hdr);
>> +	scmi_dump_header_dbg(dev, &xfer->hdr);
>> +	scmi_fetch_notification(xfer, info->desc->max_msg_size, mem);
>> +	__scmi_xfer_put(minfo, xfer);
>> +
>> +	iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE, &mem->channel_status);
>> +}
>> +
>> +static void scmi_handle_xfer_delayed_resp(struct scmi_chan_info *cinfo,
>> +					  u16 xfer_id, bool delayed_resp)
>> +{
>> +	struct scmi_xfer *xfer;
>> +	struct device *dev = cinfo->dev;
>> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>> +	struct scmi_xfers_info *minfo = &info->tx_minfo;
>> +	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>> +
>> +	/* Are we even expecting this? */
>> +	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
>> +		dev_err(dev, "message for %d is not expected!\n", xfer_id);
>> +		return;
>> +	}
>> +
>> +	xfer = &minfo->xfer_block[xfer_id];
>> +
>> +	scmi_dump_header_dbg(dev, &xfer->hdr);
>> +
>> +	scmi_fetch_response(xfer, mem);
>> +
>> +	if (delayed_resp)
>> +		complete(xfer->async_done);
>> +	else
>> +		complete(&xfer->done);
>> +}
>> +
>>  /**
>>   * scmi_rx_callback() - mailbox client callback for receive messages
>>   *
>> @@ -355,41 +416,18 @@ static void scmi_rx_callback(struct mbox_client *cl, void *m)
>>  {
>>  	u8 msg_type;
>>  	u32 msg_hdr;
>> -	u16 xfer_id;
>> -	struct scmi_xfer *xfer;
>>  	struct scmi_chan_info *cinfo = client_to_scmi_chan_info(cl);
>> -	struct device *dev = cinfo->dev;
>> -	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>> -	struct scmi_xfers_info *minfo = &info->tx_minfo;
>>  	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>>  
>>  	msg_hdr = ioread32(&mem->msg_header);
>>  	msg_type = MSG_XTRACT_TYPE(msg_hdr);
>> -	xfer_id = MSG_XTRACT_TOKEN(msg_hdr);
>>  
>>  	if (msg_type == MSG_TYPE_NOTIFICATION)
>> -		return; /* Notifications not yet supported */
>> -
>> -	/* Are we even expecting this? */
>> -	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
>> -		dev_err(dev, "message for %d is not expected!\n", xfer_id);
>> -		return;
>> -	}
>> -
>> -	xfer = &minfo->xfer_block[xfer_id];
>> -
>> -	scmi_dump_header_dbg(dev, &xfer->hdr);
>> -
>> -	scmi_fetch_response(xfer, mem);
>> -
>> -	trace_scmi_rx_done(xfer->transfer_id, xfer->hdr.id,
>> -			   xfer->hdr.protocol_id, xfer->hdr.seq,
>> -			   msg_type);
>> -
>> -	if (msg_type == MSG_TYPE_DELAYED_RESP)
>> -		complete(xfer->async_done);
>> +		scmi_handle_notification(cinfo, msg_hdr);
>>  	else
>> -		complete(&xfer->done);
>> +		scmi_handle_xfer_delayed_resp(cinfo, MSG_XTRACT_TOKEN(msg_hdr),
>> +					      msg_type);
>> +
>>  }
>>  
>>  /**
> 
> 

Re: [RFC PATCH 03/11] firmware: arm_scmi: Add support for notifications message processing

[Jonathan Cameron]

On Mon, 20 Jan 2020 12:23:25 +0000
Cristian Marussi <cristian.marussi@arm.com> wrote:

> From: Sudeep Holla <sudeep.holla@arm.com>
> 
> Add the mechanisms to distinguish notifications from delayed responses and
> to properly fetch notification messages upon reception: notifications
> processing does not continue further after the fetch phase.
> 
> Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>

Couple of bits that seem more interesting than expected inline...

> ---
>  drivers/firmware/arm_scmi/driver.c | 92 +++++++++++++++++++++---------
>  1 file changed, 65 insertions(+), 27 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 9611e8037d77..28ed1f0cb417 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -212,6 +212,15 @@ static void scmi_fetch_response(struct scmi_xfer *xfer,
>  	memcpy_fromio(xfer->rx.buf, mem->msg_payload + 4, xfer->rx.len);
>  }
>  
> +static void scmi_fetch_notification(struct scmi_xfer *xfer, size_t max_len,
> +				    struct scmi_shared_mem __iomem *mem)
> +{
> +	/* Skip only length of header in payload area i.e 4 bytes */
> +	xfer->rx.len = min_t(size_t, max_len, ioread32(&mem->length) - 4);
> +
> +	memcpy_fromio(xfer->rx.buf, mem->msg_payload, xfer->rx.len);
> +}
> +
>  /**
>   * pack_scmi_header() - packs and returns 32-bit header
>   *
> @@ -339,6 +348,58 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>  	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>  }
>  
> +static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
> +{
> +	struct scmi_xfer *xfer;
> +	struct device *dev = cinfo->dev;
> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> +	struct scmi_xfers_info *minfo = &info->rx_minfo;
> +	struct scmi_shared_mem __iomem *mem = cinfo->payload;
> +
> +	xfer = scmi_xfer_get(cinfo->handle, minfo);
> +	if (IS_ERR(xfer)) {
> +		dev_err(dev, "failed to get free message slot (%ld)\n",
> +			PTR_ERR(xfer));
> +		iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE,
> +			  &mem->channel_status);
> +		return;
> +	}
> +
> +	unpack_scmi_header(msg_hdr, &xfer->hdr);
> +	scmi_dump_header_dbg(dev, &xfer->hdr);
> +	scmi_fetch_notification(xfer, info->desc->max_msg_size, mem);
> +	__scmi_xfer_put(minfo, xfer);
> +
> +	iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE, &mem->channel_status);
> +}
> +
> +static void scmi_handle_xfer_delayed_resp(struct scmi_chan_info *cinfo,
> +					  u16 xfer_id, bool delayed_resp)

Hmm. A function called *_delayed_resp that takes a boolean to say if
it is a delayed_resp is in the category of non obvious....  Needs a rename
at the very least.

> +{
> +	struct scmi_xfer *xfer;
> +	struct device *dev = cinfo->dev;
> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> +	struct scmi_xfers_info *minfo = &info->tx_minfo;
> +	struct scmi_shared_mem __iomem *mem = cinfo->payload;
> +
> +	/* Are we even expecting this? */
> +	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
> +		dev_err(dev, "message for %d is not expected!\n", xfer_id);
> +		return;
> +	}
> +
> +	xfer = &minfo->xfer_block[xfer_id];
> +
> +	scmi_dump_header_dbg(dev, &xfer->hdr);
> +
> +	scmi_fetch_response(xfer, mem);
> +
> +	if (delayed_resp)
> +		complete(xfer->async_done);
> +	else
> +		complete(&xfer->done);
> +}
> +
>  /**
>   * scmi_rx_callback() - mailbox client callback for receive messages
>   *
> @@ -355,41 +416,18 @@ static void scmi_rx_callback(struct mbox_client *cl, void *m)
>  {
>  	u8 msg_type;
>  	u32 msg_hdr;
> -	u16 xfer_id;
> -	struct scmi_xfer *xfer;
>  	struct scmi_chan_info *cinfo = client_to_scmi_chan_info(cl);
> -	struct device *dev = cinfo->dev;
> -	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
> -	struct scmi_xfers_info *minfo = &info->tx_minfo;
>  	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>  
>  	msg_hdr = ioread32(&mem->msg_header);
>  	msg_type = MSG_XTRACT_TYPE(msg_hdr);
> -	xfer_id = MSG_XTRACT_TOKEN(msg_hdr);
>  
>  	if (msg_type == MSG_TYPE_NOTIFICATION)
> -		return; /* Notifications not yet supported */
> -
> -	/* Are we even expecting this? */
> -	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
> -		dev_err(dev, "message for %d is not expected!\n", xfer_id);
> -		return;
> -	}
> -
> -	xfer = &minfo->xfer_block[xfer_id];
> -
> -	scmi_dump_header_dbg(dev, &xfer->hdr);
> -
> -	scmi_fetch_response(xfer, mem);
> -
> -	trace_scmi_rx_done(xfer->transfer_id, xfer->hdr.id,
> -			   xfer->hdr.protocol_id, xfer->hdr.seq,
> -			   msg_type);
> -
> -	if (msg_type == MSG_TYPE_DELAYED_RESP)
> -		complete(xfer->async_done);
> +		scmi_handle_notification(cinfo, msg_hdr);
>  	else
> -		complete(&xfer->done);
> +		scmi_handle_xfer_delayed_resp(cinfo, MSG_XTRACT_TOKEN(msg_hdr),
> +					      msg_type);
First I wondered why this wasn't a switch which would make a clear distinction
between notification path and delayed response... 

However, it seems delayed_resp path also handles other values of msg_type,
though only 0 which is a command I think...

Passing a enum that I think can take 4 values, only 3 of which are defined
into a function as a boolean is 'interesting'. Don't do that.


> +
>  }
>  
>  /**

Re: [RFC PATCH 03/11] firmware: arm_scmi: Add support for notifications message processing

[Cristian Marussi]

Hi

On 27/01/2020 17:32, Jonathan Cameron wrote:
> On Mon, 20 Jan 2020 12:23:25 +0000
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
>> From: Sudeep Holla <sudeep.holla@arm.com>
>>
>> Add the mechanisms to distinguish notifications from delayed responses and
>> to properly fetch notification messages upon reception: notifications
>> processing does not continue further after the fetch phase.
>>
>> Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
>> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> 
> Couple of bits that seem more interesting than expected inline...
> 
>> ---
>>  drivers/firmware/arm_scmi/driver.c | 92 +++++++++++++++++++++---------
>>  1 file changed, 65 insertions(+), 27 deletions(-)
>>
>> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
>> index 9611e8037d77..28ed1f0cb417 100644
>> --- a/drivers/firmware/arm_scmi/driver.c
>> +++ b/drivers/firmware/arm_scmi/driver.c
>> @@ -212,6 +212,15 @@ static void scmi_fetch_response(struct scmi_xfer *xfer,
>>  	memcpy_fromio(xfer->rx.buf, mem->msg_payload + 4, xfer->rx.len);
>>  }
>>  
>> +static void scmi_fetch_notification(struct scmi_xfer *xfer, size_t max_len,
>> +				    struct scmi_shared_mem __iomem *mem)
>> +{
>> +	/* Skip only length of header in payload area i.e 4 bytes */
>> +	xfer->rx.len = min_t(size_t, max_len, ioread32(&mem->length) - 4);
>> +
>> +	memcpy_fromio(xfer->rx.buf, mem->msg_payload, xfer->rx.len);
>> +}
>> +
>>  /**
>>   * pack_scmi_header() - packs and returns 32-bit header
>>   *
>> @@ -339,6 +348,58 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>>  	spin_unlock_irqrestore(&minfo->xfer_lock, flags);
>>  }
>>  
>> +static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>> +{
>> +	struct scmi_xfer *xfer;
>> +	struct device *dev = cinfo->dev;
>> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>> +	struct scmi_xfers_info *minfo = &info->rx_minfo;
>> +	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>> +
>> +	xfer = scmi_xfer_get(cinfo->handle, minfo);
>> +	if (IS_ERR(xfer)) {
>> +		dev_err(dev, "failed to get free message slot (%ld)\n",
>> +			PTR_ERR(xfer));
>> +		iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE,
>> +			  &mem->channel_status);
>> +		return;
>> +	}
>> +
>> +	unpack_scmi_header(msg_hdr, &xfer->hdr);
>> +	scmi_dump_header_dbg(dev, &xfer->hdr);
>> +	scmi_fetch_notification(xfer, info->desc->max_msg_size, mem);
>> +	__scmi_xfer_put(minfo, xfer);
>> +
>> +	iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE, &mem->channel_status);
>> +}
>> +
>> +static void scmi_handle_xfer_delayed_resp(struct scmi_chan_info *cinfo,
>> +					  u16 xfer_id, bool delayed_resp)
> 
> Hmm. A function called *_delayed_resp that takes a boolean to say if
> it is a delayed_resp is in the category of non obvious....  Needs a rename
> at the very least.
> 

Changed logic and naming in v2, also because now v2 is on top of Viresh series
which makes transport layer independent. (in fact an additional dedicated patch
is dedicated to extend the transport layer for notifications too)


>> +{
>> +	struct scmi_xfer *xfer;
>> +	struct device *dev = cinfo->dev;
>> +	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>> +	struct scmi_xfers_info *minfo = &info->tx_minfo;
>> +	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>> +
>> +	/* Are we even expecting this? */
>> +	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
>> +		dev_err(dev, "message for %d is not expected!\n", xfer_id);
>> +		return;
>> +	}
>> +
>> +	xfer = &minfo->xfer_block[xfer_id];
>> +
>> +	scmi_dump_header_dbg(dev, &xfer->hdr);
>> +
>> +	scmi_fetch_response(xfer, mem);
>> +
>> +	if (delayed_resp)
>> +		complete(xfer->async_done);
>> +	else
>> +		complete(&xfer->done);
>> +}
>> +
>>  /**
>>   * scmi_rx_callback() - mailbox client callback for receive messages
>>   *
>> @@ -355,41 +416,18 @@ static void scmi_rx_callback(struct mbox_client *cl, void *m)
>>  {
>>  	u8 msg_type;
>>  	u32 msg_hdr;
>> -	u16 xfer_id;
>> -	struct scmi_xfer *xfer;
>>  	struct scmi_chan_info *cinfo = client_to_scmi_chan_info(cl);
>> -	struct device *dev = cinfo->dev;
>> -	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>> -	struct scmi_xfers_info *minfo = &info->tx_minfo;
>>  	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>>  
>>  	msg_hdr = ioread32(&mem->msg_header);
>>  	msg_type = MSG_XTRACT_TYPE(msg_hdr);
>> -	xfer_id = MSG_XTRACT_TOKEN(msg_hdr);
>>  
>>  	if (msg_type == MSG_TYPE_NOTIFICATION)
>> -		return; /* Notifications not yet supported */
>> -
>> -	/* Are we even expecting this? */
>> -	if (!test_bit(xfer_id, minfo->xfer_alloc_table)) {
>> -		dev_err(dev, "message for %d is not expected!\n", xfer_id);
>> -		return;
>> -	}
>> -
>> -	xfer = &minfo->xfer_block[xfer_id];
>> -
>> -	scmi_dump_header_dbg(dev, &xfer->hdr);
>> -
>> -	scmi_fetch_response(xfer, mem);
>> -
>> -	trace_scmi_rx_done(xfer->transfer_id, xfer->hdr.id,
>> -			   xfer->hdr.protocol_id, xfer->hdr.seq,
>> -			   msg_type);
>> -
>> -	if (msg_type == MSG_TYPE_DELAYED_RESP)
>> -		complete(xfer->async_done);
>> +		scmi_handle_notification(cinfo, msg_hdr);
>>  	else
>> -		complete(&xfer->done);
>> +		scmi_handle_xfer_delayed_resp(cinfo, MSG_XTRACT_TOKEN(msg_hdr),
>> +					      msg_type);
> First I wondered why this wasn't a switch which would make a clear distinction
> between notification path and delayed response... 
> 
> However, it seems delayed_resp path also handles other values of msg_type,
> though only 0 which is a command I think...
> 
> Passing a enum that I think can take 4 values, only 3 of which are defined
> into a function as a boolean is 'interesting'. Don't do that.
> 

Reworked the logic in v2

Thanks

Regards

Cristian
> 
>> +
>>  }
>>  
>>  /**
> 
> 

[RFC PATCH 04/11] firmware: arm_scmi: Add core notifications support

[Cristian Marussi]

Add basic support for SCMI Notifications, using Kernel notification chains
mechanism.

Each SCMI Protocol has a dedicated events' queue and deferred worker which
is in charge of delivering the notifications to the interested users
invoking the users' registered callbacks.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/Makefile |   2 +-
 drivers/firmware/arm_scmi/common.h |   4 +
 drivers/firmware/arm_scmi/driver.c |   2 +
 drivers/firmware/arm_scmi/notify.c | 904 +++++++++++++++++++++++++++++
 drivers/firmware/arm_scmi/notify.h |  79 +++
 5 files changed, 990 insertions(+), 1 deletion(-)
 create mode 100644 drivers/firmware/arm_scmi/notify.c
 create mode 100644 drivers/firmware/arm_scmi/notify.h

diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
index 5f298f00a82e..26587ea4661f 100644
--- a/drivers/firmware/arm_scmi/Makefile
+++ b/drivers/firmware/arm_scmi/Makefile
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-y	= scmi-bus.o scmi-driver.o scmi-protocols.o
 scmi-bus-y = bus.o
-scmi-driver-y = driver.o
+scmi-driver-y = driver.o notify.o
 scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o
 obj-$(CONFIG_ARM_SCMI_POWER_DOMAIN) += scmi_pm_domain.o
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index df35358ff324..2621c05e9149 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -6,6 +6,8 @@
  *
  * Copyright (C) 2018 ARM Ltd.
  */
+#ifndef _SCMI_COMMON_H
+#define _SCMI_COMMON_H
 
 #include <linux/bitfield.h>
 #include <linux/completion.h>
@@ -113,3 +115,5 @@ void scmi_setup_protocol_implemented(const struct scmi_handle *handle,
 				     u8 *prot_imp);
 
 int scmi_base_protocol_init(struct scmi_handle *h);
+
+#endif /* _SCMI_COMMON_H */
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 28ed1f0cb417..a43fad29de11 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -350,12 +350,14 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
 
 static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 {
+	ktime_t ts;
 	struct scmi_xfer *xfer;
 	struct device *dev = cinfo->dev;
 	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
 	struct scmi_xfers_info *minfo = &info->rx_minfo;
 	struct scmi_shared_mem __iomem *mem = cinfo->payload;
 
+	ts = ktime_get_boottime();
 	xfer = scmi_xfer_get(cinfo->handle, minfo);
 	if (IS_ERR(xfer)) {
 		dev_err(dev, "failed to get free message slot (%ld)\n",
diff --git a/drivers/firmware/arm_scmi/notify.c b/drivers/firmware/arm_scmi/notify.c
new file mode 100644
index 000000000000..da342f43021e
--- /dev/null
+++ b/drivers/firmware/arm_scmi/notify.c
@@ -0,0 +1,904 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * System Control and Management Interface (SCMI) Notification support
+ *
+ * Copyright (C) 2019 ARM Ltd.
+ *
+ * SCMI Protocol specification allows the platform to signal events to
+ * interested agents via notification messages: this in an implementation
+ * of the dispatch and delivery of such notifications to the interested users
+ * inside the Linux kernel.
+ *
+ * Each SCMI Protocol implementation, during its initialization, registers with
+ * this core notification framework its set of supported events via
+ * @scmi_register_protocol_events(), while Kernel users interested in some
+ * specific event can register their associated callbacks providing the usual
+ * notifier_block descriptor, since the notification system internally supports
+ * events delivery using customized notification chains.
+ *
+ * Given the number of possible events defined by SCMI and the extensibility
+ * of the SCMI Protocol itself, such underlying notification chains are created
+ * and destroyed dynamically on demand depending on the number of users
+ * effectively registered for an event, so that no structures or chains are
+ * allocated until at least one user has registered a notifier_block for such
+ * event. Similarly, events' generation itself is enabled at the platform level
+ * only after at least one user has registered, and it is shutdown after the
+ * last user for that event has gone.
+ *
+ * An event is identified univocally by the tuple (proto_id, evt_id, src_id)
+ * and is served by its own dedicated notification chain: given that such chain
+ * is dynamically created, the registration API simply let the users associate
+ * their callbacks with the above tuple.
+ *
+ * Here proto_id and evt_id are simply the protocol_id and message_id numbers as
+ * reported in the SCMI Protocol specification, while src_id represents an
+ * optional, protocol dependent, source identifier (like domain_id, perf_id
+ * or sensor_id and so forth), so that a user can register its callbacks for a
+ * particular event coming only from a defined source (like CPU vs GPU).
+ * When the source is not specified the user callback will be registered for
+ * all existing sources for that event (if any).
+ *
+ * Upon reception of a notification message from the platform the SCMI RX ISR
+ * passes the received message payload and some ancillary information (including
+ * an arrival timestamp in nanoseconds) to the core via @scmi_notify(), which,
+ * in turn, after having looked up the event in the registered events mapping,
+ * pushes the event-data itself on a protocol dedicated kfifo queue for deferred
+ * processing.
+ *
+ * Such dedicated protocols' queues are allocated once for all at initialization
+ * time, together with a dedicated work_item running the common delivery logic
+ * of @scmi_events_dispatcher(), so that each protocol has it own dedicated
+ * worker which, once kicked by the ISR, takes care to empty is own dedicated
+ * queue deliverying the queued items into the proper notification chain.
+ * Note that since the underlying cmwq workers run one distinct work_item per
+ * protocol and there are a number of max-active workers equal to the number of
+ * protocols, notifications processing can proceed concurrently only between
+ * events belonging to different protocols, while delivery of events within the
+ * same protocol is still strictly sequentially ordered by time of arrival; this
+ * separation effectively avoids the possibility that one particularly verbose
+ * protocol, flooding the queues with events, can cause other protocols' events
+ * to be lost or their processing starved.
+ *
+ * Event's information is converted to a custom per-event report struct before
+ * being fed as void *data param to the user callback embedded in the registered
+ * notifier_block so that it looks like as follow from the user perspective:
+ *
+ * int user_cb(struct notifier_block *nb, unsigned long event_id, void *report)
+ *
+ *
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/bug.h>
+#include <linux/bitfield.h>
+#include <linux/err.h>
+#include <linux/idr.h>
+#include <linux/kernel.h>
+#include <linux/kfifo.h>
+#include <linux/notifier.h>
+#include <linux/refcount.h>
+#include <linux/slab.h>
+#include <linux/scmi_protocol.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+#include "notify.h"
+
+#define SCMI_NOTIF_MAX_ACTIVE_QUEUES	7
+
+/*
+ * Builds an unsigned 32bit key from the given input tuple to be used as a key
+ * in IDR mappings; note that if __src_p is passed in as NULL, the returned key
+ * is built using SCMI_ALL_SRC_IDS as src_id.
+ */
+#define MAKE_EVT_KEY(__proto, __evt, __src_p)				\
+	({								\
+		u32 __evt_key, __src_id;				\
+									\
+		__src_id = ((__src_p)) ?				\
+			    (*((u32 *)(__src_p)) & SCMI_ALL_SRC_IDS) :	\
+			    SCMI_ALL_SRC_IDS;				\
+		__evt_key = ((__proto) << 24) |				\
+				((__evt) << 16) | __src_id;		\
+		__evt_key;						\
+	})
+
+#define PROTO_ID_MASK			GENMASK(31, 24)
+#define EVT_ID_MASK			GENMASK(23, 16)
+#define SRC_ID_MASK			GENMASK(15, 0)
+#define KEY_XTRACT_PROTO_ID(key)	FIELD_GET(PROTO_ID_MASK, (key))
+#define KEY_XTRACT_EVT_ID(key)		FIELD_GET(EVT_ID_MASK, (key))
+#define KEY_XTRACT_SRC_ID(key)		FIELD_GET(SRC_ID_MASK, (key))
+
+/**
+ * events_queue  - Describes a queue and its associated worker
+ *
+ * Each protocol has its own dedicated events_queue descriptor.
+ *
+ * @sz: Size in bytes of the related kfifo
+ * @kfifo: A dedicated Kernel kfifo
+ * @notify_work: A custom work item bound to this queue
+ * @wq: A reference to the related workqueue
+ */
+struct events_queue {
+	size_t			sz;
+	struct kfifo		kfifo;
+	struct work_struct	notify_work;
+	struct workqueue_struct	*wq;
+};
+
+/**
+ * scmi_registered_protocol_events_desc  - Protocol Specific information
+ *
+ * All protocols that registers at least an event have their protocol-specific
+ * information stored here, together with a pointer to the allocated
+ * events_queue.
+ *
+ * @id: Protocol ID
+ * @ops: Protocol specific and event-related operations
+ * @equeue: A reference to the associated per-protocol events_queue
+ */
+struct scmi_registered_protocol_events_desc {
+	u8					id;
+	const struct scmi_protocol_event_ops	*ops;
+	struct events_queue			*equeue;
+};
+
+/**
+ * scmi_registered_event  - Event Specific Information
+ *
+ * All registered events are represented by one of these structures.
+ *
+ * @proto: A reference to the associated protocol descriptor
+ * @evt: A reference to the associated event descriptor (as provided at
+ *       registration time)
+ * @scratch_isr: A pre-allocated buffer to be used as a scratch area by ISR
+ * @scratch_isr: A pre-allocated buffer to be used as a scratch area by the
+ *		 deferred worker
+ * @report: A pre-allocated buffer used by the deferred workr to fill a
+ *	    customized event report
+ */
+struct scmi_registered_event {
+	struct scmi_registered_protocol_events_desc	*proto;
+	const struct scmi_event				*evt;
+	void						*scratch_isr;
+	void						*scratch_bh;
+	void						*report;
+};
+
+/**
+ * scmi_event_handler  - Event handler information
+ *
+ * This structure collects all the information needed to process a received
+ * event, calling all the registered notifier callbacks.
+ * Note this an event handler is associated to a tuple of values like:
+ *	(proto_id, evt_id, src_id)
+ * through the means of a key and an IDR mapping.
+ *
+ * @evt_key: The unsigned 32bit key associated to this descriptor in the
+ *	     related IDR mapping
+ * @r_evt: A reference to the underlying registered event
+ * @chain: The notification chain dedicated to this specific event tuple
+ */
+struct scmi_event_handler {
+	u32				evt_key;
+	bool				enabled;
+	refcount_t			users;
+	struct scmi_registered_event	*r_evt;
+	struct blocking_notifier_head	chain;
+};
+
+/**
+ * scmi_event_header  - A utility header
+ *
+ * This header is prepended to each received event message payload before
+ * being queued on the related events_queue, and describes the attached event
+ * message.
+ *
+ * @timestamp: The timestamp, innanoseconds (boottime), which was associated
+ *	       to this event as soon as it entered the SCMI RX ISR
+ * @proto_id: Protocol ID
+ * @evt_id: Event ID (corresponds to the Event MsgID for this Protocol)
+ * @payld_sz: Effective size of the attached message payload which follows
+ * @payld: A reference to the included Event payload
+ */
+struct scmi_event_header {
+	u64	timestamp;
+	u8	proto_id;
+	u8	evt_id;
+	size_t	payld_sz;
+	u8	payld[];
+} __packed;
+
+/*
+ * A few IDR maps to track:
+ *
+ * - @scmi_registered_events: All event's descriptors registered by the
+ *			      protocols, together with their ancillary data
+ * - @scmi_registered_events_handlers: All event's handlers descriptors, created
+ *				       to collect all the users' notifier_block
+ *				       callbacks and related notification chains
+ * - @scmi_registered_protocols: All protocol-level specific information related
+ *				 to events' handling
+ */
+static struct idr scmi_registered_events;
+static struct idr scmi_registered_events_handlers;
+static DEFINE_MUTEX(scmi_registered_events_handler_mtx);
+static struct idr scmi_registered_protocols;
+
+/* Common Kernel cmwq workqueue used by notifications core */
+static struct workqueue_struct *scmi_notify_wq;
+
+static bool scmi_notifications_initialized;
+
+static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key);
+static void scmi_put_event_handler(struct scmi_event_handler *hndl);
+
+/**
+ * scmi_discard_bad_evt_payloadi() - Discard data from a kfifo
+ *
+ * @kq: The kfifo to act on
+ * @count: Number of bytes to flush
+ */
+static inline void scmi_discard_bad_evt_payload(struct kfifo *kq,
+						const unsigned int count)
+{
+	int i = 0;
+
+	pr_warn("SCMI Notification WQ :: skipping bad EVT Payload - %d bytes\n",
+		count);
+	/* Discard stale pending queued payload. */
+	for (i = 0; i < count; i++)
+		kfifo_skip(kq);
+}
+
+/**
+ * scmi_lookup_and_call_event_chain  - Lookup the proper chain and call it
+ *
+ * @evt_key: The event key to use to lookup the related notification chain
+ * @report: The customized event-specific report to pass down to the callbacks
+ *	    as their *data parameter.
+ */
+static inline void scmi_lookup_and_call_event_chain(u32 evt_key, void *report)
+{
+	int ret;
+	struct scmi_event_handler *hndl;
+
+	hndl = scmi_get_event_handler(evt_key);
+	if (IS_ERR_OR_NULL(hndl))
+		return;
+
+	ret = blocking_notifier_call_chain(&hndl->chain,
+					   KEY_XTRACT_EVT_ID(evt_key),
+					   report);
+	/* Notifiers are NOT supposed to cut the chain */
+	WARN_ON_ONCE(ret & NOTIFY_STOP_MASK);
+
+	scmi_put_event_handler(hndl);
+}
+
+/**
+ * scmi_events_dispatcher  - Common worker logic for all work items.
+ *
+ * In turn:
+ *  1. dequeue one pending RX notification (queued in SCMI RX ISR context)
+ *  2. generate a custom event report from the received event message
+ *  3. lookup for any registered ALL_SRC_IDs handler
+ *     - > call the related notification chain passing in the report
+ *  4. lookup for any registered specific SRC_ID handler
+ *     - > call the related notification chain passing in the report
+ *
+ * Note that:
+ * - a dedicated per-protocol kfifo queue is used: in this way an anomalous
+ *   flood of events cannot saturate other protocols' queues.
+ *
+ * - each per-protocol queue is associated to a distinct work_item, which
+ *   means, in turn, that:
+ *   + all protocol can process their dedicated queues concurrently
+ *     (since scmi_notify_wq:max_active > 1)
+ *   + anyway at most one worker instance is allowed to run on the same queue
+ *     concurrently: this ensures that we can have only one concurrent
+ *     reader/writer on the associated kfifo (needed for lock-less access)
+ *
+ * @work: The work item to use, which is associated to the proper events_queue
+ */
+static void scmi_events_dispatcher(struct work_struct *work)
+{
+	struct events_queue *equeue;
+	struct scmi_event_header eh;
+
+	equeue = container_of(work, struct events_queue, notify_work);
+	while (kfifo_out(&equeue->kfifo, &eh, sizeof(eh))) {
+		u32 src_id, evt_key;
+		unsigned int outs;
+		struct scmi_registered_event *r_evt;
+		void *report = NULL;
+
+		evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, NULL);
+		r_evt = idr_find(&scmi_registered_events, evt_key);
+		if (!r_evt) {
+			scmi_discard_bad_evt_payload(&equeue->kfifo,
+						     eh.payld_sz);
+			continue;
+		}
+
+		outs = kfifo_out(&equeue->kfifo, r_evt->scratch_bh,
+				 eh.payld_sz);
+		if (outs != eh.payld_sz) {
+			pr_warn("SCMI Notification WQ :: SKIP corrupted EVT Payload.\n");
+			continue;
+		}
+
+		/* Reset and fill custom report */
+		memset(r_evt->report, 0x00, r_evt->evt->max_report_sz);
+		report = r_evt->proto->ops->fill_custom_report(eh.evt_id,
+							eh.timestamp,
+							r_evt->scratch_bh,
+							eh.payld_sz,
+							r_evt->report,
+							&src_id);
+		if (!report)
+			continue;
+
+		/* At first search for a generic ALL src_ids handler... */
+		scmi_lookup_and_call_event_chain(evt_key, report);
+		if (src_id != SCMI_ALL_SRC_IDS) {
+			u32 *__sub_p = &src_id;
+
+			evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, __sub_p);
+			scmi_lookup_and_call_event_chain(evt_key, report);
+		}
+	}
+}
+
+/**
+ * scmi_notify  - Queues a notification for further deferred processing
+ *
+ * This is called in interrupt context to queue a received event for
+ * deferred processing.
+ *
+ * @proto_id: Protocol ID
+ * @evt_id: Event ID (msgID)
+ * @buf: Event Message Payload (without the header)
+ * @len: Event Message Payload size
+ * @ts: RX Timestamp in nanoseconds (boottime)
+ *
+ * Return: 0 on Success
+ */
+int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts)
+{
+	struct scmi_registered_event *r_evt;
+	struct scmi_event_header *eh;
+
+	if (unlikely(!scmi_notifications_initialized))
+		return 0;
+
+	r_evt = idr_find(&scmi_registered_events,
+			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
+	if (unlikely(!r_evt || !r_evt->proto->equeue))
+		return -EINVAL;
+
+	if (unlikely(len > r_evt->evt->max_payld_sz)) {
+		pr_err("SCMI Notification discard badly sized message\n");
+		return -EINVAL;
+	}
+	if (unlikely(kfifo_avail(&r_evt->proto->equeue->kfifo) <
+		     sizeof(*eh) + len)) {
+		pr_warn("SCMI Notification queue full: dropping proto_id:%d  evt_id:%d  ts:%lld\n",
+			proto_id, evt_id, ts);
+		return -ENOMEM;
+	}
+
+	/*
+	 * Note that we can use the per protocol kfifo in a lock-less manner
+	 * since we have only one concurrent reader and writer but, in order
+	 * to avoid any trouble on the reader side, here we perform one single
+	 * write, so we have to collate event header and payload in a scratch
+	 * area at first.
+	 */
+	eh = r_evt->scratch_isr;
+	eh->timestamp = ts;
+	eh->proto_id = proto_id;
+	eh->evt_id = evt_id;
+	eh->payld_sz = len;
+	memcpy(eh->payld, buf, eh->payld_sz);
+	kfifo_in(&r_evt->proto->equeue->kfifo, eh, sizeof(*eh) + eh->payld_sz);
+	queue_work(r_evt->proto->equeue->wq,
+		   &r_evt->proto->equeue->notify_work);
+
+	return 0;
+}
+
+/**
+ * scmi_allocate_events_queue  - Allocate an events_queue descriptor
+ *
+ * @sz: Size of the kfifo to initialize
+ *
+ * Return: A valid pointer to the allocated events_queue on Success
+ */
+static struct events_queue *scmi_allocate_events_queue(size_t sz)
+{
+	int ret;
+	struct events_queue *equeue;
+
+	equeue = kzalloc(sizeof(*equeue), GFP_KERNEL);
+	if (!equeue)
+		return ERR_PTR(-ENOMEM);
+	ret = kfifo_alloc(&equeue->kfifo, sz, GFP_KERNEL);
+	if (ret) {
+		kfree(equeue);
+		return ERR_PTR(ret);
+	}
+	equeue->sz = sz;
+	INIT_WORK(&equeue->notify_work, scmi_events_dispatcher);
+	equeue->wq = scmi_notify_wq;
+
+	return equeue;
+}
+
+/**
+ * scmi_deallocate_events_queue  - Deallocate am events_queue descriptor
+ *
+ * @equeue: The events_queue to free
+ */
+static void scmi_deallocate_events_queue(struct events_queue *equeue)
+{
+	kfifo_free(&equeue->kfifo);
+	kfree(equeue);
+}
+
+/**
+ * scmi_allocate_registered_protocol_desc  - Allocate a registered protocol
+ * events' descriptor
+ *
+ * Used to keep protocol specific information related to events handling for any
+ * protocl which has registered at least one event.
+ *
+ * @proto_id: Protocol ID
+ * @queue_sz: Size of the associated queue to allocate
+ * @ops: Pointer to a struct holding references to protocol specific helpers
+ *	 needed during events handling
+ */
+static struct scmi_registered_protocol_events_desc *
+scmi_allocate_registered_protocol_desc(u8 proto_id, size_t queue_sz,
+				const struct scmi_protocol_event_ops *ops)
+{
+	int ret;
+	struct scmi_registered_protocol_events_desc *pdesc;
+
+	pdesc = idr_find(&scmi_registered_protocols, proto_id);
+	if (pdesc)
+		return pdesc;
+
+	pdesc = kzalloc(sizeof(*pdesc), GFP_KERNEL);
+	if (!pdesc)
+		return ERR_PTR(-ENOMEM);
+	pdesc->id = proto_id;
+	pdesc->ops = ops;
+
+	pdesc->equeue = scmi_allocate_events_queue(queue_sz);
+	if (IS_ERR(pdesc->equeue)) {
+		kfree(pdesc);
+		return ERR_CAST(pdesc->equeue);
+	}
+
+	ret = idr_alloc(&scmi_registered_protocols, pdesc,
+			pdesc->id, pdesc->id + 1, GFP_KERNEL);
+	if (ret < 0) {
+		pr_err("SCMI Failed to allocate PDESC IDR - key:%d - err:%d\n",
+		       pdesc->id, ret);
+		scmi_deallocate_events_queue(pdesc->equeue);
+		kfree(pdesc);
+		return ERR_PTR(ret);
+	}
+
+	return pdesc;
+}
+
+/**
+ * scmi_register_protocol_events  - Register Protocol Events with the core
+ *
+ * Used by SCMI Protocols initialization code to register with the notification
+ * core the list of supported events and their description: this takes care to
+ * pre-allocate all needed scratch buffers and link the proper event queue to
+ * this event.
+ *
+ * @proto_id: Protocol ID
+ * @queue_sz: Size in bytes of the associated queue to be allocated
+ * @ops: Protocol specific event-related operations
+ * @evt: Event descriptor array
+ * @num_events: Number of events in @evt array
+ *
+ * Return: 0 on Success
+ */
+int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
+				  const struct scmi_protocol_event_ops *ops,
+				  const struct scmi_event *evt, int num_events)
+{
+	int i;
+	struct scmi_registered_protocol_events_desc *pdesc;
+
+	if (!scmi_notifications_initialized)
+		return -EAGAIN;
+
+	if (!ops || !evt)
+		return -EINVAL;
+
+	pdesc = scmi_allocate_registered_protocol_desc(proto_id, queue_sz, ops);
+	if (IS_ERR(pdesc))
+		return -ENOMEM;
+
+	for (i = 0; i < num_events; i++, evt++) {
+		int ret;
+		u32 key;
+		struct scmi_registered_event *r_evt;
+
+		r_evt = kzalloc(sizeof(*r_evt), GFP_KERNEL);
+		if (!r_evt)
+			continue;
+		r_evt->proto = pdesc;
+		r_evt->evt = evt;
+
+		r_evt->scratch_isr = kzalloc(sizeof(struct scmi_event_header) +
+					     evt->max_payld_sz, GFP_KERNEL);
+		if (!r_evt->scratch_isr)
+			goto continue_isr_fail;
+
+		r_evt->scratch_bh = kzalloc(evt->max_payld_sz, GFP_KERNEL);
+		if (!r_evt->scratch_bh)
+			goto continue_bh_fail;
+
+		r_evt->report = kzalloc(evt->max_report_sz, GFP_KERNEL);
+		if (!r_evt->report)
+			goto continue_report_fail;
+
+		key = MAKE_EVT_KEY(r_evt->proto->id, evt->evt_id, NULL);
+		ret = idr_alloc(&scmi_registered_events, r_evt,
+				key, key + 1, GFP_KERNEL);
+		if (ret < 0) {
+			pr_err("SCMI Failed to allocate EVENT IDR - key:%X - err:%d\n",
+			       key, ret);
+			goto continue_idr_fail;
+		}
+
+		pr_info("SCMI Notification registered event - key:%X\n", key);
+		continue;
+
+	/* yes, this is not nice ... */
+continue_idr_fail:
+		kfree(r_evt->report);
+continue_report_fail:
+		kfree(r_evt->scratch_bh);
+continue_bh_fail:
+		kfree(r_evt->scratch_isr);
+continue_isr_fail:
+		kfree(r_evt);
+	}
+
+	return 0;
+}
+
+/**
+ * scmi_register_event_handler  - Allocate an Event handler
+ *
+ * Allocate an event handler and related notification chain associated with
+ * the event identified by the provided event key. Fails if the associated
+ * event is unknown to the core (i.e. it had not been successfully registered
+ * upfront by some protocol)
+ *
+ * @evt_key: 32bit key uniquely bind to the event identified by the tuple
+ *	     (proto_id, evt_id, src_id)
+ *
+ * Return: the freshly allocated structure on Success
+ */
+static struct scmi_event_handler *scmi_register_event_handler(u32 evt_key)
+{
+	int id;
+	u8 proto_id, evt_id;
+	struct scmi_registered_event *r_evt;
+	struct scmi_event_handler *hndl;
+
+	proto_id = KEY_XTRACT_PROTO_ID(evt_key);
+	evt_id = KEY_XTRACT_EVT_ID(evt_key);
+	r_evt = idr_find(&scmi_registered_events,
+			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
+	if (!r_evt)
+		return ERR_PTR(-EINVAL);
+
+	hndl = kzalloc(sizeof(*hndl), GFP_KERNEL);
+	if (!hndl)
+		return ERR_PTR(-ENOMEM);
+	hndl->r_evt = r_evt;
+	hndl->evt_key = evt_key;
+	BLOCKING_INIT_NOTIFIER_HEAD(&hndl->chain);
+	refcount_set(&hndl->users, 1);
+
+	/* Register freshly allocated event handler */
+	id = idr_alloc(&scmi_registered_events_handlers, hndl,
+		       evt_key, evt_key + 1, GFP_KERNEL);
+	if (id < 0) {
+		pr_err("SCMI Failed to allocate HNDL IDR - key:%X  err:%d\n",
+		       evt_key, id);
+		kfree(hndl);
+		return ERR_PTR(id);
+	}
+
+	return hndl;
+}
+
+/**
+ * scmi_unregister_event_handler  - Free the provided Event handler
+ *
+ * @hndl: The event handler structure to free
+ */
+static void scmi_unregister_event_handler(struct scmi_event_handler *hndl)
+{
+	idr_remove(&scmi_registered_events_handlers, hndl->evt_key);
+	kfree(hndl);
+}
+
+/**
+ * __scmi_event_handler_get_ops  - Get or create an event handler
+ *
+ * After having got exclusive access to the registered events map, searches for
+ * the desired handler matching the key: when found adjust refcount, when not
+ * create and register a new handler is asked to do so.
+ * Events generation is NOT enabled within this routine since at creation time
+ * we usually want to have all setup and registered before events starts
+ * flowing.
+ *
+ * @evt_key: The event key to use
+ * @create: A boolean flag to specify if a handler must be created
+ *	    when not found
+ */
+static inline struct scmi_event_handler *
+__scmi_event_handler_get_ops(u32 evt_key, bool create)
+{
+	struct scmi_event_handler *hndl = NULL;
+
+	mutex_lock(&scmi_registered_events_handler_mtx);
+	hndl = idr_find(&scmi_registered_events_handlers, evt_key);
+	if (hndl)
+		refcount_inc(&hndl->users);
+	else if (create)
+		hndl = scmi_register_event_handler(evt_key);
+	mutex_unlock(&scmi_registered_events_handler_mtx);
+
+	return hndl;
+}
+
+static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key)
+{
+	return __scmi_event_handler_get_ops(evt_key, false);
+}
+
+static struct scmi_event_handler *scmi_get_or_create_event_handler(u32 evt_key)
+{
+	return __scmi_event_handler_get_ops(evt_key, true);
+}
+
+/**
+ * __scmi_enable_events_ops  - Enable/disable events generation
+ *
+ * @hndl: The handler specifying the events enable/disable
+ * @action: The action to perform: true->Enable false->Disable
+ *
+ * Return: True when the required @action has been successfully executed
+ */
+static inline bool __scmi_enable_events_ops(struct scmi_event_handler *hndl,
+					    bool action)
+{
+	bool ret = true;
+
+	if (hndl->enabled != action) {
+		u8 evt_id;
+		u32 src_id;
+
+		evt_id = KEY_XTRACT_EVT_ID(hndl->evt_key);
+		src_id = KEY_XTRACT_SRC_ID(hndl->evt_key);
+		ret = hndl->r_evt->proto->ops->set_notify_enabled(evt_id,
+			      src_id != SCMI_ALL_SRC_IDS ? &src_id : NULL,
+			      action);
+		if (ret)
+			hndl->enabled = action;
+	}
+
+	return ret;
+}
+
+static bool scmi_enable_events(struct scmi_event_handler *hndl)
+{
+	return __scmi_enable_events_ops(hndl, true);
+}
+
+static bool scmi_disable_events(struct scmi_event_handler *hndl)
+{
+	return __scmi_enable_events_ops(hndl, false);
+}
+
+/**
+ * scmi_put_event_handler  - Put an event handler
+ *
+ * After having got exclusive access to the registered events map, update
+ * the refcount and if @hndl is no more in use by anyone:
+ *  - disable the events generation
+ *  - unregister and free the handler itself
+ *
+ * @hndl: The handler to act upon
+ */
+static void scmi_put_event_handler(struct scmi_event_handler *hndl)
+{
+	mutex_lock(&scmi_registered_events_handler_mtx);
+	if (refcount_dec_and_test(&hndl->users)) {
+		scmi_disable_events(hndl);
+		scmi_unregister_event_handler(hndl);
+	}
+	mutex_unlock(&scmi_registered_events_handler_mtx);
+}
+
+/**
+ * scmi_register_event_notifier  - Register a notifier_block for an event
+ *
+ * Generic helper to register a notifier_block attached to a protocol event.
+ *
+ * A notifier_block @nb will be registered for each distinct event identified
+ * by the tuple (proto_id, evt_id, src_id) on a dedicated notification chain
+ * so that:
+ *
+ *	(proto_X, evt_Y, src_Z) --> chain_X_Y_Z
+ *
+ * @src_id meaning is protocol specific and identifies the origin of the event
+ * (like domain_id, sensor_id and os forth); @src_id can be NULL to signify that
+ * the caller is interested in receiving notifications from ALL the available
+ * sources for that protocol OR simply that the protocol does not support
+ * distinct sources: in these cases @nb will be attached to a generic
+ * notification chain defined for ALL src_id of that proto_id/evt_id pair like:
+ *
+ *	(proto_X, evt_Y, NULL) --> chain_X_Y_ALL
+ *
+ * Any received event will be then dispatched to both such chains if at least
+ * one user had registered an @nb on them.
+ *
+ * Note also that notification chains are created dynamically: a new chain is
+ * created each time a new distinct tuple is detected and then the provided @nb
+ * is bound to such chain; at creation time the related SCMI notifications are
+ * also enabled: this way, only when at least one user has registered its
+ * interest for a specific event tuple, the underlying notification chain is
+ * created and the related notifications are enabled in the platform.
+ *
+ * @proto_id: Protocol ID
+ * @evt_id: Event ID
+ * @src_id: Source ID
+ * @nb: A standard notifier block to register for the specified event
+ *
+ * Return: Return 0 on Success
+ */
+int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
+				 struct notifier_block *nb)
+{
+	u32 evt_key;
+	struct scmi_event_handler *hndl;
+
+	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
+	hndl = scmi_get_or_create_event_handler(evt_key);
+	if (IS_ERR_OR_NULL(hndl))
+		return PTR_ERR(hndl);
+
+	blocking_notifier_chain_register(&hndl->chain, nb);
+
+	if (!scmi_enable_events(hndl)) {
+		pr_err("SCMI Failed to ENABLE events for key:%X !\n", evt_key);
+		blocking_notifier_chain_unregister(&hndl->chain, nb);
+		scmi_put_event_handler(hndl);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * scmi_unregister_event_notifier  - Unregister a notifier_block for an event
+ *
+ * Takes care to unregister the provided @nb from the notification chain
+ * associated to the specified event and, if there are no more users for the
+ * event handler, frees also the associated event handler structures.
+ *
+ * @proto_id: Protocol ID
+ * @evt_id: Event ID
+ * @src_id: Source ID
+ * @nb: The notifier_block to unregister
+ *
+ * Return: 0 on Success
+ */
+int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
+				   struct notifier_block *nb)
+{
+	u32 evt_key;
+	struct scmi_event_handler *hndl;
+
+	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
+	hndl = scmi_get_event_handler(evt_key);
+	if (IS_ERR_OR_NULL(hndl))
+		return -EINVAL;
+
+	blocking_notifier_chain_unregister(&hndl->chain, nb);
+
+	scmi_put_event_handler(hndl);
+	/*
+	 * If this was the last user callback for this handler, this last put
+	 * will force the handler to be freed.
+	 * Note that if a call_chain walk is ongoing it will be the call_chain
+	 * put request which will finally free the handler; note also that any
+	 * operation on the inner notifier_block chain is protected on its own.
+	 */
+	scmi_put_event_handler(hndl);
+
+	return 0;
+}
+
+/**
+ * scmi_notification_init  - Initializes Notification Core Support
+ *
+ * Return: 0 on Success
+ */
+int __init scmi_notification_init(void)
+{
+	scmi_notify_wq = alloc_workqueue("scmi_notify",
+					 WQ_UNBOUND | WQ_FREEZABLE,
+					 SCMI_NOTIF_MAX_ACTIVE_QUEUES);
+	if (!scmi_notify_wq)
+		return -ENOMEM;
+	idr_init(&scmi_registered_events);
+	idr_init(&scmi_registered_events_handlers);
+	idr_init(&scmi_registered_protocols);
+
+	scmi_notifications_initialized = true;
+
+	pr_info("SCMI Notifications enabled.\n");
+
+	return 0;
+}
+
+/**
+ * scmi_notification_exit  - Shutdown and clean Notification core
+ */
+void __exit scmi_notification_exit(void)
+{
+	int k;
+	struct scmi_event_handler *hndl;
+	struct scmi_registered_protocol_events_desc *pdesc;
+	struct scmi_registered_event *r_evt;
+
+	scmi_notifications_initialized = false;
+
+	/* Disable notifications' generation */
+	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
+		scmi_disable_events(hndl);
+
+	/* Let pending work complete */
+	destroy_workqueue(scmi_notify_wq);
+
+	/* Remove ALL events handlers */
+	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
+		scmi_put_event_handler(hndl);
+	idr_destroy(&scmi_registered_events_handlers);
+
+	/* Finally remove all registered events */
+	idr_for_each_entry(&scmi_registered_events, r_evt, k) {
+		idr_remove(&scmi_registered_events, k);
+		kfree(r_evt->report);
+		kfree(r_evt->scratch_isr);
+		kfree(r_evt->scratch_bh);
+		kfree(r_evt);
+	}
+	idr_destroy(&scmi_registered_events);
+
+	/* free any remaining protocol data */
+	idr_for_each_entry(&scmi_registered_protocols, pdesc, k) {
+		idr_remove(&scmi_registered_protocols, k);
+		scmi_deallocate_events_queue(pdesc->equeue);
+		kfree(pdesc);
+	}
+	idr_destroy(&scmi_registered_protocols);
+}
diff --git a/drivers/firmware/arm_scmi/notify.h b/drivers/firmware/arm_scmi/notify.h
new file mode 100644
index 000000000000..6fa07b244b14
--- /dev/null
+++ b/drivers/firmware/arm_scmi/notify.h
@@ -0,0 +1,79 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * System Control and Management Interface (SCMI) Message Protocol
+ * notification header file containing some definitions, structures
+ * and function prototypes related to SCMI Notification handling.
+ *
+ * Copyright (C) 2019 ARM Ltd.
+ */
+#ifndef _SCMI_NOTIFY_H
+#define _SCMI_NOTIFY_H
+
+#include <linux/bug.h>
+#include <linux/notifier.h>
+#include <linux/types.h>
+
+#define	SCMI_ALL_SRC_IDS		0x0000ffffL
+
+#define MAP_EVT_TO_ENABLE_CMD(id, map)			\
+({							\
+	int ret = -1;					\
+							\
+	if (likely((id) < ARRAY_SIZE((map))))		\
+		ret = (map)[(id)];			\
+	else						\
+		WARN(1, "UN-KNOWN evt_id:%d\n", (id));	\
+	ret;						\
+})
+
+/**
+ * scmi_event  - Describes an event to be supported
+ *
+ * Each SCMI protocol, during its initialization phase, can describe the events
+ * it wishes to support in a few struct scmi_event and pass them to the core
+ * using scmi_register_protocol_events().
+ *
+ * @evt_id: Event ID
+ * @max_payld_sz: Max possible size for the payload of a notif msg of this kind
+ * @max_report_sz: Max possible size for the report of a notif msg of this kind
+ */
+struct scmi_event {
+	u8	evt_id;
+	size_t	max_payld_sz;
+	size_t	max_report_sz;
+
+};
+
+/**
+ * scmi_protocol_event_ops  - Helpers called by notification core.
+ *
+ * These are called only in process context.
+ *
+ * @set_notify_enabled: Enable/disable the required evt_id/src_id notifications
+ *			using the proper custom protocol commands.
+ *			Return true if at least one the required src_id
+ *			has been successfully enabled/disabled
+ * @fill_custom_report: fills a custom event report from the provided
+ *			event message payld identifying the event
+ *			specific src_id.
+ *			Return NULL on failure otherwise @report now fully
+ *			populated
+ */
+struct scmi_protocol_event_ops {
+	bool (*set_notify_enabled)(u8 evt_id, const u32 *src_id, bool enabled);
+	void *(*fill_custom_report)(u8 evt_id, u64 timestamp, const void *payld,
+				    size_t payld_sz, void *report, u32 *src_id);
+};
+
+int scmi_notification_init(void);
+void scmi_notification_exit(void);
+int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
+				  const struct scmi_protocol_event_ops *ops,
+				  const struct scmi_event *evt, int num_events);
+int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
+				 struct notifier_block *nb);
+int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
+				   struct notifier_block *nb);
+int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts);
+
+#endif /* _SCMI_NOTIFY_H */
-- 
2.17.1

Re: [RFC PATCH 04/11] firmware: arm_scmi: Add core notifications support

[Cristian Marussi]

On 20/01/2020 12:23, Cristian Marussi wrote:
> Add basic support for SCMI Notifications, using Kernel notification chains
> mechanism.
> 
> Each SCMI Protocol has a dedicated events' queue and deferred worker which
> is in charge of delivering the notifications to the interested users
> invoking the users' registered callbacks.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  drivers/firmware/arm_scmi/Makefile |   2 +-
>  drivers/firmware/arm_scmi/common.h |   4 +
>  drivers/firmware/arm_scmi/driver.c |   2 +

Small nit, the driver.c 2-line change is unneeded here. Ignore them.

Thanks

Cristian
>  drivers/firmware/arm_scmi/notify.c | 904 +++++++++++++++++++++++++++++
>  drivers/firmware/arm_scmi/notify.h |  79 +++
>  5 files changed, 990 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/firmware/arm_scmi/notify.c
>  create mode 100644 drivers/firmware/arm_scmi/notify.h
> 
> diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
> index 5f298f00a82e..26587ea4661f 100644
> --- a/drivers/firmware/arm_scmi/Makefile
> +++ b/drivers/firmware/arm_scmi/Makefile
> @@ -1,6 +1,6 @@
>  # SPDX-License-Identifier: GPL-2.0-only
>  obj-y	= scmi-bus.o scmi-driver.o scmi-protocols.o
>  scmi-bus-y = bus.o
> -scmi-driver-y = driver.o
> +scmi-driver-y = driver.o notify.o
>  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o
>  obj-$(CONFIG_ARM_SCMI_POWER_DOMAIN) += scmi_pm_domain.o
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index df35358ff324..2621c05e9149 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -6,6 +6,8 @@
>   *
>   * Copyright (C) 2018 ARM Ltd.
>   */
> +#ifndef _SCMI_COMMON_H
> +#define _SCMI_COMMON_H
>  
>  #include <linux/bitfield.h>
>  #include <linux/completion.h>
> @@ -113,3 +115,5 @@ void scmi_setup_protocol_implemented(const struct scmi_handle *handle,
>  				     u8 *prot_imp);
>  
>  int scmi_base_protocol_init(struct scmi_handle *h);
> +
> +#endif /* _SCMI_COMMON_H */
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 28ed1f0cb417..a43fad29de11 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -350,12 +350,14 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>  
>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  {
> +	ktime_t ts;
>  	struct scmi_xfer *xfer;
>  	struct device *dev = cinfo->dev;
>  	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>  	struct scmi_xfers_info *minfo = &info->rx_minfo;
>  	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>  
> +	ts = ktime_get_boottime();
>  	xfer = scmi_xfer_get(cinfo->handle, minfo);
>  	if (IS_ERR(xfer)) {
>  		dev_err(dev, "failed to get free message slot (%ld)\n",
> diff --git a/drivers/firmware/arm_scmi/notify.c b/drivers/firmware/arm_scmi/notify.c
> new file mode 100644
> index 000000000000..da342f43021e
> --- /dev/null
> +++ b/drivers/firmware/arm_scmi/notify.c
> @@ -0,0 +1,904 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * System Control and Management Interface (SCMI) Notification support
> + *
> + * Copyright (C) 2019 ARM Ltd.
> + *
> + * SCMI Protocol specification allows the platform to signal events to
> + * interested agents via notification messages: this in an implementation
> + * of the dispatch and delivery of such notifications to the interested users
> + * inside the Linux kernel.
> + *
> + * Each SCMI Protocol implementation, during its initialization, registers with
> + * this core notification framework its set of supported events via
> + * @scmi_register_protocol_events(), while Kernel users interested in some
> + * specific event can register their associated callbacks providing the usual
> + * notifier_block descriptor, since the notification system internally supports
> + * events delivery using customized notification chains.
> + *
> + * Given the number of possible events defined by SCMI and the extensibility
> + * of the SCMI Protocol itself, such underlying notification chains are created
> + * and destroyed dynamically on demand depending on the number of users
> + * effectively registered for an event, so that no structures or chains are
> + * allocated until at least one user has registered a notifier_block for such
> + * event. Similarly, events' generation itself is enabled at the platform level
> + * only after at least one user has registered, and it is shutdown after the
> + * last user for that event has gone.
> + *
> + * An event is identified univocally by the tuple (proto_id, evt_id, src_id)
> + * and is served by its own dedicated notification chain: given that such chain
> + * is dynamically created, the registration API simply let the users associate
> + * their callbacks with the above tuple.
> + *
> + * Here proto_id and evt_id are simply the protocol_id and message_id numbers as
> + * reported in the SCMI Protocol specification, while src_id represents an
> + * optional, protocol dependent, source identifier (like domain_id, perf_id
> + * or sensor_id and so forth), so that a user can register its callbacks for a
> + * particular event coming only from a defined source (like CPU vs GPU).
> + * When the source is not specified the user callback will be registered for
> + * all existing sources for that event (if any).
> + *
> + * Upon reception of a notification message from the platform the SCMI RX ISR
> + * passes the received message payload and some ancillary information (including
> + * an arrival timestamp in nanoseconds) to the core via @scmi_notify(), which,
> + * in turn, after having looked up the event in the registered events mapping,
> + * pushes the event-data itself on a protocol dedicated kfifo queue for deferred
> + * processing.
> + *
> + * Such dedicated protocols' queues are allocated once for all at initialization
> + * time, together with a dedicated work_item running the common delivery logic
> + * of @scmi_events_dispatcher(), so that each protocol has it own dedicated
> + * worker which, once kicked by the ISR, takes care to empty is own dedicated
> + * queue deliverying the queued items into the proper notification chain.
> + * Note that since the underlying cmwq workers run one distinct work_item per
> + * protocol and there are a number of max-active workers equal to the number of
> + * protocols, notifications processing can proceed concurrently only between
> + * events belonging to different protocols, while delivery of events within the
> + * same protocol is still strictly sequentially ordered by time of arrival; this
> + * separation effectively avoids the possibility that one particularly verbose
> + * protocol, flooding the queues with events, can cause other protocols' events
> + * to be lost or their processing starved.
> + *
> + * Event's information is converted to a custom per-event report struct before
> + * being fed as void *data param to the user callback embedded in the registered
> + * notifier_block so that it looks like as follow from the user perspective:
> + *
> + * int user_cb(struct notifier_block *nb, unsigned long event_id, void *report)
> + *
> + *
> + */
> +
> +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
> +
> +#include <linux/bug.h>
> +#include <linux/bitfield.h>
> +#include <linux/err.h>
> +#include <linux/idr.h>
> +#include <linux/kernel.h>
> +#include <linux/kfifo.h>
> +#include <linux/notifier.h>
> +#include <linux/refcount.h>
> +#include <linux/slab.h>
> +#include <linux/scmi_protocol.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +#include "notify.h"
> +
> +#define SCMI_NOTIF_MAX_ACTIVE_QUEUES	7
> +
> +/*
> + * Builds an unsigned 32bit key from the given input tuple to be used as a key
> + * in IDR mappings; note that if __src_p is passed in as NULL, the returned key
> + * is built using SCMI_ALL_SRC_IDS as src_id.
> + */
> +#define MAKE_EVT_KEY(__proto, __evt, __src_p)				\
> +	({								\
> +		u32 __evt_key, __src_id;				\
> +									\
> +		__src_id = ((__src_p)) ?				\
> +			    (*((u32 *)(__src_p)) & SCMI_ALL_SRC_IDS) :	\
> +			    SCMI_ALL_SRC_IDS;				\
> +		__evt_key = ((__proto) << 24) |				\
> +				((__evt) << 16) | __src_id;		\
> +		__evt_key;						\
> +	})
> +
> +#define PROTO_ID_MASK			GENMASK(31, 24)
> +#define EVT_ID_MASK			GENMASK(23, 16)
> +#define SRC_ID_MASK			GENMASK(15, 0)
> +#define KEY_XTRACT_PROTO_ID(key)	FIELD_GET(PROTO_ID_MASK, (key))
> +#define KEY_XTRACT_EVT_ID(key)		FIELD_GET(EVT_ID_MASK, (key))
> +#define KEY_XTRACT_SRC_ID(key)		FIELD_GET(SRC_ID_MASK, (key))
> +
> +/**
> + * events_queue  - Describes a queue and its associated worker
> + *
> + * Each protocol has its own dedicated events_queue descriptor.
> + *
> + * @sz: Size in bytes of the related kfifo
> + * @kfifo: A dedicated Kernel kfifo
> + * @notify_work: A custom work item bound to this queue
> + * @wq: A reference to the related workqueue
> + */
> +struct events_queue {
> +	size_t			sz;
> +	struct kfifo		kfifo;
> +	struct work_struct	notify_work;
> +	struct workqueue_struct	*wq;
> +};
> +
> +/**
> + * scmi_registered_protocol_events_desc  - Protocol Specific information
> + *
> + * All protocols that registers at least an event have their protocol-specific
> + * information stored here, together with a pointer to the allocated
> + * events_queue.
> + *
> + * @id: Protocol ID
> + * @ops: Protocol specific and event-related operations
> + * @equeue: A reference to the associated per-protocol events_queue
> + */
> +struct scmi_registered_protocol_events_desc {
> +	u8					id;
> +	const struct scmi_protocol_event_ops	*ops;
> +	struct events_queue			*equeue;
> +};
> +
> +/**
> + * scmi_registered_event  - Event Specific Information
> + *
> + * All registered events are represented by one of these structures.
> + *
> + * @proto: A reference to the associated protocol descriptor
> + * @evt: A reference to the associated event descriptor (as provided at
> + *       registration time)
> + * @scratch_isr: A pre-allocated buffer to be used as a scratch area by ISR
> + * @scratch_isr: A pre-allocated buffer to be used as a scratch area by the
> + *		 deferred worker
> + * @report: A pre-allocated buffer used by the deferred workr to fill a
> + *	    customized event report
> + */
> +struct scmi_registered_event {
> +	struct scmi_registered_protocol_events_desc	*proto;
> +	const struct scmi_event				*evt;
> +	void						*scratch_isr;
> +	void						*scratch_bh;
> +	void						*report;
> +};
> +
> +/**
> + * scmi_event_handler  - Event handler information
> + *
> + * This structure collects all the information needed to process a received
> + * event, calling all the registered notifier callbacks.
> + * Note this an event handler is associated to a tuple of values like:
> + *	(proto_id, evt_id, src_id)
> + * through the means of a key and an IDR mapping.
> + *
> + * @evt_key: The unsigned 32bit key associated to this descriptor in the
> + *	     related IDR mapping
> + * @r_evt: A reference to the underlying registered event
> + * @chain: The notification chain dedicated to this specific event tuple
> + */
> +struct scmi_event_handler {
> +	u32				evt_key;
> +	bool				enabled;
> +	refcount_t			users;
> +	struct scmi_registered_event	*r_evt;
> +	struct blocking_notifier_head	chain;
> +};
> +
> +/**
> + * scmi_event_header  - A utility header
> + *
> + * This header is prepended to each received event message payload before
> + * being queued on the related events_queue, and describes the attached event
> + * message.
> + *
> + * @timestamp: The timestamp, innanoseconds (boottime), which was associated
> + *	       to this event as soon as it entered the SCMI RX ISR
> + * @proto_id: Protocol ID
> + * @evt_id: Event ID (corresponds to the Event MsgID for this Protocol)
> + * @payld_sz: Effective size of the attached message payload which follows
> + * @payld: A reference to the included Event payload
> + */
> +struct scmi_event_header {
> +	u64	timestamp;
> +	u8	proto_id;
> +	u8	evt_id;
> +	size_t	payld_sz;
> +	u8	payld[];
> +} __packed;
> +
> +/*
> + * A few IDR maps to track:
> + *
> + * - @scmi_registered_events: All event's descriptors registered by the
> + *			      protocols, together with their ancillary data
> + * - @scmi_registered_events_handlers: All event's handlers descriptors, created
> + *				       to collect all the users' notifier_block
> + *				       callbacks and related notification chains
> + * - @scmi_registered_protocols: All protocol-level specific information related
> + *				 to events' handling
> + */
> +static struct idr scmi_registered_events;
> +static struct idr scmi_registered_events_handlers;
> +static DEFINE_MUTEX(scmi_registered_events_handler_mtx);
> +static struct idr scmi_registered_protocols;
> +
> +/* Common Kernel cmwq workqueue used by notifications core */
> +static struct workqueue_struct *scmi_notify_wq;
> +
> +static bool scmi_notifications_initialized;
> +
> +static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key);
> +static void scmi_put_event_handler(struct scmi_event_handler *hndl);
> +
> +/**
> + * scmi_discard_bad_evt_payloadi() - Discard data from a kfifo
> + *
> + * @kq: The kfifo to act on
> + * @count: Number of bytes to flush
> + */
> +static inline void scmi_discard_bad_evt_payload(struct kfifo *kq,
> +						const unsigned int count)
> +{
> +	int i = 0;
> +
> +	pr_warn("SCMI Notification WQ :: skipping bad EVT Payload - %d bytes\n",
> +		count);
> +	/* Discard stale pending queued payload. */
> +	for (i = 0; i < count; i++)
> +		kfifo_skip(kq);
> +}
> +
> +/**
> + * scmi_lookup_and_call_event_chain  - Lookup the proper chain and call it
> + *
> + * @evt_key: The event key to use to lookup the related notification chain
> + * @report: The customized event-specific report to pass down to the callbacks
> + *	    as their *data parameter.
> + */
> +static inline void scmi_lookup_and_call_event_chain(u32 evt_key, void *report)
> +{
> +	int ret;
> +	struct scmi_event_handler *hndl;
> +
> +	hndl = scmi_get_event_handler(evt_key);
> +	if (IS_ERR_OR_NULL(hndl))
> +		return;
> +
> +	ret = blocking_notifier_call_chain(&hndl->chain,
> +					   KEY_XTRACT_EVT_ID(evt_key),
> +					   report);
> +	/* Notifiers are NOT supposed to cut the chain */
> +	WARN_ON_ONCE(ret & NOTIFY_STOP_MASK);
> +
> +	scmi_put_event_handler(hndl);
> +}
> +
> +/**
> + * scmi_events_dispatcher  - Common worker logic for all work items.
> + *
> + * In turn:
> + *  1. dequeue one pending RX notification (queued in SCMI RX ISR context)
> + *  2. generate a custom event report from the received event message
> + *  3. lookup for any registered ALL_SRC_IDs handler
> + *     - > call the related notification chain passing in the report
> + *  4. lookup for any registered specific SRC_ID handler
> + *     - > call the related notification chain passing in the report
> + *
> + * Note that:
> + * - a dedicated per-protocol kfifo queue is used: in this way an anomalous
> + *   flood of events cannot saturate other protocols' queues.
> + *
> + * - each per-protocol queue is associated to a distinct work_item, which
> + *   means, in turn, that:
> + *   + all protocol can process their dedicated queues concurrently
> + *     (since scmi_notify_wq:max_active > 1)
> + *   + anyway at most one worker instance is allowed to run on the same queue
> + *     concurrently: this ensures that we can have only one concurrent
> + *     reader/writer on the associated kfifo (needed for lock-less access)
> + *
> + * @work: The work item to use, which is associated to the proper events_queue
> + */
> +static void scmi_events_dispatcher(struct work_struct *work)
> +{
> +	struct events_queue *equeue;
> +	struct scmi_event_header eh;
> +
> +	equeue = container_of(work, struct events_queue, notify_work);
> +	while (kfifo_out(&equeue->kfifo, &eh, sizeof(eh))) {
> +		u32 src_id, evt_key;
> +		unsigned int outs;
> +		struct scmi_registered_event *r_evt;
> +		void *report = NULL;
> +
> +		evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, NULL);
> +		r_evt = idr_find(&scmi_registered_events, evt_key);
> +		if (!r_evt) {
> +			scmi_discard_bad_evt_payload(&equeue->kfifo,
> +						     eh.payld_sz);
> +			continue;
> +		}
> +
> +		outs = kfifo_out(&equeue->kfifo, r_evt->scratch_bh,
> +				 eh.payld_sz);
> +		if (outs != eh.payld_sz) {
> +			pr_warn("SCMI Notification WQ :: SKIP corrupted EVT Payload.\n");
> +			continue;
> +		}
> +
> +		/* Reset and fill custom report */
> +		memset(r_evt->report, 0x00, r_evt->evt->max_report_sz);
> +		report = r_evt->proto->ops->fill_custom_report(eh.evt_id,
> +							eh.timestamp,
> +							r_evt->scratch_bh,
> +							eh.payld_sz,
> +							r_evt->report,
> +							&src_id);
> +		if (!report)
> +			continue;
> +
> +		/* At first search for a generic ALL src_ids handler... */
> +		scmi_lookup_and_call_event_chain(evt_key, report);
> +		if (src_id != SCMI_ALL_SRC_IDS) {
> +			u32 *__sub_p = &src_id;
> +
> +			evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, __sub_p);
> +			scmi_lookup_and_call_event_chain(evt_key, report);
> +		}
> +	}
> +}
> +
> +/**
> + * scmi_notify  - Queues a notification for further deferred processing
> + *
> + * This is called in interrupt context to queue a received event for
> + * deferred processing.
> + *
> + * @proto_id: Protocol ID
> + * @evt_id: Event ID (msgID)
> + * @buf: Event Message Payload (without the header)
> + * @len: Event Message Payload size
> + * @ts: RX Timestamp in nanoseconds (boottime)
> + *
> + * Return: 0 on Success
> + */
> +int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts)
> +{
> +	struct scmi_registered_event *r_evt;
> +	struct scmi_event_header *eh;
> +
> +	if (unlikely(!scmi_notifications_initialized))
> +		return 0;
> +
> +	r_evt = idr_find(&scmi_registered_events,
> +			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
> +	if (unlikely(!r_evt || !r_evt->proto->equeue))
> +		return -EINVAL;
> +
> +	if (unlikely(len > r_evt->evt->max_payld_sz)) {
> +		pr_err("SCMI Notification discard badly sized message\n");
> +		return -EINVAL;
> +	}
> +	if (unlikely(kfifo_avail(&r_evt->proto->equeue->kfifo) <
> +		     sizeof(*eh) + len)) {
> +		pr_warn("SCMI Notification queue full: dropping proto_id:%d  evt_id:%d  ts:%lld\n",
> +			proto_id, evt_id, ts);
> +		return -ENOMEM;
> +	}
> +
> +	/*
> +	 * Note that we can use the per protocol kfifo in a lock-less manner
> +	 * since we have only one concurrent reader and writer but, in order
> +	 * to avoid any trouble on the reader side, here we perform one single
> +	 * write, so we have to collate event header and payload in a scratch
> +	 * area at first.
> +	 */
> +	eh = r_evt->scratch_isr;
> +	eh->timestamp = ts;
> +	eh->proto_id = proto_id;
> +	eh->evt_id = evt_id;
> +	eh->payld_sz = len;
> +	memcpy(eh->payld, buf, eh->payld_sz);
> +	kfifo_in(&r_evt->proto->equeue->kfifo, eh, sizeof(*eh) + eh->payld_sz);
> +	queue_work(r_evt->proto->equeue->wq,
> +		   &r_evt->proto->equeue->notify_work);
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_allocate_events_queue  - Allocate an events_queue descriptor
> + *
> + * @sz: Size of the kfifo to initialize
> + *
> + * Return: A valid pointer to the allocated events_queue on Success
> + */
> +static struct events_queue *scmi_allocate_events_queue(size_t sz)
> +{
> +	int ret;
> +	struct events_queue *equeue;
> +
> +	equeue = kzalloc(sizeof(*equeue), GFP_KERNEL);
> +	if (!equeue)
> +		return ERR_PTR(-ENOMEM);
> +	ret = kfifo_alloc(&equeue->kfifo, sz, GFP_KERNEL);
> +	if (ret) {
> +		kfree(equeue);
> +		return ERR_PTR(ret);
> +	}
> +	equeue->sz = sz;
> +	INIT_WORK(&equeue->notify_work, scmi_events_dispatcher);
> +	equeue->wq = scmi_notify_wq;
> +
> +	return equeue;
> +}
> +
> +/**
> + * scmi_deallocate_events_queue  - Deallocate am events_queue descriptor
> + *
> + * @equeue: The events_queue to free
> + */
> +static void scmi_deallocate_events_queue(struct events_queue *equeue)
> +{
> +	kfifo_free(&equeue->kfifo);
> +	kfree(equeue);
> +}
> +
> +/**
> + * scmi_allocate_registered_protocol_desc  - Allocate a registered protocol
> + * events' descriptor
> + *
> + * Used to keep protocol specific information related to events handling for any
> + * protocl which has registered at least one event.
> + *
> + * @proto_id: Protocol ID
> + * @queue_sz: Size of the associated queue to allocate
> + * @ops: Pointer to a struct holding references to protocol specific helpers
> + *	 needed during events handling
> + */
> +static struct scmi_registered_protocol_events_desc *
> +scmi_allocate_registered_protocol_desc(u8 proto_id, size_t queue_sz,
> +				const struct scmi_protocol_event_ops *ops)
> +{
> +	int ret;
> +	struct scmi_registered_protocol_events_desc *pdesc;
> +
> +	pdesc = idr_find(&scmi_registered_protocols, proto_id);
> +	if (pdesc)
> +		return pdesc;
> +
> +	pdesc = kzalloc(sizeof(*pdesc), GFP_KERNEL);
> +	if (!pdesc)
> +		return ERR_PTR(-ENOMEM);
> +	pdesc->id = proto_id;
> +	pdesc->ops = ops;
> +
> +	pdesc->equeue = scmi_allocate_events_queue(queue_sz);
> +	if (IS_ERR(pdesc->equeue)) {
> +		kfree(pdesc);
> +		return ERR_CAST(pdesc->equeue);
> +	}
> +
> +	ret = idr_alloc(&scmi_registered_protocols, pdesc,
> +			pdesc->id, pdesc->id + 1, GFP_KERNEL);
> +	if (ret < 0) {
> +		pr_err("SCMI Failed to allocate PDESC IDR - key:%d - err:%d\n",
> +		       pdesc->id, ret);
> +		scmi_deallocate_events_queue(pdesc->equeue);
> +		kfree(pdesc);
> +		return ERR_PTR(ret);
> +	}
> +
> +	return pdesc;
> +}
> +
> +/**
> + * scmi_register_protocol_events  - Register Protocol Events with the core
> + *
> + * Used by SCMI Protocols initialization code to register with the notification
> + * core the list of supported events and their description: this takes care to
> + * pre-allocate all needed scratch buffers and link the proper event queue to
> + * this event.
> + *
> + * @proto_id: Protocol ID
> + * @queue_sz: Size in bytes of the associated queue to be allocated
> + * @ops: Protocol specific event-related operations
> + * @evt: Event descriptor array
> + * @num_events: Number of events in @evt array
> + *
> + * Return: 0 on Success
> + */
> +int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
> +				  const struct scmi_protocol_event_ops *ops,
> +				  const struct scmi_event *evt, int num_events)
> +{
> +	int i;
> +	struct scmi_registered_protocol_events_desc *pdesc;
> +
> +	if (!scmi_notifications_initialized)
> +		return -EAGAIN;
> +
> +	if (!ops || !evt)
> +		return -EINVAL;
> +
> +	pdesc = scmi_allocate_registered_protocol_desc(proto_id, queue_sz, ops);
> +	if (IS_ERR(pdesc))
> +		return -ENOMEM;
> +
> +	for (i = 0; i < num_events; i++, evt++) {
> +		int ret;
> +		u32 key;
> +		struct scmi_registered_event *r_evt;
> +
> +		r_evt = kzalloc(sizeof(*r_evt), GFP_KERNEL);
> +		if (!r_evt)
> +			continue;
> +		r_evt->proto = pdesc;
> +		r_evt->evt = evt;
> +
> +		r_evt->scratch_isr = kzalloc(sizeof(struct scmi_event_header) +
> +					     evt->max_payld_sz, GFP_KERNEL);
> +		if (!r_evt->scratch_isr)
> +			goto continue_isr_fail;
> +
> +		r_evt->scratch_bh = kzalloc(evt->max_payld_sz, GFP_KERNEL);
> +		if (!r_evt->scratch_bh)
> +			goto continue_bh_fail;
> +
> +		r_evt->report = kzalloc(evt->max_report_sz, GFP_KERNEL);
> +		if (!r_evt->report)
> +			goto continue_report_fail;
> +
> +		key = MAKE_EVT_KEY(r_evt->proto->id, evt->evt_id, NULL);
> +		ret = idr_alloc(&scmi_registered_events, r_evt,
> +				key, key + 1, GFP_KERNEL);
> +		if (ret < 0) {
> +			pr_err("SCMI Failed to allocate EVENT IDR - key:%X - err:%d\n",
> +			       key, ret);
> +			goto continue_idr_fail;
> +		}
> +
> +		pr_info("SCMI Notification registered event - key:%X\n", key);
> +		continue;
> +
> +	/* yes, this is not nice ... */
> +continue_idr_fail:
> +		kfree(r_evt->report);
> +continue_report_fail:
> +		kfree(r_evt->scratch_bh);
> +continue_bh_fail:
> +		kfree(r_evt->scratch_isr);
> +continue_isr_fail:
> +		kfree(r_evt);
> +	}
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_register_event_handler  - Allocate an Event handler
> + *
> + * Allocate an event handler and related notification chain associated with
> + * the event identified by the provided event key. Fails if the associated
> + * event is unknown to the core (i.e. it had not been successfully registered
> + * upfront by some protocol)
> + *
> + * @evt_key: 32bit key uniquely bind to the event identified by the tuple
> + *	     (proto_id, evt_id, src_id)
> + *
> + * Return: the freshly allocated structure on Success
> + */
> +static struct scmi_event_handler *scmi_register_event_handler(u32 evt_key)
> +{
> +	int id;
> +	u8 proto_id, evt_id;
> +	struct scmi_registered_event *r_evt;
> +	struct scmi_event_handler *hndl;
> +
> +	proto_id = KEY_XTRACT_PROTO_ID(evt_key);
> +	evt_id = KEY_XTRACT_EVT_ID(evt_key);
> +	r_evt = idr_find(&scmi_registered_events,
> +			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
> +	if (!r_evt)
> +		return ERR_PTR(-EINVAL);
> +
> +	hndl = kzalloc(sizeof(*hndl), GFP_KERNEL);
> +	if (!hndl)
> +		return ERR_PTR(-ENOMEM);
> +	hndl->r_evt = r_evt;
> +	hndl->evt_key = evt_key;
> +	BLOCKING_INIT_NOTIFIER_HEAD(&hndl->chain);
> +	refcount_set(&hndl->users, 1);
> +
> +	/* Register freshly allocated event handler */
> +	id = idr_alloc(&scmi_registered_events_handlers, hndl,
> +		       evt_key, evt_key + 1, GFP_KERNEL);
> +	if (id < 0) {
> +		pr_err("SCMI Failed to allocate HNDL IDR - key:%X  err:%d\n",
> +		       evt_key, id);
> +		kfree(hndl);
> +		return ERR_PTR(id);
> +	}
> +
> +	return hndl;
> +}
> +
> +/**
> + * scmi_unregister_event_handler  - Free the provided Event handler
> + *
> + * @hndl: The event handler structure to free
> + */
> +static void scmi_unregister_event_handler(struct scmi_event_handler *hndl)
> +{
> +	idr_remove(&scmi_registered_events_handlers, hndl->evt_key);
> +	kfree(hndl);
> +}
> +
> +/**
> + * __scmi_event_handler_get_ops  - Get or create an event handler
> + *
> + * After having got exclusive access to the registered events map, searches for
> + * the desired handler matching the key: when found adjust refcount, when not
> + * create and register a new handler is asked to do so.
> + * Events generation is NOT enabled within this routine since at creation time
> + * we usually want to have all setup and registered before events starts
> + * flowing.
> + *
> + * @evt_key: The event key to use
> + * @create: A boolean flag to specify if a handler must be created
> + *	    when not found
> + */
> +static inline struct scmi_event_handler *
> +__scmi_event_handler_get_ops(u32 evt_key, bool create)
> +{
> +	struct scmi_event_handler *hndl = NULL;
> +
> +	mutex_lock(&scmi_registered_events_handler_mtx);
> +	hndl = idr_find(&scmi_registered_events_handlers, evt_key);
> +	if (hndl)
> +		refcount_inc(&hndl->users);
> +	else if (create)
> +		hndl = scmi_register_event_handler(evt_key);
> +	mutex_unlock(&scmi_registered_events_handler_mtx);
> +
> +	return hndl;
> +}
> +
> +static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key)
> +{
> +	return __scmi_event_handler_get_ops(evt_key, false);
> +}
> +
> +static struct scmi_event_handler *scmi_get_or_create_event_handler(u32 evt_key)
> +{
> +	return __scmi_event_handler_get_ops(evt_key, true);
> +}
> +
> +/**
> + * __scmi_enable_events_ops  - Enable/disable events generation
> + *
> + * @hndl: The handler specifying the events enable/disable
> + * @action: The action to perform: true->Enable false->Disable
> + *
> + * Return: True when the required @action has been successfully executed
> + */
> +static inline bool __scmi_enable_events_ops(struct scmi_event_handler *hndl,
> +					    bool action)
> +{
> +	bool ret = true;
> +
> +	if (hndl->enabled != action) {
> +		u8 evt_id;
> +		u32 src_id;
> +
> +		evt_id = KEY_XTRACT_EVT_ID(hndl->evt_key);
> +		src_id = KEY_XTRACT_SRC_ID(hndl->evt_key);
> +		ret = hndl->r_evt->proto->ops->set_notify_enabled(evt_id,
> +			      src_id != SCMI_ALL_SRC_IDS ? &src_id : NULL,
> +			      action);
> +		if (ret)
> +			hndl->enabled = action;
> +	}
> +
> +	return ret;
> +}
> +
> +static bool scmi_enable_events(struct scmi_event_handler *hndl)
> +{
> +	return __scmi_enable_events_ops(hndl, true);
> +}
> +
> +static bool scmi_disable_events(struct scmi_event_handler *hndl)
> +{
> +	return __scmi_enable_events_ops(hndl, false);
> +}
> +
> +/**
> + * scmi_put_event_handler  - Put an event handler
> + *
> + * After having got exclusive access to the registered events map, update
> + * the refcount and if @hndl is no more in use by anyone:
> + *  - disable the events generation
> + *  - unregister and free the handler itself
> + *
> + * @hndl: The handler to act upon
> + */
> +static void scmi_put_event_handler(struct scmi_event_handler *hndl)
> +{
> +	mutex_lock(&scmi_registered_events_handler_mtx);
> +	if (refcount_dec_and_test(&hndl->users)) {
> +		scmi_disable_events(hndl);
> +		scmi_unregister_event_handler(hndl);
> +	}
> +	mutex_unlock(&scmi_registered_events_handler_mtx);
> +}
> +
> +/**
> + * scmi_register_event_notifier  - Register a notifier_block for an event
> + *
> + * Generic helper to register a notifier_block attached to a protocol event.
> + *
> + * A notifier_block @nb will be registered for each distinct event identified
> + * by the tuple (proto_id, evt_id, src_id) on a dedicated notification chain
> + * so that:
> + *
> + *	(proto_X, evt_Y, src_Z) --> chain_X_Y_Z
> + *
> + * @src_id meaning is protocol specific and identifies the origin of the event
> + * (like domain_id, sensor_id and os forth); @src_id can be NULL to signify that
> + * the caller is interested in receiving notifications from ALL the available
> + * sources for that protocol OR simply that the protocol does not support
> + * distinct sources: in these cases @nb will be attached to a generic
> + * notification chain defined for ALL src_id of that proto_id/evt_id pair like:
> + *
> + *	(proto_X, evt_Y, NULL) --> chain_X_Y_ALL
> + *
> + * Any received event will be then dispatched to both such chains if at least
> + * one user had registered an @nb on them.
> + *
> + * Note also that notification chains are created dynamically: a new chain is
> + * created each time a new distinct tuple is detected and then the provided @nb
> + * is bound to such chain; at creation time the related SCMI notifications are
> + * also enabled: this way, only when at least one user has registered its
> + * interest for a specific event tuple, the underlying notification chain is
> + * created and the related notifications are enabled in the platform.
> + *
> + * @proto_id: Protocol ID
> + * @evt_id: Event ID
> + * @src_id: Source ID
> + * @nb: A standard notifier block to register for the specified event
> + *
> + * Return: Return 0 on Success
> + */
> +int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
> +				 struct notifier_block *nb)
> +{
> +	u32 evt_key;
> +	struct scmi_event_handler *hndl;
> +
> +	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
> +	hndl = scmi_get_or_create_event_handler(evt_key);
> +	if (IS_ERR_OR_NULL(hndl))
> +		return PTR_ERR(hndl);
> +
> +	blocking_notifier_chain_register(&hndl->chain, nb);
> +
> +	if (!scmi_enable_events(hndl)) {
> +		pr_err("SCMI Failed to ENABLE events for key:%X !\n", evt_key);
> +		blocking_notifier_chain_unregister(&hndl->chain, nb);
> +		scmi_put_event_handler(hndl);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_unregister_event_notifier  - Unregister a notifier_block for an event
> + *
> + * Takes care to unregister the provided @nb from the notification chain
> + * associated to the specified event and, if there are no more users for the
> + * event handler, frees also the associated event handler structures.
> + *
> + * @proto_id: Protocol ID
> + * @evt_id: Event ID
> + * @src_id: Source ID
> + * @nb: The notifier_block to unregister
> + *
> + * Return: 0 on Success
> + */
> +int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
> +				   struct notifier_block *nb)
> +{
> +	u32 evt_key;
> +	struct scmi_event_handler *hndl;
> +
> +	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
> +	hndl = scmi_get_event_handler(evt_key);
> +	if (IS_ERR_OR_NULL(hndl))
> +		return -EINVAL;
> +
> +	blocking_notifier_chain_unregister(&hndl->chain, nb);
> +
> +	scmi_put_event_handler(hndl);
> +	/*
> +	 * If this was the last user callback for this handler, this last put
> +	 * will force the handler to be freed.
> +	 * Note that if a call_chain walk is ongoing it will be the call_chain
> +	 * put request which will finally free the handler; note also that any
> +	 * operation on the inner notifier_block chain is protected on its own.
> +	 */
> +	scmi_put_event_handler(hndl);
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_notification_init  - Initializes Notification Core Support
> + *
> + * Return: 0 on Success
> + */
> +int __init scmi_notification_init(void)
> +{
> +	scmi_notify_wq = alloc_workqueue("scmi_notify",
> +					 WQ_UNBOUND | WQ_FREEZABLE,
> +					 SCMI_NOTIF_MAX_ACTIVE_QUEUES);
> +	if (!scmi_notify_wq)
> +		return -ENOMEM;
> +	idr_init(&scmi_registered_events);
> +	idr_init(&scmi_registered_events_handlers);
> +	idr_init(&scmi_registered_protocols);
> +
> +	scmi_notifications_initialized = true;
> +
> +	pr_info("SCMI Notifications enabled.\n");
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_notification_exit  - Shutdown and clean Notification core
> + */
> +void __exit scmi_notification_exit(void)
> +{
> +	int k;
> +	struct scmi_event_handler *hndl;
> +	struct scmi_registered_protocol_events_desc *pdesc;
> +	struct scmi_registered_event *r_evt;
> +
> +	scmi_notifications_initialized = false;
> +
> +	/* Disable notifications' generation */
> +	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
> +		scmi_disable_events(hndl);
> +
> +	/* Let pending work complete */
> +	destroy_workqueue(scmi_notify_wq);
> +
> +	/* Remove ALL events handlers */
> +	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
> +		scmi_put_event_handler(hndl);
> +	idr_destroy(&scmi_registered_events_handlers);
> +
> +	/* Finally remove all registered events */
> +	idr_for_each_entry(&scmi_registered_events, r_evt, k) {
> +		idr_remove(&scmi_registered_events, k);
> +		kfree(r_evt->report);
> +		kfree(r_evt->scratch_isr);
> +		kfree(r_evt->scratch_bh);
> +		kfree(r_evt);
> +	}
> +	idr_destroy(&scmi_registered_events);
> +
> +	/* free any remaining protocol data */
> +	idr_for_each_entry(&scmi_registered_protocols, pdesc, k) {
> +		idr_remove(&scmi_registered_protocols, k);
> +		scmi_deallocate_events_queue(pdesc->equeue);
> +		kfree(pdesc);
> +	}
> +	idr_destroy(&scmi_registered_protocols);
> +}
> diff --git a/drivers/firmware/arm_scmi/notify.h b/drivers/firmware/arm_scmi/notify.h
> new file mode 100644
> index 000000000000..6fa07b244b14
> --- /dev/null
> +++ b/drivers/firmware/arm_scmi/notify.h
> @@ -0,0 +1,79 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * System Control and Management Interface (SCMI) Message Protocol
> + * notification header file containing some definitions, structures
> + * and function prototypes related to SCMI Notification handling.
> + *
> + * Copyright (C) 2019 ARM Ltd.
> + */
> +#ifndef _SCMI_NOTIFY_H
> +#define _SCMI_NOTIFY_H
> +
> +#include <linux/bug.h>
> +#include <linux/notifier.h>
> +#include <linux/types.h>
> +
> +#define	SCMI_ALL_SRC_IDS		0x0000ffffL
> +
> +#define MAP_EVT_TO_ENABLE_CMD(id, map)			\
> +({							\
> +	int ret = -1;					\
> +							\
> +	if (likely((id) < ARRAY_SIZE((map))))		\
> +		ret = (map)[(id)];			\
> +	else						\
> +		WARN(1, "UN-KNOWN evt_id:%d\n", (id));	\
> +	ret;						\
> +})
> +
> +/**
> + * scmi_event  - Describes an event to be supported
> + *
> + * Each SCMI protocol, during its initialization phase, can describe the events
> + * it wishes to support in a few struct scmi_event and pass them to the core
> + * using scmi_register_protocol_events().
> + *
> + * @evt_id: Event ID
> + * @max_payld_sz: Max possible size for the payload of a notif msg of this kind
> + * @max_report_sz: Max possible size for the report of a notif msg of this kind
> + */
> +struct scmi_event {
> +	u8	evt_id;
> +	size_t	max_payld_sz;
> +	size_t	max_report_sz;
> +
> +};
> +
> +/**
> + * scmi_protocol_event_ops  - Helpers called by notification core.
> + *
> + * These are called only in process context.
> + *
> + * @set_notify_enabled: Enable/disable the required evt_id/src_id notifications
> + *			using the proper custom protocol commands.
> + *			Return true if at least one the required src_id
> + *			has been successfully enabled/disabled
> + * @fill_custom_report: fills a custom event report from the provided
> + *			event message payld identifying the event
> + *			specific src_id.
> + *			Return NULL on failure otherwise @report now fully
> + *			populated
> + */
> +struct scmi_protocol_event_ops {
> +	bool (*set_notify_enabled)(u8 evt_id, const u32 *src_id, bool enabled);
> +	void *(*fill_custom_report)(u8 evt_id, u64 timestamp, const void *payld,
> +				    size_t payld_sz, void *report, u32 *src_id);
> +};
> +
> +int scmi_notification_init(void);
> +void scmi_notification_exit(void);
> +int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
> +				  const struct scmi_protocol_event_ops *ops,
> +				  const struct scmi_event *evt, int num_events);
> +int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
> +				 struct notifier_block *nb);
> +int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
> +				   struct notifier_block *nb);
> +int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts);
> +
> +#endif /* _SCMI_NOTIFY_H */
> 

Re: [RFC PATCH 04/11] firmware: arm_scmi: Add core notifications support

[Jonathan Cameron]

On Mon, 20 Jan 2020 12:23:26 +0000
Cristian Marussi <cristian.marussi@arm.com> wrote:

> Add basic support for SCMI Notifications, using Kernel notification chains
> mechanism.
> 
> Each SCMI Protocol has a dedicated events' queue and deferred worker which
> is in charge of delivering the notifications to the interested users
> invoking the users' registered callbacks.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>

I'm not keen on the huge asymmetry on the setup and tear down paths.
For instance the lack of a scmi_unregister_protocol events.

The idr usage is not standard.  They aren't the right structure
if you known the value you want.

Not a particularly thorough review otherwise. Out of time for today and
may not get back to this for a while.

Thanks,

Jonathan

> ---
>  drivers/firmware/arm_scmi/Makefile |   2 +-
>  drivers/firmware/arm_scmi/common.h |   4 +
>  drivers/firmware/arm_scmi/driver.c |   2 +
>  drivers/firmware/arm_scmi/notify.c | 904 +++++++++++++++++++++++++++++
>  drivers/firmware/arm_scmi/notify.h |  79 +++
>  5 files changed, 990 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/firmware/arm_scmi/notify.c
>  create mode 100644 drivers/firmware/arm_scmi/notify.h
> 
> diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
> index 5f298f00a82e..26587ea4661f 100644
> --- a/drivers/firmware/arm_scmi/Makefile
> +++ b/drivers/firmware/arm_scmi/Makefile
> @@ -1,6 +1,6 @@
>  # SPDX-License-Identifier: GPL-2.0-only
>  obj-y	= scmi-bus.o scmi-driver.o scmi-protocols.o
>  scmi-bus-y = bus.o
> -scmi-driver-y = driver.o
> +scmi-driver-y = driver.o notify.o
>  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o
>  obj-$(CONFIG_ARM_SCMI_POWER_DOMAIN) += scmi_pm_domain.o
> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
> index df35358ff324..2621c05e9149 100644
> --- a/drivers/firmware/arm_scmi/common.h
> +++ b/drivers/firmware/arm_scmi/common.h
> @@ -6,6 +6,8 @@
>   *
>   * Copyright (C) 2018 ARM Ltd.
>   */
> +#ifndef _SCMI_COMMON_H
> +#define _SCMI_COMMON_H
>  
>  #include <linux/bitfield.h>
>  #include <linux/completion.h>
> @@ -113,3 +115,5 @@ void scmi_setup_protocol_implemented(const struct scmi_handle *handle,
>  				     u8 *prot_imp);
>  
>  int scmi_base_protocol_init(struct scmi_handle *h);
> +
> +#endif /* _SCMI_COMMON_H */
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 28ed1f0cb417..a43fad29de11 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -350,12 +350,14 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>  
>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>  {
> +	ktime_t ts;
>  	struct scmi_xfer *xfer;
>  	struct device *dev = cinfo->dev;
>  	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>  	struct scmi_xfers_info *minfo = &info->rx_minfo;
>  	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>  
> +	ts = ktime_get_boottime();
>  	xfer = scmi_xfer_get(cinfo->handle, minfo);
>  	if (IS_ERR(xfer)) {
>  		dev_err(dev, "failed to get free message slot (%ld)\n",
> diff --git a/drivers/firmware/arm_scmi/notify.c b/drivers/firmware/arm_scmi/notify.c
> new file mode 100644
> index 000000000000..da342f43021e
> --- /dev/null
> +++ b/drivers/firmware/arm_scmi/notify.c
> @@ -0,0 +1,904 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * System Control and Management Interface (SCMI) Notification support
> + *
> + * Copyright (C) 2019 ARM Ltd.
> + *
> + * SCMI Protocol specification allows the platform to signal events to
> + * interested agents via notification messages: this in an implementation
> + * of the dispatch and delivery of such notifications to the interested users
> + * inside the Linux kernel.
> + *
> + * Each SCMI Protocol implementation, during its initialization, registers with
> + * this core notification framework its set of supported events via
> + * @scmi_register_protocol_events(), while Kernel users interested in some
> + * specific event can register their associated callbacks providing the usual
> + * notifier_block descriptor, since the notification system internally supports
> + * events delivery using customized notification chains.
> + *
> + * Given the number of possible events defined by SCMI and the extensibility
> + * of the SCMI Protocol itself, such underlying notification chains are created
> + * and destroyed dynamically on demand depending on the number of users
> + * effectively registered for an event, so that no structures or chains are
> + * allocated until at least one user has registered a notifier_block for such
> + * event. Similarly, events' generation itself is enabled at the platform level
> + * only after at least one user has registered, and it is shutdown after the
> + * last user for that event has gone.
> + *
> + * An event is identified univocally by the tuple (proto_id, evt_id, src_id)
> + * and is served by its own dedicated notification chain: given that such chain
> + * is dynamically created, the registration API simply let the users associate
> + * their callbacks with the above tuple.
> + *
> + * Here proto_id and evt_id are simply the protocol_id and message_id numbers as
> + * reported in the SCMI Protocol specification, while src_id represents an
> + * optional, protocol dependent, source identifier (like domain_id, perf_id
> + * or sensor_id and so forth), so that a user can register its callbacks for a
> + * particular event coming only from a defined source (like CPU vs GPU).
> + * When the source is not specified the user callback will be registered for
> + * all existing sources for that event (if any).
> + *
> + * Upon reception of a notification message from the platform the SCMI RX ISR
> + * passes the received message payload and some ancillary information (including
> + * an arrival timestamp in nanoseconds) to the core via @scmi_notify(), which,
> + * in turn, after having looked up the event in the registered events mapping,
> + * pushes the event-data itself on a protocol dedicated kfifo queue for deferred
> + * processing.
> + *
> + * Such dedicated protocols' queues are allocated once for all at initialization
> + * time, together with a dedicated work_item running the common delivery logic
> + * of @scmi_events_dispatcher(), so that each protocol has it own dedicated
> + * worker which, once kicked by the ISR, takes care to empty is own dedicated
> + * queue deliverying the queued items into the proper notification chain.
> + * Note that since the underlying cmwq workers run one distinct work_item per
> + * protocol and there are a number of max-active workers equal to the number of
> + * protocols, notifications processing can proceed concurrently only between
> + * events belonging to different protocols, while delivery of events within the
> + * same protocol is still strictly sequentially ordered by time of arrival; this
> + * separation effectively avoids the possibility that one particularly verbose
> + * protocol, flooding the queues with events, can cause other protocols' events
> + * to be lost or their processing starved.
> + *
> + * Event's information is converted to a custom per-event report struct before
> + * being fed as void *data param to the user callback embedded in the registered
> + * notifier_block so that it looks like as follow from the user perspective:
> + *
> + * int user_cb(struct notifier_block *nb, unsigned long event_id, void *report)
> + *
No point in having blank lines at the end of a comment block.
> + *
> + */
> +
> +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
> +
> +#include <linux/bug.h>
> +#include <linux/bitfield.h>
> +#include <linux/err.h>
> +#include <linux/idr.h>
> +#include <linux/kernel.h>
> +#include <linux/kfifo.h>
> +#include <linux/notifier.h>
> +#include <linux/refcount.h>
> +#include <linux/slab.h>
> +#include <linux/scmi_protocol.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +#include "notify.h"
> +
> +#define SCMI_NOTIF_MAX_ACTIVE_QUEUES	7
> +
> +/*
> + * Builds an unsigned 32bit key from the given input tuple to be used as a key
> + * in IDR mappings; note that if __src_p is passed in as NULL, the returned key
> + * is built using SCMI_ALL_SRC_IDS as src_id.
> + */
> +#define MAKE_EVT_KEY(__proto, __evt, __src_p)				\
> +	({								\
> +		u32 __evt_key, __src_id;				\
> +									\
> +		__src_id = ((__src_p)) ?				\
> +			    (*((u32 *)(__src_p)) & SCMI_ALL_SRC_IDS) :	\
> +			    SCMI_ALL_SRC_IDS;				\
> +		__evt_key = ((__proto) << 24) |				\
> +				((__evt) << 16) | __src_id;		\
> +		__evt_key;						\
> +	})
> +
> +#define PROTO_ID_MASK			GENMASK(31, 24)
> +#define EVT_ID_MASK			GENMASK(23, 16)
> +#define SRC_ID_MASK			GENMASK(15, 0)
> +#define KEY_XTRACT_PROTO_ID(key)	FIELD_GET(PROTO_ID_MASK, (key))
> +#define KEY_XTRACT_EVT_ID(key)		FIELD_GET(EVT_ID_MASK, (key))
> +#define KEY_XTRACT_SRC_ID(key)		FIELD_GET(SRC_ID_MASK, (key))
> +
> +/**
> + * events_queue  - Describes a queue and its associated worker

Not valid kernel-doc.    I'll assume all other instances will get fixed
and not bother talking about them.

> + *
> + * Each protocol has its own dedicated events_queue descriptor.
> + *
> + * @sz: Size in bytes of the related kfifo
> + * @kfifo: A dedicated Kernel kfifo
> + * @notify_work: A custom work item bound to this queue
> + * @wq: A reference to the related workqueue
> + */
> +struct events_queue {
> +	size_t			sz;
> +	struct kfifo		kfifo;
> +	struct work_struct	notify_work;
> +	struct workqueue_struct	*wq;
> +};
> +
> +/**
> + * scmi_registered_protocol_events_desc  - Protocol Specific information
> + *
> + * All protocols that registers at least an event have their protocol-specific
> + * information stored here, together with a pointer to the allocated
> + * events_queue.
> + *
> + * @id: Protocol ID
> + * @ops: Protocol specific and event-related operations
> + * @equeue: A reference to the associated per-protocol events_queue
> + */
> +struct scmi_registered_protocol_events_desc {
> +	u8					id;
> +	const struct scmi_protocol_event_ops	*ops;
> +	struct events_queue			*equeue;
> +};
> +
> +/**
> + * scmi_registered_event  - Event Specific Information
> + *
> + * All registered events are represented by one of these structures.
> + *
> + * @proto: A reference to the associated protocol descriptor
> + * @evt: A reference to the associated event descriptor (as provided at
> + *       registration time)
> + * @scratch_isr: A pre-allocated buffer to be used as a scratch area by ISR
> + * @scratch_isr: A pre-allocated buffer to be used as a scratch area by the
> + *		 deferred worker
scratch_bh.

> + * @report: A pre-allocated buffer used by the deferred workr to fill a

worker

> + *	    customized event report
> + */
> +struct scmi_registered_event {
> +	struct scmi_registered_protocol_events_desc	*proto;
> +	const struct scmi_event				*evt;
> +	void						*scratch_isr;
> +	void						*scratch_bh;
> +	void						*report;
> +};
> +
> +/**
> + * scmi_event_handler  - Event handler information
> + *
> + * This structure collects all the information needed to process a received
> + * event, calling all the registered notifier callbacks.
> + * Note this an event handler is associated to a tuple of values like:
> + *	(proto_id, evt_id, src_id)
> + * through the means of a key and an IDR mapping.
> + *
> + * @evt_key: The unsigned 32bit key associated to this descriptor in the
> + *	     related IDR mapping
> + * @r_evt: A reference to the underlying registered event
> + * @chain: The notification chain dedicated to this specific event tuple
> + */
> +struct scmi_event_handler {
> +	u32				evt_key;
> +	bool				enabled;
> +	refcount_t			users;
> +	struct scmi_registered_event	*r_evt;
> +	struct blocking_notifier_head	chain;
> +};
> +
> +/**
> + * scmi_event_header  - A utility header

That's not a good description.

> + *
> + * This header is prepended to each received event message payload before
> + * being queued on the related events_queue, and describes the attached event
> + * message.
> + *
> + * @timestamp: The timestamp, innanoseconds (boottime), which was associated
> + *	       to this event as soon as it entered the SCMI RX ISR
> + * @proto_id: Protocol ID
> + * @evt_id: Event ID (corresponds to the Event MsgID for this Protocol)
> + * @payld_sz: Effective size of the attached message payload which follows
> + * @payld: A reference to the included Event payload
> + */
> +struct scmi_event_header {
> +	u64	timestamp;
> +	u8	proto_id;
> +	u8	evt_id;
> +	size_t	payld_sz;
> +	u8	payld[];
> +} __packed;
> +
> +/*
> + * A few IDR maps to track:
> + *
> + * - @scmi_registered_events: All event's descriptors registered by the
> + *			      protocols, together with their ancillary data
> + * - @scmi_registered_events_handlers: All event's handlers descriptors, created
> + *				       to collect all the users' notifier_block
> + *				       callbacks and related notification chains
> + * - @scmi_registered_protocols: All protocol-level specific information related
> + *				 to events' handling
> + */
> +static struct idr scmi_registered_events;
> +static struct idr scmi_registered_events_handlers;
> +static DEFINE_MUTEX(scmi_registered_events_handler_mtx);
> +static struct idr scmi_registered_protocols;
> +
> +/* Common Kernel cmwq workqueue used by notifications core */
> +static struct workqueue_struct *scmi_notify_wq;
> +
> +static bool scmi_notifications_initialized;
> +
> +static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key);
> +static void scmi_put_event_handler(struct scmi_event_handler *hndl);
> +
> +/**
> + * scmi_discard_bad_evt_payloadi() - Discard data from a kfifo

function name is wrong.

> + *
> + * @kq: The kfifo to act on
> + * @count: Number of bytes to flush
> + */
> +static inline void scmi_discard_bad_evt_payload(struct kfifo *kq,
> +						const unsigned int count)
> +{
> +	int i = 0;
> +
> +	pr_warn("SCMI Notification WQ :: skipping bad EVT Payload - %d bytes\n",
> +		count);
> +	/* Discard stale pending queued payload. */
> +	for (i = 0; i < count; i++)
> +		kfifo_skip(kq);
> +}
> +
> +/**
> + * scmi_lookup_and_call_event_chain  - Lookup the proper chain and call it
> + *
> + * @evt_key: The event key to use to lookup the related notification chain
> + * @report: The customized event-specific report to pass down to the callbacks
> + *	    as their *data parameter.
> + */
> +static inline void scmi_lookup_and_call_event_chain(u32 evt_key, void *report)
> +{
> +	int ret;
> +	struct scmi_event_handler *hndl;
> +
> +	hndl = scmi_get_event_handler(evt_key);
> +	if (IS_ERR_OR_NULL(hndl))
> +		return;
> +
> +	ret = blocking_notifier_call_chain(&hndl->chain,
> +					   KEY_XTRACT_EVT_ID(evt_key),
> +					   report);
> +	/* Notifiers are NOT supposed to cut the chain */
> +	WARN_ON_ONCE(ret & NOTIFY_STOP_MASK);
> +
> +	scmi_put_event_handler(hndl);
> +}
> +
> +/**
> + * scmi_events_dispatcher  - Common worker logic for all work items.
> + *
> + * In turn:
> + *  1. dequeue one pending RX notification (queued in SCMI RX ISR context)
> + *  2. generate a custom event report from the received event message
> + *  3. lookup for any registered ALL_SRC_IDs handler
> + *     - > call the related notification chain passing in the report
> + *  4. lookup for any registered specific SRC_ID handler
> + *     - > call the related notification chain passing in the report
> + *
> + * Note that:
> + * - a dedicated per-protocol kfifo queue is used: in this way an anomalous
> + *   flood of events cannot saturate other protocols' queues.
> + *
> + * - each per-protocol queue is associated to a distinct work_item, which
> + *   means, in turn, that:
> + *   + all protocol can process their dedicated queues concurrently
> + *     (since scmi_notify_wq:max_active > 1)
> + *   + anyway at most one worker instance is allowed to run on the same queue
> + *     concurrently: this ensures that we can have only one concurrent
> + *     reader/writer on the associated kfifo (needed for lock-less access)
> + *
> + * @work: The work item to use, which is associated to the proper events_queue
> + */
> +static void scmi_events_dispatcher(struct work_struct *work)
> +{
> +	struct events_queue *equeue;
> +	struct scmi_event_header eh;
> +
> +	equeue = container_of(work, struct events_queue, notify_work);
> +	while (kfifo_out(&equeue->kfifo, &eh, sizeof(eh))) {
> +		u32 src_id, evt_key;
> +		unsigned int outs;
> +		struct scmi_registered_event *r_evt;
> +		void *report = NULL;
> +
> +		evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, NULL);
> +		r_evt = idr_find(&scmi_registered_events, evt_key);
> +		if (!r_evt) {
> +			scmi_discard_bad_evt_payload(&equeue->kfifo,
> +						     eh.payld_sz);
> +			continue;
> +		}
> +
> +		outs = kfifo_out(&equeue->kfifo, r_evt->scratch_bh,
> +				 eh.payld_sz);
> +		if (outs != eh.payld_sz) {
> +			pr_warn("SCMI Notification WQ :: SKIP corrupted EVT Payload.\n");
> +			continue;
> +		}
> +
> +		/* Reset and fill custom report */
> +		memset(r_evt->report, 0x00, r_evt->evt->max_report_sz);
> +		report = r_evt->proto->ops->fill_custom_report(eh.evt_id,
> +							eh.timestamp,
> +							r_evt->scratch_bh,
> +							eh.payld_sz,
> +							r_evt->report,
> +							&src_id);
> +		if (!report)
> +			continue;
> +
> +		/* At first search for a generic ALL src_ids handler... */
> +		scmi_lookup_and_call_event_chain(evt_key, report);
> +		if (src_id != SCMI_ALL_SRC_IDS) {
> +			u32 *__sub_p = &src_id;
> +
> +			evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, __sub_p);
> +			scmi_lookup_and_call_event_chain(evt_key, report);
> +		}
> +	}
> +}
> +
> +/**
> + * scmi_notify  - Queues a notification for further deferred processing
> + *
> + * This is called in interrupt context to queue a received event for
> + * deferred processing.
> + *
> + * @proto_id: Protocol ID
> + * @evt_id: Event ID (msgID)
> + * @buf: Event Message Payload (without the header)
> + * @len: Event Message Payload size
> + * @ts: RX Timestamp in nanoseconds (boottime)
> + *
> + * Return: 0 on Success
> + */
> +int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts)
> +{
> +	struct scmi_registered_event *r_evt;
> +	struct scmi_event_header *eh;
> +
> +	if (unlikely(!scmi_notifications_initialized))
> +		return 0;
> +
> +	r_evt = idr_find(&scmi_registered_events,
> +			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
> +	if (unlikely(!r_evt || !r_evt->proto->equeue))
> +		return -EINVAL;
> +
> +	if (unlikely(len > r_evt->evt->max_payld_sz)) {
> +		pr_err("SCMI Notification discard badly sized message\n");
> +		return -EINVAL;
> +	}
> +	if (unlikely(kfifo_avail(&r_evt->proto->equeue->kfifo) <
> +		     sizeof(*eh) + len)) {
> +		pr_warn("SCMI Notification queue full: dropping proto_id:%d  evt_id:%d  ts:%lld\n",
> +			proto_id, evt_id, ts);
> +		return -ENOMEM;
> +	}
> +
> +	/*
> +	 * Note that we can use the per protocol kfifo in a lock-less manner
> +	 * since we have only one concurrent reader and writer but, in order
> +	 * to avoid any trouble on the reader side, here we perform one single
> +	 * write, so we have to collate event header and payload in a scratch
> +	 * area at first.
> +	 */
> +	eh = r_evt->scratch_isr;
> +	eh->timestamp = ts;
> +	eh->proto_id = proto_id;
> +	eh->evt_id = evt_id;
> +	eh->payld_sz = len;
> +	memcpy(eh->payld, buf, eh->payld_sz);
> +	kfifo_in(&r_evt->proto->equeue->kfifo, eh, sizeof(*eh) + eh->payld_sz);
> +	queue_work(r_evt->proto->equeue->wq,
> +		   &r_evt->proto->equeue->notify_work);
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_allocate_events_queue  - Allocate an events_queue descriptor
> + *
> + * @sz: Size of the kfifo to initialize
> + *
> + * Return: A valid pointer to the allocated events_queue on Success
> + */
> +static struct events_queue *scmi_allocate_events_queue(size_t sz)
> +{
> +	int ret;
> +	struct events_queue *equeue;
> +
> +	equeue = kzalloc(sizeof(*equeue), GFP_KERNEL);
> +	if (!equeue)
> +		return ERR_PTR(-ENOMEM);
> +	ret = kfifo_alloc(&equeue->kfifo, sz, GFP_KERNEL);
> +	if (ret) {
> +		kfree(equeue);
> +		return ERR_PTR(ret);
> +	}
> +	equeue->sz = sz;
> +	INIT_WORK(&equeue->notify_work, scmi_events_dispatcher);
> +	equeue->wq = scmi_notify_wq;
> +
> +	return equeue;
> +}
> +
> +/**
> + * scmi_deallocate_events_queue  - Deallocate am events_queue descriptor
> + *
> + * @equeue: The events_queue to free
> + */
> +static void scmi_deallocate_events_queue(struct events_queue *equeue)
> +{
> +	kfifo_free(&equeue->kfifo);
> +	kfree(equeue);
> +}
> +
> +/**
> + * scmi_allocate_registered_protocol_desc  - Allocate a registered protocol
> + * events' descriptor
> + *
> + * Used to keep protocol specific information related to events handling for any
> + * protocl which has registered at least one event.

protocol

> + *
> + * @proto_id: Protocol ID
> + * @queue_sz: Size of the associated queue to allocate
> + * @ops: Pointer to a struct holding references to protocol specific helpers
> + *	 needed during events handling
> + */
> +static struct scmi_registered_protocol_events_desc *
> +scmi_allocate_registered_protocol_desc(u8 proto_id, size_t queue_sz,
> +				const struct scmi_protocol_event_ops *ops)
> +{
> +	int ret;
> +	struct scmi_registered_protocol_events_desc *pdesc;
> +
> +	pdesc = idr_find(&scmi_registered_protocols, proto_id);

Perhaps add a comment here on why this might get called multiple times
but is still safe from point of view of release path.

> +	if (pdesc)
> +		return pdesc;


> +
> +	pdesc = kzalloc(sizeof(*pdesc), GFP_KERNEL);
> +	if (!pdesc)
> +		return ERR_PTR(-ENOMEM);
> +	pdesc->id = proto_id;
> +	pdesc->ops = ops;
> +
> +	pdesc->equeue = scmi_allocate_events_queue(queue_sz);
> +	if (IS_ERR(pdesc->equeue)) {
> +		kfree(pdesc);
> +		return ERR_CAST(pdesc->equeue);
> +	}
> +
> +	ret = idr_alloc(&scmi_registered_protocols, pdesc,
> +			pdesc->id, pdesc->id + 1, GFP_KERNEL);
> +	if (ret < 0) {
> +		pr_err("SCMI Failed to allocate PDESC IDR - key:%d - err:%d\n",
> +		       pdesc->id, ret);
> +		scmi_deallocate_events_queue(pdesc->equeue);
> +		kfree(pdesc);
> +		return ERR_PTR(ret);
> +	}
> +
> +	return pdesc;
> +}
> +
> +/**
> + * scmi_register_protocol_events  - Register Protocol Events with the core
> + *
> + * Used by SCMI Protocols initialization code to register with the notification
> + * core the list of supported events and their description: this takes care to
> + * pre-allocate all needed scratch buffers and link the proper event queue to
> + * this event.
> + *
> + * @proto_id: Protocol ID
> + * @queue_sz: Size in bytes of the associated queue to be allocated
> + * @ops: Protocol specific event-related operations
> + * @evt: Event descriptor array
> + * @num_events: Number of events in @evt array
> + *
> + * Return: 0 on Success
> + */
> +int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
> +				  const struct scmi_protocol_event_ops *ops,
> +				  const struct scmi_event *evt, int num_events)
> +{
> +	int i;
> +	struct scmi_registered_protocol_events_desc *pdesc;
> +
> +	if (!scmi_notifications_initialized)
> +		return -EAGAIN;
> +
> +	if (!ops || !evt)
> +		return -EINVAL;
> +
> +	pdesc = scmi_allocate_registered_protocol_desc(proto_id, queue_sz, ops);
> +	if (IS_ERR(pdesc))
> +		return -ENOMEM;
> +
> +	for (i = 0; i < num_events; i++, evt++) {
> +		int ret;
> +		u32 key;
> +		struct scmi_registered_event *r_evt;
> +
> +		r_evt = kzalloc(sizeof(*r_evt), GFP_KERNEL);
> +		if (!r_evt)
> +			continue;
> +		r_evt->proto = pdesc;
> +		r_evt->evt = evt;
> +
> +		r_evt->scratch_isr = kzalloc(sizeof(struct scmi_event_header) +
> +					     evt->max_payld_sz, GFP_KERNEL);
> +		if (!r_evt->scratch_isr)
> +			goto continue_isr_fail;
> +
> +		r_evt->scratch_bh = kzalloc(evt->max_payld_sz, GFP_KERNEL);
> +		if (!r_evt->scratch_bh)
> +			goto continue_bh_fail;
> +
> +		r_evt->report = kzalloc(evt->max_report_sz, GFP_KERNEL);
> +		if (!r_evt->report)
> +			goto continue_report_fail;
> +
> +		key = MAKE_EVT_KEY(r_evt->proto->id, evt->evt_id, NULL);
> +		ret = idr_alloc(&scmi_registered_events, r_evt,
> +				key, key + 1, GFP_KERNEL);
> +		if (ret < 0) {
> +			pr_err("SCMI Failed to allocate EVENT IDR - key:%X - err:%d\n",
> +			       key, ret);
> +			goto continue_idr_fail;
> +		}
> +
> +		pr_info("SCMI Notification registered event - key:%X\n", key);
> +		continue;
> +
> +	/* yes, this is not nice ... */
> +continue_idr_fail:
> +		kfree(r_evt->report);
> +continue_report_fail:
> +		kfree(r_evt->scratch_bh);
> +continue_bh_fail:
> +		kfree(r_evt->scratch_isr);
> +continue_isr_fail:
> +		kfree(r_evt);
> +	}
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_register_event_handler  - Allocate an Event handler
> + *
> + * Allocate an event handler and related notification chain associated with
> + * the event identified by the provided event key. Fails if the associated
> + * event is unknown to the core (i.e. it had not been successfully registered
> + * upfront by some protocol)
> + *
> + * @evt_key: 32bit key uniquely bind to the event identified by the tuple
> + *	     (proto_id, evt_id, src_id)
> + *
> + * Return: the freshly allocated structure on Success
> + */
> +static struct scmi_event_handler *scmi_register_event_handler(u32 evt_key)
> +{
> +	int id;
> +	u8 proto_id, evt_id;
> +	struct scmi_registered_event *r_evt;
> +	struct scmi_event_handler *hndl;
> +
> +	proto_id = KEY_XTRACT_PROTO_ID(evt_key);
> +	evt_id = KEY_XTRACT_EVT_ID(evt_key);
> +	r_evt = idr_find(&scmi_registered_events,
> +			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
> +	if (!r_evt)
> +		return ERR_PTR(-EINVAL);
> +
> +	hndl = kzalloc(sizeof(*hndl), GFP_KERNEL);
> +	if (!hndl)
> +		return ERR_PTR(-ENOMEM);
> +	hndl->r_evt = r_evt;
> +	hndl->evt_key = evt_key;
> +	BLOCKING_INIT_NOTIFIER_HEAD(&hndl->chain);
> +	refcount_set(&hndl->users, 1);
> +
> +	/* Register freshly allocated event handler */
> +	id = idr_alloc(&scmi_registered_events_handlers, hndl,
> +		       evt_key, evt_key + 1, GFP_KERNEL);

Umm. That's not using an idr for what it is meant for. You aren't allocating an id, you
are using it as a poor wrapper around an xarray.

> +	if (id < 0) {
> +		pr_err("SCMI Failed to allocate HNDL IDR - key:%X  err:%d\n",
> +		       evt_key, id);
> +		kfree(hndl);
> +		return ERR_PTR(id);
> +	}
> +
> +	return hndl;
> +}
> +
> +/**
> + * scmi_unregister_event_handler  - Free the provided Event handler
> + *
> + * @hndl: The event handler structure to free
> + */
> +static void scmi_unregister_event_handler(struct scmi_event_handler *hndl)
> +{
> +	idr_remove(&scmi_registered_events_handlers, hndl->evt_key);
> +	kfree(hndl);
> +}
> +
> +/**
> + * __scmi_event_handler_get_ops  - Get or create an event handler
> + *
> + * After having got exclusive access to the registered events map, searches for
> + * the desired handler matching the key: when found adjust refcount, when not
> + * create and register a new handler is asked to do so.
> + * Events generation is NOT enabled within this routine since at creation time
> + * we usually want to have all setup and registered before events starts
> + * flowing.
> + *
> + * @evt_key: The event key to use
> + * @create: A boolean flag to specify if a handler must be created
> + *	    when not found
> + */
> +static inline struct scmi_event_handler *
> +__scmi_event_handler_get_ops(u32 evt_key, bool create)
> +{
> +	struct scmi_event_handler *hndl = NULL;
> +
> +	mutex_lock(&scmi_registered_events_handler_mtx);
> +	hndl = idr_find(&scmi_registered_events_handlers, evt_key);
> +	if (hndl)
> +		refcount_inc(&hndl->users);
> +	else if (create)
> +		hndl = scmi_register_event_handler(evt_key);
> +	mutex_unlock(&scmi_registered_events_handler_mtx);
> +
> +	return hndl;
> +}
> +
> +static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key)
> +{
> +	return __scmi_event_handler_get_ops(evt_key, false);
> +}
> +
> +static struct scmi_event_handler *scmi_get_or_create_event_handler(u32 evt_key)
> +{
> +	return __scmi_event_handler_get_ops(evt_key, true);
> +}
> +
> +/**
> + * __scmi_enable_events_ops  - Enable/disable events generation
> + *
> + * @hndl: The handler specifying the events enable/disable
> + * @action: The action to perform: true->Enable false->Disable
> + *
> + * Return: True when the required @action has been successfully executed
> + */
> +static inline bool __scmi_enable_events_ops(struct scmi_event_handler *hndl,
> +					    bool action)
> +{
> +	bool ret = true;
> +
> +	if (hndl->enabled != action) {
> +		u8 evt_id;
> +		u32 src_id;
> +
> +		evt_id = KEY_XTRACT_EVT_ID(hndl->evt_key);
> +		src_id = KEY_XTRACT_SRC_ID(hndl->evt_key);
> +		ret = hndl->r_evt->proto->ops->set_notify_enabled(evt_id,
> +			      src_id != SCMI_ALL_SRC_IDS ? &src_id : NULL,
> +			      action);
> +		if (ret)
> +			hndl->enabled = action;
> +	}
> +
> +	return ret;
> +}
> +
> +static bool scmi_enable_events(struct scmi_event_handler *hndl)
> +{
> +	return __scmi_enable_events_ops(hndl, true);
> +}
> +
> +static bool scmi_disable_events(struct scmi_event_handler *hndl)
> +{
> +	return __scmi_enable_events_ops(hndl, false);
> +}
> +
> +/**
> + * scmi_put_event_handler  - Put an event handler
> + *
> + * After having got exclusive access to the registered events map, update
> + * the refcount and if @hndl is no more in use by anyone:
> + *  - disable the events generation
> + *  - unregister and free the handler itself
> + *
> + * @hndl: The handler to act upon
> + */
> +static void scmi_put_event_handler(struct scmi_event_handler *hndl)
> +{
> +	mutex_lock(&scmi_registered_events_handler_mtx);
> +	if (refcount_dec_and_test(&hndl->users)) {
> +		scmi_disable_events(hndl);
> +		scmi_unregister_event_handler(hndl);
> +	}
> +	mutex_unlock(&scmi_registered_events_handler_mtx);
> +}
> +
> +/**
> + * scmi_register_event_notifier  - Register a notifier_block for an event
> + *
> + * Generic helper to register a notifier_block attached to a protocol event.
> + *
> + * A notifier_block @nb will be registered for each distinct event identified
> + * by the tuple (proto_id, evt_id, src_id) on a dedicated notification chain
> + * so that:
> + *
> + *	(proto_X, evt_Y, src_Z) --> chain_X_Y_Z
> + *
> + * @src_id meaning is protocol specific and identifies the origin of the event
> + * (like domain_id, sensor_id and os forth); @src_id can be NULL to signify that

so

> + * the caller is interested in receiving notifications from ALL the available
> + * sources for that protocol OR simply that the protocol does not support
> + * distinct sources: in these cases @nb will be attached to a generic
> + * notification chain defined for ALL src_id of that proto_id/evt_id pair like:
> + *
> + *	(proto_X, evt_Y, NULL) --> chain_X_Y_ALL
> + *
> + * Any received event will be then dispatched to both such chains if at least
> + * one user had registered an @nb on them.

both should be all I think.

> + *
> + * Note also that notification chains are created dynamically: a new chain is
> + * created each time a new distinct tuple is detected and then the provided @nb
> + * is bound to such chain; at creation time the related SCMI notifications are
> + * also enabled: this way, only when at least one user has registered its
> + * interest for a specific event tuple, the underlying notification chain is
> + * created and the related notifications are enabled in the platform.

That's one of the most complex bits of punctuation I've seenin a comment for
a long time.   Please break that sentence up for readability.

> + *
> + * @proto_id: Protocol ID
> + * @evt_id: Event ID
> + * @src_id: Source ID
> + * @nb: A standard notifier block to register for the specified event
> + *
> + * Return: Return 0 on Success
> + */
> +int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
> +				 struct notifier_block *nb)
> +{
> +	u32 evt_key;
> +	struct scmi_event_handler *hndl;
> +
> +	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
> +	hndl = scmi_get_or_create_event_handler(evt_key);
> +	if (IS_ERR_OR_NULL(hndl))
> +		return PTR_ERR(hndl);
> +
> +	blocking_notifier_chain_register(&hndl->chain, nb);
> +
> +	if (!scmi_enable_events(hndl)) {
> +		pr_err("SCMI Failed to ENABLE events for key:%X !\n", evt_key);
> +		blocking_notifier_chain_unregister(&hndl->chain, nb);
> +		scmi_put_event_handler(hndl);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_unregister_event_notifier  - Unregister a notifier_block for an event
> + *
> + * Takes care to unregister the provided @nb from the notification chain
> + * associated to the specified event and, if there are no more users for the
> + * event handler, frees also the associated event handler structures.
> + *
> + * @proto_id: Protocol ID
> + * @evt_id: Event ID
> + * @src_id: Source ID
> + * @nb: The notifier_block to unregister
> + *
> + * Return: 0 on Success
> + */
> +int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
> +				   struct notifier_block *nb)
> +{
> +	u32 evt_key;
> +	struct scmi_event_handler *hndl;
> +
> +	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
> +	hndl = scmi_get_event_handler(evt_key);
> +	if (IS_ERR_OR_NULL(hndl))
> +		return -EINVAL;
> +
> +	blocking_notifier_chain_unregister(&hndl->chain, nb);
> +
> +	scmi_put_event_handler(hndl);
> +	/*
> +	 * If this was the last user callback for this handler, this last put
> +	 * will force the handler to be freed.
> +	 * Note that if a call_chain walk is ongoing it will be the call_chain
> +	 * put request which will finally free the handler; note also that any
> +	 * operation on the inner notifier_block chain is protected on its own.
> +	 */
> +	scmi_put_event_handler(hndl);
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_notification_init  - Initializes Notification Core Support
> + *
> + * Return: 0 on Success
> + */
> +int __init scmi_notification_init(void)
> +{
> +	scmi_notify_wq = alloc_workqueue("scmi_notify",
> +					 WQ_UNBOUND | WQ_FREEZABLE,
> +					 SCMI_NOTIF_MAX_ACTIVE_QUEUES);
> +	if (!scmi_notify_wq)
> +		return -ENOMEM;
> +	idr_init(&scmi_registered_events);
> +	idr_init(&scmi_registered_events_handlers);
> +	idr_init(&scmi_registered_protocols);
> +
> +	scmi_notifications_initialized = true;
> +
> +	pr_info("SCMI Notifications enabled.\n");
> +
> +	return 0;
> +}
> +
> +/**
> + * scmi_notification_exit  - Shutdown and clean Notification core
> + */
> +void __exit scmi_notification_exit(void)
> +{
> +	int k;
> +	struct scmi_event_handler *hndl;
> +	struct scmi_registered_protocol_events_desc *pdesc;
> +	struct scmi_registered_event *r_evt;
> +
> +	scmi_notifications_initialized = false;
> +
> +	/* Disable notifications' generation */
> +	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
> +		scmi_disable_events(hndl);
> +
> +	/* Let pending work complete */
> +	destroy_workqueue(scmi_notify_wq);
> +
> +	/* Remove ALL events handlers */
> +	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
> +		scmi_put_event_handler(hndl);
> +	idr_destroy(&scmi_registered_events_handlers);
> +
> +	/* Finally remove all registered events */

Finally seems a bit premature here ;)

> +	idr_for_each_entry(&scmi_registered_events, r_evt, k) {
> +		idr_remove(&scmi_registered_events, k);
> +		kfree(r_evt->report);
> +		kfree(r_evt->scratch_isr);
> +		kfree(r_evt->scratch_bh);
> +		kfree(r_evt);
> +	}
> +	idr_destroy(&scmi_registered_events);
> +
> +	/* free any remaining protocol data */
> +	idr_for_each_entry(&scmi_registered_protocols, pdesc, k) {
> +		idr_remove(&scmi_registered_protocols, k);
> +		scmi_deallocate_events_queue(pdesc->equeue);
> +		kfree(pdesc);
> +	}
> +	idr_destroy(&scmi_registered_protocols);
> +}
> diff --git a/drivers/firmware/arm_scmi/notify.h b/drivers/firmware/arm_scmi/notify.h
> new file mode 100644
> index 000000000000..6fa07b244b14
> --- /dev/null
> +++ b/drivers/firmware/arm_scmi/notify.h
> @@ -0,0 +1,79 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * System Control and Management Interface (SCMI) Message Protocol
> + * notification header file containing some definitions, structures
> + * and function prototypes related to SCMI Notification handling.
> + *
> + * Copyright (C) 2019 ARM Ltd.
> + */
> +#ifndef _SCMI_NOTIFY_H
> +#define _SCMI_NOTIFY_H
> +
> +#include <linux/bug.h>
> +#include <linux/notifier.h>
> +#include <linux/types.h>
> +
> +#define	SCMI_ALL_SRC_IDS		0x0000ffffL
> +
> +#define MAP_EVT_TO_ENABLE_CMD(id, map)			\
> +({							\
> +	int ret = -1;					\
> +							\
> +	if (likely((id) < ARRAY_SIZE((map))))		\
> +		ret = (map)[(id)];			\
> +	else						\
> +		WARN(1, "UN-KNOWN evt_id:%d\n", (id));	\
> +	ret;						\
> +})
> +
> +/**
> + * scmi_event  - Describes an event to be supported

I don't think this is valid kernel-doc.  Make sure to point the scripts at the
file and check and fix any warnings.

> + *
> + * Each SCMI protocol, during its initialization phase, can describe the events
> + * it wishes to support in a few struct scmi_event and pass them to the core
> + * using scmi_register_protocol_events().
> + *
> + * @evt_id: Event ID
> + * @max_payld_sz: Max possible size for the payload of a notif msg of this kind
> + * @max_report_sz: Max possible size for the report of a notif msg of this kind
> + */
> +struct scmi_event {
> +	u8	evt_id;
> +	size_t	max_payld_sz;
> +	size_t	max_report_sz;
> +
> +};
> +
> +/**
> + * scmi_protocol_event_ops  - Helpers called by notification core.
> + *
> + * These are called only in process context.
> + *
> + * @set_notify_enabled: Enable/disable the required evt_id/src_id notifications
> + *			using the proper custom protocol commands.
> + *			Return true if at least one the required src_id
> + *			has been successfully enabled/disabled
> + * @fill_custom_report: fills a custom event report from the provided

Nitpick.  Inconsistent capitalization after the :

> + *			event message payld identifying the event
> + *			specific src_id.
> + *			Return NULL on failure otherwise @report now fully
> + *			populated
> + */
> +struct scmi_protocol_event_ops {
> +	bool (*set_notify_enabled)(u8 evt_id, const u32 *src_id, bool enabled);
> +	void *(*fill_custom_report)(u8 evt_id, u64 timestamp, const void *payld,
> +				    size_t payld_sz, void *report, u32 *src_id);
> +};
> +
> +int scmi_notification_init(void);
> +void scmi_notification_exit(void);
> +int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
> +				  const struct scmi_protocol_event_ops *ops,
> +				  const struct scmi_event *evt, int num_events);
> +int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
> +				 struct notifier_block *nb);
> +int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
> +				   struct notifier_block *nb);
> +int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts);
> +
> +#endif /* _SCMI_NOTIFY_H */

Re: [RFC PATCH 04/11] firmware: arm_scmi: Add core notifications support

[Cristian Marussi]

Hi Jonathan

Thanks for the review first of all (no matter how partial) !

On 1/27/20 6:11 PM, Jonathan Cameron wrote:
> On Mon, 20 Jan 2020 12:23:26 +0000
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
>> Add basic support for SCMI Notifications, using Kernel notification chains
>> mechanism.
>>
>> Each SCMI Protocol has a dedicated events' queue and deferred worker which
>> is in charge of delivering the notifications to the interested users
>> invoking the users' registered callbacks.
>>
>> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> 
> I'm not keen on the huge asymmetry on the setup and tear down paths.
> For instance the lack of a scmi_unregister_protocol events.
>
Yes it's something I thought about, but it seems that SCMI protocols are never
explicitly shutdown as it is now, so I bound init of notification core
to the bus/protocols going up and notification exit to the bus shutdown itself.
I'll have another thought about it, maybe reviewing a bit protocols shutdown
procedure too.

  
> The idr usage is not standard.  They aren't the right structure
> if you known the value you want.

I agree, it was the most ready-to-use in kernel hash-likish structure so I went for that,
but it is a bit of an abuse used this way....I want to switch to real hashtables in V2 to
match dynamically events described by (proto_id, evt_id, src_id) to their data structs.
  
> 
> Not a particularly thorough review otherwise. Out of time for today and
> may not get back to this for a while.
> 
> Thanks,
> 
> Jonathan
> 

Thanks a lot.

Regards

Cristian

>> ---
>>   drivers/firmware/arm_scmi/Makefile |   2 +-
>>   drivers/firmware/arm_scmi/common.h |   4 +
>>   drivers/firmware/arm_scmi/driver.c |   2 +
>>   drivers/firmware/arm_scmi/notify.c | 904 +++++++++++++++++++++++++++++
>>   drivers/firmware/arm_scmi/notify.h |  79 +++
>>   5 files changed, 990 insertions(+), 1 deletion(-)
>>   create mode 100644 drivers/firmware/arm_scmi/notify.c
>>   create mode 100644 drivers/firmware/arm_scmi/notify.h
>>
>> diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
>> index 5f298f00a82e..26587ea4661f 100644
>> --- a/drivers/firmware/arm_scmi/Makefile
>> +++ b/drivers/firmware/arm_scmi/Makefile
>> @@ -1,6 +1,6 @@
>>   # SPDX-License-Identifier: GPL-2.0-only
>>   obj-y	= scmi-bus.o scmi-driver.o scmi-protocols.o
>>   scmi-bus-y = bus.o
>> -scmi-driver-y = driver.o
>> +scmi-driver-y = driver.o notify.o
>>   scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o
>>   obj-$(CONFIG_ARM_SCMI_POWER_DOMAIN) += scmi_pm_domain.o
>> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
>> index df35358ff324..2621c05e9149 100644
>> --- a/drivers/firmware/arm_scmi/common.h
>> +++ b/drivers/firmware/arm_scmi/common.h
>> @@ -6,6 +6,8 @@
>>    *
>>    * Copyright (C) 2018 ARM Ltd.
>>    */
>> +#ifndef _SCMI_COMMON_H
>> +#define _SCMI_COMMON_H
>>   
>>   #include <linux/bitfield.h>
>>   #include <linux/completion.h>
>> @@ -113,3 +115,5 @@ void scmi_setup_protocol_implemented(const struct scmi_handle *handle,
>>   				     u8 *prot_imp);
>>   
>>   int scmi_base_protocol_init(struct scmi_handle *h);
>> +
>> +#endif /* _SCMI_COMMON_H */
>> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
>> index 28ed1f0cb417..a43fad29de11 100644
>> --- a/drivers/firmware/arm_scmi/driver.c
>> +++ b/drivers/firmware/arm_scmi/driver.c
>> @@ -350,12 +350,14 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>>   
>>   static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>>   {
>> +	ktime_t ts;
>>   	struct scmi_xfer *xfer;
>>   	struct device *dev = cinfo->dev;
>>   	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>>   	struct scmi_xfers_info *minfo = &info->rx_minfo;
>>   	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>>   
>> +	ts = ktime_get_boottime();
>>   	xfer = scmi_xfer_get(cinfo->handle, minfo);
>>   	if (IS_ERR(xfer)) {
>>   		dev_err(dev, "failed to get free message slot (%ld)\n",
>> diff --git a/drivers/firmware/arm_scmi/notify.c b/drivers/firmware/arm_scmi/notify.c
>> new file mode 100644
>> index 000000000000..da342f43021e
>> --- /dev/null
>> +++ b/drivers/firmware/arm_scmi/notify.c
>> @@ -0,0 +1,904 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * System Control and Management Interface (SCMI) Notification support
>> + *
>> + * Copyright (C) 2019 ARM Ltd.
>> + *
>> + * SCMI Protocol specification allows the platform to signal events to
>> + * interested agents via notification messages: this in an implementation
>> + * of the dispatch and delivery of such notifications to the interested users
>> + * inside the Linux kernel.
>> + *
>> + * Each SCMI Protocol implementation, during its initialization, registers with
>> + * this core notification framework its set of supported events via
>> + * @scmi_register_protocol_events(), while Kernel users interested in some
>> + * specific event can register their associated callbacks providing the usual
>> + * notifier_block descriptor, since the notification system internally supports
>> + * events delivery using customized notification chains.
>> + *
>> + * Given the number of possible events defined by SCMI and the extensibility
>> + * of the SCMI Protocol itself, such underlying notification chains are created
>> + * and destroyed dynamically on demand depending on the number of users
>> + * effectively registered for an event, so that no structures or chains are
>> + * allocated until at least one user has registered a notifier_block for such
>> + * event. Similarly, events' generation itself is enabled at the platform level
>> + * only after at least one user has registered, and it is shutdown after the
>> + * last user for that event has gone.
>> + *
>> + * An event is identified univocally by the tuple (proto_id, evt_id, src_id)
>> + * and is served by its own dedicated notification chain: given that such chain
>> + * is dynamically created, the registration API simply let the users associate
>> + * their callbacks with the above tuple.
>> + *
>> + * Here proto_id and evt_id are simply the protocol_id and message_id numbers as
>> + * reported in the SCMI Protocol specification, while src_id represents an
>> + * optional, protocol dependent, source identifier (like domain_id, perf_id
>> + * or sensor_id and so forth), so that a user can register its callbacks for a
>> + * particular event coming only from a defined source (like CPU vs GPU).
>> + * When the source is not specified the user callback will be registered for
>> + * all existing sources for that event (if any).
>> + *
>> + * Upon reception of a notification message from the platform the SCMI RX ISR
>> + * passes the received message payload and some ancillary information (including
>> + * an arrival timestamp in nanoseconds) to the core via @scmi_notify(), which,
>> + * in turn, after having looked up the event in the registered events mapping,
>> + * pushes the event-data itself on a protocol dedicated kfifo queue for deferred
>> + * processing.
>> + *
>> + * Such dedicated protocols' queues are allocated once for all at initialization
>> + * time, together with a dedicated work_item running the common delivery logic
>> + * of @scmi_events_dispatcher(), so that each protocol has it own dedicated
>> + * worker which, once kicked by the ISR, takes care to empty is own dedicated
>> + * queue deliverying the queued items into the proper notification chain.
>> + * Note that since the underlying cmwq workers run one distinct work_item per
>> + * protocol and there are a number of max-active workers equal to the number of
>> + * protocols, notifications processing can proceed concurrently only between
>> + * events belonging to different protocols, while delivery of events within the
>> + * same protocol is still strictly sequentially ordered by time of arrival; this
>> + * separation effectively avoids the possibility that one particularly verbose
>> + * protocol, flooding the queues with events, can cause other protocols' events
>> + * to be lost or their processing starved.
>> + *
>> + * Event's information is converted to a custom per-event report struct before
>> + * being fed as void *data param to the user callback embedded in the registered
>> + * notifier_block so that it looks like as follow from the user perspective:
>> + *
>> + * int user_cb(struct notifier_block *nb, unsigned long event_id, void *report)
>> + *
> No point in having blank lines at the end of a comment block.
>> + *
>> + */
>> +
>> +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
>> +
>> +#include <linux/bug.h>
>> +#include <linux/bitfield.h>
>> +#include <linux/err.h>
>> +#include <linux/idr.h>
>> +#include <linux/kernel.h>
>> +#include <linux/kfifo.h>
>> +#include <linux/notifier.h>
>> +#include <linux/refcount.h>
>> +#include <linux/slab.h>
>> +#include <linux/scmi_protocol.h>
>> +#include <linux/types.h>
>> +#include <linux/workqueue.h>
>> +
>> +#include "notify.h"
>> +
>> +#define SCMI_NOTIF_MAX_ACTIVE_QUEUES	7
>> +
>> +/*
>> + * Builds an unsigned 32bit key from the given input tuple to be used as a key
>> + * in IDR mappings; note that if __src_p is passed in as NULL, the returned key
>> + * is built using SCMI_ALL_SRC_IDS as src_id.
>> + */
>> +#define MAKE_EVT_KEY(__proto, __evt, __src_p)				\
>> +	({								\
>> +		u32 __evt_key, __src_id;				\
>> +									\
>> +		__src_id = ((__src_p)) ?				\
>> +			    (*((u32 *)(__src_p)) & SCMI_ALL_SRC_IDS) :	\
>> +			    SCMI_ALL_SRC_IDS;				\
>> +		__evt_key = ((__proto) << 24) |				\
>> +				((__evt) << 16) | __src_id;		\
>> +		__evt_key;						\
>> +	})
>> +
>> +#define PROTO_ID_MASK			GENMASK(31, 24)
>> +#define EVT_ID_MASK			GENMASK(23, 16)
>> +#define SRC_ID_MASK			GENMASK(15, 0)
>> +#define KEY_XTRACT_PROTO_ID(key)	FIELD_GET(PROTO_ID_MASK, (key))
>> +#define KEY_XTRACT_EVT_ID(key)		FIELD_GET(EVT_ID_MASK, (key))
>> +#define KEY_XTRACT_SRC_ID(key)		FIELD_GET(SRC_ID_MASK, (key))
>> +
>> +/**
>> + * events_queue  - Describes a queue and its associated worker
> 
> Not valid kernel-doc.    I'll assume all other instances will get fixed
> and not bother talking about them.
> 
>> + *
>> + * Each protocol has its own dedicated events_queue descriptor.
>> + *
>> + * @sz: Size in bytes of the related kfifo
>> + * @kfifo: A dedicated Kernel kfifo
>> + * @notify_work: A custom work item bound to this queue
>> + * @wq: A reference to the related workqueue
>> + */
>> +struct events_queue {
>> +	size_t			sz;
>> +	struct kfifo		kfifo;
>> +	struct work_struct	notify_work;
>> +	struct workqueue_struct	*wq;
>> +};
>> +
>> +/**
>> + * scmi_registered_protocol_events_desc  - Protocol Specific information
>> + *
>> + * All protocols that registers at least an event have their protocol-specific
>> + * information stored here, together with a pointer to the allocated
>> + * events_queue.
>> + *
>> + * @id: Protocol ID
>> + * @ops: Protocol specific and event-related operations
>> + * @equeue: A reference to the associated per-protocol events_queue
>> + */
>> +struct scmi_registered_protocol_events_desc {
>> +	u8					id;
>> +	const struct scmi_protocol_event_ops	*ops;
>> +	struct events_queue			*equeue;
>> +};
>> +
>> +/**
>> + * scmi_registered_event  - Event Specific Information
>> + *
>> + * All registered events are represented by one of these structures.
>> + *
>> + * @proto: A reference to the associated protocol descriptor
>> + * @evt: A reference to the associated event descriptor (as provided at
>> + *       registration time)
>> + * @scratch_isr: A pre-allocated buffer to be used as a scratch area by ISR
>> + * @scratch_isr: A pre-allocated buffer to be used as a scratch area by the
>> + *		 deferred worker
> scratch_bh.
> 
>> + * @report: A pre-allocated buffer used by the deferred workr to fill a
> 
> worker
> 
>> + *	    customized event report
>> + */
>> +struct scmi_registered_event {
>> +	struct scmi_registered_protocol_events_desc	*proto;
>> +	const struct scmi_event				*evt;
>> +	void						*scratch_isr;
>> +	void						*scratch_bh;
>> +	void						*report;
>> +};
>> +
>> +/**
>> + * scmi_event_handler  - Event handler information
>> + *
>> + * This structure collects all the information needed to process a received
>> + * event, calling all the registered notifier callbacks.
>> + * Note this an event handler is associated to a tuple of values like:
>> + *	(proto_id, evt_id, src_id)
>> + * through the means of a key and an IDR mapping.
>> + *
>> + * @evt_key: The unsigned 32bit key associated to this descriptor in the
>> + *	     related IDR mapping
>> + * @r_evt: A reference to the underlying registered event
>> + * @chain: The notification chain dedicated to this specific event tuple
>> + */
>> +struct scmi_event_handler {
>> +	u32				evt_key;
>> +	bool				enabled;
>> +	refcount_t			users;
>> +	struct scmi_registered_event	*r_evt;
>> +	struct blocking_notifier_head	chain;
>> +};
>> +
>> +/**
>> + * scmi_event_header  - A utility header
> 
> That's not a good description.
> 
>> + *
>> + * This header is prepended to each received event message payload before
>> + * being queued on the related events_queue, and describes the attached event
>> + * message.
>> + *
>> + * @timestamp: The timestamp, innanoseconds (boottime), which was associated
>> + *	       to this event as soon as it entered the SCMI RX ISR
>> + * @proto_id: Protocol ID
>> + * @evt_id: Event ID (corresponds to the Event MsgID for this Protocol)
>> + * @payld_sz: Effective size of the attached message payload which follows
>> + * @payld: A reference to the included Event payload
>> + */
>> +struct scmi_event_header {
>> +	u64	timestamp;
>> +	u8	proto_id;
>> +	u8	evt_id;
>> +	size_t	payld_sz;
>> +	u8	payld[];
>> +} __packed;
>> +
>> +/*
>> + * A few IDR maps to track:
>> + *
>> + * - @scmi_registered_events: All event's descriptors registered by the
>> + *			      protocols, together with their ancillary data
>> + * - @scmi_registered_events_handlers: All event's handlers descriptors, created
>> + *				       to collect all the users' notifier_block
>> + *				       callbacks and related notification chains
>> + * - @scmi_registered_protocols: All protocol-level specific information related
>> + *				 to events' handling
>> + */
>> +static struct idr scmi_registered_events;
>> +static struct idr scmi_registered_events_handlers;
>> +static DEFINE_MUTEX(scmi_registered_events_handler_mtx);
>> +static struct idr scmi_registered_protocols;
>> +
>> +/* Common Kernel cmwq workqueue used by notifications core */
>> +static struct workqueue_struct *scmi_notify_wq;
>> +
>> +static bool scmi_notifications_initialized;
>> +
>> +static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key);
>> +static void scmi_put_event_handler(struct scmi_event_handler *hndl);
>> +
>> +/**
>> + * scmi_discard_bad_evt_payloadi() - Discard data from a kfifo
> 
> function name is wrong.
> 
>> + *
>> + * @kq: The kfifo to act on
>> + * @count: Number of bytes to flush
>> + */
>> +static inline void scmi_discard_bad_evt_payload(struct kfifo *kq,
>> +						const unsigned int count)
>> +{
>> +	int i = 0;
>> +
>> +	pr_warn("SCMI Notification WQ :: skipping bad EVT Payload - %d bytes\n",
>> +		count);
>> +	/* Discard stale pending queued payload. */
>> +	for (i = 0; i < count; i++)
>> +		kfifo_skip(kq);
>> +}
>> +
>> +/**
>> + * scmi_lookup_and_call_event_chain  - Lookup the proper chain and call it
>> + *
>> + * @evt_key: The event key to use to lookup the related notification chain
>> + * @report: The customized event-specific report to pass down to the callbacks
>> + *	    as their *data parameter.
>> + */
>> +static inline void scmi_lookup_and_call_event_chain(u32 evt_key, void *report)
>> +{
>> +	int ret;
>> +	struct scmi_event_handler *hndl;
>> +
>> +	hndl = scmi_get_event_handler(evt_key);
>> +	if (IS_ERR_OR_NULL(hndl))
>> +		return;
>> +
>> +	ret = blocking_notifier_call_chain(&hndl->chain,
>> +					   KEY_XTRACT_EVT_ID(evt_key),
>> +					   report);
>> +	/* Notifiers are NOT supposed to cut the chain */
>> +	WARN_ON_ONCE(ret & NOTIFY_STOP_MASK);
>> +
>> +	scmi_put_event_handler(hndl);
>> +}
>> +
>> +/**
>> + * scmi_events_dispatcher  - Common worker logic for all work items.
>> + *
>> + * In turn:
>> + *  1. dequeue one pending RX notification (queued in SCMI RX ISR context)
>> + *  2. generate a custom event report from the received event message
>> + *  3. lookup for any registered ALL_SRC_IDs handler
>> + *     - > call the related notification chain passing in the report
>> + *  4. lookup for any registered specific SRC_ID handler
>> + *     - > call the related notification chain passing in the report
>> + *
>> + * Note that:
>> + * - a dedicated per-protocol kfifo queue is used: in this way an anomalous
>> + *   flood of events cannot saturate other protocols' queues.
>> + *
>> + * - each per-protocol queue is associated to a distinct work_item, which
>> + *   means, in turn, that:
>> + *   + all protocol can process their dedicated queues concurrently
>> + *     (since scmi_notify_wq:max_active > 1)
>> + *   + anyway at most one worker instance is allowed to run on the same queue
>> + *     concurrently: this ensures that we can have only one concurrent
>> + *     reader/writer on the associated kfifo (needed for lock-less access)
>> + *
>> + * @work: The work item to use, which is associated to the proper events_queue
>> + */
>> +static void scmi_events_dispatcher(struct work_struct *work)
>> +{
>> +	struct events_queue *equeue;
>> +	struct scmi_event_header eh;
>> +
>> +	equeue = container_of(work, struct events_queue, notify_work);
>> +	while (kfifo_out(&equeue->kfifo, &eh, sizeof(eh))) {
>> +		u32 src_id, evt_key;
>> +		unsigned int outs;
>> +		struct scmi_registered_event *r_evt;
>> +		void *report = NULL;
>> +
>> +		evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, NULL);
>> +		r_evt = idr_find(&scmi_registered_events, evt_key);
>> +		if (!r_evt) {
>> +			scmi_discard_bad_evt_payload(&equeue->kfifo,
>> +						     eh.payld_sz);
>> +			continue;
>> +		}
>> +
>> +		outs = kfifo_out(&equeue->kfifo, r_evt->scratch_bh,
>> +				 eh.payld_sz);
>> +		if (outs != eh.payld_sz) {
>> +			pr_warn("SCMI Notification WQ :: SKIP corrupted EVT Payload.\n");
>> +			continue;
>> +		}
>> +
>> +		/* Reset and fill custom report */
>> +		memset(r_evt->report, 0x00, r_evt->evt->max_report_sz);
>> +		report = r_evt->proto->ops->fill_custom_report(eh.evt_id,
>> +							eh.timestamp,
>> +							r_evt->scratch_bh,
>> +							eh.payld_sz,
>> +							r_evt->report,
>> +							&src_id);
>> +		if (!report)
>> +			continue;
>> +
>> +		/* At first search for a generic ALL src_ids handler... */
>> +		scmi_lookup_and_call_event_chain(evt_key, report);
>> +		if (src_id != SCMI_ALL_SRC_IDS) {
>> +			u32 *__sub_p = &src_id;
>> +
>> +			evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, __sub_p);
>> +			scmi_lookup_and_call_event_chain(evt_key, report);
>> +		}
>> +	}
>> +}
>> +
>> +/**
>> + * scmi_notify  - Queues a notification for further deferred processing
>> + *
>> + * This is called in interrupt context to queue a received event for
>> + * deferred processing.
>> + *
>> + * @proto_id: Protocol ID
>> + * @evt_id: Event ID (msgID)
>> + * @buf: Event Message Payload (without the header)
>> + * @len: Event Message Payload size
>> + * @ts: RX Timestamp in nanoseconds (boottime)
>> + *
>> + * Return: 0 on Success
>> + */
>> +int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts)
>> +{
>> +	struct scmi_registered_event *r_evt;
>> +	struct scmi_event_header *eh;
>> +
>> +	if (unlikely(!scmi_notifications_initialized))
>> +		return 0;
>> +
>> +	r_evt = idr_find(&scmi_registered_events,
>> +			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
>> +	if (unlikely(!r_evt || !r_evt->proto->equeue))
>> +		return -EINVAL;
>> +
>> +	if (unlikely(len > r_evt->evt->max_payld_sz)) {
>> +		pr_err("SCMI Notification discard badly sized message\n");
>> +		return -EINVAL;
>> +	}
>> +	if (unlikely(kfifo_avail(&r_evt->proto->equeue->kfifo) <
>> +		     sizeof(*eh) + len)) {
>> +		pr_warn("SCMI Notification queue full: dropping proto_id:%d  evt_id:%d  ts:%lld\n",
>> +			proto_id, evt_id, ts);
>> +		return -ENOMEM;
>> +	}
>> +
>> +	/*
>> +	 * Note that we can use the per protocol kfifo in a lock-less manner
>> +	 * since we have only one concurrent reader and writer but, in order
>> +	 * to avoid any trouble on the reader side, here we perform one single
>> +	 * write, so we have to collate event header and payload in a scratch
>> +	 * area at first.
>> +	 */
>> +	eh = r_evt->scratch_isr;
>> +	eh->timestamp = ts;
>> +	eh->proto_id = proto_id;
>> +	eh->evt_id = evt_id;
>> +	eh->payld_sz = len;
>> +	memcpy(eh->payld, buf, eh->payld_sz);
>> +	kfifo_in(&r_evt->proto->equeue->kfifo, eh, sizeof(*eh) + eh->payld_sz);
>> +	queue_work(r_evt->proto->equeue->wq,
>> +		   &r_evt->proto->equeue->notify_work);
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_allocate_events_queue  - Allocate an events_queue descriptor
>> + *
>> + * @sz: Size of the kfifo to initialize
>> + *
>> + * Return: A valid pointer to the allocated events_queue on Success
>> + */
>> +static struct events_queue *scmi_allocate_events_queue(size_t sz)
>> +{
>> +	int ret;
>> +	struct events_queue *equeue;
>> +
>> +	equeue = kzalloc(sizeof(*equeue), GFP_KERNEL);
>> +	if (!equeue)
>> +		return ERR_PTR(-ENOMEM);
>> +	ret = kfifo_alloc(&equeue->kfifo, sz, GFP_KERNEL);
>> +	if (ret) {
>> +		kfree(equeue);
>> +		return ERR_PTR(ret);
>> +	}
>> +	equeue->sz = sz;
>> +	INIT_WORK(&equeue->notify_work, scmi_events_dispatcher);
>> +	equeue->wq = scmi_notify_wq;
>> +
>> +	return equeue;
>> +}
>> +
>> +/**
>> + * scmi_deallocate_events_queue  - Deallocate am events_queue descriptor
>> + *
>> + * @equeue: The events_queue to free
>> + */
>> +static void scmi_deallocate_events_queue(struct events_queue *equeue)
>> +{
>> +	kfifo_free(&equeue->kfifo);
>> +	kfree(equeue);
>> +}
>> +
>> +/**
>> + * scmi_allocate_registered_protocol_desc  - Allocate a registered protocol
>> + * events' descriptor
>> + *
>> + * Used to keep protocol specific information related to events handling for any
>> + * protocl which has registered at least one event.
> 
> protocol
> 
>> + *
>> + * @proto_id: Protocol ID
>> + * @queue_sz: Size of the associated queue to allocate
>> + * @ops: Pointer to a struct holding references to protocol specific helpers
>> + *	 needed during events handling
>> + */
>> +static struct scmi_registered_protocol_events_desc *
>> +scmi_allocate_registered_protocol_desc(u8 proto_id, size_t queue_sz,
>> +				const struct scmi_protocol_event_ops *ops)
>> +{
>> +	int ret;
>> +	struct scmi_registered_protocol_events_desc *pdesc;
>> +
>> +	pdesc = idr_find(&scmi_registered_protocols, proto_id);
> 
> Perhaps add a comment here on why this might get called multiple times
> but is still safe from point of view of release path.
> 
>> +	if (pdesc)
>> +		return pdesc;
> 
> 
>> +
>> +	pdesc = kzalloc(sizeof(*pdesc), GFP_KERNEL);
>> +	if (!pdesc)
>> +		return ERR_PTR(-ENOMEM);
>> +	pdesc->id = proto_id;
>> +	pdesc->ops = ops;
>> +
>> +	pdesc->equeue = scmi_allocate_events_queue(queue_sz);
>> +	if (IS_ERR(pdesc->equeue)) {
>> +		kfree(pdesc);
>> +		return ERR_CAST(pdesc->equeue);
>> +	}
>> +
>> +	ret = idr_alloc(&scmi_registered_protocols, pdesc,
>> +			pdesc->id, pdesc->id + 1, GFP_KERNEL);
>> +	if (ret < 0) {
>> +		pr_err("SCMI Failed to allocate PDESC IDR - key:%d - err:%d\n",
>> +		       pdesc->id, ret);
>> +		scmi_deallocate_events_queue(pdesc->equeue);
>> +		kfree(pdesc);
>> +		return ERR_PTR(ret);
>> +	}
>> +
>> +	return pdesc;
>> +}
>> +
>> +/**
>> + * scmi_register_protocol_events  - Register Protocol Events with the core
>> + *
>> + * Used by SCMI Protocols initialization code to register with the notification
>> + * core the list of supported events and their description: this takes care to
>> + * pre-allocate all needed scratch buffers and link the proper event queue to
>> + * this event.
>> + *
>> + * @proto_id: Protocol ID
>> + * @queue_sz: Size in bytes of the associated queue to be allocated
>> + * @ops: Protocol specific event-related operations
>> + * @evt: Event descriptor array
>> + * @num_events: Number of events in @evt array
>> + *
>> + * Return: 0 on Success
>> + */
>> +int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
>> +				  const struct scmi_protocol_event_ops *ops,
>> +				  const struct scmi_event *evt, int num_events)
>> +{
>> +	int i;
>> +	struct scmi_registered_protocol_events_desc *pdesc;
>> +
>> +	if (!scmi_notifications_initialized)
>> +		return -EAGAIN;
>> +
>> +	if (!ops || !evt)
>> +		return -EINVAL;
>> +
>> +	pdesc = scmi_allocate_registered_protocol_desc(proto_id, queue_sz, ops);
>> +	if (IS_ERR(pdesc))
>> +		return -ENOMEM;
>> +
>> +	for (i = 0; i < num_events; i++, evt++) {
>> +		int ret;
>> +		u32 key;
>> +		struct scmi_registered_event *r_evt;
>> +
>> +		r_evt = kzalloc(sizeof(*r_evt), GFP_KERNEL);
>> +		if (!r_evt)
>> +			continue;
>> +		r_evt->proto = pdesc;
>> +		r_evt->evt = evt;
>> +
>> +		r_evt->scratch_isr = kzalloc(sizeof(struct scmi_event_header) +
>> +					     evt->max_payld_sz, GFP_KERNEL);
>> +		if (!r_evt->scratch_isr)
>> +			goto continue_isr_fail;
>> +
>> +		r_evt->scratch_bh = kzalloc(evt->max_payld_sz, GFP_KERNEL);
>> +		if (!r_evt->scratch_bh)
>> +			goto continue_bh_fail;
>> +
>> +		r_evt->report = kzalloc(evt->max_report_sz, GFP_KERNEL);
>> +		if (!r_evt->report)
>> +			goto continue_report_fail;
>> +
>> +		key = MAKE_EVT_KEY(r_evt->proto->id, evt->evt_id, NULL);
>> +		ret = idr_alloc(&scmi_registered_events, r_evt,
>> +				key, key + 1, GFP_KERNEL);
>> +		if (ret < 0) {
>> +			pr_err("SCMI Failed to allocate EVENT IDR - key:%X - err:%d\n",
>> +			       key, ret);
>> +			goto continue_idr_fail;
>> +		}
>> +
>> +		pr_info("SCMI Notification registered event - key:%X\n", key);
>> +		continue;
>> +
>> +	/* yes, this is not nice ... */
>> +continue_idr_fail:
>> +		kfree(r_evt->report);
>> +continue_report_fail:
>> +		kfree(r_evt->scratch_bh);
>> +continue_bh_fail:
>> +		kfree(r_evt->scratch_isr);
>> +continue_isr_fail:
>> +		kfree(r_evt);
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_register_event_handler  - Allocate an Event handler
>> + *
>> + * Allocate an event handler and related notification chain associated with
>> + * the event identified by the provided event key. Fails if the associated
>> + * event is unknown to the core (i.e. it had not been successfully registered
>> + * upfront by some protocol)
>> + *
>> + * @evt_key: 32bit key uniquely bind to the event identified by the tuple
>> + *	     (proto_id, evt_id, src_id)
>> + *
>> + * Return: the freshly allocated structure on Success
>> + */
>> +static struct scmi_event_handler *scmi_register_event_handler(u32 evt_key)
>> +{
>> +	int id;
>> +	u8 proto_id, evt_id;
>> +	struct scmi_registered_event *r_evt;
>> +	struct scmi_event_handler *hndl;
>> +
>> +	proto_id = KEY_XTRACT_PROTO_ID(evt_key);
>> +	evt_id = KEY_XTRACT_EVT_ID(evt_key);
>> +	r_evt = idr_find(&scmi_registered_events,
>> +			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
>> +	if (!r_evt)
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	hndl = kzalloc(sizeof(*hndl), GFP_KERNEL);
>> +	if (!hndl)
>> +		return ERR_PTR(-ENOMEM);
>> +	hndl->r_evt = r_evt;
>> +	hndl->evt_key = evt_key;
>> +	BLOCKING_INIT_NOTIFIER_HEAD(&hndl->chain);
>> +	refcount_set(&hndl->users, 1);
>> +
>> +	/* Register freshly allocated event handler */
>> +	id = idr_alloc(&scmi_registered_events_handlers, hndl,
>> +		       evt_key, evt_key + 1, GFP_KERNEL);
> 
> Umm. That's not using an idr for what it is meant for. You aren't allocating an id, you
> are using it as a poor wrapper around an xarray.
> 
>> +	if (id < 0) {
>> +		pr_err("SCMI Failed to allocate HNDL IDR - key:%X  err:%d\n",
>> +		       evt_key, id);
>> +		kfree(hndl);
>> +		return ERR_PTR(id);
>> +	}
>> +
>> +	return hndl;
>> +}
>> +
>> +/**
>> + * scmi_unregister_event_handler  - Free the provided Event handler
>> + *
>> + * @hndl: The event handler structure to free
>> + */
>> +static void scmi_unregister_event_handler(struct scmi_event_handler *hndl)
>> +{
>> +	idr_remove(&scmi_registered_events_handlers, hndl->evt_key);
>> +	kfree(hndl);
>> +}
>> +
>> +/**
>> + * __scmi_event_handler_get_ops  - Get or create an event handler
>> + *
>> + * After having got exclusive access to the registered events map, searches for
>> + * the desired handler matching the key: when found adjust refcount, when not
>> + * create and register a new handler is asked to do so.
>> + * Events generation is NOT enabled within this routine since at creation time
>> + * we usually want to have all setup and registered before events starts
>> + * flowing.
>> + *
>> + * @evt_key: The event key to use
>> + * @create: A boolean flag to specify if a handler must be created
>> + *	    when not found
>> + */
>> +static inline struct scmi_event_handler *
>> +__scmi_event_handler_get_ops(u32 evt_key, bool create)
>> +{
>> +	struct scmi_event_handler *hndl = NULL;
>> +
>> +	mutex_lock(&scmi_registered_events_handler_mtx);
>> +	hndl = idr_find(&scmi_registered_events_handlers, evt_key);
>> +	if (hndl)
>> +		refcount_inc(&hndl->users);
>> +	else if (create)
>> +		hndl = scmi_register_event_handler(evt_key);
>> +	mutex_unlock(&scmi_registered_events_handler_mtx);
>> +
>> +	return hndl;
>> +}
>> +
>> +static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key)
>> +{
>> +	return __scmi_event_handler_get_ops(evt_key, false);
>> +}
>> +
>> +static struct scmi_event_handler *scmi_get_or_create_event_handler(u32 evt_key)
>> +{
>> +	return __scmi_event_handler_get_ops(evt_key, true);
>> +}
>> +
>> +/**
>> + * __scmi_enable_events_ops  - Enable/disable events generation
>> + *
>> + * @hndl: The handler specifying the events enable/disable
>> + * @action: The action to perform: true->Enable false->Disable
>> + *
>> + * Return: True when the required @action has been successfully executed
>> + */
>> +static inline bool __scmi_enable_events_ops(struct scmi_event_handler *hndl,
>> +					    bool action)
>> +{
>> +	bool ret = true;
>> +
>> +	if (hndl->enabled != action) {
>> +		u8 evt_id;
>> +		u32 src_id;
>> +
>> +		evt_id = KEY_XTRACT_EVT_ID(hndl->evt_key);
>> +		src_id = KEY_XTRACT_SRC_ID(hndl->evt_key);
>> +		ret = hndl->r_evt->proto->ops->set_notify_enabled(evt_id,
>> +			      src_id != SCMI_ALL_SRC_IDS ? &src_id : NULL,
>> +			      action);
>> +		if (ret)
>> +			hndl->enabled = action;
>> +	}
>> +
>> +	return ret;
>> +}
>> +
>> +static bool scmi_enable_events(struct scmi_event_handler *hndl)
>> +{
>> +	return __scmi_enable_events_ops(hndl, true);
>> +}
>> +
>> +static bool scmi_disable_events(struct scmi_event_handler *hndl)
>> +{
>> +	return __scmi_enable_events_ops(hndl, false);
>> +}
>> +
>> +/**
>> + * scmi_put_event_handler  - Put an event handler
>> + *
>> + * After having got exclusive access to the registered events map, update
>> + * the refcount and if @hndl is no more in use by anyone:
>> + *  - disable the events generation
>> + *  - unregister and free the handler itself
>> + *
>> + * @hndl: The handler to act upon
>> + */
>> +static void scmi_put_event_handler(struct scmi_event_handler *hndl)
>> +{
>> +	mutex_lock(&scmi_registered_events_handler_mtx);
>> +	if (refcount_dec_and_test(&hndl->users)) {
>> +		scmi_disable_events(hndl);
>> +		scmi_unregister_event_handler(hndl);
>> +	}
>> +	mutex_unlock(&scmi_registered_events_handler_mtx);
>> +}
>> +
>> +/**
>> + * scmi_register_event_notifier  - Register a notifier_block for an event
>> + *
>> + * Generic helper to register a notifier_block attached to a protocol event.
>> + *
>> + * A notifier_block @nb will be registered for each distinct event identified
>> + * by the tuple (proto_id, evt_id, src_id) on a dedicated notification chain
>> + * so that:
>> + *
>> + *	(proto_X, evt_Y, src_Z) --> chain_X_Y_Z
>> + *
>> + * @src_id meaning is protocol specific and identifies the origin of the event
>> + * (like domain_id, sensor_id and os forth); @src_id can be NULL to signify that
> 
> so
> 
>> + * the caller is interested in receiving notifications from ALL the available
>> + * sources for that protocol OR simply that the protocol does not support
>> + * distinct sources: in these cases @nb will be attached to a generic
>> + * notification chain defined for ALL src_id of that proto_id/evt_id pair like:
>> + *
>> + *	(proto_X, evt_Y, NULL) --> chain_X_Y_ALL
>> + *
>> + * Any received event will be then dispatched to both such chains if at least
>> + * one user had registered an @nb on them.
> 
> both should be all I think.
> 
>> + *
>> + * Note also that notification chains are created dynamically: a new chain is
>> + * created each time a new distinct tuple is detected and then the provided @nb
>> + * is bound to such chain; at creation time the related SCMI notifications are
>> + * also enabled: this way, only when at least one user has registered its
>> + * interest for a specific event tuple, the underlying notification chain is
>> + * created and the related notifications are enabled in the platform.
> 
> That's one of the most complex bits of punctuation I've seenin a comment for
> a long time.   Please break that sentence up for readability.
> 
>> + *
>> + * @proto_id: Protocol ID
>> + * @evt_id: Event ID
>> + * @src_id: Source ID
>> + * @nb: A standard notifier block to register for the specified event
>> + *
>> + * Return: Return 0 on Success
>> + */
>> +int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
>> +				 struct notifier_block *nb)
>> +{
>> +	u32 evt_key;
>> +	struct scmi_event_handler *hndl;
>> +
>> +	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
>> +	hndl = scmi_get_or_create_event_handler(evt_key);
>> +	if (IS_ERR_OR_NULL(hndl))
>> +		return PTR_ERR(hndl);
>> +
>> +	blocking_notifier_chain_register(&hndl->chain, nb);
>> +
>> +	if (!scmi_enable_events(hndl)) {
>> +		pr_err("SCMI Failed to ENABLE events for key:%X !\n", evt_key);
>> +		blocking_notifier_chain_unregister(&hndl->chain, nb);
>> +		scmi_put_event_handler(hndl);
>> +		return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_unregister_event_notifier  - Unregister a notifier_block for an event
>> + *
>> + * Takes care to unregister the provided @nb from the notification chain
>> + * associated to the specified event and, if there are no more users for the
>> + * event handler, frees also the associated event handler structures.
>> + *
>> + * @proto_id: Protocol ID
>> + * @evt_id: Event ID
>> + * @src_id: Source ID
>> + * @nb: The notifier_block to unregister
>> + *
>> + * Return: 0 on Success
>> + */
>> +int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
>> +				   struct notifier_block *nb)
>> +{
>> +	u32 evt_key;
>> +	struct scmi_event_handler *hndl;
>> +
>> +	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
>> +	hndl = scmi_get_event_handler(evt_key);
>> +	if (IS_ERR_OR_NULL(hndl))
>> +		return -EINVAL;
>> +
>> +	blocking_notifier_chain_unregister(&hndl->chain, nb);
>> +
>> +	scmi_put_event_handler(hndl);
>> +	/*
>> +	 * If this was the last user callback for this handler, this last put
>> +	 * will force the handler to be freed.
>> +	 * Note that if a call_chain walk is ongoing it will be the call_chain
>> +	 * put request which will finally free the handler; note also that any
>> +	 * operation on the inner notifier_block chain is protected on its own.
>> +	 */
>> +	scmi_put_event_handler(hndl);
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_notification_init  - Initializes Notification Core Support
>> + *
>> + * Return: 0 on Success
>> + */
>> +int __init scmi_notification_init(void)
>> +{
>> +	scmi_notify_wq = alloc_workqueue("scmi_notify",
>> +					 WQ_UNBOUND | WQ_FREEZABLE,
>> +					 SCMI_NOTIF_MAX_ACTIVE_QUEUES);
>> +	if (!scmi_notify_wq)
>> +		return -ENOMEM;
>> +	idr_init(&scmi_registered_events);
>> +	idr_init(&scmi_registered_events_handlers);
>> +	idr_init(&scmi_registered_protocols);
>> +
>> +	scmi_notifications_initialized = true;
>> +
>> +	pr_info("SCMI Notifications enabled.\n");
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_notification_exit  - Shutdown and clean Notification core
>> + */
>> +void __exit scmi_notification_exit(void)
>> +{
>> +	int k;
>> +	struct scmi_event_handler *hndl;
>> +	struct scmi_registered_protocol_events_desc *pdesc;
>> +	struct scmi_registered_event *r_evt;
>> +
>> +	scmi_notifications_initialized = false;
>> +
>> +	/* Disable notifications' generation */
>> +	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
>> +		scmi_disable_events(hndl);
>> +
>> +	/* Let pending work complete */
>> +	destroy_workqueue(scmi_notify_wq);
>> +
>> +	/* Remove ALL events handlers */
>> +	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
>> +		scmi_put_event_handler(hndl);
>> +	idr_destroy(&scmi_registered_events_handlers);
>> +
>> +	/* Finally remove all registered events */
> 
> Finally seems a bit premature here ;)
> 
>> +	idr_for_each_entry(&scmi_registered_events, r_evt, k) {
>> +		idr_remove(&scmi_registered_events, k);
>> +		kfree(r_evt->report);
>> +		kfree(r_evt->scratch_isr);
>> +		kfree(r_evt->scratch_bh);
>> +		kfree(r_evt);
>> +	}
>> +	idr_destroy(&scmi_registered_events);
>> +
>> +	/* free any remaining protocol data */
>> +	idr_for_each_entry(&scmi_registered_protocols, pdesc, k) {
>> +		idr_remove(&scmi_registered_protocols, k);
>> +		scmi_deallocate_events_queue(pdesc->equeue);
>> +		kfree(pdesc);
>> +	}
>> +	idr_destroy(&scmi_registered_protocols);
>> +}
>> diff --git a/drivers/firmware/arm_scmi/notify.h b/drivers/firmware/arm_scmi/notify.h
>> new file mode 100644
>> index 000000000000..6fa07b244b14
>> --- /dev/null
>> +++ b/drivers/firmware/arm_scmi/notify.h
>> @@ -0,0 +1,79 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * System Control and Management Interface (SCMI) Message Protocol
>> + * notification header file containing some definitions, structures
>> + * and function prototypes related to SCMI Notification handling.
>> + *
>> + * Copyright (C) 2019 ARM Ltd.
>> + */
>> +#ifndef _SCMI_NOTIFY_H
>> +#define _SCMI_NOTIFY_H
>> +
>> +#include <linux/bug.h>
>> +#include <linux/notifier.h>
>> +#include <linux/types.h>
>> +
>> +#define	SCMI_ALL_SRC_IDS		0x0000ffffL
>> +
>> +#define MAP_EVT_TO_ENABLE_CMD(id, map)			\
>> +({							\
>> +	int ret = -1;					\
>> +							\
>> +	if (likely((id) < ARRAY_SIZE((map))))		\
>> +		ret = (map)[(id)];			\
>> +	else						\
>> +		WARN(1, "UN-KNOWN evt_id:%d\n", (id));	\
>> +	ret;						\
>> +})
>> +
>> +/**
>> + * scmi_event  - Describes an event to be supported
> 
> I don't think this is valid kernel-doc.  Make sure to point the scripts at the
> file and check and fix any warnings.
> 
>> + *
>> + * Each SCMI protocol, during its initialization phase, can describe the events
>> + * it wishes to support in a few struct scmi_event and pass them to the core
>> + * using scmi_register_protocol_events().
>> + *
>> + * @evt_id: Event ID
>> + * @max_payld_sz: Max possible size for the payload of a notif msg of this kind
>> + * @max_report_sz: Max possible size for the report of a notif msg of this kind
>> + */
>> +struct scmi_event {
>> +	u8	evt_id;
>> +	size_t	max_payld_sz;
>> +	size_t	max_report_sz;
>> +
>> +};
>> +
>> +/**
>> + * scmi_protocol_event_ops  - Helpers called by notification core.
>> + *
>> + * These are called only in process context.
>> + *
>> + * @set_notify_enabled: Enable/disable the required evt_id/src_id notifications
>> + *			using the proper custom protocol commands.
>> + *			Return true if at least one the required src_id
>> + *			has been successfully enabled/disabled
>> + * @fill_custom_report: fills a custom event report from the provided
> 
> Nitpick.  Inconsistent capitalization after the :
> 
>> + *			event message payld identifying the event
>> + *			specific src_id.
>> + *			Return NULL on failure otherwise @report now fully
>> + *			populated
>> + */
>> +struct scmi_protocol_event_ops {
>> +	bool (*set_notify_enabled)(u8 evt_id, const u32 *src_id, bool enabled);
>> +	void *(*fill_custom_report)(u8 evt_id, u64 timestamp, const void *payld,
>> +				    size_t payld_sz, void *report, u32 *src_id);
>> +};
>> +
>> +int scmi_notification_init(void);
>> +void scmi_notification_exit(void);
>> +int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
>> +				  const struct scmi_protocol_event_ops *ops,
>> +				  const struct scmi_event *evt, int num_events);
>> +int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
>> +				 struct notifier_block *nb);
>> +int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
>> +				   struct notifier_block *nb);
>> +int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts);
>> +
>> +#endif /* _SCMI_NOTIFY_H */
> 
> 

Re: [RFC PATCH 04/11] firmware: arm_scmi: Add core notifications support

[Cristian Marussi]

Hi

On 27/01/2020 18:11, Jonathan Cameron wrote:
> On Mon, 20 Jan 2020 12:23:26 +0000
> Cristian Marussi <cristian.marussi@arm.com> wrote:
> 
>> Add basic support for SCMI Notifications, using Kernel notification chains
>> mechanism.
>>
>> Each SCMI Protocol has a dedicated events' queue and deferred worker which
>> is in charge of delivering the notifications to the interested users
>> invoking the users' registered callbacks.
>>
>> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> 
> I'm not keen on the huge asymmetry on the setup and tear down paths.
> For instance the lack of a scmi_unregister_protocol events.
> 

In v2 I removed most of the init/cleanup asymmetry reviwewing the initialization
cleanup logic and make more use of devres groups to limit the uglyness and
complexity of endless gotos on failpath.

> The idr usage is not standard.  They aren't the right structure
> if you known the value you want.
> 

Moved to hashtables as said.


> Not a particularly thorough review otherwise. Out of time for today and
> may not get back to this for a while.
> 
> Thanks,
> 
> Jonathan
> 

Regards

Cristian

>> ---
>>  drivers/firmware/arm_scmi/Makefile |   2 +-
>>  drivers/firmware/arm_scmi/common.h |   4 +
>>  drivers/firmware/arm_scmi/driver.c |   2 +
>>  drivers/firmware/arm_scmi/notify.c | 904 +++++++++++++++++++++++++++++
>>  drivers/firmware/arm_scmi/notify.h |  79 +++
>>  5 files changed, 990 insertions(+), 1 deletion(-)
>>  create mode 100644 drivers/firmware/arm_scmi/notify.c
>>  create mode 100644 drivers/firmware/arm_scmi/notify.h
>>
>> diff --git a/drivers/firmware/arm_scmi/Makefile b/drivers/firmware/arm_scmi/Makefile
>> index 5f298f00a82e..26587ea4661f 100644
>> --- a/drivers/firmware/arm_scmi/Makefile
>> +++ b/drivers/firmware/arm_scmi/Makefile
>> @@ -1,6 +1,6 @@
>>  # SPDX-License-Identifier: GPL-2.0-only
>>  obj-y	= scmi-bus.o scmi-driver.o scmi-protocols.o
>>  scmi-bus-y = bus.o
>> -scmi-driver-y = driver.o
>> +scmi-driver-y = driver.o notify.o
>>  scmi-protocols-y = base.o clock.o perf.o power.o reset.o sensors.o
>>  obj-$(CONFIG_ARM_SCMI_POWER_DOMAIN) += scmi_pm_domain.o
>> diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
>> index df35358ff324..2621c05e9149 100644
>> --- a/drivers/firmware/arm_scmi/common.h
>> +++ b/drivers/firmware/arm_scmi/common.h
>> @@ -6,6 +6,8 @@
>>   *
>>   * Copyright (C) 2018 ARM Ltd.
>>   */
>> +#ifndef _SCMI_COMMON_H
>> +#define _SCMI_COMMON_H
>>  
>>  #include <linux/bitfield.h>
>>  #include <linux/completion.h>
>> @@ -113,3 +115,5 @@ void scmi_setup_protocol_implemented(const struct scmi_handle *handle,
>>  				     u8 *prot_imp);
>>  
>>  int scmi_base_protocol_init(struct scmi_handle *h);
>> +
>> +#endif /* _SCMI_COMMON_H */
>> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
>> index 28ed1f0cb417..a43fad29de11 100644
>> --- a/drivers/firmware/arm_scmi/driver.c
>> +++ b/drivers/firmware/arm_scmi/driver.c
>> @@ -350,12 +350,14 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
>>  
>>  static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
>>  {
>> +	ktime_t ts;
>>  	struct scmi_xfer *xfer;
>>  	struct device *dev = cinfo->dev;
>>  	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
>>  	struct scmi_xfers_info *minfo = &info->rx_minfo;
>>  	struct scmi_shared_mem __iomem *mem = cinfo->payload;
>>  
>> +	ts = ktime_get_boottime();
>>  	xfer = scmi_xfer_get(cinfo->handle, minfo);
>>  	if (IS_ERR(xfer)) {
>>  		dev_err(dev, "failed to get free message slot (%ld)\n",
>> diff --git a/drivers/firmware/arm_scmi/notify.c b/drivers/firmware/arm_scmi/notify.c
>> new file mode 100644
>> index 000000000000..da342f43021e
>> --- /dev/null
>> +++ b/drivers/firmware/arm_scmi/notify.c
>> @@ -0,0 +1,904 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * System Control and Management Interface (SCMI) Notification support
>> + *
>> + * Copyright (C) 2019 ARM Ltd.
>> + *
>> + * SCMI Protocol specification allows the platform to signal events to
>> + * interested agents via notification messages: this in an implementation
>> + * of the dispatch and delivery of such notifications to the interested users
>> + * inside the Linux kernel.
>> + *
>> + * Each SCMI Protocol implementation, during its initialization, registers with
>> + * this core notification framework its set of supported events via
>> + * @scmi_register_protocol_events(), while Kernel users interested in some
>> + * specific event can register their associated callbacks providing the usual
>> + * notifier_block descriptor, since the notification system internally supports
>> + * events delivery using customized notification chains.
>> + *
>> + * Given the number of possible events defined by SCMI and the extensibility
>> + * of the SCMI Protocol itself, such underlying notification chains are created
>> + * and destroyed dynamically on demand depending on the number of users
>> + * effectively registered for an event, so that no structures or chains are
>> + * allocated until at least one user has registered a notifier_block for such
>> + * event. Similarly, events' generation itself is enabled at the platform level
>> + * only after at least one user has registered, and it is shutdown after the
>> + * last user for that event has gone.
>> + *
>> + * An event is identified univocally by the tuple (proto_id, evt_id, src_id)
>> + * and is served by its own dedicated notification chain: given that such chain
>> + * is dynamically created, the registration API simply let the users associate
>> + * their callbacks with the above tuple.
>> + *
>> + * Here proto_id and evt_id are simply the protocol_id and message_id numbers as
>> + * reported in the SCMI Protocol specification, while src_id represents an
>> + * optional, protocol dependent, source identifier (like domain_id, perf_id
>> + * or sensor_id and so forth), so that a user can register its callbacks for a
>> + * particular event coming only from a defined source (like CPU vs GPU).
>> + * When the source is not specified the user callback will be registered for
>> + * all existing sources for that event (if any).
>> + *
>> + * Upon reception of a notification message from the platform the SCMI RX ISR
>> + * passes the received message payload and some ancillary information (including
>> + * an arrival timestamp in nanoseconds) to the core via @scmi_notify(), which,
>> + * in turn, after having looked up the event in the registered events mapping,
>> + * pushes the event-data itself on a protocol dedicated kfifo queue for deferred
>> + * processing.
>> + *
>> + * Such dedicated protocols' queues are allocated once for all at initialization
>> + * time, together with a dedicated work_item running the common delivery logic
>> + * of @scmi_events_dispatcher(), so that each protocol has it own dedicated
>> + * worker which, once kicked by the ISR, takes care to empty is own dedicated
>> + * queue deliverying the queued items into the proper notification chain.
>> + * Note that since the underlying cmwq workers run one distinct work_item per
>> + * protocol and there are a number of max-active workers equal to the number of
>> + * protocols, notifications processing can proceed concurrently only between
>> + * events belonging to different protocols, while delivery of events within the
>> + * same protocol is still strictly sequentially ordered by time of arrival; this
>> + * separation effectively avoids the possibility that one particularly verbose
>> + * protocol, flooding the queues with events, can cause other protocols' events
>> + * to be lost or their processing starved.
>> + *
>> + * Event's information is converted to a custom per-event report struct before
>> + * being fed as void *data param to the user callback embedded in the registered
>> + * notifier_block so that it looks like as follow from the user perspective:
>> + *
>> + * int user_cb(struct notifier_block *nb, unsigned long event_id, void *report)
>> + *
> No point in having blank lines at the end of a comment block.
>> + *
>> + */
>> +
>> +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
>> +
>> +#include <linux/bug.h>
>> +#include <linux/bitfield.h>
>> +#include <linux/err.h>
>> +#include <linux/idr.h>
>> +#include <linux/kernel.h>
>> +#include <linux/kfifo.h>
>> +#include <linux/notifier.h>
>> +#include <linux/refcount.h>
>> +#include <linux/slab.h>
>> +#include <linux/scmi_protocol.h>
>> +#include <linux/types.h>
>> +#include <linux/workqueue.h>
>> +
>> +#include "notify.h"
>> +
>> +#define SCMI_NOTIF_MAX_ACTIVE_QUEUES	7
>> +
>> +/*
>> + * Builds an unsigned 32bit key from the given input tuple to be used as a key
>> + * in IDR mappings; note that if __src_p is passed in as NULL, the returned key
>> + * is built using SCMI_ALL_SRC_IDS as src_id.
>> + */
>> +#define MAKE_EVT_KEY(__proto, __evt, __src_p)				\
>> +	({								\
>> +		u32 __evt_key, __src_id;				\
>> +									\
>> +		__src_id = ((__src_p)) ?				\
>> +			    (*((u32 *)(__src_p)) & SCMI_ALL_SRC_IDS) :	\
>> +			    SCMI_ALL_SRC_IDS;				\
>> +		__evt_key = ((__proto) << 24) |				\
>> +				((__evt) << 16) | __src_id;		\
>> +		__evt_key;						\
>> +	})
>> +
>> +#define PROTO_ID_MASK			GENMASK(31, 24)
>> +#define EVT_ID_MASK			GENMASK(23, 16)
>> +#define SRC_ID_MASK			GENMASK(15, 0)
>> +#define KEY_XTRACT_PROTO_ID(key)	FIELD_GET(PROTO_ID_MASK, (key))
>> +#define KEY_XTRACT_EVT_ID(key)		FIELD_GET(EVT_ID_MASK, (key))
>> +#define KEY_XTRACT_SRC_ID(key)		FIELD_GET(SRC_ID_MASK, (key))
>> +
>> +/**
>> + * events_queue  - Describes a queue and its associated worker
> 
> Not valid kernel-doc.    I'll assume all other instances will get fixed
> and not bother talking about them.
> 
>> + *
>> + * Each protocol has its own dedicated events_queue descriptor.
>> + *
>> + * @sz: Size in bytes of the related kfifo
>> + * @kfifo: A dedicated Kernel kfifo
>> + * @notify_work: A custom work item bound to this queue
>> + * @wq: A reference to the related workqueue
>> + */
>> +struct events_queue {
>> +	size_t			sz;
>> +	struct kfifo		kfifo;
>> +	struct work_struct	notify_work;
>> +	struct workqueue_struct	*wq;
>> +};
>> +
>> +/**
>> + * scmi_registered_protocol_events_desc  - Protocol Specific information
>> + *
>> + * All protocols that registers at least an event have their protocol-specific
>> + * information stored here, together with a pointer to the allocated
>> + * events_queue.
>> + *
>> + * @id: Protocol ID
>> + * @ops: Protocol specific and event-related operations
>> + * @equeue: A reference to the associated per-protocol events_queue
>> + */
>> +struct scmi_registered_protocol_events_desc {
>> +	u8					id;
>> +	const struct scmi_protocol_event_ops	*ops;
>> +	struct events_queue			*equeue;
>> +};
>> +
>> +/**
>> + * scmi_registered_event  - Event Specific Information
>> + *
>> + * All registered events are represented by one of these structures.
>> + *
>> + * @proto: A reference to the associated protocol descriptor
>> + * @evt: A reference to the associated event descriptor (as provided at
>> + *       registration time)
>> + * @scratch_isr: A pre-allocated buffer to be used as a scratch area by ISR
>> + * @scratch_isr: A pre-allocated buffer to be used as a scratch area by the
>> + *		 deferred worker
> scratch_bh.
> 
>> + * @report: A pre-allocated buffer used by the deferred workr to fill a
> 
> worker
> 
>> + *	    customized event report
>> + */
>> +struct scmi_registered_event {
>> +	struct scmi_registered_protocol_events_desc	*proto;
>> +	const struct scmi_event				*evt;
>> +	void						*scratch_isr;
>> +	void						*scratch_bh;
>> +	void						*report;
>> +};
>> +
>> +/**
>> + * scmi_event_handler  - Event handler information
>> + *
>> + * This structure collects all the information needed to process a received
>> + * event, calling all the registered notifier callbacks.
>> + * Note this an event handler is associated to a tuple of values like:
>> + *	(proto_id, evt_id, src_id)
>> + * through the means of a key and an IDR mapping.
>> + *
>> + * @evt_key: The unsigned 32bit key associated to this descriptor in the
>> + *	     related IDR mapping
>> + * @r_evt: A reference to the underlying registered event
>> + * @chain: The notification chain dedicated to this specific event tuple
>> + */
>> +struct scmi_event_handler {
>> +	u32				evt_key;
>> +	bool				enabled;
>> +	refcount_t			users;
>> +	struct scmi_registered_event	*r_evt;
>> +	struct blocking_notifier_head	chain;
>> +};
>> +
>> +/**
>> + * scmi_event_header  - A utility header
> 
> That's not a good description.
> 
>> + *
>> + * This header is prepended to each received event message payload before
>> + * being queued on the related events_queue, and describes the attached event
>> + * message.
>> + *
>> + * @timestamp: The timestamp, innanoseconds (boottime), which was associated
>> + *	       to this event as soon as it entered the SCMI RX ISR
>> + * @proto_id: Protocol ID
>> + * @evt_id: Event ID (corresponds to the Event MsgID for this Protocol)
>> + * @payld_sz: Effective size of the attached message payload which follows
>> + * @payld: A reference to the included Event payload
>> + */
>> +struct scmi_event_header {
>> +	u64	timestamp;
>> +	u8	proto_id;
>> +	u8	evt_id;
>> +	size_t	payld_sz;
>> +	u8	payld[];
>> +} __packed;
>> +
>> +/*
>> + * A few IDR maps to track:
>> + *
>> + * - @scmi_registered_events: All event's descriptors registered by the
>> + *			      protocols, together with their ancillary data
>> + * - @scmi_registered_events_handlers: All event's handlers descriptors, created
>> + *				       to collect all the users' notifier_block
>> + *				       callbacks and related notification chains
>> + * - @scmi_registered_protocols: All protocol-level specific information related
>> + *				 to events' handling
>> + */
>> +static struct idr scmi_registered_events;
>> +static struct idr scmi_registered_events_handlers;
>> +static DEFINE_MUTEX(scmi_registered_events_handler_mtx);
>> +static struct idr scmi_registered_protocols;
>> +
>> +/* Common Kernel cmwq workqueue used by notifications core */
>> +static struct workqueue_struct *scmi_notify_wq;
>> +
>> +static bool scmi_notifications_initialized;
>> +
>> +static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key);
>> +static void scmi_put_event_handler(struct scmi_event_handler *hndl);
>> +
>> +/**
>> + * scmi_discard_bad_evt_payloadi() - Discard data from a kfifo
> 
> function name is wrong.
> 
>> + *
>> + * @kq: The kfifo to act on
>> + * @count: Number of bytes to flush
>> + */
>> +static inline void scmi_discard_bad_evt_payload(struct kfifo *kq,
>> +						const unsigned int count)
>> +{
>> +	int i = 0;
>> +
>> +	pr_warn("SCMI Notification WQ :: skipping bad EVT Payload - %d bytes\n",
>> +		count);
>> +	/* Discard stale pending queued payload. */
>> +	for (i = 0; i < count; i++)
>> +		kfifo_skip(kq);
>> +}
>> +
>> +/**
>> + * scmi_lookup_and_call_event_chain  - Lookup the proper chain and call it
>> + *
>> + * @evt_key: The event key to use to lookup the related notification chain
>> + * @report: The customized event-specific report to pass down to the callbacks
>> + *	    as their *data parameter.
>> + */
>> +static inline void scmi_lookup_and_call_event_chain(u32 evt_key, void *report)
>> +{
>> +	int ret;
>> +	struct scmi_event_handler *hndl;
>> +
>> +	hndl = scmi_get_event_handler(evt_key);
>> +	if (IS_ERR_OR_NULL(hndl))
>> +		return;
>> +
>> +	ret = blocking_notifier_call_chain(&hndl->chain,
>> +					   KEY_XTRACT_EVT_ID(evt_key),
>> +					   report);
>> +	/* Notifiers are NOT supposed to cut the chain */
>> +	WARN_ON_ONCE(ret & NOTIFY_STOP_MASK);
>> +
>> +	scmi_put_event_handler(hndl);
>> +}
>> +
>> +/**
>> + * scmi_events_dispatcher  - Common worker logic for all work items.
>> + *
>> + * In turn:
>> + *  1. dequeue one pending RX notification (queued in SCMI RX ISR context)
>> + *  2. generate a custom event report from the received event message
>> + *  3. lookup for any registered ALL_SRC_IDs handler
>> + *     - > call the related notification chain passing in the report
>> + *  4. lookup for any registered specific SRC_ID handler
>> + *     - > call the related notification chain passing in the report
>> + *
>> + * Note that:
>> + * - a dedicated per-protocol kfifo queue is used: in this way an anomalous
>> + *   flood of events cannot saturate other protocols' queues.
>> + *
>> + * - each per-protocol queue is associated to a distinct work_item, which
>> + *   means, in turn, that:
>> + *   + all protocol can process their dedicated queues concurrently
>> + *     (since scmi_notify_wq:max_active > 1)
>> + *   + anyway at most one worker instance is allowed to run on the same queue
>> + *     concurrently: this ensures that we can have only one concurrent
>> + *     reader/writer on the associated kfifo (needed for lock-less access)
>> + *
>> + * @work: The work item to use, which is associated to the proper events_queue
>> + */
>> +static void scmi_events_dispatcher(struct work_struct *work)
>> +{
>> +	struct events_queue *equeue;
>> +	struct scmi_event_header eh;
>> +
>> +	equeue = container_of(work, struct events_queue, notify_work);
>> +	while (kfifo_out(&equeue->kfifo, &eh, sizeof(eh))) {
>> +		u32 src_id, evt_key;
>> +		unsigned int outs;
>> +		struct scmi_registered_event *r_evt;
>> +		void *report = NULL;
>> +
>> +		evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, NULL);
>> +		r_evt = idr_find(&scmi_registered_events, evt_key);
>> +		if (!r_evt) {
>> +			scmi_discard_bad_evt_payload(&equeue->kfifo,
>> +						     eh.payld_sz);
>> +			continue;
>> +		}
>> +
>> +		outs = kfifo_out(&equeue->kfifo, r_evt->scratch_bh,
>> +				 eh.payld_sz);
>> +		if (outs != eh.payld_sz) {
>> +			pr_warn("SCMI Notification WQ :: SKIP corrupted EVT Payload.\n");
>> +			continue;
>> +		}
>> +
>> +		/* Reset and fill custom report */
>> +		memset(r_evt->report, 0x00, r_evt->evt->max_report_sz);
>> +		report = r_evt->proto->ops->fill_custom_report(eh.evt_id,
>> +							eh.timestamp,
>> +							r_evt->scratch_bh,
>> +							eh.payld_sz,
>> +							r_evt->report,
>> +							&src_id);
>> +		if (!report)
>> +			continue;
>> +
>> +		/* At first search for a generic ALL src_ids handler... */
>> +		scmi_lookup_and_call_event_chain(evt_key, report);
>> +		if (src_id != SCMI_ALL_SRC_IDS) {
>> +			u32 *__sub_p = &src_id;
>> +
>> +			evt_key = MAKE_EVT_KEY(eh.proto_id, eh.evt_id, __sub_p);
>> +			scmi_lookup_and_call_event_chain(evt_key, report);
>> +		}
>> +	}
>> +}
>> +
>> +/**
>> + * scmi_notify  - Queues a notification for further deferred processing
>> + *
>> + * This is called in interrupt context to queue a received event for
>> + * deferred processing.
>> + *
>> + * @proto_id: Protocol ID
>> + * @evt_id: Event ID (msgID)
>> + * @buf: Event Message Payload (without the header)
>> + * @len: Event Message Payload size
>> + * @ts: RX Timestamp in nanoseconds (boottime)
>> + *
>> + * Return: 0 on Success
>> + */
>> +int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts)
>> +{
>> +	struct scmi_registered_event *r_evt;
>> +	struct scmi_event_header *eh;
>> +
>> +	if (unlikely(!scmi_notifications_initialized))
>> +		return 0;
>> +
>> +	r_evt = idr_find(&scmi_registered_events,
>> +			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
>> +	if (unlikely(!r_evt || !r_evt->proto->equeue))
>> +		return -EINVAL;
>> +
>> +	if (unlikely(len > r_evt->evt->max_payld_sz)) {
>> +		pr_err("SCMI Notification discard badly sized message\n");
>> +		return -EINVAL;
>> +	}
>> +	if (unlikely(kfifo_avail(&r_evt->proto->equeue->kfifo) <
>> +		     sizeof(*eh) + len)) {
>> +		pr_warn("SCMI Notification queue full: dropping proto_id:%d  evt_id:%d  ts:%lld\n",
>> +			proto_id, evt_id, ts);
>> +		return -ENOMEM;
>> +	}
>> +
>> +	/*
>> +	 * Note that we can use the per protocol kfifo in a lock-less manner
>> +	 * since we have only one concurrent reader and writer but, in order
>> +	 * to avoid any trouble on the reader side, here we perform one single
>> +	 * write, so we have to collate event header and payload in a scratch
>> +	 * area at first.
>> +	 */
>> +	eh = r_evt->scratch_isr;
>> +	eh->timestamp = ts;
>> +	eh->proto_id = proto_id;
>> +	eh->evt_id = evt_id;
>> +	eh->payld_sz = len;
>> +	memcpy(eh->payld, buf, eh->payld_sz);
>> +	kfifo_in(&r_evt->proto->equeue->kfifo, eh, sizeof(*eh) + eh->payld_sz);
>> +	queue_work(r_evt->proto->equeue->wq,
>> +		   &r_evt->proto->equeue->notify_work);
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_allocate_events_queue  - Allocate an events_queue descriptor
>> + *
>> + * @sz: Size of the kfifo to initialize
>> + *
>> + * Return: A valid pointer to the allocated events_queue on Success
>> + */
>> +static struct events_queue *scmi_allocate_events_queue(size_t sz)
>> +{
>> +	int ret;
>> +	struct events_queue *equeue;
>> +
>> +	equeue = kzalloc(sizeof(*equeue), GFP_KERNEL);
>> +	if (!equeue)
>> +		return ERR_PTR(-ENOMEM);
>> +	ret = kfifo_alloc(&equeue->kfifo, sz, GFP_KERNEL);
>> +	if (ret) {
>> +		kfree(equeue);
>> +		return ERR_PTR(ret);
>> +	}
>> +	equeue->sz = sz;
>> +	INIT_WORK(&equeue->notify_work, scmi_events_dispatcher);
>> +	equeue->wq = scmi_notify_wq;
>> +
>> +	return equeue;
>> +}
>> +
>> +/**
>> + * scmi_deallocate_events_queue  - Deallocate am events_queue descriptor
>> + *
>> + * @equeue: The events_queue to free
>> + */
>> +static void scmi_deallocate_events_queue(struct events_queue *equeue)
>> +{
>> +	kfifo_free(&equeue->kfifo);
>> +	kfree(equeue);
>> +}
>> +
>> +/**
>> + * scmi_allocate_registered_protocol_desc  - Allocate a registered protocol
>> + * events' descriptor
>> + *
>> + * Used to keep protocol specific information related to events handling for any
>> + * protocl which has registered at least one event.
> 
> protocol
> 
>> + *
>> + * @proto_id: Protocol ID
>> + * @queue_sz: Size of the associated queue to allocate
>> + * @ops: Pointer to a struct holding references to protocol specific helpers
>> + *	 needed during events handling
>> + */
>> +static struct scmi_registered_protocol_events_desc *
>> +scmi_allocate_registered_protocol_desc(u8 proto_id, size_t queue_sz,
>> +				const struct scmi_protocol_event_ops *ops)
>> +{
>> +	int ret;
>> +	struct scmi_registered_protocol_events_desc *pdesc;
>> +
>> +	pdesc = idr_find(&scmi_registered_protocols, proto_id);
> 
> Perhaps add a comment here on why this might get called multiple times
> but is still safe from point of view of release path.
> 
>> +	if (pdesc)
>> +		return pdesc;
> 
> 
>> +
>> +	pdesc = kzalloc(sizeof(*pdesc), GFP_KERNEL);
>> +	if (!pdesc)
>> +		return ERR_PTR(-ENOMEM);
>> +	pdesc->id = proto_id;
>> +	pdesc->ops = ops;
>> +
>> +	pdesc->equeue = scmi_allocate_events_queue(queue_sz);
>> +	if (IS_ERR(pdesc->equeue)) {
>> +		kfree(pdesc);
>> +		return ERR_CAST(pdesc->equeue);
>> +	}
>> +
>> +	ret = idr_alloc(&scmi_registered_protocols, pdesc,
>> +			pdesc->id, pdesc->id + 1, GFP_KERNEL);
>> +	if (ret < 0) {
>> +		pr_err("SCMI Failed to allocate PDESC IDR - key:%d - err:%d\n",
>> +		       pdesc->id, ret);
>> +		scmi_deallocate_events_queue(pdesc->equeue);
>> +		kfree(pdesc);
>> +		return ERR_PTR(ret);
>> +	}
>> +
>> +	return pdesc;
>> +}
>> +
>> +/**
>> + * scmi_register_protocol_events  - Register Protocol Events with the core
>> + *
>> + * Used by SCMI Protocols initialization code to register with the notification
>> + * core the list of supported events and their description: this takes care to
>> + * pre-allocate all needed scratch buffers and link the proper event queue to
>> + * this event.
>> + *
>> + * @proto_id: Protocol ID
>> + * @queue_sz: Size in bytes of the associated queue to be allocated
>> + * @ops: Protocol specific event-related operations
>> + * @evt: Event descriptor array
>> + * @num_events: Number of events in @evt array
>> + *
>> + * Return: 0 on Success
>> + */
>> +int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
>> +				  const struct scmi_protocol_event_ops *ops,
>> +				  const struct scmi_event *evt, int num_events)
>> +{
>> +	int i;
>> +	struct scmi_registered_protocol_events_desc *pdesc;
>> +
>> +	if (!scmi_notifications_initialized)
>> +		return -EAGAIN;
>> +
>> +	if (!ops || !evt)
>> +		return -EINVAL;
>> +
>> +	pdesc = scmi_allocate_registered_protocol_desc(proto_id, queue_sz, ops);
>> +	if (IS_ERR(pdesc))
>> +		return -ENOMEM;
>> +
>> +	for (i = 0; i < num_events; i++, evt++) {
>> +		int ret;
>> +		u32 key;
>> +		struct scmi_registered_event *r_evt;
>> +
>> +		r_evt = kzalloc(sizeof(*r_evt), GFP_KERNEL);
>> +		if (!r_evt)
>> +			continue;
>> +		r_evt->proto = pdesc;
>> +		r_evt->evt = evt;
>> +
>> +		r_evt->scratch_isr = kzalloc(sizeof(struct scmi_event_header) +
>> +					     evt->max_payld_sz, GFP_KERNEL);
>> +		if (!r_evt->scratch_isr)
>> +			goto continue_isr_fail;
>> +
>> +		r_evt->scratch_bh = kzalloc(evt->max_payld_sz, GFP_KERNEL);
>> +		if (!r_evt->scratch_bh)
>> +			goto continue_bh_fail;
>> +
>> +		r_evt->report = kzalloc(evt->max_report_sz, GFP_KERNEL);
>> +		if (!r_evt->report)
>> +			goto continue_report_fail;
>> +
>> +		key = MAKE_EVT_KEY(r_evt->proto->id, evt->evt_id, NULL);
>> +		ret = idr_alloc(&scmi_registered_events, r_evt,
>> +				key, key + 1, GFP_KERNEL);
>> +		if (ret < 0) {
>> +			pr_err("SCMI Failed to allocate EVENT IDR - key:%X - err:%d\n",
>> +			       key, ret);
>> +			goto continue_idr_fail;
>> +		}
>> +
>> +		pr_info("SCMI Notification registered event - key:%X\n", key);
>> +		continue;
>> +
>> +	/* yes, this is not nice ... */
>> +continue_idr_fail:
>> +		kfree(r_evt->report);
>> +continue_report_fail:
>> +		kfree(r_evt->scratch_bh);
>> +continue_bh_fail:
>> +		kfree(r_evt->scratch_isr);
>> +continue_isr_fail:
>> +		kfree(r_evt);
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_register_event_handler  - Allocate an Event handler
>> + *
>> + * Allocate an event handler and related notification chain associated with
>> + * the event identified by the provided event key. Fails if the associated
>> + * event is unknown to the core (i.e. it had not been successfully registered
>> + * upfront by some protocol)
>> + *
>> + * @evt_key: 32bit key uniquely bind to the event identified by the tuple
>> + *	     (proto_id, evt_id, src_id)
>> + *
>> + * Return: the freshly allocated structure on Success
>> + */
>> +static struct scmi_event_handler *scmi_register_event_handler(u32 evt_key)
>> +{
>> +	int id;
>> +	u8 proto_id, evt_id;
>> +	struct scmi_registered_event *r_evt;
>> +	struct scmi_event_handler *hndl;
>> +
>> +	proto_id = KEY_XTRACT_PROTO_ID(evt_key);
>> +	evt_id = KEY_XTRACT_EVT_ID(evt_key);
>> +	r_evt = idr_find(&scmi_registered_events,
>> +			 MAKE_EVT_KEY(proto_id, evt_id, NULL));
>> +	if (!r_evt)
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	hndl = kzalloc(sizeof(*hndl), GFP_KERNEL);
>> +	if (!hndl)
>> +		return ERR_PTR(-ENOMEM);
>> +	hndl->r_evt = r_evt;
>> +	hndl->evt_key = evt_key;
>> +	BLOCKING_INIT_NOTIFIER_HEAD(&hndl->chain);
>> +	refcount_set(&hndl->users, 1);
>> +
>> +	/* Register freshly allocated event handler */
>> +	id = idr_alloc(&scmi_registered_events_handlers, hndl,
>> +		       evt_key, evt_key + 1, GFP_KERNEL);
> 
> Umm. That's not using an idr for what it is meant for. You aren't allocating an id, you
> are using it as a poor wrapper around an xarray.
> 
>> +	if (id < 0) {
>> +		pr_err("SCMI Failed to allocate HNDL IDR - key:%X  err:%d\n",
>> +		       evt_key, id);
>> +		kfree(hndl);
>> +		return ERR_PTR(id);
>> +	}
>> +
>> +	return hndl;
>> +}
>> +
>> +/**
>> + * scmi_unregister_event_handler  - Free the provided Event handler
>> + *
>> + * @hndl: The event handler structure to free
>> + */
>> +static void scmi_unregister_event_handler(struct scmi_event_handler *hndl)
>> +{
>> +	idr_remove(&scmi_registered_events_handlers, hndl->evt_key);
>> +	kfree(hndl);
>> +}
>> +
>> +/**
>> + * __scmi_event_handler_get_ops  - Get or create an event handler
>> + *
>> + * After having got exclusive access to the registered events map, searches for
>> + * the desired handler matching the key: when found adjust refcount, when not
>> + * create and register a new handler is asked to do so.
>> + * Events generation is NOT enabled within this routine since at creation time
>> + * we usually want to have all setup and registered before events starts
>> + * flowing.
>> + *
>> + * @evt_key: The event key to use
>> + * @create: A boolean flag to specify if a handler must be created
>> + *	    when not found
>> + */
>> +static inline struct scmi_event_handler *
>> +__scmi_event_handler_get_ops(u32 evt_key, bool create)
>> +{
>> +	struct scmi_event_handler *hndl = NULL;
>> +
>> +	mutex_lock(&scmi_registered_events_handler_mtx);
>> +	hndl = idr_find(&scmi_registered_events_handlers, evt_key);
>> +	if (hndl)
>> +		refcount_inc(&hndl->users);
>> +	else if (create)
>> +		hndl = scmi_register_event_handler(evt_key);
>> +	mutex_unlock(&scmi_registered_events_handler_mtx);
>> +
>> +	return hndl;
>> +}
>> +
>> +static struct scmi_event_handler *scmi_get_event_handler(u32 evt_key)
>> +{
>> +	return __scmi_event_handler_get_ops(evt_key, false);
>> +}
>> +
>> +static struct scmi_event_handler *scmi_get_or_create_event_handler(u32 evt_key)
>> +{
>> +	return __scmi_event_handler_get_ops(evt_key, true);
>> +}
>> +
>> +/**
>> + * __scmi_enable_events_ops  - Enable/disable events generation
>> + *
>> + * @hndl: The handler specifying the events enable/disable
>> + * @action: The action to perform: true->Enable false->Disable
>> + *
>> + * Return: True when the required @action has been successfully executed
>> + */
>> +static inline bool __scmi_enable_events_ops(struct scmi_event_handler *hndl,
>> +					    bool action)
>> +{
>> +	bool ret = true;
>> +
>> +	if (hndl->enabled != action) {
>> +		u8 evt_id;
>> +		u32 src_id;
>> +
>> +		evt_id = KEY_XTRACT_EVT_ID(hndl->evt_key);
>> +		src_id = KEY_XTRACT_SRC_ID(hndl->evt_key);
>> +		ret = hndl->r_evt->proto->ops->set_notify_enabled(evt_id,
>> +			      src_id != SCMI_ALL_SRC_IDS ? &src_id : NULL,
>> +			      action);
>> +		if (ret)
>> +			hndl->enabled = action;
>> +	}
>> +
>> +	return ret;
>> +}
>> +
>> +static bool scmi_enable_events(struct scmi_event_handler *hndl)
>> +{
>> +	return __scmi_enable_events_ops(hndl, true);
>> +}
>> +
>> +static bool scmi_disable_events(struct scmi_event_handler *hndl)
>> +{
>> +	return __scmi_enable_events_ops(hndl, false);
>> +}
>> +
>> +/**
>> + * scmi_put_event_handler  - Put an event handler
>> + *
>> + * After having got exclusive access to the registered events map, update
>> + * the refcount and if @hndl is no more in use by anyone:
>> + *  - disable the events generation
>> + *  - unregister and free the handler itself
>> + *
>> + * @hndl: The handler to act upon
>> + */
>> +static void scmi_put_event_handler(struct scmi_event_handler *hndl)
>> +{
>> +	mutex_lock(&scmi_registered_events_handler_mtx);
>> +	if (refcount_dec_and_test(&hndl->users)) {
>> +		scmi_disable_events(hndl);
>> +		scmi_unregister_event_handler(hndl);
>> +	}
>> +	mutex_unlock(&scmi_registered_events_handler_mtx);
>> +}
>> +
>> +/**
>> + * scmi_register_event_notifier  - Register a notifier_block for an event
>> + *
>> + * Generic helper to register a notifier_block attached to a protocol event.
>> + *
>> + * A notifier_block @nb will be registered for each distinct event identified
>> + * by the tuple (proto_id, evt_id, src_id) on a dedicated notification chain
>> + * so that:
>> + *
>> + *	(proto_X, evt_Y, src_Z) --> chain_X_Y_Z
>> + *
>> + * @src_id meaning is protocol specific and identifies the origin of the event
>> + * (like domain_id, sensor_id and os forth); @src_id can be NULL to signify that
> 
> so
> 
>> + * the caller is interested in receiving notifications from ALL the available
>> + * sources for that protocol OR simply that the protocol does not support
>> + * distinct sources: in these cases @nb will be attached to a generic
>> + * notification chain defined for ALL src_id of that proto_id/evt_id pair like:
>> + *
>> + *	(proto_X, evt_Y, NULL) --> chain_X_Y_ALL
>> + *
>> + * Any received event will be then dispatched to both such chains if at least
>> + * one user had registered an @nb on them.
> 
> both should be all I think.
> 
>> + *
>> + * Note also that notification chains are created dynamically: a new chain is
>> + * created each time a new distinct tuple is detected and then the provided @nb
>> + * is bound to such chain; at creation time the related SCMI notifications are
>> + * also enabled: this way, only when at least one user has registered its
>> + * interest for a specific event tuple, the underlying notification chain is
>> + * created and the related notifications are enabled in the platform.
> 
> That's one of the most complex bits of punctuation I've seenin a comment for
> a long time.   Please break that sentence up for readability.
> 
>> + *
>> + * @proto_id: Protocol ID
>> + * @evt_id: Event ID
>> + * @src_id: Source ID
>> + * @nb: A standard notifier block to register for the specified event
>> + *
>> + * Return: Return 0 on Success
>> + */
>> +int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
>> +				 struct notifier_block *nb)
>> +{
>> +	u32 evt_key;
>> +	struct scmi_event_handler *hndl;
>> +
>> +	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
>> +	hndl = scmi_get_or_create_event_handler(evt_key);
>> +	if (IS_ERR_OR_NULL(hndl))
>> +		return PTR_ERR(hndl);
>> +
>> +	blocking_notifier_chain_register(&hndl->chain, nb);
>> +
>> +	if (!scmi_enable_events(hndl)) {
>> +		pr_err("SCMI Failed to ENABLE events for key:%X !\n", evt_key);
>> +		blocking_notifier_chain_unregister(&hndl->chain, nb);
>> +		scmi_put_event_handler(hndl);
>> +		return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_unregister_event_notifier  - Unregister a notifier_block for an event
>> + *
>> + * Takes care to unregister the provided @nb from the notification chain
>> + * associated to the specified event and, if there are no more users for the
>> + * event handler, frees also the associated event handler structures.
>> + *
>> + * @proto_id: Protocol ID
>> + * @evt_id: Event ID
>> + * @src_id: Source ID
>> + * @nb: The notifier_block to unregister
>> + *
>> + * Return: 0 on Success
>> + */
>> +int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
>> +				   struct notifier_block *nb)
>> +{
>> +	u32 evt_key;
>> +	struct scmi_event_handler *hndl;
>> +
>> +	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
>> +	hndl = scmi_get_event_handler(evt_key);
>> +	if (IS_ERR_OR_NULL(hndl))
>> +		return -EINVAL;
>> +
>> +	blocking_notifier_chain_unregister(&hndl->chain, nb);
>> +
>> +	scmi_put_event_handler(hndl);
>> +	/*
>> +	 * If this was the last user callback for this handler, this last put
>> +	 * will force the handler to be freed.
>> +	 * Note that if a call_chain walk is ongoing it will be the call_chain
>> +	 * put request which will finally free the handler; note also that any
>> +	 * operation on the inner notifier_block chain is protected on its own.
>> +	 */
>> +	scmi_put_event_handler(hndl);
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_notification_init  - Initializes Notification Core Support
>> + *
>> + * Return: 0 on Success
>> + */
>> +int __init scmi_notification_init(void)
>> +{
>> +	scmi_notify_wq = alloc_workqueue("scmi_notify",
>> +					 WQ_UNBOUND | WQ_FREEZABLE,
>> +					 SCMI_NOTIF_MAX_ACTIVE_QUEUES);
>> +	if (!scmi_notify_wq)
>> +		return -ENOMEM;
>> +	idr_init(&scmi_registered_events);
>> +	idr_init(&scmi_registered_events_handlers);
>> +	idr_init(&scmi_registered_protocols);
>> +
>> +	scmi_notifications_initialized = true;
>> +
>> +	pr_info("SCMI Notifications enabled.\n");
>> +
>> +	return 0;
>> +}
>> +
>> +/**
>> + * scmi_notification_exit  - Shutdown and clean Notification core
>> + */
>> +void __exit scmi_notification_exit(void)
>> +{
>> +	int k;
>> +	struct scmi_event_handler *hndl;
>> +	struct scmi_registered_protocol_events_desc *pdesc;
>> +	struct scmi_registered_event *r_evt;
>> +
>> +	scmi_notifications_initialized = false;
>> +
>> +	/* Disable notifications' generation */
>> +	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
>> +		scmi_disable_events(hndl);
>> +
>> +	/* Let pending work complete */
>> +	destroy_workqueue(scmi_notify_wq);
>> +
>> +	/* Remove ALL events handlers */
>> +	idr_for_each_entry(&scmi_registered_events_handlers, hndl, k)
>> +		scmi_put_event_handler(hndl);
>> +	idr_destroy(&scmi_registered_events_handlers);
>> +
>> +	/* Finally remove all registered events */
> 
> Finally seems a bit premature here ;)
> 
>> +	idr_for_each_entry(&scmi_registered_events, r_evt, k) {
>> +		idr_remove(&scmi_registered_events, k);
>> +		kfree(r_evt->report);
>> +		kfree(r_evt->scratch_isr);
>> +		kfree(r_evt->scratch_bh);
>> +		kfree(r_evt);
>> +	}
>> +	idr_destroy(&scmi_registered_events);
>> +
>> +	/* free any remaining protocol data */
>> +	idr_for_each_entry(&scmi_registered_protocols, pdesc, k) {
>> +		idr_remove(&scmi_registered_protocols, k);
>> +		scmi_deallocate_events_queue(pdesc->equeue);
>> +		kfree(pdesc);
>> +	}
>> +	idr_destroy(&scmi_registered_protocols);
>> +}
>> diff --git a/drivers/firmware/arm_scmi/notify.h b/drivers/firmware/arm_scmi/notify.h
>> new file mode 100644
>> index 000000000000..6fa07b244b14
>> --- /dev/null
>> +++ b/drivers/firmware/arm_scmi/notify.h
>> @@ -0,0 +1,79 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * System Control and Management Interface (SCMI) Message Protocol
>> + * notification header file containing some definitions, structures
>> + * and function prototypes related to SCMI Notification handling.
>> + *
>> + * Copyright (C) 2019 ARM Ltd.
>> + */
>> +#ifndef _SCMI_NOTIFY_H
>> +#define _SCMI_NOTIFY_H
>> +
>> +#include <linux/bug.h>
>> +#include <linux/notifier.h>
>> +#include <linux/types.h>
>> +
>> +#define	SCMI_ALL_SRC_IDS		0x0000ffffL
>> +
>> +#define MAP_EVT_TO_ENABLE_CMD(id, map)			\
>> +({							\
>> +	int ret = -1;					\
>> +							\
>> +	if (likely((id) < ARRAY_SIZE((map))))		\
>> +		ret = (map)[(id)];			\
>> +	else						\
>> +		WARN(1, "UN-KNOWN evt_id:%d\n", (id));	\
>> +	ret;						\
>> +})
>> +
>> +/**
>> + * scmi_event  - Describes an event to be supported
> 
> I don't think this is valid kernel-doc.  Make sure to point the scripts at the
> file and check and fix any warnings.
> 
>> + *
>> + * Each SCMI protocol, during its initialization phase, can describe the events
>> + * it wishes to support in a few struct scmi_event and pass them to the core
>> + * using scmi_register_protocol_events().
>> + *
>> + * @evt_id: Event ID
>> + * @max_payld_sz: Max possible size for the payload of a notif msg of this kind
>> + * @max_report_sz: Max possible size for the report of a notif msg of this kind
>> + */
>> +struct scmi_event {
>> +	u8	evt_id;
>> +	size_t	max_payld_sz;
>> +	size_t	max_report_sz;
>> +
>> +};
>> +
>> +/**
>> + * scmi_protocol_event_ops  - Helpers called by notification core.
>> + *
>> + * These are called only in process context.
>> + *
>> + * @set_notify_enabled: Enable/disable the required evt_id/src_id notifications
>> + *			using the proper custom protocol commands.
>> + *			Return true if at least one the required src_id
>> + *			has been successfully enabled/disabled
>> + * @fill_custom_report: fills a custom event report from the provided
> 
> Nitpick.  Inconsistent capitalization after the :
> 
>> + *			event message payld identifying the event
>> + *			specific src_id.
>> + *			Return NULL on failure otherwise @report now fully
>> + *			populated
>> + */
>> +struct scmi_protocol_event_ops {
>> +	bool (*set_notify_enabled)(u8 evt_id, const u32 *src_id, bool enabled);
>> +	void *(*fill_custom_report)(u8 evt_id, u64 timestamp, const void *payld,
>> +				    size_t payld_sz, void *report, u32 *src_id);
>> +};
>> +
>> +int scmi_notification_init(void);
>> +void scmi_notification_exit(void);
>> +int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
>> +				  const struct scmi_protocol_event_ops *ops,
>> +				  const struct scmi_event *evt, int num_events);
>> +int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
>> +				 struct notifier_block *nb);
>> +int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *sub_id,
>> +				   struct notifier_block *nb);
>> +int scmi_notify(u8 proto_id, u8 evt_id, const void *buf, size_t len, u64 ts);
>> +
>> +#endif /* _SCMI_NOTIFY_H */
> 
> 

[RFC PATCH 05/11] firmware: arm_scmi: Add notifications anti-tampering

[Cristian Marussi]

Add a simple mechanism to avoid the fact that with standard Kernel
notification chains a notifier callback, registered by a user, can
potentially stop further processing for the notification chain itself,
or mangle the *data content callback parameter along the way.

A simple notifier_block wrapper layer is introduced in order to limit the
above described tampering capabilities.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
This patch can be simply dropped from the series if the anti-tampering
mechanism itself is not deemed worth the effort
---
 drivers/firmware/arm_scmi/notify.c | 151 ++++++++++++++++++++++++++++-
 1 file changed, 147 insertions(+), 4 deletions(-)

diff --git a/drivers/firmware/arm_scmi/notify.c b/drivers/firmware/arm_scmi/notify.c
index da342f43021e..ab5033c1b03c 100644
--- a/drivers/firmware/arm_scmi/notify.c
+++ b/drivers/firmware/arm_scmi/notify.c
@@ -111,6 +111,9 @@
 #define KEY_XTRACT_EVT_ID(key)		FIELD_GET(EVT_ID_MASK, (key))
 #define KEY_XTRACT_SRC_ID(key)		FIELD_GET(SRC_ID_MASK, (key))
 
+#define	PTR_2_EVT_KEY(p)		\
+	((u32)((unsigned long)(p) & 0x00000000ffffffffL))
+
 /**
  * events_queue  - Describes a queue and its associated worker
  *
@@ -180,6 +183,7 @@ struct scmi_registered_event {
  *	     related IDR mapping
  * @r_evt: A reference to the underlying registered event
  * @chain: The notification chain dedicated to this specific event tuple
+ * @scmi_registered_nbs: An IDR map to the currently allocated wrapper nbs.
  */
 struct scmi_event_handler {
 	u32				evt_key;
@@ -187,6 +191,7 @@ struct scmi_event_handler {
 	refcount_t			users;
 	struct scmi_registered_event	*r_evt;
 	struct blocking_notifier_head	chain;
+	struct idr			scmi_registered_nbs;
 };
 
 /**
@@ -211,6 +216,26 @@ struct scmi_event_header {
 	u8	payld[];
 } __packed;
 
+/**
+ * scmi_wrapper_block  - A wrapper notifier_block
+ *
+ * Used to wrap the original notifier_block provided by the user to limit user
+ * capabilities to cut short the notification chain or mangle the *data
+ * parameters.
+ *
+ * @original_nb: The original notifier_block provided by the user
+ * @wnb: A wrapper notifier_block to be effectively registered with the chains
+ * @report_copy: A pointer to the pre-allocated report buffer effectively passed
+ *		 down to the user notifier_block as *data param
+ * @report_sz: The size of the @report_copy buffer
+ */
+struct scmi_wrapper_block {
+	struct notifier_block	*original_nb;
+	struct notifier_block	wnb;
+	void			*report_copy;
+	size_t			report_sz;
+};
+
 /*
  * A few IDR maps to track:
  *
@@ -579,6 +604,104 @@ int scmi_register_protocol_events(u8 proto_id, size_t queue_sz,
 	return 0;
 }
 
+/**
+ * scmi_wrapper_notifier_call  - A common wrapper notifier_call function
+ *
+ * This wrapper callback is what is effectively registered on the notification
+ * chains; it ensures the user provided callback which is in turn called from
+ * this cannot tamper with the notification deliveries, like returning a
+ * NOTIFY_STOP to cut the chain or mangling with the event reports passed down
+ * *data.
+ *
+ * @nb: The associated notifier block
+ * @event: The event ID
+ * @data: A custom event report
+ *
+ * Return: Alwways NOTIFY_OK
+ */
+static int scmi_wrapper_notifier_call(struct notifier_block *nb,
+				      unsigned long event, void *data)
+{
+	int ret;
+	struct scmi_wrapper_block *cwnb;
+	void *report = data;
+
+	cwnb = container_of(nb, struct scmi_wrapper_block, wnb);
+
+	if (!cwnb)
+		return NOTIFY_STOP;
+
+	if (cwnb->report_copy) {
+		memcpy(cwnb->report_copy, data, cwnb->report_sz);
+		report = cwnb->report_copy;
+	}
+	ret = cwnb->original_nb->notifier_call(cwnb->original_nb,
+					       event, report);
+
+	return NOTIFY_OK;
+}
+
+/**
+ * scmi_register_wrapper_blocking_notifier  - Register a notifier wrapper
+ *
+ * In order to avoid a user registering a notifier_block to tamper with the
+ * notification delivery process, this function builds a notifier_block wrapper
+ * and embeds into it the user provided one @nb.
+ *
+ * @hndl: The event handler to act upon
+ * @nb: the original user provided notfier_block to wrap over
+ *
+ * Return: The allocated and registered scmi_wrapper_block on Success
+ */
+static struct scmi_wrapper_block *
+scmi_register_wrapper_blocking_notifier(struct scmi_event_handler *hndl,
+					struct notifier_block *nb)
+{
+	int id;
+	struct scmi_wrapper_block *cwnb;
+
+	cwnb = kzalloc(sizeof(*cwnb), GFP_KERNEL);
+	if (!cwnb)
+		return ERR_PTR(-ENOMEM);
+
+	cwnb->report_sz = hndl->r_evt->evt->max_report_sz;
+	cwnb->report_copy = kzalloc(cwnb->report_sz, GFP_KERNEL);
+	if (!cwnb->report_copy)
+		pr_warn("SCMI Failed to allocate per-notifier report-buffer\n");
+	cwnb->original_nb = nb;
+	cwnb->wnb.notifier_call = scmi_wrapper_notifier_call;
+	cwnb->wnb.priority = nb->priority;
+
+	id = idr_alloc(&hndl->scmi_registered_nbs, cwnb, PTR_2_EVT_KEY(nb),
+		       PTR_2_EVT_KEY(nb) + 1, GFP_KERNEL);
+	if (id < 0) {
+		pr_err("SCMI Failed to allocate NB IDR - err:%d\n", id);
+		kfree(cwnb);
+		return ERR_PTR(id);
+	}
+
+	blocking_notifier_chain_register(&hndl->chain, &cwnb->wnb);
+
+	return cwnb;
+}
+
+/**
+ * scmi_unregister_wrapper_blocking_notifier  - Unregister a notifier wrapper
+ *
+ * @hndl: The event handler to act upon
+ * @cwnb: The wrapper notifier to be unregistered
+ */
+static void
+scmi_unregister_wrapper_blocking_notifier(struct scmi_event_handler *hndl,
+					  struct scmi_wrapper_block *cwnb)
+{
+	idr_remove(&hndl->scmi_registered_nbs,
+		   PTR_2_EVT_KEY(cwnb->original_nb));
+	blocking_notifier_chain_unregister(&hndl->chain, &cwnb->wnb);
+	kfree(cwnb->report_copy);
+	kfree(cwnb);
+}
+
 /**
  * scmi_register_event_handler  - Allocate an Event handler
  *
@@ -624,6 +747,8 @@ static struct scmi_event_handler *scmi_register_event_handler(u32 evt_key)
 		return ERR_PTR(id);
 	}
 
+	idr_init(&hndl->scmi_registered_nbs);
+
 	return hndl;
 }
 
@@ -779,17 +904,28 @@ int scmi_register_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
 {
 	u32 evt_key;
 	struct scmi_event_handler *hndl;
+	struct scmi_wrapper_block *cwnb;
 
 	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
 	hndl = scmi_get_or_create_event_handler(evt_key);
 	if (IS_ERR_OR_NULL(hndl))
 		return PTR_ERR(hndl);
 
-	blocking_notifier_chain_register(&hndl->chain, nb);
+	if (idr_find(&hndl->scmi_registered_nbs, PTR_2_EVT_KEY(nb))) {
+		pr_err("SCMI Duplicated NB\n");
+		scmi_put_event_handler(hndl);
+		return -EINVAL;
+	}
+
+	cwnb = scmi_register_wrapper_blocking_notifier(hndl, nb);
+	if (IS_ERR_OR_NULL(cwnb)) {
+		scmi_put_event_handler(hndl);
+		return PTR_ERR(cwnb);
+	}
 
 	if (!scmi_enable_events(hndl)) {
 		pr_err("SCMI Failed to ENABLE events for key:%X !\n", evt_key);
-		blocking_notifier_chain_unregister(&hndl->chain, nb);
+		scmi_unregister_wrapper_blocking_notifier(hndl, cwnb);
 		scmi_put_event_handler(hndl);
 		return -EINVAL;
 	}
@@ -816,15 +952,22 @@ int scmi_unregister_event_notifier(u8 proto_id, u8 evt_id, u32 *src_id,
 {
 	u32 evt_key;
 	struct scmi_event_handler *hndl;
+	struct scmi_wrapper_block *cwnb;
 
 	evt_key = MAKE_EVT_KEY(proto_id, evt_id, src_id);
 	hndl = scmi_get_event_handler(evt_key);
 	if (IS_ERR_OR_NULL(hndl))
 		return -EINVAL;
 
-	blocking_notifier_chain_unregister(&hndl->chain, nb);
-
+	cwnb = idr_find(&hndl->scmi_registered_nbs, PTR_2_EVT_KEY(nb));
+	if (!cwnb) {
+		pr_err("SCMI Unknown NB\n");
+		scmi_put_event_handler(hndl);
+		return -EINVAL;
+	}
+	scmi_unregister_wrapper_blocking_notifier(hndl, cwnb);
 	scmi_put_event_handler(hndl);
+
 	/*
 	 * If this was the last user callback for this handler, this last put
 	 * will force the handler to be freed.
-- 
2.17.1

[RFC PATCH 06/11] firmware: arm_scmi: Enable core notifications

[Cristian Marussi]

Initialize and enable SCMI Notification core support during bus/driver
init and probe phases: protocol can register their supported events
and users can register their callbacks for interested events.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/bus.c    | 3 +++
 drivers/firmware/arm_scmi/driver.c | 7 +++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c
index db55c43a2cbd..80d5ad29d205 100644
--- a/drivers/firmware/arm_scmi/bus.c
+++ b/drivers/firmware/arm_scmi/bus.c
@@ -14,6 +14,7 @@
 #include <linux/device.h>
 
 #include "common.h"
+#include "notify.h"
 
 static DEFINE_IDA(scmi_bus_id);
 static DEFINE_IDR(scmi_protocols);
@@ -234,6 +235,7 @@ static int __init scmi_bus_init(void)
 {
 	int retval;
 
+	scmi_notification_init();
 	retval = bus_register(&scmi_bus_type);
 	if (retval)
 		pr_err("scmi protocol bus register failed (%d)\n", retval);
@@ -245,6 +247,7 @@ subsys_initcall(scmi_bus_init);
 static void __exit scmi_bus_exit(void)
 {
 	scmi_devices_unregister();
+	scmi_notification_exit();
 	bus_unregister(&scmi_bus_type);
 	ida_destroy(&scmi_bus_id);
 }
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index a43fad29de11..a60559201f2d 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -28,6 +28,7 @@
 #include <linux/slab.h>
 
 #include "common.h"
+#include "notify.h"
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/scmi.h>
@@ -350,14 +351,14 @@ __scmi_xfer_put(struct scmi_xfers_info *minfo, struct scmi_xfer *xfer)
 
 static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 {
-	ktime_t ts;
+	u64 ts;
 	struct scmi_xfer *xfer;
 	struct device *dev = cinfo->dev;
 	struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
 	struct scmi_xfers_info *minfo = &info->rx_minfo;
 	struct scmi_shared_mem __iomem *mem = cinfo->payload;
 
-	ts = ktime_get_boottime();
+	ts = ktime_get_boottime_ns();
 	xfer = scmi_xfer_get(cinfo->handle, minfo);
 	if (IS_ERR(xfer)) {
 		dev_err(dev, "failed to get free message slot (%ld)\n",
@@ -370,6 +371,8 @@ static void scmi_handle_notification(struct scmi_chan_info *cinfo, u32 msg_hdr)
 	unpack_scmi_header(msg_hdr, &xfer->hdr);
 	scmi_dump_header_dbg(dev, &xfer->hdr);
 	scmi_fetch_notification(xfer, info->desc->max_msg_size, mem);
+	scmi_notify(xfer->hdr.protocol_id, xfer->hdr.id, xfer->rx.buf,
+		    xfer->rx.len, ts);
 	__scmi_xfer_put(minfo, xfer);
 
 	iowrite32(SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE, &mem->channel_status);
-- 
2.17.1

[RFC PATCH 07/11] firmware: arm_scmi: Add Power notifications support

[Cristian Marussi]

Make SCMI Power protocol register with the notification core.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/power.c | 155 +++++++++++++++++++++++++++++-
 include/linux/scmi_protocol.h     |  26 +++++
 2 files changed, 180 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/arm_scmi/power.c b/drivers/firmware/arm_scmi/power.c
index cf7f0312381b..024d6a375710 100644
--- a/drivers/firmware/arm_scmi/power.c
+++ b/drivers/firmware/arm_scmi/power.c
@@ -6,6 +6,7 @@
  */
 
 #include "common.h"
+#include "notify.h"
 
 enum scmi_power_protocol_cmd {
 	POWER_DOMAIN_ATTRIBUTES = 0x3,
@@ -48,6 +49,12 @@ struct scmi_power_state_notify {
 	__le32 notify_enable;
 };
 
+struct scmi_power_state_notify_payld {
+	__le32 agent_id;
+	__le32 domain_id;
+	__le32 power_state;
+};
+
 struct power_dom_info {
 	bool state_set_sync;
 	bool state_set_async;
@@ -61,6 +68,14 @@ struct scmi_power_info {
 	u64 stats_addr;
 	u32 stats_size;
 	struct power_dom_info *dom_info;
+	const struct scmi_handle *handle;
+};
+
+static struct scmi_power_info *pinfo;
+
+static enum scmi_power_protocol_cmd evt_2_cmd[] = {
+	POWER_STATE_NOTIFY,
+	POWER_STATE_CHANGE_REQUESTED_NOTIFY,
 };
 
 static int scmi_power_attributes_get(const struct scmi_handle *handle,
@@ -186,11 +201,130 @@ static struct scmi_power_ops power_ops = {
 	.state_get = scmi_power_state_get,
 };
 
+static int scmi_power_request_notify(const struct scmi_handle *handle,
+				     u32 domain, int message_id, bool enable)
+{
+	int ret;
+	struct scmi_xfer *t;
+	struct scmi_power_state_notify *notify;
+
+	ret = scmi_xfer_get_init(handle, message_id, SCMI_PROTOCOL_POWER,
+				 sizeof(*notify), 0, &t);
+	if (ret)
+		return ret;
+
+	notify = t->tx.buf;
+	notify->domain = cpu_to_le32(domain);
+	notify->notify_enable = enable ? cpu_to_le32(BIT(0)) : 0;
+
+	ret = scmi_do_xfer(handle, t);
+
+	scmi_xfer_put(handle, t);
+	return ret;
+}
+
+static bool scmi_power_set_notify_enabled(u8 evt_id, const u32 *src_id,
+					  bool enable)
+{
+	int ret, cmd_id;
+
+	cmd_id = MAP_EVT_TO_ENABLE_CMD(evt_id, evt_2_cmd);
+	if (cmd_id < 0)
+		return false;
+
+	if (src_id) {
+		ret = scmi_power_request_notify(pinfo->handle, *src_id,
+						cmd_id, enable);
+		if (ret)
+			pr_warn("Failed enabling SCMI Notifications - Power - evt[%X] dom[%d] - ret:%d\n",
+				evt_id, *src_id, ret);
+	} else {
+		int r, dom;
+
+		ret = 1;
+		for (dom = 0; dom < pinfo->num_domains; dom++) {
+			r = scmi_power_request_notify(pinfo->handle, dom,
+						      cmd_id, enable);
+			if (r)
+				pr_warn("Failed enabling SCMI Notifications - Power - evt[%X] dom[%d] - ret:%d\n",
+					evt_id, dom, r);
+			ret *= r;
+		}
+	}
+
+	return !ret ? true : false;
+}
+
+static void *scmi_power_fill_custom_report(u8 evt_id, u64 timestamp,
+					   const void *payld, size_t payld_sz,
+					   void *report, u32 *src_id)
+{
+	void *rep = NULL;
+
+	switch (evt_id) {
+	case POWER_STATE_CHANGED:
+	{
+		const struct scmi_power_state_notify_payld *p = payld;
+		struct scmi_power_state_changed_report *r = report;
+
+		if (sizeof(*p) != payld_sz)
+			break;
+
+		r->timestamp = timestamp;
+		r->agent_id = le32_to_cpu(p->agent_id);
+		r->domain_id = le32_to_cpu(p->domain_id);
+		r->power_state = le32_to_cpu(p->power_state);
+		*src_id = r->domain_id;
+		rep = r;
+		break;
+	}
+	case POWER_STATE_CHANGE_REQUESTED:
+	{
+		const struct scmi_power_state_notify_payld *p = payld;
+		struct scmi_power_state_change_requested_report *r = report;
+
+		if (sizeof(*p) != payld_sz)
+			break;
+
+		r->timestamp = timestamp;
+		r->agent_id = le32_to_cpu(p->agent_id);
+		r->domain_id = le32_to_cpu(p->domain_id);
+		r->power_state = le32_to_cpu(p->power_state);
+		*src_id = r->domain_id;
+		rep = r;
+		break;
+	}
+	default:
+		break;
+	}
+
+	return rep;
+}
+
+static const struct scmi_event power_events[] = {
+	{
+		.evt_id = POWER_STATE_CHANGED,
+		.max_payld_sz = 12,
+		.max_report_sz =
+			sizeof(struct scmi_power_state_changed_report),
+	},
+	{
+		.evt_id = POWER_STATE_CHANGE_REQUESTED,
+		.max_payld_sz = 12,
+		.max_report_sz =
+			sizeof(struct scmi_power_state_change_requested_report),
+	},
+};
+
+static const struct scmi_protocol_event_ops power_event_ops = {
+	.set_notify_enabled = scmi_power_set_notify_enabled,
+	.fill_custom_report = scmi_power_fill_custom_report,
+};
+
 static int scmi_power_protocol_init(struct scmi_handle *handle)
 {
 	int domain;
 	u32 version;
-	struct scmi_power_info *pinfo;
 
 	scmi_version_get(handle, SCMI_PROTOCOL_POWER, &version);
 
@@ -214,7 +348,12 @@ static int scmi_power_protocol_init(struct scmi_handle *handle)
 		scmi_power_domain_attributes_get(handle, domain, dom);
 	}
 
+	scmi_register_protocol_events(SCMI_PROTOCOL_POWER, PAGE_SIZE,
+				      &power_event_ops, power_events,
+				      ARRAY_SIZE(power_events));
+
 	pinfo->version = version;
+	pinfo->handle = handle;
 	handle->power_ops = &power_ops;
 	handle->power_priv = pinfo;
 
@@ -227,3 +366,17 @@ static int __init scmi_power_init(void)
 				      &scmi_power_protocol_init);
 }
 subsys_initcall(scmi_power_init);
+
+int scmi_register_power_event_notifier(u8 evt_id, u32 *dom_id,
+				       struct notifier_block *nb)
+{
+	return scmi_register_event_notifier(SCMI_PROTOCOL_POWER, evt_id,
+					    dom_id, nb);
+}
+
+int scmi_unregister_power_event_notifier(u8 evt_id, u32 *dom_id,
+					 struct notifier_block *nb)
+{
+	return scmi_unregister_event_notifier(SCMI_PROTOCOL_POWER, evt_id,
+					      dom_id, nb);
+}
diff --git a/include/linux/scmi_protocol.h b/include/linux/scmi_protocol.h
index 5c873a59b387..61bdfe0bebe2 100644
--- a/include/linux/scmi_protocol.h
+++ b/include/linux/scmi_protocol.h
@@ -4,8 +4,13 @@
  *
  * Copyright (C) 2018 ARM Ltd.
  */
+#ifndef _LINUX_SCMI_PROTOCOL_H
+#define _LINUX_SCMI_PROTOCOL_H
+
 #include <linux/device.h>
 #include <linux/types.h>
+#include <linux/notifier.h>
+#include <linux/ktime.h>
 
 #define SCMI_MAX_STR_SIZE	16
 #define SCMI_MAX_NUM_RATES	16
@@ -319,3 +324,24 @@ static inline void scmi_driver_unregister(struct scmi_driver *driver) {}
 typedef int (*scmi_prot_init_fn_t)(struct scmi_handle *);
 int scmi_protocol_register(int protocol_id, scmi_prot_init_fn_t fn);
 void scmi_protocol_unregister(int protocol_id);
+
+struct scmi_power_state_changed_report {
+	ktime_t	timestamp;
+	u32	agent_id;
+	u32	domain_id;
+	u32	power_state;
+};
+
+struct scmi_power_state_change_requested_report {
+	ktime_t	timestamp;
+	u32	agent_id;
+	u32	domain_id;
+	u32	power_state;
+};
+
+int scmi_register_power_event_notifier(u8 evt_id, u32 *dom_id,
+				       struct notifier_block *nb);
+int scmi_unregister_power_event_notifier(u8 evt_id, u32 *dom_id,
+					 struct notifier_block *nb);
+
+#endif /* _LINUX_SCMI_PROTOCOL_H */
-- 
2.17.1

[RFC PATCH 08/11] firmware: arm_scmi: Add Perf notifications support

[Cristian Marussi]

Make SCMI Perf protocol register with the notification core.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/perf.c | 162 ++++++++++++++++++++++++++++++-
 include/linux/scmi_protocol.h    |  20 ++++
 2 files changed, 181 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/arm_scmi/perf.c b/drivers/firmware/arm_scmi/perf.c
index 88509ec637d0..3e09ced2adc8 100644
--- a/drivers/firmware/arm_scmi/perf.c
+++ b/drivers/firmware/arm_scmi/perf.c
@@ -14,6 +14,7 @@
 #include <linux/sort.h>
 
 #include "common.h"
+#include "notify.h"
 
 enum scmi_performance_protocol_cmd {
 	PERF_DOMAIN_ATTRIBUTES = 0x3,
@@ -86,6 +87,19 @@ struct scmi_perf_notify_level_or_limits {
 	__le32 notify_enable;
 };
 
+struct scmi_perf_limits_notify_payld {
+	__le32 agent_id;
+	__le32 domain_id;
+	__le32 range_max;
+	__le32 range_min;
+};
+
+struct scmi_perf_level_notify_payld {
+	__le32 agent_id;
+	__le32 domain_id;
+	__le32 performance_level;
+};
+
 struct scmi_msg_resp_perf_describe_levels {
 	__le16 num_returned;
 	__le16 num_remaining;
@@ -156,6 +170,14 @@ struct scmi_perf_info {
 	u64 stats_addr;
 	u32 stats_size;
 	struct perf_dom_info *dom_info;
+	const struct scmi_handle *handle;
+};
+
+static struct scmi_perf_info *pinfo;
+
+static enum scmi_performance_protocol_cmd evt_2_cmd[] = {
+	PERF_NOTIFY_LIMITS,
+	PERF_NOTIFY_LEVEL,
 };
 
 static int scmi_perf_attributes_get(const struct scmi_handle *handle,
@@ -488,6 +510,29 @@ static int scmi_perf_level_get(const struct scmi_handle *handle, u32 domain,
 	return scmi_perf_mb_level_get(handle, domain, level, poll);
 }
 
+static int scmi_perf_level_limits_notify(const struct scmi_handle *handle,
+					 u32 domain, int message_id,
+					 bool enable)
+{
+	int ret;
+	struct scmi_xfer *t;
+	struct scmi_perf_notify_level_or_limits *notify;
+
+	ret = scmi_xfer_get_init(handle, message_id, SCMI_PROTOCOL_PERF,
+				 sizeof(*notify), 0, &t);
+	if (ret)
+		return ret;
+
+	notify = t->tx.buf;
+	notify->domain = cpu_to_le32(domain);
+	notify->notify_enable = enable ? cpu_to_le32(BIT(0)) : 0;
+
+	ret = scmi_do_xfer(handle, t);
+
+	scmi_xfer_put(handle, t);
+	return ret;
+}
+
 static bool scmi_perf_fc_size_is_valid(u32 msg, u32 size)
 {
 	if ((msg == PERF_LEVEL_GET || msg == PERF_LEVEL_SET) && size == 4)
@@ -710,11 +755,107 @@ static struct scmi_perf_ops perf_ops = {
 	.est_power_get = scmi_dvfs_est_power_get,
 };
 
+static bool scmi_perf_set_notify_enabled(u8 evt_id, const u32 *src_id,
+					 bool enable)
+{
+	int ret, cmd_id;
+
+	cmd_id = MAP_EVT_TO_ENABLE_CMD(evt_id, evt_2_cmd);
+	if (cmd_id < 0)
+		return false;
+
+	if (src_id) {
+		ret = scmi_perf_level_limits_notify(pinfo->handle,
+						    *src_id, cmd_id, enable);
+		if (ret)
+			pr_warn("Failed enabling SCMI Notifications - Perf - evt[%X] dom[%d] - ret:%d\n",
+				evt_id, *src_id, ret);
+	} else {
+		int r, dom;
+
+		ret = 1;
+		for (dom = 0; dom < pinfo->num_domains; dom++) {
+			r = scmi_perf_level_limits_notify(pinfo->handle, dom,
+							  cmd_id, enable);
+			if (r)
+				pr_warn("Failed enabling SCMI Notifications - Perf - evt[%X] dom[%d] - ret:%d\n",
+					evt_id, dom, r);
+			ret *= r;
+		}
+	}
+
+	return !ret ? true : false;
+}
+
+static void *scmi_perf_fill_custom_report(u8 evt_id, u64 timestamp,
+					  const void *payld, size_t payld_sz,
+					  void *report, u32 *src_id)
+{
+	void *rep = NULL;
+
+	switch (evt_id) {
+	case PERFORMANCE_LIMITS_CHANGED:
+	{
+		const struct scmi_perf_limits_notify_payld *p = payld;
+		struct scmi_perf_limits_report *r = report;
+
+		if (sizeof(*p) != payld_sz)
+			break;
+
+		r->timestamp = timestamp;
+		r->agent_id = le32_to_cpu(p->agent_id);
+		r->domain_id = le32_to_cpu(p->domain_id);
+		r->range_max = le32_to_cpu(p->range_max);
+		r->range_min = le32_to_cpu(p->range_min);
+		*src_id = r->domain_id;
+		rep = r;
+		break;
+	}
+	case PERFORMANCE_LEVEL_CHANGED:
+	{
+		const struct scmi_perf_level_notify_payld *p = payld;
+		struct scmi_perf_level_report *r = report;
+
+		if (sizeof(*p) != payld_sz)
+			break;
+
+		r->timestamp = timestamp;
+		r->agent_id = le32_to_cpu(p->agent_id);
+		r->domain_id = le32_to_cpu(p->domain_id);
+		r->performance_level = le32_to_cpu(p->performance_level);
+		*src_id = r->domain_id;
+		rep = r;
+		break;
+	}
+	default:
+		break;
+	}
+
+	return rep;
+}
+
+static const struct scmi_event perf_events[] = {
+	{
+		.evt_id = PERFORMANCE_LIMITS_CHANGED,
+		.max_payld_sz = 16,
+		.max_report_sz = sizeof(struct scmi_perf_limits_report),
+	},
+	{
+		.evt_id = PERFORMANCE_LEVEL_CHANGED,
+		.max_payld_sz = 12,
+		.max_report_sz = sizeof(struct scmi_perf_level_report),
+	},
+};
+
+static const struct scmi_protocol_event_ops perf_event_ops = {
+	.set_notify_enabled = scmi_perf_set_notify_enabled,
+	.fill_custom_report = scmi_perf_fill_custom_report,
+};
+
 static int scmi_perf_protocol_init(struct scmi_handle *handle)
 {
 	int domain;
 	u32 version;
-	struct scmi_perf_info *pinfo;
 
 	scmi_version_get(handle, SCMI_PROTOCOL_PERF, &version);
 
@@ -742,7 +883,12 @@ static int scmi_perf_protocol_init(struct scmi_handle *handle)
 			scmi_perf_domain_init_fc(handle, domain, &dom->fc_info);
 	}
 
+	scmi_register_protocol_events(SCMI_PROTOCOL_PERF, PAGE_SIZE,
+				      &perf_event_ops, perf_events,
+				      ARRAY_SIZE(perf_events));
+
 	pinfo->version = version;
+	pinfo->handle = handle;
 	handle->perf_ops = &perf_ops;
 	handle->perf_priv = pinfo;
 
@@ -755,3 +901,17 @@ static int __init scmi_perf_init(void)
 				      &scmi_perf_protocol_init);
 }
 subsys_initcall(scmi_perf_init);
+
+int scmi_register_perf_event_notifier(u8 evt_id, u32 *dom_id,
+				      struct notifier_block *nb)
+{
+	return scmi_register_event_notifier(SCMI_PROTOCOL_PERF, evt_id,
+					    dom_id, nb);
+}
+
+int scmi_unregister_perf_event_notifier(u8 evt_id, u32 *dom_id,
+					struct notifier_block *nb)
+{
+	return scmi_unregister_event_notifier(SCMI_PROTOCOL_PERF, evt_id,
+					      dom_id, nb);
+}
diff --git a/include/linux/scmi_protocol.h b/include/linux/scmi_protocol.h
index 61bdfe0bebe2..a719f3cd394b 100644
--- a/include/linux/scmi_protocol.h
+++ b/include/linux/scmi_protocol.h
@@ -344,4 +344,24 @@ int scmi_register_power_event_notifier(u8 evt_id, u32 *dom_id,
 int scmi_unregister_power_event_notifier(u8 evt_id, u32 *dom_id,
 					 struct notifier_block *nb);
 
+struct scmi_perf_limits_report {
+	ktime_t	timestamp;
+	u32	agent_id;
+	u32	domain_id;
+	u32	range_max;
+	u32	range_min;
+};
+
+struct scmi_perf_level_report {
+	ktime_t	timestamp;
+	u32	agent_id;
+	u32	domain_id;
+	u32	performance_level;
+};
+
+int scmi_register_perf_event_notifier(u8 evt_id, u32 *dom_id,
+				      struct notifier_block *nb);
+int scmi_unregister_perf_event_notifier(u8 evt_id, u32 *dom_id,
+					struct notifier_block *nb);
+
 #endif /* _LINUX_SCMI_PROTOCOL_H */
-- 
2.17.1

[RFC PATCH 09/11] firmware: arm_scmi: Add Sensor notifications support

[Cristian Marussi]

Make SCMI Sensor protocol register with the notification core.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/sensors.c | 101 +++++++++++++++++++++++++++-
 include/linux/scmi_protocol.h       |  12 ++++
 2 files changed, 112 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/arm_scmi/sensors.c b/drivers/firmware/arm_scmi/sensors.c
index db1b1ab303da..1f2f60516810 100644
--- a/drivers/firmware/arm_scmi/sensors.c
+++ b/drivers/firmware/arm_scmi/sensors.c
@@ -6,6 +6,7 @@
  */
 
 #include "common.h"
+#include "notify.h"
 
 enum scmi_sensor_protocol_cmd {
 	SENSOR_DESCRIPTION_GET = 0x3,
@@ -71,6 +72,12 @@ struct scmi_msg_sensor_reading_get {
 #define SENSOR_READ_ASYNC	BIT(0)
 };
 
+struct scmi_sensor_trip_notify_payld {
+	__le32 agent_id;
+	__le32 sensor_id;
+	__le32 trip_point_desc;
+};
+
 struct sensors_info {
 	u32 version;
 	int num_sensors;
@@ -78,8 +85,11 @@ struct sensors_info {
 	u64 reg_addr;
 	u32 reg_size;
 	struct scmi_sensor_info *sensors;
+	const struct scmi_handle *handle;
 };
 
+static struct sensors_info *sinfo;
+
 static int scmi_sensor_attributes_get(const struct scmi_handle *handle,
 				      struct sensors_info *si)
 {
@@ -276,10 +286,80 @@ static struct scmi_sensor_ops sensor_ops = {
 	.reading_get = scmi_sensor_reading_get,
 };
 
+static bool scmi_sensor_set_notify_enabled(u8 evt_id, const u32 *src_id,
+					   bool enable)
+{
+	int ret;
+
+	if (src_id) {
+		ret = scmi_sensor_trip_point_notify(sinfo->handle,
+						    *src_id, enable);
+		if (ret)
+			pr_warn("Failed enabling SCMI Notifications - Sensor - evt[%X] dom[%d] - ret:%d\n",
+				evt_id, *src_id, ret);
+	} else {
+		int r, sens;
+
+		ret = 1;
+		for (sens = 0; sens < sinfo->num_sensors; sens++) {
+			r = scmi_sensor_trip_point_notify(sinfo->handle, sens,
+							  enable);
+			if (r)
+				pr_warn("Failed enabling SCMI Notifications - Sensor - evt[%X] dom[%d] - ret:%d\n",
+					evt_id, sens, r);
+			ret *= r;
+		}
+	}
+
+	return !ret ? true : false;
+}
+
+static void *scmi_sensor_fill_custom_report(u8 evt_id, u64 timestamp,
+					   const void *payld, size_t payld_sz,
+					   void *report, u32 *src_id)
+{
+	void *rep = NULL;
+
+	switch (evt_id) {
+	case SENSOR_TRIP_POINT_EVENT:
+	{
+		const struct scmi_sensor_trip_notify_payld *p = payld;
+		struct scmi_sensor_trip_point_report *r = report;
+
+		if (sizeof(*p) != payld_sz)
+			break;
+
+		r->timestamp = timestamp;
+		r->agent_id = le32_to_cpu(p->agent_id);
+		r->sensor_id = le32_to_cpu(p->sensor_id);
+		r->trip_point_desc = le32_to_cpu(p->trip_point_desc);
+		*src_id = r->sensor_id;
+		rep = r;
+		break;
+	}
+	default:
+		break;
+	}
+
+	return rep;
+}
+
+static const struct scmi_event sensor_events[] = {
+	{
+		.evt_id = SENSOR_TRIP_POINT_EVENT,
+		.max_payld_sz = 12,
+		.max_report_sz = sizeof(struct scmi_sensor_trip_point_report),
+	},
+};
+
+static const struct scmi_protocol_event_ops sensor_event_ops = {
+	.set_notify_enabled = scmi_sensor_set_notify_enabled,
+	.fill_custom_report = scmi_sensor_fill_custom_report,
+};
+
 static int scmi_sensors_protocol_init(struct scmi_handle *handle)
 {
 	u32 version;
-	struct sensors_info *sinfo;
 
 	scmi_version_get(handle, SCMI_PROTOCOL_SENSOR, &version);
 
@@ -299,7 +379,12 @@ static int scmi_sensors_protocol_init(struct scmi_handle *handle)
 
 	scmi_sensor_description_get(handle, sinfo);
 
+	scmi_register_protocol_events(SCMI_PROTOCOL_SENSOR, PAGE_SIZE,
+				      &sensor_event_ops, sensor_events,
+				      ARRAY_SIZE(sensor_events));
+
 	sinfo->version = version;
+	sinfo->handle = handle;
 	handle->sensor_ops = &sensor_ops;
 	handle->sensor_priv = sinfo;
 
@@ -312,3 +397,17 @@ static int __init scmi_sensors_init(void)
 				      &scmi_sensors_protocol_init);
 }
 subsys_initcall(scmi_sensors_init);
+
+int scmi_register_sensor_event_notifier(u8 evt_id, u32 *sens_id,
+					struct notifier_block *nb)
+{
+	return scmi_register_event_notifier(SCMI_PROTOCOL_SENSOR, evt_id,
+					    sens_id, nb);
+}
+
+int scmi_unregister_sensor_event_notifier(u8 evt_id, u32 *sens_id,
+					  struct notifier_block *nb)
+{
+	return scmi_unregister_event_notifier(SCMI_PROTOCOL_SENSOR, evt_id,
+					      sens_id, nb);
+}
diff --git a/include/linux/scmi_protocol.h b/include/linux/scmi_protocol.h
index a719f3cd394b..833e21af52fe 100644
--- a/include/linux/scmi_protocol.h
+++ b/include/linux/scmi_protocol.h
@@ -364,4 +364,16 @@ int scmi_register_perf_event_notifier(u8 evt_id, u32 *dom_id,
 int scmi_unregister_perf_event_notifier(u8 evt_id, u32 *dom_id,
 					struct notifier_block *nb);
 
+struct scmi_sensor_trip_point_report {
+	ktime_t	timestamp;
+	u32	agent_id;
+	u32	sensor_id;
+	u32	trip_point_desc;
+};
+
+int scmi_register_sensor_event_notifier(u8 evt_id, u32 *sens_id,
+				      struct notifier_block *nb);
+int scmi_unregister_sensor_event_notifier(u8 evt_id, u32 *sens_id,
+					struct notifier_block *nb);
+
 #endif /* _LINUX_SCMI_PROTOCOL_H */
-- 
2.17.1

[RFC PATCH 10/11] firmware: arm_scmi: Add Reset notifications support

[Cristian Marussi]

Make SCMI Reset protocol register with the notification core.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/reset.c | 126 +++++++++++++++++++++++++++++-
 include/linux/scmi_protocol.h     |  11 +++
 2 files changed, 136 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/arm_scmi/reset.c b/drivers/firmware/arm_scmi/reset.c
index de73054554f3..528993d62fbb 100644
--- a/drivers/firmware/arm_scmi/reset.c
+++ b/drivers/firmware/arm_scmi/reset.c
@@ -6,6 +6,7 @@
  */
 
 #include "common.h"
+#include "notify.h"
 
 enum scmi_reset_protocol_cmd {
 	RESET_DOMAIN_ATTRIBUTES = 0x3,
@@ -40,6 +41,17 @@ struct scmi_msg_reset_domain_reset {
 #define ARCH_COLD_RESET		(ARCH_RESET_TYPE | COLD_RESET_STATE)
 };
 
+struct scmi_msg_reset_notify {
+	__le32 id;
+	__le32 event_control;
+#define RESET_TP_NOTIFY_ALL	BIT(0)
+};
+
+struct scmi_reset_issued_notify_payld {
+	__le32 domain_id;
+	__le32 reset_state;
+};
+
 struct reset_dom_info {
 	bool async_reset;
 	bool reset_notify;
@@ -51,8 +63,11 @@ struct scmi_reset_info {
 	u32 version;
 	int num_domains;
 	struct reset_dom_info *dom_info;
+	const struct scmi_handle *handle;
 };
 
+static struct scmi_reset_info *pinfo;
+
 static int scmi_reset_attributes_get(const struct scmi_handle *handle,
 				     struct scmi_reset_info *pi)
 {
@@ -190,11 +205,101 @@ static struct scmi_reset_ops reset_ops = {
 	.deassert = scmi_reset_domain_deassert,
 };
 
+static int scmi_reset_notify(const struct scmi_handle *handle, u32 domain_id,
+			     bool enable)
+{
+	int ret;
+	u32 evt_cntl = enable ? RESET_TP_NOTIFY_ALL : 0;
+	struct scmi_xfer *t;
+	struct scmi_msg_reset_notify *cfg;
+
+	ret = scmi_xfer_get_init(handle, RESET_NOTIFY,
+				 SCMI_PROTOCOL_RESET, sizeof(*cfg), 0, &t);
+	if (ret)
+		return ret;
+
+	cfg = t->tx.buf;
+	cfg->id = cpu_to_le32(domain_id);
+	cfg->event_control = cpu_to_le32(evt_cntl);
+
+	ret = scmi_do_xfer(handle, t);
+
+	scmi_xfer_put(handle, t);
+	return ret;
+}
+
+static bool scmi_reset_set_notify_enabled(u8 evt_id, const u32 *src_id,
+					  bool enable)
+{
+	int ret;
+
+	if (src_id) {
+		ret = scmi_reset_notify(pinfo->handle, *src_id, enable);
+		if (ret)
+			pr_warn("Failed enabling SCMI Notifications - Reset - evt[%X] dom[%d] - ret:%d\n",
+				evt_id, *src_id, ret);
+	} else {
+		int r, dom;
+
+		ret = 1;
+		for (dom = 0; dom < pinfo->num_domains; dom++) {
+			r = scmi_reset_notify(pinfo->handle, dom, enable);
+			if (r)
+				pr_warn("Failed enabling SCMI Notifications - Reset - evt[%X] dom[%d] - ret:%d\n",
+					evt_id, dom, r);
+			ret *= r;
+		}
+	}
+
+	return !ret ? true : false;
+}
+
+static void *scmi_reset_fill_custom_report(u8 evt_id, u64 timestamp,
+					   const void *payld, size_t payld_sz,
+					   void *report, u32 *src_id)
+{
+	void *rep = NULL;
+
+	switch (evt_id) {
+	case RESET_ISSUED:
+	{
+		const struct scmi_reset_issued_notify_payld *p = payld;
+		struct scmi_reset_issued_report *r = report;
+
+		if (sizeof(*p) != payld_sz)
+			break;
+
+		r->timestamp = timestamp;
+		r->domain_id = le32_to_cpu(p->domain_id);
+		r->reset_state = le32_to_cpu(p->reset_state);
+		*src_id = r->domain_id;
+		rep = r;
+		break;
+	}
+	default:
+		break;
+	}
+
+	return rep;
+}
+
+static const struct scmi_event reset_events[] = {
+	{
+		.evt_id = RESET_NOTIFY,
+		.max_payld_sz = 8,
+		.max_report_sz = sizeof(struct scmi_reset_issued_report),
+	},
+};
+
+static const struct scmi_protocol_event_ops reset_event_ops = {
+	.set_notify_enabled = scmi_reset_set_notify_enabled,
+	.fill_custom_report = scmi_reset_fill_custom_report,
+};
+
 static int scmi_reset_protocol_init(struct scmi_handle *handle)
 {
 	int domain;
 	u32 version;
-	struct scmi_reset_info *pinfo;
 
 	scmi_version_get(handle, SCMI_PROTOCOL_RESET, &version);
 
@@ -218,7 +323,12 @@ static int scmi_reset_protocol_init(struct scmi_handle *handle)
 		scmi_reset_domain_attributes_get(handle, domain, dom);
 	}
 
+	scmi_register_protocol_events(SCMI_PROTOCOL_RESET, PAGE_SIZE,
+				      &reset_event_ops, reset_events,
+				      ARRAY_SIZE(reset_events));
+
 	pinfo->version = version;
+	pinfo->handle = handle;
 	handle->reset_ops = &reset_ops;
 	handle->reset_priv = pinfo;
 
@@ -231,3 +341,17 @@ static int __init scmi_reset_init(void)
 				      &scmi_reset_protocol_init);
 }
 subsys_initcall(scmi_reset_init);
+
+int scmi_register_reset_event_notifier(u8 evt_id, u32 *dom_id,
+				       struct notifier_block *nb)
+{
+	return scmi_register_event_notifier(SCMI_PROTOCOL_RESET, evt_id,
+					    dom_id, nb);
+}
+
+int scmi_unregister_reset_event_notifier(u8 evt_id, u32 *dom_id,
+					 struct notifier_block *nb)
+{
+	return scmi_unregister_event_notifier(SCMI_PROTOCOL_RESET, evt_id,
+					      dom_id, nb);
+}
diff --git a/include/linux/scmi_protocol.h b/include/linux/scmi_protocol.h
index 833e21af52fe..4af258f07c1e 100644
--- a/include/linux/scmi_protocol.h
+++ b/include/linux/scmi_protocol.h
@@ -376,4 +376,15 @@ int scmi_register_sensor_event_notifier(u8 evt_id, u32 *sens_id,
 int scmi_unregister_sensor_event_notifier(u8 evt_id, u32 *sens_id,
 					struct notifier_block *nb);
 
+struct scmi_reset_issued_report {
+	ktime_t	timestamp;
+	u32	domain_id;
+	u32	reset_state;
+};
+
+int scmi_register_reset_event_notifier(u8 evt_id, u32 *dom_id,
+				       struct notifier_block *nb);
+int scmi_unregister_reset_event_notifier(u8 evt_id, u32 *dom_id,
+					 struct notifier_block *nb);
+
 #endif /* _LINUX_SCMI_PROTOCOL_H */
-- 
2.17.1

[RFC PATCH 11/11] firmware: arm_scmi: Add Base notifications support

[Cristian Marussi]

Make SCMI Base protocol register with the notification core.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 drivers/firmware/arm_scmi/base.c | 125 +++++++++++++++++++++++++++++++
 include/linux/scmi_protocol.h    |  13 ++++
 2 files changed, 138 insertions(+)

diff --git a/drivers/firmware/arm_scmi/base.c b/drivers/firmware/arm_scmi/base.c
index ce7d9203e41b..ea28b05d5af6 100644
--- a/drivers/firmware/arm_scmi/base.c
+++ b/drivers/firmware/arm_scmi/base.c
@@ -6,6 +6,7 @@
  */
 
 #include "common.h"
+#include "notify.h"
 
 enum scmi_base_protocol_cmd {
 	BASE_DISCOVER_VENDOR = 0x3,
@@ -29,6 +30,21 @@ struct scmi_msg_resp_base_attributes {
 	__le16 reserved;
 };
 
+struct scmi_msg_base_error_notify {
+	__le32 event_control;
+#define BASE_TP_NOTIFY_ALL	BIT(0)
+};
+
+struct scmi_base_error_notify_payld {
+	__le32 agent_id;
+	__le32 error_status;
+#define IS_FATAL_ERROR(x)	((x) & BIT(31))
+#define ERROR_CMD_COUNT(x)	FIELD_GET(GENMASK(9, 0), (x))
+	__le64 msg_reports[8192];
+};
+
+static const struct scmi_handle *protocol_handle;
+
 /**
  * scmi_base_attributes_get() - gets the implementation details
  *	that are associated with the base protocol.
@@ -222,6 +238,96 @@ static int scmi_base_discover_agent_get(const struct scmi_handle *handle,
 	return ret;
 }
 
+static int scmi_base_error_notify(const struct scmi_handle *handle, bool enable)
+{
+	int ret;
+	u32 evt_cntl = enable ? BASE_TP_NOTIFY_ALL : 0;
+	struct scmi_xfer *t;
+	struct scmi_msg_base_error_notify *cfg;
+
+	ret = scmi_xfer_get_init(handle, BASE_NOTIFY_ERRORS,
+				 SCMI_PROTOCOL_BASE, sizeof(*cfg), 0, &t);
+	if (ret)
+		return ret;
+
+	cfg = t->tx.buf;
+	cfg->event_control = cpu_to_le32(evt_cntl);
+
+	ret = scmi_do_xfer(handle, t);
+
+	scmi_xfer_put(handle, t);
+	return ret;
+}
+
+static bool scmi_base_set_notify_enabled(u8 evt_id, const u32 *src_id,
+					 bool enable)
+{
+	int ret;
+
+	if (!protocol_handle)
+		return false;
+
+	ret = scmi_base_error_notify(protocol_handle, enable);
+	if (ret)
+		pr_warn("Failed enabling SCMI Notifications - Base - evt[%X] ret:%d\n",
+			evt_id, ret);
+
+	return !ret ? true : false;
+}
+
+static void *scmi_base_fill_custom_report(u8 evt_id, u64 timestamp,
+					  const void *payld, size_t payld_sz,
+					  void *report, u32 *src_id)
+{
+	void *rep = NULL;
+
+	switch (evt_id) {
+	case BASE_ERROR_EVENT:
+	{
+		int i;
+		const struct scmi_base_error_notify_payld *p = payld;
+		struct scmi_base_error_report *r = report;
+
+		/*
+		 * BaseError notification payload is variable in size but
+		 * up to a maximum length determined by the struct ponted by p.
+		 * Instead payld_sz is the effective length of this notification
+		 * payload so cannot be greater of the maximum allowed size as
+		 * pointed by p.
+		 */
+		if (sizeof(*p) < payld_sz)
+			break;
+
+		r->timestamp = timestamp;
+		r->agent_id = le32_to_cpu(p->agent_id);
+		r->fatal = IS_FATAL_ERROR(le32_to_cpu(p->error_status));
+		r->cmd_count = ERROR_CMD_COUNT(le32_to_cpu(p->error_status));
+		for (i = 0; i < r->cmd_count; i++)
+			r->reports[i] = le64_to_cpu(p->msg_reports[i]);
+		*src_id = SCMI_ALL_SRC_IDS;
+		rep = r;
+		break;
+	}
+	default:
+		break;
+	}
+
+	return rep;
+}
+
+static const struct scmi_event base_events[] = {
+	{
+		.evt_id = BASE_ERROR_EVENT,
+		.max_payld_sz = 8192,
+		.max_report_sz = sizeof(struct scmi_base_error_report),
+	},
+};
+
+static const struct scmi_protocol_event_ops base_event_ops = {
+	.set_notify_enabled = scmi_base_set_notify_enabled,
+	.fill_custom_report = scmi_base_fill_custom_report,
+};
+
 int scmi_base_protocol_init(struct scmi_handle *h)
 {
 	int id, ret;
@@ -256,10 +362,29 @@ int scmi_base_protocol_init(struct scmi_handle *h)
 	dev_dbg(dev, "Found %d protocol(s) %d agent(s)\n", rev->num_protocols,
 		rev->num_agents);
 
+	scmi_register_protocol_events(SCMI_PROTOCOL_BASE, (4 * PAGE_SIZE),
+				      &base_event_ops, base_events,
+				      ARRAY_SIZE(base_events));
+
 	for (id = 0; id < rev->num_agents; id++) {
 		scmi_base_discover_agent_get(handle, id, name);
 		dev_dbg(dev, "Agent %d: %s\n", id, name);
 	}
+	protocol_handle = handle;
 
 	return 0;
 }
+
+int scmi_register_base_event_notifier(u8 evt_id, u32 *src_id,
+				      struct notifier_block *nb)
+{
+	return scmi_register_event_notifier(SCMI_PROTOCOL_BASE, evt_id,
+					    src_id, nb);
+}
+
+int scmi_unregister_base_event_notifier(u8 evt_id, u32 *src_id,
+					struct notifier_block *nb)
+{
+	return scmi_unregister_event_notifier(SCMI_PROTOCOL_BASE, evt_id,
+					      src_id, nb);
+}
diff --git a/include/linux/scmi_protocol.h b/include/linux/scmi_protocol.h
index 4af258f07c1e..24a64e9ba616 100644
--- a/include/linux/scmi_protocol.h
+++ b/include/linux/scmi_protocol.h
@@ -387,4 +387,17 @@ int scmi_register_reset_event_notifier(u8 evt_id, u32 *dom_id,
 int scmi_unregister_reset_event_notifier(u8 evt_id, u32 *dom_id,
 					 struct notifier_block *nb);
 
+struct scmi_base_error_report {
+	ktime_t	timestamp;
+	u32	agent_id;
+	bool	fatal;
+	u16	cmd_count;
+	u64	reports[8192];
+};
+
+int scmi_register_base_event_notifier(u8 evt_id, u32 *src_id,
+				      struct notifier_block *nb);
+int scmi_unregister_base_event_notifier(u8 evt_id, u32 *src_id,
+					struct notifier_block *nb);
+
 #endif /* _LINUX_SCMI_PROTOCOL_H */
-- 
2.17.1

Re: [RFC PATCH 00/11] SCMI Notifications Support

[Cristian Marussi]

Hi

On 20/01/2020 12:23, Cristian Marussi wrote:
> Hi all,
> 
> this series wants to introduce SCMI Notification Support, built on top of
> the standard Kernel notification chain subsystem.
> 
[snip]

> 
> Based on scmi-next [1], on top of:
> 
> commit 257d0e20ec4f ("include: trace: Add SCMI header with trace events")

Sorry but I've got this is wrong, the series is based in fact on top of the
very following commit on scmi-next [1]:

729d3530a504 drivers: firmware: scmi: Extend SCMI transport layer by trace events

Thanks Jim for reporting this.

Regards

Cristian

> 
> This series has been tested on JUNO with an experimental firmware only
> supporting Perf Notifications.
> 
> Any thoughts ?
> 
> Thanks
> 
> Cristian
> ----
> 
> [1] git://git.kernel.org/pub/scm/linux/kernel/git/sudeep.holla/linux.git
> 
> Cristian Marussi (8):
>   firmware: arm_scmi: Add core notifications support
>   firmware: arm_scmi: Add notifications anti-tampering
>   firmware: arm_scmi: Enable core notifications
>   firmware: arm_scmi: Add Power notifications support
>   firmware: arm_scmi: Add Perf notifications support
>   firmware: arm_scmi: Add Sensor notifications support
>   firmware: arm_scmi: Add Reset notifications support
>   firmware: arm_scmi: Add Base notifications support
> 
> Sudeep Holla (3):
>   firmware: arm_scmi: Add receive buffer support for notifications
>   firmware: arm_scmi: Update protocol commands and notification list
>   firmware: arm_scmi: Add support for notifications message processing
> 
>  drivers/firmware/arm_scmi/Makefile  |    2 +-
>  drivers/firmware/arm_scmi/base.c    |  132 ++++
>  drivers/firmware/arm_scmi/bus.c     |    3 +
>  drivers/firmware/arm_scmi/common.h  |    4 +
>  drivers/firmware/arm_scmi/driver.c  |  121 +++-
>  drivers/firmware/arm_scmi/notify.c  | 1047 +++++++++++++++++++++++++++
>  drivers/firmware/arm_scmi/notify.h  |   79 ++
>  drivers/firmware/arm_scmi/perf.c    |  167 ++++-
>  drivers/firmware/arm_scmi/power.c   |  161 +++-
>  drivers/firmware/arm_scmi/reset.c   |  126 +++-
>  drivers/firmware/arm_scmi/sensors.c |  105 ++-
>  include/linux/scmi_protocol.h       |   82 +++
>  12 files changed, 1991 insertions(+), 38 deletions(-)
>  create mode 100644 drivers/firmware/arm_scmi/notify.c
>  create mode 100644 drivers/firmware/arm_scmi/notify.h
>