[PATCH v6 0/6] drm/i915/pxp: Add missing cleanup steps for PXP global-teardown

[PATCH v6 0/6] drm/i915/pxp: Add missing cleanup steps for PXP global-teardown

[Alan Previn]

A customer issue was recently discovered and in the process a
gap in i915's PXP interaction with HW+FW architecure was also
realized. This series adds those missing pieces.

This fix includes changes where i915 calls into the mei
component interface in order to submit requests to the security
firmware during the i915's suspend_prepare flow. This change did
expose a blocking issue in the mei component side that was
discovered while testing in rev1. The issue being the mei-pxp
component driver not being able to runtime-resume while being
within the suspend_prepare callstack.


Thus, we have now included the mei patches (from Alexander) that
fixes that issue by adding a device-link based on the interface
type to ensure mei side runtime resume during the i915's
suspend_prepare call.

That said, as per request from Alexander, we seek Greg's and Tomas'
review for the mei patches (Patch 1, 2 and 3). Patch 2, although is
a change in the i915 code, is the mei component device link change.

The individual patches explain more details. Patch 7 can be ignored
as it won't be merged and is only meant to ensure the CI run's
the PXP subtests with PXP support enabled in KConfig.

Changes from prior revs:
   v1: - Dont need to teardown non-arbitration sessions (Juston).
       - Fix builds when PXP is enabled in config (Alan/CI-build).
       - Fix the broken pm-suspend-resume symmetry when we do this
         pxp-session-teardown during i915s pm_suspend_prepare by
         ensuring the init is done during i915s pm_resume_complete.
   v2: - Rebase on latest drm-tip after PXP subsytem was promoted
         to global.
       - Remove "INTEL_PXP_MAX_HWDRM_SESSIONS" unneeded (Juston Li).
       - Added mei patches that are dependencies for this series
         to successfully pass testing when PXP config is enabled.
   v3: - Added fix for mei patch when CONFIG_PM_SLEEP is off (reported
         by kernel test robot <lkp@intel.com>).
   v4: - Added "DRM_SWITCH_POWER_OFF" check and removed bail-out if
         '!i915' that wont happen in i915_pm_complete (Daniele).
       - move i915_pm_complete to appear in i915_pm_resume.
       - One more fix for mei patch when CONFIG_PM_SLEEP is off
         (reported by kernel test robot <lkp@intel.com>).
   v5: - Reworked Patch#2 on device link establishment. Don't hide
         triggering device-link behind drm_WARN, return -ENODEV if
         it fails and stash the returned device_link struct.

Alan Previn (3):
  drm/i915/pxp: Invalidate all PXP fw sessions during teardown
  drm/i915/pxp: Trigger the global teardown for before suspending
  drm/i915/pxp: Pxp hw init should be in resume_complete

Alexander Usyskin (3):
  mei: mei-me: resume device in prepare
  drm/i915/pxp: add device link between i915 and mei_pxp
  mei: clean pending read with vtag on bus

 drivers/gpu/drm/i915/i915_driver.c            | 20 +++++-
 drivers/gpu/drm/i915/pxp/intel_pxp.c          | 65 ++++++++++++++++---
 drivers/gpu/drm/i915/pxp/intel_pxp.h          |  2 +
 .../drm/i915/pxp/intel_pxp_cmd_interface_42.h | 15 +++++
 .../i915/pxp/intel_pxp_cmd_interface_cmn.h    |  3 +
 drivers/gpu/drm/i915/pxp/intel_pxp_pm.c       |  4 +-
 drivers/gpu/drm/i915/pxp/intel_pxp_pm.h       |  6 +-
 drivers/gpu/drm/i915/pxp/intel_pxp_session.c  |  8 ++-
 drivers/gpu/drm/i915/pxp/intel_pxp_session.h  |  5 ++
 drivers/gpu/drm/i915/pxp/intel_pxp_tee.c      | 46 +++++++++++++
 drivers/gpu/drm/i915/pxp/intel_pxp_types.h    |  6 ++
 drivers/misc/mei/client.c                     |  4 +-
 drivers/misc/mei/pci-me.c                     | 20 +++++-
 13 files changed, 183 insertions(+), 21 deletions(-)


base-commit: 6a023df4443d313724dc96d1fff15193bb7ec5b8
-- 
2.39.0

[PATCH v6 1/6] mei: mei-me: resume device in prepare

[Alan Previn]

From: Alexander Usyskin <alexander.usyskin@intel.com>

Asynchronous runtime resume is not possible while the system
is suspending.
The power management subsystem resumes the device only in the
suspend phase, not in the prepare phase.
Force resume device in prepare to allow drivers on mei bus
to communicate in their prepare callbacks.

Signed-off-by: Alexander Usyskin <alexander.usyskin@intel.com>
Reviewed-by: Tomas Winkler <tomas.winkler@intel.com>
---
 drivers/misc/mei/pci-me.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/mei/pci-me.c b/drivers/misc/mei/pci-me.c
index 5bf0d50d55a0..676d566f38dd 100644
--- a/drivers/misc/mei/pci-me.c
+++ b/drivers/misc/mei/pci-me.c
@@ -342,6 +342,12 @@ static void mei_me_remove(struct pci_dev *pdev)
 }
 
 #ifdef CONFIG_PM_SLEEP
+static int mei_me_pci_prepare(struct device *device)
+{
+	pm_runtime_resume(device);
+	return 0;
+}
+
 static int mei_me_pci_suspend(struct device *device)
 {
 	struct pci_dev *pdev = to_pci_dev(device);
@@ -398,7 +404,17 @@ static int mei_me_pci_resume(struct device *device)
 
 	return 0;
 }
-#endif /* CONFIG_PM_SLEEP */
+
+static void mei_me_pci_complete(struct device *device)
+{
+	pm_runtime_suspend(device);
+}
+#else /* CONFIG_PM_SLEEP */
+
+#define mei_me_pci_prepare NULL
+#define mei_me_pci_complete NULL
+
+#endif /* !CONFIG_PM_SLEEP */
 
 #ifdef CONFIG_PM
 static int mei_me_pm_runtime_idle(struct device *device)
@@ -501,6 +517,8 @@ static inline void mei_me_unset_pm_domain(struct mei_device *dev)
 }
 
 static const struct dev_pm_ops mei_me_pm_ops = {
+	.prepare = mei_me_pci_prepare,
+	.complete = mei_me_pci_complete,
 	SET_SYSTEM_SLEEP_PM_OPS(mei_me_pci_suspend,
 				mei_me_pci_resume)
 	SET_RUNTIME_PM_OPS(
-- 
2.39.0

[PATCH v6 2/6] drm/i915/pxp: add device link between i915 and mei_pxp

[Alan Previn]

From: Alexander Usyskin <alexander.usyskin@intel.com>

Add device link with i915 as consumer and mei_pxp as supplier
to ensure proper ordering of power flows.

V2: condition on absence of heci_pxp to filter out DG

Signed-off-by: Alexander Usyskin <alexander.usyskin@intel.com>
---
 drivers/gpu/drm/i915/pxp/intel_pxp_tee.c   | 11 +++++++++++
 drivers/gpu/drm/i915/pxp/intel_pxp_types.h |  6 ++++++
 2 files changed, 17 insertions(+)

diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
index 73aa8015f828..cd5b86216506 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
@@ -127,6 +127,12 @@ static int i915_pxp_tee_component_bind(struct device *i915_kdev,
 	intel_wakeref_t wakeref;
 	int ret = 0;
 
+	if (!HAS_HECI_PXP(i915)) {
+		pxp->component_dev_link = device_link_add(i915_kdev, tee_kdev, DL_FLAG_STATELESS);
+		if (drm_WARN_ON(&i915->drm, !pxp->component_dev_link))
+			return -ENODEV;
+	}
+
 	mutex_lock(&pxp->tee_mutex);
 	pxp->pxp_component = data;
 	pxp->pxp_component->tee_dev = tee_kdev;
@@ -169,6 +175,11 @@ static void i915_pxp_tee_component_unbind(struct device *i915_kdev,
 	mutex_lock(&pxp->tee_mutex);
 	pxp->pxp_component = NULL;
 	mutex_unlock(&pxp->tee_mutex);
+
+	if (pxp->component_dev_link) {
+		device_link_remove(i915_kdev, tee_kdev);
+		pxp->component_dev_link = NULL;
+	}
 }
 
 static const struct component_ops i915_pxp_tee_component_ops = {
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_types.h b/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
index 7dc5f08d1583..efd2f3915abe 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
@@ -32,6 +32,12 @@ struct intel_pxp {
 	 * which are protected by &tee_mutex.
 	 */
 	struct i915_pxp_component *pxp_component;
+
+	/**
+	 * @component_dev_link: device link of the pxp-component enforcing i915 as the
+	 * consumer. This is needed for legacy platform (TGL/ADL) full-feature usage.
+	 */
+	struct device_link *component_dev_link;
 	/**
 	 * @pxp_component_added: track if the pxp component has been added.
 	 * Set and cleared in tee init and fini functions respectively.
-- 
2.39.0

[PATCH v6 3/6] mei: clean pending read with vtag on bus

[Alan Previn]

From: Alexander Usyskin <alexander.usyskin@intel.com>

Client on bus have only one vtag map slot and should disregard the vtag
value when cleaning pending read flag.
Fixes read flow control message unexpectedly generated when
clent on bus send messages with different vtags.

Signed-off-by: Alexander Usyskin <alexander.usyskin@intel.com>
Reviewed-by: Tomas Winkler <tomas.winkler@intel.com>
---
 drivers/misc/mei/client.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/mei/client.c b/drivers/misc/mei/client.c
index 9ddb854b8155..5c19097266fe 100644
--- a/drivers/misc/mei/client.c
+++ b/drivers/misc/mei/client.c
@@ -1343,7 +1343,9 @@ static void mei_cl_reset_read_by_vtag(const struct mei_cl *cl, u8 vtag)
 	struct mei_cl_vtag *vtag_l;
 
 	list_for_each_entry(vtag_l, &cl->vtag_map, list) {
-		if (vtag_l->vtag == vtag) {
+		/* The client on bus has one fixed vtag map */
+		if ((cl->cldev && mei_cldev_enabled(cl->cldev)) ||
+		    vtag_l->vtag == vtag) {
 			vtag_l->pending_read = false;
 			break;
 		}
-- 
2.39.0

[PATCH v6 4/6] drm/i915/pxp: Invalidate all PXP fw sessions during teardown

[Alan Previn]

A gap was recently discovered where if an application did not
invalidate all of the stream keys (intentionally or not), and the
driver did a full PXP global teardown on the GT subsystem, we
find that future session creation would fail on the security
firmware's side of the equation. i915 is the entity that needs
ensure the sessions' state across both iGT and security firmware
are at a known clean point when performing a full global teardown.

Architecturally speaking, i915 should inspect all active sessions
and submit the invalidate-stream-key PXP command to the security
firmware for each of them. However, for the upstream i915 driver
we only support the arbitration session that can be created
so that will be the only session we will cleanup.

Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
Reviewed-by: Juston Li <justonli@chromium.org>
Acked-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
---
 drivers/gpu/drm/i915/pxp/intel_pxp.h          |  1 +
 .../drm/i915/pxp/intel_pxp_cmd_interface_42.h | 15 ++++++++
 .../i915/pxp/intel_pxp_cmd_interface_cmn.h    |  3 ++
 drivers/gpu/drm/i915/pxp/intel_pxp_session.c  |  2 ++
 drivers/gpu/drm/i915/pxp/intel_pxp_tee.c      | 35 +++++++++++++++++++
 5 files changed, 56 insertions(+)

diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.h b/drivers/gpu/drm/i915/pxp/intel_pxp.h
index 04440fada711..9658d3005222 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp.h
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp.h
@@ -24,6 +24,7 @@ void intel_pxp_init_hw(struct intel_pxp *pxp);
 void intel_pxp_fini_hw(struct intel_pxp *pxp);
 
 void intel_pxp_mark_termination_in_progress(struct intel_pxp *pxp);
+void intel_pxp_tee_end_arb_fw_session(struct intel_pxp *pxp, u32 arb_session_id);
 
 int intel_pxp_start(struct intel_pxp *pxp);
 
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_42.h b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_42.h
index 739f9072fa5f..26f7d9f01bf3 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_42.h
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_42.h
@@ -12,6 +12,9 @@
 /* PXP-Opcode for Init Session */
 #define PXP42_CMDID_INIT_SESSION 0x1e
 
+/* PXP-Opcode for Invalidate Stream Key */
+#define PXP42_CMDID_INVALIDATE_STREAM_KEY 0x00000007
+
 /* PXP-Input-Packet: Init Session (Arb-Session) */
 struct pxp42_create_arb_in {
 	struct pxp_cmd_header header;
@@ -25,4 +28,16 @@ struct pxp42_create_arb_out {
 	struct pxp_cmd_header header;
 } __packed;
 
+/* PXP-Input-Packet: Invalidate Stream Key */
+struct pxp42_inv_stream_key_in {
+	struct pxp_cmd_header header;
+	u32 rsvd[3];
+} __packed;
+
+/* PXP-Output-Packet: Invalidate Stream Key */
+struct pxp42_inv_stream_key_out {
+	struct pxp_cmd_header header;
+	u32 rsvd;
+} __packed;
+
 #endif /* __INTEL_PXP_FW_INTERFACE_42_H__ */
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_cmn.h b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_cmn.h
index aaa8187a0afb..ae9b151b7cb7 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_cmn.h
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_cmn.h
@@ -28,6 +28,9 @@ struct pxp_cmd_header {
 	union {
 		u32 status; /* out */
 		u32 stream_id; /* in */
+#define PXP_CMDHDR_EXTDATA_SESSION_VALID GENMASK(0, 0)
+#define PXP_CMDHDR_EXTDATA_APP_TYPE GENMASK(1, 1)
+#define PXP_CMDHDR_EXTDATA_SESSION_ID GENMASK(17, 2)
 	};
 	/* Length of the message (excluding the header) */
 	u32 buffer_len;
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
index ae413580b81a..74ed7e16e481 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
@@ -110,6 +110,8 @@ static int pxp_terminate_arb_session_and_global(struct intel_pxp *pxp)
 
 	intel_uncore_write(gt->uncore, PXP_GLOBAL_TERMINATE, 1);
 
+	intel_pxp_tee_end_arb_fw_session(pxp, ARB_SESSION);
+
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
index cd5b86216506..aa0ad46e524b 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
@@ -319,3 +319,38 @@ int intel_pxp_tee_cmd_create_arb_session(struct intel_pxp *pxp,
 
 	return ret;
 }
+
+void intel_pxp_tee_end_arb_fw_session(struct intel_pxp *pxp, u32 session_id)
+{
+	struct drm_i915_private *i915 = pxp->ctrl_gt->i915;
+	struct pxp42_inv_stream_key_in msg_in = {0};
+	struct pxp42_inv_stream_key_out msg_out = {0};
+	int ret, trials = 0;
+
+try_again:
+	memset(&msg_in, 0, sizeof(msg_in));
+	memset(&msg_out, 0, sizeof(msg_out));
+	msg_in.header.api_version = PXP_APIVER(4, 2);
+	msg_in.header.command_id = PXP42_CMDID_INVALIDATE_STREAM_KEY;
+	msg_in.header.buffer_len = sizeof(msg_in) - sizeof(msg_in.header);
+
+	msg_in.header.stream_id = FIELD_PREP(PXP_CMDHDR_EXTDATA_SESSION_VALID, 1);
+	msg_in.header.stream_id |= FIELD_PREP(PXP_CMDHDR_EXTDATA_APP_TYPE, 0);
+	msg_in.header.stream_id |= FIELD_PREP(PXP_CMDHDR_EXTDATA_SESSION_ID, session_id);
+
+	ret = intel_pxp_tee_io_message(pxp,
+				       &msg_in, sizeof(msg_in),
+				       &msg_out, sizeof(msg_out),
+				       NULL);
+
+	/* Cleanup coherency between GT and Firmware is critical, so try again if it fails */
+	if ((ret || msg_out.header.status != 0x0) && ++trials < 3)
+		goto try_again;
+
+	if (ret)
+		drm_err(&i915->drm, "Failed to send tee msg for inv-stream-key-%d, ret=[%d]\n",
+			session_id, ret);
+	else if (msg_out.header.status != 0x0)
+		drm_warn(&i915->drm, "PXP firmware failed inv-stream-key-%d with status 0x%08x\n",
+			 session_id, msg_out.header.status);
+}
-- 
2.39.0

[PATCH v6 5/6] drm/i915/pxp: Trigger the global teardown for before suspending

[Alan Previn]

A driver bug was recently discovered where the security firmware was
receiving internal HW signals indicating that session key expirations
had occurred. Architecturally, the firmware was expecting a response
from the GuC to acknowledge the event with the firmware side.
However the OS was in a suspended state and GuC had been reset.

Internal specifications actually required the driver to ensure
that all active sessions be properly cleaned up in such cases where
the system is suspended and the GuC potentially unable to respond.

This patch adds the global teardown code in i915's suspend_prepare
code path.

Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
Reviewed-by: Juston Li <justonli@chromium.org>
---
 drivers/gpu/drm/i915/pxp/intel_pxp.c         | 65 +++++++++++++++++---
 drivers/gpu/drm/i915/pxp/intel_pxp.h         |  1 +
 drivers/gpu/drm/i915/pxp/intel_pxp_pm.c      |  2 +-
 drivers/gpu/drm/i915/pxp/intel_pxp_session.c |  6 +-
 drivers/gpu/drm/i915/pxp/intel_pxp_session.h |  5 ++
 5 files changed, 66 insertions(+), 13 deletions(-)

diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.c b/drivers/gpu/drm/i915/pxp/intel_pxp.c
index cfc9af8b3d21..9d4c7724e98e 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp.c
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp.c
@@ -270,6 +270,60 @@ static bool pxp_component_bound(struct intel_pxp *pxp)
 	return bound;
 }
 
+static int __pxp_global_teardown_final(struct intel_pxp *pxp)
+{
+	if (!pxp->arb_is_valid)
+		return 0;
+	/*
+	 * To ensure synchronous and coherent session teardown completion
+	 * in response to suspend or shutdown triggers, don't use a worker.
+	 */
+	intel_pxp_mark_termination_in_progress(pxp);
+	intel_pxp_terminate(pxp, false);
+
+	if (!wait_for_completion_timeout(&pxp->termination, msecs_to_jiffies(250)))
+		return -ETIMEDOUT;
+
+	return 0;
+}
+
+static int __pxp_global_teardown_restart(struct intel_pxp *pxp)
+{
+	if (pxp->arb_is_valid)
+		return 0;
+	/*
+	 * The arb-session is currently inactive and we are doing a reset and restart
+	 * due to a runtime event. Use the worker that was designed for this.
+	 */
+	pxp_queue_termination(pxp);
+
+	if (!wait_for_completion_timeout(&pxp->termination, msecs_to_jiffies(250)))
+		return -ETIMEDOUT;
+
+	return 0;
+}
+
+void intel_pxp_end(struct intel_pxp *pxp)
+{
+	struct drm_i915_private *i915 = pxp->ctrl_gt->i915;
+	intel_wakeref_t wakeref;
+
+	if (!intel_pxp_is_enabled(pxp))
+		return;
+
+	wakeref = intel_runtime_pm_get(&i915->runtime_pm);
+
+	mutex_lock(&pxp->arb_mutex);
+
+	if (__pxp_global_teardown_final(pxp))
+		drm_dbg(&i915->drm, "PXP end timed out\n");
+
+	mutex_unlock(&pxp->arb_mutex);
+
+	intel_pxp_fini_hw(pxp);
+	intel_runtime_pm_put(&i915->runtime_pm, wakeref);
+}
+
 /*
  * the arb session is restarted from the irq work when we receive the
  * termination completion interrupt
@@ -286,16 +340,9 @@ int intel_pxp_start(struct intel_pxp *pxp)
 
 	mutex_lock(&pxp->arb_mutex);
 
-	if (pxp->arb_is_valid)
-		goto unlock;
-
-	pxp_queue_termination(pxp);
-
-	if (!wait_for_completion_timeout(&pxp->termination,
-					msecs_to_jiffies(250))) {
-		ret = -ETIMEDOUT;
+	ret = __pxp_global_teardown_restart(pxp);
+	if (ret)
 		goto unlock;
-	}
 
 	/* make sure the compiler doesn't optimize the double access */
 	barrier();
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.h b/drivers/gpu/drm/i915/pxp/intel_pxp.h
index 9658d3005222..3ded0890cd27 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp.h
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp.h
@@ -27,6 +27,7 @@ void intel_pxp_mark_termination_in_progress(struct intel_pxp *pxp);
 void intel_pxp_tee_end_arb_fw_session(struct intel_pxp *pxp, u32 arb_session_id);
 
 int intel_pxp_start(struct intel_pxp *pxp);
+void intel_pxp_end(struct intel_pxp *pxp);
 
 int intel_pxp_key_check(struct intel_pxp *pxp,
 			struct drm_i915_gem_object *obj,
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
index 892d39cc61c1..e427464aa131 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
@@ -16,7 +16,7 @@ void intel_pxp_suspend_prepare(struct intel_pxp *pxp)
 	if (!intel_pxp_is_enabled(pxp))
 		return;
 
-	pxp->arb_is_valid = false;
+	intel_pxp_end(pxp);
 
 	intel_pxp_invalidate(pxp);
 }
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
index 74ed7e16e481..448cacb0465d 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
@@ -115,11 +115,11 @@ static int pxp_terminate_arb_session_and_global(struct intel_pxp *pxp)
 	return ret;
 }
 
-static void pxp_terminate(struct intel_pxp *pxp)
+void intel_pxp_terminate(struct intel_pxp *pxp, bool post_invalidation_needs_restart)
 {
 	int ret;
 
-	pxp->hw_state_invalidated = true;
+	pxp->hw_state_invalidated = post_invalidation_needs_restart;
 
 	/*
 	 * if we fail to submit the termination there is no point in waiting for
@@ -167,7 +167,7 @@ static void pxp_session_work(struct work_struct *work)
 
 	if (events & PXP_TERMINATION_REQUEST) {
 		events &= ~PXP_TERMINATION_COMPLETE;
-		pxp_terminate(pxp);
+		intel_pxp_terminate(pxp, true);
 	}
 
 	if (events & PXP_TERMINATION_COMPLETE)
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
index 903ac52cffa1..ba5788127109 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
@@ -12,9 +12,14 @@ struct intel_pxp;
 
 #ifdef CONFIG_DRM_I915_PXP
 void intel_pxp_session_management_init(struct intel_pxp *pxp);
+void intel_pxp_terminate(struct intel_pxp *pxp, bool post_invalidation_needs_restart);
 #else
 static inline void intel_pxp_session_management_init(struct intel_pxp *pxp)
 {
 }
+
+static inline void intel_pxp_terminate(struct intel_pxp *pxp, bool post_invalidation_needs_restart)
+{
+}
 #endif
 #endif /* __INTEL_PXP_SESSION_H__ */
-- 
2.39.0

[PATCH v6 6/6] drm/i915/pxp: Pxp hw init should be in resume_complete

[Alan Previn]

During suspend flow, i915 currently achors' on the pm_suspend_prepare
callback as the location where we quiesce the entire GPU and perform
all necessary cleanup in order to go into suspend. PXP is also called
during this time to perform the arbitration session teardown (with
the assurance no additional GEM IOCTLs will come after that could
restart the session).

However, if other devices or drivers fail their suspend_prepare, the
system will not go into suspend and i915 will be expected to resume
operation. In this case, we need to re-initialize the PXP hardware
and this really should be done within the pm_resume_complete callback
which is the correct opposing function in the resume sequence to
match pm_suspend_prepare of the suspend sequence.

Because this callback is the last thing at the end of resuming
we expect little to no impact to the rest of the i915 resume sequence
with this change.

Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Acked-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
---
 drivers/gpu/drm/i915/i915_driver.c      | 20 ++++++++++++++++++--
 drivers/gpu/drm/i915/pxp/intel_pxp_pm.c |  2 +-
 drivers/gpu/drm/i915/pxp/intel_pxp_pm.h |  6 +++---
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_driver.c b/drivers/gpu/drm/i915/i915_driver.c
index 6469c7c1e154..122736b48938 100644
--- a/drivers/gpu/drm/i915/i915_driver.c
+++ b/drivers/gpu/drm/i915/i915_driver.c
@@ -1167,6 +1167,13 @@ static bool suspend_to_idle(struct drm_i915_private *dev_priv)
 	return false;
 }
 
+static void i915_drm_complete(struct drm_device *dev)
+{
+	struct drm_i915_private *i915 = to_i915(dev);
+
+	intel_pxp_resume_complete(i915->pxp);
+}
+
 static int i915_drm_prepare(struct drm_device *dev)
 {
 	struct drm_i915_private *i915 = to_i915(dev);
@@ -1367,8 +1374,6 @@ static int i915_drm_resume(struct drm_device *dev)
 
 	i915_gem_resume(dev_priv);
 
-	intel_pxp_resume(dev_priv->pxp);
-
 	intel_modeset_init_hw(dev_priv);
 	intel_init_clock_gating(dev_priv);
 	intel_hpd_init(dev_priv);
@@ -1560,6 +1565,16 @@ static int i915_pm_resume(struct device *kdev)
 	return i915_drm_resume(&i915->drm);
 }
 
+static void i915_pm_complete(struct device *kdev)
+{
+	struct drm_i915_private *i915 = kdev_to_i915(kdev);
+
+	if (i915->drm.switch_power_state == DRM_SWITCH_POWER_OFF)
+		return;
+
+	i915_drm_complete(&i915->drm);
+}
+
 /* freeze: before creating the hibernation_image */
 static int i915_pm_freeze(struct device *kdev)
 {
@@ -1780,6 +1795,7 @@ const struct dev_pm_ops i915_pm_ops = {
 	.suspend_late = i915_pm_suspend_late,
 	.resume_early = i915_pm_resume_early,
 	.resume = i915_pm_resume,
+	.complete = i915_pm_complete,
 
 	/*
 	 * S4 event handlers
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
index e427464aa131..4f836b317424 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
@@ -34,7 +34,7 @@ void intel_pxp_suspend(struct intel_pxp *pxp)
 	}
 }
 
-void intel_pxp_resume(struct intel_pxp *pxp)
+void intel_pxp_resume_complete(struct intel_pxp *pxp)
 {
 	if (!intel_pxp_is_enabled(pxp))
 		return;
diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.h b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.h
index 586be769104f..06b46f535b42 100644
--- a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.h
+++ b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.h
@@ -11,7 +11,7 @@ struct intel_pxp;
 #ifdef CONFIG_DRM_I915_PXP
 void intel_pxp_suspend_prepare(struct intel_pxp *pxp);
 void intel_pxp_suspend(struct intel_pxp *pxp);
-void intel_pxp_resume(struct intel_pxp *pxp);
+void intel_pxp_resume_complete(struct intel_pxp *pxp);
 void intel_pxp_runtime_suspend(struct intel_pxp *pxp);
 #else
 static inline void intel_pxp_suspend_prepare(struct intel_pxp *pxp)
@@ -22,7 +22,7 @@ static inline void intel_pxp_suspend(struct intel_pxp *pxp)
 {
 }
 
-static inline void intel_pxp_resume(struct intel_pxp *pxp)
+static inline void intel_pxp_resume_complete(struct intel_pxp *pxp)
 {
 }
 
@@ -32,6 +32,6 @@ static inline void intel_pxp_runtime_suspend(struct intel_pxp *pxp)
 #endif
 static inline void intel_pxp_runtime_resume(struct intel_pxp *pxp)
 {
-	intel_pxp_resume(pxp);
+	intel_pxp_resume_complete(pxp);
 }
 #endif /* __INTEL_PXP_PM_H__ */
-- 
2.39.0

Re: [PATCH v6 2/6] drm/i915/pxp: add device link between i915 and mei_pxp

[Rodrigo Vivi]

On Mon, Jan 23, 2023 at 09:31:46PM -0800, Alan Previn wrote:
> From: Alexander Usyskin <alexander.usyskin@intel.com>
> 
> Add device link with i915 as consumer and mei_pxp as supplier
> to ensure proper ordering of power flows.
> 
> V2: condition on absence of heci_pxp to filter out DG
> 
> Signed-off-by: Alexander Usyskin <alexander.usyskin@intel.com>
> ---
>  drivers/gpu/drm/i915/pxp/intel_pxp_tee.c   | 11 +++++++++++
>  drivers/gpu/drm/i915/pxp/intel_pxp_types.h |  6 ++++++
>  2 files changed, 17 insertions(+)
> 
> diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
> index 73aa8015f828..cd5b86216506 100644
> --- a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
> +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
> @@ -127,6 +127,12 @@ static int i915_pxp_tee_component_bind(struct device *i915_kdev,
>  	intel_wakeref_t wakeref;
>  	int ret = 0;
>  
> +	if (!HAS_HECI_PXP(i915)) {
> +		pxp->component_dev_link = device_link_add(i915_kdev, tee_kdev, DL_FLAG_STATELESS);
> +		if (drm_WARN_ON(&i915->drm, !pxp->component_dev_link))
> +			return -ENODEV;
> +	}
> +
>  	mutex_lock(&pxp->tee_mutex);
>  	pxp->pxp_component = data;
>  	pxp->pxp_component->tee_dev = tee_kdev;
> @@ -169,6 +175,11 @@ static void i915_pxp_tee_component_unbind(struct device *i915_kdev,
>  	mutex_lock(&pxp->tee_mutex);
>  	pxp->pxp_component = NULL;
>  	mutex_unlock(&pxp->tee_mutex);
> +
> +	if (pxp->component_dev_link) {
> +		device_link_remove(i915_kdev, tee_kdev);
> +		pxp->component_dev_link = NULL;
> +	}

use the 'del' version instead of the 'remove' one.

if (pxp->dev_link)
	device_link_del(pxp->dev_link);

>  }
>  
>  static const struct component_ops i915_pxp_tee_component_ops = {
> diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_types.h b/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
> index 7dc5f08d1583..efd2f3915abe 100644
> --- a/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
> +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
> @@ -32,6 +32,12 @@ struct intel_pxp {
>  	 * which are protected by &tee_mutex.
>  	 */
>  	struct i915_pxp_component *pxp_component;
> +
> +	/**
> +	 * @component_dev_link: device link of the pxp-component enforcing i915 as the
> +	 * consumer. This is needed for legacy platform (TGL/ADL) full-feature usage.

No need to add platform information here.

What about something shorter:

/* @dev_link: Enforce module relationship for power management ordering. */

> +	 */
> +	struct device_link *component_dev_link;

What about
     struct device_link *dev_link;

'component' is already part of the pxp struct.

>  	/**
>  	 * @pxp_component_added: track if the pxp component has been added.
>  	 * Set and cleared in tee init and fini functions respectively.
> -- 
> 2.39.0
> 

Re: [PATCH v6 5/6] drm/i915/pxp: Trigger the global teardown for before suspending

[Rodrigo Vivi]

On Mon, Jan 23, 2023 at 09:31:49PM -0800, Alan Previn wrote:
> A driver bug was recently discovered where the security firmware was
> receiving internal HW signals indicating that session key expirations
> had occurred. Architecturally, the firmware was expecting a response
> from the GuC to acknowledge the event with the firmware side.
> However the OS was in a suspended state and GuC had been reset.
> 
> Internal specifications actually required the driver to ensure
> that all active sessions be properly cleaned up in such cases where
> the system is suspended and the GuC potentially unable to respond.
> 
> This patch adds the global teardown code in i915's suspend_prepare
> code path.
> 
> Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
> Reviewed-by: Juston Li <justonli@chromium.org>

it should probably contain some revision history above so the
reviewer can know what changed from the time he reviewed and now.

it looks we have another function that I don't remember...
Juston, could you please confirm your review stands in this version?
if so feel free to add my ack with that.

> ---
>  drivers/gpu/drm/i915/pxp/intel_pxp.c         | 65 +++++++++++++++++---
>  drivers/gpu/drm/i915/pxp/intel_pxp.h         |  1 +
>  drivers/gpu/drm/i915/pxp/intel_pxp_pm.c      |  2 +-
>  drivers/gpu/drm/i915/pxp/intel_pxp_session.c |  6 +-
>  drivers/gpu/drm/i915/pxp/intel_pxp_session.h |  5 ++
>  5 files changed, 66 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.c b/drivers/gpu/drm/i915/pxp/intel_pxp.c
> index cfc9af8b3d21..9d4c7724e98e 100644
> --- a/drivers/gpu/drm/i915/pxp/intel_pxp.c
> +++ b/drivers/gpu/drm/i915/pxp/intel_pxp.c
> @@ -270,6 +270,60 @@ static bool pxp_component_bound(struct intel_pxp *pxp)
>  	return bound;
>  }
>  
> +static int __pxp_global_teardown_final(struct intel_pxp *pxp)
> +{
> +	if (!pxp->arb_is_valid)
> +		return 0;
> +	/*
> +	 * To ensure synchronous and coherent session teardown completion
> +	 * in response to suspend or shutdown triggers, don't use a worker.
> +	 */
> +	intel_pxp_mark_termination_in_progress(pxp);
> +	intel_pxp_terminate(pxp, false);
> +
> +	if (!wait_for_completion_timeout(&pxp->termination, msecs_to_jiffies(250)))
> +		return -ETIMEDOUT;
> +
> +	return 0;
> +}
> +
> +static int __pxp_global_teardown_restart(struct intel_pxp *pxp)
> +{
> +	if (pxp->arb_is_valid)
> +		return 0;
> +	/*
> +	 * The arb-session is currently inactive and we are doing a reset and restart
> +	 * due to a runtime event. Use the worker that was designed for this.
> +	 */
> +	pxp_queue_termination(pxp);
> +
> +	if (!wait_for_completion_timeout(&pxp->termination, msecs_to_jiffies(250)))
> +		return -ETIMEDOUT;
> +
> +	return 0;
> +}
> +
> +void intel_pxp_end(struct intel_pxp *pxp)
> +{
> +	struct drm_i915_private *i915 = pxp->ctrl_gt->i915;
> +	intel_wakeref_t wakeref;
> +
> +	if (!intel_pxp_is_enabled(pxp))
> +		return;
> +
> +	wakeref = intel_runtime_pm_get(&i915->runtime_pm);
> +
> +	mutex_lock(&pxp->arb_mutex);
> +
> +	if (__pxp_global_teardown_final(pxp))
> +		drm_dbg(&i915->drm, "PXP end timed out\n");
> +
> +	mutex_unlock(&pxp->arb_mutex);
> +
> +	intel_pxp_fini_hw(pxp);
> +	intel_runtime_pm_put(&i915->runtime_pm, wakeref);
> +}
> +
>  /*
>   * the arb session is restarted from the irq work when we receive the
>   * termination completion interrupt
> @@ -286,16 +340,9 @@ int intel_pxp_start(struct intel_pxp *pxp)
>  
>  	mutex_lock(&pxp->arb_mutex);
>  
> -	if (pxp->arb_is_valid)
> -		goto unlock;
> -
> -	pxp_queue_termination(pxp);
> -
> -	if (!wait_for_completion_timeout(&pxp->termination,
> -					msecs_to_jiffies(250))) {
> -		ret = -ETIMEDOUT;
> +	ret = __pxp_global_teardown_restart(pxp);
> +	if (ret)
>  		goto unlock;
> -	}
>  
>  	/* make sure the compiler doesn't optimize the double access */
>  	barrier();
> diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.h b/drivers/gpu/drm/i915/pxp/intel_pxp.h
> index 9658d3005222..3ded0890cd27 100644
> --- a/drivers/gpu/drm/i915/pxp/intel_pxp.h
> +++ b/drivers/gpu/drm/i915/pxp/intel_pxp.h
> @@ -27,6 +27,7 @@ void intel_pxp_mark_termination_in_progress(struct intel_pxp *pxp);
>  void intel_pxp_tee_end_arb_fw_session(struct intel_pxp *pxp, u32 arb_session_id);
>  
>  int intel_pxp_start(struct intel_pxp *pxp);
> +void intel_pxp_end(struct intel_pxp *pxp);
>  
>  int intel_pxp_key_check(struct intel_pxp *pxp,
>  			struct drm_i915_gem_object *obj,
> diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> index 892d39cc61c1..e427464aa131 100644
> --- a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> @@ -16,7 +16,7 @@ void intel_pxp_suspend_prepare(struct intel_pxp *pxp)
>  	if (!intel_pxp_is_enabled(pxp))
>  		return;
>  
> -	pxp->arb_is_valid = false;
> +	intel_pxp_end(pxp);
>  
>  	intel_pxp_invalidate(pxp);
>  }
> diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> index 74ed7e16e481..448cacb0465d 100644
> --- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> @@ -115,11 +115,11 @@ static int pxp_terminate_arb_session_and_global(struct intel_pxp *pxp)
>  	return ret;
>  }
>  
> -static void pxp_terminate(struct intel_pxp *pxp)
> +void intel_pxp_terminate(struct intel_pxp *pxp, bool post_invalidation_needs_restart)
>  {
>  	int ret;
>  
> -	pxp->hw_state_invalidated = true;
> +	pxp->hw_state_invalidated = post_invalidation_needs_restart;
>  
>  	/*
>  	 * if we fail to submit the termination there is no point in waiting for
> @@ -167,7 +167,7 @@ static void pxp_session_work(struct work_struct *work)
>  
>  	if (events & PXP_TERMINATION_REQUEST) {
>  		events &= ~PXP_TERMINATION_COMPLETE;
> -		pxp_terminate(pxp);
> +		intel_pxp_terminate(pxp, true);
>  	}
>  
>  	if (events & PXP_TERMINATION_COMPLETE)
> diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> index 903ac52cffa1..ba5788127109 100644
> --- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> @@ -12,9 +12,14 @@ struct intel_pxp;
>  
>  #ifdef CONFIG_DRM_I915_PXP
>  void intel_pxp_session_management_init(struct intel_pxp *pxp);
> +void intel_pxp_terminate(struct intel_pxp *pxp, bool post_invalidation_needs_restart);
>  #else
>  static inline void intel_pxp_session_management_init(struct intel_pxp *pxp)
>  {
>  }
> +
> +static inline void intel_pxp_terminate(struct intel_pxp *pxp, bool post_invalidation_needs_restart)
> +{
> +}
>  #endif
>  #endif /* __INTEL_PXP_SESSION_H__ */
> -- 
> 2.39.0
> 

Re: [PATCH v6 5/6] drm/i915/pxp: Trigger the global teardown for before suspending

[Juston Li]

On Tue, 2023-01-24 at 10:17 -0500, Rodrigo Vivi wrote:
> On Mon, Jan 23, 2023 at 09:31:49PM -0800, Alan Previn wrote:
> > A driver bug was recently discovered where the security firmware
> > was
> > receiving internal HW signals indicating that session key
> > expirations
> > had occurred. Architecturally, the firmware was expecting a
> > response
> > from the GuC to acknowledge the event with the firmware side.
> > However the OS was in a suspended state and GuC had been reset.
> > 
> > Internal specifications actually required the driver to ensure
> > that all active sessions be properly cleaned up in such cases where
> > the system is suspended and the GuC potentially unable to respond.
> > 
> > This patch adds the global teardown code in i915's suspend_prepare
> > code path.
> > 
> > Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
> > Reviewed-by: Juston Li <justonli@chromium.org>
> 
> it should probably contain some revision history above so the
> reviewer can know what changed from the time he reviewed and now.
> 
> it looks we have another function that I don't remember...
> Juston, could you please confirm your review stands in this version?
> if so feel free to add my ack with that.

Sure, v6 is:
Reviewed-by: Juston Li <justonli@chromium.org>

Appreciate the reviews/revisions Rodrigo and Alan!

> 
> > ---
> >  drivers/gpu/drm/i915/pxp/intel_pxp.c         | 65
> > +++++++++++++++++---
> >  drivers/gpu/drm/i915/pxp/intel_pxp.h         |  1 +
> >  drivers/gpu/drm/i915/pxp/intel_pxp_pm.c      |  2 +-
> >  drivers/gpu/drm/i915/pxp/intel_pxp_session.c |  6 +-
> >  drivers/gpu/drm/i915/pxp/intel_pxp_session.h |  5 ++
> >  5 files changed, 66 insertions(+), 13 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.c
> > b/drivers/gpu/drm/i915/pxp/intel_pxp.c
> > index cfc9af8b3d21..9d4c7724e98e 100644
> > --- a/drivers/gpu/drm/i915/pxp/intel_pxp.c
> > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp.c
> > @@ -270,6 +270,60 @@ static bool pxp_component_bound(struct
> > intel_pxp *pxp)
> >         return bound;
> >  }
> >  
> > +static int __pxp_global_teardown_final(struct intel_pxp *pxp)
> > +{
> > +       if (!pxp->arb_is_valid)
> > +               return 0;
> > +       /*
> > +        * To ensure synchronous and coherent session teardown
> > completion
> > +        * in response to suspend or shutdown triggers, don't use a
> > worker.
> > +        */
> > +       intel_pxp_mark_termination_in_progress(pxp);
> > +       intel_pxp_terminate(pxp, false);
> > +
> > +       if (!wait_for_completion_timeout(&pxp->termination,
> > msecs_to_jiffies(250)))
> > +               return -ETIMEDOUT;
> > +
> > +       return 0;
> > +}
> > +
> > +static int __pxp_global_teardown_restart(struct intel_pxp *pxp)
> > +{
> > +       if (pxp->arb_is_valid)
> > +               return 0;
> > +       /*
> > +        * The arb-session is currently inactive and we are doing a
> > reset and restart
> > +        * due to a runtime event. Use the worker that was designed
> > for this.
> > +        */
> > +       pxp_queue_termination(pxp);
> > +
> > +       if (!wait_for_completion_timeout(&pxp->termination,
> > msecs_to_jiffies(250)))
> > +               return -ETIMEDOUT;
> > +
> > +       return 0;
> > +}
> > +
> > +void intel_pxp_end(struct intel_pxp *pxp)
> > +{
> > +       struct drm_i915_private *i915 = pxp->ctrl_gt->i915;
> > +       intel_wakeref_t wakeref;
> > +
> > +       if (!intel_pxp_is_enabled(pxp))
> > +               return;
> > +
> > +       wakeref = intel_runtime_pm_get(&i915->runtime_pm);
> > +
> > +       mutex_lock(&pxp->arb_mutex);
> > +
> > +       if (__pxp_global_teardown_final(pxp))
> > +               drm_dbg(&i915->drm, "PXP end timed out\n");
> > +
> > +       mutex_unlock(&pxp->arb_mutex);
> > +
> > +       intel_pxp_fini_hw(pxp);
> > +       intel_runtime_pm_put(&i915->runtime_pm, wakeref);
> > +}
> > +
> >  /*
> >   * the arb session is restarted from the irq work when we receive
> > the
> >   * termination completion interrupt
> > @@ -286,16 +340,9 @@ int intel_pxp_start(struct intel_pxp *pxp)
> >  
> >         mutex_lock(&pxp->arb_mutex);
> >  
> > -       if (pxp->arb_is_valid)
> > -               goto unlock;
> > -
> > -       pxp_queue_termination(pxp);
> > -
> > -       if (!wait_for_completion_timeout(&pxp->termination,
> > -                                       msecs_to_jiffies(250))) {
> > -               ret = -ETIMEDOUT;
> > +       ret = __pxp_global_teardown_restart(pxp);
> > +       if (ret)
> >                 goto unlock;
> > -       }
> >  
> >         /* make sure the compiler doesn't optimize the double
> > access */
> >         barrier();
> > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.h
> > b/drivers/gpu/drm/i915/pxp/intel_pxp.h
> > index 9658d3005222..3ded0890cd27 100644
> > --- a/drivers/gpu/drm/i915/pxp/intel_pxp.h
> > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp.h
> > @@ -27,6 +27,7 @@ void
> > intel_pxp_mark_termination_in_progress(struct intel_pxp *pxp);
> >  void intel_pxp_tee_end_arb_fw_session(struct intel_pxp *pxp, u32
> > arb_session_id);
> >  
> >  int intel_pxp_start(struct intel_pxp *pxp);
> > +void intel_pxp_end(struct intel_pxp *pxp);
> >  
> >  int intel_pxp_key_check(struct intel_pxp *pxp,
> >                         struct drm_i915_gem_object *obj,
> > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> > b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> > index 892d39cc61c1..e427464aa131 100644
> > --- a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> > @@ -16,7 +16,7 @@ void intel_pxp_suspend_prepare(struct intel_pxp
> > *pxp)
> >         if (!intel_pxp_is_enabled(pxp))
> >                 return;
> >  
> > -       pxp->arb_is_valid = false;
> > +       intel_pxp_end(pxp);
> >  
> >         intel_pxp_invalidate(pxp);
> >  }
> > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> > b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> > index 74ed7e16e481..448cacb0465d 100644
> > --- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> > @@ -115,11 +115,11 @@ static int
> > pxp_terminate_arb_session_and_global(struct intel_pxp *pxp)
> >         return ret;
> >  }
> >  
> > -static void pxp_terminate(struct intel_pxp *pxp)
> > +void intel_pxp_terminate(struct intel_pxp *pxp, bool
> > post_invalidation_needs_restart)
> >  {
> >         int ret;
> >  
> > -       pxp->hw_state_invalidated = true;
> > +       pxp->hw_state_invalidated =
> > post_invalidation_needs_restart;
> >  
> >         /*
> >          * if we fail to submit the termination there is no point
> > in waiting for
> > @@ -167,7 +167,7 @@ static void pxp_session_work(struct work_struct
> > *work)
> >  
> >         if (events & PXP_TERMINATION_REQUEST) {
> >                 events &= ~PXP_TERMINATION_COMPLETE;
> > -               pxp_terminate(pxp);
> > +               intel_pxp_terminate(pxp, true);
> >         }
> >  
> >         if (events & PXP_TERMINATION_COMPLETE)
> > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> > b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> > index 903ac52cffa1..ba5788127109 100644
> > --- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> > @@ -12,9 +12,14 @@ struct intel_pxp;
> >  
> >  #ifdef CONFIG_DRM_I915_PXP
> >  void intel_pxp_session_management_init(struct intel_pxp *pxp);
> > +void intel_pxp_terminate(struct intel_pxp *pxp, bool
> > post_invalidation_needs_restart);
> >  #else
> >  static inline void intel_pxp_session_management_init(struct
> > intel_pxp *pxp)
> >  {
> >  }
> > +
> > +static inline void intel_pxp_terminate(struct intel_pxp *pxp, bool
> > post_invalidation_needs_restart)
> > +{
> > +}
> >  #endif
> >  #endif /* __INTEL_PXP_SESSION_H__ */
> > -- 
> > 2.39.0
> > 

Re: [Intel-gfx] [PATCH v6 5/6] drm/i915/pxp: Trigger the global teardown for before suspending

[Rodrigo Vivi]

On Tue, Jan 24, 2023 at 10:08:52AM -0800, Juston Li wrote:
> On Tue, 2023-01-24 at 10:17 -0500, Rodrigo Vivi wrote:
> > On Mon, Jan 23, 2023 at 09:31:49PM -0800, Alan Previn wrote:
> > > A driver bug was recently discovered where the security firmware
> > > was
> > > receiving internal HW signals indicating that session key
> > > expirations
> > > had occurred. Architecturally, the firmware was expecting a
> > > response
> > > from the GuC to acknowledge the event with the firmware side.
> > > However the OS was in a suspended state and GuC had been reset.
> > > 
> > > Internal specifications actually required the driver to ensure
> > > that all active sessions be properly cleaned up in such cases where
> > > the system is suspended and the GuC potentially unable to respond.
> > > 
> > > This patch adds the global teardown code in i915's suspend_prepare
> > > code path.
> > > 
> > > Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
> > > Reviewed-by: Juston Li <justonli@chromium.org>
> > 
> > it should probably contain some revision history above so the
> > reviewer can know what changed from the time he reviewed and now.
> > 
> > it looks we have another function that I don't remember...
> > Juston, could you please confirm your review stands in this version?
> > if so feel free to add my ack with that.
> 
> Sure, v6 is:
> Reviewed-by: Juston Li <justonli@chromium.org>


Acked-by: Rodrigo Vivi <rodrigo.vivi@intel.com>


> 
> Appreciate the reviews/revisions Rodrigo and Alan!
> 
> > 
> > > ---
> > >  drivers/gpu/drm/i915/pxp/intel_pxp.c         | 65
> > > +++++++++++++++++---
> > >  drivers/gpu/drm/i915/pxp/intel_pxp.h         |  1 +
> > >  drivers/gpu/drm/i915/pxp/intel_pxp_pm.c      |  2 +-
> > >  drivers/gpu/drm/i915/pxp/intel_pxp_session.c |  6 +-
> > >  drivers/gpu/drm/i915/pxp/intel_pxp_session.h |  5 ++
> > >  5 files changed, 66 insertions(+), 13 deletions(-)
> > > 
> > > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.c
> > > b/drivers/gpu/drm/i915/pxp/intel_pxp.c
> > > index cfc9af8b3d21..9d4c7724e98e 100644
> > > --- a/drivers/gpu/drm/i915/pxp/intel_pxp.c
> > > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp.c
> > > @@ -270,6 +270,60 @@ static bool pxp_component_bound(struct
> > > intel_pxp *pxp)
> > >         return bound;
> > >  }
> > >  
> > > +static int __pxp_global_teardown_final(struct intel_pxp *pxp)
> > > +{
> > > +       if (!pxp->arb_is_valid)
> > > +               return 0;
> > > +       /*
> > > +        * To ensure synchronous and coherent session teardown
> > > completion
> > > +        * in response to suspend or shutdown triggers, don't use a
> > > worker.
> > > +        */
> > > +       intel_pxp_mark_termination_in_progress(pxp);
> > > +       intel_pxp_terminate(pxp, false);
> > > +
> > > +       if (!wait_for_completion_timeout(&pxp->termination,
> > > msecs_to_jiffies(250)))
> > > +               return -ETIMEDOUT;
> > > +
> > > +       return 0;
> > > +}
> > > +
> > > +static int __pxp_global_teardown_restart(struct intel_pxp *pxp)
> > > +{
> > > +       if (pxp->arb_is_valid)
> > > +               return 0;
> > > +       /*
> > > +        * The arb-session is currently inactive and we are doing a
> > > reset and restart
> > > +        * due to a runtime event. Use the worker that was designed
> > > for this.
> > > +        */
> > > +       pxp_queue_termination(pxp);
> > > +
> > > +       if (!wait_for_completion_timeout(&pxp->termination,
> > > msecs_to_jiffies(250)))
> > > +               return -ETIMEDOUT;
> > > +
> > > +       return 0;
> > > +}
> > > +
> > > +void intel_pxp_end(struct intel_pxp *pxp)
> > > +{
> > > +       struct drm_i915_private *i915 = pxp->ctrl_gt->i915;
> > > +       intel_wakeref_t wakeref;
> > > +
> > > +       if (!intel_pxp_is_enabled(pxp))
> > > +               return;
> > > +
> > > +       wakeref = intel_runtime_pm_get(&i915->runtime_pm);
> > > +
> > > +       mutex_lock(&pxp->arb_mutex);
> > > +
> > > +       if (__pxp_global_teardown_final(pxp))
> > > +               drm_dbg(&i915->drm, "PXP end timed out\n");
> > > +
> > > +       mutex_unlock(&pxp->arb_mutex);
> > > +
> > > +       intel_pxp_fini_hw(pxp);
> > > +       intel_runtime_pm_put(&i915->runtime_pm, wakeref);
> > > +}
> > > +
> > >  /*
> > >   * the arb session is restarted from the irq work when we receive
> > > the
> > >   * termination completion interrupt
> > > @@ -286,16 +340,9 @@ int intel_pxp_start(struct intel_pxp *pxp)
> > >  
> > >         mutex_lock(&pxp->arb_mutex);
> > >  
> > > -       if (pxp->arb_is_valid)
> > > -               goto unlock;
> > > -
> > > -       pxp_queue_termination(pxp);
> > > -
> > > -       if (!wait_for_completion_timeout(&pxp->termination,
> > > -                                       msecs_to_jiffies(250))) {
> > > -               ret = -ETIMEDOUT;
> > > +       ret = __pxp_global_teardown_restart(pxp);
> > > +       if (ret)
> > >                 goto unlock;
> > > -       }
> > >  
> > >         /* make sure the compiler doesn't optimize the double
> > > access */
> > >         barrier();
> > > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.h
> > > b/drivers/gpu/drm/i915/pxp/intel_pxp.h
> > > index 9658d3005222..3ded0890cd27 100644
> > > --- a/drivers/gpu/drm/i915/pxp/intel_pxp.h
> > > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp.h
> > > @@ -27,6 +27,7 @@ void
> > > intel_pxp_mark_termination_in_progress(struct intel_pxp *pxp);
> > >  void intel_pxp_tee_end_arb_fw_session(struct intel_pxp *pxp, u32
> > > arb_session_id);
> > >  
> > >  int intel_pxp_start(struct intel_pxp *pxp);
> > > +void intel_pxp_end(struct intel_pxp *pxp);
> > >  
> > >  int intel_pxp_key_check(struct intel_pxp *pxp,
> > >                         struct drm_i915_gem_object *obj,
> > > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> > > b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> > > index 892d39cc61c1..e427464aa131 100644
> > > --- a/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> > > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_pm.c
> > > @@ -16,7 +16,7 @@ void intel_pxp_suspend_prepare(struct intel_pxp
> > > *pxp)
> > >         if (!intel_pxp_is_enabled(pxp))
> > >                 return;
> > >  
> > > -       pxp->arb_is_valid = false;
> > > +       intel_pxp_end(pxp);
> > >  
> > >         intel_pxp_invalidate(pxp);
> > >  }
> > > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> > > b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> > > index 74ed7e16e481..448cacb0465d 100644
> > > --- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> > > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c
> > > @@ -115,11 +115,11 @@ static int
> > > pxp_terminate_arb_session_and_global(struct intel_pxp *pxp)
> > >         return ret;
> > >  }
> > >  
> > > -static void pxp_terminate(struct intel_pxp *pxp)
> > > +void intel_pxp_terminate(struct intel_pxp *pxp, bool
> > > post_invalidation_needs_restart)
> > >  {
> > >         int ret;
> > >  
> > > -       pxp->hw_state_invalidated = true;
> > > +       pxp->hw_state_invalidated =
> > > post_invalidation_needs_restart;
> > >  
> > >         /*
> > >          * if we fail to submit the termination there is no point
> > > in waiting for
> > > @@ -167,7 +167,7 @@ static void pxp_session_work(struct work_struct
> > > *work)
> > >  
> > >         if (events & PXP_TERMINATION_REQUEST) {
> > >                 events &= ~PXP_TERMINATION_COMPLETE;
> > > -               pxp_terminate(pxp);
> > > +               intel_pxp_terminate(pxp, true);
> > >         }
> > >  
> > >         if (events & PXP_TERMINATION_COMPLETE)
> > > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> > > b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> > > index 903ac52cffa1..ba5788127109 100644
> > > --- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> > > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h
> > > @@ -12,9 +12,14 @@ struct intel_pxp;
> > >  
> > >  #ifdef CONFIG_DRM_I915_PXP
> > >  void intel_pxp_session_management_init(struct intel_pxp *pxp);
> > > +void intel_pxp_terminate(struct intel_pxp *pxp, bool
> > > post_invalidation_needs_restart);
> > >  #else
> > >  static inline void intel_pxp_session_management_init(struct
> > > intel_pxp *pxp)
> > >  {
> > >  }
> > > +
> > > +static inline void intel_pxp_terminate(struct intel_pxp *pxp, bool
> > > post_invalidation_needs_restart)
> > > +{
> > > +}
> > >  #endif
> > >  #endif /* __INTEL_PXP_SESSION_H__ */
> > > -- 
> > > 2.39.0
> > > 
> 

Re: [Intel-gfx] [PATCH v6 5/6] drm/i915/pxp: Trigger the global teardown for before suspending

[Teres Alexis, Alan Previn]

On Tue, 2023-01-24 at 13:43 -0500, Vivi, Rodrigo wrote:
> On Tue, Jan 24, 2023 at 10:08:52AM -0800, Juston Li wrote:
> > On Tue, 2023-01-24 at 10:17 -0500, Rodrigo Vivi wrote:
> > > On Mon, Jan 23, 2023 at 09:31:49PM -0800, Alan Previn wrote:
> > > > A driver bug was recently discovered where the security firmware
> > > > was
> > > > receiving internal HW signals indicating that session key
> > > > expirations
> > > > had occurred. Architecturally, the firmware was expecting a
> > > > response
> > > > from the GuC to acknowledge the event with the firmware side.
> > > > However the OS was in a suspended state and GuC had been reset.
> > > > 
> > > > Internal specifications actually required the driver to ensure
> > > > that all active sessions be properly cleaned up in such cases where
> > > > the system is suspended and the GuC potentially unable to respond.
> > > > 
> > > > This patch adds the global teardown code in i915's suspend_prepare
> > > > code path.
> > > > 
> > > > Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
> > > > Reviewed-by: Juston Li <justonli@chromium.org>
> > > 
> > > it should probably contain some revision history above so the
> > > reviewer can know what changed from the time he reviewed and now.
> > > 
> > > it looks we have another function that I don't remember...
> > > Juston, could you please confirm your review stands in this version?
> > > if so feel free to add my ack with that.
> > 
> > Sure, v6 is:
> > Reviewed-by: Juston Li <justonli@chromium.org>
> 
> 
> Acked-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
> 
> 
> 
Thanks guys - will respin the final one (i hope) with tweaks on patch #2.
...alan

Re: [PATCH v6 2/6] drm/i915/pxp: add device link between i915 and mei_pxp

[Teres Alexis, Alan Previn]

Thanks Rodrigo - will do.

On Tue, 2023-01-24 at 10:10 -0500, Vivi, Rodrigo wrote:
> On Mon, Jan 23, 2023 at 09:31:46PM -0800, Alan Previn wrote:
> > From: Alexander Usyskin <alexander.usyskin@intel.com>
> > 
> > Add device link with i915 as consumer and mei_pxp as supplier
> > to ensure proper ordering of power flows.
> > 
> > V2: condition on absence of heci_pxp to filter out DG
> > 
> > Signed-off-by: Alexander Usyskin <alexander.usyskin@intel.com>
> > ---
> >  drivers/gpu/drm/i915/pxp/intel_pxp_tee.c   | 11 +++++++++++
> >  drivers/gpu/drm/i915/pxp/intel_pxp_types.h |  6 ++++++
> >  2 files changed, 17 insertions(+)
> > 
> > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
> > index 73aa8015f828..cd5b86216506 100644
> > --- a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
> > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c
> > @@ -127,6 +127,12 @@ static int i915_pxp_tee_component_bind(struct device *i915_kdev,
> >         intel_wakeref_t wakeref;
> >         int ret = 0;
> >  
> > +       if (!HAS_HECI_PXP(i915)) {
> > +               pxp->component_dev_link = device_link_add(i915_kdev, tee_kdev, DL_FLAG_STATELESS);
> > +               if (drm_WARN_ON(&i915->drm, !pxp->component_dev_link))
> > +                       return -ENODEV;
> > +       }
> > +
> >         mutex_lock(&pxp->tee_mutex);
> >         pxp->pxp_component = data;
> >         pxp->pxp_component->tee_dev = tee_kdev;
> > @@ -169,6 +175,11 @@ static void i915_pxp_tee_component_unbind(struct device *i915_kdev,
> >         mutex_lock(&pxp->tee_mutex);
> >         pxp->pxp_component = NULL;
> >         mutex_unlock(&pxp->tee_mutex);
> > +
> > +       if (pxp->component_dev_link) {
> > +               device_link_remove(i915_kdev, tee_kdev);
> > +               pxp->component_dev_link = NULL;
> > +       }
> 
> use the 'del' version instead of the 'remove' one.
> 
> if (pxp->dev_link)
>         device_link_del(pxp->dev_link);
> 
> >  }
> >  
> >  static const struct component_ops i915_pxp_tee_component_ops = {
> > diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_types.h b/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
> > index 7dc5f08d1583..efd2f3915abe 100644
> > --- a/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
> > +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_types.h
> > @@ -32,6 +32,12 @@ struct intel_pxp {
> >          * which are protected by &tee_mutex.
> >          */
> >         struct i915_pxp_component *pxp_component;
> > +
> > +       /**
> > +        * @component_dev_link: device link of the pxp-component enforcing i915 as the
> > +        * consumer. This is needed for legacy platform (TGL/ADL) full-feature usage.
> 
> No need to add platform information here.
> 
> What about something shorter:
> 
> /* @dev_link: Enforce module relationship for power management ordering. */
> 
> > +        */
> > +       struct device_link *component_dev_link;
> 
> What about
>      struct device_link *dev_link;
> 
> 'component' is already part of the pxp struct.
> 
> >         /**
> >          * @pxp_component_added: track if the pxp component has been added.
> >          * Set and cleared in tee init and fini functions respectively.
> > -- 
> > 2.39.0
> >