[PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Mario Limonciello]

Currently the Linux kernel will offer s2idle regardless of whether the FADT
indicates the system should use or on X86 if the LPS0 ACPI device has been
activated.

On some non-AMD platforms s2idle can be offered even without proper
firmware support.  The power consumption may be higher in these instances
but the system otherwise properly suspends and resumes.

On AMD platforms however when the FW has been configured not to offer
s2idle some different hardware initialization has occurred such that the
system won't properly resume.

This patch series changes the kernel suspend code to conditionally decide
whether to offer s2idle.  When an AMD system is encountered all the
prerequisites will be checked before s2idle is offered to the user.

Mario Limonciello (3):
  PM: suspend: Move some structure declarations around
  PM: sleep: Don't always assume s2idle will be enabled
  acpi: sleep: Don't offer s2idle on AMD platforms without LPS0

 drivers/acpi/sleep.c      |  6 ++++++
 drivers/acpi/x86/s2idle.c | 36 ++++++++++++++++++++++++----------
 include/linux/suspend.h   |  1 +
 kernel/power/suspend.c    | 41 ++++++++++++++++++++-------------------
 4 files changed, 54 insertions(+), 30 deletions(-)

-- 
2.25.1

[PATCH 1/3] PM: suspend: Move some structure declarations around

[Mario Limonciello]

Future patches will modify members of these structures dependent
upon these locations.  Move the structure declarations first to
make those patches easier to consume.

Reviewed-by: Basavaraj Natikar <Basavaraj.Natikar@amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 drivers/acpi/x86/s2idle.c | 20 ++++++++++----------
 kernel/power/suspend.c    | 14 +++++++-------
 2 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/acpi/x86/s2idle.c b/drivers/acpi/x86/s2idle.c
index 1c48358b43ba..32ab4ba5adb9 100644
--- a/drivers/acpi/x86/s2idle.c
+++ b/drivers/acpi/x86/s2idle.c
@@ -361,6 +361,16 @@ static int validate_dsm(acpi_handle handle, const char *uuid, int rev, guid_t *d
 	return ret;
 }
 
+static const struct platform_s2idle_ops acpi_s2idle_ops_lps0 = {
+	.begin = acpi_s2idle_begin,
+	.prepare = acpi_s2idle_prepare,
+	.prepare_late = acpi_s2idle_prepare_late,
+	.wake = acpi_s2idle_wake,
+	.restore_early = acpi_s2idle_restore_early,
+	.restore = acpi_s2idle_restore,
+	.end = acpi_s2idle_end,
+};
+
 static int lps0_device_attach(struct acpi_device *adev,
 			      const struct acpi_device_id *not_used)
 {
@@ -508,16 +518,6 @@ void acpi_s2idle_restore_early(void)
 					lps0_dsm_func_mask, lps0_dsm_guid);
 }
 
-static const struct platform_s2idle_ops acpi_s2idle_ops_lps0 = {
-	.begin = acpi_s2idle_begin,
-	.prepare = acpi_s2idle_prepare,
-	.prepare_late = acpi_s2idle_prepare_late,
-	.wake = acpi_s2idle_wake,
-	.restore_early = acpi_s2idle_restore_early,
-	.restore = acpi_s2idle_restore,
-	.end = acpi_s2idle_end,
-};
-
 void acpi_s2idle_setup(void)
 {
 	acpi_scan_add_handler(&lps0_handler);
diff --git a/kernel/power/suspend.c b/kernel/power/suspend.c
index 80cc1f0f502b..597c5e65aa45 100644
--- a/kernel/power/suspend.c
+++ b/kernel/power/suspend.c
@@ -73,13 +73,6 @@ bool pm_suspend_default_s2idle(void)
 }
 EXPORT_SYMBOL_GPL(pm_suspend_default_s2idle);
 
-void s2idle_set_ops(const struct platform_s2idle_ops *ops)
-{
-	lock_system_sleep();
-	s2idle_ops = ops;
-	unlock_system_sleep();
-}
-
 static void s2idle_begin(void)
 {
 	s2idle_state = S2IDLE_STATE_NONE;
@@ -169,6 +162,13 @@ static bool valid_state(suspend_state_t state)
 		suspend_ops->enter;
 }
 
+void s2idle_set_ops(const struct platform_s2idle_ops *ops)
+{
+	lock_system_sleep();
+	s2idle_ops = ops;
+	unlock_system_sleep();
+}
+
 void __init pm_states_init(void)
 {
 	/* "mem" and "freeze" are always present in /sys/power/state. */
-- 
2.25.1

[PATCH 2/3] PM: sleep: Don't always assume s2idle will be enabled

[Mario Limonciello]

Some platforms may require firmware support for s2idle to work
properly, so allow other drivers that set s2idle ops to decide whether
to offer it on a given platform.

This should be no functional change for any platform.

Reviewed-by: Basavaraj Natikar <Basavaraj.Natikar@amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 drivers/acpi/sleep.c      |  6 ++++++
 drivers/acpi/x86/s2idle.c |  6 ++++++
 include/linux/suspend.h   |  1 +
 kernel/power/suspend.c    | 27 ++++++++++++++-------------
 4 files changed, 27 insertions(+), 13 deletions(-)

diff --git a/drivers/acpi/sleep.c b/drivers/acpi/sleep.c
index eaa47753b758..9dafdaafa5dc 100644
--- a/drivers/acpi/sleep.c
+++ b/drivers/acpi/sleep.c
@@ -800,12 +800,18 @@ void acpi_s2idle_end(void)
 	acpi_scan_lock_release();
 }
 
+static bool acpi_s2idle_valid(void)
+{
+	return true;
+}
+
 static const struct platform_s2idle_ops acpi_s2idle_ops = {
 	.begin = acpi_s2idle_begin,
 	.prepare = acpi_s2idle_prepare,
 	.wake = acpi_s2idle_wake,
 	.restore = acpi_s2idle_restore,
 	.end = acpi_s2idle_end,
+	.valid = acpi_s2idle_valid,
 };
 
 void __weak acpi_s2idle_setup(void)
diff --git a/drivers/acpi/x86/s2idle.c b/drivers/acpi/x86/s2idle.c
index 32ab4ba5adb9..a1626737e5e0 100644
--- a/drivers/acpi/x86/s2idle.c
+++ b/drivers/acpi/x86/s2idle.c
@@ -361,6 +361,11 @@ static int validate_dsm(acpi_handle handle, const char *uuid, int rev, guid_t *d
 	return ret;
 }
 
+static bool acpi_s2idle_valid(void)
+{
+	return true;
+}
+
 static const struct platform_s2idle_ops acpi_s2idle_ops_lps0 = {
 	.begin = acpi_s2idle_begin,
 	.prepare = acpi_s2idle_prepare,
@@ -369,6 +374,7 @@ static const struct platform_s2idle_ops acpi_s2idle_ops_lps0 = {
 	.restore_early = acpi_s2idle_restore_early,
 	.restore = acpi_s2idle_restore,
 	.end = acpi_s2idle_end,
+	.valid = acpi_s2idle_valid,
 };
 
 static int lps0_device_attach(struct acpi_device *adev,
diff --git a/include/linux/suspend.h b/include/linux/suspend.h
index 8af13ba60c7e..51e5bdd9be23 100644
--- a/include/linux/suspend.h
+++ b/include/linux/suspend.h
@@ -195,6 +195,7 @@ struct platform_s2idle_ops {
 	void (*restore_early)(void);
 	void (*restore)(void);
 	void (*end)(void);
+	bool (*valid)(void);
 };
 
 #ifdef CONFIG_SUSPEND
diff --git a/kernel/power/suspend.c b/kernel/power/suspend.c
index 597c5e65aa45..29d6f1738359 100644
--- a/kernel/power/suspend.c
+++ b/kernel/power/suspend.c
@@ -46,7 +46,7 @@ static const char * const mem_sleep_labels[] = {
 };
 const char *mem_sleep_states[PM_SUSPEND_MAX];
 
-suspend_state_t mem_sleep_current = PM_SUSPEND_TO_IDLE;
+suspend_state_t mem_sleep_current = PM_SUSPEND_MAX;
 suspend_state_t mem_sleep_default = PM_SUSPEND_MAX;
 suspend_state_t pm_suspend_target_state;
 EXPORT_SYMBOL_GPL(pm_suspend_target_state);
@@ -152,6 +152,10 @@ EXPORT_SYMBOL_GPL(s2idle_wake);
 
 static bool valid_state(suspend_state_t state)
 {
+	/* PM_SUSPEND_TO_IDLE may require low-level support on some platforms */
+	if (state == PM_SUSPEND_TO_IDLE)
+		return s2idle_ops && s2idle_ops->valid && s2idle_ops->valid();
+
 	/*
 	 * The PM_SUSPEND_STANDBY and PM_SUSPEND_MEM states require low-level
 	 * support and need to be valid to the low-level implementation.
@@ -166,6 +170,13 @@ void s2idle_set_ops(const struct platform_s2idle_ops *ops)
 {
 	lock_system_sleep();
 	s2idle_ops = ops;
+	if (valid_state(PM_SUSPEND_TO_IDLE)) {
+		mem_sleep_states[PM_SUSPEND_TO_IDLE] = mem_sleep_labels[PM_SUSPEND_TO_IDLE];
+		/* if supported, update to suspend to idle for default when ops set */
+		if (mem_sleep_current == PM_SUSPEND_MAX ||
+		    mem_sleep_default == PM_SUSPEND_TO_IDLE)
+			mem_sleep_current = PM_SUSPEND_TO_IDLE;
+	}
 	unlock_system_sleep();
 }
 
@@ -174,11 +185,6 @@ void __init pm_states_init(void)
 	/* "mem" and "freeze" are always present in /sys/power/state. */
 	pm_states[PM_SUSPEND_MEM] = pm_labels[PM_SUSPEND_MEM];
 	pm_states[PM_SUSPEND_TO_IDLE] = pm_labels[PM_SUSPEND_TO_IDLE];
-	/*
-	 * Suspend-to-idle should be supported even without any suspend_ops,
-	 * initialize mem_sleep_states[] accordingly here.
-	 */
-	mem_sleep_states[PM_SUSPEND_TO_IDLE] = mem_sleep_labels[PM_SUSPEND_TO_IDLE];
 }
 
 static int __init mem_sleep_default_setup(char *str)
@@ -236,11 +242,6 @@ int suspend_valid_only_mem(suspend_state_t state)
 }
 EXPORT_SYMBOL_GPL(suspend_valid_only_mem);
 
-static bool sleep_state_supported(suspend_state_t state)
-{
-	return state == PM_SUSPEND_TO_IDLE || valid_state(state);
-}
-
 static int platform_suspend_prepare(suspend_state_t state)
 {
 	return state != PM_SUSPEND_TO_IDLE && suspend_ops->prepare ?
@@ -346,7 +347,7 @@ static int suspend_prepare(suspend_state_t state)
 {
 	int error;
 
-	if (!sleep_state_supported(state))
+	if (!valid_state(state))
 		return -EPERM;
 
 	pm_prepare_console();
@@ -478,7 +479,7 @@ int suspend_devices_and_enter(suspend_state_t state)
 	int error;
 	bool wakeup = false;
 
-	if (!sleep_state_supported(state))
+	if (!valid_state(state))
 		return -ENOSYS;
 
 	pm_suspend_target_state = state;
-- 
2.25.1

[PATCH 3/3] acpi: sleep: Don't offer s2idle on AMD platforms without LPS0

[Mario Limonciello]

On some OEM platforms a BIOS option is offered that will set the
sleep mode between S3 and S2idle.  This option will change certain HW
behaviors.  When in S2idle mode, Linux works properly. However when S3 mode
is picked but the user chooses S2idle in Linux the platform may not be
properly resumed.

To avoid users getting into this situation, don't offer s2idle on AMD
systems missing the LPS0 device either by a BIOS option or by using the
acpi "no_sleep_lps0" module parameter.

Reported-by: Bjoren Dasse <bjoern.daase@gmail.com>
BugLink: https://gitlab.freedesktop.org/drm/amd/-/issues/1824
BugLink: https://bugzilla.kernel.org/show_bug.cgi?id=215387
Reviewed-by: Basavaraj Natikar <Basavaraj.Natikar@amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 drivers/acpi/x86/s2idle.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/acpi/x86/s2idle.c b/drivers/acpi/x86/s2idle.c
index a1626737e5e0..c3d35a42ac2f 100644
--- a/drivers/acpi/x86/s2idle.c
+++ b/drivers/acpi/x86/s2idle.c
@@ -363,6 +363,13 @@ static int validate_dsm(acpi_handle handle, const char *uuid, int rev, guid_t *d
 
 static bool acpi_s2idle_valid(void)
 {
+	/* AMD systems must have low level firmware support */
+	if (acpi_s2idle_vendor_amd())
+#if IS_ENABLED(CONFIG_AMD_PMC)
+		return lps0_device_handle && !sleep_no_lps0;
+#else
+		return false;
+#endif
 	return true;
 }
 
@@ -450,6 +457,9 @@ static int lps0_device_attach(struct acpi_device *adev,
 	if (!acpi_s2idle_vendor_amd())
 		acpi_ec_mark_gpe_for_wake();
 
+	/* reset these so we update valid */
+	s2idle_set_ops(&acpi_s2idle_ops_lps0);
+
 	return 0;
 }
 
-- 
2.25.1

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Rafael J. Wysocki]

On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
<mario.limonciello@amd.com> wrote:
>
> Currently the Linux kernel will offer s2idle regardless of whether the FADT
> indicates the system should use or on X86 if the LPS0 ACPI device has been
> activated.
>
> On some non-AMD platforms s2idle can be offered even without proper
> firmware support.  The power consumption may be higher in these instances
> but the system otherwise properly suspends and resumes.

Well, the idea is that s2idle should not require FW support at all.

It may not be possible to reach the minimum power level of the
platform without FW support, but that should not prevent s2idle from
being used.

> On AMD platforms however when the FW has been configured not to offer
> s2idle some different hardware initialization has occurred such that the
> system won't properly resume.

That's rather unfortunate.

Can you please share some details on what's going on in those cases?

Technically, without FW support there should be no difference between
the platform state reachable via s2idle and the platform state
reachable via runtime idle.

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Limonciello, Mario]

+Alex

On 1/11/2022 09:52, Rafael J. Wysocki wrote:
> On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
> <mario.limonciello@amd.com> wrote:
>>
>> Currently the Linux kernel will offer s2idle regardless of whether the FADT
>> indicates the system should use or on X86 if the LPS0 ACPI device has been
>> activated.
>>
>> On some non-AMD platforms s2idle can be offered even without proper
>> firmware support.  The power consumption may be higher in these instances
>> but the system otherwise properly suspends and resumes.
> 
> Well, the idea is that s2idle should not require FW support at all. >

May I ask - why?  It's an intentional design decision?

> It may not be possible to reach the minimum power level of the
> platform without FW support, but that should not prevent s2idle from
> being used.
> 
>> On AMD platforms however when the FW has been configured not to offer
>> s2idle some different hardware initialization has occurred such that the
>> system won't properly resume.
> 
> That's rather unfortunate.
> 
> Can you please share some details on what's going on in those cases?
> 
> Technically, without FW support there should be no difference between
> the platform state reachable via s2idle and the platform state
> reachable via runtime idle.

During resume there is a number of page faults that occur and during 
initialization the ring tests fail.  The graphics is unusable at this 
time as a result.

The amdgpu code actually *does* distinguish between the 3 different 
cases of S3, S0ix, and runtime suspend.

The function "amdgpu_acpi_is_s0ix_active" causes different codepaths to 
be used during the suspend routine.

In this particular case that FADT doesn't set the low power idle bit
and that function returns false meaning the s3 codepath is taken but
the hardware didn't go through a reset.

It *might* also be possible to solve this by mandating an ASIC reset in 
such a case (we didn't try).

However it comes back to my first upleveveled question - is this a case 
we really want to support and encourage?  This type of bug and 
combination of codepaths is not a case that is going to be well tested. 
This patch series will align the kernel behavior to only what AMD validates.

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Rafael J. Wysocki]

On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario
<mario.limonciello@amd.com> wrote:
>
> +Alex
>
> On 1/11/2022 09:52, Rafael J. Wysocki wrote:
> > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
> > <mario.limonciello@amd.com> wrote:
> >>
> >> Currently the Linux kernel will offer s2idle regardless of whether the FADT
> >> indicates the system should use or on X86 if the LPS0 ACPI device has been
> >> activated.
> >>
> >> On some non-AMD platforms s2idle can be offered even without proper
> >> firmware support.  The power consumption may be higher in these instances
> >> but the system otherwise properly suspends and resumes.
> >
> > Well, the idea is that s2idle should not require FW support at all. >
>
> May I ask - why?  It's an intentional design decision?

Yes, it is.

> > It may not be possible to reach the minimum power level of the
> > platform without FW support, but that should not prevent s2idle from
> > being used.
> >
> >> On AMD platforms however when the FW has been configured not to offer
> >> s2idle some different hardware initialization has occurred such that the
> >> system won't properly resume.
> >
> > That's rather unfortunate.
> >
> > Can you please share some details on what's going on in those cases?
> >
> > Technically, without FW support there should be no difference between
> > the platform state reachable via s2idle and the platform state
> > reachable via runtime idle.
>
> During resume there is a number of page faults that occur and during
> initialization the ring tests fail.  The graphics is unusable at this
> time as a result.
>
> The amdgpu code actually *does* distinguish between the 3 different
> cases of S3, S0ix, and runtime suspend.

But s2idle doesn't guarantee S0ix in any case.

> The function "amdgpu_acpi_is_s0ix_active" causes different codepaths to
> be used during the suspend routine.

Well, as I said, s2idle need not mean S0ix.

> In this particular case that FADT doesn't set the low power idle bit
> and that function returns false meaning the s3 codepath is taken but
> the hardware didn't go through a reset.

If there is a separate S3 code path, taking it when
pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect.

> It *might* also be possible to solve this by mandating an ASIC reset in
> such a case (we didn't try).

I'd rather do a PM-runtime path equivalent if the target sleep state
is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix.

> However it comes back to my first upleveveled question - is this a case
> we really want to support and encourage?  This type of bug and
> combination of codepaths is not a case that is going to be well tested.
> This patch series will align the kernel behavior to only what AMD validates.

But this does not follow the definition of s2idle and its documentation.

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Limonciello, Mario]

On 1/11/2022 11:05, Rafael J. Wysocki wrote:
> On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario
> <mario.limonciello@amd.com> wrote:
>>
>> +Alex
>>
>> On 1/11/2022 09:52, Rafael J. Wysocki wrote:
>>> On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
>>> <mario.limonciello@amd.com> wrote:
>>>>
>>>> Currently the Linux kernel will offer s2idle regardless of whether the FADT
>>>> indicates the system should use or on X86 if the LPS0 ACPI device has been
>>>> activated.
>>>>
>>>> On some non-AMD platforms s2idle can be offered even without proper
>>>> firmware support.  The power consumption may be higher in these instances
>>>> but the system otherwise properly suspends and resumes.
>>>
>>> Well, the idea is that s2idle should not require FW support at all. >
>>
>> May I ask - why?  It's an intentional design decision?
> 
> Yes, it is.
> 
>>> It may not be possible to reach the minimum power level of the
>>> platform without FW support, but that should not prevent s2idle from
>>> being used.
>>>
>>>> On AMD platforms however when the FW has been configured not to offer
>>>> s2idle some different hardware initialization has occurred such that the
>>>> system won't properly resume.
>>>
>>> That's rather unfortunate.
>>>
>>> Can you please share some details on what's going on in those cases?
>>>
>>> Technically, without FW support there should be no difference between
>>> the platform state reachable via s2idle and the platform state
>>> reachable via runtime idle.
>>
>> During resume there is a number of page faults that occur and during
>> initialization the ring tests fail.  The graphics is unusable at this
>> time as a result.
>>
>> The amdgpu code actually *does* distinguish between the 3 different
>> cases of S3, S0ix, and runtime suspend.
> 
> But s2idle doesn't guarantee S0ix in any case.
> 
>> The function "amdgpu_acpi_is_s0ix_active" causes different codepaths to
>> be used during the suspend routine.
> 
> Well, as I said, s2idle need not mean S0ix.
> 
>> In this particular case that FADT doesn't set the low power idle bit
>> and that function returns false meaning the s3 codepath is taken but
>> the hardware didn't go through a reset.
> 
> If there is a separate S3 code path, taking it when
> pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect.
> 
>> It *might* also be possible to solve this by mandating an ASIC reset in
>> such a case (we didn't try).
> 
> I'd rather do a PM-runtime path equivalent if the target sleep state
> is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix.
> 
>> However it comes back to my first upleveveled question - is this a case
>> we really want to support and encourage?  This type of bug and
>> combination of codepaths is not a case that is going to be well tested.
>> This patch series will align the kernel behavior to only what AMD validates.
> 
> But this does not follow the definition of s2idle and its documentation.

Very well, we'll work out in amdgpu what we can do.  Please close the 
kernel bugzilla (I don't have permissions).

RE: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Deucher, Alexander]

[AMD Official Use Only]

> -----Original Message-----
> From: Rafael J. Wysocki <rafael@kernel.org>
> Sent: Tuesday, January 11, 2022 12:06 PM
> To: Limonciello, Mario <Mario.Limonciello@amd.com>
> Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki
> <rjw@rjwysocki.net>; ACPI Devel Maling List <linux-acpi@vger.kernel.org>;
> S-k, Shyam-sundar <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj
> <Basavaraj.Natikar@amd.com>; Deucher, Alexander
> <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com
> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW
> 
> On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario
> <mario.limonciello@amd.com> wrote:
> >
> > +Alex
> >
> > On 1/11/2022 09:52, Rafael J. Wysocki wrote:
> > > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
> > > <mario.limonciello@amd.com> wrote:
> > >>
> > >> Currently the Linux kernel will offer s2idle regardless of whether
> > >> the FADT indicates the system should use or on X86 if the LPS0 ACPI
> > >> device has been activated.
> > >>
> > >> On some non-AMD platforms s2idle can be offered even without
> proper
> > >> firmware support.  The power consumption may be higher in these
> > >> instances but the system otherwise properly suspends and resumes.
> > >
> > > Well, the idea is that s2idle should not require FW support at all.
> > > >
> >
> > May I ask - why?  It's an intentional design decision?
> 
> Yes, it is.
> 
> > > It may not be possible to reach the minimum power level of the
> > > platform without FW support, but that should not prevent s2idle from
> > > being used.
> > >
> > >> On AMD platforms however when the FW has been configured not to
> > >> offer s2idle some different hardware initialization has occurred
> > >> such that the system won't properly resume.
> > >
> > > That's rather unfortunate.
> > >
> > > Can you please share some details on what's going on in those cases?
> > >
> > > Technically, without FW support there should be no difference
> > > between the platform state reachable via s2idle and the platform
> > > state reachable via runtime idle.
> >
> > During resume there is a number of page faults that occur and during
> > initialization the ring tests fail.  The graphics is unusable at this
> > time as a result.
> >
> > The amdgpu code actually *does* distinguish between the 3 different
> > cases of S3, S0ix, and runtime suspend.
> 
> But s2idle doesn't guarantee S0ix in any case.
> 
> > The function "amdgpu_acpi_is_s0ix_active" causes different codepaths
> > to be used during the suspend routine.
> 
> Well, as I said, s2idle need not mean S0ix.
> 
> > In this particular case that FADT doesn't set the low power idle bit
> > and that function returns false meaning the s3 codepath is taken but
> > the hardware didn't go through a reset.
> 
> If there is a separate S3 code path, taking it when pm_suspend_target_state
> == PM_SUSPEND_TO_IDLE is incorrect.
> 
> > It *might* also be possible to solve this by mandating an ASIC reset
> > in such a case (we didn't try).
> 
> I'd rather do a PM-runtime path equivalent if the target sleep state is
> PM_SUSPEND_TO_IDLE and there is no FW support for S0ix.
> 
> > However it comes back to my first upleveveled question - is this a
> > case we really want to support and encourage?  This type of bug and
> > combination of codepaths is not a case that is going to be well tested.
> > This patch series will align the kernel behavior to only what AMD validates.
> 
> But this does not follow the definition of s2idle and its documentation.

At least for devices integrated into the SoC, the power rails are controlled by the firmware in the SoC.  For S3, the power rails are cut by the FW when the platform enters S3.  For S0ix, the power rails are cut when all of the devices on the rail suspended and various conditions are met.  Also, in the case of some devices, the device has to be in a very specific state for s0ix to work properly.  The GPU is the big one here.  For S3, the entire GPU has to be re-initialized at resume.  For S0ix, the GPU's state is largely handled by the firmware and attempting to re-initialize it won't work unless you reset it.  Integrated AMD graphics don't support runtime power management, only dGPUs do.  For integrated graphics the firmware dynamically controls the power at runtime so there is no need to do anything special for runtime pm.  For dGPUs we support d3cold either via ACPI on platforms like all-in-ones and laptops or via a driver initiated sequence for add-in-cards.

What does S2idle ultimately do when all devices have suspended?  Does it enter S0ix or S3 at the end when it want to ultimately suspend the platform, or is the assumption that if all devices have suspended, the that is equivalent to S0ix or S3?  For AMD platforms, either S3 or S0ix needs to be entered for the platform to actually power down most of the power rails.  It's not clear to me what we should do for s2idle.

Alex

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Rafael J. Wysocki]

On Tue, Jan 11, 2022 at 6:32 PM Deucher, Alexander
<Alexander.Deucher@amd.com> wrote:
>
> [AMD Official Use Only]
>
> > -----Original Message-----
> > From: Rafael J. Wysocki <rafael@kernel.org>
> > Sent: Tuesday, January 11, 2022 12:06 PM
> > To: Limonciello, Mario <Mario.Limonciello@amd.com>
> > Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki
> > <rjw@rjwysocki.net>; ACPI Devel Maling List <linux-acpi@vger.kernel.org>;
> > S-k, Shyam-sundar <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj
> > <Basavaraj.Natikar@amd.com>; Deucher, Alexander
> > <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com
> > Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW
> >
> > On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario
> > <mario.limonciello@amd.com> wrote:
> > >
> > > +Alex
> > >
> > > On 1/11/2022 09:52, Rafael J. Wysocki wrote:
> > > > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
> > > > <mario.limonciello@amd.com> wrote:
> > > >>
> > > >> Currently the Linux kernel will offer s2idle regardless of whether
> > > >> the FADT indicates the system should use or on X86 if the LPS0 ACPI
> > > >> device has been activated.
> > > >>
> > > >> On some non-AMD platforms s2idle can be offered even without
> > proper
> > > >> firmware support.  The power consumption may be higher in these
> > > >> instances but the system otherwise properly suspends and resumes.
> > > >
> > > > Well, the idea is that s2idle should not require FW support at all.
> > > > >
> > >
> > > May I ask - why?  It's an intentional design decision?
> >
> > Yes, it is.
> >
> > > > It may not be possible to reach the minimum power level of the
> > > > platform without FW support, but that should not prevent s2idle from
> > > > being used.
> > > >
> > > >> On AMD platforms however when the FW has been configured not to
> > > >> offer s2idle some different hardware initialization has occurred
> > > >> such that the system won't properly resume.
> > > >
> > > > That's rather unfortunate.
> > > >
> > > > Can you please share some details on what's going on in those cases?
> > > >
> > > > Technically, without FW support there should be no difference
> > > > between the platform state reachable via s2idle and the platform
> > > > state reachable via runtime idle.
> > >
> > > During resume there is a number of page faults that occur and during
> > > initialization the ring tests fail.  The graphics is unusable at this
> > > time as a result.
> > >
> > > The amdgpu code actually *does* distinguish between the 3 different
> > > cases of S3, S0ix, and runtime suspend.
> >
> > But s2idle doesn't guarantee S0ix in any case.
> >
> > > The function "amdgpu_acpi_is_s0ix_active" causes different codepaths
> > > to be used during the suspend routine.
> >
> > Well, as I said, s2idle need not mean S0ix.
> >
> > > In this particular case that FADT doesn't set the low power idle bit
> > > and that function returns false meaning the s3 codepath is taken but
> > > the hardware didn't go through a reset.
> >
> > If there is a separate S3 code path, taking it when pm_suspend_target_state
> > == PM_SUSPEND_TO_IDLE is incorrect.
> >
> > > It *might* also be possible to solve this by mandating an ASIC reset
> > > in such a case (we didn't try).
> >
> > I'd rather do a PM-runtime path equivalent if the target sleep state is
> > PM_SUSPEND_TO_IDLE and there is no FW support for S0ix.
> >
> > > However it comes back to my first upleveveled question - is this a
> > > case we really want to support and encourage?  This type of bug and
> > > combination of codepaths is not a case that is going to be well tested.
> > > This patch series will align the kernel behavior to only what AMD validates.
> >
> > But this does not follow the definition of s2idle and its documentation.
>
> At least for devices integrated into the SoC, the power rails are controlled by the firmware in the SoC.  For S3, the power rails are cut by the FW when the platform enters S3.  For S0ix, the power rails are cut when all of the devices on the rail suspended and various conditions are met.  Also, in the case of some devices, the device has to be in a very specific state for s0ix to work properly.  The GPU is the big one here.  For S3, the entire GPU has to be re-initialized at resume.  For S0ix, the GPU's state is largely handled by the firmware and attempting to re-initialize it won't work unless you reset it.  Integrated AMD graphics don't support runtime power management, only dGPUs do.  For integrated graphics the firmware dynamically controls the power at runtime so there is no need to do anything special for runtime pm.  For dGPUs we support d3cold either via ACPI on platforms like all-in-ones and laptops or via a driver initiated sequence for add-in-cards.
>
> What does S2idle ultimately do when all devices have suspended?  Does it enter S0ix or S3 at the end when it want to ultimately suspend the platform, or is the assumption that if all devices have suspended, the that is equivalent to S0ix or S3?  For AMD platforms, either S3 or S0ix needs to be entered for the platform to actually power down most of the power rails.  It's not clear to me what we should do for s2idle.

s2idle will never enter S3, because that requires platform support and
generally is a different thing (eg. some devices need special upfront
preparation for S3, wakeup may need to be configured in a special way
etc.).

It will attempt to enter S0ix if possible, but otherwise it will just
put CPUs into the deepest available idle state and stay there until an
interrupt (or equivalent) triggers.

Physically, at a device level, s2idle is more similar to runtime
suspend than to S3, but it uses system-wide suspend callbacks and it
requires wakeup to be disabled for the devices that are not allowed to
wake up the system by user space, that is device_may_wakeup(0 returns
false (PM-runtime assumes wakeup to always be enabled for the
suspended devices that can signal wakeup).

In any case, the ACPI system state for s2idle is always S0.

RE: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Deucher, Alexander]

[Public]

> -----Original Message-----
> From: Rafael J. Wysocki <rafael@kernel.org>
> Sent: Tuesday, January 11, 2022 12:45 PM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: Rafael J. Wysocki <rafael@kernel.org>; Limonciello, Mario
> <Mario.Limonciello@amd.com>; Rafael J . Wysocki <rjw@rjwysocki.net>;
> ACPI Devel Maling List <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar
> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj
> <Basavaraj.Natikar@amd.com>; bjoern.daase@gmail.com
> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW
> 
> On Tue, Jan 11, 2022 at 6:32 PM Deucher, Alexander
> <Alexander.Deucher@amd.com> wrote:
> >
> > [AMD Official Use Only]
> >
> > > -----Original Message-----
> > > From: Rafael J. Wysocki <rafael@kernel.org>
> > > Sent: Tuesday, January 11, 2022 12:06 PM
> > > To: Limonciello, Mario <Mario.Limonciello@amd.com>
> > > Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki
> > > <rjw@rjwysocki.net>; ACPI Devel Maling List
> > > <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar
> > > <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj
> > > <Basavaraj.Natikar@amd.com>; Deucher, Alexander
> > > <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com
> > > Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/
> > > proper FW
> > >
> > > On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario
> > > <mario.limonciello@amd.com> wrote:
> > > >
> > > > +Alex
> > > >
> > > > On 1/11/2022 09:52, Rafael J. Wysocki wrote:
> > > > > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
> > > > > <mario.limonciello@amd.com> wrote:
> > > > >>
> > > > >> Currently the Linux kernel will offer s2idle regardless of
> > > > >> whether the FADT indicates the system should use or on X86 if
> > > > >> the LPS0 ACPI device has been activated.
> > > > >>
> > > > >> On some non-AMD platforms s2idle can be offered even without
> > > proper
> > > > >> firmware support.  The power consumption may be higher in these
> > > > >> instances but the system otherwise properly suspends and
> resumes.
> > > > >
> > > > > Well, the idea is that s2idle should not require FW support at all.
> > > > > >
> > > >
> > > > May I ask - why?  It's an intentional design decision?
> > >
> > > Yes, it is.
> > >
> > > > > It may not be possible to reach the minimum power level of the
> > > > > platform without FW support, but that should not prevent s2idle
> > > > > from being used.
> > > > >
> > > > >> On AMD platforms however when the FW has been configured not
> to
> > > > >> offer s2idle some different hardware initialization has
> > > > >> occurred such that the system won't properly resume.
> > > > >
> > > > > That's rather unfortunate.
> > > > >
> > > > > Can you please share some details on what's going on in those cases?
> > > > >
> > > > > Technically, without FW support there should be no difference
> > > > > between the platform state reachable via s2idle and the platform
> > > > > state reachable via runtime idle.
> > > >
> > > > During resume there is a number of page faults that occur and
> > > > during initialization the ring tests fail.  The graphics is
> > > > unusable at this time as a result.
> > > >
> > > > The amdgpu code actually *does* distinguish between the 3
> > > > different cases of S3, S0ix, and runtime suspend.
> > >
> > > But s2idle doesn't guarantee S0ix in any case.
> > >
> > > > The function "amdgpu_acpi_is_s0ix_active" causes different
> > > > codepaths to be used during the suspend routine.
> > >
> > > Well, as I said, s2idle need not mean S0ix.
> > >
> > > > In this particular case that FADT doesn't set the low power idle
> > > > bit and that function returns false meaning the s3 codepath is
> > > > taken but the hardware didn't go through a reset.
> > >
> > > If there is a separate S3 code path, taking it when
> > > pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect.
> > >
> > > > It *might* also be possible to solve this by mandating an ASIC
> > > > reset in such a case (we didn't try).
> > >
> > > I'd rather do a PM-runtime path equivalent if the target sleep state
> > > is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix.
> > >
> > > > However it comes back to my first upleveveled question - is this a
> > > > case we really want to support and encourage?  This type of bug
> > > > and combination of codepaths is not a case that is going to be well
> tested.
> > > > This patch series will align the kernel behavior to only what AMD
> validates.
> > >
> > > But this does not follow the definition of s2idle and its documentation.
> >
> > At least for devices integrated into the SoC, the power rails are controlled
> by the firmware in the SoC.  For S3, the power rails are cut by the FW when
> the platform enters S3.  For S0ix, the power rails are cut when all of the
> devices on the rail suspended and various conditions are met.  Also, in the
> case of some devices, the device has to be in a very specific state for s0ix to
> work properly.  The GPU is the big one here.  For S3, the entire GPU has to
> be re-initialized at resume.  For S0ix, the GPU's state is largely handled by the
> firmware and attempting to re-initialize it won't work unless you reset it.
> Integrated AMD graphics don't support runtime power management, only
> dGPUs do.  For integrated graphics the firmware dynamically controls the
> power at runtime so there is no need to do anything special for runtime pm.
> For dGPUs we support d3cold either via ACPI on platforms like all-in-ones
> and laptops or via a driver initiated sequence for add-in-cards.
> >
> > What does S2idle ultimately do when all devices have suspended?  Does it
> enter S0ix or S3 at the end when it want to ultimately suspend the platform,
> or is the assumption that if all devices have suspended, the that is equivalent
> to S0ix or S3?  For AMD platforms, either S3 or S0ix needs to be entered for
> the platform to actually power down most of the power rails.  It's not clear to
> me what we should do for s2idle.
> 
> s2idle will never enter S3, because that requires platform support and
> generally is a different thing (eg. some devices need special upfront
> preparation for S3, wakeup may need to be configured in a special way etc.).
> 
> It will attempt to enter S0ix if possible, but otherwise it will just put CPUs into
> the deepest available idle state and stay there until an interrupt (or
> equivalent) triggers.
> 
> Physically, at a device level, s2idle is more similar to runtime suspend than to
> S3, but it uses system-wide suspend callbacks and it requires wakeup to be
> disabled for the devices that are not allowed to wake up the system by user
> space, that is device_may_wakeup(0 returns false (PM-runtime assumes
> wakeup to always be enabled for the suspended devices that can signal
> wakeup).
> 
> In any case, the ACPI system state for s2idle is always S0.

Thanks.  In that case, it's kind of pointless on AMD platforms then since the power rails will never be turned off for most devices on the system.  Does it even make sense to expose it?  It just gives users a false sense of suspend and then they will probably complain that it uses too much power when in s2idle.

Alex

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Limonciello, Mario]

On 1/11/2022 12:30, Deucher, Alexander wrote:
> [Public]
> 
>> -----Original Message-----
>> From: Rafael J. Wysocki <rafael@kernel.org>
>> Sent: Tuesday, January 11, 2022 12:45 PM
>> To: Deucher, Alexander <Alexander.Deucher@amd.com>
>> Cc: Rafael J. Wysocki <rafael@kernel.org>; Limonciello, Mario
>> <Mario.Limonciello@amd.com>; Rafael J . Wysocki <rjw@rjwysocki.net>;
>> ACPI Devel Maling List <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar
>> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj
>> <Basavaraj.Natikar@amd.com>; bjoern.daase@gmail.com
>> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW
>>
>> On Tue, Jan 11, 2022 at 6:32 PM Deucher, Alexander
>> <Alexander.Deucher@amd.com> wrote:
>>>
>>> [AMD Official Use Only]
>>>
>>>> -----Original Message-----
>>>> From: Rafael J. Wysocki <rafael@kernel.org>
>>>> Sent: Tuesday, January 11, 2022 12:06 PM
>>>> To: Limonciello, Mario <Mario.Limonciello@amd.com>
>>>> Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki
>>>> <rjw@rjwysocki.net>; ACPI Devel Maling List
>>>> <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar
>>>> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj
>>>> <Basavaraj.Natikar@amd.com>; Deucher, Alexander
>>>> <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com
>>>> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/
>>>> proper FW
>>>>
>>>> On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario
>>>> <mario.limonciello@amd.com> wrote:
>>>>>
>>>>> +Alex
>>>>>
>>>>> On 1/11/2022 09:52, Rafael J. Wysocki wrote:
>>>>>> On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
>>>>>> <mario.limonciello@amd.com> wrote:
>>>>>>>
>>>>>>> Currently the Linux kernel will offer s2idle regardless of
>>>>>>> whether the FADT indicates the system should use or on X86 if
>>>>>>> the LPS0 ACPI device has been activated.
>>>>>>>
>>>>>>> On some non-AMD platforms s2idle can be offered even without
>>>> proper
>>>>>>> firmware support.  The power consumption may be higher in these
>>>>>>> instances but the system otherwise properly suspends and
>> resumes.
>>>>>>
>>>>>> Well, the idea is that s2idle should not require FW support at all.
>>>>>>>
>>>>>
>>>>> May I ask - why?  It's an intentional design decision?
>>>>
>>>> Yes, it is.
>>>>
>>>>>> It may not be possible to reach the minimum power level of the
>>>>>> platform without FW support, but that should not prevent s2idle
>>>>>> from being used.
>>>>>>
>>>>>>> On AMD platforms however when the FW has been configured not
>> to
>>>>>>> offer s2idle some different hardware initialization has
>>>>>>> occurred such that the system won't properly resume.
>>>>>>
>>>>>> That's rather unfortunate.
>>>>>>
>>>>>> Can you please share some details on what's going on in those cases?
>>>>>>
>>>>>> Technically, without FW support there should be no difference
>>>>>> between the platform state reachable via s2idle and the platform
>>>>>> state reachable via runtime idle.
>>>>>
>>>>> During resume there is a number of page faults that occur and
>>>>> during initialization the ring tests fail.  The graphics is
>>>>> unusable at this time as a result.
>>>>>
>>>>> The amdgpu code actually *does* distinguish between the 3
>>>>> different cases of S3, S0ix, and runtime suspend.
>>>>
>>>> But s2idle doesn't guarantee S0ix in any case.
>>>>
>>>>> The function "amdgpu_acpi_is_s0ix_active" causes different
>>>>> codepaths to be used during the suspend routine.
>>>>
>>>> Well, as I said, s2idle need not mean S0ix.
>>>>
>>>>> In this particular case that FADT doesn't set the low power idle
>>>>> bit and that function returns false meaning the s3 codepath is
>>>>> taken but the hardware didn't go through a reset.
>>>>
>>>> If there is a separate S3 code path, taking it when
>>>> pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect.
>>>>
>>>>> It *might* also be possible to solve this by mandating an ASIC
>>>>> reset in such a case (we didn't try).
>>>>
>>>> I'd rather do a PM-runtime path equivalent if the target sleep state
>>>> is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix.
>>>>
>>>>> However it comes back to my first upleveveled question - is this a
>>>>> case we really want to support and encourage?  This type of bug
>>>>> and combination of codepaths is not a case that is going to be well
>> tested.
>>>>> This patch series will align the kernel behavior to only what AMD
>> validates.
>>>>
>>>> But this does not follow the definition of s2idle and its documentation.
>>>
>>> At least for devices integrated into the SoC, the power rails are controlled
>> by the firmware in the SoC.  For S3, the power rails are cut by the FW when
>> the platform enters S3.  For S0ix, the power rails are cut when all of the
>> devices on the rail suspended and various conditions are met.  Also, in the
>> case of some devices, the device has to be in a very specific state for s0ix to
>> work properly.  The GPU is the big one here.  For S3, the entire GPU has to
>> be re-initialized at resume.  For S0ix, the GPU's state is largely handled by the
>> firmware and attempting to re-initialize it won't work unless you reset it.
>> Integrated AMD graphics don't support runtime power management, only
>> dGPUs do.  For integrated graphics the firmware dynamically controls the
>> power at runtime so there is no need to do anything special for runtime pm.
>> For dGPUs we support d3cold either via ACPI on platforms like all-in-ones
>> and laptops or via a driver initiated sequence for add-in-cards.
>>>
>>> What does S2idle ultimately do when all devices have suspended?  Does it
>> enter S0ix or S3 at the end when it want to ultimately suspend the platform,
>> or is the assumption that if all devices have suspended, the that is equivalent
>> to S0ix or S3?  For AMD platforms, either S3 or S0ix needs to be entered for
>> the platform to actually power down most of the power rails.  It's not clear to
>> me what we should do for s2idle.
>>
>> s2idle will never enter S3, because that requires platform support and
>> generally is a different thing (eg. some devices need special upfront
>> preparation for S3, wakeup may need to be configured in a special way etc.).
>>
>> It will attempt to enter S0ix if possible, but otherwise it will just put CPUs into
>> the deepest available idle state and stay there until an interrupt (or
>> equivalent) triggers.
>>
>> Physically, at a device level, s2idle is more similar to runtime suspend than to
>> S3, but it uses system-wide suspend callbacks and it requires wakeup to be
>> disabled for the devices that are not allowed to wake up the system by user
>> space, that is device_may_wakeup(0 returns false (PM-runtime assumes
>> wakeup to always be enabled for the suspended devices that can signal
>> wakeup).
>>
>> In any case, the ACPI system state for s2idle is always S0.
> 
> Thanks.  In that case, it's kind of pointless on AMD platforms then since the power rails will never be turned off for most devices on the system.  Does it even make sense to expose it?  It just gives users a false sense of suspend and then they will probably complain that it uses too much power when in s2idle.
> 
> Alex

That's why I thought my patch series made sense.

I guess another (antisocial) response would be to to return false when 
the suspend callback for amdgpu happens and dev_err mentioning that 
firmware support is needed for suspend.

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Rafael J. Wysocki]

On Tue, Jan 11, 2022 at 7:36 PM Limonciello, Mario
<mario.limonciello@amd.com> wrote:
>
> On 1/11/2022 12:30, Deucher, Alexander wrote:
> > [Public]
> >
> >> -----Original Message-----
> >> From: Rafael J. Wysocki <rafael@kernel.org>
> >> Sent: Tuesday, January 11, 2022 12:45 PM
> >> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> >> Cc: Rafael J. Wysocki <rafael@kernel.org>; Limonciello, Mario
> >> <Mario.Limonciello@amd.com>; Rafael J . Wysocki <rjw@rjwysocki.net>;
> >> ACPI Devel Maling List <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar
> >> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj
> >> <Basavaraj.Natikar@amd.com>; bjoern.daase@gmail.com
> >> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW
> >>
> >> On Tue, Jan 11, 2022 at 6:32 PM Deucher, Alexander
> >> <Alexander.Deucher@amd.com> wrote:
> >>>
> >>> [AMD Official Use Only]
> >>>
> >>>> -----Original Message-----
> >>>> From: Rafael J. Wysocki <rafael@kernel.org>
> >>>> Sent: Tuesday, January 11, 2022 12:06 PM
> >>>> To: Limonciello, Mario <Mario.Limonciello@amd.com>
> >>>> Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki
> >>>> <rjw@rjwysocki.net>; ACPI Devel Maling List
> >>>> <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar
> >>>> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj
> >>>> <Basavaraj.Natikar@amd.com>; Deucher, Alexander
> >>>> <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com
> >>>> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/
> >>>> proper FW
> >>>>
> >>>> On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario
> >>>> <mario.limonciello@amd.com> wrote:
> >>>>>
> >>>>> +Alex
> >>>>>
> >>>>> On 1/11/2022 09:52, Rafael J. Wysocki wrote:
> >>>>>> On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello
> >>>>>> <mario.limonciello@amd.com> wrote:
> >>>>>>>
> >>>>>>> Currently the Linux kernel will offer s2idle regardless of
> >>>>>>> whether the FADT indicates the system should use or on X86 if
> >>>>>>> the LPS0 ACPI device has been activated.
> >>>>>>>
> >>>>>>> On some non-AMD platforms s2idle can be offered even without
> >>>> proper
> >>>>>>> firmware support.  The power consumption may be higher in these
> >>>>>>> instances but the system otherwise properly suspends and
> >> resumes.
> >>>>>>
> >>>>>> Well, the idea is that s2idle should not require FW support at all.
> >>>>>>>
> >>>>>
> >>>>> May I ask - why?  It's an intentional design decision?
> >>>>
> >>>> Yes, it is.
> >>>>
> >>>>>> It may not be possible to reach the minimum power level of the
> >>>>>> platform without FW support, but that should not prevent s2idle
> >>>>>> from being used.
> >>>>>>
> >>>>>>> On AMD platforms however when the FW has been configured not
> >> to
> >>>>>>> offer s2idle some different hardware initialization has
> >>>>>>> occurred such that the system won't properly resume.
> >>>>>>
> >>>>>> That's rather unfortunate.
> >>>>>>
> >>>>>> Can you please share some details on what's going on in those cases?
> >>>>>>
> >>>>>> Technically, without FW support there should be no difference
> >>>>>> between the platform state reachable via s2idle and the platform
> >>>>>> state reachable via runtime idle.
> >>>>>
> >>>>> During resume there is a number of page faults that occur and
> >>>>> during initialization the ring tests fail.  The graphics is
> >>>>> unusable at this time as a result.
> >>>>>
> >>>>> The amdgpu code actually *does* distinguish between the 3
> >>>>> different cases of S3, S0ix, and runtime suspend.
> >>>>
> >>>> But s2idle doesn't guarantee S0ix in any case.
> >>>>
> >>>>> The function "amdgpu_acpi_is_s0ix_active" causes different
> >>>>> codepaths to be used during the suspend routine.
> >>>>
> >>>> Well, as I said, s2idle need not mean S0ix.
> >>>>
> >>>>> In this particular case that FADT doesn't set the low power idle
> >>>>> bit and that function returns false meaning the s3 codepath is
> >>>>> taken but the hardware didn't go through a reset.
> >>>>
> >>>> If there is a separate S3 code path, taking it when
> >>>> pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect.
> >>>>
> >>>>> It *might* also be possible to solve this by mandating an ASIC
> >>>>> reset in such a case (we didn't try).
> >>>>
> >>>> I'd rather do a PM-runtime path equivalent if the target sleep state
> >>>> is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix.
> >>>>
> >>>>> However it comes back to my first upleveveled question - is this a
> >>>>> case we really want to support and encourage?  This type of bug
> >>>>> and combination of codepaths is not a case that is going to be well
> >> tested.
> >>>>> This patch series will align the kernel behavior to only what AMD
> >> validates.
> >>>>
> >>>> But this does not follow the definition of s2idle and its documentation.
> >>>
> >>> At least for devices integrated into the SoC, the power rails are controlled
> >> by the firmware in the SoC.  For S3, the power rails are cut by the FW when
> >> the platform enters S3.  For S0ix, the power rails are cut when all of the
> >> devices on the rail suspended and various conditions are met.  Also, in the
> >> case of some devices, the device has to be in a very specific state for s0ix to
> >> work properly.  The GPU is the big one here.  For S3, the entire GPU has to
> >> be re-initialized at resume.  For S0ix, the GPU's state is largely handled by the
> >> firmware and attempting to re-initialize it won't work unless you reset it.
> >> Integrated AMD graphics don't support runtime power management, only
> >> dGPUs do.  For integrated graphics the firmware dynamically controls the
> >> power at runtime so there is no need to do anything special for runtime pm.
> >> For dGPUs we support d3cold either via ACPI on platforms like all-in-ones
> >> and laptops or via a driver initiated sequence for add-in-cards.
> >>>
> >>> What does S2idle ultimately do when all devices have suspended?  Does it
> >> enter S0ix or S3 at the end when it want to ultimately suspend the platform,
> >> or is the assumption that if all devices have suspended, the that is equivalent
> >> to S0ix or S3?  For AMD platforms, either S3 or S0ix needs to be entered for
> >> the platform to actually power down most of the power rails.  It's not clear to
> >> me what we should do for s2idle.
> >>
> >> s2idle will never enter S3, because that requires platform support and
> >> generally is a different thing (eg. some devices need special upfront
> >> preparation for S3, wakeup may need to be configured in a special way etc.).
> >>
> >> It will attempt to enter S0ix if possible, but otherwise it will just put CPUs into
> >> the deepest available idle state and stay there until an interrupt (or
> >> equivalent) triggers.
> >>
> >> Physically, at a device level, s2idle is more similar to runtime suspend than to
> >> S3, but it uses system-wide suspend callbacks and it requires wakeup to be
> >> disabled for the devices that are not allowed to wake up the system by user
> >> space, that is device_may_wakeup(0 returns false (PM-runtime assumes
> >> wakeup to always be enabled for the suspended devices that can signal
> >> wakeup).
> >>
> >> In any case, the ACPI system state for s2idle is always S0.
> >
> > Thanks.  In that case, it's kind of pointless on AMD platforms then since the power rails will never be turned off for most devices on the system.  Does it even make sense to expose it?  It just gives users a false sense of suspend and then they will probably complain that it uses too much power when in s2idle.
> >
> > Alex
>
> That's why I thought my patch series made sense.
>
> I guess another (antisocial) response would be to to return false when
> the suspend callback for amdgpu happens and dev_err mentioning that
> firmware support is needed for suspend.

If you want to avoid exposing s2idle at all in some cases, you need to
change the sysfs I/F in /sys/power/ so that it doesn't expose "s2idle"
in mem_sleep or "freeze" in state at all.  That's much more intrusive
than just failing every s2idle attempt (which is what the patches do
AFAICS).

Do you want to do that for platforms supporting S3, or for platforms
that don't support any kind of suspend at all?

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Limonciello, Mario]

>>
>> That's why I thought my patch series made sense.
>>
>> I guess another (antisocial) response would be to to return false when
>> the suspend callback for amdgpu happens and dev_err mentioning that
>> firmware support is needed for suspend.
> 
> If you want to avoid exposing s2idle at all in some cases, you need to
> change the sysfs I/F in /sys/power/ so that it doesn't expose "s2idle"
> in mem_sleep or "freeze" in state at all.  

Right now they make it so that s2idle isn't exposed in "mem_sleep", but 
"freeze" is still exposed in "state".

I'd be worried about breaking userspace with taking "freeze" out of state.

> That's much more intrusive
> than just failing every s2idle attempt (which is what the patches do
> AFAICS)
As they stand right now when the BIOS is set to S3:
# cat /sys/power/mem_sleep
[deep]

# echo freeze > /sys/power/state
Returns a failure though.

And then when it's set to Modern Standby :

# cat /sys/power/mem_sleep
[s2idle]

# echo freeze > /sys/power/state
Works

> 
> Do you want to do that for platforms supporting S3, or for platforms
> that don't support any kind of suspend at all?

You know since this series went out 
6dc8265f9803ccb7e5da804e01601f0c14f270e0 was merged.  This will probably
have cleaned up the problem state that Bjoern found in the first place.

So this is really a philosophical discussion now if it's worth offering 
s2idle in /sys/power/mem_sleep on AMD systems.

Admittedly this has been an APU notebook oriented discussion.  Platforms 
that don't support any kind of suspend (like servers) will get caught up 
in this and could no longer do s2idle either.

With the assumption that 6dc8265f9803ccb7e5da804e01601f0c14f270e0 helps 
the problem state I'm leaning back on we should add some warnings to let 
people know how to help themselves for power consumption if they did this.

We can probably put those in amdgpu though.

Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW

[Rafael J. Wysocki]

On Tue, Jan 11, 2022 at 8:35 PM Limonciello, Mario
<mario.limonciello@amd.com> wrote:
>
> >>
> >> That's why I thought my patch series made sense.
> >>
> >> I guess another (antisocial) response would be to to return false when
> >> the suspend callback for amdgpu happens and dev_err mentioning that
> >> firmware support is needed for suspend.
> >
> > If you want to avoid exposing s2idle at all in some cases, you need to
> > change the sysfs I/F in /sys/power/ so that it doesn't expose "s2idle"
> > in mem_sleep or "freeze" in state at all.
>
> Right now they make it so that s2idle isn't exposed in "mem_sleep", but
> "freeze" is still exposed in "state".
>
> I'd be worried about breaking userspace with taking "freeze" out of state.
>
> > That's much more intrusive
> > than just failing every s2idle attempt (which is what the patches do
> > AFAICS)
> As they stand right now when the BIOS is set to S3:
> # cat /sys/power/mem_sleep
> [deep]
>
> # echo freeze > /sys/power/state
> Returns a failure though.
>
> And then when it's set to Modern Standby :
>
> # cat /sys/power/mem_sleep
> [s2idle]
>
> # echo freeze > /sys/power/state
> Works
>
> >
> > Do you want to do that for platforms supporting S3, or for platforms
> > that don't support any kind of suspend at all?
>
> You know since this series went out
> 6dc8265f9803ccb7e5da804e01601f0c14f270e0 was merged.  This will probably
> have cleaned up the problem state that Bjoern found in the first place.
>
> So this is really a philosophical discussion now if it's worth offering
> s2idle in /sys/power/mem_sleep on AMD systems.

Well, if "freeze" is present in "state", then "s2idle" needs to be
there in "mem_sleep", because they are clearly documented as
alternative paths to the same functionality.

> Admittedly this has been an APU notebook oriented discussion.  Platforms
> that don't support any kind of suspend (like servers) will get caught up
> in this and could no longer do s2idle either.
>
> With the assumption that 6dc8265f9803ccb7e5da804e01601f0c14f270e0 helps
> the problem state I'm leaning back on we should add some warnings to let
> people know how to help themselves for power consumption if they did this.
>
> We can probably put those in amdgpu though.

Sounds like a better direction to me, honestly.