[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
isn't well optimized for system sleep.

What makes this driver particularly interesting is because it's a cross-SoC
driver, which sometimes means there is an ACPI PM domain attached to the i2c
device and sometimes not. The driver is being used on both x86 and ARM.

In principle, to optimize the system sleep support in i2c driver, this series
enables the proven runtime PM centric path for the i2c driver. However, to do
that the ACPI PM domain also have to collaborate and understand this behaviour.
>From earlier versions, Rafael has also pointed out that also the PM core needs
to be involved.

Therefore a number of changes, patch 1 to patch 6, makes the needed changes to
the PM core and the ACPI PM domain. In patch7 and patch 8, the i2c driver gets
optimized and is converted to the runtime PM centric path for system sleep.

It shall be noted, the behaviour of the ACPI PM domain should remain intact,
still taking benefit of using the direct_complete path during system sleep,
except for those drivers that uses the runtime PM centric path.

This series has been tested on an ARM64 Hikey board, which isn't having the
i2c device attached to the ACPI PM domain. This means that the ACPI changes
needs to be tested on some relevant Intel SoCs and it's greatly appreciated
is someone could help out with this, so is of course review comments.

Some news in v3:
	- The fix for the i2c driver [1], is now present in Linus' tree from tag
	v4.13-rc7 - and so does Rafael's tree.
	- To simplify for testers, I have published a branch [3] based upon
	Rafael's pm tree and linux-next branch.
	- Rephrased some part of the coverletter to clarify the intent of this
	series.
	- Addressed review comments from v2.

Some news in v2:
	- The v1 contained a fix for the i2c driver, this has been sent
	separately [1] and picked up for fixes by Wolfram for v4.13-rcs. However
	the fix has not yet reached Linus' tree. The changes on i2c driver
	are based upon that change.
	- To simplify for testers, I have published a branch [2] based upon
	Rafael's pm tree and linux-next branch, which also includes the above
	patch.
	- Rephrased the coverletter to clarify the intent of this series.
	- Addressed review comments from v1.

[1]
http://patchwork.ozlabs.org/patch/799803/

[2]
git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git acpi_pm_i2c_rpm_path_v2

[3]
git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git acpi_pm_i2c_rpm_path_v3

Kind regards
Ulf Hansson


Ulf Hansson (8):
  PM / Sleep: Make the runtime PM centric path known to the PM core
  PM / ACPI: Restore acpi_subsys_complete()
  PM / Sleep: Remove pm_complete_with_resume_check()
  PM / ACPI: Split code validating need for runtime resume in
    ->prepare()
  PM / ACPI: Split acpi_lpss_suspend_late|resume_early()
  PM / ACPI: Enable the runtime PM centric approach for system sleep
  i2c: designware: Don't resume device in the ->complete() callback
  i2c: designware: Deploy the runtime PM centric path for system sleep

 drivers/acpi/acpi_lpss.c                    |  79 ++++++++++++++------
 drivers/acpi/device_pm.c                    | 111 ++++++++++++++++++++++------
 drivers/base/power/generic_ops.c            |  23 ------
 drivers/base/power/main.c                   |  49 ++++++++++--
 drivers/base/power/runtime.c                |   1 +
 drivers/i2c/busses/i2c-designware-platdrv.c |  34 ++-------
 include/linux/pm.h                          |   8 +-
 7 files changed, 204 insertions(+), 101 deletions(-)

-- 
2.7.4

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
isn't well optimized for system sleep.

What makes this driver particularly interesting is because it's a cross-SoC
driver, which sometimes means there is an ACPI PM domain attached to the i2c
device and sometimes not. The driver is being used on both x86 and ARM.

In principle, to optimize the system sleep support in i2c driver, this series
enables the proven runtime PM centric path for the i2c driver. However, to do
that the ACPI PM domain also have to collaborate and understand this behaviour.
>From earlier versions, Rafael has also pointed out that also the PM core needs
to be involved.

Therefore a number of changes, patch 1 to patch 6, makes the needed changes to
the PM core and the ACPI PM domain. In patch7 and patch 8, the i2c driver gets
optimized and is converted to the runtime PM centric path for system sleep.

It shall be noted, the behaviour of the ACPI PM domain should remain intact,
still taking benefit of using the direct_complete path during system sleep,
except for those drivers that uses the runtime PM centric path.

This series has been tested on an ARM64 Hikey board, which isn't having the
i2c device attached to the ACPI PM domain. This means that the ACPI changes
needs to be tested on some relevant Intel SoCs and it's greatly appreciated
is someone could help out with this, so is of course review comments.

Some news in v3:
	- The fix for the i2c driver [1], is now present in Linus' tree from tag
	v4.13-rc7 - and so does Rafael's tree.
	- To simplify for testers, I have published a branch [3] based upon
	Rafael's pm tree and linux-next branch.
	- Rephrased some part of the coverletter to clarify the intent of this
	series.
	- Addressed review comments from v2.

Some news in v2:
	- The v1 contained a fix for the i2c driver, this has been sent
	separately [1] and picked up for fixes by Wolfram for v4.13-rcs. However
	the fix has not yet reached Linus' tree. The changes on i2c driver
	are based upon that change.
	- To simplify for testers, I have published a branch [2] based upon
	Rafael's pm tree and linux-next branch, which also includes the above
	patch.
	- Rephrased the coverletter to clarify the intent of this series.
	- Addressed review comments from v1.

[1]
http://patchwork.ozlabs.org/patch/799803/

[2]
git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git acpi_pm_i2c_rpm_path_v2

[3]
git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git acpi_pm_i2c_rpm_path_v3

Kind regards
Ulf Hansson


Ulf Hansson (8):
  PM / Sleep: Make the runtime PM centric path known to the PM core
  PM / ACPI: Restore acpi_subsys_complete()
  PM / Sleep: Remove pm_complete_with_resume_check()
  PM / ACPI: Split code validating need for runtime resume in
    ->prepare()
  PM / ACPI: Split acpi_lpss_suspend_late|resume_early()
  PM / ACPI: Enable the runtime PM centric approach for system sleep
  i2c: designware: Don't resume device in the ->complete() callback
  i2c: designware: Deploy the runtime PM centric path for system sleep

 drivers/acpi/acpi_lpss.c                    |  79 ++++++++++++++------
 drivers/acpi/device_pm.c                    | 111 ++++++++++++++++++++++------
 drivers/base/power/generic_ops.c            |  23 ------
 drivers/base/power/main.c                   |  49 ++++++++++--
 drivers/base/power/runtime.c                |   1 +
 drivers/i2c/busses/i2c-designware-platdrv.c |  34 ++-------
 include/linux/pm.h                          |   8 +-
 7 files changed, 204 insertions(+), 101 deletions(-)

-- 
2.7.4

[PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Ulf Hansson]

The main principle behind the runtime PM centric path, is to re-use the
runtime PM callbacks to implement system sleep - and while doing that also
achieve a fully optimized behaviour from PM point of view.

More precisely, avoid to wake up a device from its low power state during
system sleep, unless the device is really needed to be operational. That
does not only mean avoiding to waste power, but may also decrease system
suspend/resume time for a device.

However, using the runtime PM centric path for a device, does put some
requirements on the behaviour of the PM core and a potential PM domain that
may be attached to the device. So far, these requirements are not being
specially considered, except by the generic PM domain.

To move forward and to make it possible to deploy the runtime PM centric
path for cross SoC drivers, which may have different PM domains attached to
its devices depending on the SoC, we must address how to deal with these
requirements. This change starts by making some adoptions to the PM core,
while other parts, such as the ACPI PM domain needs to be taken care of
separately.

In the runtime PM centric path, the driver is expected to make use of the
pm_runtime_force_suspend|resume() helpers, to deploy system sleep support.
More precisely it may assign the system sleep callbacks to these helpers or
may call them from its own callbacks, in case it needs to perform
additional actions during system sleep.

In other words, the PM core must always invoke the system sleep callbacks
for the device when they are present, to allow the driver to deal with
the system sleep operations.

In case the PM core decides to run the direct_complete path for the device,
it skips invoking most of the system sleep callbacks, besides
->prepare|complete(). Therefore using the direct_complete path in
combination with the runtime PM centric patch for a device, does not play
well.

To deal with this issue, let's add a flag 'is_rpm_sleep', to the struct
dev_pm_info. The driver that deploys the runtime PM centric path, shall set
the flag for the device during ->probe(), to inform the PM core about that
it must not use the direct_complete path for the device.

Note, not allowing the direct_complete path for a device, doesn't implicit
need to propagate to the device's parent/suppliers. Therefore make the PM
core check this condition in device_suspend(), before it decides to abandon
the direct_complete path for parent/suppliers.

To make the is_rpm_sleep flag internal to the PM core, let's add two APIs.
	- dev_pm_use_rpm_sleep(): It sets the flag and should be called by
	  the driver during ->probe().
	- dev_pm_is_rpm_sleep(): Makes it possible for users of the device,
	  like a PM domain, to fetch the state of the flag.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- New patch.
	- This replaces the earlier method of adding the no_direct_complete to
	the ACPI structures, according to comments from Rafael.
	- This change also address the consern Rafael had around that
	direct_complete should not have to be disabled for parent/suppliers, in
	case a device use the runtime PM centric path for system sleep.

---
 drivers/base/power/main.c    | 49 +++++++++++++++++++++++++++++++++++++++-----
 drivers/base/power/runtime.c |  1 +
 include/linux/pm.h           |  7 +++++++
 3 files changed, 52 insertions(+), 5 deletions(-)

diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index ea1732e..865737a 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
 		if (parent) {
 			spin_lock_irq(&parent->power.lock);
 
-			dev->parent->power.direct_complete = false;
+			if (!dev->power.is_rpm_sleep)
+				dev->parent->power.direct_complete = false;
 			if (dev->power.wakeup_path
 			    && !dev->parent->power.ignore_children)
 				dev->parent->power.wakeup_path = true;
 
 			spin_unlock_irq(&parent->power.lock);
 		}
-		dpm_clear_suppliers_direct_complete(dev);
+		if (!dev->power.is_rpm_sleep)
+			dpm_clear_suppliers_direct_complete(dev);
 	}
 
 	device_unlock(dev);
@@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
 	 * A positive return value from ->prepare() means "this device appears
 	 * to be runtime-suspended and its state is fine, so if it really is
 	 * runtime-suspended, you can leave it in that state provided that you
-	 * will do the same thing with all of its descendants".  This only
-	 * applies to suspend transitions, however.
+	 * will do the same thing with all of its descendants". To allow this,
+	 * the is_rpm_sleep flag must not be set, which is default false, but
+	 * may have been changed by a driver. This only applies to suspend
+	 * transitions, however.
 	 */
 	spin_lock_irq(&dev->power.lock);
-	dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
+	dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
+				state.event == PM_EVENT_SUSPEND;
 	spin_unlock_irq(&dev->power.lock);
 	return 0;
 }
@@ -1841,6 +1846,40 @@ void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *))
 }
 EXPORT_SYMBOL_GPL(dpm_for_each_dev);
 
+/**
+ * dev_pm_use_rpm_sleep - use the runtime PM centric sleep path.
+ * @dev: the device to use the path for.
+ *
+ * The runtime PM centric path requires the system sleep callbacks for the
+ * driver of the device to be invoked. This function sets the is_rpm_sleep flag
+ * to enable this path to be used, which makes the PM core to adopt to this
+ * behaviour. More precisely the PM core makes sure to not run the
+ * direct_complete path for the chosen device during system sleep, when the
+ * is_rpm_sleep flag is set. A driver using the runtime PM centric path, shall
+ * use the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
+ * support and call this function during ->probe().
+ * The is_rpm_sleep flag is cleared when unbinding/bind-failure of the driver.
+ */
+void dev_pm_use_rpm_sleep(struct device *dev)
+{
+	dev->power.is_rpm_sleep = true;
+}
+EXPORT_SYMBOL_GPL(dev_pm_use_rpm_sleep);
+
+/**
+ * dev_pm_is_rpm_sleep - Returns the value of the is_rpm_sleep flag.
+ * @dev: the device to return the flag for.
+ *
+ * The caller of this function is typically a subsystem or a PM domain that
+ * needs to know if the runtime PM centric path is used for the device. It may
+ * be invoked during any of the system sleep phases.
+ */
+bool dev_pm_is_rpm_sleep(struct device *dev)
+{
+	return dev->power.is_rpm_sleep;
+}
+EXPORT_SYMBOL_GPL(dev_pm_is_rpm_sleep);
+
 static bool pm_ops_is_empty(const struct dev_pm_ops *ops)
 {
 	if (!ops)
diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c
index 7bcf80f..b2ab22c 100644
--- a/drivers/base/power/runtime.c
+++ b/drivers/base/power/runtime.c
@@ -1522,6 +1522,7 @@ void pm_runtime_reinit(struct device *dev)
 				pm_runtime_put(dev->parent);
 		}
 	}
+	dev->power.is_rpm_sleep = false;
 }
 
 /**
diff --git a/include/linux/pm.h b/include/linux/pm.h
index 47ded8a..5bf96d2 100644
--- a/include/linux/pm.h
+++ b/include/linux/pm.h
@@ -559,6 +559,7 @@ struct dev_pm_info {
 	bool			is_suspended:1;	/* Ditto */
 	bool			is_noirq_suspended:1;
 	bool			is_late_suspended:1;
+	bool			is_rpm_sleep:1;	/* Owned by the PM core */
 	bool			early_init:1;	/* Owned by the PM core */
 	bool			direct_complete:1;	/* Owned by the PM core */
 	spinlock_t		lock;
@@ -716,6 +717,9 @@ extern void __suspend_report_result(const char *function, void *fn, int ret);
 extern int device_pm_wait_for_dev(struct device *sub, struct device *dev);
 extern void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *));
 
+extern void dev_pm_use_rpm_sleep(struct device *dev);
+extern bool dev_pm_is_rpm_sleep(struct device *dev);
+
 extern int pm_generic_prepare(struct device *dev);
 extern int pm_generic_suspend_late(struct device *dev);
 extern int pm_generic_suspend_noirq(struct device *dev);
@@ -759,6 +763,9 @@ static inline void dpm_for_each_dev(void *data, void (*fn)(struct device *, void
 {
 }
 
+static inline void dev_pm_use_rpm_sleep(struct device *dev) {}
+static inline bool dev_pm_is_rpm_sleep(struct device *dev) { return false; }
+
 #define pm_generic_prepare		NULL
 #define pm_generic_suspend_late		NULL
 #define pm_generic_suspend_noirq	NULL
-- 
2.7.4

[PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Ulf Hansson]

The main principle behind the runtime PM centric path, is to re-use the
runtime PM callbacks to implement system sleep - and while doing that also
achieve a fully optimized behaviour from PM point of view.

More precisely, avoid to wake up a device from its low power state during
system sleep, unless the device is really needed to be operational. That
does not only mean avoiding to waste power, but may also decrease system
suspend/resume time for a device.

However, using the runtime PM centric path for a device, does put some
requirements on the behaviour of the PM core and a potential PM domain that
may be attached to the device. So far, these requirements are not being
specially considered, except by the generic PM domain.

To move forward and to make it possible to deploy the runtime PM centric
path for cross SoC drivers, which may have different PM domains attached to
its devices depending on the SoC, we must address how to deal with these
requirements. This change starts by making some adoptions to the PM core,
while other parts, such as the ACPI PM domain needs to be taken care of
separately.

In the runtime PM centric path, the driver is expected to make use of the
pm_runtime_force_suspend|resume() helpers, to deploy system sleep support.
More precisely it may assign the system sleep callbacks to these helpers or
may call them from its own callbacks, in case it needs to perform
additional actions during system sleep.

In other words, the PM core must always invoke the system sleep callbacks
for the device when they are present, to allow the driver to deal with
the system sleep operations.

In case the PM core decides to run the direct_complete path for the device,
it skips invoking most of the system sleep callbacks, besides
->prepare|complete(). Therefore using the direct_complete path in
combination with the runtime PM centric patch for a device, does not play
well.

To deal with this issue, let's add a flag 'is_rpm_sleep', to the struct
dev_pm_info. The driver that deploys the runtime PM centric path, shall set
the flag for the device during ->probe(), to inform the PM core about that
it must not use the direct_complete path for the device.

Note, not allowing the direct_complete path for a device, doesn't implicit
need to propagate to the device's parent/suppliers. Therefore make the PM
core check this condition in device_suspend(), before it decides to abandon
the direct_complete path for parent/suppliers.

To make the is_rpm_sleep flag internal to the PM core, let's add two APIs.
	- dev_pm_use_rpm_sleep(): It sets the flag and should be called by
	  the driver during ->probe().
	- dev_pm_is_rpm_sleep(): Makes it possible for users of the device,
	  like a PM domain, to fetch the state of the flag.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- New patch.
	- This replaces the earlier method of adding the no_direct_complete to
	the ACPI structures, according to comments from Rafael.
	- This change also address the consern Rafael had around that
	direct_complete should not have to be disabled for parent/suppliers, in
	case a device use the runtime PM centric path for system sleep.

---
 drivers/base/power/main.c    | 49 +++++++++++++++++++++++++++++++++++++++-----
 drivers/base/power/runtime.c |  1 +
 include/linux/pm.h           |  7 +++++++
 3 files changed, 52 insertions(+), 5 deletions(-)

diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index ea1732e..865737a 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
 		if (parent) {
 			spin_lock_irq(&parent->power.lock);
 
-			dev->parent->power.direct_complete = false;
+			if (!dev->power.is_rpm_sleep)
+				dev->parent->power.direct_complete = false;
 			if (dev->power.wakeup_path
 			    && !dev->parent->power.ignore_children)
 				dev->parent->power.wakeup_path = true;
 
 			spin_unlock_irq(&parent->power.lock);
 		}
-		dpm_clear_suppliers_direct_complete(dev);
+		if (!dev->power.is_rpm_sleep)
+			dpm_clear_suppliers_direct_complete(dev);
 	}
 
 	device_unlock(dev);
@@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
 	 * A positive return value from ->prepare() means "this device appears
 	 * to be runtime-suspended and its state is fine, so if it really is
 	 * runtime-suspended, you can leave it in that state provided that you
-	 * will do the same thing with all of its descendants".  This only
-	 * applies to suspend transitions, however.
+	 * will do the same thing with all of its descendants". To allow this,
+	 * the is_rpm_sleep flag must not be set, which is default false, but
+	 * may have been changed by a driver. This only applies to suspend
+	 * transitions, however.
 	 */
 	spin_lock_irq(&dev->power.lock);
-	dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
+	dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
+				state.event == PM_EVENT_SUSPEND;
 	spin_unlock_irq(&dev->power.lock);
 	return 0;
 }
@@ -1841,6 +1846,40 @@ void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *))
 }
 EXPORT_SYMBOL_GPL(dpm_for_each_dev);
 
+/**
+ * dev_pm_use_rpm_sleep - use the runtime PM centric sleep path.
+ * @dev: the device to use the path for.
+ *
+ * The runtime PM centric path requires the system sleep callbacks for the
+ * driver of the device to be invoked. This function sets the is_rpm_sleep flag
+ * to enable this path to be used, which makes the PM core to adopt to this
+ * behaviour. More precisely the PM core makes sure to not run the
+ * direct_complete path for the chosen device during system sleep, when the
+ * is_rpm_sleep flag is set. A driver using the runtime PM centric path, shall
+ * use the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
+ * support and call this function during ->probe().
+ * The is_rpm_sleep flag is cleared when unbinding/bind-failure of the driver.
+ */
+void dev_pm_use_rpm_sleep(struct device *dev)
+{
+	dev->power.is_rpm_sleep = true;
+}
+EXPORT_SYMBOL_GPL(dev_pm_use_rpm_sleep);
+
+/**
+ * dev_pm_is_rpm_sleep - Returns the value of the is_rpm_sleep flag.
+ * @dev: the device to return the flag for.
+ *
+ * The caller of this function is typically a subsystem or a PM domain that
+ * needs to know if the runtime PM centric path is used for the device. It may
+ * be invoked during any of the system sleep phases.
+ */
+bool dev_pm_is_rpm_sleep(struct device *dev)
+{
+	return dev->power.is_rpm_sleep;
+}
+EXPORT_SYMBOL_GPL(dev_pm_is_rpm_sleep);
+
 static bool pm_ops_is_empty(const struct dev_pm_ops *ops)
 {
 	if (!ops)
diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c
index 7bcf80f..b2ab22c 100644
--- a/drivers/base/power/runtime.c
+++ b/drivers/base/power/runtime.c
@@ -1522,6 +1522,7 @@ void pm_runtime_reinit(struct device *dev)
 				pm_runtime_put(dev->parent);
 		}
 	}
+	dev->power.is_rpm_sleep = false;
 }
 
 /**
diff --git a/include/linux/pm.h b/include/linux/pm.h
index 47ded8a..5bf96d2 100644
--- a/include/linux/pm.h
+++ b/include/linux/pm.h
@@ -559,6 +559,7 @@ struct dev_pm_info {
 	bool			is_suspended:1;	/* Ditto */
 	bool			is_noirq_suspended:1;
 	bool			is_late_suspended:1;
+	bool			is_rpm_sleep:1;	/* Owned by the PM core */
 	bool			early_init:1;	/* Owned by the PM core */
 	bool			direct_complete:1;	/* Owned by the PM core */
 	spinlock_t		lock;
@@ -716,6 +717,9 @@ extern void __suspend_report_result(const char *function, void *fn, int ret);
 extern int device_pm_wait_for_dev(struct device *sub, struct device *dev);
 extern void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *));
 
+extern void dev_pm_use_rpm_sleep(struct device *dev);
+extern bool dev_pm_is_rpm_sleep(struct device *dev);
+
 extern int pm_generic_prepare(struct device *dev);
 extern int pm_generic_suspend_late(struct device *dev);
 extern int pm_generic_suspend_noirq(struct device *dev);
@@ -759,6 +763,9 @@ static inline void dpm_for_each_dev(void *data, void (*fn)(struct device *, void
 {
 }
 
+static inline void dev_pm_use_rpm_sleep(struct device *dev) {}
+static inline bool dev_pm_is_rpm_sleep(struct device *dev) { return false; }
+
 #define pm_generic_prepare		NULL
 #define pm_generic_suspend_late		NULL
 #define pm_generic_suspend_noirq	NULL
-- 
2.7.4

[PATCH v3 2/8] PM / ACPI: Restore acpi_subsys_complete()

[Ulf Hansson]

The commit 58a1fbbb2ee8 ("PM / PCI / ACPI: Kick devices that might have
been reset by firmware"), made PCI's and ACPI's ->complete() callbacks to
be assigned to a new API called pm_complete_with_resume_check(), which was
introduced in the same change.

Later it turned out that using pm_complete_with_resume_check() isn't good
enough for PCI, as it needs additional PCI specific checks, before deciding
whether runtime resuming the device is needed when running the ->complete()
callback.

This leaves ACPI being the only user of pm_complete_with_resume_check().
Therefore let's restore ACPI's acpi_subsys_complete(), which was dropped in
commit 58a1fbbb2ee8 ("PM / PCI / ACPI: Kick devices that might have been
reset by firmware").

This enables us to remove the pm_complete_with_resume_check() API in a
following change, but it also enables ACPI to add more ACPI specific checks
in acpi_subsys_complete() if it turns out that is needed.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/acpi/acpi_lpss.c |  2 +-
 drivers/acpi/device_pm.c | 19 ++++++++++++++++++-
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c
index 032ae44..0c84d15 100644
--- a/drivers/acpi/acpi_lpss.c
+++ b/drivers/acpi/acpi_lpss.c
@@ -894,7 +894,7 @@ static struct dev_pm_domain acpi_lpss_pm_domain = {
 #ifdef CONFIG_PM
 #ifdef CONFIG_PM_SLEEP
 		.prepare = acpi_subsys_prepare,
-		.complete = pm_complete_with_resume_check,
+		.complete = acpi_subsys_complete,
 		.suspend = acpi_subsys_suspend,
 		.suspend_late = acpi_lpss_suspend_late,
 		.resume_early = acpi_lpss_resume_early,
diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index fbcc73f..632f214 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1020,6 +1020,23 @@ int acpi_subsys_prepare(struct device *dev)
 EXPORT_SYMBOL_GPL(acpi_subsys_prepare);
 
 /**
+ * acpi_subsys_complete - Finalize device's resume during system resume.
+ * @dev: Device to handle.
+ */
+void acpi_subsys_complete(struct device *dev)
+{
+	pm_generic_complete(dev);
+	/*
+	 * If the device had been runtime-suspended before the system went into
+	 * the sleep state it is going out of and it has never been resumed till
+	 * now, resume it in case the firmware powered it up.
+	 */
+	if (dev->power.direct_complete && pm_resume_via_firmware())
+		pm_request_resume(dev);
+}
+EXPORT_SYMBOL_GPL(acpi_subsys_complete);
+
+/**
  * acpi_subsys_suspend - Run the device driver's suspend callback.
  * @dev: Device to handle.
  *
@@ -1087,7 +1104,7 @@ static struct dev_pm_domain acpi_general_pm_domain = {
 		.runtime_resume = acpi_subsys_runtime_resume,
 #ifdef CONFIG_PM_SLEEP
 		.prepare = acpi_subsys_prepare,
-		.complete = pm_complete_with_resume_check,
+		.complete = acpi_subsys_complete,
 		.suspend = acpi_subsys_suspend,
 		.suspend_late = acpi_subsys_suspend_late,
 		.resume_early = acpi_subsys_resume_early,
-- 
2.7.4

[PATCH v3 2/8] PM / ACPI: Restore acpi_subsys_complete()

[Ulf Hansson]

The commit 58a1fbbb2ee8 ("PM / PCI / ACPI: Kick devices that might have
been reset by firmware"), made PCI's and ACPI's ->complete() callbacks to
be assigned to a new API called pm_complete_with_resume_check(), which was
introduced in the same change.

Later it turned out that using pm_complete_with_resume_check() isn't good
enough for PCI, as it needs additional PCI specific checks, before deciding
whether runtime resuming the device is needed when running the ->complete()
callback.

This leaves ACPI being the only user of pm_complete_with_resume_check().
Therefore let's restore ACPI's acpi_subsys_complete(), which was dropped in
commit 58a1fbbb2ee8 ("PM / PCI / ACPI: Kick devices that might have been
reset by firmware").

This enables us to remove the pm_complete_with_resume_check() API in a
following change, but it also enables ACPI to add more ACPI specific checks
in acpi_subsys_complete() if it turns out that is needed.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/acpi/acpi_lpss.c |  2 +-
 drivers/acpi/device_pm.c | 19 ++++++++++++++++++-
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c
index 032ae44..0c84d15 100644
--- a/drivers/acpi/acpi_lpss.c
+++ b/drivers/acpi/acpi_lpss.c
@@ -894,7 +894,7 @@ static struct dev_pm_domain acpi_lpss_pm_domain = {
 #ifdef CONFIG_PM
 #ifdef CONFIG_PM_SLEEP
 		.prepare = acpi_subsys_prepare,
-		.complete = pm_complete_with_resume_check,
+		.complete = acpi_subsys_complete,
 		.suspend = acpi_subsys_suspend,
 		.suspend_late = acpi_lpss_suspend_late,
 		.resume_early = acpi_lpss_resume_early,
diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index fbcc73f..632f214 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1020,6 +1020,23 @@ int acpi_subsys_prepare(struct device *dev)
 EXPORT_SYMBOL_GPL(acpi_subsys_prepare);
 
 /**
+ * acpi_subsys_complete - Finalize device's resume during system resume.
+ * @dev: Device to handle.
+ */
+void acpi_subsys_complete(struct device *dev)
+{
+	pm_generic_complete(dev);
+	/*
+	 * If the device had been runtime-suspended before the system went into
+	 * the sleep state it is going out of and it has never been resumed till
+	 * now, resume it in case the firmware powered it up.
+	 */
+	if (dev->power.direct_complete && pm_resume_via_firmware())
+		pm_request_resume(dev);
+}
+EXPORT_SYMBOL_GPL(acpi_subsys_complete);
+
+/**
  * acpi_subsys_suspend - Run the device driver's suspend callback.
  * @dev: Device to handle.
  *
@@ -1087,7 +1104,7 @@ static struct dev_pm_domain acpi_general_pm_domain = {
 		.runtime_resume = acpi_subsys_runtime_resume,
 #ifdef CONFIG_PM_SLEEP
 		.prepare = acpi_subsys_prepare,
-		.complete = pm_complete_with_resume_check,
+		.complete = acpi_subsys_complete,
 		.suspend = acpi_subsys_suspend,
 		.suspend_late = acpi_subsys_suspend_late,
 		.resume_early = acpi_subsys_resume_early,
-- 
2.7.4

[PATCH v3 3/8] PM / Sleep: Remove pm_complete_with_resume_check()

[Ulf Hansson]

According to recent changes for ACPI, the are longer any users of
pm_complete_with_resume_check(), thus let's drop it.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/base/power/generic_ops.c | 23 -----------------------
 include/linux/pm.h               |  1 -
 2 files changed, 24 deletions(-)

diff --git a/drivers/base/power/generic_ops.c b/drivers/base/power/generic_ops.c
index 07c3c4a..b2ed606 100644
--- a/drivers/base/power/generic_ops.c
+++ b/drivers/base/power/generic_ops.c
@@ -9,7 +9,6 @@
 #include <linux/pm.h>
 #include <linux/pm_runtime.h>
 #include <linux/export.h>
-#include <linux/suspend.h>
 
 #ifdef CONFIG_PM
 /**
@@ -298,26 +297,4 @@ void pm_generic_complete(struct device *dev)
 	if (drv && drv->pm && drv->pm->complete)
 		drv->pm->complete(dev);
 }
-
-/**
- * pm_complete_with_resume_check - Complete a device power transition.
- * @dev: Device to handle.
- *
- * Complete a device power transition during a system-wide power transition and
- * optionally schedule a runtime resume of the device if the system resume in
- * progress has been initated by the platform firmware and the device had its
- * power.direct_complete flag set.
- */
-void pm_complete_with_resume_check(struct device *dev)
-{
-	pm_generic_complete(dev);
-	/*
-	 * If the device had been runtime-suspended before the system went into
-	 * the sleep state it is going out of and it has never been resumed till
-	 * now, resume it in case the firmware powered it up.
-	 */
-	if (dev->power.direct_complete && pm_resume_via_firmware())
-		pm_request_resume(dev);
-}
-EXPORT_SYMBOL_GPL(pm_complete_with_resume_check);
 #endif /* CONFIG_PM_SLEEP */
diff --git a/include/linux/pm.h b/include/linux/pm.h
index 5bf96d2..c88a4ef 100644
--- a/include/linux/pm.h
+++ b/include/linux/pm.h
@@ -740,7 +740,6 @@ extern int pm_generic_poweroff_noirq(struct device *dev);
 extern int pm_generic_poweroff_late(struct device *dev);
 extern int pm_generic_poweroff(struct device *dev);
 extern void pm_generic_complete(struct device *dev);
-extern void pm_complete_with_resume_check(struct device *dev);
 
 #else /* !CONFIG_PM_SLEEP */
 
-- 
2.7.4

[PATCH v3 3/8] PM / Sleep: Remove pm_complete_with_resume_check()

[Ulf Hansson]

According to recent changes for ACPI, the are longer any users of
pm_complete_with_resume_check(), thus let's drop it.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/base/power/generic_ops.c | 23 -----------------------
 include/linux/pm.h               |  1 -
 2 files changed, 24 deletions(-)

diff --git a/drivers/base/power/generic_ops.c b/drivers/base/power/generic_ops.c
index 07c3c4a..b2ed606 100644
--- a/drivers/base/power/generic_ops.c
+++ b/drivers/base/power/generic_ops.c
@@ -9,7 +9,6 @@
 #include <linux/pm.h>
 #include <linux/pm_runtime.h>
 #include <linux/export.h>
-#include <linux/suspend.h>
 
 #ifdef CONFIG_PM
 /**
@@ -298,26 +297,4 @@ void pm_generic_complete(struct device *dev)
 	if (drv && drv->pm && drv->pm->complete)
 		drv->pm->complete(dev);
 }
-
-/**
- * pm_complete_with_resume_check - Complete a device power transition.
- * @dev: Device to handle.
- *
- * Complete a device power transition during a system-wide power transition and
- * optionally schedule a runtime resume of the device if the system resume in
- * progress has been initated by the platform firmware and the device had its
- * power.direct_complete flag set.
- */
-void pm_complete_with_resume_check(struct device *dev)
-{
-	pm_generic_complete(dev);
-	/*
-	 * If the device had been runtime-suspended before the system went into
-	 * the sleep state it is going out of and it has never been resumed till
-	 * now, resume it in case the firmware powered it up.
-	 */
-	if (dev->power.direct_complete && pm_resume_via_firmware())
-		pm_request_resume(dev);
-}
-EXPORT_SYMBOL_GPL(pm_complete_with_resume_check);
 #endif /* CONFIG_PM_SLEEP */
diff --git a/include/linux/pm.h b/include/linux/pm.h
index 5bf96d2..c88a4ef 100644
--- a/include/linux/pm.h
+++ b/include/linux/pm.h
@@ -740,7 +740,6 @@ extern int pm_generic_poweroff_noirq(struct device *dev);
 extern int pm_generic_poweroff_late(struct device *dev);
 extern int pm_generic_poweroff(struct device *dev);
 extern void pm_generic_complete(struct device *dev);
-extern void pm_complete_with_resume_check(struct device *dev);
 
 #else /* !CONFIG_PM_SLEEP */
 
-- 
2.7.4

[PATCH v3 4/8] PM / ACPI: Split code validating need for runtime resume in ->prepare()

[Ulf Hansson]

Move the code dealing with validation of whether runtime resuming the
device is needed during system suspend.

In this way it becomes more clear for what circumstances ACPI is prevented
from trying the direct_complete path.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/acpi/device_pm.c | 37 ++++++++++++++++++++++++-------------
 1 file changed, 24 insertions(+), 13 deletions(-)

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index 632f214..5181057 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -989,6 +989,27 @@ int acpi_dev_resume_early(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(acpi_dev_resume_early);
 
+static bool acpi_dev_needs_resume(struct device *dev, struct acpi_device *adev)
+{
+	u32 sys_target = acpi_target_system_state();
+	int ret, state;
+
+	if (device_may_wakeup(dev) != !!adev->wakeup.prepare_count)
+		return true;
+
+	if (sys_target == ACPI_STATE_S0)
+		return false;
+
+	if (adev->power.flags.dsw_present)
+		return true;
+
+	ret = acpi_dev_pm_get_state(dev, adev, sys_target, NULL, &state);
+	if (ret)
+		return true;
+
+	return state != adev->power.state;
+}
+
 /**
  * acpi_subsys_prepare - Prepare device for system transition to a sleep state.
  * @dev: Device to prepare.
@@ -996,26 +1017,16 @@ EXPORT_SYMBOL_GPL(acpi_dev_resume_early);
 int acpi_subsys_prepare(struct device *dev)
 {
 	struct acpi_device *adev = ACPI_COMPANION(dev);
-	u32 sys_target;
-	int ret, state;
+	int ret;
 
 	ret = pm_generic_prepare(dev);
 	if (ret < 0)
 		return ret;
 
-	if (!adev || !pm_runtime_suspended(dev)
-	    || device_may_wakeup(dev) != !!adev->wakeup.prepare_count)
-		return 0;
-
-	sys_target = acpi_target_system_state();
-	if (sys_target == ACPI_STATE_S0)
-		return 1;
-
-	if (adev->power.flags.dsw_present)
+	if (!adev || !pm_runtime_suspended(dev))
 		return 0;
 
-	ret = acpi_dev_pm_get_state(dev, adev, sys_target, NULL, &state);
-	return !ret && state == adev->power.state;
+	return !acpi_dev_needs_resume(dev, adev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_prepare);
 
-- 
2.7.4

[PATCH v3 4/8] PM / ACPI: Split code validating need for runtime resume in ->prepare()

[Ulf Hansson]

Move the code dealing with validation of whether runtime resuming the
device is needed during system suspend.

In this way it becomes more clear for what circumstances ACPI is prevented
from trying the direct_complete path.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/acpi/device_pm.c | 37 ++++++++++++++++++++++++-------------
 1 file changed, 24 insertions(+), 13 deletions(-)

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index 632f214..5181057 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -989,6 +989,27 @@ int acpi_dev_resume_early(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(acpi_dev_resume_early);
 
+static bool acpi_dev_needs_resume(struct device *dev, struct acpi_device *adev)
+{
+	u32 sys_target = acpi_target_system_state();
+	int ret, state;
+
+	if (device_may_wakeup(dev) != !!adev->wakeup.prepare_count)
+		return true;
+
+	if (sys_target == ACPI_STATE_S0)
+		return false;
+
+	if (adev->power.flags.dsw_present)
+		return true;
+
+	ret = acpi_dev_pm_get_state(dev, adev, sys_target, NULL, &state);
+	if (ret)
+		return true;
+
+	return state != adev->power.state;
+}
+
 /**
  * acpi_subsys_prepare - Prepare device for system transition to a sleep state.
  * @dev: Device to prepare.
@@ -996,26 +1017,16 @@ EXPORT_SYMBOL_GPL(acpi_dev_resume_early);
 int acpi_subsys_prepare(struct device *dev)
 {
 	struct acpi_device *adev = ACPI_COMPANION(dev);
-	u32 sys_target;
-	int ret, state;
+	int ret;
 
 	ret = pm_generic_prepare(dev);
 	if (ret < 0)
 		return ret;
 
-	if (!adev || !pm_runtime_suspended(dev)
-	    || device_may_wakeup(dev) != !!adev->wakeup.prepare_count)
-		return 0;
-
-	sys_target = acpi_target_system_state();
-	if (sys_target == ACPI_STATE_S0)
-		return 1;
-
-	if (adev->power.flags.dsw_present)
+	if (!adev || !pm_runtime_suspended(dev))
 		return 0;
 
-	ret = acpi_dev_pm_get_state(dev, adev, sys_target, NULL, &state);
-	return !ret && state == adev->power.state;
+	return !acpi_dev_needs_resume(dev, adev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_prepare);
 
-- 
2.7.4

[PATCH v3 5/8] PM / ACPI: Split acpi_lpss_suspend_late|resume_early()

[Ulf Hansson]

Move the code which is special to ACPI LPSS into separate functions. This
may clarify the code a bit, but the main purpose of this change, is instead
to prepare for additional changes on top. Ideally the following changes
should then become easier to review.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/acpi/acpi_lpss.c | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)

diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c
index 0c84d15..e726173 100644
--- a/drivers/acpi/acpi_lpss.c
+++ b/drivers/acpi/acpi_lpss.c
@@ -717,22 +717,28 @@ static void acpi_lpss_dismiss(struct device *dev)
 }
 
 #ifdef CONFIG_PM_SLEEP
-static int acpi_lpss_suspend_late(struct device *dev)
+static int lpss_suspend_late(struct device *dev)
 {
 	struct lpss_private_data *pdata = acpi_driver_data(ACPI_COMPANION(dev));
+
+	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
+		acpi_lpss_save_ctx(dev, pdata);
+
+	return acpi_dev_suspend_late(dev);
+}
+
+static int acpi_lpss_suspend_late(struct device *dev)
+{
 	int ret;
 
 	ret = pm_generic_suspend_late(dev);
 	if (ret)
 		return ret;
 
-	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
-		acpi_lpss_save_ctx(dev, pdata);
-
-	return acpi_dev_suspend_late(dev);
+	return lpss_suspend_late(dev);
 }
 
-static int acpi_lpss_resume_early(struct device *dev)
+static int lpss_resume_early(struct device *dev)
 {
 	struct lpss_private_data *pdata = acpi_driver_data(ACPI_COMPANION(dev));
 	int ret;
@@ -746,6 +752,17 @@ static int acpi_lpss_resume_early(struct device *dev)
 	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
 		acpi_lpss_restore_ctx(dev, pdata);
 
+	return 0;
+}
+
+static int acpi_lpss_resume_early(struct device *dev)
+{
+	int ret;
+
+	ret = lpss_resume_early(dev);
+	if (ret)
+		return ret;
+
 	return pm_generic_resume_early(dev);
 }
 #endif /* CONFIG_PM_SLEEP */
-- 
2.7.4

[PATCH v3 5/8] PM / ACPI: Split acpi_lpss_suspend_late|resume_early()

[Ulf Hansson]

Move the code which is special to ACPI LPSS into separate functions. This
may clarify the code a bit, but the main purpose of this change, is instead
to prepare for additional changes on top. Ideally the following changes
should then become easier to review.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/acpi/acpi_lpss.c | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)

diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c
index 0c84d15..e726173 100644
--- a/drivers/acpi/acpi_lpss.c
+++ b/drivers/acpi/acpi_lpss.c
@@ -717,22 +717,28 @@ static void acpi_lpss_dismiss(struct device *dev)
 }
 
 #ifdef CONFIG_PM_SLEEP
-static int acpi_lpss_suspend_late(struct device *dev)
+static int lpss_suspend_late(struct device *dev)
 {
 	struct lpss_private_data *pdata = acpi_driver_data(ACPI_COMPANION(dev));
+
+	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
+		acpi_lpss_save_ctx(dev, pdata);
+
+	return acpi_dev_suspend_late(dev);
+}
+
+static int acpi_lpss_suspend_late(struct device *dev)
+{
 	int ret;
 
 	ret = pm_generic_suspend_late(dev);
 	if (ret)
 		return ret;
 
-	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
-		acpi_lpss_save_ctx(dev, pdata);
-
-	return acpi_dev_suspend_late(dev);
+	return lpss_suspend_late(dev);
 }
 
-static int acpi_lpss_resume_early(struct device *dev)
+static int lpss_resume_early(struct device *dev)
 {
 	struct lpss_private_data *pdata = acpi_driver_data(ACPI_COMPANION(dev));
 	int ret;
@@ -746,6 +752,17 @@ static int acpi_lpss_resume_early(struct device *dev)
 	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
 		acpi_lpss_restore_ctx(dev, pdata);
 
+	return 0;
+}
+
+static int acpi_lpss_resume_early(struct device *dev)
+{
+	int ret;
+
+	ret = lpss_resume_early(dev);
+	if (ret)
+		return ret;
+
 	return pm_generic_resume_early(dev);
 }
 #endif /* CONFIG_PM_SLEEP */
-- 
2.7.4

[PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Ulf Hansson]

This change enables the ACPI PM domain to cope with drivers that deploys
the runtime PM centric path for system sleep.

Currently the ACPI PM domain supports the direct_complete path, which
offers some nice optimizations from PM point of view during system sleep.
However, the runtime PM centric path have some additional optimizations
that this change enables for the ACPI PM domain. Let's walk through them.

*)
The runtime PM centric path, doesn't require the device to be runtime
suspended during system suspend, when later during system resume trying to
avoid to bring it back into full power. That is the case for the
direct_complete path. This further avoids wasting power during system
resume, but should also decrease the time it takes to resume the device.

**)
When the runtime PM centric path is used, the PM core does not skip calling
any system sleep callbacks for the device, which is the case in the
direct_complete path. Based on that knowledge and relying on the driver to
do the right thing, the ACPI PM domain may avoid to always runtime resume
the device in the device_suspend() phase.

***)
In the runtime PM centric path, the device may remain runtime PM enabled
until the device_suspend_late() phase, instead of as in the direct_complete
path, in the device_suspend() phase. This is convenient if the device needs
to be runtime resumed sometime during the device_suspend() phase.

To deploy the runtime PM centric approach for the ACPI PM domain, and make
it benefit from the above optimizations, the follow changes are made.

First, the ACPI PM domain's runtime PM callbacks may be called when runtime
PM has been disabled for the device. This serves as an indication to
understand when they are running in the system sleep sequence, instead of
in the regular runtime PM path. For these cases, make the ACPI PM domain to
execute the same operations as normally being run, when the
->suspend_late() and the ->resume_early() callbacks are invoked.

Second, the ACPI PM domain's ->suspend_late() callback, shall not execute
the regular operations to put the device into low power state, when the
runtime PM centric path is used. Calling pm_generic_suspend_late() is
sufficient. Vice verse applies to the ACPI PM domain's ->resume_early()
callback.

Third, in the ACPI PM domain's ->suspend|freeze() callbacks, let's avoid
runtime resuming the device in case the runtime PM centric path is used,
unless there are ACPI PM domain specific reasons to not.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- Convert to use the PM core flag, is_rpm_sleep flag.
	- Fold in changes from patch v2 7/9, which means avoiding to runtime
	resume the device in the ACPI PM domain's ->suspend|freeze() callbacks.
	- Updated changelog to reflect new changes.

---
 drivers/acpi/acpi_lpss.c | 56 ++++++++++++++++++++++++++++++---------------
 drivers/acpi/device_pm.c | 59 ++++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 87 insertions(+), 28 deletions(-)

diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c
index e726173..2e34f69 100644
--- a/drivers/acpi/acpi_lpss.c
+++ b/drivers/acpi/acpi_lpss.c
@@ -732,7 +732,7 @@ static int acpi_lpss_suspend_late(struct device *dev)
 	int ret;
 
 	ret = pm_generic_suspend_late(dev);
-	if (ret)
+	if (ret || dev_pm_is_rpm_sleep(dev))
 		return ret;
 
 	return lpss_suspend_late(dev);
@@ -757,13 +757,22 @@ static int lpss_resume_early(struct device *dev)
 
 static int acpi_lpss_resume_early(struct device *dev)
 {
-	int ret;
+	int ret = 0;
 
-	ret = lpss_resume_early(dev);
-	if (ret)
-		return ret;
+	if (!dev_pm_is_rpm_sleep(dev))
+		ret = lpss_resume_early(dev);
 
-	return pm_generic_resume_early(dev);
+	return ret ? ret : pm_generic_resume_early(dev);
+}
+#else
+static inline int lpss_suspend_late(struct device *dev)
+{
+	return 0;
+}
+
+static inline int lpss_resume_early(struct device *dev)
+{
+	return 0;
 }
 #endif /* CONFIG_PM_SLEEP */
 
@@ -861,6 +870,9 @@ static int acpi_lpss_runtime_suspend(struct device *dev)
 	if (ret)
 		return ret;
 
+	if (!pm_runtime_enabled(dev))
+		return lpss_suspend_late(dev);
+
 	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
 		acpi_lpss_save_ctx(dev, pdata);
 
@@ -882,21 +894,29 @@ static int acpi_lpss_runtime_resume(struct device *dev)
 	struct lpss_private_data *pdata = acpi_driver_data(ACPI_COMPANION(dev));
 	int ret;
 
-	/*
-	 * This call is kept first to be in symmetry with
-	 * acpi_lpss_runtime_suspend() one.
-	 */
-	if (lpss_quirks & LPSS_QUIRK_ALWAYS_POWER_ON && iosf_mbi_available())
-		lpss_iosf_exit_d3_state();
+	if (pm_runtime_enabled(dev)) {
+		/*
+		 * This call is kept first to be in symmetry with
+		 * acpi_lpss_runtime_suspend() one.
+		 */
+		if (lpss_quirks & LPSS_QUIRK_ALWAYS_POWER_ON &&
+		    iosf_mbi_available())
+			lpss_iosf_exit_d3_state();
 
-	ret = acpi_dev_runtime_resume(dev);
-	if (ret)
-		return ret;
+		ret = acpi_dev_runtime_resume(dev);
+		if (ret)
+			return ret;
 
-	acpi_lpss_d3_to_d0_delay(pdata);
+		acpi_lpss_d3_to_d0_delay(pdata);
 
-	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
-		acpi_lpss_restore_ctx(dev, pdata);
+		if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
+			acpi_lpss_restore_ctx(dev, pdata);
+
+	} else {
+		ret = lpss_resume_early(dev);
+		if (ret)
+			return ret;
+	}
 
 	return pm_generic_runtime_resume(dev);
 }
diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index 5181057..f5c4d0e 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -913,7 +913,14 @@ EXPORT_SYMBOL_GPL(acpi_dev_runtime_resume);
 int acpi_subsys_runtime_suspend(struct device *dev)
 {
 	int ret = pm_generic_runtime_suspend(dev);
-	return ret ? ret : acpi_dev_runtime_suspend(dev);
+
+	if (ret)
+		return ret;
+
+	if (!pm_runtime_enabled(dev))
+		return acpi_dev_suspend_late(dev);
+
+	return acpi_dev_runtime_suspend(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_runtime_suspend);
 
@@ -926,7 +933,13 @@ EXPORT_SYMBOL_GPL(acpi_subsys_runtime_suspend);
  */
 int acpi_subsys_runtime_resume(struct device *dev)
 {
-	int ret = acpi_dev_runtime_resume(dev);
+	int ret = 0;
+
+	if (!pm_runtime_enabled(dev))
+		ret = acpi_dev_resume_early(dev);
+	else
+		ret = acpi_dev_runtime_resume(dev);
+
 	return ret ? ret : pm_generic_runtime_resume(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_runtime_resume);
@@ -1023,7 +1036,7 @@ int acpi_subsys_prepare(struct device *dev)
 	if (ret < 0)
 		return ret;
 
-	if (!adev || !pm_runtime_suspended(dev))
+	if (!adev || dev_pm_is_rpm_sleep(dev) || !pm_runtime_suspended(dev))
 		return 0;
 
 	return !acpi_dev_needs_resume(dev, adev);
@@ -1042,7 +1055,8 @@ void acpi_subsys_complete(struct device *dev)
 	 * the sleep state it is going out of and it has never been resumed till
 	 * now, resume it in case the firmware powered it up.
 	 */
-	if (dev->power.direct_complete && pm_resume_via_firmware())
+	if ((dev->power.direct_complete || dev_pm_is_rpm_sleep(dev)) &&
+	    pm_resume_via_firmware())
 		pm_request_resume(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_complete);
@@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
  * @dev: Device to handle.
  *
  * Follow PCI and resume devices suspended at run time before running their
- * system suspend callbacks.
+ * system suspend callbacks. However, try to avoid it in case the runtime PM
+ * centric path is used for the device and then trust the driver to do the
+ * right thing.
  */
 int acpi_subsys_suspend(struct device *dev)
 {
-	pm_runtime_resume(dev);
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+
+	if (!adev)
+		return 0;
+
+	if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
+		pm_runtime_resume(dev);
+
 	return pm_generic_suspend(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
@@ -1071,7 +1094,11 @@ EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
 int acpi_subsys_suspend_late(struct device *dev)
 {
 	int ret = pm_generic_suspend_late(dev);
-	return ret ? ret : acpi_dev_suspend_late(dev);
+
+	if (ret || dev_pm_is_rpm_sleep(dev))
+		return ret;
+
+	return acpi_dev_suspend_late(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_suspend_late);
 
@@ -1085,7 +1112,11 @@ EXPORT_SYMBOL_GPL(acpi_subsys_suspend_late);
  */
 int acpi_subsys_resume_early(struct device *dev)
 {
-	int ret = acpi_dev_resume_early(dev);
+	int ret = 0;
+
+	if (!dev_pm_is_rpm_sleep(dev))
+		ret = acpi_dev_resume_early(dev);
+
 	return ret ? ret : pm_generic_resume_early(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_resume_early);
@@ -1096,13 +1127,21 @@ EXPORT_SYMBOL_GPL(acpi_subsys_resume_early);
  */
 int acpi_subsys_freeze(struct device *dev)
 {
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+
+	if (!adev)
+		return 0;
+
 	/*
 	 * This used to be done in acpi_subsys_prepare() for all devices and
 	 * some drivers may depend on it, so do it here.  Ideally, however,
 	 * runtime-suspended devices should not be touched during freeze/thaw
-	 * transitions.
+	 * transitions. In case the runtime PM centric path is used, let's try
+	 * to avoid it.
 	 */
-	pm_runtime_resume(dev);
+	if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
+		pm_runtime_resume(dev);
+
 	return pm_generic_freeze(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_freeze);
-- 
2.7.4

[PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Ulf Hansson]

This change enables the ACPI PM domain to cope with drivers that deploys
the runtime PM centric path for system sleep.

Currently the ACPI PM domain supports the direct_complete path, which
offers some nice optimizations from PM point of view during system sleep.
However, the runtime PM centric path have some additional optimizations
that this change enables for the ACPI PM domain. Let's walk through them.

*)
The runtime PM centric path, doesn't require the device to be runtime
suspended during system suspend, when later during system resume trying to
avoid to bring it back into full power. That is the case for the
direct_complete path. This further avoids wasting power during system
resume, but should also decrease the time it takes to resume the device.

**)
When the runtime PM centric path is used, the PM core does not skip calling
any system sleep callbacks for the device, which is the case in the
direct_complete path. Based on that knowledge and relying on the driver to
do the right thing, the ACPI PM domain may avoid to always runtime resume
the device in the device_suspend() phase.

***)
In the runtime PM centric path, the device may remain runtime PM enabled
until the device_suspend_late() phase, instead of as in the direct_complete
path, in the device_suspend() phase. This is convenient if the device needs
to be runtime resumed sometime during the device_suspend() phase.

To deploy the runtime PM centric approach for the ACPI PM domain, and make
it benefit from the above optimizations, the follow changes are made.

First, the ACPI PM domain's runtime PM callbacks may be called when runtime
PM has been disabled for the device. This serves as an indication to
understand when they are running in the system sleep sequence, instead of
in the regular runtime PM path. For these cases, make the ACPI PM domain to
execute the same operations as normally being run, when the
->suspend_late() and the ->resume_early() callbacks are invoked.

Second, the ACPI PM domain's ->suspend_late() callback, shall not execute
the regular operations to put the device into low power state, when the
runtime PM centric path is used. Calling pm_generic_suspend_late() is
sufficient. Vice verse applies to the ACPI PM domain's ->resume_early()
callback.

Third, in the ACPI PM domain's ->suspend|freeze() callbacks, let's avoid
runtime resuming the device in case the runtime PM centric path is used,
unless there are ACPI PM domain specific reasons to not.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- Convert to use the PM core flag, is_rpm_sleep flag.
	- Fold in changes from patch v2 7/9, which means avoiding to runtime
	resume the device in the ACPI PM domain's ->suspend|freeze() callbacks.
	- Updated changelog to reflect new changes.

---
 drivers/acpi/acpi_lpss.c | 56 ++++++++++++++++++++++++++++++---------------
 drivers/acpi/device_pm.c | 59 ++++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 87 insertions(+), 28 deletions(-)

diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c
index e726173..2e34f69 100644
--- a/drivers/acpi/acpi_lpss.c
+++ b/drivers/acpi/acpi_lpss.c
@@ -732,7 +732,7 @@ static int acpi_lpss_suspend_late(struct device *dev)
 	int ret;
 
 	ret = pm_generic_suspend_late(dev);
-	if (ret)
+	if (ret || dev_pm_is_rpm_sleep(dev))
 		return ret;
 
 	return lpss_suspend_late(dev);
@@ -757,13 +757,22 @@ static int lpss_resume_early(struct device *dev)
 
 static int acpi_lpss_resume_early(struct device *dev)
 {
-	int ret;
+	int ret = 0;
 
-	ret = lpss_resume_early(dev);
-	if (ret)
-		return ret;
+	if (!dev_pm_is_rpm_sleep(dev))
+		ret = lpss_resume_early(dev);
 
-	return pm_generic_resume_early(dev);
+	return ret ? ret : pm_generic_resume_early(dev);
+}
+#else
+static inline int lpss_suspend_late(struct device *dev)
+{
+	return 0;
+}
+
+static inline int lpss_resume_early(struct device *dev)
+{
+	return 0;
 }
 #endif /* CONFIG_PM_SLEEP */
 
@@ -861,6 +870,9 @@ static int acpi_lpss_runtime_suspend(struct device *dev)
 	if (ret)
 		return ret;
 
+	if (!pm_runtime_enabled(dev))
+		return lpss_suspend_late(dev);
+
 	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
 		acpi_lpss_save_ctx(dev, pdata);
 
@@ -882,21 +894,29 @@ static int acpi_lpss_runtime_resume(struct device *dev)
 	struct lpss_private_data *pdata = acpi_driver_data(ACPI_COMPANION(dev));
 	int ret;
 
-	/*
-	 * This call is kept first to be in symmetry with
-	 * acpi_lpss_runtime_suspend() one.
-	 */
-	if (lpss_quirks & LPSS_QUIRK_ALWAYS_POWER_ON && iosf_mbi_available())
-		lpss_iosf_exit_d3_state();
+	if (pm_runtime_enabled(dev)) {
+		/*
+		 * This call is kept first to be in symmetry with
+		 * acpi_lpss_runtime_suspend() one.
+		 */
+		if (lpss_quirks & LPSS_QUIRK_ALWAYS_POWER_ON &&
+		    iosf_mbi_available())
+			lpss_iosf_exit_d3_state();
 
-	ret = acpi_dev_runtime_resume(dev);
-	if (ret)
-		return ret;
+		ret = acpi_dev_runtime_resume(dev);
+		if (ret)
+			return ret;
 
-	acpi_lpss_d3_to_d0_delay(pdata);
+		acpi_lpss_d3_to_d0_delay(pdata);
 
-	if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
-		acpi_lpss_restore_ctx(dev, pdata);
+		if (pdata->dev_desc->flags & LPSS_SAVE_CTX)
+			acpi_lpss_restore_ctx(dev, pdata);
+
+	} else {
+		ret = lpss_resume_early(dev);
+		if (ret)
+			return ret;
+	}
 
 	return pm_generic_runtime_resume(dev);
 }
diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index 5181057..f5c4d0e 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -913,7 +913,14 @@ EXPORT_SYMBOL_GPL(acpi_dev_runtime_resume);
 int acpi_subsys_runtime_suspend(struct device *dev)
 {
 	int ret = pm_generic_runtime_suspend(dev);
-	return ret ? ret : acpi_dev_runtime_suspend(dev);
+
+	if (ret)
+		return ret;
+
+	if (!pm_runtime_enabled(dev))
+		return acpi_dev_suspend_late(dev);
+
+	return acpi_dev_runtime_suspend(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_runtime_suspend);
 
@@ -926,7 +933,13 @@ EXPORT_SYMBOL_GPL(acpi_subsys_runtime_suspend);
  */
 int acpi_subsys_runtime_resume(struct device *dev)
 {
-	int ret = acpi_dev_runtime_resume(dev);
+	int ret = 0;
+
+	if (!pm_runtime_enabled(dev))
+		ret = acpi_dev_resume_early(dev);
+	else
+		ret = acpi_dev_runtime_resume(dev);
+
 	return ret ? ret : pm_generic_runtime_resume(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_runtime_resume);
@@ -1023,7 +1036,7 @@ int acpi_subsys_prepare(struct device *dev)
 	if (ret < 0)
 		return ret;
 
-	if (!adev || !pm_runtime_suspended(dev))
+	if (!adev || dev_pm_is_rpm_sleep(dev) || !pm_runtime_suspended(dev))
 		return 0;
 
 	return !acpi_dev_needs_resume(dev, adev);
@@ -1042,7 +1055,8 @@ void acpi_subsys_complete(struct device *dev)
 	 * the sleep state it is going out of and it has never been resumed till
 	 * now, resume it in case the firmware powered it up.
 	 */
-	if (dev->power.direct_complete && pm_resume_via_firmware())
+	if ((dev->power.direct_complete || dev_pm_is_rpm_sleep(dev)) &&
+	    pm_resume_via_firmware())
 		pm_request_resume(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_complete);
@@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
  * @dev: Device to handle.
  *
  * Follow PCI and resume devices suspended at run time before running their
- * system suspend callbacks.
+ * system suspend callbacks. However, try to avoid it in case the runtime PM
+ * centric path is used for the device and then trust the driver to do the
+ * right thing.
  */
 int acpi_subsys_suspend(struct device *dev)
 {
-	pm_runtime_resume(dev);
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+
+	if (!adev)
+		return 0;
+
+	if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
+		pm_runtime_resume(dev);
+
 	return pm_generic_suspend(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
@@ -1071,7 +1094,11 @@ EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
 int acpi_subsys_suspend_late(struct device *dev)
 {
 	int ret = pm_generic_suspend_late(dev);
-	return ret ? ret : acpi_dev_suspend_late(dev);
+
+	if (ret || dev_pm_is_rpm_sleep(dev))
+		return ret;
+
+	return acpi_dev_suspend_late(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_suspend_late);
 
@@ -1085,7 +1112,11 @@ EXPORT_SYMBOL_GPL(acpi_subsys_suspend_late);
  */
 int acpi_subsys_resume_early(struct device *dev)
 {
-	int ret = acpi_dev_resume_early(dev);
+	int ret = 0;
+
+	if (!dev_pm_is_rpm_sleep(dev))
+		ret = acpi_dev_resume_early(dev);
+
 	return ret ? ret : pm_generic_resume_early(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_resume_early);
@@ -1096,13 +1127,21 @@ EXPORT_SYMBOL_GPL(acpi_subsys_resume_early);
  */
 int acpi_subsys_freeze(struct device *dev)
 {
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+
+	if (!adev)
+		return 0;
+
 	/*
 	 * This used to be done in acpi_subsys_prepare() for all devices and
 	 * some drivers may depend on it, so do it here.  Ideally, however,
 	 * runtime-suspended devices should not be touched during freeze/thaw
-	 * transitions.
+	 * transitions. In case the runtime PM centric path is used, let's try
+	 * to avoid it.
 	 */
-	pm_runtime_resume(dev);
+	if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
+		pm_runtime_resume(dev);
+
 	return pm_generic_freeze(dev);
 }
 EXPORT_SYMBOL_GPL(acpi_subsys_freeze);
-- 
2.7.4

[PATCH v3 7/8] i2c: designware: Don't resume device in the ->complete() callback

[Ulf Hansson]

In case the PM core is able to use the direct_complete path during system
sleep for the i2c device, the device is runtime resumed in the ->complete()
callback. For ACPI platforms this is needed to synchronize the power state
of the device, while for non-ACPI platforms this is a waste.

To better deal with this, let's drop the ->complete() callback from the
i2c-dw-plat driver altogether, thus avoiding the runtime resume of the
device.

This change still plays well for the ACPI case, because the ACPI PM
domain's ->complete() callback, assigned to acpi_subsys_complete(), already
deals with runtime resuming the device in case it's needed.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/i2c/busses/i2c-designware-platdrv.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/drivers/i2c/busses/i2c-designware-platdrv.c b/drivers/i2c/busses/i2c-designware-platdrv.c
index 57248bc..38121c9 100644
--- a/drivers/i2c/busses/i2c-designware-platdrv.c
+++ b/drivers/i2c/busses/i2c-designware-platdrv.c
@@ -417,15 +417,8 @@ static int dw_i2c_plat_prepare(struct device *dev)
 {
 	return pm_runtime_suspended(dev);
 }
-
-static void dw_i2c_plat_complete(struct device *dev)
-{
-	if (dev->power.direct_complete)
-		pm_request_resume(dev);
-}
 #else
 #define dw_i2c_plat_prepare	NULL
-#define dw_i2c_plat_complete	NULL
 #endif
 
 #ifdef CONFIG_PM
@@ -461,7 +454,6 @@ static int dw_i2c_plat_suspend(struct device *dev)
 
 static const struct dev_pm_ops dw_i2c_dev_pm_ops = {
 	.prepare = dw_i2c_plat_prepare,
-	.complete = dw_i2c_plat_complete,
 	SET_SYSTEM_SLEEP_PM_OPS(dw_i2c_plat_suspend, dw_i2c_plat_resume)
 	SET_RUNTIME_PM_OPS(dw_i2c_plat_runtime_suspend,
 			   dw_i2c_plat_resume,
-- 
2.7.4

[PATCH v3 7/8] i2c: designware: Don't resume device in the ->complete() callback

[Ulf Hansson]

In case the PM core is able to use the direct_complete path during system
sleep for the i2c device, the device is runtime resumed in the ->complete()
callback. For ACPI platforms this is needed to synchronize the power state
of the device, while for non-ACPI platforms this is a waste.

To better deal with this, let's drop the ->complete() callback from the
i2c-dw-plat driver altogether, thus avoiding the runtime resume of the
device.

This change still plays well for the ACPI case, because the ACPI PM
domain's ->complete() callback, assigned to acpi_subsys_complete(), already
deals with runtime resuming the device in case it's needed.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- None.

---
 drivers/i2c/busses/i2c-designware-platdrv.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/drivers/i2c/busses/i2c-designware-platdrv.c b/drivers/i2c/busses/i2c-designware-platdrv.c
index 57248bc..38121c9 100644
--- a/drivers/i2c/busses/i2c-designware-platdrv.c
+++ b/drivers/i2c/busses/i2c-designware-platdrv.c
@@ -417,15 +417,8 @@ static int dw_i2c_plat_prepare(struct device *dev)
 {
 	return pm_runtime_suspended(dev);
 }
-
-static void dw_i2c_plat_complete(struct device *dev)
-{
-	if (dev->power.direct_complete)
-		pm_request_resume(dev);
-}
 #else
 #define dw_i2c_plat_prepare	NULL
-#define dw_i2c_plat_complete	NULL
 #endif
 
 #ifdef CONFIG_PM
@@ -461,7 +454,6 @@ static int dw_i2c_plat_suspend(struct device *dev)
 
 static const struct dev_pm_ops dw_i2c_dev_pm_ops = {
 	.prepare = dw_i2c_plat_prepare,
-	.complete = dw_i2c_plat_complete,
 	SET_SYSTEM_SLEEP_PM_OPS(dw_i2c_plat_suspend, dw_i2c_plat_resume)
 	SET_RUNTIME_PM_OPS(dw_i2c_plat_runtime_suspend,
 			   dw_i2c_plat_resume,
-- 
2.7.4

[PATCH v3 8/8] i2c: designware: Deploy the runtime PM centric path for system sleep

[Ulf Hansson]

Currently the device is runtime resumed in the i2c-dw-plat driver's
->suspend() callback, which is needed to manage system sleep properly.

The particular reason for the runtime resume is because the PM core may
unset the direct_complete flag for a parent device, in case its child
device are being system suspended before. This leads to that the
i2c-dw-plat driver's ->suspend() callback can be invoked when the device is
runtime suspended.

Runtime resuming the device in this scenario may be unnecessary, in case
when the device is already in its proper lower power state for system
sleep. This behaviour increases the time it takes to put the device into
low power state, but means also a waste of power.

Let's fix the behaviour by deploying the runtime PM centric path for system
sleep, via assigning the pm_runtime_force_suspend|resume() helpers as the
system sleep callbacks for the i2c-dw-plat driver. We must also inform the
PM core and the ACPI PM domain that we are now using the runtime PM centric
path, therefore we call dev_pm_use_rpm_sleep() during ->probe().

While deploying the runtime PM centric path, the ->prepare() callback
becomes redundant, so let's drop it.

Along with these improvements, of course we also get the other regular
benefits one get when deploying the runtime PM centric path.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- Use dev_pm_use_rpm_sleep() instead of the earlier
	acpi_disable_direct_complete() from v2.
	- Updated changelog.

---
 drivers/i2c/busses/i2c-designware-platdrv.c | 26 +++++---------------------
 1 file changed, 5 insertions(+), 21 deletions(-)

diff --git a/drivers/i2c/busses/i2c-designware-platdrv.c b/drivers/i2c/busses/i2c-designware-platdrv.c
index 38121c9..20db3c2 100644
--- a/drivers/i2c/busses/i2c-designware-platdrv.c
+++ b/drivers/i2c/busses/i2c-designware-platdrv.c
@@ -371,6 +371,7 @@ static int dw_i2c_plat_probe(struct platform_device *pdev)
 	if (ret)
 		goto exit_probe;
 
+	dev_pm_use_rpm_sleep(&pdev->dev);
 	return ret;
 
 exit_probe:
@@ -412,15 +413,6 @@ static const struct of_device_id dw_i2c_of_match[] = {
 MODULE_DEVICE_TABLE(of, dw_i2c_of_match);
 #endif
 
-#ifdef CONFIG_PM_SLEEP
-static int dw_i2c_plat_prepare(struct device *dev)
-{
-	return pm_runtime_suspended(dev);
-}
-#else
-#define dw_i2c_plat_prepare	NULL
-#endif
-
 #ifdef CONFIG_PM
 static int dw_i2c_plat_runtime_suspend(struct device *dev)
 {
@@ -433,7 +425,7 @@ static int dw_i2c_plat_runtime_suspend(struct device *dev)
 	return 0;
 }
 
-static int dw_i2c_plat_resume(struct device *dev)
+static int dw_i2c_plat_runtime_resume(struct device *dev)
 {
 	struct platform_device *pdev = to_platform_device(dev);
 	struct dw_i2c_dev *i_dev = platform_get_drvdata(pdev);
@@ -444,19 +436,11 @@ static int dw_i2c_plat_resume(struct device *dev)
 	return 0;
 }
 
-#ifdef CONFIG_PM_SLEEP
-static int dw_i2c_plat_suspend(struct device *dev)
-{
-	pm_runtime_resume(dev);
-	return dw_i2c_plat_runtime_suspend(dev);
-}
-#endif
-
 static const struct dev_pm_ops dw_i2c_dev_pm_ops = {
-	.prepare = dw_i2c_plat_prepare,
-	SET_SYSTEM_SLEEP_PM_OPS(dw_i2c_plat_suspend, dw_i2c_plat_resume)
+	SET_LATE_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+				     pm_runtime_force_resume)
 	SET_RUNTIME_PM_OPS(dw_i2c_plat_runtime_suspend,
-			   dw_i2c_plat_resume,
+			   dw_i2c_plat_runtime_resume,
 			   NULL)
 };
 
-- 
2.7.4

[PATCH v3 8/8] i2c: designware: Deploy the runtime PM centric path for system sleep

[Ulf Hansson]

Currently the device is runtime resumed in the i2c-dw-plat driver's
->suspend() callback, which is needed to manage system sleep properly.

The particular reason for the runtime resume is because the PM core may
unset the direct_complete flag for a parent device, in case its child
device are being system suspended before. This leads to that the
i2c-dw-plat driver's ->suspend() callback can be invoked when the device is
runtime suspended.

Runtime resuming the device in this scenario may be unnecessary, in case
when the device is already in its proper lower power state for system
sleep. This behaviour increases the time it takes to put the device into
low power state, but means also a waste of power.

Let's fix the behaviour by deploying the runtime PM centric path for system
sleep, via assigning the pm_runtime_force_suspend|resume() helpers as the
system sleep callbacks for the i2c-dw-plat driver. We must also inform the
PM core and the ACPI PM domain that we are now using the runtime PM centric
path, therefore we call dev_pm_use_rpm_sleep() during ->probe().

While deploying the runtime PM centric path, the ->prepare() callback
becomes redundant, so let's drop it.

Along with these improvements, of course we also get the other regular
benefits one get when deploying the runtime PM centric path.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---

Changes in v3:
	- Use dev_pm_use_rpm_sleep() instead of the earlier
	acpi_disable_direct_complete() from v2.
	- Updated changelog.

---
 drivers/i2c/busses/i2c-designware-platdrv.c | 26 +++++---------------------
 1 file changed, 5 insertions(+), 21 deletions(-)

diff --git a/drivers/i2c/busses/i2c-designware-platdrv.c b/drivers/i2c/busses/i2c-designware-platdrv.c
index 38121c9..20db3c2 100644
--- a/drivers/i2c/busses/i2c-designware-platdrv.c
+++ b/drivers/i2c/busses/i2c-designware-platdrv.c
@@ -371,6 +371,7 @@ static int dw_i2c_plat_probe(struct platform_device *pdev)
 	if (ret)
 		goto exit_probe;
 
+	dev_pm_use_rpm_sleep(&pdev->dev);
 	return ret;
 
 exit_probe:
@@ -412,15 +413,6 @@ static const struct of_device_id dw_i2c_of_match[] = {
 MODULE_DEVICE_TABLE(of, dw_i2c_of_match);
 #endif
 
-#ifdef CONFIG_PM_SLEEP
-static int dw_i2c_plat_prepare(struct device *dev)
-{
-	return pm_runtime_suspended(dev);
-}
-#else
-#define dw_i2c_plat_prepare	NULL
-#endif
-
 #ifdef CONFIG_PM
 static int dw_i2c_plat_runtime_suspend(struct device *dev)
 {
@@ -433,7 +425,7 @@ static int dw_i2c_plat_runtime_suspend(struct device *dev)
 	return 0;
 }
 
-static int dw_i2c_plat_resume(struct device *dev)
+static int dw_i2c_plat_runtime_resume(struct device *dev)
 {
 	struct platform_device *pdev = to_platform_device(dev);
 	struct dw_i2c_dev *i_dev = platform_get_drvdata(pdev);
@@ -444,19 +436,11 @@ static int dw_i2c_plat_resume(struct device *dev)
 	return 0;
 }
 
-#ifdef CONFIG_PM_SLEEP
-static int dw_i2c_plat_suspend(struct device *dev)
-{
-	pm_runtime_resume(dev);
-	return dw_i2c_plat_runtime_suspend(dev);
-}
-#endif
-
 static const struct dev_pm_ops dw_i2c_dev_pm_ops = {
-	.prepare = dw_i2c_plat_prepare,
-	SET_SYSTEM_SLEEP_PM_OPS(dw_i2c_plat_suspend, dw_i2c_plat_resume)
+	SET_LATE_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+				     pm_runtime_force_resume)
 	SET_RUNTIME_PM_OPS(dw_i2c_plat_runtime_suspend,
-			   dw_i2c_plat_resume,
+			   dw_i2c_plat_runtime_resume,
 			   NULL)
 };
 
-- 
2.7.4

Re: [PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Rafael J. Wysocki]

On Tuesday, August 29, 2017 4:56:43 PM CEST Ulf Hansson wrote:
> The main principle behind the runtime PM centric path, is to re-use the
> runtime PM callbacks to implement system sleep - and while doing that also
> achieve a fully optimized behaviour from PM point of view.
> 
> More precisely, avoid to wake up a device from its low power state during
> system sleep, unless the device is really needed to be operational. That
> does not only mean avoiding to waste power, but may also decrease system
> suspend/resume time for a device.
> 
> However, using the runtime PM centric path for a device, does put some
> requirements on the behaviour of the PM core and a potential PM domain that
> may be attached to the device. So far, these requirements are not being
> specially considered, except by the generic PM domain.
> 
> To move forward and to make it possible to deploy the runtime PM centric
> path for cross SoC drivers, which may have different PM domains attached to
> its devices depending on the SoC, we must address how to deal with these
> requirements. This change starts by making some adoptions to the PM core,
> while other parts, such as the ACPI PM domain needs to be taken care of
> separately.
> 
> In the runtime PM centric path, the driver is expected to make use of the
> pm_runtime_force_suspend|resume() helpers, to deploy system sleep support.
> More precisely it may assign the system sleep callbacks to these helpers or
> may call them from its own callbacks, in case it needs to perform
> additional actions during system sleep.
> 
> In other words, the PM core must always invoke the system sleep callbacks
> for the device when they are present, to allow the driver to deal with
> the system sleep operations.
> 
> In case the PM core decides to run the direct_complete path for the device,
> it skips invoking most of the system sleep callbacks, besides
> ->prepare|complete(). Therefore using the direct_complete path in
> combination with the runtime PM centric patch for a device, does not play
> well.
> 
> To deal with this issue, let's add a flag 'is_rpm_sleep', to the struct
> dev_pm_info. The driver that deploys the runtime PM centric path, shall set
> the flag for the device during ->probe(), to inform the PM core about that
> it must not use the direct_complete path for the device.
> 
> Note, not allowing the direct_complete path for a device, doesn't implicit
> need to propagate to the device's parent/suppliers. Therefore make the PM
> core check this condition in device_suspend(), before it decides to abandon
> the direct_complete path for parent/suppliers.
> 
> To make the is_rpm_sleep flag internal to the PM core, let's add two APIs.
> 	- dev_pm_use_rpm_sleep(): It sets the flag and should be called by
> 	  the driver during ->probe().
> 	- dev_pm_is_rpm_sleep(): Makes it possible for users of the device,
> 	  like a PM domain, to fetch the state of the flag.
> 
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
> 
> Changes in v3:
> 	- New patch.
> 	- This replaces the earlier method of adding the no_direct_complete to
> 	the ACPI structures, according to comments from Rafael.
> 	- This change also address the consern Rafael had around that
> 	direct_complete should not have to be disabled for parent/suppliers, in
> 	case a device use the runtime PM centric path for system sleep.
> 
> ---
>  drivers/base/power/main.c    | 49 +++++++++++++++++++++++++++++++++++++++-----
>  drivers/base/power/runtime.c |  1 +
>  include/linux/pm.h           |  7 +++++++
>  3 files changed, 52 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> index ea1732e..865737a 100644
> --- a/drivers/base/power/main.c
> +++ b/drivers/base/power/main.c
> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
>  		if (parent) {
>  			spin_lock_irq(&parent->power.lock);
>  
> -			dev->parent->power.direct_complete = false;
> +			if (!dev->power.is_rpm_sleep)
> +				dev->parent->power.direct_complete = false;

This doesn't look good to me at all.

It potentially breaks the fundamental assumption of the direct_complete
mechanism that no devices with that flag set will ever be runtime resumed
during system suspend.

Which can happen with this change if a device with power.is_rpm_sleep is
resumed causing the parent to be resumed too.

>  			if (dev->power.wakeup_path
>  			    && !dev->parent->power.ignore_children)
>  				dev->parent->power.wakeup_path = true;
>  
>  			spin_unlock_irq(&parent->power.lock);
>  		}
> -		dpm_clear_suppliers_direct_complete(dev);
> +		if (!dev->power.is_rpm_sleep)
> +			dpm_clear_suppliers_direct_complete(dev);

And same here.

>  	}
>  
>  	device_unlock(dev);
> @@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
>  	 * A positive return value from ->prepare() means "this device appears
>  	 * to be runtime-suspended and its state is fine, so if it really is
>  	 * runtime-suspended, you can leave it in that state provided that you
> -	 * will do the same thing with all of its descendants".  This only
> -	 * applies to suspend transitions, however.
> +	 * will do the same thing with all of its descendants". To allow this,
> +	 * the is_rpm_sleep flag must not be set, which is default false, but
> +	 * may have been changed by a driver. This only applies to suspend
> +	 * transitions, however.
>  	 */
>  	spin_lock_irq(&dev->power.lock);
> -	dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
> +	dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
> +				state.event == PM_EVENT_SUSPEND;

The only problem addressed by avoiding direct_complete is when runtime PM needs
to work during __device_suspend() for devices with direct_complete set.  You
said that this was not the case with the i2c designware driver, so I'm not sure
what the purpose of this is.

It doesn't solve any other problems at all.

>  	spin_unlock_irq(&dev->power.lock);
>  	return 0;
>  }
> @@ -1841,6 +1846,40 @@ void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *))
>  }
>  EXPORT_SYMBOL_GPL(dpm_for_each_dev);
>  
> +/**
> + * dev_pm_use_rpm_sleep - use the runtime PM centric sleep path.
> + * @dev: the device to use the path for.
> + *
> + * The runtime PM centric path requires the system sleep callbacks for the
> + * driver of the device to be invoked. This function sets the is_rpm_sleep flag
> + * to enable this path to be used, which makes the PM core to adopt to this
> + * behaviour. More precisely the PM core makes sure to not run the
> + * direct_complete path for the chosen device during system sleep, when the
> + * is_rpm_sleep flag is set. A driver using the runtime PM centric path, shall
> + * use the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
> + * support and call this function during ->probe().
> + * The is_rpm_sleep flag is cleared when unbinding/bind-failure of the driver.
> + */
> +void dev_pm_use_rpm_sleep(struct device *dev)
> +{
> +	dev->power.is_rpm_sleep = true;
> +}
> +EXPORT_SYMBOL_GPL(dev_pm_use_rpm_sleep);
> +
> +/**
> + * dev_pm_is_rpm_sleep - Returns the value of the is_rpm_sleep flag.
> + * @dev: the device to return the flag for.
> + *
> + * The caller of this function is typically a subsystem or a PM domain that
> + * needs to know if the runtime PM centric path is used for the device. It may
> + * be invoked during any of the system sleep phases.
> + */
> +bool dev_pm_is_rpm_sleep(struct device *dev)
> +{
> +	return dev->power.is_rpm_sleep;
> +}
> +EXPORT_SYMBOL_GPL(dev_pm_is_rpm_sleep);
> +
>  static bool pm_ops_is_empty(const struct dev_pm_ops *ops)
>  {
>  	if (!ops)
> diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c
> index 7bcf80f..b2ab22c 100644
> --- a/drivers/base/power/runtime.c
> +++ b/drivers/base/power/runtime.c
> @@ -1522,6 +1522,7 @@ void pm_runtime_reinit(struct device *dev)
>  				pm_runtime_put(dev->parent);
>  		}
>  	}
> +	dev->power.is_rpm_sleep = false;
>  }
>  
>  /**
> diff --git a/include/linux/pm.h b/include/linux/pm.h
> index 47ded8a..5bf96d2 100644
> --- a/include/linux/pm.h
> +++ b/include/linux/pm.h
> @@ -559,6 +559,7 @@ struct dev_pm_info {
>  	bool			is_suspended:1;	/* Ditto */
>  	bool			is_noirq_suspended:1;
>  	bool			is_late_suspended:1;
> +	bool			is_rpm_sleep:1;	/* Owned by the PM core */

It is expected to be set by drivers, so by definition it is *not* owned by the
PM core.

>  	bool			early_init:1;	/* Owned by the PM core */
>  	bool			direct_complete:1;	/* Owned by the PM core */
>  	spinlock_t		lock;
> @@ -716,6 +717,9 @@ extern void __suspend_report_result(const char *function, void *fn, int ret);
>  extern int device_pm_wait_for_dev(struct device *sub, struct device *dev);
>  extern void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *));
>  
> +extern void dev_pm_use_rpm_sleep(struct device *dev);
> +extern bool dev_pm_is_rpm_sleep(struct device *dev);
> +
>  extern int pm_generic_prepare(struct device *dev);
>  extern int pm_generic_suspend_late(struct device *dev);
>  extern int pm_generic_suspend_noirq(struct device *dev);
> @@ -759,6 +763,9 @@ static inline void dpm_for_each_dev(void *data, void (*fn)(struct device *, void
>  {
>  }
>  
> +static inline void dev_pm_use_rpm_sleep(struct device *dev) {}
> +static inline bool dev_pm_is_rpm_sleep(struct device *dev) { return false; }
> +
>  #define pm_generic_prepare		NULL
>  #define pm_generic_suspend_late		NULL
>  #define pm_generic_suspend_noirq	NULL
> 

Thanks,
Rafael

[PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Rafael J. Wysocki]

On Tuesday, August 29, 2017 4:56:43 PM CEST Ulf Hansson wrote:
> The main principle behind the runtime PM centric path, is to re-use the
> runtime PM callbacks to implement system sleep - and while doing that also
> achieve a fully optimized behaviour from PM point of view.
> 
> More precisely, avoid to wake up a device from its low power state during
> system sleep, unless the device is really needed to be operational. That
> does not only mean avoiding to waste power, but may also decrease system
> suspend/resume time for a device.
> 
> However, using the runtime PM centric path for a device, does put some
> requirements on the behaviour of the PM core and a potential PM domain that
> may be attached to the device. So far, these requirements are not being
> specially considered, except by the generic PM domain.
> 
> To move forward and to make it possible to deploy the runtime PM centric
> path for cross SoC drivers, which may have different PM domains attached to
> its devices depending on the SoC, we must address how to deal with these
> requirements. This change starts by making some adoptions to the PM core,
> while other parts, such as the ACPI PM domain needs to be taken care of
> separately.
> 
> In the runtime PM centric path, the driver is expected to make use of the
> pm_runtime_force_suspend|resume() helpers, to deploy system sleep support.
> More precisely it may assign the system sleep callbacks to these helpers or
> may call them from its own callbacks, in case it needs to perform
> additional actions during system sleep.
> 
> In other words, the PM core must always invoke the system sleep callbacks
> for the device when they are present, to allow the driver to deal with
> the system sleep operations.
> 
> In case the PM core decides to run the direct_complete path for the device,
> it skips invoking most of the system sleep callbacks, besides
> ->prepare|complete(). Therefore using the direct_complete path in
> combination with the runtime PM centric patch for a device, does not play
> well.
> 
> To deal with this issue, let's add a flag 'is_rpm_sleep', to the struct
> dev_pm_info. The driver that deploys the runtime PM centric path, shall set
> the flag for the device during ->probe(), to inform the PM core about that
> it must not use the direct_complete path for the device.
> 
> Note, not allowing the direct_complete path for a device, doesn't implicit
> need to propagate to the device's parent/suppliers. Therefore make the PM
> core check this condition in device_suspend(), before it decides to abandon
> the direct_complete path for parent/suppliers.
> 
> To make the is_rpm_sleep flag internal to the PM core, let's add two APIs.
> 	- dev_pm_use_rpm_sleep(): It sets the flag and should be called by
> 	  the driver during ->probe().
> 	- dev_pm_is_rpm_sleep(): Makes it possible for users of the device,
> 	  like a PM domain, to fetch the state of the flag.
> 
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
> 
> Changes in v3:
> 	- New patch.
> 	- This replaces the earlier method of adding the no_direct_complete to
> 	the ACPI structures, according to comments from Rafael.
> 	- This change also address the consern Rafael had around that
> 	direct_complete should not have to be disabled for parent/suppliers, in
> 	case a device use the runtime PM centric path for system sleep.
> 
> ---
>  drivers/base/power/main.c    | 49 +++++++++++++++++++++++++++++++++++++++-----
>  drivers/base/power/runtime.c |  1 +
>  include/linux/pm.h           |  7 +++++++
>  3 files changed, 52 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> index ea1732e..865737a 100644
> --- a/drivers/base/power/main.c
> +++ b/drivers/base/power/main.c
> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
>  		if (parent) {
>  			spin_lock_irq(&parent->power.lock);
>  
> -			dev->parent->power.direct_complete = false;
> +			if (!dev->power.is_rpm_sleep)
> +				dev->parent->power.direct_complete = false;

This doesn't look good to me at all.

It potentially breaks the fundamental assumption of the direct_complete
mechanism that no devices with that flag set will ever be runtime resumed
during system suspend.

Which can happen with this change if a device with power.is_rpm_sleep is
resumed causing the parent to be resumed too.

>  			if (dev->power.wakeup_path
>  			    && !dev->parent->power.ignore_children)
>  				dev->parent->power.wakeup_path = true;
>  
>  			spin_unlock_irq(&parent->power.lock);
>  		}
> -		dpm_clear_suppliers_direct_complete(dev);
> +		if (!dev->power.is_rpm_sleep)
> +			dpm_clear_suppliers_direct_complete(dev);

And same here.

>  	}
>  
>  	device_unlock(dev);
> @@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
>  	 * A positive return value from ->prepare() means "this device appears
>  	 * to be runtime-suspended and its state is fine, so if it really is
>  	 * runtime-suspended, you can leave it in that state provided that you
> -	 * will do the same thing with all of its descendants".  This only
> -	 * applies to suspend transitions, however.
> +	 * will do the same thing with all of its descendants". To allow this,
> +	 * the is_rpm_sleep flag must not be set, which is default false, but
> +	 * may have been changed by a driver. This only applies to suspend
> +	 * transitions, however.
>  	 */
>  	spin_lock_irq(&dev->power.lock);
> -	dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
> +	dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
> +				state.event == PM_EVENT_SUSPEND;

The only problem addressed by avoiding direct_complete is when runtime PM needs
to work during __device_suspend() for devices with direct_complete set.  You
said that this was not the case with the i2c designware driver, so I'm not sure
what the purpose of this is.

It doesn't solve any other problems at all.

>  	spin_unlock_irq(&dev->power.lock);
>  	return 0;
>  }
> @@ -1841,6 +1846,40 @@ void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *))
>  }
>  EXPORT_SYMBOL_GPL(dpm_for_each_dev);
>  
> +/**
> + * dev_pm_use_rpm_sleep - use the runtime PM centric sleep path.
> + * @dev: the device to use the path for.
> + *
> + * The runtime PM centric path requires the system sleep callbacks for the
> + * driver of the device to be invoked. This function sets the is_rpm_sleep flag
> + * to enable this path to be used, which makes the PM core to adopt to this
> + * behaviour. More precisely the PM core makes sure to not run the
> + * direct_complete path for the chosen device during system sleep, when the
> + * is_rpm_sleep flag is set. A driver using the runtime PM centric path, shall
> + * use the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
> + * support and call this function during ->probe().
> + * The is_rpm_sleep flag is cleared when unbinding/bind-failure of the driver.
> + */
> +void dev_pm_use_rpm_sleep(struct device *dev)
> +{
> +	dev->power.is_rpm_sleep = true;
> +}
> +EXPORT_SYMBOL_GPL(dev_pm_use_rpm_sleep);
> +
> +/**
> + * dev_pm_is_rpm_sleep - Returns the value of the is_rpm_sleep flag.
> + * @dev: the device to return the flag for.
> + *
> + * The caller of this function is typically a subsystem or a PM domain that
> + * needs to know if the runtime PM centric path is used for the device. It may
> + * be invoked during any of the system sleep phases.
> + */
> +bool dev_pm_is_rpm_sleep(struct device *dev)
> +{
> +	return dev->power.is_rpm_sleep;
> +}
> +EXPORT_SYMBOL_GPL(dev_pm_is_rpm_sleep);
> +
>  static bool pm_ops_is_empty(const struct dev_pm_ops *ops)
>  {
>  	if (!ops)
> diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c
> index 7bcf80f..b2ab22c 100644
> --- a/drivers/base/power/runtime.c
> +++ b/drivers/base/power/runtime.c
> @@ -1522,6 +1522,7 @@ void pm_runtime_reinit(struct device *dev)
>  				pm_runtime_put(dev->parent);
>  		}
>  	}
> +	dev->power.is_rpm_sleep = false;
>  }
>  
>  /**
> diff --git a/include/linux/pm.h b/include/linux/pm.h
> index 47ded8a..5bf96d2 100644
> --- a/include/linux/pm.h
> +++ b/include/linux/pm.h
> @@ -559,6 +559,7 @@ struct dev_pm_info {
>  	bool			is_suspended:1;	/* Ditto */
>  	bool			is_noirq_suspended:1;
>  	bool			is_late_suspended:1;
> +	bool			is_rpm_sleep:1;	/* Owned by the PM core */

It is expected to be set by drivers, so by definition it is *not* owned by the
PM core.

>  	bool			early_init:1;	/* Owned by the PM core */
>  	bool			direct_complete:1;	/* Owned by the PM core */
>  	spinlock_t		lock;
> @@ -716,6 +717,9 @@ extern void __suspend_report_result(const char *function, void *fn, int ret);
>  extern int device_pm_wait_for_dev(struct device *sub, struct device *dev);
>  extern void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *));
>  
> +extern void dev_pm_use_rpm_sleep(struct device *dev);
> +extern bool dev_pm_is_rpm_sleep(struct device *dev);
> +
>  extern int pm_generic_prepare(struct device *dev);
>  extern int pm_generic_suspend_late(struct device *dev);
>  extern int pm_generic_suspend_noirq(struct device *dev);
> @@ -759,6 +763,9 @@ static inline void dpm_for_each_dev(void *data, void (*fn)(struct device *, void
>  {
>  }
>  
> +static inline void dev_pm_use_rpm_sleep(struct device *dev) {}
> +static inline bool dev_pm_is_rpm_sleep(struct device *dev) { return false; }
> +
>  #define pm_generic_prepare		NULL
>  #define pm_generic_suspend_late		NULL
>  #define pm_generic_suspend_noirq	NULL
> 

Thanks,
Rafael

Re: [PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Rafael J. Wysocki]

On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
> This change enables the ACPI PM domain to cope with drivers that deploys
> the runtime PM centric path for system sleep.

[cut]

> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
>   * @dev: Device to handle.
>   *
>   * Follow PCI and resume devices suspended at run time before running their
> - * system suspend callbacks.
> + * system suspend callbacks. However, try to avoid it in case the runtime PM
> + * centric path is used for the device and then trust the driver to do the
> + * right thing.
>   */
>  int acpi_subsys_suspend(struct device *dev)
>  {
> -	pm_runtime_resume(dev);
> +	struct acpi_device *adev = ACPI_COMPANION(dev);
> +
> +	if (!adev)
> +		return 0;
> +
> +	if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
> +		pm_runtime_resume(dev);
> +
>  	return pm_generic_suspend(dev);
>  }
>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);

Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
and that's why I added the update_state thing.

Moreover, the is_rpm_sleep flag here has to mean not only that
direct_complete should not be used with the device, but also that its driver
is fine with not resuming it.

IMO it is not a good idea to use one flag for these two different things at the
same time at all.

Thanks,
Rafael

[PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Rafael J. Wysocki]

On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
> This change enables the ACPI PM domain to cope with drivers that deploys
> the runtime PM centric path for system sleep.

[cut]

> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
>   * @dev: Device to handle.
>   *
>   * Follow PCI and resume devices suspended at run time before running their
> - * system suspend callbacks.
> + * system suspend callbacks. However, try to avoid it in case the runtime PM
> + * centric path is used for the device and then trust the driver to do the
> + * right thing.
>   */
>  int acpi_subsys_suspend(struct device *dev)
>  {
> -	pm_runtime_resume(dev);
> +	struct acpi_device *adev = ACPI_COMPANION(dev);
> +
> +	if (!adev)
> +		return 0;
> +
> +	if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
> +		pm_runtime_resume(dev);
> +
>  	return pm_generic_suspend(dev);
>  }
>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);

Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
and that's why I added the update_state thing.

Moreover, the is_rpm_sleep flag here has to mean not only that
direct_complete should not be used with the device, but also that its driver
is fine with not resuming it.

IMO it is not a good idea to use one flag for these two different things at the
same time at all.

Thanks,
Rafael

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
> isn't well optimized for system sleep.
> 
> What makes this driver particularly interesting is because it's a cross-SoC
> driver, which sometimes means there is an ACPI PM domain attached to the i2c
> device and sometimes not. The driver is being used on both x86 and ARM.
> 
> In principle, to optimize the system sleep support in i2c driver, this series
> enables the proven runtime PM centric path for the i2c driver. However, to do
> that the ACPI PM domain also have to collaborate and understand this behaviour.
> From earlier versions, Rafael has also pointed out that also the PM core needs
> to be involved.

Earlier today I realized that drivers pointing their ->suspend_late and
->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
pm_runtime_force_resume(), are fundamentally incompatible with any bus type
doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.

First, it basically requires the bus type or PM domain to expect that its
->runtime_suspend callback may or may not be indirectly invoked from its
own ->suspend_late callback, depending on the driver (and analogously
for ->runtime_resume and ->resume early), which is insane.

Second, it is a layering violation, because it makes the driver effectively
override the upper layer's decisions about what code to run.

That's why I'm afraid that we've reached a dead end here. :-(

Thanks,
Rafael

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
> isn't well optimized for system sleep.
> 
> What makes this driver particularly interesting is because it's a cross-SoC
> driver, which sometimes means there is an ACPI PM domain attached to the i2c
> device and sometimes not. The driver is being used on both x86 and ARM.
> 
> In principle, to optimize the system sleep support in i2c driver, this series
> enables the proven runtime PM centric path for the i2c driver. However, to do
> that the ACPI PM domain also have to collaborate and understand this behaviour.
> From earlier versions, Rafael has also pointed out that also the PM core needs
> to be involved.

Earlier today I realized that drivers pointing their ->suspend_late and
->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
pm_runtime_force_resume(), are fundamentally incompatible with any bus type
doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.

First, it basically requires the bus type or PM domain to expect that its
->runtime_suspend callback may or may not be indirectly invoked from its
own ->suspend_late callback, depending on the driver (and analogously
for ->runtime_resume and ->resume early), which is insane.

Second, it is a layering violation, because it makes the driver effectively
override the upper layer's decisions about what code to run.

That's why I'm afraid that we've reached a dead end here. :-(

Thanks,
Rafael

Re: [PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Ulf Hansson]

On 29 August 2017 at 17:15, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Tuesday, August 29, 2017 4:56:43 PM CEST Ulf Hansson wrote:
>> The main principle behind the runtime PM centric path, is to re-use the
>> runtime PM callbacks to implement system sleep - and while doing that also
>> achieve a fully optimized behaviour from PM point of view.
>>
>> More precisely, avoid to wake up a device from its low power state during
>> system sleep, unless the device is really needed to be operational. That
>> does not only mean avoiding to waste power, but may also decrease system
>> suspend/resume time for a device.
>>
>> However, using the runtime PM centric path for a device, does put some
>> requirements on the behaviour of the PM core and a potential PM domain that
>> may be attached to the device. So far, these requirements are not being
>> specially considered, except by the generic PM domain.
>>
>> To move forward and to make it possible to deploy the runtime PM centric
>> path for cross SoC drivers, which may have different PM domains attached to
>> its devices depending on the SoC, we must address how to deal with these
>> requirements. This change starts by making some adoptions to the PM core,
>> while other parts, such as the ACPI PM domain needs to be taken care of
>> separately.
>>
>> In the runtime PM centric path, the driver is expected to make use of the
>> pm_runtime_force_suspend|resume() helpers, to deploy system sleep support.
>> More precisely it may assign the system sleep callbacks to these helpers or
>> may call them from its own callbacks, in case it needs to perform
>> additional actions during system sleep.
>>
>> In other words, the PM core must always invoke the system sleep callbacks
>> for the device when they are present, to allow the driver to deal with
>> the system sleep operations.
>>
>> In case the PM core decides to run the direct_complete path for the device,
>> it skips invoking most of the system sleep callbacks, besides
>> ->prepare|complete(). Therefore using the direct_complete path in
>> combination with the runtime PM centric patch for a device, does not play
>> well.
>>
>> To deal with this issue, let's add a flag 'is_rpm_sleep', to the struct
>> dev_pm_info. The driver that deploys the runtime PM centric path, shall set
>> the flag for the device during ->probe(), to inform the PM core about that
>> it must not use the direct_complete path for the device.
>>
>> Note, not allowing the direct_complete path for a device, doesn't implicit
>> need to propagate to the device's parent/suppliers. Therefore make the PM
>> core check this condition in device_suspend(), before it decides to abandon
>> the direct_complete path for parent/suppliers.
>>
>> To make the is_rpm_sleep flag internal to the PM core, let's add two APIs.
>>       - dev_pm_use_rpm_sleep(): It sets the flag and should be called by
>>         the driver during ->probe().
>>       - dev_pm_is_rpm_sleep(): Makes it possible for users of the device,
>>         like a PM domain, to fetch the state of the flag.
>>
>> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
>> ---
>>
>> Changes in v3:
>>       - New patch.
>>       - This replaces the earlier method of adding the no_direct_complete to
>>       the ACPI structures, according to comments from Rafael.
>>       - This change also address the consern Rafael had around that
>>       direct_complete should not have to be disabled for parent/suppliers, in
>>       case a device use the runtime PM centric path for system sleep.
>>
>> ---
>>  drivers/base/power/main.c    | 49 +++++++++++++++++++++++++++++++++++++++-----
>>  drivers/base/power/runtime.c |  1 +
>>  include/linux/pm.h           |  7 +++++++
>>  3 files changed, 52 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
>> index ea1732e..865737a 100644
>> --- a/drivers/base/power/main.c
>> +++ b/drivers/base/power/main.c
>> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
>>               if (parent) {
>>                       spin_lock_irq(&parent->power.lock);
>>
>> -                     dev->parent->power.direct_complete = false;
>> +                     if (!dev->power.is_rpm_sleep)
>> +                             dev->parent->power.direct_complete = false;
>
> This doesn't look good to me at all.
>
> It potentially breaks the fundamental assumption of the direct_complete
> mechanism that no devices with that flag set will ever be runtime resumed
> during system suspend.
>
> Which can happen with this change if a device with power.is_rpm_sleep is
> resumed causing the parent to be resumed too.

Doesn't the exact same problem you describe, already exist prior my change!?

That is, if the current device is runtime suspended and the
direct_complete flag is set for it, then __device_suspend() has
already disabled runtime PM for the device, when we reach this point.
That means, a following attempts to runtime resume the device will
fail and thus also to runtime resume its parent.

So to me, this is a different problem, related how to work out the
correct order of how to suspend devices.

>
>>                       if (dev->power.wakeup_path
>>                           && !dev->parent->power.ignore_children)
>>                               dev->parent->power.wakeup_path = true;
>>
>>                       spin_unlock_irq(&parent->power.lock);
>>               }
>> -             dpm_clear_suppliers_direct_complete(dev);
>> +             if (!dev->power.is_rpm_sleep)
>> +                     dpm_clear_suppliers_direct_complete(dev);
>
> And same here.

See comment above.

>
>>       }
>>
>>       device_unlock(dev);
>> @@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
>>        * A positive return value from ->prepare() means "this device appears
>>        * to be runtime-suspended and its state is fine, so if it really is
>>        * runtime-suspended, you can leave it in that state provided that you
>> -      * will do the same thing with all of its descendants".  This only
>> -      * applies to suspend transitions, however.
>> +      * will do the same thing with all of its descendants". To allow this,
>> +      * the is_rpm_sleep flag must not be set, which is default false, but
>> +      * may have been changed by a driver. This only applies to suspend
>> +      * transitions, however.
>>        */
>>       spin_lock_irq(&dev->power.lock);
>> -     dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
>> +     dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
>> +                             state.event == PM_EVENT_SUSPEND;
>
> The only problem addressed by avoiding direct_complete is when runtime PM needs
> to work during __device_suspend() for devices with direct_complete set.  You
> said that this was not the case with the i2c designware driver, so I'm not sure
> what the purpose of this is.

I should probably work more on my changelog, because I tried to
explain it there.

Anyway, let me quote the important parts, and this is not specific for
the i2c-designware-driver, but generic to drivers using
pm_runtime_force_suspend|resume().

"In the runtime PM centric path, the driver is expected to make use of
the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
support. More precisely it may assign the system sleep callbacks to
these helpers or may call them from its own callbacks, in case it
needs to perform additional actions during system sleep.

In other words, the PM core must always invoke the system sleep
callbacks for the device when they are present, to allow the driver to
deal with the system sleep operations."

A concrete example is drivers/spi/spi-fsl-espi.c, but one can easily find more.

>
> It doesn't solve any other problems at all.
>
>>       spin_unlock_irq(&dev->power.lock);
>>       return 0;
>>  }
>> @@ -1841,6 +1846,40 @@ void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *))
>>  }
>>  EXPORT_SYMBOL_GPL(dpm_for_each_dev);
>>
>> +/**
>> + * dev_pm_use_rpm_sleep - use the runtime PM centric sleep path.
>> + * @dev: the device to use the path for.
>> + *
>> + * The runtime PM centric path requires the system sleep callbacks for the
>> + * driver of the device to be invoked. This function sets the is_rpm_sleep flag
>> + * to enable this path to be used, which makes the PM core to adopt to this
>> + * behaviour. More precisely the PM core makes sure to not run the
>> + * direct_complete path for the chosen device during system sleep, when the
>> + * is_rpm_sleep flag is set. A driver using the runtime PM centric path, shall
>> + * use the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
>> + * support and call this function during ->probe().
>> + * The is_rpm_sleep flag is cleared when unbinding/bind-failure of the driver.
>> + */
>> +void dev_pm_use_rpm_sleep(struct device *dev)
>> +{
>> +     dev->power.is_rpm_sleep = true;
>> +}
>> +EXPORT_SYMBOL_GPL(dev_pm_use_rpm_sleep);
>> +
>> +/**
>> + * dev_pm_is_rpm_sleep - Returns the value of the is_rpm_sleep flag.
>> + * @dev: the device to return the flag for.
>> + *
>> + * The caller of this function is typically a subsystem or a PM domain that
>> + * needs to know if the runtime PM centric path is used for the device. It may
>> + * be invoked during any of the system sleep phases.
>> + */
>> +bool dev_pm_is_rpm_sleep(struct device *dev)
>> +{
>> +     return dev->power.is_rpm_sleep;
>> +}
>> +EXPORT_SYMBOL_GPL(dev_pm_is_rpm_sleep);
>> +
>>  static bool pm_ops_is_empty(const struct dev_pm_ops *ops)
>>  {
>>       if (!ops)
>> diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c
>> index 7bcf80f..b2ab22c 100644
>> --- a/drivers/base/power/runtime.c
>> +++ b/drivers/base/power/runtime.c
>> @@ -1522,6 +1522,7 @@ void pm_runtime_reinit(struct device *dev)
>>                               pm_runtime_put(dev->parent);
>>               }
>>       }
>> +     dev->power.is_rpm_sleep = false;
>>  }
>>
>>  /**
>> diff --git a/include/linux/pm.h b/include/linux/pm.h
>> index 47ded8a..5bf96d2 100644
>> --- a/include/linux/pm.h
>> +++ b/include/linux/pm.h
>> @@ -559,6 +559,7 @@ struct dev_pm_info {
>>       bool                    is_suspended:1; /* Ditto */
>>       bool                    is_noirq_suspended:1;
>>       bool                    is_late_suspended:1;
>> +     bool                    is_rpm_sleep:1; /* Owned by the PM core */
>
> It is expected to be set by drivers, so by definition it is *not* owned by the
> PM core.

Ahh, I see. I didn't quite get those comments, thanks for clarifying.

>
>>       bool                    early_init:1;   /* Owned by the PM core */
>>       bool                    direct_complete:1;      /* Owned by the PM core */
>>       spinlock_t              lock;
>> @@ -716,6 +717,9 @@ extern void __suspend_report_result(const char *function, void *fn, int ret);
>>  extern int device_pm_wait_for_dev(struct device *sub, struct device *dev);
>>  extern void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *));
>>
>> +extern void dev_pm_use_rpm_sleep(struct device *dev);
>> +extern bool dev_pm_is_rpm_sleep(struct device *dev);
>> +
>>  extern int pm_generic_prepare(struct device *dev);
>>  extern int pm_generic_suspend_late(struct device *dev);
>>  extern int pm_generic_suspend_noirq(struct device *dev);
>> @@ -759,6 +763,9 @@ static inline void dpm_for_each_dev(void *data, void (*fn)(struct device *, void
>>  {
>>  }
>>
>> +static inline void dev_pm_use_rpm_sleep(struct device *dev) {}
>> +static inline bool dev_pm_is_rpm_sleep(struct device *dev) { return false; }
>> +
>>  #define pm_generic_prepare           NULL
>>  #define pm_generic_suspend_late              NULL
>>  #define pm_generic_suspend_noirq     NULL
>>
>
> Thanks,
> Rafael
>

Thanks for reviewing!

Kind regards
Uffe

[PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Ulf Hansson]

On 29 August 2017 at 17:15, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Tuesday, August 29, 2017 4:56:43 PM CEST Ulf Hansson wrote:
>> The main principle behind the runtime PM centric path, is to re-use the
>> runtime PM callbacks to implement system sleep - and while doing that also
>> achieve a fully optimized behaviour from PM point of view.
>>
>> More precisely, avoid to wake up a device from its low power state during
>> system sleep, unless the device is really needed to be operational. That
>> does not only mean avoiding to waste power, but may also decrease system
>> suspend/resume time for a device.
>>
>> However, using the runtime PM centric path for a device, does put some
>> requirements on the behaviour of the PM core and a potential PM domain that
>> may be attached to the device. So far, these requirements are not being
>> specially considered, except by the generic PM domain.
>>
>> To move forward and to make it possible to deploy the runtime PM centric
>> path for cross SoC drivers, which may have different PM domains attached to
>> its devices depending on the SoC, we must address how to deal with these
>> requirements. This change starts by making some adoptions to the PM core,
>> while other parts, such as the ACPI PM domain needs to be taken care of
>> separately.
>>
>> In the runtime PM centric path, the driver is expected to make use of the
>> pm_runtime_force_suspend|resume() helpers, to deploy system sleep support.
>> More precisely it may assign the system sleep callbacks to these helpers or
>> may call them from its own callbacks, in case it needs to perform
>> additional actions during system sleep.
>>
>> In other words, the PM core must always invoke the system sleep callbacks
>> for the device when they are present, to allow the driver to deal with
>> the system sleep operations.
>>
>> In case the PM core decides to run the direct_complete path for the device,
>> it skips invoking most of the system sleep callbacks, besides
>> ->prepare|complete(). Therefore using the direct_complete path in
>> combination with the runtime PM centric patch for a device, does not play
>> well.
>>
>> To deal with this issue, let's add a flag 'is_rpm_sleep', to the struct
>> dev_pm_info. The driver that deploys the runtime PM centric path, shall set
>> the flag for the device during ->probe(), to inform the PM core about that
>> it must not use the direct_complete path for the device.
>>
>> Note, not allowing the direct_complete path for a device, doesn't implicit
>> need to propagate to the device's parent/suppliers. Therefore make the PM
>> core check this condition in device_suspend(), before it decides to abandon
>> the direct_complete path for parent/suppliers.
>>
>> To make the is_rpm_sleep flag internal to the PM core, let's add two APIs.
>>       - dev_pm_use_rpm_sleep(): It sets the flag and should be called by
>>         the driver during ->probe().
>>       - dev_pm_is_rpm_sleep(): Makes it possible for users of the device,
>>         like a PM domain, to fetch the state of the flag.
>>
>> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
>> ---
>>
>> Changes in v3:
>>       - New patch.
>>       - This replaces the earlier method of adding the no_direct_complete to
>>       the ACPI structures, according to comments from Rafael.
>>       - This change also address the consern Rafael had around that
>>       direct_complete should not have to be disabled for parent/suppliers, in
>>       case a device use the runtime PM centric path for system sleep.
>>
>> ---
>>  drivers/base/power/main.c    | 49 +++++++++++++++++++++++++++++++++++++++-----
>>  drivers/base/power/runtime.c |  1 +
>>  include/linux/pm.h           |  7 +++++++
>>  3 files changed, 52 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
>> index ea1732e..865737a 100644
>> --- a/drivers/base/power/main.c
>> +++ b/drivers/base/power/main.c
>> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
>>               if (parent) {
>>                       spin_lock_irq(&parent->power.lock);
>>
>> -                     dev->parent->power.direct_complete = false;
>> +                     if (!dev->power.is_rpm_sleep)
>> +                             dev->parent->power.direct_complete = false;
>
> This doesn't look good to me at all.
>
> It potentially breaks the fundamental assumption of the direct_complete
> mechanism that no devices with that flag set will ever be runtime resumed
> during system suspend.
>
> Which can happen with this change if a device with power.is_rpm_sleep is
> resumed causing the parent to be resumed too.

Doesn't the exact same problem you describe, already exist prior my change!?

That is, if the current device is runtime suspended and the
direct_complete flag is set for it, then __device_suspend() has
already disabled runtime PM for the device, when we reach this point.
That means, a following attempts to runtime resume the device will
fail and thus also to runtime resume its parent.

So to me, this is a different problem, related how to work out the
correct order of how to suspend devices.

>
>>                       if (dev->power.wakeup_path
>>                           && !dev->parent->power.ignore_children)
>>                               dev->parent->power.wakeup_path = true;
>>
>>                       spin_unlock_irq(&parent->power.lock);
>>               }
>> -             dpm_clear_suppliers_direct_complete(dev);
>> +             if (!dev->power.is_rpm_sleep)
>> +                     dpm_clear_suppliers_direct_complete(dev);
>
> And same here.

See comment above.

>
>>       }
>>
>>       device_unlock(dev);
>> @@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
>>        * A positive return value from ->prepare() means "this device appears
>>        * to be runtime-suspended and its state is fine, so if it really is
>>        * runtime-suspended, you can leave it in that state provided that you
>> -      * will do the same thing with all of its descendants".  This only
>> -      * applies to suspend transitions, however.
>> +      * will do the same thing with all of its descendants". To allow this,
>> +      * the is_rpm_sleep flag must not be set, which is default false, but
>> +      * may have been changed by a driver. This only applies to suspend
>> +      * transitions, however.
>>        */
>>       spin_lock_irq(&dev->power.lock);
>> -     dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
>> +     dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
>> +                             state.event == PM_EVENT_SUSPEND;
>
> The only problem addressed by avoiding direct_complete is when runtime PM needs
> to work during __device_suspend() for devices with direct_complete set.  You
> said that this was not the case with the i2c designware driver, so I'm not sure
> what the purpose of this is.

I should probably work more on my changelog, because I tried to
explain it there.

Anyway, let me quote the important parts, and this is not specific for
the i2c-designware-driver, but generic to drivers using
pm_runtime_force_suspend|resume().

"In the runtime PM centric path, the driver is expected to make use of
the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
support. More precisely it may assign the system sleep callbacks to
these helpers or may call them from its own callbacks, in case it
needs to perform additional actions during system sleep.

In other words, the PM core must always invoke the system sleep
callbacks for the device when they are present, to allow the driver to
deal with the system sleep operations."

A concrete example is drivers/spi/spi-fsl-espi.c, but one can easily find more.

>
> It doesn't solve any other problems at all.
>
>>       spin_unlock_irq(&dev->power.lock);
>>       return 0;
>>  }
>> @@ -1841,6 +1846,40 @@ void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *))
>>  }
>>  EXPORT_SYMBOL_GPL(dpm_for_each_dev);
>>
>> +/**
>> + * dev_pm_use_rpm_sleep - use the runtime PM centric sleep path.
>> + * @dev: the device to use the path for.
>> + *
>> + * The runtime PM centric path requires the system sleep callbacks for the
>> + * driver of the device to be invoked. This function sets the is_rpm_sleep flag
>> + * to enable this path to be used, which makes the PM core to adopt to this
>> + * behaviour. More precisely the PM core makes sure to not run the
>> + * direct_complete path for the chosen device during system sleep, when the
>> + * is_rpm_sleep flag is set. A driver using the runtime PM centric path, shall
>> + * use the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
>> + * support and call this function during ->probe().
>> + * The is_rpm_sleep flag is cleared when unbinding/bind-failure of the driver.
>> + */
>> +void dev_pm_use_rpm_sleep(struct device *dev)
>> +{
>> +     dev->power.is_rpm_sleep = true;
>> +}
>> +EXPORT_SYMBOL_GPL(dev_pm_use_rpm_sleep);
>> +
>> +/**
>> + * dev_pm_is_rpm_sleep - Returns the value of the is_rpm_sleep flag.
>> + * @dev: the device to return the flag for.
>> + *
>> + * The caller of this function is typically a subsystem or a PM domain that
>> + * needs to know if the runtime PM centric path is used for the device. It may
>> + * be invoked during any of the system sleep phases.
>> + */
>> +bool dev_pm_is_rpm_sleep(struct device *dev)
>> +{
>> +     return dev->power.is_rpm_sleep;
>> +}
>> +EXPORT_SYMBOL_GPL(dev_pm_is_rpm_sleep);
>> +
>>  static bool pm_ops_is_empty(const struct dev_pm_ops *ops)
>>  {
>>       if (!ops)
>> diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c
>> index 7bcf80f..b2ab22c 100644
>> --- a/drivers/base/power/runtime.c
>> +++ b/drivers/base/power/runtime.c
>> @@ -1522,6 +1522,7 @@ void pm_runtime_reinit(struct device *dev)
>>                               pm_runtime_put(dev->parent);
>>               }
>>       }
>> +     dev->power.is_rpm_sleep = false;
>>  }
>>
>>  /**
>> diff --git a/include/linux/pm.h b/include/linux/pm.h
>> index 47ded8a..5bf96d2 100644
>> --- a/include/linux/pm.h
>> +++ b/include/linux/pm.h
>> @@ -559,6 +559,7 @@ struct dev_pm_info {
>>       bool                    is_suspended:1; /* Ditto */
>>       bool                    is_noirq_suspended:1;
>>       bool                    is_late_suspended:1;
>> +     bool                    is_rpm_sleep:1; /* Owned by the PM core */
>
> It is expected to be set by drivers, so by definition it is *not* owned by the
> PM core.

Ahh, I see. I didn't quite get those comments, thanks for clarifying.

>
>>       bool                    early_init:1;   /* Owned by the PM core */
>>       bool                    direct_complete:1;      /* Owned by the PM core */
>>       spinlock_t              lock;
>> @@ -716,6 +717,9 @@ extern void __suspend_report_result(const char *function, void *fn, int ret);
>>  extern int device_pm_wait_for_dev(struct device *sub, struct device *dev);
>>  extern void dpm_for_each_dev(void *data, void (*fn)(struct device *, void *));
>>
>> +extern void dev_pm_use_rpm_sleep(struct device *dev);
>> +extern bool dev_pm_is_rpm_sleep(struct device *dev);
>> +
>>  extern int pm_generic_prepare(struct device *dev);
>>  extern int pm_generic_suspend_late(struct device *dev);
>>  extern int pm_generic_suspend_noirq(struct device *dev);
>> @@ -759,6 +763,9 @@ static inline void dpm_for_each_dev(void *data, void (*fn)(struct device *, void
>>  {
>>  }
>>
>> +static inline void dev_pm_use_rpm_sleep(struct device *dev) {}
>> +static inline bool dev_pm_is_rpm_sleep(struct device *dev) { return false; }
>> +
>>  #define pm_generic_prepare           NULL
>>  #define pm_generic_suspend_late              NULL
>>  #define pm_generic_suspend_noirq     NULL
>>
>
> Thanks,
> Rafael
>

Thanks for reviewing!

Kind regards
Uffe

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

On 29 August 2017 at 22:19, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
>> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
>> isn't well optimized for system sleep.
>>
>> What makes this driver particularly interesting is because it's a cross-SoC
>> driver, which sometimes means there is an ACPI PM domain attached to the i2c
>> device and sometimes not. The driver is being used on both x86 and ARM.
>>
>> In principle, to optimize the system sleep support in i2c driver, this series
>> enables the proven runtime PM centric path for the i2c driver. However, to do
>> that the ACPI PM domain also have to collaborate and understand this behaviour.
>> From earlier versions, Rafael has also pointed out that also the PM core needs
>> to be involved.
>
> Earlier today I realized that drivers pointing their ->suspend_late and
> ->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
> pm_runtime_force_resume(), are fundamentally incompatible with any bus type
> doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.
>
> First, it basically requires the bus type or PM domain to expect that its
> ->runtime_suspend callback may or may not be indirectly invoked from its
> own ->suspend_late callback, depending on the driver (and analogously
> for ->runtime_resume and ->resume early), which is insane.
>
> Second, it is a layering violation, because it makes the driver effectively
> override the upper layer's decisions about what code to run.

You are right that for more complex bus types and PM domains, those
needs to play along. So that is what I am trying to implement for the
ACPI PM domain in this series.

The generic PM domain, is simple in this regards. There is only a
minor adaptation for the ->runtime_suspend|resume() callbacks, which
avoids validating dev_pm_qos constraints during system sleep. Nothing
special is needed in ->suspend_late|noirq callbacks, etc.

For most other simple bus types, like the platform bus, spi, i2c,
amba, no particular adoptions is needed at all. Instead those just
trust the drivers to do the right thing.

Before we had the direct_complete path, using the
pm_runtime_force_suspend|resume() helpers, was the only good way for
these kind of drivers, to in an optimized manner, deal with system
sleep when runtime PM also was enabled for their devices. Now this
method has become widely deployed, unfortunate whether you like it or
not.

Besides the slightly better optimizations you get when using
pm_runtime_force_suspend|resume(), comparing to the direct_complete
path - I think it's also worth to consider, how easy it becomes for
drivers to deploy system sleep support. In many cases, only two lines
of code is needed to add system sleep support in a driver.

Now, some complex code always needs to be implemented somewhere. When
using the runtime PM centric path, that code consist of the
pm_runtime_force_suspend|resume() helpers itself - and some
adaptations in buses/PM domains in cases when those needs special
care.

My point is, the runtime PM centric path, allows us to keep the
complex part of the code at a few centralized places, instead of
having it spread/duplicated into drivers.

So yes, you could consider it insane, but to me and many others, it
seems to work out quite well.

Yeah, and the laying violation is undoubtedly the most controversial
part of the runtime PM centric path - I agree to that! The
direct_complete path don't have this, as you implemented it. :-)

On the other hand, one could consider that these upper layers, in many
cases anyway needs to play along with the behavior of the driver. So,
I guess it depends on how one see it.

>
> That's why I'm afraid that we've reached a dead end here. :-(

That's said news. Is was really hoping I could find a way to move this
forward. You don't have any other ideas on how I can adjust the series
to make you happy?

>
> Thanks,
> Rafael
>

Kind regards
Uffe

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

On 29 August 2017 at 22:19, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
>> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
>> isn't well optimized for system sleep.
>>
>> What makes this driver particularly interesting is because it's a cross-SoC
>> driver, which sometimes means there is an ACPI PM domain attached to the i2c
>> device and sometimes not. The driver is being used on both x86 and ARM.
>>
>> In principle, to optimize the system sleep support in i2c driver, this series
>> enables the proven runtime PM centric path for the i2c driver. However, to do
>> that the ACPI PM domain also have to collaborate and understand this behaviour.
>> From earlier versions, Rafael has also pointed out that also the PM core needs
>> to be involved.
>
> Earlier today I realized that drivers pointing their ->suspend_late and
> ->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
> pm_runtime_force_resume(), are fundamentally incompatible with any bus type
> doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.
>
> First, it basically requires the bus type or PM domain to expect that its
> ->runtime_suspend callback may or may not be indirectly invoked from its
> own ->suspend_late callback, depending on the driver (and analogously
> for ->runtime_resume and ->resume early), which is insane.
>
> Second, it is a layering violation, because it makes the driver effectively
> override the upper layer's decisions about what code to run.

You are right that for more complex bus types and PM domains, those
needs to play along. So that is what I am trying to implement for the
ACPI PM domain in this series.

The generic PM domain, is simple in this regards. There is only a
minor adaptation for the ->runtime_suspend|resume() callbacks, which
avoids validating dev_pm_qos constraints during system sleep. Nothing
special is needed in ->suspend_late|noirq callbacks, etc.

For most other simple bus types, like the platform bus, spi, i2c,
amba, no particular adoptions is needed at all. Instead those just
trust the drivers to do the right thing.

Before we had the direct_complete path, using the
pm_runtime_force_suspend|resume() helpers, was the only good way for
these kind of drivers, to in an optimized manner, deal with system
sleep when runtime PM also was enabled for their devices. Now this
method has become widely deployed, unfortunate whether you like it or
not.

Besides the slightly better optimizations you get when using
pm_runtime_force_suspend|resume(), comparing to the direct_complete
path - I think it's also worth to consider, how easy it becomes for
drivers to deploy system sleep support. In many cases, only two lines
of code is needed to add system sleep support in a driver.

Now, some complex code always needs to be implemented somewhere. When
using the runtime PM centric path, that code consist of the
pm_runtime_force_suspend|resume() helpers itself - and some
adaptations in buses/PM domains in cases when those needs special
care.

My point is, the runtime PM centric path, allows us to keep the
complex part of the code at a few centralized places, instead of
having it spread/duplicated into drivers.

So yes, you could consider it insane, but to me and many others, it
seems to work out quite well.

Yeah, and the laying violation is undoubtedly the most controversial
part of the runtime PM centric path - I agree to that! The
direct_complete path don't have this, as you implemented it. :-)

On the other hand, one could consider that these upper layers, in many
cases anyway needs to play along with the behavior of the driver. So,
I guess it depends on how one see it.

>
> That's why I'm afraid that we've reached a dead end here. :-(

That's said news. Is was really hoping I could find a way to move this
forward. You don't have any other ideas on how I can adjust the series
to make you happy?

>
> Thanks,
> Rafael
>

Kind regards
Uffe

Re: [PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Rafael J. Wysocki]

On Wednesday, August 30, 2017 9:13:47 AM CEST Ulf Hansson wrote:
> On 29 August 2017 at 17:15, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Tuesday, August 29, 2017 4:56:43 PM CEST Ulf Hansson wrote:
> >> The main principle behind the runtime PM centric path, is to re-use the
> >> runtime PM callbacks to implement system sleep - and while doing that also
> >> achieve a fully optimized behaviour from PM point of view.
> >>
> >> More precisely, avoid to wake up a device from its low power state during
> >> system sleep, unless the device is really needed to be operational. That
> >> does not only mean avoiding to waste power, but may also decrease system
> >> suspend/resume time for a device.
> >>
> >> However, using the runtime PM centric path for a device, does put some
> >> requirements on the behaviour of the PM core and a potential PM domain that
> >> may be attached to the device. So far, these requirements are not being
> >> specially considered, except by the generic PM domain.
> >>
> >> To move forward and to make it possible to deploy the runtime PM centric
> >> path for cross SoC drivers, which may have different PM domains attached to
> >> its devices depending on the SoC, we must address how to deal with these
> >> requirements. This change starts by making some adoptions to the PM core,
> >> while other parts, such as the ACPI PM domain needs to be taken care of
> >> separately.
> >>
> >> In the runtime PM centric path, the driver is expected to make use of the
> >> pm_runtime_force_suspend|resume() helpers, to deploy system sleep support.
> >> More precisely it may assign the system sleep callbacks to these helpers or
> >> may call them from its own callbacks, in case it needs to perform
> >> additional actions during system sleep.
> >>
> >> In other words, the PM core must always invoke the system sleep callbacks
> >> for the device when they are present, to allow the driver to deal with
> >> the system sleep operations.
> >>
> >> In case the PM core decides to run the direct_complete path for the device,
> >> it skips invoking most of the system sleep callbacks, besides
> >> ->prepare|complete(). Therefore using the direct_complete path in
> >> combination with the runtime PM centric patch for a device, does not play
> >> well.
> >>
> >> To deal with this issue, let's add a flag 'is_rpm_sleep', to the struct
> >> dev_pm_info. The driver that deploys the runtime PM centric path, shall set
> >> the flag for the device during ->probe(), to inform the PM core about that
> >> it must not use the direct_complete path for the device.
> >>
> >> Note, not allowing the direct_complete path for a device, doesn't implicit
> >> need to propagate to the device's parent/suppliers. Therefore make the PM
> >> core check this condition in device_suspend(), before it decides to abandon
> >> the direct_complete path for parent/suppliers.
> >>
> >> To make the is_rpm_sleep flag internal to the PM core, let's add two APIs.
> >>       - dev_pm_use_rpm_sleep(): It sets the flag and should be called by
> >>         the driver during ->probe().
> >>       - dev_pm_is_rpm_sleep(): Makes it possible for users of the device,
> >>         like a PM domain, to fetch the state of the flag.
> >>
> >> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> >> ---
> >>
> >> Changes in v3:
> >>       - New patch.
> >>       - This replaces the earlier method of adding the no_direct_complete to
> >>       the ACPI structures, according to comments from Rafael.
> >>       - This change also address the consern Rafael had around that
> >>       direct_complete should not have to be disabled for parent/suppliers, in
> >>       case a device use the runtime PM centric path for system sleep.
> >>
> >> ---
> >>  drivers/base/power/main.c    | 49 +++++++++++++++++++++++++++++++++++++++-----
> >>  drivers/base/power/runtime.c |  1 +
> >>  include/linux/pm.h           |  7 +++++++
> >>  3 files changed, 52 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> >> index ea1732e..865737a 100644
> >> --- a/drivers/base/power/main.c
> >> +++ b/drivers/base/power/main.c
> >> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
> >>               if (parent) {
> >>                       spin_lock_irq(&parent->power.lock);
> >>
> >> -                     dev->parent->power.direct_complete = false;
> >> +                     if (!dev->power.is_rpm_sleep)
> >> +                             dev->parent->power.direct_complete = false;
> >
> > This doesn't look good to me at all.
> >
> > It potentially breaks the fundamental assumption of the direct_complete
> > mechanism that no devices with that flag set will ever be runtime resumed
> > during system suspend.
> >
> > Which can happen with this change if a device with power.is_rpm_sleep is
> > resumed causing the parent to be resumed too.
> 
> Doesn't the exact same problem you describe, already exist prior my change!?

No, it doesn't.

> That is, if the current device is runtime suspended and the
> direct_complete flag is set for it, then __device_suspend() has
> already disabled runtime PM for the device, when we reach this point.
> That means, a following attempts to runtime resume the device will
> fail and thus also to runtime resume its parent.

That's because any devices with direct_complete set *cannot* be runtime
resumed after __device_suspend().  That's intentional and how the thing
is designed.  It cannot work differently, sorry.

> So to me, this is a different problem, related how to work out the
> correct order of how to suspend devices.

The ordering matters here, but it is not the problem.

Setting direct_complete means that PM callbacks for this device won't be
invoked going forward.  However, if the state of the device was to change
(like as a result of runtime PM resume), it would be necessary to run those
callbacks after all, but after __device_suspend() it may be too late for
that, because the ->suspend callback may have been skipped already.

The only way to address that is to prevent runtime PM resume of the device
from happening at the point the PM core decides to use direct_complete for it,
but that requires runtime PM to be disabled for it.  Moreover, since the
device is now suspended with runtime PM disabled and its children might rely
on it if they were active, the children must be suspended with runtime PM
disabled at this point too.  That's why direct_complete can only be set
for a device if it is set for the entire hierarchy below it.

Summary: If you allow a device to be runtime resumed during system susped,
direct_complete cannot be set for it and it cannot be set for its parent and
so on.

[cut]

> >> @@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
> >>        * A positive return value from ->prepare() means "this device appears
> >>        * to be runtime-suspended and its state is fine, so if it really is
> >>        * runtime-suspended, you can leave it in that state provided that you
> >> -      * will do the same thing with all of its descendants".  This only
> >> -      * applies to suspend transitions, however.
> >> +      * will do the same thing with all of its descendants". To allow this,
> >> +      * the is_rpm_sleep flag must not be set, which is default false, but
> >> +      * may have been changed by a driver. This only applies to suspend
> >> +      * transitions, however.
> >>        */
> >>       spin_lock_irq(&dev->power.lock);
> >> -     dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
> >> +     dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
> >> +                             state.event == PM_EVENT_SUSPEND;
> >
> > The only problem addressed by avoiding direct_complete is when runtime PM needs
> > to work during __device_suspend() for devices with direct_complete set.  You
> > said that this was not the case with the i2c designware driver, so I'm not sure
> > what the purpose of this is.
> 
> I should probably work more on my changelog, because I tried to
> explain it there.
> 
> Anyway, let me quote the important parts, and this is not specific for
> the i2c-designware-driver, but generic to drivers using
> pm_runtime_force_suspend|resume().
> 
> "In the runtime PM centric path, the driver is expected to make use of
> the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
> support. More precisely it may assign the system sleep callbacks to
> these helpers or may call them from its own callbacks, in case it
> needs to perform additional actions during system sleep.
> 
> In other words, the PM core must always invoke the system sleep
> callbacks for the device when they are present, to allow the driver to
> deal with the system sleep operations."
> 
> A concrete example is drivers/spi/spi-fsl-espi.c, but one can easily find more.

In fact, the new flag introduced here means "assume that that the driver of this
device points ->late_suspend and ->early_resume to pm_runtime_force_suspend()
and pm_runtime_force_resume(), respectively."  Which then implies that
direct_complete cannot be used with it and (and with its parent and so on)
and that it is not necessary to runtime resume it during system suspend.

However, there are (or at least there may be) devices that need not be
runtime resumed during system suspend even though their drivers don't use
_force_suspend/resume().

Also there are devices whose drivers don't want direct_complete to be used with
them, even though they don't use _force_suspend/resume() and they *do* need
their devices to be runtime resumed during system suspend.

Moreover, some drivers may not mind direct_complete even though their devices
need not be runtime resumed during system suspend.  All of that doesn't have
to be related to using _force_suspend/resume() at all.

So I don't see any reason whatever for combining all of these three *different*
conditions under one flag.

Thanks,
Rafael

[PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Rafael J. Wysocki]

On Wednesday, August 30, 2017 9:13:47 AM CEST Ulf Hansson wrote:
> On 29 August 2017 at 17:15, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Tuesday, August 29, 2017 4:56:43 PM CEST Ulf Hansson wrote:
> >> The main principle behind the runtime PM centric path, is to re-use the
> >> runtime PM callbacks to implement system sleep - and while doing that also
> >> achieve a fully optimized behaviour from PM point of view.
> >>
> >> More precisely, avoid to wake up a device from its low power state during
> >> system sleep, unless the device is really needed to be operational. That
> >> does not only mean avoiding to waste power, but may also decrease system
> >> suspend/resume time for a device.
> >>
> >> However, using the runtime PM centric path for a device, does put some
> >> requirements on the behaviour of the PM core and a potential PM domain that
> >> may be attached to the device. So far, these requirements are not being
> >> specially considered, except by the generic PM domain.
> >>
> >> To move forward and to make it possible to deploy the runtime PM centric
> >> path for cross SoC drivers, which may have different PM domains attached to
> >> its devices depending on the SoC, we must address how to deal with these
> >> requirements. This change starts by making some adoptions to the PM core,
> >> while other parts, such as the ACPI PM domain needs to be taken care of
> >> separately.
> >>
> >> In the runtime PM centric path, the driver is expected to make use of the
> >> pm_runtime_force_suspend|resume() helpers, to deploy system sleep support.
> >> More precisely it may assign the system sleep callbacks to these helpers or
> >> may call them from its own callbacks, in case it needs to perform
> >> additional actions during system sleep.
> >>
> >> In other words, the PM core must always invoke the system sleep callbacks
> >> for the device when they are present, to allow the driver to deal with
> >> the system sleep operations.
> >>
> >> In case the PM core decides to run the direct_complete path for the device,
> >> it skips invoking most of the system sleep callbacks, besides
> >> ->prepare|complete(). Therefore using the direct_complete path in
> >> combination with the runtime PM centric patch for a device, does not play
> >> well.
> >>
> >> To deal with this issue, let's add a flag 'is_rpm_sleep', to the struct
> >> dev_pm_info. The driver that deploys the runtime PM centric path, shall set
> >> the flag for the device during ->probe(), to inform the PM core about that
> >> it must not use the direct_complete path for the device.
> >>
> >> Note, not allowing the direct_complete path for a device, doesn't implicit
> >> need to propagate to the device's parent/suppliers. Therefore make the PM
> >> core check this condition in device_suspend(), before it decides to abandon
> >> the direct_complete path for parent/suppliers.
> >>
> >> To make the is_rpm_sleep flag internal to the PM core, let's add two APIs.
> >>       - dev_pm_use_rpm_sleep(): It sets the flag and should be called by
> >>         the driver during ->probe().
> >>       - dev_pm_is_rpm_sleep(): Makes it possible for users of the device,
> >>         like a PM domain, to fetch the state of the flag.
> >>
> >> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> >> ---
> >>
> >> Changes in v3:
> >>       - New patch.
> >>       - This replaces the earlier method of adding the no_direct_complete to
> >>       the ACPI structures, according to comments from Rafael.
> >>       - This change also address the consern Rafael had around that
> >>       direct_complete should not have to be disabled for parent/suppliers, in
> >>       case a device use the runtime PM centric path for system sleep.
> >>
> >> ---
> >>  drivers/base/power/main.c    | 49 +++++++++++++++++++++++++++++++++++++++-----
> >>  drivers/base/power/runtime.c |  1 +
> >>  include/linux/pm.h           |  7 +++++++
> >>  3 files changed, 52 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> >> index ea1732e..865737a 100644
> >> --- a/drivers/base/power/main.c
> >> +++ b/drivers/base/power/main.c
> >> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
> >>               if (parent) {
> >>                       spin_lock_irq(&parent->power.lock);
> >>
> >> -                     dev->parent->power.direct_complete = false;
> >> +                     if (!dev->power.is_rpm_sleep)
> >> +                             dev->parent->power.direct_complete = false;
> >
> > This doesn't look good to me at all.
> >
> > It potentially breaks the fundamental assumption of the direct_complete
> > mechanism that no devices with that flag set will ever be runtime resumed
> > during system suspend.
> >
> > Which can happen with this change if a device with power.is_rpm_sleep is
> > resumed causing the parent to be resumed too.
> 
> Doesn't the exact same problem you describe, already exist prior my change!?

No, it doesn't.

> That is, if the current device is runtime suspended and the
> direct_complete flag is set for it, then __device_suspend() has
> already disabled runtime PM for the device, when we reach this point.
> That means, a following attempts to runtime resume the device will
> fail and thus also to runtime resume its parent.

That's because any devices with direct_complete set *cannot* be runtime
resumed after __device_suspend().  That's intentional and how the thing
is designed.  It cannot work differently, sorry.

> So to me, this is a different problem, related how to work out the
> correct order of how to suspend devices.

The ordering matters here, but it is not the problem.

Setting direct_complete means that PM callbacks for this device won't be
invoked going forward.  However, if the state of the device was to change
(like as a result of runtime PM resume), it would be necessary to run those
callbacks after all, but after __device_suspend() it may be too late for
that, because the ->suspend callback may have been skipped already.

The only way to address that is to prevent runtime PM resume of the device
from happening at the point the PM core decides to use direct_complete for it,
but that requires runtime PM to be disabled for it.  Moreover, since the
device is now suspended with runtime PM disabled and its children might rely
on it if they were active, the children must be suspended with runtime PM
disabled at this point too.  That's why direct_complete can only be set
for a device if it is set for the entire hierarchy below it.

Summary: If you allow a device to be runtime resumed during system susped,
direct_complete cannot be set for it and it cannot be set for its parent and
so on.

[cut]

> >> @@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
> >>        * A positive return value from ->prepare() means "this device appears
> >>        * to be runtime-suspended and its state is fine, so if it really is
> >>        * runtime-suspended, you can leave it in that state provided that you
> >> -      * will do the same thing with all of its descendants".  This only
> >> -      * applies to suspend transitions, however.
> >> +      * will do the same thing with all of its descendants". To allow this,
> >> +      * the is_rpm_sleep flag must not be set, which is default false, but
> >> +      * may have been changed by a driver. This only applies to suspend
> >> +      * transitions, however.
> >>        */
> >>       spin_lock_irq(&dev->power.lock);
> >> -     dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
> >> +     dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
> >> +                             state.event == PM_EVENT_SUSPEND;
> >
> > The only problem addressed by avoiding direct_complete is when runtime PM needs
> > to work during __device_suspend() for devices with direct_complete set.  You
> > said that this was not the case with the i2c designware driver, so I'm not sure
> > what the purpose of this is.
> 
> I should probably work more on my changelog, because I tried to
> explain it there.
> 
> Anyway, let me quote the important parts, and this is not specific for
> the i2c-designware-driver, but generic to drivers using
> pm_runtime_force_suspend|resume().
> 
> "In the runtime PM centric path, the driver is expected to make use of
> the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
> support. More precisely it may assign the system sleep callbacks to
> these helpers or may call them from its own callbacks, in case it
> needs to perform additional actions during system sleep.
> 
> In other words, the PM core must always invoke the system sleep
> callbacks for the device when they are present, to allow the driver to
> deal with the system sleep operations."
> 
> A concrete example is drivers/spi/spi-fsl-espi.c, but one can easily find more.

In fact, the new flag introduced here means "assume that that the driver of this
device points ->late_suspend and ->early_resume to pm_runtime_force_suspend()
and pm_runtime_force_resume(), respectively."  Which then implies that
direct_complete cannot be used with it and (and with its parent and so on)
and that it is not necessary to runtime resume it during system suspend.

However, there are (or at least there may be) devices that need not be
runtime resumed during system suspend even though their drivers don't use
_force_suspend/resume().

Also there are devices whose drivers don't want direct_complete to be used with
them, even though they don't use _force_suspend/resume() and they *do* need
their devices to be runtime resumed during system suspend.

Moreover, some drivers may not mind direct_complete even though their devices
need not be runtime resumed during system suspend.  All of that doesn't have
to be related to using _force_suspend/resume() at all.

So I don't see any reason whatever for combining all of these three *different*
conditions under one flag.

Thanks,
Rafael

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

Disclaimer: I'm falling asleep, so I probably shouldn't reply to email right
now, but tomorrow I may not be able to get to email at all, so I'll try anyway.

On Wednesday, August 30, 2017 11:57:28 AM CEST Ulf Hansson wrote:
> On 29 August 2017 at 22:19, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
> >> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
> >> isn't well optimized for system sleep.
> >>
> >> What makes this driver particularly interesting is because it's a cross-SoC
> >> driver, which sometimes means there is an ACPI PM domain attached to the i2c
> >> device and sometimes not. The driver is being used on both x86 and ARM.
> >>
> >> In principle, to optimize the system sleep support in i2c driver, this series
> >> enables the proven runtime PM centric path for the i2c driver. However, to do
> >> that the ACPI PM domain also have to collaborate and understand this behaviour.
> >> From earlier versions, Rafael has also pointed out that also the PM core needs
> >> to be involved.
> >
> > Earlier today I realized that drivers pointing their ->suspend_late and
> > ->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
> > pm_runtime_force_resume(), are fundamentally incompatible with any bus type
> > doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.
> >
> > First, it basically requires the bus type or PM domain to expect that its
> > ->runtime_suspend callback may or may not be indirectly invoked from its
> > own ->suspend_late callback, depending on the driver (and analogously
> > for ->runtime_resume and ->resume early), which is insane.
> >
> > Second, it is a layering violation, because it makes the driver effectively
> > override the upper layer's decisions about what code to run.
> 
> You are right that for more complex bus types and PM domains, those
> needs to play along. So that is what I am trying to implement for the
> ACPI PM domain in this series.

Well, "play along" is a bit of an understatement here.

They would need to turn into horrible mess and that's not going to happen.

> The generic PM domain, is simple in this regards. There is only a
> minor adaptation for the ->runtime_suspend|resume() callbacks, which
> avoids validating dev_pm_qos constraints during system sleep. Nothing
> special is needed in ->suspend_late|noirq callbacks, etc.
> 
> For most other simple bus types, like the platform bus, spi, i2c,
> amba, no particular adoptions is needed at all. Instead those just
> trust the drivers to do the right thing.

They are the trivial ones.

> Before we had the direct_complete path, using the
> pm_runtime_force_suspend|resume() helpers, was the only good way for
> these kind of drivers, to in an optimized manner, deal with system
> sleep when runtime PM also was enabled for their devices. Now this
> method has become widely deployed, unfortunate whether you like it or
> not.

So can you please remind me why the _force_ wrappers are needed?

In particular, why can't drivers arrange their callbacks the way I did that
in https://patchwork.kernel.org/patch/9928583/ ?

> Besides the slightly better optimizations you get when using
> pm_runtime_force_suspend|resume(), comparing to the direct_complete
> path - I think it's also worth to consider, how easy it becomes for
> drivers to deploy system sleep support. In many cases, only two lines
> of code is needed to add system sleep support in a driver.

You are doing a wrong comparison here IMO.  You essentially are comparing two
bandaids with each other and arguing that one of them somehow is better.

What about doing something which is not a bandaid instead?

> Now, some complex code always needs to be implemented somewhere. When
> using the runtime PM centric path, that code consist of the
> pm_runtime_force_suspend|resume() helpers itself - and some
> adaptations in buses/PM domains in cases when those needs special
> care.
> 
> My point is, the runtime PM centric path, allows us to keep the
> complex part of the code at a few centralized places, instead of
> having it spread/duplicated into drivers.
> 
> So yes, you could consider it insane, but to me and many others, it
> seems to work out quite well.

Because it only has been used with trivial middle layer code so far
and I'm quite disappointed that you don't seem to see a problem here. :-/

I mean something like

PM core => bus type / PM domain ->suspend_late => driver ->suspend_late

is far more straightforward than

PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
	bus type / PM domain ->runtime_suspend => driver ->runtime_suspend

with the bus type / PM domain having to figure out somehow at the
->suspend_late time whether or not its ->runtume_suspend is going to be invoked
in the middle of it.

Apart from this just being aesthetically disgusting to me, which admittedly is
a matter of personal opinion, it makes debugging new driver code harder (if it
happens to not work) and reviewing it almost impossible, because now you need
to take all of the tangling between callbacks into accont and sometimes not
just for one bus type / PM domain.

> Yeah, and the laying violation is undoubtedly the most controversial
> part of the runtime PM centric path - I agree to that! The
> direct_complete path don't have this, as you implemented it. :-)
> 
> On the other hand, one could consider that these upper layers, in many
> cases anyway needs to play along with the behavior of the driver. So,
> I guess it depends on how one see it.
> 
> >
> > That's why I'm afraid that we've reached a dead end here. :-(
> 
> That's said news. Is was really hoping I could find a way to move this
> forward. You don't have any other ideas on how I can adjust the series
> to make you happy?

So to be precise, patches [2-3/8] are basically fine by me.  Patch [4/8]
sort of works too, but I'd do the splitting slightly differently and I don't
see much value in it alone.

The rest of the ACPI changes is mostly not acceptable to me, mostly because
of what is done to the PM domain's ->runtime_suspend/resume and
->suspend_late/->resume_early callbacks.

I guess the only way that could be made work for me would be by not using
_force_suspend/resume() at all, but that would defeat the point, right?

I don't like the flag too, but that might be worked out.

Also, when I looked at _force_suspend/resume() again, I got concerned.
There is stuff in there that shouldn't be necessary in a driver's
->late_suspend/->early_resume and some things in there just made me
scratch my head.

Thanks,
Rafael

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

Disclaimer: I'm falling asleep, so I probably shouldn't reply to email right
now, but tomorrow I may not be able to get to email at all, so I'll try anyway.

On Wednesday, August 30, 2017 11:57:28 AM CEST Ulf Hansson wrote:
> On 29 August 2017 at 22:19, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
> >> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
> >> isn't well optimized for system sleep.
> >>
> >> What makes this driver particularly interesting is because it's a cross-SoC
> >> driver, which sometimes means there is an ACPI PM domain attached to the i2c
> >> device and sometimes not. The driver is being used on both x86 and ARM.
> >>
> >> In principle, to optimize the system sleep support in i2c driver, this series
> >> enables the proven runtime PM centric path for the i2c driver. However, to do
> >> that the ACPI PM domain also have to collaborate and understand this behaviour.
> >> From earlier versions, Rafael has also pointed out that also the PM core needs
> >> to be involved.
> >
> > Earlier today I realized that drivers pointing their ->suspend_late and
> > ->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
> > pm_runtime_force_resume(), are fundamentally incompatible with any bus type
> > doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.
> >
> > First, it basically requires the bus type or PM domain to expect that its
> > ->runtime_suspend callback may or may not be indirectly invoked from its
> > own ->suspend_late callback, depending on the driver (and analogously
> > for ->runtime_resume and ->resume early), which is insane.
> >
> > Second, it is a layering violation, because it makes the driver effectively
> > override the upper layer's decisions about what code to run.
> 
> You are right that for more complex bus types and PM domains, those
> needs to play along. So that is what I am trying to implement for the
> ACPI PM domain in this series.

Well, "play along" is a bit of an understatement here.

They would need to turn into horrible mess and that's not going to happen.

> The generic PM domain, is simple in this regards. There is only a
> minor adaptation for the ->runtime_suspend|resume() callbacks, which
> avoids validating dev_pm_qos constraints during system sleep. Nothing
> special is needed in ->suspend_late|noirq callbacks, etc.
> 
> For most other simple bus types, like the platform bus, spi, i2c,
> amba, no particular adoptions is needed at all. Instead those just
> trust the drivers to do the right thing.

They are the trivial ones.

> Before we had the direct_complete path, using the
> pm_runtime_force_suspend|resume() helpers, was the only good way for
> these kind of drivers, to in an optimized manner, deal with system
> sleep when runtime PM also was enabled for their devices. Now this
> method has become widely deployed, unfortunate whether you like it or
> not.

So can you please remind me why the _force_ wrappers are needed?

In particular, why can't drivers arrange their callbacks the way I did that
in https://patchwork.kernel.org/patch/9928583/ ?

> Besides the slightly better optimizations you get when using
> pm_runtime_force_suspend|resume(), comparing to the direct_complete
> path - I think it's also worth to consider, how easy it becomes for
> drivers to deploy system sleep support. In many cases, only two lines
> of code is needed to add system sleep support in a driver.

You are doing a wrong comparison here IMO.  You essentially are comparing two
bandaids with each other and arguing that one of them somehow is better.

What about doing something which is not a bandaid instead?

> Now, some complex code always needs to be implemented somewhere. When
> using the runtime PM centric path, that code consist of the
> pm_runtime_force_suspend|resume() helpers itself - and some
> adaptations in buses/PM domains in cases when those needs special
> care.
> 
> My point is, the runtime PM centric path, allows us to keep the
> complex part of the code at a few centralized places, instead of
> having it spread/duplicated into drivers.
> 
> So yes, you could consider it insane, but to me and many others, it
> seems to work out quite well.

Because it only has been used with trivial middle layer code so far
and I'm quite disappointed that you don't seem to see a problem here. :-/

I mean something like

PM core => bus type / PM domain ->suspend_late => driver ->suspend_late

is far more straightforward than

PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
	bus type / PM domain ->runtime_suspend => driver ->runtime_suspend

with the bus type / PM domain having to figure out somehow at the
->suspend_late time whether or not its ->runtume_suspend is going to be invoked
in the middle of it.

Apart from this just being aesthetically disgusting to me, which admittedly is
a matter of personal opinion, it makes debugging new driver code harder (if it
happens to not work) and reviewing it almost impossible, because now you need
to take all of the tangling between callbacks into accont and sometimes not
just for one bus type / PM domain.

> Yeah, and the laying violation is undoubtedly the most controversial
> part of the runtime PM centric path - I agree to that! The
> direct_complete path don't have this, as you implemented it. :-)
> 
> On the other hand, one could consider that these upper layers, in many
> cases anyway needs to play along with the behavior of the driver. So,
> I guess it depends on how one see it.
> 
> >
> > That's why I'm afraid that we've reached a dead end here. :-(
> 
> That's said news. Is was really hoping I could find a way to move this
> forward. You don't have any other ideas on how I can adjust the series
> to make you happy?

So to be precise, patches [2-3/8] are basically fine by me.  Patch [4/8]
sort of works too, but I'd do the splitting slightly differently and I don't
see much value in it alone.

The rest of the ACPI changes is mostly not acceptable to me, mostly because
of what is done to the PM domain's ->runtime_suspend/resume and
->suspend_late/->resume_early callbacks.

I guess the only way that could be made work for me would be by not using
_force_suspend/resume() at all, but that would defeat the point, right?

I don't like the flag too, but that might be worked out.

Also, when I looked at _force_suspend/resume() again, I got concerned.
There is stuff in there that shouldn't be necessary in a driver's
->late_suspend/->early_resume and some things in there just made me
scratch my head.

Thanks,
Rafael

Re: [PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Ulf Hansson]

[...]

>> >> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
>> >> index ea1732e..865737a 100644
>> >> --- a/drivers/base/power/main.c
>> >> +++ b/drivers/base/power/main.c
>> >> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
>> >>               if (parent) {
>> >>                       spin_lock_irq(&parent->power.lock);
>> >>
>> >> -                     dev->parent->power.direct_complete = false;
>> >> +                     if (!dev->power.is_rpm_sleep)
>> >> +                             dev->parent->power.direct_complete = false;
>> >
>> > This doesn't look good to me at all.
>> >
>> > It potentially breaks the fundamental assumption of the direct_complete
>> > mechanism that no devices with that flag set will ever be runtime resumed
>> > during system suspend.
>> >
>> > Which can happen with this change if a device with power.is_rpm_sleep is
>> > resumed causing the parent to be resumed too.
>>
>> Doesn't the exact same problem you describe, already exist prior my change!?
>
> No, it doesn't.
>
>> That is, if the current device is runtime suspended and the
>> direct_complete flag is set for it, then __device_suspend() has
>> already disabled runtime PM for the device, when we reach this point.
>> That means, a following attempts to runtime resume the device will
>> fail and thus also to runtime resume its parent.
>
> That's because any devices with direct_complete set *cannot* be runtime
> resumed after __device_suspend().  That's intentional and how the thing
> is designed.  It cannot work differently, sorry.
>
>> So to me, this is a different problem, related how to work out the
>> correct order of how to suspend devices.
>
> The ordering matters here, but it is not the problem.
>
> Setting direct_complete means that PM callbacks for this device won't be
> invoked going forward.  However, if the state of the device was to change
> (like as a result of runtime PM resume), it would be necessary to run those
> callbacks after all, but after __device_suspend() it may be too late for
> that, because the ->suspend callback may have been skipped already.
>
> The only way to address that is to prevent runtime PM resume of the device
> from happening at the point the PM core decides to use direct_complete for it,
> but that requires runtime PM to be disabled for it.  Moreover, since the
> device is now suspended with runtime PM disabled and its children might rely
> on it if they were active, the children must be suspended with runtime PM
> disabled at this point too.  That's why direct_complete can only be set
> for a device if it is set for the entire hierarchy below it.

Thanks, this was a very nice description of the direct_complete path!

>
> Summary: If you allow a device to be runtime resumed during system susped,
> direct_complete cannot be set for it and it cannot be set for its parent and
> so on.

Yes, this is what it seems to boils done to!

Perhaps the ACPI PM domain is special in this case, because in its
->prepare() callback it asks the ACPI FW about whether the
direct_complete path is possible for a device. In other words, the
ACPI FW seems to be capable of providing information about if a device
may become runtime resumed during system suspend. But, really, isn't
that just a best guess? No?

On those ARM SoCs I am working on, non-ACPI, the information about
whether a device may become runtime resumed during system suspend,
most often can't be find out in a deterministic way. That's because
this information depends on how a device is being used by other
devices, which changes dynamically and from one system suspend
sequence to another. In cases like the i2c-designware-plat driver used
in Hikey (ARM64 board), it can't ever know before hand, if someone is
going to request an i2c transfer during system suspend or not.

To really deal with these devices properly, we have to suspend them in
the correct order.

My point is, doesn't the ordering problem exists also for a device,
which the PM core runs the direct_complete path for, even if the ACPI
PM domain is attached to the device? Just because runtime PM is
disabled for a device, doesn't mean the ordering issue disappears,
right?

>
> [cut]
>
>> >> @@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
>> >>        * A positive return value from ->prepare() means "this device appears
>> >>        * to be runtime-suspended and its state is fine, so if it really is
>> >>        * runtime-suspended, you can leave it in that state provided that you
>> >> -      * will do the same thing with all of its descendants".  This only
>> >> -      * applies to suspend transitions, however.
>> >> +      * will do the same thing with all of its descendants". To allow this,
>> >> +      * the is_rpm_sleep flag must not be set, which is default false, but
>> >> +      * may have been changed by a driver. This only applies to suspend
>> >> +      * transitions, however.
>> >>        */
>> >>       spin_lock_irq(&dev->power.lock);
>> >> -     dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
>> >> +     dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
>> >> +                             state.event == PM_EVENT_SUSPEND;
>> >
>> > The only problem addressed by avoiding direct_complete is when runtime PM needs
>> > to work during __device_suspend() for devices with direct_complete set.  You
>> > said that this was not the case with the i2c designware driver, so I'm not sure
>> > what the purpose of this is.
>>
>> I should probably work more on my changelog, because I tried to
>> explain it there.
>>
>> Anyway, let me quote the important parts, and this is not specific for
>> the i2c-designware-driver, but generic to drivers using
>> pm_runtime_force_suspend|resume().
>>
>> "In the runtime PM centric path, the driver is expected to make use of
>> the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
>> support. More precisely it may assign the system sleep callbacks to
>> these helpers or may call them from its own callbacks, in case it
>> needs to perform additional actions during system sleep.
>>
>> In other words, the PM core must always invoke the system sleep
>> callbacks for the device when they are present, to allow the driver to
>> deal with the system sleep operations."
>>
>> A concrete example is drivers/spi/spi-fsl-espi.c, but one can easily find more.
>
> In fact, the new flag introduced here means "assume that that the driver of this
> device points ->late_suspend and ->early_resume to pm_runtime_force_suspend()
> and pm_runtime_force_resume(), respectively."  Which then implies that
> direct_complete cannot be used with it and (and with its parent and so on)
> and that it is not necessary to runtime resume it during system suspend.
>
> However, there are (or at least there may be) devices that need not be
> runtime resumed during system suspend even though their drivers don't use
> _force_suspend/resume().
>
> Also there are devices whose drivers don't want direct_complete to be used with
> them, even though they don't use _force_suspend/resume() and they *do* need
> their devices to be runtime resumed during system suspend.
>
> Moreover, some drivers may not mind direct_complete even though their devices
> need not be runtime resumed during system suspend.  All of that doesn't have
> to be related to using _force_suspend/resume() at all.
>
> So I don't see any reason whatever for combining all of these three *different*
> conditions under one flag.

Okay!

>
> Thanks,
> Rafael
>

Kind regards
Uffe

[PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Ulf Hansson]

[...]

>> >> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
>> >> index ea1732e..865737a 100644
>> >> --- a/drivers/base/power/main.c
>> >> +++ b/drivers/base/power/main.c
>> >> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
>> >>               if (parent) {
>> >>                       spin_lock_irq(&parent->power.lock);
>> >>
>> >> -                     dev->parent->power.direct_complete = false;
>> >> +                     if (!dev->power.is_rpm_sleep)
>> >> +                             dev->parent->power.direct_complete = false;
>> >
>> > This doesn't look good to me at all.
>> >
>> > It potentially breaks the fundamental assumption of the direct_complete
>> > mechanism that no devices with that flag set will ever be runtime resumed
>> > during system suspend.
>> >
>> > Which can happen with this change if a device with power.is_rpm_sleep is
>> > resumed causing the parent to be resumed too.
>>
>> Doesn't the exact same problem you describe, already exist prior my change!?
>
> No, it doesn't.
>
>> That is, if the current device is runtime suspended and the
>> direct_complete flag is set for it, then __device_suspend() has
>> already disabled runtime PM for the device, when we reach this point.
>> That means, a following attempts to runtime resume the device will
>> fail and thus also to runtime resume its parent.
>
> That's because any devices with direct_complete set *cannot* be runtime
> resumed after __device_suspend().  That's intentional and how the thing
> is designed.  It cannot work differently, sorry.
>
>> So to me, this is a different problem, related how to work out the
>> correct order of how to suspend devices.
>
> The ordering matters here, but it is not the problem.
>
> Setting direct_complete means that PM callbacks for this device won't be
> invoked going forward.  However, if the state of the device was to change
> (like as a result of runtime PM resume), it would be necessary to run those
> callbacks after all, but after __device_suspend() it may be too late for
> that, because the ->suspend callback may have been skipped already.
>
> The only way to address that is to prevent runtime PM resume of the device
> from happening at the point the PM core decides to use direct_complete for it,
> but that requires runtime PM to be disabled for it.  Moreover, since the
> device is now suspended with runtime PM disabled and its children might rely
> on it if they were active, the children must be suspended with runtime PM
> disabled at this point too.  That's why direct_complete can only be set
> for a device if it is set for the entire hierarchy below it.

Thanks, this was a very nice description of the direct_complete path!

>
> Summary: If you allow a device to be runtime resumed during system susped,
> direct_complete cannot be set for it and it cannot be set for its parent and
> so on.

Yes, this is what it seems to boils done to!

Perhaps the ACPI PM domain is special in this case, because in its
->prepare() callback it asks the ACPI FW about whether the
direct_complete path is possible for a device. In other words, the
ACPI FW seems to be capable of providing information about if a device
may become runtime resumed during system suspend. But, really, isn't
that just a best guess? No?

On those ARM SoCs I am working on, non-ACPI, the information about
whether a device may become runtime resumed during system suspend,
most often can't be find out in a deterministic way. That's because
this information depends on how a device is being used by other
devices, which changes dynamically and from one system suspend
sequence to another. In cases like the i2c-designware-plat driver used
in Hikey (ARM64 board), it can't ever know before hand, if someone is
going to request an i2c transfer during system suspend or not.

To really deal with these devices properly, we have to suspend them in
the correct order.

My point is, doesn't the ordering problem exists also for a device,
which the PM core runs the direct_complete path for, even if the ACPI
PM domain is attached to the device? Just because runtime PM is
disabled for a device, doesn't mean the ordering issue disappears,
right?

>
> [cut]
>
>> >> @@ -1709,11 +1711,14 @@ static int device_prepare(struct device *dev, pm_message_t state)
>> >>        * A positive return value from ->prepare() means "this device appears
>> >>        * to be runtime-suspended and its state is fine, so if it really is
>> >>        * runtime-suspended, you can leave it in that state provided that you
>> >> -      * will do the same thing with all of its descendants".  This only
>> >> -      * applies to suspend transitions, however.
>> >> +      * will do the same thing with all of its descendants". To allow this,
>> >> +      * the is_rpm_sleep flag must not be set, which is default false, but
>> >> +      * may have been changed by a driver. This only applies to suspend
>> >> +      * transitions, however.
>> >>        */
>> >>       spin_lock_irq(&dev->power.lock);
>> >> -     dev->power.direct_complete = ret > 0 && state.event == PM_EVENT_SUSPEND;
>> >> +     dev->power.direct_complete = ret > 0 && !dev->power.is_rpm_sleep &&
>> >> +                             state.event == PM_EVENT_SUSPEND;
>> >
>> > The only problem addressed by avoiding direct_complete is when runtime PM needs
>> > to work during __device_suspend() for devices with direct_complete set.  You
>> > said that this was not the case with the i2c designware driver, so I'm not sure
>> > what the purpose of this is.
>>
>> I should probably work more on my changelog, because I tried to
>> explain it there.
>>
>> Anyway, let me quote the important parts, and this is not specific for
>> the i2c-designware-driver, but generic to drivers using
>> pm_runtime_force_suspend|resume().
>>
>> "In the runtime PM centric path, the driver is expected to make use of
>> the pm_runtime_force_suspend|resume() helpers, to deploy system sleep
>> support. More precisely it may assign the system sleep callbacks to
>> these helpers or may call them from its own callbacks, in case it
>> needs to perform additional actions during system sleep.
>>
>> In other words, the PM core must always invoke the system sleep
>> callbacks for the device when they are present, to allow the driver to
>> deal with the system sleep operations."
>>
>> A concrete example is drivers/spi/spi-fsl-espi.c, but one can easily find more.
>
> In fact, the new flag introduced here means "assume that that the driver of this
> device points ->late_suspend and ->early_resume to pm_runtime_force_suspend()
> and pm_runtime_force_resume(), respectively."  Which then implies that
> direct_complete cannot be used with it and (and with its parent and so on)
> and that it is not necessary to runtime resume it during system suspend.
>
> However, there are (or at least there may be) devices that need not be
> runtime resumed during system suspend even though their drivers don't use
> _force_suspend/resume().
>
> Also there are devices whose drivers don't want direct_complete to be used with
> them, even though they don't use _force_suspend/resume() and they *do* need
> their devices to be runtime resumed during system suspend.
>
> Moreover, some drivers may not mind direct_complete even though their devices
> need not be runtime resumed during system suspend.  All of that doesn't have
> to be related to using _force_suspend/resume() at all.
>
> So I don't see any reason whatever for combining all of these three *different*
> conditions under one flag.

Okay!

>
> Thanks,
> Rafael
>

Kind regards
Uffe

Re: [PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Ulf Hansson]

On 29 August 2017 at 17:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
>> This change enables the ACPI PM domain to cope with drivers that deploys
>> the runtime PM centric path for system sleep.
>
> [cut]
>
>> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
>>   * @dev: Device to handle.
>>   *
>>   * Follow PCI and resume devices suspended at run time before running their
>> - * system suspend callbacks.
>> + * system suspend callbacks. However, try to avoid it in case the runtime PM
>> + * centric path is used for the device and then trust the driver to do the
>> + * right thing.
>>   */
>>  int acpi_subsys_suspend(struct device *dev)
>>  {
>> -     pm_runtime_resume(dev);
>> +     struct acpi_device *adev = ACPI_COMPANION(dev);
>> +
>> +     if (!adev)
>> +             return 0;
>> +
>> +     if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
>> +             pm_runtime_resume(dev);
>> +
>>       return pm_generic_suspend(dev);
>>  }
>>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
>
> Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
> and that's why I added the update_state thing.
>
> Moreover, the is_rpm_sleep flag here has to mean not only that
> direct_complete should not be used with the device, but also that its driver
> is fine with not resuming it.

Let me try to explain this better. I realize the changelog is
misleading around this particular section! Huh, apologize for that!

First, patch1 makes the PM core treat the is_rpm_sleep flag as the
direct_complete isn't allowed for the device.

For that reason, when the is_rpm_sleep is set, there is no point
calling acpi_dev_needs_resume() from acpi_subsys_prepare(), but
instead that can be deferred to acpi_subsys_suspend() - because it
doesn't matter if acpi_subsys_prepare() returns 0 or 1, in either case
the acpi_subsys_suspend() will be called. That's really what goes on
here.

The end result is the same. If the acpi_dev_needs_resume() thinks that
the device needs to be runtime resumed, pm_runtime_resume() is called
for the device in acpi_subsys_suspend().

So, this has nothing to do with whether the driver "is fine with not
resuming it" thing.

>
> IMO it is not a good idea to use one flag for these two different things at the
> same time at all.

Yeah, I guess my upper comment addresses your immediate concern here?

However, there is one other thing the is_rpm_flag means. That is that
the driver has informed the ACPI PM domain, to trust the driver to
deal with system sleep, via re-using the runtime PM callbacks.
So the flag does still have two meanings, but that we can change - of course.

>
> Thanks,
> Rafael
>

Kind regards
Uffe

[PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Ulf Hansson]

On 29 August 2017 at 17:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
>> This change enables the ACPI PM domain to cope with drivers that deploys
>> the runtime PM centric path for system sleep.
>
> [cut]
>
>> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
>>   * @dev: Device to handle.
>>   *
>>   * Follow PCI and resume devices suspended at run time before running their
>> - * system suspend callbacks.
>> + * system suspend callbacks. However, try to avoid it in case the runtime PM
>> + * centric path is used for the device and then trust the driver to do the
>> + * right thing.
>>   */
>>  int acpi_subsys_suspend(struct device *dev)
>>  {
>> -     pm_runtime_resume(dev);
>> +     struct acpi_device *adev = ACPI_COMPANION(dev);
>> +
>> +     if (!adev)
>> +             return 0;
>> +
>> +     if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
>> +             pm_runtime_resume(dev);
>> +
>>       return pm_generic_suspend(dev);
>>  }
>>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
>
> Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
> and that's why I added the update_state thing.
>
> Moreover, the is_rpm_sleep flag here has to mean not only that
> direct_complete should not be used with the device, but also that its driver
> is fine with not resuming it.

Let me try to explain this better. I realize the changelog is
misleading around this particular section! Huh, apologize for that!

First, patch1 makes the PM core treat the is_rpm_sleep flag as the
direct_complete isn't allowed for the device.

For that reason, when the is_rpm_sleep is set, there is no point
calling acpi_dev_needs_resume() from acpi_subsys_prepare(), but
instead that can be deferred to acpi_subsys_suspend() - because it
doesn't matter if acpi_subsys_prepare() returns 0 or 1, in either case
the acpi_subsys_suspend() will be called. That's really what goes on
here.

The end result is the same. If the acpi_dev_needs_resume() thinks that
the device needs to be runtime resumed, pm_runtime_resume() is called
for the device in acpi_subsys_suspend().

So, this has nothing to do with whether the driver "is fine with not
resuming it" thing.

>
> IMO it is not a good idea to use one flag for these two different things at the
> same time at all.

Yeah, I guess my upper comment addresses your immediate concern here?

However, there is one other thing the is_rpm_flag means. That is that
the driver has informed the ACPI PM domain, to trust the driver to
deal with system sleep, via re-using the runtime PM callbacks.
So the flag does still have two meanings, but that we can change - of course.

>
> Thanks,
> Rafael
>

Kind regards
Uffe

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

On 31 August 2017 at 02:17, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> Disclaimer: I'm falling asleep, so I probably shouldn't reply to email right
> now, but tomorrow I may not be able to get to email at all, so I'll try anyway.
>
> On Wednesday, August 30, 2017 11:57:28 AM CEST Ulf Hansson wrote:
>> On 29 August 2017 at 22:19, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> > On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
>> >> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
>> >> isn't well optimized for system sleep.
>> >>
>> >> What makes this driver particularly interesting is because it's a cross-SoC
>> >> driver, which sometimes means there is an ACPI PM domain attached to the i2c
>> >> device and sometimes not. The driver is being used on both x86 and ARM.
>> >>
>> >> In principle, to optimize the system sleep support in i2c driver, this series
>> >> enables the proven runtime PM centric path for the i2c driver. However, to do
>> >> that the ACPI PM domain also have to collaborate and understand this behaviour.
>> >> From earlier versions, Rafael has also pointed out that also the PM core needs
>> >> to be involved.
>> >
>> > Earlier today I realized that drivers pointing their ->suspend_late and
>> > ->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
>> > pm_runtime_force_resume(), are fundamentally incompatible with any bus type
>> > doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.
>> >
>> > First, it basically requires the bus type or PM domain to expect that its
>> > ->runtime_suspend callback may or may not be indirectly invoked from its
>> > own ->suspend_late callback, depending on the driver (and analogously
>> > for ->runtime_resume and ->resume early), which is insane.
>> >
>> > Second, it is a layering violation, because it makes the driver effectively
>> > override the upper layer's decisions about what code to run.
>>
>> You are right that for more complex bus types and PM domains, those
>> needs to play along. So that is what I am trying to implement for the
>> ACPI PM domain in this series.
>
> Well, "play along" is a bit of an understatement here.
>
> They would need to turn into horrible mess and that's not going to happen.

I absolutely agree, there must be no mess what so ever!

But, I don't want to give up yet, I still believe I can make this
series into a nice couple of changes for the ACPI PM domain.
Especially if you continue giving me your guidance.

>
>> The generic PM domain, is simple in this regards. There is only a
>> minor adaptation for the ->runtime_suspend|resume() callbacks, which
>> avoids validating dev_pm_qos constraints during system sleep. Nothing
>> special is needed in ->suspend_late|noirq callbacks, etc.
>>
>> For most other simple bus types, like the platform bus, spi, i2c,
>> amba, no particular adoptions is needed at all. Instead those just
>> trust the drivers to do the right thing.
>
> They are the trivial ones.

Yes.

However, the platform bus is also very commonly used in kernel. I
think that's an important thing to consider.

>
>> Before we had the direct_complete path, using the
>> pm_runtime_force_suspend|resume() helpers, was the only good way for
>> these kind of drivers, to in an optimized manner, deal with system
>> sleep when runtime PM also was enabled for their devices. Now this
>> method has become widely deployed, unfortunate whether you like it or
>> not.
>
> So can you please remind me why the _force_ wrappers are needed?

See below.

>
> In particular, why can't drivers arrange their callbacks the way I did that
> in https://patchwork.kernel.org/patch/9928583/ ?

I was preparing a reply to that patch, but let me summarize that here instead.

Let me be clear, the patch is an improvement of the behavior of the
driver and it addresses the issues you point out in the change log.
Re-using the runtime PM callbacks for system sleep, is nice as it
avoids open coding, which is of curse also one of the reason of using
pm_runtime_force_suspend|resume().

Still there are a couple of things I am worried about in this patch.
*)
To be able to re-use the same callbacks for system sleep and runtime
PM, some boilerplate code is added to the driver, as to cope with the
different conditions inside the callbacks. That pattern would become
repeated to many drivers dealing with similar issues.

**)
The ->resume_early() callback powers on the device, in case it was
runtime resumed when the ->suspend_late() callback was invoked. That
is in many cases completely unnecessary, causing us to waste power and
increase system resume time, for absolutely no reason. However, I
understand the patch didn't try to address this, but to really fix
this, there has to be an even closer collaboration between runtime PM
and the system sleep callbacks.

So, to remind you why the pm_runtime_force_suspend|resume() helpers is
preferred, that's because both of the above two things becomes taken
care of.

>
>> Besides the slightly better optimizations you get when using
>> pm_runtime_force_suspend|resume(), comparing to the direct_complete
>> path - I think it's also worth to consider, how easy it becomes for
>> drivers to deploy system sleep support. In many cases, only two lines
>> of code is needed to add system sleep support in a driver.
>
> You are doing a wrong comparison here IMO.  You essentially are comparing two
> bandaids with each other and arguing that one of them somehow is better.

I just wanted to compare against something...

>
> What about doing something which is not a bandaid instead?

I don't have a problem working on something new, but I am not sure
what that should be.

Unless you re-consider moving forward in some form, with the current
suggested approach for the ACPI PM domain, can you give me some
pointers on what you have in mind?

To remind you of my current view, the direct_complete path is useful
for PM domains, like the ACPI PM domain as it impacts all its devices.
Using pm_runtime_force_suspend|resume() offers the next steps to
achieve a fully optimized behavior of a device during system sleep, as
already been proven by now. It would be great to both options
supported by the ACPI PM domain.

Another related thing that is causing lots of problems during system
sleep of devices, but not related to optimizations, is to have the
correct order of how to suspend/resume the devices. We have talked
about this, but it's a separate problem and it's rather a deployment
issue, than having to implements something entirely new (we have
supplies/consumers links you invented for this).

>
>> Now, some complex code always needs to be implemented somewhere. When
>> using the runtime PM centric path, that code consist of the
>> pm_runtime_force_suspend|resume() helpers itself - and some
>> adaptations in buses/PM domains in cases when those needs special
>> care.
>>
>> My point is, the runtime PM centric path, allows us to keep the
>> complex part of the code at a few centralized places, instead of
>> having it spread/duplicated into drivers.
>>
>> So yes, you could consider it insane, but to me and many others, it
>> seems to work out quite well.
>
> Because it only has been used with trivial middle layer code so far
> and I'm quite disappointed that you don't seem to see a problem here. :-/
>
> I mean something like
>
> PM core => bus type / PM domain ->suspend_late => driver ->suspend_late
>
> is far more straightforward than
>
> PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
>         bus type / PM domain ->runtime_suspend => driver ->runtime_suspend
>
> with the bus type / PM domain having to figure out somehow at the
> ->suspend_late time whether or not its ->runtume_suspend is going to be invoked
> in the middle of it.
>
> Apart from this just being aesthetically disgusting to me, which admittedly is
> a matter of personal opinion, it makes debugging new driver code harder (if it
> happens to not work) and reviewing it almost impossible, because now you need
> to take all of the tangling between callbacks into accont and sometimes not
> just for one bus type / PM domain.

I am wondering that perhaps you may be overlooking some of the
internals of runtime PM. Or maybe not? :-)

I mean, the hole thing is build upon that anyone can call runtime PM
functions to runtime resume/suspend a device. Doing that, makes the
hierarchy of the runtime PM callbacks being walked and invoked, of
course properly managed by the runtime PM core.

My point is that, the runtime PM core still controls this behavior,
even when the pm_runtime_force_suspend|resume() helpers are being
invoked. The only difference is that it allows runtime PM for the
device to be disabled, and still correctly invoked the callbacks. That
is what it is all about.

>
>> Yeah, and the laying violation is undoubtedly the most controversial
>> part of the runtime PM centric path - I agree to that! The
>> direct_complete path don't have this, as you implemented it. :-)
>>
>> On the other hand, one could consider that these upper layers, in many
>> cases anyway needs to play along with the behavior of the driver. So,
>> I guess it depends on how one see it.
>>
>> >
>> > That's why I'm afraid that we've reached a dead end here. :-(
>>
>> That's said news. Is was really hoping I could find a way to move this
>> forward. You don't have any other ideas on how I can adjust the series
>> to make you happy?
>
> So to be precise, patches [2-3/8] are basically fine by me.  Patch [4/8]
> sort of works too, but I'd do the splitting slightly differently and I don't
> see much value in it alone.
>
> The rest of the ACPI changes is mostly not acceptable to me, mostly because
> of what is done to the PM domain's ->runtime_suspend/resume and
> ->suspend_late/->resume_early callbacks.
>
> I guess the only way that could be made work for me would be by not using
> _force_suspend/resume() at all, but that would defeat the point, right?

Yes, it would.

>
> I don't like the flag too, but that might be worked out.

Yeah, I am open to any suggestion.

>
> Also, when I looked at _force_suspend/resume() again, I got concerned.
> There is stuff in there that shouldn't be necessary in a driver's
> ->late_suspend/->early_resume and some things in there just made me
> scratch my head.

Yes, there are some complexity in there, I will be happy to answer any
specific question about it.

The main thing is, that it tries to conform to the regular rules set
by the runtime PM core when runtime PM is enabled for the device - and
then apply those to the device when runtime PM has been disabled for
it.

Again, thanks for being patient and reviewing!

Kind regards
Uffe

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

On 31 August 2017 at 02:17, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> Disclaimer: I'm falling asleep, so I probably shouldn't reply to email right
> now, but tomorrow I may not be able to get to email at all, so I'll try anyway.
>
> On Wednesday, August 30, 2017 11:57:28 AM CEST Ulf Hansson wrote:
>> On 29 August 2017 at 22:19, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> > On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
>> >> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
>> >> isn't well optimized for system sleep.
>> >>
>> >> What makes this driver particularly interesting is because it's a cross-SoC
>> >> driver, which sometimes means there is an ACPI PM domain attached to the i2c
>> >> device and sometimes not. The driver is being used on both x86 and ARM.
>> >>
>> >> In principle, to optimize the system sleep support in i2c driver, this series
>> >> enables the proven runtime PM centric path for the i2c driver. However, to do
>> >> that the ACPI PM domain also have to collaborate and understand this behaviour.
>> >> From earlier versions, Rafael has also pointed out that also the PM core needs
>> >> to be involved.
>> >
>> > Earlier today I realized that drivers pointing their ->suspend_late and
>> > ->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
>> > pm_runtime_force_resume(), are fundamentally incompatible with any bus type
>> > doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.
>> >
>> > First, it basically requires the bus type or PM domain to expect that its
>> > ->runtime_suspend callback may or may not be indirectly invoked from its
>> > own ->suspend_late callback, depending on the driver (and analogously
>> > for ->runtime_resume and ->resume early), which is insane.
>> >
>> > Second, it is a layering violation, because it makes the driver effectively
>> > override the upper layer's decisions about what code to run.
>>
>> You are right that for more complex bus types and PM domains, those
>> needs to play along. So that is what I am trying to implement for the
>> ACPI PM domain in this series.
>
> Well, "play along" is a bit of an understatement here.
>
> They would need to turn into horrible mess and that's not going to happen.

I absolutely agree, there must be no mess what so ever!

But, I don't want to give up yet, I still believe I can make this
series into a nice couple of changes for the ACPI PM domain.
Especially if you continue giving me your guidance.

>
>> The generic PM domain, is simple in this regards. There is only a
>> minor adaptation for the ->runtime_suspend|resume() callbacks, which
>> avoids validating dev_pm_qos constraints during system sleep. Nothing
>> special is needed in ->suspend_late|noirq callbacks, etc.
>>
>> For most other simple bus types, like the platform bus, spi, i2c,
>> amba, no particular adoptions is needed at all. Instead those just
>> trust the drivers to do the right thing.
>
> They are the trivial ones.

Yes.

However, the platform bus is also very commonly used in kernel. I
think that's an important thing to consider.

>
>> Before we had the direct_complete path, using the
>> pm_runtime_force_suspend|resume() helpers, was the only good way for
>> these kind of drivers, to in an optimized manner, deal with system
>> sleep when runtime PM also was enabled for their devices. Now this
>> method has become widely deployed, unfortunate whether you like it or
>> not.
>
> So can you please remind me why the _force_ wrappers are needed?

See below.

>
> In particular, why can't drivers arrange their callbacks the way I did that
> in https://patchwork.kernel.org/patch/9928583/ ?

I was preparing a reply to that patch, but let me summarize that here instead.

Let me be clear, the patch is an improvement of the behavior of the
driver and it addresses the issues you point out in the change log.
Re-using the runtime PM callbacks for system sleep, is nice as it
avoids open coding, which is of curse also one of the reason of using
pm_runtime_force_suspend|resume().

Still there are a couple of things I am worried about in this patch.
*)
To be able to re-use the same callbacks for system sleep and runtime
PM, some boilerplate code is added to the driver, as to cope with the
different conditions inside the callbacks. That pattern would become
repeated to many drivers dealing with similar issues.

**)
The ->resume_early() callback powers on the device, in case it was
runtime resumed when the ->suspend_late() callback was invoked. That
is in many cases completely unnecessary, causing us to waste power and
increase system resume time, for absolutely no reason. However, I
understand the patch didn't try to address this, but to really fix
this, there has to be an even closer collaboration between runtime PM
and the system sleep callbacks.

So, to remind you why the pm_runtime_force_suspend|resume() helpers is
preferred, that's because both of the above two things becomes taken
care of.

>
>> Besides the slightly better optimizations you get when using
>> pm_runtime_force_suspend|resume(), comparing to the direct_complete
>> path - I think it's also worth to consider, how easy it becomes for
>> drivers to deploy system sleep support. In many cases, only two lines
>> of code is needed to add system sleep support in a driver.
>
> You are doing a wrong comparison here IMO.  You essentially are comparing two
> bandaids with each other and arguing that one of them somehow is better.

I just wanted to compare against something...

>
> What about doing something which is not a bandaid instead?

I don't have a problem working on something new, but I am not sure
what that should be.

Unless you re-consider moving forward in some form, with the current
suggested approach for the ACPI PM domain, can you give me some
pointers on what you have in mind?

To remind you of my current view, the direct_complete path is useful
for PM domains, like the ACPI PM domain as it impacts all its devices.
Using pm_runtime_force_suspend|resume() offers the next steps to
achieve a fully optimized behavior of a device during system sleep, as
already been proven by now. It would be great to both options
supported by the ACPI PM domain.

Another related thing that is causing lots of problems during system
sleep of devices, but not related to optimizations, is to have the
correct order of how to suspend/resume the devices. We have talked
about this, but it's a separate problem and it's rather a deployment
issue, than having to implements something entirely new (we have
supplies/consumers links you invented for this).

>
>> Now, some complex code always needs to be implemented somewhere. When
>> using the runtime PM centric path, that code consist of the
>> pm_runtime_force_suspend|resume() helpers itself - and some
>> adaptations in buses/PM domains in cases when those needs special
>> care.
>>
>> My point is, the runtime PM centric path, allows us to keep the
>> complex part of the code at a few centralized places, instead of
>> having it spread/duplicated into drivers.
>>
>> So yes, you could consider it insane, but to me and many others, it
>> seems to work out quite well.
>
> Because it only has been used with trivial middle layer code so far
> and I'm quite disappointed that you don't seem to see a problem here. :-/
>
> I mean something like
>
> PM core => bus type / PM domain ->suspend_late => driver ->suspend_late
>
> is far more straightforward than
>
> PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
>         bus type / PM domain ->runtime_suspend => driver ->runtime_suspend
>
> with the bus type / PM domain having to figure out somehow at the
> ->suspend_late time whether or not its ->runtume_suspend is going to be invoked
> in the middle of it.
>
> Apart from this just being aesthetically disgusting to me, which admittedly is
> a matter of personal opinion, it makes debugging new driver code harder (if it
> happens to not work) and reviewing it almost impossible, because now you need
> to take all of the tangling between callbacks into accont and sometimes not
> just for one bus type / PM domain.

I am wondering that perhaps you may be overlooking some of the
internals of runtime PM. Or maybe not? :-)

I mean, the hole thing is build upon that anyone can call runtime PM
functions to runtime resume/suspend a device. Doing that, makes the
hierarchy of the runtime PM callbacks being walked and invoked, of
course properly managed by the runtime PM core.

My point is that, the runtime PM core still controls this behavior,
even when the pm_runtime_force_suspend|resume() helpers are being
invoked. The only difference is that it allows runtime PM for the
device to be disabled, and still correctly invoked the callbacks. That
is what it is all about.

>
>> Yeah, and the laying violation is undoubtedly the most controversial
>> part of the runtime PM centric path - I agree to that! The
>> direct_complete path don't have this, as you implemented it. :-)
>>
>> On the other hand, one could consider that these upper layers, in many
>> cases anyway needs to play along with the behavior of the driver. So,
>> I guess it depends on how one see it.
>>
>> >
>> > That's why I'm afraid that we've reached a dead end here. :-(
>>
>> That's said news. Is was really hoping I could find a way to move this
>> forward. You don't have any other ideas on how I can adjust the series
>> to make you happy?
>
> So to be precise, patches [2-3/8] are basically fine by me.  Patch [4/8]
> sort of works too, but I'd do the splitting slightly differently and I don't
> see much value in it alone.
>
> The rest of the ACPI changes is mostly not acceptable to me, mostly because
> of what is done to the PM domain's ->runtime_suspend/resume and
> ->suspend_late/->resume_early callbacks.
>
> I guess the only way that could be made work for me would be by not using
> _force_suspend/resume() at all, but that would defeat the point, right?

Yes, it would.

>
> I don't like the flag too, but that might be worked out.

Yeah, I am open to any suggestion.

>
> Also, when I looked at _force_suspend/resume() again, I got concerned.
> There is stuff in there that shouldn't be necessary in a driver's
> ->late_suspend/->early_resume and some things in there just made me
> scratch my head.

Yes, there are some complexity in there, I will be happy to answer any
specific question about it.

The main thing is, that it tries to conform to the regular rules set
by the runtime PM core when runtime PM is enabled for the device - and
then apply those to the device when runtime PM has been disabled for
it.

Again, thanks for being patient and reviewing!

Kind regards
Uffe

Re: [PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Rafael J. Wysocki]

On Thursday, August 31, 2017 11:06:05 AM CEST Ulf Hansson wrote:
> [...]
> 
> >> >> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> >> >> index ea1732e..865737a 100644
> >> >> --- a/drivers/base/power/main.c
> >> >> +++ b/drivers/base/power/main.c
> >> >> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
> >> >>               if (parent) {
> >> >>                       spin_lock_irq(&parent->power.lock);
> >> >>
> >> >> -                     dev->parent->power.direct_complete = false;
> >> >> +                     if (!dev->power.is_rpm_sleep)
> >> >> +                             dev->parent->power.direct_complete = false;
> >> >
> >> > This doesn't look good to me at all.
> >> >
> >> > It potentially breaks the fundamental assumption of the direct_complete
> >> > mechanism that no devices with that flag set will ever be runtime resumed
> >> > during system suspend.
> >> >
> >> > Which can happen with this change if a device with power.is_rpm_sleep is
> >> > resumed causing the parent to be resumed too.
> >>
> >> Doesn't the exact same problem you describe, already exist prior my change!?
> >
> > No, it doesn't.
> >
> >> That is, if the current device is runtime suspended and the
> >> direct_complete flag is set for it, then __device_suspend() has
> >> already disabled runtime PM for the device, when we reach this point.
> >> That means, a following attempts to runtime resume the device will
> >> fail and thus also to runtime resume its parent.
> >
> > That's because any devices with direct_complete set *cannot* be runtime
> > resumed after __device_suspend().  That's intentional and how the thing
> > is designed.  It cannot work differently, sorry.
> >
> >> So to me, this is a different problem, related how to work out the
> >> correct order of how to suspend devices.
> >
> > The ordering matters here, but it is not the problem.
> >
> > Setting direct_complete means that PM callbacks for this device won't be
> > invoked going forward.  However, if the state of the device was to change
> > (like as a result of runtime PM resume), it would be necessary to run those
> > callbacks after all, but after __device_suspend() it may be too late for
> > that, because the ->suspend callback may have been skipped already.
> >
> > The only way to address that is to prevent runtime PM resume of the device
> > from happening at the point the PM core decides to use direct_complete for it,
> > but that requires runtime PM to be disabled for it.  Moreover, since the
> > device is now suspended with runtime PM disabled and its children might rely
> > on it if they were active, the children must be suspended with runtime PM
> > disabled at this point too.  That's why direct_complete can only be set
> > for a device if it is set for the entire hierarchy below it.
> 
> Thanks, this was a very nice description of the direct_complete path!
> 
> >
> > Summary: If you allow a device to be runtime resumed during system susped,
> > direct_complete cannot be set for it and it cannot be set for its parent and
> > so on.
> 
> Yes, this is what it seems to boils done to!
> 
> Perhaps the ACPI PM domain is special in this case, because in its
> ->prepare() callback it asks the ACPI FW about whether the
> direct_complete path is possible for a device. In other words, the
> ACPI FW seems to be capable of providing information about if a device
> may become runtime resumed during system suspend. But, really, isn't
> that just a best guess? No?

I wouldn't call it a guess, but it may go too far.

The part that the ACPI PM domain can figure out is whether or not the power
state of the device needs to be updated due to differences between runtime
PM and system suspend configuration requirements.  If it doesn't need to be
updated, acpi_subsys_prepare() returns 1 which may me overoptimistic.

For many devices that works, because the only possible reason why they may
need to be accessed during system suspend is to update their power state.

It may not work for some devices, though, because there may be other reasons
for accessing them during system suspend and that's where some opt-out way
is needed.

> On those ARM SoCs I am working on, non-ACPI, the information about
> whether a device may become runtime resumed during system suspend,
> most often can't be find out in a deterministic way. That's because
> this information depends on how a device is being used by other
> devices, which changes dynamically and from one system suspend
> sequence to another. In cases like the i2c-designware-plat driver used
> in Hikey (ARM64 board), it can't ever know before hand, if someone is
> going to request an i2c transfer during system suspend or not.

Well, I2C is somewhat special in that respect, because dependencies between
devices in there are not tracked AFAICS. 

> To really deal with these devices properly, we have to suspend them in
> the correct order.

Right, but in order to achieve that the right ordering actually needs
to be known.  If it is not known, there is no way to get that right in
general. :-)

The problem basically is that during system suspend you need to disable
the controller at one point, sooner or later, and keep it disabled from
that point on.   Of course, that should happen when it is guaranteed that
there won't be any new transactions going forward, but you cannot actually
guaranee that without knowing that all of its "consumers" have already
been suspended.

> My point is, doesn't the ordering problem exists also for a device,
> which the PM core runs the direct_complete path for, even if the ACPI
> PM domain is attached to the device? Just because runtime PM is
> disabled for a device, doesn't mean the ordering issue disappears,
> right?

The ordering issue is there if there are dependencies between devices the
PM core doesn't know about.  Otherwise, the rule that direct_complete can
only be used for a given device if it also is used for the entire hierarchy
below it (and for all of the device's "consumers" and everything that depends
on them too for that matter) makes it go away (it serves as a guarantee that
all of the device's "consumers" have already been suspended).

Thanks,
Rafael

[PATCH v3 1/8] PM / Sleep: Make the runtime PM centric path known to the PM core

[Rafael J. Wysocki]

On Thursday, August 31, 2017 11:06:05 AM CEST Ulf Hansson wrote:
> [...]
> 
> >> >> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> >> >> index ea1732e..865737a 100644
> >> >> --- a/drivers/base/power/main.c
> >> >> +++ b/drivers/base/power/main.c
> >> >> @@ -1549,14 +1549,16 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
> >> >>               if (parent) {
> >> >>                       spin_lock_irq(&parent->power.lock);
> >> >>
> >> >> -                     dev->parent->power.direct_complete = false;
> >> >> +                     if (!dev->power.is_rpm_sleep)
> >> >> +                             dev->parent->power.direct_complete = false;
> >> >
> >> > This doesn't look good to me at all.
> >> >
> >> > It potentially breaks the fundamental assumption of the direct_complete
> >> > mechanism that no devices with that flag set will ever be runtime resumed
> >> > during system suspend.
> >> >
> >> > Which can happen with this change if a device with power.is_rpm_sleep is
> >> > resumed causing the parent to be resumed too.
> >>
> >> Doesn't the exact same problem you describe, already exist prior my change!?
> >
> > No, it doesn't.
> >
> >> That is, if the current device is runtime suspended and the
> >> direct_complete flag is set for it, then __device_suspend() has
> >> already disabled runtime PM for the device, when we reach this point.
> >> That means, a following attempts to runtime resume the device will
> >> fail and thus also to runtime resume its parent.
> >
> > That's because any devices with direct_complete set *cannot* be runtime
> > resumed after __device_suspend().  That's intentional and how the thing
> > is designed.  It cannot work differently, sorry.
> >
> >> So to me, this is a different problem, related how to work out the
> >> correct order of how to suspend devices.
> >
> > The ordering matters here, but it is not the problem.
> >
> > Setting direct_complete means that PM callbacks for this device won't be
> > invoked going forward.  However, if the state of the device was to change
> > (like as a result of runtime PM resume), it would be necessary to run those
> > callbacks after all, but after __device_suspend() it may be too late for
> > that, because the ->suspend callback may have been skipped already.
> >
> > The only way to address that is to prevent runtime PM resume of the device
> > from happening at the point the PM core decides to use direct_complete for it,
> > but that requires runtime PM to be disabled for it.  Moreover, since the
> > device is now suspended with runtime PM disabled and its children might rely
> > on it if they were active, the children must be suspended with runtime PM
> > disabled at this point too.  That's why direct_complete can only be set
> > for a device if it is set for the entire hierarchy below it.
> 
> Thanks, this was a very nice description of the direct_complete path!
> 
> >
> > Summary: If you allow a device to be runtime resumed during system susped,
> > direct_complete cannot be set for it and it cannot be set for its parent and
> > so on.
> 
> Yes, this is what it seems to boils done to!
> 
> Perhaps the ACPI PM domain is special in this case, because in its
> ->prepare() callback it asks the ACPI FW about whether the
> direct_complete path is possible for a device. In other words, the
> ACPI FW seems to be capable of providing information about if a device
> may become runtime resumed during system suspend. But, really, isn't
> that just a best guess? No?

I wouldn't call it a guess, but it may go too far.

The part that the ACPI PM domain can figure out is whether or not the power
state of the device needs to be updated due to differences between runtime
PM and system suspend configuration requirements.  If it doesn't need to be
updated, acpi_subsys_prepare() returns 1 which may me overoptimistic.

For many devices that works, because the only possible reason why they may
need to be accessed during system suspend is to update their power state.

It may not work for some devices, though, because there may be other reasons
for accessing them during system suspend and that's where some opt-out way
is needed.

> On those ARM SoCs I am working on, non-ACPI, the information about
> whether a device may become runtime resumed during system suspend,
> most often can't be find out in a deterministic way. That's because
> this information depends on how a device is being used by other
> devices, which changes dynamically and from one system suspend
> sequence to another. In cases like the i2c-designware-plat driver used
> in Hikey (ARM64 board), it can't ever know before hand, if someone is
> going to request an i2c transfer during system suspend or not.

Well, I2C is somewhat special in that respect, because dependencies between
devices in there are not tracked AFAICS. 

> To really deal with these devices properly, we have to suspend them in
> the correct order.

Right, but in order to achieve that the right ordering actually needs
to be known.  If it is not known, there is no way to get that right in
general. :-)

The problem basically is that during system suspend you need to disable
the controller at one point, sooner or later, and keep it disabled from
that point on.   Of course, that should happen when it is guaranteed that
there won't be any new transactions going forward, but you cannot actually
guaranee that without knowing that all of its "consumers" have already
been suspended.

> My point is, doesn't the ordering problem exists also for a device,
> which the PM core runs the direct_complete path for, even if the ACPI
> PM domain is attached to the device? Just because runtime PM is
> disabled for a device, doesn't mean the ordering issue disappears,
> right?

The ordering issue is there if there are dependencies between devices the
PM core doesn't know about.  Otherwise, the rule that direct_complete can
only be used for a given device if it also is used for the entire hierarchy
below it (and for all of the device's "consumers" and everything that depends
on them too for that matter) makes it go away (it serves as a guarantee that
all of the device's "consumers" have already been suspended).

Thanks,
Rafael

Re: [PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Rafael J. Wysocki]

On Friday, September 1, 2017 10:27:05 AM CEST Ulf Hansson wrote:
> On 29 August 2017 at 17:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
> >> This change enables the ACPI PM domain to cope with drivers that deploys
> >> the runtime PM centric path for system sleep.
> >
> > [cut]
> >
> >> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
> >>   * @dev: Device to handle.
> >>   *
> >>   * Follow PCI and resume devices suspended at run time before running their
> >> - * system suspend callbacks.
> >> + * system suspend callbacks. However, try to avoid it in case the runtime PM
> >> + * centric path is used for the device and then trust the driver to do the
> >> + * right thing.
> >>   */
> >>  int acpi_subsys_suspend(struct device *dev)
> >>  {
> >> -     pm_runtime_resume(dev);
> >> +     struct acpi_device *adev = ACPI_COMPANION(dev);
> >> +
> >> +     if (!adev)
> >> +             return 0;
> >> +
> >> +     if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
> >> +             pm_runtime_resume(dev);
> >> +
> >>       return pm_generic_suspend(dev);
> >>  }
> >>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
> >
> > Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
> > and that's why I added the update_state thing.
> >
> > Moreover, the is_rpm_sleep flag here has to mean not only that
> > direct_complete should not be used with the device, but also that its driver
> > is fine with not resuming it.
> 
> Let me try to explain this better. I realize the changelog is
> misleading around this particular section! Huh, apologize for that!
> 
> First, patch1 makes the PM core treat the is_rpm_sleep flag as the
> direct_complete isn't allowed for the device.
> 
> For that reason, when the is_rpm_sleep is set, there is no point
> calling acpi_dev_needs_resume() from acpi_subsys_prepare(), but
> instead that can be deferred to acpi_subsys_suspend() - because it
> doesn't matter if acpi_subsys_prepare() returns 0 or 1, in either case
> the acpi_subsys_suspend() will be called. That's really what goes on
> here.
> 
> The end result is the same. If the acpi_dev_needs_resume() thinks that
> the device needs to be runtime resumed, pm_runtime_resume() is called
> for the device in acpi_subsys_suspend().
> 
> So, this has nothing to do with whether the driver "is fine with not
> resuming it" thing.

No, sorry.

If is_rpm_sleep was not set, the ACPI PM domain would resume the device in
acpi_subsys_suspend() regardless of the acpi_dev_needs_resume() return value.
That's what's there in the patch.  So clearly, setting is_rpm_sleep means
"this device does not need to be resumed in acpi_subsys_suspend() unless
acpi_dev_needs_resume() returns true".  Which clearly means that the driver
*is* fine with not resuming it, because if is_rpm_sleep is set, the device
in fact may not be resumed and then the driver will need to cope with that.

And note that this meaning of is_rpm_sleep is different from what it is
expected to mean to the core.

> >
> > IMO it is not a good idea to use one flag for these two different things at the
> > same time at all.
> 
> Yeah, I guess my upper comment addresses your immediate concern here?

No, they don't.

> However, there is one other thing the is_rpm_flag means. That is that
> the driver has informed the ACPI PM domain, to trust the driver to
> deal with system sleep, via re-using the runtime PM callbacks.
> So the flag does still have two meanings, but that we can change - of course.

I guess that you are referring to the use of dev_pm_is_rpm_sleep() in
acpi_subsys_suspend_late()?  That's the third thing this flag means ...

Thanks,
Rafael

[PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Rafael J. Wysocki]

On Friday, September 1, 2017 10:27:05 AM CEST Ulf Hansson wrote:
> On 29 August 2017 at 17:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
> >> This change enables the ACPI PM domain to cope with drivers that deploys
> >> the runtime PM centric path for system sleep.
> >
> > [cut]
> >
> >> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
> >>   * @dev: Device to handle.
> >>   *
> >>   * Follow PCI and resume devices suspended at run time before running their
> >> - * system suspend callbacks.
> >> + * system suspend callbacks. However, try to avoid it in case the runtime PM
> >> + * centric path is used for the device and then trust the driver to do the
> >> + * right thing.
> >>   */
> >>  int acpi_subsys_suspend(struct device *dev)
> >>  {
> >> -     pm_runtime_resume(dev);
> >> +     struct acpi_device *adev = ACPI_COMPANION(dev);
> >> +
> >> +     if (!adev)
> >> +             return 0;
> >> +
> >> +     if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
> >> +             pm_runtime_resume(dev);
> >> +
> >>       return pm_generic_suspend(dev);
> >>  }
> >>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
> >
> > Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
> > and that's why I added the update_state thing.
> >
> > Moreover, the is_rpm_sleep flag here has to mean not only that
> > direct_complete should not be used with the device, but also that its driver
> > is fine with not resuming it.
> 
> Let me try to explain this better. I realize the changelog is
> misleading around this particular section! Huh, apologize for that!
> 
> First, patch1 makes the PM core treat the is_rpm_sleep flag as the
> direct_complete isn't allowed for the device.
> 
> For that reason, when the is_rpm_sleep is set, there is no point
> calling acpi_dev_needs_resume() from acpi_subsys_prepare(), but
> instead that can be deferred to acpi_subsys_suspend() - because it
> doesn't matter if acpi_subsys_prepare() returns 0 or 1, in either case
> the acpi_subsys_suspend() will be called. That's really what goes on
> here.
> 
> The end result is the same. If the acpi_dev_needs_resume() thinks that
> the device needs to be runtime resumed, pm_runtime_resume() is called
> for the device in acpi_subsys_suspend().
> 
> So, this has nothing to do with whether the driver "is fine with not
> resuming it" thing.

No, sorry.

If is_rpm_sleep was not set, the ACPI PM domain would resume the device in
acpi_subsys_suspend() regardless of the acpi_dev_needs_resume() return value.
That's what's there in the patch.  So clearly, setting is_rpm_sleep means
"this device does not need to be resumed in acpi_subsys_suspend() unless
acpi_dev_needs_resume() returns true".  Which clearly means that the driver
*is* fine with not resuming it, because if is_rpm_sleep is set, the device
in fact may not be resumed and then the driver will need to cope with that.

And note that this meaning of is_rpm_sleep is different from what it is
expected to mean to the core.

> >
> > IMO it is not a good idea to use one flag for these two different things at the
> > same time at all.
> 
> Yeah, I guess my upper comment addresses your immediate concern here?

No, they don't.

> However, there is one other thing the is_rpm_flag means. That is that
> the driver has informed the ACPI PM domain, to trust the driver to
> deal with system sleep, via re-using the runtime PM callbacks.
> So the flag does still have two meanings, but that we can change - of course.

I guess that you are referring to the use of dev_pm_is_rpm_sleep() in
acpi_subsys_suspend_late()?  That's the third thing this flag means ...

Thanks,
Rafael

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Friday, September 1, 2017 12:42:35 PM CEST Ulf Hansson wrote:
> On 31 August 2017 at 02:17, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > Disclaimer: I'm falling asleep, so I probably shouldn't reply to email right
> > now, but tomorrow I may not be able to get to email at all, so I'll try anyway.
> >
> > On Wednesday, August 30, 2017 11:57:28 AM CEST Ulf Hansson wrote:
> >> On 29 August 2017 at 22:19, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> >> > On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
> >> >> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
> >> >> isn't well optimized for system sleep.
> >> >>
> >> >> What makes this driver particularly interesting is because it's a cross-SoC
> >> >> driver, which sometimes means there is an ACPI PM domain attached to the i2c
> >> >> device and sometimes not. The driver is being used on both x86 and ARM.
> >> >>
> >> >> In principle, to optimize the system sleep support in i2c driver, this series
> >> >> enables the proven runtime PM centric path for the i2c driver. However, to do
> >> >> that the ACPI PM domain also have to collaborate and understand this behaviour.
> >> >> From earlier versions, Rafael has also pointed out that also the PM core needs
> >> >> to be involved.
> >> >
> >> > Earlier today I realized that drivers pointing their ->suspend_late and
> >> > ->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
> >> > pm_runtime_force_resume(), are fundamentally incompatible with any bus type
> >> > doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.
> >> >
> >> > First, it basically requires the bus type or PM domain to expect that its
> >> > ->runtime_suspend callback may or may not be indirectly invoked from its
> >> > own ->suspend_late callback, depending on the driver (and analogously
> >> > for ->runtime_resume and ->resume early), which is insane.
> >> >
> >> > Second, it is a layering violation, because it makes the driver effectively
> >> > override the upper layer's decisions about what code to run.
> >>
> >> You are right that for more complex bus types and PM domains, those
> >> needs to play along. So that is what I am trying to implement for the
> >> ACPI PM domain in this series.
> >
> > Well, "play along" is a bit of an understatement here.
> >
> > They would need to turn into horrible mess and that's not going to happen.
> 
> I absolutely agree, there must be no mess what so ever!
> 
> But, I don't want to give up yet, I still believe I can make this
> series into a nice couple of changes for the ACPI PM domain.

Well, as far as I'm concerned, this is not going to get any further.

> Especially if you continue giving me your guidance.
> 
> >
> >> The generic PM domain, is simple in this regards. There is only a
> >> minor adaptation for the ->runtime_suspend|resume() callbacks, which
> >> avoids validating dev_pm_qos constraints during system sleep. Nothing
> >> special is needed in ->suspend_late|noirq callbacks, etc.
> >>
> >> For most other simple bus types, like the platform bus, spi, i2c,
> >> amba, no particular adoptions is needed at all. Instead those just
> >> trust the drivers to do the right thing.
> >
> > They are the trivial ones.
> 
> Yes.
> 
> However, the platform bus is also very commonly used in kernel. I
> think that's an important thing to consider.

Well, so is ACPI, and PCI.

And the platform bus doesn't do any kind of PM handling by itself,
whereas the above do.  Therefore trying to look at the platform bus as
an example to follow by them is rather not useful IMO.

> >
> >> Before we had the direct_complete path, using the
> >> pm_runtime_force_suspend|resume() helpers, was the only good way for
> >> these kind of drivers, to in an optimized manner, deal with system
> >> sleep when runtime PM also was enabled for their devices. Now this
> >> method has become widely deployed, unfortunate whether you like it or
> >> not.
> >
> > So can you please remind me why the _force_ wrappers are needed?
> 
> See below.
> 
> >
> > In particular, why can't drivers arrange their callbacks the way I did that
> > in https://patchwork.kernel.org/patch/9928583/ ?
> 
> I was preparing a reply to that patch, but let me summarize that here instead.
> 
> Let me be clear, the patch is an improvement of the behavior of the
> driver and it addresses the issues you point out in the change log.
> Re-using the runtime PM callbacks for system sleep, is nice as it
> avoids open coding, which is of curse also one of the reason of using
> pm_runtime_force_suspend|resume().
> 
> Still there are a couple of things I am worried about in this patch.
> *)
> To be able to re-use the same callbacks for system sleep and runtime
> PM, some boilerplate code is added to the driver, as to cope with the
> different conditions inside the callbacks. That pattern would become
> repeated to many drivers dealing with similar issues.

I'm not worried about that as long as there are good examples and
documented best practices.

There aren't any right now, which is a problem, but that certainly is
fixable.

> **)
> The ->resume_early() callback powers on the device, in case it was
> runtime resumed when the ->suspend_late() callback was invoked. That
> is in many cases completely unnecessary, causing us to waste power and
> increase system resume time, for absolutely no reason. However, I
> understand the patch didn't try to address this, but to really fix
> this, there has to be an even closer collaboration between runtime PM
> and the system sleep callbacks.

I don't quite agree and here's why.

If a device was not runtime-suspended right before system suspend, then quite
likely it was in use then.  Therefore it is quite likely to be resumed
immediately after system resume anyway.

Now, if that's just one device, it probably doesn't matter, but if there are
more devices like that, they will be resumed after system suspend when they
are accessed and quite likely they will be accessed one-by-one rather than in
parallel with each other, so the latencies related to that will add up.  In
that case it is better to resume them upfront during system resume as they will
be resumed in parallel with each other then.  And that also is *way* simpler.

This means that the benefit from avoiding to resume devices during system
resume is not quite so obvious and the whole point above is highly
questionable.

> 
> So, to remind you why the pm_runtime_force_suspend|resume() helpers is
> preferred, that's because both of the above two things becomes taken
> care of.

And that is why there is this stuff about parents and usage counters, right?

I'm not liking it at all.

> 
> >
> >> Besides the slightly better optimizations you get when using
> >> pm_runtime_force_suspend|resume(), comparing to the direct_complete
> >> path - I think it's also worth to consider, how easy it becomes for
> >> drivers to deploy system sleep support. In many cases, only two lines
> >> of code is needed to add system sleep support in a driver.
> >
> > You are doing a wrong comparison here IMO.  You essentially are comparing two
> > bandaids with each other and arguing that one of them somehow is better.
> 
> I just wanted to compare against something...
> 
> >
> > What about doing something which is not a bandaid instead?
> 
> I don't have a problem working on something new, but I am not sure
> what that should be.
> 
> Unless you re-consider moving forward in some form, with the current
> suggested approach for the ACPI PM domain, can you give me some
> pointers on what you have in mind?

Yes.

Do what was indended from the start and make drivers re-use runtime
PM callbacks for ->suspend_late and ->resume_early.

First, that can be done.

Second, it is *conceptually* much more straightforward than things like
_force_suspend/resume().

Next, the drivers have full control on what they do in that case and
can be made work with any middle-layer core easily enough.

No layering violations, no insane callback chains.

> To remind you of my current view, the direct_complete path is useful
> for PM domains, like the ACPI PM domain as it impacts all its devices.
> Using pm_runtime_force_suspend|resume() offers the next steps to

I completely disagree at this point.

So to be clear, the invocation of moddle-layer callbacks instead of
*driver* callbacks in pm_runtime_force_suspend|resume() is a grave mistake.
They would have been almost possible to work with had they just invoke
driver callbacks.

OTOH I'm starting to think that direct_complete is only theoretically useful
and may not be actually set very often in practice, whereas it adds significant
complexity to the code, so I'm not sure about it any more.

> achieve a fully optimized behavior of a device during system sleep, as
> already been proven by now. It would be great to both options
> supported by the ACPI PM domain.

No.

> Another related thing that is causing lots of problems during system
> sleep of devices, but not related to optimizations, is to have the
> correct order of how to suspend/resume the devices. We have talked
> about this, but it's a separate problem and it's rather a deployment
> issue, than having to implements something entirely new (we have
> supplies/consumers links you invented for this).

Yes, that is a separate issue.

> >
> >> Now, some complex code always needs to be implemented somewhere. When
> >> using the runtime PM centric path, that code consist of the
> >> pm_runtime_force_suspend|resume() helpers itself - and some
> >> adaptations in buses/PM domains in cases when those needs special
> >> care.
> >>
> >> My point is, the runtime PM centric path, allows us to keep the
> >> complex part of the code at a few centralized places, instead of
> >> having it spread/duplicated into drivers.
> >>
> >> So yes, you could consider it insane, but to me and many others, it
> >> seems to work out quite well.
> >
> > Because it only has been used with trivial middle layer code so far
> > and I'm quite disappointed that you don't seem to see a problem here. :-/
> >
> > I mean something like
> >
> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late
> >
> > is far more straightforward than
> >
> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
> >         bus type / PM domain ->runtime_suspend => driver ->runtime_suspend
> >
> > with the bus type / PM domain having to figure out somehow at the
> > ->suspend_late time whether or not its ->runtume_suspend is going to be invoked
> > in the middle of it.
> >
> > Apart from this just being aesthetically disgusting to me, which admittedly is
> > a matter of personal opinion, it makes debugging new driver code harder (if it
> > happens to not work) and reviewing it almost impossible, because now you need
> > to take all of the tangling between callbacks into accont and sometimes not
> > just for one bus type / PM domain.
> 
> I am wondering that perhaps you may be overlooking some of the
> internals of runtime PM. Or maybe not? :-)
> 
> I mean, the hole thing is build upon that anyone can call runtime PM
> functions to runtime resume/suspend a device.

Well, right in general, except that _force_suspend/resume() invoke
*callbacks* and *not* runtime PM functions.

> Doing that, makes the
> hierarchy of the runtime PM callbacks being walked and invoked, of
> course properly managed by the runtime PM core.
> 
> My point is that, the runtime PM core still controls this behavior,
> even when the pm_runtime_force_suspend|resume() helpers are being
> invoked. The only difference is that it allows runtime PM for the
> device to be disabled, and still correctly invoked the callbacks. That
> is what it is all about.

So why is it even useful to call ->runtime_suspend from a middle layer
in pm_runtime_force_suspend(), for example?

> >
> >> Yeah, and the laying violation is undoubtedly the most controversial
> >> part of the runtime PM centric path - I agree to that! The
> >> direct_complete path don't have this, as you implemented it. :-)
> >>
> >> On the other hand, one could consider that these upper layers, in many
> >> cases anyway needs to play along with the behavior of the driver. So,
> >> I guess it depends on how one see it.
> >>
> >> >
> >> > That's why I'm afraid that we've reached a dead end here. :-(
> >>
> >> That's said news. Is was really hoping I could find a way to move this
> >> forward. You don't have any other ideas on how I can adjust the series
> >> to make you happy?
> >
> > So to be precise, patches [2-3/8] are basically fine by me.  Patch [4/8]
> > sort of works too, but I'd do the splitting slightly differently and I don't
> > see much value in it alone.
> >
> > The rest of the ACPI changes is mostly not acceptable to me, mostly because
> > of what is done to the PM domain's ->runtime_suspend/resume and
> > ->suspend_late/->resume_early callbacks.
> >
> > I guess the only way that could be made work for me would be by not using
> > _force_suspend/resume() at all, but that would defeat the point, right?
> 
> Yes, it would.
> 
> >
> > I don't like the flag too, but that might be worked out.
> 
> Yeah, I am open to any suggestion.
> 
> >
> > Also, when I looked at _force_suspend/resume() again, I got concerned.
> > There is stuff in there that shouldn't be necessary in a driver's
> > ->late_suspend/->early_resume and some things in there just made me
> > scratch my head.
> 
> Yes, there are some complexity in there, I will be happy to answer any
> specific question about it.

OK

Of course they require runtime PM to be enabled by drivers using them as
their callbacks, but I suppose that you realize that.

Why to disabe/renable runtime PM in there in the first place?  That should
have been done by the core when these functions are intended to be called.

Second, why to use RPM_GET_CALLBACK in there?

Next, how is the parent actually runtime-resumed by pm_runtime_force_resume()
which the comment in pm_runtime_force_suspend() talks about?

> 
> The main thing is, that it tries to conform to the regular rules set
> by the runtime PM core when runtime PM is enabled for the device - and
> then apply those to the device when runtime PM has been disabled for
> it.

Sorry, I'm not sure what this means really ...

> Again, thanks for being patient and reviewing!

Well, no problem.

Thanks,
Rafael

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Friday, September 1, 2017 12:42:35 PM CEST Ulf Hansson wrote:
> On 31 August 2017 at 02:17, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > Disclaimer: I'm falling asleep, so I probably shouldn't reply to email right
> > now, but tomorrow I may not be able to get to email at all, so I'll try anyway.
> >
> > On Wednesday, August 30, 2017 11:57:28 AM CEST Ulf Hansson wrote:
> >> On 29 August 2017 at 22:19, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> >> > On Tuesday, August 29, 2017 4:56:42 PM CEST Ulf Hansson wrote:
> >> >> The i2c designware platform driver, drivers/i2c/busses/i2c-designware-platdrv.c,
> >> >> isn't well optimized for system sleep.
> >> >>
> >> >> What makes this driver particularly interesting is because it's a cross-SoC
> >> >> driver, which sometimes means there is an ACPI PM domain attached to the i2c
> >> >> device and sometimes not. The driver is being used on both x86 and ARM.
> >> >>
> >> >> In principle, to optimize the system sleep support in i2c driver, this series
> >> >> enables the proven runtime PM centric path for the i2c driver. However, to do
> >> >> that the ACPI PM domain also have to collaborate and understand this behaviour.
> >> >> From earlier versions, Rafael has also pointed out that also the PM core needs
> >> >> to be involved.
> >> >
> >> > Earlier today I realized that drivers pointing their ->suspend_late and
> >> > ->resume_early callbacks, respectively, to pm_runtime_force_suspend() and
> >> > pm_runtime_force_resume(), are fundamentally incompatible with any bus type
> >> > doing nontrivial PM and with almost any nontrivial PM domains, for two reasons.
> >> >
> >> > First, it basically requires the bus type or PM domain to expect that its
> >> > ->runtime_suspend callback may or may not be indirectly invoked from its
> >> > own ->suspend_late callback, depending on the driver (and analogously
> >> > for ->runtime_resume and ->resume early), which is insane.
> >> >
> >> > Second, it is a layering violation, because it makes the driver effectively
> >> > override the upper layer's decisions about what code to run.
> >>
> >> You are right that for more complex bus types and PM domains, those
> >> needs to play along. So that is what I am trying to implement for the
> >> ACPI PM domain in this series.
> >
> > Well, "play along" is a bit of an understatement here.
> >
> > They would need to turn into horrible mess and that's not going to happen.
> 
> I absolutely agree, there must be no mess what so ever!
> 
> But, I don't want to give up yet, I still believe I can make this
> series into a nice couple of changes for the ACPI PM domain.

Well, as far as I'm concerned, this is not going to get any further.

> Especially if you continue giving me your guidance.
> 
> >
> >> The generic PM domain, is simple in this regards. There is only a
> >> minor adaptation for the ->runtime_suspend|resume() callbacks, which
> >> avoids validating dev_pm_qos constraints during system sleep. Nothing
> >> special is needed in ->suspend_late|noirq callbacks, etc.
> >>
> >> For most other simple bus types, like the platform bus, spi, i2c,
> >> amba, no particular adoptions is needed at all. Instead those just
> >> trust the drivers to do the right thing.
> >
> > They are the trivial ones.
> 
> Yes.
> 
> However, the platform bus is also very commonly used in kernel. I
> think that's an important thing to consider.

Well, so is ACPI, and PCI.

And the platform bus doesn't do any kind of PM handling by itself,
whereas the above do.  Therefore trying to look at the platform bus as
an example to follow by them is rather not useful IMO.

> >
> >> Before we had the direct_complete path, using the
> >> pm_runtime_force_suspend|resume() helpers, was the only good way for
> >> these kind of drivers, to in an optimized manner, deal with system
> >> sleep when runtime PM also was enabled for their devices. Now this
> >> method has become widely deployed, unfortunate whether you like it or
> >> not.
> >
> > So can you please remind me why the _force_ wrappers are needed?
> 
> See below.
> 
> >
> > In particular, why can't drivers arrange their callbacks the way I did that
> > in https://patchwork.kernel.org/patch/9928583/ ?
> 
> I was preparing a reply to that patch, but let me summarize that here instead.
> 
> Let me be clear, the patch is an improvement of the behavior of the
> driver and it addresses the issues you point out in the change log.
> Re-using the runtime PM callbacks for system sleep, is nice as it
> avoids open coding, which is of curse also one of the reason of using
> pm_runtime_force_suspend|resume().
> 
> Still there are a couple of things I am worried about in this patch.
> *)
> To be able to re-use the same callbacks for system sleep and runtime
> PM, some boilerplate code is added to the driver, as to cope with the
> different conditions inside the callbacks. That pattern would become
> repeated to many drivers dealing with similar issues.

I'm not worried about that as long as there are good examples and
documented best practices.

There aren't any right now, which is a problem, but that certainly is
fixable.

> **)
> The ->resume_early() callback powers on the device, in case it was
> runtime resumed when the ->suspend_late() callback was invoked. That
> is in many cases completely unnecessary, causing us to waste power and
> increase system resume time, for absolutely no reason. However, I
> understand the patch didn't try to address this, but to really fix
> this, there has to be an even closer collaboration between runtime PM
> and the system sleep callbacks.

I don't quite agree and here's why.

If a device was not runtime-suspended right before system suspend, then quite
likely it was in use then.  Therefore it is quite likely to be resumed
immediately after system resume anyway.

Now, if that's just one device, it probably doesn't matter, but if there are
more devices like that, they will be resumed after system suspend when they
are accessed and quite likely they will be accessed one-by-one rather than in
parallel with each other, so the latencies related to that will add up.  In
that case it is better to resume them upfront during system resume as they will
be resumed in parallel with each other then.  And that also is *way* simpler.

This means that the benefit from avoiding to resume devices during system
resume is not quite so obvious and the whole point above is highly
questionable.

> 
> So, to remind you why the pm_runtime_force_suspend|resume() helpers is
> preferred, that's because both of the above two things becomes taken
> care of.

And that is why there is this stuff about parents and usage counters, right?

I'm not liking it at all.

> 
> >
> >> Besides the slightly better optimizations you get when using
> >> pm_runtime_force_suspend|resume(), comparing to the direct_complete
> >> path - I think it's also worth to consider, how easy it becomes for
> >> drivers to deploy system sleep support. In many cases, only two lines
> >> of code is needed to add system sleep support in a driver.
> >
> > You are doing a wrong comparison here IMO.  You essentially are comparing two
> > bandaids with each other and arguing that one of them somehow is better.
> 
> I just wanted to compare against something...
> 
> >
> > What about doing something which is not a bandaid instead?
> 
> I don't have a problem working on something new, but I am not sure
> what that should be.
> 
> Unless you re-consider moving forward in some form, with the current
> suggested approach for the ACPI PM domain, can you give me some
> pointers on what you have in mind?

Yes.

Do what was indended from the start and make drivers re-use runtime
PM callbacks for ->suspend_late and ->resume_early.

First, that can be done.

Second, it is *conceptually* much more straightforward than things like
_force_suspend/resume().

Next, the drivers have full control on what they do in that case and
can be made work with any middle-layer core easily enough.

No layering violations, no insane callback chains.

> To remind you of my current view, the direct_complete path is useful
> for PM domains, like the ACPI PM domain as it impacts all its devices.
> Using pm_runtime_force_suspend|resume() offers the next steps to

I completely disagree at this point.

So to be clear, the invocation of moddle-layer callbacks instead of
*driver* callbacks in pm_runtime_force_suspend|resume() is a grave mistake.
They would have been almost possible to work with had they just invoke
driver callbacks.

OTOH I'm starting to think that direct_complete is only theoretically useful
and may not be actually set very often in practice, whereas it adds significant
complexity to the code, so I'm not sure about it any more.

> achieve a fully optimized behavior of a device during system sleep, as
> already been proven by now. It would be great to both options
> supported by the ACPI PM domain.

No.

> Another related thing that is causing lots of problems during system
> sleep of devices, but not related to optimizations, is to have the
> correct order of how to suspend/resume the devices. We have talked
> about this, but it's a separate problem and it's rather a deployment
> issue, than having to implements something entirely new (we have
> supplies/consumers links you invented for this).

Yes, that is a separate issue.

> >
> >> Now, some complex code always needs to be implemented somewhere. When
> >> using the runtime PM centric path, that code consist of the
> >> pm_runtime_force_suspend|resume() helpers itself - and some
> >> adaptations in buses/PM domains in cases when those needs special
> >> care.
> >>
> >> My point is, the runtime PM centric path, allows us to keep the
> >> complex part of the code at a few centralized places, instead of
> >> having it spread/duplicated into drivers.
> >>
> >> So yes, you could consider it insane, but to me and many others, it
> >> seems to work out quite well.
> >
> > Because it only has been used with trivial middle layer code so far
> > and I'm quite disappointed that you don't seem to see a problem here. :-/
> >
> > I mean something like
> >
> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late
> >
> > is far more straightforward than
> >
> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
> >         bus type / PM domain ->runtime_suspend => driver ->runtime_suspend
> >
> > with the bus type / PM domain having to figure out somehow at the
> > ->suspend_late time whether or not its ->runtume_suspend is going to be invoked
> > in the middle of it.
> >
> > Apart from this just being aesthetically disgusting to me, which admittedly is
> > a matter of personal opinion, it makes debugging new driver code harder (if it
> > happens to not work) and reviewing it almost impossible, because now you need
> > to take all of the tangling between callbacks into accont and sometimes not
> > just for one bus type / PM domain.
> 
> I am wondering that perhaps you may be overlooking some of the
> internals of runtime PM. Or maybe not? :-)
> 
> I mean, the hole thing is build upon that anyone can call runtime PM
> functions to runtime resume/suspend a device.

Well, right in general, except that _force_suspend/resume() invoke
*callbacks* and *not* runtime PM functions.

> Doing that, makes the
> hierarchy of the runtime PM callbacks being walked and invoked, of
> course properly managed by the runtime PM core.
> 
> My point is that, the runtime PM core still controls this behavior,
> even when the pm_runtime_force_suspend|resume() helpers are being
> invoked. The only difference is that it allows runtime PM for the
> device to be disabled, and still correctly invoked the callbacks. That
> is what it is all about.

So why is it even useful to call ->runtime_suspend from a middle layer
in pm_runtime_force_suspend(), for example?

> >
> >> Yeah, and the laying violation is undoubtedly the most controversial
> >> part of the runtime PM centric path - I agree to that! The
> >> direct_complete path don't have this, as you implemented it. :-)
> >>
> >> On the other hand, one could consider that these upper layers, in many
> >> cases anyway needs to play along with the behavior of the driver. So,
> >> I guess it depends on how one see it.
> >>
> >> >
> >> > That's why I'm afraid that we've reached a dead end here. :-(
> >>
> >> That's said news. Is was really hoping I could find a way to move this
> >> forward. You don't have any other ideas on how I can adjust the series
> >> to make you happy?
> >
> > So to be precise, patches [2-3/8] are basically fine by me.  Patch [4/8]
> > sort of works too, but I'd do the splitting slightly differently and I don't
> > see much value in it alone.
> >
> > The rest of the ACPI changes is mostly not acceptable to me, mostly because
> > of what is done to the PM domain's ->runtime_suspend/resume and
> > ->suspend_late/->resume_early callbacks.
> >
> > I guess the only way that could be made work for me would be by not using
> > _force_suspend/resume() at all, but that would defeat the point, right?
> 
> Yes, it would.
> 
> >
> > I don't like the flag too, but that might be worked out.
> 
> Yeah, I am open to any suggestion.
> 
> >
> > Also, when I looked at _force_suspend/resume() again, I got concerned.
> > There is stuff in there that shouldn't be necessary in a driver's
> > ->late_suspend/->early_resume and some things in there just made me
> > scratch my head.
> 
> Yes, there are some complexity in there, I will be happy to answer any
> specific question about it.

OK

Of course they require runtime PM to be enabled by drivers using them as
their callbacks, but I suppose that you realize that.

Why to disabe/renable runtime PM in there in the first place?  That should
have been done by the core when these functions are intended to be called.

Second, why to use RPM_GET_CALLBACK in there?

Next, how is the parent actually runtime-resumed by pm_runtime_force_resume()
which the comment in pm_runtime_force_suspend() talks about?

> 
> The main thing is, that it tries to conform to the regular rules set
> by the runtime PM core when runtime PM is enabled for the device - and
> then apply those to the device when runtime PM has been disabled for
> it.

Sorry, I'm not sure what this means really ...

> Again, thanks for being patient and reviewing!

Well, no problem.

Thanks,
Rafael

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Lukas Wunner]

On Mon, Sep 04, 2017 at 02:17:21AM +0200, Rafael J. Wysocki wrote:
> OTOH I'm starting to think that direct_complete is only theoretically
> useful and may not be actually set very often in practice, whereas it
> adds significant complexity to the code, so I'm not sure about it any
> more.

That makes me come out of the woodwork as a direct_complete fan:

Runtime resuming a discrete GPU on a modern dual GPU laptop takes about
1.5 sec, runtime resuming Thunderbolt controllers more than 2 sec.
A discrete GPU easily consumes 10W, a Thunderbolt controller 2W.

So not having direct_complete would noticeably delay system suspend and
resume as well as reduce battery life.

Lukas

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Lukas Wunner]

On Mon, Sep 04, 2017 at 02:17:21AM +0200, Rafael J. Wysocki wrote:
> OTOH I'm starting to think that direct_complete is only theoretically
> useful and may not be actually set very often in practice, whereas it
> adds significant complexity to the code, so I'm not sure about it any
> more.

That makes me come out of the woodwork as a direct_complete fan:

Runtime resuming a discrete GPU on a modern dual GPU laptop takes about
1.5 sec, runtime resuming Thunderbolt controllers more than 2 sec.
A discrete GPU easily consumes 10W, a Thunderbolt controller 2W.

So not having direct_complete would noticeably delay system suspend and
resume as well as reduce battery life.

Lukas

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Monday, September 4, 2017 7:46:37 AM CEST Lukas Wunner wrote:
> On Mon, Sep 04, 2017 at 02:17:21AM +0200, Rafael J. Wysocki wrote:
> > OTOH I'm starting to think that direct_complete is only theoretically
> > useful and may not be actually set very often in practice, whereas it
> > adds significant complexity to the code, so I'm not sure about it any
> > more.
> 
> That makes me come out of the woodwork as a direct_complete fan:
> 
> Runtime resuming a discrete GPU on a modern dual GPU laptop takes about
> 1.5 sec, runtime resuming Thunderbolt controllers more than 2 sec.
> A discrete GPU easily consumes 10W, a Thunderbolt controller 2W.
> 
> So not having direct_complete would noticeably delay system suspend and
> resume as well as reduce battery life.

Well, that's a good reason for having it. :-)

Thanks,
Rafael

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Monday, September 4, 2017 7:46:37 AM CEST Lukas Wunner wrote:
> On Mon, Sep 04, 2017 at 02:17:21AM +0200, Rafael J. Wysocki wrote:
> > OTOH I'm starting to think that direct_complete is only theoretically
> > useful and may not be actually set very often in practice, whereas it
> > adds significant complexity to the code, so I'm not sure about it any
> > more.
> 
> That makes me come out of the woodwork as a direct_complete fan:
> 
> Runtime resuming a discrete GPU on a modern dual GPU laptop takes about
> 1.5 sec, runtime resuming Thunderbolt controllers more than 2 sec.
> A discrete GPU easily consumes 10W, a Thunderbolt controller 2W.
> 
> So not having direct_complete would noticeably delay system suspend and
> resume as well as reduce battery life.

Well, that's a good reason for having it. :-)

Thanks,
Rafael

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

[...]

>> > So can you please remind me why the _force_ wrappers are needed?
>>
>> See below.
>>
>> >
>> > In particular, why can't drivers arrange their callbacks the way I did that
>> > in https://patchwork.kernel.org/patch/9928583/ ?
>>
>> I was preparing a reply to that patch, but let me summarize that here instead.
>>
>> Let me be clear, the patch is an improvement of the behavior of the
>> driver and it addresses the issues you point out in the change log.
>> Re-using the runtime PM callbacks for system sleep, is nice as it
>> avoids open coding, which is of curse also one of the reason of using
>> pm_runtime_force_suspend|resume().
>>
>> Still there are a couple of things I am worried about in this patch.
>> *)
>> To be able to re-use the same callbacks for system sleep and runtime
>> PM, some boilerplate code is added to the driver, as to cope with the
>> different conditions inside the callbacks. That pattern would become
>> repeated to many drivers dealing with similar issues.
>
> I'm not worried about that as long as there are good examples and
> documented best practices.
>
> There aren't any right now, which is a problem, but that certainly is
> fixable.
>
>> **)
>> The ->resume_early() callback powers on the device, in case it was
>> runtime resumed when the ->suspend_late() callback was invoked. That
>> is in many cases completely unnecessary, causing us to waste power and
>> increase system resume time, for absolutely no reason. However, I
>> understand the patch didn't try to address this, but to really fix
>> this, there has to be an even closer collaboration between runtime PM
>> and the system sleep callbacks.
>
> I don't quite agree and here's why.
>
> If a device was not runtime-suspended right before system suspend, then quite
> likely it was in use then.  Therefore it is quite likely to be resumed
> immediately after system resume anyway.

Unfortunate, to always make that assumption, leads to a non-optimized
behavior of system sleep. I think we can do better than that!

Let me give you a concrete example, where the above assumption would
lead to an non-optimized behavior.

To put an MMC card into low power state during system suspend
(covering eMMC, SD, SDIO) the mmc core needs to send a couple of
commands over the MMC interface to the card, as to conform with the
(e)MMC/SD/SDIO spec. To do this, the mmc driver for the mmc controller
must runtime resume its device, as to be able to send the commands
over the interface.

Now, when the system resumes, there is absolutely no reason to runtime
resume the device for the MMC controller, just because it was runtime
resumed during system suspend. Instead that is better to be postponed
to when the MMC card is really needed and thus via runtime PM instead.

This scenario shouldn't be specific to only MMC controllers/cards, but
should apply to any external devices/controllers that needs some
special treatment to be put into low power state during system
suspend. Particularly also when those external devices may be left in
that low power state until those are really needed. A couple of cases
I know of pops up in my head, WiFi chips, persistent storage devices,
etc. There should be plenty.

Another common case, is when a subsystem core layer flushes a request
queue during system suspend, which may cause a controller device to be
runtime resumed. Making the assumption that, because flushing the
queue was done during system suspend, we must also power up the
controller during system resume, again would lead to a non-optimized
behavior.

>
> Now, if that's just one device, it probably doesn't matter, but if there are
> more devices like that, they will be resumed after system suspend when they
> are accessed and quite likely they will be accessed one-by-one rather than in
> parallel with each other, so the latencies related to that will add up.  In
> that case it is better to resume them upfront during system resume as they will
> be resumed in parallel with each other then.  And that also is *way* simpler.
>
> This means that the benefit from avoiding to resume devices during system
> resume is not quite so obvious and the whole point above is highly
> questionable.

I hope my reasoning above explains why I think it shouldn't be
considered as questionable.

If you like, I can also provide some real data/logs - showing you
what's happening.

>
>>
>> So, to remind you why the pm_runtime_force_suspend|resume() helpers is
>> preferred, that's because both of the above two things becomes taken
>> care of.
>
> And that is why there is this stuff about parents and usage counters, right?

Correct. Perhaps this commit tells you a little more.

commit 1d9174fbc55ec99ccbfcafa3de2528ef78a849aa
Author: Ulf Hansson <ulf.hansson@linaro.org>
Date:   Thu Oct 13 16:58:54 2016 +0200

    PM / Runtime: Defer resuming of the device in pm_runtime_force_resume()

[...]

>> >
>> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late
>> >
>> > is far more straightforward than
>> >
>> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
>> >         bus type / PM domain ->runtime_suspend => driver ->runtime_suspend
>> >
>> > with the bus type / PM domain having to figure out somehow at the
>> > ->suspend_late time whether or not its ->runtume_suspend is going to be invoked
>> > in the middle of it.
>> >
>> > Apart from this just being aesthetically disgusting to me, which admittedly is
>> > a matter of personal opinion, it makes debugging new driver code harder (if it
>> > happens to not work) and reviewing it almost impossible, because now you need
>> > to take all of the tangling between callbacks into accont and sometimes not
>> > just for one bus type / PM domain.
>>
>> I am wondering that perhaps you may be overlooking some of the
>> internals of runtime PM. Or maybe not? :-)
>>
>> I mean, the hole thing is build upon that anyone can call runtime PM
>> functions to runtime resume/suspend a device.
>
> Well, right in general, except that _force_suspend/resume() invoke
> *callbacks* and *not* runtime PM functions.

I am considering pm_runtime_force_suspend|resume() being a part of the
runtime PM API, except that those may be called only during system
sleep.

Comparing a call to pm_runtime_resume(); this may trigger rpm_resume()
to invoke the callbacks. To me, the difference is that the conditions
looked at in rpm_resume(), when runtime PM is enabled, becomes
different for system sleep when runtime PM is disabled - and that is
taken care of in pm_runtime_force_suspend|resume().

>
>> Doing that, makes the
>> hierarchy of the runtime PM callbacks being walked and invoked, of
>> course properly managed by the runtime PM core.
>>
>> My point is that, the runtime PM core still controls this behavior,
>> even when the pm_runtime_force_suspend|resume() helpers are being
>> invoked. The only difference is that it allows runtime PM for the
>> device to be disabled, and still correctly invoked the callbacks. That
>> is what it is all about.
>
> So why is it even useful to call ->runtime_suspend from a middle layer
> in pm_runtime_force_suspend(), for example?

Perhaps I don't understand the question correctly.

Anyway, the answer I think of, is probably because of the same reason
to why the runtime PM core invokes it, when it runs rpm_suspend() for
a device. My point is, we want the similar behavior.

[...]

>> >
>> > Also, when I looked at _force_suspend/resume() again, I got concerned.
>> > There is stuff in there that shouldn't be necessary in a driver's
>> > ->late_suspend/->early_resume and some things in there just made me
>> > scratch my head.
>>
>> Yes, there are some complexity in there, I will be happy to answer any
>> specific question about it.
>
> OK
>
> Of course they require runtime PM to be enabled by drivers using them as
> their callbacks, but I suppose that you realize that.
>
> Why to disabe/renable runtime PM in there in the first place?  That should
> have been done by the core when these functions are intended to be called.

The reason is because we didn't want to re-strict them to be used only
in ->suspend_late() and ->resume_early(), but also for ->suspend() and
->resume(), which is when runtime PM still is enabled.

>
> Second, why to use RPM_GET_CALLBACK in there?

To follow the same rules/hierarchy, as being done in rpm_suspend|resume().

>
> Next, how is the parent actually runtime-resumed by pm_runtime_force_resume()
> which the comment in pm_runtime_force_suspend() talks about?

I think the relevant use case here is when a parent and a child, both
have subsystems/drivers using pm_runtime_force_suspend|resume(). If
that isn't the case, we expect that the parent is always resumed
during system resume. It's a bit fragile approach, so we perhaps we
should deal with it, even if the hole thing is used as opt-in.

Anyway, let's focus on the case which I think is most relevant to your question:

A couple of conditions to start with.
*) The PM core system suspends a child prior a parent, which leads to
pm_runtime_force_suspend() being called for the child first.
**) The PM core system resumes a parents before a child, thus
pm_runtime_force_resume() is called for the parent first.

In case a child don't need to be resumed when
pm_runtime_force_resume() is called for it, likely doesn't its parent.
However, to control that, in system suspend the
pm_runtime_force_suspend() increases the usage counter for the parent,
as to indicate if it needs to be resumed when
pm_runtime_force_resume() is called for it.

Finally, when the child becomes resumed in pm_runtime_force_resume(),
pm_runtime_set_active() is called for it. This verifies that the
parent also has been resumed properly.

[...]

Kind regards
Uffe

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

[...]

>> > So can you please remind me why the _force_ wrappers are needed?
>>
>> See below.
>>
>> >
>> > In particular, why can't drivers arrange their callbacks the way I did that
>> > in https://patchwork.kernel.org/patch/9928583/ ?
>>
>> I was preparing a reply to that patch, but let me summarize that here instead.
>>
>> Let me be clear, the patch is an improvement of the behavior of the
>> driver and it addresses the issues you point out in the change log.
>> Re-using the runtime PM callbacks for system sleep, is nice as it
>> avoids open coding, which is of curse also one of the reason of using
>> pm_runtime_force_suspend|resume().
>>
>> Still there are a couple of things I am worried about in this patch.
>> *)
>> To be able to re-use the same callbacks for system sleep and runtime
>> PM, some boilerplate code is added to the driver, as to cope with the
>> different conditions inside the callbacks. That pattern would become
>> repeated to many drivers dealing with similar issues.
>
> I'm not worried about that as long as there are good examples and
> documented best practices.
>
> There aren't any right now, which is a problem, but that certainly is
> fixable.
>
>> **)
>> The ->resume_early() callback powers on the device, in case it was
>> runtime resumed when the ->suspend_late() callback was invoked. That
>> is in many cases completely unnecessary, causing us to waste power and
>> increase system resume time, for absolutely no reason. However, I
>> understand the patch didn't try to address this, but to really fix
>> this, there has to be an even closer collaboration between runtime PM
>> and the system sleep callbacks.
>
> I don't quite agree and here's why.
>
> If a device was not runtime-suspended right before system suspend, then quite
> likely it was in use then.  Therefore it is quite likely to be resumed
> immediately after system resume anyway.

Unfortunate, to always make that assumption, leads to a non-optimized
behavior of system sleep. I think we can do better than that!

Let me give you a concrete example, where the above assumption would
lead to an non-optimized behavior.

To put an MMC card into low power state during system suspend
(covering eMMC, SD, SDIO) the mmc core needs to send a couple of
commands over the MMC interface to the card, as to conform with the
(e)MMC/SD/SDIO spec. To do this, the mmc driver for the mmc controller
must runtime resume its device, as to be able to send the commands
over the interface.

Now, when the system resumes, there is absolutely no reason to runtime
resume the device for the MMC controller, just because it was runtime
resumed during system suspend. Instead that is better to be postponed
to when the MMC card is really needed and thus via runtime PM instead.

This scenario shouldn't be specific to only MMC controllers/cards, but
should apply to any external devices/controllers that needs some
special treatment to be put into low power state during system
suspend. Particularly also when those external devices may be left in
that low power state until those are really needed. A couple of cases
I know of pops up in my head, WiFi chips, persistent storage devices,
etc. There should be plenty.

Another common case, is when a subsystem core layer flushes a request
queue during system suspend, which may cause a controller device to be
runtime resumed. Making the assumption that, because flushing the
queue was done during system suspend, we must also power up the
controller during system resume, again would lead to a non-optimized
behavior.

>
> Now, if that's just one device, it probably doesn't matter, but if there are
> more devices like that, they will be resumed after system suspend when they
> are accessed and quite likely they will be accessed one-by-one rather than in
> parallel with each other, so the latencies related to that will add up.  In
> that case it is better to resume them upfront during system resume as they will
> be resumed in parallel with each other then.  And that also is *way* simpler.
>
> This means that the benefit from avoiding to resume devices during system
> resume is not quite so obvious and the whole point above is highly
> questionable.

I hope my reasoning above explains why I think it shouldn't be
considered as questionable.

If you like, I can also provide some real data/logs - showing you
what's happening.

>
>>
>> So, to remind you why the pm_runtime_force_suspend|resume() helpers is
>> preferred, that's because both of the above two things becomes taken
>> care of.
>
> And that is why there is this stuff about parents and usage counters, right?

Correct. Perhaps this commit tells you a little more.

commit 1d9174fbc55ec99ccbfcafa3de2528ef78a849aa
Author: Ulf Hansson <ulf.hansson@linaro.org>
Date:   Thu Oct 13 16:58:54 2016 +0200

    PM / Runtime: Defer resuming of the device in pm_runtime_force_resume()

[...]

>> >
>> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late
>> >
>> > is far more straightforward than
>> >
>> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
>> >         bus type / PM domain ->runtime_suspend => driver ->runtime_suspend
>> >
>> > with the bus type / PM domain having to figure out somehow at the
>> > ->suspend_late time whether or not its ->runtume_suspend is going to be invoked
>> > in the middle of it.
>> >
>> > Apart from this just being aesthetically disgusting to me, which admittedly is
>> > a matter of personal opinion, it makes debugging new driver code harder (if it
>> > happens to not work) and reviewing it almost impossible, because now you need
>> > to take all of the tangling between callbacks into accont and sometimes not
>> > just for one bus type / PM domain.
>>
>> I am wondering that perhaps you may be overlooking some of the
>> internals of runtime PM. Or maybe not? :-)
>>
>> I mean, the hole thing is build upon that anyone can call runtime PM
>> functions to runtime resume/suspend a device.
>
> Well, right in general, except that _force_suspend/resume() invoke
> *callbacks* and *not* runtime PM functions.

I am considering pm_runtime_force_suspend|resume() being a part of the
runtime PM API, except that those may be called only during system
sleep.

Comparing a call to pm_runtime_resume(); this may trigger rpm_resume()
to invoke the callbacks. To me, the difference is that the conditions
looked at in rpm_resume(), when runtime PM is enabled, becomes
different for system sleep when runtime PM is disabled - and that is
taken care of in pm_runtime_force_suspend|resume().

>
>> Doing that, makes the
>> hierarchy of the runtime PM callbacks being walked and invoked, of
>> course properly managed by the runtime PM core.
>>
>> My point is that, the runtime PM core still controls this behavior,
>> even when the pm_runtime_force_suspend|resume() helpers are being
>> invoked. The only difference is that it allows runtime PM for the
>> device to be disabled, and still correctly invoked the callbacks. That
>> is what it is all about.
>
> So why is it even useful to call ->runtime_suspend from a middle layer
> in pm_runtime_force_suspend(), for example?

Perhaps I don't understand the question correctly.

Anyway, the answer I think of, is probably because of the same reason
to why the runtime PM core invokes it, when it runs rpm_suspend() for
a device. My point is, we want the similar behavior.

[...]

>> >
>> > Also, when I looked at _force_suspend/resume() again, I got concerned.
>> > There is stuff in there that shouldn't be necessary in a driver's
>> > ->late_suspend/->early_resume and some things in there just made me
>> > scratch my head.
>>
>> Yes, there are some complexity in there, I will be happy to answer any
>> specific question about it.
>
> OK
>
> Of course they require runtime PM to be enabled by drivers using them as
> their callbacks, but I suppose that you realize that.
>
> Why to disabe/renable runtime PM in there in the first place?  That should
> have been done by the core when these functions are intended to be called.

The reason is because we didn't want to re-strict them to be used only
in ->suspend_late() and ->resume_early(), but also for ->suspend() and
->resume(), which is when runtime PM still is enabled.

>
> Second, why to use RPM_GET_CALLBACK in there?

To follow the same rules/hierarchy, as being done in rpm_suspend|resume().

>
> Next, how is the parent actually runtime-resumed by pm_runtime_force_resume()
> which the comment in pm_runtime_force_suspend() talks about?

I think the relevant use case here is when a parent and a child, both
have subsystems/drivers using pm_runtime_force_suspend|resume(). If
that isn't the case, we expect that the parent is always resumed
during system resume. It's a bit fragile approach, so we perhaps we
should deal with it, even if the hole thing is used as opt-in.

Anyway, let's focus on the case which I think is most relevant to your question:

A couple of conditions to start with.
*) The PM core system suspends a child prior a parent, which leads to
pm_runtime_force_suspend() being called for the child first.
**) The PM core system resumes a parents before a child, thus
pm_runtime_force_resume() is called for the parent first.

In case a child don't need to be resumed when
pm_runtime_force_resume() is called for it, likely doesn't its parent.
However, to control that, in system suspend the
pm_runtime_force_suspend() increases the usage counter for the parent,
as to indicate if it needs to be resumed when
pm_runtime_force_resume() is called for it.

Finally, when the child becomes resumed in pm_runtime_force_resume(),
pm_runtime_set_active() is called for it. This verifies that the
parent also has been resumed properly.

[...]

Kind regards
Uffe

Re: [PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Ulf Hansson]

On 2 September 2017 at 17:38, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Friday, September 1, 2017 10:27:05 AM CEST Ulf Hansson wrote:
>> On 29 August 2017 at 17:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> > On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
>> >> This change enables the ACPI PM domain to cope with drivers that deploys
>> >> the runtime PM centric path for system sleep.
>> >
>> > [cut]
>> >
>> >> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
>> >>   * @dev: Device to handle.
>> >>   *
>> >>   * Follow PCI and resume devices suspended at run time before running their
>> >> - * system suspend callbacks.
>> >> + * system suspend callbacks. However, try to avoid it in case the runtime PM
>> >> + * centric path is used for the device and then trust the driver to do the
>> >> + * right thing.
>> >>   */
>> >>  int acpi_subsys_suspend(struct device *dev)
>> >>  {
>> >> -     pm_runtime_resume(dev);
>> >> +     struct acpi_device *adev = ACPI_COMPANION(dev);
>> >> +
>> >> +     if (!adev)
>> >> +             return 0;
>> >> +
>> >> +     if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
>> >> +             pm_runtime_resume(dev);
>> >> +
>> >>       return pm_generic_suspend(dev);
>> >>  }
>> >>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
>> >
>> > Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
>> > and that's why I added the update_state thing.
>> >
>> > Moreover, the is_rpm_sleep flag here has to mean not only that
>> > direct_complete should not be used with the device, but also that its driver
>> > is fine with not resuming it.
>>
>> Let me try to explain this better. I realize the changelog is
>> misleading around this particular section! Huh, apologize for that!
>>
>> First, patch1 makes the PM core treat the is_rpm_sleep flag as the
>> direct_complete isn't allowed for the device.
>>
>> For that reason, when the is_rpm_sleep is set, there is no point
>> calling acpi_dev_needs_resume() from acpi_subsys_prepare(), but
>> instead that can be deferred to acpi_subsys_suspend() - because it
>> doesn't matter if acpi_subsys_prepare() returns 0 or 1, in either case
>> the acpi_subsys_suspend() will be called. That's really what goes on
>> here.
>>
>> The end result is the same. If the acpi_dev_needs_resume() thinks that
>> the device needs to be runtime resumed, pm_runtime_resume() is called
>> for the device in acpi_subsys_suspend().
>>
>> So, this has nothing to do with whether the driver "is fine with not
>> resuming it" thing.
>
> No, sorry.
>
> If is_rpm_sleep was not set, the ACPI PM domain would resume the device in
> acpi_subsys_suspend() regardless of the acpi_dev_needs_resume() return value.

Yes, I believe I forgot about one scenario, when the direct_complete
path has been abandoned by the PM core, because a child device was
suspend before and it couldn't run the direct_complete path for it?

Just to be sure, that's the case you also had in mind?

> That's what's there in the patch.  So clearly, setting is_rpm_sleep means
> "this device does not need to be resumed in acpi_subsys_suspend() unless
> acpi_dev_needs_resume() returns true".  Which clearly means that the driver
> *is* fine with not resuming it, because if is_rpm_sleep is set, the device
> in fact may not be resumed and then the driver will need to cope with that.

Yes, I understand your concern, because we may break the default
behavior of the ACPI PM domain.

So, *if* there will be a next version, I will make sure to be better
safe than sorry, and add one flag per use case.

>
> And note that this meaning of is_rpm_sleep is different from what it is
> expected to mean to the core.
>
>> >
>> > IMO it is not a good idea to use one flag for these two different things at the
>> > same time at all.
>>
>> Yeah, I guess my upper comment addresses your immediate concern here?
>
> No, they don't.
>
>> However, there is one other thing the is_rpm_flag means. That is that
>> the driver has informed the ACPI PM domain, to trust the driver to
>> deal with system sleep, via re-using the runtime PM callbacks.
>> So the flag does still have two meanings, but that we can change - of course.
>
> I guess that you are referring to the use of dev_pm_is_rpm_sleep() in
> acpi_subsys_suspend_late()?  That's the third thing this flag means ...

Yes.

Kind regards
Uffe

[PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Ulf Hansson]

On 2 September 2017 at 17:38, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Friday, September 1, 2017 10:27:05 AM CEST Ulf Hansson wrote:
>> On 29 August 2017 at 17:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> > On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
>> >> This change enables the ACPI PM domain to cope with drivers that deploys
>> >> the runtime PM centric path for system sleep.
>> >
>> > [cut]
>> >
>> >> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
>> >>   * @dev: Device to handle.
>> >>   *
>> >>   * Follow PCI and resume devices suspended at run time before running their
>> >> - * system suspend callbacks.
>> >> + * system suspend callbacks. However, try to avoid it in case the runtime PM
>> >> + * centric path is used for the device and then trust the driver to do the
>> >> + * right thing.
>> >>   */
>> >>  int acpi_subsys_suspend(struct device *dev)
>> >>  {
>> >> -     pm_runtime_resume(dev);
>> >> +     struct acpi_device *adev = ACPI_COMPANION(dev);
>> >> +
>> >> +     if (!adev)
>> >> +             return 0;
>> >> +
>> >> +     if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
>> >> +             pm_runtime_resume(dev);
>> >> +
>> >>       return pm_generic_suspend(dev);
>> >>  }
>> >>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
>> >
>> > Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
>> > and that's why I added the update_state thing.
>> >
>> > Moreover, the is_rpm_sleep flag here has to mean not only that
>> > direct_complete should not be used with the device, but also that its driver
>> > is fine with not resuming it.
>>
>> Let me try to explain this better. I realize the changelog is
>> misleading around this particular section! Huh, apologize for that!
>>
>> First, patch1 makes the PM core treat the is_rpm_sleep flag as the
>> direct_complete isn't allowed for the device.
>>
>> For that reason, when the is_rpm_sleep is set, there is no point
>> calling acpi_dev_needs_resume() from acpi_subsys_prepare(), but
>> instead that can be deferred to acpi_subsys_suspend() - because it
>> doesn't matter if acpi_subsys_prepare() returns 0 or 1, in either case
>> the acpi_subsys_suspend() will be called. That's really what goes on
>> here.
>>
>> The end result is the same. If the acpi_dev_needs_resume() thinks that
>> the device needs to be runtime resumed, pm_runtime_resume() is called
>> for the device in acpi_subsys_suspend().
>>
>> So, this has nothing to do with whether the driver "is fine with not
>> resuming it" thing.
>
> No, sorry.
>
> If is_rpm_sleep was not set, the ACPI PM domain would resume the device in
> acpi_subsys_suspend() regardless of the acpi_dev_needs_resume() return value.

Yes, I believe I forgot about one scenario, when the direct_complete
path has been abandoned by the PM core, because a child device was
suspend before and it couldn't run the direct_complete path for it?

Just to be sure, that's the case you also had in mind?

> That's what's there in the patch.  So clearly, setting is_rpm_sleep means
> "this device does not need to be resumed in acpi_subsys_suspend() unless
> acpi_dev_needs_resume() returns true".  Which clearly means that the driver
> *is* fine with not resuming it, because if is_rpm_sleep is set, the device
> in fact may not be resumed and then the driver will need to cope with that.

Yes, I understand your concern, because we may break the default
behavior of the ACPI PM domain.

So, *if* there will be a next version, I will make sure to be better
safe than sorry, and add one flag per use case.

>
> And note that this meaning of is_rpm_sleep is different from what it is
> expected to mean to the core.
>
>> >
>> > IMO it is not a good idea to use one flag for these two different things at the
>> > same time at all.
>>
>> Yeah, I guess my upper comment addresses your immediate concern here?
>
> No, they don't.
>
>> However, there is one other thing the is_rpm_flag means. That is that
>> the driver has informed the ACPI PM domain, to trust the driver to
>> deal with system sleep, via re-using the runtime PM callbacks.
>> So the flag does still have two meanings, but that we can change - of course.
>
> I guess that you are referring to the use of dev_pm_is_rpm_sleep() in
> acpi_subsys_suspend_late()?  That's the third thing this flag means ...

Yes.

Kind regards
Uffe

Re: [PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Rafael J. Wysocki]

On Monday, September 4, 2017 3:21:15 PM CEST Ulf Hansson wrote:
> On 2 September 2017 at 17:38, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Friday, September 1, 2017 10:27:05 AM CEST Ulf Hansson wrote:
> >> On 29 August 2017 at 17:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> >> > On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
> >> >> This change enables the ACPI PM domain to cope with drivers that deploys
> >> >> the runtime PM centric path for system sleep.
> >> >
> >> > [cut]
> >> >
> >> >> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
> >> >>   * @dev: Device to handle.
> >> >>   *
> >> >>   * Follow PCI and resume devices suspended at run time before running their
> >> >> - * system suspend callbacks.
> >> >> + * system suspend callbacks. However, try to avoid it in case the runtime PM
> >> >> + * centric path is used for the device and then trust the driver to do the
> >> >> + * right thing.
> >> >>   */
> >> >>  int acpi_subsys_suspend(struct device *dev)
> >> >>  {
> >> >> -     pm_runtime_resume(dev);
> >> >> +     struct acpi_device *adev = ACPI_COMPANION(dev);
> >> >> +
> >> >> +     if (!adev)
> >> >> +             return 0;
> >> >> +
> >> >> +     if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
> >> >> +             pm_runtime_resume(dev);
> >> >> +
> >> >>       return pm_generic_suspend(dev);
> >> >>  }
> >> >>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
> >> >
> >> > Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
> >> > and that's why I added the update_state thing.
> >> >
> >> > Moreover, the is_rpm_sleep flag here has to mean not only that
> >> > direct_complete should not be used with the device, but also that its driver
> >> > is fine with not resuming it.
> >>
> >> Let me try to explain this better. I realize the changelog is
> >> misleading around this particular section! Huh, apologize for that!
> >>
> >> First, patch1 makes the PM core treat the is_rpm_sleep flag as the
> >> direct_complete isn't allowed for the device.
> >>
> >> For that reason, when the is_rpm_sleep is set, there is no point
> >> calling acpi_dev_needs_resume() from acpi_subsys_prepare(), but
> >> instead that can be deferred to acpi_subsys_suspend() - because it
> >> doesn't matter if acpi_subsys_prepare() returns 0 or 1, in either case
> >> the acpi_subsys_suspend() will be called. That's really what goes on
> >> here.
> >>
> >> The end result is the same. If the acpi_dev_needs_resume() thinks that
> >> the device needs to be runtime resumed, pm_runtime_resume() is called
> >> for the device in acpi_subsys_suspend().
> >>
> >> So, this has nothing to do with whether the driver "is fine with not
> >> resuming it" thing.
> >
> > No, sorry.
> >
> > If is_rpm_sleep was not set, the ACPI PM domain would resume the device in
> > acpi_subsys_suspend() regardless of the acpi_dev_needs_resume() return value.
> 
> Yes, I believe I forgot about one scenario, when the direct_complete
> path has been abandoned by the PM core, because a child device was
> suspend before and it couldn't run the direct_complete path for it?
> 
> Just to be sure, that's the case you also had in mind?

Yes.

Thanks,
Rafael

[PATCH v3 6/8] PM / ACPI: Enable the runtime PM centric approach for system sleep

[Rafael J. Wysocki]

On Monday, September 4, 2017 3:21:15 PM CEST Ulf Hansson wrote:
> On 2 September 2017 at 17:38, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Friday, September 1, 2017 10:27:05 AM CEST Ulf Hansson wrote:
> >> On 29 August 2017 at 17:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> >> > On Tuesday, August 29, 2017 4:56:48 PM CEST Ulf Hansson wrote:
> >> >> This change enables the ACPI PM domain to cope with drivers that deploys
> >> >> the runtime PM centric path for system sleep.
> >> >
> >> > [cut]
> >> >
> >> >> @@ -1052,11 +1066,20 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
> >> >>   * @dev: Device to handle.
> >> >>   *
> >> >>   * Follow PCI and resume devices suspended at run time before running their
> >> >> - * system suspend callbacks.
> >> >> + * system suspend callbacks. However, try to avoid it in case the runtime PM
> >> >> + * centric path is used for the device and then trust the driver to do the
> >> >> + * right thing.
> >> >>   */
> >> >>  int acpi_subsys_suspend(struct device *dev)
> >> >>  {
> >> >> -     pm_runtime_resume(dev);
> >> >> +     struct acpi_device *adev = ACPI_COMPANION(dev);
> >> >> +
> >> >> +     if (!adev)
> >> >> +             return 0;
> >> >> +
> >> >> +     if (!dev_pm_is_rpm_sleep(dev) || acpi_dev_needs_resume(dev, adev))
> >> >> +             pm_runtime_resume(dev);
> >> >> +
> >> >>       return pm_generic_suspend(dev);
> >> >>  }
> >> >>  EXPORT_SYMBOL_GPL(acpi_subsys_suspend);
> >> >
> >> > Well, I tried to avoid calling acpi_dev_needs_resume() for multiple times
> >> > and that's why I added the update_state thing.
> >> >
> >> > Moreover, the is_rpm_sleep flag here has to mean not only that
> >> > direct_complete should not be used with the device, but also that its driver
> >> > is fine with not resuming it.
> >>
> >> Let me try to explain this better. I realize the changelog is
> >> misleading around this particular section! Huh, apologize for that!
> >>
> >> First, patch1 makes the PM core treat the is_rpm_sleep flag as the
> >> direct_complete isn't allowed for the device.
> >>
> >> For that reason, when the is_rpm_sleep is set, there is no point
> >> calling acpi_dev_needs_resume() from acpi_subsys_prepare(), but
> >> instead that can be deferred to acpi_subsys_suspend() - because it
> >> doesn't matter if acpi_subsys_prepare() returns 0 or 1, in either case
> >> the acpi_subsys_suspend() will be called. That's really what goes on
> >> here.
> >>
> >> The end result is the same. If the acpi_dev_needs_resume() thinks that
> >> the device needs to be runtime resumed, pm_runtime_resume() is called
> >> for the device in acpi_subsys_suspend().
> >>
> >> So, this has nothing to do with whether the driver "is fine with not
> >> resuming it" thing.
> >
> > No, sorry.
> >
> > If is_rpm_sleep was not set, the ACPI PM domain would resume the device in
> > acpi_subsys_suspend() regardless of the acpi_dev_needs_resume() return value.
> 
> Yes, I believe I forgot about one scenario, when the direct_complete
> path has been abandoned by the PM core, because a child device was
> suspend before and it couldn't run the direct_complete path for it?
> 
> Just to be sure, that's the case you also had in mind?

Yes.

Thanks,
Rafael

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Monday, September 4, 2017 2:55:37 PM CEST Ulf Hansson wrote:
> [...]
> 
> >> > So can you please remind me why the _force_ wrappers are needed?
> >>
> >> See below.
> >>
> >> >
> >> > In particular, why can't drivers arrange their callbacks the way I did that
> >> > in https://patchwork.kernel.org/patch/9928583/ ?
> >>
> >> I was preparing a reply to that patch, but let me summarize that here instead.
> >>
> >> Let me be clear, the patch is an improvement of the behavior of the
> >> driver and it addresses the issues you point out in the change log.
> >> Re-using the runtime PM callbacks for system sleep, is nice as it
> >> avoids open coding, which is of curse also one of the reason of using
> >> pm_runtime_force_suspend|resume().
> >>
> >> Still there are a couple of things I am worried about in this patch.
> >> *)
> >> To be able to re-use the same callbacks for system sleep and runtime
> >> PM, some boilerplate code is added to the driver, as to cope with the
> >> different conditions inside the callbacks. That pattern would become
> >> repeated to many drivers dealing with similar issues.
> >
> > I'm not worried about that as long as there are good examples and
> > documented best practices.
> >
> > There aren't any right now, which is a problem, but that certainly is
> > fixable.
> >
> >> **)
> >> The ->resume_early() callback powers on the device, in case it was
> >> runtime resumed when the ->suspend_late() callback was invoked. That
> >> is in many cases completely unnecessary, causing us to waste power and
> >> increase system resume time, for absolutely no reason. However, I
> >> understand the patch didn't try to address this, but to really fix
> >> this, there has to be an even closer collaboration between runtime PM
> >> and the system sleep callbacks.
> >
> > I don't quite agree and here's why.
> >
> > If a device was not runtime-suspended right before system suspend, then quite
> > likely it was in use then.  Therefore it is quite likely to be resumed
> > immediately after system resume anyway.
> 
> Unfortunate, to always make that assumption, leads to a non-optimized
> behavior of system sleep. I think we can do better than that!
> 
> Let me give you a concrete example, where the above assumption would
> lead to an non-optimized behavior.
> 
> To put an MMC card into low power state during system suspend
> (covering eMMC, SD, SDIO) the mmc core needs to send a couple of
> commands over the MMC interface to the card, as to conform with the
> (e)MMC/SD/SDIO spec. To do this, the mmc driver for the mmc controller
> must runtime resume its device, as to be able to send the commands
> over the interface.
> 
> Now, when the system resumes, there is absolutely no reason to runtime
> resume the device for the MMC controller, just because it was runtime
> resumed during system suspend. Instead that is better to be postponed
> to when the MMC card is really needed and thus via runtime PM instead.

Yes, in this particular case it makes more sense to defer the resume of
the device, but there also are cases in which doing that leads to
suboptimal behavior.

> This scenario shouldn't be specific to only MMC controllers/cards, but
> should apply to any external devices/controllers that needs some
> special treatment to be put into low power state during system
> suspend. Particularly also when those external devices may be left in
> that low power state until those are really needed. A couple of cases
> I know of pops up in my head, WiFi chips, persistent storage devices,
> etc. There should be plenty.
> 
> Another common case, is when a subsystem core layer flushes a request
> queue during system suspend, which may cause a controller device to be
> runtime resumed. Making the assumption that, because flushing the
> queue was done during system suspend, we must also power up the
> controller during system resume, again would lead to a non-optimized
> behavior.

I understand that.

However, from a driver perspective, the most straightforward thing to do
is to restore the previous state of the device during system resume,
because that guarantees correctness.  Anything else is tricky and need to
be done with extra care.  Drivers *must* know what they are doing when
they are doing such things.

> >
> > Now, if that's just one device, it probably doesn't matter, but if there are
> > more devices like that, they will be resumed after system suspend when they
> > are accessed and quite likely they will be accessed one-by-one rather than in
> > parallel with each other, so the latencies related to that will add up.  In
> > that case it is better to resume them upfront during system resume as they will
> > be resumed in parallel with each other then.  And that also is *way* simpler.
> >
> > This means that the benefit from avoiding to resume devices during system
> > resume is not quite so obvious and the whole point above is highly
> > questionable.
> 
> I hope my reasoning above explains why I think it shouldn't be
> considered as questionable.
> 
> If you like, I can also provide some real data/logs - showing you
> what's happening.
> 

That's not necessary, this behavior can be useful and there are arguments for
doing it in *some* cases, but all of this argumentation applies to devices
that aren't going to be used right after system resume.  If they *are* going
to be used then, it very well may be better to resume them as part of
system resume instead of deferring that.

The tricky part is that at the point the resume callbacks run it is not known
whether or not the device is going to be accessed shortly and the decision made
either way may be suboptimal.

[Note: I know that people mostly care about seeing the screen on, but in fact
they should *also* care about the touch panel being ready to respond to
touches, for example.  If it isn't ready and the system suspends again
after a while because of that, the experience is somehwat less than fantastic.]

> >>
> >> So, to remind you why the pm_runtime_force_suspend|resume() helpers is
> >> preferred, that's because both of the above two things becomes taken
> >> care of.
> >
> > And that is why there is this stuff about parents and usage counters, right?
> 
> Correct. Perhaps this commit tells you a little more.
> 
> commit 1d9174fbc55ec99ccbfcafa3de2528ef78a849aa
> Author: Ulf Hansson <ulf.hansson@linaro.org>
> Date:   Thu Oct 13 16:58:54 2016 +0200
> 
>     PM / Runtime: Defer resuming of the device in pm_runtime_force_resume()
> 
> [...]
> 
> >> >
> >> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late
> >> >
> >> > is far more straightforward than
> >> >
> >> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
> >> >         bus type / PM domain ->runtime_suspend => driver ->runtime_suspend
> >> >
> >> > with the bus type / PM domain having to figure out somehow at the
> >> > ->suspend_late time whether or not its ->runtume_suspend is going to be invoked
> >> > in the middle of it.
> >> >
> >> > Apart from this just being aesthetically disgusting to me, which admittedly is
> >> > a matter of personal opinion, it makes debugging new driver code harder (if it
> >> > happens to not work) and reviewing it almost impossible, because now you need
> >> > to take all of the tangling between callbacks into accont and sometimes not
> >> > just for one bus type / PM domain.
> >>
> >> I am wondering that perhaps you may be overlooking some of the
> >> internals of runtime PM. Or maybe not? :-)
> >>
> >> I mean, the hole thing is build upon that anyone can call runtime PM
> >> functions to runtime resume/suspend a device.
> >
> > Well, right in general, except that _force_suspend/resume() invoke
> > *callbacks* and *not* runtime PM functions.
> 
> I am considering pm_runtime_force_suspend|resume() being a part of the
> runtime PM API, except that those may be called only during system
> sleep.
> 
> Comparing a call to pm_runtime_resume(); this may trigger rpm_resume()
> to invoke the callbacks. To me, the difference is that the conditions
> looked at in rpm_resume(), when runtime PM is enabled, becomes
> different for system sleep when runtime PM is disabled - and that is
> taken care of in pm_runtime_force_suspend|resume().

So actually invoking runtime PM from a *driver* ->suspend callback for the
same device it was called for is fishy at best and may be a bug.  I'm not
sure why I had been thinking that it might have been fine at all.  It isn't.

The reason why is because runtime PM *potentially* involves invoking middle
layer callbacks an they generally may look like

->runtime_resume:
	(1) do A
	(2) call driver ->runtime_resume
	(3) do B

Now, a middle layer ->suspend callback generally may look like this:

->suspend:
	(1) do C
	(2) call driver ->suspend
	(3) do D

and if you stick the middle layer ->runtime_suspend invocation into the
driver ->suspend (which effectively is what running runtime PM in there means),
you get something like

do C
...
do A
call driver ->runtime_resume
do B
...
do D

and there's no guarantee whatever that "do C" can go before "do A" and
"do B" can go before "do D".  That depends on how the middle layer is designed
and there may be good reasons for how it works.

> >
> >> Doing that, makes the
> >> hierarchy of the runtime PM callbacks being walked and invoked, of
> >> course properly managed by the runtime PM core.
> >>
> >> My point is that, the runtime PM core still controls this behavior,
> >> even when the pm_runtime_force_suspend|resume() helpers are being
> >> invoked. The only difference is that it allows runtime PM for the
> >> device to be disabled, and still correctly invoked the callbacks. That
> >> is what it is all about.
> >
> > So why is it even useful to call ->runtime_suspend from a middle layer
> > in pm_runtime_force_suspend(), for example?
> 
> Perhaps I don't understand the question correctly.
> 
> Anyway, the answer I think of, is probably because of the same reason
> to why the runtime PM core invokes it, when it runs rpm_suspend() for
> a device. My point is, we want the similar behavior.

Not really.  The context is different, so why to expect the behavior to be
the same?

> [...]
> 
> >> >
> >> > Also, when I looked at _force_suspend/resume() again, I got concerned.
> >> > There is stuff in there that shouldn't be necessary in a driver's
> >> > ->late_suspend/->early_resume and some things in there just made me
> >> > scratch my head.
> >>
> >> Yes, there are some complexity in there, I will be happy to answer any
> >> specific question about it.
> >
> > OK
> >
> > Of course they require runtime PM to be enabled by drivers using them as
> > their callbacks, but I suppose that you realize that.
> >
> > Why to disabe/renable runtime PM in there in the first place?  That should
> > have been done by the core when these functions are intended to be called.
> 
> The reason is because we didn't want to re-strict them to be used only
> in ->suspend_late() and ->resume_early(), but also for ->suspend() and
> ->resume(), which is when runtime PM still is enabled.

Well, that means disabling runtime PM for some devices earlier which isn't
particularly consistent overall.

> >
> > Second, why to use RPM_GET_CALLBACK in there?
> 
> To follow the same rules/hierarchy, as being done in rpm_suspend|resume().

No, you don't use the same hierarchy, which is the key point of my objection.

You run *already* in the context of a middle layer PM callback, so by very
definition it is *not* the same situation as running runtime PM elsewhere.

This is the second or maybe even the third time I have repeated this point
and I'm not going to do so again.

> >
> > Next, how is the parent actually runtime-resumed by pm_runtime_force_resume()
> > which the comment in pm_runtime_force_suspend() talks about?
> 
> I think the relevant use case here is when a parent and a child, both
> have subsystems/drivers using pm_runtime_force_suspend|resume(). If
> that isn't the case, we expect that the parent is always resumed
> during system resume.

Why?

> It's a bit fragile approach, so we perhaps we
> should deal with it, even if the hole thing is used as opt-in.
> 
> Anyway, let's focus on the case which I think is most relevant to your question:
> 
> A couple of conditions to start with.
> *) The PM core system suspends a child prior a parent, which leads to
> pm_runtime_force_suspend() being called for the child first.
> **) The PM core system resumes a parents before a child, thus
> pm_runtime_force_resume() is called for the parent first.
> 
> In case a child don't need to be resumed when
> pm_runtime_force_resume() is called for it, likely doesn't its parent.
> However, to control that, in system suspend the
> pm_runtime_force_suspend() increases the usage counter for the parent,
> as to indicate if it needs to be resumed when
> pm_runtime_force_resume() is called for it.

OK, I see.

Why is usage_count > 1 used as the condition to trigger this behavior?

Thanks,
Rafael

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Monday, September 4, 2017 2:55:37 PM CEST Ulf Hansson wrote:
> [...]
> 
> >> > So can you please remind me why the _force_ wrappers are needed?
> >>
> >> See below.
> >>
> >> >
> >> > In particular, why can't drivers arrange their callbacks the way I did that
> >> > in https://patchwork.kernel.org/patch/9928583/ ?
> >>
> >> I was preparing a reply to that patch, but let me summarize that here instead.
> >>
> >> Let me be clear, the patch is an improvement of the behavior of the
> >> driver and it addresses the issues you point out in the change log.
> >> Re-using the runtime PM callbacks for system sleep, is nice as it
> >> avoids open coding, which is of curse also one of the reason of using
> >> pm_runtime_force_suspend|resume().
> >>
> >> Still there are a couple of things I am worried about in this patch.
> >> *)
> >> To be able to re-use the same callbacks for system sleep and runtime
> >> PM, some boilerplate code is added to the driver, as to cope with the
> >> different conditions inside the callbacks. That pattern would become
> >> repeated to many drivers dealing with similar issues.
> >
> > I'm not worried about that as long as there are good examples and
> > documented best practices.
> >
> > There aren't any right now, which is a problem, but that certainly is
> > fixable.
> >
> >> **)
> >> The ->resume_early() callback powers on the device, in case it was
> >> runtime resumed when the ->suspend_late() callback was invoked. That
> >> is in many cases completely unnecessary, causing us to waste power and
> >> increase system resume time, for absolutely no reason. However, I
> >> understand the patch didn't try to address this, but to really fix
> >> this, there has to be an even closer collaboration between runtime PM
> >> and the system sleep callbacks.
> >
> > I don't quite agree and here's why.
> >
> > If a device was not runtime-suspended right before system suspend, then quite
> > likely it was in use then.  Therefore it is quite likely to be resumed
> > immediately after system resume anyway.
> 
> Unfortunate, to always make that assumption, leads to a non-optimized
> behavior of system sleep. I think we can do better than that!
> 
> Let me give you a concrete example, where the above assumption would
> lead to an non-optimized behavior.
> 
> To put an MMC card into low power state during system suspend
> (covering eMMC, SD, SDIO) the mmc core needs to send a couple of
> commands over the MMC interface to the card, as to conform with the
> (e)MMC/SD/SDIO spec. To do this, the mmc driver for the mmc controller
> must runtime resume its device, as to be able to send the commands
> over the interface.
> 
> Now, when the system resumes, there is absolutely no reason to runtime
> resume the device for the MMC controller, just because it was runtime
> resumed during system suspend. Instead that is better to be postponed
> to when the MMC card is really needed and thus via runtime PM instead.

Yes, in this particular case it makes more sense to defer the resume of
the device, but there also are cases in which doing that leads to
suboptimal behavior.

> This scenario shouldn't be specific to only MMC controllers/cards, but
> should apply to any external devices/controllers that needs some
> special treatment to be put into low power state during system
> suspend. Particularly also when those external devices may be left in
> that low power state until those are really needed. A couple of cases
> I know of pops up in my head, WiFi chips, persistent storage devices,
> etc. There should be plenty.
> 
> Another common case, is when a subsystem core layer flushes a request
> queue during system suspend, which may cause a controller device to be
> runtime resumed. Making the assumption that, because flushing the
> queue was done during system suspend, we must also power up the
> controller during system resume, again would lead to a non-optimized
> behavior.

I understand that.

However, from a driver perspective, the most straightforward thing to do
is to restore the previous state of the device during system resume,
because that guarantees correctness.  Anything else is tricky and need to
be done with extra care.  Drivers *must* know what they are doing when
they are doing such things.

> >
> > Now, if that's just one device, it probably doesn't matter, but if there are
> > more devices like that, they will be resumed after system suspend when they
> > are accessed and quite likely they will be accessed one-by-one rather than in
> > parallel with each other, so the latencies related to that will add up.  In
> > that case it is better to resume them upfront during system resume as they will
> > be resumed in parallel with each other then.  And that also is *way* simpler.
> >
> > This means that the benefit from avoiding to resume devices during system
> > resume is not quite so obvious and the whole point above is highly
> > questionable.
> 
> I hope my reasoning above explains why I think it shouldn't be
> considered as questionable.
> 
> If you like, I can also provide some real data/logs - showing you
> what's happening.
> 

That's not necessary, this behavior can be useful and there are arguments for
doing it in *some* cases, but all of this argumentation applies to devices
that aren't going to be used right after system resume.  If they *are* going
to be used then, it very well may be better to resume them as part of
system resume instead of deferring that.

The tricky part is that at the point the resume callbacks run it is not known
whether or not the device is going to be accessed shortly and the decision made
either way may be suboptimal.

[Note: I know that people mostly care about seeing the screen on, but in fact
they should *also* care about the touch panel being ready to respond to
touches, for example.  If it isn't ready and the system suspends again
after a while because of that, the experience is somehwat less than fantastic.]

> >>
> >> So, to remind you why the pm_runtime_force_suspend|resume() helpers is
> >> preferred, that's because both of the above two things becomes taken
> >> care of.
> >
> > And that is why there is this stuff about parents and usage counters, right?
> 
> Correct. Perhaps this commit tells you a little more.
> 
> commit 1d9174fbc55ec99ccbfcafa3de2528ef78a849aa
> Author: Ulf Hansson <ulf.hansson@linaro.org>
> Date:   Thu Oct 13 16:58:54 2016 +0200
> 
>     PM / Runtime: Defer resuming of the device in pm_runtime_force_resume()
> 
> [...]
> 
> >> >
> >> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late
> >> >
> >> > is far more straightforward than
> >> >
> >> > PM core => bus type / PM domain ->suspend_late => driver ->suspend_late =>
> >> >         bus type / PM domain ->runtime_suspend => driver ->runtime_suspend
> >> >
> >> > with the bus type / PM domain having to figure out somehow at the
> >> > ->suspend_late time whether or not its ->runtume_suspend is going to be invoked
> >> > in the middle of it.
> >> >
> >> > Apart from this just being aesthetically disgusting to me, which admittedly is
> >> > a matter of personal opinion, it makes debugging new driver code harder (if it
> >> > happens to not work) and reviewing it almost impossible, because now you need
> >> > to take all of the tangling between callbacks into accont and sometimes not
> >> > just for one bus type / PM domain.
> >>
> >> I am wondering that perhaps you may be overlooking some of the
> >> internals of runtime PM. Or maybe not? :-)
> >>
> >> I mean, the hole thing is build upon that anyone can call runtime PM
> >> functions to runtime resume/suspend a device.
> >
> > Well, right in general, except that _force_suspend/resume() invoke
> > *callbacks* and *not* runtime PM functions.
> 
> I am considering pm_runtime_force_suspend|resume() being a part of the
> runtime PM API, except that those may be called only during system
> sleep.
> 
> Comparing a call to pm_runtime_resume(); this may trigger rpm_resume()
> to invoke the callbacks. To me, the difference is that the conditions
> looked at in rpm_resume(), when runtime PM is enabled, becomes
> different for system sleep when runtime PM is disabled - and that is
> taken care of in pm_runtime_force_suspend|resume().

So actually invoking runtime PM from a *driver* ->suspend callback for the
same device it was called for is fishy at best and may be a bug.  I'm not
sure why I had been thinking that it might have been fine at all.  It isn't.

The reason why is because runtime PM *potentially* involves invoking middle
layer callbacks an they generally may look like

->runtime_resume:
	(1) do A
	(2) call driver ->runtime_resume
	(3) do B

Now, a middle layer ->suspend callback generally may look like this:

->suspend:
	(1) do C
	(2) call driver ->suspend
	(3) do D

and if you stick the middle layer ->runtime_suspend invocation into the
driver ->suspend (which effectively is what running runtime PM in there means),
you get something like

do C
...
do A
call driver ->runtime_resume
do B
...
do D

and there's no guarantee whatever that "do C" can go before "do A" and
"do B" can go before "do D".  That depends on how the middle layer is designed
and there may be good reasons for how it works.

> >
> >> Doing that, makes the
> >> hierarchy of the runtime PM callbacks being walked and invoked, of
> >> course properly managed by the runtime PM core.
> >>
> >> My point is that, the runtime PM core still controls this behavior,
> >> even when the pm_runtime_force_suspend|resume() helpers are being
> >> invoked. The only difference is that it allows runtime PM for the
> >> device to be disabled, and still correctly invoked the callbacks. That
> >> is what it is all about.
> >
> > So why is it even useful to call ->runtime_suspend from a middle layer
> > in pm_runtime_force_suspend(), for example?
> 
> Perhaps I don't understand the question correctly.
> 
> Anyway, the answer I think of, is probably because of the same reason
> to why the runtime PM core invokes it, when it runs rpm_suspend() for
> a device. My point is, we want the similar behavior.

Not really.  The context is different, so why to expect the behavior to be
the same?

> [...]
> 
> >> >
> >> > Also, when I looked at _force_suspend/resume() again, I got concerned.
> >> > There is stuff in there that shouldn't be necessary in a driver's
> >> > ->late_suspend/->early_resume and some things in there just made me
> >> > scratch my head.
> >>
> >> Yes, there are some complexity in there, I will be happy to answer any
> >> specific question about it.
> >
> > OK
> >
> > Of course they require runtime PM to be enabled by drivers using them as
> > their callbacks, but I suppose that you realize that.
> >
> > Why to disabe/renable runtime PM in there in the first place?  That should
> > have been done by the core when these functions are intended to be called.
> 
> The reason is because we didn't want to re-strict them to be used only
> in ->suspend_late() and ->resume_early(), but also for ->suspend() and
> ->resume(), which is when runtime PM still is enabled.

Well, that means disabling runtime PM for some devices earlier which isn't
particularly consistent overall.

> >
> > Second, why to use RPM_GET_CALLBACK in there?
> 
> To follow the same rules/hierarchy, as being done in rpm_suspend|resume().

No, you don't use the same hierarchy, which is the key point of my objection.

You run *already* in the context of a middle layer PM callback, so by very
definition it is *not* the same situation as running runtime PM elsewhere.

This is the second or maybe even the third time I have repeated this point
and I'm not going to do so again.

> >
> > Next, how is the parent actually runtime-resumed by pm_runtime_force_resume()
> > which the comment in pm_runtime_force_suspend() talks about?
> 
> I think the relevant use case here is when a parent and a child, both
> have subsystems/drivers using pm_runtime_force_suspend|resume(). If
> that isn't the case, we expect that the parent is always resumed
> during system resume.

Why?

> It's a bit fragile approach, so we perhaps we
> should deal with it, even if the hole thing is used as opt-in.
> 
> Anyway, let's focus on the case which I think is most relevant to your question:
> 
> A couple of conditions to start with.
> *) The PM core system suspends a child prior a parent, which leads to
> pm_runtime_force_suspend() being called for the child first.
> **) The PM core system resumes a parents before a child, thus
> pm_runtime_force_resume() is called for the parent first.
> 
> In case a child don't need to be resumed when
> pm_runtime_force_resume() is called for it, likely doesn't its parent.
> However, to control that, in system suspend the
> pm_runtime_force_suspend() increases the usage counter for the parent,
> as to indicate if it needs to be resumed when
> pm_runtime_force_resume() is called for it.

OK, I see.

Why is usage_count > 1 used as the condition to trigger this behavior?

Thanks,
Rafael

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Wednesday, September 6, 2017 2:52:59 AM CEST Rafael J. Wysocki wrote:
> On Monday, September 4, 2017 2:55:37 PM CEST Ulf Hansson wrote:
> > [...]

I guess I can wrap it up, because all of the points seem to have been stated
and repeating them would not be useful.

My summary of the discussion is as follows.

It only is valid to use pm_runtime_force_suspend/resume() as *driver*
callbacks for system suspend/resume if both the driver itself and all of
the middle layers it has to work with carry out the same sequence of
operations in order to suspend the device both in runtime PM and for
system sleep (and analogously for resuming).  [The middle layers need
to meet additional conditions, but that's less relevant.]

Unfortunately, for the ACPI PM domain and the PCI bus type the situation is
different, because they generally need to do different things to suspend
devices for system sleep than they do for runtime PM (which mostly is
related to the handling of ACPI-defined sleep states and device/system
wakeup, but not limited to that).  This clearly means that drivers needing
to work with the ACPI PM domain and PCI drivers cannot use
pm_runtime_force_suspend/resume() as their PM callbacks for system
suspend/resume (quite fundamentally).

[Note that for i2c-designware-platdrv the situation is even more complicated,
because on some platforms it has to work with the ACPI PM domain (or the
ACPI LPSS driver), on some platforms its parent is a PCI device and on
some other platforms there's none of them.]

However, for drivers that need to work with the ACPI PM domain and
PCI drivers the differences in the device handling between runtime PM and
system suspend/resume are *very* often (even though not always) covered
entirely by the middle layer code.  Then, the driver itself actually
always carries out the same sequence of operations in order to suspend
the device (or to resume it, analogously).  The driver then can re-use
its runtime PM callbacks for system suspend/resume (but at the driver
level only) and it would be good to make that easy (or easier) for these
drivers somehow.

Thanks,
Rafael

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Wednesday, September 6, 2017 2:52:59 AM CEST Rafael J. Wysocki wrote:
> On Monday, September 4, 2017 2:55:37 PM CEST Ulf Hansson wrote:
> > [...]

I guess I can wrap it up, because all of the points seem to have been stated
and repeating them would not be useful.

My summary of the discussion is as follows.

It only is valid to use pm_runtime_force_suspend/resume() as *driver*
callbacks for system suspend/resume if both the driver itself and all of
the middle layers it has to work with carry out the same sequence of
operations in order to suspend the device both in runtime PM and for
system sleep (and analogously for resuming).  [The middle layers need
to meet additional conditions, but that's less relevant.]

Unfortunately, for the ACPI PM domain and the PCI bus type the situation is
different, because they generally need to do different things to suspend
devices for system sleep than they do for runtime PM (which mostly is
related to the handling of ACPI-defined sleep states and device/system
wakeup, but not limited to that).  This clearly means that drivers needing
to work with the ACPI PM domain and PCI drivers cannot use
pm_runtime_force_suspend/resume() as their PM callbacks for system
suspend/resume (quite fundamentally).

[Note that for i2c-designware-platdrv the situation is even more complicated,
because on some platforms it has to work with the ACPI PM domain (or the
ACPI LPSS driver), on some platforms its parent is a PCI device and on
some other platforms there's none of them.]

However, for drivers that need to work with the ACPI PM domain and
PCI drivers the differences in the device handling between runtime PM and
system suspend/resume are *very* often (even though not always) covered
entirely by the middle layer code.  Then, the driver itself actually
always carries out the same sequence of operations in order to suspend
the device (or to resume it, analogously).  The driver then can re-use
its runtime PM callbacks for system suspend/resume (but at the driver
level only) and it would be good to make that easy (or easier) for these
drivers somehow.

Thanks,
Rafael

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

[...]

>
>> >
>> > Now, if that's just one device, it probably doesn't matter, but if there are
>> > more devices like that, they will be resumed after system suspend when they
>> > are accessed and quite likely they will be accessed one-by-one rather than in
>> > parallel with each other, so the latencies related to that will add up.  In
>> > that case it is better to resume them upfront during system resume as they will
>> > be resumed in parallel with each other then.  And that also is *way* simpler.
>> >
>> > This means that the benefit from avoiding to resume devices during system
>> > resume is not quite so obvious and the whole point above is highly
>> > questionable.
>>
>> I hope my reasoning above explains why I think it shouldn't be
>> considered as questionable.
>>
>> If you like, I can also provide some real data/logs - showing you
>> what's happening.
>>
>
> That's not necessary, this behavior can be useful and there are arguments for
> doing it in *some* cases, but all of this argumentation applies to devices
> that aren't going to be used right after system resume.  If they *are* going
> to be used then, it very well may be better to resume them as part of
> system resume instead of deferring that.
>
> The tricky part is that at the point the resume callbacks run it is not known
> whether or not the device is going to be accessed shortly and the decision made
> either way may be suboptimal.

You have a point and it seems like this is what everything boils done
to, except for the reasons about that you dislike how
pm_runtime_force_suspend|resume() is being used by drivers.

To clarify, let me bring up yet another typical scenario, observed
often in cases when pm_runtime_force_suspend|resume is not used.

During system resume the device gets resumed, then shortly after the
system resume sequence has completed, it become runtime suspened,
because pm_runtime_put() is called in device_complete(). Then, soon
after the system has resumed, the device becomes runtime resumed
again, which is because there is a request for it to really be used.

This means we end up resuming the device, suspending it then and
resuming it again, all within a very short time frame. I guess this is
also one of those tricky cases you refer to above, because one just
can know how long after the system has resumed it takes for the device
to be requested to be used again, thus we end up runtime suspending
the device in-between.

To me, spending lot of time in the world of embedded battery driven
devices, this behavior isn't good enough, because it increases system
resume time and may waste some power. Apologize if you find me
repeating myself.

Anyway, this leads to my final question, do you want this behavior to
be better addressed by the ACPI PM domain, if it can be solved nicely,
or are you fine with how works today?

>
> [Note: I know that people mostly care about seeing the screen on, but in fact
> they should *also* care about the touch panel being ready to respond to
> touches, for example.  If it isn't ready and the system suspends again
> after a while because of that, the experience is somehwat less than fantastic.]
>

Yep!

[...]

>> Comparing a call to pm_runtime_resume(); this may trigger rpm_resume()
>> to invoke the callbacks. To me, the difference is that the conditions
>> looked at in rpm_resume(), when runtime PM is enabled, becomes
>> different for system sleep when runtime PM is disabled - and that is
>> taken care of in pm_runtime_force_suspend|resume().
>
> So actually invoking runtime PM from a *driver* ->suspend callback for the
> same device it was called for is fishy at best and may be a bug.  I'm not
> sure why I had been thinking that it might have been fine at all.  It isn't.

Huh, now you lost me. :-)

>
> The reason why is because runtime PM *potentially* involves invoking middle
> layer callbacks an they generally may look like
>
> ->runtime_resume:
>         (1) do A
>         (2) call driver ->runtime_resume
>         (3) do B
>
> Now, a middle layer ->suspend callback generally may look like this:
>
> ->suspend:
>         (1) do C
>         (2) call driver ->suspend
>         (3) do D
>
> and if you stick the middle layer ->runtime_suspend invocation into the
> driver ->suspend (which effectively is what running runtime PM in there means),
> you get something like
>
> do C
> ...
> do A
> call driver ->runtime_resume
> do B
> ...
> do D
>
> and there's no guarantee whatever that "do C" can go before "do A" and
> "do B" can go before "do D".  That depends on how the middle layer is designed
> and there may be good reasons for how it works.

For ARM SoCs, not using the ACPI PM domain, many drivers needs to be
able to use runtime PM during system suspend, simply because the PM
domain/middle layer, has no knowledge of what the driver needs to put
its device into low power state during system suspend.

For many of the simple cases, the PM domain/middle layer act
transparent to this, which means leaving what needs to be done to the
driver (platform, spi, i2c, amba, genpd etc).

I understand there may be some cases where the situation becomes more
complex and interaction between the driver and the PM domain/middle
layer is required, like what it seems for in ACPI PM domain and PCI,
but is that a reason turn the world upside down for everybody else?

Or perhaps I don't understand what your are suggesting here.

[...]

>
>> I think the relevant use case here is when a parent and a child, both
>> have subsystems/drivers using pm_runtime_force_suspend|resume(). If
>> that isn't the case, we expect that the parent is always resumed
>> during system resume.
>
> Why?

Because the child may rely on that for it to be resumed.

Moreover, the expectation is that the parent likely doesn't support
runtime PM, or that it not yet supports the optimized method of using
pm_runtime_force_suspend|resume() during system sleep, and will thus
resume its device always during system resume.

>
>> It's a bit fragile approach, so we perhaps we
>> should deal with it, even if the hole thing is used as opt-in.
>>
>> Anyway, let's focus on the case which I think is most relevant to your question:
>>
>> A couple of conditions to start with.
>> *) The PM core system suspends a child prior a parent, which leads to
>> pm_runtime_force_suspend() being called for the child first.
>> **) The PM core system resumes a parents before a child, thus
>> pm_runtime_force_resume() is called for the parent first.
>>
>> In case a child don't need to be resumed when
>> pm_runtime_force_resume() is called for it, likely doesn't its parent.
>> However, to control that, in system suspend the
>> pm_runtime_force_suspend() increases the usage counter for the parent,
>> as to indicate if it needs to be resumed when
>> pm_runtime_force_resume() is called for it.
>
> OK, I see.
>
> Why is usage_count > 1 used as the condition to trigger this behavior?

It takes into account that the PM core increases the usage count in
device_prepare(), but which isn't because it needs the device to be
operational.

Kind regards
Uffe

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

[...]

>
>> >
>> > Now, if that's just one device, it probably doesn't matter, but if there are
>> > more devices like that, they will be resumed after system suspend when they
>> > are accessed and quite likely they will be accessed one-by-one rather than in
>> > parallel with each other, so the latencies related to that will add up.  In
>> > that case it is better to resume them upfront during system resume as they will
>> > be resumed in parallel with each other then.  And that also is *way* simpler.
>> >
>> > This means that the benefit from avoiding to resume devices during system
>> > resume is not quite so obvious and the whole point above is highly
>> > questionable.
>>
>> I hope my reasoning above explains why I think it shouldn't be
>> considered as questionable.
>>
>> If you like, I can also provide some real data/logs - showing you
>> what's happening.
>>
>
> That's not necessary, this behavior can be useful and there are arguments for
> doing it in *some* cases, but all of this argumentation applies to devices
> that aren't going to be used right after system resume.  If they *are* going
> to be used then, it very well may be better to resume them as part of
> system resume instead of deferring that.
>
> The tricky part is that at the point the resume callbacks run it is not known
> whether or not the device is going to be accessed shortly and the decision made
> either way may be suboptimal.

You have a point and it seems like this is what everything boils done
to, except for the reasons about that you dislike how
pm_runtime_force_suspend|resume() is being used by drivers.

To clarify, let me bring up yet another typical scenario, observed
often in cases when pm_runtime_force_suspend|resume is not used.

During system resume the device gets resumed, then shortly after the
system resume sequence has completed, it become runtime suspened,
because pm_runtime_put() is called in device_complete(). Then, soon
after the system has resumed, the device becomes runtime resumed
again, which is because there is a request for it to really be used.

This means we end up resuming the device, suspending it then and
resuming it again, all within a very short time frame. I guess this is
also one of those tricky cases you refer to above, because one just
can know how long after the system has resumed it takes for the device
to be requested to be used again, thus we end up runtime suspending
the device in-between.

To me, spending lot of time in the world of embedded battery driven
devices, this behavior isn't good enough, because it increases system
resume time and may waste some power. Apologize if you find me
repeating myself.

Anyway, this leads to my final question, do you want this behavior to
be better addressed by the ACPI PM domain, if it can be solved nicely,
or are you fine with how works today?

>
> [Note: I know that people mostly care about seeing the screen on, but in fact
> they should *also* care about the touch panel being ready to respond to
> touches, for example.  If it isn't ready and the system suspends again
> after a while because of that, the experience is somehwat less than fantastic.]
>

Yep!

[...]

>> Comparing a call to pm_runtime_resume(); this may trigger rpm_resume()
>> to invoke the callbacks. To me, the difference is that the conditions
>> looked at in rpm_resume(), when runtime PM is enabled, becomes
>> different for system sleep when runtime PM is disabled - and that is
>> taken care of in pm_runtime_force_suspend|resume().
>
> So actually invoking runtime PM from a *driver* ->suspend callback for the
> same device it was called for is fishy at best and may be a bug.  I'm not
> sure why I had been thinking that it might have been fine at all.  It isn't.

Huh, now you lost me. :-)

>
> The reason why is because runtime PM *potentially* involves invoking middle
> layer callbacks an they generally may look like
>
> ->runtime_resume:
>         (1) do A
>         (2) call driver ->runtime_resume
>         (3) do B
>
> Now, a middle layer ->suspend callback generally may look like this:
>
> ->suspend:
>         (1) do C
>         (2) call driver ->suspend
>         (3) do D
>
> and if you stick the middle layer ->runtime_suspend invocation into the
> driver ->suspend (which effectively is what running runtime PM in there means),
> you get something like
>
> do C
> ...
> do A
> call driver ->runtime_resume
> do B
> ...
> do D
>
> and there's no guarantee whatever that "do C" can go before "do A" and
> "do B" can go before "do D".  That depends on how the middle layer is designed
> and there may be good reasons for how it works.

For ARM SoCs, not using the ACPI PM domain, many drivers needs to be
able to use runtime PM during system suspend, simply because the PM
domain/middle layer, has no knowledge of what the driver needs to put
its device into low power state during system suspend.

For many of the simple cases, the PM domain/middle layer act
transparent to this, which means leaving what needs to be done to the
driver (platform, spi, i2c, amba, genpd etc).

I understand there may be some cases where the situation becomes more
complex and interaction between the driver and the PM domain/middle
layer is required, like what it seems for in ACPI PM domain and PCI,
but is that a reason turn the world upside down for everybody else?

Or perhaps I don't understand what your are suggesting here.

[...]

>
>> I think the relevant use case here is when a parent and a child, both
>> have subsystems/drivers using pm_runtime_force_suspend|resume(). If
>> that isn't the case, we expect that the parent is always resumed
>> during system resume.
>
> Why?

Because the child may rely on that for it to be resumed.

Moreover, the expectation is that the parent likely doesn't support
runtime PM, or that it not yet supports the optimized method of using
pm_runtime_force_suspend|resume() during system sleep, and will thus
resume its device always during system resume.

>
>> It's a bit fragile approach, so we perhaps we
>> should deal with it, even if the hole thing is used as opt-in.
>>
>> Anyway, let's focus on the case which I think is most relevant to your question:
>>
>> A couple of conditions to start with.
>> *) The PM core system suspends a child prior a parent, which leads to
>> pm_runtime_force_suspend() being called for the child first.
>> **) The PM core system resumes a parents before a child, thus
>> pm_runtime_force_resume() is called for the parent first.
>>
>> In case a child don't need to be resumed when
>> pm_runtime_force_resume() is called for it, likely doesn't its parent.
>> However, to control that, in system suspend the
>> pm_runtime_force_suspend() increases the usage counter for the parent,
>> as to indicate if it needs to be resumed when
>> pm_runtime_force_resume() is called for it.
>
> OK, I see.
>
> Why is usage_count > 1 used as the condition to trigger this behavior?

It takes into account that the PM core increases the usage count in
device_prepare(), but which isn't because it needs the device to be
operational.

Kind regards
Uffe

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

On 6 September 2017 at 12:46, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Wednesday, September 6, 2017 2:52:59 AM CEST Rafael J. Wysocki wrote:
>> On Monday, September 4, 2017 2:55:37 PM CEST Ulf Hansson wrote:
>> > [...]
>
> I guess I can wrap it up, because all of the points seem to have been stated
> and repeating them would not be useful.
>
> My summary of the discussion is as follows.
>
> It only is valid to use pm_runtime_force_suspend/resume() as *driver*
> callbacks for system suspend/resume if both the driver itself and all of
> the middle layers it has to work with carry out the same sequence of
> operations in order to suspend the device both in runtime PM and for
> system sleep (and analogously for resuming).  [The middle layers need
> to meet additional conditions, but that's less relevant.]
>
> Unfortunately, for the ACPI PM domain and the PCI bus type the situation is
> different, because they generally need to do different things to suspend
> devices for system sleep than they do for runtime PM (which mostly is
> related to the handling of ACPI-defined sleep states and device/system
> wakeup, but not limited to that).  This clearly means that drivers needing
> to work with the ACPI PM domain and PCI drivers cannot use
> pm_runtime_force_suspend/resume() as their PM callbacks for system
> suspend/resume (quite fundamentally).
>
> [Note that for i2c-designware-platdrv the situation is even more complicated,
> because on some platforms it has to work with the ACPI PM domain (or the
> ACPI LPSS driver), on some platforms its parent is a PCI device and on
> some other platforms there's none of them.]

That is also why it makes it really interesting. I am guessing we will
be seeing more of these cases sooner or later.

To make it even more complex, I can guess we can expect cases when
genpd is mixed with the ACPI PM domain.

>
> However, for drivers that need to work with the ACPI PM domain and
> PCI drivers the differences in the device handling between runtime PM and
> system suspend/resume are *very* often (even though not always) covered
> entirely by the middle layer code.  Then, the driver itself actually
> always carries out the same sequence of operations in order to suspend
> the device (or to resume it, analogously).  The driver then can re-use
> its runtime PM callbacks for system suspend/resume (but at the driver
> level only) and it would be good to make that easy (or easier) for these
> drivers somehow.

This is a very nice summary so far, thanks for putting it together.

Kind regards
Uffe

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Ulf Hansson]

On 6 September 2017 at 12:46, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Wednesday, September 6, 2017 2:52:59 AM CEST Rafael J. Wysocki wrote:
>> On Monday, September 4, 2017 2:55:37 PM CEST Ulf Hansson wrote:
>> > [...]
>
> I guess I can wrap it up, because all of the points seem to have been stated
> and repeating them would not be useful.
>
> My summary of the discussion is as follows.
>
> It only is valid to use pm_runtime_force_suspend/resume() as *driver*
> callbacks for system suspend/resume if both the driver itself and all of
> the middle layers it has to work with carry out the same sequence of
> operations in order to suspend the device both in runtime PM and for
> system sleep (and analogously for resuming).  [The middle layers need
> to meet additional conditions, but that's less relevant.]
>
> Unfortunately, for the ACPI PM domain and the PCI bus type the situation is
> different, because they generally need to do different things to suspend
> devices for system sleep than they do for runtime PM (which mostly is
> related to the handling of ACPI-defined sleep states and device/system
> wakeup, but not limited to that).  This clearly means that drivers needing
> to work with the ACPI PM domain and PCI drivers cannot use
> pm_runtime_force_suspend/resume() as their PM callbacks for system
> suspend/resume (quite fundamentally).
>
> [Note that for i2c-designware-platdrv the situation is even more complicated,
> because on some platforms it has to work with the ACPI PM domain (or the
> ACPI LPSS driver), on some platforms its parent is a PCI device and on
> some other platforms there's none of them.]

That is also why it makes it really interesting. I am guessing we will
be seeing more of these cases sooner or later.

To make it even more complex, I can guess we can expect cases when
genpd is mixed with the ACPI PM domain.

>
> However, for drivers that need to work with the ACPI PM domain and
> PCI drivers the differences in the device handling between runtime PM and
> system suspend/resume are *very* often (even though not always) covered
> entirely by the middle layer code.  Then, the driver itself actually
> always carries out the same sequence of operations in order to suspend
> the device (or to resume it, analogously).  The driver then can re-use
> its runtime PM callbacks for system suspend/resume (but at the driver
> level only) and it would be good to make that easy (or easier) for these
> drivers somehow.

This is a very nice summary so far, thanks for putting it together.

Kind regards
Uffe

Re: [PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Wednesday, September 6, 2017 3:59:16 PM CEST Ulf Hansson wrote:
> On 6 September 2017 at 12:46, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Wednesday, September 6, 2017 2:52:59 AM CEST Rafael J. Wysocki wrote:
> >> On Monday, September 4, 2017 2:55:37 PM CEST Ulf Hansson wrote:
> >> > [...]
> >
> > I guess I can wrap it up, because all of the points seem to have been stated
> > and repeating them would not be useful.
> >
> > My summary of the discussion is as follows.
> >
> > It only is valid to use pm_runtime_force_suspend/resume() as *driver*
> > callbacks for system suspend/resume if both the driver itself and all of
> > the middle layers it has to work with carry out the same sequence of
> > operations in order to suspend the device both in runtime PM and for
> > system sleep (and analogously for resuming).  [The middle layers need
> > to meet additional conditions, but that's less relevant.]
> >
> > Unfortunately, for the ACPI PM domain and the PCI bus type the situation is
> > different, because they generally need to do different things to suspend
> > devices for system sleep than they do for runtime PM (which mostly is
> > related to the handling of ACPI-defined sleep states and device/system
> > wakeup, but not limited to that).  This clearly means that drivers needing
> > to work with the ACPI PM domain and PCI drivers cannot use
> > pm_runtime_force_suspend/resume() as their PM callbacks for system
> > suspend/resume (quite fundamentally).
> >
> > [Note that for i2c-designware-platdrv the situation is even more complicated,
> > because on some platforms it has to work with the ACPI PM domain (or the
> > ACPI LPSS driver), on some platforms its parent is a PCI device and on
> > some other platforms there's none of them.]
> 
> That is also why it makes it really interesting. I am guessing we will
> be seeing more of these cases sooner or later.
> 
> To make it even more complex, I can guess we can expect cases when
> genpd is mixed with the ACPI PM domain.
> 
> >
> > However, for drivers that need to work with the ACPI PM domain and
> > PCI drivers the differences in the device handling between runtime PM and
> > system suspend/resume are *very* often (even though not always) covered
> > entirely by the middle layer code.  Then, the driver itself actually
> > always carries out the same sequence of operations in order to suspend
> > the device (or to resume it, analogously).  The driver then can re-use
> > its runtime PM callbacks for system suspend/resume (but at the driver
> > level only) and it would be good to make that easy (or easier) for these
> > drivers somehow.
> 
> This is a very nice summary so far, thanks for putting it together.

No problem.

I actually have an idea on how to move forward, but let me start a new thread
for discussing that.

Thanks,
Rafael

[PATCH v3 0/8] PM / ACPI / i2c: Deploy runtime PM centric path for system sleep

[Rafael J. Wysocki]

On Wednesday, September 6, 2017 3:59:16 PM CEST Ulf Hansson wrote:
> On 6 September 2017 at 12:46, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Wednesday, September 6, 2017 2:52:59 AM CEST Rafael J. Wysocki wrote:
> >> On Monday, September 4, 2017 2:55:37 PM CEST Ulf Hansson wrote:
> >> > [...]
> >
> > I guess I can wrap it up, because all of the points seem to have been stated
> > and repeating them would not be useful.
> >
> > My summary of the discussion is as follows.
> >
> > It only is valid to use pm_runtime_force_suspend/resume() as *driver*
> > callbacks for system suspend/resume if both the driver itself and all of
> > the middle layers it has to work with carry out the same sequence of
> > operations in order to suspend the device both in runtime PM and for
> > system sleep (and analogously for resuming).  [The middle layers need
> > to meet additional conditions, but that's less relevant.]
> >
> > Unfortunately, for the ACPI PM domain and the PCI bus type the situation is
> > different, because they generally need to do different things to suspend
> > devices for system sleep than they do for runtime PM (which mostly is
> > related to the handling of ACPI-defined sleep states and device/system
> > wakeup, but not limited to that).  This clearly means that drivers needing
> > to work with the ACPI PM domain and PCI drivers cannot use
> > pm_runtime_force_suspend/resume() as their PM callbacks for system
> > suspend/resume (quite fundamentally).
> >
> > [Note that for i2c-designware-platdrv the situation is even more complicated,
> > because on some platforms it has to work with the ACPI PM domain (or the
> > ACPI LPSS driver), on some platforms its parent is a PCI device and on
> > some other platforms there's none of them.]
> 
> That is also why it makes it really interesting. I am guessing we will
> be seeing more of these cases sooner or later.
> 
> To make it even more complex, I can guess we can expect cases when
> genpd is mixed with the ACPI PM domain.
> 
> >
> > However, for drivers that need to work with the ACPI PM domain and
> > PCI drivers the differences in the device handling between runtime PM and
> > system suspend/resume are *very* often (even though not always) covered
> > entirely by the middle layer code.  Then, the driver itself actually
> > always carries out the same sequence of operations in order to suspend
> > the device (or to resume it, analogously).  The driver then can re-use
> > its runtime PM callbacks for system suspend/resume (but at the driver
> > level only) and it would be good to make that easy (or easier) for these
> > drivers somehow.
> 
> This is a very nice summary so far, thanks for putting it together.

No problem.

I actually have an idea on how to move forward, but let me start a new thread
for discussing that.

Thanks,
Rafael