[RFC] Runtime PM for host controllers

[RFC] Runtime PM for host controllers

[Kevin Cernekee]

In embedded applications, there are a number of situations in which it
is useful to completely disable a host controller in order to conserve
power.  Some typical scenarios include:

1) A SATA controller is present, but there are no disks attached.
Clock-gating the SATA PHY saves a great deal of power, with the
tradeoff that the controller will no longer be able to detect
hotplugging on the (e)SATA connector.  The user/application is willing
to forego the possibility that a disk will be hotplugged, in order to
realize the power savings.

2) Similar to #1, except there IS a disk attached.  Maybe even
mounted.  But the user/application knows that it will not need to
access the disk any time soon, so it would like to put that part of
the system to sleep.

One such case occurs on PVRs: the user presses the "power off" button
to put the PVR into a non-interactive standby mode.  The TV screen
goes black, yet the OS needs to be kept running for various reasons:
program guide downloads, network bridging/routing, general
housekeeping, etc.  But the HDD can be safely spun down, and the SATA
interface can be put into power save mode until such time that it is
needed again.

It is possible that the various Linux-based NAS devices may also want
to implement this sort of feature, to conserve power after several
hours/days of inactivity.  For many home users, it might make sense
for a NAS appliance to enter a low power state during periods of
limited usage (nighttime), but remain active on the LAN (replying to
ARPs and sending NetBIOS advertisements) so that new requests could be
serviced with a few seconds' notice.

For both NAS and PVR, the root filesystem is typically on flash, not on the HDD.

PVRs in particular are subject to increasingly strict Energy Star /
European CoC rules, so saving even a few hundred milliwatts in standby
mode is critical.

3) Cases #1 and #2, applied to other hotpluggable interfaces like USB,
MMC/SD, or (less commonly) PCIe.

The USB scenario could include situations where the USB *HCI host
controllers can be safely disabled because the physical interface has
been switched to USB device (gadget/OTG) mode.  Or disabled through a
configuration menu to conserve power - especially on a mobile phone.

It could also occur in the case of NAS devices that use external USB
drives instead of SATA drives.

4) A network interface is present, but the user/application knows that
networking will not be needed in the host's current state.  So it can
run "ifconfig eth0 down" and then put the controller and PHY into a
low-power state.


FWIW, these "extra" power save features are usually implemented in
proprietary ways by each SoC manufacturer.  They are not always part
of the general AHCI/EHCI/SDHCI/... specification.  In some cases the
official specification does provide for certain power save modes, but
more extensive savings can be achieved by clock-gating or power-gating
the entire hardware block.  It is probably best to assume they will
always need to be handled by the clk_* functions, or something
similar.


Looking through Linus' head of tree, I see:

USB runtime PM seems to focus on the individual USB devices, and does
not allow for dynamically suspending the entire host controller
AFAICT.

USB used to allow writing "suspend" to the power/control file, but
that no longer seems to be the case as of 2.6.32.  In general I don't
see any facility where the application can explicitly tell the PM core
to suspend a device (incurring a potential loss of functionality, such
as hot plug/unplug detection).

libata has some limited support for runtime PM on a per-port basis,
but on my ahci_platform controller, both ports show up as "active" and
do not seem to want to autosuspend.  One has a HDD attached; the other
is unconnected.  I do not see a way to tell libata to suspend the
whole controller, or even freeze a single port, from userland.

MMC/SD seems to be in better shape with respect to "opportunistically"
suspending the controller if it is idle for 50ms, except the design
still assumes that the hardware is able to detect card
insertion/removal through some other mechanism.  From commit
66fd8ad51:

"For Medfield, the host controller's card detect mechanism is
supplanted by an always-on GPIO which provides for card detect
wake-up."

This is not feasible on all systems, so for systems that lack this
"side channel," it would be necessary to provide a way for userland to
indicate whether it is really OK to suspend the controller and
potentially lose the ability to detect card plug/unplug events.

Several ethernet drivers (such as e1000e) allow autosuspend via
runtime PM if the interface is down.  This is probably about the best
we could hope for on a network adapter.  Although it may be worth
thinking about what to do for cases like ethtool ioctls, which could
cause problems if they result in register accesses to a clock-gated
device.  Is it better to let each driver handle the problem in its own
way, or should the net/core/ code be made aware of runtime PM?


So, my questions are:

a) Is SATA/USB/MMC host controller suspension something that can be
accomplished within the existing runtime PM framework?

b) If not, what is the best way to modify the runtime PM framework so
that it can support these use cases?

c) Is this something you would even want in mainline?

d) Or should I just go back to using "rmmod"?


Thanks in advance for your input.

Re: [RFC] Runtime PM for host controllers

[Alan Stern]

On Sun, 8 Apr 2012, Kevin Cernekee wrote:

> In embedded applications, there are a number of situations in which it
> is useful to completely disable a host controller in order to conserve
> power.  Some typical scenarios include:

<SNIP>

> Looking through Linus' head of tree, I see:
> 
> USB runtime PM seems to focus on the individual USB devices, and does
> not allow for dynamically suspending the entire host controller
> AFAICT.

That is not correct.  USB host controllers _can_ be suspended.  (Not 
all of them; many of the platform drivers don't implement controller 
suspend.  But some of them do, and almost all of the PCI-based drivers 
do.)

> USB used to allow writing "suspend" to the power/control file, but
> that no longer seems to be the case as of 2.6.32.

Correct.

>  In general I don't
> see any facility where the application can explicitly tell the PM core
> to suspend a device (incurring a potential loss of functionality, such
> as hot plug/unplug detection).

Right; there is no way to do this.  The OS will suspend a device only 
when it believes it is safe to do so.

<SNIP>

> So, my questions are:
> 
> a) Is SATA/USB/MMC host controller suspension something that can be
> accomplished within the existing runtime PM framework?

Not only _can_ it be accomplished, it already _has_ been accomplished 
(for USB at least, I'm not familiar with the others).

> b) If not, what is the best way to modify the runtime PM framework so
> that it can support these use cases?

No changes to the framework are needed.  If a subsystem or driver
doesn't support runtime suspend then support has to be added at the 
subsystem or driver level.

> c) Is this something you would even want in mainline?

Yes.  People are already working on this stuff.

> d) Or should I just go back to using "rmmod"?

I don't see how rmmod will save any power.

Alan Stern

Re: [RFC] Runtime PM for host controllers

[Kevin Cernekee]

On Mon, Apr 9, 2012 at 7:14 AM, Alan Stern <stern@rowland.harvard.edu> wrote:
>> USB runtime PM seems to focus on the individual USB devices, and does
>> not allow for dynamically suspending the entire host controller
>> AFAICT.
>
> That is not correct.  USB host controllers _can_ be suspended.  (Not
> all of them; many of the platform drivers don't implement controller
> suspend.  But some of them do, and almost all of the PCI-based drivers
> do.)

I am not clear on what is actually happening at the chip level in these cases?

i.e. is the entire USB core getting clock gated or power gated, such
that it is completely non-responsive to register accesses or device
hotplug?  Or is the driver just shutting down a few tiny pieces of the
hardware?

Is there a way to initiate USB runtime suspend on the host controller
even if there are devices plugged into the downstream ports?

>>  In general I don't
>> see any facility where the application can explicitly tell the PM core
>> to suspend a device (incurring a potential loss of functionality, such
>> as hot plug/unplug detection).
>
> Right; there is no way to do this.  The OS will suspend a device only
> when it believes it is safe to do so.

Assuming that somebody is able to come up with a satisfactory
implementation, are you open to allowing a "user-initiated suspend"
mode, in addition to "opportunistic suspend"?

>> d) Or should I just go back to using "rmmod"?
>
> I don't see how rmmod will save any power.

rmmod allows me to forcibly shut down the controller, cleanly (in
theory) disconnecting any active devices and preventing the kernel
from touching the registers.

After rmmod, I can then tell the SoC to put the whole core into an
unresponsive, low power state.

This is not a very clean way of doing things, so I would prefer to use
runtime PM if at all possible.  But I don't think my hardware is
designed in a way that is compatible with "opportunistic suspend."


Thanks.

Re: [RFC] Runtime PM for host controllers

[Mark Brown]

On Mon, Apr 09, 2012 at 10:14:06AM -0400, Alan Stern wrote:
> On Sun, 8 Apr 2012, Kevin Cernekee wrote:

> > a) Is SATA/USB/MMC host controller suspension something that can be
> > accomplished within the existing runtime PM framework?

> Not only _can_ it be accomplished, it already _has_ been accomplished 
> (for USB at least, I'm not familiar with the others).

MMC does this quite happily too, controller and device suspends are
largely orthogonal.

Re: [RFC] Runtime PM for host controllers

[Alan Stern]

On Mon, 9 Apr 2012, Kevin Cernekee wrote:

> On Mon, Apr 9, 2012 at 7:14 AM, Alan Stern <stern@rowland.harvard.edu> wrote:
> >> USB runtime PM seems to focus on the individual USB devices, and does
> >> not allow for dynamically suspending the entire host controller
> >> AFAICT.
> >
> > That is not correct.  USB host controllers _can_ be suspended.  (Not
> > all of them; many of the platform drivers don't implement controller
> > suspend.  But some of them do, and almost all of the PCI-based drivers
> > do.)
> 
> I am not clear on what is actually happening at the chip level in these cases?
> 
> i.e. is the entire USB core getting clock gated or power gated, such
> that it is completely non-responsive to register accesses or device
> hotplug?  Or is the driver just shutting down a few tiny pieces of the
> hardware?

The entire controller goes into a low-power mode.  For example, a PCI 
controller would go into D3hot.  The hardware is no longer responsive 
to register accesses, but it can generate a wakeup signal in response 
to device hotplug.

> Is there a way to initiate USB runtime suspend on the host controller
> even if there are devices plugged into the downstream ports?

Yes, provided all those devices are themselves already suspended.

> >>  In general I don't
> >> see any facility where the application can explicitly tell the PM core
> >> to suspend a device (incurring a potential loss of functionality, such
> >> as hot plug/unplug detection).
> >
> > Right; there is no way to do this.  The OS will suspend a device only
> > when it believes it is safe to do so.
> 
> Assuming that somebody is able to come up with a satisfactory
> implementation, are you open to allowing a "user-initiated suspend"
> mode, in addition to "opportunistic suspend"?

I'm not sure what you are asking.  If "use-initiated suspend"  
involves loss of functionality (such as loss of hotplug/unplug
detection) then no, it is generally not acceptable -- although the
final decision rests with the subsystem's maintainer.  If it does not
involve any such loss then no "user-initiated suspend" should be
needed, because the existing runtime suspend methods will do the job.

On the other hand, if you really want to suspend a device and don't
care about using it, then unbinding it from its driver is a good start.  
For USB devices, that alone will cause the device to be suspended.  
For PCI devices (such as host controllers), unfortunately, it isn't
sufficient -- the PCI subsystem doesn't allow driverless devices to be
suspended.  Perhaps the same is true of the platform subsystem; I'm not
sure.

If you'd like to work on a method of allowing driverless PCI or
platform devices to be runtime-suspended, that would be okay.  It 
probably wouldn't be very hard to do.

> >> d) Or should I just go back to using "rmmod"?
> >
> > I don't see how rmmod will save any power.
> 
> rmmod allows me to forcibly shut down the controller, cleanly (in
> theory) disconnecting any active devices and preventing the kernel
> from touching the registers.
> 
> After rmmod, I can then tell the SoC to put the whole core into an
> unresponsive, low power state.
> 
> This is not a very clean way of doing things, so I would prefer to use
> runtime PM if at all possible.  But I don't think my hardware is
> designed in a way that is compatible with "opportunistic suspend."

In what way is it incompatible?

Alan Stern

Re: [RFC] Runtime PM for host controllers

[Kevin Cernekee]

On Tue, Apr 10, 2012 at 7:15 AM, Alan Stern <stern@rowland.harvard.edu> wrote:
>> I am not clear on what is actually happening at the chip level in these cases?
>>
>> i.e. is the entire USB core getting clock gated or power gated, such
>> that it is completely non-responsive to register accesses or device
>> hotplug?  Or is the driver just shutting down a few tiny pieces of the
>> hardware?
>
> The entire controller goes into a low-power mode.  For example, a PCI
> controller would go into D3hot.  The hardware is no longer responsive
> to register accesses, but it can generate a wakeup signal in response
> to device hotplug.

Do you have any estimates of power savings on the host controller side?

On my systems, it would take a lot of extra power to maintain the
ability to detect device hotplug.  I am wondering if the PCI based
controller has a better way of handling it in the hardware than we do,
or if the power savings from D3hot are just very small.

>> Is there a way to initiate USB runtime suspend on the host controller
>> even if there are devices plugged into the downstream ports?
>
> Yes, provided all those devices are themselves already suspended.

But a lot of the drivers for USB devices do not even support suspend,
correct?  And many of them become unreliable if you turn it on?

What if I just want to forcibly suspend the entire USB subsystem?

>> >>  In general I don't
>> >> see any facility where the application can explicitly tell the PM core
>> >> to suspend a device (incurring a potential loss of functionality, such
>> >> as hot plug/unplug detection).
>> >
>> > Right; there is no way to do this.  The OS will suspend a device only
>> > when it believes it is safe to do so.
>>
>> Assuming that somebody is able to come up with a satisfactory
>> implementation, are you open to allowing a "user-initiated suspend"
>> mode, in addition to "opportunistic suspend"?
>
> I'm not sure what you are asking.  If "use-initiated suspend"
> involves loss of functionality (such as loss of hotplug/unplug
> detection) then no, it is generally not acceptable -- although the
> final decision rests with the subsystem's maintainer.

Who is the right person to make the call?  Rafael?

> If it does not
> involve any such loss then no "user-initiated suspend" should be
> needed, because the existing runtime suspend methods will do the job.

Right.  Unfortunately I don't really have any devices that work this
way.  Most are "all or nothing."

>> rmmod allows me to forcibly shut down the controller, cleanly (in
>> theory) disconnecting any active devices and preventing the kernel
>> from touching the registers.
>>
>> After rmmod, I can then tell the SoC to put the whole core into an
>> unresponsive, low power state.
>>
>> This is not a very clean way of doing things, so I would prefer to use
>> runtime PM if at all possible.  But I don't think my hardware is
>> designed in a way that is compatible with "opportunistic suspend."
>
> In what way is it incompatible?

My devices lose functionality when they are put into low-power modes.

Re: [RFC] Runtime PM for host controllers

[Alan Stern]

On Tue, 10 Apr 2012, Kevin Cernekee wrote:

> > The entire controller goes into a low-power mode.  For example, a PCI
> > controller would go into D3hot.  The hardware is no longer responsive
> > to register accesses, but it can generate a wakeup signal in response
> > to device hotplug.
> 
> Do you have any estimates of power savings on the host controller side?

No.  Other people may have measured it, though.

> On my systems, it would take a lot of extra power to maintain the
> ability to detect device hotplug.  I am wondering if the PCI based
> controller has a better way of handling it in the hardware than we do,
> or if the power savings from D3hot are just very small.

PCI devices have two different power wells, one for normal use and one 
for use during suspend.  As far as I know, there is no way to turn off 
power to the auxiliary well short of turning off the entire computer.  
Rafael may have a more complete picture.

> >> Is there a way to initiate USB runtime suspend on the host controller
> >> even if there are devices plugged into the downstream ports?
> >
> > Yes, provided all those devices are themselves already suspended.
> 
> But a lot of the drivers for USB devices do not even support suspend,
> correct?  And many of them become unreliable if you turn it on?

I haven't counted, but I think most of the commonly used drivers do
support it.  And most of the devices can be suspended and resumed
reliably, although there certainly are exceptions.

> What if I just want to forcibly suspend the entire USB subsystem?

Your rmmod approach will always work, but that isn't really a suspend.  
The USB subsystem doesn't provide any way to "force" a device to be
suspended against the kernel's will.

> >> Assuming that somebody is able to come up with a satisfactory
> >> implementation, are you open to allowing a "user-initiated suspend"
> >> mode, in addition to "opportunistic suspend"?
> >
> > I'm not sure what you are asking.  If "use-initiated suspend"
> > involves loss of functionality (such as loss of hotplug/unplug
> > detection) then no, it is generally not acceptable -- although the
> > final decision rests with the subsystem's maintainer.
> 
> Who is the right person to make the call?  Rafael?

The maintainer for whatever subsystem you're working on.  However,
general policy is set by Rafael and overall consensus on the linux-pm
list.

> > If it does not
> > involve any such loss then no "user-initiated suspend" should be
> > needed, because the existing runtime suspend methods will do the job.
> 
> Right.  Unfortunately I don't really have any devices that work this
> way.  Most are "all or nothing."

Is the loss of functionality you're talking about essentially the same
as removing the driver?  I don't think anybody would object to a
driverless device being put into a non-functional low-power mode, since
without a driver, it's pretty non-functional anyway.  It's easy for
userspace to unbind a device from its driver; would that then do what
you want?

> >> This is not a very clean way of doing things, so I would prefer to use
> >> runtime PM if at all possible.  But I don't think my hardware is
> >> designed in a way that is compatible with "opportunistic suspend."
> >
> > In what way is it incompatible?
> 
> My devices lose functionality when they are put into low-power modes.

What functionality do they lose?

Alan Stern

Re: [RFC] Runtime PM for host controllers

[Kevin Cernekee]

On Tue, Apr 10, 2012 at 12:48 PM, Alan Stern <stern@rowland.harvard.edu> wrote:
>> On my systems, it would take a lot of extra power to maintain the
>> ability to detect device hotplug.  I am wondering if the PCI based
>> controller has a better way of handling it in the hardware than we do,
>> or if the power savings from D3hot are just very small.
>
> PCI devices have two different power wells, one for normal use and one
> for use during suspend.  As far as I know, there is no way to turn off
> power to the auxiliary well short of turning off the entire computer.
> Rafael may have a more complete picture.

That may be the case on PCs, but on many embedded systems (mine
included), entire hardware blocks can be clock-gated or power-gated
without shutting off the rest of the system.

>> > If it does not
>> > involve any such loss then no "user-initiated suspend" should be
>> > needed, because the existing runtime suspend methods will do the job.
>>
>> Right.  Unfortunately I don't really have any devices that work this
>> way.  Most are "all or nothing."
>
> Is the loss of functionality you're talking about essentially the same
> as removing the driver?

>From a hardware perspective, yes.

> I don't think anybody would object to a
> driverless device being put into a non-functional low-power mode, since
> without a driver, it's pretty non-functional anyway.  It's easy for
> userspace to unbind a device from its driver; would that then do what
> you want?

Probably, but it seems a little clunky when we have a "runtime
suspend" framework already in place...

Unbinding/binding the device is slower than suspending it, and might
be a problem if there are mounted filesystems involved.

Going back to the NAS example I mentioned earlier - I would imagine
that merely resuming the USB/SATA controller + HDD when a new request
arrives is going to provide a much faster recovery time than rebinding
+ redetecting + remounting.

(Of course that raises issues like: what if somebody unplugs the
current HDD, then plugs in a different HDD, while the USB/SATA core is
asleep?)

>> >> This is not a very clean way of doing things, so I would prefer to use
>> >> runtime PM if at all possible.  But I don't think my hardware is
>> >> designed in a way that is compatible with "opportunistic suspend."
>> >
>> > In what way is it incompatible?
>>
>> My devices lose functionality when they are put into low-power modes.
>
> What functionality do they lose?

Everything, including register access, PHY links, and hotplug detection.

Re: [RFC] Runtime PM for host controllers

[Mark Brown]

On Tue, Apr 10, 2012 at 03:42:33PM -0700, Kevin Cernekee wrote:
> On Tue, Apr 10, 2012 at 12:48 PM, Alan Stern <stern@rowland.harvard.edu> wrote:

> > PCI devices have two different power wells, one for normal use and one
> > for use during suspend.  As far as I know, there is no way to turn off
> > power to the auxiliary well short of turning off the entire computer.
> > Rafael may have a more complete picture.

> That may be the case on PCs, but on many embedded systems (mine
> included), entire hardware blocks can be clock-gated or power-gated
> without shutting off the rest of the system.

Do you have any numbers here?  With systems that care about power
consumption on this level you tend to find there's good enough power
management within the IP block down to the level where you need to start
suspending the system itself to make a noticable impact.

Sometimes this needs to be accomplished in conjunction with things like
pin muxing - for example, configuring the pins to wake on an edge and
then having the driver detect this edge and wake up the IP itself in
order to actually handle the event.

> Probably, but it seems a little clunky when we have a "runtime
> suspend" framework already in place...

The goal of the runtime suspend framework is to be transparent to the
end user.

Re: [RFC] Runtime PM for host controllers

[Alan Stern]

On Tue, 10 Apr 2012, Kevin Cernekee wrote:

> > PCI devices have two different power wells, one for normal use and one
> > for use during suspend.  As far as I know, there is no way to turn off
> > power to the auxiliary well short of turning off the entire computer.
> > Rafael may have a more complete picture.
> 
> That may be the case on PCs,

It should be true on any system -- PC, SoC, or anything else -- that 
uses PCI.  But again, my knowledge is limited.

>  but on many embedded systems (mine
> included), entire hardware blocks can be clock-gated or power-gated
> without shutting off the rest of the system.

Then why isn't there a way to clock-gate or power-gate the parts of a 
hardware block used for data communication while leaving the circuits 
involved in wakeup detection active?  Such a capability is part of the 
EHCI specification.

> Going back to the NAS example I mentioned earlier - I would imagine
> that merely resuming the USB/SATA controller + HDD when a new request
> arrives is going to provide a much faster recovery time than rebinding
> + redetecting + remounting.

Certainly.  And leaving the USB controller suspended but powered up
would be even faster.

Furthermore, what happens when the user plugs in a new USB device, or
unplugs one that is already attached?  The USB specification is very
clear about what device suspend entails; completely turning off the
power violates the spec.

> (Of course that raises issues like: what if somebody unplugs the
> current HDD, then plugs in a different HDD, while the USB/SATA core is
> asleep?)

Right.  Although we do try to detect that sort of thing, since it can
also happen during hibernation.  But our detection methods are not 100%
accurate.

Alan Stern