[PATCH/RESEND v2 0/2] Hard disk S3 resume time optimization

[PATCH/RESEND v2 0/2] Hard disk S3 resume time optimization

[Todd E Brandt]

This patch reduces S3 resume time from 10+ seconds to less than a second
on systems with SATA drives. It does this by making ata port and scsi disk
resume non-blocking.

This is another resend of the patch I send out in early December. I've
addressed all the feedback I received from list members, so all I really
need is a yay or nay from Tejun Huo and James Bottomley (since it touches
both subsystems).

The notes below include the performance results and patch description.

[Computer One Test - 10.5X speedup on S3 resume]
Platform: Ubuntu Raring Ringtail (13.04) kernel 3.11.0-rc7
Cpu: Intel(R) Core(TM) i7-3960X CPU @ 3.30GHz
Ahci: Intel C600/X79 series chipset 6-Port SATA AHCI (r5)
Disk config (using all 6 ahci ports): 
240 GB SSD, 3 TB HD, 500 GB HD, DVD-ROM (with cd inserted), 2 TB HD, 1 TB HD
Resume time: [unpatched 11656ms] [patched 1110ms]

[Computer Two Test - 12X speedup on S3 resume]
Platform: Ubuntu Raring Ringtail (13.04) kernel 3.11.0-rc7
Cpu: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
Ahci: Intel 7 Series/C210 Series Chipset Family 6-port SATA [AHCI mode] (r4)
Disk config (using 1 ahci port): 320 GB Hard Disk
Resume time: [unpatched 5416 ms] [patched 448 ms]

[Computer Three Test - 7.8X speedup on S3 resume]
Platform: Ubuntu Raring Ringtail (13.04) kernel 3.11.0-rc7
Cpu: Intel(R) Core(TM) i7-4770S CPU @ 3.10GHz
Ahci: Intel Lynx Point 6-port SATA Controller 1 [AHCI mode] (r2)
Disk config (using 2 ahci ports): DVD-ROM (empty), 500 GB Hard Disk
Resume time: [unpatched 5385ms] [patched 688ms]

The essential issue behind hard disks' lengthy resume time is the ata port
driver blocking until the ATA port hardware is finished coming online. So
the kernel isn't really doing anything during all those seconds that the
disks are resuming, it's just blocking until the hardware says it's ready
to accept commands. Applying this patch set allows SATA disks to resume
asynchronously without holding up system resume, thus allowing the UI to
come online much more quickly. There may be a short period after resume
where the disks are still spinning up in the background, but it will only
momentarily affect applications using that disk.

The patch set has two parts which apply to ata_port_resume and sd_resume
respectively. Both are required to achieve any real performance benefit,
but they will still function independantly without a performance hit.

ata_port_resume patch (1/2):
sd_resume patch (2/2):

v2:
 [in response to Oliver Neukum]
 - Added scsi cmd error reporting through the scsi_sense_hdr
 [in response to James Bottomley]
 - unified the sd_resume code path by just making sd_resume async. Thus all
   three resume callbacks: resume, restore, and runtime-resume will use it,
   but there is a performance benefit for resume only.
 [in response to Bartlomiej Zolnierkiewicz]
 - unified the __ata_port_resume_common code path to include an async
   parameter. This way there's no need to re-implement ata_port_request_pm.

Re: [PATCH/RESEND v2 0/2] Hard disk S3 resume time optimization

[Gwendal Grignou]

On Tue, Jan 7, 2014 at 4:55 PM, Todd E Brandt
<todd.e.brandt@linux.intel.com> wrote:
>
> This patch reduces S3 resume time from 10+ seconds to less than a second
> on systems with SATA drives. It does this by making ata port and scsi disk
> resume non-blocking.
>
> This is another resend of the patch I send out in early December. I've
> addressed all the feedback I received from list members, so all I really
> need is a yay or nay from Tejun Huo and James Bottomley (since it touches
> both subsystems).
>
> The notes below include the performance results and patch description.
>
> [Computer One Test - 10.5X speedup on S3 resume]
> Platform: Ubuntu Raring Ringtail (13.04) kernel 3.11.0-rc7
> Cpu: Intel(R) Core(TM) i7-3960X CPU @ 3.30GHz
> Ahci: Intel C600/X79 series chipset 6-Port SATA AHCI (r5)
> Disk config (using all 6 ahci ports):
> 240 GB SSD, 3 TB HD, 500 GB HD, DVD-ROM (with cd inserted), 2 TB HD, 1 TB HD
> Resume time: [unpatched 11656ms] [patched 1110ms]
>
> [Computer Two Test - 12X speedup on S3 resume]
> Platform: Ubuntu Raring Ringtail (13.04) kernel 3.11.0-rc7
> Cpu: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
> Ahci: Intel 7 Series/C210 Series Chipset Family 6-port SATA [AHCI mode] (r4)
> Disk config (using 1 ahci port): 320 GB Hard Disk
> Resume time: [unpatched 5416 ms] [patched 448 ms]
>
> [Computer Three Test - 7.8X speedup on S3 resume]
> Platform: Ubuntu Raring Ringtail (13.04) kernel 3.11.0-rc7
> Cpu: Intel(R) Core(TM) i7-4770S CPU @ 3.10GHz
> Ahci: Intel Lynx Point 6-port SATA Controller 1 [AHCI mode] (r2)
> Disk config (using 2 ahci ports): DVD-ROM (empty), 500 GB Hard Disk
> Resume time: [unpatched 5385ms] [patched 688ms]
>
> The essential issue behind hard disks' lengthy resume time is the ata port
> driver blocking until the ATA port hardware is finished coming online. So
> the kernel isn't really doing anything during all those seconds that the
> disks are resuming, it's just blocking until the hardware says it's ready
> to accept commands. Applying this patch set allows SATA disks to resume
> asynchronously without holding up system resume, thus allowing the UI to
> come online much more quickly.
Won't this patch defeat staggered spinup at resume? If you have a jbod
with a smallish power supply, with a 12V rail designed for the steady
state and 1 or 2 devices spinning up at once, you may be in trouble
when all the disks will want to spinup in same second.

Gwendal.
> There may be a short period after resume
> where the disks are still spinning up in the background, but it will only
> momentarily affect applications using that disk.
>
> The patch set has two parts which apply to ata_port_resume and sd_resume
> respectively. Both are required to achieve any real performance benefit,
> but they will still function independantly without a performance hit.
>
> ata_port_resume patch (1/2):
> sd_resume patch (2/2):
>
> v2:
>  [in response to Oliver Neukum]
>  - Added scsi cmd error reporting through the scsi_sense_hdr
>  [in response to James Bottomley]
>  - unified the sd_resume code path by just making sd_resume async. Thus all
>    three resume callbacks: resume, restore, and runtime-resume will use it,
>    but there is a performance benefit for resume only.
>  [in response to Bartlomiej Zolnierkiewicz]
>  - unified the __ata_port_resume_common code path to include an async
>    parameter. This way there's no need to re-implement ata_port_request_pm.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RESEND v2 0/2] Hard disk S3 resume time optimization

[Tejun Heo]

Hello, Gwendal.

On Mon, Jan 13, 2014 at 04:36:52PM -0800, Gwendal Grignou wrote:
> Won't this patch defeat staggered spinup at resume? If you have a jbod
> with a smallish power supply, with a 12V rail designed for the steady
> state and 1 or 2 devices spinning up at once, you may be in trouble
> when all the disks will want to spinup in same second.

Yeah, I think it would.  We never fully solved the staggered spinup
problem and it probably is too late to invest into proper mechanism
for that.  I think an easy workaround would be, say, have a kernel
param which limits the max number of concurrent async wakeups and set
the value to something reasonable - say, 2, which should give us most
of benefits of async wakeup in vast majority of cases while avoiding
blowing up PSUs on larger setups.

Thanks.

-- 
tejun

Re: [PATCH/RESEND v2 0/2] Hard disk S3 resume time optimization

[Todd E Brandt]

On Tue, Jan 14, 2014 at 09:43:25AM -0500, Tejun Heo wrote:
> Hello, Gwendal.
> 
> On Mon, Jan 13, 2014 at 04:36:52PM -0800, Gwendal Grignou wrote:
> > Won't this patch defeat staggered spinup at resume? If you have a jbod
> > with a smallish power supply, with a 12V rail designed for the steady
> > state and 1 or 2 devices spinning up at once, you may be in trouble
> > when all the disks will want to spinup in same second.
> 
> Yeah, I think it would.  We never fully solved the staggered spinup
> problem and it probably is too late to invest into proper mechanism
> for that.  I think an easy workaround would be, say, have a kernel
> param which limits the max number of concurrent async wakeups and set
> the value to something reasonable - say, 2, which should give us most
> of benefits of async wakeup in vast majority of cases while avoiding
> blowing up PSUs on larger setups.

Does my patch actually have any affect on staggered spinup? I see that when
the HOST_CAP_SSS flag is detected it causes a PORT_CMD_SPIN_UP to be issued
during each port's resume. But the port resume calls themselves appear to
always be asynchronous, regardless of my patch's changes. (The ata_port
devices all have their power.async_suspend flags set, so the dpm_resume
call issues them all in parallel) So as far as I can tell I don't think 
I'm causing any harm.

I do see how my patch might hinder future attempts at staggered resume
handling, since it would effectively render the power.async_suspend flag
useless. Would it be wise for ata_port_resume to check the ata_port's 
power.async_suspend flag and only use asynchronous ata_port_request_pm 
if it's enabled?

> 
> Thanks.
> 
> -- 
> tejun

Re: [PATCH/RESEND v2 0/2] Hard disk S3 resume time optimization

[Gwendal Grignou]

You're right, staggered spinup is not enforced today when the disks
are behind several ATA ports. Only the disks behind a port multiplier
are spun up one at a time.
We are currently waiting for all the device(s) on the slowest port to
resume to continue.
Gwendal.

On Tue, Jan 14, 2014 at 1:45 PM, Todd E Brandt
<todd.e.brandt@linux.intel.com> wrote:
> On Tue, Jan 14, 2014 at 09:43:25AM -0500, Tejun Heo wrote:
>> Hello, Gwendal.
>>
>> On Mon, Jan 13, 2014 at 04:36:52PM -0800, Gwendal Grignou wrote:
>> > Won't this patch defeat staggered spinup at resume? If you have a jbod
>> > with a smallish power supply, with a 12V rail designed for the steady
>> > state and 1 or 2 devices spinning up at once, you may be in trouble
>> > when all the disks will want to spinup in same second.
>>
>> Yeah, I think it would.  We never fully solved the staggered spinup
>> problem and it probably is too late to invest into proper mechanism
>> for that.  I think an easy workaround would be, say, have a kernel
>> param which limits the max number of concurrent async wakeups and set
>> the value to something reasonable - say, 2, which should give us most
>> of benefits of async wakeup in vast majority of cases while avoiding
>> blowing up PSUs on larger setups.
>
> Does my patch actually have any affect on staggered spinup? I see that when
> the HOST_CAP_SSS flag is detected it causes a PORT_CMD_SPIN_UP to be issued
> during each port's resume. But the port resume calls themselves appear to
> always be asynchronous, regardless of my patch's changes. (The ata_port
> devices all have their power.async_suspend flags set, so the dpm_resume
> call issues them all in parallel) So as far as I can tell I don't think
> I'm causing any harm.
>
> I do see how my patch might hinder future attempts at staggered resume
> handling, since it would effectively render the power.async_suspend flag
> useless. Would it be wise for ata_port_resume to check the ata_port's
> power.async_suspend flag and only use asynchronous ata_port_request_pm
> if it's enabled?
>
>>
>> Thanks.
>>
>> --
>> tejun