linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Ingo Molnar <mingo@elte.hu>
To: Mark Lord <lkml@rtr.ca>
Cc: Pavel Machek <pavel@ucw.cz>, Mark Lord <liml@rtr.ca>,
	Thomas Gleixner <tglx@linutronix.de>,
	len.brown@intel.com, Andrew Morton <akpm@linux-foundation.org>,
	linux-kernel@vger.kernel.org,
	linux-pm@lists.linux-foundation.org, rjw@sisk.pl,
	Len Brown <lenb@kernel.org>
Subject: Re: [BUG] Strange 1-second pauses during Resume-from-RAM
Date: Fri, 16 Nov 2007 12:23:17 +0100	[thread overview]
Message-ID: <20071116112317.GA12724@elte.hu> (raw)
In-Reply-To: <20071116082146.GA14685@elte.hu>


* Ingo Molnar <mingo@elte.hu> wrote:

> once that tracer bug was fixed, the best method to generate a trace 
> was to do this:
> 
>    echo 1 > /proc/sys/kernel/stackframe_tracing
>    echo 1 > /proc/sys/kernel/syscall_tracing
>    ./trace-cmd bash -c "echo mem > /sys/power/state" > trace.txt

so here's an UP suspend+resume trace i did:

  http://redhat.com/~mingo/latency-tracing-patches/misc/trace-suspend-long.txt.bz2

tons of detail - which might be interesting to other folks as well. Fact 
is, our suspend-to-RAM+resume cycle is very, very slow, even on fast 
hardware - and this trace shows all the reasons why.

This was a fully cached system - i.e. i've done a suspend+resume before 
to warm up the caches. (not that suspend+resume does much IO normally.)

The trace shows that a suspend+resume cycle is 7.95 seconds long 
(without counting the time the box spent suspended) - ouch! This was a 
T60 with Core2Duo 1.83GHz.

For example here is where freezing starts:

    bash-2397  0.... 31686us : remove_wait_queue (vt_waitactive)
    bash-2397  0.... 31688us : freeze_processes (enter_state)
    bash-2397  0.... 31689us : printk (freeze_processes)

here is where the ACPI code triggers the suspend:

    bash-2397  0D... 1904138us : acpi_hw_low_level_write (acpi_hw_register_write)

but this is a whopping 1.9 seconds into the trace already!

first sign of life after i opened the laptop lid again:

    bash-2397  0D... 1904138us : __restore_processor_state (restore_processor_state)
    bash-2397  0D... 1904138us : enable_sep_cpu (__restore_processor_state)

(in the trace there's no delay visible - the period of time spent 
suspended is not visible to the tracer.)

One good way to start looking at such traces is to filter out 
rescheduling events alone:

  grep ': schedule <'  trace-suspend-long.txt

that gives a rough outline of what's going on:

   <idle>-0     0D... 1776566us : schedule <bash-2397> (0 20)
     bash-2397  0D... 1786748us : schedule <<idle>-0> (20 0)
 scsi_eh_-419   0D... 1786814us : schedule <bash-2397> (0 -5)
     bash-2397  0D... 1786960us : schedule <scsi_eh_-419> (-5 0)
 scsi_eh_-421   0D... 1787020us : schedule <bash-2397> (0 -5)
     bash-2397  0D... 1787125us : schedule <scsi_eh_-421> (-5 0)

so you can zoom in on the real area of interest by searching for the 
timestamp.

Hope this helps,

	Ingo

  reply	other threads:[~2007-11-16 11:24 UTC|newest]

Thread overview: 268+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2007-11-13  6:42 [BUG] New Kernel Bugs Natalie Protasevich
2007-11-13 11:15 ` Andrew Morton
2007-11-13 11:24   ` Jens Axboe
2007-11-13 11:33   ` Evgeniy Polyakov
2007-11-13 11:39   ` David Miller
2007-11-13 11:49     ` Andrew Morton
2007-11-13 11:58       ` David Miller
2007-11-13 12:12         ` Andrew Morton
2007-11-13 12:32           ` David Miller
2007-11-13 19:02             ` Andrew Morton
2007-11-13 20:00               ` Christian Kujau
2007-11-13 21:04                 ` Andrew Morton
2007-11-13 16:56                   ` Nick Piggin
2007-11-14 19:54                     ` Linus Torvalds
2007-11-14 22:22                       ` Heikki Orsila
2007-11-14 23:05                         ` Linus Torvalds
2007-11-13 21:37                   ` Adrian Bunk
2007-11-13 21:56                   ` Christian Kujau
2007-11-15  4:07                     ` Bron Gondwana
2007-11-15  4:24                       ` Linus Torvalds
2007-11-15  5:25                         ` Bron Gondwana
2007-11-15  5:35                           ` Linus Torvalds
2007-11-15  5:53                             ` Linus Torvalds
2007-11-15 11:50                               ` mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs) Bron Gondwana
2007-11-15 16:32                                 ` Linus Torvalds
2007-11-15 19:40                                   ` Peter Zijlstra
2007-11-15 20:44                                     ` Peter Zijlstra
2007-11-15 20:56                                       ` Linus Torvalds
2007-11-15 20:59                                         ` Peter Zijlstra
2007-11-15 21:12                                           ` Peter Zijlstra
2007-11-15 21:14                                         ` Linus Torvalds
2007-11-15 21:26                                           ` Linus Torvalds
2007-11-15 21:26                                           ` Peter Zijlstra
2007-11-15 21:47                                             ` Linus Torvalds
2007-11-15 22:11                                               ` Chris Friesen
2007-11-15 22:31                                                 ` Linus Torvalds
2007-11-15 22:24                                               ` Rob Mueller
2007-11-18 23:13                                                 ` Daniel Phillips
2007-11-19  3:41                                                   ` Bron Gondwana
2007-11-16  0:48                                               ` Alan Cox
2007-11-21 21:25                                               ` Jan Engelhardt
2007-11-19  3:54                                           ` Bron Gondwana
2007-11-22  3:42                                           ` [PATCH 1/1] mm: add dirty_highmem option Bron Gondwana
2007-11-26 17:53                                             ` Linus Torvalds
2007-11-27  1:30                                               ` Bron Gondwana
2007-11-27  4:54                                             ` Andrew Morton
2007-11-27  5:24                                               ` Bron Gondwana
2007-11-27  5:53                                                 ` Andrew Morton
2007-11-27 12:10                                                   ` dirty highmem calculation sysctl name (Was: [PATCH 1/1] mm: add dirty_highmem option) Bron Gondwana
2007-11-27 13:06                                                     ` [PATCH] mm/page-writeback - highmem_is_dirtyable option (replaces dirty_highmem patch) Bron Gondwana
2007-11-21 23:51                                   ` mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs) Bron Gondwana
2007-11-22  2:16                                     ` Bron Gondwana
2007-11-13 19:32             ` [BUG] New Kernel Bugs Russell King
2007-11-13 20:13               ` Adrian Bunk
2007-11-13 23:29                 ` Russell King
2007-11-13 23:38                   ` Andrew Morton
2007-11-13 20:52               ` Andrew Morton
2007-11-13 22:18                 ` Russell King
2007-11-13 22:32                   ` Andrew Morton
2007-11-13 23:09                     ` Russell King
2007-11-13 23:17                       ` Andrew Morton
2007-11-14  1:55                     ` David Miller
2007-11-14  2:27                       ` Andrew Morton
2007-11-14  3:47                         ` David Miller
2007-11-14  8:30                         ` Russell King
2007-11-14  9:55                       ` Russell King
2007-11-14 10:07                         ` David Miller
2007-11-14 11:46                           ` [alsa-devel] " Rene Herman
2007-11-14 11:56                             ` David Miller
2007-11-14 12:01                               ` David Miller
2007-11-14  8:25                                 ` Moderated list (Was: Re: [BUG] New Kernel Bugs) Takashi Iwai
2007-11-14 12:21                                   ` Rene Herman
2007-11-14  9:47                                     ` Takashi Iwai
2007-11-14 23:23                                       ` Moderated list David Miller
2007-11-15  6:09                                         ` Rene Herman
2007-11-14 12:12                                 ` [alsa-devel] [BUG] New Kernel Bugs Rene Herman
2007-11-14 12:09                               ` Rene Herman
2007-11-15  4:16                             ` Bron Gondwana
2007-11-15  5:59                               ` Rene Herman
2007-11-15 12:02                                 ` Bron Gondwana
2007-11-15 12:26                                   ` Rene Herman
2007-11-15 13:00                                     ` Jörn Engel
2007-11-15 14:29                                       ` Rene Herman
2007-11-15 13:17                                 ` Olivier Galibert
2007-11-15  9:34                                   ` Takashi Iwai
2007-11-14 19:44                           ` Russell King
2007-11-16 22:16                           ` Use *poof* for linux-omap (Was: [BUG] New Kernel Bugs) Tony Lindgren
2007-11-17  0:45                             ` Use *poof* for linux-omap David Miller
2007-11-18 20:01                               ` Tony Lindgren
2007-11-14  5:56                   ` [BUG] New Kernel Bugs Sam Ravnborg
2007-11-14  5:59                     ` Sam Ravnborg
2007-11-14  6:13                     ` David Miller
2007-11-13 13:40           ` Ingo Molnar
2007-11-13 14:08             ` Mark Lord
2007-11-13 15:24               ` Giacomo A. Catenazzi
2007-11-13 15:57                 ` Ray Lee
2007-11-13 17:01                   ` Adrian Bunk
2007-11-13 17:50                     ` Romano Giannetti
2007-11-13 22:03                       ` Frans Pop
2007-11-13 15:52               ` Benoit Boissinot
2007-11-13 16:49                 ` Ingo Molnar
2007-11-13 17:13                 ` Theodore Tso
2007-11-13 17:30                   ` Alan Cox
2007-11-13 17:33                   ` Larry Finger
2007-11-13 18:55                     ` Theodore Tso
2007-11-13 20:07                       ` Larry Finger
2007-11-13 17:56                   ` Adrian Bunk
2007-11-13 18:57                     ` Gabriel C
2007-11-14  0:41                       ` Denys Vlasenko
2007-11-14  0:39                     ` Denys Vlasenko
2007-11-14  7:27                       ` Adrian Bunk
2007-11-14  7:46                         ` Denys Vlasenko
2007-11-14 13:30                           ` Matthew Wilcox
2007-11-14 13:35                             ` Hannes Reinecke
2007-11-14 21:39                               ` Denys Vlasenko
2007-11-14 21:58                                 ` Gabriel C
2007-11-14 18:27                           ` Kok, Auke
2007-11-14 16:55                     ` Jan Evert van Grootheest
2007-11-14 23:23                   ` Daniel Barkalow
2007-11-15 15:30                     ` Theodore Tso
2007-11-15 16:19                       ` Daniel Barkalow
2007-11-16  8:20                         ` Romano Giannetti
2007-11-16 18:20                           ` Daniel Barkalow
2007-11-16 19:46                             ` Theodore Tso
2007-11-17 12:20                               ` Adrian Bunk
2007-11-18 18:01                                 ` Theodore Tso
2007-11-13 16:46               ` Ingo Molnar
2007-11-13 17:50                 ` Mark Lord
2007-11-13 18:12                   ` Adrian Bunk
2007-11-13 18:18                     ` Mark Lord
2007-11-13 18:36                       ` Adrian Bunk
2007-11-13 18:47                         ` Mark Lord
2007-11-13 19:04                           ` Adrian Bunk
2007-11-13 19:12                             ` Mark Lord
2007-11-13 19:30                               ` Adrian Bunk
2007-11-13 19:46                                 ` Russell King
2007-11-13 20:04                                   ` Adrian Bunk
2007-11-13 19:26                             ` Mark Lord
2007-11-13 20:00                               ` Adrian Bunk
2007-11-13 20:13                                 ` Mark Lord
2007-11-13 21:20                                   ` Adrian Bunk
2007-11-13 21:12                                 ` Alan Cox
2007-11-14  0:52                                   ` Chuck Ebbert
2007-11-14  1:11                                     ` Stephen Hemminger
2007-11-14  2:10                                       ` Andrew Morton
2007-11-14  1:10                       ` David Miller
2007-11-14  1:18                         ` Peter Stuge
2007-11-13 18:17                   ` Peter Zijlstra
2007-11-13 18:39                   ` Matthew Wilcox
2007-11-13 18:43                     ` Mark Lord
2007-11-13 18:49                       ` Matthew Wilcox
2007-11-13 18:54                         ` Mark Lord
2007-11-13 22:09                           ` Rafael J. Wysocki
2007-11-14 14:30                           ` Ingo Molnar
2007-11-14 14:49                             ` Larry Finger
2007-11-18 12:44                   ` size of git repository (was Re: [BUG] New Kernel Bugs) Pavel Machek
2007-11-18 12:58                     ` Rene Herman
2007-11-18 14:35                       ` James Bottomley
2007-11-18 15:19                         ` Rene Herman
2007-11-18 14:56                     ` Ingo Molnar
2007-11-19  4:43                       ` Willy Tarreau
2007-11-13 19:37               ` [BUG] New Kernel Bugs Russell King
2007-11-13 20:18                 ` Mark Lord
2007-11-13 21:33                   ` Jörn Engel
2007-11-13 21:56                     ` Andrew Morton
2007-11-13 22:24                       ` Jörn Engel
2007-11-13 22:43                         ` Andrew Morton
2007-11-13 22:29                     ` Mark Lord
2007-11-13 23:40                   ` Russell King
2007-11-14  1:56                     ` David Miller
2007-11-14  0:34               ` Denys Vlasenko
2007-11-15  3:06                 ` Neil Brown
2007-11-13 16:55             ` Randy Dunlap
2007-11-14 14:08               ` Ingo Molnar
2007-11-14 17:38                 ` Randy Dunlap
2007-11-14 18:23                   ` J. Bruce Fields
2007-11-15  2:50                     ` Neil Brown
2007-11-16  0:05                       ` J. Bruce Fields
2007-11-14 20:16                   ` Ingo Molnar
2007-11-14 20:29                     ` Randy Dunlap
2007-11-14 20:37                       ` Ingo Molnar
2007-11-14 21:05                         ` Randy Dunlap
2007-11-14 19:56                 ` David Miller
2007-11-14 20:09                   ` James Bottomley
2007-11-14 20:54                     ` Ingo Molnar
2007-11-14 20:48                   ` Ingo Molnar
2007-11-14 21:05                     ` david
2007-11-13 11:47   ` Jarek Poplawski
2007-11-13 13:58   ` Mark Lord
2007-11-13 14:18     ` Mark Lord
2007-11-13 16:08       ` Thomas Gleixner
2007-11-13 16:07     ` Thomas Gleixner
2007-11-13 17:47       ` Mark Lord
2007-11-15 16:32         ` [BUG] Strange 1-second pauses during Resume-from-RAM Mark Lord
2007-11-15 16:49           ` Ray Lee
2007-11-15 16:51             ` Mark Lord
2007-11-15 16:53               ` Mark Lord
2007-11-15 18:14           ` Pavel Machek
2007-11-15 17:31             ` Mark Lord
2007-11-15 19:34               ` Ingo Molnar
2007-11-15 19:36                 ` Ingo Molnar
2007-11-15 22:23                   ` Mark Lord
2007-11-16  5:55                     ` Ingo Molnar
2007-11-16  7:15                       ` Ingo Molnar
2007-11-16  8:21                         ` Ingo Molnar
2007-11-16 11:23                           ` Ingo Molnar [this message]
2007-11-16 11:53                             ` Mike Galbraith
2007-11-16 12:43                             ` Ingo Molnar
2007-11-16 12:58                               ` [patch] snd hda suspend latency: shorten codec read Ingo Molnar
2007-11-16 13:31                                 ` Rafael J. Wysocki
2007-11-16 14:21                                 ` Takashi Iwai
2007-11-16 19:06                         ` [BUG] Strange 1-second pauses during Resume-from-RAM Mark Lord
2007-11-16 18:35                     ` Mark Lord
2007-11-30 20:12                     ` Mark Lord
2007-11-30 12:56                   ` Jörn Engel
2007-11-30 13:35                     ` Ingo Molnar
2007-11-30 13:43                       ` Ingo Molnar
2007-11-30 18:35                         ` Jörn Engel
2007-11-30 18:46                           ` Ingo Molnar
2007-12-01 15:16                             ` Jörn Engel
2007-12-01 18:32                               ` Ingo Molnar
2007-12-01 20:47                                 ` Jörn Engel
2007-12-01 20:54                                   ` Ingo Molnar
2007-12-01 23:41                                     ` Jörn Engel
2007-12-02  8:56                                       ` Ingo Molnar
2007-12-02 11:31                                         ` Jörn Engel
2007-12-02 12:31                                           ` Jörn Engel
2007-12-02 13:57                                             ` Ingo Molnar
2007-12-02 14:46                                             ` Jörn Engel
2007-12-02 15:44                                               ` Ingo Molnar
2007-12-02 13:57                                           ` Ingo Molnar
2007-12-02 14:11                                             ` Jörn Engel
2007-12-02 15:47                                               ` Ingo Molnar
2007-12-02 19:55                                                 ` Jörn Engel
2007-12-02 20:07                                                   ` Ingo Molnar
2007-12-02 20:30                                                     ` Jörn Engel
2007-12-02 20:45                                                       ` Ingo Molnar
2007-12-02 21:08                                                         ` Jörn Engel
2007-12-02 21:10                                                         ` Jörn Engel
2007-12-02 21:19                                                           ` Ingo Molnar
2007-12-03  0:57                                                             ` Jörn Engel
2007-12-04  0:06                                                               ` Jörn Engel
2007-12-04  9:34                                                                 ` Ingo Molnar
2007-11-30 15:49                       ` Jörn Engel
2007-11-15 20:27               ` Rafael J. Wysocki
2007-11-18 16:10           ` Mark Lord
2007-11-18 16:21             ` Ingo Molnar
2007-11-18 17:37               ` Mark Lord
2007-11-13 17:54       ` [BUG] New Kernel Bugs Mark Lord
2007-11-13 22:46         ` Thomas Gleixner
2007-11-13 23:37           ` Mark Lord
2007-11-13 18:10       ` Russell King
2007-11-13 18:25         ` Alan Cox
2007-11-13 22:34           ` Russell King
2007-11-15 20:16             ` Ben Dooks
2007-11-13 15:21   ` Bartlomiej Zolnierkiewicz
2007-11-13 15:33   ` James Bottomley
2007-11-13 16:43     ` Randy Dunlap
2007-11-13 17:46     ` Martin Bligh
2007-11-13 18:47       ` Andrew Morton
2007-11-14  5:07       ` David Miller
2007-11-13 15:36   ` Alan Cox
2007-11-13 17:49   ` Jan Kara
2007-11-13 18:04   ` Russell King
2007-11-14 12:46   ` Jiri Kosina
2007-11-14 13:24   ` Pavel Machek
2007-11-14 14:14     ` Fabio Comolli
2007-11-14 19:52     ` Russell King

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20071116112317.GA12724@elte.hu \
    --to=mingo@elte.hu \
    --cc=akpm@linux-foundation.org \
    --cc=len.brown@intel.com \
    --cc=lenb@kernel.org \
    --cc=liml@rtr.ca \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pm@lists.linux-foundation.org \
    --cc=lkml@rtr.ca \
    --cc=pavel@ucw.cz \
    --cc=rjw@sisk.pl \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).