All of lore.kernel.org
 help / color / mirror / Atom feed
From: Len Brown <lenb@kernel.org>
To: Dave Chinner <david@fromorbit.com>
Cc: NeilBrown <neilb@suse.de>,
	One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	Ming Lei <tom.leiming@gmail.com>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	Linux PM List <linux-pm@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Len Brown <len.brown@intel.com>
Subject: Re: [PATCH 1/1] suspend: delete sys_sync()
Date: Sat, 20 Jun 2015 01:26:22 -0400	[thread overview]
Message-ID: <CAJvTdKn1kJ=2knUnXgO+6iK6m7Ns6q1aLT2RCeERU5+gJnzmqw@mail.gmail.com> (raw)
In-Reply-To: <20150619230720.GB16870@dastard>

>> Putting a function trace on sys_sync and executing sync manually,
>> I was able to see it take 100ms,
>> though function trace itself could be contributing to that...
>
> It would seem that way - you need to get the traces to dump to
> something that has no sync overhead....

I don't think that ftrace buffers have sync overhead --
at least not until they are copied into files.

>> Curiously, in another run, sync ran at 15ms, but sd suspend exploded to 300ms.
>> I've seen that in some other results.  Sometimes sync if fast, but sd
>> then more than makes up for it by being slow:-(
>
> Oh, I see that too. Normally That's because the filesystem hasn't
> been told to enter an idle state and so is doing metadata writeback
> IO after the sync. When that happens the sd suspend has wait for
> request queues to drain, IO to complete and device caches to flush.
> This simply cannot be avoided because suspend never tells the
> filesytems to enter an idle state....

I captured a trace of a slow sd suspend.
Apparently, it does two operations -- first  a sync_cache,
then a stop operation.  The sync was fast.   The stop command
was where all the time went.

I'll look at a more modern drive on the same system next week,
just for comparison.

> i.e. remember what I said initially in this thread about suspend
> actually needing to freeze filesystems, not just sync them?

I think with the complexity of file systems and the underlying
devices, yes, we need to think about how to efficiently
and safely suspend/resume them.

But sys_sync is too expensive to have hard-coded in the kernel suspend path.
Some machines can suspend and resume in under 10ms -- they
absolutely do not want sys_sync() hard-coded in the suspend path.

>> FYI,
>> I ran analyze_suspend.py -x2
>> from current directory /tmp, which is mounted on tmpfs,
>> but still found the 2nd sync was very slow -- 200ms
>> vs 6 - 20 ms for the sync preceding the 1st suspend.
>
> So where did that time go? As I pointed out previously, function
> trace will only tell us if the delay is data writeback or not. We
> seem to have confirmed that the delay is, indeed, writeback of dirty
> data. Now we need to identify what the dirty data belongs to: we
> need to trace individual writeback events to see what dirty inodes
> are actually being written.

I expect that analyze_suspend.py is moving data around between
the back-to-back suspends when the -x2 option is used -- will look into it.

thanks,
Len Brown, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

WARNING: multiple messages have this Message-ID (diff)
From: Len Brown <lenb@kernel.org>
To: Dave Chinner <david@fromorbit.com>
Cc: NeilBrown <neilb@suse.de>,
	One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	Ming Lei <tom.leiming@gmail.com>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	Linux PM List <linux-pm@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Len Brown <len.brown@intel.com>
Subject: Re: [PATCH 1/1] suspend: delete sys_sync()
Date: Sat, 20 Jun 2015 01:26:22 -0400	[thread overview]
Message-ID: <CAJvTdKn1kJ=2knUnXgO+6iK6m7Ns6q1aLT2RCeERU5+gJnzmqw@mail.gmail.com> (raw)
In-Reply-To: <20150619230720.GB16870@dastard>

>> Putting a function trace on sys_sync and executing sync manually,
>> I was able to see it take 100ms,
>> though function trace itself could be contributing to that...
>
> It would seem that way - you need to get the traces to dump to
> something that has no sync overhead....

I don't think that ftrace buffers have sync overhead --
at least not until they are copied into files.

>> Curiously, in another run, sync ran at 15ms, but sd suspend exploded to 300ms.
>> I've seen that in some other results.  Sometimes sync if fast, but sd
>> then more than makes up for it by being slow:-(
>
> Oh, I see that too. Normally That's because the filesystem hasn't
> been told to enter an idle state and so is doing metadata writeback
> IO after the sync. When that happens the sd suspend has wait for
> request queues to drain, IO to complete and device caches to flush.
> This simply cannot be avoided because suspend never tells the
> filesytems to enter an idle state....

I captured a trace of a slow sd suspend.
Apparently, it does two operations -- first  a sync_cache,
then a stop operation.  The sync was fast.   The stop command
was where all the time went.

I'll look at a more modern drive on the same system next week,
just for comparison.

> i.e. remember what I said initially in this thread about suspend
> actually needing to freeze filesystems, not just sync them?

I think with the complexity of file systems and the underlying
devices, yes, we need to think about how to efficiently
and safely suspend/resume them.

But sys_sync is too expensive to have hard-coded in the kernel suspend path.
Some machines can suspend and resume in under 10ms -- they
absolutely do not want sys_sync() hard-coded in the suspend path.

>> FYI,
>> I ran analyze_suspend.py -x2
>> from current directory /tmp, which is mounted on tmpfs,
>> but still found the 2nd sync was very slow -- 200ms
>> vs 6 - 20 ms for the sync preceding the 1st suspend.
>
> So where did that time go? As I pointed out previously, function
> trace will only tell us if the delay is data writeback or not. We
> seem to have confirmed that the delay is, indeed, writeback of dirty
> data. Now we need to identify what the dirty data belongs to: we
> need to trace individual writeback events to see what dirty inodes
> are actually being written.

I expect that analyze_suspend.py is moving data around between
the back-to-back suspends when the -x2 option is used -- will look into it.

thanks,
Len Brown, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-pm" in

  reply	other threads:[~2015-06-20  5:26 UTC|newest]

Thread overview: 77+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-05-08  7:08 [PATCH 1/1] suspend: delete sys_sync() Len Brown
2015-05-08 14:34 ` Alan Stern
2015-05-08 14:34   ` Alan Stern
2015-05-08 16:36   ` Len Brown
2015-05-08 19:13     ` One Thousand Gnomes
2015-05-08 19:32       ` Len Brown
2015-05-08 19:52         ` One Thousand Gnomes
2015-05-08 20:39           ` Rafael J. Wysocki
2015-05-08 20:30         ` Rafael J. Wysocki
2015-05-09 19:59           ` Alan Stern
2015-05-09 20:25             ` Henrique de Moraes Holschuh
2015-05-11 20:34               ` Len Brown
2015-05-12  6:11                 ` Oliver Neukum
2015-06-25 17:11                 ` Henrique de Moraes Holschuh
2015-06-30 20:04                   ` Len Brown
2015-07-01 12:21                     ` Henrique de Moraes Holschuh
2015-07-02  3:07                       ` Len Brown
2015-07-03  1:42                         ` Dave Chinner
2015-07-04  1:03                           ` Rafael J. Wysocki
2015-07-04  8:50                             ` Geert Uytterhoeven
2015-07-05 23:25                               ` Rafael J. Wysocki
2015-07-04 14:19                             ` Alan Stern
2015-07-05 23:28                               ` Rafael J. Wysocki
2015-07-06 11:06                                 ` Pavel Machek
2015-07-06 13:59                                   ` Rafael J. Wysocki
2015-07-07 10:25                                     ` Pavel Machek
2015-07-07 12:22                                       ` Rafael J. Wysocki
2015-07-06  0:06                             ` Dave Chinner
2015-07-06 11:11                               ` Pavel Machek
2015-07-06 13:52                               ` Rafael J. Wysocki
2015-07-07  1:17                                 ` Dave Chinner
2015-07-07 12:14                                   ` Rafael J. Wysocki
2015-07-07 13:16                                     ` Oliver Neukum
2015-07-07 14:32                                       ` Rafael J. Wysocki
2015-07-07 14:38                                         ` Oliver Neukum
2015-07-07 15:03                                           ` Alan Stern
2015-07-07 22:20                                             ` Rafael J. Wysocki
2015-07-08 11:20                                               ` Pavel Machek
2015-07-08 14:40                                                 ` Alan Stern
2015-07-08 22:04                                                   ` Rafael J. Wysocki
2015-07-07 22:11                                           ` Rafael J. Wysocki
2015-07-08  7:51                                             ` Oliver Neukum
2015-07-08 22:03                                               ` Rafael J. Wysocki
2015-07-09  7:32                                                 ` Oliver Neukum
2015-07-09 23:22                                                   ` Rafael J. Wysocki
2015-08-04 19:54                                                     ` Pavel Machek
2015-07-08 11:17                                         ` Pavel Machek
2015-07-07 13:42                                   ` Takashi Iwai
2015-07-06 10:15                             ` Ming Lei
2015-07-06 10:03           ` Pavel Machek
2015-05-11  1:44 ` Dave Chinner
2015-05-11 20:22   ` Len Brown
2015-05-12 22:34     ` Dave Chinner
2015-05-13 23:22   ` NeilBrown
2015-05-14 23:54     ` Dave Chinner
2015-05-15  0:34       ` Rafael J. Wysocki
2015-05-15  0:40         ` Ming Lei
2015-05-15  0:59           ` Rafael J. Wysocki
2015-05-15  5:13             ` Ming Lei
2015-05-15 10:35             ` One Thousand Gnomes
2015-05-18  1:57               ` NeilBrown
     [not found]                 ` <CAJvTdKn_0EZ0ZuqO2e4+ExD8kFWcy78fse4zHr3uFZODOroXEg@mail.gmail.com>
2015-06-19  1:09                   ` Dave Chinner
2015-06-19  2:35                     ` Len Brown
2015-06-19  4:31                       ` Dave Chinner
2015-06-19  6:34                         ` Len Brown
2015-06-19 23:07                           ` Dave Chinner
2015-06-19 23:07                             ` Dave Chinner
2015-06-20  5:26                             ` Len Brown [this message]
2015-06-20  5:26                               ` Len Brown
2015-05-15  1:04       ` NeilBrown
2015-05-15 14:20         ` Alan Stern
2015-05-15 14:20           ` Alan Stern
2015-05-15 14:32           ` Alan Stern
2015-05-15 14:32             ` Alan Stern
2015-05-15 14:19       ` Alan Stern
2015-05-15 14:19         ` Alan Stern
2015-07-06 10:07   ` Pavel Machek

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAJvTdKn1kJ=2knUnXgO+6iK6m7Ns6q1aLT2RCeERU5+gJnzmqw@mail.gmail.com' \
    --to=lenb@kernel.org \
    --cc=david@fromorbit.com \
    --cc=gnomes@lxorguk.ukuu.org.uk \
    --cc=len.brown@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pm@vger.kernel.org \
    --cc=neilb@suse.de \
    --cc=rafael@kernel.org \
    --cc=rjw@rjwysocki.net \
    --cc=tom.leiming@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.