Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Linus Torvalds]

On Wed, Apr 25, 2018 at 6:45 AM, tip-bot for Thomas Gleixner
<tipbot@zytor.com> wrote:
>
> As stated in the pull request for the unification of CLOCK_MONOTONIC and
> CLOCK_BOOTTIME, it was clear that we might have to revert the change.

I'm ok with the revert., but I do want the revert message to talk
about *exactly* what broke.

I don't see the reported watchdog things, and I run systemd and I ran
these patches.

So it's not just "systemd uses this for a watchdog". It must be
something specific that made it trigger for some people and not
others, and it's that specific thing that we should document so that
we know why this simplification failed.

                 Linus

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Thomas Gleixner]

On Wed, 25 Apr 2018, Linus Torvalds wrote:
> On Wed, Apr 25, 2018 at 6:45 AM, tip-bot for Thomas Gleixner
> <tipbot@zytor.com> wrote:
> >
> > As stated in the pull request for the unification of CLOCK_MONOTONIC and
> > CLOCK_BOOTTIME, it was clear that we might have to revert the change.
> 
> I'm ok with the revert., but I do want the revert message to talk
> about *exactly* what broke.
> 
> I don't see the reported watchdog things, and I run systemd and I ran
> these patches.

With suspend/resume?

> So it's not just "systemd uses this for a watchdog". It must be
> something specific that made it trigger for some people and not
> others, and it's that specific thing that we should document so that
> we know why this simplification failed.

Fair enough. Do the reporters have more information or do I have to chase
it down myself?

Thanks,

	tglx

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Linus Torvalds]

On Wed, Apr 25, 2018 at 12:09 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Wed, 25 Apr 2018, Linus Torvalds wrote:
>>
>> I don't see the reported watchdog things, and I run systemd and I ran
>> these patches.
>
> With suspend/resume?

Yes. I run them on my laptop, and while I don't do a _lot_ of
suspend/resume, I do some. My laptop right now is running a "just
after rc1" kernel, and has been for the last week, and has a couple of
suspends in that time. Type "closed my laptop, took it elsewhere,
opened it again", so more than a few minutes.

> Fair enough. Do the reporters have more information or do I have to chase
> it down myself?

I think reporters should just say what distro they are running, and
systemd versions etc, and how they debugged this. Maybe the pattern is
obvious.

For the record, I just run plain F27, obviously with my own kernel
(and a tweaked config, but it approaches just being a "make
localmodconfig" of the distro config).

               Linus

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

On Wed 2018-04-25 12:41:23, Linus Torvalds wrote:
> On Wed, Apr 25, 2018 at 12:09 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > On Wed, 25 Apr 2018, Linus Torvalds wrote:
> >>
> >> I don't see the reported watchdog things, and I run systemd and I ran
> >> these patches.
> >
> > With suspend/resume?
> 
> Yes. I run them on my laptop, and while I don't do a _lot_ of
> suspend/resume, I do some. My laptop right now is running a "just
> after rc1" kernel, and has been for the last week, and has a couple of
> suspends in that time. Type "closed my laptop, took it elsewhere,
> opened it again", so more than a few minutes.
> 
> > Fair enough. Do the reporters have more information or do I have to chase
> > it down myself?
> 
> I think reporters should just say what distro they are running, and
> systemd versions etc, and how they debugged this. Maybe the pattern is
> obvious.
> 
> For the record, I just run plain F27, obviously with my own kernel
> (and a tweaked config, but it approaches just being a "make
> localmodconfig" of the distro config).

So... I do have debian 8.10. With 4.17-rc1, I get "networking
disabled" message from network manager (and broken network), about
half of the time.

But there's more visible symptom: suspend/resume with 4.16, machine is
okay after resume. With 4.17-rc1, just after resume, display starts
dimming and screensaver will kick in. MATE desktop.

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Rafael J. Wysocki]

On Wednesday, April 25, 2018 9:41:23 PM CEST Linus Torvalds wrote:
> On Wed, Apr 25, 2018 at 12:09 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > On Wed, 25 Apr 2018, Linus Torvalds wrote:
> >>
> >> I don't see the reported watchdog things, and I run systemd and I ran
> >> these patches.
> >
> > With suspend/resume?
> 
> Yes. I run them on my laptop, and while I don't do a _lot_ of
> suspend/resume, I do some. My laptop right now is running a "just
> after rc1" kernel, and has been for the last week, and has a couple of
> suspends in that time. Type "closed my laptop, took it elsewhere,
> opened it again", so more than a few minutes.
> 
> > Fair enough. Do the reporters have more information or do I have to chase
> > it down myself?
> 
> I think reporters should just say what distro they are running, and
> systemd versions etc, and how they debugged this. Maybe the pattern is
> obvious.
> 
> For the record, I just run plain F27, obviously with my own kernel
> (and a tweaked config, but it approaches just being a "make
> localmodconfig" of the distro config).

FWIW, I run (64-bit) OpenSUSE 42.3 on the failing machine (Acer Aspire S5) with
my own kernel (custom config based on the distro one).

The problem here is that the machine doesn't resume from S2R.  It appears to
just hang somewhere around where we return to user space.  However, to trigger
the issue it needs to stay in suspend for several minutes (not sure how many
exactly).  If it is just back-to-back suspend/resume, it will resume correctly.

Thanks,
Rafael

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Rafael J. Wysocki]

On Wednesday, April 25, 2018 9:09:29 PM CEST Thomas Gleixner wrote:
> On Wed, 25 Apr 2018, Linus Torvalds wrote:
> > On Wed, Apr 25, 2018 at 6:45 AM, tip-bot for Thomas Gleixner
> > <tipbot@zytor.com> wrote:
> > >
> > > As stated in the pull request for the unification of CLOCK_MONOTONIC and
> > > CLOCK_BOOTTIME, it was clear that we might have to revert the change.
> > 
> > I'm ok with the revert., but I do want the revert message to talk
> > about *exactly* what broke.
> > 
> > I don't see the reported watchdog things, and I run systemd and I ran
> > these patches.
> 
> With suspend/resume?
> 
> > So it's not just "systemd uses this for a watchdog". It must be
> > something specific that made it trigger for some people and not
> > others, and it's that specific thing that we should document so that
> > we know why this simplification failed.
> 
> Fair enough. Do the reporters have more information or do I have to chase
> it down myself?

Well, if CLOCK_MONOTONIC did not count when the system was suspended and now
it does that due to the change in question, then this is a significant
difference in behavior likely to confuse things even in principle.

While it may not confuse things on some systems after all, does this really
matter?

Thanks,
Rafael

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Steven Rostedt]

On Wed, 25 Apr 2018 22:57:12 +0200
"Rafael J. Wysocki" <rjw@rjwysocki.net> wrote:

> FWIW, I run (64-bit) OpenSUSE 42.3 on the failing machine (Acer Aspire S5) with
> my own kernel (custom config based on the distro one).
> 
> The problem here is that the machine doesn't resume from S2R.  It appears to
> just hang somewhere around where we return to user space.  However, to trigger
> the issue it needs to stay in suspend for several minutes (not sure how many
> exactly).  If it is just back-to-back suspend/resume, it will resume correctly.

I guess the question that needs to be asked is, have you tried a kernel
with this commit reverted and it works? I'd play a bit going back and
forth between two kernels, one with the commit and one without and no
other differences, to make sure that is indeed the problem commit.

-- Steve

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Rafael J. Wysocki]

On Thursday, April 26, 2018 12:41:40 AM CEST Steven Rostedt wrote:
> On Wed, 25 Apr 2018 22:57:12 +0200
> "Rafael J. Wysocki" <rjw@rjwysocki.net> wrote:
> 
> > FWIW, I run (64-bit) OpenSUSE 42.3 on the failing machine (Acer Aspire S5) with
> > my own kernel (custom config based on the distro one).
> > 
> > The problem here is that the machine doesn't resume from S2R.  It appears to
> > just hang somewhere around where we return to user space.  However, to trigger
> > the issue it needs to stay in suspend for several minutes (not sure how many
> > exactly).  If it is just back-to-back suspend/resume, it will resume correctly.
> 
> I guess the question that needs to be asked is, have you tried a kernel
> with this commit reverted and it works?

Yes, I have.

> I'd play a bit going back and
> forth between two kernels, one with the commit and one without and no
> other differences, to make sure that is indeed the problem commit.

I've done that.

It is the problem commit.

Thanks,
Rafael

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Rafael J. Wysocki]

On Wednesday, April 25, 2018 10:19:08 PM CEST Pavel Machek wrote:
> 
> --J2SCkAp4GZ/dPZZf
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> Content-Transfer-Encoding: quoted-printable
> 
> On Wed 2018-04-25 12:41:23, Linus Torvalds wrote:
> > On Wed, Apr 25, 2018 at 12:09 PM, Thomas Gleixner <tglx@linutronix.de> wr=
> ote:
> > > On Wed, 25 Apr 2018, Linus Torvalds wrote:
> > >>
> > >> I don't see the reported watchdog things, and I run systemd and I ran
> > >> these patches.
> > >
> > > With suspend/resume?
> >=20
> > Yes. I run them on my laptop, and while I don't do a _lot_ of
> > suspend/resume, I do some. My laptop right now is running a "just
> > after rc1" kernel, and has been for the last week, and has a couple of
> > suspends in that time. Type "closed my laptop, took it elsewhere,
> > opened it again", so more than a few minutes.
> >=20
> > > Fair enough. Do the reporters have more information or do I have to cha=
> se
> > > it down myself?
> >=20
> > I think reporters should just say what distro they are running, and
> > systemd versions etc, and how they debugged this. Maybe the pattern is
> > obvious.
> >=20
> > For the record, I just run plain F27, obviously with my own kernel
> > (and a tweaked config, but it approaches just being a "make
> > localmodconfig" of the distro config).
> 
> So... I do have debian 8.10.

Does that use systemd at all?

> With 4.17-rc1, I get "networking
> disabled" message from network manager (and broken network), about
> half of the time.
> 
> But there's more visible symptom: suspend/resume with 4.16, machine is
> okay after resume. With 4.17-rc1, just after resume, display starts
> dimming and screensaver will kick in. MATE desktop.

Yeah, why not.  It thinks that you've been inactive for the duration of system
sleep. :-)

Thanks,
Rafael

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1184 bytes --]

Hi!

> > > For the record, I just run plain F27, obviously with my own kernel
> > > (and a tweaked config, but it approaches just being a "make
> > > localmodconfig" of the distro config).
> > 
> > So... I do have debian 8.10.
> 
> Does that use systemd at all?

pavel@duo:/data/l/linux$ systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ
-SECCOMP -APPARMOR

> > With 4.17-rc1, I get "networking
> > disabled" message from network manager (and broken network), about
> > half of the time.
> > 
> > But there's more visible symptom: suspend/resume with 4.16, machine is
> > okay after resume. With 4.17-rc1, just after resume, display starts
> > dimming and screensaver will kick in. MATE desktop.
> 
> Yeah, why not.  It thinks that you've been inactive for the duration of system
> sleep. :-)

Well, its rather annoying. "Wake up!" (hitting a key) "OOh, I'm slowly
waking up.." (few seconds, wakeup is not instant) "and now I'm going
back to sleep".

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Rafael J. Wysocki]

On Thursday, April 26, 2018 9:19:25 AM CEST Pavel Machek wrote:
> Hi!
> 
> > > > For the record, I just run plain F27, obviously with my own kernel
> > > > (and a tweaked config, but it approaches just being a "make
> > > > localmodconfig" of the distro config).
> > > 
> > > So... I do have debian 8.10.
> > 
> > Does that use systemd at all?
> 
> pavel@duo:/data/l/linux$ systemd --version
> systemd 215
> +PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ
> -SECCOMP -APPARMOR

OK
 
> > > With 4.17-rc1, I get "networking
> > > disabled" message from network manager (and broken network), about
> > > half of the time.

I'm not sure if NM is affected by systemd, though.

> > > But there's more visible symptom: suspend/resume with 4.16, machine is
> > > okay after resume. With 4.17-rc1, just after resume, display starts
> > > dimming and screensaver will kick in. MATE desktop.
> > 
> > Yeah, why not.  It thinks that you've been inactive for the duration of system
> > sleep. :-)
> 
> Well, its rather annoying.

Yes, it is, no question about that.

> "Wake up!" (hitting a key) "OOh, I'm slowly
> waking up.." (few seconds, wakeup is not instant) "and now I'm going
> back to sleep".

I just wanted to say that you could expect that if the clock used by it
was counting over suspend/resume.

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Rafael J. Wysocki]

On Wednesday, April 25, 2018 11:28:27 PM CEST Rafael J. Wysocki wrote:
> On Wednesday, April 25, 2018 9:09:29 PM CEST Thomas Gleixner wrote:
> > On Wed, 25 Apr 2018, Linus Torvalds wrote:
> > > On Wed, Apr 25, 2018 at 6:45 AM, tip-bot for Thomas Gleixner
> > > <tipbot@zytor.com> wrote:
> > > >
> > > > As stated in the pull request for the unification of CLOCK_MONOTONIC and
> > > > CLOCK_BOOTTIME, it was clear that we might have to revert the change.
> > > 
> > > I'm ok with the revert., but I do want the revert message to talk
> > > about *exactly* what broke.
> > > 
> > > I don't see the reported watchdog things, and I run systemd and I ran
> > > these patches.
> > 
> > With suspend/resume?
> > 
> > > So it's not just "systemd uses this for a watchdog". It must be
> > > something specific that made it trigger for some people and not
> > > others, and it's that specific thing that we should document so that
> > > we know why this simplification failed.
> > 
> > Fair enough. Do the reporters have more information or do I have to chase
> > it down myself?
> 
> Well, if CLOCK_MONOTONIC did not count when the system was suspended and now
> it does that due to the change in question, then this is a significant
> difference in behavior likely to confuse things even in principle.

FWIW, here's a list of issues reported and narrowed down to the commit
in question so far I could find:

* systemd kills daemons on resume, after >WatchdogSec seconds
  of suspending (Genki Sky).  [Verified that that's because systemd uses
  CLOCK_MONOTONIC and expects it to not include the suspend time.]

* systemd-journald misbehaves after resume:
  systemd-journald[7266]: File /var/log/journal/016627c3c4784cd4812d4b7e96a34226/system.journal corrupted or uncleanly shut down, renaming and replacing.
  (	Mike Galbraith).

* NetworkManager reports "networking disabled" and networking is broken
  after resume 50% of the time (Pavel).  [May be because of systemd.]

* MATE desktop dims the display and starts the screensaver right after
  system resume (Pavel).

* Full system hang during resume (me).  [May be due to systemd or NM or both.]

I'm not sure if all of the above can be connected to one specific thing (other
than the fact that things get confused if they expect CLOCK_MONOTONIC to not
include system suspend time).

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Thomas Gleixner]

On Thu, 26 Apr 2018, Rafael J. Wysocki wrote:
> FWIW, here's a list of issues reported and narrowed down to the commit
> in question so far I could find:
> 
> * systemd kills daemons on resume, after >WatchdogSec seconds
>   of suspending (Genki Sky).  [Verified that that's because systemd uses
>   CLOCK_MONOTONIC and expects it to not include the suspend time.]
> 
> * systemd-journald misbehaves after resume:
>   systemd-journald[7266]: File /var/log/journal/016627c3c4784cd4812d4b7e96a34226/system.journal corrupted or uncleanly shut down, renaming and replacing.
>   (	Mike Galbraith).
> 
> * NetworkManager reports "networking disabled" and networking is broken
>   after resume 50% of the time (Pavel).  [May be because of systemd.]
> 
> * MATE desktop dims the display and starts the screensaver right after
>   system resume (Pavel).
> 
> * Full system hang during resume (me).  [May be due to systemd or NM or both.]
> 
> I'm not sure if all of the above can be connected to one specific thing (other
> than the fact that things get confused if they expect CLOCK_MONOTONIC to not
> include system suspend time).

Thanks for collecting the info. I'll amend the changelog with those.

Thanks,

	tglx

Re: [tip:timers/urgent] Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2011 bytes --]

On Wed 2018-04-25 06:45:50, tip-bot for Thomas Gleixner wrote:
> Commit-ID:  757d5a9b3052a4eebd029f207f1e1df59bef7b7d
> Gitweb:     https://git.kernel.org/tip/757d5a9b3052a4eebd029f207f1e1df59bef7b7d
> Author:     Thomas Gleixner <tglx@linutronix.de>
> AuthorDate: Wed, 25 Apr 2018 15:33:38 +0200
> Committer:  Thomas Gleixner <tglx@linutronix.de>
> CommitDate: Wed, 25 Apr 2018 15:35:08 +0200
> 
> Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME
> 
> Revert commits
> 
> 92af4dcb4e1c ("tracing: Unify the "boot" and "mono" tracing clocks")
> 127bfa5f4342 ("hrtimer: Unify MONOTONIC and BOOTTIME clock behavior")
> 7250a4047aa6 ("posix-timers: Unify MONOTONIC and BOOTTIME clock behavior")
> d6c7270e913d ("timekeeping: Remove boot time specific code")
> f2d6fdbfd238 ("Input: Evdev - unify MONOTONIC and BOOTTIME clock behavior")
> d6ed449afdb3 ("timekeeping: Make the MONOTONIC clock behave like the BOOTTIME clock")
> 72199320d49d ("timekeeping: Add the new CLOCK_MONOTONIC_ACTIVE clock")
> 
> As stated in the pull request for the unification of CLOCK_MONOTONIC and
> CLOCK_BOOTTIME, it was clear that we might have to revert the change.
> 
> As reported by several folks systemd and other applications rely on the
> documented behaviour of CLOCK_MONOTONIC on Linux and break with the above
> changes. After resume daemons time out and other timeout related issues are
> observed.
> 
> It's sad, that these problems were neither catched in -next nor by those
> folks who expressed interest in this change.
> 
> Reported-by: Rafael J. Wysocki <rjw@rjwysocki.net>
> Reported-by: Genki Sky <sky@genki.is>,

Tested-by: Pavel Machek <pavel@ucw.cz>

Both screensaver problem and network problems are gone on X60 after I
applied the revert. (I did not do too many tests, but it was normally
too easy to reproduce.)

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]