From: Peter Zijlstra <peterz@infradead.org> To: Eduardo Valentin <eduval@amazon.com> Cc: Anchal Agarwal <anchalag@amazon.com>, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, x86@kernel.org, boris.ostrovsky@oracle.com, jgross@suse.com, linux-pm@vger.kernel.org, linux-mm@kvack.org, kamatam@amazon.com, sstabellini@kernel.org, konrad.wilk@oracle.co, roger.pau@citrix.com, axboe@kernel.dk, davem@davemloft.net, rjw@rjwysocki.net, len.brown@intel.com, pavel@ucw.cz, sblbir@amazon.com, xen-devel@lists.xenproject.org, vkuznets@redhat.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com, dwmw@amazon.co.uk, fllinden@amaozn.com Subject: Re: [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation Date: Mon, 13 Jan 2020 11:16:09 +0100 [thread overview] Message-ID: <20200113101609.GT2844@hirez.programming.kicks-ass.net> (raw) In-Reply-To: <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> On Fri, Jan 10, 2020 at 07:35:20AM -0800, Eduardo Valentin wrote: > Hey Peter, > > On Wed, Jan 08, 2020 at 11:50:11AM +0100, Peter Zijlstra wrote: > > On Tue, Jan 07, 2020 at 11:45:26PM +0000, Anchal Agarwal wrote: > > > From: Eduardo Valentin <eduval@amazon.com> > > > > > > System instability are seen during resume from hibernation when system > > > is under heavy CPU load. This is due to the lack of update of sched > > > clock data, and the scheduler would then think that heavy CPU hog > > > tasks need more time in CPU, causing the system to freeze > > > during the unfreezing of tasks. For example, threaded irqs, > > > and kernel processes servicing network interface may be delayed > > > for several tens of seconds, causing the system to be unreachable. > > > > > The fix for this situation is to mark the sched clock as unstable > > > as early as possible in the resume path, leaving it unstable > > > for the duration of the resume process. This will force the > > > scheduler to attempt to align the sched clock across CPUs using > > > the delta with time of day, updating sched clock data. In a post > > > hibernation event, we can then mark the sched clock as stable > > > again, avoiding unnecessary syncs with time of day on systems > > > in which TSC is reliable. > > > > This makes no frigging sense what so bloody ever. If the clock is > > stable, we don't care about sched_clock_data. When it is stable you get > > a linear function of the TSC without complicated bits on. > > > > When it is unstable, only then do we care about the sched_clock_data. > > > > Yeah, maybe what is not clear here is that we covering for situation > where clock stability changes over time, e.g. at regular boot clock is > stable, hibernation happens, then restore happens in a non-stable clock. Still confused, who marks the thing unstable? The patch seems to suggest you do yourself, but it is not at all clear why. If TSC really is unstable, then it needs to remain unstable. If the TSC really is stable then there is no point in marking is unstable. Either way something is off, and you're not telling me what. > > > Reviewed-by: Erik Quanstrom <quanstro@amazon.com> > > > Reviewed-by: Frank van der Linden <fllinden@amazon.com> > > > Reviewed-by: Balbir Singh <sblbir@amazon.com> > > > Reviewed-by: Munehisa Kamata <kamatam@amazon.com> > > > Tested-by: Anchal Agarwal <anchalag@amazon.com> > > > Signed-off-by: Eduardo Valentin <eduval@amazon.com> > > > --- > > > > NAK, the code very much relies on never getting marked stable again > > after it gets set to unstable. > > > > Well actually, at the PM_POST_HIBERNATION, we do the check and set stable if > known to be stable. > > The issue only really happens during the restoration path under scheduling pressure, > which takes forever to finish, as described in the commit. > > Do you see a better solution for this issue? I still have no clue what your actual problem is. You say scheduling goes wobbly because sched_clock_data is stale, but when stable that doesn't matter. So what is the actual problem?
WARNING: multiple messages have this Message-ID (diff)
From: Peter Zijlstra <peterz@infradead.org> To: Eduardo Valentin <eduval@amazon.com> Cc: konrad.wilk@oracle.co, x86@kernel.org, linux-mm@kvack.org, axboe@kernel.dk, pavel@ucw.cz, hpa@zytor.com, boris.ostrovsky@oracle.com, sstabellini@kernel.org, fllinden@amaozn.com, kamatam@amazon.com, mingo@redhat.com, xen-devel@lists.xenproject.org, sblbir@amazon.com, len.brown@intel.com, linux-pm@vger.kernel.org, Anchal Agarwal <anchalag@amazon.com>, bp@alien8.de, tglx@linutronix.de, jgross@suse.com, netdev@vger.kernel.org, Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com, rjw@rjwysocki.net, linux-kernel@vger.kernel.org, vkuznets@redhat.com, davem@davemloft.net, dwmw@amazon.co.uk, roger.pau@citrix.com Subject: Re: [Xen-devel] [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation Date: Mon, 13 Jan 2020 11:16:09 +0100 [thread overview] Message-ID: <20200113101609.GT2844@hirez.programming.kicks-ass.net> (raw) In-Reply-To: <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> On Fri, Jan 10, 2020 at 07:35:20AM -0800, Eduardo Valentin wrote: > Hey Peter, > > On Wed, Jan 08, 2020 at 11:50:11AM +0100, Peter Zijlstra wrote: > > On Tue, Jan 07, 2020 at 11:45:26PM +0000, Anchal Agarwal wrote: > > > From: Eduardo Valentin <eduval@amazon.com> > > > > > > System instability are seen during resume from hibernation when system > > > is under heavy CPU load. This is due to the lack of update of sched > > > clock data, and the scheduler would then think that heavy CPU hog > > > tasks need more time in CPU, causing the system to freeze > > > during the unfreezing of tasks. For example, threaded irqs, > > > and kernel processes servicing network interface may be delayed > > > for several tens of seconds, causing the system to be unreachable. > > > > > The fix for this situation is to mark the sched clock as unstable > > > as early as possible in the resume path, leaving it unstable > > > for the duration of the resume process. This will force the > > > scheduler to attempt to align the sched clock across CPUs using > > > the delta with time of day, updating sched clock data. In a post > > > hibernation event, we can then mark the sched clock as stable > > > again, avoiding unnecessary syncs with time of day on systems > > > in which TSC is reliable. > > > > This makes no frigging sense what so bloody ever. If the clock is > > stable, we don't care about sched_clock_data. When it is stable you get > > a linear function of the TSC without complicated bits on. > > > > When it is unstable, only then do we care about the sched_clock_data. > > > > Yeah, maybe what is not clear here is that we covering for situation > where clock stability changes over time, e.g. at regular boot clock is > stable, hibernation happens, then restore happens in a non-stable clock. Still confused, who marks the thing unstable? The patch seems to suggest you do yourself, but it is not at all clear why. If TSC really is unstable, then it needs to remain unstable. If the TSC really is stable then there is no point in marking is unstable. Either way something is off, and you're not telling me what. > > > Reviewed-by: Erik Quanstrom <quanstro@amazon.com> > > > Reviewed-by: Frank van der Linden <fllinden@amazon.com> > > > Reviewed-by: Balbir Singh <sblbir@amazon.com> > > > Reviewed-by: Munehisa Kamata <kamatam@amazon.com> > > > Tested-by: Anchal Agarwal <anchalag@amazon.com> > > > Signed-off-by: Eduardo Valentin <eduval@amazon.com> > > > --- > > > > NAK, the code very much relies on never getting marked stable again > > after it gets set to unstable. > > > > Well actually, at the PM_POST_HIBERNATION, we do the check and set stable if > known to be stable. > > The issue only really happens during the restoration path under scheduling pressure, > which takes forever to finish, as described in the commit. > > Do you see a better solution for this issue? I still have no clue what your actual problem is. You say scheduling goes wobbly because sched_clock_data is stale, but when stable that doesn't matter. So what is the actual problem? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
next prev parent reply other threads:[~2020-01-13 10:17 UTC|newest] Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-01-07 23:45 [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation Anchal Agarwal 2020-01-07 23:45 ` [Xen-devel] " Anchal Agarwal 2020-01-08 10:50 ` Peter Zijlstra 2020-01-08 10:50 ` [Xen-devel] " Peter Zijlstra 2020-01-10 15:35 ` Eduardo Valentin 2020-01-10 15:35 ` [Xen-devel] " Eduardo Valentin 2020-01-13 10:16 ` Peter Zijlstra [this message] 2020-01-13 10:16 ` Peter Zijlstra 2020-01-13 11:43 ` Singh, Balbir 2020-01-13 11:43 ` [Xen-devel] " Singh, Balbir 2020-01-13 11:43 ` Singh, Balbir 2020-01-13 11:48 ` Rafael J. Wysocki 2020-01-13 11:48 ` [Xen-devel] " Rafael J. Wysocki 2020-01-13 11:48 ` Rafael J. Wysocki 2020-01-13 12:42 ` Peter Zijlstra 2020-01-13 12:42 ` [Xen-devel] " Peter Zijlstra 2020-01-13 12:42 ` Peter Zijlstra 2020-01-13 21:50 ` Rafael J. Wysocki 2020-01-13 21:50 ` [Xen-devel] " Rafael J. Wysocki 2020-01-13 21:50 ` Rafael J. Wysocki 2020-01-13 23:30 ` Rafael J. Wysocki 2020-01-13 23:30 ` [Xen-devel] " Rafael J. Wysocki 2020-01-13 23:30 ` Rafael J. Wysocki 2020-01-14 19:29 ` Anchal Agarwal 2020-01-14 19:29 ` [Xen-devel] " Anchal Agarwal 2020-01-22 20:07 ` Anchal Agarwal 2020-01-22 20:07 ` [Xen-devel] " Anchal Agarwal 2020-01-23 16:27 ` Boris Ostrovsky 2020-01-23 16:27 ` [Xen-devel] " Boris Ostrovsky 2020-01-13 13:01 ` Andrew Cooper 2020-01-13 13:01 ` Andrew Cooper 2020-01-13 13:01 ` Andrew Cooper 2020-01-13 13:54 ` David Woodhouse 2020-01-13 13:54 ` David Woodhouse 2020-01-13 13:54 ` David Woodhouse 2020-01-13 15:02 ` Singh, Balbir 2020-01-13 15:02 ` Singh, Balbir 2020-01-13 15:02 ` Singh, Balbir
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200113101609.GT2844@hirez.programming.kicks-ass.net \ --to=peterz@infradead.org \ --cc=Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com \ --cc=anchalag@amazon.com \ --cc=axboe@kernel.dk \ --cc=boris.ostrovsky@oracle.com \ --cc=bp@alien8.de \ --cc=davem@davemloft.net \ --cc=dwmw@amazon.co.uk \ --cc=eduval@amazon.com \ --cc=fllinden@amaozn.com \ --cc=hpa@zytor.com \ --cc=jgross@suse.com \ --cc=kamatam@amazon.com \ --cc=konrad.wilk@oracle.co \ --cc=len.brown@intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-pm@vger.kernel.org \ --cc=mingo@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=pavel@ucw.cz \ --cc=rjw@rjwysocki.net \ --cc=roger.pau@citrix.com \ --cc=sblbir@amazon.com \ --cc=sstabellini@kernel.org \ --cc=tglx@linutronix.de \ --cc=vkuznets@redhat.com \ --cc=x86@kernel.org \ --cc=xen-devel@lists.xenproject.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.