From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31492C33CB2 for ; Tue, 14 Jan 2020 19:30:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 007C524670 for ; Tue, 14 Jan 2020 19:30:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="scxTRtqw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728847AbgANTaa (ORCPT ); Tue, 14 Jan 2020 14:30:30 -0500 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:28975 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726491AbgANTa3 (ORCPT ); Tue, 14 Jan 2020 14:30:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1579030228; x=1610566228; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=kERWQcPBD6U7cttKEBKCe+VLCzWuIvbuKzU3wI+3unU=; b=scxTRtqwvDG4gt1n++qHyLf4X19Tko65fdiRcC7WpWD5MVHiQtJtdnzo OThiQrtjSpn3uQIfrrGpeWVu2SectF68SmPqGQr1KFv8quvJLK4imwmNJ kX9anqxYP/n5V+uRG0u9NzjlsGPPJshzWE9wFIQmGnkWCFhsuPnEyPSx/ I=; IronPort-SDR: 1XWTl0ZFPeAlOX/4U8kY6SYcBuV6mXXPHzXHV+KPO++vV8ZqU64Zy3g8i6YkQ3MNzJ1c7HsLNp 881qx3hEFFTA== X-IronPort-AV: E=Sophos;i="5.70,319,1574121600"; d="scan'208";a="18710229" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1a-807d4a99.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 14 Jan 2020 19:30:15 +0000 Received: from EX13MTAUWB001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1a-807d4a99.us-east-1.amazon.com (Postfix) with ESMTPS id F0146A29E9; Tue, 14 Jan 2020 19:30:07 +0000 (UTC) Received: from EX13D10UWB004.ant.amazon.com (10.43.161.121) by EX13MTAUWB001.ant.amazon.com (10.43.161.249) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 14 Jan 2020 19:29:52 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D10UWB004.ant.amazon.com (10.43.161.121) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 14 Jan 2020 19:29:52 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 14 Jan 2020 19:29:52 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id 2E01D40E7F; Tue, 14 Jan 2020 19:29:52 +0000 (UTC) Date: Tue, 14 Jan 2020 19:29:52 +0000 From: Anchal Agarwal To: "Rafael J. Wysocki" CC: Peter Zijlstra , "Singh, Balbir" , "Valentin, Eduardo" , "boris.ostrovsky@oracle.com" , "linux-kernel@vger.kernel.org" , "Woodhouse, David" , "vkuznets@redhat.com" , "sstabellini@kernel.org" , "tglx@linutronix.de" , "linux-pm@vger.kernel.org" , "linux-mm@kvack.org" , "pavel@ucw.cz" , "axboe@kernel.dk" , "x86@kernel.org" , "roger.pau@citrix.com" , "hpa@zytor.com" , "rjw@rjwysocki.net" , "mingo@redhat.com" , "Kamata, Munehisa" , "bp@alien8.de" , "netdev@vger.kernel.org" , "konrad.wilk@oracle.co" , "len.brown@intel.com" , "davem@davemloft.net" , "fllinden@amaozn.com" , "xen-devel@lists.xenproject.org" , Subject: Re: [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation Message-ID: <20200114192952.GA26755@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> <20200113101609.GT2844@hirez.programming.kicks-ass.net> <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> <20200113124247.GG2827@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jan 14, 2020 at 12:30:02AM +0100, Rafael J. Wysocki wrote: > On Mon, Jan 13, 2020 at 10:50 PM Rafael J. Wysocki wrote: > > > > On Mon, Jan 13, 2020 at 1:43 PM Peter Zijlstra wrote: > > > > > > On Mon, Jan 13, 2020 at 11:43:18AM +0000, Singh, Balbir wrote: > > > > For your original comment, just wanted to clarify the following: > > > > > > > > 1. After hibernation, the machine can be resumed on a different but compatible > > > > host (these are VM images hibernated) > > > > 2. This means the clock between host1 and host2 can/will be different > > > > > > > > In your comments are you making the assumption that the host(s) is/are the > > > > same? Just checking the assumptions being made and being on the same page with > > > > them. > > > > > > I would expect this to be the same problem we have as regular suspend, > > > after power off the TSC will have been reset, so resume will have to > > > somehow bridge that gap. I've no idea if/how it does that. > > > > In general, this is done by timekeeping_resume() and the only special > > thing done for the TSC appears to be the tsc_verify_tsc_adjust(true) > > call in tsc_resume(). > > And I forgot about tsc_restore_sched_clock_state() that gets called > via restore_processor_state() on x86, before calling > timekeeping_resume(). > In this case tsc_verify_tsc_adjust(true) this does nothing as feature bit X86_FEATURE_TSC_ADJUST is not available to guest. I am no expert in this area, but could this be messing things up? Thanks, Anchal From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIM_ADSP_ALL,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46ADBC33C9E for ; Tue, 14 Jan 2020 19:31:06 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 18A2E24658 for ; Tue, 14 Jan 2020 19:31:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="FCki7Boy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 18A2E24658 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1irRtW-0004DU-3y; Tue, 14 Jan 2020 19:30:30 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1irRtV-0004DP-35 for xen-devel@lists.xenproject.org; Tue, 14 Jan 2020 19:30:29 +0000 X-Inumbo-ID: 516a794f-3704-11ea-843c-12813bfff9fa Received: from smtp-fw-9102.amazon.com (unknown [207.171.184.29]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 516a794f-3704-11ea-843c-12813bfff9fa; Tue, 14 Jan 2020 19:30:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1579030229; x=1610566229; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=kERWQcPBD6U7cttKEBKCe+VLCzWuIvbuKzU3wI+3unU=; b=FCki7BoyQcfgsyWCOV1o926wDltZEzP8R/VWkvO6f8I/kI5jbYjgkmjm gny2ahtfI1tAP+9d3OhyhR7eNZStdqhVy+d8kBtL8S5/lbZQdY/oOgbDO u05MvStD+m97Kv8J7gqyugBCu4VFeHbVMibzHWxr2sdl7MiIlrRT++QHK o=; IronPort-SDR: 1XWTl0ZFPeAlOX/4U8kY6SYcBuV6mXXPHzXHV+KPO++vV8ZqU64Zy3g8i6YkQ3MNzJ1c7HsLNp 881qx3hEFFTA== X-IronPort-AV: E=Sophos;i="5.70,319,1574121600"; d="scan'208";a="18710229" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1a-807d4a99.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 14 Jan 2020 19:30:15 +0000 Received: from EX13MTAUWB001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1a-807d4a99.us-east-1.amazon.com (Postfix) with ESMTPS id F0146A29E9; Tue, 14 Jan 2020 19:30:07 +0000 (UTC) Received: from EX13D10UWB004.ant.amazon.com (10.43.161.121) by EX13MTAUWB001.ant.amazon.com (10.43.161.249) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 14 Jan 2020 19:29:52 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D10UWB004.ant.amazon.com (10.43.161.121) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 14 Jan 2020 19:29:52 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 14 Jan 2020 19:29:52 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id 2E01D40E7F; Tue, 14 Jan 2020 19:29:52 +0000 (UTC) Date: Tue, 14 Jan 2020 19:29:52 +0000 From: Anchal Agarwal To: "Rafael J. Wysocki" Message-ID: <20200114192952.GA26755@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> <20200113101609.GT2844@hirez.programming.kicks-ass.net> <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> <20200113124247.GG2827@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Precedence: Bulk Subject: Re: [Xen-devel] [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: "konrad.wilk@oracle.co" , "Kamata, Munehisa" , Peter Zijlstra , "linux-mm@kvack.org" , "pavel@ucw.cz" , "hpa@zytor.com" , "boris.ostrovsky@oracle.com" , "sstabellini@kernel.org" , "fllinden@amaozn.com" , "Valentin, Eduardo" , "x86@kernel.org" , "mingo@redhat.com" , "xen-devel@lists.xenproject.org" , "Singh, Balbir" , "len.brown@intel.com" , "linux-pm@vger.kernel.org" , anchalag@amazon.com, "bp@alien8.de" , "tglx@linutronix.de" , "axboe@kernel.dk" , "netdev@vger.kernel.org" , "rjw@rjwysocki.net" , "linux-kernel@vger.kernel.org" , "vkuznets@redhat.com" , "davem@davemloft.net" , "Woodhouse, David" , "roger.pau@citrix.com" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" T24gVHVlLCBKYW4gMTQsIDIwMjAgYXQgMTI6MzA6MDJBTSArMDEwMCwgUmFmYWVsIEouIFd5c29j a2kgd3JvdGU6Cj4gT24gTW9uLCBKYW4gMTMsIDIwMjAgYXQgMTA6NTAgUE0gUmFmYWVsIEouIFd5 c29ja2kgPHJhZmFlbEBrZXJuZWwub3JnPiB3cm90ZToKPiA+Cj4gPiBPbiBNb24sIEphbiAxMywg MjAyMCBhdCAxOjQzIFBNIFBldGVyIFppamxzdHJhIDxwZXRlcnpAaW5mcmFkZWFkLm9yZz4gd3Jv dGU6Cj4gPiA+Cj4gPiA+IE9uIE1vbiwgSmFuIDEzLCAyMDIwIGF0IDExOjQzOjE4QU0gKzAwMDAs IFNpbmdoLCBCYWxiaXIgd3JvdGU6Cj4gPiA+ID4gRm9yIHlvdXIgb3JpZ2luYWwgY29tbWVudCwg anVzdCB3YW50ZWQgdG8gY2xhcmlmeSB0aGUgZm9sbG93aW5nOgo+ID4gPiA+Cj4gPiA+ID4gMS4g QWZ0ZXIgaGliZXJuYXRpb24sIHRoZSBtYWNoaW5lIGNhbiBiZSByZXN1bWVkIG9uIGEgZGlmZmVy ZW50IGJ1dCBjb21wYXRpYmxlCj4gPiA+ID4gaG9zdCAodGhlc2UgYXJlIFZNIGltYWdlcyBoaWJl cm5hdGVkKQo+ID4gPiA+IDIuIFRoaXMgbWVhbnMgdGhlIGNsb2NrIGJldHdlZW4gaG9zdDEgYW5k IGhvc3QyIGNhbi93aWxsIGJlIGRpZmZlcmVudAo+ID4gPiA+Cj4gPiA+ID4gSW4geW91ciBjb21t ZW50cyBhcmUgeW91IG1ha2luZyB0aGUgYXNzdW1wdGlvbiB0aGF0IHRoZSBob3N0KHMpIGlzL2Fy ZSB0aGUKPiA+ID4gPiBzYW1lPyBKdXN0IGNoZWNraW5nIHRoZSBhc3N1bXB0aW9ucyBiZWluZyBt YWRlIGFuZCBiZWluZyBvbiB0aGUgc2FtZSBwYWdlIHdpdGgKPiA+ID4gPiB0aGVtLgo+ID4gPgo+ ID4gPiBJIHdvdWxkIGV4cGVjdCB0aGlzIHRvIGJlIHRoZSBzYW1lIHByb2JsZW0gd2UgaGF2ZSBh cyByZWd1bGFyIHN1c3BlbmQsCj4gPiA+IGFmdGVyIHBvd2VyIG9mZiB0aGUgVFNDIHdpbGwgaGF2 ZSBiZWVuIHJlc2V0LCBzbyByZXN1bWUgd2lsbCBoYXZlIHRvCj4gPiA+IHNvbWVob3cgYnJpZGdl IHRoYXQgZ2FwLiBJJ3ZlIG5vIGlkZWEgaWYvaG93IGl0IGRvZXMgdGhhdC4KPiA+Cj4gPiBJbiBn ZW5lcmFsLCB0aGlzIGlzIGRvbmUgYnkgdGltZWtlZXBpbmdfcmVzdW1lKCkgYW5kIHRoZSBvbmx5 IHNwZWNpYWwKPiA+IHRoaW5nIGRvbmUgZm9yIHRoZSBUU0MgYXBwZWFycyB0byBiZSB0aGUgdHNj X3ZlcmlmeV90c2NfYWRqdXN0KHRydWUpCj4gPiBjYWxsIGluIHRzY19yZXN1bWUoKS4KPiAKPiBB bmQgSSBmb3Jnb3QgYWJvdXQgdHNjX3Jlc3RvcmVfc2NoZWRfY2xvY2tfc3RhdGUoKSB0aGF0IGdl dHMgY2FsbGVkCj4gdmlhIHJlc3RvcmVfcHJvY2Vzc29yX3N0YXRlKCkgb24geDg2LCBiZWZvcmUg Y2FsbGluZwo+IHRpbWVrZWVwaW5nX3Jlc3VtZSgpLgo+CkluIHRoaXMgY2FzZSB0c2NfdmVyaWZ5 X3RzY19hZGp1c3QodHJ1ZSkgdGhpcyBkb2VzIG5vdGhpbmcgYXMKZmVhdHVyZSBiaXQgWDg2X0ZF QVRVUkVfVFNDX0FESlVTVCBpcyBub3QgYXZhaWxhYmxlIHRvIGd1ZXN0LiAKSSBhbSBubyBleHBl cnQgaW4gdGhpcyBhcmVhLCBidXQgY291bGQgdGhpcyBiZSBtZXNzaW5nIHRoaW5ncyB1cD8KClRo YW5rcywKQW5jaGFsCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBsaXN0cy54ZW5wcm9qZWN0Lm9y ZwpodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL21haWxtYW4vbGlzdGluZm8veGVuLWRldmVs