From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.0 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 972B8C33CAF for ; Mon, 13 Jan 2020 13:08:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6382C2081E for ; Mon, 13 Jan 2020 13:08:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="PFWqB/E/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728731AbgAMNIQ (ORCPT ); Mon, 13 Jan 2020 08:08:16 -0500 Received: from esa3.hc3370-68.iphmx.com ([216.71.145.155]:9696 "EHLO esa3.hc3370-68.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726593AbgAMNIP (ORCPT ); Mon, 13 Jan 2020 08:08:15 -0500 X-Greylist: delayed 426 seconds by postgrey-1.27 at vger.kernel.org; Mon, 13 Jan 2020 08:08:14 EST DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1578920896; h=subject:to:cc:references:from:message-id:date: mime-version:in-reply-to:content-transfer-encoding; bh=ngkACCPTrWi5McT5n6PAauLZLAB+ALgfEQVDPw6WcrY=; b=PFWqB/E/GXwB/X0Oeh5Gm4ZrjxrrPZ9s2K2I4SfqTm38VFngAotQiKRm rQAsW88fU8Ap5uI4NAdSkQiHpwGPiyC19RKZbytVr2p/MjAR0ozCHdaiQ 8i4nAiPKOCYP7Y9cYlK56IpGF6d0RXqhjf6VAwTSs3itJaU184uIsg4ed w=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=andrew.cooper3@citrix.com; spf=Pass smtp.mailfrom=Andrew.Cooper3@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of andrew.cooper3@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="andrew.cooper3@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa3.hc3370-68.iphmx.com: domain of Andrew.Cooper3@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="Andrew.Cooper3@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: GlauahcLPcHiN+06JBb+FFer7fhPNPvyqF8yyWZbPPRkFWWBNJGIFBvz8S9tRXAkU1/OKSXilb N87abC6Y4uVDvvv6XHlqkQvbGa2yqAh/Zy3zCUL8DN5Q89i90pTh+XBvuk5sMsHe51PcK1oND1 AfmXW8js1o5PHvxveLH0UTha768NDUThPbfKNdX+wVzVxbtMXVjDGi6ILPC+ZeDOAg8q7Y6e89 y8ulSfoUwjf57z9mM1ueqD6xvKVBchjvg86GkD4dK+2YeTImmj/FVhP+jPxHpCTNTPOadGpw53 qoo= X-SBRS: 2.7 X-MesageID: 10820025 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.69,429,1571716800"; d="scan'208";a="10820025" Subject: Re: [Xen-devel] [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation To: "Singh, Balbir" , "peterz@infradead.org" , "Valentin, Eduardo" CC: "konrad.wilk@oracle.co" , "x86@kernel.org" , "len.brown@intel.com" , "linux-mm@kvack.org" , "pavel@ucw.cz" , "hpa@zytor.com" , "boris.ostrovsky@oracle.com" , "sstabellini@kernel.org" , "fllinden@amaozn.com" , "Kamata, Munehisa" , "mingo@redhat.com" , "xen-devel@lists.xenproject.org" , "axboe@kernel.dk" , "linux-pm@vger.kernel.org" , "Agarwal, Anchal" , "bp@alien8.de" , "tglx@linutronix.de" , "jgross@suse.com" , "netdev@vger.kernel.org" , "Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com" , "rjw@rjwysocki.net" , "linux-kernel@vger.kernel.org" , "vkuznets@redhat.com" , "davem@davemloft.net" , "Woodhouse, David" , "roger.pau@citrix.com" References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> <20200113101609.GT2844@hirez.programming.kicks-ass.net> <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> From: Andrew Cooper Openpgp: preference=signencrypt Autocrypt: addr=andrew.cooper3@citrix.com; prefer-encrypt=mutual; keydata= mQINBFLhNn8BEADVhE+Hb8i0GV6mihnnr/uiQQdPF8kUoFzCOPXkf7jQ5sLYeJa0cQi6Penp VtiFYznTairnVsN5J+ujSTIb+OlMSJUWV4opS7WVNnxHbFTPYZVQ3erv7NKc2iVizCRZ2Kxn srM1oPXWRic8BIAdYOKOloF2300SL/bIpeD+x7h3w9B/qez7nOin5NzkxgFoaUeIal12pXSR Q354FKFoy6Vh96gc4VRqte3jw8mPuJQpfws+Pb+swvSf/i1q1+1I4jsRQQh2m6OTADHIqg2E ofTYAEh7R5HfPx0EXoEDMdRjOeKn8+vvkAwhviWXTHlG3R1QkbE5M/oywnZ83udJmi+lxjJ5 YhQ5IzomvJ16H0Bq+TLyVLO/VRksp1VR9HxCzItLNCS8PdpYYz5TC204ViycobYU65WMpzWe LFAGn8jSS25XIpqv0Y9k87dLbctKKA14Ifw2kq5OIVu2FuX+3i446JOa2vpCI9GcjCzi3oHV e00bzYiHMIl0FICrNJU0Kjho8pdo0m2uxkn6SYEpogAy9pnatUlO+erL4LqFUO7GXSdBRbw5 gNt25XTLdSFuZtMxkY3tq8MFss5QnjhehCVPEpE6y9ZjI4XB8ad1G4oBHVGK5LMsvg22PfMJ ISWFSHoF/B5+lHkCKWkFxZ0gZn33ju5n6/FOdEx4B8cMJt+cWwARAQABtClBbmRyZXcgQ29v cGVyIDxhbmRyZXcuY29vcGVyM0BjaXRyaXguY29tPokCOgQTAQgAJAIbAwULCQgHAwUVCgkI CwUWAgMBAAIeAQIXgAUCWKD95wIZAQAKCRBlw/kGpdefoHbdD/9AIoR3k6fKl+RFiFpyAhvO 59ttDFI7nIAnlYngev2XUR3acFElJATHSDO0ju+hqWqAb8kVijXLops0gOfqt3VPZq9cuHlh IMDquatGLzAadfFx2eQYIYT+FYuMoPZy/aTUazmJIDVxP7L383grjIkn+7tAv+qeDfE+txL4 SAm1UHNvmdfgL2/lcmL3xRh7sub3nJilM93RWX1Pe5LBSDXO45uzCGEdst6uSlzYR/MEr+5Z JQQ32JV64zwvf/aKaagSQSQMYNX9JFgfZ3TKWC1KJQbX5ssoX/5hNLqxMcZV3TN7kU8I3kjK mPec9+1nECOjjJSO/h4P0sBZyIUGfguwzhEeGf4sMCuSEM4xjCnwiBwftR17sr0spYcOpqET ZGcAmyYcNjy6CYadNCnfR40vhhWuCfNCBzWnUW0lFoo12wb0YnzoOLjvfD6OL3JjIUJNOmJy RCsJ5IA/Iz33RhSVRmROu+TztwuThClw63g7+hoyewv7BemKyuU6FTVhjjW+XUWmS/FzknSi dAG+insr0746cTPpSkGl3KAXeWDGJzve7/SBBfyznWCMGaf8E2P1oOdIZRxHgWj0zNr1+ooF /PzgLPiCI4OMUttTlEKChgbUTQ+5o0P080JojqfXwbPAyumbaYcQNiH1/xYbJdOFSiBv9rpt TQTBLzDKXok86LkCDQRS4TZ/ARAAkgqudHsp+hd82UVkvgnlqZjzz2vyrYfz7bkPtXaGb9H4 Rfo7mQsEQavEBdWWjbga6eMnDqtu+FC+qeTGYebToxEyp2lKDSoAsvt8w82tIlP/EbmRbDVn 7bhjBlfRcFjVYw8uVDPptT0TV47vpoCVkTwcyb6OltJrvg/QzV9f07DJswuda1JH3/qvYu0p vjPnYvCq4NsqY2XSdAJ02HrdYPFtNyPEntu1n1KK+gJrstjtw7KsZ4ygXYrsm/oCBiVW/OgU g/XIlGErkrxe4vQvJyVwg6YH653YTX5hLLUEL1NS4TCo47RP+wi6y+TnuAL36UtK/uFyEuPy wwrDVcC4cIFhYSfsO0BumEI65yu7a8aHbGfq2lW251UcoU48Z27ZUUZd2Dr6O/n8poQHbaTd 6bJJSjzGGHZVbRP9UQ3lkmkmc0+XCHmj5WhwNNYjgbbmML7y0fsJT5RgvefAIFfHBg7fTY/i kBEimoUsTEQz+N4hbKwo1hULfVxDJStE4sbPhjbsPCrlXf6W9CxSyQ0qmZ2bXsLQYRj2xqd1 bpA+1o1j2N4/au1R/uSiUFjewJdT/LX1EklKDcQwpk06Af/N7VZtSfEJeRV04unbsKVXWZAk uAJyDDKN99ziC0Wz5kcPyVD1HNf8bgaqGDzrv3TfYjwqayRFcMf7xJaL9xXedMcAEQEAAYkC HwQYAQgACQUCUuE2fwIbDAAKCRBlw/kGpdefoG4XEACD1Qf/er8EA7g23HMxYWd3FXHThrVQ HgiGdk5Yh632vjOm9L4sd/GCEACVQKjsu98e8o3ysitFlznEns5EAAXEbITrgKWXDDUWGYxd pnjj2u+GkVdsOAGk0kxczX6s+VRBhpbBI2PWnOsRJgU2n10PZ3mZD4Xu9kU2IXYmuW+e5KCA vTArRUdCrAtIa1k01sPipPPw6dfxx2e5asy21YOytzxuWFfJTGnVxZZSCyLUO83sh6OZhJkk b9rxL9wPmpN/t2IPaEKoAc0FTQZS36wAMOXkBh24PQ9gaLJvfPKpNzGD8XWR5HHF0NLIJhgg 4ZlEXQ2fVp3XrtocHqhu4UZR4koCijgB8sB7Tb0GCpwK+C4UePdFLfhKyRdSXuvY3AHJd4CP 4JzW0Bzq/WXY3XMOzUTYApGQpnUpdOmuQSfpV9MQO+/jo7r6yPbxT7CwRS5dcQPzUiuHLK9i nvjREdh84qycnx0/6dDroYhp0DFv4udxuAvt1h4wGwTPRQZerSm4xaYegEFusyhbZrI0U9tJ B8WrhBLXDiYlyJT6zOV2yZFuW47VrLsjYnHwn27hmxTC/7tvG3euCklmkn9Sl9IAKFu29RSo d5bD8kMSCYsTqtTfT6W4A3qHGvIDta3ptLYpIAOD2sY3GYq2nf3Bbzx81wZK14JdDDHUX2Rs 6+ahAA== Message-ID: <7bb967ca-2a91-6397-9c0a-6eafd43c83ed@citrix.com> Date: Mon, 13 Jan 2020 13:01:02 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL01.citrite.net (10.69.22.125) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 13/01/2020 11:43, Singh, Balbir wrote: > On Mon, 2020-01-13 at 11:16 +0100, Peter Zijlstra wrote: >> On Fri, Jan 10, 2020 at 07:35:20AM -0800, Eduardo Valentin wrote: >>> Hey Peter, >>> >>> On Wed, Jan 08, 2020 at 11:50:11AM +0100, Peter Zijlstra wrote: >>>> On Tue, Jan 07, 2020 at 11:45:26PM +0000, Anchal Agarwal wrote: >>>>> From: Eduardo Valentin >>>>> >>>>> System instability are seen during resume from hibernation when system >>>>> is under heavy CPU load. This is due to the lack of update of sched >>>>> clock data, and the scheduler would then think that heavy CPU hog >>>>> tasks need more time in CPU, causing the system to freeze >>>>> during the unfreezing of tasks. For example, threaded irqs, >>>>> and kernel processes servicing network interface may be delayed >>>>> for several tens of seconds, causing the system to be unreachable. >>>>> The fix for this situation is to mark the sched clock as unstable >>>>> as early as possible in the resume path, leaving it unstable >>>>> for the duration of the resume process. This will force the >>>>> scheduler to attempt to align the sched clock across CPUs using >>>>> the delta with time of day, updating sched clock data. In a post >>>>> hibernation event, we can then mark the sched clock as stable >>>>> again, avoiding unnecessary syncs with time of day on systems >>>>> in which TSC is reliable. >>>> This makes no frigging sense what so bloody ever. If the clock is >>>> stable, we don't care about sched_clock_data. When it is stable you get >>>> a linear function of the TSC without complicated bits on. >>>> >>>> When it is unstable, only then do we care about the sched_clock_data. >>>> >>> Yeah, maybe what is not clear here is that we covering for situation >>> where clock stability changes over time, e.g. at regular boot clock is >>> stable, hibernation happens, then restore happens in a non-stable clock. >> Still confused, who marks the thing unstable? The patch seems to suggest >> you do yourself, but it is not at all clear why. >> >> If TSC really is unstable, then it needs to remain unstable. If the TSC >> really is stable then there is no point in marking is unstable. >> >> Either way something is off, and you're not telling me what. >> > Hi, Peter > > For your original comment, just wanted to clarify the following: > > 1. After hibernation, the machine can be resumed on a different but compatible > host (these are VM images hibernated) > 2. This means the clock between host1 and host2 can/will be different The guests TSC value is part of all save/migrate/resume state.  Given this bug, I presume you've actually discarded all register state on hibernate, and the TSC is starting again from 0? The frequency of the new TSC might very likely be different, but the scale/offset in the paravirtual clock information should let Linux's view of time stay consistent. > In your comments are you making the assumption that the host(s) is/are the > same? Just checking the assumptions being made and being on the same page with > them. TSCs are a massive source of "fun".  I'm not surprised that there are yet more bugs around. Does anyone actually know what does/should happen to the real TSC on native S4?  The default course of action should be for virtualisation to follow suit. ~Andrew From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.0 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67DA1C33CAD for ; Mon, 13 Jan 2020 13:01:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1298A21569 for ; Mon, 13 Jan 2020 13:01:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="Su2e4iPG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1298A21569 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9880C8E0006; Mon, 13 Jan 2020 08:01:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 938928E0003; Mon, 13 Jan 2020 08:01:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B1AD8E0006; Mon, 13 Jan 2020 08:01:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id 5BFDF8E0003 for ; Mon, 13 Jan 2020 08:01:11 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 134ED4DB3 for ; Mon, 13 Jan 2020 13:01:11 +0000 (UTC) X-FDA: 76372621542.22.84B0416 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id E62EF18038E67 for ; Mon, 13 Jan 2020 13:01:10 +0000 (UTC) X-HE-Tag: chair89_445a74018d763 X-Filterd-Recvd-Size: 11737 Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Mon, 13 Jan 2020 13:01:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1578920470; h=subject:to:cc:references:from:message-id:date: mime-version:in-reply-to:content-transfer-encoding; bh=ngkACCPTrWi5McT5n6PAauLZLAB+ALgfEQVDPw6WcrY=; b=Su2e4iPGELoWTwm46da+RZIlUY3XW/Mp/aXEHZI7iA3vNvF+POP4ptAT /RY1kEcrfJvBIf4RuYRF1L3xQfWIRL916j/yuGWhAlKmvpasKbDuBxV6E uy4icJudZdQuglelzNKOAGuDWSTGB+uQwqQeH/hJ0N4xr91j5Dj0eolt4 A=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=andrew.cooper3@citrix.com; spf=Pass smtp.mailfrom=Andrew.Cooper3@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of andrew.cooper3@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="andrew.cooper3@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa3.hc3370-68.iphmx.com: domain of Andrew.Cooper3@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="Andrew.Cooper3@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: GlauahcLPcHiN+06JBb+FFer7fhPNPvyqF8yyWZbPPRkFWWBNJGIFBvz8S9tRXAkU1/OKSXilb N87abC6Y4uVDvvv6XHlqkQvbGa2yqAh/Zy3zCUL8DN5Q89i90pTh+XBvuk5sMsHe51PcK1oND1 AfmXW8js1o5PHvxveLH0UTha768NDUThPbfKNdX+wVzVxbtMXVjDGi6ILPC+ZeDOAg8q7Y6e89 y8ulSfoUwjf57z9mM1ueqD6xvKVBchjvg86GkD4dK+2YeTImmj/FVhP+jPxHpCTNTPOadGpw53 qoo= X-SBRS: 2.7 X-MesageID: 10820025 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.69,429,1571716800"; d="scan'208";a="10820025" Subject: Re: [Xen-devel] [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation To: "Singh, Balbir" , "peterz@infradead.org" , "Valentin, Eduardo" CC: "konrad.wilk@oracle.co" , "x86@kernel.org" , "len.brown@intel.com" , "linux-mm@kvack.org" , "pavel@ucw.cz" , "hpa@zytor.com" , "boris.ostrovsky@oracle.com" , "sstabellini@kernel.org" , "fllinden@amaozn.com" , "Kamata, Munehisa" , "mingo@redhat.com" , "xen-devel@lists.xenproject.org" , "axboe@kernel.dk" , "linux-pm@vger.kernel.org" , "Agarwal, Anchal" , "bp@alien8.de" , "tglx@linutronix.de" , "jgross@suse.com" , "netdev@vger.kernel.org" , "Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com" , "rjw@rjwysocki.net" , "linux-kernel@vger.kernel.org" , "vkuznets@redhat.com" , "davem@davemloft.net" , "Woodhouse, David" , "roger.pau@citrix.com" References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> <20200113101609.GT2844@hirez.programming.kicks-ass.net> <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> From: Andrew Cooper Openpgp: preference=signencrypt Autocrypt: addr=andrew.cooper3@citrix.com; prefer-encrypt=mutual; keydata= mQINBFLhNn8BEADVhE+Hb8i0GV6mihnnr/uiQQdPF8kUoFzCOPXkf7jQ5sLYeJa0cQi6Penp VtiFYznTairnVsN5J+ujSTIb+OlMSJUWV4opS7WVNnxHbFTPYZVQ3erv7NKc2iVizCRZ2Kxn srM1oPXWRic8BIAdYOKOloF2300SL/bIpeD+x7h3w9B/qez7nOin5NzkxgFoaUeIal12pXSR Q354FKFoy6Vh96gc4VRqte3jw8mPuJQpfws+Pb+swvSf/i1q1+1I4jsRQQh2m6OTADHIqg2E ofTYAEh7R5HfPx0EXoEDMdRjOeKn8+vvkAwhviWXTHlG3R1QkbE5M/oywnZ83udJmi+lxjJ5 YhQ5IzomvJ16H0Bq+TLyVLO/VRksp1VR9HxCzItLNCS8PdpYYz5TC204ViycobYU65WMpzWe LFAGn8jSS25XIpqv0Y9k87dLbctKKA14Ifw2kq5OIVu2FuX+3i446JOa2vpCI9GcjCzi3oHV e00bzYiHMIl0FICrNJU0Kjho8pdo0m2uxkn6SYEpogAy9pnatUlO+erL4LqFUO7GXSdBRbw5 gNt25XTLdSFuZtMxkY3tq8MFss5QnjhehCVPEpE6y9ZjI4XB8ad1G4oBHVGK5LMsvg22PfMJ ISWFSHoF/B5+lHkCKWkFxZ0gZn33ju5n6/FOdEx4B8cMJt+cWwARAQABtClBbmRyZXcgQ29v cGVyIDxhbmRyZXcuY29vcGVyM0BjaXRyaXguY29tPokCOgQTAQgAJAIbAwULCQgHAwUVCgkI CwUWAgMBAAIeAQIXgAUCWKD95wIZAQAKCRBlw/kGpdefoHbdD/9AIoR3k6fKl+RFiFpyAhvO 59ttDFI7nIAnlYngev2XUR3acFElJATHSDO0ju+hqWqAb8kVijXLops0gOfqt3VPZq9cuHlh IMDquatGLzAadfFx2eQYIYT+FYuMoPZy/aTUazmJIDVxP7L383grjIkn+7tAv+qeDfE+txL4 SAm1UHNvmdfgL2/lcmL3xRh7sub3nJilM93RWX1Pe5LBSDXO45uzCGEdst6uSlzYR/MEr+5Z JQQ32JV64zwvf/aKaagSQSQMYNX9JFgfZ3TKWC1KJQbX5ssoX/5hNLqxMcZV3TN7kU8I3kjK mPec9+1nECOjjJSO/h4P0sBZyIUGfguwzhEeGf4sMCuSEM4xjCnwiBwftR17sr0spYcOpqET ZGcAmyYcNjy6CYadNCnfR40vhhWuCfNCBzWnUW0lFoo12wb0YnzoOLjvfD6OL3JjIUJNOmJy RCsJ5IA/Iz33RhSVRmROu+TztwuThClw63g7+hoyewv7BemKyuU6FTVhjjW+XUWmS/FzknSi dAG+insr0746cTPpSkGl3KAXeWDGJzve7/SBBfyznWCMGaf8E2P1oOdIZRxHgWj0zNr1+ooF /PzgLPiCI4OMUttTlEKChgbUTQ+5o0P080JojqfXwbPAyumbaYcQNiH1/xYbJdOFSiBv9rpt TQTBLzDKXok86LkCDQRS4TZ/ARAAkgqudHsp+hd82UVkvgnlqZjzz2vyrYfz7bkPtXaGb9H4 Rfo7mQsEQavEBdWWjbga6eMnDqtu+FC+qeTGYebToxEyp2lKDSoAsvt8w82tIlP/EbmRbDVn 7bhjBlfRcFjVYw8uVDPptT0TV47vpoCVkTwcyb6OltJrvg/QzV9f07DJswuda1JH3/qvYu0p vjPnYvCq4NsqY2XSdAJ02HrdYPFtNyPEntu1n1KK+gJrstjtw7KsZ4ygXYrsm/oCBiVW/OgU g/XIlGErkrxe4vQvJyVwg6YH653YTX5hLLUEL1NS4TCo47RP+wi6y+TnuAL36UtK/uFyEuPy wwrDVcC4cIFhYSfsO0BumEI65yu7a8aHbGfq2lW251UcoU48Z27ZUUZd2Dr6O/n8poQHbaTd 6bJJSjzGGHZVbRP9UQ3lkmkmc0+XCHmj5WhwNNYjgbbmML7y0fsJT5RgvefAIFfHBg7fTY/i kBEimoUsTEQz+N4hbKwo1hULfVxDJStE4sbPhjbsPCrlXf6W9CxSyQ0qmZ2bXsLQYRj2xqd1 bpA+1o1j2N4/au1R/uSiUFjewJdT/LX1EklKDcQwpk06Af/N7VZtSfEJeRV04unbsKVXWZAk uAJyDDKN99ziC0Wz5kcPyVD1HNf8bgaqGDzrv3TfYjwqayRFcMf7xJaL9xXedMcAEQEAAYkC HwQYAQgACQUCUuE2fwIbDAAKCRBlw/kGpdefoG4XEACD1Qf/er8EA7g23HMxYWd3FXHThrVQ HgiGdk5Yh632vjOm9L4sd/GCEACVQKjsu98e8o3ysitFlznEns5EAAXEbITrgKWXDDUWGYxd pnjj2u+GkVdsOAGk0kxczX6s+VRBhpbBI2PWnOsRJgU2n10PZ3mZD4Xu9kU2IXYmuW+e5KCA vTArRUdCrAtIa1k01sPipPPw6dfxx2e5asy21YOytzxuWFfJTGnVxZZSCyLUO83sh6OZhJkk b9rxL9wPmpN/t2IPaEKoAc0FTQZS36wAMOXkBh24PQ9gaLJvfPKpNzGD8XWR5HHF0NLIJhgg 4ZlEXQ2fVp3XrtocHqhu4UZR4koCijgB8sB7Tb0GCpwK+C4UePdFLfhKyRdSXuvY3AHJd4CP 4JzW0Bzq/WXY3XMOzUTYApGQpnUpdOmuQSfpV9MQO+/jo7r6yPbxT7CwRS5dcQPzUiuHLK9i nvjREdh84qycnx0/6dDroYhp0DFv4udxuAvt1h4wGwTPRQZerSm4xaYegEFusyhbZrI0U9tJ B8WrhBLXDiYlyJT6zOV2yZFuW47VrLsjYnHwn27hmxTC/7tvG3euCklmkn9Sl9IAKFu29RSo d5bD8kMSCYsTqtTfT6W4A3qHGvIDta3ptLYpIAOD2sY3GYq2nf3Bbzx81wZK14JdDDHUX2Rs 6+ahAA== Message-ID: <7bb967ca-2a91-6397-9c0a-6eafd43c83ed@citrix.com> Date: Mon, 13 Jan 2020 13:01:02 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> Content-Type: text/plain; charset="utf-8" Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL01.citrite.net (10.69.22.125) Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 13/01/2020 11:43, Singh, Balbir wrote: > On Mon, 2020-01-13 at 11:16 +0100, Peter Zijlstra wrote: >> On Fri, Jan 10, 2020 at 07:35:20AM -0800, Eduardo Valentin wrote: >>> Hey Peter, >>> >>> On Wed, Jan 08, 2020 at 11:50:11AM +0100, Peter Zijlstra wrote: >>>> On Tue, Jan 07, 2020 at 11:45:26PM +0000, Anchal Agarwal wrote: >>>>> From: Eduardo Valentin >>>>> >>>>> System instability are seen during resume from hibernation when sys= tem >>>>> is under heavy CPU load. This is due to the lack of update of sched >>>>> clock data, and the scheduler would then think that heavy CPU hog >>>>> tasks need more time in CPU, causing the system to freeze >>>>> during the unfreezing of tasks. For example, threaded irqs, >>>>> and kernel processes servicing network interface may be delayed >>>>> for several tens of seconds, causing the system to be unreachable. >>>>> The fix for this situation is to mark the sched clock as unstable >>>>> as early as possible in the resume path, leaving it unstable >>>>> for the duration of the resume process. This will force the >>>>> scheduler to attempt to align the sched clock across CPUs using >>>>> the delta with time of day, updating sched clock data. In a post >>>>> hibernation event, we can then mark the sched clock as stable >>>>> again, avoiding unnecessary syncs with time of day on systems >>>>> in which TSC is reliable. >>>> This makes no frigging sense what so bloody ever. If the clock is >>>> stable, we don't care about sched_clock_data. When it is stable you = get >>>> a linear function of the TSC without complicated bits on. >>>> >>>> When it is unstable, only then do we care about the sched_clock_data= . >>>> >>> Yeah, maybe what is not clear here is that we covering for situation >>> where clock stability changes over time, e.g. at regular boot clock i= s >>> stable, hibernation happens, then restore happens in a non-stable clo= ck. >> Still confused, who marks the thing unstable? The patch seems to sugge= st >> you do yourself, but it is not at all clear why. >> >> If TSC really is unstable, then it needs to remain unstable. If the TS= C >> really is stable then there is no point in marking is unstable. >> >> Either way something is off, and you're not telling me what. >> > Hi, Peter > > For your original comment, just wanted to clarify the following: > > 1. After hibernation, the machine can be resumed on a different but com= patible > host (these are VM images hibernated) > 2. This means the clock between host1 and host2 can/will be different The guests TSC value is part of all save/migrate/resume state.=C2=A0 Give= n this bug, I presume you've actually discarded all register state on hibernate, and the TSC is starting again from 0? The frequency of the new TSC might very likely be different, but the scale/offset in the paravirtual clock information should let Linux's view of time stay consistent. > In your comments are you making the assumption that the host(s) is/are = the > same? Just checking the assumptions being made and being on the same pa= ge with > them. TSCs are a massive source of "fun".=C2=A0 I'm not surprised that there ar= e yet more bugs around. Does anyone actually know what does/should happen to the real TSC on native S4?=C2=A0 The default course of action should be for virtualisatio= n to follow suit. ~Andrew From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.0 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C759C33CAD for ; Mon, 13 Jan 2020 13:01:32 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DE3CE207FF for ; Mon, 13 Jan 2020 13:01:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="Su2e4iPG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DE3CE207FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iqzLE-0003hR-B5; Mon, 13 Jan 2020 13:01:12 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iqzLC-0003hM-Fn for xen-devel@lists.xenproject.org; Mon, 13 Jan 2020 13:01:10 +0000 X-Inumbo-ID: c3f734b2-3604-11ea-8263-12813bfff9fa Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id c3f734b2-3604-11ea-8263-12813bfff9fa; Mon, 13 Jan 2020 13:01:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1578920470; h=subject:to:cc:references:from:message-id:date: mime-version:in-reply-to:content-transfer-encoding; bh=ngkACCPTrWi5McT5n6PAauLZLAB+ALgfEQVDPw6WcrY=; b=Su2e4iPGELoWTwm46da+RZIlUY3XW/Mp/aXEHZI7iA3vNvF+POP4ptAT /RY1kEcrfJvBIf4RuYRF1L3xQfWIRL916j/yuGWhAlKmvpasKbDuBxV6E uy4icJudZdQuglelzNKOAGuDWSTGB+uQwqQeH/hJ0N4xr91j5Dj0eolt4 A=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=andrew.cooper3@citrix.com; spf=Pass smtp.mailfrom=Andrew.Cooper3@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of andrew.cooper3@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="andrew.cooper3@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa3.hc3370-68.iphmx.com: domain of Andrew.Cooper3@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="Andrew.Cooper3@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: GlauahcLPcHiN+06JBb+FFer7fhPNPvyqF8yyWZbPPRkFWWBNJGIFBvz8S9tRXAkU1/OKSXilb N87abC6Y4uVDvvv6XHlqkQvbGa2yqAh/Zy3zCUL8DN5Q89i90pTh+XBvuk5sMsHe51PcK1oND1 AfmXW8js1o5PHvxveLH0UTha768NDUThPbfKNdX+wVzVxbtMXVjDGi6ILPC+ZeDOAg8q7Y6e89 y8ulSfoUwjf57z9mM1ueqD6xvKVBchjvg86GkD4dK+2YeTImmj/FVhP+jPxHpCTNTPOadGpw53 qoo= X-SBRS: 2.7 X-MesageID: 10820025 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.69,429,1571716800"; d="scan'208";a="10820025" To: "Singh, Balbir" , "peterz@infradead.org" , "Valentin, Eduardo" References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> <20200113101609.GT2844@hirez.programming.kicks-ass.net> <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> From: Andrew Cooper Openpgp: preference=signencrypt Autocrypt: addr=andrew.cooper3@citrix.com; prefer-encrypt=mutual; keydata= mQINBFLhNn8BEADVhE+Hb8i0GV6mihnnr/uiQQdPF8kUoFzCOPXkf7jQ5sLYeJa0cQi6Penp VtiFYznTairnVsN5J+ujSTIb+OlMSJUWV4opS7WVNnxHbFTPYZVQ3erv7NKc2iVizCRZ2Kxn srM1oPXWRic8BIAdYOKOloF2300SL/bIpeD+x7h3w9B/qez7nOin5NzkxgFoaUeIal12pXSR Q354FKFoy6Vh96gc4VRqte3jw8mPuJQpfws+Pb+swvSf/i1q1+1I4jsRQQh2m6OTADHIqg2E ofTYAEh7R5HfPx0EXoEDMdRjOeKn8+vvkAwhviWXTHlG3R1QkbE5M/oywnZ83udJmi+lxjJ5 YhQ5IzomvJ16H0Bq+TLyVLO/VRksp1VR9HxCzItLNCS8PdpYYz5TC204ViycobYU65WMpzWe LFAGn8jSS25XIpqv0Y9k87dLbctKKA14Ifw2kq5OIVu2FuX+3i446JOa2vpCI9GcjCzi3oHV e00bzYiHMIl0FICrNJU0Kjho8pdo0m2uxkn6SYEpogAy9pnatUlO+erL4LqFUO7GXSdBRbw5 gNt25XTLdSFuZtMxkY3tq8MFss5QnjhehCVPEpE6y9ZjI4XB8ad1G4oBHVGK5LMsvg22PfMJ ISWFSHoF/B5+lHkCKWkFxZ0gZn33ju5n6/FOdEx4B8cMJt+cWwARAQABtClBbmRyZXcgQ29v cGVyIDxhbmRyZXcuY29vcGVyM0BjaXRyaXguY29tPokCOgQTAQgAJAIbAwULCQgHAwUVCgkI CwUWAgMBAAIeAQIXgAUCWKD95wIZAQAKCRBlw/kGpdefoHbdD/9AIoR3k6fKl+RFiFpyAhvO 59ttDFI7nIAnlYngev2XUR3acFElJATHSDO0ju+hqWqAb8kVijXLops0gOfqt3VPZq9cuHlh IMDquatGLzAadfFx2eQYIYT+FYuMoPZy/aTUazmJIDVxP7L383grjIkn+7tAv+qeDfE+txL4 SAm1UHNvmdfgL2/lcmL3xRh7sub3nJilM93RWX1Pe5LBSDXO45uzCGEdst6uSlzYR/MEr+5Z JQQ32JV64zwvf/aKaagSQSQMYNX9JFgfZ3TKWC1KJQbX5ssoX/5hNLqxMcZV3TN7kU8I3kjK mPec9+1nECOjjJSO/h4P0sBZyIUGfguwzhEeGf4sMCuSEM4xjCnwiBwftR17sr0spYcOpqET ZGcAmyYcNjy6CYadNCnfR40vhhWuCfNCBzWnUW0lFoo12wb0YnzoOLjvfD6OL3JjIUJNOmJy RCsJ5IA/Iz33RhSVRmROu+TztwuThClw63g7+hoyewv7BemKyuU6FTVhjjW+XUWmS/FzknSi dAG+insr0746cTPpSkGl3KAXeWDGJzve7/SBBfyznWCMGaf8E2P1oOdIZRxHgWj0zNr1+ooF /PzgLPiCI4OMUttTlEKChgbUTQ+5o0P080JojqfXwbPAyumbaYcQNiH1/xYbJdOFSiBv9rpt TQTBLzDKXok86LkCDQRS4TZ/ARAAkgqudHsp+hd82UVkvgnlqZjzz2vyrYfz7bkPtXaGb9H4 Rfo7mQsEQavEBdWWjbga6eMnDqtu+FC+qeTGYebToxEyp2lKDSoAsvt8w82tIlP/EbmRbDVn 7bhjBlfRcFjVYw8uVDPptT0TV47vpoCVkTwcyb6OltJrvg/QzV9f07DJswuda1JH3/qvYu0p vjPnYvCq4NsqY2XSdAJ02HrdYPFtNyPEntu1n1KK+gJrstjtw7KsZ4ygXYrsm/oCBiVW/OgU g/XIlGErkrxe4vQvJyVwg6YH653YTX5hLLUEL1NS4TCo47RP+wi6y+TnuAL36UtK/uFyEuPy wwrDVcC4cIFhYSfsO0BumEI65yu7a8aHbGfq2lW251UcoU48Z27ZUUZd2Dr6O/n8poQHbaTd 6bJJSjzGGHZVbRP9UQ3lkmkmc0+XCHmj5WhwNNYjgbbmML7y0fsJT5RgvefAIFfHBg7fTY/i kBEimoUsTEQz+N4hbKwo1hULfVxDJStE4sbPhjbsPCrlXf6W9CxSyQ0qmZ2bXsLQYRj2xqd1 bpA+1o1j2N4/au1R/uSiUFjewJdT/LX1EklKDcQwpk06Af/N7VZtSfEJeRV04unbsKVXWZAk uAJyDDKN99ziC0Wz5kcPyVD1HNf8bgaqGDzrv3TfYjwqayRFcMf7xJaL9xXedMcAEQEAAYkC HwQYAQgACQUCUuE2fwIbDAAKCRBlw/kGpdefoG4XEACD1Qf/er8EA7g23HMxYWd3FXHThrVQ HgiGdk5Yh632vjOm9L4sd/GCEACVQKjsu98e8o3ysitFlznEns5EAAXEbITrgKWXDDUWGYxd pnjj2u+GkVdsOAGk0kxczX6s+VRBhpbBI2PWnOsRJgU2n10PZ3mZD4Xu9kU2IXYmuW+e5KCA vTArRUdCrAtIa1k01sPipPPw6dfxx2e5asy21YOytzxuWFfJTGnVxZZSCyLUO83sh6OZhJkk b9rxL9wPmpN/t2IPaEKoAc0FTQZS36wAMOXkBh24PQ9gaLJvfPKpNzGD8XWR5HHF0NLIJhgg 4ZlEXQ2fVp3XrtocHqhu4UZR4koCijgB8sB7Tb0GCpwK+C4UePdFLfhKyRdSXuvY3AHJd4CP 4JzW0Bzq/WXY3XMOzUTYApGQpnUpdOmuQSfpV9MQO+/jo7r6yPbxT7CwRS5dcQPzUiuHLK9i nvjREdh84qycnx0/6dDroYhp0DFv4udxuAvt1h4wGwTPRQZerSm4xaYegEFusyhbZrI0U9tJ B8WrhBLXDiYlyJT6zOV2yZFuW47VrLsjYnHwn27hmxTC/7tvG3euCklmkn9Sl9IAKFu29RSo d5bD8kMSCYsTqtTfT6W4A3qHGvIDta3ptLYpIAOD2sY3GYq2nf3Bbzx81wZK14JdDDHUX2Rs 6+ahAA== Message-ID: <7bb967ca-2a91-6397-9c0a-6eafd43c83ed@citrix.com> Date: Mon, 13 Jan 2020 13:01:02 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL01.citrite.net (10.69.22.125) Subject: Re: [Xen-devel] [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: "konrad.wilk@oracle.co" , "Kamata, Munehisa" , "len.brown@intel.com" , "linux-mm@kvack.org" , "pavel@ucw.cz" , "hpa@zytor.com" , "tglx@linutronix.de" , "sstabellini@kernel.org" , "fllinden@amaozn.com" , "x86@kernel.org" , "mingo@redhat.com" , "xen-devel@lists.xenproject.org" , "jgross@suse.com" , "linux-pm@vger.kernel.org" , "Agarwal, Anchal" , "bp@alien8.de" , "boris.ostrovsky@oracle.com" , "axboe@kernel.dk" , "netdev@vger.kernel.org" , "Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com" , "rjw@rjwysocki.net" , "linux-kernel@vger.kernel.org" , "vkuznets@redhat.com" , "davem@davemloft.net" , "Woodhouse, David" , "roger.pau@citrix.com" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" T24gMTMvMDEvMjAyMCAxMTo0MywgU2luZ2gsIEJhbGJpciB3cm90ZToKPiBPbiBNb24sIDIwMjAt MDEtMTMgYXQgMTE6MTYgKzAxMDAsIFBldGVyIFppamxzdHJhIHdyb3RlOgo+PiBPbiBGcmksIEph biAxMCwgMjAyMCBhdCAwNzozNToyMEFNIC0wODAwLCBFZHVhcmRvIFZhbGVudGluIHdyb3RlOgo+ Pj4gSGV5IFBldGVyLAo+Pj4KPj4+IE9uIFdlZCwgSmFuIDA4LCAyMDIwIGF0IDExOjUwOjExQU0g KzAxMDAsIFBldGVyIFppamxzdHJhIHdyb3RlOgo+Pj4+IE9uIFR1ZSwgSmFuIDA3LCAyMDIwIGF0 IDExOjQ1OjI2UE0gKzAwMDAsIEFuY2hhbCBBZ2Fyd2FsIHdyb3RlOgo+Pj4+PiBGcm9tOiBFZHVh cmRvIFZhbGVudGluIDxlZHV2YWxAYW1hem9uLmNvbT4KPj4+Pj4KPj4+Pj4gU3lzdGVtIGluc3Rh YmlsaXR5IGFyZSBzZWVuIGR1cmluZyByZXN1bWUgZnJvbSBoaWJlcm5hdGlvbiB3aGVuIHN5c3Rl bQo+Pj4+PiBpcyB1bmRlciBoZWF2eSBDUFUgbG9hZC4gVGhpcyBpcyBkdWUgdG8gdGhlIGxhY2sg b2YgdXBkYXRlIG9mIHNjaGVkCj4+Pj4+IGNsb2NrIGRhdGEsIGFuZCB0aGUgc2NoZWR1bGVyIHdv dWxkIHRoZW4gdGhpbmsgdGhhdCBoZWF2eSBDUFUgaG9nCj4+Pj4+IHRhc2tzIG5lZWQgbW9yZSB0 aW1lIGluIENQVSwgY2F1c2luZyB0aGUgc3lzdGVtIHRvIGZyZWV6ZQo+Pj4+PiBkdXJpbmcgdGhl IHVuZnJlZXppbmcgb2YgdGFza3MuIEZvciBleGFtcGxlLCB0aHJlYWRlZCBpcnFzLAo+Pj4+PiBh bmQga2VybmVsIHByb2Nlc3NlcyBzZXJ2aWNpbmcgbmV0d29yayBpbnRlcmZhY2UgbWF5IGJlIGRl bGF5ZWQKPj4+Pj4gZm9yIHNldmVyYWwgdGVucyBvZiBzZWNvbmRzLCBjYXVzaW5nIHRoZSBzeXN0 ZW0gdG8gYmUgdW5yZWFjaGFibGUuCj4+Pj4+IFRoZSBmaXggZm9yIHRoaXMgc2l0dWF0aW9uIGlz IHRvIG1hcmsgdGhlIHNjaGVkIGNsb2NrIGFzIHVuc3RhYmxlCj4+Pj4+IGFzIGVhcmx5IGFzIHBv c3NpYmxlIGluIHRoZSByZXN1bWUgcGF0aCwgbGVhdmluZyBpdCB1bnN0YWJsZQo+Pj4+PiBmb3Ig dGhlIGR1cmF0aW9uIG9mIHRoZSByZXN1bWUgcHJvY2Vzcy4gVGhpcyB3aWxsIGZvcmNlIHRoZQo+ Pj4+PiBzY2hlZHVsZXIgdG8gYXR0ZW1wdCB0byBhbGlnbiB0aGUgc2NoZWQgY2xvY2sgYWNyb3Nz IENQVXMgdXNpbmcKPj4+Pj4gdGhlIGRlbHRhIHdpdGggdGltZSBvZiBkYXksIHVwZGF0aW5nIHNj aGVkIGNsb2NrIGRhdGEuIEluIGEgcG9zdAo+Pj4+PiBoaWJlcm5hdGlvbiBldmVudCwgd2UgY2Fu IHRoZW4gbWFyayB0aGUgc2NoZWQgY2xvY2sgYXMgc3RhYmxlCj4+Pj4+IGFnYWluLCBhdm9pZGlu ZyB1bm5lY2Vzc2FyeSBzeW5jcyB3aXRoIHRpbWUgb2YgZGF5IG9uIHN5c3RlbXMKPj4+Pj4gaW4g d2hpY2ggVFNDIGlzIHJlbGlhYmxlLgo+Pj4+IFRoaXMgbWFrZXMgbm8gZnJpZ2dpbmcgc2Vuc2Ug d2hhdCBzbyBibG9vZHkgZXZlci4gSWYgdGhlIGNsb2NrIGlzCj4+Pj4gc3RhYmxlLCB3ZSBkb24n dCBjYXJlIGFib3V0IHNjaGVkX2Nsb2NrX2RhdGEuIFdoZW4gaXQgaXMgc3RhYmxlIHlvdSBnZXQK Pj4+PiBhIGxpbmVhciBmdW5jdGlvbiBvZiB0aGUgVFNDIHdpdGhvdXQgY29tcGxpY2F0ZWQgYml0 cyBvbi4KPj4+Pgo+Pj4+IFdoZW4gaXQgaXMgdW5zdGFibGUsIG9ubHkgdGhlbiBkbyB3ZSBjYXJl IGFib3V0IHRoZSBzY2hlZF9jbG9ja19kYXRhLgo+Pj4+Cj4+PiBZZWFoLCBtYXliZSB3aGF0IGlz IG5vdCBjbGVhciBoZXJlIGlzIHRoYXQgd2UgY292ZXJpbmcgZm9yIHNpdHVhdGlvbgo+Pj4gd2hl cmUgY2xvY2sgc3RhYmlsaXR5IGNoYW5nZXMgb3ZlciB0aW1lLCBlLmcuIGF0IHJlZ3VsYXIgYm9v dCBjbG9jayBpcwo+Pj4gc3RhYmxlLCBoaWJlcm5hdGlvbiBoYXBwZW5zLCB0aGVuIHJlc3RvcmUg aGFwcGVucyBpbiBhIG5vbi1zdGFibGUgY2xvY2suCj4+IFN0aWxsIGNvbmZ1c2VkLCB3aG8gbWFy a3MgdGhlIHRoaW5nIHVuc3RhYmxlPyBUaGUgcGF0Y2ggc2VlbXMgdG8gc3VnZ2VzdAo+PiB5b3Ug ZG8geW91cnNlbGYsIGJ1dCBpdCBpcyBub3QgYXQgYWxsIGNsZWFyIHdoeS4KPj4KPj4gSWYgVFND IHJlYWxseSBpcyB1bnN0YWJsZSwgdGhlbiBpdCBuZWVkcyB0byByZW1haW4gdW5zdGFibGUuIElm IHRoZSBUU0MKPj4gcmVhbGx5IGlzIHN0YWJsZSB0aGVuIHRoZXJlIGlzIG5vIHBvaW50IGluIG1h cmtpbmcgaXMgdW5zdGFibGUuCj4+Cj4+IEVpdGhlciB3YXkgc29tZXRoaW5nIGlzIG9mZiwgYW5k IHlvdSdyZSBub3QgdGVsbGluZyBtZSB3aGF0Lgo+Pgo+IEhpLCBQZXRlcgo+Cj4gRm9yIHlvdXIg b3JpZ2luYWwgY29tbWVudCwganVzdCB3YW50ZWQgdG8gY2xhcmlmeSB0aGUgZm9sbG93aW5nOgo+ Cj4gMS4gQWZ0ZXIgaGliZXJuYXRpb24sIHRoZSBtYWNoaW5lIGNhbiBiZSByZXN1bWVkIG9uIGEg ZGlmZmVyZW50IGJ1dCBjb21wYXRpYmxlCj4gaG9zdCAodGhlc2UgYXJlIFZNIGltYWdlcyBoaWJl cm5hdGVkKQo+IDIuIFRoaXMgbWVhbnMgdGhlIGNsb2NrIGJldHdlZW4gaG9zdDEgYW5kIGhvc3Qy IGNhbi93aWxsIGJlIGRpZmZlcmVudAoKVGhlIGd1ZXN0cyBUU0MgdmFsdWUgaXMgcGFydCBvZiBh bGwgc2F2ZS9taWdyYXRlL3Jlc3VtZSBzdGF0ZS7CoCBHaXZlbgp0aGlzIGJ1ZywgSSBwcmVzdW1l IHlvdSd2ZSBhY3R1YWxseSBkaXNjYXJkZWQgYWxsIHJlZ2lzdGVyIHN0YXRlIG9uCmhpYmVybmF0 ZSwgYW5kIHRoZSBUU0MgaXMgc3RhcnRpbmcgYWdhaW4gZnJvbSAwPwoKVGhlIGZyZXF1ZW5jeSBv ZiB0aGUgbmV3IFRTQyBtaWdodCB2ZXJ5IGxpa2VseSBiZSBkaWZmZXJlbnQsIGJ1dCB0aGUKc2Nh bGUvb2Zmc2V0IGluIHRoZSBwYXJhdmlydHVhbCBjbG9jayBpbmZvcm1hdGlvbiBzaG91bGQgbGV0 IExpbnV4J3MKdmlldyBvZiB0aW1lIHN0YXkgY29uc2lzdGVudC4KCj4gSW4geW91ciBjb21tZW50 cyBhcmUgeW91IG1ha2luZyB0aGUgYXNzdW1wdGlvbiB0aGF0IHRoZSBob3N0KHMpIGlzL2FyZSB0 aGUKPiBzYW1lPyBKdXN0IGNoZWNraW5nIHRoZSBhc3N1bXB0aW9ucyBiZWluZyBtYWRlIGFuZCBi ZWluZyBvbiB0aGUgc2FtZSBwYWdlIHdpdGgKPiB0aGVtLgoKVFNDcyBhcmUgYSBtYXNzaXZlIHNv dXJjZSBvZiAiZnVuIi7CoCBJJ20gbm90IHN1cnByaXNlZCB0aGF0IHRoZXJlIGFyZQp5ZXQgbW9y ZSBidWdzIGFyb3VuZC4KCkRvZXMgYW55b25lIGFjdHVhbGx5IGtub3cgd2hhdCBkb2VzL3Nob3Vs ZCBoYXBwZW4gdG8gdGhlIHJlYWwgVFNDIG9uCm5hdGl2ZSBTND/CoCBUaGUgZGVmYXVsdCBjb3Vy c2Ugb2YgYWN0aW9uIHNob3VsZCBiZSBmb3IgdmlydHVhbGlzYXRpb24gdG8KZm9sbG93IHN1aXQu Cgp+QW5kcmV3CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f XwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBsaXN0cy54ZW5wcm9qZWN0Lm9yZwpo dHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL21haWxtYW4vbGlzdGluZm8veGVuLWRldmVs