From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB549C33CAF for ; Mon, 13 Jan 2020 11:48:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 98B102084D for ; Mon, 13 Jan 2020 11:48:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1578916101; bh=LxnhZqYlYpH5Vy9lVIo6OhZ32kMjzMITo+VXkeln1fw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:List-ID:From; b=s7oKElx5IWZx9CvT+eCB0JmGS5Gqj921PusffA8neuLLtLL19EfftDT2YPBx2lUr5 B+XtC1fPMmhvgWCaR595/uGukDhk2U3ZaetKxg6r+vi8X+OVuDTygNPGlVM1I06Srx 1YJYqo9GEI4up17Zp9HDJALaIn/n44fyH+URUa5M= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728699AbgAMLsU (ORCPT ); Mon, 13 Jan 2020 06:48:20 -0500 Received: from mail-ot1-f67.google.com ([209.85.210.67]:42546 "EHLO mail-ot1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726180AbgAMLsU (ORCPT ); Mon, 13 Jan 2020 06:48:20 -0500 Received: by mail-ot1-f67.google.com with SMTP id 66so8635587otd.9; Mon, 13 Jan 2020 03:48:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=LxnhZqYlYpH5Vy9lVIo6OhZ32kMjzMITo+VXkeln1fw=; b=pgtg9JnHLBwCYyAZ5RFYQGgsl66M/mj7yM4MaFDspS6YKEdB6ROJMmAjIIAEXYkksF KzKUHo82O4P/9hmKRMftjCb4Bg13udYWoRS5mI7ymCEb0qYb2lsDeBQ9aJ8c03P6YYu3 RpWLNKYlDqjUndZh+iMRoeNY3reI9vvAUIlyGZWXv2/dZ88XegVLV2+nIr7XuhSrBLiF ATtFSmI6LuVhD2SVTvG9pcezEIeyuJj62e0CIabQLIBbsHYv0TvFCPjTS+bcTY3OqRt5 zu2UluqGJWQF1SD/cHUMRL91/W2L5V5U0lQnKwRg4sqLYLXhn6jHzMlRP5NmGkJWY3B5 GQTg== X-Gm-Message-State: APjAAAUxBfUnWj5LxEpGzaWPJZmSoYX9m1hVevSpYeSTEhxL4qZfH0ho tN+Wn8dJGaZVxT2lIoutfhcA2CamCqh4dboRlVE= X-Google-Smtp-Source: APXvYqw3Qgw3VCVDMPHWyRb4A+fn8GcFRFWgD+RozWxW13+tMz8RTbaH9oz22LVJAtTKMJUr2mfSKOEXlhCa9eU2PHs= X-Received: by 2002:a05:6830:4b9:: with SMTP id l25mr13198440otd.266.1578916099461; Mon, 13 Jan 2020 03:48:19 -0800 (PST) MIME-Version: 1.0 References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> <20200113101609.GT2844@hirez.programming.kicks-ass.net> <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> In-Reply-To: <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> From: "Rafael J. Wysocki" Date: Mon, 13 Jan 2020 12:48:08 +0100 Message-ID: Subject: Re: [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation To: "Singh, Balbir" Cc: "peterz@infradead.org" , "Valentin, Eduardo" , "boris.ostrovsky@oracle.com" , "linux-kernel@vger.kernel.org" , "Agarwal, Anchal" , "Woodhouse, David" , "vkuznets@redhat.com" , "sstabellini@kernel.org" , "tglx@linutronix.de" , "linux-pm@vger.kernel.org" , "Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com" , "linux-mm@kvack.org" , "jgross@suse.com" , "pavel@ucw.cz" , "axboe@kernel.dk" , "x86@kernel.org" , "roger.pau@citrix.com" , "hpa@zytor.com" , "rjw@rjwysocki.net" , "mingo@redhat.com" , "Kamata, Munehisa" , "bp@alien8.de" , "netdev@vger.kernel.org" , "konrad.wilk@oracle.co" , "len.brown@intel.com" , "davem@davemloft.net" , "fllinden@amaozn.com" , "xen-devel@lists.xenproject.org" Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jan 13, 2020 at 12:43 PM Singh, Balbir wrote: > > On Mon, 2020-01-13 at 11:16 +0100, Peter Zijlstra wrote: > > On Fri, Jan 10, 2020 at 07:35:20AM -0800, Eduardo Valentin wrote: > > > Hey Peter, > > > > > > On Wed, Jan 08, 2020 at 11:50:11AM +0100, Peter Zijlstra wrote: > > > > On Tue, Jan 07, 2020 at 11:45:26PM +0000, Anchal Agarwal wrote: > > > > > From: Eduardo Valentin > > > > > > > > > > System instability are seen during resume from hibernation when system > > > > > is under heavy CPU load. This is due to the lack of update of sched > > > > > clock data, and the scheduler would then think that heavy CPU hog > > > > > tasks need more time in CPU, causing the system to freeze > > > > > during the unfreezing of tasks. For example, threaded irqs, > > > > > and kernel processes servicing network interface may be delayed > > > > > for several tens of seconds, causing the system to be unreachable. > > > > > The fix for this situation is to mark the sched clock as unstable > > > > > as early as possible in the resume path, leaving it unstable > > > > > for the duration of the resume process. This will force the > > > > > scheduler to attempt to align the sched clock across CPUs using > > > > > the delta with time of day, updating sched clock data. In a post > > > > > hibernation event, we can then mark the sched clock as stable > > > > > again, avoiding unnecessary syncs with time of day on systems > > > > > in which TSC is reliable. > > > > > > > > This makes no frigging sense what so bloody ever. If the clock is > > > > stable, we don't care about sched_clock_data. When it is stable you get > > > > a linear function of the TSC without complicated bits on. > > > > > > > > When it is unstable, only then do we care about the sched_clock_data. > > > > > > > > > > Yeah, maybe what is not clear here is that we covering for situation > > > where clock stability changes over time, e.g. at regular boot clock is > > > stable, hibernation happens, then restore happens in a non-stable clock. > > > > Still confused, who marks the thing unstable? The patch seems to suggest > > you do yourself, but it is not at all clear why. > > > > If TSC really is unstable, then it needs to remain unstable. If the TSC > > really is stable then there is no point in marking is unstable. > > > > Either way something is off, and you're not telling me what. > > > > Hi, Peter > > For your original comment, just wanted to clarify the following: > > 1. After hibernation, the machine can be resumed on a different but compatible > host (these are VM images hibernated) > 2. This means the clock between host1 and host2 can/will be different So the problem is specific to this particular use case. I'm not sure why to impose this hack on hibernation in all cases. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0D20C3F68F for ; Mon, 13 Jan 2020 11:48:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A29F9222C2 for ; Mon, 13 Jan 2020 11:48:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A29F9222C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 15AAD8E0005; Mon, 13 Jan 2020 06:48:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 10B608E0003; Mon, 13 Jan 2020 06:48:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 021468E0005; Mon, 13 Jan 2020 06:48:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0089.hostedemail.com [216.40.44.89]) by kanga.kvack.org (Postfix) with ESMTP id E0FE98E0003 for ; Mon, 13 Jan 2020 06:48:20 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 998E5181AC9BF for ; Mon, 13 Jan 2020 11:48:20 +0000 (UTC) X-FDA: 76372437960.28.aunt47_e94662ca3f43 X-HE-Tag: aunt47_e94662ca3f43 X-Filterd-Recvd-Size: 6389 Received: from mail-ot1-f65.google.com (mail-ot1-f65.google.com [209.85.210.65]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Mon, 13 Jan 2020 11:48:20 +0000 (UTC) Received: by mail-ot1-f65.google.com with SMTP id i15so8662571oto.2 for ; Mon, 13 Jan 2020 03:48:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=LxnhZqYlYpH5Vy9lVIo6OhZ32kMjzMITo+VXkeln1fw=; b=P6nRgggoCVKF3pOJehn5AG3HcJYEqBrHR4wweD1XS3rksIm02fTDFGTVep5xA2wPln 81U47z8KWowoHb5k3ZpLLjoFfZFF7D0SuoCiPhN17yAvmiiPphIMwEUSUajcrM31jPDq RyS25CVvpyeK5hPKBIZ0cEf12tL+MRap1pHnFU8GHWvabf5ViUt/PgOr4Do9NuXCQASY +WrrdYEcRoKkGmXCkb0DZSHk/NkdEPliFlXgXT3A+E9XExfzOH/652lots508fwc2ENh AxILtoXxnYbvemuIm2BWqIHqEenAHYAOfNNPPnpInc9Ow4jsVdjFDCEJk70QCuc37d2N zewA== X-Gm-Message-State: APjAAAWrxa9yOABAH72ufyVa5z+eDs8tUCrm6VJ6DI4vLUciyf/C6Hb3 M8AIyUsfZJChe3HUctMqUdDfOS+d2iFTUNTi36Y= X-Google-Smtp-Source: APXvYqw3Qgw3VCVDMPHWyRb4A+fn8GcFRFWgD+RozWxW13+tMz8RTbaH9oz22LVJAtTKMJUr2mfSKOEXlhCa9eU2PHs= X-Received: by 2002:a05:6830:4b9:: with SMTP id l25mr13198440otd.266.1578916099461; Mon, 13 Jan 2020 03:48:19 -0800 (PST) MIME-Version: 1.0 References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> <20200113101609.GT2844@hirez.programming.kicks-ass.net> <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> In-Reply-To: <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> From: "Rafael J. Wysocki" Date: Mon, 13 Jan 2020 12:48:08 +0100 Message-ID: Subject: Re: [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation To: "Singh, Balbir" Cc: "peterz@infradead.org" , "Valentin, Eduardo" , "boris.ostrovsky@oracle.com" , "linux-kernel@vger.kernel.org" , "Agarwal, Anchal" , "Woodhouse, David" , "vkuznets@redhat.com" , "sstabellini@kernel.org" , "tglx@linutronix.de" , "linux-pm@vger.kernel.org" , "Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com" , "linux-mm@kvack.org" , "jgross@suse.com" , "pavel@ucw.cz" , "axboe@kernel.dk" , "x86@kernel.org" , "roger.pau@citrix.com" , "hpa@zytor.com" , "rjw@rjwysocki.net" , "mingo@redhat.com" , "Kamata, Munehisa" , "bp@alien8.de" , "netdev@vger.kernel.org" , "konrad.wilk@oracle.co" , "len.brown@intel.com" , "davem@davemloft.net" , "fllinden@amaozn.com" , "xen-devel@lists.xenproject.org" Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jan 13, 2020 at 12:43 PM Singh, Balbir wrote: > > On Mon, 2020-01-13 at 11:16 +0100, Peter Zijlstra wrote: > > On Fri, Jan 10, 2020 at 07:35:20AM -0800, Eduardo Valentin wrote: > > > Hey Peter, > > > > > > On Wed, Jan 08, 2020 at 11:50:11AM +0100, Peter Zijlstra wrote: > > > > On Tue, Jan 07, 2020 at 11:45:26PM +0000, Anchal Agarwal wrote: > > > > > From: Eduardo Valentin > > > > > > > > > > System instability are seen during resume from hibernation when system > > > > > is under heavy CPU load. This is due to the lack of update of sched > > > > > clock data, and the scheduler would then think that heavy CPU hog > > > > > tasks need more time in CPU, causing the system to freeze > > > > > during the unfreezing of tasks. For example, threaded irqs, > > > > > and kernel processes servicing network interface may be delayed > > > > > for several tens of seconds, causing the system to be unreachable. > > > > > The fix for this situation is to mark the sched clock as unstable > > > > > as early as possible in the resume path, leaving it unstable > > > > > for the duration of the resume process. This will force the > > > > > scheduler to attempt to align the sched clock across CPUs using > > > > > the delta with time of day, updating sched clock data. In a post > > > > > hibernation event, we can then mark the sched clock as stable > > > > > again, avoiding unnecessary syncs with time of day on systems > > > > > in which TSC is reliable. > > > > > > > > This makes no frigging sense what so bloody ever. If the clock is > > > > stable, we don't care about sched_clock_data. When it is stable you get > > > > a linear function of the TSC without complicated bits on. > > > > > > > > When it is unstable, only then do we care about the sched_clock_data. > > > > > > > > > > Yeah, maybe what is not clear here is that we covering for situation > > > where clock stability changes over time, e.g. at regular boot clock is > > > stable, hibernation happens, then restore happens in a non-stable clock. > > > > Still confused, who marks the thing unstable? The patch seems to suggest > > you do yourself, but it is not at all clear why. > > > > If TSC really is unstable, then it needs to remain unstable. If the TSC > > really is stable then there is no point in marking is unstable. > > > > Either way something is off, and you're not telling me what. > > > > Hi, Peter > > For your original comment, just wanted to clarify the following: > > 1. After hibernation, the machine can be resumed on a different but compatible > host (these are VM images hibernated) > 2. This means the clock between host1 and host2 can/will be different So the problem is specific to this particular use case. I'm not sure why to impose this hack on hibernation in all cases. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9620FC33CAD for ; Mon, 13 Jan 2020 11:48:40 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 737C72081E for ; Mon, 13 Jan 2020 11:48:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 737C72081E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iqyCr-0005Eh-K2; Mon, 13 Jan 2020 11:48:29 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iqyCq-0005EZ-Es for xen-devel@lists.xenproject.org; Mon, 13 Jan 2020 11:48:28 +0000 X-Inumbo-ID: 980b2c14-35fa-11ea-a985-bc764e2007e4 Received: from mail-ot1-f67.google.com (unknown [209.85.210.67]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 980b2c14-35fa-11ea-a985-bc764e2007e4; Mon, 13 Jan 2020 11:48:19 +0000 (UTC) Received: by mail-ot1-f67.google.com with SMTP id a15so8662135otf.1 for ; Mon, 13 Jan 2020 03:48:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=LxnhZqYlYpH5Vy9lVIo6OhZ32kMjzMITo+VXkeln1fw=; b=uWaGfKbQUcy+oJ/MA+dn4BpX9JAnbd0k1VXl9A3EoU0IkQ2+JIzwWhOQNPCFec7XLL xOKOsrxmHticXw0yVNpU9HwU0e1QTMzf0C0ara9e9jjxZLXPDjkBcqME4guqyZB7D7b5 itAZwHLOFZIfdnPxtaalTXlFwxy7x2ZdMcBKyPwxc9rOzlRhheMgN421bg4ZnIFz/BPJ 8pn6XRnSLV0eUJDv0W7LbrGv91quS8Dt8ODFr+wrAYlyJyDb1heECiV3gae6WrR/6P8H i/5Jbd1x6fHGPl7YZR/yCiKRVgiRneBpbxz6CwgMeztQI7/PMJ48lpIIf96AoY2IsF2J NDag== X-Gm-Message-State: APjAAAWk9lRgmlNx4Yb5qSrUMjDVgoWNC/WgignL/ruanlPfchkRFWeb gFcx1zMZJf7JJNLzh7+vll6DOkbyCKsJJY8OXFQ= X-Google-Smtp-Source: APXvYqw3Qgw3VCVDMPHWyRb4A+fn8GcFRFWgD+RozWxW13+tMz8RTbaH9oz22LVJAtTKMJUr2mfSKOEXlhCa9eU2PHs= X-Received: by 2002:a05:6830:4b9:: with SMTP id l25mr13198440otd.266.1578916099461; Mon, 13 Jan 2020 03:48:19 -0800 (PST) MIME-Version: 1.0 References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> <20200113101609.GT2844@hirez.programming.kicks-ass.net> <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> In-Reply-To: <857b42b2e86b2ae09a23f488daada3b1b2836116.camel@amazon.com> From: "Rafael J. Wysocki" Date: Mon, 13 Jan 2020 12:48:08 +0100 Message-ID: To: "Singh, Balbir" Subject: Re: [Xen-devel] [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: "konrad.wilk@oracle.co" , "Valentin, Eduardo" , "len.brown@intel.com" , "peterz@infradead.org" , "x86@kernel.org" , "linux-mm@kvack.org" , "pavel@ucw.cz" , "hpa@zytor.com" , "boris.ostrovsky@oracle.com" , "sstabellini@kernel.org" , "fllinden@amaozn.com" , "Kamata, Munehisa" , "mingo@redhat.com" , "xen-devel@lists.xenproject.org" , "axboe@kernel.dk" , "linux-pm@vger.kernel.org" , "Agarwal, Anchal" , "bp@alien8.de" , "tglx@linutronix.de" , "jgross@suse.com" , "netdev@vger.kernel.org" , "Woodhouse@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com" , "rjw@rjwysocki.net" , "linux-kernel@vger.kernel.org" , "vkuznets@redhat.com" , "davem@davemloft.net" , "Woodhouse, David" , "roger.pau@citrix.com" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" T24gTW9uLCBKYW4gMTMsIDIwMjAgYXQgMTI6NDMgUE0gU2luZ2gsIEJhbGJpciA8c2JsYmlyQGFt YXpvbi5jb20+IHdyb3RlOgo+Cj4gT24gTW9uLCAyMDIwLTAxLTEzIGF0IDExOjE2ICswMTAwLCBQ ZXRlciBaaWpsc3RyYSB3cm90ZToKPiA+IE9uIEZyaSwgSmFuIDEwLCAyMDIwIGF0IDA3OjM1OjIw QU0gLTA4MDAsIEVkdWFyZG8gVmFsZW50aW4gd3JvdGU6Cj4gPiA+IEhleSBQZXRlciwKPiA+ID4K PiA+ID4gT24gV2VkLCBKYW4gMDgsIDIwMjAgYXQgMTE6NTA6MTFBTSArMDEwMCwgUGV0ZXIgWmlq bHN0cmEgd3JvdGU6Cj4gPiA+ID4gT24gVHVlLCBKYW4gMDcsIDIwMjAgYXQgMTE6NDU6MjZQTSAr MDAwMCwgQW5jaGFsIEFnYXJ3YWwgd3JvdGU6Cj4gPiA+ID4gPiBGcm9tOiBFZHVhcmRvIFZhbGVu dGluIDxlZHV2YWxAYW1hem9uLmNvbT4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBTeXN0ZW0gaW5zdGFi aWxpdHkgYXJlIHNlZW4gZHVyaW5nIHJlc3VtZSBmcm9tIGhpYmVybmF0aW9uIHdoZW4gc3lzdGVt Cj4gPiA+ID4gPiBpcyB1bmRlciBoZWF2eSBDUFUgbG9hZC4gVGhpcyBpcyBkdWUgdG8gdGhlIGxh Y2sgb2YgdXBkYXRlIG9mIHNjaGVkCj4gPiA+ID4gPiBjbG9jayBkYXRhLCBhbmQgdGhlIHNjaGVk dWxlciB3b3VsZCB0aGVuIHRoaW5rIHRoYXQgaGVhdnkgQ1BVIGhvZwo+ID4gPiA+ID4gdGFza3Mg bmVlZCBtb3JlIHRpbWUgaW4gQ1BVLCBjYXVzaW5nIHRoZSBzeXN0ZW0gdG8gZnJlZXplCj4gPiA+ ID4gPiBkdXJpbmcgdGhlIHVuZnJlZXppbmcgb2YgdGFza3MuIEZvciBleGFtcGxlLCB0aHJlYWRl ZCBpcnFzLAo+ID4gPiA+ID4gYW5kIGtlcm5lbCBwcm9jZXNzZXMgc2VydmljaW5nIG5ldHdvcmsg aW50ZXJmYWNlIG1heSBiZSBkZWxheWVkCj4gPiA+ID4gPiBmb3Igc2V2ZXJhbCB0ZW5zIG9mIHNl Y29uZHMsIGNhdXNpbmcgdGhlIHN5c3RlbSB0byBiZSB1bnJlYWNoYWJsZS4KPiA+ID4gPiA+IFRo ZSBmaXggZm9yIHRoaXMgc2l0dWF0aW9uIGlzIHRvIG1hcmsgdGhlIHNjaGVkIGNsb2NrIGFzIHVu c3RhYmxlCj4gPiA+ID4gPiBhcyBlYXJseSBhcyBwb3NzaWJsZSBpbiB0aGUgcmVzdW1lIHBhdGgs IGxlYXZpbmcgaXQgdW5zdGFibGUKPiA+ID4gPiA+IGZvciB0aGUgZHVyYXRpb24gb2YgdGhlIHJl c3VtZSBwcm9jZXNzLiBUaGlzIHdpbGwgZm9yY2UgdGhlCj4gPiA+ID4gPiBzY2hlZHVsZXIgdG8g YXR0ZW1wdCB0byBhbGlnbiB0aGUgc2NoZWQgY2xvY2sgYWNyb3NzIENQVXMgdXNpbmcKPiA+ID4g PiA+IHRoZSBkZWx0YSB3aXRoIHRpbWUgb2YgZGF5LCB1cGRhdGluZyBzY2hlZCBjbG9jayBkYXRh LiBJbiBhIHBvc3QKPiA+ID4gPiA+IGhpYmVybmF0aW9uIGV2ZW50LCB3ZSBjYW4gdGhlbiBtYXJr IHRoZSBzY2hlZCBjbG9jayBhcyBzdGFibGUKPiA+ID4gPiA+IGFnYWluLCBhdm9pZGluZyB1bm5l Y2Vzc2FyeSBzeW5jcyB3aXRoIHRpbWUgb2YgZGF5IG9uIHN5c3RlbXMKPiA+ID4gPiA+IGluIHdo aWNoIFRTQyBpcyByZWxpYWJsZS4KPiA+ID4gPgo+ID4gPiA+IFRoaXMgbWFrZXMgbm8gZnJpZ2dp bmcgc2Vuc2Ugd2hhdCBzbyBibG9vZHkgZXZlci4gSWYgdGhlIGNsb2NrIGlzCj4gPiA+ID4gc3Rh YmxlLCB3ZSBkb24ndCBjYXJlIGFib3V0IHNjaGVkX2Nsb2NrX2RhdGEuIFdoZW4gaXQgaXMgc3Rh YmxlIHlvdSBnZXQKPiA+ID4gPiBhIGxpbmVhciBmdW5jdGlvbiBvZiB0aGUgVFNDIHdpdGhvdXQg Y29tcGxpY2F0ZWQgYml0cyBvbi4KPiA+ID4gPgo+ID4gPiA+IFdoZW4gaXQgaXMgdW5zdGFibGUs IG9ubHkgdGhlbiBkbyB3ZSBjYXJlIGFib3V0IHRoZSBzY2hlZF9jbG9ja19kYXRhLgo+ID4gPiA+ Cj4gPiA+Cj4gPiA+IFllYWgsIG1heWJlIHdoYXQgaXMgbm90IGNsZWFyIGhlcmUgaXMgdGhhdCB3 ZSBjb3ZlcmluZyBmb3Igc2l0dWF0aW9uCj4gPiA+IHdoZXJlIGNsb2NrIHN0YWJpbGl0eSBjaGFu Z2VzIG92ZXIgdGltZSwgZS5nLiBhdCByZWd1bGFyIGJvb3QgY2xvY2sgaXMKPiA+ID4gc3RhYmxl LCBoaWJlcm5hdGlvbiBoYXBwZW5zLCB0aGVuIHJlc3RvcmUgaGFwcGVucyBpbiBhIG5vbi1zdGFi bGUgY2xvY2suCj4gPgo+ID4gU3RpbGwgY29uZnVzZWQsIHdobyBtYXJrcyB0aGUgdGhpbmcgdW5z dGFibGU/IFRoZSBwYXRjaCBzZWVtcyB0byBzdWdnZXN0Cj4gPiB5b3UgZG8geW91cnNlbGYsIGJ1 dCBpdCBpcyBub3QgYXQgYWxsIGNsZWFyIHdoeS4KPiA+Cj4gPiBJZiBUU0MgcmVhbGx5IGlzIHVu c3RhYmxlLCB0aGVuIGl0IG5lZWRzIHRvIHJlbWFpbiB1bnN0YWJsZS4gSWYgdGhlIFRTQwo+ID4g cmVhbGx5IGlzIHN0YWJsZSB0aGVuIHRoZXJlIGlzIG5vIHBvaW50IGluIG1hcmtpbmcgaXMgdW5z dGFibGUuCj4gPgo+ID4gRWl0aGVyIHdheSBzb21ldGhpbmcgaXMgb2ZmLCBhbmQgeW91J3JlIG5v dCB0ZWxsaW5nIG1lIHdoYXQuCj4gPgo+Cj4gSGksIFBldGVyCj4KPiBGb3IgeW91ciBvcmlnaW5h bCBjb21tZW50LCBqdXN0IHdhbnRlZCB0byBjbGFyaWZ5IHRoZSBmb2xsb3dpbmc6Cj4KPiAxLiBB ZnRlciBoaWJlcm5hdGlvbiwgdGhlIG1hY2hpbmUgY2FuIGJlIHJlc3VtZWQgb24gYSBkaWZmZXJl bnQgYnV0IGNvbXBhdGlibGUKPiBob3N0ICh0aGVzZSBhcmUgVk0gaW1hZ2VzIGhpYmVybmF0ZWQp Cj4gMi4gVGhpcyBtZWFucyB0aGUgY2xvY2sgYmV0d2VlbiBob3N0MSBhbmQgaG9zdDIgY2FuL3dp bGwgYmUgZGlmZmVyZW50CgpTbyB0aGUgcHJvYmxlbSBpcyBzcGVjaWZpYyB0byB0aGlzIHBhcnRp Y3VsYXIgdXNlIGNhc2UuCgpJJ20gbm90IHN1cmUgd2h5IHRvIGltcG9zZSB0aGlzIGhhY2sgb24g aGliZXJuYXRpb24gaW4gYWxsIGNhc2VzLgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18KWGVuLWRldmVsIG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMu eGVucHJvamVjdC5vcmcKaHR0cHM6Ly9saXN0cy54ZW5wcm9qZWN0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3hlbi1kZXZlbA==