From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28E63C3405F for ; Wed, 19 Feb 2020 18:04:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F150C206DB for ; Wed, 19 Feb 2020 18:04:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="oqMj2W+q" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726722AbgBSSEp (ORCPT ); Wed, 19 Feb 2020 13:04:45 -0500 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:6594 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726514AbgBSSEp (ORCPT ); Wed, 19 Feb 2020 13:04:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1582135485; x=1613671485; h=date:from:to:cc:subject:message-id:references: mime-version:content-transfer-encoding:in-reply-to; bh=3Zqh9Z5ArCKT5eh3kytqNR+atJTyGLoFKV/QsPjClkg=; b=oqMj2W+qDZkw2Dp39HVx5C4B1C7VumFPWUfHdx8b4Q7df7y3V3tVG+2u DBLkOj4GXqIoKyZ0UDXhKSCjnW+xcjN7H/yH1EKmd0dg2I19n+IFGRLqw +eRKjGRs5QUIx/W75ME+xiICZNYLdp0ESbhdmZWprHlMA9Kph9vfqJc+8 8=; IronPort-SDR: SWoQO1RFkW7rgd2CL4S62O3VarDPqWQrz58ohCj8J+VHKH9cgFWDywhjXDZn2lMphuLg99bCok LxZKOSkF49jw== X-IronPort-AV: E=Sophos;i="5.70,461,1574121600"; d="scan'208";a="27530192" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-4ff6265a.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 19 Feb 2020 18:04:42 +0000 Received: from EX13MTAUEE002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-4ff6265a.us-west-2.amazon.com (Postfix) with ESMTPS id 170F4A268C; Wed, 19 Feb 2020 18:04:40 +0000 (UTC) Received: from EX13D08UEE003.ant.amazon.com (10.43.62.118) by EX13MTAUEE002.ant.amazon.com (10.43.62.24) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Wed, 19 Feb 2020 18:04:25 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D08UEE003.ant.amazon.com (10.43.62.118) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Wed, 19 Feb 2020 18:04:25 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Wed, 19 Feb 2020 18:04:24 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id DFEB5403C0; Wed, 19 Feb 2020 18:04:24 +0000 (UTC) Date: Wed, 19 Feb 2020 18:04:24 +0000 From: Anchal Agarwal To: Roger Pau =?iso-8859-1?Q?Monn=E9?= , , , , , , , , , , , , , , , , , , , , , , , , , , , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: Re: [RFC PATCH v3 06/12] xen-blkfront: add callbacks for PM suspend and hibernation Message-ID: <20200219180424.GA17584@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> References: <890c404c585d7790514527f0c021056a7be6e748.1581721799.git.anchalag@amazon.com> <20200217100509.GE4679@Air-de-Roger> <20200217230553.GA8100@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200218091611.GN4679@Air-de-Roger> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200218091611.GN4679@Air-de-Roger> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Feb 18, 2020 at 10:16:11AM +0100, Roger Pau Monné wrote: > On Mon, Feb 17, 2020 at 11:05:53PM +0000, Anchal Agarwal wrote: > > On Mon, Feb 17, 2020 at 11:05:09AM +0100, Roger Pau Monné wrote: > > > On Fri, Feb 14, 2020 at 11:25:34PM +0000, Anchal Agarwal wrote: > > > > From: Munehisa Kamata > > > > > > > Add freeze, thaw and restore callbacks for PM suspend and hibernation > > > > support. All frontend drivers that needs to use PM_HIBERNATION/PM_SUSPEND > > > > events, need to implement these xenbus_driver callbacks. > > > > The freeze handler stops a block-layer queue and disconnect the > > > > frontend from the backend while freeing ring_info and associated resources. > > > > The restore handler re-allocates ring_info and re-connect to the > > > > backend, so the rest of the kernel can continue to use the block device > > > > transparently. Also, the handlers are used for both PM suspend and > > > > hibernation so that we can keep the existing suspend/resume callbacks for > > > > Xen suspend without modification. Before disconnecting from backend, > > > > we need to prevent any new IO from being queued and wait for existing > > > > IO to complete. > > > > > > This is different from Xen (xenstore) initiated suspension, as in that > > > case Linux doesn't flush the rings or disconnects from the backend. > > Yes, AFAIK in xen initiated suspension backend takes care of it. > > No, in Xen initiated suspension backend doesn't take care of flushing > the rings, the frontend has a shadow copy of the ring contents and it > re-issues the requests on resume. > Yes, I meant suspension in general where both xenstore and backend knows system is going under suspension and not flushing of rings. That happens in frontend when backend indicates that state is closing and so on. I may have written it in wrong context. > > > > +static int blkfront_freeze(struct xenbus_device *dev) > > > > +{ > > > > + unsigned int i; > > > > + struct blkfront_info *info = dev_get_drvdata(&dev->dev); > > > > + struct blkfront_ring_info *rinfo; > > > > + /* This would be reasonable timeout as used in xenbus_dev_shutdown() */ > > > > + unsigned int timeout = 5 * HZ; > > > > + int err = 0; > > > > + > > > > + info->connected = BLKIF_STATE_FREEZING; > > > > + > > > > + blk_mq_freeze_queue(info->rq); > > > > + blk_mq_quiesce_queue(info->rq); > > > > + > > > > + for (i = 0; i < info->nr_rings; i++) { > > > > + rinfo = &info->rinfo[i]; > > > > + > > > > + gnttab_cancel_free_callback(&rinfo->callback); > > > > + flush_work(&rinfo->work); > > > > + } > > > > + > > > > + /* Kick the backend to disconnect */ > > > > + xenbus_switch_state(dev, XenbusStateClosing); > > > > > > Are you sure this is safe? > > > > > In my testing running multiple fio jobs, other test scenarios running > > a memory loader works fine. I did not came across a scenario that would > > have failed resume due to blkfront issues unless you can sugest some? > > AFAICT you don't wait for the in-flight requests to be finished, and > just rely on blkback to finish processing those. I'm not sure all > blkback implementations out there can guarantee that. > > The approach used by Xen initiated suspension is to re-issue the > in-flight requests when resuming. I have to admit I don't think this > is the best approach, but I would like to keep both the Xen and the PM > initiated suspension using the same logic, and hence I would request > that you try to re-use the existing resume logic (blkfront_resume). > > > > I don't think you wait for all requests pending on the ring to be > > > finished by the backend, and hence you might loose requests as the > > > ones on the ring would not be re-issued by blkfront_restore AFAICT. > > > > > AFAIU, blk_mq_freeze_queue/blk_mq_quiesce_queue should take care of no used > > request on the shared ring. Also, we I want to pause the queue and flush all > > the pending requests in the shared ring before disconnecting from backend. > > Oh, so blk_mq_freeze_queue does wait for in-flight requests to be > finished. I guess it's fine then. > Ok. > > Quiescing the queue seemed a better option here as we want to make sure ongoing > > requests dispatches are totally drained. > > I should accept that some of these notion is borrowed from how nvme freeze/unfreeze > > is done although its not apple to apple comparison. > > That's fine, but I would still like to requests that you use the same > logic (as much as possible) for both the Xen and the PM initiated > suspension. > > So you either apply this freeze/unfreeze to the Xen suspension (and > drop the re-issuing of requests on resume) or adapt the same approach > as the Xen initiated suspension. Keeping two completely different > approaches to suspension / resume on blkfront is not suitable long > term. > I agree with you on overhaul of xen suspend/resume wrt blkfront is a good idea however, IMO that is a work for future and this patch series should not be blocked for it. What do you think? > Thanks, Roger. > Thanks, Anchal From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIM_ADSP_ALL,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3C16C34056 for ; Wed, 19 Feb 2020 18:05:05 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C5608206DB for ; Wed, 19 Feb 2020 18:05:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="lH2rupTO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C5608206DB Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j4TiJ-0004RD-1k; Wed, 19 Feb 2020 18:04:47 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j4TiH-0004R3-Pj for xen-devel@lists.xenproject.org; Wed, 19 Feb 2020 18:04:45 +0000 X-Inumbo-ID: 4ed27808-5342-11ea-ade5-bc764e2007e4 Received: from smtp-fw-33001.amazon.com (unknown [207.171.190.10]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 4ed27808-5342-11ea-ade5-bc764e2007e4; Wed, 19 Feb 2020 18:04:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1582135486; x=1613671486; h=date:from:to:cc:subject:message-id:references: mime-version:content-transfer-encoding:in-reply-to; bh=3Zqh9Z5ArCKT5eh3kytqNR+atJTyGLoFKV/QsPjClkg=; b=lH2rupTOJymR4WSs4/i0nFmQigGTmc6GRkYVw5DAKnIk6l4GcXptV573 a96BpskBq+8VNmUSvmAV1r5W6TlIzf0i7B+nXYyhdZotcDUwKqPTa1T8L LKFN4UmxZIitM8IEtaBg4Y6VFGmaQFWXJaGISeT8UTqcO2xbwrJGhr4aC g=; IronPort-SDR: SWoQO1RFkW7rgd2CL4S62O3VarDPqWQrz58ohCj8J+VHKH9cgFWDywhjXDZn2lMphuLg99bCok LxZKOSkF49jw== X-IronPort-AV: E=Sophos;i="5.70,461,1574121600"; d="scan'208";a="27530192" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-4ff6265a.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 19 Feb 2020 18:04:42 +0000 Received: from EX13MTAUEE002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-4ff6265a.us-west-2.amazon.com (Postfix) with ESMTPS id 170F4A268C; Wed, 19 Feb 2020 18:04:40 +0000 (UTC) Received: from EX13D08UEE003.ant.amazon.com (10.43.62.118) by EX13MTAUEE002.ant.amazon.com (10.43.62.24) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Wed, 19 Feb 2020 18:04:25 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D08UEE003.ant.amazon.com (10.43.62.118) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Wed, 19 Feb 2020 18:04:25 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Wed, 19 Feb 2020 18:04:24 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id DFEB5403C0; Wed, 19 Feb 2020 18:04:24 +0000 (UTC) Date: Wed, 19 Feb 2020 18:04:24 +0000 From: Anchal Agarwal To: Roger Pau =?iso-8859-1?Q?Monn=E9?= , , , , , , , , , , , , , , , , , , , , , , , , , , , , Message-ID: <20200219180424.GA17584@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> References: <890c404c585d7790514527f0c021056a7be6e748.1581721799.git.anchalag@amazon.com> <20200217100509.GE4679@Air-de-Roger> <20200217230553.GA8100@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200218091611.GN4679@Air-de-Roger> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200218091611.GN4679@Air-de-Roger> User-Agent: Mutt/1.5.21 (2010-09-15) Precedence: Bulk Subject: Re: [Xen-devel] [RFC PATCH v3 06/12] xen-blkfront: add callbacks for PM suspend and hibernation X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: eduval@amazon.com, len.brown@intel.com, peterz@infradead.org, benh@kernel.crashing.org, x86@kernel.org, linux-mm@kvack.org, pavel@ucw.cz, hpa@zytor.com, tglx@linutronix.de, sstabellini@kernel.org, fllinden@amaozn.com, kamatam@amazon.com, mingo@redhat.com, xen-devel@lists.xenproject.org, sblbir@amazon.com, axboe@kernel.dk, konrad.wilk@oracle.com, bp@alien8.de, boris.ostrovsky@oracle.com, jgross@suse.com, netdev@vger.kernel.org, linux-pm@vger.kernel.org, rjw@rjwysocki.net, linux-kernel@vger.kernel.org, vkuznets@redhat.com, davem@davemloft.net, dwmw@amazon.co.uk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" T24gVHVlLCBGZWIgMTgsIDIwMjAgYXQgMTA6MTY6MTFBTSArMDEwMCwgUm9nZXIgUGF1IE1vbm7D qSB3cm90ZToKPiBPbiBNb24sIEZlYiAxNywgMjAyMCBhdCAxMTowNTo1M1BNICswMDAwLCBBbmNo YWwgQWdhcndhbCB3cm90ZToKPiA+IE9uIE1vbiwgRmViIDE3LCAyMDIwIGF0IDExOjA1OjA5QU0g KzAxMDAsIFJvZ2VyIFBhdSBNb25uw6kgd3JvdGU6Cj4gPiA+IE9uIEZyaSwgRmViIDE0LCAyMDIw IGF0IDExOjI1OjM0UE0gKzAwMDAsIEFuY2hhbCBBZ2Fyd2FsIHdyb3RlOgo+ID4gPiA+IEZyb206 IE11bmVoaXNhIEthbWF0YSA8a2FtYXRhbUBhbWF6b24uY29tCj4gPiA+ID4gCj4gPiA+ID4gQWRk IGZyZWV6ZSwgdGhhdyBhbmQgcmVzdG9yZSBjYWxsYmFja3MgZm9yIFBNIHN1c3BlbmQgYW5kIGhp YmVybmF0aW9uCj4gPiA+ID4gc3VwcG9ydC4gQWxsIGZyb250ZW5kIGRyaXZlcnMgdGhhdCBuZWVk cyB0byB1c2UgUE1fSElCRVJOQVRJT04vUE1fU1VTUEVORAo+ID4gPiA+IGV2ZW50cywgbmVlZCB0 byBpbXBsZW1lbnQgdGhlc2UgeGVuYnVzX2RyaXZlciBjYWxsYmFja3MuCj4gPiA+ID4gVGhlIGZy ZWV6ZSBoYW5kbGVyIHN0b3BzIGEgYmxvY2stbGF5ZXIgcXVldWUgYW5kIGRpc2Nvbm5lY3QgdGhl Cj4gPiA+ID4gZnJvbnRlbmQgZnJvbSB0aGUgYmFja2VuZCB3aGlsZSBmcmVlaW5nIHJpbmdfaW5m byBhbmQgYXNzb2NpYXRlZCByZXNvdXJjZXMuCj4gPiA+ID4gVGhlIHJlc3RvcmUgaGFuZGxlciBy ZS1hbGxvY2F0ZXMgcmluZ19pbmZvIGFuZCByZS1jb25uZWN0IHRvIHRoZQo+ID4gPiA+IGJhY2tl bmQsIHNvIHRoZSByZXN0IG9mIHRoZSBrZXJuZWwgY2FuIGNvbnRpbnVlIHRvIHVzZSB0aGUgYmxv Y2sgZGV2aWNlCj4gPiA+ID4gdHJhbnNwYXJlbnRseS4gQWxzbywgdGhlIGhhbmRsZXJzIGFyZSB1 c2VkIGZvciBib3RoIFBNIHN1c3BlbmQgYW5kCj4gPiA+ID4gaGliZXJuYXRpb24gc28gdGhhdCB3 ZSBjYW4ga2VlcCB0aGUgZXhpc3Rpbmcgc3VzcGVuZC9yZXN1bWUgY2FsbGJhY2tzIGZvcgo+ID4g PiA+IFhlbiBzdXNwZW5kIHdpdGhvdXQgbW9kaWZpY2F0aW9uLiBCZWZvcmUgZGlzY29ubmVjdGlu ZyBmcm9tIGJhY2tlbmQsCj4gPiA+ID4gd2UgbmVlZCB0byBwcmV2ZW50IGFueSBuZXcgSU8gZnJv bSBiZWluZyBxdWV1ZWQgYW5kIHdhaXQgZm9yIGV4aXN0aW5nCj4gPiA+ID4gSU8gdG8gY29tcGxl dGUuCj4gPiA+IAo+ID4gPiBUaGlzIGlzIGRpZmZlcmVudCBmcm9tIFhlbiAoeGVuc3RvcmUpIGlu aXRpYXRlZCBzdXNwZW5zaW9uLCBhcyBpbiB0aGF0Cj4gPiA+IGNhc2UgTGludXggZG9lc24ndCBm bHVzaCB0aGUgcmluZ3Mgb3IgZGlzY29ubmVjdHMgZnJvbSB0aGUgYmFja2VuZC4KPiA+IFllcywg QUZBSUsgaW4geGVuIGluaXRpYXRlZCBzdXNwZW5zaW9uIGJhY2tlbmQgdGFrZXMgY2FyZSBvZiBp dC4gCj4gCj4gTm8sIGluIFhlbiBpbml0aWF0ZWQgc3VzcGVuc2lvbiBiYWNrZW5kIGRvZXNuJ3Qg dGFrZSBjYXJlIG9mIGZsdXNoaW5nCj4gdGhlIHJpbmdzLCB0aGUgZnJvbnRlbmQgaGFzIGEgc2hh ZG93IGNvcHkgb2YgdGhlIHJpbmcgY29udGVudHMgYW5kIGl0Cj4gcmUtaXNzdWVzIHRoZSByZXF1 ZXN0cyBvbiByZXN1bWUuCj4gClllcywgSSBtZWFudCBzdXNwZW5zaW9uIGluIGdlbmVyYWwgd2hl cmUgYm90aCB4ZW5zdG9yZSBhbmQgYmFja2VuZCBrbm93cwpzeXN0ZW0gaXMgZ29pbmcgdW5kZXIg c3VzcGVuc2lvbiBhbmQgbm90IGZsdXNoaW5nIG9mIHJpbmdzLiBUaGF0IGhhcHBlbnMKaW4gZnJv bnRlbmQgd2hlbiBiYWNrZW5kIGluZGljYXRlcyB0aGF0IHN0YXRlIGlzIGNsb3NpbmcgYW5kIHNv IG9uLgpJIG1heSBoYXZlIHdyaXR0ZW4gaXQgaW4gd3JvbmcgY29udGV4dC4KPiA+ID4gPiArc3Rh dGljIGludCBibGtmcm9udF9mcmVlemUoc3RydWN0IHhlbmJ1c19kZXZpY2UgKmRldikKPiA+ID4g PiArewo+ID4gPiA+ICsJdW5zaWduZWQgaW50IGk7Cj4gPiA+ID4gKwlzdHJ1Y3QgYmxrZnJvbnRf aW5mbyAqaW5mbyA9IGRldl9nZXRfZHJ2ZGF0YSgmZGV2LT5kZXYpOwo+ID4gPiA+ICsJc3RydWN0 IGJsa2Zyb250X3JpbmdfaW5mbyAqcmluZm87Cj4gPiA+ID4gKwkvKiBUaGlzIHdvdWxkIGJlIHJl YXNvbmFibGUgdGltZW91dCBhcyB1c2VkIGluIHhlbmJ1c19kZXZfc2h1dGRvd24oKSAqLwo+ID4g PiA+ICsJdW5zaWduZWQgaW50IHRpbWVvdXQgPSA1ICogSFo7Cj4gPiA+ID4gKwlpbnQgZXJyID0g MDsKPiA+ID4gPiArCj4gPiA+ID4gKwlpbmZvLT5jb25uZWN0ZWQgPSBCTEtJRl9TVEFURV9GUkVF WklORzsKPiA+ID4gPiArCj4gPiA+ID4gKwlibGtfbXFfZnJlZXplX3F1ZXVlKGluZm8tPnJxKTsK PiA+ID4gPiArCWJsa19tcV9xdWllc2NlX3F1ZXVlKGluZm8tPnJxKTsKPiA+ID4gPiArCj4gPiA+ ID4gKwlmb3IgKGkgPSAwOyBpIDwgaW5mby0+bnJfcmluZ3M7IGkrKykgewo+ID4gPiA+ICsJCXJp bmZvID0gJmluZm8tPnJpbmZvW2ldOwo+ID4gPiA+ICsKPiA+ID4gPiArCQlnbnR0YWJfY2FuY2Vs X2ZyZWVfY2FsbGJhY2soJnJpbmZvLT5jYWxsYmFjayk7Cj4gPiA+ID4gKwkJZmx1c2hfd29yaygm cmluZm8tPndvcmspOwo+ID4gPiA+ICsJfQo+ID4gPiA+ICsKPiA+ID4gPiArCS8qIEtpY2sgdGhl IGJhY2tlbmQgdG8gZGlzY29ubmVjdCAqLwo+ID4gPiA+ICsJeGVuYnVzX3N3aXRjaF9zdGF0ZShk ZXYsIFhlbmJ1c1N0YXRlQ2xvc2luZyk7Cj4gPiA+IAo+ID4gPiBBcmUgeW91IHN1cmUgdGhpcyBp cyBzYWZlPwo+ID4gPiAKPiA+IEluIG15IHRlc3RpbmcgcnVubmluZyBtdWx0aXBsZSBmaW8gam9i cywgb3RoZXIgdGVzdCBzY2VuYXJpb3MgcnVubmluZwo+ID4gYSBtZW1vcnkgbG9hZGVyIHdvcmtz IGZpbmUuIEkgZGlkIG5vdCBjYW1lIGFjcm9zcyBhIHNjZW5hcmlvIHRoYXQgd291bGQKPiA+IGhh dmUgZmFpbGVkIHJlc3VtZSBkdWUgdG8gYmxrZnJvbnQgaXNzdWVzIHVubGVzcyB5b3UgY2FuIHN1 Z2VzdCBzb21lPwo+IAo+IEFGQUlDVCB5b3UgZG9uJ3Qgd2FpdCBmb3IgdGhlIGluLWZsaWdodCBy ZXF1ZXN0cyB0byBiZSBmaW5pc2hlZCwgYW5kCj4ganVzdCByZWx5IG9uIGJsa2JhY2sgdG8gZmlu aXNoIHByb2Nlc3NpbmcgdGhvc2UuIEknbSBub3Qgc3VyZSBhbGwKPiBibGtiYWNrIGltcGxlbWVu dGF0aW9ucyBvdXQgdGhlcmUgY2FuIGd1YXJhbnRlZSB0aGF0Lgo+IAo+IFRoZSBhcHByb2FjaCB1 c2VkIGJ5IFhlbiBpbml0aWF0ZWQgc3VzcGVuc2lvbiBpcyB0byByZS1pc3N1ZSB0aGUKPiBpbi1m bGlnaHQgcmVxdWVzdHMgd2hlbiByZXN1bWluZy4gSSBoYXZlIHRvIGFkbWl0IEkgZG9uJ3QgdGhp bmsgdGhpcwo+IGlzIHRoZSBiZXN0IGFwcHJvYWNoLCBidXQgSSB3b3VsZCBsaWtlIHRvIGtlZXAg Ym90aCB0aGUgWGVuIGFuZCB0aGUgUE0KPiBpbml0aWF0ZWQgc3VzcGVuc2lvbiB1c2luZyB0aGUg c2FtZSBsb2dpYywgYW5kIGhlbmNlIEkgd291bGQgcmVxdWVzdAo+IHRoYXQgeW91IHRyeSB0byBy ZS11c2UgdGhlIGV4aXN0aW5nIHJlc3VtZSBsb2dpYyAoYmxrZnJvbnRfcmVzdW1lKS4KPiAKPiA+ ID4gSSBkb24ndCB0aGluayB5b3Ugd2FpdCBmb3IgYWxsIHJlcXVlc3RzIHBlbmRpbmcgb24gdGhl IHJpbmcgdG8gYmUKPiA+ID4gZmluaXNoZWQgYnkgdGhlIGJhY2tlbmQsIGFuZCBoZW5jZSB5b3Ug bWlnaHQgbG9vc2UgcmVxdWVzdHMgYXMgdGhlCj4gPiA+IG9uZXMgb24gdGhlIHJpbmcgd291bGQg bm90IGJlIHJlLWlzc3VlZCBieSBibGtmcm9udF9yZXN0b3JlIEFGQUlDVC4KPiA+ID4gCj4gPiBB RkFJVSwgYmxrX21xX2ZyZWV6ZV9xdWV1ZS9ibGtfbXFfcXVpZXNjZV9xdWV1ZSBzaG91bGQgdGFr ZSBjYXJlIG9mIG5vIHVzZWQKPiA+IHJlcXVlc3Qgb24gdGhlIHNoYXJlZCByaW5nLiBBbHNvLCB3 ZSBJIHdhbnQgdG8gcGF1c2UgdGhlIHF1ZXVlIGFuZCBmbHVzaCBhbGwKPiA+IHRoZSBwZW5kaW5n IHJlcXVlc3RzIGluIHRoZSBzaGFyZWQgcmluZyBiZWZvcmUgZGlzY29ubmVjdGluZyBmcm9tIGJh Y2tlbmQuCj4gCj4gT2gsIHNvIGJsa19tcV9mcmVlemVfcXVldWUgZG9lcyB3YWl0IGZvciBpbi1m bGlnaHQgcmVxdWVzdHMgdG8gYmUKPiBmaW5pc2hlZC4gSSBndWVzcyBpdCdzIGZpbmUgdGhlbi4K PiAKT2suCj4gPiBRdWllc2NpbmcgdGhlIHF1ZXVlIHNlZW1lZCBhIGJldHRlciBvcHRpb24gaGVy ZSBhcyB3ZSB3YW50IHRvIG1ha2Ugc3VyZSBvbmdvaW5nCj4gPiByZXF1ZXN0cyBkaXNwYXRjaGVz IGFyZSB0b3RhbGx5IGRyYWluZWQuCj4gPiBJIHNob3VsZCBhY2NlcHQgdGhhdCBzb21lIG9mIHRo ZXNlIG5vdGlvbiBpcyBib3Jyb3dlZCBmcm9tIGhvdyBudm1lIGZyZWV6ZS91bmZyZWV6ZSAKPiA+ IGlzIGRvbmUgYWx0aG91Z2ggaXRzIG5vdCBhcHBsZSB0byBhcHBsZSBjb21wYXJpc29uLgo+IAo+ IFRoYXQncyBmaW5lLCBidXQgSSB3b3VsZCBzdGlsbCBsaWtlIHRvIHJlcXVlc3RzIHRoYXQgeW91 IHVzZSB0aGUgc2FtZQo+IGxvZ2ljIChhcyBtdWNoIGFzIHBvc3NpYmxlKSBmb3IgYm90aCB0aGUg WGVuIGFuZCB0aGUgUE0gaW5pdGlhdGVkCj4gc3VzcGVuc2lvbi4KPiAKPiBTbyB5b3UgZWl0aGVy IGFwcGx5IHRoaXMgZnJlZXplL3VuZnJlZXplIHRvIHRoZSBYZW4gc3VzcGVuc2lvbiAoYW5kCj4g ZHJvcCB0aGUgcmUtaXNzdWluZyBvZiByZXF1ZXN0cyBvbiByZXN1bWUpIG9yIGFkYXB0IHRoZSBz YW1lIGFwcHJvYWNoCj4gYXMgdGhlIFhlbiBpbml0aWF0ZWQgc3VzcGVuc2lvbi4gS2VlcGluZyB0 d28gY29tcGxldGVseSBkaWZmZXJlbnQKPiBhcHByb2FjaGVzIHRvIHN1c3BlbnNpb24gLyByZXN1 bWUgb24gYmxrZnJvbnQgaXMgbm90IHN1aXRhYmxlIGxvbmcKPiB0ZXJtLgo+IApJIGFncmVlIHdp dGggeW91IG9uIG92ZXJoYXVsIG9mIHhlbiBzdXNwZW5kL3Jlc3VtZSB3cnQgYmxrZnJvbnQgaXMg YSBnb29kCmlkZWEgaG93ZXZlciwgSU1PIHRoYXQgaXMgYSB3b3JrIGZvciBmdXR1cmUgYW5kIHRo aXMgcGF0Y2ggc2VyaWVzIHNob3VsZCAKbm90IGJlIGJsb2NrZWQgZm9yIGl0LiBXaGF0IGRvIHlv dSB0aGluaz8KPiBUaGFua3MsIFJvZ2VyLgo+IApUaGFua3MsCkFuY2hhbAoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVsIG1haWxpbmcgbGlz dApYZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcKaHR0cHM6Ly9saXN0cy54ZW5wcm9qZWN0 Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3hlbi1kZXZlbA==