From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A968C43603 for ; Mon, 16 Dec 2019 10:01:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 29C4020725 for ; Mon, 16 Dec 2019 10:01:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="RxHjRGXG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727165AbfLPKBq (ORCPT ); Mon, 16 Dec 2019 05:01:46 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:21745 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726992AbfLPKBq (ORCPT ); Mon, 16 Dec 2019 05:01:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1576490505; x=1608026505; h=from:to:cc:subject:date:message-id:mime-version: in-reply-to:content-transfer-encoding; bh=LtdCtzcshD6qXVmdgBHlLj8wpUXd5CRe7nngX+q2rNE=; b=RxHjRGXGcitU7zINvYSunUxuDMaDnNqXl2tAbj98wo9b/4uh+OEMvpqR NGML8rpEW84jk8G4EUfxBzdVYVdgnyeeg+i7vO7CXn9TdLCPZC11LOYyv CZUWOqCLHxO7Gcm19CsRCrE/9+KzqUMkdMDKXQWzhdZy30xFySaeVDpgt w=; IronPort-SDR: +nhKU2Tl8MS+eqoV/mR0K7/tezXU9G9id8VSRKkd7HdhYB1SCvtQW6h/4TOmb27zuQKsgd/HFA GWz4PZC1oE7w== X-IronPort-AV: E=Sophos;i="5.69,321,1571702400"; d="scan'208";a="7785002" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 16 Dec 2019 10:01:43 +0000 Received: from EX13MTAUEA001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com (Postfix) with ESMTPS id EE82FA1DDE; Mon, 16 Dec 2019 10:01:41 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA001.ant.amazon.com (10.43.61.82) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 16 Dec 2019 10:01:12 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.160.100) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 16 Dec 2019 10:01:07 +0000 From: SeongJae Park To: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= CC: , , , , , SeongJae Park , , , , Subject: Re: Re: [Xen-devel] [PATCH v9 2/4] xen/blkback: Squeeze page pools if a memory pressure is detected Date: Mon, 16 Dec 2019 11:00:52 +0100 Message-ID: <20191216100052.27361-1-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 In-Reply-To: <20191216093755.GJ11756@Air-de-Roger> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.43.160.100] X-ClientProxiedBy: EX13D16UWB001.ant.amazon.com (10.43.161.17) To EX13D31EUA001.ant.amazon.com (10.43.165.15) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On Mon, 16 Dec 2019 10:37:55 +0100 "Roger Pau Monné" wrote: > On Fri, Dec 13, 2019 at 03:35:44PM +0000, SeongJae Park wrote: > > Each `blkif` has a free pages pool for the grant mapping. The size of > > the pool starts from zero and is increased on demand while processing > > the I/O requests. If current I/O requests handling is finished or 100 > > milliseconds has passed since last I/O requests handling, it checks and > > shrinks the pool to not exceed the size limit, `max_buffer_pages`. > > > > Therefore, host administrators can cause memory pressure in blkback by > > attaching a large number of block devices and inducing I/O. Such > > problematic situations can be avoided by limiting the maximum number of > > devices that can be attached, but finding the optimal limit is not so > > easy. Improper set of the limit can results in memory pressure or a > > resource underutilization. This commit avoids such problematic > > situations by squeezing the pools (returns every free page in the pool > > to the system) for a while (users can set this duration via a module > > parameter) if memory pressure is detected. > > > > Discussions > > =========== > > > > The `blkback`'s original shrinking mechanism returns only pages in the > > pool which are not currently be used by `blkback` to the system. In > > other words, the pages that are not mapped with granted pages. Because > > this commit is changing only the shrink limit but still uses the same > > freeing mechanism it does not touch pages which are currently mapping > > grants. > > > > Once memory pressure is detected, this commit keeps the squeezing limit > > for a user-specified time duration. The duration should be neither too > > long nor too short. If it is too long, the squeezing incurring overhead > > can reduce the I/O performance. If it is too short, `blkback` will not > > free enough pages to reduce the memory pressure. This commit sets the > > value as `10 milliseconds` by default because it is a short time in > > terms of I/O while it is a long time in terms of memory operations. > > Also, as the original shrinking mechanism works for at least every 100 > > milliseconds, this could be a somewhat reasonable choice. I also tested > > other durations (refer to the below section for more details) and > > confirmed that 10 milliseconds is the one that works best with the test. > > That said, the proper duration depends on actual configurations and > > workloads. That's why this commit allows users to set the duration as a > > module parameter. > > > > Memory Pressure Test > > ==================== > > > > To show how this commit fixes the memory pressure situation well, I > > configured a test environment on a xen-running virtualization system. > > On the `blkfront` running guest instances, I attach a large number of > > network-backed volume devices and induce I/O to those. Meanwhile, I > > measure the number of pages that swapped in (pswpin) and out (pswpout) > > on the `blkback` running guest. The test ran twice, once for the > > `blkback` before this commit and once for that after this commit. As > > shown below, this commit has dramatically reduced the memory pressure: > > > > pswpin pswpout > > before 76,672 185,799 > > after 212 3,325 > > > > Optimal Aggressive Shrinking Duration > > ------------------------------------- > > > > To find a best squeezing duration, I repeated the test with three > > different durations (1ms, 10ms, and 100ms). The results are as below: > > > > duration pswpin pswpout > > 1 852 6,424 > > 10 212 3,325 > > 100 203 3,340 > > > > As expected, the memory pressure has decreased as the duration is > > increased, but the reduction stopped from the `10ms`. Based on this > > results, I chose the default duration as 10ms. > > > > Performance Overhead Test > > ========================= > > > > This commit could incur I/O performance degradation under severe memory > > pressure because the squeezing will require more page allocations per > > I/O. To show the overhead, I artificially made a worst-case squeezing > > situation and measured the I/O performance of a `blkfront` running > > guest. > > > > For the artificial squeezing, I set the `blkback.max_buffer_pages` using > > the `/sys/module/xen_blkback/parameters/max_buffer_pages` file. In this > > test, I set the value to `1024` and `0`. The `1024` is the default > > value. Setting the value as `0` is same to a situation doing the > > squeezing always (worst-case). > > > > For the I/O performance measurement, I run a simple `dd` command 5 times > > as below and collect the 'MB/s' results. > > > > $ for i in {1..5}; do dd if=/dev/zero of=file \ > > bs=4k count=$((256*512)); sync; done > > > > If the underlying block device is slow enough, the squeezing overhead > > could be hidden. For the reason, I do this test for both a slow block > > device and a fast block device. I use a popular cloud block storage > > service, ebs[1] as a slow device and the ramdisk block device[2] for the > > fast device. > > > > The results are as below. 'max_pgs' represents the value of the > > `blkback.max_buffer_pages` parameter. > > > > On the slow block device > > ------------------------ > > > > max_pgs Min Max Median Avg Stddev > > 0 38.7 45.8 38.7 40.12 3.1752165 > > 1024 38.7 45.8 38.7 40.12 3.1752165 > > No difference proven at 95.0% confidence > > > > On the fast block device > > ------------------------ > > > > max_pgs Min Max Median Avg Stddev > > 0 417 423 420 419.4 2.5099801 > > 1024 414 425 416 417.8 4.4384682 > > No difference proven at 95.0% confidence > > > > In short, even worst case squeezing on ramdisk based fast block device > > makes no visible performance degradation. Please note that this is just > > a very simple and minimal test. On systems using super-fast block > > devices and a special I/O workload, the results might be different. If > > you have any doubt, test on your machine with your workload to find the > > optimal squeezing duration for you. > > > > [1] https://aws.amazon.com/ebs/ > > [2] https://www.kernel.org/doc/html/latest/admin-guide/blockdev/ramdisk.html I forgot to update this section. It contains two evaluation results which has no big difference and also describes one test in wrong way (it induced direct IO to the ramdisk). For example, I would like to update this section as below: Performance Overhead Test ========================= This commit could incur I/O performance degradation under severe memory pressure because the squeezing will require more page allocations per I/O. To show the overhead, I artificially made a worst-case squeezing situation and measured the I/O performance of a `blkfront` running guest. For the artificial squeezing, I set the `blkback.max_buffer_pages` using the `/sys/module/xen_blkback/parameters/max_buffer_pages` file. In this test, I set the value to `1024` and `0`. The `1024` is the default value. Setting the value as `0` is same to a situation doing the squeezing always (worst-case). If the underlying block device is slow enough, the squeezing overhead could be hidden. For the reason, I use a fast block device, namely the rbd[1]: # xl block-attach guest phy:/dev/ram0 xvdb w For the I/O performance measurement, I run a simple `dd` command 5 times directly to the device as below and collect the 'MB/s' results. $ for i in {1..5}; do dd if=/dev/zero of=/dev/xvdb \ bs=4k count=$((256*512)); sync; done The results are as below. 'max_pgs' represents the value of the `blkback.max_buffer_pages` parameter. max_pgs Min Max Median Avg Stddev 0 417 423 420 419.4 2.5099801 1024 414 425 416 417.8 4.4384682 No difference proven at 95.0% confidence In short, even worst case squeezing on ramdisk based fast block device makes no visible performance degradation. Please note that this is just a very simple and minimal test. On systems using super-fast block devices and a special I/O workload, the results might be different. If you have any doubt, test on your machine with your workload to find the optimal squeezing duration for you. [1] https://www.kernel.org/doc/html/latest/admin-guide/blockdev/ramdisk.html > > > > Signed-off-by: SeongJae Park > > Reviewed-by: Roger Pau Monné Appreciate for your reviews. You made this patch much better! Thanks, SeongJae Park > > Thanks, Roger. > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=DKIM_ADSP_ALL,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61B61C43603 for ; Mon, 16 Dec 2019 10:01:56 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 27D69207FF for ; Mon, 16 Dec 2019 10:01:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="V3Zk7UVn" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 27D69207FF Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ignCD-0006Jv-Pr; Mon, 16 Dec 2019 10:01:45 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ignCD-0006Jn-A6 for xen-devel@lists.xenproject.org; Mon, 16 Dec 2019 10:01:45 +0000 X-Inumbo-ID: 10aa8e1a-1feb-11ea-9377-12813bfff9fa Received: from smtp-fw-6002.amazon.com (unknown [52.95.49.90]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 10aa8e1a-1feb-11ea-9377-12813bfff9fa; Mon, 16 Dec 2019 10:01:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1576490504; x=1608026504; h=from:to:cc:subject:date:message-id:mime-version: in-reply-to:content-transfer-encoding; bh=LtdCtzcshD6qXVmdgBHlLj8wpUXd5CRe7nngX+q2rNE=; b=V3Zk7UVngsR4Jr1qElyuINrcB2ZKcmi5uPY8FUkiSIkEj0ouYpH3wYdz AAlQ8OxOlac0fAgza9JdSNY/l1XvGq0rNOfo33p1yxV5Cb6CqChayTpbH 09u01f5TQBRputdmVGwZ+JQEhNM69TqqzkSON2Gn0bvnKbfQXeY0IGw+R s=; IronPort-SDR: +nhKU2Tl8MS+eqoV/mR0K7/tezXU9G9id8VSRKkd7HdhYB1SCvtQW6h/4TOmb27zuQKsgd/HFA GWz4PZC1oE7w== X-IronPort-AV: E=Sophos;i="5.69,321,1571702400"; d="scan'208";a="7785002" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 16 Dec 2019 10:01:43 +0000 Received: from EX13MTAUEA001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com (Postfix) with ESMTPS id EE82FA1DDE; Mon, 16 Dec 2019 10:01:41 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA001.ant.amazon.com (10.43.61.82) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 16 Dec 2019 10:01:12 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.160.100) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 16 Dec 2019 10:01:07 +0000 From: SeongJae Park To: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Date: Mon, 16 Dec 2019 11:00:52 +0100 Message-ID: <20191216100052.27361-1-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 In-Reply-To: <20191216093755.GJ11756@Air-de-Roger> X-Originating-IP: [10.43.160.100] X-ClientProxiedBy: EX13D16UWB001.ant.amazon.com (10.43.161.17) To EX13D31EUA001.ant.amazon.com (10.43.165.15) Precedence: Bulk Subject: Re: [Xen-devel] [PATCH v9 2/4] xen/blkback: Squeeze page pools if a memory pressure is detected X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: jgross@suse.com, axboe@kernel.dk, sjpark@amazon.com, konrad.wilk@oracle.com, pdurrant@amazon.com, SeongJae Park , linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, sj38.park@gmail.com, xen-devel@lists.xenproject.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" T24gTW9uLCAxNiBEZWMgMjAxOSAxMDozNzo1NSArMDEwMCAiUm9nZXIgUGF1IE1vbm7DqSIgPHJv Z2VyLnBhdUBjaXRyaXguY29tPiB3cm90ZToKCj4gT24gRnJpLCBEZWMgMTMsIDIwMTkgYXQgMDM6 MzU6NDRQTSArMDAwMCwgU2VvbmdKYWUgUGFyayB3cm90ZToKPiA+IEVhY2ggYGJsa2lmYCBoYXMg YSBmcmVlIHBhZ2VzIHBvb2wgZm9yIHRoZSBncmFudCBtYXBwaW5nLiAgVGhlIHNpemUgb2YKPiA+ IHRoZSBwb29sIHN0YXJ0cyBmcm9tIHplcm8gYW5kIGlzIGluY3JlYXNlZCBvbiBkZW1hbmQgd2hp bGUgcHJvY2Vzc2luZwo+ID4gdGhlIEkvTyByZXF1ZXN0cy4gIElmIGN1cnJlbnQgSS9PIHJlcXVl c3RzIGhhbmRsaW5nIGlzIGZpbmlzaGVkIG9yIDEwMAo+ID4gbWlsbGlzZWNvbmRzIGhhcyBwYXNz ZWQgc2luY2UgbGFzdCBJL08gcmVxdWVzdHMgaGFuZGxpbmcsIGl0IGNoZWNrcyBhbmQKPiA+IHNo cmlua3MgdGhlIHBvb2wgdG8gbm90IGV4Y2VlZCB0aGUgc2l6ZSBsaW1pdCwgYG1heF9idWZmZXJf cGFnZXNgLgo+ID4gCj4gPiBUaGVyZWZvcmUsIGhvc3QgYWRtaW5pc3RyYXRvcnMgY2FuIGNhdXNl IG1lbW9yeSBwcmVzc3VyZSBpbiBibGtiYWNrIGJ5Cj4gPiBhdHRhY2hpbmcgYSBsYXJnZSBudW1i ZXIgb2YgYmxvY2sgZGV2aWNlcyBhbmQgaW5kdWNpbmcgSS9PLiAgU3VjaAo+ID4gcHJvYmxlbWF0 aWMgc2l0dWF0aW9ucyBjYW4gYmUgYXZvaWRlZCBieSBsaW1pdGluZyB0aGUgbWF4aW11bSBudW1i ZXIgb2YKPiA+IGRldmljZXMgdGhhdCBjYW4gYmUgYXR0YWNoZWQsIGJ1dCBmaW5kaW5nIHRoZSBv cHRpbWFsIGxpbWl0IGlzIG5vdCBzbwo+ID4gZWFzeS4gIEltcHJvcGVyIHNldCBvZiB0aGUgbGlt aXQgY2FuIHJlc3VsdHMgaW4gbWVtb3J5IHByZXNzdXJlIG9yIGEKPiA+IHJlc291cmNlIHVuZGVy dXRpbGl6YXRpb24uICBUaGlzIGNvbW1pdCBhdm9pZHMgc3VjaCBwcm9ibGVtYXRpYwo+ID4gc2l0 dWF0aW9ucyBieSBzcXVlZXppbmcgdGhlIHBvb2xzIChyZXR1cm5zIGV2ZXJ5IGZyZWUgcGFnZSBp biB0aGUgcG9vbAo+ID4gdG8gdGhlIHN5c3RlbSkgZm9yIGEgd2hpbGUgKHVzZXJzIGNhbiBzZXQg dGhpcyBkdXJhdGlvbiB2aWEgYSBtb2R1bGUKPiA+IHBhcmFtZXRlcikgaWYgbWVtb3J5IHByZXNz dXJlIGlzIGRldGVjdGVkLgo+ID4gCj4gPiBEaXNjdXNzaW9ucwo+ID4gPT09PT09PT09PT0KPiA+ IAo+ID4gVGhlIGBibGtiYWNrYCdzIG9yaWdpbmFsIHNocmlua2luZyBtZWNoYW5pc20gcmV0dXJu cyBvbmx5IHBhZ2VzIGluIHRoZQo+ID4gcG9vbCB3aGljaCBhcmUgbm90IGN1cnJlbnRseSBiZSB1 c2VkIGJ5IGBibGtiYWNrYCB0byB0aGUgc3lzdGVtLiAgSW4KPiA+IG90aGVyIHdvcmRzLCB0aGUg cGFnZXMgdGhhdCBhcmUgbm90IG1hcHBlZCB3aXRoIGdyYW50ZWQgcGFnZXMuICBCZWNhdXNlCj4g PiB0aGlzIGNvbW1pdCBpcyBjaGFuZ2luZyBvbmx5IHRoZSBzaHJpbmsgbGltaXQgYnV0IHN0aWxs IHVzZXMgdGhlIHNhbWUKPiA+IGZyZWVpbmcgbWVjaGFuaXNtIGl0IGRvZXMgbm90IHRvdWNoIHBh Z2VzIHdoaWNoIGFyZSBjdXJyZW50bHkgbWFwcGluZwo+ID4gZ3JhbnRzLgo+ID4gCj4gPiBPbmNl IG1lbW9yeSBwcmVzc3VyZSBpcyBkZXRlY3RlZCwgdGhpcyBjb21taXQga2VlcHMgdGhlIHNxdWVl emluZyBsaW1pdAo+ID4gZm9yIGEgdXNlci1zcGVjaWZpZWQgdGltZSBkdXJhdGlvbi4gIFRoZSBk dXJhdGlvbiBzaG91bGQgYmUgbmVpdGhlciB0b28KPiA+IGxvbmcgbm9yIHRvbyBzaG9ydC4gIElm IGl0IGlzIHRvbyBsb25nLCB0aGUgc3F1ZWV6aW5nIGluY3VycmluZyBvdmVyaGVhZAo+ID4gY2Fu IHJlZHVjZSB0aGUgSS9PIHBlcmZvcm1hbmNlLiAgSWYgaXQgaXMgdG9vIHNob3J0LCBgYmxrYmFj a2Agd2lsbCBub3QKPiA+IGZyZWUgZW5vdWdoIHBhZ2VzIHRvIHJlZHVjZSB0aGUgbWVtb3J5IHBy ZXNzdXJlLiAgVGhpcyBjb21taXQgc2V0cyB0aGUKPiA+IHZhbHVlIGFzIGAxMCBtaWxsaXNlY29u ZHNgIGJ5IGRlZmF1bHQgYmVjYXVzZSBpdCBpcyBhIHNob3J0IHRpbWUgaW4KPiA+IHRlcm1zIG9m IEkvTyB3aGlsZSBpdCBpcyBhIGxvbmcgdGltZSBpbiB0ZXJtcyBvZiBtZW1vcnkgb3BlcmF0aW9u cy4KPiA+IEFsc28sIGFzIHRoZSBvcmlnaW5hbCBzaHJpbmtpbmcgbWVjaGFuaXNtIHdvcmtzIGZv ciBhdCBsZWFzdCBldmVyeSAxMDAKPiA+IG1pbGxpc2Vjb25kcywgdGhpcyBjb3VsZCBiZSBhIHNv bWV3aGF0IHJlYXNvbmFibGUgY2hvaWNlLiAgSSBhbHNvIHRlc3RlZAo+ID4gb3RoZXIgZHVyYXRp b25zIChyZWZlciB0byB0aGUgYmVsb3cgc2VjdGlvbiBmb3IgbW9yZSBkZXRhaWxzKSBhbmQKPiA+ IGNvbmZpcm1lZCB0aGF0IDEwIG1pbGxpc2Vjb25kcyBpcyB0aGUgb25lIHRoYXQgd29ya3MgYmVz dCB3aXRoIHRoZSB0ZXN0Lgo+ID4gVGhhdCBzYWlkLCB0aGUgcHJvcGVyIGR1cmF0aW9uIGRlcGVu ZHMgb24gYWN0dWFsIGNvbmZpZ3VyYXRpb25zIGFuZAo+ID4gd29ya2xvYWRzLiAgVGhhdCdzIHdo eSB0aGlzIGNvbW1pdCBhbGxvd3MgdXNlcnMgdG8gc2V0IHRoZSBkdXJhdGlvbiBhcyBhCj4gPiBt b2R1bGUgcGFyYW1ldGVyLgo+ID4gCj4gPiBNZW1vcnkgUHJlc3N1cmUgVGVzdAo+ID4gPT09PT09 PT09PT09PT09PT09PT0KPiA+IAo+ID4gVG8gc2hvdyBob3cgdGhpcyBjb21taXQgZml4ZXMgdGhl IG1lbW9yeSBwcmVzc3VyZSBzaXR1YXRpb24gd2VsbCwgSQo+ID4gY29uZmlndXJlZCBhIHRlc3Qg ZW52aXJvbm1lbnQgb24gYSB4ZW4tcnVubmluZyB2aXJ0dWFsaXphdGlvbiBzeXN0ZW0uCj4gPiBP biB0aGUgYGJsa2Zyb250YCBydW5uaW5nIGd1ZXN0IGluc3RhbmNlcywgSSBhdHRhY2ggYSBsYXJn ZSBudW1iZXIgb2YKPiA+IG5ldHdvcmstYmFja2VkIHZvbHVtZSBkZXZpY2VzIGFuZCBpbmR1Y2Ug SS9PIHRvIHRob3NlLiAgTWVhbndoaWxlLCBJCj4gPiBtZWFzdXJlIHRoZSBudW1iZXIgb2YgcGFn ZXMgdGhhdCBzd2FwcGVkIGluIChwc3dwaW4pIGFuZCBvdXQgKHBzd3BvdXQpCj4gPiBvbiB0aGUg YGJsa2JhY2tgIHJ1bm5pbmcgZ3Vlc3QuICBUaGUgdGVzdCByYW4gdHdpY2UsIG9uY2UgZm9yIHRo ZQo+ID4gYGJsa2JhY2tgIGJlZm9yZSB0aGlzIGNvbW1pdCBhbmQgb25jZSBmb3IgdGhhdCBhZnRl ciB0aGlzIGNvbW1pdC4gIEFzCj4gPiBzaG93biBiZWxvdywgdGhpcyBjb21taXQgaGFzIGRyYW1h dGljYWxseSByZWR1Y2VkIHRoZSBtZW1vcnkgcHJlc3N1cmU6Cj4gPiAKPiA+ICAgICAgICAgICAg ICAgICBwc3dwaW4gIHBzd3BvdXQKPiA+ICAgICBiZWZvcmUgICAgICA3Niw2NzIgIDE4NSw3OTkK PiA+ICAgICBhZnRlciAgICAgICAgICAyMTIgICAgMywzMjUKPiA+IAo+ID4gT3B0aW1hbCBBZ2dy ZXNzaXZlIFNocmlua2luZyBEdXJhdGlvbgo+ID4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLQo+ID4gCj4gPiBUbyBmaW5kIGEgYmVzdCBzcXVlZXppbmcgZHVyYXRpb24sIEkg cmVwZWF0ZWQgdGhlIHRlc3Qgd2l0aCB0aHJlZQo+ID4gZGlmZmVyZW50IGR1cmF0aW9ucyAoMW1z LCAxMG1zLCBhbmQgMTAwbXMpLiAgVGhlIHJlc3VsdHMgYXJlIGFzIGJlbG93Ogo+ID4gCj4gPiAg ICAgZHVyYXRpb24gICAgcHN3cGluICBwc3dwb3V0Cj4gPiAgICAgMSAgICAgICAgICAgODUyICAg ICA2LDQyNAo+ID4gICAgIDEwICAgICAgICAgIDIxMiAgICAgMywzMjUKPiA+ICAgICAxMDAgICAg ICAgICAyMDMgICAgIDMsMzQwCj4gPiAKPiA+IEFzIGV4cGVjdGVkLCB0aGUgbWVtb3J5IHByZXNz dXJlIGhhcyBkZWNyZWFzZWQgYXMgdGhlIGR1cmF0aW9uIGlzCj4gPiBpbmNyZWFzZWQsIGJ1dCB0 aGUgcmVkdWN0aW9uIHN0b3BwZWQgZnJvbSB0aGUgYDEwbXNgLiAgQmFzZWQgb24gdGhpcwo+ID4g cmVzdWx0cywgSSBjaG9zZSB0aGUgZGVmYXVsdCBkdXJhdGlvbiBhcyAxMG1zLgo+ID4gCj4gPiBQ ZXJmb3JtYW5jZSBPdmVyaGVhZCBUZXN0Cj4gPiA9PT09PT09PT09PT09PT09PT09PT09PT09Cj4g PiAKPiA+IFRoaXMgY29tbWl0IGNvdWxkIGluY3VyIEkvTyBwZXJmb3JtYW5jZSBkZWdyYWRhdGlv biB1bmRlciBzZXZlcmUgbWVtb3J5Cj4gPiBwcmVzc3VyZSBiZWNhdXNlIHRoZSBzcXVlZXppbmcg d2lsbCByZXF1aXJlIG1vcmUgcGFnZSBhbGxvY2F0aW9ucyBwZXIKPiA+IEkvTy4gIFRvIHNob3cg dGhlIG92ZXJoZWFkLCBJIGFydGlmaWNpYWxseSBtYWRlIGEgd29yc3QtY2FzZSBzcXVlZXppbmcK PiA+IHNpdHVhdGlvbiBhbmQgbWVhc3VyZWQgdGhlIEkvTyBwZXJmb3JtYW5jZSBvZiBhIGBibGtm cm9udGAgcnVubmluZwo+ID4gZ3Vlc3QuCj4gPiAKPiA+IEZvciB0aGUgYXJ0aWZpY2lhbCBzcXVl ZXppbmcsIEkgc2V0IHRoZSBgYmxrYmFjay5tYXhfYnVmZmVyX3BhZ2VzYCB1c2luZwo+ID4gdGhl IGAvc3lzL21vZHVsZS94ZW5fYmxrYmFjay9wYXJhbWV0ZXJzL21heF9idWZmZXJfcGFnZXNgIGZp bGUuICBJbiB0aGlzCj4gPiB0ZXN0LCBJIHNldCB0aGUgdmFsdWUgdG8gYDEwMjRgIGFuZCBgMGAu ICBUaGUgYDEwMjRgIGlzIHRoZSBkZWZhdWx0Cj4gPiB2YWx1ZS4gIFNldHRpbmcgdGhlIHZhbHVl IGFzIGAwYCBpcyBzYW1lIHRvIGEgc2l0dWF0aW9uIGRvaW5nIHRoZQo+ID4gc3F1ZWV6aW5nIGFs d2F5cyAod29yc3QtY2FzZSkuCj4gPiAKPiA+IEZvciB0aGUgSS9PIHBlcmZvcm1hbmNlIG1lYXN1 cmVtZW50LCBJIHJ1biBhIHNpbXBsZSBgZGRgIGNvbW1hbmQgNSB0aW1lcwo+ID4gYXMgYmVsb3cg YW5kIGNvbGxlY3QgdGhlICdNQi9zJyByZXN1bHRzLgo+ID4gCj4gPiAgICAgJCBmb3IgaSBpbiB7 MS4uNX07IGRvIGRkIGlmPS9kZXYvemVybyBvZj1maWxlIFwKPiA+ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgYnM9NGsgY291bnQ9JCgoMjU2KjUxMikpOyBzeW5jOyBkb25lCj4gPiAKPiA+ IElmIHRoZSB1bmRlcmx5aW5nIGJsb2NrIGRldmljZSBpcyBzbG93IGVub3VnaCwgdGhlIHNxdWVl emluZyBvdmVyaGVhZAo+ID4gY291bGQgYmUgaGlkZGVuLiAgRm9yIHRoZSByZWFzb24sIEkgZG8g dGhpcyB0ZXN0IGZvciBib3RoIGEgc2xvdyBibG9jawo+ID4gZGV2aWNlIGFuZCBhIGZhc3QgYmxv Y2sgZGV2aWNlLiAgSSB1c2UgYSBwb3B1bGFyIGNsb3VkIGJsb2NrIHN0b3JhZ2UKPiA+IHNlcnZp Y2UsIGVic1sxXSBhcyBhIHNsb3cgZGV2aWNlIGFuZCB0aGUgcmFtZGlzayBibG9jayBkZXZpY2Vb Ml0gZm9yIHRoZQo+ID4gZmFzdCBkZXZpY2UuCj4gPiAKPiA+IFRoZSByZXN1bHRzIGFyZSBhcyBi ZWxvdy4gICdtYXhfcGdzJyByZXByZXNlbnRzIHRoZSB2YWx1ZSBvZiB0aGUKPiA+IGBibGtiYWNr Lm1heF9idWZmZXJfcGFnZXNgIHBhcmFtZXRlci4KPiA+IAo+ID4gT24gdGhlIHNsb3cgYmxvY2sg ZGV2aWNlCj4gPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KPiA+IAo+ID4gICAgIG1heF9wZ3Mg ICBNaW4gICAgICAgTWF4ICAgICAgIE1lZGlhbiAgICAgQXZnICAgIFN0ZGRldgo+ID4gICAgIDAg ICAgICAgICAzOC43ICAgICAgNDUuOCAgICAgIDM4LjcgICAgICAgNDAuMTIgIDMuMTc1MjE2NQo+ ID4gICAgIDEwMjQgICAgICAzOC43ICAgICAgNDUuOCAgICAgIDM4LjcgICAgICAgNDAuMTIgIDMu MTc1MjE2NQo+ID4gICAgIE5vIGRpZmZlcmVuY2UgcHJvdmVuIGF0IDk1LjAlIGNvbmZpZGVuY2UK PiA+IAo+ID4gT24gdGhlIGZhc3QgYmxvY2sgZGV2aWNlCj4gPiAtLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0KPiA+IAo+ID4gICAgIG1heF9wZ3MgICBNaW4gICAgICAgTWF4ICAgICAgIE1lZGlhbiAg ICAgQXZnICAgIFN0ZGRldgo+ID4gICAgIDAgICAgICAgICA0MTcgICAgICAgNDIzICAgICAgIDQy MCAgICAgICAgNDE5LjQgIDIuNTA5OTgwMQo+ID4gICAgIDEwMjQgICAgICA0MTQgICAgICAgNDI1 ICAgICAgIDQxNiAgICAgICAgNDE3LjggIDQuNDM4NDY4Mgo+ID4gICAgIE5vIGRpZmZlcmVuY2Ug cHJvdmVuIGF0IDk1LjAlIGNvbmZpZGVuY2UKPiA+IAo+ID4gSW4gc2hvcnQsIGV2ZW4gd29yc3Qg Y2FzZSBzcXVlZXppbmcgb24gcmFtZGlzayBiYXNlZCBmYXN0IGJsb2NrIGRldmljZQo+ID4gbWFr ZXMgbm8gdmlzaWJsZSBwZXJmb3JtYW5jZSBkZWdyYWRhdGlvbi4gIFBsZWFzZSBub3RlIHRoYXQg dGhpcyBpcyBqdXN0Cj4gPiBhIHZlcnkgc2ltcGxlIGFuZCBtaW5pbWFsIHRlc3QuICBPbiBzeXN0 ZW1zIHVzaW5nIHN1cGVyLWZhc3QgYmxvY2sKPiA+IGRldmljZXMgYW5kIGEgc3BlY2lhbCBJL08g d29ya2xvYWQsIHRoZSByZXN1bHRzIG1pZ2h0IGJlIGRpZmZlcmVudC4gIElmCj4gPiB5b3UgaGF2 ZSBhbnkgZG91YnQsIHRlc3Qgb24geW91ciBtYWNoaW5lIHdpdGggeW91ciB3b3JrbG9hZCB0byBm aW5kIHRoZQo+ID4gb3B0aW1hbCBzcXVlZXppbmcgZHVyYXRpb24gZm9yIHlvdS4KPiA+IAo+ID4g WzFdIGh0dHBzOi8vYXdzLmFtYXpvbi5jb20vZWJzLwo+ID4gWzJdIGh0dHBzOi8vd3d3Lmtlcm5l bC5vcmcvZG9jL2h0bWwvbGF0ZXN0L2FkbWluLWd1aWRlL2Jsb2NrZGV2L3JhbWRpc2suaHRtbAoK SSBmb3Jnb3QgdG8gdXBkYXRlIHRoaXMgc2VjdGlvbi4gIEl0IGNvbnRhaW5zIHR3byBldmFsdWF0 aW9uIHJlc3VsdHMgd2hpY2ggaGFzCm5vIGJpZyBkaWZmZXJlbmNlIGFuZCBhbHNvIGRlc2NyaWJl cyBvbmUgdGVzdCBpbiB3cm9uZyB3YXkgKGl0IGluZHVjZWQgZGlyZWN0CklPIHRvIHRoZSByYW1k aXNrKS4gIEZvciBleGFtcGxlLCBJIHdvdWxkIGxpa2UgdG8gdXBkYXRlIHRoaXMgc2VjdGlvbiBh cyBiZWxvdzoKCiAgICBQZXJmb3JtYW5jZSBPdmVyaGVhZCBUZXN0CiAgICA9PT09PT09PT09PT09 PT09PT09PT09PT09CiAgICAKICAgIFRoaXMgY29tbWl0IGNvdWxkIGluY3VyIEkvTyBwZXJmb3Jt YW5jZSBkZWdyYWRhdGlvbiB1bmRlciBzZXZlcmUgbWVtb3J5CiAgICBwcmVzc3VyZSBiZWNhdXNl IHRoZSBzcXVlZXppbmcgd2lsbCByZXF1aXJlIG1vcmUgcGFnZSBhbGxvY2F0aW9ucyBwZXIKICAg IEkvTy4gIFRvIHNob3cgdGhlIG92ZXJoZWFkLCBJIGFydGlmaWNpYWxseSBtYWRlIGEgd29yc3Qt Y2FzZSBzcXVlZXppbmcKICAgIHNpdHVhdGlvbiBhbmQgbWVhc3VyZWQgdGhlIEkvTyBwZXJmb3Jt YW5jZSBvZiBhIGBibGtmcm9udGAgcnVubmluZwogICAgZ3Vlc3QuCiAgICAKICAgIEZvciB0aGUg YXJ0aWZpY2lhbCBzcXVlZXppbmcsIEkgc2V0IHRoZSBgYmxrYmFjay5tYXhfYnVmZmVyX3BhZ2Vz YCB1c2luZwogICAgdGhlIGAvc3lzL21vZHVsZS94ZW5fYmxrYmFjay9wYXJhbWV0ZXJzL21heF9i dWZmZXJfcGFnZXNgIGZpbGUuICBJbiB0aGlzCiAgICB0ZXN0LCBJIHNldCB0aGUgdmFsdWUgdG8g YDEwMjRgIGFuZCBgMGAuICBUaGUgYDEwMjRgIGlzIHRoZSBkZWZhdWx0CiAgICB2YWx1ZS4gIFNl dHRpbmcgdGhlIHZhbHVlIGFzIGAwYCBpcyBzYW1lIHRvIGEgc2l0dWF0aW9uIGRvaW5nIHRoZQog ICAgc3F1ZWV6aW5nIGFsd2F5cyAod29yc3QtY2FzZSkuCiAgICAKICAgIElmIHRoZSB1bmRlcmx5 aW5nIGJsb2NrIGRldmljZSBpcyBzbG93IGVub3VnaCwgdGhlIHNxdWVlemluZyBvdmVyaGVhZCBj b3VsZAogICAgYmUgaGlkZGVuLiAgRm9yIHRoZSByZWFzb24sIEkgdXNlIGEgZmFzdCBibG9jayBk ZXZpY2UsIG5hbWVseSB0aGUgcmJkWzFdOgogICAgCiAgICAgICAgIyB4bCBibG9jay1hdHRhY2gg Z3Vlc3QgcGh5Oi9kZXYvcmFtMCB4dmRiIHcKICAgIAogICAgRm9yIHRoZSBJL08gcGVyZm9ybWFu Y2UgbWVhc3VyZW1lbnQsIEkgcnVuIGEgc2ltcGxlIGBkZGAgY29tbWFuZCA1IHRpbWVzCiAgICBk aXJlY3RseSB0byB0aGUgZGV2aWNlIGFzIGJlbG93IGFuZCBjb2xsZWN0IHRoZSAnTUIvcycgcmVz dWx0cy4KICAgIAogICAgICAgICQgZm9yIGkgaW4gezEuLjV9OyBkbyBkZCBpZj0vZGV2L3plcm8g b2Y9L2Rldi94dmRiIFwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYnM9NGsgY291 bnQ9JCgoMjU2KjUxMikpOyBzeW5jOyBkb25lCiAgICAKICAgIFRoZSByZXN1bHRzIGFyZSBhcyBi ZWxvdy4gICdtYXhfcGdzJyByZXByZXNlbnRzIHRoZSB2YWx1ZSBvZiB0aGUKICAgIGBibGtiYWNr Lm1heF9idWZmZXJfcGFnZXNgIHBhcmFtZXRlci4KICAgIAogICAgICAgIG1heF9wZ3MgICBNaW4g ICAgICAgTWF4ICAgICAgIE1lZGlhbiAgICAgQXZnICAgIFN0ZGRldgogICAgICAgIDAgICAgICAg ICA0MTcgICAgICAgNDIzICAgICAgIDQyMCAgICAgICAgNDE5LjQgIDIuNTA5OTgwMQogICAgICAg IDEwMjQgICAgICA0MTQgICAgICAgNDI1ICAgICAgIDQxNiAgICAgICAgNDE3LjggIDQuNDM4NDY4 MgogICAgICAgIE5vIGRpZmZlcmVuY2UgcHJvdmVuIGF0IDk1LjAlIGNvbmZpZGVuY2UKICAgIAog ICAgSW4gc2hvcnQsIGV2ZW4gd29yc3QgY2FzZSBzcXVlZXppbmcgb24gcmFtZGlzayBiYXNlZCBm YXN0IGJsb2NrIGRldmljZQogICAgbWFrZXMgbm8gdmlzaWJsZSBwZXJmb3JtYW5jZSBkZWdyYWRh dGlvbi4gIFBsZWFzZSBub3RlIHRoYXQgdGhpcyBpcyBqdXN0IGEKICAgIHZlcnkgc2ltcGxlIGFu ZCBtaW5pbWFsIHRlc3QuICBPbiBzeXN0ZW1zIHVzaW5nIHN1cGVyLWZhc3QgYmxvY2sgZGV2aWNl cwogICAgYW5kIGEgc3BlY2lhbCBJL08gd29ya2xvYWQsIHRoZSByZXN1bHRzIG1pZ2h0IGJlIGRp ZmZlcmVudC4gIElmIHlvdSBoYXZlCiAgICBhbnkgZG91YnQsIHRlc3Qgb24geW91ciBtYWNoaW5l IHdpdGggeW91ciB3b3JrbG9hZCB0byBmaW5kIHRoZSBvcHRpbWFsCiAgICBzcXVlZXppbmcgZHVy YXRpb24gZm9yIHlvdS4KICAgIAogICAgWzFdIGh0dHBzOi8vd3d3Lmtlcm5lbC5vcmcvZG9jL2h0 bWwvbGF0ZXN0L2FkbWluLWd1aWRlL2Jsb2NrZGV2L3JhbWRpc2suaHRtbAoKPiA+IAo+ID4gU2ln bmVkLW9mZi1ieTogU2VvbmdKYWUgUGFyayA8c2pwYXJrQGFtYXpvbi5kZT4KPiAKPiBSZXZpZXdl ZC1ieTogUm9nZXIgUGF1IE1vbm7DqSA8cm9nZXIucGF1QGNpdHJpeC5jb20+CgpBcHByZWNpYXRl IGZvciB5b3VyIHJldmlld3MuICBZb3UgbWFkZSB0aGlzIHBhdGNoIG11Y2ggYmV0dGVyIQoKClRo YW5rcywKU2VvbmdKYWUgUGFyawoKPiAKPiBUaGFua3MsIFJvZ2VyLgo+IAoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVsIG1haWxpbmcgbGlz dApYZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcKaHR0cHM6Ly9saXN0cy54ZW5wcm9qZWN0 Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3hlbi1kZXZlbA==