From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C584C07E99 for ; Fri, 9 Jul 2021 11:11:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0D1C4613E1 for ; Fri, 9 Jul 2021 11:11:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230508AbhGILO3 (ORCPT ); Fri, 9 Jul 2021 07:14:29 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3376 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229861AbhGILO3 (ORCPT ); Fri, 9 Jul 2021 07:14:29 -0400 Received: from fraeml734-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4GLqyn74LDz6J6X3; Fri, 9 Jul 2021 19:03:29 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml734-chm.china.huawei.com (10.206.15.215) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 9 Jul 2021 13:11:44 +0200 Received: from [10.47.93.40] (10.47.93.40) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 9 Jul 2021 12:11:43 +0100 Subject: Re: [bug report] iommu_dma_unmap_sg() is very slow then running IO from remote numa node To: Robin Murphy , Ming Lei , , Will Deacon , , CC: References: <23e7956b-f3b5-b585-3c18-724165994051@arm.com> From: John Garry Message-ID: Date: Fri, 9 Jul 2021 12:04:24 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <23e7956b-f3b5-b585-3c18-724165994051@arm.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Originating-IP: [10.47.93.40] X-ClientProxiedBy: lhreml737-chm.china.huawei.com (10.201.108.187) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 09/07/2021 11:26, Robin Murphy wrote: > n 2021-07-09 09:38, Ming Lei wrote: >> Hello, >> >> I observed that NVMe performance is very bad when running fio on one >> CPU(aarch64) in remote numa node compared with the nvme pci numa node. >> >> Please see the test result[1] 327K vs. 34.9K. >> >> Latency trace shows that one big difference is in iommu_dma_unmap_sg(), >> 1111 nsecs vs 25437 nsecs. > > Are you able to dig down further into that? iommu_dma_unmap_sg() itself > doesn't do anything particularly special, so whatever makes a difference > is probably happening at a lower level, and I suspect there's probably > an SMMU involved. If for instance it turns out to go all the way down to > __arm_smmu_cmdq_poll_until_consumed() because polling MMIO from the > wrong node is slow, there's unlikely to be much you can do about that > other than the global "go faster" knobs (iommu.strict and > iommu.passthrough) with their associated compromises. There was also the disable_msipolling option: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c#n42 But I am not sure if that platform even supports MSI polling (or has smmu v3). You could also try iommu.forcedac=1 cmdline option. But I doubt it will help since the issue was mentioned to be NUMA related. > > Robin. > >> [1] fio test & results >> >> 1) fio test result: >> >> - run fio on local CPU >> taskset -c 0 ~/git/tools/test/nvme/io_uring 10 1 /dev/nvme1n1 4k >> + fio --bs=4k --ioengine=io_uring --fixedbufs --registerfiles --hipri >> --iodepth=64 --iodepth_batch_submit=16 --iodepth_batch_complete_min=16 >> --filename=/dev/nvme1n1 --direct=1 --runtime=10 --numjobs=1 >> --rw=randread --name=test --group_reporting >> >> IOPS: 327K >> avg latency of iommu_dma_unmap_sg(): 1111 nsecs >> >> >> - run fio on remote CPU >> taskset -c 80 ~/git/tools/test/nvme/io_uring 10 1 /dev/nvme1n1 4k >> + fio --bs=4k --ioengine=io_uring --fixedbufs --registerfiles --hipri >> --iodepth=64 --iodepth_batch_submit=16 --iodepth_batch_complete_min=16 >> --filename=/dev/nvme1n1 --direct=1 --runtime=10 --numjobs=1 >> --rw=randread --name=test --group_reporting >> >> IOPS: 34.9K >> avg latency of iommu_dma_unmap_sg(): 25437 nsecs >> >> 2) system info >> [root@ampere-mtjade-04 ~]# lscpu | grep NUMA >> NUMA node(s):                    2 >> NUMA node0 CPU(s):               0-79 >> NUMA node1 CPU(s):               80-159 >> >> lspci | grep NVMe >> 0003:01:00.0 Non-Volatile memory controller: Samsung Electronics Co >> Ltd NVMe SSD Controller SM981/PM981/PM983 >> >> [root@ampere-mtjade-04 ~]# cat /sys/block/nvme1n1/device/device/numa_node Since it's ampere, I guess it's smmu v3. BTW, if you remember, I did raise a performance issue of smmuv3 with NVMe before: https://lore.kernel.org/linux-iommu/b2a6e26d-6d0d-7f0d-f222-589812f701d2@huawei.com/ I did have this series to improve performance for systems with lots of CPUs, like above, but not accepted: https://lore.kernel.org/linux-iommu/1598018062-175608-1-git-send-email-john.garry@huawei.com/ Thanks, John From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EAC2C07E99 for ; Fri, 9 Jul 2021 11:12:20 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 74402613E1 for ; Fri, 9 Jul 2021 11:12:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74402613E1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:CC:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tE5eGSHgHuvbW57ZiAXnCKxO2gQpKoJDP+T/xtyVTuU=; b=uruU2FLdg+6jIv028WmBdCLlxV ykJDlDIC3dDuKuL367yswxtyjkevY36/H1vWG1FXcfpsoqwHx/ugi24vKEFCDI+nOnSnRAtz9kjkk 7NPTaxUtHFDtzaY07BaQHe1FJDv84yEIUMhLbGkFfORXwMVtzjzZHbYnTMYxUOjaxBOWnRmOY5+Mx S3I9LQRvU0gukrpwiT6iTHcIirTDXy3jii0I5YK0fNcBhIXCBMV9+g++7JiseHv7AesNhqVQIEgtE AD65G9P/9YIQ9hKm+GQoPxZVcdHUZnYbioVUIWK4+mT8xC6UVGEI5XU2c8fbK07r0kMf2EzeP8aUq o9oCudvg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m1oQT-001dK0-BV; Fri, 09 Jul 2021 11:12:09 +0000 Received: from frasgout.his.huawei.com ([185.176.79.56]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m1oQG-001dIA-Bq; Fri, 09 Jul 2021 11:11:58 +0000 Received: from fraeml734-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4GLqyn74LDz6J6X3; Fri, 9 Jul 2021 19:03:29 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml734-chm.china.huawei.com (10.206.15.215) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 9 Jul 2021 13:11:44 +0200 Received: from [10.47.93.40] (10.47.93.40) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 9 Jul 2021 12:11:43 +0100 Subject: Re: [bug report] iommu_dma_unmap_sg() is very slow then running IO from remote numa node To: Robin Murphy , Ming Lei , , Will Deacon , , CC: References: <23e7956b-f3b5-b585-3c18-724165994051@arm.com> From: John Garry Message-ID: Date: Fri, 9 Jul 2021 12:04:24 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <23e7956b-f3b5-b585-3c18-724165994051@arm.com> Content-Language: en-US X-Originating-IP: [10.47.93.40] X-ClientProxiedBy: lhreml737-chm.china.huawei.com (10.201.108.187) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210709_041156_743992_172AF3B4 X-CRM114-Status: GOOD ( 16.80 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org T24gMDkvMDcvMjAyMSAxMToyNiwgUm9iaW4gTXVycGh5IHdyb3RlOgo+IG4gMjAyMS0wNy0wOSAw OTozOCwgTWluZyBMZWkgd3JvdGU6Cj4+IEhlbGxvLAo+Pgo+PiBJIG9ic2VydmVkIHRoYXQgTlZN ZSBwZXJmb3JtYW5jZSBpcyB2ZXJ5IGJhZCB3aGVuIHJ1bm5pbmcgZmlvIG9uIG9uZQo+PiBDUFUo YWFyY2g2NCkgaW4gcmVtb3RlIG51bWEgbm9kZSBjb21wYXJlZCB3aXRoIHRoZSBudm1lIHBjaSBu dW1hIG5vZGUuCj4+Cj4+IFBsZWFzZSBzZWUgdGhlIHRlc3QgcmVzdWx0WzFdIDMyN0sgdnMuIDM0 LjlLLgo+Pgo+PiBMYXRlbmN5IHRyYWNlIHNob3dzIHRoYXQgb25lIGJpZyBkaWZmZXJlbmNlIGlz IGluIGlvbW11X2RtYV91bm1hcF9zZygpLAo+PiAxMTExIG5zZWNzIHZzIDI1NDM3IG5zZWNzLgo+ IAo+IEFyZSB5b3UgYWJsZSB0byBkaWcgZG93biBmdXJ0aGVyIGludG8gdGhhdD8gaW9tbXVfZG1h X3VubWFwX3NnKCkgaXRzZWxmIAo+IGRvZXNuJ3QgZG8gYW55dGhpbmcgcGFydGljdWxhcmx5IHNw ZWNpYWwsIHNvIHdoYXRldmVyIG1ha2VzIGEgZGlmZmVyZW5jZSAKPiBpcyBwcm9iYWJseSBoYXBw ZW5pbmcgYXQgYSBsb3dlciBsZXZlbCwgYW5kIEkgc3VzcGVjdCB0aGVyZSdzIHByb2JhYmx5IAo+ IGFuIFNNTVUgaW52b2x2ZWQuIElmIGZvciBpbnN0YW5jZSBpdCB0dXJucyBvdXQgdG8gZ28gYWxs IHRoZSB3YXkgZG93biB0byAKPiBfX2FybV9zbW11X2NtZHFfcG9sbF91bnRpbF9jb25zdW1lZCgp IGJlY2F1c2UgcG9sbGluZyBNTUlPIGZyb20gdGhlIAo+IHdyb25nIG5vZGUgaXMgc2xvdywgdGhl cmUncyB1bmxpa2VseSB0byBiZSBtdWNoIHlvdSBjYW4gZG8gYWJvdXQgdGhhdCAKPiBvdGhlciB0 aGFuIHRoZSBnbG9iYWwgImdvIGZhc3RlciIga25vYnMgKGlvbW11LnN0cmljdCBhbmQgCj4gaW9t bXUucGFzc3Rocm91Z2gpIHdpdGggdGhlaXIgYXNzb2NpYXRlZCBjb21wcm9taXNlcy4KClRoZXJl IHdhcyBhbHNvIHRoZSBkaXNhYmxlX21zaXBvbGxpbmcgb3B0aW9uOgoKaHR0cHM6Ly9naXQua2Vy bmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvdG9ydmFsZHMvbGludXguZ2l0L3RyZWUv ZHJpdmVycy9pb21tdS9hcm0vYXJtLXNtbXUtdjMvYXJtLXNtbXUtdjMuYyNuNDIKCkJ1dCBJIGFt IG5vdCBzdXJlIGlmIHRoYXQgcGxhdGZvcm0gZXZlbiBzdXBwb3J0cyBNU0kgcG9sbGluZyAob3Ig aGFzIApzbW11IHYzKS4KCllvdSBjb3VsZCBhbHNvIHRyeSBpb21tdS5mb3JjZWRhYz0xIGNtZGxp bmUgb3B0aW9uLiBCdXQgSSBkb3VidCBpdCB3aWxsIApoZWxwIHNpbmNlIHRoZSBpc3N1ZSB3YXMg bWVudGlvbmVkIHRvIGJlIE5VTUEgcmVsYXRlZC4KCj4gCj4gUm9iaW4uCj4gCj4+IFsxXSBmaW8g dGVzdCAmIHJlc3VsdHMKPj4KPj4gMSkgZmlvIHRlc3QgcmVzdWx0Ogo+Pgo+PiAtIHJ1biBmaW8g b24gbG9jYWwgQ1BVCj4+IHRhc2tzZXQgLWMgMCB+L2dpdC90b29scy90ZXN0L252bWUvaW9fdXJp bmcgMTAgMSAvZGV2L252bWUxbjEgNGsKPj4gKyBmaW8gLS1icz00ayAtLWlvZW5naW5lPWlvX3Vy aW5nIC0tZml4ZWRidWZzIC0tcmVnaXN0ZXJmaWxlcyAtLWhpcHJpIAo+PiAtLWlvZGVwdGg9NjQg LS1pb2RlcHRoX2JhdGNoX3N1Ym1pdD0xNiAtLWlvZGVwdGhfYmF0Y2hfY29tcGxldGVfbWluPTE2 IAo+PiAtLWZpbGVuYW1lPS9kZXYvbnZtZTFuMSAtLWRpcmVjdD0xIC0tcnVudGltZT0xMCAtLW51 bWpvYnM9MSAKPj4gLS1ydz1yYW5kcmVhZCAtLW5hbWU9dGVzdCAtLWdyb3VwX3JlcG9ydGluZwo+ Pgo+PiBJT1BTOiAzMjdLCj4+IGF2ZyBsYXRlbmN5IG9mIGlvbW11X2RtYV91bm1hcF9zZygpOiAx MTExIG5zZWNzCj4+Cj4+Cj4+IC0gcnVuIGZpbyBvbiByZW1vdGUgQ1BVCj4+IHRhc2tzZXQgLWMg ODAgfi9naXQvdG9vbHMvdGVzdC9udm1lL2lvX3VyaW5nIDEwIDEgL2Rldi9udm1lMW4xIDRrCj4+ ICsgZmlvIC0tYnM9NGsgLS1pb2VuZ2luZT1pb191cmluZyAtLWZpeGVkYnVmcyAtLXJlZ2lzdGVy ZmlsZXMgLS1oaXByaSAKPj4gLS1pb2RlcHRoPTY0IC0taW9kZXB0aF9iYXRjaF9zdWJtaXQ9MTYg LS1pb2RlcHRoX2JhdGNoX2NvbXBsZXRlX21pbj0xNiAKPj4gLS1maWxlbmFtZT0vZGV2L252bWUx bjEgLS1kaXJlY3Q9MSAtLXJ1bnRpbWU9MTAgLS1udW1qb2JzPTEgCj4+IC0tcnc9cmFuZHJlYWQg LS1uYW1lPXRlc3QgLS1ncm91cF9yZXBvcnRpbmcKPj4KPj4gSU9QUzogMzQuOUsKPj4gYXZnIGxh dGVuY3kgb2YgaW9tbXVfZG1hX3VubWFwX3NnKCk6IDI1NDM3IG5zZWNzCj4+Cj4+IDIpIHN5c3Rl bSBpbmZvCj4+IFtyb290QGFtcGVyZS1tdGphZGUtMDQgfl0jIGxzY3B1IHwgZ3JlcCBOVU1BCj4+ IE5VTUEgbm9kZShzKTrCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAyCj4+ IE5VTUEgbm9kZTAgQ1BVKHMpOsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMC03OQo+PiBO VU1BIG5vZGUxIENQVShzKTrCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDgwLTE1OQo+Pgo+ PiBsc3BjaSB8IGdyZXAgTlZNZQo+PiAwMDAzOjAxOjAwLjAgTm9uLVZvbGF0aWxlIG1lbW9yeSBj b250cm9sbGVyOiBTYW1zdW5nIEVsZWN0cm9uaWNzIENvIAo+PiBMdGQgTlZNZSBTU0QgQ29udHJv bGxlciBTTTk4MS9QTTk4MS9QTTk4Mwo+Pgo+PiBbcm9vdEBhbXBlcmUtbXRqYWRlLTA0IH5dIyBj YXQgL3N5cy9ibG9jay9udm1lMW4xL2RldmljZS9kZXZpY2UvbnVtYV9ub2RlIAoKU2luY2UgaXQn cyBhbXBlcmUsIEkgZ3Vlc3MgaXQncyBzbW11IHYzLgoKQlRXLCBpZiB5b3UgcmVtZW1iZXIsIEkg ZGlkIHJhaXNlIGEgcGVyZm9ybWFuY2UgaXNzdWUgb2Ygc21tdXYzIHdpdGggCk5WTWUgYmVmb3Jl OgpodHRwczovL2xvcmUua2VybmVsLm9yZy9saW51eC1pb21tdS9iMmE2ZTI2ZC02ZDBkLTdmMGQt ZjIyMi01ODk4MTJmNzAxZDJAaHVhd2VpLmNvbS8KCkkgZGlkIGhhdmUgdGhpcyBzZXJpZXMgdG8g aW1wcm92ZSBwZXJmb3JtYW5jZSBmb3Igc3lzdGVtcyB3aXRoIGxvdHMgb2YgCkNQVXMsIGxpa2Ug YWJvdmUsIGJ1dCBub3QgYWNjZXB0ZWQ6Cmh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2xpbnV4LWlv bW11LzE1OTgwMTgwNjItMTc1NjA4LTEtZ2l0LXNlbmQtZW1haWwtam9obi5nYXJyeUBodWF3ZWku Y29tLwoKVGhhbmtzLApKb2huCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18KTGludXgtbnZtZSBtYWlsaW5nIGxpc3QKTGludXgtbnZtZUBsaXN0cy5pbmZy YWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGlu dXgtbnZtZQo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AA23C07E99 for ; Fri, 9 Jul 2021 11:11:53 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 254DB613E1 for ; Fri, 9 Jul 2021 11:11:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 254DB613E1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id DB595405D3; Fri, 9 Jul 2021 11:11:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cfTTvRpTih8c; Fri, 9 Jul 2021 11:11:52 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id C6FD7405A0; Fri, 9 Jul 2021 11:11:51 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 98543C001B; Fri, 9 Jul 2021 11:11:51 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3183DC001A for ; Fri, 9 Jul 2021 11:11:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id AB730405C5 for ; Fri, 9 Jul 2021 11:11:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jmb0p7fntssE for ; Fri, 9 Jul 2021 11:11:47 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 20E93405A7 for ; Fri, 9 Jul 2021 11:11:47 +0000 (UTC) Received: from fraeml734-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4GLqyn74LDz6J6X3; Fri, 9 Jul 2021 19:03:29 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml734-chm.china.huawei.com (10.206.15.215) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 9 Jul 2021 13:11:44 +0200 Received: from [10.47.93.40] (10.47.93.40) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 9 Jul 2021 12:11:43 +0100 Subject: Re: [bug report] iommu_dma_unmap_sg() is very slow then running IO from remote numa node To: Robin Murphy , Ming Lei , , Will Deacon , , References: <23e7956b-f3b5-b585-3c18-724165994051@arm.com> From: John Garry Message-ID: Date: Fri, 9 Jul 2021 12:04:24 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <23e7956b-f3b5-b585-3c18-724165994051@arm.com> Content-Language: en-US X-Originating-IP: [10.47.93.40] X-ClientProxiedBy: lhreml737-chm.china.huawei.com (10.201.108.187) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Cc: linux-kernel@vger.kernel.org X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" T24gMDkvMDcvMjAyMSAxMToyNiwgUm9iaW4gTXVycGh5IHdyb3RlOgo+IG4gMjAyMS0wNy0wOSAw OTozOCwgTWluZyBMZWkgd3JvdGU6Cj4+IEhlbGxvLAo+Pgo+PiBJIG9ic2VydmVkIHRoYXQgTlZN ZSBwZXJmb3JtYW5jZSBpcyB2ZXJ5IGJhZCB3aGVuIHJ1bm5pbmcgZmlvIG9uIG9uZQo+PiBDUFUo YWFyY2g2NCkgaW4gcmVtb3RlIG51bWEgbm9kZSBjb21wYXJlZCB3aXRoIHRoZSBudm1lIHBjaSBu dW1hIG5vZGUuCj4+Cj4+IFBsZWFzZSBzZWUgdGhlIHRlc3QgcmVzdWx0WzFdIDMyN0sgdnMuIDM0 LjlLLgo+Pgo+PiBMYXRlbmN5IHRyYWNlIHNob3dzIHRoYXQgb25lIGJpZyBkaWZmZXJlbmNlIGlz IGluIGlvbW11X2RtYV91bm1hcF9zZygpLAo+PiAxMTExIG5zZWNzIHZzIDI1NDM3IG5zZWNzLgo+ IAo+IEFyZSB5b3UgYWJsZSB0byBkaWcgZG93biBmdXJ0aGVyIGludG8gdGhhdD8gaW9tbXVfZG1h X3VubWFwX3NnKCkgaXRzZWxmIAo+IGRvZXNuJ3QgZG8gYW55dGhpbmcgcGFydGljdWxhcmx5IHNw ZWNpYWwsIHNvIHdoYXRldmVyIG1ha2VzIGEgZGlmZmVyZW5jZSAKPiBpcyBwcm9iYWJseSBoYXBw ZW5pbmcgYXQgYSBsb3dlciBsZXZlbCwgYW5kIEkgc3VzcGVjdCB0aGVyZSdzIHByb2JhYmx5IAo+ IGFuIFNNTVUgaW52b2x2ZWQuIElmIGZvciBpbnN0YW5jZSBpdCB0dXJucyBvdXQgdG8gZ28gYWxs IHRoZSB3YXkgZG93biB0byAKPiBfX2FybV9zbW11X2NtZHFfcG9sbF91bnRpbF9jb25zdW1lZCgp IGJlY2F1c2UgcG9sbGluZyBNTUlPIGZyb20gdGhlIAo+IHdyb25nIG5vZGUgaXMgc2xvdywgdGhl cmUncyB1bmxpa2VseSB0byBiZSBtdWNoIHlvdSBjYW4gZG8gYWJvdXQgdGhhdCAKPiBvdGhlciB0 aGFuIHRoZSBnbG9iYWwgImdvIGZhc3RlciIga25vYnMgKGlvbW11LnN0cmljdCBhbmQgCj4gaW9t bXUucGFzc3Rocm91Z2gpIHdpdGggdGhlaXIgYXNzb2NpYXRlZCBjb21wcm9taXNlcy4KClRoZXJl IHdhcyBhbHNvIHRoZSBkaXNhYmxlX21zaXBvbGxpbmcgb3B0aW9uOgoKaHR0cHM6Ly9naXQua2Vy bmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvdG9ydmFsZHMvbGludXguZ2l0L3RyZWUv ZHJpdmVycy9pb21tdS9hcm0vYXJtLXNtbXUtdjMvYXJtLXNtbXUtdjMuYyNuNDIKCkJ1dCBJIGFt IG5vdCBzdXJlIGlmIHRoYXQgcGxhdGZvcm0gZXZlbiBzdXBwb3J0cyBNU0kgcG9sbGluZyAob3Ig aGFzIApzbW11IHYzKS4KCllvdSBjb3VsZCBhbHNvIHRyeSBpb21tdS5mb3JjZWRhYz0xIGNtZGxp bmUgb3B0aW9uLiBCdXQgSSBkb3VidCBpdCB3aWxsIApoZWxwIHNpbmNlIHRoZSBpc3N1ZSB3YXMg bWVudGlvbmVkIHRvIGJlIE5VTUEgcmVsYXRlZC4KCj4gCj4gUm9iaW4uCj4gCj4+IFsxXSBmaW8g dGVzdCAmIHJlc3VsdHMKPj4KPj4gMSkgZmlvIHRlc3QgcmVzdWx0Ogo+Pgo+PiAtIHJ1biBmaW8g b24gbG9jYWwgQ1BVCj4+IHRhc2tzZXQgLWMgMCB+L2dpdC90b29scy90ZXN0L252bWUvaW9fdXJp bmcgMTAgMSAvZGV2L252bWUxbjEgNGsKPj4gKyBmaW8gLS1icz00ayAtLWlvZW5naW5lPWlvX3Vy aW5nIC0tZml4ZWRidWZzIC0tcmVnaXN0ZXJmaWxlcyAtLWhpcHJpIAo+PiAtLWlvZGVwdGg9NjQg LS1pb2RlcHRoX2JhdGNoX3N1Ym1pdD0xNiAtLWlvZGVwdGhfYmF0Y2hfY29tcGxldGVfbWluPTE2 IAo+PiAtLWZpbGVuYW1lPS9kZXYvbnZtZTFuMSAtLWRpcmVjdD0xIC0tcnVudGltZT0xMCAtLW51 bWpvYnM9MSAKPj4gLS1ydz1yYW5kcmVhZCAtLW5hbWU9dGVzdCAtLWdyb3VwX3JlcG9ydGluZwo+ Pgo+PiBJT1BTOiAzMjdLCj4+IGF2ZyBsYXRlbmN5IG9mIGlvbW11X2RtYV91bm1hcF9zZygpOiAx MTExIG5zZWNzCj4+Cj4+Cj4+IC0gcnVuIGZpbyBvbiByZW1vdGUgQ1BVCj4+IHRhc2tzZXQgLWMg ODAgfi9naXQvdG9vbHMvdGVzdC9udm1lL2lvX3VyaW5nIDEwIDEgL2Rldi9udm1lMW4xIDRrCj4+ ICsgZmlvIC0tYnM9NGsgLS1pb2VuZ2luZT1pb191cmluZyAtLWZpeGVkYnVmcyAtLXJlZ2lzdGVy ZmlsZXMgLS1oaXByaSAKPj4gLS1pb2RlcHRoPTY0IC0taW9kZXB0aF9iYXRjaF9zdWJtaXQ9MTYg LS1pb2RlcHRoX2JhdGNoX2NvbXBsZXRlX21pbj0xNiAKPj4gLS1maWxlbmFtZT0vZGV2L252bWUx bjEgLS1kaXJlY3Q9MSAtLXJ1bnRpbWU9MTAgLS1udW1qb2JzPTEgCj4+IC0tcnc9cmFuZHJlYWQg LS1uYW1lPXRlc3QgLS1ncm91cF9yZXBvcnRpbmcKPj4KPj4gSU9QUzogMzQuOUsKPj4gYXZnIGxh dGVuY3kgb2YgaW9tbXVfZG1hX3VubWFwX3NnKCk6IDI1NDM3IG5zZWNzCj4+Cj4+IDIpIHN5c3Rl bSBpbmZvCj4+IFtyb290QGFtcGVyZS1tdGphZGUtMDQgfl0jIGxzY3B1IHwgZ3JlcCBOVU1BCj4+ IE5VTUEgbm9kZShzKTrCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAyCj4+ IE5VTUEgbm9kZTAgQ1BVKHMpOsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMC03OQo+PiBO VU1BIG5vZGUxIENQVShzKTrCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDgwLTE1OQo+Pgo+ PiBsc3BjaSB8IGdyZXAgTlZNZQo+PiAwMDAzOjAxOjAwLjAgTm9uLVZvbGF0aWxlIG1lbW9yeSBj b250cm9sbGVyOiBTYW1zdW5nIEVsZWN0cm9uaWNzIENvIAo+PiBMdGQgTlZNZSBTU0QgQ29udHJv bGxlciBTTTk4MS9QTTk4MS9QTTk4Mwo+Pgo+PiBbcm9vdEBhbXBlcmUtbXRqYWRlLTA0IH5dIyBj YXQgL3N5cy9ibG9jay9udm1lMW4xL2RldmljZS9kZXZpY2UvbnVtYV9ub2RlIAoKU2luY2UgaXQn cyBhbXBlcmUsIEkgZ3Vlc3MgaXQncyBzbW11IHYzLgoKQlRXLCBpZiB5b3UgcmVtZW1iZXIsIEkg ZGlkIHJhaXNlIGEgcGVyZm9ybWFuY2UgaXNzdWUgb2Ygc21tdXYzIHdpdGggCk5WTWUgYmVmb3Jl OgpodHRwczovL2xvcmUua2VybmVsLm9yZy9saW51eC1pb21tdS9iMmE2ZTI2ZC02ZDBkLTdmMGQt ZjIyMi01ODk4MTJmNzAxZDJAaHVhd2VpLmNvbS8KCkkgZGlkIGhhdmUgdGhpcyBzZXJpZXMgdG8g aW1wcm92ZSBwZXJmb3JtYW5jZSBmb3Igc3lzdGVtcyB3aXRoIGxvdHMgb2YgCkNQVXMsIGxpa2Ug YWJvdmUsIGJ1dCBub3QgYWNjZXB0ZWQ6Cmh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2xpbnV4LWlv bW11LzE1OTgwMTgwNjItMTc1NjA4LTEtZ2l0LXNlbmQtZW1haWwtam9obi5nYXJyeUBodWF3ZWku Y29tLwoKVGhhbmtzLApKb2huCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fXwppb21tdSBtYWlsaW5nIGxpc3QKaW9tbXVAbGlzdHMubGludXgtZm91bmRhdGlv bi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8v aW9tbXU= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D578C07E99 for ; Fri, 9 Jul 2021 11:13:38 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CB1F1613E1 for ; Fri, 9 Jul 2021 11:13:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CB1F1613E1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:CC:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=wxuXk0IGrKD2fHS62RE1I+JDLfYN5At4/ko+Ye8+aRo=; b=3bnRQjWHC73ORgFBUC5emKoLL3 EHSZ8jytbevhzbRNUBUg/SzY5DjAgruji8RKj+0iKNBiaNgmuIeCOhcUxmGddenrsuqdR1FFJqEtN WqV5ZHFY/SeP+xqmYcuvagI8l/RnhNYT5ulIU5Txy9bChI0Rj2mq4bWCNayHv0SK++E3mh1VR+UXi qdXM16CCfXTXewjfoVBykTp2bm8QJHvZMbyn1nWCRaByrEufh9mc+sgYhfxmjGbz6JPUlvi+0tUD6 8tAV5Kci1IPhJeoH/3efNez0rHWd5Z86/3SKqObi34n7Oou47OsdQlnCxtPWnuFslKLe4lgtX7Rqe B7o7xydQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m1oQK-001dJS-Eh; Fri, 09 Jul 2021 11:12:00 +0000 Received: from frasgout.his.huawei.com ([185.176.79.56]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m1oQG-001dIA-Bq; Fri, 09 Jul 2021 11:11:58 +0000 Received: from fraeml734-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4GLqyn74LDz6J6X3; Fri, 9 Jul 2021 19:03:29 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml734-chm.china.huawei.com (10.206.15.215) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 9 Jul 2021 13:11:44 +0200 Received: from [10.47.93.40] (10.47.93.40) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 9 Jul 2021 12:11:43 +0100 Subject: Re: [bug report] iommu_dma_unmap_sg() is very slow then running IO from remote numa node To: Robin Murphy , Ming Lei , , Will Deacon , , CC: References: <23e7956b-f3b5-b585-3c18-724165994051@arm.com> From: John Garry Message-ID: Date: Fri, 9 Jul 2021 12:04:24 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <23e7956b-f3b5-b585-3c18-724165994051@arm.com> Content-Language: en-US X-Originating-IP: [10.47.93.40] X-ClientProxiedBy: lhreml737-chm.china.huawei.com (10.201.108.187) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210709_041156_743992_172AF3B4 X-CRM114-Status: GOOD ( 16.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMDkvMDcvMjAyMSAxMToyNiwgUm9iaW4gTXVycGh5IHdyb3RlOgo+IG4gMjAyMS0wNy0wOSAw OTozOCwgTWluZyBMZWkgd3JvdGU6Cj4+IEhlbGxvLAo+Pgo+PiBJIG9ic2VydmVkIHRoYXQgTlZN ZSBwZXJmb3JtYW5jZSBpcyB2ZXJ5IGJhZCB3aGVuIHJ1bm5pbmcgZmlvIG9uIG9uZQo+PiBDUFUo YWFyY2g2NCkgaW4gcmVtb3RlIG51bWEgbm9kZSBjb21wYXJlZCB3aXRoIHRoZSBudm1lIHBjaSBu dW1hIG5vZGUuCj4+Cj4+IFBsZWFzZSBzZWUgdGhlIHRlc3QgcmVzdWx0WzFdIDMyN0sgdnMuIDM0 LjlLLgo+Pgo+PiBMYXRlbmN5IHRyYWNlIHNob3dzIHRoYXQgb25lIGJpZyBkaWZmZXJlbmNlIGlz IGluIGlvbW11X2RtYV91bm1hcF9zZygpLAo+PiAxMTExIG5zZWNzIHZzIDI1NDM3IG5zZWNzLgo+ IAo+IEFyZSB5b3UgYWJsZSB0byBkaWcgZG93biBmdXJ0aGVyIGludG8gdGhhdD8gaW9tbXVfZG1h X3VubWFwX3NnKCkgaXRzZWxmIAo+IGRvZXNuJ3QgZG8gYW55dGhpbmcgcGFydGljdWxhcmx5IHNw ZWNpYWwsIHNvIHdoYXRldmVyIG1ha2VzIGEgZGlmZmVyZW5jZSAKPiBpcyBwcm9iYWJseSBoYXBw ZW5pbmcgYXQgYSBsb3dlciBsZXZlbCwgYW5kIEkgc3VzcGVjdCB0aGVyZSdzIHByb2JhYmx5IAo+ IGFuIFNNTVUgaW52b2x2ZWQuIElmIGZvciBpbnN0YW5jZSBpdCB0dXJucyBvdXQgdG8gZ28gYWxs IHRoZSB3YXkgZG93biB0byAKPiBfX2FybV9zbW11X2NtZHFfcG9sbF91bnRpbF9jb25zdW1lZCgp IGJlY2F1c2UgcG9sbGluZyBNTUlPIGZyb20gdGhlIAo+IHdyb25nIG5vZGUgaXMgc2xvdywgdGhl cmUncyB1bmxpa2VseSB0byBiZSBtdWNoIHlvdSBjYW4gZG8gYWJvdXQgdGhhdCAKPiBvdGhlciB0 aGFuIHRoZSBnbG9iYWwgImdvIGZhc3RlciIga25vYnMgKGlvbW11LnN0cmljdCBhbmQgCj4gaW9t bXUucGFzc3Rocm91Z2gpIHdpdGggdGhlaXIgYXNzb2NpYXRlZCBjb21wcm9taXNlcy4KClRoZXJl IHdhcyBhbHNvIHRoZSBkaXNhYmxlX21zaXBvbGxpbmcgb3B0aW9uOgoKaHR0cHM6Ly9naXQua2Vy bmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvdG9ydmFsZHMvbGludXguZ2l0L3RyZWUv ZHJpdmVycy9pb21tdS9hcm0vYXJtLXNtbXUtdjMvYXJtLXNtbXUtdjMuYyNuNDIKCkJ1dCBJIGFt IG5vdCBzdXJlIGlmIHRoYXQgcGxhdGZvcm0gZXZlbiBzdXBwb3J0cyBNU0kgcG9sbGluZyAob3Ig aGFzIApzbW11IHYzKS4KCllvdSBjb3VsZCBhbHNvIHRyeSBpb21tdS5mb3JjZWRhYz0xIGNtZGxp bmUgb3B0aW9uLiBCdXQgSSBkb3VidCBpdCB3aWxsIApoZWxwIHNpbmNlIHRoZSBpc3N1ZSB3YXMg bWVudGlvbmVkIHRvIGJlIE5VTUEgcmVsYXRlZC4KCj4gCj4gUm9iaW4uCj4gCj4+IFsxXSBmaW8g dGVzdCAmIHJlc3VsdHMKPj4KPj4gMSkgZmlvIHRlc3QgcmVzdWx0Ogo+Pgo+PiAtIHJ1biBmaW8g b24gbG9jYWwgQ1BVCj4+IHRhc2tzZXQgLWMgMCB+L2dpdC90b29scy90ZXN0L252bWUvaW9fdXJp bmcgMTAgMSAvZGV2L252bWUxbjEgNGsKPj4gKyBmaW8gLS1icz00ayAtLWlvZW5naW5lPWlvX3Vy aW5nIC0tZml4ZWRidWZzIC0tcmVnaXN0ZXJmaWxlcyAtLWhpcHJpIAo+PiAtLWlvZGVwdGg9NjQg LS1pb2RlcHRoX2JhdGNoX3N1Ym1pdD0xNiAtLWlvZGVwdGhfYmF0Y2hfY29tcGxldGVfbWluPTE2 IAo+PiAtLWZpbGVuYW1lPS9kZXYvbnZtZTFuMSAtLWRpcmVjdD0xIC0tcnVudGltZT0xMCAtLW51 bWpvYnM9MSAKPj4gLS1ydz1yYW5kcmVhZCAtLW5hbWU9dGVzdCAtLWdyb3VwX3JlcG9ydGluZwo+ Pgo+PiBJT1BTOiAzMjdLCj4+IGF2ZyBsYXRlbmN5IG9mIGlvbW11X2RtYV91bm1hcF9zZygpOiAx MTExIG5zZWNzCj4+Cj4+Cj4+IC0gcnVuIGZpbyBvbiByZW1vdGUgQ1BVCj4+IHRhc2tzZXQgLWMg ODAgfi9naXQvdG9vbHMvdGVzdC9udm1lL2lvX3VyaW5nIDEwIDEgL2Rldi9udm1lMW4xIDRrCj4+ ICsgZmlvIC0tYnM9NGsgLS1pb2VuZ2luZT1pb191cmluZyAtLWZpeGVkYnVmcyAtLXJlZ2lzdGVy ZmlsZXMgLS1oaXByaSAKPj4gLS1pb2RlcHRoPTY0IC0taW9kZXB0aF9iYXRjaF9zdWJtaXQ9MTYg LS1pb2RlcHRoX2JhdGNoX2NvbXBsZXRlX21pbj0xNiAKPj4gLS1maWxlbmFtZT0vZGV2L252bWUx bjEgLS1kaXJlY3Q9MSAtLXJ1bnRpbWU9MTAgLS1udW1qb2JzPTEgCj4+IC0tcnc9cmFuZHJlYWQg LS1uYW1lPXRlc3QgLS1ncm91cF9yZXBvcnRpbmcKPj4KPj4gSU9QUzogMzQuOUsKPj4gYXZnIGxh dGVuY3kgb2YgaW9tbXVfZG1hX3VubWFwX3NnKCk6IDI1NDM3IG5zZWNzCj4+Cj4+IDIpIHN5c3Rl bSBpbmZvCj4+IFtyb290QGFtcGVyZS1tdGphZGUtMDQgfl0jIGxzY3B1IHwgZ3JlcCBOVU1BCj4+ IE5VTUEgbm9kZShzKTrCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAyCj4+ IE5VTUEgbm9kZTAgQ1BVKHMpOsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMC03OQo+PiBO VU1BIG5vZGUxIENQVShzKTrCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDgwLTE1OQo+Pgo+ PiBsc3BjaSB8IGdyZXAgTlZNZQo+PiAwMDAzOjAxOjAwLjAgTm9uLVZvbGF0aWxlIG1lbW9yeSBj b250cm9sbGVyOiBTYW1zdW5nIEVsZWN0cm9uaWNzIENvIAo+PiBMdGQgTlZNZSBTU0QgQ29udHJv bGxlciBTTTk4MS9QTTk4MS9QTTk4Mwo+Pgo+PiBbcm9vdEBhbXBlcmUtbXRqYWRlLTA0IH5dIyBj YXQgL3N5cy9ibG9jay9udm1lMW4xL2RldmljZS9kZXZpY2UvbnVtYV9ub2RlIAoKU2luY2UgaXQn cyBhbXBlcmUsIEkgZ3Vlc3MgaXQncyBzbW11IHYzLgoKQlRXLCBpZiB5b3UgcmVtZW1iZXIsIEkg ZGlkIHJhaXNlIGEgcGVyZm9ybWFuY2UgaXNzdWUgb2Ygc21tdXYzIHdpdGggCk5WTWUgYmVmb3Jl OgpodHRwczovL2xvcmUua2VybmVsLm9yZy9saW51eC1pb21tdS9iMmE2ZTI2ZC02ZDBkLTdmMGQt ZjIyMi01ODk4MTJmNzAxZDJAaHVhd2VpLmNvbS8KCkkgZGlkIGhhdmUgdGhpcyBzZXJpZXMgdG8g aW1wcm92ZSBwZXJmb3JtYW5jZSBmb3Igc3lzdGVtcyB3aXRoIGxvdHMgb2YgCkNQVXMsIGxpa2Ug YWJvdmUsIGJ1dCBub3QgYWNjZXB0ZWQ6Cmh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2xpbnV4LWlv bW11LzE1OTgwMTgwNjItMTc1NjA4LTEtZ2l0LXNlbmQtZW1haWwtam9obi5nYXJyeUBodWF3ZWku Y29tLwoKVGhhbmtzLApKb2huCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5l bEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4v bGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=