From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7F00C4708C for ; Fri, 28 May 2021 10:58:18 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B03E7613B5 for ; Fri, 28 May 2021 10:58:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B03E7613B5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DBE286F464; Fri, 28 May 2021 10:58:10 +0000 (UTC) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7A7FA6F464; Fri, 28 May 2021 10:58:08 +0000 (UTC) IronPort-SDR: ZmF/IFMAaxh46B4vUyNdyqL8cMicCwnXHH5da2y5OQBDWRTPS+ivH7am3unNVg9L+ftKgRWjll AnXNXiBnsf9w== X-IronPort-AV: E=McAfee;i="6200,9189,9997"; a="266821826" X-IronPort-AV: E=Sophos;i="5.83,229,1616482800"; d="scan'208";a="266821826" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2021 03:58:08 -0700 IronPort-SDR: ApjOyrII/4na76IPQH6nxFBGelByAn5NoAUKphJcfXdhTG+IcPJ3+f9J3/EiAEGjU4C0PkcUV4 M7yglSyUloUA== X-IronPort-AV: E=Sophos;i="5.83,229,1616482800"; d="scan'208";a="477885221" Received: from jdahlin-mobl1.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.92]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2021 03:58:05 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH v6 07/15] drm: Add a prefetching memcpy_from_wc Date: Fri, 28 May 2021 12:57:36 +0200 Message-Id: <20210528105744.58271-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210528105744.58271-1-thomas.hellstrom@linux.intel.com> References: <20210528105744.58271-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , =?UTF-8?q?Christian=20K=C3=B6nig?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Reading out of write-combining mapped memory is typically very slow since the CPU doesn't prefetch. However some archs have special instructions to do this. So add a best-effort memcpy_from_wc taking dma-buf-map pointer arguments that attempts to use a fast prefetching memcpy and otherwise falls back to ordinary memcopies, taking the iomem tagging into account. The code is largely copied from i915_memcpy_from_wc. Cc: Daniel Vetter Cc: Christian König Suggested-by: Daniel Vetter Signed-off-by: Thomas Hellström --- Documentation/gpu/drm-mm.rst | 2 +- drivers/gpu/drm/drm_cache.c | 138 +++++++++++++++++++++++++++++++++++ drivers/gpu/drm/drm_drv.c | 2 + include/drm/drm_cache.h | 7 ++ 4 files changed, 148 insertions(+), 1 deletion(-) diff --git a/Documentation/gpu/drm-mm.rst b/Documentation/gpu/drm-mm.rst index 21be6deadc12..c66058c5bce7 100644 --- a/Documentation/gpu/drm-mm.rst +++ b/Documentation/gpu/drm-mm.rst @@ -469,7 +469,7 @@ DRM MM Range Allocator Function References .. kernel-doc:: drivers/gpu/drm/drm_mm.c :export: -DRM Cache Handling +DRM Cache Handling and Fast WC memcpy() ================== .. kernel-doc:: drivers/gpu/drm/drm_cache.c diff --git a/drivers/gpu/drm/drm_cache.c b/drivers/gpu/drm/drm_cache.c index 79a50ef1250f..08614f7fdd8d 100644 --- a/drivers/gpu/drm/drm_cache.c +++ b/drivers/gpu/drm/drm_cache.c @@ -28,6 +28,7 @@ * Authors: Thomas Hellström */ +#include #include #include #include @@ -35,6 +36,9 @@ #include +/* A small bounce buffer that fits on the stack. */ +#define MEMCPY_BOUNCE_SIZE 128 + #if defined(CONFIG_X86) #include @@ -209,3 +213,137 @@ bool drm_need_swiotlb(int dma_bits) return max_iomem > ((u64)1 << dma_bits); } EXPORT_SYMBOL(drm_need_swiotlb); + +#ifdef CONFIG_X86 + +static DEFINE_STATIC_KEY_FALSE(has_movntdqa); + +static void __memcpy_ntdqa(void *dst, const void *src, unsigned long len) +{ + kernel_fpu_begin(); + + while (len >= 4) { + asm("movntdqa (%0), %%xmm0\n" + "movntdqa 16(%0), %%xmm1\n" + "movntdqa 32(%0), %%xmm2\n" + "movntdqa 48(%0), %%xmm3\n" + "movaps %%xmm0, (%1)\n" + "movaps %%xmm1, 16(%1)\n" + "movaps %%xmm2, 32(%1)\n" + "movaps %%xmm3, 48(%1)\n" + :: "r" (src), "r" (dst) : "memory"); + src += 64; + dst += 64; + len -= 4; + } + while (len--) { + asm("movntdqa (%0), %%xmm0\n" + "movaps %%xmm0, (%1)\n" + :: "r" (src), "r" (dst) : "memory"); + src += 16; + dst += 16; + } + + kernel_fpu_end(); +} + +/* + * __drm_memcpy_from_wc copies @len bytes from @src to @dst using + * non-temporal instructions where available. Note that all arguments + * (@src, @dst) must be aligned to 16 bytes and @len must be a multiple + * of 16. + */ +static void __drm_memcpy_from_wc(void *dst, const void *src, unsigned long len) +{ + if (unlikely(((unsigned long)dst | (unsigned long)src | len) & 15)) + memcpy(dst, src, len); + else if (likely(len)) + __memcpy_ntdqa(dst, src, len >> 4); +} +#endif + +static void memcpy_fallback(struct dma_buf_map *dst, + const struct dma_buf_map *src, + unsigned long len) +{ + if (!dst->is_iomem && !src->is_iomem) { + memcpy(dst->vaddr, src->vaddr, len); + } else if (!src->is_iomem) { + dma_buf_map_memcpy_to(dst, src->vaddr, len); + } else if (!dst->is_iomem) { + memcpy_fromio(dst->vaddr, src->vaddr_iomem, len); + } else { + /* + * Bounce size is not performance tuned, but using a + * bounce buffer like this is significantly faster than + * resorting to ioreadxx() + iowritexx(). + */ + char bounce[MEMCPY_BOUNCE_SIZE]; + void __iomem *_src = src->vaddr_iomem; + void __iomem *_dst = dst->vaddr_iomem; + + while (len >= MEMCPY_BOUNCE_SIZE) { + memcpy_fromio(bounce, _src, MEMCPY_BOUNCE_SIZE); + memcpy_toio(_dst, bounce, MEMCPY_BOUNCE_SIZE); + _src += MEMCPY_BOUNCE_SIZE; + _dst += MEMCPY_BOUNCE_SIZE; + len -= MEMCPY_BOUNCE_SIZE; + } + if (len) { + memcpy_fromio(bounce, _src, MEMCPY_BOUNCE_SIZE); + memcpy_toio(_dst, bounce, MEMCPY_BOUNCE_SIZE); + } + } +} + +/** + * drm_memcpy_from_wc - Perform the fastest available memcpy from a source + * that may be WC. + * @dst: The destination pointer + * @src: The source pointer + * @len: The size of the area o transfer in bytes + * + * Tries an arch optimized memcpy for prefetching reading out of a WC region, + * and if no such beast is available, falls back to a normal memcpy. + */ +void drm_memcpy_from_wc(struct dma_buf_map *dst, + const struct dma_buf_map *src, + unsigned long len) +{ + if (WARN_ON(in_interrupt())) + return; + + if (IS_ENABLED(CONFIG_X86) && static_branch_likely(&has_movntdqa)) { + __drm_memcpy_from_wc(dst->is_iomem ? + (void __force *)dst->vaddr_iomem : + dst->vaddr, + src->is_iomem ? + (void const __force *)src->vaddr_iomem : + src->vaddr, + len); + return; + } + + memcpy_fallback(dst, src, len); +} +EXPORT_SYMBOL(drm_memcpy_from_wc); + +#ifdef CONFIG_X86 +/** + * drm_memcpy_init_early - One time initialization of the WC memcpy code + */ +void drm_memcpy_init_early(void) +{ + /* + * Some hypervisors (e.g. KVM) don't support VEX-prefix instructions + * emulation. So don't enable movntdqa in hypervisor guest. + */ + if (static_cpu_has(X86_FEATURE_XMM4_1) && + !boot_cpu_has(X86_FEATURE_HYPERVISOR)) + static_branch_enable(&has_movntdqa); +} +#else +void drm_memcpy_init_early(void) +{ +} +#endif diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c index 3d8d68a98b95..8804ec7d3215 100644 --- a/drivers/gpu/drm/drm_drv.c +++ b/drivers/gpu/drm/drm_drv.c @@ -35,6 +35,7 @@ #include #include +#include #include #include #include @@ -1041,6 +1042,7 @@ static int __init drm_core_init(void) drm_connector_ida_init(); idr_init(&drm_minors_idr); + drm_memcpy_init_early(); ret = drm_sysfs_init(); if (ret < 0) { diff --git a/include/drm/drm_cache.h b/include/drm/drm_cache.h index e9ad4863d915..cc9de1632dd3 100644 --- a/include/drm/drm_cache.h +++ b/include/drm/drm_cache.h @@ -35,6 +35,8 @@ #include +struct dma_buf_map; + void drm_clflush_pages(struct page *pages[], unsigned long num_pages); void drm_clflush_sg(struct sg_table *st); void drm_clflush_virt_range(void *addr, unsigned long length); @@ -70,4 +72,9 @@ static inline bool drm_arch_can_wc_memory(void) #endif } +void drm_memcpy_init_early(void); + +void drm_memcpy_from_wc(struct dma_buf_map *dst, + const struct dma_buf_map *src, + unsigned long len); #endif -- 2.31.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BC36C47091 for ; Fri, 28 May 2021 10:58:18 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F335C613B5 for ; Fri, 28 May 2021 10:58:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F335C613B5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id AAC9E6F51B; Fri, 28 May 2021 10:58:10 +0000 (UTC) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7A7FA6F464; Fri, 28 May 2021 10:58:08 +0000 (UTC) IronPort-SDR: ZmF/IFMAaxh46B4vUyNdyqL8cMicCwnXHH5da2y5OQBDWRTPS+ivH7am3unNVg9L+ftKgRWjll AnXNXiBnsf9w== X-IronPort-AV: E=McAfee;i="6200,9189,9997"; a="266821826" X-IronPort-AV: E=Sophos;i="5.83,229,1616482800"; d="scan'208";a="266821826" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2021 03:58:08 -0700 IronPort-SDR: ApjOyrII/4na76IPQH6nxFBGelByAn5NoAUKphJcfXdhTG+IcPJ3+f9J3/EiAEGjU4C0PkcUV4 M7yglSyUloUA== X-IronPort-AV: E=Sophos;i="5.83,229,1616482800"; d="scan'208";a="477885221" Received: from jdahlin-mobl1.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.92]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2021 03:58:05 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Fri, 28 May 2021 12:57:36 +0200 Message-Id: <20210528105744.58271-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210528105744.58271-1-thomas.hellstrom@linux.intel.com> References: <20210528105744.58271-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v6 07/15] drm: Add a prefetching memcpy_from_wc X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , =?UTF-8?q?Christian=20K=C3=B6nig?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" UmVhZGluZyBvdXQgb2Ygd3JpdGUtY29tYmluaW5nIG1hcHBlZCBtZW1vcnkgaXMgdHlwaWNhbGx5 IHZlcnkgc2xvdwpzaW5jZSB0aGUgQ1BVIGRvZXNuJ3QgcHJlZmV0Y2guIEhvd2V2ZXIgc29tZSBh cmNocyBoYXZlIHNwZWNpYWwKaW5zdHJ1Y3Rpb25zIHRvIGRvIHRoaXMuCgpTbyBhZGQgYSBiZXN0 LWVmZm9ydCBtZW1jcHlfZnJvbV93YyB0YWtpbmcgZG1hLWJ1Zi1tYXAgcG9pbnRlcgphcmd1bWVu dHMgdGhhdCBhdHRlbXB0cyB0byB1c2UgYSBmYXN0IHByZWZldGNoaW5nIG1lbWNweSBhbmQKb3Ro ZXJ3aXNlIGZhbGxzIGJhY2sgdG8gb3JkaW5hcnkgbWVtY29waWVzLCB0YWtpbmcgdGhlIGlvbWVt IHRhZ2dpbmcKaW50byBhY2NvdW50LgoKVGhlIGNvZGUgaXMgbGFyZ2VseSBjb3BpZWQgZnJvbSBp OTE1X21lbWNweV9mcm9tX3djLgoKQ2M6IERhbmllbCBWZXR0ZXIgPGRhbmllbEBmZndsbC5jaD4K Q2M6IENocmlzdGlhbiBLw7ZuaWcgPGNocmlzdGlhbi5rb2VuaWdAYW1kLmNvbT4KU3VnZ2VzdGVk LWJ5OiBEYW5pZWwgVmV0dGVyIDxkYW5pZWxAZmZ3bGwuY2g+ClNpZ25lZC1vZmYtYnk6IFRob21h cyBIZWxsc3Ryw7ZtIDx0aG9tYXMuaGVsbHN0cm9tQGxpbnV4LmludGVsLmNvbT4KLS0tCiBEb2N1 bWVudGF0aW9uL2dwdS9kcm0tbW0ucnN0IHwgICAyICstCiBkcml2ZXJzL2dwdS9kcm0vZHJtX2Nh Y2hlLmMgIHwgMTM4ICsrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrCiBkcml2ZXJz L2dwdS9kcm0vZHJtX2Rydi5jICAgIHwgICAyICsKIGluY2x1ZGUvZHJtL2RybV9jYWNoZS5oICAg ICAgfCAgIDcgKysKIDQgZmlsZXMgY2hhbmdlZCwgMTQ4IGluc2VydGlvbnMoKyksIDEgZGVsZXRp b24oLSkKCmRpZmYgLS1naXQgYS9Eb2N1bWVudGF0aW9uL2dwdS9kcm0tbW0ucnN0IGIvRG9jdW1l bnRhdGlvbi9ncHUvZHJtLW1tLnJzdAppbmRleCAyMWJlNmRlYWRjMTIuLmM2NjA1OGM1YmNlNyAx MDA2NDQKLS0tIGEvRG9jdW1lbnRhdGlvbi9ncHUvZHJtLW1tLnJzdAorKysgYi9Eb2N1bWVudGF0 aW9uL2dwdS9kcm0tbW0ucnN0CkBAIC00NjksNyArNDY5LDcgQEAgRFJNIE1NIFJhbmdlIEFsbG9j YXRvciBGdW5jdGlvbiBSZWZlcmVuY2VzCiAuLiBrZXJuZWwtZG9jOjogZHJpdmVycy9ncHUvZHJt L2RybV9tbS5jCiAgICA6ZXhwb3J0OgogCi1EUk0gQ2FjaGUgSGFuZGxpbmcKK0RSTSBDYWNoZSBI YW5kbGluZyBhbmQgRmFzdCBXQyBtZW1jcHkoKQogPT09PT09PT09PT09PT09PT09CiAKIC4uIGtl cm5lbC1kb2M6OiBkcml2ZXJzL2dwdS9kcm0vZHJtX2NhY2hlLmMKZGlmZiAtLWdpdCBhL2RyaXZl cnMvZ3B1L2RybS9kcm1fY2FjaGUuYyBiL2RyaXZlcnMvZ3B1L2RybS9kcm1fY2FjaGUuYwppbmRl eCA3OWE1MGVmMTI1MGYuLjA4NjE0ZjdmZGQ4ZCAxMDA2NDQKLS0tIGEvZHJpdmVycy9ncHUvZHJt L2RybV9jYWNoZS5jCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9kcm1fY2FjaGUuYwpAQCAtMjgsNiAr MjgsNyBAQAogICogQXV0aG9yczogVGhvbWFzIEhlbGxzdHLDtm0gPHRob21hcy1hdC10dW5nc3Rl bmdyYXBoaWNzLWRvdC1jb20+CiAgKi8KIAorI2luY2x1ZGUgPGxpbnV4L2RtYS1idWYtbWFwLmg+ CiAjaW5jbHVkZSA8bGludXgvZXhwb3J0Lmg+CiAjaW5jbHVkZSA8bGludXgvaGlnaG1lbS5oPgog I2luY2x1ZGUgPGxpbnV4L21lbV9lbmNyeXB0Lmg+CkBAIC0zNSw2ICszNiw5IEBACiAKICNpbmNs dWRlIDxkcm0vZHJtX2NhY2hlLmg+CiAKKy8qIEEgc21hbGwgYm91bmNlIGJ1ZmZlciB0aGF0IGZp dHMgb24gdGhlIHN0YWNrLiAqLworI2RlZmluZSBNRU1DUFlfQk9VTkNFX1NJWkUgMTI4CisKICNp ZiBkZWZpbmVkKENPTkZJR19YODYpCiAjaW5jbHVkZSA8YXNtL3NtcC5oPgogCkBAIC0yMDksMyAr MjEzLDEzNyBAQCBib29sIGRybV9uZWVkX3N3aW90bGIoaW50IGRtYV9iaXRzKQogCXJldHVybiBt YXhfaW9tZW0gPiAoKHU2NCkxIDw8IGRtYV9iaXRzKTsKIH0KIEVYUE9SVF9TWU1CT0woZHJtX25l ZWRfc3dpb3RsYik7CisKKyNpZmRlZiBDT05GSUdfWDg2CisKK3N0YXRpYyBERUZJTkVfU1RBVElD X0tFWV9GQUxTRShoYXNfbW92bnRkcWEpOworCitzdGF0aWMgdm9pZCBfX21lbWNweV9udGRxYSh2 b2lkICpkc3QsIGNvbnN0IHZvaWQgKnNyYywgdW5zaWduZWQgbG9uZyBsZW4pCit7CisJa2VybmVs X2ZwdV9iZWdpbigpOworCisJd2hpbGUgKGxlbiA+PSA0KSB7CisJCWFzbSgibW92bnRkcWEJKCUw KSwgJSV4bW0wXG4iCisJCSAgICAibW92bnRkcWEgMTYoJTApLCAlJXhtbTFcbiIKKwkJICAgICJt b3ZudGRxYSAzMiglMCksICUleG1tMlxuIgorCQkgICAgIm1vdm50ZHFhIDQ4KCUwKSwgJSV4bW0z XG4iCisJCSAgICAibW92YXBzICUleG1tMCwgICAoJTEpXG4iCisJCSAgICAibW92YXBzICUleG1t MSwgMTYoJTEpXG4iCisJCSAgICAibW92YXBzICUleG1tMiwgMzIoJTEpXG4iCisJCSAgICAibW92 YXBzICUleG1tMywgNDgoJTEpXG4iCisJCSAgICA6OiAiciIgKHNyYyksICJyIiAoZHN0KSA6ICJt ZW1vcnkiKTsKKwkJc3JjICs9IDY0OworCQlkc3QgKz0gNjQ7CisJCWxlbiAtPSA0OworCX0KKwl3 aGlsZSAobGVuLS0pIHsKKwkJYXNtKCJtb3ZudGRxYSAoJTApLCAlJXhtbTBcbiIKKwkJICAgICJt b3ZhcHMgJSV4bW0wLCAoJTEpXG4iCisJCSAgICA6OiAiciIgKHNyYyksICJyIiAoZHN0KSA6ICJt ZW1vcnkiKTsKKwkJc3JjICs9IDE2OworCQlkc3QgKz0gMTY7CisJfQorCisJa2VybmVsX2ZwdV9l bmQoKTsKK30KKworLyoKKyAqIF9fZHJtX21lbWNweV9mcm9tX3djIGNvcGllcyBAbGVuIGJ5dGVz IGZyb20gQHNyYyB0byBAZHN0IHVzaW5nCisgKiBub24tdGVtcG9yYWwgaW5zdHJ1Y3Rpb25zIHdo ZXJlIGF2YWlsYWJsZS4gTm90ZSB0aGF0IGFsbCBhcmd1bWVudHMKKyAqIChAc3JjLCBAZHN0KSBt dXN0IGJlIGFsaWduZWQgdG8gMTYgYnl0ZXMgYW5kIEBsZW4gbXVzdCBiZSBhIG11bHRpcGxlCisg KiBvZiAxNi4KKyAqLworc3RhdGljIHZvaWQgX19kcm1fbWVtY3B5X2Zyb21fd2Modm9pZCAqZHN0 LCBjb25zdCB2b2lkICpzcmMsIHVuc2lnbmVkIGxvbmcgbGVuKQoreworCWlmICh1bmxpa2VseSgo KHVuc2lnbmVkIGxvbmcpZHN0IHwgKHVuc2lnbmVkIGxvbmcpc3JjIHwgbGVuKSAmIDE1KSkKKwkJ bWVtY3B5KGRzdCwgc3JjLCBsZW4pOworCWVsc2UgaWYgKGxpa2VseShsZW4pKQorCQlfX21lbWNw eV9udGRxYShkc3QsIHNyYywgbGVuID4+IDQpOworfQorI2VuZGlmCisKK3N0YXRpYyB2b2lkIG1l bWNweV9mYWxsYmFjayhzdHJ1Y3QgZG1hX2J1Zl9tYXAgKmRzdCwKKwkJCSAgICBjb25zdCBzdHJ1 Y3QgZG1hX2J1Zl9tYXAgKnNyYywKKwkJCSAgICB1bnNpZ25lZCBsb25nIGxlbikKK3sKKwlpZiAo IWRzdC0+aXNfaW9tZW0gJiYgIXNyYy0+aXNfaW9tZW0pIHsKKwkJbWVtY3B5KGRzdC0+dmFkZHIs IHNyYy0+dmFkZHIsIGxlbik7CisJfSBlbHNlIGlmICghc3JjLT5pc19pb21lbSkgeworCQlkbWFf YnVmX21hcF9tZW1jcHlfdG8oZHN0LCBzcmMtPnZhZGRyLCBsZW4pOworCX0gZWxzZSBpZiAoIWRz dC0+aXNfaW9tZW0pIHsKKwkJbWVtY3B5X2Zyb21pbyhkc3QtPnZhZGRyLCBzcmMtPnZhZGRyX2lv bWVtLCBsZW4pOworCX0gZWxzZSB7CisJCS8qCisJCSAqIEJvdW5jZSBzaXplIGlzIG5vdCBwZXJm b3JtYW5jZSB0dW5lZCwgYnV0IHVzaW5nIGEKKwkJICogYm91bmNlIGJ1ZmZlciBsaWtlIHRoaXMg aXMgc2lnbmlmaWNhbnRseSBmYXN0ZXIgdGhhbgorCQkgKiByZXNvcnRpbmcgdG8gaW9yZWFkeHgo KSArIGlvd3JpdGV4eCgpLgorCQkgKi8KKwkJY2hhciBib3VuY2VbTUVNQ1BZX0JPVU5DRV9TSVpF XTsKKwkJdm9pZCBfX2lvbWVtICpfc3JjID0gc3JjLT52YWRkcl9pb21lbTsKKwkJdm9pZCBfX2lv bWVtICpfZHN0ID0gZHN0LT52YWRkcl9pb21lbTsKKworCQl3aGlsZSAobGVuID49IE1FTUNQWV9C T1VOQ0VfU0laRSkgeworCQkJbWVtY3B5X2Zyb21pbyhib3VuY2UsIF9zcmMsIE1FTUNQWV9CT1VO Q0VfU0laRSk7CisJCQltZW1jcHlfdG9pbyhfZHN0LCBib3VuY2UsIE1FTUNQWV9CT1VOQ0VfU0la RSk7CisJCQlfc3JjICs9IE1FTUNQWV9CT1VOQ0VfU0laRTsKKwkJCV9kc3QgKz0gTUVNQ1BZX0JP VU5DRV9TSVpFOworCQkJbGVuIC09IE1FTUNQWV9CT1VOQ0VfU0laRTsKKwkJfQorCQlpZiAobGVu KSB7CisJCQltZW1jcHlfZnJvbWlvKGJvdW5jZSwgX3NyYywgTUVNQ1BZX0JPVU5DRV9TSVpFKTsK KwkJCW1lbWNweV90b2lvKF9kc3QsIGJvdW5jZSwgTUVNQ1BZX0JPVU5DRV9TSVpFKTsKKwkJfQor CX0KK30KKworLyoqCisgKiBkcm1fbWVtY3B5X2Zyb21fd2MgLSBQZXJmb3JtIHRoZSBmYXN0ZXN0 IGF2YWlsYWJsZSBtZW1jcHkgZnJvbSBhIHNvdXJjZQorICogdGhhdCBtYXkgYmUgV0MuCisgKiBA ZHN0OiBUaGUgZGVzdGluYXRpb24gcG9pbnRlcgorICogQHNyYzogVGhlIHNvdXJjZSBwb2ludGVy CisgKiBAbGVuOiBUaGUgc2l6ZSBvZiB0aGUgYXJlYSBvIHRyYW5zZmVyIGluIGJ5dGVzCisgKgor ICogVHJpZXMgYW4gYXJjaCBvcHRpbWl6ZWQgbWVtY3B5IGZvciBwcmVmZXRjaGluZyByZWFkaW5n IG91dCBvZiBhIFdDIHJlZ2lvbiwKKyAqIGFuZCBpZiBubyBzdWNoIGJlYXN0IGlzIGF2YWlsYWJs ZSwgZmFsbHMgYmFjayB0byBhIG5vcm1hbCBtZW1jcHkuCisgKi8KK3ZvaWQgZHJtX21lbWNweV9m cm9tX3djKHN0cnVjdCBkbWFfYnVmX21hcCAqZHN0LAorCQkJY29uc3Qgc3RydWN0IGRtYV9idWZf bWFwICpzcmMsCisJCQl1bnNpZ25lZCBsb25nIGxlbikKK3sKKwlpZiAoV0FSTl9PTihpbl9pbnRl cnJ1cHQoKSkpCisJCXJldHVybjsKKworCWlmIChJU19FTkFCTEVEKENPTkZJR19YODYpICYmIHN0 YXRpY19icmFuY2hfbGlrZWx5KCZoYXNfbW92bnRkcWEpKSB7CisJCV9fZHJtX21lbWNweV9mcm9t X3djKGRzdC0+aXNfaW9tZW0gPworCQkJCSAgICAgKHZvaWQgX19mb3JjZSAqKWRzdC0+dmFkZHJf aW9tZW0gOgorCQkJCSAgICAgZHN0LT52YWRkciwKKwkJCQkgICAgIHNyYy0+aXNfaW9tZW0gPwor CQkJCSAgICAgKHZvaWQgY29uc3QgX19mb3JjZSAqKXNyYy0+dmFkZHJfaW9tZW0gOgorCQkJCSAg ICAgc3JjLT52YWRkciwKKwkJCQkgICAgIGxlbik7CisJCXJldHVybjsKKwl9CisKKwltZW1jcHlf ZmFsbGJhY2soZHN0LCBzcmMsIGxlbik7Cit9CitFWFBPUlRfU1lNQk9MKGRybV9tZW1jcHlfZnJv bV93Yyk7CisKKyNpZmRlZiBDT05GSUdfWDg2CisvKioKKyAqIGRybV9tZW1jcHlfaW5pdF9lYXJs eSAtIE9uZSB0aW1lIGluaXRpYWxpemF0aW9uIG9mIHRoZSBXQyBtZW1jcHkgY29kZQorICovCit2 b2lkIGRybV9tZW1jcHlfaW5pdF9lYXJseSh2b2lkKQoreworCS8qCisJICogU29tZSBoeXBlcnZp c29ycyAoZS5nLiBLVk0pIGRvbid0IHN1cHBvcnQgVkVYLXByZWZpeCBpbnN0cnVjdGlvbnMKKwkg KiBlbXVsYXRpb24uIFNvIGRvbid0IGVuYWJsZSBtb3ZudGRxYSBpbiBoeXBlcnZpc29yIGd1ZXN0 LgorCSAqLworCWlmIChzdGF0aWNfY3B1X2hhcyhYODZfRkVBVFVSRV9YTU00XzEpICYmCisJICAg ICFib290X2NwdV9oYXMoWDg2X0ZFQVRVUkVfSFlQRVJWSVNPUikpCisJCXN0YXRpY19icmFuY2hf ZW5hYmxlKCZoYXNfbW92bnRkcWEpOworfQorI2Vsc2UKK3ZvaWQgZHJtX21lbWNweV9pbml0X2Vh cmx5KHZvaWQpCit7Cit9CisjZW5kaWYKZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9kcm1f ZHJ2LmMgYi9kcml2ZXJzL2dwdS9kcm0vZHJtX2Rydi5jCmluZGV4IDNkOGQ2OGE5OGI5NS4uODgw NGVjN2QzMjE1IDEwMDY0NAotLS0gYS9kcml2ZXJzL2dwdS9kcm0vZHJtX2Rydi5jCisrKyBiL2Ry aXZlcnMvZ3B1L2RybS9kcm1fZHJ2LmMKQEAgLTM1LDYgKzM1LDcgQEAKICNpbmNsdWRlIDxsaW51 eC9zbGFiLmg+CiAjaW5jbHVkZSA8bGludXgvc3JjdS5oPgogCisjaW5jbHVkZSA8ZHJtL2RybV9j YWNoZS5oPgogI2luY2x1ZGUgPGRybS9kcm1fY2xpZW50Lmg+CiAjaW5jbHVkZSA8ZHJtL2RybV9j b2xvcl9tZ210Lmg+CiAjaW5jbHVkZSA8ZHJtL2RybV9kcnYuaD4KQEAgLTEwNDEsNiArMTA0Miw3 IEBAIHN0YXRpYyBpbnQgX19pbml0IGRybV9jb3JlX2luaXQodm9pZCkKIAogCWRybV9jb25uZWN0 b3JfaWRhX2luaXQoKTsKIAlpZHJfaW5pdCgmZHJtX21pbm9yc19pZHIpOworCWRybV9tZW1jcHlf aW5pdF9lYXJseSgpOwogCiAJcmV0ID0gZHJtX3N5c2ZzX2luaXQoKTsKIAlpZiAocmV0IDwgMCkg ewpkaWZmIC0tZ2l0IGEvaW5jbHVkZS9kcm0vZHJtX2NhY2hlLmggYi9pbmNsdWRlL2RybS9kcm1f Y2FjaGUuaAppbmRleCBlOWFkNDg2M2Q5MTUuLmNjOWRlMTYzMmRkMyAxMDA2NDQKLS0tIGEvaW5j bHVkZS9kcm0vZHJtX2NhY2hlLmgKKysrIGIvaW5jbHVkZS9kcm0vZHJtX2NhY2hlLmgKQEAgLTM1 LDYgKzM1LDggQEAKIAogI2luY2x1ZGUgPGxpbnV4L3NjYXR0ZXJsaXN0Lmg+CiAKK3N0cnVjdCBk bWFfYnVmX21hcDsKKwogdm9pZCBkcm1fY2xmbHVzaF9wYWdlcyhzdHJ1Y3QgcGFnZSAqcGFnZXNb XSwgdW5zaWduZWQgbG9uZyBudW1fcGFnZXMpOwogdm9pZCBkcm1fY2xmbHVzaF9zZyhzdHJ1Y3Qg c2dfdGFibGUgKnN0KTsKIHZvaWQgZHJtX2NsZmx1c2hfdmlydF9yYW5nZSh2b2lkICphZGRyLCB1 bnNpZ25lZCBsb25nIGxlbmd0aCk7CkBAIC03MCw0ICs3Miw5IEBAIHN0YXRpYyBpbmxpbmUgYm9v bCBkcm1fYXJjaF9jYW5fd2NfbWVtb3J5KHZvaWQpCiAjZW5kaWYKIH0KIAordm9pZCBkcm1fbWVt Y3B5X2luaXRfZWFybHkodm9pZCk7CisKK3ZvaWQgZHJtX21lbWNweV9mcm9tX3djKHN0cnVjdCBk bWFfYnVmX21hcCAqZHN0LAorCQkJY29uc3Qgc3RydWN0IGRtYV9idWZfbWFwICpzcmMsCisJCQl1 bnNpZ25lZCBsb25nIGxlbik7CiAjZW5kaWYKLS0gCjIuMzEuMQoKX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KSW50ZWwtZ2Z4IG1haWxpbmcgbGlzdApJbnRl bC1nZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3Jn L21haWxtYW4vbGlzdGluZm8vaW50ZWwtZ2Z4Cg==