From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21D06C48BE8 for ; Fri, 11 Jun 2021 14:55:24 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E55DE613D0 for ; Fri, 11 Jun 2021 14:55:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E55DE613D0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A3B5F6EEA0; Fri, 11 Jun 2021 14:55:21 +0000 (UTC) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by gabe.freedesktop.org (Postfix) with ESMTPS id C8B146EE9E; Fri, 11 Jun 2021 14:55:18 +0000 (UTC) IronPort-SDR: DGCoJfru9O0uSyxnxYwhYxpcEEtOcbd/4tzC7lvySm+fxD/Pfi0qip2qyQnslLg0vSnKFkoSRg dZKohbyd3QvQ== X-IronPort-AV: E=McAfee;i="6200,9189,10012"; a="205355258" X-IronPort-AV: E=Sophos;i="5.83,265,1616482800"; d="scan'208";a="205355258" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jun 2021 07:55:18 -0700 IronPort-SDR: QSEBlbmSp4T3a5V/ljLLWHYPxBlY7AZIMiHvN2tCjuy++urUsiBOeqhkrMVhYfotMeUVghQiOq 6uCaZI0R6TxQ== X-IronPort-AV: E=Sophos;i="5.83,265,1616482800"; d="scan'208";a="450783217" Received: from delmer-mobl.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.23]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jun 2021 07:55:17 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 2/4] drm/i915/ttm: Adjust gem flags and caching settings after a move Date: Fri, 11 Jun 2021 16:54:57 +0200 Message-Id: <20210611145459.8576-3-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210611145459.8576-1-thomas.hellstrom@linux.intel.com> References: <20210611145459.8576-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" After a TTM move we need to update the i915 gem flags and caching settings to reflect the new placement. Also introduce gpu_binds_iomem() and cpu_maps_iomem() to clean up the various ways we previously used to detect this. Finally, initialize the TTM object reserved to be able to update flags and caching before anyone else gets hold of the object. Signed-off-by: Thomas Hellström --- v2: - Style fixes (Reported by Matthew Auld) --- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 112 +++++++++++++++++++----- 1 file changed, 90 insertions(+), 22 deletions(-) diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c index 33ab47f1e05b..45ef1d101937 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c @@ -70,6 +70,17 @@ static struct ttm_placement i915_sys_placement = { .busy_placement = &lmem0_sys_placement_flags[1], }; +static bool gpu_binds_iomem(struct ttm_resource *mem) +{ + return mem->mem_type != TTM_PL_SYSTEM; +} + +static bool cpu_maps_iomem(struct ttm_resource *mem) +{ + /* Once / if we support GGTT, this is also false for cached ttm_tts */ + return mem->mem_type != TTM_PL_SYSTEM; +} + static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj); static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo, @@ -175,6 +186,41 @@ static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj) obj->ttm.cached_io_st = NULL; } +static void +i915_ttm_adjust_domains_after_cpu_move(struct drm_i915_gem_object *obj) +{ + struct ttm_buffer_object *bo = i915_gem_to_ttm(obj); + + if (cpu_maps_iomem(bo->resource) || bo->ttm->caching != ttm_cached) { + obj->write_domain = I915_GEM_DOMAIN_WC; + obj->read_domains = I915_GEM_DOMAIN_WC; + } else { + obj->write_domain = I915_GEM_DOMAIN_CPU; + obj->read_domains = I915_GEM_DOMAIN_CPU; + } +} + +static void i915_ttm_adjust_gem_after_move(struct drm_i915_gem_object *obj) +{ + struct drm_i915_private *i915 = to_i915(obj->base.dev); + struct ttm_buffer_object *bo = i915_gem_to_ttm(obj); + unsigned int cache_level; + + obj->mem_flags &= ~(I915_BO_FLAG_STRUCT_PAGE | I915_BO_FLAG_IOMEM); + + obj->mem_flags |= cpu_maps_iomem(bo->resource) ? I915_BO_FLAG_IOMEM : + I915_BO_FLAG_STRUCT_PAGE; + + if ((HAS_LLC(i915) || HAS_SNOOP(i915)) && !gpu_binds_iomem(bo->resource) && + bo->ttm->caching == ttm_cached) { + cache_level = I915_CACHE_LLC; + } else { + cache_level = I915_CACHE_NONE; + } + + i915_gem_object_set_cache_coherency(obj, cache_level); +} + static void i915_ttm_purge(struct drm_i915_gem_object *obj) { struct ttm_buffer_object *bo = i915_gem_to_ttm(obj); @@ -190,8 +236,10 @@ static void i915_ttm_purge(struct drm_i915_gem_object *obj) /* TTM's purge interface. Note that we might be reentering. */ ret = ttm_bo_validate(bo, &place, &ctx); - if (!ret) { + obj->write_domain = 0; + obj->read_domains = 0; + i915_ttm_adjust_gem_after_move(obj); i915_ttm_free_cached_io_st(obj); obj->mm.madv = __I915_MADV_PURGED; } @@ -273,12 +321,15 @@ i915_ttm_resource_get_st(struct drm_i915_gem_object *obj, struct ttm_resource *res) { struct ttm_buffer_object *bo = i915_gem_to_ttm(obj); - struct ttm_resource_manager *man = - ttm_manager_type(bo->bdev, res->mem_type); - if (man->use_tt) + if (!gpu_binds_iomem(res)) return i915_ttm_tt_get_st(bo->ttm); + /* + * If CPU mapping differs, we need to add the ttm_tt pages to + * the resulting st. Might make sense for GGTT. + */ + GEM_WARN_ON(!cpu_maps_iomem(res)); return intel_region_ttm_node_to_st(obj->mm.region, res); } @@ -290,8 +341,6 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict, struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo); struct ttm_resource_manager *dst_man = ttm_manager_type(bo->bdev, dst_mem->mem_type); - struct ttm_resource_manager *src_man = - ttm_manager_type(bo->bdev, bo->resource->mem_type); struct intel_memory_region *dst_reg, *src_reg; union { struct ttm_kmap_iter_tt tt; @@ -332,34 +381,36 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict, if (IS_ERR(dst_st)) return PTR_ERR(dst_st); - /* If we start mapping GGTT, we can no longer use man::use_tt here. */ - dst_iter = dst_man->use_tt ? + dst_iter = !cpu_maps_iomem(dst_mem) ? ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) : ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap, dst_st, dst_reg->region.start); - src_iter = src_man->use_tt ? + src_iter = !cpu_maps_iomem(bo->resource) ? ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) : ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap, obj->ttm.cached_io_st, src_reg->region.start); ttm_move_memcpy(bo, dst_mem->num_pages, dst_iter, src_iter); + /* Below dst_mem becomes bo->resource. */ ttm_bo_move_sync_cleanup(bo, dst_mem); + i915_ttm_adjust_domains_after_cpu_move(obj); i915_ttm_free_cached_io_st(obj); - if (!dst_man->use_tt) { + if (gpu_binds_iomem(dst_mem) || cpu_maps_iomem(dst_mem)) { obj->ttm.cached_io_st = dst_st; obj->ttm.get_io_page.sg_pos = dst_st->sgl; obj->ttm.get_io_page.sg_idx = 0; } + i915_ttm_adjust_gem_after_move(obj); return 0; } static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource *mem) { - if (mem->mem_type < I915_PL_LMEM0) + if (!cpu_maps_iomem(mem)) return 0; mem->bus.caching = ttm_write_combined; @@ -421,6 +472,16 @@ static int i915_ttm_get_pages(struct drm_i915_gem_object *obj) if (ret) return ret == -ENOSPC ? -ENXIO : ret; + i915_ttm_adjust_lru(obj); + if (bo->ttm && !ttm_tt_is_populated(bo->ttm)) { + ret = ttm_tt_populate(bo->bdev, bo->ttm, &ctx); + if (ret) + return ret; + + i915_ttm_adjust_domains_after_cpu_move(obj); + i915_ttm_adjust_gem_after_move(obj); + } + /* Object either has a page vector or is an iomem object */ st = bo->ttm ? i915_ttm_tt_get_st(bo->ttm) : obj->ttm.cached_io_st; if (IS_ERR(st)) @@ -428,8 +489,6 @@ static int i915_ttm_get_pages(struct drm_i915_gem_object *obj) __i915_gem_object_set_pages(obj, st, i915_sg_dma_sizes(st->sgl)); - i915_ttm_adjust_lru(obj); - return ret; } @@ -563,6 +622,7 @@ static u64 i915_ttm_mmap_offset(struct drm_i915_gem_object *obj) const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = { .name = "i915_gem_object_ttm", + .flags = I915_GEM_OBJECT_IS_SHRINKABLE, .get_pages = i915_ttm_get_pages, .put_pages = i915_ttm_put_pages, @@ -599,6 +659,10 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem, { static struct lock_class_key lock_class; struct drm_i915_private *i915 = mem->i915; + struct ttm_operation_ctx ctx = { + .interruptible = true, + .no_wait_gpu = false, + }; enum ttm_bo_type bo_type; size_t alignment = 0; int ret; @@ -618,15 +682,14 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem, i915_gem_object_init(obj, &i915_gem_ttm_obj_ops, &lock_class, flags); i915_gem_object_init_memory_region(obj, mem); i915_gem_object_make_unshrinkable(obj); - obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT; - obj->mem_flags |= I915_BO_FLAG_IOMEM; - i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE); INIT_RADIX_TREE(&obj->ttm.get_io_page.radix, GFP_KERNEL | __GFP_NOWARN); mutex_init(&obj->ttm.get_io_page.lock); bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device : ttm_bo_type_kernel; + obj->base.vma_node.driver_private = i915_gem_to_ttm(obj); + /* * If this function fails, it will call the destructor, but * our caller still owns the object. So no freeing in the @@ -634,14 +697,19 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem, * Similarly, in delayed_destroy, we can't call ttm_bo_put() * until successful initialization. */ - obj->base.vma_node.driver_private = i915_gem_to_ttm(obj); - ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size, - bo_type, &i915_sys_placement, alignment, - true, NULL, NULL, i915_ttm_bo_destroy); + ret = ttm_bo_init_reserved(&i915->bdev, i915_gem_to_ttm(obj), size, + bo_type, &i915_sys_placement, alignment, + &ctx, NULL, NULL, i915_ttm_bo_destroy); + + if (ret) + goto out; - if (!ret) - obj->ttm.created = true; + obj->ttm.created = true; + i915_ttm_adjust_domains_after_cpu_move(obj); + i915_ttm_adjust_gem_after_move(obj); + i915_gem_object_unlock(obj); +out: /* i915 wants -ENXIO when out of memory region space. */ return (ret == -ENOSPC) ? -ENXIO : ret; } -- 2.31.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FA6DC48BE5 for ; Fri, 11 Jun 2021 14:55:21 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CE1BF613FA for ; Fri, 11 Jun 2021 14:55:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CE1BF613FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 29FF16EEA2; Fri, 11 Jun 2021 14:55:20 +0000 (UTC) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by gabe.freedesktop.org (Postfix) with ESMTPS id C8B146EE9E; Fri, 11 Jun 2021 14:55:18 +0000 (UTC) IronPort-SDR: DGCoJfru9O0uSyxnxYwhYxpcEEtOcbd/4tzC7lvySm+fxD/Pfi0qip2qyQnslLg0vSnKFkoSRg dZKohbyd3QvQ== X-IronPort-AV: E=McAfee;i="6200,9189,10012"; a="205355258" X-IronPort-AV: E=Sophos;i="5.83,265,1616482800"; d="scan'208";a="205355258" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jun 2021 07:55:18 -0700 IronPort-SDR: QSEBlbmSp4T3a5V/ljLLWHYPxBlY7AZIMiHvN2tCjuy++urUsiBOeqhkrMVhYfotMeUVghQiOq 6uCaZI0R6TxQ== X-IronPort-AV: E=Sophos;i="5.83,265,1616482800"; d="scan'208";a="450783217" Received: from delmer-mobl.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.23]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jun 2021 07:55:17 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Fri, 11 Jun 2021 16:54:57 +0200 Message-Id: <20210611145459.8576-3-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210611145459.8576-1-thomas.hellstrom@linux.intel.com> References: <20210611145459.8576-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 2/4] drm/i915/ttm: Adjust gem flags and caching settings after a move X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" QWZ0ZXIgYSBUVE0gbW92ZSB3ZSBuZWVkIHRvIHVwZGF0ZSB0aGUgaTkxNSBnZW0gZmxhZ3MgYW5k IGNhY2hpbmcKc2V0dGluZ3MgdG8gcmVmbGVjdCB0aGUgbmV3IHBsYWNlbWVudC4KQWxzbyBpbnRy b2R1Y2UgZ3B1X2JpbmRzX2lvbWVtKCkgYW5kIGNwdV9tYXBzX2lvbWVtKCkgdG8gY2xlYW4gdXAg dGhlCnZhcmlvdXMgd2F5cyB3ZSBwcmV2aW91c2x5IHVzZWQgdG8gZGV0ZWN0IHRoaXMuCkZpbmFs bHksIGluaXRpYWxpemUgdGhlIFRUTSBvYmplY3QgcmVzZXJ2ZWQgdG8gYmUgYWJsZSB0byB1cGRh dGUKZmxhZ3MgYW5kIGNhY2hpbmcgYmVmb3JlIGFueW9uZSBlbHNlIGdldHMgaG9sZCBvZiB0aGUg b2JqZWN0LgoKU2lnbmVkLW9mZi1ieTogVGhvbWFzIEhlbGxzdHLDtm0gPHRob21hcy5oZWxsc3Ry b21AbGludXguaW50ZWwuY29tPgotLS0KdjI6Ci0gU3R5bGUgZml4ZXMgKFJlcG9ydGVkIGJ5IE1h dHRoZXcgQXVsZCkKLS0tCiBkcml2ZXJzL2dwdS9kcm0vaTkxNS9nZW0vaTkxNV9nZW1fdHRtLmMg fCAxMTIgKysrKysrKysrKysrKysrKysrKy0tLS0tCiAxIGZpbGUgY2hhbmdlZCwgOTAgaW5zZXJ0 aW9ucygrKSwgMjIgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJtL2k5 MTUvZ2VtL2k5MTVfZ2VtX3R0bS5jIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ2VtL2k5MTVfZ2Vt X3R0bS5jCmluZGV4IDMzYWI0N2YxZTA1Yi4uNDVlZjFkMTAxOTM3IDEwMDY0NAotLS0gYS9kcml2 ZXJzL2dwdS9kcm0vaTkxNS9nZW0vaTkxNV9nZW1fdHRtLmMKKysrIGIvZHJpdmVycy9ncHUvZHJt L2k5MTUvZ2VtL2k5MTVfZ2VtX3R0bS5jCkBAIC03MCw2ICs3MCwxNyBAQCBzdGF0aWMgc3RydWN0 IHR0bV9wbGFjZW1lbnQgaTkxNV9zeXNfcGxhY2VtZW50ID0gewogCS5idXN5X3BsYWNlbWVudCA9 ICZsbWVtMF9zeXNfcGxhY2VtZW50X2ZsYWdzWzFdLAogfTsKIAorc3RhdGljIGJvb2wgZ3B1X2Jp bmRzX2lvbWVtKHN0cnVjdCB0dG1fcmVzb3VyY2UgKm1lbSkKK3sKKwlyZXR1cm4gbWVtLT5tZW1f dHlwZSAhPSBUVE1fUExfU1lTVEVNOworfQorCitzdGF0aWMgYm9vbCBjcHVfbWFwc19pb21lbShz dHJ1Y3QgdHRtX3Jlc291cmNlICptZW0pCit7CisJLyogT25jZSAvIGlmIHdlIHN1cHBvcnQgR0dU VCwgdGhpcyBpcyBhbHNvIGZhbHNlIGZvciBjYWNoZWQgdHRtX3R0cyAqLworCXJldHVybiBtZW0t Pm1lbV90eXBlICE9IFRUTV9QTF9TWVNURU07Cit9CisKIHN0YXRpYyB2b2lkIGk5MTVfdHRtX2Fk anVzdF9scnUoc3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKm9iaik7CiAKIHN0YXRpYyBzdHJ1 Y3QgdHRtX3R0ICppOTE1X3R0bV90dF9jcmVhdGUoc3RydWN0IHR0bV9idWZmZXJfb2JqZWN0ICpi bywKQEAgLTE3NSw2ICsxODYsNDEgQEAgc3RhdGljIHZvaWQgaTkxNV90dG1fZnJlZV9jYWNoZWRf aW9fc3Qoc3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKm9iaikKIAlvYmotPnR0bS5jYWNoZWRf aW9fc3QgPSBOVUxMOwogfQogCitzdGF0aWMgdm9pZAoraTkxNV90dG1fYWRqdXN0X2RvbWFpbnNf YWZ0ZXJfY3B1X21vdmUoc3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKm9iaikKK3sKKwlzdHJ1 Y3QgdHRtX2J1ZmZlcl9vYmplY3QgKmJvID0gaTkxNV9nZW1fdG9fdHRtKG9iaik7CisKKwlpZiAo Y3B1X21hcHNfaW9tZW0oYm8tPnJlc291cmNlKSB8fCBiby0+dHRtLT5jYWNoaW5nICE9IHR0bV9j YWNoZWQpIHsKKwkJb2JqLT53cml0ZV9kb21haW4gPSBJOTE1X0dFTV9ET01BSU5fV0M7CisJCW9i ai0+cmVhZF9kb21haW5zID0gSTkxNV9HRU1fRE9NQUlOX1dDOworCX0gZWxzZSB7CisJCW9iai0+ d3JpdGVfZG9tYWluID0gSTkxNV9HRU1fRE9NQUlOX0NQVTsKKwkJb2JqLT5yZWFkX2RvbWFpbnMg PSBJOTE1X0dFTV9ET01BSU5fQ1BVOworCX0KK30KKworc3RhdGljIHZvaWQgaTkxNV90dG1fYWRq dXN0X2dlbV9hZnRlcl9tb3ZlKHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpvYmopCit7CisJ c3RydWN0IGRybV9pOTE1X3ByaXZhdGUgKmk5MTUgPSB0b19pOTE1KG9iai0+YmFzZS5kZXYpOwor CXN0cnVjdCB0dG1fYnVmZmVyX29iamVjdCAqYm8gPSBpOTE1X2dlbV90b190dG0ob2JqKTsKKwl1 bnNpZ25lZCBpbnQgY2FjaGVfbGV2ZWw7CisKKwlvYmotPm1lbV9mbGFncyAmPSB+KEk5MTVfQk9f RkxBR19TVFJVQ1RfUEFHRSB8IEk5MTVfQk9fRkxBR19JT01FTSk7CisKKwlvYmotPm1lbV9mbGFn cyB8PSBjcHVfbWFwc19pb21lbShiby0+cmVzb3VyY2UpID8gSTkxNV9CT19GTEFHX0lPTUVNIDoK KwkJSTkxNV9CT19GTEFHX1NUUlVDVF9QQUdFOworCisJaWYgKChIQVNfTExDKGk5MTUpIHx8IEhB U19TTk9PUChpOTE1KSkgJiYgIWdwdV9iaW5kc19pb21lbShiby0+cmVzb3VyY2UpICYmCisJICAg IGJvLT50dG0tPmNhY2hpbmcgPT0gdHRtX2NhY2hlZCkgeworCQljYWNoZV9sZXZlbCA9IEk5MTVf Q0FDSEVfTExDOworCX0gZWxzZSB7CisJCWNhY2hlX2xldmVsID0gSTkxNV9DQUNIRV9OT05FOwor CX0KKworCWk5MTVfZ2VtX29iamVjdF9zZXRfY2FjaGVfY29oZXJlbmN5KG9iaiwgY2FjaGVfbGV2 ZWwpOworfQorCiBzdGF0aWMgdm9pZCBpOTE1X3R0bV9wdXJnZShzdHJ1Y3QgZHJtX2k5MTVfZ2Vt X29iamVjdCAqb2JqKQogewogCXN0cnVjdCB0dG1fYnVmZmVyX29iamVjdCAqYm8gPSBpOTE1X2dl bV90b190dG0ob2JqKTsKQEAgLTE5MCw4ICsyMzYsMTAgQEAgc3RhdGljIHZvaWQgaTkxNV90dG1f cHVyZ2Uoc3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKm9iaikKIAogCS8qIFRUTSdzIHB1cmdl IGludGVyZmFjZS4gTm90ZSB0aGF0IHdlIG1pZ2h0IGJlIHJlZW50ZXJpbmcuICovCiAJcmV0ID0g dHRtX2JvX3ZhbGlkYXRlKGJvLCAmcGxhY2UsICZjdHgpOwotCiAJaWYgKCFyZXQpIHsKKwkJb2Jq LT53cml0ZV9kb21haW4gPSAwOworCQlvYmotPnJlYWRfZG9tYWlucyA9IDA7CisJCWk5MTVfdHRt X2FkanVzdF9nZW1fYWZ0ZXJfbW92ZShvYmopOwogCQlpOTE1X3R0bV9mcmVlX2NhY2hlZF9pb19z dChvYmopOwogCQlvYmotPm1tLm1hZHYgPSBfX0k5MTVfTUFEVl9QVVJHRUQ7CiAJfQpAQCAtMjcz LDEyICszMjEsMTUgQEAgaTkxNV90dG1fcmVzb3VyY2VfZ2V0X3N0KHN0cnVjdCBkcm1faTkxNV9n ZW1fb2JqZWN0ICpvYmosCiAJCQkgc3RydWN0IHR0bV9yZXNvdXJjZSAqcmVzKQogewogCXN0cnVj dCB0dG1fYnVmZmVyX29iamVjdCAqYm8gPSBpOTE1X2dlbV90b190dG0ob2JqKTsKLQlzdHJ1Y3Qg dHRtX3Jlc291cmNlX21hbmFnZXIgKm1hbiA9Ci0JCXR0bV9tYW5hZ2VyX3R5cGUoYm8tPmJkZXYs IHJlcy0+bWVtX3R5cGUpOwogCi0JaWYgKG1hbi0+dXNlX3R0KQorCWlmICghZ3B1X2JpbmRzX2lv bWVtKHJlcykpCiAJCXJldHVybiBpOTE1X3R0bV90dF9nZXRfc3QoYm8tPnR0bSk7CiAKKwkvKgor CSAqIElmIENQVSBtYXBwaW5nIGRpZmZlcnMsIHdlIG5lZWQgdG8gYWRkIHRoZSB0dG1fdHQgcGFn ZXMgdG8KKwkgKiB0aGUgcmVzdWx0aW5nIHN0LiBNaWdodCBtYWtlIHNlbnNlIGZvciBHR1RULgor CSAqLworCUdFTV9XQVJOX09OKCFjcHVfbWFwc19pb21lbShyZXMpKTsKIAlyZXR1cm4gaW50ZWxf cmVnaW9uX3R0bV9ub2RlX3RvX3N0KG9iai0+bW0ucmVnaW9uLCByZXMpOwogfQogCkBAIC0yOTAs OCArMzQxLDYgQEAgc3RhdGljIGludCBpOTE1X3R0bV9tb3ZlKHN0cnVjdCB0dG1fYnVmZmVyX29i amVjdCAqYm8sIGJvb2wgZXZpY3QsCiAJc3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKm9iaiA9 IGk5MTVfdHRtX3RvX2dlbShibyk7CiAJc3RydWN0IHR0bV9yZXNvdXJjZV9tYW5hZ2VyICpkc3Rf bWFuID0KIAkJdHRtX21hbmFnZXJfdHlwZShiby0+YmRldiwgZHN0X21lbS0+bWVtX3R5cGUpOwot CXN0cnVjdCB0dG1fcmVzb3VyY2VfbWFuYWdlciAqc3JjX21hbiA9Ci0JCXR0bV9tYW5hZ2VyX3R5 cGUoYm8tPmJkZXYsIGJvLT5yZXNvdXJjZS0+bWVtX3R5cGUpOwogCXN0cnVjdCBpbnRlbF9tZW1v cnlfcmVnaW9uICpkc3RfcmVnLCAqc3JjX3JlZzsKIAl1bmlvbiB7CiAJCXN0cnVjdCB0dG1fa21h cF9pdGVyX3R0IHR0OwpAQCAtMzMyLDM0ICszODEsMzYgQEAgc3RhdGljIGludCBpOTE1X3R0bV9t b3ZlKHN0cnVjdCB0dG1fYnVmZmVyX29iamVjdCAqYm8sIGJvb2wgZXZpY3QsCiAJaWYgKElTX0VS Uihkc3Rfc3QpKQogCQlyZXR1cm4gUFRSX0VSUihkc3Rfc3QpOwogCi0JLyogSWYgd2Ugc3RhcnQg bWFwcGluZyBHR1RULCB3ZSBjYW4gbm8gbG9uZ2VyIHVzZSBtYW46OnVzZV90dCBoZXJlLiAqLwot CWRzdF9pdGVyID0gZHN0X21hbi0+dXNlX3R0ID8KKwlkc3RfaXRlciA9ICFjcHVfbWFwc19pb21l bShkc3RfbWVtKSA/CiAJCXR0bV9rbWFwX2l0ZXJfdHRfaW5pdCgmX2RzdF9pdGVyLnR0LCBiby0+ dHRtKSA6CiAJCXR0bV9rbWFwX2l0ZXJfaW9tYXBfaW5pdCgmX2RzdF9pdGVyLmlvLCAmZHN0X3Jl Zy0+aW9tYXAsCiAJCQkJCSBkc3Rfc3QsIGRzdF9yZWctPnJlZ2lvbi5zdGFydCk7CiAKLQlzcmNf aXRlciA9IHNyY19tYW4tPnVzZV90dCA/CisJc3JjX2l0ZXIgPSAhY3B1X21hcHNfaW9tZW0oYm8t PnJlc291cmNlKSA/CiAJCXR0bV9rbWFwX2l0ZXJfdHRfaW5pdCgmX3NyY19pdGVyLnR0LCBiby0+ dHRtKSA6CiAJCXR0bV9rbWFwX2l0ZXJfaW9tYXBfaW5pdCgmX3NyY19pdGVyLmlvLCAmc3JjX3Jl Zy0+aW9tYXAsCiAJCQkJCSBvYmotPnR0bS5jYWNoZWRfaW9fc3QsCiAJCQkJCSBzcmNfcmVnLT5y ZWdpb24uc3RhcnQpOwogCiAJdHRtX21vdmVfbWVtY3B5KGJvLCBkc3RfbWVtLT5udW1fcGFnZXMs IGRzdF9pdGVyLCBzcmNfaXRlcik7CisJLyogQmVsb3cgZHN0X21lbSBiZWNvbWVzIGJvLT5yZXNv dXJjZS4gKi8KIAl0dG1fYm9fbW92ZV9zeW5jX2NsZWFudXAoYm8sIGRzdF9tZW0pOworCWk5MTVf dHRtX2FkanVzdF9kb21haW5zX2FmdGVyX2NwdV9tb3ZlKG9iaik7CiAJaTkxNV90dG1fZnJlZV9j YWNoZWRfaW9fc3Qob2JqKTsKIAotCWlmICghZHN0X21hbi0+dXNlX3R0KSB7CisJaWYgKGdwdV9i aW5kc19pb21lbShkc3RfbWVtKSB8fCBjcHVfbWFwc19pb21lbShkc3RfbWVtKSkgewogCQlvYmot PnR0bS5jYWNoZWRfaW9fc3QgPSBkc3Rfc3Q7CiAJCW9iai0+dHRtLmdldF9pb19wYWdlLnNnX3Bv cyA9IGRzdF9zdC0+c2dsOwogCQlvYmotPnR0bS5nZXRfaW9fcGFnZS5zZ19pZHggPSAwOwogCX0K IAorCWk5MTVfdHRtX2FkanVzdF9nZW1fYWZ0ZXJfbW92ZShvYmopOwogCXJldHVybiAwOwogfQog CiBzdGF0aWMgaW50IGk5MTVfdHRtX2lvX21lbV9yZXNlcnZlKHN0cnVjdCB0dG1fZGV2aWNlICpi ZGV2LCBzdHJ1Y3QgdHRtX3Jlc291cmNlICptZW0pCiB7Ci0JaWYgKG1lbS0+bWVtX3R5cGUgPCBJ OTE1X1BMX0xNRU0wKQorCWlmICghY3B1X21hcHNfaW9tZW0obWVtKSkKIAkJcmV0dXJuIDA7CiAK IAltZW0tPmJ1cy5jYWNoaW5nID0gdHRtX3dyaXRlX2NvbWJpbmVkOwpAQCAtNDIxLDYgKzQ3Miwx NiBAQCBzdGF0aWMgaW50IGk5MTVfdHRtX2dldF9wYWdlcyhzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29i amVjdCAqb2JqKQogCWlmIChyZXQpCiAJCXJldHVybiByZXQgPT0gLUVOT1NQQyA/IC1FTlhJTyA6 IHJldDsKIAorCWk5MTVfdHRtX2FkanVzdF9scnUob2JqKTsKKwlpZiAoYm8tPnR0bSAmJiAhdHRt X3R0X2lzX3BvcHVsYXRlZChiby0+dHRtKSkgeworCQlyZXQgPSB0dG1fdHRfcG9wdWxhdGUoYm8t PmJkZXYsIGJvLT50dG0sICZjdHgpOworCQlpZiAocmV0KQorCQkJcmV0dXJuIHJldDsKKworCQlp OTE1X3R0bV9hZGp1c3RfZG9tYWluc19hZnRlcl9jcHVfbW92ZShvYmopOworCQlpOTE1X3R0bV9h ZGp1c3RfZ2VtX2FmdGVyX21vdmUob2JqKTsKKwl9CisKIAkvKiBPYmplY3QgZWl0aGVyIGhhcyBh IHBhZ2UgdmVjdG9yIG9yIGlzIGFuIGlvbWVtIG9iamVjdCAqLwogCXN0ID0gYm8tPnR0bSA/IGk5 MTVfdHRtX3R0X2dldF9zdChiby0+dHRtKSA6IG9iai0+dHRtLmNhY2hlZF9pb19zdDsKIAlpZiAo SVNfRVJSKHN0KSkKQEAgLTQyOCw4ICs0ODksNiBAQCBzdGF0aWMgaW50IGk5MTVfdHRtX2dldF9w YWdlcyhzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29iamVjdCAqb2JqKQogCiAJX19pOTE1X2dlbV9vYmpl Y3Rfc2V0X3BhZ2VzKG9iaiwgc3QsIGk5MTVfc2dfZG1hX3NpemVzKHN0LT5zZ2wpKTsKIAotCWk5 MTVfdHRtX2FkanVzdF9scnUob2JqKTsKLQogCXJldHVybiByZXQ7CiB9CiAKQEAgLTU2Myw2ICs2 MjIsNyBAQCBzdGF0aWMgdTY0IGk5MTVfdHRtX21tYXBfb2Zmc2V0KHN0cnVjdCBkcm1faTkxNV9n ZW1fb2JqZWN0ICpvYmopCiAKIGNvbnN0IHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0X29wcyBp OTE1X2dlbV90dG1fb2JqX29wcyA9IHsKIAkubmFtZSA9ICJpOTE1X2dlbV9vYmplY3RfdHRtIiwK KwkuZmxhZ3MgPSBJOTE1X0dFTV9PQkpFQ1RfSVNfU0hSSU5LQUJMRSwKIAogCS5nZXRfcGFnZXMg PSBpOTE1X3R0bV9nZXRfcGFnZXMsCiAJLnB1dF9wYWdlcyA9IGk5MTVfdHRtX3B1dF9wYWdlcywK QEAgLTU5OSw2ICs2NTksMTAgQEAgaW50IF9faTkxNV9nZW1fdHRtX29iamVjdF9pbml0KHN0cnVj dCBpbnRlbF9tZW1vcnlfcmVnaW9uICptZW0sCiB7CiAJc3RhdGljIHN0cnVjdCBsb2NrX2NsYXNz X2tleSBsb2NrX2NsYXNzOwogCXN0cnVjdCBkcm1faTkxNV9wcml2YXRlICppOTE1ID0gbWVtLT5p OTE1OworCXN0cnVjdCB0dG1fb3BlcmF0aW9uX2N0eCBjdHggPSB7CisJCS5pbnRlcnJ1cHRpYmxl ID0gdHJ1ZSwKKwkJLm5vX3dhaXRfZ3B1ID0gZmFsc2UsCisJfTsKIAllbnVtIHR0bV9ib190eXBl IGJvX3R5cGU7CiAJc2l6ZV90IGFsaWdubWVudCA9IDA7CiAJaW50IHJldDsKQEAgLTYxOCwxNSAr NjgyLDE0IEBAIGludCBfX2k5MTVfZ2VtX3R0bV9vYmplY3RfaW5pdChzdHJ1Y3QgaW50ZWxfbWVt b3J5X3JlZ2lvbiAqbWVtLAogCWk5MTVfZ2VtX29iamVjdF9pbml0KG9iaiwgJmk5MTVfZ2VtX3R0 bV9vYmpfb3BzLCAmbG9ja19jbGFzcywgZmxhZ3MpOwogCWk5MTVfZ2VtX29iamVjdF9pbml0X21l bW9yeV9yZWdpb24ob2JqLCBtZW0pOwogCWk5MTVfZ2VtX29iamVjdF9tYWtlX3Vuc2hyaW5rYWJs ZShvYmopOwotCW9iai0+cmVhZF9kb21haW5zID0gSTkxNV9HRU1fRE9NQUlOX1dDIHwgSTkxNV9H RU1fRE9NQUlOX0dUVDsKLQlvYmotPm1lbV9mbGFncyB8PSBJOTE1X0JPX0ZMQUdfSU9NRU07Ci0J aTkxNV9nZW1fb2JqZWN0X3NldF9jYWNoZV9jb2hlcmVuY3kob2JqLCBJOTE1X0NBQ0hFX05PTkUp OwogCUlOSVRfUkFESVhfVFJFRSgmb2JqLT50dG0uZ2V0X2lvX3BhZ2UucmFkaXgsIEdGUF9LRVJO RUwgfCBfX0dGUF9OT1dBUk4pOwogCW11dGV4X2luaXQoJm9iai0+dHRtLmdldF9pb19wYWdlLmxv Y2spOwogCiAJYm9fdHlwZSA9IChvYmotPmZsYWdzICYgSTkxNV9CT19BTExPQ19VU0VSKSA/IHR0 bV9ib190eXBlX2RldmljZSA6CiAJCXR0bV9ib190eXBlX2tlcm5lbDsKIAorCW9iai0+YmFzZS52 bWFfbm9kZS5kcml2ZXJfcHJpdmF0ZSA9IGk5MTVfZ2VtX3RvX3R0bShvYmopOworCiAJLyoKIAkg KiBJZiB0aGlzIGZ1bmN0aW9uIGZhaWxzLCBpdCB3aWxsIGNhbGwgdGhlIGRlc3RydWN0b3IsIGJ1 dAogCSAqIG91ciBjYWxsZXIgc3RpbGwgb3ducyB0aGUgb2JqZWN0LiBTbyBubyBmcmVlaW5nIGlu IHRoZQpAQCAtNjM0LDE0ICs2OTcsMTkgQEAgaW50IF9faTkxNV9nZW1fdHRtX29iamVjdF9pbml0 KHN0cnVjdCBpbnRlbF9tZW1vcnlfcmVnaW9uICptZW0sCiAJICogU2ltaWxhcmx5LCBpbiBkZWxh eWVkX2Rlc3Ryb3ksIHdlIGNhbid0IGNhbGwgdHRtX2JvX3B1dCgpCiAJICogdW50aWwgc3VjY2Vz c2Z1bCBpbml0aWFsaXphdGlvbi4KIAkgKi8KLQlvYmotPmJhc2Uudm1hX25vZGUuZHJpdmVyX3By aXZhdGUgPSBpOTE1X2dlbV90b190dG0ob2JqKTsKLQlyZXQgPSB0dG1fYm9faW5pdCgmaTkxNS0+ YmRldiwgaTkxNV9nZW1fdG9fdHRtKG9iaiksIHNpemUsCi0JCQkgIGJvX3R5cGUsICZpOTE1X3N5 c19wbGFjZW1lbnQsIGFsaWdubWVudCwKLQkJCSAgdHJ1ZSwgTlVMTCwgTlVMTCwgaTkxNV90dG1f Ym9fZGVzdHJveSk7CisJcmV0ID0gdHRtX2JvX2luaXRfcmVzZXJ2ZWQoJmk5MTUtPmJkZXYsIGk5 MTVfZ2VtX3RvX3R0bShvYmopLCBzaXplLAorCQkJCSAgIGJvX3R5cGUsICZpOTE1X3N5c19wbGFj ZW1lbnQsIGFsaWdubWVudCwKKwkJCQkgICAmY3R4LCBOVUxMLCBOVUxMLCBpOTE1X3R0bV9ib19k ZXN0cm95KTsKKworCWlmIChyZXQpCisJCWdvdG8gb3V0OwogCi0JaWYgKCFyZXQpCi0JCW9iai0+ dHRtLmNyZWF0ZWQgPSB0cnVlOworCW9iai0+dHRtLmNyZWF0ZWQgPSB0cnVlOworCWk5MTVfdHRt X2FkanVzdF9kb21haW5zX2FmdGVyX2NwdV9tb3ZlKG9iaik7CisJaTkxNV90dG1fYWRqdXN0X2dl bV9hZnRlcl9tb3ZlKG9iaik7CisJaTkxNV9nZW1fb2JqZWN0X3VubG9jayhvYmopOwogCitvdXQ6 CiAJLyogaTkxNSB3YW50cyAtRU5YSU8gd2hlbiBvdXQgb2YgbWVtb3J5IHJlZ2lvbiBzcGFjZS4g Ki8KIAlyZXR1cm4gKHJldCA9PSAtRU5PU1BDKSA/IC1FTlhJTyA6IHJldDsKIH0KLS0gCjIuMzEu MQoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KSW50ZWwt Z2Z4IG1haWxpbmcgbGlzdApJbnRlbC1nZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8v bGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtZ2Z4Cg==