From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBDD3C43603 for ; Thu, 20 May 2021 15:10:48 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7AA60611ED for ; Thu, 20 May 2021 15:10:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7AA60611ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 677C36F47B; Thu, 20 May 2021 15:10:41 +0000 (UTC) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by gabe.freedesktop.org (Postfix) with ESMTPS id 147296E1BE; Thu, 20 May 2021 15:10:38 +0000 (UTC) IronPort-SDR: mbX+d8G7gik3ZjQnTm7UeDCooH+Gy2d5oYicvKUlXFJOnJOUnTUmIEz14uy+OMX5R/iOSPfxew /OTjv5tV6SfA== X-IronPort-AV: E=McAfee;i="6200,9189,9989"; a="222341180" X-IronPort-AV: E=Sophos;i="5.82,313,1613462400"; d="scan'208";a="222341180" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 May 2021 08:10:00 -0700 IronPort-SDR: IOVnqwaDlb4a9XEeMYvp7bbL/kLlusFzRrnLZ7UePtd+DiS9Kd65Vs50uYbGQU8l+Ko183Z1xz TmRg/XvCANDQ== X-IronPort-AV: E=Sophos;i="5.82,313,1613462400"; d="scan'208";a="395728149" Received: from cbjoerns-mobl1.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.247]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 May 2021 08:09:58 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [RFC PATCH 1/5] drm/ttm: Add a generic TTM memcpy move for page-based iomem Date: Thu, 20 May 2021 17:09:43 +0200 Message-Id: <20210520150947.803891-2-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210520150947.803891-1-thomas.hellstrom@linux.intel.com> References: <20210520150947.803891-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , =?UTF-8?q?Christian=20K=C3=B6nig?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The internal ttm_bo_util memcpy uses ioremap functionality, and while it probably might be possible to use it for copying in- and out of sglist represented io memory, using io_mem_reserve() / io_mem_free() callbacks, that would cause problems with fault(). Instead, implement a method mapping page-by-page using kmap_local() semantics. As an additional benefit we then avoid the occasional global TLB flushes of ioremap() and consuming ioremap space, elimination of a critical point of failure and with a slight change of semantics we could also push the memcpy out async for testing and async driver development purposes. A special linear iomem iterator is introduced internally to mimic the old ioremap behaviour for code-paths that can't immediately be ported over. This adds to the code size and should be considered a temporary solution. Looking at the code we have a lot of checks for iomap tagged pointers. Ideally we should extend the core memremap functions to also accept uncached memory and kmap_local functionality. Then we could strip a lot of code. Cc: Christian König Signed-off-by: Thomas Hellström --- drivers/gpu/drm/ttm/ttm_bo_util.c | 468 ++++++++++++++++++++---------- include/drm/ttm/ttm_bo_driver.h | 94 ++++++ 2 files changed, 407 insertions(+), 155 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c index ae8b61460724..bad9b16e96ba 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_util.c +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c @@ -35,11 +35,13 @@ #include #include #include +#include #include #include #include #include #include +#include struct ttm_transfer_obj { struct ttm_buffer_object base; @@ -72,190 +74,366 @@ void ttm_mem_io_free(struct ttm_device *bdev, mem->bus.addr = NULL; } -static int ttm_resource_ioremap(struct ttm_device *bdev, - struct ttm_resource *mem, - void **virtual) +static pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp) { - int ret; - void *addr; + /* Cached mappings need no adjustment */ + if (caching == ttm_cached) + return tmp; - *virtual = NULL; - ret = ttm_mem_io_reserve(bdev, mem); - if (ret || !mem->bus.is_iomem) - return ret; +#if defined(__i386__) || defined(__x86_64__) + if (caching == ttm_write_combined) + tmp = pgprot_writecombine(tmp); + else if (boot_cpu_data.x86 > 3) + tmp = pgprot_noncached(tmp); +#endif +#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \ + defined(__powerpc__) || defined(__mips__) + if (caching == ttm_write_combined) + tmp = pgprot_writecombine(tmp); + else + tmp = pgprot_noncached(tmp); +#endif +#if defined(__sparc__) + tmp = pgprot_noncached(tmp); +#endif + return tmp; +} - if (mem->bus.addr) { - addr = mem->bus.addr; - } else { - size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT; +static void ttm_kmap_iter_tt_kmap_local(struct ttm_kmap_iter *iter, + struct dma_buf_map *dmap, + pgoff_t i) +{ + struct ttm_kmap_iter_tt *iter_tt = + container_of(iter, typeof(*iter_tt), base); - if (mem->bus.caching == ttm_write_combined) - addr = ioremap_wc(mem->bus.offset, bus_size); -#ifdef CONFIG_X86 - else if (mem->bus.caching == ttm_cached) - addr = ioremap_cache(mem->bus.offset, bus_size); -#endif - else - addr = ioremap(mem->bus.offset, bus_size); - if (!addr) { - ttm_mem_io_free(bdev, mem); - return -ENOMEM; - } + dma_buf_map_set_vaddr(dmap, kmap_local_page_prot(iter_tt->tt->pages[i], + iter_tt->prot)); +} + +static void ttm_kmap_iter_iomap_kmap_local(struct ttm_kmap_iter *iter, + struct dma_buf_map *dmap, + pgoff_t i) +{ + struct ttm_kmap_iter_iomap *iter_io = + container_of(iter, typeof(*iter_io), base); + void __iomem *addr; + +retry: + while (i >= iter_io->cache.end) { + iter_io->cache.sg = iter_io->cache.sg ? + sg_next(iter_io->cache.sg) : iter_io->st->sgl; + iter_io->cache.i = iter_io->cache.end; + iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >> + PAGE_SHIFT; + iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) - + iter_io->start; } - *virtual = addr; - return 0; + + if (i < iter_io->cache.i) { + iter_io->cache.end = 0; + iter_io->cache.sg = NULL; + goto retry; + } + + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs + + (((resource_size_t)i - iter_io->cache.i) + << PAGE_SHIFT)); + dma_buf_map_set_vaddr_iomem(dmap, addr); } -static void ttm_resource_iounmap(struct ttm_device *bdev, - struct ttm_resource *mem, - void *virtual) +static const struct ttm_kmap_iter_ops ttm_kmap_iter_tt_ops = { + .kmap_local = ttm_kmap_iter_tt_kmap_local, + .needs_unmap = true +}; + +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = { + .kmap_local = ttm_kmap_iter_iomap_kmap_local, + .needs_unmap = true +}; + +/* If needed, make unmap functionality part of ttm_kmap_iter_ops */ +static void kunmap_local_iter(struct ttm_kmap_iter *iter, + struct dma_buf_map *map) { - if (virtual && mem->bus.addr == NULL) - iounmap(virtual); - ttm_mem_io_free(bdev, mem); + if (!iter->ops->needs_unmap) + return; + + if (map->is_iomem) + io_mapping_unmap_local(map->vaddr_iomem); + else + kunmap_local(map->vaddr); } -static int ttm_copy_io_page(void *dst, void *src, unsigned long page) +/** + * ttm_move_memcpy - Helper to perform a memcpy ttm move operation. + * @bo: The struct ttm_buffer_object. + * @new_mem: The struct ttm_resource we're moving to (copy destination). + * @new_iter: A struct ttm_kmap_iter representing the destination resource. + * @old_iter: A struct ttm_kmap_iter representing the source resource. + * + * This function is intended to be able to move out async under a + * dma-fence if desired. + */ +void ttm_move_memcpy(struct ttm_buffer_object *bo, + struct ttm_resource *new_mem, + struct ttm_kmap_iter *new_iter, + struct ttm_kmap_iter *old_iter) { - uint32_t *dstP = - (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT)); - uint32_t *srcP = - (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT)); - - int i; - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i) - iowrite32(ioread32(srcP++), dstP++); - return 0; + struct ttm_device *bdev = bo->bdev; + struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type); + struct ttm_tt *ttm = bo->ttm; + struct ttm_resource *old_mem = &bo->mem; + struct ttm_resource_manager *old_man = ttm_manager_type(bdev, old_mem->mem_type); + struct dma_buf_map old_map, new_map; + pgoff_t i; + + /* Single TTM move. NOP */ + if (old_man->use_tt && man->use_tt) + return; + + /* Don't move nonexistent data. Clear destination instead. */ + if (old_man->use_tt && !man->use_tt && + (!ttm || !ttm_tt_is_populated(ttm))) { + if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC)) + return; + + for (i = 0; i < new_mem->num_pages; ++i) { + new_iter->ops->kmap_local(new_iter, &new_map, i); + if (new_map.is_iomem) + memset_io(new_map.vaddr_iomem, 0, PAGE_SIZE); + else + memset(new_map.vaddr, 0, PAGE_SIZE); + kunmap_local_iter(new_iter, &new_map); + } + return; + } + + for (i = 0; i < new_mem->num_pages; ++i) { + new_iter->ops->kmap_local(new_iter, &new_map, i); + old_iter->ops->kmap_local(old_iter, &old_map, i); + + if (!old_map.is_iomem && !new_map.is_iomem) { + memcpy(new_map.vaddr, old_map.vaddr, PAGE_SIZE); + } else if (!old_map.is_iomem) { + dma_buf_map_memcpy_to(&new_map, old_map.vaddr, + PAGE_SIZE); + } else if (!new_map.is_iomem) { + memcpy_fromio(new_map.vaddr, old_map.vaddr_iomem, + PAGE_SIZE); + } else { + int j; + u32 __iomem *src = old_map.vaddr_iomem; + u32 __iomem *dst = new_map.vaddr_iomem; + + for (j = 0; j < (PAGE_SIZE >> 2); ++j) + iowrite32(ioread32(src++), dst++); + } + kunmap_local_iter(old_iter, &old_map); + kunmap_local_iter(new_iter, &new_map); + } } +EXPORT_SYMBOL(ttm_move_memcpy); -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src, - unsigned long page, - pgprot_t prot) +/** + * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap + * @iter_io: The struct ttm_kmap_iter_iomap to initialize. + * @iomap: The struct io_mapping representing the underlying linear io_memory. + * @st: sg_table into @iomap, representing the memory of the struct + * ttm_resource. + * @start: Offset that needs to be subtracted from @st to make + * sg_dma_address(st->sgl) - @start == 0 for @iomap start. + * + * Return: Pointer to the embedded struct ttm_kmap_iter. + */ +struct ttm_kmap_iter * +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io, + struct io_mapping *iomap, + struct sg_table *st, + resource_size_t start) { - struct page *d = ttm->pages[page]; - void *dst; + iter_io->base.ops = &ttm_kmap_iter_io_ops; + iter_io->iomap = iomap; + iter_io->st = st; + iter_io->start = start; + memset(&iter_io->cache, 0, sizeof(iter_io->cache)); - if (!d) - return -ENOMEM; + return &iter_io->base; +} +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init); - src = (void *)((unsigned long)src + (page << PAGE_SHIFT)); - dst = kmap_atomic_prot(d, prot); - if (!dst) - return -ENOMEM; +/** + * ttm_kmap_iter_tt_init - Initialize a struct ttm_kmap_iter_tt + * @iter_tt: The struct ttm_kmap_iter_tt to initialize. + * @tt: Struct ttm_tt holding page pointers of the struct ttm_resource. + * + * Return: Pointer to the embedded struct ttm_kmap_iter. + */ +struct ttm_kmap_iter * +ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt, + struct ttm_tt *tt) +{ + iter_tt->base.ops = &ttm_kmap_iter_tt_ops; + iter_tt->tt = tt; + iter_tt->prot = ttm_prot_from_caching(tt->caching, PAGE_KERNEL); - memcpy_fromio(dst, src, PAGE_SIZE); + return &iter_tt->base; +} +EXPORT_SYMBOL(ttm_kmap_iter_tt_init); - kunmap_atomic(dst); +/** + * DOC: Linear io iterator + * + * This code should die in the not too near future. Best would be if we could + * make io-mapping use memremap for all io memory, and have memremap + * implement a kmap_local functionality. We could then strip a huge amount of + * code. These linear io iterators are implemented to mimic old functionality, + * and they don't use kmap_local semantics at all internally. Rather ioremap or + * friends, and at least on 32-bit they add global TLB flushes and points + * of failure. + */ - return 0; +/** + * struct ttm_kmap_iter_linear_io - Iterator specialization for linear io + * @base: The base iterator + * @dmap: Points to the starting address of the region + * @needs_unmap: Whether we need to unmap on fini + */ +struct ttm_kmap_iter_linear_io { + struct ttm_kmap_iter base; + struct dma_buf_map dmap; + bool needs_unmap; +}; + +static void ttm_kmap_iter_linear_io_kmap_local(struct ttm_kmap_iter *iter, + struct dma_buf_map *dmap, + pgoff_t i) +{ + struct ttm_kmap_iter_linear_io *iter_io = + container_of(iter, typeof(*iter_io), base); + + *dmap = iter_io->dmap; + dma_buf_map_incr(dmap, i * PAGE_SIZE); } -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst, - unsigned long page, - pgprot_t prot) +static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = { + .kmap_local = ttm_kmap_iter_linear_io_kmap_local +}; + +static struct ttm_kmap_iter * +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io, + struct ttm_device *bdev, + struct ttm_resource *mem) { - struct page *s = ttm->pages[page]; - void *src; + int ret; - if (!s) - return -ENOMEM; + ret = ttm_mem_io_reserve(bdev, mem); + if (ret) + goto out_err; + if (!mem->bus.is_iomem) { + ret = -EINVAL; + goto out_io_free; + } - dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT)); - src = kmap_atomic_prot(s, prot); - if (!src) - return -ENOMEM; + if (mem->bus.addr) { + dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr); + iter_io->needs_unmap = false; + } else { + size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT; - memcpy_toio(dst, src, PAGE_SIZE); + iter_io->needs_unmap = true; + if (mem->bus.caching == ttm_write_combined) + dma_buf_map_set_vaddr_iomem(&iter_io->dmap, + ioremap_wc(mem->bus.offset, + bus_size)); + else if (mem->bus.caching == ttm_cached) + dma_buf_map_set_vaddr(&iter_io->dmap, + memremap(mem->bus.offset, bus_size, + MEMREMAP_WB)); + else + dma_buf_map_set_vaddr_iomem(&iter_io->dmap, + ioremap(mem->bus.offset, + bus_size)); + if (dma_buf_map_is_null(&iter_io->dmap)) { + ret = -ENOMEM; + goto out_io_free; + } + } - kunmap_atomic(src); + iter_io->base.ops = &ttm_kmap_iter_linear_io_ops; + return &iter_io->base; - return 0; +out_io_free: + ttm_mem_io_free(bdev, mem); +out_err: + return ERR_PTR(ret); +} + +static void +ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io, + struct ttm_device *bdev, + struct ttm_resource *mem) +{ + if (iter_io->needs_unmap && dma_buf_map_is_set(&iter_io->dmap)) { + if (iter_io->dmap.is_iomem) + iounmap(iter_io->dmap.vaddr_iomem); + else + memunmap(iter_io->dmap.vaddr); + } + + ttm_mem_io_free(bdev, mem); } +static int ttm_bo_wait_free_node(struct ttm_buffer_object *bo, + bool dst_use_tt); + int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, struct ttm_resource *new_mem) { struct ttm_device *bdev = bo->bdev; struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type); + struct ttm_resource_manager *new_man = + ttm_manager_type(bo->bdev, new_mem->mem_type); struct ttm_tt *ttm = bo->ttm; struct ttm_resource *old_mem = &bo->mem; struct ttm_resource old_copy = *old_mem; - void *old_iomap; - void *new_iomap; + union { + struct ttm_kmap_iter_tt tt; + struct ttm_kmap_iter_linear_io io; + } _new_iter, _old_iter; + struct ttm_kmap_iter *new_iter, *old_iter; int ret; - unsigned long i; - - ret = ttm_bo_wait_ctx(bo, ctx); - if (ret) - return ret; - - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap); - if (ret) - return ret; - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap); - if (ret) - goto out; - - /* - * Single TTM move. NOP. - */ - if (old_iomap == NULL && new_iomap == NULL) - goto out2; - /* - * Don't move nonexistent data. Clear destination instead. - */ - if (old_iomap == NULL && - (ttm == NULL || (!ttm_tt_is_populated(ttm) && - !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) { - memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE); - goto out2; - } - - /* - * TTM might be null for moves within the same region. - */ if (ttm) { ret = ttm_tt_populate(bdev, ttm, ctx); if (ret) - goto out1; + return ret; } - for (i = 0; i < new_mem->num_pages; ++i) { - if (old_iomap == NULL) { - pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL); - ret = ttm_copy_ttm_io_page(ttm, new_iomap, i, - prot); - } else if (new_iomap == NULL) { - pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL); - ret = ttm_copy_io_ttm_page(ttm, old_iomap, i, - prot); - } else { - ret = ttm_copy_io_page(new_iomap, old_iomap, i); - } - if (ret) - goto out1; + new_iter = new_man->use_tt ? + ttm_kmap_iter_tt_init(&_new_iter.tt, bo->ttm) : + ttm_kmap_iter_linear_io_init(&_new_iter.io, bdev, new_mem); + if (IS_ERR(new_iter)) + return PTR_ERR(new_iter); + + old_iter = man->use_tt ? + ttm_kmap_iter_tt_init(&_old_iter.tt, bo->ttm) : + ttm_kmap_iter_linear_io_init(&_old_iter.io, bdev, old_mem); + if (IS_ERR(old_iter)) { + ret = PTR_ERR(old_iter); + goto out_old_iter; } - mb(); -out2: - old_copy = *old_mem; - ttm_bo_assign_mem(bo, new_mem); + ttm_move_memcpy(bo, new_mem, new_iter, old_iter); + old_copy = *old_mem; + ret = ttm_bo_wait_free_node(bo, new_man->use_tt); if (!man->use_tt) - ttm_bo_tt_destroy(bo); + ttm_kmap_iter_linear_io_fini(&_old_iter.io, bdev, &old_copy); +out_old_iter: + if (!new_man->use_tt) + ttm_kmap_iter_linear_io_fini(&_new_iter.io, bdev, new_mem); -out1: - ttm_resource_iounmap(bdev, old_mem, new_iomap); -out: - ttm_resource_iounmap(bdev, &old_copy, old_iomap); - - /* - * On error, keep the mm node! - */ - if (!ret) - ttm_resource_free(bo, &old_copy); return ret; } EXPORT_SYMBOL(ttm_bo_move_memcpy); @@ -336,27 +514,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res, man = ttm_manager_type(bo->bdev, res->mem_type); caching = man->use_tt ? bo->ttm->caching : res->bus.caching; - /* Cached mappings need no adjustment */ - if (caching == ttm_cached) - return tmp; - -#if defined(__i386__) || defined(__x86_64__) - if (caching == ttm_write_combined) - tmp = pgprot_writecombine(tmp); - else if (boot_cpu_data.x86 > 3) - tmp = pgprot_noncached(tmp); -#endif -#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \ - defined(__powerpc__) || defined(__mips__) - if (caching == ttm_write_combined) - tmp = pgprot_writecombine(tmp); - else - tmp = pgprot_noncached(tmp); -#endif -#if defined(__sparc__) - tmp = pgprot_noncached(tmp); -#endif - return tmp; + return ttm_prot_from_caching(caching, tmp); } EXPORT_SYMBOL(ttm_io_prot); diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h index dbccac957f8f..30f1dce987b2 100644 --- a/include/drm/ttm/ttm_bo_driver.h +++ b/include/drm/ttm/ttm_bo_driver.h @@ -332,4 +332,98 @@ int ttm_range_man_init(struct ttm_device *bdev, int ttm_range_man_fini(struct ttm_device *bdev, unsigned type); +struct dma_buf_map; +struct io_mapping; +struct sg_table; +struct scatterlist; +struct ttm_kmap_iter; + +/** + * struct ttm_kmap_iter_ops - Ops structure for a struct + * ttm_kmap_iter. + * @needs_unmap - Whether a kunmap_local is needed to balance @kmap_local. + */ +struct ttm_kmap_iter_ops { + /** + * kmap_local - Map a PAGE_SIZE part of the resource using + * kmap_local semantics. + * @res_kmap: Pointer to the struct ttm_kmap_iter representing + * the resource. + * @dmap: The struct dma_buf_map holding the virtual address after + * the operation. + * @i: The location within the resource to map. PAGE_SIZE granularity. + */ + void (*kmap_local)(struct ttm_kmap_iter *res_kmap, + struct dma_buf_map *dmap, pgoff_t i); + bool needs_unmap; +}; + +/** + * struct ttm_kmap_iter - Iterator for kmap_local type operations on a + * resource. + * @ops: Pointer to the operations struct. + * + * This struct is intended to be embedded in a resource-specific specialization + * implementing operations for the resource. + * + * Nothing stops us from extending the operations to vmap, vmap_pfn etc, + * replacing some or parts of the ttm_bo_util. cpu-map functionality. + */ +struct ttm_kmap_iter { + const struct ttm_kmap_iter_ops *ops; +}; + +/** + * struct ttm_kmap_iter_tt - Specialization for a tt (page) backed struct + * ttm_resource. + * @base: Embedded struct ttm_kmap_iter providing the usage interface + * @tt: Cached struct ttm_tt. + */ +struct ttm_kmap_iter_tt { + struct ttm_kmap_iter base; + struct ttm_tt *tt; + pgprot_t prot; +}; + +/** + * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping + + * struct sg_table backed struct ttm_resource. + * @base: Embedded struct ttm_kmap_iter providing the usage interface. + * @iomap: struct io_mapping representing the underlying linear io_memory. + * @st: sg_table into @iomap, representing the memory of the struct ttm_resource. + * @start: Offset that needs to be subtracted from @st to make + * sg_dma_address(st->sgl) - @start == 0 for @iomap start. + * @cache: Scatterlist traversal cache for fast lookups. + * @cache.sg: Pointer to the currently cached scatterlist segment. + * @cache.i: First index of @sg. PAGE_SIZE granularity. + * @cache.end: Last index + 1 of @sg. PAGE_SIZE granularity. + * @cache.offs: First offset into @iomap of @sg. PAGE_SIZE granularity. + */ +struct ttm_kmap_iter_iomap { + struct ttm_kmap_iter base; + struct io_mapping *iomap; + struct sg_table *st; + resource_size_t start; + struct { + struct scatterlist *sg; + pgoff_t i; + pgoff_t end; + pgoff_t offs; + } cache; +}; + +void ttm_move_memcpy(struct ttm_buffer_object *bo, + struct ttm_resource *new_mem, + struct ttm_kmap_iter *new_iter, + struct ttm_kmap_iter *old_iter); + +struct ttm_kmap_iter * +ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt, + struct ttm_tt *tt); + +struct ttm_kmap_iter * +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io, + struct io_mapping *iomap, + struct sg_table *st, + resource_size_t start); #endif -- 2.31.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A6B6C43462 for ; Thu, 20 May 2021 15:10:47 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5AB0D60C3D for ; Thu, 20 May 2021 15:10:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5AB0D60C3D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3676F6F47A; Thu, 20 May 2021 15:10:40 +0000 (UTC) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by gabe.freedesktop.org (Postfix) with ESMTPS id 147296E1BE; Thu, 20 May 2021 15:10:38 +0000 (UTC) IronPort-SDR: mbX+d8G7gik3ZjQnTm7UeDCooH+Gy2d5oYicvKUlXFJOnJOUnTUmIEz14uy+OMX5R/iOSPfxew /OTjv5tV6SfA== X-IronPort-AV: E=McAfee;i="6200,9189,9989"; a="222341180" X-IronPort-AV: E=Sophos;i="5.82,313,1613462400"; d="scan'208";a="222341180" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 May 2021 08:10:00 -0700 IronPort-SDR: IOVnqwaDlb4a9XEeMYvp7bbL/kLlusFzRrnLZ7UePtd+DiS9Kd65Vs50uYbGQU8l+Ko183Z1xz TmRg/XvCANDQ== X-IronPort-AV: E=Sophos;i="5.82,313,1613462400"; d="scan'208";a="395728149" Received: from cbjoerns-mobl1.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.247]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 May 2021 08:09:58 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Thu, 20 May 2021 17:09:43 +0200 Message-Id: <20210520150947.803891-2-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210520150947.803891-1-thomas.hellstrom@linux.intel.com> References: <20210520150947.803891-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [RFC PATCH 1/5] drm/ttm: Add a generic TTM memcpy move for page-based iomem X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , =?UTF-8?q?Christian=20K=C3=B6nig?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" VGhlIGludGVybmFsIHR0bV9ib191dGlsIG1lbWNweSB1c2VzIGlvcmVtYXAgZnVuY3Rpb25hbGl0 eSwgYW5kIHdoaWxlIGl0CnByb2JhYmx5IG1pZ2h0IGJlIHBvc3NpYmxlIHRvIHVzZSBpdCBmb3Ig Y29weWluZyBpbi0gYW5kIG91dCBvZgpzZ2xpc3QgcmVwcmVzZW50ZWQgaW8gbWVtb3J5LCB1c2lu ZyBpb19tZW1fcmVzZXJ2ZSgpIC8gaW9fbWVtX2ZyZWUoKQpjYWxsYmFja3MsIHRoYXQgd291bGQg Y2F1c2UgcHJvYmxlbXMgd2l0aCBmYXVsdCgpLgpJbnN0ZWFkLCBpbXBsZW1lbnQgYSBtZXRob2Qg bWFwcGluZyBwYWdlLWJ5LXBhZ2UgdXNpbmcga21hcF9sb2NhbCgpCnNlbWFudGljcy4gQXMgYW4g YWRkaXRpb25hbCBiZW5lZml0IHdlIHRoZW4gYXZvaWQgdGhlIG9jY2FzaW9uYWwgZ2xvYmFsClRM QiBmbHVzaGVzIG9mIGlvcmVtYXAoKSBhbmQgY29uc3VtaW5nIGlvcmVtYXAgc3BhY2UsIGVsaW1p bmF0aW9uIG9mIGEKY3JpdGljYWwgcG9pbnQgb2YgZmFpbHVyZSBhbmQgd2l0aCBhIHNsaWdodCBj aGFuZ2Ugb2Ygc2VtYW50aWNzIHdlIGNvdWxkCmFsc28gcHVzaCB0aGUgbWVtY3B5IG91dCBhc3lu YyBmb3IgdGVzdGluZyBhbmQgYXN5bmMgZHJpdmVyIGRldmVsb3BtZW50CnB1cnBvc2VzLgoKQSBz cGVjaWFsIGxpbmVhciBpb21lbSBpdGVyYXRvciBpcyBpbnRyb2R1Y2VkIGludGVybmFsbHkgdG8g bWltaWMgdGhlCm9sZCBpb3JlbWFwIGJlaGF2aW91ciBmb3IgY29kZS1wYXRocyB0aGF0IGNhbid0 IGltbWVkaWF0ZWx5IGJlIHBvcnRlZApvdmVyLiBUaGlzIGFkZHMgdG8gdGhlIGNvZGUgc2l6ZSBh bmQgc2hvdWxkIGJlIGNvbnNpZGVyZWQgYSB0ZW1wb3JhcnkKc29sdXRpb24uCgpMb29raW5nIGF0 IHRoZSBjb2RlIHdlIGhhdmUgYSBsb3Qgb2YgY2hlY2tzIGZvciBpb21hcCB0YWdnZWQgcG9pbnRl cnMuCklkZWFsbHkgd2Ugc2hvdWxkIGV4dGVuZCB0aGUgY29yZSBtZW1yZW1hcCBmdW5jdGlvbnMg dG8gYWxzbyBhY2NlcHQKdW5jYWNoZWQgbWVtb3J5IGFuZCBrbWFwX2xvY2FsIGZ1bmN0aW9uYWxp dHkuIFRoZW4gd2UgY291bGQgc3RyaXAgYQpsb3Qgb2YgY29kZS4KCkNjOiBDaHJpc3RpYW4gS8O2 bmlnIDxjaHJpc3RpYW4ua29lbmlnQGFtZC5jb20+ClNpZ25lZC1vZmYtYnk6IFRob21hcyBIZWxs c3Ryw7ZtIDx0aG9tYXMuaGVsbHN0cm9tQGxpbnV4LmludGVsLmNvbT4KLS0tCiBkcml2ZXJzL2dw dS9kcm0vdHRtL3R0bV9ib191dGlsLmMgfCA0NjggKysrKysrKysrKysrKysrKysrKystLS0tLS0t LS0tCiBpbmNsdWRlL2RybS90dG0vdHRtX2JvX2RyaXZlci5oICAgfCAgOTQgKysrKysrCiAyIGZp bGVzIGNoYW5nZWQsIDQwNyBpbnNlcnRpb25zKCspLCAxNTUgZGVsZXRpb25zKC0pCgpkaWZmIC0t Z2l0IGEvZHJpdmVycy9ncHUvZHJtL3R0bS90dG1fYm9fdXRpbC5jIGIvZHJpdmVycy9ncHUvZHJt L3R0bS90dG1fYm9fdXRpbC5jCmluZGV4IGFlOGI2MTQ2MDcyNC4uYmFkOWIxNmU5NmJhIDEwMDY0 NAotLS0gYS9kcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV9ib191dGlsLmMKKysrIGIvZHJpdmVycy9n cHUvZHJtL3R0bS90dG1fYm9fdXRpbC5jCkBAIC0zNSwxMSArMzUsMTMgQEAKICNpbmNsdWRlIDxs aW51eC9kbWEtYnVmLW1hcC5oPgogI2luY2x1ZGUgPGxpbnV4L2lvLmg+CiAjaW5jbHVkZSA8bGlu dXgvaGlnaG1lbS5oPgorI2luY2x1ZGUgPGxpbnV4L2lvLW1hcHBpbmcuaD4KICNpbmNsdWRlIDxs aW51eC93YWl0Lmg+CiAjaW5jbHVkZSA8bGludXgvc2xhYi5oPgogI2luY2x1ZGUgPGxpbnV4L3Zt YWxsb2MuaD4KICNpbmNsdWRlIDxsaW51eC9tb2R1bGUuaD4KICNpbmNsdWRlIDxsaW51eC9kbWEt cmVzdi5oPgorI2luY2x1ZGUgPGxpbnV4L3NjYXR0ZXJsaXN0Lmg+CiAKIHN0cnVjdCB0dG1fdHJh bnNmZXJfb2JqIHsKIAlzdHJ1Y3QgdHRtX2J1ZmZlcl9vYmplY3QgYmFzZTsKQEAgLTcyLDE5MCAr NzQsMzY2IEBAIHZvaWQgdHRtX21lbV9pb19mcmVlKHN0cnVjdCB0dG1fZGV2aWNlICpiZGV2LAog CW1lbS0+YnVzLmFkZHIgPSBOVUxMOwogfQogCi1zdGF0aWMgaW50IHR0bV9yZXNvdXJjZV9pb3Jl bWFwKHN0cnVjdCB0dG1fZGV2aWNlICpiZGV2LAotCQkJICAgICAgIHN0cnVjdCB0dG1fcmVzb3Vy Y2UgKm1lbSwKLQkJCSAgICAgICB2b2lkICoqdmlydHVhbCkKK3N0YXRpYyBwZ3Byb3RfdCB0dG1f cHJvdF9mcm9tX2NhY2hpbmcoZW51bSB0dG1fY2FjaGluZyBjYWNoaW5nLCBwZ3Byb3RfdCB0bXAp CiB7Ci0JaW50IHJldDsKLQl2b2lkICphZGRyOworCS8qIENhY2hlZCBtYXBwaW5ncyBuZWVkIG5v IGFkanVzdG1lbnQgKi8KKwlpZiAoY2FjaGluZyA9PSB0dG1fY2FjaGVkKQorCQlyZXR1cm4gdG1w OwogCi0JKnZpcnR1YWwgPSBOVUxMOwotCXJldCA9IHR0bV9tZW1faW9fcmVzZXJ2ZShiZGV2LCBt ZW0pOwotCWlmIChyZXQgfHwgIW1lbS0+YnVzLmlzX2lvbWVtKQotCQlyZXR1cm4gcmV0OworI2lm IGRlZmluZWQoX19pMzg2X18pIHx8IGRlZmluZWQoX194ODZfNjRfXykKKwlpZiAoY2FjaGluZyA9 PSB0dG1fd3JpdGVfY29tYmluZWQpCisJCXRtcCA9IHBncHJvdF93cml0ZWNvbWJpbmUodG1wKTsK KwllbHNlIGlmIChib290X2NwdV9kYXRhLng4NiA+IDMpCisJCXRtcCA9IHBncHJvdF9ub25jYWNo ZWQodG1wKTsKKyNlbmRpZgorI2lmIGRlZmluZWQoX19pYTY0X18pIHx8IGRlZmluZWQoX19hcm1f XykgfHwgZGVmaW5lZChfX2FhcmNoNjRfXykgfHwgXAorCWRlZmluZWQoX19wb3dlcnBjX18pIHx8 IGRlZmluZWQoX19taXBzX18pCisJaWYgKGNhY2hpbmcgPT0gdHRtX3dyaXRlX2NvbWJpbmVkKQor CQl0bXAgPSBwZ3Byb3Rfd3JpdGVjb21iaW5lKHRtcCk7CisJZWxzZQorCQl0bXAgPSBwZ3Byb3Rf bm9uY2FjaGVkKHRtcCk7CisjZW5kaWYKKyNpZiBkZWZpbmVkKF9fc3BhcmNfXykKKwl0bXAgPSBw Z3Byb3Rfbm9uY2FjaGVkKHRtcCk7CisjZW5kaWYKKwlyZXR1cm4gdG1wOworfQogCi0JaWYgKG1l bS0+YnVzLmFkZHIpIHsKLQkJYWRkciA9IG1lbS0+YnVzLmFkZHI7Ci0JfSBlbHNlIHsKLQkJc2l6 ZV90IGJ1c19zaXplID0gKHNpemVfdCltZW0tPm51bV9wYWdlcyA8PCBQQUdFX1NISUZUOworc3Rh dGljIHZvaWQgdHRtX2ttYXBfaXRlcl90dF9rbWFwX2xvY2FsKHN0cnVjdCB0dG1fa21hcF9pdGVy ICppdGVyLAorCQkJCQlzdHJ1Y3QgZG1hX2J1Zl9tYXAgKmRtYXAsCisJCQkJCXBnb2ZmX3QgaSkK K3sKKwlzdHJ1Y3QgdHRtX2ttYXBfaXRlcl90dCAqaXRlcl90dCA9CisJCWNvbnRhaW5lcl9vZihp dGVyLCB0eXBlb2YoKml0ZXJfdHQpLCBiYXNlKTsKIAotCQlpZiAobWVtLT5idXMuY2FjaGluZyA9 PSB0dG1fd3JpdGVfY29tYmluZWQpCi0JCQlhZGRyID0gaW9yZW1hcF93YyhtZW0tPmJ1cy5vZmZz ZXQsIGJ1c19zaXplKTsKLSNpZmRlZiBDT05GSUdfWDg2Ci0JCWVsc2UgaWYgKG1lbS0+YnVzLmNh Y2hpbmcgPT0gdHRtX2NhY2hlZCkKLQkJCWFkZHIgPSBpb3JlbWFwX2NhY2hlKG1lbS0+YnVzLm9m ZnNldCwgYnVzX3NpemUpOwotI2VuZGlmCi0JCWVsc2UKLQkJCWFkZHIgPSBpb3JlbWFwKG1lbS0+ YnVzLm9mZnNldCwgYnVzX3NpemUpOwotCQlpZiAoIWFkZHIpIHsKLQkJCXR0bV9tZW1faW9fZnJl ZShiZGV2LCBtZW0pOwotCQkJcmV0dXJuIC1FTk9NRU07Ci0JCX0KKwlkbWFfYnVmX21hcF9zZXRf dmFkZHIoZG1hcCwga21hcF9sb2NhbF9wYWdlX3Byb3QoaXRlcl90dC0+dHQtPnBhZ2VzW2ldLAor CQkJCQkJCSBpdGVyX3R0LT5wcm90KSk7Cit9CisKK3N0YXRpYyB2b2lkIHR0bV9rbWFwX2l0ZXJf aW9tYXBfa21hcF9sb2NhbChzdHJ1Y3QgdHRtX2ttYXBfaXRlciAqaXRlciwKKwkJCQkJICAgc3Ry dWN0IGRtYV9idWZfbWFwICpkbWFwLAorCQkJCQkgICBwZ29mZl90IGkpCit7CisJc3RydWN0IHR0 bV9rbWFwX2l0ZXJfaW9tYXAgKml0ZXJfaW8gPQorCQljb250YWluZXJfb2YoaXRlciwgdHlwZW9m KCppdGVyX2lvKSwgYmFzZSk7CisJdm9pZCBfX2lvbWVtICphZGRyOworCityZXRyeToKKwl3aGls ZSAoaSA+PSBpdGVyX2lvLT5jYWNoZS5lbmQpIHsKKwkJaXRlcl9pby0+Y2FjaGUuc2cgPSBpdGVy X2lvLT5jYWNoZS5zZyA/CisJCQlzZ19uZXh0KGl0ZXJfaW8tPmNhY2hlLnNnKSA6IGl0ZXJfaW8t PnN0LT5zZ2w7CisJCWl0ZXJfaW8tPmNhY2hlLmkgPSBpdGVyX2lvLT5jYWNoZS5lbmQ7CisJCWl0 ZXJfaW8tPmNhY2hlLmVuZCArPSBzZ19kbWFfbGVuKGl0ZXJfaW8tPmNhY2hlLnNnKSA+PgorCQkJ UEFHRV9TSElGVDsKKwkJaXRlcl9pby0+Y2FjaGUub2ZmcyA9IHNnX2RtYV9hZGRyZXNzKGl0ZXJf aW8tPmNhY2hlLnNnKSAtCisJCQlpdGVyX2lvLT5zdGFydDsKIAl9Ci0JKnZpcnR1YWwgPSBhZGRy OwotCXJldHVybiAwOworCisJaWYgKGkgPCBpdGVyX2lvLT5jYWNoZS5pKSB7CisJCWl0ZXJfaW8t PmNhY2hlLmVuZCA9IDA7CisJCWl0ZXJfaW8tPmNhY2hlLnNnID0gTlVMTDsKKwkJZ290byByZXRy eTsKKwl9CisKKwlhZGRyID0gaW9fbWFwcGluZ19tYXBfbG9jYWxfd2MoaXRlcl9pby0+aW9tYXAs IGl0ZXJfaW8tPmNhY2hlLm9mZnMgKworCQkJCSAgICAgICAoKChyZXNvdXJjZV9zaXplX3QpaSAt IGl0ZXJfaW8tPmNhY2hlLmkpCisJCQkJCTw8IFBBR0VfU0hJRlQpKTsKKwlkbWFfYnVmX21hcF9z ZXRfdmFkZHJfaW9tZW0oZG1hcCwgYWRkcik7CiB9CiAKLXN0YXRpYyB2b2lkIHR0bV9yZXNvdXJj ZV9pb3VubWFwKHN0cnVjdCB0dG1fZGV2aWNlICpiZGV2LAotCQkJCXN0cnVjdCB0dG1fcmVzb3Vy Y2UgKm1lbSwKLQkJCQl2b2lkICp2aXJ0dWFsKQorc3RhdGljIGNvbnN0IHN0cnVjdCB0dG1fa21h cF9pdGVyX29wcyB0dG1fa21hcF9pdGVyX3R0X29wcyA9IHsKKwkua21hcF9sb2NhbCA9IHR0bV9r bWFwX2l0ZXJfdHRfa21hcF9sb2NhbCwKKwkubmVlZHNfdW5tYXAgPSB0cnVlCit9OworCitzdGF0 aWMgY29uc3Qgc3RydWN0IHR0bV9rbWFwX2l0ZXJfb3BzIHR0bV9rbWFwX2l0ZXJfaW9fb3BzID0g eworCS5rbWFwX2xvY2FsID0gIHR0bV9rbWFwX2l0ZXJfaW9tYXBfa21hcF9sb2NhbCwKKwkubmVl ZHNfdW5tYXAgPSB0cnVlCit9OworCisvKiBJZiBuZWVkZWQsIG1ha2UgdW5tYXAgZnVuY3Rpb25h bGl0eSBwYXJ0IG9mIHR0bV9rbWFwX2l0ZXJfb3BzICovCitzdGF0aWMgdm9pZCBrdW5tYXBfbG9j YWxfaXRlcihzdHJ1Y3QgdHRtX2ttYXBfaXRlciAqaXRlciwKKwkJCSAgICAgIHN0cnVjdCBkbWFf YnVmX21hcCAqbWFwKQogewotCWlmICh2aXJ0dWFsICYmIG1lbS0+YnVzLmFkZHIgPT0gTlVMTCkK LQkJaW91bm1hcCh2aXJ0dWFsKTsKLQl0dG1fbWVtX2lvX2ZyZWUoYmRldiwgbWVtKTsKKwlpZiAo IWl0ZXItPm9wcy0+bmVlZHNfdW5tYXApCisJCXJldHVybjsKKworCWlmIChtYXAtPmlzX2lvbWVt KQorCQlpb19tYXBwaW5nX3VubWFwX2xvY2FsKG1hcC0+dmFkZHJfaW9tZW0pOworCWVsc2UKKwkJ a3VubWFwX2xvY2FsKG1hcC0+dmFkZHIpOwogfQogCi1zdGF0aWMgaW50IHR0bV9jb3B5X2lvX3Bh Z2Uodm9pZCAqZHN0LCB2b2lkICpzcmMsIHVuc2lnbmVkIGxvbmcgcGFnZSkKKy8qKgorICogdHRt X21vdmVfbWVtY3B5IC0gSGVscGVyIHRvIHBlcmZvcm0gYSBtZW1jcHkgdHRtIG1vdmUgb3BlcmF0 aW9uLgorICogQGJvOiBUaGUgc3RydWN0IHR0bV9idWZmZXJfb2JqZWN0LgorICogQG5ld19tZW06 IFRoZSBzdHJ1Y3QgdHRtX3Jlc291cmNlIHdlJ3JlIG1vdmluZyB0byAoY29weSBkZXN0aW5hdGlv bikuCisgKiBAbmV3X2l0ZXI6IEEgc3RydWN0IHR0bV9rbWFwX2l0ZXIgcmVwcmVzZW50aW5nIHRo ZSBkZXN0aW5hdGlvbiByZXNvdXJjZS4KKyAqIEBvbGRfaXRlcjogQSBzdHJ1Y3QgdHRtX2ttYXBf aXRlciByZXByZXNlbnRpbmcgdGhlIHNvdXJjZSByZXNvdXJjZS4KKyAqCisgKiBUaGlzIGZ1bmN0 aW9uIGlzIGludGVuZGVkIHRvIGJlIGFibGUgdG8gbW92ZSBvdXQgYXN5bmMgdW5kZXIgYQorICog ZG1hLWZlbmNlIGlmIGRlc2lyZWQuCisgKi8KK3ZvaWQgdHRtX21vdmVfbWVtY3B5KHN0cnVjdCB0 dG1fYnVmZmVyX29iamVjdCAqYm8sCisJCSAgICAgc3RydWN0IHR0bV9yZXNvdXJjZSAqbmV3X21l bSwKKwkJICAgICBzdHJ1Y3QgdHRtX2ttYXBfaXRlciAqbmV3X2l0ZXIsCisJCSAgICAgc3RydWN0 IHR0bV9rbWFwX2l0ZXIgKm9sZF9pdGVyKQogewotCXVpbnQzMl90ICpkc3RQID0KLQkgICAgKHVp bnQzMl90ICopICgodW5zaWduZWQgbG9uZylkc3QgKyAocGFnZSA8PCBQQUdFX1NISUZUKSk7Ci0J dWludDMyX3QgKnNyY1AgPQotCSAgICAodWludDMyX3QgKikgKCh1bnNpZ25lZCBsb25nKXNyYyAr IChwYWdlIDw8IFBBR0VfU0hJRlQpKTsKLQotCWludCBpOwotCWZvciAoaSA9IDA7IGkgPCBQQUdF X1NJWkUgLyBzaXplb2YodWludDMyX3QpOyArK2kpCi0JCWlvd3JpdGUzMihpb3JlYWQzMihzcmNQ KyspLCBkc3RQKyspOwotCXJldHVybiAwOworCXN0cnVjdCB0dG1fZGV2aWNlICpiZGV2ID0gYm8t PmJkZXY7CisJc3RydWN0IHR0bV9yZXNvdXJjZV9tYW5hZ2VyICptYW4gPSB0dG1fbWFuYWdlcl90 eXBlKGJkZXYsIG5ld19tZW0tPm1lbV90eXBlKTsKKwlzdHJ1Y3QgdHRtX3R0ICp0dG0gPSBiby0+ dHRtOworCXN0cnVjdCB0dG1fcmVzb3VyY2UgKm9sZF9tZW0gPSAmYm8tPm1lbTsKKwlzdHJ1Y3Qg dHRtX3Jlc291cmNlX21hbmFnZXIgKm9sZF9tYW4gPSB0dG1fbWFuYWdlcl90eXBlKGJkZXYsIG9s ZF9tZW0tPm1lbV90eXBlKTsKKwlzdHJ1Y3QgZG1hX2J1Zl9tYXAgb2xkX21hcCwgbmV3X21hcDsK KwlwZ29mZl90IGk7CisKKwkvKiBTaW5nbGUgVFRNIG1vdmUuIE5PUCAqLworCWlmIChvbGRfbWFu LT51c2VfdHQgJiYgbWFuLT51c2VfdHQpCisJCXJldHVybjsKKworCS8qIERvbid0IG1vdmUgbm9u ZXhpc3RlbnQgZGF0YS4gQ2xlYXIgZGVzdGluYXRpb24gaW5zdGVhZC4gKi8KKwlpZiAob2xkX21h bi0+dXNlX3R0ICYmICFtYW4tPnVzZV90dCAmJgorCSAgICAoIXR0bSB8fCAhdHRtX3R0X2lzX3Bv cHVsYXRlZCh0dG0pKSkgeworCQlpZiAodHRtICYmICEodHRtLT5wYWdlX2ZsYWdzICYgVFRNX1BB R0VfRkxBR19aRVJPX0FMTE9DKSkKKwkJCXJldHVybjsKKworCQlmb3IgKGkgPSAwOyBpIDwgbmV3 X21lbS0+bnVtX3BhZ2VzOyArK2kpIHsKKwkJCW5ld19pdGVyLT5vcHMtPmttYXBfbG9jYWwobmV3 X2l0ZXIsICZuZXdfbWFwLCBpKTsKKwkJCWlmIChuZXdfbWFwLmlzX2lvbWVtKQorCQkJCW1lbXNl dF9pbyhuZXdfbWFwLnZhZGRyX2lvbWVtLCAwLCBQQUdFX1NJWkUpOworCQkJZWxzZQorCQkJCW1l bXNldChuZXdfbWFwLnZhZGRyLCAwLCBQQUdFX1NJWkUpOworCQkJa3VubWFwX2xvY2FsX2l0ZXIo bmV3X2l0ZXIsICZuZXdfbWFwKTsKKwkJfQorCQlyZXR1cm47CisJfQorCisJZm9yIChpID0gMDsg aSA8IG5ld19tZW0tPm51bV9wYWdlczsgKytpKSB7CisJCW5ld19pdGVyLT5vcHMtPmttYXBfbG9j YWwobmV3X2l0ZXIsICZuZXdfbWFwLCBpKTsKKwkJb2xkX2l0ZXItPm9wcy0+a21hcF9sb2NhbChv bGRfaXRlciwgJm9sZF9tYXAsIGkpOworCisJCWlmICghb2xkX21hcC5pc19pb21lbSAmJiAhbmV3 X21hcC5pc19pb21lbSkgeworCQkJbWVtY3B5KG5ld19tYXAudmFkZHIsIG9sZF9tYXAudmFkZHIs IFBBR0VfU0laRSk7CisJCX0gZWxzZSBpZiAoIW9sZF9tYXAuaXNfaW9tZW0pIHsKKwkJCWRtYV9i dWZfbWFwX21lbWNweV90bygmbmV3X21hcCwgb2xkX21hcC52YWRkciwKKwkJCQkJICAgICAgUEFH RV9TSVpFKTsKKwkJfSBlbHNlIGlmICghbmV3X21hcC5pc19pb21lbSkgeworCQkJbWVtY3B5X2Zy b21pbyhuZXdfbWFwLnZhZGRyLCBvbGRfbWFwLnZhZGRyX2lvbWVtLAorCQkJCSAgICAgIFBBR0Vf U0laRSk7CisJCX0gZWxzZSB7CisJCQlpbnQgajsKKwkJCXUzMiBfX2lvbWVtICpzcmMgPSBvbGRf bWFwLnZhZGRyX2lvbWVtOworCQkJdTMyIF9faW9tZW0gKmRzdCA9IG5ld19tYXAudmFkZHJfaW9t ZW07CisKKwkJCWZvciAoaiA9IDA7IGogPCAoUEFHRV9TSVpFID4+IDIpOyArK2opCisJCQkJaW93 cml0ZTMyKGlvcmVhZDMyKHNyYysrKSwgZHN0KyspOworCQl9CisJCWt1bm1hcF9sb2NhbF9pdGVy KG9sZF9pdGVyLCAmb2xkX21hcCk7CisJCWt1bm1hcF9sb2NhbF9pdGVyKG5ld19pdGVyLCAmbmV3 X21hcCk7CisJfQogfQorRVhQT1JUX1NZTUJPTCh0dG1fbW92ZV9tZW1jcHkpOwogCi1zdGF0aWMg aW50IHR0bV9jb3B5X2lvX3R0bV9wYWdlKHN0cnVjdCB0dG1fdHQgKnR0bSwgdm9pZCAqc3JjLAot CQkJCXVuc2lnbmVkIGxvbmcgcGFnZSwKLQkJCQlwZ3Byb3RfdCBwcm90KQorLyoqCisgKiB0dG1f a21hcF9pdGVyX2lvbWFwX2luaXQgLSBJbml0aWFsaXplIGEgc3RydWN0IHR0bV9rbWFwX2l0ZXJf aW9tYXAKKyAqIEBpdGVyX2lvOiBUaGUgc3RydWN0IHR0bV9rbWFwX2l0ZXJfaW9tYXAgdG8gaW5p dGlhbGl6ZS4KKyAqIEBpb21hcDogVGhlIHN0cnVjdCBpb19tYXBwaW5nIHJlcHJlc2VudGluZyB0 aGUgdW5kZXJseWluZyBsaW5lYXIgaW9fbWVtb3J5LgorICogQHN0OiBzZ190YWJsZSBpbnRvIEBp b21hcCwgcmVwcmVzZW50aW5nIHRoZSBtZW1vcnkgb2YgdGhlIHN0cnVjdAorICogdHRtX3Jlc291 cmNlLgorICogQHN0YXJ0OiBPZmZzZXQgdGhhdCBuZWVkcyB0byBiZSBzdWJ0cmFjdGVkIGZyb20g QHN0IHRvIG1ha2UKKyAqIHNnX2RtYV9hZGRyZXNzKHN0LT5zZ2wpIC0gQHN0YXJ0ID09IDAgZm9y IEBpb21hcCBzdGFydC4KKyAqCisgKiBSZXR1cm46IFBvaW50ZXIgdG8gdGhlIGVtYmVkZGVkIHN0 cnVjdCB0dG1fa21hcF9pdGVyLgorICovCitzdHJ1Y3QgdHRtX2ttYXBfaXRlciAqCit0dG1fa21h cF9pdGVyX2lvbWFwX2luaXQoc3RydWN0IHR0bV9rbWFwX2l0ZXJfaW9tYXAgKml0ZXJfaW8sCisJ CQkgc3RydWN0IGlvX21hcHBpbmcgKmlvbWFwLAorCQkJIHN0cnVjdCBzZ190YWJsZSAqc3QsCisJ CQkgcmVzb3VyY2Vfc2l6ZV90IHN0YXJ0KQogewotCXN0cnVjdCBwYWdlICpkID0gdHRtLT5wYWdl c1twYWdlXTsKLQl2b2lkICpkc3Q7CisJaXRlcl9pby0+YmFzZS5vcHMgPSAmdHRtX2ttYXBfaXRl cl9pb19vcHM7CisJaXRlcl9pby0+aW9tYXAgPSBpb21hcDsKKwlpdGVyX2lvLT5zdCA9IHN0Owor CWl0ZXJfaW8tPnN0YXJ0ID0gc3RhcnQ7CisJbWVtc2V0KCZpdGVyX2lvLT5jYWNoZSwgMCwgc2l6 ZW9mKGl0ZXJfaW8tPmNhY2hlKSk7CiAKLQlpZiAoIWQpCi0JCXJldHVybiAtRU5PTUVNOworCXJl dHVybiAmaXRlcl9pby0+YmFzZTsKK30KK0VYUE9SVF9TWU1CT0wodHRtX2ttYXBfaXRlcl9pb21h cF9pbml0KTsKIAotCXNyYyA9ICh2b2lkICopKCh1bnNpZ25lZCBsb25nKXNyYyArIChwYWdlIDw8 IFBBR0VfU0hJRlQpKTsKLQlkc3QgPSBrbWFwX2F0b21pY19wcm90KGQsIHByb3QpOwotCWlmICgh ZHN0KQotCQlyZXR1cm4gLUVOT01FTTsKKy8qKgorICogdHRtX2ttYXBfaXRlcl90dF9pbml0IC0g SW5pdGlhbGl6ZSBhIHN0cnVjdCB0dG1fa21hcF9pdGVyX3R0CisgKiBAaXRlcl90dDogVGhlIHN0 cnVjdCB0dG1fa21hcF9pdGVyX3R0IHRvIGluaXRpYWxpemUuCisgKiBAdHQ6IFN0cnVjdCB0dG1f dHQgaG9sZGluZyBwYWdlIHBvaW50ZXJzIG9mIHRoZSBzdHJ1Y3QgdHRtX3Jlc291cmNlLgorICoK KyAqIFJldHVybjogUG9pbnRlciB0byB0aGUgZW1iZWRkZWQgc3RydWN0IHR0bV9rbWFwX2l0ZXIu CisgKi8KK3N0cnVjdCB0dG1fa21hcF9pdGVyICoKK3R0bV9rbWFwX2l0ZXJfdHRfaW5pdChzdHJ1 Y3QgdHRtX2ttYXBfaXRlcl90dCAqaXRlcl90dCwKKwkJICAgICAgc3RydWN0IHR0bV90dCAqdHQp Cit7CisJaXRlcl90dC0+YmFzZS5vcHMgPSAmdHRtX2ttYXBfaXRlcl90dF9vcHM7CisJaXRlcl90 dC0+dHQgPSB0dDsKKwlpdGVyX3R0LT5wcm90ID0gdHRtX3Byb3RfZnJvbV9jYWNoaW5nKHR0LT5j YWNoaW5nLCBQQUdFX0tFUk5FTCk7CiAKLQltZW1jcHlfZnJvbWlvKGRzdCwgc3JjLCBQQUdFX1NJ WkUpOworCXJldHVybiAmaXRlcl90dC0+YmFzZTsKK30KK0VYUE9SVF9TWU1CT0wodHRtX2ttYXBf aXRlcl90dF9pbml0KTsKIAotCWt1bm1hcF9hdG9taWMoZHN0KTsKKy8qKgorICogRE9DOiBMaW5l YXIgaW8gaXRlcmF0b3IKKyAqCisgKiBUaGlzIGNvZGUgc2hvdWxkIGRpZSBpbiB0aGUgbm90IHRv byBuZWFyIGZ1dHVyZS4gQmVzdCB3b3VsZCBiZSBpZiB3ZSBjb3VsZAorICogbWFrZSBpby1tYXBw aW5nIHVzZSBtZW1yZW1hcCBmb3IgYWxsIGlvIG1lbW9yeSwgYW5kIGhhdmUgbWVtcmVtYXAKKyAq IGltcGxlbWVudCBhIGttYXBfbG9jYWwgZnVuY3Rpb25hbGl0eS4gV2UgY291bGQgdGhlbiBzdHJp cCBhIGh1Z2UgYW1vdW50IG9mCisgKiBjb2RlLiBUaGVzZSBsaW5lYXIgaW8gaXRlcmF0b3JzIGFy ZSBpbXBsZW1lbnRlZCB0byBtaW1pYyBvbGQgZnVuY3Rpb25hbGl0eSwKKyAqIGFuZCB0aGV5IGRv bid0IHVzZSBrbWFwX2xvY2FsIHNlbWFudGljcyBhdCBhbGwgaW50ZXJuYWxseS4gUmF0aGVyIGlv cmVtYXAgb3IKKyAqIGZyaWVuZHMsIGFuZCBhdCBsZWFzdCBvbiAzMi1iaXQgdGhleSBhZGQgZ2xv YmFsIFRMQiBmbHVzaGVzIGFuZCBwb2ludHMKKyAqIG9mIGZhaWx1cmUuCisgKi8KIAotCXJldHVy biAwOworLyoqCisgKiBzdHJ1Y3QgdHRtX2ttYXBfaXRlcl9saW5lYXJfaW8gLSBJdGVyYXRvciBz cGVjaWFsaXphdGlvbiBmb3IgbGluZWFyIGlvCisgKiBAYmFzZTogVGhlIGJhc2UgaXRlcmF0b3IK KyAqIEBkbWFwOiBQb2ludHMgdG8gdGhlIHN0YXJ0aW5nIGFkZHJlc3Mgb2YgdGhlIHJlZ2lvbgor ICogQG5lZWRzX3VubWFwOiBXaGV0aGVyIHdlIG5lZWQgdG8gdW5tYXAgb24gZmluaQorICovCitz dHJ1Y3QgdHRtX2ttYXBfaXRlcl9saW5lYXJfaW8geworCXN0cnVjdCB0dG1fa21hcF9pdGVyIGJh c2U7CisJc3RydWN0IGRtYV9idWZfbWFwIGRtYXA7CisJYm9vbCBuZWVkc191bm1hcDsKK307CisK K3N0YXRpYyB2b2lkIHR0bV9rbWFwX2l0ZXJfbGluZWFyX2lvX2ttYXBfbG9jYWwoc3RydWN0IHR0 bV9rbWFwX2l0ZXIgKml0ZXIsCisJCQkJCSAgICAgICBzdHJ1Y3QgZG1hX2J1Zl9tYXAgKmRtYXAs CisJCQkJCSAgICAgICBwZ29mZl90IGkpCit7CisJc3RydWN0IHR0bV9rbWFwX2l0ZXJfbGluZWFy X2lvICppdGVyX2lvID0KKwkJY29udGFpbmVyX29mKGl0ZXIsIHR5cGVvZigqaXRlcl9pbyksIGJh c2UpOworCisJKmRtYXAgPSBpdGVyX2lvLT5kbWFwOworCWRtYV9idWZfbWFwX2luY3IoZG1hcCwg aSAqIFBBR0VfU0laRSk7CiB9CiAKLXN0YXRpYyBpbnQgdHRtX2NvcHlfdHRtX2lvX3BhZ2Uoc3Ry dWN0IHR0bV90dCAqdHRtLCB2b2lkICpkc3QsCi0JCQkJdW5zaWduZWQgbG9uZyBwYWdlLAotCQkJ CXBncHJvdF90IHByb3QpCitzdGF0aWMgY29uc3Qgc3RydWN0IHR0bV9rbWFwX2l0ZXJfb3BzIHR0 bV9rbWFwX2l0ZXJfbGluZWFyX2lvX29wcyA9IHsKKwkua21hcF9sb2NhbCA9ICB0dG1fa21hcF9p dGVyX2xpbmVhcl9pb19rbWFwX2xvY2FsCit9OworCitzdGF0aWMgc3RydWN0IHR0bV9rbWFwX2l0 ZXIgKgordHRtX2ttYXBfaXRlcl9saW5lYXJfaW9faW5pdChzdHJ1Y3QgdHRtX2ttYXBfaXRlcl9s aW5lYXJfaW8gKml0ZXJfaW8sCisJCQkgICAgIHN0cnVjdCB0dG1fZGV2aWNlICpiZGV2LAorCQkJ ICAgICBzdHJ1Y3QgdHRtX3Jlc291cmNlICptZW0pCiB7Ci0Jc3RydWN0IHBhZ2UgKnMgPSB0dG0t PnBhZ2VzW3BhZ2VdOwotCXZvaWQgKnNyYzsKKwlpbnQgcmV0OwogCi0JaWYgKCFzKQotCQlyZXR1 cm4gLUVOT01FTTsKKwlyZXQgPSB0dG1fbWVtX2lvX3Jlc2VydmUoYmRldiwgbWVtKTsKKwlpZiAo cmV0KQorCQlnb3RvIG91dF9lcnI7CisJaWYgKCFtZW0tPmJ1cy5pc19pb21lbSkgeworCQlyZXQg PSAtRUlOVkFMOworCQlnb3RvIG91dF9pb19mcmVlOworCX0KIAotCWRzdCA9ICh2b2lkICopKCh1 bnNpZ25lZCBsb25nKWRzdCArIChwYWdlIDw8IFBBR0VfU0hJRlQpKTsKLQlzcmMgPSBrbWFwX2F0 b21pY19wcm90KHMsIHByb3QpOwotCWlmICghc3JjKQotCQlyZXR1cm4gLUVOT01FTTsKKwlpZiAo bWVtLT5idXMuYWRkcikgeworCQlkbWFfYnVmX21hcF9zZXRfdmFkZHIoJml0ZXJfaW8tPmRtYXAs IG1lbS0+YnVzLmFkZHIpOworCQlpdGVyX2lvLT5uZWVkc191bm1hcCA9IGZhbHNlOworCX0gZWxz ZSB7CisJCXNpemVfdCBidXNfc2l6ZSA9IChzaXplX3QpbWVtLT5udW1fcGFnZXMgPDwgUEFHRV9T SElGVDsKIAotCW1lbWNweV90b2lvKGRzdCwgc3JjLCBQQUdFX1NJWkUpOworCQlpdGVyX2lvLT5u ZWVkc191bm1hcCA9IHRydWU7CisJCWlmIChtZW0tPmJ1cy5jYWNoaW5nID09IHR0bV93cml0ZV9j b21iaW5lZCkKKwkJCWRtYV9idWZfbWFwX3NldF92YWRkcl9pb21lbSgmaXRlcl9pby0+ZG1hcCwK KwkJCQkJCSAgICBpb3JlbWFwX3djKG1lbS0+YnVzLm9mZnNldCwKKwkJCQkJCQkgICAgICAgYnVz X3NpemUpKTsKKwkJZWxzZSBpZiAobWVtLT5idXMuY2FjaGluZyA9PSB0dG1fY2FjaGVkKQorCQkJ ZG1hX2J1Zl9tYXBfc2V0X3ZhZGRyKCZpdGVyX2lvLT5kbWFwLAorCQkJCQkgICAgICBtZW1yZW1h cChtZW0tPmJ1cy5vZmZzZXQsIGJ1c19zaXplLAorCQkJCQkJICAgICAgIE1FTVJFTUFQX1dCKSk7 CisJCWVsc2UKKwkJCWRtYV9idWZfbWFwX3NldF92YWRkcl9pb21lbSgmaXRlcl9pby0+ZG1hcCwK KwkJCQkJCSAgICBpb3JlbWFwKG1lbS0+YnVzLm9mZnNldCwKKwkJCQkJCQkgICAgYnVzX3NpemUp KTsKKwkJaWYgKGRtYV9idWZfbWFwX2lzX251bGwoJml0ZXJfaW8tPmRtYXApKSB7CisJCQlyZXQg PSAtRU5PTUVNOworCQkJZ290byBvdXRfaW9fZnJlZTsKKwkJfQorCX0KIAotCWt1bm1hcF9hdG9t aWMoc3JjKTsKKwlpdGVyX2lvLT5iYXNlLm9wcyA9ICZ0dG1fa21hcF9pdGVyX2xpbmVhcl9pb19v cHM7CisJcmV0dXJuICZpdGVyX2lvLT5iYXNlOwogCi0JcmV0dXJuIDA7CitvdXRfaW9fZnJlZToK Kwl0dG1fbWVtX2lvX2ZyZWUoYmRldiwgbWVtKTsKK291dF9lcnI6CisJcmV0dXJuIEVSUl9QVFIo cmV0KTsKK30KKworc3RhdGljIHZvaWQKK3R0bV9rbWFwX2l0ZXJfbGluZWFyX2lvX2Zpbmkoc3Ry dWN0IHR0bV9rbWFwX2l0ZXJfbGluZWFyX2lvICppdGVyX2lvLAorCQkJICAgICBzdHJ1Y3QgdHRt X2RldmljZSAqYmRldiwKKwkJCSAgICAgc3RydWN0IHR0bV9yZXNvdXJjZSAqbWVtKQoreworCWlm IChpdGVyX2lvLT5uZWVkc191bm1hcCAmJiBkbWFfYnVmX21hcF9pc19zZXQoJml0ZXJfaW8tPmRt YXApKSB7CisJCWlmIChpdGVyX2lvLT5kbWFwLmlzX2lvbWVtKQorCQkJaW91bm1hcChpdGVyX2lv LT5kbWFwLnZhZGRyX2lvbWVtKTsKKwkJZWxzZQorCQkJbWVtdW5tYXAoaXRlcl9pby0+ZG1hcC52 YWRkcik7CisJfQorCisJdHRtX21lbV9pb19mcmVlKGJkZXYsIG1lbSk7CiB9CiAKK3N0YXRpYyBp bnQgdHRtX2JvX3dhaXRfZnJlZV9ub2RlKHN0cnVjdCB0dG1fYnVmZmVyX29iamVjdCAqYm8sCisJ CQkJIGJvb2wgZHN0X3VzZV90dCk7CisKIGludCB0dG1fYm9fbW92ZV9tZW1jcHkoc3RydWN0IHR0 bV9idWZmZXJfb2JqZWN0ICpibywKIAkJICAgICAgIHN0cnVjdCB0dG1fb3BlcmF0aW9uX2N0eCAq Y3R4LAogCQkgICAgICAgc3RydWN0IHR0bV9yZXNvdXJjZSAqbmV3X21lbSkKIHsKIAlzdHJ1Y3Qg dHRtX2RldmljZSAqYmRldiA9IGJvLT5iZGV2OwogCXN0cnVjdCB0dG1fcmVzb3VyY2VfbWFuYWdl ciAqbWFuID0gdHRtX21hbmFnZXJfdHlwZShiZGV2LCBuZXdfbWVtLT5tZW1fdHlwZSk7CisJc3Ry dWN0IHR0bV9yZXNvdXJjZV9tYW5hZ2VyICpuZXdfbWFuID0KKwkJdHRtX21hbmFnZXJfdHlwZShi by0+YmRldiwgbmV3X21lbS0+bWVtX3R5cGUpOwogCXN0cnVjdCB0dG1fdHQgKnR0bSA9IGJvLT50 dG07CiAJc3RydWN0IHR0bV9yZXNvdXJjZSAqb2xkX21lbSA9ICZiby0+bWVtOwogCXN0cnVjdCB0 dG1fcmVzb3VyY2Ugb2xkX2NvcHkgPSAqb2xkX21lbTsKLQl2b2lkICpvbGRfaW9tYXA7Ci0Jdm9p ZCAqbmV3X2lvbWFwOworCXVuaW9uIHsKKwkJc3RydWN0IHR0bV9rbWFwX2l0ZXJfdHQgdHQ7CisJ CXN0cnVjdCB0dG1fa21hcF9pdGVyX2xpbmVhcl9pbyBpbzsKKwl9IF9uZXdfaXRlciwgX29sZF9p dGVyOworCXN0cnVjdCB0dG1fa21hcF9pdGVyICpuZXdfaXRlciwgKm9sZF9pdGVyOwogCWludCBy ZXQ7Ci0JdW5zaWduZWQgbG9uZyBpOwotCi0JcmV0ID0gdHRtX2JvX3dhaXRfY3R4KGJvLCBjdHgp OwotCWlmIChyZXQpCi0JCXJldHVybiByZXQ7Ci0KLQlyZXQgPSB0dG1fcmVzb3VyY2VfaW9yZW1h cChiZGV2LCBvbGRfbWVtLCAmb2xkX2lvbWFwKTsKLQlpZiAocmV0KQotCQlyZXR1cm4gcmV0Owot CXJldCA9IHR0bV9yZXNvdXJjZV9pb3JlbWFwKGJkZXYsIG5ld19tZW0sICZuZXdfaW9tYXApOwot CWlmIChyZXQpCi0JCWdvdG8gb3V0OwotCi0JLyoKLQkgKiBTaW5nbGUgVFRNIG1vdmUuIE5PUC4K LQkgKi8KLQlpZiAob2xkX2lvbWFwID09IE5VTEwgJiYgbmV3X2lvbWFwID09IE5VTEwpCi0JCWdv dG8gb3V0MjsKIAotCS8qCi0JICogRG9uJ3QgbW92ZSBub25leGlzdGVudCBkYXRhLiBDbGVhciBk ZXN0aW5hdGlvbiBpbnN0ZWFkLgotCSAqLwotCWlmIChvbGRfaW9tYXAgPT0gTlVMTCAmJgotCSAg ICAodHRtID09IE5VTEwgfHwgKCF0dG1fdHRfaXNfcG9wdWxhdGVkKHR0bSkgJiYKLQkJCSAgICAg ISh0dG0tPnBhZ2VfZmxhZ3MgJiBUVE1fUEFHRV9GTEFHX1NXQVBQRUQpKSkpIHsKLQkJbWVtc2V0 X2lvKG5ld19pb21hcCwgMCwgbmV3X21lbS0+bnVtX3BhZ2VzKlBBR0VfU0laRSk7Ci0JCWdvdG8g b3V0MjsKLQl9Ci0KLQkvKgotCSAqIFRUTSBtaWdodCBiZSBudWxsIGZvciBtb3ZlcyB3aXRoaW4g dGhlIHNhbWUgcmVnaW9uLgotCSAqLwogCWlmICh0dG0pIHsKIAkJcmV0ID0gdHRtX3R0X3BvcHVs YXRlKGJkZXYsIHR0bSwgY3R4KTsKIAkJaWYgKHJldCkKLQkJCWdvdG8gb3V0MTsKKwkJCXJldHVy biByZXQ7CiAJfQogCi0JZm9yIChpID0gMDsgaSA8IG5ld19tZW0tPm51bV9wYWdlczsgKytpKSB7 Ci0JCWlmIChvbGRfaW9tYXAgPT0gTlVMTCkgewotCQkJcGdwcm90X3QgcHJvdCA9IHR0bV9pb19w cm90KGJvLCBvbGRfbWVtLCBQQUdFX0tFUk5FTCk7Ci0JCQlyZXQgPSB0dG1fY29weV90dG1faW9f cGFnZSh0dG0sIG5ld19pb21hcCwgaSwKLQkJCQkJCSAgIHByb3QpOwotCQl9IGVsc2UgaWYgKG5l d19pb21hcCA9PSBOVUxMKSB7Ci0JCQlwZ3Byb3RfdCBwcm90ID0gdHRtX2lvX3Byb3QoYm8sIG5l d19tZW0sIFBBR0VfS0VSTkVMKTsKLQkJCXJldCA9IHR0bV9jb3B5X2lvX3R0bV9wYWdlKHR0bSwg b2xkX2lvbWFwLCBpLAotCQkJCQkJICAgcHJvdCk7Ci0JCX0gZWxzZSB7Ci0JCQlyZXQgPSB0dG1f Y29weV9pb19wYWdlKG5ld19pb21hcCwgb2xkX2lvbWFwLCBpKTsKLQkJfQotCQlpZiAocmV0KQot CQkJZ290byBvdXQxOworCW5ld19pdGVyID0gbmV3X21hbi0+dXNlX3R0ID8KKwkJdHRtX2ttYXBf aXRlcl90dF9pbml0KCZfbmV3X2l0ZXIudHQsIGJvLT50dG0pIDoKKwkJdHRtX2ttYXBfaXRlcl9s aW5lYXJfaW9faW5pdCgmX25ld19pdGVyLmlvLCBiZGV2LCBuZXdfbWVtKTsKKwlpZiAoSVNfRVJS KG5ld19pdGVyKSkKKwkJcmV0dXJuIFBUUl9FUlIobmV3X2l0ZXIpOworCisJb2xkX2l0ZXIgPSBt YW4tPnVzZV90dCA/CisJCXR0bV9rbWFwX2l0ZXJfdHRfaW5pdCgmX29sZF9pdGVyLnR0LCBiby0+ dHRtKSA6CisJCXR0bV9rbWFwX2l0ZXJfbGluZWFyX2lvX2luaXQoJl9vbGRfaXRlci5pbywgYmRl diwgb2xkX21lbSk7CisJaWYgKElTX0VSUihvbGRfaXRlcikpIHsKKwkJcmV0ID0gUFRSX0VSUihv bGRfaXRlcik7CisJCWdvdG8gb3V0X29sZF9pdGVyOwogCX0KLQltYigpOwotb3V0MjoKLQlvbGRf Y29weSA9ICpvbGRfbWVtOwogCi0JdHRtX2JvX2Fzc2lnbl9tZW0oYm8sIG5ld19tZW0pOworCXR0 bV9tb3ZlX21lbWNweShibywgbmV3X21lbSwgbmV3X2l0ZXIsIG9sZF9pdGVyKTsKKwlvbGRfY29w eSA9ICpvbGRfbWVtOworCXJldCA9IHR0bV9ib193YWl0X2ZyZWVfbm9kZShibywgbmV3X21hbi0+ dXNlX3R0KTsKIAogCWlmICghbWFuLT51c2VfdHQpCi0JCXR0bV9ib190dF9kZXN0cm95KGJvKTsK KwkJdHRtX2ttYXBfaXRlcl9saW5lYXJfaW9fZmluaSgmX29sZF9pdGVyLmlvLCBiZGV2LCAmb2xk X2NvcHkpOworb3V0X29sZF9pdGVyOgorCWlmICghbmV3X21hbi0+dXNlX3R0KQorCQl0dG1fa21h cF9pdGVyX2xpbmVhcl9pb19maW5pKCZfbmV3X2l0ZXIuaW8sIGJkZXYsIG5ld19tZW0pOwogCi1v dXQxOgotCXR0bV9yZXNvdXJjZV9pb3VubWFwKGJkZXYsIG9sZF9tZW0sIG5ld19pb21hcCk7Ci1v dXQ6Ci0JdHRtX3Jlc291cmNlX2lvdW5tYXAoYmRldiwgJm9sZF9jb3B5LCBvbGRfaW9tYXApOwot Ci0JLyoKLQkgKiBPbiBlcnJvciwga2VlcCB0aGUgbW0gbm9kZSEKLQkgKi8KLQlpZiAoIXJldCkK LQkJdHRtX3Jlc291cmNlX2ZyZWUoYm8sICZvbGRfY29weSk7CiAJcmV0dXJuIHJldDsKIH0KIEVY UE9SVF9TWU1CT0wodHRtX2JvX21vdmVfbWVtY3B5KTsKQEAgLTMzNiwyNyArNTE0LDcgQEAgcGdw cm90X3QgdHRtX2lvX3Byb3Qoc3RydWN0IHR0bV9idWZmZXJfb2JqZWN0ICpibywgc3RydWN0IHR0 bV9yZXNvdXJjZSAqcmVzLAogCW1hbiA9IHR0bV9tYW5hZ2VyX3R5cGUoYm8tPmJkZXYsIHJlcy0+ bWVtX3R5cGUpOwogCWNhY2hpbmcgPSBtYW4tPnVzZV90dCA/IGJvLT50dG0tPmNhY2hpbmcgOiBy ZXMtPmJ1cy5jYWNoaW5nOwogCi0JLyogQ2FjaGVkIG1hcHBpbmdzIG5lZWQgbm8gYWRqdXN0bWVu dCAqLwotCWlmIChjYWNoaW5nID09IHR0bV9jYWNoZWQpCi0JCXJldHVybiB0bXA7Ci0KLSNpZiBk ZWZpbmVkKF9faTM4Nl9fKSB8fCBkZWZpbmVkKF9feDg2XzY0X18pCi0JaWYgKGNhY2hpbmcgPT0g dHRtX3dyaXRlX2NvbWJpbmVkKQotCQl0bXAgPSBwZ3Byb3Rfd3JpdGVjb21iaW5lKHRtcCk7Ci0J ZWxzZSBpZiAoYm9vdF9jcHVfZGF0YS54ODYgPiAzKQotCQl0bXAgPSBwZ3Byb3Rfbm9uY2FjaGVk KHRtcCk7Ci0jZW5kaWYKLSNpZiBkZWZpbmVkKF9faWE2NF9fKSB8fCBkZWZpbmVkKF9fYXJtX18p IHx8IGRlZmluZWQoX19hYXJjaDY0X18pIHx8IFwKLSAgICBkZWZpbmVkKF9fcG93ZXJwY19fKSB8 fCBkZWZpbmVkKF9fbWlwc19fKQotCWlmIChjYWNoaW5nID09IHR0bV93cml0ZV9jb21iaW5lZCkK LQkJdG1wID0gcGdwcm90X3dyaXRlY29tYmluZSh0bXApOwotCWVsc2UKLQkJdG1wID0gcGdwcm90 X25vbmNhY2hlZCh0bXApOwotI2VuZGlmCi0jaWYgZGVmaW5lZChfX3NwYXJjX18pCi0JdG1wID0g cGdwcm90X25vbmNhY2hlZCh0bXApOwotI2VuZGlmCi0JcmV0dXJuIHRtcDsKKwlyZXR1cm4gdHRt X3Byb3RfZnJvbV9jYWNoaW5nKGNhY2hpbmcsIHRtcCk7CiB9CiBFWFBPUlRfU1lNQk9MKHR0bV9p b19wcm90KTsKIApkaWZmIC0tZ2l0IGEvaW5jbHVkZS9kcm0vdHRtL3R0bV9ib19kcml2ZXIuaCBi L2luY2x1ZGUvZHJtL3R0bS90dG1fYm9fZHJpdmVyLmgKaW5kZXggZGJjY2FjOTU3ZjhmLi4zMGYx ZGNlOTg3YjIgMTAwNjQ0Ci0tLSBhL2luY2x1ZGUvZHJtL3R0bS90dG1fYm9fZHJpdmVyLmgKKysr IGIvaW5jbHVkZS9kcm0vdHRtL3R0bV9ib19kcml2ZXIuaApAQCAtMzMyLDQgKzMzMiw5OCBAQCBp bnQgdHRtX3JhbmdlX21hbl9pbml0KHN0cnVjdCB0dG1fZGV2aWNlICpiZGV2LAogaW50IHR0bV9y YW5nZV9tYW5fZmluaShzdHJ1Y3QgdHRtX2RldmljZSAqYmRldiwKIAkJICAgICAgIHVuc2lnbmVk IHR5cGUpOwogCitzdHJ1Y3QgZG1hX2J1Zl9tYXA7CitzdHJ1Y3QgaW9fbWFwcGluZzsKK3N0cnVj dCBzZ190YWJsZTsKK3N0cnVjdCBzY2F0dGVybGlzdDsKK3N0cnVjdCB0dG1fa21hcF9pdGVyOwor CisvKioKKyAqIHN0cnVjdCB0dG1fa21hcF9pdGVyX29wcyAtIE9wcyBzdHJ1Y3R1cmUgZm9yIGEg c3RydWN0CisgKiB0dG1fa21hcF9pdGVyLgorICogQG5lZWRzX3VubWFwIC0gV2hldGhlciBhIGt1 bm1hcF9sb2NhbCBpcyBuZWVkZWQgdG8gYmFsYW5jZSBAa21hcF9sb2NhbC4KKyAqLworc3RydWN0 IHR0bV9rbWFwX2l0ZXJfb3BzIHsKKwkvKioKKwkgKiBrbWFwX2xvY2FsIC0gTWFwIGEgUEFHRV9T SVpFIHBhcnQgb2YgdGhlIHJlc291cmNlIHVzaW5nCisJICoga21hcF9sb2NhbCBzZW1hbnRpY3Mu CisJICogQHJlc19rbWFwOiBQb2ludGVyIHRvIHRoZSBzdHJ1Y3QgdHRtX2ttYXBfaXRlciByZXBy ZXNlbnRpbmcKKwkgKiB0aGUgcmVzb3VyY2UuCisJICogQGRtYXA6IFRoZSBzdHJ1Y3QgZG1hX2J1 Zl9tYXAgaG9sZGluZyB0aGUgdmlydHVhbCBhZGRyZXNzIGFmdGVyCisJICogdGhlIG9wZXJhdGlv bi4KKwkgKiBAaTogVGhlIGxvY2F0aW9uIHdpdGhpbiB0aGUgcmVzb3VyY2UgdG8gbWFwLiBQQUdF X1NJWkUgZ3JhbnVsYXJpdHkuCisJICovCisJdm9pZCAoKmttYXBfbG9jYWwpKHN0cnVjdCB0dG1f a21hcF9pdGVyICpyZXNfa21hcCwKKwkJCSAgIHN0cnVjdCBkbWFfYnVmX21hcCAqZG1hcCwgcGdv ZmZfdCBpKTsKKwlib29sIG5lZWRzX3VubWFwOworfTsKKworLyoqCisgKiBzdHJ1Y3QgdHRtX2tt YXBfaXRlciAtIEl0ZXJhdG9yIGZvciBrbWFwX2xvY2FsIHR5cGUgb3BlcmF0aW9ucyBvbiBhCisg KiByZXNvdXJjZS4KKyAqIEBvcHM6IFBvaW50ZXIgdG8gdGhlIG9wZXJhdGlvbnMgc3RydWN0Lgor ICoKKyAqIFRoaXMgc3RydWN0IGlzIGludGVuZGVkIHRvIGJlIGVtYmVkZGVkIGluIGEgcmVzb3Vy Y2Utc3BlY2lmaWMgc3BlY2lhbGl6YXRpb24KKyAqIGltcGxlbWVudGluZyBvcGVyYXRpb25zIGZv ciB0aGUgcmVzb3VyY2UuCisgKgorICogTm90aGluZyBzdG9wcyB1cyBmcm9tIGV4dGVuZGluZyB0 aGUgb3BlcmF0aW9ucyB0byB2bWFwLCB2bWFwX3BmbiBldGMsCisgKiByZXBsYWNpbmcgc29tZSBv ciBwYXJ0cyBvZiB0aGUgdHRtX2JvX3V0aWwuIGNwdS1tYXAgZnVuY3Rpb25hbGl0eS4KKyAqLwor c3RydWN0IHR0bV9rbWFwX2l0ZXIgeworCWNvbnN0IHN0cnVjdCB0dG1fa21hcF9pdGVyX29wcyAq b3BzOworfTsKKworLyoqCisgKiBzdHJ1Y3QgdHRtX2ttYXBfaXRlcl90dCAtIFNwZWNpYWxpemF0 aW9uIGZvciBhIHR0IChwYWdlKSBiYWNrZWQgc3RydWN0CisgKiB0dG1fcmVzb3VyY2UuCisgKiBA YmFzZTogRW1iZWRkZWQgc3RydWN0IHR0bV9rbWFwX2l0ZXIgcHJvdmlkaW5nIHRoZSB1c2FnZSBp bnRlcmZhY2UKKyAqIEB0dDogQ2FjaGVkIHN0cnVjdCB0dG1fdHQuCisgKi8KK3N0cnVjdCB0dG1f a21hcF9pdGVyX3R0IHsKKwlzdHJ1Y3QgdHRtX2ttYXBfaXRlciBiYXNlOworCXN0cnVjdCB0dG1f dHQgKnR0OworCXBncHJvdF90IHByb3Q7Cit9OworCisvKioKKyAqIHN0cnVjdCB0dG1fa21hcF9p dGVyX2lvbWFwIC0gU3BlY2lhbGl6YXRpb24gZm9yIGEgc3RydWN0IGlvX21hcHBpbmcgKworICog c3RydWN0IHNnX3RhYmxlIGJhY2tlZCBzdHJ1Y3QgdHRtX3Jlc291cmNlLgorICogQGJhc2U6IEVt YmVkZGVkIHN0cnVjdCB0dG1fa21hcF9pdGVyIHByb3ZpZGluZyB0aGUgdXNhZ2UgaW50ZXJmYWNl LgorICogQGlvbWFwOiBzdHJ1Y3QgaW9fbWFwcGluZyByZXByZXNlbnRpbmcgdGhlIHVuZGVybHlp bmcgbGluZWFyIGlvX21lbW9yeS4KKyAqIEBzdDogc2dfdGFibGUgaW50byBAaW9tYXAsIHJlcHJl c2VudGluZyB0aGUgbWVtb3J5IG9mIHRoZSBzdHJ1Y3QgdHRtX3Jlc291cmNlLgorICogQHN0YXJ0 OiBPZmZzZXQgdGhhdCBuZWVkcyB0byBiZSBzdWJ0cmFjdGVkIGZyb20gQHN0IHRvIG1ha2UKKyAq IHNnX2RtYV9hZGRyZXNzKHN0LT5zZ2wpIC0gQHN0YXJ0ID09IDAgZm9yIEBpb21hcCBzdGFydC4K KyAqIEBjYWNoZTogU2NhdHRlcmxpc3QgdHJhdmVyc2FsIGNhY2hlIGZvciBmYXN0IGxvb2t1cHMu CisgKiBAY2FjaGUuc2c6IFBvaW50ZXIgdG8gdGhlIGN1cnJlbnRseSBjYWNoZWQgc2NhdHRlcmxp c3Qgc2VnbWVudC4KKyAqIEBjYWNoZS5pOiBGaXJzdCBpbmRleCBvZiBAc2cuIFBBR0VfU0laRSBn cmFudWxhcml0eS4KKyAqIEBjYWNoZS5lbmQ6IExhc3QgaW5kZXggKyAxIG9mIEBzZy4gUEFHRV9T SVpFIGdyYW51bGFyaXR5LgorICogQGNhY2hlLm9mZnM6IEZpcnN0IG9mZnNldCBpbnRvIEBpb21h cCBvZiBAc2cuIFBBR0VfU0laRSBncmFudWxhcml0eS4KKyAqLworc3RydWN0IHR0bV9rbWFwX2l0 ZXJfaW9tYXAgeworCXN0cnVjdCB0dG1fa21hcF9pdGVyIGJhc2U7CisJc3RydWN0IGlvX21hcHBp bmcgKmlvbWFwOworCXN0cnVjdCBzZ190YWJsZSAqc3Q7CisJcmVzb3VyY2Vfc2l6ZV90IHN0YXJ0 OworCXN0cnVjdCB7CisJCXN0cnVjdCBzY2F0dGVybGlzdCAqc2c7CisJCXBnb2ZmX3QgaTsKKwkJ cGdvZmZfdCBlbmQ7CisJCXBnb2ZmX3Qgb2ZmczsKKwl9IGNhY2hlOworfTsKKwordm9pZCB0dG1f bW92ZV9tZW1jcHkoc3RydWN0IHR0bV9idWZmZXJfb2JqZWN0ICpibywKKwkJICAgICBzdHJ1Y3Qg dHRtX3Jlc291cmNlICpuZXdfbWVtLAorCQkgICAgIHN0cnVjdCB0dG1fa21hcF9pdGVyICpuZXdf aXRlciwKKwkJICAgICBzdHJ1Y3QgdHRtX2ttYXBfaXRlciAqb2xkX2l0ZXIpOworCitzdHJ1Y3Qg dHRtX2ttYXBfaXRlciAqCit0dG1fa21hcF9pdGVyX3R0X2luaXQoc3RydWN0IHR0bV9rbWFwX2l0 ZXJfdHQgKml0ZXJfdHQsCisJCSAgICAgIHN0cnVjdCB0dG1fdHQgKnR0KTsKKworc3RydWN0IHR0 bV9rbWFwX2l0ZXIgKgordHRtX2ttYXBfaXRlcl9pb21hcF9pbml0KHN0cnVjdCB0dG1fa21hcF9p dGVyX2lvbWFwICppdGVyX2lvLAorCQkJIHN0cnVjdCBpb19tYXBwaW5nICppb21hcCwKKwkJCSBz dHJ1Y3Qgc2dfdGFibGUgKnN0LAorCQkJIHJlc291cmNlX3NpemVfdCBzdGFydCk7CiAjZW5kaWYK LS0gCjIuMzEuMQoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18KSW50ZWwtZ2Z4IG1haWxpbmcgbGlzdApJbnRlbC1nZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3Jn Cmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtZ2Z4 Cg==