From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D41E3C47E49 for ; Mon, 28 Oct 2019 20:11:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7516E208C0 for ; Mon, 28 Oct 2019 20:11:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="ZMZ20vhD" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7516E208C0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AB42A6B026B; Mon, 28 Oct 2019 16:10:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3CC86B026C; Mon, 28 Oct 2019 16:10:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B6B96B026E; Mon, 28 Oct 2019 16:10:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0030.hostedemail.com [216.40.44.30]) by kanga.kvack.org (Postfix) with ESMTP id 364406B026B for ; Mon, 28 Oct 2019 16:10:52 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id C331A8249980 for ; Mon, 28 Oct 2019 20:10:51 +0000 (UTC) X-FDA: 76094286702.22.spot36_41a09742f0a48 X-HE-Tag: spot36_41a09742f0a48 X-Filterd-Recvd-Size: 19509 Received: from mail-qk1-f196.google.com (mail-qk1-f196.google.com [209.85.222.196]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Mon, 28 Oct 2019 20:10:50 +0000 (UTC) Received: by mail-qk1-f196.google.com with SMTP id y81so9744854qkb.7 for ; Mon, 28 Oct 2019 13:10:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kWAFXaairmju5dUouyI0fIFpVoVh+rRI3GJkG/c3jbo=; b=ZMZ20vhDqRatgd/VeNs+gakMuIFZXaMsb6XQWoUK9HXLD+PHirdDumPc851+WT2Hyf C1Iv1wiPN6IlkT/5OzHp2qTA3MHXJcFx6P54sX18cmGOfENyUr/3lvF7WB/e9idjitHy lGg8W7hcfUwYhCtTo+P5QwNkSGSXWLHrmB7UiWpkznO/5PpvIjO4rkGK5oPJe284j39Z 0SrDdsATeTt0aysE+Vjhm+ODlfdb0SYunqwjoByjas1/pFQCfKFMrcffBKQilclNwoJN Kw57TFwiQGNXYLPWr/vN3hajssQSFf3odyyUW2L2YFJ1wLFryfEwL6Mj/wjl8DvBS7u6 6HmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kWAFXaairmju5dUouyI0fIFpVoVh+rRI3GJkG/c3jbo=; b=YP5xBL4JqzAUG4UGa5jCQtjHszvpGr5zxiddVtEsJw7KC9KfsVKSG6eqxqTi4sW519 xd9/aBSiwfP0Tcm6zC/YboL2t/dFhkzZsfQUP4fFS+Dhjtxuh2Xn/sT7R757ChJqRJ5/ bK4PGc//FlfSqWaI73/5uR33Pf+68Q1w0nPwBLB7Tne6pUj0KFCozXGNcTcSNCzmte3h tEb98vRgraAnY+Izz1jpUi7JoPT18hKOZi7yuYdVNiY+3GSzOHO011S86tieAMdVVRH4 J5VY1hzsPPsgswms8iy7aeA0UZf0zue9nTq5fO7K8s/5zKcfwtskZZeKhE61aNMwQAKn RXDA== X-Gm-Message-State: APjAAAXpCsnheiEL8BgORv0sMJKLzCAzp8KtxszQJLPy1PX1OXKdmOqV MUk7kP6VhKonsdCBQJjCGRkoeiTlrpo= X-Google-Smtp-Source: APXvYqyexUtgyT46O4Xj8Sx5B6Zyp9sP915KnVYaCPjwqJLiYtfebmqJR+ZAbE7J5Hww5Pxy2DBZEA== X-Received: by 2002:a05:620a:13f0:: with SMTP id h16mr17732993qkl.310.1572293449993; Mon, 28 Oct 2019 13:10:49 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id o1sm7821379qtb.82.2019.10.28.13.10.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 28 Oct 2019 13:10:48 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iPBLf-0001hL-Mi; Mon, 28 Oct 2019 17:10:43 -0300 From: Jason Gunthorpe To: linux-mm@kvack.org, Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Alex Deucher , Ben Skeggs , Boris Ostrovsky , =?UTF-8?q?Christian=20K=C3=B6nig?= , David Zhou , Dennis Dalessandro , Juergen Gross , Mike Marciniszyn , Oleksandr Andrushchenko , Petr Cvek , Stefano Stabellini , nouveau@lists.freedesktop.org, xen-devel@lists.xenproject.org, Christoph Hellwig , Jason Gunthorpe Subject: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror Date: Mon, 28 Oct 2019 17:10:31 -0300 Message-Id: <20191028201032.6352-15-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191028201032.6352-1-jgg@ziepe.ca> References: <20191028201032.6352-1-jgg@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jason Gunthorpe Convert the collision-retry lock around hmm_range_fault to use the one no= w provided by the mmu_range notifier. Although this driver does not seem to use the collision retry lock that hmm provides correctly, it can still be converted over to use the mmu_range_notifier api instead of hmm_mirror without too much trouble. This also deletes another place where a driver is associating additional data (struct amdgpu_mn) with a mmu_struct. Cc: Alex Deucher Cc: Christian K=C3=B6nig Cc: David (ChunMing) Zhou Cc: amd-gfx@lists.freedesktop.org Signed-off-by: Jason Gunthorpe --- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 4 + drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 14 +- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 148 ++---------------- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 49 ------ drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 76 ++++----- 5 files changed, 66 insertions(+), 225 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/g= pu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index 47700302a08b7f..1bcedb9b477dce 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -1738,6 +1738,10 @@ static int update_invalid_user_pages(struct amdkfd= _process_info *process_info, return ret; } =20 + /* + * FIXME: Cannot ignore the return code, must hold + * notifier_lock + */ amdgpu_ttm_tt_get_user_pages_done(bo->tbo.ttm); =20 /* Mark the BO as valid unless it was invalidated diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd= /amdgpu/amdgpu_cs.c index 2e53feed40e230..76771f5f0b60ab 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c @@ -607,8 +607,6 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_pars= er *p, e->tv.num_shared =3D 2; =20 amdgpu_bo_list_get_list(p->bo_list, &p->validated); - if (p->bo_list->first_userptr !=3D p->bo_list->num_entries) - p->mn =3D amdgpu_mn_get(p->adev, AMDGPU_MN_TYPE_GFX); =20 INIT_LIST_HEAD(&duplicates); amdgpu_vm_get_pd_bo(&fpriv->vm, &p->validated, &p->vm_pd); @@ -1291,11 +1289,11 @@ static int amdgpu_cs_submit(struct amdgpu_cs_pars= er *p, if (r) goto error_unlock; =20 - /* No memory allocation is allowed while holding the mn lock. - * p->mn is hold until amdgpu_cs_submit is finished and fence is added - * to BOs. + /* No memory allocation is allowed while holding the notifier lock. + * The lock is held until amdgpu_cs_submit is finished and fence is + * added to BOs. */ - amdgpu_mn_lock(p->mn); + mutex_lock(&p->adev->notifier_lock); =20 /* If userptr are invalidated after amdgpu_cs_parser_bos(), return * -EAGAIN, drmIoctl in libdrm will restart the amdgpu_cs_ioctl. @@ -1338,13 +1336,13 @@ static int amdgpu_cs_submit(struct amdgpu_cs_pars= er *p, amdgpu_vm_move_to_lru_tail(p->adev, &fpriv->vm); =20 ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence); - amdgpu_mn_unlock(p->mn); + mutex_unlock(&p->adev->notifier_lock); =20 return 0; =20 error_abort: drm_sched_job_cleanup(&job->base); - amdgpu_mn_unlock(p->mn); + mutex_unlock(&p->adev->notifier_lock); =20 error_unlock: amdgpu_job_free(job); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c b/drivers/gpu/drm/amd= /amdgpu/amdgpu_mn.c index 4ffd7b90f4d907..cb718a064eb491 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c @@ -50,28 +50,6 @@ #include "amdgpu.h" #include "amdgpu_amdkfd.h" =20 -/** - * amdgpu_mn_lock - take the write side lock for this notifier - * - * @mn: our notifier - */ -void amdgpu_mn_lock(struct amdgpu_mn *mn) -{ - if (mn) - down_write(&mn->lock); -} - -/** - * amdgpu_mn_unlock - drop the write side lock for this notifier - * - * @mn: our notifier - */ -void amdgpu_mn_unlock(struct amdgpu_mn *mn) -{ - if (mn) - up_write(&mn->lock); -} - /** * amdgpu_mn_invalidate_gfx - callback to notify about mm change * @@ -82,12 +60,19 @@ void amdgpu_mn_unlock(struct amdgpu_mn *mn) * potentially dirty. */ static bool amdgpu_mn_invalidate_gfx(struct mmu_range_notifier *mrn, - const struct mmu_notifier_range *range) + const struct mmu_notifier_range *range, + unsigned long cur_seq) { struct amdgpu_bo *bo =3D container_of(mrn, struct amdgpu_bo, notifier); struct amdgpu_device *adev =3D amdgpu_ttm_adev(bo->tbo.bdev); long r; =20 + /* + * FIXME: Must hold some lock shared with + * amdgpu_ttm_tt_get_user_pages_done() + */ + mmu_range_set_seq(mrn, cur_seq); + /* FIXME: Is this necessary? */ if (!amdgpu_ttm_tt_affect_userptr(bo->tbo.ttm, range->start, range->end)) @@ -119,11 +104,18 @@ static const struct mmu_range_notifier_ops amdgpu_m= n_gfx_ops =3D { * evicting all user-mode queues of the process. */ static bool amdgpu_mn_invalidate_hsa(struct mmu_range_notifier *mrn, - const struct mmu_notifier_range *range) + const struct mmu_notifier_range *range, + unsigned long cur_seq) { struct amdgpu_bo *bo =3D container_of(mrn, struct amdgpu_bo, notifier); struct amdgpu_device *adev =3D amdgpu_ttm_adev(bo->tbo.bdev); =20 + /* + * FIXME: Must hold some lock shared with + * amdgpu_ttm_tt_get_user_pages_done() + */ + mmu_range_set_seq(mrn, cur_seq); + /* FIXME: Is this necessary? */ if (!amdgpu_ttm_tt_affect_userptr(bo->tbo.ttm, range->start, range->end)) @@ -143,92 +135,6 @@ static const struct mmu_range_notifier_ops amdgpu_mn= _hsa_ops =3D { .invalidate =3D amdgpu_mn_invalidate_hsa, }; =20 -static int amdgpu_mn_sync_pagetables(struct hmm_mirror *mirror, - const struct mmu_notifier_range *update) -{ - struct amdgpu_mn *amn =3D container_of(mirror, struct amdgpu_mn, mirror= ); - - if (!mmu_notifier_range_blockable(update)) - return false; - - down_read(&amn->lock); - up_read(&amn->lock); - return 0; -} - -/* Low bits of any reasonable mm pointer will be unused due to struct - * alignment. Use these bits to make a unique key from the mm pointer - * and notifier type. - */ -#define AMDGPU_MN_KEY(mm, type) ((unsigned long)(mm) + (type)) - -static struct hmm_mirror_ops amdgpu_hmm_mirror_ops[] =3D { - [AMDGPU_MN_TYPE_GFX] =3D { - .sync_cpu_device_pagetables =3D amdgpu_mn_sync_pagetables, - }, - [AMDGPU_MN_TYPE_HSA] =3D { - .sync_cpu_device_pagetables =3D amdgpu_mn_sync_pagetables, - }, -}; - -/** - * amdgpu_mn_get - create HMM mirror context - * - * @adev: amdgpu device pointer - * @type: type of MMU notifier context - * - * Creates a HMM mirror context for current->mm. - */ -struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, - enum amdgpu_mn_type type) -{ - struct mm_struct *mm =3D current->mm; - struct amdgpu_mn *amn; - unsigned long key =3D AMDGPU_MN_KEY(mm, type); - int r; - - mutex_lock(&adev->mn_lock); - if (down_write_killable(&mm->mmap_sem)) { - mutex_unlock(&adev->mn_lock); - return ERR_PTR(-EINTR); - } - - hash_for_each_possible(adev->mn_hash, amn, node, key) - if (AMDGPU_MN_KEY(amn->mirror.hmm->mmu_notifier.mm, - amn->type) =3D=3D key) - goto release_locks; - - amn =3D kzalloc(sizeof(*amn), GFP_KERNEL); - if (!amn) { - amn =3D ERR_PTR(-ENOMEM); - goto release_locks; - } - - amn->adev =3D adev; - init_rwsem(&amn->lock); - amn->type =3D type; - - amn->mirror.ops =3D &amdgpu_hmm_mirror_ops[type]; - r =3D hmm_mirror_register(&amn->mirror, mm); - if (r) - goto free_amn; - - hash_add(adev->mn_hash, &amn->node, AMDGPU_MN_KEY(mm, type)); - -release_locks: - up_write(&mm->mmap_sem); - mutex_unlock(&adev->mn_lock); - - return amn; - -free_amn: - up_write(&mm->mmap_sem); - mutex_unlock(&adev->mn_lock); - kfree(amn); - - return ERR_PTR(r); -} - /** * amdgpu_mn_register - register a BO for notifier updates * @@ -263,25 +169,3 @@ void amdgpu_mn_unregister(struct amdgpu_bo *bo) mmu_range_notifier_remove(&bo->notifier); bo->notifier.mm =3D NULL; } - -/* flags used by HMM internal, not related to CPU/GPU PTE flags */ -static const uint64_t hmm_range_flags[HMM_PFN_FLAG_MAX] =3D { - (1 << 0), /* HMM_PFN_VALID */ - (1 << 1), /* HMM_PFN_WRITE */ - 0 /* HMM_PFN_DEVICE_PRIVATE */ -}; - -static const uint64_t hmm_range_values[HMM_PFN_VALUE_MAX] =3D { - 0xfffffffffffffffeUL, /* HMM_PFN_ERROR */ - 0, /* HMM_PFN_NONE */ - 0xfffffffffffffffcUL /* HMM_PFN_SPECIAL */ -}; - -void amdgpu_hmm_init_range(struct hmm_range *range) -{ - if (range) { - range->flags =3D hmm_range_flags; - range->values =3D hmm_range_values; - range->pfn_shift =3D PAGE_SHIFT; - } -} diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h b/drivers/gpu/drm/amd= /amdgpu/amdgpu_mn.h index d73ab2947b22b2..a292238f75ebae 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h @@ -30,59 +30,10 @@ #include #include =20 -enum amdgpu_mn_type { - AMDGPU_MN_TYPE_GFX, - AMDGPU_MN_TYPE_HSA, -}; - -/** - * struct amdgpu_mn - * - * @adev: amdgpu device pointer - * @type: type of MMU notifier - * @work: destruction work item - * @node: hash table node to find structure by adev and mn - * @lock: rw semaphore protecting the notifier nodes - * @mirror: HMM mirror function support - * - * Data for each amdgpu device and process address space. - */ -struct amdgpu_mn { - /* constant after initialisation */ - struct amdgpu_device *adev; - enum amdgpu_mn_type type; - - /* only used on destruction */ - struct work_struct work; - - /* protected by adev->mn_lock */ - struct hlist_node node; - - /* objects protected by lock */ - struct rw_semaphore lock; - -#ifdef CONFIG_HMM_MIRROR - /* HMM mirror */ - struct hmm_mirror mirror; -#endif -}; - #if defined(CONFIG_HMM_MIRROR) -void amdgpu_mn_lock(struct amdgpu_mn *mn); -void amdgpu_mn_unlock(struct amdgpu_mn *mn); -struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, - enum amdgpu_mn_type type); int amdgpu_mn_register(struct amdgpu_bo *bo, unsigned long addr); void amdgpu_mn_unregister(struct amdgpu_bo *bo); -void amdgpu_hmm_init_range(struct hmm_range *range); #else -static inline void amdgpu_mn_lock(struct amdgpu_mn *mn) {} -static inline void amdgpu_mn_unlock(struct amdgpu_mn *mn) {} -static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev= , - enum amdgpu_mn_type type) -{ - return NULL; -} static inline int amdgpu_mn_register(struct amdgpu_bo *bo, unsigned long= addr) { DRM_WARN_ONCE("HMM_MIRROR kernel config option is not enabled, " diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/am= d/amdgpu/amdgpu_ttm.c index c0e41f1f0c2365..65d9824b54f2a9 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -773,6 +773,20 @@ struct amdgpu_ttm_tt { #endif }; =20 +#ifdef CONFIG_DRM_AMDGPU_USERPTR +/* flags used by HMM internal, not related to CPU/GPU PTE flags */ +static const uint64_t hmm_range_flags[HMM_PFN_FLAG_MAX] =3D { + (1 << 0), /* HMM_PFN_VALID */ + (1 << 1), /* HMM_PFN_WRITE */ + 0 /* HMM_PFN_DEVICE_PRIVATE */ +}; + +static const uint64_t hmm_range_values[HMM_PFN_VALUE_MAX] =3D { + 0xfffffffffffffffeUL, /* HMM_PFN_ERROR */ + 0, /* HMM_PFN_NONE */ + 0xfffffffffffffffcUL /* HMM_PFN_SPECIAL */ +}; + /** * amdgpu_ttm_tt_get_user_pages - get device accessible pages that back = user * memory and start HMM tracking CPU page table update @@ -780,29 +794,27 @@ struct amdgpu_ttm_tt { * Calling function must call amdgpu_ttm_tt_userptr_range_done() once an= d only * once afterwards to stop HMM tracking */ -#if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR) - -#define MAX_RETRY_HMM_RANGE_FAULT 16 - int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pag= es) { - struct hmm_mirror *mirror =3D bo->mn ? &bo->mn->mirror : NULL; struct ttm_tt *ttm =3D bo->tbo.ttm; struct amdgpu_ttm_tt *gtt =3D (void *)ttm; struct mm_struct *mm; + struct hmm_range *range; unsigned long start =3D gtt->userptr; struct vm_area_struct *vma; - struct hmm_range *range; unsigned long i; - uint64_t *pfns; int r =3D 0; =20 - if (unlikely(!mirror)) { - DRM_DEBUG_DRIVER("Failed to get hmm_mirror\n"); + mm =3D bo->notifier.mm; + if (unlikely(!mm)) { + DRM_DEBUG_DRIVER("BO is not registered?\n"); return -EFAULT; } =20 - mm =3D mirror->hmm->mmu_notifier.mm; + /* Another get_user_pages is running at the same time?? */ + if (WARN_ON(gtt->range)) + return -EFAULT; + if (!mmget_not_zero(mm)) /* Happens during process shutdown */ return -ESRCH; =20 @@ -811,30 +823,24 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *= bo, struct page **pages) r =3D -ENOMEM; goto out; } + range->notifier =3D &bo->notifier; + range->flags =3D hmm_range_flags; + range->values =3D hmm_range_values; + range->pfn_shift =3D PAGE_SHIFT; + range->start =3D bo->notifier.interval_tree.start; + range->end =3D bo->notifier.interval_tree.last + 1; + range->default_flags =3D hmm_range_flags[HMM_PFN_VALID]; + if (!amdgpu_ttm_tt_is_readonly(ttm)) + range->default_flags |=3D range->flags[HMM_PFN_WRITE]; =20 - pfns =3D kvmalloc_array(ttm->num_pages, sizeof(*pfns), GFP_KERNEL); - if (unlikely(!pfns)) { + range->pfns =3D kvmalloc_array(ttm->num_pages, sizeof(*range->pfns), + GFP_KERNEL); + if (unlikely(!range->pfns)) { r =3D -ENOMEM; goto out_free_ranges; } =20 - amdgpu_hmm_init_range(range); - range->default_flags =3D range->flags[HMM_PFN_VALID]; - range->default_flags |=3D amdgpu_ttm_tt_is_readonly(ttm) ? - 0 : range->flags[HMM_PFN_WRITE]; - range->pfn_flags_mask =3D 0; - range->pfns =3D pfns; - range->start =3D start; - range->end =3D start + ttm->num_pages * PAGE_SIZE; - - hmm_range_register(range, mirror); - - /* - * Just wait for range to be valid, safe to ignore return value as we - * will use the return value of hmm_range_fault() below under the - * mmap_sem to ascertain the validity of the range. - */ - hmm_range_wait_until_valid(range, HMM_RANGE_DEFAULT_TIMEOUT); + range->notifier_seq =3D mmu_range_read_begin(&bo->notifier); =20 down_read(&mm->mmap_sem); vma =3D find_vma(mm, start); @@ -855,10 +861,10 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *= bo, struct page **pages) goto out_free_pfns; =20 for (i =3D 0; i < ttm->num_pages; i++) { - pages[i] =3D hmm_device_entry_to_page(range, pfns[i]); + pages[i] =3D hmm_device_entry_to_page(range, range->pfns[i]); if (unlikely(!pages[i])) { pr_err("Page fault failed for pfn[%lu] =3D 0x%llx\n", - i, pfns[i]); + i, range->pfns[i]); r =3D -ENOMEM; =20 goto out_free_pfns; @@ -873,8 +879,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo= , struct page **pages) out_unlock: up_read(&mm->mmap_sem); out_free_pfns: - hmm_range_unregister(range); - kvfree(pfns); + kvfree(range->pfns); out_free_ranges: kfree(range); out: @@ -903,9 +908,8 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt = *ttm) "No user pages to check\n"); =20 if (gtt->range) { - r =3D hmm_range_valid(gtt->range); - hmm_range_unregister(gtt->range); - + r =3D mmu_range_read_retry(gtt->range->notifier, + gtt->range->notifier_seq); kvfree(gtt->range->pfns); kfree(gtt->range); gtt->range =3D NULL; --=20 2.23.0