From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE9EBCA9EC1 for ; Mon, 28 Oct 2019 20:11:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8DB63208C0 for ; Mon, 28 Oct 2019 20:11:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="JDfbNHty" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8DB63208C0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6BB076B026A; Mon, 28 Oct 2019 16:10:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6434B6B026C; Mon, 28 Oct 2019 16:10:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55DA76B026D; Mon, 28 Oct 2019 16:10:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0161.hostedemail.com [216.40.44.161]) by kanga.kvack.org (Postfix) with ESMTP id 20E636B026A for ; Mon, 28 Oct 2019 16:10:52 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id ABF184DC9 for ; Mon, 28 Oct 2019 20:10:51 +0000 (UTC) X-FDA: 76094286702.26.gold15_41a6b3475204a X-HE-Tag: gold15_41a6b3475204a X-Filterd-Recvd-Size: 14315 Received: from mail-qk1-f196.google.com (mail-qk1-f196.google.com [209.85.222.196]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Mon, 28 Oct 2019 20:10:51 +0000 (UTC) Received: by mail-qk1-f196.google.com with SMTP id e2so1216629qkn.5 for ; Mon, 28 Oct 2019 13:10:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PQZNLb2n/VZGpEmak/Qy43FMq9i6D/cb88Nb6FZ4HCc=; b=JDfbNHty1ny9u0tolzz+R0bxyhO0ollb7F5vPylbs5XjRPew3ElqUgkRpnO1KqyNHJ lbmxT+WbuV2iqQKttXAinvJg6mLlpZOBkINKsbyZqgOCdsmgbxz+ymyca8lTAo2EIudp RdrM54tOjIVl+x7WKO8GaCpeaojoHs5YLsjcx7foxAOaiA94okZYV8o6vQfQCwuPbxr0 yggvPWtLeAKJS+9waHFzTuYqiUemPetepD5ewVTHRTeKTa3PYZPmq7T889b0g2RSfWRi iL7jN1WDPg49bHl8qlfraAoRYGBU7o23Ymndp/Dt+umlDMRAc3zH7TPYUwjq23HTYvtx Z/OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PQZNLb2n/VZGpEmak/Qy43FMq9i6D/cb88Nb6FZ4HCc=; b=OZg47OI4Sgk5Y+T/cDSvAUmmjRVEXST1Dt48IFLKNXZMiZb6OAzXTtAvwHE/EIPA4c Dww9v/h1v6TC7mux48TR/b/tRCYzdsU5xmei6SaH37Dpp1IVeN9NZKFtecmVLUjjfDFK 9UHGyehoaiaZqIkFHCZUCdTgG8jt0pkVZPVkeaX7FlzehruTfwAbIuezcmOU/ECLoIS9 d7sEV0iaUXL8nLee30nHtDkotFeoq0POzZM4e7IAU0A38BVReBszkC9yXnXyPpsOsBXo FnrKxVscOt2dZd0OKIvnW+eL7Oqj83laZDqLRLw0vYSuazKCJGnGgfPI64YDWJTBoFOF zNrg== X-Gm-Message-State: APjAAAU2oH3B/KZwXcpKTQ/SY71qzUxFm/TfR7mTzYuHKE1uNY8p4l1x Mpr3OqJsJfPCmdPGQvVJm3kIKRWo/r0= X-Google-Smtp-Source: APXvYqzMU7VQvTju2rb3w6e+2fL+RjV7hRaY2VBwnJrbzaMQL58Q1wBtGQ3o/J/TSiYTcAN3ImKl5A== X-Received: by 2002:a37:6643:: with SMTP id a64mr18555491qkc.144.1572293450333; Mon, 28 Oct 2019 13:10:50 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id u9sm1501115qke.50.2019.10.28.13.10.43 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 28 Oct 2019 13:10:46 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iPBLf-0001h3-HR; Mon, 28 Oct 2019 17:10:43 -0300 From: Jason Gunthorpe To: linux-mm@kvack.org, Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Alex Deucher , Ben Skeggs , Boris Ostrovsky , =?UTF-8?q?Christian=20K=C3=B6nig?= , David Zhou , Dennis Dalessandro , Juergen Gross , Mike Marciniszyn , Oleksandr Andrushchenko , Petr Cvek , Stefano Stabellini , nouveau@lists.freedesktop.org, xen-devel@lists.xenproject.org, Christoph Hellwig , Jason Gunthorpe Subject: [PATCH v2 11/15] nouveau: use mmu_range_notifier instead of hmm_mirror Date: Mon, 28 Oct 2019 17:10:28 -0300 Message-Id: <20191028201032.6352-12-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191028201032.6352-1-jgg@ziepe.ca> References: <20191028201032.6352-1-jgg@ziepe.ca> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jason Gunthorpe Remove the hmm_mirror object and use the mmu_range_notifier API instead for the range, and use the normal mmu_notifier API for the general invalidation callback. While here re-organize the pagefault path so the locking pattern is clear= . nouveau is the only driver that uses a temporary range object and instead forwards nearly every invalidation range directly to the HW. While this i= s not how the mmu_range_notifier was intended to be used, the overheads on the pagefaulting path are similar to the existing hmm_mirror version. Particularly since the interval tree will be small. Cc: Ben Skeggs Cc: dri-devel@lists.freedesktop.org Cc: nouveau@lists.freedesktop.org Cc: Ralph Campbell Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/nouveau/nouveau_svm.c | 180 ++++++++++++++------------ 1 file changed, 100 insertions(+), 80 deletions(-) diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouv= eau/nouveau_svm.c index 577f8811925a59..f27317fbe36f45 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.c +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c @@ -96,8 +96,6 @@ struct nouveau_svmm { } unmanaged; =20 struct mutex mutex; - - struct hmm_mirror mirror; }; =20 #define SVMM_DBG(s,f,a...) = \ @@ -293,23 +291,11 @@ static const struct mmu_notifier_ops nouveau_mn_ops= =3D { .free_notifier =3D nouveau_svmm_free_notifier, }; =20 -static int -nouveau_svmm_sync_cpu_device_pagetables(struct hmm_mirror *mirror, - const struct mmu_notifier_range *update) -{ - return 0; -} - -static const struct hmm_mirror_ops nouveau_svmm =3D { - .sync_cpu_device_pagetables =3D nouveau_svmm_sync_cpu_device_pagetables= , -}; - void nouveau_svmm_fini(struct nouveau_svmm **psvmm) { struct nouveau_svmm *svmm =3D *psvmm; if (svmm) { - hmm_mirror_unregister(&svmm->mirror); mutex_lock(&svmm->mutex); svmm->vmm =3D NULL; mutex_unlock(&svmm->mutex); @@ -357,15 +343,10 @@ nouveau_svmm_init(struct drm_device *dev, void *dat= a, goto out_free; =20 down_write(¤t->mm->mmap_sem); - svmm->mirror.ops =3D &nouveau_svmm; - ret =3D hmm_mirror_register(&svmm->mirror, current->mm); - if (ret) - goto out_mm_unlock; - svmm->notifier.ops =3D &nouveau_mn_ops; ret =3D __mmu_notifier_register(&svmm->notifier, current->mm); if (ret) - goto out_hmm_unregister; + goto out_mm_unlock; /* Note, ownership of svmm transfers to mmu_notifier */ =20 cli->svm.svmm =3D svmm; @@ -374,8 +355,6 @@ nouveau_svmm_init(struct drm_device *dev, void *data, mutex_unlock(&cli->mutex); return 0; =20 -out_hmm_unregister: - hmm_mirror_unregister(&svmm->mirror); out_mm_unlock: up_write(¤t->mm->mmap_sem); out_free: @@ -503,43 +482,91 @@ nouveau_svm_fault_cache(struct nouveau_svm *svm, fault->inst, fault->addr, fault->access); } =20 -static inline bool -nouveau_range_done(struct hmm_range *range) +struct svm_notifier { + struct mmu_range_notifier notifier; + struct nouveau_svmm *svmm; +}; + +static bool nouveau_svm_range_invalidate(struct mmu_range_notifier *mrn, + const struct mmu_notifier_range *range, + unsigned long cur_seq) { - bool ret =3D hmm_range_valid(range); + struct svm_notifier *sn =3D + container_of(mrn, struct svm_notifier, notifier); =20 - hmm_range_unregister(range); - return ret; + /* + * serializes the update to mrn->invalidate_seq done by caller and + * prevents invalidation of the PTE from progressing while HW is being + * programmed. This is very hacky and only works because the normal + * notifier that does invalidation is always called after the range + * notifier. + */ + if (mmu_notifier_range_blockable(range)) + mutex_lock(&sn->svmm->mutex); + else if (!mutex_trylock(&sn->svmm->mutex)) + return false; + mmu_range_set_seq(mrn, cur_seq); + mutex_unlock(&sn->svmm->mutex); + return true; } =20 -static int -nouveau_range_fault(struct nouveau_svmm *svmm, struct hmm_range *range) +static const struct mmu_range_notifier_ops nouveau_svm_mrn_ops =3D { + .invalidate =3D nouveau_svm_range_invalidate, +}; + +static int nouveau_range_fault(struct nouveau_svmm *svmm, + struct nouveau_drm *drm, void *data, u32 size, + u64 *pfns, + struct svm_notifier *notifier) { + unsigned long timeout =3D + jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + /* Have HMM fault pages within the fault window to the GPU. */ + struct hmm_range range =3D { + .notifier =3D ¬ifier->notifier, + .start =3D notifier->notifier.interval_tree.start, + .end =3D notifier->notifier.interval_tree.last + 1, + .pfns =3D pfns, + .flags =3D nouveau_svm_pfn_flags, + .values =3D nouveau_svm_pfn_values, + .pfn_shift =3D NVIF_VMM_PFNMAP_V0_ADDR_SHIFT, + }; + struct mm_struct *mm =3D notifier->notifier.mm; long ret; =20 - range->default_flags =3D 0; - range->pfn_flags_mask =3D -1UL; + while (true) { + if (time_after(jiffies, timeout)) + return -EBUSY; =20 - ret =3D hmm_range_register(range, &svmm->mirror); - if (ret) { - up_read(&svmm->notifier.mm->mmap_sem); - return (int)ret; - } + range.notifier_seq =3D mmu_range_read_begin(range.notifier); + range.default_flags =3D 0; + range.pfn_flags_mask =3D -1UL; + down_read(&mm->mmap_sem); + ret =3D hmm_range_fault(&range, 0); + up_read(&mm->mmap_sem); + if (ret <=3D 0) { + if (ret =3D=3D 0 || ret =3D=3D -EBUSY) + continue; + return ret; + } =20 - if (!hmm_range_wait_until_valid(range, HMM_RANGE_DEFAULT_TIMEOUT)) { - up_read(&svmm->notifier.mm->mmap_sem); - return -EBUSY; + mutex_lock(&svmm->mutex); + if (mmu_range_read_retry(range.notifier, + range.notifier_seq)) { + mutex_unlock(&svmm->mutex); + continue; + } + break; } =20 - ret =3D hmm_range_fault(range, 0); - if (ret <=3D 0) { - if (ret =3D=3D 0) - ret =3D -EBUSY; - up_read(&svmm->notifier.mm->mmap_sem); - hmm_range_unregister(range); - return ret; - } - return 0; + nouveau_dmem_convert_pfn(drm, &range); + + svmm->vmm->vmm.object.client->super =3D true; + ret =3D nvif_object_ioctl(&svmm->vmm->vmm.object, data, size, NULL); + svmm->vmm->vmm.object.client->super =3D false; + mutex_unlock(&svmm->mutex); + + return ret; } =20 static int @@ -559,7 +586,6 @@ nouveau_svm_fault(struct nvif_notify *notify) } i; u64 phys[16]; } args; - struct hmm_range range; struct vm_area_struct *vma; u64 inst, start, limit; int fi, fn, pi, fill; @@ -615,6 +641,7 @@ nouveau_svm_fault(struct nvif_notify *notify) args.i.p.version =3D 0; =20 for (fi =3D 0; fn =3D fi + 1, fi < buffer->fault_nr; fi =3D fn) { + struct svm_notifier notifier; struct mm_struct *mm; =20 /* Cancel any faults from non-SVM channels. */ @@ -623,7 +650,6 @@ nouveau_svm_fault(struct nvif_notify *notify) continue; } SVMM_DBG(svmm, "addr %016llx", buffer->fault[fi]->addr); - mm =3D svmm->notifier.mm; =20 /* We try and group handling of faults within a small * window into a single update. @@ -637,6 +663,12 @@ nouveau_svm_fault(struct nvif_notify *notify) start =3D max_t(u64, start, svmm->unmanaged.limit); SVMM_DBG(svmm, "wndw %016llx-%016llx", start, limit); =20 + mm =3D svmm->notifier.mm; + if (!mmget_not_zero(mm)) { + nouveau_svm_fault_cancel_fault(svm, buffer->fault[fi]); + continue; + } + /* Intersect fault window with the CPU VMA, cancelling * the fault if the address is invalid. */ @@ -645,16 +677,18 @@ nouveau_svm_fault(struct nvif_notify *notify) if (!vma) { SVMM_ERR(svmm, "wndw %016llx-%016llx", start, limit); up_read(&mm->mmap_sem); + mmput(mm); nouveau_svm_fault_cancel_fault(svm, buffer->fault[fi]); continue; } start =3D max_t(u64, start, vma->vm_start); limit =3D min_t(u64, limit, vma->vm_end); + up_read(&mm->mmap_sem); SVMM_DBG(svmm, "wndw %016llx-%016llx", start, limit); =20 if (buffer->fault[fi]->addr !=3D start) { SVMM_ERR(svmm, "addr %016llx", buffer->fault[fi]->addr); - up_read(&mm->mmap_sem); + mmput(mm); nouveau_svm_fault_cancel_fault(svm, buffer->fault[fi]); continue; } @@ -710,33 +744,19 @@ nouveau_svm_fault(struct nvif_notify *notify) args.i.p.addr, args.i.p.addr + args.i.p.size, fn - fi); =20 - /* Have HMM fault pages within the fault window to the GPU. */ - range.start =3D args.i.p.addr; - range.end =3D args.i.p.addr + args.i.p.size; - range.pfns =3D args.phys; - range.flags =3D nouveau_svm_pfn_flags; - range.values =3D nouveau_svm_pfn_values; - range.pfn_shift =3D NVIF_VMM_PFNMAP_V0_ADDR_SHIFT; -again: - ret =3D nouveau_range_fault(svmm, &range); - if (ret =3D=3D 0) { - mutex_lock(&svmm->mutex); - if (!nouveau_range_done(&range)) { - mutex_unlock(&svmm->mutex); - goto again; - } - - nouveau_dmem_convert_pfn(svm->drm, &range); - - svmm->vmm->vmm.object.client->super =3D true; - ret =3D nvif_object_ioctl(&svmm->vmm->vmm.object, - &args, sizeof(args.i) + - pi * sizeof(args.phys[0]), - NULL); - svmm->vmm->vmm.object.client->super =3D false; - mutex_unlock(&svmm->mutex); - up_read(&mm->mmap_sem); + notifier.svmm =3D svmm; + notifier.notifier.ops =3D &nouveau_svm_mrn_ops; + ret =3D mmu_range_notifier_insert(¬ifier.notifier, + args.i.p.addr, args.i.p.size, + svmm->notifier.mm); + if (!ret) { + ret =3D nouveau_range_fault( + svmm, svm->drm, &args, + sizeof(args.i) + pi * sizeof(args.phys[0]), + args.phys, ¬ifier); + mmu_range_notifier_remove(¬ifier.notifier); } + mmput(mm); =20 /* Cancel any faults in the window whose pages didn't manage * to keep their valid bit, or stay writeable when required. @@ -745,10 +765,10 @@ nouveau_svm_fault(struct nvif_notify *notify) */ while (fi < fn) { struct nouveau_svm_fault *fault =3D buffer->fault[fi++]; - pi =3D (fault->addr - range.start) >> PAGE_SHIFT; + pi =3D (fault->addr - args.i.p.addr) >> PAGE_SHIFT; if (ret || - !(range.pfns[pi] & NVIF_VMM_PFNMAP_V0_V) || - (!(range.pfns[pi] & NVIF_VMM_PFNMAP_V0_W) && + !(args.phys[pi] & NVIF_VMM_PFNMAP_V0_V) || + (!(args.phys[pi] & NVIF_VMM_PFNMAP_V0_W) && fault->access !=3D 0 && fault->access !=3D 3)) { nouveau_svm_fault_cancel_fault(svm, fault); continue; --=20 2.23.0