From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8FA6C433DB for ; Tue, 9 Feb 2021 10:27:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1247064EA6 for ; Tue, 9 Feb 2021 10:27:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1247064EA6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 629136B006E; Tue, 9 Feb 2021 05:27:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DB9A6B0070; Tue, 9 Feb 2021 05:27:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F0206B0071; Tue, 9 Feb 2021 05:27:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0081.hostedemail.com [216.40.44.81]) by kanga.kvack.org (Postfix) with ESMTP id 38ACF6B006E for ; Tue, 9 Feb 2021 05:27:20 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E81795DDA for ; Tue, 9 Feb 2021 10:27:19 +0000 (UTC) X-FDA: 77798352198.22.talk83_1e01a8a27606 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id C1796180B3C94 for ; Tue, 9 Feb 2021 10:27:19 +0000 (UTC) X-HE-Tag: talk83_1e01a8a27606 X-Filterd-Recvd-Size: 6839 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 10:27:18 +0000 (UTC) Received: by mail-oi1-f181.google.com with SMTP id y199so16868793oia.4 for ; Tue, 09 Feb 2021 02:27:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IuPwe6UnW2D70azMPi7ooQm/qzJFfsCFKJ5pbRf3Rag=; b=fjNiIqJN11AmOAs1Pa1q8noSSurhramxPWrN+UmCUCz19mWIDz3e7u/5LlRSNHdThF 3SEZ7MSpAtz9+pzxH2o0Dx3sbAmj0oYCQwbWPReA/QOLUf+HZ5ccnMsn7DBtkieJMPzL BA++OnfY3vQP93lMntxdlYu6pI2Pqm/Q0zTQA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IuPwe6UnW2D70azMPi7ooQm/qzJFfsCFKJ5pbRf3Rag=; b=nZDfh5p1hDrrYhtPRPxUwQGgNOT69wL580WWDEkxiChZKikhohaOEkZ8ldMn1CK3bM 7qqWP8yHkWhy7/pSR0tcONzvcrPpArG/uCa+H1Sv8A3kz6K3iYZGYhhocIRl8VgZO8GY V8xvCpo7fxOMMmIF52JWRGAaTWblHangaFYfDNZk3nuDwkqQkzLLw00ktIopSkYIgwyJ 370zALVuKIc2ACpEhFUcU03Hoi5t+xzyvHLU0GGfq1PmikxvkzjdjnPoY8bFcsN6Txh3 9OKlCotGe3tE+LKeRgAE3VBBzGOahm7bTFAXXaHQk4LGnmB/pOFy+8+qcFX6vf2D3iig 7LmQ== X-Gm-Message-State: AOAM530XZAdFUuMtoPX6Fr1J0K68GkbTJW8KhTrvL0+a85pq5S7jvftR 2aMPsG6IPzovUVXnxahVpQ1FsA+MdbX9tdy711wgRA== X-Google-Smtp-Source: ABdhPJyir5D67fhPcTVsC/9F/To+q3sh8YkD1lFXu3j/zAvynT6ofUoVqy6egtyN+Ya2IAdtuesS+PnTgi+/Evagt+A= X-Received: by 2002:aca:1906:: with SMTP id l6mr1967569oii.101.1612866437717; Tue, 09 Feb 2021 02:27:17 -0800 (PST) MIME-Version: 1.0 References: <20210209010722.13839-1-apopple@nvidia.com> In-Reply-To: <20210209010722.13839-1-apopple@nvidia.com> From: Daniel Vetter Date: Tue, 9 Feb 2021 11:27:05 +0100 Message-ID: Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau To: Alistair Popple Cc: Linux MM , Nouveau Dev , Ben Skeggs , Andrew Morton , Linux Doc Mailing List , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, dri-devel , John Hubbard , Ralph Campbell , Jerome Glisse , Jason Gunthorpe Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 09, 2021 at 12:07:13PM +1100, Alistair Popple wrote: > This series adds support to Nouveau for atomic memory operations on OpenCL > shared virtual memory (SVM). This is achieved using the atomic PTE bits on > the GPU to only permit atomic operations to system memory when a page is > not mapped in userspace on the CPU. > > This is implemented by adding a mode to migrate_vma_pages() which unmaps > and isolates existing pages from the CPU and pins them. The original > userspace page table entries are migrated to point to device private pages > allocated by the driver. This allows the driver to enable GPU atomic access > to the page as it will receive a callback when CPU userspace needs to > access it. > > In response to this callback the driver revokes the atomic access > permission from the GPU and migrates entries to point back to the original > page. The original page is unpinned as part of the migration operation > which also returns it to the LRU. > > Patch 3 contains the bulk of the memory management changes to implement > unmap and pin. > > Patches 6-9 extend Nouveau to use the new mode to allow system wide atomics > for OpenCL SVM to be implemented on Nouveau. > > This has been tested using the latest upstream Mesa userspace with a simple > OpenCL test program which checks the results of atomic GPU operations on a > buffer whilst also writing to the same buffer from the CPU. > > Problems yet to be addressed: > > Recent changes to pin_user_pages() prevent the creation of pinned pages in > ZONE_MOVABLE. This series allows pinned pages to be created in ZONE_MOVABLE > as attempts to migrate may fail which would be fatal to userspace. > > In this case migration of the pinned page is unnecessary as the page can be > unpinned at anytime by having the driver revoke atomic permission as it > does for the migrate_to_ram() callback. However a method of calling this > when memory needs to be moved has yet to be resolved so any discussion is > welcome. Why do we need to pin for gpu atomics? You still have the callback for cpu faults, so you can move the page as needed, and hence a long-term pin sounds like the wrong approach. That would avoid all the hacking around long term pin constraints, because for real unmoveable long term pinned memory we really want to have all these checks. So I think we might be missing some other callbacks to be able to move these pages, instead of abusing longterm pins for lack of better tools. Cheers, Daniel > > Alistair Popple (9): > mm/migrate.c: Always allow device private pages to migrate > mm/migrate.c: Allow pfn flags to be passed to migrate_vma_setup() > mm/migrate: Add a unmap and pin migration mode > Documentation: Add unmap and pin to HMM > hmm-tests: Add test for unmap and pin > nouveau/dmem: Only map migrating pages > nouveau/svm: Refactor nouveau_range_fault > nouveau/dmem: Add support for multiple page types > nouveau/svm: Implement atomic SVM access > > Documentation/vm/hmm.rst | 22 +- > arch/powerpc/kvm/book3s_hv_uvmem.c | 4 +- > drivers/gpu/drm/nouveau/include/nvif/if000c.h | 1 + > drivers/gpu/drm/nouveau/nouveau_dmem.c | 190 +++++++++++++++--- > drivers/gpu/drm/nouveau/nouveau_dmem.h | 9 + > drivers/gpu/drm/nouveau/nouveau_svm.c | 148 +++++++++++--- > drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h | 1 + > .../drm/nouveau/nvkm/subdev/mmu/vmmgp100.c | 6 + > include/linux/migrate.h | 2 + > include/linux/migrate_mode.h | 1 + > lib/test_hmm.c | 109 ++++++++-- > lib/test_hmm_uapi.h | 1 + > mm/migrate.c | 82 +++++--- > tools/testing/selftests/vm/hmm-tests.c | 49 +++++ > 14 files changed, 524 insertions(+), 101 deletions(-) > > -- > 2.20.1 > -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch