From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7A9CC433E6 for ; Wed, 10 Feb 2021 12:59:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5DA1764E3B for ; Wed, 10 Feb 2021 12:59:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5DA1764E3B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D7AEE6B0006; Wed, 10 Feb 2021 07:59:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D52C86B006E; Wed, 10 Feb 2021 07:59:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C67E26B0070; Wed, 10 Feb 2021 07:59:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0095.hostedemail.com [216.40.44.95]) by kanga.kvack.org (Postfix) with ESMTP id B199D6B0006 for ; Wed, 10 Feb 2021 07:59:37 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 813C712661 for ; Wed, 10 Feb 2021 12:59:37 +0000 (UTC) X-FDA: 77802364794.04.need78_4214b7127610 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 5FFB98071952 for ; Wed, 10 Feb 2021 12:59:37 +0000 (UTC) X-HE-Tag: need78_4214b7127610 X-Filterd-Recvd-Size: 9399 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Feb 2021 12:59:36 +0000 (UTC) Received: by mail-wr1-f47.google.com with SMTP id b3so2430003wrj.5 for ; Wed, 10 Feb 2021 04:59:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:in-reply-to; bh=6MsHCb96PmdEoxri8X3RrE2xw2rPHnbybff/JS7t4rM=; b=Tstgim+g0HY1P/mkk3+/u2VrYZzdm9S0YfCCOW5tUOYsB36CYieEHqqX4gLAN0dvYl HwuWr0Ii0ZR/atXuP6UXJ5kI/MFr3fELyAgmJG9LEiHWGssayf7qVtCiTYZ270jOoGKa 40lRsXcevUNvgMYcsuDv15Ykj6bDqL5ycyu64= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to; bh=6MsHCb96PmdEoxri8X3RrE2xw2rPHnbybff/JS7t4rM=; b=tkdEFuDwDwl/31b2pYXSo3c8zqGiU9WoaGFGwby9s63PR3ZWsXkp6jbgd091It4DTm Vz6t8wpwKHeLrNP1pJ9V1YP3oUPNjYHGHUzcbsIYYLjsyoqALLUoDEUZ4sZAjQBycBAv 3XHe0i2Gg2wC2Va3BOxzr8rvGDonTWC+Grx/UD2fUKSnLbVwmTRSt/yvDmjwoqHc0BJO yiMkBV1eaA/DY7M1Q6J71u5C1gnWQCilgs7baKFm9zDB9/7ZGCLYKkSIirwN5grQQmCf 90L9OpWwn2uRTUGMhH0l9QlQn4axO9RAsm4cl8TvtbNKlg5bvovTgGnfY6rX7RTVnnD/ 7unA== X-Gm-Message-State: AOAM530TDyCLV4DfykjjvwKxCUtpLh+XjJBGSFAjUGiisLrV+VaFKqfX 0wnrHDfkuY54zS6OYA+3zRxKpw== X-Google-Smtp-Source: ABdhPJwRmPaxUgrAFJhjeq8xi1cVj/iHb/OYEDnTnx72e9/77v9DlCR5ZxyjtYtC8tHGuc5C0xMUTQ== X-Received: by 2002:a5d:4211:: with SMTP id n17mr3539933wrq.37.1612961975096; Wed, 10 Feb 2021 04:59:35 -0800 (PST) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id d10sm2955713wrn.88.2021.02.10.04.59.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Feb 2021 04:59:34 -0800 (PST) Date: Wed, 10 Feb 2021 13:59:32 +0100 From: Daniel Vetter To: John Hubbard Cc: Daniel Vetter , Alistair Popple , Linux MM , Nouveau Dev , Ben Skeggs , Andrew Morton , Linux Doc Mailing List , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, dri-devel , Ralph Campbell , Jerome Glisse , Jason Gunthorpe Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau Message-ID: Mail-Followup-To: John Hubbard , Alistair Popple , Linux MM , Nouveau Dev , Ben Skeggs , Andrew Morton , Linux Doc Mailing List , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, dri-devel , Ralph Campbell , Jerome Glisse , Jason Gunthorpe References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> <57fe0deb-8bf6-d3ee-3545-11109e946528@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <57fe0deb-8bf6-d3ee-3545-11109e946528@nvidia.com> X-Operating-System: Linux phenom 5.7.0-1-amd64 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 09, 2021 at 12:53:27PM -0800, John Hubbard wrote: > On 2/9/21 5:37 AM, Daniel Vetter wrote: > > On Tue, Feb 9, 2021 at 1:57 PM Alistair Popple wrote: > > > > > > On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: > > > > > > > > > > Recent changes to pin_user_pages() prevent the creation of pinned pages in > > > > > ZONE_MOVABLE. This series allows pinned pages to be created in > > > ZONE_MOVABLE > > > > > as attempts to migrate may fail which would be fatal to userspace. > > > > > > > > > > In this case migration of the pinned page is unnecessary as the page can > > > be > > > > > unpinned at anytime by having the driver revoke atomic permission as it > > > > > does for the migrate_to_ram() callback. However a method of calling this > > > > > when memory needs to be moved has yet to be resolved so any discussion is > > > > > welcome. > > > > > > > > Why do we need to pin for gpu atomics? You still have the callback for > > > > cpu faults, so you > > > > can move the page as needed, and hence a long-term pin sounds like the > > > > wrong approach. > > > > > > Technically a real long term unmoveable pin isn't required, because as you say > > > the page can be moved as needed at any time. However I needed some way of > > > stopping the CPU page from being freed once the userspace mappings for it had > > > been removed. Obviously I could have just used get_page() but from the > > > perspective of page migration the result is much the same as a pin - a page > > > which can't be moved because of the extra refcount. > > > > long term pin vs short term page reference aren't fully fleshed out. > > But the rule more or less is: > > - short term page reference: _must_ get released in finite time for > > migration and other things, either because you have a callback, or > > because it's just for direct I/O, which will complete. This means > > short term pins will delay migration, but not foul it complete > > > GPU atomic operations to sysmem are hard to categorize, because because application > programmers could easily write programs that do a long series of atomic operations. > Such a program would be a little weird, but it's hard to rule out. Yeah, but we can forcefully break this whenever we feel like by revoking the page, moving it, and then reinstating the gpu pte again and let it continue. If that's no possible then what we need here instead is an mlock() type of thing I think. > > - long term pin: the page cannot be moved, all migration must fail. > > Also this will have an impact on COW behaviour for fork (but not sure > > where those patches are, John Hubbard will know). > > > That would be Jason's commit 57efa1fe59576 ("mm/gup: prevent gup_fast from racing > with COW during fork"), which is in linux-next 20201216. Nice, thanks for the pointer. > > > > > > So I think for your use case here you want a) short term page > > reference to make sure it doesn't disappear plus b) callback to make > > sure migrate isn't blocked. > > > > Breaking ZONE_MOVEABLE with either allowing long term pins or failing > > migrations because you don't release your short term page reference > > isn't good. > > > > > The normal solution of registering an MMU notifier to unpin the page when it > > > needs to be moved also doesn't work as the CPU page tables now point to the > > > device-private page and hence the migration code won't call any invalidate > > > notifiers for the CPU page. > > > > Yeah you need some other callback for migration on the page directly. > > it's a bit awkward since there is one already for struct > > address_space, but that's own by the address_space/page cache, not > > HMM. So I think we need something else, maybe something for each > > ZONE_DEVICE? > > > > This direction sounds at least...possible. Using MMU notifiers instead of pins > is definitely appealing. I'm not quite clear on the callback idea above, but > overall it seems like taking advantage of the ZONE_DEVICE tracking of pages > (without having to put anything additional in each struct page), could work. > > Additional notes or ideas here are definitely welcome. Well I don't have ideas for the details tbh, just from the little I learned about how this all fits together pretending to be a pin while pretending to not be a pin just gets us back to the mess we're trying to solve with gup vs pup cleanup. And given that the pin_user_pages rollout hasn't even completed yet it's maybe way to early to already toss it out again. I think overall we should try really hard to not mix up things between memory that's pinned and memory that can be moved. Because retroactively trying to fix things up just because it was easier to get a feature going that way is very, very hard. And I think we've demonstrated countless times that "ah we just fix this with pinning, it doesn't happen often, no one will notice" always comes back to bite us later on :-) Like just looking ahead, 1GB pages are a thing, we will have to support that, migrate_page is the only way to get there, and statistically just a 1/256th chance of encountering a pinned page guarantees we'll never assemble a giant page. Cheers, Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch