From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76610C433E0 for ; Tue, 9 Feb 2021 13:40:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 004E164DF0 for ; Tue, 9 Feb 2021 13:40:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 004E164DF0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9AAB76B0005; Tue, 9 Feb 2021 08:40:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 934F86B0078; Tue, 9 Feb 2021 08:40:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AFDC6B007B; Tue, 9 Feb 2021 08:40:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0023.hostedemail.com [216.40.44.23]) by kanga.kvack.org (Postfix) with ESMTP id 5F5EE6B0005 for ; Tue, 9 Feb 2021 08:40:04 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2281B8418 for ; Tue, 9 Feb 2021 13:40:04 +0000 (UTC) X-FDA: 77798837928.23.slope93_0a06bb627607 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 0292037606 for ; Tue, 9 Feb 2021 13:40:03 +0000 (UTC) X-HE-Tag: slope93_0a06bb627607 X-Filterd-Recvd-Size: 6065 Received: from mail-oo1-f52.google.com (mail-oo1-f52.google.com [209.85.161.52]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 13:40:03 +0000 (UTC) Received: by mail-oo1-f52.google.com with SMTP id y21so4256239oot.12 for ; Tue, 09 Feb 2021 05:40:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=LhfQbnhwDGpqGDadCiKHjxS8DmSjilEDHzg3CJ532pI=; b=JigqNdgDEFgtQlKMjnmdnlDlN7M4qkzWm5dhzF1dd992sb4eU3A75DIikN86A5yKF3 +HYqLOXv3f90n0bmYdNvleTeiQcUc5RdDT1mXps9fTyTu9OPEPFxGyjMaOdfAf5jGZOp LN5zBs74rUhwch5wHpsKumLLOkAYJemIb3dfs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=LhfQbnhwDGpqGDadCiKHjxS8DmSjilEDHzg3CJ532pI=; b=LwED5i3o+f5zhHH4w1DP7c43TVwIt/IWskk/oW7mH8CBKHBfSkhKjWvYDTv0SWBL1k uP6Xm4YczcqLkeVkcoz84O2FeiWqZoxlD5/5pKHfvoZucOdYWZ4IxgKAf+7ZrOEWclhl vFg95Hu+G8glD+djRalvgfCefo0no3+S8CoQyJUGYDans592PBTBAhRSVJPq6pBZH0iX 2ZIDZD25XqzD1fb3YFw6VsEp0ub0Y3xv5yX5odP0mF4k8oCl+iieOP+SIVTxYaGEVlln yV7q9Dh9t5zxTil5YHm6lRazESD/FxFFo3le35gGNpKx4kz3hV7nEQ6XkG3mJ8qlvEm2 xEww== X-Gm-Message-State: AOAM532JuunexA4AHFPNkVOHZHW4TZ0AlaH8HAV0hg5mBKUk2De86ryL LMsGcT8lqLp21e3kQ0t/MsTM07RClh7sjXDVwItfIw== X-Google-Smtp-Source: ABdhPJxCuVVi8kRB1kkxAZJFWPpa7yPv+iZrvRDXXQ3OzMMiP+Pm+I8NUjrdp4w6xRd7m67NamfzG5rvygCAv52MFRA= X-Received: by 2002:a4a:d891:: with SMTP id b17mr15851254oov.28.1612878002267; Tue, 09 Feb 2021 05:40:02 -0800 (PST) MIME-Version: 1.0 References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> <20210209133520.GB4718@ziepe.ca> In-Reply-To: <20210209133520.GB4718@ziepe.ca> From: Daniel Vetter Date: Tue, 9 Feb 2021 14:39:51 +0100 Message-ID: Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau To: Jason Gunthorpe Cc: Alistair Popple , Linux MM , Nouveau Dev , Ben Skeggs , Andrew Morton , Linux Doc Mailing List , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, dri-devel , John Hubbard , Ralph Campbell , Jerome Glisse Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 9, 2021 at 2:35 PM Jason Gunthorpe wrote: > > On Tue, Feb 09, 2021 at 11:57:28PM +1100, Alistair Popple wrote: > > On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: > > > > > > > > Recent changes to pin_user_pages() prevent the creation of pinned pages in > > > > ZONE_MOVABLE. This series allows pinned pages to be created in > > ZONE_MOVABLE > > > > as attempts to migrate may fail which would be fatal to userspace. > > > > > > > > In this case migration of the pinned page is unnecessary as the page can > > be > > > > unpinned at anytime by having the driver revoke atomic permission as it > > > > does for the migrate_to_ram() callback. However a method of calling this > > > > when memory needs to be moved has yet to be resolved so any discussion is > > > > welcome. > > > > > > Why do we need to pin for gpu atomics? You still have the callback for > > > cpu faults, so you > > > can move the page as needed, and hence a long-term pin sounds like the > > > wrong approach. > > > > Technically a real long term unmoveable pin isn't required, because as you say > > the page can be moved as needed at any time. However I needed some way of > > stopping the CPU page from being freed once the userspace mappings for it had > > been removed. > > The issue is you took the page out of the PTE it belongs to, which > makes it orphaned and unlocatable by the rest of the mm? > > Ideally this would leave the PTE in place so everything continues to > work, just disable CPU access to it. > > Maybe some kind of special swap entry? I probably should have read the patches more in detail, I was assuming the ZONE_DEVICE is only for vram. At least I thought the requirement for gpu atomics was that the page is in vram, but maybe I'm mixing up how this works on nvidia with how it works in other places. Iirc we had a long discussion about this at lpc19 that ended with the conclusion that we must be able to migrate, and sometimes migration is blocked. But the details ellude me now. Either way ZONE_DEVICE for not vram/device memory sounds wrong. Is that really going on here? -Daniel > > I also don't much like the use of ZONE_DEVICE here, that should only > be used for actual device memory, not as a temporary proxy for CPU > pages.. Having two struct pages refer to the same physical memory is > pretty ugly. > > > The normal solution of registering an MMU notifier to unpin the page when it > > needs to be moved also doesn't work as the CPU page tables now point to the > > device-private page and hence the migration code won't call any invalidate > > notifiers for the CPU page. > > The fact the page is lost from the MM seems to be the main issue here. > > > Yes, I would like to avoid the long term pin constraints as well if possible I > > just haven't found a solution yet. Are you suggesting it might be possible to > > add a callback in the page migration logic to specially deal with moving these > > pages? > > How would migration even find the page? > > Jason -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch