From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5662C433E6 for ; Wed, 10 Feb 2021 00:28:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 965E064E58 for ; Wed, 10 Feb 2021 00:28:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234930AbhBJA2Q (ORCPT ); Tue, 9 Feb 2021 19:28:16 -0500 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:13699 "EHLO hqnvemgate25.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234149AbhBIUyK (ORCPT ); Tue, 9 Feb 2021 15:54:10 -0500 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 09 Feb 2021 12:53:28 -0800 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 9 Feb 2021 20:53:28 +0000 Received: from [10.2.50.67] (172.20.145.6) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 9 Feb 2021 20:53:28 +0000 Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau To: Daniel Vetter , Alistair Popple CC: Linux MM , Nouveau Dev , Ben Skeggs , "Andrew Morton" , Linux Doc Mailing List , Linux Kernel Mailing List , , dri-devel , Ralph Campbell , Jerome Glisse , Jason Gunthorpe References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> From: John Hubbard Message-ID: <57fe0deb-8bf6-d3ee-3545-11109e946528@nvidia.com> Date: Tue, 9 Feb 2021 12:53:27 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Thunderbird/85.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1612904008; bh=efTFNdu9GY93fBrYgfGo5YB09FPO44DdmrDbf3NcmA8=; h=Subject:To:CC:References:From:Message-ID:Date:User-Agent: MIME-Version:In-Reply-To:Content-Type:Content-Language: Content-Transfer-Encoding:X-Originating-IP:X-ClientProxiedBy; b=oOacMjOb06MbLt3MklBlfJmol0WiXwI5VrJQzUPy3VIRUABA721mwmtEk5laRaFdL 1TgYD7us6SRaillAdI1lcgGlIdyJek1dOTFVhDnyE2O3I+0OWr72QO1zqGgtsAlgZM XoXWC6bFgcqLN4YfI9Hq+NhihJS47c+q+V0qXeOTEgQtEwB92yoJZn89seCqJe43ky ex8WisDf0I2J97YumwelfGCA2keVVdcX0KJL5u61SZMitqoPN5SG/CCVlP3I7iPS1w ghlzljcQ2bKvx+UVLYtlzqvILA3AlyDKepX5K4+iznmkld1sNDTuw0sGpBLV8p9rc5 18FHL1oWI+WFA== Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2/9/21 5:37 AM, Daniel Vetter wrote: > On Tue, Feb 9, 2021 at 1:57 PM Alistair Popple wrote: >> >> On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: >>>> >>>> Recent changes to pin_user_pages() prevent the creation of pinned pages in >>>> ZONE_MOVABLE. This series allows pinned pages to be created in >> ZONE_MOVABLE >>>> as attempts to migrate may fail which would be fatal to userspace. >>>> >>>> In this case migration of the pinned page is unnecessary as the page can >> be >>>> unpinned at anytime by having the driver revoke atomic permission as it >>>> does for the migrate_to_ram() callback. However a method of calling this >>>> when memory needs to be moved has yet to be resolved so any discussion is >>>> welcome. >>> >>> Why do we need to pin for gpu atomics? You still have the callback for >>> cpu faults, so you >>> can move the page as needed, and hence a long-term pin sounds like the >>> wrong approach. >> >> Technically a real long term unmoveable pin isn't required, because as you say >> the page can be moved as needed at any time. However I needed some way of >> stopping the CPU page from being freed once the userspace mappings for it had >> been removed. Obviously I could have just used get_page() but from the >> perspective of page migration the result is much the same as a pin - a page >> which can't be moved because of the extra refcount. > > long term pin vs short term page reference aren't fully fleshed out. > But the rule more or less is: > - short term page reference: _must_ get released in finite time for > migration and other things, either because you have a callback, or > because it's just for direct I/O, which will complete. This means > short term pins will delay migration, but not foul it complete GPU atomic operations to sysmem are hard to categorize, because because application programmers could easily write programs that do a long series of atomic operations. Such a program would be a little weird, but it's hard to rule out. > > - long term pin: the page cannot be moved, all migration must fail. > Also this will have an impact on COW behaviour for fork (but not sure > where those patches are, John Hubbard will know). That would be Jason's commit 57efa1fe59576 ("mm/gup: prevent gup_fast from racing with COW during fork"), which is in linux-next 20201216. > > So I think for your use case here you want a) short term page > reference to make sure it doesn't disappear plus b) callback to make > sure migrate isn't blocked. > > Breaking ZONE_MOVEABLE with either allowing long term pins or failing > migrations because you don't release your short term page reference > isn't good. > >> The normal solution of registering an MMU notifier to unpin the page when it >> needs to be moved also doesn't work as the CPU page tables now point to the >> device-private page and hence the migration code won't call any invalidate >> notifiers for the CPU page. > > Yeah you need some other callback for migration on the page directly. > it's a bit awkward since there is one already for struct > address_space, but that's own by the address_space/page cache, not > HMM. So I think we need something else, maybe something for each > ZONE_DEVICE? > This direction sounds at least...possible. Using MMU notifiers instead of pins is definitely appealing. I'm not quite clear on the callback idea above, but overall it seems like taking advantage of the ZONE_DEVICE tracking of pages (without having to put anything additional in each struct page), could work. Additional notes or ideas here are definitely welcome. thanks, -- John Hubbard NVIDIA From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2E02C433DB for ; Tue, 9 Feb 2021 21:08:42 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 30EA464E6B for ; Tue, 9 Feb 2021 21:08:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 30EA464E6B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=nouveau-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B67E46E0C6; Tue, 9 Feb 2021 21:08:41 +0000 (UTC) X-Greylist: delayed 911 seconds by postgrey-1.36 at gabe; Tue, 09 Feb 2021 21:08:40 UTC Received: from hqnvemgate25.nvidia.com (hqnvemgate25.nvidia.com [216.228.121.64]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4366B6E0C6 for ; Tue, 9 Feb 2021 21:08:40 +0000 (UTC) Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 09 Feb 2021 12:53:28 -0800 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 9 Feb 2021 20:53:28 +0000 Received: from [10.2.50.67] (172.20.145.6) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 9 Feb 2021 20:53:28 +0000 To: Daniel Vetter , Alistair Popple References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> From: John Hubbard Message-ID: <57fe0deb-8bf6-d3ee-3545-11109e946528@nvidia.com> Date: Tue, 9 Feb 2021 12:53:27 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Thunderbird/85.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1612904008; bh=efTFNdu9GY93fBrYgfGo5YB09FPO44DdmrDbf3NcmA8=; h=Subject:To:CC:References:From:Message-ID:Date:User-Agent: MIME-Version:In-Reply-To:Content-Type:Content-Language: Content-Transfer-Encoding:X-Originating-IP:X-ClientProxiedBy; b=oOacMjOb06MbLt3MklBlfJmol0WiXwI5VrJQzUPy3VIRUABA721mwmtEk5laRaFdL 1TgYD7us6SRaillAdI1lcgGlIdyJek1dOTFVhDnyE2O3I+0OWr72QO1zqGgtsAlgZM XoXWC6bFgcqLN4YfI9Hq+NhihJS47c+q+V0qXeOTEgQtEwB92yoJZn89seCqJe43ky ex8WisDf0I2J97YumwelfGCA2keVVdcX0KJL5u61SZMitqoPN5SG/CCVlP3I7iPS1w ghlzljcQ2bKvx+UVLYtlzqvILA3AlyDKepX5K4+iznmkld1sNDTuw0sGpBLV8p9rc5 18FHL1oWI+WFA== Subject: Re: [Nouveau] [PATCH 0/9] Add support for SVM atomics in Nouveau X-BeenThere: nouveau@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Nouveau development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Gunthorpe , Ralph Campbell , Linux Doc Mailing List , Nouveau Dev , dri-devel , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, Linux MM , Ben Skeggs , Andrew Morton Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: nouveau-bounces@lists.freedesktop.org Sender: "Nouveau" On 2/9/21 5:37 AM, Daniel Vetter wrote: > On Tue, Feb 9, 2021 at 1:57 PM Alistair Popple wrote: >> >> On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: >>>> >>>> Recent changes to pin_user_pages() prevent the creation of pinned pages in >>>> ZONE_MOVABLE. This series allows pinned pages to be created in >> ZONE_MOVABLE >>>> as attempts to migrate may fail which would be fatal to userspace. >>>> >>>> In this case migration of the pinned page is unnecessary as the page can >> be >>>> unpinned at anytime by having the driver revoke atomic permission as it >>>> does for the migrate_to_ram() callback. However a method of calling this >>>> when memory needs to be moved has yet to be resolved so any discussion is >>>> welcome. >>> >>> Why do we need to pin for gpu atomics? You still have the callback for >>> cpu faults, so you >>> can move the page as needed, and hence a long-term pin sounds like the >>> wrong approach. >> >> Technically a real long term unmoveable pin isn't required, because as you say >> the page can be moved as needed at any time. However I needed some way of >> stopping the CPU page from being freed once the userspace mappings for it had >> been removed. Obviously I could have just used get_page() but from the >> perspective of page migration the result is much the same as a pin - a page >> which can't be moved because of the extra refcount. > > long term pin vs short term page reference aren't fully fleshed out. > But the rule more or less is: > - short term page reference: _must_ get released in finite time for > migration and other things, either because you have a callback, or > because it's just for direct I/O, which will complete. This means > short term pins will delay migration, but not foul it complete GPU atomic operations to sysmem are hard to categorize, because because application programmers could easily write programs that do a long series of atomic operations. Such a program would be a little weird, but it's hard to rule out. > > - long term pin: the page cannot be moved, all migration must fail. > Also this will have an impact on COW behaviour for fork (but not sure > where those patches are, John Hubbard will know). That would be Jason's commit 57efa1fe59576 ("mm/gup: prevent gup_fast from racing with COW during fork"), which is in linux-next 20201216. > > So I think for your use case here you want a) short term page > reference to make sure it doesn't disappear plus b) callback to make > sure migrate isn't blocked. > > Breaking ZONE_MOVEABLE with either allowing long term pins or failing > migrations because you don't release your short term page reference > isn't good. > >> The normal solution of registering an MMU notifier to unpin the page when it >> needs to be moved also doesn't work as the CPU page tables now point to the >> device-private page and hence the migration code won't call any invalidate >> notifiers for the CPU page. > > Yeah you need some other callback for migration on the page directly. > it's a bit awkward since there is one already for struct > address_space, but that's own by the address_space/page cache, not > HMM. So I think we need something else, maybe something for each > ZONE_DEVICE? > This direction sounds at least...possible. Using MMU notifiers instead of pins is definitely appealing. I'm not quite clear on the callback idea above, but overall it seems like taking advantage of the ZONE_DEVICE tracking of pages (without having to put anything additional in each struct page), could work. Additional notes or ideas here are definitely welcome. thanks, -- John Hubbard NVIDIA _______________________________________________ Nouveau mailing list Nouveau@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/nouveau From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4664BC433E0 for ; Tue, 9 Feb 2021 20:53:31 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AD5BD64E30 for ; Tue, 9 Feb 2021 20:53:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AD5BD64E30 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E6A9E6E0CB; Tue, 9 Feb 2021 20:53:29 +0000 (UTC) Received: from hqnvemgate25.nvidia.com (hqnvemgate25.nvidia.com [216.228.121.64]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0FD8B6E0CB for ; Tue, 9 Feb 2021 20:53:29 +0000 (UTC) Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 09 Feb 2021 12:53:28 -0800 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 9 Feb 2021 20:53:28 +0000 Received: from [10.2.50.67] (172.20.145.6) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 9 Feb 2021 20:53:28 +0000 Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau To: Daniel Vetter , Alistair Popple References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> From: John Hubbard Message-ID: <57fe0deb-8bf6-d3ee-3545-11109e946528@nvidia.com> Date: Tue, 9 Feb 2021 12:53:27 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Thunderbird/85.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1612904008; bh=efTFNdu9GY93fBrYgfGo5YB09FPO44DdmrDbf3NcmA8=; h=Subject:To:CC:References:From:Message-ID:Date:User-Agent: MIME-Version:In-Reply-To:Content-Type:Content-Language: Content-Transfer-Encoding:X-Originating-IP:X-ClientProxiedBy; b=oOacMjOb06MbLt3MklBlfJmol0WiXwI5VrJQzUPy3VIRUABA721mwmtEk5laRaFdL 1TgYD7us6SRaillAdI1lcgGlIdyJek1dOTFVhDnyE2O3I+0OWr72QO1zqGgtsAlgZM XoXWC6bFgcqLN4YfI9Hq+NhihJS47c+q+V0qXeOTEgQtEwB92yoJZn89seCqJe43ky ex8WisDf0I2J97YumwelfGCA2keVVdcX0KJL5u61SZMitqoPN5SG/CCVlP3I7iPS1w ghlzljcQ2bKvx+UVLYtlzqvILA3AlyDKepX5K4+iznmkld1sNDTuw0sGpBLV8p9rc5 18FHL1oWI+WFA== X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Gunthorpe , Ralph Campbell , Linux Doc Mailing List , Nouveau Dev , dri-devel , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, Linux MM , Jerome Glisse , Ben Skeggs , Andrew Morton Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On 2/9/21 5:37 AM, Daniel Vetter wrote: > On Tue, Feb 9, 2021 at 1:57 PM Alistair Popple wrote: >> >> On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: >>>> >>>> Recent changes to pin_user_pages() prevent the creation of pinned pages in >>>> ZONE_MOVABLE. This series allows pinned pages to be created in >> ZONE_MOVABLE >>>> as attempts to migrate may fail which would be fatal to userspace. >>>> >>>> In this case migration of the pinned page is unnecessary as the page can >> be >>>> unpinned at anytime by having the driver revoke atomic permission as it >>>> does for the migrate_to_ram() callback. However a method of calling this >>>> when memory needs to be moved has yet to be resolved so any discussion is >>>> welcome. >>> >>> Why do we need to pin for gpu atomics? You still have the callback for >>> cpu faults, so you >>> can move the page as needed, and hence a long-term pin sounds like the >>> wrong approach. >> >> Technically a real long term unmoveable pin isn't required, because as you say >> the page can be moved as needed at any time. However I needed some way of >> stopping the CPU page from being freed once the userspace mappings for it had >> been removed. Obviously I could have just used get_page() but from the >> perspective of page migration the result is much the same as a pin - a page >> which can't be moved because of the extra refcount. > > long term pin vs short term page reference aren't fully fleshed out. > But the rule more or less is: > - short term page reference: _must_ get released in finite time for > migration and other things, either because you have a callback, or > because it's just for direct I/O, which will complete. This means > short term pins will delay migration, but not foul it complete GPU atomic operations to sysmem are hard to categorize, because because application programmers could easily write programs that do a long series of atomic operations. Such a program would be a little weird, but it's hard to rule out. > > - long term pin: the page cannot be moved, all migration must fail. > Also this will have an impact on COW behaviour for fork (but not sure > where those patches are, John Hubbard will know). That would be Jason's commit 57efa1fe59576 ("mm/gup: prevent gup_fast from racing with COW during fork"), which is in linux-next 20201216. > > So I think for your use case here you want a) short term page > reference to make sure it doesn't disappear plus b) callback to make > sure migrate isn't blocked. > > Breaking ZONE_MOVEABLE with either allowing long term pins or failing > migrations because you don't release your short term page reference > isn't good. > >> The normal solution of registering an MMU notifier to unpin the page when it >> needs to be moved also doesn't work as the CPU page tables now point to the >> device-private page and hence the migration code won't call any invalidate >> notifiers for the CPU page. > > Yeah you need some other callback for migration on the page directly. > it's a bit awkward since there is one already for struct > address_space, but that's own by the address_space/page cache, not > HMM. So I think we need something else, maybe something for each > ZONE_DEVICE? > This direction sounds at least...possible. Using MMU notifiers instead of pins is definitely appealing. I'm not quite clear on the callback idea above, but overall it seems like taking advantage of the ZONE_DEVICE tracking of pages (without having to put anything additional in each struct page), could work. Additional notes or ideas here are definitely welcome. thanks, -- John Hubbard NVIDIA _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: John Hubbard Date: Tue, 09 Feb 2021 20:53:27 +0000 Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau Message-Id: <57fe0deb-8bf6-d3ee-3545-11109e946528@nvidia.com> List-Id: References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Daniel Vetter , Alistair Popple Cc: Linux MM , Nouveau Dev , Ben Skeggs , Andrew Morton , Linux Doc Mailing List , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, dri-devel , Ralph Campbell , Jerome Glisse , Jason Gunthorpe On 2/9/21 5:37 AM, Daniel Vetter wrote: > On Tue, Feb 9, 2021 at 1:57 PM Alistair Popple wrote: >> >> On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: >>>> >>>> Recent changes to pin_user_pages() prevent the creation of pinned pages in >>>> ZONE_MOVABLE. This series allows pinned pages to be created in >> ZONE_MOVABLE >>>> as attempts to migrate may fail which would be fatal to userspace. >>>> >>>> In this case migration of the pinned page is unnecessary as the page can >> be >>>> unpinned at anytime by having the driver revoke atomic permission as it >>>> does for the migrate_to_ram() callback. However a method of calling this >>>> when memory needs to be moved has yet to be resolved so any discussion is >>>> welcome. >>> >>> Why do we need to pin for gpu atomics? You still have the callback for >>> cpu faults, so you >>> can move the page as needed, and hence a long-term pin sounds like the >>> wrong approach. >> >> Technically a real long term unmoveable pin isn't required, because as you say >> the page can be moved as needed at any time. However I needed some way of >> stopping the CPU page from being freed once the userspace mappings for it had >> been removed. Obviously I could have just used get_page() but from the >> perspective of page migration the result is much the same as a pin - a page >> which can't be moved because of the extra refcount. > > long term pin vs short term page reference aren't fully fleshed out. > But the rule more or less is: > - short term page reference: _must_ get released in finite time for > migration and other things, either because you have a callback, or > because it's just for direct I/O, which will complete. This means > short term pins will delay migration, but not foul it complete GPU atomic operations to sysmem are hard to categorize, because because application programmers could easily write programs that do a long series of atomic operations. Such a program would be a little weird, but it's hard to rule out. > > - long term pin: the page cannot be moved, all migration must fail. > Also this will have an impact on COW behaviour for fork (but not sure > where those patches are, John Hubbard will know). That would be Jason's commit 57efa1fe59576 ("mm/gup: prevent gup_fast from racing with COW during fork"), which is in linux-next 20201216. > > So I think for your use case here you want a) short term page > reference to make sure it doesn't disappear plus b) callback to make > sure migrate isn't blocked. > > Breaking ZONE_MOVEABLE with either allowing long term pins or failing > migrations because you don't release your short term page reference > isn't good. > >> The normal solution of registering an MMU notifier to unpin the page when it >> needs to be moved also doesn't work as the CPU page tables now point to the >> device-private page and hence the migration code won't call any invalidate >> notifiers for the CPU page. > > Yeah you need some other callback for migration on the page directly. > it's a bit awkward since there is one already for struct > address_space, but that's own by the address_space/page cache, not > HMM. So I think we need something else, maybe something for each > ZONE_DEVICE? > This direction sounds at least...possible. Using MMU notifiers instead of pins is definitely appealing. I'm not quite clear on the callback idea above, but overall it seems like taking advantage of the ZONE_DEVICE tracking of pages (without having to put anything additional in each struct page), could work. Additional notes or ideas here are definitely welcome. thanks, -- John Hubbard NVIDIA