From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3B42C433E0 for ; Thu, 25 Jun 2020 17:25:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AEC4520773 for ; Thu, 25 Jun 2020 17:25:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="XRF2xro5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AEC4520773 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2E8536B009A; Thu, 25 Jun 2020 13:25:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 299546B009B; Thu, 25 Jun 2020 13:25:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AF616B009C; Thu, 25 Jun 2020 13:25:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id 049026B009A for ; Thu, 25 Jun 2020 13:25:56 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C532B8C87E for ; Thu, 25 Jun 2020 17:25:56 +0000 (UTC) X-FDA: 76968411912.11.box82_1401ddc26e4e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 8FA0F1808187F for ; Thu, 25 Jun 2020 17:25:51 +0000 (UTC) X-HE-Tag: box82_1401ddc26e4e X-Filterd-Recvd-Size: 6166 Received: from hqnvemgate24.nvidia.com (hqnvemgate24.nvidia.com [216.228.121.143]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Thu, 25 Jun 2020 17:25:50 +0000 (UTC) Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 25 Jun 2020 10:24:15 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 25 Jun 2020 10:25:49 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 25 Jun 2020 10:25:49 -0700 Received: from rcampbell-dev.nvidia.com (172.20.13.39) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 25 Jun 2020 17:25:39 +0000 Subject: Re: [RESEND PATCH 2/3] nouveau: fix mixed normal and device private page migration From: Ralph Campbell To: Christoph Hellwig CC: , , "Jerome Glisse" , John Hubbard , "Jason Gunthorpe" , Ben Skeggs , Andrew Morton , , Bharata B Rao References: <20200622233854.10889-1-rcampbell@nvidia.com> <20200622233854.10889-3-rcampbell@nvidia.com> <20200624072355.GB18609@lst.de> <330f6a82-d01d-db97-1dec-69346f41e707@nvidia.com> X-Nvconfidentiality: public Message-ID: Date: Thu, 25 Jun 2020 10:25:38 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: <330f6a82-d01d-db97-1dec-69346f41e707@nvidia.com> X-Originating-IP: [172.20.13.39] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1593105855; bh=uYfK6rsz5+ealU9/4gfA5Q+t0zfXrSyYkvAq9Mm4eGo=; h=X-PGP-Universal:Subject:From:To:CC:References:X-Nvconfidentiality: Message-ID:Date:User-Agent:MIME-Version:In-Reply-To: X-Originating-IP:X-ClientProxiedBy:Content-Type:Content-Language: Content-Transfer-Encoding; b=XRF2xro5q/8nFW0qjYwB2C4HyBRCQPyCPqDcMZofasBrICeXUutwqnsZyPduCtLpq /ggcpTLFzaRb8MUCEsmqC8yYPn/2pY+ZgjZwdi3UDopmO98V/URpD7d9BSuxWKPA+G 2nWK0ffA0twlTJi0w2nlPSyqlw9LesqC01C2zXdacZOl5IX+OEzPnbN5JxKfO+3mNq oLvpus8nOYzAdTC4mzG5HtY6eVcuvTk4GK+8EAbum/RRJFnP5wOzkH//3bk+0krJvj HKpTZHcZhZjtaWE+Kb3QksgDgj0vokNgKMFr+K5B2UagSY1t9LsD44MzthqGXgXuBU T0TO8BMY5JjBg== X-Rspamd-Queue-Id: 8FA0F1808187F X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Making sure to include linux-mm and Bharata B Rao for IBM's use of migrate_vma*(). On 6/24/20 11:10 AM, Ralph Campbell wrote: >=20 > On 6/24/20 12:23 AM, Christoph Hellwig wrote: >> On Mon, Jun 22, 2020 at 04:38:53PM -0700, Ralph Campbell wrote: >>> The OpenCL function clEnqueueSVMMigrateMem(), without any flags, will >>> migrate memory in the given address range to device private memory. The >>> source pages might already have been migrated to device private memory. >>> In that case, the source struct page is not checked to see if it is >>> a device private page and incorrectly computes the GPU's physical >>> address of local memory leading to data corruption. >>> Fix this by checking the source struct page and computing the correct >>> physical address. >> >> I'm really worried about all this delicate code to fix the mixed >> ranges.=C2=A0 Can't we make it clear at the migrate_vma_* level if we wa= nt >> to migrate from or two device private memory, and then skip all the work >> for regions of memory that already are in the right place?=C2=A0 This mi= ght be >> a little more work initially, but I think it leads to a much better >> API. >> >=20 > The current code does encode the direction with src_owner !=3D NULL meani= ng > device private to system memory and src_owner =3D=3D NULL meaning system > memory to device private memory. This patch would obviously defeat that > so perhaps a flag could be added to the struct migrate_vma to indicate th= e > direction but I'm unclear how that makes things less delicate. > Can you expand on what you are worried about? >=20 > The issue with invalidations might be better addressed by letting the dev= ice > driver handle device private page TLB invalidations when migrating to > system memory and changing migrate_vma_setup() to only invalidate CPU > TLB entries for normal pages being migrated to device private memory. > If a page isn't migrating, it seems inefficient to invalidate those TLB > entries. >=20 > Any other suggestions? After a night's sleep, I think this might work. What do others think? 1) Add a new MMU_NOTIFY_MIGRATE enum to mmu_notifier_event. 2) Change migrate_vma_collect() to use the new MMU_NOTIFY_MIGRATE event typ= e. 3) Modify nouveau_svmm_invalidate_range_start() to simply return (no invali= dations) for MMU_NOTIFY_MIGRATE mmu notifier callbacks. 4) Leave the src_owner check in migrate_vma_collect_pmd() for normal pages = so if the device driver is migrating normal pages to device private memory, the drive= r would set src_owner =3D NULL and already migrated device private pages would be s= kipped. Since the mmu notifier callback did nothing, the device private entries rem= ain valid in the device's MMU. migrate_vma_collect_pmd() would still invalidate the C= PU page tables for migrated normal pages. If the driver is migrating device private pages to system memory, it would = set src_owner !=3D NULL, normal pages would be skipped, but now the device driv= er has to invalidate device MMU mappings in the "alloc and copy" before doing the cop= y. This would be after migrate_vma_setup() returns so the list of migrating de= vice pages is known to the driver. The rest of the migrate_vma_pages() and migrate_vma_finalize() remain the s= ame.