From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA692C433E1 for ; Thu, 25 Jun 2020 17:31:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9C05A20789 for ; Thu, 25 Jun 2020 17:31:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="k1ee41sa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9C05A20789 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 46CC08D000B; Thu, 25 Jun 2020 13:31:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 41F116B009F; Thu, 25 Jun 2020 13:31:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30D938D000B; Thu, 25 Jun 2020 13:31:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0163.hostedemail.com [216.40.44.163]) by kanga.kvack.org (Postfix) with ESMTP id 18A5A6B009E for ; Thu, 25 Jun 2020 13:31:49 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id CC5DF181CC1A1 for ; Thu, 25 Jun 2020 17:31:48 +0000 (UTC) X-FDA: 76968426696.24.juice79_36110bf26e4e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 33F31195C7F for ; Thu, 25 Jun 2020 17:31:46 +0000 (UTC) X-HE-Tag: juice79_36110bf26e4e X-Filterd-Recvd-Size: 6315 Received: from mail-qv1-f65.google.com (mail-qv1-f65.google.com [209.85.219.65]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Thu, 25 Jun 2020 17:31:45 +0000 (UTC) Received: by mail-qv1-f65.google.com with SMTP id dp10so3168172qvb.10 for ; Thu, 25 Jun 2020 10:31:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=itEeZ1F/lzuGzui0eFbAV1XPt3YLhZ3c66RORb7DwP8=; b=k1ee41sapeBv+Ec64wWauXUcVjlspfk/nCpFrSmyjqHWPbZOpen3DKVuzSLpEZhorb SNyCh4Wrg6Li7Mt/bqu1d93/TQSYUh3V84XlhBqp/w1zTZahN5ITFnxgUvcpC9F0KKsL LJLFmCjXYwzNu7O8ELdzpKKbIPsGCZ1MYU32SbHbrX1nUnaAVkfpwwCoUiP4Y8xhI0RO JYs9DYZnyUXg8xBcNbdKvBllo8rf5989i+mk+UBsB5R9jOhaY+c5WTAbM23JoenxfNQt yR04iqXNABRySelA7WVrXzwrHeiPg4/kuV/aU7jWS6xs+3DNY2vJL+9oBZGwdHz29bOb VrbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=itEeZ1F/lzuGzui0eFbAV1XPt3YLhZ3c66RORb7DwP8=; b=XcEkPz1t0aEZXGBEDjU+8lq0J3+fdpdDnB3jgx0qydJPC25EzBR3Od29ZmVcwMgj+m EHDOwCpuj4K7NCN+y404OYcw5tnCQPKyWlKG493VxpGdsmHv4Ddd4Z5oKCG14tkLia7y d1As8OwudgHiPKnHRAcraONyleNQNSAHXyyOvivw4GTFchqyKz1KMipNwyuelY1YuUsT /yvbWvDbEuZa5roqmeXabyRKVQTL3rayQCg8SbTFfFMCd81ZwWLykW0uttnE4SGdgQxU 1cuHZJmZcCQR4onwUjkSmM8BZlpfn4it6kzcskVERtHHgcW2hUC7h6pCnlUZCxirYMs1 2pVw== X-Gm-Message-State: AOAM53355hYn9gfKg46PKZFxjuA4nrtkhy+5yH6cmsq8wyo4xXQtQ6HJ ETILBNVOMy1YDXi8SOm/BahwpA== X-Google-Smtp-Source: ABdhPJzuxsX04dp6vqU3dYcnAf6eQ492YSoYAsmCgHr0gLh6r4gTcCO/Wqy1Q1zh7JC9jxZVWY0cXw== X-Received: by 2002:ad4:4903:: with SMTP id bh3mr25904140qvb.17.1593106304974; Thu, 25 Jun 2020 10:31:44 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id e9sm1659289qtq.70.2020.06.25.10.31.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jun 2020 10:31:44 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.93) (envelope-from ) id 1joViy-00EGdW-2a; Thu, 25 Jun 2020 14:31:44 -0300 Date: Thu, 25 Jun 2020 14:31:44 -0300 From: Jason Gunthorpe To: Ralph Campbell Cc: Christoph Hellwig , nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, Jerome Glisse , John Hubbard , Ben Skeggs , Andrew Morton , linux-mm@kvack.org, Bharata B Rao Subject: Re: [RESEND PATCH 2/3] nouveau: fix mixed normal and device private page migration Message-ID: <20200625173144.GT6578@ziepe.ca> References: <20200622233854.10889-1-rcampbell@nvidia.com> <20200622233854.10889-3-rcampbell@nvidia.com> <20200624072355.GB18609@lst.de> <330f6a82-d01d-db97-1dec-69346f41e707@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 33F31195C7F X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jun 25, 2020 at 10:25:38AM -0700, Ralph Campbell wrote: > Making sure to include linux-mm and Bharata B Rao for IBM's > use of migrate_vma*(). >=20 > On 6/24/20 11:10 AM, Ralph Campbell wrote: > >=20 > > On 6/24/20 12:23 AM, Christoph Hellwig wrote: > > > On Mon, Jun 22, 2020 at 04:38:53PM -0700, Ralph Campbell wrote: > > > > The OpenCL function clEnqueueSVMMigrateMem(), without any flags, = will > > > > migrate memory in the given address range to device private memor= y. The > > > > source pages might already have been migrated to device private m= emory. > > > > In that case, the source struct page is not checked to see if it = is > > > > a device private page and incorrectly computes the GPU's physical > > > > address of local memory leading to data corruption. > > > > Fix this by checking the source struct page and computing the cor= rect > > > > physical address. > > >=20 > > > I'm really worried about all this delicate code to fix the mixed > > > ranges.=C2=A0 Can't we make it clear at the migrate_vma_* level if = we want > > > to migrate from or two device private memory, and then skip all the= work > > > for regions of memory that already are in the right place?=C2=A0 Th= is might be > > > a little more work initially, but I think it leads to a much better > > > API. > > >=20 > >=20 > > The current code does encode the direction with src_owner !=3D NULL m= eaning > > device private to system memory and src_owner =3D=3D NULL meaning sys= tem > > memory to device private memory. This patch would obviously defeat th= at > > so perhaps a flag could be added to the struct migrate_vma to indicat= e the > > direction but I'm unclear how that makes things less delicate. > > Can you expand on what you are worried about? > >=20 > > The issue with invalidations might be better addressed by letting the= device > > driver handle device private page TLB invalidations when migrating to > > system memory and changing migrate_vma_setup() to only invalidate CPU > > TLB entries for normal pages being migrated to device private memory. > > If a page isn't migrating, it seems inefficient to invalidate those T= LB > > entries. > >=20 > > Any other suggestions? >=20 > After a night's sleep, I think this might work. What do others think? >=20 > 1) Add a new MMU_NOTIFY_MIGRATE enum to mmu_notifier_event. >=20 > 2) Change migrate_vma_collect() to use the new MMU_NOTIFY_MIGRATE event= type. > > 3) Modify nouveau_svmm_invalidate_range_start() to simply return (no in= validations) > for MMU_NOTIFY_MIGRATE mmu notifier callbacks. Isn't it a bit of an assumption that migrate_vma_collect() is only used by nouveau itself? What if some other devices' device_private pages are being migrated? Jason