From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E822C47082 for ; Sat, 29 May 2021 18:37:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 81F4A61107 for ; Sat, 29 May 2021 18:37:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 81F4A61107 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B1A966B006C; Sat, 29 May 2021 14:37:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEF9A6B006E; Sat, 29 May 2021 14:37:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9699D6B0070; Sat, 29 May 2021 14:37:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0072.hostedemail.com [216.40.44.72]) by kanga.kvack.org (Postfix) with ESMTP id 4F8926B006C for ; Sat, 29 May 2021 14:37:53 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D0E7B180AD815 for ; Sat, 29 May 2021 18:37:52 +0000 (UTC) X-FDA: 78195127584.26.0D0C9F2 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf30.hostedemail.com (Postfix) with ESMTP id 50997E000242 for ; Sat, 29 May 2021 18:37:45 +0000 (UTC) Received: by mail-qv1-f45.google.com with SMTP id ee9so3590960qvb.8 for ; Sat, 29 May 2021 11:37:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:cc:references:from:subject:message-id:date:user-agent :mime-version:in-reply-to; bh=twoTbCEhVPknBNle9I5VmY9hPfyutTSbI17u0jwpKa8=; b=AUzIf1vANL4PJ+Mj5P1X9cSuxyBLENG8J5KHIoGWDODFxgkzl/VBV5cyMF2Lt4i815 baHLhVLD9x80HQZBGNXNzNCIcJD5xTDSmxeUXJ4IYxUM7HYf5EY0VVE8Cq40WQML/8g9 4YmRgANwAETngKJljQejxbyGLqgYF8BBKEK/6sEijzlMsjhxbJ8VsVY6wmdzl00C3qW+ tFOBHw4Qc/D7cGXM3IkVHZFmAwoIOr3FN7H878cSEWMWiQ7pkWUZLOyUKmB4954NgBhG 8ASU6PLuaYA57QvWeqWVWt1jR1kduMXR50Ymyz9hyX3QctMwmjH6szwUwme/V4Paz+rc s4cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:cc:references:from:subject:message-id:date :user-agent:mime-version:in-reply-to; bh=twoTbCEhVPknBNle9I5VmY9hPfyutTSbI17u0jwpKa8=; b=C+5lhe7tcbLbFgs9BpiciWMx53nzZd/qXiiF4+ZHWLu54awTja+UnfXv01IBlRpQFr t5O0SKMJwZlnBT/Q63zTrpmCKLZlVlnFDVKcooSZ8QQUyWSw3cI8zv7mG4qfGMvFMKe4 189qSM+FSPFA4Tc5j8OrEt01c9oJpC6zXdELFCLrkYe0rEH1898LUB7eMm1E9+75f5Qj QTEXepUdq5WqhvH7tXc2000dZg6pCwJdEbungldq9E1zkBR4+k1RhmsY3w7KRjEsxdUz ChU/SgxBwX4tZu5nDKlcglqtjhdFpFu1HR1rpSYijrvrnmQjO1jZ4vewyuX4hzovXqDv LI/A== X-Gm-Message-State: AOAM531YNPeSFoxC7BNkW7DOd+vuAT1s27QInb46p1Xi2c/6Mmv8ef1e JpFEJhkV+CtT7Ho2FMSt5C4= X-Google-Smtp-Source: ABdhPJz0uL8MWJGDxa1FwCsNoPlfrXg7imFfr+QweyCsRvfT9WHa40yS9tpRnyJKws2MUxelWhXfmA== X-Received: by 2002:ad4:53cc:: with SMTP id k12mr9600890qvv.49.1622313471993; Sat, 29 May 2021 11:37:51 -0700 (PDT) Received: from [192.168.2.100] (lnsm1-toronto42-142-116-203-225.internet.virginmobile.ca. [142.116.203.225]) by smtp.gmail.com with ESMTPSA id g9sm5993707qka.38.2021.05.29.11.37.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 29 May 2021 11:37:51 -0700 (PDT) To: Christoph Hellwig , Felix Kuehling Cc: Jason Gunthorpe , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, jglisse@redhat.com, amd-gfx@lists.freedesktop.org, akpm@linux-foundation.org References: <20210527230809.3701-1-Felix.Kuehling@amd.com> <20210528130816.GM1002214@nvidia.com> <20210529064124.GC15834@lst.de> From: Felix Kuehling Subject: Re: [RFC PATCH 0/5] Support DEVICE_GENERIC memory in migrate_vma_* Message-ID: <66cfeddd-11f5-0dba-841a-dcbdf4a1cee6@gmail.com> Date: Sat, 29 May 2021 14:37:49 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <20210529064124.GC15834@lst.de> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="EI7zzorYg30fi1H6VXHnRL7NAvh79DpSt" X-Rspamd-Queue-Id: 50997E000242 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=AUzIf1vA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of felixkuehling@gmail.com designates 209.85.219.45 as permitted sender) smtp.mailfrom=felixkuehling@gmail.com X-Rspamd-Server: rspam03 X-Stat-Signature: 5451h768ke554stbir54jju5k9ooa6xc X-HE-Tag: 1622313465-923739 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --EI7zzorYg30fi1H6VXHnRL7NAvh79DpSt Content-Type: multipart/mixed; boundary="ARIbLlR54FXkZwyu5Z6AqHqSmuRARcji5"; protected-headers="v1" From: Felix Kuehling To: Christoph Hellwig , Felix Kuehling Cc: Jason Gunthorpe , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, jglisse@redhat.com, amd-gfx@lists.freedesktop.org, akpm@linux-foundation.org Message-ID: <66cfeddd-11f5-0dba-841a-dcbdf4a1cee6@gmail.com> Subject: Re: [RFC PATCH 0/5] Support DEVICE_GENERIC memory in migrate_vma_* References: <20210527230809.3701-1-Felix.Kuehling@amd.com> <20210528130816.GM1002214@nvidia.com> <20210529064124.GC15834@lst.de> In-Reply-To: <20210529064124.GC15834@lst.de> --ARIbLlR54FXkZwyu5Z6AqHqSmuRARcji5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Language: en-US Am 2021-05-29 um 2:41 a.m. schrieb Christoph Hellwig: > On Fri, May 28, 2021 at 11:56:36AM -0400, Felix Kuehling wrote: >> Am 2021-05-28 um 9:08 a.m. schrieb Jason Gunthorpe: >>> On Thu, May 27, 2021 at 07:08:04PM -0400, Felix Kuehling wrote: >>>> Now we're trying to migrate data to and from that memory using the >>>> migrate_vma_* helpers so we can support page-based migration in our >>>> unified memory allocations, while also supporting CPU access to thos= e >>>> pages. >>> So you have completely coherent and indistinguishable GPU and CPU >>> memory and the need of migration is basicaly alot like NUMA policy >>> choice - get better access locality? >> Yes. For a typical GPU compute application it means the GPU gets the >> best bandwidth/latency, and the CPU can coherently access the results >> without page faults and migrations. That's especially valuable for >> applications with persistent compute kernels that want to exploit >> concurrency between CPU and GPU. > So why not expose the GPU memory as a CPUless memory node? We did consider this, and are in fact still considering it for future systems. For this system we decided not to go that way for several reason= s. For one, it means the driver would need to allocate VRAM with __alloc_pages_nodemask for its own needs (firmware blobs, page tables, etc.) and traditional BO-based memory allocation APIs. The GPU driver would compete for VRAM with other application allocations, such as malloc, mmap, page cache etc. Benchmarking and optimizing the NUMA policy for such a system with a wide variety of workloads would be a big effort. All VRAM would need to be 0-initialized at allocation time. (I know about init_on_free=3D1. In fact that's what our GPU driver does for VRAM today, but asynchronously to hide the latency. However, init_on_free is synchronous and has other drawbacks for system memory according to the documentation of config INIT_ON_FREE_DEFAULT_ON.) To make virtualization work, GPU access to its own local VRAM would need to go through the system IOMMU. Regards, =C2=A0 Felix --ARIbLlR54FXkZwyu5Z6AqHqSmuRARcji5-- --EI7zzorYg30fi1H6VXHnRL7NAvh79DpSt Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wnsEABEIACMWIQSF9iw3xYpcigJ58/LWGSsMBQyTdgUCYLKJ/QUDAAAAAAAKCRDWGSsMBQyTdlIm AP9YobaDl3U1tdF2IgIV+V3/rfI1H/rsutO1AHiKQuupCwD8C7slNVnonawvCCmmfNmkz1ubWJN2 HuBFddFzmw7eFEo= =+pp4 -----END PGP SIGNATURE----- --EI7zzorYg30fi1H6VXHnRL7NAvh79DpSt--