archive mirror
 help / color / mirror / Atom feed
From: Felix Kuehling <>
To: Christoph Hellwig <>, Felix Kuehling <>
Cc: Jason Gunthorpe <>,,,,,
Subject: Re: [RFC PATCH 0/5] Support DEVICE_GENERIC memory in migrate_vma_*
Date: Sat, 29 May 2021 14:37:49 -0400	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

[-- Attachment #1.1: Type: text/plain, Size: 2108 bytes --]

Am 2021-05-29 um 2:41 a.m. schrieb Christoph Hellwig:
> On Fri, May 28, 2021 at 11:56:36AM -0400, Felix Kuehling wrote:
>> Am 2021-05-28 um 9:08 a.m. schrieb Jason Gunthorpe:
>>> On Thu, May 27, 2021 at 07:08:04PM -0400, Felix Kuehling wrote:
>>>> Now we're trying to migrate data to and from that memory using the
>>>> migrate_vma_* helpers so we can support page-based migration in our
>>>> unified memory allocations, while also supporting CPU access to those
>>>> pages.
>>> So you have completely coherent and indistinguishable GPU and CPU
>>> memory and the need of migration is basicaly alot like NUMA policy
>>> choice - get better access locality?
>> Yes. For a typical GPU compute application it means the GPU gets the
>> best bandwidth/latency, and the CPU can coherently access the results
>> without page faults and migrations. That's especially valuable for
>> applications with persistent compute kernels that want to exploit
>> concurrency between CPU and GPU.
> So why not expose the GPU memory as a CPUless memory node?

We did consider this, and are in fact still considering it for future
systems. For this system we decided not to go that way for several reasons.

For one, it means the driver would need to allocate VRAM with
__alloc_pages_nodemask for its own needs (firmware blobs, page tables,
etc.) and traditional BO-based memory allocation APIs. The GPU driver
would compete for VRAM with other application allocations, such as
malloc, mmap, page cache etc. Benchmarking and optimizing the NUMA
policy for such a system with a wide variety of workloads would be a big

All VRAM would need to be 0-initialized at allocation time. (I know
about init_on_free=1. In fact that's what our GPU driver does for VRAM
today, but asynchronously to hide the latency. However, init_on_free is
synchronous and has other drawbacks for system memory according to the
documentation of config INIT_ON_FREE_DEFAULT_ON.)

To make virtualization work, GPU access to its own local VRAM would need
to go through the system IOMMU.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

      reply	other threads:[~2021-05-29 18:37 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-27 23:08 Felix Kuehling
2021-05-27 23:08 ` [RFC PATCH 1/5] drm/amdkfd: add SPM support for SVM Felix Kuehling
2021-05-29  6:38   ` Christoph Hellwig
2021-05-29 18:42     ` Felix Kuehling
2021-05-27 23:08 ` [RFC PATCH 2/5] drm/amdkfd: generic type as sys mem on migration to ram Felix Kuehling
2021-05-27 23:08 ` [RFC PATCH 3/5] include/linux/mm.h: helper to check zone device generic type Felix Kuehling
2021-05-27 23:08 ` [RFC PATCH 4/5] mm: add generic type support for device zone page migration Felix Kuehling
2021-05-29  6:40   ` Christoph Hellwig
2021-05-27 23:08 ` [RFC PATCH 5/5] mm: changes to unref pages with Generic type Felix Kuehling
2021-05-29  6:42   ` Christoph Hellwig
2021-05-29 18:44     ` Felix Kuehling
2021-05-28 13:08 ` [RFC PATCH 0/5] Support DEVICE_GENERIC memory in migrate_vma_* Jason Gunthorpe
2021-05-28 15:56   ` Felix Kuehling
2021-05-29  6:41     ` Christoph Hellwig
2021-05-29 18:37       ` Felix Kuehling [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [RFC PATCH 0/5] Support DEVICE_GENERIC memory in migrate_vma_*' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).