From: "Theodore Ts'o" <tytso@mit.edu>
To: Alex Sierra <alex.sierra@amd.com>
Cc: akpm@linux-foundation.org, Felix.Kuehling@amd.com,
linux-mm@kvack.org, rcampbell@nvidia.com,
linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
hch@lst.de, jgg@nvidia.com, jglisse@redhat.com
Subject: Re: [PATCH v3 0/8] Support DEVICE_GENERIC memory in migrate_vma_*
Date: Sun, 20 Jun 2021 10:14:54 -0400 [thread overview]
Message-ID: <YM9NXrGlhdp0qb7S@mit.edu> (raw)
In-Reply-To: <20210617151705.15367-1-alex.sierra@amd.com>
On Thu, Jun 17, 2021 at 10:16:57AM -0500, Alex Sierra wrote:
> v1:
> AMD is building a system architecture for the Frontier supercomputer with a
> coherent interconnect between CPUs and GPUs. This hardware architecture allows
> the CPUs to coherently access GPU device memory. We have hardware in our labs
> and we are working with our partner HPE on the BIOS, firmware and software
> for delivery to the DOE.
>
> The system BIOS advertises the GPU device memory (aka VRAM) as SPM
> (special purpose memory) in the UEFI system address map. The amdgpu driver looks
> it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC
> using devm_memremap_pages.
>
> Now we're trying to migrate data to and from that memory using the migrate_vma_*
> helpers so we can support page-based migration in our unified memory allocations,
> while also supporting CPU access to those pages.
>
> This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave
> correctly in the migrate_vma_* helpers. We are looking for feedback about this
> approach. If we're close, what's needed to make our patches acceptable upstream?
> If we're not close, any suggestions how else to achieve what we are trying to do
> (i.e. page migration and coherent CPU access to VRAM)?
Is there a way we can test the codepaths touched by this patchset? It
doesn't have to be via a complete qemu simulation of the GPU device
memory, but some way of creating MEMORY_DEVICE_GENERIC subject to
migrate_vma_* helpers so we can test for regressions moving forward.
Thanks,
- Ted
WARNING: multiple messages have this Message-ID (diff)
From: "Theodore Ts'o" <tytso@mit.edu>
To: Alex Sierra <alex.sierra@amd.com>
Cc: rcampbell@nvidia.com, Felix.Kuehling@amd.com,
amd-gfx@lists.freedesktop.org, linux-xfs@vger.kernel.org,
linux-mm@kvack.org, jglisse@redhat.com,
dri-devel@lists.freedesktop.org, jgg@nvidia.com,
akpm@linux-foundation.org, linux-ext4@vger.kernel.org,
hch@lst.de
Subject: Re: [PATCH v3 0/8] Support DEVICE_GENERIC memory in migrate_vma_*
Date: Sun, 20 Jun 2021 10:14:54 -0400 [thread overview]
Message-ID: <YM9NXrGlhdp0qb7S@mit.edu> (raw)
In-Reply-To: <20210617151705.15367-1-alex.sierra@amd.com>
On Thu, Jun 17, 2021 at 10:16:57AM -0500, Alex Sierra wrote:
> v1:
> AMD is building a system architecture for the Frontier supercomputer with a
> coherent interconnect between CPUs and GPUs. This hardware architecture allows
> the CPUs to coherently access GPU device memory. We have hardware in our labs
> and we are working with our partner HPE on the BIOS, firmware and software
> for delivery to the DOE.
>
> The system BIOS advertises the GPU device memory (aka VRAM) as SPM
> (special purpose memory) in the UEFI system address map. The amdgpu driver looks
> it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC
> using devm_memremap_pages.
>
> Now we're trying to migrate data to and from that memory using the migrate_vma_*
> helpers so we can support page-based migration in our unified memory allocations,
> while also supporting CPU access to those pages.
>
> This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave
> correctly in the migrate_vma_* helpers. We are looking for feedback about this
> approach. If we're close, what's needed to make our patches acceptable upstream?
> If we're not close, any suggestions how else to achieve what we are trying to do
> (i.e. page migration and coherent CPU access to VRAM)?
Is there a way we can test the codepaths touched by this patchset? It
doesn't have to be via a complete qemu simulation of the GPU device
memory, but some way of creating MEMORY_DEVICE_GENERIC subject to
migrate_vma_* helpers so we can test for regressions moving forward.
Thanks,
- Ted
WARNING: multiple messages have this Message-ID (diff)
From: "Theodore Ts'o" <tytso@mit.edu>
To: Alex Sierra <alex.sierra@amd.com>
Cc: rcampbell@nvidia.com, Felix.Kuehling@amd.com,
amd-gfx@lists.freedesktop.org, linux-xfs@vger.kernel.org,
linux-mm@kvack.org, jglisse@redhat.com,
dri-devel@lists.freedesktop.org, jgg@nvidia.com,
akpm@linux-foundation.org, linux-ext4@vger.kernel.org,
hch@lst.de
Subject: Re: [PATCH v3 0/8] Support DEVICE_GENERIC memory in migrate_vma_*
Date: Sun, 20 Jun 2021 10:14:54 -0400 [thread overview]
Message-ID: <YM9NXrGlhdp0qb7S@mit.edu> (raw)
In-Reply-To: <20210617151705.15367-1-alex.sierra@amd.com>
On Thu, Jun 17, 2021 at 10:16:57AM -0500, Alex Sierra wrote:
> v1:
> AMD is building a system architecture for the Frontier supercomputer with a
> coherent interconnect between CPUs and GPUs. This hardware architecture allows
> the CPUs to coherently access GPU device memory. We have hardware in our labs
> and we are working with our partner HPE on the BIOS, firmware and software
> for delivery to the DOE.
>
> The system BIOS advertises the GPU device memory (aka VRAM) as SPM
> (special purpose memory) in the UEFI system address map. The amdgpu driver looks
> it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC
> using devm_memremap_pages.
>
> Now we're trying to migrate data to and from that memory using the migrate_vma_*
> helpers so we can support page-based migration in our unified memory allocations,
> while also supporting CPU access to those pages.
>
> This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave
> correctly in the migrate_vma_* helpers. We are looking for feedback about this
> approach. If we're close, what's needed to make our patches acceptable upstream?
> If we're not close, any suggestions how else to achieve what we are trying to do
> (i.e. page migration and coherent CPU access to VRAM)?
Is there a way we can test the codepaths touched by this patchset? It
doesn't have to be via a complete qemu simulation of the GPU device
memory, but some way of creating MEMORY_DEVICE_GENERIC subject to
migrate_vma_* helpers so we can test for regressions moving forward.
Thanks,
- Ted
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
next prev parent reply other threads:[~2021-06-20 14:15 UTC|newest]
Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-17 15:16 [PATCH v3 0/8] Support DEVICE_GENERIC memory in migrate_vma_* Alex Sierra
2021-06-17 15:16 ` Alex Sierra
2021-06-17 15:16 ` Alex Sierra
2021-06-17 15:16 ` [PATCH v3 1/8] ext4/xfs: add page refcount helper Alex Sierra
2021-06-17 15:16 ` Alex Sierra
2021-06-17 15:16 ` Alex Sierra
2021-06-17 16:11 ` Darrick J. Wong
2021-06-17 16:11 ` Darrick J. Wong
2021-06-17 16:11 ` Darrick J. Wong
2021-06-17 23:52 ` Dave Chinner
2021-06-17 23:52 ` Dave Chinner
2021-06-17 23:52 ` Dave Chinner
2021-06-19 16:14 ` Sierra Guiza, Alejandro (Alex)
2021-06-19 16:14 ` Sierra Guiza, Alejandro (Alex)
2021-06-19 16:14 ` Sierra Guiza, Alejandro (Alex)
2021-06-19 18:31 ` Alex Sierra
2021-06-19 18:31 ` Alex Sierra
2021-06-19 18:31 ` Alex Sierra
2021-06-17 15:16 ` [PATCH v3 2/8] mm: remove extra ZONE_DEVICE struct page refcount Alex Sierra
2021-06-17 15:16 ` Alex Sierra
2021-06-17 15:16 ` Alex Sierra
2021-06-17 19:16 ` Ralph Campbell
2021-06-17 19:16 ` Ralph Campbell
2021-06-17 19:16 ` Ralph Campbell
2021-06-28 16:46 ` Felix Kuehling
2021-06-28 16:46 ` Felix Kuehling
2021-06-28 16:46 ` Felix Kuehling
2021-06-30 0:23 ` Ralph Campbell
2021-06-30 0:23 ` Ralph Campbell
2021-06-30 0:23 ` Ralph Campbell
2021-06-17 15:17 ` [PATCH v3 3/8] kernel: resource: lookup_resource as exported symbol Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` [PATCH v3 4/8] drm/amdkfd: add SPM support for SVM Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` [PATCH v3 5/8] drm/amdkfd: generic type as sys mem on migration to ram Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` [PATCH v3 6/8] include/linux/mm.h: helpers to check zone device generic type Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` [PATCH v3 7/8] mm: add generic type support to migrate_vma helpers Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` [PATCH v3 8/8] mm: call pgmap->ops->page_free for DEVICE_GENERIC pages Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:17 ` Alex Sierra
2021-06-17 15:56 ` [PATCH v3 0/8] Support DEVICE_GENERIC memory in migrate_vma_* Sierra Guiza, Alejandro (Alex)
2021-06-17 15:56 ` Sierra Guiza, Alejandro (Alex)
2021-06-17 15:56 ` Sierra Guiza, Alejandro (Alex)
2021-06-20 14:14 ` Theodore Ts'o [this message]
2021-06-20 14:14 ` Theodore Ts'o
2021-06-20 14:14 ` Theodore Ts'o
2021-06-23 21:49 ` Felix Kuehling
2021-06-23 21:49 ` Felix Kuehling
2021-06-23 21:49 ` Felix Kuehling
2021-06-24 5:30 ` Christoph Hellwig
2021-06-24 5:30 ` Christoph Hellwig
2021-06-24 15:08 ` Felix Kuehling
2021-06-24 15:08 ` Felix Kuehling
2021-06-24 15:08 ` Felix Kuehling
2021-07-16 15:07 ` Theodore Y. Ts'o
2021-07-16 15:07 ` Theodore Y. Ts'o
2021-07-16 15:07 ` Theodore Y. Ts'o
2021-07-16 22:14 ` Felix Kuehling
2021-07-16 22:14 ` Felix Kuehling
2021-07-16 22:14 ` Felix Kuehling
2021-07-17 19:54 ` Sierra Guiza, Alejandro (Alex)
2021-07-17 19:54 ` Sierra Guiza, Alejandro (Alex)
2021-07-17 19:54 ` Sierra Guiza, Alejandro (Alex)
2021-07-23 22:46 ` Sierra Guiza, Alejandro (Alex)
2021-07-23 22:46 ` Sierra Guiza, Alejandro (Alex)
2021-07-23 22:46 ` Sierra Guiza, Alejandro (Alex)
2021-07-30 19:02 ` Felix Kuehling
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YM9NXrGlhdp0qb7S@mit.edu \
--to=tytso@mit.edu \
--cc=Felix.Kuehling@amd.com \
--cc=akpm@linux-foundation.org \
--cc=alex.sierra@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=dri-devel@lists.freedesktop.org \
--cc=hch@lst.de \
--cc=jgg@nvidia.com \
--cc=jglisse@redhat.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-xfs@vger.kernel.org \
--cc=rcampbell@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.