From: Felix Kuehling <felix.kuehling@amd.com>
To: Philip Yang <Philip.Yang@amd.com>, amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH] drm/amdkfd: set memory limit to avoid OOM with HMM enabled
Date: Thu, 15 Apr 2021 17:51:49 -0400 [thread overview]
Message-ID: <72c15b2b-c71f-45b9-cb3c-1c56cb20bc3e@amd.com> (raw)
In-Reply-To: <20210415214332.9631-1-Philip.Yang@amd.com>
Am 2021-04-15 um 5:43 p.m. schrieb Philip Yang:
> HMM migration alloc sizeof(struct page) on system memory for each VRAM
> page, it is 1GB system memory reserved for 64GB VRAM. To avoid
> application OOM, increase system memory used size based on VRAM size of
> all GPUs, then application alloc memory will fail if system memory usage
> reach the limit.
>
> Signed-off-by: Philip Yang <Philip.Yang@amd.com>
> Reviewed-by: Oak Zeng <Oak.Zeng@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Thanks. I'll apply this to amd-staging-drm-next together with the big
HMM patch series.
Regards,
Felix
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 1 +
> drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 5 +++++
> drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 8 ++++++++
> 3 files changed, 14 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> index 025e8bade8c8..2cb7f8c30b9f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> @@ -272,6 +272,7 @@ void amdgpu_amdkfd_gpuvm_init_mem_limits(void);
> void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev,
> struct amdgpu_vm *vm);
> void amdgpu_amdkfd_unreserve_memory_limit(struct amdgpu_bo *bo);
> +void amdgpu_amdkfd_reserve_system_mem(uint64_t size);
> #else
> static inline
> void amdgpu_amdkfd_gpuvm_init_mem_limits(void)
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index d70a21ea576d..6ea1039b08a0 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -105,6 +105,11 @@ void amdgpu_amdkfd_gpuvm_init_mem_limits(void)
> (kfd_mem_limit.max_ttm_mem_limit >> 20));
> }
>
> +void amdgpu_amdkfd_reserve_system_mem(uint64_t size)
> +{
> + kfd_mem_limit.system_mem_used += size;
> +}
> +
> /* Estimate page table size needed to represent a given memory size
> *
> * With 4KB pages, we need one 8 byte PTE for each 4KB of memory
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> index 7d8659517447..1373ce9af890 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> @@ -876,6 +876,9 @@ static const struct dev_pagemap_ops svm_migrate_pgmap_ops = {
> .migrate_to_ram = svm_migrate_to_ram,
> };
>
> +/* Each VRAM page uses sizeof(struct page) on system memory */
> +#define SVM_HMM_PAGE_STRUCT_SIZE(size) ((size)/PAGE_SIZE * sizeof(struct page))
> +
> int svm_migrate_init(struct amdgpu_device *adev)
> {
> struct kfd_dev *kfddev = adev->kfd.dev;
> @@ -912,6 +915,11 @@ int svm_migrate_init(struct amdgpu_device *adev)
> return PTR_ERR(r);
> }
>
> + pr_debug("reserve %ldMB system memory for VRAM pages struct\n",
> + SVM_HMM_PAGE_STRUCT_SIZE(size) >> 20);
> +
> + amdgpu_amdkfd_reserve_system_mem(SVM_HMM_PAGE_STRUCT_SIZE(size));
> +
> pr_info("HMM registered %ldMB device memory\n", size >> 20);
>
> return 0;
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
prev parent reply other threads:[~2021-04-15 21:51 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-15 21:43 [PATCH] drm/amdkfd: set memory limit to avoid OOM with HMM enabled Philip Yang
2021-04-15 21:51 ` Felix Kuehling [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=72c15b2b-c71f-45b9-cb3c-1c56cb20bc3e@amd.com \
--to=felix.kuehling@amd.com \
--cc=Philip.Yang@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).