From: Felix Kuehling <Felix.Kuehling@amd.com>
To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org
Cc: Alex Sierra <alex.sierra@amd.com>, Philip Yang <Philip.Yang@amd.com>
Subject: [PATCH 30/34] drm/amdkfd: refine migration policy with xnack on
Date: Thu, 1 Apr 2021 00:22:24 -0400 [thread overview]
Message-ID: <20210401042228.1423-31-Felix.Kuehling@amd.com> (raw)
In-Reply-To: <20210401042228.1423-1-Felix.Kuehling@amd.com>
With xnack on, GPU vm fault handler decide the best restore location,
then migrate range to the best restore location and update GPU mapping
to recover the GPU vm fault.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 7 +-
drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 3 +-
drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 3 +
drivers/gpu/drm/amd/amdkfd/kfd_process.c | 16 +++
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 136 +++++++++++++++++++++--
5 files changed, 150 insertions(+), 15 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
index 552c4f656e2d..73701406acb3 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
@@ -479,18 +479,19 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
* svm_migrate_ram_to_vram - migrate svm range from system to device
* @prange: range structure
* @best_loc: the device to migrate to
+ * @mm: the process mm structure
*
* Context: Process context, caller hold mmap read lock, svms lock, prange lock
*
* Return:
* 0 - OK, otherwise error code
*/
-int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc)
+int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
+ struct mm_struct *mm)
{
unsigned long addr, start, end;
struct vm_area_struct *vma;
struct amdgpu_device *adev;
- struct mm_struct *mm;
int r = 0;
if (prange->actual_loc == best_loc) {
@@ -514,8 +515,6 @@ int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc)
start = prange->start << PAGE_SHIFT;
end = (prange->last + 1) << PAGE_SHIFT;
- mm = current->mm;
-
for (addr = start; addr < end;) {
unsigned long next;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
index 95fd7b21791f..9949b55d3b6a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
@@ -37,7 +37,8 @@ enum MIGRATION_COPY_DIR {
FROM_VRAM_TO_RAM
};
-int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc);
+int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
+ struct mm_struct *mm);
int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm);
unsigned long
svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index ca1b0c518d46..bce44164f1e3 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -864,6 +864,9 @@ struct kfd_process *kfd_lookup_process_by_pasid(u32 pasid);
struct kfd_process *kfd_lookup_process_by_mm(const struct mm_struct *mm);
int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id);
+int kfd_process_gpuid_from_kgd(struct kfd_process *p,
+ struct amdgpu_device *adev, uint32_t *gpuid,
+ uint32_t *gpuidx);
static inline int kfd_process_gpuid_from_gpuidx(struct kfd_process *p,
uint32_t gpuidx, uint32_t *gpuid) {
return gpuidx < p->n_pdds ? p->pdds[gpuidx]->dev->id : -EINVAL;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
index f897c1d0ea66..1d6310f63ae9 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
@@ -1673,6 +1673,22 @@ int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id)
return -EINVAL;
}
+int
+kfd_process_gpuid_from_kgd(struct kfd_process *p, struct amdgpu_device *adev,
+ uint32_t *gpuid, uint32_t *gpuidx)
+{
+ struct kgd_dev *kgd = (struct kgd_dev *)adev;
+ int i;
+
+ for (i = 0; i < p->n_pdds; i++)
+ if (p->pdds[i] && p->pdds[i]->dev->kgd == kgd) {
+ *gpuid = p->pdds[i]->dev->id;
+ *gpuidx = i;
+ return 0;
+ }
+ return -EINVAL;
+}
+
static void evict_process_worker(struct work_struct *work)
{
int ret;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 77da6c68fab2..61bf2df38e72 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1283,6 +1283,24 @@ static int svm_range_validate_and_map(struct mm_struct *mm,
if (gpuidx < MAX_GPU_INSTANCE) {
bitmap_zero(ctx.bitmap, MAX_GPU_INSTANCE);
bitmap_set(ctx.bitmap, gpuidx, 1);
+ } else if (ctx.process->xnack_enabled) {
+ bitmap_copy(ctx.bitmap, prange->bitmap_aip, MAX_GPU_INSTANCE);
+
+ /* If prefetch range to GPU, or GPU retry fault migrate range to
+ * GPU, which has ACCESS attribute to the range, create mapping
+ * on that GPU.
+ */
+ if (prange->actual_loc) {
+ gpuidx = kfd_process_gpuidx_from_gpuid(ctx.process,
+ prange->actual_loc);
+ if (gpuidx < 0) {
+ WARN_ONCE(1, "failed get device by id 0x%x\n",
+ prange->actual_loc);
+ return -EINVAL;
+ }
+ if (test_bit(gpuidx, prange->bitmap_access))
+ bitmap_set(ctx.bitmap, gpuidx, 1);
+ }
} else {
bitmap_or(ctx.bitmap, prange->bitmap_access,
prange->bitmap_aip, MAX_GPU_INSTANCE);
@@ -2013,15 +2031,75 @@ svm_range_from_addr(struct svm_range_list *svms, unsigned long addr,
return NULL;
}
+/* svm_range_best_restore_location - decide the best fault restore location
+ * @prange: svm range structure
+ * @adev: the GPU on which vm fault happened
+ *
+ * This is only called when xnack is on, to decide the best location to restore
+ * the range mapping after GPU vm fault. Caller uses the best location to do
+ * migration if actual loc is not best location, then update GPU page table
+ * mapping to the best location.
+ *
+ * If vm fault gpu is range preferred loc, the best_loc is preferred loc.
+ * If vm fault gpu idx is on range ACCESSIBLE bitmap, best_loc is vm fault gpu
+ * If vm fault gpu idx is on range ACCESSIBLE_IN_PLACE bitmap, then
+ * if range actual loc is cpu, best_loc is cpu
+ * if vm fault gpu is on xgmi same hive of range actual loc gpu, best_loc is
+ * range actual loc.
+ * Otherwise, GPU no access, best_loc is -1.
+ *
+ * Return:
+ * -1 means vm fault GPU no access
+ * 0 for CPU or GPU id
+ */
+static int32_t
+svm_range_best_restore_location(struct svm_range *prange,
+ struct amdgpu_device *adev,
+ int32_t *gpuidx)
+{
+ struct amdgpu_device *bo_adev;
+ struct kfd_process *p;
+ uint32_t gpuid;
+ int r;
+
+ p = container_of(prange->svms, struct kfd_process, svms);
+
+ r = kfd_process_gpuid_from_kgd(p, adev, &gpuid, gpuidx);
+ if (r < 0) {
+ pr_debug("failed to get gpuid from kgd\n");
+ return -1;
+ }
+
+ if (prange->preferred_loc == gpuid)
+ return prange->preferred_loc;
+
+ if (test_bit(*gpuidx, prange->bitmap_access))
+ return gpuid;
+
+ if (test_bit(*gpuidx, prange->bitmap_aip)) {
+ if (!prange->actual_loc)
+ return 0;
+
+ bo_adev = svm_range_get_adev_by_id(prange, prange->actual_loc);
+ if (amdgpu_xgmi_same_hive(adev, bo_adev))
+ return prange->actual_loc;
+ else
+ return 0;
+ }
+
+ return -1;
+}
+
int
svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
uint64_t addr)
{
- int r = 0;
struct mm_struct *mm = NULL;
- struct svm_range *prange;
struct svm_range_list *svms;
+ struct svm_range *prange;
struct kfd_process *p;
+ int32_t best_loc, gpuidx;
+ int r = 0;
p = kfd_lookup_process_by_pasid(pasid);
if (!p) {
@@ -2056,11 +2134,48 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
mutex_lock(&prange->migrate_mutex);
- r = svm_range_validate_and_map(mm, prange, MAX_GPU_INSTANCE, false, false);
- if (r)
- pr_debug("failed %d to map svms 0x%p [0x%lx 0x%lx] to gpu\n", r,
+ best_loc = svm_range_best_restore_location(prange, adev, &gpuidx);
+ if (best_loc == -1) {
+ pr_debug("svms %p failed get best restore loc [0x%lx 0x%lx]\n",
svms, prange->start, prange->last);
+ r = -EACCES;
+ goto out_unlock_range;
+ }
+
+ pr_debug("svms %p [0x%lx 0x%lx] best restore 0x%x, actual loc 0x%x\n",
+ svms, prange->start, prange->last, best_loc,
+ prange->actual_loc);
+ if (prange->actual_loc != best_loc) {
+ if (best_loc) {
+ r = svm_migrate_ram_to_vram(prange, best_loc, mm);
+ if (r) {
+ pr_debug("svm_migrate_to_vram failed (%d) at %llx, falling back to system memory\n",
+ r, addr);
+ /* Fallback to system memory if migration to
+ * VRAM failed
+ */
+ if (prange->actual_loc)
+ r = svm_migrate_vram_to_ram(prange, mm);
+ else
+ r = 0;
+ }
+ } else {
+ r = svm_migrate_vram_to_ram(prange, mm);
+ }
+ if (r) {
+ pr_debug("failed %d to migrate svms %p [0x%lx 0x%lx]\n",
+ r, svms, prange->start, prange->last);
+ goto out_unlock_range;
+ }
+ }
+
+ r = svm_range_validate_and_map(mm, prange, gpuidx, false, false);
+ if (r)
+ pr_debug("failed %d to map svms 0x%p [0x%lx 0x%lx] to gpus\n",
+ r, svms, prange->start, prange->last);
+
+out_unlock_range:
mutex_unlock(&prange->migrate_mutex);
out_unlock_svms:
mutex_unlock(&svms->lock);
@@ -2194,7 +2309,7 @@ svm_range_add(struct kfd_process *p, uint64_t start, uint64_t size,
return 0;
}
-/* svm_range_best_location - decide the best actual location
+/* svm_range_best_prefetch_location - decide the best prefetch location
* @prange: svm range structure
*
* For xnack off:
@@ -2216,7 +2331,8 @@ svm_range_add(struct kfd_process *p, uint64_t start, uint64_t size,
* Return:
* 0 for CPU or GPU id
*/
-static uint32_t svm_range_best_location(struct svm_range *prange)
+static uint32_t
+svm_range_best_prefetch_location(struct svm_range *prange)
{
DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE);
uint32_t best_loc = prange->prefetch_loc;
@@ -2318,7 +2434,7 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange,
int r = 0;
*migrated = false;
- best_loc = svm_range_best_location(prange);
+ best_loc = svm_range_best_prefetch_location(prange);
if (best_loc == KFD_IOCTL_SVM_LOCATION_UNDEFINED ||
best_loc == prange->actual_loc)
@@ -2330,10 +2446,10 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange,
if (best_loc) {
pr_debug("migrate from ram to vram\n");
- r = svm_migrate_ram_to_vram(prange, best_loc);
+ r = svm_migrate_ram_to_vram(prange, best_loc, mm);
} else {
pr_debug("migrate from vram to ram\n");
- r = svm_migrate_vram_to_ram(prange, current->mm);
+ r = svm_migrate_vram_to_ram(prange, mm);
}
if (!r)
--
2.31.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
next prev parent reply other threads:[~2021-04-01 4:23 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-01 4:21 [PATCH 00/34] Add HMM-based SVM memory manager to KFD v3 Felix Kuehling
2021-04-01 4:21 ` [PATCH 01/34] drm/amdkfd: helper to convert gpu id and idx Felix Kuehling
2021-04-01 4:21 ` [PATCH 02/34] drm/amdkfd: add svm ioctl API Felix Kuehling
2021-04-01 4:21 ` [PATCH 03/34] drm/amdkfd: register svm range Felix Kuehling
2021-04-01 10:04 ` kernel test robot
2021-04-01 4:21 ` [PATCH 04/34] drm/amdkfd: add svm ioctl GET_ATTR op Felix Kuehling
2021-04-01 4:21 ` [PATCH 05/34] drm/amdgpu: add common HMM get pages function Felix Kuehling
2021-04-01 4:22 ` [PATCH 06/34] drm/amdkfd: validate svm range system memory Felix Kuehling
2021-04-01 4:22 ` [PATCH 07/34] drm/amdkfd: deregister svm range Felix Kuehling
2021-04-01 11:48 ` kernel test robot
2021-04-01 4:22 ` [PATCH 08/34] drm/amdgpu: export vm update mapping interface Felix Kuehling
2021-04-01 4:22 ` [PATCH 09/34] drm/amdkfd: map svm range to GPUs Felix Kuehling
2021-04-01 13:28 ` kernel test robot
2021-04-01 4:22 ` [PATCH 10/34] drm/amdkfd: svm range eviction and restore Felix Kuehling
2021-04-01 15:19 ` kernel test robot
2021-04-01 4:22 ` [PATCH 11/34] drm/amdgpu: Enable retry faults unconditionally on Aldebaran Felix Kuehling
2021-04-01 4:22 ` [PATCH 12/34] drm/amdkfd: add xnack enabled flag to kfd_process Felix Kuehling
2021-04-01 4:22 ` [PATCH 13/34] drm/amdkfd: add ioctl to configure and query xnack retries Felix Kuehling
2021-04-01 4:22 ` [PATCH 14/34] drm/amdkfd: register HMM device private zone Felix Kuehling
2021-04-01 19:38 ` kernel test robot
2021-04-01 4:22 ` [PATCH 15/34] drm/amdkfd: validate vram svm range from TTM Felix Kuehling
2021-04-01 4:22 ` [PATCH 16/34] drm/amdkfd: support xgmi same hive mapping Felix Kuehling
2021-04-01 4:22 ` [PATCH 17/34] drm/amdkfd: copy memory through gart table Felix Kuehling
2021-04-01 4:22 ` [PATCH 18/34] drm/amdkfd: HMM migrate ram to vram Felix Kuehling
2021-04-01 4:22 ` [PATCH 19/34] drm/amdkfd: HMM migrate vram to ram Felix Kuehling
2021-04-01 22:37 ` kernel test robot
2021-04-01 4:22 ` [PATCH 20/34] drm/amdkfd: invalidate tables on page retry fault Felix Kuehling
2021-04-01 4:22 ` [PATCH 21/34] drm/amdgpu: enable 48-bit IH timestamp counter Felix Kuehling
2021-04-01 4:22 ` [PATCH 22/34] drm/amdkfd: page table restore through svm API Felix Kuehling
2021-04-01 4:22 ` [PATCH 23/34] drm/amdkfd: SVM API call to restore page tables Felix Kuehling
2021-04-01 4:22 ` [PATCH 24/34] drm/amdkfd: add svm_bo reference for eviction fence Felix Kuehling
2021-04-01 4:22 ` [PATCH 25/34] drm/amdgpu: add param bit flag to create SVM BOs Felix Kuehling
2021-04-01 4:22 ` [PATCH 26/34] drm/amdkfd: add svm_bo eviction mechanism support Felix Kuehling
2021-04-01 4:22 ` [PATCH 27/34] drm/amdgpu: svm bo enable_signal call condition Felix Kuehling
2021-04-01 4:22 ` [PATCH 28/34] drm/amdgpu: add svm_bo eviction to enable_signal cb Felix Kuehling
2021-04-01 4:22 ` [PATCH 29/34] drm/amdgpu: reserve fence slot to update page table Felix Kuehling
2021-04-01 4:22 ` Felix Kuehling [this message]
2021-04-01 4:22 ` [PATCH 31/34] drm/amdkfd: add svm range validate timestamp Felix Kuehling
2021-04-01 4:22 ` [PATCH 32/34] drm/amdkfd: multiple gpu migrate vram to vram Felix Kuehling
2021-04-01 4:22 ` [PATCH 33/34] drm/amdkfd: Add SVM API support capability bits Felix Kuehling
2021-04-01 4:22 ` [PATCH 34/34] drm/amdkfd: Add CONFIG_HSA_AMD_SVM Felix Kuehling
2021-04-06 1:45 [PATCH 00/34] Add HMM-based SVM memory manager to KFD v4 Felix Kuehling
2021-04-06 1:46 ` [PATCH 30/34] drm/amdkfd: refine migration policy with xnack on Felix Kuehling
2021-04-15 1:23 [PATCH 00/34] Add HMM-based SVM memory manager to KFD v5 Felix Kuehling
2021-04-15 1:23 ` [PATCH 30/34] drm/amdkfd: refine migration policy with xnack on Felix Kuehling
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210401042228.1423-31-Felix.Kuehling@amd.com \
--to=felix.kuehling@amd.com \
--cc=Philip.Yang@amd.com \
--cc=alex.sierra@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=dri-devel@lists.freedesktop.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).