* [PATCH v2 0/8] HMM profiler interface
@ 2022-01-20 23:13 Philip Yang
2022-01-20 23:13 ` [PATCH v2 1/8] drm/amdkfd: Correct SMI event read size Philip Yang
` (8 more replies)
0 siblings, 9 replies; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
The ROCm profiler would expose the data from KFD profiling APIs to
application developers to tune the applications based on how the address
range attributes affect the behavior and performance.
Per process event log use the existing SMI (system management interface)
event API. Each event log is one line of text with the event specific
information.
v2:
* Keep existing events behaviour
* Use ktime_get_boottime_ns() as timestamp to correlate with other APIs
* Use compact message layout, stick with existing message convention
* Add unmap from GPU event
Philip Yang (8):
drm/amdkfd: Correct SMI event read size
drm/amdkfd: Add KFD SMI event IDs and triggers
drm/amdkfd: Enable per process SMI event
drm/amdkfd: Add GPU recoverable fault SMI event
drm/amdkfd: add migration SMI event
drm/amdkfd: Add user queue eviction restore SMI event
drm/amdkfd: Add unmap from GPU SMI event
drm/amdkfd: Bump KFD API version for SMI profiling event
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 7 +-
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 11 +-
drivers/gpu/drm/amd/amdkfd/kfd_device.c | 4 +-
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 67 ++++---
drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 5 +-
drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 +-
drivers/gpu/drm/amd/amdkfd/kfd_process.c | 37 +++-
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 163 +++++++++++++++++-
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 19 +-
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 63 +++++--
include/uapi/linux/kfd_ioctl.h | 30 +++-
11 files changed, 343 insertions(+), 65 deletions(-)
--
2.17.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 1/8] drm/amdkfd: Correct SMI event read size
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
@ 2022-01-20 23:13 ` Philip Yang
2022-01-25 16:24 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 2/8] drm/amdkfd: Add KFD SMI event IDs and triggers Philip Yang
` (7 subsequent siblings)
8 siblings, 1 reply; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
sizeof(buf) is 8 bytes because it is defined as unsigned char *buf,
each SMI event read only copy max 8 bytes to user buffer. Correct this
by using the buf allocate size.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index 329a4c89f1e6..18ed1b72f0f7 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -81,7 +81,8 @@ static ssize_t kfd_smi_ev_read(struct file *filep, char __user *user,
struct kfd_smi_client *client = filep->private_data;
unsigned char *buf;
- buf = kmalloc_array(MAX_KFIFO_SIZE, sizeof(*buf), GFP_KERNEL);
+ size = min_t(size_t, size, MAX_KFIFO_SIZE);
+ buf = kmalloc(size, GFP_KERNEL);
if (!buf)
return -ENOMEM;
@@ -95,7 +96,7 @@ static ssize_t kfd_smi_ev_read(struct file *filep, char __user *user,
ret = -EAGAIN;
goto ret_err;
}
- to_copy = min3(size, sizeof(buf), to_copy);
+ to_copy = min(size, to_copy);
ret = kfifo_out(&client->fifo, buf, to_copy);
spin_unlock(&client->lock);
if (ret <= 0) {
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 2/8] drm/amdkfd: Add KFD SMI event IDs and triggers
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
2022-01-20 23:13 ` [PATCH v2 1/8] drm/amdkfd: Correct SMI event read size Philip Yang
@ 2022-01-20 23:13 ` Philip Yang
2022-01-25 17:05 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 3/8] drm/amdkfd: Enable per process SMI event Philip Yang
` (6 subsequent siblings)
8 siblings, 1 reply; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
Define new system management interface event IDs, migration triggers and
user queue eviction triggers, those will be implemented in the following
patches.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
include/uapi/linux/kfd_ioctl.h | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/include/uapi/linux/kfd_ioctl.h b/include/uapi/linux/kfd_ioctl.h
index af96af174dc4..de0b5bb95db3 100644
--- a/include/uapi/linux/kfd_ioctl.h
+++ b/include/uapi/linux/kfd_ioctl.h
@@ -459,10 +459,37 @@ enum kfd_smi_event {
KFD_SMI_EVENT_THERMAL_THROTTLE = 2,
KFD_SMI_EVENT_GPU_PRE_RESET = 3,
KFD_SMI_EVENT_GPU_POST_RESET = 4,
+ KFD_SMI_EVENT_MIGRATION = 5,
+ KFD_SMI_EVENT_PAGE_FAULT_START = 6,
+ KFD_SMI_EVENT_PAGE_FAULT_END = 7,
+ KFD_SMI_EVENT_QUEUE_EVICTION = 8,
+ KFD_SMI_EVENT_QUEUE_EVICTION_RESTORE = 9,
+ KFD_SMI_EVENT_UNMAP_FROM_GPU = 10,
+
+ /*
+ * max event number, as a flag bit to get events from all processes,
+ * this requires super user permission, otherwise will not be able to
+ * receive event from any process. Without this flag to receive events
+ * from same process.
+ */
+ KFD_SMI_EVENT_ALL_PROCESS = 64
};
#define KFD_SMI_EVENT_MASK_FROM_INDEX(i) (1ULL << ((i) - 1))
+enum KFD_MIGRATION_QUEUE_EVICTION_UNMAP_EVENT_TRIGGER {
+ MIGRATION_TRIGGER_PREFETCH = 1,
+ MIGRATION_TRIGGER_PAGEFAULT,
+ MIGRATION_TRIGGER_PAGEFAULT_CPU,
+ MIGRATION_TRIGGER_TTM_EVICTION,
+ SVM_RANGE_EVICTION,
+ SVM_RANGE_MIGRATION,
+ USERPTR_EVICTION,
+ TTM_EVICTION,
+ UNMAP_FROM_CPU,
+ SUSPEND_EVICTION
+};
+
struct kfd_ioctl_smi_events_args {
__u32 gpuid; /* to KFD */
__u32 anon_fd; /* from KFD */
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 3/8] drm/amdkfd: Enable per process SMI event
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
2022-01-20 23:13 ` [PATCH v2 1/8] drm/amdkfd: Correct SMI event read size Philip Yang
2022-01-20 23:13 ` [PATCH v2 2/8] drm/amdkfd: Add KFD SMI event IDs and triggers Philip Yang
@ 2022-01-20 23:13 ` Philip Yang
2022-01-27 22:59 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 4/8] drm/amdkfd: Add GPU recoverable fault " Philip Yang
` (5 subsequent siblings)
8 siblings, 1 reply; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
Process receive event from same process by default. Add a flag to be
able to receive event from all processes, this requires super user
permission.
Event with pid 0 send to all processes, to keep the default behavior of
existing SMI events.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 29 ++++++++++++++++-----
1 file changed, 22 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index 18ed1b72f0f7..68c93701c5f7 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -37,6 +37,8 @@ struct kfd_smi_client {
uint64_t events;
struct kfd_dev *dev;
spinlock_t lock;
+ pid_t pid;
+ bool suser;
};
#define MAX_KFIFO_SIZE 1024
@@ -150,16 +152,27 @@ static int kfd_smi_ev_release(struct inode *inode, struct file *filep)
return 0;
}
-static void add_event_to_kfifo(struct kfd_dev *dev, unsigned int smi_event,
- char *event_msg, int len)
+static bool kfd_smi_ev_enabled(pid_t pid, struct kfd_smi_client *client,
+ unsigned int event)
+{
+ uint64_t all = KFD_SMI_EVENT_MASK_FROM_INDEX(KFD_SMI_EVENT_ALL_PROCESS);
+ uint64_t events = READ_ONCE(client->events);
+
+ if (pid && client->pid != pid && !(client->suser && (events & all)))
+ return false;
+
+ return events & KFD_SMI_EVENT_MASK_FROM_INDEX(event);
+}
+
+static void add_event_to_kfifo(pid_t pid, struct kfd_dev *dev,
+ unsigned int smi_event, char *event_msg, int len)
{
struct kfd_smi_client *client;
rcu_read_lock();
list_for_each_entry_rcu(client, &dev->smi_clients, list) {
- if (!(READ_ONCE(client->events) &
- KFD_SMI_EVENT_MASK_FROM_INDEX(smi_event)))
+ if (!kfd_smi_ev_enabled(pid, client, smi_event))
continue;
spin_lock(&client->lock);
if (kfifo_avail(&client->fifo) >= len) {
@@ -202,7 +215,7 @@ void kfd_smi_event_update_gpu_reset(struct kfd_dev *dev, bool post_reset)
len = snprintf(fifo_in, sizeof(fifo_in), "%x %x\n", event,
dev->reset_seq_num);
- add_event_to_kfifo(dev, event, fifo_in, len);
+ add_event_to_kfifo(0, dev, event, fifo_in, len);
}
void kfd_smi_event_update_thermal_throttling(struct kfd_dev *dev,
@@ -225,7 +238,7 @@ void kfd_smi_event_update_thermal_throttling(struct kfd_dev *dev,
KFD_SMI_EVENT_THERMAL_THROTTLE, throttle_bitmask,
amdgpu_dpm_get_thermal_throttling_counter(dev->adev));
- add_event_to_kfifo(dev, KFD_SMI_EVENT_THERMAL_THROTTLE, fifo_in, len);
+ add_event_to_kfifo(0, dev, KFD_SMI_EVENT_THERMAL_THROTTLE, fifo_in, len);
}
void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid)
@@ -250,7 +263,7 @@ void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid)
len = snprintf(fifo_in, sizeof(fifo_in), "%x %x:%s\n", KFD_SMI_EVENT_VMFAULT,
task_info.pid, task_info.task_name);
- add_event_to_kfifo(dev, KFD_SMI_EVENT_VMFAULT, fifo_in, len);
+ add_event_to_kfifo(0, dev, KFD_SMI_EVENT_VMFAULT, fifo_in, len);
}
int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
@@ -282,6 +295,8 @@ int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
spin_lock_init(&client->lock);
client->events = 0;
client->dev = dev;
+ client->pid = current->pid;
+ client->suser = capable(CAP_SYS_ADMIN);
spin_lock(&dev->smi_lock);
list_add_rcu(&client->list, &dev->smi_clients);
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 4/8] drm/amdkfd: Add GPU recoverable fault SMI event
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
` (2 preceding siblings ...)
2022-01-20 23:13 ` [PATCH v2 3/8] drm/amdkfd: Enable per process SMI event Philip Yang
@ 2022-01-20 23:13 ` Philip Yang
2022-01-27 23:15 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 5/8] drm/amdkfd: add migration " Philip Yang
` (4 subsequent siblings)
8 siblings, 1 reply; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
Output timestamp when GPU recoverable fault starts, ends and duration to
recover the fault, if migration happened or only GPU page table is
updated, fault address, read or write fault.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 48 +++++++++++++++++++++
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 7 ++-
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 17 ++++++--
3 files changed, 67 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index 68c93701c5f7..080eba0d3be0 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -266,6 +266,54 @@ void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid)
add_event_to_kfifo(0, dev, KFD_SMI_EVENT_VMFAULT, fifo_in, len);
}
+static bool kfd_smi_event_duration(struct kfd_dev *dev, uint64_t ts,
+ uint64_t *duration)
+{
+ if (list_empty(&dev->smi_clients))
+ return false;
+
+ *duration = ktime_get_boottime_ns() - ts;
+ return true;
+}
+
+void kfd_smi_event_page_fault_start(struct kfd_dev *dev, pid_t pid,
+ unsigned long address, bool write_fault,
+ uint64_t ts)
+{
+ char fifo_in[64];
+ int len;
+
+ if (list_empty(&dev->smi_clients))
+ return;
+
+ len = snprintf(fifo_in, sizeof(fifo_in), "%x %lld -%d @%lx(%x) %c\n",
+ KFD_SMI_EVENT_PAGE_FAULT_START, ts, pid, address,
+ dev->id, write_fault ? 'W' : 'R');
+
+ add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_PAGE_FAULT_START, fifo_in,
+ len);
+}
+
+void kfd_smi_event_page_fault_end(struct kfd_dev *dev, pid_t pid,
+ unsigned long address, bool migration,
+ uint64_t ts)
+{
+ char fifo_in[64];
+ uint64_t duration;
+ int len;
+
+ if (!kfd_smi_event_duration(dev, ts, &duration))
+ return;
+
+ len = snprintf(fifo_in, sizeof(fifo_in),
+ "%x %lld(%lld) -%d @%lx(%x) %c\n",
+ KFD_SMI_EVENT_PAGE_FAULT_END, ktime_get_boottime_ns(),
+ duration, pid, address, dev->id, migration ? 'M' : 'm');
+
+ add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_PAGE_FAULT_END, fifo_in,
+ len);
+}
+
int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
{
struct kfd_smi_client *client;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
index bffd0c32b060..7f70db914d2c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
@@ -28,5 +28,10 @@ void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid);
void kfd_smi_event_update_thermal_throttling(struct kfd_dev *dev,
uint64_t throttle_bitmask);
void kfd_smi_event_update_gpu_reset(struct kfd_dev *dev, bool post_reset);
-
+void kfd_smi_event_page_fault_start(struct kfd_dev *dev, pid_t pid,
+ unsigned long address, bool write_fault,
+ uint64_t ts);
+void kfd_smi_event_page_fault_end(struct kfd_dev *dev, pid_t pid,
+ unsigned long address, bool migration,
+ uint64_t ts);
#endif
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 2d2cae05dbea..08b21f9759ea 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -32,6 +32,7 @@
#include "kfd_priv.h"
#include "kfd_svm.h"
#include "kfd_migrate.h"
+#include "kfd_smi_events.h"
#ifdef dev_fmt
#undef dev_fmt
@@ -1596,7 +1597,7 @@ static int svm_range_validate_and_map(struct mm_struct *mm,
svm_range_unreserve_bos(&ctx);
if (!r)
- prange->validate_timestamp = ktime_to_us(ktime_get());
+ prange->validate_timestamp = ktime_get_boottime_ns();
return r;
}
@@ -2665,11 +2666,12 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
struct svm_range_list *svms;
struct svm_range *prange;
struct kfd_process *p;
- uint64_t timestamp;
+ uint64_t timestamp = ktime_get_boottime_ns();
int32_t best_loc;
int32_t gpuidx = MAX_GPU_INSTANCE;
bool write_locked = false;
struct vm_area_struct *vma;
+ bool migration = false;
int r = 0;
if (!KFD_IS_SVM_API_SUPPORTED(adev->kfd.dev)) {
@@ -2745,9 +2747,9 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
goto out_unlock_range;
}
- timestamp = ktime_to_us(ktime_get()) - prange->validate_timestamp;
/* skip duplicate vm fault on different pages of same range */
- if (timestamp < AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING) {
+ if (div_u64(timestamp - prange->validate_timestamp, 1000000) <
+ AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING) {
pr_debug("svms 0x%p [0x%lx %lx] already restored\n",
svms, prange->start, prange->last);
r = 0;
@@ -2783,7 +2785,11 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
svms, prange->start, prange->last, best_loc,
prange->actual_loc);
+ kfd_smi_event_page_fault_start(adev->kfd.dev, p->lead_thread->pid, addr,
+ write_fault, timestamp);
+
if (prange->actual_loc != best_loc) {
+ migration = true;
if (best_loc) {
r = svm_migrate_to_vram(prange, best_loc, mm);
if (r) {
@@ -2812,6 +2818,9 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
pr_debug("failed %d to map svms 0x%p [0x%lx 0x%lx] to gpus\n",
r, svms, prange->start, prange->last);
+ kfd_smi_event_page_fault_end(adev->kfd.dev, p->lead_thread->pid, addr,
+ migration, timestamp);
+
out_unlock_range:
mutex_unlock(&prange->migrate_mutex);
out_unlock_svms:
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 5/8] drm/amdkfd: add migration SMI event
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
` (3 preceding siblings ...)
2022-01-20 23:13 ` [PATCH v2 4/8] drm/amdkfd: Add GPU recoverable fault " Philip Yang
@ 2022-01-20 23:13 ` Philip Yang
2022-01-27 23:29 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 6/8] drm/amdkfd: Add user queue eviction restore " Philip Yang
` (3 subsequent siblings)
8 siblings, 1 reply; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
After migration is finished, output timestamp when migration starts,
duration of migration, svm range address and size, GPU id of
migration source and destination and svm range attributes,
Migration trigger could be prefetch, CPU or GPU page fault and TTM
eviction.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 67 ++++++++++++++-------
drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 5 +-
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 29 +++++++++
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 5 ++
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 16 +++--
5 files changed, 91 insertions(+), 31 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
index 88db82b3d443..06fb888f87aa 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
@@ -32,6 +32,7 @@
#include "kfd_priv.h"
#include "kfd_svm.h"
#include "kfd_migrate.h"
+#include "kfd_smi_events.h"
#ifdef dev_fmt
#undef dev_fmt
@@ -402,10 +403,11 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
static long
svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
struct vm_area_struct *vma, uint64_t start,
- uint64_t end)
+ uint64_t end, uint32_t trigger)
{
uint64_t npages = (end - start) >> PAGE_SHIFT;
- struct kfd_process_device *pdd;
+ struct kfd_process_device *pdd = NULL;
+ uint64_t timestamp = ktime_get_boottime_ns();
struct dma_fence *mfence = NULL;
struct migrate_vma migrate;
unsigned long cpages = 0;
@@ -431,6 +433,10 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
migrate.dst = migrate.src + npages;
scratch = (dma_addr_t *)(migrate.dst + npages);
+ pdd = svm_range_get_pdd_by_adev(prange, adev);
+ if (!pdd)
+ goto out_free;
+
r = migrate_vma_setup(&migrate);
if (r) {
dev_err(adev->dev, "vma setup fail %d range [0x%lx 0x%lx]\n", r,
@@ -459,6 +465,11 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
svm_migrate_copy_done(adev, mfence);
migrate_vma_finalize(&migrate);
+ kfd_smi_event_migration(adev->kfd.dev, pdd->process->pasid,
+ start >> PAGE_SHIFT, end >> PAGE_SHIFT,
+ 0, adev->kfd.dev->id, prange->prefetch_loc,
+ prange->preferred_loc, trigger, timestamp);
+
svm_range_dma_unmap(adev->dev, scratch, 0, npages);
svm_range_free_dma_mappings(prange);
@@ -466,10 +477,7 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
kvfree(buf);
out:
if (!r && cpages) {
- pdd = svm_range_get_pdd_by_adev(prange, adev);
- if (pdd)
- WRITE_ONCE(pdd->page_in, pdd->page_in + cpages);
-
+ WRITE_ONCE(pdd->page_in, pdd->page_in + cpages);
return cpages;
}
return r;
@@ -480,6 +488,7 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
* @prange: range structure
* @best_loc: the device to migrate to
* @mm: the process mm structure
+ * @trigger: reason of migration
*
* Context: Process context, caller hold mmap read lock, svms lock, prange lock
*
@@ -488,7 +497,7 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
*/
static int
svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
- struct mm_struct *mm)
+ struct mm_struct *mm, uint32_t trigger)
{
unsigned long addr, start, end;
struct vm_area_struct *vma;
@@ -525,7 +534,7 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
break;
next = min(vma->vm_end, end);
- r = svm_migrate_vma_to_vram(adev, prange, vma, addr, next);
+ r = svm_migrate_vma_to_vram(adev, prange, vma, addr, next, trigger);
if (r < 0) {
pr_debug("failed %ld to migrate\n", r);
break;
@@ -641,12 +650,14 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
static long
svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
- struct vm_area_struct *vma, uint64_t start, uint64_t end)
+ struct vm_area_struct *vma, uint64_t start, uint64_t end,
+ uint32_t trigger)
{
uint64_t npages = (end - start) >> PAGE_SHIFT;
+ uint64_t timestamp = ktime_get_boottime_ns();
unsigned long upages = npages;
unsigned long cpages = 0;
- struct kfd_process_device *pdd;
+ struct kfd_process_device *pdd = NULL;
struct dma_fence *mfence = NULL;
struct migrate_vma migrate;
dma_addr_t *scratch;
@@ -671,6 +682,10 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
migrate.dst = migrate.src + npages;
scratch = (dma_addr_t *)(migrate.dst + npages);
+ pdd = svm_range_get_pdd_by_adev(prange, adev);
+ if (!pdd)
+ goto out_free;
+
r = migrate_vma_setup(&migrate);
if (r) {
dev_err(adev->dev, "vma setup fail %d range [0x%lx 0x%lx]\n", r,
@@ -701,16 +716,19 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
svm_migrate_copy_done(adev, mfence);
migrate_vma_finalize(&migrate);
+
+ kfd_smi_event_migration(adev->kfd.dev, pdd->process->pasid,
+ start >> PAGE_SHIFT, end >> PAGE_SHIFT,
+ adev->kfd.dev->id, 0, prange->prefetch_loc,
+ prange->preferred_loc, trigger, timestamp);
+
svm_range_dma_unmap(adev->dev, scratch, 0, npages);
out_free:
kvfree(buf);
out:
if (!r && cpages) {
- pdd = svm_range_get_pdd_by_adev(prange, adev);
- if (pdd)
- WRITE_ONCE(pdd->page_out, pdd->page_out + cpages);
-
+ WRITE_ONCE(pdd->page_out, pdd->page_out + cpages);
return upages;
}
return r ? r : upages;
@@ -720,13 +738,15 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
* svm_migrate_vram_to_ram - migrate svm range from device to system
* @prange: range structure
* @mm: process mm, use current->mm if NULL
+ * @trigger: reason of migration
*
* Context: Process context, caller hold mmap read lock, svms lock, prange lock
*
* Return:
* 0 - OK, otherwise error code
*/
-int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm)
+int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm,
+ uint32_t trigger)
{
struct amdgpu_device *adev;
struct vm_area_struct *vma;
@@ -764,7 +784,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm)
break;
next = min(vma->vm_end, end);
- r = svm_migrate_vma_to_ram(adev, prange, vma, addr, next);
+ r = svm_migrate_vma_to_ram(adev, prange, vma, addr, next, trigger);
if (r < 0) {
pr_debug("failed %ld to migrate\n", r);
break;
@@ -787,6 +807,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm)
* @prange: range structure
* @best_loc: the device to migrate to
* @mm: process mm, use current->mm if NULL
+ * @trigger: reason of migration
*
* Context: Process context, caller hold mmap read lock, svms lock, prange lock
*
@@ -795,7 +816,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm)
*/
static int
svm_migrate_vram_to_vram(struct svm_range *prange, uint32_t best_loc,
- struct mm_struct *mm)
+ struct mm_struct *mm, uint32_t trigger)
{
int r, retries = 3;
@@ -807,7 +828,7 @@ svm_migrate_vram_to_vram(struct svm_range *prange, uint32_t best_loc,
pr_debug("from gpu 0x%x to gpu 0x%x\n", prange->actual_loc, best_loc);
do {
- r = svm_migrate_vram_to_ram(prange, mm);
+ r = svm_migrate_vram_to_ram(prange, mm, trigger);
if (r)
return r;
} while (prange->actual_loc && --retries);
@@ -815,17 +836,17 @@ svm_migrate_vram_to_vram(struct svm_range *prange, uint32_t best_loc,
if (prange->actual_loc)
return -EDEADLK;
- return svm_migrate_ram_to_vram(prange, best_loc, mm);
+ return svm_migrate_ram_to_vram(prange, best_loc, mm, trigger);
}
int
svm_migrate_to_vram(struct svm_range *prange, uint32_t best_loc,
- struct mm_struct *mm)
+ struct mm_struct *mm, uint32_t trigger)
{
if (!prange->actual_loc)
- return svm_migrate_ram_to_vram(prange, best_loc, mm);
+ return svm_migrate_ram_to_vram(prange, best_loc, mm, trigger);
else
- return svm_migrate_vram_to_vram(prange, best_loc, mm);
+ return svm_migrate_vram_to_vram(prange, best_loc, mm, trigger);
}
@@ -894,7 +915,7 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf)
goto out_unlock_prange;
}
- r = svm_migrate_vram_to_ram(prange, mm);
+ r = svm_migrate_vram_to_ram(prange, mm, MIGRATION_TRIGGER_PAGEFAULT_CPU);
if (r)
pr_debug("failed %d migrate 0x%p [0x%lx 0x%lx] to ram\n", r,
prange, prange->start, prange->last);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
index 2f5b3394c9ed..b3f0754b32fa 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
@@ -41,8 +41,9 @@ enum MIGRATION_COPY_DIR {
};
int svm_migrate_to_vram(struct svm_range *prange, uint32_t best_loc,
- struct mm_struct *mm);
-int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm);
+ struct mm_struct *mm, uint32_t trigger);
+int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm,
+ uint32_t trigger);
unsigned long
svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index 080eba0d3be0..97393f4f3549 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -314,6 +314,35 @@ void kfd_smi_event_page_fault_end(struct kfd_dev *dev, pid_t pid,
len);
}
+void kfd_smi_event_migration(struct kfd_dev *dev, uint16_t pasid,
+ unsigned long start, unsigned long end,
+ uint32_t from, uint32_t to,
+ uint32_t prefetch_loc, uint32_t preferred_loc,
+ uint32_t trigger, uint64_t ts)
+{
+ struct amdgpu_task_info task_info = {0};
+ char fifo_in[128];
+ uint64_t duration;
+ pid_t pid;
+ int len;
+
+ if (!kfd_smi_event_duration(dev, ts, &duration))
+ return;
+
+ amdgpu_vm_get_task_info(dev->adev, pasid, &task_info);
+ if (!task_info.pid)
+ return;
+ pid = task_info.pid;
+
+ len = snprintf(fifo_in, sizeof(fifo_in),
+ "%x %lld(%lld) -%d @%lx(%lx) %x->%x %x:%x %d\n",
+ KFD_SMI_EVENT_MIGRATION, ts, duration, pid, start,
+ end - start, from, to, prefetch_loc, preferred_loc,
+ trigger);
+
+ add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_MIGRATION, fifo_in, len);
+}
+
int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
{
struct kfd_smi_client *client;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
index 7f70db914d2c..5788728f2879 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
@@ -34,4 +34,9 @@ void kfd_smi_event_page_fault_start(struct kfd_dev *dev, pid_t pid,
void kfd_smi_event_page_fault_end(struct kfd_dev *dev, pid_t pid,
unsigned long address, bool migration,
uint64_t ts);
+void kfd_smi_event_migration(struct kfd_dev *dev, uint16_t pasid,
+ unsigned long start, unsigned long end,
+ uint32_t from, uint32_t to,
+ uint32_t prefetch_loc, uint32_t preferred_loc,
+ uint32_t trigger, uint64_t ts);
#endif
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 08b21f9759ea..7dbc724364e6 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -2791,7 +2791,8 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
if (prange->actual_loc != best_loc) {
migration = true;
if (best_loc) {
- r = svm_migrate_to_vram(prange, best_loc, mm);
+ r = svm_migrate_to_vram(prange, best_loc, mm,
+ MIGRATION_TRIGGER_PAGEFAULT);
if (r) {
pr_debug("svm_migrate_to_vram failed (%d) at %llx, falling back to system memory\n",
r, addr);
@@ -2799,12 +2800,14 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
* VRAM failed
*/
if (prange->actual_loc)
- r = svm_migrate_vram_to_ram(prange, mm);
+ r = svm_migrate_vram_to_ram(prange, mm,
+ MIGRATION_TRIGGER_PAGEFAULT);
else
r = 0;
}
} else {
- r = svm_migrate_vram_to_ram(prange, mm);
+ r = svm_migrate_vram_to_ram(prange, mm,
+ MIGRATION_TRIGGER_PAGEFAULT);
}
if (r) {
pr_debug("failed %d to migrate svms %p [0x%lx 0x%lx]\n",
@@ -3126,12 +3129,12 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange,
return 0;
if (!best_loc) {
- r = svm_migrate_vram_to_ram(prange, mm);
+ r = svm_migrate_vram_to_ram(prange, mm, MIGRATION_TRIGGER_PREFETCH);
*migrated = !r;
return r;
}
- r = svm_migrate_to_vram(prange, best_loc, mm);
+ r = svm_migrate_to_vram(prange, best_loc, mm, MIGRATION_TRIGGER_PREFETCH);
*migrated = !r;
return r;
@@ -3188,7 +3191,8 @@ static void svm_range_evict_svm_bo_worker(struct work_struct *work)
mutex_lock(&prange->migrate_mutex);
do {
svm_migrate_vram_to_ram(prange,
- svm_bo->eviction_fence->mm);
+ svm_bo->eviction_fence->mm,
+ MIGRATION_TRIGGER_TTM_EVICTION);
} while (prange->actual_loc && --retries);
WARN(prange->actual_loc, "Migration failed during eviction");
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 6/8] drm/amdkfd: Add user queue eviction restore SMI event
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
` (4 preceding siblings ...)
2022-01-20 23:13 ` [PATCH v2 5/8] drm/amdkfd: add migration " Philip Yang
@ 2022-01-20 23:13 ` Philip Yang
2022-01-27 23:37 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 7/8] drm/amdkfd: Add unmap from GPU " Philip Yang
` (2 subsequent siblings)
8 siblings, 1 reply; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
Output user queue eviction and restore event. User queue eviction may be
triggered by migration, MMU notifier, TTM eviction or device suspend.
User queue restore may be rescheduled if eviction happens again while
restore.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 7 +++-
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 11 ++++--
drivers/gpu/drm/amd/amdkfd/kfd_device.c | 4 +-
drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 +-
drivers/gpu/drm/amd/amdkfd/kfd_process.c | 37 +++++++++++++++++--
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 34 +++++++++++++++++
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 4 ++
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 16 ++++++--
8 files changed, 101 insertions(+), 14 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
index ac841ae8f5cc..bd3301e2c682 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
@@ -309,6 +309,7 @@ void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev,
*/
void amdgpu_amdkfd_release_notify(struct amdgpu_bo *bo);
void amdgpu_amdkfd_reserve_system_mem(uint64_t size);
+void kfd_process_smi_event_restore_rescheduled(struct mm_struct *mm);
#else
static inline
void amdgpu_amdkfd_gpuvm_init_mem_limits(void)
@@ -325,9 +326,13 @@ static inline
void amdgpu_amdkfd_release_notify(struct amdgpu_bo *bo)
{
}
+
+static inline void kfd_process_smi_event_restore_rescheduled(struct mm_struct *mm)
+{
+}
#endif
/* KGD2KFD callbacks */
-int kgd2kfd_quiesce_mm(struct mm_struct *mm);
+int kgd2kfd_quiesce_mm(struct mm_struct *mm, uint32_t trigger);
int kgd2kfd_resume_mm(struct mm_struct *mm);
int kgd2kfd_schedule_evict_and_restore_process(struct mm_struct *mm,
struct dma_fence *fence);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 5df387c4d7fb..c44e8dc0d869 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -2066,7 +2066,7 @@ int amdgpu_amdkfd_evict_userptr(struct kgd_mem *mem,
evicted_bos = atomic_inc_return(&process_info->evicted_bos);
if (evicted_bos == 1) {
/* First eviction, stop the queues */
- r = kgd2kfd_quiesce_mm(mm);
+ r = kgd2kfd_quiesce_mm(mm, USERPTR_EVICTION);
if (r)
pr_err("Failed to quiesce KFD\n");
schedule_delayed_work(&process_info->restore_userptr_work,
@@ -2340,13 +2340,16 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work)
unlock_out:
mutex_unlock(&process_info->lock);
- mmput(mm);
- put_task_struct(usertask);
/* If validation failed, reschedule another attempt */
- if (evicted_bos)
+ if (evicted_bos) {
schedule_delayed_work(&process_info->restore_userptr_work,
msecs_to_jiffies(AMDGPU_USERPTR_RESTORE_DELAY_MS));
+
+ kfd_process_smi_event_restore_rescheduled(mm);
+ }
+ mmput(mm);
+ put_task_struct(usertask);
}
/** amdgpu_amdkfd_gpuvm_restore_process_bos - Restore all BOs for the given
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index 5a47f437b455..ffaa80447d9c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -783,7 +783,7 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry)
spin_unlock_irqrestore(&kfd->interrupt_lock, flags);
}
-int kgd2kfd_quiesce_mm(struct mm_struct *mm)
+int kgd2kfd_quiesce_mm(struct mm_struct *mm, uint32_t trigger)
{
struct kfd_process *p;
int r;
@@ -797,7 +797,7 @@ int kgd2kfd_quiesce_mm(struct mm_struct *mm)
return -ESRCH;
WARN(debug_evictions, "Evicting pid %d", p->lead_thread->pid);
- r = kfd_process_evict_queues(p);
+ r = kfd_process_evict_queues(p, trigger);
kfd_unref_process(p);
return r;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index ea68f3b3a4e9..39519084df78 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -906,7 +906,7 @@ static inline struct kfd_process_device *kfd_process_device_from_gpuidx(
}
void kfd_unref_process(struct kfd_process *p);
-int kfd_process_evict_queues(struct kfd_process *p);
+int kfd_process_evict_queues(struct kfd_process *p, uint32_t trigger);
int kfd_process_restore_queues(struct kfd_process *p);
void kfd_suspend_all_processes(void);
int kfd_resume_all_processes(void);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
index 74f162887d3b..e4ba4d537b3c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
@@ -43,6 +43,7 @@ struct mm_struct;
#include "kfd_dbgmgr.h"
#include "kfd_iommu.h"
#include "kfd_svm.h"
+#include "kfd_smi_events.h"
/*
* List of struct kfd_process (field kfd_process).
@@ -1701,7 +1702,7 @@ struct kfd_process *kfd_lookup_process_by_mm(const struct mm_struct *mm)
* Eviction is reference-counted per process-device. This means multiple
* evictions from different sources can be nested safely.
*/
-int kfd_process_evict_queues(struct kfd_process *p)
+int kfd_process_evict_queues(struct kfd_process *p, uint32_t trigger)
{
int r = 0;
int i;
@@ -1710,6 +1711,9 @@ int kfd_process_evict_queues(struct kfd_process *p)
for (i = 0; i < p->n_pdds; i++) {
struct kfd_process_device *pdd = p->pdds[i];
+ kfd_smi_event_queue_eviction(pdd->dev, p->lead_thread->pid,
+ trigger);
+
r = pdd->dev->dqm->ops.evict_process_queues(pdd->dev->dqm,
&pdd->qpd);
/* evict return -EIO if HWS is hang or asic is resetting, in this case
@@ -1734,6 +1738,11 @@ int kfd_process_evict_queues(struct kfd_process *p)
if (n_evicted == 0)
break;
+
+ kfd_smi_event_queue_eviction_restore(pdd->dev,
+ p->lead_thread->pid,
+ false);
+
if (pdd->dev->dqm->ops.restore_process_queues(pdd->dev->dqm,
&pdd->qpd))
pr_err("Failed to restore queues\n");
@@ -1753,6 +1762,10 @@ int kfd_process_restore_queues(struct kfd_process *p)
for (i = 0; i < p->n_pdds; i++) {
struct kfd_process_device *pdd = p->pdds[i];
+ kfd_smi_event_queue_eviction_restore(pdd->dev,
+ p->lead_thread->pid,
+ false);
+
r = pdd->dev->dqm->ops.restore_process_queues(pdd->dev->dqm,
&pdd->qpd);
if (r) {
@@ -1765,6 +1778,24 @@ int kfd_process_restore_queues(struct kfd_process *p)
return ret;
}
+void kfd_process_smi_event_restore_rescheduled(struct mm_struct *mm)
+{
+ struct kfd_process *p;
+ int i;
+
+ p = kfd_lookup_process_by_mm(mm);
+ if (!p)
+ return;
+
+ for (i = 0; i < p->n_pdds; i++) {
+ struct kfd_process_device *pdd = p->pdds[i];
+
+ kfd_smi_event_queue_eviction_restore(pdd->dev,
+ p->lead_thread->pid, true);
+ }
+ kfd_unref_process(p);
+}
+
int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id)
{
int i;
@@ -1814,7 +1845,7 @@ static void evict_process_worker(struct work_struct *work)
flush_delayed_work(&p->restore_work);
pr_debug("Started evicting pasid 0x%x\n", p->pasid);
- ret = kfd_process_evict_queues(p);
+ ret = kfd_process_evict_queues(p, TTM_EVICTION);
if (!ret) {
dma_fence_signal(p->ef);
dma_fence_put(p->ef);
@@ -1881,7 +1912,7 @@ void kfd_suspend_all_processes(void)
cancel_delayed_work_sync(&p->eviction_work);
cancel_delayed_work_sync(&p->restore_work);
- if (kfd_process_evict_queues(p))
+ if (kfd_process_evict_queues(p, SUSPEND_EVICTION))
pr_err("Failed to suspend process 0x%x\n", p->pasid);
dma_fence_signal(p->ef);
dma_fence_put(p->ef);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index 97393f4f3549..facc8d7627d8 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -343,6 +343,40 @@ void kfd_smi_event_migration(struct kfd_dev *dev, uint16_t pasid,
add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_MIGRATION, fifo_in, len);
}
+void kfd_smi_event_queue_eviction(struct kfd_dev *dev, pid_t pid,
+ uint32_t trigger)
+{
+ char fifo_in[64];
+ int len;
+
+ if (list_empty(&dev->smi_clients))
+ return;
+
+ len = snprintf(fifo_in, sizeof(fifo_in), "%x %lld -%d @%x %d\n",
+ KFD_SMI_EVENT_QUEUE_EVICTION, ktime_get_boottime_ns(),
+ pid, dev->id, trigger);
+
+ add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_QUEUE_EVICTION, fifo_in, len);
+}
+
+void kfd_smi_event_queue_eviction_restore(struct kfd_dev *dev, pid_t pid,
+ bool rescheduled)
+{
+ char fifo_in[64];
+ int len;
+
+ if (list_empty(&dev->smi_clients))
+ return;
+
+ len = snprintf(fifo_in, sizeof(fifo_in), "%x %lld -%d @%x %c\n",
+ KFD_SMI_EVENT_QUEUE_EVICTION_RESTORE,
+ ktime_get_boottime_ns(), pid, dev->id,
+ rescheduled ? 'r' : ' ');
+
+ add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_QUEUE_EVICTION_RESTORE,
+ fifo_in, len);
+}
+
int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
{
struct kfd_smi_client *client;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
index 5788728f2879..d85300b5af23 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
@@ -39,4 +39,8 @@ void kfd_smi_event_migration(struct kfd_dev *dev, uint16_t pasid,
uint32_t from, uint32_t to,
uint32_t prefetch_loc, uint32_t preferred_loc,
uint32_t trigger, uint64_t ts);
+void kfd_smi_event_queue_eviction(struct kfd_dev *dev, pid_t pid,
+ uint32_t trigger);
+void kfd_smi_event_queue_eviction_restore(struct kfd_dev *dev, pid_t pid,
+ bool rescheduled);
#endif
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 7dbc724364e6..30aaa9764067 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1712,7 +1712,10 @@ static void svm_range_restore_work(struct work_struct *work)
pr_debug("reschedule to restore svm range\n");
schedule_delayed_work(&svms->restore_work,
msecs_to_jiffies(AMDGPU_SVM_RANGE_RESTORE_DELAY_MS));
+
+ kfd_process_smi_event_restore_rescheduled(mm);
}
+ mmput(mm);
}
/**
@@ -1732,15 +1735,22 @@ static void svm_range_restore_work(struct work_struct *work)
*/
static int
svm_range_evict(struct svm_range *prange, struct mm_struct *mm,
- unsigned long start, unsigned long last)
+ unsigned long start, unsigned long last,
+ enum mmu_notifier_event event)
{
struct svm_range_list *svms = prange->svms;
struct svm_range *pchild;
struct kfd_process *p;
+ uint32_t trigger;
int r = 0;
p = container_of(svms, struct kfd_process, svms);
+ if (event == MMU_NOTIFY_MIGRATE)
+ trigger = SVM_RANGE_MIGRATION;
+ else
+ trigger = SVM_RANGE_EVICTION;
+
pr_debug("invalidate svms 0x%p prange [0x%lx 0x%lx] [0x%lx 0x%lx]\n",
svms, prange->start, prange->last, start, last);
@@ -1768,7 +1778,7 @@ svm_range_evict(struct svm_range *prange, struct mm_struct *mm,
prange->svms, prange->start, prange->last);
/* First eviction, stop the queues */
- r = kgd2kfd_quiesce_mm(mm);
+ r = kgd2kfd_quiesce_mm(mm, trigger);
if (r)
pr_debug("failed to quiesce KFD\n");
@@ -2303,7 +2313,7 @@ svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni,
svm_range_unmap_from_cpu(mni->mm, prange, start, last);
break;
default:
- svm_range_evict(prange, mni->mm, start, last);
+ svm_range_evict(prange, mni->mm, start, last, range->event);
break;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 7/8] drm/amdkfd: Add unmap from GPU SMI event
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
` (5 preceding siblings ...)
2022-01-20 23:13 ` [PATCH v2 6/8] drm/amdkfd: Add user queue eviction restore " Philip Yang
@ 2022-01-20 23:13 ` Philip Yang
2022-01-20 23:13 ` [PATCH v2 8/8] drm/amdkfd: Bump KFD API version for SMI profiling event Philip Yang
2022-01-21 18:49 ` [PATCH v2 0/8] HMM profiler interface Deucher, Alexander
8 siblings, 0 replies; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
SVM range unmapped from GPUs when range is unmapped from CPU, or with
xnack on when range is evicted or migrated.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 18 ++++++++++++++++++
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 3 +++
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 14 +++++++++-----
3 files changed, 30 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index facc8d7627d8..736d8d0c9666 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -377,6 +377,24 @@ void kfd_smi_event_queue_eviction_restore(struct kfd_dev *dev, pid_t pid,
fifo_in, len);
}
+void kfd_smi_event_unmap_from_gpu(struct kfd_dev *dev, pid_t pid,
+ unsigned long address, unsigned long last,
+ uint32_t trigger)
+{
+ char fifo_in[64];
+ int len;
+
+ if (list_empty(&dev->smi_clients))
+ return;
+
+ len = snprintf(fifo_in, sizeof(fifo_in), "%x %lld -%d @%lx(%lx) %x %d\n",
+ KFD_SMI_EVENT_UNMAP_FROM_GPU, ktime_get_boottime_ns(),
+ pid, address, last - address + 1, dev->id, trigger);
+
+ add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_UNMAP_FROM_GPU, fifo_in,
+ len);
+}
+
int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
{
struct kfd_smi_client *client;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
index d85300b5af23..7d348452d8c3 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
@@ -43,4 +43,7 @@ void kfd_smi_event_queue_eviction(struct kfd_dev *dev, pid_t pid,
uint32_t trigger);
void kfd_smi_event_queue_eviction_restore(struct kfd_dev *dev, pid_t pid,
bool rescheduled);
+void kfd_smi_event_unmap_from_gpu(struct kfd_dev *dev, pid_t pid,
+ unsigned long address, unsigned long last,
+ uint32_t trigger);
#endif
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 30aaa9764067..f8e6c8269743 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1191,7 +1191,7 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
static int
svm_range_unmap_from_gpus(struct svm_range *prange, unsigned long start,
- unsigned long last)
+ unsigned long last, uint32_t trigger)
{
DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE);
struct kfd_process_device *pdd;
@@ -1212,6 +1212,9 @@ svm_range_unmap_from_gpus(struct svm_range *prange, unsigned long start,
return -EINVAL;
}
+ kfd_smi_event_unmap_from_gpu(pdd->dev, p->lead_thread->pid,
+ start, last, trigger);
+
r = svm_range_unmap_from_gpu(pdd->dev->adev,
drm_priv_to_vm(pdd->drm_priv),
start, last, &fence);
@@ -1795,13 +1798,13 @@ svm_range_evict(struct svm_range *prange, struct mm_struct *mm,
s = max(start, pchild->start);
l = min(last, pchild->last);
if (l >= s)
- svm_range_unmap_from_gpus(pchild, s, l);
+ svm_range_unmap_from_gpus(pchild, s, l, trigger);
mutex_unlock(&pchild->lock);
}
s = max(start, prange->start);
l = min(last, prange->last);
if (l >= s)
- svm_range_unmap_from_gpus(prange, s, l);
+ svm_range_unmap_from_gpus(prange, s, l, trigger);
}
return r;
@@ -2214,6 +2217,7 @@ static void
svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange,
unsigned long start, unsigned long last)
{
+ uint32_t trigger = UNMAP_FROM_CPU;
struct svm_range_list *svms;
struct svm_range *pchild;
struct kfd_process *p;
@@ -2241,14 +2245,14 @@ svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange,
s = max(start, pchild->start);
l = min(last, pchild->last);
if (l >= s)
- svm_range_unmap_from_gpus(pchild, s, l);
+ svm_range_unmap_from_gpus(pchild, s, l, trigger);
svm_range_unmap_split(mm, prange, pchild, start, last);
mutex_unlock(&pchild->lock);
}
s = max(start, prange->start);
l = min(last, prange->last);
if (l >= s)
- svm_range_unmap_from_gpus(prange, s, l);
+ svm_range_unmap_from_gpus(prange, s, l, trigger);
svm_range_unmap_split(mm, prange, prange, start, last);
if (unmap_parent)
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 8/8] drm/amdkfd: Bump KFD API version for SMI profiling event
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
` (6 preceding siblings ...)
2022-01-20 23:13 ` [PATCH v2 7/8] drm/amdkfd: Add unmap from GPU " Philip Yang
@ 2022-01-20 23:13 ` Philip Yang
2022-01-21 18:49 ` [PATCH v2 0/8] HMM profiler interface Deucher, Alexander
8 siblings, 0 replies; 16+ messages in thread
From: Philip Yang @ 2022-01-20 23:13 UTC (permalink / raw)
To: amd-gfx; +Cc: Philip Yang, felix.kuehling
Indicate SMI profiling events available,
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
include/uapi/linux/kfd_ioctl.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/include/uapi/linux/kfd_ioctl.h b/include/uapi/linux/kfd_ioctl.h
index de0b5bb95db3..1236550d1375 100644
--- a/include/uapi/linux/kfd_ioctl.h
+++ b/include/uapi/linux/kfd_ioctl.h
@@ -32,9 +32,10 @@
* - 1.4 - Indicate new SRAM EDC bit in device properties
* - 1.5 - Add SVM API
* - 1.6 - Query clear flags in SVM get_attr API
+ * - 1.7 - Add SMI profiler event log
*/
#define KFD_IOCTL_MAJOR_VERSION 1
-#define KFD_IOCTL_MINOR_VERSION 6
+#define KFD_IOCTL_MINOR_VERSION 7
struct kfd_ioctl_get_version_args {
__u32 major_version; /* from KFD */
--
2.17.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2 0/8] HMM profiler interface
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
` (7 preceding siblings ...)
2022-01-20 23:13 ` [PATCH v2 8/8] drm/amdkfd: Bump KFD API version for SMI profiling event Philip Yang
@ 2022-01-21 18:49 ` Deucher, Alexander
8 siblings, 0 replies; 16+ messages in thread
From: Deucher, Alexander @ 2022-01-21 18:49 UTC (permalink / raw)
To: Yang, Philip, amd-gfx; +Cc: Kuehling, Felix
[-- Attachment #1: Type: text/plain, Size: 2189 bytes --]
[Public]
Please provide a link to the proposed userspace branch that makes use of this.
Alex
________________________________
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> on behalf of Philip Yang <Philip.Yang@amd.com>
Sent: Thursday, January 20, 2022 6:13 PM
To: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Yang, Philip <Philip.Yang@amd.com>; Kuehling, Felix <Felix.Kuehling@amd.com>
Subject: [PATCH v2 0/8] HMM profiler interface
The ROCm profiler would expose the data from KFD profiling APIs to
application developers to tune the applications based on how the address
range attributes affect the behavior and performance.
Per process event log use the existing SMI (system management interface)
event API. Each event log is one line of text with the event specific
information.
v2:
* Keep existing events behaviour
* Use ktime_get_boottime_ns() as timestamp to correlate with other APIs
* Use compact message layout, stick with existing message convention
* Add unmap from GPU event
Philip Yang (8):
drm/amdkfd: Correct SMI event read size
drm/amdkfd: Add KFD SMI event IDs and triggers
drm/amdkfd: Enable per process SMI event
drm/amdkfd: Add GPU recoverable fault SMI event
drm/amdkfd: add migration SMI event
drm/amdkfd: Add user queue eviction restore SMI event
drm/amdkfd: Add unmap from GPU SMI event
drm/amdkfd: Bump KFD API version for SMI profiling event
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 7 +-
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 11 +-
drivers/gpu/drm/amd/amdkfd/kfd_device.c | 4 +-
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 67 ++++---
drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 5 +-
drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 +-
drivers/gpu/drm/amd/amdkfd/kfd_process.c | 37 +++-
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 163 +++++++++++++++++-
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 19 +-
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 63 +++++--
include/uapi/linux/kfd_ioctl.h | 30 +++-
11 files changed, 343 insertions(+), 65 deletions(-)
--
2.17.1
[-- Attachment #2: Type: text/html, Size: 3929 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/8] drm/amdkfd: Correct SMI event read size
2022-01-20 23:13 ` [PATCH v2 1/8] drm/amdkfd: Correct SMI event read size Philip Yang
@ 2022-01-25 16:24 ` Felix Kuehling
0 siblings, 0 replies; 16+ messages in thread
From: Felix Kuehling @ 2022-01-25 16:24 UTC (permalink / raw)
To: Philip Yang, amd-gfx
Am 2022-01-20 um 18:13 schrieb Philip Yang:
> sizeof(buf) is 8 bytes because it is defined as unsigned char *buf,
> each SMI event read only copy max 8 bytes to user buffer. Correct this
> by using the buf allocate size.
>
> Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
> ---
> drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> index 329a4c89f1e6..18ed1b72f0f7 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> @@ -81,7 +81,8 @@ static ssize_t kfd_smi_ev_read(struct file *filep, char __user *user,
> struct kfd_smi_client *client = filep->private_data;
> unsigned char *buf;
>
> - buf = kmalloc_array(MAX_KFIFO_SIZE, sizeof(*buf), GFP_KERNEL);
> + size = min_t(size_t, size, MAX_KFIFO_SIZE);
> + buf = kmalloc(size, GFP_KERNEL);
> if (!buf)
> return -ENOMEM;
>
> @@ -95,7 +96,7 @@ static ssize_t kfd_smi_ev_read(struct file *filep, char __user *user,
> ret = -EAGAIN;
> goto ret_err;
> }
> - to_copy = min3(size, sizeof(buf), to_copy);
> + to_copy = min(size, to_copy);
> ret = kfifo_out(&client->fifo, buf, to_copy);
> spin_unlock(&client->lock);
> if (ret <= 0) {
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 2/8] drm/amdkfd: Add KFD SMI event IDs and triggers
2022-01-20 23:13 ` [PATCH v2 2/8] drm/amdkfd: Add KFD SMI event IDs and triggers Philip Yang
@ 2022-01-25 17:05 ` Felix Kuehling
0 siblings, 0 replies; 16+ messages in thread
From: Felix Kuehling @ 2022-01-25 17:05 UTC (permalink / raw)
To: Philip Yang, amd-gfx
Am 2022-01-20 um 18:13 schrieb Philip Yang:
> Define new system management interface event IDs, migration triggers and
> user queue eviction triggers, those will be implemented in the following
> patches.
>
> Signed-off-by: Philip Yang <Philip.Yang@amd.com>
> ---
> include/uapi/linux/kfd_ioctl.h | 27 +++++++++++++++++++++++++++
> 1 file changed, 27 insertions(+)
>
> diff --git a/include/uapi/linux/kfd_ioctl.h b/include/uapi/linux/kfd_ioctl.h
> index af96af174dc4..de0b5bb95db3 100644
> --- a/include/uapi/linux/kfd_ioctl.h
> +++ b/include/uapi/linux/kfd_ioctl.h
> @@ -459,10 +459,37 @@ enum kfd_smi_event {
> KFD_SMI_EVENT_THERMAL_THROTTLE = 2,
> KFD_SMI_EVENT_GPU_PRE_RESET = 3,
> KFD_SMI_EVENT_GPU_POST_RESET = 4,
> + KFD_SMI_EVENT_MIGRATION = 5,
> + KFD_SMI_EVENT_PAGE_FAULT_START = 6,
> + KFD_SMI_EVENT_PAGE_FAULT_END = 7,
> + KFD_SMI_EVENT_QUEUE_EVICTION = 8,
> + KFD_SMI_EVENT_QUEUE_EVICTION_RESTORE = 9,
> + KFD_SMI_EVENT_UNMAP_FROM_GPU = 10,
> +
> + /*
> + * max event number, as a flag bit to get events from all processes,
> + * this requires super user permission, otherwise will not be able to
> + * receive event from any process. Without this flag to receive events
> + * from same process.
> + */
> + KFD_SMI_EVENT_ALL_PROCESS = 64
> };
>
> #define KFD_SMI_EVENT_MASK_FROM_INDEX(i) (1ULL << ((i) - 1))
>
> +enum KFD_MIGRATION_QUEUE_EVICTION_UNMAP_EVENT_TRIGGER {
> + MIGRATION_TRIGGER_PREFETCH = 1,
> + MIGRATION_TRIGGER_PAGEFAULT,
Call this KFD_MIGRATION_TRIGGER_PAGEFAULT_GPU for consistency. Add the
KFD_ prefix to all the migration triggers.
> + MIGRATION_TRIGGER_PAGEFAULT_CPU,
> + MIGRATION_TRIGGER_TTM_EVICTION,
> + SVM_RANGE_EVICTION,
> + SVM_RANGE_MIGRATION,
> + USERPTR_EVICTION,
> + TTM_EVICTION,
> + UNMAP_FROM_CPU,
> + SUSPEND_EVICTION
Migration triggers and queue eviction triggers should be separate enums.
The eviction triggers should have a consistent prefix:
KFD_QUEUE_EVICTION_TRIGGER...
I'm not sure the distinction between SVM_RANGE_MIGRATION and
SVM_RANGE_EVICTION is useful. It should be clear from the context, since
migrations log their own events. If anything, the term
SVM_RANGE_EVICTION is misleading. It's really any MMU notifier that's
not MMU_NOTIFY_MIGRATE or MMU_NOTIFY_UNMAP.
I'd suggest the following names:
KFD_QUEUE_EVICTION_TRIGGER_SVM
KFD_QUEUE_EVICTION_TRIGGER_USERPTR
KFD_QUEUE_EVICTION_TRIGGER_TTM
KFD_QUEUE_EVICTION_TRIGGER_SUSPEND
Then there are triggers for unmapping from GPU, which seems to have some
overlap with queue eviction in svm_range_evict. But that becomes a bit
easier if you remove the distinction between SVM_RANGE_MIGRATION and
SVM_RANGE_EVICTION. Then you are left with these two unmap triggers:
KFD_SVM_UNMAP_TRIGGER_MMU_NOTIFIER
KFD_SVM_UNMAP_TRIGGER_UNMAP_FROM_CPU
Regards,
Felix
> +};
> +
> struct kfd_ioctl_smi_events_args {
> __u32 gpuid; /* to KFD */
> __u32 anon_fd; /* from KFD */
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 3/8] drm/amdkfd: Enable per process SMI event
2022-01-20 23:13 ` [PATCH v2 3/8] drm/amdkfd: Enable per process SMI event Philip Yang
@ 2022-01-27 22:59 ` Felix Kuehling
0 siblings, 0 replies; 16+ messages in thread
From: Felix Kuehling @ 2022-01-27 22:59 UTC (permalink / raw)
To: Philip Yang, amd-gfx
Am 2022-01-20 um 18:13 schrieb Philip Yang:
> Process receive event from same process by default. Add a flag to be
> able to receive event from all processes, this requires super user
> permission.
>
> Event with pid 0 send to all processes, to keep the default behavior of
> existing SMI events.
>
> Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
> ---
> drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 29 ++++++++++++++++-----
> 1 file changed, 22 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> index 18ed1b72f0f7..68c93701c5f7 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> @@ -37,6 +37,8 @@ struct kfd_smi_client {
> uint64_t events;
> struct kfd_dev *dev;
> spinlock_t lock;
> + pid_t pid;
> + bool suser;
> };
>
> #define MAX_KFIFO_SIZE 1024
> @@ -150,16 +152,27 @@ static int kfd_smi_ev_release(struct inode *inode, struct file *filep)
> return 0;
> }
>
> -static void add_event_to_kfifo(struct kfd_dev *dev, unsigned int smi_event,
> - char *event_msg, int len)
> +static bool kfd_smi_ev_enabled(pid_t pid, struct kfd_smi_client *client,
> + unsigned int event)
> +{
> + uint64_t all = KFD_SMI_EVENT_MASK_FROM_INDEX(KFD_SMI_EVENT_ALL_PROCESS);
> + uint64_t events = READ_ONCE(client->events);
> +
> + if (pid && client->pid != pid && !(client->suser && (events & all)))
> + return false;
> +
> + return events & KFD_SMI_EVENT_MASK_FROM_INDEX(event);
> +}
> +
> +static void add_event_to_kfifo(pid_t pid, struct kfd_dev *dev,
> + unsigned int smi_event, char *event_msg, int len)
> {
> struct kfd_smi_client *client;
>
> rcu_read_lock();
>
> list_for_each_entry_rcu(client, &dev->smi_clients, list) {
> - if (!(READ_ONCE(client->events) &
> - KFD_SMI_EVENT_MASK_FROM_INDEX(smi_event)))
> + if (!kfd_smi_ev_enabled(pid, client, smi_event))
> continue;
> spin_lock(&client->lock);
> if (kfifo_avail(&client->fifo) >= len) {
> @@ -202,7 +215,7 @@ void kfd_smi_event_update_gpu_reset(struct kfd_dev *dev, bool post_reset)
> len = snprintf(fifo_in, sizeof(fifo_in), "%x %x\n", event,
> dev->reset_seq_num);
>
> - add_event_to_kfifo(dev, event, fifo_in, len);
> + add_event_to_kfifo(0, dev, event, fifo_in, len);
> }
>
> void kfd_smi_event_update_thermal_throttling(struct kfd_dev *dev,
> @@ -225,7 +238,7 @@ void kfd_smi_event_update_thermal_throttling(struct kfd_dev *dev,
> KFD_SMI_EVENT_THERMAL_THROTTLE, throttle_bitmask,
> amdgpu_dpm_get_thermal_throttling_counter(dev->adev));
>
> - add_event_to_kfifo(dev, KFD_SMI_EVENT_THERMAL_THROTTLE, fifo_in, len);
> + add_event_to_kfifo(0, dev, KFD_SMI_EVENT_THERMAL_THROTTLE, fifo_in, len);
> }
>
> void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid)
> @@ -250,7 +263,7 @@ void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid)
> len = snprintf(fifo_in, sizeof(fifo_in), "%x %x:%s\n", KFD_SMI_EVENT_VMFAULT,
> task_info.pid, task_info.task_name);
>
> - add_event_to_kfifo(dev, KFD_SMI_EVENT_VMFAULT, fifo_in, len);
> + add_event_to_kfifo(0, dev, KFD_SMI_EVENT_VMFAULT, fifo_in, len);
> }
>
> int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
> @@ -282,6 +295,8 @@ int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
> spin_lock_init(&client->lock);
> client->events = 0;
> client->dev = dev;
> + client->pid = current->pid;
> + client->suser = capable(CAP_SYS_ADMIN);
>
> spin_lock(&dev->smi_lock);
> list_add_rcu(&client->list, &dev->smi_clients);
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 4/8] drm/amdkfd: Add GPU recoverable fault SMI event
2022-01-20 23:13 ` [PATCH v2 4/8] drm/amdkfd: Add GPU recoverable fault " Philip Yang
@ 2022-01-27 23:15 ` Felix Kuehling
0 siblings, 0 replies; 16+ messages in thread
From: Felix Kuehling @ 2022-01-27 23:15 UTC (permalink / raw)
To: Philip Yang, amd-gfx
Am 2022-01-20 um 18:13 schrieb Philip Yang:
> Output timestamp when GPU recoverable fault starts, ends and duration to
> recover the fault, if migration happened or only GPU page table is
> updated, fault address, read or write fault.
>
> Signed-off-by: Philip Yang <Philip.Yang@amd.com>
> ---
> drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 48 +++++++++++++++++++++
> drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 7 ++-
> drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 17 ++++++--
> 3 files changed, 67 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> index 68c93701c5f7..080eba0d3be0 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> @@ -266,6 +266,54 @@ void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid)
> add_event_to_kfifo(0, dev, KFD_SMI_EVENT_VMFAULT, fifo_in, len);
> }
>
> +static bool kfd_smi_event_duration(struct kfd_dev *dev, uint64_t ts,
> + uint64_t *duration)
> +{
> + if (list_empty(&dev->smi_clients))
> + return false;
I'm not sure what this check has to do with the duration calculation. I
think this should be checked in the caller.
But I'd actually prefer not to include the duration in the message
because it's redundant. If you have both the start and end event in the
log, the duration can be easily calculated from the time stamps by
whatever tool is used to analyze or visualize the log. We do need a way
to match the start event to the end event. I think the PID and address
should be good enough for that.
> +
> + *duration = ktime_get_boottime_ns() - ts;
> + return true;
> +}
> +
> +void kfd_smi_event_page_fault_start(struct kfd_dev *dev, pid_t pid,
> + unsigned long address, bool write_fault,
> + uint64_t ts)
> +{
> + char fifo_in[64];
> + int len;
> +
> + if (list_empty(&dev->smi_clients))
> + return;
> +
> + len = snprintf(fifo_in, sizeof(fifo_in), "%x %lld -%d @%lx(%x) %c\n",
> + KFD_SMI_EVENT_PAGE_FAULT_START, ts, pid, address,
> + dev->id, write_fault ? 'W' : 'R');
> +
> + add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_PAGE_FAULT_START, fifo_in,
> + len);
> +}
> +
> +void kfd_smi_event_page_fault_end(struct kfd_dev *dev, pid_t pid,
> + unsigned long address, bool migration,
> + uint64_t ts)
> +{
> + char fifo_in[64];
> + uint64_t duration;
> + int len;
> +
> + if (!kfd_smi_event_duration(dev, ts, &duration))
> + return;
> +
> + len = snprintf(fifo_in, sizeof(fifo_in),
> + "%x %lld(%lld) -%d @%lx(%x) %c\n",
> + KFD_SMI_EVENT_PAGE_FAULT_END, ktime_get_boottime_ns(),
> + duration, pid, address, dev->id, migration ? 'M' : 'm');
> +
> + add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_PAGE_FAULT_END, fifo_in,
> + len);
> +}
> +
> int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
> {
> struct kfd_smi_client *client;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> index bffd0c32b060..7f70db914d2c 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> @@ -28,5 +28,10 @@ void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid);
> void kfd_smi_event_update_thermal_throttling(struct kfd_dev *dev,
> uint64_t throttle_bitmask);
> void kfd_smi_event_update_gpu_reset(struct kfd_dev *dev, bool post_reset);
> -
> +void kfd_smi_event_page_fault_start(struct kfd_dev *dev, pid_t pid,
> + unsigned long address, bool write_fault,
> + uint64_t ts);
> +void kfd_smi_event_page_fault_end(struct kfd_dev *dev, pid_t pid,
> + unsigned long address, bool migration,
> + uint64_t ts);
> #endif
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> index 2d2cae05dbea..08b21f9759ea 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> @@ -32,6 +32,7 @@
> #include "kfd_priv.h"
> #include "kfd_svm.h"
> #include "kfd_migrate.h"
> +#include "kfd_smi_events.h"
>
> #ifdef dev_fmt
> #undef dev_fmt
> @@ -1596,7 +1597,7 @@ static int svm_range_validate_and_map(struct mm_struct *mm,
> svm_range_unreserve_bos(&ctx);
>
> if (!r)
> - prange->validate_timestamp = ktime_to_us(ktime_get());
> + prange->validate_timestamp = ktime_get_boottime_ns();
>
> return r;
> }
> @@ -2665,11 +2666,12 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
> struct svm_range_list *svms;
> struct svm_range *prange;
> struct kfd_process *p;
> - uint64_t timestamp;
> + uint64_t timestamp = ktime_get_boottime_ns();
> int32_t best_loc;
> int32_t gpuidx = MAX_GPU_INSTANCE;
> bool write_locked = false;
> struct vm_area_struct *vma;
> + bool migration = false;
> int r = 0;
>
> if (!KFD_IS_SVM_API_SUPPORTED(adev->kfd.dev)) {
> @@ -2745,9 +2747,9 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
> goto out_unlock_range;
> }
>
> - timestamp = ktime_to_us(ktime_get()) - prange->validate_timestamp;
> /* skip duplicate vm fault on different pages of same range */
> - if (timestamp < AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING) {
> + if (div_u64(timestamp - prange->validate_timestamp, 1000000) <
> + AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING) {
You can probably avoid the division if you redefine
AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING to be in nanoseconds.
Regards,
Felix
> pr_debug("svms 0x%p [0x%lx %lx] already restored\n",
> svms, prange->start, prange->last);
> r = 0;
> @@ -2783,7 +2785,11 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
> svms, prange->start, prange->last, best_loc,
> prange->actual_loc);
>
> + kfd_smi_event_page_fault_start(adev->kfd.dev, p->lead_thread->pid, addr,
> + write_fault, timestamp);
> +
> if (prange->actual_loc != best_loc) {
> + migration = true;
> if (best_loc) {
> r = svm_migrate_to_vram(prange, best_loc, mm);
> if (r) {
> @@ -2812,6 +2818,9 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
> pr_debug("failed %d to map svms 0x%p [0x%lx 0x%lx] to gpus\n",
> r, svms, prange->start, prange->last);
>
> + kfd_smi_event_page_fault_end(adev->kfd.dev, p->lead_thread->pid, addr,
> + migration, timestamp);
> +
> out_unlock_range:
> mutex_unlock(&prange->migrate_mutex);
> out_unlock_svms:
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 5/8] drm/amdkfd: add migration SMI event
2022-01-20 23:13 ` [PATCH v2 5/8] drm/amdkfd: add migration " Philip Yang
@ 2022-01-27 23:29 ` Felix Kuehling
0 siblings, 0 replies; 16+ messages in thread
From: Felix Kuehling @ 2022-01-27 23:29 UTC (permalink / raw)
To: Philip Yang, amd-gfx
Am 2022-01-20 um 18:13 schrieb Philip Yang:
> After migration is finished, output timestamp when migration starts,
> duration of migration, svm range address and size, GPU id of
> migration source and destination and svm range attributes,
>
> Migration trigger could be prefetch, CPU or GPU page fault and TTM
> eviction.
>
> Signed-off-by: Philip Yang <Philip.Yang@amd.com>
> ---
> drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 67 ++++++++++++++-------
> drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 5 +-
> drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 29 +++++++++
> drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 5 ++
> drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 16 +++--
> 5 files changed, 91 insertions(+), 31 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> index 88db82b3d443..06fb888f87aa 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> @@ -32,6 +32,7 @@
> #include "kfd_priv.h"
> #include "kfd_svm.h"
> #include "kfd_migrate.h"
> +#include "kfd_smi_events.h"
>
> #ifdef dev_fmt
> #undef dev_fmt
> @@ -402,10 +403,11 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
> static long
> svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
> struct vm_area_struct *vma, uint64_t start,
> - uint64_t end)
> + uint64_t end, uint32_t trigger)
> {
> uint64_t npages = (end - start) >> PAGE_SHIFT;
> - struct kfd_process_device *pdd;
> + struct kfd_process_device *pdd = NULL;
> + uint64_t timestamp = ktime_get_boottime_ns();
> struct dma_fence *mfence = NULL;
> struct migrate_vma migrate;
> unsigned long cpages = 0;
> @@ -431,6 +433,10 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
> migrate.dst = migrate.src + npages;
> scratch = (dma_addr_t *)(migrate.dst + npages);
>
> + pdd = svm_range_get_pdd_by_adev(prange, adev);
> + if (!pdd)
> + goto out_free;
> +
> r = migrate_vma_setup(&migrate);
> if (r) {
> dev_err(adev->dev, "vma setup fail %d range [0x%lx 0x%lx]\n", r,
> @@ -459,6 +465,11 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
> svm_migrate_copy_done(adev, mfence);
> migrate_vma_finalize(&migrate);
>
> + kfd_smi_event_migration(adev->kfd.dev, pdd->process->pasid,
The pasid is only used by kfd_smi_event_migration to determine the PID.
But you already know the PID here:
container_of(prange->svms, struct kfd_process, svms)->lead_thread->pid;
> + start >> PAGE_SHIFT, end >> PAGE_SHIFT,
> + 0, adev->kfd.dev->id, prange->prefetch_loc,
> + prange->preferred_loc, trigger, timestamp);
> +
> svm_range_dma_unmap(adev->dev, scratch, 0, npages);
> svm_range_free_dma_mappings(prange);
>
> @@ -466,10 +477,7 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
> kvfree(buf);
> out:
> if (!r && cpages) {
> - pdd = svm_range_get_pdd_by_adev(prange, adev);
> - if (pdd)
> - WRITE_ONCE(pdd->page_in, pdd->page_in + cpages);
> -
> + WRITE_ONCE(pdd->page_in, pdd->page_in + cpages);
> return cpages;
> }
> return r;
> @@ -480,6 +488,7 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
> * @prange: range structure
> * @best_loc: the device to migrate to
> * @mm: the process mm structure
> + * @trigger: reason of migration
> *
> * Context: Process context, caller hold mmap read lock, svms lock, prange lock
> *
> @@ -488,7 +497,7 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
> */
> static int
> svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
> - struct mm_struct *mm)
> + struct mm_struct *mm, uint32_t trigger)
> {
> unsigned long addr, start, end;
> struct vm_area_struct *vma;
> @@ -525,7 +534,7 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
> break;
>
> next = min(vma->vm_end, end);
> - r = svm_migrate_vma_to_vram(adev, prange, vma, addr, next);
> + r = svm_migrate_vma_to_vram(adev, prange, vma, addr, next, trigger);
> if (r < 0) {
> pr_debug("failed %ld to migrate\n", r);
> break;
> @@ -641,12 +650,14 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>
> static long
> svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
> - struct vm_area_struct *vma, uint64_t start, uint64_t end)
> + struct vm_area_struct *vma, uint64_t start, uint64_t end,
> + uint32_t trigger)
> {
> uint64_t npages = (end - start) >> PAGE_SHIFT;
> + uint64_t timestamp = ktime_get_boottime_ns();
> unsigned long upages = npages;
> unsigned long cpages = 0;
> - struct kfd_process_device *pdd;
> + struct kfd_process_device *pdd = NULL;
> struct dma_fence *mfence = NULL;
> struct migrate_vma migrate;
> dma_addr_t *scratch;
> @@ -671,6 +682,10 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
> migrate.dst = migrate.src + npages;
> scratch = (dma_addr_t *)(migrate.dst + npages);
>
> + pdd = svm_range_get_pdd_by_adev(prange, adev);
> + if (!pdd)
> + goto out_free;
> +
> r = migrate_vma_setup(&migrate);
> if (r) {
> dev_err(adev->dev, "vma setup fail %d range [0x%lx 0x%lx]\n", r,
> @@ -701,16 +716,19 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>
> svm_migrate_copy_done(adev, mfence);
> migrate_vma_finalize(&migrate);
> +
> + kfd_smi_event_migration(adev->kfd.dev, pdd->process->pasid,
> + start >> PAGE_SHIFT, end >> PAGE_SHIFT,
> + adev->kfd.dev->id, 0, prange->prefetch_loc,
> + prange->preferred_loc, trigger, timestamp);
> +
> svm_range_dma_unmap(adev->dev, scratch, 0, npages);
>
> out_free:
> kvfree(buf);
> out:
> if (!r && cpages) {
> - pdd = svm_range_get_pdd_by_adev(prange, adev);
> - if (pdd)
> - WRITE_ONCE(pdd->page_out, pdd->page_out + cpages);
> -
> + WRITE_ONCE(pdd->page_out, pdd->page_out + cpages);
> return upages;
> }
> return r ? r : upages;
> @@ -720,13 +738,15 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
> * svm_migrate_vram_to_ram - migrate svm range from device to system
> * @prange: range structure
> * @mm: process mm, use current->mm if NULL
> + * @trigger: reason of migration
> *
> * Context: Process context, caller hold mmap read lock, svms lock, prange lock
> *
> * Return:
> * 0 - OK, otherwise error code
> */
> -int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm)
> +int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm,
> + uint32_t trigger)
> {
> struct amdgpu_device *adev;
> struct vm_area_struct *vma;
> @@ -764,7 +784,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm)
> break;
>
> next = min(vma->vm_end, end);
> - r = svm_migrate_vma_to_ram(adev, prange, vma, addr, next);
> + r = svm_migrate_vma_to_ram(adev, prange, vma, addr, next, trigger);
> if (r < 0) {
> pr_debug("failed %ld to migrate\n", r);
> break;
> @@ -787,6 +807,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm)
> * @prange: range structure
> * @best_loc: the device to migrate to
> * @mm: process mm, use current->mm if NULL
> + * @trigger: reason of migration
> *
> * Context: Process context, caller hold mmap read lock, svms lock, prange lock
> *
> @@ -795,7 +816,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm)
> */
> static int
> svm_migrate_vram_to_vram(struct svm_range *prange, uint32_t best_loc,
> - struct mm_struct *mm)
> + struct mm_struct *mm, uint32_t trigger)
> {
> int r, retries = 3;
>
> @@ -807,7 +828,7 @@ svm_migrate_vram_to_vram(struct svm_range *prange, uint32_t best_loc,
> pr_debug("from gpu 0x%x to gpu 0x%x\n", prange->actual_loc, best_loc);
>
> do {
> - r = svm_migrate_vram_to_ram(prange, mm);
> + r = svm_migrate_vram_to_ram(prange, mm, trigger);
> if (r)
> return r;
> } while (prange->actual_loc && --retries);
> @@ -815,17 +836,17 @@ svm_migrate_vram_to_vram(struct svm_range *prange, uint32_t best_loc,
> if (prange->actual_loc)
> return -EDEADLK;
>
> - return svm_migrate_ram_to_vram(prange, best_loc, mm);
> + return svm_migrate_ram_to_vram(prange, best_loc, mm, trigger);
> }
>
> int
> svm_migrate_to_vram(struct svm_range *prange, uint32_t best_loc,
> - struct mm_struct *mm)
> + struct mm_struct *mm, uint32_t trigger)
> {
> if (!prange->actual_loc)
> - return svm_migrate_ram_to_vram(prange, best_loc, mm);
> + return svm_migrate_ram_to_vram(prange, best_loc, mm, trigger);
> else
> - return svm_migrate_vram_to_vram(prange, best_loc, mm);
> + return svm_migrate_vram_to_vram(prange, best_loc, mm, trigger);
>
> }
>
> @@ -894,7 +915,7 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf)
> goto out_unlock_prange;
> }
>
> - r = svm_migrate_vram_to_ram(prange, mm);
> + r = svm_migrate_vram_to_ram(prange, mm, MIGRATION_TRIGGER_PAGEFAULT_CPU);
> if (r)
> pr_debug("failed %d migrate 0x%p [0x%lx 0x%lx] to ram\n", r,
> prange, prange->start, prange->last);
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
> index 2f5b3394c9ed..b3f0754b32fa 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h
> @@ -41,8 +41,9 @@ enum MIGRATION_COPY_DIR {
> };
>
> int svm_migrate_to_vram(struct svm_range *prange, uint32_t best_loc,
> - struct mm_struct *mm);
> -int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm);
> + struct mm_struct *mm, uint32_t trigger);
> +int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm,
> + uint32_t trigger);
> unsigned long
> svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr);
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> index 080eba0d3be0..97393f4f3549 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> @@ -314,6 +314,35 @@ void kfd_smi_event_page_fault_end(struct kfd_dev *dev, pid_t pid,
> len);
> }
>
> +void kfd_smi_event_migration(struct kfd_dev *dev, uint16_t pasid,
> + unsigned long start, unsigned long end,
> + uint32_t from, uint32_t to,
> + uint32_t prefetch_loc, uint32_t preferred_loc,
> + uint32_t trigger, uint64_t ts)
> +{
> + struct amdgpu_task_info task_info = {0};
> + char fifo_in[128];
> + uint64_t duration;
> + pid_t pid;
> + int len;
> +
> + if (!kfd_smi_event_duration(dev, ts, &duration))
> + return;
> +
> + amdgpu_vm_get_task_info(dev->adev, pasid, &task_info);
> + if (!task_info.pid)
> + return;
> + pid = task_info.pid;
> +
> + len = snprintf(fifo_in, sizeof(fifo_in),
> + "%x %lld(%lld) -%d @%lx(%lx) %x->%x %x:%x %d\n",
> + KFD_SMI_EVENT_MIGRATION, ts, duration, pid, start,
> + end - start, from, to, prefetch_loc, preferred_loc,
> + trigger);
Would it make sense to have start and end events instead of a duration,
for consistency with the page fault events?
Regards,
Felix
> +
> + add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_MIGRATION, fifo_in, len);
> +}
> +
> int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
> {
> struct kfd_smi_client *client;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> index 7f70db914d2c..5788728f2879 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> @@ -34,4 +34,9 @@ void kfd_smi_event_page_fault_start(struct kfd_dev *dev, pid_t pid,
> void kfd_smi_event_page_fault_end(struct kfd_dev *dev, pid_t pid,
> unsigned long address, bool migration,
> uint64_t ts);
> +void kfd_smi_event_migration(struct kfd_dev *dev, uint16_t pasid,
> + unsigned long start, unsigned long end,
> + uint32_t from, uint32_t to,
> + uint32_t prefetch_loc, uint32_t preferred_loc,
> + uint32_t trigger, uint64_t ts);
> #endif
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> index 08b21f9759ea..7dbc724364e6 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> @@ -2791,7 +2791,8 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
> if (prange->actual_loc != best_loc) {
> migration = true;
> if (best_loc) {
> - r = svm_migrate_to_vram(prange, best_loc, mm);
> + r = svm_migrate_to_vram(prange, best_loc, mm,
> + MIGRATION_TRIGGER_PAGEFAULT);
> if (r) {
> pr_debug("svm_migrate_to_vram failed (%d) at %llx, falling back to system memory\n",
> r, addr);
> @@ -2799,12 +2800,14 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
> * VRAM failed
> */
> if (prange->actual_loc)
> - r = svm_migrate_vram_to_ram(prange, mm);
> + r = svm_migrate_vram_to_ram(prange, mm,
> + MIGRATION_TRIGGER_PAGEFAULT);
> else
> r = 0;
> }
> } else {
> - r = svm_migrate_vram_to_ram(prange, mm);
> + r = svm_migrate_vram_to_ram(prange, mm,
> + MIGRATION_TRIGGER_PAGEFAULT);
> }
> if (r) {
> pr_debug("failed %d to migrate svms %p [0x%lx 0x%lx]\n",
> @@ -3126,12 +3129,12 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange,
> return 0;
>
> if (!best_loc) {
> - r = svm_migrate_vram_to_ram(prange, mm);
> + r = svm_migrate_vram_to_ram(prange, mm, MIGRATION_TRIGGER_PREFETCH);
> *migrated = !r;
> return r;
> }
>
> - r = svm_migrate_to_vram(prange, best_loc, mm);
> + r = svm_migrate_to_vram(prange, best_loc, mm, MIGRATION_TRIGGER_PREFETCH);
> *migrated = !r;
>
> return r;
> @@ -3188,7 +3191,8 @@ static void svm_range_evict_svm_bo_worker(struct work_struct *work)
> mutex_lock(&prange->migrate_mutex);
> do {
> svm_migrate_vram_to_ram(prange,
> - svm_bo->eviction_fence->mm);
> + svm_bo->eviction_fence->mm,
> + MIGRATION_TRIGGER_TTM_EVICTION);
> } while (prange->actual_loc && --retries);
> WARN(prange->actual_loc, "Migration failed during eviction");
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 6/8] drm/amdkfd: Add user queue eviction restore SMI event
2022-01-20 23:13 ` [PATCH v2 6/8] drm/amdkfd: Add user queue eviction restore " Philip Yang
@ 2022-01-27 23:37 ` Felix Kuehling
0 siblings, 0 replies; 16+ messages in thread
From: Felix Kuehling @ 2022-01-27 23:37 UTC (permalink / raw)
To: Philip Yang, amd-gfx
Am 2022-01-20 um 18:13 schrieb Philip Yang:
> Output user queue eviction and restore event. User queue eviction may be
> triggered by migration, MMU notifier, TTM eviction or device suspend.
>
> User queue restore may be rescheduled if eviction happens again while
> restore.
>
> Signed-off-by: Philip Yang <Philip.Yang@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 7 +++-
> .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 11 ++++--
> drivers/gpu/drm/amd/amdkfd/kfd_device.c | 4 +-
> drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 +-
> drivers/gpu/drm/amd/amdkfd/kfd_process.c | 37 +++++++++++++++++--
> drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c | 34 +++++++++++++++++
> drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h | 4 ++
> drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 16 ++++++--
> 8 files changed, 101 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> index ac841ae8f5cc..bd3301e2c682 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> @@ -309,6 +309,7 @@ void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev,
> */
> void amdgpu_amdkfd_release_notify(struct amdgpu_bo *bo);
> void amdgpu_amdkfd_reserve_system_mem(uint64_t size);
> +void kfd_process_smi_event_restore_rescheduled(struct mm_struct *mm);
> #else
> static inline
> void amdgpu_amdkfd_gpuvm_init_mem_limits(void)
> @@ -325,9 +326,13 @@ static inline
> void amdgpu_amdkfd_release_notify(struct amdgpu_bo *bo)
> {
> }
> +
> +static inline void kfd_process_smi_event_restore_rescheduled(struct mm_struct *mm)
> +{
> +}
> #endif
> /* KGD2KFD callbacks */
> -int kgd2kfd_quiesce_mm(struct mm_struct *mm);
> +int kgd2kfd_quiesce_mm(struct mm_struct *mm, uint32_t trigger);
> int kgd2kfd_resume_mm(struct mm_struct *mm);
> int kgd2kfd_schedule_evict_and_restore_process(struct mm_struct *mm,
> struct dma_fence *fence);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index 5df387c4d7fb..c44e8dc0d869 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -2066,7 +2066,7 @@ int amdgpu_amdkfd_evict_userptr(struct kgd_mem *mem,
> evicted_bos = atomic_inc_return(&process_info->evicted_bos);
> if (evicted_bos == 1) {
> /* First eviction, stop the queues */
> - r = kgd2kfd_quiesce_mm(mm);
> + r = kgd2kfd_quiesce_mm(mm, USERPTR_EVICTION);
> if (r)
> pr_err("Failed to quiesce KFD\n");
> schedule_delayed_work(&process_info->restore_userptr_work,
> @@ -2340,13 +2340,16 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work)
>
> unlock_out:
> mutex_unlock(&process_info->lock);
> - mmput(mm);
> - put_task_struct(usertask);
>
> /* If validation failed, reschedule another attempt */
> - if (evicted_bos)
> + if (evicted_bos) {
> schedule_delayed_work(&process_info->restore_userptr_work,
> msecs_to_jiffies(AMDGPU_USERPTR_RESTORE_DELAY_MS));
> +
> + kfd_process_smi_event_restore_rescheduled(mm);
> + }
> + mmput(mm);
> + put_task_struct(usertask);
> }
>
> /** amdgpu_amdkfd_gpuvm_restore_process_bos - Restore all BOs for the given
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> index 5a47f437b455..ffaa80447d9c 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> @@ -783,7 +783,7 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry)
> spin_unlock_irqrestore(&kfd->interrupt_lock, flags);
> }
>
> -int kgd2kfd_quiesce_mm(struct mm_struct *mm)
> +int kgd2kfd_quiesce_mm(struct mm_struct *mm, uint32_t trigger)
> {
> struct kfd_process *p;
> int r;
> @@ -797,7 +797,7 @@ int kgd2kfd_quiesce_mm(struct mm_struct *mm)
> return -ESRCH;
>
> WARN(debug_evictions, "Evicting pid %d", p->lead_thread->pid);
> - r = kfd_process_evict_queues(p);
> + r = kfd_process_evict_queues(p, trigger);
>
> kfd_unref_process(p);
> return r;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> index ea68f3b3a4e9..39519084df78 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> @@ -906,7 +906,7 @@ static inline struct kfd_process_device *kfd_process_device_from_gpuidx(
> }
>
> void kfd_unref_process(struct kfd_process *p);
> -int kfd_process_evict_queues(struct kfd_process *p);
> +int kfd_process_evict_queues(struct kfd_process *p, uint32_t trigger);
> int kfd_process_restore_queues(struct kfd_process *p);
> void kfd_suspend_all_processes(void);
> int kfd_resume_all_processes(void);
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> index 74f162887d3b..e4ba4d537b3c 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> @@ -43,6 +43,7 @@ struct mm_struct;
> #include "kfd_dbgmgr.h"
> #include "kfd_iommu.h"
> #include "kfd_svm.h"
> +#include "kfd_smi_events.h"
>
> /*
> * List of struct kfd_process (field kfd_process).
> @@ -1701,7 +1702,7 @@ struct kfd_process *kfd_lookup_process_by_mm(const struct mm_struct *mm)
> * Eviction is reference-counted per process-device. This means multiple
> * evictions from different sources can be nested safely.
> */
> -int kfd_process_evict_queues(struct kfd_process *p)
> +int kfd_process_evict_queues(struct kfd_process *p, uint32_t trigger)
> {
> int r = 0;
> int i;
> @@ -1710,6 +1711,9 @@ int kfd_process_evict_queues(struct kfd_process *p)
> for (i = 0; i < p->n_pdds; i++) {
> struct kfd_process_device *pdd = p->pdds[i];
>
> + kfd_smi_event_queue_eviction(pdd->dev, p->lead_thread->pid,
> + trigger);
> +
If we're always evicting queues on all GPUs, we probably only need one
event per process, not one event per GPU.
> r = pdd->dev->dqm->ops.evict_process_queues(pdd->dev->dqm,
> &pdd->qpd);
> /* evict return -EIO if HWS is hang or asic is resetting, in this case
> @@ -1734,6 +1738,11 @@ int kfd_process_evict_queues(struct kfd_process *p)
>
> if (n_evicted == 0)
> break;
> +
> + kfd_smi_event_queue_eviction_restore(pdd->dev,
> + p->lead_thread->pid,
> + false);
> +
> if (pdd->dev->dqm->ops.restore_process_queues(pdd->dev->dqm,
> &pdd->qpd))
> pr_err("Failed to restore queues\n");
> @@ -1753,6 +1762,10 @@ int kfd_process_restore_queues(struct kfd_process *p)
> for (i = 0; i < p->n_pdds; i++) {
> struct kfd_process_device *pdd = p->pdds[i];
>
> + kfd_smi_event_queue_eviction_restore(pdd->dev,
> + p->lead_thread->pid,
> + false);
> +
> r = pdd->dev->dqm->ops.restore_process_queues(pdd->dev->dqm,
> &pdd->qpd);
> if (r) {
> @@ -1765,6 +1778,24 @@ int kfd_process_restore_queues(struct kfd_process *p)
> return ret;
> }
>
> +void kfd_process_smi_event_restore_rescheduled(struct mm_struct *mm)
> +{
> + struct kfd_process *p;
> + int i;
> +
> + p = kfd_lookup_process_by_mm(mm);
> + if (!p)
> + return;
> +
> + for (i = 0; i < p->n_pdds; i++) {
> + struct kfd_process_device *pdd = p->pdds[i];
> +
> + kfd_smi_event_queue_eviction_restore(pdd->dev,
> + p->lead_thread->pid, true);
Same as above. One event per process should be enough.
> + }
> + kfd_unref_process(p);
> +}
> +
> int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id)
> {
> int i;
> @@ -1814,7 +1845,7 @@ static void evict_process_worker(struct work_struct *work)
> flush_delayed_work(&p->restore_work);
>
> pr_debug("Started evicting pasid 0x%x\n", p->pasid);
> - ret = kfd_process_evict_queues(p);
> + ret = kfd_process_evict_queues(p, TTM_EVICTION);
> if (!ret) {
> dma_fence_signal(p->ef);
> dma_fence_put(p->ef);
> @@ -1881,7 +1912,7 @@ void kfd_suspend_all_processes(void)
> cancel_delayed_work_sync(&p->eviction_work);
> cancel_delayed_work_sync(&p->restore_work);
>
> - if (kfd_process_evict_queues(p))
> + if (kfd_process_evict_queues(p, SUSPEND_EVICTION))
> pr_err("Failed to suspend process 0x%x\n", p->pasid);
> dma_fence_signal(p->ef);
> dma_fence_put(p->ef);
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> index 97393f4f3549..facc8d7627d8 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
> @@ -343,6 +343,40 @@ void kfd_smi_event_migration(struct kfd_dev *dev, uint16_t pasid,
> add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_MIGRATION, fifo_in, len);
> }
>
> +void kfd_smi_event_queue_eviction(struct kfd_dev *dev, pid_t pid,
> + uint32_t trigger)
> +{
> + char fifo_in[64];
> + int len;
> +
> + if (list_empty(&dev->smi_clients))
> + return;
> +
> + len = snprintf(fifo_in, sizeof(fifo_in), "%x %lld -%d @%x %d\n",
> + KFD_SMI_EVENT_QUEUE_EVICTION, ktime_get_boottime_ns(),
> + pid, dev->id, trigger);
> +
> + add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_QUEUE_EVICTION, fifo_in, len);
> +}
> +
> +void kfd_smi_event_queue_eviction_restore(struct kfd_dev *dev, pid_t pid,
> + bool rescheduled)
> +{
> + char fifo_in[64];
> + int len;
> +
> + if (list_empty(&dev->smi_clients))
> + return;
> +
> + len = snprintf(fifo_in, sizeof(fifo_in), "%x %lld -%d @%x %c\n",
> + KFD_SMI_EVENT_QUEUE_EVICTION_RESTORE,
> + ktime_get_boottime_ns(), pid, dev->id,
> + rescheduled ? 'r' : ' ');
> +
> + add_event_to_kfifo(pid, dev, KFD_SMI_EVENT_QUEUE_EVICTION_RESTORE,
> + fifo_in, len);
> +}
> +
> int kfd_smi_event_open(struct kfd_dev *dev, uint32_t *fd)
> {
> struct kfd_smi_client *client;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> index 5788728f2879..d85300b5af23 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h
> @@ -39,4 +39,8 @@ void kfd_smi_event_migration(struct kfd_dev *dev, uint16_t pasid,
> uint32_t from, uint32_t to,
> uint32_t prefetch_loc, uint32_t preferred_loc,
> uint32_t trigger, uint64_t ts);
> +void kfd_smi_event_queue_eviction(struct kfd_dev *dev, pid_t pid,
> + uint32_t trigger);
> +void kfd_smi_event_queue_eviction_restore(struct kfd_dev *dev, pid_t pid,
> + bool rescheduled);
> #endif
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> index 7dbc724364e6..30aaa9764067 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> @@ -1712,7 +1712,10 @@ static void svm_range_restore_work(struct work_struct *work)
> pr_debug("reschedule to restore svm range\n");
> schedule_delayed_work(&svms->restore_work,
> msecs_to_jiffies(AMDGPU_SVM_RANGE_RESTORE_DELAY_MS));
> +
> + kfd_process_smi_event_restore_rescheduled(mm);
> }
> + mmput(mm);
Where is the matching mmget for this? If this is a fix for an old
reference-leak bug in the code, it deserves its own patch.
Regards,
Felix
> }
>
> /**
> @@ -1732,15 +1735,22 @@ static void svm_range_restore_work(struct work_struct *work)
> */
> static int
> svm_range_evict(struct svm_range *prange, struct mm_struct *mm,
> - unsigned long start, unsigned long last)
> + unsigned long start, unsigned long last,
> + enum mmu_notifier_event event)
> {
> struct svm_range_list *svms = prange->svms;
> struct svm_range *pchild;
> struct kfd_process *p;
> + uint32_t trigger;
> int r = 0;
>
> p = container_of(svms, struct kfd_process, svms);
>
> + if (event == MMU_NOTIFY_MIGRATE)
> + trigger = SVM_RANGE_MIGRATION;
> + else
> + trigger = SVM_RANGE_EVICTION;
> +
> pr_debug("invalidate svms 0x%p prange [0x%lx 0x%lx] [0x%lx 0x%lx]\n",
> svms, prange->start, prange->last, start, last);
>
> @@ -1768,7 +1778,7 @@ svm_range_evict(struct svm_range *prange, struct mm_struct *mm,
> prange->svms, prange->start, prange->last);
>
> /* First eviction, stop the queues */
> - r = kgd2kfd_quiesce_mm(mm);
> + r = kgd2kfd_quiesce_mm(mm, trigger);
> if (r)
> pr_debug("failed to quiesce KFD\n");
>
> @@ -2303,7 +2313,7 @@ svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni,
> svm_range_unmap_from_cpu(mni->mm, prange, start, last);
> break;
> default:
> - svm_range_evict(prange, mni->mm, start, last);
> + svm_range_evict(prange, mni->mm, start, last, range->event);
> break;
> }
>
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2022-01-27 23:37 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-20 23:13 [PATCH v2 0/8] HMM profiler interface Philip Yang
2022-01-20 23:13 ` [PATCH v2 1/8] drm/amdkfd: Correct SMI event read size Philip Yang
2022-01-25 16:24 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 2/8] drm/amdkfd: Add KFD SMI event IDs and triggers Philip Yang
2022-01-25 17:05 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 3/8] drm/amdkfd: Enable per process SMI event Philip Yang
2022-01-27 22:59 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 4/8] drm/amdkfd: Add GPU recoverable fault " Philip Yang
2022-01-27 23:15 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 5/8] drm/amdkfd: add migration " Philip Yang
2022-01-27 23:29 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 6/8] drm/amdkfd: Add user queue eviction restore " Philip Yang
2022-01-27 23:37 ` Felix Kuehling
2022-01-20 23:13 ` [PATCH v2 7/8] drm/amdkfd: Add unmap from GPU " Philip Yang
2022-01-20 23:13 ` [PATCH v2 8/8] drm/amdkfd: Bump KFD API version for SMI profiling event Philip Yang
2022-01-21 18:49 ` [PATCH v2 0/8] HMM profiler interface Deucher, Alexander
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.