[PATCH 0/6] Raven support for KFD v2

[PATCH 0/6] Raven support for KFD v2

[Felix Kuehling]

Raven refers to Ryzen APUs with integrated GFXv9 GPU.
This patch series completes Raven support for KFD:

* fix up memory banks info from CRAT
* support different number of SDMA engines
* workaround IOMMUv2 PPR issues
* add device info

v2: Updated patch descriptions, no code changes

Yong Zhao (6):
  drm/amdkfd: Consolidate duplicate memory banks info in topology
  drm/amdkfd: Make SDMA engine number an ASIC-dependent variable
  drm/amdkfd: Avoid flooding dmesg on Raven due to IOMMU issues
  drm/amdkfd: Workaround to accommodate Raven too many PPR issue
  drm/amdkfd: Optimize out some duplicated code in
    kfd_signal_iommu_event()
  drm/amdkfd: Enable Raven for KFD

 drivers/gpu/drm/amd/amdkfd/kfd_crat.c              | 57 +++++++++++++++++-----
 drivers/gpu/drm/amd/amdkfd/kfd_device.c            | 28 +++++++++++
 .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c  | 29 +++++++----
 .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.h  |  6 +--
 drivers/gpu/drm/amd/amdkfd/kfd_events.c            | 47 ++++++++++--------
 drivers/gpu/drm/amd/amdkfd/kfd_iommu.c             |  2 +-
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h              |  1 +
 .../gpu/drm/amd/amdkfd/kfd_process_queue_manager.c |  3 +-
 8 files changed, 126 insertions(+), 47 deletions(-)

-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 1/6] drm/amdkfd: Consolidate duplicate memory banks info in topology

[Felix Kuehling]

From: Yong Zhao <yong.zhao@amd.com>

If there are several memory banks that has the same properties in CRAT,
we aggregate them into one memory bank. This cleans up memory banks on
APUs (e.g. Raven) where the CRAT reports each memory channel as a
separate bank. This only confuses user mode, which only deals with
virtual memory.

Signed-off-by: Yong Zhao <yong.zhao@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 57 ++++++++++++++++++++++++++++-------
 1 file changed, 46 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 296b3f2..ee49960 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -189,6 +189,21 @@ static int kfd_parse_subtype_cu(struct crat_subtype_computeunit *cu,
 	return 0;
 }
 
+static struct kfd_mem_properties *
+find_subtype_mem(uint32_t heap_type, uint32_t flags, uint32_t width,
+		struct kfd_topology_device *dev)
+{
+	struct kfd_mem_properties *props;
+
+	list_for_each_entry(props, &dev->mem_props, list) {
+		if (props->heap_type == heap_type
+				&& props->flags == flags
+				&& props->width == width)
+			return props;
+	}
+
+	return NULL;
+}
 /* kfd_parse_subtype_mem - parse memory subtypes and attach it to correct
  * topology device present in the device_list
  */
@@ -197,36 +212,56 @@ static int kfd_parse_subtype_mem(struct crat_subtype_memory *mem,
 {
 	struct kfd_mem_properties *props;
 	struct kfd_topology_device *dev;
+	uint32_t heap_type;
+	uint64_t size_in_bytes;
+	uint32_t flags = 0;
+	uint32_t width;
 
 	pr_debug("Found memory entry in CRAT table with proximity_domain=%d\n",
 			mem->proximity_domain);
 	list_for_each_entry(dev, device_list, list) {
 		if (mem->proximity_domain == dev->proximity_domain) {
-			props = kfd_alloc_struct(props);
-			if (!props)
-				return -ENOMEM;
-
 			/* We're on GPU node */
 			if (dev->node_props.cpu_cores_count == 0) {
 				/* APU */
 				if (mem->visibility_type == 0)
-					props->heap_type =
+					heap_type =
 						HSA_MEM_HEAP_TYPE_FB_PRIVATE;
 				/* dGPU */
 				else
-					props->heap_type = mem->visibility_type;
+					heap_type = mem->visibility_type;
 			} else
-				props->heap_type = HSA_MEM_HEAP_TYPE_SYSTEM;
+				heap_type = HSA_MEM_HEAP_TYPE_SYSTEM;
 
 			if (mem->flags & CRAT_MEM_FLAGS_HOT_PLUGGABLE)
-				props->flags |= HSA_MEM_FLAGS_HOT_PLUGGABLE;
+				flags |= HSA_MEM_FLAGS_HOT_PLUGGABLE;
 			if (mem->flags & CRAT_MEM_FLAGS_NON_VOLATILE)
-				props->flags |= HSA_MEM_FLAGS_NON_VOLATILE;
+				flags |= HSA_MEM_FLAGS_NON_VOLATILE;
 
-			props->size_in_bytes =
+			size_in_bytes =
 				((uint64_t)mem->length_high << 32) +
 							mem->length_low;
-			props->width = mem->width;
+			width = mem->width;
+
+			/* Multiple banks of the same type are aggregated into
+			 * one. User mode doesn't care about multiple physical
+			 * memory segments. It's managed as a single virtual
+			 * heap for user mode.
+			 */
+			props = find_subtype_mem(heap_type, flags, width, dev);
+			if (props) {
+				props->size_in_bytes += size_in_bytes;
+				break;
+			}
+
+			props = kfd_alloc_struct(props);
+			if (!props)
+				return -ENOMEM;
+
+			props->heap_type = heap_type;
+			props->flags = flags;
+			props->size_in_bytes = size_in_bytes;
+			props->width = width;
 
 			dev->node_props.mem_banks_count++;
 			list_add_tail(&props->list, &dev->mem_props);
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 2/6] drm/amdkfd: Make SDMA engine number an ASIC-dependent variable

[Felix Kuehling]

From: Yong Zhao <yong.zhao@amd.com>

On Raven there is only one SDMA engine instead of previously assumed two,
so we need to adapt our code to this new scenario.

Signed-off-by: Yong Zhao <yong.zhao@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_device.c            | 12 +++++++++
 .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c  | 29 +++++++++++++++-------
 .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.h  |  6 ++---
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h              |  1 +
 .../gpu/drm/amd/amdkfd/kfd_process_queue_manager.c |  3 +--
 5 files changed, 36 insertions(+), 15 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index 8faa8db..572235c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -52,6 +52,7 @@ static const struct kfd_device_info kaveri_device_info = {
 	.supports_cwsr = false,
 	.needs_iommu_device = true,
 	.needs_pci_atomics = false,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info carrizo_device_info = {
@@ -67,6 +68,7 @@ static const struct kfd_device_info carrizo_device_info = {
 	.supports_cwsr = true,
 	.needs_iommu_device = true,
 	.needs_pci_atomics = false,
+	.num_sdma_engines = 2,
 };
 #endif
 
@@ -83,6 +85,7 @@ static const struct kfd_device_info hawaii_device_info = {
 	.supports_cwsr = false,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = false,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info tonga_device_info = {
@@ -97,6 +100,7 @@ static const struct kfd_device_info tonga_device_info = {
 	.supports_cwsr = false,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = true,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info tonga_vf_device_info = {
@@ -111,6 +115,7 @@ static const struct kfd_device_info tonga_vf_device_info = {
 	.supports_cwsr = false,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = false,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info fiji_device_info = {
@@ -125,6 +130,7 @@ static const struct kfd_device_info fiji_device_info = {
 	.supports_cwsr = true,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = true,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info fiji_vf_device_info = {
@@ -139,6 +145,7 @@ static const struct kfd_device_info fiji_vf_device_info = {
 	.supports_cwsr = true,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = false,
+	.num_sdma_engines = 2,
 };
 
 
@@ -154,6 +161,7 @@ static const struct kfd_device_info polaris10_device_info = {
 	.supports_cwsr = true,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = true,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info polaris10_vf_device_info = {
@@ -168,6 +176,7 @@ static const struct kfd_device_info polaris10_vf_device_info = {
 	.supports_cwsr = true,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = false,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info polaris11_device_info = {
@@ -182,6 +191,7 @@ static const struct kfd_device_info polaris11_device_info = {
 	.supports_cwsr = true,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = true,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info vega10_device_info = {
@@ -196,6 +206,7 @@ static const struct kfd_device_info vega10_device_info = {
 	.supports_cwsr = true,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = false,
+	.num_sdma_engines = 2,
 };
 
 static const struct kfd_device_info vega10_vf_device_info = {
@@ -210,6 +221,7 @@ static const struct kfd_device_info vega10_vf_device_info = {
 	.supports_cwsr = true,
 	.needs_iommu_device = false,
 	.needs_pci_atomics = false,
+	.num_sdma_engines = 2,
 };
 
 
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index 97c9f10..ace94d6 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -101,6 +101,17 @@ unsigned int get_pipes_per_mec(struct device_queue_manager *dqm)
 	return dqm->dev->shared_resources.num_pipe_per_mec;
 }
 
+static unsigned int get_num_sdma_engines(struct device_queue_manager *dqm)
+{
+	return dqm->dev->device_info->num_sdma_engines;
+}
+
+unsigned int get_num_sdma_queues(struct device_queue_manager *dqm)
+{
+	return dqm->dev->device_info->num_sdma_engines
+			* KFD_SDMA_QUEUES_PER_ENGINE;
+}
+
 void program_sh_mem_settings(struct device_queue_manager *dqm,
 					struct qcm_process_device *qpd)
 {
@@ -855,7 +866,7 @@ static int initialize_nocpsch(struct device_queue_manager *dqm)
 	}
 
 	dqm->vmid_bitmap = (1 << dqm->dev->vm_info.vmid_num_kfd) - 1;
-	dqm->sdma_bitmap = (1 << CIK_SDMA_QUEUES) - 1;
+	dqm->sdma_bitmap = (1 << get_num_sdma_queues(dqm)) - 1;
 
 	return 0;
 }
@@ -903,7 +914,7 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm,
 static void deallocate_sdma_queue(struct device_queue_manager *dqm,
 				unsigned int sdma_queue_id)
 {
-	if (sdma_queue_id >= CIK_SDMA_QUEUES)
+	if (sdma_queue_id >= get_num_sdma_queues(dqm))
 		return;
 	dqm->sdma_bitmap |= (1 << sdma_queue_id);
 }
@@ -923,8 +934,8 @@ static int create_sdma_queue_nocpsch(struct device_queue_manager *dqm,
 	if (retval)
 		return retval;
 
-	q->properties.sdma_queue_id = q->sdma_id / CIK_SDMA_QUEUES_PER_ENGINE;
-	q->properties.sdma_engine_id = q->sdma_id % CIK_SDMA_QUEUES_PER_ENGINE;
+	q->properties.sdma_queue_id = q->sdma_id / get_num_sdma_engines(dqm);
+	q->properties.sdma_engine_id = q->sdma_id % get_num_sdma_engines(dqm);
 
 	retval = allocate_doorbell(qpd, q);
 	if (retval)
@@ -1011,7 +1022,7 @@ static int initialize_cpsch(struct device_queue_manager *dqm)
 	dqm->queue_count = dqm->processes_count = 0;
 	dqm->sdma_queue_count = 0;
 	dqm->active_runlist = false;
-	dqm->sdma_bitmap = (1 << CIK_SDMA_QUEUES) - 1;
+	dqm->sdma_bitmap = (1 << get_num_sdma_queues(dqm)) - 1;
 
 	INIT_WORK(&dqm->hw_exception_work, kfd_process_hw_exception);
 
@@ -1142,9 +1153,9 @@ static int create_queue_cpsch(struct device_queue_manager *dqm, struct queue *q,
 		if (retval)
 			goto out_unlock;
 		q->properties.sdma_queue_id =
-			q->sdma_id / CIK_SDMA_QUEUES_PER_ENGINE;
+			q->sdma_id / get_num_sdma_engines(dqm);
 		q->properties.sdma_engine_id =
-			q->sdma_id % CIK_SDMA_QUEUES_PER_ENGINE;
+			q->sdma_id % get_num_sdma_engines(dqm);
 	}
 
 	retval = allocate_doorbell(qpd, q);
@@ -1791,8 +1802,8 @@ int dqm_debugfs_hqds(struct seq_file *m, void *data)
 		}
 	}
 
-	for (pipe = 0; pipe < CIK_SDMA_ENGINE_NUM; pipe++) {
-		for (queue = 0; queue < CIK_SDMA_QUEUES_PER_ENGINE; queue++) {
+	for (pipe = 0; pipe < get_num_sdma_engines(dqm); pipe++) {
+		for (queue = 0; queue < KFD_SDMA_QUEUES_PER_ENGINE; queue++) {
 			r = dqm->dev->kfd2kgd->hqd_sdma_dump(
 				dqm->dev->kgd, pipe, queue, &dump, &n_regs);
 			if (r)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
index 52e708c..00da316 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
@@ -33,10 +33,7 @@
 
 #define KFD_UNMAP_LATENCY_MS			(4000)
 #define QUEUE_PREEMPT_DEFAULT_TIMEOUT_MS (2 * KFD_UNMAP_LATENCY_MS + 1000)
-
-#define CIK_SDMA_QUEUES				(4)
-#define CIK_SDMA_QUEUES_PER_ENGINE		(2)
-#define CIK_SDMA_ENGINE_NUM			(2)
+#define KFD_SDMA_QUEUES_PER_ENGINE		(2)
 
 struct device_process_node {
 	struct qcm_process_device *qpd;
@@ -214,6 +211,7 @@ void program_sh_mem_settings(struct device_queue_manager *dqm,
 unsigned int get_queues_num(struct device_queue_manager *dqm);
 unsigned int get_queues_per_pipe(struct device_queue_manager *dqm);
 unsigned int get_pipes_per_mec(struct device_queue_manager *dqm);
+unsigned int get_num_sdma_queues(struct device_queue_manager *dqm);
 
 static inline unsigned int get_sh_mem_bases_32(struct kfd_process_device *pdd)
 {
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index 37d179e..ca83254 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -203,6 +203,7 @@ struct kfd_device_info {
 	bool supports_cwsr;
 	bool needs_iommu_device;
 	bool needs_pci_atomics;
+	unsigned int num_sdma_engines;
 };
 
 struct kfd_mem_obj {
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
index 1303b14..eb4e5fb 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
@@ -186,8 +186,7 @@ int pqm_create_queue(struct process_queue_manager *pqm,
 
 	switch (type) {
 	case KFD_QUEUE_TYPE_SDMA:
-		if (dev->dqm->queue_count >=
-			CIK_SDMA_QUEUES_PER_ENGINE * CIK_SDMA_ENGINE_NUM) {
+		if (dev->dqm->queue_count >= get_num_sdma_queues(dev->dqm)) {
 			pr_err("Over-subscription is not allowed for SDMA.\n");
 			retval = -EPERM;
 			goto err_create_queue;
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 3/6] drm/amdkfd: Avoid flooding dmesg on Raven due to IOMMU issues

[Felix Kuehling]

From: Yong Zhao <yong.zhao@amd.com>

On Raven Invalid PPRs (peripheral page requests) can be reported
because multiple PPRs can be still queued when memory is freed.
Apply a rate limit to avoid flooding the log in this case.

Signed-off-by: Yong Zhao <yong.zhao@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_iommu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_iommu.c b/drivers/gpu/drm/amd/amdkfd/kfd_iommu.c
index c718179..7a61f38 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_iommu.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_iommu.c
@@ -190,7 +190,7 @@ static int iommu_invalid_ppr_cb(struct pci_dev *pdev, int pasid,
 {
 	struct kfd_dev *dev;
 
-	dev_warn(kfd_device,
+	dev_warn_ratelimited(kfd_device,
 			"Invalid PPR device %x:%x.%x pasid %d address 0x%lX flags 0x%X",
 			PCI_BUS_NUM(pdev->devfn),
 			PCI_SLOT(pdev->devfn),
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 4/6] drm/amdkfd: Workaround to accommodate Raven too many PPR issue

[Felix Kuehling]

From: Yong Zhao <yong.zhao@amd.com>

On Raven multiple PPRs can be queued up by the hardware. When the
first of those requests is handled by the IOMMU driver, the memory
access succeeds. After that the application may be done with the
memory and unmap it. At that point the page table entries are
invalidated, but there are still outstanding duplicate PPRs for those
addresses. When the IOMMU driver processes those duplicate requests,
it finds invalid page table entries and triggers an invalid PPR fault.

As a workaround, don't signal invalid PPR faults on Raven to avoid
segfaulting applications that haven't done anything wrong. As a side
effect, real GPU memory access faults may go unnoticed by the
application.

Signed-off-by: Yong Zhao <yong.zhao@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_events.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
index 820133c..4dcacce 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
@@ -932,13 +932,24 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid,
 	up_read(&mm->mmap_sem);
 	mmput(mm);
 
-	mutex_lock(&p->event_mutex);
+	pr_debug("notpresent %d, noexecute %d, readonly %d\n",
+			memory_exception_data.failure.NotPresent,
+			memory_exception_data.failure.NoExecute,
+			memory_exception_data.failure.ReadOnly);
 
-	/* Lookup events by type and signal them */
-	lookup_events_by_type_and_signal(p, KFD_EVENT_TYPE_MEMORY,
-			&memory_exception_data);
+	/* Workaround on Raven to not kill the process when memory is freed
+	 * before IOMMU is able to finish processing all the excessive PPRs
+	 */
+	if (dev->device_info->asic_family != CHIP_RAVEN) {
+		mutex_lock(&p->event_mutex);
+
+		/* Lookup events by type and signal them */
+		lookup_events_by_type_and_signal(p, KFD_EVENT_TYPE_MEMORY,
+				&memory_exception_data);
+
+		mutex_unlock(&p->event_mutex);
+	}
 
-	mutex_unlock(&p->event_mutex);
 	kfd_unref_process(p);
 }
 #endif /* KFD_SUPPORT_IOMMU_V2 */
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 5/6] drm/amdkfd: Optimize out some duplicated code in kfd_signal_iommu_event()

[Felix Kuehling]

From: Yong Zhao <yong.zhao@amd.com>

memory_exception_data is already initialized for not-present faults.
It only needs to be overridden for permission faults.

Signed-off-by: Yong Zhao <yong.zhao@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_events.c | 26 +++++++++++---------------
 1 file changed, 11 insertions(+), 15 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
index 4dcacce..e9f0e0a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
@@ -911,22 +911,18 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid,
 	memory_exception_data.failure.NotPresent = 1;
 	memory_exception_data.failure.NoExecute = 0;
 	memory_exception_data.failure.ReadOnly = 0;
-	if (vma) {
-		if (vma->vm_start > address) {
-			memory_exception_data.failure.NotPresent = 1;
-			memory_exception_data.failure.NoExecute = 0;
+	if (vma && address >= vma->vm_start) {
+		memory_exception_data.failure.NotPresent = 0;
+
+		if (is_write_requested && !(vma->vm_flags & VM_WRITE))
+			memory_exception_data.failure.ReadOnly = 1;
+		else
 			memory_exception_data.failure.ReadOnly = 0;
-		} else {
-			memory_exception_data.failure.NotPresent = 0;
-			if (is_write_requested && !(vma->vm_flags & VM_WRITE))
-				memory_exception_data.failure.ReadOnly = 1;
-			else
-				memory_exception_data.failure.ReadOnly = 0;
-			if (is_execute_requested && !(vma->vm_flags & VM_EXEC))
-				memory_exception_data.failure.NoExecute = 1;
-			else
-				memory_exception_data.failure.NoExecute = 0;
-		}
+
+		if (is_execute_requested && !(vma->vm_flags & VM_EXEC))
+			memory_exception_data.failure.NoExecute = 1;
+		else
+			memory_exception_data.failure.NoExecute = 0;
 	}
 
 	up_read(&mm->mmap_sem);
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 6/6] drm/amdkfd: Enable Raven for KFD

[Felix Kuehling]

From: Yong Zhao <Yong.Zhao@amd.com>

Add DID and kfd_device_info for Raven.

Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_device.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index 572235c..1b04871 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -70,6 +70,21 @@ static const struct kfd_device_info carrizo_device_info = {
 	.needs_pci_atomics = false,
 	.num_sdma_engines = 2,
 };
+
+static const struct kfd_device_info raven_device_info = {
+	.asic_family = CHIP_RAVEN,
+	.max_pasid_bits = 16,
+	.max_no_of_hqd  = 24,
+	.doorbell_size  = 8,
+	.ih_ring_entry_size = 8 * sizeof(uint32_t),
+	.event_interrupt_class = &event_interrupt_class_v9,
+	.num_of_watch_points = 4,
+	.mqd_size_aligned = MQD_SIZE_ALIGNED,
+	.supports_cwsr = true,
+	.needs_iommu_device = true,
+	.needs_pci_atomics = true,
+	.num_sdma_engines = 1,
+};
 #endif
 
 static const struct kfd_device_info hawaii_device_info = {
@@ -259,6 +274,7 @@ static const struct kfd_deviceid supported_devices[] = {
 	{ 0x9875, &carrizo_device_info },	/* Carrizo */
 	{ 0x9876, &carrizo_device_info },	/* Carrizo */
 	{ 0x9877, &carrizo_device_info },	/* Carrizo */
+	{ 0x15DD, &raven_device_info },		/* Raven */
 #endif
 	{ 0x67A0, &hawaii_device_info },	/* Hawaii */
 	{ 0x67A1, &hawaii_device_info },	/* Hawaii */
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] Raven support for KFD v2

[Alex Deucher]

On Fri, Jul 13, 2018 at 4:17 PM, Felix Kuehling <Felix.Kuehling@amd.com> wrote:
> Raven refers to Ryzen APUs with integrated GFXv9 GPU.
> This patch series completes Raven support for KFD:
>
> * fix up memory banks info from CRAT
> * support different number of SDMA engines
> * workaround IOMMUv2 PPR issues
> * add device info
>
> v2: Updated patch descriptions, no code changes

Looks good. For the remaining patches:
Acked-by: Alex Deucher <alexander.deucher@amd.com>

>
> Yong Zhao (6):
>   drm/amdkfd: Consolidate duplicate memory banks info in topology
>   drm/amdkfd: Make SDMA engine number an ASIC-dependent variable
>   drm/amdkfd: Avoid flooding dmesg on Raven due to IOMMU issues
>   drm/amdkfd: Workaround to accommodate Raven too many PPR issue
>   drm/amdkfd: Optimize out some duplicated code in
>     kfd_signal_iommu_event()
>   drm/amdkfd: Enable Raven for KFD
>
>  drivers/gpu/drm/amd/amdkfd/kfd_crat.c              | 57 +++++++++++++++++-----
>  drivers/gpu/drm/amd/amdkfd/kfd_device.c            | 28 +++++++++++
>  .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c  | 29 +++++++----
>  .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.h  |  6 +--
>  drivers/gpu/drm/amd/amdkfd/kfd_events.c            | 47 ++++++++++--------
>  drivers/gpu/drm/amd/amdkfd/kfd_iommu.c             |  2 +-
>  drivers/gpu/drm/amd/amdkfd/kfd_priv.h              |  1 +
>  .../gpu/drm/amd/amdkfd/kfd_process_queue_manager.c |  3 +-
>  8 files changed, 126 insertions(+), 47 deletions(-)
>
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] Raven support for KFD v2

[Oded Gabbay]

Hi Felix,
Thanks for the patch-set. Applied to -next.

Oded
On Fri, Jul 13, 2018 at 11:24 PM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Fri, Jul 13, 2018 at 4:17 PM, Felix Kuehling <Felix.Kuehling@amd.com> wrote:
> > Raven refers to Ryzen APUs with integrated GFXv9 GPU.
> > This patch series completes Raven support for KFD:
> >
> > * fix up memory banks info from CRAT
> > * support different number of SDMA engines
> > * workaround IOMMUv2 PPR issues
> > * add device info
> >
> > v2: Updated patch descriptions, no code changes
>
> Looks good. For the remaining patches:
> Acked-by: Alex Deucher <alexander.deucher@amd.com>
>
> >
> > Yong Zhao (6):
> >   drm/amdkfd: Consolidate duplicate memory banks info in topology
> >   drm/amdkfd: Make SDMA engine number an ASIC-dependent variable
> >   drm/amdkfd: Avoid flooding dmesg on Raven due to IOMMU issues
> >   drm/amdkfd: Workaround to accommodate Raven too many PPR issue
> >   drm/amdkfd: Optimize out some duplicated code in
> >     kfd_signal_iommu_event()
> >   drm/amdkfd: Enable Raven for KFD
> >
> >  drivers/gpu/drm/amd/amdkfd/kfd_crat.c              | 57 +++++++++++++++++-----
> >  drivers/gpu/drm/amd/amdkfd/kfd_device.c            | 28 +++++++++++
> >  .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c  | 29 +++++++----
> >  .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.h  |  6 +--
> >  drivers/gpu/drm/amd/amdkfd/kfd_events.c            | 47 ++++++++++--------
> >  drivers/gpu/drm/amd/amdkfd/kfd_iommu.c             |  2 +-
> >  drivers/gpu/drm/amd/amdkfd/kfd_priv.h              |  1 +
> >  .../gpu/drm/amd/amdkfd/kfd_process_queue_manager.c |  3 +-
> >  8 files changed, 126 insertions(+), 47 deletions(-)
> >
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 6/6] drm/amdkfd: Enable Raven for KFD

[Paul Menzel]

[-- Attachment #1.1: Type: text/plain, Size: 6361 bytes --]

Dear Felix, dear Yong,


On 07/13/18 22:17, Felix Kühling wrote:
> From: Yong Zhao <Yong.Zhao-5C7GfCeVMHo@public.gmane.org>
> 
> Add DID and kfd_device_info for Raven.
> 
> Signed-off-by: Yong Zhao <Yong.Zhao-5C7GfCeVMHo@public.gmane.org>
> Reviewed-by: Felix Kuehling <Felix.Kuehling-5C7GfCeVMHo@public.gmane.org>
> Signed-off-by: Felix Kuehling <Felix.Kuehling-5C7GfCeVMHo@public.gmane.org>
> Acked-by: Alex Deucher <alexander.deucher-5C7GfCeVMHo@public.gmane.org>
> ---
>  drivers/gpu/drm/amd/amdkfd/kfd_device.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> index 572235c..1b04871 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> @@ -70,6 +70,21 @@ static const struct kfd_device_info carrizo_device_info = {
>  	.needs_pci_atomics = false,
>  	.num_sdma_engines = 2,
>  };
> +
> +static const struct kfd_device_info raven_device_info = {
> +	.asic_family = CHIP_RAVEN,
> +	.max_pasid_bits = 16,
> +	.max_no_of_hqd  = 24,
> +	.doorbell_size  = 8,
> +	.ih_ring_entry_size = 8 * sizeof(uint32_t),
> +	.event_interrupt_class = &event_interrupt_class_v9,
> +	.num_of_watch_points = 4,
> +	.mqd_size_aligned = MQD_SIZE_ALIGNED,
> +	.supports_cwsr = true,
> +	.needs_iommu_device = true,
> +	.needs_pci_atomics = true,
> +	.num_sdma_engines = 1,
> +};
>  #endif
>  
>  static const struct kfd_device_info hawaii_device_info = {
> @@ -259,6 +274,7 @@ static const struct kfd_deviceid supported_devices[] = {
>  	{ 0x9875, &carrizo_device_info },	/* Carrizo */
>  	{ 0x9876, &carrizo_device_info },	/* Carrizo */
>  	{ 0x9877, &carrizo_device_info },	/* Carrizo */
> +	{ 0x15DD, &raven_device_info },		/* Raven */
>  #endif
>  	{ 0x67A0, &hawaii_device_info },	/* Hawaii */
>  	{ 0x67A1, &hawaii_device_info },	/* Hawaii */


I built the branch *amdkfd-next* from [1], but on the MIS B350M MORTAR (MS-7A37)
with BIOS 1.G1 05/17/2018 it is not loaded.

```
$ uname -a
Linux tokeiihto 4.18.0-rc3-01368-gb5aa3f4aef72 #51 SMP Mon Jul 30 13:11:36 UTC 2018 x86_64 GNU/Linux
$ grep HSA /boot/config-4.18.0-rc3-01368-gb5aa3f4aef72
CONFIG_HSA_AMD=m
$ sudo dmesg | grep -i kfd
[   23.206628] calling  kfd_module_init+0x0/0x1000 [amdkfd] @ 365
[   23.209082] kfd kfd: Initialized module
[   23.209667] initcall kfd_module_init+0x0/0x1000 [amdkfd] returned 0 after 2960 usecs
[   23.212852] kfd kfd: DID 15dd is missing in supported_devices
[   23.212856] kfd kfd: kgd2kfd_probe failed
$ /sbin/modinfo amdkfd
filename:       /lib/modules/4.18.0-rc3-01368-gb5aa3f4aef72/kernel/drivers/gpu/drm/amd/amdkfd/amdkfd.ko
version:        0.7.2
license:        GPL and additional rights
description:    Standalone HSA driver for AMD's GPUs
author:         AMD Inc. and others
srcversion:     9AAD0B6E4510ACF8D20A888
depends:
retpoline:      Y 
intree:         Y 
name:           amdkfd
vermagic:       4.18.0-rc3-01368-gb5aa3f4aef72 SMP mod_unload 
sig_id:         PKCS#7
[…]
```

Any hints, what I am doing wrong?


Kind regards,

Paul


PS: Output of lspci:

```
$ lspci -nn
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d0]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d1]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d3]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15db]
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15dc]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e8]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e9]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ea]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15eb]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ec]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ed]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ee]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ef]
15:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset USB 3.1 xHCI Controller [1022:43bb] (rev 02)
15:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset SATA Controller [1022:43b7] (rev 02)
15:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
16:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
16:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
16:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
18:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
38:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] [1002:15dd] (rev c8)
38:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:15de]
38:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:15df]
38:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e0]
38:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e1]
38:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e3]
39:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 61)
```


[1]: git://people.freedesktop.org/~gabbayo/linux.git


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 6/6] drm/amdkfd: Enable Raven for KFD

[Oded Gabbay]

Do you have IOMMUv2 enabled on this chipset ?
If not, amdkfd won't include raven support.

Oded
On Mon, Jul 30, 2018 at 4:33 PM Paul Menzel
<pmenzel+amd-gfx@molgen.mpg.de> wrote:
>
> Dear Felix, dear Yong,
>
>
> On 07/13/18 22:17, Felix Kühling wrote:
> > From: Yong Zhao <Yong.Zhao@amd.com>
> >
> > Add DID and kfd_device_info for Raven.
> >
> > Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
> > Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
> > Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
> > Acked-by: Alex Deucher <alexander.deucher@amd.com>
> > ---
> >  drivers/gpu/drm/amd/amdkfd/kfd_device.c | 16 ++++++++++++++++
> >  1 file changed, 16 insertions(+)
> >
> > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> > index 572235c..1b04871 100644
> > --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> > @@ -70,6 +70,21 @@ static const struct kfd_device_info carrizo_device_info = {
> >       .needs_pci_atomics = false,
> >       .num_sdma_engines = 2,
> >  };
> > +
> > +static const struct kfd_device_info raven_device_info = {
> > +     .asic_family = CHIP_RAVEN,
> > +     .max_pasid_bits = 16,
> > +     .max_no_of_hqd  = 24,
> > +     .doorbell_size  = 8,
> > +     .ih_ring_entry_size = 8 * sizeof(uint32_t),
> > +     .event_interrupt_class = &event_interrupt_class_v9,
> > +     .num_of_watch_points = 4,
> > +     .mqd_size_aligned = MQD_SIZE_ALIGNED,
> > +     .supports_cwsr = true,
> > +     .needs_iommu_device = true,
> > +     .needs_pci_atomics = true,
> > +     .num_sdma_engines = 1,
> > +};
> >  #endif
> >
> >  static const struct kfd_device_info hawaii_device_info = {
> > @@ -259,6 +274,7 @@ static const struct kfd_deviceid supported_devices[] = {
> >       { 0x9875, &carrizo_device_info },       /* Carrizo */
> >       { 0x9876, &carrizo_device_info },       /* Carrizo */
> >       { 0x9877, &carrizo_device_info },       /* Carrizo */
> > +     { 0x15DD, &raven_device_info },         /* Raven */
> >  #endif
> >       { 0x67A0, &hawaii_device_info },        /* Hawaii */
> >       { 0x67A1, &hawaii_device_info },        /* Hawaii */
>
>
> I built the branch *amdkfd-next* from [1], but on the MIS B350M MORTAR (MS-7A37)
> with BIOS 1.G1 05/17/2018 it is not loaded.
>
> ```
> $ uname -a
> Linux tokeiihto 4.18.0-rc3-01368-gb5aa3f4aef72 #51 SMP Mon Jul 30 13:11:36 UTC 2018 x86_64 GNU/Linux
> $ grep HSA /boot/config-4.18.0-rc3-01368-gb5aa3f4aef72
> CONFIG_HSA_AMD=m
> $ sudo dmesg | grep -i kfd
> [   23.206628] calling  kfd_module_init+0x0/0x1000 [amdkfd] @ 365
> [   23.209082] kfd kfd: Initialized module
> [   23.209667] initcall kfd_module_init+0x0/0x1000 [amdkfd] returned 0 after 2960 usecs
> [   23.212852] kfd kfd: DID 15dd is missing in supported_devices
> [   23.212856] kfd kfd: kgd2kfd_probe failed
> $ /sbin/modinfo amdkfd
> filename:       /lib/modules/4.18.0-rc3-01368-gb5aa3f4aef72/kernel/drivers/gpu/drm/amd/amdkfd/amdkfd.ko
> version:        0.7.2
> license:        GPL and additional rights
> description:    Standalone HSA driver for AMD's GPUs
> author:         AMD Inc. and others
> srcversion:     9AAD0B6E4510ACF8D20A888
> depends:
> retpoline:      Y
> intree:         Y
> name:           amdkfd
> vermagic:       4.18.0-rc3-01368-gb5aa3f4aef72 SMP mod_unload
> sig_id:         PKCS#7
> […]
> ```
>
> Any hints, what I am doing wrong?
>
>
> Kind regards,
>
> Paul
>
>
> PS: Output of lspci:
>
> ```
> $ lspci -nn
> 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d0]
> 00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d1]
> 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
> 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d3]
> 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
> 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15db]
> 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15dc]
> 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
> 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
> 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e8]
> 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e9]
> 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ea]
> 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15eb]
> 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ec]
> 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ed]
> 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ee]
> 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ef]
> 15:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset USB 3.1 xHCI Controller [1022:43bb] (rev 02)
> 15:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset SATA Controller [1022:43b7] (rev 02)
> 15:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
> 16:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
> 16:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
> 16:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
> 18:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
> 38:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] [1002:15dd] (rev c8)
> 38:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:15de]
> 38:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:15df]
> 38:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e0]
> 38:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e1]
> 38:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e3]
> 39:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 61)
> ```
>
>
> [1]: git://people.freedesktop.org/~gabbayo/linux.git
>
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Raven: KFD/HSA depends on AMD_IOMMU_V2 (was: [PATCH 6/6] drm/amdkfd: Enable Raven for KFD)

[Paul Menzel]

[-- Attachment #1.1: Type: text/plain, Size: 1198 bytes --]

Dear Oded,


On 07/30/18 15:53, Oded Gabbay wrote:
> Do you have IOMMUv2 enabled on this chipset?

No, I did not. With `AMD_IOMMU_V2` selected, the device is found.

```
$ grep AMD_IOMMU /boot/config-4.18.0-rc3-01368-gb5aa3f4aef72 
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_V2=m
$ sudo dmesg | grep -i kfd
[sudo] password for paul: 
[   20.767499] calling  kfd_module_init+0x0/0x1000 [amdkfd] @ 365
[   20.770567] kfd kfd: Initialized module
[   20.771195] initcall kfd_module_init+0x0/0x1000 [amdkfd] returned 0 after 3598 usecs
[   20.893432] Modules linked in: amdkfd amd_iommu_v2 snd_hda_codec_realtek crct10dif_pclmul crc32_pclmul snd_hda_codec_hdmi snd_hda_codec_generic ghash_clmulni_intel snd_hda_intel amdgpu(+) i2c_piix4 r8169 k10temp snd_hda_codec snd_hda_core mii snd_hwdep snd_pcm snd_timer snd chash soundcore gpu_sched wmi video crc32c_intel xhci_pci xhci_hcd
[   20.948465] kfd kfd: Allocated 3969056 bytes on gart
[   20.948663] kfd kfd: added device 1002:15dd
```

> If not, amdkfd won't include raven support.

Hmm, then the warning is very confusing. Also, shouldn’t the build
system ensure that the dependencies are selected?


Kind regards,

Paul


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Raven: KFD/HSA depends on AMD_IOMMU_V2 (was: [PATCH 6/6] drm/amdkfd: Enable Raven for KFD)

[Alex Deucher]

On Mon, Jul 30, 2018 at 11:25 AM, Paul Menzel
<pmenzel+amd-gfx@molgen.mpg.de> wrote:
> Dear Oded,
>
>
> On 07/30/18 15:53, Oded Gabbay wrote:
>> Do you have IOMMUv2 enabled on this chipset?
>
> No, I did not. With `AMD_IOMMU_V2` selected, the device is found.
>
> ```
> $ grep AMD_IOMMU /boot/config-4.18.0-rc3-01368-gb5aa3f4aef72
> CONFIG_AMD_IOMMU=y
> CONFIG_AMD_IOMMU_V2=m
> $ sudo dmesg | grep -i kfd
> [sudo] password for paul:
> [   20.767499] calling  kfd_module_init+0x0/0x1000 [amdkfd] @ 365
> [   20.770567] kfd kfd: Initialized module
> [   20.771195] initcall kfd_module_init+0x0/0x1000 [amdkfd] returned 0 after 3598 usecs
> [   20.893432] Modules linked in: amdkfd amd_iommu_v2 snd_hda_codec_realtek crct10dif_pclmul crc32_pclmul snd_hda_codec_hdmi snd_hda_codec_generic ghash_clmulni_intel snd_hda_intel amdgpu(+) i2c_piix4 r8169 k10temp snd_hda_codec snd_hda_core mii snd_hwdep snd_pcm snd_timer snd chash soundcore gpu_sched wmi video crc32c_intel xhci_pci xhci_hcd
> [   20.948465] kfd kfd: Allocated 3969056 bytes on gart
> [   20.948663] kfd kfd: added device 1002:15dd
> ```
>
>> If not, amdkfd won't include raven support.
>
> Hmm, then the warning is very confusing. Also, shouldn’t the build
> system ensure that the dependencies are selected?

It's only required for APUs.  If you use a dGPU in a intel system for
example, you may not want the AMD IOMMU driver.

Alex
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Raven: KFD/HSA depends on AMD_IOMMU_V2

[Paul Menzel]

[-- Attachment #1.1: Type: text/plain, Size: 1952 bytes --]

Dear Alex,


On 07/30/18 17:32, Alex Deucher wrote:
> On Mon, Jul 30, 2018 at 11:25 AM, Paul Menzel wrote:

>> On 07/30/18 15:53, Oded Gabbay wrote:
>>> Do you have IOMMUv2 enabled on this chipset?
>>
>> No, I did not. With `AMD_IOMMU_V2` selected, the device is found.
>>
>> ```
>> $ grep AMD_IOMMU /boot/config-4.18.0-rc3-01368-gb5aa3f4aef72
>> CONFIG_AMD_IOMMU=y
>> CONFIG_AMD_IOMMU_V2=m
>> $ sudo dmesg | grep -i kfd
>> [sudo] password for paul:
>> [   20.767499] calling  kfd_module_init+0x0/0x1000 [amdkfd] @ 365
>> [   20.770567] kfd kfd: Initialized module
>> [   20.771195] initcall kfd_module_init+0x0/0x1000 [amdkfd] returned 0 after 3598 usecs
>> [   20.893432] Modules linked in: amdkfd amd_iommu_v2 snd_hda_codec_realtek crct10dif_pclmul crc32_pclmul snd_hda_codec_hdmi snd_hda_codec_generic ghash_clmulni_intel snd_hda_intel amdgpu(+) i2c_piix4 r8169 k10temp snd_hda_codec snd_hda_core mii snd_hwdep snd_pcm snd_timer snd chash soundcore gpu_sched wmi video crc32c_intel xhci_pci xhci_hcd
>> [   20.948465] kfd kfd: Allocated 3969056 bytes on gart
>> [   20.948663] kfd kfd: added device 1002:15dd
>> ```
>>
>>> If not, amdkfd won't include raven support.
>>
>> Hmm, then the warning is very confusing. Also, shouldn’t the build
>> system ensure that the dependencies are selected?
> 
> It's only required for APUs.  If you use a dGPU in a intel system for
> example, you may not want the AMD IOMMU driver.

Yes, I understand that, but couldn’t that be documented in the Kconfig
help text?

It looks like *imply* is used already, but that is not shown in
menuconfig, if I am not mistaken.

> config HSA_AMD
>         tristate "HSA kernel driver for AMD GPU devices"
>         depends on DRM_AMDGPU && X86_64
>         imply AMD_IOMMU_V2
>         select MMU_NOTIFIER
>         help
>           Enable this if you want to use HSA features on AMD GPU devices.


Kind regards,

Paul


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 6/6] drm/amdkfd: Enable Raven for KFD

[Felix Kuehling]

I'm using that branch myself successfully. For Raven (and other APU)
support, make sure you have AMD_IOMMU_V2 enabled in your kernel config.

Regards,
  Felix


On 2018-07-30 09:33 AM, Paul Menzel wrote:
> Dear Felix, dear Yong,
>
>
> On 07/13/18 22:17, Felix Kühling wrote:
>> From: Yong Zhao <Yong.Zhao@amd.com>
>>
>> Add DID and kfd_device_info for Raven.
>>
>> Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
>> Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
>> Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
>> Acked-by: Alex Deucher <alexander.deucher@amd.com>
>> ---
>>  drivers/gpu/drm/amd/amdkfd/kfd_device.c | 16 ++++++++++++++++
>>  1 file changed, 16 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
>> index 572235c..1b04871 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
>> @@ -70,6 +70,21 @@ static const struct kfd_device_info carrizo_device_info = {
>>  	.needs_pci_atomics = false,
>>  	.num_sdma_engines = 2,
>>  };
>> +
>> +static const struct kfd_device_info raven_device_info = {
>> +	.asic_family = CHIP_RAVEN,
>> +	.max_pasid_bits = 16,
>> +	.max_no_of_hqd  = 24,
>> +	.doorbell_size  = 8,
>> +	.ih_ring_entry_size = 8 * sizeof(uint32_t),
>> +	.event_interrupt_class = &event_interrupt_class_v9,
>> +	.num_of_watch_points = 4,
>> +	.mqd_size_aligned = MQD_SIZE_ALIGNED,
>> +	.supports_cwsr = true,
>> +	.needs_iommu_device = true,
>> +	.needs_pci_atomics = true,
>> +	.num_sdma_engines = 1,
>> +};
>>  #endif
>>  
>>  static const struct kfd_device_info hawaii_device_info = {
>> @@ -259,6 +274,7 @@ static const struct kfd_deviceid supported_devices[] = {
>>  	{ 0x9875, &carrizo_device_info },	/* Carrizo */
>>  	{ 0x9876, &carrizo_device_info },	/* Carrizo */
>>  	{ 0x9877, &carrizo_device_info },	/* Carrizo */
>> +	{ 0x15DD, &raven_device_info },		/* Raven */
>>  #endif
>>  	{ 0x67A0, &hawaii_device_info },	/* Hawaii */
>>  	{ 0x67A1, &hawaii_device_info },	/* Hawaii */
>
> I built the branch *amdkfd-next* from [1], but on the MIS B350M MORTAR (MS-7A37)
> with BIOS 1.G1 05/17/2018 it is not loaded.
>
> ```
> $ uname -a
> Linux tokeiihto 4.18.0-rc3-01368-gb5aa3f4aef72 #51 SMP Mon Jul 30 13:11:36 UTC 2018 x86_64 GNU/Linux
> $ grep HSA /boot/config-4.18.0-rc3-01368-gb5aa3f4aef72
> CONFIG_HSA_AMD=m
> $ sudo dmesg | grep -i kfd
> [   23.206628] calling  kfd_module_init+0x0/0x1000 [amdkfd] @ 365
> [   23.209082] kfd kfd: Initialized module
> [   23.209667] initcall kfd_module_init+0x0/0x1000 [amdkfd] returned 0 after 2960 usecs
> [   23.212852] kfd kfd: DID 15dd is missing in supported_devices
> [   23.212856] kfd kfd: kgd2kfd_probe failed
> $ /sbin/modinfo amdkfd
> filename:       /lib/modules/4.18.0-rc3-01368-gb5aa3f4aef72/kernel/drivers/gpu/drm/amd/amdkfd/amdkfd.ko
> version:        0.7.2
> license:        GPL and additional rights
> description:    Standalone HSA driver for AMD's GPUs
> author:         AMD Inc. and others
> srcversion:     9AAD0B6E4510ACF8D20A888
> depends:
> retpoline:      Y 
> intree:         Y 
> name:           amdkfd
> vermagic:       4.18.0-rc3-01368-gb5aa3f4aef72 SMP mod_unload 
> sig_id:         PKCS#7
> […]
> ```
>
> Any hints, what I am doing wrong?
>
>
> Kind regards,
>
> Paul
>
>
> PS: Output of lspci:
>
> ```
> $ lspci -nn
> 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d0]
> 00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d1]
> 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
> 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d3]
> 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
> 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15db]
> 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:15dc]
> 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
> 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
> 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e8]
> 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e9]
> 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ea]
> 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15eb]
> 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ec]
> 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ed]
> 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ee]
> 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ef]
> 15:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset USB 3.1 xHCI Controller [1022:43bb] (rev 02)
> 15:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset SATA Controller [1022:43b7] (rev 02)
> 15:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
> 16:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
> 16:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
> 16:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
> 18:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
> 38:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] [1002:15dd] (rev c8)
> 38:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:15de]
> 38:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:15df]
> 38:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e0]
> 38:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e1]
> 38:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:15e3]
> 39:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 61)
> ```
>
>
> [1]: git://people.freedesktop.org/~gabbayo/linux.git
>

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx