[PATCH 00/23] Add support AMD GPU virtualization soultion

[PATCH 00/23] Add support AMD GPU virtualization soultion

[Xiangliang Yu]

This patch series will enable AMD GPU virtualization and wires it
into amdgpu modules.

The code design is base on below ideas:
1. Make virtualization code independent;
2. Self-maintain related GPU virtualization codes;
3. Easy to maintain virtualization codes;
4. Reduce the influence on other amdgpu components;
5. Easy to support different chips;

According to above ideas, make the following changes: 
1. Create a new directory for virtualization and put all of
   related virtualization code into it;
2. Create new IP block to support different virtual features. 
3. Create new file for each different hardware;

In the end, the patch series is support serval virtualization
features:
1. Support CSA feature;
2. Support Mailbox communication with GPU hypervisor;
3. Support KIQ feature;

Xiangliang Yu (23):
  drm/amdgpu: add support kernel interface queue(KIQ)
  drm/amdgpu: add kiq into compiling
  drm/amdgpu: export KIQ interfaces
  drm/amdgpu: add new structure for KIQ memory allcation
  drm/amdgpu/gfx8: add support KIQ for FIJI/TONGA chips
  drm/amdgpu/gfx8: correct KIQ hdp flush
  drm/amdgpu: create new directory for GPU virtualization
  drm/amdgpu: add new flag for virtual function
  drm/amdgpu: enable virtualization feature for FIJI/TONGA
  drm/amdgpu: export gem va update interface
  drm/amdgpu: implement context save area(CSA) feature
  drm/amdgpu: Insert meta data during submitting IB
  drm/amdgpu/mxgpu: add support for mailbox communication
  drm/amdgpu: export two mailbox interface to amdgpu
  drm/amdgpu/mxgpu: implement register access function with KIQ
  drm/amdgpu: add flag to indicate VF runtime state
  drm/amdgpu: export vi common ip block
  drm/amdgpu: add new maroc to identify virtualization ip block
  drm/amdgpu/mxgpu: add implementation of GPU virtualization of VI
  drm/amdgpu/mxgpu: enable VI virtualization
  drm/amdgpu: change golden register program sequence of virtualization
  drm/amdgpu: do not reset gpu for virtualization
  drm/amdgpu/mxgpu: add Makefie and Kconfig for GPU virtualization

 drivers/gpu/drm/amd/amdgpu/Makefile        |   9 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  35 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  14 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |   4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c    |   6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c     |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c    | 436 ++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    |   4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   |  34 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |   8 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h     |   4 +
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c      | 240 ++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/vi.c            |   8 +-
 drivers/gpu/drm/amd/amdgpu/vi.h            |   2 +-
 drivers/gpu/drm/amd/amdgpu/vid.h           |   2 +
 drivers/gpu/drm/amd/include/amd_shared.h   |   2 +
 drivers/gpu/drm/amd/mxgpu/Kconfig          |  13 +
 drivers/gpu/drm/amd/mxgpu/Makefile         |  11 +
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c      | 105 +++++++
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h      |  93 ++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c      | 297 ++++++++++++++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c      | 110 +++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c       | 159 ++++++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c       | 472 +++++++++++++++++++++++++++++
 24 files changed, 2058 insertions(+), 12 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/Kconfig
 create mode 100644 drivers/gpu/drm/amd/mxgpu/Makefile
 create mode 100644 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c

-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 01/23] drm/amdgpu: add support kernel interface queue(KIQ)

[Xiangliang Yu]

KIQ is queue-memory based initialization method: setup KIQ queue
firstly, then send command to KIQ to setup other queues, without
accessing registers.

For virtualization, need KIQ to access virtual function registers
when running on guest mode.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
Signed-off-by: Monk Liu <Monk.Liu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c | 436 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/vid.h        |   2 +
 2 files changed, 438 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
new file mode 100644
index 0000000..e9cdc28
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
@@ -0,0 +1,436 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Xiangliang.Yu@amd.com
+ *	    Monk.Liu@amd.com
+ */
+#include "amdgpu.h"
+#include "vid.h"
+#include "vi_structs.h"
+#include "gca/gfx_8_0_d.h"
+#include "gca/gfx_8_0_sh_mask.h"
+
+enum amdgpu_kiq_irq {
+	AMDGPU_CP_KIQ_IRQ_DRIVER0 = 0,
+	AMDGPU_CP_KIQ_IRQ_LAST
+};
+
+int amdgpu_kiq_init_ring(struct amdgpu_device *adev, struct amdgpu_ring *ring,
+			 struct amdgpu_irq_src *irq)
+{
+	int r = 0;
+
+	ring->adev = NULL;
+	ring->ring_obj = NULL;
+	ring->use_doorbell = true;
+	ring->doorbell_index = AMDGPU_DOORBELL_KIQ;
+	if (adev->gfx.mec2_fw) {
+		ring->me = 2;
+		ring->pipe = 0;
+	} else {
+		ring->me = 1;
+		ring->pipe = 1;
+	}
+
+	irq->data = ring;
+	ring->queue = 0;
+	sprintf(ring->name, "kiq %d.%d.%d", ring->me, ring->pipe, ring->queue);
+	r = amdgpu_ring_init(adev, ring, 1024,
+			     irq, AMDGPU_CP_KIQ_IRQ_DRIVER0);
+	if (r)
+		dev_warn(adev->dev, "(%d) failed to init kiq ring\n", r);
+
+	return r;
+}
+
+void amdgpu_kiq_free_ring(struct amdgpu_ring *ring, struct amdgpu_irq_src *irq)
+{
+	amdgpu_ring_fini(ring);
+	irq->data = NULL;
+}
+
+void amdgpu_kiq_enable(struct amdgpu_ring *ring)
+{
+	uint32_t tmp;
+	struct amdgpu_device *adev = ring->adev;
+
+	/* tell RLC which is KIQ queue */
+	tmp = RREG32(mmRLC_CP_SCHEDULERS);
+	tmp &= 0xffffff00;
+	tmp |= (ring->me << 5) | (ring->pipe << 3) | (ring->queue);
+	WREG32(mmRLC_CP_SCHEDULERS, tmp);
+	tmp |= 0x80;
+	WREG32(mmRLC_CP_SCHEDULERS, tmp);
+}
+
+void amdgpu_kiq_start(struct amdgpu_ring *ring)
+{
+	amdgpu_ring_alloc(ring, 8);
+	/* set resources */
+	amdgpu_ring_write(ring, PACKET3(PACKET3_SET_RESOURCES, 6));
+	amdgpu_ring_write(ring, 0);	/* vmid_mask:0 queue_type:0 (KIQ) */
+	amdgpu_ring_write(ring, 0x000000FF);	/* queue mask lo */
+	amdgpu_ring_write(ring, 0);	/* queue mask hi */
+	amdgpu_ring_write(ring, 0);	/* gws mask lo */
+	amdgpu_ring_write(ring, 0);	/* gws mask hi */
+	amdgpu_ring_write(ring, 0);	/* oac mask */
+	amdgpu_ring_write(ring, 0);	/* gds heap base:0, gds heap size:0 */
+	amdgpu_ring_commit(ring);
+	udelay(50);
+}
+
+void amdgpu_kiq_map_queue(struct amdgpu_ring *kiq_ring,
+			  struct amdgpu_ring *ring)
+{
+	struct amdgpu_device *adev = kiq_ring->adev;
+	uint64_t mqd_addr, wptr_addr;
+
+	mqd_addr = amdgpu_bo_gpu_offset(ring->mqd_obj);
+	wptr_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	amdgpu_ring_alloc(kiq_ring, 8);
+
+	amdgpu_ring_write(kiq_ring, PACKET3(PACKET3_MAP_QUEUES, 5));
+	/* Q_sel:0, vmid:0, vidmem: 1, engine:0, num_Q:1*/
+	amdgpu_ring_write(kiq_ring, 0x21010000);
+	amdgpu_ring_write(kiq_ring, (ring->doorbell_index << 2) |
+			(ring->queue << 26) |
+			(ring->pipe << 29) |
+			((ring->me == 1 ? 0 : 1) << 31)); /* doorbell */
+	amdgpu_ring_write(kiq_ring, lower_32_bits(mqd_addr));
+	amdgpu_ring_write(kiq_ring, upper_32_bits(mqd_addr));
+	amdgpu_ring_write(kiq_ring, lower_32_bits(wptr_addr));
+	amdgpu_ring_write(kiq_ring, upper_32_bits(wptr_addr));
+	amdgpu_ring_commit(kiq_ring);
+	udelay(50);
+}
+
+#define MEC_HPD_SIZE	2048
+
+int amdgpu_mqd_init(struct amdgpu_device *adev,
+		    struct vi_mqd *mqd,
+		    uint64_t mqd_gpu_addr,
+		    uint64_t eop_gpu_addr,
+		    struct amdgpu_ring *ring)
+{
+	uint64_t hqd_gpu_addr, wb_gpu_addr, eop_base_addr;
+	uint32_t tmp;
+
+	mqd->header = 0xC0310800;
+	mqd->compute_pipelinestat_enable = 0x00000001;
+	mqd->compute_static_thread_mgmt_se0 = 0xffffffff;
+	mqd->compute_static_thread_mgmt_se1 = 0xffffffff;
+	mqd->compute_static_thread_mgmt_se2 = 0xffffffff;
+	mqd->compute_static_thread_mgmt_se3 = 0xffffffff;
+	mqd->compute_misc_reserved = 0x00000003;
+
+	eop_base_addr = eop_gpu_addr >> 8;
+	mqd->cp_hqd_eop_base_addr_lo = eop_base_addr;
+	mqd->cp_hqd_eop_base_addr_hi = upper_32_bits(eop_base_addr);
+
+	/* set the EOP size, register value is 2^(EOP_SIZE+1) dwords */
+	tmp = RREG32(mmCP_HQD_EOP_CONTROL);
+	tmp = REG_SET_FIELD(tmp, CP_HQD_EOP_CONTROL, EOP_SIZE,
+			(order_base_2(MEC_HPD_SIZE / 4) - 1));
+
+	mqd->cp_hqd_eop_control = tmp;
+
+	/* enable doorbell? */
+	tmp = RREG32(mmCP_HQD_PQ_DOORBELL_CONTROL);
+
+	if (ring->use_doorbell)
+		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
+					 DOORBELL_EN, 1);
+	else
+		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
+					 DOORBELL_EN, 0);
+
+	mqd->cp_hqd_pq_doorbell_control = tmp;
+
+	/* disable the queue if it's active */
+	mqd->cp_hqd_dequeue_request = 0;
+	mqd->cp_hqd_pq_rptr = 0;
+	mqd->cp_hqd_pq_wptr = 0;
+
+	/* set the pointer to the MQD */
+	mqd->cp_mqd_base_addr_lo = mqd_gpu_addr & 0xfffffffc;
+	mqd->cp_mqd_base_addr_hi = upper_32_bits(mqd_gpu_addr);
+
+	/* set MQD vmid to 0 */
+	tmp = RREG32(mmCP_MQD_CONTROL);
+	tmp = REG_SET_FIELD(tmp, CP_MQD_CONTROL, VMID, 0);
+	mqd->cp_mqd_control = tmp;
+
+	/* set the pointer to the HQD, this is similar CP_RB0_BASE/_HI */
+	hqd_gpu_addr = ring->gpu_addr >> 8;
+	mqd->cp_hqd_pq_base_lo = hqd_gpu_addr;
+	mqd->cp_hqd_pq_base_hi = upper_32_bits(hqd_gpu_addr);
+
+	/* set up the HQD, this is similar to CP_RB0_CNTL */
+	tmp = RREG32(mmCP_HQD_PQ_CONTROL);
+	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, QUEUE_SIZE,
+			    (order_base_2(ring->ring_size / 4) - 1));
+	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, RPTR_BLOCK_SIZE,
+			((order_base_2(AMDGPU_GPU_PAGE_SIZE / 4) - 1) << 8));
+#ifdef __BIG_ENDIAN
+	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, ENDIAN_SWAP, 1);
+#endif
+	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, UNORD_DISPATCH, 0);
+	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, ROQ_PQ_IB_FLIP, 0);
+	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);
+	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);
+	mqd->cp_hqd_pq_control = tmp;
+
+	/* set the wb address whether it's enabled or not */
+	wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
+	mqd->cp_hqd_pq_rptr_report_addr_hi =
+		upper_32_bits(wb_gpu_addr) & 0xffff;
+
+	/* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
+	wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
+	mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
+
+	tmp = 0;
+	/* enable the doorbell if requested */
+	if (ring->use_doorbell) {
+		tmp = RREG32(mmCP_HQD_PQ_DOORBELL_CONTROL);
+		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
+				DOORBELL_OFFSET, ring->doorbell_index);
+
+		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
+					 DOORBELL_EN, 1);
+		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
+					 DOORBELL_SOURCE, 0);
+		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
+					 DOORBELL_HIT, 0);
+	}
+
+	mqd->cp_hqd_pq_doorbell_control = tmp;
+
+	/* reset read and write pointers, similar to CP_RB0_WPTR/_RPTR */
+	ring->wptr = 0;
+	mqd->cp_hqd_pq_wptr = ring->wptr;
+	mqd->cp_hqd_pq_rptr = RREG32(mmCP_HQD_PQ_RPTR);
+
+	/* set the vmid for the queue */
+	mqd->cp_hqd_vmid = 0;
+
+	tmp = RREG32(mmCP_HQD_PERSISTENT_STATE);
+	tmp = REG_SET_FIELD(tmp, CP_HQD_PERSISTENT_STATE, PRELOAD_SIZE, 0x53);
+	mqd->cp_hqd_persistent_state = tmp;
+
+	/* activate the queue */
+	mqd->cp_hqd_active = 1;
+
+	return 0;
+}
+
+int amdgpu_kiq_init(struct amdgpu_device *adev,
+		    struct vi_mqd *mqd,
+		    struct amdgpu_ring *ring)
+{
+	uint32_t tmp;
+	int j;
+
+	/* disable wptr polling */
+	tmp = RREG32(mmCP_PQ_WPTR_POLL_CNTL);
+	tmp = REG_SET_FIELD(tmp, CP_PQ_WPTR_POLL_CNTL, EN, 0);
+	WREG32(mmCP_PQ_WPTR_POLL_CNTL, tmp);
+
+	WREG32(mmCP_HQD_EOP_BASE_ADDR, mqd->cp_hqd_eop_base_addr_lo);
+	WREG32(mmCP_HQD_EOP_BASE_ADDR_HI, mqd->cp_hqd_eop_base_addr_hi);
+
+	/* set the EOP size, register value is 2^(EOP_SIZE+1) dwords */
+	WREG32(mmCP_HQD_EOP_CONTROL, mqd->cp_hqd_eop_control);
+
+	/* enable doorbell? */
+	WREG32(mmCP_HQD_PQ_DOORBELL_CONTROL, mqd->cp_hqd_pq_doorbell_control);
+
+	/* disable the queue if it's active */
+	if (RREG32(mmCP_HQD_ACTIVE) & 1) {
+		WREG32(mmCP_HQD_DEQUEUE_REQUEST, 1);
+		for (j = 0; j < adev->usec_timeout; j++) {
+			if (!(RREG32(mmCP_HQD_ACTIVE) & 1))
+				break;
+			udelay(1);
+		}
+		WREG32(mmCP_HQD_DEQUEUE_REQUEST, mqd->cp_hqd_dequeue_request);
+		WREG32(mmCP_HQD_PQ_RPTR, mqd->cp_hqd_pq_rptr);
+		WREG32(mmCP_HQD_PQ_WPTR, mqd->cp_hqd_pq_wptr);
+	}
+
+	/* set the pointer to the MQD */
+	WREG32(mmCP_MQD_BASE_ADDR, mqd->cp_mqd_base_addr_lo);
+	WREG32(mmCP_MQD_BASE_ADDR_HI, mqd->cp_mqd_base_addr_hi);
+
+	/* set MQD vmid to 0 */
+	WREG32(mmCP_MQD_CONTROL, mqd->cp_mqd_control);
+
+	/* set the pointer to the HQD, this is similar CP_RB0_BASE/_HI */
+	WREG32(mmCP_HQD_PQ_BASE, mqd->cp_hqd_pq_base_lo);
+	WREG32(mmCP_HQD_PQ_BASE_HI, mqd->cp_hqd_pq_base_hi);
+
+	/* set up the HQD, this is similar to CP_RB0_CNTL */
+	WREG32(mmCP_HQD_PQ_CONTROL, mqd->cp_hqd_pq_control);
+
+	/* set the wb address whether it's enabled or not */
+	WREG32(mmCP_HQD_PQ_RPTR_REPORT_ADDR,
+				mqd->cp_hqd_pq_rptr_report_addr_lo);
+	WREG32(mmCP_HQD_PQ_RPTR_REPORT_ADDR_HI,
+				mqd->cp_hqd_pq_rptr_report_addr_hi);
+
+	/* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
+	WREG32(mmCP_HQD_PQ_WPTR_POLL_ADDR, mqd->cp_hqd_pq_wptr_poll_addr_lo);
+	WREG32(mmCP_HQD_PQ_WPTR_POLL_ADDR_HI, mqd->cp_hqd_pq_wptr_poll_addr_hi);
+
+	/* enable the doorbell if requested */
+	if (ring->use_doorbell) {
+		if ((adev->asic_type == CHIP_CARRIZO) ||
+				(adev->asic_type == CHIP_FIJI) ||
+				(adev->asic_type == CHIP_STONEY)) {
+			WREG32(mmCP_MEC_DOORBELL_RANGE_LOWER,
+						AMDGPU_DOORBELL_KIQ << 2);
+			WREG32(mmCP_MEC_DOORBELL_RANGE_UPPER,
+						AMDGPU_DOORBELL_MEC_RING7 << 2);
+		}
+	}
+	WREG32(mmCP_HQD_PQ_DOORBELL_CONTROL, mqd->cp_hqd_pq_doorbell_control);
+
+	/* reset read and write pointers, similar to CP_RB0_WPTR/_RPTR */
+	WREG32(mmCP_HQD_PQ_WPTR, mqd->cp_hqd_pq_wptr);
+
+	/* set the vmid for the queue */
+	WREG32(mmCP_HQD_VMID, mqd->cp_hqd_vmid);
+
+	WREG32(mmCP_HQD_PERSISTENT_STATE, mqd->cp_hqd_persistent_state);
+
+	/* activate the queue */
+	WREG32(mmCP_HQD_ACTIVE, mqd->cp_hqd_active);
+
+	if (ring->use_doorbell) {
+		tmp = RREG32(mmCP_PQ_STATUS);
+		tmp = REG_SET_FIELD(tmp, CP_PQ_STATUS, DOORBELL_ENABLE, 1);
+		WREG32(mmCP_PQ_STATUS, tmp);
+	}
+
+	return 0;
+}
+
+void amdgpu_kiq_ring_emit_fence(struct amdgpu_ring *ring, u64 addr,
+				u64 seq, unsigned int flags)
+{
+	/* we only allocate 32bit for each seq wb address */
+	if (flags & AMDGPU_FENCE_FLAG_64BIT)
+		BUG();
+
+	/* write fence seq to the "addr" */
+	amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
+	amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(0) |
+				 WRITE_DATA_DST_SEL(5) | WR_CONFIRM));
+	amdgpu_ring_write(ring, lower_32_bits(addr));
+	amdgpu_ring_write(ring, upper_32_bits(addr));
+	amdgpu_ring_write(ring, lower_32_bits(seq));
+
+	if (flags & AMDGPU_FENCE_FLAG_INT) {
+		/* set register to trigger INT */
+		amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
+		amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(0) |
+					 WRITE_DATA_DST_SEL(0) | WR_CONFIRM));
+		amdgpu_ring_write(ring, mmCPC_INT_STATUS);
+		amdgpu_ring_write(ring, 0);
+		amdgpu_ring_write(ring, 0x20000000); /* src_id is 178 */
+	}
+}
+
+static int amdgpu_kiq_set_interrupt_state(struct amdgpu_device *adev,
+					  struct amdgpu_irq_src *src,
+					  unsigned int type,
+					  enum amdgpu_interrupt_state state)
+{
+	uint32_t tmp, target;
+	struct amdgpu_ring *ring = (struct amdgpu_ring *)src->data;
+
+	BUG_ON(!ring || (ring->funcs->type != AMDGPU_RING_TYPE_KIQ));
+
+	target = (ring->me == 1) ? mmCP_ME1_PIPE0_INT_CNTL : mmCP_ME2_PIPE0_INT_CNTL;
+	target += ring->pipe;
+
+	switch (type) {
+	case AMDGPU_CP_KIQ_IRQ_DRIVER0:
+		if (state == AMDGPU_IRQ_STATE_DISABLE) {
+			tmp = RREG32(mmCPC_INT_CNTL);
+			tmp = REG_SET_FIELD(tmp, CPC_INT_CNTL,
+						 GENERIC2_INT_ENABLE, 0);
+			WREG32(mmCPC_INT_CNTL, tmp);
+
+			tmp = RREG32(target);
+			tmp = REG_SET_FIELD(tmp, CP_ME2_PIPE0_INT_CNTL,
+						 GENERIC2_INT_ENABLE, 0);
+			WREG32(target, tmp);
+		} else {
+			tmp = RREG32(mmCPC_INT_CNTL);
+			tmp = REG_SET_FIELD(tmp, CPC_INT_CNTL,
+						 GENERIC2_INT_ENABLE, 1);
+			WREG32(mmCPC_INT_CNTL, tmp);
+
+			tmp = RREG32(target);
+			tmp = REG_SET_FIELD(tmp, CP_ME2_PIPE0_INT_CNTL,
+						 GENERIC2_INT_ENABLE, 1);
+			WREG32(target, tmp);
+		}
+		break;
+	default:
+		BUG(); /* kiq only support GENERIC2_INT now */
+		break;
+	}
+	return 0;
+}
+
+static int amdgpu_kiq_irq(struct amdgpu_device *adev,
+			  struct amdgpu_irq_src *source,
+			  struct amdgpu_iv_entry *entry)
+{
+	u8 me_id, pipe_id, queue_id;
+	struct amdgpu_ring *ring = (struct amdgpu_ring *)source->data;
+
+	BUG_ON(!ring || (ring->funcs->type != AMDGPU_RING_TYPE_KIQ));
+
+	me_id = (entry->ring_id & 0x0c) >> 2;
+	pipe_id = (entry->ring_id & 0x03) >> 0;
+	queue_id = (entry->ring_id & 0x70) >> 4;
+	DRM_DEBUG("IH: CPC GENERIC2_INT, me:%d, pipe:%d, queue:%d\n",
+		   me_id, pipe_id, queue_id);
+
+	amdgpu_fence_process(ring);
+	return 0;
+}
+
+static const struct amdgpu_irq_src_funcs amdgpu_kiq_irq_funcs = {
+	.set = amdgpu_kiq_set_interrupt_state,
+	.process = amdgpu_kiq_irq,
+};
+
+void amdgpu_kiq_set_irq_funcs(struct amdgpu_irq_src *irq)
+{
+	irq->num_types = AMDGPU_CP_KIQ_IRQ_LAST;
+	irq->funcs = &amdgpu_kiq_irq_funcs;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/vid.h b/drivers/gpu/drm/amd/amdgpu/vid.h
index 11746f2..7a3863a 100644
--- a/drivers/gpu/drm/amd/amdgpu/vid.h
+++ b/drivers/gpu/drm/amd/amdgpu/vid.h
@@ -360,6 +360,8 @@
 #define	PACKET3_WAIT_ON_CE_COUNTER			0x86
 #define	PACKET3_WAIT_ON_DE_COUNTER_DIFF			0x88
 #define	PACKET3_SWITCH_BUFFER				0x8B
+#define	PACKET3_SET_RESOURCES				0xA0
+#define	PACKET3_MAP_QUEUES				0xA2
 
 #define VCE_CMD_NO_OP		0x00000000
 #define VCE_CMD_END		0x00000001
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 01/23] drm/amdgpu: add support kernel interface queue(KIQ)

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> KIQ is queue-memory based initialization method: setup KIQ queue
> firstly, then send command to KIQ to setup other queues, without
> accessing registers.
>
> For virtualization, need KIQ to access virtual function registers
> when running on guest mode.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com>

kiq is just another CP ring.  Please add the asic specific parts to
gfx_v8_0.c.  If there is any common KIQ helper code that is non asic
dependent, that should be in amdgpu_kiq.c.

Alex

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c | 436 ++++++++++++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/vid.h        |   2 +
>  2 files changed, 438 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
> new file mode 100644
> index 0000000..e9cdc28
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
> @@ -0,0 +1,436 @@
> +/*
> + * Copyright 2016 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + * Authors: Xiangliang.Yu@amd.com
> + *         Monk.Liu@amd.com
> + */
> +#include "amdgpu.h"
> +#include "vid.h"
> +#include "vi_structs.h"
> +#include "gca/gfx_8_0_d.h"
> +#include "gca/gfx_8_0_sh_mask.h"
> +
> +enum amdgpu_kiq_irq {
> +       AMDGPU_CP_KIQ_IRQ_DRIVER0 = 0,
> +       AMDGPU_CP_KIQ_IRQ_LAST
> +};
> +
> +int amdgpu_kiq_init_ring(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> +                        struct amdgpu_irq_src *irq)
> +{
> +       int r = 0;
> +
> +       ring->adev = NULL;
> +       ring->ring_obj = NULL;
> +       ring->use_doorbell = true;
> +       ring->doorbell_index = AMDGPU_DOORBELL_KIQ;
> +       if (adev->gfx.mec2_fw) {
> +               ring->me = 2;
> +               ring->pipe = 0;
> +       } else {
> +               ring->me = 1;
> +               ring->pipe = 1;
> +       }
> +
> +       irq->data = ring;
> +       ring->queue = 0;
> +       sprintf(ring->name, "kiq %d.%d.%d", ring->me, ring->pipe, ring->queue);
> +       r = amdgpu_ring_init(adev, ring, 1024,
> +                            irq, AMDGPU_CP_KIQ_IRQ_DRIVER0);
> +       if (r)
> +               dev_warn(adev->dev, "(%d) failed to init kiq ring\n", r);
> +
> +       return r;
> +}
> +
> +void amdgpu_kiq_free_ring(struct amdgpu_ring *ring, struct amdgpu_irq_src *irq)
> +{
> +       amdgpu_ring_fini(ring);
> +       irq->data = NULL;
> +}
> +
> +void amdgpu_kiq_enable(struct amdgpu_ring *ring)
> +{
> +       uint32_t tmp;
> +       struct amdgpu_device *adev = ring->adev;
> +
> +       /* tell RLC which is KIQ queue */
> +       tmp = RREG32(mmRLC_CP_SCHEDULERS);
> +       tmp &= 0xffffff00;
> +       tmp |= (ring->me << 5) | (ring->pipe << 3) | (ring->queue);
> +       WREG32(mmRLC_CP_SCHEDULERS, tmp);
> +       tmp |= 0x80;
> +       WREG32(mmRLC_CP_SCHEDULERS, tmp);
> +}
> +
> +void amdgpu_kiq_start(struct amdgpu_ring *ring)
> +{
> +       amdgpu_ring_alloc(ring, 8);
> +       /* set resources */
> +       amdgpu_ring_write(ring, PACKET3(PACKET3_SET_RESOURCES, 6));
> +       amdgpu_ring_write(ring, 0);     /* vmid_mask:0 queue_type:0 (KIQ) */
> +       amdgpu_ring_write(ring, 0x000000FF);    /* queue mask lo */
> +       amdgpu_ring_write(ring, 0);     /* queue mask hi */
> +       amdgpu_ring_write(ring, 0);     /* gws mask lo */
> +       amdgpu_ring_write(ring, 0);     /* gws mask hi */
> +       amdgpu_ring_write(ring, 0);     /* oac mask */
> +       amdgpu_ring_write(ring, 0);     /* gds heap base:0, gds heap size:0 */
> +       amdgpu_ring_commit(ring);
> +       udelay(50);
> +}
> +
> +void amdgpu_kiq_map_queue(struct amdgpu_ring *kiq_ring,
> +                         struct amdgpu_ring *ring)
> +{
> +       struct amdgpu_device *adev = kiq_ring->adev;
> +       uint64_t mqd_addr, wptr_addr;
> +
> +       mqd_addr = amdgpu_bo_gpu_offset(ring->mqd_obj);
> +       wptr_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
> +       amdgpu_ring_alloc(kiq_ring, 8);
> +
> +       amdgpu_ring_write(kiq_ring, PACKET3(PACKET3_MAP_QUEUES, 5));
> +       /* Q_sel:0, vmid:0, vidmem: 1, engine:0, num_Q:1*/
> +       amdgpu_ring_write(kiq_ring, 0x21010000);
> +       amdgpu_ring_write(kiq_ring, (ring->doorbell_index << 2) |
> +                       (ring->queue << 26) |
> +                       (ring->pipe << 29) |
> +                       ((ring->me == 1 ? 0 : 1) << 31)); /* doorbell */
> +       amdgpu_ring_write(kiq_ring, lower_32_bits(mqd_addr));
> +       amdgpu_ring_write(kiq_ring, upper_32_bits(mqd_addr));
> +       amdgpu_ring_write(kiq_ring, lower_32_bits(wptr_addr));
> +       amdgpu_ring_write(kiq_ring, upper_32_bits(wptr_addr));
> +       amdgpu_ring_commit(kiq_ring);
> +       udelay(50);
> +}
> +
> +#define MEC_HPD_SIZE   2048
> +
> +int amdgpu_mqd_init(struct amdgpu_device *adev,
> +                   struct vi_mqd *mqd,
> +                   uint64_t mqd_gpu_addr,
> +                   uint64_t eop_gpu_addr,
> +                   struct amdgpu_ring *ring)
> +{
> +       uint64_t hqd_gpu_addr, wb_gpu_addr, eop_base_addr;
> +       uint32_t tmp;
> +
> +       mqd->header = 0xC0310800;
> +       mqd->compute_pipelinestat_enable = 0x00000001;
> +       mqd->compute_static_thread_mgmt_se0 = 0xffffffff;
> +       mqd->compute_static_thread_mgmt_se1 = 0xffffffff;
> +       mqd->compute_static_thread_mgmt_se2 = 0xffffffff;
> +       mqd->compute_static_thread_mgmt_se3 = 0xffffffff;
> +       mqd->compute_misc_reserved = 0x00000003;
> +
> +       eop_base_addr = eop_gpu_addr >> 8;
> +       mqd->cp_hqd_eop_base_addr_lo = eop_base_addr;
> +       mqd->cp_hqd_eop_base_addr_hi = upper_32_bits(eop_base_addr);
> +
> +       /* set the EOP size, register value is 2^(EOP_SIZE+1) dwords */
> +       tmp = RREG32(mmCP_HQD_EOP_CONTROL);
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_EOP_CONTROL, EOP_SIZE,
> +                       (order_base_2(MEC_HPD_SIZE / 4) - 1));
> +
> +       mqd->cp_hqd_eop_control = tmp;
> +
> +       /* enable doorbell? */
> +       tmp = RREG32(mmCP_HQD_PQ_DOORBELL_CONTROL);
> +
> +       if (ring->use_doorbell)
> +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> +                                        DOORBELL_EN, 1);
> +       else
> +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> +                                        DOORBELL_EN, 0);
> +
> +       mqd->cp_hqd_pq_doorbell_control = tmp;
> +
> +       /* disable the queue if it's active */
> +       mqd->cp_hqd_dequeue_request = 0;
> +       mqd->cp_hqd_pq_rptr = 0;
> +       mqd->cp_hqd_pq_wptr = 0;
> +
> +       /* set the pointer to the MQD */
> +       mqd->cp_mqd_base_addr_lo = mqd_gpu_addr & 0xfffffffc;
> +       mqd->cp_mqd_base_addr_hi = upper_32_bits(mqd_gpu_addr);
> +
> +       /* set MQD vmid to 0 */
> +       tmp = RREG32(mmCP_MQD_CONTROL);
> +       tmp = REG_SET_FIELD(tmp, CP_MQD_CONTROL, VMID, 0);
> +       mqd->cp_mqd_control = tmp;
> +
> +       /* set the pointer to the HQD, this is similar CP_RB0_BASE/_HI */
> +       hqd_gpu_addr = ring->gpu_addr >> 8;
> +       mqd->cp_hqd_pq_base_lo = hqd_gpu_addr;
> +       mqd->cp_hqd_pq_base_hi = upper_32_bits(hqd_gpu_addr);
> +
> +       /* set up the HQD, this is similar to CP_RB0_CNTL */
> +       tmp = RREG32(mmCP_HQD_PQ_CONTROL);
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, QUEUE_SIZE,
> +                           (order_base_2(ring->ring_size / 4) - 1));
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, RPTR_BLOCK_SIZE,
> +                       ((order_base_2(AMDGPU_GPU_PAGE_SIZE / 4) - 1) << 8));
> +#ifdef __BIG_ENDIAN
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, ENDIAN_SWAP, 1);
> +#endif
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, UNORD_DISPATCH, 0);
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, ROQ_PQ_IB_FLIP, 0);
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);
> +       mqd->cp_hqd_pq_control = tmp;
> +
> +       /* set the wb address whether it's enabled or not */
> +       wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
> +       mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
> +       mqd->cp_hqd_pq_rptr_report_addr_hi =
> +               upper_32_bits(wb_gpu_addr) & 0xffff;
> +
> +       /* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
> +       wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
> +       mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
> +       mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
> +
> +       tmp = 0;
> +       /* enable the doorbell if requested */
> +       if (ring->use_doorbell) {
> +               tmp = RREG32(mmCP_HQD_PQ_DOORBELL_CONTROL);
> +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> +                               DOORBELL_OFFSET, ring->doorbell_index);
> +
> +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> +                                        DOORBELL_EN, 1);
> +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> +                                        DOORBELL_SOURCE, 0);
> +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> +                                        DOORBELL_HIT, 0);
> +       }
> +
> +       mqd->cp_hqd_pq_doorbell_control = tmp;
> +
> +       /* reset read and write pointers, similar to CP_RB0_WPTR/_RPTR */
> +       ring->wptr = 0;
> +       mqd->cp_hqd_pq_wptr = ring->wptr;
> +       mqd->cp_hqd_pq_rptr = RREG32(mmCP_HQD_PQ_RPTR);
> +
> +       /* set the vmid for the queue */
> +       mqd->cp_hqd_vmid = 0;
> +
> +       tmp = RREG32(mmCP_HQD_PERSISTENT_STATE);
> +       tmp = REG_SET_FIELD(tmp, CP_HQD_PERSISTENT_STATE, PRELOAD_SIZE, 0x53);
> +       mqd->cp_hqd_persistent_state = tmp;
> +
> +       /* activate the queue */
> +       mqd->cp_hqd_active = 1;
> +
> +       return 0;
> +}
> +
> +int amdgpu_kiq_init(struct amdgpu_device *adev,
> +                   struct vi_mqd *mqd,
> +                   struct amdgpu_ring *ring)
> +{
> +       uint32_t tmp;
> +       int j;
> +
> +       /* disable wptr polling */
> +       tmp = RREG32(mmCP_PQ_WPTR_POLL_CNTL);
> +       tmp = REG_SET_FIELD(tmp, CP_PQ_WPTR_POLL_CNTL, EN, 0);
> +       WREG32(mmCP_PQ_WPTR_POLL_CNTL, tmp);
> +
> +       WREG32(mmCP_HQD_EOP_BASE_ADDR, mqd->cp_hqd_eop_base_addr_lo);
> +       WREG32(mmCP_HQD_EOP_BASE_ADDR_HI, mqd->cp_hqd_eop_base_addr_hi);
> +
> +       /* set the EOP size, register value is 2^(EOP_SIZE+1) dwords */
> +       WREG32(mmCP_HQD_EOP_CONTROL, mqd->cp_hqd_eop_control);
> +
> +       /* enable doorbell? */
> +       WREG32(mmCP_HQD_PQ_DOORBELL_CONTROL, mqd->cp_hqd_pq_doorbell_control);
> +
> +       /* disable the queue if it's active */
> +       if (RREG32(mmCP_HQD_ACTIVE) & 1) {
> +               WREG32(mmCP_HQD_DEQUEUE_REQUEST, 1);
> +               for (j = 0; j < adev->usec_timeout; j++) {
> +                       if (!(RREG32(mmCP_HQD_ACTIVE) & 1))
> +                               break;
> +                       udelay(1);
> +               }
> +               WREG32(mmCP_HQD_DEQUEUE_REQUEST, mqd->cp_hqd_dequeue_request);
> +               WREG32(mmCP_HQD_PQ_RPTR, mqd->cp_hqd_pq_rptr);
> +               WREG32(mmCP_HQD_PQ_WPTR, mqd->cp_hqd_pq_wptr);
> +       }
> +
> +       /* set the pointer to the MQD */
> +       WREG32(mmCP_MQD_BASE_ADDR, mqd->cp_mqd_base_addr_lo);
> +       WREG32(mmCP_MQD_BASE_ADDR_HI, mqd->cp_mqd_base_addr_hi);
> +
> +       /* set MQD vmid to 0 */
> +       WREG32(mmCP_MQD_CONTROL, mqd->cp_mqd_control);
> +
> +       /* set the pointer to the HQD, this is similar CP_RB0_BASE/_HI */
> +       WREG32(mmCP_HQD_PQ_BASE, mqd->cp_hqd_pq_base_lo);
> +       WREG32(mmCP_HQD_PQ_BASE_HI, mqd->cp_hqd_pq_base_hi);
> +
> +       /* set up the HQD, this is similar to CP_RB0_CNTL */
> +       WREG32(mmCP_HQD_PQ_CONTROL, mqd->cp_hqd_pq_control);
> +
> +       /* set the wb address whether it's enabled or not */
> +       WREG32(mmCP_HQD_PQ_RPTR_REPORT_ADDR,
> +                               mqd->cp_hqd_pq_rptr_report_addr_lo);
> +       WREG32(mmCP_HQD_PQ_RPTR_REPORT_ADDR_HI,
> +                               mqd->cp_hqd_pq_rptr_report_addr_hi);
> +
> +       /* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
> +       WREG32(mmCP_HQD_PQ_WPTR_POLL_ADDR, mqd->cp_hqd_pq_wptr_poll_addr_lo);
> +       WREG32(mmCP_HQD_PQ_WPTR_POLL_ADDR_HI, mqd->cp_hqd_pq_wptr_poll_addr_hi);
> +
> +       /* enable the doorbell if requested */
> +       if (ring->use_doorbell) {
> +               if ((adev->asic_type == CHIP_CARRIZO) ||
> +                               (adev->asic_type == CHIP_FIJI) ||
> +                               (adev->asic_type == CHIP_STONEY)) {
> +                       WREG32(mmCP_MEC_DOORBELL_RANGE_LOWER,
> +                                               AMDGPU_DOORBELL_KIQ << 2);
> +                       WREG32(mmCP_MEC_DOORBELL_RANGE_UPPER,
> +                                               AMDGPU_DOORBELL_MEC_RING7 << 2);
> +               }
> +       }
> +       WREG32(mmCP_HQD_PQ_DOORBELL_CONTROL, mqd->cp_hqd_pq_doorbell_control);
> +
> +       /* reset read and write pointers, similar to CP_RB0_WPTR/_RPTR */
> +       WREG32(mmCP_HQD_PQ_WPTR, mqd->cp_hqd_pq_wptr);
> +
> +       /* set the vmid for the queue */
> +       WREG32(mmCP_HQD_VMID, mqd->cp_hqd_vmid);
> +
> +       WREG32(mmCP_HQD_PERSISTENT_STATE, mqd->cp_hqd_persistent_state);
> +
> +       /* activate the queue */
> +       WREG32(mmCP_HQD_ACTIVE, mqd->cp_hqd_active);
> +
> +       if (ring->use_doorbell) {
> +               tmp = RREG32(mmCP_PQ_STATUS);
> +               tmp = REG_SET_FIELD(tmp, CP_PQ_STATUS, DOORBELL_ENABLE, 1);
> +               WREG32(mmCP_PQ_STATUS, tmp);
> +       }
> +
> +       return 0;
> +}
> +
> +void amdgpu_kiq_ring_emit_fence(struct amdgpu_ring *ring, u64 addr,
> +                               u64 seq, unsigned int flags)
> +{
> +       /* we only allocate 32bit for each seq wb address */
> +       if (flags & AMDGPU_FENCE_FLAG_64BIT)
> +               BUG();
> +
> +       /* write fence seq to the "addr" */
> +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
> +       amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(0) |
> +                                WRITE_DATA_DST_SEL(5) | WR_CONFIRM));
> +       amdgpu_ring_write(ring, lower_32_bits(addr));
> +       amdgpu_ring_write(ring, upper_32_bits(addr));
> +       amdgpu_ring_write(ring, lower_32_bits(seq));
> +
> +       if (flags & AMDGPU_FENCE_FLAG_INT) {
> +               /* set register to trigger INT */
> +               amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
> +               amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(0) |
> +                                        WRITE_DATA_DST_SEL(0) | WR_CONFIRM));
> +               amdgpu_ring_write(ring, mmCPC_INT_STATUS);
> +               amdgpu_ring_write(ring, 0);
> +               amdgpu_ring_write(ring, 0x20000000); /* src_id is 178 */
> +       }
> +}
> +
> +static int amdgpu_kiq_set_interrupt_state(struct amdgpu_device *adev,
> +                                         struct amdgpu_irq_src *src,
> +                                         unsigned int type,
> +                                         enum amdgpu_interrupt_state state)
> +{
> +       uint32_t tmp, target;
> +       struct amdgpu_ring *ring = (struct amdgpu_ring *)src->data;
> +
> +       BUG_ON(!ring || (ring->funcs->type != AMDGPU_RING_TYPE_KIQ));
> +
> +       target = (ring->me == 1) ? mmCP_ME1_PIPE0_INT_CNTL : mmCP_ME2_PIPE0_INT_CNTL;
> +       target += ring->pipe;
> +
> +       switch (type) {
> +       case AMDGPU_CP_KIQ_IRQ_DRIVER0:
> +               if (state == AMDGPU_IRQ_STATE_DISABLE) {
> +                       tmp = RREG32(mmCPC_INT_CNTL);
> +                       tmp = REG_SET_FIELD(tmp, CPC_INT_CNTL,
> +                                                GENERIC2_INT_ENABLE, 0);
> +                       WREG32(mmCPC_INT_CNTL, tmp);
> +
> +                       tmp = RREG32(target);
> +                       tmp = REG_SET_FIELD(tmp, CP_ME2_PIPE0_INT_CNTL,
> +                                                GENERIC2_INT_ENABLE, 0);
> +                       WREG32(target, tmp);
> +               } else {
> +                       tmp = RREG32(mmCPC_INT_CNTL);
> +                       tmp = REG_SET_FIELD(tmp, CPC_INT_CNTL,
> +                                                GENERIC2_INT_ENABLE, 1);
> +                       WREG32(mmCPC_INT_CNTL, tmp);
> +
> +                       tmp = RREG32(target);
> +                       tmp = REG_SET_FIELD(tmp, CP_ME2_PIPE0_INT_CNTL,
> +                                                GENERIC2_INT_ENABLE, 1);
> +                       WREG32(target, tmp);
> +               }
> +               break;
> +       default:
> +               BUG(); /* kiq only support GENERIC2_INT now */
> +               break;
> +       }
> +       return 0;
> +}
> +
> +static int amdgpu_kiq_irq(struct amdgpu_device *adev,
> +                         struct amdgpu_irq_src *source,
> +                         struct amdgpu_iv_entry *entry)
> +{
> +       u8 me_id, pipe_id, queue_id;
> +       struct amdgpu_ring *ring = (struct amdgpu_ring *)source->data;
> +
> +       BUG_ON(!ring || (ring->funcs->type != AMDGPU_RING_TYPE_KIQ));
> +
> +       me_id = (entry->ring_id & 0x0c) >> 2;
> +       pipe_id = (entry->ring_id & 0x03) >> 0;
> +       queue_id = (entry->ring_id & 0x70) >> 4;
> +       DRM_DEBUG("IH: CPC GENERIC2_INT, me:%d, pipe:%d, queue:%d\n",
> +                  me_id, pipe_id, queue_id);
> +
> +       amdgpu_fence_process(ring);
> +       return 0;
> +}
> +
> +static const struct amdgpu_irq_src_funcs amdgpu_kiq_irq_funcs = {
> +       .set = amdgpu_kiq_set_interrupt_state,
> +       .process = amdgpu_kiq_irq,
> +};
> +
> +void amdgpu_kiq_set_irq_funcs(struct amdgpu_irq_src *irq)
> +{
> +       irq->num_types = AMDGPU_CP_KIQ_IRQ_LAST;
> +       irq->funcs = &amdgpu_kiq_irq_funcs;
> +}
> diff --git a/drivers/gpu/drm/amd/amdgpu/vid.h b/drivers/gpu/drm/amd/amdgpu/vid.h
> index 11746f2..7a3863a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vid.h
> +++ b/drivers/gpu/drm/amd/amdgpu/vid.h
> @@ -360,6 +360,8 @@
>  #define        PACKET3_WAIT_ON_CE_COUNTER                      0x86
>  #define        PACKET3_WAIT_ON_DE_COUNTER_DIFF                 0x88
>  #define        PACKET3_SWITCH_BUFFER                           0x8B
> +#define        PACKET3_SET_RESOURCES                           0xA0
> +#define        PACKET3_MAP_QUEUES                              0xA2
>
>  #define VCE_CMD_NO_OP          0x00000000
>  #define VCE_CMD_END            0x00000001
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 01/23] drm/amdgpu: add support kernel interface queue(KIQ)

[Yu, Xiangliang]

> -----Original Message-----
> From: Alex Deucher [mailto:alexdeucher@gmail.com]
> Sent: Tuesday, December 20, 2016 7:11 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> dl.SRDC_SW_GPUVirtualization
> <dl.SRDC_SW_GPUVirtualization@amd.com>; Liu, Monk
> <Monk.Liu@amd.com>
> Subject: Re: [PATCH 01/23] drm/amdgpu: add support kernel interface
> queue(KIQ)
> 
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> wrote:
> > KIQ is queue-memory based initialization method: setup KIQ queue
> > firstly, then send command to KIQ to setup other queues, without
> > accessing registers.
> >
> > For virtualization, need KIQ to access virtual function registers when
> > running on guest mode.
> >
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> 
> kiq is just another CP ring.  Please add the asic specific parts to gfx_v8_0.c.  If
> there is any common KIQ helper code that is non asic dependent, that should
> be in amdgpu_kiq.c.

You are right, I erroneously think KIQ is not relate to chips, I'll put it into gfx_v8_0.c later.

> Alex
> 
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c | 436
> ++++++++++++++++++++++++++++++++
> >  drivers/gpu/drm/amd/amdgpu/vid.h        |   2 +
> >  2 files changed, 438 insertions(+)
> >  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
> > new file mode 100644
> > index 0000000..e9cdc28
> > --- /dev/null
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
> > @@ -0,0 +1,436 @@
> > +/*
> > + * Copyright 2016 Advanced Micro Devices, Inc.
> > + *
> > + * Permission is hereby granted, free of charge, to any person
> > +obtaining a
> > + * copy of this software and associated documentation files (the
> > +"Software"),
> > + * to deal in the Software without restriction, including without
> > +limitation
> > + * the rights to use, copy, modify, merge, publish, distribute,
> > +sublicense,
> > + * and/or sell copies of the Software, and to permit persons to whom
> > +the
> > + * Software is furnished to do so, subject to the following conditions:
> > + *
> > + * The above copyright notice and this permission notice shall be
> > +included in
> > + * all copies or substantial portions of the Software.
> > + *
> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
> KIND,
> > +EXPRESS OR
> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > +MERCHANTABILITY,
> > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN
> NO EVENT
> > +SHALL
> > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
> > +DAMAGES OR
> > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
> > +OTHERWISE,
> > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
> THE USE
> > +OR
> > + * OTHER DEALINGS IN THE SOFTWARE.
> > + *
> > + * Authors: Xiangliang.Yu@amd.com
> > + *         Monk.Liu@amd.com
> > + */
> > +#include "amdgpu.h"
> > +#include "vid.h"
> > +#include "vi_structs.h"
> > +#include "gca/gfx_8_0_d.h"
> > +#include "gca/gfx_8_0_sh_mask.h"
> > +
> > +enum amdgpu_kiq_irq {
> > +       AMDGPU_CP_KIQ_IRQ_DRIVER0 = 0,
> > +       AMDGPU_CP_KIQ_IRQ_LAST
> > +};
> > +
> > +int amdgpu_kiq_init_ring(struct amdgpu_device *adev, struct
> amdgpu_ring *ring,
> > +                        struct amdgpu_irq_src *irq) {
> > +       int r = 0;
> > +
> > +       ring->adev = NULL;
> > +       ring->ring_obj = NULL;
> > +       ring->use_doorbell = true;
> > +       ring->doorbell_index = AMDGPU_DOORBELL_KIQ;
> > +       if (adev->gfx.mec2_fw) {
> > +               ring->me = 2;
> > +               ring->pipe = 0;
> > +       } else {
> > +               ring->me = 1;
> > +               ring->pipe = 1;
> > +       }
> > +
> > +       irq->data = ring;
> > +       ring->queue = 0;
> > +       sprintf(ring->name, "kiq %d.%d.%d", ring->me, ring->pipe, ring-
> >queue);
> > +       r = amdgpu_ring_init(adev, ring, 1024,
> > +                            irq, AMDGPU_CP_KIQ_IRQ_DRIVER0);
> > +       if (r)
> > +               dev_warn(adev->dev, "(%d) failed to init kiq ring\n",
> > + r);
> > +
> > +       return r;
> > +}
> > +
> > +void amdgpu_kiq_free_ring(struct amdgpu_ring *ring, struct
> > +amdgpu_irq_src *irq) {
> > +       amdgpu_ring_fini(ring);
> > +       irq->data = NULL;
> > +}
> > +
> > +void amdgpu_kiq_enable(struct amdgpu_ring *ring) {
> > +       uint32_t tmp;
> > +       struct amdgpu_device *adev = ring->adev;
> > +
> > +       /* tell RLC which is KIQ queue */
> > +       tmp = RREG32(mmRLC_CP_SCHEDULERS);
> > +       tmp &= 0xffffff00;
> > +       tmp |= (ring->me << 5) | (ring->pipe << 3) | (ring->queue);
> > +       WREG32(mmRLC_CP_SCHEDULERS, tmp);
> > +       tmp |= 0x80;
> > +       WREG32(mmRLC_CP_SCHEDULERS, tmp); }
> > +
> > +void amdgpu_kiq_start(struct amdgpu_ring *ring) {
> > +       amdgpu_ring_alloc(ring, 8);
> > +       /* set resources */
> > +       amdgpu_ring_write(ring, PACKET3(PACKET3_SET_RESOURCES, 6));
> > +       amdgpu_ring_write(ring, 0);     /* vmid_mask:0 queue_type:0 (KIQ) */
> > +       amdgpu_ring_write(ring, 0x000000FF);    /* queue mask lo */
> > +       amdgpu_ring_write(ring, 0);     /* queue mask hi */
> > +       amdgpu_ring_write(ring, 0);     /* gws mask lo */
> > +       amdgpu_ring_write(ring, 0);     /* gws mask hi */
> > +       amdgpu_ring_write(ring, 0);     /* oac mask */
> > +       amdgpu_ring_write(ring, 0);     /* gds heap base:0, gds heap size:0 */
> > +       amdgpu_ring_commit(ring);
> > +       udelay(50);
> > +}
> > +
> > +void amdgpu_kiq_map_queue(struct amdgpu_ring *kiq_ring,
> > +                         struct amdgpu_ring *ring) {
> > +       struct amdgpu_device *adev = kiq_ring->adev;
> > +       uint64_t mqd_addr, wptr_addr;
> > +
> > +       mqd_addr = amdgpu_bo_gpu_offset(ring->mqd_obj);
> > +       wptr_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
> > +       amdgpu_ring_alloc(kiq_ring, 8);
> > +
> > +       amdgpu_ring_write(kiq_ring, PACKET3(PACKET3_MAP_QUEUES, 5));
> > +       /* Q_sel:0, vmid:0, vidmem: 1, engine:0, num_Q:1*/
> > +       amdgpu_ring_write(kiq_ring, 0x21010000);
> > +       amdgpu_ring_write(kiq_ring, (ring->doorbell_index << 2) |
> > +                       (ring->queue << 26) |
> > +                       (ring->pipe << 29) |
> > +                       ((ring->me == 1 ? 0 : 1) << 31)); /* doorbell */
> > +       amdgpu_ring_write(kiq_ring, lower_32_bits(mqd_addr));
> > +       amdgpu_ring_write(kiq_ring, upper_32_bits(mqd_addr));
> > +       amdgpu_ring_write(kiq_ring, lower_32_bits(wptr_addr));
> > +       amdgpu_ring_write(kiq_ring, upper_32_bits(wptr_addr));
> > +       amdgpu_ring_commit(kiq_ring);
> > +       udelay(50);
> > +}
> > +
> > +#define MEC_HPD_SIZE   2048
> > +
> > +int amdgpu_mqd_init(struct amdgpu_device *adev,
> > +                   struct vi_mqd *mqd,
> > +                   uint64_t mqd_gpu_addr,
> > +                   uint64_t eop_gpu_addr,
> > +                   struct amdgpu_ring *ring) {
> > +       uint64_t hqd_gpu_addr, wb_gpu_addr, eop_base_addr;
> > +       uint32_t tmp;
> > +
> > +       mqd->header = 0xC0310800;
> > +       mqd->compute_pipelinestat_enable = 0x00000001;
> > +       mqd->compute_static_thread_mgmt_se0 = 0xffffffff;
> > +       mqd->compute_static_thread_mgmt_se1 = 0xffffffff;
> > +       mqd->compute_static_thread_mgmt_se2 = 0xffffffff;
> > +       mqd->compute_static_thread_mgmt_se3 = 0xffffffff;
> > +       mqd->compute_misc_reserved = 0x00000003;
> > +
> > +       eop_base_addr = eop_gpu_addr >> 8;
> > +       mqd->cp_hqd_eop_base_addr_lo = eop_base_addr;
> > +       mqd->cp_hqd_eop_base_addr_hi = upper_32_bits(eop_base_addr);
> > +
> > +       /* set the EOP size, register value is 2^(EOP_SIZE+1) dwords */
> > +       tmp = RREG32(mmCP_HQD_EOP_CONTROL);
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_EOP_CONTROL, EOP_SIZE,
> > +                       (order_base_2(MEC_HPD_SIZE / 4) - 1));
> > +
> > +       mqd->cp_hqd_eop_control = tmp;
> > +
> > +       /* enable doorbell? */
> > +       tmp = RREG32(mmCP_HQD_PQ_DOORBELL_CONTROL);
> > +
> > +       if (ring->use_doorbell)
> > +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> > +                                        DOORBELL_EN, 1);
> > +       else
> > +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> > +                                        DOORBELL_EN, 0);
> > +
> > +       mqd->cp_hqd_pq_doorbell_control = tmp;
> > +
> > +       /* disable the queue if it's active */
> > +       mqd->cp_hqd_dequeue_request = 0;
> > +       mqd->cp_hqd_pq_rptr = 0;
> > +       mqd->cp_hqd_pq_wptr = 0;
> > +
> > +       /* set the pointer to the MQD */
> > +       mqd->cp_mqd_base_addr_lo = mqd_gpu_addr & 0xfffffffc;
> > +       mqd->cp_mqd_base_addr_hi = upper_32_bits(mqd_gpu_addr);
> > +
> > +       /* set MQD vmid to 0 */
> > +       tmp = RREG32(mmCP_MQD_CONTROL);
> > +       tmp = REG_SET_FIELD(tmp, CP_MQD_CONTROL, VMID, 0);
> > +       mqd->cp_mqd_control = tmp;
> > +
> > +       /* set the pointer to the HQD, this is similar CP_RB0_BASE/_HI */
> > +       hqd_gpu_addr = ring->gpu_addr >> 8;
> > +       mqd->cp_hqd_pq_base_lo = hqd_gpu_addr;
> > +       mqd->cp_hqd_pq_base_hi = upper_32_bits(hqd_gpu_addr);
> > +
> > +       /* set up the HQD, this is similar to CP_RB0_CNTL */
> > +       tmp = RREG32(mmCP_HQD_PQ_CONTROL);
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, QUEUE_SIZE,
> > +                           (order_base_2(ring->ring_size / 4) - 1));
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL,
> RPTR_BLOCK_SIZE,
> > +                       ((order_base_2(AMDGPU_GPU_PAGE_SIZE / 4) - 1)
> > +<< 8)); #ifdef __BIG_ENDIAN
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, ENDIAN_SWAP,
> 1);
> > +#endif
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL,
> UNORD_DISPATCH, 0);
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, ROQ_PQ_IB_FLIP,
> 0);
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);
> > +       mqd->cp_hqd_pq_control = tmp;
> > +
> > +       /* set the wb address whether it's enabled or not */
> > +       wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
> > +       mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
> > +       mqd->cp_hqd_pq_rptr_report_addr_hi =
> > +               upper_32_bits(wb_gpu_addr) & 0xffff;
> > +
> > +       /* only used if
> CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
> > +       wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
> > +       mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
> > +       mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr)
> > + & 0xffff;
> > +
> > +       tmp = 0;
> > +       /* enable the doorbell if requested */
> > +       if (ring->use_doorbell) {
> > +               tmp = RREG32(mmCP_HQD_PQ_DOORBELL_CONTROL);
> > +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> > +                               DOORBELL_OFFSET,
> > + ring->doorbell_index);
> > +
> > +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> > +                                        DOORBELL_EN, 1);
> > +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> > +                                        DOORBELL_SOURCE, 0);
> > +               tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
> > +                                        DOORBELL_HIT, 0);
> > +       }
> > +
> > +       mqd->cp_hqd_pq_doorbell_control = tmp;
> > +
> > +       /* reset read and write pointers, similar to CP_RB0_WPTR/_RPTR */
> > +       ring->wptr = 0;
> > +       mqd->cp_hqd_pq_wptr = ring->wptr;
> > +       mqd->cp_hqd_pq_rptr = RREG32(mmCP_HQD_PQ_RPTR);
> > +
> > +       /* set the vmid for the queue */
> > +       mqd->cp_hqd_vmid = 0;
> > +
> > +       tmp = RREG32(mmCP_HQD_PERSISTENT_STATE);
> > +       tmp = REG_SET_FIELD(tmp, CP_HQD_PERSISTENT_STATE,
> PRELOAD_SIZE, 0x53);
> > +       mqd->cp_hqd_persistent_state = tmp;
> > +
> > +       /* activate the queue */
> > +       mqd->cp_hqd_active = 1;
> > +
> > +       return 0;
> > +}
> > +
> > +int amdgpu_kiq_init(struct amdgpu_device *adev,
> > +                   struct vi_mqd *mqd,
> > +                   struct amdgpu_ring *ring) {
> > +       uint32_t tmp;
> > +       int j;
> > +
> > +       /* disable wptr polling */
> > +       tmp = RREG32(mmCP_PQ_WPTR_POLL_CNTL);
> > +       tmp = REG_SET_FIELD(tmp, CP_PQ_WPTR_POLL_CNTL, EN, 0);
> > +       WREG32(mmCP_PQ_WPTR_POLL_CNTL, tmp);
> > +
> > +       WREG32(mmCP_HQD_EOP_BASE_ADDR, mqd-
> >cp_hqd_eop_base_addr_lo);
> > +       WREG32(mmCP_HQD_EOP_BASE_ADDR_HI,
> > + mqd->cp_hqd_eop_base_addr_hi);
> > +
> > +       /* set the EOP size, register value is 2^(EOP_SIZE+1) dwords */
> > +       WREG32(mmCP_HQD_EOP_CONTROL, mqd->cp_hqd_eop_control);
> > +
> > +       /* enable doorbell? */
> > +       WREG32(mmCP_HQD_PQ_DOORBELL_CONTROL,
> > + mqd->cp_hqd_pq_doorbell_control);
> > +
> > +       /* disable the queue if it's active */
> > +       if (RREG32(mmCP_HQD_ACTIVE) & 1) {
> > +               WREG32(mmCP_HQD_DEQUEUE_REQUEST, 1);
> > +               for (j = 0; j < adev->usec_timeout; j++) {
> > +                       if (!(RREG32(mmCP_HQD_ACTIVE) & 1))
> > +                               break;
> > +                       udelay(1);
> > +               }
> > +               WREG32(mmCP_HQD_DEQUEUE_REQUEST, mqd-
> >cp_hqd_dequeue_request);
> > +               WREG32(mmCP_HQD_PQ_RPTR, mqd->cp_hqd_pq_rptr);
> > +               WREG32(mmCP_HQD_PQ_WPTR, mqd->cp_hqd_pq_wptr);
> > +       }
> > +
> > +       /* set the pointer to the MQD */
> > +       WREG32(mmCP_MQD_BASE_ADDR, mqd->cp_mqd_base_addr_lo);
> > +       WREG32(mmCP_MQD_BASE_ADDR_HI, mqd-
> >cp_mqd_base_addr_hi);
> > +
> > +       /* set MQD vmid to 0 */
> > +       WREG32(mmCP_MQD_CONTROL, mqd->cp_mqd_control);
> > +
> > +       /* set the pointer to the HQD, this is similar CP_RB0_BASE/_HI */
> > +       WREG32(mmCP_HQD_PQ_BASE, mqd->cp_hqd_pq_base_lo);
> > +       WREG32(mmCP_HQD_PQ_BASE_HI, mqd->cp_hqd_pq_base_hi);
> > +
> > +       /* set up the HQD, this is similar to CP_RB0_CNTL */
> > +       WREG32(mmCP_HQD_PQ_CONTROL, mqd->cp_hqd_pq_control);
> > +
> > +       /* set the wb address whether it's enabled or not */
> > +       WREG32(mmCP_HQD_PQ_RPTR_REPORT_ADDR,
> > +                               mqd->cp_hqd_pq_rptr_report_addr_lo);
> > +       WREG32(mmCP_HQD_PQ_RPTR_REPORT_ADDR_HI,
> > +                               mqd->cp_hqd_pq_rptr_report_addr_hi);
> > +
> > +       /* only used if
> CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
> > +       WREG32(mmCP_HQD_PQ_WPTR_POLL_ADDR, mqd-
> >cp_hqd_pq_wptr_poll_addr_lo);
> > +       WREG32(mmCP_HQD_PQ_WPTR_POLL_ADDR_HI,
> > + mqd->cp_hqd_pq_wptr_poll_addr_hi);
> > +
> > +       /* enable the doorbell if requested */
> > +       if (ring->use_doorbell) {
> > +               if ((adev->asic_type == CHIP_CARRIZO) ||
> > +                               (adev->asic_type == CHIP_FIJI) ||
> > +                               (adev->asic_type == CHIP_STONEY)) {
> > +                       WREG32(mmCP_MEC_DOORBELL_RANGE_LOWER,
> > +                                               AMDGPU_DOORBELL_KIQ << 2);
> > +                       WREG32(mmCP_MEC_DOORBELL_RANGE_UPPER,
> > +                                               AMDGPU_DOORBELL_MEC_RING7 << 2);
> > +               }
> > +       }
> > +       WREG32(mmCP_HQD_PQ_DOORBELL_CONTROL,
> > + mqd->cp_hqd_pq_doorbell_control);
> > +
> > +       /* reset read and write pointers, similar to CP_RB0_WPTR/_RPTR */
> > +       WREG32(mmCP_HQD_PQ_WPTR, mqd->cp_hqd_pq_wptr);
> > +
> > +       /* set the vmid for the queue */
> > +       WREG32(mmCP_HQD_VMID, mqd->cp_hqd_vmid);
> > +
> > +       WREG32(mmCP_HQD_PERSISTENT_STATE,
> > + mqd->cp_hqd_persistent_state);
> > +
> > +       /* activate the queue */
> > +       WREG32(mmCP_HQD_ACTIVE, mqd->cp_hqd_active);
> > +
> > +       if (ring->use_doorbell) {
> > +               tmp = RREG32(mmCP_PQ_STATUS);
> > +               tmp = REG_SET_FIELD(tmp, CP_PQ_STATUS, DOORBELL_ENABLE,
> 1);
> > +               WREG32(mmCP_PQ_STATUS, tmp);
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +void amdgpu_kiq_ring_emit_fence(struct amdgpu_ring *ring, u64 addr,
> > +                               u64 seq, unsigned int flags) {
> > +       /* we only allocate 32bit for each seq wb address */
> > +       if (flags & AMDGPU_FENCE_FLAG_64BIT)
> > +               BUG();
> > +
> > +       /* write fence seq to the "addr" */
> > +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
> > +       amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(0) |
> > +                                WRITE_DATA_DST_SEL(5) | WR_CONFIRM));
> > +       amdgpu_ring_write(ring, lower_32_bits(addr));
> > +       amdgpu_ring_write(ring, upper_32_bits(addr));
> > +       amdgpu_ring_write(ring, lower_32_bits(seq));
> > +
> > +       if (flags & AMDGPU_FENCE_FLAG_INT) {
> > +               /* set register to trigger INT */
> > +               amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
> > +               amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(0) |
> > +                                        WRITE_DATA_DST_SEL(0) | WR_CONFIRM));
> > +               amdgpu_ring_write(ring, mmCPC_INT_STATUS);
> > +               amdgpu_ring_write(ring, 0);
> > +               amdgpu_ring_write(ring, 0x20000000); /* src_id is 178 */
> > +       }
> > +}
> > +
> > +static int amdgpu_kiq_set_interrupt_state(struct amdgpu_device *adev,
> > +                                         struct amdgpu_irq_src *src,
> > +                                         unsigned int type,
> > +                                         enum amdgpu_interrupt_state
> > +state) {
> > +       uint32_t tmp, target;
> > +       struct amdgpu_ring *ring = (struct amdgpu_ring *)src->data;
> > +
> > +       BUG_ON(!ring || (ring->funcs->type != AMDGPU_RING_TYPE_KIQ));
> > +
> > +       target = (ring->me == 1) ? mmCP_ME1_PIPE0_INT_CNTL :
> mmCP_ME2_PIPE0_INT_CNTL;
> > +       target += ring->pipe;
> > +
> > +       switch (type) {
> > +       case AMDGPU_CP_KIQ_IRQ_DRIVER0:
> > +               if (state == AMDGPU_IRQ_STATE_DISABLE) {
> > +                       tmp = RREG32(mmCPC_INT_CNTL);
> > +                       tmp = REG_SET_FIELD(tmp, CPC_INT_CNTL,
> > +                                                GENERIC2_INT_ENABLE, 0);
> > +                       WREG32(mmCPC_INT_CNTL, tmp);
> > +
> > +                       tmp = RREG32(target);
> > +                       tmp = REG_SET_FIELD(tmp, CP_ME2_PIPE0_INT_CNTL,
> > +                                                GENERIC2_INT_ENABLE, 0);
> > +                       WREG32(target, tmp);
> > +               } else {
> > +                       tmp = RREG32(mmCPC_INT_CNTL);
> > +                       tmp = REG_SET_FIELD(tmp, CPC_INT_CNTL,
> > +                                                GENERIC2_INT_ENABLE, 1);
> > +                       WREG32(mmCPC_INT_CNTL, tmp);
> > +
> > +                       tmp = RREG32(target);
> > +                       tmp = REG_SET_FIELD(tmp, CP_ME2_PIPE0_INT_CNTL,
> > +                                                GENERIC2_INT_ENABLE, 1);
> > +                       WREG32(target, tmp);
> > +               }
> > +               break;
> > +       default:
> > +               BUG(); /* kiq only support GENERIC2_INT now */
> > +               break;
> > +       }
> > +       return 0;
> > +}
> > +
> > +static int amdgpu_kiq_irq(struct amdgpu_device *adev,
> > +                         struct amdgpu_irq_src *source,
> > +                         struct amdgpu_iv_entry *entry) {
> > +       u8 me_id, pipe_id, queue_id;
> > +       struct amdgpu_ring *ring = (struct amdgpu_ring *)source->data;
> > +
> > +       BUG_ON(!ring || (ring->funcs->type != AMDGPU_RING_TYPE_KIQ));
> > +
> > +       me_id = (entry->ring_id & 0x0c) >> 2;
> > +       pipe_id = (entry->ring_id & 0x03) >> 0;
> > +       queue_id = (entry->ring_id & 0x70) >> 4;
> > +       DRM_DEBUG("IH: CPC GENERIC2_INT, me:%d, pipe:%d, queue:%d\n",
> > +                  me_id, pipe_id, queue_id);
> > +
> > +       amdgpu_fence_process(ring);
> > +       return 0;
> > +}
> > +
> > +static const struct amdgpu_irq_src_funcs amdgpu_kiq_irq_funcs = {
> > +       .set = amdgpu_kiq_set_interrupt_state,
> > +       .process = amdgpu_kiq_irq,
> > +};
> > +
> > +void amdgpu_kiq_set_irq_funcs(struct amdgpu_irq_src *irq) {
> > +       irq->num_types = AMDGPU_CP_KIQ_IRQ_LAST;
> > +       irq->funcs = &amdgpu_kiq_irq_funcs; }
> > diff --git a/drivers/gpu/drm/amd/amdgpu/vid.h
> > b/drivers/gpu/drm/amd/amdgpu/vid.h
> > index 11746f2..7a3863a 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/vid.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/vid.h
> > @@ -360,6 +360,8 @@
> >  #define        PACKET3_WAIT_ON_CE_COUNTER                      0x86
> >  #define        PACKET3_WAIT_ON_DE_COUNTER_DIFF                 0x88
> >  #define        PACKET3_SWITCH_BUFFER                           0x8B
> > +#define        PACKET3_SET_RESOURCES                           0xA0
> > +#define        PACKET3_MAP_QUEUES                              0xA2
> >
> >  #define VCE_CMD_NO_OP          0x00000000
> >  #define VCE_CMD_END            0x00000001
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 02/23] drm/amdgpu: add kiq into compiling

[Xiangliang Yu]

Add support KIQ in Makefile to make it into module.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index dd0b6db..635ff94 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
 	atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
 	amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
 	amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
-	amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
+	amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
 
 # add asic specific block
 amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

转发: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Liu, Monk]

Sorry, NAK for the KIQ implement patches.

KIQ is just another compute queue, and each  generation of compute queue initialization is different, so there is no need of a amdgpu_kiq.c file to cover the function of kiq,

Kiq is good enough to be in each gfx_x_x.c file .e.g :
KIQ  for VI is in gfx_v8_0.c
And KIQ for AI is in gfx_v9_0.c

Most important is that the "queue_init" routine of above two chips on KIQ is different (MQD descriptor as well as registers need programing ... ), please just align with the previous patch style in amd-sriov-4.6 for KIQ part

BR Monk

-----邮件原件-----
发件人: Yu, Xiangliang 
发送时间: Sunday, December 18, 2016 9:47 AM
收件人: dl.SRDC_SW_GPUVirtualization
主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling



-----Original Message-----
From: Xiangliang Yu [mailto:Xiangliang.Yu@amd.com] 
Sent: Sunday, December 18, 2016 12:16 AM
To: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>
Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com>
Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling

Add support KIQ in Makefile to make it into module.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index dd0b6db..635ff94 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
 	atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
 	amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
 	amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
-	amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
+	amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
 
 # add asic specific block
 amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Yu, Xiangliang]

> Sorry, NAK for the KIQ implement patches.
> 
> KIQ is just another compute queue, and each  generation of compute queue
> initialization is different, so there is no need of a amdgpu_kiq.c file to cover
> the function of kiq,
> 
> Kiq is good enough to be in each gfx_x_x.c file .e.g :
> KIQ  for VI is in gfx_v8_0.c
> And KIQ for AI is in gfx_v9_0.c
> 
> Most important is that the "queue_init" routine of above two chips on KIQ is
> different (MQD descriptor as well as registers need programing ... ), please
> just align with the previous patch style in amd-sriov-4.6 for KIQ part

Sure.

> BR Monk
> 
> -----邮件原件-----
> 发件人: Yu, Xiangliang
> 发送时间: Sunday, December 18, 2016 9:47 AM
> 收件人: dl.SRDC_SW_GPUVirtualization
> 主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling
> 
> 
> 
> -----Original Message-----
> From: Xiangliang Yu [mailto:Xiangliang.Yu@amd.com]
> Sent: Sunday, December 18, 2016 12:16 AM
> To: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization
> <dl.SRDC_SW_GPUVirtualization@amd.com>
> Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling
> 
> Add support KIQ in Makefile to make it into module.
> 
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile
> b/drivers/gpu/drm/amd/amdgpu/Makefile
> index dd0b6db..635ff94 100644
> --- a/drivers/gpu/drm/amd/amdgpu/Makefile
> +++ b/drivers/gpu/drm/amd/amdgpu/Makefile
> @@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
>  	atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
>  	amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
>  	amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
> -	amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
> +	amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
> 
>  # add asic specific block
>  amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o
> kv_dpm.o \
> --
> 2.7.4
> 
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: 转发: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Christian König]

Yeah, agree totally with Monk here.

If you find that you need some KIQ helpers which are independent of the 
hardware generation you can put them into amdgpu_kiq.c.

Monk, David or maybe even Michel and Nicolai could you guys take a 
closer look at the patches.

I'm currently to busy with private issues, so I just briefly skimmed 
over them, but still found some issues.

The VA update stuff seems incorrectly handled to me for example. You 
only update the page tables once and then assume that everything stays 
as it is.

Regards,
Christian.

Am 19.12.2016 um 08:13 schrieb Liu, Monk:
> Sorry, NAK for the KIQ implement patches.
>
> KIQ is just another compute queue, and each  generation of compute queue initialization is different, so there is no need of a amdgpu_kiq.c file to cover the function of kiq,
>
> Kiq is good enough to be in each gfx_x_x.c file .e.g :
> KIQ  for VI is in gfx_v8_0.c
> And KIQ for AI is in gfx_v9_0.c
>
> Most important is that the "queue_init" routine of above two chips on KIQ is different (MQD descriptor as well as registers need programing ... ), please just align with the previous patch style in amd-sriov-4.6 for KIQ part
>
> BR Monk
>
> -----邮件原件-----
> 发件人: Yu, Xiangliang
> 发送时间: Sunday, December 18, 2016 9:47 AM
> 收件人: dl.SRDC_SW_GPUVirtualization
> 主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
>
>
> -----Original Message-----
> From: Xiangliang Yu [mailto:Xiangliang.Yu@amd.com]
> Sent: Sunday, December 18, 2016 12:16 AM
> To: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>
> Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
> Add support KIQ in Makefile to make it into module.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
> index dd0b6db..635ff94 100644
> --- a/drivers/gpu/drm/amd/amdgpu/Makefile
> +++ b/drivers/gpu/drm/amd/amdgpu/Makefile
> @@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
>   	atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
>   	amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
>   	amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
> -	amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
> +	amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
>   
>   # add asic specific block
>   amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \


_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: 转发: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Alex Deucher]

On Mon, Dec 19, 2016 at 6:17 AM, Christian König
<deathsimple@vodafone.de> wrote:
> Yeah, agree totally with Monk here.
>
> If you find that you need some KIQ helpers which are independent of the
> hardware generation you can put them into amdgpu_kiq.c.
>
> Monk, David or maybe even Michel and Nicolai could you guys take a closer
> look at the patches.
>
> I'm currently to busy with private issues, so I just briefly skimmed over
> them, but still found some issues.
>
> The VA update stuff seems incorrectly handled to me for example. You only
> update the page tables once and then assume that everything stays as it is.
>

I agree with all of Christian and Monk's comments.  Please see my
comments on the patches.

Alex

> Regards,
> Christian.
>
>
> Am 19.12.2016 um 08:13 schrieb Liu, Monk:
>>
>> Sorry, NAK for the KIQ implement patches.
>>
>> KIQ is just another compute queue, and each  generation of compute queue
>> initialization is different, so there is no need of a amdgpu_kiq.c file to
>> cover the function of kiq,
>>
>> Kiq is good enough to be in each gfx_x_x.c file .e.g :
>> KIQ  for VI is in gfx_v8_0.c
>> And KIQ for AI is in gfx_v9_0.c
>>
>> Most important is that the "queue_init" routine of above two chips on KIQ
>> is different (MQD descriptor as well as registers need programing ... ),
>> please just align with the previous patch style in amd-sriov-4.6 for KIQ
>> part
>>
>> BR Monk
>>
>> -----邮件原件-----
>> 发件人: Yu, Xiangliang
>> 发送时间: Sunday, December 18, 2016 9:47 AM
>> 收件人: dl.SRDC_SW_GPUVirtualization
>> 主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>>
>>
>>
>> -----Original Message-----
>> From: Xiangliang Yu [mailto:Xiangliang.Yu@amd.com]
>> Sent: Sunday, December 18, 2016 12:16 AM
>> To: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization
>> <dl.SRDC_SW_GPUVirtualization@amd.com>
>> Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com>
>> Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>>
>> Add support KIQ in Makefile to make it into module.
>>
>> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile
>> b/drivers/gpu/drm/amd/amdgpu/Makefile
>> index dd0b6db..635ff94 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/Makefile
>> +++ b/drivers/gpu/drm/amd/amdgpu/Makefile
>> @@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
>>         atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
>>         amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
>>         amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
>> -       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
>> +       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
>>     # add asic specific block
>>   amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
>
>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

答复: 转发: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Liu, Monk]

[-- Attachment #1.1: Type: text/plain, Size: 3611 bytes --]

yeah, I'll take a deep look, we are not in hurry to submit those SRIOV patches, for the VA update part (if you mean CSA mappings) I'll pay attention on it, thanks


BR Monk

________________________________
发件人: Alex Deucher <alexdeucher@gmail.com>
发送时间: 2016年12月20日 7:38:10
收件人: Christian König
抄送: Liu, Monk; amd-gfx@lists.freedesktop.org
主题: Re: 转发: [PATCH 02/23] drm/amdgpu: add kiq into compiling

On Mon, Dec 19, 2016 at 6:17 AM, Christian König
<deathsimple@vodafone.de> wrote:
> Yeah, agree totally with Monk here.
>
> If you find that you need some KIQ helpers which are independent of the
> hardware generation you can put them into amdgpu_kiq.c.
>
> Monk, David or maybe even Michel and Nicolai could you guys take a closer
> look at the patches.
>
> I'm currently to busy with private issues, so I just briefly skimmed over
> them, but still found some issues.
>
> The VA update stuff seems incorrectly handled to me for example. You only
> update the page tables once and then assume that everything stays as it is.
>

I agree with all of Christian and Monk's comments.  Please see my
comments on the patches.

Alex

> Regards,
> Christian.
>
>
> Am 19.12.2016 um 08:13 schrieb Liu, Monk:
>>
>> Sorry, NAK for the KIQ implement patches.
>>
>> KIQ is just another compute queue, and each  generation of compute queue
>> initialization is different, so there is no need of a amdgpu_kiq.c file to
>> cover the function of kiq,
>>
>> Kiq is good enough to be in each gfx_x_x.c file .e.g :
>> KIQ  for VI is in gfx_v8_0.c
>> And KIQ for AI is in gfx_v9_0.c
>>
>> Most important is that the "queue_init" routine of above two chips on KIQ
>> is different (MQD descriptor as well as registers need programing ... ),
>> please just align with the previous patch style in amd-sriov-4.6 for KIQ
>> part
>>
>> BR Monk
>>
>> -----邮件原件-----
>> 发件人: Yu, Xiangliang
>> 发送时间: Sunday, December 18, 2016 9:47 AM
>> 收件人: dl.SRDC_SW_GPUVirtualization
>> 主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>>
>>
>>
>> -----Original Message-----
>> From: Xiangliang Yu [mailto:Xiangliang.Yu@amd.com]
>> Sent: Sunday, December 18, 2016 12:16 AM
>> To: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization
>> <dl.SRDC_SW_GPUVirtualization@amd.com>
>> Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com>
>> Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>>
>> Add support KIQ in Makefile to make it into module.
>>
>> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile
>> b/drivers/gpu/drm/amd/amdgpu/Makefile
>> index dd0b6db..635ff94 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/Makefile
>> +++ b/drivers/gpu/drm/amd/amdgpu/Makefile
>> @@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
>>         atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
>>         amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
>>         amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
>> -       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
>> +       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
>>     # add asic specific block
>>   amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
>
>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 5766 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

答复: 转发: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Liu, Monk]

[-- Attachment #1.1: Type: text/plain, Size: 3480 bytes --]


Hi Christian,


the CSA VA update should be correct, we pin it after it allocated per device during init stage, and we map it upon each process creates their vm page tables,


since this CSA is pined, I assume there is no need to validate it prior to submission


please correct me if I missed something


BR Monk

________________________________
发件人: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> 代表 Christian König <deathsimple@vodafone.de>
发送时间: 2016年12月19日 19:17:38
收件人: Liu, Monk; amd-gfx@lists.freedesktop.org
主题: Re: 转发: [PATCH 02/23] drm/amdgpu: add kiq into compiling

Yeah, agree totally with Monk here.

If you find that you need some KIQ helpers which are independent of the
hardware generation you can put them into amdgpu_kiq.c.

Monk, David or maybe even Michel and Nicolai could you guys take a
closer look at the patches.

I'm currently to busy with private issues, so I just briefly skimmed
over them, but still found some issues.

The VA update stuff seems incorrectly handled to me for example. You
only update the page tables once and then assume that everything stays
as it is.

Regards,
Christian.

Am 19.12.2016 um 08:13 schrieb Liu, Monk:
> Sorry, NAK for the KIQ implement patches.
>
> KIQ is just another compute queue, and each  generation of compute queue initialization is different, so there is no need of a amdgpu_kiq.c file to cover the function of kiq,
>
> Kiq is good enough to be in each gfx_x_x.c file .e.g :
> KIQ  for VI is in gfx_v8_0.c
> And KIQ for AI is in gfx_v9_0.c
>
> Most important is that the "queue_init" routine of above two chips on KIQ is different (MQD descriptor as well as registers need programing ... ), please just align with the previous patch style in amd-sriov-4.6 for KIQ part
>
> BR Monk
>
> -----邮件原件-----
> 发件人: Yu, Xiangliang
> 发送时间: Sunday, December 18, 2016 9:47 AM
> 收件人: dl.SRDC_SW_GPUVirtualization
> 主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
>
>
> -----Original Message-----
> From: Xiangliang Yu [mailto:Xiangliang.Yu@amd.com]
> Sent: Sunday, December 18, 2016 12:16 AM
> To: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>
> Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
> Add support KIQ in Makefile to make it into module.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
> index dd0b6db..635ff94 100644
> --- a/drivers/gpu/drm/amd/amdgpu/Makefile
> +++ b/drivers/gpu/drm/amd/amdgpu/Makefile
> @@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
>        atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
>        amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
>        amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
> -     amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
> +     amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
>
>   # add asic specific block
>   amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \


_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 5290 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Mike Lothian]

[-- Attachment #1.1: Type: text/plain, Size: 2573 bytes --]

I'd be curious to know what AI stands for in this context

On Mon, 19 Dec 2016 at 07:28 Liu, Monk <Monk.Liu-5C7GfCeVMHo@public.gmane.org> wrote:

> Sorry, NAK for the KIQ implement patches.
>
> KIQ is just another compute queue, and each  generation of compute queue
> initialization is different, so there is no need of a amdgpu_kiq.c file to
> cover the function of kiq,
>
> Kiq is good enough to be in each gfx_x_x.c file .e.g :
> KIQ  for VI is in gfx_v8_0.c
> And KIQ for AI is in gfx_v9_0.c
>
> Most important is that the "queue_init" routine of above two chips on KIQ
> is different (MQD descriptor as well as registers need programing ... ),
> please just align with the previous patch style in amd-sriov-4.6 for KIQ
> part
>
> BR Monk
>
> -----邮件原件-----
> 发件人: Yu, Xiangliang
> 发送时间: Sunday, December 18, 2016 9:47 AM
> 收件人: dl.SRDC_SW_GPUVirtualization
> 主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
>
>
> -----Original Message-----
> From: Xiangliang Yu [mailto:Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org]
> Sent: Sunday, December 18, 2016 12:16 AM
> To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org; dl.SRDC_SW_GPUVirtualization <
> dl.SRDC_SW_GPUVirtualization-5C7GfCeVMHo@public.gmane.org>
> Cc: Yu, Xiangliang <Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org>
> Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
> Add support KIQ in Makefile to make it into module.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org>
> ---
>  drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile
> b/drivers/gpu/drm/amd/amdgpu/Makefile
> index dd0b6db..635ff94 100644
> --- a/drivers/gpu/drm/amd/amdgpu/Makefile
> +++ b/drivers/gpu/drm/amd/amdgpu/Makefile
> @@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
>         atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
>         amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
>         amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
> -       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
> +       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
>
>  # add asic specific block
>  amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>

[-- Attachment #1.2: Type: text/html, Size: 4711 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Bridgman, John]

[-- Attachment #1.1: Type: text/plain, Size: 2805 bytes --]

An upcoming GPU generation (Vega).


From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf Of Mike Lothian
Sent: Monday, December 19, 2016 10:49 PM
To: Liu, Monk; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 02/23] drm/amdgpu: add kiq into compiling

I'd be curious to know what AI stands for in this context
On Mon, 19 Dec 2016 at 07:28 Liu, Monk <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>> wrote:
Sorry, NAK for the KIQ implement patches.

KIQ is just another compute queue, and each  generation of compute queue initialization is different, so there is no need of a amdgpu_kiq.c file to cover the function of kiq,

Kiq is good enough to be in each gfx_x_x.c file .e.g :
KIQ  for VI is in gfx_v8_0.c
And KIQ for AI is in gfx_v9_0.c

Most important is that the "queue_init" routine of above two chips on KIQ is different (MQD descriptor as well as registers need programing ... ), please just align with the previous patch style in amd-sriov-4.6 for KIQ part

BR Monk

-----邮件原件-----
发件人: Yu, Xiangliang
发送时间: Sunday, December 18, 2016 9:47 AM
收件人: dl.SRDC_SW_GPUVirtualization
主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling



-----Original Message-----
From: Xiangliang Yu [mailto:Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>]
Sent: Sunday, December 18, 2016 12:16 AM
To: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com<mailto:dl.SRDC_SW_GPUVirtualization@amd.com>>
Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>>
Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling

Add support KIQ in Makefile to make it into module.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>>
---
 drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index dd0b6db..635ff94 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
        atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
        amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
        amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
-       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
+       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o

 # add asic specific block
 amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
--
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 7358 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 02/23] drm/amdgpu: add kiq into compiling

[Mike Lothian]

[-- Attachment #1.1: Type: text/plain, Size: 3142 bytes --]

Ah, I've clicked now - I thought Vega was a star name

AI - Arctic Islands :D

On Tue, 20 Dec 2016 at 04:07 Bridgman, John <John.Bridgman-5C7GfCeVMHo@public.gmane.org> wrote:

> An upcoming GPU generation (Vega).
>
>
>
>
>
> *From:* amd-gfx [mailto:amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org] *On Behalf
> Of *Mike Lothian
> *Sent:* Monday, December 19, 2016 10:49 PM
> *To:* Liu, Monk; amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> *Subject:* Re: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
>
>
> I'd be curious to know what AI stands for in this context
>
> On Mon, 19 Dec 2016 at 07:28 Liu, Monk <Monk.Liu-5C7GfCeVMHo@public.gmane.org> wrote:
>
> Sorry, NAK for the KIQ implement patches.
>
> KIQ is just another compute queue, and each  generation of compute queue
> initialization is different, so there is no need of a amdgpu_kiq.c file to
> cover the function of kiq,
>
> Kiq is good enough to be in each gfx_x_x.c file .e.g :
> KIQ  for VI is in gfx_v8_0.c
> And KIQ for AI is in gfx_v9_0.c
>
> Most important is that the "queue_init" routine of above two chips on KIQ
> is different (MQD descriptor as well as registers need programing ... ),
> please just align with the previous patch style in amd-sriov-4.6 for KIQ
> part
>
> BR Monk
>
> -----邮件原件-----
> 发件人: Yu, Xiangliang
> 发送时间: Sunday, December 18, 2016 9:47 AM
> 收件人: dl.SRDC_SW_GPUVirtualization
> 主题: FW: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
>
>
> -----Original Message-----
> From: Xiangliang Yu [mailto:Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org]
> Sent: Sunday, December 18, 2016 12:16 AM
> To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org; dl.SRDC_SW_GPUVirtualization <
> dl.SRDC_SW_GPUVirtualization-5C7GfCeVMHo@public.gmane.org>
> Cc: Yu, Xiangliang <Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org>
> Subject: [PATCH 02/23] drm/amdgpu: add kiq into compiling
>
> Add support KIQ in Makefile to make it into module.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org>
> ---
>  drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile
> b/drivers/gpu/drm/amd/amdgpu/Makefile
> index dd0b6db..635ff94 100644
> --- a/drivers/gpu/drm/amd/amdgpu/Makefile
> +++ b/drivers/gpu/drm/amd/amdgpu/Makefile
> @@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
>         atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
>         amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \
>         amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
> -       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o
> +       amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_kiq.o
>
>  # add asic specific block
>  amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>
>

[-- Attachment #1.2: Type: text/html, Size: 8297 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 03/23] drm/amdgpu: export KIQ interfaces

[Xiangliang Yu]

Export KIQ interfaces so that others components can use the
interfaces to setup queues.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index b7f521a..bc25c7b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -157,6 +157,7 @@ struct amdgpu_cs_parser;
 struct amdgpu_job;
 struct amdgpu_irq_src;
 struct amdgpu_fpriv;
+struct vi_mqd;
 
 enum amdgpu_cp_irq {
 	AMDGPU_CP_IRQ_GFX_EOP = 0,
@@ -691,6 +692,25 @@ void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr);
 void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr);
 
 /*
+ * Kernel Interface Queue functions
+ */
+int amdgpu_kiq_init_ring(struct amdgpu_device *adev, struct amdgpu_ring *ring,
+			 struct amdgpu_irq_src *irq);
+void amdgpu_kiq_free_ring(struct amdgpu_ring *ring, struct amdgpu_irq_src *irq);
+void amdgpu_kiq_enable(struct amdgpu_ring *ring);
+void amdgpu_kiq_start(struct amdgpu_ring *ring);
+int amdgpu_kiq_init(struct amdgpu_device *adev, struct vi_mqd *mqd,
+		    struct amdgpu_ring *ring);
+void amdgpu_kiq_map_queue(struct amdgpu_ring *kiq_ring,
+			  struct amdgpu_ring *ring);
+int amdgpu_mqd_init(struct amdgpu_device *adev, struct vi_mqd *mqd,
+		    uint64_t mqd_gpu_addr, uint64_t eop_gpu_addr,
+		    struct amdgpu_ring *ring);
+void amdgpu_kiq_ring_emit_fence(struct amdgpu_ring *ring, u64 addr,
+				u64 seq, unsigned flags);
+void amdgpu_kiq_set_irq_funcs(struct amdgpu_irq_src *irq);
+
+/*
  * file private structure
  */
 
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 04/23] drm/amdgpu: add new structure for KIQ memory allcation

[Xiangliang Yu]

KIQ need three memory spaces: interrupt, ring and buffer object. If
want to setup KIQ, must to allocate related memory firstly.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index bc25c7b..25a3a2a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -691,6 +691,13 @@ int amdgpu_ctx_ioctl(struct drm_device *dev, void *data,
 void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr);
 void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr);
 
+struct amdgpu_kiq {
+	u64			eop_gpu_addr;
+	struct amdgpu_bo	*eop_obj;
+	struct amdgpu_ring	ring;
+	struct amdgpu_irq_src	irq;
+};
+
 /*
  * Kernel Interface Queue functions
  */
@@ -1461,6 +1468,9 @@ struct amdgpu_device {
 	/* gfx */
 	struct amdgpu_gfx		gfx;
 
+	/* KIQ */
+	struct amdgpu_kiq		kiq;
+
 	/* sdma */
 	struct amdgpu_sdma		sdma;
 
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 05/23] drm/amdgpu/gfx8: add support KIQ for FIJI/TONGA chips

[Xiangliang Yu]

FIJI/TONGA chips must enable KIQ feature to support virtualization.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
Signed-off-by: Monk Liu <Monk.Liu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 237 +++++++++++++++++++++++++++++++++-
 1 file changed, 236 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
index a7c5113..ae20cd9 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
@@ -1410,6 +1410,67 @@ static int gfx_v8_0_mec_init(struct amdgpu_device *adev)
 	return 0;
 }
 
+static void gfx_v8_0_kiq_fini(struct amdgpu_device *adev)
+{
+	int r;
+
+	if (adev->kiq.eop_obj) {
+		r = amdgpu_bo_reserve(adev->kiq.eop_obj, false);
+		if (unlikely(r != 0))
+			dev_warn(adev->dev, "(%d) reserve HPD EOP bo failed\n", r);
+		amdgpu_bo_unpin(adev->kiq.eop_obj);
+		amdgpu_bo_unreserve(adev->kiq.eop_obj);
+		amdgpu_bo_unref(&adev->kiq.eop_obj);
+		adev->kiq.eop_obj = NULL;
+	}
+}
+
+static int gfx_v8_0_kiq_init(struct amdgpu_device *adev)
+{
+	int r;
+	u32 *hpd;
+
+	if (adev->kiq.eop_obj == NULL) {
+		r = amdgpu_bo_create(adev,
+				     MEC_HPD_SIZE,
+				     PAGE_SIZE, true,
+				     AMDGPU_GEM_DOMAIN_GTT, 0, NULL, NULL,
+				     &adev->kiq.eop_obj);
+		if (r) {
+			dev_warn(adev->dev, "(%d) create HDP EOP bo failed\n", r);
+			return r;
+		}
+	}
+
+	r = amdgpu_bo_reserve(adev->kiq.eop_obj, false);
+	if (unlikely(r != 0)) {
+		gfx_v8_0_kiq_fini(adev);
+		return r;
+	}
+
+	r = amdgpu_bo_pin(adev->kiq.eop_obj, AMDGPU_GEM_DOMAIN_GTT,
+			  &adev->kiq.eop_gpu_addr);
+	if (r) {
+		dev_warn(adev->dev, "(%d) pin HDP EOP bo failed\n", r);
+		gfx_v8_0_kiq_fini(adev);
+		return r;
+	}
+
+	r = amdgpu_bo_kmap(adev->kiq.eop_obj, (void **)&hpd);
+	if (r) {
+		dev_warn(adev->dev, "(%d) map HDP EOP bo failed\n", r);
+		gfx_v8_0_kiq_fini(adev);
+		return r;
+	}
+
+	memset(hpd, 0, MEC_HPD_SIZE);
+
+	amdgpu_bo_kunmap(adev->kiq.eop_obj);
+	amdgpu_bo_unreserve(adev->kiq.eop_obj);
+
+	return 0;
+}
+
 static const u32 vgpr_init_compute_shader[] =
 {
 	0x7e000209, 0x7e020208,
@@ -1987,6 +2048,11 @@ static int gfx_v8_0_sw_init(void *handle)
 	struct amdgpu_ring *ring;
 	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
 
+	/* KIQ event */
+	r = amdgpu_irq_add_id(adev, 178, &adev->kiq.irq);
+	if (r)
+		return r;
+
 	/* EOP Event */
 	r = amdgpu_irq_add_id(adev, 181, &adev->gfx.eop_irq);
 	if (r)
@@ -2024,6 +2090,16 @@ static int gfx_v8_0_sw_init(void *handle)
 		return r;
 	}
 
+	r = gfx_v8_0_kiq_init(adev);
+	if (r) {
+		DRM_ERROR("Failed to init KIQ BOs!\n");
+		return r;
+	}
+
+	r = amdgpu_kiq_init_ring(adev, &adev->kiq.ring, &adev->kiq.irq);
+	if (r)
+		return r;
+
 	/* set up the gfx ring */
 	for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
 		ring = &adev->gfx.gfx_ring[i];
@@ -2107,7 +2183,9 @@ static int gfx_v8_0_sw_fini(void *handle)
 		amdgpu_ring_fini(&adev->gfx.gfx_ring[i]);
 	for (i = 0; i < adev->gfx.num_compute_rings; i++)
 		amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
+	amdgpu_kiq_free_ring(&adev->kiq.ring, &adev->kiq.irq);
 
+	gfx_v8_0_kiq_fini(adev);
 	gfx_v8_0_mec_fini(adev);
 	gfx_v8_0_rlc_fini(adev);
 	gfx_v8_0_free_microcode(adev);
@@ -4468,6 +4546,131 @@ static int gfx_v8_0_cp_compute_load_microcode(struct amdgpu_device *adev)
 	return 0;
 }
 
+static int gfx_v8_0_kiq_init_queue(struct amdgpu_ring *ring,
+				   struct vi_mqd *mqd,
+				   u64 mqd_gpu_addr)
+{
+	struct amdgpu_device *adev = ring->adev;
+	struct amdgpu_kiq *kiq = &adev->kiq;
+	uint64_t eop_gpu_addr;
+	bool is_kiq = false;
+
+	if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
+		is_kiq = true;
+
+	if (is_kiq) {
+		eop_gpu_addr = kiq->eop_gpu_addr;
+		amdgpu_kiq_enable(&kiq->ring);
+	} else
+		eop_gpu_addr = adev->gfx.mec.hpd_eop_gpu_addr + ring->queue * MEC_HPD_SIZE;
+
+	mutex_lock(&adev->srbm_mutex);
+	vi_srbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
+
+	amdgpu_mqd_init(adev, mqd, mqd_gpu_addr, eop_gpu_addr, ring);
+
+	if (is_kiq)
+		amdgpu_kiq_init(adev, mqd, ring);
+
+	vi_srbm_select(adev, 0, 0, 0, 0);
+	mutex_unlock(&adev->srbm_mutex);
+
+	if (is_kiq)
+		amdgpu_kiq_start(ring);
+	else
+		amdgpu_kiq_map_queue(&kiq->ring, ring);
+
+	return 0;
+}
+
+static int gfx_v8_0_kiq_map_queue(struct amdgpu_device *adev,
+				  struct amdgpu_ring *ring)
+{
+	struct vi_mqd *mqd;
+	u64 mqd_gpu_addr;
+	u32 *buf;
+	int r = 0;
+
+	r = amdgpu_bo_create(adev, sizeof(struct vi_mqd), PAGE_SIZE, true,
+			     AMDGPU_GEM_DOMAIN_GTT, 0, NULL, NULL,
+			     &ring->mqd_obj);
+	if (r) {
+		dev_warn(adev->dev, "(%d) create MQD bo failed\n", r);
+		return r;
+	}
+	r = amdgpu_bo_reserve(ring->mqd_obj, false);
+	if (unlikely(r != 0)) {
+		amdgpu_bo_unref(&ring->mqd_obj);
+		return r;
+	}
+	r = amdgpu_bo_pin(ring->mqd_obj, AMDGPU_GEM_DOMAIN_GTT,
+			  &mqd_gpu_addr);
+	if (r) {
+		dev_warn(adev->dev, "(%d) pin MQD bo failed\n", r);
+		amdgpu_bo_unreserve(ring->mqd_obj);
+		amdgpu_bo_unref(&ring->mqd_obj);
+		return r;
+	}
+	r = amdgpu_bo_kmap(ring->mqd_obj, (void **)&buf);
+	if (r) {
+		dev_warn(adev->dev, "(%d) map MQD bo failed\n", r);
+		amdgpu_bo_unpin(ring->mqd_obj);
+		amdgpu_bo_unreserve(ring->mqd_obj);
+		amdgpu_bo_unref(&ring->mqd_obj);
+		return r;
+	}
+
+	/* init the mqd struct */
+	memset(buf, 0, sizeof(struct vi_mqd));
+	mqd = (struct vi_mqd *)buf;
+
+	r = gfx_v8_0_kiq_init_queue(ring, mqd, mqd_gpu_addr);
+	if (r)
+		return r;
+
+	amdgpu_bo_kunmap(ring->mqd_obj);
+	amdgpu_bo_unreserve(ring->mqd_obj);
+
+	return 0;
+}
+
+static int gfx_v8_0_kiq_resume(struct amdgpu_device *adev)
+{
+	struct amdgpu_ring *ring = NULL;
+	int r, i;
+
+	ring = &adev->kiq.ring;
+	r = gfx_v8_0_kiq_map_queue(adev, ring);
+	if (r)
+		return r;
+
+	for (i = 0; i < adev->gfx.num_compute_rings; i++) {
+		ring = &adev->gfx.compute_ring[i];
+		r = gfx_v8_0_kiq_map_queue(adev, ring);
+		if (r)
+			return r;
+	}
+
+	gfx_v8_0_cp_compute_enable(adev, true);
+
+	for (i = 0; i < adev->gfx.num_compute_rings; i++) {
+		ring = &adev->gfx.compute_ring[i];
+
+		ring->ready = true;
+		r = amdgpu_ring_test_ring(ring);
+		if (r)
+			ring->ready = false;
+	}
+
+	ring = &adev->kiq.ring;
+	ring->ready = true;
+	r = amdgpu_ring_test_ring(ring);
+	if (r)
+		ring->ready = false;
+
+	return 0;
+}
+
 static void gfx_v8_0_cp_compute_fini(struct amdgpu_device *adev)
 {
 	int i, r;
@@ -4787,7 +4990,10 @@ static int gfx_v8_0_cp_resume(struct amdgpu_device *adev)
 	if (r)
 		return r;
 
-	r = gfx_v8_0_cp_compute_resume(adev);
+	if (amdgpu_sriov_vf(adev))
+		r = gfx_v8_0_kiq_resume(adev);
+	else
+		r = gfx_v8_0_cp_compute_resume(adev);
 	if (r)
 		return r;
 
@@ -6464,10 +6670,37 @@ static const struct amdgpu_ring_funcs gfx_v8_0_ring_funcs_compute = {
 	.pad_ib = amdgpu_ring_generic_pad_ib,
 };
 
+static const struct amdgpu_ring_funcs gfx_v8_0_ring_funcs_kiq = {
+	.type = AMDGPU_RING_TYPE_KIQ,
+	.align_mask = 0xff,
+	.nop = PACKET3(PACKET3_NOP, 0x3FFF),
+	.get_rptr = gfx_v8_0_ring_get_rptr,
+	.get_wptr = gfx_v8_0_ring_get_wptr_compute,
+	.set_wptr = gfx_v8_0_ring_set_wptr_compute,
+	.emit_frame_size =
+		20 + /* gfx_v8_0_ring_emit_gds_switch */
+		7 + /* gfx_v8_0_ring_emit_hdp_flush */
+		5 + /* gfx_v8_0_ring_emit_hdp_invalidate */
+		7 + /* gfx_v8_0_ring_emit_pipeline_sync */
+		17 + /* gfx_v8_0_ring_emit_vm_flush */
+		7 + 7 + 7, /* gfx_v8_0_ring_emit_fence_compute x3 for user fence, vm fence */
+	.emit_ib_size =	4, /* gfx_v8_0_ring_emit_ib_compute */
+	.emit_ib = gfx_v8_0_ring_emit_ib_compute,
+	.emit_fence = amdgpu_kiq_ring_emit_fence,
+	.emit_hdp_flush = gfx_v8_0_ring_emit_hdp_flush,
+	.emit_hdp_invalidate = gfx_v8_0_ring_emit_hdp_invalidate,
+	.test_ring = gfx_v8_0_ring_test_ring,
+	.test_ib = gfx_v8_0_ring_test_ib,
+	.insert_nop = amdgpu_ring_insert_nop,
+	.pad_ib = amdgpu_ring_generic_pad_ib,
+};
+
 static void gfx_v8_0_set_ring_funcs(struct amdgpu_device *adev)
 {
 	int i;
 
+	adev->kiq.ring.funcs = &gfx_v8_0_ring_funcs_kiq;
+
 	for (i = 0; i < adev->gfx.num_gfx_rings; i++)
 		adev->gfx.gfx_ring[i].funcs = &gfx_v8_0_ring_funcs_gfx;
 
@@ -6500,6 +6733,8 @@ static void gfx_v8_0_set_irq_funcs(struct amdgpu_device *adev)
 
 	adev->gfx.priv_inst_irq.num_types = 1;
 	adev->gfx.priv_inst_irq.funcs = &gfx_v8_0_priv_inst_irq_funcs;
+
+	amdgpu_kiq_set_irq_funcs(&adev->kiq.irq);
 }
 
 static void gfx_v8_0_set_rlc_funcs(struct amdgpu_device *adev)
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 06/23] drm/amdgpu/gfx8: correct KIQ hdp flush

[Xiangliang Yu]

KIQ has some behavior as compute ring.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
index ae20cd9..a5a9a8e 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
@@ -6196,7 +6196,8 @@ static void gfx_v8_0_ring_emit_hdp_flush(struct amdgpu_ring *ring)
 {
 	u32 ref_and_mask, reg_mem_engine;
 
-	if (ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) {
+	if ((ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) ||
+	    (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)) {
 		switch (ring->me) {
 		case 1:
 			ref_and_mask = GPU_HDP_FLUSH_DONE__CP2_MASK << ring->pipe;
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 06/23] drm/amdgpu/gfx8: correct KIQ hdp flush

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> KIQ has some behavior as compute ring.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>

Should be squashed into patch 5.

Alex

> ---
>  drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
> index ae20cd9..a5a9a8e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
> @@ -6196,7 +6196,8 @@ static void gfx_v8_0_ring_emit_hdp_flush(struct amdgpu_ring *ring)
>  {
>         u32 ref_and_mask, reg_mem_engine;
>
> -       if (ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) {
> +       if ((ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) ||
> +           (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)) {
>                 switch (ring->me) {
>                 case 1:
>                         ref_and_mask = GPU_HDP_FLUSH_DONE__CP2_MASK << ring->pipe;
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 06/23] drm/amdgpu/gfx8: correct KIQ hdp flush

[Yu, Xiangliang]

> -----Original Message-----
> From: Alex Deucher [mailto:alexdeucher@gmail.com]
> Sent: Tuesday, December 20, 2016 7:14 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>
> Subject: Re: [PATCH 06/23] drm/amdgpu/gfx8: correct KIQ hdp flush
> 
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> wrote:
> > KIQ has some behavior as compute ring.
> >
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> 
> Should be squashed into patch 5.

Sure. 

> 
> Alex
> 
> > ---
> >  drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
> > b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
> > index ae20cd9..a5a9a8e 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
> > @@ -6196,7 +6196,8 @@ static void gfx_v8_0_ring_emit_hdp_flush(struct
> > amdgpu_ring *ring)  {
> >         u32 ref_and_mask, reg_mem_engine;
> >
> > -       if (ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) {
> > +       if ((ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) ||
> > +           (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)) {
> >                 switch (ring->me) {
> >                 case 1:
> >                         ref_and_mask = GPU_HDP_FLUSH_DONE__CP2_MASK <<
> > ring->pipe;
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 07/23] drm/amdgpu: create new directory for GPU virtualization

[Xiangliang Yu]

Create new directory for GPU virtualization and name it with
marketing name, so it will easy to mantainer all realted virtual
code and make it independent so that will not affect other parts.

And create new pointer and interface to connect virtualization
code with amdgpu core code.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h      |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   3 +-
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c    | 104 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    |  37 +++++++++++
 4 files changed, 145 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 25a3a2a..b0a8d8f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1511,6 +1511,8 @@ struct amdgpu_device {
 	spinlock_t			gtt_list_lock;
 	struct list_head                gtt_list;
 
+	/* GPU virtualization */
+	void				*priv_data;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
index 2c37a37..8ee70f8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
@@ -54,4 +54,5 @@ static inline bool is_virtual_machine(void)
 #endif
 }
 
-#endif
\ No newline at end of file
+int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev);
+#endif
diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
new file mode 100644
index 0000000..72469e2
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
@@ -0,0 +1,104 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Author: Xiangliang.Yu@amd.com
+ *
+ */
+#include "amd_mxgpu.h"
+
+int amd_xgpu_alloc(struct amdgpu_device *adev)
+{
+	struct amd_xgpu *xgpu = NULL;
+
+	if (!adev->priv_data) {
+		xgpu = kzalloc(sizeof(struct amd_xgpu), GFP_KERNEL);
+		if (!xgpu)
+			goto error;
+
+		adev->priv_data = xgpu;
+	}
+
+	xgpu->adev = adev;
+	mutex_init(&xgpu->lock);
+
+	return 0;
+
+error:
+	DRM_ERROR("xgpu: failed to allocate xgpu data.\n");
+	return -ENOMEM;
+}
+
+void amd_xgpu_free(struct amd_xgpu *xgpu)
+{
+	struct amdgpu_device *adev = xgpu->adev;
+
+	kfree(xgpu);
+	adev->priv_data = NULL;
+}
+
+static int amd_xgpu_init_ip_blocks(struct amdgpu_device *adev)
+{
+	int r = 0;
+
+	switch (adev->asic_type) {
+	case CHIP_TONGA:
+	case CHIP_FIJI:
+		/* add IP blocks*/
+		break;
+	default:
+		DRM_ERROR("Does not support this chip.\n");
+		r = -ENODEV;
+		break;
+	}
+
+	return r;
+}
+
+static void amd_xgpu_init_default_setting(struct amdgpu_device *adev)
+{
+	/* virtual display setting */
+	adev->mode_info.num_crtc = 1;
+	/* don't need dc */
+	amdgpu_dc = 0;
+	/* disable dpm */
+	amdgpu_dpm = 0;
+}
+
+/*
+ * tell amdgpu IP block info of VF
+ */
+int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev)
+{
+	int r = 0;
+
+	if (!is_virtual_machine()) {
+		DRM_ERROR("Host OS can't support VF device.\n");
+		return -EPERM;
+	}
+
+	r = amd_xgpu_init_ip_blocks(adev);
+	if (r)
+		return r;
+
+	amd_xgpu_init_default_setting(adev);
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
new file mode 100644
index 0000000..6ab13bc
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
@@ -0,0 +1,37 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Author: Xiangliang.Yu@amd.com
+ */
+#ifndef __AMDGPU_XGPU_H__
+#define __AMDGPU_XGPU_H__
+
+#include "amdgpu.h"
+
+struct amd_xgpu {
+	struct amdgpu_device	*adev;
+	struct mutex		lock;
+	u32			reg_val_offs;
+};
+
+extern int amd_xgpu_alloc(struct amdgpu_device *adev);
+extern void amd_xgpu_free(struct amd_xgpu *xgpu);
+#endif
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

转发: [PATCH 07/23] drm/amdgpu: create new directory for GPU virtualization

[Liu, Monk]

[-- Attachment #1.1: Type: text/plain, Size: 8867 bytes --]

Forward to amd-gfx.

发件人: Yu, Xiangliang
发送时间: Monday, December 19, 2016 1:49 PM
收件人: Liu, Monk; dl.SRDC_SW_GPUVirtualization
主题: RE: [PATCH 07/23] drm/amdgpu: create new directory for GPU virtualization

Make sense.


Thanks!
Xiangliang Yu

From: Liu, Monk
Sent: Monday, December 19, 2016 1:28 PM
To: Yu, Xiangliang <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>>; dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com<mailto:dl.SRDC_SW_GPUVirtualization@amd.com>>
Subject: 答复: [PATCH 07/23] drm/amdgpu: create new directory for GPU virtualization


NAK, there is not need to do that





         spinlock_t                      gtt_list_lock;
         struct list_head                gtt_list;

+       /* GPU virtualization */
+       void                            *priv_data;
 };



just do like this:



         spinlock_t                      gtt_list_lock;
         struct list_head                gtt_list;

+       /* GPU virtualization */
+       struct amdgpu_virtualization virtualization;
 };



don't use needless pointer in amdgpu_device structures, which brings more kmalloc/kfree cost on,



you can just put "struct amdgpu_virtualization" structure definition in amdgpu_virt.h





________________________________
发件人: Yu, Xiangliang
发送时间: 2016年12月18日 9:48:30
收件人: dl.SRDC_SW_GPUVirtualization
主题: FW: [PATCH 07/23] drm/amdgpu: create new directory for GPU virtualization



-----Original Message-----
From: Xiangliang Yu [mailto:Xiangliang.Yu@amd.com]
Sent: Sunday, December 18, 2016 12:16 AM
To: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com<mailto:dl.SRDC_SW_GPUVirtualization@amd.com>>
Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>>
Subject: [PATCH 07/23] drm/amdgpu: create new directory for GPU virtualization

Create new directory for GPU virtualization and name it with marketing name, so it will easy to mantainer all realted virtual code and make it independent so that will not affect other parts.

And create new pointer and interface to connect virtualization code with amdgpu core code.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h      |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   3 +-
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c    | 104 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    |  37 +++++++++++
 4 files changed, 145 insertions(+), 1 deletion(-)  create mode 100644 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 25a3a2a..b0a8d8f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1511,6 +1511,8 @@ struct amdgpu_device {
         spinlock_t                      gtt_list_lock;
         struct list_head                gtt_list;

+       /* GPU virtualization */
+       void                            *priv_data;
 };

 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
index 2c37a37..8ee70f8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
@@ -54,4 +54,5 @@ static inline bool is_virtual_machine(void)  #endif  }

-#endif
\ No newline at end of file
+int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev); #endif
diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
new file mode 100644
index 0000000..72469e2
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
@@ -0,0 +1,104 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person
+obtaining a
+ * copy of this software and associated documentation files (the
+"Software"),
+ * to deal in the Software without restriction, including without
+limitation
+ * the rights to use, copy, modify, merge, publish, distribute,
+sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom
+the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT
+SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
+DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
+OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Author: Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>
+ *
+ */
+#include "amd_mxgpu.h"
+
+int amd_xgpu_alloc(struct amdgpu_device *adev) {
+       struct amd_xgpu *xgpu = NULL;
+
+       if (!adev->priv_data) {
+               xgpu = kzalloc(sizeof(struct amd_xgpu), GFP_KERNEL);
+               if (!xgpu)
+                       goto error;
+
+               adev->priv_data = xgpu;
+       }
+
+       xgpu->adev = adev;
+       mutex_init(&xgpu->lock);
+
+       return 0;
+
+error:
+       DRM_ERROR("xgpu: failed to allocate xgpu data.\n");
+       return -ENOMEM;
+}
+
+void amd_xgpu_free(struct amd_xgpu *xgpu) {
+       struct amdgpu_device *adev = xgpu->adev;
+
+       kfree(xgpu);
+       adev->priv_data = NULL;
+}
+
+static int amd_xgpu_init_ip_blocks(struct amdgpu_device *adev) {
+       int r = 0;
+
+       switch (adev->asic_type) {
+       case CHIP_TONGA:
+       case CHIP_FIJI:
+               /* add IP blocks*/
+               break;
+       default:
+               DRM_ERROR("Does not support this chip.\n");
+               r = -ENODEV;
+               break;
+       }
+
+       return r;
+}
+
+static void amd_xgpu_init_default_setting(struct amdgpu_device *adev) {
+       /* virtual display setting */
+       adev->mode_info.num_crtc = 1;
+       /* don't need dc */
+       amdgpu_dc = 0;
+       /* disable dpm */
+       amdgpu_dpm = 0;
+}
+
+/*
+ * tell amdgpu IP block info of VF
+ */
+int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev) {
+       int r = 0;
+
+       if (!is_virtual_machine()) {
+               DRM_ERROR("Host OS can't support VF device.\n");
+               return -EPERM;
+       }
+
+       r = amd_xgpu_init_ip_blocks(adev);
+       if (r)
+               return r;
+
+       amd_xgpu_init_default_setting(adev);
+
+       return 0;
+}
diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
new file mode 100644
index 0000000..6ab13bc
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
@@ -0,0 +1,37 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person
+obtaining a
+ * copy of this software and associated documentation files (the
+"Software"),
+ * to deal in the Software without restriction, including without
+limitation
+ * the rights to use, copy, modify, merge, publish, distribute,
+sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom
+the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT
+SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
+DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
+OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Author: Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>
+ */
+#ifndef __AMDGPU_XGPU_H__
+#define __AMDGPU_XGPU_H__
+
+#include "amdgpu.h"
+
+struct amd_xgpu {
+       struct amdgpu_device    *adev;
+       struct mutex            lock;
+       u32                     reg_val_offs;
+};
+
+extern int amd_xgpu_alloc(struct amdgpu_device *adev); extern void
+amd_xgpu_free(struct amd_xgpu *xgpu); #endif
--
2.7.4

[-- Attachment #1.2: Type: text/html, Size: 24534 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 08/23] drm/amdgpu: add new flag for virtual function

[Xiangliang Yu]

Add new flag to identify chips that is support virtualization.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/include/amd_shared.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/amd/include/amd_shared.h b/drivers/gpu/drm/amd/include/amd_shared.h
index c02469a..9e555c7 100644
--- a/drivers/gpu/drm/amd/include/amd_shared.h
+++ b/drivers/gpu/drm/amd/include/amd_shared.h
@@ -59,6 +59,7 @@ enum amd_chip_flags {
 	AMD_IS_APU      = 0x00020000UL,
 	AMD_IS_PX       = 0x00040000UL,
 	AMD_EXP_HW_SUPPORT = 0x00080000UL,
+	AMD_IS_VF	= 0x00100000UL,
 };
 
 enum amd_ip_block_type {
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

[Xiangliang Yu]

According to chip device id to set VF flag, and call virtual
interface to setup all realted IP blocks.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 4 ++--
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index c4075b7..ab8c8bb5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -1285,7 +1285,10 @@ static int amdgpu_early_init(struct amdgpu_device *adev)
 		else
 			adev->family = AMDGPU_FAMILY_VI;
 
-		r = vi_set_ip_blocks(adev);
+		if (adev->flags & AMD_IS_VF)
+			r = amd_xgpu_set_ip_blocks(adev);
+		else
+			r = vi_set_ip_blocks(adev);
 		if (r)
 			return r;
 		break;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 93c4704..5a18111 100755
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -385,13 +385,13 @@ static const struct pci_device_id pciidlist[] = {
 	{0x1002, 0x6928, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
 	{0x1002, 0x6929, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
 	{0x1002, 0x692B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
-	{0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
+	{0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA | AMD_IS_VF},
 	{0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
 	{0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
 	{0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
 	/* fiji */
 	{0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
-	{0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
+	{0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI | AMD_IS_VF},
 	/* carrizo */
 	{0x1002, 0x9870, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_CARRIZO|AMD_IS_APU},
 	{0x1002, 0x9874, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_CARRIZO|AMD_IS_APU},
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

答复: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

[Liu, Monk]

[-- Attachment #1.1: Type: text/plain, Size: 3270 bytes --]

-               r = vi_set_ip_blocks(adev);

                detect_sriov(adev);

                if (amdgpu_sriov_vf(adev))

+                      adev->flags |= AMD_IS_VF;

                 if (adev->flags & AMD_IS_VF)

+                       r = amd_xgpu_set_ip_blocks(adev);
+               else
+                       r = vi_set_ip_blocks(adev);
                 if (r)
                         return r;

________________________________
发件人: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> 代表 Xiangliang Yu <Xiangliang.Yu@amd.com>
发送时间: 2016年12月18日 0:16:31
收件人: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization
抄送: Yu, Xiangliang
主题: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

According to chip device id to set VF flag, and call virtual
interface to setup all realted IP blocks.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 4 ++--
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index c4075b7..ab8c8bb5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -1285,7 +1285,10 @@ static int amdgpu_early_init(struct amdgpu_device *adev)
                 else
                         adev->family = AMDGPU_FAMILY_VI;

-               r = vi_set_ip_blocks(adev);
+               if (adev->flags & AMD_IS_VF)
+                       r = amd_xgpu_set_ip_blocks(adev);
+               else
+                       r = vi_set_ip_blocks(adev);
                 if (r)
                         return r;
                 break;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 93c4704..5a18111 100755
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -385,13 +385,13 @@ static const struct pci_device_id pciidlist[] = {
         {0x1002, 0x6928, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
         {0x1002, 0x6929, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
         {0x1002, 0x692B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
-       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
+       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA | AMD_IS_VF},
         {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
         {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
         {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
         /* fiji */
         {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
-       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
+       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI | AMD_IS_VF},
         /* carrizo */
         {0x1002, 0x9870, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_CARRIZO|AMD_IS_APU},
         {0x1002, 0x9874, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_CARRIZO|AMD_IS_APU},
--
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 7538 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> According to chip device id to set VF flag, and call virtual
> interface to setup all realted IP blocks.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 4 ++--
>  2 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index c4075b7..ab8c8bb5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -1285,7 +1285,10 @@ static int amdgpu_early_init(struct amdgpu_device *adev)
>                 else
>                         adev->family = AMDGPU_FAMILY_VI;
>
> -               r = vi_set_ip_blocks(adev);
> +               if (adev->flags & AMD_IS_VF)
> +                       r = amd_xgpu_set_ip_blocks(adev);

As far as I can see there's no need for a special
amd_xgpu_set_ip_blocks() function.  Just handle the VF case directly
in vi_set_ip_blocks() and avoid all the extra indirection.

Alex

> +               else
> +                       r = vi_set_ip_blocks(adev);
>                 if (r)
>                         return r;
>                 break;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 93c4704..5a18111 100755
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -385,13 +385,13 @@ static const struct pci_device_id pciidlist[] = {
>         {0x1002, 0x6928, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         {0x1002, 0x6929, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         {0x1002, 0x692B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> -       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> +       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA | AMD_IS_VF},
>         {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         /* fiji */
>         {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> -       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> +       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI | AMD_IS_VF},
>         /* carrizo */
>         {0x1002, 0x9870, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_CARRIZO|AMD_IS_APU},
>         {0x1002, 0x9874, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_CARRIZO|AMD_IS_APU},
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

答复: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

[Liu, Monk]

[-- Attachment #1.1: Type: text/plain, Size: 3625 bytes --]

agree with Alex on that, the patch looks overhead on the ip blocks cherry-picking logic, we could keep it as simple as original style,


as long as we know current device is a VF device, we shall know how many ip blocks need be cherry-picked and assembled together


BR Monk

________________________________
发件人: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> 代表 Alex Deucher <alexdeucher@gmail.com>
发送时间: 2016年12月20日 7:17:26
收件人: Yu, Xiangliang
抄送: dl.SRDC_SW_GPUVirtualization; amd-gfx list
主题: Re: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> According to chip device id to set VF flag, and call virtual
> interface to setup all realted IP blocks.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 4 ++--
>  2 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index c4075b7..ab8c8bb5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -1285,7 +1285,10 @@ static int amdgpu_early_init(struct amdgpu_device *adev)
>                 else
>                         adev->family = AMDGPU_FAMILY_VI;
>
> -               r = vi_set_ip_blocks(adev);
> +               if (adev->flags & AMD_IS_VF)
> +                       r = amd_xgpu_set_ip_blocks(adev);

As far as I can see there's no need for a special
amd_xgpu_set_ip_blocks() function.  Just handle the VF case directly
in vi_set_ip_blocks() and avoid all the extra indirection.

Alex

> +               else
> +                       r = vi_set_ip_blocks(adev);
>                 if (r)
>                         return r;
>                 break;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 93c4704..5a18111 100755
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -385,13 +385,13 @@ static const struct pci_device_id pciidlist[] = {
>         {0x1002, 0x6928, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         {0x1002, 0x6929, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         {0x1002, 0x692B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> -       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> +       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA | AMD_IS_VF},
>         {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>         /* fiji */
>         {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> -       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> +       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI | AMD_IS_VF},
>         /* carrizo */
>         {0x1002, 0x9870, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_CARRIZO|AMD_IS_APU},
>         {0x1002, 0x9874, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_CARRIZO|AMD_IS_APU},
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 6941 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

[Yu, Xiangliang]

> -----Original Message-----
> From: Alex Deucher [mailto:alexdeucher@gmail.com]
> Sent: Tuesday, December 20, 2016 7:17 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>
> Subject: Re: [PATCH 09/23] drm/amdgpu: enable virtualization feature for
> FIJI/TONGA
> 
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> wrote:
> > According to chip device id to set VF flag, and call virtual interface
> > to setup all realted IP blocks.
> >
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++++-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 4 ++--
> >  2 files changed, 6 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index c4075b7..ab8c8bb5 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -1285,7 +1285,10 @@ static int amdgpu_early_init(struct
> amdgpu_device *adev)
> >                 else
> >                         adev->family = AMDGPU_FAMILY_VI;
> >
> > -               r = vi_set_ip_blocks(adev);
> > +               if (adev->flags & AMD_IS_VF)
> > +                       r = amd_xgpu_set_ip_blocks(adev);
> 
> As far as I can see there's no need for a special
> amd_xgpu_set_ip_blocks() function.  Just handle the VF case directly in
> vi_set_ip_blocks() and avoid all the extra indirection.

My idea is that all virtualization chip share one common interface as a special chip family.  It will bring some benefits: 
1. Logic code is very clear;
2. Avoid to scatter virtualization code throughout all amdgpu components;
3. Easy to support next virtualization chip without change amdgpu code;

> Alex
> 
> > +               else
> > +                       r = vi_set_ip_blocks(adev);
> >                 if (r)
> >                         return r;
> >                 break;
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > index 93c4704..5a18111 100755
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > @@ -385,13 +385,13 @@ static const struct pci_device_id pciidlist[] = {
> >         {0x1002, 0x6928, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> >         {0x1002, 0x6929, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> >         {0x1002, 0x692B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > -       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > +       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA |
> > + AMD_IS_VF},
> >         {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> >         {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> >         {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> >         /* fiji */
> >         {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> > -       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> > +       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI |
> > + AMD_IS_VF},
> >         /* carrizo */
> >         {0x1002, 0x9870, PCI_ANY_ID, PCI_ANY_ID, 0, 0,
> CHIP_CARRIZO|AMD_IS_APU},
> >         {0x1002, 0x9874, PCI_ANY_ID, PCI_ANY_ID, 0, 0,
> > CHIP_CARRIZO|AMD_IS_APU},
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

[Deucher, Alexander]

> -----Original Message-----
> From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf
> Of Yu, Xiangliang
> Sent: Tuesday, December 20, 2016 12:41 AM
> To: Alex Deucher
> Cc: dl.SRDC_SW_GPUVirtualization; amd-gfx list
> Subject: RE: [PATCH 09/23] drm/amdgpu: enable virtualization feature for
> FIJI/TONGA
> 
> 
> > -----Original Message-----
> > From: Alex Deucher [mailto:alexdeucher@gmail.com]
> > Sent: Tuesday, December 20, 2016 7:17 AM
> > To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> > Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> > dl.SRDC_SW_GPUVirtualization
> <dl.SRDC_SW_GPUVirtualization@amd.com>
> > Subject: Re: [PATCH 09/23] drm/amdgpu: enable virtualization feature for
> > FIJI/TONGA
> >
> > On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> > wrote:
> > > According to chip device id to set VF flag, and call virtual interface
> > > to setup all realted IP blocks.
> > >
> > > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > > ---
> > >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++++-
> > >  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 4 ++--
> > >  2 files changed, 6 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > > index c4075b7..ab8c8bb5 100644
> > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > > @@ -1285,7 +1285,10 @@ static int amdgpu_early_init(struct
> > amdgpu_device *adev)
> > >                 else
> > >                         adev->family = AMDGPU_FAMILY_VI;
> > >
> > > -               r = vi_set_ip_blocks(adev);
> > > +               if (adev->flags & AMD_IS_VF)
> > > +                       r = amd_xgpu_set_ip_blocks(adev);
> >
> > As far as I can see there's no need for a special
> > amd_xgpu_set_ip_blocks() function.  Just handle the VF case directly in
> > vi_set_ip_blocks() and avoid all the extra indirection.
> 
> My idea is that all virtualization chip share one common interface as a special
> chip family.  It will bring some benefits:
> 1. Logic code is very clear;
> 2. Avoid to scatter virtualization code throughout all amdgpu components;
> 3. Easy to support next virtualization chip without change amdgpu code;
> 

I don't mind having a separate IP module for special VF related setup, but I think the differences in the list of IP modules is small enough that it doesn't warrant all of the redirection.  Basically just:

if (VF) {
    add_ip_module(A);
    add_ip_module(X);
    add_ip_module(C);
} else {
    add_ip_module(A);
    add_ip_module(B);
    add_ip_module(C);
    add_ip_module(D);
}

That way it's obvious in one place which modules are present in the VF and bare metal cases without having to trace through a bunch of indirection.  It also makes it easier to update the lists if we ever rework the ip module interface or add a new IP module or something like that.

Alex

> > Alex
> >
> > > +               else
> > > +                       r = vi_set_ip_blocks(adev);
> > >                 if (r)
> > >                         return r;
> > >                 break;
> > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > index 93c4704..5a18111 100755
> > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > @@ -385,13 +385,13 @@ static const struct pci_device_id pciidlist[] = {
> > >         {0x1002, 0x6928, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > >         {0x1002, 0x6929, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > >         {0x1002, 0x692B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > -       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > +       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA |
> > > + AMD_IS_VF},
> > >         {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > >         {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > >         {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > >         /* fiji */
> > >         {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> > > -       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> > > +       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI |
> > > + AMD_IS_VF},
> > >         /* carrizo */
> > >         {0x1002, 0x9870, PCI_ANY_ID, PCI_ANY_ID, 0, 0,
> > CHIP_CARRIZO|AMD_IS_APU},
> > >         {0x1002, 0x9874, PCI_ANY_ID, PCI_ANY_ID, 0, 0,
> > > CHIP_CARRIZO|AMD_IS_APU},
> > > --
> > > 2.7.4
> > >
> > > _______________________________________________
> > > amd-gfx mailing list
> > > amd-gfx@lists.freedesktop.org
> > > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

[Yu, Xiangliang]

> -----Original Message-----
> From: Deucher, Alexander
> Sent: Tuesday, December 20, 2016 11:53 PM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>; Alex Deucher
> <alexdeucher@gmail.com>
> Cc: dl.SRDC_SW_GPUVirtualization
> <dl.SRDC_SW_GPUVirtualization@amd.com>; amd-gfx list <amd-
> gfx@lists.freedesktop.org>
> Subject: RE: [PATCH 09/23] drm/amdgpu: enable virtualization feature for
> FIJI/TONGA
> 
> > -----Original Message-----
> > From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf
> > Of Yu, Xiangliang
> > Sent: Tuesday, December 20, 2016 12:41 AM
> > To: Alex Deucher
> > Cc: dl.SRDC_SW_GPUVirtualization; amd-gfx list
> > Subject: RE: [PATCH 09/23] drm/amdgpu: enable virtualization feature
> > for FIJI/TONGA
> >
> >
> > > -----Original Message-----
> > > From: Alex Deucher [mailto:alexdeucher@gmail.com]
> > > Sent: Tuesday, December 20, 2016 7:17 AM
> > > To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> > > Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> > > dl.SRDC_SW_GPUVirtualization
> > <dl.SRDC_SW_GPUVirtualization@amd.com>
> > > Subject: Re: [PATCH 09/23] drm/amdgpu: enable virtualization feature
> > > for FIJI/TONGA
> > >
> > > On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu
> > > <Xiangliang.Yu@amd.com>
> > > wrote:
> > > > According to chip device id to set VF flag, and call virtual
> > > > interface to setup all realted IP blocks.
> > > >
> > > > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > > > ---
> > > >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++++-
> > > >  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 4 ++--
> > > >  2 files changed, 6 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > > > index c4075b7..ab8c8bb5 100644
> > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > > > @@ -1285,7 +1285,10 @@ static int amdgpu_early_init(struct
> > > amdgpu_device *adev)
> > > >                 else
> > > >                         adev->family = AMDGPU_FAMILY_VI;
> > > >
> > > > -               r = vi_set_ip_blocks(adev);
> > > > +               if (adev->flags & AMD_IS_VF)
> > > > +                       r = amd_xgpu_set_ip_blocks(adev);
> > >
> > > As far as I can see there's no need for a special
> > > amd_xgpu_set_ip_blocks() function.  Just handle the VF case directly
> > > in
> > > vi_set_ip_blocks() and avoid all the extra indirection.
> >
> > My idea is that all virtualization chip share one common interface as
> > a special chip family.  It will bring some benefits:
> > 1. Logic code is very clear;
> > 2. Avoid to scatter virtualization code throughout all amdgpu
> > components; 3. Easy to support next virtualization chip without change
> > amdgpu code;
> >
> 
> I don't mind having a separate IP module for special VF related setup, but I
> think the differences in the list of IP modules is small enough that it doesn't
> warrant all of the redirection.  Basically just:
> 
> if (VF) {
>     add_ip_module(A);
>     add_ip_module(X);
>     add_ip_module(C);
> } else {
>     add_ip_module(A);
>     add_ip_module(B);
>     add_ip_module(C);
>     add_ip_module(D);
> }
> 
> That way it's obvious in one place which modules are present in the VF and
> bare metal cases without having to trace through a bunch of indirection.  It
> also makes it easier to update the lists if we ever rework the ip module
> interface or add a new IP module or something like that.

My point is we want to centrally manage all virtualization, like as power play component does. 
For you, the code is not big difference, but for us, easy to implement new features, support new chips, and add new setting for virtualization.
Otherwise, there are lot of virtualization check in amdgpu and bring lot of effects to maintain virtualization code. 
You know, virtualization is very big feature for graphics.

Please think about from what I'm standing.

> > >
> > > > +               else
> > > > +                       r = vi_set_ip_blocks(adev);
> > > >                 if (r)
> > > >                         return r;
> > > >                 break;
> > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > > index 93c4704..5a18111 100755
> > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > > @@ -385,13 +385,13 @@ static const struct pci_device_id pciidlist[] = {
> > > >         {0x1002, 0x6928, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > >         {0x1002, 0x6929, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > >         {0x1002, 0x692B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > > -       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > > +       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA
> > > > + | AMD_IS_VF},
> > > >         {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > >         {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > >         {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
> > > >         /* fiji */
> > > >         {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> > > > -       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
> > > > +       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI |
> > > > + AMD_IS_VF},
> > > >         /* carrizo */
> > > >         {0x1002, 0x9870, PCI_ANY_ID, PCI_ANY_ID, 0, 0,
> > > CHIP_CARRIZO|AMD_IS_APU},
> > > >         {0x1002, 0x9874, PCI_ANY_ID, PCI_ANY_ID, 0, 0,
> > > > CHIP_CARRIZO|AMD_IS_APU},
> > > > --
> > > > 2.7.4
> > > >
> > > > _______________________________________________
> > > > amd-gfx mailing list
> > > > amd-gfx@lists.freedesktop.org
> > > > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 09/23] drm/amdgpu: enable virtualization feature for FIJI/TONGA

[Christian König]

Am 21.12.2016 um 02:59 schrieb Yu, Xiangliang:
>> -----Original Message-----
>> From: Deucher, Alexander
>> Sent: Tuesday, December 20, 2016 11:53 PM
>> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>; Alex Deucher
>> <alexdeucher@gmail.com>
>> Cc: dl.SRDC_SW_GPUVirtualization
>> <dl.SRDC_SW_GPUVirtualization@amd.com>; amd-gfx list <amd-
>> gfx@lists.freedesktop.org>
>> Subject: RE: [PATCH 09/23] drm/amdgpu: enable virtualization feature for
>> FIJI/TONGA
>>
>>> -----Original Message-----
>>> From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf
>>> Of Yu, Xiangliang
>>> Sent: Tuesday, December 20, 2016 12:41 AM
>>> To: Alex Deucher
>>> Cc: dl.SRDC_SW_GPUVirtualization; amd-gfx list
>>> Subject: RE: [PATCH 09/23] drm/amdgpu: enable virtualization feature
>>> for FIJI/TONGA
>>>
>>>
>>>> -----Original Message-----
>>>> From: Alex Deucher [mailto:alexdeucher@gmail.com]
>>>> Sent: Tuesday, December 20, 2016 7:17 AM
>>>> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
>>>> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
>>>> dl.SRDC_SW_GPUVirtualization
>>> <dl.SRDC_SW_GPUVirtualization@amd.com>
>>>> Subject: Re: [PATCH 09/23] drm/amdgpu: enable virtualization feature
>>>> for FIJI/TONGA
>>>>
>>>> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu
>>>> <Xiangliang.Yu@amd.com>
>>>> wrote:
>>>>> According to chip device id to set VF flag, and call virtual
>>>>> interface to setup all realted IP blocks.
>>>>>
>>>>> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
>>>>> ---
>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++++-
>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 4 ++--
>>>>>   2 files changed, 6 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>>> index c4075b7..ab8c8bb5 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>>> @@ -1285,7 +1285,10 @@ static int amdgpu_early_init(struct
>>>> amdgpu_device *adev)
>>>>>                  else
>>>>>                          adev->family = AMDGPU_FAMILY_VI;
>>>>>
>>>>> -               r = vi_set_ip_blocks(adev);
>>>>> +               if (adev->flags & AMD_IS_VF)
>>>>> +                       r = amd_xgpu_set_ip_blocks(adev);
>>>> As far as I can see there's no need for a special
>>>> amd_xgpu_set_ip_blocks() function.  Just handle the VF case directly
>>>> in
>>>> vi_set_ip_blocks() and avoid all the extra indirection.
>>> My idea is that all virtualization chip share one common interface as
>>> a special chip family.  It will bring some benefits:
>>> 1. Logic code is very clear;
>>> 2. Avoid to scatter virtualization code throughout all amdgpu
>>> components; 3. Easy to support next virtualization chip without change
>>> amdgpu code;
>>>
>> I don't mind having a separate IP module for special VF related setup, but I
>> think the differences in the list of IP modules is small enough that it doesn't
>> warrant all of the redirection.  Basically just:
>>
>> if (VF) {
>>      add_ip_module(A);
>>      add_ip_module(X);
>>      add_ip_module(C);
>> } else {
>>      add_ip_module(A);
>>      add_ip_module(B);
>>      add_ip_module(C);
>>      add_ip_module(D);
>> }
>>
>> That way it's obvious in one place which modules are present in the VF and
>> bare metal cases without having to trace through a bunch of indirection.  It
>> also makes it easier to update the lists if we ever rework the ip module
>> interface or add a new IP module or something like that.
> My point is we want to centrally manage all virtualization, like as power play component does.
> For you, the code is not big difference, but for us, easy to implement new features, support new chips, and add new setting for virtualization.
> Otherwise, there are lot of virtualization check in amdgpu and bring lot of effects to maintain virtualization code.
> You know, virtualization is very big feature for graphics.
>
> Please think about from what I'm standing.

Sorry, but I have to agree with Alex here. Powerplay is separate because 
it affects multiple components at the same time.

Virtualization doesn't justify at all having it a separate module like 
Powerplay. It's just an addition to the existing IP modules, not 
something completely new.

Additional to that it is likely that virtualization will change over 
time, so the separation like Alex suggest makes a lot of sense.

Regards,
Christian.

>
>>>>> +               else
>>>>> +                       r = vi_set_ip_blocks(adev);
>>>>>                  if (r)
>>>>>                          return r;
>>>>>                  break;
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>>>>> index 93c4704..5a18111 100755
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>>>>> @@ -385,13 +385,13 @@ static const struct pci_device_id pciidlist[] = {
>>>>>          {0x1002, 0x6928, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>>>>>          {0x1002, 0x6929, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>>>>>          {0x1002, 0x692B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>>>>> -       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>>>>> +       {0x1002, 0x692F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA
>>>>> + | AMD_IS_VF},
>>>>>          {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>>>>>          {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>>>>>          {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
>>>>>          /* fiji */
>>>>>          {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
>>>>> -       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
>>>>> +       {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI |
>>>>> + AMD_IS_VF},
>>>>>          /* carrizo */
>>>>>          {0x1002, 0x9870, PCI_ANY_ID, PCI_ANY_ID, 0, 0,
>>>> CHIP_CARRIZO|AMD_IS_APU},
>>>>>          {0x1002, 0x9874, PCI_ANY_ID, PCI_ANY_ID, 0, 0,
>>>>> CHIP_CARRIZO|AMD_IS_APU},
>>>>> --
>>>>> 2.7.4
>>>>>
>>>>> _______________________________________________
>>>>> amd-gfx mailing list
>>>>> amd-gfx@lists.freedesktop.org
>>>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>>> _______________________________________________
>>> amd-gfx mailing list
>>> amd-gfx@lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx


_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 10/23] drm/amdgpu: export gem va update interface

[Xiangliang Yu]

Context Save Area feature need to update va info so that hardware
can save state into correct area. So export it.

Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h     | 3 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 6 +++---
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index b0a8d8f..ea7f8a1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -412,6 +412,9 @@ struct amdgpu_bo {
 };
 #define gem_to_amdgpu_bo(gobj) container_of((gobj), struct amdgpu_bo, gem_base)
 
+void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
+			     struct amdgpu_bo_va *bo_va,
+			     uint32_t operation);
 void amdgpu_gem_object_free(struct drm_gem_object *obj);
 int amdgpu_gem_object_open(struct drm_gem_object *obj,
 				struct drm_file *file_priv);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index c0820de..f74427c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -490,9 +490,9 @@ static int amdgpu_gem_va_check(void *param, struct amdgpu_bo *bo)
  * Update the bo_va directly after setting it's address. Errors are not
  * vital here, so they are not reported back to userspace.
  */
-static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
-				    struct amdgpu_bo_va *bo_va,
-				    uint32_t operation)
+void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
+			     struct amdgpu_bo_va *bo_va,
+			     uint32_t operation)
 {
 	struct ttm_validate_buffer tv, *entry;
 	struct amdgpu_bo_list_entry vm_pd;
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 11/23] drm/amdgpu: implement context save area(CSA) feature

[Xiangliang Yu]

CSA is need by world switch. This patch implement CSA feature and
bind it to each VM, so hardware can save the state into the area
and restore it when running again.

Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  14 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |   8 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   |   4 +
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    |  12 ++
 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c    | 209 +++++++++++++++++++++++++++++++
 5 files changed, 247 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
index 8ee70f8..dff1248 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
@@ -24,6 +24,8 @@
 #ifndef AMDGPU_VIRT_H
 #define AMDGPU_VIRT_H
 
+struct amdgpu_vm;
+
 #define AMDGPU_SRIOV_CAPS_SRIOV_VBIOS  (1 << 0) /* vBIOS is sr-iov ready */
 #define AMDGPU_SRIOV_CAPS_ENABLE_IOV   (1 << 1) /* sr-iov is enabled on this GPU */
 #define AMDGPU_SRIOV_CAPS_IS_VF        (1 << 2) /* this GPU is a virtual function */
@@ -33,6 +35,14 @@ struct amdgpu_virtualization {
 	uint32_t virtual_caps;
 };
 
+struct amdgpu_csa {
+	struct amdgpu_bo_va		*va;
+	struct ttm_validate_buffer	tv;
+	uint64_t			reserved_top;
+	uint64_t			csa_addr;
+	uint64_t			gds_addr;
+};
+
 #define amdgpu_sriov_enabled(adev) \
 ((adev)->virtualization.virtual_caps & AMDGPU_SRIOV_CAPS_ENABLE_IOV)
 
@@ -55,4 +65,8 @@ static inline bool is_virtual_machine(void)
 }
 
 int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev);
+
+/* Context Save Area functions */
+int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
+void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
 #endif
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index d05546e..98540d9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1539,6 +1539,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	pd_size = amdgpu_vm_directory_size(adev);
 	pd_entries = amdgpu_vm_num_pdes(adev);
 
+	vm->csa.reserved_top = AMDGPU_VA_RESERVED_SIZE;
+
 	/* allocate page table array */
 	vm->page_tables = drm_calloc_large(pd_entries, sizeof(struct amdgpu_vm_pt));
 	if (vm->page_tables == NULL) {
@@ -1576,6 +1578,10 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	vm->last_eviction_counter = atomic64_read(&adev->num_evictions);
 	amdgpu_bo_unreserve(vm->page_directory);
 
+	r = amdgpu_vm_map_csa(adev, vm);
+	if (r)
+		goto error_free_page_directory;
+
 	return 0;
 
 error_free_page_directory:
@@ -1606,6 +1612,8 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	struct amdgpu_bo_va_mapping *mapping, *tmp;
 	int i;
 
+	amdgpu_vm_unmap_csa(adev, vm);
+
 	amd_sched_entity_fini(vm->entity.sched, &vm->entity);
 
 	if (!RB_EMPTY_ROOT(&vm->va)) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 42a629b..d90630a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -29,6 +29,7 @@
 #include "gpu_scheduler.h"
 #include "amdgpu_sync.h"
 #include "amdgpu_ring.h"
+#include "amdgpu_virt.h"
 
 struct amdgpu_bo_va;
 struct amdgpu_job;
@@ -109,6 +110,9 @@ struct amdgpu_vm {
 	/* Scheduler entity for page table updates */
 	struct amd_sched_entity	entity;
 
+	/* Context Save Area */
+	struct amdgpu_csa	csa;
+
 	/* client id */
 	u64                     client_id;
 };
diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
index 6ab13bc..a25e07f 100644
--- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
@@ -26,12 +26,24 @@
 
 #include "amdgpu.h"
 
+/* xgpu structures */
+struct amd_xgpu_csa {
+	struct amdgpu_bo	    *robj;
+	uint64_t		    gpu_addr;
+	uint64_t		    gds_addr;
+	int32_t			    size;
+};
+
 struct amd_xgpu {
 	struct amdgpu_device	*adev;
 	struct mutex		lock;
+	struct amd_xgpu_csa	sa;
 	u32			reg_val_offs;
 };
 
 extern int amd_xgpu_alloc(struct amdgpu_device *adev);
 extern void amd_xgpu_free(struct amd_xgpu *xgpu);
+
+extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);
+extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
 #endif
diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
new file mode 100644
index 0000000..246a747
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
@@ -0,0 +1,209 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Xiangliang.Yu@amd.com
+ *	    Monk.Liu@amd.com
+ *
+ */
+#include "amd_mxgpu.h"
+#include "vid.h"
+
+static int xgpu_init_csa(struct amdgpu_device *adev, struct amd_xgpu_csa *sa)
+{
+	int r, size;
+	void *ptr;
+
+	/* meta data (4k) + gds-gfx (4k)*/
+	size = PAGE_SIZE + adev->gds.mem.gfx_partition_size;
+
+	r = amdgpu_bo_create(adev, size, PAGE_SIZE, true,
+			     AMDGPU_GEM_DOMAIN_GTT,
+			     AMDGPU_GEM_CREATE_CPU_GTT_USWC,
+			     NULL, NULL, &sa->robj);
+	if (r) {
+		dev_err(adev->dev, "(%d) failed to allocate csa bo\n", r);
+		return r;
+	}
+
+	r = amdgpu_bo_reserve(sa->robj, true);
+	if (unlikely(r != 0))
+		goto error_free;
+
+	r = amdgpu_bo_pin(sa->robj, AMDGPU_GEM_DOMAIN_GTT, &sa->gpu_addr);
+	if (r)
+		goto error_unreserve;
+
+	r = amdgpu_bo_kmap(sa->robj, &ptr);
+	if (r)
+		goto error_unpin;
+
+	memset(ptr, 0, size);
+	amdgpu_bo_unreserve(sa->robj);
+
+	sa->size = size;
+	sa->gds_addr = sa->gpu_addr + PAGE_SIZE;
+
+	return 0;
+
+error_unpin:
+	amdgpu_bo_unpin(sa->robj);
+error_unreserve:
+	amdgpu_bo_unreserve(sa->robj);
+error_free:
+	amdgpu_bo_unref(&sa->robj);
+	return r;
+}
+
+int xgpu_allocate_csa(struct amd_xgpu *xgpu)
+{
+	struct amdgpu_device *adev = xgpu->adev;
+	struct amd_xgpu_csa *sa = &xgpu->sa;
+
+	return xgpu_init_csa(adev, sa);
+}
+
+void xgpu_destroy_csa(struct amd_xgpu_csa *sa)
+{
+	amdgpu_bo_reserve(sa->robj, true);
+	amdgpu_bo_unpin(sa->robj);
+	amdgpu_bo_unreserve(sa->robj);
+	amdgpu_bo_unref(&sa->robj);
+	sa->gpu_addr = 0;
+	sa->gds_addr = 0;
+}
+
+static int xgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
+			   struct amd_xgpu_csa *sa)
+{
+	int r;
+	uint64_t vaddr;
+	struct ww_acquire_ctx ticket;
+	struct list_head list, duplicates;
+	struct amdgpu_bo_list_entry pd;
+	struct amdgpu_bo_va *bo_va;
+
+	INIT_LIST_HEAD(&list);
+	INIT_LIST_HEAD(&duplicates);
+	INIT_LIST_HEAD(&vm->csa.tv.head);
+	vm->csa.tv.bo = &sa->robj->tbo;
+	vm->csa.tv.shared = true;
+
+	list_add(&vm->csa.tv.head, &list);
+	amdgpu_vm_get_pd_bo(vm, &list, &pd);
+
+	spin_lock(&vm->status_lock);
+	vm->csa.reserved_top -= sa->size;
+	vaddr = vm->csa.reserved_top;
+	spin_unlock(&vm->status_lock);
+
+	r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
+	if (r) {
+		DRM_ERROR("failed to reserve global CSA buffer(%d).\n", r);
+		return r;
+	}
+
+	bo_va = amdgpu_vm_bo_add(adev, vm, sa->robj);
+	if (!bo_va) {
+		DRM_ERROR("failed to create bo_va for global CSA buffer.\n");
+		return -ENOMEM;
+	}
+
+
+	r = amdgpu_vm_bo_map(adev, bo_va, vaddr, 0, sa->size,
+			     AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
+			     AMDGPU_PTE_EXECUTABLE);
+	if (r) {
+		DRM_ERROR("failed to do bo_map on global CSA buffer(%d).\n", r);
+		amdgpu_vm_bo_rmv(adev, bo_va);
+		ttm_eu_backoff_reservation(&ticket, &list);
+		kfree(bo_va);
+		return r;
+	}
+
+	ttm_eu_backoff_reservation(&ticket, &list);
+	amdgpu_gem_va_update_vm(adev, bo_va, AMDGPU_VA_OP_MAP);
+
+	vm->csa.va = bo_va;
+	vm->csa.csa_addr = vaddr;
+	vm->csa.gds_addr = vaddr + PAGE_SIZE;
+
+	return 0;
+}
+
+int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
+{
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+	struct amd_xgpu_csa *sa = NULL;
+
+	if (!xgpu)
+		return 0;
+
+	sa = &xgpu->sa;
+
+	return xgpu_vm_map_csa(adev, vm, sa);
+}
+
+static void xgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
+			      struct amd_xgpu_csa *sa)
+{
+	int r;
+	struct ww_acquire_ctx ticket;
+	struct list_head list, duplicates;
+	struct amdgpu_bo_list_entry pd;
+
+	if (!vm->csa.va)
+		return;
+
+	INIT_LIST_HEAD(&list);
+	INIT_LIST_HEAD(&duplicates);
+	list_add(&vm->csa.tv.head, &list);
+	amdgpu_vm_get_pd_bo(vm, &list, &pd);
+
+	r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
+	if (r) {
+		DRM_ERROR("failed to reserve global CSA buffer(%d).\n", r);
+		return;
+	}
+
+	amdgpu_vm_bo_rmv(adev, vm->csa.va);
+	/* maybe we don't need to do real clearing for the vm will die soon */
+	r = amdgpu_vm_clear_freed(adev, vm);
+	if (r) {
+		DRM_ERROR("failed to clear global CSA bo(%d).\n", r);
+		return;
+	}
+
+	ttm_eu_backoff_reservation(&ticket, &list);
+	vm->csa.va = NULL;
+	vm->csa.csa_addr = vm->csa.gds_addr = 0;
+}
+
+void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
+{
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+	struct amd_xgpu_csa *sa = NULL;
+
+	if (!xgpu)
+		return;
+
+	sa = &xgpu->sa;
+	xgpu_vm_unmap_csa(adev, vm, sa);
+}
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 11/23] drm/amdgpu: implement context save area(CSA) feature

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> CSA is need by world switch. This patch implement CSA feature and
> bind it to each VM, so hardware can save the state into the area
> and restore it when running again.
>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>

Isn't the CSA actually for preemption?  Preemption is useful outside
of the VF case as well so it should be untangled from the mxgpu code
so it can be utilized independently.

Alex

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  14 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |   8 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   |   4 +
>  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    |  12 ++
>  drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c    | 209 +++++++++++++++++++++++++++++++
>  5 files changed, 247 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index 8ee70f8..dff1248 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -24,6 +24,8 @@
>  #ifndef AMDGPU_VIRT_H
>  #define AMDGPU_VIRT_H
>
> +struct amdgpu_vm;
> +
>  #define AMDGPU_SRIOV_CAPS_SRIOV_VBIOS  (1 << 0) /* vBIOS is sr-iov ready */
>  #define AMDGPU_SRIOV_CAPS_ENABLE_IOV   (1 << 1) /* sr-iov is enabled on this GPU */
>  #define AMDGPU_SRIOV_CAPS_IS_VF        (1 << 2) /* this GPU is a virtual function */
> @@ -33,6 +35,14 @@ struct amdgpu_virtualization {
>         uint32_t virtual_caps;
>  };
>
> +struct amdgpu_csa {
> +       struct amdgpu_bo_va             *va;
> +       struct ttm_validate_buffer      tv;
> +       uint64_t                        reserved_top;
> +       uint64_t                        csa_addr;
> +       uint64_t                        gds_addr;
> +};
> +
>  #define amdgpu_sriov_enabled(adev) \
>  ((adev)->virtualization.virtual_caps & AMDGPU_SRIOV_CAPS_ENABLE_IOV)
>
> @@ -55,4 +65,8 @@ static inline bool is_virtual_machine(void)
>  }
>
>  int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev);
> +
> +/* Context Save Area functions */
> +int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
> +void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index d05546e..98540d9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -1539,6 +1539,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         pd_size = amdgpu_vm_directory_size(adev);
>         pd_entries = amdgpu_vm_num_pdes(adev);
>
> +       vm->csa.reserved_top = AMDGPU_VA_RESERVED_SIZE;
> +
>         /* allocate page table array */
>         vm->page_tables = drm_calloc_large(pd_entries, sizeof(struct amdgpu_vm_pt));
>         if (vm->page_tables == NULL) {
> @@ -1576,6 +1578,10 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         vm->last_eviction_counter = atomic64_read(&adev->num_evictions);
>         amdgpu_bo_unreserve(vm->page_directory);
>
> +       r = amdgpu_vm_map_csa(adev, vm);
> +       if (r)
> +               goto error_free_page_directory;
> +
>         return 0;
>
>  error_free_page_directory:
> @@ -1606,6 +1612,8 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         struct amdgpu_bo_va_mapping *mapping, *tmp;
>         int i;
>
> +       amdgpu_vm_unmap_csa(adev, vm);
> +
>         amd_sched_entity_fini(vm->entity.sched, &vm->entity);
>
>         if (!RB_EMPTY_ROOT(&vm->va)) {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index 42a629b..d90630a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -29,6 +29,7 @@
>  #include "gpu_scheduler.h"
>  #include "amdgpu_sync.h"
>  #include "amdgpu_ring.h"
> +#include "amdgpu_virt.h"
>
>  struct amdgpu_bo_va;
>  struct amdgpu_job;
> @@ -109,6 +110,9 @@ struct amdgpu_vm {
>         /* Scheduler entity for page table updates */
>         struct amd_sched_entity entity;
>
> +       /* Context Save Area */
> +       struct amdgpu_csa       csa;
> +
>         /* client id */
>         u64                     client_id;
>  };
> diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> index 6ab13bc..a25e07f 100644
> --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> @@ -26,12 +26,24 @@
>
>  #include "amdgpu.h"
>
> +/* xgpu structures */
> +struct amd_xgpu_csa {
> +       struct amdgpu_bo            *robj;
> +       uint64_t                    gpu_addr;
> +       uint64_t                    gds_addr;
> +       int32_t                     size;
> +};
> +
>  struct amd_xgpu {
>         struct amdgpu_device    *adev;
>         struct mutex            lock;
> +       struct amd_xgpu_csa     sa;
>         u32                     reg_val_offs;
>  };
>
>  extern int amd_xgpu_alloc(struct amdgpu_device *adev);
>  extern void amd_xgpu_free(struct amd_xgpu *xgpu);
> +
> +extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);
> +extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
>  #endif
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> new file mode 100644
> index 0000000..246a747
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> @@ -0,0 +1,209 @@
> +/*
> + * Copyright 2016 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + * Authors: Xiangliang.Yu@amd.com
> + *         Monk.Liu@amd.com
> + *
> + */
> +#include "amd_mxgpu.h"
> +#include "vid.h"
> +
> +static int xgpu_init_csa(struct amdgpu_device *adev, struct amd_xgpu_csa *sa)
> +{
> +       int r, size;
> +       void *ptr;
> +
> +       /* meta data (4k) + gds-gfx (4k)*/
> +       size = PAGE_SIZE + adev->gds.mem.gfx_partition_size;
> +
> +       r = amdgpu_bo_create(adev, size, PAGE_SIZE, true,
> +                            AMDGPU_GEM_DOMAIN_GTT,
> +                            AMDGPU_GEM_CREATE_CPU_GTT_USWC,
> +                            NULL, NULL, &sa->robj);
> +       if (r) {
> +               dev_err(adev->dev, "(%d) failed to allocate csa bo\n", r);
> +               return r;
> +       }
> +
> +       r = amdgpu_bo_reserve(sa->robj, true);
> +       if (unlikely(r != 0))
> +               goto error_free;
> +
> +       r = amdgpu_bo_pin(sa->robj, AMDGPU_GEM_DOMAIN_GTT, &sa->gpu_addr);
> +       if (r)
> +               goto error_unreserve;
> +
> +       r = amdgpu_bo_kmap(sa->robj, &ptr);
> +       if (r)
> +               goto error_unpin;
> +
> +       memset(ptr, 0, size);
> +       amdgpu_bo_unreserve(sa->robj);
> +
> +       sa->size = size;
> +       sa->gds_addr = sa->gpu_addr + PAGE_SIZE;
> +
> +       return 0;
> +
> +error_unpin:
> +       amdgpu_bo_unpin(sa->robj);
> +error_unreserve:
> +       amdgpu_bo_unreserve(sa->robj);
> +error_free:
> +       amdgpu_bo_unref(&sa->robj);
> +       return r;
> +}
> +
> +int xgpu_allocate_csa(struct amd_xgpu *xgpu)
> +{
> +       struct amdgpu_device *adev = xgpu->adev;
> +       struct amd_xgpu_csa *sa = &xgpu->sa;
> +
> +       return xgpu_init_csa(adev, sa);
> +}
> +
> +void xgpu_destroy_csa(struct amd_xgpu_csa *sa)
> +{
> +       amdgpu_bo_reserve(sa->robj, true);
> +       amdgpu_bo_unpin(sa->robj);
> +       amdgpu_bo_unreserve(sa->robj);
> +       amdgpu_bo_unref(&sa->robj);
> +       sa->gpu_addr = 0;
> +       sa->gds_addr = 0;
> +}
> +
> +static int xgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> +                          struct amd_xgpu_csa *sa)
> +{
> +       int r;
> +       uint64_t vaddr;
> +       struct ww_acquire_ctx ticket;
> +       struct list_head list, duplicates;
> +       struct amdgpu_bo_list_entry pd;
> +       struct amdgpu_bo_va *bo_va;
> +
> +       INIT_LIST_HEAD(&list);
> +       INIT_LIST_HEAD(&duplicates);
> +       INIT_LIST_HEAD(&vm->csa.tv.head);
> +       vm->csa.tv.bo = &sa->robj->tbo;
> +       vm->csa.tv.shared = true;
> +
> +       list_add(&vm->csa.tv.head, &list);
> +       amdgpu_vm_get_pd_bo(vm, &list, &pd);
> +
> +       spin_lock(&vm->status_lock);
> +       vm->csa.reserved_top -= sa->size;
> +       vaddr = vm->csa.reserved_top;
> +       spin_unlock(&vm->status_lock);
> +
> +       r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
> +       if (r) {
> +               DRM_ERROR("failed to reserve global CSA buffer(%d).\n", r);
> +               return r;
> +       }
> +
> +       bo_va = amdgpu_vm_bo_add(adev, vm, sa->robj);
> +       if (!bo_va) {
> +               DRM_ERROR("failed to create bo_va for global CSA buffer.\n");
> +               return -ENOMEM;
> +       }
> +
> +
> +       r = amdgpu_vm_bo_map(adev, bo_va, vaddr, 0, sa->size,
> +                            AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
> +                            AMDGPU_PTE_EXECUTABLE);
> +       if (r) {
> +               DRM_ERROR("failed to do bo_map on global CSA buffer(%d).\n", r);
> +               amdgpu_vm_bo_rmv(adev, bo_va);
> +               ttm_eu_backoff_reservation(&ticket, &list);
> +               kfree(bo_va);
> +               return r;
> +       }
> +
> +       ttm_eu_backoff_reservation(&ticket, &list);
> +       amdgpu_gem_va_update_vm(adev, bo_va, AMDGPU_VA_OP_MAP);
> +
> +       vm->csa.va = bo_va;
> +       vm->csa.csa_addr = vaddr;
> +       vm->csa.gds_addr = vaddr + PAGE_SIZE;
> +
> +       return 0;
> +}
> +
> +int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> +{
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       struct amd_xgpu_csa *sa = NULL;
> +
> +       if (!xgpu)
> +               return 0;
> +
> +       sa = &xgpu->sa;
> +
> +       return xgpu_vm_map_csa(adev, vm, sa);
> +}
> +
> +static void xgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> +                             struct amd_xgpu_csa *sa)
> +{
> +       int r;
> +       struct ww_acquire_ctx ticket;
> +       struct list_head list, duplicates;
> +       struct amdgpu_bo_list_entry pd;
> +
> +       if (!vm->csa.va)
> +               return;
> +
> +       INIT_LIST_HEAD(&list);
> +       INIT_LIST_HEAD(&duplicates);
> +       list_add(&vm->csa.tv.head, &list);
> +       amdgpu_vm_get_pd_bo(vm, &list, &pd);
> +
> +       r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
> +       if (r) {
> +               DRM_ERROR("failed to reserve global CSA buffer(%d).\n", r);
> +               return;
> +       }
> +
> +       amdgpu_vm_bo_rmv(adev, vm->csa.va);
> +       /* maybe we don't need to do real clearing for the vm will die soon */
> +       r = amdgpu_vm_clear_freed(adev, vm);
> +       if (r) {
> +               DRM_ERROR("failed to clear global CSA bo(%d).\n", r);
> +               return;
> +       }
> +
> +       ttm_eu_backoff_reservation(&ticket, &list);
> +       vm->csa.va = NULL;
> +       vm->csa.csa_addr = vm->csa.gds_addr = 0;
> +}
> +
> +void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> +{
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       struct amd_xgpu_csa *sa = NULL;
> +
> +       if (!xgpu)
> +               return;
> +
> +       sa = &xgpu->sa;
> +       xgpu_vm_unmap_csa(adev, vm, sa);
> +}
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

答复: [PATCH 11/23] drm/amdgpu: implement context save area(CSA) feature

[Liu, Monk]

[-- Attachment #1.1: Type: text/plain, Size: 13792 bytes --]

the CSA is used for world switch, and each amdgpu device should have one and only one CSA,

and this CSA will pined, and mapped to each virtual memory /process.


CP/RLCV will use this CSA buffer when preemption occurred, and will write some hardware status into this CSA buffer, within the current IB's context (that's why need do mapping for each virtual memory on CSA)


BR Monk

________________________________
发件人: Alex Deucher <alexdeucher@gmail.com>
发送时间: 2016年12月20日 7:20:09
收件人: Yu, Xiangliang
抄送: amd-gfx list; dl.SRDC_SW_GPUVirtualization; Liu, Monk
主题: Re: [PATCH 11/23] drm/amdgpu: implement context save area(CSA) feature

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> CSA is need by world switch. This patch implement CSA feature and
> bind it to each VM, so hardware can save the state into the area
> and restore it when running again.
>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>

Isn't the CSA actually for preemption?  Preemption is useful outside
of the VF case as well so it should be untangled from the mxgpu code
so it can be utilized independently.

Alex

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  14 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |   8 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   |   4 +
>  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    |  12 ++
>  drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c    | 209 +++++++++++++++++++++++++++++++
>  5 files changed, 247 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index 8ee70f8..dff1248 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -24,6 +24,8 @@
>  #ifndef AMDGPU_VIRT_H
>  #define AMDGPU_VIRT_H
>
> +struct amdgpu_vm;
> +
>  #define AMDGPU_SRIOV_CAPS_SRIOV_VBIOS  (1 << 0) /* vBIOS is sr-iov ready */
>  #define AMDGPU_SRIOV_CAPS_ENABLE_IOV   (1 << 1) /* sr-iov is enabled on this GPU */
>  #define AMDGPU_SRIOV_CAPS_IS_VF        (1 << 2) /* this GPU is a virtual function */
> @@ -33,6 +35,14 @@ struct amdgpu_virtualization {
>         uint32_t virtual_caps;
>  };
>
> +struct amdgpu_csa {
> +       struct amdgpu_bo_va             *va;
> +       struct ttm_validate_buffer      tv;
> +       uint64_t                        reserved_top;
> +       uint64_t                        csa_addr;
> +       uint64_t                        gds_addr;
> +};
> +
>  #define amdgpu_sriov_enabled(adev) \
>  ((adev)->virtualization.virtual_caps & AMDGPU_SRIOV_CAPS_ENABLE_IOV)
>
> @@ -55,4 +65,8 @@ static inline bool is_virtual_machine(void)
>  }
>
>  int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev);
> +
> +/* Context Save Area functions */
> +int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
> +void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index d05546e..98540d9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -1539,6 +1539,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         pd_size = amdgpu_vm_directory_size(adev);
>         pd_entries = amdgpu_vm_num_pdes(adev);
>
> +       vm->csa.reserved_top = AMDGPU_VA_RESERVED_SIZE;
> +
>         /* allocate page table array */
>         vm->page_tables = drm_calloc_large(pd_entries, sizeof(struct amdgpu_vm_pt));
>         if (vm->page_tables == NULL) {
> @@ -1576,6 +1578,10 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         vm->last_eviction_counter = atomic64_read(&adev->num_evictions);
>         amdgpu_bo_unreserve(vm->page_directory);
>
> +       r = amdgpu_vm_map_csa(adev, vm);
> +       if (r)
> +               goto error_free_page_directory;
> +
>         return 0;
>
>  error_free_page_directory:
> @@ -1606,6 +1612,8 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         struct amdgpu_bo_va_mapping *mapping, *tmp;
>         int i;
>
> +       amdgpu_vm_unmap_csa(adev, vm);
> +
>         amd_sched_entity_fini(vm->entity.sched, &vm->entity);
>
>         if (!RB_EMPTY_ROOT(&vm->va)) {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index 42a629b..d90630a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -29,6 +29,7 @@
>  #include "gpu_scheduler.h"
>  #include "amdgpu_sync.h"
>  #include "amdgpu_ring.h"
> +#include "amdgpu_virt.h"
>
>  struct amdgpu_bo_va;
>  struct amdgpu_job;
> @@ -109,6 +110,9 @@ struct amdgpu_vm {
>         /* Scheduler entity for page table updates */
>         struct amd_sched_entity entity;
>
> +       /* Context Save Area */
> +       struct amdgpu_csa       csa;
> +
>         /* client id */
>         u64                     client_id;
>  };
> diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> index 6ab13bc..a25e07f 100644
> --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> @@ -26,12 +26,24 @@
>
>  #include "amdgpu.h"
>
> +/* xgpu structures */
> +struct amd_xgpu_csa {
> +       struct amdgpu_bo            *robj;
> +       uint64_t                    gpu_addr;
> +       uint64_t                    gds_addr;
> +       int32_t                     size;
> +};
> +
>  struct amd_xgpu {
>         struct amdgpu_device    *adev;
>         struct mutex            lock;
> +       struct amd_xgpu_csa     sa;
>         u32                     reg_val_offs;
>  };
>
>  extern int amd_xgpu_alloc(struct amdgpu_device *adev);
>  extern void amd_xgpu_free(struct amd_xgpu *xgpu);
> +
> +extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);
> +extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
>  #endif
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> new file mode 100644
> index 0000000..246a747
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> @@ -0,0 +1,209 @@
> +/*
> + * Copyright 2016 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + * Authors: Xiangliang.Yu@amd.com
> + *         Monk.Liu@amd.com
> + *
> + */
> +#include "amd_mxgpu.h"
> +#include "vid.h"
> +
> +static int xgpu_init_csa(struct amdgpu_device *adev, struct amd_xgpu_csa *sa)
> +{
> +       int r, size;
> +       void *ptr;
> +
> +       /* meta data (4k) + gds-gfx (4k)*/
> +       size = PAGE_SIZE + adev->gds.mem.gfx_partition_size;
> +
> +       r = amdgpu_bo_create(adev, size, PAGE_SIZE, true,
> +                            AMDGPU_GEM_DOMAIN_GTT,
> +                            AMDGPU_GEM_CREATE_CPU_GTT_USWC,
> +                            NULL, NULL, &sa->robj);
> +       if (r) {
> +               dev_err(adev->dev, "(%d) failed to allocate csa bo\n", r);
> +               return r;
> +       }
> +
> +       r = amdgpu_bo_reserve(sa->robj, true);
> +       if (unlikely(r != 0))
> +               goto error_free;
> +
> +       r = amdgpu_bo_pin(sa->robj, AMDGPU_GEM_DOMAIN_GTT, &sa->gpu_addr);
> +       if (r)
> +               goto error_unreserve;
> +
> +       r = amdgpu_bo_kmap(sa->robj, &ptr);
> +       if (r)
> +               goto error_unpin;
> +
> +       memset(ptr, 0, size);
> +       amdgpu_bo_unreserve(sa->robj);
> +
> +       sa->size = size;
> +       sa->gds_addr = sa->gpu_addr + PAGE_SIZE;
> +
> +       return 0;
> +
> +error_unpin:
> +       amdgpu_bo_unpin(sa->robj);
> +error_unreserve:
> +       amdgpu_bo_unreserve(sa->robj);
> +error_free:
> +       amdgpu_bo_unref(&sa->robj);
> +       return r;
> +}
> +
> +int xgpu_allocate_csa(struct amd_xgpu *xgpu)
> +{
> +       struct amdgpu_device *adev = xgpu->adev;
> +       struct amd_xgpu_csa *sa = &xgpu->sa;
> +
> +       return xgpu_init_csa(adev, sa);
> +}
> +
> +void xgpu_destroy_csa(struct amd_xgpu_csa *sa)
> +{
> +       amdgpu_bo_reserve(sa->robj, true);
> +       amdgpu_bo_unpin(sa->robj);
> +       amdgpu_bo_unreserve(sa->robj);
> +       amdgpu_bo_unref(&sa->robj);
> +       sa->gpu_addr = 0;
> +       sa->gds_addr = 0;
> +}
> +
> +static int xgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> +                          struct amd_xgpu_csa *sa)
> +{
> +       int r;
> +       uint64_t vaddr;
> +       struct ww_acquire_ctx ticket;
> +       struct list_head list, duplicates;
> +       struct amdgpu_bo_list_entry pd;
> +       struct amdgpu_bo_va *bo_va;
> +
> +       INIT_LIST_HEAD(&list);
> +       INIT_LIST_HEAD(&duplicates);
> +       INIT_LIST_HEAD(&vm->csa.tv.head);
> +       vm->csa.tv.bo = &sa->robj->tbo;
> +       vm->csa.tv.shared = true;
> +
> +       list_add(&vm->csa.tv.head, &list);
> +       amdgpu_vm_get_pd_bo(vm, &list, &pd);
> +
> +       spin_lock(&vm->status_lock);
> +       vm->csa.reserved_top -= sa->size;
> +       vaddr = vm->csa.reserved_top;
> +       spin_unlock(&vm->status_lock);
> +
> +       r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
> +       if (r) {
> +               DRM_ERROR("failed to reserve global CSA buffer(%d).\n", r);
> +               return r;
> +       }
> +
> +       bo_va = amdgpu_vm_bo_add(adev, vm, sa->robj);
> +       if (!bo_va) {
> +               DRM_ERROR("failed to create bo_va for global CSA buffer.\n");
> +               return -ENOMEM;
> +       }
> +
> +
> +       r = amdgpu_vm_bo_map(adev, bo_va, vaddr, 0, sa->size,
> +                            AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
> +                            AMDGPU_PTE_EXECUTABLE);
> +       if (r) {
> +               DRM_ERROR("failed to do bo_map on global CSA buffer(%d).\n", r);
> +               amdgpu_vm_bo_rmv(adev, bo_va);
> +               ttm_eu_backoff_reservation(&ticket, &list);
> +               kfree(bo_va);
> +               return r;
> +       }
> +
> +       ttm_eu_backoff_reservation(&ticket, &list);
> +       amdgpu_gem_va_update_vm(adev, bo_va, AMDGPU_VA_OP_MAP);
> +
> +       vm->csa.va = bo_va;
> +       vm->csa.csa_addr = vaddr;
> +       vm->csa.gds_addr = vaddr + PAGE_SIZE;
> +
> +       return 0;
> +}
> +
> +int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> +{
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       struct amd_xgpu_csa *sa = NULL;
> +
> +       if (!xgpu)
> +               return 0;
> +
> +       sa = &xgpu->sa;
> +
> +       return xgpu_vm_map_csa(adev, vm, sa);
> +}
> +
> +static void xgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> +                             struct amd_xgpu_csa *sa)
> +{
> +       int r;
> +       struct ww_acquire_ctx ticket;
> +       struct list_head list, duplicates;
> +       struct amdgpu_bo_list_entry pd;
> +
> +       if (!vm->csa.va)
> +               return;
> +
> +       INIT_LIST_HEAD(&list);
> +       INIT_LIST_HEAD(&duplicates);
> +       list_add(&vm->csa.tv.head, &list);
> +       amdgpu_vm_get_pd_bo(vm, &list, &pd);
> +
> +       r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
> +       if (r) {
> +               DRM_ERROR("failed to reserve global CSA buffer(%d).\n", r);
> +               return;
> +       }
> +
> +       amdgpu_vm_bo_rmv(adev, vm->csa.va);
> +       /* maybe we don't need to do real clearing for the vm will die soon */
> +       r = amdgpu_vm_clear_freed(adev, vm);
> +       if (r) {
> +               DRM_ERROR("failed to clear global CSA bo(%d).\n", r);
> +               return;
> +       }
> +
> +       ttm_eu_backoff_reservation(&ticket, &list);
> +       vm->csa.va = NULL;
> +       vm->csa.csa_addr = vm->csa.gds_addr = 0;
> +}
> +
> +void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> +{
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       struct amd_xgpu_csa *sa = NULL;
> +
> +       if (!xgpu)
> +               return;
> +
> +       sa = &xgpu->sa;
> +       xgpu_vm_unmap_csa(adev, vm, sa);
> +}
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 27301 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 11/23] drm/amdgpu: implement context save area(CSA) feature

[Yu, Xiangliang]

[-- Attachment #1.1: Type: text/plain, Size: 14518 bytes --]

Thank monk’s detail expanation.
And I think this patch is only support virtualization world switch, not touch whole amdpgu preemption.

Thanks!
Xiangliang Yu

From: Liu, Monk
Sent: Tuesday, December 20, 2016 11:58 AM
To: Alex Deucher <alexdeucher@gmail.com>; Yu, Xiangliang <Xiangliang.Yu@amd.com>
Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>; dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>
Subject: 答复: [PATCH 11/23] drm/amdgpu: implement context save area(CSA) feature


the CSA is used for world switch, and each amdgpu device should have one and only one CSA,

and this CSA will pined, and mapped to each virtual memory /process.



CP/RLCV will use this CSA buffer when preemption occurred, and will write some hardware status into this CSA buffer, within the current IB's context (that's why need do mapping for each virtual memory on CSA)



BR Monk

________________________________
发件人: Alex Deucher <alexdeucher@gmail.com<mailto:alexdeucher@gmail.com>>
发送时间: 2016年12月20日 7:20:09
收件人: Yu, Xiangliang
抄送: amd-gfx list; dl.SRDC_SW_GPUVirtualization; Liu, Monk
主题: Re: [PATCH 11/23] drm/amdgpu: implement context save area(CSA) feature

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>> wrote:
> CSA is need by world switch. This patch implement CSA feature and
> bind it to each VM, so hardware can save the state into the area
> and restore it when running again.
>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>>

Isn't the CSA actually for preemption?  Preemption is useful outside
of the VF case as well so it should be untangled from the mxgpu code
so it can be utilized independently.

Alex

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  14 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |   8 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   |   4 +
>  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    |  12 ++
>  drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c    | 209 +++++++++++++++++++++++++++++++
>  5 files changed, 247 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index 8ee70f8..dff1248 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -24,6 +24,8 @@
>  #ifndef AMDGPU_VIRT_H
>  #define AMDGPU_VIRT_H
>
> +struct amdgpu_vm;
> +
>  #define AMDGPU_SRIOV_CAPS_SRIOV_VBIOS  (1 << 0) /* vBIOS is sr-iov ready */
>  #define AMDGPU_SRIOV_CAPS_ENABLE_IOV   (1 << 1) /* sr-iov is enabled on this GPU */
>  #define AMDGPU_SRIOV_CAPS_IS_VF        (1 << 2) /* this GPU is a virtual function */
> @@ -33,6 +35,14 @@ struct amdgpu_virtualization {
>         uint32_t virtual_caps;
>  };
>
> +struct amdgpu_csa {
> +       struct amdgpu_bo_va             *va;
> +       struct ttm_validate_buffer      tv;
> +       uint64_t                        reserved_top;
> +       uint64_t                        csa_addr;
> +       uint64_t                        gds_addr;
> +};
> +
>  #define amdgpu_sriov_enabled(adev) \
>  ((adev)->virtualization.virtual_caps & AMDGPU_SRIOV_CAPS_ENABLE_IOV)
>
> @@ -55,4 +65,8 @@ static inline bool is_virtual_machine(void)
>  }
>
>  int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev);
> +
> +/* Context Save Area functions */
> +int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
> +void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index d05546e..98540d9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -1539,6 +1539,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         pd_size = amdgpu_vm_directory_size(adev);
>         pd_entries = amdgpu_vm_num_pdes(adev);
>
> +       vm->csa.reserved_top = AMDGPU_VA_RESERVED_SIZE;
> +
>         /* allocate page table array */
>         vm->page_tables = drm_calloc_large(pd_entries, sizeof(struct amdgpu_vm_pt));
>         if (vm->page_tables == NULL) {
> @@ -1576,6 +1578,10 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         vm->last_eviction_counter = atomic64_read(&adev->num_evictions);
>         amdgpu_bo_unreserve(vm->page_directory);
>
> +       r = amdgpu_vm_map_csa(adev, vm);
> +       if (r)
> +               goto error_free_page_directory;
> +
>         return 0;
>
>  error_free_page_directory:
> @@ -1606,6 +1612,8 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         struct amdgpu_bo_va_mapping *mapping, *tmp;
>         int i;
>
> +       amdgpu_vm_unmap_csa(adev, vm);
> +
>         amd_sched_entity_fini(vm->entity.sched, &vm->entity);
>
>         if (!RB_EMPTY_ROOT(&vm->va)) {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index 42a629b..d90630a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -29,6 +29,7 @@
>  #include "gpu_scheduler.h"
>  #include "amdgpu_sync.h"
>  #include "amdgpu_ring.h"
> +#include "amdgpu_virt.h"
>
>  struct amdgpu_bo_va;
>  struct amdgpu_job;
> @@ -109,6 +110,9 @@ struct amdgpu_vm {
>         /* Scheduler entity for page table updates */
>         struct amd_sched_entity entity;
>
> +       /* Context Save Area */
> +       struct amdgpu_csa       csa;
> +
>         /* client id */
>         u64                     client_id;
>  };
> diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> index 6ab13bc..a25e07f 100644
> --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> @@ -26,12 +26,24 @@
>
>  #include "amdgpu.h"
>
> +/* xgpu structures */
> +struct amd_xgpu_csa {
> +       struct amdgpu_bo            *robj;
> +       uint64_t                    gpu_addr;
> +       uint64_t                    gds_addr;
> +       int32_t                     size;
> +};
> +
>  struct amd_xgpu {
>         struct amdgpu_device    *adev;
>         struct mutex            lock;
> +       struct amd_xgpu_csa     sa;
>         u32                     reg_val_offs;
>  };
>
>  extern int amd_xgpu_alloc(struct amdgpu_device *adev);
>  extern void amd_xgpu_free(struct amd_xgpu *xgpu);
> +
> +extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);
> +extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
>  #endif
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> new file mode 100644
> index 0000000..246a747
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> @@ -0,0 +1,209 @@
> +/*
> + * Copyright 2016 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + * Authors: Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>
> + *         Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>
> + *
> + */
> +#include "amd_mxgpu.h"
> +#include "vid.h"
> +
> +static int xgpu_init_csa(struct amdgpu_device *adev, struct amd_xgpu_csa *sa)
> +{
> +       int r, size;
> +       void *ptr;
> +
> +       /* meta data (4k) + gds-gfx (4k)*/
> +       size = PAGE_SIZE + adev->gds.mem.gfx_partition_size;
> +
> +       r = amdgpu_bo_create(adev, size, PAGE_SIZE, true,
> +                            AMDGPU_GEM_DOMAIN_GTT,
> +                            AMDGPU_GEM_CREATE_CPU_GTT_USWC,
> +                            NULL, NULL, &sa->robj);
> +       if (r) {
> +               dev_err(adev->dev, "(%d) failed to allocate csa bo\n", r);
> +               return r;
> +       }
> +
> +       r = amdgpu_bo_reserve(sa->robj, true);
> +       if (unlikely(r != 0))
> +               goto error_free;
> +
> +       r = amdgpu_bo_pin(sa->robj, AMDGPU_GEM_DOMAIN_GTT, &sa->gpu_addr);
> +       if (r)
> +               goto error_unreserve;
> +
> +       r = amdgpu_bo_kmap(sa->robj, &ptr);
> +       if (r)
> +               goto error_unpin;
> +
> +       memset(ptr, 0, size);
> +       amdgpu_bo_unreserve(sa->robj);
> +
> +       sa->size = size;
> +       sa->gds_addr = sa->gpu_addr + PAGE_SIZE;
> +
> +       return 0;
> +
> +error_unpin:
> +       amdgpu_bo_unpin(sa->robj);
> +error_unreserve:
> +       amdgpu_bo_unreserve(sa->robj);
> +error_free:
> +       amdgpu_bo_unref(&sa->robj);
> +       return r;
> +}
> +
> +int xgpu_allocate_csa(struct amd_xgpu *xgpu)
> +{
> +       struct amdgpu_device *adev = xgpu->adev;
> +       struct amd_xgpu_csa *sa = &xgpu->sa;
> +
> +       return xgpu_init_csa(adev, sa);
> +}
> +
> +void xgpu_destroy_csa(struct amd_xgpu_csa *sa)
> +{
> +       amdgpu_bo_reserve(sa->robj, true);
> +       amdgpu_bo_unpin(sa->robj);
> +       amdgpu_bo_unreserve(sa->robj);
> +       amdgpu_bo_unref(&sa->robj);
> +       sa->gpu_addr = 0;
> +       sa->gds_addr = 0;
> +}
> +
> +static int xgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> +                          struct amd_xgpu_csa *sa)
> +{
> +       int r;
> +       uint64_t vaddr;
> +       struct ww_acquire_ctx ticket;
> +       struct list_head list, duplicates;
> +       struct amdgpu_bo_list_entry pd;
> +       struct amdgpu_bo_va *bo_va;
> +
> +       INIT_LIST_HEAD(&list);
> +       INIT_LIST_HEAD(&duplicates);
> +       INIT_LIST_HEAD(&vm->csa.tv.head);
> +       vm->csa.tv.bo = &sa->robj->tbo;
> +       vm->csa.tv.shared = true;
> +
> +       list_add(&vm->csa.tv.head, &list);
> +       amdgpu_vm_get_pd_bo(vm, &list, &pd);
> +
> +       spin_lock(&vm->status_lock);
> +       vm->csa.reserved_top -= sa->size;
> +       vaddr = vm->csa.reserved_top;
> +       spin_unlock(&vm->status_lock);
> +
> +       r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
> +       if (r) {
> +               DRM_ERROR("failed to reserve global CSA buffer(%d).\n", r);
> +               return r;
> +       }
> +
> +       bo_va = amdgpu_vm_bo_add(adev, vm, sa->robj);
> +       if (!bo_va) {
> +               DRM_ERROR("failed to create bo_va for global CSA buffer.\n");
> +               return -ENOMEM;
> +       }
> +
> +
> +       r = amdgpu_vm_bo_map(adev, bo_va, vaddr, 0, sa->size,
> +                            AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
> +                            AMDGPU_PTE_EXECUTABLE);
> +       if (r) {
> +               DRM_ERROR("failed to do bo_map on global CSA buffer(%d).\n", r);
> +               amdgpu_vm_bo_rmv(adev, bo_va);
> +               ttm_eu_backoff_reservation(&ticket, &list);
> +               kfree(bo_va);
> +               return r;
> +       }
> +
> +       ttm_eu_backoff_reservation(&ticket, &list);
> +       amdgpu_gem_va_update_vm(adev, bo_va, AMDGPU_VA_OP_MAP);
> +
> +       vm->csa.va = bo_va;
> +       vm->csa.csa_addr = vaddr;
> +       vm->csa.gds_addr = vaddr + PAGE_SIZE;
> +
> +       return 0;
> +}
> +
> +int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> +{
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       struct amd_xgpu_csa *sa = NULL;
> +
> +       if (!xgpu)
> +               return 0;
> +
> +       sa = &xgpu->sa;
> +
> +       return xgpu_vm_map_csa(adev, vm, sa);
> +}
> +
> +static void xgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> +                             struct amd_xgpu_csa *sa)
> +{
> +       int r;
> +       struct ww_acquire_ctx ticket;
> +       struct list_head list, duplicates;
> +       struct amdgpu_bo_list_entry pd;
> +
> +       if (!vm->csa.va)
> +               return;
> +
> +       INIT_LIST_HEAD(&list);
> +       INIT_LIST_HEAD(&duplicates);
> +       list_add(&vm->csa.tv.head, &list);
> +       amdgpu_vm_get_pd_bo(vm, &list, &pd);
> +
> +       r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
> +       if (r) {
> +               DRM_ERROR("failed to reserve global CSA buffer(%d).\n", r);
> +               return;
> +       }
> +
> +       amdgpu_vm_bo_rmv(adev, vm->csa.va);
> +       /* maybe we don't need to do real clearing for the vm will die soon */
> +       r = amdgpu_vm_clear_freed(adev, vm);
> +       if (r) {
> +               DRM_ERROR("failed to clear global CSA bo(%d).\n", r);
> +               return;
> +       }
> +
> +       ttm_eu_backoff_reservation(&ticket, &list);
> +       vm->csa.va = NULL;
> +       vm->csa.csa_addr = vm->csa.gds_addr = 0;
> +}
> +
> +void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> +{
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       struct amd_xgpu_csa *sa = NULL;
> +
> +       if (!xgpu)
> +               return;
> +
> +       sa = &xgpu->sa;
> +       xgpu_vm_unmap_csa(adev, vm, sa);
> +}
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 33762 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 11/23] drm/amdgpu: implement context save area(CSA) feature

[Christian König]

[-- Attachment #1.1: Type: text/plain, Size: 16529 bytes --]

NAK on that whole approach. We previously had similar handling in radeon 
for the temporary IB buffer and it was a pain to support correctly.

The main problem is that you might then have a mismatch between the 
housekeeping structures in the VM and what is really in the page tables 
/ page directory.

Additional a concrete problem is that you simply assume that the PD and 
PTs are present when the VM is created, but that might not be correct 
under memory pressure. In this case the code below will just fail miserable.

I suggest that you go a step back and fully implement the support for 
the CSA for each context and not just a quick hack like this. If I 
remember our internal discussion correctly Monk already had promising 
locking patches for this for the general preemption support.

Regards,
Christian.

Am 20.12.2016 um 06:43 schrieb Yu, Xiangliang:
>
> Thank monk’s detail expanation.
>
> And I think this patch is only support virtualization world switch, 
> not touch whole amdpgu preemption.
>
> Thanks!
>
> Xiangliang Yu
>
> *From:*Liu, Monk
> *Sent:* Tuesday, December 20, 2016 11:58 AM
> *To:* Alex Deucher <alexdeucher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>; Yu, Xiangliang 
> <Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org>
> *Cc:* amd-gfx list <amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>; 
> dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization-5C7GfCeVMHo@public.gmane.org>
> *Subject:* 答复: [PATCH 11/23] drm/amdgpu: implement context save 
> area(CSA) feature
>
> the CSA is used for world switch, and each amdgpu device should have 
> one and only one CSA,
>
> and this CSA will pined, and mapped to each virtual memory /process.
>
> CP/RLCV will use this CSA buffer when preemption occurred, and will 
> write some hardware status into this CSA buffer, within the current 
> IB's context (that's why need do mapping for each virtual memory on CSA)
>
> BR Monk
>
> ------------------------------------------------------------------------
>
> *发件人**:*Alex Deucher <alexdeucher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org 
> <mailto:alexdeucher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>>
> *发送时间**:*2016年12月20日7:20:09
> *收件人**:*Yu, Xiangliang
> *抄送**:*amd-gfx list; dl.SRDC_SW_GPUVirtualization; Liu, Monk
> *主题**:*Re: [PATCH 11/23] drm/amdgpu: implement context save area(CSA) 
> feature
>
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org 
> <mailto:Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org>> wrote:
> > CSA is need by world switch. This patch implement CSA feature and
> > bind it to each VM, so hardware can save the state into the area
> > and restore it when running again.
> >
> > Signed-off-by: Monk Liu <Monk.Liu-5C7GfCeVMHo@public.gmane.org <mailto:Monk.Liu-5C7GfCeVMHo@public.gmane.org>>
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org 
> <mailto:Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org>>
>
> Isn't the CSA actually for preemption?  Preemption is useful outside
> of the VF case as well so it should be untangled from the mxgpu code
> so it can be utilized independently.
>
> Alex
>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  14 +++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |   8 ++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   |   4 +
> >  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    |  12 ++
> >  drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c    | 209 
> +++++++++++++++++++++++++++++++
> >  5 files changed, 247 insertions(+)
> >  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > index 8ee70f8..dff1248 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > @@ -24,6 +24,8 @@
> >  #ifndef AMDGPU_VIRT_H
> >  #define AMDGPU_VIRT_H
> >
> > +struct amdgpu_vm;
> > +
> >  #define AMDGPU_SRIOV_CAPS_SRIOV_VBIOS  (1 << 0) /* vBIOS is sr-iov 
> ready */
> >  #define AMDGPU_SRIOV_CAPS_ENABLE_IOV   (1 << 1) /* sr-iov is 
> enabled on this GPU */
> >  #define AMDGPU_SRIOV_CAPS_IS_VF        (1 << 2) /* this GPU is a 
> virtual function */
> > @@ -33,6 +35,14 @@ struct amdgpu_virtualization {
> >         uint32_t virtual_caps;
> >  };
> >
> > +struct amdgpu_csa {
> > +       struct amdgpu_bo_va             *va;
> > +       struct ttm_validate_buffer      tv;
> > +       uint64_t reserved_top;
> > +       uint64_t                        csa_addr;
> > +       uint64_t                        gds_addr;
> > +};
> > +
> >  #define amdgpu_sriov_enabled(adev) \
> >  ((adev)->virtualization.virtual_caps & AMDGPU_SRIOV_CAPS_ENABLE_IOV)
> >
> > @@ -55,4 +65,8 @@ static inline bool is_virtual_machine(void)
> >  }
> >
> >  int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev);
> > +
> > +/* Context Save Area functions */
> > +int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm 
> *vm);
> > +void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct 
> amdgpu_vm *vm);
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > index d05546e..98540d9 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > @@ -1539,6 +1539,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, 
> struct amdgpu_vm *vm)
> >         pd_size = amdgpu_vm_directory_size(adev);
> >         pd_entries = amdgpu_vm_num_pdes(adev);
> >
> > +       vm->csa.reserved_top = AMDGPU_VA_RESERVED_SIZE;
> > +
> >         /* allocate page table array */
> >         vm->page_tables = drm_calloc_large(pd_entries, sizeof(struct 
> amdgpu_vm_pt));
> >         if (vm->page_tables == NULL) {
> > @@ -1576,6 +1578,10 @@ int amdgpu_vm_init(struct amdgpu_device 
> *adev, struct amdgpu_vm *vm)
> >         vm->last_eviction_counter = atomic64_read(&adev->num_evictions);
> >         amdgpu_bo_unreserve(vm->page_directory);
> >
> > +       r = amdgpu_vm_map_csa(adev, vm);
> > +       if (r)
> > +               goto error_free_page_directory;
> > +
> >         return 0;
> >
> >  error_free_page_directory:
> > @@ -1606,6 +1612,8 @@ void amdgpu_vm_fini(struct amdgpu_device 
> *adev, struct amdgpu_vm *vm)
> >         struct amdgpu_bo_va_mapping *mapping, *tmp;
> >         int i;
> >
> > +       amdgpu_vm_unmap_csa(adev, vm);
> > +
> >         amd_sched_entity_fini(vm->entity.sched, &vm->entity);
> >
> >         if (!RB_EMPTY_ROOT(&vm->va)) {
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > index 42a629b..d90630a 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > @@ -29,6 +29,7 @@
> >  #include "gpu_scheduler.h"
> >  #include "amdgpu_sync.h"
> >  #include "amdgpu_ring.h"
> > +#include "amdgpu_virt.h"
> >
> >  struct amdgpu_bo_va;
> >  struct amdgpu_job;
> > @@ -109,6 +110,9 @@ struct amdgpu_vm {
> >         /* Scheduler entity for page table updates */
> >         struct amd_sched_entity entity;
> >
> > +       /* Context Save Area */
> > +       struct amdgpu_csa       csa;
> > +
> >         /* client id */
> >         u64                     client_id;
> >  };
> > diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h 
> b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > index 6ab13bc..a25e07f 100644
> > --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > @@ -26,12 +26,24 @@
> >
> >  #include "amdgpu.h"
> >
> > +/* xgpu structures */
> > +struct amd_xgpu_csa {
> > +       struct amdgpu_bo            *robj;
> > +       uint64_t                    gpu_addr;
> > +       uint64_t                    gds_addr;
> > +       int32_t                     size;
> > +};
> > +
> >  struct amd_xgpu {
> >         struct amdgpu_device    *adev;
> >         struct mutex            lock;
> > +       struct amd_xgpu_csa     sa;
> >         u32                     reg_val_offs;
> >  };
> >
> >  extern int amd_xgpu_alloc(struct amdgpu_device *adev);
> >  extern void amd_xgpu_free(struct amd_xgpu *xgpu);
> > +
> > +extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);
> > +extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c 
> b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> > new file mode 100644
> > index 0000000..246a747
> > --- /dev/null
> > +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> > @@ -0,0 +1,209 @@
> > +/*
> > + * Copyright 2016 Advanced Micro Devices, Inc.
> > + *
> > + * Permission is hereby granted, free of charge, to any person 
> obtaining a
> > + * copy of this software and associated documentation files (the 
> "Software"),
> > + * to deal in the Software without restriction, including without 
> limitation
> > + * the rights to use, copy, modify, merge, publish, distribute, 
> sublicense,
> > + * and/or sell copies of the Software, and to permit persons to 
> whom the
> > + * Software is furnished to do so, subject to the following conditions:
> > + *
> > + * The above copyright notice and this permission notice shall be 
> included in
> > + * all copies or substantial portions of the Software.
> > + *
> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 
> EXPRESS OR
> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
> MERCHANTABILITY,
> > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO 
> EVENT SHALL
> > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, 
> DAMAGES OR
> > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 
> OTHERWISE,
> > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE 
> USE OR
> > + * OTHER DEALINGS IN THE SOFTWARE.
> > + *
> > + * Authors: Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org <mailto:Xiangliang.Yu-5C7GfCeVMHo@public.gmane.org>
> > + * Monk.Liu-5C7GfCeVMHo@public.gmane.org <mailto:Monk.Liu-5C7GfCeVMHo@public.gmane.org>
> > + *
> > + */
> > +#include "amd_mxgpu.h"
> > +#include "vid.h"
> > +
> > +static int xgpu_init_csa(struct amdgpu_device *adev, struct 
> amd_xgpu_csa *sa)
> > +{
> > +       int r, size;
> > +       void *ptr;
> > +
> > +       /* meta data (4k) + gds-gfx (4k)*/
> > +       size = PAGE_SIZE + adev->gds.mem.gfx_partition_size;
> > +
> > +       r = amdgpu_bo_create(adev, size, PAGE_SIZE, true,
> > +                            AMDGPU_GEM_DOMAIN_GTT,
> > + AMDGPU_GEM_CREATE_CPU_GTT_USWC,
> > +                            NULL, NULL, &sa->robj);
> > +       if (r) {
> > +               dev_err(adev->dev, "(%d) failed to allocate csa 
> bo\n", r);
> > +               return r;
> > +       }
> > +
> > +       r = amdgpu_bo_reserve(sa->robj, true);
> > +       if (unlikely(r != 0))
> > +               goto error_free;
> > +
> > +       r = amdgpu_bo_pin(sa->robj, AMDGPU_GEM_DOMAIN_GTT, 
> &sa->gpu_addr);
> > +       if (r)
> > +               goto error_unreserve;
> > +
> > +       r = amdgpu_bo_kmap(sa->robj, &ptr);
> > +       if (r)
> > +               goto error_unpin;
> > +
> > +       memset(ptr, 0, size);
> > +       amdgpu_bo_unreserve(sa->robj);
> > +
> > +       sa->size = size;
> > +       sa->gds_addr = sa->gpu_addr + PAGE_SIZE;
> > +
> > +       return 0;
> > +
> > +error_unpin:
> > +       amdgpu_bo_unpin(sa->robj);
> > +error_unreserve:
> > +       amdgpu_bo_unreserve(sa->robj);
> > +error_free:
> > +       amdgpu_bo_unref(&sa->robj);
> > +       return r;
> > +}
> > +
> > +int xgpu_allocate_csa(struct amd_xgpu *xgpu)
> > +{
> > +       struct amdgpu_device *adev = xgpu->adev;
> > +       struct amd_xgpu_csa *sa = &xgpu->sa;
> > +
> > +       return xgpu_init_csa(adev, sa);
> > +}
> > +
> > +void xgpu_destroy_csa(struct amd_xgpu_csa *sa)
> > +{
> > +       amdgpu_bo_reserve(sa->robj, true);
> > +       amdgpu_bo_unpin(sa->robj);
> > +       amdgpu_bo_unreserve(sa->robj);
> > +       amdgpu_bo_unref(&sa->robj);
> > +       sa->gpu_addr = 0;
> > +       sa->gds_addr = 0;
> > +}
> > +
> > +static int xgpu_vm_map_csa(struct amdgpu_device *adev, struct 
> amdgpu_vm *vm,
> > +                          struct amd_xgpu_csa *sa)
> > +{
> > +       int r;
> > +       uint64_t vaddr;
> > +       struct ww_acquire_ctx ticket;
> > +       struct list_head list, duplicates;
> > +       struct amdgpu_bo_list_entry pd;
> > +       struct amdgpu_bo_va *bo_va;
> > +
> > +       INIT_LIST_HEAD(&list);
> > +       INIT_LIST_HEAD(&duplicates);
> > +       INIT_LIST_HEAD(&vm->csa.tv.head);
> > +       vm->csa.tv.bo = &sa->robj->tbo;
> > +       vm->csa.tv.shared = true;
> > +
> > +       list_add(&vm->csa.tv.head, &list);
> > +       amdgpu_vm_get_pd_bo(vm, &list, &pd);
> > +
> > +       spin_lock(&vm->status_lock);
> > +       vm->csa.reserved_top -= sa->size;
> > +       vaddr = vm->csa.reserved_top;
> > +       spin_unlock(&vm->status_lock);
> > +
> > +       r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
> > +       if (r) {
> > +               DRM_ERROR("failed to reserve global CSA 
> buffer(%d).\n", r);
> > +               return r;
> > +       }
> > +
> > +       bo_va = amdgpu_vm_bo_add(adev, vm, sa->robj);
> > +       if (!bo_va) {
> > +               DRM_ERROR("failed to create bo_va for global CSA 
> buffer.\n");
> > +               return -ENOMEM;
> > +       }
> > +
> > +
> > +       r = amdgpu_vm_bo_map(adev, bo_va, vaddr, 0, sa->size,
> > +                            AMDGPU_PTE_READABLE | 
> AMDGPU_PTE_WRITEABLE |
> > + AMDGPU_PTE_EXECUTABLE);
> > +       if (r) {
> > +               DRM_ERROR("failed to do bo_map on global CSA 
> buffer(%d).\n", r);
> > +               amdgpu_vm_bo_rmv(adev, bo_va);
> > + ttm_eu_backoff_reservation(&ticket, &list);
> > +               kfree(bo_va);
> > +               return r;
> > +       }
> > +
> > +       ttm_eu_backoff_reservation(&ticket, &list);
> > +       amdgpu_gem_va_update_vm(adev, bo_va, AMDGPU_VA_OP_MAP);
> > +
> > +       vm->csa.va = bo_va;
> > +       vm->csa.csa_addr = vaddr;
> > +       vm->csa.gds_addr = vaddr + PAGE_SIZE;
> > +
> > +       return 0;
> > +}
> > +
> > +int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> > +{
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +       struct amd_xgpu_csa *sa = NULL;
> > +
> > +       if (!xgpu)
> > +               return 0;
> > +
> > +       sa = &xgpu->sa;
> > +
> > +       return xgpu_vm_map_csa(adev, vm, sa);
> > +}
> > +
> > +static void xgpu_vm_unmap_csa(struct amdgpu_device *adev, struct 
> amdgpu_vm *vm,
> > +                             struct amd_xgpu_csa *sa)
> > +{
> > +       int r;
> > +       struct ww_acquire_ctx ticket;
> > +       struct list_head list, duplicates;
> > +       struct amdgpu_bo_list_entry pd;
> > +
> > +       if (!vm->csa.va)
> > +               return;
> > +
> > +       INIT_LIST_HEAD(&list);
> > +       INIT_LIST_HEAD(&duplicates);
> > +       list_add(&vm->csa.tv.head, &list);
> > +       amdgpu_vm_get_pd_bo(vm, &list, &pd);
> > +
> > +       r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates);
> > +       if (r) {
> > +               DRM_ERROR("failed to reserve global CSA 
> buffer(%d).\n", r);
> > +               return;
> > +       }
> > +
> > +       amdgpu_vm_bo_rmv(adev, vm->csa.va);
> > +       /* maybe we don't need to do real clearing for the vm will 
> die soon */
> > +       r = amdgpu_vm_clear_freed(adev, vm);
> > +       if (r) {
> > +               DRM_ERROR("failed to clear global CSA bo(%d).\n", r);
> > +               return;
> > +       }
> > +
> > +       ttm_eu_backoff_reservation(&ticket, &list);
> > +       vm->csa.va = NULL;
> > +       vm->csa.csa_addr = vm->csa.gds_addr = 0;
> > +}
> > +
> > +void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct 
> amdgpu_vm *vm)
> > +{
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +       struct amd_xgpu_csa *sa = NULL;
> > +
> > +       if (!xgpu)
> > +               return;
> > +
> > +       sa = &xgpu->sa;
> > +       xgpu_vm_unmap_csa(adev, vm, sa);
> > +}
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>
>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx



[-- Attachment #1.2: Type: text/html, Size: 36881 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 12/23] drm/amdgpu: Insert meta data during submitting IB

[Xiangliang Yu]

Virtualization world switch need each command that is submitted
into GFX with an extra entry, which will using WRITE_DATA to
fullfill CSA. In this way, CP will save CE/DE snapshots when
preemption occurred and restore it later.

Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c   |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  2 +
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    | 39 ++++++++++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c    | 88 ++++++++++++++++++++++++++++++++
 4 files changed, 131 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
index acf48de..cc35255 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
@@ -175,6 +175,8 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned num_ibs,
 	if (ring->funcs->emit_hdp_flush)
 		amdgpu_ring_emit_hdp_flush(ring);
 
+	amdgpu_gfx_ring_emit_meta_data(ring, vm);
+
 	/* always set cond_exec_polling to CONTINUE */
 	*ring->cond_exe_cpu_addr = 1;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
index dff1248..d6f57a2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
@@ -69,4 +69,6 @@ int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev);
 /* Context Save Area functions */
 int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
 void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
+void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
+				    struct amdgpu_vm *vm);
 #endif
diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
index a25e07f..cc3123b 100644
--- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
@@ -26,9 +26,48 @@
 
 #include "amdgpu.h"
 
+/* context save area structures */
+struct amdgpu_ce_ib_state {
+	uint32_t	ce_ib_completion_status;
+	uint32_t	ce_const_engine_count;
+	uint32_t	ce_ib_offset_ib1;
+	uint32_t	ce_ib_offset_ib2;
+};
+
+struct amdgpu_de_ib_state {
+	uint32_t	de_ib_completion_status;
+	uint32_t	de_const_engine_count;
+	uint32_t	de_ib_offset_ib1;
+	uint32_t	de_ib_offset_ib2;
+	uint32_t	preamble_begin_ib1;
+	uint32_t	preamble_begin_ib2;
+	uint32_t	preamble_end_ib1;
+	uint32_t	preamble_end_ib2;
+	uint32_t	draw_indirect_base_lo;
+	uint32_t	draw_indirect_base_hi;
+	uint32_t	disp_indirect_base_lo;
+	uint32_t	disp_indirect_base_hi;
+	uint32_t	gds_backup_addr_lo;
+	uint32_t	gds_backup_addr_hi;
+	uint32_t	index_base_addr_lo;
+	uint32_t	index_base_addr_hi;
+	uint32_t	sample_cntl;
+};
+
+struct amdgpu_gfx_meta_data {
+	struct amdgpu_ce_ib_state	ce_payload;
+	uint32_t			reserved1[60];
+	struct amdgpu_de_ib_state	de_payload;
+	uint32_t			de_ib_base_addr_lo;
+	uint32_t			de_ib_base_addr_hi;
+	uint32_t			reserved2[941];
+};
+
 /* xgpu structures */
 struct amd_xgpu_csa {
 	struct amdgpu_bo	    *robj;
+	struct amdgpu_ce_ib_state   ce_payload;
+	struct amdgpu_de_ib_state   de_payload;
 	uint64_t		    gpu_addr;
 	uint64_t		    gds_addr;
 	int32_t			    size;
diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
index 246a747..6d4246c 100644
--- a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
+++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
@@ -207,3 +207,91 @@ void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	sa = &xgpu->sa;
 	xgpu_vm_unmap_csa(adev, vm, sa);
 }
+
+static void amdgpu_ring_write_multiple(struct amdgpu_ring *ring,
+				       void *src, int count_dw)
+{
+	if (ring->count_dw < count_dw)
+		DRM_ERROR("writing more dwords to the ring than expected:%d.\n",
+			   count_dw);
+	else {
+		unsigned int chunk1, chunk2;
+		void *dst = (void *)&ring->ring[ring->wptr];
+
+		chunk1 = ring->ptr_mask + 1 - ring->wptr;
+		chunk1 = (chunk1 >= count_dw) ? count_dw : chunk1;
+		chunk2 = count_dw - chunk1;
+		chunk1 <<= 2;
+		chunk2 <<= 2;
+		if (chunk1) {
+			memcpy(dst, src, chunk1);
+			dst = (void *)(((uint64_t)dst + chunk1) &
+					ring->ptr_mask);
+		}
+
+		if (chunk2) {
+			src += chunk1;
+			dst = (void *)ring->ring;
+			memcpy(dst, src, chunk2);
+		}
+
+		ring->wptr += count_dw;
+		ring->wptr &= ring->ptr_mask;
+		ring->count_dw -= count_dw;
+	}
+}
+
+void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
+				    struct amdgpu_vm *vm)
+{
+	struct amdgpu_ce_ib_state *ce_payload;
+	struct amdgpu_de_ib_state *de_payload;
+	struct amd_xgpu_csa *sa = NULL;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)ring->adev->priv_data;
+	uint64_t csa_addr, gds_addr;
+	int cnt;
+
+	if (!xgpu || (ring->funcs->type != AMDGPU_RING_TYPE_GFX))
+		return;
+
+	sa = &xgpu->sa;
+
+	ce_payload = &sa->ce_payload;
+	de_payload = &sa->de_payload;
+	memset(ce_payload, 0, sizeof(*ce_payload));
+	memset(de_payload, 0, sizeof(*de_payload));
+
+	cnt = (sizeof(*ce_payload) >> 2) + 4 - 2;
+	csa_addr = vm ? vm->csa.csa_addr : sa->gpu_addr;
+	gds_addr = vm ? vm->csa.gds_addr : sa->gds_addr;
+	de_payload->gds_backup_addr_lo = lower_32_bits(gds_addr);
+	de_payload->gds_backup_addr_hi = upper_32_bits(gds_addr);
+
+	/* write CE meta data */
+	amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
+	amdgpu_ring_write(ring, WRITE_DATA_ENGINE_SEL(2) |
+				WRITE_DATA_DST_SEL(8) |
+				WR_CONFIRM | WRITE_DATA_CACHE_POLICY(0));
+	amdgpu_ring_write(ring, lower_32_bits(csa_addr +
+			  offsetof(struct amdgpu_gfx_meta_data, ce_payload)));
+	amdgpu_ring_write(ring, upper_32_bits(csa_addr +
+			  offsetof(struct amdgpu_gfx_meta_data, ce_payload)));
+
+	amdgpu_ring_write_multiple(ring, (void *)ce_payload,
+					 sizeof(*ce_payload) >> 2);
+
+	/* write DE meta data */
+	cnt = (sizeof(*de_payload) >> 2) + 4 - 2;
+
+	amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
+	amdgpu_ring_write(ring, WRITE_DATA_ENGINE_SEL(1) |
+				WRITE_DATA_DST_SEL(8) |
+				WR_CONFIRM | WRITE_DATA_CACHE_POLICY(0));
+	amdgpu_ring_write(ring, lower_32_bits(csa_addr +
+			  offsetof(struct amdgpu_gfx_meta_data, de_payload)));
+	amdgpu_ring_write(ring, upper_32_bits(csa_addr +
+			  offsetof(struct amdgpu_gfx_meta_data, de_payload)));
+
+	amdgpu_ring_write_multiple(ring, (void *)de_payload,
+					 sizeof(*de_payload) >> 2);
+}
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 12/23] drm/amdgpu: Insert meta data during submitting IB

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> Virtualization world switch need each command that is submitted
> into GFX with an extra entry, which will using WRITE_DATA to
> fullfill CSA. In this way, CP will save CE/DE snapshots when
> preemption occurred and restore it later.
>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c   |  2 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  2 +
>  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    | 39 ++++++++++++++
>  drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c    | 88 ++++++++++++++++++++++++++++++++
>  4 files changed, 131 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> index acf48de..cc35255 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> @@ -175,6 +175,8 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned num_ibs,
>         if (ring->funcs->emit_hdp_flush)
>                 amdgpu_ring_emit_hdp_flush(ring);
>
> +       amdgpu_gfx_ring_emit_meta_data(ring, vm);
> +
>         /* always set cond_exec_polling to CONTINUE */
>         *ring->cond_exe_cpu_addr = 1;
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index dff1248..d6f57a2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -69,4 +69,6 @@ int amd_xgpu_set_ip_blocks(struct amdgpu_device *adev);
>  /* Context Save Area functions */
>  int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>  void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
> +void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
> +                                   struct amdgpu_vm *vm);
>  #endif
> diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> index a25e07f..cc3123b 100644
> --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> @@ -26,9 +26,48 @@
>
>  #include "amdgpu.h"
>
> +/* context save area structures */
> +struct amdgpu_ce_ib_state {
> +       uint32_t        ce_ib_completion_status;
> +       uint32_t        ce_const_engine_count;
> +       uint32_t        ce_ib_offset_ib1;
> +       uint32_t        ce_ib_offset_ib2;
> +};
> +
> +struct amdgpu_de_ib_state {
> +       uint32_t        de_ib_completion_status;
> +       uint32_t        de_const_engine_count;
> +       uint32_t        de_ib_offset_ib1;
> +       uint32_t        de_ib_offset_ib2;
> +       uint32_t        preamble_begin_ib1;
> +       uint32_t        preamble_begin_ib2;
> +       uint32_t        preamble_end_ib1;
> +       uint32_t        preamble_end_ib2;
> +       uint32_t        draw_indirect_base_lo;
> +       uint32_t        draw_indirect_base_hi;
> +       uint32_t        disp_indirect_base_lo;
> +       uint32_t        disp_indirect_base_hi;
> +       uint32_t        gds_backup_addr_lo;
> +       uint32_t        gds_backup_addr_hi;
> +       uint32_t        index_base_addr_lo;
> +       uint32_t        index_base_addr_hi;
> +       uint32_t        sample_cntl;
> +};
> +
> +struct amdgpu_gfx_meta_data {
> +       struct amdgpu_ce_ib_state       ce_payload;
> +       uint32_t                        reserved1[60];
> +       struct amdgpu_de_ib_state       de_payload;
> +       uint32_t                        de_ib_base_addr_lo;
> +       uint32_t                        de_ib_base_addr_hi;
> +       uint32_t                        reserved2[941];
> +};
> +

These are gfx8 specific and should be moved to gfx8 module.

>  /* xgpu structures */
>  struct amd_xgpu_csa {
>         struct amdgpu_bo            *robj;
> +       struct amdgpu_ce_ib_state   ce_payload;
> +       struct amdgpu_de_ib_state   de_payload;
>         uint64_t                    gpu_addr;
>         uint64_t                    gds_addr;
>         int32_t                     size;
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> index 246a747..6d4246c 100644
> --- a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> @@ -207,3 +207,91 @@ void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>         sa = &xgpu->sa;
>         xgpu_vm_unmap_csa(adev, vm, sa);
>  }
> +
> +static void amdgpu_ring_write_multiple(struct amdgpu_ring *ring,
> +                                      void *src, int count_dw)
> +{
> +       if (ring->count_dw < count_dw)
> +               DRM_ERROR("writing more dwords to the ring than expected:%d.\n",
> +                          count_dw);
> +       else {
> +               unsigned int chunk1, chunk2;
> +               void *dst = (void *)&ring->ring[ring->wptr];
> +
> +               chunk1 = ring->ptr_mask + 1 - ring->wptr;
> +               chunk1 = (chunk1 >= count_dw) ? count_dw : chunk1;
> +               chunk2 = count_dw - chunk1;
> +               chunk1 <<= 2;
> +               chunk2 <<= 2;
> +               if (chunk1) {
> +                       memcpy(dst, src, chunk1);
> +                       dst = (void *)(((uint64_t)dst + chunk1) &
> +                                       ring->ptr_mask);
> +               }
> +
> +               if (chunk2) {
> +                       src += chunk1;
> +                       dst = (void *)ring->ring;
> +                       memcpy(dst, src, chunk2);
> +               }
> +
> +               ring->wptr += count_dw;
> +               ring->wptr &= ring->ptr_mask;
> +               ring->count_dw -= count_dw;
> +       }
> +}
> +
> +void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
> +                                   struct amdgpu_vm *vm)
> +{
> +       struct amdgpu_ce_ib_state *ce_payload;
> +       struct amdgpu_de_ib_state *de_payload;
> +       struct amd_xgpu_csa *sa = NULL;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)ring->adev->priv_data;
> +       uint64_t csa_addr, gds_addr;
> +       int cnt;
> +
> +       if (!xgpu || (ring->funcs->type != AMDGPU_RING_TYPE_GFX))
> +               return;
> +
> +       sa = &xgpu->sa;

No need to make this dependent on xgpu.  As I said in the previous
patch preemption is useful independent of xgpu.

> +
> +       ce_payload = &sa->ce_payload;
> +       de_payload = &sa->de_payload;
> +       memset(ce_payload, 0, sizeof(*ce_payload));
> +       memset(de_payload, 0, sizeof(*de_payload));
> +
> +       cnt = (sizeof(*ce_payload) >> 2) + 4 - 2;
> +       csa_addr = vm ? vm->csa.csa_addr : sa->gpu_addr;
> +       gds_addr = vm ? vm->csa.gds_addr : sa->gds_addr;
> +       de_payload->gds_backup_addr_lo = lower_32_bits(gds_addr);
> +       de_payload->gds_backup_addr_hi = upper_32_bits(gds_addr);
> +
> +       /* write CE meta data */
> +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
> +       amdgpu_ring_write(ring, WRITE_DATA_ENGINE_SEL(2) |
> +                               WRITE_DATA_DST_SEL(8) |
> +                               WR_CONFIRM | WRITE_DATA_CACHE_POLICY(0));
> +       amdgpu_ring_write(ring, lower_32_bits(csa_addr +
> +                         offsetof(struct amdgpu_gfx_meta_data, ce_payload)));
> +       amdgpu_ring_write(ring, upper_32_bits(csa_addr +
> +                         offsetof(struct amdgpu_gfx_meta_data, ce_payload)));
> +
> +       amdgpu_ring_write_multiple(ring, (void *)ce_payload,
> +                                        sizeof(*ce_payload) >> 2);
> +
> +       /* write DE meta data */
> +       cnt = (sizeof(*de_payload) >> 2) + 4 - 2;
> +
> +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
> +       amdgpu_ring_write(ring, WRITE_DATA_ENGINE_SEL(1) |
> +                               WRITE_DATA_DST_SEL(8) |
> +                               WR_CONFIRM | WRITE_DATA_CACHE_POLICY(0));
> +       amdgpu_ring_write(ring, lower_32_bits(csa_addr +
> +                         offsetof(struct amdgpu_gfx_meta_data, de_payload)));
> +       amdgpu_ring_write(ring, upper_32_bits(csa_addr +
> +                         offsetof(struct amdgpu_gfx_meta_data, de_payload)));
> +
> +       amdgpu_ring_write_multiple(ring, (void *)de_payload,
> +                                        sizeof(*de_payload) >> 2);
> +}

This function is gfx8 specific and should be moved to the gfx8 module.

Alex

> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 12/23] drm/amdgpu: Insert meta data during submitting IB

[Yu, Xiangliang]

> -----Original Message-----
> From: Alex Deucher [mailto:alexdeucher@gmail.com]
> Sent: Tuesday, December 20, 2016 7:23 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> dl.SRDC_SW_GPUVirtualization
> <dl.SRDC_SW_GPUVirtualization@amd.com>; Liu, Monk
> <Monk.Liu@amd.com>
> Subject: Re: [PATCH 12/23] drm/amdgpu: Insert meta data during submitting
> IB
> 
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> wrote:
> > Virtualization world switch need each command that is submitted into
> > GFX with an extra entry, which will using WRITE_DATA to fullfill CSA.
> > In this way, CP will save CE/DE snapshots when preemption occurred and
> > restore it later.
> >
> > Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c   |  2 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  2 +
> >  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h    | 39 ++++++++++++++
> >  drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c    | 88
> ++++++++++++++++++++++++++++++++
> >  4 files changed, 131 insertions(+)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > index acf48de..cc35255 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > @@ -175,6 +175,8 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring,
> unsigned num_ibs,
> >         if (ring->funcs->emit_hdp_flush)
> >                 amdgpu_ring_emit_hdp_flush(ring);
> >
> > +       amdgpu_gfx_ring_emit_meta_data(ring, vm);
> > +
> >         /* always set cond_exec_polling to CONTINUE */
> >         *ring->cond_exe_cpu_addr = 1;
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > index dff1248..d6f57a2 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > @@ -69,4 +69,6 @@ int amd_xgpu_set_ip_blocks(struct amdgpu_device
> > *adev);
> >  /* Context Save Area functions */
> >  int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct
> amdgpu_vm
> > *vm);  void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct
> > amdgpu_vm *vm);
> > +void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
> > +                                   struct amdgpu_vm *vm);
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > index a25e07f..cc3123b 100644
> > --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > @@ -26,9 +26,48 @@
> >
> >  #include "amdgpu.h"
> >
> > +/* context save area structures */
> > +struct amdgpu_ce_ib_state {
> > +       uint32_t        ce_ib_completion_status;
> > +       uint32_t        ce_const_engine_count;
> > +       uint32_t        ce_ib_offset_ib1;
> > +       uint32_t        ce_ib_offset_ib2;
> > +};
> > +
> > +struct amdgpu_de_ib_state {
> > +       uint32_t        de_ib_completion_status;
> > +       uint32_t        de_const_engine_count;
> > +       uint32_t        de_ib_offset_ib1;
> > +       uint32_t        de_ib_offset_ib2;
> > +       uint32_t        preamble_begin_ib1;
> > +       uint32_t        preamble_begin_ib2;
> > +       uint32_t        preamble_end_ib1;
> > +       uint32_t        preamble_end_ib2;
> > +       uint32_t        draw_indirect_base_lo;
> > +       uint32_t        draw_indirect_base_hi;
> > +       uint32_t        disp_indirect_base_lo;
> > +       uint32_t        disp_indirect_base_hi;
> > +       uint32_t        gds_backup_addr_lo;
> > +       uint32_t        gds_backup_addr_hi;
> > +       uint32_t        index_base_addr_lo;
> > +       uint32_t        index_base_addr_hi;
> > +       uint32_t        sample_cntl;
> > +};
> > +
> > +struct amdgpu_gfx_meta_data {
> > +       struct amdgpu_ce_ib_state       ce_payload;
> > +       uint32_t                        reserved1[60];
> > +       struct amdgpu_de_ib_state       de_payload;
> > +       uint32_t                        de_ib_base_addr_lo;
> > +       uint32_t                        de_ib_base_addr_hi;
> > +       uint32_t                        reserved2[941];
> > +};
> > +
> 
> These are gfx8 specific and should be moved to gfx8 module.

I think it is only relate to virtualization right now. I'm fine to move it out if support whole preemption solution later.

> 
> >  /* xgpu structures */
> >  struct amd_xgpu_csa {
> >         struct amdgpu_bo            *robj;
> > +       struct amdgpu_ce_ib_state   ce_payload;
> > +       struct amdgpu_de_ib_state   de_payload;
> >         uint64_t                    gpu_addr;
> >         uint64_t                    gds_addr;
> >         int32_t                     size;
> > diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> > b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> > index 246a747..6d4246c 100644
> > --- a/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> > +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
> > @@ -207,3 +207,91 @@ void amdgpu_vm_unmap_csa(struct
> amdgpu_device *adev, struct amdgpu_vm *vm)
> >         sa = &xgpu->sa;
> >         xgpu_vm_unmap_csa(adev, vm, sa);  }
> > +
> > +static void amdgpu_ring_write_multiple(struct amdgpu_ring *ring,
> > +                                      void *src, int count_dw) {
> > +       if (ring->count_dw < count_dw)
> > +               DRM_ERROR("writing more dwords to the ring than
> expected:%d.\n",
> > +                          count_dw);
> > +       else {
> > +               unsigned int chunk1, chunk2;
> > +               void *dst = (void *)&ring->ring[ring->wptr];
> > +
> > +               chunk1 = ring->ptr_mask + 1 - ring->wptr;
> > +               chunk1 = (chunk1 >= count_dw) ? count_dw : chunk1;
> > +               chunk2 = count_dw - chunk1;
> > +               chunk1 <<= 2;
> > +               chunk2 <<= 2;
> > +               if (chunk1) {
> > +                       memcpy(dst, src, chunk1);
> > +                       dst = (void *)(((uint64_t)dst + chunk1) &
> > +                                       ring->ptr_mask);
> > +               }
> > +
> > +               if (chunk2) {
> > +                       src += chunk1;
> > +                       dst = (void *)ring->ring;
> > +                       memcpy(dst, src, chunk2);
> > +               }
> > +
> > +               ring->wptr += count_dw;
> > +               ring->wptr &= ring->ptr_mask;
> > +               ring->count_dw -= count_dw;
> > +       }
> > +}
> > +
> > +void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
> > +                                   struct amdgpu_vm *vm) {
> > +       struct amdgpu_ce_ib_state *ce_payload;
> > +       struct amdgpu_de_ib_state *de_payload;
> > +       struct amd_xgpu_csa *sa = NULL;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)ring->adev->priv_data;
> > +       uint64_t csa_addr, gds_addr;
> > +       int cnt;
> > +
> > +       if (!xgpu || (ring->funcs->type != AMDGPU_RING_TYPE_GFX))
> > +               return;
> > +
> > +       sa = &xgpu->sa;
> 
> No need to make this dependent on xgpu.  As I said in the previous patch
> preemption is useful independent of xgpu.
> 

Ditto

> > +
> > +       ce_payload = &sa->ce_payload;
> > +       de_payload = &sa->de_payload;
> > +       memset(ce_payload, 0, sizeof(*ce_payload));
> > +       memset(de_payload, 0, sizeof(*de_payload));
> > +
> > +       cnt = (sizeof(*ce_payload) >> 2) + 4 - 2;
> > +       csa_addr = vm ? vm->csa.csa_addr : sa->gpu_addr;
> > +       gds_addr = vm ? vm->csa.gds_addr : sa->gds_addr;
> > +       de_payload->gds_backup_addr_lo = lower_32_bits(gds_addr);
> > +       de_payload->gds_backup_addr_hi = upper_32_bits(gds_addr);
> > +
> > +       /* write CE meta data */
> > +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
> > +       amdgpu_ring_write(ring, WRITE_DATA_ENGINE_SEL(2) |
> > +                               WRITE_DATA_DST_SEL(8) |
> > +                               WR_CONFIRM | WRITE_DATA_CACHE_POLICY(0));
> > +       amdgpu_ring_write(ring, lower_32_bits(csa_addr +
> > +                         offsetof(struct amdgpu_gfx_meta_data, ce_payload)));
> > +       amdgpu_ring_write(ring, upper_32_bits(csa_addr +
> > +                         offsetof(struct amdgpu_gfx_meta_data,
> > + ce_payload)));
> > +
> > +       amdgpu_ring_write_multiple(ring, (void *)ce_payload,
> > +                                        sizeof(*ce_payload) >> 2);
> > +
> > +       /* write DE meta data */
> > +       cnt = (sizeof(*de_payload) >> 2) + 4 - 2;
> > +
> > +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
> > +       amdgpu_ring_write(ring, WRITE_DATA_ENGINE_SEL(1) |
> > +                               WRITE_DATA_DST_SEL(8) |
> > +                               WR_CONFIRM | WRITE_DATA_CACHE_POLICY(0));
> > +       amdgpu_ring_write(ring, lower_32_bits(csa_addr +
> > +                         offsetof(struct amdgpu_gfx_meta_data, de_payload)));
> > +       amdgpu_ring_write(ring, upper_32_bits(csa_addr +
> > +                         offsetof(struct amdgpu_gfx_meta_data,
> > + de_payload)));
> > +
> > +       amdgpu_ring_write_multiple(ring, (void *)de_payload,
> > +                                        sizeof(*de_payload) >> 2); }
> 
> This function is gfx8 specific and should be moved to the gfx8 module.
> 
Ditto

> Alex
> 
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 13/23] drm/amdgpu/mxgpu: add support for mailbox communication

[Xiangliang Yu]

GPU guest driver send mailbox messages to hyperverisor to request
full access to all of registers and release it when access is done.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
Signed-off-by: shaoyunl <Shaoyun.Liu@amd.com>
---
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h |   3 +
 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c  | 128 ++++++++++++++++++++++++++++++++++
 2 files changed, 131 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c

diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
index cc3123b..54e7b31 100644
--- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
@@ -85,4 +85,7 @@ extern void amd_xgpu_free(struct amd_xgpu *xgpu);
 
 extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);
 extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
+
+extern int xgpu_request_full_gpu_access(struct amdgpu_device *adev, bool init);
+extern int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool init);
 #endif
diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
new file mode 100644
index 0000000..b74b501
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
@@ -0,0 +1,128 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Xiangliang.Yu@amd.com
+ */
+#include "amd_mxgpu.h"
+#include "bif/bif_5_0_d.h"
+#include "bif/bif_5_0_sh_mask.h"
+
+/* mailbox messages request */
+enum idh_request {
+	IDH_REQ_GPU_INIT_ACCESS	= 1,
+	IDH_REL_GPU_INIT_ACCESS,
+	IDH_REQ_GPU_FINI_ACCESS,
+	IDH_REL_GPU_FINI_ACCESS,
+	IDH_REQ_GPU_RESET_ACCESS
+};
+
+/* mailbox messages data */
+enum idh_event {
+	IDH_CLR_MSG_BUF = 0,
+	IDH_READY_TO_ACCESS_GPU,
+	IDH_FLR_NOTIFICATION,
+	IDH_FLR_NOTIFICATION_CMPL
+};
+
+/* mailbox state */
+enum mail_state {
+	AMDGPU_START_INIT = 1,
+	AMDGPU_END_INIT,
+	AMDGPU_START_FINI,
+	AMDGPU_END_FINI,
+};
+
+/*
+ * Mailbox communicate between PF and VF
+ */
+static int xgpu_request_reg_access(struct amdgpu_device *adev,
+				   enum mail_state state)
+{
+	u32 temp, timeout = 150000;
+	u32 req = IDH_REQ_GPU_INIT_ACCESS;
+
+	if (state == AMDGPU_START_FINI)
+		req = IDH_REQ_GPU_FINI_ACCESS;
+	else if (state != AMDGPU_START_INIT)
+		return -EINVAL;
+
+	WREG32(mmMAILBOX_MSGBUF_TRN_DW0, req);
+	WREG32(mmMAILBOX_CONTROL,
+			REG_SET_FIELD(RREG32(mmMAILBOX_CONTROL),
+			MAILBOX_CONTROL, TRN_MSG_VALID,	1));
+
+	/* wait respond from PF */
+	while (true) {
+		temp = RREG32(mmMAILBOX_MSGBUF_RCV_DW0);
+		if (temp == IDH_READY_TO_ACCESS_GPU)
+			break;
+		if (timeout == 0) {
+			pr_err("amdgpu: VF request GPU access failed(0x%x) \
+				in state %d\n", temp, state);
+			return -ETIME;
+		}
+		msleep(20);
+		timeout -= 20;
+	}
+	return 0;
+}
+
+static int xgpu_release_reg_access(struct amdgpu_device *adev,
+				   enum mail_state state)
+{
+	u32 rel = IDH_REL_GPU_INIT_ACCESS;
+
+	if (state == AMDGPU_END_FINI)
+		rel = IDH_REL_GPU_FINI_ACCESS;
+	else if (state != AMDGPU_END_INIT)
+		return -EINVAL;
+
+	WREG32(mmMAILBOX_MSGBUF_TRN_DW0, rel);
+	WREG32(mmMAILBOX_CONTROL,
+		REG_SET_FIELD(RREG32(mmMAILBOX_CONTROL),
+			MAILBOX_CONTROL, TRN_MSG_VALID,	1));
+
+	return 0;
+}
+
+int xgpu_request_full_gpu_access(struct amdgpu_device *adev, bool init)
+{
+	int r;
+
+	if (init)
+		r = xgpu_request_reg_access(adev, AMDGPU_START_INIT);
+	else
+		r = xgpu_request_reg_access(adev, AMDGPU_START_FINI);
+
+	return r;
+}
+
+int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool init)
+{
+	int r;
+
+	if (init)
+		r = xgpu_release_reg_access(adev, AMDGPU_END_INIT);
+	else
+		r = xgpu_release_reg_access(adev, AMDGPU_END_FINI);
+
+	return r;
+}
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 13/23] drm/amdgpu/mxgpu: add support for mailbox communication

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> GPU guest driver send mailbox messages to hyperverisor to request
> full access to all of registers and release it when access is done.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> Signed-off-by: shaoyunl <Shaoyun.Liu@amd.com>
> ---
>  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h |   3 +
>  drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c  | 128 ++++++++++++++++++++++++++++++++++
>  2 files changed, 131 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
>
> diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> index cc3123b..54e7b31 100644
> --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> @@ -85,4 +85,7 @@ extern void amd_xgpu_free(struct amd_xgpu *xgpu);
>
>  extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);
>  extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
> +
> +extern int xgpu_request_full_gpu_access(struct amdgpu_device *adev, bool init);
> +extern int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool init);
>  #endif
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> new file mode 100644
> index 0000000..b74b501
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> @@ -0,0 +1,128 @@
> +/*
> + * Copyright 2016 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + * Authors: Xiangliang.Yu@amd.com
> + */
> +#include "amd_mxgpu.h"
> +#include "bif/bif_5_0_d.h"
> +#include "bif/bif_5_0_sh_mask.h"
> +
> +/* mailbox messages request */
> +enum idh_request {
> +       IDH_REQ_GPU_INIT_ACCESS = 1,
> +       IDH_REL_GPU_INIT_ACCESS,
> +       IDH_REQ_GPU_FINI_ACCESS,
> +       IDH_REL_GPU_FINI_ACCESS,
> +       IDH_REQ_GPU_RESET_ACCESS
> +};
> +
> +/* mailbox messages data */
> +enum idh_event {
> +       IDH_CLR_MSG_BUF = 0,
> +       IDH_READY_TO_ACCESS_GPU,
> +       IDH_FLR_NOTIFICATION,
> +       IDH_FLR_NOTIFICATION_CMPL
> +};
> +
> +/* mailbox state */
> +enum mail_state {
> +       AMDGPU_START_INIT = 1,
> +       AMDGPU_END_INIT,
> +       AMDGPU_START_FINI,
> +       AMDGPU_END_FINI,
> +};
> +
> +/*
> + * Mailbox communicate between PF and VF
> + */
> +static int xgpu_request_reg_access(struct amdgpu_device *adev,
> +                                  enum mail_state state)
> +{
> +       u32 temp, timeout = 150000;
> +       u32 req = IDH_REQ_GPU_INIT_ACCESS;
> +
> +       if (state == AMDGPU_START_FINI)
> +               req = IDH_REQ_GPU_FINI_ACCESS;
> +       else if (state != AMDGPU_START_INIT)
> +               return -EINVAL;
> +
> +       WREG32(mmMAILBOX_MSGBUF_TRN_DW0, req);
> +       WREG32(mmMAILBOX_CONTROL,
> +                       REG_SET_FIELD(RREG32(mmMAILBOX_CONTROL),
> +                       MAILBOX_CONTROL, TRN_MSG_VALID, 1));
> +
> +       /* wait respond from PF */
> +       while (true) {
> +               temp = RREG32(mmMAILBOX_MSGBUF_RCV_DW0);
> +               if (temp == IDH_READY_TO_ACCESS_GPU)
> +                       break;
> +               if (timeout == 0) {
> +                       pr_err("amdgpu: VF request GPU access failed(0x%x) \
> +                               in state %d\n", temp, state);
> +                       return -ETIME;
> +               }
> +               msleep(20);
> +               timeout -= 20;
> +       }
> +       return 0;
> +}
> +
> +static int xgpu_release_reg_access(struct amdgpu_device *adev,
> +                                  enum mail_state state)
> +{
> +       u32 rel = IDH_REL_GPU_INIT_ACCESS;
> +
> +       if (state == AMDGPU_END_FINI)
> +               rel = IDH_REL_GPU_FINI_ACCESS;
> +       else if (state != AMDGPU_END_INIT)
> +               return -EINVAL;
> +
> +       WREG32(mmMAILBOX_MSGBUF_TRN_DW0, rel);
> +       WREG32(mmMAILBOX_CONTROL,
> +               REG_SET_FIELD(RREG32(mmMAILBOX_CONTROL),
> +                       MAILBOX_CONTROL, TRN_MSG_VALID, 1));
> +
> +       return 0;
> +}
> +

These are gfx8 specific.  they should be handled in a gfx8 specific
code.  mxgpu_mb.c should only contain asic independent code.

Alex


> +int xgpu_request_full_gpu_access(struct amdgpu_device *adev, bool init)
> +{
> +       int r;
> +
> +       if (init)
> +               r = xgpu_request_reg_access(adev, AMDGPU_START_INIT);
> +       else
> +               r = xgpu_request_reg_access(adev, AMDGPU_START_FINI);
> +
> +       return r;
> +}
> +
> +int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool init)
> +{
> +       int r;
> +
> +       if (init)
> +               r = xgpu_release_reg_access(adev, AMDGPU_END_INIT);
> +       else
> +               r = xgpu_release_reg_access(adev, AMDGPU_END_FINI);
> +
> +       return r;
> +}
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 13/23] drm/amdgpu/mxgpu: add support for mailbox communication

[Yu, Xiangliang]

> -----Original Message-----
> From: Alex Deucher [mailto:alexdeucher@gmail.com]
> Sent: Tuesday, December 20, 2016 7:25 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> dl.SRDC_SW_GPUVirtualization
> <dl.SRDC_SW_GPUVirtualization@amd.com>; Liu, Shaoyun
> <Shaoyun.Liu@amd.com>
> Subject: Re: [PATCH 13/23] drm/amdgpu/mxgpu: add support for mailbox
> communication
> 
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> wrote:
> > GPU guest driver send mailbox messages to hyperverisor to request full
> > access to all of registers and release it when access is done.
> >
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > Signed-off-by: shaoyunl <Shaoyun.Liu@amd.com>
> > ---
> >  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h |   3 +
> >  drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c  | 128
> > ++++++++++++++++++++++++++++++++++
> >  2 files changed, 131 insertions(+)
> >  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> >
> > diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > index cc3123b..54e7b31 100644
> > --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > @@ -85,4 +85,7 @@ extern void amd_xgpu_free(struct amd_xgpu *xgpu);
> >
> >  extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);  extern void
> > xgpu_destroy_csa(struct amd_xgpu_csa *csa);
> > +
> > +extern int xgpu_request_full_gpu_access(struct amdgpu_device *adev,
> > +bool init); extern int xgpu_release_full_gpu_access(struct
> > +amdgpu_device *adev, bool init);
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> > b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> > new file mode 100644
> > index 0000000..b74b501
> > --- /dev/null
> > +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> > @@ -0,0 +1,128 @@
> > +/*
> > + * Copyright 2016 Advanced Micro Devices, Inc.
> > + *
> > + * Permission is hereby granted, free of charge, to any person
> > +obtaining a
> > + * copy of this software and associated documentation files (the
> > +"Software"),
> > + * to deal in the Software without restriction, including without
> > +limitation
> > + * the rights to use, copy, modify, merge, publish, distribute,
> > +sublicense,
> > + * and/or sell copies of the Software, and to permit persons to whom
> > +the
> > + * Software is furnished to do so, subject to the following conditions:
> > + *
> > + * The above copyright notice and this permission notice shall be
> > +included in
> > + * all copies or substantial portions of the Software.
> > + *
> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
> KIND,
> > +EXPRESS OR
> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > +MERCHANTABILITY,
> > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN
> NO EVENT
> > +SHALL
> > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
> > +DAMAGES OR
> > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
> > +OTHERWISE,
> > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
> THE USE
> > +OR
> > + * OTHER DEALINGS IN THE SOFTWARE.
> > + *
> > + * Authors: Xiangliang.Yu@amd.com
> > + */
> > +#include "amd_mxgpu.h"
> > +#include "bif/bif_5_0_d.h"
> > +#include "bif/bif_5_0_sh_mask.h"
> > +
> > +/* mailbox messages request */
> > +enum idh_request {
> > +       IDH_REQ_GPU_INIT_ACCESS = 1,
> > +       IDH_REL_GPU_INIT_ACCESS,
> > +       IDH_REQ_GPU_FINI_ACCESS,
> > +       IDH_REL_GPU_FINI_ACCESS,
> > +       IDH_REQ_GPU_RESET_ACCESS
> > +};
> > +
> > +/* mailbox messages data */
> > +enum idh_event {
> > +       IDH_CLR_MSG_BUF = 0,
> > +       IDH_READY_TO_ACCESS_GPU,
> > +       IDH_FLR_NOTIFICATION,
> > +       IDH_FLR_NOTIFICATION_CMPL
> > +};
> > +
> > +/* mailbox state */
> > +enum mail_state {
> > +       AMDGPU_START_INIT = 1,
> > +       AMDGPU_END_INIT,
> > +       AMDGPU_START_FINI,
> > +       AMDGPU_END_FINI,
> > +};
> > +
> > +/*
> > + * Mailbox communicate between PF and VF  */ static int
> > +xgpu_request_reg_access(struct amdgpu_device *adev,
> > +                                  enum mail_state state) {
> > +       u32 temp, timeout = 150000;
> > +       u32 req = IDH_REQ_GPU_INIT_ACCESS;
> > +
> > +       if (state == AMDGPU_START_FINI)
> > +               req = IDH_REQ_GPU_FINI_ACCESS;
> > +       else if (state != AMDGPU_START_INIT)
> > +               return -EINVAL;
> > +
> > +       WREG32(mmMAILBOX_MSGBUF_TRN_DW0, req);
> > +       WREG32(mmMAILBOX_CONTROL,
> > +                       REG_SET_FIELD(RREG32(mmMAILBOX_CONTROL),
> > +                       MAILBOX_CONTROL, TRN_MSG_VALID, 1));
> > +
> > +       /* wait respond from PF */
> > +       while (true) {
> > +               temp = RREG32(mmMAILBOX_MSGBUF_RCV_DW0);
> > +               if (temp == IDH_READY_TO_ACCESS_GPU)
> > +                       break;
> > +               if (timeout == 0) {
> > +                       pr_err("amdgpu: VF request GPU access failed(0x%x) \
> > +                               in state %d\n", temp, state);
> > +                       return -ETIME;
> > +               }
> > +               msleep(20);
> > +               timeout -= 20;
> > +       }
> > +       return 0;
> > +}
> > +
> > +static int xgpu_release_reg_access(struct amdgpu_device *adev,
> > +                                  enum mail_state state) {
> > +       u32 rel = IDH_REL_GPU_INIT_ACCESS;
> > +
> > +       if (state == AMDGPU_END_FINI)
> > +               rel = IDH_REL_GPU_FINI_ACCESS;
> > +       else if (state != AMDGPU_END_INIT)
> > +               return -EINVAL;
> > +
> > +       WREG32(mmMAILBOX_MSGBUF_TRN_DW0, rel);
> > +       WREG32(mmMAILBOX_CONTROL,
> > +               REG_SET_FIELD(RREG32(mmMAILBOX_CONTROL),
> > +                       MAILBOX_CONTROL, TRN_MSG_VALID, 1));
> > +
> > +       return 0;
> > +}
> > +
> 
> These are gfx8 specific.  they should be handled in a gfx8 specific code.
> mxgpu_mb.c should only contain asic independent code.

I don't think so. The mailbox is only used by virtualization, and I'd like to put it into virtualization component.  

As I said in cover letter, the main aim of the series is put all virtualization code together and manage different virtualization chips and features. Do not want to mix it into amdgpu components. Otherwise the series will lost meaning.

> 
> Alex
> 
> 
> > +int xgpu_request_full_gpu_access(struct amdgpu_device *adev, bool
> > +init) {
> > +       int r;
> > +
> > +       if (init)
> > +               r = xgpu_request_reg_access(adev, AMDGPU_START_INIT);
> > +       else
> > +               r = xgpu_request_reg_access(adev, AMDGPU_START_FINI);
> > +
> > +       return r;
> > +}
> > +
> > +int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool
> > +init) {
> > +       int r;
> > +
> > +       if (init)
> > +               r = xgpu_release_reg_access(adev, AMDGPU_END_INIT);
> > +       else
> > +               r = xgpu_release_reg_access(adev, AMDGPU_END_FINI);
> > +
> > +       return r;
> > +}
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 13/23] drm/amdgpu/mxgpu: add support for mailbox communication

[Liu, Shaoyun]

Hi, 
I didn't check others changes in the  series.  For these two specific changes regarding  the mailbox communication , I have  two concerns 
1. This code should only works on SRIOV enabled VF , in your code seems you use  adev->priv_data  for the check , I would like to have  some more meaningful name than priv_data if possible . 
2. How you handle the  asic specific  MAILBOX programming ?   ex, for Vega10 support . 

Regards
Shaoyun.liu

-----Original Message-----
From: Yu, Xiangliang 
Sent: Tuesday, December 20, 2016 1:06 AM
To: Alex Deucher
Cc: amd-gfx list; dl.SRDC_SW_GPUVirtualization; Liu, Shaoyun
Subject: RE: [PATCH 13/23] drm/amdgpu/mxgpu: add support for mailbox communication


> -----Original Message-----
> From: Alex Deucher [mailto:alexdeucher@gmail.com]
> Sent: Tuesday, December 20, 2016 7:25 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>; 
> dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>; 
> Liu, Shaoyun <Shaoyun.Liu@amd.com>
> Subject: Re: [PATCH 13/23] drm/amdgpu/mxgpu: add support for mailbox 
> communication
> 
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu 
> <Xiangliang.Yu@amd.com>
> wrote:
> > GPU guest driver send mailbox messages to hyperverisor to request 
> > full access to all of registers and release it when access is done.
> >
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > Signed-off-by: shaoyunl <Shaoyun.Liu@amd.com>
> > ---
> >  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h |   3 +
> >  drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c  | 128
> > ++++++++++++++++++++++++++++++++++
> >  2 files changed, 131 insertions(+)
> >  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> >
> > diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > index cc3123b..54e7b31 100644
> > --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > @@ -85,4 +85,7 @@ extern void amd_xgpu_free(struct amd_xgpu *xgpu);
> >
> >  extern int xgpu_allocate_csa(struct amd_xgpu *xgpu);  extern void 
> > xgpu_destroy_csa(struct amd_xgpu_csa *csa);
> > +
> > +extern int xgpu_request_full_gpu_access(struct amdgpu_device *adev, 
> > +bool init); extern int xgpu_release_full_gpu_access(struct
> > +amdgpu_device *adev, bool init);
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> > b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> > new file mode 100644
> > index 0000000..b74b501
> > --- /dev/null
> > +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
> > @@ -0,0 +1,128 @@
> > +/*
> > + * Copyright 2016 Advanced Micro Devices, Inc.
> > + *
> > + * Permission is hereby granted, free of charge, to any person 
> > +obtaining a
> > + * copy of this software and associated documentation files (the 
> > +"Software"),
> > + * to deal in the Software without restriction, including without 
> > +limitation
> > + * the rights to use, copy, modify, merge, publish, distribute, 
> > +sublicense,
> > + * and/or sell copies of the Software, and to permit persons to 
> > +whom the
> > + * Software is furnished to do so, subject to the following conditions:
> > + *
> > + * The above copyright notice and this permission notice shall be 
> > +included in
> > + * all copies or substantial portions of the Software.
> > + *
> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
> KIND,
> > +EXPRESS OR
> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
> > +MERCHANTABILITY,
> > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN
> NO EVENT
> > +SHALL
> > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, 
> > +DAMAGES OR
> > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 
> > +OTHERWISE,
> > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
> THE USE
> > +OR
> > + * OTHER DEALINGS IN THE SOFTWARE.
> > + *
> > + * Authors: Xiangliang.Yu@amd.com
> > + */
> > +#include "amd_mxgpu.h"
> > +#include "bif/bif_5_0_d.h"
> > +#include "bif/bif_5_0_sh_mask.h"
> > +
> > +/* mailbox messages request */
> > +enum idh_request {
> > +       IDH_REQ_GPU_INIT_ACCESS = 1,
> > +       IDH_REL_GPU_INIT_ACCESS,
> > +       IDH_REQ_GPU_FINI_ACCESS,
> > +       IDH_REL_GPU_FINI_ACCESS,
> > +       IDH_REQ_GPU_RESET_ACCESS
> > +};
> > +
> > +/* mailbox messages data */
> > +enum idh_event {
> > +       IDH_CLR_MSG_BUF = 0,
> > +       IDH_READY_TO_ACCESS_GPU,
> > +       IDH_FLR_NOTIFICATION,
> > +       IDH_FLR_NOTIFICATION_CMPL
> > +};
> > +
> > +/* mailbox state */
> > +enum mail_state {
> > +       AMDGPU_START_INIT = 1,
> > +       AMDGPU_END_INIT,
> > +       AMDGPU_START_FINI,
> > +       AMDGPU_END_FINI,
> > +};
> > +
> > +/*
> > + * Mailbox communicate between PF and VF  */ static int 
> > +xgpu_request_reg_access(struct amdgpu_device *adev,
> > +                                  enum mail_state state) {
> > +       u32 temp, timeout = 150000;
> > +       u32 req = IDH_REQ_GPU_INIT_ACCESS;
> > +
> > +       if (state == AMDGPU_START_FINI)
> > +               req = IDH_REQ_GPU_FINI_ACCESS;
> > +       else if (state != AMDGPU_START_INIT)
> > +               return -EINVAL;
> > +
> > +       WREG32(mmMAILBOX_MSGBUF_TRN_DW0, req);
> > +       WREG32(mmMAILBOX_CONTROL,
> > +                       REG_SET_FIELD(RREG32(mmMAILBOX_CONTROL),
> > +                       MAILBOX_CONTROL, TRN_MSG_VALID, 1));
> > +
> > +       /* wait respond from PF */
> > +       while (true) {
> > +               temp = RREG32(mmMAILBOX_MSGBUF_RCV_DW0);
> > +               if (temp == IDH_READY_TO_ACCESS_GPU)
> > +                       break;
> > +               if (timeout == 0) {
> > +                       pr_err("amdgpu: VF request GPU access failed(0x%x) \
> > +                               in state %d\n", temp, state);
> > +                       return -ETIME;
> > +               }
> > +               msleep(20);
> > +               timeout -= 20;
> > +       }
> > +       return 0;
> > +}
> > +
> > +static int xgpu_release_reg_access(struct amdgpu_device *adev,
> > +                                  enum mail_state state) {
> > +       u32 rel = IDH_REL_GPU_INIT_ACCESS;
> > +
> > +       if (state == AMDGPU_END_FINI)
> > +               rel = IDH_REL_GPU_FINI_ACCESS;
> > +       else if (state != AMDGPU_END_INIT)
> > +               return -EINVAL;
> > +
> > +       WREG32(mmMAILBOX_MSGBUF_TRN_DW0, rel);
> > +       WREG32(mmMAILBOX_CONTROL,
> > +               REG_SET_FIELD(RREG32(mmMAILBOX_CONTROL),
> > +                       MAILBOX_CONTROL, TRN_MSG_VALID, 1));
> > +
> > +       return 0;
> > +}
> > +
> 
> These are gfx8 specific.  they should be handled in a gfx8 specific code.
> mxgpu_mb.c should only contain asic independent code.

I don't think so. The mailbox is only used by virtualization, and I'd like to put it into virtualization component.  

As I said in cover letter, the main aim of the series is put all virtualization code together and manage different virtualization chips and features. Do not want to mix it into amdgpu components. Otherwise the series will lost meaning.

> 
> Alex
> 
> 
> > +int xgpu_request_full_gpu_access(struct amdgpu_device *adev, bool
> > +init) {
> > +       int r;
> > +
> > +       if (init)
> > +               r = xgpu_request_reg_access(adev, AMDGPU_START_INIT);
> > +       else
> > +               r = xgpu_request_reg_access(adev, 
> > + AMDGPU_START_FINI);
> > +
> > +       return r;
> > +}
> > +
> > +int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool
> > +init) {
> > +       int r;
> > +
> > +       if (init)
> > +               r = xgpu_release_reg_access(adev, AMDGPU_END_INIT);
> > +       else
> > +               r = xgpu_release_reg_access(adev, AMDGPU_END_FINI);
> > +
> > +       return r;
> > +}
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 14/23] drm/amdgpu: export two mailbox interface to amdgpu

[Xiangliang Yu]

KFD also will access GPU registers when loading/unloading drivers,
so can't hide the mailbox in mxgpu. Right now, can't find any way
to put it into kfd interface, export it into amdgpu.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
Signed-off-by: shaoyunl <Shaoyun.Liu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c  |  4 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  4 ++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c     | 28 ++++++++++++++++++++++++++++
 3 files changed, 36 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
index 3273d8c..1f152f2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
@@ -60,6 +60,8 @@ int amdgpu_driver_unload_kms(struct drm_device *dev)
 	if (adev->rmmio == NULL)
 		goto done_free;
 
+	amdgpu_get_gpu(adev);
+
 	if (amdgpu_device_is_px(dev)) {
 		pm_runtime_get_sync(dev->dev);
 		pm_runtime_forbid(dev->dev);
@@ -139,6 +141,8 @@ int amdgpu_driver_load_kms(struct drm_device *dev, unsigned long flags)
 		pm_runtime_put_autosuspend(dev->dev);
 	}
 
+	amdgpu_put_gpu(adev);
+
 out:
 	if (r) {
 		/* balance pm_runtime_get_sync in amdgpu_driver_unload_kms */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
index d6f57a2..97aae31 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
@@ -71,4 +71,8 @@ int amdgpu_vm_map_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
 void amdgpu_vm_unmap_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
 void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
 				    struct amdgpu_vm *vm);
+
+/* get full gpu access */
+int amdgpu_get_gpu(struct amdgpu_device *adev);
+int amdgpu_put_gpu(struct amdgpu_device *adev);
 #endif
diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
index b74b501..387246c 100644
--- a/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
+++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
@@ -126,3 +126,31 @@ int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool init)
 
 	return r;
 }
+
+int amdgpu_get_gpu(struct amdgpu_device *adev)
+{
+	int r = 0;
+
+	if (!adev->priv_data)
+		return 0;
+
+	r = xgpu_request_full_gpu_access(adev, false);
+	if (r)
+		return r;
+
+	return 0;
+}
+
+int amdgpu_put_gpu(struct amdgpu_device *adev)
+{
+	int r = 0;
+
+	if (!adev->priv_data)
+		return 0;
+
+	r = xgpu_release_full_gpu_access(adev, true);
+	if (r)
+		return r;
+
+	return 0;
+}
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 15/23] drm/amdgpu/mxgpu: implement register access function with KIQ

[Xiangliang Yu]

One of important role of KIQ is provide one way to access VF
registers. This patch implement the feature and export interfaces.

Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   4 ++
 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c    | 110 +++++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
index 97aae31..c78435d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
@@ -75,4 +75,8 @@ void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
 /* get full gpu access */
 int amdgpu_get_gpu(struct amdgpu_device *adev);
 int amdgpu_put_gpu(struct amdgpu_device *adev);
+
+/* access vf registers */
+uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
+void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
 #endif
diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
new file mode 100644
index 0000000..6e1b8e3
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
@@ -0,0 +1,110 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Xiangliang.Yu@amd.com
+ *	    Monk.Liu@amd.com
+ */
+#include "amd_mxgpu.h"
+#include "vid.h"
+#include "gca/gfx_8_0_d.h"
+#include "gca/gfx_8_0_sh_mask.h"
+
+static void xgpu_kiq_ring_emit_rreg(struct amdgpu_ring *ring, u32 idx)
+{
+	struct amdgpu_device *adev = ring->adev;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+
+	amdgpu_ring_write(ring, PACKET3(PACKET3_COPY_DATA, 4));
+	amdgpu_ring_write(ring, 0 |	/* src: register*/
+				(5 << 8) |	/* dst: memory */
+				(1 << 20));	/* write confirm */
+	amdgpu_ring_write(ring, idx);
+	amdgpu_ring_write(ring, 0);
+	amdgpu_ring_write(ring, lower_32_bits(adev->wb.gpu_addr +
+				xgpu->reg_val_offs * 4));
+	amdgpu_ring_write(ring, upper_32_bits(adev->wb.gpu_addr +
+				xgpu->reg_val_offs * 4));
+}
+
+static void xgpu_kiq_ring_emit_wreg(struct amdgpu_ring *ring, u32 idx, u32 val)
+{
+	amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
+	amdgpu_ring_write(ring, (1 << 16)); /* no inc addr */
+	amdgpu_ring_write(ring, idx);
+	amdgpu_ring_write(ring, 0);
+	amdgpu_ring_write(ring, val);
+}
+
+uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg)
+{
+	signed long r;
+	uint32_t val;
+	struct fence *f;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+	struct amdgpu_kiq *kiq = &adev->kiq;
+
+	if (in_interrupt())
+		return 0;
+
+	mutex_lock(&xgpu->lock);
+	amdgpu_ring_alloc(&kiq->ring, 32);
+	amdgpu_ring_emit_hdp_flush(&kiq->ring);
+	xgpu_kiq_ring_emit_rreg(&kiq->ring, reg);
+	amdgpu_ring_emit_hdp_invalidate(&kiq->ring);
+	amdgpu_fence_emit(&kiq->ring, &f);
+	fence_get(f);
+	amdgpu_ring_commit(&kiq->ring);
+	mutex_unlock(&xgpu->lock);
+
+	r = fence_wait(f, false);
+	fence_put(f);
+	if (r)
+		DRM_ERROR("wait for kiq fence error: %ld.\n", r);
+	fence_put(f);
+
+	val = adev->wb.wb[xgpu->reg_val_offs];
+
+	return val;
+}
+
+void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v)
+{
+	signed long r;
+	struct fence *f;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+	struct amdgpu_kiq *kiq = &adev->kiq;
+
+	mutex_lock(&xgpu->lock);
+	amdgpu_ring_alloc(&kiq->ring, 32);
+	amdgpu_ring_emit_hdp_flush(&kiq->ring);
+	xgpu_kiq_ring_emit_wreg(&kiq->ring, reg, v);
+	amdgpu_ring_emit_hdp_invalidate(&kiq->ring);
+	amdgpu_fence_emit(&kiq->ring, &f);
+	fence_get(f);
+	amdgpu_ring_commit(&kiq->ring);
+	mutex_unlock(&xgpu->lock);
+
+	r = fence_wait(f, false);
+	fence_put(f);
+	if (r)
+		DRM_ERROR("wait for kiq fence error: %ld.\n", r);
+	fence_put(f);
+}
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 15/23] drm/amdgpu/mxgpu: implement register access function with KIQ

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> One of important role of KIQ is provide one way to access VF
> registers. This patch implement the feature and export interfaces.
>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   4 ++
>  drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c    | 110 +++++++++++++++++++++++++++++++
>  2 files changed, 114 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index 97aae31..c78435d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -75,4 +75,8 @@ void amdgpu_gfx_ring_emit_meta_data(struct amdgpu_ring *ring,
>  /* get full gpu access */
>  int amdgpu_get_gpu(struct amdgpu_device *adev);
>  int amdgpu_put_gpu(struct amdgpu_device *adev);
> +
> +/* access vf registers */
> +uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
> +void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
>  #endif
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> new file mode 100644
> index 0000000..6e1b8e3
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> @@ -0,0 +1,110 @@
> +/*
> + * Copyright 2016 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + * Authors: Xiangliang.Yu@amd.com
> + *         Monk.Liu@amd.com
> + */
> +#include "amd_mxgpu.h"
> +#include "vid.h"
> +#include "gca/gfx_8_0_d.h"
> +#include "gca/gfx_8_0_sh_mask.h"
> +
> +static void xgpu_kiq_ring_emit_rreg(struct amdgpu_ring *ring, u32 idx)
> +{
> +       struct amdgpu_device *adev = ring->adev;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +
> +       amdgpu_ring_write(ring, PACKET3(PACKET3_COPY_DATA, 4));
> +       amdgpu_ring_write(ring, 0 |     /* src: register*/
> +                               (5 << 8) |      /* dst: memory */
> +                               (1 << 20));     /* write confirm */
> +       amdgpu_ring_write(ring, idx);
> +       amdgpu_ring_write(ring, 0);
> +       amdgpu_ring_write(ring, lower_32_bits(adev->wb.gpu_addr +
> +                               xgpu->reg_val_offs * 4));
> +       amdgpu_ring_write(ring, upper_32_bits(adev->wb.gpu_addr +
> +                               xgpu->reg_val_offs * 4));
> +}
> +
> +static void xgpu_kiq_ring_emit_wreg(struct amdgpu_ring *ring, u32 idx, u32 val)
> +{
> +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
> +       amdgpu_ring_write(ring, (1 << 16)); /* no inc addr */
> +       amdgpu_ring_write(ring, idx);
> +       amdgpu_ring_write(ring, 0);
> +       amdgpu_ring_write(ring, val);
> +}

These should be in gfx8.

Alex

> +
> +uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg)
> +{
> +       signed long r;
> +       uint32_t val;
> +       struct fence *f;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       struct amdgpu_kiq *kiq = &adev->kiq;
> +
> +       if (in_interrupt())
> +               return 0;
> +
> +       mutex_lock(&xgpu->lock);
> +       amdgpu_ring_alloc(&kiq->ring, 32);
> +       amdgpu_ring_emit_hdp_flush(&kiq->ring);
> +       xgpu_kiq_ring_emit_rreg(&kiq->ring, reg);
> +       amdgpu_ring_emit_hdp_invalidate(&kiq->ring);
> +       amdgpu_fence_emit(&kiq->ring, &f);
> +       fence_get(f);
> +       amdgpu_ring_commit(&kiq->ring);
> +       mutex_unlock(&xgpu->lock);
> +
> +       r = fence_wait(f, false);
> +       fence_put(f);
> +       if (r)
> +               DRM_ERROR("wait for kiq fence error: %ld.\n", r);
> +       fence_put(f);
> +
> +       val = adev->wb.wb[xgpu->reg_val_offs];
> +
> +       return val;
> +}
> +
> +void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v)
> +{
> +       signed long r;
> +       struct fence *f;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       struct amdgpu_kiq *kiq = &adev->kiq;
> +
> +       mutex_lock(&xgpu->lock);
> +       amdgpu_ring_alloc(&kiq->ring, 32);
> +       amdgpu_ring_emit_hdp_flush(&kiq->ring);
> +       xgpu_kiq_ring_emit_wreg(&kiq->ring, reg, v);
> +       amdgpu_ring_emit_hdp_invalidate(&kiq->ring);
> +       amdgpu_fence_emit(&kiq->ring, &f);
> +       fence_get(f);
> +       amdgpu_ring_commit(&kiq->ring);
> +       mutex_unlock(&xgpu->lock);
> +
> +       r = fence_wait(f, false);
> +       fence_put(f);
> +       if (r)
> +               DRM_ERROR("wait for kiq fence error: %ld.\n", r);
> +       fence_put(f);
> +}
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 15/23] drm/amdgpu/mxgpu: implement register access function with KIQ

[Yu, Xiangliang]

> -----Original Message-----
> From: Alex Deucher [mailto:alexdeucher@gmail.com]
> Sent: Tuesday, December 20, 2016 7:27 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> dl.SRDC_SW_GPUVirtualization
> <dl.SRDC_SW_GPUVirtualization@amd.com>; Liu, Monk
> <Monk.Liu@amd.com>
> Subject: Re: [PATCH 15/23] drm/amdgpu/mxgpu: implement register access
> function with KIQ
> 
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> wrote:
> > One of important role of KIQ is provide one way to access VF
> > registers. This patch implement the feature and export interfaces.
> >
> > Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   4 ++
> >  drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c    | 110
> +++++++++++++++++++++++++++++++
> >  2 files changed, 114 insertions(+)
> >  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > index 97aae31..c78435d 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > @@ -75,4 +75,8 @@ void amdgpu_gfx_ring_emit_meta_data(struct
> > amdgpu_ring *ring,
> >  /* get full gpu access */
> >  int amdgpu_get_gpu(struct amdgpu_device *adev);  int
> > amdgpu_put_gpu(struct amdgpu_device *adev);
> > +
> > +/* access vf registers */
> > +uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
> > +void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg,
> > +uint32_t v);
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> > b/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> > new file mode 100644
> > index 0000000..6e1b8e3
> > --- /dev/null
> > +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> > @@ -0,0 +1,110 @@
> > +/*
> > + * Copyright 2016 Advanced Micro Devices, Inc.
> > + *
> > + * Permission is hereby granted, free of charge, to any person
> > +obtaining a
> > + * copy of this software and associated documentation files (the
> > +"Software"),
> > + * to deal in the Software without restriction, including without
> > +limitation
> > + * the rights to use, copy, modify, merge, publish, distribute,
> > +sublicense,
> > + * and/or sell copies of the Software, and to permit persons to whom
> > +the
> > + * Software is furnished to do so, subject to the following conditions:
> > + *
> > + * The above copyright notice and this permission notice shall be
> > +included in
> > + * all copies or substantial portions of the Software.
> > + *
> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
> KIND,
> > +EXPRESS OR
> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > +MERCHANTABILITY,
> > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN
> NO EVENT
> > +SHALL
> > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
> > +DAMAGES OR
> > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
> > +OTHERWISE,
> > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
> THE USE
> > +OR
> > + * OTHER DEALINGS IN THE SOFTWARE.
> > + *
> > + * Authors: Xiangliang.Yu@amd.com
> > + *         Monk.Liu@amd.com
> > + */
> > +#include "amd_mxgpu.h"
> > +#include "vid.h"
> > +#include "gca/gfx_8_0_d.h"
> > +#include "gca/gfx_8_0_sh_mask.h"
> > +
> > +static void xgpu_kiq_ring_emit_rreg(struct amdgpu_ring *ring, u32
> > +idx) {
> > +       struct amdgpu_device *adev = ring->adev;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +
> > +       amdgpu_ring_write(ring, PACKET3(PACKET3_COPY_DATA, 4));
> > +       amdgpu_ring_write(ring, 0 |     /* src: register*/
> > +                               (5 << 8) |      /* dst: memory */
> > +                               (1 << 20));     /* write confirm */
> > +       amdgpu_ring_write(ring, idx);
> > +       amdgpu_ring_write(ring, 0);
> > +       amdgpu_ring_write(ring, lower_32_bits(adev->wb.gpu_addr +
> > +                               xgpu->reg_val_offs * 4));
> > +       amdgpu_ring_write(ring, upper_32_bits(adev->wb.gpu_addr +
> > +                               xgpu->reg_val_offs * 4)); }
> > +
> > +static void xgpu_kiq_ring_emit_wreg(struct amdgpu_ring *ring, u32
> > +idx, u32 val) {
> > +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
> > +       amdgpu_ring_write(ring, (1 << 16)); /* no inc addr */
> > +       amdgpu_ring_write(ring, idx);
> > +       amdgpu_ring_write(ring, 0);
> > +       amdgpu_ring_write(ring, val);
> > +}
> 
> These should be in gfx8.

This is only for virtualization,  please see my previous patch's comment.

> 
> Alex
> 
> > +
> > +uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg) {
> > +       signed long r;
> > +       uint32_t val;
> > +       struct fence *f;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +       struct amdgpu_kiq *kiq = &adev->kiq;
> > +
> > +       if (in_interrupt())
> > +               return 0;
> > +
> > +       mutex_lock(&xgpu->lock);
> > +       amdgpu_ring_alloc(&kiq->ring, 32);
> > +       amdgpu_ring_emit_hdp_flush(&kiq->ring);
> > +       xgpu_kiq_ring_emit_rreg(&kiq->ring, reg);
> > +       amdgpu_ring_emit_hdp_invalidate(&kiq->ring);
> > +       amdgpu_fence_emit(&kiq->ring, &f);
> > +       fence_get(f);
> > +       amdgpu_ring_commit(&kiq->ring);
> > +       mutex_unlock(&xgpu->lock);
> > +
> > +       r = fence_wait(f, false);
> > +       fence_put(f);
> > +       if (r)
> > +               DRM_ERROR("wait for kiq fence error: %ld.\n", r);
> > +       fence_put(f);
> > +
> > +       val = adev->wb.wb[xgpu->reg_val_offs];
> > +
> > +       return val;
> > +}
> > +
> > +void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg,
> > +uint32_t v) {
> > +       signed long r;
> > +       struct fence *f;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +       struct amdgpu_kiq *kiq = &adev->kiq;
> > +
> > +       mutex_lock(&xgpu->lock);
> > +       amdgpu_ring_alloc(&kiq->ring, 32);
> > +       amdgpu_ring_emit_hdp_flush(&kiq->ring);
> > +       xgpu_kiq_ring_emit_wreg(&kiq->ring, reg, v);
> > +       amdgpu_ring_emit_hdp_invalidate(&kiq->ring);
> > +       amdgpu_fence_emit(&kiq->ring, &f);
> > +       fence_get(f);
> > +       amdgpu_ring_commit(&kiq->ring);
> > +       mutex_unlock(&xgpu->lock);
> > +
> > +       r = fence_wait(f, false);
> > +       fence_put(f);
> > +       if (r)
> > +               DRM_ERROR("wait for kiq fence error: %ld.\n", r);
> > +       fence_put(f);
> > +}
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 15/23] drm/amdgpu/mxgpu: implement register access function with KIQ

[Deucher, Alexander]

> -----Original Message-----
> From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf
> Of Yu, Xiangliang
> Sent: Tuesday, December 20, 2016 1:08 AM
> To: Alex Deucher
> Cc: dl.SRDC_SW_GPUVirtualization; Liu, Monk; amd-gfx list
> Subject: RE: [PATCH 15/23] drm/amdgpu/mxgpu: implement register access
> function with KIQ
> 
> 
> 
> > -----Original Message-----
> > From: Alex Deucher [mailto:alexdeucher@gmail.com]
> > Sent: Tuesday, December 20, 2016 7:27 AM
> > To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> > Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> > dl.SRDC_SW_GPUVirtualization
> > <dl.SRDC_SW_GPUVirtualization@amd.com>; Liu, Monk
> > <Monk.Liu@amd.com>
> > Subject: Re: [PATCH 15/23] drm/amdgpu/mxgpu: implement register
> access
> > function with KIQ
> >
> > On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> > wrote:
> > > One of important role of KIQ is provide one way to access VF
> > > registers. This patch implement the feature and export interfaces.
> > >
> > > Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> > > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > > ---
> > >  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   4 ++
> > >  drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c    | 110
> > +++++++++++++++++++++++++++++++
> > >  2 files changed, 114 insertions(+)
> > >  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> > >
> > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > > index 97aae31..c78435d 100644
> > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> > > @@ -75,4 +75,8 @@ void amdgpu_gfx_ring_emit_meta_data(struct
> > > amdgpu_ring *ring,
> > >  /* get full gpu access */
> > >  int amdgpu_get_gpu(struct amdgpu_device *adev);  int
> > > amdgpu_put_gpu(struct amdgpu_device *adev);
> > > +
> > > +/* access vf registers */
> > > +uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
> > > +void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg,
> > > +uint32_t v);
> > >  #endif
> > > diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> > > b/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> > > new file mode 100644
> > > index 0000000..6e1b8e3
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
> > > @@ -0,0 +1,110 @@
> > > +/*
> > > + * Copyright 2016 Advanced Micro Devices, Inc.
> > > + *
> > > + * Permission is hereby granted, free of charge, to any person
> > > +obtaining a
> > > + * copy of this software and associated documentation files (the
> > > +"Software"),
> > > + * to deal in the Software without restriction, including without
> > > +limitation
> > > + * the rights to use, copy, modify, merge, publish, distribute,
> > > +sublicense,
> > > + * and/or sell copies of the Software, and to permit persons to whom
> > > +the
> > > + * Software is furnished to do so, subject to the following conditions:
> > > + *
> > > + * The above copyright notice and this permission notice shall be
> > > +included in
> > > + * all copies or substantial portions of the Software.
> > > + *
> > > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
> > KIND,
> > > +EXPRESS OR
> > > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > > +MERCHANTABILITY,
> > > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN
> > NO EVENT
> > > +SHALL
> > > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
> > > +DAMAGES OR
> > > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
> > > +OTHERWISE,
> > > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
> > THE USE
> > > +OR
> > > + * OTHER DEALINGS IN THE SOFTWARE.
> > > + *
> > > + * Authors: Xiangliang.Yu@amd.com
> > > + *         Monk.Liu@amd.com
> > > + */
> > > +#include "amd_mxgpu.h"
> > > +#include "vid.h"
> > > +#include "gca/gfx_8_0_d.h"
> > > +#include "gca/gfx_8_0_sh_mask.h"
> > > +
> > > +static void xgpu_kiq_ring_emit_rreg(struct amdgpu_ring *ring, u32
> > > +idx) {
> > > +       struct amdgpu_device *adev = ring->adev;
> > > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > > +
> > > +       amdgpu_ring_write(ring, PACKET3(PACKET3_COPY_DATA, 4));
> > > +       amdgpu_ring_write(ring, 0 |     /* src: register*/
> > > +                               (5 << 8) |      /* dst: memory */
> > > +                               (1 << 20));     /* write confirm */
> > > +       amdgpu_ring_write(ring, idx);
> > > +       amdgpu_ring_write(ring, 0);
> > > +       amdgpu_ring_write(ring, lower_32_bits(adev->wb.gpu_addr +
> > > +                               xgpu->reg_val_offs * 4));
> > > +       amdgpu_ring_write(ring, upper_32_bits(adev->wb.gpu_addr +
> > > +                               xgpu->reg_val_offs * 4)); }
> > > +
> > > +static void xgpu_kiq_ring_emit_wreg(struct amdgpu_ring *ring, u32
> > > +idx, u32 val) {
> > > +       amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3));
> > > +       amdgpu_ring_write(ring, (1 << 16)); /* no inc addr */
> > > +       amdgpu_ring_write(ring, idx);
> > > +       amdgpu_ring_write(ring, 0);
> > > +       amdgpu_ring_write(ring, val);
> > > +}
> >
> > These should be in gfx8.
> 
> This is only for virtualization,  please see my previous patch's comment.

Right, but it's gfx8 specific.  The packet and register interface may change on other asics.  Maybe put the gfx8 specific bits in gfx_v8_0_kiq.c or something like that.

Alex

> 
> >
> > Alex
> >
> > > +
> > > +uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg) {
> > > +       signed long r;
> > > +       uint32_t val;
> > > +       struct fence *f;
> > > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > > +       struct amdgpu_kiq *kiq = &adev->kiq;
> > > +
> > > +       if (in_interrupt())
> > > +               return 0;
> > > +
> > > +       mutex_lock(&xgpu->lock);
> > > +       amdgpu_ring_alloc(&kiq->ring, 32);
> > > +       amdgpu_ring_emit_hdp_flush(&kiq->ring);
> > > +       xgpu_kiq_ring_emit_rreg(&kiq->ring, reg);
> > > +       amdgpu_ring_emit_hdp_invalidate(&kiq->ring);
> > > +       amdgpu_fence_emit(&kiq->ring, &f);
> > > +       fence_get(f);
> > > +       amdgpu_ring_commit(&kiq->ring);
> > > +       mutex_unlock(&xgpu->lock);
> > > +
> > > +       r = fence_wait(f, false);
> > > +       fence_put(f);
> > > +       if (r)
> > > +               DRM_ERROR("wait for kiq fence error: %ld.\n", r);
> > > +       fence_put(f);
> > > +
> > > +       val = adev->wb.wb[xgpu->reg_val_offs];
> > > +
> > > +       return val;
> > > +}
> > > +
> > > +void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg,
> > > +uint32_t v) {
> > > +       signed long r;
> > > +       struct fence *f;
> > > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > > +       struct amdgpu_kiq *kiq = &adev->kiq;
> > > +
> > > +       mutex_lock(&xgpu->lock);
> > > +       amdgpu_ring_alloc(&kiq->ring, 32);
> > > +       amdgpu_ring_emit_hdp_flush(&kiq->ring);
> > > +       xgpu_kiq_ring_emit_wreg(&kiq->ring, reg, v);
> > > +       amdgpu_ring_emit_hdp_invalidate(&kiq->ring);
> > > +       amdgpu_fence_emit(&kiq->ring, &f);
> > > +       fence_get(f);
> > > +       amdgpu_ring_commit(&kiq->ring);
> > > +       mutex_unlock(&xgpu->lock);
> > > +
> > > +       r = fence_wait(f, false);
> > > +       fence_put(f);
> > > +       if (r)
> > > +               DRM_ERROR("wait for kiq fence error: %ld.\n", r);
> > > +       fence_put(f);
> > > +}
> > > --
> > > 2.7.4
> > >
> > > _______________________________________________
> > > amd-gfx mailing list
> > > amd-gfx@lists.freedesktop.org
> > > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 16/23] drm/amdgpu: add flag to indicate VF runtime state

[Xiangliang Yu]

Guest driver will select using KIQ to access gpu registers if
runtime flas is set, or will use MMIO method to r/w registers
directly.

When VF's initialization is done, VF should use set runtime flag
and use KIQ method. And clear runtime flag when starting to unload
driver.

Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 6 ++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   | 5 +++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c       | 3 +++
 3 files changed, 14 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index ab8c8bb5..b3c1820 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -94,6 +94,9 @@ uint32_t amdgpu_mm_rreg(struct amdgpu_device *adev, uint32_t reg,
 {
 	uint32_t ret;
 
+	if (amdgpu_sriov_runtime(adev))
+		return amdgpu_kiq_rreg(adev, reg);
+
 	if ((reg * 4) < adev->rmmio_size && !always_indirect)
 		ret = readl(((void __iomem *)adev->rmmio) + (reg * 4));
 	else {
@@ -113,6 +116,9 @@ void amdgpu_mm_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v,
 {
 	trace_amdgpu_mm_wreg(adev->pdev->device, reg, v);
 
+	if (amdgpu_sriov_runtime(adev))
+		return amdgpu_kiq_wreg(adev, reg, v);
+
 	if ((reg * 4) < adev->rmmio_size && !always_indirect)
 		writel(v, ((void __iomem *)adev->rmmio) + (reg * 4));
 	else {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
index c78435d..eb2905e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
@@ -30,6 +30,8 @@ struct amdgpu_vm;
 #define AMDGPU_SRIOV_CAPS_ENABLE_IOV   (1 << 1) /* sr-iov is enabled on this GPU */
 #define AMDGPU_SRIOV_CAPS_IS_VF        (1 << 2) /* this GPU is a virtual function */
 #define AMDGPU_PASSTHROUGH_MODE        (1 << 3) /* thw whole GPU is pass through for VM */
+/* now GPU is not in init/fini stage */
+#define AMDGPU_SRIOV_CAPS_IS_RUNTIME	(1 << 4)
 /* GPU virtualization */
 struct amdgpu_virtualization {
 	uint32_t virtual_caps;
@@ -55,6 +57,9 @@ struct amdgpu_csa {
 #define amdgpu_passthrough(adev) \
 ((adev)->virtualization.virtual_caps & AMDGPU_PASSTHROUGH_MODE)
 
+#define amdgpu_sriov_runtime(adev) \
+((adev)->virtualization.virtual_caps & AMDGPU_SRIOV_CAPS_IS_RUNTIME)
+
 static inline bool is_virtual_machine(void)
 {
 #ifdef CONFIG_X86
diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
index 387246c..f0c7e97 100644
--- a/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
+++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
@@ -134,6 +134,7 @@ int amdgpu_get_gpu(struct amdgpu_device *adev)
 	if (!adev->priv_data)
 		return 0;
 
+	adev->virtualization.virtual_caps &= ~AMDGPU_SRIOV_CAPS_IS_RUNTIME;
 	r = xgpu_request_full_gpu_access(adev, false);
 	if (r)
 		return r;
@@ -152,5 +153,7 @@ int amdgpu_put_gpu(struct amdgpu_device *adev)
 	if (r)
 		return r;
 
+	adev->virtualization.virtual_caps |= AMDGPU_SRIOV_CAPS_IS_RUNTIME;
+
 	return 0;
 }
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 17/23] drm/amdgpu: export vi common ip block

[Xiangliang Yu]

GPU virtualization component need vim common ip block and the
block was also public before. Export it again.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/vi.c | 2 +-
 drivers/gpu/drm/amd/amdgpu/vi.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c
index df0b6a0..5229b4a2a 100644
--- a/drivers/gpu/drm/amd/amdgpu/vi.c
+++ b/drivers/gpu/drm/amd/amdgpu/vi.c
@@ -1377,7 +1377,7 @@ static const struct amd_ip_funcs vi_common_ip_funcs = {
 	.set_powergating_state = vi_common_set_powergating_state,
 };
 
-static const struct amdgpu_ip_block_version vi_common_ip_block =
+const struct amdgpu_ip_block_version vi_common_ip_block =
 {
 	.type = AMD_IP_BLOCK_TYPE_COMMON,
 	.major = 1,
diff --git a/drivers/gpu/drm/amd/amdgpu/vi.h b/drivers/gpu/drm/amd/amdgpu/vi.h
index 575d7ae..6f918f4 100644
--- a/drivers/gpu/drm/amd/amdgpu/vi.h
+++ b/drivers/gpu/drm/amd/amdgpu/vi.h
@@ -27,5 +27,5 @@
 void vi_srbm_select(struct amdgpu_device *adev,
 		    u32 me, u32 pipe, u32 queue, u32 vmid);
 int vi_set_ip_blocks(struct amdgpu_device *adev);
-
+extern const struct amdgpu_ip_block_version vi_common_ip_block;
 #endif
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 17/23] drm/amdgpu: export vi common ip block

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> GPU virtualization component need vim common ip block and the
> block was also public before. Export it again.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/vi.c | 2 +-
>  drivers/gpu/drm/amd/amdgpu/vi.h | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c
> index df0b6a0..5229b4a2a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vi.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vi.c
> @@ -1377,7 +1377,7 @@ static const struct amd_ip_funcs vi_common_ip_funcs = {
>         .set_powergating_state = vi_common_set_powergating_state,
>  };
>
> -static const struct amdgpu_ip_block_version vi_common_ip_block =
> +const struct amdgpu_ip_block_version vi_common_ip_block =
>  {
>         .type = AMD_IP_BLOCK_TYPE_COMMON,
>         .major = 1,
> diff --git a/drivers/gpu/drm/amd/amdgpu/vi.h b/drivers/gpu/drm/amd/amdgpu/vi.h
> index 575d7ae..6f918f4 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vi.h
> +++ b/drivers/gpu/drm/amd/amdgpu/vi.h
> @@ -27,5 +27,5 @@
>  void vi_srbm_select(struct amdgpu_device *adev,
>                     u32 me, u32 pipe, u32 queue, u32 vmid);
>  int vi_set_ip_blocks(struct amdgpu_device *adev);
> -
> +extern const struct amdgpu_ip_block_version vi_common_ip_block;

No need to do this if you handle the VF case directly in vi.c

Alex

>  #endif
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 18/23] drm/amdgpu: add new maroc to identify virtualization ip block

[Xiangliang Yu]

Create a new ip block for virtualization, its main role is to make
virtual code independent and flexible, and also eary to maintainer
virtualization code.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/include/amd_shared.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/amd/include/amd_shared.h b/drivers/gpu/drm/amd/include/amd_shared.h
index 9e555c7..ce36831 100644
--- a/drivers/gpu/drm/amd/include/amd_shared.h
+++ b/drivers/gpu/drm/amd/include/amd_shared.h
@@ -69,6 +69,7 @@ enum amd_ip_block_type {
 	AMD_IP_BLOCK_TYPE_SMC,
 	AMD_IP_BLOCK_TYPE_DCE,
 	AMD_IP_BLOCK_TYPE_GFX,
+	AMD_IP_BLOCK_TYPE_XGPU,
 	AMD_IP_BLOCK_TYPE_SDMA,
 	AMD_IP_BLOCK_TYPE_UVD,
 	AMD_IP_BLOCK_TYPE_VCE,
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 19/23] drm/amdgpu/mxgpu: add implementation of GPU virtualization of VI

[Xiangliang Yu]

Different chips will have different virtual behaviors, so need to
implemnt different virtual feature according to hardware design.

This patch will implemnt Vi family virtualization, it will call
CSA, mailbox interface and allocate wb offset for KIQ.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h |   2 +
 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c  | 205 ++++++++++++++++++++++++++++++++++
 2 files changed, 207 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c

diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
index 54e7b31..02a8839 100644
--- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
@@ -88,4 +88,6 @@ extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
 
 extern int xgpu_request_full_gpu_access(struct amdgpu_device *adev, bool init);
 extern int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool init);
+
+extern void xgpu_vi_add_ip_blocks(struct amdgpu_device *adev);
 #endif
diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
new file mode 100644
index 0000000..e5d517f
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
@@ -0,0 +1,205 @@
+/*
+ * Copyright 2016 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Author: Xiangliang.Yu@amd.com
+ */
+#include "amd_mxgpu.h"
+#include "vi.h"
+#include "gmc_v8_0.h"
+#include "gfx_v8_0.h"
+#include "sdma_v3_0.h"
+#include "tonga_ih.h"
+#include "gmc/gmc_8_2_d.h"
+#include "gmc/gmc_8_2_sh_mask.h"
+#include "oss/oss_3_0_d.h"
+#include "oss/oss_3_0_sh_mask.h"
+#include "bif/bif_5_0_d.h"
+#include "bif/bif_5_0_sh_mask.h"
+#include "amdgpu_powerplay.h"
+#include "dce_virtual.h"
+#include "gca/gfx_8_0_d.h"
+#include "gca/gfx_8_0_enum.h"
+#include "gca/gfx_8_0_sh_mask.h"
+#include "dce/dce_10_0_d.h"
+#include "dce/dce_10_0_sh_mask.h"
+#include "smu/smu_7_1_3_d.h"
+
+static int xgpu_vi_early_init(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+	int r;
+
+	r = amd_xgpu_alloc(adev);
+	if (r)
+		return r;
+
+	r = xgpu_request_full_gpu_access(adev, true);
+	if (r) {
+		DRM_ERROR("failed to send message to HV (%d).\n", r);
+		return r;
+	}
+
+	return 0;
+}
+
+static int xgpu_vi_late_init(void *handle)
+{
+	return 0;
+}
+
+static int xgpu_vi_sw_init(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+
+	return xgpu_allocate_csa(xgpu);
+}
+
+static int xgpu_vi_sw_fini(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+
+	xgpu_destroy_csa(&xgpu->sa);
+
+	return 0;
+}
+
+static int xgpu_vi_hw_init(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+	int r;
+
+	r = amdgpu_wb_get(adev, &xgpu->reg_val_offs);
+	if (r) {
+		DRM_ERROR("failed to alloc reg val offs(%d).\n", r);
+		return r;
+	}
+
+	return 0;
+}
+
+static int xgpu_vi_hw_fini(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+
+	amdgpu_wb_free(adev, xgpu->reg_val_offs);
+	return 0;
+}
+
+static void xgpu_vi_late_fini(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+	struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
+
+	amd_xgpu_free(xgpu);
+	adev->priv_data = NULL;
+
+	xgpu_release_full_gpu_access(adev, false);
+}
+
+static int xgpu_vi_suspend(void *handle)
+{
+	return 0;
+}
+
+static int xgpu_vi_resume(void *handle)
+{
+	return 0;
+}
+
+static bool xgpu_vi_is_idle(void *handle)
+{
+	return false;
+}
+
+static int xgpu_vi_wait_for_idle(void *handle)
+{
+	return 0;
+}
+
+static bool xgpu_vi_check_soft_reset(void *handle)
+{
+	return false;
+}
+
+static int xgpu_vi_pre_soft_reset(void *handle)
+{
+	return 0;
+}
+
+static int xgpu_vi_soft_reset(void *handle)
+{
+	return 0;
+}
+
+static int xgpu_vi_post_soft_reset(void *handle)
+{
+	return 0;
+}
+
+static int xgpu_vi_set_clockgating_state(void *handle,
+					 enum amd_clockgating_state state)
+{
+	return 0;
+}
+
+const struct amd_ip_funcs xgpu_vi_ip_funcs = {
+	.name = "xgpu-vi",
+	.early_init = xgpu_vi_early_init,
+	.late_init = xgpu_vi_late_init,
+	.sw_init = xgpu_vi_sw_init,
+	.sw_fini = xgpu_vi_sw_fini,
+	.hw_init = xgpu_vi_hw_init,
+	.hw_fini = xgpu_vi_hw_fini,
+	.late_fini = xgpu_vi_late_fini,
+	.suspend = xgpu_vi_suspend,
+	.resume = xgpu_vi_resume,
+	.is_idle = xgpu_vi_is_idle,
+	.wait_for_idle = xgpu_vi_wait_for_idle,
+	.check_soft_reset = xgpu_vi_check_soft_reset,
+	.pre_soft_reset = xgpu_vi_pre_soft_reset,
+	.soft_reset = xgpu_vi_soft_reset,
+	.post_soft_reset = xgpu_vi_post_soft_reset,
+	.set_clockgating_state = xgpu_vi_set_clockgating_state,
+};
+
+static const struct amdgpu_ip_block_version xgpu_vi_ip_block = {
+	.type = AMD_IP_BLOCK_TYPE_XGPU,
+	.major = 1,
+	.minor = 0,
+	.rev = 0,
+	.funcs = &xgpu_vi_ip_funcs,
+};
+
+void xgpu_vi_add_ip_blocks(struct amdgpu_device *adev)
+{
+	amdgpu_ip_block_add(adev, &vi_common_ip_block);
+	amdgpu_ip_block_add(adev, &gmc_v8_5_ip_block);
+	amdgpu_ip_block_add(adev, &tonga_ih_ip_block);
+	amdgpu_ip_block_add(adev, &amdgpu_pp_ip_block);
+	amdgpu_ip_block_add(adev, &dce_virtual_ip_block);
+	amdgpu_ip_block_add(adev, &gfx_v8_0_ip_block);
+	amdgpu_ip_block_add(adev, &xgpu_vi_ip_block);
+	amdgpu_ip_block_add(adev, &sdma_v3_0_ip_block);
+}
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 19/23] drm/amdgpu/mxgpu: add implementation of GPU virtualization of VI

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> Different chips will have different virtual behaviors, so need to
> implemnt different virtual feature according to hardware design.
>
> This patch will implemnt Vi family virtualization, it will call
> CSA, mailbox interface and allocate wb offset for KIQ.
>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h |   2 +
>  drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c  | 205 ++++++++++++++++++++++++++++++++++
>  2 files changed, 207 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
>
> diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> index 54e7b31..02a8839 100644
> --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> @@ -88,4 +88,6 @@ extern void xgpu_destroy_csa(struct amd_xgpu_csa *csa);
>
>  extern int xgpu_request_full_gpu_access(struct amdgpu_device *adev, bool init);
>  extern int xgpu_release_full_gpu_access(struct amdgpu_device *adev, bool init);
> +
> +extern void xgpu_vi_add_ip_blocks(struct amdgpu_device *adev);
>  #endif
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> new file mode 100644
> index 0000000..e5d517f
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> @@ -0,0 +1,205 @@
> +/*
> + * Copyright 2016 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + * Author: Xiangliang.Yu@amd.com
> + */
> +#include "amd_mxgpu.h"
> +#include "vi.h"
> +#include "gmc_v8_0.h"
> +#include "gfx_v8_0.h"
> +#include "sdma_v3_0.h"
> +#include "tonga_ih.h"
> +#include "gmc/gmc_8_2_d.h"
> +#include "gmc/gmc_8_2_sh_mask.h"
> +#include "oss/oss_3_0_d.h"
> +#include "oss/oss_3_0_sh_mask.h"
> +#include "bif/bif_5_0_d.h"
> +#include "bif/bif_5_0_sh_mask.h"
> +#include "amdgpu_powerplay.h"
> +#include "dce_virtual.h"
> +#include "gca/gfx_8_0_d.h"
> +#include "gca/gfx_8_0_enum.h"
> +#include "gca/gfx_8_0_sh_mask.h"
> +#include "dce/dce_10_0_d.h"
> +#include "dce/dce_10_0_sh_mask.h"
> +#include "smu/smu_7_1_3_d.h"

You don't need all of these register headers.  there aren't even any
registers used in this code.


> +
> +static int xgpu_vi_early_init(void *handle)
> +{
> +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> +       int r;
> +
> +       r = amd_xgpu_alloc(adev);
> +       if (r)
> +               return r;
> +
> +       r = xgpu_request_full_gpu_access(adev, true);
> +       if (r) {
> +               DRM_ERROR("failed to send message to HV (%d).\n", r);
> +               return r;
> +       }
> +
> +       return 0;
> +}
> +
> +static int xgpu_vi_late_init(void *handle)
> +{
> +       return 0;
> +}
> +
> +static int xgpu_vi_sw_init(void *handle)
> +{
> +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +
> +       return xgpu_allocate_csa(xgpu);
> +}
> +
> +static int xgpu_vi_sw_fini(void *handle)
> +{
> +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +
> +       xgpu_destroy_csa(&xgpu->sa);
> +
> +       return 0;
> +}
> +
> +static int xgpu_vi_hw_init(void *handle)
> +{
> +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +       int r;
> +
> +       r = amdgpu_wb_get(adev, &xgpu->reg_val_offs);
> +       if (r) {
> +               DRM_ERROR("failed to alloc reg val offs(%d).\n", r);
> +               return r;
> +       }
> +
> +       return 0;
> +}
> +
> +static int xgpu_vi_hw_fini(void *handle)
> +{
> +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +
> +       amdgpu_wb_free(adev, xgpu->reg_val_offs);
> +       return 0;
> +}
> +
> +static void xgpu_vi_late_fini(void *handle)
> +{
> +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> +
> +       amd_xgpu_free(xgpu);
> +       adev->priv_data = NULL;
> +
> +       xgpu_release_full_gpu_access(adev, false);
> +}

Most of this code seems like it would be better to just add directly
to the relevant code in vi.c (VF reg offset stuff, mail box requests)
and gfx_v8_0.c (CSA stuff).

> +
> +static int xgpu_vi_suspend(void *handle)
> +{
> +       return 0;
> +}
> +
> +static int xgpu_vi_resume(void *handle)
> +{
> +       return 0;
> +}
> +
> +static bool xgpu_vi_is_idle(void *handle)
> +{
> +       return false;
> +}
> +
> +static int xgpu_vi_wait_for_idle(void *handle)
> +{
> +       return 0;
> +}
> +
> +static bool xgpu_vi_check_soft_reset(void *handle)
> +{
> +       return false;
> +}
> +
> +static int xgpu_vi_pre_soft_reset(void *handle)
> +{
> +       return 0;
> +}
> +
> +static int xgpu_vi_soft_reset(void *handle)
> +{
> +       return 0;
> +}
> +
> +static int xgpu_vi_post_soft_reset(void *handle)
> +{
> +       return 0;
> +}
> +
> +static int xgpu_vi_set_clockgating_state(void *handle,
> +                                        enum amd_clockgating_state state)
> +{
> +       return 0;
> +}
> +
> +const struct amd_ip_funcs xgpu_vi_ip_funcs = {
> +       .name = "xgpu-vi",
> +       .early_init = xgpu_vi_early_init,
> +       .late_init = xgpu_vi_late_init,
> +       .sw_init = xgpu_vi_sw_init,
> +       .sw_fini = xgpu_vi_sw_fini,
> +       .hw_init = xgpu_vi_hw_init,
> +       .hw_fini = xgpu_vi_hw_fini,
> +       .late_fini = xgpu_vi_late_fini,
> +       .suspend = xgpu_vi_suspend,
> +       .resume = xgpu_vi_resume,
> +       .is_idle = xgpu_vi_is_idle,
> +       .wait_for_idle = xgpu_vi_wait_for_idle,
> +       .check_soft_reset = xgpu_vi_check_soft_reset,
> +       .pre_soft_reset = xgpu_vi_pre_soft_reset,
> +       .soft_reset = xgpu_vi_soft_reset,
> +       .post_soft_reset = xgpu_vi_post_soft_reset,
> +       .set_clockgating_state = xgpu_vi_set_clockgating_state,
> +};
> +
> +static const struct amdgpu_ip_block_version xgpu_vi_ip_block = {
> +       .type = AMD_IP_BLOCK_TYPE_XGPU,
> +       .major = 1,
> +       .minor = 0,
> +       .rev = 0,
> +       .funcs = &xgpu_vi_ip_funcs,
> +};
> +
> +void xgpu_vi_add_ip_blocks(struct amdgpu_device *adev)
> +{
> +       amdgpu_ip_block_add(adev, &vi_common_ip_block);
> +       amdgpu_ip_block_add(adev, &gmc_v8_5_ip_block);
> +       amdgpu_ip_block_add(adev, &tonga_ih_ip_block);
> +       amdgpu_ip_block_add(adev, &amdgpu_pp_ip_block);
> +       amdgpu_ip_block_add(adev, &dce_virtual_ip_block);
> +       amdgpu_ip_block_add(adev, &gfx_v8_0_ip_block);
> +       amdgpu_ip_block_add(adev, &xgpu_vi_ip_block);
> +       amdgpu_ip_block_add(adev, &sdma_v3_0_ip_block);
> +}

Handle the VF specific IP blocks directly in vi.c

> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 19/23] drm/amdgpu/mxgpu: add implementation of GPU virtualization of VI

[Yu, Xiangliang]

> -----Original Message-----
> From: Alex Deucher [mailto:alexdeucher@gmail.com]
> Sent: Tuesday, December 20, 2016 7:35 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>;
> dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>
> Subject: Re: [PATCH 19/23] drm/amdgpu/mxgpu: add implementation of
> GPU virtualization of VI
> 
> On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com>
> wrote:
> > Different chips will have different virtual behaviors, so need to
> > implemnt different virtual feature according to hardware design.
> >
> > This patch will implemnt Vi family virtualization, it will call CSA,
> > mailbox interface and allocate wb offset for KIQ.
> >
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > ---
> >  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h |   2 +
> >  drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c  | 205
> > ++++++++++++++++++++++++++++++++++
> >  2 files changed, 207 insertions(+)
> >  create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> >
> > diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > index 54e7b31..02a8839 100644
> > --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
> > @@ -88,4 +88,6 @@ extern void xgpu_destroy_csa(struct amd_xgpu_csa
> > *csa);
> >
> >  extern int xgpu_request_full_gpu_access(struct amdgpu_device *adev,
> > bool init);  extern int xgpu_release_full_gpu_access(struct
> > amdgpu_device *adev, bool init);
> > +
> > +extern void xgpu_vi_add_ip_blocks(struct amdgpu_device *adev);
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> > b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> > new file mode 100644
> > index 0000000..e5d517f
> > --- /dev/null
> > +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> > @@ -0,0 +1,205 @@
> > +/*
> > + * Copyright 2016 Advanced Micro Devices, Inc.
> > + *
> > + * Permission is hereby granted, free of charge, to any person
> > +obtaining a
> > + * copy of this software and associated documentation files (the
> > +"Software"),
> > + * to deal in the Software without restriction, including without
> > +limitation
> > + * the rights to use, copy, modify, merge, publish, distribute,
> > +sublicense,
> > + * and/or sell copies of the Software, and to permit persons to whom
> > +the
> > + * Software is furnished to do so, subject to the following conditions:
> > + *
> > + * The above copyright notice and this permission notice shall be
> > +included in
> > + * all copies or substantial portions of the Software.
> > + *
> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
> KIND,
> > +EXPRESS OR
> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > +MERCHANTABILITY,
> > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN
> NO EVENT
> > +SHALL
> > + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
> > +DAMAGES OR
> > + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
> > +OTHERWISE,
> > + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
> THE USE
> > +OR
> > + * OTHER DEALINGS IN THE SOFTWARE.
> > + *
> > + * Author: Xiangliang.Yu@amd.com
> > + */
> > +#include "amd_mxgpu.h"
> > +#include "vi.h"
> > +#include "gmc_v8_0.h"
> > +#include "gfx_v8_0.h"
> > +#include "sdma_v3_0.h"
> > +#include "tonga_ih.h"
> > +#include "gmc/gmc_8_2_d.h"
> > +#include "gmc/gmc_8_2_sh_mask.h"
> > +#include "oss/oss_3_0_d.h"
> > +#include "oss/oss_3_0_sh_mask.h"
> > +#include "bif/bif_5_0_d.h"
> > +#include "bif/bif_5_0_sh_mask.h"
> > +#include "amdgpu_powerplay.h"
> > +#include "dce_virtual.h"
> > +#include "gca/gfx_8_0_d.h"
> > +#include "gca/gfx_8_0_enum.h"
> > +#include "gca/gfx_8_0_sh_mask.h"
> > +#include "dce/dce_10_0_d.h"
> > +#include "dce/dce_10_0_sh_mask.h"
> > +#include "smu/smu_7_1_3_d.h"
> 
> You don't need all of these register headers.  there aren't even any registers
> used in this code.

The headers is need by golden setting.

> 
> 
> > +
> > +static int xgpu_vi_early_init(void *handle) {
> > +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> > +       int r;
> > +
> > +       r = amd_xgpu_alloc(adev);
> > +       if (r)
> > +               return r;
> > +
> > +       r = xgpu_request_full_gpu_access(adev, true);
> > +       if (r) {
> > +               DRM_ERROR("failed to send message to HV (%d).\n", r);
> > +               return r;
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int xgpu_vi_late_init(void *handle) {
> > +       return 0;
> > +}
> > +
> > +static int xgpu_vi_sw_init(void *handle) {
> > +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +
> > +       return xgpu_allocate_csa(xgpu); }
> > +
> > +static int xgpu_vi_sw_fini(void *handle) {
> > +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +
> > +       xgpu_destroy_csa(&xgpu->sa);
> > +
> > +       return 0;
> > +}
> > +
> > +static int xgpu_vi_hw_init(void *handle) {
> > +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +       int r;
> > +
> > +       r = amdgpu_wb_get(adev, &xgpu->reg_val_offs);
> > +       if (r) {
> > +               DRM_ERROR("failed to alloc reg val offs(%d).\n", r);
> > +               return r;
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int xgpu_vi_hw_fini(void *handle) {
> > +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +
> > +       amdgpu_wb_free(adev, xgpu->reg_val_offs);
> > +       return 0;
> > +}
> > +
> > +static void xgpu_vi_late_fini(void *handle) {
> > +       struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> > +       struct amd_xgpu *xgpu = (struct amd_xgpu *)adev->priv_data;
> > +
> > +       amd_xgpu_free(xgpu);
> > +       adev->priv_data = NULL;
> > +
> > +       xgpu_release_full_gpu_access(adev, false); }
> 
> Most of this code seems like it would be better to just add directly to the
> relevant code in vi.c (VF reg offset stuff, mail box requests) and gfx_v8_0.c
> (CSA stuff).

As I said in previous patch comments, the main aim of the series is support centralized management of virtualization chips and features. 

We don't want to interrupt amdgpu component for each commit later.
This design will bring us lot of room for support virtualization.

> 
> > +
> > +static int xgpu_vi_suspend(void *handle) {
> > +       return 0;
> > +}
> > +
> > +static int xgpu_vi_resume(void *handle) {
> > +       return 0;
> > +}
> > +
> > +static bool xgpu_vi_is_idle(void *handle) {
> > +       return false;
> > +}
> > +
> > +static int xgpu_vi_wait_for_idle(void *handle) {
> > +       return 0;
> > +}
> > +
> > +static bool xgpu_vi_check_soft_reset(void *handle) {
> > +       return false;
> > +}
> > +
> > +static int xgpu_vi_pre_soft_reset(void *handle) {
> > +       return 0;
> > +}
> > +
> > +static int xgpu_vi_soft_reset(void *handle) {
> > +       return 0;
> > +}
> > +
> > +static int xgpu_vi_post_soft_reset(void *handle) {
> > +       return 0;
> > +}
> > +
> > +static int xgpu_vi_set_clockgating_state(void *handle,
> > +                                        enum amd_clockgating_state
> > +state) {
> > +       return 0;
> > +}
> > +
> > +const struct amd_ip_funcs xgpu_vi_ip_funcs = {
> > +       .name = "xgpu-vi",
> > +       .early_init = xgpu_vi_early_init,
> > +       .late_init = xgpu_vi_late_init,
> > +       .sw_init = xgpu_vi_sw_init,
> > +       .sw_fini = xgpu_vi_sw_fini,
> > +       .hw_init = xgpu_vi_hw_init,
> > +       .hw_fini = xgpu_vi_hw_fini,
> > +       .late_fini = xgpu_vi_late_fini,
> > +       .suspend = xgpu_vi_suspend,
> > +       .resume = xgpu_vi_resume,
> > +       .is_idle = xgpu_vi_is_idle,
> > +       .wait_for_idle = xgpu_vi_wait_for_idle,
> > +       .check_soft_reset = xgpu_vi_check_soft_reset,
> > +       .pre_soft_reset = xgpu_vi_pre_soft_reset,
> > +       .soft_reset = xgpu_vi_soft_reset,
> > +       .post_soft_reset = xgpu_vi_post_soft_reset,
> > +       .set_clockgating_state = xgpu_vi_set_clockgating_state, };
> > +
> > +static const struct amdgpu_ip_block_version xgpu_vi_ip_block = {
> > +       .type = AMD_IP_BLOCK_TYPE_XGPU,
> > +       .major = 1,
> > +       .minor = 0,
> > +       .rev = 0,
> > +       .funcs = &xgpu_vi_ip_funcs,
> > +};
> > +
> > +void xgpu_vi_add_ip_blocks(struct amdgpu_device *adev) {
> > +       amdgpu_ip_block_add(adev, &vi_common_ip_block);
> > +       amdgpu_ip_block_add(adev, &gmc_v8_5_ip_block);
> > +       amdgpu_ip_block_add(adev, &tonga_ih_ip_block);
> > +       amdgpu_ip_block_add(adev, &amdgpu_pp_ip_block);
> > +       amdgpu_ip_block_add(adev, &dce_virtual_ip_block);
> > +       amdgpu_ip_block_add(adev, &gfx_v8_0_ip_block);
> > +       amdgpu_ip_block_add(adev, &xgpu_vi_ip_block);
> > +       amdgpu_ip_block_add(adev, &sdma_v3_0_ip_block); }
> 
> Handle the VF specific IP blocks directly in vi.c
> 
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 20/23] drm/amdgpu/mxgpu: enable VI virtualization

[Xiangliang Yu]

Call vi ip block setup function to enable vi virtualization.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
index 72469e2..7cacab7 100644
--- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
+++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
@@ -62,6 +62,7 @@ static int amd_xgpu_init_ip_blocks(struct amdgpu_device *adev)
 	case CHIP_TONGA:
 	case CHIP_FIJI:
 		/* add IP blocks*/
+		xgpu_vi_add_ip_blocks(adev);
 		break;
 	default:
 		DRM_ERROR("Does not support this chip.\n");
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 20/23] drm/amdgpu/mxgpu: enable VI virtualization

[Deucher, Alexander]

> -----Original Message-----
> From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf
> Of Xiangliang Yu
> Sent: Saturday, December 17, 2016 11:17 AM
> To: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization
> Cc: Yu, Xiangliang
> Subject: [PATCH 20/23] drm/amdgpu/mxgpu: enable VI virtualization
> 
> Call vi ip block setup function to enable vi virtualization.
> 
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
> b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
> index 72469e2..7cacab7 100644
> --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
> +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
> @@ -62,6 +62,7 @@ static int amd_xgpu_init_ip_blocks(struct
> amdgpu_device *adev)
>  	case CHIP_TONGA:
>  	case CHIP_FIJI:
>  		/* add IP blocks*/
> +		xgpu_vi_add_ip_blocks(adev);

Rather than going through all this indirection, just check for the VF flag in vi.c and add the appropriate IP blocks for the bare metal vs VF cases there.  It would also allow you to keep the vi_common functions static.

Alex

>  		break;
>  	default:
>  		DRM_ERROR("Does not support this chip.\n");
> --
> 2.7.4
> 
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 20/23] drm/amdgpu/mxgpu: enable VI virtualization

[Yu, Xiangliang]

> -----Original Message-----
> From: Deucher, Alexander
> Sent: Tuesday, December 20, 2016 7:08 AM
> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>; amd-
> gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization
> <dl.SRDC_SW_GPUVirtualization@amd.com>
> Cc: Yu, Xiangliang <Xiangliang.Yu@amd.com>
> Subject: RE: [PATCH 20/23] drm/amdgpu/mxgpu: enable VI virtualization
> 
> > -----Original Message-----
> > From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf
> > Of Xiangliang Yu
> > Sent: Saturday, December 17, 2016 11:17 AM
> > To: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization
> > Cc: Yu, Xiangliang
> > Subject: [PATCH 20/23] drm/amdgpu/mxgpu: enable VI virtualization
> >
> > Call vi ip block setup function to enable vi virtualization.
> >
> > Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> > ---
> >  drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
> > b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
> > index 72469e2..7cacab7 100644
> > --- a/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
> > +++ b/drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
> > @@ -62,6 +62,7 @@ static int amd_xgpu_init_ip_blocks(struct
> > amdgpu_device *adev)
> >  	case CHIP_TONGA:
> >  	case CHIP_FIJI:
> >  		/* add IP blocks*/
> > +		xgpu_vi_add_ip_blocks(adev);
> 
> Rather than going through all this indirection, just check for the VF flag in vi.c
> and add the appropriate IP blocks for the bare metal vs VF cases there.  It
> would also allow you to keep the vi_common functions static.

The method let us easy to put all virtualization implementation together, just export one interface to amdgpu. That is also the main aim of Re-org monk's sriov patches, don't want to mix virtualization into amdgpu. 

I don't why you keep vi_common function as static, I remember it is public before.

> 
> Alex
> 
> >  		break;
> >  	default:
> >  		DRM_ERROR("Does not support this chip.\n");
> > --
> > 2.7.4
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization

[Xiangliang Yu]

GPU virtualization has different sequence from normal, change it.

Signed-off-by: Frank Min <Frank.Min@amd.com>
Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   2 +
 drivers/gpu/drm/amd/amdgpu/vi.c          |   6 +
 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c     | 267 +++++++++++++++++++++++++++++++
 3 files changed, 275 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
index eb2905e..e781c9c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
@@ -84,4 +84,6 @@ int amdgpu_put_gpu(struct amdgpu_device *adev);
 /* access vf registers */
 uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
 void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
+
+void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev);
 #endif
diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c
index 5229b4a2a..0d5e807 100644
--- a/drivers/gpu/drm/amd/amdgpu/vi.c
+++ b/drivers/gpu/drm/amd/amdgpu/vi.c
@@ -285,6 +285,12 @@ static void vi_init_golden_registers(struct amdgpu_device *adev)
 	/* Some of the registers might be dependent on GRBM_GFX_INDEX */
 	mutex_lock(&adev->grbm_idx_mutex);
 
+	if (adev->flags & AMD_IS_VF) {
+		amdgpu_xgpu_init_golden_registers(adev);
+		mutex_unlock(&adev->grbm_idx_mutex);
+		return;
+	}
+
 	switch (adev->asic_type) {
 	case CHIP_TOPAZ:
 		amdgpu_program_register_sequence(adev,
diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
index e5d517f..fa1ee8f 100644
--- a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
+++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
@@ -42,6 +42,273 @@
 #include "dce/dce_10_0_sh_mask.h"
 #include "smu/smu_7_1_3_d.h"
 
+static const u32 xgpu_fiji_mgcg_cgcg_init[] = {
+	mmRLC_CGTT_MGCG_OVERRIDE, 0xffffffff, 0xffffffff,
+	mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
+	mmCB_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_BCI_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_CP_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_CPC_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_CPF_CLK_CTRL, 0xffffffff, 0x40000100,
+	mmCGTT_DRM_CLK_CTRL0, 0xffffffff, 0x00600100,
+	mmCGTT_GDS_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_IA_CLK_CTRL, 0xffffffff, 0x06000100,
+	mmCGTT_PA_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_WD_CLK_CTRL, 0xffffffff, 0x06000100,
+	mmCGTT_PC_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_RLC_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_SC_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_SPI_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_SQ_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_SQG_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL0, 0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL1, 0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL2, 0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL3, 0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL4, 0xffffffff, 0x00000100,
+	mmCGTT_TCI_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_TCP_CLK_CTRL, 0xffffffff, 0x00000100,
+	mmCGTT_VGT_CLK_CTRL, 0xffffffff, 0x06000100,
+	mmDB_CGTT_CLK_CTRL_0, 0xffffffff, 0x00000100,
+	mmTA_CGTT_CTRL, 0xffffffff, 0x00000100,
+	mmTCA_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
+	mmTCC_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
+	mmTD_CGTT_CTRL, 0xffffffff, 0x00000100,
+	mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
+	mmCGTS_SM_CTRL_REG, 0xffffffff, 0x96e00200,
+	mmCP_RB_WPTR_POLL_CNTL, 0xffffffff, 0x00900100,
+	mmRLC_CGCG_CGLS_CTRL, 0xffffffff, 0x0020003c,
+	mmPCIE_INDEX, 0xffffffff, 0x0140001c,
+	mmPCIE_DATA, 0x000f0000, 0x00000000,
+	mmSMC_IND_INDEX_4, 0xffffffff, 0xC060000C,
+	mmSMC_IND_DATA_4, 0xc0000fff, 0x00000100,
+	mmXDMA_CLOCK_GATING_CNTL, 0xffffffff, 0x00000100,
+	mmXDMA_MEM_POWER_CNTL, 0x00000101, 0x00000000,
+	mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104,
+	mmCGTT_DRM_CLK_CTRL0, 0xff000fff, 0x00000100,
+	mmHDP_XDP_CGTT_BLK_CTRL, 0xc0000fff, 0x00000104,
+	mmCP_MEM_SLP_CNTL, 0x00000001, 0x00000001,
+	mmSDMA0_CLK_CTRL, 0xff000ff0, 0x00000100,
+	mmSDMA1_CLK_CTRL, 0xff000ff0, 0x00000100,
+};
+
+static const u32 xgpu_golden_settings_fiji_a10[] = {
+	mmCB_HW_CONTROL_3, 0x000001ff, 0x00000040,
+	mmDB_DEBUG2, 0xf00fffff, 0x00000400,
+	mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
+	mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
+	mmFBC_MISC, 0x1f311fff, 0x12300000,
+	mmHDMI_CONTROL, 0x31000111, 0x00000011,
+	mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
+	mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
+	mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
+	mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
+	mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
+	mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
+	mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
+	mmTCP_ADDR_CONFIG, 0x000003ff, 0x000000ff,
+	mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
+	mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+	mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+	mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+	mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+};
+
+static const u32 xgpu_fiji_golden_common_all[] = {
+	mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
+	mmPA_SC_RASTER_CONFIG, 0xffffffff, 0x3a00161a,
+	mmPA_SC_RASTER_CONFIG_1, 0xffffffff, 0x0000002e,
+	mmGB_ADDR_CONFIG, 0xffffffff, 0x22011003,
+	mmSPI_RESOURCE_RESERVE_CU_0, 0xffffffff, 0x00000800,
+	mmSPI_RESOURCE_RESERVE_CU_1, 0xffffffff, 0x00000800,
+	mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
+	mmSPI_RESOURCE_RESERVE_EN_CU_1, 0xffffffff, 0x00007FAF,
+	mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
+	mmSPI_CONFIG_CNTL_1, 0x0000000f, 0x00000009,
+};
+
+static const u32 xgpu_tonga_mgcg_cgcg_init[] = {
+	mmRLC_CGTT_MGCG_OVERRIDE,   0xffffffff, 0xffffffff,
+	mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
+	mmCB_CGTT_SCLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_BCI_CLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_CP_CLK_CTRL,         0xffffffff, 0x00000100,
+	mmCGTT_CPC_CLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_CPF_CLK_CTRL,        0xffffffff, 0x40000100,
+	mmCGTT_DRM_CLK_CTRL0,       0xffffffff, 0x00600100,
+	mmCGTT_GDS_CLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_IA_CLK_CTRL,         0xffffffff, 0x06000100,
+	mmCGTT_PA_CLK_CTRL,         0xffffffff, 0x00000100,
+	mmCGTT_WD_CLK_CTRL,         0xffffffff, 0x06000100,
+	mmCGTT_PC_CLK_CTRL,         0xffffffff, 0x00000100,
+	mmCGTT_RLC_CLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_SC_CLK_CTRL,         0xffffffff, 0x00000100,
+	mmCGTT_SPI_CLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_SQ_CLK_CTRL,         0xffffffff, 0x00000100,
+	mmCGTT_SQG_CLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL0,        0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL1,        0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL2,        0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL3,        0xffffffff, 0x00000100,
+	mmCGTT_SX_CLK_CTRL4,        0xffffffff, 0x00000100,
+	mmCGTT_TCI_CLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_TCP_CLK_CTRL,        0xffffffff, 0x00000100,
+	mmCGTT_VGT_CLK_CTRL,        0xffffffff, 0x06000100,
+	mmDB_CGTT_CLK_CTRL_0,       0xffffffff, 0x00000100,
+	mmTA_CGTT_CTRL,             0xffffffff, 0x00000100,
+	mmTCA_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
+	mmTCC_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
+	mmTD_CGTT_CTRL,             0xffffffff, 0x00000100,
+	mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
+	mmCGTS_CU0_SP0_CTRL_REG,    0xffffffff, 0x00010000,
+	mmCGTS_CU0_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
+	mmCGTS_CU0_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
+	mmCGTS_CU0_SP1_CTRL_REG,    0xffffffff, 0x00060005,
+	mmCGTS_CU0_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
+	mmCGTS_CU1_SP0_CTRL_REG,    0xffffffff, 0x00010000,
+	mmCGTS_CU1_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
+	mmCGTS_CU1_TA_CTRL_REG,     0xffffffff, 0x00040007,
+	mmCGTS_CU1_SP1_CTRL_REG,    0xffffffff, 0x00060005,
+	mmCGTS_CU1_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
+	mmCGTS_CU2_SP0_CTRL_REG,    0xffffffff, 0x00010000,
+	mmCGTS_CU2_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
+	mmCGTS_CU2_TA_CTRL_REG,     0xffffffff, 0x00040007,
+	mmCGTS_CU2_SP1_CTRL_REG,    0xffffffff, 0x00060005,
+	mmCGTS_CU2_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
+	mmCGTS_CU3_SP0_CTRL_REG,    0xffffffff, 0x00010000,
+	mmCGTS_CU3_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
+	mmCGTS_CU3_TA_CTRL_REG,     0xffffffff, 0x00040007,
+	mmCGTS_CU3_SP1_CTRL_REG,    0xffffffff, 0x00060005,
+	mmCGTS_CU3_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
+	mmCGTS_CU4_SP0_CTRL_REG,    0xffffffff, 0x00010000,
+	mmCGTS_CU4_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
+	mmCGTS_CU4_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
+	mmCGTS_CU4_SP1_CTRL_REG,    0xffffffff, 0x00060005,
+	mmCGTS_CU4_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
+	mmCGTS_CU5_SP0_CTRL_REG,    0xffffffff, 0x00010000,
+	mmCGTS_CU5_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
+	mmCGTS_CU5_TA_CTRL_REG,     0xffffffff, 0x00040007,
+	mmCGTS_CU5_SP1_CTRL_REG,    0xffffffff, 0x00060005,
+	mmCGTS_CU5_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
+	mmCGTS_CU6_SP0_CTRL_REG,    0xffffffff, 0x00010000,
+	mmCGTS_CU6_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
+	mmCGTS_CU6_TA_CTRL_REG,     0xffffffff, 0x00040007,
+	mmCGTS_CU6_SP1_CTRL_REG,    0xffffffff, 0x00060005,
+	mmCGTS_CU6_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
+	mmCGTS_CU7_SP0_CTRL_REG,    0xffffffff, 0x00010000,
+	mmCGTS_CU7_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
+	mmCGTS_CU7_TA_CTRL_REG,     0xffffffff, 0x00040007,
+	mmCGTS_CU7_SP1_CTRL_REG,    0xffffffff, 0x00060005,
+	mmCGTS_CU7_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
+	mmCGTS_SM_CTRL_REG,         0xffffffff, 0x96e00200,
+	mmCP_RB_WPTR_POLL_CNTL,     0xffffffff, 0x00900100,
+	mmRLC_CGCG_CGLS_CTRL,       0xffffffff, 0x0020003c,
+	mmPCIE_INDEX,               0xffffffff, 0x0140001c,
+	mmPCIE_DATA,                0x000f0000, 0x00000000,
+	mmSMC_IND_INDEX_4,          0xffffffff, 0xC060000C,
+	mmSMC_IND_DATA_4,           0xc0000fff, 0x00000100,
+	mmXDMA_CLOCK_GATING_CNTL,   0xffffffff, 0x00000100,
+	mmXDMA_MEM_POWER_CNTL,      0x00000101, 0x00000000,
+	mmMC_MEM_POWER_LS,          0xffffffff, 0x00000104,
+	mmCGTT_DRM_CLK_CTRL0,       0xff000fff, 0x00000100,
+	mmHDP_XDP_CGTT_BLK_CTRL,    0xc0000fff, 0x00000104,
+	mmCP_MEM_SLP_CNTL,          0x00000001, 0x00000001,
+	mmSDMA0_CLK_CTRL,           0xff000ff0, 0x00000100,
+	mmSDMA1_CLK_CTRL,           0xff000ff0, 0x00000100,
+};
+
+static const u32 xgpu_golden_settings_tonga_a11[] = {
+	mmCB_HW_CONTROL, 0xfffdf3cf, 0x00007208,
+	mmCB_HW_CONTROL_3, 0x00000040, 0x00000040,
+	mmDB_DEBUG2, 0xf00fffff, 0x00000400,
+	mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
+	mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
+	mmFBC_MISC, 0x1f311fff, 0x12300000,
+	mmGB_GPU_ID, 0x0000000f, 0x00000000,
+	mmHDMI_CONTROL, 0x31000111, 0x00000011,
+	mmMC_ARB_WTM_GRPWT_RD, 0x00000003, 0x00000000,
+	mmMC_HUB_RDREQ_DMIF_LIMIT, 0x0000007f, 0x00000028,
+	mmMC_HUB_WDP_UMC, 0x00007fb6, 0x00000991,
+	mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
+	mmPA_SC_FIFO_DEPTH_CNTL, 0x000003ff, 0x000000fc,
+	mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
+	mmRLC_CGCG_CGLS_CTRL, 0x00000003, 0x0000003c,
+	mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
+	mmSDMA0_CLK_CTRL, 0xff000fff, 0x00000000,
+	mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
+	mmSDMA1_CLK_CTRL, 0xff000fff, 0x00000000,
+	mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
+	mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
+	mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
+	mmTCC_CTRL, 0x00100000, 0xf31fff7f,
+	mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
+	mmTCP_ADDR_CONFIG, 0x000003ff, 0x000002fb,
+	mmTCP_CHAN_STEER_HI, 0xffffffff, 0x0000543b,
+	mmTCP_CHAN_STEER_LO, 0xffffffff, 0xa9210876,
+	mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
+	mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+	mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+	mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+	mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+};
+
+static const u32 xgpu_tonga_golden_common_all[] = {
+	mmGRBM_GFX_INDEX,               0xffffffff, 0xe0000000,
+	mmPA_SC_RASTER_CONFIG,          0xffffffff, 0x16000012,
+	mmPA_SC_RASTER_CONFIG_1,        0xffffffff, 0x0000002A,
+	mmGB_ADDR_CONFIG,               0xffffffff, 0x22011002,
+	mmSPI_RESOURCE_RESERVE_CU_0,    0xffffffff, 0x00000800,
+	mmSPI_RESOURCE_RESERVE_CU_1,    0xffffffff, 0x00000800,
+	mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
+};
+
+void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev)
+{
+	switch (adev->asic_type) {
+	case CHIP_FIJI:
+		amdgpu_program_register_sequence(adev,
+						 xgpu_fiji_mgcg_cgcg_init,
+						 (const u32)ARRAY_SIZE(
+						 xgpu_fiji_mgcg_cgcg_init));
+		amdgpu_program_register_sequence(adev,
+						 xgpu_golden_settings_fiji_a10,
+						 (const u32)ARRAY_SIZE(
+						 xgpu_golden_settings_fiji_a10));
+		amdgpu_program_register_sequence(adev,
+						 xgpu_fiji_golden_common_all,
+						 (const u32)ARRAY_SIZE(
+						 xgpu_fiji_golden_common_all));
+		break;
+	case CHIP_TONGA:
+		amdgpu_program_register_sequence(adev,
+						 xgpu_tonga_mgcg_cgcg_init,
+						 (const u32)ARRAY_SIZE(
+						 xgpu_tonga_mgcg_cgcg_init));
+		amdgpu_program_register_sequence(adev,
+						 xgpu_golden_settings_tonga_a11,
+						 (const u32)ARRAY_SIZE(
+						 xgpu_golden_settings_tonga_a11));
+		amdgpu_program_register_sequence(adev,
+						 xgpu_tonga_golden_common_all,
+						 (const u32)ARRAY_SIZE(
+						 xgpu_tonga_golden_common_all));
+		break;
+	default:
+		break;
+	}
+}
+
 static int xgpu_vi_early_init(void *handle)
 {
 	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization

[Alex Deucher]

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> GPU virtualization has different sequence from normal, change it.
>
> Signed-off-by: Frank Min <Frank.Min@amd.com>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   2 +
>  drivers/gpu/drm/amd/amdgpu/vi.c          |   6 +
>  drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c     | 267 +++++++++++++++++++++++++++++++
>  3 files changed, 275 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index eb2905e..e781c9c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -84,4 +84,6 @@ int amdgpu_put_gpu(struct amdgpu_device *adev);
>  /* access vf registers */
>  uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
>  void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
> +
> +void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev);
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c
> index 5229b4a2a..0d5e807 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vi.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vi.c
> @@ -285,6 +285,12 @@ static void vi_init_golden_registers(struct amdgpu_device *adev)
>         /* Some of the registers might be dependent on GRBM_GFX_INDEX */
>         mutex_lock(&adev->grbm_idx_mutex);
>
> +       if (adev->flags & AMD_IS_VF) {
> +               amdgpu_xgpu_init_golden_registers(adev);
> +               mutex_unlock(&adev->grbm_idx_mutex);
> +               return;
> +       }
> +
>         switch (adev->asic_type) {
>         case CHIP_TOPAZ:
>                 amdgpu_program_register_sequence(adev,
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> index e5d517f..fa1ee8f 100644
> --- a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> @@ -42,6 +42,273 @@
>  #include "dce/dce_10_0_sh_mask.h"
>  #include "smu/smu_7_1_3_d.h"
>
> +static const u32 xgpu_fiji_mgcg_cgcg_init[] = {
> +       mmRLC_CGTT_MGCG_OVERRIDE, 0xffffffff, 0xffffffff,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmCB_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_BCI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CP_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CPC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CPF_CLK_CTRL, 0xffffffff, 0x40000100,
> +       mmCGTT_DRM_CLK_CTRL0, 0xffffffff, 0x00600100,
> +       mmCGTT_GDS_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_IA_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmCGTT_PA_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_WD_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmCGTT_PC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_RLC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SPI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SQ_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SQG_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL0, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL1, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL2, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL3, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL4, 0xffffffff, 0x00000100,
> +       mmCGTT_TCI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_TCP_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_VGT_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmDB_CGTT_CLK_CTRL_0, 0xffffffff, 0x00000100,
> +       mmTA_CGTT_CTRL, 0xffffffff, 0x00000100,
> +       mmTCA_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmTCC_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmTD_CGTT_CTRL, 0xffffffff, 0x00000100,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmCGTS_SM_CTRL_REG, 0xffffffff, 0x96e00200,
> +       mmCP_RB_WPTR_POLL_CNTL, 0xffffffff, 0x00900100,
> +       mmRLC_CGCG_CGLS_CTRL, 0xffffffff, 0x0020003c,
> +       mmPCIE_INDEX, 0xffffffff, 0x0140001c,
> +       mmPCIE_DATA, 0x000f0000, 0x00000000,
> +       mmSMC_IND_INDEX_4, 0xffffffff, 0xC060000C,
> +       mmSMC_IND_DATA_4, 0xc0000fff, 0x00000100,
> +       mmXDMA_CLOCK_GATING_CNTL, 0xffffffff, 0x00000100,
> +       mmXDMA_MEM_POWER_CNTL, 0x00000101, 0x00000000,
> +       mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104,
> +       mmCGTT_DRM_CLK_CTRL0, 0xff000fff, 0x00000100,
> +       mmHDP_XDP_CGTT_BLK_CTRL, 0xc0000fff, 0x00000104,
> +       mmCP_MEM_SLP_CNTL, 0x00000001, 0x00000001,
> +       mmSDMA0_CLK_CTRL, 0xff000ff0, 0x00000100,
> +       mmSDMA1_CLK_CTRL, 0xff000ff0, 0x00000100,
> +};
> +
> +static const u32 xgpu_golden_settings_fiji_a10[] = {
> +       mmCB_HW_CONTROL_3, 0x000001ff, 0x00000040,
> +       mmDB_DEBUG2, 0xf00fffff, 0x00000400,
> +       mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
> +       mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
> +       mmFBC_MISC, 0x1f311fff, 0x12300000,
> +       mmHDMI_CONTROL, 0x31000111, 0x00000011,
> +       mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
> +       mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
> +       mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
> +       mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
> +       mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
> +       mmTCP_ADDR_CONFIG, 0x000003ff, 0x000000ff,
> +       mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
> +       mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +};
> +
> +static const u32 xgpu_fiji_golden_common_all[] = {
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmPA_SC_RASTER_CONFIG, 0xffffffff, 0x3a00161a,
> +       mmPA_SC_RASTER_CONFIG_1, 0xffffffff, 0x0000002e,
> +       mmGB_ADDR_CONFIG, 0xffffffff, 0x22011003,
> +       mmSPI_RESOURCE_RESERVE_CU_0, 0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_CU_1, 0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_1, 0xffffffff, 0x00007FAF,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmSPI_CONFIG_CNTL_1, 0x0000000f, 0x00000009,
> +};
> +
> +static const u32 xgpu_tonga_mgcg_cgcg_init[] = {
> +       mmRLC_CGTT_MGCG_OVERRIDE,   0xffffffff, 0xffffffff,
> +       mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
> +       mmCB_CGTT_SCLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_BCI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_CP_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_CPC_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_CPF_CLK_CTRL,        0xffffffff, 0x40000100,
> +       mmCGTT_DRM_CLK_CTRL0,       0xffffffff, 0x00600100,
> +       mmCGTT_GDS_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_IA_CLK_CTRL,         0xffffffff, 0x06000100,
> +       mmCGTT_PA_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_WD_CLK_CTRL,         0xffffffff, 0x06000100,
> +       mmCGTT_PC_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_RLC_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SC_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_SPI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SQ_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_SQG_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL0,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL1,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL2,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL3,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL4,        0xffffffff, 0x00000100,
> +       mmCGTT_TCI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_TCP_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_VGT_CLK_CTRL,        0xffffffff, 0x06000100,
> +       mmDB_CGTT_CLK_CTRL_0,       0xffffffff, 0x00000100,
> +       mmTA_CGTT_CTRL,             0xffffffff, 0x00000100,
> +       mmTCA_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
> +       mmTCC_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
> +       mmTD_CGTT_CTRL,             0xffffffff, 0x00000100,
> +       mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
> +       mmCGTS_CU0_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU0_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU0_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
> +       mmCGTS_CU0_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU0_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU1_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU1_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU1_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU1_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU1_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU2_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU2_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU2_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU2_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU2_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU3_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU3_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU3_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU3_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU3_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU4_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU4_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU4_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
> +       mmCGTS_CU4_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU4_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU5_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU5_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU5_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU5_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU5_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU6_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU6_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU6_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU6_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU6_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU7_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU7_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU7_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU7_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU7_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_SM_CTRL_REG,         0xffffffff, 0x96e00200,
> +       mmCP_RB_WPTR_POLL_CNTL,     0xffffffff, 0x00900100,
> +       mmRLC_CGCG_CGLS_CTRL,       0xffffffff, 0x0020003c,
> +       mmPCIE_INDEX,               0xffffffff, 0x0140001c,
> +       mmPCIE_DATA,                0x000f0000, 0x00000000,
> +       mmSMC_IND_INDEX_4,          0xffffffff, 0xC060000C,
> +       mmSMC_IND_DATA_4,           0xc0000fff, 0x00000100,
> +       mmXDMA_CLOCK_GATING_CNTL,   0xffffffff, 0x00000100,
> +       mmXDMA_MEM_POWER_CNTL,      0x00000101, 0x00000000,
> +       mmMC_MEM_POWER_LS,          0xffffffff, 0x00000104,
> +       mmCGTT_DRM_CLK_CTRL0,       0xff000fff, 0x00000100,
> +       mmHDP_XDP_CGTT_BLK_CTRL,    0xc0000fff, 0x00000104,
> +       mmCP_MEM_SLP_CNTL,          0x00000001, 0x00000001,
> +       mmSDMA0_CLK_CTRL,           0xff000ff0, 0x00000100,
> +       mmSDMA1_CLK_CTRL,           0xff000ff0, 0x00000100,
> +};
> +
> +static const u32 xgpu_golden_settings_tonga_a11[] = {
> +       mmCB_HW_CONTROL, 0xfffdf3cf, 0x00007208,
> +       mmCB_HW_CONTROL_3, 0x00000040, 0x00000040,
> +       mmDB_DEBUG2, 0xf00fffff, 0x00000400,
> +       mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
> +       mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
> +       mmFBC_MISC, 0x1f311fff, 0x12300000,
> +       mmGB_GPU_ID, 0x0000000f, 0x00000000,
> +       mmHDMI_CONTROL, 0x31000111, 0x00000011,
> +       mmMC_ARB_WTM_GRPWT_RD, 0x00000003, 0x00000000,
> +       mmMC_HUB_RDREQ_DMIF_LIMIT, 0x0000007f, 0x00000028,
> +       mmMC_HUB_WDP_UMC, 0x00007fb6, 0x00000991,
> +       mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
> +       mmPA_SC_FIFO_DEPTH_CNTL, 0x000003ff, 0x000000fc,
> +       mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
> +       mmRLC_CGCG_CGLS_CTRL, 0x00000003, 0x0000003c,
> +       mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA0_CLK_CTRL, 0xff000fff, 0x00000000,
> +       mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA1_CLK_CTRL, 0xff000fff, 0x00000000,
> +       mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
> +       mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
> +       mmTCC_CTRL, 0x00100000, 0xf31fff7f,
> +       mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
> +       mmTCP_ADDR_CONFIG, 0x000003ff, 0x000002fb,
> +       mmTCP_CHAN_STEER_HI, 0xffffffff, 0x0000543b,
> +       mmTCP_CHAN_STEER_LO, 0xffffffff, 0xa9210876,
> +       mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
> +       mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +};
> +
> +static const u32 xgpu_tonga_golden_common_all[] = {
> +       mmGRBM_GFX_INDEX,               0xffffffff, 0xe0000000,
> +       mmPA_SC_RASTER_CONFIG,          0xffffffff, 0x16000012,
> +       mmPA_SC_RASTER_CONFIG_1,        0xffffffff, 0x0000002A,
> +       mmGB_ADDR_CONFIG,               0xffffffff, 0x22011002,
> +       mmSPI_RESOURCE_RESERVE_CU_0,    0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_CU_1,    0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
> +};
> +
> +void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev)
> +{
> +       switch (adev->asic_type) {
> +       case CHIP_FIJI:
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_fiji_mgcg_cgcg_init,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_fiji_mgcg_cgcg_init));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_golden_settings_fiji_a10,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_golden_settings_fiji_a10));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_fiji_golden_common_all,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_fiji_golden_common_all));
> +               break;
> +       case CHIP_TONGA:
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_tonga_mgcg_cgcg_init,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_tonga_mgcg_cgcg_init));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_golden_settings_tonga_a11,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_golden_settings_tonga_a11));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_tonga_golden_common_all,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_tonga_golden_common_all));
> +               break;
> +       default:
> +               break;
> +       }
> +}

Cleaner to just put this in vi.c.

Alex

> +
>  static int xgpu_vi_early_init(void *handle)
>  {
>         struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

答复: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization

[Liu, Monk]

[-- Attachment #1.1: Type: text/plain, Size: 18526 bytes --]

Hi Alex


I agree with you that this patch's GOLDEN setting programming should be put in VI.C, but I found a hardware issue :


original linux logic is that we set golden setting registers separately within each IP's hw init routine, but for TONGA VF, it is really strange that we must set all GOLDEN setting value to chip in one shoot (means we use one routine to programing all IP's golden setting to registers), ortherwise we found TONGA vf  just failed in RING TEST.


and I admit I don't know why (I checked windows CAIL code, it is also set all golden setting registers in one routine)


BR Monk

________________________________
发件人: Alex Deucher <alexdeucher@gmail.com>
发送时间: 2016年12月20日 7:37:06
收件人: Yu, Xiangliang
抄送: amd-gfx list; dl.SRDC_SW_GPUVirtualization; Min, Frank; Liu, Monk
主题: Re: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com> wrote:
> GPU virtualization has different sequence from normal, change it.
>
> Signed-off-by: Frank Min <Frank.Min@amd.com>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   2 +
>  drivers/gpu/drm/amd/amdgpu/vi.c          |   6 +
>  drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c     | 267 +++++++++++++++++++++++++++++++
>  3 files changed, 275 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index eb2905e..e781c9c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -84,4 +84,6 @@ int amdgpu_put_gpu(struct amdgpu_device *adev);
>  /* access vf registers */
>  uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
>  void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
> +
> +void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev);
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c
> index 5229b4a2a..0d5e807 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vi.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vi.c
> @@ -285,6 +285,12 @@ static void vi_init_golden_registers(struct amdgpu_device *adev)
>         /* Some of the registers might be dependent on GRBM_GFX_INDEX */
>         mutex_lock(&adev->grbm_idx_mutex);
>
> +       if (adev->flags & AMD_IS_VF) {
> +               amdgpu_xgpu_init_golden_registers(adev);
> +               mutex_unlock(&adev->grbm_idx_mutex);
> +               return;
> +       }
> +
>         switch (adev->asic_type) {
>         case CHIP_TOPAZ:
>                 amdgpu_program_register_sequence(adev,
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> index e5d517f..fa1ee8f 100644
> --- a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> @@ -42,6 +42,273 @@
>  #include "dce/dce_10_0_sh_mask.h"
>  #include "smu/smu_7_1_3_d.h"
>
> +static const u32 xgpu_fiji_mgcg_cgcg_init[] = {
> +       mmRLC_CGTT_MGCG_OVERRIDE, 0xffffffff, 0xffffffff,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmCB_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_BCI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CP_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CPC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CPF_CLK_CTRL, 0xffffffff, 0x40000100,
> +       mmCGTT_DRM_CLK_CTRL0, 0xffffffff, 0x00600100,
> +       mmCGTT_GDS_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_IA_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmCGTT_PA_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_WD_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmCGTT_PC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_RLC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SPI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SQ_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SQG_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL0, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL1, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL2, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL3, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL4, 0xffffffff, 0x00000100,
> +       mmCGTT_TCI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_TCP_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_VGT_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmDB_CGTT_CLK_CTRL_0, 0xffffffff, 0x00000100,
> +       mmTA_CGTT_CTRL, 0xffffffff, 0x00000100,
> +       mmTCA_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmTCC_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmTD_CGTT_CTRL, 0xffffffff, 0x00000100,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmCGTS_SM_CTRL_REG, 0xffffffff, 0x96e00200,
> +       mmCP_RB_WPTR_POLL_CNTL, 0xffffffff, 0x00900100,
> +       mmRLC_CGCG_CGLS_CTRL, 0xffffffff, 0x0020003c,
> +       mmPCIE_INDEX, 0xffffffff, 0x0140001c,
> +       mmPCIE_DATA, 0x000f0000, 0x00000000,
> +       mmSMC_IND_INDEX_4, 0xffffffff, 0xC060000C,
> +       mmSMC_IND_DATA_4, 0xc0000fff, 0x00000100,
> +       mmXDMA_CLOCK_GATING_CNTL, 0xffffffff, 0x00000100,
> +       mmXDMA_MEM_POWER_CNTL, 0x00000101, 0x00000000,
> +       mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104,
> +       mmCGTT_DRM_CLK_CTRL0, 0xff000fff, 0x00000100,
> +       mmHDP_XDP_CGTT_BLK_CTRL, 0xc0000fff, 0x00000104,
> +       mmCP_MEM_SLP_CNTL, 0x00000001, 0x00000001,
> +       mmSDMA0_CLK_CTRL, 0xff000ff0, 0x00000100,
> +       mmSDMA1_CLK_CTRL, 0xff000ff0, 0x00000100,
> +};
> +
> +static const u32 xgpu_golden_settings_fiji_a10[] = {
> +       mmCB_HW_CONTROL_3, 0x000001ff, 0x00000040,
> +       mmDB_DEBUG2, 0xf00fffff, 0x00000400,
> +       mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
> +       mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
> +       mmFBC_MISC, 0x1f311fff, 0x12300000,
> +       mmHDMI_CONTROL, 0x31000111, 0x00000011,
> +       mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
> +       mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
> +       mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
> +       mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
> +       mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
> +       mmTCP_ADDR_CONFIG, 0x000003ff, 0x000000ff,
> +       mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
> +       mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +};
> +
> +static const u32 xgpu_fiji_golden_common_all[] = {
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmPA_SC_RASTER_CONFIG, 0xffffffff, 0x3a00161a,
> +       mmPA_SC_RASTER_CONFIG_1, 0xffffffff, 0x0000002e,
> +       mmGB_ADDR_CONFIG, 0xffffffff, 0x22011003,
> +       mmSPI_RESOURCE_RESERVE_CU_0, 0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_CU_1, 0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_1, 0xffffffff, 0x00007FAF,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmSPI_CONFIG_CNTL_1, 0x0000000f, 0x00000009,
> +};
> +
> +static const u32 xgpu_tonga_mgcg_cgcg_init[] = {
> +       mmRLC_CGTT_MGCG_OVERRIDE,   0xffffffff, 0xffffffff,
> +       mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
> +       mmCB_CGTT_SCLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_BCI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_CP_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_CPC_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_CPF_CLK_CTRL,        0xffffffff, 0x40000100,
> +       mmCGTT_DRM_CLK_CTRL0,       0xffffffff, 0x00600100,
> +       mmCGTT_GDS_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_IA_CLK_CTRL,         0xffffffff, 0x06000100,
> +       mmCGTT_PA_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_WD_CLK_CTRL,         0xffffffff, 0x06000100,
> +       mmCGTT_PC_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_RLC_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SC_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_SPI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SQ_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_SQG_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL0,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL1,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL2,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL3,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL4,        0xffffffff, 0x00000100,
> +       mmCGTT_TCI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_TCP_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_VGT_CLK_CTRL,        0xffffffff, 0x06000100,
> +       mmDB_CGTT_CLK_CTRL_0,       0xffffffff, 0x00000100,
> +       mmTA_CGTT_CTRL,             0xffffffff, 0x00000100,
> +       mmTCA_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
> +       mmTCC_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
> +       mmTD_CGTT_CTRL,             0xffffffff, 0x00000100,
> +       mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
> +       mmCGTS_CU0_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU0_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU0_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
> +       mmCGTS_CU0_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU0_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU1_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU1_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU1_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU1_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU1_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU2_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU2_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU2_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU2_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU2_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU3_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU3_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU3_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU3_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU3_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU4_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU4_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU4_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
> +       mmCGTS_CU4_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU4_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU5_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU5_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU5_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU5_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU5_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU6_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU6_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU6_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU6_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU6_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU7_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU7_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU7_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU7_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU7_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_SM_CTRL_REG,         0xffffffff, 0x96e00200,
> +       mmCP_RB_WPTR_POLL_CNTL,     0xffffffff, 0x00900100,
> +       mmRLC_CGCG_CGLS_CTRL,       0xffffffff, 0x0020003c,
> +       mmPCIE_INDEX,               0xffffffff, 0x0140001c,
> +       mmPCIE_DATA,                0x000f0000, 0x00000000,
> +       mmSMC_IND_INDEX_4,          0xffffffff, 0xC060000C,
> +       mmSMC_IND_DATA_4,           0xc0000fff, 0x00000100,
> +       mmXDMA_CLOCK_GATING_CNTL,   0xffffffff, 0x00000100,
> +       mmXDMA_MEM_POWER_CNTL,      0x00000101, 0x00000000,
> +       mmMC_MEM_POWER_LS,          0xffffffff, 0x00000104,
> +       mmCGTT_DRM_CLK_CTRL0,       0xff000fff, 0x00000100,
> +       mmHDP_XDP_CGTT_BLK_CTRL,    0xc0000fff, 0x00000104,
> +       mmCP_MEM_SLP_CNTL,          0x00000001, 0x00000001,
> +       mmSDMA0_CLK_CTRL,           0xff000ff0, 0x00000100,
> +       mmSDMA1_CLK_CTRL,           0xff000ff0, 0x00000100,
> +};
> +
> +static const u32 xgpu_golden_settings_tonga_a11[] = {
> +       mmCB_HW_CONTROL, 0xfffdf3cf, 0x00007208,
> +       mmCB_HW_CONTROL_3, 0x00000040, 0x00000040,
> +       mmDB_DEBUG2, 0xf00fffff, 0x00000400,
> +       mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
> +       mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
> +       mmFBC_MISC, 0x1f311fff, 0x12300000,
> +       mmGB_GPU_ID, 0x0000000f, 0x00000000,
> +       mmHDMI_CONTROL, 0x31000111, 0x00000011,
> +       mmMC_ARB_WTM_GRPWT_RD, 0x00000003, 0x00000000,
> +       mmMC_HUB_RDREQ_DMIF_LIMIT, 0x0000007f, 0x00000028,
> +       mmMC_HUB_WDP_UMC, 0x00007fb6, 0x00000991,
> +       mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
> +       mmPA_SC_FIFO_DEPTH_CNTL, 0x000003ff, 0x000000fc,
> +       mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
> +       mmRLC_CGCG_CGLS_CTRL, 0x00000003, 0x0000003c,
> +       mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA0_CLK_CTRL, 0xff000fff, 0x00000000,
> +       mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA1_CLK_CTRL, 0xff000fff, 0x00000000,
> +       mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
> +       mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
> +       mmTCC_CTRL, 0x00100000, 0xf31fff7f,
> +       mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
> +       mmTCP_ADDR_CONFIG, 0x000003ff, 0x000002fb,
> +       mmTCP_CHAN_STEER_HI, 0xffffffff, 0x0000543b,
> +       mmTCP_CHAN_STEER_LO, 0xffffffff, 0xa9210876,
> +       mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
> +       mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +};
> +
> +static const u32 xgpu_tonga_golden_common_all[] = {
> +       mmGRBM_GFX_INDEX,               0xffffffff, 0xe0000000,
> +       mmPA_SC_RASTER_CONFIG,          0xffffffff, 0x16000012,
> +       mmPA_SC_RASTER_CONFIG_1,        0xffffffff, 0x0000002A,
> +       mmGB_ADDR_CONFIG,               0xffffffff, 0x22011002,
> +       mmSPI_RESOURCE_RESERVE_CU_0,    0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_CU_1,    0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
> +};
> +
> +void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev)
> +{
> +       switch (adev->asic_type) {
> +       case CHIP_FIJI:
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_fiji_mgcg_cgcg_init,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_fiji_mgcg_cgcg_init));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_golden_settings_fiji_a10,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_golden_settings_fiji_a10));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_fiji_golden_common_all,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_fiji_golden_common_all));
> +               break;
> +       case CHIP_TONGA:
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_tonga_mgcg_cgcg_init,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_tonga_mgcg_cgcg_init));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_golden_settings_tonga_a11,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_golden_settings_tonga_a11));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_tonga_golden_common_all,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_tonga_golden_common_all));
> +               break;
> +       default:
> +               break;
> +       }
> +}

Cleaner to just put this in vi.c.

Alex

> +
>  static int xgpu_vi_early_init(void *handle)
>  {
>         struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 37913 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization

[Yu, Xiangliang]

[-- Attachment #1.1: Type: text/plain, Size: 19139 bytes --]

Thank monk’s comments.


From: Liu, Monk
Sent: Tuesday, December 20, 2016 12:09 PM
To: Alex Deucher <alexdeucher@gmail.com>; Yu, Xiangliang <Xiangliang.Yu@amd.com>
Cc: amd-gfx list <amd-gfx@lists.freedesktop.org>; dl.SRDC_SW_GPUVirtualization <dl.SRDC_SW_GPUVirtualization@amd.com>; Min, Frank <Frank.Min@amd.com>
Subject: 答复: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization


Hi Alex



I agree with you that this patch's GOLDEN setting programming should be put in VI.C, but I found a hardware issue :



original linux logic is that we set golden setting registers separately within each IP's hw init routine, but for TONGA VF, it is really strange that we must set all GOLDEN setting value to chip in one shoot (means we use one routine to programing all IP's golden setting to registers), ortherwise we found TONGA vf  just failed in RING TEST.



and I admit I don't know why (I checked windows CAIL code, it is also set all golden setting registers in one routine)



BR Monk

________________________________
发件人: Alex Deucher <alexdeucher@gmail.com<mailto:alexdeucher@gmail.com>>
发送时间: 2016年12月20日 7:37:06
收件人: Yu, Xiangliang
抄送: amd-gfx list; dl.SRDC_SW_GPUVirtualization; Min, Frank; Liu, Monk
主题: Re: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>> wrote:
> GPU virtualization has different sequence from normal, change it.
>
> Signed-off-by: Frank Min <Frank.Min@amd.com<mailto:Frank.Min@amd.com>>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   2 +
>  drivers/gpu/drm/amd/amdgpu/vi.c          |   6 +
>  drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c     | 267 +++++++++++++++++++++++++++++++
>  3 files changed, 275 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index eb2905e..e781c9c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -84,4 +84,6 @@ int amdgpu_put_gpu(struct amdgpu_device *adev);
>  /* access vf registers */
>  uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
>  void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
> +
> +void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev);
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c
> index 5229b4a2a..0d5e807 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vi.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vi.c
> @@ -285,6 +285,12 @@ static void vi_init_golden_registers(struct amdgpu_device *adev)
>         /* Some of the registers might be dependent on GRBM_GFX_INDEX */
>         mutex_lock(&adev->grbm_idx_mutex);
>
> +       if (adev->flags & AMD_IS_VF) {
> +               amdgpu_xgpu_init_golden_registers(adev);
> +               mutex_unlock(&adev->grbm_idx_mutex);
> +               return;
> +       }
> +
>         switch (adev->asic_type) {
>         case CHIP_TOPAZ:
>                 amdgpu_program_register_sequence(adev,
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> index e5d517f..fa1ee8f 100644
> --- a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> @@ -42,6 +42,273 @@
>  #include "dce/dce_10_0_sh_mask.h"
>  #include "smu/smu_7_1_3_d.h"
>
> +static const u32 xgpu_fiji_mgcg_cgcg_init[] = {
> +       mmRLC_CGTT_MGCG_OVERRIDE, 0xffffffff, 0xffffffff,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmCB_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_BCI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CP_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CPC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CPF_CLK_CTRL, 0xffffffff, 0x40000100,
> +       mmCGTT_DRM_CLK_CTRL0, 0xffffffff, 0x00600100,
> +       mmCGTT_GDS_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_IA_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmCGTT_PA_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_WD_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmCGTT_PC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_RLC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SPI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SQ_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SQG_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL0, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL1, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL2, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL3, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL4, 0xffffffff, 0x00000100,
> +       mmCGTT_TCI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_TCP_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_VGT_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmDB_CGTT_CLK_CTRL_0, 0xffffffff, 0x00000100,
> +       mmTA_CGTT_CTRL, 0xffffffff, 0x00000100,
> +       mmTCA_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmTCC_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmTD_CGTT_CTRL, 0xffffffff, 0x00000100,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmCGTS_SM_CTRL_REG, 0xffffffff, 0x96e00200,
> +       mmCP_RB_WPTR_POLL_CNTL, 0xffffffff, 0x00900100,
> +       mmRLC_CGCG_CGLS_CTRL, 0xffffffff, 0x0020003c,
> +       mmPCIE_INDEX, 0xffffffff, 0x0140001c,
> +       mmPCIE_DATA, 0x000f0000, 0x00000000,
> +       mmSMC_IND_INDEX_4, 0xffffffff, 0xC060000C,
> +       mmSMC_IND_DATA_4, 0xc0000fff, 0x00000100,
> +       mmXDMA_CLOCK_GATING_CNTL, 0xffffffff, 0x00000100,
> +       mmXDMA_MEM_POWER_CNTL, 0x00000101, 0x00000000,
> +       mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104,
> +       mmCGTT_DRM_CLK_CTRL0, 0xff000fff, 0x00000100,
> +       mmHDP_XDP_CGTT_BLK_CTRL, 0xc0000fff, 0x00000104,
> +       mmCP_MEM_SLP_CNTL, 0x00000001, 0x00000001,
> +       mmSDMA0_CLK_CTRL, 0xff000ff0, 0x00000100,
> +       mmSDMA1_CLK_CTRL, 0xff000ff0, 0x00000100,
> +};
> +
> +static const u32 xgpu_golden_settings_fiji_a10[] = {
> +       mmCB_HW_CONTROL_3, 0x000001ff, 0x00000040,
> +       mmDB_DEBUG2, 0xf00fffff, 0x00000400,
> +       mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
> +       mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
> +       mmFBC_MISC, 0x1f311fff, 0x12300000,
> +       mmHDMI_CONTROL, 0x31000111, 0x00000011,
> +       mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
> +       mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
> +       mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
> +       mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
> +       mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
> +       mmTCP_ADDR_CONFIG, 0x000003ff, 0x000000ff,
> +       mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
> +       mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +};
> +
> +static const u32 xgpu_fiji_golden_common_all[] = {
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmPA_SC_RASTER_CONFIG, 0xffffffff, 0x3a00161a,
> +       mmPA_SC_RASTER_CONFIG_1, 0xffffffff, 0x0000002e,
> +       mmGB_ADDR_CONFIG, 0xffffffff, 0x22011003,
> +       mmSPI_RESOURCE_RESERVE_CU_0, 0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_CU_1, 0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_1, 0xffffffff, 0x00007FAF,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmSPI_CONFIG_CNTL_1, 0x0000000f, 0x00000009,
> +};
> +
> +static const u32 xgpu_tonga_mgcg_cgcg_init[] = {
> +       mmRLC_CGTT_MGCG_OVERRIDE,   0xffffffff, 0xffffffff,
> +       mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
> +       mmCB_CGTT_SCLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_BCI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_CP_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_CPC_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_CPF_CLK_CTRL,        0xffffffff, 0x40000100,
> +       mmCGTT_DRM_CLK_CTRL0,       0xffffffff, 0x00600100,
> +       mmCGTT_GDS_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_IA_CLK_CTRL,         0xffffffff, 0x06000100,
> +       mmCGTT_PA_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_WD_CLK_CTRL,         0xffffffff, 0x06000100,
> +       mmCGTT_PC_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_RLC_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SC_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_SPI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SQ_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_SQG_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL0,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL1,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL2,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL3,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL4,        0xffffffff, 0x00000100,
> +       mmCGTT_TCI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_TCP_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_VGT_CLK_CTRL,        0xffffffff, 0x06000100,
> +       mmDB_CGTT_CLK_CTRL_0,       0xffffffff, 0x00000100,
> +       mmTA_CGTT_CTRL,             0xffffffff, 0x00000100,
> +       mmTCA_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
> +       mmTCC_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
> +       mmTD_CGTT_CTRL,             0xffffffff, 0x00000100,
> +       mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
> +       mmCGTS_CU0_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU0_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU0_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
> +       mmCGTS_CU0_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU0_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU1_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU1_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU1_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU1_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU1_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU2_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU2_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU2_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU2_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU2_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU3_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU3_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU3_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU3_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU3_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU4_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU4_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU4_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
> +       mmCGTS_CU4_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU4_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU5_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU5_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU5_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU5_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU5_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU6_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU6_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU6_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU6_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU6_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU7_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU7_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU7_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU7_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU7_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_SM_CTRL_REG,         0xffffffff, 0x96e00200,
> +       mmCP_RB_WPTR_POLL_CNTL,     0xffffffff, 0x00900100,
> +       mmRLC_CGCG_CGLS_CTRL,       0xffffffff, 0x0020003c,
> +       mmPCIE_INDEX,               0xffffffff, 0x0140001c,
> +       mmPCIE_DATA,                0x000f0000, 0x00000000,
> +       mmSMC_IND_INDEX_4,          0xffffffff, 0xC060000C,
> +       mmSMC_IND_DATA_4,           0xc0000fff, 0x00000100,
> +       mmXDMA_CLOCK_GATING_CNTL,   0xffffffff, 0x00000100,
> +       mmXDMA_MEM_POWER_CNTL,      0x00000101, 0x00000000,
> +       mmMC_MEM_POWER_LS,          0xffffffff, 0x00000104,
> +       mmCGTT_DRM_CLK_CTRL0,       0xff000fff, 0x00000100,
> +       mmHDP_XDP_CGTT_BLK_CTRL,    0xc0000fff, 0x00000104,
> +       mmCP_MEM_SLP_CNTL,          0x00000001, 0x00000001,
> +       mmSDMA0_CLK_CTRL,           0xff000ff0, 0x00000100,
> +       mmSDMA1_CLK_CTRL,           0xff000ff0, 0x00000100,
> +};
> +
> +static const u32 xgpu_golden_settings_tonga_a11[] = {
> +       mmCB_HW_CONTROL, 0xfffdf3cf, 0x00007208,
> +       mmCB_HW_CONTROL_3, 0x00000040, 0x00000040,
> +       mmDB_DEBUG2, 0xf00fffff, 0x00000400,
> +       mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
> +       mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
> +       mmFBC_MISC, 0x1f311fff, 0x12300000,
> +       mmGB_GPU_ID, 0x0000000f, 0x00000000,
> +       mmHDMI_CONTROL, 0x31000111, 0x00000011,
> +       mmMC_ARB_WTM_GRPWT_RD, 0x00000003, 0x00000000,
> +       mmMC_HUB_RDREQ_DMIF_LIMIT, 0x0000007f, 0x00000028,
> +       mmMC_HUB_WDP_UMC, 0x00007fb6, 0x00000991,
> +       mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
> +       mmPA_SC_FIFO_DEPTH_CNTL, 0x000003ff, 0x000000fc,
> +       mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
> +       mmRLC_CGCG_CGLS_CTRL, 0x00000003, 0x0000003c,
> +       mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA0_CLK_CTRL, 0xff000fff, 0x00000000,
> +       mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA1_CLK_CTRL, 0xff000fff, 0x00000000,
> +       mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
> +       mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
> +       mmTCC_CTRL, 0x00100000, 0xf31fff7f,
> +       mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
> +       mmTCP_ADDR_CONFIG, 0x000003ff, 0x000002fb,
> +       mmTCP_CHAN_STEER_HI, 0xffffffff, 0x0000543b,
> +       mmTCP_CHAN_STEER_LO, 0xffffffff, 0xa9210876,
> +       mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
> +       mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +};
> +
> +static const u32 xgpu_tonga_golden_common_all[] = {
> +       mmGRBM_GFX_INDEX,               0xffffffff, 0xe0000000,
> +       mmPA_SC_RASTER_CONFIG,          0xffffffff, 0x16000012,
> +       mmPA_SC_RASTER_CONFIG_1,        0xffffffff, 0x0000002A,
> +       mmGB_ADDR_CONFIG,               0xffffffff, 0x22011002,
> +       mmSPI_RESOURCE_RESERVE_CU_0,    0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_CU_1,    0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
> +};
> +
> +void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev)
> +{
> +       switch (adev->asic_type) {
> +       case CHIP_FIJI:
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_fiji_mgcg_cgcg_init,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_fiji_mgcg_cgcg_init));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_golden_settings_fiji_a10,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_golden_settings_fiji_a10));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_fiji_golden_common_all,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_fiji_golden_common_all));
> +               break;
> +       case CHIP_TONGA:
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_tonga_mgcg_cgcg_init,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_tonga_mgcg_cgcg_init));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_golden_settings_tonga_a11,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_golden_settings_tonga_a11));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_tonga_golden_common_all,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_tonga_golden_common_all));
> +               break;
> +       default:
> +               break;
> +       }
> +}

Cleaner to just put this in vi.c.

Alex

> +
>  static int xgpu_vi_early_init(void *handle)
>  {
>         struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 44115 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization

[Deucher, Alexander]

[-- Attachment #1.1: Type: text/plain, Size: 19089 bytes --]

I don't mind keeping the list together.

Alex

From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf Of Liu, Monk
Sent: Monday, December 19, 2016 11:09 PM
To: Alex Deucher; Yu, Xiangliang
Cc: Min, Frank; dl.SRDC_SW_GPUVirtualization; amd-gfx list
Subject: 答复: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization


Hi Alex



I agree with you that this patch's GOLDEN setting programming should be put in VI.C, but I found a hardware issue :



original linux logic is that we set golden setting registers separately within each IP's hw init routine, but for TONGA VF, it is really strange that we must set all GOLDEN setting value to chip in one shoot (means we use one routine to programing all IP's golden setting to registers), ortherwise we found TONGA vf  just failed in RING TEST.



and I admit I don't know why (I checked windows CAIL code, it is also set all golden setting registers in one routine)



BR Monk

________________________________
发件人: Alex Deucher <alexdeucher@gmail.com<mailto:alexdeucher@gmail.com>>
发送时间: 2016年12月20日 7:37:06
收件人: Yu, Xiangliang
抄送: amd-gfx list; dl.SRDC_SW_GPUVirtualization; Min, Frank; Liu, Monk
主题: Re: [PATCH 21/23] drm/amdgpu: change golden register program sequence of virtualization

On Sat, Dec 17, 2016 at 11:16 AM, Xiangliang Yu <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>> wrote:
> GPU virtualization has different sequence from normal, change it.
>
> Signed-off-by: Frank Min <Frank.Min@amd.com<mailto:Frank.Min@amd.com>>
> Signed-off-by: Monk Liu <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>
> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com<mailto:Xiangliang.Yu@amd.com>>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |   2 +
>  drivers/gpu/drm/amd/amdgpu/vi.c          |   6 +
>  drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c     | 267 +++++++++++++++++++++++++++++++
>  3 files changed, 275 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index eb2905e..e781c9c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -84,4 +84,6 @@ int amdgpu_put_gpu(struct amdgpu_device *adev);
>  /* access vf registers */
>  uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
>  void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
> +
> +void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev);
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c
> index 5229b4a2a..0d5e807 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vi.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vi.c
> @@ -285,6 +285,12 @@ static void vi_init_golden_registers(struct amdgpu_device *adev)
>         /* Some of the registers might be dependent on GRBM_GFX_INDEX */
>         mutex_lock(&adev->grbm_idx_mutex);
>
> +       if (adev->flags & AMD_IS_VF) {
> +               amdgpu_xgpu_init_golden_registers(adev);
> +               mutex_unlock(&adev->grbm_idx_mutex);
> +               return;
> +       }
> +
>         switch (adev->asic_type) {
>         case CHIP_TOPAZ:
>                 amdgpu_program_register_sequence(adev,
> diff --git a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> index e5d517f..fa1ee8f 100644
> --- a/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> +++ b/drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c
> @@ -42,6 +42,273 @@
>  #include "dce/dce_10_0_sh_mask.h"
>  #include "smu/smu_7_1_3_d.h"
>
> +static const u32 xgpu_fiji_mgcg_cgcg_init[] = {
> +       mmRLC_CGTT_MGCG_OVERRIDE, 0xffffffff, 0xffffffff,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmCB_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_BCI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CP_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CPC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_CPF_CLK_CTRL, 0xffffffff, 0x40000100,
> +       mmCGTT_DRM_CLK_CTRL0, 0xffffffff, 0x00600100,
> +       mmCGTT_GDS_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_IA_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmCGTT_PA_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_WD_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmCGTT_PC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_RLC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SC_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SPI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SQ_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SQG_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL0, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL1, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL2, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL3, 0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL4, 0xffffffff, 0x00000100,
> +       mmCGTT_TCI_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_TCP_CLK_CTRL, 0xffffffff, 0x00000100,
> +       mmCGTT_VGT_CLK_CTRL, 0xffffffff, 0x06000100,
> +       mmDB_CGTT_CLK_CTRL_0, 0xffffffff, 0x00000100,
> +       mmTA_CGTT_CTRL, 0xffffffff, 0x00000100,
> +       mmTCA_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmTCC_CGTT_SCLK_CTRL, 0xffffffff, 0x00000100,
> +       mmTD_CGTT_CTRL, 0xffffffff, 0x00000100,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmCGTS_SM_CTRL_REG, 0xffffffff, 0x96e00200,
> +       mmCP_RB_WPTR_POLL_CNTL, 0xffffffff, 0x00900100,
> +       mmRLC_CGCG_CGLS_CTRL, 0xffffffff, 0x0020003c,
> +       mmPCIE_INDEX, 0xffffffff, 0x0140001c,
> +       mmPCIE_DATA, 0x000f0000, 0x00000000,
> +       mmSMC_IND_INDEX_4, 0xffffffff, 0xC060000C,
> +       mmSMC_IND_DATA_4, 0xc0000fff, 0x00000100,
> +       mmXDMA_CLOCK_GATING_CNTL, 0xffffffff, 0x00000100,
> +       mmXDMA_MEM_POWER_CNTL, 0x00000101, 0x00000000,
> +       mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104,
> +       mmCGTT_DRM_CLK_CTRL0, 0xff000fff, 0x00000100,
> +       mmHDP_XDP_CGTT_BLK_CTRL, 0xc0000fff, 0x00000104,
> +       mmCP_MEM_SLP_CNTL, 0x00000001, 0x00000001,
> +       mmSDMA0_CLK_CTRL, 0xff000ff0, 0x00000100,
> +       mmSDMA1_CLK_CTRL, 0xff000ff0, 0x00000100,
> +};
> +
> +static const u32 xgpu_golden_settings_fiji_a10[] = {
> +       mmCB_HW_CONTROL_3, 0x000001ff, 0x00000040,
> +       mmDB_DEBUG2, 0xf00fffff, 0x00000400,
> +       mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
> +       mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
> +       mmFBC_MISC, 0x1f311fff, 0x12300000,
> +       mmHDMI_CONTROL, 0x31000111, 0x00000011,
> +       mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
> +       mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
> +       mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
> +       mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
> +       mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
> +       mmTCP_ADDR_CONFIG, 0x000003ff, 0x000000ff,
> +       mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
> +       mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +};
> +
> +static const u32 xgpu_fiji_golden_common_all[] = {
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmPA_SC_RASTER_CONFIG, 0xffffffff, 0x3a00161a,
> +       mmPA_SC_RASTER_CONFIG_1, 0xffffffff, 0x0000002e,
> +       mmGB_ADDR_CONFIG, 0xffffffff, 0x22011003,
> +       mmSPI_RESOURCE_RESERVE_CU_0, 0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_CU_1, 0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_1, 0xffffffff, 0x00007FAF,
> +       mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000,
> +       mmSPI_CONFIG_CNTL_1, 0x0000000f, 0x00000009,
> +};
> +
> +static const u32 xgpu_tonga_mgcg_cgcg_init[] = {
> +       mmRLC_CGTT_MGCG_OVERRIDE,   0xffffffff, 0xffffffff,
> +       mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
> +       mmCB_CGTT_SCLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_BCI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_CP_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_CPC_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_CPF_CLK_CTRL,        0xffffffff, 0x40000100,
> +       mmCGTT_DRM_CLK_CTRL0,       0xffffffff, 0x00600100,
> +       mmCGTT_GDS_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_IA_CLK_CTRL,         0xffffffff, 0x06000100,
> +       mmCGTT_PA_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_WD_CLK_CTRL,         0xffffffff, 0x06000100,
> +       mmCGTT_PC_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_RLC_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SC_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_SPI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SQ_CLK_CTRL,         0xffffffff, 0x00000100,
> +       mmCGTT_SQG_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL0,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL1,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL2,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL3,        0xffffffff, 0x00000100,
> +       mmCGTT_SX_CLK_CTRL4,        0xffffffff, 0x00000100,
> +       mmCGTT_TCI_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_TCP_CLK_CTRL,        0xffffffff, 0x00000100,
> +       mmCGTT_VGT_CLK_CTRL,        0xffffffff, 0x06000100,
> +       mmDB_CGTT_CLK_CTRL_0,       0xffffffff, 0x00000100,
> +       mmTA_CGTT_CTRL,             0xffffffff, 0x00000100,
> +       mmTCA_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
> +       mmTCC_CGTT_SCLK_CTRL,       0xffffffff, 0x00000100,
> +       mmTD_CGTT_CTRL,             0xffffffff, 0x00000100,
> +       mmGRBM_GFX_INDEX,           0xffffffff, 0xe0000000,
> +       mmCGTS_CU0_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU0_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU0_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
> +       mmCGTS_CU0_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU0_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU1_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU1_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU1_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU1_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU1_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU2_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU2_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU2_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU2_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU2_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU3_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU3_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU3_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU3_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU3_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU4_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU4_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU4_TA_SQC_CTRL_REG, 0xffffffff, 0x00040007,
> +       mmCGTS_CU4_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU4_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU5_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU5_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU5_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU5_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU5_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU6_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU6_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU6_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU6_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU6_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_CU7_SP0_CTRL_REG,    0xffffffff, 0x00010000,
> +       mmCGTS_CU7_LDS_SQ_CTRL_REG, 0xffffffff, 0x00030002,
> +       mmCGTS_CU7_TA_CTRL_REG,     0xffffffff, 0x00040007,
> +       mmCGTS_CU7_SP1_CTRL_REG,    0xffffffff, 0x00060005,
> +       mmCGTS_CU7_TD_TCP_CTRL_REG, 0xffffffff, 0x00090008,
> +       mmCGTS_SM_CTRL_REG,         0xffffffff, 0x96e00200,
> +       mmCP_RB_WPTR_POLL_CNTL,     0xffffffff, 0x00900100,
> +       mmRLC_CGCG_CGLS_CTRL,       0xffffffff, 0x0020003c,
> +       mmPCIE_INDEX,               0xffffffff, 0x0140001c,
> +       mmPCIE_DATA,                0x000f0000, 0x00000000,
> +       mmSMC_IND_INDEX_4,          0xffffffff, 0xC060000C,
> +       mmSMC_IND_DATA_4,           0xc0000fff, 0x00000100,
> +       mmXDMA_CLOCK_GATING_CNTL,   0xffffffff, 0x00000100,
> +       mmXDMA_MEM_POWER_CNTL,      0x00000101, 0x00000000,
> +       mmMC_MEM_POWER_LS,          0xffffffff, 0x00000104,
> +       mmCGTT_DRM_CLK_CTRL0,       0xff000fff, 0x00000100,
> +       mmHDP_XDP_CGTT_BLK_CTRL,    0xc0000fff, 0x00000104,
> +       mmCP_MEM_SLP_CNTL,          0x00000001, 0x00000001,
> +       mmSDMA0_CLK_CTRL,           0xff000ff0, 0x00000100,
> +       mmSDMA1_CLK_CTRL,           0xff000ff0, 0x00000100,
> +};
> +
> +static const u32 xgpu_golden_settings_tonga_a11[] = {
> +       mmCB_HW_CONTROL, 0xfffdf3cf, 0x00007208,
> +       mmCB_HW_CONTROL_3, 0x00000040, 0x00000040,
> +       mmDB_DEBUG2, 0xf00fffff, 0x00000400,
> +       mmDCI_CLK_CNTL, 0x00000080, 0x00000000,
> +       mmFBC_DEBUG_COMP, 0x000000f0, 0x00000070,
> +       mmFBC_MISC, 0x1f311fff, 0x12300000,
> +       mmGB_GPU_ID, 0x0000000f, 0x00000000,
> +       mmHDMI_CONTROL, 0x31000111, 0x00000011,
> +       mmMC_ARB_WTM_GRPWT_RD, 0x00000003, 0x00000000,
> +       mmMC_HUB_RDREQ_DMIF_LIMIT, 0x0000007f, 0x00000028,
> +       mmMC_HUB_WDP_UMC, 0x00007fb6, 0x00000991,
> +       mmPA_SC_ENHANCE, 0xffffffff, 0x20000001,
> +       mmPA_SC_FIFO_DEPTH_CNTL, 0x000003ff, 0x000000fc,
> +       mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000,
> +       mmRLC_CGCG_CGLS_CTRL, 0x00000003, 0x0000003c,
> +       mmSDMA0_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA0_CLK_CTRL, 0xff000fff, 0x00000000,
> +       mmSDMA0_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA0_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_CHICKEN_BITS, 0xfc910007, 0x00810007,
> +       mmSDMA1_CLK_CTRL, 0xff000fff, 0x00000000,
> +       mmSDMA1_GFX_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC0_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSDMA1_RLC1_IB_CNTL, 0x800f0111, 0x00000100,
> +       mmSQ_RANDOM_WAVE_PRI, 0x001fffff, 0x000006fd,
> +       mmTA_CNTL_AUX, 0x000f000f, 0x000b0000,
> +       mmTCC_CTRL, 0x00100000, 0xf31fff7f,
> +       mmTCC_EXE_DISABLE, 0x00000002, 0x00000002,
> +       mmTCP_ADDR_CONFIG, 0x000003ff, 0x000002fb,
> +       mmTCP_CHAN_STEER_HI, 0xffffffff, 0x0000543b,
> +       mmTCP_CHAN_STEER_LO, 0xffffffff, 0xa9210876,
> +       mmVGT_RESET_DEBUG, 0x00000004, 0x00000004,
> +       mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +       mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff,
> +};
> +
> +static const u32 xgpu_tonga_golden_common_all[] = {
> +       mmGRBM_GFX_INDEX,               0xffffffff, 0xe0000000,
> +       mmPA_SC_RASTER_CONFIG,          0xffffffff, 0x16000012,
> +       mmPA_SC_RASTER_CONFIG_1,        0xffffffff, 0x0000002A,
> +       mmGB_ADDR_CONFIG,               0xffffffff, 0x22011002,
> +       mmSPI_RESOURCE_RESERVE_CU_0,    0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_CU_1,    0xffffffff, 0x00000800,
> +       mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00007FBF,
> +};
> +
> +void amdgpu_xgpu_init_golden_registers(struct amdgpu_device *adev)
> +{
> +       switch (adev->asic_type) {
> +       case CHIP_FIJI:
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_fiji_mgcg_cgcg_init,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_fiji_mgcg_cgcg_init));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_golden_settings_fiji_a10,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_golden_settings_fiji_a10));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_fiji_golden_common_all,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_fiji_golden_common_all));
> +               break;
> +       case CHIP_TONGA:
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_tonga_mgcg_cgcg_init,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_tonga_mgcg_cgcg_init));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_golden_settings_tonga_a11,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_golden_settings_tonga_a11));
> +               amdgpu_program_register_sequence(adev,
> +                                                xgpu_tonga_golden_common_all,
> +                                                (const u32)ARRAY_SIZE(
> +                                                xgpu_tonga_golden_common_all));
> +               break;
> +       default:
> +               break;
> +       }
> +}

Cleaner to just put this in vi.c.

Alex

> +
>  static int xgpu_vi_early_init(void *handle)
>  {
>         struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 45007 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 22/23] drm/amdgpu: do not reset gpu for virtualization

[Xiangliang Yu]

Current job timeout setting is not fit for virtualization, so it
will cause job timeout sometimes. Add workaround for this when
timeout happen.

Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index b3c1820..976d217 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2322,6 +2322,9 @@ int amdgpu_gpu_reset(struct amdgpu_device *adev)
 	int resched;
 	bool need_full_reset;
 
+	if (amdgpu_sriov_vf(adev))
+		return 0;
+
 	if (!amdgpu_check_soft_reset(adev)) {
 		DRM_INFO("No hardware hang detected. Did some blocks stall?\n");
 		return 0;
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 23/23] drm/amdgpu/mxgpu: add Makefie and Kconfig for GPU virtualization

[Xiangliang Yu]

Add Kconfig for user configure and add Makefile to compile GPU
virtualization code and wires it into amdpgu modules.

Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Makefile |  7 +++++++
 drivers/gpu/drm/amd/mxgpu/Kconfig   | 13 +++++++++++++
 drivers/gpu/drm/amd/mxgpu/Makefile  | 11 +++++++++++
 3 files changed, 31 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/mxgpu/Kconfig
 create mode 100644 drivers/gpu/drm/amd/mxgpu/Makefile

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index 635ff94..f67dbbf 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -88,6 +88,13 @@ amdgpu-y += \
 	amdgpu_vce.o \
 	vce_v3_0.o
 
+# MxGPU componet
+ifneq ($(CONFIG_DRM_AMD_MXGPU),)
+MxGPU_PATH = ../mxgpu
+include $(FULL_AMD_PATH)/mxgpu/Makefile
+amdgpu-y += $(AMD_Mx_GPU_FILES)
+endif
+
 # add amdkfd interfaces
 amdgpu-y += \
 	 amdgpu_amdkfd.o \
diff --git a/drivers/gpu/drm/amd/mxgpu/Kconfig b/drivers/gpu/drm/amd/mxgpu/Kconfig
new file mode 100644
index 0000000..190b575
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/Kconfig
@@ -0,0 +1,13 @@
+menu "AMD GPU virtualization Configuration"
+
+config DRM_AMD_MXGPU
+       bool "Enable AMD GPU virtualization support"
+       depends on DRM_AMDGPU
+       help
+	Choose this option if you want to enable experimental support
+	for AMD GPU virtualization.
+	This adds AMD GPU virtualization driver and wires it up into
+	the amdgpu drivers. User can load the driver in guest OS and
+	run graphics applications on AMD hardware in guest mode.
+
+endmenu
diff --git a/drivers/gpu/drm/amd/mxgpu/Makefile b/drivers/gpu/drm/amd/mxgpu/Makefile
new file mode 100644
index 0000000..1e7fc18
--- /dev/null
+++ b/drivers/gpu/drm/amd/mxgpu/Makefile
@@ -0,0 +1,11 @@
+#
+# Makefile for Mxgpu
+#
+
+subdir-ccflags-y += -I$(MxGPU_PATH)/
+
+MxGPU = amd_mxgpu.o mxgpu_kiq.o mxgpu_csa.o mxgpu_mb.o mxgpu_vi.o
+
+AMD_MxGPU = $(addprefix $(MxGPU_PATH)/,$(MxGPU))
+
+AMD_Mx_GPU_FILES := $(AMD_MxGPU)
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

答复: [PATCH 00/23] Add support AMD GPU virtualization soultion

[Liu, Monk]

[-- Attachment #1.1: Type: text/plain, Size: 4682 bytes --]

Hi xiangliang


seems only the cover page was sent to amd-gfx loops, the rest of your serials were not


BR Monk

________________________________
发件人: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> 代表 Xiangliang Yu <Xiangliang.Yu@amd.com>
发送时间: 2016年12月18日 0:16:22
收件人: amd-gfx@lists.freedesktop.org; dl.SRDC_SW_GPUVirtualization
抄送: Yu, Xiangliang
主题: [PATCH 00/23] Add support AMD GPU virtualization soultion

This patch series will enable AMD GPU virtualization and wires it
into amdgpu modules.

The code design is base on below ideas:
1. Make virtualization code independent;
2. Self-maintain related GPU virtualization codes;
3. Easy to maintain virtualization codes;
4. Reduce the influence on other amdgpu components;
5. Easy to support different chips;

According to above ideas, make the following changes:
1. Create a new directory for virtualization and put all of
   related virtualization code into it;
2. Create new IP block to support different virtual features.
3. Create new file for each different hardware;

In the end, the patch series is support serval virtualization
features:
1. Support CSA feature;
2. Support Mailbox communication with GPU hypervisor;
3. Support KIQ feature;

Xiangliang Yu (23):
  drm/amdgpu: add support kernel interface queue(KIQ)
  drm/amdgpu: add kiq into compiling
  drm/amdgpu: export KIQ interfaces
  drm/amdgpu: add new structure for KIQ memory allcation
  drm/amdgpu/gfx8: add support KIQ for FIJI/TONGA chips
  drm/amdgpu/gfx8: correct KIQ hdp flush
  drm/amdgpu: create new directory for GPU virtualization
  drm/amdgpu: add new flag for virtual function
  drm/amdgpu: enable virtualization feature for FIJI/TONGA
  drm/amdgpu: export gem va update interface
  drm/amdgpu: implement context save area(CSA) feature
  drm/amdgpu: Insert meta data during submitting IB
  drm/amdgpu/mxgpu: add support for mailbox communication
  drm/amdgpu: export two mailbox interface to amdgpu
  drm/amdgpu/mxgpu: implement register access function with KIQ
  drm/amdgpu: add flag to indicate VF runtime state
  drm/amdgpu: export vi common ip block
  drm/amdgpu: add new maroc to identify virtualization ip block
  drm/amdgpu/mxgpu: add implementation of GPU virtualization of VI
  drm/amdgpu/mxgpu: enable VI virtualization
  drm/amdgpu: change golden register program sequence of virtualization
  drm/amdgpu: do not reset gpu for virtualization
  drm/amdgpu/mxgpu: add Makefie and Kconfig for GPU virtualization

 drivers/gpu/drm/amd/amdgpu/Makefile        |   9 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  35 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  14 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |   4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c    |   6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c     |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c    | 436 ++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    |   4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   |  34 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |   8 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h     |   4 +
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c      | 240 ++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/vi.c            |   8 +-
 drivers/gpu/drm/amd/amdgpu/vi.h            |   2 +-
 drivers/gpu/drm/amd/amdgpu/vid.h           |   2 +
 drivers/gpu/drm/amd/include/amd_shared.h   |   2 +
 drivers/gpu/drm/amd/mxgpu/Kconfig          |  13 +
 drivers/gpu/drm/amd/mxgpu/Makefile         |  11 +
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c      | 105 +++++++
 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h      |  93 ++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c      | 297 ++++++++++++++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c      | 110 +++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c       | 159 ++++++++++
 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c       | 472 +++++++++++++++++++++++++++++
 24 files changed, 2058 insertions(+), 12 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_kiq.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/Kconfig
 create mode 100644 drivers/gpu/drm/amd/mxgpu/Makefile
 create mode 100644 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/amd_mxgpu.h
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_csa.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_kiq.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_mb.c
 create mode 100644 drivers/gpu/drm/amd/mxgpu/mxgpu_vi.c

--
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[-- Attachment #1.2: Type: text/html, Size: 7829 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx