[PATCH 00/39] GPU scheduler for i915 driver

[PATCH 00/39] GPU scheduler for i915 driver

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Implemented a batch buffer submission scheduler for the i915 DRM driver.

The general theory of operation is that when batch buffers are
submitted to the driver, the execbuffer() code assigns a unique seqno
value and then packages up all the information required to execute the
batch buffer at a later time. This package is given over to the
scheduler which adds it to an internal node list. The scheduler also
scans the list of objects associated with the batch buffer and
compares them against the objects already in use by other buffers in
the node list. If matches are found then the new batch buffer node is
marked as being dependent upon the matching node. The same is done for
the context object. The scheduler also bumps up the priority of such
matching nodes on the grounds that the more dependencies a given batch
buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers
in flight on the hardware at any given time. If fewer than this are
currently executing when a new node is queued, then the node is passed
straight through to the submit function. Otherwise it is simply added
to the queue and the driver returns back to user land.

As each batch buffer completes, it raises an interrupt which wakes up
the scheduler. Note that it is possible for multiple buffers to
complete before the IRQ handler gets to run. Further, the seqno values
of the individual buffers are not necessary incrementing as the
scheduler may have re-ordered their submission. However, the scheduler
keeps the list of executing buffers in order of hardware submission.
Thus it can scan through the list until a matching seqno is found and
then mark all in flight nodes from that point on as completed.

A deferred work queue is also poked by the interrupt handler. When
this wakes up it can do more involved processing such as actually
removing completed nodes from the queue and freeing up the resources
associated with them (internal memory allocations, DRM object
references, context reference, etc.). The work handler also checks the
in flight count and calls the submission code if a new slot has
appeared.

When the scheduler's submit code is called, it scans the queued node
list for the highest priority node that has no unmet dependencies.
Note that the dependency calculation is complex as it must take
inter-ring dependencies and potential preemptions into account. Note
also that in the future this will be extended to include external
dependencies such as the Android Native Sync file descriptors and/or
the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for
submission to the hardware. The in flight count is then re-checked and
a new node popped from the list if appropriate.

The scheduler also allows high priority batch buffers (e.g. from a
desktop compositor) to jump ahead of whatever is already running if
the underlying hardware supports pre-emption. In this situation, any
work that was pre-empted is returned to the queued list ready to be
resubmitted when no more high priority work is outstanding.

Various IGT tests are in progress to test the scheduler's operation
and will follow.

v2: Updated for changes in struct fence patch series and other changes
to underlying tree (e.g. removal of cliprects). Also changed priority
levels to be signed +/-1023 range and reduced mutex lock usage.

[Patches against drm-intel-nightly tree fetched 17/11/2015 with struct
fence conversion patches applied]

Dave Gordon (2):
  drm/i915: Updating assorted register and status page definitions
  drm/i915: Add scheduling priority to per-context parameters

John Harrison (37):
  drm/i915: Add total count to context status debugfs output
  drm/i915: Explicit power enable during deferred context initialisation
  drm/i915: Prelude to splitting i915_gem_do_execbuffer in two
  drm/i915: Split i915_dem_do_execbuffer() in half
  drm/i915: Re-instate request->uniq because it is extremely useful
  drm/i915: Start of GPU scheduler
  drm/i915: Prepare retire_requests to handle out-of-order seqnos
  drm/i915: Disable hardware semaphores when GPU scheduler is enabled
  drm/i915: Force MMIO flips when scheduler enabled
  drm/i915: Added scheduler hook when closing DRM file handles
  drm/i915: Added scheduler hook into i915_gem_request_notify()
  drm/i915: Added deferred work handler for scheduler
  drm/i915: Redirect execbuffer_final() via scheduler
  drm/i915: Keep the reserved space mechanism happy
  drm/i915: Added tracking/locking of batch buffer objects
  drm/i915: Hook scheduler node clean up into retire requests
  drm/i915: Added scheduler support to __wait_request() calls
  drm/i915: Added scheduler support to page fault handler
  drm/i915: Added scheduler flush calls to ring throttle and idle functions
  drm/i915: Added a module parameter for allowing scheduler overrides
  drm/i915: Support for 'unflushed' ring idle
  drm/i915: Defer seqno allocation until actual hardware submission time
  drm/i915: Added immediate submission override to scheduler
  drm/i915: Add sync wait support to scheduler
  drm/i915: Connecting execbuff fences to scheduler
  drm/i915: Added trace points to scheduler
  drm/i915: Added scheduler queue throttling by DRM file handle
  drm/i915: Added debugfs interface to scheduler tuning parameters
  drm/i915: Added debug state dump facilities to scheduler
  drm/i915: Add early exit to execbuff_final() if insufficient ring space
  drm/i915: Added scheduler statistic reporting to debugfs
  drm/i915: Added seqno values to scheduler status dump
  drm/i915: Add scheduler support functions for TDR
  drm/i915: GPU priority bumping to prevent starvation
  drm/i915: Scheduler state dump via debugfs
  drm/i915: Enable GPU scheduler by default
  drm/i915: Allow scheduler to manage inter-ring object synchronisation

 drivers/gpu/drm/i915/Makefile              |    1 +
 drivers/gpu/drm/i915/i915_debugfs.c        |  283 +++++
 drivers/gpu/drm/i915/i915_dma.c            |    6 +
 drivers/gpu/drm/i915/i915_drv.c            |    9 +
 drivers/gpu/drm/i915/i915_drv.h            |   56 +-
 drivers/gpu/drm/i915/i915_gem.c            |  174 ++-
 drivers/gpu/drm/i915/i915_gem_context.c    |   24 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  341 ++++--
 drivers/gpu/drm/i915/i915_params.c         |    4 +
 drivers/gpu/drm/i915/i915_reg.h            |   30 +-
 drivers/gpu/drm/i915/i915_scheduler.c      | 1697 ++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |  176 +++
 drivers/gpu/drm/i915/i915_trace.h          |  215 +++-
 drivers/gpu/drm/i915/intel_display.c       |   10 +-
 drivers/gpu/drm/i915/intel_lrc.c           |  162 ++-
 drivers/gpu/drm/i915/intel_lrc.h           |    1 +
 drivers/gpu/drm/i915/intel_ringbuffer.c    |   47 +-
 drivers/gpu/drm/i915/intel_ringbuffer.h    |   35 +-
 include/uapi/drm/i915_drm.h                |    1 +
 19 files changed, 3110 insertions(+), 162 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h

-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 01/39] drm/i915: Add total count to context status debugfs output

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When there are lots and lots and even more lots of contexts (e.g. when running
with execlists) it is useful to be able to immediately see what the total
context count is.

Change-Id: If9726d4df86567100ecf53867b43f4753f08bf84
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 18dfb56..ecab9b8 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1953,6 +1953,7 @@ static int i915_context_status(struct seq_file *m, void *unused)
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct intel_engine_cs *ring;
 	struct intel_context *ctx;
+	uint32_t count = 0;
 	int ret, i;
 
 	ret = mutex_lock_interruptible(&dev->struct_mutex);
@@ -1966,6 +1967,7 @@ static int i915_context_status(struct seq_file *m, void *unused)
 
 		seq_puts(m, "HW context ");
 		describe_ctx(m, ctx);
+		count++;
 		for_each_ring(ring, dev_priv, i) {
 			if (ring->default_context == ctx)
 				seq_printf(m, "(default context %s) ",
@@ -1994,6 +1996,8 @@ static int i915_context_status(struct seq_file *m, void *unused)
 		seq_putc(m, '\n');
 	}
 
+	seq_printf(m, "Total: %d contexts\n", count);
+
 	mutex_unlock(&dev->struct_mutex);
 
 	return 0;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 02/39] drm/i915: Updating assorted register and status page definitions

[John.C.Harrison]

From: Dave Gordon <david.s.gordon@intel.com>

Added various definitions that will be useful for the scheduler in general and
pre-emptive context switching in particular.

Change-Id: Ica805b94160426def51f5d520f5ce51c60864a98
For: VIZ-1587
Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
---
 drivers/gpu/drm/i915/i915_reg.h         | 30 +++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/intel_ringbuffer.h | 32 +++++++++++++++++++++++++++++---
 2 files changed, 58 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_reg.h b/drivers/gpu/drm/i915/i915_reg.h
index db72f98..b986050 100644
--- a/drivers/gpu/drm/i915/i915_reg.h
+++ b/drivers/gpu/drm/i915/i915_reg.h
@@ -259,6 +259,10 @@
 #define  MI_GLOBAL_GTT    (1<<22)
 
 #define MI_NOOP			MI_INSTR(0, 0)
+#define   MI_NOOP_WRITE_ID		(1<<22)
+#define   MI_NOOP_ID_MASK		((1<<22) - 1)
+#define   MI_NOOP_MID(id)		((id) & MI_NOOP_ID_MASK)
+#define MI_NOOP_WITH_ID(id)	MI_INSTR(0, MI_NOOP_WRITE_ID|MI_NOOP_MID(id))
 #define MI_USER_INTERRUPT	MI_INSTR(0x02, 0)
 #define MI_WAIT_FOR_EVENT       MI_INSTR(0x03, 0)
 #define   MI_WAIT_FOR_OVERLAY_FLIP	(1<<16)
@@ -276,6 +280,7 @@
 #define MI_ARB_ON_OFF		MI_INSTR(0x08, 0)
 #define   MI_ARB_ENABLE			(1<<0)
 #define   MI_ARB_DISABLE		(0<<0)
+#define MI_ARB_CHECK		MI_INSTR(0x05, 0)
 #define MI_BATCH_BUFFER_END	MI_INSTR(0x0a, 0)
 #define MI_SUSPEND_FLUSH	MI_INSTR(0x0b, 0)
 #define   MI_SUSPEND_FLUSH_EN	(1<<0)
@@ -325,6 +330,8 @@
 #define   MI_SEMAPHORE_SYNC_INVALID (3<<16)
 #define   MI_SEMAPHORE_SYNC_MASK    (3<<16)
 #define MI_SET_CONTEXT		MI_INSTR(0x18, 0)
+#define   MI_CONTEXT_ADDR_MASK		((~0)<<12)
+#define   MI_SET_CONTEXT_FLAG_MASK	((1<<12)-1)
 #define   MI_MM_SPACE_GTT		(1<<8)
 #define   MI_MM_SPACE_PHYSICAL		(0<<8)
 #define   MI_SAVE_EXT_STATE_EN		(1<<3)
@@ -344,6 +351,10 @@
 #define   MI_USE_GGTT		(1 << 22) /* g4x+ */
 #define MI_STORE_DWORD_INDEX	MI_INSTR(0x21, 1)
 #define   MI_STORE_DWORD_INDEX_SHIFT 2
+#define MI_STORE_REG_MEM	MI_INSTR(0x24, 1)
+#define   MI_STORE_REG_MEM_GTT		(1 << 22)
+#define   MI_STORE_REG_MEM_PREDICATE	(1 << 21)
+
 /* Official intel docs are somewhat sloppy concerning MI_LOAD_REGISTER_IMM:
  * - Always issue a MI_NOOP _before_ the MI_LOAD_REGISTER_IMM - otherwise hw
  *   simply ignores the register load under certain conditions.
@@ -358,7 +369,10 @@
 #define MI_FLUSH_DW		MI_INSTR(0x26, 1) /* for GEN6 */
 #define   MI_FLUSH_DW_STORE_INDEX	(1<<21)
 #define   MI_INVALIDATE_TLB		(1<<18)
+#define   MI_FLUSH_DW_OP_NONE		(0<<14)
 #define   MI_FLUSH_DW_OP_STOREDW	(1<<14)
+#define   MI_FLUSH_DW_OP_RSVD		(2<<14)
+#define   MI_FLUSH_DW_OP_STAMP		(3<<14)
 #define   MI_FLUSH_DW_OP_MASK		(3<<14)
 #define   MI_FLUSH_DW_NOTIFY		(1<<8)
 #define   MI_INVALIDATE_BSD		(1<<7)
@@ -1533,6 +1547,19 @@ enum skl_disp_power_wells {
 
 #define HSW_GTT_CACHE_EN	0x4024
 #define   GTT_CACHE_EN_ALL	0xF0007FFF
+
+/*
+ * Premption-related registers
+ */
+#define RING_UHPTR(base)	((base)+0x134)
+#define   UHPTR_GFX_ADDR_ALIGN		(0x7)
+#define   UHPTR_VALID			(0x1)
+#define RING_PREEMPT_ADDR	0x0214c
+#define   PREEMPT_BATCH_LEVEL_MASK	(0x3)
+#define BB_PREEMPT_ADDR		0x02148
+#define SBB_PREEMPT_ADDR	0x0213c
+#define RS_PREEMPT_STATUS	0x0215c
+
 #define GEN7_WR_WATERMARK	0x4028
 #define GEN7_GFX_PRIO_CTRL	0x402C
 #define ARB_MODE		0x4030
@@ -6736,7 +6763,8 @@ enum skl_disp_power_wells {
 #define  VLV_SPAREG2H				0xA194
 
 #define  GTFIFODBG				0x120000
-#define    GT_FIFO_SBDROPERR			(1<<6)
+#define    GT_FIFO_CPU_ERROR_MASK		0xf
+#define    GT_FIFO_SDDROPERR			(1<<6)
 #define    GT_FIFO_BLOBDROPERR			(1<<5)
 #define    GT_FIFO_SB_READ_ABORTERR		(1<<4)
 #define    GT_FIFO_DROPERR			(1<<3)
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h b/drivers/gpu/drm/i915/intel_ringbuffer.h
index 1987abd..48f60cc 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.h
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.h
@@ -49,6 +49,12 @@ struct  intel_hw_status_page {
 #define I915_READ_MODE(ring) I915_READ(RING_MI_MODE((ring)->mmio_base))
 #define I915_WRITE_MODE(ring, val) I915_WRITE(RING_MI_MODE((ring)->mmio_base), val)
 
+#define I915_READ_UHPTR(ring) \
+		I915_READ(RING_UHPTR((ring)->mmio_base))
+#define I915_WRITE_UHPTR(ring, val) \
+		I915_WRITE(RING_UHPTR((ring)->mmio_base), val)
+#define I915_READ_NOPID(ring) I915_READ(RING_NOPID((ring)->mmio_base))
+
 /* seqno size is actually only a uint32, but since we plan to use MI_FLUSH_DW to
  * do the writes, and that must have qw aligned offsets, simply pretend it's 8b.
  */
@@ -426,10 +432,30 @@ intel_write_status_page(struct intel_engine_cs *ring,
  * 0x20-0x2f: Reserved (Gen6+)
  *
  * The area from dword 0x30 to 0x3ff is available for driver usage.
+ *
+ * Note: in general the allocation of these indices is arbitrary, as long
+ * as they are all unique. But a few of them are used with instructions that
+ * have specific alignment requirements, those particular indices must be
+ * chosen carefully to meet those requirements. The list below shows the
+ * currently-known alignment requirements:
+ *
+ *	I915_GEM_SCRATCH_INDEX	    must be EVEN
+ */
+
+/*
+ * Tracking; these are updated by the GPU at the beginning and/or end of every
+ * batch. One pair for regular buffers, the other for preemptive ones.
  */
-#define I915_GEM_HWS_INDEX		0x30
-#define I915_GEM_HWS_SCRATCH_INDEX	0x40
-#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX << MI_STORE_DWORD_INDEX_SHIFT)
+#define I915_BATCH_DONE_SEQNO		0x30  /* Completed batch seqno        */
+#define I915_BATCH_ACTIVE_SEQNO		0x31  /* In progress batch seqno      */
+#define I915_PREEMPTIVE_DONE_SEQNO	0x32  /* Completed preemptive batch   */
+#define I915_PREEMPTIVE_ACTIVE_SEQNO	0x33  /* In progress preemptive batch */
+
+#define I915_GEM_HWS_INDEX		I915_BATCH_DONE_SEQNO	/* alias */
+#define I915_GEM_ACTIVE_SEQNO_INDEX	I915_BATCH_ACTIVE_SEQNO	/* alias */
+
+#define I915_GEM_HWS_SCRATCH_INDEX	0x38  /* QWord */
+#define I915_GEM_HWS_SCRATCH_ADDR	(I915_GEM_HWS_SCRATCH_INDEX << MI_STORE_DWORD_INDEX_SHIFT)
 
 struct intel_ringbuffer *
 intel_engine_create_ringbuffer(struct intel_engine_cs *engine, int size);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 03/39] drm/i915: Explicit power enable during deferred context initialisation

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

A later patch in this series re-organises the batch buffer submission
code. Part of that is to reduce the scope of a pm_get/put pair.
Specifically, they previously wrapped the entire submission path from
the very start to the very end, now they only wrap the actual hardware
submission part in the back half.

While that is a good thing in general, it causes a problem with the
deferred context initialisation. That is done quite early on in the
execbuf code path - it happens at context validation time rather than
context switch time. Some of the deferred work requires the power to
be enabled. Hence this patch adds an explicit power reference count to
the deferred initialisation code itself.

Change-Id: Id7b1535dfd8809a2bd5546272de2bbec39da2868
Issue: GMINL-5159
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/intel_lrc.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 76fc245..e510730 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -2470,6 +2470,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,
 	WARN_ON(ctx->legacy_hw_ctx.rcs_state != NULL);
 	WARN_ON(ctx->engine[ring->id].state);
 
+	intel_runtime_pm_get(dev->dev_private);
+
 	context_size = round_up(get_lr_context_size(ring), 4096);
 
 	/* One extra page as the sharing data between driver and GuC */
@@ -2478,7 +2480,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,
 	ctx_obj = i915_gem_alloc_object(dev, context_size);
 	if (!ctx_obj) {
 		DRM_DEBUG_DRIVER("Alloc LRC backing obj failed.\n");
-		return -ENOMEM;
+		ret = -ENOMEM;
+		goto error_pm;
 	}
 
 	ringbuf = intel_engine_create_ringbuffer(ring, 4 * PAGE_SIZE);
@@ -2524,6 +2527,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,
 		}
 		i915_add_request_no_flush(req);
 	}
+
+	intel_runtime_pm_put(dev->dev_private);
 	return 0;
 
 error_ringbuf:
@@ -2532,6 +2537,8 @@ error_deref_obj:
 	drm_gem_object_unreference(&ctx_obj->base);
 	ctx->engine[ring->id].ringbuf = NULL;
 	ctx->engine[ring->id].state = NULL;
+error_pm:
+	intel_runtime_pm_put(dev->dev_private);
 	return ret;
 }
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 04/39] drm/i915: Prelude to splitting i915_gem_do_execbuffer in two

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler decouples the submission of batch buffers to the driver with their
submission to the hardware. This basically means splitting the execbuffer()
function in half. This change rearranges some code ready for the split to occur.

Change-Id: Icc9c8afaac18821f3eb8a151a49f918f90c068a3
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 64 ++++++++++++++++++------------
 drivers/gpu/drm/i915/intel_lrc.c           | 18 ++++++---
 2 files changed, 51 insertions(+), 31 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 5f629f8..f7f1057 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -935,10 +935,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
 	if (flush_domains & I915_GEM_DOMAIN_GTT)
 		wmb();
 
-	/* Unconditionally invalidate gpu caches and ensure that we do flush
-	 * any residual writes from the previous batch.
-	 */
-	return intel_ring_invalidate_all_caches(req);
+	return 0;
 }
 
 static bool
@@ -1191,17 +1188,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	u32 instp_mask;
 	int ret;
 
-	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
-	if (ret)
-		return ret;
-
-	ret = i915_switch_context(params->request);
-	if (ret)
-		return ret;
-
-	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<ring->id),
-	     "%s didn't clear reload\n", ring->name);
-
 	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
 	instp_mask = I915_EXEC_CONSTANTS_MASK;
 	switch (instp_mode) {
@@ -1235,11 +1221,37 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 		return -EINVAL;
 	}
 
+	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
+	if (ret)
+		return ret;
+
+	i915_gem_execbuffer_move_to_active(vmas, params->request);
+
+	/* To be split into two functions here... */
+
+	intel_runtime_pm_get(dev_priv);
+
+	/*
+	 * Unconditionally invalidate gpu caches and ensure that we do flush
+	 * any residual writes from the previous batch.
+	 */
+	ret = intel_ring_invalidate_all_caches(params->request);
+	if (ret)
+		goto error;
+
+	/* Switch to the correct context for the batch */
+	ret = i915_switch_context(params->request);
+	if (ret)
+		goto error;
+
+	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<ring->id),
+	     "%s didn't clear reload\n", ring->name);
+
 	if (ring == &dev_priv->ring[RCS] &&
 	    instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_ring_begin(params->request, 4);
 		if (ret)
-			return ret;
+			goto error;
 
 		intel_ring_emit(ring, MI_NOOP);
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
@@ -1253,7 +1265,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {
 		ret = i915_reset_gen7_sol_offsets(dev, params->request);
 		if (ret)
-			return ret;
+			goto error;
 	}
 
 	exec_len   = args->batch_len;
@@ -1264,14 +1276,20 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 					exec_start, exec_len,
 					params->dispatch_flags);
 	if (ret)
-		return ret;
+		goto error;
 
 	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
 
-	i915_gem_execbuffer_move_to_active(vmas, params->request);
 	i915_gem_execbuffer_retire_commands(params);
 
-	return 0;
+error:
+	/*
+	 * intel_gpu_busy should also get a ref, so it will free when the device
+	 * is really idle.
+	 */
+	intel_runtime_pm_put(dev_priv);
+
+	return ret;
 }
 
 /**
@@ -1480,8 +1498,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		args->flags &= ~I915_EXEC_WAIT_FENCE;
 	}
 
-	intel_runtime_pm_get(dev_priv);
-
 	ret = i915_mutex_lock_interruptible(dev);
 	if (ret)
 		goto pre_mutex_err;
@@ -1688,10 +1704,6 @@ err:
 	mutex_unlock(&dev->struct_mutex);
 
 pre_mutex_err:
-	/* intel_gpu_busy should also get a ref, so it will free when the device
-	 * is really idle. */
-	intel_runtime_pm_put(dev_priv);
-
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index e510730..4bf0ee6 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -647,10 +647,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,
 	if (flush_domains & I915_GEM_DOMAIN_GTT)
 		wmb();
 
-	/* Unconditionally invalidate gpu caches and ensure that we do flush
-	 * any residual writes from the previous batch.
-	 */
-	return logical_ring_invalidate_all_caches(req);
+	return 0;
 }
 
 int intel_logical_ring_alloc_request_extras(struct drm_i915_gem_request *request)
@@ -913,6 +910,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 	if (ret)
 		return ret;
 
+	i915_gem_execbuffer_move_to_active(vmas, params->request);
+
+	/* To be split into two functions here... */
+
+	/*
+	 * Unconditionally invalidate gpu caches and ensure that we do flush
+	 * any residual writes from the previous batch.
+	 */
+	ret = logical_ring_invalidate_all_caches(params->request);
+	if (ret)
+		return ret;
+
 	if (ring == &dev_priv->ring[RCS] &&
 	    instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_logical_ring_begin(params->request, 4);
@@ -937,7 +946,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
 
-	i915_gem_execbuffer_move_to_active(vmas, params->request);
 	i915_gem_execbuffer_retire_commands(params);
 
 	return 0;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 05/39] drm/i915: Split i915_dem_do_execbuffer() in half

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Split the execbuffer() function in half. The first half collects and
validates all the information requried to process the batch buffer. It
also does all the object pinning, relocations, active list management,
etc - basically anything that must be done upfront before the IOCTL
returns and allows the user land side to start changing/freeing
things. The second half does the actual ring submission.

This change implements the split but leaves the back half being called
directly from the end of the front half.

v2: Updated due to changes in underlying tree - addition of sync fence
support and removal of cliprects.

Change-Id: I5e1c77639ce526ab2401b0323186c518bf13da0a
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  11 +++
 drivers/gpu/drm/i915/i915_gem.c            |   2 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 130 ++++++++++++++++++++---------
 drivers/gpu/drm/i915/intel_lrc.c           |  58 +++++++++----
 drivers/gpu/drm/i915/intel_lrc.h           |   1 +
 5 files changed, 146 insertions(+), 56 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 194bca0..eb00454 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1679,10 +1679,18 @@ struct i915_execbuffer_params {
 	struct drm_device               *dev;
 	struct drm_file                 *file;
 	uint32_t                        dispatch_flags;
+	uint32_t                        args_flags;
 	uint32_t                        args_batch_start_offset;
+	uint32_t                        args_batch_len;
+	uint32_t                        args_num_cliprects;
+	uint32_t                        args_DR1;
+	uint32_t                        args_DR4;
 	uint64_t                        batch_obj_vm_offset;
 	struct intel_engine_cs          *ring;
 	struct drm_i915_gem_object      *batch_obj;
+	struct drm_clip_rect            *cliprects;
+	uint32_t                        instp_mask;
+	int                             instp_mode;
 	struct intel_context            *ctx;
 	struct drm_i915_gem_request     *request;
 };
@@ -1944,6 +1952,7 @@ struct drm_i915_private {
 		int (*execbuf_submit)(struct i915_execbuffer_params *params,
 				      struct drm_i915_gem_execbuffer2 *args,
 				      struct list_head *vmas);
+		int (*execbuf_final)(struct i915_execbuffer_params *params);
 		int (*init_rings)(struct drm_device *dev);
 		void (*cleanup_ring)(struct intel_engine_cs *ring);
 		void (*stop_ring)(struct intel_engine_cs *ring);
@@ -2798,9 +2807,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,
 void i915_gem_execbuffer_move_to_active(struct list_head *vmas,
 					struct drm_i915_gem_request *req);
 void i915_gem_execbuffer_retire_commands(struct i915_execbuffer_params *params);
+void i915_gem_execbuff_release_batch_obj(struct drm_i915_gem_object *batch_obj);
 int i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 				   struct drm_i915_gem_execbuffer2 *args,
 				   struct list_head *vmas);
+int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params);
 int i915_gem_execbuffer(struct drm_device *dev, void *data,
 			struct drm_file *file_priv);
 int i915_gem_execbuffer2(struct drm_device *dev, void *data,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 0e41386..8ecc1f4 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -5256,11 +5256,13 @@ int i915_gem_init(struct drm_device *dev)
 
 	if (!i915.enable_execlists) {
 		dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission;
+		dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final;
 		dev_priv->gt.init_rings = i915_gem_init_rings;
 		dev_priv->gt.cleanup_ring = intel_cleanup_ring_buffer;
 		dev_priv->gt.stop_ring = intel_stop_ring_buffer;
 	} else {
 		dev_priv->gt.execbuf_submit = intel_execlists_submission;
+		dev_priv->gt.execbuf_final = intel_execlists_submission_final;
 		dev_priv->gt.init_rings = intel_logical_rings_init;
 		dev_priv->gt.cleanup_ring = intel_logical_ring_cleanup;
 		dev_priv->gt.stop_ring = intel_logical_ring_stop;
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index f7f1057..05c9de6 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1183,41 +1183,38 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	struct drm_device *dev = params->dev;
 	struct intel_engine_cs *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	u64 exec_start, exec_len;
-	int instp_mode;
-	u32 instp_mask;
 	int ret;
 
-	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
-	instp_mask = I915_EXEC_CONSTANTS_MASK;
-	switch (instp_mode) {
+	params->instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
+	params->instp_mask = I915_EXEC_CONSTANTS_MASK;
+	switch (params->instp_mode) {
 	case I915_EXEC_CONSTANTS_REL_GENERAL:
 	case I915_EXEC_CONSTANTS_ABSOLUTE:
 	case I915_EXEC_CONSTANTS_REL_SURFACE:
-		if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
+		if (params->instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
 			DRM_DEBUG("non-0 rel constants mode on non-RCS\n");
 			return -EINVAL;
 		}
 
-		if (instp_mode != dev_priv->relative_constants_mode) {
+		if (params->instp_mode != dev_priv->relative_constants_mode) {
 			if (INTEL_INFO(dev)->gen < 4) {
 				DRM_DEBUG("no rel constants on pre-gen4\n");
 				return -EINVAL;
 			}
 
 			if (INTEL_INFO(dev)->gen > 5 &&
-			    instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
+			    params->instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
 				DRM_DEBUG("rel surface constants mode invalid on gen5+\n");
 				return -EINVAL;
 			}
 
 			/* The HW changed the meaning on this bit on gen6 */
 			if (INTEL_INFO(dev)->gen >= 6)
-				instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
+				params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
 		}
 		break;
 	default:
-		DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode);
+		DRM_DEBUG("execbuf with unknown constants: %d\n", params->instp_mode);
 		return -EINVAL;
 	}
 
@@ -1227,7 +1224,33 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	/* To be split into two functions here... */
+	ret = dev_priv->gt.execbuf_final(params);
+	if (ret)
+		return ret;
+
+	/*
+	 * Free everything that was stored in the QE structure (until the
+	 * scheduler arrives and does it instead):
+	 */
+	if (params->dispatch_flags & I915_DISPATCH_SECURE)
+		i915_gem_execbuff_release_batch_obj(params->batch_obj);
+
+	return 0;
+}
+
+/*
+ * This is the main function for adding a batch to the ring.
+ * It is called from the scheduler, with the struct_mutex already held.
+ */
+int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
+{
+	struct drm_i915_private *dev_priv = params->dev->dev_private;
+	struct intel_engine_cs  *ring = params->ring;
+	u64 exec_start, exec_len;
+	int ret;
+
+	/* The mutex must be acquired before calling this function */
+	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
 	intel_runtime_pm_get(dev_priv);
 
@@ -1248,7 +1271,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	     "%s didn't clear reload\n", ring->name);
 
 	if (ring == &dev_priv->ring[RCS] &&
-	    instp_mode != dev_priv->relative_constants_mode) {
+	    params->instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_ring_begin(params->request, 4);
 		if (ret)
 			goto error;
@@ -1256,19 +1279,19 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 		intel_ring_emit(ring, MI_NOOP);
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, INSTPM);
-		intel_ring_emit(ring, instp_mask << 16 | instp_mode);
+		intel_ring_emit(ring, params->instp_mask << 16 | params->instp_mode);
 		intel_ring_advance(ring);
 
-		dev_priv->relative_constants_mode = instp_mode;
+		dev_priv->relative_constants_mode = params->instp_mode;
 	}
 
-	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {
-		ret = i915_reset_gen7_sol_offsets(dev, params->request);
+	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
+		ret = i915_reset_gen7_sol_offsets(params->dev, params->request);
 		if (ret)
 			goto error;
 	}
 
-	exec_len   = args->batch_len;
+	exec_len   = params->args_batch_len;
 	exec_start = params->batch_obj_vm_offset +
 		     params->args_batch_start_offset;
 
@@ -1640,6 +1663,11 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->file                    = file;
 	params->ring                    = ring;
 	params->dispatch_flags          = dispatch_flags;
+	params->args_flags              = args->flags;
+	params->args_batch_len          = args->batch_len;
+	params->args_num_cliprects      = args->num_cliprects;
+	params->args_DR1                = args->DR1;
+	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
 	params->ctx                     = ctx;
 
@@ -1659,37 +1687,50 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	}
 
 	ret = dev_priv->gt.execbuf_submit(params, args, &eb->vmas);
+	if (ret)
+		goto err_fence;
+
+	/* the request owns the ref now */
+	i915_gem_context_unreference(ctx);
 
 	if (fd_fence_complete != -1) {
-		if (ret) {
-			sync_fence_put(sync_fence);
-			put_unused_fd(fd_fence_complete);
-		} else {
-			/*
-			 * Install the fence into the pre-allocated file
-			 * descriptor to the fence object so that user land
-			 * can wait on it...
-			 */
-			i915_install_sync_fence_fd(params->request,
-						   sync_fence, fd_fence_complete);
+		/*
+		 * Install the fence into the pre-allocated file
+		 * descriptor to the fence object so that user land
+		 * can wait on it...
+		 */
+		i915_install_sync_fence_fd(params->request,
+					   sync_fence, fd_fence_complete);
 
-			/* Return the fence through the rsvd2 field */
-			args->rsvd2 = (__u64) fd_fence_complete;
-		}
+		/* Return the fence through the rsvd2 field */
+		args->rsvd2 = (__u64) fd_fence_complete;
 	}
 
-err_batch_unpin:
 	/*
-	 * FIXME: We crucially rely upon the active tracking for the (ppgtt)
-	 * batch vma for correctness. For less ugly and less fragility this
-	 * needs to be adjusted to also track the ggtt batch vma properly as
-	 * active.
+	 * The eb list is no longer required. The scheduler has extracted all
+	 * the information than needs to persist.
+	 */
+	eb_destroy(eb);
+
+	/*
+	 * Don't clean up everything that is now saved away in the queue.
+	 * Just unlock and return immediately.
 	 */
+	mutex_unlock(&dev->struct_mutex);
+
+	return 0;
+
+err_fence:
+	if (fd_fence_complete != -1) {
+		sync_fence_put(sync_fence);
+		put_unused_fd(fd_fence_complete);
+	}
+
+err_batch_unpin:
 	if (dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_object_ggtt_unpin(batch_obj);
+		i915_gem_execbuff_release_batch_obj(batch_obj);
 
 err:
-	/* the request owns the ref now */
 	i915_gem_context_unreference(ctx);
 	eb_destroy(eb);
 
@@ -1707,6 +1748,17 @@ pre_mutex_err:
 	return ret;
 }
 
+void i915_gem_execbuff_release_batch_obj(struct drm_i915_gem_object *batch_obj)
+{
+	/*
+	 * FIXME: We crucially rely upon the active tracking for the (ppgtt)
+	 * batch vma for correctness. For less ugly and less fragility this
+	 * needs to be adjusted to also track the ggtt batch vma properly as
+	 * active.
+	 */
+	i915_gem_object_ggtt_unpin(batch_obj);
+}
+
 /*
  * Legacy execbuffer just creates an exec2 list from the original exec object
  * list array and passes it to the real function.
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 4bf0ee6..4aae31c 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -869,35 +869,31 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 	struct drm_device       *dev = params->dev;
 	struct intel_engine_cs  *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct intel_ringbuffer *ringbuf = params->ctx->engine[ring->id].ringbuf;
-	u64 exec_start;
-	int instp_mode;
-	u32 instp_mask;
 	int ret;
 
-	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
-	instp_mask = I915_EXEC_CONSTANTS_MASK;
-	switch (instp_mode) {
+	params->instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
+	params->instp_mask = I915_EXEC_CONSTANTS_MASK;
+	switch (params->instp_mode) {
 	case I915_EXEC_CONSTANTS_REL_GENERAL:
 	case I915_EXEC_CONSTANTS_ABSOLUTE:
 	case I915_EXEC_CONSTANTS_REL_SURFACE:
-		if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
+		if (params->instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
 			DRM_DEBUG("non-0 rel constants mode on non-RCS\n");
 			return -EINVAL;
 		}
 
-		if (instp_mode != dev_priv->relative_constants_mode) {
-			if (instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
+		if (params->instp_mode != dev_priv->relative_constants_mode) {
+			if (params->instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
 				DRM_DEBUG("rel surface constants mode invalid on gen5+\n");
 				return -EINVAL;
 			}
 
 			/* The HW changed the meaning on this bit on gen6 */
-			instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
+			params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
 		}
 		break;
 	default:
-		DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode);
+		DRM_DEBUG("execbuf with unknown constants: %d\n", params->instp_mode);
 		return -EINVAL;
 	}
 
@@ -912,7 +908,33 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	/* To be split into two functions here... */
+	ret = dev_priv->gt.execbuf_final(params);
+	if (ret)
+		return ret;
+
+	/*
+	 * Free everything that was stored in the QE structure (until the
+	 * scheduler arrives and does it instead):
+	 */
+	if (params->dispatch_flags & I915_DISPATCH_SECURE)
+		i915_gem_execbuff_release_batch_obj(params->batch_obj);
+
+	return 0;
+}
+
+/*
+ * This is the main function for adding a batch to the ring.
+ * It is called from the scheduler, with the struct_mutex already held.
+ */
+int intel_execlists_submission_final(struct i915_execbuffer_params *params)
+{
+	struct drm_i915_private *dev_priv = params->dev->dev_private;
+	struct intel_engine_cs  *ring = params->ring;
+	u64 exec_start;
+	int ret;
+
+	/* The mutex must be acquired before calling this function */
+	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
@@ -923,7 +945,9 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 		return ret;
 
 	if (ring == &dev_priv->ring[RCS] &&
-	    instp_mode != dev_priv->relative_constants_mode) {
+	    params->instp_mode != dev_priv->relative_constants_mode) {
+		struct intel_ringbuffer *ringbuf = params->request->ringbuf;
+
 		ret = intel_logical_ring_begin(params->request, 4);
 		if (ret)
 			return ret;
@@ -931,14 +955,14 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 		intel_logical_ring_emit(ringbuf, MI_NOOP);
 		intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1));
 		intel_logical_ring_emit(ringbuf, INSTPM);
-		intel_logical_ring_emit(ringbuf, instp_mask << 16 | instp_mode);
+		intel_logical_ring_emit(ringbuf, params->instp_mask << 16 | params->instp_mode);
 		intel_logical_ring_advance(ringbuf);
 
-		dev_priv->relative_constants_mode = instp_mode;
+		dev_priv->relative_constants_mode = params->instp_mode;
 	}
 
 	exec_start = params->batch_obj_vm_offset +
-		     args->batch_start_offset;
+		     params->args_batch_start_offset;
 
 	ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags);
 	if (ret)
diff --git a/drivers/gpu/drm/i915/intel_lrc.h b/drivers/gpu/drm/i915/intel_lrc.h
index 4e60d54..8d9bad7 100644
--- a/drivers/gpu/drm/i915/intel_lrc.h
+++ b/drivers/gpu/drm/i915/intel_lrc.h
@@ -93,6 +93,7 @@ struct i915_execbuffer_params;
 int intel_execlists_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas);
+int intel_execlists_submission_final(struct i915_execbuffer_params *params);
 u32 intel_execlists_ctx_id(struct drm_i915_gem_object *ctx_obj);
 
 void intel_lrc_irq_handler(struct intel_engine_cs *ring);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 06/39] drm/i915: Re-instate request->uniq because it is extremely useful

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The seqno value cannot always be used when debugging issues via trace
points. This is because it can be reset back to start, especially
during TDR type tests. Also, when the scheduler arrives the seqno is
only valid while a given request is executing on the hardware. While
the request is simply queued waiting for submission, it's seqno value
will be zero (meaning invalid).

For: VIZ-5115
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
Reviewed-by: Tomas Elf <tomas.elf@intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h   |  5 +++++
 drivers/gpu/drm/i915/i915_gem.c   |  3 ++-
 drivers/gpu/drm/i915/i915_trace.h | 25 +++++++++++++++++--------
 3 files changed, 24 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index eb00454..5d390d9 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1963,6 +1963,8 @@ struct drm_i915_private {
 	/* perform PHY state sanity checks? */
 	bool chv_phy_assert[2];
 
+	uint32_t request_uniq;
+
 	/*
 	 * NOTE: This is the dri1/ums dungeon, don't add stuff here. Your patch
 	 * will be rejected. Instead look for a better place.
@@ -2210,6 +2212,9 @@ struct drm_i915_gem_request {
 	/** GEM sequence number associated with this request. */
 	uint32_t seqno;
 
+	/* Unique identifier which can be used for trace points & debug */
+	uint32_t uniq;
+
 	/** Position in the ringbuffer of the start of the request */
 	u32 head;
 
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 8ecc1f4..b23746a 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2862,7 +2862,7 @@ static void i915_gem_request_fence_value_str(struct fence *req_fence, char *str,
 
 	req = container_of(req_fence, typeof(*req), fence);
 
-	snprintf(str, size, "%d [%d]", req->fence.seqno, req->seqno);
+	snprintf(str, size, "%d [%d:%d]", req->fence.seqno, req->uniq, req->seqno);
 }
 
 static const struct fence_ops i915_gem_request_fops = {
@@ -3015,6 +3015,7 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring,
 
 	req->i915 = dev_priv;
 	req->ring = ring;
+	req->uniq = dev_priv->request_uniq++;
 	req->ctx  = ctx;
 	i915_gem_context_reference(req->ctx);
 
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index 41a026d..76b08fa 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -469,6 +469,7 @@ TRACE_EVENT(i915_gem_ring_sync_to,
 			     __field(u32, dev)
 			     __field(u32, sync_from)
 			     __field(u32, sync_to)
+			     __field(u32, uniq_to)
 			     __field(u32, seqno)
 			     ),
 
@@ -476,13 +477,14 @@ TRACE_EVENT(i915_gem_ring_sync_to,
 			   __entry->dev = from->dev->primary->index;
 			   __entry->sync_from = from->id;
 			   __entry->sync_to = to_req->ring->id;
+			   __entry->uniq_to = to_req->uniq;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   ),
 
-	    TP_printk("dev=%u, sync-from=%u, sync-to=%u, seqno=%u",
+	    TP_printk("dev=%u, sync-from=%u, sync-to=%u, seqno=%u, to_uniq=%u",
 		      __entry->dev,
 		      __entry->sync_from, __entry->sync_to,
-		      __entry->seqno)
+		      __entry->seqno, __entry->uniq_to)
 );
 
 TRACE_EVENT(i915_gem_ring_dispatch,
@@ -517,6 +519,7 @@ TRACE_EVENT(i915_gem_ring_flush,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, invalidate)
 			     __field(u32, flush)
 			     ),
@@ -524,12 +527,13 @@ TRACE_EVENT(i915_gem_ring_flush,
 	    TP_fast_assign(
 			   __entry->dev = req->ring->dev->primary->index;
 			   __entry->ring = req->ring->id;
+			   __entry->uniq = req->uniq;
 			   __entry->invalidate = invalidate;
 			   __entry->flush = flush;
 			   ),
 
-	    TP_printk("dev=%u, ring=%x, invalidate=%04x, flush=%04x",
-		      __entry->dev, __entry->ring,
+	    TP_printk("dev=%u, ring=%x, request=%u, invalidate=%04x, flush=%04x",
+		      __entry->dev, __entry->ring, __entry->uniq,
 		      __entry->invalidate, __entry->flush)
 );
 
@@ -540,6 +544,7 @@ DECLARE_EVENT_CLASS(i915_gem_request,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, seqno)
 			     ),
 
@@ -548,11 +553,13 @@ DECLARE_EVENT_CLASS(i915_gem_request,
 						i915_gem_request_get_ring(req);
 			   __entry->dev = ring->dev->primary->index;
 			   __entry->ring = ring->id;
+			   __entry->uniq = req ? req->uniq : 0;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   ),
 
-	    TP_printk("dev=%u, ring=%u, seqno=%u",
-		      __entry->dev, __entry->ring, __entry->seqno)
+	    TP_printk("dev=%u, ring=%u, uniq=%u, seqno=%u",
+		      __entry->dev, __entry->ring, __entry->uniq,
+		      __entry->seqno)
 );
 
 DEFINE_EVENT(i915_gem_request, i915_gem_request_add,
@@ -600,6 +607,7 @@ TRACE_EVENT(i915_gem_request_wait_begin,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, seqno)
 			     __field(bool, blocking)
 			     ),
@@ -615,13 +623,14 @@ TRACE_EVENT(i915_gem_request_wait_begin,
 						i915_gem_request_get_ring(req);
 			   __entry->dev = ring->dev->primary->index;
 			   __entry->ring = ring->id;
+			   __entry->uniq = req ? req->uniq : 0;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   __entry->blocking =
 				     mutex_is_locked(&ring->dev->struct_mutex);
 			   ),
 
-	    TP_printk("dev=%u, ring=%u, seqno=%u, blocking=%s",
-		      __entry->dev, __entry->ring,
+	    TP_printk("dev=%u, ring=%u, uniq=%u, seqno=%u, blocking=%s",
+		      __entry->dev, __entry->ring, __entry->uniq,
 		      __entry->seqno, __entry->blocking ?  "yes (NB)" : "no")
 );
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 07/39] drm/i915: Start of GPU scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Initial creation of scheduler source files. Note that this patch
implements most of the scheduler functionality but does not hook it in
to the driver yet. It also leaves the scheduler code in 'pass through'
mode so that even when it is hooked in, it will not actually do very
much. This allows the hooks to be added one at a time in byte size
chunks and only when the scheduler is finally enabled at the end does
anything start happening.

The general theory of operation is that when batch buffers are
submitted to the driver, the execbuffer() code assigns a unique
request and then packages up all the information required to execute
the batch buffer at a later time. This package is given over to the
scheduler which adds it to an internal node list. The scheduler also
scans the list of objects associated with the batch buffer and
compares them against the objects already in use by other buffers in
the node list. If matches are found then the new batch buffer node is
marked as being dependent upon the matching node. The same is done for
the context object. The scheduler also bumps up the priority of such
matching nodes on the grounds that the more dependencies a given batch
buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers
in flight on the hardware at any given time. If fewer than this are
currently executing when a new node is queued, then the node is passed
straight through to the submit function. Otherwise it is simply added
to the queue and the driver returns back to user land.

As each batch buffer completes, it raises an interrupt which wakes up
the scheduler. Note that it is possible for multiple buffers to
complete before the IRQ handler gets to run. Further, it is possible
for the seqno values to be un-ordered (particularly once pre-emption
is enabled). However, the scheduler keeps the list of executing
buffers in order of hardware submission. Thus it can scan through the
list until a matching seqno is found and then mark all in flight nodes
from that point on as completed.

A deferred work queue is also poked by the interrupt handler. When
this wakes up it can do more involved processing such as actually
removing completed nodes from the queue and freeing up the resources
associated with them (internal memory allocations, DRM object
references, context reference, etc.). The work handler also checks the
in flight count and calls the submission code if a new slot has
appeared.

When the scheduler's submit code is called, it scans the queued node
list for the highest priority node that has no unmet dependencies.
Note that the dependency calculation is complex as it must take
inter-ring dependencies and potential preemptions into account. Note
also that in the future this will be extended to include external
dependencies such as the Android Native Sync file descriptors and/or
the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for
submission to the hardware. The in flight count is then re-checked and
a new node popped from the list if appropriate.

Note that this patch does not implement pre-emptive scheduling. Only
basic scheduling by re-ordering batch buffer submission is currently
implemented.

v2: Changed priority levels to +/-1023 due to feedback from Chris
Wilson.

Removed redundant index from scheduler node.

Changed time stamps to use jiffies instead of raw monotonic. This
provides lower resolution but improved compatibility with other i915
code.

Major re-write of completion tracking code due to struct fence
conversion. The scheduler no longer has it's own private IRQ handler
but just lets the existing request code handle completion events.
Instead, the scheduler now hooks into the request notify code to be
told when a request has completed.

Reduced driver mutex locking scope. Removal of scheduler nodes no
longer grabs the mutex lock.

Change-Id: I1e08f59e650a3c2bbaaa9de7627da33849b06106
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/Makefile         |   1 +
 drivers/gpu/drm/i915/i915_drv.h       |   4 +
 drivers/gpu/drm/i915/i915_gem.c       |   5 +
 drivers/gpu/drm/i915/i915_scheduler.c | 740 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  90 +++++
 5 files changed, 840 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h

diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 15398c5..79cb38b 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -10,6 +10,7 @@ ccflags-y := -Werror
 i915-y := i915_drv.o \
 	  i915_irq.o \
 	  i915_params.o \
+	  i915_scheduler.o \
           i915_suspend.o \
 	  i915_sysfs.o \
 	  intel_csr.o \
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 5d390d9..23aed32 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {
 	struct drm_i915_gem_request     *request;
 };
 
+struct i915_scheduler;
+
 /* used in computing the new watermarks state */
 struct intel_wm_config {
 	unsigned int num_pipes_active;
@@ -1947,6 +1949,8 @@ struct drm_i915_private {
 
 	struct i915_runtime_pm pm;
 
+	struct i915_scheduler *scheduler;
+
 	/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */
 	struct {
 		int (*execbuf_submit)(struct i915_execbuffer_params *params,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index b23746a..5239340 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -38,6 +38,7 @@
 #include <linux/pci.h>
 #include <linux/dma-buf.h>
 #include <../drivers/android/sync.h>
+#include "i915_scheduler.h"
 
 #define RQ_BUG_ON(expr)
 
@@ -5277,6 +5278,10 @@ int i915_gem_init(struct drm_device *dev)
 	 */
 	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
+	ret = i915_scheduler_init(dev);
+	if (ret)
+		goto out_unlock;
+
 	ret = i915_gem_init_userptr(dev);
 	if (ret)
 		goto out_unlock;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
new file mode 100644
index 0000000..76973ab
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -0,0 +1,740 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include "i915_drv.h"
+#include "intel_drv.h"
+#include "i915_scheduler.h"
+
+static int         i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node);
+static int         i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
+						   struct i915_scheduler_queue_entry *remove);
+static int         i915_scheduler_submit(struct intel_engine_cs *ring,
+					 bool is_locked);
+static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
+					       struct intel_engine_cs *ring);
+static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
+static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
+						struct i915_scheduler_queue_entry *target,
+						uint32_t bump);
+
+int i915_scheduler_init(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	int                     r;
+
+	if (scheduler)
+		return 0;
+
+	scheduler = kzalloc(sizeof(*scheduler), GFP_KERNEL);
+	if (!scheduler)
+		return -ENOMEM;
+
+	spin_lock_init(&scheduler->lock);
+
+	for (r = 0; r < I915_NUM_RINGS; r++)
+		INIT_LIST_HEAD(&scheduler->node_queue[r]);
+
+	/* Default tuning values: */
+	scheduler->priority_level_min     = -1023;
+	scheduler->priority_level_max     = 1023;
+	scheduler->priority_level_preempt = 900;
+	scheduler->min_flying             = 2;
+
+	dev_priv->scheduler = scheduler;
+
+	return 0;
+}
+
+int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
+{
+	struct drm_i915_private *dev_priv = qe->params.dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring = qe->params.ring;
+	struct i915_scheduler_queue_entry  *node;
+	struct i915_scheduler_queue_entry  *test;
+	unsigned long       flags;
+	bool                not_flying, found;
+	int                 i, j, r;
+	int                 incomplete = 0;
+
+	BUG_ON(!scheduler);
+
+	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
+		int ret;
+
+		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
+		ret = dev_priv->gt.execbuf_final(&qe->params);
+		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
+
+		/*
+		 * Don't do any clean up on failure because the caller will
+		 * do it all anyway.
+		 */
+		if (ret)
+			return ret;
+
+		/* Free everything that is owned by the QE structure: */
+		kfree(qe->params.cliprects);
+		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
+
+		return 0;
+	}
+
+	node = kmalloc(sizeof(*node), GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	*node = *qe;
+	INIT_LIST_HEAD(&node->link);
+	node->status = i915_sqs_queued;
+	node->stamp  = jiffies;
+	i915_gem_request_reference(node->params.request);
+
+	/* Need to determine the number of incomplete entries in the list as
+	 * that will be the maximum size of the dependency list.
+	 *
+	 * Note that the allocation must not be made with the spinlock acquired
+	 * as kmalloc can sleep. However, the unlock/relock is safe because no
+	 * new entries can be queued up during the unlock as the i915 driver
+	 * mutex is still held. Entries could be removed from the list but that
+	 * just means the dep_list will be over-allocated which is fine.
+	 */
+	spin_lock_irqsave(&scheduler->lock, flags);
+	for (r = 0; r < I915_NUM_RINGS; r++) {
+		list_for_each_entry(test, &scheduler->node_queue[r], link) {
+			if (I915_SQS_IS_COMPLETE(test))
+				continue;
+
+			incomplete++;
+		}
+	}
+
+	/* Temporarily unlock to allocate memory: */
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+	if (incomplete) {
+		node->dep_list = kmalloc(sizeof(node->dep_list[0]) * incomplete,
+					 GFP_KERNEL);
+		if (!node->dep_list) {
+			kfree(node);
+			return -ENOMEM;
+		}
+	} else
+		node->dep_list = NULL;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	node->num_deps = 0;
+
+	if (node->dep_list) {
+		for (r = 0; r < I915_NUM_RINGS; r++) {
+			list_for_each_entry(test, &scheduler->node_queue[r], link) {
+				if (I915_SQS_IS_COMPLETE(test))
+					continue;
+
+				/*
+				 * Batches on the same ring for the same
+				 * context must be kept in order.
+				 */
+				found = (node->params.ctx == test->params.ctx) &&
+					(node->params.ring == test->params.ring);
+
+				/*
+				 * Batches working on the same objects must
+				 * be kept in order.
+				 */
+				for (i = 0; (i < node->num_objs) && !found; i++) {
+					for (j = 0; j < test->num_objs; j++) {
+						if (node->saved_objects[i].obj !=
+							    test->saved_objects[j].obj)
+							continue;
+
+						found = true;
+						break;
+					}
+				}
+
+				if (found) {
+					node->dep_list[node->num_deps] = test;
+					node->num_deps++;
+				}
+			}
+		}
+
+		BUG_ON(node->num_deps > incomplete);
+	}
+
+	if (node->priority > scheduler->priority_level_max)
+		node->priority = scheduler->priority_level_max;
+	else if (node->priority < scheduler->priority_level_min)
+		node->priority = scheduler->priority_level_min;
+
+	if ((node->priority > 0) && node->num_deps) {
+		i915_scheduler_priority_bump_clear(scheduler);
+
+		for (i = 0; i < node->num_deps; i++)
+			i915_scheduler_priority_bump(scheduler,
+					node->dep_list[i], node->priority);
+	}
+
+	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
+
+	not_flying = i915_scheduler_count_flying(scheduler, ring) <
+						 scheduler->min_flying;
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (not_flying)
+		i915_scheduler_submit(ring, true);
+
+	return 0;
+}
+
+static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
+{
+	struct drm_i915_private *dev_priv = node->params.dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring;
+
+	BUG_ON(!scheduler);
+	BUG_ON(!node);
+	BUG_ON(node->status != i915_sqs_popped);
+
+	ring = node->params.ring;
+
+	/* Add the node (which should currently be in state none) to the front
+	 * of the queue. This ensure that flying nodes are always held in
+	 * hardware submission order. */
+	list_add(&node->link, &scheduler->node_queue[ring->id]);
+
+	node->status = i915_sqs_flying;
+
+	if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
+		bool    success = true;
+
+		success = ring->irq_get(ring);
+		if (success)
+			scheduler->flags[ring->id] |= i915_sf_interrupts_enabled;
+		else
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * Nodes are considered valid dependencies if they are queued on any ring or
+ * if they are in flight on a different ring. In flight on the same ring is no
+ * longer interesting for non-premptive nodes as the ring serialises execution.
+ * For pre-empting nodes, all in flight dependencies are valid as they must not
+ * be jumped by the act of pre-empting.
+ *
+ * Anything that is neither queued nor flying is uninteresting.
+ */
+static inline bool i915_scheduler_is_dependency_valid(
+			struct i915_scheduler_queue_entry *node, uint32_t idx)
+{
+	struct i915_scheduler_queue_entry *dep;
+
+	dep = node->dep_list[idx];
+	if (!dep)
+		return false;
+
+	if (I915_SQS_IS_QUEUED(dep))
+		return true;
+
+	if (I915_SQS_IS_FLYING(dep)) {
+		if (node->params.ring != dep->params.ring)
+			return true;
+	}
+
+	return false;
+}
+
+static uint32_t i915_scheduler_count_flying(struct i915_scheduler *scheduler,
+					    struct intel_engine_cs *ring)
+{
+	struct i915_scheduler_queue_entry *node;
+	uint32_t                          flying = 0;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link)
+		if (I915_SQS_IS_FLYING(node))
+			flying++;
+
+	return flying;
+}
+
+/* Add a popped node back in to the queue. For example, because the ring was
+ * hung when execfinal() was called and thus the ring submission needs to be
+ * retried later. */
+static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
+{
+	BUG_ON(!node);
+	BUG_ON(!I915_SQS_IS_FLYING(node));
+
+	node->status = i915_sqs_queued;
+	node->params.request->seqno = 0;
+}
+
+/* Give up on a popped node completely. For example, because it is causing the
+ * ring to hang or is using some resource that no longer exists. */
+static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
+{
+	BUG_ON(!node);
+	BUG_ON(!I915_SQS_IS_FLYING(node));
+
+	node->status = i915_sqs_dead;
+}
+
+/*
+ * A sequence number has popped out of the hardware and the request handling
+ * code has mapped it back to a request and will mark that request complete.
+ * It also calls this function to notify the scheduler about the completion
+ * so the scheduler's node can be updated appropriately.
+ * Returns true if the request is scheduler managed, false if not.
+ */
+bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
+{
+	struct drm_i915_private *dev_priv  = to_i915(req->ring->dev);
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	/* XXX: Need to map back from request to node */
+	struct i915_scheduler_queue_entry *node = NULL;
+	unsigned long       flags;
+
+	if (!node)
+		return false;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	WARN_ON(!I915_SQS_IS_FLYING(node));
+
+	/* Node was in flight so mark it as complete. */
+	if (req->cancelled)
+		node->status = i915_sqs_dead;
+	else
+		node->status = i915_sqs_complete;
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	/*
+	 * XXX: If the in-flight list is now empty then new work should be
+	 * submitted. However, this function is called from interrupt context
+	 * and thus cannot acquire mutex locks and other such things that are
+	 * necessary for fresh submission.
+	 */
+
+	return true;
+}
+
+int i915_scheduler_remove(struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node, *node_next;
+	unsigned long       flags;
+	int                 flying = 0, queued = 0;
+	int                 ret = 0;
+	bool                do_submit;
+	uint32_t            min_seqno;
+	struct list_head    remove;
+
+	if (list_empty(&scheduler->node_queue[ring->id]))
+		return 0;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	/* /i915_scheduler_dump_locked(ring, "remove/pre");/ */
+
+	/*
+	 * In the case where the system is idle, starting 'min_seqno' from a big
+	 * number will cause all nodes to be removed as they are now back to
+	 * being in-order. However, this will be a problem if the last one to
+	 * complete was actually out-of-order as the ring seqno value will be
+	 * lower than one or more completed buffers. Thus code looking for the
+	 * completion of said buffers will wait forever.
+	 * Instead, use the hardware seqno as the starting point. This means
+	 * that some buffers might be kept around even in a completely idle
+	 * system but it should guarantee that no-one ever gets confused when
+	 * waiting for buffer completion.
+	 */
+	min_seqno = ring->get_seqno(ring, true);
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (I915_SQS_IS_QUEUED(node))
+			queued++;
+		else if (I915_SQS_IS_FLYING(node))
+			flying++;
+		else if (I915_SQS_IS_COMPLETE(node))
+			continue;
+
+		if (node->params.request->seqno == 0)
+			continue;
+
+		if (!i915_seqno_passed(node->params.request->seqno, min_seqno))
+			min_seqno = node->params.request->seqno;
+	}
+
+	INIT_LIST_HEAD(&remove);
+	list_for_each_entry_safe(node, node_next, &scheduler->node_queue[ring->id], link) {
+		/*
+		 * Only remove completed nodes which have a lower seqno than
+		 * all pending nodes. While there is the possibility of the
+		 * ring's seqno counting backwards, all higher buffers must
+		 * be remembered so that the 'i915_seqno_passed()' test can
+		 * report that they have in fact passed.
+		 *
+		 * NB: This is not true for 'dead' nodes. The GPU reset causes
+		 * the software seqno to restart from its initial value. Thus
+		 * the dead nodes must be removed even though their seqno values
+		 * are potentially vastly greater than the current ring seqno.
+		 */
+		if (!I915_SQS_IS_COMPLETE(node))
+			continue;
+
+		if (node->status != i915_sqs_dead) {
+			if (i915_seqno_passed(node->params.request->seqno, min_seqno) &&
+			    (node->params.request->seqno != min_seqno))
+				continue;
+		}
+
+		list_del(&node->link);
+		list_add(&node->link, &remove);
+
+		/* Strip the dependency info while the mutex is still locked */
+		i915_scheduler_remove_dependent(scheduler, node);
+
+		continue;
+	}
+
+	/*
+	 * No idea why but this seems to cause problems occasionally.
+	 * Note that the 'irq_put' code is internally reference counted
+	 * and spin_locked so it should be safe to call.
+	 */
+	/*if ((scheduler->flags[ring->id] & i915_sf_interrupts_enabled) &&
+	    (first_flight[ring->id] == NULL)) {
+		ring->irq_put(ring);
+		scheduler->flags[ring->id] &= ~i915_sf_interrupts_enabled;
+	}*/
+
+	/* Launch more packets now? */
+	do_submit = (queued > 0) && (flying < scheduler->min_flying);
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (!do_submit && list_empty(&remove))
+		return ret;
+
+	mutex_lock(&ring->dev->struct_mutex);
+
+	if (do_submit)
+		ret = i915_scheduler_submit(ring, true);
+
+	while (!list_empty(&remove)) {
+		node = list_first_entry(&remove, typeof(*node), link);
+		list_del(&node->link);
+
+		/* The batch buffer must be unpinned before it is unreferenced
+		 * otherwise the unpin fails with a missing vma!? */
+		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
+
+		/* Free everything that is owned by the node: */
+		i915_gem_request_unreference(node->params.request);
+		kfree(node->params.cliprects);
+		kfree(node->dep_list);
+		kfree(node);
+	}
+
+	mutex_unlock(&ring->dev->struct_mutex);
+
+	return ret;
+}
+
+static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
+{
+	struct i915_scheduler_queue_entry *node;
+	int i;
+
+	/*
+	 * Ensure circular dependencies don't cause problems and that a bump
+	 * by object usage only bumps each using buffer once:
+	 */
+	for (i = 0; i < I915_NUM_RINGS; i++) {
+		list_for_each_entry(node, &scheduler->node_queue[i], link)
+			node->bumped = false;
+	}
+}
+
+static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
+					struct i915_scheduler_queue_entry *target,
+					uint32_t bump)
+{
+	uint32_t new_priority;
+	int      i, count;
+
+	if (target->priority >= scheduler->priority_level_max)
+		return 1;
+
+	if (target->bumped)
+		return 0;
+
+	new_priority = target->priority + bump;
+	if ((new_priority <= target->priority) ||
+	    (new_priority > scheduler->priority_level_max))
+		target->priority = scheduler->priority_level_max;
+	else
+		target->priority = new_priority;
+
+	count = 1;
+	target->bumped = true;
+
+	for (i = 0; i < target->num_deps; i++) {
+		if (!target->dep_list[i])
+			continue;
+
+		if (target->dep_list[i]->bumped)
+			continue;
+
+		count += i915_scheduler_priority_bump(scheduler,
+						      target->dep_list[i],
+						      bump);
+	}
+
+	return count;
+}
+
+static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
+				    struct i915_scheduler_queue_entry **pop_node,
+				    unsigned long *flags)
+{
+	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *best;
+	struct i915_scheduler_queue_entry  *node;
+	int     ret;
+	int     i;
+	bool	any_queued;
+	bool	has_local, has_remote, only_remote;
+
+	*pop_node = NULL;
+	ret = -ENODATA;
+
+	any_queued = false;
+	only_remote = false;
+	best = NULL;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+		any_queued = true;
+
+		has_local  = false;
+		has_remote = false;
+		for (i = 0; i < node->num_deps; i++) {
+			if (!i915_scheduler_is_dependency_valid(node, i))
+				continue;
+
+			if (node->dep_list[i]->params.ring == node->params.ring)
+				has_local = true;
+			else
+				has_remote = true;
+		}
+
+		if (has_remote && !has_local)
+			only_remote = true;
+
+		if (!has_local && !has_remote) {
+			if (!best ||
+			    (node->priority > best->priority))
+				best = node;
+		}
+	}
+
+	if (best) {
+		list_del(&best->link);
+
+		INIT_LIST_HEAD(&best->link);
+		best->status  = i915_sqs_popped;
+
+		ret = 0;
+	} else {
+		/* Can only get here if:
+		 * (a) there are no buffers in the queue
+		 * (b) all queued buffers are dependent on other buffers
+		 *     e.g. on a buffer that is in flight on a different ring
+		 */
+		if (only_remote) {
+			/* The only dependent buffers are on another ring. */
+			ret = -EAGAIN;
+		} else if (any_queued) {
+			/* It seems that something has gone horribly wrong! */
+			DRM_ERROR("Broken dependency tracking on ring %d!\n",
+				  (int) ring->id);
+		}
+	}
+
+	/* i915_scheduler_dump_queue_pop(ring, best); */
+
+	*pop_node = best;
+	return ret;
+}
+
+static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
+{
+	struct drm_device   *dev = ring->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	unsigned long       flags;
+	int                 ret = 0, count = 0;
+
+	if (!was_locked) {
+		ret = i915_mutex_lock_interruptible(dev);
+		if (ret)
+			return ret;
+	}
+
+	BUG_ON(!mutex_is_locked(&dev->struct_mutex));
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	/* First time around, complain if anything unexpected occurs: */
+	ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
+	if (ret) {
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (!was_locked)
+			mutex_unlock(&dev->struct_mutex);
+
+		return ret;
+	}
+
+	do {
+		BUG_ON(!node);
+		BUG_ON(node->params.ring != ring);
+		BUG_ON(node->status != i915_sqs_popped);
+		count++;
+
+		/* The call to pop above will have removed the node from the
+		 * list. So add it back in and mark it as in flight. */
+		i915_scheduler_fly_node(node);
+
+		scheduler->flags[ring->id] |= i915_sf_submitting;
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+		ret = dev_priv->gt.execbuf_final(&node->params);
+		spin_lock_irqsave(&scheduler->lock, flags);
+		scheduler->flags[ring->id] &= ~i915_sf_submitting;
+
+		if (ret) {
+			int requeue = 1;
+
+			/* Oh dear! Either the node is broken or the ring is
+			 * busy. So need to kill the node or requeue it and try
+			 * again later as appropriate. */
+
+			switch (-ret) {
+			case ENODEV:
+			case ENOENT:
+				/* Fatal errors. Kill the node. */
+				requeue = -1;
+			break;
+
+			case EAGAIN:
+			case EBUSY:
+			case EIO:
+			case ENOMEM:
+			case ERESTARTSYS:
+			case EINTR:
+				/* Supposedly recoverable errors. */
+			break;
+
+			default:
+				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
+						 ring->name, ret);
+				/* Assume it is recoverable and hope for the best. */
+			break;
+			}
+
+			/* Check that the watchdog/reset code has not nuked
+			 * the node while we weren't looking: */
+			if (node->status == i915_sqs_dead)
+				requeue = 0;
+
+			if (requeue == 1) {
+				i915_scheduler_node_requeue(node);
+				/* No point spinning if the ring is currently
+				 * unavailable so just give up and come back
+				 * later. */
+				break;
+			} else if (requeue == -1)
+				i915_scheduler_node_kill(node);
+		}
+
+		/* Keep launching until the sky is sufficiently full. */
+		if (i915_scheduler_count_flying(scheduler, ring) >=
+						scheduler->min_flying)
+			break;
+
+		ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
+	} while (ret == 0);
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (!was_locked)
+		mutex_unlock(&dev->struct_mutex);
+
+	/* Don't complain about not being able to submit extra entries */
+	if (ret == -ENODATA)
+		ret = 0;
+
+	return (ret < 0) ? ret : count;
+}
+
+static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
+					   struct i915_scheduler_queue_entry *remove)
+{
+	struct i915_scheduler_queue_entry  *node;
+	int     i, r;
+	int     count = 0;
+
+	for (i = 0; i < remove->num_deps; i++)
+		if ((remove->dep_list[i]) &&
+		    (!I915_SQS_IS_COMPLETE(remove->dep_list[i])))
+			count++;
+	BUG_ON(count);
+
+	for (r = 0; r < I915_NUM_RINGS; r++) {
+		list_for_each_entry(node, &scheduler->node_queue[r], link) {
+			for (i = 0; i < node->num_deps; i++) {
+				if (node->dep_list[i] != remove)
+					continue;
+
+				node->dep_list[i] = NULL;
+			}
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
new file mode 100644
index 0000000..ee39b77
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -0,0 +1,90 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef _I915_SCHEDULER_H_
+#define _I915_SCHEDULER_H_
+
+enum i915_scheduler_queue_status {
+	/* Limbo: */
+	i915_sqs_none = 0,
+	/* Not yet submitted to hardware: */
+	i915_sqs_queued,
+	/* Popped from queue, ready to fly: */
+	i915_sqs_popped,
+	/* Sent to hardware for processing: */
+	i915_sqs_flying,
+	/* Finished processing on the hardware: */
+	i915_sqs_complete,
+	/* Killed by watchdog or catastrophic submission failure: */
+	i915_sqs_dead,
+	/* Limit value for use with arrays/loops */
+	i915_sqs_MAX
+};
+
+#define I915_SQS_IS_QUEUED(node)	(((node)->status == i915_sqs_queued))
+#define I915_SQS_IS_FLYING(node)	(((node)->status == i915_sqs_flying))
+#define I915_SQS_IS_COMPLETE(node)	(((node)->status == i915_sqs_complete) || \
+					 ((node)->status == i915_sqs_dead))
+
+struct i915_scheduler_obj_entry {
+	struct drm_i915_gem_object          *obj;
+};
+
+struct i915_scheduler_queue_entry {
+	struct i915_execbuffer_params       params;
+	/* -1023 = lowest priority, 0 = default, 1023 = highest */
+	int32_t                             priority;
+	struct i915_scheduler_obj_entry     *saved_objects;
+	int                                 num_objs;
+	bool                                bumped;
+	struct i915_scheduler_queue_entry   **dep_list;
+	int                                 num_deps;
+	enum i915_scheduler_queue_status    status;
+	unsigned long                       stamp;
+	struct list_head                    link;
+};
+
+struct i915_scheduler {
+	struct list_head    node_queue[I915_NUM_RINGS];
+	uint32_t            flags[I915_NUM_RINGS];
+	spinlock_t          lock;
+
+	/* Tuning parameters: */
+	int32_t             priority_level_min;
+	int32_t             priority_level_max;
+	int32_t             priority_level_preempt;
+	uint32_t            min_flying;
+};
+
+/* Flag bits for i915_scheduler::flags */
+enum {
+	i915_sf_interrupts_enabled  = (1 << 0),
+	i915_sf_submitting          = (1 << 1),
+};
+
+int         i915_scheduler_init(struct drm_device *dev);
+int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
+bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
+
+#endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 08/39] drm/i915: Prepare retire_requests to handle out-of-order seqnos

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

A major point of the GPU scheduler is that it re-orders batch buffers
after they have been submitted to the driver. This leads to requests
completing out of order. In turn, this means that the retire
processing can no longer assume that all completed entries are at the
front of the list. Rather than attempting to re-order the request list
on a regular basis, it is better to simply scan the entire list.

v2: Removed deferred free code as no longer necessary due to request
handling updates.

Change-Id: I7eb6793581d9d28eb832e0e94c116b7202fa1b26
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c | 31 +++++++++++++------------------
 1 file changed, 13 insertions(+), 18 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 5239340..d38e013 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3218,6 +3218,7 @@ void i915_gem_reset(struct drm_device *dev)
 void
 i915_gem_retire_requests_ring(struct intel_engine_cs *ring)
 {
+	struct drm_i915_gem_object *obj, *obj_next;
 	struct drm_i915_gem_request *req, *req_next;
 	LIST_HEAD(list_head);
 	unsigned long flags;
@@ -3231,37 +3232,31 @@ i915_gem_retire_requests_ring(struct intel_engine_cs *ring)
 	 */
 	i915_gem_request_notify(ring, false);
 
+	/*
+	 * Note that request entries might be out of order due to rescheduling
+	 * and pre-emption. Thus both lists must be processed in their entirety
+	 * rather than stopping at the first non-complete entry.
+	 */
+
 	/* Retire requests first as we use it above for the early return.
 	 * If we retire requests last, we may use a later seqno and so clear
 	 * the requests lists without clearing the active list, leading to
 	 * confusion.
 	 */
-	while (!list_empty(&ring->request_list)) {
-		struct drm_i915_gem_request *request;
-
-		request = list_first_entry(&ring->request_list,
-					   struct drm_i915_gem_request,
-					   list);
-
-		if (!i915_gem_request_completed(request))
-			break;
+	list_for_each_entry_safe(req, req_next, &ring->request_list, list) {
+		if (!i915_gem_request_completed(req))
+			continue;
 
-		i915_gem_request_retire(request);
+		i915_gem_request_retire(req);
 	}
 
 	/* Move any buffers on the active list that are no longer referenced
 	 * by the ringbuffer to the flushing/inactive lists as appropriate,
 	 * before we free the context associated with the requests.
 	 */
-	while (!list_empty(&ring->active_list)) {
-		struct drm_i915_gem_object *obj;
-
-		obj = list_first_entry(&ring->active_list,
-				      struct drm_i915_gem_object,
-				      ring_list[ring->id]);
-
+	list_for_each_entry_safe(obj, obj_next, &ring->active_list, ring_list[ring->id]) {
 		if (!list_empty(&obj->last_read_req[ring->id]->list))
-			break;
+			continue;
 
 		i915_gem_object_retire__read(obj, ring->id);
 	}
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 09/39] drm/i915: Disable hardware semaphores when GPU scheduler is enabled

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Hardware sempahores require seqno values to be continuously incrementing.
However, the scheduler's reordering of batch buffers means that the seqno values
going through the hardware could be out of order. Thus semaphores can not be
used.

On the other hand, the scheduler superceeds the need for hardware semaphores
anyway. Having one ring stall waiting for something to complete on another ring
is inefficient if that ring could be working on some other, independent task.
This is what the scheduler is meant to do - keep the hardware as busy as
possible by reordering batch buffers to avoid dependency stalls.

Change-Id: I95d1fceacd370455a9720d7dca55cfd0a1f6beaa
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.c         | 9 +++++++++
 drivers/gpu/drm/i915/i915_scheduler.c   | 7 +++++++
 drivers/gpu/drm/i915/i915_scheduler.h   | 1 +
 drivers/gpu/drm/i915/intel_ringbuffer.c | 4 ++++
 4 files changed, 21 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 858d58c..e6be8f5 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -34,6 +34,7 @@
 #include "i915_drv.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
+#include "i915_scheduler.h"
 
 #include <linux/console.h>
 #include <linux/module.h>
@@ -581,6 +582,14 @@ void intel_detect_pch(struct drm_device *dev)
 
 bool i915_semaphore_is_enabled(struct drm_device *dev)
 {
+	/* Hardware semaphores are not compatible with the scheduler due to the
+	 * seqno values being potentially out of order. However, semaphores are
+	 * also not required as the scheduler will handle interring dependencies
+	 * and try do so in a way that does not cause dead time on the hardware.
+	 */
+	if (i915_scheduler_is_enabled(dev))
+		return false;
+
 	if (INTEL_INFO(dev)->gen < 6)
 		return false;
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 76973ab..6386d1c 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -38,6 +38,13 @@ static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler
 						struct i915_scheduler_queue_entry *target,
 						uint32_t bump);
 
+bool i915_scheduler_is_enabled(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+
+	return dev_priv->scheduler != NULL;
+}
+
 int i915_scheduler_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index ee39b77..93faf40 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -83,6 +83,7 @@ enum {
 	i915_sf_submitting          = (1 << 1),
 };
 
+bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 1dec252..34fbe68 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -32,6 +32,7 @@
 #include <drm/i915_drm.h>
 #include "i915_trace.h"
 #include "intel_drv.h"
+#include "i915_scheduler.h"
 
 bool
 intel_ring_initialized(struct intel_engine_cs *ring)
@@ -1411,6 +1412,9 @@ gen6_ring_sync(struct drm_i915_gem_request *waiter_req,
 	u32 wait_mbox = signaller->semaphore.mbox.wait[waiter->id];
 	int ret;
 
+	/* Arithmetic on sequence numbers is unreliable with a scheduler. */
+	BUG_ON(i915_scheduler_is_enabled(signaller->dev));
+
 	/* Throughout all of the GEM code, seqno passed implies our current
 	 * seqno is >= the last seqno executed. However for hardware the
 	 * comparison is strictly greater than.
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 10/39] drm/i915: Force MMIO flips when scheduler enabled

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Change-Id: Ice071af6d88306b0d1c53bdb651a1a3e20bdc1af
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/intel_display.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 9291a1d..c00bc50 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -44,6 +44,7 @@
 #include <drm/drm_plane_helper.h>
 #include <drm/drm_rect.h>
 #include <linux/dma_remapping.h>
+#include "i915_scheduler.h"
 
 /* Primary plane formats for gen <= 3 */
 static const uint32_t i8xx_primary_formats[] = {
@@ -11135,6 +11136,8 @@ static bool use_mmio_flip(struct intel_engine_cs *ring,
 		return true;
 	else if (i915.enable_execlists)
 		return true;
+	else if (i915_scheduler_is_enabled(ring->dev))
+		return true;
 	else
 		return ring != i915_gem_request_get_ring(obj->last_write_req);
 }
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 11/39] drm/i915: Added scheduler hook when closing DRM file handles

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler decouples the submission of batch buffers to the driver with
submission of batch buffers to the hardware. Thus it is possible for an
application to submit work, then close the DRM handle and free up all the
resources that piece of work wishes to use before the work has even been
submitted to the hardware. To prevent this, the scheduler needs to be informed
of the DRM close event so that it can force through any outstanding work
attributed to that file handle.

Change-Id: I24ac056c062b075ff1cc5e2ed2d3fa8e17e85951
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_dma.c       |  3 ++
 drivers/gpu/drm/i915/i915_scheduler.c | 66 +++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  2 ++
 3 files changed, 71 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 731cf31..c2f9c03 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -46,6 +46,7 @@
 #include <linux/vga_switcheroo.h>
 #include <linux/slab.h>
 #include <acpi/video.h>
+#include "i915_scheduler.h"
 #include <linux/pm.h>
 #include <linux/pm_runtime.h>
 #include <linux/oom.h>
@@ -1250,6 +1251,8 @@ void i915_driver_lastclose(struct drm_device *dev)
 
 void i915_driver_preclose(struct drm_device *dev, struct drm_file *file)
 {
+	i915_scheduler_closefile(dev, file);
+
 	mutex_lock(&dev->struct_mutex);
 	i915_gem_context_close(dev, file);
 	i915_gem_release(dev, file);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 6386d1c..25e7ade 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -745,3 +745,69 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
 
 	return 0;
 }
+
+int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
+{
+	struct i915_scheduler_queue_entry  *node;
+	struct drm_i915_private            *dev_priv = dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	struct drm_i915_gem_request        *req;
+	struct intel_engine_cs  *ring;
+	int                     i, ret;
+	unsigned long           flags;
+	bool                    found;
+
+	if (!scheduler)
+		return 0;
+
+	for_each_ring(ring, dev_priv, i) {
+		do {
+			spin_lock_irqsave(&scheduler->lock, flags);
+
+			found = false;
+			list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+				if (I915_SQS_IS_COMPLETE(node))
+					continue;
+
+				if (node->params.file != file)
+					continue;
+
+				found = true;
+				req = node->params.request;
+				i915_gem_request_reference(req);
+				break;
+			}
+
+			spin_unlock_irqrestore(&scheduler->lock, flags);
+
+			if (found) {
+				do {
+					mutex_lock(&dev->struct_mutex);
+					ret = i915_wait_request(req);
+					mutex_unlock(&dev->struct_mutex);
+					if (ret == -EAGAIN)
+						msleep(20);
+				} while (ret == -EAGAIN);
+
+				mutex_lock(&dev->struct_mutex);
+				i915_gem_request_unreference(req);
+				mutex_unlock(&dev->struct_mutex);
+			}
+		} while (found);
+	}
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	for_each_ring(ring, dev_priv, i) {
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (node->params.file != file)
+				continue;
+
+			WARN_ON(!I915_SQS_IS_COMPLETE(node));
+
+			node->params.file = NULL;
+		}
+	}
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 93faf40..9736b8d 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -85,6 +85,8 @@ enum {
 
 bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
+int         i915_scheduler_closefile(struct drm_device *dev,
+				     struct drm_file *file);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 12/39] drm/i915: Added scheduler hook into i915_gem_request_notify()

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler needs to know when requests have completed so that it
can keep its own internal state up to date and can submit new requests
to the hardware from its queue.

v2: Updated due to changes in request handling. The operation is now
reversed from before. Rather than the scheduler being in control of
completion events, it is now the request code itself. The scheduler
merely receives a notification event. It can then optionally request
it's worker thread be woken up after all completion processing is
complete.

Change-Id: I149250a8f9382586514ca324aba1c53063b83e19
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h       |  2 ++
 drivers/gpu/drm/i915/i915_gem.c       | 16 ++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.c | 28 ++++++++++++++++++----------
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 4 files changed, 37 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 23aed32..8c576fe 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2262,6 +2262,8 @@ struct drm_i915_gem_request {
 	/** process identifier submitting this request */
 	struct pid *pid;
 
+	struct i915_scheduler_queue_entry	*scheduler_qe;
+
 	/**
 	 * The ELSP only accepts two elements at a time, so we queue
 	 * context/tail pairs on a given queue (ring->execlist_queue) until the
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index d38e013..8a2fc7c 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2785,6 +2785,7 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 {
 	struct drm_i915_gem_request *req, *req_next;
 	unsigned long flags;
+	bool wake_sched = false;
 	u32 seqno;
 
 	if (list_empty(&ring->fence_signal_list)) {
@@ -2813,6 +2814,14 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 		 */
 		list_del_init(&req->signal_link);
 
+		/*
+		 * NB: Must notify the scheduler before signalling
+		 * the node. Otherwise the node can get retired first
+		 * and call scheduler_clean() while the scheduler
+		 * thinks it is still active.
+		 */
+		wake_sched |= i915_scheduler_notify_request(req);
+
 		if (!req->cancelled) {
 			fence_signal_locked(&req->fence);
 			trace_i915_gem_request_complete(req);
@@ -2829,6 +2838,13 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 
 	if (!fence_locked)
 		spin_unlock_irqrestore(&ring->fence_lock, flags);
+
+	/* Necessary? Or does the fence_signal() call do an implicit wakeup? */
+	wake_up_all(&ring->irq_queue);
+
+	/* Final scheduler processing after all individual updates are done. */
+	if (wake_sched)
+		i915_scheduler_wakeup(ring->dev);
 }
 
 static const char *i915_gem_request_get_driver_name(struct fence *req_fence)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 25e7ade..dd9c710 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -120,6 +120,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	node->stamp  = jiffies;
 	i915_gem_request_reference(node->params.request);
 
+	BUG_ON(node->params.request->scheduler_qe);
+	node->params.request->scheduler_qe = node;
+
 	/* Need to determine the number of incomplete entries in the list as
 	 * that will be the maximum size of the dependency list.
 	 *
@@ -319,14 +322,16 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
  * code has mapped it back to a request and will mark that request complete.
  * It also calls this function to notify the scheduler about the completion
  * so the scheduler's node can be updated appropriately.
- * Returns true if the request is scheduler managed, false if not.
+ * Returns true if the request is scheduler managed, false if not. The return
+ * value is combined for all freshly completed requests and if any were true
+ * then i915_scheduler_wakeup() is called so the scheduler can do further
+ * processing (submit more work) at the end.
  */
 bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 {
 	struct drm_i915_private *dev_priv  = to_i915(req->ring->dev);
 	struct i915_scheduler   *scheduler = dev_priv->scheduler;
-	/* XXX: Need to map back from request to node */
-	struct i915_scheduler_queue_entry *node = NULL;
+	struct i915_scheduler_queue_entry *node = req->scheduler_qe;
 	unsigned long       flags;
 
 	if (!node)
@@ -344,16 +349,18 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
-	/*
-	 * XXX: If the in-flight list is now empty then new work should be
-	 * submitted. However, this function is called from interrupt context
-	 * and thus cannot acquire mutex locks and other such things that are
-	 * necessary for fresh submission.
-	 */
-
 	return true;
 }
 
+/*
+ * Called at the end of seqno interrupt processing if any request has
+ * completed that corresponds to a scheduler node.
+ */
+void i915_scheduler_wakeup(struct drm_device *dev)
+{
+	/* XXX: Need to call i915_scheduler_remove() via work handler. */
+}
+
 int i915_scheduler_remove(struct intel_engine_cs *ring)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
@@ -468,6 +475,7 @@ int i915_scheduler_remove(struct intel_engine_cs *ring)
 			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
 
 		/* Free everything that is owned by the node: */
+		node->params.request->scheduler_qe = NULL;
 		i915_gem_request_unreference(node->params.request);
 		kfree(node->params.cliprects);
 		kfree(node->dep_list);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 9736b8d..a0b38b8 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -89,5 +89,6 @@ int         i915_scheduler_closefile(struct drm_device *dev,
 				     struct drm_file *file);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
+void        i915_scheduler_wakeup(struct drm_device *dev);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 13/39] drm/i915: Added deferred work handler for scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler needs to do interrupt triggered work that is too complex
to do in the interrupt handler. Thus it requires a deferred work
handler to process such tasks asynchronously.

v2: Updated to reduce mutex lock usage. The lock is now only held for
the minimum time within the remove function rather than for the whole
of the worker thread's operation.

Change-Id: I0f7cc2b6f034a50bf8f7e368b60ad8bafd00f993
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_dma.c       |  3 +++
 drivers/gpu/drm/i915/i915_drv.h       | 10 ++++++++++
 drivers/gpu/drm/i915/i915_gem.c       |  2 ++
 drivers/gpu/drm/i915/i915_scheduler.c | 21 +++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 5 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index c2f9c03..8c9246f 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1149,6 +1149,9 @@ int i915_driver_unload(struct drm_device *dev)
 	WARN_ON(unregister_oom_notifier(&dev_priv->mm.oom_notifier));
 	unregister_shrinker(&dev_priv->mm.shrinker);
 
+	/* Cancel the scheduler work handler, which should be idle now. */
+	cancel_work_sync(&dev_priv->mm.scheduler_work);
+
 	io_mapping_free(dev_priv->gtt.mappable);
 	arch_phys_wc_del(dev_priv->gtt.mtrr);
 
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 8c576fe..9a67f7c 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1285,6 +1285,16 @@ struct i915_gem_mm {
 	struct delayed_work retire_work;
 
 	/**
+	 * New scheme is to get an interrupt after every work packet
+	 * in order to allow the low latency scheduling of pending
+	 * packets. The idea behind adding new packets to a pending
+	 * queue rather than directly into the hardware ring buffer
+	 * is to allow high priority packets to over take low priority
+	 * ones.
+	 */
+	struct work_struct scheduler_work;
+
+	/**
 	 * When we detect an idle GPU, we want to turn on
 	 * powersaving features. So once we see that there
 	 * are no more requests outstanding and no more
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 8a2fc7c..f625d88 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -5389,6 +5389,8 @@ i915_gem_load(struct drm_device *dev)
 			  i915_gem_retire_work_handler);
 	INIT_DELAYED_WORK(&dev_priv->mm.idle_work,
 			  i915_gem_idle_work_handler);
+	INIT_WORK(&dev_priv->mm.scheduler_work,
+				i915_gem_scheduler_work_handler);
 	init_waitqueue_head(&dev_priv->gpu_error.reset_queue);
 
 	dev_priv->relative_constants_mode = I915_EXEC_CONSTANTS_REL_GENERAL;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index dd9c710..7521338 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -358,10 +358,12 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
  */
 void i915_scheduler_wakeup(struct drm_device *dev)
 {
-	/* XXX: Need to call i915_scheduler_remove() via work handler. */
+	struct drm_i915_private *dev_priv  = to_i915(dev);
+
+	queue_work(dev_priv->wq, &dev_priv->mm.scheduler_work);
 }
 
-int i915_scheduler_remove(struct intel_engine_cs *ring)
+static int i915_scheduler_remove(struct intel_engine_cs *ring)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
 	struct i915_scheduler   *scheduler = dev_priv->scheduler;
@@ -487,6 +489,21 @@ int i915_scheduler_remove(struct intel_engine_cs *ring)
 	return ret;
 }
 
+void i915_gem_scheduler_work_handler(struct work_struct *work)
+{
+	struct intel_engine_cs  *ring;
+	struct drm_i915_private *dev_priv;
+	struct drm_device       *dev;
+	int                     i;
+
+	dev_priv = container_of(work, struct drm_i915_private, mm.scheduler_work);
+	dev = dev_priv->dev;
+
+	for_each_ring(ring, dev_priv, i) {
+		i915_scheduler_remove(ring);
+	}
+}
+
 static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
 {
 	struct i915_scheduler_queue_entry *node;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index a0b38b8..da095f9 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -90,5 +90,6 @@ int         i915_scheduler_closefile(struct drm_device *dev,
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
+void        i915_gem_scheduler_work_handler(struct work_struct *work);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 14/39] drm/i915: Redirect execbuffer_final() via scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Updated the execbuffer() code to pass the packaged up batch buffer information
to the scheduler rather than calling execbuffer_final() directly. The scheduler
queue() code is currently a stub which simply chains on to _final() immediately.

Change-Id: I2a19062a9e66845f2e886332fc4b5fc7ac992864
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 18 +++++++-----------
 drivers/gpu/drm/i915/intel_lrc.c           | 12 ++++--------
 2 files changed, 11 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 05c9de6..cd4aa64 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -35,6 +35,7 @@
 #include <linux/dma_remapping.h>
 #include <linux/uaccess.h>
 #include <../drivers/android/sync.h>
+#include "i915_scheduler.h"
 
 #define  __EXEC_OBJECT_HAS_PIN (1<<31)
 #define  __EXEC_OBJECT_HAS_FENCE (1<<30)
@@ -1180,6 +1181,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas)
 {
+	struct i915_scheduler_queue_entry *qe;
 	struct drm_device *dev = params->dev;
 	struct intel_engine_cs *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -1224,17 +1226,11 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	ret = dev_priv->gt.execbuf_final(params);
+	qe = container_of(params, typeof(*qe), params);
+	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
 		return ret;
 
-	/*
-	 * Free everything that was stored in the QE structure (until the
-	 * scheduler arrives and does it instead):
-	 */
-	if (params->dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_execbuff_release_batch_obj(params->batch_obj);
-
 	return 0;
 }
 
@@ -1410,8 +1406,8 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	struct intel_engine_cs *ring;
 	struct intel_context *ctx;
 	struct i915_address_space *vm;
-	struct i915_execbuffer_params params_master; /* XXX: will be removed later */
-	struct i915_execbuffer_params *params = &params_master;
+	struct i915_scheduler_queue_entry qe;
+	struct i915_execbuffer_params *params = &qe.params;
 	const u32 ctx_id = i915_execbuffer2_get_context_id(*args);
 	u32 dispatch_flags;
 	int ret;
@@ -1539,7 +1535,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	else
 		vm = &dev_priv->gtt.base;
 
-	memset(&params_master, 0x00, sizeof(params_master));
+	memset(&qe, 0x00, sizeof(qe));
 
 	eb = eb_create(args);
 	if (eb == NULL) {
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 4aae31c..2f49fb9 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -136,6 +136,7 @@
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
 #include "intel_mocs.h"
+#include "i915_scheduler.h"
 
 #define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE)
 #define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE)
@@ -866,6 +867,7 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas)
 {
+	struct i915_scheduler_queue_entry *qe;
 	struct drm_device       *dev = params->dev;
 	struct intel_engine_cs  *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -908,17 +910,11 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	ret = dev_priv->gt.execbuf_final(params);
+	qe = container_of(params, typeof(*qe), params);
+	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
 		return ret;
 
-	/*
-	 * Free everything that was stored in the QE structure (until the
-	 * scheduler arrives and does it instead):
-	 */
-	if (params->dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_execbuff_release_batch_obj(params->batch_obj);
-
 	return 0;
 }
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 15/39] drm/i915: Keep the reserved space mechanism happy

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Ring space is reserved when constructing a request to ensure that the
subsequent 'add_request()' call cannot fail due to waiting for space
on a busy or broken GPU. However, the scheduler jumps in to the middle
of the execbuffer process between request creation and request
submission. Thus it needs to cancel the reserved space when the
request is simply added to the scheduler's queue and not yet
submitted. Similarly, it needs to re-reserve the space when it finally
does want to send the batch buffer to the hardware.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  7 +++++++
 drivers/gpu/drm/i915/i915_scheduler.c      |  4 ++++
 drivers/gpu/drm/i915/intel_lrc.c           | 13 +++++++++++--
 3 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index cd4aa64..12b34d7 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1248,6 +1248,10 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	ret = intel_ring_reserve_space(params->request);
+	if (ret)
+		return ret;
+
 	intel_runtime_pm_get(dev_priv);
 
 	/*
@@ -1308,6 +1312,9 @@ error:
 	 */
 	intel_runtime_pm_put(dev_priv);
 
+	if (ret)
+		intel_ring_reserved_space_cancel(params->request->ringbuf);
+
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 7521338..79d02d7 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -91,6 +91,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
 		int ret;
 
+		intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
+
 		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
 		ret = dev_priv->gt.execbuf_final(&qe->params);
 		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
@@ -120,6 +122,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	node->stamp  = jiffies;
 	i915_gem_request_reference(node->params.request);
 
+	intel_ring_reserved_space_cancel(node->params.request->ringbuf);
+
 	BUG_ON(node->params.request->scheduler_qe);
 	node->params.request->scheduler_qe = node;
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 2f49fb9..16c5302 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -932,13 +932,17 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	ret = intel_logical_ring_reserve_space(params->request);
+	if (ret)
+		return ret;
+
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
 	ret = logical_ring_invalidate_all_caches(params->request);
 	if (ret)
-		return ret;
+		goto err;
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
@@ -962,13 +966,18 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 
 	ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags);
 	if (ret)
-		return ret;
+		goto err;
 
 	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
 	return 0;
+
+err:
+	intel_ring_reserved_space_cancel(params->request->ringbuf);
+
+	return ret;
 }
 
 void intel_execlists_retire_requests(struct intel_engine_cs *ring)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 16/39] drm/i915: Added tracking/locking of batch buffer objects

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler needs to track interdependencies between batch buffers. These are
calculated by analysing the object lists of the buffers and looking for
commonality. The scheduler also needs to keep those buffers locked long after
the initial IOCTL call has returned to user land.

Change-Id: I31e3677ecfc2c9b5a908bda6acc4850432d55f1e
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 48 ++++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.c      | 33 ++++++++++++++++++--
 2 files changed, 76 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 12b34d7..fe90571 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1417,7 +1417,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	struct i915_execbuffer_params *params = &qe.params;
 	const u32 ctx_id = i915_execbuffer2_get_context_id(*args);
 	u32 dispatch_flags;
-	int ret;
+	int ret, i;
 	bool need_relocs;
 	int fd_fence_complete = -1;
 	int fd_fence_wait = lower_32_bits(args->rsvd2);
@@ -1552,6 +1552,14 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		goto pre_mutex_err;
 	}
 
+	qe.saved_objects = kzalloc(
+			sizeof(*qe.saved_objects) * args->buffer_count,
+			GFP_KERNEL);
+	if (!qe.saved_objects) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
 	/* Look up object handles */
 	ret = eb_lookup_vmas(eb, exec, args, vm, file);
 	if (ret)
@@ -1672,7 +1680,26 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->args_DR1                = args->DR1;
 	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
-	params->ctx                     = ctx;
+
+	/*
+	 * Save away the list of objects used by this batch buffer for the
+	 * purpose of tracking inter-buffer dependencies.
+	 */
+	for (i = 0; i < args->buffer_count; i++) {
+		/*
+		 * NB: 'drm_gem_object_lookup()' increments the object's
+		 * reference count and so must be matched by a
+		 * 'drm_gem_object_unreference' call.
+		 */
+		qe.saved_objects[i].obj =
+			to_intel_bo(drm_gem_object_lookup(dev, file,
+							  exec[i].handle));
+	}
+	qe.num_objs = i;
+
+	/* Lock and save the context object as well. */
+	i915_gem_context_reference(ctx);
+	params->ctx = ctx;
 
 	if (args->flags & I915_EXEC_CREATE_FENCE) {
 		/*
@@ -1737,6 +1764,23 @@ err:
 	i915_gem_context_unreference(ctx);
 	eb_destroy(eb);
 
+	if (qe.saved_objects) {
+		/* Need to release the objects: */
+		for (i = 0; i < qe.num_objs; i++) {
+			if (!qe.saved_objects[i].obj)
+				continue;
+
+			drm_gem_object_unreference(
+					&qe.saved_objects[i].obj->base);
+		}
+
+		kfree(qe.saved_objects);
+
+		/* Context too */
+		if (params->ctx)
+			i915_gem_context_unreference(params->ctx);
+	}
+
 	/*
 	 * If the request was created but not successfully submitted then it
 	 * must be freed again. If it was submitted then it is being tracked
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 79d02d7..cd69f53 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -104,7 +104,23 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		if (ret)
 			return ret;
 
-		/* Free everything that is owned by the QE structure: */
+		/* Need to release the objects: */
+		for (i = 0; i < qe->num_objs; i++) {
+			if (!qe->saved_objects[i].obj)
+				continue;
+
+			drm_gem_object_unreference(&qe->saved_objects[i].obj->base);
+		}
+
+		kfree(qe->saved_objects);
+		qe->saved_objects = NULL;
+		qe->num_objs = 0;
+
+		/* Free the context object too: */
+		if (qe->params.ctx)
+			i915_gem_context_unreference(qe->params.ctx);
+
+		/* And anything else owned by the QE structure: */
 		kfree(qe->params.cliprects);
 		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
 			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
@@ -376,7 +392,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	int                 flying = 0, queued = 0;
 	int                 ret = 0;
 	bool                do_submit;
-	uint32_t            min_seqno;
+	uint32_t            i, min_seqno;
 	struct list_head    remove;
 
 	if (list_empty(&scheduler->node_queue[ring->id]))
@@ -480,7 +496,18 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
 			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
 
-		/* Free everything that is owned by the node: */
+		/* Release the locked buffers: */
+		for (i = 0; i < node->num_objs; i++) {
+			drm_gem_object_unreference(
+					    &node->saved_objects[i].obj->base);
+		}
+		kfree(node->saved_objects);
+
+		/* Context too: */
+		if (node->params.ctx)
+			i915_gem_context_unreference(node->params.ctx);
+
+		/* And anything else owned by the node: */
 		node->params.request->scheduler_qe = NULL;
 		i915_gem_request_unreference(node->params.request);
 		kfree(node->params.cliprects);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 17/39] drm/i915: Hook scheduler node clean up into retire requests

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler keeps its own lock on various DRM objects in order to guarantee
safe access long after the original execbuff IOCTL has completed. This is
especially important when pre-emption is enabled as the batch buffer might need
to be submitted to the hardware multiple times. This patch hooks the clean up of
these locks into the request retire function. The request can only be retired
after it has completed on the hardware and thus is no longer eligible for
re-submission. Thus there is no point holding on to the locks beyond that time.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c       |  3 +++
 drivers/gpu/drm/i915/i915_scheduler.c | 51 ++++++++++++++++++++++++-----------
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 3 files changed, 39 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index f625d88..451ae6d 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1402,6 +1402,9 @@ static void i915_gem_request_retire(struct drm_i915_gem_request *request)
 		fence_signal_locked(&request->fence);
 	}
 
+	if (request->scheduler_qe)
+		i915_gem_scheduler_clean_node(request->scheduler_qe);
+
 	i915_gem_request_unreference(request);
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index cd69f53..5dc3497 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -383,6 +383,38 @@ void i915_scheduler_wakeup(struct drm_device *dev)
 	queue_work(dev_priv->wq, &dev_priv->mm.scheduler_work);
 }
 
+void i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *node)
+{
+	uint32_t i;
+
+	if (WARN(!I915_SQS_IS_COMPLETE(node), "Incomplete node: %d!\n", node->status))
+		return;
+
+	if (node->params.batch_obj) {
+		/* The batch buffer must be unpinned before it is unreferenced
+		 * otherwise the unpin fails with a missing vma!? */
+		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
+
+		node->params.batch_obj = NULL;
+	}
+
+	/* Release the locked buffers: */
+	for (i = 0; i < node->num_objs; i++) {
+		drm_gem_object_unreference(
+				    &node->saved_objects[i].obj->base);
+	}
+	kfree(node->saved_objects);
+	node->saved_objects = NULL;
+	node->num_objs = 0;
+
+	/* Context too: */
+	if (node->params.ctx) {
+		i915_gem_context_unreference(node->params.ctx);
+		node->params.ctx = NULL;
+	}
+}
+
 static int i915_scheduler_remove(struct intel_engine_cs *ring)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
@@ -392,7 +424,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	int                 flying = 0, queued = 0;
 	int                 ret = 0;
 	bool                do_submit;
-	uint32_t            i, min_seqno;
+	uint32_t            min_seqno;
 	struct list_head    remove;
 
 	if (list_empty(&scheduler->node_queue[ring->id]))
@@ -491,21 +523,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		node = list_first_entry(&remove, typeof(*node), link);
 		list_del(&node->link);
 
-		/* The batch buffer must be unpinned before it is unreferenced
-		 * otherwise the unpin fails with a missing vma!? */
-		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
-			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
-
-		/* Release the locked buffers: */
-		for (i = 0; i < node->num_objs; i++) {
-			drm_gem_object_unreference(
-					    &node->saved_objects[i].obj->base);
-		}
-		kfree(node->saved_objects);
-
-		/* Context too: */
-		if (node->params.ctx)
-			i915_gem_context_unreference(node->params.ctx);
+		/* Free up all the DRM object references */
+		i915_gem_scheduler_clean_node(node);
 
 		/* And anything else owned by the node: */
 		node->params.request->scheduler_qe = NULL;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index da095f9..8469270 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -87,6 +87,7 @@ bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
 int         i915_scheduler_closefile(struct drm_device *dev,
 				     struct drm_file *file);
+void        i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 18/39] drm/i915: Added scheduler support to __wait_request() calls

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler can cause batch buffers, and hence requests, to be
submitted to the ring out of order and asynchronously to their
submission to the driver. Thus at the point of waiting for the
completion of a given request, it is not even guaranteed that the
request has actually been sent to the hardware yet. Even it is has
been sent, it is possible that it could be pre-empted and thus
'unsent'.

This means that it is necessary to be able to submit requests to the
hardware during the wait call itself. Unfortunately, while some
callers of __wait_request() release the mutex lock first, others do
not (and apparently can not). Hence there is the ability to deadlock
as the wait stalls for submission but the asynchronous submission is
stalled for the mutex lock.

This change hooks the scheduler in to the __wait_request() code to
ensure correct behaviour. That is, flush the target batch buffer
through to the hardware and do not deadlock waiting for something that
cannot currently be submitted. Instead, the wait call must return
EAGAIN at least as far back as necessary to release the mutex lock and
allow the scheduler's asynchronous processing to get in and handle the
pre-emption operation and eventually (re-)submit the work.

Change-Id: I31fe6bc7e38f6ffdd843fcae16e7cc8b1e52a931
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h         |   3 +-
 drivers/gpu/drm/i915/i915_gem.c         |  37 +++++++++--
 drivers/gpu/drm/i915/i915_scheduler.c   | 111 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h   |   4 ++
 drivers/gpu/drm/i915/intel_display.c    |   5 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c |   2 +-
 6 files changed, 151 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 9a67f7c..5ed600c 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -3029,7 +3029,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			unsigned reset_counter,
 			bool interruptible,
 			s64 *timeout,
-			struct intel_rps_client *rps);
+			struct intel_rps_client *rps,
+			bool is_locked);
 int __must_check i915_wait_request(struct drm_i915_gem_request *req);
 int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf);
 int __must_check
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 451ae6d..3994b47 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1207,7 +1207,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			unsigned reset_counter,
 			bool interruptible,
 			s64 *timeout,
-			struct intel_rps_client *rps)
+			struct intel_rps_client *rps,
+			bool is_locked)
 {
 	struct intel_engine_cs *ring = i915_gem_request_get_ring(req);
 	struct drm_device *dev = ring->dev;
@@ -1217,8 +1218,10 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 	DEFINE_WAIT(wait);
 	unsigned long timeout_expire;
 	s64 before, now;
-	int ret;
+	int ret = 0;
+	bool    busy;
 
+	might_sleep();
 	WARN(!intel_irqs_enabled(dev_priv), "IRQs disabled");
 
 	if (i915_gem_request_completed(req))
@@ -1269,6 +1272,22 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			break;
 		}
 
+		if (is_locked) {
+			/* If this request is being processed by the scheduler
+			 * then it is unsafe to sleep with the mutex lock held
+			 * as the scheduler may require the lock in order to
+			 * progress the request. */
+			if (i915_scheduler_is_request_tracked(req, NULL, &busy)) {
+				if (busy) {
+					ret = -EAGAIN;
+					break;
+				}
+			}
+
+			/* If the request is not tracked by the scheduler then the
+			 * regular test can be done. */
+		}
+
 		if (i915_gem_request_completed(req)) {
 			ret = 0;
 			break;
@@ -1449,13 +1468,17 @@ i915_wait_request(struct drm_i915_gem_request *req)
 
 	BUG_ON(!mutex_is_locked(&dev->struct_mutex));
 
+	ret = i915_scheduler_flush_request(req, true);
+	if (ret < 0)
+		return ret;
+
 	ret = i915_gem_check_wedge(&dev_priv->gpu_error, interruptible);
 	if (ret)
 		return ret;
 
 	ret = __i915_wait_request(req,
 				  atomic_read(&dev_priv->gpu_error.reset_counter),
-				  interruptible, NULL, NULL);
+				  interruptible, NULL, NULL, true);
 	if (ret)
 		return ret;
 
@@ -1568,7 +1591,7 @@ i915_gem_object_wait_rendering__nonblocking(struct drm_i915_gem_object *obj,
 	mutex_unlock(&dev->struct_mutex);
 	for (i = 0; ret == 0 && i < n; i++)
 		ret = __i915_wait_request(requests[i], reset_counter, true,
-					  NULL, rps);
+					  NULL, rps, false);
 	mutex_lock(&dev->struct_mutex);
 
 	for (i = 0; i < n; i++) {
@@ -3493,7 +3516,7 @@ i915_gem_wait_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 		if (ret == 0)
 			ret = __i915_wait_request(req[i], reset_counter, true,
 						  args->timeout_ns > 0 ? &args->timeout_ns : NULL,
-						  file->driver_priv);
+						  file->driver_priv, false);
 		i915_gem_request_unreference(req[i]);
 	}
 	return ret;
@@ -3526,7 +3549,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 					  atomic_read(&i915->gpu_error.reset_counter),
 					  i915->mm.interruptible,
 					  NULL,
-					  &i915->rps.semaphores);
+					  &i915->rps.semaphores, true);
 		if (ret)
 			return ret;
 
@@ -4485,7 +4508,7 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	if (target == NULL)
 		return 0;
 
-	ret = __i915_wait_request(target, reset_counter, true, NULL, NULL);
+	ret = __i915_wait_request(target, reset_counter, true, NULL, NULL, false);
 	if (ret == 0)
 		queue_delayed_work(dev_priv->wq, &dev_priv->mm.retire_work, 0);
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 5dc3497..b8b84a6 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -31,6 +31,8 @@ static int         i915_scheduler_remove_dependent(struct i915_scheduler *schedu
 						   struct i915_scheduler_queue_entry *remove);
 static int         i915_scheduler_submit(struct intel_engine_cs *ring,
 					 bool is_locked);
+static int         i915_scheduler_submit_max_priority(struct intel_engine_cs *ring,
+						      bool is_locked);
 static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
 					       struct intel_engine_cs *ring);
 static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
@@ -554,6 +556,57 @@ void i915_gem_scheduler_work_handler(struct work_struct *work)
 	}
 }
 
+int i915_scheduler_flush_request(struct drm_i915_gem_request *req,
+				 bool is_locked)
+{
+	struct drm_i915_private *dev_priv;
+	struct i915_scheduler   *scheduler;
+	unsigned long       flags;
+	int                 flush_count;
+	uint32_t            ring_id;
+
+	if (!req)
+		return -EINVAL;
+
+	dev_priv  = req->ring->dev->dev_private;
+	scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return 0;
+
+	if (!req->scheduler_qe)
+		return 0;
+
+	if (!I915_SQS_IS_QUEUED(req->scheduler_qe))
+		return 0;
+
+	ring_id = req->ring->id;
+	if (is_locked && (scheduler->flags[ring_id] & i915_sf_submitting)) {
+		/* Scheduler is busy already submitting another batch,
+		 * come back later rather than going recursive... */
+		return -EAGAIN;
+	}
+
+	if (list_empty(&scheduler->node_queue[ring_id]))
+		return 0;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	i915_scheduler_priority_bump_clear(scheduler);
+
+	flush_count = i915_scheduler_priority_bump(scheduler,
+			    req->scheduler_qe, scheduler->priority_level_max);
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (flush_count) {
+		DRM_DEBUG_DRIVER("<%s> Bumped %d entries\n", req->ring->name, flush_count);
+		flush_count = i915_scheduler_submit_max_priority(req->ring, is_locked);
+	}
+
+	return flush_count;
+}
+
 static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
 {
 	struct i915_scheduler_queue_entry *node;
@@ -607,6 +660,44 @@ static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 	return count;
 }
 
+static int i915_scheduler_submit_max_priority(struct intel_engine_cs *ring,
+					      bool is_locked)
+{
+	struct i915_scheduler_queue_entry  *node;
+	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	unsigned long	flags;
+	int             ret, count = 0;
+	bool            found;
+
+	do {
+		found = false;
+		spin_lock_irqsave(&scheduler->lock, flags);
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (!I915_SQS_IS_QUEUED(node))
+				continue;
+
+			if (node->priority < scheduler->priority_level_max)
+				continue;
+
+			found = true;
+			break;
+		}
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (!found)
+			break;
+
+		ret = i915_scheduler_submit(ring, is_locked);
+		if (ret < 0)
+			return ret;
+
+		count += ret;
+	} while (found);
+
+	return count;
+}
+
 static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 				    struct i915_scheduler_queue_entry **pop_node,
 				    unsigned long *flags)
@@ -821,6 +912,26 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
 	return 0;
 }
 
+bool i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
+				       bool *completed, bool *busy)
+{
+	struct drm_i915_private *dev_priv = req->ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return false;
+
+	if (req->scheduler_qe == NULL)
+		return false;
+
+	if (completed)
+		*completed = I915_SQS_IS_COMPLETE(req->scheduler_qe);
+	if (busy)
+		*busy      = I915_SQS_IS_QUEUED(req->scheduler_qe);
+
+	return true;
+}
+
 int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 {
 	struct i915_scheduler_queue_entry  *node;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 8469270..42371b3 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -92,5 +92,9 @@ int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *q
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
 void        i915_gem_scheduler_work_handler(struct work_struct *work);
+int         i915_scheduler_flush_request(struct drm_i915_gem_request *req,
+					 bool is_locked);
+bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
+					      bool *completed, bool *busy);
 
 #endif  /* _I915_SCHEDULER_H_ */
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index c00bc50..58c464f 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -11258,7 +11258,8 @@ static void intel_mmio_flip_work_func(struct work_struct *work)
 		WARN_ON(__i915_wait_request(mmio_flip->req,
 					    mmio_flip->crtc->reset_counter,
 					    false, NULL,
-					    &mmio_flip->i915->rps.mmioflips));
+					    &mmio_flip->i915->rps.mmioflips,
+					    false));
 		i915_gem_request_unreference(mmio_flip->req);
 	}
 
@@ -13258,7 +13259,7 @@ static int intel_atomic_prepare_commit(struct drm_device *dev,
 
 			ret = __i915_wait_request(intel_plane_state->wait_req,
 						  reset_counter, true,
-						  NULL, NULL);
+						  NULL, NULL, false);
 
 			/* Swallow -EIO errors to allow updates during hw lockup. */
 			if (ret == -EIO)
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 34fbe68..4d77886 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -2294,7 +2294,7 @@ int intel_ring_idle(struct intel_engine_cs *ring)
 	return __i915_wait_request(req,
 				   atomic_read(&to_i915(ring->dev)->gpu_error.reset_counter),
 				   to_i915(ring->dev)->mm.interruptible,
-				   NULL, NULL);
+				   NULL, NULL, true);
 }
 
 int intel_ring_alloc_request_extras(struct drm_i915_gem_request *request)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 19/39] drm/i915: Added scheduler support to page fault handler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

GPU page faults can now require scheduler operation in order to complete. For
example, in order to free up sufficient memory to handle the fault the handler
must wait for a batch buffer to complete that has not even been sent to the
hardware yet. Thus EAGAIN no longer means a GPU hang, it can occur under normal
operation.

Change-Id: Iff6bd2744ef12bb7405fbcd6b43c286caad4141f
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 3994b47..067f213 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1916,10 +1916,15 @@ out:
 		}
 	case -EAGAIN:
 		/*
-		 * EAGAIN means the gpu is hung and we'll wait for the error
-		 * handler to reset everything when re-faulting in
+		 * EAGAIN can mean the gpu is hung and we'll have to wait for
+		 * the error handler to reset everything when re-faulting in
 		 * i915_mutex_lock_interruptible.
+		 *
+		 * It can also indicate various other nonfatal errors for which
+		 * the best response is to give other threads a chance to run,
+		 * and then retry the failing operation in its entirety.
 		 */
+		/*FALLTHRU*/
 	case 0:
 	case -ERESTARTSYS:
 	case -EINTR:
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 20/39] drm/i915: Added scheduler flush calls to ring throttle and idle functions

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When requesting that all GPU work is completed, it is now necessary to
get the scheduler involved in order to flush out work that queued and
not yet submitted.

v2: Updated to add support for flushing the scheduler queue by time
stamp rather than just doing a blanket flush.

Change-Id: I95dcc2a2ee5c1a844748621c333994ddd6cf6a66
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c       | 24 ++++++++-
 drivers/gpu/drm/i915/i915_scheduler.c | 92 +++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  3 ++
 3 files changed, 118 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 067f213..16bd82c 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3768,6 +3768,10 @@ int i915_gpu_idle(struct drm_device *dev)
 
 	/* Flush everything onto the inactive list. */
 	for_each_ring(ring, dev_priv, i) {
+		ret = i915_scheduler_flush(ring, true);
+		if (ret < 0)
+			return ret;
+
 		if (!i915.enable_execlists) {
 			struct drm_i915_gem_request *req;
 
@@ -4481,7 +4485,8 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	unsigned long recent_enough = jiffies - DRM_I915_THROTTLE_JIFFIES;
 	struct drm_i915_gem_request *request, *target = NULL;
 	unsigned reset_counter;
-	int ret;
+	int i, ret;
+	struct intel_engine_cs *ring;
 
 	ret = i915_gem_wait_for_error(&dev_priv->gpu_error);
 	if (ret)
@@ -4491,6 +4496,23 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	if (ret)
 		return ret;
 
+	for_each_ring(ring, dev_priv, i) {
+		/*
+		 * Flush out scheduler entries that are getting 'stale'. Note
+		 * that the following recent_enough test will only check
+		 * against the time at which the request was submitted to the
+		 * hardware (i.e. when it left the scheduler) not the time it
+		 * was submitted to the driver.
+		 *
+		 * Also, there is not much point worring about busy return
+		 * codes from the scheduler flush call. Even if more work
+		 * cannot be submitted right now for whatever reason, we
+		 * still want to throttle against stale work that has already
+		 * been submitted.
+		 */
+		i915_scheduler_flush_stamp(ring, recent_enough, false);
+	}
+
 	spin_lock(&file_priv->mm.lock);
 	list_for_each_entry(request, &file_priv->mm.request_list, client_list) {
 		if (time_after_eq(request->emitted_jiffies, recent_enough))
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index b8b84a6..232f48a 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -607,6 +607,98 @@ int i915_scheduler_flush_request(struct drm_i915_gem_request *req,
 	return flush_count;
 }
 
+int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
+			       unsigned long target,
+			       bool is_locked)
+{
+	struct i915_scheduler_queue_entry *node;
+	struct drm_i915_private           *dev_priv;
+	struct i915_scheduler             *scheduler;
+	unsigned long       flags;
+	int                 flush_count = 0;
+
+	if (!ring)
+		return -EINVAL;
+
+	dev_priv  = ring->dev->dev_private;
+	scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return 0;
+
+	if (is_locked && (scheduler->flags[ring->id] & i915_sf_submitting)) {
+		/* Scheduler is busy already submitting another batch,
+		 * come back later rather than going recursive... */
+		return -EAGAIN;
+	}
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	i915_scheduler_priority_bump_clear(scheduler);
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+
+		if (node->stamp > target)
+			continue;
+
+		flush_count = i915_scheduler_priority_bump(scheduler,
+					node, scheduler->priority_level_max);
+	}
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (flush_count) {
+		DRM_DEBUG_DRIVER("<%s> Bumped %d entries\n", ring->name, flush_count);
+		flush_count = i915_scheduler_submit_max_priority(ring, is_locked);
+	}
+
+	return flush_count;
+}
+
+int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
+{
+	struct i915_scheduler_queue_entry *node;
+	struct drm_i915_private           *dev_priv;
+	struct i915_scheduler             *scheduler;
+	unsigned long       flags;
+	bool        found;
+	int         ret;
+	uint32_t    count = 0;
+
+	if (!ring)
+		return -EINVAL;
+
+	dev_priv  = ring->dev->dev_private;
+	scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return 0;
+
+	BUG_ON(is_locked && (scheduler->flags[ring->id] & i915_sf_submitting));
+
+	do {
+		found = false;
+		spin_lock_irqsave(&scheduler->lock, flags);
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (!I915_SQS_IS_QUEUED(node))
+				continue;
+
+			found = true;
+			break;
+		}
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (found) {
+			ret = i915_scheduler_submit(ring, is_locked);
+			if (ret < 0)
+				return ret;
+
+			count += ret;
+		}
+	} while (found);
+
+	return count;
+}
+
 static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
 {
 	struct i915_scheduler_queue_entry *node;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 42371b3..29f7248 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -92,6 +92,9 @@ int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *q
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
 void        i915_gem_scheduler_work_handler(struct work_struct *work);
+int         i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked);
+int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
+				       unsigned long stamp, bool is_locked);
 int         i915_scheduler_flush_request(struct drm_i915_gem_request *req,
 					 bool is_locked);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 21/39] drm/i915: Added a module parameter for allowing scheduler overrides

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

It can be useful to be able to disable certain features (e.g. the entire
scheduler) via a module parameter for debugging purposes. A parameter has the
advantage of not being a compile time switch but without implying that it can be
changed dynamically at runtime.

Change-Id: I92f4c832be88f5b34b49b90d6a9903fac68f7004
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h       | 1 +
 drivers/gpu/drm/i915/i915_params.c    | 4 ++++
 drivers/gpu/drm/i915/i915_scheduler.c | 5 ++++-
 drivers/gpu/drm/i915/i915_scheduler.h | 5 +++++
 4 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 5ed600c..5b893a6 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2717,6 +2717,7 @@ struct i915_params {
 	bool verbose_state_checks;
 	bool nuclear_pageflip;
 	int edp_vswing;
+	int scheduler_override;
 };
 extern struct i915_params i915 __read_mostly;
 
diff --git a/drivers/gpu/drm/i915/i915_params.c b/drivers/gpu/drm/i915/i915_params.c
index ba7274a..be2e16c 100644
--- a/drivers/gpu/drm/i915/i915_params.c
+++ b/drivers/gpu/drm/i915/i915_params.c
@@ -54,6 +54,7 @@ struct i915_params i915 __read_mostly = {
 	.edp_vswing = 0,
 	.enable_guc_submission = true,
 	.guc_log_level = -1,
+	.scheduler_override = 1,
 };
 
 module_param_named(modeset, i915.modeset, int, 0400);
@@ -191,3 +192,6 @@ MODULE_PARM_DESC(enable_guc_submission, "Enable GuC submission (default:false)")
 module_param_named(guc_log_level, i915.guc_log_level, int, 0400);
 MODULE_PARM_DESC(guc_log_level,
 	"GuC firmware logging level (-1:disabled (default), 0-3:enabled)");
+
+module_param_named(scheduler_override, i915.scheduler_override, int, 0600);
+MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (0 = none, 1 = direct submission [default])");
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 232f48a..f51676b 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -44,6 +44,9 @@ bool i915_scheduler_is_enabled(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	if (i915.scheduler_override & i915_so_direct_submit)
+		return false;
+
 	return dev_priv->scheduler != NULL;
 }
 
@@ -90,7 +93,7 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	BUG_ON(!scheduler);
 
-	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
+	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
 		intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 29f7248..7e85f80 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -83,6 +83,11 @@ enum {
 	i915_sf_submitting          = (1 << 1),
 };
 
+/* Options for 'scheduler_override' module parameter: */
+enum {
+	i915_so_direct_submit       = (1 << 0),
+};
+
 bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
 int         i915_scheduler_closefile(struct drm_device *dev,
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 22/39] drm/i915: Support for 'unflushed' ring idle

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When the seqno wraps around zero, the entire GPU is forced to be idle
for some reason (possibly only to work around issues with hardware
semaphores but no-one seems too sure!). This causes a problem if the
force idle occurs at an inopportune moment such as in the middle of
submitting a batch buffer. Specifically, it would lead to recursive
submits - submitting work requires a new seqno, the new seqno requires
idling the ring, idling the ring requires submitting work, submitting
work requires a new seqno...

This change adds a 'flush' parameter to the idle function call which
specifies whether the scheduler queues should be flushed out. I.e. is
the call intended to just idle the ring as it is right now (no flush)
or is it intended to force all outstanding work out of the system
(with flush).

In the seqno wrap case, pending work is not an issue because the next
operation will be to submit it. However, in other cases, the intention
is to make sure everything that could be done has been done.

Change-Id: I182e9a5853666c64ecc9e84d8a8b820a7f8e8836
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c         |  4 ++--
 drivers/gpu/drm/i915/intel_lrc.c        |  2 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c | 17 +++++++++++++++--
 drivers/gpu/drm/i915/intel_ringbuffer.h |  2 +-
 4 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 16bd82c..ebb8e0c 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2477,7 +2477,7 @@ i915_gem_init_seqno(struct drm_device *dev, u32 seqno)
 
 	/* Carefully retire all requests without writing to the rings */
 	for_each_ring(ring, dev_priv, i) {
-		ret = intel_ring_idle(ring);
+		ret = intel_ring_idle(ring, false);
 		if (ret)
 			return ret;
 	}
@@ -3788,7 +3788,7 @@ int i915_gpu_idle(struct drm_device *dev)
 			i915_add_request_no_flush(req);
 		}
 
-		ret = intel_ring_idle(ring);
+		ret = intel_ring_idle(ring, true);
 		if (ret)
 			return ret;
 	}
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 16c5302..191a118 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -1014,7 +1014,7 @@ void intel_logical_ring_stop(struct intel_engine_cs *ring)
 	if (!intel_ring_initialized(ring))
 		return;
 
-	ret = intel_ring_idle(ring);
+	ret = intel_ring_idle(ring, true);
 	if (ret && !i915_reset_in_progress(&to_i915(ring->dev)->gpu_error))
 		DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n",
 			  ring->name, ret);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 4d77886..2d8dc54 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -2278,9 +2278,22 @@ static void __wrap_ring_buffer(struct intel_ringbuffer *ringbuf)
 	intel_ring_update_space(ringbuf);
 }
 
-int intel_ring_idle(struct intel_engine_cs *ring)
+int intel_ring_idle(struct intel_engine_cs *ring, bool flush)
 {
 	struct drm_i915_gem_request *req;
+	int ret;
+
+	/*
+	 * NB: Must not flush the scheduler if this idle request is from
+	 * within an execbuff submission (i.e. due to 'get_seqno' calling
+	 * 'wrap_seqno' calling 'idle'). As that would lead to recursive
+	 * flushes!
+	 */
+	if (flush) {
+		ret = i915_scheduler_flush(ring, true);
+		if (ret)
+			return ret;
+	}
 
 	/* Wait upon the last request to be completed */
 	if (list_empty(&ring->request_list))
@@ -3085,7 +3098,7 @@ intel_stop_ring_buffer(struct intel_engine_cs *ring)
 	if (!intel_ring_initialized(ring))
 		return;
 
-	ret = intel_ring_idle(ring);
+	ret = intel_ring_idle(ring, true);
 	if (ret && !i915_reset_in_progress(&to_i915(ring->dev)->gpu_error))
 		DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n",
 			  ring->name, ret);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h b/drivers/gpu/drm/i915/intel_ringbuffer.h
index 48f60cc..d31c94f 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.h
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.h
@@ -488,7 +488,7 @@ void intel_ring_update_space(struct intel_ringbuffer *ringbuf);
 int intel_ring_space(struct intel_ringbuffer *ringbuf);
 bool intel_ring_stopped(struct intel_engine_cs *ring);
 
-int __must_check intel_ring_idle(struct intel_engine_cs *ring);
+int __must_check intel_ring_idle(struct intel_engine_cs *ring, bool flush);
 void intel_ring_init_seqno(struct intel_engine_cs *ring, u32 seqno);
 int intel_ring_flush_all_caches(struct drm_i915_gem_request *req);
 int intel_ring_invalidate_all_caches(struct drm_i915_gem_request *req);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 23/39] drm/i915: Defer seqno allocation until actual hardware submission time

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The seqno value is now only used for the final test for completion of a request.
It is no longer used to track the request through the software stack. Thus it is
no longer necessary to allocate the seqno immediately with the request. Instead,
it can be done lazily and left until the request is actually sent to the
hardware. This is particular advantageous with a GPU scheduler as the requests
can then be re-ordered between their creation and their hardware submission
without having out of order seqnos.

v2: i915_add_request() can't fail!

v3: combine with 'drm/i915: Assign seqno at start of exec_final()'
Various bits of code during the execbuf code path need a seqno value to be
assigned to the request. This change makes this assignment explicit at the start
of submission_final() rather than relying on an auto-generated seqno to have
happened already. This is in preparation for a future patch which changes seqno
values to be assigned lazily (during add_request).

Change-Id: I0d922b84c517611a79fa6c2b9e730d4fe3671d6a
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  1 +
 drivers/gpu/drm/i915/i915_gem.c            | 21 ++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++++++++
 drivers/gpu/drm/i915/intel_lrc.c           | 13 +++++++++++++
 4 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 5b893a6..15dee41 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2225,6 +2225,7 @@ struct drm_i915_gem_request {
 
 	/** GEM sequence number associated with this request. */
 	uint32_t seqno;
+	uint32_t reserved_seqno;
 
 	/* Unique identifier which can be used for trace points & debug */
 	uint32_t uniq;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index ebb8e0c..9c08035 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2529,6 +2529,9 @@ i915_gem_get_seqno(struct drm_device *dev, u32 *seqno)
 
 	/* reserve 0 for non-seqno */
 	if (dev_priv->next_seqno == 0) {
+		/* Why is the full re-initialisation required? Is it only for
+		 * hardware semaphores? If so, could skip it in the case where
+		 * semaphores are disabled? */
 		int ret = i915_gem_init_seqno(dev, 0);
 		if (ret)
 			return ret;
@@ -2586,6 +2589,12 @@ void __i915_add_request(struct drm_i915_gem_request *request,
 		WARN(ret, "*_ring_flush_all_caches failed: %d!\n", ret);
 	}
 
+	/* Make the request's seqno 'live': */
+	if(!request->seqno) {
+		request->seqno = request->reserved_seqno;
+		WARN_ON(request->seqno != dev_priv->last_seqno);
+	}
+
 	/* Record the position of the start of the request so that
 	 * should we detect the updated seqno part-way through the
 	 * GPU processing the request, we never over-estimate the
@@ -2835,6 +2844,9 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 
 	list_for_each_entry_safe(req, req_next, &ring->fence_signal_list, signal_link) {
 		if (!req->cancelled) {
+			/* How can this happen? */
+			WARN_ON(req->seqno == 0);
+
 			if (!i915_seqno_passed(seqno, req->seqno))
 				break;
 		}
@@ -3057,7 +3069,14 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring,
 	if (req == NULL)
 		return -ENOMEM;
 
-	ret = i915_gem_get_seqno(ring->dev, &req->seqno);
+	/*
+	 * Assign an identifier to track this request through the hardware
+	 * but don't make it live yet. It could change in the future if this
+	 * request gets overtaken. However, it still needs to be allocated
+	 * in advance because the point of submission must not fail and seqno
+	 * allocation can fail.
+	 */
+	ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
 	if (ret)
 		goto err;
 
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index fe90571..1c249d0 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1248,6 +1248,19 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Make sure the request's seqno is the latest and greatest: */
+	if(params->request->reserved_seqno != dev_priv->last_seqno) {
+		ret = i915_gem_get_seqno(ring->dev, &params->request->reserved_seqno);
+		if (ret)
+			return ret;
+	}
+	/*
+	 * And make it live because some of the execbuff submission code
+	 * requires the seqno to be available up front. */
+	WARN_ON(params->request->seqno);
+	params->request->seqno = params->request->reserved_seqno;
+	WARN_ON(params->request->seqno != dev_priv->last_seqno);
+
 	ret = intel_ring_reserve_space(params->request);
 	if (ret)
 		return ret;
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 191a118..41e20ba 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -932,6 +932,19 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Make sure the request's seqno is the latest and greatest: */
+	if(params->request->reserved_seqno != dev_priv->last_seqno) {
+		ret = i915_gem_get_seqno(ring->dev, &params->request->reserved_seqno);
+		if (ret)
+			return ret;
+	}
+	/*
+	 * And make it live because some of the execbuff submission code
+	 * requires the seqno to be available up front. */
+	WARN_ON(params->request->seqno);
+	params->request->seqno = params->request->reserved_seqno;
+	WARN_ON(params->request->seqno != dev_priv->last_seqno);
+
 	ret = intel_logical_ring_reserve_space(params->request);
 	if (ret)
 		return ret;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 24/39] drm/i915: Added immediate submission override to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

To aid with debugging issues related to the scheduler, it can be useful to
ensure that all batch buffers are submitted immediately rather than queued until
later. This change adds an override flag via the module parameter to force
instant submission.

Change-Id: I7652df53e2d3c3d77d78bebcf99856e2c53f2801
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 7 +++++--
 drivers/gpu/drm/i915/i915_scheduler.h | 1 +
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index f51676b..d4d2466 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -235,8 +235,11 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
 
-	not_flying = i915_scheduler_count_flying(scheduler, ring) <
-						 scheduler->min_flying;
+	if (i915.scheduler_override & i915_so_submit_on_queue)
+		not_flying = true;
+	else
+		not_flying = i915_scheduler_count_flying(scheduler, ring) <
+							 scheduler->min_flying;
 
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 7e85f80..fc58d49 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -86,6 +86,7 @@ enum {
 /* Options for 'scheduler_override' module parameter: */
 enum {
 	i915_so_direct_submit       = (1 << 0),
+	i915_so_submit_on_queue     = (1 << 1),
 };
 
 bool        i915_scheduler_is_enabled(struct drm_device *dev);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 25/39] drm/i915: Add sync wait support to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

There is a sync framework to allow work for multiple independent
systems to be synchronised with each other but without stalling
the CPU whether in the application or the driver. This patch adds
support for this framework to the GPU scheduler.

Batch buffers can now have sync framework fence objects associated with
them. The scheduler will look at this fence when deciding what to
submit next to the hardware. If the fence is outstanding then that
batch buffer will be passed over in preference of one that is ready to
run. If no other batches are ready then the scheduler will queue an
asynchronous callback to be woken up when the fence has been
signalled. The callback will wake the scheduler and submit the now
ready batch buffer.

v2: Updated to use the new sync_fence_is_signaled() API rather than
having to know about the internals of a fence object.

Also removed the spin lock from the fence wait callback function. This
was used to clear the 'waiting' flag. However, it causes a problem
with the TDR code. Specifically, when cancelling work packets due to a
TDR there is a code path where the fence can be signalled while the
spinlock is already held. It is not really necessary to clear the flag
anyway as it's purpose is solely to prevent multiple waits being
issued. Once the fence has been signalled, no further waits will be
attempted so it doesn't matter whether the fence is marked as having
an outstanding wait or not.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h       |   1 +
 drivers/gpu/drm/i915/i915_scheduler.c | 144 ++++++++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.h |   6 ++
 3 files changed, 146 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 15dee41..4187e75 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1698,6 +1698,7 @@ struct i915_execbuffer_params {
 	uint64_t                        batch_obj_vm_offset;
 	struct intel_engine_cs          *ring;
 	struct drm_i915_gem_object      *batch_obj;
+	struct sync_fence               *fence_wait;
 	struct drm_clip_rect            *cliprects;
 	uint32_t                        instp_mask;
 	int                             instp_mode;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index d4d2466..939dc2b 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -25,6 +25,7 @@
 #include "i915_drv.h"
 #include "intel_drv.h"
 #include "i915_scheduler.h"
+#include <../drivers/android/sync.h>
 
 static int         i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node);
 static int         i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
@@ -96,6 +97,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
+		WARN_ON(qe->params.fence_wait &&
+			(!sync_fence_is_signaled(qe->params.fence_wait)));
+
 		intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
 
 		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
@@ -130,6 +134,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
 			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
 
+		if (qe->params.fence_wait)
+			sync_fence_put(qe->params.fence_wait);
+
 		return 0;
 	}
 
@@ -531,6 +538,9 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		node = list_first_entry(&remove, typeof(*node), link);
 		list_del(&node->link);
 
+		if (node->params.fence_wait)
+			sync_fence_put(node->params.fence_wait);
+
 		/* Free up all the DRM object references */
 		i915_gem_scheduler_clean_node(node);
 
@@ -796,17 +806,94 @@ static int i915_scheduler_submit_max_priority(struct intel_engine_cs *ring,
 	return count;
 }
 
+/* Use a private structure in order to pass the 'dev' pointer through */
+struct i915_sync_fence_waiter {
+	struct sync_fence_waiter sfw;
+	struct drm_device	 *dev;
+	struct i915_scheduler_queue_entry *node;
+};
+
+/*
+ * NB: This callback can be executed at interrupt time. Further, it can be
+ * called from within the TDR reset sequence during a scheduler 'kill_all'
+ * and thus be called while the scheduler spinlock is already held. Thus
+ * it can grab neither the driver mutex nor the scheduler spinlock.
+ */
+static void i915_scheduler_wait_fence_signaled(struct sync_fence *fence,
+				       struct sync_fence_waiter *waiter)
+{
+	struct i915_sync_fence_waiter *i915_waiter;
+	struct drm_i915_private *dev_priv = NULL;
+
+	i915_waiter = container_of(waiter, struct i915_sync_fence_waiter, sfw);
+	dev_priv    = (i915_waiter && i915_waiter->dev) ?
+					    i915_waiter->dev->dev_private : NULL;
+
+	/*
+	 * NB: The callback is executed at interrupt time, thus it can not
+	 * call _submit() directly. It must go via the delayed work handler.
+	 */
+	if (dev_priv)
+		queue_work(dev_priv->wq, &dev_priv->mm.scheduler_work);
+
+	kfree(waiter);
+}
+
+static bool i915_scheduler_async_fence_wait(struct drm_device *dev,
+					    struct i915_scheduler_queue_entry *node)
+{
+	struct i915_sync_fence_waiter	*fence_waiter;
+	struct sync_fence		*fence = node->params.fence_wait;
+	int				signaled;
+	bool				success = true;
+
+	if ((node->flags & i915_qef_fence_waiting) == 0)
+		node->flags |= i915_qef_fence_waiting;
+	else
+		return true;
+
+	if (fence == NULL)
+		return false;
+
+	signaled = sync_fence_is_signaled(fence);
+	if (!signaled) {
+		fence_waiter = kmalloc(sizeof(*fence_waiter), GFP_KERNEL);
+		if (!fence_waiter) {
+			success = false;
+			goto end;
+		}
+
+		sync_fence_waiter_init(&fence_waiter->sfw,
+				i915_scheduler_wait_fence_signaled);
+		fence_waiter->node = node;
+		fence_waiter->dev = dev;
+
+		if (sync_fence_wait_async(fence, &fence_waiter->sfw)) {
+			/* an error occurred, usually this is because the
+			 * fence was signaled already */
+			signaled = sync_fence_is_signaled(fence);
+			if (!signaled) {
+				success = false;
+				goto end;
+			}
+		}
+	}
+end:
+	return success;
+}
+
 static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 				    struct i915_scheduler_queue_entry **pop_node,
 				    unsigned long *flags)
 {
 	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
 	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *best_wait, *fence_wait = NULL;
 	struct i915_scheduler_queue_entry  *best;
 	struct i915_scheduler_queue_entry  *node;
 	int     ret;
 	int     i;
-	bool	any_queued;
+	bool	signalled, any_queued;
 	bool	has_local, has_remote, only_remote;
 
 	*pop_node = NULL;
@@ -814,13 +901,20 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 
 	any_queued = false;
 	only_remote = false;
+	best_wait = NULL;
 	best = NULL;
+	signalled = true;
 
 	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
 		if (!I915_SQS_IS_QUEUED(node))
 			continue;
 		any_queued = true;
 
+		if (node->params.fence_wait)
+			signalled = sync_fence_is_signaled(node->params.fence_wait);
+		else
+			signalled = true;
+
 		has_local  = false;
 		has_remote = false;
 		for (i = 0; i < node->num_deps; i++) {
@@ -837,9 +931,15 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 			only_remote = true;
 
 		if (!has_local && !has_remote) {
-			if (!best ||
-			    (node->priority > best->priority))
-				best = node;
+			if (signalled) {
+				if (!best ||
+				    (node->priority > best->priority))
+					best = node;
+			} else {
+				if (!best_wait ||
+				    (node->priority > best_wait->priority))
+					best_wait = node;
+			}
 		}
 	}
 
@@ -855,8 +955,34 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		 * (a) there are no buffers in the queue
 		 * (b) all queued buffers are dependent on other buffers
 		 *     e.g. on a buffer that is in flight on a different ring
+		 * (c) all independent buffers are waiting on fences
 		 */
-		if (only_remote) {
+		if (best_wait) {
+			/* Need to wait for something to be signalled.
+			 *
+			 * NB: do not really want to wait on one specific fd
+			 * because there is no guarantee in the order that
+			 * blocked buffers will be signalled. Need to wait on
+			 * 'anything' and then rescan for best available, if
+			 * still nothing then wait again...
+			 *
+			 * NB 2: The wait must also wake up if someone attempts
+			 * to submit a new buffer. The new buffer might be
+			 * independent of all others and thus could jump the
+			 * queue and start running immediately.
+			 *
+			 * NB 3: Lastly, must not wait with the spinlock held!
+			 *
+			 * So rather than wait here, need to queue a deferred
+			 * wait thread and just return 'nothing to do'.
+			 *
+			 * NB 4: Can't actually do the wait here because the
+			 * spinlock is still held and the wait requires doing
+			 * a memory allocation.
+			 */
+			fence_wait = best_wait;
+			ret = -EAGAIN;
+		} else if (only_remote) {
 			/* The only dependent buffers are on another ring. */
 			ret = -EAGAIN;
 		} else if (any_queued) {
@@ -868,6 +994,14 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 
 	/* i915_scheduler_dump_queue_pop(ring, best); */
 
+	if (fence_wait) {
+		/* It should be safe to sleep now... */
+		/* NB: Need to release and reacquire the spinlock though */
+		spin_unlock_irqrestore(&scheduler->lock, *flags);
+		i915_scheduler_async_fence_wait(ring->dev, fence_wait);
+		spin_lock_irqsave(&scheduler->lock, *flags);
+	}
+
 	*pop_node = best;
 	return ret;
 }
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index fc58d49..adfa1e0 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -51,6 +51,11 @@ struct i915_scheduler_obj_entry {
 	struct drm_i915_gem_object          *obj;
 };
 
+enum i915_scheduler_queue_entry_flags {
+	/* Fence is being waited on */
+	i915_qef_fence_waiting              = (1 << 0),
+};
+
 struct i915_scheduler_queue_entry {
 	struct i915_execbuffer_params       params;
 	/* -1023 = lowest priority, 0 = default, 1023 = highest */
@@ -63,6 +68,7 @@ struct i915_scheduler_queue_entry {
 	enum i915_scheduler_queue_status    status;
 	unsigned long                       stamp;
 	struct list_head                    link;
+	uint32_t                            flags;
 };
 
 struct i915_scheduler {
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 26/39] drm/i915: Connecting execbuff fences to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler now supports sync framework fences being associated with
batch buffers. The execbuff IOCTL allows such fences to be passed in
from user land. This patch wires the two together so that the IOCTL no
longer needs to stall on the fence immediately. Instead the stall is
now swallowed by the scheduler's scheduling algorithm.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 19 ++++++++++++++++++-
 drivers/gpu/drm/i915/i915_scheduler.c      |  3 +++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 1c249d0..a247d50 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1529,7 +1529,9 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	/*
 	 * Without a GPU scheduler, any fence waits must be done up front.
 	 */
-	if (args->flags & I915_EXEC_WAIT_FENCE) {
+	if ((args->flags & I915_EXEC_WAIT_FENCE) &&
+	    (i915.scheduler_override & i915_so_direct_submit))
+	{
 		ret = i915_early_fence_wait(ring, fd_fence_wait);
 		if (ret < 0)
 			return ret;
@@ -1714,6 +1716,18 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	i915_gem_context_reference(ctx);
 	params->ctx = ctx;
 
+	if (args->flags & I915_EXEC_WAIT_FENCE) {
+		if (fd_fence_wait < 0) {
+			DRM_ERROR("Wait fence for ring %d has invalid id %d\n",
+				  (int) ring->id, fd_fence_wait);
+		} else {
+			params->fence_wait = sync_fence_fdget(fd_fence_wait);
+			if (params->fence_wait == NULL)
+				DRM_ERROR("Invalid wait fence %d\n",
+					  fd_fence_wait);
+		}
+	}
+
 	if (args->flags & I915_EXEC_CREATE_FENCE) {
 		/*
 		 * Caller has requested a sync fence.
@@ -1794,6 +1808,9 @@ err:
 			i915_gem_context_unreference(params->ctx);
 	}
 
+	if (params->fence_wait)
+		sync_fence_put(params->fence_wait);
+
 	/*
 	 * If the request was created but not successfully submitted then it
 	 * must be freed again. If it was submitted then it is being tracked
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 939dc2b..ddd21b2 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -915,6 +915,9 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		else
 			signalled = true;
 
+		if (!signalled)
+			signalled = i915_safe_to_ignore_fence(ring, node->params.fence_wait);
+
 		has_local  = false;
 		has_remote = false;
 		for (i = 0; i < node->num_deps; i++) {
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 27/39] drm/i915: Added trace points to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Added trace points to the scheduler to track all the various events,
node state transitions and other interesting things that occur.

v2: Updated for new request completion tracking implementation.

Change-Id: I9886390cfc7897bc1faf50a104bc651d8baed8a5
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   2 +
 drivers/gpu/drm/i915/i915_scheduler.c      |  24 ++++
 drivers/gpu/drm/i915/i915_trace.h          | 190 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_lrc.c           |   2 +
 4 files changed, 218 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index a247d50..e748fd3 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1226,6 +1226,8 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
+	trace_i915_gem_ring_queue(ring, params);
+
 	qe = container_of(params, typeof(*qe), params);
 	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index ddd21b2..098f69a 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -97,6 +97,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
+		trace_i915_scheduler_queue(qe->params.ring, qe);
+
 		WARN_ON(qe->params.fence_wait &&
 			(!sync_fence_is_signaled(qe->params.fence_wait)));
 
@@ -248,6 +250,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		not_flying = i915_scheduler_count_flying(scheduler, ring) <
 							 scheduler->min_flying;
 
+	trace_i915_scheduler_queue(ring, node);
+	trace_i915_scheduler_node_state_change(ring, node);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (not_flying)
@@ -275,6 +280,9 @@ static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
 
 	node->status = i915_sqs_flying;
 
+	trace_i915_scheduler_fly(ring, node);
+	trace_i915_scheduler_node_state_change(ring, node);
+
 	if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
 		bool    success = true;
 
@@ -340,6 +348,8 @@ static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
 
 	node->status = i915_sqs_queued;
 	node->params.request->seqno = 0;
+	trace_i915_scheduler_unfly(node->params.ring, node);
+	trace_i915_scheduler_node_state_change(node->params.ring, node);
 }
 
 /* Give up on a popped node completely. For example, because it is causing the
@@ -350,6 +360,8 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
 	BUG_ON(!I915_SQS_IS_FLYING(node));
 
 	node->status = i915_sqs_dead;
+	trace_i915_scheduler_unfly(node->params.ring, node);
+	trace_i915_scheduler_node_state_change(node->params.ring, node);
 }
 
 /*
@@ -369,6 +381,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	struct i915_scheduler_queue_entry *node = req->scheduler_qe;
 	unsigned long       flags;
 
+	trace_i915_scheduler_landing(req);
+
 	if (!node)
 		return false;
 
@@ -382,6 +396,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	else
 		node->status = i915_sqs_complete;
 
+	trace_i915_scheduler_node_state_change(req->ring, node);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	return true;
@@ -524,6 +540,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	/* Launch more packets now? */
 	do_submit = (queued > 0) && (flying < scheduler->min_flying);
 
+	trace_i915_scheduler_remove(ring, min_seqno, do_submit);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (!do_submit && list_empty(&remove))
@@ -538,6 +556,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		node = list_first_entry(&remove, typeof(*node), link);
 		list_del(&node->link);
 
+		trace_i915_scheduler_destroy(ring, node);
+
 		if (node->params.fence_wait)
 			sync_fence_put(node->params.fence_wait);
 
@@ -952,6 +972,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		INIT_LIST_HEAD(&best->link);
 		best->status  = i915_sqs_popped;
 
+		trace_i915_scheduler_node_state_change(ring, best);
+
 		ret = 0;
 	} else {
 		/* Can only get here if:
@@ -1005,6 +1027,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		spin_lock_irqsave(&scheduler->lock, *flags);
 	}
 
+	trace_i915_scheduler_pop_from_queue(ring, best);
+
 	*pop_node = best;
 	return ret;
 }
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index 76b08fa..73b0ee9 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -9,6 +9,7 @@
 #include "i915_drv.h"
 #include "intel_drv.h"
 #include "intel_ringbuffer.h"
+#include "i915_scheduler.h"
 
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM i915
@@ -822,6 +823,195 @@ TRACE_EVENT(switch_mm,
 		  __entry->dev, __entry->ring, __entry->to, __entry->vm)
 );
 
+TRACE_EVENT(i915_scheduler_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_fly,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_unfly,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_landing,
+	    TP_PROTO(struct drm_i915_gem_request *req),
+	    TP_ARGS(req),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     __field(u32, status)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = req->ring->id;
+			   __entry->uniq   = req->uniq;
+			   __entry->seqno  = req->seqno;
+			   __entry->status = req->scheduler_qe ? req->scheduler_qe->status : ~0U;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d, status=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno, __entry->status)
+);
+
+TRACE_EVENT(i915_scheduler_remove,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     u32 min_seqno, bool do_submit),
+	    TP_ARGS(ring, min_seqno, do_submit),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, min_seqno)
+			     __field(bool, do_submit)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring      = ring->id;
+			   __entry->min_seqno = min_seqno;
+			   __entry->do_submit = do_submit;
+			   ),
+
+	    TP_printk("ring=%d, min_seqno = %d, do_submit=%d",
+		      __entry->ring, __entry->min_seqno, __entry->do_submit)
+);
+
+TRACE_EVENT(i915_scheduler_destroy,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_pop_from_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_node_state_change,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     __field(u32, status)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = ring->id;
+			   __entry->uniq   = node ? node->params.request->uniq  : 0;
+			   __entry->seqno  = node->params.request->seqno;
+			   __entry->status = node->status;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d, status=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno, __entry->status)
+);
+
+TRACE_EVENT(i915_gem_ring_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_execbuffer_params *params),
+	    TP_ARGS(ring, params),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = ring->id;
+			   __entry->seqno  = params->request->seqno;
+			   ),
+
+	    TP_printk("ring=%d, seqno=%d", __entry->ring, __entry->seqno)
+);
+
 #endif /* _I915_TRACE_H_ */
 
 /* This part must be outside protection */
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 41e20ba..6c01f6c 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -910,6 +910,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
+	trace_i915_gem_ring_queue(ring, params);
+
 	qe = container_of(params, typeof(*qe), params);
 	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 28/39] drm/i915: Added scheduler queue throttling by DRM file handle

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler decouples the submission of batch buffers to the driver from their
subsequent submission to the hardware. This means that an application which is
continuously submitting buffers as fast as it can could potentialy flood the
driver. To prevent this, the driver now tracks how many buffers are in progress
(queued in software or executing in hardware) and limits this to a given
(tunable) number. If this number is exceeded then the queue to the driver will
return EAGAIN and thus prevent the scheduler's queue becoming arbitrarily large.

Change-Id: I83258240aec7c810db08c006a3062d46aa91363f
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  2 ++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  8 +++++++
 drivers/gpu/drm/i915/i915_scheduler.c      | 34 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |  2 ++
 4 files changed, 46 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 4187e75..4ecb6e4 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -342,6 +342,8 @@ struct drm_i915_file_private {
 	} rps;
 
 	struct intel_engine_cs *bsd_ring;
+
+	u32 scheduler_queue_length;
 };
 
 enum intel_dpll_id {
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index e748fd3..8aca374 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1857,6 +1857,10 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_is_full(file))
+		return -EAGAIN;
+
 	/* Copy in the exec list from userland */
 	exec_list = drm_malloc_ab(sizeof(*exec_list), args->buffer_count);
 	exec2_list = drm_malloc_ab(sizeof(*exec2_list), args->buffer_count);
@@ -1940,6 +1944,10 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_is_full(file))
+		return -EAGAIN;
+
 	exec2_list = kmalloc(sizeof(*exec2_list)*args->buffer_count,
 			     GFP_TEMPORARY | __GFP_NOWARN | __GFP_NORETRY);
 	if (exec2_list == NULL)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 098f69a..ff6460e 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -40,6 +40,8 @@ static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *sch
 static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 						struct i915_scheduler_queue_entry *target,
 						uint32_t bump);
+static void        i915_scheduler_file_queue_inc(struct drm_file *file);
+static void        i915_scheduler_file_queue_dec(struct drm_file *file);
 
 bool i915_scheduler_is_enabled(struct drm_device *dev)
 {
@@ -74,6 +76,7 @@ int i915_scheduler_init(struct drm_device *dev)
 	scheduler->priority_level_max     = 1023;
 	scheduler->priority_level_preempt = 900;
 	scheduler->min_flying             = 2;
+	scheduler->file_queue_max         = 64;
 
 	dev_priv->scheduler = scheduler;
 
@@ -244,6 +247,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
 
+	i915_scheduler_file_queue_inc(node->params.file);
+
 	if (i915.scheduler_override & i915_so_submit_on_queue)
 		not_flying = true;
 	else
@@ -523,6 +528,12 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		/* Strip the dependency info while the mutex is still locked */
 		i915_scheduler_remove_dependent(scheduler, node);
 
+		/* Likewise clean up the file descriptor before it might disappear. */
+		if (node->params.file) {
+			i915_scheduler_file_queue_dec(node->params.file);
+			node->params.file = NULL;
+		}
+
 		continue;
 	}
 
@@ -1256,3 +1267,26 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 
 	return 0;
 }
+
+bool i915_scheduler_file_queue_is_full(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+	struct drm_i915_private      *dev_priv  = file_priv->dev_priv;
+	struct i915_scheduler        *scheduler = dev_priv->scheduler;
+
+	return file_priv->scheduler_queue_length >= scheduler->file_queue_max;
+}
+
+static void i915_scheduler_file_queue_inc(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+
+	file_priv->scheduler_queue_length++;
+}
+
+static void i915_scheduler_file_queue_dec(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+
+	file_priv->scheduler_queue_length--;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index adfa1e0..74d7321 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -81,6 +81,7 @@ struct i915_scheduler {
 	int32_t             priority_level_max;
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
+	uint32_t            file_queue_max;
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -111,5 +112,6 @@ int         i915_scheduler_flush_request(struct drm_i915_gem_request *req,
 					 bool is_locked);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
+bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 29/39] drm/i915: Added debugfs interface to scheduler tuning parameters

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

There are various parameters within the scheduler which can be tuned
to improve performance, reduce memory footprint, etc. This change adds
support for altering these via debugfs.

v2: Updated for priorities now being signed values.

Change-Id: I6c26765269ae7173ff4d3a5c20921eaaca7c36ed
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c | 141 ++++++++++++++++++++++++++++++++++++
 1 file changed, 141 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index ecab9b8..8f1c10c 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -39,6 +39,7 @@
 #include "intel_ringbuffer.h"
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_scheduler.h"
 
 enum {
 	ACTIVE_LIST,
@@ -1122,6 +1123,141 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_next_seqno_fops,
 			i915_next_seqno_get, i915_next_seqno_set,
 			"0x%llx\n");
 
+static int
+i915_scheduler_priority_min_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_min;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_min_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->priority_level_min = (int32_t) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_min_fops,
+			i915_scheduler_priority_min_get,
+			i915_scheduler_priority_min_set,
+			"%lld\n");
+
+static int
+i915_scheduler_priority_max_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_max;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_max_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->priority_level_max = (int32_t) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_max_fops,
+			i915_scheduler_priority_max_get,
+			i915_scheduler_priority_max_set,
+			"%lld\n");
+
+static int
+i915_scheduler_priority_preempt_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_preempt;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_preempt_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->priority_level_preempt = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_preempt_fops,
+			i915_scheduler_priority_preempt_get,
+			i915_scheduler_priority_preempt_set,
+			"%lld\n");
+
+static int
+i915_scheduler_min_flying_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->min_flying;
+	return 0;
+}
+
+static int
+i915_scheduler_min_flying_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->min_flying = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_min_flying_fops,
+			i915_scheduler_min_flying_get,
+			i915_scheduler_min_flying_set,
+			"%llu\n");
+
+static int
+i915_scheduler_file_queue_max_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->file_queue_max;
+	return 0;
+}
+
+static int
+i915_scheduler_file_queue_max_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->file_queue_max = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_file_queue_max_fops,
+			i915_scheduler_file_queue_max_get,
+			i915_scheduler_file_queue_max_set,
+			"%llu\n");
+
 static int i915_frequency_info(struct seq_file *m, void *unused)
 {
 	struct drm_info_node *node = m->private;
@@ -5457,6 +5593,11 @@ static const struct i915_debugfs_files {
 	{"i915_gem_drop_caches", &i915_drop_caches_fops},
 	{"i915_error_state", &i915_error_state_fops},
 	{"i915_next_seqno", &i915_next_seqno_fops},
+	{"i915_scheduler_priority_min", &i915_scheduler_priority_min_fops},
+	{"i915_scheduler_priority_max", &i915_scheduler_priority_max_fops},
+	{"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops},
+	{"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops},
+	{"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
 	{"i915_display_crc_ctl", &i915_display_crc_ctl_fops},
 	{"i915_pri_wm_latency", &i915_pri_wm_latency_fops},
 	{"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 30/39] drm/i915: Added debug state dump facilities to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When debugging batch buffer submission issues, it is useful to be able to see
what the current state of the scheduler is. This change adds functions for
decoding the internal scheduler state and reporting it.

Change-Id: I0634168e3f3465ff023f5a673165c90b07e535b6
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 276 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  14 ++
 2 files changed, 290 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index ff6460e..5eec217 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -36,6 +36,9 @@ static int         i915_scheduler_submit_max_priority(struct intel_engine_cs *ri
 						      bool is_locked);
 static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
 					       struct intel_engine_cs *ring);
+static int         i915_scheduler_dump_locked(struct intel_engine_cs *ring,
+					      const char *msg);
+static int         i915_scheduler_dump_all_locked(struct drm_device *dev, const char *msg);
 static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
 static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 						struct i915_scheduler_queue_entry *target,
@@ -53,6 +56,116 @@ bool i915_scheduler_is_enabled(struct drm_device *dev)
 	return dev_priv->scheduler != NULL;
 }
 
+const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node)
+{
+	static char	str[50];
+	char		*ptr = str;
+
+	*(ptr++) = node->bumped ? 'B' : '-',
+	*(ptr++) = i915_gem_request_completed(node->params.request) ? 'C' : '-';
+
+	*ptr = 0;
+
+	return str;
+}
+
+char i915_scheduler_queue_status_chr(enum i915_scheduler_queue_status status)
+{
+	switch (status) {
+	case i915_sqs_none:
+	return 'N';
+
+	case i915_sqs_queued:
+	return 'Q';
+
+	case i915_sqs_popped:
+	return 'X';
+
+	case i915_sqs_flying:
+	return 'F';
+
+	case i915_sqs_complete:
+	return 'C';
+
+	case i915_sqs_dead:
+	return 'D';
+
+	default:
+	break;
+	}
+
+	return '?';
+}
+
+const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status)
+{
+	static char	str[50];
+
+	switch (status) {
+	case i915_sqs_none:
+	return "None";
+
+	case i915_sqs_queued:
+	return "Queued";
+
+	case i915_sqs_popped:
+	return "Popped";
+
+	case i915_sqs_flying:
+	return "Flying";
+
+	case i915_sqs_complete:
+	return "Complete";
+
+	case i915_sqs_dead:
+	return "Dead";
+
+	default:
+	break;
+	}
+
+	sprintf(str, "[Unknown_%d!]", status);
+	return str;
+}
+
+const char *i915_scheduler_flag_str(uint32_t flags)
+{
+	static char     str[100];
+	char           *ptr = str;
+
+	*ptr = 0;
+
+#define TEST_FLAG(flag, msg)						\
+	do {								\
+		if (flags & (flag)) {					\
+			strcpy(ptr, msg);				\
+			ptr += strlen(ptr);				\
+			flags &= ~(flag);				\
+		}							\
+	} while (0)
+
+	TEST_FLAG(i915_sf_interrupts_enabled, "IntOn|");
+	TEST_FLAG(i915_sf_submitting,         "Submitting|");
+	TEST_FLAG(i915_sf_dump_force,         "DumpForce|");
+	TEST_FLAG(i915_sf_dump_details,       "DumpDetails|");
+	TEST_FLAG(i915_sf_dump_dependencies,  "DumpDeps|");
+
+#undef TEST_FLAG
+
+	if (flags) {
+		sprintf(ptr, "Unknown_0x%X!", flags);
+		ptr += strlen(ptr);
+	}
+
+	if (ptr == str)
+		strcpy(str, "-");
+	else
+		ptr[-1] = 0;
+
+	return str;
+};
+
 int i915_scheduler_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -603,6 +716,169 @@ void i915_gem_scheduler_work_handler(struct work_struct *work)
 	}
 }
 
+int i915_scheduler_dump_all(struct drm_device *dev, const char *msg)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	unsigned long   flags;
+	int             ret;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	ret = i915_scheduler_dump_all_locked(dev, msg);
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return ret;
+}
+
+static int i915_scheduler_dump_all_locked(struct drm_device *dev, const char *msg)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring;
+	int                     i, r, ret = 0;
+
+	for_each_ring(ring, dev_priv, i) {
+		scheduler->flags[ring->id] |= i915_sf_dump_force   |
+					      i915_sf_dump_details |
+					      i915_sf_dump_dependencies;
+		r = i915_scheduler_dump_locked(ring, msg);
+		if (ret == 0)
+			ret = r;
+	}
+
+	return ret;
+}
+
+int i915_scheduler_dump(struct intel_engine_cs *ring, const char *msg)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	unsigned long   flags;
+	int             ret;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	ret = i915_scheduler_dump_locked(ring, msg);
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return ret;
+}
+
+static int i915_scheduler_dump_locked(struct intel_engine_cs *ring, const char *msg)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	int                 flying = 0, queued = 0, complete = 0, other = 0;
+	static int          old_flying = -1, old_queued = -1, old_complete = -1;
+	bool                b_dump;
+	char                brkt[2] = { '<', '>' };
+
+	if (!ring)
+		return -EINVAL;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (I915_SQS_IS_QUEUED(node))
+			queued++;
+		else if (I915_SQS_IS_FLYING(node))
+			flying++;
+		else if (I915_SQS_IS_COMPLETE(node))
+			complete++;
+		else
+			other++;
+	}
+
+	b_dump = (flying != old_flying) ||
+		 (queued != old_queued) ||
+		 (complete != old_complete);
+	if (scheduler->flags[ring->id] & i915_sf_dump_force) {
+		if (!b_dump) {
+			b_dump = true;
+			brkt[0] = '{';
+			brkt[1] = '}';
+		}
+
+		scheduler->flags[ring->id] &= ~i915_sf_dump_force;
+	}
+
+	if (b_dump) {
+		old_flying   = flying;
+		old_queued   = queued;
+		old_complete = complete;
+		DRM_DEBUG_DRIVER("<%s> Q:%02d, F:%02d, C:%02d, O:%02d, "
+				 "Flags = %s, Next = %d:%d %c%s%c\n",
+				 ring->name, queued, flying, complete, other,
+				 i915_scheduler_flag_str(scheduler->flags[ring->id]),
+				 dev_priv->request_uniq, dev_priv->next_seqno,
+				 brkt[0], msg, brkt[1]);
+	} else {
+		/*DRM_DEBUG_DRIVER("<%s> Q:%02d, F:%02d, C:%02d, O:%02d"
+				 ", Flags = %s, Next = %d:%d [%s]\n",
+				 ring->name,
+				 queued, flying, complete, other,
+				 i915_scheduler_flag_str(scheduler->flags[ring->id]),
+				 dev_priv->request_uniq, dev_priv->next_seqno, msg); */
+
+		return 0;
+	}
+
+	if (scheduler->flags[ring->id] & i915_sf_dump_details) {
+		int         i, deps;
+		uint32_t    count, counts[i915_sqs_MAX];
+
+		memset(counts, 0x00, sizeof(counts));
+
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (node->status < i915_sqs_MAX) {
+				count = counts[node->status]++;
+			} else {
+				DRM_DEBUG_DRIVER("<%s>   Unknown status: %d!\n",
+						 ring->name, node->status);
+				count = -1;
+			}
+
+			deps = 0;
+			for (i = 0; i < node->num_deps; i++)
+				if (i915_scheduler_is_dependency_valid(node, i))
+					deps++;
+
+			DRM_DEBUG_DRIVER("<%s>   %c:%02d> uniq = %d, seqno"
+					 " = %d/%s, deps = %d / %d, fence = %p/%d, %s [pri = "
+					 "%4d]\n", ring->name,
+					 i915_scheduler_queue_status_chr(node->status),
+					 count,
+					 node->params.request->uniq,
+					 node->params.request->seqno,
+					 node->params.ring->name,
+					 deps, node->num_deps,
+					 node->params.fence_wait,
+					 node->params.fence_wait ? sync_fence_is_signaled(node->params.fence_wait) : 0,
+					 i915_qe_state_str(node),
+					 node->priority);
+
+			if ((scheduler->flags[ring->id] & i915_sf_dump_dependencies)
+				== 0)
+				continue;
+
+			for (i = 0; i < node->num_deps; i++)
+				if (node->dep_list[i])
+					DRM_DEBUG_DRIVER("<%s>       |-%c:"
+						"%02d%c uniq = %d, seqno = %d/%s, %s [pri = %4d]\n",
+						ring->name,
+						i915_scheduler_queue_status_chr(node->dep_list[i]->status),
+						i,
+						i915_scheduler_is_dependency_valid(node, i)
+							? '>' : '#',
+						node->dep_list[i]->params.request->uniq,
+						node->dep_list[i]->params.request->seqno,
+						node->dep_list[i]->params.ring->name,
+						i915_qe_state_str(node->dep_list[i]),
+						node->dep_list[i]->priority);
+		}
+	}
+
+	return 0;
+}
+
 int i915_scheduler_flush_request(struct drm_i915_gem_request *req,
 				 bool is_locked)
 {
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 74d7321..b5518b5 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -41,6 +41,9 @@ enum i915_scheduler_queue_status {
 	/* Limit value for use with arrays/loops */
 	i915_sqs_MAX
 };
+char i915_scheduler_queue_status_chr(enum i915_scheduler_queue_status status);
+const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status);
 
 #define I915_SQS_IS_QUEUED(node)	(((node)->status == i915_sqs_queued))
 #define I915_SQS_IS_FLYING(node)	(((node)->status == i915_sqs_flying))
@@ -70,6 +73,7 @@ struct i915_scheduler_queue_entry {
 	struct list_head                    link;
 	uint32_t                            flags;
 };
+const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node);
 
 struct i915_scheduler {
 	struct list_head    node_queue[I915_NUM_RINGS];
@@ -86,9 +90,16 @@ struct i915_scheduler {
 
 /* Flag bits for i915_scheduler::flags */
 enum {
+	/* Internal state */
 	i915_sf_interrupts_enabled  = (1 << 0),
 	i915_sf_submitting          = (1 << 1),
+
+	/* Dump/debug flags */
+	i915_sf_dump_force          = (1 << 8),
+	i915_sf_dump_details        = (1 << 9),
+	i915_sf_dump_dependencies   = (1 << 10),
 };
+const char *i915_scheduler_flag_str(uint32_t flags);
 
 /* Options for 'scheduler_override' module parameter: */
 enum {
@@ -110,6 +121,9 @@ int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 				       unsigned long stamp, bool is_locked);
 int         i915_scheduler_flush_request(struct drm_i915_gem_request *req,
 					 bool is_locked);
+int         i915_scheduler_dump(struct intel_engine_cs *ring,
+				const char *msg);
+int         i915_scheduler_dump_all(struct drm_device *dev, const char *msg);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
 bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 31/39] drm/i915: Add early exit to execbuff_final() if insufficient ring space

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

One of the major purposes of the GPU scheduler is to avoid stalling the CPU when
the GPU is busy and unable to accept more work. This change adds support to the
ring submission code to allow a ring space check to be performed before
attempting to submit a batch buffer to the hardware. If insufficient space is
available then the scheduler can go away and come back later, letting the CPU
get on with other work, rather than stalling and waiting for the hardware to
catch up.

Change-Id: I267159ce1150cb6714d34a49b841bcbe4bf66326
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 42 ++++++++++++++++------
 drivers/gpu/drm/i915/intel_lrc.c           | 57 +++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/intel_ringbuffer.c    | 24 +++++++++++++
 drivers/gpu/drm/i915/intel_ringbuffer.h    |  1 +
 4 files changed, 109 insertions(+), 15 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 8aca374..ca57147 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1101,25 +1101,19 @@ i915_reset_gen7_sol_offsets(struct drm_device *dev,
 {
 	struct intel_engine_cs *ring = req->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	int ret, i;
+	int i;
 
 	if (!IS_GEN7(dev) || ring != &dev_priv->ring[RCS]) {
 		DRM_DEBUG("sol reset is gen7/rcs only\n");
 		return -EINVAL;
 	}
 
-	ret = intel_ring_begin(req, 4 * 3);
-	if (ret)
-		return ret;
-
 	for (i = 0; i < 4; i++) {
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, GEN7_SO_WRITE_OFFSET(i));
 		intel_ring_emit(ring, 0);
 	}
 
-	intel_ring_advance(ring);
-
 	return 0;
 }
 
@@ -1246,6 +1240,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	struct intel_engine_cs  *ring = params->ring;
 	u64 exec_start, exec_len;
 	int ret;
+	uint32_t min_space;
 
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
@@ -1267,8 +1262,36 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	if (ret)
 		return ret;
 
+	/*
+	 * It would be a bad idea to run out of space while writing commands
+	 * to the ring. One of the major aims of the scheduler is to not stall
+	 * at any point for any reason. However, doing an early exit half way
+	 * through submission could result in a partial sequence being written
+	 * which would leave the engine in an unknown state. Therefore, check in
+	 * advance that there will be enough space for the entire submission
+	 * whether emitted by the code below OR by any other functions that may
+	 * be executed before the end of final().
+	 *
+	 * NB: This test deliberately overestimates, because that's easier than
+	 * tracing every potential path that could be taken!
+	 *
+	 * Current measurements suggest that we may need to emit up to 744 bytes
+	 * (186 dwords), so this is rounded up to 256 dwords here. Then we double
+	 * that to get the free space requirement, because the block isn't allowed
+	 * to span the transition from the end to the beginning of the ring.
+	 */
+#define I915_BATCH_EXEC_MAX_LEN         256	/* max dwords emitted here	*/
+	min_space = I915_BATCH_EXEC_MAX_LEN * 2 * sizeof(uint32_t);
+	ret = intel_ring_test_space(params->request->ringbuf, min_space);
+	if (ret)
+		goto early_error;
+
 	intel_runtime_pm_get(dev_priv);
 
+	ret = intel_ring_begin(params->request, I915_BATCH_EXEC_MAX_LEN);
+	if (ret)
+		goto error;
+
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
@@ -1287,10 +1310,6 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_ring_begin(params->request, 4);
-		if (ret)
-			goto error;
-
 		intel_ring_emit(ring, MI_NOOP);
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, INSTPM);
@@ -1327,6 +1346,7 @@ error:
 	 */
 	intel_runtime_pm_put(dev_priv);
 
+early_error:
 	if (ret)
 		intel_ring_reserved_space_cancel(params->request->ringbuf);
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 6c01f6c..b5abea4 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -226,6 +226,30 @@ static void lrc_setup_hardware_status_page(struct intel_engine_cs *ring,
 		struct drm_i915_gem_object *default_ctx_obj);
 
 
+/* Test to see if the ring has sufficient space to submit a given piece of work
+ * without causing a stall */
+static int logical_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space)
+{
+	//struct intel_engine_cs *ring = ringbuf->ring;
+	//struct drm_device *dev = ring->dev;
+	//struct drm_i915_private *dev_priv = dev->dev_private;
+
+	if (ringbuf->space < min_space) {
+		/* Need to update the actual ring space. Otherwise, the system
+		 * hangs forever testing a software copy of the space value that
+		 * never changes!
+		 */
+		//ringbuf->head  = I915_READ_HEAD(ring);
+		//ringbuf->space = intel_ring_space(ringbuf);
+		intel_ring_update_space(ringbuf);
+
+		if (ringbuf->space < min_space)
+			return -EAGAIN;
+	}
+
+	return 0;
+}
+
 /**
  * intel_sanitize_enable_execlists() - sanitize i915.enable_execlists
  * @dev: DRM device.
@@ -930,6 +954,7 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	struct intel_engine_cs  *ring = params->ring;
 	u64 exec_start;
 	int ret;
+	uint32_t min_space;
 
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
@@ -952,6 +977,34 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 		return ret;
 
 	/*
+	 * It would be a bad idea to run out of space while writing commands
+	 * to the ring. One of the major aims of the scheduler is to not stall
+	 * at any point for any reason. However, doing an early exit half way
+	 * through submission could result in a partial sequence being written
+	 * which would leave the engine in an unknown state. Therefore, check in
+	 * advance that there will be enough space for the entire submission
+	 * whether emitted by the code below OR by any other functions that may
+	 * be executed before the end of final().
+	 *
+	 * NB: This test deliberately overestimates, because that's easier than
+	 * tracing every potential path that could be taken!
+	 *
+	 * Current measurements suggest that we may need to emit up to ??? bytes
+	 * (186 dwords), so this is rounded up to 256 dwords here. Then we double
+	 * that to get the free space requirement, because the block isn't allowed
+	 * to span the transition from the end to the beginning of the ring.
+	 */
+#define I915_BATCH_EXEC_MAX_LEN         256	/* max dwords emitted here	*/
+	min_space = I915_BATCH_EXEC_MAX_LEN * 2 * sizeof(uint32_t);
+	ret = logical_ring_test_space(params->request->ringbuf, min_space);
+	if (ret)
+		goto err;
+
+	ret = intel_logical_ring_begin(params->request, I915_BATCH_EXEC_MAX_LEN);
+	if (ret)
+		goto err;
+
+	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
@@ -963,10 +1016,6 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	    params->instp_mode != dev_priv->relative_constants_mode) {
 		struct intel_ringbuffer *ringbuf = params->request->ringbuf;
 
-		ret = intel_logical_ring_begin(params->request, 4);
-		if (ret)
-			return ret;
-
 		intel_logical_ring_emit(ringbuf, MI_NOOP);
 		intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1));
 		intel_logical_ring_emit(ringbuf, INSTPM);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 2d8dc54..97b9dfa 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -2472,6 +2472,30 @@ int intel_ring_cacheline_align(struct drm_i915_gem_request *req)
 	return 0;
 }
 
+/* Test to see if the ring has sufficient space to submit a given piece of work
+ * without causing a stall */
+int intel_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space)
+{
+	struct drm_i915_private *dev_priv = ringbuf->ring->dev->dev_private;
+
+	/* There is a separate LRC version of this code. */
+	BUG_ON(i915.enable_execlists);
+
+	if (ringbuf->space < min_space) {
+		/* Need to update the actual ring space. Otherwise, the system
+		 * hangs forever testing a software copy of the space value that
+		 * never changes!
+		 */
+		ringbuf->head  = I915_READ_HEAD(ringbuf->ring);
+		ringbuf->space = intel_ring_space(ringbuf);
+
+		if (ringbuf->space < min_space)
+			return -EAGAIN;
+	}
+
+	return 0;
+}
+
 void intel_ring_init_seqno(struct intel_engine_cs *ring, u32 seqno)
 {
 	struct drm_device *dev = ring->dev;
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h b/drivers/gpu/drm/i915/intel_ringbuffer.h
index d31c94f..c472f8a 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.h
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.h
@@ -469,6 +469,7 @@ void intel_cleanup_ring_buffer(struct intel_engine_cs *ring);
 
 int intel_ring_alloc_request_extras(struct drm_i915_gem_request *request);
 
+int intel_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space);
 int __must_check intel_ring_begin(struct drm_i915_gem_request *req, int n);
 int __must_check intel_ring_cacheline_align(struct drm_i915_gem_request *req);
 static inline void intel_ring_emit(struct intel_engine_cs *ring,
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 32/39] drm/i915: Added scheduler statistic reporting to debugfs

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

It is useful for know what the scheduler is doing for both debugging
and performance analysis purposes. This change adds a bunch of
counters and such that keep track of various scheduler operations
(batches submitted, completed, flush requests, etc.). The data can
then be read in userland via the debugfs mechanism.

v2: Updated to match changes to scheduler implementation.

Change-Id: I3266c631cd70c9eeb2c235f88f493e60462f85d7
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c        | 77 ++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 11 +++-
 drivers/gpu/drm/i915/i915_scheduler.c      | 87 +++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_scheduler.h      | 36 +++++++++++++
 4 files changed, 203 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 8f1c10c..9e7d67d 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -3603,6 +3603,82 @@ static int i915_drrs_status(struct seq_file *m, void *unused)
 	return 0;
 }
 
+static int i915_scheduler_info(struct seq_file *m, void *unused)
+{
+	struct drm_info_node *node = (struct drm_info_node *) m->private;
+	struct drm_device *dev = node->minor->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_stats *stats = scheduler->stats;
+	struct i915_scheduler_stats_nodes node_stats[I915_NUM_RINGS];
+	struct intel_engine_cs *ring;
+	char   str[50 * (I915_NUM_RINGS + 1)], name[50], *ptr;
+	int ret, i, r;
+
+	ret = mutex_lock_interruptible(&dev->mode_config.mutex);
+	if (ret)
+		return ret;
+
+#define PRINT_VAR(name, fmt, var)					\
+	do {								\
+		sprintf(str, "%-22s", name);				\
+		ptr = str + strlen(str);				\
+		for_each_ring(ring, dev_priv, r) {			\
+			sprintf(ptr, " %10" fmt, var);			\
+			ptr += strlen(ptr);				\
+		}							\
+		seq_printf(m, "%s\n", str);				\
+	} while (0)
+
+	PRINT_VAR("Ring name:",             "s", dev_priv->ring[r].name);
+	PRINT_VAR("  Ring seqno",           "d", ring->get_seqno(ring, false));
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Batch submissions:\n");
+	PRINT_VAR("  Queued",               "u", stats[r].queued);
+	PRINT_VAR("  Submitted",            "u", stats[r].submitted);
+	PRINT_VAR("  Completed",            "u", stats[r].completed);
+	PRINT_VAR("  Expired",              "u", stats[r].expired);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Flush counts:\n");
+	PRINT_VAR("  By object",            "u", stats[r].flush_obj);
+	PRINT_VAR("  By request",           "u", stats[r].flush_req);
+	PRINT_VAR("  By stamp",             "u", stats[r].flush_stamp);
+	PRINT_VAR("  Blanket",              "u", stats[r].flush_all);
+	PRINT_VAR("  Entries bumped",       "u", stats[r].flush_bump);
+	PRINT_VAR("  Entries submitted",    "u", stats[r].flush_submit);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Miscellaneous:\n");
+	PRINT_VAR("  ExecEarly retry",      "u", stats[r].exec_early);
+	PRINT_VAR("  ExecFinal requeue",    "u", stats[r].exec_again);
+	PRINT_VAR("  ExecFinal killed",     "u", stats[r].exec_dead);
+	PRINT_VAR("  Fence wait",           "u", stats[r].fence_wait);
+	PRINT_VAR("  Fence wait again",     "u", stats[r].fence_again);
+	PRINT_VAR("  Fence wait ignore",    "u", stats[r].fence_ignore);
+	PRINT_VAR("  Fence supplied",       "u", stats[r].fence_got);
+	PRINT_VAR("  Hung flying",          "u", stats[r].kill_flying);
+	PRINT_VAR("  Hung queued",          "u", stats[r].kill_queued);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Queue contents:\n");
+	for_each_ring(ring, dev_priv, i)
+		i915_scheduler_query_stats(ring, node_stats + ring->id);
+
+	for (i = 0; i < (i915_sqs_MAX + 1); i++) {
+		sprintf(name, "  %s", i915_scheduler_queue_status_str(i));
+		PRINT_VAR(name, "d", node_stats[r].counts[i]);
+	}
+	seq_putc(m, '\n');
+
+#undef PRINT_VAR
+
+	mutex_unlock(&dev->mode_config.mutex);
+
+	return 0;
+}
+
 struct pipe_crc_info {
 	const char *name;
 	struct drm_device *dev;
@@ -5571,6 +5647,7 @@ static const struct drm_info_list i915_debugfs_list[] = {
 	{"i915_semaphore_status", i915_semaphore_status, 0},
 	{"i915_shared_dplls_info", i915_shared_dplls_info, 0},
 	{"i915_dp_mst_info", i915_dp_mst_info, 0},
+	{"i915_scheduler_info", i915_scheduler_info, 0},
 	{"i915_wa_registers", i915_wa_registers, 0},
 	{"i915_ddb_info", i915_ddb_info, 0},
 	{"i915_sseu_status", i915_sseu_status, 0},
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index ca57147..dfd9c29 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1422,13 +1422,20 @@ static int i915_early_fence_wait(struct intel_engine_cs *ring, int fence_fd)
 	}
 
 	if (!sync_fence_is_signaled(fence)) {
+		struct drm_i915_private *dev_priv = ring->dev->dev_private;
+		struct i915_scheduler *scheduler = dev_priv->scheduler;
+
 		/*
 		 * Wait forever for the fence to be signalled. This is safe
 		 * because the the mutex lock has not yet been acquired and
 		 * the wait is interruptible.
 		 */
-		if (!i915_safe_to_ignore_fence(ring, fence))
+		if (i915_safe_to_ignore_fence(ring, fence))
+			scheduler->stats[ring->id].fence_ignore++;
+		else {
+			scheduler->stats[ring->id].fence_wait++;
 			ret = sync_fence_wait(fence, -1);
+		}
 	}
 
 	sync_fence_put(fence);
@@ -1844,6 +1851,8 @@ err:
 	mutex_unlock(&dev->struct_mutex);
 
 pre_mutex_err:
+	dev_priv->scheduler->stats[ring->id].exec_early++;
+
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 5eec217..2acc798 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -121,6 +121,9 @@ const char *i915_scheduler_queue_status_str(
 	case i915_sqs_dead:
 	return "Dead";
 
+	case i915_sqs_MAX:
+	return "Invalid";
+
 	default:
 	break;
 	}
@@ -210,9 +213,14 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	BUG_ON(!scheduler);
 
+	if (qe->params.fence_wait)
+		scheduler->stats[ring->id].fence_got++;
+
 	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
+		scheduler->stats[qe->params.ring->id].queued++;
+
 		trace_i915_scheduler_queue(qe->params.ring, qe);
 
 		WARN_ON(qe->params.fence_wait &&
@@ -222,6 +230,7 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
 		ret = dev_priv->gt.execbuf_final(&qe->params);
+		scheduler->stats[qe->params.ring->id].submitted++;
 		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
 
 		/*
@@ -255,6 +264,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		if (qe->params.fence_wait)
 			sync_fence_put(qe->params.fence_wait);
 
+		scheduler->stats[qe->params.ring->id].expired++;
+
 		return 0;
 	}
 
@@ -368,6 +379,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		not_flying = i915_scheduler_count_flying(scheduler, ring) <
 							 scheduler->min_flying;
 
+	scheduler->stats[ring->id].queued++;
+
 	trace_i915_scheduler_queue(ring, node);
 	trace_i915_scheduler_node_state_change(ring, node);
 
@@ -472,7 +485,8 @@ static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
 
 /* Give up on a popped node completely. For example, because it is causing the
  * ring to hang or is using some resource that no longer exists. */
-static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
+static void i915_scheduler_node_kill(struct i915_scheduler *scheduler,
+				     struct i915_scheduler_queue_entry *node)
 {
 	BUG_ON(!node);
 	BUG_ON(!I915_SQS_IS_FLYING(node));
@@ -480,6 +494,8 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
 	node->status = i915_sqs_dead;
 	trace_i915_scheduler_unfly(node->params.ring, node);
 	trace_i915_scheduler_node_state_change(node->params.ring, node);
+
+	scheduler->stats[node->params.ring->id].kill_flying++;
 }
 
 /*
@@ -509,10 +525,13 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	WARN_ON(!I915_SQS_IS_FLYING(node));
 
 	/* Node was in flight so mark it as complete. */
-	if (req->cancelled)
+	if (req->cancelled) {
 		node->status = i915_sqs_dead;
-	else
+		scheduler->stats[req->ring->id].kill_flying++;
+	} else {
 		node->status = i915_sqs_complete;
+		scheduler->stats[req->ring->id].completed++;
+	}
 
 	trace_i915_scheduler_node_state_change(req->ring, node);
 
@@ -637,6 +656,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 
 		list_del(&node->link);
 		list_add(&node->link, &remove);
+		scheduler->stats[ring->id].expired++;
 
 		/* Strip the dependency info while the mutex is still locked */
 		i915_scheduler_remove_dependent(scheduler, node);
@@ -879,6 +899,35 @@ static int i915_scheduler_dump_locked(struct intel_engine_cs *ring, const char *
 	return 0;
 }
 
+int i915_scheduler_query_stats(struct intel_engine_cs *ring,
+			       struct i915_scheduler_stats_nodes *stats)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	unsigned long   flags;
+
+	memset(stats, 0x00, sizeof(*stats));
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (node->status >= i915_sqs_MAX) {
+			DRM_DEBUG_DRIVER("Invalid node state: %d! [uniq = %d, seqno = %d]\n",
+					 node->status, node->params.request->uniq, node->params.request->seqno);
+
+			stats->counts[i915_sqs_MAX]++;
+			continue;
+		}
+
+		stats->counts[node->status]++;
+	}
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return 0;
+}
+
 int i915_scheduler_flush_request(struct drm_i915_gem_request *req,
 				 bool is_locked)
 {
@@ -915,16 +964,21 @@ int i915_scheduler_flush_request(struct drm_i915_gem_request *req,
 
 	spin_lock_irqsave(&scheduler->lock, flags);
 
+	scheduler->stats[ring_id].flush_req++;
+
 	i915_scheduler_priority_bump_clear(scheduler);
 
 	flush_count = i915_scheduler_priority_bump(scheduler,
 			    req->scheduler_qe, scheduler->priority_level_max);
+	scheduler->stats[ring_id].flush_bump += flush_count;
 
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (flush_count) {
 		DRM_DEBUG_DRIVER("<%s> Bumped %d entries\n", req->ring->name, flush_count);
 		flush_count = i915_scheduler_submit_max_priority(req->ring, is_locked);
+		if (flush_count > 0)
+			scheduler->stats[ring_id].flush_submit += flush_count;
 	}
 
 	return flush_count;
@@ -956,6 +1010,7 @@ int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 	}
 
 	spin_lock_irqsave(&scheduler->lock, flags);
+	scheduler->stats[ring->id].flush_stamp++;
 	i915_scheduler_priority_bump_clear(scheduler);
 	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
 		if (!I915_SQS_IS_QUEUED(node))
@@ -966,12 +1021,15 @@ int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 
 		flush_count = i915_scheduler_priority_bump(scheduler,
 					node, scheduler->priority_level_max);
+		scheduler->stats[ring->id].flush_bump += flush_count;
 	}
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (flush_count) {
 		DRM_DEBUG_DRIVER("<%s> Bumped %d entries\n", ring->name, flush_count);
 		flush_count = i915_scheduler_submit_max_priority(ring, is_locked);
+		if (flush_count > 0)
+			scheduler->stats[ring->id].flush_submit += flush_count;
 	}
 
 	return flush_count;
@@ -998,6 +1056,8 @@ int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
 
 	BUG_ON(is_locked && (scheduler->flags[ring->id] & i915_sf_submitting));
 
+	scheduler->stats[ring->id].flush_all++;
+
 	do {
 		found = false;
 		spin_lock_irqsave(&scheduler->lock, flags);
@@ -1012,6 +1072,7 @@ int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
 
 		if (found) {
 			ret = i915_scheduler_submit(ring, is_locked);
+			scheduler->stats[ring->id].flush_submit++;
 			if (ret < 0)
 				return ret;
 
@@ -1149,15 +1210,20 @@ static void i915_scheduler_wait_fence_signaled(struct sync_fence *fence,
 static bool i915_scheduler_async_fence_wait(struct drm_device *dev,
 					    struct i915_scheduler_queue_entry *node)
 {
+	struct drm_i915_private         *dev_priv = node->params.ring->dev->dev_private;
+	struct i915_scheduler           *scheduler = dev_priv->scheduler;
 	struct i915_sync_fence_waiter	*fence_waiter;
 	struct sync_fence		*fence = node->params.fence_wait;
 	int				signaled;
 	bool				success = true;
 
-	if ((node->flags & i915_qef_fence_waiting) == 0)
+	if ((node->flags & i915_qef_fence_waiting) == 0) {
 		node->flags |= i915_qef_fence_waiting;
-	else
+		scheduler->stats[node->params.ring->id].fence_wait++;
+	} else {
+		scheduler->stats[node->params.ring->id].fence_again++;
 		return true;
+	}
 
 	if (fence == NULL)
 		return false;
@@ -1222,8 +1288,10 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		else
 			signalled = true;
 
-		if (!signalled)
+		if (!signalled) {
 			signalled = i915_safe_to_ignore_fence(ring, node->params.fence_wait);
+			scheduler->stats[node->params.ring->id].fence_ignore++;
+		}
 
 		has_local  = false;
 		has_remote = false;
@@ -1360,6 +1428,8 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 		 * list. So add it back in and mark it as in flight. */
 		i915_scheduler_fly_node(node);
 
+		scheduler->stats[ring->id].submitted++;
+
 		scheduler->flags[ring->id] |= i915_sf_submitting;
 		spin_unlock_irqrestore(&scheduler->lock, flags);
 		ret = dev_priv->gt.execbuf_final(&node->params);
@@ -1378,6 +1448,7 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 			case ENOENT:
 				/* Fatal errors. Kill the node. */
 				requeue = -1;
+				scheduler->stats[ring->id].exec_dead++;
 			break;
 
 			case EAGAIN:
@@ -1387,12 +1458,14 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 			case ERESTARTSYS:
 			case EINTR:
 				/* Supposedly recoverable errors. */
+				scheduler->stats[ring->id].exec_again++;
 			break;
 
 			default:
 				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
 						 ring->name, ret);
 				/* Assume it is recoverable and hope for the best. */
+				scheduler->stats[ring->id].exec_again++;
 			break;
 			}
 
@@ -1408,7 +1481,7 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 				 * later. */
 				break;
 			} else if (requeue == -1)
-				i915_scheduler_node_kill(node);
+				i915_scheduler_node_kill(scheduler, node);
 		}
 
 		/* Keep launching until the sky is sufficiently full. */
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index b5518b5..ea70a3f 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -75,6 +75,37 @@ struct i915_scheduler_queue_entry {
 };
 const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node);
 
+struct i915_scheduler_stats_nodes {
+	uint32_t	counts[i915_sqs_MAX + 1];
+};
+
+struct i915_scheduler_stats {
+	/* Batch buffer counts: */
+	uint32_t            queued;
+	uint32_t            submitted;
+	uint32_t            completed;
+	uint32_t            expired;
+
+	/* Other stuff: */
+	uint32_t            flush_obj;
+	uint32_t            flush_req;
+	uint32_t            flush_stamp;
+	uint32_t            flush_all;
+	uint32_t            flush_bump;
+	uint32_t            flush_submit;
+
+	uint32_t            exec_early;
+	uint32_t            exec_again;
+	uint32_t            exec_dead;
+	uint32_t            kill_flying;
+	uint32_t            kill_queued;
+
+	uint32_t            fence_wait;
+	uint32_t            fence_again;
+	uint32_t            fence_ignore;
+	uint32_t            fence_got;
+};
+
 struct i915_scheduler {
 	struct list_head    node_queue[I915_NUM_RINGS];
 	uint32_t            flags[I915_NUM_RINGS];
@@ -86,6 +117,9 @@ struct i915_scheduler {
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
 	uint32_t            file_queue_max;
+
+	/* Statistics: */
+	struct i915_scheduler_stats     stats[I915_NUM_RINGS];
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -126,6 +160,8 @@ int         i915_scheduler_dump(struct intel_engine_cs *ring,
 int         i915_scheduler_dump_all(struct drm_device *dev, const char *msg);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
+int         i915_scheduler_query_stats(struct intel_engine_cs *ring,
+				       struct i915_scheduler_stats_nodes *stats);
 bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 33/39] drm/i915: Added seqno values to scheduler status dump

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

It is useful to be able to see what seqnos have actually popped out of the
hardware when viewing the scheduler status.

Change-Id: Ie93e51c64328be2606b8b43440f6344d5f225426
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 10 ++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 2 files changed, 11 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 2acc798..ebb9ab3 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -153,6 +153,7 @@ const char *i915_scheduler_flag_str(uint32_t flags)
 	TEST_FLAG(i915_sf_dump_force,         "DumpForce|");
 	TEST_FLAG(i915_sf_dump_details,       "DumpDetails|");
 	TEST_FLAG(i915_sf_dump_dependencies,  "DumpDeps|");
+	TEST_FLAG(i915_sf_dump_seqno,         "DumpSeqno|");
 
 #undef TEST_FLAG
 
@@ -760,6 +761,7 @@ static int i915_scheduler_dump_all_locked(struct drm_device *dev, const char *ms
 	for_each_ring(ring, dev_priv, i) {
 		scheduler->flags[ring->id] |= i915_sf_dump_force   |
 					      i915_sf_dump_details |
+					      i915_sf_dump_seqno   |
 					      i915_sf_dump_dependencies;
 		r = i915_scheduler_dump_locked(ring, msg);
 		if (ret == 0)
@@ -841,6 +843,14 @@ static int i915_scheduler_dump_locked(struct intel_engine_cs *ring, const char *
 		return 0;
 	}
 
+	if (scheduler->flags[ring->id] & i915_sf_dump_seqno) {
+		uint32_t    seqno;
+
+		seqno    = ring->get_seqno(ring, true);
+
+		DRM_DEBUG_DRIVER("<%s> Seqno = %d\n", ring->name, seqno);
+	}
+
 	if (scheduler->flags[ring->id] & i915_sf_dump_details) {
 		int         i, deps;
 		uint32_t    count, counts[i915_sqs_MAX];
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index ea70a3f..46d2503 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -132,6 +132,7 @@ enum {
 	i915_sf_dump_force          = (1 << 8),
 	i915_sf_dump_details        = (1 << 9),
 	i915_sf_dump_dependencies   = (1 << 10),
+	i915_sf_dump_seqno          = (1 << 11),
 };
 const char *i915_scheduler_flag_str(uint32_t flags);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 34/39] drm/i915: Add scheduler support functions for TDR

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Change-Id: I720463f01c4edd3579ce52e315a85e4d7874d7e5
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 31 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 2 files changed, 32 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index ebb9ab3..1f1dfa78 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -1627,6 +1627,37 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 	return 0;
 }
 
+/*
+ * Used by TDR to distinguish hung rings (not moving but with work to do)
+ * from idle rings (not moving because there is nothing to do).
+ */
+bool i915_scheduler_is_ring_flying(struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry *node;
+	unsigned long   flags;
+	bool            found = false;
+
+	/* With the scheduler in bypass mode, no information can be returned. */
+	if (i915.scheduler_override & i915_so_direct_submit) {
+		return true;
+	}
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (I915_SQS_IS_FLYING(node)) {
+			found = true;
+			break;
+		}
+	}
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return found;
+}
+
 bool i915_scheduler_file_queue_is_full(struct drm_file *file)
 {
 	struct drm_i915_file_private *file_priv = file->driver_priv;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 46d2503..adaefa6 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -150,6 +150,7 @@ void        i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *nod
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
+bool        i915_scheduler_is_ring_flying(struct intel_engine_cs *ring);
 void        i915_gem_scheduler_work_handler(struct work_struct *work);
 int         i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked);
 int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 35/39] drm/i915: GPU priority bumping to prevent starvation

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

If a high priority task was to continuously submit batch buffers to
the driver, it could starve out any lower priority task from getting
any GPU time at all. To prevent this, the priority of a queued batch
buffer is bumped each time it does not get submitted to the hardware.

v2: Updated for signed priority values.

Change-Id: I0319c7d2f306c61a283f03edda9b5d09a6d3b621
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c   | 28 ++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.c | 14 ++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 3 files changed, 43 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 9e7d67d..66a2202 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1178,6 +1178,33 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_max_fops,
 			"%lld\n");
 
 static int
+i915_scheduler_priority_bump_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_bump;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_bump_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->priority_level_bump = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_bump_fops,
+			i915_scheduler_priority_bump_get,
+			i915_scheduler_priority_bump_set,
+			"%lld\n");
+
+static int
 i915_scheduler_priority_preempt_get(void *data, u64 *val)
 {
 	struct drm_device       *dev       = data;
@@ -5672,6 +5699,7 @@ static const struct i915_debugfs_files {
 	{"i915_next_seqno", &i915_next_seqno_fops},
 	{"i915_scheduler_priority_min", &i915_scheduler_priority_min_fops},
 	{"i915_scheduler_priority_max", &i915_scheduler_priority_max_fops},
+	{"i915_scheduler_priority_bump", &i915_scheduler_priority_bump_fops},
 	{"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops},
 	{"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops},
 	{"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 1f1dfa78..9035cdb 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -191,6 +191,7 @@ int i915_scheduler_init(struct drm_device *dev)
 	/* Default tuning values: */
 	scheduler->priority_level_min     = -1023;
 	scheduler->priority_level_max     = 1023;
+	scheduler->priority_level_bump    = 50;
 	scheduler->priority_level_preempt = 900;
 	scheduler->min_flying             = 2;
 	scheduler->file_queue_max         = 64;
@@ -1502,6 +1503,19 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 		ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
 	} while (ret == 0);
 
+	/*
+	 * Bump the priority of everything that was not submitted to prevent
+	 * starvation of low priority tasks by a spamming high priority task.
+	 */
+	i915_scheduler_priority_bump_clear(scheduler);
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+
+		i915_scheduler_priority_bump(scheduler, node,
+					     scheduler->priority_level_bump);
+	}
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (!was_locked)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index adaefa6..6e0dfa9 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -114,6 +114,7 @@ struct i915_scheduler {
 	/* Tuning parameters: */
 	int32_t             priority_level_min;
 	int32_t             priority_level_max;
+	int32_t             priority_level_bump;
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
 	uint32_t            file_queue_max;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 36/39] drm/i915: Scheduler state dump via debugfs

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Added a facility for triggering the scheduler state dump via a debugfs
entry.

v2: New patch in series.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c   | 33 +++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.c |  9 +++++----
 drivers/gpu/drm/i915/i915_scheduler.h |  6 ++++++
 3 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 66a2202..1c1c4bd 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1285,6 +1285,38 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_file_queue_max_fops,
 			i915_scheduler_file_queue_max_set,
 			"%llu\n");
 
+static int
+i915_scheduler_dump_flags_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = scheduler->dump_flags;
+
+	return 0;
+}
+
+static int
+i915_scheduler_dump_flags_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->dump_flags = lower_32_bits(val) & i915_sf_dump_mask;
+
+	if (val & 1)
+		i915_scheduler_dump_all(dev, "DebugFS");
+
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_dump_flags_fops,
+			i915_scheduler_dump_flags_get,
+			i915_scheduler_dump_flags_set,
+			"0x%llx\n");
+
 static int i915_frequency_info(struct seq_file *m, void *unused)
 {
 	struct drm_info_node *node = m->private;
@@ -5703,6 +5735,7 @@ static const struct i915_debugfs_files {
 	{"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops},
 	{"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops},
 	{"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
+	{"i915_scheduler_dump_flags", &i915_scheduler_dump_flags_fops},
 	{"i915_display_crc_ctl", &i915_display_crc_ctl_fops},
 	{"i915_pri_wm_latency", &i915_pri_wm_latency_fops},
 	{"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 9035cdb..16d067e 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -195,6 +195,10 @@ int i915_scheduler_init(struct drm_device *dev)
 	scheduler->priority_level_preempt = 900;
 	scheduler->min_flying             = 2;
 	scheduler->file_queue_max         = 64;
+	scheduler->dump_flags             = i915_sf_dump_force   |
+					    i915_sf_dump_details |
+					    i915_sf_dump_seqno   |
+					    i915_sf_dump_dependencies;
 
 	dev_priv->scheduler = scheduler;
 
@@ -760,10 +764,7 @@ static int i915_scheduler_dump_all_locked(struct drm_device *dev, const char *ms
 	int                     i, r, ret = 0;
 
 	for_each_ring(ring, dev_priv, i) {
-		scheduler->flags[ring->id] |= i915_sf_dump_force   |
-					      i915_sf_dump_details |
-					      i915_sf_dump_seqno   |
-					      i915_sf_dump_dependencies;
+		scheduler->flags[ring->id] |= scheduler->dump_flags & i915_sf_dump_mask;
 		r = i915_scheduler_dump_locked(ring, msg);
 		if (ret == 0)
 			ret = r;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 6e0dfa9..5257f5c 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -118,6 +118,7 @@ struct i915_scheduler {
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
 	uint32_t            file_queue_max;
+	uint32_t            dump_flags;
 
 	/* Statistics: */
 	struct i915_scheduler_stats     stats[I915_NUM_RINGS];
@@ -134,6 +135,11 @@ enum {
 	i915_sf_dump_details        = (1 << 9),
 	i915_sf_dump_dependencies   = (1 << 10),
 	i915_sf_dump_seqno          = (1 << 11),
+
+	i915_sf_dump_mask           = i915_sf_dump_force        |
+				      i915_sf_dump_details      |
+				      i915_sf_dump_dependencies |
+				      i915_sf_dump_seqno,
 };
 const char *i915_scheduler_flag_str(uint32_t flags);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 37/39] drm/i915: Enable GPU scheduler by default

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Now that all the scheduler patches have been applied, it is safe to enable.

Change-Id: I128042e85a30fca765ce1eb46c837c62dee66089
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_params.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_params.c b/drivers/gpu/drm/i915/i915_params.c
index be2e16c..7db0f83 100644
--- a/drivers/gpu/drm/i915/i915_params.c
+++ b/drivers/gpu/drm/i915/i915_params.c
@@ -54,7 +54,7 @@ struct i915_params i915 __read_mostly = {
 	.edp_vswing = 0,
 	.enable_guc_submission = true,
 	.guc_log_level = -1,
-	.scheduler_override = 1,
+	.scheduler_override = 0,
 };
 
 module_param_named(modeset, i915.modeset, int, 0400);
@@ -194,4 +194,4 @@ MODULE_PARM_DESC(guc_log_level,
 	"GuC firmware logging level (-1:disabled (default), 0-3:enabled)");
 
 module_param_named(scheduler_override, i915.scheduler_override, int, 0600);
-MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (0 = none, 1 = direct submission [default])");
+MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (default: 0)");
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 38/39] drm/i915: Add scheduling priority to per-context parameters

[John.C.Harrison]

From: Dave Gordon <david.s.gordon@intel.com>

Added an interface for user land applications/libraries/services to
set their GPU scheduler priority. This extends the existing context
parameter IOCTL interface to add a scheduler priority parameter. The
range is +/-1023 with +ve numbers meaning higher priority. Only
system processes may set a higher priority than the default (zero),
normal applications may only lower theirs.

v2: New patch in series.

For: VIZ-1587
Signed-off-by: Dave Gordon <David.S.Gordon@Intel.com>
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            | 14 ++++++++++++++
 drivers/gpu/drm/i915/i915_gem_context.c    | 24 ++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  3 +++
 include/uapi/drm/i915_drm.h                |  1 +
 4 files changed, 42 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 4ecb6e4..9de831b 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -843,6 +843,19 @@ struct i915_ctx_hang_stats {
 	bool banned;
 };
 
+/*
+ * User-settable GFX scheduler priorities are on a scale of -1023 (I don't
+ * care about running) to +1023 (I'm the most important thing in existence)
+ * with zero being the default. Any process may decrease its scheduling
+ * priority, but only a sufficiently privileged process may increase it
+ * beyond zero.
+ */
+
+struct i915_ctx_sched_info {
+	/* Scheduling priority */
+	int32_t priority;
+};
+
 struct i915_fence_timeline {
 	char        name[32];
 	unsigned    fence_context;
@@ -883,6 +896,7 @@ struct intel_context {
 	int flags;
 	struct drm_i915_file_private *file_priv;
 	struct i915_ctx_hang_stats hang_stats;
+	struct i915_ctx_sched_info sched_info;
 	struct i915_hw_ppgtt *ppgtt;
 
 	/* Legacy ring buffer submission */
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 2798ddc..b226e6d 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -946,6 +946,9 @@ int i915_gem_context_getparam_ioctl(struct drm_device *dev, void *data,
 		else
 			args->value = to_i915(dev)->gtt.base.total;
 		break;
+	case I915_CONTEXT_PARAM_PRIORITY:
+		args->value = (__u64) ctx->sched_info.priority;
+		break;
 	default:
 		ret = -EINVAL;
 		break;
@@ -983,6 +986,7 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
 		else
 			ctx->hang_stats.ban_period_seconds = args->value;
 		break;
+
 	case I915_CONTEXT_PARAM_NO_ZEROMAP:
 		if (args->size) {
 			ret = -EINVAL;
@@ -991,6 +995,26 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
 			ctx->flags |= args->value ? CONTEXT_NO_ZEROMAP : 0;
 		}
 		break;
+
+	case I915_CONTEXT_PARAM_PRIORITY:
+	{
+		int32_t	priority = (int32_t) args->value;
+		struct drm_i915_private *dev_priv  = dev->dev_private;
+		struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+		if (args->size)
+			ret = -EINVAL;
+		else if ((priority > scheduler->priority_level_max) ||
+			 (priority < scheduler->priority_level_min))
+			ret = -EINVAL;
+		else if ((priority > 0) &&
+			 !capable(CAP_SYS_ADMIN))
+			ret = -EPERM;
+		else
+			ctx->sched_info.priority = priority;
+		break;
+	}
+
 	default:
 		ret = -EINVAL;
 		break;
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index dfd9c29..8553297 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1725,6 +1725,9 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
 
+	/* Start with the context's priority level */
+	qe.priority = ctx->sched_info.priority;
+
 	/*
 	 * Save away the list of objects used by this batch buffer for the
 	 * purpose of tracking inter-buffer dependencies.
diff --git a/include/uapi/drm/i915_drm.h b/include/uapi/drm/i915_drm.h
index 86f7921..5f35f89 100644
--- a/include/uapi/drm/i915_drm.h
+++ b/include/uapi/drm/i915_drm.h
@@ -1138,6 +1138,7 @@ struct drm_i915_gem_context_param {
 #define I915_CONTEXT_PARAM_BAN_PERIOD	0x1
 #define I915_CONTEXT_PARAM_NO_ZEROMAP	0x2
 #define I915_CONTEXT_PARAM_GTT_SIZE	0x3
+#define I915_CONTEXT_PARAM_PRIORITY	0x4
 	__u64 value;
 };
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 39/39] drm/i915: Allow scheduler to manage inter-ring object synchronisation

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler has always tracked batch buffer dependencies based on
DRM object usage. This means that it will not submit a batch on one
ring that has outstanding dependencies still executing on other rings.
This is exactly the same synchronisation performed by
i915_gem_object_sync() using hardware semaphores where available and
CPU stalls where not (e.g. in execlist mode and/or on Gen8 hardware).

Unfortunately, when a batch buffer is submitted to the driver the
_object_sync() call happens first. Thus in case where hardware
semaphores are disabled, the driver has already stalled until the
dependency has been resolved.

This patch adds an optimisation to _object_sync() to ignore the
synchronisation in the case where it will subsequently be handled by
the scheduler. This removes the driver stall and (in the single
application case) provides near hardware semaphore performance even
when hardware semaphores are disabled. In a busy system where there is
other work that can be executed on the stalling ring, it provides
better than hardware semaphore performance as it removes the stall
from both the driver and from the hardware. There is also a theory
that this method should improve power usage as hardware semaphores are
apparently not very power efficient - the stalled ring does not go
into as low a power a state as when it is genuinely idle.

The optimisation is to check whether both ends of the synchronisation
are batch buffer requests. If they are, then the scheduler will have
the inter-dependency tracked and managed. If one or other end is not a
batch buffer request (e.g. a page flip) then the code falls back to
the CPU stall or hardware semaphore as appropriate.

To check whether the existing usage is a batch buffer, the code simply
calls the 'are you tracking this request' function of the scheduler on
the object's last_read_req member. To check whether the new usage is a
batch buffer, a flag is passed in from the caller.

Change-Id: Idc16e19b5a4dc8b3782ce9db44dd3df445f396c1
Issue: VIZ-5566
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  2 +-
 drivers/gpu/drm/i915/i915_gem.c            | 17 ++++++++++++++---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  2 +-
 drivers/gpu/drm/i915/intel_display.c       |  2 +-
 drivers/gpu/drm/i915/intel_lrc.c           |  2 +-
 5 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 9de831b..809816a 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2971,7 +2971,7 @@ int __must_check i915_mutex_lock_interruptible(struct drm_device *dev);
 #endif
 int i915_gem_object_sync(struct drm_i915_gem_object *obj,
 			 struct intel_engine_cs *to,
-			 struct drm_i915_gem_request **to_req);
+			 struct drm_i915_gem_request **to_req, bool to_batch);
 void i915_vma_move_to_active(struct i915_vma *vma,
 			     struct drm_i915_gem_request *req);
 int i915_gem_dumb_create(struct drm_file *file_priv,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 9c08035..ab6064e 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3555,7 +3555,7 @@ static int
 __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 		       struct intel_engine_cs *to,
 		       struct drm_i915_gem_request *from_req,
-		       struct drm_i915_gem_request **to_req)
+		       struct drm_i915_gem_request **to_req, bool to_batch)
 {
 	struct intel_engine_cs *from;
 	int ret;
@@ -3567,6 +3567,15 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 	if (i915_gem_request_completed(from_req))
 		return 0;
 
+	/*
+	 * The scheduler will manage inter-ring object dependencies
+	 * as long as both to and from requests are scheduler managed
+	 * (i.e. batch buffers).
+	 */
+	if (to_batch &&
+	    i915_scheduler_is_request_tracked(from_req, NULL, NULL))
+		return 0;
+
 	if (!i915_semaphore_is_enabled(obj->base.dev)) {
 		struct drm_i915_private *i915 = to_i915(obj->base.dev);
 		ret = __i915_wait_request(from_req,
@@ -3617,6 +3626,8 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
  * @to_req: request we wish to use the object for. See below.
  *          This will be allocated and returned if a request is
  *          required but not passed in.
+ * @to_batch: is the sync request on behalf of batch buffer submission?
+ * If so then the scheduler can (potentially) manage the synchronisation.
  *
  * This code is meant to abstract object synchronization with the GPU.
  * Calling with NULL implies synchronizing the object with the CPU
@@ -3647,7 +3658,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 int
 i915_gem_object_sync(struct drm_i915_gem_object *obj,
 		     struct intel_engine_cs *to,
-		     struct drm_i915_gem_request **to_req)
+		     struct drm_i915_gem_request **to_req, bool to_batch)
 {
 	const bool readonly = obj->base.pending_write_domain == 0;
 	struct drm_i915_gem_request *req[I915_NUM_RINGS];
@@ -3669,7 +3680,7 @@ i915_gem_object_sync(struct drm_i915_gem_object *obj,
 				req[n++] = obj->last_read_req[i];
 	}
 	for (i = 0; i < n; i++) {
-		ret = __i915_gem_object_sync(obj, to, req[i], to_req);
+		ret = __i915_gem_object_sync(obj, to, req[i], to_req, to_batch);
 		if (ret)
 			return ret;
 	}
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 8553297..20693c3 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -919,7 +919,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
 		struct drm_i915_gem_object *obj = vma->obj;
 
 		if (obj->active & other_rings) {
-			ret = i915_gem_object_sync(obj, req->ring, &req);
+			ret = i915_gem_object_sync(obj, req->ring, &req, true);
 			if (ret)
 				return ret;
 		}
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 58c464f..3612562 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -11487,7 +11487,7 @@ static int intel_crtc_page_flip(struct drm_crtc *crtc,
 	 * into the display plane and skip any waits.
 	 */
 	if (!mmio_flip) {
-		ret = i915_gem_object_sync(obj, ring, &request);
+		ret = i915_gem_object_sync(obj, ring, &request, false);
 		if (ret)
 			goto cleanup_pending;
 	}
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index b5abea4..7a3fb219 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -658,7 +658,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,
 		struct drm_i915_gem_object *obj = vma->obj;
 
 		if (obj->active & other_rings) {
-			ret = i915_gem_object_sync(obj, req->ring, &req);
+			ret = i915_gem_object_sync(obj, req->ring, &req, true);
 			if (ret)
 				return ret;
 		}
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 05/40] drm/i915: Split i915_dem_do_execbuffer() in half

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Split the execbuffer() function in half. The first half collects and
validates all the information requried to process the batch buffer. It
also does all the object pinning, relocations, active list management,
etc - basically anything that must be done upfront before the IOCTL
returns and allows the user land side to start changing/freeing
things. The second half does the actual ring submission.

This change implements the split but leaves the back half being called
directly from the end of the front half.

v2: Updated due to changes in underlying tree - addition of sync fence
support and removal of cliprects.

v3: Moved local 'ringbuf' variable to make later patches in the
series a bit neater.

Change-Id: I5e1c77639ce526ab2401b0323186c518bf13da0a
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  11 +++
 drivers/gpu/drm/i915/i915_gem.c            |   2 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 130 ++++++++++++++++++++---------
 drivers/gpu/drm/i915/intel_lrc.c           |  57 +++++++++----
 drivers/gpu/drm/i915/intel_lrc.h           |   1 +
 5 files changed, 145 insertions(+), 56 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 194bca0..eb00454 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1679,10 +1679,18 @@ struct i915_execbuffer_params {
 	struct drm_device               *dev;
 	struct drm_file                 *file;
 	uint32_t                        dispatch_flags;
+	uint32_t                        args_flags;
 	uint32_t                        args_batch_start_offset;
+	uint32_t                        args_batch_len;
+	uint32_t                        args_num_cliprects;
+	uint32_t                        args_DR1;
+	uint32_t                        args_DR4;
 	uint64_t                        batch_obj_vm_offset;
 	struct intel_engine_cs          *ring;
 	struct drm_i915_gem_object      *batch_obj;
+	struct drm_clip_rect            *cliprects;
+	uint32_t                        instp_mask;
+	int                             instp_mode;
 	struct intel_context            *ctx;
 	struct drm_i915_gem_request     *request;
 };
@@ -1944,6 +1952,7 @@ struct drm_i915_private {
 		int (*execbuf_submit)(struct i915_execbuffer_params *params,
 				      struct drm_i915_gem_execbuffer2 *args,
 				      struct list_head *vmas);
+		int (*execbuf_final)(struct i915_execbuffer_params *params);
 		int (*init_rings)(struct drm_device *dev);
 		void (*cleanup_ring)(struct intel_engine_cs *ring);
 		void (*stop_ring)(struct intel_engine_cs *ring);
@@ -2798,9 +2807,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,
 void i915_gem_execbuffer_move_to_active(struct list_head *vmas,
 					struct drm_i915_gem_request *req);
 void i915_gem_execbuffer_retire_commands(struct i915_execbuffer_params *params);
+void i915_gem_execbuff_release_batch_obj(struct drm_i915_gem_object *batch_obj);
 int i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 				   struct drm_i915_gem_execbuffer2 *args,
 				   struct list_head *vmas);
+int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params);
 int i915_gem_execbuffer(struct drm_device *dev, void *data,
 			struct drm_file *file_priv);
 int i915_gem_execbuffer2(struct drm_device *dev, void *data,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 3c88678..b9501ca 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -5257,11 +5257,13 @@ int i915_gem_init(struct drm_device *dev)
 
 	if (!i915.enable_execlists) {
 		dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission;
+		dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final;
 		dev_priv->gt.init_rings = i915_gem_init_rings;
 		dev_priv->gt.cleanup_ring = intel_cleanup_ring_buffer;
 		dev_priv->gt.stop_ring = intel_stop_ring_buffer;
 	} else {
 		dev_priv->gt.execbuf_submit = intel_execlists_submission;
+		dev_priv->gt.execbuf_final = intel_execlists_submission_final;
 		dev_priv->gt.init_rings = intel_logical_rings_init;
 		dev_priv->gt.cleanup_ring = intel_logical_ring_cleanup;
 		dev_priv->gt.stop_ring = intel_logical_ring_stop;
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index f7f1057..05c9de6 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1183,41 +1183,38 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	struct drm_device *dev = params->dev;
 	struct intel_engine_cs *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	u64 exec_start, exec_len;
-	int instp_mode;
-	u32 instp_mask;
 	int ret;
 
-	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
-	instp_mask = I915_EXEC_CONSTANTS_MASK;
-	switch (instp_mode) {
+	params->instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
+	params->instp_mask = I915_EXEC_CONSTANTS_MASK;
+	switch (params->instp_mode) {
 	case I915_EXEC_CONSTANTS_REL_GENERAL:
 	case I915_EXEC_CONSTANTS_ABSOLUTE:
 	case I915_EXEC_CONSTANTS_REL_SURFACE:
-		if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
+		if (params->instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
 			DRM_DEBUG("non-0 rel constants mode on non-RCS\n");
 			return -EINVAL;
 		}
 
-		if (instp_mode != dev_priv->relative_constants_mode) {
+		if (params->instp_mode != dev_priv->relative_constants_mode) {
 			if (INTEL_INFO(dev)->gen < 4) {
 				DRM_DEBUG("no rel constants on pre-gen4\n");
 				return -EINVAL;
 			}
 
 			if (INTEL_INFO(dev)->gen > 5 &&
-			    instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
+			    params->instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
 				DRM_DEBUG("rel surface constants mode invalid on gen5+\n");
 				return -EINVAL;
 			}
 
 			/* The HW changed the meaning on this bit on gen6 */
 			if (INTEL_INFO(dev)->gen >= 6)
-				instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
+				params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
 		}
 		break;
 	default:
-		DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode);
+		DRM_DEBUG("execbuf with unknown constants: %d\n", params->instp_mode);
 		return -EINVAL;
 	}
 
@@ -1227,7 +1224,33 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	/* To be split into two functions here... */
+	ret = dev_priv->gt.execbuf_final(params);
+	if (ret)
+		return ret;
+
+	/*
+	 * Free everything that was stored in the QE structure (until the
+	 * scheduler arrives and does it instead):
+	 */
+	if (params->dispatch_flags & I915_DISPATCH_SECURE)
+		i915_gem_execbuff_release_batch_obj(params->batch_obj);
+
+	return 0;
+}
+
+/*
+ * This is the main function for adding a batch to the ring.
+ * It is called from the scheduler, with the struct_mutex already held.
+ */
+int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
+{
+	struct drm_i915_private *dev_priv = params->dev->dev_private;
+	struct intel_engine_cs  *ring = params->ring;
+	u64 exec_start, exec_len;
+	int ret;
+
+	/* The mutex must be acquired before calling this function */
+	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
 	intel_runtime_pm_get(dev_priv);
 
@@ -1248,7 +1271,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	     "%s didn't clear reload\n", ring->name);
 
 	if (ring == &dev_priv->ring[RCS] &&
-	    instp_mode != dev_priv->relative_constants_mode) {
+	    params->instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_ring_begin(params->request, 4);
 		if (ret)
 			goto error;
@@ -1256,19 +1279,19 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 		intel_ring_emit(ring, MI_NOOP);
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, INSTPM);
-		intel_ring_emit(ring, instp_mask << 16 | instp_mode);
+		intel_ring_emit(ring, params->instp_mask << 16 | params->instp_mode);
 		intel_ring_advance(ring);
 
-		dev_priv->relative_constants_mode = instp_mode;
+		dev_priv->relative_constants_mode = params->instp_mode;
 	}
 
-	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {
-		ret = i915_reset_gen7_sol_offsets(dev, params->request);
+	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
+		ret = i915_reset_gen7_sol_offsets(params->dev, params->request);
 		if (ret)
 			goto error;
 	}
 
-	exec_len   = args->batch_len;
+	exec_len   = params->args_batch_len;
 	exec_start = params->batch_obj_vm_offset +
 		     params->args_batch_start_offset;
 
@@ -1640,6 +1663,11 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->file                    = file;
 	params->ring                    = ring;
 	params->dispatch_flags          = dispatch_flags;
+	params->args_flags              = args->flags;
+	params->args_batch_len          = args->batch_len;
+	params->args_num_cliprects      = args->num_cliprects;
+	params->args_DR1                = args->DR1;
+	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
 	params->ctx                     = ctx;
 
@@ -1659,37 +1687,50 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	}
 
 	ret = dev_priv->gt.execbuf_submit(params, args, &eb->vmas);
+	if (ret)
+		goto err_fence;
+
+	/* the request owns the ref now */
+	i915_gem_context_unreference(ctx);
 
 	if (fd_fence_complete != -1) {
-		if (ret) {
-			sync_fence_put(sync_fence);
-			put_unused_fd(fd_fence_complete);
-		} else {
-			/*
-			 * Install the fence into the pre-allocated file
-			 * descriptor to the fence object so that user land
-			 * can wait on it...
-			 */
-			i915_install_sync_fence_fd(params->request,
-						   sync_fence, fd_fence_complete);
+		/*
+		 * Install the fence into the pre-allocated file
+		 * descriptor to the fence object so that user land
+		 * can wait on it...
+		 */
+		i915_install_sync_fence_fd(params->request,
+					   sync_fence, fd_fence_complete);
 
-			/* Return the fence through the rsvd2 field */
-			args->rsvd2 = (__u64) fd_fence_complete;
-		}
+		/* Return the fence through the rsvd2 field */
+		args->rsvd2 = (__u64) fd_fence_complete;
 	}
 
-err_batch_unpin:
 	/*
-	 * FIXME: We crucially rely upon the active tracking for the (ppgtt)
-	 * batch vma for correctness. For less ugly and less fragility this
-	 * needs to be adjusted to also track the ggtt batch vma properly as
-	 * active.
+	 * The eb list is no longer required. The scheduler has extracted all
+	 * the information than needs to persist.
+	 */
+	eb_destroy(eb);
+
+	/*
+	 * Don't clean up everything that is now saved away in the queue.
+	 * Just unlock and return immediately.
 	 */
+	mutex_unlock(&dev->struct_mutex);
+
+	return 0;
+
+err_fence:
+	if (fd_fence_complete != -1) {
+		sync_fence_put(sync_fence);
+		put_unused_fd(fd_fence_complete);
+	}
+
+err_batch_unpin:
 	if (dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_object_ggtt_unpin(batch_obj);
+		i915_gem_execbuff_release_batch_obj(batch_obj);
 
 err:
-	/* the request owns the ref now */
 	i915_gem_context_unreference(ctx);
 	eb_destroy(eb);
 
@@ -1707,6 +1748,17 @@ pre_mutex_err:
 	return ret;
 }
 
+void i915_gem_execbuff_release_batch_obj(struct drm_i915_gem_object *batch_obj)
+{
+	/*
+	 * FIXME: We crucially rely upon the active tracking for the (ppgtt)
+	 * batch vma for correctness. For less ugly and less fragility this
+	 * needs to be adjusted to also track the ggtt batch vma properly as
+	 * active.
+	 */
+	i915_gem_object_ggtt_unpin(batch_obj);
+}
+
 /*
  * Legacy execbuffer just creates an exec2 list from the original exec object
  * list array and passes it to the real function.
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 4bf0ee6..88d57b7 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -869,35 +869,31 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 	struct drm_device       *dev = params->dev;
 	struct intel_engine_cs  *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct intel_ringbuffer *ringbuf = params->ctx->engine[ring->id].ringbuf;
-	u64 exec_start;
-	int instp_mode;
-	u32 instp_mask;
 	int ret;
 
-	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
-	instp_mask = I915_EXEC_CONSTANTS_MASK;
-	switch (instp_mode) {
+	params->instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
+	params->instp_mask = I915_EXEC_CONSTANTS_MASK;
+	switch (params->instp_mode) {
 	case I915_EXEC_CONSTANTS_REL_GENERAL:
 	case I915_EXEC_CONSTANTS_ABSOLUTE:
 	case I915_EXEC_CONSTANTS_REL_SURFACE:
-		if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
+		if (params->instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
 			DRM_DEBUG("non-0 rel constants mode on non-RCS\n");
 			return -EINVAL;
 		}
 
-		if (instp_mode != dev_priv->relative_constants_mode) {
-			if (instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
+		if (params->instp_mode != dev_priv->relative_constants_mode) {
+			if (params->instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
 				DRM_DEBUG("rel surface constants mode invalid on gen5+\n");
 				return -EINVAL;
 			}
 
 			/* The HW changed the meaning on this bit on gen6 */
-			instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
+			params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
 		}
 		break;
 	default:
-		DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode);
+		DRM_DEBUG("execbuf with unknown constants: %d\n", params->instp_mode);
 		return -EINVAL;
 	}
 
@@ -912,7 +908,34 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	/* To be split into two functions here... */
+	ret = dev_priv->gt.execbuf_final(params);
+	if (ret)
+		return ret;
+
+	/*
+	 * Free everything that was stored in the QE structure (until the
+	 * scheduler arrives and does it instead):
+	 */
+	if (params->dispatch_flags & I915_DISPATCH_SECURE)
+		i915_gem_execbuff_release_batch_obj(params->batch_obj);
+
+	return 0;
+}
+
+/*
+ * This is the main function for adding a batch to the ring.
+ * It is called from the scheduler, with the struct_mutex already held.
+ */
+int intel_execlists_submission_final(struct i915_execbuffer_params *params)
+{
+	struct drm_i915_private *dev_priv = params->dev->dev_private;
+	struct intel_ringbuffer *ringbuf = params->request->ringbuf;
+	struct intel_engine_cs *ring = params->ring;
+	u64 exec_start;
+	int ret;
+
+	/* The mutex must be acquired before calling this function */
+	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
@@ -923,7 +946,7 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 		return ret;
 
 	if (ring == &dev_priv->ring[RCS] &&
-	    instp_mode != dev_priv->relative_constants_mode) {
+	    params->instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_logical_ring_begin(params->request, 4);
 		if (ret)
 			return ret;
@@ -931,14 +954,14 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 		intel_logical_ring_emit(ringbuf, MI_NOOP);
 		intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1));
 		intel_logical_ring_emit(ringbuf, INSTPM);
-		intel_logical_ring_emit(ringbuf, instp_mask << 16 | instp_mode);
+		intel_logical_ring_emit(ringbuf, params->instp_mask << 16 | params->instp_mode);
 		intel_logical_ring_advance(ringbuf);
 
-		dev_priv->relative_constants_mode = instp_mode;
+		dev_priv->relative_constants_mode = params->instp_mode;
 	}
 
 	exec_start = params->batch_obj_vm_offset +
-		     args->batch_start_offset;
+		     params->args_batch_start_offset;
 
 	ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags);
 	if (ret)
diff --git a/drivers/gpu/drm/i915/intel_lrc.h b/drivers/gpu/drm/i915/intel_lrc.h
index 4e60d54..8d9bad7 100644
--- a/drivers/gpu/drm/i915/intel_lrc.h
+++ b/drivers/gpu/drm/i915/intel_lrc.h
@@ -93,6 +93,7 @@ struct i915_execbuffer_params;
 int intel_execlists_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas);
+int intel_execlists_submission_final(struct i915_execbuffer_params *params);
 u32 intel_execlists_ctx_id(struct drm_i915_gem_object *ctx_obj);
 
 void intel_lrc_irq_handler(struct intel_engine_cs *ring);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 06/40] drm/i915: Cache request pointer in *_submission_final()

[John.C.Harrison]

From: Dave Gordon <david.s.gordon@intel.com>

Keep a local copy of the request pointer in the _final() functions
rather than dereferencing the params block repeatedly.

v3: New patch in series.

For: VIZ-1587
Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++------
 drivers/gpu/drm/i915/intel_lrc.c           | 11 ++++++-----
 2 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 05c9de6..e38310f 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1245,6 +1245,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 {
 	struct drm_i915_private *dev_priv = params->dev->dev_private;
+	struct drm_i915_gem_request *req = params->request;
 	struct intel_engine_cs  *ring = params->ring;
 	u64 exec_start, exec_len;
 	int ret;
@@ -1258,12 +1259,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
-	ret = intel_ring_invalidate_all_caches(params->request);
+	ret = intel_ring_invalidate_all_caches(req);
 	if (ret)
 		goto error;
 
 	/* Switch to the correct context for the batch */
-	ret = i915_switch_context(params->request);
+	ret = i915_switch_context(req);
 	if (ret)
 		goto error;
 
@@ -1272,7 +1273,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_ring_begin(params->request, 4);
+		ret = intel_ring_begin(req, 4);
 		if (ret)
 			goto error;
 
@@ -1286,7 +1287,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	}
 
 	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
-		ret = i915_reset_gen7_sol_offsets(params->dev, params->request);
+		ret = i915_reset_gen7_sol_offsets(params->dev, req);
 		if (ret)
 			goto error;
 	}
@@ -1295,13 +1296,13 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	exec_start = params->batch_obj_vm_offset +
 		     params->args_batch_start_offset;
 
-	ret = ring->dispatch_execbuffer(params->request,
+	ret = ring->dispatch_execbuffer(req,
 					exec_start, exec_len,
 					params->dispatch_flags);
 	if (ret)
 		goto error;
 
-	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
+	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 88d57b7..b98ea3d 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -929,7 +929,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 {
 	struct drm_i915_private *dev_priv = params->dev->dev_private;
-	struct intel_ringbuffer *ringbuf = params->request->ringbuf;
+	struct drm_i915_gem_request *req = params->request;
+	struct intel_ringbuffer *ringbuf = req->ringbuf;
 	struct intel_engine_cs *ring = params->ring;
 	u64 exec_start;
 	int ret;
@@ -941,13 +942,13 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
-	ret = logical_ring_invalidate_all_caches(params->request);
+	ret = logical_ring_invalidate_all_caches(req);
 	if (ret)
 		return ret;
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_logical_ring_begin(params->request, 4);
+		ret = intel_logical_ring_begin(req, 4);
 		if (ret)
 			return ret;
 
@@ -963,11 +964,11 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	exec_start = params->batch_obj_vm_offset +
 		     params->args_batch_start_offset;
 
-	ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags);
+	ret = ring->emit_bb_start(req, exec_start, params->dispatch_flags);
 	if (ret)
 		return ret;
 
-	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
+	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 08/40] drm/i915: Start of GPU scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Initial creation of scheduler source files. Note that this patch
implements most of the scheduler functionality but does not hook it in
to the driver yet. It also leaves the scheduler code in 'pass through'
mode so that even when it is hooked in, it will not actually do very
much. This allows the hooks to be added one at a time in byte size
chunks and only when the scheduler is finally enabled at the end does
anything start happening.

The general theory of operation is that when batch buffers are
submitted to the driver, the execbuffer() code assigns a unique
request and then packages up all the information required to execute
the batch buffer at a later time. This package is given over to the
scheduler which adds it to an internal node list. The scheduler also
scans the list of objects associated with the batch buffer and
compares them against the objects already in use by other buffers in
the node list. If matches are found then the new batch buffer node is
marked as being dependent upon the matching node. The same is done for
the context object. The scheduler also bumps up the priority of such
matching nodes on the grounds that the more dependencies a given batch
buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers
in flight on the hardware at any given time. If fewer than this are
currently executing when a new node is queued, then the node is passed
straight through to the submit function. Otherwise it is simply added
to the queue and the driver returns back to user land.

As each batch buffer completes, it raises an interrupt which wakes up
the scheduler. Note that it is possible for multiple buffers to
complete before the IRQ handler gets to run. Further, it is possible
for the seqno values to be un-ordered (particularly once pre-emption
is enabled). However, the scheduler keeps the list of executing
buffers in order of hardware submission. Thus it can scan through the
list until a matching seqno is found and then mark all in flight nodes
from that point on as completed.

A deferred work queue is also poked by the interrupt handler. When
this wakes up it can do more involved processing such as actually
removing completed nodes from the queue and freeing up the resources
associated with them (internal memory allocations, DRM object
references, context reference, etc.). The work handler also checks the
in flight count and calls the submission code if a new slot has
appeared.

When the scheduler's submit code is called, it scans the queued node
list for the highest priority node that has no unmet dependencies.
Note that the dependency calculation is complex as it must take
inter-ring dependencies and potential preemptions into account. Note
also that in the future this will be extended to include external
dependencies such as the Android Native Sync file descriptors and/or
the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for
submission to the hardware. The in flight count is then re-checked and
a new node popped from the list if appropriate.

Note that this patch does not implement pre-emptive scheduling. Only
basic scheduling by re-ordering batch buffer submission is currently
implemented.

v2: Changed priority levels to +/-1023 due to feedback from Chris
Wilson.

Removed redundant index from scheduler node.

Changed time stamps to use jiffies instead of raw monotonic. This
provides lower resolution but improved compatibility with other i915
code.

Major re-write of completion tracking code due to struct fence
conversion. The scheduler no longer has it's own private IRQ handler
but just lets the existing request code handle completion events.
Instead, the scheduler now hooks into the request notify code to be
told when a request has completed.

Reduced driver mutex locking scope. Removal of scheduler nodes no
longer grabs the mutex lock.

v3: Refactor of dependency generation to make the code more readable.
Also added in read-read optimisation support - i.e., don't treat a
shared read-only buffer as being a dependency.

Allowed the killing of queued nodes rather than only flying ones.

Change-Id: I1e08f59e650a3c2bbaaa9de7627da33849b06106
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/Makefile         |   1 +
 drivers/gpu/drm/i915/i915_drv.h       |   4 +
 drivers/gpu/drm/i915/i915_gem.c       |   5 +
 drivers/gpu/drm/i915/i915_scheduler.c | 763 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  91 ++++
 5 files changed, 864 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h

diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 15398c5..79cb38b 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -10,6 +10,7 @@ ccflags-y := -Werror
 i915-y := i915_drv.o \
 	  i915_irq.o \
 	  i915_params.o \
+	  i915_scheduler.o \
           i915_suspend.o \
 	  i915_sysfs.o \
 	  intel_csr.o \
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 5d390d9..23aed32 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {
 	struct drm_i915_gem_request     *request;
 };
 
+struct i915_scheduler;
+
 /* used in computing the new watermarks state */
 struct intel_wm_config {
 	unsigned int num_pipes_active;
@@ -1947,6 +1949,8 @@ struct drm_i915_private {
 
 	struct i915_runtime_pm pm;
 
+	struct i915_scheduler *scheduler;
+
 	/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */
 	struct {
 		int (*execbuf_submit)(struct i915_execbuffer_params *params,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 280cff0..36a11d2 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -38,6 +38,7 @@
 #include <linux/pci.h>
 #include <linux/dma-buf.h>
 #include <../drivers/android/sync.h>
+#include "i915_scheduler.h"
 
 #define RQ_BUG_ON(expr)
 
@@ -5278,6 +5279,10 @@ int i915_gem_init(struct drm_device *dev)
 	 */
 	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
+	ret = i915_scheduler_init(dev);
+	if (ret)
+		goto out_unlock;
+
 	ret = i915_gem_init_userptr(dev);
 	if (ret)
 		goto out_unlock;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
new file mode 100644
index 0000000..2e2c590
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -0,0 +1,763 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include "i915_drv.h"
+#include "intel_drv.h"
+#include "i915_scheduler.h"
+
+static int         i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node);
+static int         i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
+						   struct i915_scheduler_queue_entry *remove);
+static int         i915_scheduler_submit(struct intel_engine_cs *ring,
+					 bool is_locked);
+static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
+					       struct intel_engine_cs *ring);
+static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
+static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
+						struct i915_scheduler_queue_entry *target,
+						uint32_t bump);
+
+int i915_scheduler_init(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	int                     r;
+
+	if (scheduler)
+		return 0;
+
+	scheduler = kzalloc(sizeof(*scheduler), GFP_KERNEL);
+	if (!scheduler)
+		return -ENOMEM;
+
+	spin_lock_init(&scheduler->lock);
+
+	for (r = 0; r < I915_NUM_RINGS; r++)
+		INIT_LIST_HEAD(&scheduler->node_queue[r]);
+
+	/* Default tuning values: */
+	scheduler->priority_level_min     = -1023;
+	scheduler->priority_level_max     = 1023;
+	scheduler->priority_level_preempt = 900;
+	scheduler->min_flying             = 2;
+
+	dev_priv->scheduler = scheduler;
+
+	return 0;
+}
+
+static void i915_generate_dependencies(struct i915_scheduler *scheduler,
+				       struct i915_scheduler_queue_entry *node,
+				       uint32_t ring)
+{
+	struct i915_scheduler_obj_entry *this, *that;
+	struct i915_scheduler_queue_entry  *test;
+	int                 i, j;
+	bool                found;
+
+	list_for_each_entry(test, &scheduler->node_queue[ring], link) {
+		if (I915_SQS_IS_COMPLETE(test))
+			continue;
+
+		/*
+		 * Batches on the same ring for the same
+		 * context must be kept in order.
+		 */
+		found = (node->params.ctx == test->params.ctx) &&
+			(node->params.ring == test->params.ring);
+
+		if (found)
+			goto depends;
+
+		/*
+		 * Batches working on the same objects must
+		 * be kept in order.
+		 */
+		for (i = 0; (i < node->num_objs) && !found; i++) {
+			this = node->saved_objects + i;
+
+			for (j = 0; j < test->num_objs; j++) {
+				that = test->saved_objects + j;
+
+				if (this->obj != that->obj)
+					continue;
+
+				/* Only need to worry about writes */
+				if (this->read_only && that->read_only)
+					continue;
+
+				found = true;
+				break;
+			}
+		}
+
+		if (!found)
+			continue;
+
+depends:
+		node->dep_list[node->num_deps] = test;
+		node->num_deps++;
+	}
+}
+
+int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
+{
+	struct drm_i915_private *dev_priv = qe->params.dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring = qe->params.ring;
+	struct i915_scheduler_queue_entry  *node;
+	struct i915_scheduler_queue_entry  *test;
+	unsigned long       flags;
+	bool                not_flying;
+	int                 i, r;
+	int                 incomplete = 0;
+
+	BUG_ON(!scheduler);
+
+	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
+		int ret;
+
+		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
+		ret = dev_priv->gt.execbuf_final(&qe->params);
+		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
+
+		/*
+		 * Don't do any clean up on failure because the caller will
+		 * do it all anyway.
+		 */
+		if (ret)
+			return ret;
+
+		/* Free everything that is owned by the QE structure: */
+		kfree(qe->params.cliprects);
+		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
+
+		return 0;
+	}
+
+	node = kmalloc(sizeof(*node), GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	*node = *qe;
+	INIT_LIST_HEAD(&node->link);
+	node->status = i915_sqs_queued;
+	node->stamp  = jiffies;
+	i915_gem_request_reference(node->params.request);
+
+	/* Need to determine the number of incomplete entries in the list as
+	 * that will be the maximum size of the dependency list.
+	 *
+	 * Note that the allocation must not be made with the spinlock acquired
+	 * as kmalloc can sleep. However, the unlock/relock is safe because no
+	 * new entries can be queued up during the unlock as the i915 driver
+	 * mutex is still held. Entries could be removed from the list but that
+	 * just means the dep_list will be over-allocated which is fine.
+	 */
+	spin_lock_irqsave(&scheduler->lock, flags);
+	for (r = 0; r < I915_NUM_RINGS; r++) {
+		list_for_each_entry(test, &scheduler->node_queue[r], link) {
+			if (I915_SQS_IS_COMPLETE(test))
+				continue;
+
+			incomplete++;
+		}
+	}
+
+	/* Temporarily unlock to allocate memory: */
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+	if (incomplete) {
+		node->dep_list = kmalloc(sizeof(node->dep_list[0]) * incomplete,
+					 GFP_KERNEL);
+		if (!node->dep_list) {
+			kfree(node);
+			return -ENOMEM;
+		}
+	} else
+		node->dep_list = NULL;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	node->num_deps = 0;
+
+	if (node->dep_list) {
+		for (r = 0; r < I915_NUM_RINGS; r++)
+			i915_generate_dependencies(scheduler, node, r);
+
+		BUG_ON(node->num_deps > incomplete);
+	}
+
+	if (node->priority > scheduler->priority_level_max)
+		node->priority = scheduler->priority_level_max;
+	else if (node->priority < scheduler->priority_level_min)
+		node->priority = scheduler->priority_level_min;
+
+	if ((node->priority > 0) && node->num_deps) {
+		i915_scheduler_priority_bump_clear(scheduler);
+
+		for (i = 0; i < node->num_deps; i++)
+			i915_scheduler_priority_bump(scheduler,
+					node->dep_list[i], node->priority);
+	}
+
+	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
+
+	not_flying = i915_scheduler_count_flying(scheduler, ring) <
+						 scheduler->min_flying;
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (not_flying)
+		i915_scheduler_submit(ring, true);
+
+	return 0;
+}
+
+static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
+{
+	struct drm_i915_private *dev_priv = node->params.dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring;
+
+	BUG_ON(!scheduler);
+	BUG_ON(!node);
+	BUG_ON(node->status != i915_sqs_popped);
+
+	ring = node->params.ring;
+
+	/* Add the node (which should currently be in state none) to the front
+	 * of the queue. This ensure that flying nodes are always held in
+	 * hardware submission order. */
+	list_add(&node->link, &scheduler->node_queue[ring->id]);
+
+	node->status = i915_sqs_flying;
+
+	if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
+		bool    success = true;
+
+		success = ring->irq_get(ring);
+		if (success)
+			scheduler->flags[ring->id] |= i915_sf_interrupts_enabled;
+		else
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * Nodes are considered valid dependencies if they are queued on any ring or
+ * if they are in flight on a different ring. In flight on the same ring is no
+ * longer interesting for non-premptive nodes as the ring serialises execution.
+ * For pre-empting nodes, all in flight dependencies are valid as they must not
+ * be jumped by the act of pre-empting.
+ *
+ * Anything that is neither queued nor flying is uninteresting.
+ */
+static inline bool i915_scheduler_is_dependency_valid(
+			struct i915_scheduler_queue_entry *node, uint32_t idx)
+{
+	struct i915_scheduler_queue_entry *dep;
+
+	dep = node->dep_list[idx];
+	if (!dep)
+		return false;
+
+	if (I915_SQS_IS_QUEUED(dep))
+		return true;
+
+	if (I915_SQS_IS_FLYING(dep)) {
+		if (node->params.ring != dep->params.ring)
+			return true;
+	}
+
+	return false;
+}
+
+static uint32_t i915_scheduler_count_flying(struct i915_scheduler *scheduler,
+					    struct intel_engine_cs *ring)
+{
+	struct i915_scheduler_queue_entry *node;
+	uint32_t                          flying = 0;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link)
+		if (I915_SQS_IS_FLYING(node))
+			flying++;
+
+	return flying;
+}
+
+/* Add a popped node back in to the queue. For example, because the ring was
+ * hung when execfinal() was called and thus the ring submission needs to be
+ * retried later. */
+static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
+{
+	BUG_ON(!node);
+	BUG_ON(!I915_SQS_IS_FLYING(node));
+
+	node->status = i915_sqs_queued;
+	node->params.request->seqno = 0;
+}
+
+/* Give up on a node completely. For example, because it is causing the
+ * ring to hang or is using some resource that no longer exists. */
+static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
+{
+	BUG_ON(!node);
+	BUG_ON(I915_SQS_IS_COMPLETE(node));
+
+	node->status = i915_sqs_dead;
+}
+
+/*
+ * A sequence number has popped out of the hardware and the request handling
+ * code has mapped it back to a request and will mark that request complete.
+ * It also calls this function to notify the scheduler about the completion
+ * so the scheduler's node can be updated appropriately.
+ * Returns true if the request is scheduler managed, false if not.
+ */
+bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
+{
+	struct drm_i915_private *dev_priv  = to_i915(req->ring->dev);
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	/* XXX: Need to map back from request to node */
+	struct i915_scheduler_queue_entry *node = NULL;
+	unsigned long       flags;
+
+	if (!node)
+		return false;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	WARN_ON(!I915_SQS_IS_FLYING(node));
+
+	/* Node was in flight so mark it as complete. */
+	if (req->cancelled)
+		node->status = i915_sqs_dead;
+	else
+		node->status = i915_sqs_complete;
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	/*
+	 * XXX: If the in-flight list is now empty then new work should be
+	 * submitted. However, this function is called from interrupt context
+	 * and thus cannot acquire mutex locks and other such things that are
+	 * necessary for fresh submission.
+	 */
+
+	return true;
+}
+
+int i915_scheduler_remove(struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node, *node_next;
+	unsigned long       flags;
+	int                 flying = 0, queued = 0;
+	int                 ret = 0;
+	bool                do_submit;
+	uint32_t            min_seqno;
+	struct list_head    remove;
+
+	if (list_empty(&scheduler->node_queue[ring->id]))
+		return 0;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	/* /i915_scheduler_dump_locked(ring, "remove/pre");/ */
+
+	/*
+	 * In the case where the system is idle, starting 'min_seqno' from a big
+	 * number will cause all nodes to be removed as they are now back to
+	 * being in-order. However, this will be a problem if the last one to
+	 * complete was actually out-of-order as the ring seqno value will be
+	 * lower than one or more completed buffers. Thus code looking for the
+	 * completion of said buffers will wait forever.
+	 * Instead, use the hardware seqno as the starting point. This means
+	 * that some buffers might be kept around even in a completely idle
+	 * system but it should guarantee that no-one ever gets confused when
+	 * waiting for buffer completion.
+	 */
+	min_seqno = ring->get_seqno(ring, true);
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (I915_SQS_IS_QUEUED(node))
+			queued++;
+		else if (I915_SQS_IS_FLYING(node))
+			flying++;
+		else if (I915_SQS_IS_COMPLETE(node))
+			continue;
+
+		if (node->params.request->seqno == 0)
+			continue;
+
+		if (!i915_seqno_passed(node->params.request->seqno, min_seqno))
+			min_seqno = node->params.request->seqno;
+	}
+
+	INIT_LIST_HEAD(&remove);
+	list_for_each_entry_safe(node, node_next, &scheduler->node_queue[ring->id], link) {
+		/*
+		 * Only remove completed nodes which have a lower seqno than
+		 * all pending nodes. While there is the possibility of the
+		 * ring's seqno counting backwards, all higher buffers must
+		 * be remembered so that the 'i915_seqno_passed()' test can
+		 * report that they have in fact passed.
+		 *
+		 * NB: This is not true for 'dead' nodes. The GPU reset causes
+		 * the software seqno to restart from its initial value. Thus
+		 * the dead nodes must be removed even though their seqno values
+		 * are potentially vastly greater than the current ring seqno.
+		 */
+		if (!I915_SQS_IS_COMPLETE(node))
+			continue;
+
+		if (node->status != i915_sqs_dead) {
+			if (i915_seqno_passed(node->params.request->seqno, min_seqno) &&
+			    (node->params.request->seqno != min_seqno))
+				continue;
+		}
+
+		list_del(&node->link);
+		list_add(&node->link, &remove);
+
+		/* Strip the dependency info while the mutex is still locked */
+		i915_scheduler_remove_dependent(scheduler, node);
+
+		continue;
+	}
+
+	/*
+	 * No idea why but this seems to cause problems occasionally.
+	 * Note that the 'irq_put' code is internally reference counted
+	 * and spin_locked so it should be safe to call.
+	 */
+	/*if ((scheduler->flags[ring->id] & i915_sf_interrupts_enabled) &&
+	    (first_flight[ring->id] == NULL)) {
+		ring->irq_put(ring);
+		scheduler->flags[ring->id] &= ~i915_sf_interrupts_enabled;
+	}*/
+
+	/* Launch more packets now? */
+	do_submit = (queued > 0) && (flying < scheduler->min_flying);
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (!do_submit && list_empty(&remove))
+		return ret;
+
+	mutex_lock(&ring->dev->struct_mutex);
+
+	if (do_submit)
+		ret = i915_scheduler_submit(ring, true);
+
+	while (!list_empty(&remove)) {
+		node = list_first_entry(&remove, typeof(*node), link);
+		list_del(&node->link);
+
+		/* The batch buffer must be unpinned before it is unreferenced
+		 * otherwise the unpin fails with a missing vma!? */
+		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
+
+		/* Free everything that is owned by the node: */
+		i915_gem_request_unreference(node->params.request);
+		kfree(node->params.cliprects);
+		kfree(node->dep_list);
+		kfree(node);
+	}
+
+	mutex_unlock(&ring->dev->struct_mutex);
+
+	return ret;
+}
+
+static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
+{
+	struct i915_scheduler_queue_entry *node;
+	int i;
+
+	/*
+	 * Ensure circular dependencies don't cause problems and that a bump
+	 * by object usage only bumps each using buffer once:
+	 */
+	for (i = 0; i < I915_NUM_RINGS; i++) {
+		list_for_each_entry(node, &scheduler->node_queue[i], link)
+			node->bumped = false;
+	}
+}
+
+static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
+					struct i915_scheduler_queue_entry *target,
+					uint32_t bump)
+{
+	uint32_t new_priority;
+	int      i, count;
+
+	if (target->priority >= scheduler->priority_level_max)
+		return 1;
+
+	if (target->bumped)
+		return 0;
+
+	new_priority = target->priority + bump;
+	if ((new_priority <= target->priority) ||
+	    (new_priority > scheduler->priority_level_max))
+		target->priority = scheduler->priority_level_max;
+	else
+		target->priority = new_priority;
+
+	count = 1;
+	target->bumped = true;
+
+	for (i = 0; i < target->num_deps; i++) {
+		if (!target->dep_list[i])
+			continue;
+
+		if (target->dep_list[i]->bumped)
+			continue;
+
+		count += i915_scheduler_priority_bump(scheduler,
+						      target->dep_list[i],
+						      bump);
+	}
+
+	return count;
+}
+
+static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
+				    struct i915_scheduler_queue_entry **pop_node,
+				    unsigned long *flags)
+{
+	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *best;
+	struct i915_scheduler_queue_entry  *node;
+	int     ret;
+	int     i;
+	bool	any_queued;
+	bool	has_local, has_remote, only_remote;
+
+	*pop_node = NULL;
+	ret = -ENODATA;
+
+	any_queued = false;
+	only_remote = false;
+	best = NULL;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+		any_queued = true;
+
+		has_local  = false;
+		has_remote = false;
+		for (i = 0; i < node->num_deps; i++) {
+			if (!i915_scheduler_is_dependency_valid(node, i))
+				continue;
+
+			if (node->dep_list[i]->params.ring == node->params.ring)
+				has_local = true;
+			else
+				has_remote = true;
+		}
+
+		if (has_remote && !has_local)
+			only_remote = true;
+
+		if (!has_local && !has_remote) {
+			if (!best ||
+			    (node->priority > best->priority))
+				best = node;
+		}
+	}
+
+	if (best) {
+		list_del(&best->link);
+
+		INIT_LIST_HEAD(&best->link);
+		best->status  = i915_sqs_popped;
+
+		ret = 0;
+	} else {
+		/* Can only get here if:
+		 * (a) there are no buffers in the queue
+		 * (b) all queued buffers are dependent on other buffers
+		 *     e.g. on a buffer that is in flight on a different ring
+		 */
+		if (only_remote) {
+			/* The only dependent buffers are on another ring. */
+			ret = -EAGAIN;
+		} else if (any_queued) {
+			/* It seems that something has gone horribly wrong! */
+			DRM_ERROR("Broken dependency tracking on ring %d!\n",
+				  (int) ring->id);
+		}
+	}
+
+	/* i915_scheduler_dump_queue_pop(ring, best); */
+
+	*pop_node = best;
+	return ret;
+}
+
+static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
+{
+	struct drm_device   *dev = ring->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	unsigned long       flags;
+	int                 ret = 0, count = 0;
+
+	if (!was_locked) {
+		ret = i915_mutex_lock_interruptible(dev);
+		if (ret)
+			return ret;
+	}
+
+	BUG_ON(!mutex_is_locked(&dev->struct_mutex));
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	/* First time around, complain if anything unexpected occurs: */
+	ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
+	if (ret) {
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (!was_locked)
+			mutex_unlock(&dev->struct_mutex);
+
+		return ret;
+	}
+
+	do {
+		BUG_ON(!node);
+		BUG_ON(node->params.ring != ring);
+		BUG_ON(node->status != i915_sqs_popped);
+		count++;
+
+		/* The call to pop above will have removed the node from the
+		 * list. So add it back in and mark it as in flight. */
+		i915_scheduler_fly_node(node);
+
+		scheduler->flags[ring->id] |= i915_sf_submitting;
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+		ret = dev_priv->gt.execbuf_final(&node->params);
+		spin_lock_irqsave(&scheduler->lock, flags);
+		scheduler->flags[ring->id] &= ~i915_sf_submitting;
+
+		if (ret) {
+			int requeue = 1;
+
+			/* Oh dear! Either the node is broken or the ring is
+			 * busy. So need to kill the node or requeue it and try
+			 * again later as appropriate. */
+
+			switch (-ret) {
+			case ENODEV:
+			case ENOENT:
+				/* Fatal errors. Kill the node. */
+				requeue = -1;
+			break;
+
+			case EAGAIN:
+			case EBUSY:
+			case EIO:
+			case ENOMEM:
+			case ERESTARTSYS:
+			case EINTR:
+				/* Supposedly recoverable errors. */
+			break;
+
+			default:
+				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
+						 ring->name, ret);
+				/* Assume it is recoverable and hope for the best. */
+			break;
+			}
+
+			/* Check that the watchdog/reset code has not nuked
+			 * the node while we weren't looking: */
+			if (node->status == i915_sqs_dead)
+				requeue = 0;
+
+			if (requeue == 1) {
+				i915_scheduler_node_requeue(node);
+				/* No point spinning if the ring is currently
+				 * unavailable so just give up and come back
+				 * later. */
+				break;
+			} else if (requeue == -1)
+				i915_scheduler_node_kill(node);
+		}
+
+		/* Keep launching until the sky is sufficiently full. */
+		if (i915_scheduler_count_flying(scheduler, ring) >=
+						scheduler->min_flying)
+			break;
+
+		ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
+	} while (ret == 0);
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (!was_locked)
+		mutex_unlock(&dev->struct_mutex);
+
+	/* Don't complain about not being able to submit extra entries */
+	if (ret == -ENODATA)
+		ret = 0;
+
+	return (ret < 0) ? ret : count;
+}
+
+static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
+					   struct i915_scheduler_queue_entry *remove)
+{
+	struct i915_scheduler_queue_entry  *node;
+	int     i, r;
+	int     count = 0;
+
+	for (i = 0; i < remove->num_deps; i++)
+		if ((remove->dep_list[i]) &&
+		    (!I915_SQS_IS_COMPLETE(remove->dep_list[i])))
+			count++;
+	BUG_ON(count);
+
+	for (r = 0; r < I915_NUM_RINGS; r++) {
+		list_for_each_entry(node, &scheduler->node_queue[r], link) {
+			for (i = 0; i < node->num_deps; i++) {
+				if (node->dep_list[i] != remove)
+					continue;
+
+				node->dep_list[i] = NULL;
+			}
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
new file mode 100644
index 0000000..00dc7f3
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef _I915_SCHEDULER_H_
+#define _I915_SCHEDULER_H_
+
+enum i915_scheduler_queue_status {
+	/* Limbo: */
+	i915_sqs_none = 0,
+	/* Not yet submitted to hardware: */
+	i915_sqs_queued,
+	/* Popped from queue, ready to fly: */
+	i915_sqs_popped,
+	/* Sent to hardware for processing: */
+	i915_sqs_flying,
+	/* Finished processing on the hardware: */
+	i915_sqs_complete,
+	/* Killed by watchdog or catastrophic submission failure: */
+	i915_sqs_dead,
+	/* Limit value for use with arrays/loops */
+	i915_sqs_MAX
+};
+
+#define I915_SQS_IS_QUEUED(node)	(((node)->status == i915_sqs_queued))
+#define I915_SQS_IS_FLYING(node)	(((node)->status == i915_sqs_flying))
+#define I915_SQS_IS_COMPLETE(node)	(((node)->status == i915_sqs_complete) || \
+					 ((node)->status == i915_sqs_dead))
+
+struct i915_scheduler_obj_entry {
+	struct drm_i915_gem_object          *obj;
+	bool                                read_only;
+};
+
+struct i915_scheduler_queue_entry {
+	struct i915_execbuffer_params       params;
+	/* -1023 = lowest priority, 0 = default, 1023 = highest */
+	int32_t                             priority;
+	struct i915_scheduler_obj_entry     *saved_objects;
+	int                                 num_objs;
+	bool                                bumped;
+	struct i915_scheduler_queue_entry   **dep_list;
+	int                                 num_deps;
+	enum i915_scheduler_queue_status    status;
+	unsigned long                       stamp;
+	struct list_head                    link;
+};
+
+struct i915_scheduler {
+	struct list_head    node_queue[I915_NUM_RINGS];
+	uint32_t            flags[I915_NUM_RINGS];
+	spinlock_t          lock;
+
+	/* Tuning parameters: */
+	int32_t             priority_level_min;
+	int32_t             priority_level_max;
+	int32_t             priority_level_preempt;
+	uint32_t            min_flying;
+};
+
+/* Flag bits for i915_scheduler::flags */
+enum {
+	i915_sf_interrupts_enabled  = (1 << 0),
+	i915_sf_submitting          = (1 << 1),
+};
+
+int         i915_scheduler_init(struct drm_device *dev);
+int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
+bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
+
+#endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 12/40] drm/i915: Added scheduler hook when closing DRM file handles

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler decouples the submission of batch buffers to the driver
with submission of batch buffers to the hardware. Thus it is possible
for an application to close its DRM file handle while there is still
work outstanding. That means the scheduler needs to know about file
close events so it can remove the file pointer from such orphaned
batch buffers and not attempt to dereference it later.

v3: Updated to not wait for outstanding work to complete but merely
remove the file handle reference. The wait was getting excessively
complicated with inter-ring dependencies, pre-emption, and other such
issues.

Change-Id: I24ac056c062b075ff1cc5e2ed2d3fa8e17e85951
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_dma.c       |  3 +++
 drivers/gpu/drm/i915/i915_scheduler.c | 35 +++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  2 ++
 3 files changed, 40 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 731cf31..c2f9c03 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -46,6 +46,7 @@
 #include <linux/vga_switcheroo.h>
 #include <linux/slab.h>
 #include <acpi/video.h>
+#include "i915_scheduler.h"
 #include <linux/pm.h>
 #include <linux/pm_runtime.h>
 #include <linux/oom.h>
@@ -1250,6 +1251,8 @@ void i915_driver_lastclose(struct drm_device *dev)
 
 void i915_driver_preclose(struct drm_device *dev, struct drm_file *file)
 {
+	i915_scheduler_closefile(dev, file);
+
 	mutex_lock(&dev->struct_mutex);
 	i915_gem_context_close(dev, file);
 	i915_gem_release(dev, file);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 344760e..5aafc96 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -768,3 +768,38 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
 
 	return 0;
 }
+
+int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
+{
+	struct i915_scheduler_queue_entry  *node;
+	struct drm_i915_private            *dev_priv = dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring;
+	int                     i;
+	unsigned long           flags;
+
+	if (!scheduler)
+		return 0;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	for_each_ring(ring, dev_priv, i) {
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (node->params.file != file)
+				continue;
+
+			if(!I915_SQS_IS_COMPLETE(node))
+				DRM_DEBUG_DRIVER("Closing file handle with outstanding work: %d:%d/%d on %s\n",
+						 node->params.request->uniq,
+						 node->params.request->seqno,
+						 node->status,
+						 ring->name);
+
+			node->params.file = NULL;
+		}
+	}
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 2d50d83..02ac6f2 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -86,6 +86,8 @@ enum {
 
 bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
+int         i915_scheduler_closefile(struct drm_device *dev,
+				     struct drm_file *file);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 16/40] drm/i915: Keep the reserved space mechanism happy

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Ring space is reserved when constructing a request to ensure that the
subsequent 'add_request()' call cannot fail due to waiting for space
on a busy or broken GPU. However, the scheduler jumps in to the middle
of the execbuffer process between request creation and request
submission. Thus it needs to cancel the reserved space when the
request is simply added to the scheduler's queue and not yet
submitted. Similarly, it needs to re-reserve the space when it finally
does want to send the batch buffer to the hardware.

v3: Updated to use locally cached request pointer.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  7 +++++++
 drivers/gpu/drm/i915/i915_scheduler.c      |  4 ++++
 drivers/gpu/drm/i915/intel_lrc.c           | 13 +++++++++++--
 3 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index b5d618a..2c7a395 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1249,6 +1249,10 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	ret = intel_ring_reserve_space(req);
+	if (ret)
+		return ret;
+
 	intel_runtime_pm_get(dev_priv);
 
 	/*
@@ -1309,6 +1313,9 @@ error:
 	 */
 	intel_runtime_pm_put(dev_priv);
 
+	if (ret)
+		intel_ring_reserved_space_cancel(req->ringbuf);
+
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 0e657cf..9d1475f 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -145,6 +145,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
 		int ret;
 
+		intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
+
 		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
 		ret = dev_priv->gt.execbuf_final(&qe->params);
 		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
@@ -174,6 +176,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	node->stamp  = jiffies;
 	i915_gem_request_reference(node->params.request);
 
+	intel_ring_reserved_space_cancel(node->params.request->ringbuf);
+
 	BUG_ON(node->params.request->scheduler_qe);
 	node->params.request->scheduler_qe = node;
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index f14d9b2..ebc951e 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -934,13 +934,17 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	ret = intel_logical_ring_reserve_space(req);
+	if (ret)
+		return ret;
+
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
 	ret = logical_ring_invalidate_all_caches(req);
 	if (ret)
-		return ret;
+		goto err;
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
@@ -962,13 +966,18 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 
 	ret = ring->emit_bb_start(req, exec_start, params->dispatch_flags);
 	if (ret)
-		return ret;
+		goto err;
 
 	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
 	return 0;
+
+err:
+	intel_ring_reserved_space_cancel(params->request->ringbuf);
+
+	return ret;
 }
 
 void intel_execlists_retire_requests(struct intel_engine_cs *ring)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 17/40] drm/i915: Added tracking/locking of batch buffer objects

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler needs to track interdependencies between batch buffers.
These are calculated by analysing the object lists of the buffers and
looking for commonality. The scheduler also needs to keep those
buffers locked long after the initial IOCTL call has returned to user
land.

v3: Updated to support read-read optimisation.

Change-Id: I31e3677ecfc2c9b5a908bda6acc4850432d55f1e
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 52 ++++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.c      | 33 +++++++++++++++++--
 2 files changed, 80 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 2c7a395..0908699 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1418,7 +1418,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	struct i915_execbuffer_params *params = &qe.params;
 	const u32 ctx_id = i915_execbuffer2_get_context_id(*args);
 	u32 dispatch_flags;
-	int ret;
+	int ret, i;
 	bool need_relocs;
 	int fd_fence_complete = -1;
 	int fd_fence_wait = lower_32_bits(args->rsvd2);
@@ -1553,6 +1553,14 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		goto pre_mutex_err;
 	}
 
+	qe.saved_objects = kzalloc(
+			sizeof(*qe.saved_objects) * args->buffer_count,
+			GFP_KERNEL);
+	if (!qe.saved_objects) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
 	/* Look up object handles */
 	ret = eb_lookup_vmas(eb, exec, args, vm, file);
 	if (ret)
@@ -1673,7 +1681,30 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->args_DR1                = args->DR1;
 	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
-	params->ctx                     = ctx;
+
+	/*
+	 * Save away the list of objects used by this batch buffer for the
+	 * purpose of tracking inter-buffer dependencies.
+	 */
+	for (i = 0; i < args->buffer_count; i++) {
+		struct drm_i915_gem_object *obj;
+
+		/*
+		 * NB: 'drm_gem_object_lookup()' increments the object's
+		 * reference count and so must be matched by a
+		 * 'drm_gem_object_unreference' call.
+		 */
+		obj = to_intel_bo(drm_gem_object_lookup(dev, file,
+							  exec[i].handle));
+		qe.saved_objects[i].obj       = obj;
+		qe.saved_objects[i].read_only = obj->base.pending_write_domain == 0;
+
+	}
+	qe.num_objs = i;
+
+	/* Lock and save the context object as well. */
+	i915_gem_context_reference(ctx);
+	params->ctx = ctx;
 
 	if (args->flags & I915_EXEC_CREATE_FENCE) {
 		/*
@@ -1738,6 +1769,23 @@ err:
 	i915_gem_context_unreference(ctx);
 	eb_destroy(eb);
 
+	if (qe.saved_objects) {
+		/* Need to release the objects: */
+		for (i = 0; i < qe.num_objs; i++) {
+			if (!qe.saved_objects[i].obj)
+				continue;
+
+			drm_gem_object_unreference(
+					&qe.saved_objects[i].obj->base);
+		}
+
+		kfree(qe.saved_objects);
+
+		/* Context too */
+		if (params->ctx)
+			i915_gem_context_unreference(params->ctx);
+	}
+
 	/*
 	 * If the request was created but not successfully submitted then it
 	 * must be freed again. If it was submitted then it is being tracked
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 9d1475f..300cd89 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -158,7 +158,23 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		if (ret)
 			return ret;
 
-		/* Free everything that is owned by the QE structure: */
+		/* Need to release the objects: */
+		for (i = 0; i < qe->num_objs; i++) {
+			if (!qe->saved_objects[i].obj)
+				continue;
+
+			drm_gem_object_unreference(&qe->saved_objects[i].obj->base);
+		}
+
+		kfree(qe->saved_objects);
+		qe->saved_objects = NULL;
+		qe->num_objs = 0;
+
+		/* Free the context object too: */
+		if (qe->params.ctx)
+			i915_gem_context_unreference(qe->params.ctx);
+
+		/* And anything else owned by the QE structure: */
 		kfree(qe->params.cliprects);
 		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
 			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
@@ -399,7 +415,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	int                 flying = 0, queued = 0;
 	int                 ret = 0;
 	bool                do_submit;
-	uint32_t            min_seqno;
+	uint32_t            i, min_seqno;
 	struct list_head    remove;
 
 	if (list_empty(&scheduler->node_queue[ring->id]))
@@ -503,7 +519,18 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
 			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
 
-		/* Free everything that is owned by the node: */
+		/* Release the locked buffers: */
+		for (i = 0; i < node->num_objs; i++) {
+			drm_gem_object_unreference(
+					    &node->saved_objects[i].obj->base);
+		}
+		kfree(node->saved_objects);
+
+		/* Context too: */
+		if (node->params.ctx)
+			i915_gem_context_unreference(node->params.ctx);
+
+		/* And anything else owned by the node: */
 		node->params.request->scheduler_qe = NULL;
 		i915_gem_request_unreference(node->params.request);
 		kfree(node->params.cliprects);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 18/40] drm/i915: Hook scheduler node clean up into retire requests

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler keeps its own lock on various DRM objects in order to
guarantee safe access long after the original execbuff IOCTL has
completed. This is especially important when pre-emption is enabled as
the batch buffer might need to be submitted to the hardware multiple
times. This patch hooks the clean up of these locks into the request
retire function. The request can only be retired after it has
completed on the hardware and thus is no longer eligible for
re-submission. Thus there is no point holding on to the locks beyond
that time.

v3: Updated to not WARN when cleaning a node that is being cancelled.
The clean will happen later so skipping it at the point of
cancellation is fine.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c       |  3 ++
 drivers/gpu/drm/i915/i915_scheduler.c | 54 ++++++++++++++++++++++++-----------
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 3 files changed, 42 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index dc5f3fe..349ff58 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1402,6 +1402,9 @@ static void i915_gem_request_retire(struct drm_i915_gem_request *request)
 		fence_signal_locked(&request->fence);
 	}
 
+	if (request->scheduler_qe)
+		i915_gem_scheduler_clean_node(request->scheduler_qe);
+
 	i915_gem_request_unreference(request);
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 300cd89..f88c871 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -406,6 +406,41 @@ void i915_scheduler_wakeup(struct drm_device *dev)
 	queue_work(dev_priv->wq, &dev_priv->mm.scheduler_work);
 }
 
+void i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *node)
+{
+	uint32_t i;
+
+	if (!I915_SQS_IS_COMPLETE(node)) {
+		WARN(!node->params.request->cancelled,
+		     "Cleaning active node: %d!\n", node->status);
+		return;
+	}
+
+	if (node->params.batch_obj) {
+		/* The batch buffer must be unpinned before it is unreferenced
+		 * otherwise the unpin fails with a missing vma!? */
+		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
+
+		node->params.batch_obj = NULL;
+	}
+
+	/* Release the locked buffers: */
+	for (i = 0; i < node->num_objs; i++) {
+		drm_gem_object_unreference(
+				    &node->saved_objects[i].obj->base);
+	}
+	kfree(node->saved_objects);
+	node->saved_objects = NULL;
+	node->num_objs = 0;
+
+	/* Context too: */
+	if (node->params.ctx) {
+		i915_gem_context_unreference(node->params.ctx);
+		node->params.ctx = NULL;
+	}
+}
+
 static int i915_scheduler_remove(struct intel_engine_cs *ring)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
@@ -415,7 +450,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	int                 flying = 0, queued = 0;
 	int                 ret = 0;
 	bool                do_submit;
-	uint32_t            i, min_seqno;
+	uint32_t            min_seqno;
 	struct list_head    remove;
 
 	if (list_empty(&scheduler->node_queue[ring->id]))
@@ -514,21 +549,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		node = list_first_entry(&remove, typeof(*node), link);
 		list_del(&node->link);
 
-		/* The batch buffer must be unpinned before it is unreferenced
-		 * otherwise the unpin fails with a missing vma!? */
-		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
-			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
-
-		/* Release the locked buffers: */
-		for (i = 0; i < node->num_objs; i++) {
-			drm_gem_object_unreference(
-					    &node->saved_objects[i].obj->base);
-		}
-		kfree(node->saved_objects);
-
-		/* Context too: */
-		if (node->params.ctx)
-			i915_gem_context_unreference(node->params.ctx);
+		/* Free up all the DRM object references */
+		i915_gem_scheduler_clean_node(node);
 
 		/* And anything else owned by the node: */
 		node->params.request->scheduler_qe = NULL;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 56f68e5..54d87fb 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -88,6 +88,7 @@ bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
 int         i915_scheduler_closefile(struct drm_device *dev,
 				     struct drm_file *file);
+void        i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 19/40] drm/i915: Added scheduler support to __wait_request() calls

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler can cause batch buffers, and hence requests, to be
submitted to the ring out of order and asynchronously to their
submission to the driver. Thus at the point of waiting for the
completion of a given request, it is not even guaranteed that the
request has actually been sent to the hardware yet. Even it is has
been sent, it is possible that it could be pre-empted and thus
'unsent'.

This means that it is necessary to be able to submit requests to the
hardware during the wait call itself. Unfortunately, while some
callers of __wait_request() release the mutex lock first, others do
not (and apparently can not). Hence there is the ability to deadlock
as the wait stalls for submission but the asynchronous submission is
stalled for the mutex lock.

This change hooks the scheduler in to the __wait_request() code to
ensure correct behaviour. That is, flush the target batch buffer
through to the hardware and do not deadlock waiting for something that
cannot currently be submitted. Instead, the wait call must return
EAGAIN at least as far back as necessary to release the mutex lock and
allow the scheduler's asynchronous processing to get in and handle the
pre-emption operation and eventually (re-)submit the work.

v3: Removed the explicit scheduler flush from i915_wait_request().
This is no longer necessary and was causing unintended changes to the
scheduler priority level which broke a validation team test.

Change-Id: I31fe6bc7e38f6ffdd843fcae16e7cc8b1e52a931
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h         |  3 ++-
 drivers/gpu/drm/i915/i915_gem.c         | 33 ++++++++++++++++++++++++++-------
 drivers/gpu/drm/i915/i915_scheduler.c   | 20 ++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h   |  2 ++
 drivers/gpu/drm/i915/intel_display.c    |  5 +++--
 drivers/gpu/drm/i915/intel_ringbuffer.c |  2 +-
 6 files changed, 54 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 9a67f7c..5ed600c 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -3029,7 +3029,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			unsigned reset_counter,
 			bool interruptible,
 			s64 *timeout,
-			struct intel_rps_client *rps);
+			struct intel_rps_client *rps,
+			bool is_locked);
 int __must_check i915_wait_request(struct drm_i915_gem_request *req);
 int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf);
 int __must_check
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 349ff58..784000b 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1207,7 +1207,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			unsigned reset_counter,
 			bool interruptible,
 			s64 *timeout,
-			struct intel_rps_client *rps)
+			struct intel_rps_client *rps,
+			bool is_locked)
 {
 	struct intel_engine_cs *ring = i915_gem_request_get_ring(req);
 	struct drm_device *dev = ring->dev;
@@ -1217,8 +1218,10 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 	DEFINE_WAIT(wait);
 	unsigned long timeout_expire;
 	s64 before, now;
-	int ret;
+	int ret = 0;
+	bool    busy;
 
+	might_sleep();
 	WARN(!intel_irqs_enabled(dev_priv), "IRQs disabled");
 
 	if (i915_gem_request_completed(req))
@@ -1269,6 +1272,22 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			break;
 		}
 
+		if (is_locked) {
+			/* If this request is being processed by the scheduler
+			 * then it is unsafe to sleep with the mutex lock held
+			 * as the scheduler may require the lock in order to
+			 * progress the request. */
+			if (i915_scheduler_is_request_tracked(req, NULL, &busy)) {
+				if (busy) {
+					ret = -EAGAIN;
+					break;
+				}
+			}
+
+			/* If the request is not tracked by the scheduler then the
+			 * regular test can be done. */
+		}
+
 		if (i915_gem_request_completed(req)) {
 			ret = 0;
 			break;
@@ -1455,7 +1474,7 @@ i915_wait_request(struct drm_i915_gem_request *req)
 
 	ret = __i915_wait_request(req,
 				  atomic_read(&dev_priv->gpu_error.reset_counter),
-				  interruptible, NULL, NULL);
+				  interruptible, NULL, NULL, true);
 	if (ret)
 		return ret;
 
@@ -1568,7 +1587,7 @@ i915_gem_object_wait_rendering__nonblocking(struct drm_i915_gem_object *obj,
 	mutex_unlock(&dev->struct_mutex);
 	for (i = 0; ret == 0 && i < n; i++)
 		ret = __i915_wait_request(requests[i], reset_counter, true,
-					  NULL, rps);
+					  NULL, rps, false);
 	mutex_lock(&dev->struct_mutex);
 
 	for (i = 0; i < n; i++) {
@@ -3494,7 +3513,7 @@ i915_gem_wait_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 		if (ret == 0)
 			ret = __i915_wait_request(req[i], reset_counter, true,
 						  args->timeout_ns > 0 ? &args->timeout_ns : NULL,
-						  file->driver_priv);
+						  file->driver_priv, false);
 		i915_gem_request_unreference(req[i]);
 	}
 	return ret;
@@ -3527,7 +3546,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 					  atomic_read(&i915->gpu_error.reset_counter),
 					  i915->mm.interruptible,
 					  NULL,
-					  &i915->rps.semaphores);
+					  &i915->rps.semaphores, true);
 		if (ret)
 			return ret;
 
@@ -4486,7 +4505,7 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	if (target == NULL)
 		return 0;
 
-	ret = __i915_wait_request(target, reset_counter, true, NULL, NULL);
+	ret = __i915_wait_request(target, reset_counter, true, NULL, NULL, false);
 	if (ret == 0)
 		queue_delayed_work(dev_priv->wq, &dev_priv->mm.retire_work, 0);
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index f88c871..386f157 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -847,6 +847,26 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
 	return 0;
 }
 
+bool i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
+				       bool *completed, bool *busy)
+{
+	struct drm_i915_private *dev_priv = req->ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return false;
+
+	if (req->scheduler_qe == NULL)
+		return false;
+
+	if (completed)
+		*completed = I915_SQS_IS_COMPLETE(req->scheduler_qe);
+	if (busy)
+		*busy      = I915_SQS_IS_QUEUED(req->scheduler_qe);
+
+	return true;
+}
+
 int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 {
 	struct i915_scheduler_queue_entry  *node;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 54d87fb..c3e7ac6 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -93,5 +93,7 @@ int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *q
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
 void        i915_gem_scheduler_work_handler(struct work_struct *work);
+bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
+					      bool *completed, bool *busy);
 
 #endif  /* _I915_SCHEDULER_H_ */
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index c00bc50..58c464f 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -11258,7 +11258,8 @@ static void intel_mmio_flip_work_func(struct work_struct *work)
 		WARN_ON(__i915_wait_request(mmio_flip->req,
 					    mmio_flip->crtc->reset_counter,
 					    false, NULL,
-					    &mmio_flip->i915->rps.mmioflips));
+					    &mmio_flip->i915->rps.mmioflips,
+					    false));
 		i915_gem_request_unreference(mmio_flip->req);
 	}
 
@@ -13258,7 +13259,7 @@ static int intel_atomic_prepare_commit(struct drm_device *dev,
 
 			ret = __i915_wait_request(intel_plane_state->wait_req,
 						  reset_counter, true,
-						  NULL, NULL);
+						  NULL, NULL, false);
 
 			/* Swallow -EIO errors to allow updates during hw lockup. */
 			if (ret == -EIO)
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 34fbe68..4d77886 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -2294,7 +2294,7 @@ int intel_ring_idle(struct intel_engine_cs *ring)
 	return __i915_wait_request(req,
 				   atomic_read(&to_i915(ring->dev)->gpu_error.reset_counter),
 				   to_i915(ring->dev)->mm.interruptible,
-				   NULL, NULL);
+				   NULL, NULL, true);
 }
 
 int intel_ring_alloc_request_extras(struct drm_i915_gem_request *request)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 21/40] drm/i915: Added scheduler flush calls to ring throttle and idle functions

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When requesting that all GPU work is completed, it is now necessary to
get the scheduler involved in order to flush out work that queued and
not yet submitted.

v2: Updated to add support for flushing the scheduler queue by time
stamp rather than just doing a blanket flush.

v3: Moved submit_max_priority() to this patch from an earlier patch
is it is no longer required in the other.

Change-Id: I95dcc2a2ee5c1a844748621c333994ddd6cf6a66
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c       |  24 ++++++-
 drivers/gpu/drm/i915/i915_scheduler.c | 132 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |   3 +
 3 files changed, 158 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 1a05c97..541ed9a 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3765,6 +3765,10 @@ int i915_gpu_idle(struct drm_device *dev)
 
 	/* Flush everything onto the inactive list. */
 	for_each_ring(ring, dev_priv, i) {
+		ret = i915_scheduler_flush(ring, true);
+		if (ret < 0)
+			return ret;
+
 		if (!i915.enable_execlists) {
 			struct drm_i915_gem_request *req;
 
@@ -4478,7 +4482,8 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	unsigned long recent_enough = jiffies - DRM_I915_THROTTLE_JIFFIES;
 	struct drm_i915_gem_request *request, *target = NULL;
 	unsigned reset_counter;
-	int ret;
+	int i, ret;
+	struct intel_engine_cs *ring;
 
 	ret = i915_gem_wait_for_error(&dev_priv->gpu_error);
 	if (ret)
@@ -4488,6 +4493,23 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	if (ret)
 		return ret;
 
+	for_each_ring(ring, dev_priv, i) {
+		/*
+		 * Flush out scheduler entries that are getting 'stale'. Note
+		 * that the following recent_enough test will only check
+		 * against the time at which the request was submitted to the
+		 * hardware (i.e. when it left the scheduler) not the time it
+		 * was submitted to the driver.
+		 *
+		 * Also, there is not much point worring about busy return
+		 * codes from the scheduler flush call. Even if more work
+		 * cannot be submitted right now for whatever reason, we
+		 * still want to throttle against stale work that has already
+		 * been submitted.
+		 */
+		i915_scheduler_flush_stamp(ring, recent_enough, false);
+	}
+
 	spin_lock(&file_priv->mm.lock);
 	list_for_each_entry(request, &file_priv->mm.request_list, client_list) {
 		if (time_after_eq(request->emitted_jiffies, recent_enough))
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 386f157..c13dbc3 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -31,6 +31,8 @@ static int         i915_scheduler_remove_dependent(struct i915_scheduler *schedu
 						   struct i915_scheduler_queue_entry *remove);
 static int         i915_scheduler_submit(struct intel_engine_cs *ring,
 					 bool is_locked);
+static int         i915_scheduler_submit_max_priority(struct intel_engine_cs *ring,
+						      bool is_locked);
 static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
 					       struct intel_engine_cs *ring);
 static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
@@ -580,6 +582,98 @@ void i915_gem_scheduler_work_handler(struct work_struct *work)
 	}
 }
 
+int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
+			       unsigned long target,
+			       bool is_locked)
+{
+	struct i915_scheduler_queue_entry *node;
+	struct drm_i915_private           *dev_priv;
+	struct i915_scheduler             *scheduler;
+	unsigned long       flags;
+	int                 flush_count = 0;
+
+	if (!ring)
+		return -EINVAL;
+
+	dev_priv  = ring->dev->dev_private;
+	scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return 0;
+
+	if (is_locked && (scheduler->flags[ring->id] & i915_sf_submitting)) {
+		/* Scheduler is busy already submitting another batch,
+		 * come back later rather than going recursive... */
+		return -EAGAIN;
+	}
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	i915_scheduler_priority_bump_clear(scheduler);
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+
+		if (node->stamp > target)
+			continue;
+
+		flush_count = i915_scheduler_priority_bump(scheduler,
+					node, scheduler->priority_level_max);
+	}
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (flush_count) {
+		DRM_DEBUG_DRIVER("<%s> Bumped %d entries\n", ring->name, flush_count);
+		flush_count = i915_scheduler_submit_max_priority(ring, is_locked);
+	}
+
+	return flush_count;
+}
+
+int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
+{
+	struct i915_scheduler_queue_entry *node;
+	struct drm_i915_private           *dev_priv;
+	struct i915_scheduler             *scheduler;
+	unsigned long       flags;
+	bool        found;
+	int         ret;
+	uint32_t    count = 0;
+
+	if (!ring)
+		return -EINVAL;
+
+	dev_priv  = ring->dev->dev_private;
+	scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return 0;
+
+	BUG_ON(is_locked && (scheduler->flags[ring->id] & i915_sf_submitting));
+
+	do {
+		found = false;
+		spin_lock_irqsave(&scheduler->lock, flags);
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (!I915_SQS_IS_QUEUED(node))
+				continue;
+
+			found = true;
+			break;
+		}
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (found) {
+			ret = i915_scheduler_submit(ring, is_locked);
+			if (ret < 0)
+				return ret;
+
+			count += ret;
+		}
+	} while (found);
+
+	return count;
+}
+
 static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
 {
 	struct i915_scheduler_queue_entry *node;
@@ -633,6 +727,44 @@ static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 	return count;
 }
 
+static int i915_scheduler_submit_max_priority(struct intel_engine_cs *ring,
+					      bool is_locked)
+{
+	struct i915_scheduler_queue_entry  *node;
+	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	unsigned long	flags;
+	int             ret, count = 0;
+	bool            found;
+
+	do {
+		found = false;
+		spin_lock_irqsave(&scheduler->lock, flags);
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (!I915_SQS_IS_QUEUED(node))
+				continue;
+
+			if (node->priority < scheduler->priority_level_max)
+				continue;
+
+			found = true;
+			break;
+		}
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (!found)
+			break;
+
+		ret = i915_scheduler_submit(ring, is_locked);
+		if (ret < 0)
+			return ret;
+
+		count += ret;
+	} while (found);
+
+	return count;
+}
+
 static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 				    struct i915_scheduler_queue_entry **pop_node,
 				    unsigned long *flags)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index c3e7ac6..7654013 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -93,6 +93,9 @@ int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *q
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
 void        i915_gem_scheduler_work_handler(struct work_struct *work);
+int         i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked);
+int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
+				       unsigned long stamp, bool is_locked);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 24/40] drm/i915: Defer seqno allocation until actual hardware submission time

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The seqno value is now only used for the final test for completion of
a request. It is no longer used to track the request through the
software stack. Thus it is no longer necessary to allocate the seqno
immediately with the request. Instead, it can be done lazily and left
until the request is actually sent to the hardware. This is particular
advantageous with a GPU scheduler as the requests can then be
re-ordered between their creation and their hardware submission
without having out of order seqnos.

v2: i915_add_request() can't fail!

Combine with 'drm/i915: Assign seqno at start of exec_final()'
Various bits of code during the execbuf code path need a seqno value
to be assigned to the request. This change makes this assignment
explicit at the start of submission_final() rather than relying on an
auto-generated seqno to have happened already. This is in preparation
for a future patch which changes seqno values to be assigned lazily
(during add_request).

v3: Updated to use locally cached request pointer.

Change-Id: I0d922b84c517611a79fa6c2b9e730d4fe3671d6a
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  1 +
 drivers/gpu/drm/i915/i915_gem.c            | 21 ++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++++++++
 drivers/gpu/drm/i915/intel_lrc.c           | 13 +++++++++++++
 4 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 5b893a6..15dee41 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2225,6 +2225,7 @@ struct drm_i915_gem_request {
 
 	/** GEM sequence number associated with this request. */
 	uint32_t seqno;
+	uint32_t reserved_seqno;
 
 	/* Unique identifier which can be used for trace points & debug */
 	uint32_t uniq;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 99e5b1d0..1fb45c2 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2525,6 +2525,9 @@ i915_gem_get_seqno(struct drm_device *dev, u32 *seqno)
 
 	/* reserve 0 for non-seqno */
 	if (dev_priv->next_seqno == 0) {
+		/* Why is the full re-initialisation required? Is it only for
+		 * hardware semaphores? If so, could skip it in the case where
+		 * semaphores are disabled? */
 		int ret = i915_gem_init_seqno(dev, 0);
 		if (ret)
 			return ret;
@@ -2582,6 +2585,12 @@ void __i915_add_request(struct drm_i915_gem_request *request,
 		WARN(ret, "*_ring_flush_all_caches failed: %d!\n", ret);
 	}
 
+	/* Make the request's seqno 'live': */
+	if(!request->seqno) {
+		request->seqno = request->reserved_seqno;
+		WARN_ON(request->seqno != dev_priv->last_seqno);
+	}
+
 	/* Record the position of the start of the request so that
 	 * should we detect the updated seqno part-way through the
 	 * GPU processing the request, we never over-estimate the
@@ -2830,6 +2839,9 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 
 	list_for_each_entry_safe(req, req_next, &ring->fence_signal_list, signal_link) {
 		if (!req->cancelled) {
+			/* How can this happen? */
+			WARN_ON(req->seqno == 0);
+
 			if (!i915_seqno_passed(seqno, req->seqno))
 				break;
 		}
@@ -3054,7 +3066,14 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring,
 	if (req == NULL)
 		return -ENOMEM;
 
-	ret = i915_gem_get_seqno(ring->dev, &req->seqno);
+	/*
+	 * Assign an identifier to track this request through the hardware
+	 * but don't make it live yet. It could change in the future if this
+	 * request gets overtaken. However, it still needs to be allocated
+	 * in advance because the point of submission must not fail and seqno
+	 * allocation can fail.
+	 */
+	ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
 	if (ret)
 		goto err;
 
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 0908699..7970958 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1249,6 +1249,19 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Make sure the request's seqno is the latest and greatest: */
+	if(req->reserved_seqno != dev_priv->last_seqno) {
+		ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
+		if (ret)
+			return ret;
+	}
+	/*
+	 * And make it live because some of the execbuff submission code
+	 * requires the seqno to be available up front. */
+	WARN_ON(req->seqno);
+	req->seqno = req->reserved_seqno;
+	WARN_ON(req->seqno != dev_priv->last_seqno);
+
 	ret = intel_ring_reserve_space(req);
 	if (ret)
 		return ret;
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 85bef51..d187f9a 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -934,6 +934,19 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Make sure the request's seqno is the latest and greatest: */
+	if(req->reserved_seqno != dev_priv->last_seqno) {
+		ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
+		if (ret)
+			return ret;
+	}
+	/*
+	 * And make it live because some of the execbuff submission code
+	 * requires the seqno to be available up front. */
+	WARN_ON(req->seqno);
+	req->seqno = req->reserved_seqno;
+	WARN_ON(req->seqno != dev_priv->last_seqno);
+
 	ret = intel_logical_ring_reserve_space(req);
 	if (ret)
 		return ret;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 28/40] drm/i915: Added trace points to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Added trace points to the scheduler to track all the various events,
node state transitions and other interesting things that occur.

v2: Updated for new request completion tracking implementation.

v3: Updated for changes to node kill code.

Change-Id: I9886390cfc7897bc1faf50a104bc651d8baed8a5
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   2 +
 drivers/gpu/drm/i915/i915_scheduler.c      |  26 ++++
 drivers/gpu/drm/i915/i915_trace.h          | 190 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_lrc.c           |   2 +
 4 files changed, 220 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index fdaede3..b358b21 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1226,6 +1226,8 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
+	trace_i915_gem_ring_queue(ring, params);
+
 	qe = container_of(params, typeof(*qe), params);
 	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 39aa702..4736f0f 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -151,6 +151,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
+		trace_i915_scheduler_queue(qe->params.ring, qe);
+
 		WARN_ON(qe->params.fence_wait &&
 			(!sync_fence_is_signaled(qe->params.fence_wait)));
 
@@ -271,6 +273,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		not_flying = i915_scheduler_count_flying(scheduler, ring) <
 							 scheduler->min_flying;
 
+	trace_i915_scheduler_queue(ring, node);
+	trace_i915_scheduler_node_state_change(ring, node);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (not_flying)
@@ -298,6 +303,9 @@ static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
 
 	node->status = i915_sqs_flying;
 
+	trace_i915_scheduler_fly(ring, node);
+	trace_i915_scheduler_node_state_change(ring, node);
+
 	if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
 		bool    success = true;
 
@@ -363,6 +371,8 @@ static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
 
 	node->status = i915_sqs_queued;
 	node->params.request->seqno = 0;
+	trace_i915_scheduler_unfly(node->params.ring, node);
+	trace_i915_scheduler_node_state_change(node->params.ring, node);
 }
 
 /* Give up on a node completely. For example, because it is causing the
@@ -372,7 +382,11 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
 	BUG_ON(!node);
 	BUG_ON(I915_SQS_IS_COMPLETE(node));
 
+	if (I915_SQS_IS_FLYING(node))
+		trace_i915_scheduler_unfly(node->params.ring, node);
+
 	node->status = i915_sqs_dead;
+	trace_i915_scheduler_node_state_change(node->params.ring, node);
 }
 
 /*
@@ -392,6 +406,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	struct i915_scheduler_queue_entry *node = req->scheduler_qe;
 	unsigned long       flags;
 
+	trace_i915_scheduler_landing(req);
+
 	if (!node)
 		return false;
 
@@ -405,6 +421,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	else
 		node->status = i915_sqs_complete;
 
+	trace_i915_scheduler_node_state_change(req->ring, node);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	return true;
@@ -550,6 +568,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	/* Launch more packets now? */
 	do_submit = (queued > 0) && (flying < scheduler->min_flying);
 
+	trace_i915_scheduler_remove(ring, min_seqno, do_submit);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (!do_submit && list_empty(&remove))
@@ -564,6 +584,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		node = list_first_entry(&remove, typeof(*node), link);
 		list_del(&node->link);
 
+		trace_i915_scheduler_destroy(ring, node);
+
 		if (node->params.fence_wait)
 			sync_fence_put(node->params.fence_wait);
 
@@ -927,6 +949,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		INIT_LIST_HEAD(&best->link);
 		best->status  = i915_sqs_popped;
 
+		trace_i915_scheduler_node_state_change(ring, best);
+
 		ret = 0;
 	} else {
 		/* Can only get here if:
@@ -980,6 +1004,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		spin_lock_irqsave(&scheduler->lock, *flags);
 	}
 
+	trace_i915_scheduler_pop_from_queue(ring, best);
+
 	*pop_node = best;
 	return ret;
 }
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index 76b08fa..73b0ee9 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -9,6 +9,7 @@
 #include "i915_drv.h"
 #include "intel_drv.h"
 #include "intel_ringbuffer.h"
+#include "i915_scheduler.h"
 
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM i915
@@ -822,6 +823,195 @@ TRACE_EVENT(switch_mm,
 		  __entry->dev, __entry->ring, __entry->to, __entry->vm)
 );
 
+TRACE_EVENT(i915_scheduler_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_fly,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_unfly,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_landing,
+	    TP_PROTO(struct drm_i915_gem_request *req),
+	    TP_ARGS(req),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     __field(u32, status)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = req->ring->id;
+			   __entry->uniq   = req->uniq;
+			   __entry->seqno  = req->seqno;
+			   __entry->status = req->scheduler_qe ? req->scheduler_qe->status : ~0U;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d, status=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno, __entry->status)
+);
+
+TRACE_EVENT(i915_scheduler_remove,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     u32 min_seqno, bool do_submit),
+	    TP_ARGS(ring, min_seqno, do_submit),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, min_seqno)
+			     __field(bool, do_submit)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring      = ring->id;
+			   __entry->min_seqno = min_seqno;
+			   __entry->do_submit = do_submit;
+			   ),
+
+	    TP_printk("ring=%d, min_seqno = %d, do_submit=%d",
+		      __entry->ring, __entry->min_seqno, __entry->do_submit)
+);
+
+TRACE_EVENT(i915_scheduler_destroy,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_pop_from_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_node_state_change,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     __field(u32, status)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = ring->id;
+			   __entry->uniq   = node ? node->params.request->uniq  : 0;
+			   __entry->seqno  = node->params.request->seqno;
+			   __entry->status = node->status;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d, status=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno, __entry->status)
+);
+
+TRACE_EVENT(i915_gem_ring_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_execbuffer_params *params),
+	    TP_ARGS(ring, params),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = ring->id;
+			   __entry->seqno  = params->request->seqno;
+			   ),
+
+	    TP_printk("ring=%d, seqno=%d", __entry->ring, __entry->seqno)
+);
+
 #endif /* _I915_TRACE_H_ */
 
 /* This part must be outside protection */
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index d187f9a..1fa3228 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -910,6 +910,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
+	trace_i915_gem_ring_queue(ring, params);
+
 	qe = container_of(params, typeof(*qe), params);
 	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 29/40] drm/i915: Added scheduler queue throttling by DRM file handle

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler decouples the submission of batch buffers to the driver
from their subsequent submission to the hardware. This means that an
application which is continuously submitting buffers as fast as it can
could potentialy flood the driver. To prevent this, the driver now
tracks how many buffers are in progress (queued in software or
executing in hardware) and limits this to a given (tunable) number. If
this number is exceeded then the queue to the driver will return
EAGAIN and thus prevent the scheduler's queue becoming arbitrarily
large.

v3: Added a missing decrement of the file queue counter.

Change-Id: I83258240aec7c810db08c006a3062d46aa91363f
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  2 ++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  8 +++++++
 drivers/gpu/drm/i915/i915_scheduler.c      | 35 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |  2 ++
 4 files changed, 47 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 4187e75..4ecb6e4 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -342,6 +342,8 @@ struct drm_i915_file_private {
 	} rps;
 
 	struct intel_engine_cs *bsd_ring;
+
+	u32 scheduler_queue_length;
 };
 
 enum intel_dpll_id {
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index b358b21..8ba426f 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1862,6 +1862,10 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_is_full(file))
+		return -EAGAIN;
+
 	/* Copy in the exec list from userland */
 	exec_list = drm_malloc_ab(sizeof(*exec_list), args->buffer_count);
 	exec2_list = drm_malloc_ab(sizeof(*exec2_list), args->buffer_count);
@@ -1945,6 +1949,10 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_is_full(file))
+		return -EAGAIN;
+
 	exec2_list = kmalloc(sizeof(*exec2_list)*args->buffer_count,
 			     GFP_TEMPORARY | __GFP_NOWARN | __GFP_NORETRY);
 	if (exec2_list == NULL)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 4736f0f..e6e1bd967 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -40,6 +40,8 @@ static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *sch
 static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 						struct i915_scheduler_queue_entry *target,
 						uint32_t bump);
+static void        i915_scheduler_file_queue_inc(struct drm_file *file);
+static void        i915_scheduler_file_queue_dec(struct drm_file *file);
 
 bool i915_scheduler_is_enabled(struct drm_device *dev)
 {
@@ -74,6 +76,7 @@ int i915_scheduler_init(struct drm_device *dev)
 	scheduler->priority_level_max     = 1023;
 	scheduler->priority_level_preempt = 900;
 	scheduler->min_flying             = 2;
+	scheduler->file_queue_max         = 64;
 
 	dev_priv->scheduler = scheduler;
 
@@ -267,6 +270,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
 
+	i915_scheduler_file_queue_inc(node->params.file);
+
 	if (i915.scheduler_override & i915_so_submit_on_queue)
 		not_flying = true;
 	else
@@ -551,6 +556,12 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		/* Strip the dependency info while the mutex is still locked */
 		i915_scheduler_remove_dependent(scheduler, node);
 
+		/* Likewise clean up the file descriptor before it might disappear. */
+		if (node->params.file) {
+			i915_scheduler_file_queue_dec(node->params.file);
+			node->params.file = NULL;
+		}
+
 		continue;
 	}
 
@@ -1194,6 +1205,7 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 						 node->status,
 						 ring->name);
 
+			i915_scheduler_file_queue_dec(node->params.file);
 			node->params.file = NULL;
 		}
 	}
@@ -1202,3 +1214,26 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 
 	return 0;
 }
+
+bool i915_scheduler_file_queue_is_full(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+	struct drm_i915_private      *dev_priv  = file_priv->dev_priv;
+	struct i915_scheduler        *scheduler = dev_priv->scheduler;
+
+	return file_priv->scheduler_queue_length >= scheduler->file_queue_max;
+}
+
+static void i915_scheduler_file_queue_inc(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+
+	file_priv->scheduler_queue_length++;
+}
+
+static void i915_scheduler_file_queue_dec(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+
+	file_priv->scheduler_queue_length--;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index b10f30c..5de0506 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -82,6 +82,7 @@ struct i915_scheduler {
 	int32_t             priority_level_max;
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
+	uint32_t            file_queue_max;
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -110,5 +111,6 @@ int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 				       unsigned long stamp, bool is_locked);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
+bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 31/40] drm/i915: Added debug state dump facilities to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When debugging batch buffer submission issues, it is useful to be able
to see what the current state of the scheduler is. This change adds
functions for decoding the internal scheduler state and reporting it.

v3: Updated a debug message with the new state_str() function.

Change-Id: I0634168e3f3465ff023f5a673165c90b07e535b6
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 280 +++++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_scheduler.h |  14 ++
 2 files changed, 292 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index e6e1bd967..be2430d 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -36,6 +36,9 @@ static int         i915_scheduler_submit_max_priority(struct intel_engine_cs *ri
 						      bool is_locked);
 static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
 					       struct intel_engine_cs *ring);
+static int         i915_scheduler_dump_locked(struct intel_engine_cs *ring,
+					      const char *msg);
+static int         i915_scheduler_dump_all_locked(struct drm_device *dev, const char *msg);
 static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
 static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 						struct i915_scheduler_queue_entry *target,
@@ -53,6 +56,116 @@ bool i915_scheduler_is_enabled(struct drm_device *dev)
 	return dev_priv->scheduler != NULL;
 }
 
+const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node)
+{
+	static char	str[50];
+	char		*ptr = str;
+
+	*(ptr++) = node->bumped ? 'B' : '-',
+	*(ptr++) = i915_gem_request_completed(node->params.request) ? 'C' : '-';
+
+	*ptr = 0;
+
+	return str;
+}
+
+char i915_scheduler_queue_status_chr(enum i915_scheduler_queue_status status)
+{
+	switch (status) {
+	case i915_sqs_none:
+	return 'N';
+
+	case i915_sqs_queued:
+	return 'Q';
+
+	case i915_sqs_popped:
+	return 'X';
+
+	case i915_sqs_flying:
+	return 'F';
+
+	case i915_sqs_complete:
+	return 'C';
+
+	case i915_sqs_dead:
+	return 'D';
+
+	default:
+	break;
+	}
+
+	return '?';
+}
+
+const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status)
+{
+	static char	str[50];
+
+	switch (status) {
+	case i915_sqs_none:
+	return "None";
+
+	case i915_sqs_queued:
+	return "Queued";
+
+	case i915_sqs_popped:
+	return "Popped";
+
+	case i915_sqs_flying:
+	return "Flying";
+
+	case i915_sqs_complete:
+	return "Complete";
+
+	case i915_sqs_dead:
+	return "Dead";
+
+	default:
+	break;
+	}
+
+	sprintf(str, "[Unknown_%d!]", status);
+	return str;
+}
+
+const char *i915_scheduler_flag_str(uint32_t flags)
+{
+	static char     str[100];
+	char           *ptr = str;
+
+	*ptr = 0;
+
+#define TEST_FLAG(flag, msg)						\
+	do {								\
+		if (flags & (flag)) {					\
+			strcpy(ptr, msg);				\
+			ptr += strlen(ptr);				\
+			flags &= ~(flag);				\
+		}							\
+	} while (0)
+
+	TEST_FLAG(i915_sf_interrupts_enabled, "IntOn|");
+	TEST_FLAG(i915_sf_submitting,         "Submitting|");
+	TEST_FLAG(i915_sf_dump_force,         "DumpForce|");
+	TEST_FLAG(i915_sf_dump_details,       "DumpDetails|");
+	TEST_FLAG(i915_sf_dump_dependencies,  "DumpDeps|");
+
+#undef TEST_FLAG
+
+	if (flags) {
+		sprintf(ptr, "Unknown_0x%X!", flags);
+		ptr += strlen(ptr);
+	}
+
+	if (ptr == str)
+		strcpy(str, "-");
+	else
+		ptr[-1] = 0;
+
+	return str;
+};
+
 int i915_scheduler_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -631,6 +744,169 @@ void i915_gem_scheduler_work_handler(struct work_struct *work)
 	}
 }
 
+int i915_scheduler_dump_all(struct drm_device *dev, const char *msg)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	unsigned long   flags;
+	int             ret;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	ret = i915_scheduler_dump_all_locked(dev, msg);
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return ret;
+}
+
+static int i915_scheduler_dump_all_locked(struct drm_device *dev, const char *msg)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring;
+	int                     i, r, ret = 0;
+
+	for_each_ring(ring, dev_priv, i) {
+		scheduler->flags[ring->id] |= i915_sf_dump_force   |
+					      i915_sf_dump_details |
+					      i915_sf_dump_dependencies;
+		r = i915_scheduler_dump_locked(ring, msg);
+		if (ret == 0)
+			ret = r;
+	}
+
+	return ret;
+}
+
+int i915_scheduler_dump(struct intel_engine_cs *ring, const char *msg)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	unsigned long   flags;
+	int             ret;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	ret = i915_scheduler_dump_locked(ring, msg);
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return ret;
+}
+
+static int i915_scheduler_dump_locked(struct intel_engine_cs *ring, const char *msg)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	int                 flying = 0, queued = 0, complete = 0, other = 0;
+	static int          old_flying = -1, old_queued = -1, old_complete = -1;
+	bool                b_dump;
+	char                brkt[2] = { '<', '>' };
+
+	if (!ring)
+		return -EINVAL;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (I915_SQS_IS_QUEUED(node))
+			queued++;
+		else if (I915_SQS_IS_FLYING(node))
+			flying++;
+		else if (I915_SQS_IS_COMPLETE(node))
+			complete++;
+		else
+			other++;
+	}
+
+	b_dump = (flying != old_flying) ||
+		 (queued != old_queued) ||
+		 (complete != old_complete);
+	if (scheduler->flags[ring->id] & i915_sf_dump_force) {
+		if (!b_dump) {
+			b_dump = true;
+			brkt[0] = '{';
+			brkt[1] = '}';
+		}
+
+		scheduler->flags[ring->id] &= ~i915_sf_dump_force;
+	}
+
+	if (b_dump) {
+		old_flying   = flying;
+		old_queued   = queued;
+		old_complete = complete;
+		DRM_DEBUG_DRIVER("<%s> Q:%02d, F:%02d, C:%02d, O:%02d, "
+				 "Flags = %s, Next = %d:%d %c%s%c\n",
+				 ring->name, queued, flying, complete, other,
+				 i915_scheduler_flag_str(scheduler->flags[ring->id]),
+				 dev_priv->request_uniq, dev_priv->next_seqno,
+				 brkt[0], msg, brkt[1]);
+	} else {
+		/*DRM_DEBUG_DRIVER("<%s> Q:%02d, F:%02d, C:%02d, O:%02d"
+				 ", Flags = %s, Next = %d:%d [%s]\n",
+				 ring->name,
+				 queued, flying, complete, other,
+				 i915_scheduler_flag_str(scheduler->flags[ring->id]),
+				 dev_priv->request_uniq, dev_priv->next_seqno, msg); */
+
+		return 0;
+	}
+
+	if (scheduler->flags[ring->id] & i915_sf_dump_details) {
+		int         i, deps;
+		uint32_t    count, counts[i915_sqs_MAX];
+
+		memset(counts, 0x00, sizeof(counts));
+
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (node->status < i915_sqs_MAX) {
+				count = counts[node->status]++;
+			} else {
+				DRM_DEBUG_DRIVER("<%s>   Unknown status: %d!\n",
+						 ring->name, node->status);
+				count = -1;
+			}
+
+			deps = 0;
+			for (i = 0; i < node->num_deps; i++)
+				if (i915_scheduler_is_dependency_valid(node, i))
+					deps++;
+
+			DRM_DEBUG_DRIVER("<%s>   %c:%02d> uniq = %d, seqno"
+					 " = %d/%s, deps = %d / %d, fence = %p/%d, %s [pri = "
+					 "%4d]\n", ring->name,
+					 i915_scheduler_queue_status_chr(node->status),
+					 count,
+					 node->params.request->uniq,
+					 node->params.request->seqno,
+					 node->params.ring->name,
+					 deps, node->num_deps,
+					 node->params.fence_wait,
+					 node->params.fence_wait ? sync_fence_is_signaled(node->params.fence_wait) : 0,
+					 i915_qe_state_str(node),
+					 node->priority);
+
+			if ((scheduler->flags[ring->id] & i915_sf_dump_dependencies)
+				== 0)
+				continue;
+
+			for (i = 0; i < node->num_deps; i++)
+				if (node->dep_list[i])
+					DRM_DEBUG_DRIVER("<%s>       |-%c:"
+						"%02d%c uniq = %d, seqno = %d/%s, %s [pri = %4d]\n",
+						ring->name,
+						i915_scheduler_queue_status_chr(node->dep_list[i]->status),
+						i,
+						i915_scheduler_is_dependency_valid(node, i)
+							? '>' : '#',
+						node->dep_list[i]->params.request->uniq,
+						node->dep_list[i]->params.request->seqno,
+						node->dep_list[i]->params.ring->name,
+						i915_qe_state_str(node->dep_list[i]),
+						node->dep_list[i]->priority);
+		}
+	}
+
+	return 0;
+}
+
 int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 			       unsigned long target,
 			       bool is_locked)
@@ -1199,10 +1475,10 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 				continue;
 
 			if(!I915_SQS_IS_COMPLETE(node))
-				DRM_DEBUG_DRIVER("Closing file handle with outstanding work: %d:%d/%d on %s\n",
+				DRM_DEBUG_DRIVER("Closing file handle with outstanding work: %d:%d/%s on %s\n",
 						 node->params.request->uniq,
 						 node->params.request->seqno,
-						 node->status,
+						 i915_qe_state_str(node),
 						 ring->name);
 
 			i915_scheduler_file_queue_dec(node->params.file);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 5de0506..1bf2fa6 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -41,6 +41,9 @@ enum i915_scheduler_queue_status {
 	/* Limit value for use with arrays/loops */
 	i915_sqs_MAX
 };
+char i915_scheduler_queue_status_chr(enum i915_scheduler_queue_status status);
+const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status);
 
 #define I915_SQS_IS_QUEUED(node)	(((node)->status == i915_sqs_queued))
 #define I915_SQS_IS_FLYING(node)	(((node)->status == i915_sqs_flying))
@@ -71,6 +74,7 @@ struct i915_scheduler_queue_entry {
 	struct list_head                    link;
 	uint32_t                            flags;
 };
+const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node);
 
 struct i915_scheduler {
 	struct list_head    node_queue[I915_NUM_RINGS];
@@ -87,9 +91,16 @@ struct i915_scheduler {
 
 /* Flag bits for i915_scheduler::flags */
 enum {
+	/* Internal state */
 	i915_sf_interrupts_enabled  = (1 << 0),
 	i915_sf_submitting          = (1 << 1),
+
+	/* Dump/debug flags */
+	i915_sf_dump_force          = (1 << 8),
+	i915_sf_dump_details        = (1 << 9),
+	i915_sf_dump_dependencies   = (1 << 10),
 };
+const char *i915_scheduler_flag_str(uint32_t flags);
 
 /* Options for 'scheduler_override' module parameter: */
 enum {
@@ -109,6 +120,9 @@ void        i915_gem_scheduler_work_handler(struct work_struct *work);
 int         i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked);
 int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 				       unsigned long stamp, bool is_locked);
+int         i915_scheduler_dump(struct intel_engine_cs *ring,
+				const char *msg);
+int         i915_scheduler_dump_all(struct drm_device *dev, const char *msg);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
 bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 32/40] drm/i915: Add early exit to execbuff_final() if insufficient ring space

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

One of the major purposes of the GPU scheduler is to avoid stalling
the CPU when the GPU is busy and unable to accept more work. This
change adds support to the ring submission code to allow a ring space
check to be performed before attempting to submit a batch buffer to
the hardware. If insufficient space is available then the scheduler
can go away and come back later, letting the CPU get on with other
work, rather than stalling and waiting for the hardware to catch up.

v3: Updated to use locally cached request pointer.

Change-Id: I267159ce1150cb6714d34a49b841bcbe4bf66326
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 42 ++++++++++++++++------
 drivers/gpu/drm/i915/intel_lrc.c           | 57 +++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/intel_ringbuffer.c    | 24 +++++++++++++
 drivers/gpu/drm/i915/intel_ringbuffer.h    |  1 +
 4 files changed, 109 insertions(+), 15 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 8ba426f..bf9d804 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1101,25 +1101,19 @@ i915_reset_gen7_sol_offsets(struct drm_device *dev,
 {
 	struct intel_engine_cs *ring = req->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	int ret, i;
+	int i;
 
 	if (!IS_GEN7(dev) || ring != &dev_priv->ring[RCS]) {
 		DRM_DEBUG("sol reset is gen7/rcs only\n");
 		return -EINVAL;
 	}
 
-	ret = intel_ring_begin(req, 4 * 3);
-	if (ret)
-		return ret;
-
 	for (i = 0; i < 4; i++) {
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, GEN7_SO_WRITE_OFFSET(i));
 		intel_ring_emit(ring, 0);
 	}
 
-	intel_ring_advance(ring);
-
 	return 0;
 }
 
@@ -1247,6 +1241,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	struct intel_engine_cs  *ring = params->ring;
 	u64 exec_start, exec_len;
 	int ret;
+	uint32_t min_space;
 
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
@@ -1268,8 +1263,36 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	if (ret)
 		return ret;
 
+	/*
+	 * It would be a bad idea to run out of space while writing commands
+	 * to the ring. One of the major aims of the scheduler is to not stall
+	 * at any point for any reason. However, doing an early exit half way
+	 * through submission could result in a partial sequence being written
+	 * which would leave the engine in an unknown state. Therefore, check in
+	 * advance that there will be enough space for the entire submission
+	 * whether emitted by the code below OR by any other functions that may
+	 * be executed before the end of final().
+	 *
+	 * NB: This test deliberately overestimates, because that's easier than
+	 * tracing every potential path that could be taken!
+	 *
+	 * Current measurements suggest that we may need to emit up to 744 bytes
+	 * (186 dwords), so this is rounded up to 256 dwords here. Then we double
+	 * that to get the free space requirement, because the block isn't allowed
+	 * to span the transition from the end to the beginning of the ring.
+	 */
+#define I915_BATCH_EXEC_MAX_LEN         256	/* max dwords emitted here	*/
+	min_space = I915_BATCH_EXEC_MAX_LEN * 2 * sizeof(uint32_t);
+	ret = intel_ring_test_space(req->ringbuf, min_space);
+	if (ret)
+		goto early_error;
+
 	intel_runtime_pm_get(dev_priv);
 
+	ret = intel_ring_begin(req, I915_BATCH_EXEC_MAX_LEN);
+	if (ret)
+		goto error;
+
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
@@ -1288,10 +1311,6 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_ring_begin(req, 4);
-		if (ret)
-			goto error;
-
 		intel_ring_emit(ring, MI_NOOP);
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, INSTPM);
@@ -1328,6 +1347,7 @@ error:
 	 */
 	intel_runtime_pm_put(dev_priv);
 
+early_error:
 	if (ret)
 		intel_ring_reserved_space_cancel(req->ringbuf);
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 1fa3228..d6acd2d6 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -226,6 +226,30 @@ static void lrc_setup_hardware_status_page(struct intel_engine_cs *ring,
 		struct drm_i915_gem_object *default_ctx_obj);
 
 
+/* Test to see if the ring has sufficient space to submit a given piece of work
+ * without causing a stall */
+static int logical_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space)
+{
+	//struct intel_engine_cs *ring = ringbuf->ring;
+	//struct drm_device *dev = ring->dev;
+	//struct drm_i915_private *dev_priv = dev->dev_private;
+
+	if (ringbuf->space < min_space) {
+		/* Need to update the actual ring space. Otherwise, the system
+		 * hangs forever testing a software copy of the space value that
+		 * never changes!
+		 */
+		//ringbuf->head  = I915_READ_HEAD(ring);
+		//ringbuf->space = intel_ring_space(ringbuf);
+		intel_ring_update_space(ringbuf);
+
+		if (ringbuf->space < min_space)
+			return -EAGAIN;
+	}
+
+	return 0;
+}
+
 /**
  * intel_sanitize_enable_execlists() - sanitize i915.enable_execlists
  * @dev: DRM device.
@@ -932,6 +956,7 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	struct intel_engine_cs *ring = params->ring;
 	u64 exec_start;
 	int ret;
+	uint32_t min_space;
 
 	/* The mutex must be acquired before calling this function */
 	BUG_ON(!mutex_is_locked(&params->dev->struct_mutex));
@@ -954,6 +979,34 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 		return ret;
 
 	/*
+	 * It would be a bad idea to run out of space while writing commands
+	 * to the ring. One of the major aims of the scheduler is to not stall
+	 * at any point for any reason. However, doing an early exit half way
+	 * through submission could result in a partial sequence being written
+	 * which would leave the engine in an unknown state. Therefore, check in
+	 * advance that there will be enough space for the entire submission
+	 * whether emitted by the code below OR by any other functions that may
+	 * be executed before the end of final().
+	 *
+	 * NB: This test deliberately overestimates, because that's easier than
+	 * tracing every potential path that could be taken!
+	 *
+	 * Current measurements suggest that we may need to emit up to ??? bytes
+	 * (186 dwords), so this is rounded up to 256 dwords here. Then we double
+	 * that to get the free space requirement, because the block isn't allowed
+	 * to span the transition from the end to the beginning of the ring.
+	 */
+#define I915_BATCH_EXEC_MAX_LEN         256	/* max dwords emitted here	*/
+	min_space = I915_BATCH_EXEC_MAX_LEN * 2 * sizeof(uint32_t);
+	ret = logical_ring_test_space(ringbuf, min_space);
+	if (ret)
+		goto err;
+
+	ret = intel_logical_ring_begin(req, I915_BATCH_EXEC_MAX_LEN);
+	if (ret)
+		goto err;
+
+	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
@@ -963,10 +1016,6 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_logical_ring_begin(req, 4);
-		if (ret)
-			return ret;
-
 		intel_logical_ring_emit(ringbuf, MI_NOOP);
 		intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1));
 		intel_logical_ring_emit(ringbuf, INSTPM);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 2d8dc54..97b9dfa 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -2472,6 +2472,30 @@ int intel_ring_cacheline_align(struct drm_i915_gem_request *req)
 	return 0;
 }
 
+/* Test to see if the ring has sufficient space to submit a given piece of work
+ * without causing a stall */
+int intel_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space)
+{
+	struct drm_i915_private *dev_priv = ringbuf->ring->dev->dev_private;
+
+	/* There is a separate LRC version of this code. */
+	BUG_ON(i915.enable_execlists);
+
+	if (ringbuf->space < min_space) {
+		/* Need to update the actual ring space. Otherwise, the system
+		 * hangs forever testing a software copy of the space value that
+		 * never changes!
+		 */
+		ringbuf->head  = I915_READ_HEAD(ringbuf->ring);
+		ringbuf->space = intel_ring_space(ringbuf);
+
+		if (ringbuf->space < min_space)
+			return -EAGAIN;
+	}
+
+	return 0;
+}
+
 void intel_ring_init_seqno(struct intel_engine_cs *ring, u32 seqno)
 {
 	struct drm_device *dev = ring->dev;
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h b/drivers/gpu/drm/i915/intel_ringbuffer.h
index d31c94f..c472f8a 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.h
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.h
@@ -469,6 +469,7 @@ void intel_cleanup_ring_buffer(struct intel_engine_cs *ring);
 
 int intel_ring_alloc_request_extras(struct drm_i915_gem_request *request);
 
+int intel_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space);
 int __must_check intel_ring_begin(struct drm_i915_gem_request *req, int n);
 int __must_check intel_ring_cacheline_align(struct drm_i915_gem_request *req);
 static inline void intel_ring_emit(struct intel_engine_cs *ring,
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 33/40] drm/i915: Added scheduler statistic reporting to debugfs

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

It is useful for know what the scheduler is doing for both debugging
and performance analysis purposes. This change adds a bunch of
counters and such that keep track of various scheduler operations
(batches submitted, completed, flush requests, etc.). The data can
then be read in userland via the debugfs mechanism.

v2: Updated to match changes to scheduler implementation.

v3: Updated for changes to kill code and flush code.

Change-Id: I3266c631cd70c9eeb2c235f88f493e60462f85d7
For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c        | 77 +++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 11 +++-
 drivers/gpu/drm/i915/i915_scheduler.c      | 85 +++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_scheduler.h      | 36 +++++++++++++
 4 files changed, 200 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 8f1c10c..9e7d67d 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -3603,6 +3603,82 @@ static int i915_drrs_status(struct seq_file *m, void *unused)
 	return 0;
 }
 
+static int i915_scheduler_info(struct seq_file *m, void *unused)
+{
+	struct drm_info_node *node = (struct drm_info_node *) m->private;
+	struct drm_device *dev = node->minor->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_stats *stats = scheduler->stats;
+	struct i915_scheduler_stats_nodes node_stats[I915_NUM_RINGS];
+	struct intel_engine_cs *ring;
+	char   str[50 * (I915_NUM_RINGS + 1)], name[50], *ptr;
+	int ret, i, r;
+
+	ret = mutex_lock_interruptible(&dev->mode_config.mutex);
+	if (ret)
+		return ret;
+
+#define PRINT_VAR(name, fmt, var)					\
+	do {								\
+		sprintf(str, "%-22s", name);				\
+		ptr = str + strlen(str);				\
+		for_each_ring(ring, dev_priv, r) {			\
+			sprintf(ptr, " %10" fmt, var);			\
+			ptr += strlen(ptr);				\
+		}							\
+		seq_printf(m, "%s\n", str);				\
+	} while (0)
+
+	PRINT_VAR("Ring name:",             "s", dev_priv->ring[r].name);
+	PRINT_VAR("  Ring seqno",           "d", ring->get_seqno(ring, false));
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Batch submissions:\n");
+	PRINT_VAR("  Queued",               "u", stats[r].queued);
+	PRINT_VAR("  Submitted",            "u", stats[r].submitted);
+	PRINT_VAR("  Completed",            "u", stats[r].completed);
+	PRINT_VAR("  Expired",              "u", stats[r].expired);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Flush counts:\n");
+	PRINT_VAR("  By object",            "u", stats[r].flush_obj);
+	PRINT_VAR("  By request",           "u", stats[r].flush_req);
+	PRINT_VAR("  By stamp",             "u", stats[r].flush_stamp);
+	PRINT_VAR("  Blanket",              "u", stats[r].flush_all);
+	PRINT_VAR("  Entries bumped",       "u", stats[r].flush_bump);
+	PRINT_VAR("  Entries submitted",    "u", stats[r].flush_submit);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Miscellaneous:\n");
+	PRINT_VAR("  ExecEarly retry",      "u", stats[r].exec_early);
+	PRINT_VAR("  ExecFinal requeue",    "u", stats[r].exec_again);
+	PRINT_VAR("  ExecFinal killed",     "u", stats[r].exec_dead);
+	PRINT_VAR("  Fence wait",           "u", stats[r].fence_wait);
+	PRINT_VAR("  Fence wait again",     "u", stats[r].fence_again);
+	PRINT_VAR("  Fence wait ignore",    "u", stats[r].fence_ignore);
+	PRINT_VAR("  Fence supplied",       "u", stats[r].fence_got);
+	PRINT_VAR("  Hung flying",          "u", stats[r].kill_flying);
+	PRINT_VAR("  Hung queued",          "u", stats[r].kill_queued);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Queue contents:\n");
+	for_each_ring(ring, dev_priv, i)
+		i915_scheduler_query_stats(ring, node_stats + ring->id);
+
+	for (i = 0; i < (i915_sqs_MAX + 1); i++) {
+		sprintf(name, "  %s", i915_scheduler_queue_status_str(i));
+		PRINT_VAR(name, "d", node_stats[r].counts[i]);
+	}
+	seq_putc(m, '\n');
+
+#undef PRINT_VAR
+
+	mutex_unlock(&dev->mode_config.mutex);
+
+	return 0;
+}
+
 struct pipe_crc_info {
 	const char *name;
 	struct drm_device *dev;
@@ -5571,6 +5647,7 @@ static const struct drm_info_list i915_debugfs_list[] = {
 	{"i915_semaphore_status", i915_semaphore_status, 0},
 	{"i915_shared_dplls_info", i915_shared_dplls_info, 0},
 	{"i915_dp_mst_info", i915_dp_mst_info, 0},
+	{"i915_scheduler_info", i915_scheduler_info, 0},
 	{"i915_wa_registers", i915_wa_registers, 0},
 	{"i915_ddb_info", i915_ddb_info, 0},
 	{"i915_sseu_status", i915_sseu_status, 0},
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index bf9d804..c40ceea 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1423,13 +1423,20 @@ static int i915_early_fence_wait(struct intel_engine_cs *ring, int fence_fd)
 	}
 
 	if (!sync_fence_is_signaled(fence)) {
+		struct drm_i915_private *dev_priv = ring->dev->dev_private;
+		struct i915_scheduler *scheduler = dev_priv->scheduler;
+
 		/*
 		 * Wait forever for the fence to be signalled. This is safe
 		 * because the the mutex lock has not yet been acquired and
 		 * the wait is interruptible.
 		 */
-		if (!i915_safe_to_ignore_fence(ring, fence))
+		if (i915_safe_to_ignore_fence(ring, fence))
+			scheduler->stats[ring->id].fence_ignore++;
+		else {
+			scheduler->stats[ring->id].fence_wait++;
 			ret = sync_fence_wait(fence, -1);
+		}
 	}
 
 	sync_fence_put(fence);
@@ -1849,6 +1856,8 @@ err:
 	mutex_unlock(&dev->struct_mutex);
 
 pre_mutex_err:
+	dev_priv->scheduler->stats[ring->id].exec_early++;
+
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index be2430d..8c7a9e6 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -121,6 +121,9 @@ const char *i915_scheduler_queue_status_str(
 	case i915_sqs_dead:
 	return "Dead";
 
+	case i915_sqs_MAX:
+	return "Invalid";
+
 	default:
 	break;
 	}
@@ -264,9 +267,14 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	BUG_ON(!scheduler);
 
+	if (qe->params.fence_wait)
+		scheduler->stats[ring->id].fence_got++;
+
 	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
+		scheduler->stats[qe->params.ring->id].queued++;
+
 		trace_i915_scheduler_queue(qe->params.ring, qe);
 
 		WARN_ON(qe->params.fence_wait &&
@@ -276,6 +284,7 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
 		ret = dev_priv->gt.execbuf_final(&qe->params);
+		scheduler->stats[qe->params.ring->id].submitted++;
 		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
 
 		/*
@@ -309,6 +318,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		if (qe->params.fence_wait)
 			sync_fence_put(qe->params.fence_wait);
 
+		scheduler->stats[qe->params.ring->id].expired++;
+
 		return 0;
 	}
 
@@ -391,6 +402,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		not_flying = i915_scheduler_count_flying(scheduler, ring) <
 							 scheduler->min_flying;
 
+	scheduler->stats[ring->id].queued++;
+
 	trace_i915_scheduler_queue(ring, node);
 	trace_i915_scheduler_node_state_change(ring, node);
 
@@ -495,13 +508,17 @@ static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
 
 /* Give up on a node completely. For example, because it is causing the
  * ring to hang or is using some resource that no longer exists. */
-static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
+static void i915_scheduler_node_kill(struct i915_scheduler *scheduler,
+				     struct i915_scheduler_queue_entry *node)
 {
 	BUG_ON(!node);
 	BUG_ON(I915_SQS_IS_COMPLETE(node));
 
-	if (I915_SQS_IS_FLYING(node))
+	if (I915_SQS_IS_FLYING(node)) {
 		trace_i915_scheduler_unfly(node->params.ring, node);
+		scheduler->stats[node->params.ring->id].kill_flying++;
+	} else
+		scheduler->stats[node->params.ring->id].kill_queued++;
 
 	node->status = i915_sqs_dead;
 	trace_i915_scheduler_node_state_change(node->params.ring, node);
@@ -534,10 +551,13 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	WARN_ON(!I915_SQS_IS_FLYING(node));
 
 	/* Node was in flight so mark it as complete. */
-	if (req->cancelled)
+	if (req->cancelled) {
 		node->status = i915_sqs_dead;
-	else
+		scheduler->stats[req->ring->id].kill_flying++;
+	} else {
 		node->status = i915_sqs_complete;
+		scheduler->stats[req->ring->id].completed++;
+	}
 
 	trace_i915_scheduler_node_state_change(req->ring, node);
 
@@ -665,6 +685,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 
 		list_del(&node->link);
 		list_add(&node->link, &remove);
+		scheduler->stats[ring->id].expired++;
 
 		/* Strip the dependency info while the mutex is still locked */
 		i915_scheduler_remove_dependent(scheduler, node);
@@ -907,6 +928,35 @@ static int i915_scheduler_dump_locked(struct intel_engine_cs *ring, const char *
 	return 0;
 }
 
+int i915_scheduler_query_stats(struct intel_engine_cs *ring,
+			       struct i915_scheduler_stats_nodes *stats)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	unsigned long   flags;
+
+	memset(stats, 0x00, sizeof(*stats));
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (node->status >= i915_sqs_MAX) {
+			DRM_DEBUG_DRIVER("Invalid node state: %d! [uniq = %d, seqno = %d]\n",
+					 node->status, node->params.request->uniq, node->params.request->seqno);
+
+			stats->counts[i915_sqs_MAX]++;
+			continue;
+		}
+
+		stats->counts[node->status]++;
+	}
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return 0;
+}
+
 int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 			       unsigned long target,
 			       bool is_locked)
@@ -933,6 +983,7 @@ int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 	}
 
 	spin_lock_irqsave(&scheduler->lock, flags);
+	scheduler->stats[ring->id].flush_stamp++;
 	i915_scheduler_priority_bump_clear(scheduler);
 	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
 		if (!I915_SQS_IS_QUEUED(node))
@@ -943,12 +994,15 @@ int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 
 		flush_count = i915_scheduler_priority_bump(scheduler,
 					node, scheduler->priority_level_max);
+		scheduler->stats[ring->id].flush_bump += flush_count;
 	}
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (flush_count) {
 		DRM_DEBUG_DRIVER("<%s> Bumped %d entries\n", ring->name, flush_count);
 		flush_count = i915_scheduler_submit_max_priority(ring, is_locked);
+		if (flush_count > 0)
+			scheduler->stats[ring->id].flush_submit += flush_count;
 	}
 
 	return flush_count;
@@ -975,6 +1029,8 @@ int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
 
 	BUG_ON(is_locked && (scheduler->flags[ring->id] & i915_sf_submitting));
 
+	scheduler->stats[ring->id].flush_all++;
+
 	do {
 		found = false;
 		spin_lock_irqsave(&scheduler->lock, flags);
@@ -989,6 +1045,7 @@ int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
 
 		if (found) {
 			ret = i915_scheduler_submit(ring, is_locked);
+			scheduler->stats[ring->id].flush_submit++;
 			if (ret < 0)
 				return ret;
 
@@ -1126,15 +1183,20 @@ static void i915_scheduler_wait_fence_signaled(struct sync_fence *fence,
 static bool i915_scheduler_async_fence_wait(struct drm_device *dev,
 					    struct i915_scheduler_queue_entry *node)
 {
+	struct drm_i915_private         *dev_priv = node->params.ring->dev->dev_private;
+	struct i915_scheduler           *scheduler = dev_priv->scheduler;
 	struct i915_sync_fence_waiter	*fence_waiter;
 	struct sync_fence		*fence = node->params.fence_wait;
 	int				signaled;
 	bool				success = true;
 
-	if ((node->flags & i915_qef_fence_waiting) == 0)
+	if ((node->flags & i915_qef_fence_waiting) == 0) {
 		node->flags |= i915_qef_fence_waiting;
-	else
+		scheduler->stats[node->params.ring->id].fence_wait++;
+	} else {
+		scheduler->stats[node->params.ring->id].fence_again++;
 		return true;
+	}
 
 	if (fence == NULL)
 		return false;
@@ -1199,8 +1261,10 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		else
 			signalled = true;
 
-		if (!signalled)
+		if (!signalled) {
 			signalled = i915_safe_to_ignore_fence(ring, node->params.fence_wait);
+			scheduler->stats[node->params.ring->id].fence_ignore++;
+		}
 
 		has_local  = false;
 		has_remote = false;
@@ -1337,6 +1401,8 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 		 * list. So add it back in and mark it as in flight. */
 		i915_scheduler_fly_node(node);
 
+		scheduler->stats[ring->id].submitted++;
+
 		scheduler->flags[ring->id] |= i915_sf_submitting;
 		spin_unlock_irqrestore(&scheduler->lock, flags);
 		ret = dev_priv->gt.execbuf_final(&node->params);
@@ -1355,6 +1421,7 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 			case ENOENT:
 				/* Fatal errors. Kill the node. */
 				requeue = -1;
+				scheduler->stats[ring->id].exec_dead++;
 			break;
 
 			case EAGAIN:
@@ -1364,12 +1431,14 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 			case ERESTARTSYS:
 			case EINTR:
 				/* Supposedly recoverable errors. */
+				scheduler->stats[ring->id].exec_again++;
 			break;
 
 			default:
 				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
 						 ring->name, ret);
 				/* Assume it is recoverable and hope for the best. */
+				scheduler->stats[ring->id].exec_again++;
 			break;
 			}
 
@@ -1385,7 +1454,7 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 				 * later. */
 				break;
 			} else if (requeue == -1)
-				i915_scheduler_node_kill(node);
+				i915_scheduler_node_kill(scheduler, node);
 		}
 
 		/* Keep launching until the sky is sufficiently full. */
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 1bf2fa6..9396ab4 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -76,6 +76,37 @@ struct i915_scheduler_queue_entry {
 };
 const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node);
 
+struct i915_scheduler_stats_nodes {
+	uint32_t	counts[i915_sqs_MAX + 1];
+};
+
+struct i915_scheduler_stats {
+	/* Batch buffer counts: */
+	uint32_t            queued;
+	uint32_t            submitted;
+	uint32_t            completed;
+	uint32_t            expired;
+
+	/* Other stuff: */
+	uint32_t            flush_obj;
+	uint32_t            flush_req;
+	uint32_t            flush_stamp;
+	uint32_t            flush_all;
+	uint32_t            flush_bump;
+	uint32_t            flush_submit;
+
+	uint32_t            exec_early;
+	uint32_t            exec_again;
+	uint32_t            exec_dead;
+	uint32_t            kill_flying;
+	uint32_t            kill_queued;
+
+	uint32_t            fence_wait;
+	uint32_t            fence_again;
+	uint32_t            fence_ignore;
+	uint32_t            fence_got;
+};
+
 struct i915_scheduler {
 	struct list_head    node_queue[I915_NUM_RINGS];
 	uint32_t            flags[I915_NUM_RINGS];
@@ -87,6 +118,9 @@ struct i915_scheduler {
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
 	uint32_t            file_queue_max;
+
+	/* Statistics: */
+	struct i915_scheduler_stats     stats[I915_NUM_RINGS];
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -125,6 +159,8 @@ int         i915_scheduler_dump(struct intel_engine_cs *ring,
 int         i915_scheduler_dump_all(struct drm_device *dev, const char *msg);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
+int         i915_scheduler_query_stats(struct intel_engine_cs *ring,
+				       struct i915_scheduler_stats_nodes *stats);
 bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH 00/40] GPU scheduler for i915 driver

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Implemented a batch buffer submission scheduler for the i915 DRM driver.

The general theory of operation is that when batch buffers are
submitted to the driver, the execbuffer() code assigns a unique seqno
value and then packages up all the information required to execute the
batch buffer at a later time. This package is given over to the
scheduler which adds it to an internal node list. The scheduler also
scans the list of objects associated with the batch buffer and
compares them against the objects already in use by other buffers in
the node list. If matches are found then the new batch buffer node is
marked as being dependent upon the matching node. The same is done for
the context object. The scheduler also bumps up the priority of such
matching nodes on the grounds that the more dependencies a given batch
buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers
in flight on the hardware at any given time. If fewer than this are
currently executing when a new node is queued, then the node is passed
straight through to the submit function. Otherwise it is simply added
to the queue and the driver returns back to user land.

As each batch buffer completes, it raises an interrupt which wakes up
the scheduler. Note that it is possible for multiple buffers to
complete before the IRQ handler gets to run. Further, the seqno values
of the individual buffers are not necessary incrementing as the
scheduler may have re-ordered their submission. However, the scheduler
keeps the list of executing buffers in order of hardware submission.
Thus it can scan through the list until a matching seqno is found and
then mark all in flight nodes from that point on as completed.

A deferred work queue is also poked by the interrupt handler. When
this wakes up it can do more involved processing such as actually
removing completed nodes from the queue and freeing up the resources
associated with them (internal memory allocations, DRM object
references, context reference, etc.). The work handler also checks the
in flight count and calls the submission code if a new slot has
appeared.

When the scheduler's submit code is called, it scans the queued node
list for the highest priority node that has no unmet dependencies.
Note that the dependency calculation is complex as it must take
inter-ring dependencies and potential preemptions into account. Note
also that in the future this will be extended to include external
dependencies such as the Android Native Sync file descriptors and/or
the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for
submission to the hardware. The in flight count is then re-checked and
a new node popped from the list if appropriate.

The scheduler also allows high priority batch buffers (e.g. from a
desktop compositor) to jump ahead of whatever is already running if
the underlying hardware supports pre-emption. In this situation, any
work that was pre-empted is returned to the queued list ready to be
resubmitted when no more high priority work is outstanding.

Various IGT tests are in progress to test the scheduler's operation
and will follow.

v2: Updated for changes in struct fence patch series and other changes
to underlying tree (e.g. removal of cliprects). Also changed priority
levels to be signed +/-1023 range and reduced mutex lock usage.

v3: More reuse of cached pointers rather than repeated dereferencing
(David Gordon).

Moved the dependency generation code out to a seperate function for
easier readability. Also added in support for the read-read
optimisation.

Major simplification of the DRM file close handler.

Fixed up an overzealous WARN.

Removed unnecessary flushing of the scheduler queue when waiting for a
request.


[Patches against drm-intel-nightly tree fetched 17/11/2015 with struct
fence conversion patches applied]

Dave Gordon (3):
  drm/i915: Updating assorted register and status page definitions
  drm/i915: Cache request pointer in *_submission_final()
  drm/i915: Add scheduling priority to per-context parameters

John Harrison (37):
  drm/i915: Add total count to context status debugfs output
  drm/i915: Explicit power enable during deferred context initialisation
  drm/i915: Prelude to splitting i915_gem_do_execbuffer in two
  drm/i915: Split i915_dem_do_execbuffer() in half
  drm/i915: Re-instate request->uniq because it is extremely useful
  drm/i915: Start of GPU scheduler
  drm/i915: Prepare retire_requests to handle out-of-order seqnos
  drm/i915: Disable hardware semaphores when GPU scheduler is enabled
  drm/i915: Force MMIO flips when scheduler enabled
  drm/i915: Added scheduler hook when closing DRM file handles
  drm/i915: Added scheduler hook into i915_gem_request_notify()
  drm/i915: Added deferred work handler for scheduler
  drm/i915: Redirect execbuffer_final() via scheduler
  drm/i915: Keep the reserved space mechanism happy
  drm/i915: Added tracking/locking of batch buffer objects
  drm/i915: Hook scheduler node clean up into retire requests
  drm/i915: Added scheduler support to __wait_request() calls
  drm/i915: Added scheduler support to page fault handler
  drm/i915: Added scheduler flush calls to ring throttle and idle functions
  drm/i915: Added a module parameter for allowing scheduler overrides
  drm/i915: Support for 'unflushed' ring idle
  drm/i915: Defer seqno allocation until actual hardware submission time
  drm/i915: Added immediate submission override to scheduler
  drm/i915: Add sync wait support to scheduler
  drm/i915: Connecting execbuff fences to scheduler
  drm/i915: Added trace points to scheduler
  drm/i915: Added scheduler queue throttling by DRM file handle
  drm/i915: Added debugfs interface to scheduler tuning parameters
  drm/i915: Added debug state dump facilities to scheduler
  drm/i915: Add early exit to execbuff_final() if insufficient ring space
  drm/i915: Added scheduler statistic reporting to debugfs
  drm/i915: Added seqno values to scheduler status dump
  drm/i915: Add scheduler support functions for TDR
  drm/i915: GPU priority bumping to prevent starvation
  drm/i915: Scheduler state dump via debugfs
  drm/i915: Enable GPU scheduler by default
  drm/i915: Allow scheduler to manage inter-ring object synchronisation

 drivers/gpu/drm/i915/Makefile              |    1 +
 drivers/gpu/drm/i915/i915_debugfs.c        |  283 +++++
 drivers/gpu/drm/i915/i915_dma.c            |    6 +
 drivers/gpu/drm/i915/i915_drv.c            |    9 +
 drivers/gpu/drm/i915/i915_drv.h            |   56 +-
 drivers/gpu/drm/i915/i915_gem.c            |  170 ++-
 drivers/gpu/drm/i915/i915_gem_context.c    |   24 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  350 ++++--
 drivers/gpu/drm/i915/i915_params.c         |    4 +
 drivers/gpu/drm/i915/i915_reg.h            |   30 +-
 drivers/gpu/drm/i915/i915_scheduler.c      | 1640 ++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |  175 +++
 drivers/gpu/drm/i915/i915_trace.h          |  215 +++-
 drivers/gpu/drm/i915/intel_display.c       |   10 +-
 drivers/gpu/drm/i915/intel_lrc.c           |  166 ++-
 drivers/gpu/drm/i915/intel_lrc.h           |    1 +
 drivers/gpu/drm/i915/intel_ringbuffer.c    |   47 +-
 drivers/gpu/drm/i915/intel_ringbuffer.h    |   35 +-
 include/uapi/drm/i915_drm.h                |    1 +
 19 files changed, 3057 insertions(+), 166 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h

-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH 01/39] drm/i915: Add total count to context status debugfs output

[Joonas Lahtinen]

On ma, 2015-11-23 at 11:38 +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> When there are lots and lots and even more lots of contexts (e.g.
> when running
> with execlists) it is useful to be able to immediately see what the
> total
> context count is.
> 
> Change-Id: If9726d4df86567100ecf53867b43f4753f08bf84
> For: VIZ-1587

With below notes;

Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>

> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>  drivers/gpu/drm/i915/i915_debugfs.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/gpu/drm/i915/i915_debugfs.c
> b/drivers/gpu/drm/i915/i915_debugfs.c
> index 18dfb56..ecab9b8 100644
> --- a/drivers/gpu/drm/i915/i915_debugfs.c
> +++ b/drivers/gpu/drm/i915/i915_debugfs.c
> @@ -1953,6 +1953,7 @@ static int i915_context_status(struct seq_file
> *m, void *unused)
>  	struct drm_i915_private *dev_priv = dev->dev_private;
>  	struct intel_engine_cs *ring;
>  	struct intel_context *ctx;
> +	uint32_t count = 0;

Should really be just of type 'int'. Overall the driver code use 'int'
for counting.

>  	int ret, i;
>  
>  	ret = mutex_lock_interruptible(&dev->struct_mutex);
> @@ -1966,6 +1967,7 @@ static int i915_context_status(struct seq_file
> *m, void *unused)
>  
>  		seq_puts(m, "HW context ");
>  		describe_ctx(m, ctx);
> +		count++;
>  		for_each_ring(ring, dev_priv, i) {
>  			if (ring->default_context == ctx)
>  				seq_printf(m, "(default context %s)
> ",
> @@ -1994,6 +1996,8 @@ static int i915_context_status(struct seq_file
> *m, void *unused)
>  		seq_putc(m, '\n');
>  	}
>  
> +	seq_printf(m, "Total: %d contexts\n", count);
> +

If the above variable is not changed, %d is wrong for uint32_t and unsi
gned int, should be %u.

>  	mutex_unlock(&dev->struct_mutex);
>  
>  	return 0;
-- 
Joonas Lahtinen
Open Source Technology Center
Intel Corporation

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH 02/39] drm/i915: Updating assorted register and status page definitions

[Joonas Lahtinen]

Hi,

Comments below.

On ma, 2015-11-23 at 11:38 +0000, John.C.Harrison@Intel.com wrote:
> From: Dave Gordon <david.s.gordon@intel.com>
> 
> Added various definitions that will be useful for the scheduler in
> general and
> pre-emptive context switching in particular.
> 
> Change-Id: Ica805b94160426def51f5d520f5ce51c60864a98
> For: VIZ-1587
> Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
> ---
>  drivers/gpu/drm/i915/i915_reg.h         | 30
> +++++++++++++++++++++++++++++-
>  drivers/gpu/drm/i915/intel_ringbuffer.h | 32
> +++++++++++++++++++++++++++++---
>  2 files changed, 58 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/i915_reg.h
> b/drivers/gpu/drm/i915/i915_reg.h
> index db72f98..b986050 100644
> --- a/drivers/gpu/drm/i915/i915_reg.h
> +++ b/drivers/gpu/drm/i915/i915_reg.h
> @@ -259,6 +259,10 @@
>  #define  MI_GLOBAL_GTT    (1<<22)
>  
>  #define MI_NOOP			MI_INSTR(0, 0)
> +#define   MI_NOOP_WRITE_ID		(1<<22)
> +#define   MI_NOOP_ID_MASK		((1<<22) - 1)
> +#define   MI_NOOP_MID(id)		((id) & MI_NOOP_ID_MASK)

I'd stay consistent with the style of other _MASK/_SHIFT pairs
elsewhere in this header. I find this rather unintuitive compared to
0xff notation especially if it would be with nonzero shift. MI_NOOP_MID
is extra, too.

Most of these new definitions are not used anywhere in the series?
Partially due to that I would prefer these to be introduced in smaller
patches prior to first use, at least to an extent that one patch per
series, with only stuff being used. Would make reviewing much easier to
know they have been actually tested too.

Daniel, how do you want to see the register updates? As much added at once as possible, or only needed?

> +#define MI_NOOP_WITH_ID(id)	MI_INSTR(0,
> MI_NOOP_WRITE_ID|MI_NOOP_MID(id))
>  #define MI_USER_INTERRUPT	MI_INSTR(0x02, 0)
>  #define MI_WAIT_FOR_EVENT       MI_INSTR(0x03, 0)
>  #define   MI_WAIT_FOR_OVERLAY_FLIP	(1<<16)
> @@ -276,6 +280,7 @@
>  #define MI_ARB_ON_OFF		MI_INSTR(0x08, 0)
>  #define   MI_ARB_ENABLE			(1<<0)
>  #define   MI_ARB_DISABLE		(0<<0)
> +#define MI_ARB_CHECK		MI_INSTR(0x05, 0)
>  #define MI_BATCH_BUFFER_END	MI_INSTR(0x0a, 0)
>  #define MI_SUSPEND_FLUSH	MI_INSTR(0x0b, 0)
>  #define   MI_SUSPEND_FLUSH_EN	(1<<0)
> @@ -325,6 +330,8 @@
>  #define   MI_SEMAPHORE_SYNC_INVALID (3<<16)
>  #define   MI_SEMAPHORE_SYNC_MASK    (3<<16)
>  #define MI_SET_CONTEXT		MI_INSTR(0x18, 0)
> +#define   MI_CONTEXT_ADDR_MASK		((~0)<<12)
> +#define   MI_SET_CONTEXT_FLAG_MASK	((1<<12)-1)
>  #define   MI_MM_SPACE_GTT		(1<<8)
>  #define   MI_MM_SPACE_PHYSICAL		(0<<8)
>  #define   MI_SAVE_EXT_STATE_EN		(1<<3)
> @@ -344,6 +351,10 @@
>  #define   MI_USE_GGTT		(1 << 22) /* g4x+ */
>  #define MI_STORE_DWORD_INDEX	MI_INSTR(0x21, 1)
>  #define   MI_STORE_DWORD_INDEX_SHIFT 2
> +#define MI_STORE_REG_MEM	MI_INSTR(0x24, 1)
> +#define   MI_STORE_REG_MEM_GTT		(1 << 22)
> +#define   MI_STORE_REG_MEM_PREDICATE	(1 << 21)
> +
>  /* Official intel docs are somewhat sloppy concerning
> MI_LOAD_REGISTER_IMM:
>   * - Always issue a MI_NOOP _before_ the MI_LOAD_REGISTER_IMM -
> otherwise hw
>   *   simply ignores the register load under certain conditions.
> @@ -358,7 +369,10 @@
>  #define MI_FLUSH_DW		MI_INSTR(0x26, 1) /* for GEN6 */
>  #define   MI_FLUSH_DW_STORE_INDEX	(1<<21)
>  #define   MI_INVALIDATE_TLB		(1<<18)
> +#define   MI_FLUSH_DW_OP_NONE		(0<<14)
>  #define   MI_FLUSH_DW_OP_STOREDW	(1<<14)
> +#define   MI_FLUSH_DW_OP_RSVD		(2<<14)
> +#define   MI_FLUSH_DW_OP_STAMP		(3<<14)
>  #define   MI_FLUSH_DW_OP_MASK		(3<<14)
>  #define   MI_FLUSH_DW_NOTIFY		(1<<8)
>  #define   MI_INVALIDATE_BSD		(1<<7)
> @@ -1533,6 +1547,19 @@ enum skl_disp_power_wells {
>  
>  #define HSW_GTT_CACHE_EN	0x4024
>  #define   GTT_CACHE_EN_ALL	0xF0007FFF
> +
> +/*
> + * Premption-related registers
> + */
> +#define RING_UHPTR(base)	((base)+0x134)
> +#define   UHPTR_GFX_ADDR_ALIGN		(0x7)
> +#define   UHPTR_VALID			(0x1)
> +#define RING_PREEMPT_ADDR	0x0214c
> +#define   PREEMPT_BATCH_LEVEL_MASK	(0x3)
> +#define BB_PREEMPT_ADDR		0x02148
> +#define SBB_PREEMPT_ADDR	0x0213c
> +#define RS_PREEMPT_STATUS	0x0215c
> +
>  #define GEN7_WR_WATERMARK	0x4028
>  #define GEN7_GFX_PRIO_CTRL	0x402C
>  #define ARB_MODE		0x4030
> @@ -6736,7 +6763,8 @@ enum skl_disp_power_wells {
>  #define  VLV_SPAREG2H				0xA194
>  
>  #define  GTFIFODBG				0x120000
> -#define    GT_FIFO_SBDROPERR			(1<<6)
> +#define    GT_FIFO_CPU_ERROR_MASK		0xf
> +#define    GT_FIFO_SDDROPERR			(1<<6)
>  #define    GT_FIFO_BLOBDROPERR			(1<<5)
>  #define    GT_FIFO_SB_READ_ABORTERR		(1<<4)
>  #define    GT_FIFO_DROPERR			(1<<3)
> diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h
> b/drivers/gpu/drm/i915/intel_ringbuffer.h
> index 1987abd..48f60cc 100644
> --- a/drivers/gpu/drm/i915/intel_ringbuffer.h
> +++ b/drivers/gpu/drm/i915/intel_ringbuffer.h
> @@ -49,6 +49,12 @@ struct  intel_hw_status_page {
>  #define I915_READ_MODE(ring) I915_READ(RING_MI_MODE((ring)
> ->mmio_base))
>  #define I915_WRITE_MODE(ring, val) I915_WRITE(RING_MI_MODE((ring)
> ->mmio_base), val)
>  
> +#define I915_READ_UHPTR(ring) \
> +		I915_READ(RING_UHPTR((ring)->mmio_base))
> +#define I915_WRITE_UHPTR(ring, val) \
> +		I915_WRITE(RING_UHPTR((ring)->mmio_base), val)
> +#define I915_READ_NOPID(ring) I915_READ(RING_NOPID((ring)
> ->mmio_base))
> +
>  /* seqno size is actually only a uint32, but since we plan to use
> MI_FLUSH_DW to
>   * do the writes, and that must have qw aligned offsets, simply
> pretend it's 8b.
>   */
> @@ -426,10 +432,30 @@ intel_write_status_page(struct intel_engine_cs
> *ring,
>   * 0x20-0x2f: Reserved (Gen6+)
>   *
>   * The area from dword 0x30 to 0x3ff is available for driver usage.
> + *
> + * Note: in general the allocation of these indices is arbitrary, as
> long
> + * as they are all unique. But a few of them are used with
> instructions that
> + * have specific alignment requirements, those particular indices
> must be
> + * chosen carefully to meet those requirements. The list below shows
> the
> + * currently-known alignment requirements:
> + *
> + *	I915_GEM_SCRATCH_INDEX	    must be EVEN
> + */
> +
> +/*
> + * Tracking; these are updated by the GPU at the beginning and/or
> end of every
> + * batch. One pair for regular buffers, the other for preemptive
> ones.
>   */
> -#define I915_GEM_HWS_INDEX		0x30
> -#define I915_GEM_HWS_SCRATCH_INDEX	0x40
> -#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX <<
> MI_STORE_DWORD_INDEX_SHIFT)
> +#define I915_BATCH_DONE_SEQNO		0x30  /* Completed
> batch seqno        */
> +#define I915_BATCH_ACTIVE_SEQNO		0x31  /* In progress
> batch seqno      */
> +#define I915_PREEMPTIVE_DONE_SEQNO	0x32  /* Completed
> preemptive batch   */
> +#define I915_PREEMPTIVE_ACTIVE_SEQNO	0x33  /* In progress
> preemptive batch */
> +
> +#define I915_GEM_HWS_INDEX		I915_BATCH_DONE_SEQNO	
> /* alias */
> +#define I915_GEM_ACTIVE_SEQNO_INDEX	I915_BATCH_ACTIVE_SEQNO	
> /* alias */
> +
> +#define I915_GEM_HWS_SCRATCH_INDEX	0x38  /* QWord */
> +#define I915_GEM_HWS_SCRATCH_ADDR	(I915_GEM_HWS_SCRATCH_INDEX
> << MI_STORE_DWORD_INDEX_SHIFT)
>  
>  struct intel_ringbuffer *
>  intel_engine_create_ringbuffer(struct intel_engine_cs *engine, int
> size);
-- 
Joonas Lahtinen
Open Source Technology Center
Intel Corporation

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH 03/39] drm/i915: Explicit power enable during deferred context initialisation

[Joonas Lahtinen]

On ma, 2015-11-23 at 11:38 +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> A later patch in this series re-organises the batch buffer submission
> code. Part of that is to reduce the scope of a pm_get/put pair.
> Specifically, they previously wrapped the entire submission path from
> the very start to the very end, now they only wrap the actual
> hardware
> submission part in the back half.
> 
> While that is a good thing in general, it causes a problem with the
> deferred context initialisation. That is done quite early on in the
> execbuf code path - it happens at context validation time rather than
> context switch time. Some of the deferred work requires the power to
> be enabled. Hence this patch adds an explicit power reference count
> to
> the deferred initialisation code itself.
> 
> Change-Id: Id7b1535dfd8809a2bd5546272de2bbec39da2868
> Issue: GMINL-5159

Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>

> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>  drivers/gpu/drm/i915/intel_lrc.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/i915/intel_lrc.c
> b/drivers/gpu/drm/i915/intel_lrc.c
> index 76fc245..e510730 100644
> --- a/drivers/gpu/drm/i915/intel_lrc.c
> +++ b/drivers/gpu/drm/i915/intel_lrc.c
> @@ -2470,6 +2470,8 @@ int intel_lr_context_deferred_alloc(struct
> intel_context *ctx,
>  	WARN_ON(ctx->legacy_hw_ctx.rcs_state != NULL);
>  	WARN_ON(ctx->engine[ring->id].state);
>  
> +	intel_runtime_pm_get(dev->dev_private);
> +
>  	context_size = round_up(get_lr_context_size(ring), 4096);
>  
>  	/* One extra page as the sharing data between driver and GuC
> */
> @@ -2478,7 +2480,8 @@ int intel_lr_context_deferred_alloc(struct
> intel_context *ctx,
>  	ctx_obj = i915_gem_alloc_object(dev, context_size);
>  	if (!ctx_obj) {
>  		DRM_DEBUG_DRIVER("Alloc LRC backing obj failed.\n");
> -		return -ENOMEM;
> +		ret = -ENOMEM;
> +		goto error_pm;
>  	}
>  
>  	ringbuf = intel_engine_create_ringbuffer(ring, 4 *
> PAGE_SIZE);
> @@ -2524,6 +2527,8 @@ int intel_lr_context_deferred_alloc(struct
> intel_context *ctx,
>  		}
>  		i915_add_request_no_flush(req);
>  	}
> +
> +	intel_runtime_pm_put(dev->dev_private);
>  	return 0;
>  
>  error_ringbuf:
> @@ -2532,6 +2537,8 @@ error_deref_obj:
>  	drm_gem_object_unreference(&ctx_obj->base);
>  	ctx->engine[ring->id].ringbuf = NULL;
>  	ctx->engine[ring->id].state = NULL;
> +error_pm:
> +	intel_runtime_pm_put(dev->dev_private);
>  	return ret;
>  }
>  
-- 
Joonas Lahtinen
Open Source Technology Center
Intel Corporation

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH 02/39] drm/i915: Updating assorted register and status page definitions

[Daniel Vetter]

On Fri, Jan 08, 2016 at 02:26:40PM +0200, Joonas Lahtinen wrote:
> Hi,
> 
> Comments below.
> 
> On ma, 2015-11-23 at 11:38 +0000, John.C.Harrison@Intel.com wrote:
> > From: Dave Gordon <david.s.gordon@intel.com>
> > 
> > Added various definitions that will be useful for the scheduler in
> > general and
> > pre-emptive context switching in particular.
> > 
> > Change-Id: Ica805b94160426def51f5d520f5ce51c60864a98
> > For: VIZ-1587
> > Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
> > ---
> >  drivers/gpu/drm/i915/i915_reg.h         | 30
> > +++++++++++++++++++++++++++++-
> >  drivers/gpu/drm/i915/intel_ringbuffer.h | 32
> > +++++++++++++++++++++++++++++---
> >  2 files changed, 58 insertions(+), 4 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/i915/i915_reg.h
> > b/drivers/gpu/drm/i915/i915_reg.h
> > index db72f98..b986050 100644
> > --- a/drivers/gpu/drm/i915/i915_reg.h
> > +++ b/drivers/gpu/drm/i915/i915_reg.h
> > @@ -259,6 +259,10 @@
> >  #define  MI_GLOBAL_GTT    (1<<22)
> >  
> >  #define MI_NOOP			MI_INSTR(0, 0)
> > +#define   MI_NOOP_WRITE_ID		(1<<22)
> > +#define   MI_NOOP_ID_MASK		((1<<22) - 1)
> > +#define   MI_NOOP_MID(id)		((id) & MI_NOOP_ID_MASK)
> 
> I'd stay consistent with the style of other _MASK/_SHIFT pairs
> elsewhere in this header. I find this rather unintuitive compared to
> 0xff notation especially if it would be with nonzero shift. MI_NOOP_MID
> is extra, too.
> 
> Most of these new definitions are not used anywhere in the series?
> Partially due to that I would prefer these to be introduced in smaller
> patches prior to first use, at least to an extent that one patch per
> series, with only stuff being used. Would make reviewing much easier to
> know they have been actually tested too.
> 
> Daniel, how do you want to see the register updates? As much added at once as possible, or only needed?

I'm fine with a separate patch to just update them all. We do that
regularly for feature work.
-Daniel

> 
> > +#define MI_NOOP_WITH_ID(id)	MI_INSTR(0,
> > MI_NOOP_WRITE_ID|MI_NOOP_MID(id))
> >  #define MI_USER_INTERRUPT	MI_INSTR(0x02, 0)
> >  #define MI_WAIT_FOR_EVENT       MI_INSTR(0x03, 0)
> >  #define   MI_WAIT_FOR_OVERLAY_FLIP	(1<<16)
> > @@ -276,6 +280,7 @@
> >  #define MI_ARB_ON_OFF		MI_INSTR(0x08, 0)
> >  #define   MI_ARB_ENABLE			(1<<0)
> >  #define   MI_ARB_DISABLE		(0<<0)
> > +#define MI_ARB_CHECK		MI_INSTR(0x05, 0)
> >  #define MI_BATCH_BUFFER_END	MI_INSTR(0x0a, 0)
> >  #define MI_SUSPEND_FLUSH	MI_INSTR(0x0b, 0)
> >  #define   MI_SUSPEND_FLUSH_EN	(1<<0)
> > @@ -325,6 +330,8 @@
> >  #define   MI_SEMAPHORE_SYNC_INVALID (3<<16)
> >  #define   MI_SEMAPHORE_SYNC_MASK    (3<<16)
> >  #define MI_SET_CONTEXT		MI_INSTR(0x18, 0)
> > +#define   MI_CONTEXT_ADDR_MASK		((~0)<<12)
> > +#define   MI_SET_CONTEXT_FLAG_MASK	((1<<12)-1)
> >  #define   MI_MM_SPACE_GTT		(1<<8)
> >  #define   MI_MM_SPACE_PHYSICAL		(0<<8)
> >  #define   MI_SAVE_EXT_STATE_EN		(1<<3)
> > @@ -344,6 +351,10 @@
> >  #define   MI_USE_GGTT		(1 << 22) /* g4x+ */
> >  #define MI_STORE_DWORD_INDEX	MI_INSTR(0x21, 1)
> >  #define   MI_STORE_DWORD_INDEX_SHIFT 2
> > +#define MI_STORE_REG_MEM	MI_INSTR(0x24, 1)
> > +#define   MI_STORE_REG_MEM_GTT		(1 << 22)
> > +#define   MI_STORE_REG_MEM_PREDICATE	(1 << 21)
> > +
> >  /* Official intel docs are somewhat sloppy concerning
> > MI_LOAD_REGISTER_IMM:
> >   * - Always issue a MI_NOOP _before_ the MI_LOAD_REGISTER_IMM -
> > otherwise hw
> >   *   simply ignores the register load under certain conditions.
> > @@ -358,7 +369,10 @@
> >  #define MI_FLUSH_DW		MI_INSTR(0x26, 1) /* for GEN6 */
> >  #define   MI_FLUSH_DW_STORE_INDEX	(1<<21)
> >  #define   MI_INVALIDATE_TLB		(1<<18)
> > +#define   MI_FLUSH_DW_OP_NONE		(0<<14)
> >  #define   MI_FLUSH_DW_OP_STOREDW	(1<<14)
> > +#define   MI_FLUSH_DW_OP_RSVD		(2<<14)
> > +#define   MI_FLUSH_DW_OP_STAMP		(3<<14)
> >  #define   MI_FLUSH_DW_OP_MASK		(3<<14)
> >  #define   MI_FLUSH_DW_NOTIFY		(1<<8)
> >  #define   MI_INVALIDATE_BSD		(1<<7)
> > @@ -1533,6 +1547,19 @@ enum skl_disp_power_wells {
> >  
> >  #define HSW_GTT_CACHE_EN	0x4024
> >  #define   GTT_CACHE_EN_ALL	0xF0007FFF
> > +
> > +/*
> > + * Premption-related registers
> > + */
> > +#define RING_UHPTR(base)	((base)+0x134)
> > +#define   UHPTR_GFX_ADDR_ALIGN		(0x7)
> > +#define   UHPTR_VALID			(0x1)
> > +#define RING_PREEMPT_ADDR	0x0214c
> > +#define   PREEMPT_BATCH_LEVEL_MASK	(0x3)
> > +#define BB_PREEMPT_ADDR		0x02148
> > +#define SBB_PREEMPT_ADDR	0x0213c
> > +#define RS_PREEMPT_STATUS	0x0215c
> > +
> >  #define GEN7_WR_WATERMARK	0x4028
> >  #define GEN7_GFX_PRIO_CTRL	0x402C
> >  #define ARB_MODE		0x4030
> > @@ -6736,7 +6763,8 @@ enum skl_disp_power_wells {
> >  #define  VLV_SPAREG2H				0xA194
> >  
> >  #define  GTFIFODBG				0x120000
> > -#define    GT_FIFO_SBDROPERR			(1<<6)
> > +#define    GT_FIFO_CPU_ERROR_MASK		0xf
> > +#define    GT_FIFO_SDDROPERR			(1<<6)
> >  #define    GT_FIFO_BLOBDROPERR			(1<<5)
> >  #define    GT_FIFO_SB_READ_ABORTERR		(1<<4)
> >  #define    GT_FIFO_DROPERR			(1<<3)
> > diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h
> > b/drivers/gpu/drm/i915/intel_ringbuffer.h
> > index 1987abd..48f60cc 100644
> > --- a/drivers/gpu/drm/i915/intel_ringbuffer.h
> > +++ b/drivers/gpu/drm/i915/intel_ringbuffer.h
> > @@ -49,6 +49,12 @@ struct  intel_hw_status_page {
> >  #define I915_READ_MODE(ring) I915_READ(RING_MI_MODE((ring)
> > ->mmio_base))
> >  #define I915_WRITE_MODE(ring, val) I915_WRITE(RING_MI_MODE((ring)
> > ->mmio_base), val)
> >  
> > +#define I915_READ_UHPTR(ring) \
> > +		I915_READ(RING_UHPTR((ring)->mmio_base))
> > +#define I915_WRITE_UHPTR(ring, val) \
> > +		I915_WRITE(RING_UHPTR((ring)->mmio_base), val)
> > +#define I915_READ_NOPID(ring) I915_READ(RING_NOPID((ring)
> > ->mmio_base))
> > +
> >  /* seqno size is actually only a uint32, but since we plan to use
> > MI_FLUSH_DW to
> >   * do the writes, and that must have qw aligned offsets, simply
> > pretend it's 8b.
> >   */
> > @@ -426,10 +432,30 @@ intel_write_status_page(struct intel_engine_cs
> > *ring,
> >   * 0x20-0x2f: Reserved (Gen6+)
> >   *
> >   * The area from dword 0x30 to 0x3ff is available for driver usage.
> > + *
> > + * Note: in general the allocation of these indices is arbitrary, as
> > long
> > + * as they are all unique. But a few of them are used with
> > instructions that
> > + * have specific alignment requirements, those particular indices
> > must be
> > + * chosen carefully to meet those requirements. The list below shows
> > the
> > + * currently-known alignment requirements:
> > + *
> > + *	I915_GEM_SCRATCH_INDEX	    must be EVEN
> > + */
> > +
> > +/*
> > + * Tracking; these are updated by the GPU at the beginning and/or
> > end of every
> > + * batch. One pair for regular buffers, the other for preemptive
> > ones.
> >   */
> > -#define I915_GEM_HWS_INDEX		0x30
> > -#define I915_GEM_HWS_SCRATCH_INDEX	0x40
> > -#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX <<
> > MI_STORE_DWORD_INDEX_SHIFT)
> > +#define I915_BATCH_DONE_SEQNO		0x30  /* Completed
> > batch seqno        */
> > +#define I915_BATCH_ACTIVE_SEQNO		0x31  /* In progress
> > batch seqno      */
> > +#define I915_PREEMPTIVE_DONE_SEQNO	0x32  /* Completed
> > preemptive batch   */
> > +#define I915_PREEMPTIVE_ACTIVE_SEQNO	0x33  /* In progress
> > preemptive batch */
> > +
> > +#define I915_GEM_HWS_INDEX		I915_BATCH_DONE_SEQNO	
> > /* alias */
> > +#define I915_GEM_ACTIVE_SEQNO_INDEX	I915_BATCH_ACTIVE_SEQNO	
> > /* alias */
> > +
> > +#define I915_GEM_HWS_SCRATCH_INDEX	0x38  /* QWord */
> > +#define I915_GEM_HWS_SCRATCH_ADDR	(I915_GEM_HWS_SCRATCH_INDEX
> > << MI_STORE_DWORD_INDEX_SHIFT)
> >  
> >  struct intel_ringbuffer *
> >  intel_engine_create_ringbuffer(struct intel_engine_cs *engine, int
> > size);
> -- 
> Joonas Lahtinen
> Open Source Technology Center
> Intel Corporation
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 00/38] GPU scheduler for i915 driver

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Implemented a batch buffer submission scheduler for the i915 DRM driver.

The general theory of operation is that when batch buffers are
submitted to the driver, the execbuffer() code assigns a unique seqno
value and then packages up all the information required to execute the
batch buffer at a later time. This package is given over to the
scheduler which adds it to an internal node list. The scheduler also
scans the list of objects associated with the batch buffer and
compares them against the objects already in use by other buffers in
the node list. If matches are found then the new batch buffer node is
marked as being dependent upon the matching node. The same is done for
the context object. The scheduler also bumps up the priority of such
matching nodes on the grounds that the more dependencies a given batch
buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers
in flight on the hardware at any given time. If fewer than this are
currently executing when a new node is queued, then the node is passed
straight through to the submit function. Otherwise it is simply added
to the queue and the driver returns back to user land.

As each batch buffer completes, it raises an interrupt which wakes up
the scheduler. Note that it is possible for multiple buffers to
complete before the IRQ handler gets to run. Further, the seqno values
of the individual buffers are not necessary incrementing as the
scheduler may have re-ordered their submission. However, the scheduler
keeps the list of executing buffers in order of hardware submission.
Thus it can scan through the list until a matching seqno is found and
then mark all in flight nodes from that point on as completed.

A deferred work queue is also poked by the interrupt handler. When
this wakes up it can do more involved processing such as actually
removing completed nodes from the queue and freeing up the resources
associated with them (internal memory allocations, DRM object
references, context reference, etc.). The work handler also checks the
in flight count and calls the submission code if a new slot has
appeared.

When the scheduler's submit code is called, it scans the queued node
list for the highest priority node that has no unmet dependencies.
Note that the dependency calculation is complex as it must take
inter-ring dependencies and potential preemptions into account. Note
also that in the future this will be extended to include external
dependencies such as the Android Native Sync file descriptors and/or
the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for
submission to the hardware. The in flight count is then re-checked and
a new node popped from the list if appropriate.

The scheduler also allows high priority batch buffers (e.g. from a
desktop compositor) to jump ahead of whatever is already running if
the underlying hardware supports pre-emption. In this situation, any
work that was pre-empted is returned to the queued list ready to be
resubmitted when no more high priority work is outstanding.

Various IGT tests are in progress to test the scheduler's operation
and will follow.

v2: Updated for changes in struct fence patch series and other changes
to underlying tree (e.g. removal of cliprects). Also changed priority
levels to be signed +/-1023 range and reduced mutex lock usage.

v3: More reuse of cached pointers rather than repeated dereferencing
(David Gordon).

Moved the dependency generation code out to a seperate function for
easier readability. Also added in support for the read-read
optimisation.

Major simplification of the DRM file close handler.

Fixed up an overzealous WARN.

Removed unnecessary flushing of the scheduler queue when waiting for a
request.

v4: Removed user land fence/sync integration as this is dependent upon
the de-staging of the Android sync code. That de-staging is now being
done by someone else. The sync support will be added back in to the
scheduler in a separate patch series which must wait for the
de-staging to be landed first.

Added support for killing batches from contexts that were banned after
the batches were submitted to the scheduler.

Changed various comments to fix typos, update to reflect changes to
the code, correct formatting and line wrapping, etc. Also wrapped
various long lines and twiddled white space to keep the style checker
happy.

Changed a bunch of BUG_ONs to WARN_ONs as apparently the latter are
preferred.

Used the correct array memory allocator function (kmalloc_array
instead of kmalloc).

Fixed a variable type (review comment by Joonas).

Fixed a WARN_ON firing incorrectly when removing killed nodes from the
scheduler's queue.

Dropped register definition update patch from this series. The changes
are all for pre-emption so it makes more sense for it to be part of
that series instead.

[Patches against drm-intel-nightly tree fetched 17/11/2015 with struct
fence conversion patches applied]

Dave Gordon (2):
  drm/i915: Cache request pointer in *_submission_final()
  drm/i915: Add scheduling priority to per-context parameters

John Harrison (36):
  drm/i915: Add total count to context status debugfs output
  drm/i915: Explicit power enable during deferred context initialisation
  drm/i915: Prelude to splitting i915_gem_do_execbuffer in two
  drm/i915: Split i915_dem_do_execbuffer() in half
  drm/i915: Re-instate request->uniq because it is extremely useful
  drm/i915: Start of GPU scheduler
  drm/i915: Prepare retire_requests to handle out-of-order seqnos
  drm/i915: Disable hardware semaphores when GPU scheduler is enabled
  drm/i915: Force MMIO flips when scheduler enabled
  drm/i915: Added scheduler hook when closing DRM file handles
  drm/i915: Added scheduler hook into i915_gem_request_notify()
  drm/i915: Added deferred work handler for scheduler
  drm/i915: Redirect execbuffer_final() via scheduler
  drm/i915: Keep the reserved space mechanism happy
  drm/i915: Added tracking/locking of batch buffer objects
  drm/i915: Hook scheduler node clean up into retire requests
  drm/i915: Added scheduler support to __wait_request() calls
  drm/i915: Added scheduler support to page fault handler
  drm/i915: Added scheduler flush calls to ring throttle and idle functions
  drm/i915: Added a module parameter for allowing scheduler overrides
  drm/i915: Support for 'unflushed' ring idle
  drm/i915: Defer seqno allocation until actual hardware submission time
  drm/i915: Added immediate submission override to scheduler
  drm/i915: Added trace points to scheduler
  drm/i915: Added scheduler queue throttling by DRM file handle
  drm/i915: Added debugfs interface to scheduler tuning parameters
  drm/i915: Added debug state dump facilities to scheduler
  drm/i915: Add early exit to execbuff_final() if insufficient ring space
  drm/i915: Added scheduler statistic reporting to debugfs
  drm/i915: Added seqno values to scheduler status dump
  drm/i915: Add scheduler support functions for TDR
  drm/i915: GPU priority bumping to prevent starvation
  drm/i915: Scheduler state dump via debugfs
  drm/i915: Enable GPU scheduler by default
  drm/i915: Add support for retro-actively banning batch buffers
  drm/i915: Allow scheduler to manage inter-ring object synchronisation

 drivers/gpu/drm/i915/Makefile              |    1 +
 drivers/gpu/drm/i915/i915_debugfs.c        |  280 ++++-
 drivers/gpu/drm/i915/i915_dma.c            |    6 +
 drivers/gpu/drm/i915/i915_drv.c            |    9 +
 drivers/gpu/drm/i915/i915_drv.h            |   55 +-
 drivers/gpu/drm/i915/i915_gem.c            |  177 +++-
 drivers/gpu/drm/i915/i915_gem_context.c    |   24 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  301 ++++--
 drivers/gpu/drm/i915/i915_params.c         |    4 +
 drivers/gpu/drm/i915/i915_scheduler.c      | 1530 ++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |  164 +++
 drivers/gpu/drm/i915/i915_trace.h          |  218 +++-
 drivers/gpu/drm/i915/intel_display.c       |   10 +-
 drivers/gpu/drm/i915/intel_lrc.c           |  170 +++-
 drivers/gpu/drm/i915/intel_lrc.h           |    1 +
 drivers/gpu/drm/i915/intel_ringbuffer.c    |   49 +-
 drivers/gpu/drm/i915/intel_ringbuffer.h    |    3 +-
 include/uapi/drm/i915_drm.h                |    1 +
 18 files changed, 2856 insertions(+), 147 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h

-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 01/38] drm/i915: Add total count to context status debugfs output

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When there are lots and lots and even more lots of contexts (e.g. when
running with execlists) it is useful to be able to immediately see
what the total context count is.

v4: Re-typed a variable (review feedback from Joonas)

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index b54d99e..e55294c 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1953,7 +1953,7 @@ static int i915_context_status(struct seq_file *m, void *unused)
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct intel_engine_cs *ring;
 	struct intel_context *ctx;
-	int ret, i;
+	int ret, i, count = 0;
 
 	ret = mutex_lock_interruptible(&dev->struct_mutex);
 	if (ret)
@@ -1966,6 +1966,7 @@ static int i915_context_status(struct seq_file *m, void *unused)
 
 		seq_puts(m, "HW context ");
 		describe_ctx(m, ctx);
+		count++;
 		for_each_ring(ring, dev_priv, i) {
 			if (ring->default_context == ctx)
 				seq_printf(m, "(default context %s) ",
@@ -1994,6 +1995,8 @@ static int i915_context_status(struct seq_file *m, void *unused)
 		seq_putc(m, '\n');
 	}
 
+	seq_printf(m, "Total: %d contexts\n", count);
+
 	mutex_unlock(&dev->struct_mutex);
 
 	return 0;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

A later patch in this series re-organises the batch buffer submission
code. Part of that is to reduce the scope of a pm_get/put pair.
Specifically, they previously wrapped the entire submission path from
the very start to the very end, now they only wrap the actual hardware
submission part in the back half.

While that is a good thing in general, it causes a problem with the
deferred context initialisation. That is done quite early on in the
execbuf code path - it happens at context validation time rather than
context switch time. Some of the deferred work requires the power to
be enabled. Hence this patch adds an explicit power reference count to
the deferred initialisation code itself.

Issue: GMINL-5159
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
---
 drivers/gpu/drm/i915/intel_lrc.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 76fc245..e510730 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -2470,6 +2470,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,
 	WARN_ON(ctx->legacy_hw_ctx.rcs_state != NULL);
 	WARN_ON(ctx->engine[ring->id].state);
 
+	intel_runtime_pm_get(dev->dev_private);
+
 	context_size = round_up(get_lr_context_size(ring), 4096);
 
 	/* One extra page as the sharing data between driver and GuC */
@@ -2478,7 +2480,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,
 	ctx_obj = i915_gem_alloc_object(dev, context_size);
 	if (!ctx_obj) {
 		DRM_DEBUG_DRIVER("Alloc LRC backing obj failed.\n");
-		return -ENOMEM;
+		ret = -ENOMEM;
+		goto error_pm;
 	}
 
 	ringbuf = intel_engine_create_ringbuffer(ring, 4 * PAGE_SIZE);
@@ -2524,6 +2527,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,
 		}
 		i915_add_request_no_flush(req);
 	}
+
+	intel_runtime_pm_put(dev->dev_private);
 	return 0;
 
 error_ringbuf:
@@ -2532,6 +2537,8 @@ error_deref_obj:
 	drm_gem_object_unreference(&ctx_obj->base);
 	ctx->engine[ring->id].ringbuf = NULL;
 	ctx->engine[ring->id].state = NULL;
+error_pm:
+	intel_runtime_pm_put(dev->dev_private);
 	return ret;
 }
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 03/38] drm/i915: Prelude to splitting i915_gem_do_execbuffer in two

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler decouples the submission of batch buffers to the driver
with their submission to the hardware. This basically means splitting
the execbuffer() function in half. This change rearranges some code
ready for the split to occur.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 63 ++++++++++++++++++------------
 drivers/gpu/drm/i915/intel_lrc.c           | 18 ++++++---
 2 files changed, 51 insertions(+), 30 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index bfc4c17..0eca2b6 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -933,10 +933,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
 	if (flush_domains & I915_GEM_DOMAIN_GTT)
 		wmb();
 
-	/* Unconditionally invalidate gpu caches and ensure that we do flush
-	 * any residual writes from the previous batch.
-	 */
-	return intel_ring_invalidate_all_caches(req);
+	return 0;
 }
 
 static bool
@@ -1189,17 +1186,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	u32 instp_mask;
 	int ret;
 
-	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
-	if (ret)
-		return ret;
-
-	ret = i915_switch_context(params->request);
-	if (ret)
-		return ret;
-
-	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<ring->id),
-	     "%s didn't clear reload\n", ring->name);
-
 	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
 	instp_mask = I915_EXEC_CONSTANTS_MASK;
 	switch (instp_mode) {
@@ -1233,11 +1219,37 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 		return -EINVAL;
 	}
 
+	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
+	if (ret)
+		return ret;
+
+	i915_gem_execbuffer_move_to_active(vmas, params->request);
+
+	/* To be split into two functions here... */
+
+	intel_runtime_pm_get(dev_priv);
+
+	/*
+	 * Unconditionally invalidate gpu caches and ensure that we do flush
+	 * any residual writes from the previous batch.
+	 */
+	ret = intel_ring_invalidate_all_caches(params->request);
+	if (ret)
+		goto error;
+
+	/* Switch to the correct context for the batch */
+	ret = i915_switch_context(params->request);
+	if (ret)
+		goto error;
+
+	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<ring->id),
+	     "%s didn't clear reload\n", ring->name);
+
 	if (ring == &dev_priv->ring[RCS] &&
 	    instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_ring_begin(params->request, 4);
 		if (ret)
-			return ret;
+			goto error;
 
 		intel_ring_emit(ring, MI_NOOP);
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
@@ -1251,7 +1263,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {
 		ret = i915_reset_gen7_sol_offsets(dev, params->request);
 		if (ret)
-			return ret;
+			goto error;
 	}
 
 	exec_len   = args->batch_len;
@@ -1262,14 +1274,20 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 					exec_start, exec_len,
 					params->dispatch_flags);
 	if (ret)
-		return ret;
+		goto error;
 
 	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
 
-	i915_gem_execbuffer_move_to_active(vmas, params->request);
 	i915_gem_execbuffer_retire_commands(params);
 
-	return 0;
+error:
+	/*
+	 * intel_gpu_busy should also get a ref, so it will free when the device
+	 * is really idle.
+	 */
+	intel_runtime_pm_put(dev_priv);
+
+	return ret;
 }
 
 /**
@@ -1424,8 +1442,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		dispatch_flags |= I915_DISPATCH_RS;
 	}
 
-	intel_runtime_pm_get(dev_priv);
-
 	ret = i915_mutex_lock_interruptible(dev);
 	if (ret)
 		goto pre_mutex_err;
@@ -1599,9 +1615,6 @@ err:
 	mutex_unlock(&dev->struct_mutex);
 
 pre_mutex_err:
-	/* intel_gpu_busy should also get a ref, so it will free when the device
-	 * is really idle. */
-	intel_runtime_pm_put(dev_priv);
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index e510730..4bf0ee6 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -647,10 +647,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,
 	if (flush_domains & I915_GEM_DOMAIN_GTT)
 		wmb();
 
-	/* Unconditionally invalidate gpu caches and ensure that we do flush
-	 * any residual writes from the previous batch.
-	 */
-	return logical_ring_invalidate_all_caches(req);
+	return 0;
 }
 
 int intel_logical_ring_alloc_request_extras(struct drm_i915_gem_request *request)
@@ -913,6 +910,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 	if (ret)
 		return ret;
 
+	i915_gem_execbuffer_move_to_active(vmas, params->request);
+
+	/* To be split into two functions here... */
+
+	/*
+	 * Unconditionally invalidate gpu caches and ensure that we do flush
+	 * any residual writes from the previous batch.
+	 */
+	ret = logical_ring_invalidate_all_caches(params->request);
+	if (ret)
+		return ret;
+
 	if (ring == &dev_priv->ring[RCS] &&
 	    instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_logical_ring_begin(params->request, 4);
@@ -937,7 +946,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
 
-	i915_gem_execbuffer_move_to_active(vmas, params->request);
 	i915_gem_execbuffer_retire_commands(params);
 
 	return 0;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 04/38] drm/i915: Split i915_dem_do_execbuffer() in half

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Split the execbuffer() function in half. The first half collects and
validates all the information required to process the batch buffer. It
also does all the object pinning, relocations, active list management,
etc - basically anything that must be done upfront before the IOCTL
returns and allows the user land side to start changing/freeing
things. The second half does the actual ring submission.

This change implements the split but leaves the back half being called
directly from the end of the front half.

v2: Updated due to changes in underlying tree - addition of sync fence
support and removal of cliprects.

v3: Moved local 'ringbuf' variable to make later patches in the
series a bit neater.

v4: Corrected a typo in the commit message and downgraded a BUG_ON to
a WARN_ON as the latter is preferred. Also removed all the external
sync/fence support as that will now be a separate patch series.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  11 ++++
 drivers/gpu/drm/i915/i915_gem.c            |   2 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 101 ++++++++++++++++++++++-------
 drivers/gpu/drm/i915/intel_lrc.c           |  57 +++++++++++-----
 drivers/gpu/drm/i915/intel_lrc.h           |   1 +
 5 files changed, 130 insertions(+), 42 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index acfe25f..2520459 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1679,10 +1679,18 @@ struct i915_execbuffer_params {
 	struct drm_device               *dev;
 	struct drm_file                 *file;
 	uint32_t                        dispatch_flags;
+	uint32_t                        args_flags;
 	uint32_t                        args_batch_start_offset;
+	uint32_t                        args_batch_len;
+	uint32_t                        args_num_cliprects;
+	uint32_t                        args_DR1;
+	uint32_t                        args_DR4;
 	uint64_t                        batch_obj_vm_offset;
 	struct intel_engine_cs          *ring;
 	struct drm_i915_gem_object      *batch_obj;
+	struct drm_clip_rect            *cliprects;
+	uint32_t                        instp_mask;
+	int                             instp_mode;
 	struct intel_context            *ctx;
 	struct drm_i915_gem_request     *request;
 };
@@ -1944,6 +1952,7 @@ struct drm_i915_private {
 		int (*execbuf_submit)(struct i915_execbuffer_params *params,
 				      struct drm_i915_gem_execbuffer2 *args,
 				      struct list_head *vmas);
+		int (*execbuf_final)(struct i915_execbuffer_params *params);
 		int (*init_rings)(struct drm_device *dev);
 		void (*cleanup_ring)(struct intel_engine_cs *ring);
 		void (*stop_ring)(struct intel_engine_cs *ring);
@@ -2790,9 +2799,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,
 void i915_gem_execbuffer_move_to_active(struct list_head *vmas,
 					struct drm_i915_gem_request *req);
 void i915_gem_execbuffer_retire_commands(struct i915_execbuffer_params *params);
+void i915_gem_execbuff_release_batch_obj(struct drm_i915_gem_object *batch_obj);
 int i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 				   struct drm_i915_gem_execbuffer2 *args,
 				   struct list_head *vmas);
+int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params);
 int i915_gem_execbuffer(struct drm_device *dev, void *data,
 			struct drm_file *file_priv);
 int i915_gem_execbuffer2(struct drm_device *dev, void *data,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index e8ec49e..5bf7da6 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -5220,11 +5220,13 @@ int i915_gem_init(struct drm_device *dev)
 
 	if (!i915.enable_execlists) {
 		dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission;
+		dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final;
 		dev_priv->gt.init_rings = i915_gem_init_rings;
 		dev_priv->gt.cleanup_ring = intel_cleanup_ring_buffer;
 		dev_priv->gt.stop_ring = intel_stop_ring_buffer;
 	} else {
 		dev_priv->gt.execbuf_submit = intel_execlists_submission;
+		dev_priv->gt.execbuf_final = intel_execlists_submission_final;
 		dev_priv->gt.init_rings = intel_logical_rings_init;
 		dev_priv->gt.cleanup_ring = intel_logical_ring_cleanup;
 		dev_priv->gt.stop_ring = intel_logical_ring_stop;
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 0eca2b6..eff171d 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1181,41 +1181,38 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	struct drm_device *dev = params->dev;
 	struct intel_engine_cs *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	u64 exec_start, exec_len;
-	int instp_mode;
-	u32 instp_mask;
 	int ret;
 
-	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
-	instp_mask = I915_EXEC_CONSTANTS_MASK;
-	switch (instp_mode) {
+	params->instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
+	params->instp_mask = I915_EXEC_CONSTANTS_MASK;
+	switch (params->instp_mode) {
 	case I915_EXEC_CONSTANTS_REL_GENERAL:
 	case I915_EXEC_CONSTANTS_ABSOLUTE:
 	case I915_EXEC_CONSTANTS_REL_SURFACE:
-		if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
+		if (params->instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
 			DRM_DEBUG("non-0 rel constants mode on non-RCS\n");
 			return -EINVAL;
 		}
 
-		if (instp_mode != dev_priv->relative_constants_mode) {
+		if (params->instp_mode != dev_priv->relative_constants_mode) {
 			if (INTEL_INFO(dev)->gen < 4) {
 				DRM_DEBUG("no rel constants on pre-gen4\n");
 				return -EINVAL;
 			}
 
 			if (INTEL_INFO(dev)->gen > 5 &&
-			    instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
+			    params->instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
 				DRM_DEBUG("rel surface constants mode invalid on gen5+\n");
 				return -EINVAL;
 			}
 
 			/* The HW changed the meaning on this bit on gen6 */
 			if (INTEL_INFO(dev)->gen >= 6)
-				instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
+				params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
 		}
 		break;
 	default:
-		DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode);
+		DRM_DEBUG("execbuf with unknown constants: %d\n", params->instp_mode);
 		return -EINVAL;
 	}
 
@@ -1225,7 +1222,33 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	/* To be split into two functions here... */
+	ret = dev_priv->gt.execbuf_final(params);
+	if (ret)
+		return ret;
+
+	/*
+	 * Free everything that was stored in the QE structure (until the
+	 * scheduler arrives and does it instead):
+	 */
+	if (params->dispatch_flags & I915_DISPATCH_SECURE)
+		i915_gem_execbuff_release_batch_obj(params->batch_obj);
+
+	return 0;
+}
+
+/*
+ * This is the main function for adding a batch to the ring.
+ * It is called from the scheduler, with the struct_mutex already held.
+ */
+int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
+{
+	struct drm_i915_private *dev_priv = params->dev->dev_private;
+	struct intel_engine_cs  *ring = params->ring;
+	u64 exec_start, exec_len;
+	int ret;
+
+	/* The mutex must be acquired before calling this function */
+	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
 	intel_runtime_pm_get(dev_priv);
 
@@ -1246,7 +1269,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	     "%s didn't clear reload\n", ring->name);
 
 	if (ring == &dev_priv->ring[RCS] &&
-	    instp_mode != dev_priv->relative_constants_mode) {
+	    params->instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_ring_begin(params->request, 4);
 		if (ret)
 			goto error;
@@ -1254,19 +1277,19 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 		intel_ring_emit(ring, MI_NOOP);
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, INSTPM);
-		intel_ring_emit(ring, instp_mask << 16 | instp_mode);
+		intel_ring_emit(ring, params->instp_mask << 16 | params->instp_mode);
 		intel_ring_advance(ring);
 
-		dev_priv->relative_constants_mode = instp_mode;
+		dev_priv->relative_constants_mode = params->instp_mode;
 	}
 
-	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {
-		ret = i915_reset_gen7_sol_offsets(dev, params->request);
+	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
+		ret = i915_reset_gen7_sol_offsets(params->dev, params->request);
 		if (ret)
 			goto error;
 	}
 
-	exec_len   = args->batch_len;
+	exec_len   = params->args_batch_len;
 	exec_start = params->batch_obj_vm_offset +
 		     params->args_batch_start_offset;
 
@@ -1584,23 +1607,40 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->file                    = file;
 	params->ring                    = ring;
 	params->dispatch_flags          = dispatch_flags;
+	params->args_flags              = args->flags;
+	params->args_batch_len          = args->batch_len;
+	params->args_num_cliprects      = args->num_cliprects;
+	params->args_DR1                = args->DR1;
+	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
 	params->ctx                     = ctx;
 
 	ret = dev_priv->gt.execbuf_submit(params, args, &eb->vmas);
+	if (ret)
+		goto err_batch_unpin;
+
+	/* the request owns the ref now */
+	i915_gem_context_unreference(ctx);
 
-err_batch_unpin:
 	/*
-	 * FIXME: We crucially rely upon the active tracking for the (ppgtt)
-	 * batch vma for correctness. For less ugly and less fragility this
-	 * needs to be adjusted to also track the ggtt batch vma properly as
-	 * active.
+	 * The eb list is no longer required. The scheduler has extracted all
+	 * the information than needs to persist.
 	 */
+	eb_destroy(eb);
+
+	/*
+	 * Don't clean up everything that is now saved away in the queue.
+	 * Just unlock and return immediately.
+	 */
+	mutex_unlock(&dev->struct_mutex);
+
+	return 0;
+
+err_batch_unpin:
 	if (dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_object_ggtt_unpin(batch_obj);
+		i915_gem_execbuff_release_batch_obj(batch_obj);
 
 err:
-	/* the request owns the ref now */
 	i915_gem_context_unreference(ctx);
 	eb_destroy(eb);
 
@@ -1618,6 +1658,17 @@ pre_mutex_err:
 	return ret;
 }
 
+void i915_gem_execbuff_release_batch_obj(struct drm_i915_gem_object *batch_obj)
+{
+	/*
+	 * FIXME: We crucially rely upon the active tracking for the (ppgtt)
+	 * batch vma for correctness. For less ugly and less fragility this
+	 * needs to be adjusted to also track the ggtt batch vma properly as
+	 * active.
+	 */
+	i915_gem_object_ggtt_unpin(batch_obj);
+}
+
 /*
  * Legacy execbuffer just creates an exec2 list from the original exec object
  * list array and passes it to the real function.
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 4bf0ee6..014180c 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -869,35 +869,31 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 	struct drm_device       *dev = params->dev;
 	struct intel_engine_cs  *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct intel_ringbuffer *ringbuf = params->ctx->engine[ring->id].ringbuf;
-	u64 exec_start;
-	int instp_mode;
-	u32 instp_mask;
 	int ret;
 
-	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
-	instp_mask = I915_EXEC_CONSTANTS_MASK;
-	switch (instp_mode) {
+	params->instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
+	params->instp_mask = I915_EXEC_CONSTANTS_MASK;
+	switch (params->instp_mode) {
 	case I915_EXEC_CONSTANTS_REL_GENERAL:
 	case I915_EXEC_CONSTANTS_ABSOLUTE:
 	case I915_EXEC_CONSTANTS_REL_SURFACE:
-		if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
+		if (params->instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
 			DRM_DEBUG("non-0 rel constants mode on non-RCS\n");
 			return -EINVAL;
 		}
 
-		if (instp_mode != dev_priv->relative_constants_mode) {
-			if (instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
+		if (params->instp_mode != dev_priv->relative_constants_mode) {
+			if (params->instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
 				DRM_DEBUG("rel surface constants mode invalid on gen5+\n");
 				return -EINVAL;
 			}
 
 			/* The HW changed the meaning on this bit on gen6 */
-			instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
+			params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
 		}
 		break;
 	default:
-		DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode);
+		DRM_DEBUG("execbuf with unknown constants: %d\n", params->instp_mode);
 		return -EINVAL;
 	}
 
@@ -912,7 +908,34 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	/* To be split into two functions here... */
+	ret = dev_priv->gt.execbuf_final(params);
+	if (ret)
+		return ret;
+
+	/*
+	 * Free everything that was stored in the QE structure (until the
+	 * scheduler arrives and does it instead):
+	 */
+	if (params->dispatch_flags & I915_DISPATCH_SECURE)
+		i915_gem_execbuff_release_batch_obj(params->batch_obj);
+
+	return 0;
+}
+
+/*
+ * This is the main function for adding a batch to the ring.
+ * It is called from the scheduler, with the struct_mutex already held.
+ */
+int intel_execlists_submission_final(struct i915_execbuffer_params *params)
+{
+	struct drm_i915_private *dev_priv = params->dev->dev_private;
+	struct intel_ringbuffer *ringbuf = params->request->ringbuf;
+	struct intel_engine_cs *ring = params->ring;
+	u64 exec_start;
+	int ret;
+
+	/* The mutex must be acquired before calling this function */
+	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
@@ -923,7 +946,7 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 		return ret;
 
 	if (ring == &dev_priv->ring[RCS] &&
-	    instp_mode != dev_priv->relative_constants_mode) {
+	    params->instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_logical_ring_begin(params->request, 4);
 		if (ret)
 			return ret;
@@ -931,14 +954,14 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 		intel_logical_ring_emit(ringbuf, MI_NOOP);
 		intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1));
 		intel_logical_ring_emit(ringbuf, INSTPM);
-		intel_logical_ring_emit(ringbuf, instp_mask << 16 | instp_mode);
+		intel_logical_ring_emit(ringbuf, params->instp_mask << 16 | params->instp_mode);
 		intel_logical_ring_advance(ringbuf);
 
-		dev_priv->relative_constants_mode = instp_mode;
+		dev_priv->relative_constants_mode = params->instp_mode;
 	}
 
 	exec_start = params->batch_obj_vm_offset +
-		     args->batch_start_offset;
+		     params->args_batch_start_offset;
 
 	ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags);
 	if (ret)
diff --git a/drivers/gpu/drm/i915/intel_lrc.h b/drivers/gpu/drm/i915/intel_lrc.h
index 4e60d54..8d9bad7 100644
--- a/drivers/gpu/drm/i915/intel_lrc.h
+++ b/drivers/gpu/drm/i915/intel_lrc.h
@@ -93,6 +93,7 @@ struct i915_execbuffer_params;
 int intel_execlists_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas);
+int intel_execlists_submission_final(struct i915_execbuffer_params *params);
 u32 intel_execlists_ctx_id(struct drm_i915_gem_object *ctx_obj);
 
 void intel_lrc_irq_handler(struct intel_engine_cs *ring);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 05/38] drm/i915: Cache request pointer in *_submission_final()

[John.C.Harrison]

From: Dave Gordon <david.s.gordon@intel.com>

Keep a local copy of the request pointer in the _final() functions
rather than dereferencing the params block repeatedly.

v3: New patch in series.

For: VIZ-1587
Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++------
 drivers/gpu/drm/i915/intel_lrc.c           | 11 ++++++-----
 2 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index eff171d..0ad32f6 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1243,6 +1243,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 {
 	struct drm_i915_private *dev_priv = params->dev->dev_private;
+	struct drm_i915_gem_request *req = params->request;
 	struct intel_engine_cs  *ring = params->ring;
 	u64 exec_start, exec_len;
 	int ret;
@@ -1256,12 +1257,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
-	ret = intel_ring_invalidate_all_caches(params->request);
+	ret = intel_ring_invalidate_all_caches(req);
 	if (ret)
 		goto error;
 
 	/* Switch to the correct context for the batch */
-	ret = i915_switch_context(params->request);
+	ret = i915_switch_context(req);
 	if (ret)
 		goto error;
 
@@ -1270,7 +1271,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_ring_begin(params->request, 4);
+		ret = intel_ring_begin(req, 4);
 		if (ret)
 			goto error;
 
@@ -1284,7 +1285,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	}
 
 	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
-		ret = i915_reset_gen7_sol_offsets(params->dev, params->request);
+		ret = i915_reset_gen7_sol_offsets(params->dev, req);
 		if (ret)
 			goto error;
 	}
@@ -1293,13 +1294,13 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	exec_start = params->batch_obj_vm_offset +
 		     params->args_batch_start_offset;
 
-	ret = ring->dispatch_execbuffer(params->request,
+	ret = ring->dispatch_execbuffer(req,
 					exec_start, exec_len,
 					params->dispatch_flags);
 	if (ret)
 		goto error;
 
-	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
+	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 014180c..a344a3a 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -929,7 +929,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 {
 	struct drm_i915_private *dev_priv = params->dev->dev_private;
-	struct intel_ringbuffer *ringbuf = params->request->ringbuf;
+	struct drm_i915_gem_request *req = params->request;
+	struct intel_ringbuffer *ringbuf = req->ringbuf;
 	struct intel_engine_cs *ring = params->ring;
 	u64 exec_start;
 	int ret;
@@ -941,13 +942,13 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
-	ret = logical_ring_invalidate_all_caches(params->request);
+	ret = logical_ring_invalidate_all_caches(req);
 	if (ret)
 		return ret;
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_logical_ring_begin(params->request, 4);
+		ret = intel_logical_ring_begin(req, 4);
 		if (ret)
 			return ret;
 
@@ -963,11 +964,11 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	exec_start = params->batch_obj_vm_offset +
 		     params->args_batch_start_offset;
 
-	ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags);
+	ret = ring->emit_bb_start(req, exec_start, params->dispatch_flags);
 	if (ret)
 		return ret;
 
-	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
+	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 06/38] drm/i915: Re-instate request->uniq because it is extremely useful

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The seqno value cannot always be used when debugging issues via trace
points. This is because it can be reset back to start, especially
during TDR type tests. Also, when the scheduler arrives the seqno is
only valid while a given request is executing on the hardware. While
the request is simply queued waiting for submission, it's seqno value
will be zero (meaning invalid).

v4: Wrapped a long line to keep the style checker happy.

For: VIZ-5115
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
Reviewed-by: Tomas Elf <tomas.elf@intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h   |  5 +++++
 drivers/gpu/drm/i915/i915_gem.c   |  4 +++-
 drivers/gpu/drm/i915/i915_trace.h | 25 +++++++++++++++++--------
 3 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 2520459..10dffdd 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1963,6 +1963,8 @@ struct drm_i915_private {
 	/* perform PHY state sanity checks? */
 	bool chv_phy_assert[2];
 
+	uint32_t request_uniq;
+
 	/*
 	 * NOTE: This is the dri1/ums dungeon, don't add stuff here. Your patch
 	 * will be rejected. Instead look for a better place.
@@ -2210,6 +2212,9 @@ struct drm_i915_gem_request {
 	/** GEM sequence number associated with this request. */
 	uint32_t seqno;
 
+	/* Unique identifier which can be used for trace points & debug */
+	uint32_t uniq;
+
 	/** Position in the ringbuffer of the start of the request */
 	u32 head;
 
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 5bf7da6..cff3768 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2904,7 +2904,8 @@ static void i915_gem_request_fence_value_str(struct fence *req_fence,
 
 	req = container_of(req_fence, typeof(*req), fence);
 
-	snprintf(str, size, "%d [%d]", req->fence.seqno, req->seqno);
+	snprintf(str, size, "%d [%d:%d]", req->fence.seqno, req->uniq,
+		 req->seqno);
 }
 
 static const struct fence_ops i915_gem_request_fops = {
@@ -2980,6 +2981,7 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring,
 
 	req->i915 = dev_priv;
 	req->ring = ring;
+	req->uniq = dev_priv->request_uniq++;
 	req->ctx  = ctx;
 	i915_gem_context_reference(req->ctx);
 
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index b3ae894..4c03754 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -469,6 +469,7 @@ TRACE_EVENT(i915_gem_ring_sync_to,
 			     __field(u32, dev)
 			     __field(u32, sync_from)
 			     __field(u32, sync_to)
+			     __field(u32, uniq_to)
 			     __field(u32, seqno)
 			     ),
 
@@ -476,13 +477,14 @@ TRACE_EVENT(i915_gem_ring_sync_to,
 			   __entry->dev = from->dev->primary->index;
 			   __entry->sync_from = from->id;
 			   __entry->sync_to = to_req->ring->id;
+			   __entry->uniq_to = to_req->uniq;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   ),
 
-	    TP_printk("dev=%u, sync-from=%u, sync-to=%u, seqno=%u",
+	    TP_printk("dev=%u, sync-from=%u, sync-to=%u, seqno=%u, to_uniq=%u",
 		      __entry->dev,
 		      __entry->sync_from, __entry->sync_to,
-		      __entry->seqno)
+		      __entry->seqno, __entry->uniq_to)
 );
 
 TRACE_EVENT(i915_gem_ring_dispatch,
@@ -517,6 +519,7 @@ TRACE_EVENT(i915_gem_ring_flush,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, invalidate)
 			     __field(u32, flush)
 			     ),
@@ -524,12 +527,13 @@ TRACE_EVENT(i915_gem_ring_flush,
 	    TP_fast_assign(
 			   __entry->dev = req->ring->dev->primary->index;
 			   __entry->ring = req->ring->id;
+			   __entry->uniq = req->uniq;
 			   __entry->invalidate = invalidate;
 			   __entry->flush = flush;
 			   ),
 
-	    TP_printk("dev=%u, ring=%x, invalidate=%04x, flush=%04x",
-		      __entry->dev, __entry->ring,
+	    TP_printk("dev=%u, ring=%x, request=%u, invalidate=%04x, flush=%04x",
+		      __entry->dev, __entry->ring, __entry->uniq,
 		      __entry->invalidate, __entry->flush)
 );
 
@@ -540,6 +544,7 @@ DECLARE_EVENT_CLASS(i915_gem_request,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, seqno)
 			     ),
 
@@ -548,11 +553,13 @@ DECLARE_EVENT_CLASS(i915_gem_request,
 						i915_gem_request_get_ring(req);
 			   __entry->dev = ring->dev->primary->index;
 			   __entry->ring = ring->id;
+			   __entry->uniq = req ? req->uniq : 0;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   ),
 
-	    TP_printk("dev=%u, ring=%u, seqno=%u",
-		      __entry->dev, __entry->ring, __entry->seqno)
+	    TP_printk("dev=%u, ring=%u, uniq=%u, seqno=%u",
+		      __entry->dev, __entry->ring, __entry->uniq,
+		      __entry->seqno)
 );
 
 DEFINE_EVENT(i915_gem_request, i915_gem_request_add,
@@ -601,6 +608,7 @@ TRACE_EVENT(i915_gem_request_wait_begin,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, seqno)
 			     __field(bool, blocking)
 			     ),
@@ -616,13 +624,14 @@ TRACE_EVENT(i915_gem_request_wait_begin,
 						i915_gem_request_get_ring(req);
 			   __entry->dev = ring->dev->primary->index;
 			   __entry->ring = ring->id;
+			   __entry->uniq = req ? req->uniq : 0;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   __entry->blocking =
 				     mutex_is_locked(&ring->dev->struct_mutex);
 			   ),
 
-	    TP_printk("dev=%u, ring=%u, seqno=%u, blocking=%s",
-		      __entry->dev, __entry->ring,
+	    TP_printk("dev=%u, ring=%u, uniq=%u, seqno=%u, blocking=%s",
+		      __entry->dev, __entry->ring, __entry->uniq,
 		      __entry->seqno, __entry->blocking ?  "yes (NB)" : "no")
 );
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 07/38] drm/i915: Start of GPU scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Initial creation of scheduler source files. Note that this patch
implements most of the scheduler functionality but does not hook it in
to the driver yet. It also leaves the scheduler code in 'pass through'
mode so that even when it is hooked in, it will not actually do very
much. This allows the hooks to be added one at a time in bite size
chunks and only when the scheduler is finally enabled at the end does
anything start happening.

The general theory of operation is that when batch buffers are
submitted to the driver, the execbuffer() code packages up all the
information required to execute the batch buffer at a later time. This
package is given over to the scheduler which adds it to an internal
node list. The scheduler also scans the list of objects associated
with the batch buffer and compares them against the objects already in
use by other buffers in the node list. If matches are found then the
new batch buffer node is marked as being dependent upon the matching
node. The same is done for the context object. The scheduler also
bumps up the priority of such matching nodes on the grounds that the
more dependencies a given batch buffer has the more important it is
likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers
in flight on the hardware at any given time. If fewer than this are
currently executing when a new node is queued, then the node is passed
straight through to the submit function. Otherwise it is simply added
to the queue and the driver returns back to user land.

The scheduler is notified when each batch buffer completes and updates
its internal tracking accordingly. At the end of the completion
interrupt processing, if any scheduler tracked batches were processed,
the scheduler's deferred worker thread is woken up. This can do more
involved processing such as actually removing completed nodes from the
queue and freeing up the resources associated with them (internal
memory allocations, DRM object references, context reference, etc.).
The work handler also checks the in flight count and calls the
submission code if a new slot has appeared.

When the scheduler's submit code is called, it scans the queued node
list for the highest priority node that has no unmet dependencies.
Note that the dependency calculation is complex as it must take
inter-ring dependencies and potential preemptions into account. Note
also that in the future this will be extended to include external
dependencies such as the Android Native Sync file descriptors and/or
the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for
submission to the hardware. The in flight count is then re-checked and
a new node popped from the list if appropriate.

Note that this patch does not implement pre-emptive scheduling. Only
basic scheduling by re-ordering batch buffer submission is currently
implemented. Pre-emption of actively executing batch buffers comes in
the next patch series.

v2: Changed priority levels to +/-1023 due to feedback from Chris
Wilson.

Removed redundant index from scheduler node.

Changed time stamps to use jiffies instead of raw monotonic. This
provides lower resolution but improved compatibility with other i915
code.

Major re-write of completion tracking code due to struct fence
conversion. The scheduler no longer has it's own private IRQ handler
but just lets the existing request code handle completion events.
Instead, the scheduler now hooks into the request notify code to be
told when a request has completed.

Reduced driver mutex locking scope. Removal of scheduler nodes no
longer grabs the mutex lock.

v3: Refactor of dependency generation to make the code more readable.
Also added in read-read optimisation support - i.e., don't treat a
shared read-only buffer as being a dependency.

Allowed the killing of queued nodes rather than only flying ones.

v4: Updated the commit message to better reflect the current state of
the code. Downgraded some BUG_ONs to WARN_ONs. Used the correct array
memory allocator function (kmalloc_array instead of kmalloc).
Corrected the format of some comments. Wrapped some lines differently
to keep the style checker happy.

Fixed a WARN_ON when killing nodes. The dependency removal code checks
that nodes being destroyed do not have any oustanding dependencies
(which would imply they should not have been executed yet). In the
case of nodes being destroyed, e.g. due to context banning, then this
might well be the case - they have not been executed and do indeed
have outstanding dependencies.

Re-instated the code to disble interrupts when not in use. The
underlying problem causing broken IRQ reference counts seems to have
been fixed now.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/Makefile         |   1 +
 drivers/gpu/drm/i915/i915_drv.h       |   4 +
 drivers/gpu/drm/i915/i915_gem.c       |   5 +
 drivers/gpu/drm/i915/i915_scheduler.c | 797 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  91 ++++
 5 files changed, 898 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h

diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 15398c5..79cb38b 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -10,6 +10,7 @@ ccflags-y := -Werror
 i915-y := i915_drv.o \
 	  i915_irq.o \
 	  i915_params.o \
+	  i915_scheduler.o \
           i915_suspend.o \
 	  i915_sysfs.o \
 	  intel_csr.o \
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 10dffdd..38f423b 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {
 	struct drm_i915_gem_request     *request;
 };
 
+struct i915_scheduler;
+
 /* used in computing the new watermarks state */
 struct intel_wm_config {
 	unsigned int num_pipes_active;
@@ -1947,6 +1949,8 @@ struct drm_i915_private {
 
 	struct i915_runtime_pm pm;
 
+	struct i915_scheduler *scheduler;
+
 	/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */
 	struct {
 		int (*execbuf_submit)(struct i915_execbuffer_params *params,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index cff3768..47aa85b 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -37,6 +37,7 @@
 #include <linux/swap.h>
 #include <linux/pci.h>
 #include <linux/dma-buf.h>
+#include "i915_scheduler.h"
 
 #define RQ_BUG_ON(expr)
 
@@ -5242,6 +5243,10 @@ int i915_gem_init(struct drm_device *dev)
 	 */
 	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
+	ret = i915_scheduler_init(dev);
+	if (ret)
+		goto out_unlock;
+
 	ret = i915_gem_init_userptr(dev);
 	if (ret)
 		goto out_unlock;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
new file mode 100644
index 0000000..8cb9063
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -0,0 +1,797 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include "i915_drv.h"
+#include "intel_drv.h"
+#include "i915_scheduler.h"
+
+static int         i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node);
+static int         i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
+						   struct i915_scheduler_queue_entry *remove);
+static int         i915_scheduler_submit(struct intel_engine_cs *ring,
+					 bool is_locked);
+static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
+					       struct intel_engine_cs *ring);
+static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
+static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
+						struct i915_scheduler_queue_entry *target,
+						uint32_t bump);
+
+int i915_scheduler_init(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	int                     r;
+
+	if (scheduler)
+		return 0;
+
+	scheduler = kzalloc(sizeof(*scheduler), GFP_KERNEL);
+	if (!scheduler)
+		return -ENOMEM;
+
+	spin_lock_init(&scheduler->lock);
+
+	for (r = 0; r < I915_NUM_RINGS; r++)
+		INIT_LIST_HEAD(&scheduler->node_queue[r]);
+
+	/* Default tuning values: */
+	scheduler->priority_level_min     = -1023;
+	scheduler->priority_level_max     = 1023;
+	scheduler->priority_level_preempt = 900;
+	scheduler->min_flying             = 2;
+
+	dev_priv->scheduler = scheduler;
+
+	return 0;
+}
+
+static void i915_generate_dependencies(struct i915_scheduler *scheduler,
+				       struct i915_scheduler_queue_entry *node,
+				       uint32_t ring)
+{
+	struct i915_scheduler_obj_entry *this, *that;
+	struct i915_scheduler_queue_entry  *test;
+	int                 i, j;
+	bool                found;
+
+	list_for_each_entry(test, &scheduler->node_queue[ring], link) {
+		if (I915_SQS_IS_COMPLETE(test))
+			continue;
+
+		/*
+		 * Batches on the same ring for the same
+		 * context must be kept in order.
+		 */
+		found = (node->params.ctx == test->params.ctx) &&
+			(node->params.ring == test->params.ring);
+
+		if (found)
+			goto depends;
+
+		/*
+		 * Batches working on the same objects must
+		 * be kept in order.
+		 */
+		for (i = 0; (i < node->num_objs) && !found; i++) {
+			this = node->saved_objects + i;
+
+			for (j = 0; j < test->num_objs; j++) {
+				that = test->saved_objects + j;
+
+				if (this->obj != that->obj)
+					continue;
+
+				/* Only need to worry about writes */
+				if (this->read_only && that->read_only)
+					continue;
+
+				found = true;
+				break;
+			}
+		}
+
+		if (!found)
+			continue;
+
+depends:
+		node->dep_list[node->num_deps] = test;
+		node->num_deps++;
+	}
+}
+
+int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
+{
+	struct drm_i915_private *dev_priv = qe->params.dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring = qe->params.ring;
+	struct i915_scheduler_queue_entry  *node;
+	struct i915_scheduler_queue_entry  *test;
+	unsigned long       flags;
+	bool                not_flying;
+	int                 i, r;
+	int                 incomplete = 0;
+
+	WARN_ON(!scheduler);
+
+	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
+		int ret;
+
+		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
+		ret = dev_priv->gt.execbuf_final(&qe->params);
+		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
+
+		/*
+		 * Don't do any clean up on failure because the caller will
+		 * do it all anyway.
+		 */
+		if (ret)
+			return ret;
+
+		/* Free everything that is owned by the QE structure: */
+		kfree(qe->params.cliprects);
+		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
+
+		return 0;
+	}
+
+	node = kmalloc(sizeof(*node), GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	*node = *qe;
+	INIT_LIST_HEAD(&node->link);
+	node->status = i915_sqs_queued;
+	node->stamp  = jiffies;
+	i915_gem_request_reference(node->params.request);
+
+	/* Need to determine the number of incomplete entries in the list as
+	 * that will be the maximum size of the dependency list.
+	 *
+	 * Note that the allocation must not be made with the spinlock acquired
+	 * as kmalloc can sleep. However, the unlock/relock is safe because no
+	 * new entries can be queued up during the unlock as the i915 driver
+	 * mutex is still held. Entries could be removed from the list but that
+	 * just means the dep_list will be over-allocated which is fine.
+	 */
+	spin_lock_irqsave(&scheduler->lock, flags);
+	for (r = 0; r < I915_NUM_RINGS; r++) {
+		list_for_each_entry(test, &scheduler->node_queue[r], link) {
+			if (I915_SQS_IS_COMPLETE(test))
+				continue;
+
+			incomplete++;
+		}
+	}
+
+	/* Temporarily unlock to allocate memory: */
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+	if (incomplete) {
+		node->dep_list = kmalloc_array(incomplete,
+					       sizeof(*node->dep_list),
+					       GFP_KERNEL);
+		if (!node->dep_list) {
+			kfree(node);
+			return -ENOMEM;
+		}
+	} else
+		node->dep_list = NULL;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	node->num_deps = 0;
+
+	if (node->dep_list) {
+		for (r = 0; r < I915_NUM_RINGS; r++)
+			i915_generate_dependencies(scheduler, node, r);
+
+		WARN_ON(node->num_deps > incomplete);
+	}
+
+	if (node->priority > scheduler->priority_level_max)
+		node->priority = scheduler->priority_level_max;
+	else if (node->priority < scheduler->priority_level_min)
+		node->priority = scheduler->priority_level_min;
+
+	if ((node->priority > 0) && node->num_deps) {
+		i915_scheduler_priority_bump_clear(scheduler);
+
+		for (i = 0; i < node->num_deps; i++)
+			i915_scheduler_priority_bump(scheduler,
+					node->dep_list[i], node->priority);
+	}
+
+	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
+
+	not_flying = i915_scheduler_count_flying(scheduler, ring) <
+						 scheduler->min_flying;
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (not_flying)
+		i915_scheduler_submit(ring, true);
+
+	return 0;
+}
+
+static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
+{
+	struct drm_i915_private *dev_priv = node->params.dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring;
+
+	WARN_ON(!scheduler);
+	WARN_ON(!node);
+	WARN_ON(node->status != i915_sqs_popped);
+
+	ring = node->params.ring;
+
+	/*
+	 * Add the node (which should currently be in state popped) to the
+	 * front of the queue. This ensure that flying nodes are always held
+	 * in hardware submission order.
+	 */
+	list_add(&node->link, &scheduler->node_queue[ring->id]);
+
+	node->status = i915_sqs_flying;
+
+	if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
+		bool    success = true;
+
+		success = ring->irq_get(ring);
+		if (success)
+			scheduler->flags[ring->id] |= i915_sf_interrupts_enabled;
+		else
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * Nodes are considered valid dependencies if they are queued on any ring or
+ * if they are in flight on a different ring. In flight on the same ring is no
+ * longer interesting for non-premptive nodes as the ring serialises execution.
+ * For pre-empting nodes, all in flight dependencies are valid as they must not
+ * be jumped by the act of pre-empting.
+ *
+ * Anything that is neither queued nor flying is uninteresting.
+ */
+static inline bool i915_scheduler_is_dependency_valid(
+			struct i915_scheduler_queue_entry *node, uint32_t idx)
+{
+	struct i915_scheduler_queue_entry *dep;
+
+	dep = node->dep_list[idx];
+	if (!dep)
+		return false;
+
+	if (I915_SQS_IS_QUEUED(dep))
+		return true;
+
+	if (I915_SQS_IS_FLYING(dep)) {
+		if (node->params.ring != dep->params.ring)
+			return true;
+	}
+
+	return false;
+}
+
+static uint32_t i915_scheduler_count_flying(struct i915_scheduler *scheduler,
+					    struct intel_engine_cs *ring)
+{
+	struct i915_scheduler_queue_entry *node;
+	uint32_t                          flying = 0;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link)
+		if (I915_SQS_IS_FLYING(node))
+			flying++;
+
+	return flying;
+}
+
+/*
+ * Add a popped node back in to the queue. For example, because the ring was
+ * hung when execfinal() was called and thus the ring submission needs to be
+ * retried later.
+ */
+static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
+{
+	WARN_ON(!node);
+	WARN_ON(!I915_SQS_IS_FLYING(node));
+
+	/* Seqno will be reassigned on relaunch */
+	node->params.request->seqno = 0;
+	node->status = i915_sqs_queued;
+}
+
+/*
+ * Give up on a node completely. For example, because it is causing the
+ * ring to hang or is using some resource that no longer exists.
+ */
+static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
+{
+	WARN_ON(!node);
+	WARN_ON(I915_SQS_IS_COMPLETE(node));
+
+	node->status = i915_sqs_dead;
+}
+
+/*
+ * A sequence number has popped out of the hardware and the request handling
+ * code has mapped it back to a request and will mark that request complete.
+ * It also calls this function to notify the scheduler about the completion
+ * so the scheduler's node can be updated appropriately.
+ * Returns true if the request is scheduler managed, false if not.
+ */
+bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
+{
+	struct drm_i915_private *dev_priv  = to_i915(req->ring->dev);
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	/* XXX: Need to map back from request to node */
+	struct i915_scheduler_queue_entry *node = NULL;
+	unsigned long       flags;
+
+	if (!node)
+		return false;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	WARN_ON(!I915_SQS_IS_FLYING(node));
+
+	/* Node was in flight so mark it as complete. */
+	if (req->cancelled)
+		node->status = i915_sqs_dead;
+	else
+		node->status = i915_sqs_complete;
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	/*
+	 * XXX: If the in-flight list is now empty then new work should be
+	 * submitted. However, this function is called from interrupt context
+	 * and thus cannot acquire mutex locks and other such things that are
+	 * necessary for fresh submission.
+	 */
+
+	return true;
+}
+
+int i915_scheduler_remove(struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node, *node_next;
+	unsigned long       flags;
+	int                 flying = 0, queued = 0;
+	int                 ret = 0;
+	bool                do_submit;
+	uint32_t            min_seqno;
+	struct list_head    remove;
+
+	if (list_empty(&scheduler->node_queue[ring->id]))
+		return 0;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	/* /i915_scheduler_dump_locked(ring, "remove/pre");/ */
+
+	/*
+	 * In the case where the system is idle, starting 'min_seqno' from a big
+	 * number will cause all nodes to be removed as they are now back to
+	 * being in-order. However, this will be a problem if the last one to
+	 * complete was actually out-of-order as the ring seqno value will be
+	 * lower than one or more completed buffers. Thus code looking for the
+	 * completion of said buffers will wait forever.
+	 * Instead, use the hardware seqno as the starting point. This means
+	 * that some buffers might be kept around even in a completely idle
+	 * system but it should guarantee that no-one ever gets confused when
+	 * waiting for buffer completion.
+	 */
+	min_seqno = ring->get_seqno(ring, true);
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (I915_SQS_IS_QUEUED(node))
+			queued++;
+		else if (I915_SQS_IS_FLYING(node))
+			flying++;
+		else if (I915_SQS_IS_COMPLETE(node))
+			continue;
+
+		if (node->params.request->seqno == 0)
+			continue;
+
+		if (!i915_seqno_passed(node->params.request->seqno, min_seqno))
+			min_seqno = node->params.request->seqno;
+	}
+
+	INIT_LIST_HEAD(&remove);
+	list_for_each_entry_safe(node, node_next, &scheduler->node_queue[ring->id], link) {
+		/*
+		 * Only remove completed nodes which have a lower seqno than
+		 * all pending nodes. While there is the possibility of the
+		 * ring's seqno counting backwards, all higher buffers must
+		 * be remembered so that the 'i915_seqno_passed()' test can
+		 * report that they have in fact passed.
+		 *
+		 * NB: This is not true for 'dead' nodes. The GPU reset causes
+		 * the software seqno to restart from its initial value. Thus
+		 * the dead nodes must be removed even though their seqno values
+		 * are potentially vastly greater than the current ring seqno.
+		 */
+		if (!I915_SQS_IS_COMPLETE(node))
+			continue;
+
+		if (node->status != i915_sqs_dead) {
+			if (i915_seqno_passed(node->params.request->seqno, min_seqno) &&
+			    (node->params.request->seqno != min_seqno))
+				continue;
+		}
+
+		list_del(&node->link);
+		list_add(&node->link, &remove);
+
+		/* Strip the dependency info while the mutex is still locked */
+		i915_scheduler_remove_dependent(scheduler, node);
+
+		continue;
+	}
+
+	/*
+	 * Release the interrupt reference count if there are no longer any
+	 * nodes to worry about.
+	 */
+	if (!flying && !queued &&
+	    (scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
+		ring->irq_put(ring);
+		scheduler->flags[ring->id] &= ~i915_sf_interrupts_enabled;
+	}
+
+	/* Launch more packets now? */
+	do_submit = (queued > 0) && (flying < scheduler->min_flying);
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (!do_submit && list_empty(&remove))
+		return ret;
+
+	mutex_lock(&ring->dev->struct_mutex);
+
+	if (do_submit)
+		ret = i915_scheduler_submit(ring, true);
+
+	while (!list_empty(&remove)) {
+		node = list_first_entry(&remove, typeof(*node), link);
+		list_del(&node->link);
+
+		/*
+		 * The batch buffer must be unpinned before it is unreferenced
+		 * otherwise the unpin fails with a missing vma!?
+		 */
+		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
+
+		/* Free everything that is owned by the node: */
+		i915_gem_request_unreference(node->params.request);
+		kfree(node->params.cliprects);
+		kfree(node->dep_list);
+		kfree(node);
+	}
+
+	mutex_unlock(&ring->dev->struct_mutex);
+
+	return ret;
+}
+
+static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
+{
+	struct i915_scheduler_queue_entry *node;
+	int i;
+
+	/*
+	 * Ensure circular dependencies don't cause problems and that a bump
+	 * by object usage only bumps each using buffer once:
+	 */
+	for (i = 0; i < I915_NUM_RINGS; i++) {
+		list_for_each_entry(node, &scheduler->node_queue[i], link)
+			node->bumped = false;
+	}
+}
+
+static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
+				struct i915_scheduler_queue_entry *target,
+				uint32_t bump)
+{
+	uint32_t new_priority;
+	int      i, count;
+
+	if (target->priority >= scheduler->priority_level_max)
+		return 1;
+
+	if (target->bumped)
+		return 0;
+
+	new_priority = target->priority + bump;
+	if ((new_priority <= target->priority) ||
+	    (new_priority > scheduler->priority_level_max))
+		target->priority = scheduler->priority_level_max;
+	else
+		target->priority = new_priority;
+
+	count = 1;
+	target->bumped = true;
+
+	for (i = 0; i < target->num_deps; i++) {
+		if (!target->dep_list[i])
+			continue;
+
+		if (target->dep_list[i]->bumped)
+			continue;
+
+		count += i915_scheduler_priority_bump(scheduler,
+						      target->dep_list[i],
+						      bump);
+	}
+
+	return count;
+}
+
+static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
+				struct i915_scheduler_queue_entry **pop_node,
+				unsigned long *flags)
+{
+	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *best;
+	struct i915_scheduler_queue_entry  *node;
+	int     ret;
+	int     i;
+	bool	any_queued;
+	bool	has_local, has_remote, only_remote;
+
+	*pop_node = NULL;
+	ret = -ENODATA;
+
+	any_queued = false;
+	only_remote = false;
+	best = NULL;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+		any_queued = true;
+
+		has_local  = false;
+		has_remote = false;
+		for (i = 0; i < node->num_deps; i++) {
+			if (!i915_scheduler_is_dependency_valid(node, i))
+				continue;
+
+			if (node->dep_list[i]->params.ring == node->params.ring)
+				has_local = true;
+			else
+				has_remote = true;
+		}
+
+		if (has_remote && !has_local)
+			only_remote = true;
+
+		if (!has_local && !has_remote) {
+			if (!best ||
+			    (node->priority > best->priority))
+				best = node;
+		}
+	}
+
+	if (best) {
+		list_del(&best->link);
+
+		INIT_LIST_HEAD(&best->link);
+		best->status  = i915_sqs_popped;
+
+		ret = 0;
+	} else {
+		/* Can only get here if:
+		 * (a) there are no buffers in the queue
+		 * (b) all queued buffers are dependent on other buffers
+		 *     e.g. on a buffer that is in flight on a different ring
+		 */
+		if (only_remote) {
+			/* The only dependent buffers are on another ring. */
+			ret = -EAGAIN;
+		} else if (any_queued) {
+			/* It seems that something has gone horribly wrong! */
+			DRM_ERROR("Broken dependency tracking on ring %d!\n",
+				  (int) ring->id);
+		}
+	}
+
+	/* i915_scheduler_dump_queue_pop(ring, best); */
+
+	*pop_node = best;
+	return ret;
+}
+
+static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
+{
+	struct drm_device   *dev = ring->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	unsigned long       flags;
+	int                 ret = 0, count = 0;
+
+	if (!was_locked) {
+		ret = i915_mutex_lock_interruptible(dev);
+		if (ret)
+			return ret;
+	}
+
+	WARN_ON(!mutex_is_locked(&dev->struct_mutex));
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	/* First time around, complain if anything unexpected occurs: */
+	ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
+	if (ret) {
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (!was_locked)
+			mutex_unlock(&dev->struct_mutex);
+
+		return ret;
+	}
+
+	do {
+		WARN_ON(!node);
+		WARN_ON(node->params.ring != ring);
+		WARN_ON(node->status != i915_sqs_popped);
+		count++;
+
+		/*
+		 * The call to pop above will have removed the node from the
+		 * list. So add it back in and mark it as in flight.
+		 */
+		i915_scheduler_fly_node(node);
+
+		scheduler->flags[ring->id] |= i915_sf_submitting;
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+		ret = dev_priv->gt.execbuf_final(&node->params);
+		spin_lock_irqsave(&scheduler->lock, flags);
+		scheduler->flags[ring->id] &= ~i915_sf_submitting;
+
+		if (ret) {
+			int requeue = 1;
+
+			/*
+			 * Oh dear! Either the node is broken or the ring is
+			 * busy. So need to kill the node or requeue it and try
+			 * again later as appropriate.
+			 */
+
+			switch (-ret) {
+			case ENODEV:
+			case ENOENT:
+				/* Fatal errors. Kill the node. */
+				requeue = -1;
+			break;
+
+			case EAGAIN:
+			case EBUSY:
+			case EIO:
+			case ENOMEM:
+			case ERESTARTSYS:
+			case EINTR:
+				/* Supposedly recoverable errors. */
+			break;
+
+			default:
+				/*
+				 * Assume the error is recoverable and hope
+				 * for the best.
+				 */
+				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
+						 ring->name, ret);
+			break;
+			}
+
+			/*
+			 * Check that the watchdog/reset code has not nuked
+			 * the node while we weren't looking:
+			 */
+			if (node->status == i915_sqs_dead)
+				requeue = 0;
+
+			if (requeue == 1) {
+				i915_scheduler_node_requeue(node);
+				/*
+				 * No point spinning if the ring is currently
+				 * unavailable so just give up and come back
+				 * later.
+				 */
+				break;
+			} else if (requeue == -1)
+				i915_scheduler_node_kill(node);
+		}
+
+		/* Keep launching until the sky is sufficiently full. */
+		if (i915_scheduler_count_flying(scheduler, ring) >=
+						scheduler->min_flying)
+			break;
+
+		ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
+	} while (ret == 0);
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (!was_locked)
+		mutex_unlock(&dev->struct_mutex);
+
+	/* Don't complain about not being able to submit extra entries */
+	if (ret == -ENODATA)
+		ret = 0;
+
+	return (ret < 0) ? ret : count;
+}
+
+static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
+				struct i915_scheduler_queue_entry *remove)
+{
+	struct i915_scheduler_queue_entry  *node;
+	int     i, r;
+	int     count = 0;
+
+	/*
+	 * Ensure that a node is not being removed which is still dependent
+	 * upon other (not completed) work. If that happens, it implies
+	 * something has gone very wrong with the dependency tracking! Note
+	 * that there is no need to worry if this node has been explicitly
+	 * killed for some reason - it might be being killed before it got
+	 * sent to the hardware.
+	 */
+	if (remove->status != i915_sqs_dead) {
+		for (i = 0; i < remove->num_deps; i++)
+			if ((remove->dep_list[i]) &&
+			    (!I915_SQS_IS_COMPLETE(remove->dep_list[i])))
+				count++;
+		WARN_ON(count);
+	}
+
+	/*
+	 * Remove this node from the dependency lists of any other node which
+	 * might be waiting on it.
+	 */
+	for (r = 0; r < I915_NUM_RINGS; r++) {
+		list_for_each_entry(node, &scheduler->node_queue[r], link) {
+			for (i = 0; i < node->num_deps; i++) {
+				if (node->dep_list[i] != remove)
+					continue;
+
+				node->dep_list[i] = NULL;
+			}
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
new file mode 100644
index 0000000..00dc7f3
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef _I915_SCHEDULER_H_
+#define _I915_SCHEDULER_H_
+
+enum i915_scheduler_queue_status {
+	/* Limbo: */
+	i915_sqs_none = 0,
+	/* Not yet submitted to hardware: */
+	i915_sqs_queued,
+	/* Popped from queue, ready to fly: */
+	i915_sqs_popped,
+	/* Sent to hardware for processing: */
+	i915_sqs_flying,
+	/* Finished processing on the hardware: */
+	i915_sqs_complete,
+	/* Killed by watchdog or catastrophic submission failure: */
+	i915_sqs_dead,
+	/* Limit value for use with arrays/loops */
+	i915_sqs_MAX
+};
+
+#define I915_SQS_IS_QUEUED(node)	(((node)->status == i915_sqs_queued))
+#define I915_SQS_IS_FLYING(node)	(((node)->status == i915_sqs_flying))
+#define I915_SQS_IS_COMPLETE(node)	(((node)->status == i915_sqs_complete) || \
+					 ((node)->status == i915_sqs_dead))
+
+struct i915_scheduler_obj_entry {
+	struct drm_i915_gem_object          *obj;
+	bool                                read_only;
+};
+
+struct i915_scheduler_queue_entry {
+	struct i915_execbuffer_params       params;
+	/* -1023 = lowest priority, 0 = default, 1023 = highest */
+	int32_t                             priority;
+	struct i915_scheduler_obj_entry     *saved_objects;
+	int                                 num_objs;
+	bool                                bumped;
+	struct i915_scheduler_queue_entry   **dep_list;
+	int                                 num_deps;
+	enum i915_scheduler_queue_status    status;
+	unsigned long                       stamp;
+	struct list_head                    link;
+};
+
+struct i915_scheduler {
+	struct list_head    node_queue[I915_NUM_RINGS];
+	uint32_t            flags[I915_NUM_RINGS];
+	spinlock_t          lock;
+
+	/* Tuning parameters: */
+	int32_t             priority_level_min;
+	int32_t             priority_level_max;
+	int32_t             priority_level_preempt;
+	uint32_t            min_flying;
+};
+
+/* Flag bits for i915_scheduler::flags */
+enum {
+	i915_sf_interrupts_enabled  = (1 << 0),
+	i915_sf_submitting          = (1 << 1),
+};
+
+int         i915_scheduler_init(struct drm_device *dev);
+int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
+bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
+
+#endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 08/38] drm/i915: Prepare retire_requests to handle out-of-order seqnos

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

A major point of the GPU scheduler is that it re-orders batch buffers
after they have been submitted to the driver. This leads to requests
completing out of order. In turn, this means that the retire
processing can no longer assume that all completed entries are at the
front of the list. Rather than attempting to re-order the request list
on a regular basis, it is better to simply scan the entire list.

v2: Removed deferred free code as no longer necessary due to request
handling updates.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c | 31 +++++++++++++------------------
 1 file changed, 13 insertions(+), 18 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 47aa85b..d4f1d63 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3184,6 +3184,7 @@ void i915_gem_reset(struct drm_device *dev)
 void
 i915_gem_retire_requests_ring(struct intel_engine_cs *ring)
 {
+	struct drm_i915_gem_object *obj, *obj_next;
 	struct drm_i915_gem_request *req, *req_next;
 	LIST_HEAD(list_head);
 
@@ -3196,37 +3197,31 @@ i915_gem_retire_requests_ring(struct intel_engine_cs *ring)
 	 */
 	i915_gem_request_notify(ring, false);
 
+	/*
+	 * Note that request entries might be out of order due to rescheduling
+	 * and pre-emption. Thus both lists must be processed in their entirety
+	 * rather than stopping at the first non-complete entry.
+	 */
+
 	/* Retire requests first as we use it above for the early return.
 	 * If we retire requests last, we may use a later seqno and so clear
 	 * the requests lists without clearing the active list, leading to
 	 * confusion.
 	 */
-	while (!list_empty(&ring->request_list)) {
-		struct drm_i915_gem_request *request;
-
-		request = list_first_entry(&ring->request_list,
-					   struct drm_i915_gem_request,
-					   list);
-
-		if (!i915_gem_request_completed(request))
-			break;
+	list_for_each_entry_safe(req, req_next, &ring->request_list, list) {
+		if (!i915_gem_request_completed(req))
+			continue;
 
-		i915_gem_request_retire(request);
+		i915_gem_request_retire(req);
 	}
 
 	/* Move any buffers on the active list that are no longer referenced
 	 * by the ringbuffer to the flushing/inactive lists as appropriate,
 	 * before we free the context associated with the requests.
 	 */
-	while (!list_empty(&ring->active_list)) {
-		struct drm_i915_gem_object *obj;
-
-		obj = list_first_entry(&ring->active_list,
-				      struct drm_i915_gem_object,
-				      ring_list[ring->id]);
-
+	list_for_each_entry_safe(obj, obj_next, &ring->active_list, ring_list[ring->id]) {
 		if (!list_empty(&obj->last_read_req[ring->id]->list))
-			break;
+			continue;
 
 		i915_gem_object_retire__read(obj, ring->id);
 	}
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 09/38] drm/i915: Disable hardware semaphores when GPU scheduler is enabled

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Hardware sempahores require seqno values to be continuously
incrementing. However, the scheduler's reordering of batch buffers
means that the seqno values going through the hardware could be out of
order. Thus semaphores can not be used.

On the other hand, the scheduler superceeds the need for hardware
semaphores anyway. Having one ring stall waiting for something to
complete on another ring is inefficient if that ring could be working
on some other, independent task. This is what the scheduler is meant
to do - keep the hardware as busy as possible by reordering batch
buffers to avoid dependency stalls.

v4: Downgraded a BUG_ON to WARN_ON as the latter is preferred.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.c         | 9 +++++++++
 drivers/gpu/drm/i915/i915_scheduler.c   | 7 +++++++
 drivers/gpu/drm/i915/i915_scheduler.h   | 1 +
 drivers/gpu/drm/i915/intel_ringbuffer.c | 4 ++++
 4 files changed, 21 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 858d58c..e6be8f5 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -34,6 +34,7 @@
 #include "i915_drv.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
+#include "i915_scheduler.h"
 
 #include <linux/console.h>
 #include <linux/module.h>
@@ -581,6 +582,14 @@ void intel_detect_pch(struct drm_device *dev)
 
 bool i915_semaphore_is_enabled(struct drm_device *dev)
 {
+	/* Hardware semaphores are not compatible with the scheduler due to the
+	 * seqno values being potentially out of order. However, semaphores are
+	 * also not required as the scheduler will handle interring dependencies
+	 * and try do so in a way that does not cause dead time on the hardware.
+	 */
+	if (i915_scheduler_is_enabled(dev))
+		return false;
+
 	if (INTEL_INFO(dev)->gen < 6)
 		return false;
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 8cb9063..fc48955 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -38,6 +38,13 @@ static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler
 						struct i915_scheduler_queue_entry *target,
 						uint32_t bump);
 
+bool i915_scheduler_is_enabled(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+
+	return dev_priv->scheduler != NULL;
+}
+
 int i915_scheduler_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 00dc7f3..2d50d83 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -84,6 +84,7 @@ enum {
 	i915_sf_submitting          = (1 << 1),
 };
 
+bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 1dec252..a93bbce 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -32,6 +32,7 @@
 #include <drm/i915_drm.h>
 #include "i915_trace.h"
 #include "intel_drv.h"
+#include "i915_scheduler.h"
 
 bool
 intel_ring_initialized(struct intel_engine_cs *ring)
@@ -1411,6 +1412,9 @@ gen6_ring_sync(struct drm_i915_gem_request *waiter_req,
 	u32 wait_mbox = signaller->semaphore.mbox.wait[waiter->id];
 	int ret;
 
+	/* Arithmetic on sequence numbers is unreliable with a scheduler. */
+	WARN_ON(i915_scheduler_is_enabled(signaller->dev));
+
 	/* Throughout all of the GEM code, seqno passed implies our current
 	 * seqno is >= the last seqno executed. However for hardware the
 	 * comparison is strictly greater than.
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 10/38] drm/i915: Force MMIO flips when scheduler enabled

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

MMIO flips are the preferred mechanism now but more importantly, pipe
based flips cause issues for the scheduler. Specifically, submitting
work to the rings around the side of the scheduler could cause that
work to be lost if the scheduler generates a pre-emption event on that
ring.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/intel_display.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 9291a1d..c00bc50 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -44,6 +44,7 @@
 #include <drm/drm_plane_helper.h>
 #include <drm/drm_rect.h>
 #include <linux/dma_remapping.h>
+#include "i915_scheduler.h"
 
 /* Primary plane formats for gen <= 3 */
 static const uint32_t i8xx_primary_formats[] = {
@@ -11135,6 +11136,8 @@ static bool use_mmio_flip(struct intel_engine_cs *ring,
 		return true;
 	else if (i915.enable_execlists)
 		return true;
+	else if (i915_scheduler_is_enabled(ring->dev))
+		return true;
 	else
 		return ring != i915_gem_request_get_ring(obj->last_write_req);
 }
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 11/38] drm/i915: Added scheduler hook when closing DRM file handles

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler decouples the submission of batch buffers to the driver
with submission of batch buffers to the hardware. Thus it is possible
for an application to close its DRM file handle while there is still
work outstanding. That means the scheduler needs to know about file
close events so it can remove the file pointer from such orphaned
batch buffers and not attempt to dereference it later.

v3: Updated to not wait for outstanding work to complete but merely
remove the file handle reference. The wait was getting excessively
complicated with inter-ring dependencies, pre-emption, and other such
issues.

v4: Changed some white space to keep the style checker happy.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_dma.c       |  3 +++
 drivers/gpu/drm/i915/i915_scheduler.c | 35 +++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  2 ++
 3 files changed, 40 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 731cf31..c2f9c03 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -46,6 +46,7 @@
 #include <linux/vga_switcheroo.h>
 #include <linux/slab.h>
 #include <acpi/video.h>
+#include "i915_scheduler.h"
 #include <linux/pm.h>
 #include <linux/pm_runtime.h>
 #include <linux/oom.h>
@@ -1250,6 +1251,8 @@ void i915_driver_lastclose(struct drm_device *dev)
 
 void i915_driver_preclose(struct drm_device *dev, struct drm_file *file)
 {
+	i915_scheduler_closefile(dev, file);
+
 	mutex_lock(&dev->struct_mutex);
 	i915_gem_context_close(dev, file);
 	i915_gem_release(dev, file);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index fc48955..26cd088 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -802,3 +802,38 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
 
 	return 0;
 }
+
+int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
+{
+	struct i915_scheduler_queue_entry  *node;
+	struct drm_i915_private            *dev_priv = dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring;
+	int                     i;
+	unsigned long           flags;
+
+	if (!scheduler)
+		return 0;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	for_each_ring(ring, dev_priv, i) {
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (node->params.file != file)
+				continue;
+
+			if (!I915_SQS_IS_COMPLETE(node))
+				DRM_DEBUG_DRIVER("Closing file handle with outstanding work: %d:%d/%d on %s\n",
+						 node->params.request->uniq,
+						 node->params.request->seqno,
+						 node->status,
+						 ring->name);
+
+			node->params.file = NULL;
+		}
+	}
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 2d50d83..02ac6f2 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -86,6 +86,8 @@ enum {
 
 bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
+int         i915_scheduler_closefile(struct drm_device *dev,
+				     struct drm_file *file);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 12/38] drm/i915: Added scheduler hook into i915_gem_request_notify()

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler needs to know when requests have completed so that it
can keep its own internal state up to date and can submit new requests
to the hardware from its queue.

v2: Updated due to changes in request handling. The operation is now
reversed from before. Rather than the scheduler being in control of
completion events, it is now the request code itself. The scheduler
merely receives a notification event. It can then optionally request
it's worker thread be woken up after all completion processing is
complete.

v4: Downgraded a BUG_ON to a WARN_ON as the latter is preferred.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h       |  2 ++
 drivers/gpu/drm/i915/i915_gem.c       | 16 ++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.c | 28 ++++++++++++++++++----------
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 4 files changed, 37 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 38f423b..ac4d44b 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2262,6 +2262,8 @@ struct drm_i915_gem_request {
 	/** process identifier submitting this request */
 	struct pid *pid;
 
+	struct i915_scheduler_queue_entry	*scheduler_qe;
+
 	/**
 	 * The ELSP only accepts two elements at a time, so we queue
 	 * context/tail pairs on a given queue (ring->execlist_queue) until the
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index d4f1d63..3d109b4 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2817,6 +2817,7 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 {
 	struct drm_i915_gem_request *req, *req_next;
 	unsigned long flags;
+	bool wake_sched = false;
 	u32 seqno;
 
 	if (list_empty(&ring->fence_signal_list)) {
@@ -2852,6 +2853,14 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 		 */
 		list_del_init(&req->signal_link);
 
+		/*
+		 * NB: Must notify the scheduler before signalling
+		 * the node. Otherwise the node can get retired first
+		 * and call scheduler_clean() while the scheduler
+		 * thinks it is still active.
+		 */
+		wake_sched |= i915_scheduler_notify_request(req);
+
 		if (!req->cancelled) {
 			fence_signal_locked(&req->fence);
 			trace_i915_gem_request_complete(req);
@@ -2868,6 +2877,13 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 
 	if (!fence_locked)
 		spin_unlock_irqrestore(&ring->fence_lock, flags);
+
+	/* Necessary? Or does the fence_signal() call do an implicit wakeup? */
+	wake_up_all(&ring->irq_queue);
+
+	/* Final scheduler processing after all individual updates are done. */
+	if (wake_sched)
+		i915_scheduler_wakeup(ring->dev);
 }
 
 static const char *i915_gem_request_get_driver_name(struct fence *req_fence)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 26cd088..ac0a6172 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -174,6 +174,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	node->stamp  = jiffies;
 	i915_gem_request_reference(node->params.request);
 
+	WARN_ON(node->params.request->scheduler_qe);
+	node->params.request->scheduler_qe = node;
+
 	/* Need to determine the number of incomplete entries in the list as
 	 * that will be the maximum size of the dependency list.
 	 *
@@ -350,14 +353,16 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
  * code has mapped it back to a request and will mark that request complete.
  * It also calls this function to notify the scheduler about the completion
  * so the scheduler's node can be updated appropriately.
- * Returns true if the request is scheduler managed, false if not.
+ * Returns true if the request is scheduler managed, false if not. The return
+ * value is combined for all freshly completed requests and if any were true
+ * then i915_scheduler_wakeup() is called so the scheduler can do further
+ * processing (submit more work) at the end.
  */
 bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 {
 	struct drm_i915_private *dev_priv  = to_i915(req->ring->dev);
 	struct i915_scheduler   *scheduler = dev_priv->scheduler;
-	/* XXX: Need to map back from request to node */
-	struct i915_scheduler_queue_entry *node = NULL;
+	struct i915_scheduler_queue_entry *node = req->scheduler_qe;
 	unsigned long       flags;
 
 	if (!node)
@@ -375,16 +380,18 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
-	/*
-	 * XXX: If the in-flight list is now empty then new work should be
-	 * submitted. However, this function is called from interrupt context
-	 * and thus cannot acquire mutex locks and other such things that are
-	 * necessary for fresh submission.
-	 */
-
 	return true;
 }
 
+/*
+ * Called at the end of seqno interrupt processing if any request has
+ * completed that corresponds to a scheduler node.
+ */
+void i915_scheduler_wakeup(struct drm_device *dev)
+{
+	/* XXX: Need to call i915_scheduler_remove() via work handler. */
+}
+
 int i915_scheduler_remove(struct intel_engine_cs *ring)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
@@ -500,6 +507,7 @@ int i915_scheduler_remove(struct intel_engine_cs *ring)
 			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
 
 		/* Free everything that is owned by the node: */
+		node->params.request->scheduler_qe = NULL;
 		i915_gem_request_unreference(node->params.request);
 		kfree(node->params.cliprects);
 		kfree(node->dep_list);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 02ac6f2..9f54786 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -90,5 +90,6 @@ int         i915_scheduler_closefile(struct drm_device *dev,
 				     struct drm_file *file);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
+void        i915_scheduler_wakeup(struct drm_device *dev);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 13/38] drm/i915: Added deferred work handler for scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler needs to do interrupt triggered work that is too complex
to do in the interrupt handler. Thus it requires a deferred work
handler to process such tasks asynchronously.

v2: Updated to reduce mutex lock usage. The lock is now only held for
the minimum time within the remove function rather than for the whole
of the worker thread's operation.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_dma.c       |  3 +++
 drivers/gpu/drm/i915/i915_drv.h       | 10 ++++++++++
 drivers/gpu/drm/i915/i915_gem.c       |  2 ++
 drivers/gpu/drm/i915/i915_scheduler.c | 21 +++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 5 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index c2f9c03..8c9246f 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1149,6 +1149,9 @@ int i915_driver_unload(struct drm_device *dev)
 	WARN_ON(unregister_oom_notifier(&dev_priv->mm.oom_notifier));
 	unregister_shrinker(&dev_priv->mm.shrinker);
 
+	/* Cancel the scheduler work handler, which should be idle now. */
+	cancel_work_sync(&dev_priv->mm.scheduler_work);
+
 	io_mapping_free(dev_priv->gtt.mappable);
 	arch_phys_wc_del(dev_priv->gtt.mtrr);
 
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index ac4d44b..31bf349 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1285,6 +1285,16 @@ struct i915_gem_mm {
 	struct delayed_work retire_work;
 
 	/**
+	 * New scheme is to get an interrupt after every work packet
+	 * in order to allow the low latency scheduling of pending
+	 * packets. The idea behind adding new packets to a pending
+	 * queue rather than directly into the hardware ring buffer
+	 * is to allow high priority packets to over take low priority
+	 * ones.
+	 */
+	struct work_struct scheduler_work;
+
+	/**
 	 * When we detect an idle GPU, we want to turn on
 	 * powersaving features. So once we see that there
 	 * are no more requests outstanding and no more
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 3d109b4..f396393 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -5354,6 +5354,8 @@ i915_gem_load(struct drm_device *dev)
 			  i915_gem_retire_work_handler);
 	INIT_DELAYED_WORK(&dev_priv->mm.idle_work,
 			  i915_gem_idle_work_handler);
+	INIT_WORK(&dev_priv->mm.scheduler_work,
+				i915_gem_scheduler_work_handler);
 	init_waitqueue_head(&dev_priv->gpu_error.reset_queue);
 
 	dev_priv->relative_constants_mode = I915_EXEC_CONSTANTS_REL_GENERAL;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index ac0a6172..c94c941 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -389,10 +389,12 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
  */
 void i915_scheduler_wakeup(struct drm_device *dev)
 {
-	/* XXX: Need to call i915_scheduler_remove() via work handler. */
+	struct drm_i915_private *dev_priv  = to_i915(dev);
+
+	queue_work(dev_priv->wq, &dev_priv->mm.scheduler_work);
 }
 
-int i915_scheduler_remove(struct intel_engine_cs *ring)
+static int i915_scheduler_remove(struct intel_engine_cs *ring)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
 	struct i915_scheduler   *scheduler = dev_priv->scheduler;
@@ -519,6 +521,21 @@ int i915_scheduler_remove(struct intel_engine_cs *ring)
 	return ret;
 }
 
+void i915_gem_scheduler_work_handler(struct work_struct *work)
+{
+	struct intel_engine_cs  *ring;
+	struct drm_i915_private *dev_priv;
+	struct drm_device       *dev;
+	int                     i;
+
+	dev_priv = container_of(work, struct drm_i915_private, mm.scheduler_work);
+	dev = dev_priv->dev;
+
+	for_each_ring(ring, dev_priv, i) {
+		i915_scheduler_remove(ring);
+	}
+}
+
 static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
 {
 	struct i915_scheduler_queue_entry *node;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 9f54786..56f68e5 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -91,5 +91,6 @@ int         i915_scheduler_closefile(struct drm_device *dev,
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
+void        i915_gem_scheduler_work_handler(struct work_struct *work);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 14/38] drm/i915: Redirect execbuffer_final() via scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Updated the execbuffer() code to pass the packaged up batch buffer
information to the scheduler rather than calling execbuffer_final()
directly. The scheduler queue() code is currently a stub which simply
chains on to _final() immediately.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 18 +++++++-----------
 drivers/gpu/drm/i915/intel_lrc.c           | 12 ++++--------
 2 files changed, 11 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 0ad32f6..0e09680 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -33,6 +33,7 @@
 #include "intel_drv.h"
 #include <linux/dma_remapping.h>
 #include <linux/uaccess.h>
+#include "i915_scheduler.h"
 
 #define  __EXEC_OBJECT_HAS_PIN (1<<31)
 #define  __EXEC_OBJECT_HAS_FENCE (1<<30)
@@ -1178,6 +1179,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas)
 {
+	struct i915_scheduler_queue_entry *qe;
 	struct drm_device *dev = params->dev;
 	struct intel_engine_cs *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -1222,17 +1224,11 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	ret = dev_priv->gt.execbuf_final(params);
+	qe = container_of(params, typeof(*qe), params);
+	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
 		return ret;
 
-	/*
-	 * Free everything that was stored in the QE structure (until the
-	 * scheduler arrives and does it instead):
-	 */
-	if (params->dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_execbuff_release_batch_obj(params->batch_obj);
-
 	return 0;
 }
 
@@ -1377,8 +1373,8 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	struct intel_engine_cs *ring;
 	struct intel_context *ctx;
 	struct i915_address_space *vm;
-	struct i915_execbuffer_params params_master; /* XXX: will be removed later */
-	struct i915_execbuffer_params *params = &params_master;
+	struct i915_scheduler_queue_entry qe;
+	struct i915_execbuffer_params *params = &qe.params;
 	const u32 ctx_id = i915_execbuffer2_get_context_id(*args);
 	u32 dispatch_flags;
 	int ret;
@@ -1484,7 +1480,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	else
 		vm = &dev_priv->gtt.base;
 
-	memset(&params_master, 0x00, sizeof(params_master));
+	memset(&qe, 0x00, sizeof(qe));
 
 	eb = eb_create(args);
 	if (eb == NULL) {
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index a344a3a..1ff552a 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -136,6 +136,7 @@
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
 #include "intel_mocs.h"
+#include "i915_scheduler.h"
 
 #define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE)
 #define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE)
@@ -866,6 +867,7 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas)
 {
+	struct i915_scheduler_queue_entry *qe;
 	struct drm_device       *dev = params->dev;
 	struct intel_engine_cs  *ring = params->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -908,17 +910,11 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	ret = dev_priv->gt.execbuf_final(params);
+	qe = container_of(params, typeof(*qe), params);
+	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
 		return ret;
 
-	/*
-	 * Free everything that was stored in the QE structure (until the
-	 * scheduler arrives and does it instead):
-	 */
-	if (params->dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_execbuff_release_batch_obj(params->batch_obj);
-
 	return 0;
 }
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 15/38] drm/i915: Keep the reserved space mechanism happy

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Ring space is reserved when constructing a request to ensure that the
subsequent 'add_request()' call cannot fail due to waiting for space
on a busy or broken GPU. However, the scheduler jumps in to the middle
of the execbuffer process between request creation and request
submission. Thus it needs to cancel the reserved space when the
request is simply added to the scheduler's queue and not yet
submitted. Similarly, it needs to re-reserve the space when it finally
does want to send the batch buffer to the hardware.

v3: Updated to use locally cached request pointer.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  7 +++++++
 drivers/gpu/drm/i915/i915_scheduler.c      |  4 ++++
 drivers/gpu/drm/i915/intel_lrc.c           | 13 +++++++++++--
 3 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 0e09680..13c6217 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1247,6 +1247,10 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	ret = intel_ring_reserve_space(req);
+	if (ret)
+		return ret;
+
 	intel_runtime_pm_get(dev_priv);
 
 	/*
@@ -1307,6 +1311,9 @@ error:
 	 */
 	intel_runtime_pm_put(dev_priv);
 
+	if (ret)
+		intel_ring_reserved_space_cancel(req->ringbuf);
+
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index c94c941..3c282c6 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -145,6 +145,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
 		int ret;
 
+		intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
+
 		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
 		ret = dev_priv->gt.execbuf_final(&qe->params);
 		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
@@ -174,6 +176,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	node->stamp  = jiffies;
 	i915_gem_request_reference(node->params.request);
 
+	intel_ring_reserved_space_cancel(node->params.request->ringbuf);
+
 	WARN_ON(node->params.request->scheduler_qe);
 	node->params.request->scheduler_qe = node;
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 1ff552a..7ea4d96 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -934,13 +934,17 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	ret = intel_logical_ring_reserve_space(req);
+	if (ret)
+		return ret;
+
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
 	ret = logical_ring_invalidate_all_caches(req);
 	if (ret)
-		return ret;
+		goto err;
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
@@ -962,13 +966,18 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 
 	ret = ring->emit_bb_start(req, exec_start, params->dispatch_flags);
 	if (ret)
-		return ret;
+		goto err;
 
 	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
 	return 0;
+
+err:
+	intel_ring_reserved_space_cancel(params->request->ringbuf);
+
+	return ret;
 }
 
 void intel_execlists_retire_requests(struct intel_engine_cs *ring)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 16/38] drm/i915: Added tracking/locking of batch buffer objects

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler needs to track interdependencies between batch buffers.
These are calculated by analysing the object lists of the buffers and
looking for commonality. The scheduler also needs to keep those
buffers locked long after the initial IOCTL call has returned to user
land.

v3: Updated to support read-read optimisation.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 52 ++++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.c      | 33 +++++++++++++++++--
 2 files changed, 80 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 13c6217..3c84b97 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1384,7 +1384,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	struct i915_execbuffer_params *params = &qe.params;
 	const u32 ctx_id = i915_execbuffer2_get_context_id(*args);
 	u32 dispatch_flags;
-	int ret;
+	int ret, i;
 	bool need_relocs;
 
 	if (!i915_gem_check_execbuffer(args))
@@ -1497,6 +1497,14 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		goto pre_mutex_err;
 	}
 
+	qe.saved_objects = kzalloc(
+			sizeof(*qe.saved_objects) * args->buffer_count,
+			GFP_KERNEL);
+	if (!qe.saved_objects) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
 	/* Look up object handles */
 	ret = eb_lookup_vmas(eb, exec, args, vm, file);
 	if (ret)
@@ -1617,7 +1625,30 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->args_DR1                = args->DR1;
 	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
-	params->ctx                     = ctx;
+
+	/*
+	 * Save away the list of objects used by this batch buffer for the
+	 * purpose of tracking inter-buffer dependencies.
+	 */
+	for (i = 0; i < args->buffer_count; i++) {
+		struct drm_i915_gem_object *obj;
+
+		/*
+		 * NB: 'drm_gem_object_lookup()' increments the object's
+		 * reference count and so must be matched by a
+		 * 'drm_gem_object_unreference' call.
+		 */
+		obj = to_intel_bo(drm_gem_object_lookup(dev, file,
+							  exec[i].handle));
+		qe.saved_objects[i].obj       = obj;
+		qe.saved_objects[i].read_only = obj->base.pending_write_domain == 0;
+
+	}
+	qe.num_objs = i;
+
+	/* Lock and save the context object as well. */
+	i915_gem_context_reference(ctx);
+	params->ctx = ctx;
 
 	ret = dev_priv->gt.execbuf_submit(params, args, &eb->vmas);
 	if (ret)
@@ -1648,6 +1679,23 @@ err:
 	i915_gem_context_unreference(ctx);
 	eb_destroy(eb);
 
+	if (qe.saved_objects) {
+		/* Need to release the objects: */
+		for (i = 0; i < qe.num_objs; i++) {
+			if (!qe.saved_objects[i].obj)
+				continue;
+
+			drm_gem_object_unreference(
+					&qe.saved_objects[i].obj->base);
+		}
+
+		kfree(qe.saved_objects);
+
+		/* Context too */
+		if (params->ctx)
+			i915_gem_context_unreference(params->ctx);
+	}
+
 	/*
 	 * If the request was created but not successfully submitted then it
 	 * must be freed again. If it was submitted then it is being tracked
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 3c282c6..067ef33 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -158,7 +158,23 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		if (ret)
 			return ret;
 
-		/* Free everything that is owned by the QE structure: */
+		/* Need to release the objects: */
+		for (i = 0; i < qe->num_objs; i++) {
+			if (!qe->saved_objects[i].obj)
+				continue;
+
+			drm_gem_object_unreference(&qe->saved_objects[i].obj->base);
+		}
+
+		kfree(qe->saved_objects);
+		qe->saved_objects = NULL;
+		qe->num_objs = 0;
+
+		/* Free the context object too: */
+		if (qe->params.ctx)
+			i915_gem_context_unreference(qe->params.ctx);
+
+		/* And anything else owned by the QE structure: */
 		kfree(qe->params.cliprects);
 		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
 			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
@@ -407,7 +423,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	int                 flying = 0, queued = 0;
 	int                 ret = 0;
 	bool                do_submit;
-	uint32_t            min_seqno;
+	uint32_t            i, min_seqno;
 	struct list_head    remove;
 
 	if (list_empty(&scheduler->node_queue[ring->id]))
@@ -512,7 +528,18 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
 			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
 
-		/* Free everything that is owned by the node: */
+		/* Release the locked buffers: */
+		for (i = 0; i < node->num_objs; i++) {
+			drm_gem_object_unreference(
+					    &node->saved_objects[i].obj->base);
+		}
+		kfree(node->saved_objects);
+
+		/* Context too: */
+		if (node->params.ctx)
+			i915_gem_context_unreference(node->params.ctx);
+
+		/* And anything else owned by the node: */
 		node->params.request->scheduler_qe = NULL;
 		i915_gem_request_unreference(node->params.request);
 		kfree(node->params.cliprects);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 17/38] drm/i915: Hook scheduler node clean up into retire requests

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler keeps its own lock on various DRM objects in order to
guarantee safe access long after the original execbuff IOCTL has
completed. This is especially important when pre-emption is enabled as
the batch buffer might need to be submitted to the hardware multiple
times. This patch hooks the clean up of these locks into the request
retire function. The request can only be retired after it has
completed on the hardware and thus is no longer eligible for
re-submission. Thus there is no point holding on to the locks beyond
that time.

v3: Updated to not WARN when cleaning a node that is being cancelled.
The clean will happen later so skipping it at the point of
cancellation is fine.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c       |  3 ++
 drivers/gpu/drm/i915/i915_scheduler.c | 58 ++++++++++++++++++++++++-----------
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 3 files changed, 44 insertions(+), 18 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index f396393..ce0551a 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1435,6 +1435,9 @@ static void i915_gem_request_retire(struct drm_i915_gem_request *request)
 		fence_signal_locked(&request->fence);
 	}
 
+	if (request->scheduler_qe)
+		i915_gem_scheduler_clean_node(request->scheduler_qe);
+
 	i915_gem_request_unreference(request);
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 067ef33..0b7f1df 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -414,6 +414,43 @@ void i915_scheduler_wakeup(struct drm_device *dev)
 	queue_work(dev_priv->wq, &dev_priv->mm.scheduler_work);
 }
 
+void i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *node)
+{
+	uint32_t i;
+
+	if (!I915_SQS_IS_COMPLETE(node)) {
+		WARN(!node->params.request->cancelled,
+		     "Cleaning active node: %d!\n", node->status);
+		return;
+	}
+
+	if (node->params.batch_obj) {
+		/*
+		 * The batch buffer must be unpinned before it is unreferenced
+		 * otherwise the unpin fails with a missing vma!?
+		 */
+		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
+
+		node->params.batch_obj = NULL;
+	}
+
+	/* Release the locked buffers: */
+	for (i = 0; i < node->num_objs; i++) {
+		drm_gem_object_unreference(
+				    &node->saved_objects[i].obj->base);
+	}
+	kfree(node->saved_objects);
+	node->saved_objects = NULL;
+	node->num_objs = 0;
+
+	/* Context too: */
+	if (node->params.ctx) {
+		i915_gem_context_unreference(node->params.ctx);
+		node->params.ctx = NULL;
+	}
+}
+
 static int i915_scheduler_remove(struct intel_engine_cs *ring)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
@@ -423,7 +460,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	int                 flying = 0, queued = 0;
 	int                 ret = 0;
 	bool                do_submit;
-	uint32_t            i, min_seqno;
+	uint32_t            min_seqno;
 	struct list_head    remove;
 
 	if (list_empty(&scheduler->node_queue[ring->id]))
@@ -521,23 +558,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		node = list_first_entry(&remove, typeof(*node), link);
 		list_del(&node->link);
 
-		/*
-		 * The batch buffer must be unpinned before it is unreferenced
-		 * otherwise the unpin fails with a missing vma!?
-		 */
-		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
-			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
-
-		/* Release the locked buffers: */
-		for (i = 0; i < node->num_objs; i++) {
-			drm_gem_object_unreference(
-					    &node->saved_objects[i].obj->base);
-		}
-		kfree(node->saved_objects);
-
-		/* Context too: */
-		if (node->params.ctx)
-			i915_gem_context_unreference(node->params.ctx);
+		/* Free up all the DRM object references */
+		i915_gem_scheduler_clean_node(node);
 
 		/* And anything else owned by the node: */
 		node->params.request->scheduler_qe = NULL;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 56f68e5..54d87fb 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -88,6 +88,7 @@ bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
 int         i915_scheduler_closefile(struct drm_device *dev,
 				     struct drm_file *file);
+void        i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 18/38] drm/i915: Added scheduler support to __wait_request() calls

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler can cause batch buffers, and hence requests, to be
submitted to the ring out of order and asynchronously to their
submission to the driver. Thus at the point of waiting for the
completion of a given request, it is not even guaranteed that the
request has actually been sent to the hardware yet. Even it is has
been sent, it is possible that it could be pre-empted and thus
'unsent'.

This means that it is necessary to be able to submit requests to the
hardware during the wait call itself. Unfortunately, while some
callers of __wait_request() release the mutex lock first, others do
not (and apparently can not). Hence there is the ability to deadlock
as the wait stalls for submission but the asynchronous submission is
stalled for the mutex lock.

This change hooks the scheduler in to the __wait_request() code to
ensure correct behaviour. That is, flush the target batch buffer
through to the hardware and do not deadlock waiting for something that
cannot currently be submitted. Instead, the wait call must return
EAGAIN at least as far back as necessary to release the mutex lock and
allow the scheduler's asynchronous processing to get in and handle the
pre-emption operation and eventually (re-)submit the work.

v3: Removed the explicit scheduler flush from i915_wait_request().
This is no longer necessary and was causing unintended changes to the
scheduler priority level which broke a validation team test.

v4: Corrected the format of some comments to keep the style checker
happy.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h         |  3 ++-
 drivers/gpu/drm/i915/i915_gem.c         | 37 ++++++++++++++++++++++++++-------
 drivers/gpu/drm/i915/i915_scheduler.c   | 20 ++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h   |  2 ++
 drivers/gpu/drm/i915/intel_display.c    |  5 +++--
 drivers/gpu/drm/i915/intel_ringbuffer.c |  2 +-
 6 files changed, 58 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 31bf349..2906bb0 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -3021,7 +3021,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			unsigned reset_counter,
 			bool interruptible,
 			s64 *timeout,
-			struct intel_rps_client *rps);
+			struct intel_rps_client *rps,
+			bool is_locked);
 int __must_check i915_wait_request(struct drm_i915_gem_request *req);
 int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf);
 int __must_check
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index ce0551a..59024a3 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1221,7 +1221,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			unsigned reset_counter,
 			bool interruptible,
 			s64 *timeout,
-			struct intel_rps_client *rps)
+			struct intel_rps_client *rps,
+			bool is_locked)
 {
 	struct intel_engine_cs *ring = i915_gem_request_get_ring(req);
 	struct drm_device *dev = ring->dev;
@@ -1230,8 +1231,10 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 	DEFINE_WAIT(wait);
 	unsigned long timeout_expire;
 	s64 before, now;
-	int ret;
+	int ret = 0;
+	bool    busy;
 
+	might_sleep();
 	WARN(!intel_irqs_enabled(dev_priv), "IRQs disabled");
 
 	if (i915_gem_request_completed(req))
@@ -1276,6 +1279,26 @@ int __i915_wait_request(struct drm_i915_gem_request *req,
 			break;
 		}
 
+		if (is_locked) {
+			/*
+			 * If this request is being processed by the scheduler
+			 * then it is unsafe to sleep with the mutex lock held
+			 * as the scheduler may require the lock in order to
+			 * progress the request.
+			 */
+			if (i915_scheduler_is_request_tracked(req, NULL, &busy)) {
+				if (busy) {
+					ret = -EAGAIN;
+					break;
+				}
+			}
+
+			/*
+			 * If the request is not tracked by the scheduler
+			 * then the regular test can be done.
+			 */
+		}
+
 		if (i915_gem_request_completed(req)) {
 			ret = 0;
 			break;
@@ -1488,7 +1511,7 @@ i915_wait_request(struct drm_i915_gem_request *req)
 
 	ret = __i915_wait_request(req,
 				  atomic_read(&dev_priv->gpu_error.reset_counter),
-				  interruptible, NULL, NULL);
+				  interruptible, NULL, NULL, true);
 	if (ret)
 		return ret;
 
@@ -1601,7 +1624,7 @@ i915_gem_object_wait_rendering__nonblocking(struct drm_i915_gem_object *obj,
 	mutex_unlock(&dev->struct_mutex);
 	for (i = 0; ret == 0 && i < n; i++)
 		ret = __i915_wait_request(requests[i], reset_counter, true,
-					  NULL, rps);
+					  NULL, rps, false);
 	mutex_lock(&dev->struct_mutex);
 
 	for (i = 0; i < n; i++) {
@@ -3458,7 +3481,7 @@ i915_gem_wait_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 		if (ret == 0)
 			ret = __i915_wait_request(req[i], reset_counter, true,
 						  args->timeout_ns > 0 ? &args->timeout_ns : NULL,
-						  file->driver_priv);
+						  file->driver_priv, false);
 		i915_gem_request_unreference(req[i]);
 	}
 	return ret;
@@ -3491,7 +3514,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 					  atomic_read(&i915->gpu_error.reset_counter),
 					  i915->mm.interruptible,
 					  NULL,
-					  &i915->rps.semaphores);
+					  &i915->rps.semaphores, true);
 		if (ret)
 			return ret;
 
@@ -4450,7 +4473,7 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	if (target == NULL)
 		return 0;
 
-	ret = __i915_wait_request(target, reset_counter, true, NULL, NULL);
+	ret = __i915_wait_request(target, reset_counter, true, NULL, NULL, false);
 	if (ret == 0)
 		queue_delayed_work(dev_priv->wq, &dev_priv->mm.retire_work, 0);
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 0b7f1df..8c05dd0 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -881,6 +881,26 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
 	return 0;
 }
 
+bool i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
+				       bool *completed, bool *busy)
+{
+	struct drm_i915_private *dev_priv = req->ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return false;
+
+	if (req->scheduler_qe == NULL)
+		return false;
+
+	if (completed)
+		*completed = I915_SQS_IS_COMPLETE(req->scheduler_qe);
+	if (busy)
+		*busy      = I915_SQS_IS_QUEUED(req->scheduler_qe);
+
+	return true;
+}
+
 int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 {
 	struct i915_scheduler_queue_entry  *node;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 54d87fb..c3e7ac6 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -93,5 +93,7 @@ int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *q
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
 void        i915_gem_scheduler_work_handler(struct work_struct *work);
+bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
+					      bool *completed, bool *busy);
 
 #endif  /* _I915_SCHEDULER_H_ */
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index c00bc50..58c464f 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -11258,7 +11258,8 @@ static void intel_mmio_flip_work_func(struct work_struct *work)
 		WARN_ON(__i915_wait_request(mmio_flip->req,
 					    mmio_flip->crtc->reset_counter,
 					    false, NULL,
-					    &mmio_flip->i915->rps.mmioflips));
+					    &mmio_flip->i915->rps.mmioflips,
+					    false));
 		i915_gem_request_unreference(mmio_flip->req);
 	}
 
@@ -13258,7 +13259,7 @@ static int intel_atomic_prepare_commit(struct drm_device *dev,
 
 			ret = __i915_wait_request(intel_plane_state->wait_req,
 						  reset_counter, true,
-						  NULL, NULL);
+						  NULL, NULL, false);
 
 			/* Swallow -EIO errors to allow updates during hw lockup. */
 			if (ret == -EIO)
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index a93bbce..0ae70a6 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -2294,7 +2294,7 @@ int intel_ring_idle(struct intel_engine_cs *ring)
 	return __i915_wait_request(req,
 				   atomic_read(&to_i915(ring->dev)->gpu_error.reset_counter),
 				   to_i915(ring->dev)->mm.interruptible,
-				   NULL, NULL);
+				   NULL, NULL, true);
 }
 
 int intel_ring_alloc_request_extras(struct drm_i915_gem_request *request)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 19/38] drm/i915: Added scheduler support to page fault handler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

GPU page faults can now require scheduler operation in order to
complete. For example, in order to free up sufficient memory to handle
the fault the handler must wait for a batch buffer to complete that
has not even been sent to the hardware yet. Thus EAGAIN no longer
means a GPU hang, it can occur under normal operation.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 59024a3..7716462 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1949,10 +1949,15 @@ out:
 		}
 	case -EAGAIN:
 		/*
-		 * EAGAIN means the gpu is hung and we'll wait for the error
-		 * handler to reset everything when re-faulting in
+		 * EAGAIN can mean the gpu is hung and we'll have to wait for
+		 * the error handler to reset everything when re-faulting in
 		 * i915_mutex_lock_interruptible.
+		 *
+		 * It can also indicate various other nonfatal errors for which
+		 * the best response is to give other threads a chance to run,
+		 * and then retry the failing operation in its entirety.
 		 */
+		/*FALLTHRU*/
 	case 0:
 	case -ERESTARTSYS:
 	case -EINTR:
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 20/38] drm/i915: Added scheduler flush calls to ring throttle and idle functions

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When requesting that all GPU work is completed, it is now necessary to
get the scheduler involved in order to flush out work that queued and
not yet submitted.

v2: Updated to add support for flushing the scheduler queue by time
stamp rather than just doing a blanket flush.

v3: Moved submit_max_priority() to this patch from an earlier patch
is it is no longer required in the other.

v4: Corrected the format of a comment to keep the style checker happy.
Downgraded a BUG_ON to a WARN_ON as the latter is preferred.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c       |  24 +++++-
 drivers/gpu/drm/i915/i915_scheduler.c | 134 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |   3 +
 3 files changed, 160 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 7716462..6ea0896 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3733,6 +3733,10 @@ int i915_gpu_idle(struct drm_device *dev)
 
 	/* Flush everything onto the inactive list. */
 	for_each_ring(ring, dev_priv, i) {
+		ret = i915_scheduler_flush(ring, true);
+		if (ret < 0)
+			return ret;
+
 		if (!i915.enable_execlists) {
 			struct drm_i915_gem_request *req;
 
@@ -4446,7 +4450,8 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	unsigned long recent_enough = jiffies - DRM_I915_THROTTLE_JIFFIES;
 	struct drm_i915_gem_request *request, *target = NULL;
 	unsigned reset_counter;
-	int ret;
+	int i, ret;
+	struct intel_engine_cs *ring;
 
 	ret = i915_gem_wait_for_error(&dev_priv->gpu_error);
 	if (ret)
@@ -4456,6 +4461,23 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	if (ret)
 		return ret;
 
+	for_each_ring(ring, dev_priv, i) {
+		/*
+		 * Flush out scheduler entries that are getting 'stale'. Note
+		 * that the following recent_enough test will only check
+		 * against the time at which the request was submitted to the
+		 * hardware (i.e. when it left the scheduler) not the time it
+		 * was submitted to the driver.
+		 *
+		 * Also, there is not much point worring about busy return
+		 * codes from the scheduler flush call. Even if more work
+		 * cannot be submitted right now for whatever reason, we
+		 * still want to throttle against stale work that has already
+		 * been submitted.
+		 */
+		i915_scheduler_flush_stamp(ring, recent_enough, false);
+	}
+
 	spin_lock(&file_priv->mm.lock);
 	list_for_each_entry(request, &file_priv->mm.request_list, client_list) {
 		if (time_after_eq(request->emitted_jiffies, recent_enough))
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 8c05dd0..bd52752 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -31,6 +31,8 @@ static int         i915_scheduler_remove_dependent(struct i915_scheduler *schedu
 						   struct i915_scheduler_queue_entry *remove);
 static int         i915_scheduler_submit(struct intel_engine_cs *ring,
 					 bool is_locked);
+static int         i915_scheduler_submit_max_priority(struct intel_engine_cs *ring,
+						      bool is_locked);
 static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
 					       struct intel_engine_cs *ring);
 static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
@@ -589,6 +591,100 @@ void i915_gem_scheduler_work_handler(struct work_struct *work)
 	}
 }
 
+int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
+			       unsigned long target,
+			       bool is_locked)
+{
+	struct i915_scheduler_queue_entry *node;
+	struct drm_i915_private           *dev_priv;
+	struct i915_scheduler             *scheduler;
+	unsigned long       flags;
+	int                 flush_count = 0;
+
+	if (!ring)
+		return -EINVAL;
+
+	dev_priv  = ring->dev->dev_private;
+	scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return 0;
+
+	if (is_locked && (scheduler->flags[ring->id] & i915_sf_submitting)) {
+		/*
+		 * Scheduler is busy already submitting another batch,
+		 * come back later rather than going recursive...
+		 */
+		return -EAGAIN;
+	}
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	i915_scheduler_priority_bump_clear(scheduler);
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+
+		if (node->stamp > target)
+			continue;
+
+		flush_count = i915_scheduler_priority_bump(scheduler,
+					node, scheduler->priority_level_max);
+	}
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	if (flush_count) {
+		DRM_DEBUG_DRIVER("<%s> Bumped %d entries\n", ring->name, flush_count);
+		flush_count = i915_scheduler_submit_max_priority(ring, is_locked);
+	}
+
+	return flush_count;
+}
+
+int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
+{
+	struct i915_scheduler_queue_entry *node;
+	struct drm_i915_private           *dev_priv;
+	struct i915_scheduler             *scheduler;
+	unsigned long       flags;
+	bool        found;
+	int         ret;
+	uint32_t    count = 0;
+
+	if (!ring)
+		return -EINVAL;
+
+	dev_priv  = ring->dev->dev_private;
+	scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return 0;
+
+	WARN_ON(is_locked && (scheduler->flags[ring->id] & i915_sf_submitting));
+
+	do {
+		found = false;
+		spin_lock_irqsave(&scheduler->lock, flags);
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (!I915_SQS_IS_QUEUED(node))
+				continue;
+
+			found = true;
+			break;
+		}
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (found) {
+			ret = i915_scheduler_submit(ring, is_locked);
+			if (ret < 0)
+				return ret;
+
+			count += ret;
+		}
+	} while (found);
+
+	return count;
+}
+
 static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
 {
 	struct i915_scheduler_queue_entry *node;
@@ -642,6 +738,44 @@ static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 	return count;
 }
 
+static int i915_scheduler_submit_max_priority(struct intel_engine_cs *ring,
+					      bool is_locked)
+{
+	struct i915_scheduler_queue_entry  *node;
+	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler              *scheduler = dev_priv->scheduler;
+	unsigned long	flags;
+	int             ret, count = 0;
+	bool            found;
+
+	do {
+		found = false;
+		spin_lock_irqsave(&scheduler->lock, flags);
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (!I915_SQS_IS_QUEUED(node))
+				continue;
+
+			if (node->priority < scheduler->priority_level_max)
+				continue;
+
+			found = true;
+			break;
+		}
+		spin_unlock_irqrestore(&scheduler->lock, flags);
+
+		if (!found)
+			break;
+
+		ret = i915_scheduler_submit(ring, is_locked);
+		if (ret < 0)
+			return ret;
+
+		count += ret;
+	} while (found);
+
+	return count;
+}
+
 static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 				struct i915_scheduler_queue_entry **pop_node,
 				unsigned long *flags)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index c3e7ac6..7654013 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -93,6 +93,9 @@ int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *q
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
 void        i915_gem_scheduler_work_handler(struct work_struct *work);
+int         i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked);
+int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
+				       unsigned long stamp, bool is_locked);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 21/38] drm/i915: Added a module parameter for allowing scheduler overrides

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

It can be useful to be able to disable certain features (e.g. the
entire scheduler) via a module parameter for debugging purposes. A
parameter has the advantage of not being a compile time switch but
without implying that it can be changed dynamically at runtime.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h       | 1 +
 drivers/gpu/drm/i915/i915_params.c    | 4 ++++
 drivers/gpu/drm/i915/i915_scheduler.c | 5 ++++-
 drivers/gpu/drm/i915/i915_scheduler.h | 5 +++++
 4 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 2906bb0..8d6d07a 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2709,6 +2709,7 @@ struct i915_params {
 	bool verbose_state_checks;
 	bool nuclear_pageflip;
 	int edp_vswing;
+	int scheduler_override;
 };
 extern struct i915_params i915 __read_mostly;
 
diff --git a/drivers/gpu/drm/i915/i915_params.c b/drivers/gpu/drm/i915/i915_params.c
index ba7274a..be2e16c 100644
--- a/drivers/gpu/drm/i915/i915_params.c
+++ b/drivers/gpu/drm/i915/i915_params.c
@@ -54,6 +54,7 @@ struct i915_params i915 __read_mostly = {
 	.edp_vswing = 0,
 	.enable_guc_submission = true,
 	.guc_log_level = -1,
+	.scheduler_override = 1,
 };
 
 module_param_named(modeset, i915.modeset, int, 0400);
@@ -191,3 +192,6 @@ MODULE_PARM_DESC(enable_guc_submission, "Enable GuC submission (default:false)")
 module_param_named(guc_log_level, i915.guc_log_level, int, 0400);
 MODULE_PARM_DESC(guc_log_level,
 	"GuC firmware logging level (-1:disabled (default), 0-3:enabled)");
+
+module_param_named(scheduler_override, i915.scheduler_override, int, 0600);
+MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (0 = none, 1 = direct submission [default])");
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index bd52752..bfde3a2 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -44,6 +44,9 @@ bool i915_scheduler_is_enabled(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	if (i915.scheduler_override & i915_so_direct_submit)
+		return false;
+
 	return dev_priv->scheduler != NULL;
 }
 
@@ -144,7 +147,7 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	WARN_ON(!scheduler);
 
-	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
+	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
 		intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 7654013..cc9205b 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -84,6 +84,11 @@ enum {
 	i915_sf_submitting          = (1 << 1),
 };
 
+/* Options for 'scheduler_override' module parameter: */
+enum {
+	i915_so_direct_submit       = (1 << 0),
+};
+
 bool        i915_scheduler_is_enabled(struct drm_device *dev);
 int         i915_scheduler_init(struct drm_device *dev);
 int         i915_scheduler_closefile(struct drm_device *dev,
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 22/38] drm/i915: Support for 'unflushed' ring idle

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When the seqno wraps around zero, the entire GPU is forced to be idle
for some reason (possibly only to work around issues with hardware
semaphores but no-one seems too sure!). This causes a problem if the
force idle occurs at an inopportune moment such as in the middle of
submitting a batch buffer. Specifically, it would lead to recursive
submits - submitting work requires a new seqno, the new seqno requires
idling the ring, idling the ring requires submitting work, submitting
work requires a new seqno...

This change adds a 'flush' parameter to the idle function call which
specifies whether the scheduler queues should be flushed out. I.e. is
the call intended to just idle the ring as it is right now (no flush)
or is it intended to force all outstanding work out of the system
(with flush).

In the seqno wrap case, pending work is not an issue because the next
operation will be to submit it. However, in other cases, the intention
is to make sure everything that could be done has been done.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c         |  4 ++--
 drivers/gpu/drm/i915/intel_lrc.c        |  2 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c | 17 +++++++++++++++--
 drivers/gpu/drm/i915/intel_ringbuffer.h |  2 +-
 4 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 6ea0896..073b334 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2510,7 +2510,7 @@ i915_gem_init_seqno(struct drm_device *dev, u32 seqno)
 
 	/* Carefully retire all requests without writing to the rings */
 	for_each_ring(ring, dev_priv, i) {
-		ret = intel_ring_idle(ring);
+		ret = intel_ring_idle(ring, false);
 		if (ret)
 			return ret;
 	}
@@ -3753,7 +3753,7 @@ int i915_gpu_idle(struct drm_device *dev)
 			i915_add_request_no_flush(req);
 		}
 
-		ret = intel_ring_idle(ring);
+		ret = intel_ring_idle(ring, true);
 		if (ret)
 			return ret;
 	}
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 7ea4d96..d8ddf18 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -1014,7 +1014,7 @@ void intel_logical_ring_stop(struct intel_engine_cs *ring)
 	if (!intel_ring_initialized(ring))
 		return;
 
-	ret = intel_ring_idle(ring);
+	ret = intel_ring_idle(ring, true);
 	if (ret && !i915_reset_in_progress(&to_i915(ring->dev)->gpu_error))
 		DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n",
 			  ring->name, ret);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 0ae70a6..3f3d928 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -2278,9 +2278,22 @@ static void __wrap_ring_buffer(struct intel_ringbuffer *ringbuf)
 	intel_ring_update_space(ringbuf);
 }
 
-int intel_ring_idle(struct intel_engine_cs *ring)
+int intel_ring_idle(struct intel_engine_cs *ring, bool flush)
 {
 	struct drm_i915_gem_request *req;
+	int ret;
+
+	/*
+	 * NB: Must not flush the scheduler if this idle request is from
+	 * within an execbuff submission (i.e. due to 'get_seqno' calling
+	 * 'wrap_seqno' calling 'idle'). As that would lead to recursive
+	 * flushes!
+	 */
+	if (flush) {
+		ret = i915_scheduler_flush(ring, true);
+		if (ret)
+			return ret;
+	}
 
 	/* Wait upon the last request to be completed */
 	if (list_empty(&ring->request_list))
@@ -3085,7 +3098,7 @@ intel_stop_ring_buffer(struct intel_engine_cs *ring)
 	if (!intel_ring_initialized(ring))
 		return;
 
-	ret = intel_ring_idle(ring);
+	ret = intel_ring_idle(ring, true);
 	if (ret && !i915_reset_in_progress(&to_i915(ring->dev)->gpu_error))
 		DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n",
 			  ring->name, ret);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h b/drivers/gpu/drm/i915/intel_ringbuffer.h
index a103019..136a039 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.h
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.h
@@ -465,7 +465,7 @@ void intel_ring_update_space(struct intel_ringbuffer *ringbuf);
 int intel_ring_space(struct intel_ringbuffer *ringbuf);
 bool intel_ring_stopped(struct intel_engine_cs *ring);
 
-int __must_check intel_ring_idle(struct intel_engine_cs *ring);
+int __must_check intel_ring_idle(struct intel_engine_cs *ring, bool flush);
 void intel_ring_init_seqno(struct intel_engine_cs *ring, u32 seqno);
 int intel_ring_flush_all_caches(struct drm_i915_gem_request *req);
 int intel_ring_invalidate_all_caches(struct drm_i915_gem_request *req);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 23/38] drm/i915: Defer seqno allocation until actual hardware submission time

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The seqno value is now only used for the final test for completion of
a request. It is no longer used to track the request through the
software stack. Thus it is no longer necessary to allocate the seqno
immediately with the request. Instead, it can be done lazily and left
until the request is actually sent to the hardware. This is particular
advantageous with a GPU scheduler as the requests can then be
re-ordered between their creation and their hardware submission
without having out of order seqnos.

v2: i915_add_request() can't fail!

Combine with 'drm/i915: Assign seqno at start of exec_final()'
Various bits of code during the execbuf code path need a seqno value
to be assigned to the request. This change makes this assignment
explicit at the start of submission_final() rather than relying on an
auto-generated seqno to have happened already. This is in preparation
for a future patch which changes seqno values to be assigned lazily
(during add_request).

v3: Updated to use locally cached request pointer.

v4: Changed some white space and comment formatting to keep the style
checker happy.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  1 +
 drivers/gpu/drm/i915/i915_gem.c            | 23 ++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 14 ++++++++++++++
 drivers/gpu/drm/i915/intel_lrc.c           | 14 ++++++++++++++
 4 files changed, 51 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 8d6d07a..a304918 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2225,6 +2225,7 @@ struct drm_i915_gem_request {
 
 	/** GEM sequence number associated with this request. */
 	uint32_t seqno;
+	uint32_t reserved_seqno;
 
 	/* Unique identifier which can be used for trace points & debug */
 	uint32_t uniq;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 073b334..b37dbd7 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2562,6 +2562,11 @@ i915_gem_get_seqno(struct drm_device *dev, u32 *seqno)
 
 	/* reserve 0 for non-seqno */
 	if (dev_priv->next_seqno == 0) {
+		/*
+		 * Why is the full re-initialisation required? Is it only for
+		 * hardware semaphores? If so, could skip it in the case where
+		 * semaphores are disabled?
+		 */
 		int ret = i915_gem_init_seqno(dev, 0);
 		if (ret)
 			return ret;
@@ -2619,6 +2624,12 @@ void __i915_add_request(struct drm_i915_gem_request *request,
 		WARN(ret, "*_ring_flush_all_caches failed: %d!\n", ret);
 	}
 
+	/* Make the request's seqno 'live': */
+	if (!request->seqno) {
+		request->seqno = request->reserved_seqno;
+		WARN_ON(request->seqno != dev_priv->last_seqno);
+	}
+
 	/* Record the position of the start of the request so that
 	 * should we detect the updated seqno part-way through the
 	 * GPU processing the request, we never over-estimate the
@@ -2874,6 +2885,9 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)
 
 	list_for_each_entry_safe(req, req_next, &ring->fence_signal_list, signal_link) {
 		if (!req->cancelled) {
+			/* How can this happen? */
+			WARN_ON(req->seqno == 0);
+
 			if (!i915_seqno_passed(seqno, req->seqno))
 				break;
 		}
@@ -3023,7 +3037,14 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring,
 	if (req == NULL)
 		return -ENOMEM;
 
-	ret = i915_gem_get_seqno(ring->dev, &req->seqno);
+	/*
+	 * Assign an identifier to track this request through the hardware
+	 * but don't make it live yet. It could change in the future if this
+	 * request gets overtaken. However, it still needs to be allocated
+	 * in advance because the point of submission must not fail and seqno
+	 * allocation can fail.
+	 */
+	ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
 	if (ret)
 		goto err;
 
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 3c84b97..5e0e707 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1247,6 +1247,20 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Make sure the request's seqno is the latest and greatest: */
+	if (req->reserved_seqno != dev_priv->last_seqno) {
+		ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
+		if (ret)
+			return ret;
+	}
+	/*
+	 * And make it live because some of the execbuff submission code
+	 * requires the seqno to be available up front.
+	 */
+	WARN_ON(req->seqno);
+	req->seqno = req->reserved_seqno;
+	WARN_ON(req->seqno != dev_priv->last_seqno);
+
 	ret = intel_ring_reserve_space(req);
 	if (ret)
 		return ret;
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index d8ddf18..4037336 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -934,6 +934,20 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Make sure the request's seqno is the latest and greatest: */
+	if (req->reserved_seqno != dev_priv->last_seqno) {
+		ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
+		if (ret)
+			return ret;
+	}
+	/*
+	 * And make it live because some of the execbuff submission code
+	 * requires the seqno to be available up front.
+	 */
+	WARN_ON(req->seqno);
+	req->seqno = req->reserved_seqno;
+	WARN_ON(req->seqno != dev_priv->last_seqno);
+
 	ret = intel_logical_ring_reserve_space(req);
 	if (ret)
 		return ret;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 24/38] drm/i915: Added immediate submission override to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

To aid with debugging issues related to the scheduler, it can be
useful to ensure that all batch buffers are submitted immediately
rather than queued until later. This change adds an override flag via
the module parameter to force instant submission.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 7 +++++--
 drivers/gpu/drm/i915/i915_scheduler.h | 1 +
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index bfde3a2..4c1f92d 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -259,8 +259,11 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
 
-	not_flying = i915_scheduler_count_flying(scheduler, ring) <
-						 scheduler->min_flying;
+	if (i915.scheduler_override & i915_so_submit_on_queue)
+		not_flying = true;
+	else
+		not_flying = i915_scheduler_count_flying(scheduler, ring) <
+							 scheduler->min_flying;
 
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index cc9205b..e7f67c5 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -87,6 +87,7 @@ enum {
 /* Options for 'scheduler_override' module parameter: */
 enum {
 	i915_so_direct_submit       = (1 << 0),
+	i915_so_submit_on_queue     = (1 << 1),
 };
 
 bool        i915_scheduler_is_enabled(struct drm_device *dev);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 25/38] drm/i915: Added trace points to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Added trace points to the scheduler to track all the various events,
node state transitions and other interesting things that occur.

v2: Updated for new request completion tracking implementation.

v3: Updated for changes to node kill code.

v4: Wrapped some long lines to keep the style checker happy.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   2 +
 drivers/gpu/drm/i915/i915_scheduler.c      |  26 ++++
 drivers/gpu/drm/i915/i915_trace.h          | 193 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_lrc.c           |   2 +
 4 files changed, 223 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 5e0e707..87a8059 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1224,6 +1224,8 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
+	trace_i915_gem_ring_queue(ring, params);
+
 	qe = container_of(params, typeof(*qe), params);
 	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 4c1f92d..02b458f 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -150,6 +150,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
+		trace_i915_scheduler_queue(qe->params.ring, qe);
+
 		intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
 
 		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
@@ -265,6 +267,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		not_flying = i915_scheduler_count_flying(scheduler, ring) <
 							 scheduler->min_flying;
 
+	trace_i915_scheduler_queue(ring, node);
+	trace_i915_scheduler_node_state_change(ring, node);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (not_flying)
@@ -294,6 +299,9 @@ static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
 
 	node->status = i915_sqs_flying;
 
+	trace_i915_scheduler_fly(ring, node);
+	trace_i915_scheduler_node_state_change(ring, node);
+
 	if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
 		bool    success = true;
 
@@ -362,6 +370,8 @@ static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
 	/* Seqno will be reassigned on relaunch */
 	node->params.request->seqno = 0;
 	node->status = i915_sqs_queued;
+	trace_i915_scheduler_unfly(node->params.ring, node);
+	trace_i915_scheduler_node_state_change(node->params.ring, node);
 }
 
 /*
@@ -373,7 +383,11 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
 	WARN_ON(!node);
 	WARN_ON(I915_SQS_IS_COMPLETE(node));
 
+	if (I915_SQS_IS_FLYING(node))
+		trace_i915_scheduler_unfly(node->params.ring, node);
+
 	node->status = i915_sqs_dead;
+	trace_i915_scheduler_node_state_change(node->params.ring, node);
 }
 
 /*
@@ -393,6 +407,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	struct i915_scheduler_queue_entry *node = req->scheduler_qe;
 	unsigned long       flags;
 
+	trace_i915_scheduler_landing(req);
+
 	if (!node)
 		return false;
 
@@ -406,6 +422,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	else
 		node->status = i915_sqs_complete;
 
+	trace_i915_scheduler_node_state_change(req->ring, node);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	return true;
@@ -552,6 +570,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 	/* Launch more packets now? */
 	do_submit = (queued > 0) && (flying < scheduler->min_flying);
 
+	trace_i915_scheduler_remove(ring, min_seqno, do_submit);
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (!do_submit && list_empty(&remove))
@@ -566,6 +586,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		node = list_first_entry(&remove, typeof(*node), link);
 		list_del(&node->link);
 
+		trace_i915_scheduler_destroy(ring, node);
+
 		/* Free up all the DRM object references */
 		i915_gem_scheduler_clean_node(node);
 
@@ -835,6 +857,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 		INIT_LIST_HEAD(&best->link);
 		best->status  = i915_sqs_popped;
 
+		trace_i915_scheduler_node_state_change(ring, best);
+
 		ret = 0;
 	} else {
 		/* Can only get here if:
@@ -854,6 +878,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
 
 	/* i915_scheduler_dump_queue_pop(ring, best); */
 
+	trace_i915_scheduler_pop_from_queue(ring, best);
+
 	*pop_node = best;
 	return ret;
 }
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index 4c03754..6f33982 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -9,6 +9,7 @@
 #include "i915_drv.h"
 #include "intel_drv.h"
 #include "intel_ringbuffer.h"
+#include "i915_scheduler.h"
 
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM i915
@@ -823,6 +824,198 @@ TRACE_EVENT(switch_mm,
 		  __entry->dev, __entry->ring, __entry->to, __entry->vm)
 );
 
+TRACE_EVENT(i915_scheduler_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_fly,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_unfly,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_landing,
+	    TP_PROTO(struct drm_i915_gem_request *req),
+	    TP_ARGS(req),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     __field(u32, status)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = req->ring->id;
+			   __entry->uniq   = req->uniq;
+			   __entry->seqno  = req->seqno;
+			   __entry->status = req->scheduler_qe ?
+						req->scheduler_qe->status : ~0U;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d, status=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno,
+		      __entry->status)
+);
+
+TRACE_EVENT(i915_scheduler_remove,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     u32 min_seqno, bool do_submit),
+	    TP_ARGS(ring, min_seqno, do_submit),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, min_seqno)
+			     __field(bool, do_submit)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring      = ring->id;
+			   __entry->min_seqno = min_seqno;
+			   __entry->do_submit = do_submit;
+			   ),
+
+	    TP_printk("ring=%d, min_seqno = %d, do_submit=%d",
+		      __entry->ring, __entry->min_seqno, __entry->do_submit)
+);
+
+TRACE_EVENT(i915_scheduler_destroy,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_pop_from_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring  = ring->id;
+			   __entry->uniq  = node ? node->params.request->uniq  : 0;
+			   __entry->seqno = node ? node->params.request->seqno : 0;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno)
+);
+
+TRACE_EVENT(i915_scheduler_node_state_change,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_scheduler_queue_entry *node),
+	    TP_ARGS(ring, node),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, uniq)
+			     __field(u32, seqno)
+			     __field(u32, status)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = ring->id;
+			   __entry->uniq   = node ? node->params.request->uniq  : 0;
+			   __entry->seqno  = node->params.request->seqno;
+			   __entry->status = node->status;
+			   ),
+
+	    TP_printk("ring=%d, uniq=%d, seqno=%d, status=%d",
+		      __entry->ring, __entry->uniq, __entry->seqno,
+		      __entry->status)
+);
+
+TRACE_EVENT(i915_gem_ring_queue,
+	    TP_PROTO(struct intel_engine_cs *ring,
+		     struct i915_execbuffer_params *params),
+	    TP_ARGS(ring, params),
+
+	    TP_STRUCT__entry(
+			     __field(u32, ring)
+			     __field(u32, seqno)
+			     ),
+
+	    TP_fast_assign(
+			   __entry->ring   = ring->id;
+			   __entry->seqno  = params->request->seqno;
+			   ),
+
+	    TP_printk("ring=%d, seqno=%d", __entry->ring, __entry->seqno)
+);
+
 #endif /* _I915_TRACE_H_ */
 
 /* This part must be outside protection */
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 4037336..92d9b03 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -910,6 +910,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
+	trace_i915_gem_ring_queue(ring, params);
+
 	qe = container_of(params, typeof(*qe), params);
 	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 26/38] drm/i915: Added scheduler queue throttling by DRM file handle

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler decouples the submission of batch buffers to the driver
from their subsequent submission to the hardware. This means that an
application which is continuously submitting buffers as fast as it can
could potentialy flood the driver. To prevent this, the driver now
tracks how many buffers are in progress (queued in software or
executing in hardware) and limits this to a given (tunable) number. If
this number is exceeded then the queue to the driver will return
EAGAIN and thus prevent the scheduler's queue becoming arbitrarily
large.

v3: Added a missing decrement of the file queue counter.

v4: Updated a comment.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  2 ++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  8 +++++++
 drivers/gpu/drm/i915/i915_scheduler.c      | 35 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |  2 ++
 4 files changed, 47 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index a304918..1df2a2f 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -342,6 +342,8 @@ struct drm_i915_file_private {
 	} rps;
 
 	struct intel_engine_cs *bsd_ring;
+
+	u32 scheduler_queue_length;
 };
 
 enum intel_dpll_id {
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 87a8059..de8607e 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1756,6 +1756,10 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_is_full(file))
+		return -EAGAIN;
+
 	/* Copy in the exec list from userland */
 	exec_list = drm_malloc_ab(sizeof(*exec_list), args->buffer_count);
 	exec2_list = drm_malloc_ab(sizeof(*exec2_list), args->buffer_count);
@@ -1844,6 +1848,10 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_is_full(file))
+		return -EAGAIN;
+
 	exec2_list = kmalloc(sizeof(*exec2_list)*args->buffer_count,
 			     GFP_TEMPORARY | __GFP_NOWARN | __GFP_NORETRY);
 	if (exec2_list == NULL)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 02b458f..be5df22 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -39,6 +39,8 @@ static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *sch
 static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 						struct i915_scheduler_queue_entry *target,
 						uint32_t bump);
+static void        i915_scheduler_file_queue_inc(struct drm_file *file);
+static void        i915_scheduler_file_queue_dec(struct drm_file *file);
 
 bool i915_scheduler_is_enabled(struct drm_device *dev)
 {
@@ -73,6 +75,7 @@ int i915_scheduler_init(struct drm_device *dev)
 	scheduler->priority_level_max     = 1023;
 	scheduler->priority_level_preempt = 900;
 	scheduler->min_flying             = 2;
+	scheduler->file_queue_max         = 64;
 
 	dev_priv->scheduler = scheduler;
 
@@ -261,6 +264,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 
 	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
 
+	i915_scheduler_file_queue_inc(node->params.file);
+
 	if (i915.scheduler_override & i915_so_submit_on_queue)
 		not_flying = true;
 	else
@@ -554,6 +559,12 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 		/* Strip the dependency info while the mutex is still locked */
 		i915_scheduler_remove_dependent(scheduler, node);
 
+		/* Likewise clean up the file pointer. */
+		if (node->params.file) {
+			i915_scheduler_file_queue_dec(node->params.file);
+			node->params.file = NULL;
+		}
+
 		continue;
 	}
 
@@ -1093,6 +1104,7 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 						 node->status,
 						 ring->name);
 
+			i915_scheduler_file_queue_dec(node->params.file);
 			node->params.file = NULL;
 		}
 	}
@@ -1101,3 +1113,26 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 
 	return 0;
 }
+
+bool i915_scheduler_file_queue_is_full(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+	struct drm_i915_private      *dev_priv  = file_priv->dev_priv;
+	struct i915_scheduler        *scheduler = dev_priv->scheduler;
+
+	return file_priv->scheduler_queue_length >= scheduler->file_queue_max;
+}
+
+static void i915_scheduler_file_queue_inc(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+
+	file_priv->scheduler_queue_length++;
+}
+
+static void i915_scheduler_file_queue_dec(struct drm_file *file)
+{
+	struct drm_i915_file_private *file_priv = file->driver_priv;
+
+	file_priv->scheduler_queue_length--;
+}
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index e7f67c5..4cfc1fe 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -76,6 +76,7 @@ struct i915_scheduler {
 	int32_t             priority_level_max;
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
+	uint32_t            file_queue_max;
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -104,5 +105,6 @@ int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 				       unsigned long stamp, bool is_locked);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
+bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 27/38] drm/i915: Added debugfs interface to scheduler tuning parameters

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

There are various parameters within the scheduler which can be tuned
to improve performance, reduce memory footprint, etc. This change adds
support for altering these via debugfs.

v2: Updated for priorities now being signed values.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c | 141 ++++++++++++++++++++++++++++++++++++
 1 file changed, 141 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index e55294c..611e691 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -39,6 +39,7 @@
 #include "intel_ringbuffer.h"
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_scheduler.h"
 
 enum {
 	ACTIVE_LIST,
@@ -1122,6 +1123,141 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_next_seqno_fops,
 			i915_next_seqno_get, i915_next_seqno_set,
 			"0x%llx\n");
 
+static int
+i915_scheduler_priority_min_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_min;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_min_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->priority_level_min = (int32_t) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_min_fops,
+			i915_scheduler_priority_min_get,
+			i915_scheduler_priority_min_set,
+			"%lld\n");
+
+static int
+i915_scheduler_priority_max_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_max;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_max_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->priority_level_max = (int32_t) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_max_fops,
+			i915_scheduler_priority_max_get,
+			i915_scheduler_priority_max_set,
+			"%lld\n");
+
+static int
+i915_scheduler_priority_preempt_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_preempt;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_preempt_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->priority_level_preempt = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_preempt_fops,
+			i915_scheduler_priority_preempt_get,
+			i915_scheduler_priority_preempt_set,
+			"%lld\n");
+
+static int
+i915_scheduler_min_flying_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->min_flying;
+	return 0;
+}
+
+static int
+i915_scheduler_min_flying_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->min_flying = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_min_flying_fops,
+			i915_scheduler_min_flying_get,
+			i915_scheduler_min_flying_set,
+			"%llu\n");
+
+static int
+i915_scheduler_file_queue_max_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->file_queue_max;
+	return 0;
+}
+
+static int
+i915_scheduler_file_queue_max_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->file_queue_max = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_file_queue_max_fops,
+			i915_scheduler_file_queue_max_get,
+			i915_scheduler_file_queue_max_set,
+			"%llu\n");
+
 static int i915_frequency_info(struct seq_file *m, void *unused)
 {
 	struct drm_info_node *node = m->private;
@@ -5456,6 +5592,11 @@ static const struct i915_debugfs_files {
 	{"i915_gem_drop_caches", &i915_drop_caches_fops},
 	{"i915_error_state", &i915_error_state_fops},
 	{"i915_next_seqno", &i915_next_seqno_fops},
+	{"i915_scheduler_priority_min", &i915_scheduler_priority_min_fops},
+	{"i915_scheduler_priority_max", &i915_scheduler_priority_max_fops},
+	{"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops},
+	{"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops},
+	{"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
 	{"i915_display_crc_ctl", &i915_display_crc_ctl_fops},
 	{"i915_pri_wm_latency", &i915_pri_wm_latency_fops},
 	{"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 28/38] drm/i915: Added debug state dump facilities to scheduler

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

When debugging batch buffer submission issues, it is useful to be able
to see what the current state of the scheduler is. This change adds
functions for decoding the internal scheduler state and reporting it.

v3: Updated a debug message with the new state_str() function.

v4: Wrapped some long lines to keep the style checker happy. Removed
the fence/sync code as that will now be part of a separate patch series.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 281 +++++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_scheduler.h |  14 ++
 2 files changed, 293 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index be5df22..2b4b74f 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -35,6 +35,10 @@ static int         i915_scheduler_submit_max_priority(struct intel_engine_cs *ri
 						      bool is_locked);
 static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
 					       struct intel_engine_cs *ring);
+static int         i915_scheduler_dump_locked(struct intel_engine_cs *ring,
+					      const char *msg);
+static int         i915_scheduler_dump_all_locked(struct drm_device *dev,
+						  const char *msg);
 static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
 static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
 						struct i915_scheduler_queue_entry *target,
@@ -52,6 +56,116 @@ bool i915_scheduler_is_enabled(struct drm_device *dev)
 	return dev_priv->scheduler != NULL;
 }
 
+const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node)
+{
+	static char	str[50];
+	char		*ptr = str;
+
+	*(ptr++) = node->bumped ? 'B' : '-',
+	*(ptr++) = i915_gem_request_completed(node->params.request) ? 'C' : '-';
+
+	*ptr = 0;
+
+	return str;
+}
+
+char i915_scheduler_queue_status_chr(enum i915_scheduler_queue_status status)
+{
+	switch (status) {
+	case i915_sqs_none:
+	return 'N';
+
+	case i915_sqs_queued:
+	return 'Q';
+
+	case i915_sqs_popped:
+	return 'X';
+
+	case i915_sqs_flying:
+	return 'F';
+
+	case i915_sqs_complete:
+	return 'C';
+
+	case i915_sqs_dead:
+	return 'D';
+
+	default:
+	break;
+	}
+
+	return '?';
+}
+
+const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status)
+{
+	static char	str[50];
+
+	switch (status) {
+	case i915_sqs_none:
+	return "None";
+
+	case i915_sqs_queued:
+	return "Queued";
+
+	case i915_sqs_popped:
+	return "Popped";
+
+	case i915_sqs_flying:
+	return "Flying";
+
+	case i915_sqs_complete:
+	return "Complete";
+
+	case i915_sqs_dead:
+	return "Dead";
+
+	default:
+	break;
+	}
+
+	sprintf(str, "[Unknown_%d!]", status);
+	return str;
+}
+
+const char *i915_scheduler_flag_str(uint32_t flags)
+{
+	static char     str[100];
+	char           *ptr = str;
+
+	*ptr = 0;
+
+#define TEST_FLAG(flag, msg)						\
+	do {								\
+		if (flags & (flag)) {					\
+			strcpy(ptr, msg);				\
+			ptr += strlen(ptr);				\
+			flags &= ~(flag);				\
+		}							\
+	} while (0)
+
+	TEST_FLAG(i915_sf_interrupts_enabled, "IntOn|");
+	TEST_FLAG(i915_sf_submitting,         "Submitting|");
+	TEST_FLAG(i915_sf_dump_force,         "DumpForce|");
+	TEST_FLAG(i915_sf_dump_details,       "DumpDetails|");
+	TEST_FLAG(i915_sf_dump_dependencies,  "DumpDeps|");
+
+#undef TEST_FLAG
+
+	if (flags) {
+		sprintf(ptr, "Unknown_0x%X!", flags);
+		ptr += strlen(ptr);
+	}
+
+	if (ptr == str)
+		strcpy(str, "-");
+	else
+		ptr[-1] = 0;
+
+	return str;
+};
+
 int i915_scheduler_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -630,6 +744,169 @@ void i915_gem_scheduler_work_handler(struct work_struct *work)
 	}
 }
 
+int i915_scheduler_dump_all(struct drm_device *dev, const char *msg)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	unsigned long   flags;
+	int             ret;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	ret = i915_scheduler_dump_all_locked(dev, msg);
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return ret;
+}
+
+static int i915_scheduler_dump_all_locked(struct drm_device *dev,
+					  const char *msg)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs  *ring;
+	int                     i, r, ret = 0;
+
+	for_each_ring(ring, dev_priv, i) {
+		scheduler->flags[ring->id] |= i915_sf_dump_force   |
+					      i915_sf_dump_details |
+					      i915_sf_dump_dependencies;
+		r = i915_scheduler_dump_locked(ring, msg);
+		if (ret == 0)
+			ret = r;
+	}
+
+	return ret;
+}
+
+int i915_scheduler_dump(struct intel_engine_cs *ring, const char *msg)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	unsigned long   flags;
+	int             ret;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+	ret = i915_scheduler_dump_locked(ring, msg);
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return ret;
+}
+
+static int i915_scheduler_dump_locked(struct intel_engine_cs *ring,
+				      const char *msg)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	int                 flying = 0, queued = 0, complete = 0, other = 0;
+	static int          old_flying = -1, old_queued = -1, old_complete = -1;
+	bool                b_dump;
+	char                brkt[2] = { '<', '>' };
+
+	if (!ring)
+		return -EINVAL;
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (I915_SQS_IS_QUEUED(node))
+			queued++;
+		else if (I915_SQS_IS_FLYING(node))
+			flying++;
+		else if (I915_SQS_IS_COMPLETE(node))
+			complete++;
+		else
+			other++;
+	}
+
+	b_dump = (flying != old_flying) ||
+		 (queued != old_queued) ||
+		 (complete != old_complete);
+	if (scheduler->flags[ring->id] & i915_sf_dump_force) {
+		if (!b_dump) {
+			b_dump = true;
+			brkt[0] = '{';
+			brkt[1] = '}';
+		}
+
+		scheduler->flags[ring->id] &= ~i915_sf_dump_force;
+	}
+
+	if (b_dump) {
+		old_flying   = flying;
+		old_queued   = queued;
+		old_complete = complete;
+		DRM_DEBUG_DRIVER("<%s> Q:%02d, F:%02d, C:%02d, O:%02d, "
+				 "Flags = %s, Next = %d:%d %c%s%c\n",
+				 ring->name, queued, flying, complete, other,
+				 i915_scheduler_flag_str(scheduler->flags[ring->id]),
+				 dev_priv->request_uniq, dev_priv->next_seqno,
+				 brkt[0], msg, brkt[1]);
+	} else {
+		/*DRM_DEBUG_DRIVER("<%s> Q:%02d, F:%02d, C:%02d, O:%02d"
+				 ", Flags = %s, Next = %d:%d [%s]\n",
+				 ring->name,
+				 queued, flying, complete, other,
+				 i915_scheduler_flag_str(scheduler->flags[ring->id]),
+				 dev_priv->request_uniq, dev_priv->next_seqno, msg); */
+
+		return 0;
+	}
+
+	if (scheduler->flags[ring->id] & i915_sf_dump_details) {
+		int         i, deps;
+		uint32_t    count, counts[i915_sqs_MAX];
+
+		memset(counts, 0x00, sizeof(counts));
+
+		list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+			if (node->status < i915_sqs_MAX) {
+				count = counts[node->status]++;
+			} else {
+				DRM_DEBUG_DRIVER("<%s>   Unknown status: %d!\n",
+						 ring->name, node->status);
+				count = -1;
+			}
+
+			deps = 0;
+			for (i = 0; i < node->num_deps; i++)
+				if (i915_scheduler_is_dependency_valid(node, i))
+					deps++;
+
+			DRM_DEBUG_DRIVER("<%s>   %c:%02d> uniq = %d, seqno"
+					 " = %d/%s, deps = %d / %d, %s [pri = "
+					 "%4d]\n", ring->name,
+					 i915_scheduler_queue_status_chr(node->status),
+					 count,
+					 node->params.request->uniq,
+					 node->params.request->seqno,
+					 node->params.ring->name,
+					 deps, node->num_deps,
+					 i915_qe_state_str(node),
+					 node->priority);
+
+			if ((scheduler->flags[ring->id] & i915_sf_dump_dependencies)
+				== 0)
+				continue;
+
+			for (i = 0; i < node->num_deps; i++)
+				if (node->dep_list[i])
+					DRM_DEBUG_DRIVER("<%s>       |-%c:"
+						"%02d%c uniq = %d, seqno = %d/%s, %s [pri = %4d]\n",
+						ring->name,
+						i915_scheduler_queue_status_chr(node->dep_list[i]->status),
+						i,
+						i915_scheduler_is_dependency_valid(node, i)
+							? '>' : '#',
+						node->dep_list[i]->params.request->uniq,
+						node->dep_list[i]->params.request->seqno,
+						node->dep_list[i]->params.ring->name,
+						i915_qe_state_str(node->dep_list[i]),
+						node->dep_list[i]->priority);
+		}
+	}
+
+	return 0;
+}
+
 int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 			       unsigned long target,
 			       bool is_locked)
@@ -1098,10 +1375,10 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 				continue;
 
 			if (!I915_SQS_IS_COMPLETE(node))
-				DRM_DEBUG_DRIVER("Closing file handle with outstanding work: %d:%d/%d on %s\n",
+				DRM_DEBUG_DRIVER("Closing file handle with outstanding work: %d:%d/%s on %s\n",
 						 node->params.request->uniq,
 						 node->params.request->seqno,
-						 node->status,
+						 i915_qe_state_str(node),
 						 ring->name);
 
 			i915_scheduler_file_queue_dec(node->params.file);
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 4cfc1fe..990f094 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -41,6 +41,9 @@ enum i915_scheduler_queue_status {
 	/* Limit value for use with arrays/loops */
 	i915_sqs_MAX
 };
+char i915_scheduler_queue_status_chr(enum i915_scheduler_queue_status status);
+const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status);
 
 #define I915_SQS_IS_QUEUED(node)	(((node)->status == i915_sqs_queued))
 #define I915_SQS_IS_FLYING(node)	(((node)->status == i915_sqs_flying))
@@ -65,6 +68,7 @@ struct i915_scheduler_queue_entry {
 	unsigned long                       stamp;
 	struct list_head                    link;
 };
+const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node);
 
 struct i915_scheduler {
 	struct list_head    node_queue[I915_NUM_RINGS];
@@ -81,9 +85,16 @@ struct i915_scheduler {
 
 /* Flag bits for i915_scheduler::flags */
 enum {
+	/* Internal state */
 	i915_sf_interrupts_enabled  = (1 << 0),
 	i915_sf_submitting          = (1 << 1),
+
+	/* Dump/debug flags */
+	i915_sf_dump_force          = (1 << 8),
+	i915_sf_dump_details        = (1 << 9),
+	i915_sf_dump_dependencies   = (1 << 10),
 };
+const char *i915_scheduler_flag_str(uint32_t flags);
 
 /* Options for 'scheduler_override' module parameter: */
 enum {
@@ -103,6 +114,9 @@ void        i915_gem_scheduler_work_handler(struct work_struct *work);
 int         i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked);
 int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 				       unsigned long stamp, bool is_locked);
+int         i915_scheduler_dump(struct intel_engine_cs *ring,
+				const char *msg);
+int         i915_scheduler_dump_all(struct drm_device *dev, const char *msg);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
 bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 29/38] drm/i915: Add early exit to execbuff_final() if insufficient ring space

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

One of the major purposes of the GPU scheduler is to avoid stalling
the CPU when the GPU is busy and unable to accept more work. This
change adds support to the ring submission code to allow a ring space
check to be performed before attempting to submit a batch buffer to
the hardware. If insufficient space is available then the scheduler
can go away and come back later, letting the CPU get on with other
work, rather than stalling and waiting for the hardware to catch up.

v3: Updated to use locally cached request pointer.

v4: Line wrapped some comments differently to keep the style checker
happy. Downgraded a BUG_ON to a WARN_ON as the latter is preferred.

Removed some obsolete, commented out code.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 42 +++++++++++++++++------
 drivers/gpu/drm/i915/intel_lrc.c           | 54 +++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/intel_ringbuffer.c    | 26 ++++++++++++++
 drivers/gpu/drm/i915/intel_ringbuffer.h    |  1 +
 4 files changed, 108 insertions(+), 15 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index de8607e..6336479 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1099,25 +1099,19 @@ i915_reset_gen7_sol_offsets(struct drm_device *dev,
 {
 	struct intel_engine_cs *ring = req->ring;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	int ret, i;
+	int i;
 
 	if (!IS_GEN7(dev) || ring != &dev_priv->ring[RCS]) {
 		DRM_DEBUG("sol reset is gen7/rcs only\n");
 		return -EINVAL;
 	}
 
-	ret = intel_ring_begin(req, 4 * 3);
-	if (ret)
-		return ret;
-
 	for (i = 0; i < 4; i++) {
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, GEN7_SO_WRITE_OFFSET(i));
 		intel_ring_emit(ring, 0);
 	}
 
-	intel_ring_advance(ring);
-
 	return 0;
 }
 
@@ -1245,6 +1239,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	struct intel_engine_cs  *ring = params->ring;
 	u64 exec_start, exec_len;
 	int ret;
+	uint32_t min_space;
 
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
@@ -1267,8 +1262,36 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	if (ret)
 		return ret;
 
+	/*
+	 * It would be a bad idea to run out of space while writing commands
+	 * to the ring. One of the major aims of the scheduler is to not
+	 * stall at any point for any reason. However, doing an early exit
+	 * half way through submission could result in a partial sequence
+	 * being written which would leave the engine in an unknown state.
+	 * Therefore, check in advance that there will be enough space for
+	 * the entire submission whether emitted by the code below OR by any
+	 * other functions that may be executed before the end of final().
+	 *
+	 * NB: This test deliberately overestimates, because that's easier
+	 * than tracing every potential path that could be taken!
+	 *
+	 * Current measurements suggest that we may need to emit up to 186
+	 * dwords, so this is rounded up to 256 here. Then double that to get
+	 * the free space requirement, because the block is not allowed to
+	 * span the transition from the end to the beginning of the ring.
+	 */
+#define I915_BATCH_EXEC_MAX_LEN         256	/* max dwords emitted here */
+	min_space = I915_BATCH_EXEC_MAX_LEN * 2 * sizeof(uint32_t);
+	ret = intel_ring_test_space(req->ringbuf, min_space);
+	if (ret)
+		goto early_error;
+
 	intel_runtime_pm_get(dev_priv);
 
+	ret = intel_ring_begin(req, I915_BATCH_EXEC_MAX_LEN);
+	if (ret)
+		goto error;
+
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
@@ -1287,10 +1310,6 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_ring_begin(req, 4);
-		if (ret)
-			goto error;
-
 		intel_ring_emit(ring, MI_NOOP);
 		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit(ring, INSTPM);
@@ -1327,6 +1346,7 @@ error:
 	 */
 	intel_runtime_pm_put(dev_priv);
 
+early_error:
 	if (ret)
 		intel_ring_reserved_space_cancel(req->ringbuf);
 
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 92d9b03..ae5dd0a 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -226,6 +226,27 @@ static void lrc_setup_hardware_status_page(struct intel_engine_cs *ring,
 		struct drm_i915_gem_object *default_ctx_obj);
 
 
+/*
+ * Test to see if the ring has sufficient space to submit a given piece
+ * of work without causing a stall
+ */
+static int logical_ring_test_space(struct intel_ringbuffer *ringbuf,
+				   int min_space)
+{
+	if (ringbuf->space < min_space) {
+		/* Need to update the actual ring space. Otherwise, the system
+		 * hangs forever testing a software copy of the space value that
+		 * never changes!
+		 */
+		intel_ring_update_space(ringbuf);
+
+		if (ringbuf->space < min_space)
+			return -EAGAIN;
+	}
+
+	return 0;
+}
+
 /**
  * intel_sanitize_enable_execlists() - sanitize i915.enable_execlists
  * @dev: DRM device.
@@ -932,6 +953,7 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	struct intel_engine_cs *ring = params->ring;
 	u64 exec_start;
 	int ret;
+	uint32_t min_space;
 
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
@@ -955,6 +977,34 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 		return ret;
 
 	/*
+	 * It would be a bad idea to run out of space while writing commands
+	 * to the ring. One of the major aims of the scheduler is to not
+	 * stall at any point for any reason. However, doing an early exit
+	 * half way through submission could result in a partial sequence
+	 * being written which would leave the engine in an unknown state.
+	 * Therefore, check in advance that there will be enough space for
+	 * the entire submission whether emitted by the code below OR by any
+	 * other functions that may be executed before the end of final().
+	 *
+	 * NB: This test deliberately overestimates, because that's easier
+	 * than tracing every potential path that could be taken!
+	 *
+	 * Current measurements suggest that we may need to emit up to 186
+	 * dwords, so this is rounded up to 256 here. Then double that to get
+	 * the free space requirement, because the block is not allowed to
+	 * span the transition from the end to the beginning of the ring.
+	 */
+#define I915_BATCH_EXEC_MAX_LEN         256	/* max dwords emitted here */
+	min_space = I915_BATCH_EXEC_MAX_LEN * 2 * sizeof(uint32_t);
+	ret = logical_ring_test_space(ringbuf, min_space);
+	if (ret)
+		goto err;
+
+	ret = intel_logical_ring_begin(req, I915_BATCH_EXEC_MAX_LEN);
+	if (ret)
+		goto err;
+
+	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
@@ -964,10 +1014,6 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 
 	if (ring == &dev_priv->ring[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_logical_ring_begin(req, 4);
-		if (ret)
-			return ret;
-
 		intel_logical_ring_emit(ringbuf, MI_NOOP);
 		intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1));
 		intel_logical_ring_emit(ringbuf, INSTPM);
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 3f3d928..477235c 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -2472,6 +2472,32 @@ int intel_ring_cacheline_align(struct drm_i915_gem_request *req)
 	return 0;
 }
 
+/*
+ * Test to see if the ring has sufficient space to submit a given piece
+ * of work without causing a stall
+ */
+int intel_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space)
+{
+	struct drm_i915_private *dev_priv = ringbuf->ring->dev->dev_private;
+
+	/* There is a separate LRC version of this code. */
+	WARN_ON(i915.enable_execlists);
+
+	if (ringbuf->space < min_space) {
+		/* Need to update the actual ring space. Otherwise, the system
+		 * hangs forever testing a software copy of the space value that
+		 * never changes!
+		 */
+		ringbuf->head  = I915_READ_HEAD(ringbuf->ring);
+		ringbuf->space = intel_ring_space(ringbuf);
+
+		if (ringbuf->space < min_space)
+			return -EAGAIN;
+	}
+
+	return 0;
+}
+
 void intel_ring_init_seqno(struct intel_engine_cs *ring, u32 seqno)
 {
 	struct drm_device *dev = ring->dev;
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h b/drivers/gpu/drm/i915/intel_ringbuffer.h
index 136a039..bc447aa 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.h
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.h
@@ -446,6 +446,7 @@ void intel_cleanup_ring_buffer(struct intel_engine_cs *ring);
 
 int intel_ring_alloc_request_extras(struct drm_i915_gem_request *request);
 
+int intel_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space);
 int __must_check intel_ring_begin(struct drm_i915_gem_request *req, int n);
 int __must_check intel_ring_cacheline_align(struct drm_i915_gem_request *req);
 static inline void intel_ring_emit(struct intel_engine_cs *ring,
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 30/38] drm/i915: Added scheduler statistic reporting to debugfs

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

It is useful for know what the scheduler is doing for both debugging
and performance analysis purposes. This change adds a bunch of
counters and such that keep track of various scheduler operations
(batches submitted, completed, flush requests, etc.). The data can
then be read in userland via the debugfs mechanism.

v2: Updated to match changes to scheduler implementation.

v3: Updated for changes to kill code and flush code.

v4: Removed the fence/sync code as that will be part of a separate
patch series. Wrapped a long line to keep the style checker happy.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c        | 73 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  2 +
 drivers/gpu/drm/i915/i915_scheduler.c      | 70 ++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.h      | 31 +++++++++++++
 4 files changed, 171 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 611e691..73d0f97 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -3602,6 +3602,78 @@ static int i915_drrs_status(struct seq_file *m, void *unused)
 	return 0;
 }
 
+static int i915_scheduler_info(struct seq_file *m, void *unused)
+{
+	struct drm_info_node *node = (struct drm_info_node *) m->private;
+	struct drm_device *dev = node->minor->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_stats *stats = scheduler->stats;
+	struct i915_scheduler_stats_nodes node_stats[I915_NUM_RINGS];
+	struct intel_engine_cs *ring;
+	char   str[50 * (I915_NUM_RINGS + 1)], name[50], *ptr;
+	int ret, i, r;
+
+	ret = mutex_lock_interruptible(&dev->mode_config.mutex);
+	if (ret)
+		return ret;
+
+#define PRINT_VAR(name, fmt, var)					\
+	do {								\
+		sprintf(str, "%-22s", name);				\
+		ptr = str + strlen(str);				\
+		for_each_ring(ring, dev_priv, r) {			\
+			sprintf(ptr, " %10" fmt, var);			\
+			ptr += strlen(ptr);				\
+		}							\
+		seq_printf(m, "%s\n", str);				\
+	} while (0)
+
+	PRINT_VAR("Ring name:",             "s", dev_priv->ring[r].name);
+	PRINT_VAR("  Ring seqno",           "d", ring->get_seqno(ring, false));
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Batch submissions:\n");
+	PRINT_VAR("  Queued",               "u", stats[r].queued);
+	PRINT_VAR("  Submitted",            "u", stats[r].submitted);
+	PRINT_VAR("  Completed",            "u", stats[r].completed);
+	PRINT_VAR("  Expired",              "u", stats[r].expired);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Flush counts:\n");
+	PRINT_VAR("  By object",            "u", stats[r].flush_obj);
+	PRINT_VAR("  By request",           "u", stats[r].flush_req);
+	PRINT_VAR("  By stamp",             "u", stats[r].flush_stamp);
+	PRINT_VAR("  Blanket",              "u", stats[r].flush_all);
+	PRINT_VAR("  Entries bumped",       "u", stats[r].flush_bump);
+	PRINT_VAR("  Entries submitted",    "u", stats[r].flush_submit);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Miscellaneous:\n");
+	PRINT_VAR("  ExecEarly retry",      "u", stats[r].exec_early);
+	PRINT_VAR("  ExecFinal requeue",    "u", stats[r].exec_again);
+	PRINT_VAR("  ExecFinal killed",     "u", stats[r].exec_dead);
+	PRINT_VAR("  Hung flying",          "u", stats[r].kill_flying);
+	PRINT_VAR("  Hung queued",          "u", stats[r].kill_queued);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Queue contents:\n");
+	for_each_ring(ring, dev_priv, i)
+		i915_scheduler_query_stats(ring, node_stats + ring->id);
+
+	for (i = 0; i < (i915_sqs_MAX + 1); i++) {
+		sprintf(name, "  %s", i915_scheduler_queue_status_str(i));
+		PRINT_VAR(name, "d", node_stats[r].counts[i]);
+	}
+	seq_putc(m, '\n');
+
+#undef PRINT_VAR
+
+	mutex_unlock(&dev->mode_config.mutex);
+
+	return 0;
+}
+
 struct pipe_crc_info {
 	const char *name;
 	struct drm_device *dev;
@@ -5570,6 +5642,7 @@ static const struct drm_info_list i915_debugfs_list[] = {
 	{"i915_semaphore_status", i915_semaphore_status, 0},
 	{"i915_shared_dplls_info", i915_shared_dplls_info, 0},
 	{"i915_dp_mst_info", i915_dp_mst_info, 0},
+	{"i915_scheduler_info", i915_scheduler_info, 0},
 	{"i915_wa_registers", i915_wa_registers, 0},
 	{"i915_ddb_info", i915_ddb_info, 0},
 	{"i915_sseu_status", i915_sseu_status, 0},
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 6336479..a752b7f 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1743,6 +1743,8 @@ err:
 	mutex_unlock(&dev->struct_mutex);
 
 pre_mutex_err:
+	dev_priv->scheduler->stats[ring->id].exec_early++;
+
 	return ret;
 }
 
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 2b4b74f..666e142 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -121,6 +121,9 @@ const char *i915_scheduler_queue_status_str(
 	case i915_sqs_dead:
 	return "Dead";
 
+	case i915_sqs_MAX:
+	return "Invalid";
+
 	default:
 	break;
 	}
@@ -267,12 +270,15 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	if (i915.scheduler_override & i915_so_direct_submit) {
 		int ret;
 
+		scheduler->stats[qe->params.ring->id].queued++;
+
 		trace_i915_scheduler_queue(qe->params.ring, qe);
 
 		intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
 
 		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
 		ret = dev_priv->gt.execbuf_final(&qe->params);
+		scheduler->stats[qe->params.ring->id].submitted++;
 		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
 
 		/*
@@ -303,6 +309,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
 			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
 
+		scheduler->stats[qe->params.ring->id].expired++;
+
 		return 0;
 	}
 
@@ -386,6 +394,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 		not_flying = i915_scheduler_count_flying(scheduler, ring) <
 							 scheduler->min_flying;
 
+	scheduler->stats[ring->id].queued++;
+
 	trace_i915_scheduler_queue(ring, node);
 	trace_i915_scheduler_node_state_change(ring, node);
 
@@ -497,13 +507,17 @@ static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
  * Give up on a node completely. For example, because it is causing the
  * ring to hang or is using some resource that no longer exists.
  */
-static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
+static void i915_scheduler_node_kill(struct i915_scheduler *scheduler,
+				     struct i915_scheduler_queue_entry *node)
 {
 	WARN_ON(!node);
 	WARN_ON(I915_SQS_IS_COMPLETE(node));
 
-	if (I915_SQS_IS_FLYING(node))
+	if (I915_SQS_IS_FLYING(node)) {
 		trace_i915_scheduler_unfly(node->params.ring, node);
+		scheduler->stats[node->params.ring->id].kill_flying++;
+	} else
+		scheduler->stats[node->params.ring->id].kill_queued++;
 
 	node->status = i915_sqs_dead;
 	trace_i915_scheduler_node_state_change(node->params.ring, node);
@@ -536,10 +550,13 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	WARN_ON(!I915_SQS_IS_FLYING(node));
 
 	/* Node was in flight so mark it as complete. */
-	if (req->cancelled)
+	if (req->cancelled) {
 		node->status = i915_sqs_dead;
-	else
+		scheduler->stats[req->ring->id].kill_flying++;
+	} else {
 		node->status = i915_sqs_complete;
+		scheduler->stats[req->ring->id].completed++;
+	}
 
 	trace_i915_scheduler_node_state_change(req->ring, node);
 
@@ -669,6 +686,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)
 
 		list_del(&node->link);
 		list_add(&node->link, &remove);
+		scheduler->stats[ring->id].expired++;
 
 		/* Strip the dependency info while the mutex is still locked */
 		i915_scheduler_remove_dependent(scheduler, node);
@@ -907,6 +925,36 @@ static int i915_scheduler_dump_locked(struct intel_engine_cs *ring,
 	return 0;
 }
 
+int i915_scheduler_query_stats(struct intel_engine_cs *ring,
+			       struct i915_scheduler_stats_nodes *stats)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry  *node;
+	unsigned long   flags;
+
+	memset(stats, 0x00, sizeof(*stats));
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (node->status >= i915_sqs_MAX) {
+			DRM_DEBUG_DRIVER("Invalid node state: %d! [uniq = %d, seqno = %d]\n",
+					 node->status, node->params.request->uniq,
+					 node->params.request->seqno);
+
+			stats->counts[i915_sqs_MAX]++;
+			continue;
+		}
+
+		stats->counts[node->status]++;
+	}
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return 0;
+}
+
 int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 			       unsigned long target,
 			       bool is_locked)
@@ -935,6 +983,7 @@ int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 	}
 
 	spin_lock_irqsave(&scheduler->lock, flags);
+	scheduler->stats[ring->id].flush_stamp++;
 	i915_scheduler_priority_bump_clear(scheduler);
 	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
 		if (!I915_SQS_IS_QUEUED(node))
@@ -945,12 +994,15 @@ int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
 
 		flush_count = i915_scheduler_priority_bump(scheduler,
 					node, scheduler->priority_level_max);
+		scheduler->stats[ring->id].flush_bump += flush_count;
 	}
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (flush_count) {
 		DRM_DEBUG_DRIVER("<%s> Bumped %d entries\n", ring->name, flush_count);
 		flush_count = i915_scheduler_submit_max_priority(ring, is_locked);
+		if (flush_count > 0)
+			scheduler->stats[ring->id].flush_submit += flush_count;
 	}
 
 	return flush_count;
@@ -977,6 +1029,8 @@ int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
 
 	WARN_ON(is_locked && (scheduler->flags[ring->id] & i915_sf_submitting));
 
+	scheduler->stats[ring->id].flush_all++;
+
 	do {
 		found = false;
 		spin_lock_irqsave(&scheduler->lock, flags);
@@ -991,6 +1045,7 @@ int i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked)
 
 		if (found) {
 			ret = i915_scheduler_submit(ring, is_locked);
+			scheduler->stats[ring->id].flush_submit++;
 			if (ret < 0)
 				return ret;
 
@@ -1214,6 +1269,8 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 		 */
 		i915_scheduler_fly_node(node);
 
+		scheduler->stats[ring->id].submitted++;
+
 		scheduler->flags[ring->id] |= i915_sf_submitting;
 		spin_unlock_irqrestore(&scheduler->lock, flags);
 		ret = dev_priv->gt.execbuf_final(&node->params);
@@ -1234,6 +1291,7 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 			case ENOENT:
 				/* Fatal errors. Kill the node. */
 				requeue = -1;
+				scheduler->stats[ring->id].exec_dead++;
 			break;
 
 			case EAGAIN:
@@ -1243,6 +1301,7 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 			case ERESTARTSYS:
 			case EINTR:
 				/* Supposedly recoverable errors. */
+				scheduler->stats[ring->id].exec_again++;
 			break;
 
 			default:
@@ -1252,6 +1311,7 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 				 */
 				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
 						 ring->name, ret);
+				scheduler->stats[ring->id].exec_again++;
 			break;
 			}
 
@@ -1271,7 +1331,7 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 				 */
 				break;
 			} else if (requeue == -1)
-				i915_scheduler_node_kill(node);
+				i915_scheduler_node_kill(scheduler, node);
 		}
 
 		/* Keep launching until the sky is sufficiently full. */
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 990f094..eff5d8f 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -70,6 +70,32 @@ struct i915_scheduler_queue_entry {
 };
 const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node);
 
+struct i915_scheduler_stats_nodes {
+	uint32_t	counts[i915_sqs_MAX + 1];
+};
+
+struct i915_scheduler_stats {
+	/* Batch buffer counts: */
+	uint32_t            queued;
+	uint32_t            submitted;
+	uint32_t            completed;
+	uint32_t            expired;
+
+	/* Other stuff: */
+	uint32_t            flush_obj;
+	uint32_t            flush_req;
+	uint32_t            flush_stamp;
+	uint32_t            flush_all;
+	uint32_t            flush_bump;
+	uint32_t            flush_submit;
+
+	uint32_t            exec_early;
+	uint32_t            exec_again;
+	uint32_t            exec_dead;
+	uint32_t            kill_flying;
+	uint32_t            kill_queued;
+};
+
 struct i915_scheduler {
 	struct list_head    node_queue[I915_NUM_RINGS];
 	uint32_t            flags[I915_NUM_RINGS];
@@ -81,6 +107,9 @@ struct i915_scheduler {
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
 	uint32_t            file_queue_max;
+
+	/* Statistics: */
+	struct i915_scheduler_stats     stats[I915_NUM_RINGS];
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -119,6 +148,8 @@ int         i915_scheduler_dump(struct intel_engine_cs *ring,
 int         i915_scheduler_dump_all(struct drm_device *dev, const char *msg);
 bool        i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
 					      bool *completed, bool *busy);
+int         i915_scheduler_query_stats(struct intel_engine_cs *ring,
+				       struct i915_scheduler_stats_nodes *stats);
 bool        i915_scheduler_file_queue_is_full(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 31/38] drm/i915: Added seqno values to scheduler status dump

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

It is useful to be able to see what seqnos have actually popped out of
the hardware when viewing the scheduler status.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 10 ++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 2 files changed, 11 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 666e142..c93f69e 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -153,6 +153,7 @@ const char *i915_scheduler_flag_str(uint32_t flags)
 	TEST_FLAG(i915_sf_dump_force,         "DumpForce|");
 	TEST_FLAG(i915_sf_dump_details,       "DumpDetails|");
 	TEST_FLAG(i915_sf_dump_dependencies,  "DumpDeps|");
+	TEST_FLAG(i915_sf_dump_seqno,         "DumpSeqno|");
 
 #undef TEST_FLAG
 
@@ -787,6 +788,7 @@ static int i915_scheduler_dump_all_locked(struct drm_device *dev,
 	for_each_ring(ring, dev_priv, i) {
 		scheduler->flags[ring->id] |= i915_sf_dump_force   |
 					      i915_sf_dump_details |
+					      i915_sf_dump_seqno   |
 					      i915_sf_dump_dependencies;
 		r = i915_scheduler_dump_locked(ring, msg);
 		if (ret == 0)
@@ -869,6 +871,14 @@ static int i915_scheduler_dump_locked(struct intel_engine_cs *ring,
 		return 0;
 	}
 
+	if (scheduler->flags[ring->id] & i915_sf_dump_seqno) {
+		uint32_t    seqno;
+
+		seqno    = ring->get_seqno(ring, true);
+
+		DRM_DEBUG_DRIVER("<%s> Seqno = %d\n", ring->name, seqno);
+	}
+
 	if (scheduler->flags[ring->id] & i915_sf_dump_details) {
 		int         i, deps;
 		uint32_t    count, counts[i915_sqs_MAX];
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index eff5d8f..1cf40fc 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -122,6 +122,7 @@ enum {
 	i915_sf_dump_force          = (1 << 8),
 	i915_sf_dump_details        = (1 << 9),
 	i915_sf_dump_dependencies   = (1 << 10),
+	i915_sf_dump_seqno          = (1 << 11),
 };
 const char *i915_scheduler_flag_str(uint32_t flags);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 32/38] drm/i915: Add scheduler support functions for TDR

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The TDR code needs to know what the scheduler is up to in order to
work out whether a ring is really hung or not.

v4: Removed some unnecessary braces to keep the style checker happy.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 30 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 2 files changed, 31 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index c93f69e..d19590c 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -1461,6 +1461,36 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 	return 0;
 }
 
+/*
+ * Used by TDR to distinguish hung rings (not moving but with work to do)
+ * from idle rings (not moving because there is nothing to do).
+ */
+bool i915_scheduler_is_ring_flying(struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry *node;
+	unsigned long   flags;
+	bool            found = false;
+
+	/* With the scheduler in bypass mode, no information can be returned. */
+	if (i915.scheduler_override & i915_so_direct_submit)
+		return true;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (I915_SQS_IS_FLYING(node)) {
+			found = true;
+			break;
+		}
+	}
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return found;
+}
+
 bool i915_scheduler_file_queue_is_full(struct drm_file *file)
 {
 	struct drm_i915_file_private *file_priv = file->driver_priv;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 1cf40fc..73bf321 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -140,6 +140,7 @@ void        i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *nod
 int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void        i915_scheduler_wakeup(struct drm_device *dev);
+bool        i915_scheduler_is_ring_flying(struct intel_engine_cs *ring);
 void        i915_gem_scheduler_work_handler(struct work_struct *work);
 int         i915_scheduler_flush(struct intel_engine_cs *ring, bool is_locked);
 int         i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 33/38] drm/i915: GPU priority bumping to prevent starvation

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

If a high priority task was to continuously submit batch buffers to
the driver, it could starve out any lower priority task from getting
any GPU time at all. To prevent this, the priority of a queued batch
buffer is bumped each time it does not get submitted to the hardware.

v2: Updated for signed priority values.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c   | 28 ++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.c | 14 ++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 3 files changed, 43 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 73d0f97..1c4c6fe 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1178,6 +1178,33 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_max_fops,
 			"%lld\n");
 
 static int
+i915_scheduler_priority_bump_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_bump;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_bump_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->priority_level_bump = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_bump_fops,
+			i915_scheduler_priority_bump_get,
+			i915_scheduler_priority_bump_set,
+			"%lld\n");
+
+static int
 i915_scheduler_priority_preempt_get(void *data, u64 *val)
 {
 	struct drm_device       *dev       = data;
@@ -5667,6 +5694,7 @@ static const struct i915_debugfs_files {
 	{"i915_next_seqno", &i915_next_seqno_fops},
 	{"i915_scheduler_priority_min", &i915_scheduler_priority_min_fops},
 	{"i915_scheduler_priority_max", &i915_scheduler_priority_max_fops},
+	{"i915_scheduler_priority_bump", &i915_scheduler_priority_bump_fops},
 	{"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops},
 	{"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops},
 	{"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index d19590c..3edf856 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -191,6 +191,7 @@ int i915_scheduler_init(struct drm_device *dev)
 	/* Default tuning values: */
 	scheduler->priority_level_min     = -1023;
 	scheduler->priority_level_max     = 1023;
+	scheduler->priority_level_bump    = 50;
 	scheduler->priority_level_preempt = 900;
 	scheduler->min_flying             = 2;
 	scheduler->file_queue_max         = 64;
@@ -1352,6 +1353,19 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
 		ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
 	} while (ret == 0);
 
+	/*
+	 * Bump the priority of everything that was not submitted to prevent
+	 * starvation of low priority tasks by a spamming high priority task.
+	 */
+	i915_scheduler_priority_bump_clear(scheduler);
+	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+
+		i915_scheduler_priority_bump(scheduler, node,
+					     scheduler->priority_level_bump);
+	}
+
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
 	if (!was_locked)
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index 73bf321..d7aa542 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -104,6 +104,7 @@ struct i915_scheduler {
 	/* Tuning parameters: */
 	int32_t             priority_level_min;
 	int32_t             priority_level_max;
+	int32_t             priority_level_bump;
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
 	uint32_t            file_queue_max;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 34/38] drm/i915: Scheduler state dump via debugfs

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Added a facility for triggering the scheduler state dump via a debugfs
entry.

v2: New patch in series.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_debugfs.c   | 33 +++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.c |  9 +++++----
 drivers/gpu/drm/i915/i915_scheduler.h |  6 ++++++
 3 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 1c4c6fe..a99f9c5 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1285,6 +1285,38 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_file_queue_max_fops,
 			i915_scheduler_file_queue_max_set,
 			"%llu\n");
 
+static int
+i915_scheduler_dump_flags_get(void *data, u64 *val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	*val = scheduler->dump_flags;
+
+	return 0;
+}
+
+static int
+i915_scheduler_dump_flags_set(void *data, u64 val)
+{
+	struct drm_device       *dev       = data;
+	struct drm_i915_private *dev_priv  = dev->dev_private;
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+	scheduler->dump_flags = lower_32_bits(val) & i915_sf_dump_mask;
+
+	if (val & 1)
+		i915_scheduler_dump_all(dev, "DebugFS");
+
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_dump_flags_fops,
+			i915_scheduler_dump_flags_get,
+			i915_scheduler_dump_flags_set,
+			"0x%llx\n");
+
 static int i915_frequency_info(struct seq_file *m, void *unused)
 {
 	struct drm_info_node *node = m->private;
@@ -5698,6 +5730,7 @@ static const struct i915_debugfs_files {
 	{"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops},
 	{"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops},
 	{"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
+	{"i915_scheduler_dump_flags", &i915_scheduler_dump_flags_fops},
 	{"i915_display_crc_ctl", &i915_display_crc_ctl_fops},
 	{"i915_pri_wm_latency", &i915_pri_wm_latency_fops},
 	{"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
index 3edf856..8426927 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.c
+++ b/drivers/gpu/drm/i915/i915_scheduler.c
@@ -195,6 +195,10 @@ int i915_scheduler_init(struct drm_device *dev)
 	scheduler->priority_level_preempt = 900;
 	scheduler->min_flying             = 2;
 	scheduler->file_queue_max         = 64;
+	scheduler->dump_flags             = i915_sf_dump_force   |
+					    i915_sf_dump_details |
+					    i915_sf_dump_seqno   |
+					    i915_sf_dump_dependencies;
 
 	dev_priv->scheduler = scheduler;
 
@@ -787,10 +791,7 @@ static int i915_scheduler_dump_all_locked(struct drm_device *dev,
 	int                     i, r, ret = 0;
 
 	for_each_ring(ring, dev_priv, i) {
-		scheduler->flags[ring->id] |= i915_sf_dump_force   |
-					      i915_sf_dump_details |
-					      i915_sf_dump_seqno   |
-					      i915_sf_dump_dependencies;
+		scheduler->flags[ring->id] |= scheduler->dump_flags & i915_sf_dump_mask;
 		r = i915_scheduler_dump_locked(ring, msg);
 		if (ret == 0)
 			ret = r;
diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
index d7aa542..3d1484d 100644
--- a/drivers/gpu/drm/i915/i915_scheduler.h
+++ b/drivers/gpu/drm/i915/i915_scheduler.h
@@ -108,6 +108,7 @@ struct i915_scheduler {
 	int32_t             priority_level_preempt;
 	uint32_t            min_flying;
 	uint32_t            file_queue_max;
+	uint32_t            dump_flags;
 
 	/* Statistics: */
 	struct i915_scheduler_stats     stats[I915_NUM_RINGS];
@@ -124,6 +125,11 @@ enum {
 	i915_sf_dump_details        = (1 << 9),
 	i915_sf_dump_dependencies   = (1 << 10),
 	i915_sf_dump_seqno          = (1 << 11),
+
+	i915_sf_dump_mask           = i915_sf_dump_force        |
+				      i915_sf_dump_details      |
+				      i915_sf_dump_dependencies |
+				      i915_sf_dump_seqno,
 };
 const char *i915_scheduler_flag_str(uint32_t flags);
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 35/38] drm/i915: Enable GPU scheduler by default

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Now that all the scheduler patches have been applied, it is safe to enable.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_params.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_params.c b/drivers/gpu/drm/i915/i915_params.c
index be2e16c..7db0f83 100644
--- a/drivers/gpu/drm/i915/i915_params.c
+++ b/drivers/gpu/drm/i915/i915_params.c
@@ -54,7 +54,7 @@ struct i915_params i915 __read_mostly = {
 	.edp_vswing = 0,
 	.enable_guc_submission = true,
 	.guc_log_level = -1,
-	.scheduler_override = 1,
+	.scheduler_override = 0,
 };
 
 module_param_named(modeset, i915.modeset, int, 0400);
@@ -194,4 +194,4 @@ MODULE_PARM_DESC(guc_log_level,
 	"GuC firmware logging level (-1:disabled (default), 0-3:enabled)");
 
 module_param_named(scheduler_override, i915.scheduler_override, int, 0600);
-MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (0 = none, 1 = direct submission [default])");
+MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (default: 0)");
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 36/38] drm/i915: Add scheduling priority to per-context parameters

[John.C.Harrison]

From: Dave Gordon <david.s.gordon@intel.com>

Added an interface for user land applications/libraries/services to
set their GPU scheduler priority. This extends the existing context
parameter IOCTL interface to add a scheduler priority parameter. The
range is +/-1023 with +ve numbers meaning higher priority. Only
system processes may set a higher priority than the default (zero),
normal applications may only lower theirs.

v2: New patch in series.

For: VIZ-1587
Signed-off-by: Dave Gordon <David.S.Gordon@Intel.com>
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            | 14 ++++++++++++++
 drivers/gpu/drm/i915/i915_gem_context.c    | 24 ++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  3 +++
 include/uapi/drm/i915_drm.h                |  1 +
 4 files changed, 42 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 1df2a2f..f7cd9dc 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -843,6 +843,19 @@ struct i915_ctx_hang_stats {
 	bool banned;
 };
 
+/*
+ * User-settable GFX scheduler priorities are on a scale of -1023 (I don't
+ * care about running) to +1023 (I'm the most important thing in existence)
+ * with zero being the default. Any process may decrease its scheduling
+ * priority, but only a sufficiently privileged process may increase it
+ * beyond zero.
+ */
+
+struct i915_ctx_sched_info {
+	/* Scheduling priority */
+	int32_t priority;
+};
+
 struct i915_fence_timeline {
 	char        name[32];
 	unsigned    fence_context;
@@ -883,6 +896,7 @@ struct intel_context {
 	int flags;
 	struct drm_i915_file_private *file_priv;
 	struct i915_ctx_hang_stats hang_stats;
+	struct i915_ctx_sched_info sched_info;
 	struct i915_hw_ppgtt *ppgtt;
 
 	/* Legacy ring buffer submission */
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 4570edd..84aa942 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -947,6 +947,9 @@ int i915_gem_context_getparam_ioctl(struct drm_device *dev, void *data,
 		else
 			args->value = to_i915(dev)->gtt.base.total;
 		break;
+	case I915_CONTEXT_PARAM_PRIORITY:
+		args->value = (__u64) ctx->sched_info.priority;
+		break;
 	default:
 		ret = -EINVAL;
 		break;
@@ -984,6 +987,7 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
 		else
 			ctx->hang_stats.ban_period_seconds = args->value;
 		break;
+
 	case I915_CONTEXT_PARAM_NO_ZEROMAP:
 		if (args->size) {
 			ret = -EINVAL;
@@ -992,6 +996,26 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
 			ctx->flags |= args->value ? CONTEXT_NO_ZEROMAP : 0;
 		}
 		break;
+
+	case I915_CONTEXT_PARAM_PRIORITY:
+	{
+		int32_t	priority = (int32_t) args->value;
+		struct drm_i915_private *dev_priv  = dev->dev_private;
+		struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+		if (args->size)
+			ret = -EINVAL;
+		else if ((priority > scheduler->priority_level_max) ||
+			 (priority < scheduler->priority_level_min))
+			ret = -EINVAL;
+		else if ((priority > 0) &&
+			 !capable(CAP_SYS_ADMIN))
+			ret = -EPERM;
+		else
+			ctx->sched_info.priority = priority;
+		break;
+	}
+
 	default:
 		ret = -EINVAL;
 		break;
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index a752b7f..069a0a4 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1662,6 +1662,9 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
 
+	/* Start with the context's priority level */
+	qe.priority = ctx->sched_info.priority;
+
 	/*
 	 * Save away the list of objects used by this batch buffer for the
 	 * purpose of tracking inter-buffer dependencies.
diff --git a/include/uapi/drm/i915_drm.h b/include/uapi/drm/i915_drm.h
index 67cebe6..86a255c 100644
--- a/include/uapi/drm/i915_drm.h
+++ b/include/uapi/drm/i915_drm.h
@@ -1128,6 +1128,7 @@ struct drm_i915_gem_context_param {
 #define I915_CONTEXT_PARAM_BAN_PERIOD	0x1
 #define I915_CONTEXT_PARAM_NO_ZEROMAP	0x2
 #define I915_CONTEXT_PARAM_GTT_SIZE	0x3
+#define I915_CONTEXT_PARAM_PRIORITY	0x4
 	__u64 value;
 };
 
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 37/38] drm/i915: Add support for retro-actively banning batch buffers

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

If a given context submits too many hanging batch buffers then it will
be banned and no further batch buffers will be accepted for it.
However, it is possible that a large number of buffers may already
have been accepted and are sat in the scheduler waiting to be
executed. This patch adds a late ban check to ensure that these will
also be discarded.

v4: New patch in series.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 6 ++++++
 drivers/gpu/drm/i915/intel_lrc.c           | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 069a0a4..cb076e1 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1244,6 +1244,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Check the context wasn't banned between submission and execution: */
+	if (params->ctx->hang_stats.banned) {
+		DRM_DEBUG("Trying to execute for banned context!\n");
+		return -ENOENT;
+	}
+
 	/* Make sure the request's seqno is the latest and greatest: */
 	if (req->reserved_seqno != dev_priv->last_seqno) {
 		ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index ae5dd0a..b2861aa 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -958,6 +958,12 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Check the context wasn't banned between submission and execution: */
+	if (params->ctx->hang_stats.banned) {
+		DRM_DEBUG("Trying to execute for banned context!\n");
+		return -ENOENT;
+	}
+
 	/* Make sure the request's seqno is the latest and greatest: */
 	if (req->reserved_seqno != dev_priv->last_seqno) {
 		ret = i915_gem_get_seqno(ring->dev, &req->reserved_seqno);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 38/38] drm/i915: Allow scheduler to manage inter-ring object synchronisation

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The scheduler has always tracked batch buffer dependencies based on
DRM object usage. This means that it will not submit a batch on one
ring that has outstanding dependencies still executing on other rings.
This is exactly the same synchronisation performed by
i915_gem_object_sync() using hardware semaphores where available and
CPU stalls where not (e.g. in execlist mode and/or on Gen8 hardware).

Unfortunately, when a batch buffer is submitted to the driver the
_object_sync() call happens first. Thus in case where hardware
semaphores are disabled, the driver has already stalled until the
dependency has been resolved.

This patch adds an optimisation to _object_sync() to ignore the
synchronisation in the case where it will subsequently be handled by
the scheduler. This removes the driver stall and (in the single
application case) provides near hardware semaphore performance even
when hardware semaphores are disabled. In a busy system where there is
other work that can be executed on the stalling ring, it provides
better than hardware semaphore performance as it removes the stall
from both the driver and from the hardware. There is also a theory
that this method should improve power usage as hardware semaphores are
apparently not very power efficient - the stalled ring does not go
into as low a power a state as when it is genuinely idle.

The optimisation is to check whether both ends of the synchronisation
are batch buffer requests. If they are, then the scheduler will have
the inter-dependency tracked and managed. If one or other end is not a
batch buffer request (e.g. a page flip) then the code falls back to
the CPU stall or hardware semaphore as appropriate.

To check whether the existing usage is a batch buffer, the code simply
calls the 'are you tracking this request' function of the scheduler on
the object's last_read_req member. To check whether the new usage is a
batch buffer, a flag is passed in from the caller.

Issue: VIZ-5566
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h            |  2 +-
 drivers/gpu/drm/i915/i915_gem.c            | 17 ++++++++++++++---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  2 +-
 drivers/gpu/drm/i915/intel_display.c       |  2 +-
 drivers/gpu/drm/i915/intel_lrc.c           |  2 +-
 5 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index f7cd9dc..20c8a06 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2962,7 +2962,7 @@ int __must_check i915_mutex_lock_interruptible(struct drm_device *dev);
 #endif
 int i915_gem_object_sync(struct drm_i915_gem_object *obj,
 			 struct intel_engine_cs *to,
-			 struct drm_i915_gem_request **to_req);
+			 struct drm_i915_gem_request **to_req, bool to_batch);
 void i915_vma_move_to_active(struct i915_vma *vma,
 			     struct drm_i915_gem_request *req);
 int i915_gem_dumb_create(struct drm_file *file_priv,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index b37dbd7..05f55db 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3522,7 +3522,7 @@ static int
 __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 		       struct intel_engine_cs *to,
 		       struct drm_i915_gem_request *from_req,
-		       struct drm_i915_gem_request **to_req)
+		       struct drm_i915_gem_request **to_req, bool to_batch)
 {
 	struct intel_engine_cs *from;
 	int ret;
@@ -3534,6 +3534,15 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 	if (i915_gem_request_completed(from_req))
 		return 0;
 
+	/*
+	 * The scheduler will manage inter-ring object dependencies
+	 * as long as both to and from requests are scheduler managed
+	 * (i.e. batch buffers).
+	 */
+	if (to_batch &&
+	    i915_scheduler_is_request_tracked(from_req, NULL, NULL))
+		return 0;
+
 	if (!i915_semaphore_is_enabled(obj->base.dev)) {
 		struct drm_i915_private *i915 = to_i915(obj->base.dev);
 		ret = __i915_wait_request(from_req,
@@ -3584,6 +3593,8 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
  * @to_req: request we wish to use the object for. See below.
  *          This will be allocated and returned if a request is
  *          required but not passed in.
+ * @to_batch: is the sync request on behalf of batch buffer submission?
+ * If so then the scheduler can (potentially) manage the synchronisation.
  *
  * This code is meant to abstract object synchronization with the GPU.
  * Calling with NULL implies synchronizing the object with the CPU
@@ -3614,7 +3625,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 int
 i915_gem_object_sync(struct drm_i915_gem_object *obj,
 		     struct intel_engine_cs *to,
-		     struct drm_i915_gem_request **to_req)
+		     struct drm_i915_gem_request **to_req, bool to_batch)
 {
 	const bool readonly = obj->base.pending_write_domain == 0;
 	struct drm_i915_gem_request *req[I915_NUM_RINGS];
@@ -3636,7 +3647,7 @@ i915_gem_object_sync(struct drm_i915_gem_object *obj,
 				req[n++] = obj->last_read_req[i];
 	}
 	for (i = 0; i < n; i++) {
-		ret = __i915_gem_object_sync(obj, to, req[i], to_req);
+		ret = __i915_gem_object_sync(obj, to, req[i], to_req, to_batch);
 		if (ret)
 			return ret;
 	}
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index cb076e1..3059475 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -917,7 +917,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
 		struct drm_i915_gem_object *obj = vma->obj;
 
 		if (obj->active & other_rings) {
-			ret = i915_gem_object_sync(obj, req->ring, &req);
+			ret = i915_gem_object_sync(obj, req->ring, &req, true);
 			if (ret)
 				return ret;
 		}
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 58c464f..3612562 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -11487,7 +11487,7 @@ static int intel_crtc_page_flip(struct drm_crtc *crtc,
 	 * into the display plane and skip any waits.
 	 */
 	if (!mmio_flip) {
-		ret = i915_gem_object_sync(obj, ring, &request);
+		ret = i915_gem_object_sync(obj, ring, &request, false);
 		if (ret)
 			goto cleanup_pending;
 	}
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index b2861aa..8a6583f 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -655,7 +655,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,
 		struct drm_i915_gem_object *obj = vma->obj;
 
 		if (obj->active & other_rings) {
-			ret = i915_gem_object_sync(obj, req->ring, &req);
+			ret = i915_gem_object_sync(obj, req->ring, &req, true);
 			if (ret)
 				return ret;
 		}
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH] igt/gem_ctx_param_basic: Updated to support scheduler priority interface

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

The GPU scheduler has added an execution priority level to the context
object. There is an IOCTL interface to allow user apps/libraries to
set this priority. This patch updates the context paramter IOCTL test
to include the new interface.

For: VIZ-1587
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 lib/ioctl_wrappers.h        |  1 +
 tests/gem_ctx_param_basic.c | 34 +++++++++++++++++++++++++++++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/lib/ioctl_wrappers.h b/lib/ioctl_wrappers.h
index e575bb8..bf71bfd 100644
--- a/lib/ioctl_wrappers.h
+++ b/lib/ioctl_wrappers.h
@@ -92,6 +92,7 @@ struct local_i915_gem_context_param {
 #define LOCAL_CONTEXT_PARAM_BAN_PERIOD	0x1
 #define LOCAL_CONTEXT_PARAM_NO_ZEROMAP	0x2
 #define LOCAL_CONTEXT_PARAM_GTT_SIZE	0x3
+#define LOCAL_CONTEXT_PARAM_PRIORITY	0x4
 	uint64_t value;
 };
 void gem_context_require_ban_period(int fd);
diff --git a/tests/gem_ctx_param_basic.c b/tests/gem_ctx_param_basic.c
index b75800c..585a1a8 100644
--- a/tests/gem_ctx_param_basic.c
+++ b/tests/gem_ctx_param_basic.c
@@ -147,10 +147,42 @@ igt_main
 		TEST_SUCCESS(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM);
 	}
 
+	ctx_param.param = LOCAL_CONTEXT_PARAM_PRIORITY;
+
+	igt_subtest("priority-root-set") {
+		ctx_param.context = ctx;
+		ctx_param.value = 2048;
+		TEST_FAIL(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM, EINVAL);
+		ctx_param.value = -2048;
+		TEST_FAIL(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM, EINVAL);
+		ctx_param.value = 512;
+		TEST_SUCCESS(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM);
+		ctx_param.value = -512;
+		TEST_SUCCESS(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM);
+		ctx_param.value = 0;
+		TEST_SUCCESS(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM);
+	}
+
+	igt_subtest("priority-non-root-set") {
+		igt_fork(child, 1) {
+			igt_drop_root();
+
+			ctx_param.context = ctx;
+			ctx_param.value = 512;
+			TEST_FAIL(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM, EPERM);
+			ctx_param.value = -512;
+			TEST_SUCCESS(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM);
+			ctx_param.value = 0;
+			TEST_SUCCESS(LOCAL_IOCTL_I915_GEM_CONTEXT_SETPARAM);
+		}
+
+		igt_waitchildren();
+	}
+
 	/* NOTE: This testcase intentionally tests for the next free parameter
 	 * to catch ABI extensions. Don't "fix" this testcase without adding all
 	 * the tests for the new param first. */
-	ctx_param.param = LOCAL_CONTEXT_PARAM_GTT_SIZE + 1;
+	ctx_param.param = LOCAL_CONTEXT_PARAM_PRIORITY + 1;
 
 	igt_subtest("invalid-param-get") {
 		ctx_param.context = ctx;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 04/38] drm/i915: Split i915_dem_do_execbuffer() in half

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:33PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> Split the execbuffer() function in half. The first half collects and
> validates all the information required to process the batch buffer. It
> also does all the object pinning, relocations, active list management,
> etc - basically anything that must be done upfront before the IOCTL
> returns and allows the user land side to start changing/freeing
> things. The second half does the actual ring submission.

I don't get this at all. The point of requests is that GEM constructed a
request, which could be used to pass along all the implicit GEM
synchronisation points and the explict ones, along with the ringbuffer
to execute, to an engine that could then submit it. For legacy, the request
was inline and so added immediately to the ring (but that is an
implementation detail, there is nothing stopping us from using a chained
batch to implement a ring per context), for execlists the request is
queued for future execution. A scheduler was meant to sit in the middle
and determine the order in which requests were executed, but that should
be almost transparent to the high level code tracking the requests.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 06/38] drm/i915: Re-instate request->uniq because it is extremely useful

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:35PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> The seqno value cannot always be used when debugging issues via trace
> points. This is because it can be reset back to start, especially
> during TDR type tests. Also, when the scheduler arrives the seqno is
> only valid while a given request is executing on the hardware. While
> the request is simply queued waiting for submission, it's seqno value
> will be zero (meaning invalid).

Even with per-context seqno that can be assigned before execution as we
know that requests within a context cannot be reordered?
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 38/38] drm/i915: Allow scheduler to manage inter-ring object synchronisation

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:43:07PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> The scheduler has always tracked batch buffer dependencies based on
> DRM object usage. This means that it will not submit a batch on one
> ring that has outstanding dependencies still executing on other rings.
> This is exactly the same synchronisation performed by
> i915_gem_object_sync() using hardware semaphores where available and
> CPU stalls where not (e.g. in execlist mode and/or on Gen8 hardware).
> 
> Unfortunately, when a batch buffer is submitted to the driver the
> _object_sync() call happens first. Thus in case where hardware
> semaphores are disabled, the driver has already stalled until the
> dependency has been resolved.

But this should just add the dependency to the request in the scheduler
callback for i915_gem_object_sync_to, or better renamed as
i915_gem_request_submit_after. Without a scheduler we can do the
optimisation of doing that work inline, with a scheduler we can just
track the dependency.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 08/38] drm/i915: Prepare retire_requests to handle out-of-order seqnos

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:37PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> A major point of the GPU scheduler is that it re-orders batch buffers
> after they have been submitted to the driver. This leads to requests
> completing out of order. In turn, this means that the retire
> processing can no longer assume that all completed entries are at the
> front of the list. Rather than attempting to re-order the request list
> on a regular basis, it is better to simply scan the entire list.

This is a major misstep. Just think in terms of per-context timelines,
and retirment order within those timelines being consistent..
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 12/38] drm/i915: Added scheduler hook into i915_gem_request_notify()

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:41PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> The scheduler needs to know when requests have completed so that it
> can keep its own internal state up to date and can submit new requests
> to the hardware from its queue.

Why would you reuse the user interrupt rather than introduce a
context-switch interrupt using the pipe_notify/dword_notify (yes, it can
be done by fixing up the current code). In the case of execlists you
wouldn't even need to add another interrupt vector as you could just
overload the execlists submission routine. For legacy, this would at
least let you reduce the interrupt rate from per batch to per context
switch, and keep the logic separate for user request tracking.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 10/38] drm/i915: Force MMIO flips when scheduler enabled

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:39PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> MMIO flips are the preferred mechanism now but more importantly,

Says who?

> pipe
> based flips cause issues for the scheduler. Specifically, submitting
> work to the rings around the side of the scheduler could cause that
> work to be lost if the scheduler generates a pre-emption event on that
> ring.

That just says that you haven't designed for the ability to schedule a
flip into the scheduler, including handling the priority bump that might
required to hit the deadline.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 20/38] drm/i915: Added scheduler flush calls to ring throttle and idle functions

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:49PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> When requesting that all GPU work is completed, it is now necessary to
> get the scheduler involved in order to flush out work that queued and
> not yet submitted.

But why is this needed over and above waiting on the request? Why do we
actually need to flush the work as the scheduler will get to the request
eventually (one hopes!)?  Why the priority bump? That would not be
intended for either the idle (wait until everybody has finished) or
throttling.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 21/38] drm/i915: Added a module parameter for allowing scheduler overrides

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:50PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> It can be useful to be able to disable certain features (e.g. the
> entire scheduler) via a module parameter for debugging purposes. A
> parameter has the advantage of not being a compile time switch but
> without implying that it can be changed dynamically at runtime.

> +module_param_named(scheduler_override, i915.scheduler_override, int, 0600);
> +MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (0 = none, 1 = direct submission [default])");

Is this consistent with the other *enable* booleans?
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 18/38] drm/i915: Added scheduler support to __wait_request() calls

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:47PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> The scheduler can cause batch buffers, and hence requests, to be
> submitted to the ring out of order and asynchronously to their
> submission to the driver. Thus at the point of waiting for the
> completion of a given request, it is not even guaranteed that the
> request has actually been sent to the hardware yet. Even it is has
> been sent, it is possible that it could be pre-empted and thus
> 'unsent'.
> 
> This means that it is necessary to be able to submit requests to the
> hardware during the wait call itself. Unfortunately, while some
> callers of __wait_request() release the mutex lock first, others do
> not (and apparently can not). Hence there is the ability to deadlock
> as the wait stalls for submission but the asynchronous submission is
> stalled for the mutex lock.

That is a nonsequitor. Do you mean to say that unless we take action
inside GEM, the request will never be submitted to hardware by the
scheduler?
 
> This change hooks the scheduler in to the __wait_request() code to
> ensure correct behaviour. That is, flush the target batch buffer
> through to the hardware and do not deadlock waiting for something that
> cannot currently be submitted.

The dependencies are known during request construction, how could we
generate a cyclic graph? The scheduler itself does not need the
struct_mutex (other than the buggy code), so GEM holding the
struct_mutex will not prevent the scheduler from eventually submitting
the request we are waiting for. So as far as I can see, you are papering
over your own bugs.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 00/38] GPU scheduler for i915 driver

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:29PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> Implemented a batch buffer submission scheduler for the i915 DRM driver.

I've lost track of the number of patches that are a result of not having
per-context seqno and could be eliminated.
 
> The general theory of operation is that when batch buffers are
> submitted to the driver, the execbuffer() code assigns a unique seqno
> value and then packages up all the information required to execute the
> batch buffer at a later time. This package is given over to the
> scheduler which adds it to an internal node list. The scheduler also
> scans the list of objects associated with the batch buffer and
> compares them against the objects already in use by other buffers in
> the node list. If matches are found then the new batch buffer node is
> marked as being dependent upon the matching node. The same is done for
> the context object. The scheduler also bumps up the priority of such
> matching nodes on the grounds that the more dependencies a given batch
> buffer has the more important it is likely to be.

The implicit synchronisation rules for GEM are best left for GEM.
Through the existing mechansim for synchronising requests, you can also
gather the information required to compute the dependency graph of the
new request. Adding the explicit synchronisation can then be done at the
same juncture.

> The scheduler aims to have a given (tuneable) number of batch buffers
> in flight on the hardware at any given time. If fewer than this are
> currently executing when a new node is queued, then the node is passed
> straight through to the submit function. Otherwise it is simply added
> to the queue and the driver returns back to user land.
> 
> As each batch buffer completes, it raises an interrupt which wakes up
> the scheduler. Note that it is possible for multiple buffers to
> complete before the IRQ handler gets to run. Further, the seqno values
> of the individual buffers are not necessary incrementing as the
> scheduler may have re-ordered their submission. However, the scheduler
> keeps the list of executing buffers in order of hardware submission.
> Thus it can scan through the list until a matching seqno is found and
> then mark all in flight nodes from that point on as completed.
> 
> A deferred work queue is also poked by the interrupt handler. When
> this wakes up it can do more involved processing such as actually
> removing completed nodes from the queue and freeing up the resources
> associated with them (internal memory allocations, DRM object
> references, context reference, etc.). The work handler also checks the
> in flight count and calls the submission code if a new slot has
> appeared.

No. Leave GEM code to GEM. Just handle scheduling of requests, avoid the
struct_mutex and let GEM tidy up after the requests it is tracking. Create
a kthread for the scheduler, long running tasks are not meant to be on the
system_wq. A kthread also allows you to set a rtpriority.
 
> When the scheduler's submit code is called, it scans the queued node
> list for the highest priority node that has no unmet dependencies.
> Note that the dependency calculation is complex as it must take
> inter-ring dependencies and potential preemptions into account. Note
> also that in the future this will be extended to include external
> dependencies such as the Android Native Sync file descriptors and/or
> the linux dma-buff synchronisation scheme.

(You can skip the note since it is just checking if a dependency is a
struct fence and whether that has been signalled, that is not any more
complex than the current request checking.)

> If a suitable node is found then it is sent to execbuff_final() for
> submission to the hardware. The in flight count is then re-checked and
> a new node popped from the list if appropriate.

That was the wrong callback to break up. You just wanted an
engine->submit_request(). But then if you look at execlists, you will
see a way to marry the two such that the scheduler has neglible overhead
above and beyond the already considerable overhead of execlistss. With
legacy, you will have to introduce the cost of interrupt driven
scheduling, but you can borrow an idea or two from execlists to mitigate
that somewhat (i.e. context switch interrupts rather than reusing the
user interrupt after every batch).
 
> The scheduler also allows high priority batch buffers (e.g. from a
> desktop compositor) to jump ahead of whatever is already running if
> the underlying hardware supports pre-emption. In this situation, any
> work that was pre-empted is returned to the queued list ready to be
> resubmitted when no more high priority work is outstanding.

You could actually demonstrate that in execlists without adding a full
blown scheduler.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[Chris Wilson]

On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> A later patch in this series re-organises the batch buffer submission
> code. Part of that is to reduce the scope of a pm_get/put pair.
> Specifically, they previously wrapped the entire submission path from
> the very start to the very end, now they only wrap the actual hardware
> submission part in the back half.

However, as you haven't fixed the ordering issue that requires rpm_get
before struct_mutex, this is broken. When we have rpm fixed, you don't
need this patch as we can take the wakeref around the GSM access itself.
We still want the prolonged rpm wakeref for the GPU activity, ofc.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 00/38] GPU scheduler for i915 driver

[Tian, Kevin]

> From: John.C.Harrison@Intel.com
> Sent: Tuesday, January 12, 2016 2:42 AM
> 
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> Implemented a batch buffer submission scheduler for the i915 DRM driver.
> 
> The general theory of operation is that when batch buffers are
> submitted to the driver, the execbuffer() code assigns a unique seqno
> value and then packages up all the information required to execute the
> batch buffer at a later time. This package is given over to the
> scheduler which adds it to an internal node list. The scheduler also
> scans the list of objects associated with the batch buffer and
> compares them against the objects already in use by other buffers in
> the node list. If matches are found then the new batch buffer node is
> marked as being dependent upon the matching node. The same is done for
> the context object. The scheduler also bumps up the priority of such
> matching nodes on the grounds that the more dependencies a given batch
> buffer has the more important it is likely to be.
> 

A curious question. Is this new GPU scheduler still useful when GuC
is enabled? Sorry if this Q. has been answered before.

Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[John Harrison]

On 12/01/2016 00:20, Chris Wilson wrote:
> On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> A later patch in this series re-organises the batch buffer submission
>> code. Part of that is to reduce the scope of a pm_get/put pair.
>> Specifically, they previously wrapped the entire submission path from
>> the very start to the very end, now they only wrap the actual hardware
>> submission part in the back half.
> However, as you haven't fixed the ordering issue that requires rpm_get
> before struct_mutex, this is broken.
Why does 'intel_runtime_pm_get' require the struct mutex to be held? It 
has certainly not complained at me about trying to do stuff without it.


>   When we have rpm fixed, you don't
> need this patch as we can take the wakeref around the GSM access itself.
> We still want the prolonged rpm wakeref for the GPU activity, ofc.
> -Chris
>

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 06/38] drm/i915: Re-instate request->uniq because it is extremely useful

[John Harrison]

On 11/01/2016 22:04, Chris Wilson wrote:
> On Mon, Jan 11, 2016 at 06:42:35PM +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> The seqno value cannot always be used when debugging issues via trace
>> points. This is because it can be reset back to start, especially
>> during TDR type tests. Also, when the scheduler arrives the seqno is
>> only valid while a given request is executing on the hardware. While
>> the request is simply queued waiting for submission, it's seqno value
>> will be zero (meaning invalid).
> Even with per-context seqno that can be assigned before execution as we
> know that requests within a context cannot be reordered?
> -Chris
>
Firstly, we do not have per-context seqno values at the moment. However, 
even if we did that would make a per request unique value even more 
useful as the only way to identify a given request then would be through 
a context pointer and seqno pair which would make scanning through debug 
traces even worse.

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 10/38] drm/i915: Force MMIO flips when scheduler enabled

[John Harrison]

On 11/01/2016 22:16, Chris Wilson wrote:
> On Mon, Jan 11, 2016 at 06:42:39PM +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> MMIO flips are the preferred mechanism now but more importantly,
> Says who?

I asked this exact question at the linux architecture forum quite some 
time ago - does the scheduler need to worry about managing non-batch 
buffer work such as page flips. The answer from everyone present was no, 
MMIO flips are the way to go so don't over complicate the scheduler 
trying to support ring flips. Indeed, execlist mode already forces MMIO 
flips anyway.

>> pipe
>> based flips cause issues for the scheduler. Specifically, submitting
>> work to the rings around the side of the scheduler could cause that
>> work to be lost if the scheduler generates a pre-emption event on that
>> ring.
> That just says that you haven't designed for the ability to schedule a
> flip into the scheduler, including handling the priority bump that might
> required to hit the deadline.
> -Chris
>

Initially I was doing that. I was told to drop that work.

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 12/38] drm/i915: Added scheduler hook into i915_gem_request_notify()

[John Harrison]

On 11/01/2016 22:14, Chris Wilson wrote:
> On Mon, Jan 11, 2016 at 06:42:41PM +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> The scheduler needs to know when requests have completed so that it
>> can keep its own internal state up to date and can submit new requests
>> to the hardware from its queue.
> Why would you reuse the user interrupt rather than introduce a
> context-switch interrupt using the pipe_notify/dword_notify (yes, it can
> be done by fixing up the current code). In the case of execlists you
> wouldn't even need to add another interrupt vector as you could just
> overload the execlists submission routine. For legacy, this would at
> least let you reduce the interrupt rate from per batch to per context
> switch, and keep the logic separate for user request tracking.
> -Chris
>

One of the scheduler's design goals is to throttle the number of batches 
outstanding for any given context in order to improve responsiveness. In 
order to track this, it needs to know about each individual batch 
completion not the merged context completion (which could be for an 
arbitrarily large number of batches due to coalescing in the 
execlist/GuC layer).

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[Chris Wilson]

On Tue, Jan 12, 2016 at 11:11:20AM +0000, John Harrison wrote:
> On 12/01/2016 00:20, Chris Wilson wrote:
> >On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison@Intel.com wrote:
> >>From: John Harrison <John.C.Harrison@Intel.com>
> >>
> >>A later patch in this series re-organises the batch buffer submission
> >>code. Part of that is to reduce the scope of a pm_get/put pair.
> >>Specifically, they previously wrapped the entire submission path from
> >>the very start to the very end, now they only wrap the actual hardware
> >>submission part in the back half.
> >However, as you haven't fixed the ordering issue that requires rpm_get
> >before struct_mutex, this is broken.
> Why does 'intel_runtime_pm_get' require the struct mutex to be held?
> It has certainly not complained at me about trying to do stuff
> without it.

Because it depends upon the struct_mutex and rpm doesn't have sufficient
lockdep integration to be able to warn about using rpm from the
incorrect contexts.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 18/38] drm/i915: Added scheduler support to __wait_request() calls

[John Harrison]

On 11/01/2016 23:14, Chris Wilson wrote:
> On Mon, Jan 11, 2016 at 06:42:47PM +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> The scheduler can cause batch buffers, and hence requests, to be
>> submitted to the ring out of order and asynchronously to their
>> submission to the driver. Thus at the point of waiting for the
>> completion of a given request, it is not even guaranteed that the
>> request has actually been sent to the hardware yet. Even it is has
>> been sent, it is possible that it could be pre-empted and thus
>> 'unsent'.
>>
>> This means that it is necessary to be able to submit requests to the
>> hardware during the wait call itself. Unfortunately, while some
>> callers of __wait_request() release the mutex lock first, others do
>> not (and apparently can not). Hence there is the ability to deadlock
>> as the wait stalls for submission but the asynchronous submission is
>> stalled for the mutex lock.
> That is a nonsequitor. Do you mean to say that unless we take action
> inside GEM, the request will never be submitted to hardware by the
> scheduler?

Potentially. The scheduler holds on to batches inside its internal queue 
for later submission. It can also preempt batches that have already been 
sent to the hardware. Thus the wait call might be waiting on a batch 
with has or has not been submitted but even it is currently executing, 
it might get kicked out and need re-submitting. That submission requires 
calling the back end submission part of the driver - legacy ring buffer, 
execlist or GuC. Those back ends all require grabbing the mutex lock.

>   
>> This change hooks the scheduler in to the __wait_request() code to
>> ensure correct behaviour. That is, flush the target batch buffer
>> through to the hardware and do not deadlock waiting for something that
>> cannot currently be submitted.
> The dependencies are known during request construction, how could we
> generate a cyclic graph?
It is not so much a cyclic graph as a bouncing one. As noted above, even 
a batch buffer which is currently executing could get preempted and need 
to be resubmitted again while someone is waiting one it.

>   The scheduler itself does not need the
> struct_mutex (other than the buggy code),
The scheduler internally does not need the mutex lock at all - it has 
its own private spinlock for critical data. However, the back end 
submission paths do currently require the mutex lock.

>   so GEM holding the
> struct_mutex will not prevent the scheduler from eventually submitting
> the request we are waiting for. So as far as I can see, you are papering
> over your own bugs.
> -Chris
>

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 21/38] drm/i915: Added a module parameter for allowing scheduler overrides

[John Harrison]

On 11/01/2016 22:24, Chris Wilson wrote:
> On Mon, Jan 11, 2016 at 06:42:50PM +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> It can be useful to be able to disable certain features (e.g. the
>> entire scheduler) via a module parameter for debugging purposes. A
>> parameter has the advantage of not being a compile time switch but
>> without implying that it can be changed dynamically at runtime.
>> +module_param_named(scheduler_override, i915.scheduler_override, int, 0600);
>> +MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (0 = none, 1 = direct submission [default])");
> Is this consistent with the other *enable* booleans?
Initially there were a whole bunch of override flags for 
disabling/tweaking specific bits of the scheduler's operation. Since 
these extras now only exist in an internal debugging patch that is not 
going to be upstreamed, I guess it probably should be simplified to a 
bool rather than a flags word.


> -Chris
>

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 38/38] drm/i915: Allow scheduler to manage inter-ring object synchronisation

[John Harrison]

On 11/01/2016 22:07, Chris Wilson wrote:
> On Mon, Jan 11, 2016 at 06:43:07PM +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> The scheduler has always tracked batch buffer dependencies based on
>> DRM object usage. This means that it will not submit a batch on one
>> ring that has outstanding dependencies still executing on other rings.
>> This is exactly the same synchronisation performed by
>> i915_gem_object_sync() using hardware semaphores where available and
>> CPU stalls where not (e.g. in execlist mode and/or on Gen8 hardware).
>>
>> Unfortunately, when a batch buffer is submitted to the driver the
>> _object_sync() call happens first. Thus in case where hardware
>> semaphores are disabled, the driver has already stalled until the
>> dependency has been resolved.
> But this should just add the dependency to the request in the scheduler
> callback for i915_gem_object_sync_to, or better renamed as
> i915_gem_request_submit_after. Without a scheduler we can do the
> optimisation of doing that work inline, with a scheduler we can just
> track the dependency.
That's the whole point. The scheduler is already tracking the 
dependencies between batch buffers and will ensure that everything 
happens in the correct order. The problem is that the object sync code 
is manually forcing that order before the batch buffers even get to the 
scheduler, either through hardware semaphores (which have power and 
performance penalties) or CPU stalling (which is just a performance 
issue). Hence this patch is saying that if the dependency between the 
objects is something the scheduler knows about, i.e. it is batch buffer 
based, then don't bother doing the synchronisation up front. Just assume 
the scheduler will do it internally later.


> -Chris
>

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 00/38] GPU scheduler for i915 driver

[John Harrison]

On 12/01/2016 04:37, Tian, Kevin wrote:
>> From: John.C.Harrison@Intel.com
>> Sent: Tuesday, January 12, 2016 2:42 AM
>>
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> Implemented a batch buffer submission scheduler for the i915 DRM driver.
>>
>> The general theory of operation is that when batch buffers are
>> submitted to the driver, the execbuffer() code assigns a unique seqno
>> value and then packages up all the information required to execute the
>> batch buffer at a later time. This package is given over to the
>> scheduler which adds it to an internal node list. The scheduler also
>> scans the list of objects associated with the batch buffer and
>> compares them against the objects already in use by other buffers in
>> the node list. If matches are found then the new batch buffer node is
>> marked as being dependent upon the matching node. The same is done for
>> the context object. The scheduler also bumps up the priority of such
>> matching nodes on the grounds that the more dependencies a given batch
>> buffer has the more important it is likely to be.
>>
> A curious question. Is this new GPU scheduler still useful when GuC
> is enabled? Sorry if this Q. has been answered before.
Yes. The scheduler works with any back end submission mechanism - legacy 
ring buffer, execlist or Guc. Indeed, the pre-emption support (next 
patch series in the set) currently requires the GuC. Execlist support is 
possible but just not currently planned due to time constraints. Legacy 
ring buffer pre-emption is very different and a lot more work for very 
little benefit - pre-execlist hardware does not support very much in the 
way of pre-emption facilities.

The GuC itself does not really do much in the way of scheduling. It does 
know about the dependencies between batch buffers, for example, so 
cannot re-order work according to priority. Adding such support without 
still having large chunks of kernel driver support is a currently 
unscoped and unplanning task.


>
> Thanks
> Kevin

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[John Harrison]

On 12/01/2016 11:28, Chris Wilson wrote:
> On Tue, Jan 12, 2016 at 11:11:20AM +0000, John Harrison wrote:
>> On 12/01/2016 00:20, Chris Wilson wrote:
>>> On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison@Intel.com wrote:
>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>
>>>> A later patch in this series re-organises the batch buffer submission
>>>> code. Part of that is to reduce the scope of a pm_get/put pair.
>>>> Specifically, they previously wrapped the entire submission path from
>>>> the very start to the very end, now they only wrap the actual hardware
>>>> submission part in the back half.
>>> However, as you haven't fixed the ordering issue that requires rpm_get
>>> before struct_mutex, this is broken.
>> Why does 'intel_runtime_pm_get' require the struct mutex to be held?
>> It has certainly not complained at me about trying to do stuff
>> without it.
> Because it depends upon the struct_mutex and rpm doesn't have sufficient
> lockdep integration to be able to warn about using rpm from the
> incorrect contexts.
Where? What does the 'pm_runtime_get_sync' call turn into? There are 
already other places in the driver which call intel_runtime_pm_get() 
immediately after grabbing the mutex lock. Also, the description comment 
for _pm_get() does not mention anything about mutexes at all.

> -Chris
>

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 21/38] drm/i915: Added a module parameter for allowing scheduler overrides

[Chris Wilson]

On Tue, Jan 12, 2016 at 11:34:45AM +0000, John Harrison wrote:
> On 11/01/2016 22:24, Chris Wilson wrote:
> >On Mon, Jan 11, 2016 at 06:42:50PM +0000, John.C.Harrison@Intel.com wrote:
> >>From: John Harrison <John.C.Harrison@Intel.com>
> >>
> >>It can be useful to be able to disable certain features (e.g. the
> >>entire scheduler) via a module parameter for debugging purposes. A
> >>parameter has the advantage of not being a compile time switch but
> >>without implying that it can be changed dynamically at runtime.
> >>+module_param_named(scheduler_override, i915.scheduler_override, int, 0600);
> >>+MODULE_PARM_DESC(scheduler_override, "Scheduler override mask (0 = none, 1 = direct submission [default])");
> >Is this consistent with the other *enable* booleans?
> Initially there were a whole bunch of override flags for
> disabling/tweaking specific bits of the scheduler's operation. Since
> these extras now only exist in an internal debugging patch that is
> not going to be upstreamed, I guess it probably should be simplified
> to a bool rather than a flags word.

Yes, later on I saw the unsigned flags. Those should be restricted to a
debugfs interface, not a user parameter. module options are an
invitation for the world to fiddle, and they will.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 00/38] GPU scheduler for i915 driver

[Dave Gordon]

On 12/01/2016 11:43, John Harrison wrote:
> On 12/01/2016 04:37, Tian, Kevin wrote:
>>> From: John.C.Harrison@Intel.com
>>> Sent: Tuesday, January 12, 2016 2:42 AM
>>>
>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>
>>> Implemented a batch buffer submission scheduler for the i915 DRM 
>>> driver.
>>>
>>> The general theory of operation is that when batch buffers are
>>> submitted to the driver, the execbuffer() code assigns a unique seqno
>>> value and then packages up all the information required to execute the
>>> batch buffer at a later time. This package is given over to the
>>> scheduler which adds it to an internal node list. The scheduler also
>>> scans the list of objects associated with the batch buffer and
>>> compares them against the objects already in use by other buffers in
>>> the node list. If matches are found then the new batch buffer node is
>>> marked as being dependent upon the matching node. The same is done for
>>> the context object. The scheduler also bumps up the priority of such
>>> matching nodes on the grounds that the more dependencies a given batch
>>> buffer has the more important it is likely to be.
>>>
>> A curious question. Is this new GPU scheduler still useful when GuC
>> is enabled? Sorry if this Q. has been answered before.
> Yes. The scheduler works with any back end submission mechanism - 
> legacy ring buffer, execlist or Guc. Indeed, the pre-emption support 
> (next patch series in the set) currently requires the GuC. Execlist 
> support is possible but just not currently planned due to time 
> constraints. Legacy ring buffer pre-emption is very different and a 
> lot more work for very little benefit - pre-execlist hardware does not 
> support very much in the way of pre-emption facilities.
We have previously implemented preemption on gen7 ringbuffer, but just 
as a proof of concept and we're not going to push that upstream. 
Ringbuffer mode can in any case only support co-operative preemption, 
whereas execlist and GuC modes don't require (much) cooperation from 
preemptible tasks.
>
>
> The GuC itself does not really do much in the way of scheduling. It 
> does know about the 
In the line above, John meant "does NOT know"!

.Dave.
> dependencies between batch buffers, for example, so cannot re-order 
> work according to priority. Adding such support without still having 
> large chunks of kernel driver support is a currently unscoped and 
> unplanning task.
>
>
>>
>> Thanks
>> Kevin
>
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[Daniel Vetter]

On Tue, Jan 12, 2016 at 11:50:34AM +0000, John Harrison wrote:
> On 12/01/2016 11:28, Chris Wilson wrote:
> >On Tue, Jan 12, 2016 at 11:11:20AM +0000, John Harrison wrote:
> >>On 12/01/2016 00:20, Chris Wilson wrote:
> >>>On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison@Intel.com wrote:
> >>>>From: John Harrison <John.C.Harrison@Intel.com>
> >>>>
> >>>>A later patch in this series re-organises the batch buffer submission
> >>>>code. Part of that is to reduce the scope of a pm_get/put pair.
> >>>>Specifically, they previously wrapped the entire submission path from
> >>>>the very start to the very end, now they only wrap the actual hardware
> >>>>submission part in the back half.
> >>>However, as you haven't fixed the ordering issue that requires rpm_get
> >>>before struct_mutex, this is broken.
> >>Why does 'intel_runtime_pm_get' require the struct mutex to be held?
> >>It has certainly not complained at me about trying to do stuff
> >>without it.
> >Because it depends upon the struct_mutex and rpm doesn't have sufficient
> >lockdep integration to be able to warn about using rpm from the
> >incorrect contexts.
> Where? What does the 'pm_runtime_get_sync' call turn into? There are already
> other places in the driver which call intel_runtime_pm_get() immediately
> after grabbing the mutex lock. Also, the description comment for _pm_get()
> does not mention anything about mutexes at all.

If you nest rpm_get within dev->struct_mutex that's a bug and could
deadlock. Where does this happen? And for any such place we need a new
subtest in pm_rpm.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 10/38] drm/i915: Force MMIO flips when scheduler enabled

[Daniel Vetter]

On Tue, Jan 12, 2016 at 11:19:26AM +0000, John Harrison wrote:
> On 11/01/2016 22:16, Chris Wilson wrote:
> >On Mon, Jan 11, 2016 at 06:42:39PM +0000, John.C.Harrison@Intel.com wrote:
> >>From: John Harrison <John.C.Harrison@Intel.com>
> >>
> >>MMIO flips are the preferred mechanism now but more importantly,
> >Says who?
> 
> I asked this exact question at the linux architecture forum quite some time
> ago - does the scheduler need to worry about managing non-batch buffer work
> such as page flips. The answer from everyone present was no, MMIO flips are
> the way to go so don't over complicate the scheduler trying to support ring
> flips. Indeed, execlist mode already forces MMIO flips anyway.

Atomic will kill CS flips. We can mourn them and scream about the loss,
but imo best is to just skip that all and move on to acceptance. So mmio
flips (or well, atomic flips) is still the way to go for everything.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[John Harrison]

On 12/01/2016 14:04, Daniel Vetter wrote:
> On Tue, Jan 12, 2016 at 11:50:34AM +0000, John Harrison wrote:
>> On 12/01/2016 11:28, Chris Wilson wrote:
>>> On Tue, Jan 12, 2016 at 11:11:20AM +0000, John Harrison wrote:
>>>> On 12/01/2016 00:20, Chris Wilson wrote:
>>>>> On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison@Intel.com wrote:
>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>
>>>>>> A later patch in this series re-organises the batch buffer submission
>>>>>> code. Part of that is to reduce the scope of a pm_get/put pair.
>>>>>> Specifically, they previously wrapped the entire submission path from
>>>>>> the very start to the very end, now they only wrap the actual hardware
>>>>>> submission part in the back half.
>>>>> However, as you haven't fixed the ordering issue that requires rpm_get
>>>>> before struct_mutex, this is broken.
>>>> Why does 'intel_runtime_pm_get' require the struct mutex to be held?
>>>> It has certainly not complained at me about trying to do stuff
>>>> without it.
>>> Because it depends upon the struct_mutex and rpm doesn't have sufficient
>>> lockdep integration to be able to warn about using rpm from the
>>> incorrect contexts.
>> Where? What does the 'pm_runtime_get_sync' call turn into? There are already
>> other places in the driver which call intel_runtime_pm_get() immediately
>> after grabbing the mutex lock. Also, the description comment for _pm_get()
>> does not mention anything about mutexes at all.
> If you nest rpm_get within dev->struct_mutex that's a bug and could
> deadlock. Where does this happen? And for any such place we need a new
> subtest in pm_rpm.

The first two hits when grepping the driver are in 
'i915_gem_seqno_info()' and 'i915_interrupt_info()' in i915_debugfs.c. 
Both say:
     ret = mutex_lock_interruptible(&dev->struct_mutex);
     if (ret)
         return ret;
     intel_runtime_pm_get(dev_priv);


> -Daniel

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[Daniel Vetter]

On Tue, Jan 12, 2016 at 02:21:51PM +0000, John Harrison wrote:
> On 12/01/2016 14:04, Daniel Vetter wrote:
> >On Tue, Jan 12, 2016 at 11:50:34AM +0000, John Harrison wrote:
> >>On 12/01/2016 11:28, Chris Wilson wrote:
> >>>On Tue, Jan 12, 2016 at 11:11:20AM +0000, John Harrison wrote:
> >>>>On 12/01/2016 00:20, Chris Wilson wrote:
> >>>>>On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison@Intel.com wrote:
> >>>>>>From: John Harrison <John.C.Harrison@Intel.com>
> >>>>>>
> >>>>>>A later patch in this series re-organises the batch buffer submission
> >>>>>>code. Part of that is to reduce the scope of a pm_get/put pair.
> >>>>>>Specifically, they previously wrapped the entire submission path from
> >>>>>>the very start to the very end, now they only wrap the actual hardware
> >>>>>>submission part in the back half.
> >>>>>However, as you haven't fixed the ordering issue that requires rpm_get
> >>>>>before struct_mutex, this is broken.
> >>>>Why does 'intel_runtime_pm_get' require the struct mutex to be held?
> >>>>It has certainly not complained at me about trying to do stuff
> >>>>without it.
> >>>Because it depends upon the struct_mutex and rpm doesn't have sufficient
> >>>lockdep integration to be able to warn about using rpm from the
> >>>incorrect contexts.
> >>Where? What does the 'pm_runtime_get_sync' call turn into? There are already
> >>other places in the driver which call intel_runtime_pm_get() immediately
> >>after grabbing the mutex lock. Also, the description comment for _pm_get()
> >>does not mention anything about mutexes at all.
> >If you nest rpm_get within dev->struct_mutex that's a bug and could
> >deadlock. Where does this happen? And for any such place we need a new
> >subtest in pm_rpm.
> 
> The first two hits when grepping the driver are in 'i915_gem_seqno_info()'
> and 'i915_interrupt_info()' in i915_debugfs.c. Both say:
>     ret = mutex_lock_interruptible(&dev->struct_mutex);
>     if (ret)
>         return ret;
>     intel_runtime_pm_get(dev_priv);

Yeah that's totally bonkers and will deadlock if the device is actually
suspend.

/me goes and files JIRAs

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[Imre Deak]

On ti, 2016-01-12 at 16:35 +0100, Daniel Vetter wrote:
> On Tue, Jan 12, 2016 at 02:21:51PM +0000, John Harrison wrote:
> > On 12/01/2016 14:04, Daniel Vetter wrote:
> > > On Tue, Jan 12, 2016 at 11:50:34AM +0000, John Harrison wrote:
> > > > On 12/01/2016 11:28, Chris Wilson wrote:
> > > > > On Tue, Jan 12, 2016 at 11:11:20AM +0000, John Harrison
> > > > > wrote:
> > > > > > On 12/01/2016 00:20, Chris Wilson wrote:
> > > > > > > On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison
> > > > > > > @Intel.com wrote:
> > > > > > > > From: John Harrison <John.C.Harrison@Intel.com>
> > > > > > > > 
> > > > > > > > A later patch in this series re-organises the batch
> > > > > > > > buffer submission
> > > > > > > > code. Part of that is to reduce the scope of a
> > > > > > > > pm_get/put pair.
> > > > > > > > Specifically, they previously wrapped the entire
> > > > > > > > submission path from
> > > > > > > > the very start to the very end, now they only wrap the
> > > > > > > > actual hardware
> > > > > > > > submission part in the back half.
> > > > > > > However, as you haven't fixed the ordering issue that
> > > > > > > requires rpm_get
> > > > > > > before struct_mutex, this is broken.
> > > > > > Why does 'intel_runtime_pm_get' require the struct mutex to
> > > > > > be held?
> > > > > > It has certainly not complained at me about trying to do
> > > > > > stuff
> > > > > > without it.
> > > > > Because it depends upon the struct_mutex and rpm doesn't have
> > > > > sufficient
> > > > > lockdep integration to be able to warn about using rpm from
> > > > > the
> > > > > incorrect contexts.
> > > > Where? What does the 'pm_runtime_get_sync' call turn into?
> > > > There are already
> > > > other places in the driver which call intel_runtime_pm_get()
> > > > immediately
> > > > after grabbing the mutex lock. Also, the description comment
> > > > for _pm_get()
> > > > does not mention anything about mutexes at all.
> > > If you nest rpm_get within dev->struct_mutex that's a bug and
> > > could
> > > deadlock. Where does this happen? And for any such place we need
> > > a new
> > > subtest in pm_rpm.
> > 
> > The first two hits when grepping the driver are in
> > 'i915_gem_seqno_info()'
> > and 'i915_interrupt_info()' in i915_debugfs.c. Both say:
> >     ret = mutex_lock_interruptible(&dev->struct_mutex);
> >     if (ret)
> >         return ret;
> >     intel_runtime_pm_get(dev_priv);
> 
> Yeah that's totally bonkers and will deadlock if the device is
> actually
> suspend.

It won't deadlock, runtime resume doesn't need struct mutex. Runtime
suspend needs it, but we return -EAGAIN from there if it's already
held. That in turn will delay the suspend.

--Imre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[Daniel Vetter]

On Tue, Jan 12, 2016 at 05:59:46PM +0200, Imre Deak wrote:
> On ti, 2016-01-12 at 16:35 +0100, Daniel Vetter wrote:
> > On Tue, Jan 12, 2016 at 02:21:51PM +0000, John Harrison wrote:
> > > On 12/01/2016 14:04, Daniel Vetter wrote:
> > > > On Tue, Jan 12, 2016 at 11:50:34AM +0000, John Harrison wrote:
> > > > > On 12/01/2016 11:28, Chris Wilson wrote:
> > > > > > On Tue, Jan 12, 2016 at 11:11:20AM +0000, John Harrison
> > > > > > wrote:
> > > > > > > On 12/01/2016 00:20, Chris Wilson wrote:
> > > > > > > > On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison
> > > > > > > > @Intel.com wrote:
> > > > > > > > > From: John Harrison <John.C.Harrison@Intel.com>
> > > > > > > > > 
> > > > > > > > > A later patch in this series re-organises the batch
> > > > > > > > > buffer submission
> > > > > > > > > code. Part of that is to reduce the scope of a
> > > > > > > > > pm_get/put pair.
> > > > > > > > > Specifically, they previously wrapped the entire
> > > > > > > > > submission path from
> > > > > > > > > the very start to the very end, now they only wrap the
> > > > > > > > > actual hardware
> > > > > > > > > submission part in the back half.
> > > > > > > > However, as you haven't fixed the ordering issue that
> > > > > > > > requires rpm_get
> > > > > > > > before struct_mutex, this is broken.
> > > > > > > Why does 'intel_runtime_pm_get' require the struct mutex to
> > > > > > > be held?
> > > > > > > It has certainly not complained at me about trying to do
> > > > > > > stuff
> > > > > > > without it.
> > > > > > Because it depends upon the struct_mutex and rpm doesn't have
> > > > > > sufficient
> > > > > > lockdep integration to be able to warn about using rpm from
> > > > > > the
> > > > > > incorrect contexts.
> > > > > Where? What does the 'pm_runtime_get_sync' call turn into?
> > > > > There are already
> > > > > other places in the driver which call intel_runtime_pm_get()
> > > > > immediately
> > > > > after grabbing the mutex lock. Also, the description comment
> > > > > for _pm_get()
> > > > > does not mention anything about mutexes at all.
> > > > If you nest rpm_get within dev->struct_mutex that's a bug and
> > > > could
> > > > deadlock. Where does this happen? And for any such place we need
> > > > a new
> > > > subtest in pm_rpm.
> > > 
> > > The first two hits when grepping the driver are in
> > > 'i915_gem_seqno_info()'
> > > and 'i915_interrupt_info()' in i915_debugfs.c. Both say:
> > >     ret = mutex_lock_interruptible(&dev->struct_mutex);
> > >     if (ret)
> > >         return ret;
> > >     intel_runtime_pm_get(dev_priv);
> > 
> > Yeah that's totally bonkers and will deadlock if the device is
> > actually
> > suspend.
> 
> It won't deadlock, runtime resume doesn't need struct mutex. Runtime
> suspend needs it, but we return -EAGAIN from there if it's already
> held. That in turn will delay the suspend.

You're right, why do I always mix this up. But this just got me thinking
about possible lockdep annotations. We could place a
might_lock(rpm_lockdep_key) within rpm_get. And fake-acquire that around
the actual ->resume callback. We could even do this in the rpm core ...

That would put all my worries on this topic at easy I think, and I could
finally stop confusing everyone ;-)

Or did I again miss the obvious?
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation

[Chris Wilson]

On Tue, Jan 12, 2016 at 05:11:00PM +0100, Daniel Vetter wrote:
> On Tue, Jan 12, 2016 at 05:59:46PM +0200, Imre Deak wrote:
> > On ti, 2016-01-12 at 16:35 +0100, Daniel Vetter wrote:
> > > On Tue, Jan 12, 2016 at 02:21:51PM +0000, John Harrison wrote:
> > > > On 12/01/2016 14:04, Daniel Vetter wrote:
> > > > > On Tue, Jan 12, 2016 at 11:50:34AM +0000, John Harrison wrote:
> > > > > > On 12/01/2016 11:28, Chris Wilson wrote:
> > > > > > > On Tue, Jan 12, 2016 at 11:11:20AM +0000, John Harrison
> > > > > > > wrote:
> > > > > > > > On 12/01/2016 00:20, Chris Wilson wrote:
> > > > > > > > > On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison
> > > > > > > > > @Intel.com wrote:
> > > > > > > > > > From: John Harrison <John.C.Harrison@Intel.com>
> > > > > > > > > > 
> > > > > > > > > > A later patch in this series re-organises the batch
> > > > > > > > > > buffer submission
> > > > > > > > > > code. Part of that is to reduce the scope of a
> > > > > > > > > > pm_get/put pair.
> > > > > > > > > > Specifically, they previously wrapped the entire
> > > > > > > > > > submission path from
> > > > > > > > > > the very start to the very end, now they only wrap the
> > > > > > > > > > actual hardware
> > > > > > > > > > submission part in the back half.
> > > > > > > > > However, as you haven't fixed the ordering issue that
> > > > > > > > > requires rpm_get
> > > > > > > > > before struct_mutex, this is broken.
> > > > > > > > Why does 'intel_runtime_pm_get' require the struct mutex to
> > > > > > > > be held?
> > > > > > > > It has certainly not complained at me about trying to do
> > > > > > > > stuff
> > > > > > > > without it.
> > > > > > > Because it depends upon the struct_mutex and rpm doesn't have
> > > > > > > sufficient
> > > > > > > lockdep integration to be able to warn about using rpm from
> > > > > > > the
> > > > > > > incorrect contexts.
> > > > > > Where? What does the 'pm_runtime_get_sync' call turn into?
> > > > > > There are already
> > > > > > other places in the driver which call intel_runtime_pm_get()
> > > > > > immediately
> > > > > > after grabbing the mutex lock. Also, the description comment
> > > > > > for _pm_get()
> > > > > > does not mention anything about mutexes at all.
> > > > > If you nest rpm_get within dev->struct_mutex that's a bug and
> > > > > could
> > > > > deadlock. Where does this happen? And for any such place we need
> > > > > a new
> > > > > subtest in pm_rpm.
> > > > 
> > > > The first two hits when grepping the driver are in
> > > > 'i915_gem_seqno_info()'
> > > > and 'i915_interrupt_info()' in i915_debugfs.c. Both say:
> > > >     ret = mutex_lock_interruptible(&dev->struct_mutex);
> > > >     if (ret)
> > > >         return ret;
> > > >     intel_runtime_pm_get(dev_priv);
> > > 
> > > Yeah that's totally bonkers and will deadlock if the device is
> > > actually
> > > suspend.
> > 
> > It won't deadlock, runtime resume doesn't need struct mutex. Runtime
> > suspend needs it, but we return -EAGAIN from there if it's already
> > held. That in turn will delay the suspend.
> 
> You're right, why do I always mix this up. But this just got me thinking
> about possible lockdep annotations. We could place a
> might_lock(rpm_lockdep_key) within rpm_get. And fake-acquire that around
> the actual ->resume callback. We could even do this in the rpm core ...

Or we could make progress on eliminating the need for struct_mutex in
rpm resume/suspend/whenevner :)
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 10/38] drm/i915: Force MMIO flips when scheduler enabled

[Chris Wilson]

On Tue, Jan 12, 2016 at 03:07:03PM +0100, Daniel Vetter wrote:
> On Tue, Jan 12, 2016 at 11:19:26AM +0000, John Harrison wrote:
> > On 11/01/2016 22:16, Chris Wilson wrote:
> > >On Mon, Jan 11, 2016 at 06:42:39PM +0000, John.C.Harrison@Intel.com wrote:
> > >>From: John Harrison <John.C.Harrison@Intel.com>
> > >>
> > >>MMIO flips are the preferred mechanism now but more importantly,
> > >Says who?
> > 
> > I asked this exact question at the linux architecture forum quite some time
> > ago - does the scheduler need to worry about managing non-batch buffer work
> > such as page flips. The answer from everyone present was no, MMIO flips are
> > the way to go so don't over complicate the scheduler trying to support ring
> > flips. Indeed, execlist mode already forces MMIO flips anyway.

Two wrongs do not make a right, as they say. CS flips work very nicely
with execlists.

> Atomic will kill CS flips. We can mourn them and scream about the loss,
> but imo best is to just skip that all and move on to acceptance. So mmio
> flips (or well, atomic flips) is still the way to go for everything.

The real issue I think here is that not trying to feed a request into the
scheduler for the flip has lead to a poor interface into the scheduler.
For a CS flip request, we know the ordering, it's contents, we have to
choose the context though, but we have a good idea of the deadline which
gives a good challenge to a scheduler. That was my take.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 00/38] GPU scheduler for i915 driver

[Tian, Kevin]

> From: Gordon, David S
> Sent: Tuesday, January 12, 2016 9:49 PM
> 
> On 12/01/2016 11:43, John Harrison wrote:
> > On 12/01/2016 04:37, Tian, Kevin wrote:
> >>> From: John.C.Harrison@Intel.com
> >>> Sent: Tuesday, January 12, 2016 2:42 AM
> >>>
> >>> From: John Harrison <John.C.Harrison@Intel.com>
> >>>
> >>> Implemented a batch buffer submission scheduler for the i915 DRM
> >>> driver.
> >>>
> >>> The general theory of operation is that when batch buffers are
> >>> submitted to the driver, the execbuffer() code assigns a unique seqno
> >>> value and then packages up all the information required to execute the
> >>> batch buffer at a later time. This package is given over to the
> >>> scheduler which adds it to an internal node list. The scheduler also
> >>> scans the list of objects associated with the batch buffer and
> >>> compares them against the objects already in use by other buffers in
> >>> the node list. If matches are found then the new batch buffer node is
> >>> marked as being dependent upon the matching node. The same is done for
> >>> the context object. The scheduler also bumps up the priority of such
> >>> matching nodes on the grounds that the more dependencies a given batch
> >>> buffer has the more important it is likely to be.
> >>>
> >> A curious question. Is this new GPU scheduler still useful when GuC
> >> is enabled? Sorry if this Q. has been answered before.
> > Yes. The scheduler works with any back end submission mechanism -
> > legacy ring buffer, execlist or Guc. Indeed, the pre-emption support
> > (next patch series in the set) currently requires the GuC. Execlist
> > support is possible but just not currently planned due to time
> > constraints. Legacy ring buffer pre-emption is very different and a
> > lot more work for very little benefit - pre-execlist hardware does not
> > support very much in the way of pre-emption facilities.
> We have previously implemented preemption on gen7 ringbuffer, but just
> as a proof of concept and we're not going to push that upstream.
> Ringbuffer mode can in any case only support co-operative preemption,
> whereas execlist and GuC modes don't require (much) cooperation from
> preemptible tasks.
> >
> >
> > The GuC itself does not really do much in the way of scheduling. It
> > does know about the
> In the line above, John meant "does NOT know"!
> 
> .Dave.
> > dependencies between batch buffers, for example, so cannot re-order
> > work according to priority. Adding such support without still having
> > large chunks of kernel driver support is a currently unscoped and
> > unplanning task.
> >

Thank you both for clarification. It makes sense.

Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 10/38] drm/i915: Force MMIO flips when scheduler enabled

[John Harrison]

On 12/01/2016 21:53, Chris Wilson wrote:
> On Tue, Jan 12, 2016 at 03:07:03PM +0100, Daniel Vetter wrote:
>> On Tue, Jan 12, 2016 at 11:19:26AM +0000, John Harrison wrote:
>>> On 11/01/2016 22:16, Chris Wilson wrote:
>>>> On Mon, Jan 11, 2016 at 06:42:39PM +0000, John.C.Harrison@Intel.com wrote:
>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>
>>>>> MMIO flips are the preferred mechanism now but more importantly,
>>>> Says who?
>>> I asked this exact question at the linux architecture forum quite some time
>>> ago - does the scheduler need to worry about managing non-batch buffer work
>>> such as page flips. The answer from everyone present was no, MMIO flips are
>>> the way to go so don't over complicate the scheduler trying to support ring
>>> flips. Indeed, execlist mode already forces MMIO flips anyway.
> Two wrongs do not make a right, as they say. CS flips work very nicely
> with execlists.
They might have done at one point but if you don't test it then it don't 
work and right now it ain't being tested because:
     static bool use_mmio_flip(struct intel_engine_cs *ring,
               struct drm_i915_gem_object *obj)
     ...
         else if (i915.enable_execlists)
             return true;


>
>> Atomic will kill CS flips. We can mourn them and scream about the loss,
>> but imo best is to just skip that all and move on to acceptance. So mmio
>> flips (or well, atomic flips) is still the way to go for everything.
> The real issue I think here is that not trying to feed a request into the
> scheduler for the flip has lead to a poor interface into the scheduler.
> For a CS flip request, we know the ordering, it's contents, we have to
> choose the context though, but we have a good idea of the deadline which
> gives a good challenge to a scheduler. That was my take.
> -Chris
>

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 10/38] drm/i915: Force MMIO flips when scheduler enabled

[Chris Wilson]

On Wed, Jan 13, 2016 at 12:37:44PM +0000, John Harrison wrote:
> On 12/01/2016 21:53, Chris Wilson wrote:
> >On Tue, Jan 12, 2016 at 03:07:03PM +0100, Daniel Vetter wrote:
> >>On Tue, Jan 12, 2016 at 11:19:26AM +0000, John Harrison wrote:
> >>>On 11/01/2016 22:16, Chris Wilson wrote:
> >>>>On Mon, Jan 11, 2016 at 06:42:39PM +0000, John.C.Harrison@Intel.com wrote:
> >>>>>From: John Harrison <John.C.Harrison@Intel.com>
> >>>>>
> >>>>>MMIO flips are the preferred mechanism now but more importantly,
> >>>>Says who?
> >>>I asked this exact question at the linux architecture forum quite some time
> >>>ago - does the scheduler need to worry about managing non-batch buffer work
> >>>such as page flips. The answer from everyone present was no, MMIO flips are
> >>>the way to go so don't over complicate the scheduler trying to support ring
> >>>flips. Indeed, execlist mode already forces MMIO flips anyway.
> >Two wrongs do not make a right, as they say. CS flips work very nicely
> >with execlists.
> They might have done at one point but if you don't test it then it
> don't work and right now it ain't being tested because:
>     static bool use_mmio_flip(struct intel_engine_cs *ring,
>               struct drm_i915_gem_object *obj)
>     ...
>         else if (i915.enable_execlists)
>             return true;

Indeed, but that is not what I have in my kernels.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 07/38] drm/i915: Start of GPU scheduler

[Joonas Lahtinen]

Hi,

Comments below this pre text.

Many of the comments are related to the indent and style of the code.
That stuff is important to fix for future maintainability. In order for
the future review to be more effective, I'd like to next see a v5 of
the series where the code quality concerns have been addressed, patches
squashed to be actual reviewable chunks and appropriate kerneldoc being
added.

To give an idea of proper slicing of patches, first produce a no-op
scheduler, adding the extra function calls where needed and still
keeping the scheduling completely linear. Second patch could introduce
out of order submitting, third one priority bumping, fourth pre-empting 
and so on. That way, each patch extends the functionality and is itself
already mergeable. That way I've been able to go through and understand
the existing code, and I can actually review (other than just nag about
indent and coding style) if the changes are appropriate to bring in the
functionality desired.

In the current split, for me or anyone who did not participate writing
the code, it is otherwise too confusing to try to guess what future
changes might make each piece of code make sense, and which will be
redundant in the future too. There is no value in splitting code to
chunks that are not itself functional.

Regards, Joonas

On Mon, 2016-01-11 at 18:42 +0000, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> Initial creation of scheduler source files. Note that this patch
> implements most of the scheduler functionality but does not hook it
> in
> to the driver yet. It also leaves the scheduler code in 'pass
> through'
> mode so that even when it is hooked in, it will not actually do very
> much. This allows the hooks to be added one at a time in bite size
> chunks and only when the scheduler is finally enabled at the end does
> anything start happening.
> 
> The general theory of operation is that when batch buffers are
> submitted to the driver, the execbuffer() code packages up all the
> information required to execute the batch buffer at a later time.
> This
> package is given over to the scheduler which adds it to an internal
> node list. The scheduler also scans the list of objects associated
> with the batch buffer and compares them against the objects already
> in
> use by other buffers in the node list. If matches are found then the
> new batch buffer node is marked as being dependent upon the matching
> node. The same is done for the context object. The scheduler also
> bumps up the priority of such matching nodes on the grounds that the
> more dependencies a given batch buffer has the more important it is
> likely to be.
> 
> The scheduler aims to have a given (tuneable) number of batch buffers
> in flight on the hardware at any given time. If fewer than this are
> currently executing when a new node is queued, then the node is
> passed
> straight through to the submit function. Otherwise it is simply added
> to the queue and the driver returns back to user land.
> 
> The scheduler is notified when each batch buffer completes and
> updates
> its internal tracking accordingly. At the end of the completion
> interrupt processing, if any scheduler tracked batches were
> processed,
> the scheduler's deferred worker thread is woken up. This can do more
> involved processing such as actually removing completed nodes from
> the
> queue and freeing up the resources associated with them (internal
> memory allocations, DRM object references, context reference, etc.).
> The work handler also checks the in flight count and calls the
> submission code if a new slot has appeared.
> 
> When the scheduler's submit code is called, it scans the queued node
> list for the highest priority node that has no unmet dependencies.
> Note that the dependency calculation is complex as it must take
> inter-ring dependencies and potential preemptions into account. Note
> also that in the future this will be extended to include external
> dependencies such as the Android Native Sync file descriptors and/or
> the linux dma-buff synchronisation scheme.
> 
> If a suitable node is found then it is sent to execbuff_final() for
> submission to the hardware. The in flight count is then re-checked
> and
> a new node popped from the list if appropriate.
> 
> Note that this patch does not implement pre-emptive scheduling. Only
> basic scheduling by re-ordering batch buffer submission is currently
> implemented. Pre-emption of actively executing batch buffers comes in
> the next patch series.
> 
> v2: Changed priority levels to +/-1023 due to feedback from Chris
> Wilson.
> 
> Removed redundant index from scheduler node.
> 
> Changed time stamps to use jiffies instead of raw monotonic. This
> provides lower resolution but improved compatibility with other i915
> code.
> 
> Major re-write of completion tracking code due to struct fence
> conversion. The scheduler no longer has it's own private IRQ handler
> but just lets the existing request code handle completion events.
> Instead, the scheduler now hooks into the request notify code to be
> told when a request has completed.
> 
> Reduced driver mutex locking scope. Removal of scheduler nodes no
> longer grabs the mutex lock.
> 
> v3: Refactor of dependency generation to make the code more readable.
> Also added in read-read optimisation support - i.e., don't treat a
> shared read-only buffer as being a dependency.
> 
> Allowed the killing of queued nodes rather than only flying ones.
> 
> v4: Updated the commit message to better reflect the current state of
> the code. Downgraded some BUG_ONs to WARN_ONs. Used the correct array
> memory allocator function (kmalloc_array instead of kmalloc).
> Corrected the format of some comments. Wrapped some lines differently
> to keep the style checker happy.
> 
> Fixed a WARN_ON when killing nodes. The dependency removal code
> checks
> that nodes being destroyed do not have any oustanding dependencies
> (which would imply they should not have been executed yet). In the
> case of nodes being destroyed, e.g. due to context banning, then this
> might well be the case - they have not been executed and do indeed
> have outstanding dependencies.
> 
> Re-instated the code to disble interrupts when not in use. The
> underlying problem causing broken IRQ reference counts seems to have
> been fixed now.
> 
> For: VIZ-1587
> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>  drivers/gpu/drm/i915/Makefile         |   1 +
>  drivers/gpu/drm/i915/i915_drv.h       |   4 +
>  drivers/gpu/drm/i915/i915_gem.c       |   5 +
>  drivers/gpu/drm/i915/i915_scheduler.c | 797
> ++++++++++++++++++++++++++++++++++
>  drivers/gpu/drm/i915/i915_scheduler.h |  91 ++++
>  5 files changed, 898 insertions(+)
>  create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
>  create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
> 
> diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
> index 15398c5..79cb38b 100644
> --- a/drivers/gpu/drm/i915/Makefile
> +++ b/drivers/gpu/drm/i915/Makefile
> @@ -10,6 +10,7 @@ ccflags-y := -Werror
>  i915-y := i915_drv.o \
>  	  i915_irq.o \
>  	  i915_params.o \
> +	  i915_scheduler.o \
>            i915_suspend.o \
>  	  i915_sysfs.o \
>  	  intel_csr.o \
> diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
> index 10dffdd..38f423b 100644
> --- a/drivers/gpu/drm/i915/i915_drv.h
> +++ b/drivers/gpu/drm/i915/i915_drv.h
> @@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {
>  	struct drm_i915_gem_request     *request;
>  };
>  
> +struct i915_scheduler;
> +

Rather add "i915_scheduler.h" include at the top and eliminate circular
include dependencies. This is needed for the next comment.

>  /* used in computing the new watermarks state */
>  struct intel_wm_config {
>  	unsigned int num_pipes_active;
> @@ -1947,6 +1949,8 @@ struct drm_i915_private {
>  
>  	struct i915_runtime_pm pm;
>  
> +	struct i915_scheduler *scheduler;
> +

As the scheduler is going to be enabled on all platforms to an extent,
no point in making it a pointer. Just making it member like "pm" is the
best, this also requires the above include change.

>  	/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */
>  	struct {
>  		int (*execbuf_submit)(struct i915_execbuffer_params *params,
> diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
> index cff3768..47aa85b 100644
> --- a/drivers/gpu/drm/i915/i915_gem.c
> +++ b/drivers/gpu/drm/i915/i915_gem.c
> @@ -37,6 +37,7 @@
>  #include 
>  #include 
>  #include 
> +#include "i915_scheduler.h"
>  

This should go before any <linux/...> not to mask missing includes from
the header itself, so correct place would be right after #include
"i915_trace.h" 

>  #define RQ_BUG_ON(expr)
>  
> @@ -5242,6 +5243,10 @@ int i915_gem_init(struct drm_device *dev)
>  	 */
>  	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
>  
> +	ret = i915_scheduler_init(dev);
> +	if (ret)
> +		goto out_unlock;
> +
>  	ret = i915_gem_init_userptr(dev);
>  	if (ret)
>  		goto out_unlock;
> diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
> new file mode 100644
> index 0000000..8cb9063
> --- /dev/null
> +++ b/drivers/gpu/drm/i915/i915_scheduler.c
> @@ -0,0 +1,797 @@
> +/*
> + * Copyright (c) 2014 Intel Corporation
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice (including the next
> + * paragraph) shall be included in all copies or substantial portions of the
> + * Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
> + * IN THE SOFTWARE.
> + *
> + */
> +
> +#include "i915_drv.h"
> +#include "intel_drv.h"
> +#include "i915_scheduler.h"

Again, this include should be the top one, should not require any
includes before it.

> +
> +static int         i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node);
> +static int         i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
> +						   struct i915_scheduler_queue_entry *remove);
> +static int         i915_scheduler_submit(struct intel_engine_cs *ring,
> +					 bool is_locked);
> +static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
> +					       struct intel_engine_cs *ring);
> +static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
> +static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
> +						struct i915_scheduler_queue_entry *target,
> +						uint32_t bump);

Do not indent the function names like this, it becomes unmaintainable
and messy very fast if somebody adds a new function with a more complex
return type, which is a very likely thing to happen.

What I would do is move all the helper functions here at the top and
order them so that the forward declarations are not needed, at least
i915_scheduler_fly_node is not used before its definition.

> +
> +int i915_scheduler_init(struct drm_device *dev)
> +{
> +	struct drm_i915_private *dev_priv = dev->dev_private;
> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
> +	int                     r;

Again, this indent is a no-go. I'll not mention it on further
functions, assume it to be fixed for next revision.

> +
> +	if (scheduler)
> +		return 0;
> +
> +	scheduler = kzalloc(sizeof(*scheduler), GFP_KERNEL);
> +	if (!scheduler)
> +		return -ENOMEM;
> +
> +	spin_lock_init(&scheduler->lock);
> +
> +	for (r = 0; r < I915_NUM_RINGS; r++)
> +		INIT_LIST_HEAD(&scheduler->node_queue[r]);
> +
> +	/* Default tuning values: */
> +	scheduler->priority_level_min     = -1023;
> +	scheduler->priority_level_max     = 1023;
> +	scheduler->priority_level_preempt = 900;
> +	scheduler->min_flying             = 2;

This kind of indent is tolerable because it is a contained code block,
but not needed either.

> +
> +	dev_priv->scheduler = scheduler;
> +
> +	return 0;
> +}
> +
> +static void i915_generate_dependencies(struct i915_scheduler *scheduler,
> +				       struct i915_scheduler_queue_entry *node,
> +				       uint32_t ring)
> +{
> +	struct i915_scheduler_obj_entry *this, *that;
> +	struct i915_scheduler_queue_entry  *test;
> +	int                 i, j;
> +	bool                found;
> +
> +	list_for_each_entry(test, &scheduler->node_queue[ring], link) {
> +		if (I915_SQS_IS_COMPLETE(test))
> +			continue;
> +
> +		/*
> +		 * Batches on the same ring for the same
> +		 * context must be kept in order.
> +		 */
> +		found = (node->params.ctx == test->params.ctx) &&
> +			(node->params.ring == test->params.ring);
> +
> +		if (found)
> +			goto depends;
> +

This is not needed.

> +		/*
> +		 * Batches working on the same objects must
> +		 * be kept in order.
> +		 */
> +		for (i = 0; (i < node->num_objs) && !found; i++) {

As the test is here already                         ---^

> +			this = node->saved_objects + i;
> +
> +			for (j = 0; j < test->num_objs; j++) {
> +				that = test->saved_objects + j;
> +
> +				if (this->obj != that->obj)
> +					continue;

How about VMAs? There might be multiple mappings to an object, isn't it
enough to depend on the required VMA instead of the whole object?

> +
> +				/* Only need to worry about writes */
> +				if (this->read_only && that->read_only)
> +					continue;
> +
> +				found = true;
> +				break;
> +			}
> +		}
> +

The following block is not needed.

> +		if (!found)
> +			continue;
> +
> +depends:

Rather like this, in order to avoid a goto label;
if (found) {
...


> +		node->dep_list[node->num_deps] = test;
> +		node->num_deps++;
> +	}
> +}
> +

Please add a brief kerneldoc above each function in the header, it's
required. Adding it to non-trivial inline helper functions too will
make reviewing much easier.

> +int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
> +{
> +	struct drm_i915_private *dev_priv = qe->params.dev->dev_private;
> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
> +	struct intel_engine_cs  *ring = qe->params.ring;
> +	struct i915_scheduler_queue_entry  *node;
> +	struct i915_scheduler_queue_entry  *test;
> +	unsigned long       flags;
> +	bool                not_flying;
> +	int                 i, r;
> +	int                 incomplete = 0;
> +
> +	WARN_ON(!scheduler);
> +

This kind of situations should have a be a BUG_ON, because scheduler
being zero is literally going to cause an OOPS in the next dereference
which is going to happen unconditionally. WARN + OOPS is kind of what
BUG_ON should be used avoid. But this should be removed anyway after
scheduler is made a data member of dev_priv.

> +	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {

I assume this is going to be addressed in a future commit. Could have
been introduced in this patch, too.

> +		int ret;
> +
> +		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
> +		ret = dev_priv->gt.execbuf_final(&qe->params);
> +		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
> +

The kerneldoc should mention locking requirements of this function.

> +		/*
> +		 * Don't do any clean up on failure because the caller will
> +		 * do it all anyway.
> +		 */
> +		if (ret)
> +			return ret;
> +
> +		/* Free everything that is owned by the QE structure: */
> +		kfree(qe->params.cliprects);
> +		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
> +			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
> +
> +		return 0;

Above piece of code looks like its own function, so it should probably
be one.

> +	}
> +
> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
> +	if (!node)
> +		return -ENOMEM;
> +
> +	*node = *qe;

Any reason we can't simply move ownership of qe? If not, I'd rather
make a clone function

> +	INIT_LIST_HEAD(&node->link);
> +	node->status = i915_sqs_queued;
> +	node->stamp  = jiffies;
> +	i915_gem_request_reference(node->params.request);
> +
> +	/* Need to determine the number of incomplete entries in the list as
> +	 * that will be the maximum size of the dependency list.
> +	 *
> +	 * Note that the allocation must not be made with the spinlock acquired
> +	 * as kmalloc can sleep. However, the unlock/relock is safe because no
> +	 * new entries can be queued up during the unlock as the i915 driver
> +	 * mutex is still held. Entries could be removed from the list but that
> +	 * just means the dep_list will be over-allocated which is fine.
> +	 */
> +	spin_lock_irqsave(&scheduler->lock, flags);
> +	for (r = 0; r < I915_NUM_RINGS; r++) {
> +		list_for_each_entry(test, &scheduler->node_queue[r], link) {
> +			if (I915_SQS_IS_COMPLETE(test))
> +				continue;
> +
> +			incomplete++;
> +		}
> +	}
> +
> +	/* Temporarily unlock to allocate memory: */
> +	spin_unlock_irqrestore(&scheduler->lock, flags);

I'd make the above piece of code a helper, these stats are to be
counted for debugfs anyway, too?

> +	if (incomplete) {
> +		node->dep_list = kmalloc_array(incomplete,
> +					       sizeof(*node->dep_list),
> +					       GFP_KERNEL);
> +		if (!node->dep_list) {
> +			kfree(node);
> +			return -ENOMEM;
> +		}
> +	} else
> +		node->dep_list = NULL;
> +
> +	spin_lock_irqsave(&scheduler->lock, flags);
> +	node->num_deps = 0;
> +
> +	if (node->dep_list) {
> +		for (r = 0; r < I915_NUM_RINGS; r++)
> +			i915_generate_dependencies(scheduler, node, r);
> +
> +		WARN_ON(node->num_deps > incomplete);

Wouldn't this condition again lead to a crash? If so, should be BUG_ON
to cause that crash as early as possible. WARN_ON is only good if there
is a way of coping with the situation and no imminent system crash is
bound to happen.

> +	}
> +
> +	if (node->priority > scheduler->priority_level_max)
> +		node->priority = scheduler->priority_level_max;
> +	else if (node->priority < scheduler->priority_level_min)
> +		node->priority = scheduler->priority_level_min;
> +

There is clamp_val macro in linux/kernel.h .

> +	if ((node->priority > 0) && node->num_deps) {
> +		i915_scheduler_priority_bump_clear(scheduler);
> +
> +		for (i = 0; i < node->num_deps; i++)
> +			i915_scheduler_priority_bump(scheduler,
> +					node->dep_list[i], node->priority);
> +	}
> +
> +	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
> +
> +	not_flying = i915_scheduler_count_flying(scheduler, ring) <
> +						 scheduler->min_flying;
> +
> +	spin_unlock_irqrestore(&scheduler->lock, flags);
> +
> +	if (not_flying)
> +		i915_scheduler_submit(ring, true);
> +
> +	return 0;
> +}
> +
> +static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
> +{
> +	struct drm_i915_private *dev_priv = node->params.dev->dev_private;
> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
> +	struct intel_engine_cs  *ring;
> +
> +	WARN_ON(!scheduler);
> +	WARN_ON(!node);
> +	WARN_ON(node->status != i915_sqs_popped);

Other states had their I915_SQS_IS_* macro, why some don't?

> +
> +	ring = node->params.ring;
> +
> +	/*
> +	 * Add the node (which should currently be in state popped) to the
> +	 * front of the queue. This ensure that flying nodes are always held
> +	 * in hardware submission order.
> +	 */
> +	list_add(&node->link, &scheduler->node_queue[ring->id]);
> +
> +	node->status = i915_sqs_flying;
> +
> +	if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
> +		bool    success = true;
> +
> +		success = ring->irq_get(ring);
> +		if (success)
> +			scheduler->flags[ring->id] |= i915_sf_interrupts_enabled;
> +		else
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * Nodes are considered valid dependencies if they are queued on any ring or
> + * if they are in flight on a different ring. In flight on the same ring is no
> + * longer interesting for non-premptive nodes as the ring serialises execution.
> + * For pre-empting nodes, all in flight dependencies are valid as they must not
> + * be jumped by the act of pre-empting.
> + *
> + * Anything that is neither queued nor flying is uninteresting.
> + */
> +static inline bool i915_scheduler_is_dependency_valid(
> +			struct i915_scheduler_queue_entry *node, uint32_t idx)
> +{
> +	struct i915_scheduler_queue_entry *dep;
> +
> +	dep = node->dep_list[idx];
> +	if (!dep)
> +		return false;
> +
> +	if (I915_SQS_IS_QUEUED(dep))
> +		return true;
> +
> +	if (I915_SQS_IS_FLYING(dep)) {
> +		if (node->params.ring != dep->params.ring)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +static uint32_t i915_scheduler_count_flying(struct i915_scheduler *scheduler,
> +					    struct intel_engine_cs *ring)
> +{
> +	struct i915_scheduler_queue_entry *node;
> +	uint32_t                          flying = 0;
> +
> +	list_for_each_entry(node, &scheduler->node_queue[ring->id], link)

This loop keeps popping up, it could use a define similar to the ones
in i915_drv.h ;

#define for_each_hpd_pin(__pin) \

> +		if (I915_SQS_IS_FLYING(node))
> +			flying++;
> +
> +	return flying;
> +}
> +
> +/*
> + * Add a popped node back in to the queue. For example, because the ring was
> + * hung when execfinal() was called and thus the ring submission needs to be
> + * retried later.
> + */
> +static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
> +{
> +	WARN_ON(!node);

	if (WARN_ON(!node))
		return;

Or rather no check at all, it's going to crash anyway even in the
calling function if there's NULL, and it's internal function. It's
relevant to check if the userspace.

> +	WARN_ON(!I915_SQS_IS_FLYING(node));
> +
> +	/* Seqno will be reassigned on relaunch */
> +	node->params.request->seqno = 0;
> +	node->status = i915_sqs_queued;
> +}
> +
> +/*
> + * Give up on a node completely. For example, because it is causing the
> + * ring to hang or is using some resource that no longer exists.
> + */
> +static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
> +{
> +	WARN_ON(!node);
> +	WARN_ON(I915_SQS_IS_COMPLETE(node));
> +
> +	node->status = i915_sqs_dead;
> +}
> +
> +/*
> + * A sequence number has popped out of the hardware and the request handling
> + * code has mapped it back to a request and will mark that request complete.
> + * It also calls this function to notify the scheduler about the completion
> + * so the scheduler's node can be updated appropriately.
> + * Returns true if the request is scheduler managed, false if not.
> + */
> +bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
> +{
> +	struct drm_i915_private *dev_priv  = to_i915(req->ring->dev);
> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
> +	/* XXX: Need to map back from request to node */
> +	struct i915_scheduler_queue_entry *node = NULL;
> +	unsigned long       flags;
> +
> +	if (!node)
> +		return false;

Not so sure if slicing the series down to an extent that functions are
impossible to review, was a good idea. Idea is to slice things down, to
reviewable pieces. It's hard to predict or keep looking forward the
series what is going to come.

> +
> +	spin_lock_irqsave(&scheduler->lock, flags);
> +
> +	WARN_ON(!I915_SQS_IS_FLYING(node));
> +
> +	/* Node was in flight so mark it as complete. */
> +	if (req->cancelled)
> +		node->status = i915_sqs_dead;
> +	else
> +		node->status = i915_sqs_complete;
> +
> +	spin_unlock_irqrestore(&scheduler->lock, flags);
> +
> +	/*
> +	 * XXX: If the in-flight list is now empty then new work should be
> +	 * submitted. However, this function is called from interrupt context
> +	 * and thus cannot acquire mutex locks and other such things that are
> +	 * necessary for fresh submission.
> +	 */
> +
> +	return true;
> +}
> +
> +int i915_scheduler_remove(struct intel_engine_cs *ring)
> +{
> +	struct drm_i915_private *dev_priv = ring->dev->dev_private;
> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
> +	struct i915_scheduler_queue_entry  *node, *node_next;
> +	unsigned long       flags;
> +	int                 flying = 0, queued = 0;
> +	int                 ret = 0;
> +	bool                do_submit;
> +	uint32_t            min_seqno;
> +	struct list_head    remove;
> +
> +	if (list_empty(&scheduler->node_queue[ring->id]))
> +		return 0;
> +
> +	spin_lock_irqsave(&scheduler->lock, flags);
> +
> +	/* /i915_scheduler_dump_locked(ring, "remove/pre");/ */
> +

This should not be here at all.

> +	/*
> +	 * In the case where the system is idle, starting 'min_seqno' from a big
> +	 * number will cause all nodes to be removed as they are now back to
> +	 * being in-order. However, this will be a problem if the last one to
> +	 * complete was actually out-of-order as the ring seqno value will be
> +	 * lower than one or more completed buffers. Thus code looking for the
> +	 * completion of said buffers will wait forever.
> +	 * Instead, use the hardware seqno as the starting point. This means
> +	 * that some buffers might be kept around even in a completely idle
> +	 * system but it should guarantee that no-one ever gets confused when
> +	 * waiting for buffer completion.
> +	 */
> +	min_seqno = ring->get_seqno(ring, true);
> +
> +	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
> +		if (I915_SQS_IS_QUEUED(node))
> +			queued++;
> +		else if (I915_SQS_IS_FLYING(node))
> +			flying++;
> +		else if (I915_SQS_IS_COMPLETE(node))
> +			continue;
> +
> +		if (node->params.request->seqno == 0)
> +			continue;
> +
> +		if (!i915_seqno_passed(node->params.request->seqno, min_seqno))
> +			min_seqno = node->params.request->seqno;
> +	}

Couldn't these values be kept cached, instead of counting them at each
function?

> +
> +	INIT_LIST_HEAD(&remove);
> +	list_for_each_entry_safe(node, node_next, &scheduler->node_queue[ring->id], link) {
> +		/*
> +		 * Only remove completed nodes which have a lower seqno than
> +		 * all pending nodes. While there is the possibility of the
> +		 * ring's seqno counting backwards, all higher buffers must
> +		 * be remembered so that the 'i915_seqno_passed()' test can
> +		 * report that they have in fact passed.
> +		 *
> +		 * NB: This is not true for 'dead' nodes. The GPU reset causes
> +		 * the software seqno to restart from its initial value. Thus
> +		 * the dead nodes must be removed even though their seqno values
> +		 * are potentially vastly greater than the current ring seqno.
> +		 */
> +		if (!I915_SQS_IS_COMPLETE(node))
> +			continue;
> +
> +		if (node->status != i915_sqs_dead) {
> +			if (i915_seqno_passed(node->params.request->seqno, min_seqno) &&
> +			    (node->params.request->seqno != min_seqno))
> +				continue;
> +		}
> +
> +		list_del(&node->link);
> +		list_add(&node->link, &remove);
> +
> +		/* Strip the dependency info while the mutex is still locked */
> +		i915_scheduler_remove_dependent(scheduler, node);
> +
> +		continue;
> +	}
> +
> +	/*
> +	 * Release the interrupt reference count if there are no longer any
> +	 * nodes to worry about.
> +	 */
> +	if (!flying && !queued &&
> +	    (scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
> +		ring->irq_put(ring);
> +		scheduler->flags[ring->id] &= ~i915_sf_interrupts_enabled;
> +	}
> +
> +	/* Launch more packets now? */
> +	do_submit = (queued > 0) && (flying < scheduler->min_flying);
> +
> +	spin_unlock_irqrestore(&scheduler->lock, flags);
> +
> +	if (!do_submit && list_empty(&remove))
> +		return ret;
> +
> +	mutex_lock(&ring->dev->struct_mutex);
> +
> +	if (do_submit)
> +		ret = i915_scheduler_submit(ring, true);

Confusing to have this at a remove function. Function naming needs to
be reconsidered or moved out from here.

> +
> +	while (!list_empty(&remove)) {
> +		node = list_first_entry(&remove, typeof(*node), link);
> +		list_del(&node->link);
> +
> +		/*
> +		 * The batch buffer must be unpinned before it is unreferenced
> +		 * otherwise the unpin fails with a missing vma!?
> +		 */priority_bump_clear
> +		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
> +			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
> +
> +		/* Free everything that is owned by the node: */
> +		i915_gem_request_unreference(node->params.request);
> +		kfree(node->params.cliprects);
> +		kfree(node->dep_list);
> +		kfree(node);
> +	}
> +
> +	mutex_unlock(&ring->dev->struct_mutex);
> +
> +	return ret;
> +}
> +
> +static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
> +{
> +	struct i915_scheduler_queue_entry *node;
> +	int i;
> +
> +	/*
> +	 * Ensure circular dependencies don't cause problems and that a bump
> +	 * by object usage only bumps each using buffer once:
> +	 */
> +	for (i = 0; i < I915_NUM_RINGS; i++) {
> +		list_for_each_entry(node, &scheduler->node_queue[i], link)
> +			node->bumped = false;
> +	}
> +}
> +
> +static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
> +				struct i915_scheduler_queue_entry *target,
> +				uint32_t bump)
> +{
> +	uint32_t new_priority;
> +	int      i, count;
> +
> +	if (target->priority >= scheduler->priority_level_max)
> +		return 1;
> +
> +	if (target->bumped)
> +		return 0;
> +
> +	new_priority = target->priority + bump;
> +	if ((new_priority <= target->priority) ||
> +	    (new_priority > scheduler->priority_level_max))
> +		target->priority = scheduler->priority_level_max;
> +	else
> +		target->priority = new_priority;
> +
> +	count = 1;
> +	target->bumped = true;
> +
> +	for (i = 0; i < target->num_deps; i++) {
> +		if (!target->dep_list[i])
> +			continue;
> +
> +		if (target->dep_list[i]->bumped)
> +			continue;
> +
> +		count += i915_scheduler_priority_bump(scheduler,
> +						      target->dep_list[i],
> +						      bump);
> +	}
> +
> +	return count;
> +}
> +
> +static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
> +				struct i915_scheduler_queue_entry **pop_node,
> +				unsigned long *flags)
> +{
> +	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
> +	struct i915_scheduler              *scheduler = dev_priv->scheduler;
> +	struct i915_scheduler_queue_entry  *best;
> +	struct i915_scheduler_queue_entry  *node;
> +	int     ret;
> +	int     i;
> +	bool	any_queued;
> +	bool	has_local, has_remote, only_remote;
> +
> +	*pop_node = NULL;
> +	ret = -ENODATA;
> +
> +	any_queued = false;
> +	only_remote = false;
> +	best = NULL;

These should just be initialized in-place. But looking at the code
forward. 

> +
> +	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
> +		if (!I915_SQS_IS_QUEUED(node))
> +			continue;
> +		any_queued = true;
> +
> +		has_local  = false;
> +		has_remote = false;
> +		for (i = 0; i < node->num_deps; i++) {
> +			if (!i915_scheduler_is_dependency_valid(node, i))
> +				continue;
> +
> +			if (node->dep_list[i]->params.ring == node->params.ring)
> +				has_local = true;
> +			else
> +				has_remote = true;
> +		}
> +
> +		if (has_remote && !has_local)
> +			only_remote = true;
> +
> +		if (!has_local && !has_remote) {
> +			if (!best ||
> +			    (node->priority > best->priority))
> +				best = node;
> +		}
> +	}
> +
> +	if (best) {
> +		list_del(&best->link);
> +
> +		INIT_LIST_HEAD(&best->link);
> +		best->status  = i915_sqs_popped;
> +
> +		ret = 0;
> +	} else {
> +		/* Can only get here if:
> +		 * (a) there are no buffers in the queue
> +		 * (b) all queued buffers are dependent on other buffers
> +		 *     e.g. on a buffer that is in flight on a different ring
> +		 */
> +		if (only_remote) {
> +			/* The only dependent buffers are on another ring. */
> +			ret = -EAGAIN;
> +		} else if (any_queued) {
> +			/* It seems that something has gone horribly wrong! */
> +			DRM_ERROR("Broken dependency tracking on ring %d!\n",
> +				  (int) ring->id);
> +		}
> +	}
> +
> +	/* i915_scheduler_dump_queue_pop(ring, best); */
> +
> +	*pop_node = best;
> +	return ret;
> +}
> +
> +static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
> +{
> +	struct drm_device   *dev = ring->dev;
> +	struct drm_i915_private *dev_priv = dev->dev_private;
> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
> +	struct i915_scheduler_queue_entry  *node;
> +	unsigned long       flags;
> +	int                 ret = 0, count = 0;
> +
> +	if (!was_locked) {
> +		ret = i915_mutex_lock_interruptible(dev);
> +		if (ret)
> +			return ret;
> +	}
> +

I don't really fancy this construct. Should be moved outside of this
function for proper lockdep tracking.

> +	WARN_ON(!mutex_is_locked(&dev->struct_mutex));
> +
> +	spin_lock_irqsave(&scheduler->lock, flags);
> +
> +	/* First time around, complain if anything unexpected occurs: */
> +	ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
> +	if (ret) {
> +		spin_unlock_irqrestore(&scheduler->lock, flags);
> +
> +		if (!was_locked)
> +			mutex_unlock(&dev->struct_mutex);
> +
> +		return ret;
> +	}
> +

Dropping the was_locked stuff, this should become a proper goto error
label. e.g. out_unlock

> +	do {
> +		WARN_ON(!node);
> +		WARN_ON(node->params.ring != ring);
> +		WARN_ON(node->status != i915_sqs_popped);
> +		count++;
> +
> +		/*
> +		 * The call to pop above will have removed the node from the
> +		 * list. So add it back in and mark it as in flight.
> +		 */
> +		i915_scheduler_fly_node(node);

Why do we want to pull an object out of the list inside spin lock and
push it back immediately in our critical code path? Seems like a waste
for no obvious gain at this point. Why do not we rather just select an
entry and modify it in-place, if it's going to stay in the same queue
anyway.

> +
> +		scheduler->flags[ring->id] |= i915_sf_submitting;
> +		spin_unlock_irqrestore(&scheduler->lock, flags);
> +		ret = dev_priv->gt.execbuf_final(&node->params);
> +		spin_lock_irqsave(&scheduler->lock, flags);
> +		scheduler->flags[ring->id] &= ~i915_sf_submitting;
> +
> +		if (ret) {
> +			int requeue = 1;

Multipurpose variable, not really a good idea. And as commented
further, should not exist at all.

> +
> +			/*
> +			 * Oh dear! Either the node is broken or the ring is
> +			 * busy. So need to kill the node or requeue it and try
> +			 * again later as appropriate.
> +			 */
> +
> +			switch (-ret) {
> +			case ENODEV:
> +			case ENOENT:
> +				/* Fatal errors. Kill the node. */
> +				requeue = -1;
> +			break;

"break" indent is wrong.

> +
> +			case EAGAIN:
> +			case EBUSY:
> +			case EIO:
> +			case ENOMEM:
> +			case ERESTARTSYS:
> +			case EINTR:
> +				/* Supposedly recoverable errors. */
> +			break;
> +
> +			default:
> +				/*
> +				 * Assume the error is recoverable and hope
> +				 * for the best.
> +				 */
> +				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
> +						 ring->name, ret);

There's MISSING_CASE macro, should use it.

> +			break;
> +			}
> +

Just move the code below this point to the switch, no point having a
switch to categorize your options and then doing bunch of ifs to
execute code that could be in switch.

> +			/*
> +			 * Check that the watchdog/reset code has not nuked
> +			 * the node while we weren't looking:
> +			 */
> +			if (node->status == i915_sqs_dead)
> +				requeue = 0;
> +
> +			if (requeue == 1) {
> +				i915_scheduler_node_requeue(node);
> +				/*
> +				 * No point spinning if the ring is currently
> +				 * unavailable so just give up and come back
> +				 * later.
> +				 */
> +				break;
> +			} else if (requeue == -1)
> +				i915_scheduler_node_kill(node);
> +		}
> +

Ending here, this actual submission of a single node could go to its
own helper function, these functions now become too long to follow,
although they really are not doing anything complicated.

> +		/* Keep launching until the sky is sufficiently full. */
> +		if (i915_scheduler_count_flying(scheduler, ring) >=
> +						scheduler->min_flying)
> +			break;
> +
> +		ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
> +	} while (ret == 0);
> +
> +	spin_unlock_irqrestore(&scheduler->lock, flags);
> +
> +	if (!was_locked)
> +		mutex_unlock(&dev->struct_mutex);
> +
> +	/* Don't complain about not being able to submit extra entries */
> +	if (ret == -ENODATA)
> +		ret = 0;
> +
> +	return (ret < 0) ? ret : count;

This is a combined error and success path, keeping to the convention of
kernel drivers is preferred;

	if (ret != -ENODATA)
		goto out_foo;

	return count;

out_unlock:
	spin_unlock(...);
out_foo:
	return ret;



> +}
> +
> +static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
> +				struct i915_scheduler_queue_entry *remove)
> +{
> +	struct i915_scheduler_queue_entry  *node;
> +	int     i, r;
> +	int     count = 0;
> +
> +	/*
> +	 * Ensure that a node is not being removed which is still dependent
> +	 * upon other (not completed) work. If that happens, it implies
> +	 * something has gone very wrong with the dependency tracking! Note
> +	 * that there is no need to worry if this node has been explicitly
> +	 * killed for some reason - it might be being killed before it got
> +	 * sent to the hardware.
> +	 */
> +	if (remove->status != i915_sqs_dead) {
> +		for (i = 0; i < remove->num_deps; i++)
> +			if ((remove->dep_list[i]) &&
> +			    (!I915_SQS_IS_COMPLETE(remove->dep_list[i])))
> +				count++;
> +		WARN_ON(count);
> +	}
> +
> +	/*
> +	 * Remove this node from the dependency lists of any other node which
> +	 * might be waiting on it.
> +	 */
> +	for (r = 0; r < I915_NUM_RINGS; r++) {
> +		list_for_each_entry(node, &scheduler->node_queue[r], link) {
> +			for (i = 0; i < node->num_deps; i++) {
> +				if (node->dep_list[i] != remove)
> +					continue;
> +
> +				node->dep_list[i] = NULL;
> +			}
> +		}
> +	}
> +
> +	return 0;
> +}
> diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
> new file mode 100644
> index 0000000..00dc7f3
> --- /dev/null
> +++ b/drivers/gpu/drm/i915/i915_scheduler.h
> @@ -0,0 +1,91 @@
> +/*
> + * Copyright (c) 2014 Intel Corporation
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice (including the next
> + * paragraph) shall be included in all copies or substantial portions of the
> + * Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
> + * IN THE SOFTWARE.
> + *
> + */
> +
> +#ifndef _I915_SCHEDULER_H_
> +#define _I915_SCHEDULER_H_
> +
> +enum i915_scheduler_queue_status {
> +	/* Limbo: */
> +	i915_sqs_none = 0,
> +	/* Not yet submitted to hardware: */
> +	i915_sqs_queued,
> +	/* Popped from queue, ready to fly: */
> +	i915_sqs_popped,
> +	/* Sent to hardware for processing: */
> +	i915_sqs_flying,
> +	/* Finished processing on the hardware: */
> +	i915_sqs_complete,
> +	/* Killed by watchdog or catastrophic submission failure: */
> +	i915_sqs_dead,
> +	/* Limit value for use with arrays/loops */
> +	i915_sqs_MAX
> +};

These should be UPPERCASE_FOR_ENUM_VALUES . See i915_drv.h for samples.

> +
> +#define I915_SQS_IS_QUEUED(node)	(((node)->status == i915_sqs_queued))
> +#define I915_SQS_IS_FLYING(node)	(((node)->status == i915_sqs_flying))
> +#define I915_SQS_IS_COMPLETE(node)	(((node)->status == i915_sqs_complete) || \
> +					 ((node)->status == i915_sqs_dead))
> +

Might be slightly confusing that name is IS_COMPLETE and there is
actual COMPLETE value. Rather have the test like IS_DONE and then test
for COMPLETE or DEAD, no confusion. Also, some states have their IS_*
macro and others don't, is there going to be more?

> +struct i915_scheduler_obj_entry {
> +	struct drm_i915_gem_object          *obj;
> +	bool                                read_only;
> +};
> +
> +struct i915_scheduler_queue_entry {
> +	struct i915_execbuffer_params       params;
> +	/* -1023 = lowest priority, 0 = default, 1023 = highest */
> +	int32_t                             priority;
> +	struct i915_scheduler_obj_entry     *saved_objects;
> +	int                                 num_objs;
> +	bool                                bumped;
> +	struct i915_scheduler_queue_entry   **dep_list;
> +	int                                 num_deps;
> +	enum i915_scheduler_queue_status    status;
> +	unsigned long                       stamp;
> +	struct list_head                    link;
> +};
> +
> +struct i915_scheduler {
> +	struct list_head    node_queue[I915_NUM_RINGS];
> +	uint32_t            flags[I915_NUM_RINGS];
> +	spinlock_t          lock;
> +
> +	/* Tuning parameters: */
> +	int32_t             priority_level_min;
> +	int32_t             priority_level_max;
> +	int32_t             priority_level_preempt;
> +	uint32_t            min_flying;
> +};
> +
> +/* Flag bits for i915_scheduler::flags */
> +enum {
> +	i915_sf_interrupts_enabled  = (1 << 0),
> +	i915_sf_submitting          = (1 << 1),

Again, should be uppercase. Also, enums to the beginning of file.

> +};
> +
> +int         i915_scheduler_init(struct drm_device *dev);
> +int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
> +bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
> +
> +#endif  /* _I915_SCHEDULER_H_ */
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 03/38] drm/i915: Prelude to splitting i915_gem_do_execbuffer in two

[Jesse Barnes]

On 01/11/2016 10:42 AM, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> The scheduler decouples the submission of batch buffers to the driver
> with their submission to the hardware. This basically means splitting
> the execbuffer() function in half. This change rearranges some code
> ready for the split to occur.
> 
> For: VIZ-1587
> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>  drivers/gpu/drm/i915/i915_gem_execbuffer.c | 63 ++++++++++++++++++------------
>  drivers/gpu/drm/i915/intel_lrc.c           | 18 ++++++---
>  2 files changed, 51 insertions(+), 30 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> index bfc4c17..0eca2b6 100644
> --- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> +++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> @@ -933,10 +933,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
>  	if (flush_domains & I915_GEM_DOMAIN_GTT)
>  		wmb();
>  
> -	/* Unconditionally invalidate gpu caches and ensure that we do flush
> -	 * any residual writes from the previous batch.
> -	 */
> -	return intel_ring_invalidate_all_caches(req);
> +	return 0;
>  }
>  
>  static bool
> @@ -1189,17 +1186,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  	u32 instp_mask;
>  	int ret;
>  
> -	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
> -	if (ret)
> -		return ret;
> -
> -	ret = i915_switch_context(params->request);
> -	if (ret)
> -		return ret;
> -
> -	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<ring->id),
> -	     "%s didn't clear reload\n", ring->name);
> -
>  	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
>  	instp_mask = I915_EXEC_CONSTANTS_MASK;
>  	switch (instp_mode) {
> @@ -1233,11 +1219,37 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  		return -EINVAL;
>  	}
>  
> +	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
> +	if (ret)
> +		return ret;
> +
> +	i915_gem_execbuffer_move_to_active(vmas, params->request);
> +
> +	/* To be split into two functions here... */
> +
> +	intel_runtime_pm_get(dev_priv);
> +
> +	/*
> +	 * Unconditionally invalidate gpu caches and ensure that we do flush
> +	 * any residual writes from the previous batch.
> +	 */
> +	ret = intel_ring_invalidate_all_caches(params->request);
> +	if (ret)
> +		goto error;
> +
> +	/* Switch to the correct context for the batch */
> +	ret = i915_switch_context(params->request);
> +	if (ret)
> +		goto error;
> +
> +	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<ring->id),
> +	     "%s didn't clear reload\n", ring->name);
> +
>  	if (ring == &dev_priv->ring[RCS] &&
>  	    instp_mode != dev_priv->relative_constants_mode) {
>  		ret = intel_ring_begin(params->request, 4);
>  		if (ret)
> -			return ret;
> +			goto error;
>  
>  		intel_ring_emit(ring, MI_NOOP);
>  		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
> @@ -1251,7 +1263,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {
>  		ret = i915_reset_gen7_sol_offsets(dev, params->request);
>  		if (ret)
> -			return ret;
> +			goto error;
>  	}
>  
>  	exec_len   = args->batch_len;
> @@ -1262,14 +1274,20 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  					exec_start, exec_len,
>  					params->dispatch_flags);
>  	if (ret)
> -		return ret;
> +		goto error;
>  
>  	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
>  
> -	i915_gem_execbuffer_move_to_active(vmas, params->request);
>  	i915_gem_execbuffer_retire_commands(params);
>  
> -	return 0;
> +error:
> +	/*
> +	 * intel_gpu_busy should also get a ref, so it will free when the device
> +	 * is really idle.
> +	 */
> +	intel_runtime_pm_put(dev_priv);
> +
> +	return ret;
>  }
>  
>  /**
> @@ -1424,8 +1442,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
>  		dispatch_flags |= I915_DISPATCH_RS;
>  	}
>  
> -	intel_runtime_pm_get(dev_priv);
> -
>  	ret = i915_mutex_lock_interruptible(dev);
>  	if (ret)
>  		goto pre_mutex_err;
> @@ -1599,9 +1615,6 @@ err:
>  	mutex_unlock(&dev->struct_mutex);
>  
>  pre_mutex_err:
> -	/* intel_gpu_busy should also get a ref, so it will free when the device
> -	 * is really idle. */
> -	intel_runtime_pm_put(dev_priv);
>  	return ret;
>  }
>  
> diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
> index e510730..4bf0ee6 100644
> --- a/drivers/gpu/drm/i915/intel_lrc.c
> +++ b/drivers/gpu/drm/i915/intel_lrc.c
> @@ -647,10 +647,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,
>  	if (flush_domains & I915_GEM_DOMAIN_GTT)
>  		wmb();
>  
> -	/* Unconditionally invalidate gpu caches and ensure that we do flush
> -	 * any residual writes from the previous batch.
> -	 */
> -	return logical_ring_invalidate_all_caches(req);
> +	return 0;
>  }
>  
>  int intel_logical_ring_alloc_request_extras(struct drm_i915_gem_request *request)
> @@ -913,6 +910,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>  	if (ret)
>  		return ret;
>  
> +	i915_gem_execbuffer_move_to_active(vmas, params->request);
> +
> +	/* To be split into two functions here... */
> +
> +	/*
> +	 * Unconditionally invalidate gpu caches and ensure that we do flush
> +	 * any residual writes from the previous batch.
> +	 */
> +	ret = logical_ring_invalidate_all_caches(params->request);
> +	if (ret)
> +		return ret;
> +
>  	if (ring == &dev_priv->ring[RCS] &&
>  	    instp_mode != dev_priv->relative_constants_mode) {
>  		ret = intel_logical_ring_begin(params->request, 4);
> @@ -937,7 +946,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>  
>  	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
>  
> -	i915_gem_execbuffer_move_to_active(vmas, params->request);
>  	i915_gem_execbuffer_retire_commands(params);
>  
>  	return 0;
> 

Do we need to do anything if the cache invalidation fails like move the buffers back off the active list?  The order changed here, so I'm wondering.

If that's not a problem, then:
Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 04/38] drm/i915: Split i915_dem_do_execbuffer() in half

[Jesse Barnes]

On 01/11/2016 10:42 AM, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> Split the execbuffer() function in half. The first half collects and
> validates all the information required to process the batch buffer. It
> also does all the object pinning, relocations, active list management,
> etc - basically anything that must be done upfront before the IOCTL
> returns and allows the user land side to start changing/freeing
> things. The second half does the actual ring submission.
> 
> This change implements the split but leaves the back half being called
> directly from the end of the front half.
> 
> v2: Updated due to changes in underlying tree - addition of sync fence
> support and removal of cliprects.
> 
> v3: Moved local 'ringbuf' variable to make later patches in the
> series a bit neater.
> 
> v4: Corrected a typo in the commit message and downgraded a BUG_ON to
> a WARN_ON as the latter is preferred. Also removed all the external
> sync/fence support as that will now be a separate patch series.
> 
> For: VIZ-1587
> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>  drivers/gpu/drm/i915/i915_drv.h            |  11 ++++
>  drivers/gpu/drm/i915/i915_gem.c            |   2 +
>  drivers/gpu/drm/i915/i915_gem_execbuffer.c | 101 ++++++++++++++++++++++-------
>  drivers/gpu/drm/i915/intel_lrc.c           |  57 +++++++++++-----
>  drivers/gpu/drm/i915/intel_lrc.h           |   1 +
>  5 files changed, 130 insertions(+), 42 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
> index acfe25f..2520459 100644
> --- a/drivers/gpu/drm/i915/i915_drv.h
> +++ b/drivers/gpu/drm/i915/i915_drv.h
> @@ -1679,10 +1679,18 @@ struct i915_execbuffer_params {
>  	struct drm_device               *dev;
>  	struct drm_file                 *file;
>  	uint32_t                        dispatch_flags;
> +	uint32_t                        args_flags;
>  	uint32_t                        args_batch_start_offset;
> +	uint32_t                        args_batch_len;
> +	uint32_t                        args_num_cliprects;
> +	uint32_t                        args_DR1;
> +	uint32_t                        args_DR4;
>  	uint64_t                        batch_obj_vm_offset;
>  	struct intel_engine_cs          *ring;
>  	struct drm_i915_gem_object      *batch_obj;
> +	struct drm_clip_rect            *cliprects;
> +	uint32_t                        instp_mask;
> +	int                             instp_mode;
>  	struct intel_context            *ctx;
>  	struct drm_i915_gem_request     *request;
>  };
> @@ -1944,6 +1952,7 @@ struct drm_i915_private {
>  		int (*execbuf_submit)(struct i915_execbuffer_params *params,
>  				      struct drm_i915_gem_execbuffer2 *args,
>  				      struct list_head *vmas);
> +		int (*execbuf_final)(struct i915_execbuffer_params *params);
>  		int (*init_rings)(struct drm_device *dev);
>  		void (*cleanup_ring)(struct intel_engine_cs *ring);
>  		void (*stop_ring)(struct intel_engine_cs *ring);
> @@ -2790,9 +2799,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,
>  void i915_gem_execbuffer_move_to_active(struct list_head *vmas,
>  					struct drm_i915_gem_request *req);
>  void i915_gem_execbuffer_retire_commands(struct i915_execbuffer_params *params);
> +void i915_gem_execbuff_release_batch_obj(struct drm_i915_gem_object *batch_obj);
>  int i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  				   struct drm_i915_gem_execbuffer2 *args,
>  				   struct list_head *vmas);
> +int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params);
>  int i915_gem_execbuffer(struct drm_device *dev, void *data,
>  			struct drm_file *file_priv);
>  int i915_gem_execbuffer2(struct drm_device *dev, void *data,
> diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
> index e8ec49e..5bf7da6 100644
> --- a/drivers/gpu/drm/i915/i915_gem.c
> +++ b/drivers/gpu/drm/i915/i915_gem.c
> @@ -5220,11 +5220,13 @@ int i915_gem_init(struct drm_device *dev)
>  
>  	if (!i915.enable_execlists) {
>  		dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission;
> +		dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final;
>  		dev_priv->gt.init_rings = i915_gem_init_rings;
>  		dev_priv->gt.cleanup_ring = intel_cleanup_ring_buffer;
>  		dev_priv->gt.stop_ring = intel_stop_ring_buffer;
>  	} else {
>  		dev_priv->gt.execbuf_submit = intel_execlists_submission;
> +		dev_priv->gt.execbuf_final = intel_execlists_submission_final;
>  		dev_priv->gt.init_rings = intel_logical_rings_init;
>  		dev_priv->gt.cleanup_ring = intel_logical_ring_cleanup;
>  		dev_priv->gt.stop_ring = intel_logical_ring_stop;
> diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> index 0eca2b6..eff171d 100644
> --- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> +++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> @@ -1181,41 +1181,38 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  	struct drm_device *dev = params->dev;
>  	struct intel_engine_cs *ring = params->ring;
>  	struct drm_i915_private *dev_priv = dev->dev_private;
> -	u64 exec_start, exec_len;
> -	int instp_mode;
> -	u32 instp_mask;
>  	int ret;
>  
> -	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
> -	instp_mask = I915_EXEC_CONSTANTS_MASK;
> -	switch (instp_mode) {
> +	params->instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
> +	params->instp_mask = I915_EXEC_CONSTANTS_MASK;
> +	switch (params->instp_mode) {
>  	case I915_EXEC_CONSTANTS_REL_GENERAL:
>  	case I915_EXEC_CONSTANTS_ABSOLUTE:
>  	case I915_EXEC_CONSTANTS_REL_SURFACE:
> -		if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
> +		if (params->instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
>  			DRM_DEBUG("non-0 rel constants mode on non-RCS\n");
>  			return -EINVAL;
>  		}
>  
> -		if (instp_mode != dev_priv->relative_constants_mode) {
> +		if (params->instp_mode != dev_priv->relative_constants_mode) {
>  			if (INTEL_INFO(dev)->gen < 4) {
>  				DRM_DEBUG("no rel constants on pre-gen4\n");
>  				return -EINVAL;
>  			}
>  
>  			if (INTEL_INFO(dev)->gen > 5 &&
> -			    instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
> +			    params->instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
>  				DRM_DEBUG("rel surface constants mode invalid on gen5+\n");
>  				return -EINVAL;
>  			}
>  
>  			/* The HW changed the meaning on this bit on gen6 */
>  			if (INTEL_INFO(dev)->gen >= 6)
> -				instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
> +				params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
>  		}
>  		break;
>  	default:
> -		DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode);
> +		DRM_DEBUG("execbuf with unknown constants: %d\n", params->instp_mode);
>  		return -EINVAL;
>  	}
>  
> @@ -1225,7 +1222,33 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  
>  	i915_gem_execbuffer_move_to_active(vmas, params->request);
>  
> -	/* To be split into two functions here... */
> +	ret = dev_priv->gt.execbuf_final(params);
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * Free everything that was stored in the QE structure (until the
> +	 * scheduler arrives and does it instead):
> +	 */
> +	if (params->dispatch_flags & I915_DISPATCH_SECURE)
> +		i915_gem_execbuff_release_batch_obj(params->batch_obj);
> +
> +	return 0;
> +}
> +
> +/*
> + * This is the main function for adding a batch to the ring.
> + * It is called from the scheduler, with the struct_mutex already held.
> + */
> +int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
> +{
> +	struct drm_i915_private *dev_priv = params->dev->dev_private;
> +	struct intel_engine_cs  *ring = params->ring;
> +	u64 exec_start, exec_len;
> +	int ret;
> +
> +	/* The mutex must be acquired before calling this function */
> +	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
>  
>  	intel_runtime_pm_get(dev_priv);
>  
> @@ -1246,7 +1269,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  	     "%s didn't clear reload\n", ring->name);
>  
>  	if (ring == &dev_priv->ring[RCS] &&
> -	    instp_mode != dev_priv->relative_constants_mode) {
> +	    params->instp_mode != dev_priv->relative_constants_mode) {
>  		ret = intel_ring_begin(params->request, 4);
>  		if (ret)
>  			goto error;
> @@ -1254,19 +1277,19 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  		intel_ring_emit(ring, MI_NOOP);
>  		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
>  		intel_ring_emit(ring, INSTPM);
> -		intel_ring_emit(ring, instp_mask << 16 | instp_mode);
> +		intel_ring_emit(ring, params->instp_mask << 16 | params->instp_mode);
>  		intel_ring_advance(ring);
>  
> -		dev_priv->relative_constants_mode = instp_mode;
> +		dev_priv->relative_constants_mode = params->instp_mode;
>  	}
>  
> -	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {
> -		ret = i915_reset_gen7_sol_offsets(dev, params->request);
> +	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
> +		ret = i915_reset_gen7_sol_offsets(params->dev, params->request);
>  		if (ret)
>  			goto error;
>  	}
>  
> -	exec_len   = args->batch_len;
> +	exec_len   = params->args_batch_len;
>  	exec_start = params->batch_obj_vm_offset +
>  		     params->args_batch_start_offset;
>  
> @@ -1584,23 +1607,40 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
>  	params->file                    = file;
>  	params->ring                    = ring;
>  	params->dispatch_flags          = dispatch_flags;
> +	params->args_flags              = args->flags;
> +	params->args_batch_len          = args->batch_len;
> +	params->args_num_cliprects      = args->num_cliprects;
> +	params->args_DR1                = args->DR1;
> +	params->args_DR4                = args->DR4;
>  	params->batch_obj               = batch_obj;
>  	params->ctx                     = ctx;
>  
>  	ret = dev_priv->gt.execbuf_submit(params, args, &eb->vmas);
> +	if (ret)
> +		goto err_batch_unpin;
> +
> +	/* the request owns the ref now */
> +	i915_gem_context_unreference(ctx);
>  
> -err_batch_unpin:
>  	/*
> -	 * FIXME: We crucially rely upon the active tracking for the (ppgtt)
> -	 * batch vma for correctness. For less ugly and less fragility this
> -	 * needs to be adjusted to also track the ggtt batch vma properly as
> -	 * active.
> +	 * The eb list is no longer required. The scheduler has extracted all
> +	 * the information than needs to persist.
>  	 */
> +	eb_destroy(eb);
> +
> +	/*
> +	 * Don't clean up everything that is now saved away in the queue.
> +	 * Just unlock and return immediately.
> +	 */
> +	mutex_unlock(&dev->struct_mutex);
> +
> +	return 0;
> +
> +err_batch_unpin:
>  	if (dispatch_flags & I915_DISPATCH_SECURE)
> -		i915_gem_object_ggtt_unpin(batch_obj);
> +		i915_gem_execbuff_release_batch_obj(batch_obj);
>  
>  err:
> -	/* the request owns the ref now */
>  	i915_gem_context_unreference(ctx);
>  	eb_destroy(eb);
>  
> @@ -1618,6 +1658,17 @@ pre_mutex_err:
>  	return ret;
>  }
>  
> +void i915_gem_execbuff_release_batch_obj(struct drm_i915_gem_object *batch_obj)
> +{
> +	/*
> +	 * FIXME: We crucially rely upon the active tracking for the (ppgtt)
> +	 * batch vma for correctness. For less ugly and less fragility this
> +	 * needs to be adjusted to also track the ggtt batch vma properly as
> +	 * active.
> +	 */
> +	i915_gem_object_ggtt_unpin(batch_obj);
> +}
> +
>  /*
>   * Legacy execbuffer just creates an exec2 list from the original exec object
>   * list array and passes it to the real function.
> diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
> index 4bf0ee6..014180c 100644
> --- a/drivers/gpu/drm/i915/intel_lrc.c
> +++ b/drivers/gpu/drm/i915/intel_lrc.c
> @@ -869,35 +869,31 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>  	struct drm_device       *dev = params->dev;
>  	struct intel_engine_cs  *ring = params->ring;
>  	struct drm_i915_private *dev_priv = dev->dev_private;
> -	struct intel_ringbuffer *ringbuf = params->ctx->engine[ring->id].ringbuf;
> -	u64 exec_start;
> -	int instp_mode;
> -	u32 instp_mask;
>  	int ret;
>  
> -	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
> -	instp_mask = I915_EXEC_CONSTANTS_MASK;
> -	switch (instp_mode) {
> +	params->instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
> +	params->instp_mask = I915_EXEC_CONSTANTS_MASK;
> +	switch (params->instp_mode) {
>  	case I915_EXEC_CONSTANTS_REL_GENERAL:
>  	case I915_EXEC_CONSTANTS_ABSOLUTE:
>  	case I915_EXEC_CONSTANTS_REL_SURFACE:
> -		if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
> +		if (params->instp_mode != 0 && ring != &dev_priv->ring[RCS]) {
>  			DRM_DEBUG("non-0 rel constants mode on non-RCS\n");
>  			return -EINVAL;
>  		}
>  
> -		if (instp_mode != dev_priv->relative_constants_mode) {
> -			if (instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
> +		if (params->instp_mode != dev_priv->relative_constants_mode) {
> +			if (params->instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) {
>  				DRM_DEBUG("rel surface constants mode invalid on gen5+\n");
>  				return -EINVAL;
>  			}
>  
>  			/* The HW changed the meaning on this bit on gen6 */
> -			instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
> +			params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE;
>  		}
>  		break;
>  	default:
> -		DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode);
> +		DRM_DEBUG("execbuf with unknown constants: %d\n", params->instp_mode);
>  		return -EINVAL;
>  	}
>  
> @@ -912,7 +908,34 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>  
>  	i915_gem_execbuffer_move_to_active(vmas, params->request);
>  
> -	/* To be split into two functions here... */
> +	ret = dev_priv->gt.execbuf_final(params);
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * Free everything that was stored in the QE structure (until the
> +	 * scheduler arrives and does it instead):
> +	 */
> +	if (params->dispatch_flags & I915_DISPATCH_SECURE)
> +		i915_gem_execbuff_release_batch_obj(params->batch_obj);
> +
> +	return 0;
> +}
> +
> +/*
> + * This is the main function for adding a batch to the ring.
> + * It is called from the scheduler, with the struct_mutex already held.
> + */
> +int intel_execlists_submission_final(struct i915_execbuffer_params *params)
> +{
> +	struct drm_i915_private *dev_priv = params->dev->dev_private;
> +	struct intel_ringbuffer *ringbuf = params->request->ringbuf;
> +	struct intel_engine_cs *ring = params->ring;
> +	u64 exec_start;
> +	int ret;
> +
> +	/* The mutex must be acquired before calling this function */
> +	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
>  
>  	/*
>  	 * Unconditionally invalidate gpu caches and ensure that we do flush
> @@ -923,7 +946,7 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>  		return ret;
>  
>  	if (ring == &dev_priv->ring[RCS] &&
> -	    instp_mode != dev_priv->relative_constants_mode) {
> +	    params->instp_mode != dev_priv->relative_constants_mode) {
>  		ret = intel_logical_ring_begin(params->request, 4);
>  		if (ret)
>  			return ret;
> @@ -931,14 +954,14 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>  		intel_logical_ring_emit(ringbuf, MI_NOOP);
>  		intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1));
>  		intel_logical_ring_emit(ringbuf, INSTPM);
> -		intel_logical_ring_emit(ringbuf, instp_mask << 16 | instp_mode);
> +		intel_logical_ring_emit(ringbuf, params->instp_mask << 16 | params->instp_mode);
>  		intel_logical_ring_advance(ringbuf);
>  
> -		dev_priv->relative_constants_mode = instp_mode;
> +		dev_priv->relative_constants_mode = params->instp_mode;
>  	}
>  
>  	exec_start = params->batch_obj_vm_offset +
> -		     args->batch_start_offset;
> +		     params->args_batch_start_offset;
>  
>  	ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags);
>  	if (ret)
> diff --git a/drivers/gpu/drm/i915/intel_lrc.h b/drivers/gpu/drm/i915/intel_lrc.h
> index 4e60d54..8d9bad7 100644
> --- a/drivers/gpu/drm/i915/intel_lrc.h
> +++ b/drivers/gpu/drm/i915/intel_lrc.h
> @@ -93,6 +93,7 @@ struct i915_execbuffer_params;
>  int intel_execlists_submission(struct i915_execbuffer_params *params,
>  			       struct drm_i915_gem_execbuffer2 *args,
>  			       struct list_head *vmas);
> +int intel_execlists_submission_final(struct i915_execbuffer_params *params);
>  u32 intel_execlists_ctx_id(struct drm_i915_gem_object *ctx_obj);
>  
>  void intel_lrc_irq_handler(struct intel_engine_cs *ring);

Just a nitpick on naming; I think prepare/commit might be better than submit/submit_final.  But you don't have to respin just for that.

Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 05/38] drm/i915: Cache request pointer in *_submission_final()

[Jesse Barnes]

On 01/11/2016 10:42 AM, John.C.Harrison@Intel.com wrote:
> From: Dave Gordon <david.s.gordon@intel.com>
> 
> Keep a local copy of the request pointer in the _final() functions
> rather than dereferencing the params block repeatedly.
> 
> v3: New patch in series.
> 
> For: VIZ-1587
> Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>  drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++------
>  drivers/gpu/drm/i915/intel_lrc.c           | 11 ++++++-----
>  2 files changed, 13 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> index eff171d..0ad32f6 100644
> --- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> +++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> @@ -1243,6 +1243,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>  int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
>  {
>  	struct drm_i915_private *dev_priv = params->dev->dev_private;
> +	struct drm_i915_gem_request *req = params->request;
>  	struct intel_engine_cs  *ring = params->ring;
>  	u64 exec_start, exec_len;
>  	int ret;
> @@ -1256,12 +1257,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
>  	 * Unconditionally invalidate gpu caches and ensure that we do flush
>  	 * any residual writes from the previous batch.
>  	 */
> -	ret = intel_ring_invalidate_all_caches(params->request);
> +	ret = intel_ring_invalidate_all_caches(req);
>  	if (ret)
>  		goto error;
>  
>  	/* Switch to the correct context for the batch */
> -	ret = i915_switch_context(params->request);
> +	ret = i915_switch_context(req);
>  	if (ret)
>  		goto error;
>  
> @@ -1270,7 +1271,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
>  
>  	if (ring == &dev_priv->ring[RCS] &&
>  	    params->instp_mode != dev_priv->relative_constants_mode) {
> -		ret = intel_ring_begin(params->request, 4);
> +		ret = intel_ring_begin(req, 4);
>  		if (ret)
>  			goto error;
>  
> @@ -1284,7 +1285,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
>  	}
>  
>  	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
> -		ret = i915_reset_gen7_sol_offsets(params->dev, params->request);
> +		ret = i915_reset_gen7_sol_offsets(params->dev, req);
>  		if (ret)
>  			goto error;
>  	}
> @@ -1293,13 +1294,13 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
>  	exec_start = params->batch_obj_vm_offset +
>  		     params->args_batch_start_offset;
>  
> -	ret = ring->dispatch_execbuffer(params->request,
> +	ret = ring->dispatch_execbuffer(req,
>  					exec_start, exec_len,
>  					params->dispatch_flags);
>  	if (ret)
>  		goto error;
>  
> -	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
> +	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
>  
>  	i915_gem_execbuffer_retire_commands(params);
>  
> diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
> index 014180c..a344a3a 100644
> --- a/drivers/gpu/drm/i915/intel_lrc.c
> +++ b/drivers/gpu/drm/i915/intel_lrc.c
> @@ -929,7 +929,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>  int intel_execlists_submission_final(struct i915_execbuffer_params *params)
>  {
>  	struct drm_i915_private *dev_priv = params->dev->dev_private;
> -	struct intel_ringbuffer *ringbuf = params->request->ringbuf;
> +	struct drm_i915_gem_request *req = params->request;
> +	struct intel_ringbuffer *ringbuf = req->ringbuf;
>  	struct intel_engine_cs *ring = params->ring;
>  	u64 exec_start;
>  	int ret;
> @@ -941,13 +942,13 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
>  	 * Unconditionally invalidate gpu caches and ensure that we do flush
>  	 * any residual writes from the previous batch.
>  	 */
> -	ret = logical_ring_invalidate_all_caches(params->request);
> +	ret = logical_ring_invalidate_all_caches(req);
>  	if (ret)
>  		return ret;
>  
>  	if (ring == &dev_priv->ring[RCS] &&
>  	    params->instp_mode != dev_priv->relative_constants_mode) {
> -		ret = intel_logical_ring_begin(params->request, 4);
> +		ret = intel_logical_ring_begin(req, 4);
>  		if (ret)
>  			return ret;
>  
> @@ -963,11 +964,11 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
>  	exec_start = params->batch_obj_vm_offset +
>  		     params->args_batch_start_offset;
>  
> -	ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags);
> +	ret = ring->emit_bb_start(req, exec_start, params->dispatch_flags);
>  	if (ret)
>  		return ret;
>  
> -	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
> +	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
>  
>  	i915_gem_execbuffer_retire_commands(params);
>  
> 

Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 08/38] drm/i915: Prepare retire_requests to handle out-of-order seqnos

[Jesse Barnes]

On 01/11/2016 02:10 PM, Chris Wilson wrote:
> On Mon, Jan 11, 2016 at 06:42:37PM +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> A major point of the GPU scheduler is that it re-orders batch buffers
>> after they have been submitted to the driver. This leads to requests
>> completing out of order. In turn, this means that the retire
>> processing can no longer assume that all completed entries are at the
>> front of the list. Rather than attempting to re-order the request list
>> on a regular basis, it is better to simply scan the entire list.
> 
> This is a major misstep. Just think in terms of per-context timelines,
> and retirment order within those timelines being consistent..

I think you're just re-iterating the desire for per-context timelines.  We all want this, but after already implementing several big reworks, I don't think it's fair to make John do this.  When we move that direction after this lands, we can simplify/drop code where possible (sure maybe it's more total churn, but my guess is we'll probably find other things to refactor as well, so it doesn't matter too much).

Jesse

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 03/38] drm/i915: Prelude to splitting i915_gem_do_execbuffer in two

[John Harrison]

On 04/02/2016 17:01, Jesse Barnes wrote:
> On 01/11/2016 10:42 AM, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> The scheduler decouples the submission of batch buffers to the driver
>> with their submission to the hardware. This basically means splitting
>> the execbuffer() function in half. This change rearranges some code
>> ready for the split to occur.
>>
>> For: VIZ-1587
>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>> ---
>>   drivers/gpu/drm/i915/i915_gem_execbuffer.c | 63 ++++++++++++++++++------------
>>   drivers/gpu/drm/i915/intel_lrc.c           | 18 ++++++---
>>   2 files changed, 51 insertions(+), 30 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
>> index bfc4c17..0eca2b6 100644
>> --- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
>> +++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
>> @@ -933,10 +933,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
>>   	if (flush_domains & I915_GEM_DOMAIN_GTT)
>>   		wmb();
>>   
>> -	/* Unconditionally invalidate gpu caches and ensure that we do flush
>> -	 * any residual writes from the previous batch.
>> -	 */
>> -	return intel_ring_invalidate_all_caches(req);
>> +	return 0;
>>   }
>>   
>>   static bool
>> @@ -1189,17 +1186,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>>   	u32 instp_mask;
>>   	int ret;
>>   
>> -	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
>> -	if (ret)
>> -		return ret;
>> -
>> -	ret = i915_switch_context(params->request);
>> -	if (ret)
>> -		return ret;
>> -
>> -	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<ring->id),
>> -	     "%s didn't clear reload\n", ring->name);
>> -
>>   	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
>>   	instp_mask = I915_EXEC_CONSTANTS_MASK;
>>   	switch (instp_mode) {
>> @@ -1233,11 +1219,37 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>>   		return -EINVAL;
>>   	}
>>   
>> +	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
>> +	if (ret)
>> +		return ret;
>> +
>> +	i915_gem_execbuffer_move_to_active(vmas, params->request);
>> +
>> +	/* To be split into two functions here... */
>> +
>> +	intel_runtime_pm_get(dev_priv);
>> +
>> +	/*
>> +	 * Unconditionally invalidate gpu caches and ensure that we do flush
>> +	 * any residual writes from the previous batch.
>> +	 */
>> +	ret = intel_ring_invalidate_all_caches(params->request);
>> +	if (ret)
>> +		goto error;
>> +
>> +	/* Switch to the correct context for the batch */
>> +	ret = i915_switch_context(params->request);
>> +	if (ret)
>> +		goto error;
>> +
>> +	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<ring->id),
>> +	     "%s didn't clear reload\n", ring->name);
>> +
>>   	if (ring == &dev_priv->ring[RCS] &&
>>   	    instp_mode != dev_priv->relative_constants_mode) {
>>   		ret = intel_ring_begin(params->request, 4);
>>   		if (ret)
>> -			return ret;
>> +			goto error;
>>   
>>   		intel_ring_emit(ring, MI_NOOP);
>>   		intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
>> @@ -1251,7 +1263,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>>   	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {
>>   		ret = i915_reset_gen7_sol_offsets(dev, params->request);
>>   		if (ret)
>> -			return ret;
>> +			goto error;
>>   	}
>>   
>>   	exec_len   = args->batch_len;
>> @@ -1262,14 +1274,20 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
>>   					exec_start, exec_len,
>>   					params->dispatch_flags);
>>   	if (ret)
>> -		return ret;
>> +		goto error;
>>   
>>   	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
>>   
>> -	i915_gem_execbuffer_move_to_active(vmas, params->request);
>>   	i915_gem_execbuffer_retire_commands(params);
>>   
>> -	return 0;
>> +error:
>> +	/*
>> +	 * intel_gpu_busy should also get a ref, so it will free when the device
>> +	 * is really idle.
>> +	 */
>> +	intel_runtime_pm_put(dev_priv);
>> +
>> +	return ret;
>>   }
>>   
>>   /**
>> @@ -1424,8 +1442,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
>>   		dispatch_flags |= I915_DISPATCH_RS;
>>   	}
>>   
>> -	intel_runtime_pm_get(dev_priv);
>> -
>>   	ret = i915_mutex_lock_interruptible(dev);
>>   	if (ret)
>>   		goto pre_mutex_err;
>> @@ -1599,9 +1615,6 @@ err:
>>   	mutex_unlock(&dev->struct_mutex);
>>   
>>   pre_mutex_err:
>> -	/* intel_gpu_busy should also get a ref, so it will free when the device
>> -	 * is really idle. */
>> -	intel_runtime_pm_put(dev_priv);
>>   	return ret;
>>   }
>>   
>> diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
>> index e510730..4bf0ee6 100644
>> --- a/drivers/gpu/drm/i915/intel_lrc.c
>> +++ b/drivers/gpu/drm/i915/intel_lrc.c
>> @@ -647,10 +647,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,
>>   	if (flush_domains & I915_GEM_DOMAIN_GTT)
>>   		wmb();
>>   
>> -	/* Unconditionally invalidate gpu caches and ensure that we do flush
>> -	 * any residual writes from the previous batch.
>> -	 */
>> -	return logical_ring_invalidate_all_caches(req);
>> +	return 0;
>>   }
>>   
>>   int intel_logical_ring_alloc_request_extras(struct drm_i915_gem_request *request)
>> @@ -913,6 +910,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>>   	if (ret)
>>   		return ret;
>>   
>> +	i915_gem_execbuffer_move_to_active(vmas, params->request);
>> +
>> +	/* To be split into two functions here... */
>> +
>> +	/*
>> +	 * Unconditionally invalidate gpu caches and ensure that we do flush
>> +	 * any residual writes from the previous batch.
>> +	 */
>> +	ret = logical_ring_invalidate_all_caches(params->request);
>> +	if (ret)
>> +		return ret;
>> +
>>   	if (ring == &dev_priv->ring[RCS] &&
>>   	    instp_mode != dev_priv->relative_constants_mode) {
>>   		ret = intel_logical_ring_begin(params->request, 4);
>> @@ -937,7 +946,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
>>   
>>   	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
>>   
>> -	i915_gem_execbuffer_move_to_active(vmas, params->request);
>>   	i915_gem_execbuffer_retire_commands(params);
>>   
>>   	return 0;
>>
> Do we need to do anything if the cache invalidation fails like move the buffers back off the active list?  The order changed here, so I'm wondering.

I don't believe so. The objects are all tracked by request. The request 
itself is cancelled rather than added to the request list. So the next 
run of i915_gem_retire_requests_ring() will do all the necessary clean 
up. Potentially we could do something explicit here to speed the process 
up. However, this is only an intermediate step and there is no nice and 
simple move_to_inactive function to call. Once the scheduler arrives, a 
failed backend submission would be retried by the scheduler. So objects 
must not be cleaned up as that would break the re-submission.

> If that's not a problem, then:
> Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 07/38] drm/i915: Start of GPU scheduler

[John Harrison]

On 20/01/2016 13:18, Joonas Lahtinen wrote:
> Hi,
>
> Comments below this pre text.
>
> Many of the comments are related to the indent and style of the code.
> That stuff is important to fix for future maintainability. In order for
> the future review to be more effective, I'd like to next see a v5 of
> the series where the code quality concerns have been addressed, patches
> squashed to be actual reviewable chunks and appropriate kerneldoc being
> added.
>
> To give an idea of proper slicing of patches, first produce a no-op
> scheduler, adding the extra function calls where needed and still
> keeping the scheduling completely linear. Second patch could introduce
> out of order submitting, third one priority bumping, fourth pre-empting
> and so on. That way, each patch extends the functionality and is itself
> already mergeable. That way I've been able to go through and understand
> the existing code, and I can actually review (other than just nag about
> indent and coding style) if the changes are appropriate to bring in the
> functionality desired.
>
> In the current split, for me or anyone who did not participate writing
> the code, it is otherwise too confusing to try to guess what future
> changes might make each piece of code make sense, and which will be
> redundant in the future too. There is no value in splitting code to
> chunks that are not itself functional.
>
> Regards, Joonas
>
> On Mon, 2016-01-11 at 18:42 +0000, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> Initial creation of scheduler source files. Note that this patch
>> implements most of the scheduler functionality but does not hook it
>> in
>> to the driver yet. It also leaves the scheduler code in 'pass
>> through'
>> mode so that even when it is hooked in, it will not actually do very
>> much. This allows the hooks to be added one at a time in bite size
>> chunks and only when the scheduler is finally enabled at the end does
>> anything start happening.
>>
>> The general theory of operation is that when batch buffers are
>> submitted to the driver, the execbuffer() code packages up all the
>> information required to execute the batch buffer at a later time.
>> This
>> package is given over to the scheduler which adds it to an internal
>> node list. The scheduler also scans the list of objects associated
>> with the batch buffer and compares them against the objects already
>> in
>> use by other buffers in the node list. If matches are found then the
>> new batch buffer node is marked as being dependent upon the matching
>> node. The same is done for the context object. The scheduler also
>> bumps up the priority of such matching nodes on the grounds that the
>> more dependencies a given batch buffer has the more important it is
>> likely to be.
>>
>> The scheduler aims to have a given (tuneable) number of batch buffers
>> in flight on the hardware at any given time. If fewer than this are
>> currently executing when a new node is queued, then the node is
>> passed
>> straight through to the submit function. Otherwise it is simply added
>> to the queue and the driver returns back to user land.
>>
>> The scheduler is notified when each batch buffer completes and
>> updates
>> its internal tracking accordingly. At the end of the completion
>> interrupt processing, if any scheduler tracked batches were
>> processed,
>> the scheduler's deferred worker thread is woken up. This can do more
>> involved processing such as actually removing completed nodes from
>> the
>> queue and freeing up the resources associated with them (internal
>> memory allocations, DRM object references, context reference, etc.).
>> The work handler also checks the in flight count and calls the
>> submission code if a new slot has appeared.
>>
>> When the scheduler's submit code is called, it scans the queued node
>> list for the highest priority node that has no unmet dependencies.
>> Note that the dependency calculation is complex as it must take
>> inter-ring dependencies and potential preemptions into account. Note
>> also that in the future this will be extended to include external
>> dependencies such as the Android Native Sync file descriptors and/or
>> the linux dma-buff synchronisation scheme.
>>
>> If a suitable node is found then it is sent to execbuff_final() for
>> submission to the hardware. The in flight count is then re-checked
>> and
>> a new node popped from the list if appropriate.
>>
>> Note that this patch does not implement pre-emptive scheduling. Only
>> basic scheduling by re-ordering batch buffer submission is currently
>> implemented. Pre-emption of actively executing batch buffers comes in
>> the next patch series.
>>
>> v2: Changed priority levels to +/-1023 due to feedback from Chris
>> Wilson.
>>
>> Removed redundant index from scheduler node.
>>
>> Changed time stamps to use jiffies instead of raw monotonic. This
>> provides lower resolution but improved compatibility with other i915
>> code.
>>
>> Major re-write of completion tracking code due to struct fence
>> conversion. The scheduler no longer has it's own private IRQ handler
>> but just lets the existing request code handle completion events.
>> Instead, the scheduler now hooks into the request notify code to be
>> told when a request has completed.
>>
>> Reduced driver mutex locking scope. Removal of scheduler nodes no
>> longer grabs the mutex lock.
>>
>> v3: Refactor of dependency generation to make the code more readable.
>> Also added in read-read optimisation support - i.e., don't treat a
>> shared read-only buffer as being a dependency.
>>
>> Allowed the killing of queued nodes rather than only flying ones.
>>
>> v4: Updated the commit message to better reflect the current state of
>> the code. Downgraded some BUG_ONs to WARN_ONs. Used the correct array
>> memory allocator function (kmalloc_array instead of kmalloc).
>> Corrected the format of some comments. Wrapped some lines differently
>> to keep the style checker happy.
>>
>> Fixed a WARN_ON when killing nodes. The dependency removal code
>> checks
>> that nodes being destroyed do not have any oustanding dependencies
>> (which would imply they should not have been executed yet). In the
>> case of nodes being destroyed, e.g. due to context banning, then this
>> might well be the case - they have not been executed and do indeed
>> have outstanding dependencies.
>>
>> Re-instated the code to disble interrupts when not in use. The
>> underlying problem causing broken IRQ reference counts seems to have
>> been fixed now.
>>
>> For: VIZ-1587
>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>> ---
>>   drivers/gpu/drm/i915/Makefile         |   1 +
>>   drivers/gpu/drm/i915/i915_drv.h       |   4 +
>>   drivers/gpu/drm/i915/i915_gem.c       |   5 +
>>   drivers/gpu/drm/i915/i915_scheduler.c | 797
>> ++++++++++++++++++++++++++++++++++
>>   drivers/gpu/drm/i915/i915_scheduler.h |  91 ++++
>>   5 files changed, 898 insertions(+)
>>   create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
>>   create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
>>
>> diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
>> index 15398c5..79cb38b 100644
>> --- a/drivers/gpu/drm/i915/Makefile
>> +++ b/drivers/gpu/drm/i915/Makefile
>> @@ -10,6 +10,7 @@ ccflags-y := -Werror
>>   i915-y := i915_drv.o \
>>   	  i915_irq.o \
>>   	  i915_params.o \
>> +	  i915_scheduler.o \
>>             i915_suspend.o \
>>   	  i915_sysfs.o \
>>   	  intel_csr.o \
>> diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
>> index 10dffdd..38f423b 100644
>> --- a/drivers/gpu/drm/i915/i915_drv.h
>> +++ b/drivers/gpu/drm/i915/i915_drv.h
>> @@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {
>>   	struct drm_i915_gem_request     *request;
>>   };
>>   
>> +struct i915_scheduler;
>> +
> Rather add "i915_scheduler.h" include at the top and eliminate circular
> include dependencies. This is needed for the next comment.
>
>>   /* used in computing the new watermarks state */
>>   struct intel_wm_config {
>>   	unsigned int num_pipes_active;
>> @@ -1947,6 +1949,8 @@ struct drm_i915_private {
>>   
>>   	struct i915_runtime_pm pm;
>>   
>> +	struct i915_scheduler *scheduler;
>> +
> As the scheduler is going to be enabled on all platforms to an extent,
> no point in making it a pointer. Just making it member like "pm" is the
> best, this also requires the above include change.
>
>>   	/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */
>>   	struct {
>>   		int (*execbuf_submit)(struct i915_execbuffer_params *params,
>> diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
>> index cff3768..47aa85b 100644
>> --- a/drivers/gpu/drm/i915/i915_gem.c
>> +++ b/drivers/gpu/drm/i915/i915_gem.c
>> @@ -37,6 +37,7 @@
>>   #include
>>   #include
>>   #include
>> +#include "i915_scheduler.h"
>>   
> This should go before any <linux/...> not to mask missing includes from
> the header itself, so correct place would be right after #include
> "i915_trace.h"
>
>>   #define RQ_BUG_ON(expr)
>>   
>> @@ -5242,6 +5243,10 @@ int i915_gem_init(struct drm_device *dev)
>>   	 */
>>   	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
>>   
>> +	ret = i915_scheduler_init(dev);
>> +	if (ret)
>> +		goto out_unlock;
>> +
>>   	ret = i915_gem_init_userptr(dev);
>>   	if (ret)
>>   		goto out_unlock;
>> diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c
>> new file mode 100644
>> index 0000000..8cb9063
>> --- /dev/null
>> +++ b/drivers/gpu/drm/i915/i915_scheduler.c
>> @@ -0,0 +1,797 @@
>> +/*
>> + * Copyright (c) 2014 Intel Corporation
>> + *
>> + * Permission is hereby granted, free of charge, to any person obtaining a
>> + * copy of this software and associated documentation files (the "Software"),
>> + * to deal in the Software without restriction, including without limitation
>> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
>> + * and/or sell copies of the Software, and to permit persons to whom the
>> + * Software is furnished to do so, subject to the following conditions:
>> + *
>> + * The above copyright notice and this permission notice (including the next
>> + * paragraph) shall be included in all copies or substantial portions of the
>> + * Software.
>> + *
>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
>> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
>> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
>> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
>> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
>> + * IN THE SOFTWARE.
>> + *
>> + */
>> +
>> +#include "i915_drv.h"
>> +#include "intel_drv.h"
>> +#include "i915_scheduler.h"
> Again, this include should be the top one, should not require any
> includes before it.
>
>> +
>> +static int         i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node);
>> +static int         i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
>> +						   struct i915_scheduler_queue_entry *remove);
>> +static int         i915_scheduler_submit(struct intel_engine_cs *ring,
>> +					 bool is_locked);
>> +static uint32_t    i915_scheduler_count_flying(struct i915_scheduler *scheduler,
>> +					       struct intel_engine_cs *ring);
>> +static void        i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler);
>> +static int         i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
>> +						struct i915_scheduler_queue_entry *target,
>> +						uint32_t bump);
> Do not indent the function names like this, it becomes unmaintainable
> and messy very fast if somebody adds a new function with a more complex
> return type, which is a very likely thing to happen.
>
> What I would do is move all the helper functions here at the top and
> order them so that the forward declarations are not needed, at least
> i915_scheduler_fly_node is not used before its definition.
>
>> +
>> +int i915_scheduler_init(struct drm_device *dev)
>> +{
>> +	struct drm_i915_private *dev_priv = dev->dev_private;
>> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
>> +	int                     r;
> Again, this indent is a no-go. I'll not mention it on further
> functions, assume it to be fixed for next revision.
>
>> +
>> +	if (scheduler)
>> +		return 0;
>> +
>> +	scheduler = kzalloc(sizeof(*scheduler), GFP_KERNEL);
>> +	if (!scheduler)
>> +		return -ENOMEM;
>> +
>> +	spin_lock_init(&scheduler->lock);
>> +
>> +	for (r = 0; r < I915_NUM_RINGS; r++)
>> +		INIT_LIST_HEAD(&scheduler->node_queue[r]);
>> +
>> +	/* Default tuning values: */
>> +	scheduler->priority_level_min     = -1023;
>> +	scheduler->priority_level_max     = 1023;
>> +	scheduler->priority_level_preempt = 900;
>> +	scheduler->min_flying             = 2;
> This kind of indent is tolerable because it is a contained code block,
> but not needed either.
>
>> +
>> +	dev_priv->scheduler = scheduler;
>> +
>> +	return 0;
>> +}
>> +
>> +static void i915_generate_dependencies(struct i915_scheduler *scheduler,
>> +				       struct i915_scheduler_queue_entry *node,
>> +				       uint32_t ring)
>> +{
>> +	struct i915_scheduler_obj_entry *this, *that;
>> +	struct i915_scheduler_queue_entry  *test;
>> +	int                 i, j;
>> +	bool                found;
>> +
>> +	list_for_each_entry(test, &scheduler->node_queue[ring], link) {
>> +		if (I915_SQS_IS_COMPLETE(test))
>> +			continue;
>> +
>> +		/*
>> +		 * Batches on the same ring for the same
>> +		 * context must be kept in order.
>> +		 */
>> +		found = (node->params.ctx == test->params.ctx) &&
>> +			(node->params.ring == test->params.ring);
>> +
>> +		if (found)
>> +			goto depends;
>> +
> This is not needed.
Yeah, this code block has been re-worked a number of times with things 
being added and/or removed. I guess it had gotten a bit messy by now.
>
>> +		/*
>> +		 * Batches working on the same objects must
>> +		 * be kept in order.
>> +		 */
>> +		for (i = 0; (i < node->num_objs) && !found; i++) {
> As the test is here already                         ---^
>
>> +			this = node->saved_objects + i;
>> +
>> +			for (j = 0; j < test->num_objs; j++) {
>> +				that = test->saved_objects + j;
>> +
>> +				if (this->obj != that->obj)
>> +					continue;
> How about VMAs? There might be multiple mappings to an object, isn't it
> enough to depend on the required VMA instead of the whole object?
The object is what we get coming in from user land through the IOCTL. So 
why make things more complicated? If there are multiple VMAs referring 
to the same object then we can't just track an individual VMA as that 
would loose the dependency on all the other VMAs. Just because the 
object is mapped to someone else's address space doesn't mean that this 
batch buffer can't overwrite data they are reading.

>
>> +
>> +				/* Only need to worry about writes */
>> +				if (this->read_only && that->read_only)
>> +					continue;
>> +
>> +				found = true;
>> +				break;
>> +			}
>> +		}
>> +
> The following block is not needed.
>
>> +		if (!found)
>> +			continue;
>> +
>> +depends:
> Rather like this, in order to avoid a goto label;
> if (found) {
> ...
>
>
>> +		node->dep_list[node->num_deps] = test;
>> +		node->num_deps++;
>> +	}
>> +}
>> +
> Please add a brief kerneldoc above each function in the header, it's
> required. Adding it to non-trivial inline helper functions too will
> make reviewing much easier.
>
>> +int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
>> +{
>> +	struct drm_i915_private *dev_priv = qe->params.dev->dev_private;
>> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
>> +	struct intel_engine_cs  *ring = qe->params.ring;
>> +	struct i915_scheduler_queue_entry  *node;
>> +	struct i915_scheduler_queue_entry  *test;
>> +	unsigned long       flags;
>> +	bool                not_flying;
>> +	int                 i, r;
>> +	int                 incomplete = 0;
>> +
>> +	WARN_ON(!scheduler);
>> +
> This kind of situations should have a be a BUG_ON, because scheduler
> being zero is literally going to cause an OOPS in the next dereference
> which is going to happen unconditionally. WARN + OOPS is kind of what
> BUG_ON should be used avoid. But this should be removed anyway after
> scheduler is made a data member of dev_priv.

The WARNs were originally BUGs but Daniel Vetter had the opposite 
demand. His view was the driver should never BUG under any 
circumstances. A WARN followed by an oops is better than a BUG because 
maybe it won't actually oops.


>
>> +	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
> I assume this is going to be addressed in a future commit. Could have
> been introduced in this patch, too.
>
>> +		int ret;
>> +
>> +		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
>> +		ret = dev_priv->gt.execbuf_final(&qe->params);
>> +		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
>> +
> The kerneldoc should mention locking requirements of this function.
>
>> +		/*
>> +		 * Don't do any clean up on failure because the caller will
>> +		 * do it all anyway.
>> +		 */
>> +		if (ret)
>> +			return ret;
>> +
>> +		/* Free everything that is owned by the QE structure: */
>> +		kfree(qe->params.cliprects);
>> +		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
>> +			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
>> +
>> +		return 0;
> Above piece of code looks like its own function, so it should probably
> be one.
>
>> +	}
>> +
>> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
>> +	if (!node)
>> +		return -ENOMEM;
>> +
>> +	*node = *qe;
> Any reason we can't simply move ownership of qe? If not, I'd rather
> make a clone function

The qe pointer passed in is a reference to a stack local object in the 
execbuff code path. Thus ownership cannot be transferred. Doing it this 
way keeps the execbuff code nice and simple and all the dynamic memory 
management and list tracking is self contained within the scheduler.

>
>> +	INIT_LIST_HEAD(&node->link);
>> +	node->status = i915_sqs_queued;
>> +	node->stamp  = jiffies;
>> +	i915_gem_request_reference(node->params.request);
>> +
>> +	/* Need to determine the number of incomplete entries in the list as
>> +	 * that will be the maximum size of the dependency list.
>> +	 *
>> +	 * Note that the allocation must not be made with the spinlock acquired
>> +	 * as kmalloc can sleep. However, the unlock/relock is safe because no
>> +	 * new entries can be queued up during the unlock as the i915 driver
>> +	 * mutex is still held. Entries could be removed from the list but that
>> +	 * just means the dep_list will be over-allocated which is fine.
>> +	 */
>> +	spin_lock_irqsave(&scheduler->lock, flags);
>> +	for (r = 0; r < I915_NUM_RINGS; r++) {
>> +		list_for_each_entry(test, &scheduler->node_queue[r], link) {
>> +			if (I915_SQS_IS_COMPLETE(test))
>> +				continue;
>> +
>> +			incomplete++;
>> +		}
>> +	}
>> +
>> +	/* Temporarily unlock to allocate memory: */
>> +	spin_unlock_irqrestore(&scheduler->lock, flags);
> I'd make the above piece of code a helper, these stats are to be
> counted for debugfs anyway, too?
The debugfs code counts all states, not just complete, and requires a 
scheduler stats object. Also, the above explanation about locking is not 
applicable in the debugfs case, only here. And it makes much more sense 
to me to keep the comment and the code all together in one very obvious 
and understandable block than splitting it up.


>
>> +	if (incomplete) {
>> +		node->dep_list = kmalloc_array(incomplete,
>> +					       sizeof(*node->dep_list),
>> +					       GFP_KERNEL);
>> +		if (!node->dep_list) {
>> +			kfree(node);
>> +			return -ENOMEM;
>> +		}
>> +	} else
>> +		node->dep_list = NULL;
>> +
>> +	spin_lock_irqsave(&scheduler->lock, flags);
>> +	node->num_deps = 0;
>> +
>> +	if (node->dep_list) {
>> +		for (r = 0; r < I915_NUM_RINGS; r++)
>> +			i915_generate_dependencies(scheduler, node, r);
>> +
>> +		WARN_ON(node->num_deps > incomplete);
> Wouldn't this condition again lead to a crash? If so, should be BUG_ON
> to cause that crash as early as possible. WARN_ON is only good if there
> is a way of coping with the situation and no imminent system crash is
> bound to happen.
>
>> +	}
>> +
>> +	if (node->priority > scheduler->priority_level_max)
>> +		node->priority = scheduler->priority_level_max;
>> +	else if (node->priority < scheduler->priority_level_min)
>> +		node->priority = scheduler->priority_level_min;
>> +
> There is clamp_val macro in linux/kernel.h .
>
>> +	if ((node->priority > 0) && node->num_deps) {
>> +		i915_scheduler_priority_bump_clear(scheduler);
>> +
>> +		for (i = 0; i < node->num_deps; i++)
>> +			i915_scheduler_priority_bump(scheduler,
>> +					node->dep_list[i], node->priority);
>> +	}
>> +
>> +	list_add_tail(&node->link, &scheduler->node_queue[ring->id]);
>> +
>> +	not_flying = i915_scheduler_count_flying(scheduler, ring) <
>> +						 scheduler->min_flying;
>> +
>> +	spin_unlock_irqrestore(&scheduler->lock, flags);
>> +
>> +	if (not_flying)
>> +		i915_scheduler_submit(ring, true);
>> +
>> +	return 0;
>> +}
>> +
>> +static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
>> +{
>> +	struct drm_i915_private *dev_priv = node->params.dev->dev_private;
>> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
>> +	struct intel_engine_cs  *ring;
>> +
>> +	WARN_ON(!scheduler);
>> +	WARN_ON(!node);
>> +	WARN_ON(node->status != i915_sqs_popped);
> Other states had their I915_SQS_IS_* macro, why some don't?
The purpose of the macro is to allow the combining of individual states 
into classes. E.g. dead and complete can both be considered complete for 
the majority of cases. Only in certain situations do you need to know 
that it really was dead. Hence most places that don't really care just 
use the merged macros, whereas places like this that do care use the 
explicit enum value.

>
>> +
>> +	ring = node->params.ring;
>> +
>> +	/*
>> +	 * Add the node (which should currently be in state popped) to the
>> +	 * front of the queue. This ensure that flying nodes are always held
>> +	 * in hardware submission order.
>> +	 */
>> +	list_add(&node->link, &scheduler->node_queue[ring->id]);
>> +
>> +	node->status = i915_sqs_flying;
>> +
>> +	if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
>> +		bool    success = true;
>> +
>> +		success = ring->irq_get(ring);
>> +		if (success)
>> +			scheduler->flags[ring->id] |= i915_sf_interrupts_enabled;
>> +		else
>> +			return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +/*
>> + * Nodes are considered valid dependencies if they are queued on any ring or
>> + * if they are in flight on a different ring. In flight on the same ring is no
>> + * longer interesting for non-premptive nodes as the ring serialises execution.
>> + * For pre-empting nodes, all in flight dependencies are valid as they must not
>> + * be jumped by the act of pre-empting.
>> + *
>> + * Anything that is neither queued nor flying is uninteresting.
>> + */
>> +static inline bool i915_scheduler_is_dependency_valid(
>> +			struct i915_scheduler_queue_entry *node, uint32_t idx)
>> +{
>> +	struct i915_scheduler_queue_entry *dep;
>> +
>> +	dep = node->dep_list[idx];
>> +	if (!dep)
>> +		return false;
>> +
>> +	if (I915_SQS_IS_QUEUED(dep))
>> +		return true;
>> +
>> +	if (I915_SQS_IS_FLYING(dep)) {
>> +		if (node->params.ring != dep->params.ring)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +static uint32_t i915_scheduler_count_flying(struct i915_scheduler *scheduler,
>> +					    struct intel_engine_cs *ring)
>> +{
>> +	struct i915_scheduler_queue_entry *node;
>> +	uint32_t                          flying = 0;
>> +
>> +	list_for_each_entry(node, &scheduler->node_queue[ring->id], link)
> This loop keeps popping up, it could use a define similar to the ones
> in i915_drv.h ;
>
> #define for_each_hpd_pin(__pin) \
>
>> +		if (I915_SQS_IS_FLYING(node))
>> +			flying++;
>> +
>> +	return flying;
>> +}
>> +
>> +/*
>> + * Add a popped node back in to the queue. For example, because the ring was
>> + * hung when execfinal() was called and thus the ring submission needs to be
>> + * retried later.
>> + */
>> +static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node)
>> +{
>> +	WARN_ON(!node);
> 	if (WARN_ON(!node))
> 		return;
>
> Or rather no check at all, it's going to crash anyway even in the
> calling function if there's NULL, and it's internal function. It's
> relevant to check if the userspace.
>
>> +	WARN_ON(!I915_SQS_IS_FLYING(node));
>> +
>> +	/* Seqno will be reassigned on relaunch */
>> +	node->params.request->seqno = 0;
>> +	node->status = i915_sqs_queued;
>> +}
>> +
>> +/*
>> + * Give up on a node completely. For example, because it is causing the
>> + * ring to hang or is using some resource that no longer exists.
>> + */
>> +static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)
>> +{
>> +	WARN_ON(!node);
>> +	WARN_ON(I915_SQS_IS_COMPLETE(node));
>> +
>> +	node->status = i915_sqs_dead;
>> +}
>> +
>> +/*
>> + * A sequence number has popped out of the hardware and the request handling
>> + * code has mapped it back to a request and will mark that request complete.
>> + * It also calls this function to notify the scheduler about the completion
>> + * so the scheduler's node can be updated appropriately.
>> + * Returns true if the request is scheduler managed, false if not.
>> + */
>> +bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
>> +{
>> +	struct drm_i915_private *dev_priv  = to_i915(req->ring->dev);
>> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
>> +	/* XXX: Need to map back from request to node */
>> +	struct i915_scheduler_queue_entry *node = NULL;
>> +	unsigned long       flags;
>> +
>> +	if (!node)
>> +		return false;
> Not so sure if slicing the series down to an extent that functions are
> impossible to review, was a good idea. Idea is to slice things down, to
> reviewable pieces. It's hard to predict or keep looking forward the
> series what is going to come.
>
>> +
>> +	spin_lock_irqsave(&scheduler->lock, flags);
>> +
>> +	WARN_ON(!I915_SQS_IS_FLYING(node));
>> +
>> +	/* Node was in flight so mark it as complete. */
>> +	if (req->cancelled)
>> +		node->status = i915_sqs_dead;
>> +	else
>> +		node->status = i915_sqs_complete;
>> +
>> +	spin_unlock_irqrestore(&scheduler->lock, flags);
>> +
>> +	/*
>> +	 * XXX: If the in-flight list is now empty then new work should be
>> +	 * submitted. However, this function is called from interrupt context
>> +	 * and thus cannot acquire mutex locks and other such things that are
>> +	 * necessary for fresh submission.
>> +	 */
>> +
>> +	return true;
>> +}
>> +
>> +int i915_scheduler_remove(struct intel_engine_cs *ring)
>> +{
>> +	struct drm_i915_private *dev_priv = ring->dev->dev_private;
>> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
>> +	struct i915_scheduler_queue_entry  *node, *node_next;
>> +	unsigned long       flags;
>> +	int                 flying = 0, queued = 0;
>> +	int                 ret = 0;
>> +	bool                do_submit;
>> +	uint32_t            min_seqno;
>> +	struct list_head    remove;
>> +
>> +	if (list_empty(&scheduler->node_queue[ring->id]))
>> +		return 0;
>> +
>> +	spin_lock_irqsave(&scheduler->lock, flags);
>> +
>> +	/* /i915_scheduler_dump_locked(ring, "remove/pre");/ */
>> +
> This should not be here at all.
Oops. Old debug code. Hadn't spotted it was still there.

>
>> +	/*
>> +	 * In the case where the system is idle, starting 'min_seqno' from a big
>> +	 * number will cause all nodes to be removed as they are now back to
>> +	 * being in-order. However, this will be a problem if the last one to
>> +	 * complete was actually out-of-order as the ring seqno value will be
>> +	 * lower than one or more completed buffers. Thus code looking for the
>> +	 * completion of said buffers will wait forever.
>> +	 * Instead, use the hardware seqno as the starting point. This means
>> +	 * that some buffers might be kept around even in a completely idle
>> +	 * system but it should guarantee that no-one ever gets confused when
>> +	 * waiting for buffer completion.
>> +	 */
>> +	min_seqno = ring->get_seqno(ring, true);
>> +
>> +	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
>> +		if (I915_SQS_IS_QUEUED(node))
>> +			queued++;
>> +		else if (I915_SQS_IS_FLYING(node))
>> +			flying++;
>> +		else if (I915_SQS_IS_COMPLETE(node))
>> +			continue;
>> +
>> +		if (node->params.request->seqno == 0)
>> +			continue;
>> +
>> +		if (!i915_seqno_passed(node->params.request->seqno, min_seqno))
>> +			min_seqno = node->params.request->seqno;
>> +	}
> Couldn't these values be kept cached, instead of counting them at each
> function?
The 'queued' and flying totals could be kept cached but min_seqno is 
dependent upon the state of the hardware so needs to be recalculated. In 
which case calculating the totals here is trivial and avoids having 
extra code elsewhere to keep them up to date.

>
>> +
>> +	INIT_LIST_HEAD(&remove);
>> +	list_for_each_entry_safe(node, node_next, &scheduler->node_queue[ring->id], link) {
>> +		/*
>> +		 * Only remove completed nodes which have a lower seqno than
>> +		 * all pending nodes. While there is the possibility of the
>> +		 * ring's seqno counting backwards, all higher buffers must
>> +		 * be remembered so that the 'i915_seqno_passed()' test can
>> +		 * report that they have in fact passed.
>> +		 *
>> +		 * NB: This is not true for 'dead' nodes. The GPU reset causes
>> +		 * the software seqno to restart from its initial value. Thus
>> +		 * the dead nodes must be removed even though their seqno values
>> +		 * are potentially vastly greater than the current ring seqno.
>> +		 */
>> +		if (!I915_SQS_IS_COMPLETE(node))
>> +			continue;
>> +
>> +		if (node->status != i915_sqs_dead) {
>> +			if (i915_seqno_passed(node->params.request->seqno, min_seqno) &&
>> +			    (node->params.request->seqno != min_seqno))
>> +				continue;
>> +		}
>> +
>> +		list_del(&node->link);
>> +		list_add(&node->link, &remove);
>> +
>> +		/* Strip the dependency info while the mutex is still locked */
>> +		i915_scheduler_remove_dependent(scheduler, node);
>> +
>> +		continue;
>> +	}
>> +
>> +	/*
>> +	 * Release the interrupt reference count if there are no longer any
>> +	 * nodes to worry about.
>> +	 */
>> +	if (!flying && !queued &&
>> +	    (scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {
>> +		ring->irq_put(ring);
>> +		scheduler->flags[ring->id] &= ~i915_sf_interrupts_enabled;
>> +	}
>> +
>> +	/* Launch more packets now? */
>> +	do_submit = (queued > 0) && (flying < scheduler->min_flying);
>> +
>> +	spin_unlock_irqrestore(&scheduler->lock, flags);
>> +
>> +	if (!do_submit && list_empty(&remove))
>> +		return ret;
>> +
>> +	mutex_lock(&ring->dev->struct_mutex);
>> +
>> +	if (do_submit)
>> +		ret = i915_scheduler_submit(ring, true);
> Confusing to have this at a remove function. Function naming needs to
> be reconsidered or moved out from here.
>
>> +
>> +	while (!list_empty(&remove)) {
>> +		node = list_first_entry(&remove, typeof(*node), link);
>> +		list_del(&node->link);
>> +
>> +		/*
>> +		 * The batch buffer must be unpinned before it is unreferenced
>> +		 * otherwise the unpin fails with a missing vma!?
>> +		 */priority_bump_clear
>> +		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
>> +			i915_gem_execbuff_release_batch_obj(node->params.batch_obj);
>> +
>> +		/* Free everything that is owned by the node: */
>> +		i915_gem_request_unreference(node->params.request);
>> +		kfree(node->params.cliprects);
>> +		kfree(node->dep_list);
>> +		kfree(node);
>> +	}
>> +
>> +	mutex_unlock(&ring->dev->struct_mutex);
>> +
>> +	return ret;
>> +}
>> +
>> +static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
>> +{
>> +	struct i915_scheduler_queue_entry *node;
>> +	int i;
>> +
>> +	/*
>> +	 * Ensure circular dependencies don't cause problems and that a bump
>> +	 * by object usage only bumps each using buffer once:
>> +	 */
>> +	for (i = 0; i < I915_NUM_RINGS; i++) {
>> +		list_for_each_entry(node, &scheduler->node_queue[i], link)
>> +			node->bumped = false;
>> +	}
>> +}
>> +
>> +static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
>> +				struct i915_scheduler_queue_entry *target,
>> +				uint32_t bump)
>> +{
>> +	uint32_t new_priority;
>> +	int      i, count;
>> +
>> +	if (target->priority >= scheduler->priority_level_max)
>> +		return 1;
>> +
>> +	if (target->bumped)
>> +		return 0;
>> +
>> +	new_priority = target->priority + bump;
>> +	if ((new_priority <= target->priority) ||
>> +	    (new_priority > scheduler->priority_level_max))
>> +		target->priority = scheduler->priority_level_max;
>> +	else
>> +		target->priority = new_priority;
>> +
>> +	count = 1;
>> +	target->bumped = true;
>> +
>> +	for (i = 0; i < target->num_deps; i++) {
>> +		if (!target->dep_list[i])
>> +			continue;
>> +
>> +		if (target->dep_list[i]->bumped)
>> +			continue;
>> +
>> +		count += i915_scheduler_priority_bump(scheduler,
>> +						      target->dep_list[i],
>> +						      bump);
>> +	}
>> +
>> +	return count;
>> +}
>> +
>> +static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,
>> +				struct i915_scheduler_queue_entry **pop_node,
>> +				unsigned long *flags)
>> +{
>> +	struct drm_i915_private            *dev_priv = ring->dev->dev_private;
>> +	struct i915_scheduler              *scheduler = dev_priv->scheduler;
>> +	struct i915_scheduler_queue_entry  *best;
>> +	struct i915_scheduler_queue_entry  *node;
>> +	int     ret;
>> +	int     i;
>> +	bool	any_queued;
>> +	bool	has_local, has_remote, only_remote;
>> +
>> +	*pop_node = NULL;
>> +	ret = -ENODATA;
>> +
>> +	any_queued = false;
>> +	only_remote = false;
>> +	best = NULL;
> These should just be initialized in-place. But looking at the code
> forward.
>
>> +
>> +	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
>> +		if (!I915_SQS_IS_QUEUED(node))
>> +			continue;
>> +		any_queued = true;
>> +
>> +		has_local  = false;
>> +		has_remote = false;
>> +		for (i = 0; i < node->num_deps; i++) {
>> +			if (!i915_scheduler_is_dependency_valid(node, i))
>> +				continue;
>> +
>> +			if (node->dep_list[i]->params.ring == node->params.ring)
>> +				has_local = true;
>> +			else
>> +				has_remote = true;
>> +		}
>> +
>> +		if (has_remote && !has_local)
>> +			only_remote = true;
>> +
>> +		if (!has_local && !has_remote) {
>> +			if (!best ||
>> +			    (node->priority > best->priority))
>> +				best = node;
>> +		}
>> +	}
>> +
>> +	if (best) {
>> +		list_del(&best->link);
>> +
>> +		INIT_LIST_HEAD(&best->link);
>> +		best->status  = i915_sqs_popped;
>> +
>> +		ret = 0;
>> +	} else {
>> +		/* Can only get here if:
>> +		 * (a) there are no buffers in the queue
>> +		 * (b) all queued buffers are dependent on other buffers
>> +		 *     e.g. on a buffer that is in flight on a different ring
>> +		 */
>> +		if (only_remote) {
>> +			/* The only dependent buffers are on another ring. */
>> +			ret = -EAGAIN;
>> +		} else if (any_queued) {
>> +			/* It seems that something has gone horribly wrong! */
>> +			DRM_ERROR("Broken dependency tracking on ring %d!\n",
>> +				  (int) ring->id);
>> +		}
>> +	}
>> +
>> +	/* i915_scheduler_dump_queue_pop(ring, best); */
>> +
>> +	*pop_node = best;
>> +	return ret;
>> +}
>> +
>> +static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked)
>> +{
>> +	struct drm_device   *dev = ring->dev;
>> +	struct drm_i915_private *dev_priv = dev->dev_private;
>> +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
>> +	struct i915_scheduler_queue_entry  *node;
>> +	unsigned long       flags;
>> +	int                 ret = 0, count = 0;
>> +
>> +	if (!was_locked) {
>> +		ret = i915_mutex_lock_interruptible(dev);
>> +		if (ret)
>> +			return ret;
>> +	}
>> +
> I don't really fancy this construct. Should be moved outside of this
> function for proper lockdep tracking.
>
>> +	WARN_ON(!mutex_is_locked(&dev->struct_mutex));
>> +
>> +	spin_lock_irqsave(&scheduler->lock, flags);
>> +
>> +	/* First time around, complain if anything unexpected occurs: */
>> +	ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
>> +	if (ret) {
>> +		spin_unlock_irqrestore(&scheduler->lock, flags);
>> +
>> +		if (!was_locked)
>> +			mutex_unlock(&dev->struct_mutex);
>> +
>> +		return ret;
>> +	}
>> +
> Dropping the was_locked stuff, this should become a proper goto error
> label. e.g. out_unlock
>
>> +	do {
>> +		WARN_ON(!node);
>> +		WARN_ON(node->params.ring != ring);
>> +		WARN_ON(node->status != i915_sqs_popped);
>> +		count++;
>> +
>> +		/*
>> +		 * The call to pop above will have removed the node from the
>> +		 * list. So add it back in and mark it as in flight.
>> +		 */
>> +		i915_scheduler_fly_node(node);
> Why do we want to pull an object out of the list inside spin lock and
> push it back immediately in our critical code path? Seems like a waste
> for no obvious gain at this point. Why do not we rather just select an
> entry and modify it in-place, if it's going to stay in the same queue
> anyway.
The list order is significant. The element must be moved to the front to 
keep the submitted items in submission order. Doing it this way also 
keeps the code nicely partitioned and easier to understand/maintain. 
Plus, there is a plan to optimise the code by splitting the one single 
list into three separate ones - queued, flying, complete. If/when that 
happens, the element will have to be removed from one list and added to 
another.

>
>> +
>> +		scheduler->flags[ring->id] |= i915_sf_submitting;
>> +		spin_unlock_irqrestore(&scheduler->lock, flags);
>> +		ret = dev_priv->gt.execbuf_final(&node->params);
>> +		spin_lock_irqsave(&scheduler->lock, flags);
>> +		scheduler->flags[ring->id] &= ~i915_sf_submitting;
>> +
>> +		if (ret) {
>> +			int requeue = 1;
> Multipurpose variable, not really a good idea. And as commented
> further, should not exist at all.
>
>> +
>> +			/*
>> +			 * Oh dear! Either the node is broken or the ring is
>> +			 * busy. So need to kill the node or requeue it and try
>> +			 * again later as appropriate.
>> +			 */
>> +
>> +			switch (-ret) {
>> +			case ENODEV:
>> +			case ENOENT:
>> +				/* Fatal errors. Kill the node. */
>> +				requeue = -1;
>> +			break;
> "break" indent is wrong.
>
>> +
>> +			case EAGAIN:
>> +			case EBUSY:
>> +			case EIO:
>> +			case ENOMEM:
>> +			case ERESTARTSYS:
>> +			case EINTR:
>> +				/* Supposedly recoverable errors. */
>> +			break;
>> +
>> +			default:
>> +				/*
>> +				 * Assume the error is recoverable and hope
>> +				 * for the best.
>> +				 */
>> +				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
>> +						 ring->name, ret);
> There's MISSING_CASE macro, should use it.
>
>> +			break;
>> +			}
>> +
> Just move the code below this point to the switch, no point having a
> switch to categorize your options and then doing bunch of ifs to
> execute code that could be in switch.
One of the 'if' paths is to break out of the while loop. Can't do that 
from inside the switch.

>> +			/*
>> +			 * Check that the watchdog/reset code has not nuked
>> +			 * the node while we weren't looking:
>> +			 */
>> +			if (node->status == i915_sqs_dead)
>> +				requeue = 0;
>> +
>> +			if (requeue == 1) {
>> +				i915_scheduler_node_requeue(node);
>> +				/*
>> +				 * No point spinning if the ring is currently
>> +				 * unavailable so just give up and come back
>> +				 * later.
>> +				 */
>> +				break;
>> +			} else if (requeue == -1)
>> +				i915_scheduler_node_kill(node);
>> +		}
>> +
> Ending here, this actual submission of a single node could go to its
> own helper function, these functions now become too long to follow,
> although they really are not doing anything complicated.
>
>> +		/* Keep launching until the sky is sufficiently full. */
>> +		if (i915_scheduler_count_flying(scheduler, ring) >=
>> +						scheduler->min_flying)
>> +			break;
>> +
>> +		ret = i915_scheduler_pop_from_queue_locked(ring, &node, &flags);
>> +	} while (ret == 0);
>> +
>> +	spin_unlock_irqrestore(&scheduler->lock, flags);
>> +
>> +	if (!was_locked)
>> +		mutex_unlock(&dev->struct_mutex);
>> +
>> +	/* Don't complain about not being able to submit extra entries */
>> +	if (ret == -ENODATA)
>> +		ret = 0;
>> +
>> +	return (ret < 0) ? ret : count;
> This is a combined error and success path, keeping to the convention of
> kernel drivers is preferred;
>
> 	if (ret != -ENODATA)
> 		goto out_foo;
>
> 	return count;
>
> out_unlock:
> 	spin_unlock(...);
> out_foo:
> 	return ret;
>
>
>
>> +}
>> +
>> +static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
>> +				struct i915_scheduler_queue_entry *remove)
>> +{
>> +	struct i915_scheduler_queue_entry  *node;
>> +	int     i, r;
>> +	int     count = 0;
>> +
>> +	/*
>> +	 * Ensure that a node is not being removed which is still dependent
>> +	 * upon other (not completed) work. If that happens, it implies
>> +	 * something has gone very wrong with the dependency tracking! Note
>> +	 * that there is no need to worry if this node has been explicitly
>> +	 * killed for some reason - it might be being killed before it got
>> +	 * sent to the hardware.
>> +	 */
>> +	if (remove->status != i915_sqs_dead) {
>> +		for (i = 0; i < remove->num_deps; i++)
>> +			if ((remove->dep_list[i]) &&
>> +			    (!I915_SQS_IS_COMPLETE(remove->dep_list[i])))
>> +				count++;
>> +		WARN_ON(count);
>> +	}
>> +
>> +	/*
>> +	 * Remove this node from the dependency lists of any other node which
>> +	 * might be waiting on it.
>> +	 */
>> +	for (r = 0; r < I915_NUM_RINGS; r++) {
>> +		list_for_each_entry(node, &scheduler->node_queue[r], link) {
>> +			for (i = 0; i < node->num_deps; i++) {
>> +				if (node->dep_list[i] != remove)
>> +					continue;
>> +
>> +				node->dep_list[i] = NULL;
>> +			}
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h
>> new file mode 100644
>> index 0000000..00dc7f3
>> --- /dev/null
>> +++ b/drivers/gpu/drm/i915/i915_scheduler.h
>> @@ -0,0 +1,91 @@
>> +/*
>> + * Copyright (c) 2014 Intel Corporation
>> + *
>> + * Permission is hereby granted, free of charge, to any person obtaining a
>> + * copy of this software and associated documentation files (the "Software"),
>> + * to deal in the Software without restriction, including without limitation
>> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
>> + * and/or sell copies of the Software, and to permit persons to whom the
>> + * Software is furnished to do so, subject to the following conditions:
>> + *
>> + * The above copyright notice and this permission notice (including the next
>> + * paragraph) shall be included in all copies or substantial portions of the
>> + * Software.
>> + *
>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
>> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
>> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
>> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
>> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
>> + * IN THE SOFTWARE.
>> + *
>> + */
>> +
>> +#ifndef _I915_SCHEDULER_H_
>> +#define _I915_SCHEDULER_H_
>> +
>> +enum i915_scheduler_queue_status {
>> +	/* Limbo: */
>> +	i915_sqs_none = 0,
>> +	/* Not yet submitted to hardware: */
>> +	i915_sqs_queued,
>> +	/* Popped from queue, ready to fly: */
>> +	i915_sqs_popped,
>> +	/* Sent to hardware for processing: */
>> +	i915_sqs_flying,
>> +	/* Finished processing on the hardware: */
>> +	i915_sqs_complete,
>> +	/* Killed by watchdog or catastrophic submission failure: */
>> +	i915_sqs_dead,
>> +	/* Limit value for use with arrays/loops */
>> +	i915_sqs_MAX
>> +};
> These should be UPPERCASE_FOR_ENUM_VALUES . See i915_drv.h for samples.
>
>> +
>> +#define I915_SQS_IS_QUEUED(node)	(((node)->status == i915_sqs_queued))
>> +#define I915_SQS_IS_FLYING(node)	(((node)->status == i915_sqs_flying))
>> +#define I915_SQS_IS_COMPLETE(node)	(((node)->status == i915_sqs_complete) || \
>> +					 ((node)->status == i915_sqs_dead))
>> +
> Might be slightly confusing that name is IS_COMPLETE and there is
> actual COMPLETE value. Rather have the test like IS_DONE and then test
> for COMPLETE or DEAD, no confusion. Also, some states have their IS_*
> macro and others don't, is there going to be more?
>
>> +struct i915_scheduler_obj_entry {
>> +	struct drm_i915_gem_object          *obj;
>> +	bool                                read_only;
>> +};
>> +
>> +struct i915_scheduler_queue_entry {
>> +	struct i915_execbuffer_params       params;
>> +	/* -1023 = lowest priority, 0 = default, 1023 = highest */
>> +	int32_t                             priority;
>> +	struct i915_scheduler_obj_entry     *saved_objects;
>> +	int                                 num_objs;
>> +	bool                                bumped;
>> +	struct i915_scheduler_queue_entry   **dep_list;
>> +	int                                 num_deps;
>> +	enum i915_scheduler_queue_status    status;
>> +	unsigned long                       stamp;
>> +	struct list_head                    link;
>> +};
>> +
>> +struct i915_scheduler {
>> +	struct list_head    node_queue[I915_NUM_RINGS];
>> +	uint32_t            flags[I915_NUM_RINGS];
>> +	spinlock_t          lock;
>> +
>> +	/* Tuning parameters: */
>> +	int32_t             priority_level_min;
>> +	int32_t             priority_level_max;
>> +	int32_t             priority_level_preempt;
>> +	uint32_t            min_flying;
>> +};
>> +
>> +/* Flag bits for i915_scheduler::flags */
>> +enum {
>> +	i915_sf_interrupts_enabled  = (1 << 0),
>> +	i915_sf_submitting          = (1 << 1),
> Again, should be uppercase. Also, enums to the beginning of file.
>
>> +};
>> +
>> +int         i915_scheduler_init(struct drm_device *dev);
>> +int         i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
>> +bool        i915_scheduler_notify_request(struct drm_i915_gem_request *req);
>> +
>> +#endif  /* _I915_SCHEDULER_H_ */

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 07/38] drm/i915: Start of GPU scheduler

[Joonas Lahtinen]

Hi,

Adding Daniel as CC to comment below.

On to, 2016-02-18 at 14:22 +0000, John Harrison wrote:
> On 20/01/2016 13:18, Joonas Lahtinen wrote:
> > On Mon, 2016-01-11 at 18:42 +0000, John.C.Harrison@Intel.com wrote:
> > > From: John Harrison <John.C.Harrison@Intel.com>
> > > 

<SNIP>

> > > 
> > > +			this = node->saved_objects + i;
> > > +
> > > +			for (j = 0; j < test->num_objs; j++) {
> > > +				that = test->saved_objects + j;
> > > +
> > > +				if (this->obj != that->obj)
> > > +					continue;
> > How about VMAs? There might be multiple mappings to an object, isn't it
> > enough to depend on the required VMA instead of the whole object?
> The object is what we get coming in from user land through the IOCTL. So 
> why make things more complicated? If there are multiple VMAs referring 
> to the same object then we can't just track an individual VMA as that 
> would loose the dependency on all the other VMAs. Just because the 
> object is mapped to someone else's address space doesn't mean that this 
> batch buffer can't overwrite data they are reading.
> 

Right, makes sense.

> > > +int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
> > > +{
> > > +	struct drm_i915_private *dev_priv = qe->params.dev->dev_private;
> > > +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
> > > +	struct intel_engine_cs  *ring = qe->params.ring;
> > > +	struct i915_scheduler_queue_entry  *node;
> > > +	struct i915_scheduler_queue_entry  *test;
> > > +	unsigned long       flags;
> > > +	bool                not_flying;
> > > +	int                 i, r;
> > > +	int                 incomplete = 0;
> > > +
> > > +	WARN_ON(!scheduler);
> > > +
> > This kind of situations should have a be a BUG_ON, because scheduler
> > being zero is literally going to cause an OOPS in the next dereference
> > which is going to happen unconditionally. WARN + OOPS is kind of what
> > BUG_ON should be used avoid. But this should be removed anyway after
> > scheduler is made a data member of dev_priv.
> 
> The WARNs were originally BUGs but Daniel Vetter had the opposite 
> demand. His view was the driver should never BUG under any 
> circumstances. A WARN followed by an oops is better than a BUG because 
> maybe it won't actually oops.
> 

WARN_ON is better than BUG_ON when there won't be an immediate OOPS.
But if you're doing a null pointer dereference like here if scheduler
is NULL, it is 100% sure it is going to either OOPS or cause horrible
undefined behaviour (on non-x86 platform).

Something like;

	if (WARN_ON(!a))
		return -ENODEV;

Could be what Daniel meant (if the error is propagated up and somehow
dealt with), but;

	WARN_ON(!a);
	a->foo = bar;

Should simply be BUG_ON(!a), because otherwise it'll just end up being
WARNING + OOPS right after each other.

> 
> > 
> > > +	if (1/*i915.scheduler_override & i915_so_direct_submit*/) {
> > I assume this is going to be addressed in a future commit. Could have
> > been introduced in this patch, too.
> > 
> > > +		int ret;
> > > +
> > > +		scheduler->flags[qe->params.ring->id] |= i915_sf_submitting;
> > > +		ret = dev_priv->gt.execbuf_final(&qe->params);
> > > +		scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
> > > +
> > The kerneldoc should mention locking requirements of this function.
> > 
> > > +		/*
> > > +		 * Don't do any clean up on failure because the caller will
> > > +		 * do it all anyway.
> > > +		 */
> > > +		if (ret)
> > > +			return ret;
> > > +
> > > +		/* Free everything that is owned by the QE structure: */
> > > +		kfree(qe->params.cliprects);
> > > +		if (qe->params.dispatch_flags & I915_DISPATCH_SECURE)
> > > +			i915_gem_execbuff_release_batch_obj(qe->params.batch_obj);
> > > +
> > > +		return 0;
> > Above piece of code looks like its own function, so it should probably
> > be one.
> > 
> > > +	}
> > > +
> > > +	node = kmalloc(sizeof(*node), GFP_KERNEL);
> > > +	if (!node)
> > > +		return -ENOMEM;
> > > +
> > > +	*node = *qe;
> > Any reason we can't simply move ownership of qe? If not, I'd rather
> > make a clone function
> 
> The qe pointer passed in is a reference to a stack local object in the 
> execbuff code path. Thus ownership cannot be transferred. Doing it this 
> way keeps the execbuff code nice and simple and all the dynamic memory 
> management and list tracking is self contained within the scheduler.
> 

I would indicate this with const qualifier in the function parameter.

> > > +
> > > +static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node)
> > > +{
> > > +	struct drm_i915_private *dev_priv = node->params.dev->dev_private;
> > > +	struct i915_scheduler   *scheduler = dev_priv->scheduler;
> > > +	struct intel_engine_cs  *ring;
> > > +
> > > +	WARN_ON(!scheduler);
> > > +	WARN_ON(!node);
> > > +	WARN_ON(node->status != i915_sqs_popped);
> > Other states had their I915_SQS_IS_* macro, why some don't?
> The purpose of the macro is to allow the combining of individual states 
> into classes. E.g. dead and complete can both be considered complete for 
> the majority of cases. Only in certain situations do you need to know 
> that it really was dead. Hence most places that don't really care just 
> use the merged macros, whereas places like this that do care use the 
> explicit enum value.
> 

Right, just asking cause having a macro might increase consistency.

Also here, we have WARN_ON(!node), and then next line dereferencing
node, which again is surely better off as BUG_ON.

Although it's not unreasonable to expect function to OOPS if you pass
null pointer. I think only the higher calling level should have
if(WARN_ON(!node)) return ... or BUG_ON construct, before calling this
function.

Only calls coming from userland need to be treated with such care that
anything can be passed, not internal functions. We're not having
BUG_ON(!dev) or BUG_ON(!dev_priv) around either.

> > > +	/*
> > > +	 * In the case where the system is idle, starting 'min_seqno' from a big
> > > +	 * number will cause all nodes to be removed as they are now back to
> > > +	 * being in-order. However, this will be a problem if the last one to
> > > +	 * complete was actually out-of-order as the ring seqno value will be
> > > +	 * lower than one or more completed buffers. Thus code looking for the
> > > +	 * completion of said buffers will wait forever.
> > > +	 * Instead, use the hardware seqno as the starting point. This means
> > > +	 * that some buffers might be kept around even in a completely idle
> > > +	 * system but it should guarantee that no-one ever gets confused when
> > > +	 * waiting for buffer completion.
> > > +	 */
> > > +	min_seqno = ring->get_seqno(ring, true);
> > > +
> > > +	list_for_each_entry(node, &scheduler->node_queue[ring->id], link) {
> > > +		if (I915_SQS_IS_QUEUED(node))
> > > +			queued++;
> > > +		else if (I915_SQS_IS_FLYING(node))
> > > +			flying++;
> > > +		else if (I915_SQS_IS_COMPLETE(node))
> > > +			continue;
> > > +
> > > +		if (node->params.request->seqno == 0)
> > > +			continue;
> > > +
> > > +		if (!i915_seqno_passed(node->params.request->seqno, min_seqno))
> > > +			min_seqno = node->params.request->seqno;
> > > +	}
> > Couldn't these values be kept cached, instead of counting them at each
> > function?
> The 'queued' and flying totals could be kept cached but min_seqno is 
> dependent upon the state of the hardware so needs to be recalculated. In 
> which case calculating the totals here is trivial and avoids having 
> extra code elsewhere to keep them up to date.
> 

Ok.

Btw, for_each_scheduler_node or such would be a great macro. Those are
very much preferred for readability.

$ fgrep for_each_ drivers/gpu/drm/i915/* | wc -l
525

> > 
> > > +
> > > +	INIT_LIST_HEAD(&remove);
> > > +	list_for_each_entry_safe(node, node_next, &scheduler->node_queue[ring->id], link) {
> > > +		/*
> > > +		 * Only remove completed nodes which have a lower seqno than
> > > +		 * all pending nodes. While there is the possibility of the
> > > +		 * ring's seqno counting backwards, all higher buffers must
> > > +		 * be remembered so that the 'i915_seqno_passed()' test can
> > > +		 * report that they have in fact passed.
> > > +		 *
> > > +		 * NB: This is not true for 'dead' nodes. The GPU reset causes
> > > +		 * the software seqno to restart from its initial value. Thus
> > > +		 * the dead nodes must be removed even though their seqno values
> > > +		 * are potentially vastly greater than the current ring seqno.
> > > +		 */
> > > +		if (!I915_SQS_IS_COMPLETE(node))
> > > +			continue;
> > > +
> > > +		if (node->status != i915_sqs_dead) {
> > > +			if (i915_seqno_passed(node->params.request->seqno, min_seqno) &&
> > > +			    (node->params.request->seqno != min_seqno))
> > > +				continue;
> > > +		}
> > > +
> > > +		list_del(&node->link);
> > > +		list_add(&node->link, &remove);
> > > +
> > > +		/* Strip the dependency info while the mutex is still locked */
> > > +		i915_scheduler_remove_dependent(scheduler, node);
> > > +
> > > +		continue;
> > > +	}

<SNIP>

> > > +	do {
> > > +		WARN_ON(!node);
> > > +		WARN_ON(node->params.ring != ring);
> > > +		WARN_ON(node->status != i915_sqs_popped);
> > > +		count++;
> > > +
> > > +		/*
> > > +		 * The call to pop above will have removed the node from the
> > > +		 * list. So add it back in and mark it as in flight.
> > > +		 */
> > > +		i915_scheduler_fly_node(node);
> > Why do we want to pull an object out of the list inside spin lock and
> > push it back immediately in our critical code path? Seems like a waste
> > for no obvious gain at this point. Why do not we rather just select an
> > entry and modify it in-place, if it's going to stay in the same queue
> > anyway.
> The list order is significant. The element must be moved to the front to 
> keep the submitted items in submission order. Doing it this way also 
> keeps the code nicely partitioned and easier to understand/maintain. 
> Plus, there is a plan to optimise the code by splitting the one single 
> list into three separate ones - queued, flying, complete. If/when that 
> happens, the element will have to be removed from one list and added to 
> another.

Ok.

> 
> > 
> > > +
> > > +		scheduler->flags[ring->id] |= i915_sf_submitting;
> > > +		spin_unlock_irqrestore(&scheduler->lock, flags);
> > > +		ret = dev_priv->gt.execbuf_final(&node->params);
> > > +		spin_lock_irqsave(&scheduler->lock, flags);
> > > +		scheduler->flags[ring->id] &= ~i915_sf_submitting;
> > > +
> > > +		if (ret) {
> > > +			int requeue = 1;
> > Multipurpose variable, not really a good idea. And as commented
> > further, should not exist at all.
> > 
> > > +
> > > +			/*
> > > +			 * Oh dear! Either the node is broken or the ring is
> > > +			 * busy. So need to kill the node or requeue it and try
> > > +			 * again later as appropriate.
> > > +			 */
> > > +
> > > +			switch (-ret) {
> > > +			case ENODEV:
> > > +			case ENOENT:
> > > +				/* Fatal errors. Kill the node. */
> > > +				requeue = -1;
> > > +			break;
> > "break" indent is wrong.
> > 
> > > +
> > > +			case EAGAIN:
> > > +			case EBUSY:
> > > +			case EIO:
> > > +			case ENOMEM:
> > > +			case ERESTARTSYS:
> > > +			case EINTR:
> > > +				/* Supposedly recoverable errors. */
> > > +			break;
> > > +
> > > +			default:
> > > +				/*
> > > +				 * Assume the error is recoverable and hope
> > > +				 * for the best.
> > > +				 */
> > > +				DRM_DEBUG_DRIVER("<%s> Got unexpected error from execfinal(): %d!\n",
> > > +						 ring->name, ret);
> > There's MISSING_CASE macro, should use it.
> > 
> > > +			break;
> > > +			}
> > > +
> > Just move the code below this point to the switch, no point having a
> > switch to categorize your options and then doing bunch of ifs to
> > execute code that could be in switch.
> One of the 'if' paths is to break out of the while loop. Can't do that 
> from inside the switch.

I do think the code could still be simplified, even if it involved a
carefully placed goto.

> 
> > > +			/*
> > > +			 * Check that the watchdog/reset code has not nuked
> > > +			 * the node while we weren't looking:
> > > +			 */
> > > +			if (node->status == i915_sqs_dead)
> > > +				requeue = 0;

Btw, just noticed, should some locking of node occur? Is it handled
gracefully if right after this check, the status is changed, or do we
have a race condition here?

> 
-- 
Joonas Lahtinen
Open Source Technology Center
Intel Corporation
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx