[PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Ben Widawsky]

The primary goal of these patches is to introduce what I've started
calling, "prelocations" on Broadwell. A prelocation is like a
relocation, except not. When a GPU client specifies a prelocation, it is
instructing the kernel where in the GPU address the buffer should be
mapped. The mechanic works very similarly to a relocation except it uses
the execbuffer object to obtain the offset, and bind if needed. If a GPU
client uses only prelocations, the relocation process can be entirely
skipped. This sounds like a big win initially, but realistically with
full PPGTT and 48b address space it's unlikely to noticeably improve
anything. Doing this work leaves the address space allocation up to
libc/malloc [1] instead of drm_mm which I believe has some upside due to
the hits on creating new VMAs. Not specific to prelocations, dynamic
page table allocations by themselves can save measurable memory on systems
running multiple GPU clients. As previously mentioned, this kind of thing is
needed for OCL 2.0 SVM. One other advantage I've discussed with Ken... [2].

The difficult part to enable this [for 64b platforms] is supporting the
48b address space. As mentioned in previous versions of this cover
letter, and my blog post [3], it's not feasible to allocate the entire 48b
address space's page tables. Dynamic page table allocation and teardown
required a lot of plumbing and rework, and to make the interfaces as
neat as possible, I also had to put a good deal of work into GEN7 PPGTT
well. The other really difficult part is taking the malloc'd memory and
turning it into GPU usable pages. Luckily, Chris already did that for me
with userptr, so I simply reused his work.

The kernel patches are lightly tested at best. Previous iterations of
this series were more thoroughly tested, but enough has changed since
then that I would assume the code is unstable. If miraculously it is
almost stable, there are still a lot of cosmetic things to clean up, and
a performance optimization to reduce re-mapping already mapped objects.
I started on a patch to do this but ran into too many stability problems
(See Optimize PDP loads from previous posts). It's likely memory leaks
are introduced with the dynamic page tables; plugging those would nice.
One could also implement the reaper I refer to in the comments.

With the kernel prelocation support are the libdrm patches, an
intel-gpu-tools test, and a mesa patch. Some parts of the code are in
rough shape, and were meant for demonstration only. The userspace
components in particular were mostly meant as sample code. [4]

The series is fundamental 5 parts with some bleeding between 2-3, and
3-4.

1. [00-18] Provide fixes to make a stable branch for test with full
PPGTT.  I've previously posted this as a separate series. In the
meanwhile, many similar fixes have gone in, and some of these may be
dropped. So this is mostly here for completeness.

2. [19-42] Rework code to avoid as much future churn
as possible.  Nothing special here. Some of this is arguably #3.

3. [43-46] Make page table allocations dynamic. I tried to keep this
generic, but since the current code supported very specific page table
depths, it's really mostly GEN7.

4. [47-67] GEN8 dynamic page table support with 64b page table support.
This was very hard to split up, and is definitely the majority of the
work.

5. [68] A basic SVM interface.  I opted not to use create2 IOCTL since
there are patches for that already, and I wanted to have something
that's as reusable as possible.  X. the rest are
workaround/libdrm/mesa/igt

Kernel:
http://cgit.freedesktop.org/~bwidawsk/drm-intel/log/?h=prelocate
libdrm:
http://cgit.freedesktop.org/~bwidawsk/drm/log/?h=prelocate
mesa:
http://cgit.freedesktop.org/~bwidawsk/mesa/log/?h=prelocate
IGT:
http://cgit.freedesktop.org/~bwidawsk/intel-gpu-tools/log/?h=prelocate

Final thoughts:
* Due to time pressure, the ability to go back and test on GEN7 was lost.
The original patches I posted back in March did work fine on GEN7, but I
cannot speak to the quality now. That said, I did the work, so I figured
I may as well provide it. For the sake of progress, someone should
test/fix GEN7, or simply drop the GEN7 support.

* Broadwell is currently hanging with this patch series when I run piglit.
I have gone through plenty of software bugs, and this current hang is
baffling. Therefore I think it makes sense to either parameterize, or
CONFIG_ dynamic page table allocations until that's solved.

* Again on the stability, there are a lot of extra flushes introduced as a
result of this series. I believe if we can figure out the case of some
of these issues, we can remove some flushes.

* I haven't tested aliasing PPGTT only in a while. Someone should do that.

* I'll bet 32b is broken.

* A lot of issues I had were related to the complexities when dealing with
legacy contexts. It's possible, and I am hopeful that with execlists
these issues go away, and so do the hangs.

* The patches have been rebased SOOOOO many times that they really need to
be reviewed closely to make sure they're bisectable. They were at one
time, but I doubt it's the case now.



[1] We have to use mmap in certain situations due to a hardware
limitation. I'm not sure how libc manages these things together. I hope
it's efficient...

[2] We can potentially always set the state base to be 0, and rely on HW
contexts to save restore this information, thus eliminating this
non-pipelined state upload. It turns out this is not possible for all
cases because of hardware limitations, but it's a neat idea that someone
can possibly turn into something useful. It's also probably a premature
optimization given how many PIPE CONTROL stalls we have.

[3] https://bwidawsk.net/blog/index.php/2014/07/future-ppgtt-part-4-dynamic-page-table-allocations-64-bit-address-space-gpu-mirroring-and-yeah-something-about-relocs-too/

[4] This was the best I could do on short notice. I won't be improving,
rebasing, or fixing these patches any longer, but someone is welcome to take
them over. Consider this my parting gift before I go on sabbatical [tomorrow].

--

Ben Widawsky (68):
  drm/i915: Split up do_switch
  drm/i915: Extract l3 remapping out of ctx switch
  drm/i915/ppgtt: Load address space after mi_set_context
  drm/i915: Fix another another use-after-free in do_switch
  drm/i915/ctx: Return earlier on failure
  drm/i915/error: vma error capture prettyify
  drm/i915/error: Do a better job of disambiguating VMAs
  drm/i915/error: Capture vmas instead of BOs
  drm/i915: Add some extra guards in evict_vm
  drm/i915: Make an uninterruptible evict
  drm/i915: More correct (slower) ppgtt cleanup
  drm/i915: Defer PPGTT cleanup
  drm/i915/bdw: Enable full PPGTT
  drm/i915: Get the error state over the wire (HACKish)
  drm/i915/gen8: Invalidate TLBs before PDP reload
  drm/i915: Remove false assertion in ppgtt_release
  Revert "drm/i915/bdw: Use timeout mode for RC6 on bdw"
  drm/i915/trace: Fix offsets for 64b
  drm/i915: Wrap VMA binding
  drm/i915: Make pin global flags explicit
  drm/i915: Split out aliasing binds
  drm/i915: fix gtt_total_entries()
  drm/i915: Rename to GEN8_LEGACY_PDPES
  drm/i915: Split out verbose PPGTT dumping
  drm/i915: s/pd/pdpe, s/pt/pde
  drm/i915: rename map/unmap to dma_map/unmap
  drm/i915: Setup less PPGTT on failed pagedir
  drm/i915: clean up PPGTT init error path
  drm/i915: Un-hardcode number of page directories
  drm/i915: Make gen6_write_pdes gen6_map_page_tables
  drm/i915: Range clearing is PPGTT agnostic
  drm/i915: Page table helpers, and define renames
  drm/i915: construct page table abstractions
  drm/i915: Complete page table structures
  drm/i915: Create page table allocators
  drm/i915: Generalize GEN6 mapping
  drm/i915: Clean up pagetable DMA map & unmap
  drm/i915: Always dma map page table allocations
  drm/i915: Consolidate dma mappings
  drm/i915: Always dma map page directory allocations
  drm/i915: Track GEN6 page table usage
  drm/i915: Extract context switch skip logic
  drm/i915: Track page table reload need
  drm/i915: Initialize all contexts
  drm/i915: Finish gen6/7 dynamic page table allocation
  drm/i915/bdw: Use dynamic allocation idioms on free
  drm/i915/bdw: pagedirs rework allocation
  drm/i915/bdw: pagetable allocation rework
  drm/i915/bdw: Make the pdp switch a bit less hacky
  drm/i915: num_pd_pages/num_pd_entries isn't useful
  drm/i915: Extract PPGTT param from pagedir alloc
  drm/i915/bdw: Split out mappings
  drm/i915/bdw: begin bitmap tracking
  drm/i915/bdw: Dynamic page table allocations
  drm/i915/bdw: Make pdp allocation more dynamic
  drm/i915/bdw: Abstract PDP usage
  drm/i915/bdw: Add dynamic page trace events
  drm/i915/bdw: Add ppgtt info for dynamic pages
  drm/i915/bdw: implement alloc/teardown for 4lvl
  drm/i915/bdw: Add 4 level switching infrastructure
  drm/i915/bdw: Generalize PTE writing for GEN8 PPGTT
  drm/i915: Plumb sg_iter through va allocation ->maps
  drm/i915: Introduce map and unmap for VMAs
  drm/i915: Depend exclusively on map and unmap_vma
  drm/i915: Expand error state's address width to 64b
  drm/i915/bdw: Flip the 48b switch
  drm/i915: Provide a soft_pin hook
  XXX: drm/i915: Unexplained workarounds

 drivers/gpu/drm/i915/i915_debugfs.c        |  114 +-
 drivers/gpu/drm/i915/i915_drv.h            |   61 +-
 drivers/gpu/drm/i915/i915_gem.c            |  231 +++-
 drivers/gpu/drm/i915/i915_gem_context.c    |  276 ++++-
 drivers/gpu/drm/i915/i915_gem_evict.c      |   39 +-
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   27 +-
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 1838 +++++++++++++++++++++-------
 drivers/gpu/drm/i915/i915_gem_gtt.h        |  379 +++++-
 drivers/gpu/drm/i915/i915_gem_stolen.c     |    2 +-
 drivers/gpu/drm/i915/i915_gem_userptr.c    |    7 +-
 drivers/gpu/drm/i915/i915_gpu_error.c      |  171 ++-
 drivers/gpu/drm/i915/i915_reg.h            |    1 +
 drivers/gpu/drm/i915/i915_sysfs.c          |    2 +-
 drivers/gpu/drm/i915/i915_trace.h          |  156 ++-
 drivers/gpu/drm/i915/intel_pm.c            |   16 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c    |    2 +-
 include/uapi/drm/i915_drm.h                |    3 +-
 17 files changed, 2588 insertions(+), 737 deletions(-)

-- 
2.0.4

[PATCH 01/68] drm/i915: Split up do_switch

[Ben Widawsky]

There are two important reasons for this patch. It should make the
existing code a lot more readable. It also makes the next patch much
easier to understand in my opinion. There are 2 main variables that
effect this function, leaving 4 permutations:
ring: RCS vs !RCS
PPGTT: full or not

I didn't find extracting the full PPGTT usage to be very beneficial at
this point, but it may be in the future.

This was originally recommended by Daniel Vetter, and in this case, I
agree. There was no intentional behavioral change.

v2: Change the pin assertion to be GGTT only. This is more accurate.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 76 +++++++++++++++++++++------------
 1 file changed, 49 insertions(+), 27 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 3b99390..16aebc6 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -609,31 +609,57 @@ mi_set_context(struct intel_engine_cs *ring,
 	return ret;
 }
 
-static int do_switch(struct intel_engine_cs *ring,
-		     struct intel_context *to)
+static void do_switch_fini_common(struct intel_engine_cs *ring,
+				  struct intel_context *from,
+				  struct intel_context *to)
+{
+	if (likely(from))
+		i915_gem_context_unreference(from);
+	i915_gem_context_reference(to);
+	ring->last_context = to;
+}
+
+static int do_switch_xcs(struct intel_engine_cs *ring,
+			 struct intel_context *from,
+			 struct intel_context *to)
+{
+	struct i915_hw_ppgtt *ppgtt = ctx_to_ppgtt(to);
+	int ret;
+
+	BUG_ON(from && from->legacy_hw_ctx.rcs_state != NULL);
+
+	if (USES_FULL_PPGTT(dev)) {
+		ret = ppgtt->switch_mm(ppgtt, ring, false);
+		if (ret)
+			return ret;
+	}
+
+	if (from)
+		do_switch_fini_common(ring, from, to);
+
+	return 0;
+}
+
+static int do_switch_rcs(struct intel_engine_cs *ring,
+			 struct intel_context *from,
+			 struct intel_context *to)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
-	struct intel_context *from = ring->last_context;
 	struct i915_hw_ppgtt *ppgtt = ctx_to_ppgtt(to);
 	u32 hw_flags = 0;
 	bool uninitialized = false;
 	int ret, i;
 
-	if (from != NULL && ring == &dev_priv->ring[RCS]) {
+	if (from != NULL) {
 		BUG_ON(from->legacy_hw_ctx.rcs_state == NULL);
 		BUG_ON(!i915_gem_obj_is_pinned(from->legacy_hw_ctx.rcs_state));
 	}
 
-	if (from == to && !to->remap_slice)
-		return 0;
-
 	/* Trying to pin first makes error handling easier. */
-	if (ring == &dev_priv->ring[RCS]) {
-		ret = i915_gem_obj_ggtt_pin(to->legacy_hw_ctx.rcs_state,
-					    get_context_alignment(ring->dev), 0);
-		if (ret)
-			return ret;
-	}
+	ret = i915_gem_obj_ggtt_pin(to->legacy_hw_ctx.rcs_state,
+				    get_context_alignment(ring->dev), 0);
+	if (ret)
+		return ret;
 
 	/*
 	 * Pin can switch back to the default context if we end up calling into
@@ -648,12 +674,6 @@ static int do_switch(struct intel_engine_cs *ring,
 			goto unpin_out;
 	}
 
-	if (ring != &dev_priv->ring[RCS]) {
-		if (from)
-			i915_gem_context_unreference(from);
-		goto done;
-	}
-
 	/*
 	 * Clear this page out of any CPU caches for coherent swap-in/out. Note
 	 * that thanks to write = false in this call and us not setting any gpu
@@ -712,15 +732,11 @@ static int do_switch(struct intel_engine_cs *ring,
 
 		/* obj is kept alive until the next request by its active ref */
 		i915_gem_object_ggtt_unpin(from->legacy_hw_ctx.rcs_state);
-		i915_gem_context_unreference(from);
 	}
 
 	uninitialized = !to->legacy_hw_ctx.initialized && from == NULL;
 	to->legacy_hw_ctx.initialized = true;
-
-done:
-	i915_gem_context_reference(to);
-	ring->last_context = to;
+	do_switch_fini_common(ring, from, to);
 
 	if (uninitialized) {
 		ret = i915_gem_render_state_init(ring);
@@ -731,8 +747,7 @@ done:
 	return 0;
 
 unpin_out:
-	if (ring->id == RCS)
-		i915_gem_object_ggtt_unpin(to->legacy_hw_ctx.rcs_state);
+	i915_gem_object_ggtt_unpin(to->legacy_hw_ctx.rcs_state);
 	return ret;
 }
 
@@ -750,6 +765,7 @@ int i915_switch_context(struct intel_engine_cs *ring,
 			struct intel_context *to)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
+	struct intel_context *from = ring->last_context;
 
 	WARN_ON(!mutex_is_locked(&dev_priv->dev->struct_mutex));
 
@@ -763,7 +779,13 @@ int i915_switch_context(struct intel_engine_cs *ring,
 		return 0;
 	}
 
-	return do_switch(ring, to);
+	if (from == to && !to->remap_slice)
+		return 0;
+
+	if (ring->id == RCS)
+		return do_switch_rcs(ring, from, to);
+	else
+		return do_switch_xcs(ring, from, to);
 }
 
 static bool hw_context_enabled(struct drm_device *dev)
-- 
2.0.4

[PATCH 02/68] drm/i915: Extract l3 remapping out of ctx switch

[Ben Widawsky]

This is just a cosmetic change to try to put do_switch_rcs on a diet. As
it stands, the function was quite complex, and error prone.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 32 ++++++++++++++++++++------------
 1 file changed, 20 insertions(+), 12 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 16aebc6..5a46ae3 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -640,6 +640,24 @@ static int do_switch_xcs(struct intel_engine_cs *ring,
 	return 0;
 }
 
+static void remap_l3(struct intel_engine_cs *ring,
+		     struct intel_context *ctx)
+{
+	int ret, i;
+
+	for (i = 0; i < MAX_L3_SLICES; i++) {
+		if (!(ctx->remap_slice & (1<<i)))
+			continue;
+
+		ret = i915_gem_l3_remap(ring, i);
+		/* If it failed, try again next round */
+		if (ret)
+			DRM_DEBUG_DRIVER("L3 remapping failed\n");
+		else
+			ctx->remap_slice &= ~(1<<i);
+	}
+}
+
 static int do_switch_rcs(struct intel_engine_cs *ring,
 			 struct intel_context *from,
 			 struct intel_context *to)
@@ -648,7 +666,7 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 	struct i915_hw_ppgtt *ppgtt = ctx_to_ppgtt(to);
 	u32 hw_flags = 0;
 	bool uninitialized = false;
-	int ret, i;
+	int ret;
 
 	if (from != NULL) {
 		BUG_ON(from->legacy_hw_ctx.rcs_state == NULL);
@@ -699,17 +717,7 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 	if (ret)
 		goto unpin_out;
 
-	for (i = 0; i < MAX_L3_SLICES; i++) {
-		if (!(to->remap_slice & (1<<i)))
-			continue;
-
-		ret = i915_gem_l3_remap(ring, i);
-		/* If it failed, try again next round */
-		if (ret)
-			DRM_DEBUG_DRIVER("L3 remapping failed\n");
-		else
-			to->remap_slice &= ~(1<<i);
-	}
+	remap_l3(ring, to);
 
 	/* The backing object for the context is done after switching to the
 	 * *next* context. Therefore we cannot retire the previous context until
-- 
2.0.4

[PATCH 03/68] drm/i915/ppgtt: Load address space after mi_set_context

[Ben Widawsky]

The simple explanation is, the docs say to do this for GEN8. Perhaps we
want to do this for GEN7 too, I am not certain.

PDPs are saved and restored with context. Contexts (without execlists)
only exist on the render ring. The docs say that PDPs are not power
context save/restored.  I've learned that this actually means something
which SW doesn't care about. So pretend the statement doesn't exist.
For non RCS, nothing changes.

All this patch now does is change the ordering of LRI vs MI_SET_CONTEXT
for the initialization of the context. I do this because the docs say to
do it, and frankly, I cannot reason why it is necessary. I've thought
about it a lot, and tried, without success, to get a reason from design.
The answer I got more or less says, "gen7 is different than gen8." I've
given up, and am adding this little bit of code to make it in sync with
the docs.

v2: Completely rewritten commit message that addresses the requests
Ville made for v1
Only load PDPs for initial context load (Ville)

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 5a46ae3..c9aa3e6 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -666,6 +666,7 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 	struct i915_hw_ppgtt *ppgtt = ctx_to_ppgtt(to);
 	u32 hw_flags = 0;
 	bool uninitialized = false;
+	bool needs_pd_load = (INTEL_INFO(ring->dev)->gen < 8) && USES_FULL_PPGTT(ring->dev);
 	int ret;
 
 	if (from != NULL) {
@@ -686,7 +687,10 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 	 */
 	from = ring->last_context;
 
-	if (USES_FULL_PPGTT(ring->dev)) {
+	if (needs_pd_load) {
+		/* Older GENs still want the load first, "PP_DCLV followed by
+		 * PP_DIR_BASE register through Load Register Immediate commands
+		 * in Ring Buffer before submitting a context."*/
 		ret = ppgtt->switch_mm(ppgtt, ring, false);
 		if (ret)
 			goto unpin_out;
@@ -710,13 +714,34 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 		vma->bind_vma(vma, to->legacy_hw_ctx.rcs_state->cache_level, GLOBAL_BIND);
 	}
 
-	if (!to->legacy_hw_ctx.initialized || i915_gem_context_is_default(to))
+	if (!to->legacy_hw_ctx.initialized || i915_gem_context_is_default(to)) {
 		hw_flags |= MI_RESTORE_INHIBIT;
+		needs_pd_load = USES_FULL_PPGTT(ring->dev) && IS_GEN8(ring->dev);
+	}
 
 	ret = mi_set_context(ring, to, hw_flags);
 	if (ret)
 		goto unpin_out;
 
+	/* GEN8 does *not* require an explicit reload if the PDPs have been
+	 * setup, and we do not wish to move them.
+	 *
+	 * XXX: If we implemented page directory eviction code, this
+	 * optimization needs to be removed.
+	 */
+	if (needs_pd_load) {
+		ret = ppgtt->switch_mm(ppgtt, ring, false);
+		/* The hardware context switch is emitted, but we haven't
+		 * actually changed the state - so it's probably safe to bail
+		 * here. Still, let the user know something dangerous has
+		 * happened.
+		 */
+		if (ret) {
+			DRM_ERROR("Failed to change address space on context switch\n");
+			goto unpin_out;
+		}
+	}
+
 	remap_l3(ring, to);
 
 	/* The backing object for the context is done after switching to the
-- 
2.0.4

[PATCH 04/68] drm/i915: Fix another another use-after-free in do_switch

[Ben Widawsky]

See the following for many more details.

commit acc240d41ea1ab9c488a79219fb313b5b46265ae
Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date:   Thu Dec 5 15:42:34 2013 +0100

    drm/i915: Fix use-after-free in do_switch

In this case, the issue is only for full PPGTT:
do_switch
  context_unref
    ppgtt_release
      i915_gpu_idle
	switch_to_default
	from changes to default context

This could be backported to the pre do_switch cleanup I did in this
series. However, it's much cleaner and more obvious as a patch on top,
so I'd really like to do this as a post cleanup patch.

v2: There was a bug in the original patch where the ring->last_context
was set too early. I am not sure how this wasn't being hit when I sent
this previously.

Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index c9aa3e6..0ce8fc9 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -609,14 +609,18 @@ mi_set_context(struct intel_engine_cs *ring,
 	return ret;
 }
 
-static void do_switch_fini_common(struct intel_engine_cs *ring,
-				  struct intel_context *from,
-				  struct intel_context *to)
+static struct intel_context *do_switch_fini_common(struct intel_engine_cs *ring,
+						   struct intel_context *from,
+						   struct intel_context *to)
 {
+	struct intel_context *ret;
 	if (likely(from))
 		i915_gem_context_unreference(from);
 	i915_gem_context_reference(to);
+	ret = ring->last_context;
 	ring->last_context = to;
+
+	return ret;
 }
 
 static int do_switch_xcs(struct intel_engine_cs *ring,
@@ -762,14 +766,20 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 		 */
 		from->legacy_hw_ctx.rcs_state->dirty = 1;
 		BUG_ON(from->legacy_hw_ctx.rcs_state->ring != ring);
-
-		/* obj is kept alive until the next request by its active ref */
-		i915_gem_object_ggtt_unpin(from->legacy_hw_ctx.rcs_state);
 	}
 
 	uninitialized = !to->legacy_hw_ctx.initialized && from == NULL;
 	to->legacy_hw_ctx.initialized = true;
-	do_switch_fini_common(ring, from, to);
+	/* From may have disappeared again after the context unref */
+	from = do_switch_fini_common(ring, from, to);
+	if (from != NULL) {
+		/* obj is kept alive until the next request by its active ref.
+		 * XXX: The context needs to be unpinned last, or else we risk
+		 * hitting evict/idle on the ppgtt free, which will call back
+		 * into this, and we'll get a double unpin on this context
+		 */
+		i915_gem_object_ggtt_unpin(from->legacy_hw_ctx.rcs_state);
+	}
 
 	if (uninitialized) {
 		ret = i915_gem_render_state_init(ring);
-- 
2.0.4

[PATCH 05/68] drm/i915/ctx: Return earlier on failure

[Ben Widawsky]

As what was correctly debugged here:
commit acc240d41ea1ab9c488a79219fb313b5b46265ae
Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date:   Thu Dec 5 15:42:34 2013 +0100

    drm/i915: Fix use-after-free in do_switch

It then becomes apparent that the default context cannot be the context
being switched to for context switch because it is always bound. It
follows that if the ring->last_context (from) has changed after the
bind_to_gtt, it will always be the default context - this is commented
in the code block.

This assertion will help catch issues without our logic sooner than
letting the system move long (which is possible for some time).

I really want this to be a BUG(), but I also want the patch to get
merged. I think the fact that none of the ERRNOs make any sense at all
is just more evidence that this shouldn't be a WARN.

//Cc: Ian Lister (don't have current email address)
Cc: Rafael Barbalho <rafael.barbalho@intel.com>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 0ce8fc9..34bf177 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -691,6 +691,15 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 	 */
 	from = ring->last_context;
 
+	/* The only context which 'from' can be, if it was changed, is the default
+	 * context. The default context cannot end up in evict everything (as
+	 * commented above) because it is always pinned.
+	 */
+	if (WARN_ON(from == to)) {
+		ret = -EPERM;
+		goto unpin_out;
+	}
+
 	if (needs_pd_load) {
 		/* Older GENs still want the load first, "PP_DCLV followed by
 		 * PP_DIR_BASE register through Load Register Immediate commands
-- 
2.0.4

[PATCH 06/68] drm/i915/error: vma error capture prettyify

[Ben Widawsky]

Rename some variables, and clean up the code a bit to make things
clearer in our error capture.

There isn't an intentional functional change here.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gpu_error.c | 55 ++++++++++++++++++++---------------
 1 file changed, 32 insertions(+), 23 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index eab41f9..470cc97 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -1032,63 +1032,72 @@ static void i915_gem_record_rings(struct drm_device *dev,
 	}
 }
 
-/* FIXME: Since pin count/bound list is global, we duplicate what we capture per
+/**
+ * i915_gem_capture_vm() - Capture a VMs error state.
+ * @error:	The main error structure
+ * @vm:		The address space we're capturing.
+ * @vm_ndx:	Which vm without the buffer array
+ *
+ * FIXME: Since pin count/bound list is global, we duplicate what we capture per
  * VM.
  */
 static void i915_gem_capture_vm(struct drm_i915_private *dev_priv,
 				struct drm_i915_error_state *error,
 				struct i915_address_space *vm,
-				const int ndx)
+				const int vm_ndx)
 {
 	struct drm_i915_error_buffer *active_bo = NULL, *pinned_bo = NULL;
 	struct drm_i915_gem_object *obj;
 	struct i915_vma *vma;
-	int i;
+	int active_vma_count = 0;
 
-	i = 0;
 	list_for_each_entry(vma, &vm->active_list, mm_list)
-		i++;
-	error->active_bo_count[ndx] = i;
+		active_vma_count++;
+
+	error->active_bo_count[vm_ndx] = active_vma_count;
+
 	list_for_each_entry(obj, &dev_priv->mm.bound_list, global_list)
 		if (i915_gem_obj_is_pinned(obj))
-			i++;
-	error->pinned_bo_count[ndx] = i - error->active_bo_count[ndx];
+			active_vma_count++;
+
+	/* XXX: this is an incorrect measurement of pinned BOs */
+	error->pinned_bo_count[vm_ndx] = active_vma_count - error->active_bo_count[vm_ndx];
 
-	if (i) {
-		active_bo = kcalloc(i, sizeof(*active_bo), GFP_ATOMIC);
+	if (active_vma_count) {
+		active_bo = kcalloc(active_vma_count, sizeof(*active_bo), GFP_ATOMIC);
 		if (active_bo)
-			pinned_bo = active_bo + error->active_bo_count[ndx];
+			pinned_bo = active_bo + error->active_bo_count[vm_ndx];
 	}
 
 	if (active_bo)
-		error->active_bo_count[ndx] =
+		error->active_bo_count[vm_ndx] =
 			capture_active_bo(active_bo,
-					  error->active_bo_count[ndx],
+					  error->active_bo_count[vm_ndx],
 					  &vm->active_list);
 
 	if (pinned_bo)
-		error->pinned_bo_count[ndx] =
+		error->pinned_bo_count[vm_ndx] =
 			capture_pinned_bo(pinned_bo,
-					  error->pinned_bo_count[ndx],
+					  error->pinned_bo_count[vm_ndx],
 					  &dev_priv->mm.bound_list);
-	error->active_bo[ndx] = active_bo;
-	error->pinned_bo[ndx] = pinned_bo;
+	error->active_bo[vm_ndx] = active_bo;
+	error->pinned_bo[vm_ndx] = pinned_bo;
 }
 
 static void i915_gem_capture_buffers(struct drm_i915_private *dev_priv,
 				     struct drm_i915_error_state *error)
 {
 	struct i915_address_space *vm;
-	int cnt = 0, i = 0;
+	int vm_count = 0, i = 0;
 
 	list_for_each_entry(vm, &dev_priv->vm_list, global_link)
-		cnt++;
+		vm_count++;
 
-	error->active_bo = kcalloc(cnt, sizeof(*error->active_bo), GFP_ATOMIC);
-	error->pinned_bo = kcalloc(cnt, sizeof(*error->pinned_bo), GFP_ATOMIC);
-	error->active_bo_count = kcalloc(cnt, sizeof(*error->active_bo_count),
+	error->active_bo = kcalloc(vm_count, sizeof(*error->active_bo), GFP_ATOMIC);
+	error->pinned_bo = kcalloc(vm_count, sizeof(*error->pinned_bo), GFP_ATOMIC);
+	error->active_bo_count = kcalloc(vm_count, sizeof(*error->active_bo_count),
 					 GFP_ATOMIC);
-	error->pinned_bo_count = kcalloc(cnt, sizeof(*error->pinned_bo_count),
+	error->pinned_bo_count = kcalloc(vm_count, sizeof(*error->pinned_bo_count),
 					 GFP_ATOMIC);
 
 	list_for_each_entry(vm, &dev_priv->vm_list, global_link)
-- 
2.0.4

[PATCH 07/68] drm/i915/error: Do a better job of disambiguating VMAs

[Ben Widawsky]

Some of the original PPGTT patches in this area where unmerged, and this
left a lot of confusion in our error capture with regard to which vm/obj
we want to capture. There have been at least a couple of patches from
Chris, and myself to try to fix this up; so here is another shot. Nobody
running without full PPGTT is effected by this, and that is probably why
nobody has bothered to fix it yet.

Instead of using any of the global lists to find the VMAs we want to
capture, we use the union of the active, and the inactive list in the
VM. This allows us to replace our capture_bo with capture_vma, and know
all the VMAs we want to capture are valid.

I could have probably figured out a way to reuse mm_list. As we've had
bugs here before in the shrinker, I think the best way forward is to get
it working, and then optimize it later.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c   |  1 +
 drivers/gpu/drm/i915/i915_gem_gtt.h   |  2 ++
 drivers/gpu/drm/i915/i915_gpu_error.c | 39 ++++++++++++++++++++++-------------
 3 files changed, 28 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index b4b7cfd..f2ece5f 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -2105,6 +2105,7 @@ static struct i915_vma *__i915_gem_vma_create(struct drm_i915_gem_object *obj,
 		return ERR_PTR(-ENOMEM);
 
 	INIT_LIST_HEAD(&vma->vma_link);
+	INIT_LIST_HEAD(&vma->pin_capture_link);
 	INIT_LIST_HEAD(&vma->mm_list);
 	INIT_LIST_HEAD(&vma->exec_list);
 	vma->vm = vm;
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 666c938..379cf16ea 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -126,6 +126,8 @@ struct i915_vma {
 
 	struct list_head vma_link; /* Link in the object's VMA list */
 
+	struct list_head pin_capture_link; /* Link in the error capture */
+
 	/** This vma's place in the batchbuffer or on the eviction list */
 	struct list_head exec_list;
 
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index 470cc97..5827915 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -685,14 +685,14 @@ static u32 capture_active_bo(struct drm_i915_error_buffer *err,
 static u32 capture_pinned_bo(struct drm_i915_error_buffer *err,
 			     int count, struct list_head *head)
 {
-	struct drm_i915_gem_object *obj;
+	struct i915_vma *vma;
 	int i = 0;
 
-	list_for_each_entry(obj, head, global_list) {
-		if (!i915_gem_obj_is_pinned(obj))
+	list_for_each_entry(vma, head, pin_capture_link) {
+		if (!i915_gem_obj_is_pinned(vma->obj))
 			continue;
 
-		capture_bo(err++, obj);
+		capture_bo(err++, vma->obj);
 		if (++i == count)
 			break;
 	}
@@ -1047,21 +1047,32 @@ static void i915_gem_capture_vm(struct drm_i915_private *dev_priv,
 				const int vm_ndx)
 {
 	struct drm_i915_error_buffer *active_bo = NULL, *pinned_bo = NULL;
-	struct drm_i915_gem_object *obj;
 	struct i915_vma *vma;
 	int active_vma_count = 0;
+	int vma_pin_count = 0;
+	LIST_HEAD(pinned_vma);
 
-	list_for_each_entry(vma, &vm->active_list, mm_list)
+	list_for_each_entry(vma, &vm->active_list, mm_list) {
 		active_vma_count++;
+		if (vma->pin_count) {
+			vma_pin_count++;
+			list_move_tail(&vma->pin_capture_link, &pinned_vma);
+		}
+	}
 
-	error->active_bo_count[vm_ndx] = active_vma_count;
-
-	list_for_each_entry(obj, &dev_priv->mm.bound_list, global_list)
-		if (i915_gem_obj_is_pinned(obj))
-			active_vma_count++;
+	list_for_each_entry(vma, &vm->inactive_list, mm_list) {
+		/* Certain objects may be on the inactive list, but pinned, when
+		 * in the global GGTT. */
+		if (WARN_ON(!i915_is_ggtt(vm) &&
+			    vma->pin_count &&
+			    !(vma->exec_entry->flags & (1<<31)))) { /* FIXME: need the actual flag */
+			vma_pin_count++;
+			list_move_tail(&vma->pin_capture_link, &pinned_vma);
+		}
+	}
 
-	/* XXX: this is an incorrect measurement of pinned BOs */
-	error->pinned_bo_count[vm_ndx] = active_vma_count - error->active_bo_count[vm_ndx];
+	error->active_bo_count[vm_ndx] = active_vma_count;
+	error->pinned_bo_count[vm_ndx] = vma_pin_count;
 
 	if (active_vma_count) {
 		active_bo = kcalloc(active_vma_count, sizeof(*active_bo), GFP_ATOMIC);
@@ -1079,7 +1090,7 @@ static void i915_gem_capture_vm(struct drm_i915_private *dev_priv,
 		error->pinned_bo_count[vm_ndx] =
 			capture_pinned_bo(pinned_bo,
 					  error->pinned_bo_count[vm_ndx],
-					  &dev_priv->mm.bound_list);
+					  &pinned_vma);
 	error->active_bo[vm_ndx] = active_bo;
 	error->pinned_bo[vm_ndx] = pinned_bo;
 }
-- 
2.0.4

[PATCH 08/68] drm/i915/error: Capture vmas instead of BOs

[Ben Widawsky]

To follow up on the last patch, we can now capture the VMAs instead of
the BOs. The hope if we get more accurate error capture while debugging
PPGTT.

Note that this does not impact the previous argument about whether to
capture all VMAs, or just the guilty VMA. This merely allows the code to
do whatever we chose later.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h       |  1 +
 drivers/gpu/drm/i915/i915_gpu_error.c | 53 +++++++++++++++++++----------------
 2 files changed, 30 insertions(+), 24 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index fab97bc..4f217f3 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -317,6 +317,7 @@ struct drm_i915_error_state {
 	char error_msg[128];
 	u32 reset_count;
 	u32 suspend_count;
+	u32 vm_count;
 
 	/* Generic register state */
 	u32 eir;
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index 5827915..82508dd 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -393,15 +393,19 @@ int i915_error_state_to_str(struct drm_i915_error_state_buf *m,
 		i915_ring_error_state(m, dev, &error->ring[i]);
 	}
 
-	if (error->active_bo)
-		print_error_buffers(m, "Active",
-				    error->active_bo[0],
-				    error->active_bo_count[0]);
+	for (i = 0; i < error->vm_count; i++) {
+		if (error->active_bo[i])
+			print_error_buffers(m, "Active",
+					    error->active_bo[i],
+					    error->active_bo_count[i]);
+	}
 
-	if (error->pinned_bo)
-		print_error_buffers(m, "Pinned",
-				    error->pinned_bo[0],
-				    error->pinned_bo_count[0]);
+	for (i = 0; i < error->vm_count; i++) {
+		if (error->pinned_bo[i])
+			print_error_buffers(m, "Pinned",
+					    error->pinned_bo[i],
+					    error->pinned_bo_count[i]);
+	}
 
 	for (i = 0; i < ARRAY_SIZE(error->ring); i++) {
 		obj = error->ring[i].batchbuffer;
@@ -643,22 +647,23 @@ unwind:
 	i915_error_object_create_sized((dev_priv), (src), &(dev_priv)->gtt.base, \
 				       (src)->base.size>>PAGE_SHIFT)
 
-static void capture_bo(struct drm_i915_error_buffer *err,
-		       struct drm_i915_gem_object *obj)
+static void capture_vma(struct drm_i915_error_buffer *err, struct i915_vma *vma)
 {
+	struct drm_i915_gem_object *obj = vma->obj;
+
 	err->size = obj->base.size;
 	err->name = obj->base.name;
 	err->rseqno = obj->last_read_seqno;
 	err->wseqno = obj->last_write_seqno;
-	err->gtt_offset = i915_gem_obj_ggtt_offset(obj);
+	err->gtt_offset = vma->node.start;
 	err->read_domains = obj->base.read_domains;
 	err->write_domain = obj->base.write_domain;
 	err->fence_reg = obj->fence_reg;
-	err->pinned = 0;
-	if (i915_gem_obj_is_pinned(obj))
-		err->pinned = 1;
-	if (obj->user_pin_count > 0)
+	if (obj->user_pin_count > 0) {
+		WARN_ON(i915_is_ggtt(vma->vm));
 		err->pinned = -1;
+	} else
+		err->pinned = !!vma->pin_count;
 	err->tiling = obj->tiling_mode;
 	err->dirty = obj->dirty;
 	err->purgeable = obj->madv != I915_MADV_WILLNEED;
@@ -674,7 +679,7 @@ static u32 capture_active_bo(struct drm_i915_error_buffer *err,
 	int i = 0;
 
 	list_for_each_entry(vma, head, mm_list) {
-		capture_bo(err++, vma->obj);
+		capture_vma(err++, vma);
 		if (++i == count)
 			break;
 	}
@@ -689,10 +694,10 @@ static u32 capture_pinned_bo(struct drm_i915_error_buffer *err,
 	int i = 0;
 
 	list_for_each_entry(vma, head, pin_capture_link) {
-		if (!i915_gem_obj_is_pinned(vma->obj))
+		if (!vma->pin_count)
 			continue;
 
-		capture_bo(err++, vma->obj);
+		capture_vma(err++, vma);
 		if (++i == count)
 			break;
 	}
@@ -1099,16 +1104,16 @@ static void i915_gem_capture_buffers(struct drm_i915_private *dev_priv,
 				     struct drm_i915_error_state *error)
 {
 	struct i915_address_space *vm;
-	int vm_count = 0, i = 0;
+	int i = 0;
 
 	list_for_each_entry(vm, &dev_priv->vm_list, global_link)
-		vm_count++;
+		error->vm_count++;
 
-	error->active_bo = kcalloc(vm_count, sizeof(*error->active_bo), GFP_ATOMIC);
-	error->pinned_bo = kcalloc(vm_count, sizeof(*error->pinned_bo), GFP_ATOMIC);
-	error->active_bo_count = kcalloc(vm_count, sizeof(*error->active_bo_count),
+	error->active_bo = kcalloc(error->vm_count, sizeof(*error->active_bo), GFP_ATOMIC);
+	error->pinned_bo = kcalloc(error->vm_count, sizeof(*error->pinned_bo), GFP_ATOMIC);
+	error->active_bo_count = kcalloc(error->vm_count, sizeof(*error->active_bo_count),
 					 GFP_ATOMIC);
-	error->pinned_bo_count = kcalloc(vm_count, sizeof(*error->pinned_bo_count),
+	error->pinned_bo_count = kcalloc(error->vm_count, sizeof(*error->pinned_bo_count),
 					 GFP_ATOMIC);
 
 	list_for_each_entry(vm, &dev_priv->vm_list, global_link)
-- 
2.0.4

[PATCH 09/68] drm/i915: Add some extra guards in evict_vm

[Ben Widawsky]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_evict.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_evict.c b/drivers/gpu/drm/i915/i915_gem_evict.c
index bbf4b12..38297d3 100644
--- a/drivers/gpu/drm/i915/i915_gem_evict.c
+++ b/drivers/gpu/drm/i915/i915_gem_evict.c
@@ -214,6 +214,7 @@ int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle)
 	struct i915_vma *vma, *next;
 	int ret;
 
+	BUG_ON(!mutex_is_locked(&vm->dev->struct_mutex));
 	trace_i915_gem_evict_vm(vm);
 
 	if (do_idle) {
@@ -222,11 +223,15 @@ int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle)
 			return ret;
 
 		i915_gem_retire_requests(vm->dev);
+
+		WARN_ON(!list_empty(&vm->active_list));
 	}
 
-	list_for_each_entry_safe(vma, next, &vm->inactive_list, mm_list)
+	list_for_each_entry_safe(vma, next, &vm->inactive_list, mm_list) {
+		WARN_ON(!i915_is_ggtt(vm) && vma->pin_count);
 		if (vma->pin_count == 0)
 			WARN_ON(i915_vma_unbind(vma));
+	}
 
 	return 0;
 }
-- 
2.0.4

[PATCH 10/68] drm/i915: Make an uninterruptible evict

[Ben Widawsky]

There are no users of this yet, but the idea is presented and split out
to find bugs.

Also, while here, return -ERESTARTSYS to the caller, in case they want
to do something with it.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h            |  2 +-
 drivers/gpu/drm/i915/i915_gem_context.c    |  5 ++---
 drivers/gpu/drm/i915/i915_gem_evict.c      | 32 ++++++++++++++++++++++--------
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  2 +-
 4 files changed, 28 insertions(+), 13 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 4f217f3..c5c8753 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2553,7 +2553,7 @@ int __must_check i915_gem_evict_something(struct drm_device *dev,
 					  unsigned long start,
 					  unsigned long end,
 					  unsigned flags);
-int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle);
+int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle, bool interruptible);
 int i915_gem_evict_everything(struct drm_device *dev);
 
 /* belongs in i915_gem_gtt.h */
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 34bf177..4d47bcb 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -121,11 +121,10 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 			if (WARN_ON(list_empty(&vma->vma_link) ||
 				    list_is_singular(&vma->vma_link)))
 				break;
-
-		i915_gem_evict_vm(&ppgtt->base, true);
+		i915_gem_evict_vm(&ppgtt->base, true, true);
 	} else {
 		i915_gem_retire_requests(dev);
-		i915_gem_evict_vm(&ppgtt->base, false);
+		i915_gem_evict_vm(&ppgtt->base, false, true);
 	}
 
 	ppgtt->base.cleanup(&ppgtt->base);
diff --git a/drivers/gpu/drm/i915/i915_gem_evict.c b/drivers/gpu/drm/i915/i915_gem_evict.c
index 38297d3..ac8d90f 100644
--- a/drivers/gpu/drm/i915/i915_gem_evict.c
+++ b/drivers/gpu/drm/i915/i915_gem_evict.c
@@ -197,8 +197,9 @@ found:
 /**
  * i915_gem_evict_vm - Evict all idle vmas from a vm
  *
- * @vm: Address space to cleanse
- * @do_idle: Boolean directing whether to idle first.
+ * @vm:			Address space to cleanse
+ * @do_idle:		Boolean directing whether to idle first.
+ * @interruptible:	How to wait
  *
  * This function evicts all idles vmas from a vm. If all unpinned vmas should be
  * evicted the @do_idle needs to be set to true.
@@ -209,18 +210,24 @@ found:
  * To clarify: This is for freeing up virtual address space, not for freeing
  * memory in e.g. the shrinker.
  */
-int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle)
+int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle, bool interruptible)
 {
+	struct drm_i915_private *dev_priv = to_i915(vm->dev);
+
 	struct i915_vma *vma, *next;
+	bool was_intr = dev_priv->mm.interruptible;
 	int ret;
 
 	BUG_ON(!mutex_is_locked(&vm->dev->struct_mutex));
 	trace_i915_gem_evict_vm(vm);
 
+	if (!interruptible)
+		dev_priv->mm.interruptible = false;
+
 	if (do_idle) {
 		ret = i915_gpu_idle(vm->dev);
 		if (ret)
-			return ret;
+			goto out;
 
 		i915_gem_retire_requests(vm->dev);
 
@@ -229,11 +236,20 @@ int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle)
 
 	list_for_each_entry_safe(vma, next, &vm->inactive_list, mm_list) {
 		WARN_ON(!i915_is_ggtt(vm) && vma->pin_count);
-		if (vma->pin_count == 0)
-			WARN_ON(i915_vma_unbind(vma));
+		if (vma->pin_count == 0) {
+			ret = i915_vma_unbind(vma);
+			if (ret == -ERESTARTSYS) {
+				BUG_ON(!interruptible);
+				goto out;
+			} else if (ret)
+				WARN(1, "Failed to unbind vma %d\n", ret);
+		}
 	}
+	ret = 0;
 
-	return 0;
+out:
+	dev_priv->mm.interruptible = was_intr;
+	return ret;
 }
 
 /**
@@ -276,7 +292,7 @@ i915_gem_evict_everything(struct drm_device *dev)
 
 	/* Having flushed everything, unbind() should never raise an error */
 	list_for_each_entry(vm, &dev_priv->vm_list, global_link)
-		WARN_ON(i915_gem_evict_vm(vm, false));
+		WARN_ON(i915_gem_evict_vm(vm, false, true));
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 60998fc..1420aeb 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -722,7 +722,7 @@ err:
 		list_for_each_entry(vma, vmas, exec_list)
 			i915_gem_execbuffer_unreserve_vma(vma);
 
-		ret = i915_gem_evict_vm(vm, true);
+		ret = i915_gem_evict_vm(vm, true, true);
 		if (ret)
 			return ret;
 	} while (1);
-- 
2.0.4

[PATCH 11/68] drm/i915: More correct (slower) ppgtt cleanup

[Ben Widawsky]

If a VM still have objects which are bound (exactly: have a node
reserved in the drm_mm), and we are in the middle of a reset, we have no
hope of the standard methods fixing the situation (ring idle won't
work). We must therefore let the reset handler take it's course, and
then we can resume tearing down the VM.

This logic very much duplicates^Wresembles the logic in our wait for
error code. I've decided to leave it as open coded because I expect this
bit of code to require tweaks and changes over time.

Interruptions via signal causes a really similar problem.

This should deprecate the need for the yet unmerged patch from Chris
(and an identical patch from me, which was first!!):
drm/i915: Prevent signals from interrupting close()

I have a followup patch to implement deferred free, before you complain.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 51 +++++++++++++++++++++++++++++++--
 1 file changed, 48 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 4d47bcb..e48a3bb 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -101,6 +101,32 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 	struct drm_device *dev = ppgtt->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct i915_address_space *vm = &ppgtt->base;
+	bool do_idle = false;
+	int ret;
+
+	/* If we get here while in reset, we need to let the reset handler run
+	 * first, or else our VM teardown isn't going to go smoothly. There are
+	 * a could of options at this point, but letting the reset handler do
+	 * it's thing is the most desirable. The reset handler will take care of
+	 * retiring the stuck requests.
+	 */
+	if (i915_reset_in_progress(&dev_priv->gpu_error)) {
+		mutex_unlock(&dev->struct_mutex);
+#define EXIT_COND (!i915_reset_in_progress(&dev_priv->gpu_error) || \
+		   i915_terminally_wedged(&dev_priv->gpu_error))
+		ret = wait_event_timeout(dev_priv->gpu_error.reset_queue,
+					 EXIT_COND,
+					 10 * HZ);
+		if (!ret) {
+			/* it's unlikely idling will solve anything, but it
+			 * shouldn't hurt to try. */
+			do_idle = true;
+			/* TODO: go down kicking and screaming harder */
+		}
+#undef EXIT_COND
+
+		mutex_lock(&dev->struct_mutex);
+	}
 
 	if (ppgtt == dev_priv->mm.aliasing_ppgtt ||
 	    (list_empty(&vm->active_list) && list_empty(&vm->inactive_list))) {
@@ -117,14 +143,33 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 	if (!list_empty(&vm->active_list)) {
 		struct i915_vma *vma;
 
+		do_idle = true;
 		list_for_each_entry(vma, &vm->active_list, mm_list)
 			if (WARN_ON(list_empty(&vma->vma_link) ||
 				    list_is_singular(&vma->vma_link)))
 				break;
-		i915_gem_evict_vm(&ppgtt->base, true, true);
-	} else {
+	} else
 		i915_gem_retire_requests(dev);
-		i915_gem_evict_vm(&ppgtt->base, false, true);
+
+	/* We have a problem here where VM teardown cannot be interrupted, or
+	 * else the ppgtt cleanup will fail. As an example, a precisely timed
+	 * SIGKILL could leads to an OOPS, or worse. There are two options:
+	 * 1. Make the eviction uninterruptible
+	 * 2. Defer the eviction if it was interrupted.
+	 *
+	 * Option #1 is not the friendliest, but it's the easiest to implement,
+	 * and least error prone.
+	 * TODO: Implement option 2
+	 */
+	ret = i915_gem_evict_vm(&ppgtt->base, do_idle, !do_idle);
+	if (ret == -ERESTARTSYS)
+		ret = i915_gem_evict_vm(&ppgtt->base, do_idle, false);
+	WARN_ON(ret);
+	WARN_ON(!list_empty(&vm->active_list));
+
+	/* This is going to blow up badly if the mm is unclean */
+	if (WARN_ON(!list_empty(&ppgtt->base.mm.head_node.node_list))) {
+		/* TODO: go down kicking and screaming harder++ */
 	}
 
 	ppgtt->base.cleanup(&ppgtt->base);
-- 
2.0.4

[PATCH 12/68] drm/i915: Defer PPGTT cleanup

[Ben Widawsky]

The last patch made PPGTT free cases correct. It left a major problem
though where in many cases it was possible to have to idle the GPU in
order to destroy a VM. This is really unfortunate as it is stalling the
active GPU process for the dying GPU process.

The workqueue grew very tricky. I left in my original wait based version
as #if 0, and used Chris' recommendation with the seqno check. I haven't
measure one vs the other, but I am in favor of the code as it is. I am
just leaving the old code for people to observe.

NOTE: I don't expect this patch to be merged as the thing it fixes will
be superseded by the PPGTT refcounting. However, at this time I do not
have a working and tested solution, other than mine. So it's still here.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h         |  10 +++
 drivers/gpu/drm/i915/i915_gem.c         | 110 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_context.c |  40 ++++++++----
 drivers/gpu/drm/i915/i915_gem_gtt.c     |   2 +-
 drivers/gpu/drm/i915/i915_gem_gtt.h     |   2 +
 5 files changed, 150 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index c5c8753..56d193e 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1129,6 +1129,15 @@ struct i915_gem_mm {
 	struct delayed_work idle_work;
 
 	/**
+	 * PPGTT freeing often happens during interruptible times (fd close,
+	 * execbuf, busy_ioctl). It therefore becomes difficult to clean up the
+	 * PPGTT when the refcount reaches 0 if a signal comes in. This
+	 * workqueue defers the cleanup of the VM to a later time, and allows
+	 * userspace to continue on.
+	 */
+	struct delayed_work ppgtt_work;
+
+	/**
 	 * Are we in a non-interruptible section of code like
 	 * modesetting?
 	 */
@@ -1520,6 +1529,7 @@ struct drm_i915_private {
 	struct mutex modeset_restore_lock;
 
 	struct list_head vm_list; /* Global list of all address spaces */
+	struct list_head ppgtt_free_list;
 	struct i915_gtt gtt; /* VM representing the global address space */
 
 	struct i915_gem_mm mm;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 20743bd..4ad2205 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2707,6 +2707,112 @@ i915_gem_idle_work_handler(struct work_struct *work)
 	intel_mark_idle(dev_priv->dev);
 }
 
+static void
+i915_gem_ppgtt_work_handler(struct work_struct *work)
+{
+	struct drm_i915_private *dev_priv =
+		container_of(work, typeof(*dev_priv), mm.ppgtt_work.work);
+	struct i915_address_space *vm, *v;
+#if 0
+	unsigned reset_counter;
+	int ret;
+#endif
+
+	if (!mutex_trylock(&dev_priv->dev->struct_mutex)) {
+		queue_delayed_work(dev_priv->wq, &dev_priv->mm.ppgtt_work,
+				   round_jiffies_up_relative(HZ));
+		return;
+	}
+
+	list_for_each_entry_safe(vm, v, &dev_priv->ppgtt_free_list, global_link) {
+		struct i915_hw_ppgtt *ppgtt = container_of(vm, struct i915_hw_ppgtt, base);
+		struct i915_vma *vma = NULL, *vma_active = NULL, *vma_inactive = NULL;
+
+		/* The following attempts to find the newest (most recently
+		 * activated/inactivated) VMA.
+		 */
+		if (!list_empty(&ppgtt->base.active_list)) {
+			vma_active = list_last_entry(&ppgtt->base.active_list,
+						     typeof(*vma), mm_list);
+		}
+		if (!list_empty(&ppgtt->base.inactive_list)) {
+			vma_inactive = list_last_entry(&ppgtt->base.inactive_list,
+						       typeof(*vma), mm_list);
+		}
+
+		if (vma_active)
+			vma = vma_active;
+		else if (vma_inactive)
+			vma = vma_inactive;
+
+		/* Sanity check */
+		if (vma_active && vma_inactive) {
+			if (WARN_ON(vma_active->retire_seqno <= vma_inactive->retire_seqno))
+				vma = vma_inactive;
+		}
+
+		if (!vma)
+			goto finish;
+
+		/* Another sanity check */
+		if (WARN_ON(!vma->retire_seqno))
+			continue;
+
+		/* NOTE: We could wait here for the seqno, but that makes things
+		 * significantly more complex. */
+		if (!i915_seqno_passed(vma->retire_ring->get_seqno(vma->retire_ring, true), vma->retire_seqno))
+			break;
+
+#if 0
+		reset_counter = atomic_read(&dev_priv->gpu_error.reset_counter);
+		mutex_unlock(&dev_priv->dev->struct_mutex);
+		ret = __wait_seqno(vma->retire_ring, vma->retire_seqno,
+				   reset_counter, true, NULL, NULL);
+
+		if (i915_mutex_lock_interruptible(dev_priv->dev)) {
+			queue_delayed_work(dev_priv->wq, &dev_priv->mm.ppgtt_work,
+					   round_jiffies_up_relative(HZ));
+			return;
+		}
+
+		if (ret)
+			continue;
+
+#endif
+finish:
+		/* If the object was destroyed while our VMA was dangling, the
+		 * object free code did the synchronous cleanup for us.
+		 * Therefore every node on this list should have a living object
+		 * (and VMA), but let's try not to use it in case future code
+		 * allows a VMA to outlive an object.
+		 */
+		while (!list_empty(&ppgtt->base.mm.head_node.node_list)) {
+			struct drm_mm_node *node;
+			struct i915_vma *vma;
+			struct drm_i915_gem_object *obj;
+
+			node = list_first_entry(&ppgtt->base.mm.head_node.node_list,
+						struct drm_mm_node,
+						node_list);
+			vma = container_of(node, struct i915_vma, node);
+			obj = vma->obj;
+
+			drm_mm_remove_node(node);
+			i915_gem_vma_destroy(vma);
+			i915_gem_object_unpin_pages(obj);
+		}
+
+		ppgtt->base.cleanup(&ppgtt->base);
+		kfree(ppgtt);
+	}
+
+	if (!list_empty(&dev_priv->ppgtt_free_list))
+		queue_delayed_work(dev_priv->wq, &dev_priv->mm.ppgtt_work,
+				   round_jiffies_up_relative(HZ));
+
+	mutex_unlock(&dev_priv->dev->struct_mutex);
+}
+
 /**
  * Ensures that an object will eventually get non-busy by flushing any required
  * write domains, emitting any outstanding lazy request and retiring and
@@ -4553,6 +4659,7 @@ i915_gem_suspend(struct drm_device *dev)
 	mutex_unlock(&dev->struct_mutex);
 
 	del_timer_sync(&dev_priv->gpu_error.hangcheck_timer);
+	cancel_delayed_work_sync(&dev_priv->mm.ppgtt_work);
 	cancel_delayed_work_sync(&dev_priv->mm.retire_work);
 	flush_delayed_work(&dev_priv->mm.idle_work);
 
@@ -4894,6 +5001,7 @@ i915_gem_load(struct drm_device *dev)
 				  NULL);
 
 	INIT_LIST_HEAD(&dev_priv->vm_list);
+	INIT_LIST_HEAD(&dev_priv->ppgtt_free_list);
 	i915_init_vm(dev_priv, &dev_priv->gtt.base);
 
 	INIT_LIST_HEAD(&dev_priv->context_list);
@@ -4908,6 +5016,8 @@ i915_gem_load(struct drm_device *dev)
 			  i915_gem_retire_work_handler);
 	INIT_DELAYED_WORK(&dev_priv->mm.idle_work,
 			  i915_gem_idle_work_handler);
+	INIT_DELAYED_WORK(&dev_priv->mm.ppgtt_work,
+			  i915_gem_ppgtt_work_handler);
 	init_waitqueue_head(&dev_priv->gpu_error.reset_queue);
 
 	/* On GEN3 we really need to make sure the ARB C3 LP bit is set */
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index e48a3bb..61b36f9 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -96,7 +96,7 @@
 #define GEN6_CONTEXT_ALIGN (64<<10)
 #define GEN7_CONTEXT_ALIGN 4096
 
-static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
+static int do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 {
 	struct drm_device *dev = ppgtt->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -131,7 +131,7 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 	if (ppgtt == dev_priv->mm.aliasing_ppgtt ||
 	    (list_empty(&vm->active_list) && list_empty(&vm->inactive_list))) {
 		ppgtt->base.cleanup(&ppgtt->base);
-		return;
+		return 0;
 	}
 
 	/*
@@ -153,17 +153,30 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 
 	/* We have a problem here where VM teardown cannot be interrupted, or
 	 * else the ppgtt cleanup will fail. As an example, a precisely timed
-	 * SIGKILL could leads to an OOPS, or worse. There are two options:
-	 * 1. Make the eviction uninterruptible
-	 * 2. Defer the eviction if it was interrupted.
-	 *
-	 * Option #1 is not the friendliest, but it's the easiest to implement,
-	 * and least error prone.
-	 * TODO: Implement option 2
+	 * SIGKILL could leads to an OOPS, or worse. The real solution is to
+	 * properly track the VMA <-> OBJ relationship. This temporary bandaid
+	 * will simply defer the free until we know the seqno has passed for
+	 * this VMA.
 	 */
 	ret = i915_gem_evict_vm(&ppgtt->base, do_idle, !do_idle);
-	if (ret == -ERESTARTSYS)
-		ret = i915_gem_evict_vm(&ppgtt->base, do_idle, false);
+	if (ret == -ERESTARTSYS) {
+		struct drm_mm_node *entry;
+		/* First mark all VMAs */
+		drm_mm_for_each_node(entry, &ppgtt->base.mm) {
+			struct i915_vma *vma = container_of(entry, struct i915_vma, node);
+			vma->retire_seqno = vma->obj->last_read_seqno;
+			vma->retire_ring = vma->obj->ring;
+			/* It's okay to lose the object, we just can't lose the
+			 * VMA */
+		}
+
+		list_move_tail(&ppgtt->base.global_link, &dev_priv->ppgtt_free_list);
+		queue_delayed_work(dev_priv->wq,
+				   &dev_priv->mm.ppgtt_work,
+				   round_jiffies_up_relative(HZ));
+		return -EAGAIN;
+	}
+
 	WARN_ON(ret);
 	WARN_ON(!list_empty(&vm->active_list));
 
@@ -173,6 +186,7 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 	}
 
 	ppgtt->base.cleanup(&ppgtt->base);
+	return 0;
 }
 
 static void ppgtt_release(struct kref *kref)
@@ -180,8 +194,8 @@ static void ppgtt_release(struct kref *kref)
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(kref, struct i915_hw_ppgtt, ref);
 
-	do_ppgtt_cleanup(ppgtt);
-	kfree(ppgtt);
+	if (!do_ppgtt_cleanup(ppgtt))
+		kfree(ppgtt);
 }
 
 static size_t get_context_alignment(struct drm_device *dev)
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index f2ece5f..254895d 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1019,7 +1019,7 @@ static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
 		container_of(vm, struct i915_hw_ppgtt, base);
 
 	list_del(&vm->global_link);
-	drm_mm_takedown(&ppgtt->base.mm);
+	drm_mm_takedown(&vm->mm);
 	drm_mm_remove_node(&ppgtt->node);
 
 	gen6_ppgtt_unmap_pages(ppgtt);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 379cf16ea..593b5d0 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -137,6 +137,8 @@ struct i915_vma {
 	struct hlist_node exec_node;
 	unsigned long exec_handle;
 	struct drm_i915_gem_exec_object2 *exec_entry;
+	uint32_t retire_seqno; /* Last active seqno at context desruction */
+	struct intel_engine_cs *retire_ring; /* Last ring for retire_seqno */
 
 	/**
 	 * How many users have pinned this object in GTT space. The following
-- 
2.0.4

[PATCH 13/68] drm/i915/bdw: Enable full PPGTT

[Ben Widawsky]

Broadwell is perfectly capable of full PPGTT. I've been using it for
some time, and seen no especially ill effects.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>

Conflicts:
	drivers/gpu/drm/i915/i915_drv.h
---
 drivers/gpu/drm/i915/i915_drv.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 56d193e..02d81b03 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2075,7 +2075,7 @@ struct drm_i915_cmd_table {
 
 #define HAS_HW_CONTEXTS(dev)	(INTEL_INFO(dev)->gen >= 6)
 #define HAS_ALIASING_PPGTT(dev)	(INTEL_INFO(dev)->gen >= 6)
-#define HAS_PPGTT(dev)		(INTEL_INFO(dev)->gen >= 7 && !IS_GEN8(dev))
+#define HAS_PPGTT(dev)		(INTEL_INFO(dev)->gen >= 7 && !IS_CHERRYVIEW(dev))
 #define USES_PPGTT(dev)		(i915.enable_ppgtt)
 #define USES_FULL_PPGTT(dev)	(i915.enable_ppgtt == 2)
 
-- 
2.0.4

[PATCH 14/68] drm/i915: Get the error state over the wire (HACKish)

[Ben Widawsky]

I was dealing with a bug recently where the system would hard hang
somewhere between hangcheck and reset. There was time after error
collection to actually get my error state out, but I couldn't get the
reads to work.

This patch is also useful for when reset kills the machine, and you want
to keep reset enabled but still get error state.

Since I found the patch pretty useful, I decided to clean it up and
submit it. It was mostly meant as a one-off hack originally though.

If a maintainer decides it's useful, then here it is.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c   |  2 +-
 drivers/gpu/drm/i915/i915_drv.h       |  3 ++-
 drivers/gpu/drm/i915/i915_gpu_error.c | 31 +++++++++++++++++++++++++------
 drivers/gpu/drm/i915/i915_sysfs.c     |  2 +-
 4 files changed, 29 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 330caa1..16ae700 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -931,7 +931,7 @@ static ssize_t i915_error_state_read(struct file *file, char __user *userbuf,
 	if (ret)
 		return ret;
 
-	ret = i915_error_state_to_str(&error_str, error_priv);
+	ret = i915_error_state_to_str(&error_str, error_priv->dev, error_priv->error);
 	if (ret)
 		goto out;
 
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 02d81b03..04c9e2c 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2619,7 +2619,8 @@ static inline void intel_display_crc_init(struct drm_device *dev) {}
 __printf(2, 3)
 void i915_error_printf(struct drm_i915_error_state_buf *e, const char *f, ...);
 int i915_error_state_to_str(struct drm_i915_error_state_buf *estr,
-			    const struct i915_error_state_file_priv *error);
+			    struct drm_device *dev,
+			    const struct drm_i915_error_state *error);
 int i915_error_state_buf_init(struct drm_i915_error_state_buf *eb,
 			      size_t count, loff_t pos);
 static inline void i915_error_state_buf_release(
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index 82508dd..c391268 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -184,8 +184,22 @@ static void i915_error_puts(struct drm_i915_error_state_buf *e,
 	__i915_error_advance(e, len);
 }
 
-#define err_printf(e, ...) i915_error_printf(e, __VA_ARGS__)
-#define err_puts(e, s) i915_error_puts(e, s)
+
+static bool wire = false;
+#define err_printf(e, ...) do {				\
+	if (wire) {					\
+		printk(__VA_ARGS__);			\
+	} else {					\
+		i915_error_printf(e, __VA_ARGS__);	\
+	}						\
+} while (0)
+#define err_puts(e, s) do {				\
+	if (wire) {					\
+		printk(s);				\
+	} else {					\
+		i915_error_puts(e, s);			\
+	}						\
+} while (0)
 
 static void print_error_buffers(struct drm_i915_error_state_buf *m,
 				const char *name,
@@ -242,7 +256,7 @@ static const char *hangcheck_action_to_str(enum intel_ring_hangcheck_action a)
 
 static void i915_ring_error_state(struct drm_i915_error_state_buf *m,
 				  struct drm_device *dev,
-				  struct drm_i915_error_ring *ring)
+				  const struct drm_i915_error_ring *ring)
 {
 	if (!ring->valid)
 		return;
@@ -324,11 +338,10 @@ static void print_error_obj(struct drm_i915_error_state_buf *m,
 }
 
 int i915_error_state_to_str(struct drm_i915_error_state_buf *m,
-			    const struct i915_error_state_file_priv *error_priv)
+			    struct drm_device *dev,
+			    const struct drm_i915_error_state *error)
 {
-	struct drm_device *dev = error_priv->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct drm_i915_error_state *error = error_priv->error;
 	struct drm_i915_error_object *obj;
 	int i, j, offset, elt;
 	int max_hangcheck_score;
@@ -1266,6 +1279,12 @@ void i915_capture_error_state(struct drm_device *dev, bool wedged,
 	spin_lock_irqsave(&dev_priv->gpu_error.lock, flags);
 	if (dev_priv->gpu_error.first_error == NULL) {
 		dev_priv->gpu_error.first_error = error;
+#ifdef PUSH_TO_WIRE
+		/* Probably racy, but this is emergency debug */
+		wire = true;
+		i915_error_state_to_str(NULL, dev, error);
+		wire = false;
+#endif
 		error = NULL;
 	}
 	spin_unlock_irqrestore(&dev_priv->gpu_error.lock, flags);
diff --git a/drivers/gpu/drm/i915/i915_sysfs.c b/drivers/gpu/drm/i915/i915_sysfs.c
index ae7fd8f..b559781 100644
--- a/drivers/gpu/drm/i915/i915_sysfs.c
+++ b/drivers/gpu/drm/i915/i915_sysfs.c
@@ -547,7 +547,7 @@ static ssize_t error_state_read(struct file *filp, struct kobject *kobj,
 	error_priv.dev = dev;
 	i915_error_state_get(dev, &error_priv);
 
-	ret = i915_error_state_to_str(&error_str, &error_priv);
+	ret = i915_error_state_to_str(&error_str, dev, error_priv.error);
 	if (ret)
 		goto out;
 
-- 
2.0.4

[PATCH 15/68] drm/i915/gen8: Invalidate TLBs before PDP reload

[Ben Widawsky]

This is a spec requirement for all rings.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 61b36f9..ef256ae 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -892,6 +892,9 @@ int i915_switch_context(struct intel_engine_cs *ring,
 	if (from == to && !to->remap_slice)
 		return 0;
 
+	if (IS_GEN8(ring->dev))
+		WARN_ON(ring->flush(ring, I915_GEM_GPU_DOMAINS, 0));
+
 	if (ring->id == RCS)
 		return do_switch_rcs(ring, from, to);
 	else
-- 
2.0.4

[PATCH 16/68] drm/i915: Remove false assertion in ppgtt_release

[Ben Widawsky]

Originally the thought for the assertion was that if there are no real
VMAs (died during execbuf), or there is only 1 VMA, but the VMA is on
the active list, it's a bug. The former case is pretty obvious. The
later case simply meant to assert the context unref/object retire
interactions were working properly

There is a flaw in the logic of the second when an object has multiple
VMAs. If there are multiple VMAs, it's possible that the object
continually had it's seqno increased as it was used by another context.
In this case, the context ref will die, but the VMA will not be taking
off the active list because of the missing retire seqno for a VMA.

Like some of the other fixes I've submitted recently, this should be
fixed by the eventual work Daniel will do.

This is pretty easy to reproduce whenever mesa uses the blit engine.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index ef256ae..a5c7d5d 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -145,8 +145,7 @@ static int do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 
 		do_idle = true;
 		list_for_each_entry(vma, &vm->active_list, mm_list)
-			if (WARN_ON(list_empty(&vma->vma_link) ||
-				    list_is_singular(&vma->vma_link)))
+			if (WARN_ON(list_empty(&vma->vma_link)))
 				break;
 	} else
 		i915_gem_retire_requests(dev);
-- 
2.0.4

[PATCH 17/68] Revert "drm/i915/bdw: Use timeout mode for RC6 on bdw"

[Ben Widawsky]

This reverts commit 0d68b25e9ceb344fe2f93373b1c0311d33814265.

At one time I bisected reset breakage to this patch by using a mesa that is
guaranteed to generate a hang when using the fs, and then running the following
test case:

./bin/shader_runner  tests/shaders/glsl-algebraic-add-zero.shader_test -auto
./bin/shader_runner  tests/shaders/glsl-fs-texture2d.shader_test -auto
./bin/shader_runner  tests/shaders/glsl-algebraic-add-zero.shader_test -auto

The symptom is that after the first GPU hang, all subsequent commands
will hang the GPU.

Oddly at some point I believe this revert stopped fixing the issue, but I am
leaving it in the series to minimize variables.

---
 drivers/gpu/drm/i915/intel_pm.c | 16 ++++------------
 1 file changed, 4 insertions(+), 12 deletions(-)

diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 41de760..3255c10 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -3658,23 +3658,15 @@ static void gen8_enable_rps(struct drm_device *dev)
 	for_each_ring(ring, dev_priv, unused)
 		I915_WRITE(RING_MAX_IDLE(ring->mmio_base), 10);
 	I915_WRITE(GEN6_RC_SLEEP, 0);
-	if (IS_BROADWELL(dev))
-		I915_WRITE(GEN6_RC6_THRESHOLD, 625); /* 800us/1.28 for TO */
-	else
-		I915_WRITE(GEN6_RC6_THRESHOLD, 50000); /* 50/125ms per EI */
+	I915_WRITE(GEN6_RC6_THRESHOLD, 50000); /* 50/125ms per EI */
 
 	/* 3: Enable RC6 */
 	if (intel_enable_rc6(dev) & INTEL_RC6_ENABLE)
 		rc6_mask = GEN6_RC_CTL_RC6_ENABLE;
 	intel_print_rc6_info(dev, rc6_mask);
-	if (IS_BROADWELL(dev))
-		I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
-				GEN7_RC_CTL_TO_MODE |
-				rc6_mask);
-	else
-		I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
-				GEN6_RC_CTL_EI_MODE(1) |
-				rc6_mask);
+	I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
+				    GEN6_RC_CTL_EI_MODE(1) |
+				    rc6_mask);
 
 	/* 4 Program defaults and thresholds for RPS*/
 	I915_WRITE(GEN6_RPNSWREQ,
-- 
2.0.4

[PATCH 18/68] drm/i915/trace: Fix offsets for 64b

[Ben Widawsky]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_trace.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index f5aa006..cbf5521 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -115,7 +115,7 @@ TRACE_EVENT(i915_vma_bind,
 	    TP_STRUCT__entry(
 			     __field(struct drm_i915_gem_object *, obj)
 			     __field(struct i915_address_space *, vm)
-			     __field(u32, offset)
+			     __field(u64, offset)
 			     __field(u32, size)
 			     __field(unsigned, flags)
 			     ),
@@ -128,7 +128,7 @@ TRACE_EVENT(i915_vma_bind,
 			   __entry->flags = flags;
 			   ),
 
-	    TP_printk("obj=%p, offset=%08x size=%x%s vm=%p",
+	    TP_printk("obj=%p, offset=%016llx size=%x%s vm=%p",
 		      __entry->obj, __entry->offset, __entry->size,
 		      __entry->flags & PIN_MAPPABLE ? ", mappable" : "",
 		      __entry->vm)
@@ -141,7 +141,7 @@ TRACE_EVENT(i915_vma_unbind,
 	    TP_STRUCT__entry(
 			     __field(struct drm_i915_gem_object *, obj)
 			     __field(struct i915_address_space *, vm)
-			     __field(u32, offset)
+			     __field(u64, offset)
 			     __field(u32, size)
 			     ),
 
@@ -152,7 +152,7 @@ TRACE_EVENT(i915_vma_unbind,
 			   __entry->size = vma->node.size;
 			   ),
 
-	    TP_printk("obj=%p, offset=%08x size=%x vm=%p",
+	    TP_printk("obj=%p, offset=%016llx size=%x vm=%p",
 		      __entry->obj, __entry->offset, __entry->size, __entry->vm)
 );
 
-- 
2.0.4

[PATCH 19/68] drm/i915: Wrap VMA binding

[Ben Widawsky]

This will be useful for some upcoming patches which do more platform
specific work. Having it in one central place just makes things a bit
cleaner and easier.

NOTE: I didn't actually end up using this patch for the intended
purpose, but I thought it was a nice patch to keep around.

v2: s/i915_gem_bind_vma/i915_gem_vma_bind/
s/i915_gem_unbind_vma/i915_gem_vma_unbind/
(Chris)

v3: Missed one spot

v4: Don't change the trace events (Daniel)

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h            |  3 +++
 drivers/gpu/drm/i915/i915_gem.c            | 12 ++++++------
 drivers/gpu/drm/i915/i915_gem_context.c    |  2 +-
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  5 +++--
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 13 ++++++++++++-
 5 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 04c9e2c..d1750d5 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2461,6 +2461,9 @@ bool i915_gem_obj_bound(struct drm_i915_gem_object *o,
 			struct i915_address_space *vm);
 unsigned long i915_gem_obj_size(struct drm_i915_gem_object *o,
 				struct i915_address_space *vm);
+void i915_gem_vma_bind(struct i915_vma *vma, enum i915_cache_level,
+		       unsigned flags);
+void i915_gem_vma_unbind(struct i915_vma *vma);
 struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,
 				     struct i915_address_space *vm);
 struct i915_vma *
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 4ad2205..5f66939 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3031,7 +3031,7 @@ int i915_vma_unbind(struct i915_vma *vma)
 
 	trace_i915_vma_unbind(vma);
 
-	vma->unbind_vma(vma);
+	i915_gem_vma_unbind(vma);
 
 	list_del_init(&vma->mm_list);
 	/* Avoid an unnecessary call to unbind on rebind. */
@@ -3585,8 +3585,8 @@ search_free:
 	WARN_ON(flags & PIN_MAPPABLE && !obj->map_and_fenceable);
 
 	trace_i915_vma_bind(vma, flags);
-	vma->bind_vma(vma, obj->cache_level,
-		      flags & (PIN_MAPPABLE | PIN_GLOBAL) ? GLOBAL_BIND : 0);
+	i915_gem_vma_bind(vma, obj->cache_level,
+			  flags & (PIN_MAPPABLE | PIN_GLOBAL) ? GLOBAL_BIND : 0);
 
 	i915_gem_verify_gtt(dev);
 	return vma;
@@ -3797,8 +3797,8 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 
 		list_for_each_entry(vma, &obj->vma_list, vma_link)
 			if (drm_mm_node_allocated(&vma->node))
-				vma->bind_vma(vma, cache_level,
-					      obj->has_global_gtt_mapping ? GLOBAL_BIND : 0);
+				i915_gem_vma_bind(vma, cache_level,
+						  obj->has_global_gtt_mapping ? GLOBAL_BIND : 0);
 	}
 
 	list_for_each_entry(vma, &obj->vma_list, vma_link)
@@ -4199,7 +4199,7 @@ i915_gem_object_pin(struct drm_i915_gem_object *obj,
 	}
 
 	if (flags & PIN_GLOBAL && !obj->has_global_gtt_mapping)
-		vma->bind_vma(vma, obj->cache_level, GLOBAL_BIND);
+		i915_gem_vma_bind(vma, obj->cache_level, GLOBAL_BIND);
 
 	vma->pin_count++;
 	if (flags & PIN_MAPPABLE)
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index a5c7d5d..51b517e 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -781,7 +781,7 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 	if (!to->legacy_hw_ctx.rcs_state->has_global_gtt_mapping) {
 		struct i915_vma *vma = i915_gem_obj_to_vma(to->legacy_hw_ctx.rcs_state,
 							   &dev_priv->gtt.base);
-		vma->bind_vma(vma, to->legacy_hw_ctx.rcs_state->cache_level, GLOBAL_BIND);
+		i915_gem_vma_bind(vma, to->legacy_hw_ctx.rcs_state->cache_level, GLOBAL_BIND);
 	}
 
 	if (!to->legacy_hw_ctx.initialized || i915_gem_context_is_default(to)) {
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 1420aeb..884ec39 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -376,7 +376,8 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 		struct i915_vma *vma =
 			list_first_entry(&target_i915_obj->vma_list,
 					 typeof(*vma), vma_link);
-		vma->bind_vma(vma, target_i915_obj->cache_level, GLOBAL_BIND);
+		i915_gem_vma_bind(vma, target_i915_obj->cache_level,
+				  GLOBAL_BIND);
 	}
 
 	/* Validate that the target is in a valid r/w GPU domain */
@@ -1392,7 +1393,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		 * allocate space first */
 		struct i915_vma *vma = i915_gem_obj_to_ggtt(batch_obj);
 		BUG_ON(!vma);
-		vma->bind_vma(vma, batch_obj->cache_level, GLOBAL_BIND);
+		i915_gem_vma_bind(vma, batch_obj->cache_level, GLOBAL_BIND);
 	}
 
 	if (flags & I915_DISPATCH_SECURE)
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 254895d..2a75bce 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1343,7 +1343,7 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 		 * without telling our object about it. So we need to fake it.
 		 */
 		obj->has_global_gtt_mapping = 0;
-		vma->bind_vma(vma, obj->cache_level, GLOBAL_BIND);
+		i915_gem_vma_bind(vma, obj->cache_level, GLOBAL_BIND);
 	}
 
 
@@ -2097,6 +2097,17 @@ int i915_gem_gtt_init(struct drm_device *dev)
 	return 0;
 }
 
+void i915_gem_vma_bind(struct i915_vma *vma, enum i915_cache_level cache_level,
+		       unsigned flags)
+{
+	vma->bind_vma(vma, cache_level, flags);
+}
+
+void i915_gem_vma_unbind(struct i915_vma *vma)
+{
+	vma->unbind_vma(vma);
+}
+
 static struct i915_vma *__i915_gem_vma_create(struct drm_i915_gem_object *obj,
 					      struct i915_address_space *vm)
 {
-- 
2.0.4

[PATCH 20/68] drm/i915: Make pin global flags explicit

[Ben Widawsky]

The driver currently lets callers pin global, and then tries to do
things correctly inside the function. Doing so has two downsides:
1. It's not possible to exclusively pin to a global, or an aliasing
address space.
2. It's difficult to read, and understand.

The eventual goal when realized should fix both of the issues. This patch
which should have no functional impact begins to address these issues
without intentionally breaking things.

v2: Replace PIN_GLOBAL with PIN_ALIASING in _pin(). Copy paste error

v3: Rebased/reworked with flag conflict from negative relocations

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h            | 14 ++++++++------
 drivers/gpu/drm/i915/i915_gem.c            | 31 +++++++++++++++++++++++-------
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  8 ++++++--
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 12 ++++++++++--
 drivers/gpu/drm/i915/i915_gem_gtt.h        |  6 +++++-
 5 files changed, 53 insertions(+), 18 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index d1750d5..0db17f8 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2300,11 +2300,13 @@ void i915_init_vm(struct drm_i915_private *dev_priv,
 void i915_gem_free_object(struct drm_gem_object *obj);
 void i915_gem_vma_destroy(struct i915_vma *vma);
 
-#define PIN_MAPPABLE 0x1
-#define PIN_NONBLOCK 0x2
-#define PIN_GLOBAL 0x4
-#define PIN_OFFSET_BIAS 0x8
-#define PIN_OFFSET_MASK (~4095)
+#define PIN_MAPPABLE	(1<<0)
+#define PIN_NONBLOCK	(1<<1)
+#define PIN_GLOBAL	(1<<2)
+#define PIN_ALIASING	(1<<3)
+#define PIN_GLOBAL_ALIASED (PIN_ALIASING | PIN_GLOBAL)
+#define PIN_OFFSET_BIAS (1<<4)
+#define PIN_OFFSET_MASK (PAGE_MASK)
 int __must_check i915_gem_object_pin(struct drm_i915_gem_object *obj,
 				     struct i915_address_space *vm,
 				     uint32_t alignment,
@@ -2511,7 +2513,7 @@ i915_gem_obj_ggtt_pin(struct drm_i915_gem_object *obj,
 		      uint32_t alignment,
 		      unsigned flags)
 {
-	return i915_gem_object_pin(obj, obj_to_ggtt(obj), alignment, flags | PIN_GLOBAL);
+	return i915_gem_object_pin(obj, obj_to_ggtt(obj), alignment, flags | PIN_GLOBAL_ALIASED);
 }
 
 static inline int
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 5f66939..1dd5f43 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3496,8 +3496,12 @@ i915_gem_object_bind_to_vm(struct drm_i915_gem_object *obj,
 	unsigned long end =
 		flags & PIN_MAPPABLE ? dev_priv->gtt.mappable_end : vm->total;
 	struct i915_vma *vma;
+	u32 vma_bind_flags = 0;
 	int ret;
 
+	if (WARN_ON((flags & (PIN_MAPPABLE | PIN_GLOBAL)) == PIN_MAPPABLE))
+		flags |= PIN_GLOBAL;
+
 	fence_size = i915_gem_get_gtt_size(dev,
 					   obj->base.size,
 					   obj->tiling_mode);
@@ -3584,9 +3588,11 @@ search_free:
 
 	WARN_ON(flags & PIN_MAPPABLE && !obj->map_and_fenceable);
 
+	if (flags & PIN_GLOBAL_ALIASED)
+		vma_bind_flags = GLOBAL_BIND | ALIASING_BIND;
+
 	trace_i915_vma_bind(vma, flags);
-	i915_gem_vma_bind(vma, obj->cache_level,
-			  flags & (PIN_MAPPABLE | PIN_GLOBAL) ? GLOBAL_BIND : 0);
+	i915_gem_vma_bind(vma, obj->cache_level, vma_bind_flags);
 
 	i915_gem_verify_gtt(dev);
 	return vma;
@@ -3796,9 +3802,14 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 		}
 
 		list_for_each_entry(vma, &obj->vma_list, vma_link)
-			if (drm_mm_node_allocated(&vma->node))
-				i915_gem_vma_bind(vma, cache_level,
-						  obj->has_global_gtt_mapping ? GLOBAL_BIND : 0);
+			if (drm_mm_node_allocated(&vma->node)) {
+				u32 bind_flags = 0;
+				if (obj->has_global_gtt_mapping)
+					bind_flags |= GLOBAL_BIND;
+				if (obj->has_aliasing_ppgtt_mapping)
+					bind_flags |= ALIASING_BIND;
+				i915_gem_vma_bind(vma, cache_level, bind_flags);
+			}
 	}
 
 	list_for_each_entry(vma, &obj->vma_list, vma_link)
@@ -4198,8 +4209,14 @@ i915_gem_object_pin(struct drm_i915_gem_object *obj,
 			return PTR_ERR(vma);
 	}
 
-	if (flags & PIN_GLOBAL && !obj->has_global_gtt_mapping)
-		i915_gem_vma_bind(vma, obj->cache_level, GLOBAL_BIND);
+	if (flags & PIN_GLOBAL_ALIASED) {
+		u32 bind_flags = 0;
+		if (flags & PIN_GLOBAL && !obj->has_global_gtt_mapping)
+			bind_flags |= GLOBAL_BIND;
+		if (flags & PIN_ALIASING && !obj->has_aliasing_ppgtt_mapping)
+			bind_flags |= ALIASING_BIND;
+		i915_gem_vma_bind(vma, obj->cache_level, bind_flags);
+	}
 
 	vma->pin_count++;
 	if (flags & PIN_MAPPABLE)
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 884ec39..0c7adb8 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -377,7 +377,7 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 			list_first_entry(&target_i915_obj->vma_list,
 					 typeof(*vma), vma_link);
 		i915_gem_vma_bind(vma, target_i915_obj->cache_level,
-				  GLOBAL_BIND);
+				  GLOBAL_BIND | ALIASING_BIND);
 	}
 
 	/* Validate that the target is in a valid r/w GPU domain */
@@ -564,6 +564,7 @@ i915_gem_execbuffer_reserve_vma(struct i915_vma *vma,
 	if (need_fence || need_reloc_mappable(vma))
 		flags |= PIN_MAPPABLE;
 
+	/* FIXME: What kind of bind does Chris want? */
 	if (entry->flags & EXEC_OBJECT_NEEDS_GTT)
 		flags |= PIN_GLOBAL;
 	if (entry->flags & __EXEC_OBJECT_NEEDS_BIAS)
@@ -1393,7 +1394,10 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		 * allocate space first */
 		struct i915_vma *vma = i915_gem_obj_to_ggtt(batch_obj);
 		BUG_ON(!vma);
-		i915_gem_vma_bind(vma, batch_obj->cache_level, GLOBAL_BIND);
+		/* FIXME: Current secure dispatch code actually uses PPGTT. We
+		 * need to fix this eventually */
+		i915_gem_vma_bind(vma, batch_obj->cache_level,
+				  GLOBAL_BIND | ALIASING_BIND);
 	}
 
 	if (flags & I915_DISPATCH_SECURE)
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 2a75bce..0fb24b5 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1342,8 +1342,16 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 		 * Unfortunately above, we've just wiped out the mappings
 		 * without telling our object about it. So we need to fake it.
 		 */
-		obj->has_global_gtt_mapping = 0;
-		i915_gem_vma_bind(vma, obj->cache_level, GLOBAL_BIND);
+		if (obj->has_global_gtt_mapping || obj->has_aliasing_ppgtt_mapping) {
+			u32 bind_flags = 0;
+			if (obj->has_global_gtt_mapping)
+				        bind_flags |= GLOBAL_BIND;
+			if (obj->has_aliasing_ppgtt_mapping)
+				        bind_flags |= ALIASING_BIND;
+			obj->has_global_gtt_mapping = 0;
+			obj->has_aliasing_ppgtt_mapping = 0;
+			i915_gem_vma_bind(vma, obj->cache_level, bind_flags);
+		}
 	}
 
 
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 593b5d0..6785060 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -157,8 +157,12 @@ struct i915_vma {
 	 * setting the valid PTE entries to a reserved scratch page. */
 	void (*unbind_vma)(struct i915_vma *vma);
 	/* Map an object into an address space with the given cache flags. */
+
+/* Only use this if you know you want a strictly global binding */
 #define GLOBAL_BIND (1<<0)
-#define PTE_READ_ONLY (1<<1)
+/* Only use this if you know you want a strictly aliased binding */
+#define ALIASING_BIND (1<<1)
+#define PTE_READ_ONLY (1<<2)
 	void (*bind_vma)(struct i915_vma *vma,
 			 enum i915_cache_level cache_level,
 			 u32 flags);
-- 
2.0.4

[PATCH 21/68] drm/i915: Split out aliasing binds

[Ben Widawsky]

This patch finishes off  actually separating the aliasing and global
finds. Prior to this, all global binds would be aliased. Now if aliasing
binds are required, they must be explicitly asked for. So far, we have
no users of this outside of execbuf - but Mika has already submitted a
patch requiring just this.

A nice benefit of this is we should no longer be able to clobber GTT
only objects from the aliasing PPGTT.

v2: Only add aliasing binds for the GGTT/Aliasing PPGTT at execbuf

v3: Rebase resolution with changed size of flags

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h            | 2 +-
 drivers/gpu/drm/i915/i915_gem.c            | 6 ++++--
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 5 +++--
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 3 +++
 4 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 0db17f8..651ad7f 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2513,7 +2513,7 @@ i915_gem_obj_ggtt_pin(struct drm_i915_gem_object *obj,
 		      uint32_t alignment,
 		      unsigned flags)
 {
-	return i915_gem_object_pin(obj, obj_to_ggtt(obj), alignment, flags | PIN_GLOBAL_ALIASED);
+	return i915_gem_object_pin(obj, obj_to_ggtt(obj), alignment, flags | PIN_GLOBAL);
 }
 
 static inline int
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 1dd5f43..6413f3a 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3588,8 +3588,10 @@ search_free:
 
 	WARN_ON(flags & PIN_MAPPABLE && !obj->map_and_fenceable);
 
-	if (flags & PIN_GLOBAL_ALIASED)
-		vma_bind_flags = GLOBAL_BIND | ALIASING_BIND;
+	if (flags & PIN_ALIASING)
+		vma_bind_flags = ALIASING_BIND;
+	if (flags & PIN_GLOBAL)
+		vma_bind_flags = GLOBAL_BIND;
 
 	trace_i915_vma_bind(vma, flags);
 	i915_gem_vma_bind(vma, obj->cache_level, vma_bind_flags);
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 0c7adb8..caccee9 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -552,10 +552,11 @@ i915_gem_execbuffer_reserve_vma(struct i915_vma *vma,
 	struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;
 	bool has_fenced_gpu_access = INTEL_INFO(ring->dev)->gen < 4;
 	bool need_fence;
-	uint64_t flags;
+	uint64_t flags = 0;
 	int ret;
 
-	flags = 0;
+	if (i915_is_ggtt(vma->vm))
+		flags = PIN_ALIASING;
 
 	need_fence =
 		has_fenced_gpu_access &&
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 0fb24b5..e187dc1 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1603,6 +1603,9 @@ static void ggtt_bind_vma(struct i915_vma *vma,
 		}
 	}
 
+	if (!(flags & ALIASING_BIND))
+		return;
+
 	if (dev_priv->mm.aliasing_ppgtt &&
 	    (!obj->has_aliasing_ppgtt_mapping ||
 	     (cache_level != obj->cache_level))) {
-- 
2.0.4

[PATCH 22/68] drm/i915: fix gtt_total_entries()

[Ben Widawsky]

It's useful to have it not as a macro for some upcoming work. Generally
since we try to avoid macros anyway, I think it doesn't hurt to put this
as its own patch.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c    | 4 ++--
 drivers/gpu/drm/i915/i915_gem_gtt.h    | 7 +++++--
 drivers/gpu/drm/i915/i915_gem_stolen.c | 2 +-
 3 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index e187dc1..d914ca8 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1495,7 +1495,7 @@ static void gen8_ggtt_clear_range(struct i915_address_space *vm,
 	unsigned num_entries = length >> PAGE_SHIFT;
 	gen8_gtt_pte_t scratch_pte, __iomem *gtt_base =
 		(gen8_gtt_pte_t __iomem *) dev_priv->gtt.gsm + first_entry;
-	const int max_entries = gtt_total_entries(dev_priv->gtt) - first_entry;
+	const int max_entries = gtt_total_entries(&dev_priv->gtt) - first_entry;
 	int i;
 
 	if (WARN(num_entries > max_entries,
@@ -1521,7 +1521,7 @@ static void gen6_ggtt_clear_range(struct i915_address_space *vm,
 	unsigned num_entries = length >> PAGE_SHIFT;
 	gen6_gtt_pte_t scratch_pte, __iomem *gtt_base =
 		(gen6_gtt_pte_t __iomem *) dev_priv->gtt.gsm + first_entry;
-	const int max_entries = gtt_total_entries(dev_priv->gtt) - first_entry;
+	const int max_entries = gtt_total_entries(&dev_priv->gtt) - first_entry;
 	int i;
 
 	if (WARN(num_entries > max_entries,
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 6785060..6b764b8 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -38,8 +38,6 @@ typedef uint32_t gen6_gtt_pte_t;
 typedef uint64_t gen8_gtt_pte_t;
 typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
 
-#define gtt_total_entries(gtt) ((gtt).base.total >> PAGE_SHIFT)
-
 #define I915_PPGTT_PT_ENTRIES		(PAGE_SIZE / sizeof(gen6_gtt_pte_t))
 /* gen6-hsw has bit 11-4 for physical addr bit 39-32 */
 #define GEN6_GTT_ADDR_ENCODE(addr)	((addr) | (((addr) >> 28) & 0xff0))
@@ -280,6 +278,11 @@ void i915_gem_init_global_gtt(struct drm_device *dev);
 void i915_gem_setup_global_gtt(struct drm_device *dev, unsigned long start,
 			       unsigned long mappable_end, unsigned long end);
 
+static inline size_t gtt_total_entries(struct i915_gtt *gtt)
+{
+	return gtt->base.total >> PAGE_SHIFT;
+}
+
 int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt);
 
 void i915_check_and_clear_faults(struct drm_device *dev);
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index 21c025a..c9f916c 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -90,7 +90,7 @@ static unsigned long i915_stolen_to_physical(struct drm_device *dev)
 				(gtt_start & PGTBL_ADDRESS_HI_MASK) << 28;
 		else
 			gtt_start &= PGTBL_ADDRESS_LO_MASK;
-		gtt_end = gtt_start + gtt_total_entries(dev_priv->gtt) * 4;
+		gtt_end = gtt_start + gtt_total_entries(&dev_priv->gtt) * 4;
 
 		if (gtt_start >= stolen[0].start && gtt_start < stolen[0].end)
 			stolen[0].end = gtt_start;
-- 
2.0.4

[PATCH 23/68] drm/i915: Rename to GEN8_LEGACY_PDPES

[Ben Widawsky]

In gen8, 32b PPGTT has always had one "pdp" (it doesn't actually have
one, but it resembles having one). The #define was confusing as is, and
using "PDPE" is a much better description.

sed -i 's/GEN8_LEGACY_PDPS/GEN8_LEGACY_PDPES/' drivers/gpu/drm/i915/*.[ch]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 6 +++---
 drivers/gpu/drm/i915/i915_gem_gtt.h | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index d914ca8..ab863bb 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -311,7 +311,7 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	pt_vaddr = NULL;
 
 	for_each_sg_page(pages->sgl, &sg_iter, pages->nents, 0) {
-		if (WARN_ON(pdpe >= GEN8_LEGACY_PDPS))
+		if (WARN_ON(pdpe >= GEN8_LEGACY_PDPES))
 			break;
 
 		if (pt_vaddr == NULL)
@@ -425,7 +425,7 @@ bail:
 static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt,
 					   const int max_pdp)
 {
-	struct page **pt_pages[GEN8_LEGACY_PDPS];
+	struct page **pt_pages[GEN8_LEGACY_PDPES];
 	int i, ret;
 
 	for (i = 0; i < max_pdp; i++) {
@@ -476,7 +476,7 @@ static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
 		return -ENOMEM;
 
 	ppgtt->num_pd_pages = 1 << get_order(max_pdp << PAGE_SHIFT);
-	BUG_ON(ppgtt->num_pd_pages > GEN8_LEGACY_PDPS);
+	BUG_ON(ppgtt->num_pd_pages > GEN8_LEGACY_PDPES);
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 6b764b8..4af3150 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -84,7 +84,7 @@ typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
 #define GEN8_PDE_MASK			0x1ff
 #define GEN8_PTE_SHIFT			12
 #define GEN8_PTE_MASK			0x1ff
-#define GEN8_LEGACY_PDPS		4
+#define GEN8_LEGACY_PDPES		4
 #define GEN8_PTES_PER_PAGE		(PAGE_SIZE / sizeof(gen8_gtt_pte_t))
 #define GEN8_PDES_PER_PAGE		(PAGE_SIZE / sizeof(gen8_ppgtt_pde_t))
 
@@ -252,12 +252,12 @@ struct i915_hw_ppgtt {
 	unsigned num_pd_pages; /* gen8+ */
 	union {
 		struct page **pt_pages;
-		struct page **gen8_pt_pages[GEN8_LEGACY_PDPS];
+		struct page **gen8_pt_pages[GEN8_LEGACY_PDPES];
 	};
 	struct page *pd_pages;
 	union {
 		uint32_t pd_offset;
-		dma_addr_t pd_dma_addr[GEN8_LEGACY_PDPS];
+		dma_addr_t pd_dma_addr[GEN8_LEGACY_PDPES];
 	};
 	union {
 		dma_addr_t *pt_dma_addr;
-- 
2.0.4

[PATCH 24/68] drm/i915: Split out verbose PPGTT dumping

[Ben Widawsky]

There often is not enough memory to dump the full contents of the PPGTT.
As a temporary bandage, to continue getting valuable basic PPGTT info,
wrap the dangerous, memory hungry part inside of a new verbose version
of the debugfs file.

Also while here we can split out the PPGTT print function so it's more
reusable.

I'd really like to get PPGTT info into our error state, but I found it too
difficult to make work in the limited time I have. Maybe Mika can find a way.

v2: Get the info for the non-default contexts. Merge a patch from Chris
into this patch (Chris). All credit goes to him.

References: 20140320115742.GA4463@nuc-i3427.alporthouse.com
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c | 52 +++++++++++++++++++++++--------------
 1 file changed, 32 insertions(+), 20 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 16ae700..d2977cf 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1811,22 +1811,12 @@ static int i915_swizzle_info(struct seq_file *m, void *data)
 	return 0;
 }
 
-static int per_file_ctx(int id, void *ptr, void *data)
+static void print_ppgtt(struct seq_file *m, struct i915_hw_ppgtt *ppgtt)
 {
-	struct intel_context *ctx = ptr;
-	struct seq_file *m = data;
-	struct i915_hw_ppgtt *ppgtt = ctx_to_ppgtt(ctx);
-
-	if (i915_gem_context_is_default(ctx))
-		seq_puts(m, "  default context:\n");
-	else
-		seq_printf(m, "  context %d:\n", ctx->user_handle);
-	ppgtt->debug_dump(ppgtt, m);
-
-	return 0;
+	seq_printf(m, "pd gtt offset: 0x%08x\n", ppgtt->pd_offset);
 }
 
-static void gen8_ppgtt_info(struct seq_file *m, struct drm_device *dev)
+static void gen8_ppgtt_info(struct seq_file *m, struct drm_device *dev, int verbose)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct intel_engine_cs *ring;
@@ -1850,7 +1840,26 @@ static void gen8_ppgtt_info(struct seq_file *m, struct drm_device *dev)
 	}
 }
 
-static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev)
+static int per_file_ctx(int id, void *ptr, void *data)
+{
+	struct intel_context *ctx = ptr;
+	struct seq_file *m = data;
+	bool verbose = (unsigned long)data & 1;
+	struct i915_hw_ppgtt *ppgtt = ctx_to_ppgtt(ctx);
+
+	if (i915_gem_context_is_default(ctx))
+		seq_puts(m, "  default context:\n");
+	else
+		seq_printf(m, "  context %d:\n", ctx->user_handle);
+
+	print_ppgtt(m, ppgtt);
+	if (verbose)
+		ppgtt->debug_dump(ppgtt, m);
+
+	return 0;
+}
+
+static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev, bool verbose)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct intel_engine_cs *ring;
@@ -1872,9 +1881,9 @@ static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev)
 		struct i915_hw_ppgtt *ppgtt = dev_priv->mm.aliasing_ppgtt;
 
 		seq_puts(m, "aliasing PPGTT:\n");
-		seq_printf(m, "pd gtt offset: 0x%08x\n", ppgtt->pd_offset);
-
-		ppgtt->debug_dump(ppgtt, m);
+		print_ppgtt(m, ppgtt);
+		if (verbose)
+			ppgtt->debug_dump(ppgtt, m);
 	} else
 		return;
 
@@ -1883,7 +1892,8 @@ static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev)
 
 		seq_printf(m, "proc: %s\n",
 			   get_pid_task(file->pid, PIDTYPE_PID)->comm);
-		idr_for_each(&file_priv->context_idr, per_file_ctx, m);
+		idr_for_each(&file_priv->context_idr, per_file_ctx,
+			     (void *)((unsigned long)m | verbose));
 	}
 	seq_printf(m, "ECOCHK: 0x%08x\n", I915_READ(GAM_ECOCHK));
 }
@@ -1893,6 +1903,7 @@ static int i915_ppgtt_info(struct seq_file *m, void *data)
 	struct drm_info_node *node = m->private;
 	struct drm_device *dev = node->minor->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	bool verbose = node->info_ent->data ? true : false;
 
 	int ret = mutex_lock_interruptible(&dev->struct_mutex);
 	if (ret)
@@ -1900,9 +1911,9 @@ static int i915_ppgtt_info(struct seq_file *m, void *data)
 	intel_runtime_pm_get(dev_priv);
 
 	if (INTEL_INFO(dev)->gen >= 8)
-		gen8_ppgtt_info(m, dev);
+		gen8_ppgtt_info(m, dev, verbose);
 	else if (INTEL_INFO(dev)->gen >= 6)
-		gen6_ppgtt_info(m, dev);
+		gen6_ppgtt_info(m, dev, verbose);
 
 	intel_runtime_pm_put(dev_priv);
 	mutex_unlock(&dev->struct_mutex);
@@ -3966,6 +3977,7 @@ static const struct drm_info_list i915_debugfs_list[] = {
 	{"i915_gen6_forcewake_count", i915_gen6_forcewake_count_info, 0},
 	{"i915_swizzle_info", i915_swizzle_info, 0},
 	{"i915_ppgtt_info", i915_ppgtt_info, 0},
+	{"i915_ppgtt_verbose_info", i915_ppgtt_info, 0, (void *)1},
 	{"i915_llc", i915_llc, 0},
 	{"i915_edp_psr_status", i915_edp_psr_status, 0},
 	{"i915_sink_crc_eDP1", i915_sink_crc, 0},
-- 
2.0.4

[PATCH 25/68] drm/i915: s/pd/pdpe, s/pt/pde

[Ben Widawsky]

The actual correct way to think about this with the new style of page
table data structures is as the actual entry that is being indexed into
the array. "pd", and "pt" aren't representative of what the operation is
doing.

The clarity here will improve the readability of future patches.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index ab863bb..861df21 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -506,40 +506,40 @@ static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
 }
 
 static int gen8_ppgtt_setup_page_directories(struct i915_hw_ppgtt *ppgtt,
-					     const int pd)
+					     const int pdpe)
 {
 	dma_addr_t pd_addr;
 	int ret;
 
 	pd_addr = pci_map_page(ppgtt->base.dev->pdev,
-			       &ppgtt->pd_pages[pd], 0,
+			       &ppgtt->pd_pages[pdpe], 0,
 			       PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
 
 	ret = pci_dma_mapping_error(ppgtt->base.dev->pdev, pd_addr);
 	if (ret)
 		return ret;
 
-	ppgtt->pd_dma_addr[pd] = pd_addr;
+	ppgtt->pd_dma_addr[pdpe] = pd_addr;
 
 	return 0;
 }
 
 static int gen8_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt,
-					const int pd,
-					const int pt)
+					const int pdpe,
+					const int pde)
 {
 	dma_addr_t pt_addr;
 	struct page *p;
 	int ret;
 
-	p = ppgtt->gen8_pt_pages[pd][pt];
+	p = ppgtt->gen8_pt_pages[pdpe][pde];
 	pt_addr = pci_map_page(ppgtt->base.dev->pdev,
 			       p, 0, PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
 	ret = pci_dma_mapping_error(ppgtt->base.dev->pdev, pt_addr);
 	if (ret)
 		return ret;
 
-	ppgtt->gen8_pt_dma_addr[pd][pt] = pt_addr;
+	ppgtt->gen8_pt_dma_addr[pdpe][pde] = pt_addr;
 
 	return 0;
 }
-- 
2.0.4

[PATCH 26/68] drm/i915: rename map/unmap to dma_map/unmap

[Ben Widawsky]

Upcoming patches will use the terms map and unmap in references to the
page table entries. Having this distinction will really help with code
clarity at that point.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 861df21..6e15c82 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -364,7 +364,7 @@ static void gen8_ppgtt_free(const struct i915_hw_ppgtt *ppgtt)
 	__free_pages(ppgtt->pd_pages, get_order(ppgtt->num_pd_pages << PAGE_SHIFT));
 }
 
-static void gen8_ppgtt_unmap_pages(struct i915_hw_ppgtt *ppgtt)
+static void gen8_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 {
 	struct pci_dev *hwdev = ppgtt->base.dev->pdev;
 	int i, j;
@@ -395,7 +395,7 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
 	list_del(&vm->global_link);
 	drm_mm_takedown(&vm->mm);
 
-	gen8_ppgtt_unmap_pages(ppgtt);
+	gen8_ppgtt_dma_unmap_pages(ppgtt);
 	gen8_ppgtt_free(ppgtt);
 }
 
@@ -622,7 +622,7 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	return 0;
 
 bail:
-	gen8_ppgtt_unmap_pages(ppgtt);
+	gen8_ppgtt_dma_unmap_pages(ppgtt);
 	gen8_ppgtt_free(ppgtt);
 	return ret;
 }
@@ -991,7 +991,7 @@ static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 		kunmap_atomic(pt_vaddr);
 }
 
-static void gen6_ppgtt_unmap_pages(struct i915_hw_ppgtt *ppgtt)
+static void gen6_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 {
 	int i;
 
@@ -1022,7 +1022,7 @@ static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
 	drm_mm_takedown(&vm->mm);
 	drm_mm_remove_node(&ppgtt->node);
 
-	gen6_ppgtt_unmap_pages(ppgtt);
+	gen6_ppgtt_dma_unmap_pages(ppgtt);
 	gen6_ppgtt_free(ppgtt);
 }
 
@@ -1122,7 +1122,7 @@ static int gen6_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt)
 				       PCI_DMA_BIDIRECTIONAL);
 
 		if (pci_dma_mapping_error(dev->pdev, pt_addr)) {
-			gen6_ppgtt_unmap_pages(ppgtt);
+			gen6_ppgtt_dma_unmap_pages(ppgtt);
 			return -EIO;
 		}
 
-- 
2.0.4

[PATCH 27/68] drm/i915: Setup less PPGTT on failed pagedir

[Ben Widawsky]

The current code will both potentially print a WARN, and setup part of
the PPGTT structure. Neither of these harm the current code, it is
simply for clarity, and to perhaps prevent later bugs, or weird
debug messages.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 6e15c82..46140e8 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1057,11 +1057,14 @@ alloc:
 		goto alloc;
 	}
 
+	if (ret)
+		return ret;
+
 	if (ppgtt->node.start < dev_priv->gtt.mappable_end)
 		DRM_DEBUG("Forced to use aperture for PDEs\n");
 
 	ppgtt->num_pd_entries = GEN6_PPGTT_PD_ENTRIES;
-	return ret;
+	return 0;
 }
 
 static int gen6_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
-- 
2.0.4

[PATCH 28/68] drm/i915: clean up PPGTT init error path

[Ben Widawsky]

The old code (I'm having trouble finding the commit) had a reason for
doing things when there was an error, and would continue on, thus the
!ret. For the newer code however, this looks completely silly.

Follow the normal idiom of if (ret) return ret.

Also, put the pde wiring in the gen specific init, now that GEN8 exists.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 46140e8..151ec39 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1174,6 +1174,8 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->pd_offset =
 		ppgtt->node.start / PAGE_SIZE * sizeof(gen6_gtt_pte_t);
 
+	gen6_write_pdes(ppgtt);
+
 	ppgtt->base.clear_range(&ppgtt->base, 0, ppgtt->base.total, true);
 
 	DRM_DEBUG_DRIVER("Allocated pde space (%ldM) at GTT entry: %lx\n",
@@ -1198,20 +1200,14 @@ int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 	else
 		BUG();
 
-	if (!ret) {
-		struct drm_i915_private *dev_priv = dev->dev_private;
-		kref_init(&ppgtt->ref);
-		drm_mm_init(&ppgtt->base.mm, ppgtt->base.start,
-			    ppgtt->base.total);
-		i915_init_vm(dev_priv, &ppgtt->base);
-		if (INTEL_INFO(dev)->gen < 8) {
-			gen6_write_pdes(ppgtt);
-			DRM_DEBUG("Adding PPGTT at offset %x\n",
-				  ppgtt->pd_offset << 10);
-		}
-	}
+	if (ret)
+		return ret;
 
-	return ret;
+	kref_init(&ppgtt->ref);
+	drm_mm_init(&ppgtt->base.mm, ppgtt->base.start, ppgtt->base.total);
+	i915_init_vm(dev_priv, &ppgtt->base);
+
+	return 0;
 }
 
 static void
-- 
2.0.4

[PATCH 29/68] drm/i915: Un-hardcode number of page directories

[Ben Widawsky]

trivial.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 4af3150..0199c5a 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -261,7 +261,7 @@ struct i915_hw_ppgtt {
 	};
 	union {
 		dma_addr_t *pt_dma_addr;
-		dma_addr_t *gen8_pt_dma_addr[4];
+		dma_addr_t *gen8_pt_dma_addr[GEN8_LEGACY_PDPES];
 	};
 
 	struct intel_context *ctx;
-- 
2.0.4

[PATCH 30/68] drm/i915: Make gen6_write_pdes gen6_map_page_tables

[Ben Widawsky]

Split out single mappings which will help with upcoming work. Also while
here, rename the function because it is a better description - but this
function is going away soon.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 39 ++++++++++++++++++++++---------------
 1 file changed, 23 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 151ec39..ca7ddb6 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -683,26 +683,33 @@ static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 	}
 }
 
-static void gen6_write_pdes(struct i915_hw_ppgtt *ppgtt)
+static void gen6_map_single(struct i915_hw_ppgtt *ppgtt,
+			    const unsigned pde_index,
+			    dma_addr_t daddr)
 {
 	struct drm_i915_private *dev_priv = ppgtt->base.dev->dev_private;
-	gen6_gtt_pte_t __iomem *pd_addr;
 	uint32_t pd_entry;
+	gen6_gtt_pte_t __iomem *pd_addr =
+		(gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm + ppgtt->pd_offset / sizeof(gen6_gtt_pte_t);
+
+	pd_entry = GEN6_PDE_ADDR_ENCODE(daddr);
+	pd_entry |= GEN6_PDE_VALID;
+
+	writel(pd_entry, pd_addr + pde_index);
+}
+
+/* Map all the page tables found in the ppgtt structure to incrementing page
+ * directories. */
+static void gen6_map_page_tables(struct i915_hw_ppgtt *ppgtt)
+{
+	struct drm_i915_private *dev_priv = ppgtt->base.dev->dev_private;
 	int i;
 
 	WARN_ON(ppgtt->pd_offset & 0x3f);
-	pd_addr = (gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm +
-		ppgtt->pd_offset / sizeof(gen6_gtt_pte_t);
-	for (i = 0; i < ppgtt->num_pd_entries; i++) {
-		dma_addr_t pt_addr;
-
-		pt_addr = ppgtt->pt_dma_addr[i];
-		pd_entry = GEN6_PDE_ADDR_ENCODE(pt_addr);
-		pd_entry |= GEN6_PDE_VALID;
+	for (i = 0; i < ppgtt->num_pd_entries; i++)
+		gen6_map_single(ppgtt, i, ppgtt->pt_dma_addr[i]);
 
-		writel(pd_entry, pd_addr + i);
-	}
-	readl(pd_addr);
+	readl(dev_priv->gtt.gsm);
 }
 
 static uint32_t get_pd_offset(struct i915_hw_ppgtt *ppgtt)
@@ -1174,7 +1181,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->pd_offset =
 		ppgtt->node.start / PAGE_SIZE * sizeof(gen6_gtt_pte_t);
 
-	gen6_write_pdes(ppgtt);
+	gen6_map_page_tables(ppgtt);
 
 	ppgtt->base.clear_range(&ppgtt->base, 0, ppgtt->base.total, true);
 
@@ -1367,11 +1374,11 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 		/* TODO: Perhaps it shouldn't be gen6 specific */
 		if (i915_is_ggtt(vm)) {
 			if (dev_priv->mm.aliasing_ppgtt)
-				gen6_write_pdes(dev_priv->mm.aliasing_ppgtt);
+				gen6_map_page_tables(dev_priv->mm.aliasing_ppgtt);
 			continue;
 		}
 
-		gen6_write_pdes(container_of(vm, struct i915_hw_ppgtt, base));
+		gen6_map_page_tables(container_of(vm, struct i915_hw_ppgtt, base));
 	}
 
 	i915_gem_chipset_flush(dev);
-- 
2.0.4

[PATCH 31/68] drm/i915: Range clearing is PPGTT agnostic

[Ben Widawsky]

Therefore we can do it from our general init function. Eventually, I
hope to have a lot more commonality like this. It won't arrive yet, but
this was a nice easy one.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index ca7ddb6..886c9c3 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -612,8 +612,6 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->base.start = 0;
 	ppgtt->base.total = ppgtt->num_pd_entries * GEN8_PTES_PER_PAGE * PAGE_SIZE;
 
-	ppgtt->base.clear_range(&ppgtt->base, 0, ppgtt->base.total, true);
-
 	DRM_DEBUG_DRIVER("Allocated %d pages for page directories (%d wasted)\n",
 			 ppgtt->num_pd_pages, ppgtt->num_pd_pages - max_pdp);
 	DRM_DEBUG_DRIVER("Allocated %d pages for page tables (%lld wasted)\n",
@@ -1183,8 +1181,6 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 
 	gen6_map_page_tables(ppgtt);
 
-	ppgtt->base.clear_range(&ppgtt->base, 0, ppgtt->base.total, true);
-
 	DRM_DEBUG_DRIVER("Allocated pde space (%ldM) at GTT entry: %lx\n",
 			 ppgtt->node.size >> 20,
 			 ppgtt->node.start / PAGE_SIZE);
@@ -1212,6 +1208,7 @@ int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 
 	kref_init(&ppgtt->ref);
 	drm_mm_init(&ppgtt->base.mm, ppgtt->base.start, ppgtt->base.total);
+	ppgtt->base.clear_range(&ppgtt->base, 0, ppgtt->base.total, true);
 	i915_init_vm(dev_priv, &ppgtt->base);
 
 	return 0;
-- 
2.0.4

[PATCH 32/68] drm/i915: Page table helpers, and define renames

[Ben Widawsky]

These page table helpers make the code much cleaner. There is some
room to use the arch/x86 header files. The reason I've opted not to is
in several cases, the definitions are dictated by the CONFIG_ options
which do not always indicate the restrictions in the GPU. While here,
clean up the defines to have more concise names, and consolidate between
gen6 and gen8 where appropriate.

v2: Use I915_PAGE_SIZE to remove PAGE_SIZE dep in the new code (Jesse)
Fix bugged I915_PTE_MASK define, which was unused (Chris)
BUG_ON bad length/size - taking directly from Chris (Chris)
define NUM_PTE (Chris)

I've made a lot of tiny errors in these helpers. Often I'd correct an
error only to introduce another one. While IGT was capable of catching
them, the tests often took a while to catch, and where hard/slow to
debug in the kernel. As a result, to test this, I compiled
i915_gem_gtt.h in userspace, and ran tests from userspace. What follows
isn't by any means complete, but it was able to catch lot of bugs. Gen8
is also untested, but since the current code is almost identical, I feel
pretty comfortable with that.

void test_pte(uint32_t base) {
        uint32_t ret;
        assert_pte_index((base + 0), 0);
        assert_pte_index((base + 1), 0);
        assert_pte_index((base + 0x1000), 1);
        assert_pte_index((base + (1<<22)), 0);
        assert_pte_index((base + ((1<<22) - 1)), 1023);
        assert_pte_index((base + (1<<21)), 512);

        assert_pte_count(base + 0, 0, 0);
        assert_pte_count(base + 0, 1, 1);
        assert_pte_count(base + 0, 0x1000, 1);
        assert_pte_count(base + 0, 0x1001, 2);
        assert_pte_count(base + 0, 1<<21, 512);

        assert_pte_count(base + 0, 1<<22, 1024);
        assert_pte_count(base + 0, (1<<22) - 1, 1024);
        assert_pte_count(base + (1<<21), 1<<22, 512);
        assert_pte_count(base + (1<<21), (1<<22)+1, 512);
        assert_pte_count(base + (1<<21), 10<<22, 512);
}

void test_pde(uint32_t base) {
        assert(gen6_pde_index(base + 0) == 0);
        assert(gen6_pde_index(base + 1) == 0);
        assert(gen6_pde_index(base + (1<<21)) == 0);
        assert(gen6_pde_index(base + (1<<22)) == 1);
        assert(gen6_pde_index(base + ((256<<22)))== 256);
        assert(gen6_pde_index(base + ((512<<22))) == 0);
        assert(gen6_pde_index(base + ((513<<22))) == 1); /* This is
actually not possible on gen6 */

        assert(gen6_pde_count(base + 0, 0) == 0);
        assert(gen6_pde_count(base + 0, 1) == 1);
        assert(gen6_pde_count(base + 0, 1<<21) == 1);
        assert(gen6_pde_count(base + 0, 1<<22) == 1);
        assert(gen6_pde_count(base + 0, (1<<22) + 0x1000) == 2);
        assert(gen6_pde_count(base + 0x1000, 1<<22) == 2);
        assert(gen6_pde_count(base + 0, 511<<22) == 511);
        assert(gen6_pde_count(base + 0, 512<<22) == 512);
        assert(gen6_pde_count(base + 0x1000, 512<<22) == 512);
        assert(gen6_pde_count(base + (1<<22), 512<<22) == 511);
}

int main()
{
        test_pde(0);
        while (1)
                test_pte(rand() & ~((1<<22) - 1));

        return 0;
}

v3: Some small rebase conflicts resolved

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c |  89 +++++++++++++-------------
 drivers/gpu/drm/i915/i915_gem_gtt.h | 123 +++++++++++++++++++++++++++++++++---
 2 files changed, 156 insertions(+), 56 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 886c9c3..5a62ef1 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -240,7 +240,7 @@ static int gen8_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	int i, ret;
 
 	/* bit of a hack to find the actual last used pd */
-	int used_pd = ppgtt->num_pd_entries / GEN8_PDES_PER_PAGE;
+	int used_pd = ppgtt->num_pd_entries / I915_PDES_PER_PD;
 
 	for (i = used_pd - 1; i >= 0; i--) {
 		dma_addr_t addr = ppgtt->pd_dma_addr[i];
@@ -260,9 +260,9 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 	gen8_gtt_pte_t *pt_vaddr, scratch_pte;
-	unsigned pdpe = start >> GEN8_PDPE_SHIFT & GEN8_PDPE_MASK;
-	unsigned pde = start >> GEN8_PDE_SHIFT & GEN8_PDE_MASK;
-	unsigned pte = start >> GEN8_PTE_SHIFT & GEN8_PTE_MASK;
+	unsigned pdpe = gen8_pdpe_index(start);
+	unsigned pde = gen8_pde_index(start);
+	unsigned pte = gen8_pte_index(start);
 	unsigned num_entries = length >> PAGE_SHIFT;
 	unsigned last_pte, i;
 
@@ -273,8 +273,8 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 		struct page *page_table = ppgtt->gen8_pt_pages[pdpe][pde];
 
 		last_pte = pte + num_entries;
-		if (last_pte > GEN8_PTES_PER_PAGE)
-			last_pte = GEN8_PTES_PER_PAGE;
+		if (last_pte > GEN8_PTES_PER_PT)
+			last_pte = GEN8_PTES_PER_PT;
 
 		pt_vaddr = kmap_atomic(page_table);
 
@@ -288,7 +288,7 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 		kunmap_atomic(pt_vaddr);
 
 		pte = 0;
-		if (++pde == GEN8_PDES_PER_PAGE) {
+		if (++pde == I915_PDES_PER_PD) {
 			pdpe++;
 			pde = 0;
 		}
@@ -303,9 +303,9 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 	gen8_gtt_pte_t *pt_vaddr;
-	unsigned pdpe = start >> GEN8_PDPE_SHIFT & GEN8_PDPE_MASK;
-	unsigned pde = start >> GEN8_PDE_SHIFT & GEN8_PDE_MASK;
-	unsigned pte = start >> GEN8_PTE_SHIFT & GEN8_PTE_MASK;
+	unsigned pdpe = gen8_pdpe_index(start);
+	unsigned pde = gen8_pde_index(start);
+	unsigned pte = gen8_pte_index(start);
 	struct sg_page_iter sg_iter;
 
 	pt_vaddr = NULL;
@@ -320,12 +320,12 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 		pt_vaddr[pte] =
 			gen8_pte_encode(sg_page_iter_dma_address(&sg_iter),
 					cache_level, true);
-		if (++pte == GEN8_PTES_PER_PAGE) {
+		if (++pte == GEN8_PTES_PER_PT) {
 			if (!HAS_LLC(ppgtt->base.dev))
 				drm_clflush_virt_range(pt_vaddr, PAGE_SIZE);
 			kunmap_atomic(pt_vaddr);
 			pt_vaddr = NULL;
-			if (++pde == GEN8_PDES_PER_PAGE) {
+			if (++pde == I915_PDES_PER_PD) {
 				pdpe++;
 				pde = 0;
 			}
@@ -346,7 +346,7 @@ static void gen8_free_page_tables(struct page **pt_pages)
 	if (pt_pages == NULL)
 		return;
 
-	for (i = 0; i < GEN8_PDES_PER_PAGE; i++)
+	for (i = 0; i < I915_PDES_PER_PD; i++)
 		if (pt_pages[i])
 			__free_pages(pt_pages[i], 0);
 }
@@ -378,7 +378,7 @@ static void gen8_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 		pci_unmap_page(hwdev, ppgtt->pd_dma_addr[i], PAGE_SIZE,
 			       PCI_DMA_BIDIRECTIONAL);
 
-		for (j = 0; j < GEN8_PDES_PER_PAGE; j++) {
+		for (j = 0; j < I915_PDES_PER_PD; j++) {
 			dma_addr_t addr = ppgtt->gen8_pt_dma_addr[i][j];
 			if (addr)
 				pci_unmap_page(hwdev, addr, PAGE_SIZE,
@@ -404,11 +404,11 @@ static struct page **__gen8_alloc_page_tables(void)
 	struct page **pt_pages;
 	int i;
 
-	pt_pages = kcalloc(GEN8_PDES_PER_PAGE, sizeof(struct page *), GFP_KERNEL);
+	pt_pages = kcalloc(I915_PDES_PER_PD, sizeof(struct page *), GFP_KERNEL);
 	if (!pt_pages)
 		return ERR_PTR(-ENOMEM);
 
-	for (i = 0; i < GEN8_PDES_PER_PAGE; i++) {
+	for (i = 0; i < I915_PDES_PER_PD; i++) {
 		pt_pages[i] = alloc_page(GFP_KERNEL);
 		if (!pt_pages[i])
 			goto bail;
@@ -458,7 +458,7 @@ static int gen8_ppgtt_allocate_dma(struct i915_hw_ppgtt *ppgtt)
 	int i;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		ppgtt->gen8_pt_dma_addr[i] = kcalloc(GEN8_PDES_PER_PAGE,
+		ppgtt->gen8_pt_dma_addr[i] = kcalloc(I915_PDES_PER_PD,
 						     sizeof(dma_addr_t),
 						     GFP_KERNEL);
 		if (!ppgtt->gen8_pt_dma_addr[i])
@@ -496,7 +496,7 @@ static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
 		return ret;
 	}
 
-	ppgtt->num_pd_entries = max_pdp * GEN8_PDES_PER_PAGE;
+	ppgtt->num_pd_entries = max_pdp * I915_PDES_PER_PD;
 
 	ret = gen8_ppgtt_allocate_dma(ppgtt);
 	if (ret)
@@ -557,7 +557,7 @@ static int gen8_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt,
 static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 {
 	const int max_pdp = DIV_ROUND_UP(size, 1 << 30);
-	const int min_pt_pages = GEN8_PDES_PER_PAGE * max_pdp;
+	const int min_pt_pages = I915_PDES_PER_PD * max_pdp;
 	int i, j, ret;
 
 	if (size % (1<<30))
@@ -576,7 +576,7 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 		if (ret)
 			goto bail;
 
-		for (j = 0; j < GEN8_PDES_PER_PAGE; j++) {
+		for (j = 0; j < I915_PDES_PER_PD; j++) {
 			ret = gen8_ppgtt_setup_page_tables(ppgtt, i, j);
 			if (ret)
 				goto bail;
@@ -594,7 +594,7 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	for (i = 0; i < max_pdp; i++) {
 		gen8_ppgtt_pde_t *pd_vaddr;
 		pd_vaddr = kmap_atomic(&ppgtt->pd_pages[i]);
-		for (j = 0; j < GEN8_PDES_PER_PAGE; j++) {
+		for (j = 0; j < I915_PDES_PER_PD; j++) {
 			dma_addr_t addr = ppgtt->gen8_pt_dma_addr[i][j];
 			pd_vaddr[j] = gen8_pde_encode(ppgtt->base.dev, addr,
 						      I915_CACHE_LLC);
@@ -610,7 +610,7 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
 	ppgtt->base.start = 0;
-	ppgtt->base.total = ppgtt->num_pd_entries * GEN8_PTES_PER_PAGE * PAGE_SIZE;
+	ppgtt->base.total = ppgtt->num_pd_entries * GEN8_PTES_PER_PT * PAGE_SIZE;
 
 	DRM_DEBUG_DRIVER("Allocated %d pages for page directories (%d wasted)\n",
 			 ppgtt->num_pd_pages, ppgtt->num_pd_pages - max_pdp);
@@ -656,9 +656,9 @@ static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 		seq_printf(m, "\tPDE: %x\n", pd_entry);
 
 		pt_vaddr = kmap_atomic(ppgtt->pt_pages[pde]);
-		for (pte = 0; pte < I915_PPGTT_PT_ENTRIES; pte+=4) {
+		for (pte = 0; pte < GEN6_PTES_PER_PT; pte+=4) {
 			unsigned long va =
-				(pde * PAGE_SIZE * I915_PPGTT_PT_ENTRIES) +
+				(pde * PAGE_SIZE * GEN6_PTES_PER_PT) +
 				(pte * PAGE_SIZE);
 			int i;
 			bool found = false;
@@ -937,29 +937,28 @@ static void gen6_ppgtt_clear_range(struct i915_address_space *vm,
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 	gen6_gtt_pte_t *pt_vaddr, scratch_pte;
-	unsigned first_entry = start >> PAGE_SHIFT;
+	unsigned pde = gen6_pde_index(start);
 	unsigned num_entries = length >> PAGE_SHIFT;
-	unsigned act_pt = first_entry / I915_PPGTT_PT_ENTRIES;
-	unsigned first_pte = first_entry % I915_PPGTT_PT_ENTRIES;
+	unsigned pte = gen6_pte_index(start);
 	unsigned last_pte, i;
 
 	scratch_pte = vm->pte_encode(vm->scratch.addr, I915_CACHE_LLC, true, 0);
 
 	while (num_entries) {
-		last_pte = first_pte + num_entries;
-		if (last_pte > I915_PPGTT_PT_ENTRIES)
-			last_pte = I915_PPGTT_PT_ENTRIES;
+		last_pte = pte + num_entries;
+		if (last_pte > GEN6_PTES_PER_PT)
+			last_pte = GEN6_PTES_PER_PT;
 
-		pt_vaddr = kmap_atomic(ppgtt->pt_pages[act_pt]);
+		pt_vaddr = kmap_atomic(ppgtt->pt_pages[pde]);
 
-		for (i = first_pte; i < last_pte; i++)
+		for (i = pte; i < last_pte; i++)
 			pt_vaddr[i] = scratch_pte;
 
 		kunmap_atomic(pt_vaddr);
 
-		num_entries -= last_pte - first_pte;
-		first_pte = 0;
-		act_pt++;
+		num_entries -= last_pte - pte;
+		pte = 0;
+		pde++;
 	}
 }
 
@@ -971,25 +970,23 @@ static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 	gen6_gtt_pte_t *pt_vaddr;
-	unsigned first_entry = start >> PAGE_SHIFT;
-	unsigned act_pt = first_entry / I915_PPGTT_PT_ENTRIES;
-	unsigned act_pte = first_entry % I915_PPGTT_PT_ENTRIES;
+	unsigned pde = gen6_pde_index(start);
+	unsigned pte = gen6_pte_index(start);
 	struct sg_page_iter sg_iter;
 
 	pt_vaddr = NULL;
 	for_each_sg_page(pages->sgl, &sg_iter, pages->nents, 0) {
 		if (pt_vaddr == NULL)
-			pt_vaddr = kmap_atomic(ppgtt->pt_pages[act_pt]);
+			pt_vaddr = kmap_atomic(ppgtt->pt_pages[pde]);
 
-		pt_vaddr[act_pte] =
+		pt_vaddr[pte] =
 			vm->pte_encode(sg_page_iter_dma_address(&sg_iter),
 				       cache_level, true, flags);
-
-		if (++act_pte == I915_PPGTT_PT_ENTRIES) {
+		if (++pte == GEN6_PTES_PER_PT) {
 			kunmap_atomic(pt_vaddr);
 			pt_vaddr = NULL;
-			act_pt++;
-			act_pte = 0;
+			pde++;
+			pte = 0;
 		}
 	}
 	if (pt_vaddr)
@@ -1068,7 +1065,7 @@ alloc:
 	if (ppgtt->node.start < dev_priv->gtt.mappable_end)
 		DRM_DEBUG("Forced to use aperture for PDEs\n");
 
-	ppgtt->num_pd_entries = GEN6_PPGTT_PD_ENTRIES;
+	ppgtt->num_pd_entries = I915_PDES_PER_PD;
 	return 0;
 }
 
@@ -1173,7 +1170,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->base.insert_entries = gen6_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen6_ppgtt_cleanup;
 	ppgtt->base.start = 0;
-	ppgtt->base.total =  ppgtt->num_pd_entries * I915_PPGTT_PT_ENTRIES * PAGE_SIZE;
+	ppgtt->base.total =  ppgtt->num_pd_entries * GEN6_PTES_PER_PT * PAGE_SIZE;
 	ppgtt->debug_dump = gen6_dump_ppgtt;
 
 	ppgtt->pd_offset =
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 0199c5a..465549f 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -38,8 +38,16 @@ typedef uint32_t gen6_gtt_pte_t;
 typedef uint64_t gen8_gtt_pte_t;
 typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
 
-#define I915_PPGTT_PT_ENTRIES		(PAGE_SIZE / sizeof(gen6_gtt_pte_t))
-/* gen6-hsw has bit 11-4 for physical addr bit 39-32 */
+/* GEN Agnostic defines */
+#define I915_PAGE_SIZE			4096
+#define I915_PDES_PER_PD		512
+#define I915_PTE_MASK			(I915_PAGE_SIZE-1)
+#define I915_PDE_MASK			(I915_PDES_PER_PD-1)
+
+/* GEN6 PPGTT resembles a 2 level page table:
+ * 31:22 | 21:12 |  11:0
+ *  PDE  |  PTE  | offset
+ */
 #define GEN6_GTT_ADDR_ENCODE(addr)	((addr) | (((addr) >> 28) & 0xff0))
 #define GEN6_PTE_ADDR_ENCODE(addr)	GEN6_GTT_ADDR_ENCODE(addr)
 #define GEN6_PDE_ADDR_ENCODE(addr)	GEN6_GTT_ADDR_ENCODE(addr)
@@ -47,13 +55,16 @@ typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
 #define GEN6_PTE_UNCACHED		(1 << 1)
 #define GEN6_PTE_VALID			(1 << 0)
 
-#define GEN6_PPGTT_PD_ENTRIES		512
-#define GEN6_PD_SIZE			(GEN6_PPGTT_PD_ENTRIES * PAGE_SIZE)
+#define GEN6_PD_SIZE			(I915_PDES_PER_PD * PAGE_SIZE)
 #define GEN6_PD_ALIGN			(PAGE_SIZE * 16)
 #define GEN6_PDE_VALID			(1 << 0)
 
 #define GEN7_PTE_CACHE_L3_LLC		(3 << 1)
 
+#define GEN6_PDE_SHIFT			22
+#define GEN6_PTES_PER_PT		(PAGE_SIZE / sizeof(gen6_gtt_pte_t))
+#define NUM_PTE(pde_shift)		(1 << (pde_shift - PAGE_SHIFT))
+
 #define BYT_PTE_SNOOPED_BY_CPU_CACHES	(1 << 2)
 #define BYT_PTE_WRITEABLE		(1 << 1)
 
@@ -72,6 +83,14 @@ typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
 #define HSW_GTT_ADDR_ENCODE(addr)	((addr) | (((addr) >> 28) & 0x7f0))
 #define HSW_PTE_ADDR_ENCODE(addr)	HSW_GTT_ADDR_ENCODE(addr)
 
+#define PPAT_UNCACHED_INDEX		(_PAGE_PWT | _PAGE_PCD)
+#define PPAT_CACHED_PDE_INDEX		0 /* WB LLC */
+#define PPAT_CACHED_INDEX		_PAGE_PAT /* WB LLCeLLC */
+#define PPAT_DISPLAY_ELLC_INDEX		_PAGE_PCD /* WT eLLC */
+
+#define GEN8_LEGACY_PDPES		4
+#define GEN8_PTES_PER_PT		(PAGE_SIZE / sizeof(gen8_gtt_pte_t))
+
 /* GEN8 legacy style address is defined as a 3 level page table:
  * 31:30 | 29:21 | 20:12 |  11:0
  * PDPE  |  PDE  |  PTE  | offset
@@ -81,12 +100,6 @@ typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
 #define GEN8_PDPE_SHIFT			30
 #define GEN8_PDPE_MASK			0x3
 #define GEN8_PDE_SHIFT			21
-#define GEN8_PDE_MASK			0x1ff
-#define GEN8_PTE_SHIFT			12
-#define GEN8_PTE_MASK			0x1ff
-#define GEN8_LEGACY_PDPES		4
-#define GEN8_PTES_PER_PAGE		(PAGE_SIZE / sizeof(gen8_gtt_pte_t))
-#define GEN8_PDES_PER_PAGE		(PAGE_SIZE / sizeof(gen8_ppgtt_pde_t))
 
 #define PPAT_UNCACHED_INDEX		(_PAGE_PWT | _PAGE_PCD)
 #define PPAT_CACHED_PDE_INDEX		0 /* WB LLC */
@@ -273,6 +286,96 @@ struct i915_hw_ppgtt {
 	void (*debug_dump)(struct i915_hw_ppgtt *ppgtt, struct seq_file *m);
 };
 
+static inline uint32_t i915_pte_index(uint64_t address, uint32_t pde_shift)
+{
+	const uint32_t mask = NUM_PTE(pde_shift) - 1;
+	return (address >> PAGE_SHIFT) & mask;
+}
+
+/* Helper to counts the number of PTEs within the given length. This count does
+ * not cross a page table boundary, so the max value would be
+ * GEN6_PTES_PER_PT for GEN6, and GEN8_PTES_PER_PT for GEN8.
+ */
+static inline size_t i915_pte_count(uint64_t addr, size_t length,
+				    uint32_t pde_shift)
+{
+	const uint64_t mask = ~((1 << pde_shift) - 1);
+	uint64_t end;
+
+	BUG_ON(length == 0);
+	BUG_ON(offset_in_page(addr|length));
+
+	end = addr + length;
+
+	if ((addr & mask) != (end & mask))
+		return NUM_PTE(pde_shift) - i915_pte_index(addr, pde_shift);
+
+	return i915_pte_index(end, pde_shift) - i915_pte_index(addr, pde_shift);
+}
+
+static inline uint32_t i915_pde_index(uint64_t addr, uint32_t shift)
+{
+	return (addr >> shift) & I915_PDE_MASK;
+}
+
+static inline size_t i915_pde_count(uint64_t addr, uint64_t length,
+				    uint32_t pde_shift)
+{
+	const uint32_t pdp_shift = pde_shift + 9;
+	const uint64_t mask = ~((1 << pdp_shift) - 1);
+	uint64_t end;
+
+	BUG_ON(length == 0);
+	BUG_ON(offset_in_page(addr|length));
+
+	end = addr + length;
+
+	if ((addr & mask) != (end & mask))
+		return I915_PDES_PER_PD - i915_pde_index(addr, pde_shift);
+
+	return i915_pde_index(end, pde_shift) - i915_pde_index(addr, pde_shift);
+}
+
+static inline uint32_t gen6_pte_index(uint32_t addr)
+{
+	return i915_pte_index(addr, GEN6_PDE_SHIFT);
+}
+
+static inline size_t gen6_pte_count(uint32_t addr, uint32_t length)
+{
+	return i915_pte_count(addr, length, GEN6_PDE_SHIFT);
+}
+
+static inline uint32_t gen6_pde_index(uint32_t addr)
+{
+	return i915_pde_index(addr, GEN6_PDE_SHIFT);
+}
+
+static inline size_t gen6_pde_count(uint32_t addr, uint32_t length)
+{
+	return i915_pde_count(addr, length, GEN6_PDE_SHIFT);
+}
+
+static inline uint32_t gen8_pte_index(uint64_t address)
+{
+	return i915_pte_index(address, GEN8_PDE_SHIFT);
+}
+
+static inline uint32_t gen8_pde_index(uint64_t address)
+{
+	return i915_pde_index(address, GEN8_PDE_SHIFT);
+}
+
+static inline uint32_t gen8_pdpe_index(uint64_t address)
+{
+	return (address >> GEN8_PDPE_SHIFT) & GEN8_PDPE_MASK;
+}
+
+static inline uint32_t gen8_pml4e_index(uint64_t address)
+{
+	BUG();
+}
+
 int i915_gem_gtt_init(struct drm_device *dev);
 void i915_gem_init_global_gtt(struct drm_device *dev);
 void i915_gem_setup_global_gtt(struct drm_device *dev, unsigned long start,
-- 
2.0.4

[PATCH 33/68] drm/i915: construct page table abstractions

[Ben Widawsky]

Thus far we've opted to make complex code requiring difficult review. In
the future, the code is only going to become more complex, and as such
we'll take the hit now and start to encapsulate things.

To help transition the code nicely there is some wasted space in gen6/7.
This will be ameliorated shortly.

NOTE: The pun in the subject was intentional.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>

Conflicts:
	drivers/gpu/drm/i915/i915_drv.h
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 174 ++++++++++++++++++------------------
 drivers/gpu/drm/i915/i915_gem_gtt.h |  23 +++--
 2 files changed, 104 insertions(+), 93 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 5a62ef1..b832d01 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -270,7 +270,8 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 				      I915_CACHE_LLC, use_scratch);
 
 	while (num_entries) {
-		struct page *page_table = ppgtt->gen8_pt_pages[pdpe][pde];
+		struct i915_pagedir *pd = &ppgtt->pdp.pagedir[pdpe];
+		struct page *page_table = pd->page_tables[pde].page;
 
 		last_pte = pte + num_entries;
 		if (last_pte > GEN8_PTES_PER_PT)
@@ -314,8 +315,11 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 		if (WARN_ON(pdpe >= GEN8_LEGACY_PDPES))
 			break;
 
-		if (pt_vaddr == NULL)
-			pt_vaddr = kmap_atomic(ppgtt->gen8_pt_pages[pdpe][pde]);
+		if (pt_vaddr == NULL) {
+			struct i915_pagedir *pd = &ppgtt->pdp.pagedir[pdpe];
+			struct page *page_table = pd->page_tables[pde].page;
+			pt_vaddr = kmap_atomic(page_table);
+		}
 
 		pt_vaddr[pte] =
 			gen8_pte_encode(sg_page_iter_dma_address(&sg_iter),
@@ -339,29 +343,33 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	}
 }
 
-static void gen8_free_page_tables(struct page **pt_pages)
+static void gen8_free_page_tables(struct i915_pagedir *pd)
 {
 	int i;
 
-	if (pt_pages == NULL)
+	if (pd->page_tables == NULL)
 		return;
 
 	for (i = 0; i < I915_PDES_PER_PD; i++)
-		if (pt_pages[i])
-			__free_pages(pt_pages[i], 0);
+		if (pd->page_tables[i].page)
+			__free_page(pd->page_tables[i].page);
+}
+
+static void gen8_free_page_directories(struct i915_pagedir *pd)
+{
+	kfree(pd->page_tables);
+	__free_page(pd->page);
 }
 
-static void gen8_ppgtt_free(const struct i915_hw_ppgtt *ppgtt)
+static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
 	int i;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		gen8_free_page_tables(ppgtt->gen8_pt_pages[i]);
-		kfree(ppgtt->gen8_pt_pages[i]);
+		gen8_free_page_tables(&ppgtt->pdp.pagedir[i]);
+		gen8_free_page_directories(&ppgtt->pdp.pagedir[i]);
 		kfree(ppgtt->gen8_pt_dma_addr[i]);
 	}
-
-	__free_pages(ppgtt->pd_pages, get_order(ppgtt->num_pd_pages << PAGE_SHIFT));
 }
 
 static void gen8_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
@@ -399,86 +407,73 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
 	gen8_ppgtt_free(ppgtt);
 }
 
-static struct page **__gen8_alloc_page_tables(void)
+static int gen8_ppgtt_allocate_dma(struct i915_hw_ppgtt *ppgtt)
 {
-	struct page **pt_pages;
 	int i;
 
-	pt_pages = kcalloc(I915_PDES_PER_PD, sizeof(struct page *), GFP_KERNEL);
-	if (!pt_pages)
-		return ERR_PTR(-ENOMEM);
-
-	for (i = 0; i < I915_PDES_PER_PD; i++) {
-		pt_pages[i] = alloc_page(GFP_KERNEL);
-		if (!pt_pages[i])
-			goto bail;
+	for (i = 0; i < ppgtt->num_pd_pages; i++) {
+		ppgtt->gen8_pt_dma_addr[i] = kcalloc(I915_PDES_PER_PD,
+						     sizeof(dma_addr_t),
+						     GFP_KERNEL);
+		if (!ppgtt->gen8_pt_dma_addr[i])
+			return -ENOMEM;
 	}
 
-	return pt_pages;
-
-bail:
-	gen8_free_page_tables(pt_pages);
-	kfree(pt_pages);
-	return ERR_PTR(-ENOMEM);
+	return 0;
 }
 
-static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt,
-					   const int max_pdp)
+static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 {
-	struct page **pt_pages[GEN8_LEGACY_PDPES];
-	int i, ret;
+	int i, j;
 
-	for (i = 0; i < max_pdp; i++) {
-		pt_pages[i] = __gen8_alloc_page_tables();
-		if (IS_ERR(pt_pages[i])) {
-			ret = PTR_ERR(pt_pages[i]);
-			goto unwind_out;
+	for (i = 0; i < ppgtt->num_pd_pages; i++) {
+		for (j = 0; j < I915_PDES_PER_PD; j++) {
+			struct i915_pagetab *pt = &ppgtt->pdp.pagedir[i].page_tables[j];
+			pt->page = alloc_page(GFP_KERNEL | __GFP_ZERO);
+			if (!pt->page)
+				goto unwind_out;
 		}
 	}
 
-	/* NB: Avoid touching gen8_pt_pages until last to keep the allocation,
-	 * "atomic" - for cleanup purposes.
-	 */
-	for (i = 0; i < max_pdp; i++)
-		ppgtt->gen8_pt_pages[i] = pt_pages[i];
-
 	return 0;
 
 unwind_out:
-	while (i--) {
-		gen8_free_page_tables(pt_pages[i]);
-		kfree(pt_pages[i]);
-	}
+	while (i--)
+		gen8_free_page_tables(&ppgtt->pdp.pagedir[i]);
 
-	return ret;
+	return -ENOMEM;
 }
 
-static int gen8_ppgtt_allocate_dma(struct i915_hw_ppgtt *ppgtt)
+static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
+						const int max_pdp)
 {
 	int i;
 
-	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		ppgtt->gen8_pt_dma_addr[i] = kcalloc(I915_PDES_PER_PD,
-						     sizeof(dma_addr_t),
-						     GFP_KERNEL);
-		if (!ppgtt->gen8_pt_dma_addr[i])
-			return -ENOMEM;
-	}
+	for (i = 0; i < max_pdp; i++) {
+		struct i915_pagetab *pt;
+		pt = kcalloc(I915_PDES_PER_PD, sizeof(*pt), GFP_KERNEL);
+		if (!pt)
+			goto unwind_out;
 
-	return 0;
-}
+		ppgtt->pdp.pagedir[i].page = alloc_page(GFP_KERNEL);
+		if (!ppgtt->pdp.pagedir[i].page)
+			goto unwind_out;
 
-static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
-						const int max_pdp)
-{
-	ppgtt->pd_pages = alloc_pages(GFP_KERNEL, get_order(max_pdp << PAGE_SHIFT));
-	if (!ppgtt->pd_pages)
-		return -ENOMEM;
+		ppgtt->pdp.pagedir[i].page_tables = pt;
+	}
 
-	ppgtt->num_pd_pages = 1 << get_order(max_pdp << PAGE_SHIFT);
+	ppgtt->num_pd_pages = max_pdp;
 	BUG_ON(ppgtt->num_pd_pages > GEN8_LEGACY_PDPES);
 
 	return 0;
+
+unwind_out:
+	while (i--) {
+		kfree(ppgtt->pdp.pagedir[i].page_tables);
+		__free_page(ppgtt->pdp.pagedir[i].page);
+	}
+
+	return -ENOMEM;
 }
 
 static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
@@ -490,18 +485,19 @@ static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
 	if (ret)
 		return ret;
 
-	ret = gen8_ppgtt_allocate_page_tables(ppgtt, max_pdp);
-	if (ret) {
-		__free_pages(ppgtt->pd_pages, get_order(max_pdp << PAGE_SHIFT));
-		return ret;
-	}
+	ret = gen8_ppgtt_allocate_page_tables(ppgtt);
+	if (ret)
+		goto err_out;
 
 	ppgtt->num_pd_entries = max_pdp * I915_PDES_PER_PD;
 
 	ret = gen8_ppgtt_allocate_dma(ppgtt);
-	if (ret)
-		gen8_ppgtt_free(ppgtt);
+	if (!ret)
+		return ret;
 
+	/* TODO: Check this for all cases */
+err_out:
+	gen8_ppgtt_free(ppgtt);
 	return ret;
 }
 
@@ -512,7 +508,7 @@ static int gen8_ppgtt_setup_page_directories(struct i915_hw_ppgtt *ppgtt,
 	int ret;
 
 	pd_addr = pci_map_page(ppgtt->base.dev->pdev,
-			       &ppgtt->pd_pages[pdpe], 0,
+			       ppgtt->pdp.pagedir[pdpe].page, 0,
 			       PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
 
 	ret = pci_dma_mapping_error(ppgtt->base.dev->pdev, pd_addr);
@@ -532,7 +528,7 @@ static int gen8_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt,
 	struct page *p;
 	int ret;
 
-	p = ppgtt->gen8_pt_pages[pdpe][pde];
+	p = ppgtt->pdp.pagedir[pdpe].page_tables[pde].page;
 	pt_addr = pci_map_page(ppgtt->base.dev->pdev,
 			       p, 0, PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
 	ret = pci_dma_mapping_error(ppgtt->base.dev->pdev, pt_addr);
@@ -593,7 +589,7 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	 */
 	for (i = 0; i < max_pdp; i++) {
 		gen8_ppgtt_pde_t *pd_vaddr;
-		pd_vaddr = kmap_atomic(&ppgtt->pd_pages[i]);
+		pd_vaddr = kmap_atomic(ppgtt->pdp.pagedir[i].page);
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
 			dma_addr_t addr = ppgtt->gen8_pt_dma_addr[i][j];
 			pd_vaddr[j] = gen8_pde_encode(ppgtt->base.dev, addr,
@@ -655,7 +651,7 @@ static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 				   expected);
 		seq_printf(m, "\tPDE: %x\n", pd_entry);
 
-		pt_vaddr = kmap_atomic(ppgtt->pt_pages[pde]);
+		pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde].page);
 		for (pte = 0; pte < GEN6_PTES_PER_PT; pte+=4) {
 			unsigned long va =
 				(pde * PAGE_SIZE * GEN6_PTES_PER_PT) +
@@ -949,7 +945,7 @@ static void gen6_ppgtt_clear_range(struct i915_address_space *vm,
 		if (last_pte > GEN6_PTES_PER_PT)
 			last_pte = GEN6_PTES_PER_PT;
 
-		pt_vaddr = kmap_atomic(ppgtt->pt_pages[pde]);
+		pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde].page);
 
 		for (i = pte; i < last_pte; i++)
 			pt_vaddr[i] = scratch_pte;
@@ -977,7 +973,7 @@ static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 	pt_vaddr = NULL;
 	for_each_sg_page(pages->sgl, &sg_iter, pages->nents, 0) {
 		if (pt_vaddr == NULL)
-			pt_vaddr = kmap_atomic(ppgtt->pt_pages[pde]);
+			pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde].page);
 
 		pt_vaddr[pte] =
 			vm->pte_encode(sg_page_iter_dma_address(&sg_iter),
@@ -1011,8 +1007,8 @@ static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 
 	kfree(ppgtt->pt_dma_addr);
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
-		__free_page(ppgtt->pt_pages[i]);
-	kfree(ppgtt->pt_pages);
+		__free_page(ppgtt->pd.page_tables[i].page);
+	kfree(ppgtt->pd.page_tables);
 }
 
 static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
@@ -1071,22 +1067,22 @@ alloc:
 
 static int gen6_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 {
+	struct i915_pagetab *pt;
 	int i;
 
-	ppgtt->pt_pages = kcalloc(ppgtt->num_pd_entries, sizeof(struct page *),
-				  GFP_KERNEL);
-
-	if (!ppgtt->pt_pages)
+	pt = kcalloc(ppgtt->num_pd_entries, sizeof(*pt), GFP_KERNEL);
+	if (!pt)
 		return -ENOMEM;
 
 	for (i = 0; i < ppgtt->num_pd_entries; i++) {
-		ppgtt->pt_pages[i] = alloc_page(GFP_KERNEL);
-		if (!ppgtt->pt_pages[i]) {
+		pt[i].page = alloc_page(GFP_KERNEL);
+		if (!pt->page) {
 			gen6_ppgtt_free(ppgtt);
 			return -ENOMEM;
 		}
 	}
 
+	ppgtt->pd.page_tables = pt;
 	return 0;
 }
 
@@ -1121,9 +1117,11 @@ static int gen6_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt)
 	int i;
 
 	for (i = 0; i < ppgtt->num_pd_entries; i++) {
+		struct page *page;
 		dma_addr_t pt_addr;
 
-		pt_addr = pci_map_page(dev->pdev, ppgtt->pt_pages[i], 0, 4096,
+		page = ppgtt->pd.page_tables[i].page;
+		pt_addr = pci_map_page(dev->pdev, page, 0, 4096,
 				       PCI_DMA_BIDIRECTIONAL);
 
 		if (pci_dma_mapping_error(dev->pdev, pt_addr)) {
@@ -1170,7 +1168,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->base.insert_entries = gen6_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen6_ppgtt_cleanup;
 	ppgtt->base.start = 0;
-	ppgtt->base.total =  ppgtt->num_pd_entries * GEN6_PTES_PER_PT * PAGE_SIZE;
+	ppgtt->base.total = ppgtt->num_pd_entries * GEN6_PTES_PER_PT * PAGE_SIZE;
 	ppgtt->debug_dump = gen6_dump_ppgtt;
 
 	ppgtt->pd_offset =
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 465549f..ee1c317 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -257,6 +257,20 @@ struct i915_gtt {
 			  unsigned long *mappable_end);
 };
 
+struct i915_pagetab {
+	struct page *page;
+};
+
+struct i915_pagedir {
+	struct page *page; /* NULL for GEN6-GEN7 */
+	struct i915_pagetab *page_tables;
+};
+
+struct i915_pagedirpo {
+	/* struct page *page; */
+	struct i915_pagedir pagedir[GEN8_LEGACY_PDPES];
+};
+
 struct i915_hw_ppgtt {
 	struct i915_address_space base;
 	struct kref ref;
@@ -264,11 +278,6 @@ struct i915_hw_ppgtt {
 	unsigned num_pd_entries;
 	unsigned num_pd_pages; /* gen8+ */
 	union {
-		struct page **pt_pages;
-		struct page **gen8_pt_pages[GEN8_LEGACY_PDPES];
-	};
-	struct page *pd_pages;
-	union {
 		uint32_t pd_offset;
 		dma_addr_t pd_dma_addr[GEN8_LEGACY_PDPES];
 	};
@@ -276,6 +285,10 @@ struct i915_hw_ppgtt {
 		dma_addr_t *pt_dma_addr;
 		dma_addr_t *gen8_pt_dma_addr[GEN8_LEGACY_PDPES];
 	};
+	union {
+		struct i915_pagedirpo pdp;
+		struct i915_pagedir pd;
+	};
 
 	struct intel_context *ctx;
 
-- 
2.0.4

[PATCH 34/68] drm/i915: Complete page table structures

[Ben Widawsky]

Move the remaining members over to the new page table structures.

This can be squashed with the previous commit if desire. The reasoning
is the same as that patch. I simply felt it is easier to review if split.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>

Conflicts:
	drivers/gpu/drm/i915/i915_drv.h
	drivers/gpu/drm/i915/i915_gem_gtt.c
---
 drivers/gpu/drm/i915/i915_debugfs.c |  2 +-
 drivers/gpu/drm/i915/i915_gem_gtt.c | 85 +++++++++++++------------------------
 drivers/gpu/drm/i915/i915_gem_gtt.h | 14 +++---
 3 files changed, 37 insertions(+), 64 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index d2977cf..ccecf00 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1813,7 +1813,7 @@ static int i915_swizzle_info(struct seq_file *m, void *data)
 
 static void print_ppgtt(struct seq_file *m, struct i915_hw_ppgtt *ppgtt)
 {
-	seq_printf(m, "pd gtt offset: 0x%08x\n", ppgtt->pd_offset);
+	seq_printf(m, "pd gtt offset: 0x%08x\n", ppgtt->pd.pd_offset);
 }
 
 static void gen8_ppgtt_info(struct seq_file *m, struct drm_device *dev, int verbose)
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index b832d01..c9d3d3c 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -243,7 +243,7 @@ static int gen8_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	int used_pd = ppgtt->num_pd_entries / I915_PDES_PER_PD;
 
 	for (i = used_pd - 1; i >= 0; i--) {
-		dma_addr_t addr = ppgtt->pd_dma_addr[i];
+		dma_addr_t addr = ppgtt->pdp.pagedir[i].daddr;
 		ret = gen8_write_pdp(ring, i, addr, synchronous);
 		if (ret)
 			return ret;
@@ -368,7 +368,6 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
 		gen8_free_page_tables(&ppgtt->pdp.pagedir[i]);
 		gen8_free_page_directories(&ppgtt->pdp.pagedir[i]);
-		kfree(ppgtt->gen8_pt_dma_addr[i]);
 	}
 }
 
@@ -380,14 +379,14 @@ static void gen8_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
 		/* TODO: In the future we'll support sparse mappings, so this
 		 * will have to change. */
-		if (!ppgtt->pd_dma_addr[i])
+		if (!ppgtt->pdp.pagedir[i].daddr)
 			continue;
 
-		pci_unmap_page(hwdev, ppgtt->pd_dma_addr[i], PAGE_SIZE,
+		pci_unmap_page(hwdev, ppgtt->pdp.pagedir[i].daddr, PAGE_SIZE,
 			       PCI_DMA_BIDIRECTIONAL);
 
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			dma_addr_t addr = ppgtt->gen8_pt_dma_addr[i][j];
+			dma_addr_t addr = ppgtt->pdp.pagedir[i].page_tables[j].daddr;
 			if (addr)
 				pci_unmap_page(hwdev, addr, PAGE_SIZE,
 					       PCI_DMA_BIDIRECTIONAL);
@@ -407,31 +406,18 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
 	gen8_ppgtt_free(ppgtt);
 }
 
-static int gen8_ppgtt_allocate_dma(struct i915_hw_ppgtt *ppgtt)
-{
-	int i;
-
-	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		ppgtt->gen8_pt_dma_addr[i] = kcalloc(I915_PDES_PER_PD,
-						     sizeof(dma_addr_t),
-						     GFP_KERNEL);
-		if (!ppgtt->gen8_pt_dma_addr[i])
-			return -ENOMEM;
-	}
-
-	return 0;
-}
-
 static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 {
 	int i, j;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
+		struct i915_pagedir *pd = &ppgtt->pdp.pagedir[i];
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			struct i915_pagetab *pt = &ppgtt->pdp.pagedir[i].page_tables[j];
+			struct i915_pagetab *pt = &pd->page_tables[j];
 			pt->page = alloc_page(GFP_KERNEL | __GFP_ZERO);
 			if (!pt->page)
 				goto unwind_out;
+
 		}
 	}
 
@@ -491,9 +477,7 @@ static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
 
 	ppgtt->num_pd_entries = max_pdp * I915_PDES_PER_PD;
 
-	ret = gen8_ppgtt_allocate_dma(ppgtt);
-	if (!ret)
-		return ret;
+	return 0;
 
 	/* TODO: Check this for all cases */
 err_out:
@@ -515,7 +499,7 @@ static int gen8_ppgtt_setup_page_directories(struct i915_hw_ppgtt *ppgtt,
 	if (ret)
 		return ret;
 
-	ppgtt->pd_dma_addr[pdpe] = pd_addr;
+	ppgtt->pdp.pagedir[pdpe].daddr = pd_addr;
 
 	return 0;
 }
@@ -525,17 +509,18 @@ static int gen8_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt,
 					const int pde)
 {
 	dma_addr_t pt_addr;
-	struct page *p;
+	struct i915_pagedir *pd = &ppgtt->pdp.pagedir[pdpe];
+	struct i915_pagetab *pt = &pd->page_tables[pde];
+	struct page *p = pt->page;
 	int ret;
 
-	p = ppgtt->pdp.pagedir[pdpe].page_tables[pde].page;
 	pt_addr = pci_map_page(ppgtt->base.dev->pdev,
 			       p, 0, PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
 	ret = pci_dma_mapping_error(ppgtt->base.dev->pdev, pt_addr);
 	if (ret)
 		return ret;
 
-	ppgtt->gen8_pt_dma_addr[pdpe][pde] = pt_addr;
+	pt->daddr = pt_addr;
 
 	return 0;
 }
@@ -591,7 +576,7 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 		gen8_ppgtt_pde_t *pd_vaddr;
 		pd_vaddr = kmap_atomic(ppgtt->pdp.pagedir[i].page);
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			dma_addr_t addr = ppgtt->gen8_pt_dma_addr[i][j];
+			dma_addr_t addr = ppgtt->pdp.pagedir[i].page_tables[j].daddr;
 			pd_vaddr[j] = gen8_pde_encode(ppgtt->base.dev, addr,
 						      I915_CACHE_LLC);
 		}
@@ -633,14 +618,15 @@ static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 	scratch_pte = vm->pte_encode(vm->scratch.addr, I915_CACHE_LLC, true, 0);
 
 	pd_addr = (gen6_gtt_pte_t __iomem *)dev_priv->gtt.gsm +
-		ppgtt->pd_offset / sizeof(gen6_gtt_pte_t);
+		ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t);
 
 	seq_printf(m, "  VM %p (pd_offset %x-%x):\n", vm,
-		   ppgtt->pd_offset, ppgtt->pd_offset + ppgtt->num_pd_entries);
+		   ppgtt->pd.pd_offset,
+		   ppgtt->pd.pd_offset + ppgtt->num_pd_entries);
 	for (pde = 0; pde < ppgtt->num_pd_entries; pde++) {
 		u32 expected;
 		gen6_gtt_pte_t *pt_vaddr;
-		dma_addr_t pt_addr = ppgtt->pt_dma_addr[pde];
+		dma_addr_t pt_addr = ppgtt->pd.page_tables[pde].daddr;
 		pd_entry = readl(pd_addr + pde);
 		expected = (GEN6_PDE_ADDR_ENCODE(pt_addr) | GEN6_PDE_VALID);
 
@@ -683,8 +669,8 @@ static void gen6_map_single(struct i915_hw_ppgtt *ppgtt,
 {
 	struct drm_i915_private *dev_priv = ppgtt->base.dev->dev_private;
 	uint32_t pd_entry;
-	gen6_gtt_pte_t __iomem *pd_addr =
-		(gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm + ppgtt->pd_offset / sizeof(gen6_gtt_pte_t);
+	gen6_gtt_pte_t __iomem *pd_addr = (gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm;
+	pd_addr	+= ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t);
 
 	pd_entry = GEN6_PDE_ADDR_ENCODE(daddr);
 	pd_entry |= GEN6_PDE_VALID;
@@ -699,18 +685,18 @@ static void gen6_map_page_tables(struct i915_hw_ppgtt *ppgtt)
 	struct drm_i915_private *dev_priv = ppgtt->base.dev->dev_private;
 	int i;
 
-	WARN_ON(ppgtt->pd_offset & 0x3f);
+	WARN_ON(ppgtt->pd.pd_offset & 0x3f);
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
-		gen6_map_single(ppgtt, i, ppgtt->pt_dma_addr[i]);
+		gen6_map_single(ppgtt, i, ppgtt->pd.page_tables[i].daddr);
 
 	readl(dev_priv->gtt.gsm);
 }
 
 static uint32_t get_pd_offset(struct i915_hw_ppgtt *ppgtt)
 {
-	BUG_ON(ppgtt->pd_offset & 0x3f);
+	BUG_ON(ppgtt->pd.pd_offset & 0x3f);
 
-	return (ppgtt->pd_offset / 64) << 16;
+	return (ppgtt->pd.pd_offset / 64) << 16;
 }
 
 static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,
@@ -993,19 +979,16 @@ static void gen6_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 {
 	int i;
 
-	if (ppgtt->pt_dma_addr) {
-		for (i = 0; i < ppgtt->num_pd_entries; i++)
-			pci_unmap_page(ppgtt->base.dev->pdev,
-				       ppgtt->pt_dma_addr[i],
-				       4096, PCI_DMA_BIDIRECTIONAL);
-	}
+	for (i = 0; i < ppgtt->num_pd_entries; i++)
+		pci_unmap_page(ppgtt->base.dev->pdev,
+			       ppgtt->pd.page_tables[i].daddr,
+			       4096, PCI_DMA_BIDIRECTIONAL);
 }
 
 static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
 	int i;
 
-	kfree(ppgtt->pt_dma_addr);
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
 		__free_page(ppgtt->pd.page_tables[i].page);
 	kfree(ppgtt->pd.page_tables);
@@ -1100,14 +1083,6 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
 		return ret;
 	}
 
-	ppgtt->pt_dma_addr = kcalloc(ppgtt->num_pd_entries, sizeof(dma_addr_t),
-				     GFP_KERNEL);
-	if (!ppgtt->pt_dma_addr) {
-		drm_mm_remove_node(&ppgtt->node);
-		gen6_ppgtt_free(ppgtt);
-		return -ENOMEM;
-	}
-
 	return 0;
 }
 
@@ -1129,7 +1104,7 @@ static int gen6_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt)
 			return -EIO;
 		}
 
-		ppgtt->pt_dma_addr[i] = pt_addr;
+		ppgtt->pd.page_tables[i].daddr = pt_addr;
 	}
 
 	return 0;
@@ -1171,7 +1146,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->base.total = ppgtt->num_pd_entries * GEN6_PTES_PER_PT * PAGE_SIZE;
 	ppgtt->debug_dump = gen6_dump_ppgtt;
 
-	ppgtt->pd_offset =
+	ppgtt->pd.pd_offset =
 		ppgtt->node.start / PAGE_SIZE * sizeof(gen6_gtt_pte_t);
 
 	gen6_map_page_tables(ppgtt);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index ee1c317..2e776ee 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -259,10 +259,16 @@ struct i915_gtt {
 
 struct i915_pagetab {
 	struct page *page;
+	dma_addr_t daddr;
 };
 
 struct i915_pagedir {
 	struct page *page; /* NULL for GEN6-GEN7 */
+	union {
+		uint32_t pd_offset;
+		dma_addr_t daddr;
+	};
+
 	struct i915_pagetab *page_tables;
 };
 
@@ -278,14 +284,6 @@ struct i915_hw_ppgtt {
 	unsigned num_pd_entries;
 	unsigned num_pd_pages; /* gen8+ */
 	union {
-		uint32_t pd_offset;
-		dma_addr_t pd_dma_addr[GEN8_LEGACY_PDPES];
-	};
-	union {
-		dma_addr_t *pt_dma_addr;
-		dma_addr_t *gen8_pt_dma_addr[GEN8_LEGACY_PDPES];
-	};
-	union {
 		struct i915_pagedirpo pdp;
 		struct i915_pagedir pd;
 	};
-- 
2.0.4

[PATCH 35/68] drm/i915: Create page table allocators

[Ben Widawsky]

As we move toward dynamic page table allocation, it becomes much easier
to manage our data structures if break do things less coarsely by
breaking up all of our actions into individual tasks.  This makes the
code easier to write, read, and verify.

Aside from the dissection of the allocation functions, the patch
statically allocates the page table structures without a page directory.
This remains the same for all platforms,

The patch itself should not have much functional difference. The primary
noticeable difference is the fact that page tables are no longer
allocated, but rather statically declared as part of the page directory.
This has non-zero overhead, but things gain non-trivial complexity as a
result.

This patch exists for a few reasons:
1. Splitting out the functions allows easily combining GEN6 and GEN8
code. Page tables have no difference based on GEN8. As we'll see in a
future patch when we add the DMA mappings to the allocations, it
requires only one small change to make work, and error handling should
just fall into place.

2. Unless we always want to allocate all page tables under a given PDE,
we'll have to eventually break this up into an array of pointers (or
pointer to pointer).

3. Having the discrete functions is easier to review, and understand.
All allocations and frees now take place in just a couple of locations.
Reviewing, and catching leaks should be easy.

4. Less important: the GFP flags are confined to one location, which
makes playing around with such things trivial.

v2: Updated commit message to explain why this patch exists

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 226 +++++++++++++++++++++++-------------
 drivers/gpu/drm/i915/i915_gem_gtt.h |   4 +-
 2 files changed, 147 insertions(+), 83 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index c9d3d3c..532e560 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -203,6 +203,102 @@ static gen6_gtt_pte_t iris_pte_encode(dma_addr_t addr,
 	return pte;
 }
 
+static void free_pt_single(struct i915_pagetab *pt)
+{
+	if (WARN_ON(!pt->page))
+		return;
+	__free_page(pt->page);
+	kfree(pt);
+}
+
+static struct i915_pagetab *alloc_pt_single(void)
+{
+	struct i915_pagetab *pt;
+
+	pt = kzalloc(sizeof(*pt), GFP_KERNEL);
+	if (!pt)
+		return ERR_PTR(-ENOMEM);
+
+	pt->page = alloc_page(GFP_KERNEL | __GFP_ZERO);
+	if (!pt->page) {
+		kfree(pt);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	return pt;
+}
+
+/**
+ * alloc_pt_range() - Allocate a multiple page tables
+ * @pd:		The page directory which will have at least @count entries
+ *		available to point to the allocated page tables.
+ * @pde:	First page directory entry for which we are allocating.
+ * @count:	Number of pages to allocate.
+ *
+ * Allocates multiple page table pages and sets the appropriate entries in the
+ * page table structure within the page directory. Function cleans up after
+ * itself on any failures.
+ *
+ * Return: 0 if allocation succeeded.
+ */
+static int alloc_pt_range(struct i915_pagedir *pd, uint16_t pde, size_t count)
+{
+	int i, ret;
+
+	/* 512 is the max page tables per pagedir on any platform.
+	 * TODO: make WARN after patch series is done
+	 */
+	BUG_ON(pde + count > I915_PDES_PER_PD);
+
+	for (i = pde; i < pde + count; i++) {
+		struct i915_pagetab *pt = alloc_pt_single();
+		if (IS_ERR(pt)) {
+			ret = PTR_ERR(pt);
+			goto err_out;
+		}
+		WARN(pd->page_tables[i],
+		     "Leaking page directory entry %d (%pa)\n",
+		     i, pd->page_tables[i]);
+		pd->page_tables[i] = pt;
+	}
+
+	return 0;
+
+err_out:
+	while (i--)
+		free_pt_single(pd->page_tables[i]);
+	return ret;
+}
+
+static void __free_pd_single(struct i915_pagedir *pd)
+{
+	__free_page(pd->page);
+	kfree(pd);
+}
+
+#define free_pd_single(pd) do { \
+	if ((pd)->page) { \
+		__free_pd_single(pd); \
+	} \
+} while (0)
+
+static struct i915_pagedir *alloc_pd_single(void)
+{
+	struct i915_pagedir *pd;
+
+	pd = kzalloc(sizeof(*pd), GFP_KERNEL);
+	if (!pd)
+		return ERR_PTR(-ENOMEM);
+
+	pd->page = alloc_page(GFP_KERNEL | __GFP_ZERO);
+	if (!pd->page) {
+		kfree(pd);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	return pd;
+}
+
 /* Broadwell Page Directory Pointer Descriptors */
 static int gen8_write_pdp(struct intel_engine_cs *ring, unsigned entry,
 			   uint64_t val, bool synchronous)
@@ -243,7 +339,7 @@ static int gen8_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	int used_pd = ppgtt->num_pd_entries / I915_PDES_PER_PD;
 
 	for (i = used_pd - 1; i >= 0; i--) {
-		dma_addr_t addr = ppgtt->pdp.pagedir[i].daddr;
+		dma_addr_t addr = ppgtt->pdp.pagedir[i]->daddr;
 		ret = gen8_write_pdp(ring, i, addr, synchronous);
 		if (ret)
 			return ret;
@@ -270,8 +366,9 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 				      I915_CACHE_LLC, use_scratch);
 
 	while (num_entries) {
-		struct i915_pagedir *pd = &ppgtt->pdp.pagedir[pdpe];
-		struct page *page_table = pd->page_tables[pde].page;
+		struct i915_pagedir *pd = ppgtt->pdp.pagedir[pdpe];
+		struct i915_pagetab *pt = pd->page_tables[pde];
+		struct page *page_table = pt->page;
 
 		last_pte = pte + num_entries;
 		if (last_pte > GEN8_PTES_PER_PT)
@@ -316,8 +413,9 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 			break;
 
 		if (pt_vaddr == NULL) {
-			struct i915_pagedir *pd = &ppgtt->pdp.pagedir[pdpe];
-			struct page *page_table = pd->page_tables[pde].page;
+			struct i915_pagedir *pd = ppgtt->pdp.pagedir[pdpe];
+			struct i915_pagetab *pt = pd->page_tables[pde];
+			struct page *page_table = pt->page;
 			pt_vaddr = kmap_atomic(page_table);
 		}
 
@@ -347,18 +445,13 @@ static void gen8_free_page_tables(struct i915_pagedir *pd)
 {
 	int i;
 
-	if (pd->page_tables == NULL)
+	if (!pd->page)
 		return;
 
-	for (i = 0; i < I915_PDES_PER_PD; i++)
-		if (pd->page_tables[i].page)
-			__free_page(pd->page_tables[i].page);
-}
-
-static void gen8_free_page_directories(struct i915_pagedir *pd)
-{
-	kfree(pd->page_tables);
-	__free_page(pd->page);
+	for (i = 0; i < I915_PDES_PER_PD; i++) {
+		free_pt_single(pd->page_tables[i]);
+		pd->page_tables[i] = NULL;
+	}
 }
 
 static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
@@ -366,8 +459,8 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	int i;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		gen8_free_page_tables(&ppgtt->pdp.pagedir[i]);
-		gen8_free_page_directories(&ppgtt->pdp.pagedir[i]);
+		gen8_free_page_tables(ppgtt->pdp.pagedir[i]);
+		free_pd_single(ppgtt->pdp.pagedir[i]);
 	}
 }
 
@@ -379,14 +472,16 @@ static void gen8_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
 		/* TODO: In the future we'll support sparse mappings, so this
 		 * will have to change. */
-		if (!ppgtt->pdp.pagedir[i].daddr)
+		if (!ppgtt->pdp.pagedir[i]->daddr)
 			continue;
 
-		pci_unmap_page(hwdev, ppgtt->pdp.pagedir[i].daddr, PAGE_SIZE,
+		pci_unmap_page(hwdev, ppgtt->pdp.pagedir[i]->daddr, PAGE_SIZE,
 			       PCI_DMA_BIDIRECTIONAL);
 
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			dma_addr_t addr = ppgtt->pdp.pagedir[i].page_tables[j].daddr;
+			struct i915_pagedir *pd = ppgtt->pdp.pagedir[i];
+			struct i915_pagetab *pt =  pd->page_tables[j];
+			dma_addr_t addr = pt->daddr;
 			if (addr)
 				pci_unmap_page(hwdev, addr, PAGE_SIZE,
 					       PCI_DMA_BIDIRECTIONAL);
@@ -408,24 +503,20 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
 
 static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 {
-	int i, j;
+	int i, ret;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		struct i915_pagedir *pd = &ppgtt->pdp.pagedir[i];
-		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			struct i915_pagetab *pt = &pd->page_tables[j];
-			pt->page = alloc_page(GFP_KERNEL | __GFP_ZERO);
-			if (!pt->page)
-				goto unwind_out;
-
-		}
+		ret = alloc_pt_range(ppgtt->pdp.pagedir[i],
+				     0, I915_PDES_PER_PD);
+		if (ret)
+			goto unwind_out;
 	}
 
 	return 0;
 
 unwind_out:
 	while (i--)
-		gen8_free_page_tables(&ppgtt->pdp.pagedir[i]);
+		gen8_free_page_tables(ppgtt->pdp.pagedir[i]);
 
 	return -ENOMEM;
 }
@@ -436,16 +527,9 @@ static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
 	int i;
 
 	for (i = 0; i < max_pdp; i++) {
-		struct i915_pagetab *pt;
-		pt = kcalloc(I915_PDES_PER_PD, sizeof(*pt), GFP_KERNEL);
-		if (!pt)
+		ppgtt->pdp.pagedir[i] = alloc_pd_single();
+		if (IS_ERR(ppgtt->pdp.pagedir[i]))
 			goto unwind_out;
-
-		ppgtt->pdp.pagedir[i].page = alloc_page(GFP_KERNEL);
-		if (!ppgtt->pdp.pagedir[i].page)
-			goto unwind_out;
-
-		ppgtt->pdp.pagedir[i].page_tables = pt;
 	}
 
 	ppgtt->num_pd_pages = max_pdp;
@@ -454,10 +538,8 @@ static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
 	return 0;
 
 unwind_out:
-	while (i--) {
-		kfree(ppgtt->pdp.pagedir[i].page_tables);
-		__free_page(ppgtt->pdp.pagedir[i].page);
-	}
+	while (i--)
+		free_pd_single(ppgtt->pdp.pagedir[i]);
 
 	return -ENOMEM;
 }
@@ -492,14 +574,14 @@ static int gen8_ppgtt_setup_page_directories(struct i915_hw_ppgtt *ppgtt,
 	int ret;
 
 	pd_addr = pci_map_page(ppgtt->base.dev->pdev,
-			       ppgtt->pdp.pagedir[pdpe].page, 0,
+			       ppgtt->pdp.pagedir[pdpe]->page, 0,
 			       PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
 
 	ret = pci_dma_mapping_error(ppgtt->base.dev->pdev, pd_addr);
 	if (ret)
 		return ret;
 
-	ppgtt->pdp.pagedir[pdpe].daddr = pd_addr;
+	ppgtt->pdp.pagedir[pdpe]->daddr = pd_addr;
 
 	return 0;
 }
@@ -509,8 +591,8 @@ static int gen8_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt,
 					const int pde)
 {
 	dma_addr_t pt_addr;
-	struct i915_pagedir *pd = &ppgtt->pdp.pagedir[pdpe];
-	struct i915_pagetab *pt = &pd->page_tables[pde];
+	struct i915_pagedir *pd = ppgtt->pdp.pagedir[pdpe];
+	struct i915_pagetab *pt = pd->page_tables[pde];
 	struct page *p = pt->page;
 	int ret;
 
@@ -573,10 +655,12 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	 * will never need to touch the PDEs again.
 	 */
 	for (i = 0; i < max_pdp; i++) {
+		struct i915_pagedir *pd = ppgtt->pdp.pagedir[i];
 		gen8_ppgtt_pde_t *pd_vaddr;
-		pd_vaddr = kmap_atomic(ppgtt->pdp.pagedir[i].page);
+		pd_vaddr = kmap_atomic(ppgtt->pdp.pagedir[i]->page);
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			dma_addr_t addr = ppgtt->pdp.pagedir[i].page_tables[j].daddr;
+			struct i915_pagetab *pt = pd->page_tables[j];
+			dma_addr_t addr = pt->daddr;
 			pd_vaddr[j] = gen8_pde_encode(ppgtt->base.dev, addr,
 						      I915_CACHE_LLC);
 		}
@@ -626,7 +710,7 @@ static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 	for (pde = 0; pde < ppgtt->num_pd_entries; pde++) {
 		u32 expected;
 		gen6_gtt_pte_t *pt_vaddr;
-		dma_addr_t pt_addr = ppgtt->pd.page_tables[pde].daddr;
+		dma_addr_t pt_addr = ppgtt->pd.page_tables[pde]->daddr;
 		pd_entry = readl(pd_addr + pde);
 		expected = (GEN6_PDE_ADDR_ENCODE(pt_addr) | GEN6_PDE_VALID);
 
@@ -637,7 +721,7 @@ static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 				   expected);
 		seq_printf(m, "\tPDE: %x\n", pd_entry);
 
-		pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde].page);
+		pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde]->page);
 		for (pte = 0; pte < GEN6_PTES_PER_PT; pte+=4) {
 			unsigned long va =
 				(pde * PAGE_SIZE * GEN6_PTES_PER_PT) +
@@ -687,7 +771,7 @@ static void gen6_map_page_tables(struct i915_hw_ppgtt *ppgtt)
 
 	WARN_ON(ppgtt->pd.pd_offset & 0x3f);
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
-		gen6_map_single(ppgtt, i, ppgtt->pd.page_tables[i].daddr);
+		gen6_map_single(ppgtt, i, ppgtt->pd.page_tables[i]->daddr);
 
 	readl(dev_priv->gtt.gsm);
 }
@@ -931,7 +1015,7 @@ static void gen6_ppgtt_clear_range(struct i915_address_space *vm,
 		if (last_pte > GEN6_PTES_PER_PT)
 			last_pte = GEN6_PTES_PER_PT;
 
-		pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde].page);
+		pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde]->page);
 
 		for (i = pte; i < last_pte; i++)
 			pt_vaddr[i] = scratch_pte;
@@ -959,7 +1043,7 @@ static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 	pt_vaddr = NULL;
 	for_each_sg_page(pages->sgl, &sg_iter, pages->nents, 0) {
 		if (pt_vaddr == NULL)
-			pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde].page);
+			pt_vaddr = kmap_atomic(ppgtt->pd.page_tables[pde]->page);
 
 		pt_vaddr[pte] =
 			vm->pte_encode(sg_page_iter_dma_address(&sg_iter),
@@ -981,7 +1065,7 @@ static void gen6_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
 		pci_unmap_page(ppgtt->base.dev->pdev,
-			       ppgtt->pd.page_tables[i].daddr,
+			       ppgtt->pd.page_tables[i]->daddr,
 			       4096, PCI_DMA_BIDIRECTIONAL);
 }
 
@@ -990,8 +1074,9 @@ static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	int i;
 
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
-		__free_page(ppgtt->pd.page_tables[i].page);
-	kfree(ppgtt->pd.page_tables);
+		free_pt_single(ppgtt->pd.page_tables[i]);
+
+	free_pd_single(&ppgtt->pd);
 }
 
 static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
@@ -1048,27 +1133,6 @@ alloc:
 	return 0;
 }
 
-static int gen6_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
-{
-	struct i915_pagetab *pt;
-	int i;
-
-	pt = kcalloc(ppgtt->num_pd_entries, sizeof(*pt), GFP_KERNEL);
-	if (!pt)
-		return -ENOMEM;
-
-	for (i = 0; i < ppgtt->num_pd_entries; i++) {
-		pt[i].page = alloc_page(GFP_KERNEL);
-		if (!pt->page) {
-			gen6_ppgtt_free(ppgtt);
-			return -ENOMEM;
-		}
-	}
-
-	ppgtt->pd.page_tables = pt;
-	return 0;
-}
-
 static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
 {
 	int ret;
@@ -1077,7 +1141,7 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
 	if (ret)
 		return ret;
 
-	ret = gen6_ppgtt_allocate_page_tables(ppgtt);
+	ret = alloc_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries);
 	if (ret) {
 		drm_mm_remove_node(&ppgtt->node);
 		return ret;
@@ -1095,7 +1159,7 @@ static int gen6_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt)
 		struct page *page;
 		dma_addr_t pt_addr;
 
-		page = ppgtt->pd.page_tables[i].page;
+		page = ppgtt->pd.page_tables[i]->page;
 		pt_addr = pci_map_page(dev->pdev, page, 0, 4096,
 				       PCI_DMA_BIDIRECTIONAL);
 
@@ -1104,7 +1168,7 @@ static int gen6_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt)
 			return -EIO;
 		}
 
-		ppgtt->pd.page_tables[i].daddr = pt_addr;
+		ppgtt->pd.page_tables[i]->daddr = pt_addr;
 	}
 
 	return 0;
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 2e776ee..7e5afb7 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -269,12 +269,12 @@ struct i915_pagedir {
 		dma_addr_t daddr;
 	};
 
-	struct i915_pagetab *page_tables;
+	struct i915_pagetab *page_tables[I915_PDES_PER_PD]; /* PDEs */
 };
 
 struct i915_pagedirpo {
 	/* struct page *page; */
-	struct i915_pagedir pagedir[GEN8_LEGACY_PDPES];
+	struct i915_pagedir *pagedir[GEN8_LEGACY_PDPES];
 };
 
 struct i915_hw_ppgtt {
-- 
2.0.4

[PATCH 36/68] drm/i915: Generalize GEN6 mapping

[Ben Widawsky]

Having a more general way of doing mappings will allow the ability to
easy map and unmap a specific page table. Specifically in this case, we
pass down the page directory + entry, and the page table to map. This
works similarly to the x86 code.

The same work will need to happen for GEN8. At that point I will try to
combine functionality.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 61 +++++++++++++++++++------------------
 drivers/gpu/drm/i915/i915_gem_gtt.h |  2 ++
 2 files changed, 34 insertions(+), 29 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 532e560..8df3b15 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -692,18 +692,13 @@ bail:
 
 static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 {
-	struct drm_i915_private *dev_priv = ppgtt->base.dev->dev_private;
 	struct i915_address_space *vm = &ppgtt->base;
-	gen6_gtt_pte_t __iomem *pd_addr;
 	gen6_gtt_pte_t scratch_pte;
 	uint32_t pd_entry;
 	int pte, pde;
 
 	scratch_pte = vm->pte_encode(vm->scratch.addr, I915_CACHE_LLC, true, 0);
 
-	pd_addr = (gen6_gtt_pte_t __iomem *)dev_priv->gtt.gsm +
-		ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t);
-
 	seq_printf(m, "  VM %p (pd_offset %x-%x):\n", vm,
 		   ppgtt->pd.pd_offset,
 		   ppgtt->pd.pd_offset + ppgtt->num_pd_entries);
@@ -711,7 +706,7 @@ static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 		u32 expected;
 		gen6_gtt_pte_t *pt_vaddr;
 		dma_addr_t pt_addr = ppgtt->pd.page_tables[pde]->daddr;
-		pd_entry = readl(pd_addr + pde);
+		pd_entry = readl(ppgtt->pd_addr + pde);
 		expected = (GEN6_PDE_ADDR_ENCODE(pt_addr) | GEN6_PDE_VALID);
 
 		if (pd_entry != expected)
@@ -747,39 +742,43 @@ static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 	}
 }
 
-static void gen6_map_single(struct i915_hw_ppgtt *ppgtt,
-			    const unsigned pde_index,
-			    dma_addr_t daddr)
+/* Map pde (index) from the page directory @pd to the page table @pt */
+static void gen6_map_single(struct i915_pagedir *pd,
+			    const int pde, struct i915_pagetab *pt)
 {
-	struct drm_i915_private *dev_priv = ppgtt->base.dev->dev_private;
-	uint32_t pd_entry;
-	gen6_gtt_pte_t __iomem *pd_addr = (gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm;
-	pd_addr	+= ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t);
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(pd, struct i915_hw_ppgtt, pd);
+	u32 pd_entry;
 
-	pd_entry = GEN6_PDE_ADDR_ENCODE(daddr);
+	pd_entry = GEN6_PDE_ADDR_ENCODE(pt->daddr);
 	pd_entry |= GEN6_PDE_VALID;
 
-	writel(pd_entry, pd_addr + pde_index);
+	writel(pd_entry, ppgtt->pd_addr + pde);
+
+	/* XXX: Caller needs to make sure the write completes if necessary */
 }
 
 /* Map all the page tables found in the ppgtt structure to incrementing page
  * directories. */
-static void gen6_map_page_tables(struct i915_hw_ppgtt *ppgtt)
+static void gen6_map_page_range(struct drm_i915_private *dev_priv,
+				struct i915_pagedir *pd, unsigned pde, size_t n)
 {
-	struct drm_i915_private *dev_priv = ppgtt->base.dev->dev_private;
-	int i;
+	if (WARN_ON(pde + n > I915_PDES_PER_PD))
+		n = I915_PDES_PER_PD - pde;
 
-	WARN_ON(ppgtt->pd.pd_offset & 0x3f);
-	for (i = 0; i < ppgtt->num_pd_entries; i++)
-		gen6_map_single(ppgtt, i, ppgtt->pd.page_tables[i]->daddr);
+	n += pde;
+
+	for (; pde < n; pde++)
+		gen6_map_single(pd, pde, pd->page_tables[pde]);
 
+	/* Make sure write is complete before other code can use this page
+	 * table. Also require for WC mapped PTEs */
 	readl(dev_priv->gtt.gsm);
 }
 
 static uint32_t get_pd_offset(struct i915_hw_ppgtt *ppgtt)
 {
 	BUG_ON(ppgtt->pd.pd_offset & 0x3f);
-
 	return (ppgtt->pd.pd_offset / 64) << 16;
 }
 
@@ -1213,7 +1212,10 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->pd.pd_offset =
 		ppgtt->node.start / PAGE_SIZE * sizeof(gen6_gtt_pte_t);
 
-	gen6_map_page_tables(ppgtt);
+	ppgtt->pd_addr = (gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm +
+		ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t);
+
+	gen6_map_page_range(dev_priv, &ppgtt->pd, 0, ppgtt->num_pd_entries);
 
 	DRM_DEBUG_DRIVER("Allocated pde space (%ldM) at GTT entry: %lx\n",
 			 ppgtt->node.size >> 20,
@@ -1403,13 +1405,14 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 
 	list_for_each_entry(vm, &dev_priv->vm_list, global_link) {
 		/* TODO: Perhaps it shouldn't be gen6 specific */
-		if (i915_is_ggtt(vm)) {
-			if (dev_priv->mm.aliasing_ppgtt)
-				gen6_map_page_tables(dev_priv->mm.aliasing_ppgtt);
-			continue;
-		}
 
-		gen6_map_page_tables(container_of(vm, struct i915_hw_ppgtt, base));
+		struct i915_hw_ppgtt *ppgtt =
+			container_of(vm, struct i915_hw_ppgtt, base);
+
+		if (i915_is_ggtt(vm))
+			ppgtt = dev_priv->mm.aliasing_ppgtt;
+
+		gen6_map_page_range(dev_priv, &ppgtt->pd, 0, ppgtt->num_pd_entries);
 	}
 
 	i915_gem_chipset_flush(dev);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 7e5afb7..adee18c 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -290,6 +290,8 @@ struct i915_hw_ppgtt {
 
 	struct intel_context *ctx;
 
+	gen6_gtt_pte_t __iomem *pd_addr;
+
 	int (*enable)(struct i915_hw_ppgtt *ppgtt);
 	int (*switch_mm)(struct i915_hw_ppgtt *ppgtt,
 			 struct intel_engine_cs *ring,
-- 
2.0.4

[PATCH 37/68] drm/i915: Clean up pagetable DMA map & unmap

[Ben Widawsky]

Map and unmap are common operations across all generations for
pagetables. With a simple helper, we can get a nice net code reduction
as well as simplified complexity.

There is some room for optimization here, for instance with the multiple
page mapping, that can be done in one pci_map operation. In that case
however, the max value we'll ever see there is 512, and so I believe the
simpler code makes this a worthwhile trade-off. Also, the range mapping
functions are place holders to help transition the code. Eventually,
mapping will only occur during a page allocation which will always be a
discrete operation.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 147 +++++++++++++++++++++---------------
 1 file changed, 85 insertions(+), 62 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 8df3b15..4bd1e07 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -203,6 +203,76 @@ static gen6_gtt_pte_t iris_pte_encode(dma_addr_t addr,
 	return pte;
 }
 
+#define dma_unmap_pt_single(pt, dev) do { \
+	pci_unmap_page((dev)->pdev, (pt)->daddr, 4096, PCI_DMA_BIDIRECTIONAL); \
+} while (0);
+
+
+static void dma_unmap_pt_range(struct i915_pagedir *pd,
+			       unsigned pde, size_t n,
+			       struct drm_device *dev)
+{
+	if (WARN_ON(pde + n > I915_PDES_PER_PD))
+		n = I915_PDES_PER_PD - pde;
+
+	n += pde;
+
+	for (; pde < n; pde++)
+		dma_unmap_pt_single(pd->page_tables[pde], dev);
+}
+
+/**
+ * dma_map_pt_single() - Create a dma mapping for a page table
+ * @pt:		Page table to get a DMA map for
+ * @dev:	drm device
+ *
+ * Page table allocations are unified across all gens. They always require a
+ * single 4k allocation, as well as a DMA mapping.
+ *
+ * Return: 0 if success.
+ */
+static int dma_map_pt_single(struct i915_pagetab *pt, struct drm_device *dev)
+{
+	struct page *page;
+	dma_addr_t pt_addr;
+	int ret;
+
+	page = pt->page;
+	pt_addr = pci_map_page(dev->pdev, page, 0, 4096,
+			       PCI_DMA_BIDIRECTIONAL);
+
+	ret = pci_dma_mapping_error(dev->pdev, pt_addr);
+	if (ret)
+		return ret;
+
+	pt->daddr = pt_addr;
+
+	return 0;
+}
+
+static int dma_map_pt_range(struct i915_pagedir *pd,
+			    unsigned pde, size_t n,
+			    struct drm_device *dev)
+{
+	const int first = pde;
+
+	if (WARN_ON(pde + n > I915_PDES_PER_PD))
+		n = I915_PDES_PER_PD - pde;
+
+	n += pde;
+
+	for (; pde < n; pde++) {
+		int ret;
+		ret = dma_map_pt_single(pd->page_tables[pde], dev);
+		if (ret) {
+			dma_unmap_pt_range(pd, first, pde, dev);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
 static void free_pt_single(struct i915_pagetab *pt)
 {
 	if (WARN_ON(!pt->page))
@@ -211,7 +281,7 @@ static void free_pt_single(struct i915_pagetab *pt)
 	kfree(pt);
 }
 
-static struct i915_pagetab *alloc_pt_single(void)
+static struct i915_pagetab *alloc_pt_single(struct drm_device *dev)
 {
 	struct i915_pagetab *pt;
 
@@ -234,6 +304,7 @@ static struct i915_pagetab *alloc_pt_single(void)
  *		available to point to the allocated page tables.
  * @pde:	First page directory entry for which we are allocating.
  * @count:	Number of pages to allocate.
+ * @dev		DRM device used for DMA mapping.
  *
  * Allocates multiple page table pages and sets the appropriate entries in the
  * page table structure within the page directory. Function cleans up after
@@ -241,7 +312,8 @@ static struct i915_pagetab *alloc_pt_single(void)
  *
  * Return: 0 if allocation succeeded.
  */
-static int alloc_pt_range(struct i915_pagedir *pd, uint16_t pde, size_t count)
+static int alloc_pt_range(struct i915_pagedir *pd, uint16_t pde, size_t count,
+			  struct drm_device *dev)
 {
 	int i, ret;
 
@@ -251,7 +323,7 @@ static int alloc_pt_range(struct i915_pagedir *pd, uint16_t pde, size_t count)
 	BUG_ON(pde + count > I915_PDES_PER_PD);
 
 	for (i = pde; i < pde + count; i++) {
-		struct i915_pagetab *pt = alloc_pt_single();
+		struct i915_pagetab *pt = alloc_pt_single(dev);
 		if (IS_ERR(pt)) {
 			ret = PTR_ERR(pt);
 			goto err_out;
@@ -507,7 +579,7 @@ static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
 		ret = alloc_pt_range(ppgtt->pdp.pagedir[i],
-				     0, I915_PDES_PER_PD);
+				     0, I915_PDES_PER_PD, ppgtt->base.dev);
 		if (ret)
 			goto unwind_out;
 	}
@@ -586,27 +658,6 @@ static int gen8_ppgtt_setup_page_directories(struct i915_hw_ppgtt *ppgtt,
 	return 0;
 }
 
-static int gen8_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt,
-					const int pdpe,
-					const int pde)
-{
-	dma_addr_t pt_addr;
-	struct i915_pagedir *pd = ppgtt->pdp.pagedir[pdpe];
-	struct i915_pagetab *pt = pd->page_tables[pde];
-	struct page *p = pt->page;
-	int ret;
-
-	pt_addr = pci_map_page(ppgtt->base.dev->pdev,
-			       p, 0, PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
-	ret = pci_dma_mapping_error(ppgtt->base.dev->pdev, pt_addr);
-	if (ret)
-		return ret;
-
-	pt->daddr = pt_addr;
-
-	return 0;
-}
-
 /**
  * GEN8 legacy ppgtt programming is accomplished through a max 4 PDP registers
  * with a net effect resembling a 2-level page table in normal x86 terms. Each
@@ -635,12 +686,15 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	 * 2. Create DMA mappings for the page directories and page tables.
 	 */
 	for (i = 0; i < max_pdp; i++) {
+		struct i915_pagedir *pd;
 		ret = gen8_ppgtt_setup_page_directories(ppgtt, i);
 		if (ret)
 			goto bail;
 
+		pd = ppgtt->pdp.pagedir[i];
+
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			ret = gen8_ppgtt_setup_page_tables(ppgtt, i, j);
+			ret = dma_map_pt_single(pd->page_tables[j], ppgtt->base.dev);
 			if (ret)
 				goto bail;
 		}
@@ -1058,16 +1112,6 @@ static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 		kunmap_atomic(pt_vaddr);
 }
 
-static void gen6_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
-{
-	int i;
-
-	for (i = 0; i < ppgtt->num_pd_entries; i++)
-		pci_unmap_page(ppgtt->base.dev->pdev,
-			       ppgtt->pd.page_tables[i]->daddr,
-			       4096, PCI_DMA_BIDIRECTIONAL);
-}
-
 static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
 	int i;
@@ -1087,7 +1131,7 @@ static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
 	drm_mm_takedown(&vm->mm);
 	drm_mm_remove_node(&ppgtt->node);
 
-	gen6_ppgtt_dma_unmap_pages(ppgtt);
+	dma_unmap_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries, vm->dev);
 	gen6_ppgtt_free(ppgtt);
 }
 
@@ -1140,7 +1184,8 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
 	if (ret)
 		return ret;
 
-	ret = alloc_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries);
+	ret = alloc_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries,
+			     ppgtt->base.dev);
 	if (ret) {
 		drm_mm_remove_node(&ppgtt->node);
 		return ret;
@@ -1149,29 +1194,6 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
 	return 0;
 }
 
-static int gen6_ppgtt_setup_page_tables(struct i915_hw_ppgtt *ppgtt)
-{
-	struct drm_device *dev = ppgtt->base.dev;
-	int i;
-
-	for (i = 0; i < ppgtt->num_pd_entries; i++) {
-		struct page *page;
-		dma_addr_t pt_addr;
-
-		page = ppgtt->pd.page_tables[i]->page;
-		pt_addr = pci_map_page(dev->pdev, page, 0, 4096,
-				       PCI_DMA_BIDIRECTIONAL);
-
-		if (pci_dma_mapping_error(dev->pdev, pt_addr)) {
-			gen6_ppgtt_dma_unmap_pages(ppgtt);
-			return -EIO;
-		}
-
-		ppgtt->pd.page_tables[i]->daddr = pt_addr;
-	}
-
-	return 0;
-}
 
 static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 {
@@ -1196,7 +1218,8 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	if (ret)
 		return ret;
 
-	ret = gen6_ppgtt_setup_page_tables(ppgtt);
+	ret = dma_map_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries,
+			       ppgtt->base.dev);
 	if (ret) {
 		gen6_ppgtt_free(ppgtt);
 		return ret;
-- 
2.0.4

[PATCH 38/68] drm/i915: Always dma map page table allocations

[Ben Widawsky]

There is never a case where we don't want to do it. Since we've broken
up the allocations into nice clean helper functions, it's both easy and
obvious to do the dma mapping at the same time.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 78 ++++++++-----------------------------
 1 file changed, 17 insertions(+), 61 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 4bd1e07..2346cb7 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -207,20 +207,6 @@ static gen6_gtt_pte_t iris_pte_encode(dma_addr_t addr,
 	pci_unmap_page((dev)->pdev, (pt)->daddr, 4096, PCI_DMA_BIDIRECTIONAL); \
 } while (0);
 
-
-static void dma_unmap_pt_range(struct i915_pagedir *pd,
-			       unsigned pde, size_t n,
-			       struct drm_device *dev)
-{
-	if (WARN_ON(pde + n > I915_PDES_PER_PD))
-		n = I915_PDES_PER_PD - pde;
-
-	n += pde;
-
-	for (; pde < n; pde++)
-		dma_unmap_pt_single(pd->page_tables[pde], dev);
-}
-
 /**
  * dma_map_pt_single() - Create a dma mapping for a page table
  * @pt:		Page table to get a DMA map for
@@ -250,33 +236,12 @@ static int dma_map_pt_single(struct i915_pagetab *pt, struct drm_device *dev)
 	return 0;
 }
 
-static int dma_map_pt_range(struct i915_pagedir *pd,
-			    unsigned pde, size_t n,
-			    struct drm_device *dev)
-{
-	const int first = pde;
-
-	if (WARN_ON(pde + n > I915_PDES_PER_PD))
-		n = I915_PDES_PER_PD - pde;
-
-	n += pde;
-
-	for (; pde < n; pde++) {
-		int ret;
-		ret = dma_map_pt_single(pd->page_tables[pde], dev);
-		if (ret) {
-			dma_unmap_pt_range(pd, first, pde, dev);
-			return ret;
-		}
-	}
-
-	return 0;
-}
-
-static void free_pt_single(struct i915_pagetab *pt)
+static void free_pt_single(struct i915_pagetab *pt, struct drm_device *dev)
 {
 	if (WARN_ON(!pt->page))
 		return;
+
+	dma_unmap_pt_single(pt, dev);
 	__free_page(pt->page);
 	kfree(pt);
 }
@@ -284,6 +249,7 @@ static void free_pt_single(struct i915_pagetab *pt)
 static struct i915_pagetab *alloc_pt_single(struct drm_device *dev)
 {
 	struct i915_pagetab *pt;
+	int ret;
 
 	pt = kzalloc(sizeof(*pt), GFP_KERNEL);
 	if (!pt)
@@ -295,6 +261,13 @@ static struct i915_pagetab *alloc_pt_single(struct drm_device *dev)
 		return ERR_PTR(-ENOMEM);
 	}
 
+	ret = dma_map_pt_single(pt, dev);
+	if (ret) {
+		__free_page(pt->page);
+		kfree(pt);
+		return ERR_PTR(ret);
+	}
+
 	return pt;
 }
 
@@ -338,7 +311,7 @@ static int alloc_pt_range(struct i915_pagedir *pd, uint16_t pde, size_t count,
 
 err_out:
 	while (i--)
-		free_pt_single(pd->page_tables[i]);
+		free_pt_single(pd->page_tables[i], dev);
 	return ret;
 }
 
@@ -513,7 +486,7 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	}
 }
 
-static void gen8_free_page_tables(struct i915_pagedir *pd)
+static void gen8_free_page_tables(struct i915_pagedir *pd, struct drm_device *dev)
 {
 	int i;
 
@@ -521,7 +494,7 @@ static void gen8_free_page_tables(struct i915_pagedir *pd)
 		return;
 
 	for (i = 0; i < I915_PDES_PER_PD; i++) {
-		free_pt_single(pd->page_tables[i]);
+		free_pt_single(pd->page_tables[i], dev);
 		pd->page_tables[i] = NULL;
 	}
 }
@@ -531,7 +504,7 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	int i;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		gen8_free_page_tables(ppgtt->pdp.pagedir[i]);
+		gen8_free_page_tables(ppgtt->pdp.pagedir[i], ppgtt->base.dev);
 		free_pd_single(ppgtt->pdp.pagedir[i]);
 	}
 }
@@ -588,7 +561,7 @@ static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 
 unwind_out:
 	while (i--)
-		gen8_free_page_tables(ppgtt->pdp.pagedir[i]);
+		gen8_free_page_tables(ppgtt->pdp.pagedir[i], ppgtt->base.dev);
 
 	return -ENOMEM;
 }
@@ -686,18 +659,9 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	 * 2. Create DMA mappings for the page directories and page tables.
 	 */
 	for (i = 0; i < max_pdp; i++) {
-		struct i915_pagedir *pd;
 		ret = gen8_ppgtt_setup_page_directories(ppgtt, i);
 		if (ret)
 			goto bail;
-
-		pd = ppgtt->pdp.pagedir[i];
-
-		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			ret = dma_map_pt_single(pd->page_tables[j], ppgtt->base.dev);
-			if (ret)
-				goto bail;
-		}
 	}
 
 	/*
@@ -1117,7 +1081,7 @@ static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	int i;
 
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
-		free_pt_single(ppgtt->pd.page_tables[i]);
+		free_pt_single(ppgtt->pd.page_tables[i], ppgtt->base.dev);
 
 	free_pd_single(&ppgtt->pd);
 }
@@ -1131,7 +1095,6 @@ static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
 	drm_mm_takedown(&vm->mm);
 	drm_mm_remove_node(&ppgtt->node);
 
-	dma_unmap_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries, vm->dev);
 	gen6_ppgtt_free(ppgtt);
 }
 
@@ -1218,13 +1181,6 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	if (ret)
 		return ret;
 
-	ret = dma_map_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries,
-			       ppgtt->base.dev);
-	if (ret) {
-		gen6_ppgtt_free(ppgtt);
-		return ret;
-	}
-
 	ppgtt->base.clear_range = gen6_ppgtt_clear_range;
 	ppgtt->base.insert_entries = gen6_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen6_ppgtt_cleanup;
-- 
2.0.4

[PATCH 39/68] drm/i915: Consolidate dma mappings

[Ben Widawsky]

With a little bit of macro magic, and the fact that every page
table/dir/etc. we wish to map will have a page, and daddr member, we can
greatly simplify and reduce code.

The patch introduces an i915_dma_map/unmap which has the same semantics
as pci_map_page, but is 1 line, and doesn't require newlines, or local
variables to make it fit cleanly.

Notice that even the page allocation shares this same attribute. For
now, I am leaving that code untouched because the macro version would be
a bit on the big side - but it's a nice cleanup as well (IMO)

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 56 ++++++++++++-------------------------
 1 file changed, 18 insertions(+), 38 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 2346cb7..205d5c6 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -203,45 +203,33 @@ static gen6_gtt_pte_t iris_pte_encode(dma_addr_t addr,
 	return pte;
 }
 
-#define dma_unmap_pt_single(pt, dev) do { \
-	pci_unmap_page((dev)->pdev, (pt)->daddr, 4096, PCI_DMA_BIDIRECTIONAL); \
+#define i915_dma_unmap_single(px, dev) do { \
+	pci_unmap_page((dev)->pdev, (px)->daddr, 4096, PCI_DMA_BIDIRECTIONAL); \
 } while (0);
 
 /**
- * dma_map_pt_single() - Create a dma mapping for a page table
- * @pt:		Page table to get a DMA map for
+ * i915_dma_map_px_single() - Create a dma mapping for a page table/dir/etc.
+ * @px:		Page table/dir/etc to get a DMA map for
  * @dev:	drm device
  *
  * Page table allocations are unified across all gens. They always require a
- * single 4k allocation, as well as a DMA mapping.
+ * single 4k allocation, as well as a DMA mapping. If we keep the structs
+ * symmetric here, the simple macro covers us for every page table type.
  *
  * Return: 0 if success.
  */
-static int dma_map_pt_single(struct i915_pagetab *pt, struct drm_device *dev)
-{
-	struct page *page;
-	dma_addr_t pt_addr;
-	int ret;
-
-	page = pt->page;
-	pt_addr = pci_map_page(dev->pdev, page, 0, 4096,
-			       PCI_DMA_BIDIRECTIONAL);
-
-	ret = pci_dma_mapping_error(dev->pdev, pt_addr);
-	if (ret)
-		return ret;
-
-	pt->daddr = pt_addr;
-
-	return 0;
-}
+#define i915_dma_map_px_single(px, dev) \
+	pci_dma_mapping_error((dev)->pdev, \
+			      (px)->daddr = pci_map_page((dev)->pdev, \
+							 (px)->page, 0, 4096, \
+							 PCI_DMA_BIDIRECTIONAL))
 
 static void free_pt_single(struct i915_pagetab *pt, struct drm_device *dev)
 {
 	if (WARN_ON(!pt->page))
 		return;
 
-	dma_unmap_pt_single(pt, dev);
+	i915_dma_unmap_single(pt, dev);
 	__free_page(pt->page);
 	kfree(pt);
 }
@@ -261,7 +249,7 @@ static struct i915_pagetab *alloc_pt_single(struct drm_device *dev)
 		return ERR_PTR(-ENOMEM);
 	}
 
-	ret = dma_map_pt_single(pt, dev);
+	ret = i915_dma_map_px_single(pt, dev);
 	if (ret) {
 		__free_page(pt->page);
 		kfree(pt);
@@ -511,7 +499,7 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 
 static void gen8_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 {
-	struct pci_dev *hwdev = ppgtt->base.dev->pdev;
+	struct drm_device *dev = ppgtt->base.dev;
 	int i, j;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
@@ -520,16 +508,14 @@ static void gen8_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
 		if (!ppgtt->pdp.pagedir[i]->daddr)
 			continue;
 
-		pci_unmap_page(hwdev, ppgtt->pdp.pagedir[i]->daddr, PAGE_SIZE,
-			       PCI_DMA_BIDIRECTIONAL);
+		i915_dma_unmap_single(ppgtt->pdp.pagedir[i], dev);
 
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
 			struct i915_pagedir *pd = ppgtt->pdp.pagedir[i];
 			struct i915_pagetab *pt =  pd->page_tables[j];
 			dma_addr_t addr = pt->daddr;
 			if (addr)
-				pci_unmap_page(hwdev, addr, PAGE_SIZE,
-					       PCI_DMA_BIDIRECTIONAL);
+				i915_dma_unmap_single(pt, dev);
 		}
 	}
 }
@@ -615,19 +601,13 @@ err_out:
 static int gen8_ppgtt_setup_page_directories(struct i915_hw_ppgtt *ppgtt,
 					     const int pdpe)
 {
-	dma_addr_t pd_addr;
 	int ret;
 
-	pd_addr = pci_map_page(ppgtt->base.dev->pdev,
-			       ppgtt->pdp.pagedir[pdpe]->page, 0,
-			       PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
-
-	ret = pci_dma_mapping_error(ppgtt->base.dev->pdev, pd_addr);
+	ret = i915_dma_map_px_single(ppgtt->pdp.pagedir[pdpe],
+				     ppgtt->base.dev);
 	if (ret)
 		return ret;
 
-	ppgtt->pdp.pagedir[pdpe]->daddr = pd_addr;
-
 	return 0;
 }
 
-- 
2.0.4

[PATCH 40/68] drm/i915: Always dma map page directory allocations

[Ben Widawsky]

Similar to the patch a few back in the series, we can always map and
unmap page directories when we do their allocation and teardown. Page
directory pages only exist on gen8+, so this should only effect behavior
on those platforms.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 79 +++++++++----------------------------
 1 file changed, 19 insertions(+), 60 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 205d5c6..094a82f 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -303,21 +303,23 @@ err_out:
 	return ret;
 }
 
-static void __free_pd_single(struct i915_pagedir *pd)
+static void __free_pd_single(struct i915_pagedir *pd, struct drm_device *dev)
 {
+	i915_dma_unmap_single(pd, dev);
 	__free_page(pd->page);
 	kfree(pd);
 }
 
-#define free_pd_single(pd) do { \
+#define free_pd_single(pd, dev) do { \
 	if ((pd)->page) { \
-		__free_pd_single(pd); \
+		__free_pd_single(pd, dev); \
 	} \
 } while (0)
 
-static struct i915_pagedir *alloc_pd_single(void)
+static struct i915_pagedir *alloc_pd_single(struct drm_device *dev)
 {
 	struct i915_pagedir *pd;
+	int ret;
 
 	pd = kzalloc(sizeof(*pd), GFP_KERNEL);
 	if (!pd)
@@ -329,6 +331,13 @@ static struct i915_pagedir *alloc_pd_single(void)
 		return ERR_PTR(-ENOMEM);
 	}
 
+	ret = i915_dma_map_px_single(pd, dev);
+	if (ret) {
+		__free_page(pd->page);
+		kfree(pd);
+		return ERR_PTR(ret);
+	}
+
 	return pd;
 }
 
@@ -493,30 +502,7 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
 		gen8_free_page_tables(ppgtt->pdp.pagedir[i], ppgtt->base.dev);
-		free_pd_single(ppgtt->pdp.pagedir[i]);
-	}
-}
-
-static void gen8_ppgtt_dma_unmap_pages(struct i915_hw_ppgtt *ppgtt)
-{
-	struct drm_device *dev = ppgtt->base.dev;
-	int i, j;
-
-	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		/* TODO: In the future we'll support sparse mappings, so this
-		 * will have to change. */
-		if (!ppgtt->pdp.pagedir[i]->daddr)
-			continue;
-
-		i915_dma_unmap_single(ppgtt->pdp.pagedir[i], dev);
-
-		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			struct i915_pagedir *pd = ppgtt->pdp.pagedir[i];
-			struct i915_pagetab *pt =  pd->page_tables[j];
-			dma_addr_t addr = pt->daddr;
-			if (addr)
-				i915_dma_unmap_single(pt, dev);
-		}
+		free_pd_single(ppgtt->pdp.pagedir[i], ppgtt->base.dev);
 	}
 }
 
@@ -528,7 +514,6 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
 	list_del(&vm->global_link);
 	drm_mm_takedown(&vm->mm);
 
-	gen8_ppgtt_dma_unmap_pages(ppgtt);
 	gen8_ppgtt_free(ppgtt);
 }
 
@@ -558,7 +543,7 @@ static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
 	int i;
 
 	for (i = 0; i < max_pdp; i++) {
-		ppgtt->pdp.pagedir[i] = alloc_pd_single();
+		ppgtt->pdp.pagedir[i] = alloc_pd_single(ppgtt->base.dev);
 		if (IS_ERR(ppgtt->pdp.pagedir[i]))
 			goto unwind_out;
 	}
@@ -570,7 +555,8 @@ static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
 
 unwind_out:
 	while (i--)
-		free_pd_single(ppgtt->pdp.pagedir[i]);
+		free_pd_single(ppgtt->pdp.pagedir[i],
+			       ppgtt->base.dev);
 
 	return -ENOMEM;
 }
@@ -598,19 +584,6 @@ err_out:
 	return ret;
 }
 
-static int gen8_ppgtt_setup_page_directories(struct i915_hw_ppgtt *ppgtt,
-					     const int pdpe)
-{
-	int ret;
-
-	ret = i915_dma_map_px_single(ppgtt->pdp.pagedir[pdpe],
-				     ppgtt->base.dev);
-	if (ret)
-		return ret;
-
-	return 0;
-}
-
 /**
  * GEN8 legacy ppgtt programming is accomplished through a max 4 PDP registers
  * with a net effect resembling a 2-level page table in normal x86 terms. Each
@@ -636,16 +609,7 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 		return ret;
 
 	/*
-	 * 2. Create DMA mappings for the page directories and page tables.
-	 */
-	for (i = 0; i < max_pdp; i++) {
-		ret = gen8_ppgtt_setup_page_directories(ppgtt, i);
-		if (ret)
-			goto bail;
-	}
-
-	/*
-	 * 3. Map all the page directory entires to point to the page tables
+	 * 2. Map all the page directory entires to point to the page tables
 	 * we've allocated.
 	 *
 	 * For now, the PPGTT helper functions all require that the PDEs are
@@ -681,11 +645,6 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 			 ppgtt->num_pd_entries,
 			 (ppgtt->num_pd_entries - min_pt_pages) + size % (1<<30));
 	return 0;
-
-bail:
-	gen8_ppgtt_dma_unmap_pages(ppgtt);
-	gen8_ppgtt_free(ppgtt);
-	return ret;
 }
 
 static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
@@ -1063,7 +1022,7 @@ static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
 		free_pt_single(ppgtt->pd.page_tables[i], ppgtt->base.dev);
 
-	free_pd_single(&ppgtt->pd);
+	free_pd_single(&ppgtt->pd, ppgtt->base.dev);
 }
 
 static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
-- 
2.0.4

[PATCH 41/68] drm/i915: Track GEN6 page table usage

[Ben Widawsky]

Instead of implementing the full tracking + dynamic allocation, this
patch does a bit less than half of the work, by tracking and warning on
unexpected conditions. The tracking itself follows which PTEs within a
page table are currently being used for objects. The next patch will
modify this to actually allocate the page tables only when necessary.

With the current patch there isn't much in the way of making a gen
agnostic range allocation function. However, in the next patch we'll add
more specificity which makes having separate functions a bit easier to
manage.

Notice that aliasing PPGTT is not managed here. The patch which actually
begins dynamic allocation/teardown explains the reasoning forthis.

v2: s/pdp.pagedir/pdp.pagedirs
Make a scratch page allocation helper

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 202 ++++++++++++++++++++++++++++--------
 drivers/gpu/drm/i915/i915_gem_gtt.h | 117 +++++++++++++--------
 2 files changed, 230 insertions(+), 89 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 094a82f..e7adbff 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -62,10 +62,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)
 	return HAS_ALIASING_PPGTT(dev) ? 1 : 0;
 }
 
-
-static void ppgtt_bind_vma(struct i915_vma *vma,
-			   enum i915_cache_level cache_level,
-			   u32 flags);
+static int ppgtt_bind_vma(struct i915_vma *vma,
+			  enum i915_cache_level cache_level,
+			  u32 flags);
 static void ppgtt_unbind_vma(struct i915_vma *vma);
 static int gen8_ppgtt_enable(struct i915_hw_ppgtt *ppgtt);
 
@@ -224,37 +223,78 @@ static gen6_gtt_pte_t iris_pte_encode(dma_addr_t addr,
 							 (px)->page, 0, 4096, \
 							 PCI_DMA_BIDIRECTIONAL))
 
-static void free_pt_single(struct i915_pagetab *pt, struct drm_device *dev)
+static void __free_pt_single(struct i915_pagetab *pt, struct drm_device *dev,
+			     int scratch)
 {
+	if (WARN(scratch ^ pt->scratch,
+		 "Tried to free scratch = %d. Is scratch = %d\n",
+		 scratch, pt->scratch))
+		return;
+
 	if (WARN_ON(!pt->page))
 		return;
 
+	if (!scratch) {
+		const size_t count = INTEL_INFO(dev)->gen >= 8 ?
+			GEN8_PTES_PER_PT : GEN6_PTES_PER_PT;
+		WARN(!bitmap_empty(pt->used_ptes, count),
+		     "Free page table with %d used pages\n",
+		     bitmap_weight(pt->used_ptes, count));
+	}
+
 	i915_dma_unmap_single(pt, dev);
 	__free_page(pt->page);
+	kfree(pt->used_ptes);
 	kfree(pt);
 }
 
+#define free_pt_single(pt, dev) \
+	__free_pt_single(pt, dev, false)
+#define free_pt_scratch(pt, dev) \
+	__free_pt_single(pt, dev, true)
+
 static struct i915_pagetab *alloc_pt_single(struct drm_device *dev)
 {
 	struct i915_pagetab *pt;
-	int ret;
+	const size_t count = INTEL_INFO(dev)->gen >= 8 ?
+		GEN8_PTES_PER_PT : GEN6_PTES_PER_PT;
+	int ret = -ENOMEM;
 
 	pt = kzalloc(sizeof(*pt), GFP_KERNEL);
 	if (!pt)
 		return ERR_PTR(-ENOMEM);
 
+	pt->used_ptes = kcalloc(BITS_TO_LONGS(count), sizeof(*pt->used_ptes),
+				GFP_KERNEL);
+
+	if (!pt->used_ptes)
+		goto fail_bitmap;
+
 	pt->page = alloc_page(GFP_KERNEL | __GFP_ZERO);
-	if (!pt->page) {
-		kfree(pt);
-		return ERR_PTR(-ENOMEM);
-	}
+	if (!pt->page)
+		goto fail_page;
 
 	ret = i915_dma_map_px_single(pt, dev);
-	if (ret) {
-		__free_page(pt->page);
-		kfree(pt);
-		return ERR_PTR(ret);
-	}
+	if (ret)
+		goto fail_dma;
+
+	return pt;
+
+fail_dma:
+	__free_page(pt->page);
+fail_page:
+	kfree(pt->used_ptes);
+fail_bitmap:
+	kfree(pt);
+
+	return ERR_PTR(ret);
+}
+
+static inline struct i915_pagetab *alloc_pt_scratch(struct drm_device *dev)
+{
+	struct i915_pagetab *pt = alloc_pt_single(dev);
+	if (!IS_ERR(pt))
+		pt->scratch = 1;
 
 	return pt;
 }
@@ -381,7 +421,7 @@ static int gen8_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	int used_pd = ppgtt->num_pd_entries / I915_PDES_PER_PD;
 
 	for (i = used_pd - 1; i >= 0; i--) {
-		dma_addr_t addr = ppgtt->pdp.pagedir[i]->daddr;
+		dma_addr_t addr = ppgtt->pdp.pagedirs[i]->daddr;
 		ret = gen8_write_pdp(ring, i, addr, synchronous);
 		if (ret)
 			return ret;
@@ -408,7 +448,7 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 				      I915_CACHE_LLC, use_scratch);
 
 	while (num_entries) {
-		struct i915_pagedir *pd = ppgtt->pdp.pagedir[pdpe];
+		struct i915_pagedir *pd = ppgtt->pdp.pagedirs[pdpe];
 		struct i915_pagetab *pt = pd->page_tables[pde];
 		struct page *page_table = pt->page;
 
@@ -455,7 +495,7 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 			break;
 
 		if (pt_vaddr == NULL) {
-			struct i915_pagedir *pd = ppgtt->pdp.pagedir[pdpe];
+			struct i915_pagedir *pd = ppgtt->pdp.pagedirs[pdpe];
 			struct i915_pagetab *pt = pd->page_tables[pde];
 			struct page *page_table = pt->page;
 			pt_vaddr = kmap_atomic(page_table);
@@ -501,8 +541,8 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	int i;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		gen8_free_page_tables(ppgtt->pdp.pagedir[i], ppgtt->base.dev);
-		free_pd_single(ppgtt->pdp.pagedir[i], ppgtt->base.dev);
+		gen8_free_page_tables(ppgtt->pdp.pagedirs[i], ppgtt->base.dev);
+		free_pd_single(ppgtt->pdp.pagedirs[i], ppgtt->base.dev);
 	}
 }
 
@@ -522,7 +562,7 @@ static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 	int i, ret;
 
 	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		ret = alloc_pt_range(ppgtt->pdp.pagedir[i],
+		ret = alloc_pt_range(ppgtt->pdp.pagedirs[i],
 				     0, I915_PDES_PER_PD, ppgtt->base.dev);
 		if (ret)
 			goto unwind_out;
@@ -532,7 +572,7 @@ static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 
 unwind_out:
 	while (i--)
-		gen8_free_page_tables(ppgtt->pdp.pagedir[i], ppgtt->base.dev);
+		gen8_free_page_tables(ppgtt->pdp.pagedirs[i], ppgtt->base.dev);
 
 	return -ENOMEM;
 }
@@ -543,8 +583,8 @@ static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
 	int i;
 
 	for (i = 0; i < max_pdp; i++) {
-		ppgtt->pdp.pagedir[i] = alloc_pd_single(ppgtt->base.dev);
-		if (IS_ERR(ppgtt->pdp.pagedir[i]))
+		ppgtt->pdp.pagedirs[i] = alloc_pd_single(ppgtt->base.dev);
+		if (IS_ERR(ppgtt->pdp.pagedirs[i]))
 			goto unwind_out;
 	}
 
@@ -555,7 +595,7 @@ static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
 
 unwind_out:
 	while (i--)
-		free_pd_single(ppgtt->pdp.pagedir[i],
+		free_pd_single(ppgtt->pdp.pagedirs[i],
 			       ppgtt->base.dev);
 
 	return -ENOMEM;
@@ -617,9 +657,9 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	 * will never need to touch the PDEs again.
 	 */
 	for (i = 0; i < max_pdp; i++) {
-		struct i915_pagedir *pd = ppgtt->pdp.pagedir[i];
+		struct i915_pagedir *pd = ppgtt->pdp.pagedirs[i];
 		gen8_ppgtt_pde_t *pd_vaddr;
-		pd_vaddr = kmap_atomic(ppgtt->pdp.pagedir[i]->page);
+		pd_vaddr = kmap_atomic(ppgtt->pdp.pagedirs[i]->page);
 		for (j = 0; j < I915_PDES_PER_PD; j++) {
 			struct i915_pagetab *pt = pd->page_tables[j];
 			dma_addr_t addr = pt->daddr;
@@ -718,15 +758,13 @@ static void gen6_map_single(struct i915_pagedir *pd,
 /* Map all the page tables found in the ppgtt structure to incrementing page
  * directories. */
 static void gen6_map_page_range(struct drm_i915_private *dev_priv,
-				struct i915_pagedir *pd, unsigned pde, size_t n)
+				struct i915_pagedir *pd, uint32_t start, uint32_t length)
 {
-	if (WARN_ON(pde + n > I915_PDES_PER_PD))
-		n = I915_PDES_PER_PD - pde;
-
-	n += pde;
+	struct i915_pagetab *pt;
+	uint32_t pde, temp;
 
-	for (; pde < n; pde++)
-		gen6_map_single(pd, pde, pd->page_tables[pde]);
+	gen6_for_each_pde(pt, pd, start, length, temp, pde)
+		gen6_map_single(pd, pde, pt);
 
 	/* Make sure write is complete before other code can use this page
 	 * table. Also require for WC mapped PTEs */
@@ -1015,6 +1053,51 @@ static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 		kunmap_atomic(pt_vaddr);
 }
 
+static int gen6_alloc_va_range(struct i915_address_space *vm,
+			       uint64_t start, uint64_t length)
+{
+	struct i915_hw_ppgtt *ppgtt =
+		        container_of(vm, struct i915_hw_ppgtt, base);
+	struct i915_pagetab *pt;
+	uint32_t pde, temp;
+
+	gen6_for_each_pde(pt, &ppgtt->pd, start, length, temp, pde) {
+		int j;
+
+		DECLARE_BITMAP(tmp_bitmap, GEN6_PTES_PER_PT);
+		bitmap_zero(tmp_bitmap, GEN6_PTES_PER_PT);
+		bitmap_set(tmp_bitmap, gen6_pte_index(start),
+			   gen6_pte_count(start, length));
+
+		/* TODO: To be done in the next patch. Map the page/insert
+		 * entries here */
+		for_each_set_bit(j, tmp_bitmap, GEN6_PTES_PER_PT) {
+			if (test_bit(j, pt->used_ptes)) {
+				/* Check that we're changing cache levels */
+			}
+		}
+
+		bitmap_or(pt->used_ptes, pt->used_ptes, tmp_bitmap,
+			  GEN6_PTES_PER_PT);
+	}
+
+	return 0;
+}
+
+static void gen6_teardown_va_range(struct i915_address_space *vm,
+				   uint64_t start, uint64_t length)
+{
+	struct i915_hw_ppgtt *ppgtt =
+		        container_of(vm, struct i915_hw_ppgtt, base);
+	struct i915_pagetab *pt;
+	uint32_t pde, temp;
+
+	gen6_for_each_pde(pt, &ppgtt->pd, start, length, temp, pde) {
+		bitmap_clear(pt->used_ptes, gen6_pte_index(start),
+			     gen6_pte_count(start, length));
+	}
+}
+
 static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
 	int i;
@@ -1022,6 +1105,7 @@ static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
 		free_pt_single(ppgtt->pd.page_tables[i], ppgtt->base.dev);
 
+	free_pt_scratch(ppgtt->scratch_pt, ppgtt->base.dev);
 	free_pd_single(&ppgtt->pd, ppgtt->base.dev);
 }
 
@@ -1049,6 +1133,9 @@ static int gen6_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt)
 	 * size. We allocate at the top of the GTT to avoid fragmentation.
 	 */
 	BUG_ON(!drm_mm_initialized(&dev_priv->gtt.base.mm));
+	ppgtt->scratch_pt = alloc_pt_scratch(ppgtt->base.dev);
+	if (IS_ERR(ppgtt->scratch_pt))
+		return PTR_ERR(ppgtt->scratch_pt);
 alloc:
 	ret = drm_mm_insert_node_in_range_generic(&dev_priv->gtt.base.mm,
 						  &ppgtt->node, GEN6_PD_SIZE,
@@ -1062,20 +1149,25 @@ alloc:
 					       0, dev_priv->gtt.base.total,
 					       0);
 		if (ret)
-			return ret;
+			goto err_out;
 
 		retried = true;
 		goto alloc;
 	}
 
 	if (ret)
-		return ret;
+		goto err_out;
+
 
 	if (ppgtt->node.start < dev_priv->gtt.mappable_end)
 		DRM_DEBUG("Forced to use aperture for PDEs\n");
 
 	ppgtt->num_pd_entries = I915_PDES_PER_PD;
 	return 0;
+
+err_out:
+	free_pt_scratch(ppgtt->scratch_pt, ppgtt->base.dev);
+	return ret;
 }
 
 static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
@@ -1120,6 +1212,8 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	if (ret)
 		return ret;
 
+	ppgtt->base.allocate_va_range = gen6_alloc_va_range;
+	ppgtt->base.teardown_va_range = gen6_teardown_va_range;
 	ppgtt->base.clear_range = gen6_ppgtt_clear_range;
 	ppgtt->base.insert_entries = gen6_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen6_ppgtt_cleanup;
@@ -1133,7 +1227,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->pd_addr = (gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm +
 		ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t);
 
-	gen6_map_page_range(dev_priv, &ppgtt->pd, 0, ppgtt->num_pd_entries);
+	gen6_map_page_range(dev_priv, &ppgtt->pd, 0, ppgtt->base.total);
 
 	DRM_DEBUG_DRIVER("Allocated pde space (%ldM) at GTT entry: %lx\n",
 			 ppgtt->node.size >> 20,
@@ -1168,17 +1262,28 @@ int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 	return 0;
 }
 
-static void
+static int
 ppgtt_bind_vma(struct i915_vma *vma,
 	       enum i915_cache_level cache_level,
 	       u32 flags)
 {
+	int ret;
+
 	/* Currently applicable only to VLV */
 	if (vma->obj->gt_ro)
 		flags |= PTE_READ_ONLY;
 
+	if (vma->vm->allocate_va_range) {
+		ret = vma->vm->allocate_va_range(vma->vm,
+						 vma->node.start,
+						 vma->node.size);
+		if (ret)
+			return ret;
+	}
+
 	vma->vm->insert_entries(vma->vm, vma->obj->pages, vma->node.start,
 				cache_level, flags);
+	return 0;
 }
 
 static void ppgtt_unbind_vma(struct i915_vma *vma)
@@ -1187,6 +1292,9 @@ static void ppgtt_unbind_vma(struct i915_vma *vma)
 			     vma->node.start,
 			     vma->obj->base.size,
 			     true);
+	if (vma->vm->teardown_va_range)
+		vma->vm->teardown_va_range(vma->vm,
+					   vma->node.start, vma->node.size);
 }
 
 extern int intel_iommu_gfx_mapped;
@@ -1495,9 +1603,9 @@ static void gen6_ggtt_clear_range(struct i915_address_space *vm,
 }
 
 
-static void i915_ggtt_bind_vma(struct i915_vma *vma,
-			       enum i915_cache_level cache_level,
-			       u32 unused)
+static int i915_ggtt_bind_vma(struct i915_vma *vma,
+			      enum i915_cache_level cache_level,
+			      u32 unused)
 {
 	const unsigned long entry = vma->node.start >> PAGE_SHIFT;
 	unsigned int flags = (cache_level == I915_CACHE_NONE) ?
@@ -1506,6 +1614,8 @@ static void i915_ggtt_bind_vma(struct i915_vma *vma,
 	BUG_ON(!i915_is_ggtt(vma->vm));
 	intel_gtt_insert_sg_entries(vma->obj->pages, entry, flags);
 	vma->obj->has_global_gtt_mapping = 1;
+
+	return 0;
 }
 
 static void i915_ggtt_clear_range(struct i915_address_space *vm,
@@ -1528,9 +1638,9 @@ static void i915_ggtt_unbind_vma(struct i915_vma *vma)
 	intel_gtt_clear_range(first, size);
 }
 
-static void ggtt_bind_vma(struct i915_vma *vma,
-			  enum i915_cache_level cache_level,
-			  u32 flags)
+static int ggtt_bind_vma(struct i915_vma *vma,
+			 enum i915_cache_level cache_level,
+			 u32 flags)
 {
 	struct drm_device *dev = vma->vm->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -1562,7 +1672,7 @@ static void ggtt_bind_vma(struct i915_vma *vma,
 	}
 
 	if (!(flags & ALIASING_BIND))
-		return;
+		return 0;
 
 	if (dev_priv->mm.aliasing_ppgtt &&
 	    (!obj->has_aliasing_ppgtt_mapping ||
@@ -1574,6 +1684,8 @@ static void ggtt_bind_vma(struct i915_vma *vma,
 					    cache_level, flags);
 		vma->obj->has_aliasing_ppgtt_mapping = 1;
 	}
+
+	return 0;
 }
 
 static void ggtt_unbind_vma(struct i915_vma *vma)
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index adee18c..3729222 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -174,9 +174,33 @@ struct i915_vma {
 /* Only use this if you know you want a strictly aliased binding */
 #define ALIASING_BIND (1<<1)
 #define PTE_READ_ONLY (1<<2)
-	void (*bind_vma)(struct i915_vma *vma,
-			 enum i915_cache_level cache_level,
-			 u32 flags);
+	int (*bind_vma)(struct i915_vma *vma,
+			enum i915_cache_level cache_level,
+			u32 flags);
+};
+
+
+struct i915_pagetab {
+	struct page *page;
+	dma_addr_t daddr;
+
+	unsigned long *used_ptes;
+	unsigned int scratch:1;
+};
+
+struct i915_pagedir {
+	struct page *page; /* NULL for GEN6-GEN7 */
+	union {
+		uint32_t pd_offset;
+		dma_addr_t daddr;
+	};
+
+	struct i915_pagetab *page_tables[I915_PDES_PER_PD];
+};
+
+struct i915_pagedirpo {
+	/* struct page *page; */
+	struct i915_pagedir *pagedirs[GEN8_LEGACY_PDPES];
 };
 
 struct i915_address_space {
@@ -218,6 +242,12 @@ struct i915_address_space {
 	gen6_gtt_pte_t (*pte_encode)(dma_addr_t addr,
 				     enum i915_cache_level level,
 				     bool valid, u32 flags); /* Create a valid PTE */
+	int (*allocate_va_range)(struct i915_address_space *vm,
+				 uint64_t start,
+				 uint64_t length);
+	void (*teardown_va_range)(struct i915_address_space *vm,
+				  uint64_t start,
+				  uint64_t length);
 	void (*clear_range)(struct i915_address_space *vm,
 			    uint64_t start,
 			    uint64_t length,
@@ -229,6 +259,30 @@ struct i915_address_space {
 	void (*cleanup)(struct i915_address_space *vm);
 };
 
+struct i915_hw_ppgtt {
+	struct i915_address_space base;
+	struct kref ref;
+	struct drm_mm_node node;
+	unsigned num_pd_entries;
+	unsigned num_pd_pages; /* gen8+ */
+	union {
+		struct i915_pagedirpo pdp;
+		struct i915_pagedir pd;
+	};
+
+	struct i915_pagetab *scratch_pt;
+
+	struct intel_context *ctx;
+
+	gen6_gtt_pte_t __iomem *pd_addr;
+
+	int (*enable)(struct i915_hw_ppgtt *ppgtt);
+	int (*switch_mm)(struct i915_hw_ppgtt *ppgtt,
+			 struct intel_engine_cs *ring,
+			 bool synchronous);
+	void (*debug_dump)(struct i915_hw_ppgtt *ppgtt, struct seq_file *m);
+};
+
 /* The Graphics Translation Table is the way in which GEN hardware translates a
  * Graphics Virtual Address into a Physical Address. In addition to the normal
  * collateral associated with any va->pa translations GEN hardware also has a
@@ -257,47 +311,22 @@ struct i915_gtt {
 			  unsigned long *mappable_end);
 };
 
-struct i915_pagetab {
-	struct page *page;
-	dma_addr_t daddr;
-};
-
-struct i915_pagedir {
-	struct page *page; /* NULL for GEN6-GEN7 */
-	union {
-		uint32_t pd_offset;
-		dma_addr_t daddr;
-	};
-
-	struct i915_pagetab *page_tables[I915_PDES_PER_PD]; /* PDEs */
-};
-
-struct i915_pagedirpo {
-	/* struct page *page; */
-	struct i915_pagedir *pagedir[GEN8_LEGACY_PDPES];
-};
-
-struct i915_hw_ppgtt {
-	struct i915_address_space base;
-	struct kref ref;
-	struct drm_mm_node node;
-	unsigned num_pd_entries;
-	unsigned num_pd_pages; /* gen8+ */
-	union {
-		struct i915_pagedirpo pdp;
-		struct i915_pagedir pd;
-	};
-
-	struct intel_context *ctx;
-
-	gen6_gtt_pte_t __iomem *pd_addr;
-
-	int (*enable)(struct i915_hw_ppgtt *ppgtt);
-	int (*switch_mm)(struct i915_hw_ppgtt *ppgtt,
-			 struct intel_engine_cs *ring,
-			 bool synchronous);
-	void (*debug_dump)(struct i915_hw_ppgtt *ppgtt, struct seq_file *m);
-};
+/* For each pde iterates over every pde between from start until start + length.
+ * If start, and start+length are not perfectly divisible, the macro will round
+ * down, and up as needed. The macro modifies pde, start, and length. Dev is
+ * only used to differentiate shift values. Temp is temp.  On gen6/7, start = 0,
+ * and length = 2G effectively iterates over every PDE in the system. On gen8+
+ * it simply iterates over every page directory entry in a page directory.
+ *
+ * XXX: temp is not actually needed, but it saves doing the ALIGN operation.
+ */
+#define gen6_for_each_pde(pt, pd, start, length, temp, iter) \
+	for (iter = gen6_pde_index(start), pt = (pd)->page_tables[iter]; \
+	     length > 0 && iter < I915_PDES_PER_PD; \
+	     pt = (pd)->page_tables[++iter], \
+	     temp = ALIGN(start+1, 1 << GEN6_PDE_SHIFT) - start, \
+	     temp = min(temp, (unsigned)length), \
+	     start += temp, length -= temp)
 
 static inline uint32_t i915_pte_index(uint64_t address, uint32_t pde_shift)
 {
-- 
2.0.4

[PATCH 42/68] drm/i915: Extract context switch skip logic

[Ben Widawsky]

We have some fanciness coming up. This patch just breaks out the logic.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 51b517e..6d341ef 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -860,6 +860,16 @@ unpin_out:
 	return ret;
 }
 
+static inline bool should_skip_switch(struct intel_engine_cs *ring,
+				      struct intel_context *from,
+				      struct intel_context *to)
+{
+	if (from == to && !to->remap_slice)
+		return true;
+
+	return false;
+}
+
 /**
  * i915_switch_context() - perform a GPU context switch.
  * @ring: ring for which we'll execute the context switch
-- 
2.0.4

[PATCH 43/68] drm/i915: Track page table reload need

[Ben Widawsky]

This patch was formerly known as, "Force pd restore when PDEs change,
gen6-7." I had to change the name because it is needed for GEN8 too.

The real issue this is trying to solve is when a new object is mapped
into the current address space. The GPU does not snoop the new mapping
so we must do the gen specific action to reload the page tables.

GEN8 and GEN7 do differ in the way they load page tables for the RCS.
GEN8 does so with the context restore, while GEN7 requires the proper
load commands in the command streamer. Non-render is similar for both.

Caveat for GEN7
The docs say you cannot change the PDEs of a currently running context.
We never map new PDEs of a running context, and expect them to be
present - so I think this is okay. (We can unmap, but this should also
be okay since we only unmap unreferenced objects that the GPU shouldn't
be tryingto va->pa xlate.) The MI_SET_CONTEXT command does have a flag
to signal that even if the context is the same, force a reload. It's
unclear exactly what this does, but I have a hunch it's the right thing
to do.

The logic assumes that we always emit a context switch after mapping new
PDEs, and before we submit a batch. This is the case today, and has been
the case since the inception of hardware contexts. A note in the comment
let's the user know.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>

squash! drm/i915: Force pd restore when PDEs change, gen6-7

It's not just for gen8. If the current context has mappings change, we
need a context reload to switch
---
 drivers/gpu/drm/i915/i915_gem_context.c    | 55 ++++++++++++++++++++----------
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  6 ++++
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 16 ++++++++-
 drivers/gpu/drm/i915/i915_gem_gtt.h        |  2 ++
 4 files changed, 60 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 6d341ef..dade4e2 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -693,6 +693,8 @@ static int do_switch_xcs(struct intel_engine_cs *ring,
 		ret = ppgtt->switch_mm(ppgtt, ring, false);
 		if (ret)
 			return ret;
+		/* Context switch on GEN8 will reload pds also */
+		clear_bit(ring->id, &to->vm->pd_reload_mask);
 	}
 
 	if (from)
@@ -719,6 +721,25 @@ static void remap_l3(struct intel_engine_cs *ring,
 	}
 }
 
+static inline bool should_skip_switch(struct intel_engine_cs *ring,
+				      struct intel_context *from,
+				      struct intel_context *to)
+{
+	if (to->remap_slice)
+		return false;
+
+	if (from == to && !test_bit(ring->id, &to->vm->pd_reload_mask))
+		return true;
+
+	return false;
+}
+
+static bool
+needs_pd_load(struct intel_engine_cs *ring, struct intel_context *to)
+{
+	return (USES_FULL_PPGTT(ring->dev) || &to->vm->pd_reload_mask);
+}
+
 static int do_switch_rcs(struct intel_engine_cs *ring,
 			 struct intel_context *from,
 			 struct intel_context *to)
@@ -727,7 +748,6 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 	struct i915_hw_ppgtt *ppgtt = ctx_to_ppgtt(to);
 	u32 hw_flags = 0;
 	bool uninitialized = false;
-	bool needs_pd_load = (INTEL_INFO(ring->dev)->gen < 8) && USES_FULL_PPGTT(ring->dev);
 	int ret;
 
 	if (from != NULL) {
@@ -752,18 +772,21 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 	 * context. The default context cannot end up in evict everything (as
 	 * commented above) because it is always pinned.
 	 */
-	if (WARN_ON(from == to)) {
+	if (WARN_ON(from == to && should_skip_switch(ring, from, to))) {
 		ret = -EPERM;
 		goto unpin_out;
 	}
 
-	if (needs_pd_load) {
+	if (INTEL_INFO(ring->dev)->gen < 8 && needs_pd_load(ring, to)) {
 		/* Older GENs still want the load first, "PP_DCLV followed by
 		 * PP_DIR_BASE register through Load Register Immediate commands
 		 * in Ring Buffer before submitting a context."*/
 		ret = ppgtt->switch_mm(ppgtt, ring, false);
 		if (ret)
 			goto unpin_out;
+
+		/* Doing a PD load always reloads the page dirs */
+		clear_bit(ring->id, &to->vm->pd_reload_mask);
 	}
 
 	/*
@@ -784,22 +807,28 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 		i915_gem_vma_bind(vma, to->legacy_hw_ctx.rcs_state->cache_level, GLOBAL_BIND);
 	}
 
-	if (!to->legacy_hw_ctx.initialized || i915_gem_context_is_default(to)) {
+	if (!to->legacy_hw_ctx.initialized || i915_gem_context_is_default(to))
 		hw_flags |= MI_RESTORE_INHIBIT;
-		needs_pd_load = USES_FULL_PPGTT(ring->dev) && IS_GEN8(ring->dev);
-	}
+	else if (test_and_clear_bit(ring->id, &to->vm->pd_reload_mask))
+		hw_flags |= MI_FORCE_RESTORE;
 
 	ret = mi_set_context(ring, to, hw_flags);
 	if (ret)
 		goto unpin_out;
 
+	if (IS_GEN8(ring->dev)) {
+		/* Setting the context is enough to reload pds on gen8 */
+		clear_bit(ring->id, &to->vm->pd_reload_mask);
+	}
+
 	/* GEN8 does *not* require an explicit reload if the PDPs have been
 	 * setup, and we do not wish to move them.
 	 *
 	 * XXX: If we implemented page directory eviction code, this
 	 * optimization needs to be removed.
 	 */
-	if (needs_pd_load) {
+	if (IS_GEN8(ring->dev) && needs_pd_load(ring, to)) {
+		/* Doing a PD load always reloads the page dirs */
 		ret = ppgtt->switch_mm(ppgtt, ring, false);
 		/* The hardware context switch is emitted, but we haven't
 		 * actually changed the state - so it's probably safe to bail
@@ -860,16 +889,6 @@ unpin_out:
 	return ret;
 }
 
-static inline bool should_skip_switch(struct intel_engine_cs *ring,
-				      struct intel_context *from,
-				      struct intel_context *to)
-{
-	if (from == to && !to->remap_slice)
-		return true;
-
-	return false;
-}
-
 /**
  * i915_switch_context() - perform a GPU context switch.
  * @ring: ring for which we'll execute the context switch
@@ -898,7 +917,7 @@ int i915_switch_context(struct intel_engine_cs *ring,
 		return 0;
 	}
 
-	if (from == to && !to->remap_slice)
+	if (should_skip_switch(ring, from, to))
 		return 0;
 
 	if (IS_GEN8(ring->dev))
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index caccee9..fdd68d6 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -1096,6 +1096,8 @@ legacy_ringbuffer_submission(struct drm_device *dev, struct drm_file *file,
 	if (ret)
 		goto error;
 
+	WARN(ctx->vm->pd_reload_mask & (1<<ring->id), "%s didn't clear reload\n", ring->name);
+
 	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
 	instp_mask = I915_EXEC_CONSTANTS_MASK;
 	switch (instp_mode) {
@@ -1347,6 +1349,10 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	if (ret)
 		goto err;
 
+	/* XXX: Reserve has possibly change PDEs which means we must do a
+	 * context switch before we can coherently read some of the reserved
+	 * VMAs. */
+
 	/* The objects are in their final locations, apply the relocations. */
 	if (need_relocs)
 		ret = i915_gem_execbuffer_relocate(eb);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index e7adbff..c3df359 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1262,6 +1262,15 @@ int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 	return 0;
 }
 
+/* PDE TLBs are a pain invalidate pre GEN8. It requires a context reload. If we
+ * are switching between contexts with the same LRCA, we also must do a force
+ * restore.
+ */
+#define ppgtt_invalidate_tlbs(vm) do {\
+	/* If current vm != vm, */ \
+	vm->pd_reload_mask = INTEL_INFO(vm->dev)->ring_mask; \
+} while(0)
+
 static int
 ppgtt_bind_vma(struct i915_vma *vma,
 	       enum i915_cache_level cache_level,
@@ -1279,10 +1288,13 @@ ppgtt_bind_vma(struct i915_vma *vma,
 						 vma->node.size);
 		if (ret)
 			return ret;
+
+		ppgtt_invalidate_tlbs(vma->vm);
 	}
 
 	vma->vm->insert_entries(vma->vm, vma->obj->pages, vma->node.start,
 				cache_level, flags);
+
 	return 0;
 }
 
@@ -1292,9 +1304,11 @@ static void ppgtt_unbind_vma(struct i915_vma *vma)
 			     vma->node.start,
 			     vma->obj->base.size,
 			     true);
-	if (vma->vm->teardown_va_range)
+	if (vma->vm->teardown_va_range) {
 		vma->vm->teardown_va_range(vma->vm,
 					   vma->node.start, vma->node.size);
+		ppgtt_invalidate_tlbs(vma->vm);
+	}
 }
 
 extern int intel_iommu_gfx_mapped;
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 3729222..c89930d 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -215,6 +215,8 @@ struct i915_address_space {
 		struct page *page;
 	} scratch;
 
+	unsigned long pd_reload_mask;
+
 	/**
 	 * List of objects currently involved in rendering.
 	 *
-- 
2.0.4

[PATCH 44/68] drm/i915: Initialize all contexts

[Ben Widawsky]

The problem is we're going to switch to a new context, which could be
the default context. The plan was to use restore inhibit, which would be
fine, except if we are using dynamic page tables (which we will). If we
use dynamic page tables and we don't load new page tables, the previous
page tables might go away, and future operations will fault.

CTXA runs.
switch to default, restore inhibit
CTXA dies and has its address space taken away.
Run CTXB, tries to save using the context A's address space - this
fails.

The general solution is to make sure every context has it's own state,
and its own address space. For cases when we must restore inhibit, first
thing we do is load a valid address space. I thought this would be
enough, but apparently there are references within the context itself
which will refer to the old address space - therefore, we also must
reinitialize.

It was tricky to track this down as we don't have much insight into what
happens in a context save.

This is required for the next patch which enables dynamic page tables.

I need to go back and test this pre-GEN8

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index dade4e2..59c5173 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -807,9 +807,13 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 		i915_gem_vma_bind(vma, to->legacy_hw_ctx.rcs_state->cache_level, GLOBAL_BIND);
 	}
 
-	if (!to->legacy_hw_ctx.initialized || i915_gem_context_is_default(to))
+	if (!to->legacy_hw_ctx.initialized) {
 		hw_flags |= MI_RESTORE_INHIBIT;
-	else if (test_and_clear_bit(ring->id, &to->vm->pd_reload_mask))
+		/* NB: If we inhibit the restore, the context is not allowed to
+		 * die because future work may end up depending on valid address
+		 * space. This means we must enforce that a page table load
+		 * occur when this occurs. */
+	} else if (test_and_clear_bit(ring->id, &to->vm->pd_reload_mask))
 		hw_flags |= MI_FORCE_RESTORE;
 
 	ret = mi_set_context(ring, to, hw_flags);
@@ -823,12 +827,11 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 
 	/* GEN8 does *not* require an explicit reload if the PDPs have been
 	 * setup, and we do not wish to move them.
-	 *
-	 * XXX: If we implemented page directory eviction code, this
-	 * optimization needs to be removed.
 	 */
-	if (IS_GEN8(ring->dev) && needs_pd_load(ring, to)) {
-		/* Doing a PD load always reloads the page dirs */
+	if (IS_GEN8(ring->dev) && (hw_flags & MI_RESTORE_INHIBIT)) {
+		/* We have a valid page directory (scratch) to switch to. This
+		 * allows the old VM to be freed. Note that if anything occurs
+		 * between the set context, and here, we are f*cked */
 		ret = ppgtt->switch_mm(ppgtt, ring, false);
 		/* The hardware context switch is emitted, but we haven't
 		 * actually changed the state - so it's probably safe to bail
@@ -863,7 +866,7 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 		BUG_ON(from->legacy_hw_ctx.rcs_state->ring != ring);
 	}
 
-	uninitialized = !to->legacy_hw_ctx.initialized && from == NULL;
+	uninitialized = !to->legacy_hw_ctx.initialized;
 	to->legacy_hw_ctx.initialized = true;
 	/* From may have disappeared again after the context unref */
 	from = do_switch_fini_common(ring, from, to);
-- 
2.0.4

[PATCH 45/68] drm/i915: Finish gen6/7 dynamic page table allocation

[Ben Widawsky]

This patch continues on the idea from the previous patch. From here on,
in the steady state, PDEs are all pointing to the scratch page table (as
recommended in the spec). When an object is allocated in the VA range,
the code will determine if we need to allocate a page for the page
table. Similarly when the object is destroyed, we will remove, and free
the page table pointing the PDE back to the scratch page.

Following patches will work to unify the code a bit as we bring in GEN8
support. GEN6 and GEN8 are different enough that I had a hard time to
get to this point with as much common code as I do.

The aliasing PPGTT must pre-allocate all of the page tables. There are a
few reasons for this. Two trivial ones: aliasing ppgtt goes through the
ggtt paths, so it's hard to maintain, we currently do not restore the
default context (assuming the previous force reload is indeed
necessary). Most importantly though, the only way (it seems from
empirical evidence) to invalidate the CS TLBs on non-render ring is to
either use ring sync (which requires actually stopping the rings in
order to synchronize when the sync completes vs. where you are in
execution), or to reload DCLV.  Since without full PPGTT we do not ever
reload the DCLV register, there is no good way to achieve this. The
simplest solution is just to not support dynamic page table
creation/destruction in the aliasing PPGTT.

We could always reload DCLV, but this seems like quite a bit of excess
overhead only to save at most 2MB-4k of memory for the aliasing PPGTT
page tables.

v2: Make the page table bitmap declared inside the function (Chris)
Simplify the way scratching address space works.
Move the alloc/teardown tracepoints up a level in the call stack so that
both all implementations get the trace.

v3: Updated trace event to spit out a name

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c     |  19 ++++-
 drivers/gpu/drm/i915/i915_drv.h         |   7 ++
 drivers/gpu/drm/i915/i915_gem_context.c |   2 +-
 drivers/gpu/drm/i915/i915_gem_gtt.c     | 119 +++++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_gem_gtt.h     |   2 +-
 drivers/gpu/drm/i915/i915_trace.h       | 116 +++++++++++++++++++++++++++++++
 6 files changed, 254 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index ccecf00..47baf50 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1811,9 +1811,25 @@ static int i915_swizzle_info(struct seq_file *m, void *data)
 	return 0;
 }
 
+static size_t gen6_ppgtt_count_pt_pages(struct i915_hw_ppgtt *ppgtt)
+{
+	struct i915_pagedir *pd = &ppgtt->pd;
+	struct i915_pagetab **pt = &pd->page_tables[0];
+	size_t cnt = 0;
+	int i;
+
+	for (i = 0; i < ppgtt->num_pd_entries; i++) {
+		if (pt[i] != ppgtt->scratch_pt)
+			cnt++;
+	}
+
+	return cnt;
+}
+
 static void print_ppgtt(struct seq_file *m, struct i915_hw_ppgtt *ppgtt)
 {
 	seq_printf(m, "pd gtt offset: 0x%08x\n", ppgtt->pd.pd_offset);
+	seq_printf(m, "\tpd pages: %zu\n", gen6_ppgtt_count_pt_pages(ppgtt));
 }
 
 static void gen8_ppgtt_info(struct seq_file *m, struct drm_device *dev, int verbose)
@@ -1877,6 +1893,8 @@ static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev, bool ver
 		seq_printf(m, "PP_DIR_BASE_READ: 0x%08x\n", I915_READ(RING_PP_DIR_BASE_READ(ring)));
 		seq_printf(m, "PP_DIR_DCLV: 0x%08x\n", I915_READ(RING_PP_DIR_DCLV(ring)));
 	}
+	seq_printf(m, "ECOCHK: 0x%08x\n\n", I915_READ(GAM_ECOCHK));
+
 	if (dev_priv->mm.aliasing_ppgtt) {
 		struct i915_hw_ppgtt *ppgtt = dev_priv->mm.aliasing_ppgtt;
 
@@ -1895,7 +1913,6 @@ static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev, bool ver
 		idr_for_each(&file_priv->context_idr, per_file_ctx,
 			     (void *)((unsigned long)m | verbose));
 	}
-	seq_printf(m, "ECOCHK: 0x%08x\n", I915_READ(GAM_ECOCHK));
 }
 
 static int i915_ppgtt_info(struct seq_file *m, void *data)
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 651ad7f..beb9a66 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2491,6 +2491,13 @@ static inline bool i915_is_ggtt(struct i915_address_space *vm)
 	return vm == ggtt;
 }
 
+static inline bool i915_is_aliasing_ppgtt(struct i915_address_space *vm)
+{
+	struct i915_address_space *appgtt =
+		&((struct drm_i915_private *)(vm)->dev->dev_private)->mm.aliasing_ppgtt->base;
+	return vm == appgtt;
+}
+
 static inline bool i915_gem_obj_ggtt_bound(struct drm_i915_gem_object *obj)
 {
 	return i915_gem_obj_bound(obj, obj_to_ggtt(obj));
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 59c5173..7d81904 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -293,7 +293,7 @@ create_vm_for_ctx(struct drm_device *dev, struct intel_context *ctx)
 	if (!ppgtt)
 		return ERR_PTR(-ENOMEM);
 
-	ret = i915_gem_init_ppgtt(dev, ppgtt);
+	ret = i915_gem_init_ppgtt(dev, ppgtt, ctx->file_priv == NULL);
 	if (ret) {
 		kfree(ppgtt);
 		return ERR_PTR(ret);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index c3df359..14fc8f2 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1056,10 +1056,47 @@ static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 static int gen6_alloc_va_range(struct i915_address_space *vm,
 			       uint64_t start, uint64_t length)
 {
+	DECLARE_BITMAP(new_page_tables, I915_PDES_PER_PD);
+	struct drm_device *dev = vm->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct i915_hw_ppgtt *ppgtt =
 		        container_of(vm, struct i915_hw_ppgtt, base);
 	struct i915_pagetab *pt;
+	const uint32_t start_save = start, length_save = length;
 	uint32_t pde, temp;
+	int ret;
+
+	BUG_ON(upper_32_bits(start));
+
+	bitmap_zero(new_page_tables, I915_PDES_PER_PD);
+
+	/* The allocation is done in two stages so that we can bail out with
+	 * minimal amount of pain. The first stage finds new page tables that
+	 * need allocation. The second stage marks use ptes within the page
+	 * tables.
+	 */
+	gen6_for_each_pde(pt, &ppgtt->pd, start, length, temp, pde) {
+		if (pt != ppgtt->scratch_pt) {
+			WARN_ON(bitmap_empty(pt->used_ptes, GEN6_PTES_PER_PT));
+			continue;
+		}
+
+		/* We've already allocated a page table */
+		WARN_ON(!bitmap_empty(pt->used_ptes, GEN6_PTES_PER_PT));
+
+		pt = alloc_pt_single(dev);
+		if (IS_ERR(pt)) {
+			ret = PTR_ERR(pt);
+			goto unwind_out;
+		}
+
+		ppgtt->pd.page_tables[pde] = pt;
+		set_bit(pde, new_page_tables);
+		trace_i915_pagetable_alloc(vm, pde, start, GEN6_PDE_SHIFT);
+	}
+
+	start = start_save;
+	length = length_save;
 
 	gen6_for_each_pde(pt, &ppgtt->pd, start, length, temp, pde) {
 		int j;
@@ -1077,11 +1114,32 @@ static int gen6_alloc_va_range(struct i915_address_space *vm,
 			}
 		}
 
-		bitmap_or(pt->used_ptes, pt->used_ptes, tmp_bitmap,
+		if (test_and_clear_bit(pde, new_page_tables))
+			gen6_map_single(&ppgtt->pd, pde, pt);
+
+		trace_i915_pagetable_map(vm, pde, pt,
+					 gen6_pte_index(start),
+					 gen6_pte_count(start, length),
+					 GEN6_PTES_PER_PT);
+		bitmap_or(pt->used_ptes, tmp_bitmap, pt->used_ptes,
 			  GEN6_PTES_PER_PT);
 	}
 
+	WARN_ON(!bitmap_empty(new_page_tables, I915_PDES_PER_PD));
+
+	/* Make sure write is complete before other code can use this page
+	 * table. Also require for WC mapped PTEs */
+	readl(dev_priv->gtt.gsm);
+
 	return 0;
+
+unwind_out:
+	for_each_set_bit(pde, new_page_tables, I915_PDES_PER_PD) {
+		struct i915_pagetab *pt = ppgtt->pd.page_tables[pde];
+		ppgtt->pd.page_tables[pde] = NULL;
+		free_pt_single(pt, vm->dev);
+	}
+	return ret;
 }
 
 static void gen6_teardown_va_range(struct i915_address_space *vm,
@@ -1093,8 +1151,27 @@ static void gen6_teardown_va_range(struct i915_address_space *vm,
 	uint32_t pde, temp;
 
 	gen6_for_each_pde(pt, &ppgtt->pd, start, length, temp, pde) {
+
+		if (WARN(pt == ppgtt->scratch_pt,
+		    "Tried to teardown scratch page vm %p. pde %u: %llx-%llx\n",
+		    vm, pde, start, start + length))
+			continue;
+
+		trace_i915_pagetable_unmap(vm, pde, pt,
+					   gen6_pte_index(start),
+					   gen6_pte_count(start, length),
+					   GEN6_PTES_PER_PT);
+
 		bitmap_clear(pt->used_ptes, gen6_pte_index(start),
 			     gen6_pte_count(start, length));
+
+		if (bitmap_empty(pt->used_ptes, GEN6_PTES_PER_PT)) {
+			trace_i915_pagetable_destroy(vm, pde,
+						     start & GENMASK_ULL(64, GEN6_PDE_SHIFT),
+						     GEN6_PDE_SHIFT);
+			gen6_map_single(&ppgtt->pd, pde, ppgtt->scratch_pt);
+			ppgtt->pd.page_tables[pde] = ppgtt->scratch_pt;
+		}
 	}
 }
 
@@ -1102,9 +1179,13 @@ static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
 	int i;
 
-	for (i = 0; i < ppgtt->num_pd_entries; i++)
-		free_pt_single(ppgtt->pd.page_tables[i], ppgtt->base.dev);
+	for (i = 0; i < ppgtt->num_pd_entries; i++) {
+		struct i915_pagetab *pt = ppgtt->pd.page_tables[i];
+		if (pt != ppgtt->scratch_pt)
+			free_pt_single(ppgtt->pd.page_tables[i], ppgtt->base.dev);
+	}
 
+	/* Consider putting this as part of pd free. */
 	free_pt_scratch(ppgtt->scratch_pt, ppgtt->base.dev);
 	free_pd_single(&ppgtt->pd, ppgtt->base.dev);
 }
@@ -1170,7 +1251,7 @@ err_out:
 	return ret;
 }
 
-static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
+static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt, bool preallocate_pt)
 {
 	int ret;
 
@@ -1178,9 +1259,13 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
 	if (ret)
 		return ret;
 
+	if (!preallocate_pt)
+		return 0;
+
 	ret = alloc_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries,
 			     ppgtt->base.dev);
 	if (ret) {
+		free_pt_scratch(ppgtt->scratch_pt, ppgtt->base.dev);
 		drm_mm_remove_node(&ppgtt->node);
 		return ret;
 	}
@@ -1188,8 +1273,17 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt)
 	return 0;
 }
 
+static void gen6_scratch_va_range(struct i915_hw_ppgtt *ppgtt,
+				  uint64_t start, uint64_t length)
+{
+	struct i915_pagetab *unused;
+	uint32_t pde, temp;
+
+	gen6_for_each_pde(unused, &ppgtt->pd, start, length, temp, pde)
+		ppgtt->pd.page_tables[pde] = ppgtt->scratch_pt;
+}
 
-static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
+static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt, bool aliasing)
 {
 	struct drm_device *dev = ppgtt->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -1208,7 +1302,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	} else
 		BUG();
 
-	ret = gen6_ppgtt_alloc(ppgtt);
+	ret = gen6_ppgtt_alloc(ppgtt, aliasing);
 	if (ret)
 		return ret;
 
@@ -1227,6 +1321,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->pd_addr = (gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm +
 		ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t);
 
+	if (!aliasing)
+		gen6_scratch_va_range(ppgtt, 0, ppgtt->base.total);
+
 	gen6_map_page_range(dev_priv, &ppgtt->pd, 0, ppgtt->base.total);
 
 	DRM_DEBUG_DRIVER("Allocated pde space (%ldM) at GTT entry: %lx\n",
@@ -1236,7 +1333,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	return 0;
 }
 
-int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
+int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt, bool aliasing)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	int ret = 0;
@@ -1245,7 +1342,7 @@ int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 	ppgtt->base.scratch = dev_priv->gtt.base.scratch;
 
 	if (INTEL_INFO(dev)->gen < 8)
-		ret = gen6_ppgtt_init(ppgtt);
+		ret = gen6_ppgtt_init(ppgtt, aliasing);
 	else if (IS_GEN8(dev))
 		ret = gen8_ppgtt_init(ppgtt, dev_priv->gtt.base.total);
 	else
@@ -1283,6 +1380,8 @@ ppgtt_bind_vma(struct i915_vma *vma,
 		flags |= PTE_READ_ONLY;
 
 	if (vma->vm->allocate_va_range) {
+		trace_i915_va_alloc(vma->vm, vma->node.start, vma->node.size,
+				    VM_TO_TRACE_NAME(vma->vm));
 		ret = vma->vm->allocate_va_range(vma->vm,
 						 vma->node.start,
 						 vma->node.size);
@@ -1305,6 +1404,10 @@ static void ppgtt_unbind_vma(struct i915_vma *vma)
 			     vma->obj->base.size,
 			     true);
 	if (vma->vm->teardown_va_range) {
+		trace_i915_va_teardown(vma->vm,
+				       vma->node.start, vma->node.size,
+				       VM_TO_TRACE_NAME(vma->vm));
+
 		vma->vm->teardown_va_range(vma->vm,
 					   vma->node.start, vma->node.size);
 		ppgtt_invalidate_tlbs(vma->vm);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index c89930d..73968a3 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -430,7 +430,7 @@ static inline size_t gtt_total_entries(struct i915_gtt *gtt)
 	return gtt->base.total >> PAGE_SHIFT;
 }
 
-int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt);
+int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt, bool aliasing);
 
 void i915_check_and_clear_faults(struct drm_device *dev);
 void i915_gem_suspend_gtt_mappings(struct drm_device *dev);
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index cbf5521..2d21c54 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -156,6 +156,122 @@ TRACE_EVENT(i915_vma_unbind,
 		      __entry->obj, __entry->offset, __entry->size, __entry->vm)
 );
 
+#define VM_TO_TRACE_NAME(vm) \
+	(i915_is_ggtt(vm) ? "GGTT" : \
+	 i915_is_aliasing_ppgtt(vm) ? "Aliasing PPGTT" : \
+				      "Private VM")
+
+DECLARE_EVENT_CLASS(i915_va,
+	TP_PROTO(struct i915_address_space *vm, u64 start, u64 length, const char *name),
+	TP_ARGS(vm, start, length, name),
+
+	TP_STRUCT__entry(
+		__field(struct i915_address_space *, vm)
+		__field(u64, start)
+		__field(u64, end)
+		__string(name, name)
+	),
+
+	TP_fast_assign(
+		__entry->vm = vm;
+		__entry->start = start;
+		__entry->end = start + length;
+		__assign_str(name, name);
+	),
+
+	TP_printk("vm=%p (%s), 0x%llx-0x%llx",
+		  __entry->vm, __get_str(name),  __entry->start, __entry->end)
+);
+
+DEFINE_EVENT(i915_va, i915_va_alloc,
+	     TP_PROTO(struct i915_address_space *vm, u64 start, u64 length, const char *name),
+	     TP_ARGS(vm, start, length, name)
+);
+
+DEFINE_EVENT(i915_va, i915_va_teardown,
+	     TP_PROTO(struct i915_address_space *vm, u64 start, u64 length, const char *name),
+	     TP_ARGS(vm, start, length, name)
+);
+
+DECLARE_EVENT_CLASS(i915_pagetable,
+	TP_PROTO(struct i915_address_space *vm, u32 pde, u64 start, u64 pde_shift),
+	TP_ARGS(vm, pde, start, pde_shift),
+
+	TP_STRUCT__entry(
+		__field(struct i915_address_space *, vm)
+		__field(u32, pde)
+		__field(u64, start)
+		__field(u64, end)
+	),
+
+	TP_fast_assign(
+		__entry->vm = vm;
+		__entry->pde = pde;
+		__entry->start = start;
+		__entry->end = (start + (1ULL << pde_shift)) & ~((1ULL << pde_shift)-1);
+	),
+
+	TP_printk("vm=%p, pde=%d (0x%llx-0x%llx)",
+		  __entry->vm, __entry->pde, __entry->start, __entry->end)
+);
+
+DEFINE_EVENT(i915_pagetable, i915_pagetable_alloc,
+	     TP_PROTO(struct i915_address_space *vm, u32 pde, u64 start, u64 pde_shift),
+	     TP_ARGS(vm, pde, start, pde_shift)
+);
+
+DEFINE_EVENT(i915_pagetable, i915_pagetable_destroy,
+	     TP_PROTO(struct i915_address_space *vm, u32 pde, u64 start, u64 pde_shift),
+	     TP_ARGS(vm, pde, start, pde_shift)
+);
+
+/* Avoid extra math because we only support two sizes. The format is defined by
+ * bitmap_scnprintf. Each 32 bits is 8 HEX digits followed by comma */
+#define TRACE_PT_SIZE(bits) \
+	((((bits) == 1024) ? 288 : 144) + 1)
+
+DECLARE_EVENT_CLASS(i915_pagetable_update,
+	TP_PROTO(struct i915_address_space *vm, u32 pde,
+		 struct i915_pagetab *pt, u32 first, u32 len, size_t bits),
+	TP_ARGS(vm, pde, pt, first, len, bits),
+
+	TP_STRUCT__entry(
+		__field(struct i915_address_space *, vm)
+		__field(u32, pde)
+		__field(u32, first)
+		__field(u32, last)
+		__dynamic_array(char, cur_ptes, TRACE_PT_SIZE(bits))
+	),
+
+	TP_fast_assign(
+		__entry->vm = vm;
+		__entry->pde = pde;
+		__entry->first = first;
+		__entry->last = first + len;
+
+		bitmap_scnprintf(__get_str(cur_ptes),
+				 TRACE_PT_SIZE(bits),
+				 pt->used_ptes,
+				 bits);
+	),
+
+	TP_printk("vm=%p, pde=%d, updating %u:%u\t%s",
+		  __entry->vm, __entry->pde, __entry->last, __entry->first,
+		  __get_str(cur_ptes))
+);
+
+DEFINE_EVENT(i915_pagetable_update, i915_pagetable_map,
+	TP_PROTO(struct i915_address_space *vm, u32 pde,
+		 struct i915_pagetab *pt, u32 first, u32 len, size_t bits),
+	TP_ARGS(vm, pde, pt, first, len, bits)
+);
+
+DEFINE_EVENT(i915_pagetable_update, i915_pagetable_unmap,
+	TP_PROTO(struct i915_address_space *vm, u32 pde,
+		 struct i915_pagetab *pt, u32 first, u32 len, size_t bits),
+	TP_ARGS(vm, pde, pt, first, len, bits)
+);
+
 TRACE_EVENT(i915_gem_object_change_domain,
 	    TP_PROTO(struct drm_i915_gem_object *obj, u32 old_read, u32 old_write),
 	    TP_ARGS(obj, old_read, old_write),
-- 
2.0.4

[PATCH 46/68] drm/i915/bdw: Use dynamic allocation idioms on free

[Ben Widawsky]

The page directory freer is left here for now as it's still useful given
that GEN8 still preallocates. Once the allocation functions are broken
up into more discrete chunks, we'll follow suit and destroy this
leftover piece.

comments

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 45 ++++++++++++++++++++++++-------------
 drivers/gpu/drm/i915/i915_gem_gtt.h | 26 +++++++++++++++++++++
 2 files changed, 55 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 14fc8f2..65b1c58 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -523,27 +523,40 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	}
 }
 
-static void gen8_free_page_tables(struct i915_pagedir *pd, struct drm_device *dev)
+static void gen8_teardown_va_range(struct i915_address_space *vm,
+				   uint64_t start, uint64_t length)
 {
-	int i;
-
-	if (!pd->page)
-		return;
-
-	for (i = 0; i < I915_PDES_PER_PD; i++) {
-		free_pt_single(pd->page_tables[i], dev);
-		pd->page_tables[i] = NULL;
+	struct i915_hw_ppgtt *ppgtt =
+		        container_of(vm, struct i915_hw_ppgtt, base);
+	struct i915_pagedir *pd;
+	struct i915_pagetab *pt;
+	uint64_t temp;
+	uint32_t pdpe, pde;
+
+	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
+		uint64_t pd_len = gen8_clamp_pd(start, length);
+		uint64_t pd_start = start;
+		gen8_for_each_pde(pt, pd, pd_start, pd_len, temp, pde) {
+			free_pt_single(pt, vm->dev);
+		}
+		free_pd_single(pd, vm->dev);
 	}
 }
 
-static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
+/* This function will die soon */
+static void gen8_free_full_pagedir(struct i915_hw_ppgtt *ppgtt, int i)
 {
-	int i;
+	gen8_teardown_va_range(&ppgtt->base,
+			       i << GEN8_PDPE_SHIFT,
+			       (1 << GEN8_PDPE_SHIFT));
+}
 
-	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		gen8_free_page_tables(ppgtt->pdp.pagedirs[i], ppgtt->base.dev);
-		free_pd_single(ppgtt->pdp.pagedirs[i], ppgtt->base.dev);
-	}
+static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
+{
+	trace_i915_va_teardown(&ppgtt->base,
+			       ppgtt->base.start, ppgtt->base.total);
+	gen8_teardown_va_range(&ppgtt->base,
+			       ppgtt->base.start, ppgtt->base.total);
 }
 
 static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
@@ -572,7 +585,7 @@ static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
 
 unwind_out:
 	while (i--)
-		gen8_free_page_tables(ppgtt->pdp.pagedirs[i], ppgtt->base.dev);
+		gen8_free_full_pagedir(ppgtt, i);
 
 	return -ENOMEM;
 }
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 73968a3..2cc2e87 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -400,6 +400,32 @@ static inline size_t gen6_pde_count(uint32_t addr, uint32_t length)
 	return i915_pde_count(addr, length, GEN6_PDE_SHIFT);
 }
 
+#define gen8_for_each_pde(pt, pd, start, length, temp, iter)		\
+	for (iter = gen8_pde_index(start), pt = (pd)->page_tables[iter]; \
+	     length > 0 && iter < I915_PDES_PER_PD;			\
+	     pt = (pd)->page_tables[++iter],				\
+	     temp = ALIGN(start+1, 1 << GEN8_PDE_SHIFT) - start,	\
+	     temp = min(temp, length),					\
+	     start += temp, length -= temp)
+
+#define gen8_for_each_pdpe(pd, pdp, start, length, temp, iter)		\
+	for (iter = gen8_pdpe_index(start), pd = (pdp)->pagedirs[iter];	\
+	     length > 0 && iter < GEN8_LEGACY_PDPES;			\
+	     pd = (pdp)->pagedirs[++iter],				\
+	     temp = ALIGN(start+1, 1 << GEN8_PDPE_SHIFT) - start,	\
+	     temp = min(temp, length),					\
+	     start += temp, length -= temp)
+
+/* Clamp length to the next pagedir boundary */
+static inline uint64_t gen8_clamp_pd(uint64_t start, uint64_t length)
+{
+	uint64_t next_pd = ALIGN(start + 1, 1 << GEN8_PDPE_SHIFT);
+	if (next_pd > (start + length))
+		return length;
+
+	return next_pd - start;
+}
+
 static inline uint32_t gen8_pte_index(uint64_t address)
 {
 	return i915_pte_index(address, GEN8_PDE_SHIFT);
-- 
2.0.4

[PATCH 47/68] drm/i915/bdw: pagedirs rework allocation

[Ben Widawsky]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 43 ++++++++++++++++++++++++++-----------
 1 file changed, 31 insertions(+), 12 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 65b1c58..5447a99 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -538,8 +538,10 @@ static void gen8_teardown_va_range(struct i915_address_space *vm,
 		uint64_t pd_start = start;
 		gen8_for_each_pde(pt, pd, pd_start, pd_len, temp, pde) {
 			free_pt_single(pt, vm->dev);
+			pd->page_tables[pde] = NULL;
 		}
 		free_pd_single(pd, vm->dev);
+		ppgtt->pdp.pagedirs[pdpe] = NULL;
 	}
 }
 
@@ -590,26 +592,40 @@ unwind_out:
 	return -ENOMEM;
 }
 
-static int gen8_ppgtt_allocate_page_directories(struct i915_hw_ppgtt *ppgtt,
-						const int max_pdp)
+static int gen8_ppgtt_alloc_pagedirs(struct i915_pagedirpo *pdp,
+				     uint64_t start,
+				     uint64_t length)
 {
-	int i;
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(pdp, struct i915_hw_ppgtt, pdp);
+	struct i915_pagedir *unused;
+	uint64_t temp;
+	uint32_t pdpe;
 
-	for (i = 0; i < max_pdp; i++) {
-		ppgtt->pdp.pagedirs[i] = alloc_pd_single(ppgtt->base.dev);
-		if (IS_ERR(ppgtt->pdp.pagedirs[i]))
+	/* FIXME: PPGTT container_of won't work for 64b */
+	BUG_ON((start + length) > 0x800000000ULL);
+
+	gen8_for_each_pdpe(unused, pdp, start, length, temp, pdpe) {
+		BUG_ON(unused);
+		pdp->pagedirs[pdpe] = alloc_pd_single(ppgtt->base.dev);
+		if (IS_ERR(ppgtt->pdp.pagedirs[pdpe]))
 			goto unwind_out;
+
+		ppgtt->num_pd_pages++;
 	}
 
-	ppgtt->num_pd_pages = max_pdp;
 	BUG_ON(ppgtt->num_pd_pages > GEN8_LEGACY_PDPES);
 
 	return 0;
 
 unwind_out:
-	while (i--)
-		free_pd_single(ppgtt->pdp.pagedirs[i],
+	while (pdpe--) {
+		free_pd_single(ppgtt->pdp.pagedirs[pdpe],
 			       ppgtt->base.dev);
+		ppgtt->num_pd_pages--;
+	}
+
+	WARN_ON(ppgtt->num_pd_pages);
 
 	return -ENOMEM;
 }
@@ -619,7 +635,8 @@ static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
 {
 	int ret;
 
-	ret = gen8_ppgtt_allocate_page_directories(ppgtt, max_pdp);
+	ret = gen8_ppgtt_alloc_pagedirs(&ppgtt->pdp, ppgtt->base.start,
+					ppgtt->base.total);
 	if (ret)
 		return ret;
 
@@ -656,6 +673,10 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	if (size % (1<<30))
 		DRM_INFO("Pages will be wasted unless GTT size (%llu) is divisible by 1GB\n", size);
 
+	ppgtt->base.start = 0;
+	ppgtt->base.total = size;
+	BUG_ON(ppgtt->base.total == 0);
+
 	/* 1. Do all our allocations for page directories and page tables. */
 	ret = gen8_ppgtt_alloc(ppgtt, max_pdp);
 	if (ret)
@@ -689,8 +710,6 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
 	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
-	ppgtt->base.start = 0;
-	ppgtt->base.total = ppgtt->num_pd_entries * GEN8_PTES_PER_PT * PAGE_SIZE;
 
 	DRM_DEBUG_DRIVER("Allocated %d pages for page directories (%d wasted)\n",
 			 ppgtt->num_pd_pages, ppgtt->num_pd_pages - max_pdp);
-- 
2.0.4

[PATCH 48/68] drm/i915/bdw: pagetable allocation rework

[Ben Widawsky]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 54 ++++++++++++++++++++-----------------
 drivers/gpu/drm/i915/i915_gem_gtt.h | 10 +++++++
 2 files changed, 39 insertions(+), 25 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 5447a99..0426222 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -545,14 +545,6 @@ static void gen8_teardown_va_range(struct i915_address_space *vm,
 	}
 }
 
-/* This function will die soon */
-static void gen8_free_full_pagedir(struct i915_hw_ppgtt *ppgtt, int i)
-{
-	gen8_teardown_va_range(&ppgtt->base,
-			       i << GEN8_PDPE_SHIFT,
-			       (1 << GEN8_PDPE_SHIFT));
-}
-
 static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
 	trace_i915_va_teardown(&ppgtt->base,
@@ -572,22 +564,27 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
 	gen8_ppgtt_free(ppgtt);
 }
 
-static int gen8_ppgtt_allocate_page_tables(struct i915_hw_ppgtt *ppgtt)
+static int gen8_ppgtt_alloc_pagetabs(struct i915_pagedir *pd,
+				     uint64_t start,
+				     uint64_t length,
+				     struct drm_device *dev)
 {
-	int i, ret;
+	struct i915_pagetab *unused;
+	uint64_t temp;
+	uint32_t pde;
 
-	for (i = 0; i < ppgtt->num_pd_pages; i++) {
-		ret = alloc_pt_range(ppgtt->pdp.pagedirs[i],
-				     0, I915_PDES_PER_PD, ppgtt->base.dev);
-		if (ret)
+	gen8_for_each_pde(unused, pd, start, length, temp, pde) {
+		BUG_ON(unused);
+		pd->page_tables[pde] = alloc_pt_single(dev);
+		if (IS_ERR(pd->page_tables[pde]))
 			goto unwind_out;
 	}
 
 	return 0;
 
 unwind_out:
-	while (i--)
-		gen8_free_full_pagedir(ppgtt, i);
+	while (pde--)
+		free_pt_single(pd->page_tables[pde], dev);
 
 	return -ENOMEM;
 }
@@ -631,20 +628,28 @@ unwind_out:
 }
 
 static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
-			    const int max_pdp)
+			    uint64_t start,
+			    uint64_t length)
 {
+	struct i915_pagedir *pd;
+	uint64_t temp;
+	uint32_t pdpe;
 	int ret;
 
-	ret = gen8_ppgtt_alloc_pagedirs(&ppgtt->pdp, ppgtt->base.start,
-					ppgtt->base.total);
+	ret = gen8_ppgtt_alloc_pagedirs(&ppgtt->pdp, start, length);
 	if (ret)
 		return ret;
 
-	ret = gen8_ppgtt_allocate_page_tables(ppgtt);
-	if (ret)
-		goto err_out;
+	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
+		ret = gen8_ppgtt_alloc_pagetabs(pd, start, length,
+						ppgtt->base.dev);
+		if (ret)
+			goto err_out;
+
+		ppgtt->num_pd_entries += I915_PDES_PER_PD;
+	}
 
-	ppgtt->num_pd_entries = max_pdp * I915_PDES_PER_PD;
+	BUG_ON(pdpe > ppgtt->num_pd_pages);
 
 	return 0;
 
@@ -675,10 +680,9 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 
 	ppgtt->base.start = 0;
 	ppgtt->base.total = size;
-	BUG_ON(ppgtt->base.total == 0);
 
 	/* 1. Do all our allocations for page directories and page tables. */
-	ret = gen8_ppgtt_alloc(ppgtt, max_pdp);
+	ret = gen8_ppgtt_alloc(ppgtt, ppgtt->base.start, ppgtt->base.total);
 	if (ret)
 		return ret;
 
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 2cc2e87..d635528 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -416,6 +416,16 @@ static inline size_t gen6_pde_count(uint32_t addr, uint32_t length)
 	     temp = min(temp, length),					\
 	     start += temp, length -= temp)
 
+/* Clamp length to the next pagetab boundary */
+static inline uint64_t gen8_clamp_pt(uint64_t start, uint64_t length)
+{
+	uint64_t next_pt = ALIGN(start + 1, 1 << GEN8_PDE_SHIFT);
+	if (next_pt > (start + length))
+		return length;
+
+	return next_pt - start;
+}
+
 /* Clamp length to the next pagedir boundary */
 static inline uint64_t gen8_clamp_pd(uint64_t start, uint64_t length)
 {
-- 
2.0.4

[PATCH 49/68] drm/i915/bdw: Make the pdp switch a bit less hacky

[Ben Widawsky]

One important part of this patch is we now write a scratch page
directory into any unused PDP descriptors. This matters for 2 reasons,
first, it's not clear we're allowed to just use 0, or an invalid
pointer, and second, we must wipe out any previous contents from the last
context.

The latter point only matters with full PPGTT. The former point would
only effect 32b platforms, or platforms with less than 4GB memory.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 34 +++++++++++++++++++++-------------
 drivers/gpu/drm/i915/i915_gem_gtt.h |  5 ++++-
 2 files changed, 25 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 0426222..5875071 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -382,8 +382,10 @@ static struct i915_pagedir *alloc_pd_single(struct drm_device *dev)
 }
 
 /* Broadwell Page Directory Pointer Descriptors */
-static int gen8_write_pdp(struct intel_engine_cs *ring, unsigned entry,
-			   uint64_t val, bool synchronous)
+static int gen8_write_pdp(struct intel_engine_cs *ring,
+			  unsigned entry,
+			  dma_addr_t addr,
+			  bool synchronous)
 {
 	struct drm_i915_private *dev_priv = ring->dev->dev_private;
 	int ret;
@@ -391,8 +393,8 @@ static int gen8_write_pdp(struct intel_engine_cs *ring, unsigned entry,
 	BUG_ON(entry >= 4);
 
 	if (synchronous) {
-		I915_WRITE(GEN8_RING_PDP_UDW(ring, entry), val >> 32);
-		I915_WRITE(GEN8_RING_PDP_LDW(ring, entry), (u32)val);
+		I915_WRITE(GEN8_RING_PDP_UDW(ring, entry), upper_32_bits(addr));
+		I915_WRITE(GEN8_RING_PDP_LDW(ring, entry), lower_32_bits(addr));
 		return 0;
 	}
 
@@ -402,10 +404,10 @@ static int gen8_write_pdp(struct intel_engine_cs *ring, unsigned entry,
 
 	intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 	intel_ring_emit(ring, GEN8_RING_PDP_UDW(ring, entry));
-	intel_ring_emit(ring, (u32)(val >> 32));
+	intel_ring_emit(ring, upper_32_bits(addr));
 	intel_ring_emit(ring, MI_LOAD_REGISTER_IMM(1));
 	intel_ring_emit(ring, GEN8_RING_PDP_LDW(ring, entry));
-	intel_ring_emit(ring, (u32)(val));
+	intel_ring_emit(ring, lower_32_bits(addr));
 	intel_ring_advance(ring);
 
 	return 0;
@@ -417,12 +419,12 @@ static int gen8_mm_switch(struct i915_hw_ppgtt *ppgtt,
 {
 	int i, ret;
 
-	/* bit of a hack to find the actual last used pd */
-	int used_pd = ppgtt->num_pd_entries / I915_PDES_PER_PD;
-
-	for (i = used_pd - 1; i >= 0; i--) {
-		dma_addr_t addr = ppgtt->pdp.pagedirs[i]->daddr;
-		ret = gen8_write_pdp(ring, i, addr, synchronous);
+	for (i = GEN8_LEGACY_PDPES - 1; i >= 0; i--) {
+		struct i915_pagedir *pd = ppgtt->pdp.pagedirs[i];
+		dma_addr_t pd_daddr = pd ? pd->daddr : ppgtt->scratch_pd->daddr;
+		/* The page directory might be NULL, but we need to clear out
+		 * whatever the previous context might have used. */
+		ret = gen8_write_pdp(ring, i, pd_daddr, synchronous);
 		if (ret)
 			return ret;
 	}
@@ -681,10 +683,16 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->base.start = 0;
 	ppgtt->base.total = size;
 
+	ppgtt->scratch_pd = alloc_pt_scratch(ppgtt->base.dev);
+	if (IS_ERR(ppgtt->scratch_pd))
+		return PTR_ERR(ppgtt->scratch_pd);
+
 	/* 1. Do all our allocations for page directories and page tables. */
 	ret = gen8_ppgtt_alloc(ppgtt, ppgtt->base.start, ppgtt->base.total);
-	if (ret)
+	if (ret) {
+		free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
 		return ret;
+	}
 
 	/*
 	 * 2. Map all the page directory entires to point to the page tables
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index d635528..3c28365 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -272,7 +272,10 @@ struct i915_hw_ppgtt {
 		struct i915_pagedir pd;
 	};
 
-	struct i915_pagetab *scratch_pt;
+	union {
+		struct i915_pagetab *scratch_pt;
+		struct i915_pagetab *scratch_pd; /* Just need the daddr */
+	};
 
 	struct intel_context *ctx;
 
-- 
2.0.4

[PATCH 50/68] drm/i915: num_pd_pages/num_pd_entries isn't useful

[Ben Widawsky]

These values are never quite useful for dynamic allocations of the page
tables. Getting rid of them will help prevent later confusion.

TODO: this probably needs to be earlier in the series

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c | 11 ++++-----
 drivers/gpu/drm/i915/i915_gem_gtt.c | 45 ++++++++++---------------------------
 drivers/gpu/drm/i915/i915_gem_gtt.h |  7 ++++--
 3 files changed, 21 insertions(+), 42 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 47baf50..09a64e6 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1813,13 +1813,12 @@ static int i915_swizzle_info(struct seq_file *m, void *data)
 
 static size_t gen6_ppgtt_count_pt_pages(struct i915_hw_ppgtt *ppgtt)
 {
-	struct i915_pagedir *pd = &ppgtt->pd;
-	struct i915_pagetab **pt = &pd->page_tables[0];
+	struct i915_pagetab *pt;
 	size_t cnt = 0;
-	int i;
+	uint32_t useless;
 
-	for (i = 0; i < ppgtt->num_pd_entries; i++) {
-		if (pt[i] != ppgtt->scratch_pt)
+	gen6_for_all_pdes(pt, ppgtt, useless) {
+		if (pt != ppgtt->scratch_pt)
 			cnt++;
 	}
 
@@ -1842,8 +1841,6 @@ static void gen8_ppgtt_info(struct seq_file *m, struct drm_device *dev, int verb
 	if (!ppgtt)
 		return;
 
-	seq_printf(m, "Page directories: %d\n", ppgtt->num_pd_pages);
-	seq_printf(m, "Page tables: %d\n", ppgtt->num_pd_entries);
 	for_each_ring(ring, dev_priv, unused) {
 		seq_printf(m, "%s\n", ring->name);
 		for (i = 0; i < 4; i++) {
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 5875071..d8e0a62 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -609,22 +609,14 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_pagedirpo *pdp,
 		pdp->pagedirs[pdpe] = alloc_pd_single(ppgtt->base.dev);
 		if (IS_ERR(ppgtt->pdp.pagedirs[pdpe]))
 			goto unwind_out;
-
-		ppgtt->num_pd_pages++;
 	}
 
-	BUG_ON(ppgtt->num_pd_pages > GEN8_LEGACY_PDPES);
-
 	return 0;
 
 unwind_out:
-	while (pdpe--) {
+	while (pdpe--)
 		free_pd_single(ppgtt->pdp.pagedirs[pdpe],
 			       ppgtt->base.dev);
-		ppgtt->num_pd_pages--;
-	}
-
-	WARN_ON(ppgtt->num_pd_pages);
 
 	return -ENOMEM;
 }
@@ -647,12 +639,8 @@ static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
 						ppgtt->base.dev);
 		if (ret)
 			goto err_out;
-
-		ppgtt->num_pd_entries += I915_PDES_PER_PD;
 	}
 
-	BUG_ON(pdpe > ppgtt->num_pd_pages);
-
 	return 0;
 
 	/* TODO: Check this for all cases */
@@ -674,7 +662,6 @@ err_out:
 static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 {
 	const int max_pdp = DIV_ROUND_UP(size, 1 << 30);
-	const int min_pt_pages = I915_PDES_PER_PD * max_pdp;
 	int i, j, ret;
 
 	if (size % (1<<30))
@@ -723,27 +710,21 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
 
-	DRM_DEBUG_DRIVER("Allocated %d pages for page directories (%d wasted)\n",
-			 ppgtt->num_pd_pages, ppgtt->num_pd_pages - max_pdp);
-	DRM_DEBUG_DRIVER("Allocated %d pages for page tables (%lld wasted)\n",
-			 ppgtt->num_pd_entries,
-			 (ppgtt->num_pd_entries - min_pt_pages) + size % (1<<30));
 	return 0;
 }
 
 static void gen6_dump_ppgtt(struct i915_hw_ppgtt *ppgtt, struct seq_file *m)
 {
 	struct i915_address_space *vm = &ppgtt->base;
+	struct i915_pagetab *unused;
 	gen6_gtt_pte_t scratch_pte;
 	uint32_t pd_entry;
-	int pte, pde;
+	uint32_t  pte, pde, temp;
+	uint32_t start = ppgtt->base.start, length = ppgtt->base.total;
 
 	scratch_pte = vm->pte_encode(vm->scratch.addr, I915_CACHE_LLC, true, 0);
 
-	seq_printf(m, "  VM %p (pd_offset %x-%x):\n", vm,
-		   ppgtt->pd.pd_offset,
-		   ppgtt->pd.pd_offset + ppgtt->num_pd_entries);
-	for (pde = 0; pde < ppgtt->num_pd_entries; pde++) {
+	gen6_for_each_pde(unused, &ppgtt->pd, start, length, temp, pde) {
 		u32 expected;
 		gen6_gtt_pte_t *pt_vaddr;
 		dma_addr_t pt_addr = ppgtt->pd.page_tables[pde]->daddr;
@@ -1221,12 +1202,12 @@ static void gen6_teardown_va_range(struct i915_address_space *vm,
 
 static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
-	int i;
+	struct i915_pagetab *pt;
+	uint32_t pde;
 
-	for (i = 0; i < ppgtt->num_pd_entries; i++) {
-		struct i915_pagetab *pt = ppgtt->pd.page_tables[i];
+	gen6_for_all_pdes(pt, ppgtt, pde) {
 		if (pt != ppgtt->scratch_pt)
-			free_pt_single(ppgtt->pd.page_tables[i], ppgtt->base.dev);
+			free_pt_single(pt, ppgtt->base.dev);
 	}
 
 	/* Consider putting this as part of pd free. */
@@ -1287,7 +1268,6 @@ alloc:
 	if (ppgtt->node.start < dev_priv->gtt.mappable_end)
 		DRM_DEBUG("Forced to use aperture for PDEs\n");
 
-	ppgtt->num_pd_entries = I915_PDES_PER_PD;
 	return 0;
 
 err_out:
@@ -1306,8 +1286,7 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt, bool preallocate_pt)
 	if (!preallocate_pt)
 		return 0;
 
-	ret = alloc_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries,
-			     ppgtt->base.dev);
+	ret = alloc_pt_range(&ppgtt->pd, 0, I915_PDES_PER_PD, ppgtt->base.dev);
 	if (ret) {
 		free_pt_scratch(ppgtt->scratch_pt, ppgtt->base.dev);
 		drm_mm_remove_node(&ppgtt->node);
@@ -1356,7 +1335,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt, bool aliasing)
 	ppgtt->base.insert_entries = gen6_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen6_ppgtt_cleanup;
 	ppgtt->base.start = 0;
-	ppgtt->base.total = ppgtt->num_pd_entries * GEN6_PTES_PER_PT * PAGE_SIZE;
+	ppgtt->base.total = I915_PDES_PER_PD * GEN6_PTES_PER_PT * PAGE_SIZE;
 	ppgtt->debug_dump = gen6_dump_ppgtt;
 
 	ppgtt->pd.pd_offset =
@@ -1599,7 +1578,7 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 		if (i915_is_ggtt(vm))
 			ppgtt = dev_priv->mm.aliasing_ppgtt;
 
-		gen6_map_page_range(dev_priv, &ppgtt->pd, 0, ppgtt->num_pd_entries);
+		gen6_map_page_range(dev_priv, &ppgtt->pd, 0, I915_PDES_PER_PD);
 	}
 
 	i915_gem_chipset_flush(dev);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 3c28365..18a0b68 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -265,8 +265,6 @@ struct i915_hw_ppgtt {
 	struct i915_address_space base;
 	struct kref ref;
 	struct drm_mm_node node;
-	unsigned num_pd_entries;
-	unsigned num_pd_pages; /* gen8+ */
 	union {
 		struct i915_pagedirpo pdp;
 		struct i915_pagedir pd;
@@ -333,6 +331,11 @@ struct i915_gtt {
 	     temp = min(temp, (unsigned)length), \
 	     start += temp, length -= temp)
 
+#define gen6_for_all_pdes(pt, ppgtt, iter)  \
+	for (iter = 0, pt = ppgtt->pd.page_tables[iter];			\
+	     iter < gen6_pde_index(ppgtt->base.total);			\
+	     pt =  ppgtt->pd.page_tables[++iter])
+
 static inline uint32_t i915_pte_index(uint64_t address, uint32_t pde_shift)
 {
 	const uint32_t mask = NUM_PTE(pde_shift) - 1;
-- 
2.0.4

[PATCH 51/68] drm/i915: Extract PPGTT param from pagedir alloc

[Ben Widawsky]

Now that we don't need to trace num_pd_pages, we may as well kill all
need for the PPGTT structure in the alloc_pagedirs. This is very useful
for when we move to 48b addressing, and the PDP isn't the root of the
page table structure.

The param is replaced with drm_device, which is an unavoidable wart
throughout the series. (in other words, not extra flagrant).

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index d8e0a62..731782a 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -593,10 +593,9 @@ unwind_out:
 
 static int gen8_ppgtt_alloc_pagedirs(struct i915_pagedirpo *pdp,
 				     uint64_t start,
-				     uint64_t length)
+				     uint64_t length,
+				     struct drm_device *dev)
 {
-	struct i915_hw_ppgtt *ppgtt =
-		container_of(pdp, struct i915_hw_ppgtt, pdp);
 	struct i915_pagedir *unused;
 	uint64_t temp;
 	uint32_t pdpe;
@@ -606,8 +605,8 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_pagedirpo *pdp,
 
 	gen8_for_each_pdpe(unused, pdp, start, length, temp, pdpe) {
 		BUG_ON(unused);
-		pdp->pagedirs[pdpe] = alloc_pd_single(ppgtt->base.dev);
-		if (IS_ERR(ppgtt->pdp.pagedirs[pdpe]))
+		pdp->pagedirs[pdpe] = alloc_pd_single(dev);
+		if (IS_ERR(pdp->pagedirs[pdpe]))
 			goto unwind_out;
 	}
 
@@ -615,8 +614,7 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_pagedirpo *pdp,
 
 unwind_out:
 	while (pdpe--)
-		free_pd_single(ppgtt->pdp.pagedirs[pdpe],
-			       ppgtt->base.dev);
+		free_pd_single(pdp->pagedirs[pdpe], dev);
 
 	return -ENOMEM;
 }
@@ -630,7 +628,8 @@ static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
 	uint32_t pdpe;
 	int ret;
 
-	ret = gen8_ppgtt_alloc_pagedirs(&ppgtt->pdp, start, length);
+	ret = gen8_ppgtt_alloc_pagedirs(&ppgtt->pdp, start, length,
+					ppgtt->base.dev);
 	if (ret)
 		return ret;
 
-- 
2.0.4

[PATCH 52/68] drm/i915/bdw: Split out mappings

[Ben Widawsky]

When we do dynamic page table allocations for gen8, we'll need to have
more control over how and when we map page tables, similar to gen6.

This patch adds the functionality and calls it at init, which should
have no functional change.

The PDPEs are still a special case for now. We'll need a function for
that in the future as well.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 96 ++++++++++++++++++++-----------------
 1 file changed, 52 insertions(+), 44 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 731782a..02ddac4 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -525,6 +525,36 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	}
 }
 
+static void __gen8_do_map_pt(gen8_ppgtt_pde_t *pde,
+			     struct i915_pagetab *pt,
+			     struct drm_device *dev)
+{
+	gen8_ppgtt_pde_t entry =
+		gen8_pde_encode(dev, pt->daddr, I915_CACHE_LLC);
+	*pde = entry;
+}
+
+/* It's likely we'll map more than one pagetable at a time. This function will
+ * save us unnecessary kmap calls, but do no more functionally than multiple
+ * calls to map_pt. */
+static void gen8_map_pagetable_range(struct i915_pagedir *pd,
+				     uint64_t start,
+				     uint64_t length,
+				     struct drm_device *dev)
+{
+	gen8_ppgtt_pde_t *pagedir = kmap_atomic(pd->page);
+	struct i915_pagetab *pt;
+	uint64_t temp, pde;
+
+	gen8_for_each_pde(pt, pd, start, length, temp, pde)
+		__gen8_do_map_pt(pagedir + pde, pt, dev);
+
+	if (!HAS_LLC(dev))
+		drm_clflush_virt_range(pagedir, PAGE_SIZE);
+
+	kunmap_atomic(pagedir);
+}
+
 static void gen8_teardown_va_range(struct i915_address_space *vm,
 				   uint64_t start, uint64_t length)
 {
@@ -549,8 +579,6 @@ static void gen8_teardown_va_range(struct i915_address_space *vm,
 
 static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
-	trace_i915_va_teardown(&ppgtt->base,
-			       ppgtt->base.start, ppgtt->base.total);
 	gen8_teardown_va_range(&ppgtt->base,
 			       ppgtt->base.start, ppgtt->base.total);
 }
@@ -619,11 +647,14 @@ unwind_out:
 	return -ENOMEM;
 }
 
-static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
-			    uint64_t start,
-			    uint64_t length)
+static int gen8_alloc_va_range(struct i915_address_space *vm,
+			       uint64_t start,
+			       uint64_t length)
 {
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(vm, struct i915_hw_ppgtt, base);
 	struct i915_pagedir *pd;
+	const uint64_t orig_start = start;
 	uint64_t temp;
 	uint32_t pdpe;
 	int ret;
@@ -642,9 +673,8 @@ static int gen8_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt,
 
 	return 0;
 
-	/* TODO: Check this for all cases */
 err_out:
-	gen8_ppgtt_free(ppgtt);
+	gen8_teardown_va_range(vm, orig_start, start);
 	return ret;
 }
 
@@ -654,60 +684,38 @@ err_out:
  * PDP represents 1GB of memory 4 * 512 * 512 * 4096 = 4GB legacy 32b address
  * space.
  *
- * FIXME: split allocation into smaller pieces. For now we only ever do this
- * once, but with full PPGTT, the multiple contiguous allocations will be bad.
- * TODO: Do something with the size parameter
  */
 static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 {
-	const int max_pdp = DIV_ROUND_UP(size, 1 << 30);
-	int i, j, ret;
-
-	if (size % (1<<30))
-		DRM_INFO("Pages will be wasted unless GTT size (%llu) is divisible by 1GB\n", size);
+	struct i915_pagedir *pd;
+	uint64_t temp, start = 0;
+	const uint64_t orig_length = size;
+	uint32_t pdpe;
+	int ret;
 
 	ppgtt->base.start = 0;
 	ppgtt->base.total = size;
+	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
+	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
+	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
+	ppgtt->enable = gen8_ppgtt_enable;
+	ppgtt->switch_mm = gen8_mm_switch;
 
 	ppgtt->scratch_pd = alloc_pt_scratch(ppgtt->base.dev);
 	if (IS_ERR(ppgtt->scratch_pd))
 		return PTR_ERR(ppgtt->scratch_pd);
 
-	/* 1. Do all our allocations for page directories and page tables. */
-	ret = gen8_ppgtt_alloc(ppgtt, ppgtt->base.start, ppgtt->base.total);
+	ret = gen8_alloc_va_range(&ppgtt->base, start, size);
 	if (ret) {
 		free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
 		return ret;
 	}
 
-	/*
-	 * 2. Map all the page directory entires to point to the page tables
-	 * we've allocated.
-	 *
-	 * For now, the PPGTT helper functions all require that the PDEs are
-	 * plugged in correctly. So we do that now/here. For aliasing PPGTT, we
-	 * will never need to touch the PDEs again.
-	 */
-	for (i = 0; i < max_pdp; i++) {
-		struct i915_pagedir *pd = ppgtt->pdp.pagedirs[i];
-		gen8_ppgtt_pde_t *pd_vaddr;
-		pd_vaddr = kmap_atomic(ppgtt->pdp.pagedirs[i]->page);
-		for (j = 0; j < I915_PDES_PER_PD; j++) {
-			struct i915_pagetab *pt = pd->page_tables[j];
-			dma_addr_t addr = pt->daddr;
-			pd_vaddr[j] = gen8_pde_encode(ppgtt->base.dev, addr,
-						      I915_CACHE_LLC);
-		}
-		if (!HAS_LLC(ppgtt->base.dev))
-			drm_clflush_virt_range(pd_vaddr, PAGE_SIZE);
-		kunmap_atomic(pd_vaddr);
-	}
+	start = 0;
+	size = orig_length;
 
-	ppgtt->enable = gen8_ppgtt_enable;
-	ppgtt->switch_mm = gen8_mm_switch;
-	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
-	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
-	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
+	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, size, temp, pdpe)
+		gen8_map_pagetable_range(pd, start, size, ppgtt->base.dev);
 
 	return 0;
 }
-- 
2.0.4

[PATCH 53/68] drm/i915/bdw: begin bitmap tracking

[Ben Widawsky]

Like with gen6/7, we can enable bitmap tracking with all the
preallocations to make sure things actually don't blow up.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 101 +++++++++++++++++++++++++++++++-----
 drivers/gpu/drm/i915/i915_gem_gtt.h |  12 +++++
 2 files changed, 99 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 02ddac4..3e43875 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -345,8 +345,12 @@ err_out:
 
 static void __free_pd_single(struct i915_pagedir *pd, struct drm_device *dev)
 {
+	WARN(!bitmap_empty(pd->used_pdes, I915_PDES_PER_PD),
+	     "Free page directory with %d used pages\n",
+	     bitmap_weight(pd->used_pdes, I915_PDES_PER_PD));
 	i915_dma_unmap_single(pd, dev);
 	__free_page(pd->page);
+	kfree(pd->used_pdes);
 	kfree(pd);
 }
 
@@ -359,26 +363,35 @@ static void __free_pd_single(struct i915_pagedir *pd, struct drm_device *dev)
 static struct i915_pagedir *alloc_pd_single(struct drm_device *dev)
 {
 	struct i915_pagedir *pd;
-	int ret;
+	int ret = -ENOMEM;
 
 	pd = kzalloc(sizeof(*pd), GFP_KERNEL);
 	if (!pd)
 		return ERR_PTR(-ENOMEM);
 
+	pd->used_pdes = kcalloc(BITS_TO_LONGS(I915_PDES_PER_PD),
+				sizeof(*pd->used_pdes), GFP_KERNEL);
+	if (!pd->used_pdes)
+		goto free_pd;
+
 	pd->page = alloc_page(GFP_KERNEL | __GFP_ZERO);
-	if (!pd->page) {
-		kfree(pd);
-		return ERR_PTR(-ENOMEM);
-	}
+	if (!pd->page)
+		goto free_bitmap;
 
 	ret = i915_dma_map_px_single(pd, dev);
-	if (ret) {
-		__free_page(pd->page);
-		kfree(pd);
-		return ERR_PTR(ret);
-	}
+	if (ret)
+		goto free_page;
 
 	return pd;
+
+free_page:
+	__free_page(pd->page);
+free_bitmap:
+	kfree(pd->used_pdes);
+free_pd:
+	kfree(pd);
+
+	return ERR_PTR(ret);
 }
 
 /* Broadwell Page Directory Pointer Descriptors */
@@ -568,12 +581,48 @@ static void gen8_teardown_va_range(struct i915_address_space *vm,
 	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
 		uint64_t pd_len = gen8_clamp_pd(start, length);
 		uint64_t pd_start = start;
+
+		/* Page directories might not be present since the macro rounds
+		 * down, and up.
+		 */
+		if (!pd) {
+			WARN(test_bit(pdpe, ppgtt->pdp.used_pdpes),
+			     "PDPE %d is not allocated, but is reserved (%p)\n",
+			     pdpe, vm);
+			continue;
+		} else {
+			WARN(!test_bit(pdpe, ppgtt->pdp.used_pdpes),
+			     "PDPE %d not reserved, but is allocated (%p)",
+			     pdpe, vm);
+		}
+
 		gen8_for_each_pde(pt, pd, pd_start, pd_len, temp, pde) {
-			free_pt_single(pt, vm->dev);
-			pd->page_tables[pde] = NULL;
+			if (!pt) {
+				WARN(test_bit(pde, pd->used_pdes),
+				     "PDE %d is not allocated, but is reserved (%p)\n",
+				     pde, vm);
+				continue;
+			} else
+				WARN(!test_bit(pde, pd->used_pdes),
+				     "PDE %d not reserved, but is allocated (%p)",
+				     pde, vm);
+
+			bitmap_clear(pt->used_ptes,
+				     gen8_pte_index(pd_start),
+				     gen8_pte_count(pd_start, pd_len));
+
+			if (bitmap_empty(pt->used_ptes, GEN8_PTES_PER_PT)) {
+				free_pt_single(pt, vm->dev);
+				pd->page_tables[pde] = NULL;
+				WARN_ON(!test_and_clear_bit(pde, pd->used_pdes));
+			}
+		}
+
+		if (bitmap_empty(pd->used_pdes, I915_PDES_PER_PD)) {
+			free_pd_single(pd, vm->dev);
+			ppgtt->pdp.pagedirs[pdpe] = NULL;
+			WARN_ON(!test_and_clear_bit(pdpe, ppgtt->pdp.used_pdpes));
 		}
-		free_pd_single(pd, vm->dev);
-		ppgtt->pdp.pagedirs[pdpe] = NULL;
 	}
 }
 
@@ -619,6 +668,7 @@ unwind_out:
 	return -ENOMEM;
 }
 
+/* bitmap of new pagedirs */
 static int gen8_ppgtt_alloc_pagedirs(struct i915_pagedirpo *pdp,
 				     uint64_t start,
 				     uint64_t length,
@@ -634,6 +684,7 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_pagedirpo *pdp,
 	gen8_for_each_pdpe(unused, pdp, start, length, temp, pdpe) {
 		BUG_ON(unused);
 		pdp->pagedirs[pdpe] = alloc_pd_single(dev);
+
 		if (IS_ERR(pdp->pagedirs[pdpe]))
 			goto unwind_out;
 	}
@@ -655,10 +706,12 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 		container_of(vm, struct i915_hw_ppgtt, base);
 	struct i915_pagedir *pd;
 	const uint64_t orig_start = start;
+	const uint64_t orig_length = length;
 	uint64_t temp;
 	uint32_t pdpe;
 	int ret;
 
+	/* Do the allocations first so we can easily bail out */
 	ret = gen8_ppgtt_alloc_pagedirs(&ppgtt->pdp, start, length,
 					ppgtt->base.dev);
 	if (ret)
@@ -671,6 +724,26 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 			goto err_out;
 	}
 
+	/* Now mark everything we've touched as used. This doesn't allow for
+	 * robust error checking, but it makes the code a hell of a lot simpler.
+	 */
+	start = orig_start;
+	length = orig_length;
+
+	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
+		struct i915_pagetab *pt;
+		uint64_t pd_len = gen8_clamp_pd(start, length);
+		uint64_t pd_start = start;
+		uint32_t pde;
+		gen8_for_each_pde(pt, &ppgtt->pd, pd_start, pd_len, temp, pde) {
+			bitmap_set(pd->page_tables[pde]->used_ptes,
+				   gen8_pte_index(start),
+				   gen8_pte_count(start, length));
+			set_bit(pde, pd->used_pdes);
+		}
+		set_bit(pdpe, ppgtt->pdp.used_pdpes);
+	}
+
 	return 0;
 
 err_out:
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 18a0b68..b92b1fb 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -195,11 +195,13 @@ struct i915_pagedir {
 		dma_addr_t daddr;
 	};
 
+	unsigned long *used_pdes;
 	struct i915_pagetab *page_tables[I915_PDES_PER_PD];
 };
 
 struct i915_pagedirpo {
 	/* struct page *page; */
+	DECLARE_BITMAP(used_pdpes, GEN8_LEGACY_PDPES);
 	struct i915_pagedir *pagedirs[GEN8_LEGACY_PDPES];
 };
 
@@ -462,6 +464,16 @@ static inline uint32_t gen8_pml4e_index(uint64_t address)
 	BUG();
 }
 
+static inline size_t gen8_pte_count(uint64_t addr, uint64_t length)
+{
+	return i915_pte_count(addr, length, GEN8_PDE_SHIFT);
+}
+
+static inline size_t gen8_pde_count(uint64_t addr, uint64_t length)
+{
+	return i915_pde_count(addr, length, GEN8_PDE_SHIFT);
+}
+
 int i915_gem_gtt_init(struct drm_device *dev);
 void i915_gem_init_global_gtt(struct drm_device *dev);
 void i915_gem_setup_global_gtt(struct drm_device *dev, unsigned long start,
-- 
2.0.4

[PATCH 54/68] drm/i915/bdw: Dynamic page table allocations

[Ben Widawsky]

This finishes off the dynamic page tables allocations, in the legacy 3
level style that already exists. Most everything has already been setup
to this point, the patch finishes off the enabling by setting the
appropriate function pointers.

Zombie tracking:
This could be a separate patch, but I found it helpful for debugging.
Since we write page tables asynchronously with respect to the GPU using
them, we can't actually free the page tables until we know the GPU won't
use them. With this patch, that is always when the context dies.  It
would be possible to write a reaper to go through zombies and clean them
up when under memory pressure. That exercise is left for the reader.

Scratch unused pages:
The object pages can get freed even if a page table still points to
them.  Like the zombie fix, we need to make sure we don't let our GPU
access arbitrary memory when we've unmapped things.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 371 ++++++++++++++++++++++++++++++------
 drivers/gpu/drm/i915/i915_gem_gtt.h |  16 +-
 2 files changed, 324 insertions(+), 63 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 3e43875..84e139d 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -538,7 +538,7 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	}
 }
 
-static void __gen8_do_map_pt(gen8_ppgtt_pde_t *pde,
+static void __gen8_do_map_pt(gen8_ppgtt_pde_t * const pde,
 			     struct i915_pagetab *pt,
 			     struct drm_device *dev)
 {
@@ -555,7 +555,7 @@ static void gen8_map_pagetable_range(struct i915_pagedir *pd,
 				     uint64_t length,
 				     struct drm_device *dev)
 {
-	gen8_ppgtt_pde_t *pagedir = kmap_atomic(pd->page);
+	gen8_ppgtt_pde_t * const pagedir = kmap_atomic(pd->page);
 	struct i915_pagetab *pt;
 	uint64_t temp, pde;
 
@@ -568,8 +568,9 @@ static void gen8_map_pagetable_range(struct i915_pagedir *pd,
 	kunmap_atomic(pagedir);
 }
 
-static void gen8_teardown_va_range(struct i915_address_space *vm,
-				   uint64_t start, uint64_t length)
+static void __gen8_teardown_va_range(struct i915_address_space *vm,
+				     uint64_t start, uint64_t length,
+				     bool dead)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		        container_of(vm, struct i915_hw_ppgtt, base);
@@ -591,6 +592,13 @@ static void gen8_teardown_va_range(struct i915_address_space *vm,
 			     pdpe, vm);
 			continue;
 		} else {
+			if (dead && pd->zombie) {
+				WARN_ON(test_bit(pdpe, ppgtt->pdp.used_pdpes));
+				free_pd_single(pd, vm->dev);
+				ppgtt->pdp.pagedirs[pdpe] = NULL;
+				continue;
+			}
+
 			WARN(!test_bit(pdpe, ppgtt->pdp.used_pdpes),
 			     "PDPE %d not reserved, but is allocated (%p)",
 			     pdpe, vm);
@@ -602,34 +610,65 @@ static void gen8_teardown_va_range(struct i915_address_space *vm,
 				     "PDE %d is not allocated, but is reserved (%p)\n",
 				     pde, vm);
 				continue;
-			} else
+			} else {
+				if (dead && pt->zombie) {
+					WARN_ON(test_bit(pde, pd->used_pdes));
+					free_pt_single(pt, vm->dev);
+					pd->page_tables[pde] = NULL;
+					continue;
+				}
 				WARN(!test_bit(pde, pd->used_pdes),
 				     "PDE %d not reserved, but is allocated (%p)",
 				     pde, vm);
+			}
 
 			bitmap_clear(pt->used_ptes,
 				     gen8_pte_index(pd_start),
 				     gen8_pte_count(pd_start, pd_len));
 
+
 			if (bitmap_empty(pt->used_ptes, GEN8_PTES_PER_PT)) {
+				WARN_ON(!test_and_clear_bit(pde, pd->used_pdes));
+				if (!dead) {
+					pt->zombie = 1;
+					continue;
+				}
 				free_pt_single(pt, vm->dev);
 				pd->page_tables[pde] = NULL;
-				WARN_ON(!test_and_clear_bit(pde, pd->used_pdes));
+
 			}
 		}
 
+		gen8_ppgtt_clear_range(vm, pd_start, pd_len, true);
+
 		if (bitmap_empty(pd->used_pdes, I915_PDES_PER_PD)) {
+			WARN_ON(!test_and_clear_bit(pdpe, ppgtt->pdp.used_pdpes));
+			if (!dead) {
+				/* We've unmapped a possibly live context. Make
+				 * note of it so we can clean it up later. */
+				pd->zombie = 1;
+				continue;
+			}
 			free_pd_single(pd, vm->dev);
 			ppgtt->pdp.pagedirs[pdpe] = NULL;
-			WARN_ON(!test_and_clear_bit(pdpe, ppgtt->pdp.used_pdpes));
 		}
 	}
 }
 
+static void gen8_teardown_va_range(struct i915_address_space *vm,
+				   uint64_t start, uint64_t length)
+{
+	__gen8_teardown_va_range(vm, start, length, false);
+}
+
 static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
-	gen8_teardown_va_range(&ppgtt->base,
-			       ppgtt->base.start, ppgtt->base.total);
+	trace_i915_va_teardown(&ppgtt->base,
+			       ppgtt->base.start, ppgtt->base.total,
+			       VM_TO_TRACE_NAME(&ppgtt->base));
+	__gen8_teardown_va_range(&ppgtt->base,
+				 ppgtt->base.start, ppgtt->base.total,
+				 true);
 }
 
 static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
@@ -643,58 +682,167 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
 	gen8_ppgtt_free(ppgtt);
 }
 
-static int gen8_ppgtt_alloc_pagetabs(struct i915_pagedir *pd,
+/**
+ * gen8_ppgtt_alloc_pagetabs() - Allocate page tables for VA range.
+ * @ppgtt:	Master ppgtt structure.
+ * @pd:		Page directory for this address range.
+ * @start:	Starting virtual address to begin allocations.
+ * @length	Size of the allocations.
+ * @new_pts:	Bitmap set by function with new allocations. Likely used by the
+ *		caller to free on error.
+ *
+ * Allocate the required number of page tables. Extremely similar to
+ * gen8_ppgtt_alloc_pagedirs(). The main difference is here we are limited by
+ * the page directory boundary (instead of the page directory pointer). That
+ * boundary is 1GB virtual. Therefore, unlike gen8_ppgtt_alloc_pagedirs(), it is
+ * possible, and likely that the caller will need to use multiple calls of this
+ * function to achieve the appropriate allocation.
+ *
+ * Return: 0 if success; negative error code otherwise.
+ */
+static int gen8_ppgtt_alloc_pagetabs(struct i915_hw_ppgtt *ppgtt,
+				     struct i915_pagedir *pd,
 				     uint64_t start,
 				     uint64_t length,
-				     struct drm_device *dev)
+				     unsigned long *new_pts)
 {
-	struct i915_pagetab *unused;
+	struct i915_pagetab *pt;
 	uint64_t temp;
 	uint32_t pde;
 
-	gen8_for_each_pde(unused, pd, start, length, temp, pde) {
-		BUG_ON(unused);
-		pd->page_tables[pde] = alloc_pt_single(dev);
-		if (IS_ERR(pd->page_tables[pde]))
+	gen8_for_each_pde(pt, pd, start, length, temp, pde) {
+		/* Don't reallocate page tables */
+		if (pt) {
+			/* Scratch is never allocated this way */
+			WARN_ON(pt->scratch);
+			/* If there is a zombie, we can reuse it and save time
+			 * on the allocation. If we clear the zombie status and
+			 * the caller somehow fails, we'll probably hit some
+			 * assertions, so it's up to them to fix up the bitmaps.
+			 */
+			continue;
+		}
+
+		pt = alloc_pt_single(ppgtt->base.dev);
+		if (IS_ERR(pt))
 			goto unwind_out;
+
+		pd->page_tables[pde] = pt;
+		set_bit(pde, new_pts);
 	}
 
 	return 0;
 
 unwind_out:
-	while (pde--)
-		free_pt_single(pd->page_tables[pde], dev);
+	for_each_set_bit(pde, new_pts, I915_PDES_PER_PD)
+		free_pt_single(pd->page_tables[pde], ppgtt->base.dev);
 
 	return -ENOMEM;
 }
 
-/* bitmap of new pagedirs */
-static int gen8_ppgtt_alloc_pagedirs(struct i915_pagedirpo *pdp,
+/**
+ * gen8_ppgtt_alloc_pagedirs() - Allocate page directories for VA range.
+ * @ppgtt:	Master ppgtt structure.
+ * @pdp:	Page directory pointer for this address range.
+ * @start:	Starting virtual address to begin allocations.
+ * @length	Size of the allocations.
+ * @new_pds	Bitmap set by function with new allocations. Likely used by the
+ *		caller to free on error.
+ *
+ * Allocate the required number of page directories starting at the pde index of
+ * @start, and ending at the pde index @start + @length. This function will skip
+ * over already allocated page directories within the range, and only allocate
+ * new ones, setting the appropriate pointer within the pdp as well as the
+ * correct position in the bitmap @new_pds.
+ *
+ * The function will only allocate the pages within the range for a give page
+ * directory pointer. In other words, if @start + @length straddles a virtually
+ * addressed PDP boundary (512GB for 4k pages), there will be more allocations
+ * required by the caller, This is not currently possible, and the BUG in the
+ * code will prevent it.
+ *
+ * Return: 0 if success; negative error code otherwise.
+ */
+static int gen8_ppgtt_alloc_pagedirs(struct i915_hw_ppgtt *ppgtt,
+				     struct i915_pagedirpo *pdp,
 				     uint64_t start,
 				     uint64_t length,
-				     struct drm_device *dev)
+				     unsigned long *new_pds)
 {
-	struct i915_pagedir *unused;
+	struct i915_pagedir *pd;
 	uint64_t temp;
 	uint32_t pdpe;
 
+	BUG_ON(!bitmap_empty(new_pds, GEN8_LEGACY_PDPES));
+
 	/* FIXME: PPGTT container_of won't work for 64b */
 	BUG_ON((start + length) > 0x800000000ULL);
 
-	gen8_for_each_pdpe(unused, pdp, start, length, temp, pdpe) {
-		BUG_ON(unused);
-		pdp->pagedirs[pdpe] = alloc_pd_single(dev);
+	gen8_for_each_pdpe(pd, pdp, start, length, temp, pdpe) {
+		if (pd)
+			continue;
 
-		if (IS_ERR(pdp->pagedirs[pdpe]))
+		pd = alloc_pd_single(ppgtt->base.dev);
+		if (IS_ERR(pd))
 			goto unwind_out;
+
+		pdp->pagedirs[pdpe] = pd;
+		set_bit(pdpe, new_pds);
 	}
 
 	return 0;
 
 unwind_out:
-	while (pdpe--)
-		free_pd_single(pdp->pagedirs[pdpe], dev);
+	for_each_set_bit(pdpe, new_pds, GEN8_LEGACY_PDPES)
+		free_pd_single(pdp->pagedirs[pdpe], ppgtt->base.dev);
+
+	return -ENOMEM;
+}
+
+static inline void
+free_gen8_temp_bitmaps(unsigned long *new_pds, unsigned long **new_pts)
+{
+	int i;
+	for (i = 0; i < GEN8_LEGACY_PDPES; i++)
+		kfree(new_pts[i]);
+	kfree(new_pts);
+	kfree(new_pds);
+}
+
+/* Fills in the page directory bitmap, ant the array of page tables bitmap. Both
+ * of these are based on the number of PDPEs in the system.
+ */
+int __must_check alloc_gen8_temp_bitmaps(unsigned long **new_pds,
+					 unsigned long ***new_pts)
+{
+	int i;
+	unsigned long *pds;
+	unsigned long **pts;
+
+	pds = kcalloc(BITS_TO_LONGS(GEN8_LEGACY_PDPES), sizeof(unsigned long), GFP_KERNEL);
+	if (!pds)
+		return -ENOMEM;
+
+	pts = kcalloc(I915_PDES_PER_PD, sizeof(unsigned long *), GFP_KERNEL);
+	if (!pts) {
+		kfree(pds);
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < GEN8_LEGACY_PDPES; i++) {
+		pts[i] = kcalloc(BITS_TO_LONGS(I915_PDES_PER_PD),
+				 sizeof(unsigned long), GFP_KERNEL);
+		if (!pts[i])
+			goto err_out;
+	}
 
+	*new_pds = pds;
+	*new_pts = (unsigned long **)pts;
+
+	return 0;
+
+err_out:
+	free_gen8_temp_bitmaps(pds, pts);
 	return -ENOMEM;
 }
 
@@ -704,6 +852,7 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
+	unsigned long *new_page_dirs, **new_page_tables;
 	struct i915_pagedir *pd;
 	const uint64_t orig_start = start;
 	const uint64_t orig_length = length;
@@ -711,43 +860,103 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 	uint32_t pdpe;
 	int ret;
 
-	/* Do the allocations first so we can easily bail out */
-	ret = gen8_ppgtt_alloc_pagedirs(&ppgtt->pdp, start, length,
-					ppgtt->base.dev);
+#ifndef CONFIG_64BIT
+	/* Disallow 64b address on 32b platforms. Nothing is wrong with doing
+	 * this in hardware, but a lot of the drm code is not prepared to handle
+	 * 64b offset on 32b platforms. */
+	if (start + length > 0x100000000ULL)
+		return -E2BIG;
+#endif
+
+	/* Wrap is never okay since we can only represent 48b, and we don't
+	 * actually use the other side of the canonical address space.
+	 */
+	if (WARN_ON(start + length < start))
+		return -ERANGE;
+
+	ret = alloc_gen8_temp_bitmaps(&new_page_dirs, &new_page_tables);
 	if (ret)
 		return ret;
 
+	/* Do the allocations first so we can easily bail out */
+	ret = gen8_ppgtt_alloc_pagedirs(ppgtt, &ppgtt->pdp, start, length,
+					new_page_dirs);
+	if (ret) {
+		free_gen8_temp_bitmaps(new_page_dirs, new_page_tables);
+		return ret;
+	}
+
+	/* For every page directory referenced, allocate page tables */
 	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
-		ret = gen8_ppgtt_alloc_pagetabs(pd, start, length,
-						ppgtt->base.dev);
+		bitmap_zero(new_page_tables[pdpe], I915_PDES_PER_PD);
+		ret = gen8_ppgtt_alloc_pagetabs(ppgtt, pd, start, length,
+						new_page_tables[pdpe]);
 		if (ret)
 			goto err_out;
 	}
 
-	/* Now mark everything we've touched as used. This doesn't allow for
-	 * robust error checking, but it makes the code a hell of a lot simpler.
-	 */
 	start = orig_start;
 	length = orig_length;
 
+	/* Allocations have completed successfully, so set the bitmaps, and do
+	 * the mappings. */
 	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
+		gen8_ppgtt_pde_t *const pagedir = kmap_atomic(pd->page);
 		struct i915_pagetab *pt;
 		uint64_t pd_len = gen8_clamp_pd(start, length);
 		uint64_t pd_start = start;
 		uint32_t pde;
-		gen8_for_each_pde(pt, &ppgtt->pd, pd_start, pd_len, temp, pde) {
-			bitmap_set(pd->page_tables[pde]->used_ptes,
-				   gen8_pte_index(start),
-				   gen8_pte_count(start, length));
+
+		/* Every pd should be allocated, we just did that above. */
+		BUG_ON(!pd);
+
+		gen8_for_each_pde(pt, pd, pd_start, pd_len, temp, pde) {
+			/* Same reasoning as pd */
+			BUG_ON(!pt);
+			BUG_ON(!pd_len);
+			BUG_ON(!gen8_pte_count(pd_start, pd_len));
+
+			/* Set our used ptes within the page table */
+			bitmap_set(pt->used_ptes,
+				   gen8_pte_index(pd_start),
+				   gen8_pte_count(pd_start, pd_len));
+
+			/* Our pde is now pointing to the pagetable, pt */
 			set_bit(pde, pd->used_pdes);
+
+			/* Map the PDE to the page table */
+			__gen8_do_map_pt(pagedir + pde, pt, vm->dev);
+
+			/* NB: We haven't yet mapped ptes to pages. At this
+			 * point we're still relying on insert_entries() */
+
+			/* No longer possible this page table is a zombie */
+			pt->zombie = 0;
 		}
+
+		if (!HAS_LLC(vm->dev))
+			drm_clflush_virt_range(pagedir, PAGE_SIZE);
+
+		kunmap_atomic(pagedir);
+
 		set_bit(pdpe, ppgtt->pdp.used_pdpes);
+		/* This pd is officially not a zombie either */
+		ppgtt->pdp.pagedirs[pdpe]->zombie = 0;
 	}
 
+	free_gen8_temp_bitmaps(new_page_dirs, new_page_tables);
 	return 0;
 
 err_out:
-	gen8_teardown_va_range(vm, orig_start, start);
+	while (pdpe--) {
+		for_each_set_bit(temp, new_page_tables[pdpe], I915_PDES_PER_PD)
+			free_pt_single(pd->page_tables[temp], vm->dev);
+	}
+
+	for_each_set_bit(pdpe, new_page_dirs, GEN8_LEGACY_PDPES)
+		free_pd_single(ppgtt->pdp.pagedirs[pdpe], vm->dev);
+
+	free_gen8_temp_bitmaps(new_page_dirs, new_page_tables);
 	return ret;
 }
 
@@ -758,38 +967,69 @@ err_out:
  * space.
  *
  */
-static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, uint64_t size)
+static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 {
-	struct i915_pagedir *pd;
-	uint64_t temp, start = 0;
-	const uint64_t orig_length = size;
-	uint32_t pdpe;
-	int ret;
+	ppgtt->scratch_pd = alloc_pt_scratch(ppgtt->base.dev);
+	if (IS_ERR(ppgtt->scratch_pd))
+		return PTR_ERR(ppgtt->scratch_pd);
 
 	ppgtt->base.start = 0;
 	ppgtt->base.total = size;
-	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
-	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
+	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
+
 	ppgtt->enable = gen8_ppgtt_enable;
 	ppgtt->switch_mm = gen8_mm_switch;
 
-	ppgtt->scratch_pd = alloc_pt_scratch(ppgtt->base.dev);
-	if (IS_ERR(ppgtt->scratch_pd))
-		return PTR_ERR(ppgtt->scratch_pd);
+	return 0;
+}
+
+static int gen8_aliasing_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
+{
+	struct drm_device *dev = ppgtt->base.dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_pagedir *pd;
+	uint64_t temp, start = 0, size = dev_priv->gtt.base.total;
+	uint32_t pdpe;
+	int ret;
 
+	ret = gen8_ppgtt_init_common(ppgtt, dev_priv->gtt.base.total);
+	if (ret)
+		return ret;
+
+	/* Aliasing PPGTT has to always work and be mapped because of the way we
+	 * use RESTORE_INHIBIT in the context switch. This will be fixed
+	 * eventually. */
 	ret = gen8_alloc_va_range(&ppgtt->base, start, size);
 	if (ret) {
 		free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
 		return ret;
 	}
 
-	start = 0;
-	size = orig_length;
-
 	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, size, temp, pdpe)
 		gen8_map_pagetable_range(pd, start, size, ppgtt->base.dev);
 
+	ppgtt->base.allocate_va_range = NULL;
+	ppgtt->base.teardown_va_range = NULL;
+	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
+
+	return 0;
+}
+
+static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
+{
+	struct drm_device *dev = ppgtt->base.dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	int ret;
+
+	ret = gen8_ppgtt_init_common(ppgtt, dev_priv->gtt.base.total);
+	if (ret)
+		return ret;
+
+	ppgtt->base.allocate_va_range = gen8_alloc_va_range;
+	ppgtt->base.teardown_va_range = gen8_teardown_va_range;
+	ppgtt->base.clear_range = NULL;
+
 	return 0;
 }
 
@@ -1446,8 +1686,10 @@ int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt, boo
 
 	if (INTEL_INFO(dev)->gen < 8)
 		ret = gen6_ppgtt_init(ppgtt, aliasing);
+	else if (IS_GEN8(dev) && aliasing)
+		ret = gen8_aliasing_ppgtt_init(ppgtt);
 	else if (IS_GEN8(dev))
-		ret = gen8_ppgtt_init(ppgtt, dev_priv->gtt.base.total);
+		ret = gen8_ppgtt_init(ppgtt);
 	else
 		BUG();
 
@@ -1456,7 +1698,8 @@ int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt, boo
 
 	kref_init(&ppgtt->ref);
 	drm_mm_init(&ppgtt->base.mm, ppgtt->base.start, ppgtt->base.total);
-	ppgtt->base.clear_range(&ppgtt->base, 0, ppgtt->base.total, true);
+	if (ppgtt->base.clear_range)
+		ppgtt->base.clear_range(&ppgtt->base, 0, ppgtt->base.total, true);
 	i915_init_vm(dev_priv, &ppgtt->base);
 
 	return 0;
@@ -1502,10 +1745,7 @@ ppgtt_bind_vma(struct i915_vma *vma,
 
 static void ppgtt_unbind_vma(struct i915_vma *vma)
 {
-	vma->vm->clear_range(vma->vm,
-			     vma->node.start,
-			     vma->obj->base.size,
-			     true);
+	WARN_ON(vma->vm->teardown_va_range && vma->vm->clear_range);
 	if (vma->vm->teardown_va_range) {
 		trace_i915_va_teardown(vma->vm,
 				       vma->node.start, vma->node.size,
@@ -1514,7 +1754,14 @@ static void ppgtt_unbind_vma(struct i915_vma *vma)
 		vma->vm->teardown_va_range(vma->vm,
 					   vma->node.start, vma->node.size);
 		ppgtt_invalidate_tlbs(vma->vm);
-	}
+	} else if (vma->vm->clear_range) {
+		vma->vm->clear_range(vma->vm,
+				     vma->node.start,
+				     vma->obj->base.size,
+				     true);
+	} else
+		BUG();
+
 }
 
 extern int intel_iommu_gfx_mapped;
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index b92b1fb..d9759c7 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -179,13 +179,26 @@ struct i915_vma {
 			u32 flags);
 };
 
-
+/* Zombies. We write page tables with the CPU, and hardware switches them with
+ * the GPU. As such, the only time we can safely remove a page table is when we
+ * know the context is idle. Since we have no good way to do this, we use the
+ * zombie.
+ *
+ * Under memory pressure, if the system is idle, zombies may be reaped.
+ *
+ * There are 3 states a page table can be in (not including scratch)
+ *  bitmap = 0, zombie = 0: unallocated
+ *  bitmap = 1, zombie = 0: allocated
+ *  bitmap = 0, zombie = 1: zombie
+ *  bitmap = 1, zombie = 1: invalid
+ */
 struct i915_pagetab {
 	struct page *page;
 	dma_addr_t daddr;
 
 	unsigned long *used_ptes;
 	unsigned int scratch:1;
+	unsigned zombie:1;
 };
 
 struct i915_pagedir {
@@ -197,6 +210,7 @@ struct i915_pagedir {
 
 	unsigned long *used_pdes;
 	struct i915_pagetab *page_tables[I915_PDES_PER_PD];
+	unsigned zombie:1;
 };
 
 struct i915_pagedirpo {
-- 
2.0.4

[PATCH 55/68] drm/i915/bdw: Make pdp allocation more dynamic

[Ben Widawsky]

This transitional patch doesn't do much for the existing code. However,
it should make upcoming patches to use the full 48b address space a bit
easier to swallow. The patch also introduces the PML4, ie. the new top
level structure of the page tables.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     |   5 ++
 drivers/gpu/drm/i915/i915_gem_gtt.c | 114 +++++++++++++++++++++++++++++-------
 drivers/gpu/drm/i915/i915_gem_gtt.h |  40 +++++++++----
 3 files changed, 128 insertions(+), 31 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index beb9a66..ff921e6 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2078,6 +2078,11 @@ struct drm_i915_cmd_table {
 #define HAS_PPGTT(dev)		(INTEL_INFO(dev)->gen >= 7 && !IS_CHERRYVIEW(dev))
 #define USES_PPGTT(dev)		(i915.enable_ppgtt)
 #define USES_FULL_PPGTT(dev)	(i915.enable_ppgtt == 2)
+#ifdef CONFIG_64BIT
+# define HAS_48B_PPGTT(dev)	(IS_BROADWELL(dev) && false)
+#else
+# define HAS_48B_PPGTT(dev)	false
+#endif
 
 #define HAS_OVERLAY(dev)		(INTEL_INFO(dev)->has_overlay)
 #define OVERLAY_NEEDS_PHYSICAL(dev)	(INTEL_INFO(dev)->overlay_needs_physical)
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 84e139d..8e15842 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -394,6 +394,45 @@ free_pd:
 	return ERR_PTR(ret);
 }
 
+static void __pdp_fini(struct i915_pagedirpo *pdp)
+{
+	kfree(pdp->used_pdpes);
+	kfree(pdp->pagedirs);
+	/* HACK */
+	pdp->pagedirs = NULL;
+}
+
+static void free_pdp_single(struct i915_pagedirpo *pdp,
+			    struct drm_device *dev)
+{
+	__pdp_fini(pdp);
+	if (HAS_48B_PPGTT(dev))
+		kfree(pdp);
+}
+
+static int __pdp_init(struct i915_pagedirpo *pdp,
+		      struct drm_device *dev)
+{
+	size_t pdpes = I915_PDPES_PER_PDP(dev);
+
+	pdp->used_pdpes = kcalloc(BITS_TO_LONGS(pdpes),
+				  sizeof(unsigned long),
+				  GFP_KERNEL);
+	if (!pdp->used_pdpes)
+		return -ENOMEM;
+
+	pdp->pagedirs = kcalloc(pdpes, sizeof(*pdp->pagedirs), GFP_KERNEL);
+	if (!pdp->pagedirs) {
+		kfree(pdp->used_pdpes);
+		/* the PDP might be the statically allocated top level. Keep it
+		 * as clean as possible */
+		pdp->used_pdpes = NULL;
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
 /* Broadwell Page Directory Pointer Descriptors */
 static int gen8_write_pdp(struct intel_engine_cs *ring,
 			  unsigned entry,
@@ -432,7 +471,7 @@ static int gen8_mm_switch(struct i915_hw_ppgtt *ppgtt,
 {
 	int i, ret;
 
-	for (i = GEN8_LEGACY_PDPES - 1; i >= 0; i--) {
+	for (i = 3; i >= 0; i--) {
 		struct i915_pagedir *pd = ppgtt->pdp.pagedirs[i];
 		dma_addr_t pd_daddr = pd ? pd->daddr : ppgtt->scratch_pd->daddr;
 		/* The page directory might be NULL, but we need to clear out
@@ -506,9 +545,6 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	pt_vaddr = NULL;
 
 	for_each_sg_page(pages->sgl, &sg_iter, pages->nents, 0) {
-		if (WARN_ON(pdpe >= GEN8_LEGACY_PDPES))
-			break;
-
 		if (pt_vaddr == NULL) {
 			struct i915_pagedir *pd = ppgtt->pdp.pagedirs[pdpe];
 			struct i915_pagetab *pt = pd->page_tables[pde];
@@ -574,11 +610,17 @@ static void __gen8_teardown_va_range(struct i915_address_space *vm,
 {
 	struct i915_hw_ppgtt *ppgtt =
 		        container_of(vm, struct i915_hw_ppgtt, base);
+	struct drm_device *dev = vm->dev;
 	struct i915_pagedir *pd;
 	struct i915_pagetab *pt;
 	uint64_t temp;
 	uint32_t pdpe, pde;
 
+	if (!ppgtt->pdp.pagedirs) {
+		/* If pagedirs are already free, there is nothing to do.*/
+		return;
+	}
+
 	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
 		uint64_t pd_len = gen8_clamp_pd(start, length);
 		uint64_t pd_start = start;
@@ -633,9 +675,8 @@ static void __gen8_teardown_va_range(struct i915_address_space *vm,
 					pt->zombie = 1;
 					continue;
 				}
-				free_pt_single(pt, vm->dev);
+				free_pt_single(pt, dev);
 				pd->page_tables[pde] = NULL;
-
 			}
 		}
 
@@ -649,10 +690,15 @@ static void __gen8_teardown_va_range(struct i915_address_space *vm,
 				pd->zombie = 1;
 				continue;
 			}
-			free_pd_single(pd, vm->dev);
+			free_pd_single(pd, dev);
 			ppgtt->pdp.pagedirs[pdpe] = NULL;
 		}
 	}
+
+	if (bitmap_empty(ppgtt->pdp.used_pdpes, I915_PDPES_PER_PDP(dev))) {
+		/* TODO: When pagetables are fully dynamic:
+		free_pdp_single(&ppgtt->pdp, dev); */
+	}
 }
 
 static void gen8_teardown_va_range(struct i915_address_space *vm,
@@ -669,6 +715,9 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	__gen8_teardown_va_range(&ppgtt->base,
 				 ppgtt->base.start, ppgtt->base.total,
 				 true);
+	WARN_ON(!bitmap_empty(ppgtt->pdp.used_pdpes,
+			      I915_PDPES_PER_PDP(ppgtt->base.dev)));
+	free_pdp_single(&ppgtt->pdp, ppgtt->base.dev);
 }
 
 static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
@@ -769,11 +818,13 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_hw_ppgtt *ppgtt,
 				     uint64_t length,
 				     unsigned long *new_pds)
 {
+	struct drm_device *dev = ppgtt->base.dev;
 	struct i915_pagedir *pd;
 	uint64_t temp;
 	uint32_t pdpe;
+	size_t pdpes =  I915_PDPES_PER_PDP(ppgtt->base.dev);
 
-	BUG_ON(!bitmap_empty(new_pds, GEN8_LEGACY_PDPES));
+	BUG_ON(!bitmap_empty(new_pds, pdpes));
 
 	/* FIXME: PPGTT container_of won't work for 64b */
 	BUG_ON((start + length) > 0x800000000ULL);
@@ -793,17 +844,18 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_hw_ppgtt *ppgtt,
 	return 0;
 
 unwind_out:
-	for_each_set_bit(pdpe, new_pds, GEN8_LEGACY_PDPES)
+	for_each_set_bit(pdpe, new_pds, pdpes)
 		free_pd_single(pdp->pagedirs[pdpe], ppgtt->base.dev);
 
 	return -ENOMEM;
 }
 
 static inline void
-free_gen8_temp_bitmaps(unsigned long *new_pds, unsigned long **new_pts)
+free_gen8_temp_bitmaps(unsigned long *new_pds, unsigned long **new_pts,
+		       size_t pdpes)
 {
 	int i;
-	for (i = 0; i < GEN8_LEGACY_PDPES; i++)
+	for (i = 0; i < pdpes; i++)
 		kfree(new_pts[i]);
 	kfree(new_pts);
 	kfree(new_pds);
@@ -813,13 +865,14 @@ free_gen8_temp_bitmaps(unsigned long *new_pds, unsigned long **new_pts)
  * of these are based on the number of PDPEs in the system.
  */
 int __must_check alloc_gen8_temp_bitmaps(unsigned long **new_pds,
-					 unsigned long ***new_pts)
+					 unsigned long ***new_pts,
+					 size_t pdpes)
 {
 	int i;
 	unsigned long *pds;
 	unsigned long **pts;
 
-	pds = kcalloc(BITS_TO_LONGS(GEN8_LEGACY_PDPES), sizeof(unsigned long), GFP_KERNEL);
+	pds = kcalloc(BITS_TO_LONGS(pdpes), sizeof(unsigned long), GFP_KERNEL);
 	if (!pds)
 		return -ENOMEM;
 
@@ -829,7 +882,7 @@ int __must_check alloc_gen8_temp_bitmaps(unsigned long **new_pds,
 		return -ENOMEM;
 	}
 
-	for (i = 0; i < GEN8_LEGACY_PDPES; i++) {
+	for (i = 0; i < pdpes; i++) {
 		pts[i] = kcalloc(BITS_TO_LONGS(I915_PDES_PER_PD),
 				 sizeof(unsigned long), GFP_KERNEL);
 		if (!pts[i])
@@ -842,7 +895,7 @@ int __must_check alloc_gen8_temp_bitmaps(unsigned long **new_pds,
 	return 0;
 
 err_out:
-	free_gen8_temp_bitmaps(pds, pts);
+	free_gen8_temp_bitmaps(pds, pts, pdpes);
 	return -ENOMEM;
 }
 
@@ -853,11 +906,13 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 	unsigned long *new_page_dirs, **new_page_tables;
+	struct drm_device *dev = vm->dev;
 	struct i915_pagedir *pd;
 	const uint64_t orig_start = start;
 	const uint64_t orig_length = length;
 	uint64_t temp;
 	uint32_t pdpe;
+	size_t pdpes = I915_PDPES_PER_PDP(dev);
 	int ret;
 
 #ifndef CONFIG_64BIT
@@ -874,7 +929,7 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 	if (WARN_ON(start + length < start))
 		return -ERANGE;
 
-	ret = alloc_gen8_temp_bitmaps(&new_page_dirs, &new_page_tables);
+	ret = alloc_gen8_temp_bitmaps(&new_page_dirs, &new_page_tables, pdpes);
 	if (ret)
 		return ret;
 
@@ -882,7 +937,7 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 	ret = gen8_ppgtt_alloc_pagedirs(ppgtt, &ppgtt->pdp, start, length,
 					new_page_dirs);
 	if (ret) {
-		free_gen8_temp_bitmaps(new_page_dirs, new_page_tables);
+		free_gen8_temp_bitmaps(new_page_dirs, new_page_tables, pdpes);
 		return ret;
 	}
 
@@ -944,7 +999,7 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 		ppgtt->pdp.pagedirs[pdpe]->zombie = 0;
 	}
 
-	free_gen8_temp_bitmaps(new_page_dirs, new_page_tables);
+	free_gen8_temp_bitmaps(new_page_dirs, new_page_tables, pdpes);
 	return 0;
 
 err_out:
@@ -953,13 +1008,19 @@ err_out:
 			free_pt_single(pd->page_tables[temp], vm->dev);
 	}
 
-	for_each_set_bit(pdpe, new_page_dirs, GEN8_LEGACY_PDPES)
+	for_each_set_bit(pdpe, new_page_dirs, pdpes)
 		free_pd_single(ppgtt->pdp.pagedirs[pdpe], vm->dev);
 
-	free_gen8_temp_bitmaps(new_page_dirs, new_page_tables);
+	free_gen8_temp_bitmaps(new_page_dirs, new_page_tables, pdpes);
 	return ret;
 }
 
+static void gen8_ppgtt_fini_common(struct i915_hw_ppgtt *ppgtt)
+{
+	free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
+	free_pdp_single(&ppgtt->pdp, ppgtt->base.dev);
+}
+
 /**
  * GEN8 legacy ppgtt programming is accomplished through a max 4 PDP registers
  * with a net effect resembling a 2-level page table in normal x86 terms. Each
@@ -981,6 +1042,17 @@ static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->enable = gen8_ppgtt_enable;
 	ppgtt->switch_mm = gen8_mm_switch;
 
+	if (!HAS_48B_PPGTT(ppgtt->base.dev)) {
+		int ret = __pdp_init(&ppgtt->pdp, false);
+		if (ret) {
+			free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
+			return ret;
+		}
+
+		ppgtt->switch_mm = gen8_mm_switch;
+	} else
+		BUG(); /* Not yet implemented */
+
 	return 0;
 }
 
@@ -1002,7 +1074,7 @@ static int gen8_aliasing_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	 * eventually. */
 	ret = gen8_alloc_va_range(&ppgtt->base, start, size);
 	if (ret) {
-		free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
+		gen8_ppgtt_fini_common(ppgtt);
 		return ret;
 	}
 
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index d9759c7..95b5d16 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -88,7 +88,6 @@ typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
 #define PPAT_CACHED_INDEX		_PAGE_PAT /* WB LLCeLLC */
 #define PPAT_DISPLAY_ELLC_INDEX		_PAGE_PCD /* WT eLLC */
 
-#define GEN8_LEGACY_PDPES		4
 #define GEN8_PTES_PER_PT		(PAGE_SIZE / sizeof(gen8_gtt_pte_t))
 
 /* GEN8 legacy style address is defined as a 3 level page table:
@@ -97,8 +96,17 @@ typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
  * The difference as compared to normal x86 3 level page table is the PDPEs are
  * programmed via register.
  */
+#ifdef CONFIG_64BIT
+# define I915_PDPES_PER_PDP(dev) (HAS_48B_PPGTT(dev) ? 512 : 4)
+#else
+# define I915_PDPES_PER_PDP		4
+#endif
+#define GEN8_PML4ES_PER_PML4		512
+#define GEN8_PML4E_SHIFT		39
 #define GEN8_PDPE_SHIFT			30
-#define GEN8_PDPE_MASK			0x3
+/* NB: GEN8_PDPE_MASK is untrue for 32b platforms, but it has no impact on 32b page
+ * tables */
+#define GEN8_PDPE_MASK			0x1ff
 #define GEN8_PDE_SHIFT			21
 
 #define PPAT_UNCACHED_INDEX		(_PAGE_PWT | _PAGE_PCD)
@@ -214,9 +222,17 @@ struct i915_pagedir {
 };
 
 struct i915_pagedirpo {
-	/* struct page *page; */
-	DECLARE_BITMAP(used_pdpes, GEN8_LEGACY_PDPES);
-	struct i915_pagedir *pagedirs[GEN8_LEGACY_PDPES];
+	struct page *page;
+	dma_addr_t daddr;
+	unsigned long *used_pdpes;
+	struct i915_pagedir **pagedirs;
+};
+
+struct i915_pml4 {
+	struct page *page;
+	dma_addr_t daddr;
+	DECLARE_BITMAP(used_pml4es, GEN8_PML4ES_PER_PML4);
+	struct i915_pagedirpo *pdps[GEN8_PML4ES_PER_PML4];
 };
 
 struct i915_address_space {
@@ -282,8 +298,9 @@ struct i915_hw_ppgtt {
 	struct kref ref;
 	struct drm_mm_node node;
 	union {
-		struct i915_pagedirpo pdp;
-		struct i915_pagedir pd;
+		struct i915_pml4 pml4;		/* GEN8+ & 64b PPGTT */
+		struct i915_pagedirpo pdp;	/* GEN8+ */
+		struct i915_pagedir pd;		/* GEN6-7 */
 	};
 
 	union {
@@ -430,14 +447,17 @@ static inline size_t gen6_pde_count(uint32_t addr, uint32_t length)
 	     temp = min(temp, length),					\
 	     start += temp, length -= temp)
 
-#define gen8_for_each_pdpe(pd, pdp, start, length, temp, iter)		\
-	for (iter = gen8_pdpe_index(start), pd = (pdp)->pagedirs[iter];	\
-	     length > 0 && iter < GEN8_LEGACY_PDPES;			\
+#define gen8_for_each_pdpe_e(pd, pdp, start, length, temp, iter, b)	\
+	for (iter = gen8_pdpe_index(start), pd = (pdp)->pagedirs[iter]; \
+	     length > 0 && (iter < b);					\
 	     pd = (pdp)->pagedirs[++iter],				\
 	     temp = ALIGN(start+1, 1 << GEN8_PDPE_SHIFT) - start,	\
 	     temp = min(temp, length),					\
 	     start += temp, length -= temp)
 
+#define gen8_for_each_pdpe(pd, pdp, start, length, temp, iter)		\
+	gen8_for_each_pdpe_e(pd, pdp, start, length, temp, iter, I915_PDPES_PER_PDP(dev))
+
 /* Clamp length to the next pagetab boundary */
 static inline uint64_t gen8_clamp_pt(uint64_t start, uint64_t length)
 {
-- 
2.0.4

[PATCH 56/68] drm/i915/bdw: Abstract PDP usage

[Ben Widawsky]

Up until now, ppgtt->pdp has always been the root of our page tables.
Legacy 32b addresses acted like it had 1 PDP with 4 PDPEs.

In preparation for 4 level page tables, we need to stop use ppgtt->pdp
directly unless we know it's what we want. The future structure will use
ppgtt->pml4 for the top level, and the pdp is just one of the entries
being pointed to by a pml4e.

This patch addresses some carelessness done throughout development wrt
assumptions made of the root page tables.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 160 ++++++++++++++++++++----------------
 1 file changed, 88 insertions(+), 72 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 8e15842..7cc6cf9 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -491,6 +491,7 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
+	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
 	gen8_gtt_pte_t *pt_vaddr, scratch_pte;
 	unsigned pdpe = gen8_pdpe_index(start);
 	unsigned pde = gen8_pde_index(start);
@@ -502,7 +503,7 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 				      I915_CACHE_LLC, use_scratch);
 
 	while (num_entries) {
-		struct i915_pagedir *pd = ppgtt->pdp.pagedirs[pdpe];
+		struct i915_pagedir *pd = pdp->pagedirs[pdpe];
 		struct i915_pagetab *pt = pd->page_tables[pde];
 		struct page *page_table = pt->page;
 
@@ -536,6 +537,7 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
+	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
 	gen8_gtt_pte_t *pt_vaddr;
 	unsigned pdpe = gen8_pdpe_index(start);
 	unsigned pde = gen8_pde_index(start);
@@ -546,7 +548,7 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 
 	for_each_sg_page(pages->sgl, &sg_iter, pages->nents, 0) {
 		if (pt_vaddr == NULL) {
-			struct i915_pagedir *pd = ppgtt->pdp.pagedirs[pdpe];
+			struct i915_pagedir *pd = pdp->pagedirs[pdpe];
 			struct i915_pagetab *pt = pd->page_tables[pde];
 			struct page *page_table = pt->page;
 			pt_vaddr = kmap_atomic(page_table);
@@ -604,24 +606,26 @@ static void gen8_map_pagetable_range(struct i915_pagedir *pd,
 	kunmap_atomic(pagedir);
 }
 
-static void __gen8_teardown_va_range(struct i915_address_space *vm,
-				     uint64_t start, uint64_t length,
-				     bool dead)
+static void gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
+					struct i915_pagedirpo *pdp,
+					uint64_t start, uint64_t length,
+					bool dead)
 {
-	struct i915_hw_ppgtt *ppgtt =
-		        container_of(vm, struct i915_hw_ppgtt, base);
 	struct drm_device *dev = vm->dev;
 	struct i915_pagedir *pd;
 	struct i915_pagetab *pt;
 	uint64_t temp;
 	uint32_t pdpe, pde;
 
-	if (!ppgtt->pdp.pagedirs) {
+	BUG_ON(!pdp);
+	if (!pdp->pagedirs) {
+		WARN(!bitmap_empty(pdp->used_pdpes, I915_PDPES_PER_PDP(dev)),
+		     "Page directory leak detected\n");
 		/* If pagedirs are already free, there is nothing to do.*/
 		return;
 	}
 
-	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
+	gen8_for_each_pdpe(pd, pdp, start, length, temp, pdpe) {
 		uint64_t pd_len = gen8_clamp_pd(start, length);
 		uint64_t pd_start = start;
 
@@ -629,19 +633,19 @@ static void __gen8_teardown_va_range(struct i915_address_space *vm,
 		 * down, and up.
 		 */
 		if (!pd) {
-			WARN(test_bit(pdpe, ppgtt->pdp.used_pdpes),
+			WARN(test_bit(pdpe, pdp->used_pdpes),
 			     "PDPE %d is not allocated, but is reserved (%p)\n",
 			     pdpe, vm);
 			continue;
 		} else {
 			if (dead && pd->zombie) {
-				WARN_ON(test_bit(pdpe, ppgtt->pdp.used_pdpes));
+				WARN_ON(test_bit(pdpe, pdp->used_pdpes));
 				free_pd_single(pd, vm->dev);
-				ppgtt->pdp.pagedirs[pdpe] = NULL;
+				pdp->pagedirs[pdpe] = NULL;
 				continue;
 			}
 
-			WARN(!test_bit(pdpe, ppgtt->pdp.used_pdpes),
+			WARN(!test_bit(pdpe, pdp->used_pdpes),
 			     "PDPE %d not reserved, but is allocated (%p)",
 			     pdpe, vm);
 		}
@@ -683,7 +687,7 @@ static void __gen8_teardown_va_range(struct i915_address_space *vm,
 		gen8_ppgtt_clear_range(vm, pd_start, pd_len, true);
 
 		if (bitmap_empty(pd->used_pdes, I915_PDES_PER_PD)) {
-			WARN_ON(!test_and_clear_bit(pdpe, ppgtt->pdp.used_pdpes));
+			WARN_ON(!test_and_clear_bit(pdpe, pdp->used_pdpes));
 			if (!dead) {
 				/* We've unmapped a possibly live context. Make
 				 * note of it so we can clean it up later. */
@@ -691,20 +695,32 @@ static void __gen8_teardown_va_range(struct i915_address_space *vm,
 				continue;
 			}
 			free_pd_single(pd, dev);
-			ppgtt->pdp.pagedirs[pdpe] = NULL;
+			pdp->pagedirs[pdpe] = NULL;
 		}
 	}
 
-	if (bitmap_empty(ppgtt->pdp.used_pdpes, I915_PDPES_PER_PDP(dev))) {
-		/* TODO: When pagetables are fully dynamic:
-		free_pdp_single(&ppgtt->pdp, dev); */
-	}
+	if (dead && bitmap_empty(pdp->used_pdpes, I915_PDPES_PER_PDP(dev)))
+		free_pdp_single(pdp, dev);
+}
+
+static void gen8_teardown_va_range_4lvl(struct i915_address_space *vm,
+					struct i915_pml4 *pml4,
+					uint64_t start, uint64_t length,
+					bool dead)
+{
+	BUG();
 }
 
 static void gen8_teardown_va_range(struct i915_address_space *vm,
 				   uint64_t start, uint64_t length)
 {
-	__gen8_teardown_va_range(vm, start, length, false);
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(vm, struct i915_hw_ppgtt, base);
+
+	if (!HAS_48B_PPGTT(vm->dev))
+		gen8_teardown_va_range_3lvl(vm, &ppgtt->pdp, start, length, false);
+	else
+		gen8_teardown_va_range_4lvl(vm, &ppgtt->pml4, start, length, false);
 }
 
 static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
@@ -712,12 +728,10 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	trace_i915_va_teardown(&ppgtt->base,
 			       ppgtt->base.start, ppgtt->base.total,
 			       VM_TO_TRACE_NAME(&ppgtt->base));
-	__gen8_teardown_va_range(&ppgtt->base,
-				 ppgtt->base.start, ppgtt->base.total,
-				 true);
-	WARN_ON(!bitmap_empty(ppgtt->pdp.used_pdpes,
-			      I915_PDPES_PER_PDP(ppgtt->base.dev)));
-	free_pdp_single(&ppgtt->pdp, ppgtt->base.dev);
+	gen8_teardown_va_range_3lvl(&ppgtt->base, &ppgtt->pdp,
+				    ppgtt->base.start, ppgtt->base.total,
+				    true);
+	BUG_ON(ppgtt->pdp.pagedirs); /* FIXME: 48b */
 }
 
 static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
@@ -733,7 +747,7 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
 
 /**
  * gen8_ppgtt_alloc_pagetabs() - Allocate page tables for VA range.
- * @ppgtt:	Master ppgtt structure.
+ * @vm:		Master vm structure.
  * @pd:		Page directory for this address range.
  * @start:	Starting virtual address to begin allocations.
  * @length	Size of the allocations.
@@ -749,12 +763,13 @@ static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
  *
  * Return: 0 if success; negative error code otherwise.
  */
-static int gen8_ppgtt_alloc_pagetabs(struct i915_hw_ppgtt *ppgtt,
+static int gen8_ppgtt_alloc_pagetabs(struct i915_address_space *vm,
 				     struct i915_pagedir *pd,
 				     uint64_t start,
 				     uint64_t length,
 				     unsigned long *new_pts)
 {
+	struct drm_device *dev = vm->dev;
 	struct i915_pagetab *pt;
 	uint64_t temp;
 	uint32_t pde;
@@ -772,7 +787,7 @@ static int gen8_ppgtt_alloc_pagetabs(struct i915_hw_ppgtt *ppgtt,
 			continue;
 		}
 
-		pt = alloc_pt_single(ppgtt->base.dev);
+		pt = alloc_pt_single(dev);
 		if (IS_ERR(pt))
 			goto unwind_out;
 
@@ -784,14 +799,14 @@ static int gen8_ppgtt_alloc_pagetabs(struct i915_hw_ppgtt *ppgtt,
 
 unwind_out:
 	for_each_set_bit(pde, new_pts, I915_PDES_PER_PD)
-		free_pt_single(pd->page_tables[pde], ppgtt->base.dev);
+		free_pt_single(pd->page_tables[pde], dev);
 
 	return -ENOMEM;
 }
 
 /**
  * gen8_ppgtt_alloc_pagedirs() - Allocate page directories for VA range.
- * @ppgtt:	Master ppgtt structure.
+ * @vm:		Master vm structure.
  * @pdp:	Page directory pointer for this address range.
  * @start:	Starting virtual address to begin allocations.
  * @length	Size of the allocations.
@@ -812,17 +827,17 @@ unwind_out:
  *
  * Return: 0 if success; negative error code otherwise.
  */
-static int gen8_ppgtt_alloc_pagedirs(struct i915_hw_ppgtt *ppgtt,
+static int gen8_ppgtt_alloc_pagedirs(struct i915_address_space *vm,
 				     struct i915_pagedirpo *pdp,
 				     uint64_t start,
 				     uint64_t length,
 				     unsigned long *new_pds)
 {
-	struct drm_device *dev = ppgtt->base.dev;
+	struct drm_device *dev = vm->dev;
 	struct i915_pagedir *pd;
 	uint64_t temp;
 	uint32_t pdpe;
-	size_t pdpes =  I915_PDPES_PER_PDP(ppgtt->base.dev);
+	size_t pdpes =  I915_PDPES_PER_PDP(vm->dev);
 
 	BUG_ON(!bitmap_empty(new_pds, pdpes));
 
@@ -833,7 +848,7 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_hw_ppgtt *ppgtt,
 		if (pd)
 			continue;
 
-		pd = alloc_pd_single(ppgtt->base.dev);
+		pd = alloc_pd_single(dev);
 		if (IS_ERR(pd))
 			goto unwind_out;
 
@@ -845,7 +860,7 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_hw_ppgtt *ppgtt,
 
 unwind_out:
 	for_each_set_bit(pdpe, new_pds, pdpes)
-		free_pd_single(pdp->pagedirs[pdpe], ppgtt->base.dev);
+		free_pd_single(pdp->pagedirs[pdpe], dev);
 
 	return -ENOMEM;
 }
@@ -899,12 +914,11 @@ err_out:
 	return -ENOMEM;
 }
 
-static int gen8_alloc_va_range(struct i915_address_space *vm,
-			       uint64_t start,
-			       uint64_t length)
+static int gen8_alloc_va_range_3lvl(struct i915_address_space *vm,
+				    struct i915_pagedirpo *pdp,
+				    uint64_t start,
+				    uint64_t length)
 {
-	struct i915_hw_ppgtt *ppgtt =
-		container_of(vm, struct i915_hw_ppgtt, base);
 	unsigned long *new_page_dirs, **new_page_tables;
 	struct drm_device *dev = vm->dev;
 	struct i915_pagedir *pd;
@@ -934,18 +948,15 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 		return ret;
 
 	/* Do the allocations first so we can easily bail out */
-	ret = gen8_ppgtt_alloc_pagedirs(ppgtt, &ppgtt->pdp, start, length,
-					new_page_dirs);
+	ret = gen8_ppgtt_alloc_pagedirs(vm, pdp, start, length, new_page_dirs);
 	if (ret) {
 		free_gen8_temp_bitmaps(new_page_dirs, new_page_tables, pdpes);
 		return ret;
 	}
 
-	/* For every page directory referenced, allocate page tables */
-	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
+	gen8_for_each_pdpe(pd, pdp, start, length, temp, pdpe) {
 		bitmap_zero(new_page_tables[pdpe], I915_PDES_PER_PD);
-		ret = gen8_ppgtt_alloc_pagetabs(ppgtt, pd, start, length,
-						new_page_tables[pdpe]);
+		ret = gen8_ppgtt_alloc_pagetabs(vm, pd, start, length, new_page_tables[pdpe]);
 		if (ret)
 			goto err_out;
 	}
@@ -953,10 +964,7 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 	start = orig_start;
 	length = orig_length;
 
-	/* Allocations have completed successfully, so set the bitmaps, and do
-	 * the mappings. */
-	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
-		gen8_ppgtt_pde_t *const pagedir = kmap_atomic(pd->page);
+	gen8_for_each_pdpe(pd, pdp, start, length, temp, pdpe) {
 		struct i915_pagetab *pt;
 		uint64_t pd_len = gen8_clamp_pd(start, length);
 		uint64_t pd_start = start;
@@ -978,25 +986,12 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 
 			/* Our pde is now pointing to the pagetable, pt */
 			set_bit(pde, pd->used_pdes);
-
-			/* Map the PDE to the page table */
-			__gen8_do_map_pt(pagedir + pde, pt, vm->dev);
-
-			/* NB: We haven't yet mapped ptes to pages. At this
-			 * point we're still relying on insert_entries() */
-
-			/* No longer possible this page table is a zombie */
 			pt->zombie = 0;
 		}
 
-		if (!HAS_LLC(vm->dev))
-			drm_clflush_virt_range(pagedir, PAGE_SIZE);
-
-		kunmap_atomic(pagedir);
-
-		set_bit(pdpe, ppgtt->pdp.used_pdpes);
-		/* This pd is officially not a zombie either */
-		ppgtt->pdp.pagedirs[pdpe]->zombie = 0;
+		set_bit(pdpe, pdp->used_pdpes);
+		gen8_map_pagetable_range(pd, start, length, dev);
+		pd->zombie = 0;
 	}
 
 	free_gen8_temp_bitmaps(new_page_dirs, new_page_tables, pdpes);
@@ -1005,16 +1000,36 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 err_out:
 	while (pdpe--) {
 		for_each_set_bit(temp, new_page_tables[pdpe], I915_PDES_PER_PD)
-			free_pt_single(pd->page_tables[temp], vm->dev);
+			free_pt_single(pd->page_tables[temp], dev);
 	}
 
 	for_each_set_bit(pdpe, new_page_dirs, pdpes)
-		free_pd_single(ppgtt->pdp.pagedirs[pdpe], vm->dev);
+		free_pd_single(pdp->pagedirs[pdpe], dev);
 
 	free_gen8_temp_bitmaps(new_page_dirs, new_page_tables, pdpes);
 	return ret;
 }
 
+static int __noreturn gen8_alloc_va_range_4lvl(struct i915_address_space *vm,
+					       struct i915_pml4 *pml4,
+					       uint64_t start,
+					       uint64_t length)
+{
+	BUG();
+}
+
+static int gen8_alloc_va_range(struct i915_address_space *vm,
+			       uint64_t start, uint64_t length)
+{
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(vm, struct i915_hw_ppgtt, base);
+
+	if (!HAS_48B_PPGTT(vm->dev))
+		return gen8_alloc_va_range_3lvl(vm, &ppgtt->pdp, start, length);
+	else
+		return gen8_alloc_va_range_4lvl(vm, &ppgtt->pml4, start, length);
+}
+
 static void gen8_ppgtt_fini_common(struct i915_hw_ppgtt *ppgtt)
 {
 	free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
@@ -1060,12 +1075,13 @@ static int gen8_aliasing_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 {
 	struct drm_device *dev = ppgtt->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
 	struct i915_pagedir *pd;
 	uint64_t temp, start = 0, size = dev_priv->gtt.base.total;
 	uint32_t pdpe;
 	int ret;
 
-	ret = gen8_ppgtt_init_common(ppgtt, dev_priv->gtt.base.total);
+	ret = gen8_ppgtt_init_common(ppgtt, size);
 	if (ret)
 		return ret;
 
@@ -1078,8 +1094,8 @@ static int gen8_aliasing_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 		return ret;
 	}
 
-	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, size, temp, pdpe)
-		gen8_map_pagetable_range(pd, start, size, ppgtt->base.dev);
+	gen8_for_each_pdpe(pd, pdp, start, size, temp, pdpe)
+		gen8_map_pagetable_range(pd, start, size, dev);
 
 	ppgtt->base.allocate_va_range = NULL;
 	ppgtt->base.teardown_va_range = NULL;
-- 
2.0.4

[PATCH 57/68] drm/i915/bdw: Add dynamic page trace events

[Ben Widawsky]

This works the same as GEN6.

I was disappointed that I need to pass vm around now, but it's not so
much uglier than the drm_device, and having the vm in trace events is
hugely important.

QUESTION: Now that I've rebased this on the zombie change, we probably want
to call it teardown and track unmaps as opposed to destruction.

v2: Consolidate pagetable/pagedirectory events

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 35 +++++++++++++++++++++++++++--------
 drivers/gpu/drm/i915/i915_trace.h   | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 7cc6cf9..679fe0d 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -588,19 +588,24 @@ static void __gen8_do_map_pt(gen8_ppgtt_pde_t * const pde,
 /* It's likely we'll map more than one pagetable at a time. This function will
  * save us unnecessary kmap calls, but do no more functionally than multiple
  * calls to map_pt. */
-static void gen8_map_pagetable_range(struct i915_pagedir *pd,
+static void gen8_map_pagetable_range(struct i915_address_space *vm,
+				     struct i915_pagedir *pd,
 				     uint64_t start,
-				     uint64_t length,
-				     struct drm_device *dev)
+				     uint64_t length)
 {
 	gen8_ppgtt_pde_t * const pagedir = kmap_atomic(pd->page);
 	struct i915_pagetab *pt;
 	uint64_t temp, pde;
 
-	gen8_for_each_pde(pt, pd, start, length, temp, pde)
-		__gen8_do_map_pt(pagedir + pde, pt, dev);
+	gen8_for_each_pde(pt, pd, start, length, temp, pde) {
+		__gen8_do_map_pt(pagedir + pde, pt, vm->dev);
+		trace_i915_pagetable_map(vm, pde, pt,
+					 gen8_pte_index(start),
+					 gen8_pte_count(start, length),
+					 GEN8_PTES_PER_PT);
+	}
 
-	if (!HAS_LLC(dev))
+	if (!HAS_LLC(vm->dev))
 		drm_clflush_virt_range(pagedir, PAGE_SIZE);
 
 	kunmap_atomic(pagedir);
@@ -668,6 +673,11 @@ static void gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 				     pde, vm);
 			}
 
+			trace_i915_pagetable_unmap(vm, pde, pt,
+						   gen8_pte_index(pd_start),
+						   gen8_pte_count(pd_start, pd_len),
+						   GEN8_PTES_PER_PT);
+
 			bitmap_clear(pt->used_ptes,
 				     gen8_pte_index(pd_start),
 				     gen8_pte_count(pd_start, pd_len));
@@ -680,6 +690,10 @@ static void gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 					continue;
 				}
 				free_pt_single(pt, dev);
+				trace_i915_pagetable_destroy(vm,
+							     pde,
+							     pd_start & GENMASK_ULL(64, GEN8_PDE_SHIFT),
+							     GEN8_PDE_SHIFT);
 				pd->page_tables[pde] = NULL;
 			}
 		}
@@ -696,6 +710,9 @@ static void gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 			}
 			free_pd_single(pd, dev);
 			pdp->pagedirs[pdpe] = NULL;
+			trace_i915_pagedirectory_destroy(vm, pdpe,
+							 start & GENMASK_ULL(64, GEN8_PDPE_SHIFT),
+							 GEN8_PDPE_SHIFT);
 		}
 	}
 
@@ -793,6 +810,7 @@ static int gen8_ppgtt_alloc_pagetabs(struct i915_address_space *vm,
 
 		pd->page_tables[pde] = pt;
 		set_bit(pde, new_pts);
+		trace_i915_pagetable_alloc(vm, pde, start, GEN8_PDE_SHIFT);
 	}
 
 	return 0;
@@ -854,6 +872,7 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_address_space *vm,
 
 		pdp->pagedirs[pdpe] = pd;
 		set_bit(pdpe, new_pds);
+		trace_i915_pagedirectory_alloc(vm, pdpe, start, GEN8_PDPE_SHIFT);
 	}
 
 	return 0;
@@ -990,7 +1009,7 @@ static int gen8_alloc_va_range_3lvl(struct i915_address_space *vm,
 		}
 
 		set_bit(pdpe, pdp->used_pdpes);
-		gen8_map_pagetable_range(pd, start, length, dev);
+		gen8_map_pagetable_range(vm, pd, start, length);
 		pd->zombie = 0;
 	}
 
@@ -1095,7 +1114,7 @@ static int gen8_aliasing_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	}
 
 	gen8_for_each_pdpe(pd, pdp, start, size, temp, pdpe)
-		gen8_map_pagetable_range(pd, start, size, dev);
+		gen8_map_pagetable_range(&ppgtt->base, pd,start, size);
 
 	ppgtt->base.allocate_va_range = NULL;
 	ppgtt->base.teardown_va_range = NULL;
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index 2d21c54..c6137de 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -225,6 +225,38 @@ DEFINE_EVENT(i915_pagetable, i915_pagetable_destroy,
 	     TP_ARGS(vm, pde, start, pde_shift)
 );
 
+DEFINE_EVENT_PRINT(i915_pagetable, i915_pagedirectory_alloc,
+		   TP_PROTO(struct i915_address_space *vm, u32 pdpe, u64 start, u64 pdpe_shift),
+		   TP_ARGS(vm, pdpe, start, pdpe_shift),
+
+		   TP_printk("vm=%p, pdpe=%d (0x%llx-0x%llx)",
+			     __entry->vm, __entry->pde, __entry->start, __entry->end)
+);
+
+DEFINE_EVENT_PRINT(i915_pagetable, i915_pagedirectory_destroy,
+		   TP_PROTO(struct i915_address_space *vm, u32 pdpe, u64 start, u64 pdpe_shift),
+		   TP_ARGS(vm, pdpe, start, pdpe_shift),
+
+		   TP_printk("vm=%p, pdpe=%d (0x%llx-0x%llx)",
+			     __entry->vm, __entry->pde, __entry->start, __entry->end)
+);
+
+DEFINE_EVENT_PRINT(i915_pagetable, i915_pagedirpo_alloc,
+		   TP_PROTO(struct i915_address_space *vm, u32 pml4e, u64 start, u64 pml4e_shift),
+		   TP_ARGS(vm, pml4e, start, pml4e_shift),
+
+		   TP_printk("vm=%p, pml4e=%d (0x%llx-0x%llx)",
+			     __entry->vm, __entry->pde, __entry->start, __entry->end)
+);
+
+DEFINE_EVENT_PRINT(i915_pagetable, i915_pagedirpo_destroy,
+		   TP_PROTO(struct i915_address_space *vm, u32 pml4e, u64 start, u64 pml4e_shift),
+		   TP_ARGS(vm, pml4e, start, pml4e_shift),
+
+		   TP_printk("vm=%p, pml4e=%d (0x%llx-0x%llx)",
+			     __entry->vm, __entry->pde, __entry->start, __entry->end)
+);
+
 /* Avoid extra math because we only support two sizes. The format is defined by
  * bitmap_scnprintf. Each 32 bits is 8 HEX digits followed by comma */
 #define TRACE_PT_SIZE(bits) \
-- 
2.0.4

[PATCH 58/68] drm/i915/bdw: Add ppgtt info for dynamic pages

[Ben Widawsky]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c | 52 ++++++++++++++++++++++++++++---------
 drivers/gpu/drm/i915/i915_gem_gtt.c | 33 +++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_gtt.h |  9 +++++++
 3 files changed, 82 insertions(+), 12 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 09a64e6..1322718 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1825,10 +1825,38 @@ static size_t gen6_ppgtt_count_pt_pages(struct i915_hw_ppgtt *ppgtt)
 	return cnt;
 }
 
+static void gen8_ppgtt_debugfs_counter(struct i915_pagedirpo *pdp,
+				       struct i915_pagedir *pd,
+				       struct i915_pagetab *pt,
+				       unsigned pdpe,
+				       unsigned pde,
+				       void *data)
+{
+	if (!pd || !pt)
+		return;
+
+	(*(size_t *)data)++;
+}
+
+static size_t gen8_ppgtt_count_pt_pages(struct i915_hw_ppgtt *ppgtt)
+{
+	size_t count = 0;
+
+	gen8_for_every_pdpe_pde(ppgtt, gen8_ppgtt_debugfs_counter, &count);
+
+	return count;
+}
+
 static void print_ppgtt(struct seq_file *m, struct i915_hw_ppgtt *ppgtt)
 {
-	seq_printf(m, "pd gtt offset: 0x%08x\n", ppgtt->pd.pd_offset);
-	seq_printf(m, "\tpd pages: %zu\n", gen6_ppgtt_count_pt_pages(ppgtt));
+	struct drm_device *dev = ppgtt->base.dev;
+
+	if (INTEL_INFO(dev)->gen < 8) {
+		seq_printf(m, "\tpd pages: %zu\n", gen6_ppgtt_count_pt_pages(ppgtt));
+		seq_printf(m, "pd gtt offset: 0x%08x\n", ppgtt->pd.pd_offset);
+	} else {
+		seq_printf(m, "\tpage table overhead: %zu pages\n", gen8_ppgtt_count_pt_pages(ppgtt));
+	}
 }
 
 static void gen8_ppgtt_info(struct seq_file *m, struct drm_device *dev, int verbose)
@@ -1876,7 +1904,6 @@ static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev, bool ver
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct intel_engine_cs *ring;
-	struct drm_file *file;
 	int i;
 
 	if (INTEL_INFO(dev)->gen == 6)
@@ -1901,15 +1928,6 @@ static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev, bool ver
 			ppgtt->debug_dump(ppgtt, m);
 	} else
 		return;
-
-	list_for_each_entry_reverse(file, &dev->filelist, lhead) {
-		struct drm_i915_file_private *file_priv = file->driver_priv;
-
-		seq_printf(m, "proc: %s\n",
-			   get_pid_task(file->pid, PIDTYPE_PID)->comm);
-		idr_for_each(&file_priv->context_idr, per_file_ctx,
-			     (void *)((unsigned long)m | verbose));
-	}
 }
 
 static int i915_ppgtt_info(struct seq_file *m, void *data)
@@ -1918,6 +1936,7 @@ static int i915_ppgtt_info(struct seq_file *m, void *data)
 	struct drm_device *dev = node->minor->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	bool verbose = node->info_ent->data ? true : false;
+	struct drm_file *file;
 
 	int ret = mutex_lock_interruptible(&dev->struct_mutex);
 	if (ret)
@@ -1929,6 +1948,15 @@ static int i915_ppgtt_info(struct seq_file *m, void *data)
 	else if (INTEL_INFO(dev)->gen >= 6)
 		gen6_ppgtt_info(m, dev, verbose);
 
+	list_for_each_entry_reverse(file, &dev->filelist, lhead) {
+		struct drm_i915_file_private *file_priv = file->driver_priv;
+
+		seq_printf(m, "\nproc: %s\n",
+			   get_pid_task(file->pid, PIDTYPE_PID)->comm);
+		idr_for_each(&file_priv->context_idr, per_file_ctx,
+			     (void *)((unsigned long)m | verbose));
+	}
+
 	intel_runtime_pm_put(dev_priv);
 	mutex_unlock(&dev->struct_mutex);
 
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 679fe0d..43df3ee 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -2151,6 +2151,39 @@ static void gen8_ggtt_clear_range(struct i915_address_space *vm,
 	readl(gtt_base);
 }
 
+void gen8_for_every_pdpe_pde(struct i915_hw_ppgtt *ppgtt,
+			     void (*callback)(struct i915_pagedirpo *pdp,
+					      struct i915_pagedir *pd,
+					      struct i915_pagetab *pt,
+					      unsigned pdpe,
+					      unsigned pde,
+					      void *data),
+			     void *data)
+{
+	struct drm_device *dev = ppgtt->base.dev;
+	uint64_t start = ppgtt->base.start;
+	uint64_t length = ppgtt->base.total;
+	uint64_t pdpe, pde, temp;
+
+	struct i915_pagedir *pd;
+	struct i915_pagetab *pt;
+
+	gen8_for_each_pdpe(pd, &ppgtt->pdp, start, length, temp, pdpe) {
+		uint64_t pd_start = start, pd_length = length;
+		int i;
+
+		if (pd == NULL) {
+			for (i = 0; i < I915_PDES_PER_PD; i++)
+				callback(&ppgtt->pdp, NULL, NULL, pdpe, i, data);
+			continue;
+		}
+
+		gen8_for_each_pde(pt, pd, pd_start, pd_length, temp, pde) {
+			callback(&ppgtt->pdp, pd, pt, pdpe, pde, data);
+		}
+	}
+}
+
 static void gen6_ggtt_clear_range(struct i915_address_space *vm,
 				  uint64_t start,
 				  uint64_t length,
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 95b5d16..9d90995 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -508,6 +508,15 @@ static inline size_t gen8_pde_count(uint64_t addr, uint64_t length)
 	return i915_pde_count(addr, length, GEN8_PDE_SHIFT);
 }
 
+void gen8_for_every_pdpe_pde(struct i915_hw_ppgtt *ppgtt,
+			     void (*callback)(struct i915_pagedirpo *pdp,
+					      struct i915_pagedir *pd,
+					      struct i915_pagetab *pt,
+					      unsigned pdpe,
+					      unsigned pde,
+					      void *data),
+			     void *data);
+
 int i915_gem_gtt_init(struct drm_device *dev);
 void i915_gem_init_global_gtt(struct drm_device *dev);
 void i915_gem_setup_global_gtt(struct drm_device *dev, unsigned long start,
-- 
2.0.4

[PATCH 59/68] drm/i915/bdw: implement alloc/teardown for 4lvl

[Ben Widawsky]

The code for 4lvl works just as one would expect, and nicely it is able
to call into the existing 3lvl page table code to handle all of the
lower levels.

PML4 has no special attributes. We do not track its zombie status
because there will always be a PML4. So simply initialize it at
creation, and destroy it at teardown. (A similar argument can be made
for PDPs when not using sparse addresses).

Almost none of the fanciness here will exercised since the switch isn't
flipped until later.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 246 +++++++++++++++++++++++++++++++-----
 drivers/gpu/drm/i915/i915_gem_gtt.h |  14 +-
 2 files changed, 229 insertions(+), 31 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 43df3ee..9b3358f 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -405,9 +405,12 @@ static void __pdp_fini(struct i915_pagedirpo *pdp)
 static void free_pdp_single(struct i915_pagedirpo *pdp,
 			    struct drm_device *dev)
 {
-	__pdp_fini(pdp);
-	if (HAS_48B_PPGTT(dev))
+	if (HAS_48B_PPGTT(dev)) {
+		__pdp_fini(pdp);
+		i915_dma_unmap_single(pdp, dev);
+		__free_page(pdp->page);
 		kfree(pdp);
+	}
 }
 
 static int __pdp_init(struct i915_pagedirpo *pdp,
@@ -433,6 +436,60 @@ static int __pdp_init(struct i915_pagedirpo *pdp,
 	return 0;
 }
 
+static struct i915_pagedirpo *alloc_pdp_single(struct i915_hw_ppgtt *ppgtt,
+					       struct i915_pml4 *pml4)
+{
+	struct drm_device *dev = ppgtt->base.dev;
+	struct i915_pagedirpo *pdp;
+	int ret;
+
+	BUG_ON(!HAS_48B_PPGTT(dev));
+
+	pdp = kmalloc(sizeof(*pdp), GFP_KERNEL);
+	if (!pdp)
+		return ERR_PTR(-ENOMEM);
+
+	pdp->page = alloc_page(GFP_KERNEL | GFP_DMA32 | __GFP_ZERO);
+	if (!pdp->page) {
+		kfree(pdp);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	ret = __pdp_init(pdp, dev);
+	if (ret) {
+		__free_page(pdp->page);
+		kfree(pdp);
+		return ERR_PTR(ret);
+	}
+
+	i915_dma_map_px_single(pdp, dev);
+
+	return pdp;
+}
+
+static void pml4_fini(struct i915_pml4 *pml4)
+{
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(pml4, struct i915_hw_ppgtt, pml4);
+	i915_dma_unmap_single(pml4, ppgtt->base.dev);
+	__free_page(pml4->page);
+	/* HACK */
+	pml4->page = NULL;
+}
+
+static int pml4_init(struct i915_hw_ppgtt *ppgtt)
+{
+	struct i915_pml4 *pml4 = &ppgtt->pml4;
+
+	pml4->page = alloc_page(GFP_KERNEL | __GFP_ZERO);
+	if (!pml4->page)
+		return -ENOMEM;
+
+	i915_dma_map_px_single(pml4, ppgtt->base.dev);
+
+	return 0;
+}
+
 /* Broadwell Page Directory Pointer Descriptors */
 static int gen8_write_pdp(struct intel_engine_cs *ring,
 			  unsigned entry,
@@ -611,7 +668,7 @@ static void gen8_map_pagetable_range(struct i915_address_space *vm,
 	kunmap_atomic(pagedir);
 }
 
-static void gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
+static bool gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 					struct i915_pagedirpo *pdp,
 					uint64_t start, uint64_t length,
 					bool dead)
@@ -620,14 +677,23 @@ static void gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 	struct i915_pagedir *pd;
 	struct i915_pagetab *pt;
 	uint64_t temp;
-	uint32_t pdpe, pde;
+	uint32_t pdpe, pde, orig_start = start;
 
 	BUG_ON(!pdp);
+
+	if (pdp->zombie) {
+		free_pdp_single(pdp, dev);
+		trace_i915_pagedirpo_destroy(vm, 0,
+					     orig_start & GENMASK_ULL(64, GEN8_PML4E_SHIFT),
+					     GEN8_PML4E_SHIFT);
+		return true;
+	}
+
 	if (!pdp->pagedirs) {
 		WARN(!bitmap_empty(pdp->used_pdpes, I915_PDPES_PER_PDP(dev)),
 		     "Page directory leak detected\n");
 		/* If pagedirs are already free, there is nothing to do.*/
-		return;
+		return false;
 	}
 
 	gen8_for_each_pdpe(pd, pdp, start, length, temp, pdpe) {
@@ -716,8 +782,18 @@ static void gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 		}
 	}
 
-	if (dead && bitmap_empty(pdp->used_pdpes, I915_PDPES_PER_PDP(dev)))
-		free_pdp_single(pdp, dev);
+	if (bitmap_empty(pdp->used_pdpes, I915_PDPES_PER_PDP(dev))) {
+		if (!dead) {
+			pdp->zombie = 1;
+		} else {
+			free_pdp_single(pdp, dev);
+			trace_i915_pagedirpo_destroy(vm, 0,
+						     orig_start & GENMASK_ULL(64, GEN8_PML4E_SHIFT),
+						     GEN8_PML4E_SHIFT);
+		}
+		return true;
+	}
+	return false;
 }
 
 static void gen8_teardown_va_range_4lvl(struct i915_address_space *vm,
@@ -725,19 +801,49 @@ static void gen8_teardown_va_range_4lvl(struct i915_address_space *vm,
 					uint64_t start, uint64_t length,
 					bool dead)
 {
-	BUG();
+	struct i915_pagedirpo *pdp;
+	uint64_t temp, pml4e;
+
+	gen8_for_each_pml4e(pdp, pml4, start, length, temp, pml4e) {
+		if (!pdp)
+			continue;
+
+		if (gen8_teardown_va_range_3lvl(vm, pdp, start, length, dead)) {
+			clear_bit(pml4e, pml4->used_pml4es);
+			pml4->pdps[pml4e] = NULL;
+		}
+
+		WARN_ON(!test_bit(pml4e, pml4->used_pml4es) && !pdp->zombie);
+		WARN_ON(test_bit(pml4e, pml4->used_pml4es) && pdp->zombie);
+	}
 }
 
-static void gen8_teardown_va_range(struct i915_address_space *vm,
-				   uint64_t start, uint64_t length)
+static void __gen8_teardown_va_range(struct i915_address_space *vm,
+				     uint64_t start, uint64_t length,
+				     bool dead)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 
-	if (!HAS_48B_PPGTT(vm->dev))
-		gen8_teardown_va_range_3lvl(vm, &ppgtt->pdp, start, length, false);
-	else
-		gen8_teardown_va_range_4lvl(vm, &ppgtt->pml4, start, length, false);
+	if (!HAS_48B_PPGTT(vm->dev)) {
+		gen8_teardown_va_range_3lvl(vm, &ppgtt->pdp, start, length, dead);
+		if (dead) {
+			WARN_ON(!bitmap_empty(ppgtt->pdp.used_pdpes, I915_PDPES_PER_PDP(vm->dev)));
+			__pdp_fini(&ppgtt->pdp);
+		}
+	} else {
+		gen8_teardown_va_range_4lvl(vm, &ppgtt->pml4, start, length, dead);
+		if (dead) {
+			WARN_ON(!bitmap_empty(ppgtt->pml4.used_pml4es, GEN8_PML4ES_PER_PML4));
+			pml4_fini(&ppgtt->pml4);
+		}
+	}
+}
+
+static void gen8_teardown_va_range(struct i915_address_space *vm,
+				   uint64_t start, uint64_t length)
+{
+	__gen8_teardown_va_range(vm, start, length, false);
 }
 
 static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
@@ -745,10 +851,12 @@ static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 	trace_i915_va_teardown(&ppgtt->base,
 			       ppgtt->base.start, ppgtt->base.total,
 			       VM_TO_TRACE_NAME(&ppgtt->base));
-	gen8_teardown_va_range_3lvl(&ppgtt->base, &ppgtt->pdp,
-				    ppgtt->base.start, ppgtt->base.total,
-				    true);
-	BUG_ON(ppgtt->pdp.pagedirs); /* FIXME: 48b */
+	__gen8_teardown_va_range(&ppgtt->base,
+				 ppgtt->base.start, ppgtt->base.total, true);
+	if (!HAS_48B_PPGTT(ppgtt->base.dev))
+		BUG_ON(ppgtt->pdp.pagedirs);
+	else
+		BUG_ON(ppgtt->pml4.page);
 }
 
 static void gen8_ppgtt_cleanup(struct i915_address_space *vm)
@@ -1029,12 +1137,81 @@ err_out:
 	return ret;
 }
 
-static int __noreturn gen8_alloc_va_range_4lvl(struct i915_address_space *vm,
-					       struct i915_pml4 *pml4,
-					       uint64_t start,
-					       uint64_t length)
+static int gen8_alloc_va_range_4lvl(struct i915_address_space *vm,
+				    struct i915_pml4 *pml4,
+				    uint64_t start,
+				    uint64_t length)
 {
-	BUG();
+	DECLARE_BITMAP(new_pdps, GEN8_PML4ES_PER_PML4);
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(vm, struct i915_hw_ppgtt, base);
+	struct i915_pagedirpo *pdp;
+	const uint64_t orig_start = start;
+	const uint64_t orig_length = length;
+	uint64_t temp, pml4e;
+
+	/* Do the pml4 allocations first, so we don't need to track the newly
+	 * allocated tables below the pdp */
+	bitmap_zero(new_pdps, GEN8_PML4ES_PER_PML4);
+
+	/* The pagedirectory and pagetable allocations are done in the shared 3
+	 * and 4 level code. Just allocate the pdps.
+	 */
+	gen8_for_each_pml4e(pdp, pml4, start, length, temp, pml4e) {
+		if (!pdp) {
+			WARN_ON(test_bit(pml4e, pml4->used_pml4es));
+			pdp = alloc_pdp_single(ppgtt, pml4);
+			if (IS_ERR(pdp))
+				goto err_alloc;
+
+			pml4->pdps[pml4e] = pdp;
+			set_bit(pml4e, new_pdps);
+			trace_i915_pagedirpo_alloc(&ppgtt->base, pml4e,
+						   pml4e << GEN8_PML4E_SHIFT,
+						   GEN8_PML4E_SHIFT);
+
+		} else {
+			WARN(!pdp->zombie &&
+			     !test_bit(pml4e, pml4->used_pml4es), "%lld %p", pml4e, vm);
+		}
+	}
+
+	WARN(bitmap_weight(new_pdps, GEN8_PML4ES_PER_PML4) > 2,
+	     "The allocation has spanned more than 512GB. "
+	     "It is highly likely this is incorrect.");
+
+	start = orig_start;
+	length = orig_length;
+
+	gen8_for_each_pml4e(pdp, pml4, start, length, temp, pml4e) {
+		int ret;
+
+		BUG_ON(!pdp);
+
+		ret = gen8_alloc_va_range_3lvl(vm, pdp, start, length);
+		if (ret)
+			goto err_out;
+	}
+
+	bitmap_or(pml4->used_pml4es, new_pdps, pml4->used_pml4es,
+		  GEN8_PML4ES_PER_PML4);
+
+	for_each_set_bit(pml4e, pml4->used_pml4es, GEN8_PML4ES_PER_PML4)
+		pml4->pdps[pml4e]->zombie = 0;
+
+	return 0;
+
+err_out:
+	/* This will teardown more than we allocated. It should be fine, and
+	 * makes code simpler. */
+	start = orig_start;
+	length = orig_length;
+	gen8_for_each_pml4e(pdp, pml4, start, length, temp, pml4e)
+		gen8_teardown_va_range_3lvl(vm, pdp, start, length, false);
+
+err_alloc:
+	for_each_set_bit(pml4e, new_pdps, GEN8_PML4ES_PER_PML4)
+		free_pdp_single(pdp, vm->dev);
 }
 
 static int gen8_alloc_va_range(struct i915_address_space *vm,
@@ -1043,16 +1220,19 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 
-	if (!HAS_48B_PPGTT(vm->dev))
-		return gen8_alloc_va_range_3lvl(vm, &ppgtt->pdp, start, length);
-	else
+	if (HAS_48B_PPGTT(vm->dev))
 		return gen8_alloc_va_range_4lvl(vm, &ppgtt->pml4, start, length);
+	else
+		return gen8_alloc_va_range_3lvl(vm, &ppgtt->pdp, start, length);
 }
 
 static void gen8_ppgtt_fini_common(struct i915_hw_ppgtt *ppgtt)
 {
 	free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
-	free_pdp_single(&ppgtt->pdp, ppgtt->base.dev);
+	if (HAS_48B_PPGTT(ppgtt->base.dev))
+		pml4_fini(&ppgtt->pml4);
+	else
+		free_pdp_single(&ppgtt->pdp, ppgtt->base.dev);
 }
 
 /**
@@ -1076,7 +1256,13 @@ static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->enable = gen8_ppgtt_enable;
 	ppgtt->switch_mm = gen8_mm_switch;
 
-	if (!HAS_48B_PPGTT(ppgtt->base.dev)) {
+	if (HAS_48B_PPGTT(ppgtt->base.dev)) {
+		int ret = pml4_init(ppgtt);
+		if (ret) {
+			free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
+			return ret;
+		}
+	} else {
 		int ret = __pdp_init(&ppgtt->pdp, false);
 		if (ret) {
 			free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
@@ -1084,8 +1270,8 @@ static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 		}
 
 		ppgtt->switch_mm = gen8_mm_switch;
-	} else
-		BUG(); /* Not yet implemented */
+		trace_i915_pagedirpo_alloc(&ppgtt->base, 0, 0, GEN8_PML4E_SHIFT);
+	}
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 9d90995..ba103bd 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -103,6 +103,7 @@ typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
 #endif
 #define GEN8_PML4ES_PER_PML4		512
 #define GEN8_PML4E_SHIFT		39
+#define GEN8_PML4E_MASK			(GEN8_PML4ES_PER_PML4 - 1)
 #define GEN8_PDPE_SHIFT			30
 /* NB: GEN8_PDPE_MASK is untrue for 32b platforms, but it has no impact on 32b page
  * tables */
@@ -226,6 +227,7 @@ struct i915_pagedirpo {
 	dma_addr_t daddr;
 	unsigned long *used_pdpes;
 	struct i915_pagedir **pagedirs;
+	unsigned zombie:1;
 };
 
 struct i915_pml4 {
@@ -233,6 +235,7 @@ struct i915_pml4 {
 	dma_addr_t daddr;
 	DECLARE_BITMAP(used_pml4es, GEN8_PML4ES_PER_PML4);
 	struct i915_pagedirpo *pdps[GEN8_PML4ES_PER_PML4];
+	/* Don't bother tracking zombie. Just always leave it around */
 };
 
 struct i915_address_space {
@@ -455,9 +458,18 @@ static inline size_t gen6_pde_count(uint32_t addr, uint32_t length)
 	     temp = min(temp, length),					\
 	     start += temp, length -= temp)
 
+#define gen8_for_each_pml4e(pdp, pml4, start, length, temp, iter)	\
+	for (iter = gen8_pml4e_index(start), pdp = (pml4)->pdps[iter];	\
+	     length > 0 && iter < GEN8_PML4ES_PER_PML4;			\
+	     pdp = (pml4)->pdps[++iter],				\
+	     temp = ALIGN(start+1, 1ULL << GEN8_PML4E_SHIFT) - start,	\
+	     temp = min(temp, length),					\
+	     start += temp, length -= temp)
+
 #define gen8_for_each_pdpe(pd, pdp, start, length, temp, iter)		\
 	gen8_for_each_pdpe_e(pd, pdp, start, length, temp, iter, I915_PDPES_PER_PDP(dev))
 
+
 /* Clamp length to the next pagetab boundary */
 static inline uint64_t gen8_clamp_pt(uint64_t start, uint64_t length)
 {
@@ -495,7 +507,7 @@ static inline uint32_t gen8_pdpe_index(uint64_t address)
 
 static inline uint32_t gen8_pml4e_index(uint64_t address)
 {
-	BUG();
+	return (address >> GEN8_PML4E_SHIFT) & GEN8_PML4E_MASK;
 }
 
 static inline size_t gen8_pte_count(uint64_t addr, uint64_t length)
-- 
2.0.4

[PATCH 60/68] drm/i915/bdw: Add 4 level switching infrastructure

[Ben Widawsky]

Map is easy, it's the same register as the PDP descriptor 0, but it only
has one entry. Also, the mapping code is now trivial thanks to all of
the prep patches.

This is the mostly just cleaning up lose ends of 4lvl before we can
cleanup all the code to rip out the legacy page table update functions
insert and clear entries.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 60 ++++++++++++++++++++++++++++++++-----
 drivers/gpu/drm/i915/i915_gem_gtt.h |  4 ++-
 drivers/gpu/drm/i915/i915_reg.h     |  1 +
 3 files changed, 57 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 9b3358f..12c42ea 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -103,6 +103,9 @@ static inline gen8_ppgtt_pde_t gen8_pde_encode(struct drm_device *dev,
 	return pde;
 }
 
+#define gen8_pdpe_encode gen8_pde_encode
+#define gen8_pml4e_encode gen8_pde_encode
+
 static gen6_gtt_pte_t snb_pte_encode(dma_addr_t addr,
 				     enum i915_cache_level level,
 				     bool valid, u32 unused)
@@ -522,9 +525,9 @@ static int gen8_write_pdp(struct intel_engine_cs *ring,
 	return 0;
 }
 
-static int gen8_mm_switch(struct i915_hw_ppgtt *ppgtt,
-			  struct intel_engine_cs *ring,
-			  bool synchronous)
+static int gen8_legacy_mm_switch(struct i915_hw_ppgtt *ppgtt,
+				 struct intel_engine_cs *ring,
+				 bool synchronous)
 {
 	int i, ret;
 
@@ -541,6 +544,13 @@ static int gen8_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	return 0;
 }
 
+static int gen8_48b_mm_switch(struct i915_hw_ppgtt *ppgtt,
+			      struct intel_engine_cs *ring,
+			      bool synchronous)
+{
+	return gen8_write_pdp(ring, 0, ppgtt->pml4.daddr, synchronous);
+}
+
 static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 				   uint64_t start,
 				   uint64_t length,
@@ -668,6 +678,37 @@ static void gen8_map_pagetable_range(struct i915_address_space *vm,
 	kunmap_atomic(pagedir);
 }
 
+static void gen8_map_page_directory(struct i915_pagedirpo *pdp,
+				    struct i915_pagedir *pd,
+				    int index,
+				    struct drm_device *dev)
+{
+	gen8_ppgtt_pdpe_t *pagedirpo;
+	gen8_ppgtt_pdpe_t pdpe;
+
+	/* We do not need to clflush because no platform requiring flush
+	 * supports 64b pagetables. */
+	if (!HAS_48B_PPGTT(dev))
+		return;
+
+	pagedirpo = kmap_atomic(pdp->page);
+	pdpe = gen8_pdpe_encode(dev, pd->daddr, I915_CACHE_LLC);
+	pagedirpo[index] = pdpe;
+	kunmap_atomic(pagedirpo);
+}
+
+static void gen8_map_page_directory_pointer(struct i915_pml4 *pml4,
+					    struct i915_pagedirpo *pdp,
+					    int index,
+					    struct drm_device *dev)
+{
+	gen8_ppgtt_pml4e_t *pagemap = kmap_atomic(pml4->page);
+	gen8_ppgtt_pml4e_t pml4e = gen8_pml4e_encode(dev, pdp->daddr, I915_CACHE_LLC);
+	BUG_ON(!HAS_48B_PPGTT(dev));
+	pagemap[index] = pml4e;
+	kunmap_atomic(pagemap);
+}
+
 static bool gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 					struct i915_pagedirpo *pdp,
 					uint64_t start, uint64_t length,
@@ -1118,6 +1159,7 @@ static int gen8_alloc_va_range_3lvl(struct i915_address_space *vm,
 
 		set_bit(pdpe, pdp->used_pdpes);
 		gen8_map_pagetable_range(vm, pd, start, length);
+		gen8_map_page_directory(pdp, pd, pdpe, dev);
 		pd->zombie = 0;
 	}
 
@@ -1191,6 +1233,8 @@ static int gen8_alloc_va_range_4lvl(struct i915_address_space *vm,
 		ret = gen8_alloc_va_range_3lvl(vm, pdp, start, length);
 		if (ret)
 			goto err_out;
+
+		gen8_map_page_directory_pointer(pml4, pdp, pml4e, vm->dev);
 	}
 
 	bitmap_or(pml4->used_pml4es, new_pdps, pml4->used_pml4es,
@@ -1252,9 +1296,7 @@ static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->base.total = size;
 	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
 	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
-
 	ppgtt->enable = gen8_ppgtt_enable;
-	ppgtt->switch_mm = gen8_mm_switch;
 
 	if (HAS_48B_PPGTT(ppgtt->base.dev)) {
 		int ret = pml4_init(ppgtt);
@@ -1262,6 +1304,8 @@ static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 			free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
 			return ret;
 		}
+
+		ppgtt->switch_mm = gen8_48b_mm_switch;
 	} else {
 		int ret = __pdp_init(&ppgtt->pdp, false);
 		if (ret) {
@@ -1269,7 +1313,7 @@ static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 			return ret;
 		}
 
-		ppgtt->switch_mm = gen8_mm_switch;
+		ppgtt->switch_mm = gen8_legacy_mm_switch;
 		trace_i915_pagedirpo_alloc(&ppgtt->base, 0, 0, GEN8_PML4E_SHIFT);
 	}
 
@@ -1299,6 +1343,7 @@ static int gen8_aliasing_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 		return ret;
 	}
 
+	/* FIXME: PML4 */
 	gen8_for_each_pdpe(pd, pdp, start, size, temp, pdpe)
 		gen8_map_pagetable_range(&ppgtt->base, pd,start, size);
 
@@ -1536,8 +1581,9 @@ static int gen8_ppgtt_enable(struct i915_hw_ppgtt *ppgtt)
 	int j, ret;
 
 	for_each_ring(ring, dev_priv, j) {
+		u32 four_level = HAS_48B_PPGTT(dev) ? GEN8_GFX_PPGTT_64B : 0;
 		I915_WRITE(RING_MODE_GEN7(ring),
-			   _MASKED_BIT_ENABLE(GFX_PPGTT_ENABLE));
+			   _MASKED_BIT_ENABLE(GFX_PPGTT_ENABLE | four_level));
 
 		/* We promise to do a switch later with FULL PPGTT. If this is
 		 * aliasing, this is the one and only switch we'll do */
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index ba103bd..f4c611e 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -36,7 +36,9 @@
 
 typedef uint32_t gen6_gtt_pte_t;
 typedef uint64_t gen8_gtt_pte_t;
-typedef gen8_gtt_pte_t gen8_ppgtt_pde_t;
+typedef gen8_gtt_pte_t		gen8_ppgtt_pde_t;
+typedef gen8_ppgtt_pde_t	gen8_ppgtt_pdpe_t;
+typedef gen8_ppgtt_pdpe_t	gen8_ppgtt_pml4e_t;
 
 /* GEN Agnostic defines */
 #define I915_PAGE_SIZE			4096
diff --git a/drivers/gpu/drm/i915/i915_reg.h b/drivers/gpu/drm/i915/i915_reg.h
index 7a6cc69..afb2515 100644
--- a/drivers/gpu/drm/i915/i915_reg.h
+++ b/drivers/gpu/drm/i915/i915_reg.h
@@ -1227,6 +1227,7 @@ enum punit_power_well {
 #define   GFX_REPLAY_MODE		(1<<11)
 #define   GFX_PSMI_GRANULARITY		(1<<10)
 #define   GFX_PPGTT_ENABLE		(1<<9)
+#define   GEN8_GFX_PPGTT_64B		(1<<7)
 
 #define VLV_DISPLAY_BASE 0x180000
 #define VLV_MIPI_BASE VLV_DISPLAY_BASE
-- 
2.0.4

[PATCH 61/68] drm/i915/bdw: Generalize PTE writing for GEN8 PPGTT

[Ben Widawsky]

The insert_entries function was the function used to write PTEs. For the
PPGTT it was "hardcoded" to only understand two level page tables, which
was the case for GEN7. We can reuse this for 4 level page tables, and
remove the concept of insert_entries, which was never viable past 2
level page tables anyway, but it requires a bit of rework to make the
function a bit more generic.

This patch begins the generalization work, and it will be heavily used
upon when the 48b code is complete. The patch series attempts to make
each function which touches a part of code specific to the page table
level and here is no exception. Having extra variables (such as the
PPGTT) distracts and provides room to add bugs since the function
shouldn't be touching anything in the higher order page tables.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 53 +++++++++++++++++++++++++------------
 1 file changed, 36 insertions(+), 17 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 12c42ea..116d13f 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -551,23 +551,19 @@ static int gen8_48b_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	return gen8_write_pdp(ring, 0, ppgtt->pml4.daddr, synchronous);
 }
 
-static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
-				   uint64_t start,
-				   uint64_t length,
-				   bool use_scratch)
+static void gen8_ppgtt_clear_pte_range(struct i915_pagedirpo *pdp,
+				       uint64_t start,
+				       uint64_t length,
+				       gen8_gtt_pte_t scratch_pte,
+				       const bool flush)
 {
-	struct i915_hw_ppgtt *ppgtt =
-		container_of(vm, struct i915_hw_ppgtt, base);
-	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
-	gen8_gtt_pte_t *pt_vaddr, scratch_pte;
+	gen8_gtt_pte_t *pt_vaddr;
 	unsigned pdpe = gen8_pdpe_index(start);
 	unsigned pde = gen8_pde_index(start);
 	unsigned pte = gen8_pte_index(start);
 	unsigned num_entries = length >> PAGE_SHIFT;
 	unsigned last_pte, i;
 
-	scratch_pte = gen8_pte_encode(ppgtt->base.scratch.addr,
-				      I915_CACHE_LLC, use_scratch);
 
 	while (num_entries) {
 		struct i915_pagedir *pd = pdp->pagedirs[pdpe];
@@ -585,7 +581,7 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 			num_entries--;
 		}
 
-		if (!HAS_LLC(ppgtt->base.dev))
+		if (flush)
 			drm_clflush_virt_range(pt_vaddr, PAGE_SIZE);
 		kunmap_atomic(pt_vaddr);
 
@@ -597,14 +593,25 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 	}
 }
 
-static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
-				      struct sg_table *pages,
-				      uint64_t start,
-				      enum i915_cache_level cache_level, u32 unused)
+static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
+				   uint64_t start,
+				   uint64_t length,
+				   bool use_scratch)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
+	gen8_gtt_pte_t scratch_pte = gen8_pte_encode(ppgtt->base.scratch.addr,
+						     I915_CACHE_LLC, use_scratch);
+	gen8_ppgtt_clear_pte_range(pdp, start, length, scratch_pte, !HAS_LLC(vm->dev));
+}
+
+static void gen8_ppgtt_insert_pte_entries(struct i915_pagedirpo *pdp,
+					  struct sg_table *pages,
+					  uint64_t start,
+					  enum i915_cache_level cache_level,
+					  const bool flush)
+{
 	gen8_gtt_pte_t *pt_vaddr;
 	unsigned pdpe = gen8_pdpe_index(start);
 	unsigned pde = gen8_pde_index(start);
@@ -625,7 +632,7 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 			gen8_pte_encode(sg_page_iter_dma_address(&sg_iter),
 					cache_level, true);
 		if (++pte == GEN8_PTES_PER_PT) {
-			if (!HAS_LLC(ppgtt->base.dev))
+			if (flush)
 				drm_clflush_virt_range(pt_vaddr, PAGE_SIZE);
 			kunmap_atomic(pt_vaddr);
 			pt_vaddr = NULL;
@@ -637,12 +644,24 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 		}
 	}
 	if (pt_vaddr) {
-		if (!HAS_LLC(ppgtt->base.dev))
+		if (flush)
 			drm_clflush_virt_range(pt_vaddr, PAGE_SIZE);
 		kunmap_atomic(pt_vaddr);
 	}
 }
 
+static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
+				      struct sg_table *pages,
+				      uint64_t start,
+				      enum i915_cache_level cache_level,
+				      u32 unused)
+{
+	struct i915_hw_ppgtt *ppgtt = container_of(vm, struct i915_hw_ppgtt, base);
+	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
+
+	gen8_ppgtt_insert_pte_entries(pdp, pages, start, cache_level, !HAS_LLC(vm->dev));
+}
+
 static void __gen8_do_map_pt(gen8_ppgtt_pde_t * const pde,
 			     struct i915_pagetab *pt,
 			     struct drm_device *dev)
-- 
2.0.4

[PATCH 62/68] drm/i915: Plumb sg_iter through va allocation ->maps

[Ben Widawsky]

As a step towards implementing 4 levels, while not discarding the
existing pte map functions, we need to pass the sg_iter through. The
current function understands to the page directory granularity. An
object's pages may span the page directory, and so using the iter
directly as we write the PTEs allows the iterator to stay coherent
through a VMA mapping operation spanning multiple page table levels.

This looks really ugly for now, but once we move toward not separately
allocating and updating PTEs, it will make a lot of sense.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 46 +++++++++++++++++++++++--------------
 1 file changed, 29 insertions(+), 17 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 116d13f..d73f5f5 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -607,7 +607,7 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 }
 
 static void gen8_ppgtt_insert_pte_entries(struct i915_pagedirpo *pdp,
-					  struct sg_table *pages,
+					  struct sg_page_iter *sg_iter,
 					  uint64_t start,
 					  enum i915_cache_level cache_level,
 					  const bool flush)
@@ -616,11 +616,10 @@ static void gen8_ppgtt_insert_pte_entries(struct i915_pagedirpo *pdp,
 	unsigned pdpe = gen8_pdpe_index(start);
 	unsigned pde = gen8_pde_index(start);
 	unsigned pte = gen8_pte_index(start);
-	struct sg_page_iter sg_iter;
 
 	pt_vaddr = NULL;
 
-	for_each_sg_page(pages->sgl, &sg_iter, pages->nents, 0) {
+	while (__sg_page_iter_next(sg_iter)) {
 		if (pt_vaddr == NULL) {
 			struct i915_pagedir *pd = pdp->pagedirs[pdpe];
 			struct i915_pagetab *pt = pd->page_tables[pde];
@@ -629,7 +628,7 @@ static void gen8_ppgtt_insert_pte_entries(struct i915_pagedirpo *pdp,
 		}
 
 		pt_vaddr[pte] =
-			gen8_pte_encode(sg_page_iter_dma_address(&sg_iter),
+			gen8_pte_encode(sg_page_iter_dma_address(sg_iter),
 					cache_level, true);
 		if (++pte == GEN8_PTES_PER_PT) {
 			if (flush)
@@ -658,8 +657,10 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 {
 	struct i915_hw_ppgtt *ppgtt = container_of(vm, struct i915_hw_ppgtt, base);
 	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
+	struct sg_page_iter sg_iter;
 
-	gen8_ppgtt_insert_pte_entries(pdp, pages, start, cache_level, !HAS_LLC(vm->dev));
+	__sg_page_iter_start(&sg_iter, pages->sgl, sg_nents(pages->sgl), 0);
+	gen8_ppgtt_insert_pte_entries(pdp, &sg_iter, start, cache_level, !HAS_LLC(vm->dev));
 }
 
 static void __gen8_do_map_pt(gen8_ppgtt_pde_t * const pde,
@@ -1101,10 +1102,12 @@ err_out:
 	return -ENOMEM;
 }
 
-static int gen8_alloc_va_range_3lvl(struct i915_address_space *vm,
-				    struct i915_pagedirpo *pdp,
-				    uint64_t start,
-				    uint64_t length)
+static int __gen8_alloc_vma_range_3lvl(struct i915_address_space *vm,
+				       struct i915_pagedirpo *pdp,
+				       struct sg_page_iter *sg_iter,
+				       uint64_t start,
+				       uint64_t length,
+				       u32 flags)
 {
 	unsigned long *new_page_dirs, **new_page_tables;
 	struct drm_device *dev = vm->dev;
@@ -1171,7 +1174,11 @@ static int gen8_alloc_va_range_3lvl(struct i915_address_space *vm,
 				   gen8_pte_index(pd_start),
 				   gen8_pte_count(pd_start, pd_len));
 
-			/* Our pde is now pointing to the pagetable, pt */
+			if (sg_iter) {
+				BUG_ON(!sg_iter->__nents);
+				gen8_ppgtt_insert_pte_entries(pdp, sg_iter, pd_start,
+							      flags, !HAS_LLC(vm->dev));
+			}
 			set_bit(pde, pd->used_pdes);
 			pt->zombie = 0;
 		}
@@ -1198,10 +1205,12 @@ err_out:
 	return ret;
 }
 
-static int gen8_alloc_va_range_4lvl(struct i915_address_space *vm,
-				    struct i915_pml4 *pml4,
-				    uint64_t start,
-				    uint64_t length)
+static int __gen8_alloc_vma_range_4lvl(struct i915_address_space *vm,
+				       struct i915_pml4 *pml4,
+				       struct sg_page_iter *sg_iter,
+				       uint64_t start,
+				       uint64_t length,
+				       u32 flags)
 {
 	DECLARE_BITMAP(new_pdps, GEN8_PML4ES_PER_PML4);
 	struct i915_hw_ppgtt *ppgtt =
@@ -1249,7 +1258,8 @@ static int gen8_alloc_va_range_4lvl(struct i915_address_space *vm,
 
 		BUG_ON(!pdp);
 
-		ret = gen8_alloc_va_range_3lvl(vm, pdp, start, length);
+		ret = __gen8_alloc_vma_range_3lvl(vm, pdp, sg_iter,
+						  start, length, flags);
 		if (ret)
 			goto err_out;
 
@@ -1284,9 +1294,11 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 		container_of(vm, struct i915_hw_ppgtt, base);
 
 	if (HAS_48B_PPGTT(vm->dev))
-		return gen8_alloc_va_range_4lvl(vm, &ppgtt->pml4, start, length);
+		return __gen8_alloc_vma_range_4lvl(vm, &ppgtt->pml4, NULL,
+						   start, length, 0);
 	else
-		return gen8_alloc_va_range_3lvl(vm, &ppgtt->pdp, start, length);
+		return __gen8_alloc_vma_range_3lvl(vm, &ppgtt->pdp, NULL,
+						   start, length, 0);
 }
 
 static void gen8_ppgtt_fini_common(struct i915_hw_ppgtt *ppgtt)
-- 
2.0.4

[PATCH 63/68] drm/i915: Introduce map and unmap for VMAs

[Ben Widawsky]

It does what the title says by switching over to a single map function
instead of a va allocation followed by PTE writes. It still keeps the
insert_entries crutch to make risk as low as possible. It will be gone
soon. To me, this has always been the most sensible way to treat VMAs
and it's not the first time I've written this patch.

XXX: This patch was never tested pre-GEN8. After rebase it was compile
tested only on GEN8.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 78 +++++++++++++++++++++++++++++++++----
 drivers/gpu/drm/i915/i915_gem_gtt.h |  2 +
 2 files changed, 72 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index d73f5f5..3b3f844 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -609,6 +609,7 @@ static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
 static void gen8_ppgtt_insert_pte_entries(struct i915_pagedirpo *pdp,
 					  struct sg_page_iter *sg_iter,
 					  uint64_t start,
+					  size_t pages,
 					  enum i915_cache_level cache_level,
 					  const bool flush)
 {
@@ -619,7 +620,7 @@ static void gen8_ppgtt_insert_pte_entries(struct i915_pagedirpo *pdp,
 
 	pt_vaddr = NULL;
 
-	while (__sg_page_iter_next(sg_iter)) {
+	while (pages-- && __sg_page_iter_next(sg_iter)) {
 		if (pt_vaddr == NULL) {
 			struct i915_pagedir *pd = pdp->pagedirs[pdpe];
 			struct i915_pagetab *pt = pd->page_tables[pde];
@@ -660,7 +661,9 @@ static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
 	struct sg_page_iter sg_iter;
 
 	__sg_page_iter_start(&sg_iter, pages->sgl, sg_nents(pages->sgl), 0);
-	gen8_ppgtt_insert_pte_entries(pdp, &sg_iter, start, cache_level, !HAS_LLC(vm->dev));
+	gen8_ppgtt_insert_pte_entries(pdp, &sg_iter, start,
+				      sg_nents(pages->sgl),
+				      cache_level, !HAS_LLC(vm->dev));
 }
 
 static void __gen8_do_map_pt(gen8_ppgtt_pde_t * const pde,
@@ -907,6 +910,11 @@ static void gen8_teardown_va_range(struct i915_address_space *vm,
 	__gen8_teardown_va_range(vm, start, length, false);
 }
 
+static void gen8_unmap_vma(struct i915_vma *vma)
+{
+	gen8_teardown_va_range(vma->vm, vma->node.start, vma->node.size);
+}
+
 static void gen8_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
 	trace_i915_va_teardown(&ppgtt->base,
@@ -1177,6 +1185,7 @@ static int __gen8_alloc_vma_range_3lvl(struct i915_address_space *vm,
 			if (sg_iter) {
 				BUG_ON(!sg_iter->__nents);
 				gen8_ppgtt_insert_pte_entries(pdp, sg_iter, pd_start,
+							      gen8_pte_count(pd_start, pd_len),
 							      flags, !HAS_LLC(vm->dev));
 			}
 			set_bit(pde, pd->used_pdes);
@@ -1301,6 +1310,28 @@ static int gen8_alloc_va_range(struct i915_address_space *vm,
 						   start, length, 0);
 }
 
+static int gen8_map_vma(struct i915_vma *vma,
+		       enum i915_cache_level cache_level,
+		       u32 unused)
+
+{
+	struct i915_address_space *vm = vma->vm;
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(vm, struct i915_hw_ppgtt, base);
+	struct sg_page_iter sg_iter;
+
+	__sg_page_iter_start(&sg_iter, vma->obj->pages->sgl,
+			     sg_nents(vma->obj->pages->sgl), 0);
+	if (HAS_48B_PPGTT(vm->dev))
+		return __gen8_alloc_vma_range_4lvl(vm, &ppgtt->pml4, &sg_iter,
+						   vma->node.start,
+						   vma->node.size, 0);
+	else
+		return __gen8_alloc_vma_range_3lvl(vm, &ppgtt->pdp, &sg_iter,
+						   vma->node.start,
+						   vma->node.size, 0);
+}
+
 static void gen8_ppgtt_fini_common(struct i915_hw_ppgtt *ppgtt)
 {
 	free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
@@ -1326,7 +1357,6 @@ static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
 	ppgtt->base.start = 0;
 	ppgtt->base.total = size;
 	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
-	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
 	ppgtt->enable = gen8_ppgtt_enable;
 
 	if (HAS_48B_PPGTT(ppgtt->base.dev)) {
@@ -1380,6 +1410,9 @@ static int gen8_aliasing_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 
 	ppgtt->base.allocate_va_range = NULL;
 	ppgtt->base.teardown_va_range = NULL;
+	ppgtt->base.map_vma = NULL;
+	ppgtt->base.unmap_vma = NULL;
+	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
 	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
 
 	return 0;
@@ -1395,9 +1428,12 @@ static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	if (ret)
 		return ret;
 
-	ppgtt->base.allocate_va_range = gen8_alloc_va_range;
-	ppgtt->base.teardown_va_range = gen8_teardown_va_range;
-	ppgtt->base.clear_range = NULL;
+	ppgtt->base.allocate_va_range = NULL;
+	ppgtt->base.teardown_va_range = NULL;
+	ppgtt->base.map_vma = gen8_map_vma;
+	ppgtt->base.unmap_vma = gen8_unmap_vma;
+	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
+	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
 
 	return 0;
 }
@@ -1857,6 +1893,13 @@ unwind_out:
 	return ret;
 }
 
+static int gen6_map_vma(struct i915_vma *vma,
+			enum i915_cache_level cache_level,
+			u32 flags)
+{
+	return gen6_alloc_va_range(vma->vm, vma->node.start, vma->node.size);
+}
+
 static void gen6_teardown_va_range(struct i915_address_space *vm,
 				   uint64_t start, uint64_t length)
 {
@@ -1890,6 +1933,11 @@ static void gen6_teardown_va_range(struct i915_address_space *vm,
 	}
 }
 
+static void gen6_unmap_vma(struct i915_vma *vma)
+{
+	gen6_teardown_va_range(vma->vm, vma->node.start, vma->node.size);
+}
+
 static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt)
 {
 	struct i915_pagetab *pt;
@@ -2019,8 +2067,13 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt, bool aliasing)
 	if (ret)
 		return ret;
 
-	ppgtt->base.allocate_va_range = gen6_alloc_va_range;
-	ppgtt->base.teardown_va_range = gen6_teardown_va_range;
+	if (aliasing) {
+		ppgtt->base.allocate_va_range = gen6_alloc_va_range;
+		ppgtt->base.teardown_va_range = gen6_teardown_va_range;
+	} else {
+		ppgtt->base.map_vma = gen6_map_vma;
+		ppgtt->base.unmap_vma = gen6_unmap_vma;
+	}
 	ppgtt->base.clear_range = gen6_ppgtt_clear_range;
 	ppgtt->base.insert_entries = gen6_ppgtt_insert_entries;
 	ppgtt->base.cleanup = gen6_ppgtt_cleanup;
@@ -2095,6 +2148,15 @@ ppgtt_bind_vma(struct i915_vma *vma,
 	if (vma->obj->gt_ro)
 		flags |= PTE_READ_ONLY;
 
+	if (vma->vm->map_vma) {
+		trace_i915_va_alloc(vma->vm, vma->node.start, vma->node.size,
+				    VM_TO_TRACE_NAME(vma->vm));
+		ret = vma->vm->map_vma(vma, cache_level, flags);
+		if (!ret)
+			ppgtt_invalidate_tlbs(vma->vm);
+		return ret;
+	}
+
 	if (vma->vm->allocate_va_range) {
 		trace_i915_va_alloc(vma->vm, vma->node.start, vma->node.size,
 				    VM_TO_TRACE_NAME(vma->vm));
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index f4c611e..d2cd9cc 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -295,6 +295,8 @@ struct i915_address_space {
 			       struct sg_table *st,
 			       uint64_t start,
 			       enum i915_cache_level cache_level, u32 flags);
+	int (*map_vma)(struct i915_vma *vma, enum i915_cache_level cache_level, u32 flags);
+	void (*unmap_vma)(struct i915_vma *vma);
 	void (*cleanup)(struct i915_address_space *vm);
 };
 
-- 
2.0.4

[PATCH 64/68] drm/i915: Depend exclusively on map and unmap_vma

[Ben Widawsky]

Drop both the insert and clear entries, as well as the allocate/teardown
of the va range. The former was short sighted, and the latter was never
meant to be permanent.

XXX: Like the previous few patches, this was never tested pre GEN8, and
not tested individually on GEN8 post rebase.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 398 +++++++++++++-----------------------
 drivers/gpu/drm/i915/i915_gem_gtt.h |  23 +--
 2 files changed, 147 insertions(+), 274 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 3b3f844..5c23f5b 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -593,19 +593,6 @@ static void gen8_ppgtt_clear_pte_range(struct i915_pagedirpo *pdp,
 	}
 }
 
-static void gen8_ppgtt_clear_range(struct i915_address_space *vm,
-				   uint64_t start,
-				   uint64_t length,
-				   bool use_scratch)
-{
-	struct i915_hw_ppgtt *ppgtt =
-		container_of(vm, struct i915_hw_ppgtt, base);
-	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
-	gen8_gtt_pte_t scratch_pte = gen8_pte_encode(ppgtt->base.scratch.addr,
-						     I915_CACHE_LLC, use_scratch);
-	gen8_ppgtt_clear_pte_range(pdp, start, length, scratch_pte, !HAS_LLC(vm->dev));
-}
-
 static void gen8_ppgtt_insert_pte_entries(struct i915_pagedirpo *pdp,
 					  struct sg_page_iter *sg_iter,
 					  uint64_t start,
@@ -650,22 +637,6 @@ static void gen8_ppgtt_insert_pte_entries(struct i915_pagedirpo *pdp,
 	}
 }
 
-static void gen8_ppgtt_insert_entries(struct i915_address_space *vm,
-				      struct sg_table *pages,
-				      uint64_t start,
-				      enum i915_cache_level cache_level,
-				      u32 unused)
-{
-	struct i915_hw_ppgtt *ppgtt = container_of(vm, struct i915_hw_ppgtt, base);
-	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
-	struct sg_page_iter sg_iter;
-
-	__sg_page_iter_start(&sg_iter, pages->sgl, sg_nents(pages->sgl), 0);
-	gen8_ppgtt_insert_pte_entries(pdp, &sg_iter, start,
-				      sg_nents(pages->sgl),
-				      cache_level, !HAS_LLC(vm->dev));
-}
-
 static void __gen8_do_map_pt(gen8_ppgtt_pde_t * const pde,
 			     struct i915_pagetab *pt,
 			     struct drm_device *dev)
@@ -732,6 +703,8 @@ static void gen8_map_page_directory_pointer(struct i915_pml4 *pml4,
 	kunmap_atomic(pagemap);
 }
 
+/* Returns true if the PDP(s) has been freed and the caller could potentially
+ * cleanup. */
 static bool gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 					struct i915_pagedirpo *pdp,
 					uint64_t start, uint64_t length,
@@ -742,6 +715,8 @@ static bool gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 	struct i915_pagetab *pt;
 	uint64_t temp;
 	uint32_t pdpe, pde, orig_start = start;
+	gen8_gtt_pte_t scratch = gen8_pte_encode(vm->scratch.addr,
+						 I915_CACHE_LLC, true);
 
 	BUG_ON(!pdp);
 
@@ -826,9 +801,10 @@ static bool gen8_teardown_va_range_3lvl(struct i915_address_space *vm,
 							     GEN8_PDE_SHIFT);
 				pd->page_tables[pde] = NULL;
 			}
+
+			gen8_ppgtt_clear_pte_range(pdp, pd_start, pd_len, scratch, !HAS_LLC(vm->dev));
 		}
 
-		gen8_ppgtt_clear_range(vm, pd_start, pd_len, true);
 
 		if (bitmap_empty(pd->used_pdes, I915_PDES_PER_PD)) {
 			WARN_ON(!test_and_clear_bit(pdpe, pdp->used_pdpes));
@@ -868,6 +844,7 @@ static void gen8_teardown_va_range_4lvl(struct i915_address_space *vm,
 	struct i915_pagedirpo *pdp;
 	uint64_t temp, pml4e;
 
+	BUG_ON(I915_PDPES_PER_PDP(vm->dev) != 512);
 	gen8_for_each_pml4e(pdp, pml4, start, length, temp, pml4e) {
 		if (!pdp)
 			continue;
@@ -1110,14 +1087,15 @@ err_out:
 	return -ENOMEM;
 }
 
-static int __gen8_alloc_vma_range_3lvl(struct i915_address_space *vm,
-				       struct i915_pagedirpo *pdp,
+static int __gen8_alloc_vma_range_3lvl(struct i915_pagedirpo *pdp,
+				       struct i915_vma *vma,
 				       struct sg_page_iter *sg_iter,
 				       uint64_t start,
 				       uint64_t length,
 				       u32 flags)
 {
 	unsigned long *new_page_dirs, **new_page_tables;
+	struct i915_address_space *vm = vma->vm;
 	struct drm_device *dev = vm->dev;
 	struct i915_pagedir *pd;
 	const uint64_t orig_start = start;
@@ -1127,6 +1105,8 @@ static int __gen8_alloc_vma_range_3lvl(struct i915_address_space *vm,
 	size_t pdpes = I915_PDPES_PER_PDP(dev);
 	int ret;
 
+	BUG_ON(!sg_iter->sg);
+
 #ifndef CONFIG_64BIT
 	/* Disallow 64b address on 32b platforms. Nothing is wrong with doing
 	 * this in hardware, but a lot of the drm code is not prepared to handle
@@ -1176,18 +1156,16 @@ static int __gen8_alloc_vma_range_3lvl(struct i915_address_space *vm,
 			BUG_ON(!pt);
 			BUG_ON(!pd_len);
 			BUG_ON(!gen8_pte_count(pd_start, pd_len));
+			BUG_ON(!sg_iter->__nents);
 
 			/* Set our used ptes within the page table */
 			bitmap_set(pt->used_ptes,
 				   gen8_pte_index(pd_start),
 				   gen8_pte_count(pd_start, pd_len));
 
-			if (sg_iter) {
-				BUG_ON(!sg_iter->__nents);
-				gen8_ppgtt_insert_pte_entries(pdp, sg_iter, pd_start,
-							      gen8_pte_count(pd_start, pd_len),
-							      flags, !HAS_LLC(vm->dev));
-			}
+			gen8_ppgtt_insert_pte_entries(pdp, sg_iter, pd_start,
+						      gen8_pte_count(pd_start, pd_len),
+						      flags, !HAS_LLC(vm->dev));
 			set_bit(pde, pd->used_pdes);
 			pt->zombie = 0;
 		}
@@ -1214,20 +1192,21 @@ err_out:
 	return ret;
 }
 
-static int __gen8_alloc_vma_range_4lvl(struct i915_address_space *vm,
-				       struct i915_pml4 *pml4,
+static int __gen8_alloc_vma_range_4lvl(struct i915_pml4 *pml4,
+				       struct i915_vma *vma,
 				       struct sg_page_iter *sg_iter,
-				       uint64_t start,
-				       uint64_t length,
 				       u32 flags)
 {
 	DECLARE_BITMAP(new_pdps, GEN8_PML4ES_PER_PML4);
+	struct i915_address_space *vm = vma->vm;
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 	struct i915_pagedirpo *pdp;
+	uint64_t start = vma->node.start, length = vma->node.size;
 	const uint64_t orig_start = start;
 	const uint64_t orig_length = length;
 	uint64_t temp, pml4e;
+	int ret;
 
 	/* Do the pml4 allocations first, so we don't need to track the newly
 	 * allocated tables below the pdp */
@@ -1263,11 +1242,9 @@ static int __gen8_alloc_vma_range_4lvl(struct i915_address_space *vm,
 	length = orig_length;
 
 	gen8_for_each_pml4e(pdp, pml4, start, length, temp, pml4e) {
-		int ret;
-
 		BUG_ON(!pdp);
 
-		ret = __gen8_alloc_vma_range_3lvl(vm, pdp, sg_iter,
+		ret = __gen8_alloc_vma_range_3lvl(pdp, vma, sg_iter,
 						  start, length, flags);
 		if (ret)
 			goto err_out;
@@ -1294,146 +1271,80 @@ err_out:
 err_alloc:
 	for_each_set_bit(pml4e, new_pdps, GEN8_PML4ES_PER_PML4)
 		free_pdp_single(pdp, vm->dev);
-}
-
-static int gen8_alloc_va_range(struct i915_address_space *vm,
-			       uint64_t start, uint64_t length)
-{
-	struct i915_hw_ppgtt *ppgtt =
-		container_of(vm, struct i915_hw_ppgtt, base);
 
-	if (HAS_48B_PPGTT(vm->dev))
-		return __gen8_alloc_vma_range_4lvl(vm, &ppgtt->pml4, NULL,
-						   start, length, 0);
-	else
-		return __gen8_alloc_vma_range_3lvl(vm, &ppgtt->pdp, NULL,
-						   start, length, 0);
+	return ret;
 }
 
 static int gen8_map_vma(struct i915_vma *vma,
 		       enum i915_cache_level cache_level,
-		       u32 unused)
-
+		       u32 flags)
 {
 	struct i915_address_space *vm = vma->vm;
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(vm, struct i915_hw_ppgtt, base);
 	struct sg_page_iter sg_iter;
 
-	__sg_page_iter_start(&sg_iter, vma->obj->pages->sgl,
-			     sg_nents(vma->obj->pages->sgl), 0);
-	if (HAS_48B_PPGTT(vm->dev))
-		return __gen8_alloc_vma_range_4lvl(vm, &ppgtt->pml4, &sg_iter,
-						   vma->node.start,
-						   vma->node.size, 0);
+	__sg_page_iter_start(&sg_iter, vma->obj->pages->sgl, sg_nents(vma->obj->pages->sgl), 0);
+	if (HAS_48B_PPGTT(vma->vm->dev))
+		return __gen8_alloc_vma_range_4lvl(&ppgtt->pml4, vma, &sg_iter, flags);
 	else
-		return __gen8_alloc_vma_range_3lvl(vm, &ppgtt->pdp, &sg_iter,
+		return __gen8_alloc_vma_range_3lvl(&ppgtt->pdp, vma, &sg_iter,
 						   vma->node.start,
-						   vma->node.size, 0);
+						   vma->node.size,
+						   flags);
 }
 
-static void gen8_ppgtt_fini_common(struct i915_hw_ppgtt *ppgtt)
-{
-	free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
-	if (HAS_48B_PPGTT(ppgtt->base.dev))
-		pml4_fini(&ppgtt->pml4);
-	else
-		free_pdp_single(&ppgtt->pdp, ppgtt->base.dev);
-}
-
-/**
- * GEN8 legacy ppgtt programming is accomplished through a max 4 PDP registers
- * with a net effect resembling a 2-level page table in normal x86 terms. Each
- * PDP represents 1GB of memory 4 * 512 * 512 * 4096 = 4GB legacy 32b address
- * space.
- *
- */
-static int gen8_ppgtt_init_common(struct i915_hw_ppgtt *ppgtt, uint64_t size)
+static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt, bool aliasing)
 {
 	ppgtt->scratch_pd = alloc_pt_scratch(ppgtt->base.dev);
 	if (IS_ERR(ppgtt->scratch_pd))
 		return PTR_ERR(ppgtt->scratch_pd);
 
 	ppgtt->base.start = 0;
-	ppgtt->base.total = size;
-	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
-	ppgtt->enable = gen8_ppgtt_enable;
 
+	/* In the case of 3 levels we need a page directory scratch page (each
+	 * PDP entry can point to that. 4 Levels requires a single PML4 page. In
+	 * the struct definition, they are all the same anyway...
+	 */
 	if (HAS_48B_PPGTT(ppgtt->base.dev)) {
 		int ret = pml4_init(ppgtt);
 		if (ret) {
-			free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
+			free_pt_scratch(ppgtt->scratch_pml4, ppgtt->base.dev);
 			return ret;
 		}
-
+		ppgtt->base.total = (1ULL<<48);
 		ppgtt->switch_mm = gen8_48b_mm_switch;
+		/* NB: Aliasing PPGTT always aliases the GGTT which has a max of
+		 * 4GB. However, if we used 3 level page tables instead, it
+		 * would require changing the GFX_MODE register on a switch
+		 * which adds complexity. Instead we can use a 4 level table,
+		 * and only populate the low 4GB. */
+		if (aliasing) {
+			/* Make it 32b even if GGTT is less to get all 4 PDPs */
+			ppgtt->base.total = (1ULL<<32);
+		}
 	} else {
+		/* PDP doesn't need an actually page */
 		int ret = __pdp_init(&ppgtt->pdp, false);
 		if (ret) {
 			free_pt_scratch(ppgtt->scratch_pd, ppgtt->base.dev);
 			return ret;
 		}
-
 		ppgtt->switch_mm = gen8_legacy_mm_switch;
-		trace_i915_pagedirpo_alloc(&ppgtt->base, 0, 0, GEN8_PML4E_SHIFT);
+		ppgtt->base.total = (1ULL<<32);
 	}
 
-	return 0;
-}
-
-static int gen8_aliasing_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
-{
-	struct drm_device *dev = ppgtt->base.dev;
-	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct i915_pagedirpo *pdp = &ppgtt->pdp; /* FIXME: 48b */
-	struct i915_pagedir *pd;
-	uint64_t temp, start = 0, size = dev_priv->gtt.base.total;
-	uint32_t pdpe;
-	int ret;
-
-	ret = gen8_ppgtt_init_common(ppgtt, size);
-	if (ret)
-		return ret;
-
-	/* Aliasing PPGTT has to always work and be mapped because of the way we
-	 * use RESTORE_INHIBIT in the context switch. This will be fixed
-	 * eventually. */
-	ret = gen8_alloc_va_range(&ppgtt->base, start, size);
-	if (ret) {
-		gen8_ppgtt_fini_common(ppgtt);
-		return ret;
+	if (aliasing) {
+		struct i915_address_space *vm = &ppgtt->base;
+		struct drm_i915_private *dev_priv = to_i915(vm->dev);
+		ppgtt->base.total = dev_priv->gtt.base.total;
+		WARN_ON(dev_priv->gtt.base.start != 0);
 	}
 
-	/* FIXME: PML4 */
-	gen8_for_each_pdpe(pd, pdp, start, size, temp, pdpe)
-		gen8_map_pagetable_range(&ppgtt->base, pd,start, size);
-
-	ppgtt->base.allocate_va_range = NULL;
-	ppgtt->base.teardown_va_range = NULL;
-	ppgtt->base.map_vma = NULL;
-	ppgtt->base.unmap_vma = NULL;
-	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
-	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
-
-	return 0;
-}
-
-static int gen8_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
-{
-	struct drm_device *dev = ppgtt->base.dev;
-	struct drm_i915_private *dev_priv = dev->dev_private;
-	int ret;
-
-	ret = gen8_ppgtt_init_common(ppgtt, dev_priv->gtt.base.total);
-	if (ret)
-		return ret;
-
-	ppgtt->base.allocate_va_range = NULL;
-	ppgtt->base.teardown_va_range = NULL;
+	ppgtt->base.cleanup = gen8_ppgtt_cleanup;
 	ppgtt->base.map_vma = gen8_map_vma;
 	ppgtt->base.unmap_vma = gen8_unmap_vma;
-	ppgtt->base.insert_entries = gen8_ppgtt_insert_entries;
-	ppgtt->base.clear_range = gen8_ppgtt_clear_range;
+	ppgtt->enable = gen8_ppgtt_enable;
 
 	return 0;
 }
@@ -1804,8 +1715,10 @@ static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 		kunmap_atomic(pt_vaddr);
 }
 
-static int gen6_alloc_va_range(struct i915_address_space *vm,
-			       uint64_t start, uint64_t length)
+static int _gen6_map_vma(struct i915_address_space *vm,
+			 struct i915_vma *vma,
+			 enum i915_cache_level cache_level,
+			 u32 flags)
 {
 	DECLARE_BITMAP(new_page_tables, I915_PDES_PER_PD);
 	struct drm_device *dev = vm->dev;
@@ -1813,6 +1726,7 @@ static int gen6_alloc_va_range(struct i915_address_space *vm,
 	struct i915_hw_ppgtt *ppgtt =
 		        container_of(vm, struct i915_hw_ppgtt, base);
 	struct i915_pagetab *pt;
+	uint32_t start = vma->node.start, length = vma->node.size;
 	const uint32_t start_save = start, length_save = length;
 	uint32_t pde, temp;
 	int ret;
@@ -1882,6 +1796,9 @@ static int gen6_alloc_va_range(struct i915_address_space *vm,
 	 * table. Also require for WC mapped PTEs */
 	readl(dev_priv->gtt.gsm);
 
+	gen6_ppgtt_insert_entries(vm, vma->obj->pages, vma->node.start,
+				  cache_level, flags);
+
 	return 0;
 
 unwind_out:
@@ -1897,14 +1814,16 @@ static int gen6_map_vma(struct i915_vma *vma,
 			enum i915_cache_level cache_level,
 			u32 flags)
 {
-	return gen6_alloc_va_range(vma->vm, vma->node.start, vma->node.size);
+	return _gen6_map_vma(vma->vm, vma, cache_level, flags);
 }
 
 static void gen6_teardown_va_range(struct i915_address_space *vm,
 				   uint64_t start, uint64_t length)
+
 {
 	struct i915_hw_ppgtt *ppgtt =
 		        container_of(vm, struct i915_hw_ppgtt, base);
+	const uint32_t orig_start = start, orig_length = length;
 	struct i915_pagetab *pt;
 	uint32_t pde, temp;
 
@@ -1931,6 +1850,8 @@ static void gen6_teardown_va_range(struct i915_address_space *vm,
 			ppgtt->pd.page_tables[pde] = ppgtt->scratch_pt;
 		}
 	}
+
+	gen6_ppgtt_clear_range(vm, orig_start, orig_length, true);
 }
 
 static void gen6_unmap_vma(struct i915_vma *vma)
@@ -2067,15 +1988,8 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt, bool aliasing)
 	if (ret)
 		return ret;
 
-	if (aliasing) {
-		ppgtt->base.allocate_va_range = gen6_alloc_va_range;
-		ppgtt->base.teardown_va_range = gen6_teardown_va_range;
-	} else {
-		ppgtt->base.map_vma = gen6_map_vma;
-		ppgtt->base.unmap_vma = gen6_unmap_vma;
-	}
-	ppgtt->base.clear_range = gen6_ppgtt_clear_range;
-	ppgtt->base.insert_entries = gen6_ppgtt_insert_entries;
+	ppgtt->base.map_vma = gen6_map_vma;
+	ppgtt->base.unmap_vma = gen6_unmap_vma;
 	ppgtt->base.cleanup = gen6_ppgtt_cleanup;
 	ppgtt->base.start = 0;
 	ppgtt->base.total = I915_PDES_PER_PD * GEN6_PTES_PER_PT * PAGE_SIZE;
@@ -2087,8 +2001,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt, bool aliasing)
 	ppgtt->pd_addr = (gen6_gtt_pte_t __iomem*)dev_priv->gtt.gsm +
 		ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t);
 
-	if (!aliasing)
-		gen6_scratch_va_range(ppgtt, 0, ppgtt->base.total);
+	gen6_scratch_va_range(ppgtt, 0, ppgtt->base.total);
 
 	gen6_map_page_range(dev_priv, &ppgtt->pd, 0, ppgtt->base.total);
 
@@ -2109,20 +2022,20 @@ int i915_gem_init_ppgtt(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt, boo
 
 	if (INTEL_INFO(dev)->gen < 8)
 		ret = gen6_ppgtt_init(ppgtt, aliasing);
-	else if (IS_GEN8(dev) && aliasing)
-		ret = gen8_aliasing_ppgtt_init(ppgtt);
 	else if (IS_GEN8(dev))
-		ret = gen8_ppgtt_init(ppgtt);
+		ret = gen8_ppgtt_init(ppgtt, aliasing);
 	else
 		BUG();
 
 	if (ret)
 		return ret;
 
+	BUG_ON(ppgtt->base.total < dev_priv->gtt.base.total && aliasing);
+	if (aliasing)
+		ppgtt->base.total = dev_priv->gtt.base.total;
+
 	kref_init(&ppgtt->ref);
 	drm_mm_init(&ppgtt->base.mm, ppgtt->base.start, ppgtt->base.total);
-	if (ppgtt->base.clear_range)
-		ppgtt->base.clear_range(&ppgtt->base, 0, ppgtt->base.total, true);
 	i915_init_vm(dev_priv, &ppgtt->base);
 
 	return 0;
@@ -2144,56 +2057,26 @@ ppgtt_bind_vma(struct i915_vma *vma,
 {
 	int ret;
 
+	BUG_ON(!vma->vm->map_vma);
+
 	/* Currently applicable only to VLV */
 	if (vma->obj->gt_ro)
 		flags |= PTE_READ_ONLY;
 
-	if (vma->vm->map_vma) {
-		trace_i915_va_alloc(vma->vm, vma->node.start, vma->node.size,
-				    VM_TO_TRACE_NAME(vma->vm));
-		ret = vma->vm->map_vma(vma, cache_level, flags);
-		if (!ret)
-			ppgtt_invalidate_tlbs(vma->vm);
-		return ret;
-	}
-
-	if (vma->vm->allocate_va_range) {
-		trace_i915_va_alloc(vma->vm, vma->node.start, vma->node.size,
-				    VM_TO_TRACE_NAME(vma->vm));
-		ret = vma->vm->allocate_va_range(vma->vm,
-						 vma->node.start,
-						 vma->node.size);
-		if (ret)
-			return ret;
-
+	trace_i915_va_alloc(vma->vm, vma->node.start, vma->node.size,
+			    VM_TO_TRACE_NAME(vma->vm));
+	ret = vma->vm->map_vma(vma, cache_level, flags);
+	if (!ret)
 		ppgtt_invalidate_tlbs(vma->vm);
-	}
-
-	vma->vm->insert_entries(vma->vm, vma->obj->pages, vma->node.start,
-				cache_level, flags);
-
-	return 0;
+	return ret;
 }
 
 static void ppgtt_unbind_vma(struct i915_vma *vma)
 {
-	WARN_ON(vma->vm->teardown_va_range && vma->vm->clear_range);
-	if (vma->vm->teardown_va_range) {
-		trace_i915_va_teardown(vma->vm,
-				       vma->node.start, vma->node.size,
-				       VM_TO_TRACE_NAME(vma->vm));
-
-		vma->vm->teardown_va_range(vma->vm,
-					   vma->node.start, vma->node.size);
-		ppgtt_invalidate_tlbs(vma->vm);
-	} else if (vma->vm->clear_range) {
-		vma->vm->clear_range(vma->vm,
-				     vma->node.start,
-				     vma->obj->base.size,
-				     true);
-	} else
-		BUG();
-
+	trace_i915_va_teardown(vma->vm, vma->node.start, vma->node.size,
+			       VM_TO_TRACE_NAME(vma->vm));
+	vma->vm->unmap_vma(vma);
+	ppgtt_invalidate_tlbs(vma->vm);
 }
 
 extern int intel_iommu_gfx_mapped;
@@ -2275,10 +2158,10 @@ void i915_gem_suspend_gtt_mappings(struct drm_device *dev)
 
 	i915_check_and_clear_faults(dev);
 
-	dev_priv->gtt.base.clear_range(&dev_priv->gtt.base,
-				       dev_priv->gtt.base.start,
-				       dev_priv->gtt.base.total,
-				       true);
+	dev_priv->gtt.clear_range(&dev_priv->gtt,
+				  dev_priv->gtt.base.start,
+				  dev_priv->gtt.base.total,
+				  true);
 }
 
 void i915_gem_restore_gtt_mappings(struct drm_device *dev)
@@ -2290,10 +2173,10 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 	i915_check_and_clear_faults(dev);
 
 	/* First fill our portion of the GTT with scratch pages */
-	dev_priv->gtt.base.clear_range(&dev_priv->gtt.base,
-				       dev_priv->gtt.base.start,
-				       dev_priv->gtt.base.total,
-				       true);
+	dev_priv->gtt.clear_range(&dev_priv->gtt,
+				  dev_priv->gtt.base.start,
+				  dev_priv->gtt.base.total,
+				  true);
 
 	list_for_each_entry(obj, &dev_priv->mm.bound_list, global_list) {
 		struct i915_vma *vma = i915_gem_obj_to_vma(obj,
@@ -2366,15 +2249,16 @@ static inline void gen8_set_pte(void __iomem *addr, gen8_gtt_pte_t pte)
 #endif
 }
 
-static void gen8_ggtt_insert_entries(struct i915_address_space *vm,
+static void gen8_ggtt_insert_entries(struct i915_gtt *gtt,
 				     struct sg_table *st,
 				     uint64_t start,
 				     enum i915_cache_level level, u32 unused)
 {
-	struct drm_i915_private *dev_priv = vm->dev->dev_private;
+	struct drm_i915_private *dev_priv =
+		container_of(gtt, struct drm_i915_private, gtt);
 	unsigned first_entry = start >> PAGE_SHIFT;
 	gen8_gtt_pte_t __iomem *gtt_entries =
-		(gen8_gtt_pte_t __iomem *)dev_priv->gtt.gsm + first_entry;
+		(gen8_gtt_pte_t __iomem *)gtt->gsm + first_entry;
 	int i = 0;
 	struct sg_page_iter sg_iter;
 	dma_addr_t addr = 0; /* shut up gcc */
@@ -2412,22 +2296,23 @@ static void gen8_ggtt_insert_entries(struct i915_address_space *vm,
  * within the global GTT as well as accessible by the GPU through the GMADR
  * mapped BAR (dev_priv->mm.gtt->gtt).
  */
-static void gen6_ggtt_insert_entries(struct i915_address_space *vm,
+static void gen6_ggtt_insert_entries(struct i915_gtt *gtt,
 				     struct sg_table *st,
 				     uint64_t start,
 				     enum i915_cache_level level, u32 flags)
 {
-	struct drm_i915_private *dev_priv = vm->dev->dev_private;
+	struct drm_i915_private *dev_priv =
+		container_of(gtt, struct drm_i915_private, gtt);
 	unsigned first_entry = start >> PAGE_SHIFT;
 	gen6_gtt_pte_t __iomem *gtt_entries =
-		(gen6_gtt_pte_t __iomem *)dev_priv->gtt.gsm + first_entry;
+		(gen6_gtt_pte_t __iomem *)gtt->gsm + first_entry;
 	int i = 0;
 	struct sg_page_iter sg_iter;
 	dma_addr_t addr = 0;
 
 	for_each_sg_page(st->sgl, &sg_iter, st->nents, 0) {
 		addr = sg_page_iter_dma_address(&sg_iter);
-		iowrite32(vm->pte_encode(addr, level, true, flags), &gtt_entries[i]);
+		iowrite32(gtt->base.pte_encode(addr, level, true, flags), &gtt_entries[i]);
 		i++;
 	}
 
@@ -2438,8 +2323,8 @@ static void gen6_ggtt_insert_entries(struct i915_address_space *vm,
 	 * hardware should work, we must keep this posting read for paranoia.
 	 */
 	if (i != 0) {
-		unsigned long gtt = readl(&gtt_entries[i-1]);
-		WARN_ON(gtt != vm->pte_encode(addr, level, true, flags));
+		unsigned long pte = readl(&gtt_entries[i-1]);
+		WARN_ON(pte != gtt->base.pte_encode(addr, level, true, flags));
 	}
 
 	/* This next bit makes the above posting read even more important. We
@@ -2450,17 +2335,16 @@ static void gen6_ggtt_insert_entries(struct i915_address_space *vm,
 	POSTING_READ(GFX_FLSH_CNTL_GEN6);
 }
 
-static void gen8_ggtt_clear_range(struct i915_address_space *vm,
+static void gen8_ggtt_clear_range(struct i915_gtt *gtt,
 				  uint64_t start,
 				  uint64_t length,
 				  bool use_scratch)
 {
-	struct drm_i915_private *dev_priv = vm->dev->dev_private;
 	unsigned first_entry = start >> PAGE_SHIFT;
 	unsigned num_entries = length >> PAGE_SHIFT;
 	gen8_gtt_pte_t scratch_pte, __iomem *gtt_base =
-		(gen8_gtt_pte_t __iomem *) dev_priv->gtt.gsm + first_entry;
-	const int max_entries = gtt_total_entries(&dev_priv->gtt) - first_entry;
+		(gen8_gtt_pte_t __iomem *) gtt->gsm + first_entry;
+	const int max_entries = gtt_total_entries(gtt) - first_entry;
 	int i;
 
 	if (WARN(num_entries > max_entries,
@@ -2468,7 +2352,7 @@ static void gen8_ggtt_clear_range(struct i915_address_space *vm,
 		 first_entry, num_entries, max_entries))
 		num_entries = max_entries;
 
-	scratch_pte = gen8_pte_encode(vm->scratch.addr,
+	scratch_pte = gen8_pte_encode(gtt->base.scratch.addr,
 				      I915_CACHE_LLC,
 				      use_scratch);
 	for (i = 0; i < num_entries; i++)
@@ -2509,17 +2393,16 @@ void gen8_for_every_pdpe_pde(struct i915_hw_ppgtt *ppgtt,
 	}
 }
 
-static void gen6_ggtt_clear_range(struct i915_address_space *vm,
+static void gen6_ggtt_clear_range(struct i915_gtt *gtt,
 				  uint64_t start,
 				  uint64_t length,
 				  bool use_scratch)
 {
-	struct drm_i915_private *dev_priv = vm->dev->dev_private;
 	unsigned first_entry = start >> PAGE_SHIFT;
 	unsigned num_entries = length >> PAGE_SHIFT;
 	gen6_gtt_pte_t scratch_pte, __iomem *gtt_base =
-		(gen6_gtt_pte_t __iomem *) dev_priv->gtt.gsm + first_entry;
-	const int max_entries = gtt_total_entries(&dev_priv->gtt) - first_entry;
+		(gen6_gtt_pte_t __iomem *) gtt->gsm + first_entry;
+	const int max_entries = gtt_total_entries(gtt) - first_entry;
 	int i;
 
 	if (WARN(num_entries > max_entries,
@@ -2527,7 +2410,8 @@ static void gen6_ggtt_clear_range(struct i915_address_space *vm,
 		 first_entry, num_entries, max_entries))
 		num_entries = max_entries;
 
-	scratch_pte = vm->pte_encode(vm->scratch.addr, I915_CACHE_LLC, use_scratch, 0);
+	scratch_pte = gtt->base.pte_encode(gtt->base.scratch.addr,
+					   I915_CACHE_LLC, use_scratch, 0);
 
 	for (i = 0; i < num_entries; i++)
 		iowrite32(scratch_pte, &gtt_base[i]);
@@ -2550,7 +2434,7 @@ static int i915_ggtt_bind_vma(struct i915_vma *vma,
 	return 0;
 }
 
-static void i915_ggtt_clear_range(struct i915_address_space *vm,
+static void i915_ggtt_clear_range(struct i915_gtt *gunused,
 				  uint64_t start,
 				  uint64_t length,
 				  bool unused)
@@ -2596,9 +2480,10 @@ static int ggtt_bind_vma(struct i915_vma *vma,
 	if (!dev_priv->mm.aliasing_ppgtt || flags & GLOBAL_BIND) {
 		if (!obj->has_global_gtt_mapping ||
 		    (cache_level != obj->cache_level)) {
-			vma->vm->insert_entries(vma->vm, obj->pages,
-						vma->node.start,
-						cache_level, flags);
+			struct i915_gtt *gtt = &dev_priv->gtt;
+			gtt->insert_entries(gtt, obj->pages,
+					    vma->node.start,
+					    cache_level, flags);
 			obj->has_global_gtt_mapping = 1;
 		}
 	}
@@ -2609,11 +2494,7 @@ static int ggtt_bind_vma(struct i915_vma *vma,
 	if (dev_priv->mm.aliasing_ppgtt &&
 	    (!obj->has_aliasing_ppgtt_mapping ||
 	     (cache_level != obj->cache_level))) {
-		struct i915_hw_ppgtt *appgtt = dev_priv->mm.aliasing_ppgtt;
-		appgtt->base.insert_entries(&appgtt->base,
-					    vma->obj->pages,
-					    vma->node.start,
-					    cache_level, flags);
+		ppgtt_bind_vma(vma, cache_level, flags);
 		vma->obj->has_aliasing_ppgtt_mapping = 1;
 	}
 
@@ -2622,24 +2503,20 @@ static int ggtt_bind_vma(struct i915_vma *vma,
 
 static void ggtt_unbind_vma(struct i915_vma *vma)
 {
-	struct drm_device *dev = vma->vm->dev;
-	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct drm_i915_gem_object *obj = vma->obj;
+	struct drm_device *dev = obj->base.dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_gtt *gtt = &dev_priv->gtt;
+
+	BUG_ON(vma->vm != &gtt->base);
 
 	if (obj->has_global_gtt_mapping) {
-		vma->vm->clear_range(vma->vm,
-				     vma->node.start,
-				     obj->base.size,
-				     true);
+		gtt->clear_range(gtt, vma->node.start, obj->base.size, true);
 		obj->has_global_gtt_mapping = 0;
 	}
 
 	if (obj->has_aliasing_ppgtt_mapping) {
-		struct i915_hw_ppgtt *appgtt = dev_priv->mm.aliasing_ppgtt;
-		appgtt->base.clear_range(&appgtt->base,
-					 vma->node.start,
-					 obj->base.size,
-					 true);
+		ppgtt_unbind_vma(vma);
 		obj->has_aliasing_ppgtt_mapping = 0;
 	}
 }
@@ -2692,7 +2569,8 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 	 * of the aperture.
 	 */
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct i915_address_space *ggtt_vm = &dev_priv->gtt.base;
+	struct i915_gtt *gtt = &dev_priv->gtt;
+	struct i915_address_space *ggtt_vm = &gtt->base;
 	struct drm_mm_node *entry;
 	struct drm_i915_gem_object *obj;
 	unsigned long hole_start, hole_end;
@@ -2725,12 +2603,12 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 	drm_mm_for_each_hole(entry, &ggtt_vm->mm, hole_start, hole_end) {
 		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
 			      hole_start, hole_end);
-		ggtt_vm->clear_range(ggtt_vm, hole_start,
-				     hole_end - hole_start, true);
+		gtt->clear_range(gtt, hole_start,
+				 hole_end - hole_start, true);
 	}
 
 	/* And finally clear the reserved guard page */
-	ggtt_vm->clear_range(ggtt_vm, end - PAGE_SIZE, PAGE_SIZE, true);
+	gtt->clear_range(gtt, end - PAGE_SIZE, PAGE_SIZE, true);
 }
 
 void i915_gem_init_global_gtt(struct drm_device *dev)
@@ -2961,8 +2839,8 @@ static int gen8_gmch_probe(struct drm_device *dev,
 
 	ret = ggtt_probe_common(dev, gtt_size);
 
-	dev_priv->gtt.base.clear_range = gen8_ggtt_clear_range;
-	dev_priv->gtt.base.insert_entries = gen8_ggtt_insert_entries;
+	dev_priv->gtt.clear_range = gen8_ggtt_clear_range;
+	dev_priv->gtt.insert_entries = gen8_ggtt_insert_entries;
 
 	return ret;
 }
@@ -3001,8 +2879,8 @@ static int gen6_gmch_probe(struct drm_device *dev,
 
 	ret = ggtt_probe_common(dev, gtt_size);
 
-	dev_priv->gtt.base.clear_range = gen6_ggtt_clear_range;
-	dev_priv->gtt.base.insert_entries = gen6_ggtt_insert_entries;
+	dev_priv->gtt.clear_range = gen6_ggtt_clear_range;
+	dev_priv->gtt.insert_entries = gen6_ggtt_insert_entries;
 
 	return ret;
 }
@@ -3038,7 +2916,7 @@ static int i915_gmch_probe(struct drm_device *dev,
 	intel_gtt_get(gtt_total, stolen, mappable_base, mappable_end);
 
 	dev_priv->gtt.do_idle_maps = needs_idle_maps(dev_priv->dev);
-	dev_priv->gtt.base.clear_range = i915_ggtt_clear_range;
+	dev_priv->gtt.clear_range = i915_ggtt_clear_range;
 
 	if (unlikely(dev_priv->gtt.do_idle_maps))
 		DRM_INFO("applying Ironlake quirks for intel_iommu\n");
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index d2cd9cc..92acd95 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -281,20 +281,6 @@ struct i915_address_space {
 	gen6_gtt_pte_t (*pte_encode)(dma_addr_t addr,
 				     enum i915_cache_level level,
 				     bool valid, u32 flags); /* Create a valid PTE */
-	int (*allocate_va_range)(struct i915_address_space *vm,
-				 uint64_t start,
-				 uint64_t length);
-	void (*teardown_va_range)(struct i915_address_space *vm,
-				  uint64_t start,
-				  uint64_t length);
-	void (*clear_range)(struct i915_address_space *vm,
-			    uint64_t start,
-			    uint64_t length,
-			    bool use_scratch);
-	void (*insert_entries)(struct i915_address_space *vm,
-			       struct sg_table *st,
-			       uint64_t start,
-			       enum i915_cache_level cache_level, u32 flags);
 	int (*map_vma)(struct i915_vma *vma, enum i915_cache_level cache_level, u32 flags);
 	void (*unmap_vma)(struct i915_vma *vma);
 	void (*cleanup)(struct i915_address_space *vm);
@@ -313,6 +299,7 @@ struct i915_hw_ppgtt {
 	union {
 		struct i915_pagetab *scratch_pt;
 		struct i915_pagetab *scratch_pd; /* Just need the daddr */
+		struct i915_pagetab *scratch_pml4;
 	};
 
 	struct intel_context *ctx;
@@ -352,6 +339,14 @@ struct i915_gtt {
 	int (*gtt_probe)(struct drm_device *dev, size_t *gtt_total,
 			  size_t *stolen, phys_addr_t *mappable_base,
 			  unsigned long *mappable_end);
+	void (*insert_entries)(struct i915_gtt *gtt,
+			       struct sg_table *st,
+			       uint64_t start,
+			       enum i915_cache_level cache_level, u32 flags);
+	void (*clear_range)(struct i915_gtt *gtt,
+			    uint64_t start,
+			    uint64_t length,
+			    bool use_scratch);
 };
 
 /* For each pde iterates over every pde between from start until start + length.
-- 
2.0.4

[PATCH 65/68] drm/i915: Expand error state's address width to 64b

[Ben Widawsky]

v2: 0 pad the new 8B fields or else intel_error_decode has a hard time.
Note, regardless we need an igt update.

v3: Make reloc_offset 64b also.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h       |  4 ++--
 drivers/gpu/drm/i915/i915_gpu_error.c | 17 +++++++++--------
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index ff921e6..8d46993 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -375,7 +375,7 @@ struct drm_i915_error_state {
 
 		struct drm_i915_error_object {
 			int page_count;
-			u32 gtt_offset;
+			u64 gtt_offset;
 			u32 *pages[0];
 		} *ringbuffer, *batchbuffer, *wa_batchbuffer, *ctx, *hws_page;
 
@@ -400,7 +400,7 @@ struct drm_i915_error_state {
 		u32 size;
 		u32 name;
 		u32 rseqno, wseqno;
-		u32 gtt_offset;
+		u64 gtt_offset;
 		u32 read_domains;
 		u32 write_domain;
 		s32 fence_reg:I915_MAX_NUM_FENCE_BITS;
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index c391268..00b6176 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -209,7 +209,7 @@ static void print_error_buffers(struct drm_i915_error_state_buf *m,
 	err_printf(m, "%s [%d]:\n", name, count);
 
 	while (count--) {
-		err_printf(m, "  %08x %8u %02x %02x %x %x",
+		err_printf(m, "  %016llx %8u %02x %02x %x %x",
 			   err->gtt_offset,
 			   err->size,
 			   err->read_domains,
@@ -428,7 +428,7 @@ int i915_error_state_to_str(struct drm_i915_error_state_buf *m,
 				err_printf(m, " (submitted by %s [%d])",
 					   error->ring[i].comm,
 					   error->ring[i].pid);
-			err_printf(m, " --- gtt_offset = 0x%08x\n",
+			err_printf(m, " --- gtt_offset = 0x%016llx\n",
 				   obj->gtt_offset);
 			print_error_obj(m, obj);
 		}
@@ -436,7 +436,8 @@ int i915_error_state_to_str(struct drm_i915_error_state_buf *m,
 		obj = error->ring[i].wa_batchbuffer;
 		if (obj) {
 			err_printf(m, "%s (w/a) --- gtt_offset = 0x%08x\n",
-				   dev_priv->ring[i].name, obj->gtt_offset);
+				   dev_priv->ring[i].name,
+				   lower_32_bits(obj->gtt_offset));
 			print_error_obj(m, obj);
 		}
 
@@ -455,14 +456,14 @@ int i915_error_state_to_str(struct drm_i915_error_state_buf *m,
 		if ((obj = error->ring[i].ringbuffer)) {
 			err_printf(m, "%s --- ringbuffer = 0x%08x\n",
 				   dev_priv->ring[i].name,
-				   obj->gtt_offset);
+				   lower_32_bits(obj->gtt_offset));
 			print_error_obj(m, obj);
 		}
 
 		if ((obj = error->ring[i].hws_page)) {
 			err_printf(m, "%s --- HW Status = 0x%08x\n",
 				   dev_priv->ring[i].name,
-				   obj->gtt_offset);
+				   lower_32_bits(obj->gtt_offset));
 			offset = 0;
 			for (elt = 0; elt < PAGE_SIZE/16; elt += 4) {
 				err_printf(m, "[%04x] %08x %08x %08x %08x\n",
@@ -478,13 +479,13 @@ int i915_error_state_to_str(struct drm_i915_error_state_buf *m,
 		if ((obj = error->ring[i].ctx)) {
 			err_printf(m, "%s --- HW Context = 0x%08x\n",
 				   dev_priv->ring[i].name,
-				   obj->gtt_offset);
+				   lower_32_bits(obj->gtt_offset));
 			print_error_obj(m, obj);
 		}
 	}
 
 	if ((obj = error->semaphore_obj)) {
-		err_printf(m, "Semaphore page = 0x%08x\n", obj->gtt_offset);
+		err_printf(m, "Semaphore page = 0x%016llx\n", obj->gtt_offset);
 		for (elt = 0; elt < PAGE_SIZE/16; elt += 4) {
 			err_printf(m, "[%04x] %08x %08x %08x %08x\n",
 				   elt * 4,
@@ -580,7 +581,7 @@ i915_error_object_create_sized(struct drm_i915_private *dev_priv,
 {
 	struct drm_i915_error_object *dst;
 	int i;
-	u32 reloc_offset;
+	u64 reloc_offset;
 
 	if (src == NULL || src->pages == NULL)
 		return NULL;
-- 
2.0.4

[PATCH 66/68] drm/i915/bdw: Flip the 48b switch

[Ben Widawsky]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     | 2 +-
 drivers/gpu/drm/i915/i915_gem_gtt.c | 3 ---
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 8d46993..00d9ab3 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2079,7 +2079,7 @@ struct drm_i915_cmd_table {
 #define USES_PPGTT(dev)		(i915.enable_ppgtt)
 #define USES_FULL_PPGTT(dev)	(i915.enable_ppgtt == 2)
 #ifdef CONFIG_64BIT
-# define HAS_48B_PPGTT(dev)	(IS_BROADWELL(dev) && false)
+# define HAS_48B_PPGTT(dev)	IS_BROADWELL(dev)
 #else
 # define HAS_48B_PPGTT(dev)	false
 #endif
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 5c23f5b..f5b8aea 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -1013,9 +1013,6 @@ static int gen8_ppgtt_alloc_pagedirs(struct i915_address_space *vm,
 
 	BUG_ON(!bitmap_empty(new_pds, pdpes));
 
-	/* FIXME: PPGTT container_of won't work for 64b */
-	BUG_ON((start + length) > 0x800000000ULL);
-
 	gen8_for_each_pdpe(pd, pdp, start, length, temp, pdpe) {
 		if (pd)
 			continue;
-- 
2.0.4

[PATCH 67/68] drm/i915: Provide a soft_pin hook

[Ben Widawsky]

This patch is a proof of concept hack which repurposes the MSB of the
size field in created. Userptr already has the gup code, and all we need
to do is reuse it.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h            | 12 ++++-
 drivers/gpu/drm/i915/i915_gem.c            | 86 +++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  5 +-
 drivers/gpu/drm/i915/i915_gem_gtt.h        |  1 +
 drivers/gpu/drm/i915/i915_gem_userptr.c    |  7 +--
 include/uapi/drm/i915_drm.h                |  3 +-
 6 files changed, 100 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 00d9ab3..cfad8c1 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2298,8 +2298,14 @@ void *i915_gem_object_alloc(struct drm_device *dev);
 void i915_gem_object_free(struct drm_i915_gem_object *obj);
 void i915_gem_object_init(struct drm_i915_gem_object *obj,
 			 const struct drm_i915_gem_object_ops *ops);
-struct drm_i915_gem_object *i915_gem_alloc_object(struct drm_device *dev,
-						  size_t size);
+struct drm_i915_gem_object *_i915_gem_alloc_object(struct drm_device *dev,
+						  size_t size,
+						  bool user_obj);
+#define i915_gem_alloc_object(dev, size) _i915_gem_alloc_object(dev, size, false)
+int i915_gem_userptr_init__mmu_notifier(struct drm_i915_gem_object *obj, unsigned flags);
+int i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj);
+void i915_gem_userptr_put_pages(struct drm_i915_gem_object *obj);
+#define i915_is_soft_pinned(vma) ((vma)->obj->ops->get_pages == i915_gem_userptr_get_pages)
 void i915_init_vm(struct drm_i915_private *dev_priv,
 		  struct i915_address_space *vm);
 void i915_gem_free_object(struct drm_gem_object *obj);
@@ -2311,11 +2317,13 @@ void i915_gem_vma_destroy(struct i915_vma *vma);
 #define PIN_ALIASING	(1<<3)
 #define PIN_GLOBAL_ALIASED (PIN_ALIASING | PIN_GLOBAL)
 #define PIN_OFFSET_BIAS (1<<4)
+#define PIN_SOFT	(1<<5)
 #define PIN_OFFSET_MASK (PAGE_MASK)
 int __must_check i915_gem_object_pin(struct drm_i915_gem_object *obj,
 				     struct i915_address_space *vm,
 				     uint32_t alignment,
 				     uint64_t flags);
+int i915_vma_softpin(struct i915_vma *vma, uint64_t start, uint64_t size);
 int __must_check i915_vma_unbind(struct i915_vma *vma);
 int i915_gem_object_put_pages(struct drm_i915_gem_object *obj);
 void i915_gem_release_all_mmaps(struct drm_i915_private *dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 6413f3a..75d2454 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -347,21 +347,33 @@ static int
 i915_gem_create(struct drm_file *file,
 		struct drm_device *dev,
 		uint64_t size,
+		uint64_t loc,
 		uint32_t *handle_p)
 {
 	struct drm_i915_gem_object *obj;
 	int ret;
 	u32 handle;
+	bool user_obj = false;
+
+	if (size & BIT_ULL(63)) {
+		if (!HAS_48B_PPGTT(dev) || !USES_FULL_PPGTT(dev))
+			return -EINVAL;
+		size &= ~BIT_ULL(63);
+		user_obj = true;
+	}
 
 	size = roundup(size, PAGE_SIZE);
 	if (size == 0)
 		return -EINVAL;
 
 	/* Allocate the new object */
-	obj = i915_gem_alloc_object(dev, size);
+	obj = _i915_gem_alloc_object(dev, size, user_obj);
 	if (obj == NULL)
 		return -ENOMEM;
 
+	if (user_obj)
+		 obj->userptr.ptr = loc;
+
 	ret = drm_gem_handle_create(file, &obj->base, &handle);
 	/* drop reference from allocate - handle holds it now */
 	drm_gem_object_unreference_unlocked(&obj->base);
@@ -380,7 +392,7 @@ i915_gem_dumb_create(struct drm_file *file,
 	/* have to work out size/pitch and return them */
 	args->pitch = ALIGN(args->width * DIV_ROUND_UP(args->bpp, 8), 64);
 	args->size = args->pitch * args->height;
-	return i915_gem_create(file, dev,
+	return i915_gem_create(file, dev, 0,
 			       args->size, &args->handle);
 }
 
@@ -392,9 +404,10 @@ i915_gem_create_ioctl(struct drm_device *dev, void *data,
 		      struct drm_file *file)
 {
 	struct drm_i915_gem_create *args = data;
+	uint64_t location = ((uint64_t)args->handle) << 32 | args->pad;
 
 	return i915_gem_create(file, dev,
-			       args->size, &args->handle);
+			       args->size, location, &args->handle);
 }
 
 static inline int
@@ -3038,7 +3051,10 @@ int i915_vma_unbind(struct i915_vma *vma)
 	if (i915_is_ggtt(vma->vm))
 		obj->map_and_fenceable = true;
 
-	drm_mm_remove_node(&vma->node);
+	if (i915_is_soft_pinned(vma))
+		vma->node.allocated = 0;
+	else
+		drm_mm_remove_node(&vma->node);
 	i915_gem_vma_destroy(vma);
 
 	/* Since the unbound list is global, only move to that list if
@@ -3544,6 +3560,8 @@ i915_gem_object_bind_to_vm(struct drm_i915_gem_object *obj,
 	if (IS_ERR(vma))
 		goto err_unpin;
 
+	BUG_ON(i915_is_soft_pinned(vma));
+
 search_free:
 	ret = drm_mm_insert_node_in_range_generic(&vm->mm, &vma->node,
 						  size, alignment,
@@ -4227,6 +4245,43 @@ i915_gem_object_pin(struct drm_i915_gem_object *obj,
 	return 0;
 }
 
+int i915_vma_softpin(struct i915_vma *vma, uint64_t start, uint64_t size)
+{
+	struct drm_i915_private *dev_priv = to_i915(vma->vm->dev);
+	struct drm_i915_gem_object *obj = vma->obj;
+	int ret;
+
+	BUG_ON(!i915_is_soft_pinned(vma));
+	ret = i915_gem_object_get_pages(obj);
+	if (ret)
+		return ret;
+
+	i915_gem_object_pin_pages(obj);
+
+	ret = i915_gem_gtt_prepare_object(obj);
+	if (ret)
+		goto err_release_pages;
+
+	vma->node.allocated = 1;
+
+	list_move_tail(&obj->global_list, &dev_priv->mm.bound_list);
+	list_add_tail(&vma->mm_list, &vma->vm->inactive_list);
+
+	vma->node.start = start;
+	vma->node.size = size;
+	trace_i915_vma_bind(vma, PIN_SOFT);
+	i915_gem_vma_bind(vma, obj->cache_level, SOFT_PINNED);
+	vma->pin_count++;
+
+	return 0;
+
+err_release_pages:
+	i915_gem_vma_destroy(vma);
+	i915_gem_object_unpin_pages(obj);
+
+	return ret;
+}
+
 void
 i915_gem_object_ggtt_unpin(struct drm_i915_gem_object *obj)
 {
@@ -4473,8 +4528,16 @@ static const struct drm_i915_gem_object_ops i915_gem_object_ops = {
 	.put_pages = i915_gem_object_put_pages_gtt,
 };
 
-struct drm_i915_gem_object *i915_gem_alloc_object(struct drm_device *dev,
-						  size_t size)
+/* Soft pinned objects are those which have user pages, and an offset defined by
+ * userspace
+ */
+static const struct drm_i915_gem_object_ops i915_gem_soft_pin_object_ops = {
+	.get_pages = i915_gem_userptr_get_pages,
+	.put_pages = i915_gem_userptr_put_pages,
+};
+
+struct drm_i915_gem_object *_i915_gem_alloc_object(struct drm_device *dev,
+						   size_t size, bool user_obj)
 {
 	struct drm_i915_gem_object *obj;
 	struct address_space *mapping;
@@ -4499,7 +4562,16 @@ struct drm_i915_gem_object *i915_gem_alloc_object(struct drm_device *dev,
 	mapping = file_inode(obj->base.filp)->i_mapping;
 	mapping_set_gfp_mask(mapping, mask);
 
-	i915_gem_object_init(obj, &i915_gem_object_ops);
+	if (user_obj) {
+		i915_gem_object_init(obj, &i915_gem_soft_pin_object_ops);
+		obj->userptr.mm = get_task_mm(current);
+		if (i915_gem_userptr_init__mmu_notifier(obj, 0)) {
+			i915_gem_object_free(obj);
+			drm_gem_object_unreference_unlocked(&obj->base);
+			return NULL;
+		}
+	} else
+		i915_gem_object_init(obj, &i915_gem_object_ops);
 
 	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
 	obj->base.read_domains = I915_GEM_DOMAIN_CPU;
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index fdd68d6..c1761f0 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -571,7 +571,10 @@ i915_gem_execbuffer_reserve_vma(struct i915_vma *vma,
 	if (entry->flags & __EXEC_OBJECT_NEEDS_BIAS)
 		flags |= BATCH_OFFSET_BIAS | PIN_OFFSET_BIAS;
 
-	ret = i915_gem_object_pin(obj, vma->vm, entry->alignment, flags);
+	if (i915_is_soft_pinned(vma))
+		ret = i915_vma_softpin(vma, entry->offset, vma->obj->base.size);
+	else
+		ret = i915_gem_object_pin(obj, vma->vm, entry->alignment, flags);
 	if (ret)
 		return ret;
 
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index 92acd95..e4ab4ba 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -185,6 +185,7 @@ struct i915_vma {
 /* Only use this if you know you want a strictly aliased binding */
 #define ALIASING_BIND (1<<1)
 #define PTE_READ_ONLY (1<<2)
+#define SOFT_PINNED   (1<<3) /* Just debug for now */
 	int (*bind_vma)(struct i915_vma *vma,
 			enum i915_cache_level cache_level,
 			u32 flags);
diff --git a/drivers/gpu/drm/i915/i915_gem_userptr.c b/drivers/gpu/drm/i915/i915_gem_userptr.c
index fe69fc8..ff7aea6 100644
--- a/drivers/gpu/drm/i915/i915_gem_userptr.c
+++ b/drivers/gpu/drm/i915/i915_gem_userptr.c
@@ -341,7 +341,7 @@ i915_gem_userptr_release__mmu_notifier(struct drm_i915_gem_object *obj)
 	obj->userptr.mn = NULL;
 }
 
-static int
+int
 i915_gem_userptr_init__mmu_notifier(struct drm_i915_gem_object *obj,
 				    unsigned flags)
 {
@@ -519,7 +519,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work)
 	kfree(work);
 }
 
-static int
+int
 i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
 {
 	const int num_pages = obj->base.size >> PAGE_SHIFT;
@@ -598,6 +598,7 @@ i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
 					get_task_struct(work->task);
 
 					INIT_WORK(&work->work, __i915_gem_userptr_get_pages_worker);
+					WARN_ON(!obj->userptr.ptr);
 					schedule_work(&work->work);
 				} else
 					ret = -ENOMEM;
@@ -621,7 +622,7 @@ i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
 	return ret;
 }
 
-static void
+void
 i915_gem_userptr_put_pages(struct drm_i915_gem_object *obj)
 {
 	struct scatterlist *sg;
diff --git a/include/uapi/drm/i915_drm.h b/include/uapi/drm/i915_drm.h
index ff57f07..413a4fb 100644
--- a/include/uapi/drm/i915_drm.h
+++ b/include/uapi/drm/i915_drm.h
@@ -662,7 +662,8 @@ struct drm_i915_gem_exec_object2 {
 #define EXEC_OBJECT_NEEDS_FENCE (1<<0)
 #define EXEC_OBJECT_NEEDS_GTT	(1<<1)
 #define EXEC_OBJECT_WRITE	(1<<2)
-#define __EXEC_OBJECT_UNKNOWN_FLAGS -(EXEC_OBJECT_WRITE<<1)
+#define EXEC_OBJECT_SOFT_PINNED (1<<3)
+#define __EXEC_OBJECT_UNKNOWN_FLAGS -(EXEC_OBJECT_SOFT_PINNED<<1)
 	__u64 flags;
 
 	__u64 rsvd1;
-- 
2.0.4

[PATCH 68/68] XXX: drm/i915: Unexplained workarounds

[Ben Widawsky]

1. Always force invalidate. This doesn't fix any bugs, but it makes the
time to failure longer.

2. Make TLB validation explicit

I can't say I've spent too much time with these, however, it seems if I
use each individually, I get no observable stability improvement. When I
use them together though, I do (still hangs in the end)
---
 drivers/gpu/drm/i915/i915_gem_context.c | 2 ++
 drivers/gpu/drm/i915/intel_ringbuffer.c | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 7d81904..195bc0b 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -813,6 +813,8 @@ static int do_switch_rcs(struct intel_engine_cs *ring,
 		 * die because future work may end up depending on valid address
 		 * space. This means we must enforce that a page table load
 		 * occur when this occurs. */
+	} else if (IS_GEN8(ring->dev) && USES_FULL_PPGTT(ring->dev)) {
+		hw_flags |= MI_FORCE_RESTORE;
 	} else if (test_and_clear_bit(ring->id, &to->vm->pd_reload_mask))
 		hw_flags |= MI_FORCE_RESTORE;
 
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 117543e..b6ed038 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -645,7 +645,7 @@ static int init_render_ring(struct intel_engine_cs *ring)
 			   _MASKED_BIT_ENABLE(GFX_TLB_INVALIDATE_EXPLICIT));
 
 	/* WaBCSVCSTlbInvalidationMode:ivb,vlv,hsw */
-	if (IS_GEN7(dev))
+	if (IS_GEN7(dev) || IS_GEN8(dev))
 		I915_WRITE(GFX_MODE_GEN7,
 			   _MASKED_BIT_ENABLE(GFX_TLB_INVALIDATE_EXPLICIT) |
 			   _MASKED_BIT_ENABLE(GFX_REPLAY_MODE));
-- 
2.0.4

[PATCH 1/2] intel: Split out bo allocation

[Ben Widawsky]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 intel/intel_bufmgr_gem.c | 69 ++++++++++++++++++++++++++++--------------------
 1 file changed, 40 insertions(+), 29 deletions(-)

diff --git a/intel/intel_bufmgr_gem.c b/intel/intel_bufmgr_gem.c
index 0e1cb0d..d7d3769 100644
--- a/intel/intel_bufmgr_gem.c
+++ b/intel/intel_bufmgr_gem.c
@@ -636,6 +636,43 @@ drm_intel_gem_bo_cache_purge_bucket(drm_intel_bufmgr_gem *bufmgr_gem,
 	}
 }
 
+static drm_intel_bo_gem *
+__bo_alloc(drm_intel_bufmgr_gem *bufmgr_gem, unsigned long size)
+{
+	struct drm_i915_gem_create create;
+	drm_intel_bo_gem *bo_gem;
+	int ret;
+
+	bo_gem = calloc(1, sizeof(*bo_gem));
+	if (!bo_gem)
+		return NULL;
+
+	bo_gem->bo.size = size;
+
+	VG_CLEAR(create);
+	create.size = size;
+
+	ret = drmIoctl(bufmgr_gem->fd,
+		       DRM_IOCTL_I915_GEM_CREATE,
+		       &create);
+	bo_gem->gem_handle = create.handle;
+	bo_gem->bo.handle = bo_gem->gem_handle;
+	if (ret != 0) {
+		free(bo_gem);
+		return NULL;
+	}
+	bo_gem->bo.bufmgr = (drm_intel_bufmgr *)bufmgr_gem;
+
+	bo_gem->tiling_mode = I915_TILING_NONE;
+	bo_gem->swizzle_mode = I915_BIT_6_SWIZZLE_NONE;
+	bo_gem->stride = 0;
+
+	DRMINITLISTHEAD(&bo_gem->name_list);
+	DRMINITLISTHEAD(&bo_gem->vma_list);
+
+	return bo_gem;
+}
+
 static drm_intel_bo *
 drm_intel_gem_bo_alloc_internal(drm_intel_bufmgr *bufmgr,
 				const char *name,
@@ -647,7 +684,6 @@ drm_intel_gem_bo_alloc_internal(drm_intel_bufmgr *bufmgr,
 	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bufmgr;
 	drm_intel_bo_gem *bo_gem;
 	unsigned int page_size = getpagesize();
-	int ret;
 	struct drm_intel_gem_bo_bucket *bucket;
 	bool alloc_from_cache;
 	unsigned long bo_size;
@@ -720,41 +756,16 @@ retry:
 	pthread_mutex_unlock(&bufmgr_gem->lock);
 
 	if (!alloc_from_cache) {
-		struct drm_i915_gem_create create;
-
-		bo_gem = calloc(1, sizeof(*bo_gem));
+		bo_gem = __bo_alloc(bufmgr_gem, bo_size);
 		if (!bo_gem)
 			return NULL;
 
-		bo_gem->bo.size = bo_size;
-
-		VG_CLEAR(create);
-		create.size = bo_size;
-
-		ret = drmIoctl(bufmgr_gem->fd,
-			       DRM_IOCTL_I915_GEM_CREATE,
-			       &create);
-		bo_gem->gem_handle = create.handle;
-		bo_gem->bo.handle = bo_gem->gem_handle;
-		if (ret != 0) {
-			free(bo_gem);
-			return NULL;
-		}
-		bo_gem->bo.bufmgr = bufmgr;
-
-		bo_gem->tiling_mode = I915_TILING_NONE;
-		bo_gem->swizzle_mode = I915_BIT_6_SWIZZLE_NONE;
-		bo_gem->stride = 0;
-
 		if (drm_intel_gem_bo_set_tiling_internal(&bo_gem->bo,
 							 tiling_mode,
 							 stride)) {
-		    drm_intel_gem_bo_free(&bo_gem->bo);
-		    return NULL;
+			drm_intel_gem_bo_free(&bo_gem->bo);
+			return NULL;
 		}
-
-		DRMINITLISTHEAD(&bo_gem->name_list);
-		DRMINITLISTHEAD(&bo_gem->vma_list);
 	}
 
 	bo_gem->name = name;
-- 
2.0.4

[PATCH 2/2] intel: Add prelocation support

[Ben Widawsky]

Words

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 intel/intel_bufmgr.h     |   8 ++++
 intel/intel_bufmgr_gem.c | 102 +++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 102 insertions(+), 8 deletions(-)

diff --git a/intel/intel_bufmgr.h b/intel/intel_bufmgr.h
index 9383c72..e4ecc44 100644
--- a/intel/intel_bufmgr.h
+++ b/intel/intel_bufmgr.h
@@ -88,6 +88,8 @@ struct _drm_intel_bo {
 	 * Last seen card virtual address (offset from the beginning of the
 	 * aperture) for the object.  This should be used to fill relocation
 	 * entries when calling drm_intel_bo_emit_reloc()
+	 *
+	 * This is also useful when prelocating an object.
 	 */
 	uint64_t offset64;
 };
@@ -106,6 +108,8 @@ typedef struct _drm_intel_aub_annotation {
 } drm_intel_aub_annotation;
 
 #define BO_ALLOC_FOR_RENDER (1<<0)
+#define BO_ALLOC_PRELOCATE  (1<<1)
+#define BO_ALLOC_PRELOCATE_32  (1<<2)
 
 drm_intel_bo *drm_intel_bo_alloc(drm_intel_bufmgr *bufmgr, const char *name,
 				 unsigned long size, unsigned int alignment);
@@ -119,6 +123,10 @@ drm_intel_bo *drm_intel_bo_alloc_tiled(drm_intel_bufmgr *bufmgr,
 				       uint32_t *tiling_mode,
 				       unsigned long *pitch,
 				       unsigned long flags);
+drm_intel_bo *drm_intel_bo_alloc_prelocated(drm_intel_bufmgr *bufmgr,
+					    const char *name,
+					    unsigned long size,
+					    int low);
 void drm_intel_bo_reference(drm_intel_bo *bo);
 void drm_intel_bo_unreference(drm_intel_bo *bo);
 int drm_intel_bo_map(drm_intel_bo *bo, int write_enable);
diff --git a/intel/intel_bufmgr_gem.c b/intel/intel_bufmgr_gem.c
index d7d3769..5a2a9bd 100644
--- a/intel/intel_bufmgr_gem.c
+++ b/intel/intel_bufmgr_gem.c
@@ -221,6 +221,11 @@ struct _drm_intel_bo_gem {
 	 */
 	bool idle;
 
+	/** Fillme in */
+	#define PRELOCATE_MMAP 1
+	#define PRELOCATE_MALLOC 2
+	int prelocated;
+
 	/**
 	 * Size in bytes of this buffer and its relocation descendents.
 	 *
@@ -489,7 +494,10 @@ drm_intel_add_validate_buffer2(drm_intel_bo *bo, int need_fence)
 	bufmgr_gem->exec2_objects[index].relocation_count = bo_gem->reloc_count;
 	bufmgr_gem->exec2_objects[index].relocs_ptr = (uintptr_t)bo_gem->relocs;
 	bufmgr_gem->exec2_objects[index].alignment = 0;
-	bufmgr_gem->exec2_objects[index].offset = 0;
+	if (bo_gem->prelocated)
+		bufmgr_gem->exec2_objects[index].offset = bo->offset64;
+	else
+		bufmgr_gem->exec2_objects[index].offset = 0;
 	bufmgr_gem->exec_bos[index] = bo;
 	bufmgr_gem->exec2_objects[index].flags = 0;
 	bufmgr_gem->exec2_objects[index].rsvd1 = 0;
@@ -637,9 +645,10 @@ drm_intel_gem_bo_cache_purge_bucket(drm_intel_bufmgr_gem *bufmgr_gem,
 }
 
 static drm_intel_bo_gem *
-__bo_alloc(drm_intel_bufmgr_gem *bufmgr_gem, unsigned long size)
+__bo_alloc(drm_intel_bufmgr_gem *bufmgr_gem, unsigned long size, bool prelocate, bool low32)
 {
 	struct drm_i915_gem_create create;
+	drm_intel_bo *bo;
 	drm_intel_bo_gem *bo_gem;
 	int ret;
 
@@ -647,10 +656,35 @@ __bo_alloc(drm_intel_bufmgr_gem *bufmgr_gem, unsigned long size)
 	if (!bo_gem)
 		return NULL;
 
+	bo = (drm_intel_bo *)bo_gem;
+
 	bo_gem->bo.size = size;
 
 	VG_CLEAR(create);
 	create.size = size;
+	/* FIXME: This is a gross hack to repurpose the create args */
+	if (prelocate) {
+		create.size |= (1ULL << 63);
+		if (low32) {
+			bo->offset64 = (uint64_t)mmap(NULL, size,
+					PROT_READ | PROT_WRITE,
+					MAP_ANONYMOUS | MAP_PRIVATE | MAP_32BIT,
+					-1, 0);
+			bo_gem->prelocated = PRELOCATE_MALLOC;
+		} else {
+			bo->offset64 = (uint64_t)aligned_alloc(getpagesize(), size);
+			bo_gem->prelocated = PRELOCATE_MMAP;
+		}
+		if (!bo->offset64) {
+			DBG("Couldn't allocate %ld address space for object. %s\n",
+			    size, strerror(errno));
+			free(bo_gem);
+			return NULL;
+		}
+		create.handle = bo->offset64 >> 32;
+		create.pad = bo->offset64;
+	} else
+		bo->offset64 = 0x1;
 
 	ret = drmIoctl(bufmgr_gem->fd,
 		       DRM_IOCTL_I915_GEM_CREATE,
@@ -658,6 +692,10 @@ __bo_alloc(drm_intel_bufmgr_gem *bufmgr_gem, unsigned long size)
 	bo_gem->gem_handle = create.handle;
 	bo_gem->bo.handle = bo_gem->gem_handle;
 	if (ret != 0) {
+		if (prelocate && low32)
+			munmap((void *)bo->offset64, size);
+		else if (prelocate)
+			free((void *)bo->offset64);
 		free(bo_gem);
 		return NULL;
 	}
@@ -687,10 +725,17 @@ drm_intel_gem_bo_alloc_internal(drm_intel_bufmgr *bufmgr,
 	struct drm_intel_gem_bo_bucket *bucket;
 	bool alloc_from_cache;
 	unsigned long bo_size;
-	bool for_render = false;
+	bool for_render = false, prelocate = false, low = false;
 
 	if (flags & BO_ALLOC_FOR_RENDER)
 		for_render = true;
+	if (flags & BO_ALLOC_PRELOCATE) {
+		if (flags & BO_ALLOC_PRELOCATE_32)
+			low = true;
+		prelocate = true;
+		bo_size = size;
+		goto skip_cache;
+	}
 
 	/* Round the allocated size up to a power of two number of pages. */
 	bucket = drm_intel_gem_bo_bucket_for_size(bufmgr_gem, size);
@@ -756,7 +801,8 @@ retry:
 	pthread_mutex_unlock(&bufmgr_gem->lock);
 
 	if (!alloc_from_cache) {
-		bo_gem = __bo_alloc(bufmgr_gem, bo_size);
+skip_cache:
+		bo_gem = __bo_alloc(bufmgr_gem, bo_size, prelocate, low);
 		if (!bo_gem)
 			return NULL;
 
@@ -774,7 +820,7 @@ retry:
 	bo_gem->reloc_tree_fences = 0;
 	bo_gem->used_as_reloc_target = false;
 	bo_gem->has_error = false;
-	bo_gem->reusable = true;
+	bo_gem->reusable = !prelocate;
 	bo_gem->aub_annotations = NULL;
 	bo_gem->aub_annotation_count = 0;
 
@@ -859,6 +905,25 @@ drm_intel_gem_bo_alloc_tiled(drm_intel_bufmgr *bufmgr, const char *name,
 					       tiling, stride);
 }
 
+drm_public drm_intel_bo *
+drm_intel_bo_alloc_prelocated(drm_intel_bufmgr *bufmgr,
+			      const char *name,
+			      unsigned long size,
+			      int low)
+{
+	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bufmgr;
+	int flag = BO_ALLOC_PRELOCATE;
+	/* FIXME: Need to replace this with a paramcheck */
+	if (bufmgr_gem->gen < 8 || !bufmgr_gem->has_llc)
+		return NULL;
+
+	if (low)
+		flag |= BO_ALLOC_PRELOCATE_32;
+
+	return drm_intel_gem_bo_alloc_internal(bufmgr, name, size,
+					       flag, I915_TILING_NONE, 0);
+}
+
 /**
  * Returns a drm_intel_bo wrapping the given buffer object handle.
  *
@@ -964,7 +1029,7 @@ drm_intel_gem_bo_free(drm_intel_bo *bo)
 	int ret;
 
 	DRMLISTDEL(&bo_gem->vma_list);
-	if (bo_gem->mem_virtual) {
+	if (bo_gem->mem_virtual && !bo_gem->prelocated) {
 		VG(VALGRIND_FREELIKE_BLOCK(bo_gem->mem_virtual, 0));
 		munmap(bo_gem->mem_virtual, bo_gem->bo.size);
 		bufmgr_gem->vma_count--;
@@ -982,6 +1047,12 @@ drm_intel_gem_bo_free(drm_intel_bo *bo)
 		DBG("DRM_IOCTL_GEM_CLOSE %d failed (%s): %s\n",
 		    bo_gem->gem_handle, bo_gem->name, strerror(errno));
 	}
+
+	if (bo_gem->prelocated == PRELOCATE_MMAP)
+		munmap((void *)bo->offset64, bo->size);
+	else if (bo_gem->prelocated == PRELOCATE_MALLOC)
+		free((void *)bo->offset64);
+
 	free(bo_gem->aub_annotations);
 	free(bo);
 }
@@ -1190,7 +1261,9 @@ static int drm_intel_gem_bo_map(drm_intel_bo *bo, int write_enable)
 	if (bo_gem->map_count++ == 0)
 		drm_intel_gem_bo_open_vma(bufmgr_gem, bo_gem);
 
-	if (!bo_gem->mem_virtual) {
+	if (bo_gem->prelocated) {
+		bo_gem->mem_virtual = (void *)bo->offset64;
+	} else if (!bo_gem->mem_virtual) {
 		struct drm_i915_gem_mmap mmap_arg;
 
 		DBG("bo_map: %d (%s), map_count=%d\n",
@@ -1683,6 +1756,17 @@ do_bo_emit_reloc(drm_intel_bo *bo, uint32_t offset,
 		return -ENOMEM;
 	}
 
+	/* If the target we're trying point to was a prelocated target, then we
+	 * can skip actually telling the kernel about the relocation. Userspace
+	 * is expected to use offset64 */
+	if (target_bo_gem->prelocated) {
+		assert(target_bo->offset64 != 0x1);
+		assert(target_bo->offset64 != 0); // temp hack
+		if (bo_gem->validate_index == -1)
+			drm_intel_add_validate_buffer2(target_bo, false);
+		return 0;
+	}
+
 	/* We never use HW fences for rendering on 965+ */
 	if (bufmgr_gem->gen >= 4)
 		need_fence = false;
@@ -1863,7 +1947,6 @@ drm_intel_gem_bo_process_reloc2(drm_intel_bo *bo)
 	}
 }
 
-
 static void
 drm_intel_update_buffer_offsets(drm_intel_bufmgr_gem *bufmgr_gem)
 {
@@ -1894,6 +1977,9 @@ drm_intel_update_buffer_offsets2 (drm_intel_bufmgr_gem *bufmgr_gem)
 		drm_intel_bo *bo = bufmgr_gem->exec_bos[i];
 		drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *)bo;
 
+		if (bo_gem->prelocated)
+			continue;
+
 		/* Update the buffer offset */
 		if (bufmgr_gem->exec2_objects[i].offset != bo->offset64) {
 			DBG("BO %d (%s) migrated: 0x%08lx -> 0x%08llx\n",
-- 
2.0.4

[PATCH] i965: First step toward prelocation

[Ben Widawsky]

This was a quick proof of concept to show the new API for prelocating
buffers.

It needs way more testing, to not ifdef the no-relocs, and to do a
libdrm ABI dep bump.
---
 src/mesa/drivers/dri/i965/Makefile.am               | 1 +
 src/mesa/drivers/dri/i965/brw_performance_monitor.c | 6 +++---
 src/mesa/drivers/dri/i965/brw_program.c             | 5 +++--
 src/mesa/drivers/dri/i965/brw_queryobj.c            | 6 +++---
 src/mesa/drivers/dri/i965/brw_state_cache.c         | 4 ++--
 src/mesa/drivers/dri/i965/intel_batchbuffer.c       | 3 +++
 src/mesa/drivers/dri/i965/intel_batchbuffer.h       | 8 ++++++++
 7 files changed, 23 insertions(+), 10 deletions(-)

diff --git a/src/mesa/drivers/dri/i965/Makefile.am b/src/mesa/drivers/dri/i965/Makefile.am
index 5809dc6..4b20d36 100644
--- a/src/mesa/drivers/dri/i965/Makefile.am
+++ b/src/mesa/drivers/dri/i965/Makefile.am
@@ -24,6 +24,7 @@
 include Makefile.sources
 
 AM_CFLAGS = \
+        -DNO_RELOC \
 	-I$(top_srcdir)/include \
 	-I$(top_srcdir)/src/ \
 	-I$(top_srcdir)/src/mapi \
diff --git a/src/mesa/drivers/dri/i965/brw_performance_monitor.c b/src/mesa/drivers/dri/i965/brw_performance_monitor.c
index edfa3d2..e30c527 100644
--- a/src/mesa/drivers/dri/i965/brw_performance_monitor.c
+++ b/src/mesa/drivers/dri/i965/brw_performance_monitor.c
@@ -1105,13 +1105,13 @@ brw_begin_perf_monitor(struct gl_context *ctx,
        * wasting memory for contexts that don't use performance monitors.
        */
       if (!brw->perfmon.bookend_bo) {
-         brw->perfmon.bookend_bo = drm_intel_bo_alloc(brw->bufmgr,
+         brw->perfmon.bookend_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr,
                                                       "OA bookend BO",
                                                       BOOKEND_BO_SIZE_BYTES, 64);
       }
 
       monitor->oa_bo =
-         drm_intel_bo_alloc(brw->bufmgr, "perf. monitor OA bo", 4096, 64);
+         drm_intel_bo_alloc_wrapper(brw->bufmgr, "perf. monitor OA bo", 4096, 64);
 #ifdef DEBUG
       /* Pre-filling the BO helps debug whether writes landed. */
       drm_intel_bo_map(monitor->oa_bo, true);
@@ -1146,7 +1146,7 @@ brw_begin_perf_monitor(struct gl_context *ctx,
 
    if (monitor_needs_statistics_registers(brw, m)) {
       monitor->pipeline_stats_bo =
-         drm_intel_bo_alloc(brw->bufmgr, "perf. monitor stats bo", 4096, 64);
+         drm_intel_bo_alloc_wrapper(brw->bufmgr, "perf. monitor stats bo", 4096, 64);
 
       /* Take starting snapshots. */
       snapshot_statistics_registers(brw, monitor, 0);
diff --git a/src/mesa/drivers/dri/i965/brw_program.c b/src/mesa/drivers/dri/i965/brw_program.c
index d782b4f..74ff40c 100644
--- a/src/mesa/drivers/dri/i965/brw_program.c
+++ b/src/mesa/drivers/dri/i965/brw_program.c
@@ -43,6 +43,7 @@
 
 #include "brw_context.h"
 #include "brw_wm.h"
+#include "intel_batchbuffer.h"
 
 static unsigned
 get_new_program_id(struct intel_screen *screen)
@@ -242,7 +243,7 @@ brw_get_scratch_bo(struct brw_context *brw,
    }
 
    if (!old_bo) {
-      *scratch_bo = drm_intel_bo_alloc(brw->bufmgr, "scratch bo", size, 4096);
+      *scratch_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "scratch bo", size, 4096);
    }
 }
 
@@ -265,7 +266,7 @@ void
 brw_init_shader_time(struct brw_context *brw)
 {
    const int max_entries = 4096;
-   brw->shader_time.bo = drm_intel_bo_alloc(brw->bufmgr, "shader time",
+   brw->shader_time.bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "shader time",
                                             max_entries * SHADER_TIME_STRIDE,
                                             4096);
    brw->shader_time.shader_programs = rzalloc_array(brw, struct gl_shader_program *,
diff --git a/src/mesa/drivers/dri/i965/brw_queryobj.c b/src/mesa/drivers/dri/i965/brw_queryobj.c
index c053c34..cf5a2a5 100644
--- a/src/mesa/drivers/dri/i965/brw_queryobj.c
+++ b/src/mesa/drivers/dri/i965/brw_queryobj.c
@@ -230,7 +230,7 @@ brw_begin_query(struct gl_context *ctx, struct gl_query_object *q)
        * the system was doing other work, such as running other applications.
        */
       drm_intel_bo_unreference(query->bo);
-      query->bo = drm_intel_bo_alloc(brw->bufmgr, "timer query", 4096, 4096);
+      query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "timer query", 4096, 4096);
       brw_write_timestamp(brw, query->bo, 0);
       break;
 
@@ -388,7 +388,7 @@ ensure_bo_has_space(struct gl_context *ctx, struct brw_query_object *query)
          brw_queryobj_get_results(ctx, query);
       }
 
-      query->bo = drm_intel_bo_alloc(brw->bufmgr, "query", 4096, 1);
+      query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "query", 4096, 1);
       query->last_index = 0;
    }
 }
@@ -474,7 +474,7 @@ brw_query_counter(struct gl_context *ctx, struct gl_query_object *q)
    assert(q->Target == GL_TIMESTAMP);
 
    drm_intel_bo_unreference(query->bo);
-   query->bo = drm_intel_bo_alloc(brw->bufmgr, "timestamp query", 4096, 4096);
+   query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "timestamp query", 4096, 4096);
    brw_write_timestamp(brw, query->bo, 0);
 }
 
diff --git a/src/mesa/drivers/dri/i965/brw_state_cache.c b/src/mesa/drivers/dri/i965/brw_state_cache.c
index b0986ea..daf5a11 100644
--- a/src/mesa/drivers/dri/i965/brw_state_cache.c
+++ b/src/mesa/drivers/dri/i965/brw_state_cache.c
@@ -171,7 +171,7 @@ brw_cache_new_bo(struct brw_cache *cache, uint32_t new_size)
    struct brw_context *brw = cache->brw;
    drm_intel_bo *new_bo;
 
-   new_bo = drm_intel_bo_alloc(brw->bufmgr, "program cache", new_size, 64);
+   new_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "program cache", new_size, 64);
 
    /* Copy any existing data that needs to be saved. */
    if (cache->next_offset != 0) {
@@ -335,7 +335,7 @@ brw_init_caches(struct brw_context *brw)
    cache->items =
       calloc(1, cache->size * sizeof(struct brw_cache_item *));
 
-   cache->bo = drm_intel_bo_alloc(brw->bufmgr,
+   cache->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr,
 				  "program cache",
 				  4096, 64);
 
diff --git a/src/mesa/drivers/dri/i965/intel_batchbuffer.c b/src/mesa/drivers/dri/i965/intel_batchbuffer.c
index 71dc268..50834c2 100644
--- a/src/mesa/drivers/dri/i965/intel_batchbuffer.c
+++ b/src/mesa/drivers/dri/i965/intel_batchbuffer.c
@@ -253,6 +253,9 @@ do_flush_locked(struct brw_context *brw)
 
    if (!brw->intelScreen->no_hw) {
       int flags;
+#ifdef NO_RELOC
+      flags |= I915_EXEC_NO_RELOC;
+#endif
 
       if (brw->gen >= 6 && batch->ring == BLT_RING) {
          flags = I915_EXEC_BLT;
diff --git a/src/mesa/drivers/dri/i965/intel_batchbuffer.h b/src/mesa/drivers/dri/i965/intel_batchbuffer.h
index 7bdd836..2670d22 100644
--- a/src/mesa/drivers/dri/i965/intel_batchbuffer.h
+++ b/src/mesa/drivers/dri/i965/intel_batchbuffer.h
@@ -11,6 +11,14 @@
 extern "C" {
 #endif
 
+#ifdef NO_RELOC
+ #define drm_intel_bo_alloc_wrapper(bufmgr, name, size, align) \
+   drm_intel_bo_alloc_prelocated(bufmgr, name, size, 1)
+#else
+ #define drm_intel_bo_alloc_wrapper drm_intel_bo_alloc
+#endif
+
+
 /**
  * Number of bytes to reserve for commands necessary to complete a batch.
  *
-- 
2.0.4

[PATCH] no_reloc: test case

[Ben Widawsky]

needs libdrm
---
 tests/Makefile.am        |   4 +-
 tests/Makefile.sources   |   1 +
 tests/gem_exec_noreloc.c | 172 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 175 insertions(+), 2 deletions(-)
 create mode 100644 tests/gem_exec_noreloc.c

diff --git a/tests/Makefile.am b/tests/Makefile.am
index a2fba51..5c06ab3 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -27,8 +27,8 @@ multi-tests.txt: Makefile.sources
 	@echo ${multi_kernel_tests} >> $@
 	@echo END TESTLIST >> $@
 
-EXTRA_PROGRAMS = $(TESTS_progs) $(TESTS_progs_M) $(HANG) $(TESTS_testsuite)
-EXTRA_DIST = $(TESTS_scripts) $(TESTS_scripts_M) $(scripts) $(IMAGES) $(common_files)
+bin_PROGRAMS = $(TESTS_progs) $(TESTS_progs_M) $(HANG) $(TESTS_testsuite)
+dist_bin_SCRIPTS = $(TESTS_scripts) $(TESTS_scripts_M) $(scripts) $(IMAGES) $(common_files)
 
 CLEANFILES = $(EXTRA_PROGRAMS) single-tests.txt multi-tests.txt
 
diff --git a/tests/Makefile.sources b/tests/Makefile.sources
index 698e290..cbc3bb9 100644
--- a/tests/Makefile.sources
+++ b/tests/Makefile.sources
@@ -31,6 +31,7 @@ TESTS_progs_M = \
 	gem_evict_everything \
 	gem_exec_bad_domains \
 	gem_exec_faulting_reloc \
+	gem_exec_noreloc \
 	gem_exec_nop \
 	gem_exec_params \
 	gem_exec_parse \
diff --git a/tests/gem_exec_noreloc.c b/tests/gem_exec_noreloc.c
new file mode 100644
index 0000000..3552246
--- /dev/null
+++ b/tests/gem_exec_noreloc.c
@@ -0,0 +1,172 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <limits.h>
+#include <signal.h>
+#include <errno.h>
+#include <drm.h>
+
+#include "ioctl_wrappers.h"
+#include "drmtest.h"
+#include "igt_core.h"
+#include "igt_aux.h"
+#include "intel_bufmgr.h"
+#include "intel_batchbuffer.h"
+#include "intel_io.h"
+#include "intel_chipset.h"
+
+#define PATTERN 0xdefaca7e
+
+int fd;
+static struct intel_batchbuffer *batch_3d;
+
+static void
+set_bo(drm_intel_bo *bo, uint32_t val, int width, int height)
+{
+	int size = width * height;
+	uint32_t *vaddr;
+
+	drm_intel_bo_map(bo, true);
+	vaddr = bo->virtual;
+	while (size--)
+		*vaddr++ = val;
+	drm_intel_bo_unmap(bo);
+}
+
+static void
+cmp_bo(drm_intel_bo *bo, uint32_t val, int width, int height)
+{
+	int size = width * height;
+	uint32_t *vaddr;
+
+	drm_intel_bo_map(bo, false);
+	vaddr = bo->virtual;
+	while (size--)
+		igt_assert(*vaddr++ == val);
+	drm_intel_bo_unmap(bo);
+}
+
+
+static void init_buffer(drm_intel_bufmgr *bufmgr,
+                        struct igt_buf *buf,
+                        drm_intel_bo *bo,
+                        int width, int height)
+{
+	buf->bo = bo;
+	buf->size = width * height * 4;
+	igt_assert(buf->bo);
+	buf->tiling = I915_TILING_NONE;
+	buf->num_tiles = width * height * 4;
+	buf->stride = width * 4;
+}
+
+static drm_intel_bo *
+create_bo(drm_intel_bufmgr *bufmgr, int width, int height)
+{
+	drm_intel_bo *bo;
+
+	bo = drm_intel_bo_alloc_prelocated(bufmgr, "soft bo", width * height * 4);
+	igt_assert(bo);
+
+	set_bo(bo, PATTERN, width, height);
+
+	return bo;
+}
+
+static void release_bo(drm_intel_bo *bo)
+{
+	drm_intel_gem_bo_unmap_gtt(bo);
+	drm_intel_bo_unreference(bo);
+}
+
+static void render_copyfunc(struct igt_buf *src,
+			    struct igt_buf *dst,
+			    int width,
+			    int height)
+{
+	const int src_x = 0, src_y = 0, dst_x = 0, dst_y = 0;
+	igt_render_copyfunc_t rendercopy = igt_get_render_copyfunc(intel_get_drm_devid(fd));
+
+	igt_assert(rendercopy);
+
+	if (rendercopy) {
+		rendercopy(batch_3d, NULL,
+			   src, src_x, src_y,
+			   width, height,
+			   dst, dst_x, dst_y);
+		intel_batchbuffer_flush(batch_3d);
+	}
+}
+
+static struct intel_batchbuffer *
+noreloc_batchbuffer_alloc(drm_intel_bufmgr *bufmgr, uint32_t devid)
+{
+	struct intel_batchbuffer *batch = calloc(sizeof(*batch), 1);
+	igt_assert(batch);
+
+	batch->bufmgr = bufmgr;
+	batch->devid = devid;
+	batch->bo = drm_intel_bo_alloc_prelocated(bufmgr, "batchbuffer", 4096);
+	igt_assert(batch->bo);
+	memset(batch->buffer, 0, sizeof(batch->buffer));
+	batch->ptr = batch->buffer;
+
+	return batch;
+}
+
+int main(int argc, char **argv)
+{
+	struct igt_buf src, dest;
+	drm_intel_bo *from, *to;
+	drm_intel_bufmgr *bufmgr;
+
+	fd = drm_open_any();
+	igt_assert(fd >= 0);
+
+	bufmgr = drm_intel_bufmgr_gem_init(fd, 4096);
+	batch_3d = noreloc_batchbuffer_alloc(bufmgr, intel_get_drm_devid(fd));
+	igt_assert(batch_3d);
+
+	from = create_bo(bufmgr, 4096, 4096);
+	to = create_bo(bufmgr, 4096, 4096);
+	igt_assert(from);
+	igt_assert(to);
+	init_buffer(bufmgr, &src, from, 4096, 4096);
+	init_buffer(bufmgr, &dest,  to, 4096, 4096);
+
+	render_copyfunc(&src, &dest, 4096, 4096);
+	cmp_bo(to, PATTERN, 4096, 4096);
+
+	release_bo(to);
+	release_bo(from);
+
+	return 0;
+}
-- 
2.0.4

Re: [Intel-gfx] [PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Chris Wilson]

On Thu, Aug 21, 2014 at 08:11:23PM -0700, Ben Widawsky wrote:
> The primary goal of these patches is to introduce what I've started
> calling, "prelocations" on Broadwell. A prelocation is like a
> relocation, except not. When a GPU client specifies a prelocation, it is
> instructing the kernel where in the GPU address the buffer should be
> mapped. The mechanic works very similarly to a relocation except it uses
> the execbuffer object to obtain the offset, and bind if needed.

You are mixing two APIs. One to preallocate an offset at creation
and one to presume relocations during execbuffer. I'd much rather keep
the flexible execbuffer approach outlined and first submitted a couple of
years ago.

> If a GPU
> client uses only prelocations, the relocation process can be entirely
> skipped. This sounds like a big win initially,

Close to zero if the client uses existing interfaces.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: [PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Kenneth Graunke]

[-- Attachment #1.1: Type: text/plain, Size: 1955 bytes --]

On Friday, August 22, 2014 07:30:37 AM Chris Wilson wrote:
> On Thu, Aug 21, 2014 at 08:11:23PM -0700, Ben Widawsky wrote:
> > The primary goal of these patches is to introduce what I've started
> > calling, "prelocations" on Broadwell. A prelocation is like a
> > relocation, except not. When a GPU client specifies a prelocation, it is
> > instructing the kernel where in the GPU address the buffer should be
> > mapped. The mechanic works very similarly to a relocation except it uses
> > the execbuffer object to obtain the offset, and bind if needed.
> 
> You are mixing two APIs. One to preallocate an offset at creation
> and one to presume relocations during execbuffer. I'd much rather keep
> the flexible execbuffer approach outlined and first submitted a couple of
> years ago.
> 
> > If a GPU
> > client uses only prelocations, the relocation process can be entirely
> > skipped. This sounds like a big win initially,
> 
> Close to zero if the client uses existing interfaces.
> -Chris

Chris,

I don't know if you've seen Ben's libdrm and Mesa patches, but with a few patches to libdrm and virtually zero Mesa changes, he's apparently eliminated our need to do any relocations for the 3D driver.  It wasn't invasive at all---I was surprised.

With both the CPU and GPU using 48-bit addressing, using the same virtual address on both sides and never changing it seems quite appealing.  I'm not sure why we would need to do anything different than that.

As I understand it, we still need to let the kernel know what buffers we need pinned during the course of the batchbuffer, since we can't take a page fault and fetch them as needed.  Reusing the existing relocation list but just not doing relocations seems like a simple way to do that without having to invent much new API...

What is the 'flexible execbuffer' approach you mention from a few years back?  I don't remember hearing about it (sorry)...

--Ken

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Chris Wilson]

On Thu, Aug 21, 2014 at 11:59:04PM -0700, Kenneth Graunke wrote:
> On Friday, August 22, 2014 07:30:37 AM Chris Wilson wrote:
> > On Thu, Aug 21, 2014 at 08:11:23PM -0700, Ben Widawsky wrote:
> > > The primary goal of these patches is to introduce what I've started
> > > calling, "prelocations" on Broadwell. A prelocation is like a
> > > relocation, except not. When a GPU client specifies a prelocation, it is
> > > instructing the kernel where in the GPU address the buffer should be
> > > mapped. The mechanic works very similarly to a relocation except it uses
> > > the execbuffer object to obtain the offset, and bind if needed.
> > 
> > You are mixing two APIs. One to preallocate an offset at creation
> > and one to presume relocations during execbuffer. I'd much rather keep
> > the flexible execbuffer approach outlined and first submitted a couple of
> > years ago.
> > 
> > > If a GPU
> > > client uses only prelocations, the relocation process can be entirely
> > > skipped. This sounds like a big win initially,
> > 
> > Close to zero if the client uses existing interfaces.
> > -Chris
> 
> Chris,
> 
> I don't know if you've seen Ben's libdrm and Mesa patches, but with a few patches to libdrm and virtually zero Mesa changes, he's apparently eliminated our need to do any relocations for the 3D driver.  It wasn't invasive at all---I was surprised.

Indeed, you could do everything inside libdrm with the code I posted 2
years ago.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: [Mesa-dev] [PATCH] i965: First step toward prelocation

[Alex Deucher]

On Thu, Aug 21, 2014 at 11:12 PM, Ben Widawsky
<benjamin.widawsky@intel.com> wrote:
> This was a quick proof of concept to show the new API for prelocating
> buffers.
>

What are prelocated buffers?

Alex

> It needs way more testing, to not ifdef the no-relocs, and to do a
> libdrm ABI dep bump.
> ---
>  src/mesa/drivers/dri/i965/Makefile.am               | 1 +
>  src/mesa/drivers/dri/i965/brw_performance_monitor.c | 6 +++---
>  src/mesa/drivers/dri/i965/brw_program.c             | 5 +++--
>  src/mesa/drivers/dri/i965/brw_queryobj.c            | 6 +++---
>  src/mesa/drivers/dri/i965/brw_state_cache.c         | 4 ++--
>  src/mesa/drivers/dri/i965/intel_batchbuffer.c       | 3 +++
>  src/mesa/drivers/dri/i965/intel_batchbuffer.h       | 8 ++++++++
>  7 files changed, 23 insertions(+), 10 deletions(-)
>
> diff --git a/src/mesa/drivers/dri/i965/Makefile.am b/src/mesa/drivers/dri/i965/Makefile.am
> index 5809dc6..4b20d36 100644
> --- a/src/mesa/drivers/dri/i965/Makefile.am
> +++ b/src/mesa/drivers/dri/i965/Makefile.am
> @@ -24,6 +24,7 @@
>  include Makefile.sources
>
>  AM_CFLAGS = \
> +        -DNO_RELOC \
>         -I$(top_srcdir)/include \
>         -I$(top_srcdir)/src/ \
>         -I$(top_srcdir)/src/mapi \
> diff --git a/src/mesa/drivers/dri/i965/brw_performance_monitor.c b/src/mesa/drivers/dri/i965/brw_performance_monitor.c
> index edfa3d2..e30c527 100644
> --- a/src/mesa/drivers/dri/i965/brw_performance_monitor.c
> +++ b/src/mesa/drivers/dri/i965/brw_performance_monitor.c
> @@ -1105,13 +1105,13 @@ brw_begin_perf_monitor(struct gl_context *ctx,
>         * wasting memory for contexts that don't use performance monitors.
>         */
>        if (!brw->perfmon.bookend_bo) {
> -         brw->perfmon.bookend_bo = drm_intel_bo_alloc(brw->bufmgr,
> +         brw->perfmon.bookend_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr,
>                                                        "OA bookend BO",
>                                                        BOOKEND_BO_SIZE_BYTES, 64);
>        }
>
>        monitor->oa_bo =
> -         drm_intel_bo_alloc(brw->bufmgr, "perf. monitor OA bo", 4096, 64);
> +         drm_intel_bo_alloc_wrapper(brw->bufmgr, "perf. monitor OA bo", 4096, 64);
>  #ifdef DEBUG
>        /* Pre-filling the BO helps debug whether writes landed. */
>        drm_intel_bo_map(monitor->oa_bo, true);
> @@ -1146,7 +1146,7 @@ brw_begin_perf_monitor(struct gl_context *ctx,
>
>     if (monitor_needs_statistics_registers(brw, m)) {
>        monitor->pipeline_stats_bo =
> -         drm_intel_bo_alloc(brw->bufmgr, "perf. monitor stats bo", 4096, 64);
> +         drm_intel_bo_alloc_wrapper(brw->bufmgr, "perf. monitor stats bo", 4096, 64);
>
>        /* Take starting snapshots. */
>        snapshot_statistics_registers(brw, monitor, 0);
> diff --git a/src/mesa/drivers/dri/i965/brw_program.c b/src/mesa/drivers/dri/i965/brw_program.c
> index d782b4f..74ff40c 100644
> --- a/src/mesa/drivers/dri/i965/brw_program.c
> +++ b/src/mesa/drivers/dri/i965/brw_program.c
> @@ -43,6 +43,7 @@
>
>  #include "brw_context.h"
>  #include "brw_wm.h"
> +#include "intel_batchbuffer.h"
>
>  static unsigned
>  get_new_program_id(struct intel_screen *screen)
> @@ -242,7 +243,7 @@ brw_get_scratch_bo(struct brw_context *brw,
>     }
>
>     if (!old_bo) {
> -      *scratch_bo = drm_intel_bo_alloc(brw->bufmgr, "scratch bo", size, 4096);
> +      *scratch_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "scratch bo", size, 4096);
>     }
>  }
>
> @@ -265,7 +266,7 @@ void
>  brw_init_shader_time(struct brw_context *brw)
>  {
>     const int max_entries = 4096;
> -   brw->shader_time.bo = drm_intel_bo_alloc(brw->bufmgr, "shader time",
> +   brw->shader_time.bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "shader time",
>                                              max_entries * SHADER_TIME_STRIDE,
>                                              4096);
>     brw->shader_time.shader_programs = rzalloc_array(brw, struct gl_shader_program *,
> diff --git a/src/mesa/drivers/dri/i965/brw_queryobj.c b/src/mesa/drivers/dri/i965/brw_queryobj.c
> index c053c34..cf5a2a5 100644
> --- a/src/mesa/drivers/dri/i965/brw_queryobj.c
> +++ b/src/mesa/drivers/dri/i965/brw_queryobj.c
> @@ -230,7 +230,7 @@ brw_begin_query(struct gl_context *ctx, struct gl_query_object *q)
>         * the system was doing other work, such as running other applications.
>         */
>        drm_intel_bo_unreference(query->bo);
> -      query->bo = drm_intel_bo_alloc(brw->bufmgr, "timer query", 4096, 4096);
> +      query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "timer query", 4096, 4096);
>        brw_write_timestamp(brw, query->bo, 0);
>        break;
>
> @@ -388,7 +388,7 @@ ensure_bo_has_space(struct gl_context *ctx, struct brw_query_object *query)
>           brw_queryobj_get_results(ctx, query);
>        }
>
> -      query->bo = drm_intel_bo_alloc(brw->bufmgr, "query", 4096, 1);
> +      query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "query", 4096, 1);
>        query->last_index = 0;
>     }
>  }
> @@ -474,7 +474,7 @@ brw_query_counter(struct gl_context *ctx, struct gl_query_object *q)
>     assert(q->Target == GL_TIMESTAMP);
>
>     drm_intel_bo_unreference(query->bo);
> -   query->bo = drm_intel_bo_alloc(brw->bufmgr, "timestamp query", 4096, 4096);
> +   query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "timestamp query", 4096, 4096);
>     brw_write_timestamp(brw, query->bo, 0);
>  }
>
> diff --git a/src/mesa/drivers/dri/i965/brw_state_cache.c b/src/mesa/drivers/dri/i965/brw_state_cache.c
> index b0986ea..daf5a11 100644
> --- a/src/mesa/drivers/dri/i965/brw_state_cache.c
> +++ b/src/mesa/drivers/dri/i965/brw_state_cache.c
> @@ -171,7 +171,7 @@ brw_cache_new_bo(struct brw_cache *cache, uint32_t new_size)
>     struct brw_context *brw = cache->brw;
>     drm_intel_bo *new_bo;
>
> -   new_bo = drm_intel_bo_alloc(brw->bufmgr, "program cache", new_size, 64);
> +   new_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "program cache", new_size, 64);
>
>     /* Copy any existing data that needs to be saved. */
>     if (cache->next_offset != 0) {
> @@ -335,7 +335,7 @@ brw_init_caches(struct brw_context *brw)
>     cache->items =
>        calloc(1, cache->size * sizeof(struct brw_cache_item *));
>
> -   cache->bo = drm_intel_bo_alloc(brw->bufmgr,
> +   cache->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr,
>                                   "program cache",
>                                   4096, 64);
>
> diff --git a/src/mesa/drivers/dri/i965/intel_batchbuffer.c b/src/mesa/drivers/dri/i965/intel_batchbuffer.c
> index 71dc268..50834c2 100644
> --- a/src/mesa/drivers/dri/i965/intel_batchbuffer.c
> +++ b/src/mesa/drivers/dri/i965/intel_batchbuffer.c
> @@ -253,6 +253,9 @@ do_flush_locked(struct brw_context *brw)
>
>     if (!brw->intelScreen->no_hw) {
>        int flags;
> +#ifdef NO_RELOC
> +      flags |= I915_EXEC_NO_RELOC;
> +#endif
>
>        if (brw->gen >= 6 && batch->ring == BLT_RING) {
>           flags = I915_EXEC_BLT;
> diff --git a/src/mesa/drivers/dri/i965/intel_batchbuffer.h b/src/mesa/drivers/dri/i965/intel_batchbuffer.h
> index 7bdd836..2670d22 100644
> --- a/src/mesa/drivers/dri/i965/intel_batchbuffer.h
> +++ b/src/mesa/drivers/dri/i965/intel_batchbuffer.h
> @@ -11,6 +11,14 @@
>  extern "C" {
>  #endif
>
> +#ifdef NO_RELOC
> + #define drm_intel_bo_alloc_wrapper(bufmgr, name, size, align) \
> +   drm_intel_bo_alloc_prelocated(bufmgr, name, size, 1)
> +#else
> + #define drm_intel_bo_alloc_wrapper drm_intel_bo_alloc
> +#endif
> +
> +
>  /**
>   * Number of bytes to reserve for commands necessary to complete a batch.
>   *
> --
> 2.0.4
>
> _______________________________________________
> mesa-dev mailing list
> mesa-dev@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/mesa-dev

Re: [PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Daniel Vetter]

On Fri, Aug 22, 2014 at 9:03 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
>> > > If a GPU
>> > > client uses only prelocations, the relocation process can be entirely
>> > > skipped. This sounds like a big win initially,
>> >
>> > Close to zero if the client uses existing interfaces.
>> > -Chris
>>
>> Chris,
>>
>> I don't know if you've seen Ben's libdrm and Mesa patches, but with a few patches to libdrm and virtually zero Mesa changes, he's apparently eliminated our need to do any relocations for the 3D driver.  It wasn't invasive at all---I was surprised.
>
> Indeed, you could do everything inside libdrm with the code I posted 2
> years ago.

I915_EXEC_NO_RELOC can be used to tell the kernel that it doesn't need
to walk all the reloc tables (if nothing moved) because userspace
didn't go insane and reuse reloc trees. So you'd need to implement a
flag + a libdrm function to set that (iirc mesa has been non-stupid
since years). And yeah I kinda expect any new reloc-less thing to get
benchmarked against an implementation using that, since the 48bit
specific thing proposed looks like a fairly short-lived stop-gap, and
since the current no-reloc we already have would work everywhere. And
yeah I've been poking people to look at this for years. too.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Intel-gfx] [PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Chris Wilson]

On Fri, Aug 22, 2014 at 03:30:12PM +0200, Daniel Vetter wrote:
> On Fri, Aug 22, 2014 at 9:03 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> >> > > If a GPU
> >> > > client uses only prelocations, the relocation process can be entirely
> >> > > skipped. This sounds like a big win initially,
> >> >
> >> > Close to zero if the client uses existing interfaces.
> >> > -Chris
> >>
> >> Chris,
> >>
> >> I don't know if you've seen Ben's libdrm and Mesa patches, but with a few patches to libdrm and virtually zero Mesa changes, he's apparently eliminated our need to do any relocations for the 3D driver.  It wasn't invasive at all---I was surprised.
> >
> > Indeed, you could do everything inside libdrm with the code I posted 2
> > years ago.
> 
> I915_EXEC_NO_RELOC can be used to tell the kernel that it doesn't need
> to walk all the reloc tables (if nothing moved) because userspace
> didn't go insane and reuse reloc trees. So you'd need to implement a
> flag + a libdrm function to set that (iirc mesa has been non-stupid
> since years). And yeah I kinda expect any new reloc-less thing to get
> benchmarked against an implementation using that, since the 48bit
> specific thing proposed looks like a fairly short-lived stop-gap, and
> since the current no-reloc we already have would work everywhere. And
> yeah I've been poking people to look at this for years. too.

Here, I was referring to soft-pinning. The API here is essentially
comprised of two parts:

1: a pin into the vm upon creation
2: implicit no-relocation upon execbuffer

By making those two steps independent, the API as I see is, is more
flexible and powerful.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: [PATCH] i965: First step toward prelocation

[Ben Widawsky]

On Fri, Aug 22, 2014 at 08:15:28AM -0400, Alex Deucher wrote:
> On Thu, Aug 21, 2014 at 11:12 PM, Ben Widawsky
> <benjamin.widawsky@intel.com> wrote:
> > This was a quick proof of concept to show the new API for prelocating
> > buffers.
> >
> 
> What are prelocated buffers?

http://lists.freedesktop.org/archives/mesa-dev/2014-August/066432.html

> 
> Alex
> 
> > It needs way more testing, to not ifdef the no-relocs, and to do a
> > libdrm ABI dep bump.
> > ---
> >  src/mesa/drivers/dri/i965/Makefile.am               | 1 +
> >  src/mesa/drivers/dri/i965/brw_performance_monitor.c | 6 +++---
> >  src/mesa/drivers/dri/i965/brw_program.c             | 5 +++--
> >  src/mesa/drivers/dri/i965/brw_queryobj.c            | 6 +++---
> >  src/mesa/drivers/dri/i965/brw_state_cache.c         | 4 ++--
> >  src/mesa/drivers/dri/i965/intel_batchbuffer.c       | 3 +++
> >  src/mesa/drivers/dri/i965/intel_batchbuffer.h       | 8 ++++++++
> >  7 files changed, 23 insertions(+), 10 deletions(-)
> >
> > diff --git a/src/mesa/drivers/dri/i965/Makefile.am b/src/mesa/drivers/dri/i965/Makefile.am
> > index 5809dc6..4b20d36 100644
> > --- a/src/mesa/drivers/dri/i965/Makefile.am
> > +++ b/src/mesa/drivers/dri/i965/Makefile.am
> > @@ -24,6 +24,7 @@
> >  include Makefile.sources
> >
> >  AM_CFLAGS = \
> > +        -DNO_RELOC \
> >         -I$(top_srcdir)/include \
> >         -I$(top_srcdir)/src/ \
> >         -I$(top_srcdir)/src/mapi \
> > diff --git a/src/mesa/drivers/dri/i965/brw_performance_monitor.c b/src/mesa/drivers/dri/i965/brw_performance_monitor.c
> > index edfa3d2..e30c527 100644
> > --- a/src/mesa/drivers/dri/i965/brw_performance_monitor.c
> > +++ b/src/mesa/drivers/dri/i965/brw_performance_monitor.c
> > @@ -1105,13 +1105,13 @@ brw_begin_perf_monitor(struct gl_context *ctx,
> >         * wasting memory for contexts that don't use performance monitors.
> >         */
> >        if (!brw->perfmon.bookend_bo) {
> > -         brw->perfmon.bookend_bo = drm_intel_bo_alloc(brw->bufmgr,
> > +         brw->perfmon.bookend_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr,
> >                                                        "OA bookend BO",
> >                                                        BOOKEND_BO_SIZE_BYTES, 64);
> >        }
> >
> >        monitor->oa_bo =
> > -         drm_intel_bo_alloc(brw->bufmgr, "perf. monitor OA bo", 4096, 64);
> > +         drm_intel_bo_alloc_wrapper(brw->bufmgr, "perf. monitor OA bo", 4096, 64);
> >  #ifdef DEBUG
> >        /* Pre-filling the BO helps debug whether writes landed. */
> >        drm_intel_bo_map(monitor->oa_bo, true);
> > @@ -1146,7 +1146,7 @@ brw_begin_perf_monitor(struct gl_context *ctx,
> >
> >     if (monitor_needs_statistics_registers(brw, m)) {
> >        monitor->pipeline_stats_bo =
> > -         drm_intel_bo_alloc(brw->bufmgr, "perf. monitor stats bo", 4096, 64);
> > +         drm_intel_bo_alloc_wrapper(brw->bufmgr, "perf. monitor stats bo", 4096, 64);
> >
> >        /* Take starting snapshots. */
> >        snapshot_statistics_registers(brw, monitor, 0);
> > diff --git a/src/mesa/drivers/dri/i965/brw_program.c b/src/mesa/drivers/dri/i965/brw_program.c
> > index d782b4f..74ff40c 100644
> > --- a/src/mesa/drivers/dri/i965/brw_program.c
> > +++ b/src/mesa/drivers/dri/i965/brw_program.c
> > @@ -43,6 +43,7 @@
> >
> >  #include "brw_context.h"
> >  #include "brw_wm.h"
> > +#include "intel_batchbuffer.h"
> >
> >  static unsigned
> >  get_new_program_id(struct intel_screen *screen)
> > @@ -242,7 +243,7 @@ brw_get_scratch_bo(struct brw_context *brw,
> >     }
> >
> >     if (!old_bo) {
> > -      *scratch_bo = drm_intel_bo_alloc(brw->bufmgr, "scratch bo", size, 4096);
> > +      *scratch_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "scratch bo", size, 4096);
> >     }
> >  }
> >
> > @@ -265,7 +266,7 @@ void
> >  brw_init_shader_time(struct brw_context *brw)
> >  {
> >     const int max_entries = 4096;
> > -   brw->shader_time.bo = drm_intel_bo_alloc(brw->bufmgr, "shader time",
> > +   brw->shader_time.bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "shader time",
> >                                              max_entries * SHADER_TIME_STRIDE,
> >                                              4096);
> >     brw->shader_time.shader_programs = rzalloc_array(brw, struct gl_shader_program *,
> > diff --git a/src/mesa/drivers/dri/i965/brw_queryobj.c b/src/mesa/drivers/dri/i965/brw_queryobj.c
> > index c053c34..cf5a2a5 100644
> > --- a/src/mesa/drivers/dri/i965/brw_queryobj.c
> > +++ b/src/mesa/drivers/dri/i965/brw_queryobj.c
> > @@ -230,7 +230,7 @@ brw_begin_query(struct gl_context *ctx, struct gl_query_object *q)
> >         * the system was doing other work, such as running other applications.
> >         */
> >        drm_intel_bo_unreference(query->bo);
> > -      query->bo = drm_intel_bo_alloc(brw->bufmgr, "timer query", 4096, 4096);
> > +      query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "timer query", 4096, 4096);
> >        brw_write_timestamp(brw, query->bo, 0);
> >        break;
> >
> > @@ -388,7 +388,7 @@ ensure_bo_has_space(struct gl_context *ctx, struct brw_query_object *query)
> >           brw_queryobj_get_results(ctx, query);
> >        }
> >
> > -      query->bo = drm_intel_bo_alloc(brw->bufmgr, "query", 4096, 1);
> > +      query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "query", 4096, 1);
> >        query->last_index = 0;
> >     }
> >  }
> > @@ -474,7 +474,7 @@ brw_query_counter(struct gl_context *ctx, struct gl_query_object *q)
> >     assert(q->Target == GL_TIMESTAMP);
> >
> >     drm_intel_bo_unreference(query->bo);
> > -   query->bo = drm_intel_bo_alloc(brw->bufmgr, "timestamp query", 4096, 4096);
> > +   query->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "timestamp query", 4096, 4096);
> >     brw_write_timestamp(brw, query->bo, 0);
> >  }
> >
> > diff --git a/src/mesa/drivers/dri/i965/brw_state_cache.c b/src/mesa/drivers/dri/i965/brw_state_cache.c
> > index b0986ea..daf5a11 100644
> > --- a/src/mesa/drivers/dri/i965/brw_state_cache.c
> > +++ b/src/mesa/drivers/dri/i965/brw_state_cache.c
> > @@ -171,7 +171,7 @@ brw_cache_new_bo(struct brw_cache *cache, uint32_t new_size)
> >     struct brw_context *brw = cache->brw;
> >     drm_intel_bo *new_bo;
> >
> > -   new_bo = drm_intel_bo_alloc(brw->bufmgr, "program cache", new_size, 64);
> > +   new_bo = drm_intel_bo_alloc_wrapper(brw->bufmgr, "program cache", new_size, 64);
> >
> >     /* Copy any existing data that needs to be saved. */
> >     if (cache->next_offset != 0) {
> > @@ -335,7 +335,7 @@ brw_init_caches(struct brw_context *brw)
> >     cache->items =
> >        calloc(1, cache->size * sizeof(struct brw_cache_item *));
> >
> > -   cache->bo = drm_intel_bo_alloc(brw->bufmgr,
> > +   cache->bo = drm_intel_bo_alloc_wrapper(brw->bufmgr,
> >                                   "program cache",
> >                                   4096, 64);
> >
> > diff --git a/src/mesa/drivers/dri/i965/intel_batchbuffer.c b/src/mesa/drivers/dri/i965/intel_batchbuffer.c
> > index 71dc268..50834c2 100644
> > --- a/src/mesa/drivers/dri/i965/intel_batchbuffer.c
> > +++ b/src/mesa/drivers/dri/i965/intel_batchbuffer.c
> > @@ -253,6 +253,9 @@ do_flush_locked(struct brw_context *brw)
> >
> >     if (!brw->intelScreen->no_hw) {
> >        int flags;
> > +#ifdef NO_RELOC
> > +      flags |= I915_EXEC_NO_RELOC;
> > +#endif
> >
> >        if (brw->gen >= 6 && batch->ring == BLT_RING) {
> >           flags = I915_EXEC_BLT;
> > diff --git a/src/mesa/drivers/dri/i965/intel_batchbuffer.h b/src/mesa/drivers/dri/i965/intel_batchbuffer.h
> > index 7bdd836..2670d22 100644
> > --- a/src/mesa/drivers/dri/i965/intel_batchbuffer.h
> > +++ b/src/mesa/drivers/dri/i965/intel_batchbuffer.h
> > @@ -11,6 +11,14 @@
> >  extern "C" {
> >  #endif
> >
> > +#ifdef NO_RELOC
> > + #define drm_intel_bo_alloc_wrapper(bufmgr, name, size, align) \
> > +   drm_intel_bo_alloc_prelocated(bufmgr, name, size, 1)
> > +#else
> > + #define drm_intel_bo_alloc_wrapper drm_intel_bo_alloc
> > +#endif
> > +
> > +
> >  /**
> >   * Number of bytes to reserve for commands necessary to complete a batch.
> >   *
> > --
> > 2.0.4
> >
> > _______________________________________________
> > mesa-dev mailing list
> > mesa-dev@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/mesa-dev

-- 
Ben Widawsky, Intel Open Source Technology Center

Re: [PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Daniel Vetter]

On Fri, Aug 22, 2014 at 3:38 PM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> On Fri, Aug 22, 2014 at 03:30:12PM +0200, Daniel Vetter wrote:
>> On Fri, Aug 22, 2014 at 9:03 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
>> >> > > If a GPU
>> >> > > client uses only prelocations, the relocation process can be entirely
>> >> > > skipped. This sounds like a big win initially,
>> >> >
>> >> > Close to zero if the client uses existing interfaces.
>> >> > -Chris
>> >>
>> >> Chris,
>> >>
>> >> I don't know if you've seen Ben's libdrm and Mesa patches, but with a few patches to libdrm and virtually zero Mesa changes, he's apparently eliminated our need to do any relocations for the 3D driver.  It wasn't invasive at all---I was surprised.
>> >
>> > Indeed, you could do everything inside libdrm with the code I posted 2
>> > years ago.
>>
>> I915_EXEC_NO_RELOC can be used to tell the kernel that it doesn't need
>> to walk all the reloc tables (if nothing moved) because userspace
>> didn't go insane and reuse reloc trees. So you'd need to implement a
>> flag + a libdrm function to set that (iirc mesa has been non-stupid
>> since years). And yeah I kinda expect any new reloc-less thing to get
>> benchmarked against an implementation using that, since the 48bit
>> specific thing proposed looks like a fairly short-lived stop-gap, and
>> since the current no-reloc we already have would work everywhere. And
>> yeah I've been poking people to look at this for years. too.
>
> Here, I was referring to soft-pinning. The API here is essentially
> comprised of two parts:
>
> 1: a pin into the vm upon creation
> 2: implicit no-relocation upon execbuffer
>
> By making those two steps independent, the API as I see is, is more
> flexible and powerful.

Well I admit to not having read the patches over the terrible wifi
here, but I presumed Ben's patches


-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Intel-gfx] [PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Daniel Vetter]

On Fri, Aug 22, 2014 at 3:38 PM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> On Fri, Aug 22, 2014 at 03:30:12PM +0200, Daniel Vetter wrote:
>> On Fri, Aug 22, 2014 at 9:03 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
>> >> > > If a GPU
>> >> > > client uses only prelocations, the relocation process can be entirely
>> >> > > skipped. This sounds like a big win initially,
>> >> >
>> >> > Close to zero if the client uses existing interfaces.
>> >> > -Chris
>> >>
>> >> Chris,
>> >>
>> >> I don't know if you've seen Ben's libdrm and Mesa patches, but with a few patches to libdrm and virtually zero Mesa changes, he's apparently eliminated our need to do any relocations for the 3D driver.  It wasn't invasive at all---I was surprised.
>> >
>> > Indeed, you could do everything inside libdrm with the code I posted 2
>> > years ago.
>>
>> I915_EXEC_NO_RELOC can be used to tell the kernel that it doesn't need
>> to walk all the reloc tables (if nothing moved) because userspace
>> didn't go insane and reuse reloc trees. So you'd need to implement a
>> flag + a libdrm function to set that (iirc mesa has been non-stupid
>> since years). And yeah I kinda expect any new reloc-less thing to get
>> benchmarked against an implementation using that, since the 48bit
>> specific thing proposed looks like a fairly short-lived stop-gap, and
>> since the current no-reloc we already have would work everywhere. And
>> yeah I've been poking people to look at this for years. too.
>
> Here, I was referring to soft-pinning. The API here is essentially
> comprised of two parts:
>
> 1: a pin into the vm upon creation
> 2: implicit no-relocation upon execbuffer
>
> By making those two steps independent, the API as I see is, is more
> flexible and powerful.

Well I admit to not having read the patches over the terrible wifi
here, but I presumed Ben's patches did implement softpin. I guess I've
made a mess of all of this now. In any case I still want to see
relative improvements over what we have since the prelocated stuff
looks like a gen8 oneshot. And we still can't do relocation-less
execbuf because the gpu can't fault, so I'm not sure at all whether
this is actually useful for opencl 2.0.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [PATCH 00/68] Broadwell 48b addressing and prelocations (no relocs)

[Jesse Barnes]

On Fri, 22 Aug 2014 22:38:22 +0200
Daniel Vetter <daniel@ffwll.ch> wrote:

> On Fri, Aug 22, 2014 at 3:38 PM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> > On Fri, Aug 22, 2014 at 03:30:12PM +0200, Daniel Vetter wrote:
> >> On Fri, Aug 22, 2014 at 9:03 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> >> >> > > If a GPU
> >> >> > > client uses only prelocations, the relocation process can be entirely
> >> >> > > skipped. This sounds like a big win initially,
> >> >> >
> >> >> > Close to zero if the client uses existing interfaces.
> >> >> > -Chris
> >> >>
> >> >> Chris,
> >> >>
> >> >> I don't know if you've seen Ben's libdrm and Mesa patches, but with a few patches to libdrm and virtually zero Mesa changes, he's apparently eliminated our need to do any relocations for the 3D driver.  It wasn't invasive at all---I was surprised.
> >> >
> >> > Indeed, you could do everything inside libdrm with the code I posted 2
> >> > years ago.
> >>
> >> I915_EXEC_NO_RELOC can be used to tell the kernel that it doesn't need
> >> to walk all the reloc tables (if nothing moved) because userspace
> >> didn't go insane and reuse reloc trees. So you'd need to implement a
> >> flag + a libdrm function to set that (iirc mesa has been non-stupid
> >> since years). And yeah I kinda expect any new reloc-less thing to get
> >> benchmarked against an implementation using that, since the 48bit
> >> specific thing proposed looks like a fairly short-lived stop-gap, and
> >> since the current no-reloc we already have would work everywhere. And
> >> yeah I've been poking people to look at this for years. too.
> >
> > Here, I was referring to soft-pinning. The API here is essentially
> > comprised of two parts:
> >
> > 1: a pin into the vm upon creation
> > 2: implicit no-relocation upon execbuffer
> >
> > By making those two steps independent, the API as I see is, is more
> > flexible and powerful.
> 
> Well I admit to not having read the patches over the terrible wifi
> here, but I presumed Ben's patches did implement softpin. I guess I've
> made a mess of all of this now. In any case I still want to see
> relative improvements over what we have since the prelocated stuff
> looks like a gen8 oneshot. And we still can't do relocation-less
> execbuf because the gpu can't fault, so I'm not sure at all whether
> this is actually useful for opencl 2.0.

It is.  OCL 2.0 has two modes of operation: bufferless and buffered.
Both modes require CPU/GPU pointer sharing, but in the latter case
for us the kernel GPU driver will be involved in all allocations.

I'm not sure whether this is BDW only either, so don't shoot it down or
discount it based on that.

-- 
Jesse Barnes, Intel Open Source Technology Center

Re: [PATCH 17/68] Revert "drm/i915/bdw: Use timeout mode for RC6 on bdw"

[Rodrigo Vivi]

I'm wondering how many hangs we could have fixed with this patch.

Althougth it can end up in worst RC6 residency this is what spec shows.
From our experience with SNB it is always better to stay with spec, mainly
on threshold values.

So,

Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>


On Thu, Aug 21, 2014 at 8:11 PM, Ben Widawsky
<benjamin.widawsky@intel.com> wrote:
> This reverts commit 0d68b25e9ceb344fe2f93373b1c0311d33814265.
>
> At one time I bisected reset breakage to this patch by using a mesa that is
> guaranteed to generate a hang when using the fs, and then running the following
> test case:
>
> ./bin/shader_runner  tests/shaders/glsl-algebraic-add-zero.shader_test -auto
> ./bin/shader_runner  tests/shaders/glsl-fs-texture2d.shader_test -auto
> ./bin/shader_runner  tests/shaders/glsl-algebraic-add-zero.shader_test -auto
>
> The symptom is that after the first GPU hang, all subsequent commands
> will hang the GPU.
>
> Oddly at some point I believe this revert stopped fixing the issue, but I am
> leaving it in the series to minimize variables.
>
> ---
>  drivers/gpu/drm/i915/intel_pm.c | 16 ++++------------
>  1 file changed, 4 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
> index 41de760..3255c10 100644
> --- a/drivers/gpu/drm/i915/intel_pm.c
> +++ b/drivers/gpu/drm/i915/intel_pm.c
> @@ -3658,23 +3658,15 @@ static void gen8_enable_rps(struct drm_device *dev)
>         for_each_ring(ring, dev_priv, unused)
>                 I915_WRITE(RING_MAX_IDLE(ring->mmio_base), 10);
>         I915_WRITE(GEN6_RC_SLEEP, 0);
> -       if (IS_BROADWELL(dev))
> -               I915_WRITE(GEN6_RC6_THRESHOLD, 625); /* 800us/1.28 for TO */
> -       else
> -               I915_WRITE(GEN6_RC6_THRESHOLD, 50000); /* 50/125ms per EI */
> +       I915_WRITE(GEN6_RC6_THRESHOLD, 50000); /* 50/125ms per EI */
>
>         /* 3: Enable RC6 */
>         if (intel_enable_rc6(dev) & INTEL_RC6_ENABLE)
>                 rc6_mask = GEN6_RC_CTL_RC6_ENABLE;
>         intel_print_rc6_info(dev, rc6_mask);
> -       if (IS_BROADWELL(dev))
> -               I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
> -                               GEN7_RC_CTL_TO_MODE |
> -                               rc6_mask);
> -       else
> -               I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
> -                               GEN6_RC_CTL_EI_MODE(1) |
> -                               rc6_mask);
> +       I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
> +                                   GEN6_RC_CTL_EI_MODE(1) |
> +                                   rc6_mask);
>
>         /* 4 Program defaults and thresholds for RPS*/
>         I915_WRITE(GEN6_RPNSWREQ,
> --
> 2.0.4
>
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx



-- 
Rodrigo Vivi
Blog: http://blog.vivi.eng.br
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH 17/68] Revert "drm/i915/bdw: Use timeout mode for RC6 on bdw"

[Rodrigo Vivi]

Please ignore my command and this revert. It seems original version
that is currently applied is a needed Workaround.

On Fri, Oct 31, 2014 at 12:45 PM, Rodrigo Vivi <rodrigo.vivi@gmail.com> wrote:
> I'm wondering how many hangs we could have fixed with this patch.
>
> Althougth it can end up in worst RC6 residency this is what spec shows.
> From our experience with SNB it is always better to stay with spec, mainly
> on threshold values.
>
> So,
>
> Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
>
>
> On Thu, Aug 21, 2014 at 8:11 PM, Ben Widawsky
> <benjamin.widawsky@intel.com> wrote:
>> This reverts commit 0d68b25e9ceb344fe2f93373b1c0311d33814265.
>>
>> At one time I bisected reset breakage to this patch by using a mesa that is
>> guaranteed to generate a hang when using the fs, and then running the following
>> test case:
>>
>> ./bin/shader_runner  tests/shaders/glsl-algebraic-add-zero.shader_test -auto
>> ./bin/shader_runner  tests/shaders/glsl-fs-texture2d.shader_test -auto
>> ./bin/shader_runner  tests/shaders/glsl-algebraic-add-zero.shader_test -auto
>>
>> The symptom is that after the first GPU hang, all subsequent commands
>> will hang the GPU.
>>
>> Oddly at some point I believe this revert stopped fixing the issue, but I am
>> leaving it in the series to minimize variables.
>>
>> ---
>>  drivers/gpu/drm/i915/intel_pm.c | 16 ++++------------
>>  1 file changed, 4 insertions(+), 12 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
>> index 41de760..3255c10 100644
>> --- a/drivers/gpu/drm/i915/intel_pm.c
>> +++ b/drivers/gpu/drm/i915/intel_pm.c
>> @@ -3658,23 +3658,15 @@ static void gen8_enable_rps(struct drm_device *dev)
>>         for_each_ring(ring, dev_priv, unused)
>>                 I915_WRITE(RING_MAX_IDLE(ring->mmio_base), 10);
>>         I915_WRITE(GEN6_RC_SLEEP, 0);
>> -       if (IS_BROADWELL(dev))
>> -               I915_WRITE(GEN6_RC6_THRESHOLD, 625); /* 800us/1.28 for TO */
>> -       else
>> -               I915_WRITE(GEN6_RC6_THRESHOLD, 50000); /* 50/125ms per EI */
>> +       I915_WRITE(GEN6_RC6_THRESHOLD, 50000); /* 50/125ms per EI */
>>
>>         /* 3: Enable RC6 */
>>         if (intel_enable_rc6(dev) & INTEL_RC6_ENABLE)
>>                 rc6_mask = GEN6_RC_CTL_RC6_ENABLE;
>>         intel_print_rc6_info(dev, rc6_mask);
>> -       if (IS_BROADWELL(dev))
>> -               I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
>> -                               GEN7_RC_CTL_TO_MODE |
>> -                               rc6_mask);
>> -       else
>> -               I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
>> -                               GEN6_RC_CTL_EI_MODE(1) |
>> -                               rc6_mask);
>> +       I915_WRITE(GEN6_RC_CONTROL, GEN6_RC_CTL_HW_ENABLE |
>> +                                   GEN6_RC_CTL_EI_MODE(1) |
>> +                                   rc6_mask);
>>
>>         /* 4 Program defaults and thresholds for RPS*/
>>         I915_WRITE(GEN6_RPNSWREQ,
>> --
>> 2.0.4
>>
>> _______________________________________________
>> Intel-gfx mailing list
>> Intel-gfx@lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
>
>
>
> --
> Rodrigo Vivi
> Blog: http://blog.vivi.eng.br



-- 
Rodrigo Vivi
Blog: http://blog.vivi.eng.br
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx