* [PATCH v4 00/15] drm/i915: Move LMEM (VRAM) management over to TTM
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
This is an initial patch series to move discrete memory management over to
TTM. It will be followed up shortly with adding more functionality.
The buddy allocator is temporarily removed along with its selftests and
It is replaced with the TTM range manager and some selftests are adjusted
to account for introduced fragmentation. Work is ongoing to reintroduce the
buddy allocator as a TTM resource manager.
A new memcpy ttm move is introduced that uses kmap_local() functionality
rather than vmap(). Among other things stated in the patch commit message
it helps us deal with page-pased LMEM memory. It is generic enough to replace
the ttm memcpy move with some additional work if so desired. On x86 it also
enables prefetching reads from write-combined memory.
Finally the old i915 gem object LMEM backend is replaced with a
i915 gem object TTM backend and some additional i915 gem object ops are
introduced to support the added functionality.
Currently it is used only to support management and eviction of the LMEM
region, but work is underway to extend the support to system memory. In this
way we use TTM the way it was originally intended, having the GPU binding
taken care of by driver code.
Intention is to follow up with
- System memory support
- Pipelined accelerated moves / migration
- Re-added buddy allocator in the TTM framework
v2:
- Add patches to move pagefaulting over to TTM
- Break out TTM changes to separate patches
- Address various review comments as detailed in the affected patches
v3:
- Drop TTM pagefaulting patches for now due changing approach due to a NAK.
- Address feedback on TTM patches
- Move the new TTM memcpy functionality into TTM.
- Move fast WC memcpy to drm
- Various fixes all over the place as shown in patch commit messages.
v4:
- Re-add TTM pagefaulting patches.
- Addressed review feedback mainly from Matthew Auld
- Fixed the mock ttm device code that was using an incorrect approach.
Cc: Christian König <christian.koenig@amd.com>
Maarten Lankhorst (3):
drm/i915: Disable mmap ioctl for gen12+
drm/vma: Add a driver_private member to vma_node.
drm/i915: Use ttm mmap handling for ttm bo's.
Thomas Hellström (12):
drm/i915: Untangle the vma pages_mutex
drm/i915: Don't free shared locks while shared
drm/i915: Fix i915_sg_page_sizes to record dma segments rather than
physical pages
drm/i915/ttm Initialize the ttm device and memory managers
drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
drm/ttm: Add a generic TTM memcpy move for page-based iomem
drm, drm/i915: Move the memcpy_from_wc functionality to core drm
drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
drm/ttm, drm/amdgpu: Allow the driver some control over swapping
drm/i915/ttm: Introduce a TTM i915 gem object backend
drm/i915/lmem: Verify checks for lmem residency
drivers/gpu/drm/Makefile | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +
drivers/gpu/drm/drm_drv.c | 2 +
drivers/gpu/drm/drm_gem.c | 9 -
.../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 +-
drivers/gpu/drm/i915/Kconfig | 1 +
drivers/gpu/drm/i915/Makefile | 4 +-
drivers/gpu/drm/i915/display/intel_display.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_create.c | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 +-
.../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 71 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.h | 5 -
drivers/gpu/drm/i915/gem/i915_gem_mman.c | 85 +-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 154 +++-
drivers/gpu/drm/i915/gem/i915_gem_object.h | 19 +-
.../gpu/drm/i915/gem/i915_gem_object_types.h | 52 +-
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_phys.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 126 +--
drivers/gpu/drm/i915/gem/i915_gem_region.h | 4 -
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.h | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 649 ++++++++++++++
drivers/gpu/drm/i915/gem/i915_gem_ttm.h | 48 ++
drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +-
.../drm/i915/gem/selftests/i915_gem_mman.c | 90 +-
drivers/gpu/drm/i915/gt/intel_ggtt.c | 19 +-
drivers/gpu/drm/i915/gt/intel_gt.c | 2 -
drivers/gpu/drm/i915/gt/intel_gtt.c | 45 +-
drivers/gpu/drm/i915/gt/intel_gtt.h | 28 +-
drivers/gpu/drm/i915/gt/intel_ppgtt.c | 2 +-
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 30 +-
drivers/gpu/drm/i915/gt/selftest_reset.c | 7 +-
drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 +-
drivers/gpu/drm/i915/i915_buddy.c | 435 ----------
drivers/gpu/drm/i915/i915_buddy.h | 131 ---
drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
drivers/gpu/drm/i915/i915_drv.c | 15 +-
drivers/gpu/drm/i915/i915_drv.h | 8 +-
drivers/gpu/drm/i915/i915_gem.c | 6 +-
drivers/gpu/drm/i915/i915_globals.c | 1 -
drivers/gpu/drm/i915/i915_globals.h | 1 -
drivers/gpu/drm/i915/i915_gpu_error.c | 8 +-
drivers/gpu/drm/i915/i915_memcpy.h | 34 -
drivers/gpu/drm/i915/i915_scatterlist.c | 70 ++
drivers/gpu/drm/i915/i915_scatterlist.h | 20 +-
drivers/gpu/drm/i915/i915_vma.c | 29 +-
drivers/gpu/drm/i915/intel_memory_region.c | 181 ++--
drivers/gpu/drm/i915/intel_memory_region.h | 45 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 220 +++++
drivers/gpu/drm/i915/intel_region_ttm.h | 37 +
drivers/gpu/drm/i915/selftests/i915_buddy.c | 789 ------------------
.../drm/i915/selftests/i915_mock_selftests.h | 1 -
drivers/gpu/drm/i915/selftests/igt_mmap.c | 25 +-
drivers/gpu/drm/i915/selftests/igt_mmap.h | 12 +-
.../drm/i915/selftests/intel_memory_region.c | 140 +---
.../gpu/drm/i915/selftests/mock_gem_device.c | 10 +
drivers/gpu/drm/i915/selftests/mock_region.c | 70 +-
drivers/gpu/drm/ttm/ttm_bo.c | 63 +-
drivers/gpu/drm/ttm/ttm_bo_util.c | 347 ++++----
drivers/gpu/drm/ttm/ttm_module.c | 35 +
drivers/gpu/drm/ttm/ttm_resource.c | 193 +++++
drivers/gpu/drm/ttm/ttm_tt.c | 50 ++
include/drm/drm_memcpy.h | 68 ++
include/drm/drm_vma_manager.h | 2 +-
include/drm/ttm/ttm_bo_driver.h | 28 +
include/drm/ttm/ttm_caching.h | 2 +
include/drm/ttm/ttm_kmap_iter.h | 61 ++
include/drm/ttm/ttm_resource.h | 61 ++
include/drm/ttm/ttm_tt.h | 26 +
72 files changed, 2551 insertions(+), 2259 deletions(-)
rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.c
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.h
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.c
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.h
delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.c
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.h
delete mode 100644 drivers/gpu/drm/i915/selftests/i915_buddy.c
create mode 100644 include/drm/drm_memcpy.h
create mode 100644 include/drm/ttm/ttm_kmap_iter.h
--
2.31.1
^ permalink raw reply [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 00/15] drm/i915: Move LMEM (VRAM) management over to TTM
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
This is an initial patch series to move discrete memory management over to
TTM. It will be followed up shortly with adding more functionality.
The buddy allocator is temporarily removed along with its selftests and
It is replaced with the TTM range manager and some selftests are adjusted
to account for introduced fragmentation. Work is ongoing to reintroduce the
buddy allocator as a TTM resource manager.
A new memcpy ttm move is introduced that uses kmap_local() functionality
rather than vmap(). Among other things stated in the patch commit message
it helps us deal with page-pased LMEM memory. It is generic enough to replace
the ttm memcpy move with some additional work if so desired. On x86 it also
enables prefetching reads from write-combined memory.
Finally the old i915 gem object LMEM backend is replaced with a
i915 gem object TTM backend and some additional i915 gem object ops are
introduced to support the added functionality.
Currently it is used only to support management and eviction of the LMEM
region, but work is underway to extend the support to system memory. In this
way we use TTM the way it was originally intended, having the GPU binding
taken care of by driver code.
Intention is to follow up with
- System memory support
- Pipelined accelerated moves / migration
- Re-added buddy allocator in the TTM framework
v2:
- Add patches to move pagefaulting over to TTM
- Break out TTM changes to separate patches
- Address various review comments as detailed in the affected patches
v3:
- Drop TTM pagefaulting patches for now due changing approach due to a NAK.
- Address feedback on TTM patches
- Move the new TTM memcpy functionality into TTM.
- Move fast WC memcpy to drm
- Various fixes all over the place as shown in patch commit messages.
v4:
- Re-add TTM pagefaulting patches.
- Addressed review feedback mainly from Matthew Auld
- Fixed the mock ttm device code that was using an incorrect approach.
Cc: Christian König <christian.koenig@amd.com>
Maarten Lankhorst (3):
drm/i915: Disable mmap ioctl for gen12+
drm/vma: Add a driver_private member to vma_node.
drm/i915: Use ttm mmap handling for ttm bo's.
Thomas Hellström (12):
drm/i915: Untangle the vma pages_mutex
drm/i915: Don't free shared locks while shared
drm/i915: Fix i915_sg_page_sizes to record dma segments rather than
physical pages
drm/i915/ttm Initialize the ttm device and memory managers
drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
drm/ttm: Add a generic TTM memcpy move for page-based iomem
drm, drm/i915: Move the memcpy_from_wc functionality to core drm
drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
drm/ttm, drm/amdgpu: Allow the driver some control over swapping
drm/i915/ttm: Introduce a TTM i915 gem object backend
drm/i915/lmem: Verify checks for lmem residency
drivers/gpu/drm/Makefile | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +
drivers/gpu/drm/drm_drv.c | 2 +
drivers/gpu/drm/drm_gem.c | 9 -
.../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 +-
drivers/gpu/drm/i915/Kconfig | 1 +
drivers/gpu/drm/i915/Makefile | 4 +-
drivers/gpu/drm/i915/display/intel_display.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_create.c | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 +-
.../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 71 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.h | 5 -
drivers/gpu/drm/i915/gem/i915_gem_mman.c | 85 +-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 154 +++-
drivers/gpu/drm/i915/gem/i915_gem_object.h | 19 +-
.../gpu/drm/i915/gem/i915_gem_object_types.h | 52 +-
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_phys.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 126 +--
drivers/gpu/drm/i915/gem/i915_gem_region.h | 4 -
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.h | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 649 ++++++++++++++
drivers/gpu/drm/i915/gem/i915_gem_ttm.h | 48 ++
drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +-
.../drm/i915/gem/selftests/i915_gem_mman.c | 90 +-
drivers/gpu/drm/i915/gt/intel_ggtt.c | 19 +-
drivers/gpu/drm/i915/gt/intel_gt.c | 2 -
drivers/gpu/drm/i915/gt/intel_gtt.c | 45 +-
drivers/gpu/drm/i915/gt/intel_gtt.h | 28 +-
drivers/gpu/drm/i915/gt/intel_ppgtt.c | 2 +-
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 30 +-
drivers/gpu/drm/i915/gt/selftest_reset.c | 7 +-
drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 +-
drivers/gpu/drm/i915/i915_buddy.c | 435 ----------
drivers/gpu/drm/i915/i915_buddy.h | 131 ---
drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
drivers/gpu/drm/i915/i915_drv.c | 15 +-
drivers/gpu/drm/i915/i915_drv.h | 8 +-
drivers/gpu/drm/i915/i915_gem.c | 6 +-
drivers/gpu/drm/i915/i915_globals.c | 1 -
drivers/gpu/drm/i915/i915_globals.h | 1 -
drivers/gpu/drm/i915/i915_gpu_error.c | 8 +-
drivers/gpu/drm/i915/i915_memcpy.h | 34 -
drivers/gpu/drm/i915/i915_scatterlist.c | 70 ++
drivers/gpu/drm/i915/i915_scatterlist.h | 20 +-
drivers/gpu/drm/i915/i915_vma.c | 29 +-
drivers/gpu/drm/i915/intel_memory_region.c | 181 ++--
drivers/gpu/drm/i915/intel_memory_region.h | 45 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 220 +++++
drivers/gpu/drm/i915/intel_region_ttm.h | 37 +
drivers/gpu/drm/i915/selftests/i915_buddy.c | 789 ------------------
.../drm/i915/selftests/i915_mock_selftests.h | 1 -
drivers/gpu/drm/i915/selftests/igt_mmap.c | 25 +-
drivers/gpu/drm/i915/selftests/igt_mmap.h | 12 +-
.../drm/i915/selftests/intel_memory_region.c | 140 +---
.../gpu/drm/i915/selftests/mock_gem_device.c | 10 +
drivers/gpu/drm/i915/selftests/mock_region.c | 70 +-
drivers/gpu/drm/ttm/ttm_bo.c | 63 +-
drivers/gpu/drm/ttm/ttm_bo_util.c | 347 ++++----
drivers/gpu/drm/ttm/ttm_module.c | 35 +
drivers/gpu/drm/ttm/ttm_resource.c | 193 +++++
drivers/gpu/drm/ttm/ttm_tt.c | 50 ++
include/drm/drm_memcpy.h | 68 ++
include/drm/drm_vma_manager.h | 2 +-
include/drm/ttm/ttm_bo_driver.h | 28 +
include/drm/ttm/ttm_caching.h | 2 +
include/drm/ttm/ttm_kmap_iter.h | 61 ++
include/drm/ttm/ttm_resource.h | 61 ++
include/drm/ttm/ttm_tt.h | 26 +
72 files changed, 2551 insertions(+), 2259 deletions(-)
rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.c
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.h
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.c
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.h
delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.c
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.h
delete mode 100644 drivers/gpu/drm/i915/selftests/i915_buddy.c
create mode 100644 include/drm/drm_memcpy.h
create mode 100644 include/drm/ttm/ttm_kmap_iter.h
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v4 01/15] drm/i915: Untangle the vma pages_mutex
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
Any sleeping dma_resv lock taken while the vma pages_mutex is held
will cause a lockdep splat.
Move the i915_gem_object_pin_pages() call out of the pages_mutex
critical section.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/i915_vma.c | 29 +++++++++++++++++------------
1 file changed, 17 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index a6cd0fa62847..f2b5912fc542 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -800,32 +800,37 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
static int vma_get_pages(struct i915_vma *vma)
{
int err = 0;
+ bool pinned_pages = false;
if (atomic_add_unless(&vma->pages_count, 1, 0))
return 0;
+ if (vma->obj) {
+ err = i915_gem_object_pin_pages(vma->obj);
+ if (err)
+ return err;
+ pinned_pages = true;
+ }
+
/* Allocations ahoy! */
- if (mutex_lock_interruptible(&vma->pages_mutex))
- return -EINTR;
+ if (mutex_lock_interruptible(&vma->pages_mutex)) {
+ err = -EINTR;
+ goto unpin;
+ }
if (!atomic_read(&vma->pages_count)) {
- if (vma->obj) {
- err = i915_gem_object_pin_pages(vma->obj);
- if (err)
- goto unlock;
- }
-
err = vma->ops->set_pages(vma);
- if (err) {
- if (vma->obj)
- i915_gem_object_unpin_pages(vma->obj);
+ if (err)
goto unlock;
- }
+ pinned_pages = false;
}
atomic_inc(&vma->pages_count);
unlock:
mutex_unlock(&vma->pages_mutex);
+unpin:
+ if (pinned_pages)
+ __i915_gem_object_unpin_pages(vma->obj);
return err;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 01/15] drm/i915: Untangle the vma pages_mutex
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
Any sleeping dma_resv lock taken while the vma pages_mutex is held
will cause a lockdep splat.
Move the i915_gem_object_pin_pages() call out of the pages_mutex
critical section.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/i915_vma.c | 29 +++++++++++++++++------------
1 file changed, 17 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index a6cd0fa62847..f2b5912fc542 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -800,32 +800,37 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
static int vma_get_pages(struct i915_vma *vma)
{
int err = 0;
+ bool pinned_pages = false;
if (atomic_add_unless(&vma->pages_count, 1, 0))
return 0;
+ if (vma->obj) {
+ err = i915_gem_object_pin_pages(vma->obj);
+ if (err)
+ return err;
+ pinned_pages = true;
+ }
+
/* Allocations ahoy! */
- if (mutex_lock_interruptible(&vma->pages_mutex))
- return -EINTR;
+ if (mutex_lock_interruptible(&vma->pages_mutex)) {
+ err = -EINTR;
+ goto unpin;
+ }
if (!atomic_read(&vma->pages_count)) {
- if (vma->obj) {
- err = i915_gem_object_pin_pages(vma->obj);
- if (err)
- goto unlock;
- }
-
err = vma->ops->set_pages(vma);
- if (err) {
- if (vma->obj)
- i915_gem_object_unpin_pages(vma->obj);
+ if (err)
goto unlock;
- }
+ pinned_pages = false;
}
atomic_inc(&vma->pages_count);
unlock:
mutex_unlock(&vma->pages_mutex);
+unpin:
+ if (pinned_pages)
+ __i915_gem_object_unpin_pages(vma->obj);
return err;
}
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 02/15] drm/i915: Don't free shared locks while shared
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
We are currently sharing the VM reservation locks across a number of
gem objects with page-table memory. Since TTM will individiualize the
reservation locks when freeing objects, including accessing the shared
locks, make sure that the shared locks are not freed until that is done.
For PPGTT we add an additional refcount, for GGTT we take additional
measures to make sure objects sharing the GGTT reservation lock are
freed at GGTT takedown
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
v2: Try harder to make sure objects sharing the GGTT reservation lock are
freed at GGTT takedown.
v3: Use a pointer to the vm to indicate that an object shares a reservation
object from that vm, rather than a pointer to the reservation object itself.
---
drivers/gpu/drm/i915/gem/i915_gem_object.c | 3 ++
.../gpu/drm/i915/gem/i915_gem_object_types.h | 4 ++
drivers/gpu/drm/i915/gt/intel_ggtt.c | 19 ++++++--
drivers/gpu/drm/i915/gt/intel_gtt.c | 45 +++++++++++++++----
drivers/gpu/drm/i915/gt/intel_gtt.h | 28 +++++++++++-
drivers/gpu/drm/i915/gt/intel_ppgtt.c | 2 +-
drivers/gpu/drm/i915/i915_drv.c | 5 +++
7 files changed, 93 insertions(+), 13 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 28144410df86..2be6109d0093 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -252,6 +252,9 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915,
if (obj->mm.n_placements > 1)
kfree(obj->mm.placements);
+ if (obj->shares_resv_from)
+ i915_vm_resv_put(obj->shares_resv_from);
+
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
cond_resched();
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 0727d0c76aa0..0415f99b6b95 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -149,6 +149,10 @@ struct drm_i915_gem_object {
* when i915_gem_ww_ctx_backoff() or i915_gem_ww_ctx_fini() are called.
*/
struct list_head obj_link;
+ /**
+ * @shared_resv_from: The object shares the resv from this vm.
+ */
+ struct i915_address_space *shares_resv_from;
union {
struct rcu_head rcu;
diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt.c b/drivers/gpu/drm/i915/gt/intel_ggtt.c
index 35069ca5d7de..10c23a749a95 100644
--- a/drivers/gpu/drm/i915/gt/intel_ggtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ggtt.c
@@ -746,7 +746,6 @@ static void ggtt_cleanup_hw(struct i915_ggtt *ggtt)
mutex_unlock(&ggtt->vm.mutex);
i915_address_space_fini(&ggtt->vm);
- dma_resv_fini(&ggtt->vm.resv);
arch_phys_wc_del(ggtt->mtrr);
@@ -768,6 +767,19 @@ void i915_ggtt_driver_release(struct drm_i915_private *i915)
ggtt_cleanup_hw(ggtt);
}
+/**
+ * i915_ggtt_driver_late_release - Cleanup of GGTT that needs to be done after
+ * all free objects have been drained.
+ * @i915: i915 device
+ */
+void i915_ggtt_driver_late_release(struct drm_i915_private *i915)
+{
+ struct i915_ggtt *ggtt = &i915->ggtt;
+
+ GEM_WARN_ON(kref_read(&ggtt->vm.resv_ref) != 1);
+ dma_resv_fini(&ggtt->vm._resv);
+}
+
static unsigned int gen6_get_total_gtt_size(u16 snb_gmch_ctl)
{
snb_gmch_ctl >>= SNB_GMCH_GGMS_SHIFT;
@@ -829,6 +841,7 @@ static int ggtt_probe_common(struct i915_ggtt *ggtt, u64 size)
return -ENOMEM;
}
+ kref_init(&ggtt->vm.resv_ref);
ret = setup_scratch_page(&ggtt->vm);
if (ret) {
drm_err(&i915->drm, "Scratch setup failed\n");
@@ -1135,7 +1148,7 @@ static int ggtt_probe_hw(struct i915_ggtt *ggtt, struct intel_gt *gt)
ggtt->vm.gt = gt;
ggtt->vm.i915 = i915;
ggtt->vm.dma = i915->drm.dev;
- dma_resv_init(&ggtt->vm.resv);
+ dma_resv_init(&ggtt->vm._resv);
if (INTEL_GEN(i915) <= 5)
ret = i915_gmch_probe(ggtt);
@@ -1144,7 +1157,7 @@ static int ggtt_probe_hw(struct i915_ggtt *ggtt, struct intel_gt *gt)
else
ret = gen8_gmch_probe(ggtt);
if (ret) {
- dma_resv_fini(&ggtt->vm.resv);
+ dma_resv_fini(&ggtt->vm._resv);
return ret;
}
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.c b/drivers/gpu/drm/i915/gt/intel_gtt.c
index 9b98f9d9faa3..94849567143d 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.c
@@ -22,8 +22,11 @@ struct drm_i915_gem_object *alloc_pt_lmem(struct i915_address_space *vm, int sz)
* object underneath, with the idea that one object_lock() will lock
* them all at once.
*/
- if (!IS_ERR(obj))
- obj->base.resv = &vm->resv;
+ if (!IS_ERR(obj)) {
+ obj->base.resv = i915_vm_resv_get(vm);
+ obj->shares_resv_from = vm;
+ }
+
return obj;
}
@@ -40,8 +43,11 @@ struct drm_i915_gem_object *alloc_pt_dma(struct i915_address_space *vm, int sz)
* object underneath, with the idea that one object_lock() will lock
* them all at once.
*/
- if (!IS_ERR(obj))
- obj->base.resv = &vm->resv;
+ if (!IS_ERR(obj)) {
+ obj->base.resv = i915_vm_resv_get(vm);
+ obj->shares_resv_from = vm;
+ }
+
return obj;
}
@@ -102,7 +108,7 @@ void __i915_vm_close(struct i915_address_space *vm)
int i915_vm_lock_objects(struct i915_address_space *vm,
struct i915_gem_ww_ctx *ww)
{
- if (vm->scratch[0]->base.resv == &vm->resv) {
+ if (vm->scratch[0]->base.resv == &vm->_resv) {
return i915_gem_object_lock(vm->scratch[0], ww);
} else {
struct i915_ppgtt *ppgtt = i915_vm_to_ppgtt(vm);
@@ -118,6 +124,22 @@ void i915_address_space_fini(struct i915_address_space *vm)
mutex_destroy(&vm->mutex);
}
+/**
+ * i915_vm_resv_release - Final struct i915_address_space destructor
+ * @kref: Pointer to the &i915_address_space.resv_ref member.
+ *
+ * This function is called when the last lock sharer no longer shares the
+ * &i915_address_space._resv lock.
+ */
+void i915_vm_resv_release(struct kref *kref)
+{
+ struct i915_address_space *vm =
+ container_of(kref, typeof(*vm), resv_ref);
+
+ dma_resv_fini(&vm->_resv);
+ kfree(vm);
+}
+
static void __i915_vm_release(struct work_struct *work)
{
struct i915_address_space *vm =
@@ -125,9 +147,8 @@ static void __i915_vm_release(struct work_struct *work)
vm->cleanup(vm);
i915_address_space_fini(vm);
- dma_resv_fini(&vm->resv);
- kfree(vm);
+ i915_vm_resv_put(vm);
}
void i915_vm_release(struct kref *kref)
@@ -144,6 +165,14 @@ void i915_vm_release(struct kref *kref)
void i915_address_space_init(struct i915_address_space *vm, int subclass)
{
kref_init(&vm->ref);
+
+ /*
+ * Special case for GGTT that has already done an early
+ * kref_init here.
+ */
+ if (!kref_read(&vm->resv_ref))
+ kref_init(&vm->resv_ref);
+
INIT_RCU_WORK(&vm->rcu, __i915_vm_release);
atomic_set(&vm->open, 1);
@@ -170,7 +199,7 @@ void i915_address_space_init(struct i915_address_space *vm, int subclass)
might_alloc(GFP_KERNEL);
mutex_release(&vm->mutex.dep_map, _THIS_IP_);
}
- dma_resv_init(&vm->resv);
+ dma_resv_init(&vm->_resv);
GEM_BUG_ON(!vm->total);
drm_mm_init(&vm->mm, 0, vm->total);
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.h b/drivers/gpu/drm/i915/gt/intel_gtt.h
index ca00b45827b7..f39be66e84f6 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.h
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.h
@@ -245,7 +245,9 @@ struct i915_address_space {
atomic_t open;
struct mutex mutex; /* protects vma and our lists */
- struct dma_resv resv; /* reservation lock for all pd objects, and buffer pool */
+
+ struct kref resv_ref; /* kref to keep the reservation lock alive. */
+ struct dma_resv _resv; /* reservation lock for all pd objects, and buffer pool */
#define VM_CLASS_GGTT 0
#define VM_CLASS_PPGTT 1
#define VM_CLASS_DPT 2
@@ -404,13 +406,36 @@ i915_vm_get(struct i915_address_space *vm)
return vm;
}
+/**
+ * i915_vm_resv_get - Obtain a reference on the vm's reservation lock
+ * @vm: The vm whose reservation lock we want to share.
+ *
+ * Return: A pointer to the vm's reservation lock.
+ */
+static inline struct dma_resv *i915_vm_resv_get(struct i915_address_space *vm)
+{
+ kref_get(&vm->resv_ref);
+ return &vm->_resv;
+}
+
void i915_vm_release(struct kref *kref);
+void i915_vm_resv_release(struct kref *kref);
+
static inline void i915_vm_put(struct i915_address_space *vm)
{
kref_put(&vm->ref, i915_vm_release);
}
+/**
+ * i915_vm_resv_put - Release a reference on the vm's reservation lock
+ * @resv: Pointer to a reservation lock obtained from i915_vm_resv_get()
+ */
+static inline void i915_vm_resv_put(struct i915_address_space *vm)
+{
+ kref_put(&vm->resv_ref, i915_vm_resv_release);
+}
+
static inline struct i915_address_space *
i915_vm_open(struct i915_address_space *vm)
{
@@ -506,6 +531,7 @@ void i915_ggtt_enable_guc(struct i915_ggtt *ggtt);
void i915_ggtt_disable_guc(struct i915_ggtt *ggtt);
int i915_init_ggtt(struct drm_i915_private *i915);
void i915_ggtt_driver_release(struct drm_i915_private *i915);
+void i915_ggtt_driver_late_release(struct drm_i915_private *i915);
static inline bool i915_ggtt_has_aperture(const struct i915_ggtt *ggtt)
{
diff --git a/drivers/gpu/drm/i915/gt/intel_ppgtt.c b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
index 4e3d80c2295c..aee3a8929245 100644
--- a/drivers/gpu/drm/i915/gt/intel_ppgtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
@@ -307,7 +307,7 @@ void ppgtt_init(struct i915_ppgtt *ppgtt, struct intel_gt *gt)
ppgtt->vm.dma = i915->drm.dev;
ppgtt->vm.total = BIT_ULL(INTEL_INFO(i915)->ppgtt_size);
- dma_resv_init(&ppgtt->vm.resv);
+ dma_resv_init(&ppgtt->vm._resv);
i915_address_space_init(&ppgtt->vm, VM_CLASS_PPGTT);
ppgtt->vm.vma_ops.bind_vma = ppgtt_bind_vma;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 2f06bb7b3ed2..d82a99e128cf 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -631,6 +631,8 @@ static int i915_driver_hw_probe(struct drm_i915_private *dev_priv)
intel_memory_regions_driver_release(dev_priv);
err_ggtt:
i915_ggtt_driver_release(dev_priv);
+ i915_gem_drain_freed_objects(dev_priv);
+ i915_ggtt_driver_late_release(dev_priv);
err_perf:
i915_perf_fini(dev_priv);
return ret;
@@ -880,6 +882,8 @@ int i915_driver_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
i915_driver_hw_remove(i915);
intel_memory_regions_driver_release(i915);
i915_ggtt_driver_release(i915);
+ i915_gem_drain_freed_objects(i915);
+ i915_ggtt_driver_late_release(i915);
out_cleanup_mmio:
i915_driver_mmio_release(i915);
out_runtime_pm_put:
@@ -936,6 +940,7 @@ static void i915_driver_release(struct drm_device *dev)
intel_memory_regions_driver_release(dev_priv);
i915_ggtt_driver_release(dev_priv);
i915_gem_drain_freed_objects(dev_priv);
+ i915_ggtt_driver_late_release(dev_priv);
i915_driver_mmio_release(dev_priv);
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 02/15] drm/i915: Don't free shared locks while shared
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
We are currently sharing the VM reservation locks across a number of
gem objects with page-table memory. Since TTM will individiualize the
reservation locks when freeing objects, including accessing the shared
locks, make sure that the shared locks are not freed until that is done.
For PPGTT we add an additional refcount, for GGTT we take additional
measures to make sure objects sharing the GGTT reservation lock are
freed at GGTT takedown
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
v2: Try harder to make sure objects sharing the GGTT reservation lock are
freed at GGTT takedown.
v3: Use a pointer to the vm to indicate that an object shares a reservation
object from that vm, rather than a pointer to the reservation object itself.
---
drivers/gpu/drm/i915/gem/i915_gem_object.c | 3 ++
.../gpu/drm/i915/gem/i915_gem_object_types.h | 4 ++
drivers/gpu/drm/i915/gt/intel_ggtt.c | 19 ++++++--
drivers/gpu/drm/i915/gt/intel_gtt.c | 45 +++++++++++++++----
drivers/gpu/drm/i915/gt/intel_gtt.h | 28 +++++++++++-
drivers/gpu/drm/i915/gt/intel_ppgtt.c | 2 +-
drivers/gpu/drm/i915/i915_drv.c | 5 +++
7 files changed, 93 insertions(+), 13 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 28144410df86..2be6109d0093 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -252,6 +252,9 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915,
if (obj->mm.n_placements > 1)
kfree(obj->mm.placements);
+ if (obj->shares_resv_from)
+ i915_vm_resv_put(obj->shares_resv_from);
+
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
cond_resched();
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 0727d0c76aa0..0415f99b6b95 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -149,6 +149,10 @@ struct drm_i915_gem_object {
* when i915_gem_ww_ctx_backoff() or i915_gem_ww_ctx_fini() are called.
*/
struct list_head obj_link;
+ /**
+ * @shared_resv_from: The object shares the resv from this vm.
+ */
+ struct i915_address_space *shares_resv_from;
union {
struct rcu_head rcu;
diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt.c b/drivers/gpu/drm/i915/gt/intel_ggtt.c
index 35069ca5d7de..10c23a749a95 100644
--- a/drivers/gpu/drm/i915/gt/intel_ggtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ggtt.c
@@ -746,7 +746,6 @@ static void ggtt_cleanup_hw(struct i915_ggtt *ggtt)
mutex_unlock(&ggtt->vm.mutex);
i915_address_space_fini(&ggtt->vm);
- dma_resv_fini(&ggtt->vm.resv);
arch_phys_wc_del(ggtt->mtrr);
@@ -768,6 +767,19 @@ void i915_ggtt_driver_release(struct drm_i915_private *i915)
ggtt_cleanup_hw(ggtt);
}
+/**
+ * i915_ggtt_driver_late_release - Cleanup of GGTT that needs to be done after
+ * all free objects have been drained.
+ * @i915: i915 device
+ */
+void i915_ggtt_driver_late_release(struct drm_i915_private *i915)
+{
+ struct i915_ggtt *ggtt = &i915->ggtt;
+
+ GEM_WARN_ON(kref_read(&ggtt->vm.resv_ref) != 1);
+ dma_resv_fini(&ggtt->vm._resv);
+}
+
static unsigned int gen6_get_total_gtt_size(u16 snb_gmch_ctl)
{
snb_gmch_ctl >>= SNB_GMCH_GGMS_SHIFT;
@@ -829,6 +841,7 @@ static int ggtt_probe_common(struct i915_ggtt *ggtt, u64 size)
return -ENOMEM;
}
+ kref_init(&ggtt->vm.resv_ref);
ret = setup_scratch_page(&ggtt->vm);
if (ret) {
drm_err(&i915->drm, "Scratch setup failed\n");
@@ -1135,7 +1148,7 @@ static int ggtt_probe_hw(struct i915_ggtt *ggtt, struct intel_gt *gt)
ggtt->vm.gt = gt;
ggtt->vm.i915 = i915;
ggtt->vm.dma = i915->drm.dev;
- dma_resv_init(&ggtt->vm.resv);
+ dma_resv_init(&ggtt->vm._resv);
if (INTEL_GEN(i915) <= 5)
ret = i915_gmch_probe(ggtt);
@@ -1144,7 +1157,7 @@ static int ggtt_probe_hw(struct i915_ggtt *ggtt, struct intel_gt *gt)
else
ret = gen8_gmch_probe(ggtt);
if (ret) {
- dma_resv_fini(&ggtt->vm.resv);
+ dma_resv_fini(&ggtt->vm._resv);
return ret;
}
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.c b/drivers/gpu/drm/i915/gt/intel_gtt.c
index 9b98f9d9faa3..94849567143d 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.c
@@ -22,8 +22,11 @@ struct drm_i915_gem_object *alloc_pt_lmem(struct i915_address_space *vm, int sz)
* object underneath, with the idea that one object_lock() will lock
* them all at once.
*/
- if (!IS_ERR(obj))
- obj->base.resv = &vm->resv;
+ if (!IS_ERR(obj)) {
+ obj->base.resv = i915_vm_resv_get(vm);
+ obj->shares_resv_from = vm;
+ }
+
return obj;
}
@@ -40,8 +43,11 @@ struct drm_i915_gem_object *alloc_pt_dma(struct i915_address_space *vm, int sz)
* object underneath, with the idea that one object_lock() will lock
* them all at once.
*/
- if (!IS_ERR(obj))
- obj->base.resv = &vm->resv;
+ if (!IS_ERR(obj)) {
+ obj->base.resv = i915_vm_resv_get(vm);
+ obj->shares_resv_from = vm;
+ }
+
return obj;
}
@@ -102,7 +108,7 @@ void __i915_vm_close(struct i915_address_space *vm)
int i915_vm_lock_objects(struct i915_address_space *vm,
struct i915_gem_ww_ctx *ww)
{
- if (vm->scratch[0]->base.resv == &vm->resv) {
+ if (vm->scratch[0]->base.resv == &vm->_resv) {
return i915_gem_object_lock(vm->scratch[0], ww);
} else {
struct i915_ppgtt *ppgtt = i915_vm_to_ppgtt(vm);
@@ -118,6 +124,22 @@ void i915_address_space_fini(struct i915_address_space *vm)
mutex_destroy(&vm->mutex);
}
+/**
+ * i915_vm_resv_release - Final struct i915_address_space destructor
+ * @kref: Pointer to the &i915_address_space.resv_ref member.
+ *
+ * This function is called when the last lock sharer no longer shares the
+ * &i915_address_space._resv lock.
+ */
+void i915_vm_resv_release(struct kref *kref)
+{
+ struct i915_address_space *vm =
+ container_of(kref, typeof(*vm), resv_ref);
+
+ dma_resv_fini(&vm->_resv);
+ kfree(vm);
+}
+
static void __i915_vm_release(struct work_struct *work)
{
struct i915_address_space *vm =
@@ -125,9 +147,8 @@ static void __i915_vm_release(struct work_struct *work)
vm->cleanup(vm);
i915_address_space_fini(vm);
- dma_resv_fini(&vm->resv);
- kfree(vm);
+ i915_vm_resv_put(vm);
}
void i915_vm_release(struct kref *kref)
@@ -144,6 +165,14 @@ void i915_vm_release(struct kref *kref)
void i915_address_space_init(struct i915_address_space *vm, int subclass)
{
kref_init(&vm->ref);
+
+ /*
+ * Special case for GGTT that has already done an early
+ * kref_init here.
+ */
+ if (!kref_read(&vm->resv_ref))
+ kref_init(&vm->resv_ref);
+
INIT_RCU_WORK(&vm->rcu, __i915_vm_release);
atomic_set(&vm->open, 1);
@@ -170,7 +199,7 @@ void i915_address_space_init(struct i915_address_space *vm, int subclass)
might_alloc(GFP_KERNEL);
mutex_release(&vm->mutex.dep_map, _THIS_IP_);
}
- dma_resv_init(&vm->resv);
+ dma_resv_init(&vm->_resv);
GEM_BUG_ON(!vm->total);
drm_mm_init(&vm->mm, 0, vm->total);
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.h b/drivers/gpu/drm/i915/gt/intel_gtt.h
index ca00b45827b7..f39be66e84f6 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.h
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.h
@@ -245,7 +245,9 @@ struct i915_address_space {
atomic_t open;
struct mutex mutex; /* protects vma and our lists */
- struct dma_resv resv; /* reservation lock for all pd objects, and buffer pool */
+
+ struct kref resv_ref; /* kref to keep the reservation lock alive. */
+ struct dma_resv _resv; /* reservation lock for all pd objects, and buffer pool */
#define VM_CLASS_GGTT 0
#define VM_CLASS_PPGTT 1
#define VM_CLASS_DPT 2
@@ -404,13 +406,36 @@ i915_vm_get(struct i915_address_space *vm)
return vm;
}
+/**
+ * i915_vm_resv_get - Obtain a reference on the vm's reservation lock
+ * @vm: The vm whose reservation lock we want to share.
+ *
+ * Return: A pointer to the vm's reservation lock.
+ */
+static inline struct dma_resv *i915_vm_resv_get(struct i915_address_space *vm)
+{
+ kref_get(&vm->resv_ref);
+ return &vm->_resv;
+}
+
void i915_vm_release(struct kref *kref);
+void i915_vm_resv_release(struct kref *kref);
+
static inline void i915_vm_put(struct i915_address_space *vm)
{
kref_put(&vm->ref, i915_vm_release);
}
+/**
+ * i915_vm_resv_put - Release a reference on the vm's reservation lock
+ * @resv: Pointer to a reservation lock obtained from i915_vm_resv_get()
+ */
+static inline void i915_vm_resv_put(struct i915_address_space *vm)
+{
+ kref_put(&vm->resv_ref, i915_vm_resv_release);
+}
+
static inline struct i915_address_space *
i915_vm_open(struct i915_address_space *vm)
{
@@ -506,6 +531,7 @@ void i915_ggtt_enable_guc(struct i915_ggtt *ggtt);
void i915_ggtt_disable_guc(struct i915_ggtt *ggtt);
int i915_init_ggtt(struct drm_i915_private *i915);
void i915_ggtt_driver_release(struct drm_i915_private *i915);
+void i915_ggtt_driver_late_release(struct drm_i915_private *i915);
static inline bool i915_ggtt_has_aperture(const struct i915_ggtt *ggtt)
{
diff --git a/drivers/gpu/drm/i915/gt/intel_ppgtt.c b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
index 4e3d80c2295c..aee3a8929245 100644
--- a/drivers/gpu/drm/i915/gt/intel_ppgtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
@@ -307,7 +307,7 @@ void ppgtt_init(struct i915_ppgtt *ppgtt, struct intel_gt *gt)
ppgtt->vm.dma = i915->drm.dev;
ppgtt->vm.total = BIT_ULL(INTEL_INFO(i915)->ppgtt_size);
- dma_resv_init(&ppgtt->vm.resv);
+ dma_resv_init(&ppgtt->vm._resv);
i915_address_space_init(&ppgtt->vm, VM_CLASS_PPGTT);
ppgtt->vm.vma_ops.bind_vma = ppgtt_bind_vma;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 2f06bb7b3ed2..d82a99e128cf 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -631,6 +631,8 @@ static int i915_driver_hw_probe(struct drm_i915_private *dev_priv)
intel_memory_regions_driver_release(dev_priv);
err_ggtt:
i915_ggtt_driver_release(dev_priv);
+ i915_gem_drain_freed_objects(dev_priv);
+ i915_ggtt_driver_late_release(dev_priv);
err_perf:
i915_perf_fini(dev_priv);
return ret;
@@ -880,6 +882,8 @@ int i915_driver_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
i915_driver_hw_remove(i915);
intel_memory_regions_driver_release(i915);
i915_ggtt_driver_release(i915);
+ i915_gem_drain_freed_objects(i915);
+ i915_ggtt_driver_late_release(i915);
out_cleanup_mmio:
i915_driver_mmio_release(i915);
out_runtime_pm_put:
@@ -936,6 +940,7 @@ static void i915_driver_release(struct drm_device *dev)
intel_memory_regions_driver_release(dev_priv);
i915_ggtt_driver_release(dev_priv);
i915_gem_drain_freed_objects(dev_priv);
+ i915_ggtt_driver_late_release(dev_priv);
i915_driver_mmio_release(dev_priv);
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 03/15] drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
All users of this function actually want the dma segment sizes, but that's
not what's calculated. Fix that and rename the function to
i915_sg_dma_sizes to reflect what's calculated.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_phys.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +-
drivers/gpu/drm/i915/i915_scatterlist.h | 16 ++++++++++++----
4 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index ccede73c6465..616c3a2f1baf 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -209,7 +209,7 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
if (IS_ERR(pages))
return PTR_ERR(pages);
- sg_page_sizes = i915_sg_page_sizes(pages->sgl);
+ sg_page_sizes = i915_sg_dma_sizes(pages->sgl);
__i915_gem_object_set_pages(obj, pages, sg_page_sizes);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_phys.c b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
index 51a05e62875d..be72ad0634ba 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_phys.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
@@ -207,7 +207,7 @@ static int i915_gem_object_shmem_to_phys(struct drm_i915_gem_object *obj)
err_xfer:
if (!IS_ERR_OR_NULL(pages)) {
- unsigned int sg_page_sizes = i915_sg_page_sizes(pages->sgl);
+ unsigned int sg_page_sizes = i915_sg_dma_sizes(pages->sgl);
__i915_gem_object_set_pages(obj, pages, sg_page_sizes);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
index a657b99ec760..602f0ed983ec 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
@@ -173,7 +173,7 @@ static int i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
goto err;
}
- sg_page_sizes = i915_sg_page_sizes(st->sgl);
+ sg_page_sizes = i915_sg_dma_sizes(st->sgl);
__i915_gem_object_set_pages(obj, st, sg_page_sizes);
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.h b/drivers/gpu/drm/i915/i915_scatterlist.h
index 9cb26a224034..b96baad66a3a 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.h
+++ b/drivers/gpu/drm/i915/i915_scatterlist.h
@@ -101,15 +101,23 @@ static inline struct scatterlist *__sg_next(struct scatterlist *sg)
(((__iter).curr += PAGE_SIZE) >= (__iter).max) ? \
(__iter) = __sgt_iter(__sg_next((__iter).sgp), false), 0 : 0)
-static inline unsigned int i915_sg_page_sizes(struct scatterlist *sg)
+/**
+ * i915_sg_dma_sizes - Record the dma segment sizes of a scatterlist
+ * @sg: The scatterlist
+ *
+ * Return: An unsigned int with segment sizes logically or'ed together.
+ * A caller can use this information to determine what hardware page table
+ * entry sizes can be used to map the memory represented by the scatterlist.
+ */
+static inline unsigned int i915_sg_dma_sizes(struct scatterlist *sg)
{
unsigned int page_sizes;
page_sizes = 0;
- while (sg) {
+ while (sg && sg_dma_len(sg)) {
GEM_BUG_ON(sg->offset);
- GEM_BUG_ON(!IS_ALIGNED(sg->length, PAGE_SIZE));
- page_sizes |= sg->length;
+ GEM_BUG_ON(!IS_ALIGNED(sg_dma_len(sg), PAGE_SIZE));
+ page_sizes |= sg_dma_len(sg);
sg = __sg_next(sg);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 03/15] drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
All users of this function actually want the dma segment sizes, but that's
not what's calculated. Fix that and rename the function to
i915_sg_dma_sizes to reflect what's calculated.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_phys.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +-
drivers/gpu/drm/i915/i915_scatterlist.h | 16 ++++++++++++----
4 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index ccede73c6465..616c3a2f1baf 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -209,7 +209,7 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
if (IS_ERR(pages))
return PTR_ERR(pages);
- sg_page_sizes = i915_sg_page_sizes(pages->sgl);
+ sg_page_sizes = i915_sg_dma_sizes(pages->sgl);
__i915_gem_object_set_pages(obj, pages, sg_page_sizes);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_phys.c b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
index 51a05e62875d..be72ad0634ba 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_phys.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
@@ -207,7 +207,7 @@ static int i915_gem_object_shmem_to_phys(struct drm_i915_gem_object *obj)
err_xfer:
if (!IS_ERR_OR_NULL(pages)) {
- unsigned int sg_page_sizes = i915_sg_page_sizes(pages->sgl);
+ unsigned int sg_page_sizes = i915_sg_dma_sizes(pages->sgl);
__i915_gem_object_set_pages(obj, pages, sg_page_sizes);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
index a657b99ec760..602f0ed983ec 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
@@ -173,7 +173,7 @@ static int i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
goto err;
}
- sg_page_sizes = i915_sg_page_sizes(st->sgl);
+ sg_page_sizes = i915_sg_dma_sizes(st->sgl);
__i915_gem_object_set_pages(obj, st, sg_page_sizes);
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.h b/drivers/gpu/drm/i915/i915_scatterlist.h
index 9cb26a224034..b96baad66a3a 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.h
+++ b/drivers/gpu/drm/i915/i915_scatterlist.h
@@ -101,15 +101,23 @@ static inline struct scatterlist *__sg_next(struct scatterlist *sg)
(((__iter).curr += PAGE_SIZE) >= (__iter).max) ? \
(__iter) = __sgt_iter(__sg_next((__iter).sgp), false), 0 : 0)
-static inline unsigned int i915_sg_page_sizes(struct scatterlist *sg)
+/**
+ * i915_sg_dma_sizes - Record the dma segment sizes of a scatterlist
+ * @sg: The scatterlist
+ *
+ * Return: An unsigned int with segment sizes logically or'ed together.
+ * A caller can use this information to determine what hardware page table
+ * entry sizes can be used to map the memory represented by the scatterlist.
+ */
+static inline unsigned int i915_sg_dma_sizes(struct scatterlist *sg)
{
unsigned int page_sizes;
page_sizes = 0;
- while (sg) {
+ while (sg && sg_dma_len(sg)) {
GEM_BUG_ON(sg->offset);
- GEM_BUG_ON(!IS_ALIGNED(sg->length, PAGE_SIZE));
- page_sizes |= sg->length;
+ GEM_BUG_ON(!IS_ALIGNED(sg_dma_len(sg), PAGE_SIZE));
+ page_sizes |= sg_dma_len(sg);
sg = __sg_next(sg);
}
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 04/15] drm/i915/ttm Initialize the ttm device and memory managers
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Temporarily remove the buddy allocator and related selftests
and hook up the TTM range manager for i915 regions.
Also modify the mock region selftests somewhat to account for a
fragmenting manager.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com> #v2
---
v2:
- Fix an error unwind in lmem_get_pages() (Reported by Matthew Auld)
- Break out and modify usage of i915_sg_dma_sizes() (Reported by Mattew Auld)
- Break out TTM changes to a separate patch (Reported by Christian König)
v3:
- Fix the same error unwind in mock_region_get_pages()
(Reported by Matthew Auld)
- Don't rely on new TTM functionality, but create a mock TTM device,
(Reported by Christian König)
v4:
- Use mock_gem_device rather than creating a separate ttm_device per
region.
---
drivers/gpu/drm/i915/Kconfig | 1 +
drivers/gpu/drm/i915/Makefile | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 59 +-
.../gpu/drm/i915/gem/i915_gem_object_types.h | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 120 ---
drivers/gpu/drm/i915/gem/i915_gem_region.h | 4 -
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.h | 9 +-
drivers/gpu/drm/i915/gt/intel_gt.c | 2 -
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 27 +-
drivers/gpu/drm/i915/i915_buddy.c | 435 ----------
drivers/gpu/drm/i915/i915_buddy.h | 131 ---
drivers/gpu/drm/i915/i915_drv.c | 8 +
drivers/gpu/drm/i915/i915_drv.h | 8 +-
drivers/gpu/drm/i915/i915_gem.c | 1 +
drivers/gpu/drm/i915/i915_globals.c | 1 -
drivers/gpu/drm/i915/i915_globals.h | 1 -
drivers/gpu/drm/i915/i915_scatterlist.c | 70 ++
drivers/gpu/drm/i915/i915_scatterlist.h | 4 +
drivers/gpu/drm/i915/intel_memory_region.c | 180 ++--
drivers/gpu/drm/i915/intel_memory_region.h | 44 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 220 +++++
drivers/gpu/drm/i915/intel_region_ttm.h | 32 +
drivers/gpu/drm/i915/selftests/i915_buddy.c | 789 ------------------
.../drm/i915/selftests/i915_mock_selftests.h | 1 -
.../drm/i915/selftests/intel_memory_region.c | 133 +--
.../gpu/drm/i915/selftests/mock_gem_device.c | 10 +
drivers/gpu/drm/i915/selftests/mock_region.c | 70 +-
30 files changed, 631 insertions(+), 1754 deletions(-)
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.c
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.h
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.c
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.h
delete mode 100644 drivers/gpu/drm/i915/selftests/i915_buddy.c
diff --git a/drivers/gpu/drm/i915/Kconfig b/drivers/gpu/drm/i915/Kconfig
index 93f4d059fc89..61ff5c178714 100644
--- a/drivers/gpu/drm/i915/Kconfig
+++ b/drivers/gpu/drm/i915/Kconfig
@@ -27,6 +27,7 @@ config DRM_I915
select SND_HDA_I915 if SND_HDA_CORE
select CEC_CORE if CEC_NOTIFIER
select VMAP_PFN
+ select DRM_TTM
help
Choose this option if you have a system that has "Intel Graphics
Media Accelerator" or "HD Graphics" integrated graphics,
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 6947495bf34b..4f22cac1c49b 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -50,6 +50,7 @@ i915-y += i915_drv.o \
intel_memory_region.o \
intel_pch.o \
intel_pm.o \
+ intel_region_ttm.o \
intel_runtime_pm.o \
intel_sideband.o \
intel_step.o \
@@ -160,7 +161,6 @@ gem-y += \
i915-y += \
$(gem-y) \
i915_active.o \
- i915_buddy.o \
i915_cmd_parser.o \
i915_gem_evict.o \
i915_gem_gtt.o \
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index f44bdd08f7cb..3b4aa28a076d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -4,16 +4,71 @@
*/
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_lmem.h"
#include "i915_drv.h"
+static void lmem_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *pages)
+{
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ obj->mm.dirty = false;
+ sg_free_table(pages);
+ kfree(pages);
+}
+
+static int lmem_get_pages(struct drm_i915_gem_object *obj)
+{
+ unsigned int flags;
+ struct sg_table *pages;
+
+ flags = I915_ALLOC_MIN_PAGE_SIZE;
+ if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
+ flags |= I915_ALLOC_CONTIGUOUS;
+
+ obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
+ obj->base.size,
+ flags);
+ if (IS_ERR(obj->mm.st_mm_node))
+ return PTR_ERR(obj->mm.st_mm_node);
+
+ /* Range manager is always contigous */
+ if (obj->mm.region->is_range_manager)
+ obj->flags |= I915_BO_ALLOC_CONTIGUOUS;
+ pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
+ if (IS_ERR(pages)) {
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ return PTR_ERR(pages);
+ }
+
+ __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
+
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
+ void __iomem *vaddr =
+ i915_gem_object_lmem_io_map(obj, 0, obj->base.size);
+
+ if (!vaddr) {
+ struct sg_table *pages =
+ __i915_gem_object_unset_pages(obj);
+
+ if (!IS_ERR_OR_NULL(pages))
+ lmem_put_pages(obj, pages);
+ }
+
+ memset_io(vaddr, 0, obj->base.size);
+ io_mapping_unmap(vaddr);
+ }
+
+ return 0;
+}
+
const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
.name = "i915_gem_object_lmem",
.flags = I915_GEM_OBJECT_HAS_IOMEM,
- .get_pages = i915_gem_object_get_pages_buddy,
- .put_pages = i915_gem_object_put_pages_buddy,
+ .get_pages = lmem_get_pages,
+ .put_pages = lmem_put_pages,
.release = i915_gem_object_release_memory_region,
};
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 0415f99b6b95..f5b46d11e6e6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -235,10 +235,12 @@ struct drm_i915_gem_object {
* Memory region for this object.
*/
struct intel_memory_region *region;
+
/**
- * List of memory region blocks allocated for this object.
+ * Memory manager node allocated for this object.
*/
- struct list_head blocks;
+ void *st_mm_node;
+
/**
* Element within memory_region->objects or region->purgeable
* if the object is marked as DONTNEED. Access is protected by
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
index 7361971c177d..6444e097016d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
@@ -475,7 +475,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
might_sleep();
GEM_BUG_ON(n >= obj->base.size >> PAGE_SHIFT);
- GEM_BUG_ON(!i915_gem_object_has_pinned_pages(obj));
+ if (!i915_gem_object_has_pinned_pages(obj))
+ assert_object_held(obj);
/* As we iterate forward through the sg, we record each entry in a
* radixtree for quick repeated (backwards) lookups. If we have seen
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.c b/drivers/gpu/drm/i915/gem/i915_gem_region.c
index ce8fcfc54079..f25e6646c5b7 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.c
@@ -8,129 +8,9 @@
#include "i915_drv.h"
#include "i915_trace.h"
-void
-i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj,
- struct sg_table *pages)
-{
- __intel_memory_region_put_pages_buddy(obj->mm.region, &obj->mm.blocks);
-
- obj->mm.dirty = false;
- sg_free_table(pages);
- kfree(pages);
-}
-
-int
-i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj)
-{
- const u64 max_segment = i915_sg_segment_size();
- struct intel_memory_region *mem = obj->mm.region;
- struct list_head *blocks = &obj->mm.blocks;
- resource_size_t size = obj->base.size;
- resource_size_t prev_end;
- struct i915_buddy_block *block;
- unsigned int flags;
- struct sg_table *st;
- struct scatterlist *sg;
- unsigned int sg_page_sizes;
- int ret;
-
- st = kmalloc(sizeof(*st), GFP_KERNEL);
- if (!st)
- return -ENOMEM;
-
- if (sg_alloc_table(st, size >> PAGE_SHIFT, GFP_KERNEL)) {
- kfree(st);
- return -ENOMEM;
- }
-
- flags = I915_ALLOC_MIN_PAGE_SIZE;
- if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
- flags |= I915_ALLOC_CONTIGUOUS;
-
- ret = __intel_memory_region_get_pages_buddy(mem, size, flags, blocks);
- if (ret)
- goto err_free_sg;
-
- GEM_BUG_ON(list_empty(blocks));
-
- sg = st->sgl;
- st->nents = 0;
- sg_page_sizes = 0;
- prev_end = (resource_size_t)-1;
-
- list_for_each_entry(block, blocks, link) {
- u64 block_size, offset;
-
- block_size = min_t(u64, size,
- i915_buddy_block_size(&mem->mm, block));
- offset = i915_buddy_block_offset(block);
-
- while (block_size) {
- u64 len;
-
- if (offset != prev_end || sg->length >= max_segment) {
- if (st->nents) {
- sg_page_sizes |= sg->length;
- sg = __sg_next(sg);
- }
-
- sg_dma_address(sg) = mem->region.start + offset;
- sg_dma_len(sg) = 0;
- sg->length = 0;
- st->nents++;
- }
-
- len = min(block_size, max_segment - sg->length);
- sg->length += len;
- sg_dma_len(sg) += len;
-
- offset += len;
- block_size -= len;
-
- prev_end = offset;
- }
- }
-
- sg_page_sizes |= sg->length;
- sg_mark_end(sg);
- i915_sg_trim(st);
-
- /* Intended for kernel internal use only */
- if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
- struct scatterlist *sg;
- unsigned long i;
-
- for_each_sg(st->sgl, sg, st->nents, i) {
- unsigned int length;
- void __iomem *vaddr;
- dma_addr_t daddr;
-
- daddr = sg_dma_address(sg);
- daddr -= mem->region.start;
- length = sg_dma_len(sg);
-
- vaddr = io_mapping_map_wc(&mem->iomap, daddr, length);
- memset64((void __force *)vaddr, 0, length / sizeof(u64));
- io_mapping_unmap(vaddr);
- }
-
- wmb();
- }
-
- __i915_gem_object_set_pages(obj, st, sg_page_sizes);
-
- return 0;
-
-err_free_sg:
- sg_free_table(st);
- kfree(st);
- return ret;
-}
-
void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
struct intel_memory_region *mem)
{
- INIT_LIST_HEAD(&obj->mm.blocks);
obj->mm.region = intel_memory_region_get(mem);
if (obj->base.size <= mem->min_page_size)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.h b/drivers/gpu/drm/i915/gem/i915_gem_region.h
index ebddc86d78f7..84fcb3297400 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.h
@@ -12,10 +12,6 @@ struct intel_memory_region;
struct drm_i915_gem_object;
struct sg_table;
-int i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj);
-void i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj,
- struct sg_table *pages);
-
void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
struct intel_memory_region *mem);
void i915_gem_object_release_memory_region(struct drm_i915_gem_object *obj);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
index a9bfa66c8da1..5d16c4462fda 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
@@ -628,11 +628,13 @@ static const struct intel_memory_region_ops shmem_region_ops = {
.init_object = shmem_object_init,
};
-struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915)
+struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915,
+ u16 type, u16 instance)
{
return intel_memory_region_create(i915, 0,
totalram_pages() << PAGE_SHIFT,
PAGE_SIZE, 0,
+ type, instance,
&shmem_region_ops);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
index b5553fc3ac4d..092d7a21de82 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
@@ -772,7 +772,8 @@ static const struct intel_memory_region_ops i915_region_stolen_lmem_ops = {
};
struct intel_memory_region *
-i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance)
{
struct intel_uncore *uncore = &i915->uncore;
struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
@@ -790,6 +791,7 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
mem = intel_memory_region_create(i915, lmem_base, lmem_size,
I915_GTT_PAGE_SIZE_4K, io_start,
+ type, instance,
&i915_region_stolen_lmem_ops);
if (IS_ERR(mem))
return mem;
@@ -811,14 +813,15 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
}
struct intel_memory_region*
-i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance)
{
struct intel_memory_region *mem;
mem = intel_memory_region_create(i915,
intel_graphics_stolen_res.start,
resource_size(&intel_graphics_stolen_res),
- PAGE_SIZE, 0,
+ PAGE_SIZE, 0, type, instance,
&i915_region_stolen_smem_ops);
if (IS_ERR(mem))
return mem;
@@ -826,7 +829,6 @@ i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
intel_memory_region_set_name(mem, "stolen-system");
mem->private = true;
-
return mem;
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.h b/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
index 2bec6c367b9c..ccdf7befc571 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
@@ -21,8 +21,13 @@ int i915_gem_stolen_insert_node_in_range(struct drm_i915_private *dev_priv,
u64 end);
void i915_gem_stolen_remove_node(struct drm_i915_private *dev_priv,
struct drm_mm_node *node);
-struct intel_memory_region *i915_gem_stolen_smem_setup(struct drm_i915_private *i915);
-struct intel_memory_region *i915_gem_stolen_lmem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance);
+struct intel_memory_region *
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance);
+
struct drm_i915_gem_object *
i915_gem_object_create_stolen(struct drm_i915_private *dev_priv,
resource_size_t size);
diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 8d77dcbad059..3f88ecdee031 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -68,8 +68,6 @@ int intel_gt_probe_lmem(struct intel_gt *gt)
id = INTEL_REGION_LMEM;
mem->id = id;
- mem->type = INTEL_MEMORY_LOCAL;
- mem->instance = 0;
intel_memory_region_set_name(mem, "local%u", mem->instance);
diff --git a/drivers/gpu/drm/i915/gt/intel_region_lmem.c b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
index 73fceb0c25fc..f7366b054f8e 100644
--- a/drivers/gpu/drm/i915/gt/intel_region_lmem.c
+++ b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
@@ -5,6 +5,8 @@
#include "i915_drv.h"
#include "intel_memory_region.h"
+#include "intel_region_lmem.h"
+#include "intel_region_ttm.h"
#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
#include "intel_region_lmem.h"
@@ -66,9 +68,9 @@ static void release_fake_lmem_bar(struct intel_memory_region *mem)
static void
region_lmem_release(struct intel_memory_region *mem)
{
- release_fake_lmem_bar(mem);
+ intel_region_ttm_fini(mem);
io_mapping_fini(&mem->iomap);
- intel_memory_region_release_buddy(mem);
+ release_fake_lmem_bar(mem);
}
static int
@@ -83,12 +85,21 @@ region_lmem_init(struct intel_memory_region *mem)
if (!io_mapping_init_wc(&mem->iomap,
mem->io_start,
- resource_size(&mem->region)))
- return -EIO;
+ resource_size(&mem->region))) {
+ ret = -EIO;
+ goto out_no_io;
+ }
- ret = intel_memory_region_init_buddy(mem);
+ ret = intel_region_ttm_init(mem);
if (ret)
- io_mapping_fini(&mem->iomap);
+ goto out_no_buddy;
+
+ return 0;
+
+out_no_buddy:
+ io_mapping_fini(&mem->iomap);
+out_no_io:
+ release_fake_lmem_bar(mem);
return ret;
}
@@ -127,6 +138,8 @@ intel_gt_setup_fake_lmem(struct intel_gt *gt)
mappable_end,
PAGE_SIZE,
io_start,
+ INTEL_MEMORY_LOCAL,
+ 0,
&intel_region_lmem_ops);
if (!IS_ERR(mem)) {
drm_info(&i915->drm, "Intel graphics fake LMEM: %pR\n",
@@ -198,6 +211,8 @@ static struct intel_memory_region *setup_lmem(struct intel_gt *gt)
lmem_size,
I915_GTT_PAGE_SIZE_4K,
io_start,
+ INTEL_MEMORY_LOCAL,
+ 0,
&intel_region_lmem_ops);
if (IS_ERR(mem))
return mem;
diff --git a/drivers/gpu/drm/i915/i915_buddy.c b/drivers/gpu/drm/i915/i915_buddy.c
deleted file mode 100644
index 3a2f6eecb2fc..000000000000
--- a/drivers/gpu/drm/i915/i915_buddy.c
+++ /dev/null
@@ -1,435 +0,0 @@
-// SPDX-License-Identifier: MIT
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#include <linux/kmemleak.h>
-#include <linux/slab.h>
-
-#include "i915_buddy.h"
-
-#include "i915_gem.h"
-#include "i915_globals.h"
-#include "i915_utils.h"
-
-static struct i915_global_block {
- struct i915_global base;
- struct kmem_cache *slab_blocks;
-} global;
-
-static void i915_global_buddy_shrink(void)
-{
- kmem_cache_shrink(global.slab_blocks);
-}
-
-static void i915_global_buddy_exit(void)
-{
- kmem_cache_destroy(global.slab_blocks);
-}
-
-static struct i915_global_block global = { {
- .shrink = i915_global_buddy_shrink,
- .exit = i915_global_buddy_exit,
-} };
-
-int __init i915_global_buddy_init(void)
-{
- global.slab_blocks = KMEM_CACHE(i915_buddy_block, SLAB_HWCACHE_ALIGN);
- if (!global.slab_blocks)
- return -ENOMEM;
-
- i915_global_register(&global.base);
- return 0;
-}
-
-static struct i915_buddy_block *i915_block_alloc(struct i915_buddy_block *parent,
- unsigned int order,
- u64 offset)
-{
- struct i915_buddy_block *block;
-
- GEM_BUG_ON(order > I915_BUDDY_MAX_ORDER);
-
- block = kmem_cache_zalloc(global.slab_blocks, GFP_KERNEL);
- if (!block)
- return NULL;
-
- block->header = offset;
- block->header |= order;
- block->parent = parent;
-
- GEM_BUG_ON(block->header & I915_BUDDY_HEADER_UNUSED);
- return block;
-}
-
-static void i915_block_free(struct i915_buddy_block *block)
-{
- kmem_cache_free(global.slab_blocks, block);
-}
-
-static void mark_allocated(struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_ALLOCATED;
-
- list_del(&block->link);
-}
-
-static void mark_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_FREE;
-
- list_add(&block->link,
- &mm->free_list[i915_buddy_block_order(block)]);
-}
-
-static void mark_split(struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_SPLIT;
-
- list_del(&block->link);
-}
-
-int i915_buddy_init(struct i915_buddy_mm *mm, u64 size, u64 chunk_size)
-{
- unsigned int i;
- u64 offset;
-
- if (size < chunk_size)
- return -EINVAL;
-
- if (chunk_size < PAGE_SIZE)
- return -EINVAL;
-
- if (!is_power_of_2(chunk_size))
- return -EINVAL;
-
- size = round_down(size, chunk_size);
-
- mm->size = size;
- mm->chunk_size = chunk_size;
- mm->max_order = ilog2(size) - ilog2(chunk_size);
-
- GEM_BUG_ON(mm->max_order > I915_BUDDY_MAX_ORDER);
-
- mm->free_list = kmalloc_array(mm->max_order + 1,
- sizeof(struct list_head),
- GFP_KERNEL);
- if (!mm->free_list)
- return -ENOMEM;
-
- for (i = 0; i <= mm->max_order; ++i)
- INIT_LIST_HEAD(&mm->free_list[i]);
-
- mm->n_roots = hweight64(size);
-
- mm->roots = kmalloc_array(mm->n_roots,
- sizeof(struct i915_buddy_block *),
- GFP_KERNEL);
- if (!mm->roots)
- goto out_free_list;
-
- offset = 0;
- i = 0;
-
- /*
- * Split into power-of-two blocks, in case we are given a size that is
- * not itself a power-of-two.
- */
- do {
- struct i915_buddy_block *root;
- unsigned int order;
- u64 root_size;
-
- root_size = rounddown_pow_of_two(size);
- order = ilog2(root_size) - ilog2(chunk_size);
-
- root = i915_block_alloc(NULL, order, offset);
- if (!root)
- goto out_free_roots;
-
- mark_free(mm, root);
-
- GEM_BUG_ON(i > mm->max_order);
- GEM_BUG_ON(i915_buddy_block_size(mm, root) < chunk_size);
-
- mm->roots[i] = root;
-
- offset += root_size;
- size -= root_size;
- i++;
- } while (size);
-
- return 0;
-
-out_free_roots:
- while (i--)
- i915_block_free(mm->roots[i]);
- kfree(mm->roots);
-out_free_list:
- kfree(mm->free_list);
- return -ENOMEM;
-}
-
-void i915_buddy_fini(struct i915_buddy_mm *mm)
-{
- int i;
-
- for (i = 0; i < mm->n_roots; ++i) {
- GEM_WARN_ON(!i915_buddy_block_is_free(mm->roots[i]));
- i915_block_free(mm->roots[i]);
- }
-
- kfree(mm->roots);
- kfree(mm->free_list);
-}
-
-static int split_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- unsigned int block_order = i915_buddy_block_order(block) - 1;
- u64 offset = i915_buddy_block_offset(block);
-
- GEM_BUG_ON(!i915_buddy_block_is_free(block));
- GEM_BUG_ON(!i915_buddy_block_order(block));
-
- block->left = i915_block_alloc(block, block_order, offset);
- if (!block->left)
- return -ENOMEM;
-
- block->right = i915_block_alloc(block, block_order,
- offset + (mm->chunk_size << block_order));
- if (!block->right) {
- i915_block_free(block->left);
- return -ENOMEM;
- }
-
- mark_free(mm, block->left);
- mark_free(mm, block->right);
-
- mark_split(block);
-
- return 0;
-}
-
-static struct i915_buddy_block *
-get_buddy(struct i915_buddy_block *block)
-{
- struct i915_buddy_block *parent;
-
- parent = block->parent;
- if (!parent)
- return NULL;
-
- if (parent->left == block)
- return parent->right;
-
- return parent->left;
-}
-
-static void __i915_buddy_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *parent;
-
- while ((parent = block->parent)) {
- struct i915_buddy_block *buddy;
-
- buddy = get_buddy(block);
-
- if (!i915_buddy_block_is_free(buddy))
- break;
-
- list_del(&buddy->link);
-
- i915_block_free(block);
- i915_block_free(buddy);
-
- block = parent;
- }
-
- mark_free(mm, block);
-}
-
-void i915_buddy_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- GEM_BUG_ON(!i915_buddy_block_is_allocated(block));
- __i915_buddy_free(mm, block);
-}
-
-void i915_buddy_free_list(struct i915_buddy_mm *mm, struct list_head *objects)
-{
- struct i915_buddy_block *block, *on;
-
- list_for_each_entry_safe(block, on, objects, link) {
- i915_buddy_free(mm, block);
- cond_resched();
- }
- INIT_LIST_HEAD(objects);
-}
-
-/*
- * Allocate power-of-two block. The order value here translates to:
- *
- * 0 = 2^0 * mm->chunk_size
- * 1 = 2^1 * mm->chunk_size
- * 2 = 2^2 * mm->chunk_size
- * ...
- */
-struct i915_buddy_block *
-i915_buddy_alloc(struct i915_buddy_mm *mm, unsigned int order)
-{
- struct i915_buddy_block *block = NULL;
- unsigned int i;
- int err;
-
- for (i = order; i <= mm->max_order; ++i) {
- block = list_first_entry_or_null(&mm->free_list[i],
- struct i915_buddy_block,
- link);
- if (block)
- break;
- }
-
- if (!block)
- return ERR_PTR(-ENOSPC);
-
- GEM_BUG_ON(!i915_buddy_block_is_free(block));
-
- while (i != order) {
- err = split_block(mm, block);
- if (unlikely(err))
- goto out_free;
-
- /* Go low */
- block = block->left;
- i--;
- }
-
- mark_allocated(block);
- kmemleak_update_trace(block);
- return block;
-
-out_free:
- if (i != order)
- __i915_buddy_free(mm, block);
- return ERR_PTR(err);
-}
-
-static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
-{
- return s1 <= e2 && e1 >= s2;
-}
-
-static inline bool contains(u64 s1, u64 e1, u64 s2, u64 e2)
-{
- return s1 <= s2 && e1 >= e2;
-}
-
-/*
- * Allocate range. Note that it's safe to chain together multiple alloc_ranges
- * with the same blocks list.
- *
- * Intended for pre-allocating portions of the address space, for example to
- * reserve a block for the initial framebuffer or similar, hence the expectation
- * here is that i915_buddy_alloc() is still the main vehicle for
- * allocations, so if that's not the case then the drm_mm range allocator is
- * probably a much better fit, and so you should probably go use that instead.
- */
-int i915_buddy_alloc_range(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 start, u64 size)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_block *buddy;
- LIST_HEAD(allocated);
- LIST_HEAD(dfs);
- u64 end;
- int err;
- int i;
-
- if (size < mm->chunk_size)
- return -EINVAL;
-
- if (!IS_ALIGNED(size | start, mm->chunk_size))
- return -EINVAL;
-
- if (range_overflows(start, size, mm->size))
- return -EINVAL;
-
- for (i = 0; i < mm->n_roots; ++i)
- list_add_tail(&mm->roots[i]->tmp_link, &dfs);
-
- end = start + size - 1;
-
- do {
- u64 block_start;
- u64 block_end;
-
- block = list_first_entry_or_null(&dfs,
- struct i915_buddy_block,
- tmp_link);
- if (!block)
- break;
-
- list_del(&block->tmp_link);
-
- block_start = i915_buddy_block_offset(block);
- block_end = block_start + i915_buddy_block_size(mm, block) - 1;
-
- if (!overlaps(start, end, block_start, block_end))
- continue;
-
- if (i915_buddy_block_is_allocated(block)) {
- err = -ENOSPC;
- goto err_free;
- }
-
- if (contains(start, end, block_start, block_end)) {
- if (!i915_buddy_block_is_free(block)) {
- err = -ENOSPC;
- goto err_free;
- }
-
- mark_allocated(block);
- list_add_tail(&block->link, &allocated);
- continue;
- }
-
- if (!i915_buddy_block_is_split(block)) {
- err = split_block(mm, block);
- if (unlikely(err))
- goto err_undo;
- }
-
- list_add(&block->right->tmp_link, &dfs);
- list_add(&block->left->tmp_link, &dfs);
- } while (1);
-
- list_splice_tail(&allocated, blocks);
- return 0;
-
-err_undo:
- /*
- * We really don't want to leave around a bunch of split blocks, since
- * bigger is better, so make sure we merge everything back before we
- * free the allocated blocks.
- */
- buddy = get_buddy(block);
- if (buddy &&
- (i915_buddy_block_is_free(block) &&
- i915_buddy_block_is_free(buddy)))
- __i915_buddy_free(mm, block);
-
-err_free:
- i915_buddy_free_list(mm, &allocated);
- return err;
-}
-
-#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
-#include "selftests/i915_buddy.c"
-#endif
diff --git a/drivers/gpu/drm/i915/i915_buddy.h b/drivers/gpu/drm/i915/i915_buddy.h
deleted file mode 100644
index 9ce5200f4001..000000000000
--- a/drivers/gpu/drm/i915/i915_buddy.h
+++ /dev/null
@@ -1,131 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#ifndef __I915_BUDDY_H__
-#define __I915_BUDDY_H__
-
-#include <linux/bitops.h>
-#include <linux/list.h>
-
-struct i915_buddy_block {
-#define I915_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
-#define I915_BUDDY_HEADER_STATE GENMASK_ULL(11, 10)
-#define I915_BUDDY_ALLOCATED (1 << 10)
-#define I915_BUDDY_FREE (2 << 10)
-#define I915_BUDDY_SPLIT (3 << 10)
-/* Free to be used, if needed in the future */
-#define I915_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
-#define I915_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
- u64 header;
-
- struct i915_buddy_block *left;
- struct i915_buddy_block *right;
- struct i915_buddy_block *parent;
-
- void *private; /* owned by creator */
-
- /*
- * While the block is allocated by the user through i915_buddy_alloc*,
- * the user has ownership of the link, for example to maintain within
- * a list, if so desired. As soon as the block is freed with
- * i915_buddy_free* ownership is given back to the mm.
- */
- struct list_head link;
- struct list_head tmp_link;
-};
-
-/* Order-zero must be at least PAGE_SIZE */
-#define I915_BUDDY_MAX_ORDER (63 - PAGE_SHIFT)
-
-/*
- * Binary Buddy System.
- *
- * Locking should be handled by the user, a simple mutex around
- * i915_buddy_alloc* and i915_buddy_free* should suffice.
- */
-struct i915_buddy_mm {
- /* Maintain a free list for each order. */
- struct list_head *free_list;
-
- /*
- * Maintain explicit binary tree(s) to track the allocation of the
- * address space. This gives us a simple way of finding a buddy block
- * and performing the potentially recursive merge step when freeing a
- * block. Nodes are either allocated or free, in which case they will
- * also exist on the respective free list.
- */
- struct i915_buddy_block **roots;
-
- /*
- * Anything from here is public, and remains static for the lifetime of
- * the mm. Everything above is considered do-not-touch.
- */
- unsigned int n_roots;
- unsigned int max_order;
-
- /* Must be at least PAGE_SIZE */
- u64 chunk_size;
- u64 size;
-};
-
-static inline u64
-i915_buddy_block_offset(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_OFFSET;
-}
-
-static inline unsigned int
-i915_buddy_block_order(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_ORDER;
-}
-
-static inline unsigned int
-i915_buddy_block_state(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_STATE;
-}
-
-static inline bool
-i915_buddy_block_is_allocated(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_ALLOCATED;
-}
-
-static inline bool
-i915_buddy_block_is_free(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_FREE;
-}
-
-static inline bool
-i915_buddy_block_is_split(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_SPLIT;
-}
-
-static inline u64
-i915_buddy_block_size(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- return mm->chunk_size << i915_buddy_block_order(block);
-}
-
-int i915_buddy_init(struct i915_buddy_mm *mm, u64 size, u64 chunk_size);
-
-void i915_buddy_fini(struct i915_buddy_mm *mm);
-
-struct i915_buddy_block *
-i915_buddy_alloc(struct i915_buddy_mm *mm, unsigned int order);
-
-int i915_buddy_alloc_range(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 start, u64 size);
-
-void i915_buddy_free(struct i915_buddy_mm *mm, struct i915_buddy_block *block);
-
-void i915_buddy_free_list(struct i915_buddy_mm *mm, struct list_head *objects);
-
-#endif
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index d82a99e128cf..30c349137be2 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -84,6 +84,7 @@
#include "intel_gvt.h"
#include "intel_memory_region.h"
#include "intel_pm.h"
+#include "intel_region_ttm.h"
#include "intel_sideband.h"
#include "vlv_suspend.h"
@@ -335,6 +336,10 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
if (ret < 0)
goto err_workqueues;
+ ret = intel_region_ttm_device_init(dev_priv);
+ if (ret)
+ goto err_ttm;
+
intel_wopcm_init_early(&dev_priv->wopcm);
intel_gt_init_early(&dev_priv->gt, dev_priv);
@@ -359,6 +364,8 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
err_gem:
i915_gem_cleanup_early(dev_priv);
intel_gt_driver_late_release(&dev_priv->gt);
+ intel_region_ttm_device_fini(dev_priv);
+err_ttm:
vlv_suspend_cleanup(dev_priv);
err_workqueues:
i915_workqueues_cleanup(dev_priv);
@@ -376,6 +383,7 @@ static void i915_driver_late_release(struct drm_i915_private *dev_priv)
intel_power_domains_cleanup(dev_priv);
i915_gem_cleanup_early(dev_priv);
intel_gt_driver_late_release(&dev_priv->gt);
+ intel_region_ttm_device_fini(dev_priv);
vlv_suspend_cleanup(dev_priv);
i915_workqueues_cleanup(dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 9cb02618ba15..3a985865db41 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -59,6 +59,7 @@
#include <drm/drm_atomic.h>
#include <drm/drm_connector.h>
#include <drm/i915_mei_hdcp_interface.h>
+#include <drm/ttm/ttm_device.h>
#include "i915_params.h"
#include "i915_reg.h"
@@ -778,6 +779,7 @@ struct intel_cdclk_config {
struct i915_selftest_stash {
atomic_t counter;
+ struct ida mock_region_instances;
};
struct drm_i915_private {
@@ -1167,6 +1169,9 @@ struct drm_i915_private {
/* Mutex to protect the above hdcp component related values. */
struct mutex hdcp_comp_mutex;
+ /* The TTM device structure. */
+ struct ttm_device bdev;
+
I915_SELFTEST_DECLARE(struct i915_selftest_stash selftest;)
/*
@@ -1760,7 +1765,8 @@ void i915_gem_cleanup_userptr(struct drm_i915_private *dev_priv);
void i915_gem_init_early(struct drm_i915_private *dev_priv);
void i915_gem_cleanup_early(struct drm_i915_private *dev_priv);
-struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915,
+ u16 type, u16 instance);
static inline void i915_gem_drain_freed_objects(struct drm_i915_private *i915)
{
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index cffd7f4f87dc..0993d706f067 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1108,6 +1108,7 @@ int i915_gem_init(struct drm_i915_private *dev_priv)
}
i915_gem_drain_freed_objects(dev_priv);
+
return ret;
}
diff --git a/drivers/gpu/drm/i915/i915_globals.c b/drivers/gpu/drm/i915/i915_globals.c
index 3aa213684293..77f1911c463b 100644
--- a/drivers/gpu/drm/i915/i915_globals.c
+++ b/drivers/gpu/drm/i915/i915_globals.c
@@ -87,7 +87,6 @@ static void __i915_globals_cleanup(void)
static __initconst int (* const initfn[])(void) = {
i915_global_active_init,
- i915_global_buddy_init,
i915_global_context_init,
i915_global_gem_context_init,
i915_global_objects_init,
diff --git a/drivers/gpu/drm/i915/i915_globals.h b/drivers/gpu/drm/i915/i915_globals.h
index b2f5cd9b9b1a..2d199f411a4a 100644
--- a/drivers/gpu/drm/i915/i915_globals.h
+++ b/drivers/gpu/drm/i915/i915_globals.h
@@ -27,7 +27,6 @@ void i915_globals_exit(void);
/* constructors */
int i915_global_active_init(void);
-int i915_global_buddy_init(void);
int i915_global_context_init(void);
int i915_global_gem_context_init(void);
int i915_global_objects_init(void);
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.c b/drivers/gpu/drm/i915/i915_scatterlist.c
index cc6b3846a8c7..69e9e6c3135e 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.c
+++ b/drivers/gpu/drm/i915/i915_scatterlist.c
@@ -6,6 +6,10 @@
#include "i915_scatterlist.h"
+#include <drm/drm_mm.h>
+
+#include <linux/slab.h>
+
bool i915_sg_trim(struct sg_table *orig_st)
{
struct sg_table new_st;
@@ -34,6 +38,72 @@ bool i915_sg_trim(struct sg_table *orig_st)
return true;
}
+/**
+ * i915_sg_from_mm_node - Create an sg_table from a struct drm_mm_node
+ * @node: The drm_mm_node.
+ * @region_start: An offset to add to the dma addresses of the sg list.
+ *
+ * Create a struct sg_table, initializing it from a struct drm_mm_node,
+ * taking a maximum segment length into account, splitting into segments
+ * if necessary.
+ *
+ * Return: A pointer to a kmalloced struct sg_table on success, negative
+ * error code cast to an error pointer on failure.
+ */
+struct sg_table *i915_sg_from_mm_node(const struct drm_mm_node *node,
+ u64 region_start)
+{
+ const u64 max_segment = SZ_1G; /* Do we have a limit on this? */
+ u64 segment_pages = max_segment >> PAGE_SHIFT;
+ u64 block_size, offset, prev_end;
+ struct sg_table *st;
+ struct scatterlist *sg;
+
+ st = kmalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return ERR_PTR(-ENOMEM);
+
+ if (sg_alloc_table(st, DIV_ROUND_UP(node->size, segment_pages),
+ GFP_KERNEL)) {
+ kfree(st);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ sg = st->sgl;
+ st->nents = 0;
+ prev_end = (resource_size_t)-1;
+ block_size = node->size << PAGE_SHIFT;
+ offset = node->start << PAGE_SHIFT;
+
+ while (block_size) {
+ u64 len;
+
+ if (offset != prev_end || sg->length >= max_segment) {
+ if (st->nents)
+ sg = __sg_next(sg);
+
+ sg_dma_address(sg) = region_start + offset;
+ sg_dma_len(sg) = 0;
+ sg->length = 0;
+ st->nents++;
+ }
+
+ len = min(block_size, max_segment - sg->length);
+ sg->length += len;
+ sg_dma_len(sg) += len;
+
+ offset += len;
+ block_size -= len;
+
+ prev_end = offset;
+ }
+
+ sg_mark_end(sg);
+ i915_sg_trim(st);
+
+ return st;
+}
+
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/scatterlist.c"
#endif
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.h b/drivers/gpu/drm/i915/i915_scatterlist.h
index b96baad66a3a..5acca45ea981 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.h
+++ b/drivers/gpu/drm/i915/i915_scatterlist.h
@@ -13,6 +13,8 @@
#include "i915_gem.h"
+struct drm_mm_node;
+
/*
* Optimised SGL iterator for GEM objects
*/
@@ -141,4 +143,6 @@ static inline unsigned int i915_sg_segment_size(void)
bool i915_sg_trim(struct sg_table *orig_st);
+struct sg_table *i915_sg_from_mm_node(const struct drm_mm_node *node,
+ u64 region_start);
#endif
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index d98e8b81d322..4092cc987679 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -28,6 +28,11 @@ static const struct {
},
};
+struct intel_region_reserve {
+ struct list_head link;
+ void *node;
+};
+
struct intel_memory_region *
intel_memory_region_lookup(struct drm_i915_private *i915,
u16 class, u16 instance)
@@ -58,146 +63,61 @@ intel_memory_region_by_type(struct drm_i915_private *i915,
return NULL;
}
-static u64
-intel_memory_region_free_pages(struct intel_memory_region *mem,
- struct list_head *blocks)
+/**
+ * intel_memory_region_unreserve - Unreserve all previously reserved
+ * ranges
+ * @mem: The region containing the reserved ranges.
+ */
+void intel_memory_region_unreserve(struct intel_memory_region *mem)
{
- struct i915_buddy_block *block, *on;
- u64 size = 0;
+ struct intel_region_reserve *reserve, *next;
- list_for_each_entry_safe(block, on, blocks, link) {
- size += i915_buddy_block_size(&mem->mm, block);
- i915_buddy_free(&mem->mm, block);
- }
- INIT_LIST_HEAD(blocks);
+ if (!mem->priv_ops || !mem->priv_ops->free)
+ return;
- return size;
-}
-
-void
-__intel_memory_region_put_pages_buddy(struct intel_memory_region *mem,
- struct list_head *blocks)
-{
mutex_lock(&mem->mm_lock);
- mem->avail += intel_memory_region_free_pages(mem, blocks);
- mutex_unlock(&mem->mm_lock);
-}
-
-void
-__intel_memory_region_put_block_buddy(struct i915_buddy_block *block)
-{
- struct list_head blocks;
-
- INIT_LIST_HEAD(&blocks);
- list_add(&block->link, &blocks);
- __intel_memory_region_put_pages_buddy(block->private, &blocks);
-}
-
-int
-__intel_memory_region_get_pages_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags,
- struct list_head *blocks)
-{
- unsigned int min_order = 0;
- unsigned long n_pages;
-
- GEM_BUG_ON(!IS_ALIGNED(size, mem->mm.chunk_size));
- GEM_BUG_ON(!list_empty(blocks));
-
- if (flags & I915_ALLOC_MIN_PAGE_SIZE) {
- min_order = ilog2(mem->min_page_size) -
- ilog2(mem->mm.chunk_size);
- }
-
- if (flags & I915_ALLOC_CONTIGUOUS) {
- size = roundup_pow_of_two(size);
- min_order = ilog2(size) - ilog2(mem->mm.chunk_size);
+ list_for_each_entry_safe(reserve, next, &mem->reserved, link) {
+ list_del(&reserve->link);
+ mem->priv_ops->free(mem, reserve->node);
+ kfree(reserve);
}
-
- if (size > mem->mm.size)
- return -E2BIG;
-
- n_pages = size >> ilog2(mem->mm.chunk_size);
-
- mutex_lock(&mem->mm_lock);
-
- do {
- struct i915_buddy_block *block;
- unsigned int order;
-
- order = fls(n_pages) - 1;
- GEM_BUG_ON(order > mem->mm.max_order);
- GEM_BUG_ON(order < min_order);
-
- do {
- block = i915_buddy_alloc(&mem->mm, order);
- if (!IS_ERR(block))
- break;
-
- if (order-- == min_order)
- goto err_free_blocks;
- } while (1);
-
- n_pages -= BIT(order);
-
- block->private = mem;
- list_add_tail(&block->link, blocks);
-
- if (!n_pages)
- break;
- } while (1);
-
- mem->avail -= size;
mutex_unlock(&mem->mm_lock);
- return 0;
-
-err_free_blocks:
- intel_memory_region_free_pages(mem, blocks);
- mutex_unlock(&mem->mm_lock);
- return -ENXIO;
}
-struct i915_buddy_block *
-__intel_memory_region_get_block_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags)
+/**
+ * intel_memory_region_reserve - Reserve a memory range
+ * @mem: The region for which we want to reserve a range.
+ * @offset: Start of the range to reserve.
+ * @size: The size of the range to reserve.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_memory_region_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size)
{
- struct i915_buddy_block *block;
- LIST_HEAD(blocks);
int ret;
+ struct intel_region_reserve *reserve;
- ret = __intel_memory_region_get_pages_buddy(mem, size, flags, &blocks);
- if (ret)
- return ERR_PTR(ret);
+ if (!mem->priv_ops || !mem->priv_ops->reserve)
+ return -EINVAL;
- block = list_first_entry(&blocks, typeof(*block), link);
- list_del_init(&block->link);
- return block;
-}
+ reserve = kzalloc(sizeof(*reserve), GFP_KERNEL);
+ if (!reserve)
+ return -ENOMEM;
-int intel_memory_region_init_buddy(struct intel_memory_region *mem)
-{
- return i915_buddy_init(&mem->mm, resource_size(&mem->region),
- PAGE_SIZE);
-}
-
-void intel_memory_region_release_buddy(struct intel_memory_region *mem)
-{
- i915_buddy_free_list(&mem->mm, &mem->reserved);
- i915_buddy_fini(&mem->mm);
-}
-
-int intel_memory_region_reserve(struct intel_memory_region *mem,
- u64 offset, u64 size)
-{
- int ret;
+ reserve->node = mem->priv_ops->reserve(mem, offset, size);
+ if (IS_ERR(reserve->node)) {
+ ret = PTR_ERR(reserve->node);
+ kfree(reserve);
+ return ret;
+ }
mutex_lock(&mem->mm_lock);
- ret = i915_buddy_alloc_range(&mem->mm, &mem->reserved, offset, size);
+ list_add_tail(&reserve->link, &mem->reserved);
mutex_unlock(&mem->mm_lock);
- return ret;
+ return 0;
}
struct intel_memory_region *
@@ -206,6 +126,8 @@ intel_memory_region_create(struct drm_i915_private *i915,
resource_size_t size,
resource_size_t min_page_size,
resource_size_t io_start,
+ u16 type,
+ u16 instance,
const struct intel_memory_region_ops *ops)
{
struct intel_memory_region *mem;
@@ -222,6 +144,8 @@ intel_memory_region_create(struct drm_i915_private *i915,
mem->ops = ops;
mem->total = size;
mem->avail = mem->total;
+ mem->type = type;
+ mem->instance = instance;
mutex_init(&mem->objects.lock);
INIT_LIST_HEAD(&mem->objects.list);
@@ -259,6 +183,7 @@ static void __intel_memory_region_destroy(struct kref *kref)
struct intel_memory_region *mem =
container_of(kref, typeof(*mem), kref);
+ intel_memory_region_unreserve(mem);
if (mem->ops->release)
mem->ops->release(mem);
@@ -296,15 +221,15 @@ int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
instance = intel_region_map[i].instance;
switch (type) {
case INTEL_MEMORY_SYSTEM:
- mem = i915_gem_shmem_setup(i915);
+ mem = i915_gem_shmem_setup(i915, type, instance);
break;
case INTEL_MEMORY_STOLEN_LOCAL:
- mem = i915_gem_stolen_lmem_setup(i915);
+ mem = i915_gem_stolen_lmem_setup(i915, type, instance);
if (!IS_ERR(mem))
i915->mm.stolen_region = mem;
break;
case INTEL_MEMORY_STOLEN_SYSTEM:
- mem = i915_gem_stolen_smem_setup(i915);
+ mem = i915_gem_stolen_smem_setup(i915, type, instance);
if (!IS_ERR(mem))
i915->mm.stolen_region = mem;
break;
@@ -321,9 +246,6 @@ int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
}
mem->id = i;
- mem->type = type;
- mem->instance = instance;
-
i915->mm.regions[i] = mem;
}
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index d24ce5a0b30b..e69cde13daf2 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -13,8 +13,6 @@
#include <drm/drm_mm.h>
#include <drm/i915_drm.h>
-#include "i915_buddy.h"
-
struct drm_i915_private;
struct drm_i915_gem_object;
struct intel_memory_region;
@@ -25,6 +23,7 @@ enum intel_memory_type {
INTEL_MEMORY_LOCAL = I915_MEMORY_CLASS_DEVICE,
INTEL_MEMORY_STOLEN_SYSTEM,
INTEL_MEMORY_STOLEN_LOCAL,
+ INTEL_MEMORY_MOCK,
};
enum intel_region_id {
@@ -59,10 +58,19 @@ struct intel_memory_region_ops {
unsigned int flags);
};
+struct intel_memory_region_private_ops {
+ void *(*reserve)(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size);
+ void (*free)(struct intel_memory_region *mem,
+ void *node);
+};
+
struct intel_memory_region {
struct drm_i915_private *i915;
const struct intel_memory_region_ops *ops;
+ const struct intel_memory_region_private_ops *priv_ops;
struct io_mapping iomap;
struct resource region;
@@ -70,7 +78,6 @@ struct intel_memory_region {
/* For fake LMEM */
struct drm_mm_node fake_mappable;
- struct i915_buddy_mm mm;
struct mutex mm_lock;
struct kref kref;
@@ -95,36 +102,26 @@ struct intel_memory_region {
struct list_head list;
struct list_head purgeable;
} objects;
+
+ size_t chunk_size;
+ unsigned int max_order;
+ bool is_range_manager;
+
+ void *region_private;
};
struct intel_memory_region *
intel_memory_region_lookup(struct drm_i915_private *i915,
u16 class, u16 instance);
-int intel_memory_region_init_buddy(struct intel_memory_region *mem);
-void intel_memory_region_release_buddy(struct intel_memory_region *mem);
-
-int __intel_memory_region_get_pages_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags,
- struct list_head *blocks);
-struct i915_buddy_block *
-__intel_memory_region_get_block_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags);
-void __intel_memory_region_put_pages_buddy(struct intel_memory_region *mem,
- struct list_head *blocks);
-void __intel_memory_region_put_block_buddy(struct i915_buddy_block *block);
-
-int intel_memory_region_reserve(struct intel_memory_region *mem,
- u64 offset, u64 size);
-
struct intel_memory_region *
intel_memory_region_create(struct drm_i915_private *i915,
resource_size_t start,
resource_size_t size,
resource_size_t min_page_size,
resource_size_t io_start,
+ u16 type,
+ u16 instance,
const struct intel_memory_region_ops *ops);
struct intel_memory_region *
@@ -141,4 +138,9 @@ __printf(2, 3) void
intel_memory_region_set_name(struct intel_memory_region *mem,
const char *fmt, ...);
+void intel_memory_region_unreserve(struct intel_memory_region *mem);
+
+int intel_memory_region_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size);
#endif
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
new file mode 100644
index 000000000000..c8ac118c21f6
--- /dev/null
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -0,0 +1,220 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_device.h>
+
+#include "i915_drv.h"
+#include "i915_scatterlist.h"
+
+#include "intel_region_ttm.h"
+
+/**
+ * DOC: TTM support structure
+ *
+ * The code in this file deals with setting up memory managers for TTM
+ * LMEM and MOCK regions and converting the output from
+ * the managers to struct sg_table, Basically providing the mapping from
+ * i915 GEM regions to TTM memory types and resource managers.
+ */
+
+/* A Zero-initialized driver for now. We don't have a TTM backend yet. */
+static struct ttm_device_funcs i915_ttm_bo_driver;
+
+/**
+ * intel_region_ttm_device_init - Initialize a TTM device
+ * @dev_priv: Pointer to an i915 device private structure.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_region_ttm_device_init(struct drm_i915_private *dev_priv)
+{
+ struct drm_device *drm = &dev_priv->drm;
+
+ return ttm_device_init(&dev_priv->bdev, &i915_ttm_bo_driver,
+ drm->dev, drm->anon_inode->i_mapping,
+ drm->vma_offset_manager, false, false);
+}
+
+/**
+ * intel_region_ttm_device_fini - Finalize a TTM device
+ * @dev_priv: Pointer to an i915 device private structure.
+ */
+void intel_region_ttm_device_fini(struct drm_i915_private *dev_priv)
+{
+ ttm_device_fini(&dev_priv->bdev);
+}
+
+/*
+ * Map the i915 memory regions to TTM memory types. We use the
+ * driver-private types for now, reserving TTM_PL_VRAM for stolen
+ * memory and TTM_PL_TT for GGTT use if decided to implement this.
+ */
+static int intel_region_to_ttm_type(struct intel_memory_region *mem)
+{
+ int type;
+
+ GEM_BUG_ON(mem->type != INTEL_MEMORY_LOCAL &&
+ mem->type != INTEL_MEMORY_MOCK);
+
+ type = mem->instance + TTM_PL_PRIV;
+ GEM_BUG_ON(type >= TTM_NUM_MEM_TYPES);
+
+ return type;
+}
+
+static void *intel_region_ttm_node_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size)
+{
+ struct ttm_resource_manager *man = mem->region_private;
+ struct ttm_place place = {};
+ struct ttm_resource res = {};
+ struct ttm_buffer_object mock_bo = {};
+ int ret;
+
+ /*
+ * Having to use a mock_bo is unfortunate but stems from some
+ * drivers having private managers that insist to know what the
+ * allocate memory is intended for, using it to send private
+ * data to the manager. Also recently the bo has been used to send
+ * alignment info to the manager. Assume that apart from the latter,
+ * none of the managers we use will ever access the buffer object
+ * members, hoping we can pass the alignment info in the
+ * struct ttm_place in the future.
+ */
+
+ place.fpfn = offset >> PAGE_SHIFT;
+ place.lpfn = place.fpfn + (size >> PAGE_SHIFT);
+ res.num_pages = size >> PAGE_SHIFT;
+ ret = man->func->alloc(man, &mock_bo, &place, &res);
+ if (ret == -ENOSPC)
+ ret = -ENXIO;
+
+ return ret ? ERR_PTR(ret) : res.mm_node;
+}
+
+/**
+ * intel_region_ttm_node_free - Free a node allocated from a resource manager
+ * @mem: The region the node was allocated from.
+ * @node: The opaque node representing an allocation.
+ */
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node)
+{
+ struct ttm_resource_manager *man = mem->region_private;
+ struct ttm_resource res = {};
+
+ res.mm_node = node;
+ man->func->free(man, &res);
+}
+
+static const struct intel_memory_region_private_ops priv_ops = {
+ .reserve = intel_region_ttm_node_reserve,
+ .free = intel_region_ttm_node_free,
+};
+
+int intel_region_ttm_init(struct intel_memory_region *mem)
+{
+ struct ttm_device *bdev = &mem->i915->bdev;
+ int mem_type = intel_region_to_ttm_type(mem);
+ int ret;
+
+ ret = ttm_range_man_init(bdev, mem_type, false,
+ resource_size(&mem->region) >> PAGE_SHIFT);
+ if (ret)
+ return ret;
+
+ mem->chunk_size = PAGE_SIZE;
+ mem->max_order =
+ get_order(rounddown_pow_of_two(resource_size(&mem->region)));
+ mem->is_range_manager = true;
+ mem->priv_ops = &priv_ops;
+ mem->region_private = ttm_manager_type(bdev, mem_type);
+
+ return 0;
+}
+
+/**
+ * intel_region_ttm_fini - Finalize a TTM region.
+ * @mem: The memory region
+ *
+ * This functions takes down the TTM resource manager associated with the
+ * memory region, and if it was registered with the TTM device,
+ * removes that registration.
+ */
+void intel_region_ttm_fini(struct intel_memory_region *mem)
+{
+ int ret;
+
+ ret = ttm_range_man_fini(&mem->i915->bdev,
+ intel_region_to_ttm_type(mem));
+ GEM_WARN_ON(ret);
+ mem->region_private = NULL;
+}
+
+/**
+ * intel_region_ttm_node_to_st - Convert an opaque TTM resource manager node
+ * to an sg_table.
+ * @mem: The memory region.
+ * @node: The resource manager node obtained from the TTM resource manager.
+ *
+ * The gem backends typically use sg-tables for operations on the underlying
+ * io_memory. So provide a way for the backends to translate the
+ * nodes they are handed from TTM to sg-tables.
+ *
+ * Return: A malloced sg_table on success, an error pointer on failure.
+ */
+struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
+ void *node)
+{
+ return i915_sg_from_mm_node(node, mem->region.start);
+}
+
+/**
+ * intel_region_ttm_node_alloc - Allocate memory resources from a region
+ * @mem: The memory region,
+ * @size: The requested size in bytes
+ * @flags: Allocation flags
+ *
+ * This functionality is provided only for callers that need to allocate
+ * memory from standalone TTM range managers, without the TTM eviction
+ * functionality. Don't use if you are not completely sure that's the
+ * case. The returned opaque node can be converted to an sg_table using
+ * intel_region_ttm_node_to_st(), and can be freed using
+ * intel_region_ttm_node_free().
+ *
+ * Return: A valid pointer on success, an error pointer on failure.
+ */
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags)
+{
+ struct ttm_resource_manager *man = mem->region_private;
+ struct ttm_place place = {};
+ struct ttm_resource res = {};
+ struct ttm_buffer_object mock_bo = {};
+ int ret;
+
+ /*
+ * We ignore the flags for now since we're using the range
+ * manager and contigous and min page size would be fulfilled
+ * by default if size is min page size aligned.
+ */
+ res.num_pages = size >> PAGE_SHIFT;
+
+ if (mem->is_range_manager) {
+ if (size >= SZ_1G)
+ mock_bo.page_alignment = SZ_1G >> PAGE_SHIFT;
+ else if (size >= SZ_2M)
+ mock_bo.page_alignment = SZ_2M >> PAGE_SHIFT;
+ else if (size >= SZ_64K)
+ mock_bo.page_alignment = SZ_64K >> PAGE_SHIFT;
+ }
+
+ ret = man->func->alloc(man, &mock_bo, &place, &res);
+ if (ret == -ENOSPC)
+ ret = -ENXIO;
+ return ret ? ERR_PTR(ret) : res.mm_node;
+}
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.h b/drivers/gpu/drm/i915/intel_region_ttm.h
new file mode 100644
index 000000000000..1c82c6c3429d
--- /dev/null
+++ b/drivers/gpu/drm/i915/intel_region_ttm.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef _INTEL_REGION_TTM_H_
+#define _INTEL_REGION_TTM_H_
+
+#include <linux/types.h>
+
+#include "i915_selftest.h"
+
+struct drm_i915_private;
+struct intel_memory_region;
+
+int intel_region_ttm_device_init(struct drm_i915_private *dev_priv);
+
+void intel_region_ttm_device_fini(struct drm_i915_private *dev_priv);
+
+int intel_region_ttm_init(struct intel_memory_region *mem);
+
+void intel_region_ttm_fini(struct intel_memory_region *mem);
+
+struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
+ void *node);
+
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags);
+
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node);
+#endif /* _INTEL_REGION_TTM_H_ */
diff --git a/drivers/gpu/drm/i915/selftests/i915_buddy.c b/drivers/gpu/drm/i915/selftests/i915_buddy.c
deleted file mode 100644
index f0f5c4df8dbc..000000000000
--- a/drivers/gpu/drm/i915/selftests/i915_buddy.c
+++ /dev/null
@@ -1,789 +0,0 @@
-// SPDX-License-Identifier: MIT
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#include <linux/prime_numbers.h>
-
-#include "../i915_selftest.h"
-#include "i915_random.h"
-
-static void __igt_dump_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block,
- bool buddy)
-{
- pr_err("block info: header=%llx, state=%u, order=%d, offset=%llx size=%llx root=%s buddy=%s\n",
- block->header,
- i915_buddy_block_state(block),
- i915_buddy_block_order(block),
- i915_buddy_block_offset(block),
- i915_buddy_block_size(mm, block),
- yesno(!block->parent),
- yesno(buddy));
-}
-
-static void igt_dump_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *buddy;
-
- __igt_dump_block(mm, block, false);
-
- buddy = get_buddy(block);
- if (buddy)
- __igt_dump_block(mm, buddy, true);
-}
-
-static int igt_check_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *buddy;
- unsigned int block_state;
- u64 block_size;
- u64 offset;
- int err = 0;
-
- block_state = i915_buddy_block_state(block);
-
- if (block_state != I915_BUDDY_ALLOCATED &&
- block_state != I915_BUDDY_FREE &&
- block_state != I915_BUDDY_SPLIT) {
- pr_err("block state mismatch\n");
- err = -EINVAL;
- }
-
- block_size = i915_buddy_block_size(mm, block);
- offset = i915_buddy_block_offset(block);
-
- if (block_size < mm->chunk_size) {
- pr_err("block size smaller than min size\n");
- err = -EINVAL;
- }
-
- if (!is_power_of_2(block_size)) {
- pr_err("block size not power of two\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(block_size, mm->chunk_size)) {
- pr_err("block size not aligned to min size\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(offset, mm->chunk_size)) {
- pr_err("block offset not aligned to min size\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(offset, block_size)) {
- pr_err("block offset not aligned to block size\n");
- err = -EINVAL;
- }
-
- buddy = get_buddy(block);
-
- if (!buddy && block->parent) {
- pr_err("buddy has gone fishing\n");
- err = -EINVAL;
- }
-
- if (buddy) {
- if (i915_buddy_block_offset(buddy) != (offset ^ block_size)) {
- pr_err("buddy has wrong offset\n");
- err = -EINVAL;
- }
-
- if (i915_buddy_block_size(mm, buddy) != block_size) {
- pr_err("buddy size mismatch\n");
- err = -EINVAL;
- }
-
- if (i915_buddy_block_state(buddy) == block_state &&
- block_state == I915_BUDDY_FREE) {
- pr_err("block and its buddy are free\n");
- err = -EINVAL;
- }
- }
-
- return err;
-}
-
-static int igt_check_blocks(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 expected_size,
- bool is_contiguous)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_block *prev;
- u64 total;
- int err = 0;
-
- block = NULL;
- prev = NULL;
- total = 0;
-
- list_for_each_entry(block, blocks, link) {
- err = igt_check_block(mm, block);
-
- if (!i915_buddy_block_is_allocated(block)) {
- pr_err("block not allocated\n"),
- err = -EINVAL;
- }
-
- if (is_contiguous && prev) {
- u64 prev_block_size;
- u64 prev_offset;
- u64 offset;
-
- prev_offset = i915_buddy_block_offset(prev);
- prev_block_size = i915_buddy_block_size(mm, prev);
- offset = i915_buddy_block_offset(block);
-
- if (offset != (prev_offset + prev_block_size)) {
- pr_err("block offset mismatch\n");
- err = -EINVAL;
- }
- }
-
- if (err)
- break;
-
- total += i915_buddy_block_size(mm, block);
- prev = block;
- }
-
- if (!err) {
- if (total != expected_size) {
- pr_err("size mismatch, expected=%llx, found=%llx\n",
- expected_size, total);
- err = -EINVAL;
- }
- return err;
- }
-
- if (prev) {
- pr_err("prev block, dump:\n");
- igt_dump_block(mm, prev);
- }
-
- if (block) {
- pr_err("bad block, dump:\n");
- igt_dump_block(mm, block);
- }
-
- return err;
-}
-
-static int igt_check_mm(struct i915_buddy_mm *mm)
-{
- struct i915_buddy_block *root;
- struct i915_buddy_block *prev;
- unsigned int i;
- u64 total;
- int err = 0;
-
- if (!mm->n_roots) {
- pr_err("n_roots is zero\n");
- return -EINVAL;
- }
-
- if (mm->n_roots != hweight64(mm->size)) {
- pr_err("n_roots mismatch, n_roots=%u, expected=%lu\n",
- mm->n_roots, hweight64(mm->size));
- return -EINVAL;
- }
-
- root = NULL;
- prev = NULL;
- total = 0;
-
- for (i = 0; i < mm->n_roots; ++i) {
- struct i915_buddy_block *block;
- unsigned int order;
-
- root = mm->roots[i];
- if (!root) {
- pr_err("root(%u) is NULL\n", i);
- err = -EINVAL;
- break;
- }
-
- err = igt_check_block(mm, root);
-
- if (!i915_buddy_block_is_free(root)) {
- pr_err("root not free\n");
- err = -EINVAL;
- }
-
- order = i915_buddy_block_order(root);
-
- if (!i) {
- if (order != mm->max_order) {
- pr_err("max order root missing\n");
- err = -EINVAL;
- }
- }
-
- if (prev) {
- u64 prev_block_size;
- u64 prev_offset;
- u64 offset;
-
- prev_offset = i915_buddy_block_offset(prev);
- prev_block_size = i915_buddy_block_size(mm, prev);
- offset = i915_buddy_block_offset(root);
-
- if (offset != (prev_offset + prev_block_size)) {
- pr_err("root offset mismatch\n");
- err = -EINVAL;
- }
- }
-
- block = list_first_entry_or_null(&mm->free_list[order],
- struct i915_buddy_block,
- link);
- if (block != root) {
- pr_err("root mismatch at order=%u\n", order);
- err = -EINVAL;
- }
-
- if (err)
- break;
-
- prev = root;
- total += i915_buddy_block_size(mm, root);
- }
-
- if (!err) {
- if (total != mm->size) {
- pr_err("expected mm size=%llx, found=%llx\n", mm->size,
- total);
- err = -EINVAL;
- }
- return err;
- }
-
- if (prev) {
- pr_err("prev root(%u), dump:\n", i - 1);
- igt_dump_block(mm, prev);
- }
-
- if (root) {
- pr_err("bad root(%u), dump:\n", i);
- igt_dump_block(mm, root);
- }
-
- return err;
-}
-
-static void igt_mm_config(u64 *size, u64 *chunk_size)
-{
- I915_RND_STATE(prng);
- u32 s, ms;
-
- /* Nothing fancy, just try to get an interesting bit pattern */
-
- prandom_seed_state(&prng, i915_selftest.random_seed);
-
- /* Let size be a random number of pages up to 8 GB (2M pages) */
- s = 1 + i915_prandom_u32_max_state((BIT(33 - 12)) - 1, &prng);
- /* Let the chunk size be a random power of 2 less than size */
- ms = BIT(i915_prandom_u32_max_state(ilog2(s), &prng));
- /* Round size down to the chunk size */
- s &= -ms;
-
- /* Convert from pages to bytes */
- *chunk_size = (u64)ms << 12;
- *size = (u64)s << 12;
-}
-
-static int igt_buddy_alloc_smoke(void *arg)
-{
- struct i915_buddy_mm mm;
- IGT_TIMEOUT(end_time);
- I915_RND_STATE(prng);
- u64 chunk_size;
- u64 mm_size;
- int *order;
- int err, i;
-
- igt_mm_config(&mm_size, &chunk_size);
-
- pr_info("buddy_init with size=%llx, chunk_size=%llx\n", mm_size, chunk_size);
-
- err = i915_buddy_init(&mm, mm_size, chunk_size);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
-
- order = i915_random_order(mm.max_order + 1, &prng);
- if (!order)
- goto out_fini;
-
- for (i = 0; i <= mm.max_order; ++i) {
- struct i915_buddy_block *block;
- int max_order = order[i];
- bool timeout = false;
- LIST_HEAD(blocks);
- int order;
- u64 total;
-
- err = igt_check_mm(&mm);
- if (err) {
- pr_err("pre-mm check failed, abort\n");
- break;
- }
-
- pr_info("filling from max_order=%u\n", max_order);
-
- order = max_order;
- total = 0;
-
- do {
-retry:
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- err = PTR_ERR(block);
- if (err == -ENOMEM) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- } else {
- if (order--) {
- err = 0;
- goto retry;
- }
-
- pr_err("buddy_alloc with order=%d failed(%d)\n",
- order, err);
- }
-
- break;
- }
-
- list_add_tail(&block->link, &blocks);
-
- if (i915_buddy_block_order(block) != order) {
- pr_err("buddy_alloc order mismatch\n");
- err = -EINVAL;
- break;
- }
-
- total += i915_buddy_block_size(&mm, block);
-
- if (__igt_timeout(end_time, NULL)) {
- timeout = true;
- break;
- }
- } while (total < mm.size);
-
- if (!err)
- err = igt_check_blocks(&mm, &blocks, total, false);
-
- i915_buddy_free_list(&mm, &blocks);
-
- if (!err) {
- err = igt_check_mm(&mm);
- if (err)
- pr_err("post-mm check failed\n");
- }
-
- if (err || timeout)
- break;
-
- cond_resched();
- }
-
- if (err == -ENOMEM)
- err = 0;
-
- kfree(order);
-out_fini:
- i915_buddy_fini(&mm);
-
- return err;
-}
-
-static int igt_buddy_alloc_pessimistic(void *arg)
-{
- const unsigned int max_order = 16;
- struct i915_buddy_block *block, *bn;
- struct i915_buddy_mm mm;
- unsigned int order;
- LIST_HEAD(blocks);
- int err;
-
- /*
- * Create a pot-sized mm, then allocate one of each possible
- * order within. This should leave the mm with exactly one
- * page left.
- */
-
- err = i915_buddy_init(&mm, PAGE_SIZE << max_order, PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (order = 0; order < max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
-
- list_add_tail(&block->link, &blocks);
- }
-
- /* And now the last remaining block available */
- block = i915_buddy_alloc(&mm, 0);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM on final alloc\n");
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &blocks);
-
- /* Should be completely full! */
- for (order = max_order; order--; ) {
- block = i915_buddy_alloc(&mm, order);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at order %d, it should be full!",
- order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
- block = list_last_entry(&blocks, typeof(*block), link);
- list_del(&block->link);
- i915_buddy_free(&mm, block);
-
- /* As we free in increasing size, we make available larger blocks */
- order = 1;
- list_for_each_entry_safe(block, bn, &blocks, link) {
- list_del(&block->link);
- i915_buddy_free(&mm, block);
-
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc (realloc) hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
- i915_buddy_free(&mm, block);
- order++;
- }
-
- /* To confirm, now the whole mm should be available */
- block = i915_buddy_alloc(&mm, max_order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc (realloc) hit -ENOMEM with order=%d\n",
- max_order);
- err = PTR_ERR(block);
- goto err;
- }
- i915_buddy_free(&mm, block);
-
-err:
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_optimistic(void *arg)
-{
- const int max_order = 16;
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- LIST_HEAD(blocks);
- int order;
- int err;
-
- /*
- * Create a mm with one block of each order available, and
- * try to allocate them all.
- */
-
- err = i915_buddy_init(&mm,
- PAGE_SIZE * ((1 << (max_order + 1)) - 1),
- PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (order = 0; order <= max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
-
- list_add_tail(&block->link, &blocks);
- }
-
- /* Should be completely full! */
- block = i915_buddy_alloc(&mm, 0);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded, it should be full!");
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
-
-err:
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_pathological(void *arg)
-{
- const int max_order = 16;
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- LIST_HEAD(blocks);
- LIST_HEAD(holes);
- int order, top;
- int err;
-
- /*
- * Create a pot-sized mm, then allocate one of each possible
- * order within. This should leave the mm with exactly one
- * page left. Free the largest block, then whittle down again.
- * Eventually we will have a fully 50% fragmented mm.
- */
-
- err = i915_buddy_init(&mm, PAGE_SIZE << max_order, PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (top = max_order; top; top--) {
- /* Make room by freeing the largest allocated block */
- block = list_first_entry_or_null(&blocks, typeof(*block), link);
- if (block) {
- list_del(&block->link);
- i915_buddy_free(&mm, block);
- }
-
- for (order = top; order--; ) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d, top=%d\n",
- order, top);
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &blocks);
- }
-
- /* There should be one final page for this sub-allocation */
- block = i915_buddy_alloc(&mm, 0);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM for hole\n");
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &holes);
-
- block = i915_buddy_alloc(&mm, top);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at top-order %d/%d, it should be full!",
- top, max_order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
- i915_buddy_free_list(&mm, &holes);
-
- /* Nothing larger than blocks of chunk_size now available */
- for (order = 1; order <= max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at order %d, it should be full!",
- order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
-err:
- list_splice_tail(&holes, &blocks);
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_range(void *arg)
-{
- struct i915_buddy_mm mm;
- unsigned long page_num;
- LIST_HEAD(blocks);
- u64 chunk_size;
- u64 offset;
- u64 size;
- u64 rem;
- int err;
-
- igt_mm_config(&size, &chunk_size);
-
- pr_info("buddy_init with size=%llx, chunk_size=%llx\n", size, chunk_size);
-
- err = i915_buddy_init(&mm, size, chunk_size);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
-
- err = igt_check_mm(&mm);
- if (err) {
- pr_err("pre-mm check failed, abort, abort, abort!\n");
- goto err_fini;
- }
-
- rem = mm.size;
- offset = 0;
-
- for_each_prime_number_from(page_num, 1, ULONG_MAX - 1) {
- struct i915_buddy_block *block;
- LIST_HEAD(tmp);
-
- size = min(page_num * mm.chunk_size, rem);
-
- err = i915_buddy_alloc_range(&mm, &tmp, offset, size);
- if (err) {
- if (err == -ENOMEM) {
- pr_info("alloc_range hit -ENOMEM with size=%llx\n",
- size);
- } else {
- pr_err("alloc_range with offset=%llx, size=%llx failed(%d)\n",
- offset, size, err);
- }
-
- break;
- }
-
- block = list_first_entry_or_null(&tmp,
- struct i915_buddy_block,
- link);
- if (!block) {
- pr_err("alloc_range has no blocks\n");
- err = -EINVAL;
- break;
- }
-
- if (i915_buddy_block_offset(block) != offset) {
- pr_err("alloc_range start offset mismatch, found=%llx, expected=%llx\n",
- i915_buddy_block_offset(block), offset);
- err = -EINVAL;
- }
-
- if (!err)
- err = igt_check_blocks(&mm, &tmp, size, true);
-
- list_splice_tail(&tmp, &blocks);
-
- if (err)
- break;
-
- offset += size;
-
- rem -= size;
- if (!rem)
- break;
-
- cond_resched();
- }
-
- if (err == -ENOMEM)
- err = 0;
-
- i915_buddy_free_list(&mm, &blocks);
-
- if (!err) {
- err = igt_check_mm(&mm);
- if (err)
- pr_err("post-mm check failed\n");
- }
-
-err_fini:
- i915_buddy_fini(&mm);
-
- return err;
-}
-
-static int igt_buddy_alloc_limit(void *arg)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- const u64 size = U64_MAX;
- int err;
-
- err = i915_buddy_init(&mm, size, PAGE_SIZE);
- if (err)
- return err;
-
- if (mm.max_order != I915_BUDDY_MAX_ORDER) {
- pr_err("mm.max_order(%d) != %d\n",
- mm.max_order, I915_BUDDY_MAX_ORDER);
- err = -EINVAL;
- goto out_fini;
- }
-
- block = i915_buddy_alloc(&mm, mm.max_order);
- if (IS_ERR(block)) {
- err = PTR_ERR(block);
- goto out_fini;
- }
-
- if (i915_buddy_block_order(block) != mm.max_order) {
- pr_err("block order(%d) != %d\n",
- i915_buddy_block_order(block), mm.max_order);
- err = -EINVAL;
- goto out_free;
- }
-
- if (i915_buddy_block_size(&mm, block) !=
- BIT_ULL(mm.max_order) * PAGE_SIZE) {
- pr_err("block size(%llu) != %llu\n",
- i915_buddy_block_size(&mm, block),
- BIT_ULL(mm.max_order) * PAGE_SIZE);
- err = -EINVAL;
- goto out_free;
- }
-
-out_free:
- i915_buddy_free(&mm, block);
-out_fini:
- i915_buddy_fini(&mm);
- return err;
-}
-
-int i915_buddy_mock_selftests(void)
-{
- static const struct i915_subtest tests[] = {
- SUBTEST(igt_buddy_alloc_pessimistic),
- SUBTEST(igt_buddy_alloc_optimistic),
- SUBTEST(igt_buddy_alloc_pathological),
- SUBTEST(igt_buddy_alloc_smoke),
- SUBTEST(igt_buddy_alloc_range),
- SUBTEST(igt_buddy_alloc_limit),
- };
-
- return i915_subtests(tests, NULL);
-}
diff --git a/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h b/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
index 3db34d3eea58..34e5caf38093 100644
--- a/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
+++ b/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
@@ -33,5 +33,4 @@ selftest(evict, i915_gem_evict_mock_selftests)
selftest(gtt, i915_gem_gtt_mock_selftests)
selftest(hugepages, i915_gem_huge_page_mock_selftests)
selftest(contexts, i915_gem_context_mock_selftests)
-selftest(buddy, i915_buddy_mock_selftests)
selftest(memory_region, intel_memory_region_mock_selftests)
diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
index f85fd8cbfbf5..c85d516b85cd 100644
--- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
@@ -57,9 +57,10 @@ static int igt_mock_fill(void *arg)
LIST_HEAD(objects);
int err = 0;
- page_size = mem->mm.chunk_size;
- max_pages = div64_u64(total, page_size);
+ page_size = mem->chunk_size;
rem = total;
+retry:
+ max_pages = div64_u64(rem, page_size);
for_each_prime_number_from(page_num, 1, max_pages) {
resource_size_t size = page_num * page_size;
@@ -85,6 +86,11 @@ static int igt_mock_fill(void *arg)
err = 0;
if (err == -ENXIO) {
if (page_num * page_size <= rem) {
+ if (mem->is_range_manager && max_pages > 1) {
+ max_pages >>= 1;
+ goto retry;
+ }
+
pr_err("%s failed, space still left in region\n",
__func__);
err = -EINVAL;
@@ -199,12 +205,18 @@ static int igt_mock_reserve(void *arg)
do {
u32 size = i915_prandom_u32_max_state(cur_avail, &prng);
+retry:
size = max_t(u32, round_up(size, PAGE_SIZE), PAGE_SIZE);
obj = igt_object_create(mem, &objects, size, 0);
if (IS_ERR(obj)) {
- if (PTR_ERR(obj) == -ENXIO)
+ if (PTR_ERR(obj) == -ENXIO) {
+ if (mem->is_range_manager &&
+ size > mem->chunk_size) {
+ size >>= 1;
+ goto retry;
+ }
break;
-
+ }
err = PTR_ERR(obj);
goto out_close;
}
@@ -220,7 +232,7 @@ static int igt_mock_reserve(void *arg)
out_close:
kfree(order);
close_objects(mem, &objects);
- i915_buddy_free_list(&mem->mm, &mem->reserved);
+ intel_memory_region_unreserve(mem);
return err;
}
@@ -240,7 +252,7 @@ static int igt_mock_contiguous(void *arg)
total = resource_size(&mem->region);
/* Min size */
- obj = igt_object_create(mem, &objects, mem->mm.chunk_size,
+ obj = igt_object_create(mem, &objects, mem->chunk_size,
I915_BO_ALLOC_CONTIGUOUS);
if (IS_ERR(obj))
return PTR_ERR(obj);
@@ -321,14 +333,16 @@ static int igt_mock_contiguous(void *arg)
min = target;
target = total >> 1;
- /* Make sure we can still allocate all the fragmented space */
- obj = igt_object_create(mem, &objects, target, 0);
- if (IS_ERR(obj)) {
- err = PTR_ERR(obj);
- goto err_close_objects;
- }
+ if (!mem->is_range_manager) {
+ /* Make sure we can still allocate all the fragmented space */
+ obj = igt_object_create(mem, &objects, target, 0);
+ if (IS_ERR(obj)) {
+ err = PTR_ERR(obj);
+ goto err_close_objects;
+ }
- igt_object_release(obj);
+ igt_object_release(obj);
+ }
/*
* Even though we have enough free space, we don't have a big enough
@@ -348,7 +362,7 @@ static int igt_mock_contiguous(void *arg)
}
target >>= 1;
- } while (target >= mem->mm.chunk_size);
+ } while (target >= mem->chunk_size);
err_close_objects:
list_splice_tail(&holes, &objects);
@@ -368,7 +382,7 @@ static int igt_mock_splintered_region(void *arg)
/*
* Sanity check we can still allocate everything even if the
- * mm.max_order != mm.size. i.e our starting address space size is not a
+ * max_order != mm.size. i.e our starting address space size is not a
* power-of-two.
*/
@@ -377,17 +391,10 @@ static int igt_mock_splintered_region(void *arg)
if (IS_ERR(mem))
return PTR_ERR(mem);
- if (mem->mm.size != size) {
- pr_err("%s size mismatch(%llu != %llu)\n",
- __func__, mem->mm.size, size);
- err = -EINVAL;
- goto out_put;
- }
-
expected_order = get_order(rounddown_pow_of_two(size));
- if (mem->mm.max_order != expected_order) {
+ if (mem->max_order != expected_order) {
pr_err("%s order mismatch(%u != %u)\n",
- __func__, mem->mm.max_order, expected_order);
+ __func__, mem->max_order, expected_order);
err = -EINVAL;
goto out_put;
}
@@ -408,12 +415,15 @@ static int igt_mock_splintered_region(void *arg)
* sure that does indeed hold true.
*/
- obj = igt_object_create(mem, &objects, size, I915_BO_ALLOC_CONTIGUOUS);
- if (!IS_ERR(obj)) {
- pr_err("%s too large contiguous allocation was not rejected\n",
- __func__);
- err = -EINVAL;
- goto out_close;
+ if (!mem->is_range_manager) {
+ obj = igt_object_create(mem, &objects, size,
+ I915_BO_ALLOC_CONTIGUOUS);
+ if (!IS_ERR(obj)) {
+ pr_err("%s too large contiguous allocation was not rejected\n",
+ __func__);
+ err = -EINVAL;
+ goto out_close;
+ }
}
obj = igt_object_create(mem, &objects, rounddown_pow_of_two(size),
@@ -432,68 +442,6 @@ static int igt_mock_splintered_region(void *arg)
return err;
}
-#ifndef SZ_8G
-#define SZ_8G BIT_ULL(33)
-#endif
-
-static int igt_mock_max_segment(void *arg)
-{
- const unsigned int max_segment = i915_sg_segment_size();
- struct intel_memory_region *mem = arg;
- struct drm_i915_private *i915 = mem->i915;
- struct drm_i915_gem_object *obj;
- struct i915_buddy_block *block;
- struct scatterlist *sg;
- LIST_HEAD(objects);
- u64 size;
- int err = 0;
-
- /*
- * While we may create very large contiguous blocks, we may need
- * to break those down for consumption elsewhere. In particular,
- * dma-mapping with scatterlist elements have an implicit limit of
- * UINT_MAX on each element.
- */
-
- size = SZ_8G;
- mem = mock_region_create(i915, 0, size, PAGE_SIZE, 0);
- if (IS_ERR(mem))
- return PTR_ERR(mem);
-
- obj = igt_object_create(mem, &objects, size, 0);
- if (IS_ERR(obj)) {
- err = PTR_ERR(obj);
- goto out_put;
- }
-
- size = 0;
- list_for_each_entry(block, &obj->mm.blocks, link) {
- if (i915_buddy_block_size(&mem->mm, block) > size)
- size = i915_buddy_block_size(&mem->mm, block);
- }
- if (size < max_segment) {
- pr_err("%s: Failed to create a huge contiguous block [> %u], largest block %lld\n",
- __func__, max_segment, size);
- err = -EINVAL;
- goto out_close;
- }
-
- for (sg = obj->mm.pages->sgl; sg; sg = sg_next(sg)) {
- if (sg->length > max_segment) {
- pr_err("%s: Created an oversized scatterlist entry, %u > %u\n",
- __func__, sg->length, max_segment);
- err = -EINVAL;
- goto out_close;
- }
- }
-
-out_close:
- close_objects(mem, &objects);
-out_put:
- intel_memory_region_put(mem);
- return err;
-}
-
static int igt_gpu_write_dw(struct intel_context *ce,
struct i915_vma *vma,
u32 dword,
@@ -1098,7 +1046,6 @@ int intel_memory_region_mock_selftests(void)
SUBTEST(igt_mock_fill),
SUBTEST(igt_mock_contiguous),
SUBTEST(igt_mock_splintered_region),
- SUBTEST(igt_mock_max_segment),
};
struct intel_memory_region *mem;
struct drm_i915_private *i915;
diff --git a/drivers/gpu/drm/i915/selftests/mock_gem_device.c b/drivers/gpu/drm/i915/selftests/mock_gem_device.c
index cf40004bc92a..d189c4bd4bef 100644
--- a/drivers/gpu/drm/i915/selftests/mock_gem_device.c
+++ b/drivers/gpu/drm/i915/selftests/mock_gem_device.c
@@ -32,6 +32,7 @@
#include "gt/intel_gt_requests.h"
#include "gt/mock_engine.h"
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "mock_request.h"
#include "mock_gem_device.h"
@@ -70,6 +71,7 @@ static void mock_device_release(struct drm_device *dev)
mock_fini_ggtt(&i915->ggtt);
destroy_workqueue(i915->wq);
+ intel_region_ttm_device_fini(i915);
intel_gt_driver_late_release(&i915->gt);
intel_memory_regions_driver_release(i915);
@@ -116,6 +118,7 @@ struct drm_i915_private *mock_gem_device(void)
#endif
struct drm_i915_private *i915;
struct pci_dev *pdev;
+ int ret;
pdev = kzalloc(sizeof(*pdev), GFP_KERNEL);
if (!pdev)
@@ -178,6 +181,10 @@ struct drm_i915_private *mock_gem_device(void)
atomic_inc(&i915->gt.wakeref.count); /* disable; no hw support */
i915->gt.awake = -ENODEV;
+ ret = intel_region_ttm_device_init(i915);
+ if (ret)
+ goto err_ttm;
+
i915->wq = alloc_ordered_workqueue("mock", 0);
if (!i915->wq)
goto err_drv;
@@ -201,6 +208,7 @@ struct drm_i915_private *mock_gem_device(void)
intel_engines_driver_register(i915);
i915->do_release = true;
+ ida_init(&i915->selftest.mock_region_instances);
return i915;
@@ -209,6 +217,8 @@ struct drm_i915_private *mock_gem_device(void)
err_unlock:
destroy_workqueue(i915->wq);
err_drv:
+ intel_region_ttm_device_fini(i915);
+err_ttm:
intel_gt_driver_late_release(&i915->gt);
intel_memory_regions_driver_release(i915);
drm_mode_config_cleanup(&i915->drm);
diff --git a/drivers/gpu/drm/i915/selftests/mock_region.c b/drivers/gpu/drm/i915/selftests/mock_region.c
index 5d2d010a1e22..eafc5a04975c 100644
--- a/drivers/gpu/drm/i915/selftests/mock_region.c
+++ b/drivers/gpu/drm/i915/selftests/mock_region.c
@@ -1,17 +1,56 @@
// SPDX-License-Identifier: MIT
/*
- * Copyright © 2019 Intel Corporation
+ * Copyright © 2019-2021 Intel Corporation
*/
+#include <linux/scatterlist.h>
+
+#include <drm/ttm/ttm_placement.h>
+
#include "gem/i915_gem_region.h"
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "mock_region.h"
+static void mock_region_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *pages)
+{
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ sg_free_table(pages);
+ kfree(pages);
+}
+
+static int mock_region_get_pages(struct drm_i915_gem_object *obj)
+{
+ unsigned int flags;
+ struct sg_table *pages;
+
+ flags = I915_ALLOC_MIN_PAGE_SIZE;
+ if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
+ flags |= I915_ALLOC_CONTIGUOUS;
+
+ obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
+ obj->base.size,
+ flags);
+ if (IS_ERR(obj->mm.st_mm_node))
+ return PTR_ERR(obj->mm.st_mm_node);
+
+ pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
+ if (IS_ERR(pages)) {
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ return PTR_ERR(pages);
+ }
+
+ __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
+
+ return 0;
+}
+
static const struct drm_i915_gem_object_ops mock_region_obj_ops = {
.name = "mock-region",
- .get_pages = i915_gem_object_get_pages_buddy,
- .put_pages = i915_gem_object_put_pages_buddy,
+ .get_pages = mock_region_get_pages,
+ .put_pages = mock_region_put_pages,
.release = i915_gem_object_release_memory_region,
};
@@ -23,7 +62,7 @@ static int mock_object_init(struct intel_memory_region *mem,
static struct lock_class_key lock_class;
struct drm_i915_private *i915 = mem->i915;
- if (size > mem->mm.size)
+ if (size > resource_size(&mem->region))
return -E2BIG;
drm_gem_private_object_init(&i915->drm, &obj->base, size);
@@ -38,9 +77,18 @@ static int mock_object_init(struct intel_memory_region *mem,
return 0;
}
+static void mock_region_fini(struct intel_memory_region *mem)
+{
+ struct drm_i915_private *i915 = mem->i915;
+ int instance = mem->instance;
+
+ intel_region_ttm_fini(mem);
+ ida_free(&i915->selftest.mock_region_instances, instance);
+}
+
static const struct intel_memory_region_ops mock_region_ops = {
- .init = intel_memory_region_init_buddy,
- .release = intel_memory_region_release_buddy,
+ .init = intel_region_ttm_init,
+ .release = mock_region_fini,
.init_object = mock_object_init,
};
@@ -51,6 +99,14 @@ mock_region_create(struct drm_i915_private *i915,
resource_size_t min_page_size,
resource_size_t io_start)
{
+ int instance = ida_alloc_max(&i915->selftest.mock_region_instances,
+ TTM_NUM_MEM_TYPES - TTM_PL_PRIV - 1,
+ GFP_KERNEL);
+
+ if (instance < 0)
+ return ERR_PTR(instance);
+
return intel_memory_region_create(i915, start, size, min_page_size,
- io_start, &mock_region_ops);
+ io_start, INTEL_MEMORY_MOCK, instance,
+ &mock_region_ops);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 04/15] drm/i915/ttm Initialize the ttm device and memory managers
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Temporarily remove the buddy allocator and related selftests
and hook up the TTM range manager for i915 regions.
Also modify the mock region selftests somewhat to account for a
fragmenting manager.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com> #v2
---
v2:
- Fix an error unwind in lmem_get_pages() (Reported by Matthew Auld)
- Break out and modify usage of i915_sg_dma_sizes() (Reported by Mattew Auld)
- Break out TTM changes to a separate patch (Reported by Christian König)
v3:
- Fix the same error unwind in mock_region_get_pages()
(Reported by Matthew Auld)
- Don't rely on new TTM functionality, but create a mock TTM device,
(Reported by Christian König)
v4:
- Use mock_gem_device rather than creating a separate ttm_device per
region.
---
drivers/gpu/drm/i915/Kconfig | 1 +
drivers/gpu/drm/i915/Makefile | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 59 +-
.../gpu/drm/i915/gem/i915_gem_object_types.h | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 120 ---
drivers/gpu/drm/i915/gem/i915_gem_region.h | 4 -
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.h | 9 +-
drivers/gpu/drm/i915/gt/intel_gt.c | 2 -
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 27 +-
drivers/gpu/drm/i915/i915_buddy.c | 435 ----------
drivers/gpu/drm/i915/i915_buddy.h | 131 ---
drivers/gpu/drm/i915/i915_drv.c | 8 +
drivers/gpu/drm/i915/i915_drv.h | 8 +-
drivers/gpu/drm/i915/i915_gem.c | 1 +
drivers/gpu/drm/i915/i915_globals.c | 1 -
drivers/gpu/drm/i915/i915_globals.h | 1 -
drivers/gpu/drm/i915/i915_scatterlist.c | 70 ++
drivers/gpu/drm/i915/i915_scatterlist.h | 4 +
drivers/gpu/drm/i915/intel_memory_region.c | 180 ++--
drivers/gpu/drm/i915/intel_memory_region.h | 44 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 220 +++++
drivers/gpu/drm/i915/intel_region_ttm.h | 32 +
drivers/gpu/drm/i915/selftests/i915_buddy.c | 789 ------------------
.../drm/i915/selftests/i915_mock_selftests.h | 1 -
.../drm/i915/selftests/intel_memory_region.c | 133 +--
.../gpu/drm/i915/selftests/mock_gem_device.c | 10 +
drivers/gpu/drm/i915/selftests/mock_region.c | 70 +-
30 files changed, 631 insertions(+), 1754 deletions(-)
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.c
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.h
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.c
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.h
delete mode 100644 drivers/gpu/drm/i915/selftests/i915_buddy.c
diff --git a/drivers/gpu/drm/i915/Kconfig b/drivers/gpu/drm/i915/Kconfig
index 93f4d059fc89..61ff5c178714 100644
--- a/drivers/gpu/drm/i915/Kconfig
+++ b/drivers/gpu/drm/i915/Kconfig
@@ -27,6 +27,7 @@ config DRM_I915
select SND_HDA_I915 if SND_HDA_CORE
select CEC_CORE if CEC_NOTIFIER
select VMAP_PFN
+ select DRM_TTM
help
Choose this option if you have a system that has "Intel Graphics
Media Accelerator" or "HD Graphics" integrated graphics,
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 6947495bf34b..4f22cac1c49b 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -50,6 +50,7 @@ i915-y += i915_drv.o \
intel_memory_region.o \
intel_pch.o \
intel_pm.o \
+ intel_region_ttm.o \
intel_runtime_pm.o \
intel_sideband.o \
intel_step.o \
@@ -160,7 +161,6 @@ gem-y += \
i915-y += \
$(gem-y) \
i915_active.o \
- i915_buddy.o \
i915_cmd_parser.o \
i915_gem_evict.o \
i915_gem_gtt.o \
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index f44bdd08f7cb..3b4aa28a076d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -4,16 +4,71 @@
*/
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_lmem.h"
#include "i915_drv.h"
+static void lmem_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *pages)
+{
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ obj->mm.dirty = false;
+ sg_free_table(pages);
+ kfree(pages);
+}
+
+static int lmem_get_pages(struct drm_i915_gem_object *obj)
+{
+ unsigned int flags;
+ struct sg_table *pages;
+
+ flags = I915_ALLOC_MIN_PAGE_SIZE;
+ if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
+ flags |= I915_ALLOC_CONTIGUOUS;
+
+ obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
+ obj->base.size,
+ flags);
+ if (IS_ERR(obj->mm.st_mm_node))
+ return PTR_ERR(obj->mm.st_mm_node);
+
+ /* Range manager is always contigous */
+ if (obj->mm.region->is_range_manager)
+ obj->flags |= I915_BO_ALLOC_CONTIGUOUS;
+ pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
+ if (IS_ERR(pages)) {
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ return PTR_ERR(pages);
+ }
+
+ __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
+
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
+ void __iomem *vaddr =
+ i915_gem_object_lmem_io_map(obj, 0, obj->base.size);
+
+ if (!vaddr) {
+ struct sg_table *pages =
+ __i915_gem_object_unset_pages(obj);
+
+ if (!IS_ERR_OR_NULL(pages))
+ lmem_put_pages(obj, pages);
+ }
+
+ memset_io(vaddr, 0, obj->base.size);
+ io_mapping_unmap(vaddr);
+ }
+
+ return 0;
+}
+
const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
.name = "i915_gem_object_lmem",
.flags = I915_GEM_OBJECT_HAS_IOMEM,
- .get_pages = i915_gem_object_get_pages_buddy,
- .put_pages = i915_gem_object_put_pages_buddy,
+ .get_pages = lmem_get_pages,
+ .put_pages = lmem_put_pages,
.release = i915_gem_object_release_memory_region,
};
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 0415f99b6b95..f5b46d11e6e6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -235,10 +235,12 @@ struct drm_i915_gem_object {
* Memory region for this object.
*/
struct intel_memory_region *region;
+
/**
- * List of memory region blocks allocated for this object.
+ * Memory manager node allocated for this object.
*/
- struct list_head blocks;
+ void *st_mm_node;
+
/**
* Element within memory_region->objects or region->purgeable
* if the object is marked as DONTNEED. Access is protected by
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
index 7361971c177d..6444e097016d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
@@ -475,7 +475,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
might_sleep();
GEM_BUG_ON(n >= obj->base.size >> PAGE_SHIFT);
- GEM_BUG_ON(!i915_gem_object_has_pinned_pages(obj));
+ if (!i915_gem_object_has_pinned_pages(obj))
+ assert_object_held(obj);
/* As we iterate forward through the sg, we record each entry in a
* radixtree for quick repeated (backwards) lookups. If we have seen
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.c b/drivers/gpu/drm/i915/gem/i915_gem_region.c
index ce8fcfc54079..f25e6646c5b7 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.c
@@ -8,129 +8,9 @@
#include "i915_drv.h"
#include "i915_trace.h"
-void
-i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj,
- struct sg_table *pages)
-{
- __intel_memory_region_put_pages_buddy(obj->mm.region, &obj->mm.blocks);
-
- obj->mm.dirty = false;
- sg_free_table(pages);
- kfree(pages);
-}
-
-int
-i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj)
-{
- const u64 max_segment = i915_sg_segment_size();
- struct intel_memory_region *mem = obj->mm.region;
- struct list_head *blocks = &obj->mm.blocks;
- resource_size_t size = obj->base.size;
- resource_size_t prev_end;
- struct i915_buddy_block *block;
- unsigned int flags;
- struct sg_table *st;
- struct scatterlist *sg;
- unsigned int sg_page_sizes;
- int ret;
-
- st = kmalloc(sizeof(*st), GFP_KERNEL);
- if (!st)
- return -ENOMEM;
-
- if (sg_alloc_table(st, size >> PAGE_SHIFT, GFP_KERNEL)) {
- kfree(st);
- return -ENOMEM;
- }
-
- flags = I915_ALLOC_MIN_PAGE_SIZE;
- if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
- flags |= I915_ALLOC_CONTIGUOUS;
-
- ret = __intel_memory_region_get_pages_buddy(mem, size, flags, blocks);
- if (ret)
- goto err_free_sg;
-
- GEM_BUG_ON(list_empty(blocks));
-
- sg = st->sgl;
- st->nents = 0;
- sg_page_sizes = 0;
- prev_end = (resource_size_t)-1;
-
- list_for_each_entry(block, blocks, link) {
- u64 block_size, offset;
-
- block_size = min_t(u64, size,
- i915_buddy_block_size(&mem->mm, block));
- offset = i915_buddy_block_offset(block);
-
- while (block_size) {
- u64 len;
-
- if (offset != prev_end || sg->length >= max_segment) {
- if (st->nents) {
- sg_page_sizes |= sg->length;
- sg = __sg_next(sg);
- }
-
- sg_dma_address(sg) = mem->region.start + offset;
- sg_dma_len(sg) = 0;
- sg->length = 0;
- st->nents++;
- }
-
- len = min(block_size, max_segment - sg->length);
- sg->length += len;
- sg_dma_len(sg) += len;
-
- offset += len;
- block_size -= len;
-
- prev_end = offset;
- }
- }
-
- sg_page_sizes |= sg->length;
- sg_mark_end(sg);
- i915_sg_trim(st);
-
- /* Intended for kernel internal use only */
- if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
- struct scatterlist *sg;
- unsigned long i;
-
- for_each_sg(st->sgl, sg, st->nents, i) {
- unsigned int length;
- void __iomem *vaddr;
- dma_addr_t daddr;
-
- daddr = sg_dma_address(sg);
- daddr -= mem->region.start;
- length = sg_dma_len(sg);
-
- vaddr = io_mapping_map_wc(&mem->iomap, daddr, length);
- memset64((void __force *)vaddr, 0, length / sizeof(u64));
- io_mapping_unmap(vaddr);
- }
-
- wmb();
- }
-
- __i915_gem_object_set_pages(obj, st, sg_page_sizes);
-
- return 0;
-
-err_free_sg:
- sg_free_table(st);
- kfree(st);
- return ret;
-}
-
void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
struct intel_memory_region *mem)
{
- INIT_LIST_HEAD(&obj->mm.blocks);
obj->mm.region = intel_memory_region_get(mem);
if (obj->base.size <= mem->min_page_size)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.h b/drivers/gpu/drm/i915/gem/i915_gem_region.h
index ebddc86d78f7..84fcb3297400 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.h
@@ -12,10 +12,6 @@ struct intel_memory_region;
struct drm_i915_gem_object;
struct sg_table;
-int i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj);
-void i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj,
- struct sg_table *pages);
-
void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
struct intel_memory_region *mem);
void i915_gem_object_release_memory_region(struct drm_i915_gem_object *obj);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
index a9bfa66c8da1..5d16c4462fda 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
@@ -628,11 +628,13 @@ static const struct intel_memory_region_ops shmem_region_ops = {
.init_object = shmem_object_init,
};
-struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915)
+struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915,
+ u16 type, u16 instance)
{
return intel_memory_region_create(i915, 0,
totalram_pages() << PAGE_SHIFT,
PAGE_SIZE, 0,
+ type, instance,
&shmem_region_ops);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
index b5553fc3ac4d..092d7a21de82 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
@@ -772,7 +772,8 @@ static const struct intel_memory_region_ops i915_region_stolen_lmem_ops = {
};
struct intel_memory_region *
-i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance)
{
struct intel_uncore *uncore = &i915->uncore;
struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
@@ -790,6 +791,7 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
mem = intel_memory_region_create(i915, lmem_base, lmem_size,
I915_GTT_PAGE_SIZE_4K, io_start,
+ type, instance,
&i915_region_stolen_lmem_ops);
if (IS_ERR(mem))
return mem;
@@ -811,14 +813,15 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
}
struct intel_memory_region*
-i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance)
{
struct intel_memory_region *mem;
mem = intel_memory_region_create(i915,
intel_graphics_stolen_res.start,
resource_size(&intel_graphics_stolen_res),
- PAGE_SIZE, 0,
+ PAGE_SIZE, 0, type, instance,
&i915_region_stolen_smem_ops);
if (IS_ERR(mem))
return mem;
@@ -826,7 +829,6 @@ i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
intel_memory_region_set_name(mem, "stolen-system");
mem->private = true;
-
return mem;
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.h b/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
index 2bec6c367b9c..ccdf7befc571 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
@@ -21,8 +21,13 @@ int i915_gem_stolen_insert_node_in_range(struct drm_i915_private *dev_priv,
u64 end);
void i915_gem_stolen_remove_node(struct drm_i915_private *dev_priv,
struct drm_mm_node *node);
-struct intel_memory_region *i915_gem_stolen_smem_setup(struct drm_i915_private *i915);
-struct intel_memory_region *i915_gem_stolen_lmem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance);
+struct intel_memory_region *
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance);
+
struct drm_i915_gem_object *
i915_gem_object_create_stolen(struct drm_i915_private *dev_priv,
resource_size_t size);
diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 8d77dcbad059..3f88ecdee031 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -68,8 +68,6 @@ int intel_gt_probe_lmem(struct intel_gt *gt)
id = INTEL_REGION_LMEM;
mem->id = id;
- mem->type = INTEL_MEMORY_LOCAL;
- mem->instance = 0;
intel_memory_region_set_name(mem, "local%u", mem->instance);
diff --git a/drivers/gpu/drm/i915/gt/intel_region_lmem.c b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
index 73fceb0c25fc..f7366b054f8e 100644
--- a/drivers/gpu/drm/i915/gt/intel_region_lmem.c
+++ b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
@@ -5,6 +5,8 @@
#include "i915_drv.h"
#include "intel_memory_region.h"
+#include "intel_region_lmem.h"
+#include "intel_region_ttm.h"
#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
#include "intel_region_lmem.h"
@@ -66,9 +68,9 @@ static void release_fake_lmem_bar(struct intel_memory_region *mem)
static void
region_lmem_release(struct intel_memory_region *mem)
{
- release_fake_lmem_bar(mem);
+ intel_region_ttm_fini(mem);
io_mapping_fini(&mem->iomap);
- intel_memory_region_release_buddy(mem);
+ release_fake_lmem_bar(mem);
}
static int
@@ -83,12 +85,21 @@ region_lmem_init(struct intel_memory_region *mem)
if (!io_mapping_init_wc(&mem->iomap,
mem->io_start,
- resource_size(&mem->region)))
- return -EIO;
+ resource_size(&mem->region))) {
+ ret = -EIO;
+ goto out_no_io;
+ }
- ret = intel_memory_region_init_buddy(mem);
+ ret = intel_region_ttm_init(mem);
if (ret)
- io_mapping_fini(&mem->iomap);
+ goto out_no_buddy;
+
+ return 0;
+
+out_no_buddy:
+ io_mapping_fini(&mem->iomap);
+out_no_io:
+ release_fake_lmem_bar(mem);
return ret;
}
@@ -127,6 +138,8 @@ intel_gt_setup_fake_lmem(struct intel_gt *gt)
mappable_end,
PAGE_SIZE,
io_start,
+ INTEL_MEMORY_LOCAL,
+ 0,
&intel_region_lmem_ops);
if (!IS_ERR(mem)) {
drm_info(&i915->drm, "Intel graphics fake LMEM: %pR\n",
@@ -198,6 +211,8 @@ static struct intel_memory_region *setup_lmem(struct intel_gt *gt)
lmem_size,
I915_GTT_PAGE_SIZE_4K,
io_start,
+ INTEL_MEMORY_LOCAL,
+ 0,
&intel_region_lmem_ops);
if (IS_ERR(mem))
return mem;
diff --git a/drivers/gpu/drm/i915/i915_buddy.c b/drivers/gpu/drm/i915/i915_buddy.c
deleted file mode 100644
index 3a2f6eecb2fc..000000000000
--- a/drivers/gpu/drm/i915/i915_buddy.c
+++ /dev/null
@@ -1,435 +0,0 @@
-// SPDX-License-Identifier: MIT
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#include <linux/kmemleak.h>
-#include <linux/slab.h>
-
-#include "i915_buddy.h"
-
-#include "i915_gem.h"
-#include "i915_globals.h"
-#include "i915_utils.h"
-
-static struct i915_global_block {
- struct i915_global base;
- struct kmem_cache *slab_blocks;
-} global;
-
-static void i915_global_buddy_shrink(void)
-{
- kmem_cache_shrink(global.slab_blocks);
-}
-
-static void i915_global_buddy_exit(void)
-{
- kmem_cache_destroy(global.slab_blocks);
-}
-
-static struct i915_global_block global = { {
- .shrink = i915_global_buddy_shrink,
- .exit = i915_global_buddy_exit,
-} };
-
-int __init i915_global_buddy_init(void)
-{
- global.slab_blocks = KMEM_CACHE(i915_buddy_block, SLAB_HWCACHE_ALIGN);
- if (!global.slab_blocks)
- return -ENOMEM;
-
- i915_global_register(&global.base);
- return 0;
-}
-
-static struct i915_buddy_block *i915_block_alloc(struct i915_buddy_block *parent,
- unsigned int order,
- u64 offset)
-{
- struct i915_buddy_block *block;
-
- GEM_BUG_ON(order > I915_BUDDY_MAX_ORDER);
-
- block = kmem_cache_zalloc(global.slab_blocks, GFP_KERNEL);
- if (!block)
- return NULL;
-
- block->header = offset;
- block->header |= order;
- block->parent = parent;
-
- GEM_BUG_ON(block->header & I915_BUDDY_HEADER_UNUSED);
- return block;
-}
-
-static void i915_block_free(struct i915_buddy_block *block)
-{
- kmem_cache_free(global.slab_blocks, block);
-}
-
-static void mark_allocated(struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_ALLOCATED;
-
- list_del(&block->link);
-}
-
-static void mark_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_FREE;
-
- list_add(&block->link,
- &mm->free_list[i915_buddy_block_order(block)]);
-}
-
-static void mark_split(struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_SPLIT;
-
- list_del(&block->link);
-}
-
-int i915_buddy_init(struct i915_buddy_mm *mm, u64 size, u64 chunk_size)
-{
- unsigned int i;
- u64 offset;
-
- if (size < chunk_size)
- return -EINVAL;
-
- if (chunk_size < PAGE_SIZE)
- return -EINVAL;
-
- if (!is_power_of_2(chunk_size))
- return -EINVAL;
-
- size = round_down(size, chunk_size);
-
- mm->size = size;
- mm->chunk_size = chunk_size;
- mm->max_order = ilog2(size) - ilog2(chunk_size);
-
- GEM_BUG_ON(mm->max_order > I915_BUDDY_MAX_ORDER);
-
- mm->free_list = kmalloc_array(mm->max_order + 1,
- sizeof(struct list_head),
- GFP_KERNEL);
- if (!mm->free_list)
- return -ENOMEM;
-
- for (i = 0; i <= mm->max_order; ++i)
- INIT_LIST_HEAD(&mm->free_list[i]);
-
- mm->n_roots = hweight64(size);
-
- mm->roots = kmalloc_array(mm->n_roots,
- sizeof(struct i915_buddy_block *),
- GFP_KERNEL);
- if (!mm->roots)
- goto out_free_list;
-
- offset = 0;
- i = 0;
-
- /*
- * Split into power-of-two blocks, in case we are given a size that is
- * not itself a power-of-two.
- */
- do {
- struct i915_buddy_block *root;
- unsigned int order;
- u64 root_size;
-
- root_size = rounddown_pow_of_two(size);
- order = ilog2(root_size) - ilog2(chunk_size);
-
- root = i915_block_alloc(NULL, order, offset);
- if (!root)
- goto out_free_roots;
-
- mark_free(mm, root);
-
- GEM_BUG_ON(i > mm->max_order);
- GEM_BUG_ON(i915_buddy_block_size(mm, root) < chunk_size);
-
- mm->roots[i] = root;
-
- offset += root_size;
- size -= root_size;
- i++;
- } while (size);
-
- return 0;
-
-out_free_roots:
- while (i--)
- i915_block_free(mm->roots[i]);
- kfree(mm->roots);
-out_free_list:
- kfree(mm->free_list);
- return -ENOMEM;
-}
-
-void i915_buddy_fini(struct i915_buddy_mm *mm)
-{
- int i;
-
- for (i = 0; i < mm->n_roots; ++i) {
- GEM_WARN_ON(!i915_buddy_block_is_free(mm->roots[i]));
- i915_block_free(mm->roots[i]);
- }
-
- kfree(mm->roots);
- kfree(mm->free_list);
-}
-
-static int split_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- unsigned int block_order = i915_buddy_block_order(block) - 1;
- u64 offset = i915_buddy_block_offset(block);
-
- GEM_BUG_ON(!i915_buddy_block_is_free(block));
- GEM_BUG_ON(!i915_buddy_block_order(block));
-
- block->left = i915_block_alloc(block, block_order, offset);
- if (!block->left)
- return -ENOMEM;
-
- block->right = i915_block_alloc(block, block_order,
- offset + (mm->chunk_size << block_order));
- if (!block->right) {
- i915_block_free(block->left);
- return -ENOMEM;
- }
-
- mark_free(mm, block->left);
- mark_free(mm, block->right);
-
- mark_split(block);
-
- return 0;
-}
-
-static struct i915_buddy_block *
-get_buddy(struct i915_buddy_block *block)
-{
- struct i915_buddy_block *parent;
-
- parent = block->parent;
- if (!parent)
- return NULL;
-
- if (parent->left == block)
- return parent->right;
-
- return parent->left;
-}
-
-static void __i915_buddy_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *parent;
-
- while ((parent = block->parent)) {
- struct i915_buddy_block *buddy;
-
- buddy = get_buddy(block);
-
- if (!i915_buddy_block_is_free(buddy))
- break;
-
- list_del(&buddy->link);
-
- i915_block_free(block);
- i915_block_free(buddy);
-
- block = parent;
- }
-
- mark_free(mm, block);
-}
-
-void i915_buddy_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- GEM_BUG_ON(!i915_buddy_block_is_allocated(block));
- __i915_buddy_free(mm, block);
-}
-
-void i915_buddy_free_list(struct i915_buddy_mm *mm, struct list_head *objects)
-{
- struct i915_buddy_block *block, *on;
-
- list_for_each_entry_safe(block, on, objects, link) {
- i915_buddy_free(mm, block);
- cond_resched();
- }
- INIT_LIST_HEAD(objects);
-}
-
-/*
- * Allocate power-of-two block. The order value here translates to:
- *
- * 0 = 2^0 * mm->chunk_size
- * 1 = 2^1 * mm->chunk_size
- * 2 = 2^2 * mm->chunk_size
- * ...
- */
-struct i915_buddy_block *
-i915_buddy_alloc(struct i915_buddy_mm *mm, unsigned int order)
-{
- struct i915_buddy_block *block = NULL;
- unsigned int i;
- int err;
-
- for (i = order; i <= mm->max_order; ++i) {
- block = list_first_entry_or_null(&mm->free_list[i],
- struct i915_buddy_block,
- link);
- if (block)
- break;
- }
-
- if (!block)
- return ERR_PTR(-ENOSPC);
-
- GEM_BUG_ON(!i915_buddy_block_is_free(block));
-
- while (i != order) {
- err = split_block(mm, block);
- if (unlikely(err))
- goto out_free;
-
- /* Go low */
- block = block->left;
- i--;
- }
-
- mark_allocated(block);
- kmemleak_update_trace(block);
- return block;
-
-out_free:
- if (i != order)
- __i915_buddy_free(mm, block);
- return ERR_PTR(err);
-}
-
-static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
-{
- return s1 <= e2 && e1 >= s2;
-}
-
-static inline bool contains(u64 s1, u64 e1, u64 s2, u64 e2)
-{
- return s1 <= s2 && e1 >= e2;
-}
-
-/*
- * Allocate range. Note that it's safe to chain together multiple alloc_ranges
- * with the same blocks list.
- *
- * Intended for pre-allocating portions of the address space, for example to
- * reserve a block for the initial framebuffer or similar, hence the expectation
- * here is that i915_buddy_alloc() is still the main vehicle for
- * allocations, so if that's not the case then the drm_mm range allocator is
- * probably a much better fit, and so you should probably go use that instead.
- */
-int i915_buddy_alloc_range(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 start, u64 size)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_block *buddy;
- LIST_HEAD(allocated);
- LIST_HEAD(dfs);
- u64 end;
- int err;
- int i;
-
- if (size < mm->chunk_size)
- return -EINVAL;
-
- if (!IS_ALIGNED(size | start, mm->chunk_size))
- return -EINVAL;
-
- if (range_overflows(start, size, mm->size))
- return -EINVAL;
-
- for (i = 0; i < mm->n_roots; ++i)
- list_add_tail(&mm->roots[i]->tmp_link, &dfs);
-
- end = start + size - 1;
-
- do {
- u64 block_start;
- u64 block_end;
-
- block = list_first_entry_or_null(&dfs,
- struct i915_buddy_block,
- tmp_link);
- if (!block)
- break;
-
- list_del(&block->tmp_link);
-
- block_start = i915_buddy_block_offset(block);
- block_end = block_start + i915_buddy_block_size(mm, block) - 1;
-
- if (!overlaps(start, end, block_start, block_end))
- continue;
-
- if (i915_buddy_block_is_allocated(block)) {
- err = -ENOSPC;
- goto err_free;
- }
-
- if (contains(start, end, block_start, block_end)) {
- if (!i915_buddy_block_is_free(block)) {
- err = -ENOSPC;
- goto err_free;
- }
-
- mark_allocated(block);
- list_add_tail(&block->link, &allocated);
- continue;
- }
-
- if (!i915_buddy_block_is_split(block)) {
- err = split_block(mm, block);
- if (unlikely(err))
- goto err_undo;
- }
-
- list_add(&block->right->tmp_link, &dfs);
- list_add(&block->left->tmp_link, &dfs);
- } while (1);
-
- list_splice_tail(&allocated, blocks);
- return 0;
-
-err_undo:
- /*
- * We really don't want to leave around a bunch of split blocks, since
- * bigger is better, so make sure we merge everything back before we
- * free the allocated blocks.
- */
- buddy = get_buddy(block);
- if (buddy &&
- (i915_buddy_block_is_free(block) &&
- i915_buddy_block_is_free(buddy)))
- __i915_buddy_free(mm, block);
-
-err_free:
- i915_buddy_free_list(mm, &allocated);
- return err;
-}
-
-#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
-#include "selftests/i915_buddy.c"
-#endif
diff --git a/drivers/gpu/drm/i915/i915_buddy.h b/drivers/gpu/drm/i915/i915_buddy.h
deleted file mode 100644
index 9ce5200f4001..000000000000
--- a/drivers/gpu/drm/i915/i915_buddy.h
+++ /dev/null
@@ -1,131 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#ifndef __I915_BUDDY_H__
-#define __I915_BUDDY_H__
-
-#include <linux/bitops.h>
-#include <linux/list.h>
-
-struct i915_buddy_block {
-#define I915_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
-#define I915_BUDDY_HEADER_STATE GENMASK_ULL(11, 10)
-#define I915_BUDDY_ALLOCATED (1 << 10)
-#define I915_BUDDY_FREE (2 << 10)
-#define I915_BUDDY_SPLIT (3 << 10)
-/* Free to be used, if needed in the future */
-#define I915_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
-#define I915_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
- u64 header;
-
- struct i915_buddy_block *left;
- struct i915_buddy_block *right;
- struct i915_buddy_block *parent;
-
- void *private; /* owned by creator */
-
- /*
- * While the block is allocated by the user through i915_buddy_alloc*,
- * the user has ownership of the link, for example to maintain within
- * a list, if so desired. As soon as the block is freed with
- * i915_buddy_free* ownership is given back to the mm.
- */
- struct list_head link;
- struct list_head tmp_link;
-};
-
-/* Order-zero must be at least PAGE_SIZE */
-#define I915_BUDDY_MAX_ORDER (63 - PAGE_SHIFT)
-
-/*
- * Binary Buddy System.
- *
- * Locking should be handled by the user, a simple mutex around
- * i915_buddy_alloc* and i915_buddy_free* should suffice.
- */
-struct i915_buddy_mm {
- /* Maintain a free list for each order. */
- struct list_head *free_list;
-
- /*
- * Maintain explicit binary tree(s) to track the allocation of the
- * address space. This gives us a simple way of finding a buddy block
- * and performing the potentially recursive merge step when freeing a
- * block. Nodes are either allocated or free, in which case they will
- * also exist on the respective free list.
- */
- struct i915_buddy_block **roots;
-
- /*
- * Anything from here is public, and remains static for the lifetime of
- * the mm. Everything above is considered do-not-touch.
- */
- unsigned int n_roots;
- unsigned int max_order;
-
- /* Must be at least PAGE_SIZE */
- u64 chunk_size;
- u64 size;
-};
-
-static inline u64
-i915_buddy_block_offset(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_OFFSET;
-}
-
-static inline unsigned int
-i915_buddy_block_order(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_ORDER;
-}
-
-static inline unsigned int
-i915_buddy_block_state(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_STATE;
-}
-
-static inline bool
-i915_buddy_block_is_allocated(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_ALLOCATED;
-}
-
-static inline bool
-i915_buddy_block_is_free(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_FREE;
-}
-
-static inline bool
-i915_buddy_block_is_split(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_SPLIT;
-}
-
-static inline u64
-i915_buddy_block_size(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- return mm->chunk_size << i915_buddy_block_order(block);
-}
-
-int i915_buddy_init(struct i915_buddy_mm *mm, u64 size, u64 chunk_size);
-
-void i915_buddy_fini(struct i915_buddy_mm *mm);
-
-struct i915_buddy_block *
-i915_buddy_alloc(struct i915_buddy_mm *mm, unsigned int order);
-
-int i915_buddy_alloc_range(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 start, u64 size);
-
-void i915_buddy_free(struct i915_buddy_mm *mm, struct i915_buddy_block *block);
-
-void i915_buddy_free_list(struct i915_buddy_mm *mm, struct list_head *objects);
-
-#endif
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index d82a99e128cf..30c349137be2 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -84,6 +84,7 @@
#include "intel_gvt.h"
#include "intel_memory_region.h"
#include "intel_pm.h"
+#include "intel_region_ttm.h"
#include "intel_sideband.h"
#include "vlv_suspend.h"
@@ -335,6 +336,10 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
if (ret < 0)
goto err_workqueues;
+ ret = intel_region_ttm_device_init(dev_priv);
+ if (ret)
+ goto err_ttm;
+
intel_wopcm_init_early(&dev_priv->wopcm);
intel_gt_init_early(&dev_priv->gt, dev_priv);
@@ -359,6 +364,8 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
err_gem:
i915_gem_cleanup_early(dev_priv);
intel_gt_driver_late_release(&dev_priv->gt);
+ intel_region_ttm_device_fini(dev_priv);
+err_ttm:
vlv_suspend_cleanup(dev_priv);
err_workqueues:
i915_workqueues_cleanup(dev_priv);
@@ -376,6 +383,7 @@ static void i915_driver_late_release(struct drm_i915_private *dev_priv)
intel_power_domains_cleanup(dev_priv);
i915_gem_cleanup_early(dev_priv);
intel_gt_driver_late_release(&dev_priv->gt);
+ intel_region_ttm_device_fini(dev_priv);
vlv_suspend_cleanup(dev_priv);
i915_workqueues_cleanup(dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 9cb02618ba15..3a985865db41 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -59,6 +59,7 @@
#include <drm/drm_atomic.h>
#include <drm/drm_connector.h>
#include <drm/i915_mei_hdcp_interface.h>
+#include <drm/ttm/ttm_device.h>
#include "i915_params.h"
#include "i915_reg.h"
@@ -778,6 +779,7 @@ struct intel_cdclk_config {
struct i915_selftest_stash {
atomic_t counter;
+ struct ida mock_region_instances;
};
struct drm_i915_private {
@@ -1167,6 +1169,9 @@ struct drm_i915_private {
/* Mutex to protect the above hdcp component related values. */
struct mutex hdcp_comp_mutex;
+ /* The TTM device structure. */
+ struct ttm_device bdev;
+
I915_SELFTEST_DECLARE(struct i915_selftest_stash selftest;)
/*
@@ -1760,7 +1765,8 @@ void i915_gem_cleanup_userptr(struct drm_i915_private *dev_priv);
void i915_gem_init_early(struct drm_i915_private *dev_priv);
void i915_gem_cleanup_early(struct drm_i915_private *dev_priv);
-struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915,
+ u16 type, u16 instance);
static inline void i915_gem_drain_freed_objects(struct drm_i915_private *i915)
{
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index cffd7f4f87dc..0993d706f067 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1108,6 +1108,7 @@ int i915_gem_init(struct drm_i915_private *dev_priv)
}
i915_gem_drain_freed_objects(dev_priv);
+
return ret;
}
diff --git a/drivers/gpu/drm/i915/i915_globals.c b/drivers/gpu/drm/i915/i915_globals.c
index 3aa213684293..77f1911c463b 100644
--- a/drivers/gpu/drm/i915/i915_globals.c
+++ b/drivers/gpu/drm/i915/i915_globals.c
@@ -87,7 +87,6 @@ static void __i915_globals_cleanup(void)
static __initconst int (* const initfn[])(void) = {
i915_global_active_init,
- i915_global_buddy_init,
i915_global_context_init,
i915_global_gem_context_init,
i915_global_objects_init,
diff --git a/drivers/gpu/drm/i915/i915_globals.h b/drivers/gpu/drm/i915/i915_globals.h
index b2f5cd9b9b1a..2d199f411a4a 100644
--- a/drivers/gpu/drm/i915/i915_globals.h
+++ b/drivers/gpu/drm/i915/i915_globals.h
@@ -27,7 +27,6 @@ void i915_globals_exit(void);
/* constructors */
int i915_global_active_init(void);
-int i915_global_buddy_init(void);
int i915_global_context_init(void);
int i915_global_gem_context_init(void);
int i915_global_objects_init(void);
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.c b/drivers/gpu/drm/i915/i915_scatterlist.c
index cc6b3846a8c7..69e9e6c3135e 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.c
+++ b/drivers/gpu/drm/i915/i915_scatterlist.c
@@ -6,6 +6,10 @@
#include "i915_scatterlist.h"
+#include <drm/drm_mm.h>
+
+#include <linux/slab.h>
+
bool i915_sg_trim(struct sg_table *orig_st)
{
struct sg_table new_st;
@@ -34,6 +38,72 @@ bool i915_sg_trim(struct sg_table *orig_st)
return true;
}
+/**
+ * i915_sg_from_mm_node - Create an sg_table from a struct drm_mm_node
+ * @node: The drm_mm_node.
+ * @region_start: An offset to add to the dma addresses of the sg list.
+ *
+ * Create a struct sg_table, initializing it from a struct drm_mm_node,
+ * taking a maximum segment length into account, splitting into segments
+ * if necessary.
+ *
+ * Return: A pointer to a kmalloced struct sg_table on success, negative
+ * error code cast to an error pointer on failure.
+ */
+struct sg_table *i915_sg_from_mm_node(const struct drm_mm_node *node,
+ u64 region_start)
+{
+ const u64 max_segment = SZ_1G; /* Do we have a limit on this? */
+ u64 segment_pages = max_segment >> PAGE_SHIFT;
+ u64 block_size, offset, prev_end;
+ struct sg_table *st;
+ struct scatterlist *sg;
+
+ st = kmalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return ERR_PTR(-ENOMEM);
+
+ if (sg_alloc_table(st, DIV_ROUND_UP(node->size, segment_pages),
+ GFP_KERNEL)) {
+ kfree(st);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ sg = st->sgl;
+ st->nents = 0;
+ prev_end = (resource_size_t)-1;
+ block_size = node->size << PAGE_SHIFT;
+ offset = node->start << PAGE_SHIFT;
+
+ while (block_size) {
+ u64 len;
+
+ if (offset != prev_end || sg->length >= max_segment) {
+ if (st->nents)
+ sg = __sg_next(sg);
+
+ sg_dma_address(sg) = region_start + offset;
+ sg_dma_len(sg) = 0;
+ sg->length = 0;
+ st->nents++;
+ }
+
+ len = min(block_size, max_segment - sg->length);
+ sg->length += len;
+ sg_dma_len(sg) += len;
+
+ offset += len;
+ block_size -= len;
+
+ prev_end = offset;
+ }
+
+ sg_mark_end(sg);
+ i915_sg_trim(st);
+
+ return st;
+}
+
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/scatterlist.c"
#endif
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.h b/drivers/gpu/drm/i915/i915_scatterlist.h
index b96baad66a3a..5acca45ea981 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.h
+++ b/drivers/gpu/drm/i915/i915_scatterlist.h
@@ -13,6 +13,8 @@
#include "i915_gem.h"
+struct drm_mm_node;
+
/*
* Optimised SGL iterator for GEM objects
*/
@@ -141,4 +143,6 @@ static inline unsigned int i915_sg_segment_size(void)
bool i915_sg_trim(struct sg_table *orig_st);
+struct sg_table *i915_sg_from_mm_node(const struct drm_mm_node *node,
+ u64 region_start);
#endif
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index d98e8b81d322..4092cc987679 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -28,6 +28,11 @@ static const struct {
},
};
+struct intel_region_reserve {
+ struct list_head link;
+ void *node;
+};
+
struct intel_memory_region *
intel_memory_region_lookup(struct drm_i915_private *i915,
u16 class, u16 instance)
@@ -58,146 +63,61 @@ intel_memory_region_by_type(struct drm_i915_private *i915,
return NULL;
}
-static u64
-intel_memory_region_free_pages(struct intel_memory_region *mem,
- struct list_head *blocks)
+/**
+ * intel_memory_region_unreserve - Unreserve all previously reserved
+ * ranges
+ * @mem: The region containing the reserved ranges.
+ */
+void intel_memory_region_unreserve(struct intel_memory_region *mem)
{
- struct i915_buddy_block *block, *on;
- u64 size = 0;
+ struct intel_region_reserve *reserve, *next;
- list_for_each_entry_safe(block, on, blocks, link) {
- size += i915_buddy_block_size(&mem->mm, block);
- i915_buddy_free(&mem->mm, block);
- }
- INIT_LIST_HEAD(blocks);
+ if (!mem->priv_ops || !mem->priv_ops->free)
+ return;
- return size;
-}
-
-void
-__intel_memory_region_put_pages_buddy(struct intel_memory_region *mem,
- struct list_head *blocks)
-{
mutex_lock(&mem->mm_lock);
- mem->avail += intel_memory_region_free_pages(mem, blocks);
- mutex_unlock(&mem->mm_lock);
-}
-
-void
-__intel_memory_region_put_block_buddy(struct i915_buddy_block *block)
-{
- struct list_head blocks;
-
- INIT_LIST_HEAD(&blocks);
- list_add(&block->link, &blocks);
- __intel_memory_region_put_pages_buddy(block->private, &blocks);
-}
-
-int
-__intel_memory_region_get_pages_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags,
- struct list_head *blocks)
-{
- unsigned int min_order = 0;
- unsigned long n_pages;
-
- GEM_BUG_ON(!IS_ALIGNED(size, mem->mm.chunk_size));
- GEM_BUG_ON(!list_empty(blocks));
-
- if (flags & I915_ALLOC_MIN_PAGE_SIZE) {
- min_order = ilog2(mem->min_page_size) -
- ilog2(mem->mm.chunk_size);
- }
-
- if (flags & I915_ALLOC_CONTIGUOUS) {
- size = roundup_pow_of_two(size);
- min_order = ilog2(size) - ilog2(mem->mm.chunk_size);
+ list_for_each_entry_safe(reserve, next, &mem->reserved, link) {
+ list_del(&reserve->link);
+ mem->priv_ops->free(mem, reserve->node);
+ kfree(reserve);
}
-
- if (size > mem->mm.size)
- return -E2BIG;
-
- n_pages = size >> ilog2(mem->mm.chunk_size);
-
- mutex_lock(&mem->mm_lock);
-
- do {
- struct i915_buddy_block *block;
- unsigned int order;
-
- order = fls(n_pages) - 1;
- GEM_BUG_ON(order > mem->mm.max_order);
- GEM_BUG_ON(order < min_order);
-
- do {
- block = i915_buddy_alloc(&mem->mm, order);
- if (!IS_ERR(block))
- break;
-
- if (order-- == min_order)
- goto err_free_blocks;
- } while (1);
-
- n_pages -= BIT(order);
-
- block->private = mem;
- list_add_tail(&block->link, blocks);
-
- if (!n_pages)
- break;
- } while (1);
-
- mem->avail -= size;
mutex_unlock(&mem->mm_lock);
- return 0;
-
-err_free_blocks:
- intel_memory_region_free_pages(mem, blocks);
- mutex_unlock(&mem->mm_lock);
- return -ENXIO;
}
-struct i915_buddy_block *
-__intel_memory_region_get_block_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags)
+/**
+ * intel_memory_region_reserve - Reserve a memory range
+ * @mem: The region for which we want to reserve a range.
+ * @offset: Start of the range to reserve.
+ * @size: The size of the range to reserve.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_memory_region_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size)
{
- struct i915_buddy_block *block;
- LIST_HEAD(blocks);
int ret;
+ struct intel_region_reserve *reserve;
- ret = __intel_memory_region_get_pages_buddy(mem, size, flags, &blocks);
- if (ret)
- return ERR_PTR(ret);
+ if (!mem->priv_ops || !mem->priv_ops->reserve)
+ return -EINVAL;
- block = list_first_entry(&blocks, typeof(*block), link);
- list_del_init(&block->link);
- return block;
-}
+ reserve = kzalloc(sizeof(*reserve), GFP_KERNEL);
+ if (!reserve)
+ return -ENOMEM;
-int intel_memory_region_init_buddy(struct intel_memory_region *mem)
-{
- return i915_buddy_init(&mem->mm, resource_size(&mem->region),
- PAGE_SIZE);
-}
-
-void intel_memory_region_release_buddy(struct intel_memory_region *mem)
-{
- i915_buddy_free_list(&mem->mm, &mem->reserved);
- i915_buddy_fini(&mem->mm);
-}
-
-int intel_memory_region_reserve(struct intel_memory_region *mem,
- u64 offset, u64 size)
-{
- int ret;
+ reserve->node = mem->priv_ops->reserve(mem, offset, size);
+ if (IS_ERR(reserve->node)) {
+ ret = PTR_ERR(reserve->node);
+ kfree(reserve);
+ return ret;
+ }
mutex_lock(&mem->mm_lock);
- ret = i915_buddy_alloc_range(&mem->mm, &mem->reserved, offset, size);
+ list_add_tail(&reserve->link, &mem->reserved);
mutex_unlock(&mem->mm_lock);
- return ret;
+ return 0;
}
struct intel_memory_region *
@@ -206,6 +126,8 @@ intel_memory_region_create(struct drm_i915_private *i915,
resource_size_t size,
resource_size_t min_page_size,
resource_size_t io_start,
+ u16 type,
+ u16 instance,
const struct intel_memory_region_ops *ops)
{
struct intel_memory_region *mem;
@@ -222,6 +144,8 @@ intel_memory_region_create(struct drm_i915_private *i915,
mem->ops = ops;
mem->total = size;
mem->avail = mem->total;
+ mem->type = type;
+ mem->instance = instance;
mutex_init(&mem->objects.lock);
INIT_LIST_HEAD(&mem->objects.list);
@@ -259,6 +183,7 @@ static void __intel_memory_region_destroy(struct kref *kref)
struct intel_memory_region *mem =
container_of(kref, typeof(*mem), kref);
+ intel_memory_region_unreserve(mem);
if (mem->ops->release)
mem->ops->release(mem);
@@ -296,15 +221,15 @@ int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
instance = intel_region_map[i].instance;
switch (type) {
case INTEL_MEMORY_SYSTEM:
- mem = i915_gem_shmem_setup(i915);
+ mem = i915_gem_shmem_setup(i915, type, instance);
break;
case INTEL_MEMORY_STOLEN_LOCAL:
- mem = i915_gem_stolen_lmem_setup(i915);
+ mem = i915_gem_stolen_lmem_setup(i915, type, instance);
if (!IS_ERR(mem))
i915->mm.stolen_region = mem;
break;
case INTEL_MEMORY_STOLEN_SYSTEM:
- mem = i915_gem_stolen_smem_setup(i915);
+ mem = i915_gem_stolen_smem_setup(i915, type, instance);
if (!IS_ERR(mem))
i915->mm.stolen_region = mem;
break;
@@ -321,9 +246,6 @@ int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
}
mem->id = i;
- mem->type = type;
- mem->instance = instance;
-
i915->mm.regions[i] = mem;
}
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index d24ce5a0b30b..e69cde13daf2 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -13,8 +13,6 @@
#include <drm/drm_mm.h>
#include <drm/i915_drm.h>
-#include "i915_buddy.h"
-
struct drm_i915_private;
struct drm_i915_gem_object;
struct intel_memory_region;
@@ -25,6 +23,7 @@ enum intel_memory_type {
INTEL_MEMORY_LOCAL = I915_MEMORY_CLASS_DEVICE,
INTEL_MEMORY_STOLEN_SYSTEM,
INTEL_MEMORY_STOLEN_LOCAL,
+ INTEL_MEMORY_MOCK,
};
enum intel_region_id {
@@ -59,10 +58,19 @@ struct intel_memory_region_ops {
unsigned int flags);
};
+struct intel_memory_region_private_ops {
+ void *(*reserve)(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size);
+ void (*free)(struct intel_memory_region *mem,
+ void *node);
+};
+
struct intel_memory_region {
struct drm_i915_private *i915;
const struct intel_memory_region_ops *ops;
+ const struct intel_memory_region_private_ops *priv_ops;
struct io_mapping iomap;
struct resource region;
@@ -70,7 +78,6 @@ struct intel_memory_region {
/* For fake LMEM */
struct drm_mm_node fake_mappable;
- struct i915_buddy_mm mm;
struct mutex mm_lock;
struct kref kref;
@@ -95,36 +102,26 @@ struct intel_memory_region {
struct list_head list;
struct list_head purgeable;
} objects;
+
+ size_t chunk_size;
+ unsigned int max_order;
+ bool is_range_manager;
+
+ void *region_private;
};
struct intel_memory_region *
intel_memory_region_lookup(struct drm_i915_private *i915,
u16 class, u16 instance);
-int intel_memory_region_init_buddy(struct intel_memory_region *mem);
-void intel_memory_region_release_buddy(struct intel_memory_region *mem);
-
-int __intel_memory_region_get_pages_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags,
- struct list_head *blocks);
-struct i915_buddy_block *
-__intel_memory_region_get_block_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags);
-void __intel_memory_region_put_pages_buddy(struct intel_memory_region *mem,
- struct list_head *blocks);
-void __intel_memory_region_put_block_buddy(struct i915_buddy_block *block);
-
-int intel_memory_region_reserve(struct intel_memory_region *mem,
- u64 offset, u64 size);
-
struct intel_memory_region *
intel_memory_region_create(struct drm_i915_private *i915,
resource_size_t start,
resource_size_t size,
resource_size_t min_page_size,
resource_size_t io_start,
+ u16 type,
+ u16 instance,
const struct intel_memory_region_ops *ops);
struct intel_memory_region *
@@ -141,4 +138,9 @@ __printf(2, 3) void
intel_memory_region_set_name(struct intel_memory_region *mem,
const char *fmt, ...);
+void intel_memory_region_unreserve(struct intel_memory_region *mem);
+
+int intel_memory_region_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size);
#endif
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
new file mode 100644
index 000000000000..c8ac118c21f6
--- /dev/null
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -0,0 +1,220 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_device.h>
+
+#include "i915_drv.h"
+#include "i915_scatterlist.h"
+
+#include "intel_region_ttm.h"
+
+/**
+ * DOC: TTM support structure
+ *
+ * The code in this file deals with setting up memory managers for TTM
+ * LMEM and MOCK regions and converting the output from
+ * the managers to struct sg_table, Basically providing the mapping from
+ * i915 GEM regions to TTM memory types and resource managers.
+ */
+
+/* A Zero-initialized driver for now. We don't have a TTM backend yet. */
+static struct ttm_device_funcs i915_ttm_bo_driver;
+
+/**
+ * intel_region_ttm_device_init - Initialize a TTM device
+ * @dev_priv: Pointer to an i915 device private structure.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_region_ttm_device_init(struct drm_i915_private *dev_priv)
+{
+ struct drm_device *drm = &dev_priv->drm;
+
+ return ttm_device_init(&dev_priv->bdev, &i915_ttm_bo_driver,
+ drm->dev, drm->anon_inode->i_mapping,
+ drm->vma_offset_manager, false, false);
+}
+
+/**
+ * intel_region_ttm_device_fini - Finalize a TTM device
+ * @dev_priv: Pointer to an i915 device private structure.
+ */
+void intel_region_ttm_device_fini(struct drm_i915_private *dev_priv)
+{
+ ttm_device_fini(&dev_priv->bdev);
+}
+
+/*
+ * Map the i915 memory regions to TTM memory types. We use the
+ * driver-private types for now, reserving TTM_PL_VRAM for stolen
+ * memory and TTM_PL_TT for GGTT use if decided to implement this.
+ */
+static int intel_region_to_ttm_type(struct intel_memory_region *mem)
+{
+ int type;
+
+ GEM_BUG_ON(mem->type != INTEL_MEMORY_LOCAL &&
+ mem->type != INTEL_MEMORY_MOCK);
+
+ type = mem->instance + TTM_PL_PRIV;
+ GEM_BUG_ON(type >= TTM_NUM_MEM_TYPES);
+
+ return type;
+}
+
+static void *intel_region_ttm_node_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size)
+{
+ struct ttm_resource_manager *man = mem->region_private;
+ struct ttm_place place = {};
+ struct ttm_resource res = {};
+ struct ttm_buffer_object mock_bo = {};
+ int ret;
+
+ /*
+ * Having to use a mock_bo is unfortunate but stems from some
+ * drivers having private managers that insist to know what the
+ * allocate memory is intended for, using it to send private
+ * data to the manager. Also recently the bo has been used to send
+ * alignment info to the manager. Assume that apart from the latter,
+ * none of the managers we use will ever access the buffer object
+ * members, hoping we can pass the alignment info in the
+ * struct ttm_place in the future.
+ */
+
+ place.fpfn = offset >> PAGE_SHIFT;
+ place.lpfn = place.fpfn + (size >> PAGE_SHIFT);
+ res.num_pages = size >> PAGE_SHIFT;
+ ret = man->func->alloc(man, &mock_bo, &place, &res);
+ if (ret == -ENOSPC)
+ ret = -ENXIO;
+
+ return ret ? ERR_PTR(ret) : res.mm_node;
+}
+
+/**
+ * intel_region_ttm_node_free - Free a node allocated from a resource manager
+ * @mem: The region the node was allocated from.
+ * @node: The opaque node representing an allocation.
+ */
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node)
+{
+ struct ttm_resource_manager *man = mem->region_private;
+ struct ttm_resource res = {};
+
+ res.mm_node = node;
+ man->func->free(man, &res);
+}
+
+static const struct intel_memory_region_private_ops priv_ops = {
+ .reserve = intel_region_ttm_node_reserve,
+ .free = intel_region_ttm_node_free,
+};
+
+int intel_region_ttm_init(struct intel_memory_region *mem)
+{
+ struct ttm_device *bdev = &mem->i915->bdev;
+ int mem_type = intel_region_to_ttm_type(mem);
+ int ret;
+
+ ret = ttm_range_man_init(bdev, mem_type, false,
+ resource_size(&mem->region) >> PAGE_SHIFT);
+ if (ret)
+ return ret;
+
+ mem->chunk_size = PAGE_SIZE;
+ mem->max_order =
+ get_order(rounddown_pow_of_two(resource_size(&mem->region)));
+ mem->is_range_manager = true;
+ mem->priv_ops = &priv_ops;
+ mem->region_private = ttm_manager_type(bdev, mem_type);
+
+ return 0;
+}
+
+/**
+ * intel_region_ttm_fini - Finalize a TTM region.
+ * @mem: The memory region
+ *
+ * This functions takes down the TTM resource manager associated with the
+ * memory region, and if it was registered with the TTM device,
+ * removes that registration.
+ */
+void intel_region_ttm_fini(struct intel_memory_region *mem)
+{
+ int ret;
+
+ ret = ttm_range_man_fini(&mem->i915->bdev,
+ intel_region_to_ttm_type(mem));
+ GEM_WARN_ON(ret);
+ mem->region_private = NULL;
+}
+
+/**
+ * intel_region_ttm_node_to_st - Convert an opaque TTM resource manager node
+ * to an sg_table.
+ * @mem: The memory region.
+ * @node: The resource manager node obtained from the TTM resource manager.
+ *
+ * The gem backends typically use sg-tables for operations on the underlying
+ * io_memory. So provide a way for the backends to translate the
+ * nodes they are handed from TTM to sg-tables.
+ *
+ * Return: A malloced sg_table on success, an error pointer on failure.
+ */
+struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
+ void *node)
+{
+ return i915_sg_from_mm_node(node, mem->region.start);
+}
+
+/**
+ * intel_region_ttm_node_alloc - Allocate memory resources from a region
+ * @mem: The memory region,
+ * @size: The requested size in bytes
+ * @flags: Allocation flags
+ *
+ * This functionality is provided only for callers that need to allocate
+ * memory from standalone TTM range managers, without the TTM eviction
+ * functionality. Don't use if you are not completely sure that's the
+ * case. The returned opaque node can be converted to an sg_table using
+ * intel_region_ttm_node_to_st(), and can be freed using
+ * intel_region_ttm_node_free().
+ *
+ * Return: A valid pointer on success, an error pointer on failure.
+ */
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags)
+{
+ struct ttm_resource_manager *man = mem->region_private;
+ struct ttm_place place = {};
+ struct ttm_resource res = {};
+ struct ttm_buffer_object mock_bo = {};
+ int ret;
+
+ /*
+ * We ignore the flags for now since we're using the range
+ * manager and contigous and min page size would be fulfilled
+ * by default if size is min page size aligned.
+ */
+ res.num_pages = size >> PAGE_SHIFT;
+
+ if (mem->is_range_manager) {
+ if (size >= SZ_1G)
+ mock_bo.page_alignment = SZ_1G >> PAGE_SHIFT;
+ else if (size >= SZ_2M)
+ mock_bo.page_alignment = SZ_2M >> PAGE_SHIFT;
+ else if (size >= SZ_64K)
+ mock_bo.page_alignment = SZ_64K >> PAGE_SHIFT;
+ }
+
+ ret = man->func->alloc(man, &mock_bo, &place, &res);
+ if (ret == -ENOSPC)
+ ret = -ENXIO;
+ return ret ? ERR_PTR(ret) : res.mm_node;
+}
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.h b/drivers/gpu/drm/i915/intel_region_ttm.h
new file mode 100644
index 000000000000..1c82c6c3429d
--- /dev/null
+++ b/drivers/gpu/drm/i915/intel_region_ttm.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef _INTEL_REGION_TTM_H_
+#define _INTEL_REGION_TTM_H_
+
+#include <linux/types.h>
+
+#include "i915_selftest.h"
+
+struct drm_i915_private;
+struct intel_memory_region;
+
+int intel_region_ttm_device_init(struct drm_i915_private *dev_priv);
+
+void intel_region_ttm_device_fini(struct drm_i915_private *dev_priv);
+
+int intel_region_ttm_init(struct intel_memory_region *mem);
+
+void intel_region_ttm_fini(struct intel_memory_region *mem);
+
+struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
+ void *node);
+
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags);
+
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node);
+#endif /* _INTEL_REGION_TTM_H_ */
diff --git a/drivers/gpu/drm/i915/selftests/i915_buddy.c b/drivers/gpu/drm/i915/selftests/i915_buddy.c
deleted file mode 100644
index f0f5c4df8dbc..000000000000
--- a/drivers/gpu/drm/i915/selftests/i915_buddy.c
+++ /dev/null
@@ -1,789 +0,0 @@
-// SPDX-License-Identifier: MIT
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#include <linux/prime_numbers.h>
-
-#include "../i915_selftest.h"
-#include "i915_random.h"
-
-static void __igt_dump_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block,
- bool buddy)
-{
- pr_err("block info: header=%llx, state=%u, order=%d, offset=%llx size=%llx root=%s buddy=%s\n",
- block->header,
- i915_buddy_block_state(block),
- i915_buddy_block_order(block),
- i915_buddy_block_offset(block),
- i915_buddy_block_size(mm, block),
- yesno(!block->parent),
- yesno(buddy));
-}
-
-static void igt_dump_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *buddy;
-
- __igt_dump_block(mm, block, false);
-
- buddy = get_buddy(block);
- if (buddy)
- __igt_dump_block(mm, buddy, true);
-}
-
-static int igt_check_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *buddy;
- unsigned int block_state;
- u64 block_size;
- u64 offset;
- int err = 0;
-
- block_state = i915_buddy_block_state(block);
-
- if (block_state != I915_BUDDY_ALLOCATED &&
- block_state != I915_BUDDY_FREE &&
- block_state != I915_BUDDY_SPLIT) {
- pr_err("block state mismatch\n");
- err = -EINVAL;
- }
-
- block_size = i915_buddy_block_size(mm, block);
- offset = i915_buddy_block_offset(block);
-
- if (block_size < mm->chunk_size) {
- pr_err("block size smaller than min size\n");
- err = -EINVAL;
- }
-
- if (!is_power_of_2(block_size)) {
- pr_err("block size not power of two\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(block_size, mm->chunk_size)) {
- pr_err("block size not aligned to min size\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(offset, mm->chunk_size)) {
- pr_err("block offset not aligned to min size\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(offset, block_size)) {
- pr_err("block offset not aligned to block size\n");
- err = -EINVAL;
- }
-
- buddy = get_buddy(block);
-
- if (!buddy && block->parent) {
- pr_err("buddy has gone fishing\n");
- err = -EINVAL;
- }
-
- if (buddy) {
- if (i915_buddy_block_offset(buddy) != (offset ^ block_size)) {
- pr_err("buddy has wrong offset\n");
- err = -EINVAL;
- }
-
- if (i915_buddy_block_size(mm, buddy) != block_size) {
- pr_err("buddy size mismatch\n");
- err = -EINVAL;
- }
-
- if (i915_buddy_block_state(buddy) == block_state &&
- block_state == I915_BUDDY_FREE) {
- pr_err("block and its buddy are free\n");
- err = -EINVAL;
- }
- }
-
- return err;
-}
-
-static int igt_check_blocks(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 expected_size,
- bool is_contiguous)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_block *prev;
- u64 total;
- int err = 0;
-
- block = NULL;
- prev = NULL;
- total = 0;
-
- list_for_each_entry(block, blocks, link) {
- err = igt_check_block(mm, block);
-
- if (!i915_buddy_block_is_allocated(block)) {
- pr_err("block not allocated\n"),
- err = -EINVAL;
- }
-
- if (is_contiguous && prev) {
- u64 prev_block_size;
- u64 prev_offset;
- u64 offset;
-
- prev_offset = i915_buddy_block_offset(prev);
- prev_block_size = i915_buddy_block_size(mm, prev);
- offset = i915_buddy_block_offset(block);
-
- if (offset != (prev_offset + prev_block_size)) {
- pr_err("block offset mismatch\n");
- err = -EINVAL;
- }
- }
-
- if (err)
- break;
-
- total += i915_buddy_block_size(mm, block);
- prev = block;
- }
-
- if (!err) {
- if (total != expected_size) {
- pr_err("size mismatch, expected=%llx, found=%llx\n",
- expected_size, total);
- err = -EINVAL;
- }
- return err;
- }
-
- if (prev) {
- pr_err("prev block, dump:\n");
- igt_dump_block(mm, prev);
- }
-
- if (block) {
- pr_err("bad block, dump:\n");
- igt_dump_block(mm, block);
- }
-
- return err;
-}
-
-static int igt_check_mm(struct i915_buddy_mm *mm)
-{
- struct i915_buddy_block *root;
- struct i915_buddy_block *prev;
- unsigned int i;
- u64 total;
- int err = 0;
-
- if (!mm->n_roots) {
- pr_err("n_roots is zero\n");
- return -EINVAL;
- }
-
- if (mm->n_roots != hweight64(mm->size)) {
- pr_err("n_roots mismatch, n_roots=%u, expected=%lu\n",
- mm->n_roots, hweight64(mm->size));
- return -EINVAL;
- }
-
- root = NULL;
- prev = NULL;
- total = 0;
-
- for (i = 0; i < mm->n_roots; ++i) {
- struct i915_buddy_block *block;
- unsigned int order;
-
- root = mm->roots[i];
- if (!root) {
- pr_err("root(%u) is NULL\n", i);
- err = -EINVAL;
- break;
- }
-
- err = igt_check_block(mm, root);
-
- if (!i915_buddy_block_is_free(root)) {
- pr_err("root not free\n");
- err = -EINVAL;
- }
-
- order = i915_buddy_block_order(root);
-
- if (!i) {
- if (order != mm->max_order) {
- pr_err("max order root missing\n");
- err = -EINVAL;
- }
- }
-
- if (prev) {
- u64 prev_block_size;
- u64 prev_offset;
- u64 offset;
-
- prev_offset = i915_buddy_block_offset(prev);
- prev_block_size = i915_buddy_block_size(mm, prev);
- offset = i915_buddy_block_offset(root);
-
- if (offset != (prev_offset + prev_block_size)) {
- pr_err("root offset mismatch\n");
- err = -EINVAL;
- }
- }
-
- block = list_first_entry_or_null(&mm->free_list[order],
- struct i915_buddy_block,
- link);
- if (block != root) {
- pr_err("root mismatch at order=%u\n", order);
- err = -EINVAL;
- }
-
- if (err)
- break;
-
- prev = root;
- total += i915_buddy_block_size(mm, root);
- }
-
- if (!err) {
- if (total != mm->size) {
- pr_err("expected mm size=%llx, found=%llx\n", mm->size,
- total);
- err = -EINVAL;
- }
- return err;
- }
-
- if (prev) {
- pr_err("prev root(%u), dump:\n", i - 1);
- igt_dump_block(mm, prev);
- }
-
- if (root) {
- pr_err("bad root(%u), dump:\n", i);
- igt_dump_block(mm, root);
- }
-
- return err;
-}
-
-static void igt_mm_config(u64 *size, u64 *chunk_size)
-{
- I915_RND_STATE(prng);
- u32 s, ms;
-
- /* Nothing fancy, just try to get an interesting bit pattern */
-
- prandom_seed_state(&prng, i915_selftest.random_seed);
-
- /* Let size be a random number of pages up to 8 GB (2M pages) */
- s = 1 + i915_prandom_u32_max_state((BIT(33 - 12)) - 1, &prng);
- /* Let the chunk size be a random power of 2 less than size */
- ms = BIT(i915_prandom_u32_max_state(ilog2(s), &prng));
- /* Round size down to the chunk size */
- s &= -ms;
-
- /* Convert from pages to bytes */
- *chunk_size = (u64)ms << 12;
- *size = (u64)s << 12;
-}
-
-static int igt_buddy_alloc_smoke(void *arg)
-{
- struct i915_buddy_mm mm;
- IGT_TIMEOUT(end_time);
- I915_RND_STATE(prng);
- u64 chunk_size;
- u64 mm_size;
- int *order;
- int err, i;
-
- igt_mm_config(&mm_size, &chunk_size);
-
- pr_info("buddy_init with size=%llx, chunk_size=%llx\n", mm_size, chunk_size);
-
- err = i915_buddy_init(&mm, mm_size, chunk_size);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
-
- order = i915_random_order(mm.max_order + 1, &prng);
- if (!order)
- goto out_fini;
-
- for (i = 0; i <= mm.max_order; ++i) {
- struct i915_buddy_block *block;
- int max_order = order[i];
- bool timeout = false;
- LIST_HEAD(blocks);
- int order;
- u64 total;
-
- err = igt_check_mm(&mm);
- if (err) {
- pr_err("pre-mm check failed, abort\n");
- break;
- }
-
- pr_info("filling from max_order=%u\n", max_order);
-
- order = max_order;
- total = 0;
-
- do {
-retry:
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- err = PTR_ERR(block);
- if (err == -ENOMEM) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- } else {
- if (order--) {
- err = 0;
- goto retry;
- }
-
- pr_err("buddy_alloc with order=%d failed(%d)\n",
- order, err);
- }
-
- break;
- }
-
- list_add_tail(&block->link, &blocks);
-
- if (i915_buddy_block_order(block) != order) {
- pr_err("buddy_alloc order mismatch\n");
- err = -EINVAL;
- break;
- }
-
- total += i915_buddy_block_size(&mm, block);
-
- if (__igt_timeout(end_time, NULL)) {
- timeout = true;
- break;
- }
- } while (total < mm.size);
-
- if (!err)
- err = igt_check_blocks(&mm, &blocks, total, false);
-
- i915_buddy_free_list(&mm, &blocks);
-
- if (!err) {
- err = igt_check_mm(&mm);
- if (err)
- pr_err("post-mm check failed\n");
- }
-
- if (err || timeout)
- break;
-
- cond_resched();
- }
-
- if (err == -ENOMEM)
- err = 0;
-
- kfree(order);
-out_fini:
- i915_buddy_fini(&mm);
-
- return err;
-}
-
-static int igt_buddy_alloc_pessimistic(void *arg)
-{
- const unsigned int max_order = 16;
- struct i915_buddy_block *block, *bn;
- struct i915_buddy_mm mm;
- unsigned int order;
- LIST_HEAD(blocks);
- int err;
-
- /*
- * Create a pot-sized mm, then allocate one of each possible
- * order within. This should leave the mm with exactly one
- * page left.
- */
-
- err = i915_buddy_init(&mm, PAGE_SIZE << max_order, PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (order = 0; order < max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
-
- list_add_tail(&block->link, &blocks);
- }
-
- /* And now the last remaining block available */
- block = i915_buddy_alloc(&mm, 0);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM on final alloc\n");
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &blocks);
-
- /* Should be completely full! */
- for (order = max_order; order--; ) {
- block = i915_buddy_alloc(&mm, order);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at order %d, it should be full!",
- order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
- block = list_last_entry(&blocks, typeof(*block), link);
- list_del(&block->link);
- i915_buddy_free(&mm, block);
-
- /* As we free in increasing size, we make available larger blocks */
- order = 1;
- list_for_each_entry_safe(block, bn, &blocks, link) {
- list_del(&block->link);
- i915_buddy_free(&mm, block);
-
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc (realloc) hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
- i915_buddy_free(&mm, block);
- order++;
- }
-
- /* To confirm, now the whole mm should be available */
- block = i915_buddy_alloc(&mm, max_order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc (realloc) hit -ENOMEM with order=%d\n",
- max_order);
- err = PTR_ERR(block);
- goto err;
- }
- i915_buddy_free(&mm, block);
-
-err:
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_optimistic(void *arg)
-{
- const int max_order = 16;
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- LIST_HEAD(blocks);
- int order;
- int err;
-
- /*
- * Create a mm with one block of each order available, and
- * try to allocate them all.
- */
-
- err = i915_buddy_init(&mm,
- PAGE_SIZE * ((1 << (max_order + 1)) - 1),
- PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (order = 0; order <= max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
-
- list_add_tail(&block->link, &blocks);
- }
-
- /* Should be completely full! */
- block = i915_buddy_alloc(&mm, 0);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded, it should be full!");
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
-
-err:
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_pathological(void *arg)
-{
- const int max_order = 16;
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- LIST_HEAD(blocks);
- LIST_HEAD(holes);
- int order, top;
- int err;
-
- /*
- * Create a pot-sized mm, then allocate one of each possible
- * order within. This should leave the mm with exactly one
- * page left. Free the largest block, then whittle down again.
- * Eventually we will have a fully 50% fragmented mm.
- */
-
- err = i915_buddy_init(&mm, PAGE_SIZE << max_order, PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (top = max_order; top; top--) {
- /* Make room by freeing the largest allocated block */
- block = list_first_entry_or_null(&blocks, typeof(*block), link);
- if (block) {
- list_del(&block->link);
- i915_buddy_free(&mm, block);
- }
-
- for (order = top; order--; ) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d, top=%d\n",
- order, top);
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &blocks);
- }
-
- /* There should be one final page for this sub-allocation */
- block = i915_buddy_alloc(&mm, 0);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM for hole\n");
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &holes);
-
- block = i915_buddy_alloc(&mm, top);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at top-order %d/%d, it should be full!",
- top, max_order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
- i915_buddy_free_list(&mm, &holes);
-
- /* Nothing larger than blocks of chunk_size now available */
- for (order = 1; order <= max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at order %d, it should be full!",
- order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
-err:
- list_splice_tail(&holes, &blocks);
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_range(void *arg)
-{
- struct i915_buddy_mm mm;
- unsigned long page_num;
- LIST_HEAD(blocks);
- u64 chunk_size;
- u64 offset;
- u64 size;
- u64 rem;
- int err;
-
- igt_mm_config(&size, &chunk_size);
-
- pr_info("buddy_init with size=%llx, chunk_size=%llx\n", size, chunk_size);
-
- err = i915_buddy_init(&mm, size, chunk_size);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
-
- err = igt_check_mm(&mm);
- if (err) {
- pr_err("pre-mm check failed, abort, abort, abort!\n");
- goto err_fini;
- }
-
- rem = mm.size;
- offset = 0;
-
- for_each_prime_number_from(page_num, 1, ULONG_MAX - 1) {
- struct i915_buddy_block *block;
- LIST_HEAD(tmp);
-
- size = min(page_num * mm.chunk_size, rem);
-
- err = i915_buddy_alloc_range(&mm, &tmp, offset, size);
- if (err) {
- if (err == -ENOMEM) {
- pr_info("alloc_range hit -ENOMEM with size=%llx\n",
- size);
- } else {
- pr_err("alloc_range with offset=%llx, size=%llx failed(%d)\n",
- offset, size, err);
- }
-
- break;
- }
-
- block = list_first_entry_or_null(&tmp,
- struct i915_buddy_block,
- link);
- if (!block) {
- pr_err("alloc_range has no blocks\n");
- err = -EINVAL;
- break;
- }
-
- if (i915_buddy_block_offset(block) != offset) {
- pr_err("alloc_range start offset mismatch, found=%llx, expected=%llx\n",
- i915_buddy_block_offset(block), offset);
- err = -EINVAL;
- }
-
- if (!err)
- err = igt_check_blocks(&mm, &tmp, size, true);
-
- list_splice_tail(&tmp, &blocks);
-
- if (err)
- break;
-
- offset += size;
-
- rem -= size;
- if (!rem)
- break;
-
- cond_resched();
- }
-
- if (err == -ENOMEM)
- err = 0;
-
- i915_buddy_free_list(&mm, &blocks);
-
- if (!err) {
- err = igt_check_mm(&mm);
- if (err)
- pr_err("post-mm check failed\n");
- }
-
-err_fini:
- i915_buddy_fini(&mm);
-
- return err;
-}
-
-static int igt_buddy_alloc_limit(void *arg)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- const u64 size = U64_MAX;
- int err;
-
- err = i915_buddy_init(&mm, size, PAGE_SIZE);
- if (err)
- return err;
-
- if (mm.max_order != I915_BUDDY_MAX_ORDER) {
- pr_err("mm.max_order(%d) != %d\n",
- mm.max_order, I915_BUDDY_MAX_ORDER);
- err = -EINVAL;
- goto out_fini;
- }
-
- block = i915_buddy_alloc(&mm, mm.max_order);
- if (IS_ERR(block)) {
- err = PTR_ERR(block);
- goto out_fini;
- }
-
- if (i915_buddy_block_order(block) != mm.max_order) {
- pr_err("block order(%d) != %d\n",
- i915_buddy_block_order(block), mm.max_order);
- err = -EINVAL;
- goto out_free;
- }
-
- if (i915_buddy_block_size(&mm, block) !=
- BIT_ULL(mm.max_order) * PAGE_SIZE) {
- pr_err("block size(%llu) != %llu\n",
- i915_buddy_block_size(&mm, block),
- BIT_ULL(mm.max_order) * PAGE_SIZE);
- err = -EINVAL;
- goto out_free;
- }
-
-out_free:
- i915_buddy_free(&mm, block);
-out_fini:
- i915_buddy_fini(&mm);
- return err;
-}
-
-int i915_buddy_mock_selftests(void)
-{
- static const struct i915_subtest tests[] = {
- SUBTEST(igt_buddy_alloc_pessimistic),
- SUBTEST(igt_buddy_alloc_optimistic),
- SUBTEST(igt_buddy_alloc_pathological),
- SUBTEST(igt_buddy_alloc_smoke),
- SUBTEST(igt_buddy_alloc_range),
- SUBTEST(igt_buddy_alloc_limit),
- };
-
- return i915_subtests(tests, NULL);
-}
diff --git a/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h b/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
index 3db34d3eea58..34e5caf38093 100644
--- a/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
+++ b/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
@@ -33,5 +33,4 @@ selftest(evict, i915_gem_evict_mock_selftests)
selftest(gtt, i915_gem_gtt_mock_selftests)
selftest(hugepages, i915_gem_huge_page_mock_selftests)
selftest(contexts, i915_gem_context_mock_selftests)
-selftest(buddy, i915_buddy_mock_selftests)
selftest(memory_region, intel_memory_region_mock_selftests)
diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
index f85fd8cbfbf5..c85d516b85cd 100644
--- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
@@ -57,9 +57,10 @@ static int igt_mock_fill(void *arg)
LIST_HEAD(objects);
int err = 0;
- page_size = mem->mm.chunk_size;
- max_pages = div64_u64(total, page_size);
+ page_size = mem->chunk_size;
rem = total;
+retry:
+ max_pages = div64_u64(rem, page_size);
for_each_prime_number_from(page_num, 1, max_pages) {
resource_size_t size = page_num * page_size;
@@ -85,6 +86,11 @@ static int igt_mock_fill(void *arg)
err = 0;
if (err == -ENXIO) {
if (page_num * page_size <= rem) {
+ if (mem->is_range_manager && max_pages > 1) {
+ max_pages >>= 1;
+ goto retry;
+ }
+
pr_err("%s failed, space still left in region\n",
__func__);
err = -EINVAL;
@@ -199,12 +205,18 @@ static int igt_mock_reserve(void *arg)
do {
u32 size = i915_prandom_u32_max_state(cur_avail, &prng);
+retry:
size = max_t(u32, round_up(size, PAGE_SIZE), PAGE_SIZE);
obj = igt_object_create(mem, &objects, size, 0);
if (IS_ERR(obj)) {
- if (PTR_ERR(obj) == -ENXIO)
+ if (PTR_ERR(obj) == -ENXIO) {
+ if (mem->is_range_manager &&
+ size > mem->chunk_size) {
+ size >>= 1;
+ goto retry;
+ }
break;
-
+ }
err = PTR_ERR(obj);
goto out_close;
}
@@ -220,7 +232,7 @@ static int igt_mock_reserve(void *arg)
out_close:
kfree(order);
close_objects(mem, &objects);
- i915_buddy_free_list(&mem->mm, &mem->reserved);
+ intel_memory_region_unreserve(mem);
return err;
}
@@ -240,7 +252,7 @@ static int igt_mock_contiguous(void *arg)
total = resource_size(&mem->region);
/* Min size */
- obj = igt_object_create(mem, &objects, mem->mm.chunk_size,
+ obj = igt_object_create(mem, &objects, mem->chunk_size,
I915_BO_ALLOC_CONTIGUOUS);
if (IS_ERR(obj))
return PTR_ERR(obj);
@@ -321,14 +333,16 @@ static int igt_mock_contiguous(void *arg)
min = target;
target = total >> 1;
- /* Make sure we can still allocate all the fragmented space */
- obj = igt_object_create(mem, &objects, target, 0);
- if (IS_ERR(obj)) {
- err = PTR_ERR(obj);
- goto err_close_objects;
- }
+ if (!mem->is_range_manager) {
+ /* Make sure we can still allocate all the fragmented space */
+ obj = igt_object_create(mem, &objects, target, 0);
+ if (IS_ERR(obj)) {
+ err = PTR_ERR(obj);
+ goto err_close_objects;
+ }
- igt_object_release(obj);
+ igt_object_release(obj);
+ }
/*
* Even though we have enough free space, we don't have a big enough
@@ -348,7 +362,7 @@ static int igt_mock_contiguous(void *arg)
}
target >>= 1;
- } while (target >= mem->mm.chunk_size);
+ } while (target >= mem->chunk_size);
err_close_objects:
list_splice_tail(&holes, &objects);
@@ -368,7 +382,7 @@ static int igt_mock_splintered_region(void *arg)
/*
* Sanity check we can still allocate everything even if the
- * mm.max_order != mm.size. i.e our starting address space size is not a
+ * max_order != mm.size. i.e our starting address space size is not a
* power-of-two.
*/
@@ -377,17 +391,10 @@ static int igt_mock_splintered_region(void *arg)
if (IS_ERR(mem))
return PTR_ERR(mem);
- if (mem->mm.size != size) {
- pr_err("%s size mismatch(%llu != %llu)\n",
- __func__, mem->mm.size, size);
- err = -EINVAL;
- goto out_put;
- }
-
expected_order = get_order(rounddown_pow_of_two(size));
- if (mem->mm.max_order != expected_order) {
+ if (mem->max_order != expected_order) {
pr_err("%s order mismatch(%u != %u)\n",
- __func__, mem->mm.max_order, expected_order);
+ __func__, mem->max_order, expected_order);
err = -EINVAL;
goto out_put;
}
@@ -408,12 +415,15 @@ static int igt_mock_splintered_region(void *arg)
* sure that does indeed hold true.
*/
- obj = igt_object_create(mem, &objects, size, I915_BO_ALLOC_CONTIGUOUS);
- if (!IS_ERR(obj)) {
- pr_err("%s too large contiguous allocation was not rejected\n",
- __func__);
- err = -EINVAL;
- goto out_close;
+ if (!mem->is_range_manager) {
+ obj = igt_object_create(mem, &objects, size,
+ I915_BO_ALLOC_CONTIGUOUS);
+ if (!IS_ERR(obj)) {
+ pr_err("%s too large contiguous allocation was not rejected\n",
+ __func__);
+ err = -EINVAL;
+ goto out_close;
+ }
}
obj = igt_object_create(mem, &objects, rounddown_pow_of_two(size),
@@ -432,68 +442,6 @@ static int igt_mock_splintered_region(void *arg)
return err;
}
-#ifndef SZ_8G
-#define SZ_8G BIT_ULL(33)
-#endif
-
-static int igt_mock_max_segment(void *arg)
-{
- const unsigned int max_segment = i915_sg_segment_size();
- struct intel_memory_region *mem = arg;
- struct drm_i915_private *i915 = mem->i915;
- struct drm_i915_gem_object *obj;
- struct i915_buddy_block *block;
- struct scatterlist *sg;
- LIST_HEAD(objects);
- u64 size;
- int err = 0;
-
- /*
- * While we may create very large contiguous blocks, we may need
- * to break those down for consumption elsewhere. In particular,
- * dma-mapping with scatterlist elements have an implicit limit of
- * UINT_MAX on each element.
- */
-
- size = SZ_8G;
- mem = mock_region_create(i915, 0, size, PAGE_SIZE, 0);
- if (IS_ERR(mem))
- return PTR_ERR(mem);
-
- obj = igt_object_create(mem, &objects, size, 0);
- if (IS_ERR(obj)) {
- err = PTR_ERR(obj);
- goto out_put;
- }
-
- size = 0;
- list_for_each_entry(block, &obj->mm.blocks, link) {
- if (i915_buddy_block_size(&mem->mm, block) > size)
- size = i915_buddy_block_size(&mem->mm, block);
- }
- if (size < max_segment) {
- pr_err("%s: Failed to create a huge contiguous block [> %u], largest block %lld\n",
- __func__, max_segment, size);
- err = -EINVAL;
- goto out_close;
- }
-
- for (sg = obj->mm.pages->sgl; sg; sg = sg_next(sg)) {
- if (sg->length > max_segment) {
- pr_err("%s: Created an oversized scatterlist entry, %u > %u\n",
- __func__, sg->length, max_segment);
- err = -EINVAL;
- goto out_close;
- }
- }
-
-out_close:
- close_objects(mem, &objects);
-out_put:
- intel_memory_region_put(mem);
- return err;
-}
-
static int igt_gpu_write_dw(struct intel_context *ce,
struct i915_vma *vma,
u32 dword,
@@ -1098,7 +1046,6 @@ int intel_memory_region_mock_selftests(void)
SUBTEST(igt_mock_fill),
SUBTEST(igt_mock_contiguous),
SUBTEST(igt_mock_splintered_region),
- SUBTEST(igt_mock_max_segment),
};
struct intel_memory_region *mem;
struct drm_i915_private *i915;
diff --git a/drivers/gpu/drm/i915/selftests/mock_gem_device.c b/drivers/gpu/drm/i915/selftests/mock_gem_device.c
index cf40004bc92a..d189c4bd4bef 100644
--- a/drivers/gpu/drm/i915/selftests/mock_gem_device.c
+++ b/drivers/gpu/drm/i915/selftests/mock_gem_device.c
@@ -32,6 +32,7 @@
#include "gt/intel_gt_requests.h"
#include "gt/mock_engine.h"
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "mock_request.h"
#include "mock_gem_device.h"
@@ -70,6 +71,7 @@ static void mock_device_release(struct drm_device *dev)
mock_fini_ggtt(&i915->ggtt);
destroy_workqueue(i915->wq);
+ intel_region_ttm_device_fini(i915);
intel_gt_driver_late_release(&i915->gt);
intel_memory_regions_driver_release(i915);
@@ -116,6 +118,7 @@ struct drm_i915_private *mock_gem_device(void)
#endif
struct drm_i915_private *i915;
struct pci_dev *pdev;
+ int ret;
pdev = kzalloc(sizeof(*pdev), GFP_KERNEL);
if (!pdev)
@@ -178,6 +181,10 @@ struct drm_i915_private *mock_gem_device(void)
atomic_inc(&i915->gt.wakeref.count); /* disable; no hw support */
i915->gt.awake = -ENODEV;
+ ret = intel_region_ttm_device_init(i915);
+ if (ret)
+ goto err_ttm;
+
i915->wq = alloc_ordered_workqueue("mock", 0);
if (!i915->wq)
goto err_drv;
@@ -201,6 +208,7 @@ struct drm_i915_private *mock_gem_device(void)
intel_engines_driver_register(i915);
i915->do_release = true;
+ ida_init(&i915->selftest.mock_region_instances);
return i915;
@@ -209,6 +217,8 @@ struct drm_i915_private *mock_gem_device(void)
err_unlock:
destroy_workqueue(i915->wq);
err_drv:
+ intel_region_ttm_device_fini(i915);
+err_ttm:
intel_gt_driver_late_release(&i915->gt);
intel_memory_regions_driver_release(i915);
drm_mode_config_cleanup(&i915->drm);
diff --git a/drivers/gpu/drm/i915/selftests/mock_region.c b/drivers/gpu/drm/i915/selftests/mock_region.c
index 5d2d010a1e22..eafc5a04975c 100644
--- a/drivers/gpu/drm/i915/selftests/mock_region.c
+++ b/drivers/gpu/drm/i915/selftests/mock_region.c
@@ -1,17 +1,56 @@
// SPDX-License-Identifier: MIT
/*
- * Copyright © 2019 Intel Corporation
+ * Copyright © 2019-2021 Intel Corporation
*/
+#include <linux/scatterlist.h>
+
+#include <drm/ttm/ttm_placement.h>
+
#include "gem/i915_gem_region.h"
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "mock_region.h"
+static void mock_region_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *pages)
+{
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ sg_free_table(pages);
+ kfree(pages);
+}
+
+static int mock_region_get_pages(struct drm_i915_gem_object *obj)
+{
+ unsigned int flags;
+ struct sg_table *pages;
+
+ flags = I915_ALLOC_MIN_PAGE_SIZE;
+ if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
+ flags |= I915_ALLOC_CONTIGUOUS;
+
+ obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
+ obj->base.size,
+ flags);
+ if (IS_ERR(obj->mm.st_mm_node))
+ return PTR_ERR(obj->mm.st_mm_node);
+
+ pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
+ if (IS_ERR(pages)) {
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ return PTR_ERR(pages);
+ }
+
+ __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
+
+ return 0;
+}
+
static const struct drm_i915_gem_object_ops mock_region_obj_ops = {
.name = "mock-region",
- .get_pages = i915_gem_object_get_pages_buddy,
- .put_pages = i915_gem_object_put_pages_buddy,
+ .get_pages = mock_region_get_pages,
+ .put_pages = mock_region_put_pages,
.release = i915_gem_object_release_memory_region,
};
@@ -23,7 +62,7 @@ static int mock_object_init(struct intel_memory_region *mem,
static struct lock_class_key lock_class;
struct drm_i915_private *i915 = mem->i915;
- if (size > mem->mm.size)
+ if (size > resource_size(&mem->region))
return -E2BIG;
drm_gem_private_object_init(&i915->drm, &obj->base, size);
@@ -38,9 +77,18 @@ static int mock_object_init(struct intel_memory_region *mem,
return 0;
}
+static void mock_region_fini(struct intel_memory_region *mem)
+{
+ struct drm_i915_private *i915 = mem->i915;
+ int instance = mem->instance;
+
+ intel_region_ttm_fini(mem);
+ ida_free(&i915->selftest.mock_region_instances, instance);
+}
+
static const struct intel_memory_region_ops mock_region_ops = {
- .init = intel_memory_region_init_buddy,
- .release = intel_memory_region_release_buddy,
+ .init = intel_region_ttm_init,
+ .release = mock_region_fini,
.init_object = mock_object_init,
};
@@ -51,6 +99,14 @@ mock_region_create(struct drm_i915_private *i915,
resource_size_t min_page_size,
resource_size_t io_start)
{
+ int instance = ida_alloc_max(&i915->selftest.mock_region_instances,
+ TTM_NUM_MEM_TYPES - TTM_PL_PRIV - 1,
+ GFP_KERNEL);
+
+ if (instance < 0)
+ return ERR_PTR(instance);
+
return intel_memory_region_create(i915, start, size, min_page_size,
- io_start, &mock_region_ops);
+ io_start, INTEL_MEMORY_MOCK, instance,
+ &mock_region_ops);
}
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 05/15] drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
Embed a struct ttm_buffer_object into the i915 gem object, making sure
we alias the gem object part. It's a bit unfortunate that the
struct ttm_buffer_ojbect embeds a gem object since we otherwise could
make the TTM part private to the TTM backend, and use the usual
i915 gem object for the other backends.
To make this a bit more storage efficient for the other backends,
we'd have to use a pointer for the gem object which would require
a lot of changes in the driver. We postpone that for later.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_object.c | 7 +++++++
drivers/gpu/drm/i915/gem/i915_gem_object_types.h | 12 +++++++++++-
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 2be6109d0093..5706d471692d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -62,6 +62,13 @@ void i915_gem_object_init(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_object_ops *ops,
struct lock_class_key *key, unsigned flags)
{
+ /*
+ * A gem object is embedded both in a struct ttm_buffer_object :/ and
+ * in a drm_i915_gem_object. Make sure they are aliased.
+ */
+ BUILD_BUG_ON(offsetof(typeof(*obj), base) !=
+ offsetof(typeof(*obj), __do_not_access.base));
+
spin_lock_init(&obj->vma.lock);
INIT_LIST_HEAD(&obj->vma.list);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index f5b46d11e6e6..d047ea126029 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -10,6 +10,7 @@
#include <linux/mmu_notifier.h>
#include <drm/drm_gem.h>
+#include <drm/ttm/ttm_bo_api.h>
#include <uapi/drm/i915_drm.h>
#include "i915_active.h"
@@ -99,7 +100,16 @@ struct i915_gem_object_page_iter {
};
struct drm_i915_gem_object {
- struct drm_gem_object base;
+ /*
+ * We might have reason to revisit the below since it wastes
+ * a lot of space for non-ttm gem objects.
+ * In any case, always use the accessors for the ttm_buffer_object
+ * when accessing it.
+ */
+ union {
+ struct drm_gem_object base;
+ struct ttm_buffer_object __do_not_access;
+ };
const struct drm_i915_gem_object_ops *ops;
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 05/15] drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
Embed a struct ttm_buffer_object into the i915 gem object, making sure
we alias the gem object part. It's a bit unfortunate that the
struct ttm_buffer_ojbect embeds a gem object since we otherwise could
make the TTM part private to the TTM backend, and use the usual
i915 gem object for the other backends.
To make this a bit more storage efficient for the other backends,
we'd have to use a pointer for the gem object which would require
a lot of changes in the driver. We postpone that for later.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_object.c | 7 +++++++
drivers/gpu/drm/i915/gem/i915_gem_object_types.h | 12 +++++++++++-
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 2be6109d0093..5706d471692d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -62,6 +62,13 @@ void i915_gem_object_init(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_object_ops *ops,
struct lock_class_key *key, unsigned flags)
{
+ /*
+ * A gem object is embedded both in a struct ttm_buffer_object :/ and
+ * in a drm_i915_gem_object. Make sure they are aliased.
+ */
+ BUILD_BUG_ON(offsetof(typeof(*obj), base) !=
+ offsetof(typeof(*obj), __do_not_access.base));
+
spin_lock_init(&obj->vma.lock);
INIT_LIST_HEAD(&obj->vma.list);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index f5b46d11e6e6..d047ea126029 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -10,6 +10,7 @@
#include <linux/mmu_notifier.h>
#include <drm/drm_gem.h>
+#include <drm/ttm/ttm_bo_api.h>
#include <uapi/drm/i915_drm.h>
#include "i915_active.h"
@@ -99,7 +100,16 @@ struct i915_gem_object_page_iter {
};
struct drm_i915_gem_object {
- struct drm_gem_object base;
+ /*
+ * We might have reason to revisit the below since it wastes
+ * a lot of space for non-ttm gem objects.
+ * In any case, always use the accessors for the ttm_buffer_object
+ * when accessing it.
+ */
+ union {
+ struct drm_gem_object base;
+ struct ttm_buffer_object __do_not_access;
+ };
const struct drm_i915_gem_object_ops *ops;
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 06/15] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
The internal ttm_bo_util memcpy uses ioremap functionality, and while it
probably might be possible to use it for copying in- and out of
sglist represented io memory, using io_mem_reserve() / io_mem_free()
callbacks, that would cause problems with fault().
Instead, implement a method mapping page-by-page using kmap_local()
semantics. As an additional benefit we then avoid the occasional global
TLB flushes of ioremap() and consuming ioremap space, elimination of a
critical point of failure and with a slight change of semantics we could
also push the memcpy out async for testing and async driver development
purposes.
A special linear iomem iterator is introduced internally to mimic the
old ioremap behaviour for code-paths that can't immediately be ported
over. This adds to the code size and should be considered a temporary
solution.
Looking at the code we have a lot of checks for iomap tagged pointers.
Ideally we should extend the core memremap functions to also accept
uncached memory and kmap_local functionality. Then we could strip a
lot of code.
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v3:
- Split up in various TTM files and addressed review comments by
Christian König. Tested and fixed legacy iomap memcpy path on i915.
v4:
- Fix an uninitialized variable
Reported by: kernel test robot <lkp@intel.com>
Reported by: Dan Carpenter <dan.carpenter@oracle.com>
- Minor change to the ttm_move_memcpy() interface.
- Gracefully handle lack of memremap() support on memcpy
(Reported by Matthew Auld)
- Minor style fix (Reported by Matthew Auld)
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 280 ++++++++++-------------------
drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
drivers/gpu/drm/ttm/ttm_resource.c | 193 ++++++++++++++++++++
drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
include/drm/ttm/ttm_bo_driver.h | 28 +++
include/drm/ttm/ttm_caching.h | 2 +
include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
include/drm/ttm/ttm_resource.h | 61 +++++++
include/drm/ttm/ttm_tt.h | 16 ++
9 files changed, 536 insertions(+), 182 deletions(-)
create mode 100644 include/drm/ttm/ttm_kmap_iter.h
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index ae8b61460724..6ac7744a1a5c 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
mem->bus.addr = NULL;
}
-static int ttm_resource_ioremap(struct ttm_device *bdev,
- struct ttm_resource *mem,
- void **virtual)
+/**
+ * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
+ * @bo: The struct ttm_buffer_object.
+ * @new_mem: The struct ttm_resource we're moving to (copy destination).
+ * @new_iter: A struct ttm_kmap_iter representing the destination resource.
+ * @src_iter: A struct ttm_kmap_iter representing the source resource.
+ *
+ * This function is intended to be able to move out async under a
+ * dma-fence if desired.
+ */
+void ttm_move_memcpy(struct ttm_buffer_object *bo,
+ pgoff_t num_pages,
+ struct ttm_kmap_iter *dst_iter,
+ struct ttm_kmap_iter *src_iter)
{
- int ret;
- void *addr;
-
- *virtual = NULL;
- ret = ttm_mem_io_reserve(bdev, mem);
- if (ret || !mem->bus.is_iomem)
- return ret;
+ const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
+ const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
+ struct ttm_tt *ttm = bo->ttm;
+ struct dma_buf_map src_map, dst_map;
+ pgoff_t i;
- if (mem->bus.addr) {
- addr = mem->bus.addr;
- } else {
- size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
+ /* Single TTM move. NOP */
+ if (dst_ops->maps_tt && src_ops->maps_tt)
+ return;
- if (mem->bus.caching == ttm_write_combined)
- addr = ioremap_wc(mem->bus.offset, bus_size);
-#ifdef CONFIG_X86
- else if (mem->bus.caching == ttm_cached)
- addr = ioremap_cache(mem->bus.offset, bus_size);
-#endif
- else
- addr = ioremap(mem->bus.offset, bus_size);
- if (!addr) {
- ttm_mem_io_free(bdev, mem);
- return -ENOMEM;
+ /* Don't move nonexistent data. Clear destination instead. */
+ if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
+ if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
+ return;
+
+ for (i = 0; i < num_pages; ++i) {
+ dst_ops->map_local(dst_iter, &dst_map, i);
+ if (dst_map.is_iomem)
+ memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
+ else
+ memset(dst_map.vaddr, 0, PAGE_SIZE);
+ if (dst_ops->unmap_local)
+ dst_ops->unmap_local(dst_iter, &dst_map);
}
+ return;
}
- *virtual = addr;
- return 0;
-}
-
-static void ttm_resource_iounmap(struct ttm_device *bdev,
- struct ttm_resource *mem,
- void *virtual)
-{
- if (virtual && mem->bus.addr == NULL)
- iounmap(virtual);
- ttm_mem_io_free(bdev, mem);
-}
-
-static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
-{
- uint32_t *dstP =
- (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
- uint32_t *srcP =
- (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
-
- int i;
- for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
- iowrite32(ioread32(srcP++), dstP++);
- return 0;
-}
-
-static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
- unsigned long page,
- pgprot_t prot)
-{
- struct page *d = ttm->pages[page];
- void *dst;
-
- if (!d)
- return -ENOMEM;
-
- src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
- dst = kmap_atomic_prot(d, prot);
- if (!dst)
- return -ENOMEM;
-
- memcpy_fromio(dst, src, PAGE_SIZE);
-
- kunmap_atomic(dst);
-
- return 0;
-}
-
-static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
- unsigned long page,
- pgprot_t prot)
-{
- struct page *s = ttm->pages[page];
- void *src;
- if (!s)
- return -ENOMEM;
-
- dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
- src = kmap_atomic_prot(s, prot);
- if (!src)
- return -ENOMEM;
-
- memcpy_toio(dst, src, PAGE_SIZE);
-
- kunmap_atomic(src);
+ for (i = 0; i < num_pages; ++i) {
+ dst_ops->map_local(dst_iter, &dst_map, i);
+ src_ops->map_local(src_iter, &src_map, i);
+
+ if (!src_map.is_iomem && !dst_map.is_iomem) {
+ memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
+ } else if (!src_map.is_iomem) {
+ dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
+ PAGE_SIZE);
+ } else if (!dst_map.is_iomem) {
+ memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
+ PAGE_SIZE);
+ } else {
+ int j;
+ u32 __iomem *src = src_map.vaddr_iomem;
+ u32 __iomem *dst = dst_map.vaddr_iomem;
- return 0;
+ for (j = 0; j < (PAGE_SIZE / sizeof(u32)); ++j)
+ iowrite32(ioread32(src++), dst++);
+ }
+ if (src_ops->unmap_local)
+ src_ops->unmap_local(src_iter, &src_map);
+ if (dst_ops->unmap_local)
+ dst_ops->unmap_local(dst_iter, &dst_map);
+ }
}
+EXPORT_SYMBOL(ttm_move_memcpy);
int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
struct ttm_operation_ctx *ctx,
- struct ttm_resource *new_mem)
+ struct ttm_resource *dst_mem)
{
struct ttm_device *bdev = bo->bdev;
- struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
+ struct ttm_resource_manager *dst_man =
+ ttm_manager_type(bo->bdev, dst_mem->mem_type);
struct ttm_tt *ttm = bo->ttm;
- struct ttm_resource *old_mem = &bo->mem;
- struct ttm_resource old_copy = *old_mem;
- void *old_iomap;
- void *new_iomap;
- int ret;
- unsigned long i;
-
- ret = ttm_bo_wait_ctx(bo, ctx);
- if (ret)
- return ret;
-
- ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
- if (ret)
- return ret;
- ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
- if (ret)
- goto out;
-
- /*
- * Single TTM move. NOP.
- */
- if (old_iomap == NULL && new_iomap == NULL)
- goto out2;
-
- /*
- * Don't move nonexistent data. Clear destination instead.
- */
- if (old_iomap == NULL &&
- (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
- !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
- memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
- goto out2;
- }
+ struct ttm_resource *src_mem = &bo->mem;
+ struct ttm_resource_manager *src_man =
+ ttm_manager_type(bdev, src_mem->mem_type);
+ struct ttm_resource src_copy = *src_mem;
+ union {
+ struct ttm_kmap_iter_tt tt;
+ struct ttm_kmap_iter_linear_io io;
+ } _dst_iter, _src_iter;
+ struct ttm_kmap_iter *dst_iter, *src_iter;
+ int ret = 0;
- /*
- * TTM might be null for moves within the same region.
- */
- if (ttm) {
+ if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
+ dst_man->use_tt)) {
ret = ttm_tt_populate(bdev, ttm, ctx);
if (ret)
- goto out1;
+ return ret;
}
- for (i = 0; i < new_mem->num_pages; ++i) {
- if (old_iomap == NULL) {
- pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
- ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
- prot);
- } else if (new_iomap == NULL) {
- pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
- ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
- prot);
- } else {
- ret = ttm_copy_io_page(new_iomap, old_iomap, i);
- }
- if (ret)
- goto out1;
+ dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
+ if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
+ dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
+ if (IS_ERR(dst_iter))
+ return PTR_ERR(dst_iter);
+
+ src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
+ if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
+ src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
+ if (IS_ERR(src_iter)) {
+ ret = PTR_ERR(src_iter);
+ goto out_src_iter;
}
- mb();
-out2:
- old_copy = *old_mem;
- ttm_bo_assign_mem(bo, new_mem);
-
- if (!man->use_tt)
- ttm_bo_tt_destroy(bo);
+ ttm_move_memcpy(bo, dst_mem->num_pages, dst_iter, src_iter);
+ src_copy = *src_mem;
+ ttm_bo_move_sync_cleanup(bo, dst_mem);
-out1:
- ttm_resource_iounmap(bdev, old_mem, new_iomap);
-out:
- ttm_resource_iounmap(bdev, &old_copy, old_iomap);
+ if (!src_iter->ops->maps_tt)
+ ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
+out_src_iter:
+ if (!dst_iter->ops->maps_tt)
+ ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
- /*
- * On error, keep the mm node!
- */
- if (!ret)
- ttm_resource_free(bo, &old_copy);
return ret;
}
EXPORT_SYMBOL(ttm_bo_move_memcpy);
@@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
man = ttm_manager_type(bo->bdev, res->mem_type);
caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
- /* Cached mappings need no adjustment */
- if (caching == ttm_cached)
- return tmp;
-
-#if defined(__i386__) || defined(__x86_64__)
- if (caching == ttm_write_combined)
- tmp = pgprot_writecombine(tmp);
- else if (boot_cpu_data.x86 > 3)
- tmp = pgprot_noncached(tmp);
-#endif
-#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
- defined(__powerpc__) || defined(__mips__)
- if (caching == ttm_write_combined)
- tmp = pgprot_writecombine(tmp);
- else
- tmp = pgprot_noncached(tmp);
-#endif
-#if defined(__sparc__)
- tmp = pgprot_noncached(tmp);
-#endif
- return tmp;
+ return ttm_prot_from_caching(caching, tmp);
}
EXPORT_SYMBOL(ttm_io_prot);
diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
index 56b0efdba1a9..997c458f68a9 100644
--- a/drivers/gpu/drm/ttm/ttm_module.c
+++ b/drivers/gpu/drm/ttm/ttm_module.c
@@ -31,12 +31,47 @@
*/
#include <linux/module.h>
#include <linux/device.h>
+#include <linux/pgtable.h>
#include <linux/sched.h>
#include <linux/debugfs.h>
#include <drm/drm_sysfs.h>
+#include <drm/ttm/ttm_caching.h>
#include "ttm_module.h"
+/**
+ * ttm_prot_from_caching - Modify the page protection according to the
+ * ttm cacing mode
+ * @caching: The ttm caching mode
+ * @tmp: The original page protection
+ *
+ * Return: The modified page protection
+ */
+pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
+{
+ /* Cached mappings need no adjustment */
+ if (caching == ttm_cached)
+ return tmp;
+
+#if defined(__i386__) || defined(__x86_64__)
+ if (caching == ttm_write_combined)
+ tmp = pgprot_writecombine(tmp);
+ else if (boot_cpu_data.x86 > 3)
+ tmp = pgprot_noncached(tmp);
+#endif
+#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
+ defined(__powerpc__) || defined(__mips__)
+ if (caching == ttm_write_combined)
+ tmp = pgprot_writecombine(tmp);
+ else
+ tmp = pgprot_noncached(tmp);
+#endif
+#if defined(__sparc__)
+ tmp = pgprot_noncached(tmp);
+#endif
+ return tmp;
+}
+
struct dentry *ttm_debugfs_root;
static int __init ttm_init(void)
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 59e2b7157e41..931bcd1a9b9b 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -22,6 +22,10 @@
* Authors: Christian König
*/
+#include <linux/dma-buf-map.h>
+#include <linux/io-mapping.h>
+#include <linux/scatterlist.h>
+
#include <drm/ttm/ttm_resource.h>
#include <drm/ttm/ttm_bo_driver.h>
@@ -147,3 +151,192 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
man->func->debug(man, p);
}
EXPORT_SYMBOL(ttm_resource_manager_debug);
+
+static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_iomap *iter_io =
+ container_of(iter, typeof(*iter_io), base);
+ void __iomem *addr;
+
+retry:
+ while (i >= iter_io->cache.end) {
+ iter_io->cache.sg = iter_io->cache.sg ?
+ sg_next(iter_io->cache.sg) : iter_io->st->sgl;
+ iter_io->cache.i = iter_io->cache.end;
+ iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
+ PAGE_SHIFT;
+ iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
+ iter_io->start;
+ }
+
+ if (i < iter_io->cache.i) {
+ iter_io->cache.end = 0;
+ iter_io->cache.sg = NULL;
+ goto retry;
+ }
+
+ addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
+ (((resource_size_t)i - iter_io->cache.i)
+ << PAGE_SHIFT));
+ dma_buf_map_set_vaddr_iomem(dmap, addr);
+}
+
+static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *map)
+{
+ io_mapping_unmap_local(map->vaddr_iomem);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
+ .map_local = ttm_kmap_iter_iomap_map_local,
+ .unmap_local = ttm_kmap_iter_iomap_unmap_local,
+ .maps_tt = false,
+};
+
+/**
+ * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
+ * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
+ * @iomap: The struct io_mapping representing the underlying linear io_memory.
+ * @st: sg_table into @iomap, representing the memory of the struct
+ * ttm_resource.
+ * @start: Offset that needs to be subtracted from @st to make
+ * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
+ *
+ * Return: Pointer to the embedded struct ttm_kmap_iter.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start)
+{
+ iter_io->base.ops = &ttm_kmap_iter_io_ops;
+ iter_io->iomap = iomap;
+ iter_io->st = st;
+ iter_io->start = start;
+ memset(&iter_io->cache, 0, sizeof(iter_io->cache));
+
+ return &iter_io->base;
+}
+EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
+
+/**
+ * DOC: Linear io iterator
+ *
+ * This code should die in the not too near future. Best would be if we could
+ * make io-mapping use memremap for all io memory, and have memremap
+ * implement a kmap_local functionality. We could then strip a huge amount of
+ * code. These linear io iterators are implemented to mimic old functionality,
+ * and they don't use kmap_local semantics at all internally. Rather ioremap or
+ * friends, and at least on 32-bit they add global TLB flushes and points
+ * of failure.
+ */
+
+static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_linear_io *iter_io =
+ container_of(iter, typeof(*iter_io), base);
+
+ *dmap = iter_io->dmap;
+ dma_buf_map_incr(dmap, i * PAGE_SIZE);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
+ .map_local = ttm_kmap_iter_linear_io_map_local,
+ .maps_tt = false,
+};
+
+/**
+ * ttm_kmap_iter_linear_io_init - Initialize an iterator for linear io memory
+ * @iter_io: The iterator to initialize
+ * @bdev: The TTM device
+ * @mem: The ttm resource representing the iomap.
+ *
+ * This function is for internal TTM use only. It sets up a memcpy kmap iterator
+ * pointing at a linear chunk of io memory.
+ *
+ * Return: A pointer to the embedded struct ttm_kmap_iter or error pointer on
+ * failure.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem)
+{
+ int ret;
+
+ ret = ttm_mem_io_reserve(bdev, mem);
+ if (ret)
+ goto out_err;
+ if (!mem->bus.is_iomem) {
+ ret = -EINVAL;
+ goto out_io_free;
+ }
+
+ if (mem->bus.addr) {
+ dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
+ iter_io->needs_unmap = false;
+ } else {
+ size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
+
+ iter_io->needs_unmap = true;
+ memset(&iter_io->dmap, 0, sizeof(iter_io->dmap));
+ if (mem->bus.caching == ttm_write_combined)
+ dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
+ ioremap_wc(mem->bus.offset,
+ bus_size));
+ else if (mem->bus.caching == ttm_cached)
+ dma_buf_map_set_vaddr(&iter_io->dmap,
+ memremap(mem->bus.offset, bus_size,
+ MEMREMAP_WB |
+ MEMREMAP_WT |
+ MEMREMAP_WC));
+
+ /* If uncached requested or if mapping cached or wc failed */
+ if (dma_buf_map_is_null(&iter_io->dmap))
+ dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
+ ioremap(mem->bus.offset,
+ bus_size));
+
+ if (dma_buf_map_is_null(&iter_io->dmap)) {
+ ret = -ENOMEM;
+ goto out_io_free;
+ }
+ }
+
+ iter_io->base.ops = &ttm_kmap_iter_linear_io_ops;
+ return &iter_io->base;
+
+out_io_free:
+ ttm_mem_io_free(bdev, mem);
+out_err:
+ return ERR_PTR(ret);
+}
+
+/**
+ * ttm_kmap_iter_linear_io_fini - Clean up an iterator for linear io memory
+ * @iter_io: The iterator to initialize
+ * @bdev: The TTM device
+ * @mem: The ttm resource representing the iomap.
+ *
+ * This function is for internal TTM use only. It cleans up a memcpy kmap
+ * iterator initialized by ttm_kmap_iter_linear_io_init.
+ */
+void
+ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem)
+{
+ if (iter_io->needs_unmap && dma_buf_map_is_set(&iter_io->dmap)) {
+ if (iter_io->dmap.is_iomem)
+ iounmap(iter_io->dmap.vaddr_iomem);
+ else
+ memunmap(iter_io->dmap.vaddr);
+ }
+
+ ttm_mem_io_free(bdev, mem);
+}
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 539e0232cb3b..0e41227116b1 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -433,3 +433,45 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages)
if (!ttm_dma32_pages_limit)
ttm_dma32_pages_limit = num_dma32_pages;
}
+
+static void ttm_kmap_iter_tt_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_tt *iter_tt =
+ container_of(iter, typeof(*iter_tt), base);
+
+ dma_buf_map_set_vaddr(dmap, kmap_local_page_prot(iter_tt->tt->pages[i],
+ iter_tt->prot));
+}
+
+static void ttm_kmap_iter_tt_unmap_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *map)
+{
+ kunmap_local(map->vaddr);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_tt_ops = {
+ .map_local = ttm_kmap_iter_tt_map_local,
+ .unmap_local = ttm_kmap_iter_tt_unmap_local,
+ .maps_tt = true,
+};
+
+/**
+ * ttm_kmap_iter_tt_init - Initialize a struct ttm_kmap_iter_tt
+ * @iter_tt: The struct ttm_kmap_iter_tt to initialize.
+ * @tt: Struct ttm_tt holding page pointers of the struct ttm_resource.
+ *
+ * Return: Pointer to the embedded struct ttm_kmap_iter.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
+ struct ttm_tt *tt)
+{
+ iter_tt->base.ops = &ttm_kmap_iter_tt_ops;
+ iter_tt->tt = tt;
+ iter_tt->prot = ttm_prot_from_caching(tt->caching, PAGE_KERNEL);
+
+ return &iter_tt->base;
+}
+EXPORT_SYMBOL(ttm_kmap_iter_tt_init);
diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
index dbccac957f8f..fdbeac78c236 100644
--- a/include/drm/ttm/ttm_bo_driver.h
+++ b/include/drm/ttm/ttm_bo_driver.h
@@ -40,6 +40,7 @@
#include <drm/ttm/ttm_device.h>
#include "ttm_bo_api.h"
+#include "ttm_kmap_iter.h"
#include "ttm_placement.h"
#include "ttm_tt.h"
#include "ttm_pool.h"
@@ -272,6 +273,23 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
bool pipeline,
struct ttm_resource *new_mem);
+/**
+ * ttm_bo_move_accel_cleanup.
+ *
+ * @bo: A pointer to a struct ttm_buffer_object.
+ * @new_mem: struct ttm_resource indicating where to move.
+ *
+ * Special case of ttm_bo_move_accel_cleanup where the bo is guaranteed
+ * by the caller to be idle. Typically used after memcpy buffer moves.
+ */
+static inline void ttm_bo_move_sync_cleanup(struct ttm_buffer_object *bo,
+ struct ttm_resource *new_mem)
+{
+ int ret = ttm_bo_move_accel_cleanup(bo, NULL, true, false, new_mem);
+
+ WARN_ON(ret);
+}
+
/**
* ttm_bo_pipeline_gutting.
*
@@ -332,4 +350,14 @@ int ttm_range_man_init(struct ttm_device *bdev,
int ttm_range_man_fini(struct ttm_device *bdev,
unsigned type);
+void ttm_move_memcpy(struct ttm_buffer_object *bo,
+ pgoff_t num_pages,
+ struct ttm_kmap_iter *dst_iter,
+ struct ttm_kmap_iter *src_iter);
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start);
#endif
diff --git a/include/drm/ttm/ttm_caching.h b/include/drm/ttm/ttm_caching.h
index a0b4a49fa432..3c9dd65f5aaf 100644
--- a/include/drm/ttm/ttm_caching.h
+++ b/include/drm/ttm/ttm_caching.h
@@ -33,4 +33,6 @@ enum ttm_caching {
ttm_cached
};
+pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp);
+
#endif
diff --git a/include/drm/ttm/ttm_kmap_iter.h b/include/drm/ttm/ttm_kmap_iter.h
new file mode 100644
index 000000000000..8bb00fd39d6c
--- /dev/null
+++ b/include/drm/ttm/ttm_kmap_iter.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef __TTM_KMAP_ITER_H__
+#define __TTM_KMAP_ITER_H__
+
+#include <linux/types.h>
+
+struct ttm_kmap_iter;
+struct dma_buf_map;
+
+/**
+ * struct ttm_kmap_iter_ops - Ops structure for a struct
+ * ttm_kmap_iter.
+ * @maps_tt: Whether the iterator maps TT memory directly, as opposed
+ * mapping a TT through an aperture. Both these modes have
+ * struct ttm_resource_manager::use_tt set, but the latter typically
+ * returns is_iomem == true from ttm_mem_io_reserve.
+ */
+struct ttm_kmap_iter_ops {
+ /**
+ * kmap_local() - Map a PAGE_SIZE part of the resource using
+ * kmap_local semantics.
+ * @res_iter: Pointer to the struct ttm_kmap_iter representing
+ * the resource.
+ * @dmap: The struct dma_buf_map holding the virtual address after
+ * the operation.
+ * @i: The location within the resource to map. PAGE_SIZE granularity.
+ */
+ void (*map_local)(struct ttm_kmap_iter *res_iter,
+ struct dma_buf_map *dmap, pgoff_t i);
+ /**
+ * unmap_local() - Unmap a PAGE_SIZE part of the resource previously
+ * mapped using kmap_local.
+ * @res_iter: Pointer to the struct ttm_kmap_iter representing
+ * the resource.
+ * @dmap: The struct dma_buf_map holding the virtual address after
+ * the operation.
+ */
+ void (*unmap_local)(struct ttm_kmap_iter *res_iter,
+ struct dma_buf_map *dmap);
+ bool maps_tt;
+};
+
+/**
+ * struct ttm_kmap_iter - Iterator for kmap_local type operations on a
+ * resource.
+ * @ops: Pointer to the operations struct.
+ *
+ * This struct is intended to be embedded in a resource-specific specialization
+ * implementing operations for the resource.
+ *
+ * Nothing stops us from extending the operations to vmap, vmap_pfn etc,
+ * replacing some or parts of the ttm_bo_util. cpu-map functionality.
+ */
+struct ttm_kmap_iter {
+ const struct ttm_kmap_iter_ops *ops;
+};
+
+#endif /* __TTM_KMAP_ITER_H__ */
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index 890b9d369519..b8dc0bdb0da5 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -27,9 +27,11 @@
#include <linux/types.h>
#include <linux/mutex.h>
+#include <linux/dma-buf-map.h>
#include <linux/dma-fence.h>
#include <drm/drm_print.h>
#include <drm/ttm/ttm_caching.h>
+#include <drm/ttm/ttm_kmap_iter.h>
#define TTM_MAX_BO_PRIORITY 4U
@@ -38,6 +40,10 @@ struct ttm_resource_manager;
struct ttm_resource;
struct ttm_place;
struct ttm_buffer_object;
+struct dma_buf_map;
+struct io_mapping;
+struct sg_table;
+struct scatterlist;
struct ttm_resource_manager_func {
/**
@@ -176,6 +182,45 @@ struct ttm_resource {
struct ttm_bus_placement bus;
};
+/**
+ * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
+ * struct sg_table backed struct ttm_resource.
+ * @base: Embedded struct ttm_kmap_iter providing the usage interface.
+ * @iomap: struct io_mapping representing the underlying linear io_memory.
+ * @st: sg_table into @iomap, representing the memory of the struct ttm_resource.
+ * @start: Offset that needs to be subtracted from @st to make
+ * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
+ * @cache: Scatterlist traversal cache for fast lookups.
+ * @cache.sg: Pointer to the currently cached scatterlist segment.
+ * @cache.i: First index of @sg. PAGE_SIZE granularity.
+ * @cache.end: Last index + 1 of @sg. PAGE_SIZE granularity.
+ * @cache.offs: First offset into @iomap of @sg. PAGE_SIZE granularity.
+ */
+struct ttm_kmap_iter_iomap {
+ struct ttm_kmap_iter base;
+ struct io_mapping *iomap;
+ struct sg_table *st;
+ resource_size_t start;
+ struct {
+ struct scatterlist *sg;
+ pgoff_t i;
+ pgoff_t end;
+ pgoff_t offs;
+ } cache;
+};
+
+/**
+ * struct ttm_kmap_iter_linear_io - Iterator specialization for linear io
+ * @base: The base iterator
+ * @dmap: Points to the starting address of the region
+ * @needs_unmap: Whether we need to unmap on fini
+ */
+struct ttm_kmap_iter_linear_io {
+ struct ttm_kmap_iter base;
+ struct dma_buf_map dmap;
+ bool needs_unmap;
+};
+
/**
* ttm_resource_manager_set_used
*
@@ -237,4 +282,20 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
void ttm_resource_manager_debug(struct ttm_resource_manager *man,
struct drm_printer *p);
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start);
+
+struct ttm_kmap_iter_linear_io;
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem);
+
+void ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem);
#endif
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 134d09ef7766..3102059db726 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -29,6 +29,7 @@
#include <linux/types.h>
#include <drm/ttm/ttm_caching.h>
+#include <drm/ttm/ttm_kmap_iter.h>
struct ttm_bo_device;
struct ttm_tt;
@@ -69,6 +70,18 @@ struct ttm_tt {
enum ttm_caching caching;
};
+/**
+ * struct ttm_kmap_iter_tt - Specialization of a mappig iterator for a tt.
+ * @base: Embedded struct ttm_kmap_iter providing the usage interface
+ * @tt: Cached struct ttm_tt.
+ * @prot: Cached page protection for mapping.
+ */
+struct ttm_kmap_iter_tt {
+ struct ttm_kmap_iter base;
+ struct ttm_tt *tt;
+ pgprot_t prot;
+};
+
static inline bool ttm_tt_is_populated(struct ttm_tt *tt)
{
return tt->page_flags & TTM_PAGE_FLAG_PRIV_POPULATED;
@@ -159,6 +172,9 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
+struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
+ struct ttm_tt *tt);
+
#if IS_ENABLED(CONFIG_AGP)
#include <linux/agp_backend.h>
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 06/15] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
The internal ttm_bo_util memcpy uses ioremap functionality, and while it
probably might be possible to use it for copying in- and out of
sglist represented io memory, using io_mem_reserve() / io_mem_free()
callbacks, that would cause problems with fault().
Instead, implement a method mapping page-by-page using kmap_local()
semantics. As an additional benefit we then avoid the occasional global
TLB flushes of ioremap() and consuming ioremap space, elimination of a
critical point of failure and with a slight change of semantics we could
also push the memcpy out async for testing and async driver development
purposes.
A special linear iomem iterator is introduced internally to mimic the
old ioremap behaviour for code-paths that can't immediately be ported
over. This adds to the code size and should be considered a temporary
solution.
Looking at the code we have a lot of checks for iomap tagged pointers.
Ideally we should extend the core memremap functions to also accept
uncached memory and kmap_local functionality. Then we could strip a
lot of code.
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v3:
- Split up in various TTM files and addressed review comments by
Christian König. Tested and fixed legacy iomap memcpy path on i915.
v4:
- Fix an uninitialized variable
Reported by: kernel test robot <lkp@intel.com>
Reported by: Dan Carpenter <dan.carpenter@oracle.com>
- Minor change to the ttm_move_memcpy() interface.
- Gracefully handle lack of memremap() support on memcpy
(Reported by Matthew Auld)
- Minor style fix (Reported by Matthew Auld)
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 280 ++++++++++-------------------
drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
drivers/gpu/drm/ttm/ttm_resource.c | 193 ++++++++++++++++++++
drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
include/drm/ttm/ttm_bo_driver.h | 28 +++
include/drm/ttm/ttm_caching.h | 2 +
include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
include/drm/ttm/ttm_resource.h | 61 +++++++
include/drm/ttm/ttm_tt.h | 16 ++
9 files changed, 536 insertions(+), 182 deletions(-)
create mode 100644 include/drm/ttm/ttm_kmap_iter.h
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index ae8b61460724..6ac7744a1a5c 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
mem->bus.addr = NULL;
}
-static int ttm_resource_ioremap(struct ttm_device *bdev,
- struct ttm_resource *mem,
- void **virtual)
+/**
+ * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
+ * @bo: The struct ttm_buffer_object.
+ * @new_mem: The struct ttm_resource we're moving to (copy destination).
+ * @new_iter: A struct ttm_kmap_iter representing the destination resource.
+ * @src_iter: A struct ttm_kmap_iter representing the source resource.
+ *
+ * This function is intended to be able to move out async under a
+ * dma-fence if desired.
+ */
+void ttm_move_memcpy(struct ttm_buffer_object *bo,
+ pgoff_t num_pages,
+ struct ttm_kmap_iter *dst_iter,
+ struct ttm_kmap_iter *src_iter)
{
- int ret;
- void *addr;
-
- *virtual = NULL;
- ret = ttm_mem_io_reserve(bdev, mem);
- if (ret || !mem->bus.is_iomem)
- return ret;
+ const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
+ const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
+ struct ttm_tt *ttm = bo->ttm;
+ struct dma_buf_map src_map, dst_map;
+ pgoff_t i;
- if (mem->bus.addr) {
- addr = mem->bus.addr;
- } else {
- size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
+ /* Single TTM move. NOP */
+ if (dst_ops->maps_tt && src_ops->maps_tt)
+ return;
- if (mem->bus.caching == ttm_write_combined)
- addr = ioremap_wc(mem->bus.offset, bus_size);
-#ifdef CONFIG_X86
- else if (mem->bus.caching == ttm_cached)
- addr = ioremap_cache(mem->bus.offset, bus_size);
-#endif
- else
- addr = ioremap(mem->bus.offset, bus_size);
- if (!addr) {
- ttm_mem_io_free(bdev, mem);
- return -ENOMEM;
+ /* Don't move nonexistent data. Clear destination instead. */
+ if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
+ if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
+ return;
+
+ for (i = 0; i < num_pages; ++i) {
+ dst_ops->map_local(dst_iter, &dst_map, i);
+ if (dst_map.is_iomem)
+ memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
+ else
+ memset(dst_map.vaddr, 0, PAGE_SIZE);
+ if (dst_ops->unmap_local)
+ dst_ops->unmap_local(dst_iter, &dst_map);
}
+ return;
}
- *virtual = addr;
- return 0;
-}
-
-static void ttm_resource_iounmap(struct ttm_device *bdev,
- struct ttm_resource *mem,
- void *virtual)
-{
- if (virtual && mem->bus.addr == NULL)
- iounmap(virtual);
- ttm_mem_io_free(bdev, mem);
-}
-
-static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
-{
- uint32_t *dstP =
- (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
- uint32_t *srcP =
- (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
-
- int i;
- for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
- iowrite32(ioread32(srcP++), dstP++);
- return 0;
-}
-
-static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
- unsigned long page,
- pgprot_t prot)
-{
- struct page *d = ttm->pages[page];
- void *dst;
-
- if (!d)
- return -ENOMEM;
-
- src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
- dst = kmap_atomic_prot(d, prot);
- if (!dst)
- return -ENOMEM;
-
- memcpy_fromio(dst, src, PAGE_SIZE);
-
- kunmap_atomic(dst);
-
- return 0;
-}
-
-static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
- unsigned long page,
- pgprot_t prot)
-{
- struct page *s = ttm->pages[page];
- void *src;
- if (!s)
- return -ENOMEM;
-
- dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
- src = kmap_atomic_prot(s, prot);
- if (!src)
- return -ENOMEM;
-
- memcpy_toio(dst, src, PAGE_SIZE);
-
- kunmap_atomic(src);
+ for (i = 0; i < num_pages; ++i) {
+ dst_ops->map_local(dst_iter, &dst_map, i);
+ src_ops->map_local(src_iter, &src_map, i);
+
+ if (!src_map.is_iomem && !dst_map.is_iomem) {
+ memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
+ } else if (!src_map.is_iomem) {
+ dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
+ PAGE_SIZE);
+ } else if (!dst_map.is_iomem) {
+ memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
+ PAGE_SIZE);
+ } else {
+ int j;
+ u32 __iomem *src = src_map.vaddr_iomem;
+ u32 __iomem *dst = dst_map.vaddr_iomem;
- return 0;
+ for (j = 0; j < (PAGE_SIZE / sizeof(u32)); ++j)
+ iowrite32(ioread32(src++), dst++);
+ }
+ if (src_ops->unmap_local)
+ src_ops->unmap_local(src_iter, &src_map);
+ if (dst_ops->unmap_local)
+ dst_ops->unmap_local(dst_iter, &dst_map);
+ }
}
+EXPORT_SYMBOL(ttm_move_memcpy);
int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
struct ttm_operation_ctx *ctx,
- struct ttm_resource *new_mem)
+ struct ttm_resource *dst_mem)
{
struct ttm_device *bdev = bo->bdev;
- struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
+ struct ttm_resource_manager *dst_man =
+ ttm_manager_type(bo->bdev, dst_mem->mem_type);
struct ttm_tt *ttm = bo->ttm;
- struct ttm_resource *old_mem = &bo->mem;
- struct ttm_resource old_copy = *old_mem;
- void *old_iomap;
- void *new_iomap;
- int ret;
- unsigned long i;
-
- ret = ttm_bo_wait_ctx(bo, ctx);
- if (ret)
- return ret;
-
- ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
- if (ret)
- return ret;
- ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
- if (ret)
- goto out;
-
- /*
- * Single TTM move. NOP.
- */
- if (old_iomap == NULL && new_iomap == NULL)
- goto out2;
-
- /*
- * Don't move nonexistent data. Clear destination instead.
- */
- if (old_iomap == NULL &&
- (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
- !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
- memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
- goto out2;
- }
+ struct ttm_resource *src_mem = &bo->mem;
+ struct ttm_resource_manager *src_man =
+ ttm_manager_type(bdev, src_mem->mem_type);
+ struct ttm_resource src_copy = *src_mem;
+ union {
+ struct ttm_kmap_iter_tt tt;
+ struct ttm_kmap_iter_linear_io io;
+ } _dst_iter, _src_iter;
+ struct ttm_kmap_iter *dst_iter, *src_iter;
+ int ret = 0;
- /*
- * TTM might be null for moves within the same region.
- */
- if (ttm) {
+ if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
+ dst_man->use_tt)) {
ret = ttm_tt_populate(bdev, ttm, ctx);
if (ret)
- goto out1;
+ return ret;
}
- for (i = 0; i < new_mem->num_pages; ++i) {
- if (old_iomap == NULL) {
- pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
- ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
- prot);
- } else if (new_iomap == NULL) {
- pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
- ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
- prot);
- } else {
- ret = ttm_copy_io_page(new_iomap, old_iomap, i);
- }
- if (ret)
- goto out1;
+ dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
+ if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
+ dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
+ if (IS_ERR(dst_iter))
+ return PTR_ERR(dst_iter);
+
+ src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
+ if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
+ src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
+ if (IS_ERR(src_iter)) {
+ ret = PTR_ERR(src_iter);
+ goto out_src_iter;
}
- mb();
-out2:
- old_copy = *old_mem;
- ttm_bo_assign_mem(bo, new_mem);
-
- if (!man->use_tt)
- ttm_bo_tt_destroy(bo);
+ ttm_move_memcpy(bo, dst_mem->num_pages, dst_iter, src_iter);
+ src_copy = *src_mem;
+ ttm_bo_move_sync_cleanup(bo, dst_mem);
-out1:
- ttm_resource_iounmap(bdev, old_mem, new_iomap);
-out:
- ttm_resource_iounmap(bdev, &old_copy, old_iomap);
+ if (!src_iter->ops->maps_tt)
+ ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
+out_src_iter:
+ if (!dst_iter->ops->maps_tt)
+ ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
- /*
- * On error, keep the mm node!
- */
- if (!ret)
- ttm_resource_free(bo, &old_copy);
return ret;
}
EXPORT_SYMBOL(ttm_bo_move_memcpy);
@@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
man = ttm_manager_type(bo->bdev, res->mem_type);
caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
- /* Cached mappings need no adjustment */
- if (caching == ttm_cached)
- return tmp;
-
-#if defined(__i386__) || defined(__x86_64__)
- if (caching == ttm_write_combined)
- tmp = pgprot_writecombine(tmp);
- else if (boot_cpu_data.x86 > 3)
- tmp = pgprot_noncached(tmp);
-#endif
-#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
- defined(__powerpc__) || defined(__mips__)
- if (caching == ttm_write_combined)
- tmp = pgprot_writecombine(tmp);
- else
- tmp = pgprot_noncached(tmp);
-#endif
-#if defined(__sparc__)
- tmp = pgprot_noncached(tmp);
-#endif
- return tmp;
+ return ttm_prot_from_caching(caching, tmp);
}
EXPORT_SYMBOL(ttm_io_prot);
diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
index 56b0efdba1a9..997c458f68a9 100644
--- a/drivers/gpu/drm/ttm/ttm_module.c
+++ b/drivers/gpu/drm/ttm/ttm_module.c
@@ -31,12 +31,47 @@
*/
#include <linux/module.h>
#include <linux/device.h>
+#include <linux/pgtable.h>
#include <linux/sched.h>
#include <linux/debugfs.h>
#include <drm/drm_sysfs.h>
+#include <drm/ttm/ttm_caching.h>
#include "ttm_module.h"
+/**
+ * ttm_prot_from_caching - Modify the page protection according to the
+ * ttm cacing mode
+ * @caching: The ttm caching mode
+ * @tmp: The original page protection
+ *
+ * Return: The modified page protection
+ */
+pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
+{
+ /* Cached mappings need no adjustment */
+ if (caching == ttm_cached)
+ return tmp;
+
+#if defined(__i386__) || defined(__x86_64__)
+ if (caching == ttm_write_combined)
+ tmp = pgprot_writecombine(tmp);
+ else if (boot_cpu_data.x86 > 3)
+ tmp = pgprot_noncached(tmp);
+#endif
+#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
+ defined(__powerpc__) || defined(__mips__)
+ if (caching == ttm_write_combined)
+ tmp = pgprot_writecombine(tmp);
+ else
+ tmp = pgprot_noncached(tmp);
+#endif
+#if defined(__sparc__)
+ tmp = pgprot_noncached(tmp);
+#endif
+ return tmp;
+}
+
struct dentry *ttm_debugfs_root;
static int __init ttm_init(void)
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 59e2b7157e41..931bcd1a9b9b 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -22,6 +22,10 @@
* Authors: Christian König
*/
+#include <linux/dma-buf-map.h>
+#include <linux/io-mapping.h>
+#include <linux/scatterlist.h>
+
#include <drm/ttm/ttm_resource.h>
#include <drm/ttm/ttm_bo_driver.h>
@@ -147,3 +151,192 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
man->func->debug(man, p);
}
EXPORT_SYMBOL(ttm_resource_manager_debug);
+
+static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_iomap *iter_io =
+ container_of(iter, typeof(*iter_io), base);
+ void __iomem *addr;
+
+retry:
+ while (i >= iter_io->cache.end) {
+ iter_io->cache.sg = iter_io->cache.sg ?
+ sg_next(iter_io->cache.sg) : iter_io->st->sgl;
+ iter_io->cache.i = iter_io->cache.end;
+ iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
+ PAGE_SHIFT;
+ iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
+ iter_io->start;
+ }
+
+ if (i < iter_io->cache.i) {
+ iter_io->cache.end = 0;
+ iter_io->cache.sg = NULL;
+ goto retry;
+ }
+
+ addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
+ (((resource_size_t)i - iter_io->cache.i)
+ << PAGE_SHIFT));
+ dma_buf_map_set_vaddr_iomem(dmap, addr);
+}
+
+static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *map)
+{
+ io_mapping_unmap_local(map->vaddr_iomem);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
+ .map_local = ttm_kmap_iter_iomap_map_local,
+ .unmap_local = ttm_kmap_iter_iomap_unmap_local,
+ .maps_tt = false,
+};
+
+/**
+ * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
+ * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
+ * @iomap: The struct io_mapping representing the underlying linear io_memory.
+ * @st: sg_table into @iomap, representing the memory of the struct
+ * ttm_resource.
+ * @start: Offset that needs to be subtracted from @st to make
+ * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
+ *
+ * Return: Pointer to the embedded struct ttm_kmap_iter.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start)
+{
+ iter_io->base.ops = &ttm_kmap_iter_io_ops;
+ iter_io->iomap = iomap;
+ iter_io->st = st;
+ iter_io->start = start;
+ memset(&iter_io->cache, 0, sizeof(iter_io->cache));
+
+ return &iter_io->base;
+}
+EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
+
+/**
+ * DOC: Linear io iterator
+ *
+ * This code should die in the not too near future. Best would be if we could
+ * make io-mapping use memremap for all io memory, and have memremap
+ * implement a kmap_local functionality. We could then strip a huge amount of
+ * code. These linear io iterators are implemented to mimic old functionality,
+ * and they don't use kmap_local semantics at all internally. Rather ioremap or
+ * friends, and at least on 32-bit they add global TLB flushes and points
+ * of failure.
+ */
+
+static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_linear_io *iter_io =
+ container_of(iter, typeof(*iter_io), base);
+
+ *dmap = iter_io->dmap;
+ dma_buf_map_incr(dmap, i * PAGE_SIZE);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
+ .map_local = ttm_kmap_iter_linear_io_map_local,
+ .maps_tt = false,
+};
+
+/**
+ * ttm_kmap_iter_linear_io_init - Initialize an iterator for linear io memory
+ * @iter_io: The iterator to initialize
+ * @bdev: The TTM device
+ * @mem: The ttm resource representing the iomap.
+ *
+ * This function is for internal TTM use only. It sets up a memcpy kmap iterator
+ * pointing at a linear chunk of io memory.
+ *
+ * Return: A pointer to the embedded struct ttm_kmap_iter or error pointer on
+ * failure.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem)
+{
+ int ret;
+
+ ret = ttm_mem_io_reserve(bdev, mem);
+ if (ret)
+ goto out_err;
+ if (!mem->bus.is_iomem) {
+ ret = -EINVAL;
+ goto out_io_free;
+ }
+
+ if (mem->bus.addr) {
+ dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
+ iter_io->needs_unmap = false;
+ } else {
+ size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
+
+ iter_io->needs_unmap = true;
+ memset(&iter_io->dmap, 0, sizeof(iter_io->dmap));
+ if (mem->bus.caching == ttm_write_combined)
+ dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
+ ioremap_wc(mem->bus.offset,
+ bus_size));
+ else if (mem->bus.caching == ttm_cached)
+ dma_buf_map_set_vaddr(&iter_io->dmap,
+ memremap(mem->bus.offset, bus_size,
+ MEMREMAP_WB |
+ MEMREMAP_WT |
+ MEMREMAP_WC));
+
+ /* If uncached requested or if mapping cached or wc failed */
+ if (dma_buf_map_is_null(&iter_io->dmap))
+ dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
+ ioremap(mem->bus.offset,
+ bus_size));
+
+ if (dma_buf_map_is_null(&iter_io->dmap)) {
+ ret = -ENOMEM;
+ goto out_io_free;
+ }
+ }
+
+ iter_io->base.ops = &ttm_kmap_iter_linear_io_ops;
+ return &iter_io->base;
+
+out_io_free:
+ ttm_mem_io_free(bdev, mem);
+out_err:
+ return ERR_PTR(ret);
+}
+
+/**
+ * ttm_kmap_iter_linear_io_fini - Clean up an iterator for linear io memory
+ * @iter_io: The iterator to initialize
+ * @bdev: The TTM device
+ * @mem: The ttm resource representing the iomap.
+ *
+ * This function is for internal TTM use only. It cleans up a memcpy kmap
+ * iterator initialized by ttm_kmap_iter_linear_io_init.
+ */
+void
+ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem)
+{
+ if (iter_io->needs_unmap && dma_buf_map_is_set(&iter_io->dmap)) {
+ if (iter_io->dmap.is_iomem)
+ iounmap(iter_io->dmap.vaddr_iomem);
+ else
+ memunmap(iter_io->dmap.vaddr);
+ }
+
+ ttm_mem_io_free(bdev, mem);
+}
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 539e0232cb3b..0e41227116b1 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -433,3 +433,45 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages)
if (!ttm_dma32_pages_limit)
ttm_dma32_pages_limit = num_dma32_pages;
}
+
+static void ttm_kmap_iter_tt_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_tt *iter_tt =
+ container_of(iter, typeof(*iter_tt), base);
+
+ dma_buf_map_set_vaddr(dmap, kmap_local_page_prot(iter_tt->tt->pages[i],
+ iter_tt->prot));
+}
+
+static void ttm_kmap_iter_tt_unmap_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *map)
+{
+ kunmap_local(map->vaddr);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_tt_ops = {
+ .map_local = ttm_kmap_iter_tt_map_local,
+ .unmap_local = ttm_kmap_iter_tt_unmap_local,
+ .maps_tt = true,
+};
+
+/**
+ * ttm_kmap_iter_tt_init - Initialize a struct ttm_kmap_iter_tt
+ * @iter_tt: The struct ttm_kmap_iter_tt to initialize.
+ * @tt: Struct ttm_tt holding page pointers of the struct ttm_resource.
+ *
+ * Return: Pointer to the embedded struct ttm_kmap_iter.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
+ struct ttm_tt *tt)
+{
+ iter_tt->base.ops = &ttm_kmap_iter_tt_ops;
+ iter_tt->tt = tt;
+ iter_tt->prot = ttm_prot_from_caching(tt->caching, PAGE_KERNEL);
+
+ return &iter_tt->base;
+}
+EXPORT_SYMBOL(ttm_kmap_iter_tt_init);
diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
index dbccac957f8f..fdbeac78c236 100644
--- a/include/drm/ttm/ttm_bo_driver.h
+++ b/include/drm/ttm/ttm_bo_driver.h
@@ -40,6 +40,7 @@
#include <drm/ttm/ttm_device.h>
#include "ttm_bo_api.h"
+#include "ttm_kmap_iter.h"
#include "ttm_placement.h"
#include "ttm_tt.h"
#include "ttm_pool.h"
@@ -272,6 +273,23 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
bool pipeline,
struct ttm_resource *new_mem);
+/**
+ * ttm_bo_move_accel_cleanup.
+ *
+ * @bo: A pointer to a struct ttm_buffer_object.
+ * @new_mem: struct ttm_resource indicating where to move.
+ *
+ * Special case of ttm_bo_move_accel_cleanup where the bo is guaranteed
+ * by the caller to be idle. Typically used after memcpy buffer moves.
+ */
+static inline void ttm_bo_move_sync_cleanup(struct ttm_buffer_object *bo,
+ struct ttm_resource *new_mem)
+{
+ int ret = ttm_bo_move_accel_cleanup(bo, NULL, true, false, new_mem);
+
+ WARN_ON(ret);
+}
+
/**
* ttm_bo_pipeline_gutting.
*
@@ -332,4 +350,14 @@ int ttm_range_man_init(struct ttm_device *bdev,
int ttm_range_man_fini(struct ttm_device *bdev,
unsigned type);
+void ttm_move_memcpy(struct ttm_buffer_object *bo,
+ pgoff_t num_pages,
+ struct ttm_kmap_iter *dst_iter,
+ struct ttm_kmap_iter *src_iter);
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start);
#endif
diff --git a/include/drm/ttm/ttm_caching.h b/include/drm/ttm/ttm_caching.h
index a0b4a49fa432..3c9dd65f5aaf 100644
--- a/include/drm/ttm/ttm_caching.h
+++ b/include/drm/ttm/ttm_caching.h
@@ -33,4 +33,6 @@ enum ttm_caching {
ttm_cached
};
+pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp);
+
#endif
diff --git a/include/drm/ttm/ttm_kmap_iter.h b/include/drm/ttm/ttm_kmap_iter.h
new file mode 100644
index 000000000000..8bb00fd39d6c
--- /dev/null
+++ b/include/drm/ttm/ttm_kmap_iter.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef __TTM_KMAP_ITER_H__
+#define __TTM_KMAP_ITER_H__
+
+#include <linux/types.h>
+
+struct ttm_kmap_iter;
+struct dma_buf_map;
+
+/**
+ * struct ttm_kmap_iter_ops - Ops structure for a struct
+ * ttm_kmap_iter.
+ * @maps_tt: Whether the iterator maps TT memory directly, as opposed
+ * mapping a TT through an aperture. Both these modes have
+ * struct ttm_resource_manager::use_tt set, but the latter typically
+ * returns is_iomem == true from ttm_mem_io_reserve.
+ */
+struct ttm_kmap_iter_ops {
+ /**
+ * kmap_local() - Map a PAGE_SIZE part of the resource using
+ * kmap_local semantics.
+ * @res_iter: Pointer to the struct ttm_kmap_iter representing
+ * the resource.
+ * @dmap: The struct dma_buf_map holding the virtual address after
+ * the operation.
+ * @i: The location within the resource to map. PAGE_SIZE granularity.
+ */
+ void (*map_local)(struct ttm_kmap_iter *res_iter,
+ struct dma_buf_map *dmap, pgoff_t i);
+ /**
+ * unmap_local() - Unmap a PAGE_SIZE part of the resource previously
+ * mapped using kmap_local.
+ * @res_iter: Pointer to the struct ttm_kmap_iter representing
+ * the resource.
+ * @dmap: The struct dma_buf_map holding the virtual address after
+ * the operation.
+ */
+ void (*unmap_local)(struct ttm_kmap_iter *res_iter,
+ struct dma_buf_map *dmap);
+ bool maps_tt;
+};
+
+/**
+ * struct ttm_kmap_iter - Iterator for kmap_local type operations on a
+ * resource.
+ * @ops: Pointer to the operations struct.
+ *
+ * This struct is intended to be embedded in a resource-specific specialization
+ * implementing operations for the resource.
+ *
+ * Nothing stops us from extending the operations to vmap, vmap_pfn etc,
+ * replacing some or parts of the ttm_bo_util. cpu-map functionality.
+ */
+struct ttm_kmap_iter {
+ const struct ttm_kmap_iter_ops *ops;
+};
+
+#endif /* __TTM_KMAP_ITER_H__ */
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index 890b9d369519..b8dc0bdb0da5 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -27,9 +27,11 @@
#include <linux/types.h>
#include <linux/mutex.h>
+#include <linux/dma-buf-map.h>
#include <linux/dma-fence.h>
#include <drm/drm_print.h>
#include <drm/ttm/ttm_caching.h>
+#include <drm/ttm/ttm_kmap_iter.h>
#define TTM_MAX_BO_PRIORITY 4U
@@ -38,6 +40,10 @@ struct ttm_resource_manager;
struct ttm_resource;
struct ttm_place;
struct ttm_buffer_object;
+struct dma_buf_map;
+struct io_mapping;
+struct sg_table;
+struct scatterlist;
struct ttm_resource_manager_func {
/**
@@ -176,6 +182,45 @@ struct ttm_resource {
struct ttm_bus_placement bus;
};
+/**
+ * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
+ * struct sg_table backed struct ttm_resource.
+ * @base: Embedded struct ttm_kmap_iter providing the usage interface.
+ * @iomap: struct io_mapping representing the underlying linear io_memory.
+ * @st: sg_table into @iomap, representing the memory of the struct ttm_resource.
+ * @start: Offset that needs to be subtracted from @st to make
+ * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
+ * @cache: Scatterlist traversal cache for fast lookups.
+ * @cache.sg: Pointer to the currently cached scatterlist segment.
+ * @cache.i: First index of @sg. PAGE_SIZE granularity.
+ * @cache.end: Last index + 1 of @sg. PAGE_SIZE granularity.
+ * @cache.offs: First offset into @iomap of @sg. PAGE_SIZE granularity.
+ */
+struct ttm_kmap_iter_iomap {
+ struct ttm_kmap_iter base;
+ struct io_mapping *iomap;
+ struct sg_table *st;
+ resource_size_t start;
+ struct {
+ struct scatterlist *sg;
+ pgoff_t i;
+ pgoff_t end;
+ pgoff_t offs;
+ } cache;
+};
+
+/**
+ * struct ttm_kmap_iter_linear_io - Iterator specialization for linear io
+ * @base: The base iterator
+ * @dmap: Points to the starting address of the region
+ * @needs_unmap: Whether we need to unmap on fini
+ */
+struct ttm_kmap_iter_linear_io {
+ struct ttm_kmap_iter base;
+ struct dma_buf_map dmap;
+ bool needs_unmap;
+};
+
/**
* ttm_resource_manager_set_used
*
@@ -237,4 +282,20 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
void ttm_resource_manager_debug(struct ttm_resource_manager *man,
struct drm_printer *p);
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start);
+
+struct ttm_kmap_iter_linear_io;
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem);
+
+void ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem);
#endif
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 134d09ef7766..3102059db726 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -29,6 +29,7 @@
#include <linux/types.h>
#include <drm/ttm/ttm_caching.h>
+#include <drm/ttm/ttm_kmap_iter.h>
struct ttm_bo_device;
struct ttm_tt;
@@ -69,6 +70,18 @@ struct ttm_tt {
enum ttm_caching caching;
};
+/**
+ * struct ttm_kmap_iter_tt - Specialization of a mappig iterator for a tt.
+ * @base: Embedded struct ttm_kmap_iter providing the usage interface
+ * @tt: Cached struct ttm_tt.
+ * @prot: Cached page protection for mapping.
+ */
+struct ttm_kmap_iter_tt {
+ struct ttm_kmap_iter base;
+ struct ttm_tt *tt;
+ pgprot_t prot;
+};
+
static inline bool ttm_tt_is_populated(struct ttm_tt *tt)
{
return tt->page_flags & TTM_PAGE_FLAG_PRIV_POPULATED;
@@ -159,6 +172,9 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
+struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
+ struct ttm_tt *tt);
+
#if IS_ENABLED(CONFIG_AGP)
#include <linux/agp_backend.h>
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 07/15] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
Cc: Thomas Hellström, Matthew Auld, Christian König, Daniel Vetter
Memcpy from wc will be used as well by TTM memcpy.
Move it to core drm, and make the interface do the right thing
even on !X86.
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Dave Airlie <airlied@gmail.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v4:
- Fix !X86 path (Reported by Matthew Auld)
---
drivers/gpu/drm/Makefile | 2 +-
drivers/gpu/drm/drm_drv.c | 2 +
.../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 ++++++++++++-----
drivers/gpu/drm/i915/Makefile | 1 -
.../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 5 +-
drivers/gpu/drm/i915/gt/selftest_reset.c | 7 +-
drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 +--
drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
drivers/gpu/drm/i915/i915_drv.c | 2 -
drivers/gpu/drm/i915/i915_gpu_error.c | 8 +--
drivers/gpu/drm/i915/i915_memcpy.h | 34 ----------
.../drm/i915/selftests/intel_memory_region.c | 7 +-
include/drm/drm_memcpy.h | 68 +++++++++++++++++++
14 files changed, 142 insertions(+), 76 deletions(-)
rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
create mode 100644 include/drm/drm_memcpy.h
diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
index a91cc7684904..f3ab8586c3d7 100644
--- a/drivers/gpu/drm/Makefile
+++ b/drivers/gpu/drm/Makefile
@@ -18,7 +18,7 @@ drm-y := drm_aperture.o drm_auth.o drm_cache.o \
drm_dumb_buffers.o drm_mode_config.o drm_vblank.o \
drm_syncobj.o drm_lease.o drm_writeback.o drm_client.o \
drm_client_modeset.o drm_atomic_uapi.o drm_hdcp.o \
- drm_managed.o drm_vblank_work.o
+ drm_managed.o drm_vblank_work.o drm_memcpy.o \
drm-$(CONFIG_DRM_LEGACY) += drm_agpsupport.o drm_bufs.o drm_context.o drm_dma.o \
drm_legacy_misc.o drm_lock.o drm_memory.o drm_scatter.o \
diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index 3d8d68a98b95..351cc2900cf1 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -40,6 +40,7 @@
#include <drm/drm_drv.h>
#include <drm/drm_file.h>
#include <drm/drm_managed.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_mode_object.h>
#include <drm/drm_print.h>
@@ -1041,6 +1042,7 @@ static int __init drm_core_init(void)
drm_connector_ida_init();
idr_init(&drm_minors_idr);
+ drm_memcpy_init_early();
ret = drm_sysfs_init();
if (ret < 0) {
diff --git a/drivers/gpu/drm/i915/i915_memcpy.c b/drivers/gpu/drm/drm_memcpy.c
similarity index 70%
rename from drivers/gpu/drm/i915/i915_memcpy.c
rename to drivers/gpu/drm/drm_memcpy.c
index 1b021a4902de..740377749caa 100644
--- a/drivers/gpu/drm/i915/i915_memcpy.c
+++ b/drivers/gpu/drm/drm_memcpy.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: MIT
/*
* Copyright © 2016 Intel Corporation
*
@@ -22,16 +23,12 @@
*
*/
+#ifdef CONFIG_X86
+#include <linux/dma-buf-map.h>
#include <linux/kernel.h>
#include <asm/fpu/api.h>
-#include "i915_memcpy.h"
-
-#if IS_ENABLED(CONFIG_DRM_I915_DEBUG)
-#define CI_BUG_ON(expr) BUG_ON(expr)
-#else
-#define CI_BUG_ON(expr) BUILD_BUG_ON_INVALID(expr)
-#endif
+#include "drm/drm_memcpy.h"
static DEFINE_STATIC_KEY_FALSE(has_movntdqa);
@@ -94,23 +91,24 @@ static void __memcpy_ntdqu(void *dst, const void *src, unsigned long len)
}
/**
- * i915_memcpy_from_wc: perform an accelerated *aligned* read from WC
+ * drm_memcpy_from_wc: perform an accelerated *aligned* read from WC
* @dst: destination pointer
* @src: source pointer
* @len: how many bytes to copy
*
- * i915_memcpy_from_wc copies @len bytes from @src to @dst using
+ * drm_memcpy_from_wc copies @len bytes from @src to @dst using
* non-temporal instructions where available. Note that all arguments
* (@src, @dst) must be aligned to 16 bytes and @len must be a multiple
* of 16.
*
* To test whether accelerated reads from WC are supported, use
- * i915_memcpy_from_wc(NULL, NULL, 0);
+ * drm_memcpy_from_wc(NULL, NULL, 0);
+ * This interface is intended for memremapped memory without the __iomem tag.
*
* Returns true if the copy was successful, false if the preconditions
* are not met.
*/
-bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
{
if (unlikely(((unsigned long)dst | (unsigned long)src | len) & 15))
return false;
@@ -123,24 +121,53 @@ bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
return false;
}
+EXPORT_SYMBOL(drm_memcpy_from_wc);
/**
- * i915_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
+ * drm_memcpy_from_wc_dbm: perform an accelerated *aligned* read from WC with
+ * struct dma_buf_map arguments.
+ * @dst: destination map
+ * @src: source map
+ * @len: how many bytes to copy
+ *
+ * This is identical to drm_memcpy_from_wc, except it's intended for
+ * potentially ioremapped memory rather than memremapped memory.
+ *
+ * Returns true if the copy was successful, false if the preconditions
+ * are not met.
+ */
+bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
+ const struct dma_buf_map *src,
+ unsigned long len)
+{
+ /* For X86 we can safely drop __iomem */
+ return drm_memcpy_from_wc(dst->is_iomem ?
+ (void __force *)dst->vaddr_iomem :
+ dst->vaddr,
+ src->is_iomem ?
+ (void const __force *)src->vaddr_iomem :
+ src->vaddr,
+ len);
+}
+EXPORT_SYMBOL(drm_memcpy_from_wc_dbm);
+
+/**
+ * drm_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
* @dst: destination pointer
* @src: source pointer
* @len: how many bytes to copy
*
- * Like i915_memcpy_from_wc(), the unaligned variant copies @len bytes from
+ * Like drm_memcpy_from_wc(), the unaligned variant copies @len bytes from
* @src to @dst using * non-temporal instructions where available, but
* accepts that its arguments may not be aligned, but are valid for the
* potential 16-byte read past the end.
+ *
+ * This interface is intended for mremapped memory without the __iomem tag.
*/
-void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
{
unsigned long addr;
- CI_BUG_ON(!i915_has_memcpy_from_wc());
-
addr = (unsigned long)src;
if (!IS_ALIGNED(addr, 16)) {
unsigned long x = min(ALIGN(addr, 16) - addr, len);
@@ -155,8 +182,9 @@ void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len
if (likely(len))
__memcpy_ntdqu(dst, src, DIV_ROUND_UP(len, 16));
}
+EXPORT_SYMBOL(drm_unaligned_memcpy_from_wc);
-void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
+void drm_memcpy_init_early(void)
{
/*
* Some hypervisors (e.g. KVM) don't support VEX-prefix instructions
@@ -166,3 +194,4 @@ void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
!boot_cpu_has(X86_FEATURE_HYPERVISOR))
static_branch_enable(&has_movntdqa);
}
+#endif
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 4f22cac1c49b..ebc19bd5fff4 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -61,7 +61,6 @@ i915-y += i915_drv.o \
# core library code
i915-y += \
dma_resv_utils.o \
- i915_memcpy.o \
i915_mm.o \
i915_sw_fence.o \
i915_sw_fence_work.o \
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index 297143511f99..77285e421fb8 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -10,6 +10,7 @@
#include <linux/uaccess.h>
#include <drm/drm_syncobj.h>
+#include <drm/drm_memcpy.h>
#include "display/intel_frontbuffer.h"
@@ -28,7 +29,6 @@
#include "i915_sw_fence_work.h"
#include "i915_trace.h"
#include "i915_user_extensions.h"
-#include "i915_memcpy.h"
struct eb_vma {
struct i915_vma *vma;
@@ -2503,7 +2503,7 @@ static int eb_parse_pipeline(struct i915_execbuffer *eb,
!(batch->cache_coherent & I915_BO_CACHE_COHERENT_FOR_READ);
pw->batch_map = ERR_PTR(-ENODEV);
- if (needs_clflush && i915_has_memcpy_from_wc())
+ if (needs_clflush && drm_has_memcpy_from_wc())
pw->batch_map = i915_gem_object_pin_map(batch, I915_MAP_WC);
if (IS_ERR(pw->batch_map)) {
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 5706d471692d..e9247afb0320 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -24,6 +24,8 @@
#include <linux/sched/mm.h>
+#include <drm/drm_memcpy.h>
+
#include "display/intel_frontbuffer.h"
#include "i915_drv.h"
#include "i915_gem_clflush.h"
@@ -31,7 +33,6 @@
#include "i915_gem_mman.h"
#include "i915_gem_object.h"
#include "i915_globals.h"
-#include "i915_memcpy.h"
#include "i915_trace.h"
static struct i915_global_object {
@@ -374,7 +375,7 @@ i915_gem_object_read_from_page_iomap(struct drm_i915_gem_object *obj, u64 offset
PAGE_SIZE);
src_ptr = src_map + offset_in_page(offset);
- if (!i915_memcpy_from_wc(dst, (void __force *)src_ptr, size))
+ if (!drm_memcpy_from_wc(dst, (void __force *)src_ptr, size))
memcpy_fromio(dst, src_ptr, size);
io_mapping_unmap(src_map);
diff --git a/drivers/gpu/drm/i915/gt/selftest_reset.c b/drivers/gpu/drm/i915/gt/selftest_reset.c
index 8784257ec808..92ada67a3835 100644
--- a/drivers/gpu/drm/i915/gt/selftest_reset.c
+++ b/drivers/gpu/drm/i915/gt/selftest_reset.c
@@ -5,9 +5,10 @@
#include <linux/crc32.h>
+#include <drm/drm_memcpy.h>
+
#include "gem/i915_gem_stolen.h"
-#include "i915_memcpy.h"
#include "i915_selftest.h"
#include "intel_gpu_commands.h"
#include "selftests/igt_reset.h"
@@ -99,7 +100,7 @@ __igt_reset_stolen(struct intel_gt *gt,
memset_io(s, STACK_MAGIC, PAGE_SIZE);
in = (void __force *)s;
- if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
+ if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
in = tmp;
crc[page] = crc32_le(0, in, PAGE_SIZE);
@@ -135,7 +136,7 @@ __igt_reset_stolen(struct intel_gt *gt,
PAGE_SIZE);
in = (void __force *)s;
- if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
+ if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
in = tmp;
x = crc32_le(0, in, PAGE_SIZE);
diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
index c36d5eb5bbb9..f045e42be6ca 100644
--- a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
+++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
@@ -5,9 +5,10 @@
#include <linux/debugfs.h>
+#include <drm/drm_memcpy.h>
+
#include "gt/intel_gt.h"
#include "i915_drv.h"
-#include "i915_memcpy.h"
#include "intel_guc_log.h"
static void guc_log_capture_logs(struct intel_guc_log *log);
@@ -295,13 +296,13 @@ static void guc_read_update_log_buffer(struct intel_guc_log *log)
/* Just copy the newly written data */
if (read_offset > write_offset) {
- i915_memcpy_from_wc(dst_data, src_data, write_offset);
+ drm_memcpy_from_wc(dst_data, src_data, write_offset);
bytes_to_copy = buffer_size - read_offset;
} else {
bytes_to_copy = write_offset - read_offset;
}
- i915_memcpy_from_wc(dst_data + read_offset,
- src_data + read_offset, bytes_to_copy);
+ drm_memcpy_from_wc(dst_data + read_offset,
+ src_data + read_offset, bytes_to_copy);
src_data += buffer_size;
dst_data += buffer_size;
@@ -569,7 +570,7 @@ int intel_guc_log_relay_open(struct intel_guc_log *log)
* it should be present on the chipsets supporting GuC based
* submisssions.
*/
- if (!i915_has_memcpy_from_wc()) {
+ if (!drm_has_memcpy_from_wc()) {
ret = -ENXIO;
goto out_unlock;
}
diff --git a/drivers/gpu/drm/i915/i915_cmd_parser.c b/drivers/gpu/drm/i915/i915_cmd_parser.c
index 5b4b2bd46e7c..98653f1a2b1d 100644
--- a/drivers/gpu/drm/i915/i915_cmd_parser.c
+++ b/drivers/gpu/drm/i915/i915_cmd_parser.c
@@ -24,12 +24,12 @@
* Brad Volkin <bradley.d.volkin@intel.com>
*
*/
+#include <drm/drm_memcpy.h>
#include "gt/intel_engine.h"
#include "gt/intel_gpu_commands.h"
#include "i915_drv.h"
-#include "i915_memcpy.h"
/**
* DOC: batch buffer command parser
@@ -1152,7 +1152,7 @@ static u32 *copy_batch(struct drm_i915_gem_object *dst_obj,
if (src) {
GEM_BUG_ON(!needs_clflush);
- i915_unaligned_memcpy_from_wc(dst, src + offset, length);
+ drm_unaligned_memcpy_from_wc(dst, src + offset, length);
} else {
struct scatterlist *sg;
void *ptr;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 30c349137be2..68639ed0bdec 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -72,7 +72,6 @@
#include "i915_drv.h"
#include "i915_ioc32.h"
#include "i915_irq.h"
-#include "i915_memcpy.h"
#include "i915_perf.h"
#include "i915_query.h"
#include "i915_suspend.h"
@@ -325,7 +324,6 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
mutex_init(&dev_priv->pps_mutex);
mutex_init(&dev_priv->hdcp_comp_mutex);
- i915_memcpy_init_early(dev_priv);
intel_runtime_pm_init_early(&dev_priv->runtime_pm);
ret = i915_workqueues_init(dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index 8b964e355cb5..7c1b44545bab 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -34,6 +34,7 @@
#include <linux/utsname.h>
#include <linux/zlib.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_print.h>
#include "display/intel_dmc.h"
@@ -46,7 +47,6 @@
#include "i915_drv.h"
#include "i915_gpu_error.h"
-#include "i915_memcpy.h"
#include "i915_scatterlist.h"
#define ALLOW_FAIL (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN)
@@ -255,7 +255,7 @@ static bool compress_init(struct i915_vma_compress *c)
}
c->tmp = NULL;
- if (i915_has_memcpy_from_wc())
+ if (drm_has_memcpy_from_wc())
c->tmp = pool_alloc(&c->pool, ALLOW_FAIL);
return true;
@@ -295,7 +295,7 @@ static int compress_page(struct i915_vma_compress *c,
struct z_stream_s *zstream = &c->zstream;
zstream->next_in = src;
- if (wc && c->tmp && i915_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
+ if (wc && c->tmp && drm_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
zstream->next_in = c->tmp;
zstream->avail_in = PAGE_SIZE;
@@ -395,7 +395,7 @@ static int compress_page(struct i915_vma_compress *c,
if (!ptr)
return -ENOMEM;
- if (!(wc && i915_memcpy_from_wc(ptr, src, PAGE_SIZE)))
+ if (!(wc && drm_memcpy_from_wc(ptr, src, PAGE_SIZE)))
memcpy(ptr, src, PAGE_SIZE);
dst->pages[dst->page_count++] = ptr;
cond_resched();
diff --git a/drivers/gpu/drm/i915/i915_memcpy.h b/drivers/gpu/drm/i915/i915_memcpy.h
deleted file mode 100644
index 3df063a3293b..000000000000
--- a/drivers/gpu/drm/i915/i915_memcpy.h
+++ /dev/null
@@ -1,34 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#ifndef __I915_MEMCPY_H__
-#define __I915_MEMCPY_H__
-
-#include <linux/types.h>
-
-struct drm_i915_private;
-
-void i915_memcpy_init_early(struct drm_i915_private *i915);
-
-bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len);
-void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
-
-/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
- * as well as SSE4.1 support. i915_memcpy_from_wc() will report if it cannot
- * perform the operation. To check beforehand, pass in the parameters to
- * to i915_can_memcpy_from_wc() - since we only care about the low 4 bits,
- * you only need to pass in the minor offsets, page-aligned pointers are
- * always valid.
- *
- * For just checking for SSE4.1, in the foreknowledge that the future use
- * will be correctly aligned, just use i915_has_memcpy_from_wc().
- */
-#define i915_can_memcpy_from_wc(dst, src, len) \
- i915_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
-
-#define i915_has_memcpy_from_wc() \
- i915_memcpy_from_wc(NULL, NULL, 0)
-
-#endif /* __I915_MEMCPY_H__ */
diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
index c85d516b85cd..6bb399e9be78 100644
--- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
@@ -6,6 +6,8 @@
#include <linux/prime_numbers.h>
#include <linux/sort.h>
+#include <drm/drm_memcpy.h>
+
#include "../i915_selftest.h"
#include "mock_drm.h"
@@ -20,7 +22,6 @@
#include "gem/selftests/mock_context.h"
#include "gt/intel_engine_user.h"
#include "gt/intel_gt.h"
-#include "i915_memcpy.h"
#include "selftests/igt_flush_test.h"
#include "selftests/i915_random.h"
@@ -901,7 +902,7 @@ static inline void igt_memcpy(void *dst, const void *src, size_t size)
static inline void igt_memcpy_from_wc(void *dst, const void *src, size_t size)
{
- i915_memcpy_from_wc(dst, src, size);
+ drm_memcpy_from_wc(dst, src, size);
}
static int _perf_memcpy(struct intel_memory_region *src_mr,
@@ -925,7 +926,7 @@ static int _perf_memcpy(struct intel_memory_region *src_mr,
{
"memcpy_from_wc",
igt_memcpy_from_wc,
- !i915_has_memcpy_from_wc(),
+ !drm_has_memcpy_from_wc(),
},
};
struct drm_i915_gem_object *src, *dst;
diff --git a/include/drm/drm_memcpy.h b/include/drm/drm_memcpy.h
new file mode 100644
index 000000000000..fe5ed1e89ce6
--- /dev/null
+++ b/include/drm/drm_memcpy.h
@@ -0,0 +1,68 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __DRM_MEMCPY_H__
+#define __DRM_MEMCPY_H__
+
+#include <linux/types.h>
+
+struct dma_buf_map;
+
+#ifdef CONFIG_X86
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len);
+bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
+ const struct dma_buf_map *src,
+ unsigned long len);
+void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
+
+/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
+ * as well as SSE4.1 support. drm_memcpy_from_wc() will report if it cannot
+ * perform the operation. To check beforehand, pass in the parameters to
+ * drm_can_memcpy_from_wc() - since we only care about the low 4 bits,
+ * you only need to pass in the minor offsets, page-aligned pointers are
+ * always valid.
+ *
+ * For just checking for SSE4.1, in the foreknowledge that the future use
+ * will be correctly aligned, just use drm_has_memcpy_from_wc().
+ */
+#define drm_can_memcpy_from_wc(dst, src, len) \
+ drm_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
+
+#define drm_has_memcpy_from_wc() \
+ drm_memcpy_from_wc(NULL, NULL, 0)
+
+void drm_memcpy_init_early(void);
+
+#else
+
+static inline
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+{
+ return false;
+}
+
+static inline
+bool drm_memcpy_from_wc_dbm(void *dst, const void *src, unsigned long len)
+{
+ return false;
+}
+
+static inline
+bool drm_can_memcpy_from_wc_dbm(void *dst, const void *src, unsigned long len)
+{
+ return false;
+}
+
+static inline
+bool drm_has_memcpy_from_wc(void)
+{
+ return false;
+}
+
+#define drm_has_memcpy_from_wc() (false)
+#define drm_unaligned_memcpy_from_wc(_dst, _src, _len) WARN_ON(1)
+#define drm_memcpy_init_early() do {} while (0)
+#endif /* CONFIG_X86 */
+#endif /* __DRM_MEMCPY_H__ */
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 07/15] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
Cc: Thomas Hellström, Matthew Auld, Christian König, Daniel Vetter
Memcpy from wc will be used as well by TTM memcpy.
Move it to core drm, and make the interface do the right thing
even on !X86.
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Dave Airlie <airlied@gmail.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v4:
- Fix !X86 path (Reported by Matthew Auld)
---
drivers/gpu/drm/Makefile | 2 +-
drivers/gpu/drm/drm_drv.c | 2 +
.../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 ++++++++++++-----
drivers/gpu/drm/i915/Makefile | 1 -
.../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 5 +-
drivers/gpu/drm/i915/gt/selftest_reset.c | 7 +-
drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 +--
drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
drivers/gpu/drm/i915/i915_drv.c | 2 -
drivers/gpu/drm/i915/i915_gpu_error.c | 8 +--
drivers/gpu/drm/i915/i915_memcpy.h | 34 ----------
.../drm/i915/selftests/intel_memory_region.c | 7 +-
include/drm/drm_memcpy.h | 68 +++++++++++++++++++
14 files changed, 142 insertions(+), 76 deletions(-)
rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
create mode 100644 include/drm/drm_memcpy.h
diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
index a91cc7684904..f3ab8586c3d7 100644
--- a/drivers/gpu/drm/Makefile
+++ b/drivers/gpu/drm/Makefile
@@ -18,7 +18,7 @@ drm-y := drm_aperture.o drm_auth.o drm_cache.o \
drm_dumb_buffers.o drm_mode_config.o drm_vblank.o \
drm_syncobj.o drm_lease.o drm_writeback.o drm_client.o \
drm_client_modeset.o drm_atomic_uapi.o drm_hdcp.o \
- drm_managed.o drm_vblank_work.o
+ drm_managed.o drm_vblank_work.o drm_memcpy.o \
drm-$(CONFIG_DRM_LEGACY) += drm_agpsupport.o drm_bufs.o drm_context.o drm_dma.o \
drm_legacy_misc.o drm_lock.o drm_memory.o drm_scatter.o \
diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index 3d8d68a98b95..351cc2900cf1 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -40,6 +40,7 @@
#include <drm/drm_drv.h>
#include <drm/drm_file.h>
#include <drm/drm_managed.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_mode_object.h>
#include <drm/drm_print.h>
@@ -1041,6 +1042,7 @@ static int __init drm_core_init(void)
drm_connector_ida_init();
idr_init(&drm_minors_idr);
+ drm_memcpy_init_early();
ret = drm_sysfs_init();
if (ret < 0) {
diff --git a/drivers/gpu/drm/i915/i915_memcpy.c b/drivers/gpu/drm/drm_memcpy.c
similarity index 70%
rename from drivers/gpu/drm/i915/i915_memcpy.c
rename to drivers/gpu/drm/drm_memcpy.c
index 1b021a4902de..740377749caa 100644
--- a/drivers/gpu/drm/i915/i915_memcpy.c
+++ b/drivers/gpu/drm/drm_memcpy.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: MIT
/*
* Copyright © 2016 Intel Corporation
*
@@ -22,16 +23,12 @@
*
*/
+#ifdef CONFIG_X86
+#include <linux/dma-buf-map.h>
#include <linux/kernel.h>
#include <asm/fpu/api.h>
-#include "i915_memcpy.h"
-
-#if IS_ENABLED(CONFIG_DRM_I915_DEBUG)
-#define CI_BUG_ON(expr) BUG_ON(expr)
-#else
-#define CI_BUG_ON(expr) BUILD_BUG_ON_INVALID(expr)
-#endif
+#include "drm/drm_memcpy.h"
static DEFINE_STATIC_KEY_FALSE(has_movntdqa);
@@ -94,23 +91,24 @@ static void __memcpy_ntdqu(void *dst, const void *src, unsigned long len)
}
/**
- * i915_memcpy_from_wc: perform an accelerated *aligned* read from WC
+ * drm_memcpy_from_wc: perform an accelerated *aligned* read from WC
* @dst: destination pointer
* @src: source pointer
* @len: how many bytes to copy
*
- * i915_memcpy_from_wc copies @len bytes from @src to @dst using
+ * drm_memcpy_from_wc copies @len bytes from @src to @dst using
* non-temporal instructions where available. Note that all arguments
* (@src, @dst) must be aligned to 16 bytes and @len must be a multiple
* of 16.
*
* To test whether accelerated reads from WC are supported, use
- * i915_memcpy_from_wc(NULL, NULL, 0);
+ * drm_memcpy_from_wc(NULL, NULL, 0);
+ * This interface is intended for memremapped memory without the __iomem tag.
*
* Returns true if the copy was successful, false if the preconditions
* are not met.
*/
-bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
{
if (unlikely(((unsigned long)dst | (unsigned long)src | len) & 15))
return false;
@@ -123,24 +121,53 @@ bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
return false;
}
+EXPORT_SYMBOL(drm_memcpy_from_wc);
/**
- * i915_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
+ * drm_memcpy_from_wc_dbm: perform an accelerated *aligned* read from WC with
+ * struct dma_buf_map arguments.
+ * @dst: destination map
+ * @src: source map
+ * @len: how many bytes to copy
+ *
+ * This is identical to drm_memcpy_from_wc, except it's intended for
+ * potentially ioremapped memory rather than memremapped memory.
+ *
+ * Returns true if the copy was successful, false if the preconditions
+ * are not met.
+ */
+bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
+ const struct dma_buf_map *src,
+ unsigned long len)
+{
+ /* For X86 we can safely drop __iomem */
+ return drm_memcpy_from_wc(dst->is_iomem ?
+ (void __force *)dst->vaddr_iomem :
+ dst->vaddr,
+ src->is_iomem ?
+ (void const __force *)src->vaddr_iomem :
+ src->vaddr,
+ len);
+}
+EXPORT_SYMBOL(drm_memcpy_from_wc_dbm);
+
+/**
+ * drm_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
* @dst: destination pointer
* @src: source pointer
* @len: how many bytes to copy
*
- * Like i915_memcpy_from_wc(), the unaligned variant copies @len bytes from
+ * Like drm_memcpy_from_wc(), the unaligned variant copies @len bytes from
* @src to @dst using * non-temporal instructions where available, but
* accepts that its arguments may not be aligned, but are valid for the
* potential 16-byte read past the end.
+ *
+ * This interface is intended for mremapped memory without the __iomem tag.
*/
-void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
{
unsigned long addr;
- CI_BUG_ON(!i915_has_memcpy_from_wc());
-
addr = (unsigned long)src;
if (!IS_ALIGNED(addr, 16)) {
unsigned long x = min(ALIGN(addr, 16) - addr, len);
@@ -155,8 +182,9 @@ void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len
if (likely(len))
__memcpy_ntdqu(dst, src, DIV_ROUND_UP(len, 16));
}
+EXPORT_SYMBOL(drm_unaligned_memcpy_from_wc);
-void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
+void drm_memcpy_init_early(void)
{
/*
* Some hypervisors (e.g. KVM) don't support VEX-prefix instructions
@@ -166,3 +194,4 @@ void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
!boot_cpu_has(X86_FEATURE_HYPERVISOR))
static_branch_enable(&has_movntdqa);
}
+#endif
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 4f22cac1c49b..ebc19bd5fff4 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -61,7 +61,6 @@ i915-y += i915_drv.o \
# core library code
i915-y += \
dma_resv_utils.o \
- i915_memcpy.o \
i915_mm.o \
i915_sw_fence.o \
i915_sw_fence_work.o \
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index 297143511f99..77285e421fb8 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -10,6 +10,7 @@
#include <linux/uaccess.h>
#include <drm/drm_syncobj.h>
+#include <drm/drm_memcpy.h>
#include "display/intel_frontbuffer.h"
@@ -28,7 +29,6 @@
#include "i915_sw_fence_work.h"
#include "i915_trace.h"
#include "i915_user_extensions.h"
-#include "i915_memcpy.h"
struct eb_vma {
struct i915_vma *vma;
@@ -2503,7 +2503,7 @@ static int eb_parse_pipeline(struct i915_execbuffer *eb,
!(batch->cache_coherent & I915_BO_CACHE_COHERENT_FOR_READ);
pw->batch_map = ERR_PTR(-ENODEV);
- if (needs_clflush && i915_has_memcpy_from_wc())
+ if (needs_clflush && drm_has_memcpy_from_wc())
pw->batch_map = i915_gem_object_pin_map(batch, I915_MAP_WC);
if (IS_ERR(pw->batch_map)) {
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 5706d471692d..e9247afb0320 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -24,6 +24,8 @@
#include <linux/sched/mm.h>
+#include <drm/drm_memcpy.h>
+
#include "display/intel_frontbuffer.h"
#include "i915_drv.h"
#include "i915_gem_clflush.h"
@@ -31,7 +33,6 @@
#include "i915_gem_mman.h"
#include "i915_gem_object.h"
#include "i915_globals.h"
-#include "i915_memcpy.h"
#include "i915_trace.h"
static struct i915_global_object {
@@ -374,7 +375,7 @@ i915_gem_object_read_from_page_iomap(struct drm_i915_gem_object *obj, u64 offset
PAGE_SIZE);
src_ptr = src_map + offset_in_page(offset);
- if (!i915_memcpy_from_wc(dst, (void __force *)src_ptr, size))
+ if (!drm_memcpy_from_wc(dst, (void __force *)src_ptr, size))
memcpy_fromio(dst, src_ptr, size);
io_mapping_unmap(src_map);
diff --git a/drivers/gpu/drm/i915/gt/selftest_reset.c b/drivers/gpu/drm/i915/gt/selftest_reset.c
index 8784257ec808..92ada67a3835 100644
--- a/drivers/gpu/drm/i915/gt/selftest_reset.c
+++ b/drivers/gpu/drm/i915/gt/selftest_reset.c
@@ -5,9 +5,10 @@
#include <linux/crc32.h>
+#include <drm/drm_memcpy.h>
+
#include "gem/i915_gem_stolen.h"
-#include "i915_memcpy.h"
#include "i915_selftest.h"
#include "intel_gpu_commands.h"
#include "selftests/igt_reset.h"
@@ -99,7 +100,7 @@ __igt_reset_stolen(struct intel_gt *gt,
memset_io(s, STACK_MAGIC, PAGE_SIZE);
in = (void __force *)s;
- if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
+ if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
in = tmp;
crc[page] = crc32_le(0, in, PAGE_SIZE);
@@ -135,7 +136,7 @@ __igt_reset_stolen(struct intel_gt *gt,
PAGE_SIZE);
in = (void __force *)s;
- if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
+ if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
in = tmp;
x = crc32_le(0, in, PAGE_SIZE);
diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
index c36d5eb5bbb9..f045e42be6ca 100644
--- a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
+++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
@@ -5,9 +5,10 @@
#include <linux/debugfs.h>
+#include <drm/drm_memcpy.h>
+
#include "gt/intel_gt.h"
#include "i915_drv.h"
-#include "i915_memcpy.h"
#include "intel_guc_log.h"
static void guc_log_capture_logs(struct intel_guc_log *log);
@@ -295,13 +296,13 @@ static void guc_read_update_log_buffer(struct intel_guc_log *log)
/* Just copy the newly written data */
if (read_offset > write_offset) {
- i915_memcpy_from_wc(dst_data, src_data, write_offset);
+ drm_memcpy_from_wc(dst_data, src_data, write_offset);
bytes_to_copy = buffer_size - read_offset;
} else {
bytes_to_copy = write_offset - read_offset;
}
- i915_memcpy_from_wc(dst_data + read_offset,
- src_data + read_offset, bytes_to_copy);
+ drm_memcpy_from_wc(dst_data + read_offset,
+ src_data + read_offset, bytes_to_copy);
src_data += buffer_size;
dst_data += buffer_size;
@@ -569,7 +570,7 @@ int intel_guc_log_relay_open(struct intel_guc_log *log)
* it should be present on the chipsets supporting GuC based
* submisssions.
*/
- if (!i915_has_memcpy_from_wc()) {
+ if (!drm_has_memcpy_from_wc()) {
ret = -ENXIO;
goto out_unlock;
}
diff --git a/drivers/gpu/drm/i915/i915_cmd_parser.c b/drivers/gpu/drm/i915/i915_cmd_parser.c
index 5b4b2bd46e7c..98653f1a2b1d 100644
--- a/drivers/gpu/drm/i915/i915_cmd_parser.c
+++ b/drivers/gpu/drm/i915/i915_cmd_parser.c
@@ -24,12 +24,12 @@
* Brad Volkin <bradley.d.volkin@intel.com>
*
*/
+#include <drm/drm_memcpy.h>
#include "gt/intel_engine.h"
#include "gt/intel_gpu_commands.h"
#include "i915_drv.h"
-#include "i915_memcpy.h"
/**
* DOC: batch buffer command parser
@@ -1152,7 +1152,7 @@ static u32 *copy_batch(struct drm_i915_gem_object *dst_obj,
if (src) {
GEM_BUG_ON(!needs_clflush);
- i915_unaligned_memcpy_from_wc(dst, src + offset, length);
+ drm_unaligned_memcpy_from_wc(dst, src + offset, length);
} else {
struct scatterlist *sg;
void *ptr;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 30c349137be2..68639ed0bdec 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -72,7 +72,6 @@
#include "i915_drv.h"
#include "i915_ioc32.h"
#include "i915_irq.h"
-#include "i915_memcpy.h"
#include "i915_perf.h"
#include "i915_query.h"
#include "i915_suspend.h"
@@ -325,7 +324,6 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
mutex_init(&dev_priv->pps_mutex);
mutex_init(&dev_priv->hdcp_comp_mutex);
- i915_memcpy_init_early(dev_priv);
intel_runtime_pm_init_early(&dev_priv->runtime_pm);
ret = i915_workqueues_init(dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index 8b964e355cb5..7c1b44545bab 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -34,6 +34,7 @@
#include <linux/utsname.h>
#include <linux/zlib.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_print.h>
#include "display/intel_dmc.h"
@@ -46,7 +47,6 @@
#include "i915_drv.h"
#include "i915_gpu_error.h"
-#include "i915_memcpy.h"
#include "i915_scatterlist.h"
#define ALLOW_FAIL (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN)
@@ -255,7 +255,7 @@ static bool compress_init(struct i915_vma_compress *c)
}
c->tmp = NULL;
- if (i915_has_memcpy_from_wc())
+ if (drm_has_memcpy_from_wc())
c->tmp = pool_alloc(&c->pool, ALLOW_FAIL);
return true;
@@ -295,7 +295,7 @@ static int compress_page(struct i915_vma_compress *c,
struct z_stream_s *zstream = &c->zstream;
zstream->next_in = src;
- if (wc && c->tmp && i915_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
+ if (wc && c->tmp && drm_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
zstream->next_in = c->tmp;
zstream->avail_in = PAGE_SIZE;
@@ -395,7 +395,7 @@ static int compress_page(struct i915_vma_compress *c,
if (!ptr)
return -ENOMEM;
- if (!(wc && i915_memcpy_from_wc(ptr, src, PAGE_SIZE)))
+ if (!(wc && drm_memcpy_from_wc(ptr, src, PAGE_SIZE)))
memcpy(ptr, src, PAGE_SIZE);
dst->pages[dst->page_count++] = ptr;
cond_resched();
diff --git a/drivers/gpu/drm/i915/i915_memcpy.h b/drivers/gpu/drm/i915/i915_memcpy.h
deleted file mode 100644
index 3df063a3293b..000000000000
--- a/drivers/gpu/drm/i915/i915_memcpy.h
+++ /dev/null
@@ -1,34 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#ifndef __I915_MEMCPY_H__
-#define __I915_MEMCPY_H__
-
-#include <linux/types.h>
-
-struct drm_i915_private;
-
-void i915_memcpy_init_early(struct drm_i915_private *i915);
-
-bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len);
-void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
-
-/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
- * as well as SSE4.1 support. i915_memcpy_from_wc() will report if it cannot
- * perform the operation. To check beforehand, pass in the parameters to
- * to i915_can_memcpy_from_wc() - since we only care about the low 4 bits,
- * you only need to pass in the minor offsets, page-aligned pointers are
- * always valid.
- *
- * For just checking for SSE4.1, in the foreknowledge that the future use
- * will be correctly aligned, just use i915_has_memcpy_from_wc().
- */
-#define i915_can_memcpy_from_wc(dst, src, len) \
- i915_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
-
-#define i915_has_memcpy_from_wc() \
- i915_memcpy_from_wc(NULL, NULL, 0)
-
-#endif /* __I915_MEMCPY_H__ */
diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
index c85d516b85cd..6bb399e9be78 100644
--- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
@@ -6,6 +6,8 @@
#include <linux/prime_numbers.h>
#include <linux/sort.h>
+#include <drm/drm_memcpy.h>
+
#include "../i915_selftest.h"
#include "mock_drm.h"
@@ -20,7 +22,6 @@
#include "gem/selftests/mock_context.h"
#include "gt/intel_engine_user.h"
#include "gt/intel_gt.h"
-#include "i915_memcpy.h"
#include "selftests/igt_flush_test.h"
#include "selftests/i915_random.h"
@@ -901,7 +902,7 @@ static inline void igt_memcpy(void *dst, const void *src, size_t size)
static inline void igt_memcpy_from_wc(void *dst, const void *src, size_t size)
{
- i915_memcpy_from_wc(dst, src, size);
+ drm_memcpy_from_wc(dst, src, size);
}
static int _perf_memcpy(struct intel_memory_region *src_mr,
@@ -925,7 +926,7 @@ static int _perf_memcpy(struct intel_memory_region *src_mr,
{
"memcpy_from_wc",
igt_memcpy_from_wc,
- !i915_has_memcpy_from_wc(),
+ !drm_has_memcpy_from_wc(),
},
};
struct drm_i915_gem_object *src, *dst;
diff --git a/include/drm/drm_memcpy.h b/include/drm/drm_memcpy.h
new file mode 100644
index 000000000000..fe5ed1e89ce6
--- /dev/null
+++ b/include/drm/drm_memcpy.h
@@ -0,0 +1,68 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __DRM_MEMCPY_H__
+#define __DRM_MEMCPY_H__
+
+#include <linux/types.h>
+
+struct dma_buf_map;
+
+#ifdef CONFIG_X86
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len);
+bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
+ const struct dma_buf_map *src,
+ unsigned long len);
+void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
+
+/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
+ * as well as SSE4.1 support. drm_memcpy_from_wc() will report if it cannot
+ * perform the operation. To check beforehand, pass in the parameters to
+ * drm_can_memcpy_from_wc() - since we only care about the low 4 bits,
+ * you only need to pass in the minor offsets, page-aligned pointers are
+ * always valid.
+ *
+ * For just checking for SSE4.1, in the foreknowledge that the future use
+ * will be correctly aligned, just use drm_has_memcpy_from_wc().
+ */
+#define drm_can_memcpy_from_wc(dst, src, len) \
+ drm_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
+
+#define drm_has_memcpy_from_wc() \
+ drm_memcpy_from_wc(NULL, NULL, 0)
+
+void drm_memcpy_init_early(void);
+
+#else
+
+static inline
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+{
+ return false;
+}
+
+static inline
+bool drm_memcpy_from_wc_dbm(void *dst, const void *src, unsigned long len)
+{
+ return false;
+}
+
+static inline
+bool drm_can_memcpy_from_wc_dbm(void *dst, const void *src, unsigned long len)
+{
+ return false;
+}
+
+static inline
+bool drm_has_memcpy_from_wc(void)
+{
+ return false;
+}
+
+#define drm_has_memcpy_from_wc() (false)
+#define drm_unaligned_memcpy_from_wc(_dst, _src, _len) WARN_ON(1)
+#define drm_memcpy_init_early() do {} while (0)
+#endif /* CONFIG_X86 */
+#endif /* __DRM_MEMCPY_H__ */
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 08/15] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
Cc: Thomas Hellström, Christian König, Daniel Vetter
Use fast wc memcpy for reading out of wc memory for TTM bo moves.
Cc: Dave Airlie <airlied@gmail.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
--
v4:
- Clarify when we try drm_memcpy_from_wc_dbm (Reported by Matthew Auld)
- Be paranoid about when drm_memcpy_from_wc_dbm may fail (Reported by
Matthew Auld)
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 6ac7744a1a5c..ebff603a97f4 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -31,6 +31,7 @@
#include <drm/ttm/ttm_bo_driver.h>
#include <drm/ttm/ttm_placement.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_vma_manager.h>
#include <linux/dma-buf-map.h>
#include <linux/io.h>
@@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
struct ttm_tt *ttm = bo->ttm;
struct dma_buf_map src_map, dst_map;
+ bool wc_memcpy;
pgoff_t i;
/* Single TTM move. NOP */
@@ -114,11 +116,21 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
return;
}
+ /*
+ * Condition this on src being WC if needed. However i915 perf
+ * selftest indicates that for PAGE_SIZE chunks, wc_memcpy
+ * outperforms memcpy() on all cases except WB->WB where results
+ * are similar.
+ */
+ wc_memcpy = drm_has_memcpy_from_wc();
+
for (i = 0; i < num_pages; ++i) {
dst_ops->map_local(dst_iter, &dst_map, i);
src_ops->map_local(src_iter, &src_map, i);
- if (!src_map.is_iomem && !dst_map.is_iomem) {
+ if (wc_memcpy && drm_memcpy_from_wc_dbm(&dst_map, &src_map, PAGE_SIZE)) {
+ ;
+ } else if (!src_map.is_iomem && !dst_map.is_iomem) {
memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
} else if (!src_map.is_iomem) {
dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 08/15] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
Cc: Thomas Hellström, Christian König, Daniel Vetter
Use fast wc memcpy for reading out of wc memory for TTM bo moves.
Cc: Dave Airlie <airlied@gmail.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
--
v4:
- Clarify when we try drm_memcpy_from_wc_dbm (Reported by Matthew Auld)
- Be paranoid about when drm_memcpy_from_wc_dbm may fail (Reported by
Matthew Auld)
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 6ac7744a1a5c..ebff603a97f4 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -31,6 +31,7 @@
#include <drm/ttm/ttm_bo_driver.h>
#include <drm/ttm/ttm_placement.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_vma_manager.h>
#include <linux/dma-buf-map.h>
#include <linux/io.h>
@@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
struct ttm_tt *ttm = bo->ttm;
struct dma_buf_map src_map, dst_map;
+ bool wc_memcpy;
pgoff_t i;
/* Single TTM move. NOP */
@@ -114,11 +116,21 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
return;
}
+ /*
+ * Condition this on src being WC if needed. However i915 perf
+ * selftest indicates that for PAGE_SIZE chunks, wc_memcpy
+ * outperforms memcpy() on all cases except WB->WB where results
+ * are similar.
+ */
+ wc_memcpy = drm_has_memcpy_from_wc();
+
for (i = 0; i < num_pages; ++i) {
dst_ops->map_local(dst_iter, &dst_map, i);
src_ops->map_local(src_iter, &src_map, i);
- if (!src_map.is_iomem && !dst_map.is_iomem) {
+ if (wc_memcpy && drm_memcpy_from_wc_dbm(&dst_map, &src_map, PAGE_SIZE)) {
+ ;
+ } else if (!src_map.is_iomem && !dst_map.is_iomem) {
memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
} else if (!src_map.is_iomem) {
dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 09/15] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
create a ghost object and push it out to delayed destroy.
Fix this by adding a path for idle, and document the function.
Also avoid having the bo end up in a bad state vulnerable to user-space
triggered kernel BUGs if the call to ttm_tt_create() fails.
Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v4:
- Clarify why we mark bo for clearing after ttm_bo_pipeline_gutting()
(Reported by Matthew Auld)
---
drivers/gpu/drm/ttm/ttm_bo.c | 20 +++++------
drivers/gpu/drm/ttm/ttm_bo_util.c | 55 ++++++++++++++++++++++++++++---
drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
include/drm/ttm/ttm_tt.h | 10 ++++++
4 files changed, 76 insertions(+), 14 deletions(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 51a94fd63bd7..be0406466460 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -501,10 +501,15 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
bdev->funcs->evict_flags(bo, &placement);
if (!placement.num_placement && !placement.num_busy_placement) {
- ttm_bo_wait(bo, false, false);
+ ret = ttm_bo_wait(bo, true, false);
+ if (ret)
+ return ret;
- ttm_bo_cleanup_memtype_use(bo);
- return ttm_tt_create(bo, false);
+ /*
+ * Since we've already synced, this frees backing store
+ * immediately.
+ */
+ return ttm_bo_pipeline_gutting(bo);
}
ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
@@ -976,13 +981,8 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
/*
* Remove the backing store if no placement is given.
*/
- if (!placement->num_placement && !placement->num_busy_placement) {
- ret = ttm_bo_pipeline_gutting(bo);
- if (ret)
- return ret;
-
- return ttm_tt_create(bo, false);
- }
+ if (!placement->num_placement && !placement->num_busy_placement)
+ return ttm_bo_pipeline_gutting(bo);
/*
* Check whether we need to move buffer.
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index ebff603a97f4..4cca932f1c0e 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -590,26 +590,73 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
}
EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
+/**
+ * ttm_bo_pipeline_gutting - purge the contents of a bo
+ * @bo: The buffer object
+ *
+ * Purge the contents of a bo, async if the bo is not idle.
+ * After a successful call, the bo is left unpopulated in
+ * system placement. The function may wait uninterruptible
+ * for idle on OOM.
+ *
+ * Return: 0 if successful, negative error code on failure.
+ */
int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
{
static const struct ttm_place sys_mem = { .mem_type = TTM_PL_SYSTEM };
struct ttm_buffer_object *ghost;
+ struct ttm_tt *ttm;
int ret;
- ret = ttm_buffer_object_transfer(bo, &ghost);
+ /* If already idle, no need for ghost object dance. */
+ ret = ttm_bo_wait(bo, false, true);
+ if (ret != -EBUSY) {
+ if (!bo->ttm) {
+ /* See comment below about clearing. */
+ ret = ttm_tt_create(bo, true);
+ if (ret)
+ return ret;
+ } else {
+ ttm_tt_unpopulate(bo->bdev, bo->ttm);
+ if (bo->type == ttm_bo_type_device)
+ ttm_tt_mark_for_clear(bo->ttm);
+ }
+ ttm_resource_free(bo, &bo->mem);
+ ttm_resource_alloc(bo, &sys_mem, &bo->mem);
+
+ return 0;
+ }
+
+ /*
+ * We need an unpopulated ttm_tt after giving our current one,
+ * if any, to the ghost object. And we can't afford to fail
+ * creating one *after* the operation. If the bo subsequently gets
+ * resurrected, make sure it's cleared (if ttm_bo_type_device)
+ * to avoid leaking sensitive information to user-space.
+ */
+
+ ttm = bo->ttm;
+ bo->ttm = NULL;
+ ret = ttm_tt_create(bo, true);
+ swap(bo->ttm, ttm);
if (ret)
return ret;
+ ret = ttm_buffer_object_transfer(bo, &ghost);
+ if (ret) {
+ ttm_tt_destroy(bo->bdev, ttm);
+ return ret;
+ }
+
ret = dma_resv_copy_fences(&ghost->base._resv, bo->base.resv);
/* Last resort, wait for the BO to be idle when we are OOM */
if (ret)
ttm_bo_wait(bo, false, false);
- ttm_resource_alloc(bo, &sys_mem, &bo->mem);
- bo->ttm = NULL;
-
dma_resv_unlock(&ghost->base._resv);
ttm_bo_put(ghost);
+ bo->ttm = ttm;
+ ttm_resource_alloc(bo, &sys_mem, &bo->mem);
return 0;
}
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 0e41227116b1..913b330a234b 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -134,6 +134,11 @@ void ttm_tt_destroy_common(struct ttm_device *bdev, struct ttm_tt *ttm)
}
EXPORT_SYMBOL(ttm_tt_destroy_common);
+void ttm_tt_mark_for_clear(struct ttm_tt *ttm)
+{
+ ttm->page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
+}
+
void ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
{
bdev->funcs->ttm_tt_destroy(bdev, ttm);
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 3102059db726..daa9c4cf48bb 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -170,6 +170,16 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, struct ttm_oper
*/
void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
+/**
+ * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
+ *
+ * @ttm: Pointer to the ttm_tt structure
+ *
+ * Marks pages for clearing so that the next time the page vector is
+ * populated, the pages will be cleared.
+ */
+void ttm_tt_mark_for_clear(struct ttm_tt *ttm);
+
void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 09/15] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
create a ghost object and push it out to delayed destroy.
Fix this by adding a path for idle, and document the function.
Also avoid having the bo end up in a bad state vulnerable to user-space
triggered kernel BUGs if the call to ttm_tt_create() fails.
Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v4:
- Clarify why we mark bo for clearing after ttm_bo_pipeline_gutting()
(Reported by Matthew Auld)
---
drivers/gpu/drm/ttm/ttm_bo.c | 20 +++++------
drivers/gpu/drm/ttm/ttm_bo_util.c | 55 ++++++++++++++++++++++++++++---
drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
include/drm/ttm/ttm_tt.h | 10 ++++++
4 files changed, 76 insertions(+), 14 deletions(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 51a94fd63bd7..be0406466460 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -501,10 +501,15 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
bdev->funcs->evict_flags(bo, &placement);
if (!placement.num_placement && !placement.num_busy_placement) {
- ttm_bo_wait(bo, false, false);
+ ret = ttm_bo_wait(bo, true, false);
+ if (ret)
+ return ret;
- ttm_bo_cleanup_memtype_use(bo);
- return ttm_tt_create(bo, false);
+ /*
+ * Since we've already synced, this frees backing store
+ * immediately.
+ */
+ return ttm_bo_pipeline_gutting(bo);
}
ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
@@ -976,13 +981,8 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
/*
* Remove the backing store if no placement is given.
*/
- if (!placement->num_placement && !placement->num_busy_placement) {
- ret = ttm_bo_pipeline_gutting(bo);
- if (ret)
- return ret;
-
- return ttm_tt_create(bo, false);
- }
+ if (!placement->num_placement && !placement->num_busy_placement)
+ return ttm_bo_pipeline_gutting(bo);
/*
* Check whether we need to move buffer.
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index ebff603a97f4..4cca932f1c0e 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -590,26 +590,73 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
}
EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
+/**
+ * ttm_bo_pipeline_gutting - purge the contents of a bo
+ * @bo: The buffer object
+ *
+ * Purge the contents of a bo, async if the bo is not idle.
+ * After a successful call, the bo is left unpopulated in
+ * system placement. The function may wait uninterruptible
+ * for idle on OOM.
+ *
+ * Return: 0 if successful, negative error code on failure.
+ */
int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
{
static const struct ttm_place sys_mem = { .mem_type = TTM_PL_SYSTEM };
struct ttm_buffer_object *ghost;
+ struct ttm_tt *ttm;
int ret;
- ret = ttm_buffer_object_transfer(bo, &ghost);
+ /* If already idle, no need for ghost object dance. */
+ ret = ttm_bo_wait(bo, false, true);
+ if (ret != -EBUSY) {
+ if (!bo->ttm) {
+ /* See comment below about clearing. */
+ ret = ttm_tt_create(bo, true);
+ if (ret)
+ return ret;
+ } else {
+ ttm_tt_unpopulate(bo->bdev, bo->ttm);
+ if (bo->type == ttm_bo_type_device)
+ ttm_tt_mark_for_clear(bo->ttm);
+ }
+ ttm_resource_free(bo, &bo->mem);
+ ttm_resource_alloc(bo, &sys_mem, &bo->mem);
+
+ return 0;
+ }
+
+ /*
+ * We need an unpopulated ttm_tt after giving our current one,
+ * if any, to the ghost object. And we can't afford to fail
+ * creating one *after* the operation. If the bo subsequently gets
+ * resurrected, make sure it's cleared (if ttm_bo_type_device)
+ * to avoid leaking sensitive information to user-space.
+ */
+
+ ttm = bo->ttm;
+ bo->ttm = NULL;
+ ret = ttm_tt_create(bo, true);
+ swap(bo->ttm, ttm);
if (ret)
return ret;
+ ret = ttm_buffer_object_transfer(bo, &ghost);
+ if (ret) {
+ ttm_tt_destroy(bo->bdev, ttm);
+ return ret;
+ }
+
ret = dma_resv_copy_fences(&ghost->base._resv, bo->base.resv);
/* Last resort, wait for the BO to be idle when we are OOM */
if (ret)
ttm_bo_wait(bo, false, false);
- ttm_resource_alloc(bo, &sys_mem, &bo->mem);
- bo->ttm = NULL;
-
dma_resv_unlock(&ghost->base._resv);
ttm_bo_put(ghost);
+ bo->ttm = ttm;
+ ttm_resource_alloc(bo, &sys_mem, &bo->mem);
return 0;
}
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 0e41227116b1..913b330a234b 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -134,6 +134,11 @@ void ttm_tt_destroy_common(struct ttm_device *bdev, struct ttm_tt *ttm)
}
EXPORT_SYMBOL(ttm_tt_destroy_common);
+void ttm_tt_mark_for_clear(struct ttm_tt *ttm)
+{
+ ttm->page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
+}
+
void ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
{
bdev->funcs->ttm_tt_destroy(bdev, ttm);
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 3102059db726..daa9c4cf48bb 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -170,6 +170,16 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, struct ttm_oper
*/
void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
+/**
+ * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
+ *
+ * @ttm: Pointer to the ttm_tt structure
+ *
+ * Marks pages for clearing so that the next time the page vector is
+ * populated, the pages will be cleared.
+ */
+void ttm_tt_mark_for_clear(struct ttm_tt *ttm);
+
void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
We are calling the eviction_valuable driver callback at eviction time to
determine whether we actually can evict a buffer object.
The upcoming i915 TTM backend needs the same functionality for swapout,
and that might actually be beneficial to other drivers as well.
Add an eviction_valuable call also in the swapout path. Try to keep the
current behaviour for all drivers by returning true if the buffer object
is already in the TTM_PL_SYSTEM placement. We change behaviour for the
case where a buffer object is in a TT backed placement when swapped out,
in which case the drivers normal eviction_valuable path is run.
Finally make sure we don't try to swapout a bo that was recently purged
and therefore unpopulated.
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v3:
- Don't export ttm_tt_unpopulate
- Fix confusion reading the locked pointer instead of the value
pointed to in ttm_bo_evict_swapout_allowable (Reported by
Maarten Lankhorst)
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
drivers/gpu/drm/ttm/ttm_bo.c | 43 ++++++++++++++++---------
drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
3 files changed, 34 insertions(+), 16 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 3bc3aebfef7c..45d194bffc3f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1348,6 +1348,10 @@ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
struct dma_fence *f;
int i;
+ /* Swapout? */
+ if (bo->mem.mem_type == TTM_PL_SYSTEM)
+ return true;
+
if (bo->type == ttm_bo_type_kernel &&
!amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
return false;
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index be0406466460..1b2d062266ed 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -536,6 +536,10 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
const struct ttm_place *place)
{
+ dma_resv_assert_held(bo->base.resv);
+ if (bo->mem.mem_type == TTM_PL_SYSTEM)
+ return true;
+
/* Don't evict this BO if it's outside of the
* requested placement range
*/
@@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
* b. Otherwise, trylock it.
*/
static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
- struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
+ struct ttm_operation_ctx *ctx,
+ const struct ttm_place *place,
+ bool *locked, bool *busy)
{
bool ret = false;
@@ -576,6 +582,14 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
*busy = !ret;
}
+ if (ret && place && !bo->bdev->funcs->eviction_valuable(bo, place)) {
+ ret = false;
+ if (*locked) {
+ dma_resv_unlock(bo->base.resv);
+ *locked = false;
+ }
+ }
+
return ret;
}
@@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
list_for_each_entry(bo, &man->lru[i], lru) {
bool busy;
- if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
- &busy)) {
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
+ &locked, &busy)) {
if (busy && !busy_bo && ticket !=
dma_resv_locking_ctx(bo->base.resv))
busy_bo = bo;
continue;
}
- if (place && !bdev->funcs->eviction_valuable(bo,
- place)) {
- if (locked)
- dma_resv_unlock(bo->base.resv);
- continue;
- }
if (!ttm_bo_get_unless_zero(bo)) {
if (locked)
dma_resv_unlock(bo->base.resv);
@@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
gfp_t gfp_flags)
{
+ struct ttm_place place = {};
bool locked;
int ret;
- if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
+ /*
+ * While the bo may already reside in SYSTEM placement, set
+ * SYSTEM as new placement to cover also the move further below.
+ * The driver may use the fact that we're moving from SYSTEM
+ * as an indication that we're about to swap out.
+ */
+ place.mem_type = TTM_PL_SYSTEM;
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
return -EBUSY;
if (!ttm_bo_get_unless_zero(bo)) {
@@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
if (bo->mem.mem_type != TTM_PL_SYSTEM) {
struct ttm_operation_ctx ctx = { false, false };
struct ttm_resource evict_mem;
- struct ttm_place place, hop;
-
- memset(&place, 0, sizeof(place));
- memset(&hop, 0, sizeof(hop));
-
- place.mem_type = TTM_PL_SYSTEM;
+ struct ttm_place hop = {};
ret = ttm_resource_alloc(bo, &place, &evict_mem);
if (unlikely(ret))
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 913b330a234b..d9793cbb6d13 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,
struct page *to_page;
int i, ret;
+ if (!ttm_tt_is_populated(ttm))
+ return 0;
+
swap_storage = shmem_file_setup("ttm swap", size, 0);
if (IS_ERR(swap_storage)) {
pr_err("Failed allocating swap storage\n");
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
We are calling the eviction_valuable driver callback at eviction time to
determine whether we actually can evict a buffer object.
The upcoming i915 TTM backend needs the same functionality for swapout,
and that might actually be beneficial to other drivers as well.
Add an eviction_valuable call also in the swapout path. Try to keep the
current behaviour for all drivers by returning true if the buffer object
is already in the TTM_PL_SYSTEM placement. We change behaviour for the
case where a buffer object is in a TT backed placement when swapped out,
in which case the drivers normal eviction_valuable path is run.
Finally make sure we don't try to swapout a bo that was recently purged
and therefore unpopulated.
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v3:
- Don't export ttm_tt_unpopulate
- Fix confusion reading the locked pointer instead of the value
pointed to in ttm_bo_evict_swapout_allowable (Reported by
Maarten Lankhorst)
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
drivers/gpu/drm/ttm/ttm_bo.c | 43 ++++++++++++++++---------
drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
3 files changed, 34 insertions(+), 16 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 3bc3aebfef7c..45d194bffc3f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1348,6 +1348,10 @@ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
struct dma_fence *f;
int i;
+ /* Swapout? */
+ if (bo->mem.mem_type == TTM_PL_SYSTEM)
+ return true;
+
if (bo->type == ttm_bo_type_kernel &&
!amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
return false;
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index be0406466460..1b2d062266ed 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -536,6 +536,10 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
const struct ttm_place *place)
{
+ dma_resv_assert_held(bo->base.resv);
+ if (bo->mem.mem_type == TTM_PL_SYSTEM)
+ return true;
+
/* Don't evict this BO if it's outside of the
* requested placement range
*/
@@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
* b. Otherwise, trylock it.
*/
static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
- struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
+ struct ttm_operation_ctx *ctx,
+ const struct ttm_place *place,
+ bool *locked, bool *busy)
{
bool ret = false;
@@ -576,6 +582,14 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
*busy = !ret;
}
+ if (ret && place && !bo->bdev->funcs->eviction_valuable(bo, place)) {
+ ret = false;
+ if (*locked) {
+ dma_resv_unlock(bo->base.resv);
+ *locked = false;
+ }
+ }
+
return ret;
}
@@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
list_for_each_entry(bo, &man->lru[i], lru) {
bool busy;
- if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
- &busy)) {
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
+ &locked, &busy)) {
if (busy && !busy_bo && ticket !=
dma_resv_locking_ctx(bo->base.resv))
busy_bo = bo;
continue;
}
- if (place && !bdev->funcs->eviction_valuable(bo,
- place)) {
- if (locked)
- dma_resv_unlock(bo->base.resv);
- continue;
- }
if (!ttm_bo_get_unless_zero(bo)) {
if (locked)
dma_resv_unlock(bo->base.resv);
@@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
gfp_t gfp_flags)
{
+ struct ttm_place place = {};
bool locked;
int ret;
- if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
+ /*
+ * While the bo may already reside in SYSTEM placement, set
+ * SYSTEM as new placement to cover also the move further below.
+ * The driver may use the fact that we're moving from SYSTEM
+ * as an indication that we're about to swap out.
+ */
+ place.mem_type = TTM_PL_SYSTEM;
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
return -EBUSY;
if (!ttm_bo_get_unless_zero(bo)) {
@@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
if (bo->mem.mem_type != TTM_PL_SYSTEM) {
struct ttm_operation_ctx ctx = { false, false };
struct ttm_resource evict_mem;
- struct ttm_place place, hop;
-
- memset(&place, 0, sizeof(place));
- memset(&hop, 0, sizeof(hop));
-
- place.mem_type = TTM_PL_SYSTEM;
+ struct ttm_place hop = {};
ret = ttm_resource_alloc(bo, &place, &evict_mem);
if (unlikely(ret))
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 913b330a234b..d9793cbb6d13 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,
struct page *to_page;
int i, ret;
+ if (!ttm_tt_is_populated(ttm))
+ return 0;
+
swap_storage = shmem_file_setup("ttm swap", size, 0);
if (IS_ERR(swap_storage)) {
pr_err("Failed allocating swap storage\n");
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 11/15] drm/i915/ttm: Introduce a TTM i915 gem object backend
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Most logical place to introduce TTM buffer objects is as an i915
gem object backend. We need to add some ops to account for added
functionality like delayed delete and LRU list manipulation.
Initially we support only LMEM and SYSTEM memory, but SYSTEM
(which in this case means evicted LMEM objects) is not
visible to i915 GEM yet. The plan is to move the i915 gem system region
over to the TTM system memory type in upcoming patches.
We set up GPU bindings directly both from LMEM and from the system region,
as there is no need to use the legacy TTM_TT memory type. We reserve
that for future porting of GGTT bindings to TTM.
Remove the old lmem backend.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v2:
- Break out needed TTM functionality to a separate patch (Reported by
Christian König).
- Fix an unhandled error (Reported by Matthew Auld and Maarten Lankhorst)
- Remove a stray leftover sg_table allocation (Reported by Matthew Auld)
- Use ttm_tt_unpopulate() rather than ttm_tt_destroy() in the purge path
as some TTM functionality relies on having a ttm_tt present for !is_iomem.
v3:
- Use ttm_bo_type_device for userspace visible objects so that TTM can
allocate an address space offset for mmap'ing.
- Fix up the destruction path (Reported by Matthew Auld)
- Use ttm_bo_validate() for purging (Reported by Christian König)
- Create ttm_tts write-combined as they are currently for eviction only and
we want to maintain consistent write-combined caching for bos that are
not in system only. (Suggested by Daniel Vetter)
- Make struct ttm_placements static.
- Add the ttm device funcs/ops to i915_gem_ttm.h for the region code.
- Rename new->dst and old->src. Check for swapin in the move callback.
v4:
- Adapt to small interface change in ttm_move_memcpy.
- Use a function to pull out the ttm driver from the backend.
---
drivers/gpu/drm/i915/Makefile | 1 +
drivers/gpu/drm/i915/gem/i915_gem_create.c | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 84 ---
drivers/gpu/drm/i915/gem/i915_gem_lmem.h | 5 -
drivers/gpu/drm/i915/gem/i915_gem_object.c | 125 ++--
drivers/gpu/drm/i915/gem/i915_gem_object.h | 9 +
.../gpu/drm/i915/gem/i915_gem_object_types.h | 27 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 541 ++++++++++++++++++
drivers/gpu/drm/i915/gem/i915_gem_ttm.h | 48 ++
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 3 +-
drivers/gpu/drm/i915/i915_gem.c | 5 +-
drivers/gpu/drm/i915/intel_memory_region.c | 1 -
drivers/gpu/drm/i915/intel_memory_region.h | 1 -
drivers/gpu/drm/i915/intel_region_ttm.c | 8 +-
drivers/gpu/drm/i915/intel_region_ttm.h | 11 +-
16 files changed, 731 insertions(+), 153 deletions(-)
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.c
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.h
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index ebc19bd5fff4..9f9cd5c085c3 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -154,6 +154,7 @@ gem-y += \
gem/i915_gem_stolen.o \
gem/i915_gem_throttle.o \
gem/i915_gem_tiling.o \
+ gem/i915_gem_ttm.o \
gem/i915_gem_userptr.o \
gem/i915_gem_wait.o \
gem/i915_gemfs.o
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_create.c b/drivers/gpu/drm/i915/gem/i915_gem_create.c
index 548ddf39d853..93bf63bbaff1 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_create.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_create.c
@@ -85,13 +85,10 @@ i915_gem_setup(struct drm_i915_gem_object *obj, u64 size)
return -E2BIG;
/*
- * For now resort to CPU based clearing for device local-memory, in the
- * near future this will use the blitter engine for accelerated, GPU
- * based clearing.
+ * I915_BO_ALLOC_USER will make sure the object is cleared before
+ * any user access.
*/
- flags = 0;
- if (mr->type == INTEL_MEMORY_LOCAL)
- flags = I915_BO_ALLOC_CPU_CLEAR;
+ flags = I915_BO_ALLOC_USER;
ret = mr->ops->init_object(mr, obj, size, flags);
if (ret)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 3b4aa28a076d..2b8cd15de1d9 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -4,74 +4,10 @@
*/
#include "intel_memory_region.h"
-#include "intel_region_ttm.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_lmem.h"
#include "i915_drv.h"
-static void lmem_put_pages(struct drm_i915_gem_object *obj,
- struct sg_table *pages)
-{
- intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
- obj->mm.dirty = false;
- sg_free_table(pages);
- kfree(pages);
-}
-
-static int lmem_get_pages(struct drm_i915_gem_object *obj)
-{
- unsigned int flags;
- struct sg_table *pages;
-
- flags = I915_ALLOC_MIN_PAGE_SIZE;
- if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
- flags |= I915_ALLOC_CONTIGUOUS;
-
- obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
- obj->base.size,
- flags);
- if (IS_ERR(obj->mm.st_mm_node))
- return PTR_ERR(obj->mm.st_mm_node);
-
- /* Range manager is always contigous */
- if (obj->mm.region->is_range_manager)
- obj->flags |= I915_BO_ALLOC_CONTIGUOUS;
- pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
- if (IS_ERR(pages)) {
- intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
- return PTR_ERR(pages);
- }
-
- __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
-
- if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
- void __iomem *vaddr =
- i915_gem_object_lmem_io_map(obj, 0, obj->base.size);
-
- if (!vaddr) {
- struct sg_table *pages =
- __i915_gem_object_unset_pages(obj);
-
- if (!IS_ERR_OR_NULL(pages))
- lmem_put_pages(obj, pages);
- }
-
- memset_io(vaddr, 0, obj->base.size);
- io_mapping_unmap(vaddr);
- }
-
- return 0;
-}
-
-const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
- .name = "i915_gem_object_lmem",
- .flags = I915_GEM_OBJECT_HAS_IOMEM,
-
- .get_pages = lmem_get_pages,
- .put_pages = lmem_put_pages,
- .release = i915_gem_object_release_memory_region,
-};
-
void __iomem *
i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
unsigned long n,
@@ -103,23 +39,3 @@ i915_gem_object_create_lmem(struct drm_i915_private *i915,
return i915_gem_object_create_region(i915->mm.regions[INTEL_REGION_LMEM],
size, flags);
}
-
-int __i915_gem_lmem_object_init(struct intel_memory_region *mem,
- struct drm_i915_gem_object *obj,
- resource_size_t size,
- unsigned int flags)
-{
- static struct lock_class_key lock_class;
- struct drm_i915_private *i915 = mem->i915;
-
- drm_gem_private_object_init(&i915->drm, &obj->base, size);
- i915_gem_object_init(obj, &i915_gem_lmem_obj_ops, &lock_class, flags);
-
- obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
-
- i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
-
- i915_gem_object_init_memory_region(obj, mem);
-
- return 0;
-}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
index fac6bc5a5ebb..ea76fd11ccb0 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
@@ -26,9 +26,4 @@ i915_gem_object_create_lmem(struct drm_i915_private *i915,
resource_size_t size,
unsigned int flags);
-int __i915_gem_lmem_object_init(struct intel_memory_region *mem,
- struct drm_i915_gem_object *obj,
- resource_size_t size,
- unsigned int flags);
-
#endif /* !__I915_GEM_LMEM_H */
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index e9247afb0320..df2b4e6b9bcc 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -173,7 +173,7 @@ static void i915_gem_close_object(struct drm_gem_object *gem, struct drm_file *f
}
}
-static void __i915_gem_free_object_rcu(struct rcu_head *head)
+void __i915_gem_free_object_rcu(struct rcu_head *head)
{
struct drm_i915_gem_object *obj =
container_of(head, typeof(*obj), rcu);
@@ -209,59 +209,69 @@ static void __i915_gem_object_free_mmaps(struct drm_i915_gem_object *obj)
}
}
-static void __i915_gem_free_objects(struct drm_i915_private *i915,
- struct llist_node *freed)
+void __i915_gem_free_object(struct drm_i915_gem_object *obj)
{
- struct drm_i915_gem_object *obj, *on;
+ trace_i915_gem_object_destroy(obj);
- llist_for_each_entry_safe(obj, on, freed, freed) {
- trace_i915_gem_object_destroy(obj);
+ if (!list_empty(&obj->vma.list)) {
+ struct i915_vma *vma;
+
+ /*
+ * Note that the vma keeps an object reference while
+ * it is active, so it *should* not sleep while we
+ * destroy it. Our debug code errs insits it *might*.
+ * For the moment, play along.
+ */
+ spin_lock(&obj->vma.lock);
+ while ((vma = list_first_entry_or_null(&obj->vma.list,
+ struct i915_vma,
+ obj_link))) {
+ GEM_BUG_ON(vma->obj != obj);
+ spin_unlock(&obj->vma.lock);
- if (!list_empty(&obj->vma.list)) {
- struct i915_vma *vma;
+ __i915_vma_put(vma);
- /*
- * Note that the vma keeps an object reference while
- * it is active, so it *should* not sleep while we
- * destroy it. Our debug code errs insits it *might*.
- * For the moment, play along.
- */
spin_lock(&obj->vma.lock);
- while ((vma = list_first_entry_or_null(&obj->vma.list,
- struct i915_vma,
- obj_link))) {
- GEM_BUG_ON(vma->obj != obj);
- spin_unlock(&obj->vma.lock);
+ }
+ spin_unlock(&obj->vma.lock);
+ }
- __i915_vma_put(vma);
+ __i915_gem_object_free_mmaps(obj);
- spin_lock(&obj->vma.lock);
- }
- spin_unlock(&obj->vma.lock);
- }
+ GEM_BUG_ON(!list_empty(&obj->lut_list));
- __i915_gem_object_free_mmaps(obj);
+ atomic_set(&obj->mm.pages_pin_count, 0);
+ __i915_gem_object_put_pages(obj);
+ GEM_BUG_ON(i915_gem_object_has_pages(obj));
+ bitmap_free(obj->bit_17);
- GEM_BUG_ON(!list_empty(&obj->lut_list));
+ if (obj->base.import_attach)
+ drm_prime_gem_destroy(&obj->base, NULL);
- atomic_set(&obj->mm.pages_pin_count, 0);
- __i915_gem_object_put_pages(obj);
- GEM_BUG_ON(i915_gem_object_has_pages(obj));
- bitmap_free(obj->bit_17);
+ drm_gem_free_mmap_offset(&obj->base);
- if (obj->base.import_attach)
- drm_prime_gem_destroy(&obj->base, NULL);
+ if (obj->ops->release)
+ obj->ops->release(obj);
- drm_gem_free_mmap_offset(&obj->base);
+ if (obj->mm.n_placements > 1)
+ kfree(obj->mm.placements);
- if (obj->ops->release)
- obj->ops->release(obj);
+ if (obj->shares_resv_from)
+ i915_vm_resv_put(obj->shares_resv_from);
+}
- if (obj->mm.n_placements > 1)
- kfree(obj->mm.placements);
+static void __i915_gem_free_objects(struct drm_i915_private *i915,
+ struct llist_node *freed)
+{
+ struct drm_i915_gem_object *obj, *on;
- if (obj->shares_resv_from)
- i915_vm_resv_put(obj->shares_resv_from);
+ llist_for_each_entry_safe(obj, on, freed, freed) {
+ might_sleep();
+ if (obj->ops->delayed_free) {
+ obj->ops->delayed_free(obj);
+ continue;
+ }
+ __i915_gem_free_object(obj);
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
@@ -319,6 +329,7 @@ static void i915_gem_free_object(struct drm_gem_object *gem_obj)
* worker and performing frees directly from subsequent allocations for
* crude but effective memory throttling.
*/
+
if (llist_add(&obj->freed, &i915->mm.free_list))
queue_work(i915->wq, &i915->mm.free_work);
}
@@ -411,6 +422,42 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset,
return 0;
}
+/**
+ * i915_gem_object_evictable - Whether object is likely evictable after unbind.
+ * @obj: The object to check
+ *
+ * This function checks whether the object is likely unvictable after unbind.
+ * If the object is not locked when checking, the result is only advisory.
+ * If the object is locked when checking, and the function returns true,
+ * then an eviction should indeed be possible. But since unlocked vma
+ * unpinning and unbinding is currently possible, the object can actually
+ * become evictable even if this function returns false.
+ *
+ * Return: true if the object may be evictable. False otherwise.
+ */
+bool i915_gem_object_evictable(struct drm_i915_gem_object *obj)
+{
+ struct i915_vma *vma;
+ int pin_count = atomic_read(&obj->mm.pages_pin_count);
+
+ if (!pin_count)
+ return true;
+
+ spin_lock(&obj->vma.lock);
+ list_for_each_entry(vma, &obj->vma.list, obj_link) {
+ if (i915_vma_is_pinned(vma)) {
+ spin_unlock(&obj->vma.lock);
+ return false;
+ }
+ if (atomic_read(&vma->pages_count))
+ pin_count--;
+ }
+ spin_unlock(&obj->vma.lock);
+ GEM_WARN_ON(pin_count < 0);
+
+ return pin_count == 0;
+}
+
void i915_gem_init__objects(struct drm_i915_private *i915)
{
INIT_WORK(&i915->mm.free_work, __i915_gem_free_work);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index 2ebd79537aea..ae5930e307d5 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -200,6 +200,9 @@ static inline bool i915_gem_object_trylock(struct drm_i915_gem_object *obj)
static inline void i915_gem_object_unlock(struct drm_i915_gem_object *obj)
{
+ if (obj->ops->adjust_lru)
+ obj->ops->adjust_lru(obj);
+
dma_resv_unlock(obj->base.resv);
}
@@ -587,6 +590,12 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset,
bool i915_gem_object_is_shmem(const struct drm_i915_gem_object *obj);
+void __i915_gem_free_object_rcu(struct rcu_head *head);
+
+void __i915_gem_free_object(struct drm_i915_gem_object *obj);
+
+bool i915_gem_object_evictable(struct drm_i915_gem_object *obj);
+
#ifdef CONFIG_MMU_NOTIFIER
static inline bool
i915_gem_object_is_userptr(struct drm_i915_gem_object *obj)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index d047ea126029..68313474e6a6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -63,6 +63,20 @@ struct drm_i915_gem_object_ops {
const struct drm_i915_gem_pwrite *arg);
int (*dmabuf_export)(struct drm_i915_gem_object *obj);
+
+ /**
+ * adjust_lru - notify that the madvise value was updated
+ * @obj: The gem object
+ *
+ * The madvise value may have been updated, or object was recently
+ * referenced so act accordingly (Perhaps changing an LRU list etc).
+ */
+ void (*adjust_lru)(struct drm_i915_gem_object *obj);
+
+ /**
+ * delayed_free - Override the default delayed free implementation
+ */
+ void (*delayed_free)(struct drm_i915_gem_object *obj);
void (*release)(struct drm_i915_gem_object *obj);
const char *name; /* friendly name for debug, e.g. lockdep classes */
@@ -187,12 +201,14 @@ struct drm_i915_gem_object {
#define I915_BO_ALLOC_VOLATILE BIT(1)
#define I915_BO_ALLOC_STRUCT_PAGE BIT(2)
#define I915_BO_ALLOC_CPU_CLEAR BIT(3)
+#define I915_BO_ALLOC_USER BIT(4)
#define I915_BO_ALLOC_FLAGS (I915_BO_ALLOC_CONTIGUOUS | \
I915_BO_ALLOC_VOLATILE | \
I915_BO_ALLOC_STRUCT_PAGE | \
- I915_BO_ALLOC_CPU_CLEAR)
-#define I915_BO_READONLY BIT(4)
-#define I915_TILING_QUIRK_BIT 5 /* unknown swizzling; do not release! */
+ I915_BO_ALLOC_CPU_CLEAR | \
+ I915_BO_ALLOC_USER)
+#define I915_BO_READONLY BIT(5)
+#define I915_TILING_QUIRK_BIT 6 /* unknown swizzling; do not release! */
/*
* Is the object to be mapped as read-only to the GPU
@@ -310,6 +326,11 @@ struct drm_i915_gem_object {
bool dirty:1;
} mm;
+ struct {
+ struct sg_table *cached_io_st;
+ bool created:1;
+ } ttm;
+
/** Record of address bit 17 of each page at last unbind. */
unsigned long *bit_17;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.c b/drivers/gpu/drm/i915/gem/i915_gem_region.c
index f25e6646c5b7..d1f1840540dd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.c
@@ -18,11 +18,7 @@ void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
mutex_lock(&mem->objects.lock);
- if (obj->flags & I915_BO_ALLOC_VOLATILE)
- list_add(&obj->mm.region_link, &mem->objects.purgeable);
- else
- list_add(&obj->mm.region_link, &mem->objects.list);
-
+ list_add(&obj->mm.region_link, &mem->objects.list);
mutex_unlock(&mem->objects.lock);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
new file mode 100644
index 000000000000..17598930a99e
--- /dev/null
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -0,0 +1,541 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_placement.h>
+
+#include "i915_drv.h"
+#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
+
+#include "gem/i915_gem_object.h"
+#include "gem/i915_gem_region.h"
+#include "gem/i915_gem_ttm.h"
+
+#define I915_PL_LMEM0 TTM_PL_PRIV
+#define I915_PL_SYSTEM TTM_PL_SYSTEM
+#define I915_PL_STOLEN TTM_PL_VRAM
+#define I915_PL_GGTT TTM_PL_TT
+
+#define I915_TTM_PRIO_PURGE 0
+#define I915_TTM_PRIO_NO_PAGES 1
+#define I915_TTM_PRIO_HAS_PAGES 2
+
+/**
+ * struct i915_ttm_tt - TTM page vector with additional private information
+ * @ttm: The base TTM page vector.
+ * @dev: The struct device used for dma mapping and unmapping.
+ * @cached_st: The cached scatter-gather table.
+ *
+ * Note that DMA may be going on right up to the point where the page-
+ * vector is unpopulated in delayed destroy. Hence keep the
+ * scatter-gather table mapped and cached up to that point. This is
+ * different from the cached gem object io scatter-gather table which
+ * doesn't have an associated dma mapping.
+ */
+struct i915_ttm_tt {
+ struct ttm_tt ttm;
+ struct device *dev;
+ struct sg_table *cached_st;
+};
+
+static const struct ttm_place lmem0_sys_placement_flags[] = {
+ {
+ .fpfn = 0,
+ .lpfn = 0,
+ .mem_type = I915_PL_LMEM0,
+ .flags = 0,
+ }, {
+ .fpfn = 0,
+ .lpfn = 0,
+ .mem_type = I915_PL_SYSTEM,
+ .flags = 0,
+ }
+};
+
+static struct ttm_placement i915_lmem0_placement = {
+ .num_placement = 1,
+ .placement = &lmem0_sys_placement_flags[0],
+ .num_busy_placement = 1,
+ .busy_placement = &lmem0_sys_placement_flags[0],
+};
+
+static struct ttm_placement i915_sys_placement = {
+ .num_placement = 1,
+ .placement = &lmem0_sys_placement_flags[1],
+ .num_busy_placement = 1,
+ .busy_placement = &lmem0_sys_placement_flags[1],
+};
+
+static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj);
+
+static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo,
+ uint32_t page_flags)
+{
+ struct ttm_resource_manager *man =
+ ttm_manager_type(bo->bdev, bo->mem.mem_type);
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ struct i915_ttm_tt *i915_tt;
+ int ret;
+
+ i915_tt = kzalloc(sizeof(*i915_tt), GFP_KERNEL);
+ if (!i915_tt)
+ return NULL;
+
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR &&
+ man->use_tt)
+ page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
+
+ ret = ttm_tt_init(&i915_tt->ttm, bo, page_flags, ttm_write_combined);
+ if (ret) {
+ kfree(i915_tt);
+ return NULL;
+ }
+
+ i915_tt->dev = obj->base.dev->dev;
+
+ return &i915_tt->ttm;
+}
+
+static void i915_ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+
+ if (i915_tt->cached_st) {
+ dma_unmap_sgtable(i915_tt->dev, i915_tt->cached_st,
+ DMA_BIDIRECTIONAL, 0);
+ sg_free_table(i915_tt->cached_st);
+ kfree(i915_tt->cached_st);
+ i915_tt->cached_st = NULL;
+ }
+ ttm_pool_free(&bdev->pool, ttm);
+}
+
+static void i915_ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+
+ ttm_tt_destroy_common(bdev, ttm);
+ kfree(i915_tt);
+}
+
+static bool i915_ttm_eviction_valuable(struct ttm_buffer_object *bo,
+ const struct ttm_place *place)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ /* Will do for now. Our pinned objects are still on TTM's LRU lists */
+ if (!i915_gem_object_evictable(obj))
+ return false;
+
+ /* This isn't valid with a buddy allocator */
+ return ttm_bo_eviction_valuable(bo, place);
+}
+
+static void i915_ttm_evict_flags(struct ttm_buffer_object *bo,
+ struct ttm_placement *placement)
+{
+ *placement = i915_sys_placement;
+}
+
+static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ int ret;
+
+ ret = i915_gem_object_unbind(obj, I915_GEM_OBJECT_UNBIND_ACTIVE);
+ if (ret)
+ return ret;
+
+ ret = __i915_gem_object_put_pages(obj);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
+{
+ if (obj->ttm.cached_io_st) {
+ sg_free_table(obj->ttm.cached_io_st);
+ kfree(obj->ttm.cached_io_st);
+ obj->ttm.cached_io_st = NULL;
+ }
+}
+
+static void i915_ttm_purge(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_operation_ctx ctx = {
+ .interruptible = true,
+ .no_wait_gpu = false,
+ };
+ struct ttm_placement place = {};
+ int ret;
+
+ if (obj->mm.madv == __I915_MADV_PURGED)
+ return;
+
+ /* TTM's purge interface. Note that we might be reentering. */
+ ret = ttm_bo_validate(bo, &place, &ctx);
+
+ if (!ret) {
+ i915_ttm_free_cached_io_st(obj);
+ obj->mm.madv = __I915_MADV_PURGED;
+ }
+}
+
+static void i915_ttm_swap_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ int ret = i915_ttm_move_notify(bo);
+
+ GEM_WARN_ON(ret);
+ GEM_WARN_ON(obj->ttm.cached_io_st);
+ if (!ret && obj->mm.madv != I915_MADV_WILLNEED)
+ i915_ttm_purge(obj);
+}
+
+static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ if (likely(obj)) {
+ /* This releases all gem object bindings to the backend. */
+ __i915_gem_free_object(obj);
+ }
+}
+
+static struct intel_memory_region *
+i915_ttm_region(struct ttm_device *bdev, int ttm_mem_type)
+{
+ struct drm_i915_private *i915 = container_of(bdev, typeof(*i915), bdev);
+
+ /* There's some room for optimization here... */
+ GEM_BUG_ON(ttm_mem_type != I915_PL_SYSTEM &&
+ ttm_mem_type < I915_PL_LMEM0);
+ if (ttm_mem_type == I915_PL_SYSTEM)
+ return intel_memory_region_lookup(i915, INTEL_MEMORY_SYSTEM,
+ 0);
+
+ return intel_memory_region_lookup(i915, INTEL_MEMORY_LOCAL,
+ ttm_mem_type - I915_PL_LMEM0);
+}
+
+static struct sg_table *i915_ttm_tt_get_st(struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+ struct scatterlist *sg;
+ struct sg_table *st;
+ int ret;
+
+ if (i915_tt->cached_st)
+ return i915_tt->cached_st;
+
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return ERR_PTR(-ENOMEM);
+
+ sg = __sg_alloc_table_from_pages
+ (st, ttm->pages, ttm->num_pages, 0,
+ (unsigned long)ttm->num_pages << PAGE_SHIFT,
+ i915_sg_segment_size(), NULL, 0, GFP_KERNEL);
+ if (IS_ERR(sg)) {
+ kfree(st);
+ return ERR_CAST(sg);
+ }
+
+ ret = dma_map_sgtable(i915_tt->dev, st, DMA_BIDIRECTIONAL, 0);
+ if (ret) {
+ sg_free_table(st);
+ kfree(st);
+ return ERR_PTR(ret);
+ }
+
+ i915_tt->cached_st = st;
+ return st;
+}
+
+static struct sg_table *
+i915_ttm_resource_get_st(struct drm_i915_gem_object *obj,
+ struct ttm_resource *res)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_resource_manager *man =
+ ttm_manager_type(bo->bdev, res->mem_type);
+
+ if (man->use_tt)
+ return i915_ttm_tt_get_st(bo->ttm);
+
+ return intel_region_ttm_node_to_st(obj->mm.region, res->mm_node);
+}
+
+static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
+ struct ttm_operation_ctx *ctx,
+ struct ttm_resource *dst_mem,
+ struct ttm_place *hop)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ struct ttm_resource_manager *dst_man =
+ ttm_manager_type(bo->bdev, dst_mem->mem_type);
+ struct ttm_resource_manager *src_man =
+ ttm_manager_type(bo->bdev, bo->mem.mem_type);
+ struct intel_memory_region *dst_reg, *src_reg;
+ union {
+ struct ttm_kmap_iter_tt tt;
+ struct ttm_kmap_iter_iomap io;
+ } _dst_iter, _src_iter;
+ struct ttm_kmap_iter *dst_iter, *src_iter;
+ struct sg_table *dst_st;
+ int ret;
+
+ dst_reg = i915_ttm_region(bo->bdev, dst_mem->mem_type);
+ src_reg = i915_ttm_region(bo->bdev, bo->mem.mem_type);
+ GEM_BUG_ON(!dst_reg || !src_reg);
+
+ /* Sync for now. We could do the actual copy async. */
+ ret = ttm_bo_wait_ctx(bo, ctx);
+ if (ret)
+ return ret;
+
+ ret = i915_ttm_move_notify(bo);
+ if (ret)
+ return ret;
+
+ if (obj->mm.madv != I915_MADV_WILLNEED) {
+ i915_ttm_purge(obj);
+ ttm_resource_free(bo, dst_mem);
+ return 0;
+ }
+
+ /* Populate ttm with pages if needed. Typically system memory. */
+ if (bo->ttm && (dst_man->use_tt ||
+ (bo->ttm->page_flags & TTM_PAGE_FLAG_SWAPPED))) {
+ ret = ttm_tt_populate(bo->bdev, bo->ttm, ctx);
+ if (ret)
+ return ret;
+ }
+
+ dst_st = i915_ttm_resource_get_st(obj, dst_mem);
+ if (IS_ERR(dst_st))
+ return PTR_ERR(dst_st);
+
+ /* If we start mapping GGTT, we can no longer use man::use_tt here. */
+ dst_iter = dst_man->use_tt ?
+ ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) :
+ ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
+ dst_st, dst_reg->region.start);
+
+ src_iter = src_man->use_tt ?
+ ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) :
+ ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
+ obj->ttm.cached_io_st,
+ src_reg->region.start);
+
+ ttm_move_memcpy(bo, dst_mem->num_pages, dst_iter, src_iter);
+ ttm_bo_move_sync_cleanup(bo, dst_mem);
+ i915_ttm_free_cached_io_st(obj);
+
+ if (!dst_man->use_tt)
+ obj->ttm.cached_io_st = dst_st;
+
+ return 0;
+}
+
+static struct ttm_device_funcs i915_ttm_bo_driver = {
+ .ttm_tt_create = i915_ttm_tt_create,
+ .ttm_tt_unpopulate = i915_ttm_tt_unpopulate,
+ .ttm_tt_destroy = i915_ttm_tt_destroy,
+ .eviction_valuable = i915_ttm_eviction_valuable,
+ .evict_flags = i915_ttm_evict_flags,
+ .move = i915_ttm_move,
+ .verify_access = NULL,
+ .swap_notify = i915_ttm_swap_notify,
+ .delete_mem_notify = i915_ttm_delete_mem_notify,
+};
+
+/**
+ * i915_ttm_driver - Return a pointer to the TTM device funcs
+ *
+ * Return: Pointer to statically allocated TTM device funcs.
+ */
+struct ttm_device_funcs *i915_ttm_driver(void)
+{
+ return &i915_ttm_bo_driver;
+}
+
+static int i915_ttm_get_pages(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_operation_ctx ctx = {
+ .interruptible = true,
+ .no_wait_gpu = false,
+ };
+ struct sg_table *st;
+ int ret;
+
+ /* Move to the requested placement. */
+ ret = ttm_bo_validate(bo, &i915_lmem0_placement, &ctx);
+ if (ret)
+ return ret == -ENOSPC ? -ENXIO : ret;
+
+ /* Object either has a page vector or is an iomem object */
+ st = bo->ttm ? i915_ttm_tt_get_st(bo->ttm) : obj->ttm.cached_io_st;
+ if (IS_ERR(st))
+ return PTR_ERR(st);
+
+ __i915_gem_object_set_pages(obj, st, i915_sg_dma_sizes(st->sgl));
+
+ i915_ttm_adjust_lru(obj);
+
+ return ret;
+}
+
+static void i915_ttm_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *st)
+{
+ /*
+ * We're currently not called from a shrinker, so put_pages()
+ * typically means the object is about to destroyed, or called
+ * from move_notify(). So just avoid doing much for now.
+ * If the object is not destroyed next, The TTM eviction logic
+ * and shrinkers will move it out if needed.
+ */
+
+ i915_ttm_adjust_lru(obj);
+}
+
+static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+
+ /*
+ * Don't manipulate the TTM LRUs while in TTM bo destruction.
+ * We're called through i915_ttm_delete_mem_notify().
+ */
+ if (!kref_read(&bo->kref))
+ return;
+
+ /*
+ * Put on the correct LRU list depending on the MADV status
+ */
+ spin_lock(&bo->bdev->lru_lock);
+ if (obj->mm.madv != I915_MADV_WILLNEED) {
+ bo->priority = I915_TTM_PRIO_PURGE;
+ } else if (!i915_gem_object_has_pages(obj)) {
+ if (bo->priority < I915_TTM_PRIO_HAS_PAGES)
+ bo->priority = I915_TTM_PRIO_HAS_PAGES;
+ } else {
+ if (bo->priority > I915_TTM_PRIO_NO_PAGES)
+ bo->priority = I915_TTM_PRIO_NO_PAGES;
+ }
+
+ ttm_bo_move_to_lru_tail(bo, &bo->mem, NULL);
+ spin_unlock(&bo->bdev->lru_lock);
+}
+
+/*
+ * TTM-backed gem object destruction requires some clarification.
+ * Basically we have two possibilities here. We can either rely on the
+ * i915 delayed destruction and put the TTM object when the object
+ * is idle. This would be detected by TTM which would bypass the
+ * TTM delayed destroy handling. The other approach is to put the TTM
+ * object early and rely on the TTM destroyed handling, and then free
+ * the leftover parts of the GEM object once TTM's destroyed list handling is
+ * complete. For now, we rely on the latter for two reasons:
+ * a) TTM can evict an object even when it's on the delayed destroy list,
+ * which in theory allows for complete eviction.
+ * b) There is work going on in TTM to allow freeing an object even when
+ * it's not idle, and using the TTM destroyed list handling could help us
+ * benefit from that.
+ */
+static void i915_ttm_delayed_free(struct drm_i915_gem_object *obj)
+{
+ if (obj->ttm.created) {
+ ttm_bo_put(i915_gem_to_ttm(obj));
+ } else {
+ __i915_gem_free_object(obj);
+ call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
+ }
+}
+
+static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
+ .name = "i915_gem_object_ttm",
+ .flags = I915_GEM_OBJECT_HAS_IOMEM,
+
+ .get_pages = i915_ttm_get_pages,
+ .put_pages = i915_ttm_put_pages,
+ .truncate = i915_ttm_purge,
+ .adjust_lru = i915_ttm_adjust_lru,
+ .delayed_free = i915_ttm_delayed_free,
+};
+
+void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ i915_gem_object_release_memory_region(obj);
+ if (obj->ttm.created)
+ call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
+}
+
+/**
+ * __i915_gem_ttm_object_init - Initialize a ttm-backed i915 gem object
+ * @mem: The initial memory region for the object.
+ * @obj: The gem object.
+ * @size: Object size in bytes.
+ * @flags: gem object flags.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
+ struct drm_i915_gem_object *obj,
+ resource_size_t size,
+ unsigned int flags)
+{
+ static struct lock_class_key lock_class;
+ struct drm_i915_private *i915 = mem->i915;
+ enum ttm_bo_type bo_type;
+ size_t alignment = 0;
+ int ret;
+
+ /* Adjust alignment to GPU- and CPU huge page sizes. */
+
+ if (mem->is_range_manager) {
+ if (size >= SZ_1G)
+ alignment = SZ_1G >> PAGE_SHIFT;
+ else if (size >= SZ_2M)
+ alignment = SZ_2M >> PAGE_SHIFT;
+ else if (size >= SZ_64K)
+ alignment = SZ_64K >> PAGE_SHIFT;
+ }
+
+ drm_gem_private_object_init(&i915->drm, &obj->base, size);
+ i915_gem_object_init(obj, &i915_gem_ttm_obj_ops, &lock_class, flags);
+ i915_gem_object_init_memory_region(obj, mem);
+ i915_gem_object_make_unshrinkable(obj);
+ obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
+
+ bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device :
+ ttm_bo_type_kernel;
+
+ /*
+ * If this function fails, it will call the destructor, but
+ * our caller still owns the object. So no freeing in the
+ * destructor until obj->ttm.created is true.
+ * Similarly, in delayed_destroy, we can't call ttm_bo_put()
+ * until successful initialization.
+ */
+ ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size,
+ bo_type, &i915_sys_placement, alignment,
+ true, NULL, NULL, i915_ttm_bo_destroy);
+
+ if (!ret)
+ obj->ttm.created = true;
+
+ /* i915 wants -ENXIO when out of memory region space. */
+ return (ret == -ENOSPC) ? -ENXIO : ret;
+}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.h b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h
new file mode 100644
index 000000000000..b8d3dcbb50df
--- /dev/null
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef _I915_GEM_TTM_H_
+#define _I915_GEM_TTM_H_
+
+#include "gem/i915_gem_object_types.h"
+
+/**
+ * i915_gem_to_ttm - Convert a struct drm_i915_gem_object to a
+ * struct ttm_buffer_object.
+ * @obj: Pointer to the gem object.
+ *
+ * Return: Pointer to the embedded struct ttm_buffer_object.
+ */
+static inline struct ttm_buffer_object *
+i915_gem_to_ttm(struct drm_i915_gem_object *obj)
+{
+ return &obj->__do_not_access;
+}
+
+/*
+ * i915 ttm gem object destructor. Internal use only.
+ */
+void i915_ttm_bo_destroy(struct ttm_buffer_object *bo);
+
+/**
+ * i915_ttm_to_gem - Convert a struct ttm_buffer_object to an embedding
+ * struct drm_i915_gem_object.
+ *
+ * Return: Pointer to the embedding struct ttm_buffer_object, or NULL
+ * if the object was not an i915 ttm object.
+ */
+static inline struct drm_i915_gem_object *
+i915_ttm_to_gem(struct ttm_buffer_object *bo)
+{
+ if (GEM_WARN_ON(bo->destroy != i915_ttm_bo_destroy))
+ return NULL;
+
+ return container_of(bo, struct drm_i915_gem_object, __do_not_access);
+}
+
+int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
+ struct drm_i915_gem_object *obj,
+ resource_size_t size,
+ unsigned int flags);
+#endif
diff --git a/drivers/gpu/drm/i915/gt/intel_region_lmem.c b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
index f7366b054f8e..4ae1f717a94c 100644
--- a/drivers/gpu/drm/i915/gt/intel_region_lmem.c
+++ b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
@@ -9,6 +9,7 @@
#include "intel_region_ttm.h"
#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
+#include "gem/i915_gem_ttm.h"
#include "intel_region_lmem.h"
static int init_fake_lmem_bar(struct intel_memory_region *mem)
@@ -107,7 +108,7 @@ region_lmem_init(struct intel_memory_region *mem)
static const struct intel_memory_region_ops intel_region_lmem_ops = {
.init = region_lmem_init,
.release = region_lmem_release,
- .init_object = __i915_gem_lmem_object_init,
+ .init_object = __i915_gem_ttm_object_init,
};
struct intel_memory_region *
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 0993d706f067..40fe9a147318 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1005,8 +1005,11 @@ i915_gem_madvise_ioctl(struct drm_device *dev, void *data,
}
}
- if (obj->mm.madv != __I915_MADV_PURGED)
+ if (obj->mm.madv != __I915_MADV_PURGED) {
obj->mm.madv = args->madv;
+ if (obj->ops->adjust_lru)
+ obj->ops->adjust_lru(obj);
+ }
if (i915_gem_object_has_pages(obj)) {
unsigned long flags;
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index 4092cc987679..bd27e897d4d0 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -149,7 +149,6 @@ intel_memory_region_create(struct drm_i915_private *i915,
mutex_init(&mem->objects.lock);
INIT_LIST_HEAD(&mem->objects.list);
- INIT_LIST_HEAD(&mem->objects.purgeable);
INIT_LIST_HEAD(&mem->reserved);
mutex_init(&mem->mm_lock);
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index e69cde13daf2..7b5fa97c0b59 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -100,7 +100,6 @@ struct intel_memory_region {
struct {
struct mutex lock; /* Protects access to objects */
struct list_head list;
- struct list_head purgeable;
} objects;
size_t chunk_size;
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
index c8ac118c21f6..0b41a1545570 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.c
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -10,6 +10,7 @@
#include "intel_region_ttm.h"
+#include "gem/i915_gem_ttm.h" /* For the funcs/ops export only */
/**
* DOC: TTM support structure
*
@@ -19,9 +20,6 @@
* i915 GEM regions to TTM memory types and resource managers.
*/
-/* A Zero-initialized driver for now. We don't have a TTM backend yet. */
-static struct ttm_device_funcs i915_ttm_bo_driver;
-
/**
* intel_region_ttm_device_init - Initialize a TTM device
* @dev_priv: Pointer to an i915 device private structure.
@@ -32,7 +30,7 @@ int intel_region_ttm_device_init(struct drm_i915_private *dev_priv)
{
struct drm_device *drm = &dev_priv->drm;
- return ttm_device_init(&dev_priv->bdev, &i915_ttm_bo_driver,
+ return ttm_device_init(&dev_priv->bdev, i915_ttm_driver(),
drm->dev, drm->anon_inode->i_mapping,
drm->vma_offset_manager, false, false);
}
@@ -172,6 +170,7 @@ struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
return i915_sg_from_mm_node(node, mem->region.start);
}
+#ifdef CONFIG_DRM_I915_SELFTEST
/**
* intel_region_ttm_node_alloc - Allocate memory resources from a region
* @mem: The memory region,
@@ -218,3 +217,4 @@ void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
ret = -ENXIO;
return ret ? ERR_PTR(ret) : res.mm_node;
}
+#endif
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.h b/drivers/gpu/drm/i915/intel_region_ttm.h
index 1c82c6c3429d..eaa3eccfa252 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.h
+++ b/drivers/gpu/drm/i915/intel_region_ttm.h
@@ -11,6 +11,7 @@
struct drm_i915_private;
struct intel_memory_region;
+struct ttm_device_funcs;
int intel_region_ttm_device_init(struct drm_i915_private *dev_priv);
@@ -23,10 +24,14 @@ void intel_region_ttm_fini(struct intel_memory_region *mem);
struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
void *node);
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node);
+
+struct ttm_device_funcs *i915_ttm_driver(void);
+
+#ifdef CONFIG_DRM_I915_SELFTEST
void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
resource_size_t size,
unsigned int flags);
-
-void intel_region_ttm_node_free(struct intel_memory_region *mem,
- void *node);
+#endif
#endif /* _INTEL_REGION_TTM_H_ */
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 11/15] drm/i915/ttm: Introduce a TTM i915 gem object backend
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Most logical place to introduce TTM buffer objects is as an i915
gem object backend. We need to add some ops to account for added
functionality like delayed delete and LRU list manipulation.
Initially we support only LMEM and SYSTEM memory, but SYSTEM
(which in this case means evicted LMEM objects) is not
visible to i915 GEM yet. The plan is to move the i915 gem system region
over to the TTM system memory type in upcoming patches.
We set up GPU bindings directly both from LMEM and from the system region,
as there is no need to use the legacy TTM_TT memory type. We reserve
that for future porting of GGTT bindings to TTM.
Remove the old lmem backend.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v2:
- Break out needed TTM functionality to a separate patch (Reported by
Christian König).
- Fix an unhandled error (Reported by Matthew Auld and Maarten Lankhorst)
- Remove a stray leftover sg_table allocation (Reported by Matthew Auld)
- Use ttm_tt_unpopulate() rather than ttm_tt_destroy() in the purge path
as some TTM functionality relies on having a ttm_tt present for !is_iomem.
v3:
- Use ttm_bo_type_device for userspace visible objects so that TTM can
allocate an address space offset for mmap'ing.
- Fix up the destruction path (Reported by Matthew Auld)
- Use ttm_bo_validate() for purging (Reported by Christian König)
- Create ttm_tts write-combined as they are currently for eviction only and
we want to maintain consistent write-combined caching for bos that are
not in system only. (Suggested by Daniel Vetter)
- Make struct ttm_placements static.
- Add the ttm device funcs/ops to i915_gem_ttm.h for the region code.
- Rename new->dst and old->src. Check for swapin in the move callback.
v4:
- Adapt to small interface change in ttm_move_memcpy.
- Use a function to pull out the ttm driver from the backend.
---
drivers/gpu/drm/i915/Makefile | 1 +
drivers/gpu/drm/i915/gem/i915_gem_create.c | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 84 ---
drivers/gpu/drm/i915/gem/i915_gem_lmem.h | 5 -
drivers/gpu/drm/i915/gem/i915_gem_object.c | 125 ++--
drivers/gpu/drm/i915/gem/i915_gem_object.h | 9 +
.../gpu/drm/i915/gem/i915_gem_object_types.h | 27 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 541 ++++++++++++++++++
drivers/gpu/drm/i915/gem/i915_gem_ttm.h | 48 ++
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 3 +-
drivers/gpu/drm/i915/i915_gem.c | 5 +-
drivers/gpu/drm/i915/intel_memory_region.c | 1 -
drivers/gpu/drm/i915/intel_memory_region.h | 1 -
drivers/gpu/drm/i915/intel_region_ttm.c | 8 +-
drivers/gpu/drm/i915/intel_region_ttm.h | 11 +-
16 files changed, 731 insertions(+), 153 deletions(-)
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.c
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.h
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index ebc19bd5fff4..9f9cd5c085c3 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -154,6 +154,7 @@ gem-y += \
gem/i915_gem_stolen.o \
gem/i915_gem_throttle.o \
gem/i915_gem_tiling.o \
+ gem/i915_gem_ttm.o \
gem/i915_gem_userptr.o \
gem/i915_gem_wait.o \
gem/i915_gemfs.o
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_create.c b/drivers/gpu/drm/i915/gem/i915_gem_create.c
index 548ddf39d853..93bf63bbaff1 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_create.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_create.c
@@ -85,13 +85,10 @@ i915_gem_setup(struct drm_i915_gem_object *obj, u64 size)
return -E2BIG;
/*
- * For now resort to CPU based clearing for device local-memory, in the
- * near future this will use the blitter engine for accelerated, GPU
- * based clearing.
+ * I915_BO_ALLOC_USER will make sure the object is cleared before
+ * any user access.
*/
- flags = 0;
- if (mr->type == INTEL_MEMORY_LOCAL)
- flags = I915_BO_ALLOC_CPU_CLEAR;
+ flags = I915_BO_ALLOC_USER;
ret = mr->ops->init_object(mr, obj, size, flags);
if (ret)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 3b4aa28a076d..2b8cd15de1d9 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -4,74 +4,10 @@
*/
#include "intel_memory_region.h"
-#include "intel_region_ttm.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_lmem.h"
#include "i915_drv.h"
-static void lmem_put_pages(struct drm_i915_gem_object *obj,
- struct sg_table *pages)
-{
- intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
- obj->mm.dirty = false;
- sg_free_table(pages);
- kfree(pages);
-}
-
-static int lmem_get_pages(struct drm_i915_gem_object *obj)
-{
- unsigned int flags;
- struct sg_table *pages;
-
- flags = I915_ALLOC_MIN_PAGE_SIZE;
- if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
- flags |= I915_ALLOC_CONTIGUOUS;
-
- obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
- obj->base.size,
- flags);
- if (IS_ERR(obj->mm.st_mm_node))
- return PTR_ERR(obj->mm.st_mm_node);
-
- /* Range manager is always contigous */
- if (obj->mm.region->is_range_manager)
- obj->flags |= I915_BO_ALLOC_CONTIGUOUS;
- pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
- if (IS_ERR(pages)) {
- intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
- return PTR_ERR(pages);
- }
-
- __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
-
- if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
- void __iomem *vaddr =
- i915_gem_object_lmem_io_map(obj, 0, obj->base.size);
-
- if (!vaddr) {
- struct sg_table *pages =
- __i915_gem_object_unset_pages(obj);
-
- if (!IS_ERR_OR_NULL(pages))
- lmem_put_pages(obj, pages);
- }
-
- memset_io(vaddr, 0, obj->base.size);
- io_mapping_unmap(vaddr);
- }
-
- return 0;
-}
-
-const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
- .name = "i915_gem_object_lmem",
- .flags = I915_GEM_OBJECT_HAS_IOMEM,
-
- .get_pages = lmem_get_pages,
- .put_pages = lmem_put_pages,
- .release = i915_gem_object_release_memory_region,
-};
-
void __iomem *
i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
unsigned long n,
@@ -103,23 +39,3 @@ i915_gem_object_create_lmem(struct drm_i915_private *i915,
return i915_gem_object_create_region(i915->mm.regions[INTEL_REGION_LMEM],
size, flags);
}
-
-int __i915_gem_lmem_object_init(struct intel_memory_region *mem,
- struct drm_i915_gem_object *obj,
- resource_size_t size,
- unsigned int flags)
-{
- static struct lock_class_key lock_class;
- struct drm_i915_private *i915 = mem->i915;
-
- drm_gem_private_object_init(&i915->drm, &obj->base, size);
- i915_gem_object_init(obj, &i915_gem_lmem_obj_ops, &lock_class, flags);
-
- obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
-
- i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
-
- i915_gem_object_init_memory_region(obj, mem);
-
- return 0;
-}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
index fac6bc5a5ebb..ea76fd11ccb0 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
@@ -26,9 +26,4 @@ i915_gem_object_create_lmem(struct drm_i915_private *i915,
resource_size_t size,
unsigned int flags);
-int __i915_gem_lmem_object_init(struct intel_memory_region *mem,
- struct drm_i915_gem_object *obj,
- resource_size_t size,
- unsigned int flags);
-
#endif /* !__I915_GEM_LMEM_H */
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index e9247afb0320..df2b4e6b9bcc 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -173,7 +173,7 @@ static void i915_gem_close_object(struct drm_gem_object *gem, struct drm_file *f
}
}
-static void __i915_gem_free_object_rcu(struct rcu_head *head)
+void __i915_gem_free_object_rcu(struct rcu_head *head)
{
struct drm_i915_gem_object *obj =
container_of(head, typeof(*obj), rcu);
@@ -209,59 +209,69 @@ static void __i915_gem_object_free_mmaps(struct drm_i915_gem_object *obj)
}
}
-static void __i915_gem_free_objects(struct drm_i915_private *i915,
- struct llist_node *freed)
+void __i915_gem_free_object(struct drm_i915_gem_object *obj)
{
- struct drm_i915_gem_object *obj, *on;
+ trace_i915_gem_object_destroy(obj);
- llist_for_each_entry_safe(obj, on, freed, freed) {
- trace_i915_gem_object_destroy(obj);
+ if (!list_empty(&obj->vma.list)) {
+ struct i915_vma *vma;
+
+ /*
+ * Note that the vma keeps an object reference while
+ * it is active, so it *should* not sleep while we
+ * destroy it. Our debug code errs insits it *might*.
+ * For the moment, play along.
+ */
+ spin_lock(&obj->vma.lock);
+ while ((vma = list_first_entry_or_null(&obj->vma.list,
+ struct i915_vma,
+ obj_link))) {
+ GEM_BUG_ON(vma->obj != obj);
+ spin_unlock(&obj->vma.lock);
- if (!list_empty(&obj->vma.list)) {
- struct i915_vma *vma;
+ __i915_vma_put(vma);
- /*
- * Note that the vma keeps an object reference while
- * it is active, so it *should* not sleep while we
- * destroy it. Our debug code errs insits it *might*.
- * For the moment, play along.
- */
spin_lock(&obj->vma.lock);
- while ((vma = list_first_entry_or_null(&obj->vma.list,
- struct i915_vma,
- obj_link))) {
- GEM_BUG_ON(vma->obj != obj);
- spin_unlock(&obj->vma.lock);
+ }
+ spin_unlock(&obj->vma.lock);
+ }
- __i915_vma_put(vma);
+ __i915_gem_object_free_mmaps(obj);
- spin_lock(&obj->vma.lock);
- }
- spin_unlock(&obj->vma.lock);
- }
+ GEM_BUG_ON(!list_empty(&obj->lut_list));
- __i915_gem_object_free_mmaps(obj);
+ atomic_set(&obj->mm.pages_pin_count, 0);
+ __i915_gem_object_put_pages(obj);
+ GEM_BUG_ON(i915_gem_object_has_pages(obj));
+ bitmap_free(obj->bit_17);
- GEM_BUG_ON(!list_empty(&obj->lut_list));
+ if (obj->base.import_attach)
+ drm_prime_gem_destroy(&obj->base, NULL);
- atomic_set(&obj->mm.pages_pin_count, 0);
- __i915_gem_object_put_pages(obj);
- GEM_BUG_ON(i915_gem_object_has_pages(obj));
- bitmap_free(obj->bit_17);
+ drm_gem_free_mmap_offset(&obj->base);
- if (obj->base.import_attach)
- drm_prime_gem_destroy(&obj->base, NULL);
+ if (obj->ops->release)
+ obj->ops->release(obj);
- drm_gem_free_mmap_offset(&obj->base);
+ if (obj->mm.n_placements > 1)
+ kfree(obj->mm.placements);
- if (obj->ops->release)
- obj->ops->release(obj);
+ if (obj->shares_resv_from)
+ i915_vm_resv_put(obj->shares_resv_from);
+}
- if (obj->mm.n_placements > 1)
- kfree(obj->mm.placements);
+static void __i915_gem_free_objects(struct drm_i915_private *i915,
+ struct llist_node *freed)
+{
+ struct drm_i915_gem_object *obj, *on;
- if (obj->shares_resv_from)
- i915_vm_resv_put(obj->shares_resv_from);
+ llist_for_each_entry_safe(obj, on, freed, freed) {
+ might_sleep();
+ if (obj->ops->delayed_free) {
+ obj->ops->delayed_free(obj);
+ continue;
+ }
+ __i915_gem_free_object(obj);
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
@@ -319,6 +329,7 @@ static void i915_gem_free_object(struct drm_gem_object *gem_obj)
* worker and performing frees directly from subsequent allocations for
* crude but effective memory throttling.
*/
+
if (llist_add(&obj->freed, &i915->mm.free_list))
queue_work(i915->wq, &i915->mm.free_work);
}
@@ -411,6 +422,42 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset,
return 0;
}
+/**
+ * i915_gem_object_evictable - Whether object is likely evictable after unbind.
+ * @obj: The object to check
+ *
+ * This function checks whether the object is likely unvictable after unbind.
+ * If the object is not locked when checking, the result is only advisory.
+ * If the object is locked when checking, and the function returns true,
+ * then an eviction should indeed be possible. But since unlocked vma
+ * unpinning and unbinding is currently possible, the object can actually
+ * become evictable even if this function returns false.
+ *
+ * Return: true if the object may be evictable. False otherwise.
+ */
+bool i915_gem_object_evictable(struct drm_i915_gem_object *obj)
+{
+ struct i915_vma *vma;
+ int pin_count = atomic_read(&obj->mm.pages_pin_count);
+
+ if (!pin_count)
+ return true;
+
+ spin_lock(&obj->vma.lock);
+ list_for_each_entry(vma, &obj->vma.list, obj_link) {
+ if (i915_vma_is_pinned(vma)) {
+ spin_unlock(&obj->vma.lock);
+ return false;
+ }
+ if (atomic_read(&vma->pages_count))
+ pin_count--;
+ }
+ spin_unlock(&obj->vma.lock);
+ GEM_WARN_ON(pin_count < 0);
+
+ return pin_count == 0;
+}
+
void i915_gem_init__objects(struct drm_i915_private *i915)
{
INIT_WORK(&i915->mm.free_work, __i915_gem_free_work);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index 2ebd79537aea..ae5930e307d5 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -200,6 +200,9 @@ static inline bool i915_gem_object_trylock(struct drm_i915_gem_object *obj)
static inline void i915_gem_object_unlock(struct drm_i915_gem_object *obj)
{
+ if (obj->ops->adjust_lru)
+ obj->ops->adjust_lru(obj);
+
dma_resv_unlock(obj->base.resv);
}
@@ -587,6 +590,12 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset,
bool i915_gem_object_is_shmem(const struct drm_i915_gem_object *obj);
+void __i915_gem_free_object_rcu(struct rcu_head *head);
+
+void __i915_gem_free_object(struct drm_i915_gem_object *obj);
+
+bool i915_gem_object_evictable(struct drm_i915_gem_object *obj);
+
#ifdef CONFIG_MMU_NOTIFIER
static inline bool
i915_gem_object_is_userptr(struct drm_i915_gem_object *obj)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index d047ea126029..68313474e6a6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -63,6 +63,20 @@ struct drm_i915_gem_object_ops {
const struct drm_i915_gem_pwrite *arg);
int (*dmabuf_export)(struct drm_i915_gem_object *obj);
+
+ /**
+ * adjust_lru - notify that the madvise value was updated
+ * @obj: The gem object
+ *
+ * The madvise value may have been updated, or object was recently
+ * referenced so act accordingly (Perhaps changing an LRU list etc).
+ */
+ void (*adjust_lru)(struct drm_i915_gem_object *obj);
+
+ /**
+ * delayed_free - Override the default delayed free implementation
+ */
+ void (*delayed_free)(struct drm_i915_gem_object *obj);
void (*release)(struct drm_i915_gem_object *obj);
const char *name; /* friendly name for debug, e.g. lockdep classes */
@@ -187,12 +201,14 @@ struct drm_i915_gem_object {
#define I915_BO_ALLOC_VOLATILE BIT(1)
#define I915_BO_ALLOC_STRUCT_PAGE BIT(2)
#define I915_BO_ALLOC_CPU_CLEAR BIT(3)
+#define I915_BO_ALLOC_USER BIT(4)
#define I915_BO_ALLOC_FLAGS (I915_BO_ALLOC_CONTIGUOUS | \
I915_BO_ALLOC_VOLATILE | \
I915_BO_ALLOC_STRUCT_PAGE | \
- I915_BO_ALLOC_CPU_CLEAR)
-#define I915_BO_READONLY BIT(4)
-#define I915_TILING_QUIRK_BIT 5 /* unknown swizzling; do not release! */
+ I915_BO_ALLOC_CPU_CLEAR | \
+ I915_BO_ALLOC_USER)
+#define I915_BO_READONLY BIT(5)
+#define I915_TILING_QUIRK_BIT 6 /* unknown swizzling; do not release! */
/*
* Is the object to be mapped as read-only to the GPU
@@ -310,6 +326,11 @@ struct drm_i915_gem_object {
bool dirty:1;
} mm;
+ struct {
+ struct sg_table *cached_io_st;
+ bool created:1;
+ } ttm;
+
/** Record of address bit 17 of each page at last unbind. */
unsigned long *bit_17;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.c b/drivers/gpu/drm/i915/gem/i915_gem_region.c
index f25e6646c5b7..d1f1840540dd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.c
@@ -18,11 +18,7 @@ void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
mutex_lock(&mem->objects.lock);
- if (obj->flags & I915_BO_ALLOC_VOLATILE)
- list_add(&obj->mm.region_link, &mem->objects.purgeable);
- else
- list_add(&obj->mm.region_link, &mem->objects.list);
-
+ list_add(&obj->mm.region_link, &mem->objects.list);
mutex_unlock(&mem->objects.lock);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
new file mode 100644
index 000000000000..17598930a99e
--- /dev/null
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -0,0 +1,541 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_placement.h>
+
+#include "i915_drv.h"
+#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
+
+#include "gem/i915_gem_object.h"
+#include "gem/i915_gem_region.h"
+#include "gem/i915_gem_ttm.h"
+
+#define I915_PL_LMEM0 TTM_PL_PRIV
+#define I915_PL_SYSTEM TTM_PL_SYSTEM
+#define I915_PL_STOLEN TTM_PL_VRAM
+#define I915_PL_GGTT TTM_PL_TT
+
+#define I915_TTM_PRIO_PURGE 0
+#define I915_TTM_PRIO_NO_PAGES 1
+#define I915_TTM_PRIO_HAS_PAGES 2
+
+/**
+ * struct i915_ttm_tt - TTM page vector with additional private information
+ * @ttm: The base TTM page vector.
+ * @dev: The struct device used for dma mapping and unmapping.
+ * @cached_st: The cached scatter-gather table.
+ *
+ * Note that DMA may be going on right up to the point where the page-
+ * vector is unpopulated in delayed destroy. Hence keep the
+ * scatter-gather table mapped and cached up to that point. This is
+ * different from the cached gem object io scatter-gather table which
+ * doesn't have an associated dma mapping.
+ */
+struct i915_ttm_tt {
+ struct ttm_tt ttm;
+ struct device *dev;
+ struct sg_table *cached_st;
+};
+
+static const struct ttm_place lmem0_sys_placement_flags[] = {
+ {
+ .fpfn = 0,
+ .lpfn = 0,
+ .mem_type = I915_PL_LMEM0,
+ .flags = 0,
+ }, {
+ .fpfn = 0,
+ .lpfn = 0,
+ .mem_type = I915_PL_SYSTEM,
+ .flags = 0,
+ }
+};
+
+static struct ttm_placement i915_lmem0_placement = {
+ .num_placement = 1,
+ .placement = &lmem0_sys_placement_flags[0],
+ .num_busy_placement = 1,
+ .busy_placement = &lmem0_sys_placement_flags[0],
+};
+
+static struct ttm_placement i915_sys_placement = {
+ .num_placement = 1,
+ .placement = &lmem0_sys_placement_flags[1],
+ .num_busy_placement = 1,
+ .busy_placement = &lmem0_sys_placement_flags[1],
+};
+
+static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj);
+
+static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo,
+ uint32_t page_flags)
+{
+ struct ttm_resource_manager *man =
+ ttm_manager_type(bo->bdev, bo->mem.mem_type);
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ struct i915_ttm_tt *i915_tt;
+ int ret;
+
+ i915_tt = kzalloc(sizeof(*i915_tt), GFP_KERNEL);
+ if (!i915_tt)
+ return NULL;
+
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR &&
+ man->use_tt)
+ page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
+
+ ret = ttm_tt_init(&i915_tt->ttm, bo, page_flags, ttm_write_combined);
+ if (ret) {
+ kfree(i915_tt);
+ return NULL;
+ }
+
+ i915_tt->dev = obj->base.dev->dev;
+
+ return &i915_tt->ttm;
+}
+
+static void i915_ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+
+ if (i915_tt->cached_st) {
+ dma_unmap_sgtable(i915_tt->dev, i915_tt->cached_st,
+ DMA_BIDIRECTIONAL, 0);
+ sg_free_table(i915_tt->cached_st);
+ kfree(i915_tt->cached_st);
+ i915_tt->cached_st = NULL;
+ }
+ ttm_pool_free(&bdev->pool, ttm);
+}
+
+static void i915_ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+
+ ttm_tt_destroy_common(bdev, ttm);
+ kfree(i915_tt);
+}
+
+static bool i915_ttm_eviction_valuable(struct ttm_buffer_object *bo,
+ const struct ttm_place *place)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ /* Will do for now. Our pinned objects are still on TTM's LRU lists */
+ if (!i915_gem_object_evictable(obj))
+ return false;
+
+ /* This isn't valid with a buddy allocator */
+ return ttm_bo_eviction_valuable(bo, place);
+}
+
+static void i915_ttm_evict_flags(struct ttm_buffer_object *bo,
+ struct ttm_placement *placement)
+{
+ *placement = i915_sys_placement;
+}
+
+static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ int ret;
+
+ ret = i915_gem_object_unbind(obj, I915_GEM_OBJECT_UNBIND_ACTIVE);
+ if (ret)
+ return ret;
+
+ ret = __i915_gem_object_put_pages(obj);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
+{
+ if (obj->ttm.cached_io_st) {
+ sg_free_table(obj->ttm.cached_io_st);
+ kfree(obj->ttm.cached_io_st);
+ obj->ttm.cached_io_st = NULL;
+ }
+}
+
+static void i915_ttm_purge(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_operation_ctx ctx = {
+ .interruptible = true,
+ .no_wait_gpu = false,
+ };
+ struct ttm_placement place = {};
+ int ret;
+
+ if (obj->mm.madv == __I915_MADV_PURGED)
+ return;
+
+ /* TTM's purge interface. Note that we might be reentering. */
+ ret = ttm_bo_validate(bo, &place, &ctx);
+
+ if (!ret) {
+ i915_ttm_free_cached_io_st(obj);
+ obj->mm.madv = __I915_MADV_PURGED;
+ }
+}
+
+static void i915_ttm_swap_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ int ret = i915_ttm_move_notify(bo);
+
+ GEM_WARN_ON(ret);
+ GEM_WARN_ON(obj->ttm.cached_io_st);
+ if (!ret && obj->mm.madv != I915_MADV_WILLNEED)
+ i915_ttm_purge(obj);
+}
+
+static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ if (likely(obj)) {
+ /* This releases all gem object bindings to the backend. */
+ __i915_gem_free_object(obj);
+ }
+}
+
+static struct intel_memory_region *
+i915_ttm_region(struct ttm_device *bdev, int ttm_mem_type)
+{
+ struct drm_i915_private *i915 = container_of(bdev, typeof(*i915), bdev);
+
+ /* There's some room for optimization here... */
+ GEM_BUG_ON(ttm_mem_type != I915_PL_SYSTEM &&
+ ttm_mem_type < I915_PL_LMEM0);
+ if (ttm_mem_type == I915_PL_SYSTEM)
+ return intel_memory_region_lookup(i915, INTEL_MEMORY_SYSTEM,
+ 0);
+
+ return intel_memory_region_lookup(i915, INTEL_MEMORY_LOCAL,
+ ttm_mem_type - I915_PL_LMEM0);
+}
+
+static struct sg_table *i915_ttm_tt_get_st(struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+ struct scatterlist *sg;
+ struct sg_table *st;
+ int ret;
+
+ if (i915_tt->cached_st)
+ return i915_tt->cached_st;
+
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return ERR_PTR(-ENOMEM);
+
+ sg = __sg_alloc_table_from_pages
+ (st, ttm->pages, ttm->num_pages, 0,
+ (unsigned long)ttm->num_pages << PAGE_SHIFT,
+ i915_sg_segment_size(), NULL, 0, GFP_KERNEL);
+ if (IS_ERR(sg)) {
+ kfree(st);
+ return ERR_CAST(sg);
+ }
+
+ ret = dma_map_sgtable(i915_tt->dev, st, DMA_BIDIRECTIONAL, 0);
+ if (ret) {
+ sg_free_table(st);
+ kfree(st);
+ return ERR_PTR(ret);
+ }
+
+ i915_tt->cached_st = st;
+ return st;
+}
+
+static struct sg_table *
+i915_ttm_resource_get_st(struct drm_i915_gem_object *obj,
+ struct ttm_resource *res)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_resource_manager *man =
+ ttm_manager_type(bo->bdev, res->mem_type);
+
+ if (man->use_tt)
+ return i915_ttm_tt_get_st(bo->ttm);
+
+ return intel_region_ttm_node_to_st(obj->mm.region, res->mm_node);
+}
+
+static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
+ struct ttm_operation_ctx *ctx,
+ struct ttm_resource *dst_mem,
+ struct ttm_place *hop)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ struct ttm_resource_manager *dst_man =
+ ttm_manager_type(bo->bdev, dst_mem->mem_type);
+ struct ttm_resource_manager *src_man =
+ ttm_manager_type(bo->bdev, bo->mem.mem_type);
+ struct intel_memory_region *dst_reg, *src_reg;
+ union {
+ struct ttm_kmap_iter_tt tt;
+ struct ttm_kmap_iter_iomap io;
+ } _dst_iter, _src_iter;
+ struct ttm_kmap_iter *dst_iter, *src_iter;
+ struct sg_table *dst_st;
+ int ret;
+
+ dst_reg = i915_ttm_region(bo->bdev, dst_mem->mem_type);
+ src_reg = i915_ttm_region(bo->bdev, bo->mem.mem_type);
+ GEM_BUG_ON(!dst_reg || !src_reg);
+
+ /* Sync for now. We could do the actual copy async. */
+ ret = ttm_bo_wait_ctx(bo, ctx);
+ if (ret)
+ return ret;
+
+ ret = i915_ttm_move_notify(bo);
+ if (ret)
+ return ret;
+
+ if (obj->mm.madv != I915_MADV_WILLNEED) {
+ i915_ttm_purge(obj);
+ ttm_resource_free(bo, dst_mem);
+ return 0;
+ }
+
+ /* Populate ttm with pages if needed. Typically system memory. */
+ if (bo->ttm && (dst_man->use_tt ||
+ (bo->ttm->page_flags & TTM_PAGE_FLAG_SWAPPED))) {
+ ret = ttm_tt_populate(bo->bdev, bo->ttm, ctx);
+ if (ret)
+ return ret;
+ }
+
+ dst_st = i915_ttm_resource_get_st(obj, dst_mem);
+ if (IS_ERR(dst_st))
+ return PTR_ERR(dst_st);
+
+ /* If we start mapping GGTT, we can no longer use man::use_tt here. */
+ dst_iter = dst_man->use_tt ?
+ ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) :
+ ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
+ dst_st, dst_reg->region.start);
+
+ src_iter = src_man->use_tt ?
+ ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) :
+ ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
+ obj->ttm.cached_io_st,
+ src_reg->region.start);
+
+ ttm_move_memcpy(bo, dst_mem->num_pages, dst_iter, src_iter);
+ ttm_bo_move_sync_cleanup(bo, dst_mem);
+ i915_ttm_free_cached_io_st(obj);
+
+ if (!dst_man->use_tt)
+ obj->ttm.cached_io_st = dst_st;
+
+ return 0;
+}
+
+static struct ttm_device_funcs i915_ttm_bo_driver = {
+ .ttm_tt_create = i915_ttm_tt_create,
+ .ttm_tt_unpopulate = i915_ttm_tt_unpopulate,
+ .ttm_tt_destroy = i915_ttm_tt_destroy,
+ .eviction_valuable = i915_ttm_eviction_valuable,
+ .evict_flags = i915_ttm_evict_flags,
+ .move = i915_ttm_move,
+ .verify_access = NULL,
+ .swap_notify = i915_ttm_swap_notify,
+ .delete_mem_notify = i915_ttm_delete_mem_notify,
+};
+
+/**
+ * i915_ttm_driver - Return a pointer to the TTM device funcs
+ *
+ * Return: Pointer to statically allocated TTM device funcs.
+ */
+struct ttm_device_funcs *i915_ttm_driver(void)
+{
+ return &i915_ttm_bo_driver;
+}
+
+static int i915_ttm_get_pages(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_operation_ctx ctx = {
+ .interruptible = true,
+ .no_wait_gpu = false,
+ };
+ struct sg_table *st;
+ int ret;
+
+ /* Move to the requested placement. */
+ ret = ttm_bo_validate(bo, &i915_lmem0_placement, &ctx);
+ if (ret)
+ return ret == -ENOSPC ? -ENXIO : ret;
+
+ /* Object either has a page vector or is an iomem object */
+ st = bo->ttm ? i915_ttm_tt_get_st(bo->ttm) : obj->ttm.cached_io_st;
+ if (IS_ERR(st))
+ return PTR_ERR(st);
+
+ __i915_gem_object_set_pages(obj, st, i915_sg_dma_sizes(st->sgl));
+
+ i915_ttm_adjust_lru(obj);
+
+ return ret;
+}
+
+static void i915_ttm_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *st)
+{
+ /*
+ * We're currently not called from a shrinker, so put_pages()
+ * typically means the object is about to destroyed, or called
+ * from move_notify(). So just avoid doing much for now.
+ * If the object is not destroyed next, The TTM eviction logic
+ * and shrinkers will move it out if needed.
+ */
+
+ i915_ttm_adjust_lru(obj);
+}
+
+static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+
+ /*
+ * Don't manipulate the TTM LRUs while in TTM bo destruction.
+ * We're called through i915_ttm_delete_mem_notify().
+ */
+ if (!kref_read(&bo->kref))
+ return;
+
+ /*
+ * Put on the correct LRU list depending on the MADV status
+ */
+ spin_lock(&bo->bdev->lru_lock);
+ if (obj->mm.madv != I915_MADV_WILLNEED) {
+ bo->priority = I915_TTM_PRIO_PURGE;
+ } else if (!i915_gem_object_has_pages(obj)) {
+ if (bo->priority < I915_TTM_PRIO_HAS_PAGES)
+ bo->priority = I915_TTM_PRIO_HAS_PAGES;
+ } else {
+ if (bo->priority > I915_TTM_PRIO_NO_PAGES)
+ bo->priority = I915_TTM_PRIO_NO_PAGES;
+ }
+
+ ttm_bo_move_to_lru_tail(bo, &bo->mem, NULL);
+ spin_unlock(&bo->bdev->lru_lock);
+}
+
+/*
+ * TTM-backed gem object destruction requires some clarification.
+ * Basically we have two possibilities here. We can either rely on the
+ * i915 delayed destruction and put the TTM object when the object
+ * is idle. This would be detected by TTM which would bypass the
+ * TTM delayed destroy handling. The other approach is to put the TTM
+ * object early and rely on the TTM destroyed handling, and then free
+ * the leftover parts of the GEM object once TTM's destroyed list handling is
+ * complete. For now, we rely on the latter for two reasons:
+ * a) TTM can evict an object even when it's on the delayed destroy list,
+ * which in theory allows for complete eviction.
+ * b) There is work going on in TTM to allow freeing an object even when
+ * it's not idle, and using the TTM destroyed list handling could help us
+ * benefit from that.
+ */
+static void i915_ttm_delayed_free(struct drm_i915_gem_object *obj)
+{
+ if (obj->ttm.created) {
+ ttm_bo_put(i915_gem_to_ttm(obj));
+ } else {
+ __i915_gem_free_object(obj);
+ call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
+ }
+}
+
+static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
+ .name = "i915_gem_object_ttm",
+ .flags = I915_GEM_OBJECT_HAS_IOMEM,
+
+ .get_pages = i915_ttm_get_pages,
+ .put_pages = i915_ttm_put_pages,
+ .truncate = i915_ttm_purge,
+ .adjust_lru = i915_ttm_adjust_lru,
+ .delayed_free = i915_ttm_delayed_free,
+};
+
+void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ i915_gem_object_release_memory_region(obj);
+ if (obj->ttm.created)
+ call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
+}
+
+/**
+ * __i915_gem_ttm_object_init - Initialize a ttm-backed i915 gem object
+ * @mem: The initial memory region for the object.
+ * @obj: The gem object.
+ * @size: Object size in bytes.
+ * @flags: gem object flags.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
+ struct drm_i915_gem_object *obj,
+ resource_size_t size,
+ unsigned int flags)
+{
+ static struct lock_class_key lock_class;
+ struct drm_i915_private *i915 = mem->i915;
+ enum ttm_bo_type bo_type;
+ size_t alignment = 0;
+ int ret;
+
+ /* Adjust alignment to GPU- and CPU huge page sizes. */
+
+ if (mem->is_range_manager) {
+ if (size >= SZ_1G)
+ alignment = SZ_1G >> PAGE_SHIFT;
+ else if (size >= SZ_2M)
+ alignment = SZ_2M >> PAGE_SHIFT;
+ else if (size >= SZ_64K)
+ alignment = SZ_64K >> PAGE_SHIFT;
+ }
+
+ drm_gem_private_object_init(&i915->drm, &obj->base, size);
+ i915_gem_object_init(obj, &i915_gem_ttm_obj_ops, &lock_class, flags);
+ i915_gem_object_init_memory_region(obj, mem);
+ i915_gem_object_make_unshrinkable(obj);
+ obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
+
+ bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device :
+ ttm_bo_type_kernel;
+
+ /*
+ * If this function fails, it will call the destructor, but
+ * our caller still owns the object. So no freeing in the
+ * destructor until obj->ttm.created is true.
+ * Similarly, in delayed_destroy, we can't call ttm_bo_put()
+ * until successful initialization.
+ */
+ ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size,
+ bo_type, &i915_sys_placement, alignment,
+ true, NULL, NULL, i915_ttm_bo_destroy);
+
+ if (!ret)
+ obj->ttm.created = true;
+
+ /* i915 wants -ENXIO when out of memory region space. */
+ return (ret == -ENOSPC) ? -ENXIO : ret;
+}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.h b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h
new file mode 100644
index 000000000000..b8d3dcbb50df
--- /dev/null
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef _I915_GEM_TTM_H_
+#define _I915_GEM_TTM_H_
+
+#include "gem/i915_gem_object_types.h"
+
+/**
+ * i915_gem_to_ttm - Convert a struct drm_i915_gem_object to a
+ * struct ttm_buffer_object.
+ * @obj: Pointer to the gem object.
+ *
+ * Return: Pointer to the embedded struct ttm_buffer_object.
+ */
+static inline struct ttm_buffer_object *
+i915_gem_to_ttm(struct drm_i915_gem_object *obj)
+{
+ return &obj->__do_not_access;
+}
+
+/*
+ * i915 ttm gem object destructor. Internal use only.
+ */
+void i915_ttm_bo_destroy(struct ttm_buffer_object *bo);
+
+/**
+ * i915_ttm_to_gem - Convert a struct ttm_buffer_object to an embedding
+ * struct drm_i915_gem_object.
+ *
+ * Return: Pointer to the embedding struct ttm_buffer_object, or NULL
+ * if the object was not an i915 ttm object.
+ */
+static inline struct drm_i915_gem_object *
+i915_ttm_to_gem(struct ttm_buffer_object *bo)
+{
+ if (GEM_WARN_ON(bo->destroy != i915_ttm_bo_destroy))
+ return NULL;
+
+ return container_of(bo, struct drm_i915_gem_object, __do_not_access);
+}
+
+int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
+ struct drm_i915_gem_object *obj,
+ resource_size_t size,
+ unsigned int flags);
+#endif
diff --git a/drivers/gpu/drm/i915/gt/intel_region_lmem.c b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
index f7366b054f8e..4ae1f717a94c 100644
--- a/drivers/gpu/drm/i915/gt/intel_region_lmem.c
+++ b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
@@ -9,6 +9,7 @@
#include "intel_region_ttm.h"
#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
+#include "gem/i915_gem_ttm.h"
#include "intel_region_lmem.h"
static int init_fake_lmem_bar(struct intel_memory_region *mem)
@@ -107,7 +108,7 @@ region_lmem_init(struct intel_memory_region *mem)
static const struct intel_memory_region_ops intel_region_lmem_ops = {
.init = region_lmem_init,
.release = region_lmem_release,
- .init_object = __i915_gem_lmem_object_init,
+ .init_object = __i915_gem_ttm_object_init,
};
struct intel_memory_region *
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 0993d706f067..40fe9a147318 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1005,8 +1005,11 @@ i915_gem_madvise_ioctl(struct drm_device *dev, void *data,
}
}
- if (obj->mm.madv != __I915_MADV_PURGED)
+ if (obj->mm.madv != __I915_MADV_PURGED) {
obj->mm.madv = args->madv;
+ if (obj->ops->adjust_lru)
+ obj->ops->adjust_lru(obj);
+ }
if (i915_gem_object_has_pages(obj)) {
unsigned long flags;
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index 4092cc987679..bd27e897d4d0 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -149,7 +149,6 @@ intel_memory_region_create(struct drm_i915_private *i915,
mutex_init(&mem->objects.lock);
INIT_LIST_HEAD(&mem->objects.list);
- INIT_LIST_HEAD(&mem->objects.purgeable);
INIT_LIST_HEAD(&mem->reserved);
mutex_init(&mem->mm_lock);
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index e69cde13daf2..7b5fa97c0b59 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -100,7 +100,6 @@ struct intel_memory_region {
struct {
struct mutex lock; /* Protects access to objects */
struct list_head list;
- struct list_head purgeable;
} objects;
size_t chunk_size;
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
index c8ac118c21f6..0b41a1545570 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.c
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -10,6 +10,7 @@
#include "intel_region_ttm.h"
+#include "gem/i915_gem_ttm.h" /* For the funcs/ops export only */
/**
* DOC: TTM support structure
*
@@ -19,9 +20,6 @@
* i915 GEM regions to TTM memory types and resource managers.
*/
-/* A Zero-initialized driver for now. We don't have a TTM backend yet. */
-static struct ttm_device_funcs i915_ttm_bo_driver;
-
/**
* intel_region_ttm_device_init - Initialize a TTM device
* @dev_priv: Pointer to an i915 device private structure.
@@ -32,7 +30,7 @@ int intel_region_ttm_device_init(struct drm_i915_private *dev_priv)
{
struct drm_device *drm = &dev_priv->drm;
- return ttm_device_init(&dev_priv->bdev, &i915_ttm_bo_driver,
+ return ttm_device_init(&dev_priv->bdev, i915_ttm_driver(),
drm->dev, drm->anon_inode->i_mapping,
drm->vma_offset_manager, false, false);
}
@@ -172,6 +170,7 @@ struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
return i915_sg_from_mm_node(node, mem->region.start);
}
+#ifdef CONFIG_DRM_I915_SELFTEST
/**
* intel_region_ttm_node_alloc - Allocate memory resources from a region
* @mem: The memory region,
@@ -218,3 +217,4 @@ void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
ret = -ENXIO;
return ret ? ERR_PTR(ret) : res.mm_node;
}
+#endif
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.h b/drivers/gpu/drm/i915/intel_region_ttm.h
index 1c82c6c3429d..eaa3eccfa252 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.h
+++ b/drivers/gpu/drm/i915/intel_region_ttm.h
@@ -11,6 +11,7 @@
struct drm_i915_private;
struct intel_memory_region;
+struct ttm_device_funcs;
int intel_region_ttm_device_init(struct drm_i915_private *dev_priv);
@@ -23,10 +24,14 @@ void intel_region_ttm_fini(struct intel_memory_region *mem);
struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
void *node);
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node);
+
+struct ttm_device_funcs *i915_ttm_driver(void);
+
+#ifdef CONFIG_DRM_I915_SELFTEST
void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
resource_size_t size,
unsigned int flags);
-
-void intel_region_ttm_node_free(struct intel_memory_region *mem,
- void *node);
+#endif
#endif /* _INTEL_REGION_TTM_H_ */
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 12/15] drm/i915/lmem: Verify checks for lmem residency
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Since objects can be migrated or evicted when not pinned or locked,
update the checks for lmem residency or future residency so that
the value returned is not immediately stale.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v2: Simplify i915_gem_object_migratable() (Reported by Mattew Auld)
---
drivers/gpu/drm/i915/display/intel_display.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 42 +++++++++++++++++++-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 18 +++++++++
drivers/gpu/drm/i915/gem/i915_gem_object.h | 4 ++
4 files changed, 64 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/display/intel_display.c b/drivers/gpu/drm/i915/display/intel_display.c
index 0bb2e582c87f..26e4daaf4027 100644
--- a/drivers/gpu/drm/i915/display/intel_display.c
+++ b/drivers/gpu/drm/i915/display/intel_display.c
@@ -11733,7 +11733,7 @@ intel_user_framebuffer_create(struct drm_device *dev,
/* object is backed with LMEM for discrete */
i915 = to_i915(obj->base.dev);
- if (HAS_LMEM(i915) && !i915_gem_object_is_lmem(obj)) {
+ if (HAS_LMEM(i915) && !i915_gem_object_validates_to_lmem(obj)) {
/* object is "remote", not in local memory */
i915_gem_object_put(obj);
return ERR_PTR(-EREMOTE);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 2b8cd15de1d9..d539dffa1554 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -23,10 +23,50 @@ i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
return io_mapping_map_wc(&obj->mm.region->iomap, offset, size);
}
+/**
+ * i915_gem_object_validates_to_lmem - Whether the object is resident in
+ * lmem when pages are present.
+ * @obj: The object to check.
+ *
+ * Migratable objects residency may change from under us if the object is
+ * not pinned or locked. This function is intended to be used to check whether
+ * the object can only reside in lmem when pages are present.
+ *
+ * Return: Whether the object is always resident in lmem when pages are
+ * present.
+ */
+bool i915_gem_object_validates_to_lmem(struct drm_i915_gem_object *obj)
+{
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+
+ return !i915_gem_object_migratable(obj) &&
+ mr && (mr->type == INTEL_MEMORY_LOCAL ||
+ mr->type == INTEL_MEMORY_STOLEN_LOCAL);
+}
+
+/**
+ * i915_gem_object_is_lmem - Whether the object is resident in
+ * lmem
+ * @obj: The object to check.
+ *
+ * Even if an object is allowed to migrate and change memory region,
+ * this function checks whether it will always be present in lmem when
+ * valid *or* if that's not the case, whether it's currently resident in lmem.
+ * For migratable and evictable objects, the latter only makes sense when
+ * the object is locked.
+ *
+ * Return: Whether the object migratable but resident in lmem, or not
+ * migratable and will be present in lmem when valid.
+ */
bool i915_gem_object_is_lmem(struct drm_i915_gem_object *obj)
{
- struct intel_memory_region *mr = obj->mm.region;
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+#ifdef CONFIG_LOCKDEP
+ if (i915_gem_object_migratable(obj) &&
+ i915_gem_object_evictable(obj))
+ assert_object_held(obj);
+#endif
return mr && (mr->type == INTEL_MEMORY_LOCAL ||
mr->type == INTEL_MEMORY_STOLEN_LOCAL);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index df2b4e6b9bcc..c8bb6fb1dba3 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -458,6 +458,24 @@ bool i915_gem_object_evictable(struct drm_i915_gem_object *obj)
return pin_count == 0;
}
+/**
+ * i915_gem_object_migratable - Whether the object is migratable out of the
+ * current region.
+ * @obj: Pointer to the object.
+ *
+ * Return: Whether the object is allowed to be resident in other
+ * regions than the current while pages are present.
+ */
+bool i915_gem_object_migratable(struct drm_i915_gem_object *obj)
+{
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+
+ if (!mr)
+ return false;
+
+ return obj->mm.n_placements > 1;
+}
+
void i915_gem_init__objects(struct drm_i915_private *i915)
{
INIT_WORK(&i915->mm.free_work, __i915_gem_free_work);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index ae5930e307d5..a3ad8cf4eefd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -596,6 +596,10 @@ void __i915_gem_free_object(struct drm_i915_gem_object *obj);
bool i915_gem_object_evictable(struct drm_i915_gem_object *obj);
+bool i915_gem_object_migratable(struct drm_i915_gem_object *obj);
+
+bool i915_gem_object_validates_to_lmem(struct drm_i915_gem_object *obj);
+
#ifdef CONFIG_MMU_NOTIFIER
static inline bool
i915_gem_object_is_userptr(struct drm_i915_gem_object *obj)
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 12/15] drm/i915/lmem: Verify checks for lmem residency
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Since objects can be migrated or evicted when not pinned or locked,
update the checks for lmem residency or future residency so that
the value returned is not immediately stale.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v2: Simplify i915_gem_object_migratable() (Reported by Mattew Auld)
---
drivers/gpu/drm/i915/display/intel_display.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 42 +++++++++++++++++++-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 18 +++++++++
drivers/gpu/drm/i915/gem/i915_gem_object.h | 4 ++
4 files changed, 64 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/display/intel_display.c b/drivers/gpu/drm/i915/display/intel_display.c
index 0bb2e582c87f..26e4daaf4027 100644
--- a/drivers/gpu/drm/i915/display/intel_display.c
+++ b/drivers/gpu/drm/i915/display/intel_display.c
@@ -11733,7 +11733,7 @@ intel_user_framebuffer_create(struct drm_device *dev,
/* object is backed with LMEM for discrete */
i915 = to_i915(obj->base.dev);
- if (HAS_LMEM(i915) && !i915_gem_object_is_lmem(obj)) {
+ if (HAS_LMEM(i915) && !i915_gem_object_validates_to_lmem(obj)) {
/* object is "remote", not in local memory */
i915_gem_object_put(obj);
return ERR_PTR(-EREMOTE);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 2b8cd15de1d9..d539dffa1554 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -23,10 +23,50 @@ i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
return io_mapping_map_wc(&obj->mm.region->iomap, offset, size);
}
+/**
+ * i915_gem_object_validates_to_lmem - Whether the object is resident in
+ * lmem when pages are present.
+ * @obj: The object to check.
+ *
+ * Migratable objects residency may change from under us if the object is
+ * not pinned or locked. This function is intended to be used to check whether
+ * the object can only reside in lmem when pages are present.
+ *
+ * Return: Whether the object is always resident in lmem when pages are
+ * present.
+ */
+bool i915_gem_object_validates_to_lmem(struct drm_i915_gem_object *obj)
+{
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+
+ return !i915_gem_object_migratable(obj) &&
+ mr && (mr->type == INTEL_MEMORY_LOCAL ||
+ mr->type == INTEL_MEMORY_STOLEN_LOCAL);
+}
+
+/**
+ * i915_gem_object_is_lmem - Whether the object is resident in
+ * lmem
+ * @obj: The object to check.
+ *
+ * Even if an object is allowed to migrate and change memory region,
+ * this function checks whether it will always be present in lmem when
+ * valid *or* if that's not the case, whether it's currently resident in lmem.
+ * For migratable and evictable objects, the latter only makes sense when
+ * the object is locked.
+ *
+ * Return: Whether the object migratable but resident in lmem, or not
+ * migratable and will be present in lmem when valid.
+ */
bool i915_gem_object_is_lmem(struct drm_i915_gem_object *obj)
{
- struct intel_memory_region *mr = obj->mm.region;
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+#ifdef CONFIG_LOCKDEP
+ if (i915_gem_object_migratable(obj) &&
+ i915_gem_object_evictable(obj))
+ assert_object_held(obj);
+#endif
return mr && (mr->type == INTEL_MEMORY_LOCAL ||
mr->type == INTEL_MEMORY_STOLEN_LOCAL);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index df2b4e6b9bcc..c8bb6fb1dba3 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -458,6 +458,24 @@ bool i915_gem_object_evictable(struct drm_i915_gem_object *obj)
return pin_count == 0;
}
+/**
+ * i915_gem_object_migratable - Whether the object is migratable out of the
+ * current region.
+ * @obj: Pointer to the object.
+ *
+ * Return: Whether the object is allowed to be resident in other
+ * regions than the current while pages are present.
+ */
+bool i915_gem_object_migratable(struct drm_i915_gem_object *obj)
+{
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+
+ if (!mr)
+ return false;
+
+ return obj->mm.n_placements > 1;
+}
+
void i915_gem_init__objects(struct drm_i915_private *i915)
{
INIT_WORK(&i915->mm.free_work, __i915_gem_free_work);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index ae5930e307d5..a3ad8cf4eefd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -596,6 +596,10 @@ void __i915_gem_free_object(struct drm_i915_gem_object *obj);
bool i915_gem_object_evictable(struct drm_i915_gem_object *obj);
+bool i915_gem_object_migratable(struct drm_i915_gem_object *obj);
+
+bool i915_gem_object_validates_to_lmem(struct drm_i915_gem_object *obj);
+
#ifdef CONFIG_MMU_NOTIFIER
static inline bool
i915_gem_object_is_userptr(struct drm_i915_gem_object *obj)
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 13/15] drm/i915: Disable mmap ioctl for gen12+
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
The paltform should exclusively use mmap_offset, one less path to worry
about for discrete.
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_mman.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
index f6fe5cb01438..fd1c9714f8d8 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
@@ -56,10 +56,17 @@ int
i915_gem_mmap_ioctl(struct drm_device *dev, void *data,
struct drm_file *file)
{
+ struct drm_i915_private *i915 = to_i915(dev);
struct drm_i915_gem_mmap *args = data;
struct drm_i915_gem_object *obj;
unsigned long addr;
+ /* mmap ioctl is disallowed for all platforms after TGL-LP. This also
+ * covers all platforms with local memory.
+ */
+ if (INTEL_GEN(i915) >= 12 && !IS_TIGERLAKE(i915))
+ return -EOPNOTSUPP;
+
if (args->flags & ~(I915_MMAP_WC))
return -EINVAL;
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 13/15] drm/i915: Disable mmap ioctl for gen12+
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
The paltform should exclusively use mmap_offset, one less path to worry
about for discrete.
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_mman.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
index f6fe5cb01438..fd1c9714f8d8 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
@@ -56,10 +56,17 @@ int
i915_gem_mmap_ioctl(struct drm_device *dev, void *data,
struct drm_file *file)
{
+ struct drm_i915_private *i915 = to_i915(dev);
struct drm_i915_gem_mmap *args = data;
struct drm_i915_gem_object *obj;
unsigned long addr;
+ /* mmap ioctl is disallowed for all platforms after TGL-LP. This also
+ * covers all platforms with local memory.
+ */
+ if (INTEL_GEN(i915) >= 12 && !IS_TIGERLAKE(i915))
+ return -EOPNOTSUPP;
+
if (args->flags & ~(I915_MMAP_WC))
return -EINVAL;
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 14/15] drm/vma: Add a driver_private member to vma_node.
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
This allows drivers to distinguish between different types of vma_node's.
The readonly flag was unused and is thus removed.
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/drm_gem.c | 9 ---------
include/drm/drm_vma_manager.h | 2 +-
2 files changed, 1 insertion(+), 10 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
index 9989425e9875..e710e79069f6 100644
--- a/drivers/gpu/drm/drm_gem.c
+++ b/drivers/gpu/drm/drm_gem.c
@@ -1149,15 +1149,6 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)
return -EACCES;
}
- if (node->readonly) {
- if (vma->vm_flags & VM_WRITE) {
- drm_gem_object_put(obj);
- return -EINVAL;
- }
-
- vma->vm_flags &= ~VM_MAYWRITE;
- }
-
ret = drm_gem_mmap_obj(obj, drm_vma_node_size(node) << PAGE_SHIFT,
vma);
diff --git a/include/drm/drm_vma_manager.h b/include/drm/drm_vma_manager.h
index 76ac5e97a559..4f8c35206f7c 100644
--- a/include/drm/drm_vma_manager.h
+++ b/include/drm/drm_vma_manager.h
@@ -53,7 +53,7 @@ struct drm_vma_offset_node {
rwlock_t vm_lock;
struct drm_mm_node vm_node;
struct rb_root vm_files;
- bool readonly:1;
+ void *driver_private;
};
struct drm_vma_offset_manager {
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 14/15] drm/vma: Add a driver_private member to vma_node.
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
This allows drivers to distinguish between different types of vma_node's.
The readonly flag was unused and is thus removed.
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/drm_gem.c | 9 ---------
include/drm/drm_vma_manager.h | 2 +-
2 files changed, 1 insertion(+), 10 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
index 9989425e9875..e710e79069f6 100644
--- a/drivers/gpu/drm/drm_gem.c
+++ b/drivers/gpu/drm/drm_gem.c
@@ -1149,15 +1149,6 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)
return -EACCES;
}
- if (node->readonly) {
- if (vma->vm_flags & VM_WRITE) {
- drm_gem_object_put(obj);
- return -EINVAL;
- }
-
- vma->vm_flags &= ~VM_MAYWRITE;
- }
-
ret = drm_gem_mmap_obj(obj, drm_vma_node_size(node) << PAGE_SHIFT,
vma);
diff --git a/include/drm/drm_vma_manager.h b/include/drm/drm_vma_manager.h
index 76ac5e97a559..4f8c35206f7c 100644
--- a/include/drm/drm_vma_manager.h
+++ b/include/drm/drm_vma_manager.h
@@ -53,7 +53,7 @@ struct drm_vma_offset_node {
rwlock_t vm_lock;
struct drm_mm_node vm_node;
struct rb_root vm_files;
- bool readonly:1;
+ void *driver_private;
};
struct drm_vma_offset_manager {
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [PATCH v4 15/15] drm/i915: Use ttm mmap handling for ttm bo's.
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 11:32 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Use the ttm handlers for servicing page faults, and vm_access.
We do our own validation of read-only access, otherwise use the
ttm handlers as much as possible.
Because the ttm handlers expect the vma_node at vma->base, we slightly
need to massage the mmap handlers to look at vma_node->driver_private
to fetch the bo, if it's NULL, we assume i915's normal mmap_offset uapi
is used.
This is the easiest way to achieve compatibility without changing ttm's
semantics.
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_mman.c | 78 +++++++----
drivers/gpu/drm/i915/gem/i915_gem_object.h | 6 +-
.../gpu/drm/i915/gem/i915_gem_object_types.h | 3 +
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 122 +++++++++++++++++-
.../drm/i915/gem/selftests/i915_gem_mman.c | 90 +++++++------
drivers/gpu/drm/i915/selftests/igt_mmap.c | 25 +++-
drivers/gpu/drm/i915/selftests/igt_mmap.h | 12 +-
8 files changed, 247 insertions(+), 92 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
index fd1c9714f8d8..af04ea593091 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
@@ -19,6 +19,7 @@
#include "i915_gem_mman.h"
#include "i915_trace.h"
#include "i915_user_extensions.h"
+#include "i915_gem_ttm.h"
#include "i915_vma.h"
static inline bool
@@ -622,6 +623,8 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
struct i915_mmap_offset *mmo;
int err;
+ GEM_BUG_ON(obj->ops->mmap_offset || obj->ops->mmap_ops);
+
mmo = lookup_mmo(obj, mmap_type);
if (mmo)
goto out;
@@ -664,40 +667,47 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
}
static int
-__assign_mmap_offset(struct drm_file *file,
- u32 handle,
+__assign_mmap_offset(struct drm_i915_gem_object *obj,
enum i915_mmap_type mmap_type,
- u64 *offset)
+ u64 *offset, struct drm_file *file)
{
- struct drm_i915_gem_object *obj;
struct i915_mmap_offset *mmo;
- int err;
- obj = i915_gem_object_lookup(file, handle);
- if (!obj)
- return -ENOENT;
+ if (i915_gem_object_never_mmap(obj))
+ return -ENODEV;
- if (i915_gem_object_never_mmap(obj)) {
- err = -ENODEV;
- goto out;
+ if (obj->ops->mmap_offset) {
+ *offset = obj->ops->mmap_offset(obj);
+ return 0;
}
if (mmap_type != I915_MMAP_TYPE_GTT &&
!i915_gem_object_has_struct_page(obj) &&
- !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM)) {
- err = -ENODEV;
- goto out;
- }
+ !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM))
+ return -ENODEV;
mmo = mmap_offset_attach(obj, mmap_type, file);
- if (IS_ERR(mmo)) {
- err = PTR_ERR(mmo);
- goto out;
- }
+ if (IS_ERR(mmo))
+ return PTR_ERR(mmo);
*offset = drm_vma_node_offset_addr(&mmo->vma_node);
- err = 0;
-out:
+ return 0;
+}
+
+static int
+__assign_mmap_offset_handle(struct drm_file *file,
+ u32 handle,
+ enum i915_mmap_type mmap_type,
+ u64 *offset)
+{
+ struct drm_i915_gem_object *obj;
+ int err;
+
+ obj = i915_gem_object_lookup(file, handle);
+ if (!obj)
+ return -ENOENT;
+
+ err = __assign_mmap_offset(obj, mmap_type, offset, file);
i915_gem_object_put(obj);
return err;
}
@@ -717,7 +727,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
else
mmap_type = I915_MMAP_TYPE_GTT;
- return __assign_mmap_offset(file, handle, mmap_type, offset);
+ return __assign_mmap_offset_handle(file, handle, mmap_type, offset);
}
/**
@@ -785,7 +795,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
return -EINVAL;
}
- return __assign_mmap_offset(file, args->handle, type, &args->offset);
+ return __assign_mmap_offset_handle(file, args->handle, type, &args->offset);
}
static void vm_open(struct vm_area_struct *vma)
@@ -889,8 +899,16 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
* destroyed and will be invalid when the vma manager lock
* is released.
*/
- mmo = container_of(node, struct i915_mmap_offset, vma_node);
- obj = i915_gem_object_get_rcu(mmo->obj);
+ if (!node->driver_private) {
+ mmo = container_of(node, struct i915_mmap_offset, vma_node);
+ obj = i915_gem_object_get_rcu(mmo->obj);
+
+ GEM_BUG_ON(obj && obj->ops->mmap_ops);
+ } else {
+ obj = i915_gem_object_get_rcu(container_of(node, struct drm_i915_gem_object, base.vma_node));
+
+ GEM_BUG_ON(obj && !obj->ops->mmap_ops);
+ }
}
drm_vma_offset_unlock_lookup(dev->vma_offset_manager);
rcu_read_unlock();
@@ -912,7 +930,6 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
}
vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP;
- vma->vm_private_data = mmo;
/*
* We keep the ref on mmo->obj, not vm_file, but we require
@@ -926,6 +943,15 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
/* Drop the initial creation reference, the vma is now holding one. */
fput(anon);
+ if (obj->ops->mmap_ops) {
+ vma->vm_page_prot = pgprot_decrypted(vm_get_page_prot(vma->vm_flags));
+ vma->vm_ops = obj->ops->mmap_ops;
+ vma->vm_private_data = node->driver_private;
+ return 0;
+ }
+
+ vma->vm_private_data = mmo;
+
switch (mmo->mmap_type) {
case I915_MMAP_TYPE_WC:
vma->vm_page_prot =
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index a3ad8cf4eefd..ff59e6c640e6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -342,14 +342,14 @@ struct scatterlist *
__i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
struct i915_gem_object_page_iter *iter,
unsigned int n,
- unsigned int *offset, bool allow_alloc);
+ unsigned int *offset, bool allow_alloc, bool dma);
static inline struct scatterlist *
i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
unsigned int n,
unsigned int *offset, bool allow_alloc)
{
- return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc);
+ return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc, false);
}
static inline struct scatterlist *
@@ -357,7 +357,7 @@ i915_gem_object_get_sg_dma(struct drm_i915_gem_object *obj,
unsigned int n,
unsigned int *offset, bool allow_alloc)
{
- return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc);
+ return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc, true);
}
struct page *
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 68313474e6a6..2a23b77424b3 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -61,6 +61,7 @@ struct drm_i915_gem_object_ops {
const struct drm_i915_gem_pread *arg);
int (*pwrite)(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_pwrite *arg);
+ u64 (*mmap_offset)(struct drm_i915_gem_object *obj);
int (*dmabuf_export)(struct drm_i915_gem_object *obj);
@@ -79,6 +80,7 @@ struct drm_i915_gem_object_ops {
void (*delayed_free)(struct drm_i915_gem_object *obj);
void (*release)(struct drm_i915_gem_object *obj);
+ const struct vm_operations_struct *mmap_ops;
const char *name; /* friendly name for debug, e.g. lockdep classes */
};
@@ -328,6 +330,7 @@ struct drm_i915_gem_object {
struct {
struct sg_table *cached_io_st;
+ struct i915_gem_object_page_iter get_io_page;
bool created:1;
} ttm;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
index 6444e097016d..086005c1c7ea 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
@@ -467,9 +467,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
struct i915_gem_object_page_iter *iter,
unsigned int n,
unsigned int *offset,
- bool allow_alloc)
+ bool allow_alloc, bool dma)
{
- const bool dma = iter == &obj->mm.get_dma_page;
struct scatterlist *sg;
unsigned int idx, count;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
index 17598930a99e..d0be957326e0 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -13,6 +13,7 @@
#include "gem/i915_gem_object.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_ttm.h"
+#include "gem/i915_gem_mman.h"
#define I915_PL_LMEM0 TTM_PL_PRIV
#define I915_PL_SYSTEM TTM_PL_SYSTEM
@@ -158,11 +159,20 @@ static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
{
- if (obj->ttm.cached_io_st) {
- sg_free_table(obj->ttm.cached_io_st);
- kfree(obj->ttm.cached_io_st);
- obj->ttm.cached_io_st = NULL;
- }
+ struct radix_tree_iter iter;
+ void __rcu **slot;
+
+ if (!obj->ttm.cached_io_st)
+ return;
+
+ rcu_read_lock();
+ radix_tree_for_each_slot(slot, &obj->ttm.get_io_page.radix, &iter, 0)
+ radix_tree_delete(&obj->ttm.get_io_page.radix, iter.index);
+ rcu_read_unlock();
+
+ sg_free_table(obj->ttm.cached_io_st);
+ kfree(obj->ttm.cached_io_st);
+ obj->ttm.cached_io_st = NULL;
}
static void i915_ttm_purge(struct drm_i915_gem_object *obj)
@@ -338,12 +348,42 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
ttm_bo_move_sync_cleanup(bo, dst_mem);
i915_ttm_free_cached_io_st(obj);
- if (!dst_man->use_tt)
+ if (!dst_man->use_tt) {
obj->ttm.cached_io_st = dst_st;
+ obj->ttm.get_io_page.sg_pos = dst_st->sgl;
+ obj->ttm.get_io_page.sg_idx = 0;
+ }
return 0;
}
+static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource *mem)
+{
+ if (mem->mem_type < I915_PL_LMEM0)
+ return 0;
+
+ /* We may need to revisit this later, but this allows all caching to be used in mmap */
+ mem->bus.caching = ttm_cached;
+ mem->bus.is_iomem = true;
+
+ return 0;
+}
+
+static unsigned long i915_ttm_io_mem_pfn(struct ttm_buffer_object *bo,
+ unsigned long page_offset)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ unsigned long base = obj->mm.region->iomap.base - obj->mm.region->region.start;
+ struct scatterlist *sg;
+ unsigned int ofs;
+
+ GEM_WARN_ON(bo->ttm);
+
+ sg = __i915_gem_object_get_sg(obj, &obj->ttm.get_io_page, page_offset, &ofs, true, true);
+
+ return ((base + sg_dma_address(sg)) >> PAGE_SHIFT) + ofs;
+}
+
static struct ttm_device_funcs i915_ttm_bo_driver = {
.ttm_tt_create = i915_ttm_tt_create,
.ttm_tt_unpopulate = i915_ttm_tt_unpopulate,
@@ -354,6 +394,8 @@ static struct ttm_device_funcs i915_ttm_bo_driver = {
.verify_access = NULL,
.swap_notify = i915_ttm_swap_notify,
.delete_mem_notify = i915_ttm_delete_mem_notify,
+ .io_mem_reserve = i915_ttm_io_mem_reserve,
+ .io_mem_pfn = i915_ttm_io_mem_pfn,
};
/**
@@ -461,7 +503,68 @@ static void i915_ttm_delayed_free(struct drm_i915_gem_object *obj)
}
}
-static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
+static vm_fault_t vm_fault_ttm(struct vm_fault *vmf)
+{
+ struct vm_area_struct *area = vmf->vma;
+ struct drm_i915_gem_object *obj =
+ i915_ttm_to_gem(area->vm_private_data);
+
+ /* Sanity check that we allow writing into this object */
+ if (unlikely(i915_gem_object_is_readonly(obj) &&
+ area->vm_flags & VM_WRITE))
+ return VM_FAULT_SIGBUS;
+
+ return ttm_bo_vm_fault(vmf);
+}
+
+static int
+vm_access_ttm(struct vm_area_struct *area, unsigned long addr,
+ void *buf, int len, int write)
+{
+ struct drm_i915_gem_object *obj =
+ i915_ttm_to_gem(area->vm_private_data);
+
+ if (i915_gem_object_is_readonly(obj) && write)
+ return -EACCES;
+
+ return ttm_bo_vm_access(area, addr, buf, len, write);
+}
+
+static void ttm_vm_open(struct vm_area_struct *vma)
+{
+ struct drm_i915_gem_object *obj =
+ i915_ttm_to_gem(vma->vm_private_data);
+
+ GEM_BUG_ON(!obj);
+ i915_gem_object_get(obj);
+}
+
+static void ttm_vm_close(struct vm_area_struct *vma)
+{
+ struct drm_i915_gem_object *obj =
+ i915_ttm_to_gem(vma->vm_private_data);
+
+ GEM_BUG_ON(!obj);
+ i915_gem_object_put(obj);
+}
+
+
+static const struct vm_operations_struct vm_ops_ttm = {
+ .fault = vm_fault_ttm,
+ .access = vm_access_ttm,
+ .open = ttm_vm_open,
+ .close = ttm_vm_close,
+};
+
+static u64 i915_ttm_mmap_offset(struct drm_i915_gem_object *obj)
+{
+ /* The ttm_bo must be allocated with I915_BO_ALLOC_USER */
+ GEM_BUG_ON(!drm_mm_node_allocated(&obj->base.vma_node.vm_node));
+
+ return drm_vma_node_offset_addr(&obj->base.vma_node);
+}
+
+const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
.name = "i915_gem_object_ttm",
.flags = I915_GEM_OBJECT_HAS_IOMEM,
@@ -470,6 +573,8 @@ static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
.truncate = i915_ttm_purge,
.adjust_lru = i915_ttm_adjust_lru,
.delayed_free = i915_ttm_delayed_free,
+ .mmap_offset = i915_ttm_mmap_offset,
+ .mmap_ops = &vm_ops_ttm,
};
void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
@@ -518,6 +623,8 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
i915_gem_object_make_unshrinkable(obj);
obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
+ INIT_RADIX_TREE(&obj->ttm.get_io_page.radix, GFP_KERNEL | __GFP_NOWARN);
+ mutex_init(&obj->ttm.get_io_page.lock);
bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device :
ttm_bo_type_kernel;
@@ -529,6 +636,7 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
* Similarly, in delayed_destroy, we can't call ttm_bo_put()
* until successful initialization.
*/
+ obj->base.vma_node.driver_private = i915_gem_to_ttm(obj);
ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size,
bo_type, &i915_sys_placement, alignment,
true, NULL, NULL, i915_ttm_bo_destroy);
diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
index 05a3b29f545e..ca69a29b7f2a 100644
--- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
@@ -578,16 +578,17 @@ static bool assert_mmap_offset(struct drm_i915_private *i915,
int expected)
{
struct drm_i915_gem_object *obj;
- struct i915_mmap_offset *mmo;
+ u64 offset;
+ int ret;
obj = i915_gem_object_create_internal(i915, size);
if (IS_ERR(obj))
- return false;
+ return expected && expected == PTR_ERR(obj);
- mmo = mmap_offset_attach(obj, I915_MMAP_OFFSET_GTT, NULL);
+ ret = __assign_mmap_offset(obj, I915_MMAP_TYPE_GTT, &offset, NULL);
i915_gem_object_put(obj);
- return PTR_ERR_OR_ZERO(mmo) == expected;
+ return ret == expected;
}
static void disable_retire_worker(struct drm_i915_private *i915)
@@ -622,8 +623,8 @@ static int igt_mmap_offset_exhaustion(void *arg)
struct drm_mm *mm = &i915->drm.vma_offset_manager->vm_addr_space_mm;
struct drm_i915_gem_object *obj;
struct drm_mm_node *hole, *next;
- struct i915_mmap_offset *mmo;
int loop, err = 0;
+ u64 offset;
/* Disable background reaper */
disable_retire_worker(i915);
@@ -684,13 +685,13 @@ static int igt_mmap_offset_exhaustion(void *arg)
obj = i915_gem_object_create_internal(i915, PAGE_SIZE);
if (IS_ERR(obj)) {
err = PTR_ERR(obj);
+ pr_err("Unable to create object for reclaimed hole\n");
goto out;
}
- mmo = mmap_offset_attach(obj, I915_MMAP_OFFSET_GTT, NULL);
- if (IS_ERR(mmo)) {
+ err = __assign_mmap_offset(obj, I915_MMAP_TYPE_GTT, &offset, NULL);
+ if (err) {
pr_err("Unable to insert object into reclaimed hole\n");
- err = PTR_ERR(mmo);
goto err_obj;
}
@@ -865,10 +866,10 @@ static int __igt_mmap(struct drm_i915_private *i915,
struct drm_i915_gem_object *obj,
enum i915_mmap_type type)
{
- struct i915_mmap_offset *mmo;
struct vm_area_struct *area;
unsigned long addr;
int err, i;
+ u64 offset;
if (!can_mmap(obj, type))
return 0;
@@ -879,11 +880,11 @@ static int __igt_mmap(struct drm_i915_private *i915,
if (err)
return err;
- mmo = mmap_offset_attach(obj, type, NULL);
- if (IS_ERR(mmo))
- return PTR_ERR(mmo);
+ err = __assign_mmap_offset(obj, type, &offset, NULL);
+ if (err)
+ return err;
- addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
+ addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
if (IS_ERR_VALUE(addr))
return addr;
@@ -897,13 +898,6 @@ static int __igt_mmap(struct drm_i915_private *i915,
goto out_unmap;
}
- if (area->vm_private_data != mmo) {
- pr_err("%s: vm_area_struct did not point back to our mmap_offset object!\n",
- obj->mm.region->name);
- err = -EINVAL;
- goto out_unmap;
- }
-
for (i = 0; i < obj->base.size / sizeof(u32); i++) {
u32 __user *ux = u64_to_user_ptr((u64)(addr + i * sizeof(*ux)));
u32 x;
@@ -961,7 +955,7 @@ static int igt_mmap(void *arg)
struct drm_i915_gem_object *obj;
int err;
- obj = i915_gem_object_create_region(mr, sizes[i], 0);
+ obj = i915_gem_object_create_region(mr, sizes[i], I915_BO_ALLOC_USER);
if (obj == ERR_PTR(-ENODEV))
continue;
@@ -1004,12 +998,12 @@ static int __igt_mmap_access(struct drm_i915_private *i915,
struct drm_i915_gem_object *obj,
enum i915_mmap_type type)
{
- struct i915_mmap_offset *mmo;
unsigned long __user *ptr;
unsigned long A, B;
unsigned long x, y;
unsigned long addr;
int err;
+ u64 offset;
memset(&A, 0xAA, sizeof(A));
memset(&B, 0xBB, sizeof(B));
@@ -1017,11 +1011,11 @@ static int __igt_mmap_access(struct drm_i915_private *i915,
if (!can_mmap(obj, type) || !can_access(obj))
return 0;
- mmo = mmap_offset_attach(obj, type, NULL);
- if (IS_ERR(mmo))
- return PTR_ERR(mmo);
+ err = __assign_mmap_offset(obj, type, &offset, NULL);
+ if (err)
+ return err;
- addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
+ addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
if (IS_ERR_VALUE(addr))
return addr;
ptr = (unsigned long __user *)addr;
@@ -1081,7 +1075,7 @@ static int igt_mmap_access(void *arg)
struct drm_i915_gem_object *obj;
int err;
- obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
+ obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
if (obj == ERR_PTR(-ENODEV))
continue;
@@ -1111,11 +1105,11 @@ static int __igt_mmap_gpu(struct drm_i915_private *i915,
enum i915_mmap_type type)
{
struct intel_engine_cs *engine;
- struct i915_mmap_offset *mmo;
unsigned long addr;
u32 __user *ux;
u32 bbe;
int err;
+ u64 offset;
/*
* Verify that the mmap access into the backing store aligns with
@@ -1132,11 +1126,11 @@ static int __igt_mmap_gpu(struct drm_i915_private *i915,
if (err)
return err;
- mmo = mmap_offset_attach(obj, type, NULL);
- if (IS_ERR(mmo))
- return PTR_ERR(mmo);
+ err = __assign_mmap_offset(obj, type, &offset, NULL);
+ if (err)
+ return err;
- addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
+ addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
if (IS_ERR_VALUE(addr))
return addr;
@@ -1226,7 +1220,7 @@ static int igt_mmap_gpu(void *arg)
struct drm_i915_gem_object *obj;
int err;
- obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
+ obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
if (obj == ERR_PTR(-ENODEV))
continue;
@@ -1303,18 +1297,18 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
struct drm_i915_gem_object *obj,
enum i915_mmap_type type)
{
- struct i915_mmap_offset *mmo;
unsigned long addr;
int err;
+ u64 offset;
if (!can_mmap(obj, type))
return 0;
- mmo = mmap_offset_attach(obj, type, NULL);
- if (IS_ERR(mmo))
- return PTR_ERR(mmo);
+ err = __assign_mmap_offset(obj, type, &offset, NULL);
+ if (err)
+ return err;
- addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
+ addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
if (IS_ERR_VALUE(addr))
return addr;
@@ -1350,10 +1344,20 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
}
}
- err = check_absent(addr, obj->base.size);
- if (err) {
- pr_err("%s: was not absent\n", obj->mm.region->name);
- goto out_unmap;
+ if (!obj->ops->mmap_ops) {
+ err = check_absent(addr, obj->base.size);
+ if (err) {
+ pr_err("%s: was not absent\n", obj->mm.region->name);
+ goto out_unmap;
+ }
+ } else {
+ /* ttm allows access to evicted regions by design */
+
+ err = check_present(addr, obj->base.size);
+ if (err) {
+ pr_err("%s: was not present\n", obj->mm.region->name);
+ goto out_unmap;
+ }
}
out_unmap:
@@ -1371,7 +1375,7 @@ static int igt_mmap_revoke(void *arg)
struct drm_i915_gem_object *obj;
int err;
- obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
+ obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
if (obj == ERR_PTR(-ENODEV))
continue;
diff --git a/drivers/gpu/drm/i915/selftests/igt_mmap.c b/drivers/gpu/drm/i915/selftests/igt_mmap.c
index 583a4ff8b8c9..e8286c28de91 100644
--- a/drivers/gpu/drm/i915/selftests/igt_mmap.c
+++ b/drivers/gpu/drm/i915/selftests/igt_mmap.c
@@ -9,15 +9,28 @@
#include "i915_drv.h"
#include "igt_mmap.h"
-unsigned long igt_mmap_node(struct drm_i915_private *i915,
- struct drm_vma_offset_node *node,
- unsigned long addr,
- unsigned long prot,
- unsigned long flags)
+unsigned long igt_mmap_offset(struct drm_i915_private *i915,
+ u64 offset,
+ unsigned long size,
+ unsigned long prot,
+ unsigned long flags)
{
+ struct drm_vma_offset_node *node;
struct file *file;
+ unsigned long addr;
int err;
+ /* no need to refcount, we own this object */
+ drm_vma_offset_lock_lookup(i915->drm.vma_offset_manager);
+ node = drm_vma_offset_exact_lookup_locked(i915->drm.vma_offset_manager,
+ offset / PAGE_SIZE, size / PAGE_SIZE);
+ drm_vma_offset_unlock_lookup(i915->drm.vma_offset_manager);
+
+ if (GEM_WARN_ON(!node)) {
+ pr_info("Failed to lookup %Lx\n", offset);
+ return -ENOENT;
+ }
+
/* Pretend to open("/dev/dri/card0") */
file = mock_drm_getfile(i915->drm.primary, O_RDWR);
if (IS_ERR(file))
@@ -29,7 +42,7 @@ unsigned long igt_mmap_node(struct drm_i915_private *i915,
goto out_file;
}
- addr = vm_mmap(file, addr, drm_vma_node_size(node) << PAGE_SHIFT,
+ addr = vm_mmap(file, 0, drm_vma_node_size(node) << PAGE_SHIFT,
prot, flags, drm_vma_node_offset_addr(node));
drm_vma_node_revoke(node, file->private_data);
diff --git a/drivers/gpu/drm/i915/selftests/igt_mmap.h b/drivers/gpu/drm/i915/selftests/igt_mmap.h
index 6e716cb59d7e..acbe34d81a6d 100644
--- a/drivers/gpu/drm/i915/selftests/igt_mmap.h
+++ b/drivers/gpu/drm/i915/selftests/igt_mmap.h
@@ -7,13 +7,15 @@
#ifndef IGT_MMAP_H
#define IGT_MMAP_H
+#include <linux/types.h>
+
struct drm_i915_private;
struct drm_vma_offset_node;
-unsigned long igt_mmap_node(struct drm_i915_private *i915,
- struct drm_vma_offset_node *node,
- unsigned long addr,
- unsigned long prot,
- unsigned long flags);
+unsigned long igt_mmap_offset(struct drm_i915_private *i915,
+ u64 offset,
+ unsigned long size,
+ unsigned long prot,
+ unsigned long flags);
#endif /* IGT_MMAP_H */
--
2.31.1
^ permalink raw reply related [flat|nested] 58+ messages in thread
* [Intel-gfx] [PATCH v4 15/15] drm/i915: Use ttm mmap handling for ttm bo's.
@ 2021-05-26 11:32 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 11:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Use the ttm handlers for servicing page faults, and vm_access.
We do our own validation of read-only access, otherwise use the
ttm handlers as much as possible.
Because the ttm handlers expect the vma_node at vma->base, we slightly
need to massage the mmap handlers to look at vma_node->driver_private
to fetch the bo, if it's NULL, we assume i915's normal mmap_offset uapi
is used.
This is the easiest way to achieve compatibility without changing ttm's
semantics.
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_mman.c | 78 +++++++----
drivers/gpu/drm/i915/gem/i915_gem_object.h | 6 +-
.../gpu/drm/i915/gem/i915_gem_object_types.h | 3 +
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 122 +++++++++++++++++-
.../drm/i915/gem/selftests/i915_gem_mman.c | 90 +++++++------
drivers/gpu/drm/i915/selftests/igt_mmap.c | 25 +++-
drivers/gpu/drm/i915/selftests/igt_mmap.h | 12 +-
8 files changed, 247 insertions(+), 92 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
index fd1c9714f8d8..af04ea593091 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
@@ -19,6 +19,7 @@
#include "i915_gem_mman.h"
#include "i915_trace.h"
#include "i915_user_extensions.h"
+#include "i915_gem_ttm.h"
#include "i915_vma.h"
static inline bool
@@ -622,6 +623,8 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
struct i915_mmap_offset *mmo;
int err;
+ GEM_BUG_ON(obj->ops->mmap_offset || obj->ops->mmap_ops);
+
mmo = lookup_mmo(obj, mmap_type);
if (mmo)
goto out;
@@ -664,40 +667,47 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
}
static int
-__assign_mmap_offset(struct drm_file *file,
- u32 handle,
+__assign_mmap_offset(struct drm_i915_gem_object *obj,
enum i915_mmap_type mmap_type,
- u64 *offset)
+ u64 *offset, struct drm_file *file)
{
- struct drm_i915_gem_object *obj;
struct i915_mmap_offset *mmo;
- int err;
- obj = i915_gem_object_lookup(file, handle);
- if (!obj)
- return -ENOENT;
+ if (i915_gem_object_never_mmap(obj))
+ return -ENODEV;
- if (i915_gem_object_never_mmap(obj)) {
- err = -ENODEV;
- goto out;
+ if (obj->ops->mmap_offset) {
+ *offset = obj->ops->mmap_offset(obj);
+ return 0;
}
if (mmap_type != I915_MMAP_TYPE_GTT &&
!i915_gem_object_has_struct_page(obj) &&
- !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM)) {
- err = -ENODEV;
- goto out;
- }
+ !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM))
+ return -ENODEV;
mmo = mmap_offset_attach(obj, mmap_type, file);
- if (IS_ERR(mmo)) {
- err = PTR_ERR(mmo);
- goto out;
- }
+ if (IS_ERR(mmo))
+ return PTR_ERR(mmo);
*offset = drm_vma_node_offset_addr(&mmo->vma_node);
- err = 0;
-out:
+ return 0;
+}
+
+static int
+__assign_mmap_offset_handle(struct drm_file *file,
+ u32 handle,
+ enum i915_mmap_type mmap_type,
+ u64 *offset)
+{
+ struct drm_i915_gem_object *obj;
+ int err;
+
+ obj = i915_gem_object_lookup(file, handle);
+ if (!obj)
+ return -ENOENT;
+
+ err = __assign_mmap_offset(obj, mmap_type, offset, file);
i915_gem_object_put(obj);
return err;
}
@@ -717,7 +727,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
else
mmap_type = I915_MMAP_TYPE_GTT;
- return __assign_mmap_offset(file, handle, mmap_type, offset);
+ return __assign_mmap_offset_handle(file, handle, mmap_type, offset);
}
/**
@@ -785,7 +795,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
return -EINVAL;
}
- return __assign_mmap_offset(file, args->handle, type, &args->offset);
+ return __assign_mmap_offset_handle(file, args->handle, type, &args->offset);
}
static void vm_open(struct vm_area_struct *vma)
@@ -889,8 +899,16 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
* destroyed and will be invalid when the vma manager lock
* is released.
*/
- mmo = container_of(node, struct i915_mmap_offset, vma_node);
- obj = i915_gem_object_get_rcu(mmo->obj);
+ if (!node->driver_private) {
+ mmo = container_of(node, struct i915_mmap_offset, vma_node);
+ obj = i915_gem_object_get_rcu(mmo->obj);
+
+ GEM_BUG_ON(obj && obj->ops->mmap_ops);
+ } else {
+ obj = i915_gem_object_get_rcu(container_of(node, struct drm_i915_gem_object, base.vma_node));
+
+ GEM_BUG_ON(obj && !obj->ops->mmap_ops);
+ }
}
drm_vma_offset_unlock_lookup(dev->vma_offset_manager);
rcu_read_unlock();
@@ -912,7 +930,6 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
}
vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP;
- vma->vm_private_data = mmo;
/*
* We keep the ref on mmo->obj, not vm_file, but we require
@@ -926,6 +943,15 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
/* Drop the initial creation reference, the vma is now holding one. */
fput(anon);
+ if (obj->ops->mmap_ops) {
+ vma->vm_page_prot = pgprot_decrypted(vm_get_page_prot(vma->vm_flags));
+ vma->vm_ops = obj->ops->mmap_ops;
+ vma->vm_private_data = node->driver_private;
+ return 0;
+ }
+
+ vma->vm_private_data = mmo;
+
switch (mmo->mmap_type) {
case I915_MMAP_TYPE_WC:
vma->vm_page_prot =
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index a3ad8cf4eefd..ff59e6c640e6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -342,14 +342,14 @@ struct scatterlist *
__i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
struct i915_gem_object_page_iter *iter,
unsigned int n,
- unsigned int *offset, bool allow_alloc);
+ unsigned int *offset, bool allow_alloc, bool dma);
static inline struct scatterlist *
i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
unsigned int n,
unsigned int *offset, bool allow_alloc)
{
- return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc);
+ return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc, false);
}
static inline struct scatterlist *
@@ -357,7 +357,7 @@ i915_gem_object_get_sg_dma(struct drm_i915_gem_object *obj,
unsigned int n,
unsigned int *offset, bool allow_alloc)
{
- return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc);
+ return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc, true);
}
struct page *
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 68313474e6a6..2a23b77424b3 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -61,6 +61,7 @@ struct drm_i915_gem_object_ops {
const struct drm_i915_gem_pread *arg);
int (*pwrite)(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_pwrite *arg);
+ u64 (*mmap_offset)(struct drm_i915_gem_object *obj);
int (*dmabuf_export)(struct drm_i915_gem_object *obj);
@@ -79,6 +80,7 @@ struct drm_i915_gem_object_ops {
void (*delayed_free)(struct drm_i915_gem_object *obj);
void (*release)(struct drm_i915_gem_object *obj);
+ const struct vm_operations_struct *mmap_ops;
const char *name; /* friendly name for debug, e.g. lockdep classes */
};
@@ -328,6 +330,7 @@ struct drm_i915_gem_object {
struct {
struct sg_table *cached_io_st;
+ struct i915_gem_object_page_iter get_io_page;
bool created:1;
} ttm;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
index 6444e097016d..086005c1c7ea 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
@@ -467,9 +467,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
struct i915_gem_object_page_iter *iter,
unsigned int n,
unsigned int *offset,
- bool allow_alloc)
+ bool allow_alloc, bool dma)
{
- const bool dma = iter == &obj->mm.get_dma_page;
struct scatterlist *sg;
unsigned int idx, count;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
index 17598930a99e..d0be957326e0 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -13,6 +13,7 @@
#include "gem/i915_gem_object.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_ttm.h"
+#include "gem/i915_gem_mman.h"
#define I915_PL_LMEM0 TTM_PL_PRIV
#define I915_PL_SYSTEM TTM_PL_SYSTEM
@@ -158,11 +159,20 @@ static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
{
- if (obj->ttm.cached_io_st) {
- sg_free_table(obj->ttm.cached_io_st);
- kfree(obj->ttm.cached_io_st);
- obj->ttm.cached_io_st = NULL;
- }
+ struct radix_tree_iter iter;
+ void __rcu **slot;
+
+ if (!obj->ttm.cached_io_st)
+ return;
+
+ rcu_read_lock();
+ radix_tree_for_each_slot(slot, &obj->ttm.get_io_page.radix, &iter, 0)
+ radix_tree_delete(&obj->ttm.get_io_page.radix, iter.index);
+ rcu_read_unlock();
+
+ sg_free_table(obj->ttm.cached_io_st);
+ kfree(obj->ttm.cached_io_st);
+ obj->ttm.cached_io_st = NULL;
}
static void i915_ttm_purge(struct drm_i915_gem_object *obj)
@@ -338,12 +348,42 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
ttm_bo_move_sync_cleanup(bo, dst_mem);
i915_ttm_free_cached_io_st(obj);
- if (!dst_man->use_tt)
+ if (!dst_man->use_tt) {
obj->ttm.cached_io_st = dst_st;
+ obj->ttm.get_io_page.sg_pos = dst_st->sgl;
+ obj->ttm.get_io_page.sg_idx = 0;
+ }
return 0;
}
+static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource *mem)
+{
+ if (mem->mem_type < I915_PL_LMEM0)
+ return 0;
+
+ /* We may need to revisit this later, but this allows all caching to be used in mmap */
+ mem->bus.caching = ttm_cached;
+ mem->bus.is_iomem = true;
+
+ return 0;
+}
+
+static unsigned long i915_ttm_io_mem_pfn(struct ttm_buffer_object *bo,
+ unsigned long page_offset)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ unsigned long base = obj->mm.region->iomap.base - obj->mm.region->region.start;
+ struct scatterlist *sg;
+ unsigned int ofs;
+
+ GEM_WARN_ON(bo->ttm);
+
+ sg = __i915_gem_object_get_sg(obj, &obj->ttm.get_io_page, page_offset, &ofs, true, true);
+
+ return ((base + sg_dma_address(sg)) >> PAGE_SHIFT) + ofs;
+}
+
static struct ttm_device_funcs i915_ttm_bo_driver = {
.ttm_tt_create = i915_ttm_tt_create,
.ttm_tt_unpopulate = i915_ttm_tt_unpopulate,
@@ -354,6 +394,8 @@ static struct ttm_device_funcs i915_ttm_bo_driver = {
.verify_access = NULL,
.swap_notify = i915_ttm_swap_notify,
.delete_mem_notify = i915_ttm_delete_mem_notify,
+ .io_mem_reserve = i915_ttm_io_mem_reserve,
+ .io_mem_pfn = i915_ttm_io_mem_pfn,
};
/**
@@ -461,7 +503,68 @@ static void i915_ttm_delayed_free(struct drm_i915_gem_object *obj)
}
}
-static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
+static vm_fault_t vm_fault_ttm(struct vm_fault *vmf)
+{
+ struct vm_area_struct *area = vmf->vma;
+ struct drm_i915_gem_object *obj =
+ i915_ttm_to_gem(area->vm_private_data);
+
+ /* Sanity check that we allow writing into this object */
+ if (unlikely(i915_gem_object_is_readonly(obj) &&
+ area->vm_flags & VM_WRITE))
+ return VM_FAULT_SIGBUS;
+
+ return ttm_bo_vm_fault(vmf);
+}
+
+static int
+vm_access_ttm(struct vm_area_struct *area, unsigned long addr,
+ void *buf, int len, int write)
+{
+ struct drm_i915_gem_object *obj =
+ i915_ttm_to_gem(area->vm_private_data);
+
+ if (i915_gem_object_is_readonly(obj) && write)
+ return -EACCES;
+
+ return ttm_bo_vm_access(area, addr, buf, len, write);
+}
+
+static void ttm_vm_open(struct vm_area_struct *vma)
+{
+ struct drm_i915_gem_object *obj =
+ i915_ttm_to_gem(vma->vm_private_data);
+
+ GEM_BUG_ON(!obj);
+ i915_gem_object_get(obj);
+}
+
+static void ttm_vm_close(struct vm_area_struct *vma)
+{
+ struct drm_i915_gem_object *obj =
+ i915_ttm_to_gem(vma->vm_private_data);
+
+ GEM_BUG_ON(!obj);
+ i915_gem_object_put(obj);
+}
+
+
+static const struct vm_operations_struct vm_ops_ttm = {
+ .fault = vm_fault_ttm,
+ .access = vm_access_ttm,
+ .open = ttm_vm_open,
+ .close = ttm_vm_close,
+};
+
+static u64 i915_ttm_mmap_offset(struct drm_i915_gem_object *obj)
+{
+ /* The ttm_bo must be allocated with I915_BO_ALLOC_USER */
+ GEM_BUG_ON(!drm_mm_node_allocated(&obj->base.vma_node.vm_node));
+
+ return drm_vma_node_offset_addr(&obj->base.vma_node);
+}
+
+const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
.name = "i915_gem_object_ttm",
.flags = I915_GEM_OBJECT_HAS_IOMEM,
@@ -470,6 +573,8 @@ static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
.truncate = i915_ttm_purge,
.adjust_lru = i915_ttm_adjust_lru,
.delayed_free = i915_ttm_delayed_free,
+ .mmap_offset = i915_ttm_mmap_offset,
+ .mmap_ops = &vm_ops_ttm,
};
void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
@@ -518,6 +623,8 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
i915_gem_object_make_unshrinkable(obj);
obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
+ INIT_RADIX_TREE(&obj->ttm.get_io_page.radix, GFP_KERNEL | __GFP_NOWARN);
+ mutex_init(&obj->ttm.get_io_page.lock);
bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device :
ttm_bo_type_kernel;
@@ -529,6 +636,7 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
* Similarly, in delayed_destroy, we can't call ttm_bo_put()
* until successful initialization.
*/
+ obj->base.vma_node.driver_private = i915_gem_to_ttm(obj);
ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size,
bo_type, &i915_sys_placement, alignment,
true, NULL, NULL, i915_ttm_bo_destroy);
diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
index 05a3b29f545e..ca69a29b7f2a 100644
--- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
@@ -578,16 +578,17 @@ static bool assert_mmap_offset(struct drm_i915_private *i915,
int expected)
{
struct drm_i915_gem_object *obj;
- struct i915_mmap_offset *mmo;
+ u64 offset;
+ int ret;
obj = i915_gem_object_create_internal(i915, size);
if (IS_ERR(obj))
- return false;
+ return expected && expected == PTR_ERR(obj);
- mmo = mmap_offset_attach(obj, I915_MMAP_OFFSET_GTT, NULL);
+ ret = __assign_mmap_offset(obj, I915_MMAP_TYPE_GTT, &offset, NULL);
i915_gem_object_put(obj);
- return PTR_ERR_OR_ZERO(mmo) == expected;
+ return ret == expected;
}
static void disable_retire_worker(struct drm_i915_private *i915)
@@ -622,8 +623,8 @@ static int igt_mmap_offset_exhaustion(void *arg)
struct drm_mm *mm = &i915->drm.vma_offset_manager->vm_addr_space_mm;
struct drm_i915_gem_object *obj;
struct drm_mm_node *hole, *next;
- struct i915_mmap_offset *mmo;
int loop, err = 0;
+ u64 offset;
/* Disable background reaper */
disable_retire_worker(i915);
@@ -684,13 +685,13 @@ static int igt_mmap_offset_exhaustion(void *arg)
obj = i915_gem_object_create_internal(i915, PAGE_SIZE);
if (IS_ERR(obj)) {
err = PTR_ERR(obj);
+ pr_err("Unable to create object for reclaimed hole\n");
goto out;
}
- mmo = mmap_offset_attach(obj, I915_MMAP_OFFSET_GTT, NULL);
- if (IS_ERR(mmo)) {
+ err = __assign_mmap_offset(obj, I915_MMAP_TYPE_GTT, &offset, NULL);
+ if (err) {
pr_err("Unable to insert object into reclaimed hole\n");
- err = PTR_ERR(mmo);
goto err_obj;
}
@@ -865,10 +866,10 @@ static int __igt_mmap(struct drm_i915_private *i915,
struct drm_i915_gem_object *obj,
enum i915_mmap_type type)
{
- struct i915_mmap_offset *mmo;
struct vm_area_struct *area;
unsigned long addr;
int err, i;
+ u64 offset;
if (!can_mmap(obj, type))
return 0;
@@ -879,11 +880,11 @@ static int __igt_mmap(struct drm_i915_private *i915,
if (err)
return err;
- mmo = mmap_offset_attach(obj, type, NULL);
- if (IS_ERR(mmo))
- return PTR_ERR(mmo);
+ err = __assign_mmap_offset(obj, type, &offset, NULL);
+ if (err)
+ return err;
- addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
+ addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
if (IS_ERR_VALUE(addr))
return addr;
@@ -897,13 +898,6 @@ static int __igt_mmap(struct drm_i915_private *i915,
goto out_unmap;
}
- if (area->vm_private_data != mmo) {
- pr_err("%s: vm_area_struct did not point back to our mmap_offset object!\n",
- obj->mm.region->name);
- err = -EINVAL;
- goto out_unmap;
- }
-
for (i = 0; i < obj->base.size / sizeof(u32); i++) {
u32 __user *ux = u64_to_user_ptr((u64)(addr + i * sizeof(*ux)));
u32 x;
@@ -961,7 +955,7 @@ static int igt_mmap(void *arg)
struct drm_i915_gem_object *obj;
int err;
- obj = i915_gem_object_create_region(mr, sizes[i], 0);
+ obj = i915_gem_object_create_region(mr, sizes[i], I915_BO_ALLOC_USER);
if (obj == ERR_PTR(-ENODEV))
continue;
@@ -1004,12 +998,12 @@ static int __igt_mmap_access(struct drm_i915_private *i915,
struct drm_i915_gem_object *obj,
enum i915_mmap_type type)
{
- struct i915_mmap_offset *mmo;
unsigned long __user *ptr;
unsigned long A, B;
unsigned long x, y;
unsigned long addr;
int err;
+ u64 offset;
memset(&A, 0xAA, sizeof(A));
memset(&B, 0xBB, sizeof(B));
@@ -1017,11 +1011,11 @@ static int __igt_mmap_access(struct drm_i915_private *i915,
if (!can_mmap(obj, type) || !can_access(obj))
return 0;
- mmo = mmap_offset_attach(obj, type, NULL);
- if (IS_ERR(mmo))
- return PTR_ERR(mmo);
+ err = __assign_mmap_offset(obj, type, &offset, NULL);
+ if (err)
+ return err;
- addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
+ addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
if (IS_ERR_VALUE(addr))
return addr;
ptr = (unsigned long __user *)addr;
@@ -1081,7 +1075,7 @@ static int igt_mmap_access(void *arg)
struct drm_i915_gem_object *obj;
int err;
- obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
+ obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
if (obj == ERR_PTR(-ENODEV))
continue;
@@ -1111,11 +1105,11 @@ static int __igt_mmap_gpu(struct drm_i915_private *i915,
enum i915_mmap_type type)
{
struct intel_engine_cs *engine;
- struct i915_mmap_offset *mmo;
unsigned long addr;
u32 __user *ux;
u32 bbe;
int err;
+ u64 offset;
/*
* Verify that the mmap access into the backing store aligns with
@@ -1132,11 +1126,11 @@ static int __igt_mmap_gpu(struct drm_i915_private *i915,
if (err)
return err;
- mmo = mmap_offset_attach(obj, type, NULL);
- if (IS_ERR(mmo))
- return PTR_ERR(mmo);
+ err = __assign_mmap_offset(obj, type, &offset, NULL);
+ if (err)
+ return err;
- addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
+ addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
if (IS_ERR_VALUE(addr))
return addr;
@@ -1226,7 +1220,7 @@ static int igt_mmap_gpu(void *arg)
struct drm_i915_gem_object *obj;
int err;
- obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
+ obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
if (obj == ERR_PTR(-ENODEV))
continue;
@@ -1303,18 +1297,18 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
struct drm_i915_gem_object *obj,
enum i915_mmap_type type)
{
- struct i915_mmap_offset *mmo;
unsigned long addr;
int err;
+ u64 offset;
if (!can_mmap(obj, type))
return 0;
- mmo = mmap_offset_attach(obj, type, NULL);
- if (IS_ERR(mmo))
- return PTR_ERR(mmo);
+ err = __assign_mmap_offset(obj, type, &offset, NULL);
+ if (err)
+ return err;
- addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
+ addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
if (IS_ERR_VALUE(addr))
return addr;
@@ -1350,10 +1344,20 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
}
}
- err = check_absent(addr, obj->base.size);
- if (err) {
- pr_err("%s: was not absent\n", obj->mm.region->name);
- goto out_unmap;
+ if (!obj->ops->mmap_ops) {
+ err = check_absent(addr, obj->base.size);
+ if (err) {
+ pr_err("%s: was not absent\n", obj->mm.region->name);
+ goto out_unmap;
+ }
+ } else {
+ /* ttm allows access to evicted regions by design */
+
+ err = check_present(addr, obj->base.size);
+ if (err) {
+ pr_err("%s: was not present\n", obj->mm.region->name);
+ goto out_unmap;
+ }
}
out_unmap:
@@ -1371,7 +1375,7 @@ static int igt_mmap_revoke(void *arg)
struct drm_i915_gem_object *obj;
int err;
- obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
+ obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
if (obj == ERR_PTR(-ENODEV))
continue;
diff --git a/drivers/gpu/drm/i915/selftests/igt_mmap.c b/drivers/gpu/drm/i915/selftests/igt_mmap.c
index 583a4ff8b8c9..e8286c28de91 100644
--- a/drivers/gpu/drm/i915/selftests/igt_mmap.c
+++ b/drivers/gpu/drm/i915/selftests/igt_mmap.c
@@ -9,15 +9,28 @@
#include "i915_drv.h"
#include "igt_mmap.h"
-unsigned long igt_mmap_node(struct drm_i915_private *i915,
- struct drm_vma_offset_node *node,
- unsigned long addr,
- unsigned long prot,
- unsigned long flags)
+unsigned long igt_mmap_offset(struct drm_i915_private *i915,
+ u64 offset,
+ unsigned long size,
+ unsigned long prot,
+ unsigned long flags)
{
+ struct drm_vma_offset_node *node;
struct file *file;
+ unsigned long addr;
int err;
+ /* no need to refcount, we own this object */
+ drm_vma_offset_lock_lookup(i915->drm.vma_offset_manager);
+ node = drm_vma_offset_exact_lookup_locked(i915->drm.vma_offset_manager,
+ offset / PAGE_SIZE, size / PAGE_SIZE);
+ drm_vma_offset_unlock_lookup(i915->drm.vma_offset_manager);
+
+ if (GEM_WARN_ON(!node)) {
+ pr_info("Failed to lookup %Lx\n", offset);
+ return -ENOENT;
+ }
+
/* Pretend to open("/dev/dri/card0") */
file = mock_drm_getfile(i915->drm.primary, O_RDWR);
if (IS_ERR(file))
@@ -29,7 +42,7 @@ unsigned long igt_mmap_node(struct drm_i915_private *i915,
goto out_file;
}
- addr = vm_mmap(file, addr, drm_vma_node_size(node) << PAGE_SHIFT,
+ addr = vm_mmap(file, 0, drm_vma_node_size(node) << PAGE_SHIFT,
prot, flags, drm_vma_node_offset_addr(node));
drm_vma_node_revoke(node, file->private_data);
diff --git a/drivers/gpu/drm/i915/selftests/igt_mmap.h b/drivers/gpu/drm/i915/selftests/igt_mmap.h
index 6e716cb59d7e..acbe34d81a6d 100644
--- a/drivers/gpu/drm/i915/selftests/igt_mmap.h
+++ b/drivers/gpu/drm/i915/selftests/igt_mmap.h
@@ -7,13 +7,15 @@
#ifndef IGT_MMAP_H
#define IGT_MMAP_H
+#include <linux/types.h>
+
struct drm_i915_private;
struct drm_vma_offset_node;
-unsigned long igt_mmap_node(struct drm_i915_private *i915,
- struct drm_vma_offset_node *node,
- unsigned long addr,
- unsigned long prot,
- unsigned long flags);
+unsigned long igt_mmap_offset(struct drm_i915_private *i915,
+ u64 offset,
+ unsigned long size,
+ unsigned long prot,
+ unsigned long flags);
#endif /* IGT_MMAP_H */
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 58+ messages in thread
* Re: [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 13:26 ` Christian König
-1 siblings, 0 replies; 58+ messages in thread
From: Christian König @ 2021-05-26 13:26 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel
Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> We are calling the eviction_valuable driver callback at eviction time to
> determine whether we actually can evict a buffer object.
> The upcoming i915 TTM backend needs the same functionality for swapout,
> and that might actually be beneficial to other drivers as well.
>
> Add an eviction_valuable call also in the swapout path. Try to keep the
> current behaviour for all drivers by returning true if the buffer object
> is already in the TTM_PL_SYSTEM placement. We change behaviour for the
> case where a buffer object is in a TT backed placement when swapped out,
> in which case the drivers normal eviction_valuable path is run.
>
> Finally make sure we don't try to swapout a bo that was recently purged
> and therefore unpopulated.
>
> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> v3:
> - Don't export ttm_tt_unpopulate
> - Fix confusion reading the locked pointer instead of the value
> pointed to in ttm_bo_evict_swapout_allowable (Reported by
> Maarten Lankhorst)
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
> drivers/gpu/drm/ttm/ttm_bo.c | 43 ++++++++++++++++---------
> drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
> 3 files changed, 34 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> index 3bc3aebfef7c..45d194bffc3f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> @@ -1348,6 +1348,10 @@ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> struct dma_fence *f;
> int i;
>
> + /* Swapout? */
> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> + return true;
> +
> if (bo->type == ttm_bo_type_kernel &&
> !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
> return false;
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index be0406466460..1b2d062266ed 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
> bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> const struct ttm_place *place)
> {
> + dma_resv_assert_held(bo->base.resv);
> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> + return true;
> +
> /* Don't evict this BO if it's outside of the
> * requested placement range
> */
> @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
> * b. Otherwise, trylock it.
> */
> static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
> - struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
> + struct ttm_operation_ctx *ctx,
> + const struct ttm_place *place,
> + bool *locked, bool *busy)
> {
> bool ret = false;
>
> @@ -576,6 +582,14 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
> *busy = !ret;
> }
>
> + if (ret && place && !bo->bdev->funcs->eviction_valuable(bo, place)) {
> + ret = false;
> + if (*locked) {
> + dma_resv_unlock(bo->base.resv);
> + *locked = false;
> + }
> + }
> +
> return ret;
> }
>
> @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
> list_for_each_entry(bo, &man->lru[i], lru) {
> bool busy;
>
> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
> - &busy)) {
> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
> + &locked, &busy)) {
> if (busy && !busy_bo && ticket !=
> dma_resv_locking_ctx(bo->base.resv))
> busy_bo = bo;
> continue;
> }
>
> - if (place && !bdev->funcs->eviction_valuable(bo,
> - place)) {
> - if (locked)
> - dma_resv_unlock(bo->base.resv);
> - continue;
> - }
> if (!ttm_bo_get_unless_zero(bo)) {
> if (locked)
> dma_resv_unlock(bo->base.resv);
> @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
> int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
> gfp_t gfp_flags)
> {
> + struct ttm_place place = {};
> bool locked;
> int ret;
>
> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
> + /*
> + * While the bo may already reside in SYSTEM placement, set
> + * SYSTEM as new placement to cover also the move further below.
> + * The driver may use the fact that we're moving from SYSTEM
> + * as an indication that we're about to swap out.
> + */
> + place.mem_type = TTM_PL_SYSTEM;
> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
> return -EBUSY;
>
> if (!ttm_bo_get_unless_zero(bo)) {
> @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
> if (bo->mem.mem_type != TTM_PL_SYSTEM) {
> struct ttm_operation_ctx ctx = { false, false };
> struct ttm_resource evict_mem;
> - struct ttm_place place, hop;
> -
> - memset(&place, 0, sizeof(place));
> - memset(&hop, 0, sizeof(hop));
> -
> - place.mem_type = TTM_PL_SYSTEM;
> + struct ttm_place hop = {};
I would stick with memset because of the padding reasons.
>
> ret = ttm_resource_alloc(bo, &place, &evict_mem);
> if (unlikely(ret))
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index 913b330a234b..d9793cbb6d13 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,
> struct page *to_page;
> int i, ret;
>
> + if (!ttm_tt_is_populated(ttm))
> + return 0;
> +
This here is just because of a bug in the higher level function.
I've just pushed the fix for that to drm-misc-fixes, so maybe drop that
here as soon as this is backmerged.
Apart from that patch looks good to me.
Christian.
> swap_storage = shmem_file_setup("ttm swap", size, 0);
> if (IS_ERR(swap_storage)) {
> pr_err("Failed allocating swap storage\n");
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
@ 2021-05-26 13:26 ` Christian König
0 siblings, 0 replies; 58+ messages in thread
From: Christian König @ 2021-05-26 13:26 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel
Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> We are calling the eviction_valuable driver callback at eviction time to
> determine whether we actually can evict a buffer object.
> The upcoming i915 TTM backend needs the same functionality for swapout,
> and that might actually be beneficial to other drivers as well.
>
> Add an eviction_valuable call also in the swapout path. Try to keep the
> current behaviour for all drivers by returning true if the buffer object
> is already in the TTM_PL_SYSTEM placement. We change behaviour for the
> case where a buffer object is in a TT backed placement when swapped out,
> in which case the drivers normal eviction_valuable path is run.
>
> Finally make sure we don't try to swapout a bo that was recently purged
> and therefore unpopulated.
>
> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> v3:
> - Don't export ttm_tt_unpopulate
> - Fix confusion reading the locked pointer instead of the value
> pointed to in ttm_bo_evict_swapout_allowable (Reported by
> Maarten Lankhorst)
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
> drivers/gpu/drm/ttm/ttm_bo.c | 43 ++++++++++++++++---------
> drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
> 3 files changed, 34 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> index 3bc3aebfef7c..45d194bffc3f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> @@ -1348,6 +1348,10 @@ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> struct dma_fence *f;
> int i;
>
> + /* Swapout? */
> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> + return true;
> +
> if (bo->type == ttm_bo_type_kernel &&
> !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
> return false;
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index be0406466460..1b2d062266ed 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
> bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> const struct ttm_place *place)
> {
> + dma_resv_assert_held(bo->base.resv);
> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> + return true;
> +
> /* Don't evict this BO if it's outside of the
> * requested placement range
> */
> @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
> * b. Otherwise, trylock it.
> */
> static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
> - struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
> + struct ttm_operation_ctx *ctx,
> + const struct ttm_place *place,
> + bool *locked, bool *busy)
> {
> bool ret = false;
>
> @@ -576,6 +582,14 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
> *busy = !ret;
> }
>
> + if (ret && place && !bo->bdev->funcs->eviction_valuable(bo, place)) {
> + ret = false;
> + if (*locked) {
> + dma_resv_unlock(bo->base.resv);
> + *locked = false;
> + }
> + }
> +
> return ret;
> }
>
> @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
> list_for_each_entry(bo, &man->lru[i], lru) {
> bool busy;
>
> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
> - &busy)) {
> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
> + &locked, &busy)) {
> if (busy && !busy_bo && ticket !=
> dma_resv_locking_ctx(bo->base.resv))
> busy_bo = bo;
> continue;
> }
>
> - if (place && !bdev->funcs->eviction_valuable(bo,
> - place)) {
> - if (locked)
> - dma_resv_unlock(bo->base.resv);
> - continue;
> - }
> if (!ttm_bo_get_unless_zero(bo)) {
> if (locked)
> dma_resv_unlock(bo->base.resv);
> @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
> int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
> gfp_t gfp_flags)
> {
> + struct ttm_place place = {};
> bool locked;
> int ret;
>
> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
> + /*
> + * While the bo may already reside in SYSTEM placement, set
> + * SYSTEM as new placement to cover also the move further below.
> + * The driver may use the fact that we're moving from SYSTEM
> + * as an indication that we're about to swap out.
> + */
> + place.mem_type = TTM_PL_SYSTEM;
> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
> return -EBUSY;
>
> if (!ttm_bo_get_unless_zero(bo)) {
> @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
> if (bo->mem.mem_type != TTM_PL_SYSTEM) {
> struct ttm_operation_ctx ctx = { false, false };
> struct ttm_resource evict_mem;
> - struct ttm_place place, hop;
> -
> - memset(&place, 0, sizeof(place));
> - memset(&hop, 0, sizeof(hop));
> -
> - place.mem_type = TTM_PL_SYSTEM;
> + struct ttm_place hop = {};
I would stick with memset because of the padding reasons.
>
> ret = ttm_resource_alloc(bo, &place, &evict_mem);
> if (unlikely(ret))
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index 913b330a234b..d9793cbb6d13 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,
> struct page *to_page;
> int i, ret;
>
> + if (!ttm_tt_is_populated(ttm))
> + return 0;
> +
This here is just because of a bug in the higher level function.
I've just pushed the fix for that to drm-misc-fixes, so maybe drop that
here as soon as this is backmerged.
Apart from that patch looks good to me.
Christian.
> swap_storage = shmem_file_setup("ttm swap", size, 0);
> if (IS_ERR(swap_storage)) {
> pr_err("Failed allocating swap storage\n");
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 07/15] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 14:27 ` Christian König
-1 siblings, 0 replies; 58+ messages in thread
From: Christian König @ 2021-05-26 14:27 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel; +Cc: Daniel Vetter, Matthew Auld
Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> Memcpy from wc will be used as well by TTM memcpy.
> Move it to core drm, and make the interface do the right thing
> even on !X86.
>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Cc: Dave Airlie <airlied@gmail.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Acked-by: Christian König <christian.koenig@amd.com>
> ---
> v4:
> - Fix !X86 path (Reported by Matthew Auld)
> ---
> drivers/gpu/drm/Makefile | 2 +-
> drivers/gpu/drm/drm_drv.c | 2 +
> .../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 ++++++++++++-----
> drivers/gpu/drm/i915/Makefile | 1 -
> .../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
> drivers/gpu/drm/i915/gem/i915_gem_object.c | 5 +-
> drivers/gpu/drm/i915/gt/selftest_reset.c | 7 +-
> drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 +--
> drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
> drivers/gpu/drm/i915/i915_drv.c | 2 -
> drivers/gpu/drm/i915/i915_gpu_error.c | 8 +--
> drivers/gpu/drm/i915/i915_memcpy.h | 34 ----------
> .../drm/i915/selftests/intel_memory_region.c | 7 +-
> include/drm/drm_memcpy.h | 68 +++++++++++++++++++
> 14 files changed, 142 insertions(+), 76 deletions(-)
> rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
> delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
> create mode 100644 include/drm/drm_memcpy.h
>
> diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
> index a91cc7684904..f3ab8586c3d7 100644
> --- a/drivers/gpu/drm/Makefile
> +++ b/drivers/gpu/drm/Makefile
> @@ -18,7 +18,7 @@ drm-y := drm_aperture.o drm_auth.o drm_cache.o \
> drm_dumb_buffers.o drm_mode_config.o drm_vblank.o \
> drm_syncobj.o drm_lease.o drm_writeback.o drm_client.o \
> drm_client_modeset.o drm_atomic_uapi.o drm_hdcp.o \
> - drm_managed.o drm_vblank_work.o
> + drm_managed.o drm_vblank_work.o drm_memcpy.o \
>
> drm-$(CONFIG_DRM_LEGACY) += drm_agpsupport.o drm_bufs.o drm_context.o drm_dma.o \
> drm_legacy_misc.o drm_lock.o drm_memory.o drm_scatter.o \
> diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> index 3d8d68a98b95..351cc2900cf1 100644
> --- a/drivers/gpu/drm/drm_drv.c
> +++ b/drivers/gpu/drm/drm_drv.c
> @@ -40,6 +40,7 @@
> #include <drm/drm_drv.h>
> #include <drm/drm_file.h>
> #include <drm/drm_managed.h>
> +#include <drm/drm_memcpy.h>
> #include <drm/drm_mode_object.h>
> #include <drm/drm_print.h>
>
> @@ -1041,6 +1042,7 @@ static int __init drm_core_init(void)
>
> drm_connector_ida_init();
> idr_init(&drm_minors_idr);
> + drm_memcpy_init_early();
>
> ret = drm_sysfs_init();
> if (ret < 0) {
> diff --git a/drivers/gpu/drm/i915/i915_memcpy.c b/drivers/gpu/drm/drm_memcpy.c
> similarity index 70%
> rename from drivers/gpu/drm/i915/i915_memcpy.c
> rename to drivers/gpu/drm/drm_memcpy.c
> index 1b021a4902de..740377749caa 100644
> --- a/drivers/gpu/drm/i915/i915_memcpy.c
> +++ b/drivers/gpu/drm/drm_memcpy.c
> @@ -1,3 +1,4 @@
> +// SPDX-License-Identifier: MIT
> /*
> * Copyright © 2016 Intel Corporation
> *
> @@ -22,16 +23,12 @@
> *
> */
>
> +#ifdef CONFIG_X86
> +#include <linux/dma-buf-map.h>
> #include <linux/kernel.h>
> #include <asm/fpu/api.h>
>
> -#include "i915_memcpy.h"
> -
> -#if IS_ENABLED(CONFIG_DRM_I915_DEBUG)
> -#define CI_BUG_ON(expr) BUG_ON(expr)
> -#else
> -#define CI_BUG_ON(expr) BUILD_BUG_ON_INVALID(expr)
> -#endif
> +#include "drm/drm_memcpy.h"
>
> static DEFINE_STATIC_KEY_FALSE(has_movntdqa);
>
> @@ -94,23 +91,24 @@ static void __memcpy_ntdqu(void *dst, const void *src, unsigned long len)
> }
>
> /**
> - * i915_memcpy_from_wc: perform an accelerated *aligned* read from WC
> + * drm_memcpy_from_wc: perform an accelerated *aligned* read from WC
> * @dst: destination pointer
> * @src: source pointer
> * @len: how many bytes to copy
> *
> - * i915_memcpy_from_wc copies @len bytes from @src to @dst using
> + * drm_memcpy_from_wc copies @len bytes from @src to @dst using
> * non-temporal instructions where available. Note that all arguments
> * (@src, @dst) must be aligned to 16 bytes and @len must be a multiple
> * of 16.
> *
> * To test whether accelerated reads from WC are supported, use
> - * i915_memcpy_from_wc(NULL, NULL, 0);
> + * drm_memcpy_from_wc(NULL, NULL, 0);
> + * This interface is intended for memremapped memory without the __iomem tag.
> *
> * Returns true if the copy was successful, false if the preconditions
> * are not met.
> */
> -bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> {
> if (unlikely(((unsigned long)dst | (unsigned long)src | len) & 15))
> return false;
> @@ -123,24 +121,53 @@ bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
>
> return false;
> }
> +EXPORT_SYMBOL(drm_memcpy_from_wc);
>
> /**
> - * i915_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
> + * drm_memcpy_from_wc_dbm: perform an accelerated *aligned* read from WC with
> + * struct dma_buf_map arguments.
> + * @dst: destination map
> + * @src: source map
> + * @len: how many bytes to copy
> + *
> + * This is identical to drm_memcpy_from_wc, except it's intended for
> + * potentially ioremapped memory rather than memremapped memory.
> + *
> + * Returns true if the copy was successful, false if the preconditions
> + * are not met.
> + */
> +bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
> + const struct dma_buf_map *src,
> + unsigned long len)
> +{
> + /* For X86 we can safely drop __iomem */
> + return drm_memcpy_from_wc(dst->is_iomem ?
> + (void __force *)dst->vaddr_iomem :
> + dst->vaddr,
> + src->is_iomem ?
> + (void const __force *)src->vaddr_iomem :
> + src->vaddr,
> + len);
> +}
> +EXPORT_SYMBOL(drm_memcpy_from_wc_dbm);
> +
> +/**
> + * drm_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
> * @dst: destination pointer
> * @src: source pointer
> * @len: how many bytes to copy
> *
> - * Like i915_memcpy_from_wc(), the unaligned variant copies @len bytes from
> + * Like drm_memcpy_from_wc(), the unaligned variant copies @len bytes from
> * @src to @dst using * non-temporal instructions where available, but
> * accepts that its arguments may not be aligned, but are valid for the
> * potential 16-byte read past the end.
> + *
> + * This interface is intended for mremapped memory without the __iomem tag.
> */
> -void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> +void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> {
> unsigned long addr;
>
> - CI_BUG_ON(!i915_has_memcpy_from_wc());
> -
> addr = (unsigned long)src;
> if (!IS_ALIGNED(addr, 16)) {
> unsigned long x = min(ALIGN(addr, 16) - addr, len);
> @@ -155,8 +182,9 @@ void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len
> if (likely(len))
> __memcpy_ntdqu(dst, src, DIV_ROUND_UP(len, 16));
> }
> +EXPORT_SYMBOL(drm_unaligned_memcpy_from_wc);
>
> -void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
> +void drm_memcpy_init_early(void)
> {
> /*
> * Some hypervisors (e.g. KVM) don't support VEX-prefix instructions
> @@ -166,3 +194,4 @@ void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
> !boot_cpu_has(X86_FEATURE_HYPERVISOR))
> static_branch_enable(&has_movntdqa);
> }
> +#endif
> diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
> index 4f22cac1c49b..ebc19bd5fff4 100644
> --- a/drivers/gpu/drm/i915/Makefile
> +++ b/drivers/gpu/drm/i915/Makefile
> @@ -61,7 +61,6 @@ i915-y += i915_drv.o \
> # core library code
> i915-y += \
> dma_resv_utils.o \
> - i915_memcpy.o \
> i915_mm.o \
> i915_sw_fence.o \
> i915_sw_fence_work.o \
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
> index 297143511f99..77285e421fb8 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
> @@ -10,6 +10,7 @@
> #include <linux/uaccess.h>
>
> #include <drm/drm_syncobj.h>
> +#include <drm/drm_memcpy.h>
>
> #include "display/intel_frontbuffer.h"
>
> @@ -28,7 +29,6 @@
> #include "i915_sw_fence_work.h"
> #include "i915_trace.h"
> #include "i915_user_extensions.h"
> -#include "i915_memcpy.h"
>
> struct eb_vma {
> struct i915_vma *vma;
> @@ -2503,7 +2503,7 @@ static int eb_parse_pipeline(struct i915_execbuffer *eb,
> !(batch->cache_coherent & I915_BO_CACHE_COHERENT_FOR_READ);
>
> pw->batch_map = ERR_PTR(-ENODEV);
> - if (needs_clflush && i915_has_memcpy_from_wc())
> + if (needs_clflush && drm_has_memcpy_from_wc())
> pw->batch_map = i915_gem_object_pin_map(batch, I915_MAP_WC);
>
> if (IS_ERR(pw->batch_map)) {
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
> index 5706d471692d..e9247afb0320 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
> @@ -24,6 +24,8 @@
>
> #include <linux/sched/mm.h>
>
> +#include <drm/drm_memcpy.h>
> +
> #include "display/intel_frontbuffer.h"
> #include "i915_drv.h"
> #include "i915_gem_clflush.h"
> @@ -31,7 +33,6 @@
> #include "i915_gem_mman.h"
> #include "i915_gem_object.h"
> #include "i915_globals.h"
> -#include "i915_memcpy.h"
> #include "i915_trace.h"
>
> static struct i915_global_object {
> @@ -374,7 +375,7 @@ i915_gem_object_read_from_page_iomap(struct drm_i915_gem_object *obj, u64 offset
> PAGE_SIZE);
>
> src_ptr = src_map + offset_in_page(offset);
> - if (!i915_memcpy_from_wc(dst, (void __force *)src_ptr, size))
> + if (!drm_memcpy_from_wc(dst, (void __force *)src_ptr, size))
> memcpy_fromio(dst, src_ptr, size);
>
> io_mapping_unmap(src_map);
> diff --git a/drivers/gpu/drm/i915/gt/selftest_reset.c b/drivers/gpu/drm/i915/gt/selftest_reset.c
> index 8784257ec808..92ada67a3835 100644
> --- a/drivers/gpu/drm/i915/gt/selftest_reset.c
> +++ b/drivers/gpu/drm/i915/gt/selftest_reset.c
> @@ -5,9 +5,10 @@
>
> #include <linux/crc32.h>
>
> +#include <drm/drm_memcpy.h>
> +
> #include "gem/i915_gem_stolen.h"
>
> -#include "i915_memcpy.h"
> #include "i915_selftest.h"
> #include "intel_gpu_commands.h"
> #include "selftests/igt_reset.h"
> @@ -99,7 +100,7 @@ __igt_reset_stolen(struct intel_gt *gt,
> memset_io(s, STACK_MAGIC, PAGE_SIZE);
>
> in = (void __force *)s;
> - if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
> + if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
> in = tmp;
> crc[page] = crc32_le(0, in, PAGE_SIZE);
>
> @@ -135,7 +136,7 @@ __igt_reset_stolen(struct intel_gt *gt,
> PAGE_SIZE);
>
> in = (void __force *)s;
> - if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
> + if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
> in = tmp;
> x = crc32_le(0, in, PAGE_SIZE);
>
> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
> index c36d5eb5bbb9..f045e42be6ca 100644
> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
> @@ -5,9 +5,10 @@
>
> #include <linux/debugfs.h>
>
> +#include <drm/drm_memcpy.h>
> +
> #include "gt/intel_gt.h"
> #include "i915_drv.h"
> -#include "i915_memcpy.h"
> #include "intel_guc_log.h"
>
> static void guc_log_capture_logs(struct intel_guc_log *log);
> @@ -295,13 +296,13 @@ static void guc_read_update_log_buffer(struct intel_guc_log *log)
>
> /* Just copy the newly written data */
> if (read_offset > write_offset) {
> - i915_memcpy_from_wc(dst_data, src_data, write_offset);
> + drm_memcpy_from_wc(dst_data, src_data, write_offset);
> bytes_to_copy = buffer_size - read_offset;
> } else {
> bytes_to_copy = write_offset - read_offset;
> }
> - i915_memcpy_from_wc(dst_data + read_offset,
> - src_data + read_offset, bytes_to_copy);
> + drm_memcpy_from_wc(dst_data + read_offset,
> + src_data + read_offset, bytes_to_copy);
>
> src_data += buffer_size;
> dst_data += buffer_size;
> @@ -569,7 +570,7 @@ int intel_guc_log_relay_open(struct intel_guc_log *log)
> * it should be present on the chipsets supporting GuC based
> * submisssions.
> */
> - if (!i915_has_memcpy_from_wc()) {
> + if (!drm_has_memcpy_from_wc()) {
> ret = -ENXIO;
> goto out_unlock;
> }
> diff --git a/drivers/gpu/drm/i915/i915_cmd_parser.c b/drivers/gpu/drm/i915/i915_cmd_parser.c
> index 5b4b2bd46e7c..98653f1a2b1d 100644
> --- a/drivers/gpu/drm/i915/i915_cmd_parser.c
> +++ b/drivers/gpu/drm/i915/i915_cmd_parser.c
> @@ -24,12 +24,12 @@
> * Brad Volkin <bradley.d.volkin@intel.com>
> *
> */
> +#include <drm/drm_memcpy.h>
>
> #include "gt/intel_engine.h"
> #include "gt/intel_gpu_commands.h"
>
> #include "i915_drv.h"
> -#include "i915_memcpy.h"
>
> /**
> * DOC: batch buffer command parser
> @@ -1152,7 +1152,7 @@ static u32 *copy_batch(struct drm_i915_gem_object *dst_obj,
>
> if (src) {
> GEM_BUG_ON(!needs_clflush);
> - i915_unaligned_memcpy_from_wc(dst, src + offset, length);
> + drm_unaligned_memcpy_from_wc(dst, src + offset, length);
> } else {
> struct scatterlist *sg;
> void *ptr;
> diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
> index 30c349137be2..68639ed0bdec 100644
> --- a/drivers/gpu/drm/i915/i915_drv.c
> +++ b/drivers/gpu/drm/i915/i915_drv.c
> @@ -72,7 +72,6 @@
> #include "i915_drv.h"
> #include "i915_ioc32.h"
> #include "i915_irq.h"
> -#include "i915_memcpy.h"
> #include "i915_perf.h"
> #include "i915_query.h"
> #include "i915_suspend.h"
> @@ -325,7 +324,6 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
> mutex_init(&dev_priv->pps_mutex);
> mutex_init(&dev_priv->hdcp_comp_mutex);
>
> - i915_memcpy_init_early(dev_priv);
> intel_runtime_pm_init_early(&dev_priv->runtime_pm);
>
> ret = i915_workqueues_init(dev_priv);
> diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
> index 8b964e355cb5..7c1b44545bab 100644
> --- a/drivers/gpu/drm/i915/i915_gpu_error.c
> +++ b/drivers/gpu/drm/i915/i915_gpu_error.c
> @@ -34,6 +34,7 @@
> #include <linux/utsname.h>
> #include <linux/zlib.h>
>
> +#include <drm/drm_memcpy.h>
> #include <drm/drm_print.h>
>
> #include "display/intel_dmc.h"
> @@ -46,7 +47,6 @@
>
> #include "i915_drv.h"
> #include "i915_gpu_error.h"
> -#include "i915_memcpy.h"
> #include "i915_scatterlist.h"
>
> #define ALLOW_FAIL (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN)
> @@ -255,7 +255,7 @@ static bool compress_init(struct i915_vma_compress *c)
> }
>
> c->tmp = NULL;
> - if (i915_has_memcpy_from_wc())
> + if (drm_has_memcpy_from_wc())
> c->tmp = pool_alloc(&c->pool, ALLOW_FAIL);
>
> return true;
> @@ -295,7 +295,7 @@ static int compress_page(struct i915_vma_compress *c,
> struct z_stream_s *zstream = &c->zstream;
>
> zstream->next_in = src;
> - if (wc && c->tmp && i915_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
> + if (wc && c->tmp && drm_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
> zstream->next_in = c->tmp;
> zstream->avail_in = PAGE_SIZE;
>
> @@ -395,7 +395,7 @@ static int compress_page(struct i915_vma_compress *c,
> if (!ptr)
> return -ENOMEM;
>
> - if (!(wc && i915_memcpy_from_wc(ptr, src, PAGE_SIZE)))
> + if (!(wc && drm_memcpy_from_wc(ptr, src, PAGE_SIZE)))
> memcpy(ptr, src, PAGE_SIZE);
> dst->pages[dst->page_count++] = ptr;
> cond_resched();
> diff --git a/drivers/gpu/drm/i915/i915_memcpy.h b/drivers/gpu/drm/i915/i915_memcpy.h
> deleted file mode 100644
> index 3df063a3293b..000000000000
> --- a/drivers/gpu/drm/i915/i915_memcpy.h
> +++ /dev/null
> @@ -1,34 +0,0 @@
> -/* SPDX-License-Identifier: MIT */
> -/*
> - * Copyright © 2019 Intel Corporation
> - */
> -
> -#ifndef __I915_MEMCPY_H__
> -#define __I915_MEMCPY_H__
> -
> -#include <linux/types.h>
> -
> -struct drm_i915_private;
> -
> -void i915_memcpy_init_early(struct drm_i915_private *i915);
> -
> -bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> -void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> -
> -/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
> - * as well as SSE4.1 support. i915_memcpy_from_wc() will report if it cannot
> - * perform the operation. To check beforehand, pass in the parameters to
> - * to i915_can_memcpy_from_wc() - since we only care about the low 4 bits,
> - * you only need to pass in the minor offsets, page-aligned pointers are
> - * always valid.
> - *
> - * For just checking for SSE4.1, in the foreknowledge that the future use
> - * will be correctly aligned, just use i915_has_memcpy_from_wc().
> - */
> -#define i915_can_memcpy_from_wc(dst, src, len) \
> - i915_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
> -
> -#define i915_has_memcpy_from_wc() \
> - i915_memcpy_from_wc(NULL, NULL, 0)
> -
> -#endif /* __I915_MEMCPY_H__ */
> diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
> index c85d516b85cd..6bb399e9be78 100644
> --- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
> +++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
> @@ -6,6 +6,8 @@
> #include <linux/prime_numbers.h>
> #include <linux/sort.h>
>
> +#include <drm/drm_memcpy.h>
> +
> #include "../i915_selftest.h"
>
> #include "mock_drm.h"
> @@ -20,7 +22,6 @@
> #include "gem/selftests/mock_context.h"
> #include "gt/intel_engine_user.h"
> #include "gt/intel_gt.h"
> -#include "i915_memcpy.h"
> #include "selftests/igt_flush_test.h"
> #include "selftests/i915_random.h"
>
> @@ -901,7 +902,7 @@ static inline void igt_memcpy(void *dst, const void *src, size_t size)
>
> static inline void igt_memcpy_from_wc(void *dst, const void *src, size_t size)
> {
> - i915_memcpy_from_wc(dst, src, size);
> + drm_memcpy_from_wc(dst, src, size);
> }
>
> static int _perf_memcpy(struct intel_memory_region *src_mr,
> @@ -925,7 +926,7 @@ static int _perf_memcpy(struct intel_memory_region *src_mr,
> {
> "memcpy_from_wc",
> igt_memcpy_from_wc,
> - !i915_has_memcpy_from_wc(),
> + !drm_has_memcpy_from_wc(),
> },
> };
> struct drm_i915_gem_object *src, *dst;
> diff --git a/include/drm/drm_memcpy.h b/include/drm/drm_memcpy.h
> new file mode 100644
> index 000000000000..fe5ed1e89ce6
> --- /dev/null
> +++ b/include/drm/drm_memcpy.h
> @@ -0,0 +1,68 @@
> +/* SPDX-License-Identifier: MIT */
> +/*
> + * Copyright © 2019 Intel Corporation
> + */
> +
> +#ifndef __DRM_MEMCPY_H__
> +#define __DRM_MEMCPY_H__
> +
> +#include <linux/types.h>
> +
> +struct dma_buf_map;
> +
> +#ifdef CONFIG_X86
> +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> +bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
> + const struct dma_buf_map *src,
> + unsigned long len);
> +void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> +
> +/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
> + * as well as SSE4.1 support. drm_memcpy_from_wc() will report if it cannot
> + * perform the operation. To check beforehand, pass in the parameters to
> + * drm_can_memcpy_from_wc() - since we only care about the low 4 bits,
> + * you only need to pass in the minor offsets, page-aligned pointers are
> + * always valid.
> + *
> + * For just checking for SSE4.1, in the foreknowledge that the future use
> + * will be correctly aligned, just use drm_has_memcpy_from_wc().
> + */
> +#define drm_can_memcpy_from_wc(dst, src, len) \
> + drm_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
> +
> +#define drm_has_memcpy_from_wc() \
> + drm_memcpy_from_wc(NULL, NULL, 0)
> +
> +void drm_memcpy_init_early(void);
> +
> +#else
> +
> +static inline
> +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> +{
> + return false;
> +}
> +
> +static inline
> +bool drm_memcpy_from_wc_dbm(void *dst, const void *src, unsigned long len)
> +{
> + return false;
> +}
> +
> +static inline
> +bool drm_can_memcpy_from_wc_dbm(void *dst, const void *src, unsigned long len)
> +{
> + return false;
> +}
> +
> +static inline
> +bool drm_has_memcpy_from_wc(void)
> +{
> + return false;
> +}
> +
> +#define drm_has_memcpy_from_wc() (false)
> +#define drm_unaligned_memcpy_from_wc(_dst, _src, _len) WARN_ON(1)
> +#define drm_memcpy_init_early() do {} while (0)
> +#endif /* CONFIG_X86 */
> +#endif /* __DRM_MEMCPY_H__ */
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 07/15] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
@ 2021-05-26 14:27 ` Christian König
0 siblings, 0 replies; 58+ messages in thread
From: Christian König @ 2021-05-26 14:27 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel; +Cc: Daniel Vetter, Matthew Auld
Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> Memcpy from wc will be used as well by TTM memcpy.
> Move it to core drm, and make the interface do the right thing
> even on !X86.
>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Cc: Dave Airlie <airlied@gmail.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Acked-by: Christian König <christian.koenig@amd.com>
> ---
> v4:
> - Fix !X86 path (Reported by Matthew Auld)
> ---
> drivers/gpu/drm/Makefile | 2 +-
> drivers/gpu/drm/drm_drv.c | 2 +
> .../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 ++++++++++++-----
> drivers/gpu/drm/i915/Makefile | 1 -
> .../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
> drivers/gpu/drm/i915/gem/i915_gem_object.c | 5 +-
> drivers/gpu/drm/i915/gt/selftest_reset.c | 7 +-
> drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 +--
> drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
> drivers/gpu/drm/i915/i915_drv.c | 2 -
> drivers/gpu/drm/i915/i915_gpu_error.c | 8 +--
> drivers/gpu/drm/i915/i915_memcpy.h | 34 ----------
> .../drm/i915/selftests/intel_memory_region.c | 7 +-
> include/drm/drm_memcpy.h | 68 +++++++++++++++++++
> 14 files changed, 142 insertions(+), 76 deletions(-)
> rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
> delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
> create mode 100644 include/drm/drm_memcpy.h
>
> diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
> index a91cc7684904..f3ab8586c3d7 100644
> --- a/drivers/gpu/drm/Makefile
> +++ b/drivers/gpu/drm/Makefile
> @@ -18,7 +18,7 @@ drm-y := drm_aperture.o drm_auth.o drm_cache.o \
> drm_dumb_buffers.o drm_mode_config.o drm_vblank.o \
> drm_syncobj.o drm_lease.o drm_writeback.o drm_client.o \
> drm_client_modeset.o drm_atomic_uapi.o drm_hdcp.o \
> - drm_managed.o drm_vblank_work.o
> + drm_managed.o drm_vblank_work.o drm_memcpy.o \
>
> drm-$(CONFIG_DRM_LEGACY) += drm_agpsupport.o drm_bufs.o drm_context.o drm_dma.o \
> drm_legacy_misc.o drm_lock.o drm_memory.o drm_scatter.o \
> diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> index 3d8d68a98b95..351cc2900cf1 100644
> --- a/drivers/gpu/drm/drm_drv.c
> +++ b/drivers/gpu/drm/drm_drv.c
> @@ -40,6 +40,7 @@
> #include <drm/drm_drv.h>
> #include <drm/drm_file.h>
> #include <drm/drm_managed.h>
> +#include <drm/drm_memcpy.h>
> #include <drm/drm_mode_object.h>
> #include <drm/drm_print.h>
>
> @@ -1041,6 +1042,7 @@ static int __init drm_core_init(void)
>
> drm_connector_ida_init();
> idr_init(&drm_minors_idr);
> + drm_memcpy_init_early();
>
> ret = drm_sysfs_init();
> if (ret < 0) {
> diff --git a/drivers/gpu/drm/i915/i915_memcpy.c b/drivers/gpu/drm/drm_memcpy.c
> similarity index 70%
> rename from drivers/gpu/drm/i915/i915_memcpy.c
> rename to drivers/gpu/drm/drm_memcpy.c
> index 1b021a4902de..740377749caa 100644
> --- a/drivers/gpu/drm/i915/i915_memcpy.c
> +++ b/drivers/gpu/drm/drm_memcpy.c
> @@ -1,3 +1,4 @@
> +// SPDX-License-Identifier: MIT
> /*
> * Copyright © 2016 Intel Corporation
> *
> @@ -22,16 +23,12 @@
> *
> */
>
> +#ifdef CONFIG_X86
> +#include <linux/dma-buf-map.h>
> #include <linux/kernel.h>
> #include <asm/fpu/api.h>
>
> -#include "i915_memcpy.h"
> -
> -#if IS_ENABLED(CONFIG_DRM_I915_DEBUG)
> -#define CI_BUG_ON(expr) BUG_ON(expr)
> -#else
> -#define CI_BUG_ON(expr) BUILD_BUG_ON_INVALID(expr)
> -#endif
> +#include "drm/drm_memcpy.h"
>
> static DEFINE_STATIC_KEY_FALSE(has_movntdqa);
>
> @@ -94,23 +91,24 @@ static void __memcpy_ntdqu(void *dst, const void *src, unsigned long len)
> }
>
> /**
> - * i915_memcpy_from_wc: perform an accelerated *aligned* read from WC
> + * drm_memcpy_from_wc: perform an accelerated *aligned* read from WC
> * @dst: destination pointer
> * @src: source pointer
> * @len: how many bytes to copy
> *
> - * i915_memcpy_from_wc copies @len bytes from @src to @dst using
> + * drm_memcpy_from_wc copies @len bytes from @src to @dst using
> * non-temporal instructions where available. Note that all arguments
> * (@src, @dst) must be aligned to 16 bytes and @len must be a multiple
> * of 16.
> *
> * To test whether accelerated reads from WC are supported, use
> - * i915_memcpy_from_wc(NULL, NULL, 0);
> + * drm_memcpy_from_wc(NULL, NULL, 0);
> + * This interface is intended for memremapped memory without the __iomem tag.
> *
> * Returns true if the copy was successful, false if the preconditions
> * are not met.
> */
> -bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> {
> if (unlikely(((unsigned long)dst | (unsigned long)src | len) & 15))
> return false;
> @@ -123,24 +121,53 @@ bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
>
> return false;
> }
> +EXPORT_SYMBOL(drm_memcpy_from_wc);
>
> /**
> - * i915_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
> + * drm_memcpy_from_wc_dbm: perform an accelerated *aligned* read from WC with
> + * struct dma_buf_map arguments.
> + * @dst: destination map
> + * @src: source map
> + * @len: how many bytes to copy
> + *
> + * This is identical to drm_memcpy_from_wc, except it's intended for
> + * potentially ioremapped memory rather than memremapped memory.
> + *
> + * Returns true if the copy was successful, false if the preconditions
> + * are not met.
> + */
> +bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
> + const struct dma_buf_map *src,
> + unsigned long len)
> +{
> + /* For X86 we can safely drop __iomem */
> + return drm_memcpy_from_wc(dst->is_iomem ?
> + (void __force *)dst->vaddr_iomem :
> + dst->vaddr,
> + src->is_iomem ?
> + (void const __force *)src->vaddr_iomem :
> + src->vaddr,
> + len);
> +}
> +EXPORT_SYMBOL(drm_memcpy_from_wc_dbm);
> +
> +/**
> + * drm_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
> * @dst: destination pointer
> * @src: source pointer
> * @len: how many bytes to copy
> *
> - * Like i915_memcpy_from_wc(), the unaligned variant copies @len bytes from
> + * Like drm_memcpy_from_wc(), the unaligned variant copies @len bytes from
> * @src to @dst using * non-temporal instructions where available, but
> * accepts that its arguments may not be aligned, but are valid for the
> * potential 16-byte read past the end.
> + *
> + * This interface is intended for mremapped memory without the __iomem tag.
> */
> -void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> +void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> {
> unsigned long addr;
>
> - CI_BUG_ON(!i915_has_memcpy_from_wc());
> -
> addr = (unsigned long)src;
> if (!IS_ALIGNED(addr, 16)) {
> unsigned long x = min(ALIGN(addr, 16) - addr, len);
> @@ -155,8 +182,9 @@ void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len
> if (likely(len))
> __memcpy_ntdqu(dst, src, DIV_ROUND_UP(len, 16));
> }
> +EXPORT_SYMBOL(drm_unaligned_memcpy_from_wc);
>
> -void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
> +void drm_memcpy_init_early(void)
> {
> /*
> * Some hypervisors (e.g. KVM) don't support VEX-prefix instructions
> @@ -166,3 +194,4 @@ void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
> !boot_cpu_has(X86_FEATURE_HYPERVISOR))
> static_branch_enable(&has_movntdqa);
> }
> +#endif
> diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
> index 4f22cac1c49b..ebc19bd5fff4 100644
> --- a/drivers/gpu/drm/i915/Makefile
> +++ b/drivers/gpu/drm/i915/Makefile
> @@ -61,7 +61,6 @@ i915-y += i915_drv.o \
> # core library code
> i915-y += \
> dma_resv_utils.o \
> - i915_memcpy.o \
> i915_mm.o \
> i915_sw_fence.o \
> i915_sw_fence_work.o \
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
> index 297143511f99..77285e421fb8 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
> @@ -10,6 +10,7 @@
> #include <linux/uaccess.h>
>
> #include <drm/drm_syncobj.h>
> +#include <drm/drm_memcpy.h>
>
> #include "display/intel_frontbuffer.h"
>
> @@ -28,7 +29,6 @@
> #include "i915_sw_fence_work.h"
> #include "i915_trace.h"
> #include "i915_user_extensions.h"
> -#include "i915_memcpy.h"
>
> struct eb_vma {
> struct i915_vma *vma;
> @@ -2503,7 +2503,7 @@ static int eb_parse_pipeline(struct i915_execbuffer *eb,
> !(batch->cache_coherent & I915_BO_CACHE_COHERENT_FOR_READ);
>
> pw->batch_map = ERR_PTR(-ENODEV);
> - if (needs_clflush && i915_has_memcpy_from_wc())
> + if (needs_clflush && drm_has_memcpy_from_wc())
> pw->batch_map = i915_gem_object_pin_map(batch, I915_MAP_WC);
>
> if (IS_ERR(pw->batch_map)) {
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
> index 5706d471692d..e9247afb0320 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
> @@ -24,6 +24,8 @@
>
> #include <linux/sched/mm.h>
>
> +#include <drm/drm_memcpy.h>
> +
> #include "display/intel_frontbuffer.h"
> #include "i915_drv.h"
> #include "i915_gem_clflush.h"
> @@ -31,7 +33,6 @@
> #include "i915_gem_mman.h"
> #include "i915_gem_object.h"
> #include "i915_globals.h"
> -#include "i915_memcpy.h"
> #include "i915_trace.h"
>
> static struct i915_global_object {
> @@ -374,7 +375,7 @@ i915_gem_object_read_from_page_iomap(struct drm_i915_gem_object *obj, u64 offset
> PAGE_SIZE);
>
> src_ptr = src_map + offset_in_page(offset);
> - if (!i915_memcpy_from_wc(dst, (void __force *)src_ptr, size))
> + if (!drm_memcpy_from_wc(dst, (void __force *)src_ptr, size))
> memcpy_fromio(dst, src_ptr, size);
>
> io_mapping_unmap(src_map);
> diff --git a/drivers/gpu/drm/i915/gt/selftest_reset.c b/drivers/gpu/drm/i915/gt/selftest_reset.c
> index 8784257ec808..92ada67a3835 100644
> --- a/drivers/gpu/drm/i915/gt/selftest_reset.c
> +++ b/drivers/gpu/drm/i915/gt/selftest_reset.c
> @@ -5,9 +5,10 @@
>
> #include <linux/crc32.h>
>
> +#include <drm/drm_memcpy.h>
> +
> #include "gem/i915_gem_stolen.h"
>
> -#include "i915_memcpy.h"
> #include "i915_selftest.h"
> #include "intel_gpu_commands.h"
> #include "selftests/igt_reset.h"
> @@ -99,7 +100,7 @@ __igt_reset_stolen(struct intel_gt *gt,
> memset_io(s, STACK_MAGIC, PAGE_SIZE);
>
> in = (void __force *)s;
> - if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
> + if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
> in = tmp;
> crc[page] = crc32_le(0, in, PAGE_SIZE);
>
> @@ -135,7 +136,7 @@ __igt_reset_stolen(struct intel_gt *gt,
> PAGE_SIZE);
>
> in = (void __force *)s;
> - if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
> + if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
> in = tmp;
> x = crc32_le(0, in, PAGE_SIZE);
>
> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
> index c36d5eb5bbb9..f045e42be6ca 100644
> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
> @@ -5,9 +5,10 @@
>
> #include <linux/debugfs.h>
>
> +#include <drm/drm_memcpy.h>
> +
> #include "gt/intel_gt.h"
> #include "i915_drv.h"
> -#include "i915_memcpy.h"
> #include "intel_guc_log.h"
>
> static void guc_log_capture_logs(struct intel_guc_log *log);
> @@ -295,13 +296,13 @@ static void guc_read_update_log_buffer(struct intel_guc_log *log)
>
> /* Just copy the newly written data */
> if (read_offset > write_offset) {
> - i915_memcpy_from_wc(dst_data, src_data, write_offset);
> + drm_memcpy_from_wc(dst_data, src_data, write_offset);
> bytes_to_copy = buffer_size - read_offset;
> } else {
> bytes_to_copy = write_offset - read_offset;
> }
> - i915_memcpy_from_wc(dst_data + read_offset,
> - src_data + read_offset, bytes_to_copy);
> + drm_memcpy_from_wc(dst_data + read_offset,
> + src_data + read_offset, bytes_to_copy);
>
> src_data += buffer_size;
> dst_data += buffer_size;
> @@ -569,7 +570,7 @@ int intel_guc_log_relay_open(struct intel_guc_log *log)
> * it should be present on the chipsets supporting GuC based
> * submisssions.
> */
> - if (!i915_has_memcpy_from_wc()) {
> + if (!drm_has_memcpy_from_wc()) {
> ret = -ENXIO;
> goto out_unlock;
> }
> diff --git a/drivers/gpu/drm/i915/i915_cmd_parser.c b/drivers/gpu/drm/i915/i915_cmd_parser.c
> index 5b4b2bd46e7c..98653f1a2b1d 100644
> --- a/drivers/gpu/drm/i915/i915_cmd_parser.c
> +++ b/drivers/gpu/drm/i915/i915_cmd_parser.c
> @@ -24,12 +24,12 @@
> * Brad Volkin <bradley.d.volkin@intel.com>
> *
> */
> +#include <drm/drm_memcpy.h>
>
> #include "gt/intel_engine.h"
> #include "gt/intel_gpu_commands.h"
>
> #include "i915_drv.h"
> -#include "i915_memcpy.h"
>
> /**
> * DOC: batch buffer command parser
> @@ -1152,7 +1152,7 @@ static u32 *copy_batch(struct drm_i915_gem_object *dst_obj,
>
> if (src) {
> GEM_BUG_ON(!needs_clflush);
> - i915_unaligned_memcpy_from_wc(dst, src + offset, length);
> + drm_unaligned_memcpy_from_wc(dst, src + offset, length);
> } else {
> struct scatterlist *sg;
> void *ptr;
> diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
> index 30c349137be2..68639ed0bdec 100644
> --- a/drivers/gpu/drm/i915/i915_drv.c
> +++ b/drivers/gpu/drm/i915/i915_drv.c
> @@ -72,7 +72,6 @@
> #include "i915_drv.h"
> #include "i915_ioc32.h"
> #include "i915_irq.h"
> -#include "i915_memcpy.h"
> #include "i915_perf.h"
> #include "i915_query.h"
> #include "i915_suspend.h"
> @@ -325,7 +324,6 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
> mutex_init(&dev_priv->pps_mutex);
> mutex_init(&dev_priv->hdcp_comp_mutex);
>
> - i915_memcpy_init_early(dev_priv);
> intel_runtime_pm_init_early(&dev_priv->runtime_pm);
>
> ret = i915_workqueues_init(dev_priv);
> diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
> index 8b964e355cb5..7c1b44545bab 100644
> --- a/drivers/gpu/drm/i915/i915_gpu_error.c
> +++ b/drivers/gpu/drm/i915/i915_gpu_error.c
> @@ -34,6 +34,7 @@
> #include <linux/utsname.h>
> #include <linux/zlib.h>
>
> +#include <drm/drm_memcpy.h>
> #include <drm/drm_print.h>
>
> #include "display/intel_dmc.h"
> @@ -46,7 +47,6 @@
>
> #include "i915_drv.h"
> #include "i915_gpu_error.h"
> -#include "i915_memcpy.h"
> #include "i915_scatterlist.h"
>
> #define ALLOW_FAIL (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN)
> @@ -255,7 +255,7 @@ static bool compress_init(struct i915_vma_compress *c)
> }
>
> c->tmp = NULL;
> - if (i915_has_memcpy_from_wc())
> + if (drm_has_memcpy_from_wc())
> c->tmp = pool_alloc(&c->pool, ALLOW_FAIL);
>
> return true;
> @@ -295,7 +295,7 @@ static int compress_page(struct i915_vma_compress *c,
> struct z_stream_s *zstream = &c->zstream;
>
> zstream->next_in = src;
> - if (wc && c->tmp && i915_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
> + if (wc && c->tmp && drm_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
> zstream->next_in = c->tmp;
> zstream->avail_in = PAGE_SIZE;
>
> @@ -395,7 +395,7 @@ static int compress_page(struct i915_vma_compress *c,
> if (!ptr)
> return -ENOMEM;
>
> - if (!(wc && i915_memcpy_from_wc(ptr, src, PAGE_SIZE)))
> + if (!(wc && drm_memcpy_from_wc(ptr, src, PAGE_SIZE)))
> memcpy(ptr, src, PAGE_SIZE);
> dst->pages[dst->page_count++] = ptr;
> cond_resched();
> diff --git a/drivers/gpu/drm/i915/i915_memcpy.h b/drivers/gpu/drm/i915/i915_memcpy.h
> deleted file mode 100644
> index 3df063a3293b..000000000000
> --- a/drivers/gpu/drm/i915/i915_memcpy.h
> +++ /dev/null
> @@ -1,34 +0,0 @@
> -/* SPDX-License-Identifier: MIT */
> -/*
> - * Copyright © 2019 Intel Corporation
> - */
> -
> -#ifndef __I915_MEMCPY_H__
> -#define __I915_MEMCPY_H__
> -
> -#include <linux/types.h>
> -
> -struct drm_i915_private;
> -
> -void i915_memcpy_init_early(struct drm_i915_private *i915);
> -
> -bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> -void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> -
> -/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
> - * as well as SSE4.1 support. i915_memcpy_from_wc() will report if it cannot
> - * perform the operation. To check beforehand, pass in the parameters to
> - * to i915_can_memcpy_from_wc() - since we only care about the low 4 bits,
> - * you only need to pass in the minor offsets, page-aligned pointers are
> - * always valid.
> - *
> - * For just checking for SSE4.1, in the foreknowledge that the future use
> - * will be correctly aligned, just use i915_has_memcpy_from_wc().
> - */
> -#define i915_can_memcpy_from_wc(dst, src, len) \
> - i915_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
> -
> -#define i915_has_memcpy_from_wc() \
> - i915_memcpy_from_wc(NULL, NULL, 0)
> -
> -#endif /* __I915_MEMCPY_H__ */
> diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
> index c85d516b85cd..6bb399e9be78 100644
> --- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
> +++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
> @@ -6,6 +6,8 @@
> #include <linux/prime_numbers.h>
> #include <linux/sort.h>
>
> +#include <drm/drm_memcpy.h>
> +
> #include "../i915_selftest.h"
>
> #include "mock_drm.h"
> @@ -20,7 +22,6 @@
> #include "gem/selftests/mock_context.h"
> #include "gt/intel_engine_user.h"
> #include "gt/intel_gt.h"
> -#include "i915_memcpy.h"
> #include "selftests/igt_flush_test.h"
> #include "selftests/i915_random.h"
>
> @@ -901,7 +902,7 @@ static inline void igt_memcpy(void *dst, const void *src, size_t size)
>
> static inline void igt_memcpy_from_wc(void *dst, const void *src, size_t size)
> {
> - i915_memcpy_from_wc(dst, src, size);
> + drm_memcpy_from_wc(dst, src, size);
> }
>
> static int _perf_memcpy(struct intel_memory_region *src_mr,
> @@ -925,7 +926,7 @@ static int _perf_memcpy(struct intel_memory_region *src_mr,
> {
> "memcpy_from_wc",
> igt_memcpy_from_wc,
> - !i915_has_memcpy_from_wc(),
> + !drm_has_memcpy_from_wc(),
> },
> };
> struct drm_i915_gem_object *src, *dst;
> diff --git a/include/drm/drm_memcpy.h b/include/drm/drm_memcpy.h
> new file mode 100644
> index 000000000000..fe5ed1e89ce6
> --- /dev/null
> +++ b/include/drm/drm_memcpy.h
> @@ -0,0 +1,68 @@
> +/* SPDX-License-Identifier: MIT */
> +/*
> + * Copyright © 2019 Intel Corporation
> + */
> +
> +#ifndef __DRM_MEMCPY_H__
> +#define __DRM_MEMCPY_H__
> +
> +#include <linux/types.h>
> +
> +struct dma_buf_map;
> +
> +#ifdef CONFIG_X86
> +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> +bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
> + const struct dma_buf_map *src,
> + unsigned long len);
> +void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> +
> +/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
> + * as well as SSE4.1 support. drm_memcpy_from_wc() will report if it cannot
> + * perform the operation. To check beforehand, pass in the parameters to
> + * drm_can_memcpy_from_wc() - since we only care about the low 4 bits,
> + * you only need to pass in the minor offsets, page-aligned pointers are
> + * always valid.
> + *
> + * For just checking for SSE4.1, in the foreknowledge that the future use
> + * will be correctly aligned, just use drm_has_memcpy_from_wc().
> + */
> +#define drm_can_memcpy_from_wc(dst, src, len) \
> + drm_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
> +
> +#define drm_has_memcpy_from_wc() \
> + drm_memcpy_from_wc(NULL, NULL, 0)
> +
> +void drm_memcpy_init_early(void);
> +
> +#else
> +
> +static inline
> +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
> +{
> + return false;
> +}
> +
> +static inline
> +bool drm_memcpy_from_wc_dbm(void *dst, const void *src, unsigned long len)
> +{
> + return false;
> +}
> +
> +static inline
> +bool drm_can_memcpy_from_wc_dbm(void *dst, const void *src, unsigned long len)
> +{
> + return false;
> +}
> +
> +static inline
> +bool drm_has_memcpy_from_wc(void)
> +{
> + return false;
> +}
> +
> +#define drm_has_memcpy_from_wc() (false)
> +#define drm_unaligned_memcpy_from_wc(_dst, _src, _len) WARN_ON(1)
> +#define drm_memcpy_init_early() do {} while (0)
> +#endif /* CONFIG_X86 */
> +#endif /* __DRM_MEMCPY_H__ */
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 09/15] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 14:32 ` Christian König
-1 siblings, 0 replies; 58+ messages in thread
From: Christian König @ 2021-05-26 14:32 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel
Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
> create a ghost object and push it out to delayed destroy.
> Fix this by adding a path for idle, and document the function.
>
> Also avoid having the bo end up in a bad state vulnerable to user-space
> triggered kernel BUGs if the call to ttm_tt_create() fails.
>
> Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> v4:
> - Clarify why we mark bo for clearing after ttm_bo_pipeline_gutting()
> (Reported by Matthew Auld)
> ---
> drivers/gpu/drm/ttm/ttm_bo.c | 20 +++++------
> drivers/gpu/drm/ttm/ttm_bo_util.c | 55 ++++++++++++++++++++++++++++---
> drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
> include/drm/ttm/ttm_tt.h | 10 ++++++
> 4 files changed, 76 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index 51a94fd63bd7..be0406466460 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -501,10 +501,15 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
> bdev->funcs->evict_flags(bo, &placement);
>
> if (!placement.num_placement && !placement.num_busy_placement) {
> - ttm_bo_wait(bo, false, false);
> + ret = ttm_bo_wait(bo, true, false);
> + if (ret)
> + return ret;
>
> - ttm_bo_cleanup_memtype_use(bo);
> - return ttm_tt_create(bo, false);
> + /*
> + * Since we've already synced, this frees backing store
> + * immediately.
> + */
> + return ttm_bo_pipeline_gutting(bo);
> }
>
> ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
> @@ -976,13 +981,8 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
> /*
> * Remove the backing store if no placement is given.
> */
> - if (!placement->num_placement && !placement->num_busy_placement) {
> - ret = ttm_bo_pipeline_gutting(bo);
> - if (ret)
> - return ret;
> -
> - return ttm_tt_create(bo, false);
> - }
> + if (!placement->num_placement && !placement->num_busy_placement)
> + return ttm_bo_pipeline_gutting(bo);
>
> /*
> * Check whether we need to move buffer.
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index ebff603a97f4..4cca932f1c0e 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -590,26 +590,73 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
> }
> EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
>
> +/**
> + * ttm_bo_pipeline_gutting - purge the contents of a bo
> + * @bo: The buffer object
> + *
> + * Purge the contents of a bo, async if the bo is not idle.
> + * After a successful call, the bo is left unpopulated in
> + * system placement. The function may wait uninterruptible
> + * for idle on OOM.
> + *
> + * Return: 0 if successful, negative error code on failure.
> + */
> int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
> {
> static const struct ttm_place sys_mem = { .mem_type = TTM_PL_SYSTEM };
> struct ttm_buffer_object *ghost;
> + struct ttm_tt *ttm;
> int ret;
>
> - ret = ttm_buffer_object_transfer(bo, &ghost);
> + /* If already idle, no need for ghost object dance. */
> + ret = ttm_bo_wait(bo, false, true);
> + if (ret != -EBUSY) {
> + if (!bo->ttm) {
> + /* See comment below about clearing. */
> + ret = ttm_tt_create(bo, true);
> + if (ret)
> + return ret;
> + } else {
> + ttm_tt_unpopulate(bo->bdev, bo->ttm);
> + if (bo->type == ttm_bo_type_device)
> + ttm_tt_mark_for_clear(bo->ttm);
> + }
> + ttm_resource_free(bo, &bo->mem);
> + ttm_resource_alloc(bo, &sys_mem, &bo->mem);
> +
> + return 0;
> + }
> +
> + /*
> + * We need an unpopulated ttm_tt after giving our current one,
> + * if any, to the ghost object. And we can't afford to fail
> + * creating one *after* the operation. If the bo subsequently gets
> + * resurrected, make sure it's cleared (if ttm_bo_type_device)
> + * to avoid leaking sensitive information to user-space.
> + */
> +
> + ttm = bo->ttm;
> + bo->ttm = NULL;
> + ret = ttm_tt_create(bo, true);
> + swap(bo->ttm, ttm);
> if (ret)
> return ret;
>
> + ret = ttm_buffer_object_transfer(bo, &ghost);
> + if (ret) {
> + ttm_tt_destroy(bo->bdev, ttm);
> + return ret;
> + }
> +
> ret = dma_resv_copy_fences(&ghost->base._resv, bo->base.resv);
> /* Last resort, wait for the BO to be idle when we are OOM */
> if (ret)
> ttm_bo_wait(bo, false, false);
>
> - ttm_resource_alloc(bo, &sys_mem, &bo->mem);
> - bo->ttm = NULL;
> -
> dma_resv_unlock(&ghost->base._resv);
> ttm_bo_put(ghost);
> + bo->ttm = ttm;
> + ttm_resource_alloc(bo, &sys_mem, &bo->mem);
>
> return 0;
> }
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index 0e41227116b1..913b330a234b 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -134,6 +134,11 @@ void ttm_tt_destroy_common(struct ttm_device *bdev, struct ttm_tt *ttm)
> }
> EXPORT_SYMBOL(ttm_tt_destroy_common);
>
> +void ttm_tt_mark_for_clear(struct ttm_tt *ttm)
> +{
> + ttm->page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
> +}
> +
> void ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
> {
> bdev->funcs->ttm_tt_destroy(bdev, ttm);
> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> index 3102059db726..daa9c4cf48bb 100644
> --- a/include/drm/ttm/ttm_tt.h
> +++ b/include/drm/ttm/ttm_tt.h
> @@ -170,6 +170,16 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, struct ttm_oper
> */
> void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
>
> +/**
> + * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
> + *
> + * @ttm: Pointer to the ttm_tt structure
> + *
> + * Marks pages for clearing so that the next time the page vector is
> + * populated, the pages will be cleared.
> + */
> +void ttm_tt_mark_for_clear(struct ttm_tt *ttm);
Either implement the function directly here as static (it's a one liner
anyway) or move the documentation to the implementation.
Apart from that the patch is Reviewed-by: Christian König
<christian.koenig@amd.com>
Regards,
Christian.
> +
> void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
>
> struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 09/15] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
@ 2021-05-26 14:32 ` Christian König
0 siblings, 0 replies; 58+ messages in thread
From: Christian König @ 2021-05-26 14:32 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel
Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
> create a ghost object and push it out to delayed destroy.
> Fix this by adding a path for idle, and document the function.
>
> Also avoid having the bo end up in a bad state vulnerable to user-space
> triggered kernel BUGs if the call to ttm_tt_create() fails.
>
> Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> v4:
> - Clarify why we mark bo for clearing after ttm_bo_pipeline_gutting()
> (Reported by Matthew Auld)
> ---
> drivers/gpu/drm/ttm/ttm_bo.c | 20 +++++------
> drivers/gpu/drm/ttm/ttm_bo_util.c | 55 ++++++++++++++++++++++++++++---
> drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
> include/drm/ttm/ttm_tt.h | 10 ++++++
> 4 files changed, 76 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index 51a94fd63bd7..be0406466460 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -501,10 +501,15 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
> bdev->funcs->evict_flags(bo, &placement);
>
> if (!placement.num_placement && !placement.num_busy_placement) {
> - ttm_bo_wait(bo, false, false);
> + ret = ttm_bo_wait(bo, true, false);
> + if (ret)
> + return ret;
>
> - ttm_bo_cleanup_memtype_use(bo);
> - return ttm_tt_create(bo, false);
> + /*
> + * Since we've already synced, this frees backing store
> + * immediately.
> + */
> + return ttm_bo_pipeline_gutting(bo);
> }
>
> ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
> @@ -976,13 +981,8 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
> /*
> * Remove the backing store if no placement is given.
> */
> - if (!placement->num_placement && !placement->num_busy_placement) {
> - ret = ttm_bo_pipeline_gutting(bo);
> - if (ret)
> - return ret;
> -
> - return ttm_tt_create(bo, false);
> - }
> + if (!placement->num_placement && !placement->num_busy_placement)
> + return ttm_bo_pipeline_gutting(bo);
>
> /*
> * Check whether we need to move buffer.
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index ebff603a97f4..4cca932f1c0e 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -590,26 +590,73 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
> }
> EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
>
> +/**
> + * ttm_bo_pipeline_gutting - purge the contents of a bo
> + * @bo: The buffer object
> + *
> + * Purge the contents of a bo, async if the bo is not idle.
> + * After a successful call, the bo is left unpopulated in
> + * system placement. The function may wait uninterruptible
> + * for idle on OOM.
> + *
> + * Return: 0 if successful, negative error code on failure.
> + */
> int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
> {
> static const struct ttm_place sys_mem = { .mem_type = TTM_PL_SYSTEM };
> struct ttm_buffer_object *ghost;
> + struct ttm_tt *ttm;
> int ret;
>
> - ret = ttm_buffer_object_transfer(bo, &ghost);
> + /* If already idle, no need for ghost object dance. */
> + ret = ttm_bo_wait(bo, false, true);
> + if (ret != -EBUSY) {
> + if (!bo->ttm) {
> + /* See comment below about clearing. */
> + ret = ttm_tt_create(bo, true);
> + if (ret)
> + return ret;
> + } else {
> + ttm_tt_unpopulate(bo->bdev, bo->ttm);
> + if (bo->type == ttm_bo_type_device)
> + ttm_tt_mark_for_clear(bo->ttm);
> + }
> + ttm_resource_free(bo, &bo->mem);
> + ttm_resource_alloc(bo, &sys_mem, &bo->mem);
> +
> + return 0;
> + }
> +
> + /*
> + * We need an unpopulated ttm_tt after giving our current one,
> + * if any, to the ghost object. And we can't afford to fail
> + * creating one *after* the operation. If the bo subsequently gets
> + * resurrected, make sure it's cleared (if ttm_bo_type_device)
> + * to avoid leaking sensitive information to user-space.
> + */
> +
> + ttm = bo->ttm;
> + bo->ttm = NULL;
> + ret = ttm_tt_create(bo, true);
> + swap(bo->ttm, ttm);
> if (ret)
> return ret;
>
> + ret = ttm_buffer_object_transfer(bo, &ghost);
> + if (ret) {
> + ttm_tt_destroy(bo->bdev, ttm);
> + return ret;
> + }
> +
> ret = dma_resv_copy_fences(&ghost->base._resv, bo->base.resv);
> /* Last resort, wait for the BO to be idle when we are OOM */
> if (ret)
> ttm_bo_wait(bo, false, false);
>
> - ttm_resource_alloc(bo, &sys_mem, &bo->mem);
> - bo->ttm = NULL;
> -
> dma_resv_unlock(&ghost->base._resv);
> ttm_bo_put(ghost);
> + bo->ttm = ttm;
> + ttm_resource_alloc(bo, &sys_mem, &bo->mem);
>
> return 0;
> }
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index 0e41227116b1..913b330a234b 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -134,6 +134,11 @@ void ttm_tt_destroy_common(struct ttm_device *bdev, struct ttm_tt *ttm)
> }
> EXPORT_SYMBOL(ttm_tt_destroy_common);
>
> +void ttm_tt_mark_for_clear(struct ttm_tt *ttm)
> +{
> + ttm->page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
> +}
> +
> void ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
> {
> bdev->funcs->ttm_tt_destroy(bdev, ttm);
> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> index 3102059db726..daa9c4cf48bb 100644
> --- a/include/drm/ttm/ttm_tt.h
> +++ b/include/drm/ttm/ttm_tt.h
> @@ -170,6 +170,16 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, struct ttm_oper
> */
> void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
>
> +/**
> + * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
> + *
> + * @ttm: Pointer to the ttm_tt structure
> + *
> + * Marks pages for clearing so that the next time the page vector is
> + * populated, the pages will be cleared.
> + */
> +void ttm_tt_mark_for_clear(struct ttm_tt *ttm);
Either implement the function directly here as static (it's a one liner
anyway) or move the documentation to the implementation.
Apart from that the patch is Reviewed-by: Christian König
<christian.koenig@amd.com>
Regards,
Christian.
> +
> void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
>
> struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for drm/i915: Move LMEM (VRAM) management over to TTM (rev4)
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
` (15 preceding siblings ...)
(?)
@ 2021-05-26 16:20 ` Patchwork
-1 siblings, 0 replies; 58+ messages in thread
From: Patchwork @ 2021-05-26 16:20 UTC (permalink / raw)
To: Thomas Hellström; +Cc: intel-gfx
== Series Details ==
Series: drm/i915: Move LMEM (VRAM) management over to TTM (rev4)
URL : https://patchwork.freedesktop.org/series/90022/
State : warning
== Summary ==
$ dim checkpatch origin/drm-tip
a8d995049a9a drm/i915: Untangle the vma pages_mutex
565669662415 drm/i915: Don't free shared locks while shared
34211cad1c61 drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages
9e94246f54ff drm/i915/ttm Initialize the ttm device and memory managers
-:480: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#480:
deleted file mode 100644
total: 0 errors, 1 warnings, 0 checks, 1531 lines checked
8c9f9a941096 drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
512ac8fd774b drm/ttm: Add a generic TTM memcpy move for page-based iomem
-:384: CHECK:ARCH_DEFINES: architecture specific defines should be avoided
#384: FILE: drivers/gpu/drm/ttm/ttm_module.c:56:
+#if defined(__i386__) || defined(__x86_64__)
-:727: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#727:
new file mode 100644
total: 0 errors, 1 warnings, 1 checks, 840 lines checked
1a9c04facf2f drm, drm/i915: Move the memcpy_from_wc functionality to core drm
-:56: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#56:
rename from drivers/gpu/drm/i915/i915_memcpy.c
total: 0 errors, 1 warnings, 0 checks, 431 lines checked
aa4e527660b2 drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
a0467168f3dd drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
4eb13de59af2 drm/ttm, drm/amdgpu: Allow the driver some control over swapping
d74440bb1681 drm/i915/ttm: Introduce a TTM i915 gem object backend
-:449: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#449:
new file mode 100644
total: 0 errors, 1 warnings, 0 checks, 1043 lines checked
078b4a721184 drm/i915/lmem: Verify checks for lmem residency
e1a4df82cfae drm/i915: Disable mmap ioctl for gen12+
89fe83848800 drm/vma: Add a driver_private member to vma_node.
7e57f71fdb25 drm/i915: Use ttm mmap handling for ttm bo's.
-:141: WARNING:LONG_LINE: line length of 117 exceeds 100 columns
#141: FILE: drivers/gpu/drm/i915/gem/i915_gem_mman.c:908:
+ obj = i915_gem_object_get_rcu(container_of(node, struct drm_i915_gem_object, base.vma_node));
-:357: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis
#357: FILE: drivers/gpu/drm/i915/gem/i915_gem_ttm.c:522:
+vm_access_ttm(struct vm_area_struct *area, unsigned long addr,
+ void *buf, int len, int write)
-:386: CHECK:LINE_SPACING: Please don't use multiple blank lines
#386: FILE: drivers/gpu/drm/i915/gem/i915_gem_ttm.c:551:
+
+
-:702: WARNING:PRINTF_L: %Lx is non-standard C, use %llx
#702: FILE: drivers/gpu/drm/i915/selftests/igt_mmap.c:30:
+ pr_info("Failed to lookup %Lx\n", offset);
total: 0 errors, 2 warnings, 2 checks, 649 lines checked
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* [Intel-gfx] ✗ Fi.CI.SPARSE: warning for drm/i915: Move LMEM (VRAM) management over to TTM (rev4)
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
` (16 preceding siblings ...)
(?)
@ 2021-05-26 16:23 ` Patchwork
-1 siblings, 0 replies; 58+ messages in thread
From: Patchwork @ 2021-05-26 16:23 UTC (permalink / raw)
To: Thomas Hellström; +Cc: intel-gfx
== Series Details ==
Series: drm/i915: Move LMEM (VRAM) management over to TTM (rev4)
URL : https://patchwork.freedesktop.org/series/90022/
State : warning
== Summary ==
$ dim sparse --fast origin/drm-tip
Sparse version: v0.6.2
Fast mode used, each commit won't be checked separately.
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1345:25: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1345:25: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1345:25: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1346:17: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1346:17: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1346:17: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1405:17: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1405:17: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1405:17: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_device.c:293:16: error: incompatible types in comparison expression (different type sizes):
+drivers/gpu/drm/amd/amdgpu/amdgpu_device.c:293:16: unsigned long *
+drivers/gpu/drm/amd/amdgpu/amdgpu_device.c:293:16: unsigned long long *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:275:25: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:275:25: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:275:25: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:276:17: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:276:17: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:276:17: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:330:17: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:330:17: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:330:17: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h:90:56: error: marked inline, but without a definition
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_in
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* [Intel-gfx] ✓ Fi.CI.BAT: success for drm/i915: Move LMEM (VRAM) management over to TTM (rev4)
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
` (17 preceding siblings ...)
(?)
@ 2021-05-26 16:51 ` Patchwork
-1 siblings, 0 replies; 58+ messages in thread
From: Patchwork @ 2021-05-26 16:51 UTC (permalink / raw)
To: Thomas Hellström; +Cc: intel-gfx
[-- Attachment #1.1: Type: text/plain, Size: 8957 bytes --]
== Series Details ==
Series: drm/i915: Move LMEM (VRAM) management over to TTM (rev4)
URL : https://patchwork.freedesktop.org/series/90022/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_10135 -> Patchwork_20203
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/index.html
Possible new issues
-------------------
Here are the unknown changes that may have been introduced in Patchwork_20203:
### IGT changes ###
#### Suppressed ####
The following results come from untrusted machines, tests, or statuses.
They do not affect the overall result.
* igt@prime_self_import@basic-with_two_bos:
- {fi-rkl-11500t}: NOTRUN -> [FAIL][1] +6 similar issues
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-rkl-11500t/igt@prime_self_import@basic-with_two_bos.html
* igt@prime_vgem@basic-fence-flip:
- {fi-rkl-11500t}: NOTRUN -> [SKIP][2] +11 similar issues
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-rkl-11500t/igt@prime_vgem@basic-fence-flip.html
Known issues
------------
Here are the changes found in Patchwork_20203 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@amdgpu/amd_cs_nop@sync-fork-compute0:
- fi-snb-2600: NOTRUN -> [SKIP][3] ([fdo#109271]) +17 similar issues
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-snb-2600/igt@amdgpu/amd_cs_nop@sync-fork-compute0.html
* igt@i915_selftest@live@execlists:
- fi-bsw-kefka: NOTRUN -> [INCOMPLETE][4] ([i915#2782] / [i915#2940] / [i915#3462])
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-bsw-kefka/igt@i915_selftest@live@execlists.html
* igt@kms_chamelium@hdmi-edid-read:
- fi-bsw-kefka: NOTRUN -> [SKIP][5] ([fdo#109271] / [fdo#111827]) +8 similar issues
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-bsw-kefka/igt@kms_chamelium@hdmi-edid-read.html
* igt@kms_frontbuffer_tracking@basic:
- fi-tgl-u2: [PASS][6] -> [FAIL][7] ([i915#2416])
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-tgl-u2/igt@kms_frontbuffer_tracking@basic.html
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-tgl-u2/igt@kms_frontbuffer_tracking@basic.html
* igt@kms_psr@cursor_plane_move:
- fi-bsw-kefka: NOTRUN -> [SKIP][8] ([fdo#109271]) +14 similar issues
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-bsw-kefka/igt@kms_psr@cursor_plane_move.html
* igt@runner@aborted:
- fi-bsw-kefka: NOTRUN -> [FAIL][9] ([fdo#109271] / [i915#1436])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-bsw-kefka/igt@runner@aborted.html
#### Possible fixes ####
* igt@i915_selftest@live@hangcheck:
- fi-snb-2600: [INCOMPLETE][10] ([i915#2782]) -> [PASS][11]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-snb-2600/igt@i915_selftest@live@hangcheck.html
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-snb-2600/igt@i915_selftest@live@hangcheck.html
#### Warnings ####
* igt@runner@aborted:
- fi-cfl-8700k: [FAIL][12] ([i915#2426] / [i915#3363]) -> [FAIL][13] ([i915#3363])
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-cfl-8700k/igt@runner@aborted.html
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-cfl-8700k/igt@runner@aborted.html
- fi-skl-6600u: [FAIL][14] ([i915#1436] / [i915#2426] / [i915#3363]) -> [FAIL][15] ([i915#1436] / [i915#3363])
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-skl-6600u/igt@runner@aborted.html
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-skl-6600u/igt@runner@aborted.html
- fi-glk-dsi: [FAIL][16] ([i915#3363] / [k.org#202321]) -> [FAIL][17] ([i915#2426] / [i915#3363] / [k.org#202321])
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-glk-dsi/igt@runner@aborted.html
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-glk-dsi/igt@runner@aborted.html
- fi-kbl-r: [FAIL][18] ([i915#1436] / [i915#3363]) -> [FAIL][19] ([i915#1436] / [i915#2426] / [i915#3363])
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-kbl-r/igt@runner@aborted.html
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-kbl-r/igt@runner@aborted.html
- fi-kbl-soraka: [FAIL][20] ([i915#1436] / [i915#2426] / [i915#3363]) -> [FAIL][21] ([i915#1436] / [i915#3363])
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-kbl-soraka/igt@runner@aborted.html
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-kbl-soraka/igt@runner@aborted.html
- fi-kbl-guc: [FAIL][22] ([i915#1436] / [i915#2426] / [i915#3363]) -> [FAIL][23] ([i915#1436] / [i915#3363])
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-kbl-guc/igt@runner@aborted.html
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-kbl-guc/igt@runner@aborted.html
- fi-cfl-guc: [FAIL][24] ([i915#3363]) -> [FAIL][25] ([i915#2426] / [i915#3363])
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-cfl-guc/igt@runner@aborted.html
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-cfl-guc/igt@runner@aborted.html
- fi-kbl-7567u: [FAIL][26] ([i915#1436] / [i915#2426] / [i915#3363]) -> [FAIL][27] ([i915#1436] / [i915#3363])
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/fi-kbl-7567u/igt@runner@aborted.html
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/fi-kbl-7567u/igt@runner@aborted.html
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#109285]: https://bugs.freedesktop.org/show_bug.cgi?id=109285
[fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
[i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
[i915#1436]: https://gitlab.freedesktop.org/drm/intel/issues/1436
[i915#2190]: https://gitlab.freedesktop.org/drm/intel/issues/2190
[i915#2416]: https://gitlab.freedesktop.org/drm/intel/issues/2416
[i915#2426]: https://gitlab.freedesktop.org/drm/intel/issues/2426
[i915#2782]: https://gitlab.freedesktop.org/drm/intel/issues/2782
[i915#2940]: https://gitlab.freedesktop.org/drm/intel/issues/2940
[i915#3012]: https://gitlab.freedesktop.org/drm/intel/issues/3012
[i915#3276]: https://gitlab.freedesktop.org/drm/intel/issues/3276
[i915#3277]: https://gitlab.freedesktop.org/drm/intel/issues/3277
[i915#3282]: https://gitlab.freedesktop.org/drm/intel/issues/3282
[i915#3283]: https://gitlab.freedesktop.org/drm/intel/issues/3283
[i915#3363]: https://gitlab.freedesktop.org/drm/intel/issues/3363
[i915#3462]: https://gitlab.freedesktop.org/drm/intel/issues/3462
[i915#533]: https://gitlab.freedesktop.org/drm/intel/issues/533
[k.org#202321]: https://bugzilla.kernel.org/show_bug.cgi?id=202321
Participating hosts (43 -> 41)
------------------------------
Additional (2): fi-bsw-kefka fi-rkl-11500t
Missing (4): fi-ilk-m540 fi-bsw-cyan fi-bdw-samus fi-hsw-4200u
Build changes
-------------
* Linux: CI_DRM_10135 -> Patchwork_20203
CI-20190529: 20190529
CI_DRM_10135: 3471adc5883339a844d766db0f8d859009387094 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_6094: f62d8953c0bc5ed68ea978662e62f9dbb46cf101 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
Patchwork_20203: 7e57f71fdb2547ab704f9d3365559d871b04aa8d @ git://anongit.freedesktop.org/gfx-ci/linux
== Linux commits ==
7e57f71fdb25 drm/i915: Use ttm mmap handling for ttm bo's.
89fe83848800 drm/vma: Add a driver_private member to vma_node.
e1a4df82cfae drm/i915: Disable mmap ioctl for gen12+
078b4a721184 drm/i915/lmem: Verify checks for lmem residency
d74440bb1681 drm/i915/ttm: Introduce a TTM i915 gem object backend
4eb13de59af2 drm/ttm, drm/amdgpu: Allow the driver some control over swapping
a0467168f3dd drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
aa4e527660b2 drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
1a9c04facf2f drm, drm/i915: Move the memcpy_from_wc functionality to core drm
512ac8fd774b drm/ttm: Add a generic TTM memcpy move for page-based iomem
8c9f9a941096 drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
9e94246f54ff drm/i915/ttm Initialize the ttm device and memory managers
34211cad1c61 drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages
565669662415 drm/i915: Don't free shared locks while shared
a8d995049a9a drm/i915: Untangle the vma pages_mutex
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/index.html
[-- Attachment #1.2: Type: text/html, Size: 11800 bytes --]
[-- Attachment #2: Type: text/plain, Size: 160 bytes --]
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 13/15] drm/i915: Disable mmap ioctl for gen12+
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 17:28 ` Thomas Hellström (Intel)
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström (Intel) @ 2021-05-26 17:28 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel, Maarten Lankhorst
On 5/26/21 1:32 PM, Thomas Hellström wrote:
> From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>
> The paltform should exclusively use mmap_offset, one less path to worry
Hmm, Thought this was fixed, but s/paltform/platform/
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 13/15] drm/i915: Disable mmap ioctl for gen12+
@ 2021-05-26 17:28 ` Thomas Hellström (Intel)
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström (Intel) @ 2021-05-26 17:28 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel, Maarten Lankhorst
On 5/26/21 1:32 PM, Thomas Hellström wrote:
> From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>
> The paltform should exclusively use mmap_offset, one less path to worry
Hmm, Thought this was fixed, but s/paltform/platform/
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 15/15] drm/i915: Use ttm mmap handling for ttm bo's.
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 17:40 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 17:40 UTC (permalink / raw)
To: intel-gfx, dri-devel, Maarten Lankhorst
On 5/26/21 1:32 PM, Thomas Hellström wrote:
> From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>
> Use the ttm handlers for servicing page faults, and vm_access.
>
> We do our own validation of read-only access, otherwise use the
> ttm handlers as much as possible.
>
> Because the ttm handlers expect the vma_node at vma->base, we slightly
> need to massage the mmap handlers to look at vma_node->driver_private
> to fetch the bo, if it's NULL, we assume i915's normal mmap_offset uapi
> is used.
>
> This is the easiest way to achieve compatibility without changing ttm's
> semantics.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> ---
> drivers/gpu/drm/i915/gem/i915_gem_mman.c | 78 +++++++----
> drivers/gpu/drm/i915/gem/i915_gem_object.h | 6 +-
> .../gpu/drm/i915/gem/i915_gem_object_types.h | 3 +
> drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
> drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 122 +++++++++++++++++-
> .../drm/i915/gem/selftests/i915_gem_mman.c | 90 +++++++------
> drivers/gpu/drm/i915/selftests/igt_mmap.c | 25 +++-
> drivers/gpu/drm/i915/selftests/igt_mmap.h | 12 +-
> 8 files changed, 247 insertions(+), 92 deletions(-)
There are a couple of checkpatch.pl --strict warnings/checks with this
patch.
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> index fd1c9714f8d8..af04ea593091 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> @@ -19,6 +19,7 @@
> #include "i915_gem_mman.h"
> #include "i915_trace.h"
> #include "i915_user_extensions.h"
> +#include "i915_gem_ttm.h"
> #include "i915_vma.h"
>
> static inline bool
> @@ -622,6 +623,8 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
> struct i915_mmap_offset *mmo;
> int err;
>
> + GEM_BUG_ON(obj->ops->mmap_offset || obj->ops->mmap_ops);
> +
> mmo = lookup_mmo(obj, mmap_type);
> if (mmo)
> goto out;
> @@ -664,40 +667,47 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
> }
>
> static int
> -__assign_mmap_offset(struct drm_file *file,
> - u32 handle,
> +__assign_mmap_offset(struct drm_i915_gem_object *obj,
> enum i915_mmap_type mmap_type,
> - u64 *offset)
> + u64 *offset, struct drm_file *file)
> {
> - struct drm_i915_gem_object *obj;
> struct i915_mmap_offset *mmo;
> - int err;
>
> - obj = i915_gem_object_lookup(file, handle);
> - if (!obj)
> - return -ENOENT;
> + if (i915_gem_object_never_mmap(obj))
> + return -ENODEV;
>
> - if (i915_gem_object_never_mmap(obj)) {
> - err = -ENODEV;
> - goto out;
> + if (obj->ops->mmap_offset) {
> + *offset = obj->ops->mmap_offset(obj);
> + return 0;
> }
>
> if (mmap_type != I915_MMAP_TYPE_GTT &&
> !i915_gem_object_has_struct_page(obj) &&
> - !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM)) {
> - err = -ENODEV;
> - goto out;
> - }
> + !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM))
> + return -ENODEV;
>
> mmo = mmap_offset_attach(obj, mmap_type, file);
> - if (IS_ERR(mmo)) {
> - err = PTR_ERR(mmo);
> - goto out;
> - }
> + if (IS_ERR(mmo))
> + return PTR_ERR(mmo);
>
> *offset = drm_vma_node_offset_addr(&mmo->vma_node);
> - err = 0;
> -out:
> + return 0;
> +}
> +
> +static int
> +__assign_mmap_offset_handle(struct drm_file *file,
> + u32 handle,
> + enum i915_mmap_type mmap_type,
> + u64 *offset)
> +{
> + struct drm_i915_gem_object *obj;
> + int err;
> +
> + obj = i915_gem_object_lookup(file, handle);
> + if (!obj)
> + return -ENOENT;
> +
> + err = __assign_mmap_offset(obj, mmap_type, offset, file);
> i915_gem_object_put(obj);
> return err;
> }
> @@ -717,7 +727,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
> else
> mmap_type = I915_MMAP_TYPE_GTT;
>
> - return __assign_mmap_offset(file, handle, mmap_type, offset);
> + return __assign_mmap_offset_handle(file, handle, mmap_type, offset);
> }
>
> /**
> @@ -785,7 +795,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
> return -EINVAL;
> }
>
> - return __assign_mmap_offset(file, args->handle, type, &args->offset);
> + return __assign_mmap_offset_handle(file, args->handle, type, &args->offset);
> }
>
> static void vm_open(struct vm_area_struct *vma)
> @@ -889,8 +899,16 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
> * destroyed and will be invalid when the vma manager lock
> * is released.
> */
> - mmo = container_of(node, struct i915_mmap_offset, vma_node);
> - obj = i915_gem_object_get_rcu(mmo->obj);
> + if (!node->driver_private) {
> + mmo = container_of(node, struct i915_mmap_offset, vma_node);
> + obj = i915_gem_object_get_rcu(mmo->obj);
> +
> + GEM_BUG_ON(obj && obj->ops->mmap_ops);
> + } else {
> + obj = i915_gem_object_get_rcu(container_of(node, struct drm_i915_gem_object, base.vma_node));
> +
> + GEM_BUG_ON(obj && !obj->ops->mmap_ops);
> + }
> }
> drm_vma_offset_unlock_lookup(dev->vma_offset_manager);
> rcu_read_unlock();
> @@ -912,7 +930,6 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
> }
>
> vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP;
> - vma->vm_private_data = mmo;
>
> /*
> * We keep the ref on mmo->obj, not vm_file, but we require
> @@ -926,6 +943,15 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
> /* Drop the initial creation reference, the vma is now holding one. */
> fput(anon);
>
> + if (obj->ops->mmap_ops) {
> + vma->vm_page_prot = pgprot_decrypted(vm_get_page_prot(vma->vm_flags));
> + vma->vm_ops = obj->ops->mmap_ops;
> + vma->vm_private_data = node->driver_private;
> + return 0;
> + }
> +
> + vma->vm_private_data = mmo;
> +
> switch (mmo->mmap_type) {
> case I915_MMAP_TYPE_WC:
> vma->vm_page_prot =
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
> index a3ad8cf4eefd..ff59e6c640e6 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
> @@ -342,14 +342,14 @@ struct scatterlist *
> __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
> struct i915_gem_object_page_iter *iter,
> unsigned int n,
> - unsigned int *offset, bool allow_alloc);
> + unsigned int *offset, bool allow_alloc, bool dma);
>
> static inline struct scatterlist *
> i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
> unsigned int n,
> unsigned int *offset, bool allow_alloc)
> {
> - return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc);
> + return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc, false);
> }
>
> static inline struct scatterlist *
> @@ -357,7 +357,7 @@ i915_gem_object_get_sg_dma(struct drm_i915_gem_object *obj,
> unsigned int n,
> unsigned int *offset, bool allow_alloc)
> {
> - return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc);
> + return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc, true);
> }
>
> struct page *
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
> index 68313474e6a6..2a23b77424b3 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
> @@ -61,6 +61,7 @@ struct drm_i915_gem_object_ops {
> const struct drm_i915_gem_pread *arg);
> int (*pwrite)(struct drm_i915_gem_object *obj,
> const struct drm_i915_gem_pwrite *arg);
> + u64 (*mmap_offset)(struct drm_i915_gem_object *obj);
>
> int (*dmabuf_export)(struct drm_i915_gem_object *obj);
>
> @@ -79,6 +80,7 @@ struct drm_i915_gem_object_ops {
> void (*delayed_free)(struct drm_i915_gem_object *obj);
> void (*release)(struct drm_i915_gem_object *obj);
>
> + const struct vm_operations_struct *mmap_ops;
> const char *name; /* friendly name for debug, e.g. lockdep classes */
> };
>
> @@ -328,6 +330,7 @@ struct drm_i915_gem_object {
>
> struct {
> struct sg_table *cached_io_st;
> + struct i915_gem_object_page_iter get_io_page;
> bool created:1;
> } ttm;
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> index 6444e097016d..086005c1c7ea 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> @@ -467,9 +467,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
> struct i915_gem_object_page_iter *iter,
> unsigned int n,
> unsigned int *offset,
> - bool allow_alloc)
> + bool allow_alloc, bool dma)
> {
> - const bool dma = iter == &obj->mm.get_dma_page;
> struct scatterlist *sg;
> unsigned int idx, count;
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> index 17598930a99e..d0be957326e0 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> @@ -13,6 +13,7 @@
> #include "gem/i915_gem_object.h"
> #include "gem/i915_gem_region.h"
> #include "gem/i915_gem_ttm.h"
> +#include "gem/i915_gem_mman.h"
>
> #define I915_PL_LMEM0 TTM_PL_PRIV
> #define I915_PL_SYSTEM TTM_PL_SYSTEM
> @@ -158,11 +159,20 @@ static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
>
> static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
> {
> - if (obj->ttm.cached_io_st) {
> - sg_free_table(obj->ttm.cached_io_st);
> - kfree(obj->ttm.cached_io_st);
> - obj->ttm.cached_io_st = NULL;
> - }
> + struct radix_tree_iter iter;
> + void __rcu **slot;
> +
> + if (!obj->ttm.cached_io_st)
> + return;
> +
> + rcu_read_lock();
> + radix_tree_for_each_slot(slot, &obj->ttm.get_io_page.radix, &iter, 0)
> + radix_tree_delete(&obj->ttm.get_io_page.radix, iter.index);
> + rcu_read_unlock();
> +
> + sg_free_table(obj->ttm.cached_io_st);
> + kfree(obj->ttm.cached_io_st);
> + obj->ttm.cached_io_st = NULL;
> }
>
> static void i915_ttm_purge(struct drm_i915_gem_object *obj)
> @@ -338,12 +348,42 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
> ttm_bo_move_sync_cleanup(bo, dst_mem);
> i915_ttm_free_cached_io_st(obj);
>
> - if (!dst_man->use_tt)
> + if (!dst_man->use_tt) {
> obj->ttm.cached_io_st = dst_st;
> + obj->ttm.get_io_page.sg_pos = dst_st->sgl;
> + obj->ttm.get_io_page.sg_idx = 0;
> + }
>
> return 0;
> }
>
> +static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource *mem)
> +{
> + if (mem->mem_type < I915_PL_LMEM0)
> + return 0;
> +
> + /* We may need to revisit this later, but this allows all caching to be used in mmap */
> + mem->bus.caching = ttm_cached;
Since we're now using the TTM bo offsets, we might as well just make
this ttm_write_combined now.
> + mem->bus.is_iomem = true;
> +
> + return 0;
> +}
> +
> +static unsigned long i915_ttm_io_mem_pfn(struct ttm_buffer_object *bo,
> + unsigned long page_offset)
> +{
> + struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
> + unsigned long base = obj->mm.region->iomap.base - obj->mm.region->region.start;
> + struct scatterlist *sg;
> + unsigned int ofs;
> +
> + GEM_WARN_ON(bo->ttm);
> +
> + sg = __i915_gem_object_get_sg(obj, &obj->ttm.get_io_page, page_offset, &ofs, true, true);
> +
> + return ((base + sg_dma_address(sg)) >> PAGE_SHIFT) + ofs;
> +}
> +
> static struct ttm_device_funcs i915_ttm_bo_driver = {
> .ttm_tt_create = i915_ttm_tt_create,
> .ttm_tt_unpopulate = i915_ttm_tt_unpopulate,
> @@ -354,6 +394,8 @@ static struct ttm_device_funcs i915_ttm_bo_driver = {
> .verify_access = NULL,
> .swap_notify = i915_ttm_swap_notify,
> .delete_mem_notify = i915_ttm_delete_mem_notify,
> + .io_mem_reserve = i915_ttm_io_mem_reserve,
> + .io_mem_pfn = i915_ttm_io_mem_pfn,
> };
>
> /**
> @@ -461,7 +503,68 @@ static void i915_ttm_delayed_free(struct drm_i915_gem_object *obj)
> }
> }
>
> -static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
> +static vm_fault_t vm_fault_ttm(struct vm_fault *vmf)
> +{
> + struct vm_area_struct *area = vmf->vma;
> + struct drm_i915_gem_object *obj =
> + i915_ttm_to_gem(area->vm_private_data);
> +
> + /* Sanity check that we allow writing into this object */
> + if (unlikely(i915_gem_object_is_readonly(obj) &&
> + area->vm_flags & VM_WRITE))
> + return VM_FAULT_SIGBUS;
> +
> + return ttm_bo_vm_fault(vmf);
> +}
> +
> +static int
> +vm_access_ttm(struct vm_area_struct *area, unsigned long addr,
> + void *buf, int len, int write)
> +{
> + struct drm_i915_gem_object *obj =
> + i915_ttm_to_gem(area->vm_private_data);
> +
> + if (i915_gem_object_is_readonly(obj) && write)
> + return -EACCES;
> +
> + return ttm_bo_vm_access(area, addr, buf, len, write);
> +}
> +
> +static void ttm_vm_open(struct vm_area_struct *vma)
> +{
> + struct drm_i915_gem_object *obj =
> + i915_ttm_to_gem(vma->vm_private_data);
> +
> + GEM_BUG_ON(!obj);
> + i915_gem_object_get(obj);
> +}
> +
> +static void ttm_vm_close(struct vm_area_struct *vma)
> +{
> + struct drm_i915_gem_object *obj =
> + i915_ttm_to_gem(vma->vm_private_data);
> +
> + GEM_BUG_ON(!obj);
> + i915_gem_object_put(obj);
> +}
> +
> +
> +static const struct vm_operations_struct vm_ops_ttm = {
> + .fault = vm_fault_ttm,
> + .access = vm_access_ttm,
> + .open = ttm_vm_open,
> + .close = ttm_vm_close,
> +};
> +
> +static u64 i915_ttm_mmap_offset(struct drm_i915_gem_object *obj)
> +{
> + /* The ttm_bo must be allocated with I915_BO_ALLOC_USER */
> + GEM_BUG_ON(!drm_mm_node_allocated(&obj->base.vma_node.vm_node));
> +
> + return drm_vma_node_offset_addr(&obj->base.vma_node);
> +}
> +
> +const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
> .name = "i915_gem_object_ttm",
> .flags = I915_GEM_OBJECT_HAS_IOMEM,
>
> @@ -470,6 +573,8 @@ static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
> .truncate = i915_ttm_purge,
> .adjust_lru = i915_ttm_adjust_lru,
> .delayed_free = i915_ttm_delayed_free,
> + .mmap_offset = i915_ttm_mmap_offset,
> + .mmap_ops = &vm_ops_ttm,
> };
>
> void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
Put a mutex_destroy(&obj->ttm.get_io_page.lock) here?
> @@ -518,6 +623,8 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
> i915_gem_object_make_unshrinkable(obj);
> obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
> i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
> + INIT_RADIX_TREE(&obj->ttm.get_io_page.radix, GFP_KERNEL | __GFP_NOWARN);
> + mutex_init(&obj->ttm.get_io_page.lock);
>
> bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device :
> ttm_bo_type_kernel;
> @@ -529,6 +636,7 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
> * Similarly, in delayed_destroy, we can't call ttm_bo_put()
> * until successful initialization.
> */
> + obj->base.vma_node.driver_private = i915_gem_to_ttm(obj);
> ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size,
> bo_type, &i915_sys_placement, alignment,
> true, NULL, NULL, i915_ttm_bo_destroy);
> diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
> index 05a3b29f545e..ca69a29b7f2a 100644
> --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
> +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
> @@ -578,16 +578,17 @@ static bool assert_mmap_offset(struct drm_i915_private *i915,
> int expected)
> {
> struct drm_i915_gem_object *obj;
> - struct i915_mmap_offset *mmo;
> + u64 offset;
> + int ret;
>
> obj = i915_gem_object_create_internal(i915, size);
> if (IS_ERR(obj))
> - return false;
> + return expected && expected == PTR_ERR(obj);
>
> - mmo = mmap_offset_attach(obj, I915_MMAP_OFFSET_GTT, NULL);
> + ret = __assign_mmap_offset(obj, I915_MMAP_TYPE_GTT, &offset, NULL);
> i915_gem_object_put(obj);
>
> - return PTR_ERR_OR_ZERO(mmo) == expected;
> + return ret == expected;
> }
>
> static void disable_retire_worker(struct drm_i915_private *i915)
> @@ -622,8 +623,8 @@ static int igt_mmap_offset_exhaustion(void *arg)
> struct drm_mm *mm = &i915->drm.vma_offset_manager->vm_addr_space_mm;
> struct drm_i915_gem_object *obj;
> struct drm_mm_node *hole, *next;
> - struct i915_mmap_offset *mmo;
> int loop, err = 0;
> + u64 offset;
>
> /* Disable background reaper */
> disable_retire_worker(i915);
> @@ -684,13 +685,13 @@ static int igt_mmap_offset_exhaustion(void *arg)
> obj = i915_gem_object_create_internal(i915, PAGE_SIZE);
> if (IS_ERR(obj)) {
> err = PTR_ERR(obj);
> + pr_err("Unable to create object for reclaimed hole\n");
> goto out;
> }
>
> - mmo = mmap_offset_attach(obj, I915_MMAP_OFFSET_GTT, NULL);
> - if (IS_ERR(mmo)) {
> + err = __assign_mmap_offset(obj, I915_MMAP_TYPE_GTT, &offset, NULL);
> + if (err) {
> pr_err("Unable to insert object into reclaimed hole\n");
> - err = PTR_ERR(mmo);
> goto err_obj;
> }
>
> @@ -865,10 +866,10 @@ static int __igt_mmap(struct drm_i915_private *i915,
> struct drm_i915_gem_object *obj,
> enum i915_mmap_type type)
> {
> - struct i915_mmap_offset *mmo;
> struct vm_area_struct *area;
> unsigned long addr;
> int err, i;
> + u64 offset;
>
> if (!can_mmap(obj, type))
> return 0;
> @@ -879,11 +880,11 @@ static int __igt_mmap(struct drm_i915_private *i915,
> if (err)
> return err;
>
> - mmo = mmap_offset_attach(obj, type, NULL);
> - if (IS_ERR(mmo))
> - return PTR_ERR(mmo);
> + err = __assign_mmap_offset(obj, type, &offset, NULL);
> + if (err)
> + return err;
>
> - addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
> + addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
> if (IS_ERR_VALUE(addr))
> return addr;
>
> @@ -897,13 +898,6 @@ static int __igt_mmap(struct drm_i915_private *i915,
> goto out_unmap;
> }
>
> - if (area->vm_private_data != mmo) {
> - pr_err("%s: vm_area_struct did not point back to our mmap_offset object!\n",
> - obj->mm.region->name);
> - err = -EINVAL;
> - goto out_unmap;
> - }
> -
> for (i = 0; i < obj->base.size / sizeof(u32); i++) {
> u32 __user *ux = u64_to_user_ptr((u64)(addr + i * sizeof(*ux)));
> u32 x;
> @@ -961,7 +955,7 @@ static int igt_mmap(void *arg)
> struct drm_i915_gem_object *obj;
> int err;
>
> - obj = i915_gem_object_create_region(mr, sizes[i], 0);
> + obj = i915_gem_object_create_region(mr, sizes[i], I915_BO_ALLOC_USER);
> if (obj == ERR_PTR(-ENODEV))
> continue;
>
> @@ -1004,12 +998,12 @@ static int __igt_mmap_access(struct drm_i915_private *i915,
> struct drm_i915_gem_object *obj,
> enum i915_mmap_type type)
> {
> - struct i915_mmap_offset *mmo;
> unsigned long __user *ptr;
> unsigned long A, B;
> unsigned long x, y;
> unsigned long addr;
> int err;
> + u64 offset;
>
> memset(&A, 0xAA, sizeof(A));
> memset(&B, 0xBB, sizeof(B));
> @@ -1017,11 +1011,11 @@ static int __igt_mmap_access(struct drm_i915_private *i915,
> if (!can_mmap(obj, type) || !can_access(obj))
> return 0;
>
> - mmo = mmap_offset_attach(obj, type, NULL);
> - if (IS_ERR(mmo))
> - return PTR_ERR(mmo);
> + err = __assign_mmap_offset(obj, type, &offset, NULL);
> + if (err)
> + return err;
>
> - addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
> + addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
> if (IS_ERR_VALUE(addr))
> return addr;
> ptr = (unsigned long __user *)addr;
> @@ -1081,7 +1075,7 @@ static int igt_mmap_access(void *arg)
> struct drm_i915_gem_object *obj;
> int err;
>
> - obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
> + obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
> if (obj == ERR_PTR(-ENODEV))
> continue;
>
> @@ -1111,11 +1105,11 @@ static int __igt_mmap_gpu(struct drm_i915_private *i915,
> enum i915_mmap_type type)
> {
> struct intel_engine_cs *engine;
> - struct i915_mmap_offset *mmo;
> unsigned long addr;
> u32 __user *ux;
> u32 bbe;
> int err;
> + u64 offset;
>
> /*
> * Verify that the mmap access into the backing store aligns with
> @@ -1132,11 +1126,11 @@ static int __igt_mmap_gpu(struct drm_i915_private *i915,
> if (err)
> return err;
>
> - mmo = mmap_offset_attach(obj, type, NULL);
> - if (IS_ERR(mmo))
> - return PTR_ERR(mmo);
> + err = __assign_mmap_offset(obj, type, &offset, NULL);
> + if (err)
> + return err;
>
> - addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
> + addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
> if (IS_ERR_VALUE(addr))
> return addr;
>
> @@ -1226,7 +1220,7 @@ static int igt_mmap_gpu(void *arg)
> struct drm_i915_gem_object *obj;
> int err;
>
> - obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
> + obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
> if (obj == ERR_PTR(-ENODEV))
> continue;
>
> @@ -1303,18 +1297,18 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
> struct drm_i915_gem_object *obj,
> enum i915_mmap_type type)
> {
> - struct i915_mmap_offset *mmo;
> unsigned long addr;
> int err;
> + u64 offset;
>
> if (!can_mmap(obj, type))
> return 0;
>
> - mmo = mmap_offset_attach(obj, type, NULL);
> - if (IS_ERR(mmo))
> - return PTR_ERR(mmo);
> + err = __assign_mmap_offset(obj, type, &offset, NULL);
> + if (err)
> + return err;
>
> - addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
> + addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
> if (IS_ERR_VALUE(addr))
> return addr;
>
> @@ -1350,10 +1344,20 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
> }
> }
>
> - err = check_absent(addr, obj->base.size);
> - if (err) {
> - pr_err("%s: was not absent\n", obj->mm.region->name);
> - goto out_unmap;
> + if (!obj->ops->mmap_ops) {
> + err = check_absent(addr, obj->base.size);
> + if (err) {
> + pr_err("%s: was not absent\n", obj->mm.region->name);
> + goto out_unmap;
> + }
> + } else {
> + /* ttm allows access to evicted regions by design */
> +
> + err = check_present(addr, obj->base.size);
> + if (err) {
> + pr_err("%s: was not present\n", obj->mm.region->name);
> + goto out_unmap;
> + }
> }
>
> out_unmap:
> @@ -1371,7 +1375,7 @@ static int igt_mmap_revoke(void *arg)
> struct drm_i915_gem_object *obj;
> int err;
>
> - obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
> + obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
> if (obj == ERR_PTR(-ENODEV))
> continue;
>
> diff --git a/drivers/gpu/drm/i915/selftests/igt_mmap.c b/drivers/gpu/drm/i915/selftests/igt_mmap.c
> index 583a4ff8b8c9..e8286c28de91 100644
> --- a/drivers/gpu/drm/i915/selftests/igt_mmap.c
> +++ b/drivers/gpu/drm/i915/selftests/igt_mmap.c
> @@ -9,15 +9,28 @@
> #include "i915_drv.h"
> #include "igt_mmap.h"
>
> -unsigned long igt_mmap_node(struct drm_i915_private *i915,
> - struct drm_vma_offset_node *node,
> - unsigned long addr,
> - unsigned long prot,
> - unsigned long flags)
> +unsigned long igt_mmap_offset(struct drm_i915_private *i915,
> + u64 offset,
> + unsigned long size,
> + unsigned long prot,
> + unsigned long flags)
> {
> + struct drm_vma_offset_node *node;
> struct file *file;
> + unsigned long addr;
> int err;
>
> + /* no need to refcount, we own this object */
> + drm_vma_offset_lock_lookup(i915->drm.vma_offset_manager);
> + node = drm_vma_offset_exact_lookup_locked(i915->drm.vma_offset_manager,
> + offset / PAGE_SIZE, size / PAGE_SIZE);
> + drm_vma_offset_unlock_lookup(i915->drm.vma_offset_manager);
> +
> + if (GEM_WARN_ON(!node)) {
> + pr_info("Failed to lookup %Lx\n", offset);
Checkpatch warning here iirc.
> + return -ENOENT;
> + }
> +
> /* Pretend to open("/dev/dri/card0") */
> file = mock_drm_getfile(i915->drm.primary, O_RDWR);
> if (IS_ERR(file))
> @@ -29,7 +42,7 @@ unsigned long igt_mmap_node(struct drm_i915_private *i915,
> goto out_file;
> }
>
> - addr = vm_mmap(file, addr, drm_vma_node_size(node) << PAGE_SHIFT,
> + addr = vm_mmap(file, 0, drm_vma_node_size(node) << PAGE_SHIFT,
> prot, flags, drm_vma_node_offset_addr(node));
>
> drm_vma_node_revoke(node, file->private_data);
> diff --git a/drivers/gpu/drm/i915/selftests/igt_mmap.h b/drivers/gpu/drm/i915/selftests/igt_mmap.h
> index 6e716cb59d7e..acbe34d81a6d 100644
> --- a/drivers/gpu/drm/i915/selftests/igt_mmap.h
> +++ b/drivers/gpu/drm/i915/selftests/igt_mmap.h
> @@ -7,13 +7,15 @@
> #ifndef IGT_MMAP_H
> #define IGT_MMAP_H
>
> +#include <linux/types.h>
> +
> struct drm_i915_private;
> struct drm_vma_offset_node;
>
> -unsigned long igt_mmap_node(struct drm_i915_private *i915,
> - struct drm_vma_offset_node *node,
> - unsigned long addr,
> - unsigned long prot,
> - unsigned long flags);
> +unsigned long igt_mmap_offset(struct drm_i915_private *i915,
> + u64 offset,
> + unsigned long size,
> + unsigned long prot,
> + unsigned long flags);
>
> #endif /* IGT_MMAP_H */
/Thomas
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 15/15] drm/i915: Use ttm mmap handling for ttm bo's.
@ 2021-05-26 17:40 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-26 17:40 UTC (permalink / raw)
To: intel-gfx, dri-devel, Maarten Lankhorst
On 5/26/21 1:32 PM, Thomas Hellström wrote:
> From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>
> Use the ttm handlers for servicing page faults, and vm_access.
>
> We do our own validation of read-only access, otherwise use the
> ttm handlers as much as possible.
>
> Because the ttm handlers expect the vma_node at vma->base, we slightly
> need to massage the mmap handlers to look at vma_node->driver_private
> to fetch the bo, if it's NULL, we assume i915's normal mmap_offset uapi
> is used.
>
> This is the easiest way to achieve compatibility without changing ttm's
> semantics.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> ---
> drivers/gpu/drm/i915/gem/i915_gem_mman.c | 78 +++++++----
> drivers/gpu/drm/i915/gem/i915_gem_object.h | 6 +-
> .../gpu/drm/i915/gem/i915_gem_object_types.h | 3 +
> drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
> drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 122 +++++++++++++++++-
> .../drm/i915/gem/selftests/i915_gem_mman.c | 90 +++++++------
> drivers/gpu/drm/i915/selftests/igt_mmap.c | 25 +++-
> drivers/gpu/drm/i915/selftests/igt_mmap.h | 12 +-
> 8 files changed, 247 insertions(+), 92 deletions(-)
There are a couple of checkpatch.pl --strict warnings/checks with this
patch.
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> index fd1c9714f8d8..af04ea593091 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> @@ -19,6 +19,7 @@
> #include "i915_gem_mman.h"
> #include "i915_trace.h"
> #include "i915_user_extensions.h"
> +#include "i915_gem_ttm.h"
> #include "i915_vma.h"
>
> static inline bool
> @@ -622,6 +623,8 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
> struct i915_mmap_offset *mmo;
> int err;
>
> + GEM_BUG_ON(obj->ops->mmap_offset || obj->ops->mmap_ops);
> +
> mmo = lookup_mmo(obj, mmap_type);
> if (mmo)
> goto out;
> @@ -664,40 +667,47 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
> }
>
> static int
> -__assign_mmap_offset(struct drm_file *file,
> - u32 handle,
> +__assign_mmap_offset(struct drm_i915_gem_object *obj,
> enum i915_mmap_type mmap_type,
> - u64 *offset)
> + u64 *offset, struct drm_file *file)
> {
> - struct drm_i915_gem_object *obj;
> struct i915_mmap_offset *mmo;
> - int err;
>
> - obj = i915_gem_object_lookup(file, handle);
> - if (!obj)
> - return -ENOENT;
> + if (i915_gem_object_never_mmap(obj))
> + return -ENODEV;
>
> - if (i915_gem_object_never_mmap(obj)) {
> - err = -ENODEV;
> - goto out;
> + if (obj->ops->mmap_offset) {
> + *offset = obj->ops->mmap_offset(obj);
> + return 0;
> }
>
> if (mmap_type != I915_MMAP_TYPE_GTT &&
> !i915_gem_object_has_struct_page(obj) &&
> - !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM)) {
> - err = -ENODEV;
> - goto out;
> - }
> + !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM))
> + return -ENODEV;
>
> mmo = mmap_offset_attach(obj, mmap_type, file);
> - if (IS_ERR(mmo)) {
> - err = PTR_ERR(mmo);
> - goto out;
> - }
> + if (IS_ERR(mmo))
> + return PTR_ERR(mmo);
>
> *offset = drm_vma_node_offset_addr(&mmo->vma_node);
> - err = 0;
> -out:
> + return 0;
> +}
> +
> +static int
> +__assign_mmap_offset_handle(struct drm_file *file,
> + u32 handle,
> + enum i915_mmap_type mmap_type,
> + u64 *offset)
> +{
> + struct drm_i915_gem_object *obj;
> + int err;
> +
> + obj = i915_gem_object_lookup(file, handle);
> + if (!obj)
> + return -ENOENT;
> +
> + err = __assign_mmap_offset(obj, mmap_type, offset, file);
> i915_gem_object_put(obj);
> return err;
> }
> @@ -717,7 +727,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
> else
> mmap_type = I915_MMAP_TYPE_GTT;
>
> - return __assign_mmap_offset(file, handle, mmap_type, offset);
> + return __assign_mmap_offset_handle(file, handle, mmap_type, offset);
> }
>
> /**
> @@ -785,7 +795,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
> return -EINVAL;
> }
>
> - return __assign_mmap_offset(file, args->handle, type, &args->offset);
> + return __assign_mmap_offset_handle(file, args->handle, type, &args->offset);
> }
>
> static void vm_open(struct vm_area_struct *vma)
> @@ -889,8 +899,16 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
> * destroyed and will be invalid when the vma manager lock
> * is released.
> */
> - mmo = container_of(node, struct i915_mmap_offset, vma_node);
> - obj = i915_gem_object_get_rcu(mmo->obj);
> + if (!node->driver_private) {
> + mmo = container_of(node, struct i915_mmap_offset, vma_node);
> + obj = i915_gem_object_get_rcu(mmo->obj);
> +
> + GEM_BUG_ON(obj && obj->ops->mmap_ops);
> + } else {
> + obj = i915_gem_object_get_rcu(container_of(node, struct drm_i915_gem_object, base.vma_node));
> +
> + GEM_BUG_ON(obj && !obj->ops->mmap_ops);
> + }
> }
> drm_vma_offset_unlock_lookup(dev->vma_offset_manager);
> rcu_read_unlock();
> @@ -912,7 +930,6 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
> }
>
> vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP;
> - vma->vm_private_data = mmo;
>
> /*
> * We keep the ref on mmo->obj, not vm_file, but we require
> @@ -926,6 +943,15 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
> /* Drop the initial creation reference, the vma is now holding one. */
> fput(anon);
>
> + if (obj->ops->mmap_ops) {
> + vma->vm_page_prot = pgprot_decrypted(vm_get_page_prot(vma->vm_flags));
> + vma->vm_ops = obj->ops->mmap_ops;
> + vma->vm_private_data = node->driver_private;
> + return 0;
> + }
> +
> + vma->vm_private_data = mmo;
> +
> switch (mmo->mmap_type) {
> case I915_MMAP_TYPE_WC:
> vma->vm_page_prot =
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
> index a3ad8cf4eefd..ff59e6c640e6 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
> @@ -342,14 +342,14 @@ struct scatterlist *
> __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
> struct i915_gem_object_page_iter *iter,
> unsigned int n,
> - unsigned int *offset, bool allow_alloc);
> + unsigned int *offset, bool allow_alloc, bool dma);
>
> static inline struct scatterlist *
> i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
> unsigned int n,
> unsigned int *offset, bool allow_alloc)
> {
> - return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc);
> + return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc, false);
> }
>
> static inline struct scatterlist *
> @@ -357,7 +357,7 @@ i915_gem_object_get_sg_dma(struct drm_i915_gem_object *obj,
> unsigned int n,
> unsigned int *offset, bool allow_alloc)
> {
> - return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc);
> + return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc, true);
> }
>
> struct page *
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
> index 68313474e6a6..2a23b77424b3 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
> @@ -61,6 +61,7 @@ struct drm_i915_gem_object_ops {
> const struct drm_i915_gem_pread *arg);
> int (*pwrite)(struct drm_i915_gem_object *obj,
> const struct drm_i915_gem_pwrite *arg);
> + u64 (*mmap_offset)(struct drm_i915_gem_object *obj);
>
> int (*dmabuf_export)(struct drm_i915_gem_object *obj);
>
> @@ -79,6 +80,7 @@ struct drm_i915_gem_object_ops {
> void (*delayed_free)(struct drm_i915_gem_object *obj);
> void (*release)(struct drm_i915_gem_object *obj);
>
> + const struct vm_operations_struct *mmap_ops;
> const char *name; /* friendly name for debug, e.g. lockdep classes */
> };
>
> @@ -328,6 +330,7 @@ struct drm_i915_gem_object {
>
> struct {
> struct sg_table *cached_io_st;
> + struct i915_gem_object_page_iter get_io_page;
> bool created:1;
> } ttm;
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> index 6444e097016d..086005c1c7ea 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> @@ -467,9 +467,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
> struct i915_gem_object_page_iter *iter,
> unsigned int n,
> unsigned int *offset,
> - bool allow_alloc)
> + bool allow_alloc, bool dma)
> {
> - const bool dma = iter == &obj->mm.get_dma_page;
> struct scatterlist *sg;
> unsigned int idx, count;
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> index 17598930a99e..d0be957326e0 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> @@ -13,6 +13,7 @@
> #include "gem/i915_gem_object.h"
> #include "gem/i915_gem_region.h"
> #include "gem/i915_gem_ttm.h"
> +#include "gem/i915_gem_mman.h"
>
> #define I915_PL_LMEM0 TTM_PL_PRIV
> #define I915_PL_SYSTEM TTM_PL_SYSTEM
> @@ -158,11 +159,20 @@ static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
>
> static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
> {
> - if (obj->ttm.cached_io_st) {
> - sg_free_table(obj->ttm.cached_io_st);
> - kfree(obj->ttm.cached_io_st);
> - obj->ttm.cached_io_st = NULL;
> - }
> + struct radix_tree_iter iter;
> + void __rcu **slot;
> +
> + if (!obj->ttm.cached_io_st)
> + return;
> +
> + rcu_read_lock();
> + radix_tree_for_each_slot(slot, &obj->ttm.get_io_page.radix, &iter, 0)
> + radix_tree_delete(&obj->ttm.get_io_page.radix, iter.index);
> + rcu_read_unlock();
> +
> + sg_free_table(obj->ttm.cached_io_st);
> + kfree(obj->ttm.cached_io_st);
> + obj->ttm.cached_io_st = NULL;
> }
>
> static void i915_ttm_purge(struct drm_i915_gem_object *obj)
> @@ -338,12 +348,42 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
> ttm_bo_move_sync_cleanup(bo, dst_mem);
> i915_ttm_free_cached_io_st(obj);
>
> - if (!dst_man->use_tt)
> + if (!dst_man->use_tt) {
> obj->ttm.cached_io_st = dst_st;
> + obj->ttm.get_io_page.sg_pos = dst_st->sgl;
> + obj->ttm.get_io_page.sg_idx = 0;
> + }
>
> return 0;
> }
>
> +static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource *mem)
> +{
> + if (mem->mem_type < I915_PL_LMEM0)
> + return 0;
> +
> + /* We may need to revisit this later, but this allows all caching to be used in mmap */
> + mem->bus.caching = ttm_cached;
Since we're now using the TTM bo offsets, we might as well just make
this ttm_write_combined now.
> + mem->bus.is_iomem = true;
> +
> + return 0;
> +}
> +
> +static unsigned long i915_ttm_io_mem_pfn(struct ttm_buffer_object *bo,
> + unsigned long page_offset)
> +{
> + struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
> + unsigned long base = obj->mm.region->iomap.base - obj->mm.region->region.start;
> + struct scatterlist *sg;
> + unsigned int ofs;
> +
> + GEM_WARN_ON(bo->ttm);
> +
> + sg = __i915_gem_object_get_sg(obj, &obj->ttm.get_io_page, page_offset, &ofs, true, true);
> +
> + return ((base + sg_dma_address(sg)) >> PAGE_SHIFT) + ofs;
> +}
> +
> static struct ttm_device_funcs i915_ttm_bo_driver = {
> .ttm_tt_create = i915_ttm_tt_create,
> .ttm_tt_unpopulate = i915_ttm_tt_unpopulate,
> @@ -354,6 +394,8 @@ static struct ttm_device_funcs i915_ttm_bo_driver = {
> .verify_access = NULL,
> .swap_notify = i915_ttm_swap_notify,
> .delete_mem_notify = i915_ttm_delete_mem_notify,
> + .io_mem_reserve = i915_ttm_io_mem_reserve,
> + .io_mem_pfn = i915_ttm_io_mem_pfn,
> };
>
> /**
> @@ -461,7 +503,68 @@ static void i915_ttm_delayed_free(struct drm_i915_gem_object *obj)
> }
> }
>
> -static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
> +static vm_fault_t vm_fault_ttm(struct vm_fault *vmf)
> +{
> + struct vm_area_struct *area = vmf->vma;
> + struct drm_i915_gem_object *obj =
> + i915_ttm_to_gem(area->vm_private_data);
> +
> + /* Sanity check that we allow writing into this object */
> + if (unlikely(i915_gem_object_is_readonly(obj) &&
> + area->vm_flags & VM_WRITE))
> + return VM_FAULT_SIGBUS;
> +
> + return ttm_bo_vm_fault(vmf);
> +}
> +
> +static int
> +vm_access_ttm(struct vm_area_struct *area, unsigned long addr,
> + void *buf, int len, int write)
> +{
> + struct drm_i915_gem_object *obj =
> + i915_ttm_to_gem(area->vm_private_data);
> +
> + if (i915_gem_object_is_readonly(obj) && write)
> + return -EACCES;
> +
> + return ttm_bo_vm_access(area, addr, buf, len, write);
> +}
> +
> +static void ttm_vm_open(struct vm_area_struct *vma)
> +{
> + struct drm_i915_gem_object *obj =
> + i915_ttm_to_gem(vma->vm_private_data);
> +
> + GEM_BUG_ON(!obj);
> + i915_gem_object_get(obj);
> +}
> +
> +static void ttm_vm_close(struct vm_area_struct *vma)
> +{
> + struct drm_i915_gem_object *obj =
> + i915_ttm_to_gem(vma->vm_private_data);
> +
> + GEM_BUG_ON(!obj);
> + i915_gem_object_put(obj);
> +}
> +
> +
> +static const struct vm_operations_struct vm_ops_ttm = {
> + .fault = vm_fault_ttm,
> + .access = vm_access_ttm,
> + .open = ttm_vm_open,
> + .close = ttm_vm_close,
> +};
> +
> +static u64 i915_ttm_mmap_offset(struct drm_i915_gem_object *obj)
> +{
> + /* The ttm_bo must be allocated with I915_BO_ALLOC_USER */
> + GEM_BUG_ON(!drm_mm_node_allocated(&obj->base.vma_node.vm_node));
> +
> + return drm_vma_node_offset_addr(&obj->base.vma_node);
> +}
> +
> +const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
> .name = "i915_gem_object_ttm",
> .flags = I915_GEM_OBJECT_HAS_IOMEM,
>
> @@ -470,6 +573,8 @@ static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
> .truncate = i915_ttm_purge,
> .adjust_lru = i915_ttm_adjust_lru,
> .delayed_free = i915_ttm_delayed_free,
> + .mmap_offset = i915_ttm_mmap_offset,
> + .mmap_ops = &vm_ops_ttm,
> };
>
> void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
Put a mutex_destroy(&obj->ttm.get_io_page.lock) here?
> @@ -518,6 +623,8 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
> i915_gem_object_make_unshrinkable(obj);
> obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
> i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
> + INIT_RADIX_TREE(&obj->ttm.get_io_page.radix, GFP_KERNEL | __GFP_NOWARN);
> + mutex_init(&obj->ttm.get_io_page.lock);
>
> bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device :
> ttm_bo_type_kernel;
> @@ -529,6 +636,7 @@ int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
> * Similarly, in delayed_destroy, we can't call ttm_bo_put()
> * until successful initialization.
> */
> + obj->base.vma_node.driver_private = i915_gem_to_ttm(obj);
> ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size,
> bo_type, &i915_sys_placement, alignment,
> true, NULL, NULL, i915_ttm_bo_destroy);
> diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
> index 05a3b29f545e..ca69a29b7f2a 100644
> --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
> +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
> @@ -578,16 +578,17 @@ static bool assert_mmap_offset(struct drm_i915_private *i915,
> int expected)
> {
> struct drm_i915_gem_object *obj;
> - struct i915_mmap_offset *mmo;
> + u64 offset;
> + int ret;
>
> obj = i915_gem_object_create_internal(i915, size);
> if (IS_ERR(obj))
> - return false;
> + return expected && expected == PTR_ERR(obj);
>
> - mmo = mmap_offset_attach(obj, I915_MMAP_OFFSET_GTT, NULL);
> + ret = __assign_mmap_offset(obj, I915_MMAP_TYPE_GTT, &offset, NULL);
> i915_gem_object_put(obj);
>
> - return PTR_ERR_OR_ZERO(mmo) == expected;
> + return ret == expected;
> }
>
> static void disable_retire_worker(struct drm_i915_private *i915)
> @@ -622,8 +623,8 @@ static int igt_mmap_offset_exhaustion(void *arg)
> struct drm_mm *mm = &i915->drm.vma_offset_manager->vm_addr_space_mm;
> struct drm_i915_gem_object *obj;
> struct drm_mm_node *hole, *next;
> - struct i915_mmap_offset *mmo;
> int loop, err = 0;
> + u64 offset;
>
> /* Disable background reaper */
> disable_retire_worker(i915);
> @@ -684,13 +685,13 @@ static int igt_mmap_offset_exhaustion(void *arg)
> obj = i915_gem_object_create_internal(i915, PAGE_SIZE);
> if (IS_ERR(obj)) {
> err = PTR_ERR(obj);
> + pr_err("Unable to create object for reclaimed hole\n");
> goto out;
> }
>
> - mmo = mmap_offset_attach(obj, I915_MMAP_OFFSET_GTT, NULL);
> - if (IS_ERR(mmo)) {
> + err = __assign_mmap_offset(obj, I915_MMAP_TYPE_GTT, &offset, NULL);
> + if (err) {
> pr_err("Unable to insert object into reclaimed hole\n");
> - err = PTR_ERR(mmo);
> goto err_obj;
> }
>
> @@ -865,10 +866,10 @@ static int __igt_mmap(struct drm_i915_private *i915,
> struct drm_i915_gem_object *obj,
> enum i915_mmap_type type)
> {
> - struct i915_mmap_offset *mmo;
> struct vm_area_struct *area;
> unsigned long addr;
> int err, i;
> + u64 offset;
>
> if (!can_mmap(obj, type))
> return 0;
> @@ -879,11 +880,11 @@ static int __igt_mmap(struct drm_i915_private *i915,
> if (err)
> return err;
>
> - mmo = mmap_offset_attach(obj, type, NULL);
> - if (IS_ERR(mmo))
> - return PTR_ERR(mmo);
> + err = __assign_mmap_offset(obj, type, &offset, NULL);
> + if (err)
> + return err;
>
> - addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
> + addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
> if (IS_ERR_VALUE(addr))
> return addr;
>
> @@ -897,13 +898,6 @@ static int __igt_mmap(struct drm_i915_private *i915,
> goto out_unmap;
> }
>
> - if (area->vm_private_data != mmo) {
> - pr_err("%s: vm_area_struct did not point back to our mmap_offset object!\n",
> - obj->mm.region->name);
> - err = -EINVAL;
> - goto out_unmap;
> - }
> -
> for (i = 0; i < obj->base.size / sizeof(u32); i++) {
> u32 __user *ux = u64_to_user_ptr((u64)(addr + i * sizeof(*ux)));
> u32 x;
> @@ -961,7 +955,7 @@ static int igt_mmap(void *arg)
> struct drm_i915_gem_object *obj;
> int err;
>
> - obj = i915_gem_object_create_region(mr, sizes[i], 0);
> + obj = i915_gem_object_create_region(mr, sizes[i], I915_BO_ALLOC_USER);
> if (obj == ERR_PTR(-ENODEV))
> continue;
>
> @@ -1004,12 +998,12 @@ static int __igt_mmap_access(struct drm_i915_private *i915,
> struct drm_i915_gem_object *obj,
> enum i915_mmap_type type)
> {
> - struct i915_mmap_offset *mmo;
> unsigned long __user *ptr;
> unsigned long A, B;
> unsigned long x, y;
> unsigned long addr;
> int err;
> + u64 offset;
>
> memset(&A, 0xAA, sizeof(A));
> memset(&B, 0xBB, sizeof(B));
> @@ -1017,11 +1011,11 @@ static int __igt_mmap_access(struct drm_i915_private *i915,
> if (!can_mmap(obj, type) || !can_access(obj))
> return 0;
>
> - mmo = mmap_offset_attach(obj, type, NULL);
> - if (IS_ERR(mmo))
> - return PTR_ERR(mmo);
> + err = __assign_mmap_offset(obj, type, &offset, NULL);
> + if (err)
> + return err;
>
> - addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
> + addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
> if (IS_ERR_VALUE(addr))
> return addr;
> ptr = (unsigned long __user *)addr;
> @@ -1081,7 +1075,7 @@ static int igt_mmap_access(void *arg)
> struct drm_i915_gem_object *obj;
> int err;
>
> - obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
> + obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
> if (obj == ERR_PTR(-ENODEV))
> continue;
>
> @@ -1111,11 +1105,11 @@ static int __igt_mmap_gpu(struct drm_i915_private *i915,
> enum i915_mmap_type type)
> {
> struct intel_engine_cs *engine;
> - struct i915_mmap_offset *mmo;
> unsigned long addr;
> u32 __user *ux;
> u32 bbe;
> int err;
> + u64 offset;
>
> /*
> * Verify that the mmap access into the backing store aligns with
> @@ -1132,11 +1126,11 @@ static int __igt_mmap_gpu(struct drm_i915_private *i915,
> if (err)
> return err;
>
> - mmo = mmap_offset_attach(obj, type, NULL);
> - if (IS_ERR(mmo))
> - return PTR_ERR(mmo);
> + err = __assign_mmap_offset(obj, type, &offset, NULL);
> + if (err)
> + return err;
>
> - addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
> + addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
> if (IS_ERR_VALUE(addr))
> return addr;
>
> @@ -1226,7 +1220,7 @@ static int igt_mmap_gpu(void *arg)
> struct drm_i915_gem_object *obj;
> int err;
>
> - obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
> + obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
> if (obj == ERR_PTR(-ENODEV))
> continue;
>
> @@ -1303,18 +1297,18 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
> struct drm_i915_gem_object *obj,
> enum i915_mmap_type type)
> {
> - struct i915_mmap_offset *mmo;
> unsigned long addr;
> int err;
> + u64 offset;
>
> if (!can_mmap(obj, type))
> return 0;
>
> - mmo = mmap_offset_attach(obj, type, NULL);
> - if (IS_ERR(mmo))
> - return PTR_ERR(mmo);
> + err = __assign_mmap_offset(obj, type, &offset, NULL);
> + if (err)
> + return err;
>
> - addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED);
> + addr = igt_mmap_offset(i915, offset, obj->base.size, PROT_WRITE, MAP_SHARED);
> if (IS_ERR_VALUE(addr))
> return addr;
>
> @@ -1350,10 +1344,20 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
> }
> }
>
> - err = check_absent(addr, obj->base.size);
> - if (err) {
> - pr_err("%s: was not absent\n", obj->mm.region->name);
> - goto out_unmap;
> + if (!obj->ops->mmap_ops) {
> + err = check_absent(addr, obj->base.size);
> + if (err) {
> + pr_err("%s: was not absent\n", obj->mm.region->name);
> + goto out_unmap;
> + }
> + } else {
> + /* ttm allows access to evicted regions by design */
> +
> + err = check_present(addr, obj->base.size);
> + if (err) {
> + pr_err("%s: was not present\n", obj->mm.region->name);
> + goto out_unmap;
> + }
> }
>
> out_unmap:
> @@ -1371,7 +1375,7 @@ static int igt_mmap_revoke(void *arg)
> struct drm_i915_gem_object *obj;
> int err;
>
> - obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0);
> + obj = i915_gem_object_create_region(mr, PAGE_SIZE, I915_BO_ALLOC_USER);
> if (obj == ERR_PTR(-ENODEV))
> continue;
>
> diff --git a/drivers/gpu/drm/i915/selftests/igt_mmap.c b/drivers/gpu/drm/i915/selftests/igt_mmap.c
> index 583a4ff8b8c9..e8286c28de91 100644
> --- a/drivers/gpu/drm/i915/selftests/igt_mmap.c
> +++ b/drivers/gpu/drm/i915/selftests/igt_mmap.c
> @@ -9,15 +9,28 @@
> #include "i915_drv.h"
> #include "igt_mmap.h"
>
> -unsigned long igt_mmap_node(struct drm_i915_private *i915,
> - struct drm_vma_offset_node *node,
> - unsigned long addr,
> - unsigned long prot,
> - unsigned long flags)
> +unsigned long igt_mmap_offset(struct drm_i915_private *i915,
> + u64 offset,
> + unsigned long size,
> + unsigned long prot,
> + unsigned long flags)
> {
> + struct drm_vma_offset_node *node;
> struct file *file;
> + unsigned long addr;
> int err;
>
> + /* no need to refcount, we own this object */
> + drm_vma_offset_lock_lookup(i915->drm.vma_offset_manager);
> + node = drm_vma_offset_exact_lookup_locked(i915->drm.vma_offset_manager,
> + offset / PAGE_SIZE, size / PAGE_SIZE);
> + drm_vma_offset_unlock_lookup(i915->drm.vma_offset_manager);
> +
> + if (GEM_WARN_ON(!node)) {
> + pr_info("Failed to lookup %Lx\n", offset);
Checkpatch warning here iirc.
> + return -ENOENT;
> + }
> +
> /* Pretend to open("/dev/dri/card0") */
> file = mock_drm_getfile(i915->drm.primary, O_RDWR);
> if (IS_ERR(file))
> @@ -29,7 +42,7 @@ unsigned long igt_mmap_node(struct drm_i915_private *i915,
> goto out_file;
> }
>
> - addr = vm_mmap(file, addr, drm_vma_node_size(node) << PAGE_SHIFT,
> + addr = vm_mmap(file, 0, drm_vma_node_size(node) << PAGE_SHIFT,
> prot, flags, drm_vma_node_offset_addr(node));
>
> drm_vma_node_revoke(node, file->private_data);
> diff --git a/drivers/gpu/drm/i915/selftests/igt_mmap.h b/drivers/gpu/drm/i915/selftests/igt_mmap.h
> index 6e716cb59d7e..acbe34d81a6d 100644
> --- a/drivers/gpu/drm/i915/selftests/igt_mmap.h
> +++ b/drivers/gpu/drm/i915/selftests/igt_mmap.h
> @@ -7,13 +7,15 @@
> #ifndef IGT_MMAP_H
> #define IGT_MMAP_H
>
> +#include <linux/types.h>
> +
> struct drm_i915_private;
> struct drm_vma_offset_node;
>
> -unsigned long igt_mmap_node(struct drm_i915_private *i915,
> - struct drm_vma_offset_node *node,
> - unsigned long addr,
> - unsigned long prot,
> - unsigned long flags);
> +unsigned long igt_mmap_offset(struct drm_i915_private *i915,
> + u64 offset,
> + unsigned long size,
> + unsigned long prot,
> + unsigned long flags);
>
> #endif /* IGT_MMAP_H */
/Thomas
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* [Intel-gfx] ✓ Fi.CI.IGT: success for drm/i915: Move LMEM (VRAM) management over to TTM (rev4)
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
` (18 preceding siblings ...)
(?)
@ 2021-05-27 1:57 ` Patchwork
-1 siblings, 0 replies; 58+ messages in thread
From: Patchwork @ 2021-05-27 1:57 UTC (permalink / raw)
To: Thomas Hellström; +Cc: intel-gfx
[-- Attachment #1.1: Type: text/plain, Size: 30279 bytes --]
== Series Details ==
Series: drm/i915: Move LMEM (VRAM) management over to TTM (rev4)
URL : https://patchwork.freedesktop.org/series/90022/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_10135_full -> Patchwork_20203_full
====================================================
Summary
-------
**SUCCESS**
No regressions found.
Known issues
------------
Here are the changes found in Patchwork_20203_full that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@feature_discovery@display-3x:
- shard-tglb: NOTRUN -> [SKIP][1] ([i915#1839])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@feature_discovery@display-3x.html
* igt@gem_ctx_isolation@preservation-s3@vecs0:
- shard-kbl: [PASS][2] -> [DMESG-WARN][3] ([i915#180]) +5 similar issues
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl4/igt@gem_ctx_isolation@preservation-s3@vecs0.html
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-kbl2/igt@gem_ctx_isolation@preservation-s3@vecs0.html
* igt@gem_ctx_persistence@clone:
- shard-snb: NOTRUN -> [SKIP][4] ([fdo#109271] / [i915#1099]) +5 similar issues
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-snb6/igt@gem_ctx_persistence@clone.html
* igt@gem_exec_capture@pi@rcs0:
- shard-skl: NOTRUN -> [INCOMPLETE][5] ([i915#2369])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl2/igt@gem_exec_capture@pi@rcs0.html
* igt@gem_exec_fair@basic-deadline:
- shard-skl: NOTRUN -> [FAIL][6] ([i915#2846])
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl4/igt@gem_exec_fair@basic-deadline.html
* igt@gem_exec_fair@basic-none-share@rcs0:
- shard-iclb: [PASS][7] -> [FAIL][8] ([i915#2842])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-iclb2/igt@gem_exec_fair@basic-none-share@rcs0.html
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb8/igt@gem_exec_fair@basic-none-share@rcs0.html
* igt@gem_exec_fair@basic-none@vecs0:
- shard-apl: NOTRUN -> [FAIL][9] ([i915#2842] / [i915#3468])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl3/igt@gem_exec_fair@basic-none@vecs0.html
* igt@gem_exec_fair@basic-pace-share@rcs0:
- shard-glk: [PASS][10] -> [FAIL][11] ([i915#2842]) +1 similar issue
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-glk4/igt@gem_exec_fair@basic-pace-share@rcs0.html
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-glk6/igt@gem_exec_fair@basic-pace-share@rcs0.html
* igt@gem_exec_fair@basic-pace@vcs0:
- shard-kbl: [PASS][12] -> [FAIL][13] ([i915#2842]) +1 similar issue
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl2/igt@gem_exec_fair@basic-pace@vcs0.html
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-kbl1/igt@gem_exec_fair@basic-pace@vcs0.html
- shard-tglb: [PASS][14] -> [FAIL][15] ([i915#2842])
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-tglb2/igt@gem_exec_fair@basic-pace@vcs0.html
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb2/igt@gem_exec_fair@basic-pace@vcs0.html
* igt@gem_exec_fair@basic-pace@vecs0:
- shard-kbl: [PASS][16] -> [SKIP][17] ([fdo#109271]) +1 similar issue
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl2/igt@gem_exec_fair@basic-pace@vecs0.html
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-kbl1/igt@gem_exec_fair@basic-pace@vecs0.html
* igt@gem_mmap_gtt@cpuset-basic-small-copy-odd:
- shard-snb: NOTRUN -> [INCOMPLETE][18] ([i915#2055] / [i915#3468])
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-snb7/igt@gem_mmap_gtt@cpuset-basic-small-copy-odd.html
- shard-skl: NOTRUN -> [INCOMPLETE][19] ([i915#198] / [i915#2910])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl4/igt@gem_mmap_gtt@cpuset-basic-small-copy-odd.html
* igt@gem_mmap_gtt@cpuset-medium-copy-xy:
- shard-glk: [PASS][20] -> [INCOMPLETE][21] ([i915#3468])
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-glk4/igt@gem_mmap_gtt@cpuset-medium-copy-xy.html
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-glk9/igt@gem_mmap_gtt@cpuset-medium-copy-xy.html
* igt@gem_mmap_gtt@fault-concurrent-y:
- shard-apl: NOTRUN -> [INCOMPLETE][22] ([i915#3468])
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl3/igt@gem_mmap_gtt@fault-concurrent-y.html
* igt@gem_mmap_gtt@medium-copy-xy:
- shard-kbl: NOTRUN -> [INCOMPLETE][23] ([i915#2502] / [i915#3468])
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-kbl7/igt@gem_mmap_gtt@medium-copy-xy.html
* igt@gem_mmap_offset@clear:
- shard-iclb: [PASS][24] -> [FAIL][25] ([i915#3160])
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-iclb6/igt@gem_mmap_offset@clear.html
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@gem_mmap_offset@clear.html
* igt@gem_ppgtt@flink-and-close-vma-leak:
- shard-glk: [PASS][26] -> [FAIL][27] ([i915#644])
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-glk8/igt@gem_ppgtt@flink-and-close-vma-leak.html
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-glk6/igt@gem_ppgtt@flink-and-close-vma-leak.html
* igt@gem_pread@exhaustion:
- shard-apl: NOTRUN -> [WARN][28] ([i915#2658])
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl3/igt@gem_pread@exhaustion.html
- shard-snb: NOTRUN -> [WARN][29] ([i915#2658])
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-snb2/igt@gem_pread@exhaustion.html
* igt@gem_pwrite@basic-exhaustion:
- shard-skl: NOTRUN -> [WARN][30] ([i915#2658])
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl7/igt@gem_pwrite@basic-exhaustion.html
* igt@gem_userptr_blits@unsync-unmap:
- shard-iclb: NOTRUN -> [SKIP][31] ([i915#3297])
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb8/igt@gem_userptr_blits@unsync-unmap.html
* igt@gen7_exec_parse@batch-without-end:
- shard-iclb: NOTRUN -> [SKIP][32] ([fdo#109289])
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@gen7_exec_parse@batch-without-end.html
- shard-tglb: NOTRUN -> [SKIP][33] ([fdo#109289]) +1 similar issue
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb7/igt@gen7_exec_parse@batch-without-end.html
* igt@gen9_exec_parse@bb-large:
- shard-apl: NOTRUN -> [FAIL][34] ([i915#3296])
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl1/igt@gen9_exec_parse@bb-large.html
* igt@gen9_exec_parse@bb-start-out:
- shard-iclb: NOTRUN -> [SKIP][35] ([fdo#112306])
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@gen9_exec_parse@bb-start-out.html
* igt@i915_pm_dc@dc6-psr:
- shard-tglb: NOTRUN -> [FAIL][36] ([i915#454])
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb7/igt@i915_pm_dc@dc6-psr.html
* igt@i915_pm_lpsp@kms-lpsp@kms-lpsp-dp:
- shard-apl: NOTRUN -> [SKIP][37] ([fdo#109271] / [i915#1937])
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl3/igt@i915_pm_lpsp@kms-lpsp@kms-lpsp-dp.html
* igt@kms_big_joiner@basic:
- shard-apl: NOTRUN -> [SKIP][38] ([fdo#109271] / [i915#2705])
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl7/igt@kms_big_joiner@basic.html
* igt@kms_ccs@pipe-c-random-ccs-data:
- shard-skl: NOTRUN -> [SKIP][39] ([fdo#109271] / [fdo#111304]) +1 similar issue
[39]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl2/igt@kms_ccs@pipe-c-random-ccs-data.html
* igt@kms_ccs@pipe-d-crc-primary-rotation-180:
- shard-iclb: NOTRUN -> [SKIP][40] ([fdo#109278]) +8 similar issues
[40]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@kms_ccs@pipe-d-crc-primary-rotation-180.html
* igt@kms_chamelium@dp-frame-dump:
- shard-iclb: NOTRUN -> [SKIP][41] ([fdo#109284] / [fdo#111827]) +2 similar issues
[41]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@kms_chamelium@dp-frame-dump.html
* igt@kms_chamelium@dp-mode-timings:
- shard-apl: NOTRUN -> [SKIP][42] ([fdo#109271] / [fdo#111827]) +27 similar issues
[42]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl7/igt@kms_chamelium@dp-mode-timings.html
* igt@kms_chamelium@vga-hpd:
- shard-skl: NOTRUN -> [SKIP][43] ([fdo#109271] / [fdo#111827]) +7 similar issues
[43]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl2/igt@kms_chamelium@vga-hpd.html
* igt@kms_chamelium@vga-hpd-without-ddc:
- shard-snb: NOTRUN -> [SKIP][44] ([fdo#109271] / [fdo#111827]) +19 similar issues
[44]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-snb2/igt@kms_chamelium@vga-hpd-without-ddc.html
* igt@kms_color_chamelium@pipe-a-ctm-0-75:
- shard-tglb: NOTRUN -> [SKIP][45] ([fdo#109284] / [fdo#111827]) +4 similar issues
[45]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@kms_color_chamelium@pipe-a-ctm-0-75.html
* igt@kms_color_chamelium@pipe-d-gamma:
- shard-iclb: NOTRUN -> [SKIP][46] ([fdo#109278] / [fdo#109284] / [fdo#111827]) +1 similar issue
[46]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@kms_color_chamelium@pipe-d-gamma.html
* igt@kms_content_protection@dp-mst-lic-type-0:
- shard-tglb: NOTRUN -> [SKIP][47] ([i915#3116])
[47]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@kms_content_protection@dp-mst-lic-type-0.html
* igt@kms_content_protection@lic:
- shard-apl: NOTRUN -> [TIMEOUT][48] ([i915#1319]) +1 similar issue
[48]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl1/igt@kms_content_protection@lic.html
* igt@kms_content_protection@uevent:
- shard-apl: NOTRUN -> [FAIL][49] ([i915#2105])
[49]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl3/igt@kms_content_protection@uevent.html
* igt@kms_cursor_crc@pipe-a-cursor-32x10-rapid-movement:
- shard-tglb: NOTRUN -> [SKIP][50] ([i915#3359]) +1 similar issue
[50]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb7/igt@kms_cursor_crc@pipe-a-cursor-32x10-rapid-movement.html
* igt@kms_cursor_crc@pipe-a-cursor-32x32-sliding:
- shard-tglb: NOTRUN -> [SKIP][51] ([i915#3319])
[51]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@kms_cursor_crc@pipe-a-cursor-32x32-sliding.html
* igt@kms_cursor_crc@pipe-b-cursor-512x512-rapid-movement:
- shard-tglb: NOTRUN -> [SKIP][52] ([fdo#109279] / [i915#3359])
[52]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@kms_cursor_crc@pipe-b-cursor-512x512-rapid-movement.html
* igt@kms_cursor_crc@pipe-b-cursor-512x512-sliding:
- shard-iclb: NOTRUN -> [SKIP][53] ([fdo#109278] / [fdo#109279])
[53]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@kms_cursor_crc@pipe-b-cursor-512x512-sliding.html
* igt@kms_cursor_crc@pipe-b-cursor-64x21-onscreen:
- shard-apl: NOTRUN -> [FAIL][54] ([i915#3444])
[54]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl2/igt@kms_cursor_crc@pipe-b-cursor-64x21-onscreen.html
* igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions:
- shard-skl: [PASS][55] -> [FAIL][56] ([i915#2346])
[55]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl10/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions.html
[56]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl3/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions.html
* igt@kms_dp_dsc@basic-dsc-enable-edp:
- shard-tglb: NOTRUN -> [CRASH][57] ([i915#3494])
[57]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@kms_dp_dsc@basic-dsc-enable-edp.html
* igt@kms_draw_crc@draw-method-xrgb2101010-render-xtiled:
- shard-skl: [PASS][58] -> [DMESG-WARN][59] ([i915#1982]) +1 similar issue
[58]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl5/igt@kms_draw_crc@draw-method-xrgb2101010-render-xtiled.html
[59]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl5/igt@kms_draw_crc@draw-method-xrgb2101010-render-xtiled.html
* igt@kms_flip@2x-flip-vs-rmfb:
- shard-tglb: NOTRUN -> [SKIP][60] ([fdo#111825]) +6 similar issues
[60]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@kms_flip@2x-flip-vs-rmfb.html
* igt@kms_flip@2x-wf_vblank-ts-check:
- shard-iclb: NOTRUN -> [SKIP][61] ([fdo#109274])
[61]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@kms_flip@2x-wf_vblank-ts-check.html
* igt@kms_flip@flip-vs-expired-vblank@b-edp1:
- shard-skl: [PASS][62] -> [FAIL][63] ([i915#79])
[62]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl1/igt@kms_flip@flip-vs-expired-vblank@b-edp1.html
[63]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl10/igt@kms_flip@flip-vs-expired-vblank@b-edp1.html
* igt@kms_flip@flip-vs-suspend-interruptible@b-dp1:
- shard-apl: NOTRUN -> [DMESG-WARN][64] ([i915#180])
[64]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl7/igt@kms_flip@flip-vs-suspend-interruptible@b-dp1.html
* igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile:
- shard-skl: NOTRUN -> [SKIP][65] ([fdo#109271] / [i915#2642])
[65]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl2/igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile.html
* igt@kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile:
- shard-apl: NOTRUN -> [SKIP][66] ([fdo#109271] / [i915#2642])
[66]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl7/igt@kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile.html
* igt@kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs:
- shard-apl: NOTRUN -> [SKIP][67] ([fdo#109271] / [i915#2672])
[67]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl1/igt@kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs.html
* igt@kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-pwrite:
- shard-snb: NOTRUN -> [SKIP][68] ([fdo#109271]) +418 similar issues
[68]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-snb2/igt@kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-pwrite.html
* igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-blt:
- shard-skl: NOTRUN -> [SKIP][69] ([fdo#109271]) +74 similar issues
[69]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl2/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-blt.html
* igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-render:
- shard-kbl: NOTRUN -> [SKIP][70] ([fdo#109271])
[70]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-kbl7/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-render.html
* igt@kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-pwrite:
- shard-iclb: NOTRUN -> [SKIP][71] ([fdo#109280]) +5 similar issues
[71]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-pwrite.html
* igt@kms_hdr@bpc-switch:
- shard-skl: [PASS][72] -> [FAIL][73] ([i915#1188])
[72]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl10/igt@kms_hdr@bpc-switch.html
[73]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl6/igt@kms_hdr@bpc-switch.html
* igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-d:
- shard-apl: NOTRUN -> [SKIP][74] ([fdo#109271] / [i915#533]) +1 similar issue
[74]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl7/igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-d.html
* igt@kms_pipe_crc_basic@nonblocking-crc-pipe-d-frame-sequence:
- shard-skl: NOTRUN -> [SKIP][75] ([fdo#109271] / [i915#533]) +1 similar issue
[75]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl7/igt@kms_pipe_crc_basic@nonblocking-crc-pipe-d-frame-sequence.html
* igt@kms_plane_alpha_blend@pipe-a-alpha-basic:
- shard-apl: NOTRUN -> [FAIL][76] ([fdo#108145] / [i915#265]) +3 similar issues
[76]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl2/igt@kms_plane_alpha_blend@pipe-a-alpha-basic.html
* igt@kms_plane_alpha_blend@pipe-b-alpha-opaque-fb:
- shard-skl: NOTRUN -> [FAIL][77] ([fdo#108145] / [i915#265])
[77]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl7/igt@kms_plane_alpha_blend@pipe-b-alpha-opaque-fb.html
* igt@kms_plane_alpha_blend@pipe-b-alpha-transparent-fb:
- shard-apl: NOTRUN -> [FAIL][78] ([i915#265])
[78]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl1/igt@kms_plane_alpha_blend@pipe-b-alpha-transparent-fb.html
* igt@kms_psr2_sf@overlay-plane-update-sf-dmg-area-2:
- shard-apl: NOTRUN -> [SKIP][79] ([fdo#109271] / [i915#658]) +4 similar issues
[79]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl1/igt@kms_psr2_sf@overlay-plane-update-sf-dmg-area-2.html
* igt@kms_psr2_sf@plane-move-sf-dmg-area-2:
- shard-skl: NOTRUN -> [SKIP][80] ([fdo#109271] / [i915#658]) +3 similar issues
[80]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl4/igt@kms_psr2_sf@plane-move-sf-dmg-area-2.html
* igt@kms_psr@psr2_cursor_plane_onoff:
- shard-iclb: NOTRUN -> [SKIP][81] ([fdo#109441])
[81]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@kms_psr@psr2_cursor_plane_onoff.html
* igt@kms_psr@psr2_primary_mmap_cpu:
- shard-iclb: [PASS][82] -> [SKIP][83] ([fdo#109441]) +2 similar issues
[82]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-iclb2/igt@kms_psr@psr2_primary_mmap_cpu.html
[83]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb1/igt@kms_psr@psr2_primary_mmap_cpu.html
* igt@kms_vblank@pipe-d-ts-continuation-idle:
- shard-apl: NOTRUN -> [SKIP][84] ([fdo#109271]) +284 similar issues
[84]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl7/igt@kms_vblank@pipe-d-ts-continuation-idle.html
* igt@kms_writeback@writeback-check-output:
- shard-apl: NOTRUN -> [SKIP][85] ([fdo#109271] / [i915#2437]) +2 similar issues
[85]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl1/igt@kms_writeback@writeback-check-output.html
* igt@kms_writeback@writeback-pixel-formats:
- shard-iclb: NOTRUN -> [SKIP][86] ([i915#2437])
[86]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@kms_writeback@writeback-pixel-formats.html
* igt@nouveau_crc@pipe-b-source-outp-complete:
- shard-tglb: NOTRUN -> [SKIP][87] ([i915#2530])
[87]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@nouveau_crc@pipe-b-source-outp-complete.html
* igt@perf@polling-small-buf:
- shard-skl: [PASS][88] -> [FAIL][89] ([i915#1722])
[88]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl4/igt@perf@polling-small-buf.html
[89]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl3/igt@perf@polling-small-buf.html
* igt@prime_nv_pcopy@test3_3:
- shard-iclb: NOTRUN -> [SKIP][90] ([fdo#109291])
[90]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@prime_nv_pcopy@test3_3.html
- shard-tglb: NOTRUN -> [SKIP][91] ([fdo#109291]) +1 similar issue
[91]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb7/igt@prime_nv_pcopy@test3_3.html
* igt@syncobj_timeline@etime-multi-wait-for-submit-unsubmitted:
- shard-glk: [PASS][92] -> [DMESG-WARN][93] ([i915#118] / [i915#95])
[92]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-glk7/igt@syncobj_timeline@etime-multi-wait-for-submit-unsubmitted.html
[93]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-glk6/igt@syncobj_timeline@etime-multi-wait-for-submit-unsubmitted.html
* igt@sysfs_clients@fair-0:
- shard-apl: NOTRUN -> [SKIP][94] ([fdo#109271] / [i915#2994]) +2 similar issues
[94]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-apl3/igt@sysfs_clients@fair-0.html
* igt@sysfs_clients@fair-3:
- shard-skl: NOTRUN -> [SKIP][95] ([fdo#109271] / [i915#2994])
[95]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl7/igt@sysfs_clients@fair-3.html
* igt@sysfs_heartbeat_interval@mixed@vecs0:
- shard-skl: [PASS][96] -> [FAIL][97] ([i915#1731])
[96]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl4/igt@sysfs_heartbeat_interval@mixed@vecs0.html
[97]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl3/igt@sysfs_heartbeat_interval@mixed@vecs0.html
#### Possible fixes ####
* igt@gem_exec_fair@basic-pace-share@rcs0:
- shard-tglb: [FAIL][98] ([i915#2842]) -> [PASS][99]
[98]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-tglb6/igt@gem_exec_fair@basic-pace-share@rcs0.html
[99]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@gem_exec_fair@basic-pace-share@rcs0.html
* igt@gem_exec_fair@basic-pace@rcs0:
- shard-kbl: [FAIL][100] ([i915#2842]) -> [PASS][101]
[100]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl2/igt@gem_exec_fair@basic-pace@rcs0.html
[101]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-kbl1/igt@gem_exec_fair@basic-pace@rcs0.html
* igt@gem_exec_fence@parallel@bcs0:
- shard-glk: [DMESG-WARN][102] ([i915#118] / [i915#95]) -> [PASS][103]
[102]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-glk8/igt@gem_exec_fence@parallel@bcs0.html
[103]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-glk7/igt@gem_exec_fence@parallel@bcs0.html
* igt@gem_mmap_gtt@big-copy:
- shard-glk: [FAIL][104] ([i915#307]) -> [PASS][105]
[104]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-glk2/igt@gem_mmap_gtt@big-copy.html
[105]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-glk6/igt@gem_mmap_gtt@big-copy.html
* igt@gem_mmap_gtt@cpuset-basic-small-copy-xy:
- shard-tglb: [INCOMPLETE][106] ([i915#3468]) -> [PASS][107]
[106]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-tglb2/igt@gem_mmap_gtt@cpuset-basic-small-copy-xy.html
[107]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb6/igt@gem_mmap_gtt@cpuset-basic-small-copy-xy.html
* igt@gem_mmap_gtt@cpuset-big-copy:
- shard-iclb: [FAIL][108] ([i915#2428]) -> [PASS][109]
[108]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-iclb3/igt@gem_mmap_gtt@cpuset-big-copy.html
[109]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb7/igt@gem_mmap_gtt@cpuset-big-copy.html
* igt@gem_mmap_gtt@cpuset-medium-copy-xy:
- shard-tglb: [INCOMPLETE][110] ([i915#3468] / [i915#750]) -> [PASS][111]
[110]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-tglb1/igt@gem_mmap_gtt@cpuset-medium-copy-xy.html
[111]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-tglb7/igt@gem_mmap_gtt@cpuset-medium-copy-xy.html
* igt@gem_mmap_gtt@medium-copy-xy:
- shard-iclb: [INCOMPLETE][112] ([i915#2502] / [i915#3468]) -> [PASS][113]
[112]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-iclb7/igt@gem_mmap_gtt@medium-copy-xy.html
[113]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb3/igt@gem_mmap_gtt@medium-copy-xy.html
* igt@i915_module_load@reload-with-fault-injection:
- shard-snb: [INCOMPLETE][114] ([i915#2880]) -> [PASS][115]
[114]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-snb6/igt@i915_module_load@reload-with-fault-injection.html
[115]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-snb6/igt@i915_module_load@reload-with-fault-injection.html
* igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size:
- shard-skl: [FAIL][116] ([i915#2346] / [i915#533]) -> [PASS][117]
[116]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl1/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size.html
[117]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl7/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size.html
* igt@kms_flip@flip-vs-expired-vblank@a-edp1:
- shard-skl: [FAIL][118] ([i915#79]) -> [PASS][119]
[118]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl1/igt@kms_flip@flip-vs-expired-vblank@a-edp1.html
[119]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl10/igt@kms_flip@flip-vs-expired-vblank@a-edp1.html
* igt@kms_flip@flip-vs-suspend@a-edp1:
- shard-skl: [INCOMPLETE][120] ([i915#146] / [i915#198]) -> [PASS][121]
[120]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl1/igt@kms_flip@flip-vs-suspend@a-edp1.html
[121]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl7/igt@kms_flip@flip-vs-suspend@a-edp1.html
* igt@kms_hdr@bpc-switch-suspend:
- shard-kbl: [DMESG-WARN][122] ([i915#180]) -> [PASS][123] +3 similar issues
[122]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl4/igt@kms_hdr@bpc-switch-suspend.html
[123]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-kbl7/igt@kms_hdr@bpc-switch-suspend.html
* igt@kms_plane_alpha_blend@pipe-b-coverage-7efc:
- shard-skl: [FAIL][124] ([fdo#108145] / [i915#265]) -> [PASS][125]
[124]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl5/igt@kms_plane_alpha_blend@pipe-b-coverage-7efc.html
[125]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl5/igt@kms_plane_alpha_blend@pipe-b-coverage-7efc.html
* igt@kms_vblank@pipe-c-ts-continuation-suspend:
- shard-skl: [INCOMPLETE][126] ([i915#198] / [i915#2828]) -> [PASS][127]
[126]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-skl1/igt@kms_vblank@pipe-c-ts-continuation-suspend.html
[127]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-skl4/igt@kms_vblank@pipe-c-ts-continuation-suspend.html
#### Warnings ####
* igt@kms_psr2_sf@overlay-primary-update-sf-dmg-area-3:
- shard-iclb: [SKIP][128] ([i915#2920]) -> [SKIP][129] ([i915#658]) +2 similar issues
[128]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-iclb2/igt@kms_psr2_sf@overlay-primary-update-sf-dmg-area-3.html
[129]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-iclb1/igt@kms_psr2_sf@overlay-primary-update-sf-dmg-area-3.html
* igt@runner@aborted:
- shard-kbl: ([FAIL][130], [FAIL][131], [FAIL][132], [FAIL][133], [FAIL][134], [FAIL][135], [FAIL][136], [FAIL][137], [FAIL][138], [FAIL][139], [FAIL][140], [FAIL][141], [FAIL][142], [FAIL][143], [FAIL][144]) ([i915#1436] / [i915#180] / [i915#1814] / [i915#2722] / [i915#3002] / [i915#3363] / [i915#92]) -> ([FAIL][145], [FAIL][146], [FAIL][147], [FAIL][148], [FAIL][149], [FAIL][150], [FAIL][151], [FAIL][152], [FAIL][153], [FAIL][154], [FAIL][155], [FAIL][156], [FAIL][157], [FAIL][158], [FAIL][159], [FAIL][160], [FAIL][161], [FAIL][162]) ([i915#1436] / [i915#180] / [i915#1814] / [i915#2722] / [i915#3002] / [i915#3363] / [i915#602] / [i915#92])
[130]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl7/igt@runner@aborted.html
[131]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl4/igt@runner@aborted.html
[132]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl4/igt@runner@aborted.html
[133]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl1/igt@runner@aborted.html
[134]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl4/igt@runner@aborted.html
[135]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl2/igt@runner@aborted.html
[136]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl4/igt@runner@aborted.html
[137]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl7/igt@runner@aborted.html
[138]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl4/igt@runner@aborted.html
[139]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl2/igt@runner@aborted.html
[140]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl7/igt@runner@aborted.html
[141]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl1/igt@runner@aborted.html
[142]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl2/igt@runner@aborted.html
[143]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl3/igt@runner@aborted.html
[144]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10135/shard-kbl3/igt@runner@aborted.html
[145]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/shard-kbl7/igt@runner@aborted.html
[146]: https://intel-gfx-ci.01.org/tr
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20203/index.html
[-- Attachment #1.2: Type: text/html, Size: 34110 bytes --]
[-- Attachment #2: Type: text/plain, Size: 160 bytes --]
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
2021-05-26 13:26 ` [Intel-gfx] " Christian König
@ 2021-05-27 7:33 ` Thomas Hellström (Intel)
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström (Intel) @ 2021-05-27 7:33 UTC (permalink / raw)
To: Christian König, Thomas Hellström, intel-gfx, dri-devel
Hi, Christian,
Thanks for reviewing.
On 5/26/21 3:26 PM, Christian König wrote:
> Am 26.05.21 um 13:32 schrieb Thomas Hellström:
>> We are calling the eviction_valuable driver callback at eviction time to
>> determine whether we actually can evict a buffer object.
>> The upcoming i915 TTM backend needs the same functionality for swapout,
>> and that might actually be beneficial to other drivers as well.
>>
>> Add an eviction_valuable call also in the swapout path. Try to keep the
>> current behaviour for all drivers by returning true if the buffer object
>> is already in the TTM_PL_SYSTEM placement. We change behaviour for the
>> case where a buffer object is in a TT backed placement when swapped out,
>> in which case the drivers normal eviction_valuable path is run.
>>
>> Finally make sure we don't try to swapout a bo that was recently purged
>> and therefore unpopulated.
>>
>> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>> Cc: Christian König <christian.koenig@amd.com>
>> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>> ---
>> v3:
>> - Don't export ttm_tt_unpopulate
>> - Fix confusion reading the locked pointer instead of the value
>> pointed to in ttm_bo_evict_swapout_allowable (Reported by
>> Maarten Lankhorst)
>> ---
>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
>> drivers/gpu/drm/ttm/ttm_bo.c | 43 ++++++++++++++++---------
>> drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
>> 3 files changed, 34 insertions(+), 16 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> index 3bc3aebfef7c..45d194bffc3f 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> @@ -1348,6 +1348,10 @@ static bool
>> amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
>> struct dma_fence *f;
>> int i;
>> + /* Swapout? */
>> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
>> + return true;
>> +
>> if (bo->type == ttm_bo_type_kernel &&
>> !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
>> return false;
>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
>> index be0406466460..1b2d062266ed 100644
>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>> @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct ttm_buffer_object
>> *bo,
>> bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
>> const struct ttm_place *place)
>> {
>> + dma_resv_assert_held(bo->base.resv);
>> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
>> + return true;
>> +
>> /* Don't evict this BO if it's outside of the
>> * requested placement range
>> */
>> @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
>> * b. Otherwise, trylock it.
>> */
>> static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object
>> *bo,
>> - struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
>> + struct ttm_operation_ctx *ctx,
>> + const struct ttm_place *place,
>> + bool *locked, bool *busy)
>> {
>> bool ret = false;
>> @@ -576,6 +582,14 @@ static bool
>> ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
>> *busy = !ret;
>> }
>> + if (ret && place && !bo->bdev->funcs->eviction_valuable(bo,
>> place)) {
>> + ret = false;
>> + if (*locked) {
>> + dma_resv_unlock(bo->base.resv);
>> + *locked = false;
>> + }
>> + }
>> +
>> return ret;
>> }
>> @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
>> list_for_each_entry(bo, &man->lru[i], lru) {
>> bool busy;
>> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
>> - &busy)) {
>> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
>> + &locked, &busy)) {
>> if (busy && !busy_bo && ticket !=
>> dma_resv_locking_ctx(bo->base.resv))
>> busy_bo = bo;
>> continue;
>> }
>> - if (place && !bdev->funcs->eviction_valuable(bo,
>> - place)) {
>> - if (locked)
>> - dma_resv_unlock(bo->base.resv);
>> - continue;
>> - }
>> if (!ttm_bo_get_unless_zero(bo)) {
>> if (locked)
>> dma_resv_unlock(bo->base.resv);
>> @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
>> int ttm_bo_swapout(struct ttm_buffer_object *bo, struct
>> ttm_operation_ctx *ctx,
>> gfp_t gfp_flags)
>> {
>> + struct ttm_place place = {};
>> bool locked;
>> int ret;
>> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
>> + /*
>> + * While the bo may already reside in SYSTEM placement, set
>> + * SYSTEM as new placement to cover also the move further below.
>> + * The driver may use the fact that we're moving from SYSTEM
>> + * as an indication that we're about to swap out.
>> + */
>> + place.mem_type = TTM_PL_SYSTEM;
>> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked,
>> NULL))
>> return -EBUSY;
>> if (!ttm_bo_get_unless_zero(bo)) {
>> @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct ttm_buffer_object
>> *bo, struct ttm_operation_ctx *ctx,
>> if (bo->mem.mem_type != TTM_PL_SYSTEM) {
>> struct ttm_operation_ctx ctx = { false, false };
>> struct ttm_resource evict_mem;
>> - struct ttm_place place, hop;
>> -
>> - memset(&place, 0, sizeof(place));
>> - memset(&hop, 0, sizeof(hop));
>> -
>> - place.mem_type = TTM_PL_SYSTEM;
>> + struct ttm_place hop = {};
>
> I would stick with memset because of the padding reasons.
>
>> ret = ttm_resource_alloc(bo, &place, &evict_mem);
>> if (unlikely(ret))
>> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
>> index 913b330a234b..d9793cbb6d13 100644
>> --- a/drivers/gpu/drm/ttm/ttm_tt.c
>> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
>> @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev,
>> struct ttm_tt *ttm,
>> struct page *to_page;
>> int i, ret;
>> + if (!ttm_tt_is_populated(ttm))
>> + return 0;
>> +
>
> This here is just because of a bug in the higher level function.
>
> I've just pushed the fix for that to drm-misc-fixes, so maybe drop
> that here as soon as this is backmerged.
>
That code doesn't look correct to me. In ttm_device_swapout only the lru
lock is held, and the bo->ttm pointer is protected by the resv lock,
meaning that bo->ttm can disappear at any time in that function, so
while an advisory reading bo->ttm using READ_ONCE() is ok, dereferencing
the bo->ttm pointer without reservation held is illegal and may send you
into recently freed memory.
For an example, consider
thread A. Selects bo for eviction, moves to system lru, creates ttm
Thread B locks lru in swapout code. finds bo->ttm NON_NULL,
thread A tries to evict bo, fails, destroys the ttm.
Thread B derefs freed memory.
But even relying on that there were no such example in the ttm core
itself, not adhering to the protection of bo->ttm makes the code
extremely fragile and IMHO needs fixing.
Also as a secondary note, a driver is in principle free to do things in
the swap notifier that may result in an unpopulated ttm so IMHO a late
check is needed here.
So ack to keep the above?
Thanks,
Thomas
>
> Christian.
>
>
>> swap_storage = shmem_file_setup("ttm swap", size, 0);
>> if (IS_ERR(swap_storage)) {
>> pr_err("Failed allocating swap storage\n");
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
@ 2021-05-27 7:33 ` Thomas Hellström (Intel)
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström (Intel) @ 2021-05-27 7:33 UTC (permalink / raw)
To: Christian König, Thomas Hellström, intel-gfx, dri-devel
Hi, Christian,
Thanks for reviewing.
On 5/26/21 3:26 PM, Christian König wrote:
> Am 26.05.21 um 13:32 schrieb Thomas Hellström:
>> We are calling the eviction_valuable driver callback at eviction time to
>> determine whether we actually can evict a buffer object.
>> The upcoming i915 TTM backend needs the same functionality for swapout,
>> and that might actually be beneficial to other drivers as well.
>>
>> Add an eviction_valuable call also in the swapout path. Try to keep the
>> current behaviour for all drivers by returning true if the buffer object
>> is already in the TTM_PL_SYSTEM placement. We change behaviour for the
>> case where a buffer object is in a TT backed placement when swapped out,
>> in which case the drivers normal eviction_valuable path is run.
>>
>> Finally make sure we don't try to swapout a bo that was recently purged
>> and therefore unpopulated.
>>
>> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>> Cc: Christian König <christian.koenig@amd.com>
>> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>> ---
>> v3:
>> - Don't export ttm_tt_unpopulate
>> - Fix confusion reading the locked pointer instead of the value
>> pointed to in ttm_bo_evict_swapout_allowable (Reported by
>> Maarten Lankhorst)
>> ---
>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
>> drivers/gpu/drm/ttm/ttm_bo.c | 43 ++++++++++++++++---------
>> drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
>> 3 files changed, 34 insertions(+), 16 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> index 3bc3aebfef7c..45d194bffc3f 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> @@ -1348,6 +1348,10 @@ static bool
>> amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
>> struct dma_fence *f;
>> int i;
>> + /* Swapout? */
>> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
>> + return true;
>> +
>> if (bo->type == ttm_bo_type_kernel &&
>> !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
>> return false;
>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
>> index be0406466460..1b2d062266ed 100644
>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>> @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct ttm_buffer_object
>> *bo,
>> bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
>> const struct ttm_place *place)
>> {
>> + dma_resv_assert_held(bo->base.resv);
>> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
>> + return true;
>> +
>> /* Don't evict this BO if it's outside of the
>> * requested placement range
>> */
>> @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
>> * b. Otherwise, trylock it.
>> */
>> static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object
>> *bo,
>> - struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
>> + struct ttm_operation_ctx *ctx,
>> + const struct ttm_place *place,
>> + bool *locked, bool *busy)
>> {
>> bool ret = false;
>> @@ -576,6 +582,14 @@ static bool
>> ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
>> *busy = !ret;
>> }
>> + if (ret && place && !bo->bdev->funcs->eviction_valuable(bo,
>> place)) {
>> + ret = false;
>> + if (*locked) {
>> + dma_resv_unlock(bo->base.resv);
>> + *locked = false;
>> + }
>> + }
>> +
>> return ret;
>> }
>> @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
>> list_for_each_entry(bo, &man->lru[i], lru) {
>> bool busy;
>> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
>> - &busy)) {
>> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
>> + &locked, &busy)) {
>> if (busy && !busy_bo && ticket !=
>> dma_resv_locking_ctx(bo->base.resv))
>> busy_bo = bo;
>> continue;
>> }
>> - if (place && !bdev->funcs->eviction_valuable(bo,
>> - place)) {
>> - if (locked)
>> - dma_resv_unlock(bo->base.resv);
>> - continue;
>> - }
>> if (!ttm_bo_get_unless_zero(bo)) {
>> if (locked)
>> dma_resv_unlock(bo->base.resv);
>> @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
>> int ttm_bo_swapout(struct ttm_buffer_object *bo, struct
>> ttm_operation_ctx *ctx,
>> gfp_t gfp_flags)
>> {
>> + struct ttm_place place = {};
>> bool locked;
>> int ret;
>> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
>> + /*
>> + * While the bo may already reside in SYSTEM placement, set
>> + * SYSTEM as new placement to cover also the move further below.
>> + * The driver may use the fact that we're moving from SYSTEM
>> + * as an indication that we're about to swap out.
>> + */
>> + place.mem_type = TTM_PL_SYSTEM;
>> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked,
>> NULL))
>> return -EBUSY;
>> if (!ttm_bo_get_unless_zero(bo)) {
>> @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct ttm_buffer_object
>> *bo, struct ttm_operation_ctx *ctx,
>> if (bo->mem.mem_type != TTM_PL_SYSTEM) {
>> struct ttm_operation_ctx ctx = { false, false };
>> struct ttm_resource evict_mem;
>> - struct ttm_place place, hop;
>> -
>> - memset(&place, 0, sizeof(place));
>> - memset(&hop, 0, sizeof(hop));
>> -
>> - place.mem_type = TTM_PL_SYSTEM;
>> + struct ttm_place hop = {};
>
> I would stick with memset because of the padding reasons.
>
>> ret = ttm_resource_alloc(bo, &place, &evict_mem);
>> if (unlikely(ret))
>> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
>> index 913b330a234b..d9793cbb6d13 100644
>> --- a/drivers/gpu/drm/ttm/ttm_tt.c
>> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
>> @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev,
>> struct ttm_tt *ttm,
>> struct page *to_page;
>> int i, ret;
>> + if (!ttm_tt_is_populated(ttm))
>> + return 0;
>> +
>
> This here is just because of a bug in the higher level function.
>
> I've just pushed the fix for that to drm-misc-fixes, so maybe drop
> that here as soon as this is backmerged.
>
That code doesn't look correct to me. In ttm_device_swapout only the lru
lock is held, and the bo->ttm pointer is protected by the resv lock,
meaning that bo->ttm can disappear at any time in that function, so
while an advisory reading bo->ttm using READ_ONCE() is ok, dereferencing
the bo->ttm pointer without reservation held is illegal and may send you
into recently freed memory.
For an example, consider
thread A. Selects bo for eviction, moves to system lru, creates ttm
Thread B locks lru in swapout code. finds bo->ttm NON_NULL,
thread A tries to evict bo, fails, destroys the ttm.
Thread B derefs freed memory.
But even relying on that there were no such example in the ttm core
itself, not adhering to the protection of bo->ttm makes the code
extremely fragile and IMHO needs fixing.
Also as a secondary note, a driver is in principle free to do things in
the swap notifier that may result in an unpopulated ttm so IMHO a late
check is needed here.
So ack to keep the above?
Thanks,
Thomas
>
> Christian.
>
>
>> swap_storage = shmem_file_setup("ttm swap", size, 0);
>> if (IS_ERR(swap_storage)) {
>> pr_err("Failed allocating swap storage\n");
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 14/15] drm/vma: Add a driver_private member to vma_node.
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-27 8:16 ` Thomas Hellström (Intel)
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström (Intel) @ 2021-05-27 8:16 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel, Maarten Lankhorst
Hi Maarten,
On 5/26/21 1:32 PM, Thomas Hellström wrote:
> From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>
> This allows drivers to distinguish between different types of vma_node's.
> The readonly flag was unused and is thus removed.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
This one was acked by danvet on IRC, but he wanted a more verbose commit
message stating that it would be temporary until all mmap-capable gem
obj backends could be moved over to TTM for discrete.
/Thomas
> ---
> drivers/gpu/drm/drm_gem.c | 9 ---------
> include/drm/drm_vma_manager.h | 2 +-
> 2 files changed, 1 insertion(+), 10 deletions(-)
>
> diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
> index 9989425e9875..e710e79069f6 100644
> --- a/drivers/gpu/drm/drm_gem.c
> +++ b/drivers/gpu/drm/drm_gem.c
> @@ -1149,15 +1149,6 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)
> return -EACCES;
> }
>
> - if (node->readonly) {
> - if (vma->vm_flags & VM_WRITE) {
> - drm_gem_object_put(obj);
> - return -EINVAL;
> - }
> -
> - vma->vm_flags &= ~VM_MAYWRITE;
> - }
> -
> ret = drm_gem_mmap_obj(obj, drm_vma_node_size(node) << PAGE_SHIFT,
> vma);
>
> diff --git a/include/drm/drm_vma_manager.h b/include/drm/drm_vma_manager.h
> index 76ac5e97a559..4f8c35206f7c 100644
> --- a/include/drm/drm_vma_manager.h
> +++ b/include/drm/drm_vma_manager.h
> @@ -53,7 +53,7 @@ struct drm_vma_offset_node {
> rwlock_t vm_lock;
> struct drm_mm_node vm_node;
> struct rb_root vm_files;
> - bool readonly:1;
> + void *driver_private;
> };
>
> struct drm_vma_offset_manager {
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 14/15] drm/vma: Add a driver_private member to vma_node.
@ 2021-05-27 8:16 ` Thomas Hellström (Intel)
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström (Intel) @ 2021-05-27 8:16 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel, Maarten Lankhorst
Hi Maarten,
On 5/26/21 1:32 PM, Thomas Hellström wrote:
> From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>
> This allows drivers to distinguish between different types of vma_node's.
> The readonly flag was unused and is thus removed.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
This one was acked by danvet on IRC, but he wanted a more verbose commit
message stating that it would be temporary until all mmap-capable gem
obj backends could be moved over to TTM for discrete.
/Thomas
> ---
> drivers/gpu/drm/drm_gem.c | 9 ---------
> include/drm/drm_vma_manager.h | 2 +-
> 2 files changed, 1 insertion(+), 10 deletions(-)
>
> diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
> index 9989425e9875..e710e79069f6 100644
> --- a/drivers/gpu/drm/drm_gem.c
> +++ b/drivers/gpu/drm/drm_gem.c
> @@ -1149,15 +1149,6 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)
> return -EACCES;
> }
>
> - if (node->readonly) {
> - if (vma->vm_flags & VM_WRITE) {
> - drm_gem_object_put(obj);
> - return -EINVAL;
> - }
> -
> - vma->vm_flags &= ~VM_MAYWRITE;
> - }
> -
> ret = drm_gem_mmap_obj(obj, drm_vma_node_size(node) << PAGE_SHIFT,
> vma);
>
> diff --git a/include/drm/drm_vma_manager.h b/include/drm/drm_vma_manager.h
> index 76ac5e97a559..4f8c35206f7c 100644
> --- a/include/drm/drm_vma_manager.h
> +++ b/include/drm/drm_vma_manager.h
> @@ -53,7 +53,7 @@ struct drm_vma_offset_node {
> rwlock_t vm_lock;
> struct drm_mm_node vm_node;
> struct rb_root vm_files;
> - bool readonly:1;
> + void *driver_private;
> };
>
> struct drm_vma_offset_manager {
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 15/15] drm/i915: Use ttm mmap handling for ttm bo's.
2021-05-26 17:40 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-27 11:11 ` Maarten Lankhorst
-1 siblings, 0 replies; 58+ messages in thread
From: Maarten Lankhorst @ 2021-05-27 11:11 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel
Op 2021-05-26 om 19:40 schreef Thomas Hellström:
>
> On 5/26/21 1:32 PM, Thomas Hellström wrote:
>> From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>>
>> Use the ttm handlers for servicing page faults, and vm_access.
>>
>> We do our own validation of read-only access, otherwise use the
>> ttm handlers as much as possible.
>>
>> Because the ttm handlers expect the vma_node at vma->base, we slightly
>> need to massage the mmap handlers to look at vma_node->driver_private
>> to fetch the bo, if it's NULL, we assume i915's normal mmap_offset uapi
>> is used.
>>
>> This is the easiest way to achieve compatibility without changing ttm's
>> semantics.
>>
>> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>> ---
>> drivers/gpu/drm/i915/gem/i915_gem_mman.c | 78 +++++++----
>> drivers/gpu/drm/i915/gem/i915_gem_object.h | 6 +-
>> .../gpu/drm/i915/gem/i915_gem_object_types.h | 3 +
>> drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
>> drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 122 +++++++++++++++++-
>> .../drm/i915/gem/selftests/i915_gem_mman.c | 90 +++++++------
>> drivers/gpu/drm/i915/selftests/igt_mmap.c | 25 +++-
>> drivers/gpu/drm/i915/selftests/igt_mmap.h | 12 +-
>> 8 files changed, 247 insertions(+), 92 deletions(-)
>
> There are a couple of checkpatch.pl --strict warnings/checks with this patch.
>
>
>>
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>> index fd1c9714f8d8..af04ea593091 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>> @@ -19,6 +19,7 @@
>> #include "i915_gem_mman.h"
>> #include "i915_trace.h"
>> #include "i915_user_extensions.h"
>> +#include "i915_gem_ttm.h"
>> #include "i915_vma.h"
>> static inline bool
>> @@ -622,6 +623,8 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
>> struct i915_mmap_offset *mmo;
>> int err;
>> + GEM_BUG_ON(obj->ops->mmap_offset || obj->ops->mmap_ops);
>> +
>> mmo = lookup_mmo(obj, mmap_type);
>> if (mmo)
>> goto out;
>> @@ -664,40 +667,47 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
>> }
>> static int
>> -__assign_mmap_offset(struct drm_file *file,
>> - u32 handle,
>> +__assign_mmap_offset(struct drm_i915_gem_object *obj,
>> enum i915_mmap_type mmap_type,
>> - u64 *offset)
>> + u64 *offset, struct drm_file *file)
>> {
>> - struct drm_i915_gem_object *obj;
>> struct i915_mmap_offset *mmo;
>> - int err;
>> - obj = i915_gem_object_lookup(file, handle);
>> - if (!obj)
>> - return -ENOENT;
>> + if (i915_gem_object_never_mmap(obj))
>> + return -ENODEV;
>> - if (i915_gem_object_never_mmap(obj)) {
>> - err = -ENODEV;
>> - goto out;
>> + if (obj->ops->mmap_offset) {
>> + *offset = obj->ops->mmap_offset(obj);
>> + return 0;
>> }
>> if (mmap_type != I915_MMAP_TYPE_GTT &&
>> !i915_gem_object_has_struct_page(obj) &&
>> - !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM)) {
>> - err = -ENODEV;
>> - goto out;
>> - }
>> + !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM))
>> + return -ENODEV;
>> mmo = mmap_offset_attach(obj, mmap_type, file);
>> - if (IS_ERR(mmo)) {
>> - err = PTR_ERR(mmo);
>> - goto out;
>> - }
>> + if (IS_ERR(mmo))
>> + return PTR_ERR(mmo);
>> *offset = drm_vma_node_offset_addr(&mmo->vma_node);
>> - err = 0;
>> -out:
>> + return 0;
>> +}
>> +
>> +static int
>> +__assign_mmap_offset_handle(struct drm_file *file,
>> + u32 handle,
>> + enum i915_mmap_type mmap_type,
>> + u64 *offset)
>> +{
>> + struct drm_i915_gem_object *obj;
>> + int err;
>> +
>> + obj = i915_gem_object_lookup(file, handle);
>> + if (!obj)
>> + return -ENOENT;
>> +
>> + err = __assign_mmap_offset(obj, mmap_type, offset, file);
>> i915_gem_object_put(obj);
>> return err;
>> }
>> @@ -717,7 +727,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>> else
>> mmap_type = I915_MMAP_TYPE_GTT;
>> - return __assign_mmap_offset(file, handle, mmap_type, offset);
>> + return __assign_mmap_offset_handle(file, handle, mmap_type, offset);
>> }
>> /**
>> @@ -785,7 +795,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>> return -EINVAL;
>> }
>> - return __assign_mmap_offset(file, args->handle, type, &args->offset);
>> + return __assign_mmap_offset_handle(file, args->handle, type, &args->offset);
>> }
>> static void vm_open(struct vm_area_struct *vma)
>> @@ -889,8 +899,16 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
>> * destroyed and will be invalid when the vma manager lock
>> * is released.
>> */
>> - mmo = container_of(node, struct i915_mmap_offset, vma_node);
>> - obj = i915_gem_object_get_rcu(mmo->obj);
>> + if (!node->driver_private) {
>> + mmo = container_of(node, struct i915_mmap_offset, vma_node);
>> + obj = i915_gem_object_get_rcu(mmo->obj);
>> +
>> + GEM_BUG_ON(obj && obj->ops->mmap_ops);
>> + } else {
>> + obj = i915_gem_object_get_rcu(container_of(node, struct drm_i915_gem_object, base.vma_node));
>> +
>> + GEM_BUG_ON(obj && !obj->ops->mmap_ops);
>> + }
>> }
>> drm_vma_offset_unlock_lookup(dev->vma_offset_manager);
>> rcu_read_unlock();
>> @@ -912,7 +930,6 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
>> }
>> vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP;
>> - vma->vm_private_data = mmo;
>> /*
>> * We keep the ref on mmo->obj, not vm_file, but we require
>> @@ -926,6 +943,15 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
>> /* Drop the initial creation reference, the vma is now holding one. */
>> fput(anon);
>> + if (obj->ops->mmap_ops) {
>> + vma->vm_page_prot = pgprot_decrypted(vm_get_page_prot(vma->vm_flags));
>> + vma->vm_ops = obj->ops->mmap_ops;
>> + vma->vm_private_data = node->driver_private;
>> + return 0;
>> + }
>> +
>> + vma->vm_private_data = mmo;
>> +
>> switch (mmo->mmap_type) {
>> case I915_MMAP_TYPE_WC:
>> vma->vm_page_prot =
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
>> index a3ad8cf4eefd..ff59e6c640e6 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
>> @@ -342,14 +342,14 @@ struct scatterlist *
>> __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
>> struct i915_gem_object_page_iter *iter,
>> unsigned int n,
>> - unsigned int *offset, bool allow_alloc);
>> + unsigned int *offset, bool allow_alloc, bool dma);
>> static inline struct scatterlist *
>> i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
>> unsigned int n,
>> unsigned int *offset, bool allow_alloc)
>> {
>> - return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc);
>> + return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc, false);
>> }
>> static inline struct scatterlist *
>> @@ -357,7 +357,7 @@ i915_gem_object_get_sg_dma(struct drm_i915_gem_object *obj,
>> unsigned int n,
>> unsigned int *offset, bool allow_alloc)
>> {
>> - return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc);
>> + return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc, true);
>> }
>> struct page *
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
>> index 68313474e6a6..2a23b77424b3 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
>> @@ -61,6 +61,7 @@ struct drm_i915_gem_object_ops {
>> const struct drm_i915_gem_pread *arg);
>> int (*pwrite)(struct drm_i915_gem_object *obj,
>> const struct drm_i915_gem_pwrite *arg);
>> + u64 (*mmap_offset)(struct drm_i915_gem_object *obj);
>> int (*dmabuf_export)(struct drm_i915_gem_object *obj);
>> @@ -79,6 +80,7 @@ struct drm_i915_gem_object_ops {
>> void (*delayed_free)(struct drm_i915_gem_object *obj);
>> void (*release)(struct drm_i915_gem_object *obj);
>> + const struct vm_operations_struct *mmap_ops;
>> const char *name; /* friendly name for debug, e.g. lockdep classes */
>> };
>> @@ -328,6 +330,7 @@ struct drm_i915_gem_object {
>> struct {
>> struct sg_table *cached_io_st;
>> + struct i915_gem_object_page_iter get_io_page;
>> bool created:1;
>> } ttm;
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
>> index 6444e097016d..086005c1c7ea 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
>> @@ -467,9 +467,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
>> struct i915_gem_object_page_iter *iter,
>> unsigned int n,
>> unsigned int *offset,
>> - bool allow_alloc)
>> + bool allow_alloc, bool dma)
>> {
>> - const bool dma = iter == &obj->mm.get_dma_page;
>> struct scatterlist *sg;
>> unsigned int idx, count;
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
>> index 17598930a99e..d0be957326e0 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
>> @@ -13,6 +13,7 @@
>> #include "gem/i915_gem_object.h"
>> #include "gem/i915_gem_region.h"
>> #include "gem/i915_gem_ttm.h"
>> +#include "gem/i915_gem_mman.h"
>> #define I915_PL_LMEM0 TTM_PL_PRIV
>> #define I915_PL_SYSTEM TTM_PL_SYSTEM
>> @@ -158,11 +159,20 @@ static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
>> static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
>> {
>> - if (obj->ttm.cached_io_st) {
>> - sg_free_table(obj->ttm.cached_io_st);
>> - kfree(obj->ttm.cached_io_st);
>> - obj->ttm.cached_io_st = NULL;
>> - }
>> + struct radix_tree_iter iter;
>> + void __rcu **slot;
>> +
>> + if (!obj->ttm.cached_io_st)
>> + return;
>> +
>> + rcu_read_lock();
>> + radix_tree_for_each_slot(slot, &obj->ttm.get_io_page.radix, &iter, 0)
>> + radix_tree_delete(&obj->ttm.get_io_page.radix, iter.index);
>> + rcu_read_unlock();
>> +
>> + sg_free_table(obj->ttm.cached_io_st);
>> + kfree(obj->ttm.cached_io_st);
>> + obj->ttm.cached_io_st = NULL;
>> }
>> static void i915_ttm_purge(struct drm_i915_gem_object *obj)
>> @@ -338,12 +348,42 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
>> ttm_bo_move_sync_cleanup(bo, dst_mem);
>> i915_ttm_free_cached_io_st(obj);
>> - if (!dst_man->use_tt)
>> + if (!dst_man->use_tt) {
>> obj->ttm.cached_io_st = dst_st;
>> + obj->ttm.get_io_page.sg_pos = dst_st->sgl;
>> + obj->ttm.get_io_page.sg_idx = 0;
>> + }
>> return 0;
>> }
>> +static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource *mem)
>> +{
>> + if (mem->mem_type < I915_PL_LMEM0)
>> + return 0;
>> +
>> + /* We may need to revisit this later, but this allows all caching to be used in mmap */
>> + mem->bus.caching = ttm_cached;
>
> Since we're now using the TTM bo offsets, we might as well just make this ttm_write_combined now.
Correct. That would be the correct value. TTM will use the correct caching that way.
~Maarten
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 15/15] drm/i915: Use ttm mmap handling for ttm bo's.
@ 2021-05-27 11:11 ` Maarten Lankhorst
0 siblings, 0 replies; 58+ messages in thread
From: Maarten Lankhorst @ 2021-05-27 11:11 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel
Op 2021-05-26 om 19:40 schreef Thomas Hellström:
>
> On 5/26/21 1:32 PM, Thomas Hellström wrote:
>> From: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>>
>> Use the ttm handlers for servicing page faults, and vm_access.
>>
>> We do our own validation of read-only access, otherwise use the
>> ttm handlers as much as possible.
>>
>> Because the ttm handlers expect the vma_node at vma->base, we slightly
>> need to massage the mmap handlers to look at vma_node->driver_private
>> to fetch the bo, if it's NULL, we assume i915's normal mmap_offset uapi
>> is used.
>>
>> This is the easiest way to achieve compatibility without changing ttm's
>> semantics.
>>
>> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>> ---
>> drivers/gpu/drm/i915/gem/i915_gem_mman.c | 78 +++++++----
>> drivers/gpu/drm/i915/gem/i915_gem_object.h | 6 +-
>> .../gpu/drm/i915/gem/i915_gem_object_types.h | 3 +
>> drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
>> drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 122 +++++++++++++++++-
>> .../drm/i915/gem/selftests/i915_gem_mman.c | 90 +++++++------
>> drivers/gpu/drm/i915/selftests/igt_mmap.c | 25 +++-
>> drivers/gpu/drm/i915/selftests/igt_mmap.h | 12 +-
>> 8 files changed, 247 insertions(+), 92 deletions(-)
>
> There are a couple of checkpatch.pl --strict warnings/checks with this patch.
>
>
>>
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>> index fd1c9714f8d8..af04ea593091 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>> @@ -19,6 +19,7 @@
>> #include "i915_gem_mman.h"
>> #include "i915_trace.h"
>> #include "i915_user_extensions.h"
>> +#include "i915_gem_ttm.h"
>> #include "i915_vma.h"
>> static inline bool
>> @@ -622,6 +623,8 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
>> struct i915_mmap_offset *mmo;
>> int err;
>> + GEM_BUG_ON(obj->ops->mmap_offset || obj->ops->mmap_ops);
>> +
>> mmo = lookup_mmo(obj, mmap_type);
>> if (mmo)
>> goto out;
>> @@ -664,40 +667,47 @@ mmap_offset_attach(struct drm_i915_gem_object *obj,
>> }
>> static int
>> -__assign_mmap_offset(struct drm_file *file,
>> - u32 handle,
>> +__assign_mmap_offset(struct drm_i915_gem_object *obj,
>> enum i915_mmap_type mmap_type,
>> - u64 *offset)
>> + u64 *offset, struct drm_file *file)
>> {
>> - struct drm_i915_gem_object *obj;
>> struct i915_mmap_offset *mmo;
>> - int err;
>> - obj = i915_gem_object_lookup(file, handle);
>> - if (!obj)
>> - return -ENOENT;
>> + if (i915_gem_object_never_mmap(obj))
>> + return -ENODEV;
>> - if (i915_gem_object_never_mmap(obj)) {
>> - err = -ENODEV;
>> - goto out;
>> + if (obj->ops->mmap_offset) {
>> + *offset = obj->ops->mmap_offset(obj);
>> + return 0;
>> }
>> if (mmap_type != I915_MMAP_TYPE_GTT &&
>> !i915_gem_object_has_struct_page(obj) &&
>> - !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM)) {
>> - err = -ENODEV;
>> - goto out;
>> - }
>> + !i915_gem_object_type_has(obj, I915_GEM_OBJECT_HAS_IOMEM))
>> + return -ENODEV;
>> mmo = mmap_offset_attach(obj, mmap_type, file);
>> - if (IS_ERR(mmo)) {
>> - err = PTR_ERR(mmo);
>> - goto out;
>> - }
>> + if (IS_ERR(mmo))
>> + return PTR_ERR(mmo);
>> *offset = drm_vma_node_offset_addr(&mmo->vma_node);
>> - err = 0;
>> -out:
>> + return 0;
>> +}
>> +
>> +static int
>> +__assign_mmap_offset_handle(struct drm_file *file,
>> + u32 handle,
>> + enum i915_mmap_type mmap_type,
>> + u64 *offset)
>> +{
>> + struct drm_i915_gem_object *obj;
>> + int err;
>> +
>> + obj = i915_gem_object_lookup(file, handle);
>> + if (!obj)
>> + return -ENOENT;
>> +
>> + err = __assign_mmap_offset(obj, mmap_type, offset, file);
>> i915_gem_object_put(obj);
>> return err;
>> }
>> @@ -717,7 +727,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>> else
>> mmap_type = I915_MMAP_TYPE_GTT;
>> - return __assign_mmap_offset(file, handle, mmap_type, offset);
>> + return __assign_mmap_offset_handle(file, handle, mmap_type, offset);
>> }
>> /**
>> @@ -785,7 +795,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>> return -EINVAL;
>> }
>> - return __assign_mmap_offset(file, args->handle, type, &args->offset);
>> + return __assign_mmap_offset_handle(file, args->handle, type, &args->offset);
>> }
>> static void vm_open(struct vm_area_struct *vma)
>> @@ -889,8 +899,16 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
>> * destroyed and will be invalid when the vma manager lock
>> * is released.
>> */
>> - mmo = container_of(node, struct i915_mmap_offset, vma_node);
>> - obj = i915_gem_object_get_rcu(mmo->obj);
>> + if (!node->driver_private) {
>> + mmo = container_of(node, struct i915_mmap_offset, vma_node);
>> + obj = i915_gem_object_get_rcu(mmo->obj);
>> +
>> + GEM_BUG_ON(obj && obj->ops->mmap_ops);
>> + } else {
>> + obj = i915_gem_object_get_rcu(container_of(node, struct drm_i915_gem_object, base.vma_node));
>> +
>> + GEM_BUG_ON(obj && !obj->ops->mmap_ops);
>> + }
>> }
>> drm_vma_offset_unlock_lookup(dev->vma_offset_manager);
>> rcu_read_unlock();
>> @@ -912,7 +930,6 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
>> }
>> vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP;
>> - vma->vm_private_data = mmo;
>> /*
>> * We keep the ref on mmo->obj, not vm_file, but we require
>> @@ -926,6 +943,15 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma)
>> /* Drop the initial creation reference, the vma is now holding one. */
>> fput(anon);
>> + if (obj->ops->mmap_ops) {
>> + vma->vm_page_prot = pgprot_decrypted(vm_get_page_prot(vma->vm_flags));
>> + vma->vm_ops = obj->ops->mmap_ops;
>> + vma->vm_private_data = node->driver_private;
>> + return 0;
>> + }
>> +
>> + vma->vm_private_data = mmo;
>> +
>> switch (mmo->mmap_type) {
>> case I915_MMAP_TYPE_WC:
>> vma->vm_page_prot =
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
>> index a3ad8cf4eefd..ff59e6c640e6 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
>> @@ -342,14 +342,14 @@ struct scatterlist *
>> __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
>> struct i915_gem_object_page_iter *iter,
>> unsigned int n,
>> - unsigned int *offset, bool allow_alloc);
>> + unsigned int *offset, bool allow_alloc, bool dma);
>> static inline struct scatterlist *
>> i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
>> unsigned int n,
>> unsigned int *offset, bool allow_alloc)
>> {
>> - return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc);
>> + return __i915_gem_object_get_sg(obj, &obj->mm.get_page, n, offset, allow_alloc, false);
>> }
>> static inline struct scatterlist *
>> @@ -357,7 +357,7 @@ i915_gem_object_get_sg_dma(struct drm_i915_gem_object *obj,
>> unsigned int n,
>> unsigned int *offset, bool allow_alloc)
>> {
>> - return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc);
>> + return __i915_gem_object_get_sg(obj, &obj->mm.get_dma_page, n, offset, allow_alloc, true);
>> }
>> struct page *
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
>> index 68313474e6a6..2a23b77424b3 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
>> @@ -61,6 +61,7 @@ struct drm_i915_gem_object_ops {
>> const struct drm_i915_gem_pread *arg);
>> int (*pwrite)(struct drm_i915_gem_object *obj,
>> const struct drm_i915_gem_pwrite *arg);
>> + u64 (*mmap_offset)(struct drm_i915_gem_object *obj);
>> int (*dmabuf_export)(struct drm_i915_gem_object *obj);
>> @@ -79,6 +80,7 @@ struct drm_i915_gem_object_ops {
>> void (*delayed_free)(struct drm_i915_gem_object *obj);
>> void (*release)(struct drm_i915_gem_object *obj);
>> + const struct vm_operations_struct *mmap_ops;
>> const char *name; /* friendly name for debug, e.g. lockdep classes */
>> };
>> @@ -328,6 +330,7 @@ struct drm_i915_gem_object {
>> struct {
>> struct sg_table *cached_io_st;
>> + struct i915_gem_object_page_iter get_io_page;
>> bool created:1;
>> } ttm;
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
>> index 6444e097016d..086005c1c7ea 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
>> @@ -467,9 +467,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
>> struct i915_gem_object_page_iter *iter,
>> unsigned int n,
>> unsigned int *offset,
>> - bool allow_alloc)
>> + bool allow_alloc, bool dma)
>> {
>> - const bool dma = iter == &obj->mm.get_dma_page;
>> struct scatterlist *sg;
>> unsigned int idx, count;
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
>> index 17598930a99e..d0be957326e0 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
>> @@ -13,6 +13,7 @@
>> #include "gem/i915_gem_object.h"
>> #include "gem/i915_gem_region.h"
>> #include "gem/i915_gem_ttm.h"
>> +#include "gem/i915_gem_mman.h"
>> #define I915_PL_LMEM0 TTM_PL_PRIV
>> #define I915_PL_SYSTEM TTM_PL_SYSTEM
>> @@ -158,11 +159,20 @@ static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
>> static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
>> {
>> - if (obj->ttm.cached_io_st) {
>> - sg_free_table(obj->ttm.cached_io_st);
>> - kfree(obj->ttm.cached_io_st);
>> - obj->ttm.cached_io_st = NULL;
>> - }
>> + struct radix_tree_iter iter;
>> + void __rcu **slot;
>> +
>> + if (!obj->ttm.cached_io_st)
>> + return;
>> +
>> + rcu_read_lock();
>> + radix_tree_for_each_slot(slot, &obj->ttm.get_io_page.radix, &iter, 0)
>> + radix_tree_delete(&obj->ttm.get_io_page.radix, iter.index);
>> + rcu_read_unlock();
>> +
>> + sg_free_table(obj->ttm.cached_io_st);
>> + kfree(obj->ttm.cached_io_st);
>> + obj->ttm.cached_io_st = NULL;
>> }
>> static void i915_ttm_purge(struct drm_i915_gem_object *obj)
>> @@ -338,12 +348,42 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
>> ttm_bo_move_sync_cleanup(bo, dst_mem);
>> i915_ttm_free_cached_io_st(obj);
>> - if (!dst_man->use_tt)
>> + if (!dst_man->use_tt) {
>> obj->ttm.cached_io_st = dst_st;
>> + obj->ttm.get_io_page.sg_pos = dst_st->sgl;
>> + obj->ttm.get_io_page.sg_idx = 0;
>> + }
>> return 0;
>> }
>> +static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource *mem)
>> +{
>> + if (mem->mem_type < I915_PL_LMEM0)
>> + return 0;
>> +
>> + /* We may need to revisit this later, but this allows all caching to be used in mmap */
>> + mem->bus.caching = ttm_cached;
>
> Since we're now using the TTM bo offsets, we might as well just make this ttm_write_combined now.
Correct. That would be the correct value. TTM will use the correct caching that way.
~Maarten
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
2021-05-27 7:33 ` [Intel-gfx] " Thomas Hellström (Intel)
@ 2021-05-27 12:36 ` Christian König
-1 siblings, 0 replies; 58+ messages in thread
From: Christian König @ 2021-05-27 12:36 UTC (permalink / raw)
To: Thomas Hellström (Intel),
Thomas Hellström, intel-gfx, dri-devel
Am 27.05.21 um 09:33 schrieb Thomas Hellström (Intel):
> Hi, Christian,
>
> Thanks for reviewing.
>
> On 5/26/21 3:26 PM, Christian König wrote:
>> Am 26.05.21 um 13:32 schrieb Thomas Hellström:
>>> We are calling the eviction_valuable driver callback at eviction
>>> time to
>>> determine whether we actually can evict a buffer object.
>>> The upcoming i915 TTM backend needs the same functionality for swapout,
>>> and that might actually be beneficial to other drivers as well.
>>>
>>> Add an eviction_valuable call also in the swapout path. Try to keep the
>>> current behaviour for all drivers by returning true if the buffer
>>> object
>>> is already in the TTM_PL_SYSTEM placement. We change behaviour for the
>>> case where a buffer object is in a TT backed placement when swapped
>>> out,
>>> in which case the drivers normal eviction_valuable path is run.
>>>
>>> Finally make sure we don't try to swapout a bo that was recently purged
>>> and therefore unpopulated.
>>>
>>> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>>> Cc: Christian König <christian.koenig@amd.com>
>>> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>>> ---
>>> v3:
>>> - Don't export ttm_tt_unpopulate
>>> - Fix confusion reading the locked pointer instead of the value
>>> pointed to in ttm_bo_evict_swapout_allowable (Reported by
>>> Maarten Lankhorst)
>>> ---
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
>>> drivers/gpu/drm/ttm/ttm_bo.c | 43
>>> ++++++++++++++++---------
>>> drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
>>> 3 files changed, 34 insertions(+), 16 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> index 3bc3aebfef7c..45d194bffc3f 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> @@ -1348,6 +1348,10 @@ static bool
>>> amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
>>> struct dma_fence *f;
>>> int i;
>>> + /* Swapout? */
>>> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
>>> + return true;
>>> +
>>> if (bo->type == ttm_bo_type_kernel &&
>>> !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
>>> return false;
>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
>>> b/drivers/gpu/drm/ttm/ttm_bo.c
>>> index be0406466460..1b2d062266ed 100644
>>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>>> @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct
>>> ttm_buffer_object *bo,
>>> bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
>>> const struct ttm_place *place)
>>> {
>>> + dma_resv_assert_held(bo->base.resv);
>>> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
>>> + return true;
>>> +
>>> /* Don't evict this BO if it's outside of the
>>> * requested placement range
>>> */
>>> @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
>>> * b. Otherwise, trylock it.
>>> */
>>> static bool ttm_bo_evict_swapout_allowable(struct
>>> ttm_buffer_object *bo,
>>> - struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
>>> + struct ttm_operation_ctx *ctx,
>>> + const struct ttm_place *place,
>>> + bool *locked, bool *busy)
>>> {
>>> bool ret = false;
>>> @@ -576,6 +582,14 @@ static bool
>>> ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
>>> *busy = !ret;
>>> }
>>> + if (ret && place && !bo->bdev->funcs->eviction_valuable(bo,
>>> place)) {
>>> + ret = false;
>>> + if (*locked) {
>>> + dma_resv_unlock(bo->base.resv);
>>> + *locked = false;
>>> + }
>>> + }
>>> +
>>> return ret;
>>> }
>>> @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device
>>> *bdev,
>>> list_for_each_entry(bo, &man->lru[i], lru) {
>>> bool busy;
>>> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
>>> - &busy)) {
>>> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
>>> + &locked, &busy)) {
>>> if (busy && !busy_bo && ticket !=
>>> dma_resv_locking_ctx(bo->base.resv))
>>> busy_bo = bo;
>>> continue;
>>> }
>>> - if (place && !bdev->funcs->eviction_valuable(bo,
>>> - place)) {
>>> - if (locked)
>>> - dma_resv_unlock(bo->base.resv);
>>> - continue;
>>> - }
>>> if (!ttm_bo_get_unless_zero(bo)) {
>>> if (locked)
>>> dma_resv_unlock(bo->base.resv);
>>> @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
>>> int ttm_bo_swapout(struct ttm_buffer_object *bo, struct
>>> ttm_operation_ctx *ctx,
>>> gfp_t gfp_flags)
>>> {
>>> + struct ttm_place place = {};
>>> bool locked;
>>> int ret;
>>> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
>>> + /*
>>> + * While the bo may already reside in SYSTEM placement, set
>>> + * SYSTEM as new placement to cover also the move further below.
>>> + * The driver may use the fact that we're moving from SYSTEM
>>> + * as an indication that we're about to swap out.
>>> + */
>>> + place.mem_type = TTM_PL_SYSTEM;
>>> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked,
>>> NULL))
>>> return -EBUSY;
>>> if (!ttm_bo_get_unless_zero(bo)) {
>>> @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct ttm_buffer_object
>>> *bo, struct ttm_operation_ctx *ctx,
>>> if (bo->mem.mem_type != TTM_PL_SYSTEM) {
>>> struct ttm_operation_ctx ctx = { false, false };
>>> struct ttm_resource evict_mem;
>>> - struct ttm_place place, hop;
>>> -
>>> - memset(&place, 0, sizeof(place));
>>> - memset(&hop, 0, sizeof(hop));
>>> -
>>> - place.mem_type = TTM_PL_SYSTEM;
>>> + struct ttm_place hop = {};
>>
>> I would stick with memset because of the padding reasons.
>>
>>> ret = ttm_resource_alloc(bo, &place, &evict_mem);
>>> if (unlikely(ret))
>>> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
>>> b/drivers/gpu/drm/ttm/ttm_tt.c
>>> index 913b330a234b..d9793cbb6d13 100644
>>> --- a/drivers/gpu/drm/ttm/ttm_tt.c
>>> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
>>> @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev,
>>> struct ttm_tt *ttm,
>>> struct page *to_page;
>>> int i, ret;
>>> + if (!ttm_tt_is_populated(ttm))
>>> + return 0;
>>> +
>>
>> This here is just because of a bug in the higher level function.
>>
>> I've just pushed the fix for that to drm-misc-fixes, so maybe drop
>> that here as soon as this is backmerged.
>>
> That code doesn't look correct to me. In ttm_device_swapout only the
> lru lock is held, and the bo->ttm pointer is protected by the resv
> lock, meaning that bo->ttm can disappear at any time in that function,
> so while an advisory reading bo->ttm using READ_ONCE() is ok,
> dereferencing the bo->ttm pointer without reservation held is illegal
> and may send you into recently freed memory.
>
> For an example, consider
>
> thread A. Selects bo for eviction, moves to system lru, creates ttm
> Thread B locks lru in swapout code. finds bo->ttm NON_NULL,
> thread A tries to evict bo, fails, destroys the ttm.
> Thread B derefs freed memory.
>
> But even relying on that there were no such example in the ttm core
> itself, not adhering to the protection of bo->ttm makes the code
> extremely fragile and IMHO needs fixing.
>
> Also as a secondary note, a driver is in principle free to do things
> in the swap notifier that may result in an unpopulated ttm so IMHO a
> late check is needed here.
>
> So ack to keep the above?
Oh, really good point. Haven't thought about that for the quick fix.
I think for the short term we need to protect TT destruction by the
spinlock.
The problem doing it here is that you end up in an endless loop currently.
E.g. you trylock and inspect the same BO over and over again.
Need to double check the code to see if that can somehow be avoided.
Christian.
>
> Thanks,
>
> Thomas
>
>
>>
>> Christian.
>>
>>
>>> swap_storage = shmem_file_setup("ttm swap", size, 0);
>>> if (IS_ERR(swap_storage)) {
>>> pr_err("Failed allocating swap storage\n");
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
@ 2021-05-27 12:36 ` Christian König
0 siblings, 0 replies; 58+ messages in thread
From: Christian König @ 2021-05-27 12:36 UTC (permalink / raw)
To: Thomas Hellström (Intel),
Thomas Hellström, intel-gfx, dri-devel
Am 27.05.21 um 09:33 schrieb Thomas Hellström (Intel):
> Hi, Christian,
>
> Thanks for reviewing.
>
> On 5/26/21 3:26 PM, Christian König wrote:
>> Am 26.05.21 um 13:32 schrieb Thomas Hellström:
>>> We are calling the eviction_valuable driver callback at eviction
>>> time to
>>> determine whether we actually can evict a buffer object.
>>> The upcoming i915 TTM backend needs the same functionality for swapout,
>>> and that might actually be beneficial to other drivers as well.
>>>
>>> Add an eviction_valuable call also in the swapout path. Try to keep the
>>> current behaviour for all drivers by returning true if the buffer
>>> object
>>> is already in the TTM_PL_SYSTEM placement. We change behaviour for the
>>> case where a buffer object is in a TT backed placement when swapped
>>> out,
>>> in which case the drivers normal eviction_valuable path is run.
>>>
>>> Finally make sure we don't try to swapout a bo that was recently purged
>>> and therefore unpopulated.
>>>
>>> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>>> Cc: Christian König <christian.koenig@amd.com>
>>> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>>> ---
>>> v3:
>>> - Don't export ttm_tt_unpopulate
>>> - Fix confusion reading the locked pointer instead of the value
>>> pointed to in ttm_bo_evict_swapout_allowable (Reported by
>>> Maarten Lankhorst)
>>> ---
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
>>> drivers/gpu/drm/ttm/ttm_bo.c | 43
>>> ++++++++++++++++---------
>>> drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
>>> 3 files changed, 34 insertions(+), 16 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> index 3bc3aebfef7c..45d194bffc3f 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> @@ -1348,6 +1348,10 @@ static bool
>>> amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
>>> struct dma_fence *f;
>>> int i;
>>> + /* Swapout? */
>>> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
>>> + return true;
>>> +
>>> if (bo->type == ttm_bo_type_kernel &&
>>> !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
>>> return false;
>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
>>> b/drivers/gpu/drm/ttm/ttm_bo.c
>>> index be0406466460..1b2d062266ed 100644
>>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>>> @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct
>>> ttm_buffer_object *bo,
>>> bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
>>> const struct ttm_place *place)
>>> {
>>> + dma_resv_assert_held(bo->base.resv);
>>> + if (bo->mem.mem_type == TTM_PL_SYSTEM)
>>> + return true;
>>> +
>>> /* Don't evict this BO if it's outside of the
>>> * requested placement range
>>> */
>>> @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
>>> * b. Otherwise, trylock it.
>>> */
>>> static bool ttm_bo_evict_swapout_allowable(struct
>>> ttm_buffer_object *bo,
>>> - struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
>>> + struct ttm_operation_ctx *ctx,
>>> + const struct ttm_place *place,
>>> + bool *locked, bool *busy)
>>> {
>>> bool ret = false;
>>> @@ -576,6 +582,14 @@ static bool
>>> ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
>>> *busy = !ret;
>>> }
>>> + if (ret && place && !bo->bdev->funcs->eviction_valuable(bo,
>>> place)) {
>>> + ret = false;
>>> + if (*locked) {
>>> + dma_resv_unlock(bo->base.resv);
>>> + *locked = false;
>>> + }
>>> + }
>>> +
>>> return ret;
>>> }
>>> @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device
>>> *bdev,
>>> list_for_each_entry(bo, &man->lru[i], lru) {
>>> bool busy;
>>> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
>>> - &busy)) {
>>> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
>>> + &locked, &busy)) {
>>> if (busy && !busy_bo && ticket !=
>>> dma_resv_locking_ctx(bo->base.resv))
>>> busy_bo = bo;
>>> continue;
>>> }
>>> - if (place && !bdev->funcs->eviction_valuable(bo,
>>> - place)) {
>>> - if (locked)
>>> - dma_resv_unlock(bo->base.resv);
>>> - continue;
>>> - }
>>> if (!ttm_bo_get_unless_zero(bo)) {
>>> if (locked)
>>> dma_resv_unlock(bo->base.resv);
>>> @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
>>> int ttm_bo_swapout(struct ttm_buffer_object *bo, struct
>>> ttm_operation_ctx *ctx,
>>> gfp_t gfp_flags)
>>> {
>>> + struct ttm_place place = {};
>>> bool locked;
>>> int ret;
>>> - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
>>> + /*
>>> + * While the bo may already reside in SYSTEM placement, set
>>> + * SYSTEM as new placement to cover also the move further below.
>>> + * The driver may use the fact that we're moving from SYSTEM
>>> + * as an indication that we're about to swap out.
>>> + */
>>> + place.mem_type = TTM_PL_SYSTEM;
>>> + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked,
>>> NULL))
>>> return -EBUSY;
>>> if (!ttm_bo_get_unless_zero(bo)) {
>>> @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct ttm_buffer_object
>>> *bo, struct ttm_operation_ctx *ctx,
>>> if (bo->mem.mem_type != TTM_PL_SYSTEM) {
>>> struct ttm_operation_ctx ctx = { false, false };
>>> struct ttm_resource evict_mem;
>>> - struct ttm_place place, hop;
>>> -
>>> - memset(&place, 0, sizeof(place));
>>> - memset(&hop, 0, sizeof(hop));
>>> -
>>> - place.mem_type = TTM_PL_SYSTEM;
>>> + struct ttm_place hop = {};
>>
>> I would stick with memset because of the padding reasons.
>>
>>> ret = ttm_resource_alloc(bo, &place, &evict_mem);
>>> if (unlikely(ret))
>>> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
>>> b/drivers/gpu/drm/ttm/ttm_tt.c
>>> index 913b330a234b..d9793cbb6d13 100644
>>> --- a/drivers/gpu/drm/ttm/ttm_tt.c
>>> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
>>> @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev,
>>> struct ttm_tt *ttm,
>>> struct page *to_page;
>>> int i, ret;
>>> + if (!ttm_tt_is_populated(ttm))
>>> + return 0;
>>> +
>>
>> This here is just because of a bug in the higher level function.
>>
>> I've just pushed the fix for that to drm-misc-fixes, so maybe drop
>> that here as soon as this is backmerged.
>>
> That code doesn't look correct to me. In ttm_device_swapout only the
> lru lock is held, and the bo->ttm pointer is protected by the resv
> lock, meaning that bo->ttm can disappear at any time in that function,
> so while an advisory reading bo->ttm using READ_ONCE() is ok,
> dereferencing the bo->ttm pointer without reservation held is illegal
> and may send you into recently freed memory.
>
> For an example, consider
>
> thread A. Selects bo for eviction, moves to system lru, creates ttm
> Thread B locks lru in swapout code. finds bo->ttm NON_NULL,
> thread A tries to evict bo, fails, destroys the ttm.
> Thread B derefs freed memory.
>
> But even relying on that there were no such example in the ttm core
> itself, not adhering to the protection of bo->ttm makes the code
> extremely fragile and IMHO needs fixing.
>
> Also as a secondary note, a driver is in principle free to do things
> in the swap notifier that may result in an unpopulated ttm so IMHO a
> late check is needed here.
>
> So ack to keep the above?
Oh, really good point. Haven't thought about that for the quick fix.
I think for the short term we need to protect TT destruction by the
spinlock.
The problem doing it here is that you end up in an endless loop currently.
E.g. you trylock and inspect the same BO over and over again.
Need to double check the code to see if that can somehow be avoided.
Christian.
>
> Thanks,
>
> Thomas
>
>
>>
>> Christian.
>>
>>
>>> swap_storage = shmem_file_setup("ttm swap", size, 0);
>>> if (IS_ERR(swap_storage)) {
>>> pr_err("Failed allocating swap storage\n");
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
2021-05-27 12:36 ` [Intel-gfx] " Christian König
@ 2021-05-27 13:52 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-27 13:52 UTC (permalink / raw)
To: Christian König, Thomas Hellström (Intel),
intel-gfx, dri-devel
On Thu, 2021-05-27 at 14:36 +0200, Christian König wrote:
> Am 27.05.21 um 09:33 schrieb Thomas Hellström (Intel):
> > Hi, Christian,
> >
> > Thanks for reviewing.
> >
> > On 5/26/21 3:26 PM, Christian König wrote:
> > > Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> > > > We are calling the eviction_valuable driver callback at eviction
> > > > time to
> > > > determine whether we actually can evict a buffer object.
> > > > The upcoming i915 TTM backend needs the same functionality for
> > > > swapout,
> > > > and that might actually be beneficial to other drivers as well.
> > > >
> > > > Add an eviction_valuable call also in the swapout path. Try to
> > > > keep the
> > > > current behaviour for all drivers by returning true if the buffer
> > > > object
> > > > is already in the TTM_PL_SYSTEM placement. We change behaviour
> > > > for the
> > > > case where a buffer object is in a TT backed placement when
> > > > swapped
> > > > out,
> > > > in which case the drivers normal eviction_valuable path is run.
> > > >
> > > > Finally make sure we don't try to swapout a bo that was recently
> > > > purged
> > > > and therefore unpopulated.
> > > >
> > > > Reviewed-by: Maarten Lankhorst <
> > > > maarten.lankhorst@linux.intel.com>
> > > > Cc: Christian König <christian.koenig@amd.com>
> > > > Signed-off-by: Thomas Hellström <
> > > > thomas.hellstrom@linux.intel.com>
> > > > ---
> > > > v3:
> > > > - Don't export ttm_tt_unpopulate
> > > > - Fix confusion reading the locked pointer instead of the value
> > > > pointed to in ttm_bo_evict_swapout_allowable (Reported by
> > > > Maarten Lankhorst)
> > > > ---
> > > > drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
> > > > drivers/gpu/drm/ttm/ttm_bo.c | 43
> > > > ++++++++++++++++---------
> > > > drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
> > > > 3 files changed, 34 insertions(+), 16 deletions(-)
> > > >
> > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > index 3bc3aebfef7c..45d194bffc3f 100644
> > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > @@ -1348,6 +1348,10 @@ static bool
> > > > amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> > > > struct dma_fence *f;
> > > > int i;
> > > > + /* Swapout? */
> > > > + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> > > > + return true;
> > > > +
> > > > if (bo->type == ttm_bo_type_kernel &&
> > > > !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
> > > > return false;
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > index be0406466460..1b2d062266ed 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct
> > > > ttm_buffer_object *bo,
> > > > bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> > > > const struct ttm_place *place)
> > > > {
> > > > + dma_resv_assert_held(bo->base.resv);
> > > > + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> > > > + return true;
> > > > +
> > > > /* Don't evict this BO if it's outside of the
> > > > * requested placement range
> > > > */
> > > > @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
> > > > * b. Otherwise, trylock it.
> > > > */
> > > > static bool ttm_bo_evict_swapout_allowable(struct
> > > > ttm_buffer_object *bo,
> > > > - struct ttm_operation_ctx *ctx, bool *locked, bool
> > > > *busy)
> > > > + struct ttm_operation_ctx *ctx,
> > > > + const struct ttm_place *place,
> > > > + bool *locked, bool *busy)
> > > > {
> > > > bool ret = false;
> > > > @@ -576,6 +582,14 @@ static bool
> > > > ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
> > > > *busy = !ret;
> > > > }
> > > > + if (ret && place && !bo->bdev->funcs-
> > > > >eviction_valuable(bo,
> > > > place)) {
> > > > + ret = false;
> > > > + if (*locked) {
> > > > + dma_resv_unlock(bo->base.resv);
> > > > + *locked = false;
> > > > + }
> > > > + }
> > > > +
> > > > return ret;
> > > > }
> > > > @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device
> > > > *bdev,
> > > > list_for_each_entry(bo, &man->lru[i], lru) {
> > > > bool busy;
> > > > - if (!ttm_bo_evict_swapout_allowable(bo, ctx,
> > > > &locked,
> > > > - &busy)) {
> > > > + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
> > > > + &locked, &busy)) {
> > > > if (busy && !busy_bo && ticket !=
> > > > dma_resv_locking_ctx(bo->base.resv))
> > > > busy_bo = bo;
> > > > continue;
> > > > }
> > > > - if (place && !bdev->funcs->eviction_valuable(bo,
> > > > - place)) {
> > > > - if (locked)
> > > > - dma_resv_unlock(bo->base.resv);
> > > > - continue;
> > > > - }
> > > > if (!ttm_bo_get_unless_zero(bo)) {
> > > > if (locked)
> > > > dma_resv_unlock(bo->base.resv);
> > > > @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
> > > > int ttm_bo_swapout(struct ttm_buffer_object *bo, struct
> > > > ttm_operation_ctx *ctx,
> > > > gfp_t gfp_flags)
> > > > {
> > > > + struct ttm_place place = {};
> > > > bool locked;
> > > > int ret;
> > > > - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
> > > > NULL))
> > > > + /*
> > > > + * While the bo may already reside in SYSTEM placement, set
> > > > + * SYSTEM as new placement to cover also the move further
> > > > below.
> > > > + * The driver may use the fact that we're moving from SYSTEM
> > > > + * as an indication that we're about to swap out.
> > > > + */
> > > > + place.mem_type = TTM_PL_SYSTEM;
> > > > + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place,
> > > > &locked,
> > > > NULL))
> > > > return -EBUSY;
> > > > if (!ttm_bo_get_unless_zero(bo)) {
> > > > @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct
> > > > ttm_buffer_object
> > > > *bo, struct ttm_operation_ctx *ctx,
> > > > if (bo->mem.mem_type != TTM_PL_SYSTEM) {
> > > > struct ttm_operation_ctx ctx = { false, false };
> > > > struct ttm_resource evict_mem;
> > > > - struct ttm_place place, hop;
> > > > -
> > > > - memset(&place, 0, sizeof(place));
> > > > - memset(&hop, 0, sizeof(hop));
> > > > -
> > > > - place.mem_type = TTM_PL_SYSTEM;
> > > > + struct ttm_place hop = {};
> > >
> > > I would stick with memset because of the padding reasons.
> > >
> > > > ret = ttm_resource_alloc(bo, &place, &evict_mem);
> > > > if (unlikely(ret))
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > index 913b330a234b..d9793cbb6d13 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev,
> > > > struct ttm_tt *ttm,
> > > > struct page *to_page;
> > > > int i, ret;
> > > > + if (!ttm_tt_is_populated(ttm))
> > > > + return 0;
> > > > +
> > >
> > > This here is just because of a bug in the higher level function.
> > >
> > > I've just pushed the fix for that to drm-misc-fixes, so maybe drop
> > > that here as soon as this is backmerged.
> > >
> > That code doesn't look correct to me. In ttm_device_swapout only the
> > lru lock is held, and the bo->ttm pointer is protected by the resv
> > lock, meaning that bo->ttm can disappear at any time in that
> > function,
> > so while an advisory reading bo->ttm using READ_ONCE() is ok,
> > dereferencing the bo->ttm pointer without reservation held is illegal
> > and may send you into recently freed memory.
> >
> > For an example, consider
> >
> > thread A. Selects bo for eviction, moves to system lru, creates ttm
> > Thread B locks lru in swapout code. finds bo->ttm NON_NULL,
> > thread A tries to evict bo, fails, destroys the ttm.
> > Thread B derefs freed memory.
> >
> > But even relying on that there were no such example in the ttm core
> > itself, not adhering to the protection of bo->ttm makes the code
> > extremely fragile and IMHO needs fixing.
> >
> > Also as a secondary note, a driver is in principle free to do things
> > in the swap notifier that may result in an unpopulated ttm so IMHO a
> > late check is needed here.
> >
> > So ack to keep the above?
>
> Oh, really good point. Haven't thought about that for the quick fix.
>
> I think for the short term we need to protect TT destruction by the
> spinlock.
To avoid more locking complexity,
I think it would be easy to just defer the code that derefs the ttm
(except the new unpopulated check) to ttm_bo_swapout() after we've
taken the resv trylock, but before taking the kref, returning -EBUSY if
conditions for swapping are not met.
>
> The problem doing it here is that you end up in an endless loop
> currently.
>
> E.g. you trylock and inspect the same BO over and over again.
>
> Need to double check the code to see if that can somehow be avoided.
Well ttm_tt_swapout happily returns 0, so the BO gets pulled off the
LRU anyway so I think that shouldn't happen.
The only thing that becomes incorrect is the num_pages return in
ttm_device_swapout(). OTOH, the caller shouldn't care whether we
actually swapped out or whether the bo losts its pages in swap_notify.
/Thomas
>
> Christian.
>
> >
> > Thanks,
> >
> > Thomas
> >
> >
> > >
> > > Christian.
> > >
> > >
> > > > swap_storage = shmem_file_setup("ttm swap", size, 0);
> > > > if (IS_ERR(swap_storage)) {
> > > > pr_err("Failed allocating swap storage\n");
>
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
@ 2021-05-27 13:52 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-27 13:52 UTC (permalink / raw)
To: Christian König, Thomas Hellström (Intel),
intel-gfx, dri-devel
On Thu, 2021-05-27 at 14:36 +0200, Christian König wrote:
> Am 27.05.21 um 09:33 schrieb Thomas Hellström (Intel):
> > Hi, Christian,
> >
> > Thanks for reviewing.
> >
> > On 5/26/21 3:26 PM, Christian König wrote:
> > > Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> > > > We are calling the eviction_valuable driver callback at eviction
> > > > time to
> > > > determine whether we actually can evict a buffer object.
> > > > The upcoming i915 TTM backend needs the same functionality for
> > > > swapout,
> > > > and that might actually be beneficial to other drivers as well.
> > > >
> > > > Add an eviction_valuable call also in the swapout path. Try to
> > > > keep the
> > > > current behaviour for all drivers by returning true if the buffer
> > > > object
> > > > is already in the TTM_PL_SYSTEM placement. We change behaviour
> > > > for the
> > > > case where a buffer object is in a TT backed placement when
> > > > swapped
> > > > out,
> > > > in which case the drivers normal eviction_valuable path is run.
> > > >
> > > > Finally make sure we don't try to swapout a bo that was recently
> > > > purged
> > > > and therefore unpopulated.
> > > >
> > > > Reviewed-by: Maarten Lankhorst <
> > > > maarten.lankhorst@linux.intel.com>
> > > > Cc: Christian König <christian.koenig@amd.com>
> > > > Signed-off-by: Thomas Hellström <
> > > > thomas.hellstrom@linux.intel.com>
> > > > ---
> > > > v3:
> > > > - Don't export ttm_tt_unpopulate
> > > > - Fix confusion reading the locked pointer instead of the value
> > > > pointed to in ttm_bo_evict_swapout_allowable (Reported by
> > > > Maarten Lankhorst)
> > > > ---
> > > > drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
> > > > drivers/gpu/drm/ttm/ttm_bo.c | 43
> > > > ++++++++++++++++---------
> > > > drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
> > > > 3 files changed, 34 insertions(+), 16 deletions(-)
> > > >
> > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > index 3bc3aebfef7c..45d194bffc3f 100644
> > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > @@ -1348,6 +1348,10 @@ static bool
> > > > amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> > > > struct dma_fence *f;
> > > > int i;
> > > > + /* Swapout? */
> > > > + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> > > > + return true;
> > > > +
> > > > if (bo->type == ttm_bo_type_kernel &&
> > > > !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
> > > > return false;
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > index be0406466460..1b2d062266ed 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct
> > > > ttm_buffer_object *bo,
> > > > bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> > > > const struct ttm_place *place)
> > > > {
> > > > + dma_resv_assert_held(bo->base.resv);
> > > > + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> > > > + return true;
> > > > +
> > > > /* Don't evict this BO if it's outside of the
> > > > * requested placement range
> > > > */
> > > > @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
> > > > * b. Otherwise, trylock it.
> > > > */
> > > > static bool ttm_bo_evict_swapout_allowable(struct
> > > > ttm_buffer_object *bo,
> > > > - struct ttm_operation_ctx *ctx, bool *locked, bool
> > > > *busy)
> > > > + struct ttm_operation_ctx *ctx,
> > > > + const struct ttm_place *place,
> > > > + bool *locked, bool *busy)
> > > > {
> > > > bool ret = false;
> > > > @@ -576,6 +582,14 @@ static bool
> > > > ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
> > > > *busy = !ret;
> > > > }
> > > > + if (ret && place && !bo->bdev->funcs-
> > > > >eviction_valuable(bo,
> > > > place)) {
> > > > + ret = false;
> > > > + if (*locked) {
> > > > + dma_resv_unlock(bo->base.resv);
> > > > + *locked = false;
> > > > + }
> > > > + }
> > > > +
> > > > return ret;
> > > > }
> > > > @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device
> > > > *bdev,
> > > > list_for_each_entry(bo, &man->lru[i], lru) {
> > > > bool busy;
> > > > - if (!ttm_bo_evict_swapout_allowable(bo, ctx,
> > > > &locked,
> > > > - &busy)) {
> > > > + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
> > > > + &locked, &busy)) {
> > > > if (busy && !busy_bo && ticket !=
> > > > dma_resv_locking_ctx(bo->base.resv))
> > > > busy_bo = bo;
> > > > continue;
> > > > }
> > > > - if (place && !bdev->funcs->eviction_valuable(bo,
> > > > - place)) {
> > > > - if (locked)
> > > > - dma_resv_unlock(bo->base.resv);
> > > > - continue;
> > > > - }
> > > > if (!ttm_bo_get_unless_zero(bo)) {
> > > > if (locked)
> > > > dma_resv_unlock(bo->base.resv);
> > > > @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
> > > > int ttm_bo_swapout(struct ttm_buffer_object *bo, struct
> > > > ttm_operation_ctx *ctx,
> > > > gfp_t gfp_flags)
> > > > {
> > > > + struct ttm_place place = {};
> > > > bool locked;
> > > > int ret;
> > > > - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
> > > > NULL))
> > > > + /*
> > > > + * While the bo may already reside in SYSTEM placement, set
> > > > + * SYSTEM as new placement to cover also the move further
> > > > below.
> > > > + * The driver may use the fact that we're moving from SYSTEM
> > > > + * as an indication that we're about to swap out.
> > > > + */
> > > > + place.mem_type = TTM_PL_SYSTEM;
> > > > + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place,
> > > > &locked,
> > > > NULL))
> > > > return -EBUSY;
> > > > if (!ttm_bo_get_unless_zero(bo)) {
> > > > @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct
> > > > ttm_buffer_object
> > > > *bo, struct ttm_operation_ctx *ctx,
> > > > if (bo->mem.mem_type != TTM_PL_SYSTEM) {
> > > > struct ttm_operation_ctx ctx = { false, false };
> > > > struct ttm_resource evict_mem;
> > > > - struct ttm_place place, hop;
> > > > -
> > > > - memset(&place, 0, sizeof(place));
> > > > - memset(&hop, 0, sizeof(hop));
> > > > -
> > > > - place.mem_type = TTM_PL_SYSTEM;
> > > > + struct ttm_place hop = {};
> > >
> > > I would stick with memset because of the padding reasons.
> > >
> > > > ret = ttm_resource_alloc(bo, &place, &evict_mem);
> > > > if (unlikely(ret))
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > index 913b330a234b..d9793cbb6d13 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev,
> > > > struct ttm_tt *ttm,
> > > > struct page *to_page;
> > > > int i, ret;
> > > > + if (!ttm_tt_is_populated(ttm))
> > > > + return 0;
> > > > +
> > >
> > > This here is just because of a bug in the higher level function.
> > >
> > > I've just pushed the fix for that to drm-misc-fixes, so maybe drop
> > > that here as soon as this is backmerged.
> > >
> > That code doesn't look correct to me. In ttm_device_swapout only the
> > lru lock is held, and the bo->ttm pointer is protected by the resv
> > lock, meaning that bo->ttm can disappear at any time in that
> > function,
> > so while an advisory reading bo->ttm using READ_ONCE() is ok,
> > dereferencing the bo->ttm pointer without reservation held is illegal
> > and may send you into recently freed memory.
> >
> > For an example, consider
> >
> > thread A. Selects bo for eviction, moves to system lru, creates ttm
> > Thread B locks lru in swapout code. finds bo->ttm NON_NULL,
> > thread A tries to evict bo, fails, destroys the ttm.
> > Thread B derefs freed memory.
> >
> > But even relying on that there were no such example in the ttm core
> > itself, not adhering to the protection of bo->ttm makes the code
> > extremely fragile and IMHO needs fixing.
> >
> > Also as a secondary note, a driver is in principle free to do things
> > in the swap notifier that may result in an unpopulated ttm so IMHO a
> > late check is needed here.
> >
> > So ack to keep the above?
>
> Oh, really good point. Haven't thought about that for the quick fix.
>
> I think for the short term we need to protect TT destruction by the
> spinlock.
To avoid more locking complexity,
I think it would be easy to just defer the code that derefs the ttm
(except the new unpopulated check) to ttm_bo_swapout() after we've
taken the resv trylock, but before taking the kref, returning -EBUSY if
conditions for swapping are not met.
>
> The problem doing it here is that you end up in an endless loop
> currently.
>
> E.g. you trylock and inspect the same BO over and over again.
>
> Need to double check the code to see if that can somehow be avoided.
Well ttm_tt_swapout happily returns 0, so the BO gets pulled off the
LRU anyway so I think that shouldn't happen.
The only thing that becomes incorrect is the num_pages return in
ttm_device_swapout(). OTOH, the caller shouldn't care whether we
actually swapped out or whether the bo losts its pages in swap_notify.
/Thomas
>
> Christian.
>
> >
> > Thanks,
> >
> > Thomas
> >
> >
> > >
> > > Christian.
> > >
> > >
> > > > swap_storage = shmem_file_setup("ttm swap", size, 0);
> > > > if (IS_ERR(swap_storage)) {
> > > > pr_err("Failed allocating swap storage\n");
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
2021-05-27 13:52 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-27 14:21 ` Thomas Hellström
-1 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-27 14:21 UTC (permalink / raw)
To: Christian König, Thomas Hellström (Intel),
intel-gfx, dri-devel
On Thu, 2021-05-27 at 15:52 +0200, Thomas Hellström wrote:
> On Thu, 2021-05-27 at 14:36 +0200, Christian König wrote:
> > Am 27.05.21 um 09:33 schrieb Thomas Hellström (Intel):
> > > Hi, Christian,
> > >
> > > Thanks for reviewing.
> > >
> > > On 5/26/21 3:26 PM, Christian König wrote:
> > > > Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> > > > > We are calling the eviction_valuable driver callback at
> > > > > eviction
> > > > > time to
> > > > > determine whether we actually can evict a buffer object.
> > > > > The upcoming i915 TTM backend needs the same functionality
> > > > > for
> > > > > swapout,
> > > > > and that might actually be beneficial to other drivers as
> > > > > well.
> > > > >
> > > > > Add an eviction_valuable call also in the swapout path. Try
> > > > > to
> > > > > keep the
> > > > > current behaviour for all drivers by returning true if the
> > > > > buffer
> > > > > object
> > > > > is already in the TTM_PL_SYSTEM placement. We change
> > > > > behaviour
> > > > > for the
> > > > > case where a buffer object is in a TT backed placement when
> > > > > swapped
> > > > > out,
> > > > > in which case the drivers normal eviction_valuable path is
> > > > > run.
> > > > >
> > > > > Finally make sure we don't try to swapout a bo that was
> > > > > recently
> > > > > purged
> > > > > and therefore unpopulated.
> > > > >
> > > > > Reviewed-by: Maarten Lankhorst <
> > > > > maarten.lankhorst@linux.intel.com>
> > > > > Cc: Christian König <christian.koenig@amd.com>
> > > > > Signed-off-by: Thomas Hellström <
> > > > > thomas.hellstrom@linux.intel.com>
> > > > > ---
> > > > > v3:
> > > > > - Don't export ttm_tt_unpopulate
> > > > > - Fix confusion reading the locked pointer instead of the
> > > > > value
> > > > > pointed to in ttm_bo_evict_swapout_allowable (Reported by
> > > > > Maarten Lankhorst)
> > > > > ---
> > > > > drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
> > > > > drivers/gpu/drm/ttm/ttm_bo.c | 43
> > > > > ++++++++++++++++---------
> > > > > drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
> > > > > 3 files changed, 34 insertions(+), 16 deletions(-)
> > > > >
> > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > > index 3bc3aebfef7c..45d194bffc3f 100644
> > > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > > @@ -1348,6 +1348,10 @@ static bool
> > > > > amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> > > > > struct dma_fence *f;
> > > > > int i;
> > > > > + /* Swapout? */
> > > > > + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> > > > > + return true;
> > > > > +
> > > > > if (bo->type == ttm_bo_type_kernel &&
> > > > > !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
> > > > > return false;
> > > > > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > index be0406466460..1b2d062266ed 100644
> > > > > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct
> > > > > ttm_buffer_object *bo,
> > > > > bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> > > > > const struct ttm_place *place)
> > > > > {
> > > > > + dma_resv_assert_held(bo->base.resv);
> > > > > + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> > > > > + return true;
> > > > > +
> > > > > /* Don't evict this BO if it's outside of the
> > > > > * requested placement range
> > > > > */
> > > > > @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
> > > > > * b. Otherwise, trylock it.
> > > > > */
> > > > > static bool ttm_bo_evict_swapout_allowable(struct
> > > > > ttm_buffer_object *bo,
> > > > > - struct ttm_operation_ctx *ctx, bool *locked,
> > > > > bool
> > > > > *busy)
> > > > > + struct ttm_operation_ctx *ctx,
> > > > > + const struct ttm_place *place,
> > > > > + bool *locked, bool *busy)
> > > > > {
> > > > > bool ret = false;
> > > > > @@ -576,6 +582,14 @@ static bool
> > > > > ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
> > > > > *busy = !ret;
> > > > > }
> > > > > + if (ret && place && !bo->bdev->funcs-
> > > > > > eviction_valuable(bo,
> > > > > place)) {
> > > > > + ret = false;
> > > > > + if (*locked) {
> > > > > + dma_resv_unlock(bo->base.resv);
> > > > > + *locked = false;
> > > > > + }
> > > > > + }
> > > > > +
> > > > > return ret;
> > > > > }
> > > > > @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct
> > > > > ttm_device
> > > > > *bdev,
> > > > > list_for_each_entry(bo, &man->lru[i], lru) {
> > > > > bool busy;
> > > > > - if (!ttm_bo_evict_swapout_allowable(bo, ctx,
> > > > > &locked,
> > > > > - &busy)) {
> > > > > + if (!ttm_bo_evict_swapout_allowable(bo, ctx,
> > > > > place,
> > > > > + &locked, &busy)) {
> > > > > if (busy && !busy_bo && ticket !=
> > > > > dma_resv_locking_ctx(bo->base.resv))
> > > > > busy_bo = bo;
> > > > > continue;
> > > > > }
> > > > > - if (place && !bdev->funcs-
> > > > > >eviction_valuable(bo,
> > > > > - place)) {
> > > > > - if (locked)
> > > > > - dma_resv_unlock(bo->base.resv);
> > > > > - continue;
> > > > > - }
> > > > > if (!ttm_bo_get_unless_zero(bo)) {
> > > > > if (locked)
> > > > > dma_resv_unlock(bo->base.resv);
> > > > > @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
> > > > > int ttm_bo_swapout(struct ttm_buffer_object *bo, struct
> > > > > ttm_operation_ctx *ctx,
> > > > > gfp_t gfp_flags)
> > > > > {
> > > > > + struct ttm_place place = {};
> > > > > bool locked;
> > > > > int ret;
> > > > > - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
> > > > > NULL))
> > > > > + /*
> > > > > + * While the bo may already reside in SYSTEM placement,
> > > > > set
> > > > > + * SYSTEM as new placement to cover also the move
> > > > > further
> > > > > below.
> > > > > + * The driver may use the fact that we're moving from
> > > > > SYSTEM
> > > > > + * as an indication that we're about to swap out.
> > > > > + */
> > > > > + place.mem_type = TTM_PL_SYSTEM;
> > > > > + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place,
> > > > > &locked,
> > > > > NULL))
> > > > > return -EBUSY;
> > > > > if (!ttm_bo_get_unless_zero(bo)) {
> > > > > @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct
> > > > > ttm_buffer_object
> > > > > *bo, struct ttm_operation_ctx *ctx,
> > > > > if (bo->mem.mem_type != TTM_PL_SYSTEM) {
> > > > > struct ttm_operation_ctx ctx = { false, false };
> > > > > struct ttm_resource evict_mem;
> > > > > - struct ttm_place place, hop;
> > > > > -
> > > > > - memset(&place, 0, sizeof(place));
> > > > > - memset(&hop, 0, sizeof(hop));
> > > > > -
> > > > > - place.mem_type = TTM_PL_SYSTEM;
> > > > > + struct ttm_place hop = {};
> > > >
> > > > I would stick with memset because of the padding reasons.
> > > >
> > > > > ret = ttm_resource_alloc(bo, &place, &evict_mem);
> > > > > if (unlikely(ret))
> > > > > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > index 913b330a234b..d9793cbb6d13 100644
> > > > > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device
> > > > > *bdev,
> > > > > struct ttm_tt *ttm,
> > > > > struct page *to_page;
> > > > > int i, ret;
> > > > > + if (!ttm_tt_is_populated(ttm))
> > > > > + return 0;
> > > > > +
> > > >
> > > > This here is just because of a bug in the higher level
> > > > function.
> > > >
> > > > I've just pushed the fix for that to drm-misc-fixes, so maybe
> > > > drop
> > > > that here as soon as this is backmerged.
> > > >
> > > That code doesn't look correct to me. In ttm_device_swapout only
> > > the
> > > lru lock is held, and the bo->ttm pointer is protected by the
> > > resv
> > > lock, meaning that bo->ttm can disappear at any time in that
> > > function,
> > > so while an advisory reading bo->ttm using READ_ONCE() is ok,
> > > dereferencing the bo->ttm pointer without reservation held is
> > > illegal
> > > and may send you into recently freed memory.
> > >
> > > For an example, consider
> > >
> > > thread A. Selects bo for eviction, moves to system lru, creates
> > > ttm
> > > Thread B locks lru in swapout code. finds bo->ttm NON_NULL,
> > > thread A tries to evict bo, fails, destroys the ttm.
> > > Thread B derefs freed memory.
> > >
> > > But even relying on that there were no such example in the ttm
> > > core
> > > itself, not adhering to the protection of bo->ttm makes the code
> > > extremely fragile and IMHO needs fixing.
> > >
> > > Also as a secondary note, a driver is in principle free to do
> > > things
> > > in the swap notifier that may result in an unpopulated ttm so
> > > IMHO a
> > > late check is needed here.
> > >
> > > So ack to keep the above?
> >
> > Oh, really good point. Haven't thought about that for the quick
> > fix.
> >
> > spinlock.
>
> To avoid more locking complexity,
> I think it would be easy to just defer the code that derefs the ttm
> (except the new unpopulated check) to ttm_bo_swapout() after we've
> taken the resv trylock, but before taking the kref, returning -EBUSY
> if
> conditions for swapping are not met.
>
> >
> > The problem doing it here is that you end up in an endless loop
> > currently.
> >
> > E.g. you trylock and inspect the same BO over and over again.
> >
> > Need to double check the code to see if that can somehow be
> > avoided.
>
> Well ttm_tt_swapout happily returns 0, so the BO gets pulled off the
> LRU anyway so I think that shouldn't happen.
>
> The only thing that becomes incorrect is the num_pages return in
> ttm_device_swapout(). OTOH, the caller shouldn't care whether we
> actually swapped out or whether the bo losts its pages in
> swap_notify.
>
> /Thomas
>
Sent a quick patch that might do the trick. Completely untested.
/Thomas
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [Intel-gfx] [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
@ 2021-05-27 14:21 ` Thomas Hellström
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Hellström @ 2021-05-27 14:21 UTC (permalink / raw)
To: Christian König, Thomas Hellström (Intel),
intel-gfx, dri-devel
On Thu, 2021-05-27 at 15:52 +0200, Thomas Hellström wrote:
> On Thu, 2021-05-27 at 14:36 +0200, Christian König wrote:
> > Am 27.05.21 um 09:33 schrieb Thomas Hellström (Intel):
> > > Hi, Christian,
> > >
> > > Thanks for reviewing.
> > >
> > > On 5/26/21 3:26 PM, Christian König wrote:
> > > > Am 26.05.21 um 13:32 schrieb Thomas Hellström:
> > > > > We are calling the eviction_valuable driver callback at
> > > > > eviction
> > > > > time to
> > > > > determine whether we actually can evict a buffer object.
> > > > > The upcoming i915 TTM backend needs the same functionality
> > > > > for
> > > > > swapout,
> > > > > and that might actually be beneficial to other drivers as
> > > > > well.
> > > > >
> > > > > Add an eviction_valuable call also in the swapout path. Try
> > > > > to
> > > > > keep the
> > > > > current behaviour for all drivers by returning true if the
> > > > > buffer
> > > > > object
> > > > > is already in the TTM_PL_SYSTEM placement. We change
> > > > > behaviour
> > > > > for the
> > > > > case where a buffer object is in a TT backed placement when
> > > > > swapped
> > > > > out,
> > > > > in which case the drivers normal eviction_valuable path is
> > > > > run.
> > > > >
> > > > > Finally make sure we don't try to swapout a bo that was
> > > > > recently
> > > > > purged
> > > > > and therefore unpopulated.
> > > > >
> > > > > Reviewed-by: Maarten Lankhorst <
> > > > > maarten.lankhorst@linux.intel.com>
> > > > > Cc: Christian König <christian.koenig@amd.com>
> > > > > Signed-off-by: Thomas Hellström <
> > > > > thomas.hellstrom@linux.intel.com>
> > > > > ---
> > > > > v3:
> > > > > - Don't export ttm_tt_unpopulate
> > > > > - Fix confusion reading the locked pointer instead of the
> > > > > value
> > > > > pointed to in ttm_bo_evict_swapout_allowable (Reported by
> > > > > Maarten Lankhorst)
> > > > > ---
> > > > > drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
> > > > > drivers/gpu/drm/ttm/ttm_bo.c | 43
> > > > > ++++++++++++++++---------
> > > > > drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
> > > > > 3 files changed, 34 insertions(+), 16 deletions(-)
> > > > >
> > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > > index 3bc3aebfef7c..45d194bffc3f 100644
> > > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > > > > @@ -1348,6 +1348,10 @@ static bool
> > > > > amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> > > > > struct dma_fence *f;
> > > > > int i;
> > > > > + /* Swapout? */
> > > > > + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> > > > > + return true;
> > > > > +
> > > > > if (bo->type == ttm_bo_type_kernel &&
> > > > > !amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
> > > > > return false;
> > > > > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > index be0406466460..1b2d062266ed 100644
> > > > > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > @@ -536,6 +536,10 @@ static int ttm_bo_evict(struct
> > > > > ttm_buffer_object *bo,
> > > > > bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
> > > > > const struct ttm_place *place)
> > > > > {
> > > > > + dma_resv_assert_held(bo->base.resv);
> > > > > + if (bo->mem.mem_type == TTM_PL_SYSTEM)
> > > > > + return true;
> > > > > +
> > > > > /* Don't evict this BO if it's outside of the
> > > > > * requested placement range
> > > > > */
> > > > > @@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
> > > > > * b. Otherwise, trylock it.
> > > > > */
> > > > > static bool ttm_bo_evict_swapout_allowable(struct
> > > > > ttm_buffer_object *bo,
> > > > > - struct ttm_operation_ctx *ctx, bool *locked,
> > > > > bool
> > > > > *busy)
> > > > > + struct ttm_operation_ctx *ctx,
> > > > > + const struct ttm_place *place,
> > > > > + bool *locked, bool *busy)
> > > > > {
> > > > > bool ret = false;
> > > > > @@ -576,6 +582,14 @@ static bool
> > > > > ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
> > > > > *busy = !ret;
> > > > > }
> > > > > + if (ret && place && !bo->bdev->funcs-
> > > > > > eviction_valuable(bo,
> > > > > place)) {
> > > > > + ret = false;
> > > > > + if (*locked) {
> > > > > + dma_resv_unlock(bo->base.resv);
> > > > > + *locked = false;
> > > > > + }
> > > > > + }
> > > > > +
> > > > > return ret;
> > > > > }
> > > > > @@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct
> > > > > ttm_device
> > > > > *bdev,
> > > > > list_for_each_entry(bo, &man->lru[i], lru) {
> > > > > bool busy;
> > > > > - if (!ttm_bo_evict_swapout_allowable(bo, ctx,
> > > > > &locked,
> > > > > - &busy)) {
> > > > > + if (!ttm_bo_evict_swapout_allowable(bo, ctx,
> > > > > place,
> > > > > + &locked, &busy)) {
> > > > > if (busy && !busy_bo && ticket !=
> > > > > dma_resv_locking_ctx(bo->base.resv))
> > > > > busy_bo = bo;
> > > > > continue;
> > > > > }
> > > > > - if (place && !bdev->funcs-
> > > > > >eviction_valuable(bo,
> > > > > - place)) {
> > > > > - if (locked)
> > > > > - dma_resv_unlock(bo->base.resv);
> > > > > - continue;
> > > > > - }
> > > > > if (!ttm_bo_get_unless_zero(bo)) {
> > > > > if (locked)
> > > > > dma_resv_unlock(bo->base.resv);
> > > > > @@ -1140,10 +1148,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
> > > > > int ttm_bo_swapout(struct ttm_buffer_object *bo, struct
> > > > > ttm_operation_ctx *ctx,
> > > > > gfp_t gfp_flags)
> > > > > {
> > > > > + struct ttm_place place = {};
> > > > > bool locked;
> > > > > int ret;
> > > > > - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
> > > > > NULL))
> > > > > + /*
> > > > > + * While the bo may already reside in SYSTEM placement,
> > > > > set
> > > > > + * SYSTEM as new placement to cover also the move
> > > > > further
> > > > > below.
> > > > > + * The driver may use the fact that we're moving from
> > > > > SYSTEM
> > > > > + * as an indication that we're about to swap out.
> > > > > + */
> > > > > + place.mem_type = TTM_PL_SYSTEM;
> > > > > + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place,
> > > > > &locked,
> > > > > NULL))
> > > > > return -EBUSY;
> > > > > if (!ttm_bo_get_unless_zero(bo)) {
> > > > > @@ -1168,12 +1184,7 @@ int ttm_bo_swapout(struct
> > > > > ttm_buffer_object
> > > > > *bo, struct ttm_operation_ctx *ctx,
> > > > > if (bo->mem.mem_type != TTM_PL_SYSTEM) {
> > > > > struct ttm_operation_ctx ctx = { false, false };
> > > > > struct ttm_resource evict_mem;
> > > > > - struct ttm_place place, hop;
> > > > > -
> > > > > - memset(&place, 0, sizeof(place));
> > > > > - memset(&hop, 0, sizeof(hop));
> > > > > -
> > > > > - place.mem_type = TTM_PL_SYSTEM;
> > > > > + struct ttm_place hop = {};
> > > >
> > > > I would stick with memset because of the padding reasons.
> > > >
> > > > > ret = ttm_resource_alloc(bo, &place, &evict_mem);
> > > > > if (unlikely(ret))
> > > > > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > index 913b330a234b..d9793cbb6d13 100644
> > > > > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > @@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device
> > > > > *bdev,
> > > > > struct ttm_tt *ttm,
> > > > > struct page *to_page;
> > > > > int i, ret;
> > > > > + if (!ttm_tt_is_populated(ttm))
> > > > > + return 0;
> > > > > +
> > > >
> > > > This here is just because of a bug in the higher level
> > > > function.
> > > >
> > > > I've just pushed the fix for that to drm-misc-fixes, so maybe
> > > > drop
> > > > that here as soon as this is backmerged.
> > > >
> > > That code doesn't look correct to me. In ttm_device_swapout only
> > > the
> > > lru lock is held, and the bo->ttm pointer is protected by the
> > > resv
> > > lock, meaning that bo->ttm can disappear at any time in that
> > > function,
> > > so while an advisory reading bo->ttm using READ_ONCE() is ok,
> > > dereferencing the bo->ttm pointer without reservation held is
> > > illegal
> > > and may send you into recently freed memory.
> > >
> > > For an example, consider
> > >
> > > thread A. Selects bo for eviction, moves to system lru, creates
> > > ttm
> > > Thread B locks lru in swapout code. finds bo->ttm NON_NULL,
> > > thread A tries to evict bo, fails, destroys the ttm.
> > > Thread B derefs freed memory.
> > >
> > > But even relying on that there were no such example in the ttm
> > > core
> > > itself, not adhering to the protection of bo->ttm makes the code
> > > extremely fragile and IMHO needs fixing.
> > >
> > > Also as a secondary note, a driver is in principle free to do
> > > things
> > > in the swap notifier that may result in an unpopulated ttm so
> > > IMHO a
> > > late check is needed here.
> > >
> > > So ack to keep the above?
> >
> > Oh, really good point. Haven't thought about that for the quick
> > fix.
> >
> > spinlock.
>
> To avoid more locking complexity,
> I think it would be easy to just defer the code that derefs the ttm
> (except the new unpopulated check) to ttm_bo_swapout() after we've
> taken the resv trylock, but before taking the kref, returning -EBUSY
> if
> conditions for swapping are not met.
>
> >
> > The problem doing it here is that you end up in an endless loop
> > currently.
> >
> > E.g. you trylock and inspect the same BO over and over again.
> >
> > Need to double check the code to see if that can somehow be
> > avoided.
>
> Well ttm_tt_swapout happily returns 0, so the BO gets pulled off the
> LRU anyway so I think that shouldn't happen.
>
> The only thing that becomes incorrect is the num_pages return in
> ttm_device_swapout(). OTOH, the caller shouldn't care whether we
> actually swapped out or whether the bo losts its pages in
> swap_notify.
>
> /Thomas
>
Sent a quick patch that might do the trick. Completely untested.
/Thomas
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 58+ messages in thread
end of thread, other threads:[~2021-05-27 14:21 UTC | newest]
Thread overview: 58+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-26 11:32 [PATCH v4 00/15] drm/i915: Move LMEM (VRAM) management over to TTM Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 01/15] drm/i915: Untangle the vma pages_mutex Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 02/15] drm/i915: Don't free shared locks while shared Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 03/15] drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 04/15] drm/i915/ttm Initialize the ttm device and memory managers Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 05/15] drm/i915/ttm: Embed a ttm buffer object in the i915 gem object Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 06/15] drm/ttm: Add a generic TTM memcpy move for page-based iomem Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 07/15] drm, drm/i915: Move the memcpy_from_wc functionality to core drm Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 14:27 ` Christian König
2021-05-26 14:27 ` [Intel-gfx] " Christian König
2021-05-26 11:32 ` [PATCH v4 08/15] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 09/15] drm/ttm: Document and optimize ttm_bo_pipeline_gutting() Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 14:32 ` Christian König
2021-05-26 14:32 ` [Intel-gfx] " Christian König
2021-05-26 11:32 ` [PATCH v4 10/15] drm/ttm, drm/amdgpu: Allow the driver some control over swapping Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 13:26 ` Christian König
2021-05-26 13:26 ` [Intel-gfx] " Christian König
2021-05-27 7:33 ` Thomas Hellström (Intel)
2021-05-27 7:33 ` [Intel-gfx] " Thomas Hellström (Intel)
2021-05-27 12:36 ` Christian König
2021-05-27 12:36 ` [Intel-gfx] " Christian König
2021-05-27 13:52 ` Thomas Hellström
2021-05-27 13:52 ` [Intel-gfx] " Thomas Hellström
2021-05-27 14:21 ` Thomas Hellström
2021-05-27 14:21 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 11/15] drm/i915/ttm: Introduce a TTM i915 gem object backend Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 12/15] drm/i915/lmem: Verify checks for lmem residency Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 11:32 ` [PATCH v4 13/15] drm/i915: Disable mmap ioctl for gen12+ Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 17:28 ` Thomas Hellström (Intel)
2021-05-26 17:28 ` [Intel-gfx] " Thomas Hellström (Intel)
2021-05-26 11:32 ` [PATCH v4 14/15] drm/vma: Add a driver_private member to vma_node Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-27 8:16 ` Thomas Hellström (Intel)
2021-05-27 8:16 ` [Intel-gfx] " Thomas Hellström (Intel)
2021-05-26 11:32 ` [PATCH v4 15/15] drm/i915: Use ttm mmap handling for ttm bo's Thomas Hellström
2021-05-26 11:32 ` [Intel-gfx] " Thomas Hellström
2021-05-26 17:40 ` Thomas Hellström
2021-05-26 17:40 ` [Intel-gfx] " Thomas Hellström
2021-05-27 11:11 ` Maarten Lankhorst
2021-05-27 11:11 ` [Intel-gfx] " Maarten Lankhorst
2021-05-26 16:20 ` [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for drm/i915: Move LMEM (VRAM) management over to TTM (rev4) Patchwork
2021-05-26 16:23 ` [Intel-gfx] ✗ Fi.CI.SPARSE: " Patchwork
2021-05-26 16:51 ` [Intel-gfx] ✓ Fi.CI.BAT: success " Patchwork
2021-05-27 1:57 ` [Intel-gfx] ✓ Fi.CI.IGT: " Patchwork
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.