* [PATCH 00/11] Introduce drm evictable lru
@ 2023-11-02 4:32 Oak Zeng
2023-11-02 4:32 ` [RFC 01/11] drm/ttm: re-parameter ttm_device_init Oak Zeng
` (11 more replies)
0 siblings, 12 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:32 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
We plan to implement xe driver's shared virtual memory
manager (aka SVM) without buffer object concept. This
means we won't build our shared virtual memory manager
upon TTM infrastructure like amdgpu does.
Even though this approach is more efficient, it does
create a problem for memory eviction when there is
memory pressure: memory allocated by SVM and memory
allocated by TTM should be able to mutually evict
from each other. TTM's resource manager maintains
a LRU list for each memory type and this list is used
to pick up the memory eviction victim. Since we don't
use TTM for SVM implementation, SVM allocated memory
can't be added to TTM resource manager's LRU list. Thus
SVM allocated memory and TTM allocated memory are not
mutually evictable.
See more discussion on this topic here:
https://www.spinics.net/lists/dri-devel/msg410740.html
This series solve this problem by creating a shared
LRU list b/t SVM and TTM, or any other resource manager.
The basic idea is, abstract a drm_lru_entity structure
which is supposed to be embedded in ttm_resource structure,
or any other resource manager. The resource LRU list is a
list of drm_lru_entity. drm_lru_entity has eviction function
pointers which can be used to call back drivers' specific
eviction function to evict a memory resource.
Introduce global drm_lru_manager to struct drm_device
to manage LRU lists. Each memory type or memory region
can have a LRU list. TTM resource manager's LRU list functions
including bulk move functions are moved to drm lru manager.
drm lru manager provides a evict_first function to evict
the first memory resource from LRU list. This function can
be called from TTM, SVM or any other resource manager, so
all the memory allocated in the drm sub-system can be mutually
evicted.
The lru_lock is also moved from struct ttm_device to struct
drm_device.
Opens:
1) memory accounting: currently the ttm resource manager's
memory accounting functions is kept at ttm resource manager.
Since memory accounting should be cross TTM and SVM, it should
be ideally in the drm lru manager layer. This will be polished
in the future.
2) eviction callback function interface: The current eviction
function interface is designed to meet TTM memory eviction
requirements. When SVM is in the picture, this interface
need to be futher tunned to meet SVM requirement also.
This series is not tested and it is only compiled for xe
driver. Some minor changes are needed for other driver
such as amdgpu, nouveau etc. I intended to send this out
as a request for comment series to get some early feedback,
to see whether this is the right direction to go. I will
futher polish this series after a direction is agreed.
Oak Zeng (11):
drm/ttm: re-parameter ttm_device_init
drm: move lru_lock from ttm_device to drm_device
drm: introduce drm evictable LRU
drm: Add evict function pointer to drm lru entity
drm: Replace ttm macros with drm macros
drm/ttm: Set lru manager to ttm resource manager
drm/ttm: re-parameterize a few ttm functions
drm: Initialize drm lru manager
drm/ttm: Use drm LRU manager iterator
drm/ttm: Implement ttm memory evict functions
drm/ttm: Write ttm functions using drm lru manager functions
drivers/gpu/drm/Makefile | 1 +
drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 +
.../gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c | 6 +
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 10 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 6 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 10 +-
drivers/gpu/drm/drm_drv.c | 1 +
drivers/gpu/drm/drm_evictable_lru.c | 266 ++++++++++++++++++
drivers/gpu/drm/drm_gem_vram_helper.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 6 +-
drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 10 +
drivers/gpu/drm/i915/intel_region_ttm.c | 4 +-
drivers/gpu/drm/i915/selftests/mock_region.c | 2 +-
drivers/gpu/drm/loongson/lsdc_ttm.c | 10 +-
drivers/gpu/drm/nouveau/nouveau_ttm.c | 22 +-
drivers/gpu/drm/qxl/qxl_ttm.c | 6 +-
drivers/gpu/drm/radeon/radeon_ttm.c | 10 +-
drivers/gpu/drm/ttm/tests/ttm_device_test.c | 2 +-
drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c | 2 +-
drivers/gpu/drm/ttm/ttm_bo.c | 247 ++++++++++++----
drivers/gpu/drm/ttm/ttm_bo_util.c | 20 +-
drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
drivers/gpu/drm/ttm/ttm_device.c | 55 ++--
drivers/gpu/drm/ttm/ttm_module.h | 3 +-
drivers/gpu/drm/ttm/ttm_range_manager.c | 14 +-
drivers/gpu/drm/ttm/ttm_resource.c | 242 +++-------------
drivers/gpu/drm/ttm/ttm_sys_manager.c | 8 +-
drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 2 +-
drivers/gpu/drm/vmwgfx/vmwgfx_bo.h | 2 +-
drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 6 +-
.../gpu/drm/vmwgfx/vmwgfx_system_manager.c | 6 +
drivers/gpu/drm/xe/xe_bo.c | 48 ++--
drivers/gpu/drm/xe/xe_bo.h | 5 +-
drivers/gpu/drm/xe/xe_device.c | 2 +-
drivers/gpu/drm/xe/xe_dma_buf.c | 4 +-
drivers/gpu/drm/xe/xe_exec.c | 6 +-
drivers/gpu/drm/xe/xe_migrate.c | 6 +-
drivers/gpu/drm/xe/xe_res_cursor.h | 10 +-
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 8 +-
drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 18 +-
drivers/gpu/drm/xe/xe_vm.c | 6 +-
drivers/gpu/drm/xe/xe_vm_types.h | 2 +-
include/drm/drm_device.h | 12 +
include/drm/drm_evictable_lru.h | 260 +++++++++++++++++
include/drm/ttm/ttm_bo.h | 10 +-
include/drm/ttm/ttm_device.h | 13 +-
include/drm/ttm/ttm_range_manager.h | 17 +-
include/drm/ttm/ttm_resource.h | 117 +++-----
49 files changed, 1042 insertions(+), 501 deletions(-)
create mode 100644 drivers/gpu/drm/drm_evictable_lru.c
create mode 100644 include/drm/drm_evictable_lru.h
--
2.26.3
^ permalink raw reply [flat|nested] 19+ messages in thread
* [RFC 01/11] drm/ttm: re-parameter ttm_device_init
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
@ 2023-11-02 4:32 ` Oak Zeng
2023-11-02 4:32 ` [RFC 02/11] drm: move lru_lock from ttm_device to drm_device Oak Zeng
` (10 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:32 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
Change the 3rd parameter of ttm_device_init from
struct device * to struct drm_device *. This is
a prepare work for moving lru_lock from ttm_device
to drm_device.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +-
drivers/gpu/drm/drm_gem_vram_helper.c | 2 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 2 +-
drivers/gpu/drm/loongson/lsdc_ttm.c | 2 +-
drivers/gpu/drm/nouveau/nouveau_ttm.c | 2 +-
drivers/gpu/drm/radeon/radeon_ttm.c | 2 +-
drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c | 2 +-
drivers/gpu/drm/ttm/ttm_device.c | 7 ++++---
drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 2 +-
drivers/gpu/drm/xe/xe_device.c | 2 +-
include/drm/ttm/ttm_device.h | 3 ++-
11 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 4e51dce3aab5..5cdbc901cbe2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1817,7 +1817,7 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
mutex_init(&adev->mman.gtt_window_lock);
/* No others user of address space so set it to 0 */
- r = ttm_device_init(&adev->mman.bdev, &amdgpu_bo_driver, adev->dev,
+ r = ttm_device_init(&adev->mman.bdev, &amdgpu_bo_driver, adev_to_drm(adev),
adev_to_drm(adev)->anon_inode->i_mapping,
adev_to_drm(adev)->vma_offset_manager,
adev->need_swiotlb,
diff --git a/drivers/gpu/drm/drm_gem_vram_helper.c b/drivers/gpu/drm/drm_gem_vram_helper.c
index b67eafa55715..56749e40459f 100644
--- a/drivers/gpu/drm/drm_gem_vram_helper.c
+++ b/drivers/gpu/drm/drm_gem_vram_helper.c
@@ -1002,7 +1002,7 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev,
vmm->vram_base = vram_base;
vmm->vram_size = vram_size;
- ret = ttm_device_init(&vmm->bdev, &bo_driver, dev->dev,
+ ret = ttm_device_init(&vmm->bdev, &bo_driver, dev,
dev->anon_inode->i_mapping,
dev->vma_offset_manager,
false, true);
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
index bf6097e7433d..b845782c9859 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.c
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -33,7 +33,7 @@ int intel_region_ttm_device_init(struct drm_i915_private *dev_priv)
struct drm_device *drm = &dev_priv->drm;
return ttm_device_init(&dev_priv->bdev, i915_ttm_driver(),
- drm->dev, drm->anon_inode->i_mapping,
+ drm, drm->anon_inode->i_mapping,
drm->vma_offset_manager, false, false);
}
diff --git a/drivers/gpu/drm/loongson/lsdc_ttm.c b/drivers/gpu/drm/loongson/lsdc_ttm.c
index bf79dc55afa4..bd68cb9366b5 100644
--- a/drivers/gpu/drm/loongson/lsdc_ttm.c
+++ b/drivers/gpu/drm/loongson/lsdc_ttm.c
@@ -548,7 +548,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev)
unsigned long num_gtt_pages;
int ret;
- ret = ttm_device_init(&ldev->bdev, &lsdc_bo_driver, ddev->dev,
+ ret = ttm_device_init(&ldev->bdev, &lsdc_bo_driver, ddev,
ddev->anon_inode->i_mapping,
ddev->vma_offset_manager, false, true);
if (ret)
diff --git a/drivers/gpu/drm/nouveau/nouveau_ttm.c b/drivers/gpu/drm/nouveau/nouveau_ttm.c
index 486f39f31a38..831918437850 100644
--- a/drivers/gpu/drm/nouveau/nouveau_ttm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_ttm.c
@@ -299,7 +299,7 @@ nouveau_ttm_init(struct nouveau_drm *drm)
drm->agp.cma = pci->agp.cma;
}
- ret = ttm_device_init(&drm->ttm.bdev, &nouveau_bo_driver, drm->dev->dev,
+ ret = ttm_device_init(&drm->ttm.bdev, &nouveau_bo_driver, dev,
dev->anon_inode->i_mapping,
dev->vma_offset_manager,
drm_need_swiotlb(drm->client.mmu.dmabits),
diff --git a/drivers/gpu/drm/radeon/radeon_ttm.c b/drivers/gpu/drm/radeon/radeon_ttm.c
index 4eb83ccc4906..77ca50187162 100644
--- a/drivers/gpu/drm/radeon/radeon_ttm.c
+++ b/drivers/gpu/drm/radeon/radeon_ttm.c
@@ -688,7 +688,7 @@ int radeon_ttm_init(struct radeon_device *rdev)
int r;
/* No others user of address space so set it to 0 */
- r = ttm_device_init(&rdev->mman.bdev, &radeon_bo_driver, rdev->dev,
+ r = ttm_device_init(&rdev->mman.bdev, &radeon_bo_driver, rdev->ddev,
rdev->ddev->anon_inode->i_mapping,
rdev->ddev->vma_offset_manager,
rdev->need_swiotlb,
diff --git a/drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c b/drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c
index 81661d8827aa..63eb6fdc3460 100644
--- a/drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c
+++ b/drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c
@@ -16,7 +16,7 @@ int ttm_device_kunit_init(struct ttm_test_devices *priv,
struct drm_device *drm = priv->drm;
int err;
- err = ttm_device_init(ttm, &ttm_dev_funcs, drm->dev,
+ err = ttm_device_init(ttm, &ttm_dev_funcs, drm,
drm->anon_inode->i_mapping,
drm->vma_offset_manager,
use_dma_alloc, use_dma32);
diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c
index 7726a72befc5..12014788b595 100644
--- a/drivers/gpu/drm/ttm/ttm_device.c
+++ b/drivers/gpu/drm/ttm/ttm_device.c
@@ -179,7 +179,7 @@ EXPORT_SYMBOL(ttm_device_swapout);
*
* @bdev: A pointer to a struct ttm_device to initialize.
* @funcs: Function table for the device.
- * @dev: The core kernel device pointer for DMA mappings and allocations.
+ * @drm: drm_device pointer
* @mapping: The address space to use for this bo.
* @vma_manager: A pointer to a vma manager.
* @use_dma_alloc: If coherent DMA allocation API should be used.
@@ -190,7 +190,7 @@ EXPORT_SYMBOL(ttm_device_swapout);
* !0: Failure.
*/
int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *funcs,
- struct device *dev, struct address_space *mapping,
+ struct drm_device *drm, struct address_space *mapping,
struct drm_vma_offset_manager *vma_manager,
bool use_dma_alloc, bool use_dma32)
{
@@ -213,7 +213,8 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func
bdev->funcs = funcs;
ttm_sys_man_init(bdev);
- ttm_pool_init(&bdev->pool, dev, NUMA_NO_NODE, use_dma_alloc, use_dma32);
+ ttm_pool_init(&bdev->pool, drm?drm->dev:NULL, NUMA_NO_NODE,
+ use_dma_alloc, use_dma32);
bdev->vma_manager = vma_manager;
spin_lock_init(&bdev->lru_lock);
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
index 8b24ecf60e3e..cf1c1f16102a 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
@@ -1047,7 +1047,7 @@ static int vmw_driver_load(struct vmw_private *dev_priv, u32 pci_id)
}
ret = ttm_device_init(&dev_priv->bdev, &vmw_bo_driver,
- dev_priv->drm.dev,
+ &dev_priv->drm,
dev_priv->drm.anon_inode->i_mapping,
dev_priv->drm.vma_offset_manager,
dev_priv->map_mode == vmw_dma_alloc_coherent,
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index a964dc205da9..3377bfd5b1a1 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -191,7 +191,7 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
if (IS_ERR(xe))
return xe;
- err = ttm_device_init(&xe->ttm, &xe_ttm_funcs, xe->drm.dev,
+ err = ttm_device_init(&xe->ttm, &xe_ttm_funcs, &xe->drm,
xe->drm.anon_inode->i_mapping,
xe->drm.vma_offset_manager, false, false);
if (WARN_ON(err))
diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h
index c22f30535c84..bab868d55383 100644
--- a/include/drm/ttm/ttm_device.h
+++ b/include/drm/ttm/ttm_device.h
@@ -29,6 +29,7 @@
#include <linux/workqueue.h>
#include <drm/ttm/ttm_resource.h>
#include <drm/ttm/ttm_pool.h>
+#include <drm/drm_device.h>
struct ttm_device;
struct ttm_placement;
@@ -288,7 +289,7 @@ static inline void ttm_set_driver_manager(struct ttm_device *bdev, int type,
}
int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *funcs,
- struct device *dev, struct address_space *mapping,
+ struct drm_device *drm, struct address_space *mapping,
struct drm_vma_offset_manager *vma_manager,
bool use_dma_alloc, bool use_dma32);
void ttm_device_fini(struct ttm_device *bdev);
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 02/11] drm: move lru_lock from ttm_device to drm_device
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
2023-11-02 4:32 ` [RFC 01/11] drm/ttm: re-parameter ttm_device_init Oak Zeng
@ 2023-11-02 4:32 ` Oak Zeng
2023-11-02 12:53 ` Christian König
2023-11-02 4:32 ` [RFC 03/11] drm: introduce drm evictable LRU Oak Zeng
` (9 subsequent siblings)
11 siblings, 1 reply; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:32 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
In the coming patches, we will share the lru list b/t
ttm bo based memory allocator and hmm/svm based memory
allocator. Thus lru_lock (which is used mainly to protect
the lru list) is moved from struct ttm_device to struct
drm_device, so this lock can be shared b/t those two
memory allocators.
To minimize code change, struct ttm_device still hold
a weak reference of lru_lock, so ttm layer can still
reference to this lock easily.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 4 +-
drivers/gpu/drm/drm_drv.c | 1 +
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 4 +-
drivers/gpu/drm/ttm/ttm_bo.c | 40 +++++++++----------
drivers/gpu/drm/ttm/ttm_device.c | 18 ++++-----
drivers/gpu/drm/ttm/ttm_resource.c | 42 ++++++++++----------
drivers/gpu/drm/xe/xe_bo.c | 4 +-
drivers/gpu/drm/xe/xe_exec.c | 4 +-
drivers/gpu/drm/xe/xe_vm.c | 4 +-
include/drm/drm_device.h | 5 +++
include/drm/ttm/ttm_bo.h | 4 +-
include/drm/ttm/ttm_device.h | 4 +-
13 files changed, 72 insertions(+), 66 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index f5daadcec865..747bcad86d5d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -368,9 +368,9 @@ int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct drm_exec *exec,
void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev,
struct amdgpu_vm *vm)
{
- spin_lock(&adev->mman.bdev.lru_lock);
+ spin_lock(adev->mman.bdev.lru_lock);
ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
- spin_unlock(&adev->mman.bdev.lru_lock);
+ spin_unlock(adev->mman.bdev.lru_lock);
}
/* Create scheduler entities for page table updates */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index c7085a747b03..b83e1741905e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
@@ -290,9 +290,9 @@ static void amdgpu_vram_mgr_do_reserve(struct ttm_resource_manager *man)
vis_usage = amdgpu_vram_mgr_vis_size(adev, block);
atomic64_add(vis_usage, &mgr->vis_usage);
- spin_lock(&man->bdev->lru_lock);
+ spin_lock(man->bdev->lru_lock);
man->usage += rsv->size;
- spin_unlock(&man->bdev->lru_lock);
+ spin_unlock(man->bdev->lru_lock);
list_move(&rsv->blocks, &mgr->reserved_pages);
}
}
diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index 3eda026ffac6..1943c38815aa 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -623,6 +623,7 @@ static int drm_dev_init(struct drm_device *dev,
INIT_LIST_HEAD(&dev->managed.resources);
spin_lock_init(&dev->managed.lock);
+ spin_lock_init(&dev->lru_lock);
/* no per-device feature limits by default */
dev->driver_features = ~0u;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
index 9227f8146a58..c46f54f83f54 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -984,7 +984,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
/*
* Put on the correct LRU list depending on the MADV status
*/
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
if (shrinkable) {
/* Try to keep shmem_tt from being considered for shrinking. */
bo->priority = TTM_MAX_BO_PRIORITY - 1;
@@ -1013,7 +1013,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
}
ttm_bo_move_to_lru_tail(bo);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
}
/*
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index e58b7e249816..26e0555bad0c 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -68,7 +68,7 @@ static void ttm_bo_mem_space_debug(struct ttm_buffer_object *bo,
* @bo: The buffer object.
*
* Move this BO to the tail of all lru lists used to lookup and reserve an
- * object. This function must be called with struct ttm_global::lru_lock
+ * object. This function must be called with struct drm_device::lru_lock
* held, and is used to make a BO less likely to be considered for eviction.
*/
void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo)
@@ -102,13 +102,13 @@ void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo,
if (bo->bulk_move == bulk)
return;
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
if (bo->resource)
ttm_resource_del_bulk_move(bo->resource, bo);
bo->bulk_move = bulk;
if (bo->resource)
ttm_resource_add_bulk_move(bo->resource, bo);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
}
EXPORT_SYMBOL(ttm_bo_set_bulk_move);
@@ -202,9 +202,9 @@ static int ttm_bo_individualize_resv(struct ttm_buffer_object *bo)
* reference it any more. The only tricky case is the trylock on
* the resv object while holding the lru_lock.
*/
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
bo->base.resv = &bo->base._resv;
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
}
return r;
@@ -255,7 +255,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
if (unlock_resv)
dma_resv_unlock(bo->base.resv);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP,
interruptible,
@@ -266,7 +266,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
else if (lret == 0)
return -EBUSY;
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
if (unlock_resv && !dma_resv_trylock(bo->base.resv)) {
/*
* We raced, and lost, someone else holds the reservation now,
@@ -276,7 +276,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
* delayed destruction would succeed, so just return success
* here.
*/
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
return 0;
}
ret = 0;
@@ -285,11 +285,11 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
if (ret) {
if (unlock_resv)
dma_resv_unlock(bo->base.resv);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
return ret;
}
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
ttm_bo_cleanup_memtype_use(bo);
if (unlock_resv)
@@ -351,7 +351,7 @@ static void ttm_bo_release(struct kref *kref)
ttm_bo_flush_all_fences(bo);
bo->deleted = true;
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
/*
* Make pinned bos immediately available to
@@ -367,7 +367,7 @@ static void ttm_bo_release(struct kref *kref)
}
kref_init(&bo->kref);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
INIT_WORK(&bo->delayed_delete, ttm_bo_delayed_delete);
queue_work(bdev->wq, &bo->delayed_delete);
@@ -598,7 +598,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
bool locked = false;
int ret;
- spin_lock(&bdev->lru_lock);
+ spin_lock(bdev->lru_lock);
ttm_resource_manager_for_each_res(man, &cursor, res) {
bool busy;
@@ -621,7 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
if (!bo) {
if (busy_bo && !ttm_bo_get_unless_zero(busy_bo))
busy_bo = NULL;
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket);
if (busy_bo)
ttm_bo_put(busy_bo);
@@ -635,7 +635,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
return ret;
}
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
ret = ttm_bo_evict(bo, ctx);
if (locked)
@@ -658,11 +658,11 @@ void ttm_bo_pin(struct ttm_buffer_object *bo)
{
dma_resv_assert_held(bo->base.resv);
WARN_ON_ONCE(!kref_read(&bo->kref));
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
if (bo->resource)
ttm_resource_del_bulk_move(bo->resource, bo);
++bo->pin_count;
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
}
EXPORT_SYMBOL(ttm_bo_pin);
@@ -679,11 +679,11 @@ void ttm_bo_unpin(struct ttm_buffer_object *bo)
if (WARN_ON_ONCE(!bo->pin_count))
return;
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
--bo->pin_count;
if (bo->resource)
ttm_resource_add_bulk_move(bo->resource, bo);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
}
EXPORT_SYMBOL(ttm_bo_unpin);
@@ -1156,7 +1156,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
}
/* TODO: Cleanup the locking */
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
/*
* Move to system cached
diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c
index 12014788b595..d18eca86ebd6 100644
--- a/drivers/gpu/drm/ttm/ttm_device.c
+++ b/drivers/gpu/drm/ttm/ttm_device.c
@@ -147,7 +147,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
unsigned i;
int ret;
- spin_lock(&bdev->lru_lock);
+ spin_lock(bdev->lru_lock);
for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) {
man = ttm_manager_type(bdev, i);
if (!man || !man->use_tt)
@@ -169,7 +169,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
return ret;
}
}
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
return 0;
}
EXPORT_SYMBOL(ttm_device_swapout);
@@ -217,7 +217,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func
use_dma_alloc, use_dma32);
bdev->vma_manager = vma_manager;
- spin_lock_init(&bdev->lru_lock);
+ bdev->lru_lock = &drm->lru_lock;
INIT_LIST_HEAD(&bdev->pinned);
bdev->dev_mapping = mapping;
mutex_lock(&ttm_global_mutex);
@@ -244,11 +244,11 @@ void ttm_device_fini(struct ttm_device *bdev)
drain_workqueue(bdev->wq);
destroy_workqueue(bdev->wq);
- spin_lock(&bdev->lru_lock);
+ spin_lock(bdev->lru_lock);
for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i)
if (list_empty(&man->lru[0]))
pr_debug("Swap list %d was clean\n", i);
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
ttm_pool_fini(&bdev->pool);
ttm_global_release();
@@ -260,7 +260,7 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,
{
struct ttm_resource *res;
- spin_lock(&bdev->lru_lock);
+ spin_lock(bdev->lru_lock);
while ((res = list_first_entry_or_null(list, typeof(*res), lru))) {
struct ttm_buffer_object *bo = res->bo;
@@ -269,15 +269,15 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,
continue;
list_del_init(&res->lru);
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
if (bo->ttm)
ttm_tt_unpopulate(bo->bdev, bo->ttm);
ttm_bo_put(bo);
- spin_lock(&bdev->lru_lock);
+ spin_lock(bdev->lru_lock);
}
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
}
void ttm_device_clear_dma_mappings(struct ttm_device *bdev)
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 46ff9c75bb12..6ada77f51fba 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -48,7 +48,7 @@ EXPORT_SYMBOL(ttm_lru_bulk_move_init);
* @bulk: bulk move structure
*
* Bulk move BOs to the LRU tail, only valid to use when driver makes sure that
- * resource order never changes. Should be called with &ttm_device.lru_lock held.
+ * resource order never changes. Should be called with &drm_device.lru_lock held.
*/
void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
{
@@ -62,7 +62,7 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
if (!pos->first)
continue;
- lockdep_assert_held(&pos->first->bo->bdev->lru_lock);
+ lockdep_assert_held(pos->first->bo->bdev->lru_lock);
dma_resv_assert_held(pos->first->bo->base.resv);
dma_resv_assert_held(pos->last->bo->base.resv);
@@ -148,7 +148,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res)
struct ttm_buffer_object *bo = res->bo;
struct ttm_device *bdev = bo->bdev;
- lockdep_assert_held(&bo->bdev->lru_lock);
+ lockdep_assert_held(bo->bdev->lru_lock);
if (bo->pin_count) {
list_move_tail(&res->lru, &bdev->pinned);
@@ -191,13 +191,13 @@ void ttm_resource_init(struct ttm_buffer_object *bo,
res->bo = bo;
man = ttm_manager_type(bo->bdev, place->mem_type);
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
if (bo->pin_count)
list_add_tail(&res->lru, &bo->bdev->pinned);
else
list_add_tail(&res->lru, &man->lru[bo->priority]);
man->usage += res->size;
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
}
EXPORT_SYMBOL(ttm_resource_init);
@@ -216,10 +216,10 @@ void ttm_resource_fini(struct ttm_resource_manager *man,
{
struct ttm_device *bdev = man->bdev;
- spin_lock(&bdev->lru_lock);
+ spin_lock(bdev->lru_lock);
list_del_init(&res->lru);
man->usage -= res->size;
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
}
EXPORT_SYMBOL(ttm_resource_fini);
@@ -235,9 +235,9 @@ int ttm_resource_alloc(struct ttm_buffer_object *bo,
if (ret)
return ret;
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
ttm_resource_add_bulk_move(*res_ptr, bo);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
return 0;
}
@@ -248,9 +248,9 @@ void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res)
if (!*res)
return;
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
ttm_resource_del_bulk_move(*res, bo);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
man = ttm_manager_type(bo->bdev, (*res)->mem_type);
man->func->free(man, *res);
*res = NULL;
@@ -368,9 +368,9 @@ bool ttm_resource_compat(struct ttm_resource *res,
void ttm_resource_set_bo(struct ttm_resource *res,
struct ttm_buffer_object *bo)
{
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
res->bo = bo;
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
}
/**
@@ -424,18 +424,18 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
* Can't use standard list traversal since we're unlocking.
*/
- spin_lock(&bdev->lru_lock);
+ spin_lock(bdev->lru_lock);
for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) {
while (!list_empty(&man->lru[i])) {
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
ret = ttm_mem_evict_first(bdev, man, NULL, &ctx,
NULL);
if (ret)
return ret;
- spin_lock(&bdev->lru_lock);
+ spin_lock(bdev->lru_lock);
}
}
- spin_unlock(&bdev->lru_lock);
+ spin_unlock(bdev->lru_lock);
spin_lock(&man->move_lock);
fence = dma_fence_get(man->move);
@@ -463,9 +463,9 @@ uint64_t ttm_resource_manager_usage(struct ttm_resource_manager *man)
{
uint64_t usage;
- spin_lock(&man->bdev->lru_lock);
+ spin_lock(man->bdev->lru_lock);
usage = man->usage;
- spin_unlock(&man->bdev->lru_lock);
+ spin_unlock(man->bdev->lru_lock);
return usage;
}
EXPORT_SYMBOL(ttm_resource_manager_usage);
@@ -502,7 +502,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man,
{
struct ttm_resource *res;
- lockdep_assert_held(&man->bdev->lru_lock);
+ lockdep_assert_held(man->bdev->lru_lock);
for (cursor->priority = 0; cursor->priority < TTM_MAX_BO_PRIORITY;
++cursor->priority)
@@ -526,7 +526,7 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,
struct ttm_resource_cursor *cursor,
struct ttm_resource *res)
{
- lockdep_assert_held(&man->bdev->lru_lock);
+ lockdep_assert_held(man->bdev->lru_lock);
list_for_each_entry_continue(res, &man->lru[cursor->priority], lru)
return res;
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 25fdc04627ca..827f798cccc0 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -946,9 +946,9 @@ static bool xe_ttm_bo_lock_in_destructor(struct ttm_buffer_object *ttm_bo)
* the ttm_bo refcount is zero at this point. So trylocking *should*
* always succeed here, as long as we hold the lru lock.
*/
- spin_lock(&ttm_bo->bdev->lru_lock);
+ spin_lock(ttm_bo->bdev->lru_lock);
locked = dma_resv_trylock(ttm_bo->base.resv);
- spin_unlock(&ttm_bo->bdev->lru_lock);
+ spin_unlock(ttm_bo->bdev->lru_lock);
XE_WARN_ON(!locked);
return locked;
diff --git a/drivers/gpu/drm/xe/xe_exec.c b/drivers/gpu/drm/xe/xe_exec.c
index 890fadb0a93e..dafebdfb2368 100644
--- a/drivers/gpu/drm/xe/xe_exec.c
+++ b/drivers/gpu/drm/xe/xe_exec.c
@@ -370,9 +370,9 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
xe_vm_reactivate_rebind(vm);
if (!err && !xe_vm_no_dma_fences(vm)) {
- spin_lock(&xe->ttm.lru_lock);
+ spin_lock(xe->ttm.lru_lock);
ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
- spin_unlock(&xe->ttm.lru_lock);
+ spin_unlock(xe->ttm.lru_lock);
}
err_repin:
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index a6a0f17fec1d..44e038276d41 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -651,9 +651,9 @@ static void preempt_rebind_work_func(struct work_struct *w)
#undef retry_required
- spin_lock(&vm->xe->ttm.lru_lock);
+ spin_lock(vm->xe->ttm.lru_lock);
ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
- spin_unlock(&vm->xe->ttm.lru_lock);
+ spin_unlock(vm->xe->ttm.lru_lock);
/* Point of no return. */
arm_preempt_fences(vm, &preempt_fences);
diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
index 7cf4afae2e79..d0b5f42786be 100644
--- a/include/drm/drm_device.h
+++ b/include/drm/drm_device.h
@@ -326,6 +326,11 @@ struct drm_device {
*/
struct list_head debugfs_list;
+ /**
+ * @lru_lock: Protection for the per manager LRU and destroy lists.
+ */
+ spinlock_t lru_lock;
+
/* Everything below here is for legacy driver, never use! */
/* private: */
#if IS_ENABLED(CONFIG_DRM_LEGACY)
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index 0223a41a64b2..49f32df32204 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -290,9 +290,9 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo);
static inline void
ttm_bo_move_to_lru_tail_unlocked(struct ttm_buffer_object *bo)
{
- spin_lock(&bo->bdev->lru_lock);
+ spin_lock(bo->bdev->lru_lock);
ttm_bo_move_to_lru_tail(bo);
- spin_unlock(&bo->bdev->lru_lock);
+ spin_unlock(bo->bdev->lru_lock);
}
static inline void ttm_bo_assign_mem(struct ttm_buffer_object *bo,
diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h
index bab868d55383..4d29e96bd892 100644
--- a/include/drm/ttm/ttm_device.h
+++ b/include/drm/ttm/ttm_device.h
@@ -248,9 +248,9 @@ struct ttm_device {
struct ttm_pool pool;
/**
- * @lru_lock: Protection for the per manager LRU and ddestroy lists.
+ * @lru_lock: Weak reference to drm_device::lru_lock.
*/
- spinlock_t lru_lock;
+ spinlock_t *lru_lock;
/**
* @pinned: Buffer objects which are pinned and so not on any LRU list.
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 03/11] drm: introduce drm evictable LRU
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
2023-11-02 4:32 ` [RFC 01/11] drm/ttm: re-parameter ttm_device_init Oak Zeng
2023-11-02 4:32 ` [RFC 02/11] drm: move lru_lock from ttm_device to drm_device Oak Zeng
@ 2023-11-02 4:32 ` Oak Zeng
2023-11-02 13:23 ` Christian König
2023-11-02 4:32 ` [RFC 04/11] drm: Add evict function pointer to drm lru entity Oak Zeng
` (8 subsequent siblings)
11 siblings, 1 reply; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:32 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
drm LRU manager is introuced for resource eviction purpose. It maintains
a LRU list per resource type. It provides functions to add or remove
resource to or from the list. It also provides function to retrieve the
first entity on the LRU list.
drm LRU manager also provides functions for bulk moving resources
on the LRU lists.
drm LRU manager also does very basic memory accounting function, i.e.,
LRU manager keeps a size of this resource type and a usage member
for how much of resource has been added to this LRU manager's LRU
list. TTM resource manager memory accounting functoins such as
struct ttm_resource_manager::size and struct ttm_resource_manger::usage
are still kept. In the future, when SVM codes are in the picture,
those memory accounting functions need some rework to consider
the memory used by both TTM and SVM.
For one device, a global drm LRU manager per resource type should be
created/initialized at device initialization time. Drm LRU manager
instances are embedded in struct drm_device.
It is pretty much moving some of the ttm resource manager functions
to the drm layer. The reason of this code refactory is, we want to
create a single LRU list for memory allocated from BO(buffer object)
based driver and hmm/svm(shared virtual memory) based driver, thus BO
driver and svm driver can evict memory from each other.
Previously the LRU list in TTM resource manager (lru field in struct
ttm_reource_manager) is coupled with ttm_buffer_object concept, i.e.,
each ttm resource is backed by a ttm_buffer_object and the LRU list
is essentially a list of ttm_buffer_object. Due to this behavior, the
TTM resource manager can't be used by hmm/svm driver as we don't plan
to have the BO concept for the hmm/svm implemenation. So we decouple
the evictable LRU list from the BO concept in this series.
The design goal of drm lru manager is to make it as lean as possible.
So each lru entity only has a list node member used to link this entity
to the evictable LRU list, and the basic resource size/type/priority
of this entity. It doesn't have any driver specify information. A lru
entity also has a function pointer of evict function. This is used to
implement ttm or svm specific eviction function. A lru entity is supposed
to be embedded in a driver specific structure such as struct
ttm_resource, see the usage in the next patch of this series.
The ttm resource manager, and some of the ttm_bo functions such as
ttm_mem_evict_first will be rewriten using the new drm lru manager
library, see the next patch in this series.
The future hmm/svm implemenation will call lru manager function to add
hmm/svm allocations to the shared evictable lru list.
Lock design: previously ttm_resource LRU list is protected by a device
global ttm_device::lru_lock (bdev->lru_lock in codes). This lock also
protects ttm_buffer_object::pin_count, ttm_resource_manager::usage,
ttm_resource::bo, ttm_device::pinned list etc. With this refactory,
lru_lock is moved out of ttm_device and is added to struct drm_deive, so
it can be shared b/t ttm code and svm code.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/Makefile | 1 +
drivers/gpu/drm/drm_evictable_lru.c | 232 ++++++++++++++++++++++++++++
include/drm/drm_device.h | 7 +
include/drm/drm_evictable_lru.h | 188 ++++++++++++++++++++++
4 files changed, 428 insertions(+)
create mode 100644 drivers/gpu/drm/drm_evictable_lru.c
create mode 100644 include/drm/drm_evictable_lru.h
diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
index 1ad88efb1752..13953b0d271b 100644
--- a/drivers/gpu/drm/Makefile
+++ b/drivers/gpu/drm/Makefile
@@ -46,6 +46,7 @@ drm-y := \
drm_vblank_work.o \
drm_vma_manager.o \
drm_gpuva_mgr.o \
+ drm_evictable_lru.o \
drm_writeback.o
drm-$(CONFIG_DRM_LEGACY) += \
drm_agpsupport.o \
diff --git a/drivers/gpu/drm/drm_evictable_lru.c b/drivers/gpu/drm/drm_evictable_lru.c
new file mode 100644
index 000000000000..2ba9105cca03
--- /dev/null
+++ b/drivers/gpu/drm/drm_evictable_lru.c
@@ -0,0 +1,232 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2023 Intel Corporation
+ */
+
+#include <linux/lockdep.h>
+#include <linux/container_of.h>
+#include <drm/drm_evictable_lru.h>
+#include <drm/drm_device.h>
+
+static inline struct drm_lru_mgr *entity_to_mgr(struct drm_lru_entity *entity)
+{
+ struct drm_lru_mgr *mgr;
+
+ mgr = &entity->drm->lru_mgr[entity->mem_type];
+ BUG_ON(!mgr->used);
+
+ return mgr;
+}
+
+void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm,
+ uint32_t mem_type, uint64_t size, uint32_t priority)
+{
+ entity->drm = drm;
+ entity->mem_type = mem_type;
+ entity->size = size;
+ entity->priority = priority;
+ INIT_LIST_HEAD(&entity->lru);
+}
+
+/**
+ * drm_lru_mgr_init
+ *
+ * @mgr: drm lru manager to init
+ * @size: size of the resource managed by this manager
+ * @lock: pointer of the global lru_lock
+ *
+ * Initialize a drm lru manager
+ */
+void drm_lru_mgr_init(struct drm_lru_mgr *mgr, uint64_t size, spinlock_t *lock)
+{
+ unsigned j;
+
+ mgr->used = true;
+ mgr->size = size;
+ mgr->usage = 0;
+ mgr->lru_lock = lock;
+
+ for(j = 0; j < DRM_MAX_LRU_PRIORITY; j++)
+ INIT_LIST_HEAD(&mgr->lru[j]);
+}
+
+void drm_lru_bulk_move_init(struct drm_lru_bulk_move *bulk_move)
+{
+ memset(bulk_move, 0, sizeof(*bulk_move));
+}
+
+/**
+ * drm_lru_first
+ *
+ * @mgr: drm lru manager to iterate over
+ * @cursor: cursor of the current position
+ *
+ * Returns the first entity in drm lru manager
+ */
+struct drm_lru_entity *
+drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor)
+{
+ struct drm_lru_entity *entity;
+
+ lockdep_assert_held(mgr->lru_lock);
+
+ for(cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY; ++cursor->priority)
+ list_for_each_entry(entity, &mgr->lru[cursor->priority], lru)
+ return entity;
+
+ return NULL;
+}
+
+/**
+ * drm_lru_next
+ *
+ * @mgr: drm lru manager to iterate over
+ * @cursor: cursor of the current position
+ * @entity: the current lru entity pointer
+ *
+ * Returns the next entity from drm lru manager
+ */
+struct drm_lru_entity *
+drm_lru_next(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor,
+ struct drm_lru_entity *entity)
+{
+ lockdep_assert_held(mgr->lru_lock);
+
+ list_for_each_entry_continue(entity, &mgr->lru[cursor->priority], lru)
+ return entity;
+
+ for(++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY; ++cursor->priority)
+ list_for_each_entry(entity, &mgr->lru[cursor->priority], lru)
+ return entity;
+
+ return NULL;
+}
+
+/**
+ * drm_lru_move_to_tail
+ *
+ * @entity: the lru entity to move to lru tail
+ *
+ * Move a lru entity to lru tail
+ */
+void drm_lru_move_to_tail(struct drm_lru_entity * entity)
+{
+ struct list_head *lru;
+ struct drm_lru_mgr *mgr;
+
+ mgr = entity_to_mgr(entity);
+ lockdep_assert_held(mgr->lru_lock);
+ lru = &mgr->lru[entity->priority];
+ list_move_tail(&entity->lru, lru);
+}
+
+/**
+ * drm_lru_bulk_move_range_tail
+ *
+ * @range: bulk move range
+ * @entity: lru_entity to move
+ *
+ * Move a lru_entity to the tail of a bulk move range
+ */
+void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range,
+ struct drm_lru_entity *entity)
+{
+ if (entity == range->last)
+ return;
+
+ if (entity == range->first)
+ range->first = container_of(entity->lru.next, struct drm_lru_entity, lru);
+
+ if (range->last)
+ list_move(&entity->lru, &range->last->lru);
+
+ range->last = entity;
+}
+EXPORT_SYMBOL(drm_lru_bulk_move_range_tail);
+
+/**
+ * drm_lru_bulk_move_tail - bulk move range of entities to the LRU tail.
+ *
+ * @bulk: bulk_move structure
+ *
+ * Bulk move entities to the LRU tail, only valid to use when driver makes sure that
+ * resource order never changes.
+ */
+void drm_lru_bulk_move_tail(struct drm_lru_bulk_move *bulk)
+{
+
+ unsigned i, j;
+
+ for (i = 0; i < DRM_NUM_MEM_TYPES; ++i) {
+ for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) {
+ struct drm_lru_bulk_move_range *range = &bulk->range[i][j];
+ struct drm_lru_mgr *mgr;
+
+ if (!range->first)
+ continue;
+
+ mgr = entity_to_mgr(range->first);
+ lockdep_assert_held(mgr->lru_lock);
+ list_bulk_move_tail(&mgr->lru[range->first->priority], &range->first->lru,
+ &range->last->lru);
+ }
+ }
+}
+EXPORT_SYMBOL(drm_lru_bulk_move_tail);
+
+/**
+ * drm_lru_add_bulk_move
+ *
+ * @entity: the lru entity to add to the bulk move range
+ * @bulk_move: the bulk move ranges to add the entity
+ *
+ * Add a lru entity to the tail of a bulk move range
+ */
+void drm_lru_add_bulk_move(struct drm_lru_entity *entity,
+ struct drm_lru_bulk_move *bulk_move)
+{
+ struct drm_lru_bulk_move_range *range;
+
+ range = &bulk_move->range[entity->mem_type][entity->priority];
+
+ if (!range->first) {
+ range->first = entity;
+ range->last = entity;
+ return;
+ }
+
+ drm_lru_bulk_move_range_tail(range, entity);
+}
+
+EXPORT_SYMBOL(drm_lru_add_bulk_move);
+/**
+ * drm_lru_del_bulk_move
+ *
+ * @entity: the lru entity to move from the bulk move range
+ * @bulk_move: the bulk move ranges to move the entity out of
+ *
+ * Move a lru entity out of bulk move range. This doesn't
+ * delete entity from lru manager's lru list.
+ */
+void drm_lru_del_bulk_move(struct drm_lru_entity *entity,
+ struct drm_lru_bulk_move *bulk_move)
+{
+ struct drm_lru_bulk_move_range *range;
+
+ range = &bulk_move->range[entity->mem_type][entity->priority];
+
+ if (unlikely(WARN_ON(!range->first || !range->last) ||
+ (range->first == entity && range->last == entity))) {
+ range->first = NULL;
+ range->last = NULL;
+ } else if (range->first == entity) {
+ range->first = container_of(entity->lru.next,
+ struct drm_lru_entity, lru);
+ } else if (range->last == entity) {
+ range->last = container_of(entity->lru.prev,
+ struct drm_lru_entity, lru);
+ } else {
+ list_move(&entity->lru, &range->last->lru);
+ }
+}
+EXPORT_SYMBOL(drm_lru_del_bulk_move);
diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
index d0b5f42786be..1bdcd34d3f6b 100644
--- a/include/drm/drm_device.h
+++ b/include/drm/drm_device.h
@@ -8,6 +8,7 @@
#include <drm/drm_legacy.h>
#include <drm/drm_mode_config.h>
+#include <drm/drm_evictable_lru.h>
struct drm_driver;
struct drm_minor;
@@ -331,6 +332,12 @@ struct drm_device {
*/
spinlock_t lru_lock;
+ /**
+ * @lru_mgr: Device global lru managers per memory type or memory
+ * region. Each lru manager manages a lru list of this memory type.
+ */
+ struct drm_lru_mgr lru_mgr[DRM_NUM_MEM_TYPES];
+
/* Everything below here is for legacy driver, never use! */
/* private: */
#if IS_ENABLED(CONFIG_DRM_LEGACY)
diff --git a/include/drm/drm_evictable_lru.h b/include/drm/drm_evictable_lru.h
new file mode 100644
index 000000000000..3fd6bd2475d9
--- /dev/null
+++ b/include/drm/drm_evictable_lru.h
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2023 Intel Corporation
+ */
+
+#ifndef _DRM_EVICTABLE_LRU_H_
+#define _DRM_EVICTABLE_LRU_H_
+
+#include <linux/list.h>
+#include <linux/spinlock_types.h>
+#include <linux/spinlock.h>
+
+struct drm_device;
+
+#define DRM_MAX_LRU_PRIORITY 4
+#define DRM_NUM_MEM_TYPES 8
+
+/**
+ * struct drm_lru_entity
+ *
+ * @drm: drm device that this entity belongs to
+ * @mem_type: The memory type that this entity belongs to
+ * @size: resource size of this entity
+ * @priority: The priority of this entity
+ * @lru: least recent used list node, see &drm_lru_mgr.lru
+ *
+ * This structure represents an entity in drm_lru_mgr's
+ * list. This structure is supposed to be embedded in
+ * user's data structure.
+ */
+struct drm_lru_entity {
+ struct drm_device *drm;
+ uint32_t mem_type;
+ uint64_t size;
+ uint32_t priority;
+ struct list_head lru;
+};
+
+/**
+ * struct drm_lru_mgr
+ *
+ * @used: whether this lru manager is used or not
+ * @size: size of the resource
+ * @usage: how much resource has been used
+ * @lru_lock: a weak reference to the global lru_lock
+ * @lru: least recent used list, per priority
+ *
+ * This structure maintains all the buffer allocations
+ * in a least recent used list, so a victim for eviction
+ * can be easily found.
+ */
+struct drm_lru_mgr {
+ bool used;
+ uint64_t size;
+ uint64_t usage;
+ spinlock_t *lru_lock;
+ struct list_head lru[DRM_MAX_LRU_PRIORITY];
+};
+
+/**
+ * struct drm_lru_cursor
+ *
+ * @priority: the current priority
+ *
+ * Cursor to iterate over all entities in lru manager.
+ */
+struct drm_lru_cursor {
+ unsigned priority;
+};
+
+/**
+ * struct drm_lru_bulk_move_range
+ *
+ * @first: the first entity in the range
+ * @last: the last entity in the range
+ *
+ * Range of entities on a lru list.
+ */
+struct drm_lru_bulk_move_range
+{
+ struct drm_lru_entity *first;
+ struct drm_lru_entity *last;
+};
+
+/**
+ * struct drm_lru_bulk_move
+ *
+ * @range: An array of bulk move range, each corelates to the drm_lru_mgr's
+ * lru list of the same memory type and same priority.
+ *
+ * A collection of bulk_move range which can be used to move drm_lru_entity
+ * on the lru list in a bulk way. It should be initialized through
+ * drm_lru_bulk_move_init. Add/delete a drm_lru_entity to bulk move should call
+ * drm_lru_add_bulk_move/drm_lru_del_bulk_move.
+ */
+struct drm_lru_bulk_move {
+ struct drm_lru_bulk_move_range range[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY];
+};
+
+
+
+/**
+ * drm_lru_add_entity
+ *
+ * @entity: the lru entity to add
+ * @mgr: the drm lru manager
+ * @priority: specify which priority list to add
+ *
+ * Add an entity to lru list
+ */
+static inline void drm_lru_add_entity(struct drm_lru_entity *entity,
+ struct drm_lru_mgr *mgr, unsigned priority)
+{
+ lockdep_assert_held(mgr->lru_lock);
+ list_add_tail(&entity->lru, &mgr->lru[priority]);
+ mgr->usage += entity->size;
+}
+
+/**
+ * drm_lru_remove_entity
+ *
+ * @entity: the lru entity to remove
+ * @mgr: the drm lru manager
+ *
+ * Remove an entity from lru list
+ */
+static inline void drm_lru_remove_entity(struct drm_lru_entity *entity,
+ struct drm_lru_mgr *mgr)
+{
+ lockdep_assert_held(mgr->lru_lock);
+ list_del_init(&entity->lru);
+ mgr->usage -= entity->size;
+}
+
+/**
+ * drm_lru_mgr_fini
+ *
+ * @mgr: the drm lru manager
+ *
+ * de-initialize a lru manager
+ */
+static inline void drm_lru_mgr_fini(struct drm_lru_mgr *mgr)
+{
+ mgr->used = false;
+}
+
+void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm,
+ uint32_t mem_type, uint64_t size, uint32_t priority);
+
+struct drm_lru_entity *
+drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor);
+
+struct drm_lru_entity *
+drm_lru_next(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor,
+ struct drm_lru_entity *entity);
+
+void drm_lru_mgr_init(struct drm_lru_mgr *mgr, uint64_t size,
+ spinlock_t *lru_lock);
+
+void drm_lru_move_to_tail(struct drm_lru_entity * entity);
+
+void drm_lru_bulk_move_init(struct drm_lru_bulk_move *bulk_move);
+
+
+void drm_lru_bulk_move_tail(struct drm_lru_bulk_move *bulk);
+
+void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range,
+ struct drm_lru_entity *entity);
+
+void drm_lru_add_bulk_move(struct drm_lru_entity *entity,
+ struct drm_lru_bulk_move *bulk_move);
+
+void drm_lru_del_bulk_move(struct drm_lru_entity *entity,
+ struct drm_lru_bulk_move *bulk_move);
+/**
+ * drm_lru_for_each_entity
+ *
+ * @mgr: the drm lru manager
+ * @cursor: cursor for the current position
+ * @entity: the current drm_lru_entity
+ *
+ * Iterate over all entities in drm lru manager
+ */
+#define drm_lru_for_each_entity(mgr, cursor, entity) \
+ for (entity = drm_lru_first(mgr, cursor); entity; \
+ entity = drm_lru_next(mgr, cursor, entity))
+
+#endif
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 04/11] drm: Add evict function pointer to drm lru entity
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (2 preceding siblings ...)
2023-11-02 4:32 ` [RFC 03/11] drm: introduce drm evictable LRU Oak Zeng
@ 2023-11-02 4:32 ` Oak Zeng
2023-11-02 4:33 ` [RFC 05/11] drm: Replace ttm macros with drm macros Oak Zeng
` (7 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:32 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
Drm lru manager provides generic functions to manage lru list,
and function to evict a lru entity. But how to evict an entity
is implemented in an entity's sub-class. This patch introduces
a few function pointers to drm lru entity for this purpose. Those
functions are abstracted from the current ttm resource eviction
process. They need to be tunned in the future when svm code comes
into the picture.
Also implemented a drm_lru_evict_first function to evict the first
lru entity from lru manager. Both ttm and svm codes are supposed
to call this function to evict the first resource from lru list.
This way ttm and svm codes can mutually evict resources from each
other.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/drm_evictable_lru.c | 36 +++++++++++++-
include/drm/drm_evictable_lru.h | 74 ++++++++++++++++++++++++++++-
2 files changed, 108 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/drm_evictable_lru.c b/drivers/gpu/drm/drm_evictable_lru.c
index 2ba9105cca03..7b62cae2dfea 100644
--- a/drivers/gpu/drm/drm_evictable_lru.c
+++ b/drivers/gpu/drm/drm_evictable_lru.c
@@ -19,13 +19,15 @@ static inline struct drm_lru_mgr *entity_to_mgr(struct drm_lru_entity *entity)
}
void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm,
- uint32_t mem_type, uint64_t size, uint32_t priority)
+ uint32_t mem_type, uint64_t size, uint32_t priority,
+ struct drm_lru_evict_func *evict_func)
{
entity->drm = drm;
entity->mem_type = mem_type;
entity->size = size;
entity->priority = priority;
INIT_LIST_HEAD(&entity->lru);
+ entity->evict_func = evict_func;
}
/**
@@ -230,3 +232,35 @@ void drm_lru_del_bulk_move(struct drm_lru_entity *entity,
}
}
EXPORT_SYMBOL(drm_lru_del_bulk_move);
+
+int drm_lru_evict_first(struct drm_lru_mgr *mgr,
+ const struct drm_lru_evict_ctx *evict_ctx)
+{
+ struct drm_lru_entity *entity, *busy_entity = NULL;
+ struct drm_lru_cursor cursor;
+ bool locked = false, busy = false, found = false;
+
+ spin_lock(mgr->lru_lock);
+
+ /* First find a victim to evict*/
+ drm_lru_for_each_entity(mgr, &cursor, entity) {
+ if (!entity->evict_func->evict_allowable(entity,
+ evict_ctx, &busy, &locked)) {
+ if (!busy_entity && busy)
+ busy_entity = entity;
+ continue;
+ }
+ found = true;
+ break;
+ }
+
+ /* We didn't find a victim, but we found a busy entity, i.e.,
+ * other clients hold a reservation lock of this entity. Try
+ * to wait and evict this busy entity.
+ */
+ if (!found && busy_entity)
+ return busy_entity->evict_func->evict_busy_entity(busy_entity, evict_ctx);
+
+ /* If here, we found a victim to evict*/
+ return entity->evict_func->evict_entity(entity, evict_ctx, locked);
+}
diff --git a/include/drm/drm_evictable_lru.h b/include/drm/drm_evictable_lru.h
index 3fd6bd2475d9..7f49964f2f9b 100644
--- a/include/drm/drm_evictable_lru.h
+++ b/include/drm/drm_evictable_lru.h
@@ -15,6 +15,12 @@ struct drm_device;
#define DRM_MAX_LRU_PRIORITY 4
#define DRM_NUM_MEM_TYPES 8
+struct drm_lru_evict_ctx {
+ void *data1;
+ void *data2;
+ void *data3;
+};
+
/**
* struct drm_lru_entity
*
@@ -23,6 +29,7 @@ struct drm_device;
* @size: resource size of this entity
* @priority: The priority of this entity
* @lru: least recent used list node, see &drm_lru_mgr.lru
+ * @evict_func: functions to evict this entity
*
* This structure represents an entity in drm_lru_mgr's
* list. This structure is supposed to be embedded in
@@ -34,6 +41,7 @@ struct drm_lru_entity {
uint64_t size;
uint32_t priority;
struct list_head lru;
+ struct drm_lru_evict_func *evict_func;
};
/**
@@ -97,7 +105,67 @@ struct drm_lru_bulk_move {
struct drm_lru_bulk_move_range range[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY];
};
+struct drm_lru_evict_func {
+ /**
+ * evict_allowable
+ *
+ * @lru_entity: The struct ttm_resource::lru_entity when this resource is
+ * added to drm lru list.
+ * @evict_ctx: eviction context. This is opaque data to drm lru layer. It is
+ * passed to drm lru layer through drm_lru_evict_first function and drm lru
+ * layer just pass it back to ttm or svm code by calling some ttm or svm
+ * callback functions.
+ * @busy: used to return whether the current resource is busy (i.e., locked
+ * by other clients)
+ * @locked: used to return whether this resource is locked during this check,
+ * i.e., successfully trylocked bo's dma reservation object
+ *
+ * Check whether we are allowed to evict a memory resource. Return true if we
+ * are allowed to evict resource; otherwise false.
+ *
+ * When this function returns true, a resource reference counter is hold. This
+ * reference counter need to be released after evict operation later on.
+ *
+ * This function should be called with lru_lock hold.
+ */
+ bool (*evict_allowable)(struct drm_lru_entity *lru_entity,
+ const struct drm_lru_evict_ctx *evict_ctx,
+ bool *busy, bool *locked);
+ /**
+ * evict_busy_entity
+ *
+ * @lru_entity: The struct ttm_resource::lru_entity when this resource is
+ * added to drm lru list.
+ * @evict_ctx: eviction context. This is opaque data to drm lru layer. It is
+ * passed to drm lru layer through drm_lru_evict_first function and drm lru
+ * layer just pass it back to ttm or svm code by calling some ttm or svm
+ * callback functions.
+ *
+ * Evict a busy memory resource.
+ * This function should be called with lru_lock hold.
+ */
+ int (*evict_busy_entity)(struct drm_lru_entity *lru_entity,
+ const struct drm_lru_evict_ctx *evict_ctx);
+
+ /**
+ * evict_entity
+ *
+ * @lru_entity: The struct ttm_resource::lru_entity when this resource is
+ * added to drm lru list.
+ * @evict_ctx: eviction context. This is opaque data to drm lru layer. It is
+ * passed to drm lru layer through drm_lru_evict_first function and drm lru
+ * layer just pass it back to ttm or svm code by calling some ttm or svm
+ * callback functions.
+ * @locked: whether this resource is dma-reserved (if reserved, we need to
+ * unreserve it in this function)
+ *
+ * Evict a memory resource corresponding to a lru_entity. This should be
+ * called holding lru_lock
+ */
+ int (*evict_entity)(struct drm_lru_entity *lru_entity,
+ const struct drm_lru_evict_ctx *evict_ctx, bool locked);
+};
/**
* drm_lru_add_entity
@@ -145,7 +213,8 @@ static inline void drm_lru_mgr_fini(struct drm_lru_mgr *mgr)
}
void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm,
- uint32_t mem_type, uint64_t size, uint32_t priority);
+ uint32_t mem_type, uint64_t size, uint32_t priority,
+ struct drm_lru_evict_func *evict_func);
struct drm_lru_entity *
drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor);
@@ -172,6 +241,9 @@ void drm_lru_add_bulk_move(struct drm_lru_entity *entity,
void drm_lru_del_bulk_move(struct drm_lru_entity *entity,
struct drm_lru_bulk_move *bulk_move);
+
+int drm_lru_evict_first(struct drm_lru_mgr *mgr,
+ const struct drm_lru_evict_ctx *evict_ctx);
/**
* drm_lru_for_each_entity
*
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 05/11] drm: Replace ttm macros with drm macros
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (3 preceding siblings ...)
2023-11-02 4:32 ` [RFC 04/11] drm: Add evict function pointer to drm lru entity Oak Zeng
@ 2023-11-02 4:33 ` Oak Zeng
2023-11-02 4:33 ` [RFC 06/11] drm/ttm: Set lru manager to ttm resource manager Oak Zeng
` (6 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:33 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
TTM_MAX_BO_PRIORITY and TTM_NUM_MEM_TYPES are move from ttm to
drm, so:
s/TTM_MAX_BO_PRIORITY/DRM_MAX_LRU_PRIORITY
s/TTM_NUM_MEM_TYPES/DRM_NUM_MEM_TYPES
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 2 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 2 +-
drivers/gpu/drm/i915/selftests/mock_region.c | 2 +-
drivers/gpu/drm/ttm/ttm_device.c | 8 ++++----
drivers/gpu/drm/ttm/ttm_resource.c | 12 ++++++------
drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 2 +-
drivers/gpu/drm/vmwgfx/vmwgfx_bo.h | 2 +-
drivers/gpu/drm/xe/xe_bo.h | 2 +-
include/drm/ttm/ttm_device.h | 6 +++---
include/drm/ttm/ttm_range_manager.h | 4 ++--
include/drm/ttm/ttm_resource.h | 10 ++++------
11 files changed, 25 insertions(+), 27 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
index c46f54f83f54..228dbea60949 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -987,7 +987,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
spin_lock(bo->bdev->lru_lock);
if (shrinkable) {
/* Try to keep shmem_tt from being considered for shrinking. */
- bo->priority = TTM_MAX_BO_PRIORITY - 1;
+ bo->priority = DRM_MAX_LRU_PRIORITY - 1;
} else if (obj->mm.madv != I915_MADV_WILLNEED) {
bo->priority = I915_TTM_PRIO_PURGE;
} else if (!i915_gem_object_has_pages(obj)) {
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
index b845782c9859..f75520c2ba59 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.c
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -63,7 +63,7 @@ int intel_region_to_ttm_type(const struct intel_memory_region *mem)
return TTM_PL_SYSTEM;
type = mem->instance + TTM_PL_PRIV;
- GEM_BUG_ON(type >= TTM_NUM_MEM_TYPES);
+ GEM_BUG_ON(type >= DRM_NUM_MEM_TYPES);
return type;
}
diff --git a/drivers/gpu/drm/i915/selftests/mock_region.c b/drivers/gpu/drm/i915/selftests/mock_region.c
index 6324eb32d4dd..6ea0e6bec812 100644
--- a/drivers/gpu/drm/i915/selftests/mock_region.c
+++ b/drivers/gpu/drm/i915/selftests/mock_region.c
@@ -111,7 +111,7 @@ mock_region_create(struct drm_i915_private *i915,
resource_size_t io_size)
{
int instance = ida_alloc_max(&i915->selftest.mock_region_instances,
- TTM_NUM_MEM_TYPES - TTM_PL_PRIV - 1,
+ DRM_NUM_MEM_TYPES - TTM_PL_PRIV - 1,
GFP_KERNEL);
if (instance < 0)
diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c
index d18eca86ebd6..e8c8006ba748 100644
--- a/drivers/gpu/drm/ttm/ttm_device.c
+++ b/drivers/gpu/drm/ttm/ttm_device.c
@@ -148,7 +148,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
int ret;
spin_lock(bdev->lru_lock);
- for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) {
+ for (i = TTM_PL_SYSTEM; i < DRM_NUM_MEM_TYPES; ++i) {
man = ttm_manager_type(bdev, i);
if (!man || !man->use_tt)
continue;
@@ -245,7 +245,7 @@ void ttm_device_fini(struct ttm_device *bdev)
destroy_workqueue(bdev->wq);
spin_lock(bdev->lru_lock);
- for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i)
+ for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i)
if (list_empty(&man->lru[0]))
pr_debug("Swap list %d was clean\n", i);
spin_unlock(bdev->lru_lock);
@@ -287,12 +287,12 @@ void ttm_device_clear_dma_mappings(struct ttm_device *bdev)
ttm_device_clear_lru_dma_mappings(bdev, &bdev->pinned);
- for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) {
+ for (i = TTM_PL_SYSTEM; i < DRM_NUM_MEM_TYPES; ++i) {
man = ttm_manager_type(bdev, i);
if (!man || !man->use_tt)
continue;
- for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j)
+ for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j)
ttm_device_clear_lru_dma_mappings(bdev, &man->lru[j]);
}
}
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 6ada77f51fba..05eef866065e 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -54,8 +54,8 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
{
unsigned i, j;
- for (i = 0; i < TTM_NUM_MEM_TYPES; ++i) {
- for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) {
+ for (i = 0; i < DRM_NUM_MEM_TYPES; ++i) {
+ for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) {
struct ttm_lru_bulk_move_pos *pos = &bulk->pos[i][j];
struct ttm_resource_manager *man;
@@ -393,7 +393,7 @@ void ttm_resource_manager_init(struct ttm_resource_manager *man,
man->size = size;
man->usage = 0;
- for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i)
+ for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i)
INIT_LIST_HEAD(&man->lru[i]);
man->move = NULL;
}
@@ -425,7 +425,7 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
*/
spin_lock(bdev->lru_lock);
- for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) {
+ for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i) {
while (!list_empty(&man->lru[i])) {
spin_unlock(bdev->lru_lock);
ret = ttm_mem_evict_first(bdev, man, NULL, &ctx,
@@ -504,7 +504,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man,
lockdep_assert_held(man->bdev->lru_lock);
- for (cursor->priority = 0; cursor->priority < TTM_MAX_BO_PRIORITY;
+ for (cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY;
++cursor->priority)
list_for_each_entry(res, &man->lru[cursor->priority], lru)
return res;
@@ -531,7 +531,7 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,
list_for_each_entry_continue(res, &man->lru[cursor->priority], lru)
return res;
- for (++cursor->priority; cursor->priority < TTM_MAX_BO_PRIORITY;
+ for (++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY;
++cursor->priority)
list_for_each_entry(res, &man->lru[cursor->priority], lru)
return res;
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
index c43853597776..9efde9ba9fe2 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
@@ -383,7 +383,7 @@ static int vmw_bo_init(struct vmw_private *dev_priv,
memset(vmw_bo, 0, sizeof(*vmw_bo));
- BUILD_BUG_ON(TTM_MAX_BO_PRIORITY <= 3);
+ BUILD_BUG_ON(DRM_MAX_LRU_PRIORITY <= 3);
vmw_bo->tbo.priority = 3;
vmw_bo->res_tree = RB_ROOT;
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.h b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.h
index 1d433fceed3d..c43a82c8afb9 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.h
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.h
@@ -82,7 +82,7 @@ struct vmw_bo {
struct ttm_bo_kmap_obj map;
struct rb_root res_tree;
- u32 res_prios[TTM_MAX_BO_PRIORITY];
+ u32 res_prios[DRM_MAX_LRU_PRIORITY];
atomic_t cpu_writers;
/* Not ref-counted. Protected by binding_mutex */
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 9097bcc13209..9918b2d630e1 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -72,7 +72,7 @@
#define XE_PL_TT TTM_PL_TT
#define XE_PL_VRAM0 TTM_PL_VRAM
#define XE_PL_VRAM1 (XE_PL_VRAM0 + 1)
-#define XE_PL_STOLEN (TTM_NUM_MEM_TYPES - 1)
+#define XE_PL_STOLEN (DRM_NUM_MEM_TYPES - 1)
#define XE_BO_PROPS_INVALID (-1)
diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h
index 4d29e96bd892..a75e820dc671 100644
--- a/include/drm/ttm/ttm_device.h
+++ b/include/drm/ttm/ttm_device.h
@@ -235,7 +235,7 @@ struct ttm_device {
/**
* @man_drv: An array of resource_managers, one per resource type.
*/
- struct ttm_resource_manager *man_drv[TTM_NUM_MEM_TYPES];
+ struct ttm_resource_manager *man_drv[DRM_NUM_MEM_TYPES];
/**
* @vma_manager: Address space manager for finding BOs to mmap.
@@ -277,14 +277,14 @@ static inline struct ttm_resource_manager *
ttm_manager_type(struct ttm_device *bdev, int mem_type)
{
BUILD_BUG_ON(__builtin_constant_p(mem_type)
- && mem_type >= TTM_NUM_MEM_TYPES);
+ && mem_type >= DRM_NUM_MEM_TYPES);
return bdev->man_drv[mem_type];
}
static inline void ttm_set_driver_manager(struct ttm_device *bdev, int type,
struct ttm_resource_manager *manager)
{
- BUILD_BUG_ON(__builtin_constant_p(type) && type >= TTM_NUM_MEM_TYPES);
+ BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES);
bdev->man_drv[type] = manager;
}
diff --git a/include/drm/ttm/ttm_range_manager.h b/include/drm/ttm/ttm_range_manager.h
index 7963b957e9ef..becdb88c4d84 100644
--- a/include/drm/ttm/ttm_range_manager.h
+++ b/include/drm/ttm/ttm_range_manager.h
@@ -43,14 +43,14 @@ static __always_inline int ttm_range_man_init(struct ttm_device *bdev,
unsigned int type, bool use_tt,
unsigned long p_size)
{
- BUILD_BUG_ON(__builtin_constant_p(type) && type >= TTM_NUM_MEM_TYPES);
+ BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES);
return ttm_range_man_init_nocheck(bdev, type, use_tt, p_size);
}
static __always_inline int ttm_range_man_fini(struct ttm_device *bdev,
unsigned int type)
{
- BUILD_BUG_ON(__builtin_constant_p(type) && type >= TTM_NUM_MEM_TYPES);
+ BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES);
return ttm_range_man_fini_nocheck(bdev, type);
}
#endif
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index 78a226eba953..92241c2374fa 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -34,9 +34,7 @@
#include <drm/drm_print.h>
#include <drm/ttm/ttm_caching.h>
#include <drm/ttm/ttm_kmap_iter.h>
-
-#define TTM_MAX_BO_PRIORITY 4U
-#define TTM_NUM_MEM_TYPES 8
+#include <drm/drm_evictable_lru.h>
struct ttm_device;
struct ttm_resource_manager;
@@ -167,7 +165,7 @@ struct ttm_resource_manager {
/*
* Protected by the bdev->lru_lock.
*/
- struct list_head lru[TTM_MAX_BO_PRIORITY];
+ struct list_head lru[DRM_MAX_LRU_PRIORITY];
/**
* @usage: How much of the resources are used, protected by the
@@ -253,7 +251,7 @@ struct ttm_lru_bulk_move_pos {
* ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move().
*/
struct ttm_lru_bulk_move {
- struct ttm_lru_bulk_move_pos pos[TTM_NUM_MEM_TYPES][TTM_MAX_BO_PRIORITY];
+ struct ttm_lru_bulk_move_pos pos[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY];
};
/**
@@ -309,7 +307,7 @@ ttm_resource_manager_set_used(struct ttm_resource_manager *man, bool used)
{
int i;
- for (i = 0; i < TTM_MAX_BO_PRIORITY; i++)
+ for (i = 0; i < DRM_MAX_LRU_PRIORITY; i++)
WARN_ON(!list_empty(&man->lru[i]));
man->use_type = used;
}
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 06/11] drm/ttm: Set lru manager to ttm resource manager
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (4 preceding siblings ...)
2023-11-02 4:33 ` [RFC 05/11] drm: Replace ttm macros with drm macros Oak Zeng
@ 2023-11-02 4:33 ` Oak Zeng
2023-11-02 4:33 ` [RFC 07/11] drm/ttm: re-parameterize a few ttm functions Oak Zeng
` (5 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:33 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
Add a weak reference of lru manager to ttm resource manager,
and add a function to set lru manager for ttm resource manager.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
include/drm/ttm/ttm_resource.h | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index 92241c2374fa..e4fc1ada5236 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -46,6 +46,7 @@ struct iosys_map;
struct io_mapping;
struct sg_table;
struct scatterlist;
+struct drm_lru_mgr;
struct ttm_resource_manager_func {
/**
@@ -172,6 +173,12 @@ struct ttm_resource_manager {
* bdev->lru_lock.
*/
uint64_t usage;
+
+ /**
+ * @lru_mgr: weak reference of the lru manager that manages lru
+ * list for this ttm resource manager.
+ */
+ struct drm_lru_mgr *lru_mgr;
};
/**
@@ -326,6 +333,18 @@ static inline bool ttm_resource_manager_used(struct ttm_resource_manager *man)
return man->use_type;
}
+/**
+ * ttm_resource_manager_set_lru_mgr
+ *
+ * @man: ttm resource manager
+ * @mgr: pointing to lru manager
+ */
+static inline void
+ttm_resource_manager_set_lru_mgr(struct ttm_resource_manager *man, struct drm_lru_mgr *mgr)
+{
+ man->lru_mgr = mgr;
+}
+
/**
* ttm_resource_manager_cleanup
*
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 07/11] drm/ttm: re-parameterize a few ttm functions
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (5 preceding siblings ...)
2023-11-02 4:33 ` [RFC 06/11] drm/ttm: Set lru manager to ttm resource manager Oak Zeng
@ 2023-11-02 4:33 ` Oak Zeng
2023-11-02 4:33 ` [RFC 08/11] drm: Initialize drm lru manager Oak Zeng
` (4 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:33 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
Add a struct drm_device *drm parameter to function
ttm_range_man_init, ttm_range_man_fini, ttm_sys_man_init,
and ttm_sys_man_free. This drm parameter will be used
in the coming patches to retrieve and initialize drm
lru manager.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 8 ++++----
drivers/gpu/drm/drm_gem_vram_helper.c | 8 ++++----
drivers/gpu/drm/loongson/lsdc_ttm.c | 8 ++++----
drivers/gpu/drm/nouveau/nouveau_ttm.c | 8 ++++----
drivers/gpu/drm/qxl/qxl_ttm.c | 6 +++---
drivers/gpu/drm/radeon/radeon_ttm.c | 8 ++++----
drivers/gpu/drm/ttm/ttm_device.c | 2 +-
drivers/gpu/drm/ttm/ttm_module.h | 3 ++-
drivers/gpu/drm/ttm/ttm_range_manager.c | 6 ++++--
drivers/gpu/drm/ttm/ttm_sys_manager.c | 6 +++++-
drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 4 ++--
include/drm/ttm/ttm_range_manager.h | 13 +++++++------
12 files changed, 44 insertions(+), 36 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 5cdbc901cbe2..cc0736f82a80 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -75,7 +75,7 @@ static int amdgpu_ttm_init_on_chip(struct amdgpu_device *adev,
unsigned int type,
uint64_t size_in_page)
{
- return ttm_range_man_init(&adev->mman.bdev, type,
+ return ttm_range_man_init(adev_to_drm(adev), &adev->mman.bdev, type,
false, size_in_page);
}
@@ -2026,9 +2026,9 @@ void amdgpu_ttm_fini(struct amdgpu_device *adev)
amdgpu_vram_mgr_fini(adev);
amdgpu_gtt_mgr_fini(adev);
amdgpu_preempt_mgr_fini(adev);
- ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_GDS);
- ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_GWS);
- ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_OA);
+ ttm_range_man_fini(adev_to_drm(adev), &adev->mman.bdev, AMDGPU_PL_GDS);
+ ttm_range_man_fini(adev_to_drm(adev), &adev->mman.bdev, AMDGPU_PL_GWS);
+ ttm_range_man_fini(adev_to_drm(adev), &adev->mman.bdev, AMDGPU_PL_OA);
ttm_device_fini(&adev->mman.bdev);
adev->mman.initialized = false;
DRM_INFO("amdgpu: ttm finalized\n");
diff --git a/drivers/gpu/drm/drm_gem_vram_helper.c b/drivers/gpu/drm/drm_gem_vram_helper.c
index 56749e40459f..5b18db72cc96 100644
--- a/drivers/gpu/drm/drm_gem_vram_helper.c
+++ b/drivers/gpu/drm/drm_gem_vram_helper.c
@@ -1009,7 +1009,7 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev,
if (ret)
return ret;
- ret = ttm_range_man_init(&vmm->bdev, TTM_PL_VRAM,
+ ret = ttm_range_man_init(dev, &vmm->bdev, TTM_PL_VRAM,
false, vram_size >> PAGE_SHIFT);
if (ret)
return ret;
@@ -1017,9 +1017,9 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev,
return 0;
}
-static void drm_vram_mm_cleanup(struct drm_vram_mm *vmm)
+static void drm_vram_mm_cleanup(struct drm_device *drm, struct drm_vram_mm *vmm)
{
- ttm_range_man_fini(&vmm->bdev, TTM_PL_VRAM);
+ ttm_range_man_fini(drm, &vmm->bdev, TTM_PL_VRAM);
ttm_device_fini(&vmm->bdev);
}
@@ -1056,7 +1056,7 @@ static void drm_vram_helper_release_mm(struct drm_device *dev)
if (!dev->vram_mm)
return;
- drm_vram_mm_cleanup(dev->vram_mm);
+ drm_vram_mm_cleanup(dev, dev->vram_mm);
kfree(dev->vram_mm);
dev->vram_mm = NULL;
}
diff --git a/drivers/gpu/drm/loongson/lsdc_ttm.c b/drivers/gpu/drm/loongson/lsdc_ttm.c
index bd68cb9366b5..f7f226314a09 100644
--- a/drivers/gpu/drm/loongson/lsdc_ttm.c
+++ b/drivers/gpu/drm/loongson/lsdc_ttm.c
@@ -533,8 +533,8 @@ static void lsdc_ttm_fini(struct drm_device *ddev, void *data)
{
struct lsdc_device *ldev = (struct lsdc_device *)data;
- ttm_range_man_fini(&ldev->bdev, TTM_PL_VRAM);
- ttm_range_man_fini(&ldev->bdev, TTM_PL_TT);
+ ttm_range_man_fini(ddev, &ldev->bdev, TTM_PL_VRAM);
+ ttm_range_man_fini(ddev, &ldev->bdev, TTM_PL_TT);
ttm_device_fini(&ldev->bdev);
@@ -556,7 +556,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev)
num_vram_pages = ldev->vram_size >> PAGE_SHIFT;
- ret = ttm_range_man_init(&ldev->bdev, TTM_PL_VRAM, false, num_vram_pages);
+ ret = ttm_range_man_init(&ldev->base, &ldev->bdev, TTM_PL_VRAM, false, num_vram_pages);
if (unlikely(ret))
return ret;
@@ -567,7 +567,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev)
num_gtt_pages = ldev->gtt_size >> PAGE_SHIFT;
- ret = ttm_range_man_init(&ldev->bdev, TTM_PL_TT, true, num_gtt_pages);
+ ret = ttm_range_man_init(&ldev->base, &ldev->bdev, TTM_PL_TT, true, num_gtt_pages);
if (unlikely(ret))
return ret;
diff --git a/drivers/gpu/drm/nouveau/nouveau_ttm.c b/drivers/gpu/drm/nouveau/nouveau_ttm.c
index 831918437850..1898f27f0510 100644
--- a/drivers/gpu/drm/nouveau/nouveau_ttm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_ttm.c
@@ -194,7 +194,7 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm)
ttm_resource_manager_set_used(man, true);
return 0;
} else {
- return ttm_range_man_init(&drm->ttm.bdev, TTM_PL_VRAM, false,
+ return ttm_range_man_init(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM, false,
drm->gem.vram_available >> PAGE_SHIFT);
}
}
@@ -211,7 +211,7 @@ nouveau_ttm_fini_vram(struct nouveau_drm *drm)
ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_VRAM, NULL);
kfree(man);
} else
- ttm_range_man_fini(&drm->ttm.bdev, TTM_PL_VRAM);
+ ttm_range_man_fini(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM);
}
static int
@@ -226,7 +226,7 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm)
else if (!drm->agp.bridge)
func = &nv04_gart_manager;
else
- return ttm_range_man_init(&drm->ttm.bdev, TTM_PL_TT, true,
+ return ttm_range_man_init(drm->dev, &drm->ttm.bdev, TTM_PL_TT, true,
size_pages);
man = kzalloc(sizeof(*man), GFP_KERNEL);
@@ -248,7 +248,7 @@ nouveau_ttm_fini_gtt(struct nouveau_drm *drm)
if (drm->client.device.info.family < NV_DEVICE_INFO_V0_TESLA &&
drm->agp.bridge)
- ttm_range_man_fini(&drm->ttm.bdev, TTM_PL_TT);
+ ttm_range_man_fini(drm->dev, &drm->ttm.bdev, TTM_PL_TT);
else {
ttm_resource_manager_set_used(man, false);
ttm_resource_manager_evict_all(&drm->ttm.bdev, man);
diff --git a/drivers/gpu/drm/qxl/qxl_ttm.c b/drivers/gpu/drm/qxl/qxl_ttm.c
index 1a82629bce3f..9f984afca831 100644
--- a/drivers/gpu/drm/qxl/qxl_ttm.c
+++ b/drivers/gpu/drm/qxl/qxl_ttm.c
@@ -186,7 +186,7 @@ static int qxl_ttm_init_mem_type(struct qxl_device *qdev,
unsigned int type,
uint64_t size)
{
- return ttm_range_man_init(&qdev->mman.bdev, type, false, size);
+ return ttm_range_man_init(&qdev->ddev, &qdev->mman.bdev, type, false, size);
}
int qxl_ttm_init(struct qxl_device *qdev)
@@ -227,8 +227,8 @@ int qxl_ttm_init(struct qxl_device *qdev)
void qxl_ttm_fini(struct qxl_device *qdev)
{
- ttm_range_man_fini(&qdev->mman.bdev, TTM_PL_VRAM);
- ttm_range_man_fini(&qdev->mman.bdev, TTM_PL_PRIV);
+ ttm_range_man_fini(&qdev->ddev, &qdev->mman.bdev, TTM_PL_VRAM);
+ ttm_range_man_fini(&qdev->ddev, &qdev->mman.bdev, TTM_PL_PRIV);
ttm_device_fini(&qdev->mman.bdev);
DRM_INFO("qxl: ttm finalized\n");
}
diff --git a/drivers/gpu/drm/radeon/radeon_ttm.c b/drivers/gpu/drm/radeon/radeon_ttm.c
index 77ca50187162..5ab3f229082e 100644
--- a/drivers/gpu/drm/radeon/radeon_ttm.c
+++ b/drivers/gpu/drm/radeon/radeon_ttm.c
@@ -68,13 +68,13 @@ struct radeon_device *radeon_get_rdev(struct ttm_device *bdev)
static int radeon_ttm_init_vram(struct radeon_device *rdev)
{
- return ttm_range_man_init(&rdev->mman.bdev, TTM_PL_VRAM,
+ return ttm_range_man_init(rdev->ddev, &rdev->mman.bdev, TTM_PL_VRAM,
false, rdev->mc.real_vram_size >> PAGE_SHIFT);
}
static int radeon_ttm_init_gtt(struct radeon_device *rdev)
{
- return ttm_range_man_init(&rdev->mman.bdev, TTM_PL_TT,
+ return ttm_range_man_init(rdev->ddev, &rdev->mman.bdev, TTM_PL_TT,
true, rdev->mc.gtt_size >> PAGE_SHIFT);
}
@@ -753,8 +753,8 @@ void radeon_ttm_fini(struct radeon_device *rdev)
}
radeon_bo_unref(&rdev->stolen_vga_memory);
}
- ttm_range_man_fini(&rdev->mman.bdev, TTM_PL_VRAM);
- ttm_range_man_fini(&rdev->mman.bdev, TTM_PL_TT);
+ ttm_range_man_fini(rdev->ddev, &rdev->mman.bdev, TTM_PL_VRAM);
+ ttm_range_man_fini(rdev->ddev, &rdev->mman.bdev, TTM_PL_TT);
ttm_device_fini(&rdev->mman.bdev);
radeon_gart_fini(rdev);
rdev->mman.initialized = false;
diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c
index e8c8006ba748..393c3e27016e 100644
--- a/drivers/gpu/drm/ttm/ttm_device.c
+++ b/drivers/gpu/drm/ttm/ttm_device.c
@@ -212,7 +212,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func
bdev->funcs = funcs;
- ttm_sys_man_init(bdev);
+ ttm_sys_man_init(drm, bdev);
ttm_pool_init(&bdev->pool, drm?drm->dev:NULL, NUMA_NO_NODE,
use_dma_alloc, use_dma32);
diff --git a/drivers/gpu/drm/ttm/ttm_module.h b/drivers/gpu/drm/ttm/ttm_module.h
index 767fe22aed48..6c1443704a35 100644
--- a/drivers/gpu/drm/ttm/ttm_module.h
+++ b/drivers/gpu/drm/ttm/ttm_module.h
@@ -35,9 +35,10 @@
struct dentry;
struct ttm_device;
+struct drm_device;
extern struct dentry *ttm_debugfs_root;
-void ttm_sys_man_init(struct ttm_device *bdev);
+void ttm_sys_man_init(struct drm_device *drm, struct ttm_device *bdev);
#endif /* _TTM_MODULE_H_ */
diff --git a/drivers/gpu/drm/ttm/ttm_range_manager.c b/drivers/gpu/drm/ttm/ttm_range_manager.c
index ae11d07eb63a..db1ae370580d 100644
--- a/drivers/gpu/drm/ttm/ttm_range_manager.c
+++ b/drivers/gpu/drm/ttm/ttm_range_manager.c
@@ -166,6 +166,7 @@ static const struct ttm_resource_manager_func ttm_range_manager_func = {
* ttm_range_man_init_nocheck - Initialise a generic range manager for the
* selected memory type.
*
+ * @drm: drm device
* @bdev: ttm device
* @type: memory manager type
* @use_tt: if the memory manager uses tt
@@ -175,7 +176,7 @@ static const struct ttm_resource_manager_func ttm_range_manager_func = {
*
* Return: %0 on success or a negative error code on failure
*/
-int ttm_range_man_init_nocheck(struct ttm_device *bdev,
+int ttm_range_man_init_nocheck(struct drm_device *drm, struct ttm_device *bdev,
unsigned type, bool use_tt,
unsigned long p_size)
{
@@ -206,12 +207,13 @@ EXPORT_SYMBOL(ttm_range_man_init_nocheck);
* ttm_range_man_fini_nocheck - Remove the generic range manager from a slot
* and tear it down.
*
+ * @drm: drm device
* @bdev: ttm device
* @type: memory manager type
*
* Return: %0 on success or a negative error code on failure
*/
-int ttm_range_man_fini_nocheck(struct ttm_device *bdev,
+int ttm_range_man_fini_nocheck(struct drm_device *drm, struct ttm_device *bdev,
unsigned type)
{
struct ttm_resource_manager *man = ttm_manager_type(bdev, type);
diff --git a/drivers/gpu/drm/ttm/ttm_sys_manager.c b/drivers/gpu/drm/ttm/ttm_sys_manager.c
index 2ced169513cb..f0f026d40a69 100644
--- a/drivers/gpu/drm/ttm/ttm_sys_manager.c
+++ b/drivers/gpu/drm/ttm/ttm_sys_manager.c
@@ -20,6 +20,10 @@ static int ttm_sys_man_alloc(struct ttm_resource_manager *man,
return 0;
}
+/* FIXME: Need to call drm_lru_mgr_fini. This requires a
+ * struct drm_device *drm parameter which need the change
+ * the definition of struct ttm_resource_manager_func. A
+ * very intrusive change. Leave it for now */
static void ttm_sys_man_free(struct ttm_resource_manager *man,
struct ttm_resource *res)
{
@@ -32,7 +36,7 @@ static const struct ttm_resource_manager_func ttm_sys_manager_func = {
.free = ttm_sys_man_free,
};
-void ttm_sys_man_init(struct ttm_device *bdev)
+void ttm_sys_man_init(struct drm_device *drm, struct ttm_device *bdev)
{
struct ttm_resource_manager *man = &bdev->sysman;
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
index cf1c1f16102a..1a0c161e9977 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
@@ -708,7 +708,7 @@ static int vmw_dma_masks(struct vmw_private *dev_priv)
static int vmw_vram_manager_init(struct vmw_private *dev_priv)
{
int ret;
- ret = ttm_range_man_init(&dev_priv->bdev, TTM_PL_VRAM, false,
+ ret = ttm_range_man_init(&dev_priv->drm, &dev_priv->bdev, TTM_PL_VRAM, false,
dev_priv->vram_size >> PAGE_SHIFT);
ttm_resource_manager_set_used(ttm_manager_type(&dev_priv->bdev, TTM_PL_VRAM), false);
return ret;
@@ -716,7 +716,7 @@ static int vmw_vram_manager_init(struct vmw_private *dev_priv)
static void vmw_vram_manager_fini(struct vmw_private *dev_priv)
{
- ttm_range_man_fini(&dev_priv->bdev, TTM_PL_VRAM);
+ ttm_range_man_fini(&dev_priv->drm, &dev_priv->bdev, TTM_PL_VRAM);
}
static int vmw_setup_pci_resources(struct vmw_private *dev,
diff --git a/include/drm/ttm/ttm_range_manager.h b/include/drm/ttm/ttm_range_manager.h
index becdb88c4d84..33cb5016bde6 100644
--- a/include/drm/ttm/ttm_range_manager.h
+++ b/include/drm/ttm/ttm_range_manager.h
@@ -34,23 +34,24 @@ to_ttm_range_mgr_node(struct ttm_resource *res)
return container_of(res, struct ttm_range_mgr_node, base);
}
-int ttm_range_man_init_nocheck(struct ttm_device *bdev,
+int ttm_range_man_init_nocheck(struct drm_device *drm, struct ttm_device *bdev,
unsigned type, bool use_tt,
unsigned long p_size);
-int ttm_range_man_fini_nocheck(struct ttm_device *bdev,
+int ttm_range_man_fini_nocheck(struct drm_device *drm, struct ttm_device *bdev,
unsigned type);
-static __always_inline int ttm_range_man_init(struct ttm_device *bdev,
+static __always_inline int ttm_range_man_init(struct drm_device *drm,
+ struct ttm_device *bdev,
unsigned int type, bool use_tt,
unsigned long p_size)
{
BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES);
- return ttm_range_man_init_nocheck(bdev, type, use_tt, p_size);
+ return ttm_range_man_init_nocheck(drm, bdev, type, use_tt, p_size);
}
-static __always_inline int ttm_range_man_fini(struct ttm_device *bdev,
+static __always_inline int ttm_range_man_fini(struct drm_device *drm, struct ttm_device *bdev,
unsigned int type)
{
BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES);
- return ttm_range_man_fini_nocheck(bdev, type);
+ return ttm_range_man_fini_nocheck(drm, bdev, type);
}
#endif
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 08/11] drm: Initialize drm lru manager
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (6 preceding siblings ...)
2023-11-02 4:33 ` [RFC 07/11] drm/ttm: re-parameterize a few ttm functions Oak Zeng
@ 2023-11-02 4:33 ` Oak Zeng
2023-11-02 4:33 ` [RFC 09/11] drm/ttm: Use drm LRU manager iterator Oak Zeng
` (3 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:33 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
Initialize lru_mgr for each memory type or memory region. Also set
ttm_resource_manager's weak reference to drm lru manager.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 ++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c | 6 ++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 ++++++
drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 10 ++++++++++
drivers/gpu/drm/nouveau/nouveau_ttm.c | 12 ++++++++++++
drivers/gpu/drm/ttm/ttm_range_manager.c | 6 ++++++
drivers/gpu/drm/ttm/ttm_sys_manager.c | 2 ++
drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c | 6 ++++++
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 6 ++++++
drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 6 ++++++
10 files changed, 66 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c
index 44367f03316f..57e8b1688977 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c
@@ -278,6 +278,7 @@ int amdgpu_gtt_mgr_init(struct amdgpu_device *adev, uint64_t gtt_size)
{
struct amdgpu_gtt_mgr *mgr = &adev->mman.gtt_mgr;
struct ttm_resource_manager *man = &mgr->manager;
+ struct drm_device *drm = adev_to_drm(adev);
uint64_t start, size;
man->use_tt = true;
@@ -292,6 +293,9 @@ int amdgpu_gtt_mgr_init(struct amdgpu_device *adev, uint64_t gtt_size)
ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_TT, &mgr->manager);
ttm_resource_manager_set_used(man, true);
+
+ drm_lru_mgr_init(&drm->lru_mgr[TTM_PL_TT], gtt_size, &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[TTM_PL_TT]);
return 0;
}
@@ -307,6 +311,7 @@ void amdgpu_gtt_mgr_fini(struct amdgpu_device *adev)
{
struct amdgpu_gtt_mgr *mgr = &adev->mman.gtt_mgr;
struct ttm_resource_manager *man = &mgr->manager;
+ struct drm_device *drm = adev_to_drm(adev);
int ret;
ttm_resource_manager_set_used(man, false);
@@ -321,4 +326,5 @@ void amdgpu_gtt_mgr_fini(struct amdgpu_device *adev)
ttm_resource_manager_cleanup(man);
ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_TT, NULL);
+ drm_lru_mgr_fini(&drm->lru_mgr[TTM_PL_TT]);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c
index e8adfd0a570a..f989aca2bfc4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c
@@ -100,6 +100,7 @@ static const struct ttm_resource_manager_func amdgpu_preempt_mgr_func = {
int amdgpu_preempt_mgr_init(struct amdgpu_device *adev)
{
struct ttm_resource_manager *man = &adev->mman.preempt_mgr;
+ struct drm_device *drm = adev_to_drm(adev);
int ret;
man->use_tt = true;
@@ -115,6 +116,9 @@ int amdgpu_preempt_mgr_init(struct amdgpu_device *adev)
ttm_set_driver_manager(&adev->mman.bdev, AMDGPU_PL_PREEMPT, man);
ttm_resource_manager_set_used(man, true);
+
+ drm_lru_mgr_init(&drm->lru_mgr[AMDGPU_PL_PREEMPT], (1 << 30), &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[VMW_PL_SYSTEM]);
return 0;
}
@@ -129,6 +133,7 @@ int amdgpu_preempt_mgr_init(struct amdgpu_device *adev)
void amdgpu_preempt_mgr_fini(struct amdgpu_device *adev)
{
struct ttm_resource_manager *man = &adev->mman.preempt_mgr;
+ struct drm_device *drm = adev_to_drm(adev);
int ret;
ttm_resource_manager_set_used(man, false);
@@ -141,4 +146,5 @@ void amdgpu_preempt_mgr_fini(struct amdgpu_device *adev)
ttm_resource_manager_cleanup(man);
ttm_set_driver_manager(&adev->mman.bdev, AMDGPU_PL_PREEMPT, NULL);
+ drm_lru_mgr_fini(&drm->lru_mgr[AMDGPU_PL_PREEMPT]);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index b83e1741905e..0792d22508c9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
@@ -884,6 +884,7 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev)
{
struct amdgpu_vram_mgr *mgr = &adev->mman.vram_mgr;
struct ttm_resource_manager *man = &mgr->manager;
+ struct drm_device *drm = adev_to_drm(adev);
int err;
ttm_resource_manager_init(man, &adev->mman.bdev,
@@ -907,6 +908,9 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev)
ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_VRAM, &mgr->manager);
ttm_resource_manager_set_used(man, true);
+ drm_lru_mgr_init(&drm->lru_mgr[TTM_PL_VRAM], adev->gmc.real_vram_size,
+ &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[TTM_PL_VRAM]);
return 0;
}
@@ -922,6 +926,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev)
{
struct amdgpu_vram_mgr *mgr = &adev->mman.vram_mgr;
struct ttm_resource_manager *man = &mgr->manager;
+ struct drm_device *drm = adev_to_drm(adev);
int ret;
struct amdgpu_vram_reservation *rsv, *temp;
@@ -945,4 +950,5 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev)
ttm_resource_manager_cleanup(man);
ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_VRAM, NULL);
+ drm_lru_mgr_fini(&drm->lru_mgr[TTM_PL_VRAM]);
}
diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
index a1bc804cfa15..968f9c01152e 100644
--- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
+++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
@@ -304,6 +304,9 @@ int i915_ttm_buddy_man_init(struct ttm_device *bdev,
{
struct ttm_resource_manager *man;
struct i915_ttm_buddy_manager *bman;
+ struct drm_i915_private *i915 = container_of(bdev,
+ struct drm_i915_private, bdev);
+ struct drm_device *drm = &i915->drm;
int err;
bman = kzalloc(sizeof(*bman), GFP_KERNEL);
@@ -329,6 +332,9 @@ int i915_ttm_buddy_man_init(struct ttm_device *bdev,
ttm_resource_manager_set_used(man, true);
ttm_set_driver_manager(bdev, type, man);
+ drm_lru_mgr_init(&drm->lru_mgr[type], bman->mm.size >> PAGE_SHIFT, &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[type]);
+
return 0;
err_free_bman:
@@ -350,6 +356,9 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type)
{
struct ttm_resource_manager *man = ttm_manager_type(bdev, type);
struct i915_ttm_buddy_manager *bman = to_buddy_manager(man);
+ struct drm_i915_private *i915 = container_of(bdev,
+ struct drm_i915_private, bdev);
+ struct drm_device *drm = &i915->drm;
struct drm_buddy *mm = &bman->mm;
int ret;
@@ -369,6 +378,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type)
mutex_unlock(&bman->lock);
ttm_resource_manager_cleanup(man);
+ drm_lru_mgr_fini(&drm->lru_mgr[type]);
kfree(bman);
return 0;
diff --git a/drivers/gpu/drm/nouveau/nouveau_ttm.c b/drivers/gpu/drm/nouveau/nouveau_ttm.c
index 1898f27f0510..b0936c235ff6 100644
--- a/drivers/gpu/drm/nouveau/nouveau_ttm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_ttm.c
@@ -182,6 +182,7 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm)
{
if (drm->client.device.info.family >= NV_DEVICE_INFO_V0_TESLA) {
struct ttm_resource_manager *man = kzalloc(sizeof(*man), GFP_KERNEL);
+ struct drm_device *drm_dev = drm->dev;
if (!man)
return -ENOMEM;
@@ -192,6 +193,9 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm)
drm->gem.vram_available >> PAGE_SHIFT);
ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_VRAM, man);
ttm_resource_manager_set_used(man, true);
+ drm_lru_mgr_init(&drm_dev->lru_mgr[TTM_PL_VRAM],
+ drm->gem.vram_available >> PAGE_SHIFT, &drm_dev->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm_dev->lru_mgr[TTM_PL_VRAM]);
return 0;
} else {
return ttm_range_man_init(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM, false,
@@ -205,10 +209,13 @@ nouveau_ttm_fini_vram(struct nouveau_drm *drm)
struct ttm_resource_manager *man = ttm_manager_type(&drm->ttm.bdev, TTM_PL_VRAM);
if (drm->client.device.info.family >= NV_DEVICE_INFO_V0_TESLA) {
+ struct drm_device *drm_dev = drm->dev;
+
ttm_resource_manager_set_used(man, false);
ttm_resource_manager_evict_all(&drm->ttm.bdev, man);
ttm_resource_manager_cleanup(man);
ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_VRAM, NULL);
+ drm_lru_mgr_fini(&drm_dev->lru_mgr[TTM_PL_VRAM]);
kfree(man);
} else
ttm_range_man_fini(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM);
@@ -220,6 +227,7 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm)
struct ttm_resource_manager *man;
unsigned long size_pages = drm->gem.gart_available >> PAGE_SHIFT;
const struct ttm_resource_manager_func *func = NULL;
+ struct drm_device *drm_dev = drm->dev;
if (drm->client.device.info.family >= NV_DEVICE_INFO_V0_TESLA)
func = &nouveau_gart_manager;
@@ -238,6 +246,8 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm)
ttm_resource_manager_init(man, &drm->ttm.bdev, size_pages);
ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_TT, man);
ttm_resource_manager_set_used(man, true);
+ drm_lru_mgr_init(&drm_dev->lru_mgr[TTM_PL_TT], size_pages, &drm_dev->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm_dev->lru_mgr[TTM_PL_TT]);
return 0;
}
@@ -245,6 +255,7 @@ static void
nouveau_ttm_fini_gtt(struct nouveau_drm *drm)
{
struct ttm_resource_manager *man = ttm_manager_type(&drm->ttm.bdev, TTM_PL_TT);
+ struct drm_device *drm_dev = drm->dev;
if (drm->client.device.info.family < NV_DEVICE_INFO_V0_TESLA &&
drm->agp.bridge)
@@ -254,6 +265,7 @@ nouveau_ttm_fini_gtt(struct nouveau_drm *drm)
ttm_resource_manager_evict_all(&drm->ttm.bdev, man);
ttm_resource_manager_cleanup(man);
ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_TT, NULL);
+ drm_lru_mgr_fini(&drm_dev->lru_mgr[TTM_PL_TT]);
kfree(man);
}
}
diff --git a/drivers/gpu/drm/ttm/ttm_range_manager.c b/drivers/gpu/drm/ttm/ttm_range_manager.c
index db1ae370580d..898ede7d25c4 100644
--- a/drivers/gpu/drm/ttm/ttm_range_manager.c
+++ b/drivers/gpu/drm/ttm/ttm_range_manager.c
@@ -36,6 +36,7 @@
#include <drm/drm_mm.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
+#include <drm/drm_evictable_lru.h>
/*
* Currently we use a spinlock for the lock, but a mutex *may* be
@@ -199,6 +200,10 @@ int ttm_range_man_init_nocheck(struct drm_device *drm, struct ttm_device *bdev,
ttm_set_driver_manager(bdev, type, &rman->manager);
ttm_resource_manager_set_used(man, true);
+
+ drm_lru_mgr_init(&drm->lru_mgr[type], p_size, &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[type]);
+
return 0;
}
EXPORT_SYMBOL(ttm_range_man_init_nocheck);
@@ -236,6 +241,7 @@ int ttm_range_man_fini_nocheck(struct drm_device *drm, struct ttm_device *bdev,
ttm_resource_manager_cleanup(man);
ttm_set_driver_manager(bdev, type, NULL);
+ drm_lru_mgr_fini(&drm->lru_mgr[type]);
kfree(rman);
return 0;
}
diff --git a/drivers/gpu/drm/ttm/ttm_sys_manager.c b/drivers/gpu/drm/ttm/ttm_sys_manager.c
index f0f026d40a69..db410c7f73fe 100644
--- a/drivers/gpu/drm/ttm/ttm_sys_manager.c
+++ b/drivers/gpu/drm/ttm/ttm_sys_manager.c
@@ -50,4 +50,6 @@ void ttm_sys_man_init(struct drm_device *drm, struct ttm_device *bdev)
ttm_resource_manager_init(man, bdev, 0);
ttm_set_driver_manager(bdev, TTM_PL_SYSTEM, man);
ttm_resource_manager_set_used(man, true);
+ drm_lru_mgr_init(&drm->lru_mgr[TTM_PL_SYSTEM], 0, &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[TTM_PL_SYSTEM]);
}
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c b/drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c
index ee7964cbdaca..102296399e00 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c
@@ -62,6 +62,7 @@ int vmw_sys_man_init(struct vmw_private *dev_priv)
struct ttm_device *bdev = &dev_priv->bdev;
struct ttm_resource_manager *man =
kzalloc(sizeof(*man), GFP_KERNEL);
+ struct drm_device *drm = &dev_priv->drm;
if (!man)
return -ENOMEM;
@@ -72,6 +73,9 @@ int vmw_sys_man_init(struct vmw_private *dev_priv)
ttm_resource_manager_init(man, bdev, 0);
ttm_set_driver_manager(bdev, VMW_PL_SYSTEM, man);
ttm_resource_manager_set_used(man, true);
+
+ drm_lru_mgr_init(&drm->lru_mgr[WMW_PL_SYSTEM], 0, &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[VMW_PL_SYSTEM]);
return 0;
}
@@ -79,6 +83,7 @@ void vmw_sys_man_fini(struct vmw_private *dev_priv)
{
struct ttm_resource_manager *man = ttm_manager_type(&dev_priv->bdev,
VMW_PL_SYSTEM);
+ struct drm_device *drm = &dev_priv->drm;
ttm_resource_manager_evict_all(&dev_priv->bdev, man);
@@ -86,5 +91,6 @@ void vmw_sys_man_fini(struct vmw_private *dev_priv)
ttm_resource_manager_cleanup(man);
ttm_set_driver_manager(&dev_priv->bdev, VMW_PL_SYSTEM, NULL);
+ drm_lru_mgr_fini(&drm->lru_mgr[VMW_PL_SYSTEM]);
kfree(man);
}
diff --git a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
index 3e1fa0c832ca..ace42852a419 100644
--- a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
@@ -96,11 +96,13 @@ static void ttm_sys_mgr_fini(struct drm_device *drm, void *arg)
ttm_resource_manager_cleanup(man);
ttm_set_driver_manager(&xe->ttm, XE_PL_TT, NULL);
+ drm_lru_mgr_fini(&drm->lru_mgr[XE_PL_TT]);
}
int xe_ttm_sys_mgr_init(struct xe_device *xe)
{
struct ttm_resource_manager *man = &xe->mem.sys_mgr;
+ struct drm_device *drm = &xe->drm;
struct sysinfo si;
u64 gtt_size;
@@ -114,5 +116,9 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)
ttm_resource_manager_init(man, &xe->ttm, gtt_size >> PAGE_SHIFT);
ttm_set_driver_manager(&xe->ttm, XE_PL_TT, man);
ttm_resource_manager_set_used(man, true);
+
+ drm_lru_mgr_init(&drm->lru_mgr[XE_PL_TT], gtt_size >> PAGE_SHIFT, &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[XE_PL_TT]);
+
return drmm_add_action_or_reset(&xe->drm, ttm_sys_mgr_fini, xe);
}
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index 06a54c8bd46f..a3c1bf555c06 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -328,6 +328,8 @@ static void ttm_vram_mgr_fini(struct drm_device *dev, void *arg)
ttm_resource_manager_cleanup(&mgr->manager);
ttm_set_driver_manager(&xe->ttm, mgr->mem_type, NULL);
+
+ drm_lru_mgr_fini(&dev->lru_mgr[mgr->mem_type]);
}
int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr,
@@ -335,6 +337,7 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr,
u64 default_page_size)
{
struct ttm_resource_manager *man = &mgr->manager;
+ struct drm_device *drm = &xe->drm;
int err;
man->func = &xe_ttm_vram_mgr_func;
@@ -350,6 +353,9 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr,
ttm_set_driver_manager(&xe->ttm, mem_type, &mgr->manager);
ttm_resource_manager_set_used(&mgr->manager, true);
+ drm_lru_mgr_init(&drm->lru_mgr[mem_type], size, &drm->lru_lock);
+ ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[mem_type]);
+
return drmm_add_action_or_reset(&xe->drm, ttm_vram_mgr_fini, mgr);
}
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 09/11] drm/ttm: Use drm LRU manager iterator
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (7 preceding siblings ...)
2023-11-02 4:33 ` [RFC 08/11] drm: Initialize drm lru manager Oak Zeng
@ 2023-11-02 4:33 ` Oak Zeng
2023-11-02 4:33 ` [RFC 10/11] drm/ttm: Implement ttm memory evict functions Oak Zeng
` (2 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:33 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
Since TTM resource LRU list is moved to drm LRU manager layer,
use drm lru manager iterator instead of TTM resource manager
iterator. TTM resource manager iterator is deleted. No function
change.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/ttm/ttm_bo.c | 7 ++--
drivers/gpu/drm/ttm/ttm_device.c | 10 ++++--
drivers/gpu/drm/ttm/ttm_resource.c | 51 ------------------------------
include/drm/ttm/ttm_resource.h | 33 ++-----------------
4 files changed, 14 insertions(+), 87 deletions(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 26e0555bad0c..4a5ffa920665 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -43,6 +43,7 @@
#include <linux/module.h>
#include <linux/atomic.h>
#include <linux/dma-resv.h>
+#include <drm/drm_evictable_lru.h>
#include "ttm_module.h"
@@ -593,15 +594,17 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
struct ww_acquire_ctx *ticket)
{
struct ttm_buffer_object *bo = NULL, *busy_bo = NULL;
- struct ttm_resource_cursor cursor;
+ struct drm_lru_cursor cursor;
struct ttm_resource *res;
+ struct drm_lru_entity *entity;
bool locked = false;
int ret;
spin_lock(bdev->lru_lock);
- ttm_resource_manager_for_each_res(man, &cursor, res) {
+ drm_lru_for_each_entity(man->lru_mgr, &cursor, entity) {
bool busy;
+ res = container_of(entity, struct ttm_resource, lru_entity);
if (!ttm_bo_evict_swapout_allowable(res->bo, ctx, place,
&locked, &busy)) {
if (busy && !busy_bo && ticket !=
diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c
index 393c3e27016e..881662d69aba 100644
--- a/drivers/gpu/drm/ttm/ttm_device.c
+++ b/drivers/gpu/drm/ttm/ttm_device.c
@@ -33,6 +33,7 @@
#include <drm/ttm/ttm_device.h>
#include <drm/ttm/ttm_tt.h>
#include <drm/ttm/ttm_placement.h>
+#include <drm/drm_evictable_lru.h>
#include "ttm_module.h"
@@ -141,7 +142,8 @@ int ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags)
int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
gfp_t gfp_flags)
{
- struct ttm_resource_cursor cursor;
+ struct drm_lru_cursor cursor;
+ struct drm_lru_entity *entity;
struct ttm_resource_manager *man;
struct ttm_resource *res;
unsigned i;
@@ -153,10 +155,12 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
if (!man || !man->use_tt)
continue;
- ttm_resource_manager_for_each_res(man, &cursor, res) {
- struct ttm_buffer_object *bo = res->bo;
+ drm_lru_for_each_entity(man->lru_mgr, &cursor, entity) {
+ struct ttm_buffer_object *bo;
uint32_t num_pages;
+ res = container_of(entity, struct ttm_resource, lru_entity);
+ bo = res->bo;
if (!bo || bo->resource != res)
continue;
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 05eef866065e..0c6e0dbeff07 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -488,57 +488,6 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
}
EXPORT_SYMBOL(ttm_resource_manager_debug);
-/**
- * ttm_resource_manager_first
- *
- * @man: resource manager to iterate over
- * @cursor: cursor to record the position
- *
- * Returns the first resource from the resource manager.
- */
-struct ttm_resource *
-ttm_resource_manager_first(struct ttm_resource_manager *man,
- struct ttm_resource_cursor *cursor)
-{
- struct ttm_resource *res;
-
- lockdep_assert_held(man->bdev->lru_lock);
-
- for (cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY;
- ++cursor->priority)
- list_for_each_entry(res, &man->lru[cursor->priority], lru)
- return res;
-
- return NULL;
-}
-
-/**
- * ttm_resource_manager_next
- *
- * @man: resource manager to iterate over
- * @cursor: cursor to record the position
- * @res: the current resource pointer
- *
- * Returns the next resource from the resource manager.
- */
-struct ttm_resource *
-ttm_resource_manager_next(struct ttm_resource_manager *man,
- struct ttm_resource_cursor *cursor,
- struct ttm_resource *res)
-{
- lockdep_assert_held(man->bdev->lru_lock);
-
- list_for_each_entry_continue(res, &man->lru[cursor->priority], lru)
- return res;
-
- for (++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY;
- ++cursor->priority)
- list_for_each_entry(res, &man->lru[cursor->priority], lru)
- return res;
-
- return NULL;
-}
-
static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
struct iosys_map *dmap,
pgoff_t i)
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index e4fc1ada5236..c2528cec12e6 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -207,6 +207,7 @@ struct ttm_bus_placement {
* @placement: Placement flags.
* @bus: Placement on io bus accessible to the CPU
* @bo: weak reference to the BO, protected by ttm_device::lru_lock
+ * @lru_entity: lru entity for this ttm resource.
*
* Structure indicating the placement and space resources used by a
* buffer object.
@@ -223,17 +224,7 @@ struct ttm_resource {
* @lru: Least recently used list, see &ttm_resource_manager.lru
*/
struct list_head lru;
-};
-
-/**
- * struct ttm_resource_cursor
- *
- * @priority: the current priority
- *
- * Cursor to iterate over the resources in a manager.
- */
-struct ttm_resource_cursor {
- unsigned int priority;
+ struct drm_lru_entity lru_entity;
};
/**
@@ -402,26 +393,6 @@ uint64_t ttm_resource_manager_usage(struct ttm_resource_manager *man);
void ttm_resource_manager_debug(struct ttm_resource_manager *man,
struct drm_printer *p);
-struct ttm_resource *
-ttm_resource_manager_first(struct ttm_resource_manager *man,
- struct ttm_resource_cursor *cursor);
-struct ttm_resource *
-ttm_resource_manager_next(struct ttm_resource_manager *man,
- struct ttm_resource_cursor *cursor,
- struct ttm_resource *res);
-
-/**
- * ttm_resource_manager_for_each_res - iterate over all resources
- * @man: the resource manager
- * @cursor: struct ttm_resource_cursor for the current position
- * @res: the current resource
- *
- * Iterate over all the evictable resources in a resource manager.
- */
-#define ttm_resource_manager_for_each_res(man, cursor, res) \
- for (res = ttm_resource_manager_first(man, cursor); res; \
- res = ttm_resource_manager_next(man, cursor, res))
-
struct ttm_kmap_iter *
ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
struct io_mapping *iomap,
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 10/11] drm/ttm: Implement ttm memory evict functions
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (8 preceding siblings ...)
2023-11-02 4:33 ` [RFC 09/11] drm/ttm: Use drm LRU manager iterator Oak Zeng
@ 2023-11-02 4:33 ` Oak Zeng
2023-11-02 4:33 ` [RFC 11/11] drm/ttm: Write ttm functions using drm lru manager functions Oak Zeng
2023-12-21 13:12 ` [PATCH 00/11] Introduce drm evictable lru Thomas Hellström
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:33 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
Implement ttm_mem_evict_valuable, ttm_mem_evict_entity and
ttm_mem_evict_busy_entity. Those are callback functions from
drm lru manager. Register those functions during drm lru entity
initialization. Those 3 functions are splitted from original
ttm_mem_evict_first function.
Reimplemented ttm_mem_evict_first function using drm_lru_evict_first
function. For now, drm_lru_evict_first just calls back to above 3
functions which are splitted from ttm_mem_evict_first function, so
there is no function change. In the future, when SVM code is added,
drm_lru_evict_first function can also calls into SVM resource eviction
functions, thus TTM and SVM can mutually evict resources from each
other.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/ttm/ttm_bo.c | 192 ++++++++++++++++++++++++++++-------
include/drm/ttm/ttm_bo.h | 2 +
2 files changed, 158 insertions(+), 36 deletions(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 4a5ffa920665..9ec7a246e2ad 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -587,50 +587,148 @@ static int ttm_mem_evict_wait_busy(struct ttm_buffer_object *busy_bo,
return r == -EDEADLK ? -EBUSY : r;
}
-int ttm_mem_evict_first(struct ttm_device *bdev,
- struct ttm_resource_manager *man,
- const struct ttm_place *place,
- struct ttm_operation_ctx *ctx,
- struct ww_acquire_ctx *ticket)
+struct ttm_mem_evict_ctx {
+ const struct ttm_place *place;
+ struct ttm_operation_ctx *ctx;
+ struct ww_acquire_ctx *ticket;
+};
+
+/**
+ * ttm_mem_evict_allowable
+ *
+ * @lru_entity: The struct ttm_resource::lru_entity when this resource is
+ * added to drm lru list.
+ * @place: The preferred ttm placement where we want to evict memory for
+ * more memory space. If the current ttm_resource doesn't match the preferred
+ * placement, then there is no need to evict the current resource.
+ * @ctx: ttm operation context
+ * @ticket: dma reservation's context used to lock resource
+ * @busy: used to return whether the current resource is busy (i.e., locked
+ * by other clients)
+ * @locked: used to return whether this resource is locked during this check,
+ * i.e., successfully trylocked bo's dma reservation object
+ *
+ * Check whether we are allowed to evict a memory resource. Return true if we
+ * are allowed to evict resource; otherwise false.
+ *
+ * When this function returns true, a resource reference counter (bo's reference)
+ * is hold. This reference counter need to be released after evict operation later
+ * on.
+ *
+ * This function should be called with lru_lock hold.
+ */
+bool ttm_mem_evict_allowable(struct drm_lru_entity *lru_entity,
+ const struct drm_lru_evict_ctx *lru_evict_ctx,
+ bool *busy, bool *locked)
{
- struct ttm_buffer_object *bo = NULL, *busy_bo = NULL;
- struct drm_lru_cursor cursor;
struct ttm_resource *res;
- struct drm_lru_entity *entity;
- bool locked = false;
- int ret;
+ struct ttm_buffer_object *bo = NULL;
+ struct ttm_device *bdev;
+ const struct ttm_place *place;
+ struct ttm_operation_ctx *ctx;
+ struct ww_acquire_ctx *ticket;
+ struct ttm_mem_evict_ctx *evict_ctx;
- spin_lock(bdev->lru_lock);
- drm_lru_for_each_entity(man->lru_mgr, &cursor, entity) {
- bool busy;
+ evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx;
+ place = evict_ctx->place;
+ ctx = evict_ctx->ctx;
+ ticket = evict_ctx->ticket;
- res = container_of(entity, struct ttm_resource, lru_entity);
- if (!ttm_bo_evict_swapout_allowable(res->bo, ctx, place,
- &locked, &busy)) {
- if (busy && !busy_bo && ticket !=
- dma_resv_locking_ctx(res->bo->base.resv))
- busy_bo = res->bo;
- continue;
- }
+ res = container_of(lru_entity, struct ttm_resource, lru_entity);
+ bo = res->bo;
+ bdev = bo->bdev;
- if (ttm_bo_get_unless_zero(res->bo)) {
- bo = res->bo;
- break;
- }
- if (locked)
- dma_resv_unlock(res->bo->base.resv);
- }
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, place, locked, busy)) {
+ if (busy && ticket != dma_resv_locking_ctx(bo->base.resv))
+ *busy = true;
- if (!bo) {
- if (busy_bo && !ttm_bo_get_unless_zero(busy_bo))
- busy_bo = NULL;
- spin_unlock(bdev->lru_lock);
- ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket);
- if (busy_bo)
- ttm_bo_put(busy_bo);
- return ret;
+ return false;
}
+ if (ttm_bo_get_unless_zero(bo))
+ return true;
+
+ if (locked)
+ dma_resv_unlock(bo->base.resv);
+
+ return false;
+}
+
+/**
+ * ttm_mem_evict_busy_entity
+ *
+ * @lru_entity: The struct ttm_resource::lru_entity when this resource is
+ * added to drm lru list.
+ * @ctx: ttm operation context
+ * @ticket: dma reservation's context used to lock resource
+ *
+ * Evict a busy memory resource.
+ * This function should be called with lru_lock hold.
+ */
+int ttm_mem_evict_busy_entity(struct drm_lru_entity *lru_entity,
+ const struct drm_lru_evict_ctx *lru_evict_ctx)
+{
+ struct ttm_resource *res;
+ struct ttm_buffer_object *bo = NULL;
+ struct ttm_device *bdev;
+ int ret;
+ struct ttm_operation_ctx *ctx;
+ struct ww_acquire_ctx *ticket;
+ struct ttm_mem_evict_ctx *evict_ctx;
+
+ evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx;
+ ctx = evict_ctx->ctx;
+ ticket = evict_ctx->ticket;
+
+ res = container_of(lru_entity, struct ttm_resource, lru_entity);
+ bo = res->bo;
+ bdev = bo->bdev;
+
+ if (bo && !ttm_bo_get_unless_zero(bo))
+ bo = NULL;
+ spin_unlock(bdev->lru_lock);
+ ret = ttm_mem_evict_wait_busy(bo, ctx, ticket);
+ /* FIXME: this is code copied originally from ttm_mem_evict_first.
+ * 1) Shouldn't we ttm_bo_evict this bo also? Otherwise how can we
+ * make any memory space?
+ * 2) We also need to check whether this busy entity is in the same
+ * ttm_place as specified in lru_evict_ctx::place; if not, there is
+ * no help to wait this busy entity.
+ */
+ if (bo)
+ ttm_bo_put(bo);
+
+ return ret;
+}
+
+/**
+ * @lru_entity: The struct ttm_resource::lru_entity when this resource is
+ * added to drm lru list.
+ * @ctx: ttm operation context
+ * @locked: whether this resource is dma-reserved (if reserved, we need to
+ * unreserve it in this function)
+ *
+ * Evict a memory resource corresponding to a lru_entity. This should be
+ * called holding lru_lock
+ *
+ */
+int ttm_mem_evict_entity(struct drm_lru_entity *lru_entity,
+ const struct drm_lru_evict_ctx *lru_evict_ctx, bool locked)
+{
+ struct ttm_resource *res;
+ struct ttm_buffer_object *bo = NULL;
+ struct ttm_device *bdev;
+ int ret;
+ struct ttm_operation_ctx *ctx;
+ struct ttm_mem_evict_ctx *evict_ctx;
+
+ evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx;
+ ctx = evict_ctx->ctx;
+
+ res = container_of(lru_entity, struct ttm_resource, lru_entity);
+ bo = res->bo;
+ bdev = bo->bdev;
+
if (bo->deleted) {
ret = ttm_bo_cleanup_refs(bo, ctx->interruptible,
ctx->no_wait_gpu, locked);
@@ -650,6 +748,28 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
return ret;
}
+struct drm_lru_evict_func ttm_evict_func = {
+ .evict_allowable = ttm_mem_evict_allowable,
+ .evict_busy_entity = ttm_mem_evict_busy_entity,
+ .evict_entity = ttm_mem_evict_entity
+};
+EXPORT_SYMBOL(ttm_evict_func);
+
+int ttm_mem_evict_first(struct ttm_device *bdev,
+ struct ttm_resource_manager *man,
+ const struct ttm_place *place,
+ struct ttm_operation_ctx *ctx,
+ struct ww_acquire_ctx *ticket)
+{
+ struct drm_lru_evict_ctx evict_ctx = {
+ .data1 = place,
+ .data2 = ctx,
+ .data3 = ticket
+ };
+
+ return drm_lru_evict_first(man->lru_mgr, &evict_ctx);
+}
+
/**
* ttm_bo_pin - Pin the buffer object.
* @bo: The buffer object to pin
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index 49f32df32204..223b198fe371 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -50,6 +50,7 @@ struct ttm_place;
struct ttm_resource;
struct ttm_resource_manager;
struct ttm_tt;
+struct drm_lru_evict_func;
/**
* enum ttm_bo_type
@@ -424,4 +425,5 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
pgprot_t tmp);
void ttm_bo_tt_destroy(struct ttm_buffer_object *bo);
+extern struct drm_lru_evict_func ttm_evict_func;
#endif
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [RFC 11/11] drm/ttm: Write ttm functions using drm lru manager functions
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (9 preceding siblings ...)
2023-11-02 4:33 ` [RFC 10/11] drm/ttm: Implement ttm memory evict functions Oak Zeng
@ 2023-11-02 4:33 ` Oak Zeng
2023-12-21 13:12 ` [PATCH 00/11] Introduce drm evictable lru Thomas Hellström
11 siblings, 0 replies; 19+ messages in thread
From: Oak Zeng @ 2023-11-02 4:33 UTC (permalink / raw)
To: dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty, christian.koenig
Replace struct ttm_resource::lru with drm lru entity. Replace
struct ttm_resource_manager::lru[] with drm lru manager. Remove
ttm_lru_bulk_move functions and definitions as those are moved
to drm lru manager.
Some of ttm resource, ttm bo and ttm device functions are re-written
using drm lru manager functions.
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +-
drivers/gpu/drm/ttm/tests/ttm_device_test.c | 2 +-
drivers/gpu/drm/ttm/ttm_bo.c | 20 +--
drivers/gpu/drm/ttm/ttm_bo_util.c | 20 +--
drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
drivers/gpu/drm/ttm/ttm_device.c | 10 +-
drivers/gpu/drm/ttm/ttm_range_manager.c | 2 +-
drivers/gpu/drm/ttm/ttm_resource.c | 155 ++++----------------
drivers/gpu/drm/xe/xe_bo.c | 44 +++---
drivers/gpu/drm/xe/xe_bo.h | 3 +-
drivers/gpu/drm/xe/xe_dma_buf.c | 4 +-
drivers/gpu/drm/xe/xe_exec.c | 2 +-
drivers/gpu/drm/xe/xe_migrate.c | 6 +-
drivers/gpu/drm/xe/xe_res_cursor.h | 10 +-
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 2 +-
drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 12 +-
drivers/gpu/drm/xe/xe_vm.c | 2 +-
drivers/gpu/drm/xe/xe_vm_types.h | 2 +-
include/drm/ttm/ttm_bo.h | 4 +-
include/drm/ttm/ttm_resource.h | 59 ++------
21 files changed, 112 insertions(+), 253 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 747bcad86d5d..c977c00e986a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -369,7 +369,7 @@ void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev,
struct amdgpu_vm *vm)
{
spin_lock(adev->mman.bdev.lru_lock);
- ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
+ drm_lru_bulk_move_tail(&vm->lru_bulk_move);
spin_unlock(adev->mman.bdev.lru_lock);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 204ab13184ed..fec545b5d154 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -337,7 +337,7 @@ struct amdgpu_vm {
struct amdgpu_task_info task_info;
/* Store positions of group of BOs */
- struct ttm_lru_bulk_move lru_bulk_move;
+ struct drm_lru_bulk_move lru_bulk_move;
/* Flag to indicate if VM is used for compute */
bool is_compute_context;
diff --git a/drivers/gpu/drm/ttm/tests/ttm_device_test.c b/drivers/gpu/drm/ttm/tests/ttm_device_test.c
index b1b423b68cdf..a62ca31b55df 100644
--- a/drivers/gpu/drm/ttm/tests/ttm_device_test.c
+++ b/drivers/gpu/drm/ttm/tests/ttm_device_test.c
@@ -90,7 +90,7 @@ static void ttm_device_fini_basic(struct kunit *test)
ttm_device_fini(ttm_dev);
KUNIT_ASSERT_FALSE(test, man->use_type);
- KUNIT_ASSERT_TRUE(test, list_empty(&man->lru[0]));
+ KUNIT_ASSERT_TRUE(test, list_empty(&man->lru_mgr->lru[0]));
KUNIT_ASSERT_NULL(test, ttm_dev->man_drv[TTM_PL_SYSTEM]);
}
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 9ec7a246e2ad..d44ca5e51dff 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -92,11 +92,11 @@ EXPORT_SYMBOL(ttm_bo_move_to_lru_tail);
* resulting in much less overhead of maintaining the LRU.
* The only requirement is that the resources stay together on the LRU and are
* never separated. This is enforces by setting the bulk_move structure on a BO.
- * ttm_lru_bulk_move_tail() should be used to move all resources to the tail of
+ * drm_lru_bulk_move_tail() should be used to move all resources to the tail of
* their LRU list.
*/
void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo,
- struct ttm_lru_bulk_move *bulk)
+ struct drm_lru_bulk_move *bulk)
{
dma_resv_assert_held(bo->base.resv);
@@ -122,8 +122,8 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
bool old_use_tt, new_use_tt;
int ret;
- old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
- new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
+ old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->lru_entity.mem_type)->use_tt;
+ new_use_tt = ttm_manager_type(bdev, mem->lru_entity.mem_type)->use_tt;
ttm_bo_unmap_virtual(bo);
@@ -139,7 +139,7 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
if (ret)
goto out_err;
- if (mem->mem_type != TTM_PL_SYSTEM) {
+ if (mem->lru_entity.mem_type != TTM_PL_SYSTEM) {
ret = ttm_tt_populate(bo->bdev, bo->ttm, ctx);
if (ret)
goto out_err;
@@ -492,7 +492,7 @@ bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
struct ttm_device *bdev = bo->bdev;
dma_resv_assert_held(bo->base.resv);
- if (bo->resource->mem_type == TTM_PL_SYSTEM)
+ if (bo->resource->lru_entity.mem_type == TTM_PL_SYSTEM)
return true;
/* Don't evict this BO if it's outside of the
@@ -540,7 +540,7 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
*busy = !ret;
}
- if (ret && place && (bo->resource->mem_type != place->mem_type ||
+ if (ret && place && (bo->resource->lru_entity.mem_type != place->mem_type ||
!bo->bdev->funcs->eviction_valuable(bo, place))) {
ret = false;
if (*locked) {
@@ -1039,7 +1039,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
/*
* We might need to add a TTM.
*/
- if (!bo->resource || bo->resource->mem_type == TTM_PL_SYSTEM) {
+ if (!bo->resource || bo->resource->lru_entity.mem_type == TTM_PL_SYSTEM) {
ret = ttm_tt_create(bo, true);
if (ret)
return ret;
@@ -1259,7 +1259,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
* as an indication that we're about to swap out.
*/
memset(&place, 0, sizeof(place));
- place.mem_type = bo->resource->mem_type;
+ place.mem_type = bo->resource->lru_entity.mem_type;
if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
return -EBUSY;
@@ -1284,7 +1284,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
/*
* Move to system cached
*/
- if (bo->resource->mem_type != TTM_PL_SYSTEM) {
+ if (bo->resource->lru_entity.mem_type != TTM_PL_SYSTEM) {
struct ttm_resource *evict_mem;
struct ttm_place hop;
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index fd9fd3d15101..7176fbfca5eb 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -145,7 +145,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
{
struct ttm_device *bdev = bo->bdev;
struct ttm_resource_manager *dst_man =
- ttm_manager_type(bo->bdev, dst_mem->mem_type);
+ ttm_manager_type(bo->bdev, dst_mem->lru_entity.mem_type);
struct ttm_tt *ttm = bo->ttm;
struct ttm_resource *src_mem = bo->resource;
struct ttm_resource_manager *src_man;
@@ -160,7 +160,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
if (WARN_ON(!src_mem))
return -EINVAL;
- src_man = ttm_manager_type(bdev, src_mem->mem_type);
+ src_man = ttm_manager_type(bdev, src_mem->lru_entity.mem_type);
if (ttm && ((ttm->page_flags & TTM_TT_FLAG_SWAPPED) ||
dst_man->use_tt)) {
ret = ttm_tt_populate(bdev, ttm, ctx);
@@ -184,7 +184,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
clear = src_iter->ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm));
if (!(clear && ttm && !(ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC)))
- ttm_move_memcpy(clear, PFN_UP(dst_mem->size), dst_iter, src_iter);
+ ttm_move_memcpy(clear, PFN_UP(dst_mem->lru_entity.size), dst_iter, src_iter);
if (!src_iter->ops->maps_tt)
ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, src_mem);
@@ -293,7 +293,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
struct ttm_resource_manager *man;
enum ttm_caching caching;
- man = ttm_manager_type(bo->bdev, res->mem_type);
+ man = ttm_manager_type(bo->bdev, res->lru_entity.mem_type);
caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
return ttm_prot_from_caching(caching, tmp);
@@ -393,9 +393,9 @@ int ttm_bo_kmap(struct ttm_buffer_object *bo,
map->virtual = NULL;
map->bo = bo;
- if (num_pages > PFN_UP(bo->resource->size))
+ if (num_pages > PFN_UP(bo->resource->lru_entity.size))
return -EINVAL;
- if ((start_page + num_pages) > PFN_UP(bo->resource->size))
+ if ((start_page + num_pages) > PFN_UP(bo->resource->lru_entity.size))
return -EINVAL;
ret = ttm_mem_io_reserve(bo->bdev, bo->resource);
@@ -607,7 +607,7 @@ static void ttm_bo_move_pipeline_evict(struct ttm_buffer_object *bo,
struct ttm_device *bdev = bo->bdev;
struct ttm_resource_manager *from;
- from = ttm_manager_type(bdev, bo->resource->mem_type);
+ from = ttm_manager_type(bdev, bo->resource->lru_entity.mem_type);
/**
* BO doesn't have a TTM we need to bind/unbind. Just remember
@@ -646,8 +646,8 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
struct ttm_resource *new_mem)
{
struct ttm_device *bdev = bo->bdev;
- struct ttm_resource_manager *from = ttm_manager_type(bdev, bo->resource->mem_type);
- struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
+ struct ttm_resource_manager *from = ttm_manager_type(bdev, bo->resource->lru_entity.mem_type);
+ struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->lru_entity.mem_type);
int ret = 0;
dma_resv_add_fence(bo->base.resv, fence, DMA_RESV_USAGE_KERNEL);
@@ -680,7 +680,7 @@ void ttm_bo_move_sync_cleanup(struct ttm_buffer_object *bo,
struct ttm_resource *new_mem)
{
struct ttm_device *bdev = bo->bdev;
- struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
+ struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->lru_entity.mem_type);
int ret;
ret = ttm_bo_wait_free_node(bo, man->use_tt);
diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c
index 4212b8c91dd4..0e550430fa85 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_vm.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c
@@ -421,7 +421,7 @@ int ttm_bo_vm_access(struct vm_area_struct *vma, unsigned long addr,
if (ret)
return ret;
- switch (bo->resource->mem_type) {
+ switch (bo->resource->lru_entity.mem_type) {
case TTM_PL_SYSTEM:
fallthrough;
case TTM_PL_TT:
diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c
index 881662d69aba..7e5bfdffc08d 100644
--- a/drivers/gpu/drm/ttm/ttm_device.c
+++ b/drivers/gpu/drm/ttm/ttm_device.c
@@ -250,7 +250,7 @@ void ttm_device_fini(struct ttm_device *bdev)
spin_lock(bdev->lru_lock);
for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i)
- if (list_empty(&man->lru[0]))
+ if (list_empty(&man->lru_mgr->lru[0]))
pr_debug("Swap list %d was clean\n", i);
spin_unlock(bdev->lru_lock);
@@ -263,16 +263,18 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,
struct list_head *list)
{
struct ttm_resource *res;
+ struct drm_lru_entity *entity;
spin_lock(bdev->lru_lock);
- while ((res = list_first_entry_or_null(list, typeof(*res), lru))) {
+ while ((entity = list_first_entry_or_null(list, typeof(*entity), lru))){
+ res = container_of(entity, struct ttm_resource, lru_entity);
struct ttm_buffer_object *bo = res->bo;
/* Take ref against racing releases once lru_lock is unlocked */
if (!ttm_bo_get_unless_zero(bo))
continue;
- list_del_init(&res->lru);
+ list_del_init(&entity->lru);
spin_unlock(bdev->lru_lock);
if (bo->ttm)
@@ -297,7 +299,7 @@ void ttm_device_clear_dma_mappings(struct ttm_device *bdev)
continue;
for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j)
- ttm_device_clear_lru_dma_mappings(bdev, &man->lru[j]);
+ ttm_device_clear_lru_dma_mappings(bdev, &man->lru_mgr->lru[j]);
}
}
EXPORT_SYMBOL(ttm_device_clear_dma_mappings);
diff --git a/drivers/gpu/drm/ttm/ttm_range_manager.c b/drivers/gpu/drm/ttm/ttm_range_manager.c
index 898ede7d25c4..afdea64ffc10 100644
--- a/drivers/gpu/drm/ttm/ttm_range_manager.c
+++ b/drivers/gpu/drm/ttm/ttm_range_manager.c
@@ -84,7 +84,7 @@ static int ttm_range_man_alloc(struct ttm_resource_manager *man,
spin_lock(&rman->lock);
ret = drm_mm_insert_node_in_range(mm, &node->mm_nodes[0],
- PFN_UP(node->base.size),
+ PFN_UP(node->base.lru_entity.size),
bo->page_alignment, 0,
place->fpfn, lpfn, mode);
spin_unlock(&rman->lock);
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 0c6e0dbeff07..b6ff3b9e0614 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -30,108 +30,12 @@
#include <drm/ttm/ttm_placement.h>
#include <drm/ttm/ttm_resource.h>
-/**
- * ttm_lru_bulk_move_init - initialize a bulk move structure
- * @bulk: the structure to init
- *
- * For now just memset the structure to zero.
- */
-void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk)
-{
- memset(bulk, 0, sizeof(*bulk));
-}
-EXPORT_SYMBOL(ttm_lru_bulk_move_init);
-
-/**
- * ttm_lru_bulk_move_tail - bulk move range of resources to the LRU tail.
- *
- * @bulk: bulk move structure
- *
- * Bulk move BOs to the LRU tail, only valid to use when driver makes sure that
- * resource order never changes. Should be called with &drm_device.lru_lock held.
- */
-void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
-{
- unsigned i, j;
-
- for (i = 0; i < DRM_NUM_MEM_TYPES; ++i) {
- for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) {
- struct ttm_lru_bulk_move_pos *pos = &bulk->pos[i][j];
- struct ttm_resource_manager *man;
-
- if (!pos->first)
- continue;
-
- lockdep_assert_held(pos->first->bo->bdev->lru_lock);
- dma_resv_assert_held(pos->first->bo->base.resv);
- dma_resv_assert_held(pos->last->bo->base.resv);
-
- man = ttm_manager_type(pos->first->bo->bdev, i);
- list_bulk_move_tail(&man->lru[j], &pos->first->lru,
- &pos->last->lru);
- }
- }
-}
-EXPORT_SYMBOL(ttm_lru_bulk_move_tail);
-
-/* Return the bulk move pos object for this resource */
-static struct ttm_lru_bulk_move_pos *
-ttm_lru_bulk_move_pos(struct ttm_lru_bulk_move *bulk, struct ttm_resource *res)
-{
- return &bulk->pos[res->mem_type][res->bo->priority];
-}
-
-/* Move the resource to the tail of the bulk move range */
-static void ttm_lru_bulk_move_pos_tail(struct ttm_lru_bulk_move_pos *pos,
- struct ttm_resource *res)
-{
- if (pos->last != res) {
- if (pos->first == res)
- pos->first = list_next_entry(res, lru);
- list_move(&res->lru, &pos->last->lru);
- pos->last = res;
- }
-}
-
-/* Add the resource to a bulk_move cursor */
-static void ttm_lru_bulk_move_add(struct ttm_lru_bulk_move *bulk,
- struct ttm_resource *res)
-{
- struct ttm_lru_bulk_move_pos *pos = ttm_lru_bulk_move_pos(bulk, res);
-
- if (!pos->first) {
- pos->first = res;
- pos->last = res;
- } else {
- ttm_lru_bulk_move_pos_tail(pos, res);
- }
-}
-
-/* Remove the resource from a bulk_move range */
-static void ttm_lru_bulk_move_del(struct ttm_lru_bulk_move *bulk,
- struct ttm_resource *res)
-{
- struct ttm_lru_bulk_move_pos *pos = ttm_lru_bulk_move_pos(bulk, res);
-
- if (unlikely(WARN_ON(!pos->first || !pos->last) ||
- (pos->first == res && pos->last == res))) {
- pos->first = NULL;
- pos->last = NULL;
- } else if (pos->first == res) {
- pos->first = list_next_entry(res, lru);
- } else if (pos->last == res) {
- pos->last = list_prev_entry(res, lru);
- } else {
- list_move(&res->lru, &pos->last->lru);
- }
-}
-
/* Add the resource to a bulk move if the BO is configured for it */
void ttm_resource_add_bulk_move(struct ttm_resource *res,
struct ttm_buffer_object *bo)
{
if (bo->bulk_move && !bo->pin_count)
- ttm_lru_bulk_move_add(bo->bulk_move, res);
+ drm_lru_add_bulk_move(&res->lru_entity, bo->bulk_move);
}
/* Remove the resource from a bulk move if the BO is configured for it */
@@ -139,7 +43,7 @@ void ttm_resource_del_bulk_move(struct ttm_resource *res,
struct ttm_buffer_object *bo)
{
if (bo->bulk_move && !bo->pin_count)
- ttm_lru_bulk_move_del(bo->bulk_move, res);
+ drm_lru_del_bulk_move(&res->lru_entity, bo->bulk_move);
}
/* Move a resource to the LRU or bulk tail */
@@ -150,20 +54,16 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res)
lockdep_assert_held(bo->bdev->lru_lock);
- if (bo->pin_count) {
- list_move_tail(&res->lru, &bdev->pinned);
+ if (bo->pin_count)
+ list_move_tail(&res->lru_entity.lru, &bdev->pinned);
- } else if (bo->bulk_move) {
- struct ttm_lru_bulk_move_pos *pos =
- ttm_lru_bulk_move_pos(bo->bulk_move, res);
+ else if (bo->bulk_move) {
+ struct drm_lru_bulk_move_range *range =
+ &bo->bulk_move->range[res->lru_entity.mem_type][res->lru_entity.priority];
- ttm_lru_bulk_move_pos_tail(pos, res);
- } else {
- struct ttm_resource_manager *man;
-
- man = ttm_manager_type(bdev, res->mem_type);
- list_move_tail(&res->lru, &man->lru[bo->priority]);
- }
+ drm_lru_bulk_move_range_tail(range, &res->lru_entity);
+ } else
+ drm_lru_move_to_tail(&res->lru_entity);
}
/**
@@ -181,22 +81,22 @@ void ttm_resource_init(struct ttm_buffer_object *bo,
struct ttm_resource_manager *man;
res->start = 0;
- res->size = bo->base.size;
- res->mem_type = place->mem_type;
res->placement = place->flags;
res->bus.addr = NULL;
res->bus.offset = 0;
res->bus.is_iomem = false;
res->bus.caching = ttm_cached;
res->bo = bo;
+ drm_lru_entity_init(&res->lru_entity, bo->base.dev, place->mem_type,
+ bo->base.size, bo->priority, &ttm_evict_func);
man = ttm_manager_type(bo->bdev, place->mem_type);
spin_lock(bo->bdev->lru_lock);
if (bo->pin_count)
- list_add_tail(&res->lru, &bo->bdev->pinned);
+ list_add_tail(&res->lru_entity.lru, &bo->bdev->pinned);
else
- list_add_tail(&res->lru, &man->lru[bo->priority]);
- man->usage += res->size;
+ drm_lru_add_entity(&res->lru_entity, man->lru_mgr, bo->priority);
+ man->usage += res->lru_entity.size;
spin_unlock(bo->bdev->lru_lock);
}
EXPORT_SYMBOL(ttm_resource_init);
@@ -217,8 +117,8 @@ void ttm_resource_fini(struct ttm_resource_manager *man,
struct ttm_device *bdev = man->bdev;
spin_lock(bdev->lru_lock);
- list_del_init(&res->lru);
- man->usage -= res->size;
+ drm_lru_remove_entity(&res->lru_entity, man->lru_mgr);
+ man->usage -= res->lru_entity.size;
spin_unlock(bdev->lru_lock);
}
EXPORT_SYMBOL(ttm_resource_fini);
@@ -251,7 +151,7 @@ void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res)
spin_lock(bo->bdev->lru_lock);
ttm_resource_del_bulk_move(*res, bo);
spin_unlock(bo->bdev->lru_lock);
- man = ttm_manager_type(bo->bdev, (*res)->mem_type);
+ man = ttm_manager_type(bo->bdev, (*res)->lru_entity.mem_type);
man->func->free(man, *res);
*res = NULL;
}
@@ -280,7 +180,7 @@ bool ttm_resource_intersects(struct ttm_device *bdev,
if (!res)
return false;
- man = ttm_manager_type(bdev, res->mem_type);
+ man = ttm_manager_type(bdev, res->lru_entity.mem_type);
if (!place || !man->func->intersects)
return true;
@@ -309,7 +209,7 @@ bool ttm_resource_compatible(struct ttm_device *bdev,
if (!res || !place)
return false;
- man = ttm_manager_type(bdev, res->mem_type);
+ man = ttm_manager_type(bdev, res->lru_entity.mem_type);
if (!man->func->compatible)
return true;
@@ -333,7 +233,7 @@ static bool ttm_resource_places_compat(struct ttm_resource *res,
if (!ttm_resource_compatible(bdev, res, heap, bo->base.size))
continue;
- if ((res->mem_type == heap->mem_type) &&
+ if ((res->lru_entity.mem_type == heap->mem_type) &&
(!(heap->flags & TTM_PL_FLAG_CONTIGUOUS) ||
(res->placement & TTM_PL_FLAG_CONTIGUOUS)))
return true;
@@ -386,15 +286,10 @@ void ttm_resource_manager_init(struct ttm_resource_manager *man,
struct ttm_device *bdev,
uint64_t size)
{
- unsigned i;
-
spin_lock_init(&man->move_lock);
man->bdev = bdev;
man->size = size;
man->usage = 0;
-
- for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i)
- INIT_LIST_HEAD(&man->lru[i]);
man->move = NULL;
}
EXPORT_SYMBOL(ttm_resource_manager_init);
@@ -426,7 +321,7 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
spin_lock(bdev->lru_lock);
for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i) {
- while (!list_empty(&man->lru[i])) {
+ while (!list_empty(&man->lru_mgr->lru[i])) {
spin_unlock(bdev->lru_lock);
ret = ttm_mem_evict_first(bdev, man, NULL, &ctx,
NULL);
@@ -622,10 +517,10 @@ ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
if (mem->bus.caching == ttm_write_combined)
iosys_map_set_vaddr_iomem(&iter_io->dmap,
ioremap_wc(mem->bus.offset,
- mem->size));
+ mem->lru_entity.size));
else if (mem->bus.caching == ttm_cached)
iosys_map_set_vaddr(&iter_io->dmap,
- memremap(mem->bus.offset, mem->size,
+ memremap(mem->bus.offset, mem->lru_entity.size,
MEMREMAP_WB |
MEMREMAP_WT |
MEMREMAP_WC));
@@ -634,7 +529,7 @@ ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
if (iosys_map_is_null(&iter_io->dmap))
iosys_map_set_vaddr_iomem(&iter_io->dmap,
ioremap(mem->bus.offset,
- mem->size));
+ mem->lru_entity.size));
if (iosys_map_is_null(&iter_io->dmap)) {
ret = -ENOMEM;
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 827f798cccc0..1cae43532eac 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -61,12 +61,12 @@ bool mem_type_is_vram(u32 mem_type)
static bool resource_is_stolen_vram(struct xe_device *xe, struct ttm_resource *res)
{
- return res->mem_type == XE_PL_STOLEN && IS_DGFX(xe);
+ return res->lru_entity.mem_type == XE_PL_STOLEN && IS_DGFX(xe);
}
static bool resource_is_vram(struct ttm_resource *res)
{
- return mem_type_is_vram(res->mem_type);
+ return mem_type_is_vram(res->lru_entity.mem_type);
}
bool xe_bo_is_vram(struct xe_bo *bo)
@@ -77,7 +77,7 @@ bool xe_bo_is_vram(struct xe_bo *bo)
bool xe_bo_is_stolen(struct xe_bo *bo)
{
- return bo->ttm.resource->mem_type == XE_PL_STOLEN;
+ return bo->ttm.resource->lru_entity.mem_type == XE_PL_STOLEN;
}
/**
@@ -118,7 +118,7 @@ mem_type_to_tile(struct xe_device *xe, u32 mem_type)
*/
struct xe_tile *xe_bo_to_tile(struct xe_bo *bo)
{
- return mem_type_to_tile(xe_bo_device(bo), bo->ttm.resource->mem_type);
+ return mem_type_to_tile(xe_bo_device(bo), bo->ttm.resource->lru_entity.mem_type);
}
static void try_add_system(struct xe_bo *bo, struct ttm_place *places,
@@ -259,7 +259,7 @@ static void xe_evict_flags(struct ttm_buffer_object *tbo,
*/
bo = ttm_to_xe_bo(tbo);
- switch (tbo->resource->mem_type) {
+ switch (tbo->resource->lru_entity.mem_type) {
case XE_PL_VRAM0:
case XE_PL_VRAM1:
case XE_PL_STOLEN:
@@ -410,17 +410,17 @@ static int xe_ttm_io_mem_reserve(struct ttm_device *bdev,
{
struct xe_device *xe = ttm_to_xe_device(bdev);
- switch (mem->mem_type) {
+ switch (mem->lru_entity.mem_type) {
case XE_PL_SYSTEM:
case XE_PL_TT:
return 0;
case XE_PL_VRAM0:
case XE_PL_VRAM1: {
- struct xe_tile *tile = mem_type_to_tile(xe, mem->mem_type);
+ struct xe_tile *tile = mem_type_to_tile(xe, mem->lru_entity.mem_type);
struct xe_ttm_vram_mgr_resource *vres =
to_xe_ttm_vram_mgr_resource(mem);
- if (vres->used_visible_size < mem->size)
+ if (vres->used_visible_size < mem->lru_entity.size)
return -EINVAL;
mem->bus.offset = mem->start << PAGE_SHIFT;
@@ -546,7 +546,7 @@ static int xe_bo_move_dmabuf(struct ttm_buffer_object *ttm_bo,
XE_WARN_ON(!attach);
XE_WARN_ON(!ttm_bo->ttm);
- if (new_res->mem_type == XE_PL_SYSTEM)
+ if (new_res->lru_entity.mem_type == XE_PL_SYSTEM)
goto out;
if (ttm_bo->sg) {
@@ -620,7 +620,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
struct xe_device *xe = ttm_to_xe_device(ttm_bo->bdev);
struct xe_bo *bo = ttm_to_xe_bo(ttm_bo);
struct ttm_resource *old_mem = ttm_bo->resource;
- u32 old_mem_type = old_mem ? old_mem->mem_type : XE_PL_SYSTEM;
+ u32 old_mem_type = old_mem ? old_mem->lru_entity.mem_type : XE_PL_SYSTEM;
struct ttm_tt *ttm = ttm_bo->ttm;
struct xe_tile *tile = NULL;
struct dma_fence *fence;
@@ -652,7 +652,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
if ((move_lacks_source && !needs_clear) ||
(old_mem_type == XE_PL_SYSTEM &&
- new_mem->mem_type == XE_PL_TT)) {
+ new_mem->lru_entity.mem_type == XE_PL_TT)) {
ttm_bo_move_null(ttm_bo, new_mem);
goto out;
}
@@ -662,7 +662,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
* TTM_PL_FLAG_TEMPORARY, should just be a dummy move.
*/
if (old_mem_type == XE_PL_TT &&
- new_mem->mem_type == XE_PL_TT) {
+ new_mem->lru_entity.mem_type == XE_PL_TT) {
ttm_bo_move_null(ttm_bo, new_mem);
goto out;
}
@@ -674,7 +674,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
}
if (old_mem_type == XE_PL_TT &&
- new_mem->mem_type == XE_PL_SYSTEM) {
+ new_mem->lru_entity.mem_type == XE_PL_SYSTEM) {
long timeout = dma_resv_wait_timeout(ttm_bo->base.resv,
DMA_RESV_USAGE_BOOKKEEP,
true,
@@ -690,7 +690,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
if (!move_lacks_source &&
((old_mem_type == XE_PL_SYSTEM && resource_is_vram(new_mem)) ||
(mem_type_is_vram(old_mem_type) &&
- new_mem->mem_type == XE_PL_SYSTEM))) {
+ new_mem->lru_entity.mem_type == XE_PL_SYSTEM))) {
hop->fpfn = 0;
hop->lpfn = 0;
hop->mem_type = XE_PL_TT;
@@ -702,7 +702,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
if (bo->tile)
tile = bo->tile;
else if (resource_is_vram(new_mem))
- tile = mem_type_to_tile(xe, new_mem->mem_type);
+ tile = mem_type_to_tile(xe, new_mem->lru_entity.mem_type);
else if (mem_type_is_vram(old_mem_type))
tile = mem_type_to_tile(xe, old_mem_type);
@@ -777,7 +777,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
}
xe_device_mem_access_put(xe);
- trace_printk("new_mem->mem_type=%d\n", new_mem->mem_type);
+ trace_printk("new_mem->lru_entity.mem_type=%d\n", new_mem->lru_entity.mem_type);
out:
return ret;
@@ -918,10 +918,10 @@ static unsigned long xe_ttm_io_mem_pfn(struct ttm_buffer_object *ttm_bo,
{
struct xe_device *xe = ttm_to_xe_device(ttm_bo->bdev);
struct xe_bo *bo = ttm_to_xe_bo(ttm_bo);
- struct xe_tile *tile = mem_type_to_tile(xe, ttm_bo->resource->mem_type);
+ struct xe_tile *tile = mem_type_to_tile(xe, ttm_bo->resource->lru_entity.mem_type);
struct xe_res_cursor cursor;
- if (ttm_bo->resource->mem_type == XE_PL_STOLEN)
+ if (ttm_bo->resource->lru_entity.mem_type == XE_PL_STOLEN)
return xe_ttm_stolen_io_offset(bo, page_offset << PAGE_SHIFT) >> PAGE_SHIFT;
xe_res_first(ttm_bo->resource, (u64)page_offset << PAGE_SHIFT, 0, &cursor);
@@ -1183,7 +1183,7 @@ void xe_bo_free(struct xe_bo *bo)
struct xe_bo *__xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo,
struct xe_tile *tile, struct dma_resv *resv,
- struct ttm_lru_bulk_move *bulk, size_t size,
+ struct drm_lru_bulk_move *bulk, size_t size,
enum ttm_bo_type type, u32 flags)
{
struct ttm_operation_ctx ctx = {
@@ -1452,9 +1452,9 @@ struct xe_bo *xe_bo_create_from_data(struct xe_device *xe, struct xe_tile *tile,
uint64_t vram_region_gpu_offset(struct ttm_resource *res)
{
struct xe_device *xe = ttm_to_xe_device(res->bo->bdev);
- struct xe_tile *tile = mem_type_to_tile(xe, res->mem_type);
+ struct xe_tile *tile = mem_type_to_tile(xe, res->lru_entity.mem_type);
- if (res->mem_type == XE_PL_STOLEN)
+ if (res->lru_entity.mem_type == XE_PL_STOLEN)
return xe_ttm_stolen_gpu_offset(xe);
return tile->mem.vram.dpa_base;
@@ -1960,7 +1960,7 @@ int xe_bo_migrate(struct xe_bo *bo, u32 mem_type)
xe_bo_assert_held(bo);
- if (bo->ttm.resource->mem_type == mem_type)
+ if (bo->ttm.resource->lru_entity.mem_type == mem_type)
return 0;
if (xe_bo_is_pinned(bo))
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 9918b2d630e1..3ab17c81fe6e 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -9,6 +9,7 @@
#include "xe_bo_types.h"
#include "xe_macros.h"
#include "xe_vm_types.h"
+#include <drm/drm_evictable_lru.h>
#define XE_DEFAULT_GTT_SIZE_MB 3072ULL /* 3GB by default */
@@ -83,7 +84,7 @@ void xe_bo_free(struct xe_bo *bo);
struct xe_bo *__xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo,
struct xe_tile *tile, struct dma_resv *resv,
- struct ttm_lru_bulk_move *bulk, size_t size,
+ struct drm_lru_bulk_move *bulk, size_t size,
enum ttm_bo_type type, u32 flags);
struct xe_bo *
xe_bo_create_locked_range(struct xe_device *xe,
diff --git a/drivers/gpu/drm/xe/xe_dma_buf.c b/drivers/gpu/drm/xe/xe_dma_buf.c
index 09343b8b3e96..f22a4e0388d4 100644
--- a/drivers/gpu/drm/xe/xe_dma_buf.c
+++ b/drivers/gpu/drm/xe/xe_dma_buf.c
@@ -82,7 +82,7 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach,
if (!xe_bo_is_pinned(bo)) {
if (!attach->peer2peer ||
- bo->ttm.resource->mem_type == XE_PL_SYSTEM) {
+ bo->ttm.resource->lru_entity.mem_type == XE_PL_SYSTEM) {
if (xe_bo_can_migrate(bo, XE_PL_TT))
r = xe_bo_migrate(bo, XE_PL_TT);
else
@@ -92,7 +92,7 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach,
return ERR_PTR(r);
}
- switch (bo->ttm.resource->mem_type) {
+ switch (bo->ttm.resource->lru_entity.mem_type) {
case XE_PL_TT:
sgt = drm_prime_pages_to_sg(obj->dev,
bo->ttm.ttm->pages,
diff --git a/drivers/gpu/drm/xe/xe_exec.c b/drivers/gpu/drm/xe/xe_exec.c
index dafebdfb2368..100385e71a94 100644
--- a/drivers/gpu/drm/xe/xe_exec.c
+++ b/drivers/gpu/drm/xe/xe_exec.c
@@ -371,7 +371,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
if (!err && !xe_vm_no_dma_fences(vm)) {
spin_lock(xe->ttm.lru_lock);
- ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
+ drm_lru_bulk_move_tail(&vm->lru_bulk_move);
spin_unlock(xe->ttm.lru_lock);
}
diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index ee8bc5f3ba3d..fdf2a448457d 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -643,8 +643,8 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
u64 src_L0, dst_L0;
int pass = 0;
int err;
- bool src_is_vram = mem_type_is_vram(src->mem_type);
- bool dst_is_vram = mem_type_is_vram(dst->mem_type);
+ bool src_is_vram = mem_type_is_vram(src->lru_entity.mem_type);
+ bool dst_is_vram = mem_type_is_vram(dst->lru_entity.mem_type);
bool copy_ccs = xe_device_has_flat_ccs(xe) &&
xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
@@ -895,7 +895,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,
struct xe_bo *bo,
struct ttm_resource *dst)
{
- bool clear_vram = mem_type_is_vram(dst->mem_type);
+ bool clear_vram = mem_type_is_vram(dst->lru_entity.mem_type);
struct xe_gt *gt = m->tile->primary_gt;
struct xe_device *xe = gt_to_xe(gt);
struct dma_fence *fence = NULL;
diff --git a/drivers/gpu/drm/xe/xe_res_cursor.h b/drivers/gpu/drm/xe/xe_res_cursor.h
index 5cb4b66a5d74..64c5549f4d44 100644
--- a/drivers/gpu/drm/xe/xe_res_cursor.h
+++ b/drivers/gpu/drm/xe/xe_res_cursor.h
@@ -53,8 +53,8 @@ static struct drm_buddy *xe_res_get_buddy(struct ttm_resource *res)
struct xe_device *xe = ttm_to_xe_device(res->bo->bdev);
struct ttm_resource_manager *mgr;
- if (res->mem_type != XE_PL_STOLEN)
- return &xe->tiles[res->mem_type - XE_PL_VRAM0].mem.vram_mgr->mm;
+ if (res->lru_entity.mem_type != XE_PL_STOLEN)
+ return &xe->tiles[res->lru_entity.mem_type - XE_PL_VRAM0].mem.vram_mgr->mm;
mgr = ttm_manager_type(&xe->ttm, XE_PL_STOLEN);
@@ -79,9 +79,9 @@ static inline void xe_res_first(struct ttm_resource *res,
if (!res)
goto fallback;
- XE_WARN_ON(start + size > res->size);
+ XE_WARN_ON(start + size > res->lru_entity.size);
- cur->mem_type = res->mem_type;
+ cur->mem_type = res->lru_entity.mem_type;
switch (cur->mem_type) {
case XE_PL_STOLEN:
@@ -128,7 +128,7 @@ static inline void xe_res_first(struct ttm_resource *res,
cur->remaining = size;
cur->node = NULL;
cur->mem_type = XE_PL_TT;
- XE_WARN_ON(res && start + size > res->size);
+ XE_WARN_ON(res && start + size > res->lru_entity.size);
}
static inline void __xe_res_sg_next(struct xe_res_cursor *cur)
diff --git a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
index ace42852a419..7aa179e73d50 100644
--- a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
@@ -48,7 +48,7 @@ static int xe_ttm_sys_mgr_new(struct ttm_resource_manager *man,
}
node->base.mm_nodes[0].start = 0;
- node->base.mm_nodes[0].size = PFN_UP(node->base.base.size);
+ node->base.mm_nodes[0].size = PFN_UP(node->base.base.lru_entity.size);
node->base.base.start = XE_BO_INVALID_OFFSET;
*res = &node->base.base;
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index a3c1bf555c06..06a469cc8490 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -83,11 +83,11 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
if (place->fpfn || lpfn != man->size >> PAGE_SHIFT)
vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
- if (WARN_ON(!vres->base.size)) {
+ if (WARN_ON(!vres->base.lru_entity.size)) {
err = -EINVAL;
goto error_fini;
}
- size = vres->base.size;
+ size = vres->base.lru_entity.size;
min_page_size = mgr->default_page_size;
if (tbo->page_alignment)
@@ -150,8 +150,8 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
} while (remaining_size);
if (place->flags & TTM_PL_FLAG_CONTIGUOUS) {
- if (!drm_buddy_block_trim(mm, vres->base.size, &vres->blocks))
- size = vres->base.size;
+ if (!drm_buddy_block_trim(mm, vres->base.lru_entity.size, &vres->blocks))
+ size = vres->base.lru_entity.size;
}
if (lpfn <= mgr->visible_size >> PAGE_SHIFT) {
@@ -378,14 +378,14 @@ int xe_ttm_vram_mgr_alloc_sgt(struct xe_device *xe,
enum dma_data_direction dir,
struct sg_table **sgt)
{
- struct xe_tile *tile = &xe->tiles[res->mem_type - XE_PL_VRAM0];
+ struct xe_tile *tile = &xe->tiles[res->lru_entity.mem_type - XE_PL_VRAM0];
struct xe_ttm_vram_mgr_resource *vres = to_xe_ttm_vram_mgr_resource(res);
struct xe_res_cursor cursor;
struct scatterlist *sg;
int num_entries = 0;
int i, r;
- if (vres->used_visible_size < res->size)
+ if (vres->used_visible_size < res->lru_entity.size)
return -EOPNOTSUPP;
*sgt = kmalloc(sizeof(**sgt), GFP_KERNEL);
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 44e038276d41..f96009b02800 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -652,7 +652,7 @@ static void preempt_rebind_work_func(struct work_struct *w)
#undef retry_required
spin_lock(vm->xe->ttm.lru_lock);
- ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
+ drm_lru_bulk_move_tail(&vm->lru_bulk_move);
spin_unlock(vm->xe->ttm.lru_lock);
/* Point of no return. */
diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h
index 52e5eaed91c3..784b07660fff 100644
--- a/drivers/gpu/drm/xe/xe_vm_types.h
+++ b/drivers/gpu/drm/xe/xe_vm_types.h
@@ -149,7 +149,7 @@ struct xe_vm {
struct dma_resv resv;
/** @lru_bulk_move: Bulk LRU move list for this VM's BOs */
- struct ttm_lru_bulk_move lru_bulk_move;
+ struct drm_lru_bulk_move lru_bulk_move;
u64 size;
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index 223b198fe371..f4d939ee174c 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -118,7 +118,7 @@ struct ttm_buffer_object {
struct ttm_resource *resource;
struct ttm_tt *ttm;
bool deleted;
- struct ttm_lru_bulk_move *bulk_move;
+ struct drm_lru_bulk_move *bulk_move;
unsigned priority;
unsigned pin_count;
@@ -355,7 +355,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
struct ttm_operation_ctx *ctx);
void ttm_bo_put(struct ttm_buffer_object *bo);
void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo,
- struct ttm_lru_bulk_move *bulk);
+ struct drm_lru_bulk_move *bulk);
bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
const struct ttm_place *place);
int ttm_bo_init_reserved(struct ttm_device *bdev, struct ttm_buffer_object *bo,
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index c2528cec12e6..2401a7510ef6 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -47,6 +47,7 @@ struct io_mapping;
struct sg_table;
struct scatterlist;
struct drm_lru_mgr;
+struct drm_lru_entity;
struct ttm_resource_manager_func {
/**
@@ -143,7 +144,7 @@ struct ttm_resource_manager_func {
* @func: structure pointer implementing the range manager. See above
* @move_lock: lock for move fence
* @move: The fence of the last pipelined move operation.
- * @lru: The lru list for this memory type.
+ * @lru_mgr: The lru manager for this ttm_resource_manager
*
* This structure is used to identify and manage memory types for a device.
*/
@@ -163,14 +164,9 @@ struct ttm_resource_manager {
*/
struct dma_fence *move;
- /*
- * Protected by the bdev->lru_lock.
- */
- struct list_head lru[DRM_MAX_LRU_PRIORITY];
-
/**
* @usage: How much of the resources are used, protected by the
- * bdev->lru_lock.
+ * drm_device::lru_lock.
*/
uint64_t usage;
@@ -202,8 +198,6 @@ struct ttm_bus_placement {
* struct ttm_resource
*
* @start: Start of the allocation.
- * @size: Actual size of resource in bytes.
- * @mem_type: Resource type of the allocation.
* @placement: Placement flags.
* @bus: Placement on io bus accessible to the CPU
* @bo: weak reference to the BO, protected by ttm_device::lru_lock
@@ -214,44 +208,12 @@ struct ttm_bus_placement {
*/
struct ttm_resource {
unsigned long start;
- size_t size;
- uint32_t mem_type;
uint32_t placement;
struct ttm_bus_placement bus;
struct ttm_buffer_object *bo;
-
- /**
- * @lru: Least recently used list, see &ttm_resource_manager.lru
- */
- struct list_head lru;
struct drm_lru_entity lru_entity;
};
-/**
- * struct ttm_lru_bulk_move_pos
- *
- * @first: first res in the bulk move range
- * @last: last res in the bulk move range
- *
- * Range of resources for a lru bulk move.
- */
-struct ttm_lru_bulk_move_pos {
- struct ttm_resource *first;
- struct ttm_resource *last;
-};
-
-/**
- * struct ttm_lru_bulk_move
- *
- * @pos: first/last lru entry for resources in the each domain/priority
- *
- * Container for the current bulk move state. Should be used with
- * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move().
- */
-struct ttm_lru_bulk_move {
- struct ttm_lru_bulk_move_pos pos[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY];
-};
-
/**
* struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
* struct sg_table backed struct ttm_resource.
@@ -306,7 +268,7 @@ ttm_resource_manager_set_used(struct ttm_resource_manager *man, bool used)
int i;
for (i = 0; i < DRM_MAX_LRU_PRIORITY; i++)
- WARN_ON(!list_empty(&man->lru[i]));
+ WARN_ON(!list_empty(&man->lru_mgr->lru[i]));
man->use_type = used;
}
@@ -350,13 +312,6 @@ ttm_resource_manager_cleanup(struct ttm_resource_manager *man)
man->move = NULL;
}
-void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk);
-void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk);
-
-void ttm_resource_add_bulk_move(struct ttm_resource *res,
- struct ttm_buffer_object *bo);
-void ttm_resource_del_bulk_move(struct ttm_resource *res,
- struct ttm_buffer_object *bo);
void ttm_resource_move_to_lru_tail(struct ttm_resource *res);
void ttm_resource_init(struct ttm_buffer_object *bo,
@@ -382,6 +337,12 @@ bool ttm_resource_compat(struct ttm_resource *res,
void ttm_resource_set_bo(struct ttm_resource *res,
struct ttm_buffer_object *bo);
+void ttm_resource_add_bulk_move(struct ttm_resource *res,
+ struct ttm_buffer_object *bo);
+
+void ttm_resource_del_bulk_move(struct ttm_resource *res,
+ struct ttm_buffer_object *bo);
+
void ttm_resource_manager_init(struct ttm_resource_manager *man,
struct ttm_device *bdev,
uint64_t size);
--
2.26.3
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [RFC 02/11] drm: move lru_lock from ttm_device to drm_device
2023-11-02 4:32 ` [RFC 02/11] drm: move lru_lock from ttm_device to drm_device Oak Zeng
@ 2023-11-02 12:53 ` Christian König
2023-11-03 3:26 ` Zeng, Oak
0 siblings, 1 reply; 19+ messages in thread
From: Christian König @ 2023-11-02 12:53 UTC (permalink / raw)
To: Oak Zeng, dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty
Am 02.11.23 um 05:32 schrieb Oak Zeng:
> In the coming patches, we will share the lru list b/t
> ttm bo based memory allocator and hmm/svm based memory
> allocator. Thus lru_lock (which is used mainly to protect
> the lru list) is moved from struct ttm_device to struct
> drm_device, so this lock can be shared b/t those two
> memory allocators.
>
> To minimize code change, struct ttm_device still hold
> a weak reference of lru_lock, so ttm layer can still
> reference to this lock easily.
I would rather like to see drm_device to become the base class of
ttm_device.
Similar to how drm_gem_object is the base class of ttm_buffer_object.
That is probably a bit more work, but would also eliminate some of the
duplicate house keeping we currently have (e.g. bdev pointer in
ttm_buffer_object etc...).
Moving then stuff from the ttm_device into the drm_device becomes trivial.
Regards,
Christian.
>
> Signed-off-by: Oak Zeng <oak.zeng@intel.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 4 +-
> drivers/gpu/drm/drm_drv.c | 1 +
> drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 4 +-
> drivers/gpu/drm/ttm/ttm_bo.c | 40 +++++++++----------
> drivers/gpu/drm/ttm/ttm_device.c | 18 ++++-----
> drivers/gpu/drm/ttm/ttm_resource.c | 42 ++++++++++----------
> drivers/gpu/drm/xe/xe_bo.c | 4 +-
> drivers/gpu/drm/xe/xe_exec.c | 4 +-
> drivers/gpu/drm/xe/xe_vm.c | 4 +-
> include/drm/drm_device.h | 5 +++
> include/drm/ttm/ttm_bo.h | 4 +-
> include/drm/ttm/ttm_device.h | 4 +-
> 13 files changed, 72 insertions(+), 66 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index f5daadcec865..747bcad86d5d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -368,9 +368,9 @@ int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct drm_exec *exec,
> void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev,
> struct amdgpu_vm *vm)
> {
> - spin_lock(&adev->mman.bdev.lru_lock);
> + spin_lock(adev->mman.bdev.lru_lock);
> ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
> - spin_unlock(&adev->mman.bdev.lru_lock);
> + spin_unlock(adev->mman.bdev.lru_lock);
> }
>
> /* Create scheduler entities for page table updates */
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> index c7085a747b03..b83e1741905e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> @@ -290,9 +290,9 @@ static void amdgpu_vram_mgr_do_reserve(struct ttm_resource_manager *man)
>
> vis_usage = amdgpu_vram_mgr_vis_size(adev, block);
> atomic64_add(vis_usage, &mgr->vis_usage);
> - spin_lock(&man->bdev->lru_lock);
> + spin_lock(man->bdev->lru_lock);
> man->usage += rsv->size;
> - spin_unlock(&man->bdev->lru_lock);
> + spin_unlock(man->bdev->lru_lock);
> list_move(&rsv->blocks, &mgr->reserved_pages);
> }
> }
> diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> index 3eda026ffac6..1943c38815aa 100644
> --- a/drivers/gpu/drm/drm_drv.c
> +++ b/drivers/gpu/drm/drm_drv.c
> @@ -623,6 +623,7 @@ static int drm_dev_init(struct drm_device *dev,
>
> INIT_LIST_HEAD(&dev->managed.resources);
> spin_lock_init(&dev->managed.lock);
> + spin_lock_init(&dev->lru_lock);
>
> /* no per-device feature limits by default */
> dev->driver_features = ~0u;
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> index 9227f8146a58..c46f54f83f54 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> @@ -984,7 +984,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
> /*
> * Put on the correct LRU list depending on the MADV status
> */
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> if (shrinkable) {
> /* Try to keep shmem_tt from being considered for shrinking. */
> bo->priority = TTM_MAX_BO_PRIORITY - 1;
> @@ -1013,7 +1013,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
> }
>
> ttm_bo_move_to_lru_tail(bo);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> }
>
> /*
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index e58b7e249816..26e0555bad0c 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -68,7 +68,7 @@ static void ttm_bo_mem_space_debug(struct ttm_buffer_object *bo,
> * @bo: The buffer object.
> *
> * Move this BO to the tail of all lru lists used to lookup and reserve an
> - * object. This function must be called with struct ttm_global::lru_lock
> + * object. This function must be called with struct drm_device::lru_lock
> * held, and is used to make a BO less likely to be considered for eviction.
> */
> void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo)
> @@ -102,13 +102,13 @@ void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo,
> if (bo->bulk_move == bulk)
> return;
>
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> if (bo->resource)
> ttm_resource_del_bulk_move(bo->resource, bo);
> bo->bulk_move = bulk;
> if (bo->resource)
> ttm_resource_add_bulk_move(bo->resource, bo);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> }
> EXPORT_SYMBOL(ttm_bo_set_bulk_move);
>
> @@ -202,9 +202,9 @@ static int ttm_bo_individualize_resv(struct ttm_buffer_object *bo)
> * reference it any more. The only tricky case is the trylock on
> * the resv object while holding the lru_lock.
> */
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> bo->base.resv = &bo->base._resv;
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> }
>
> return r;
> @@ -255,7 +255,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
>
> if (unlock_resv)
> dma_resv_unlock(bo->base.resv);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
>
> lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP,
> interruptible,
> @@ -266,7 +266,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
> else if (lret == 0)
> return -EBUSY;
>
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> if (unlock_resv && !dma_resv_trylock(bo->base.resv)) {
> /*
> * We raced, and lost, someone else holds the reservation now,
> @@ -276,7 +276,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
> * delayed destruction would succeed, so just return success
> * here.
> */
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> return 0;
> }
> ret = 0;
> @@ -285,11 +285,11 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
> if (ret) {
> if (unlock_resv)
> dma_resv_unlock(bo->base.resv);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> return ret;
> }
>
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> ttm_bo_cleanup_memtype_use(bo);
>
> if (unlock_resv)
> @@ -351,7 +351,7 @@ static void ttm_bo_release(struct kref *kref)
> ttm_bo_flush_all_fences(bo);
> bo->deleted = true;
>
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
>
> /*
> * Make pinned bos immediately available to
> @@ -367,7 +367,7 @@ static void ttm_bo_release(struct kref *kref)
> }
>
> kref_init(&bo->kref);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
>
> INIT_WORK(&bo->delayed_delete, ttm_bo_delayed_delete);
> queue_work(bdev->wq, &bo->delayed_delete);
> @@ -598,7 +598,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
> bool locked = false;
> int ret;
>
> - spin_lock(&bdev->lru_lock);
> + spin_lock(bdev->lru_lock);
> ttm_resource_manager_for_each_res(man, &cursor, res) {
> bool busy;
>
> @@ -621,7 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
> if (!bo) {
> if (busy_bo && !ttm_bo_get_unless_zero(busy_bo))
> busy_bo = NULL;
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
> ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket);
> if (busy_bo)
> ttm_bo_put(busy_bo);
> @@ -635,7 +635,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
> return ret;
> }
>
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
>
> ret = ttm_bo_evict(bo, ctx);
> if (locked)
> @@ -658,11 +658,11 @@ void ttm_bo_pin(struct ttm_buffer_object *bo)
> {
> dma_resv_assert_held(bo->base.resv);
> WARN_ON_ONCE(!kref_read(&bo->kref));
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> if (bo->resource)
> ttm_resource_del_bulk_move(bo->resource, bo);
> ++bo->pin_count;
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> }
> EXPORT_SYMBOL(ttm_bo_pin);
>
> @@ -679,11 +679,11 @@ void ttm_bo_unpin(struct ttm_buffer_object *bo)
> if (WARN_ON_ONCE(!bo->pin_count))
> return;
>
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> --bo->pin_count;
> if (bo->resource)
> ttm_resource_add_bulk_move(bo->resource, bo);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> }
> EXPORT_SYMBOL(ttm_bo_unpin);
>
> @@ -1156,7 +1156,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
> }
>
> /* TODO: Cleanup the locking */
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
>
> /*
> * Move to system cached
> diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c
> index 12014788b595..d18eca86ebd6 100644
> --- a/drivers/gpu/drm/ttm/ttm_device.c
> +++ b/drivers/gpu/drm/ttm/ttm_device.c
> @@ -147,7 +147,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
> unsigned i;
> int ret;
>
> - spin_lock(&bdev->lru_lock);
> + spin_lock(bdev->lru_lock);
> for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) {
> man = ttm_manager_type(bdev, i);
> if (!man || !man->use_tt)
> @@ -169,7 +169,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
> return ret;
> }
> }
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
> return 0;
> }
> EXPORT_SYMBOL(ttm_device_swapout);
> @@ -217,7 +217,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func
> use_dma_alloc, use_dma32);
>
> bdev->vma_manager = vma_manager;
> - spin_lock_init(&bdev->lru_lock);
> + bdev->lru_lock = &drm->lru_lock;
> INIT_LIST_HEAD(&bdev->pinned);
> bdev->dev_mapping = mapping;
> mutex_lock(&ttm_global_mutex);
> @@ -244,11 +244,11 @@ void ttm_device_fini(struct ttm_device *bdev)
> drain_workqueue(bdev->wq);
> destroy_workqueue(bdev->wq);
>
> - spin_lock(&bdev->lru_lock);
> + spin_lock(bdev->lru_lock);
> for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i)
> if (list_empty(&man->lru[0]))
> pr_debug("Swap list %d was clean\n", i);
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
>
> ttm_pool_fini(&bdev->pool);
> ttm_global_release();
> @@ -260,7 +260,7 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,
> {
> struct ttm_resource *res;
>
> - spin_lock(&bdev->lru_lock);
> + spin_lock(bdev->lru_lock);
> while ((res = list_first_entry_or_null(list, typeof(*res), lru))) {
> struct ttm_buffer_object *bo = res->bo;
>
> @@ -269,15 +269,15 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,
> continue;
>
> list_del_init(&res->lru);
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
>
> if (bo->ttm)
> ttm_tt_unpopulate(bo->bdev, bo->ttm);
>
> ttm_bo_put(bo);
> - spin_lock(&bdev->lru_lock);
> + spin_lock(bdev->lru_lock);
> }
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
> }
>
> void ttm_device_clear_dma_mappings(struct ttm_device *bdev)
> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
> index 46ff9c75bb12..6ada77f51fba 100644
> --- a/drivers/gpu/drm/ttm/ttm_resource.c
> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
> @@ -48,7 +48,7 @@ EXPORT_SYMBOL(ttm_lru_bulk_move_init);
> * @bulk: bulk move structure
> *
> * Bulk move BOs to the LRU tail, only valid to use when driver makes sure that
> - * resource order never changes. Should be called with &ttm_device.lru_lock held.
> + * resource order never changes. Should be called with &drm_device.lru_lock held.
> */
> void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
> {
> @@ -62,7 +62,7 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
> if (!pos->first)
> continue;
>
> - lockdep_assert_held(&pos->first->bo->bdev->lru_lock);
> + lockdep_assert_held(pos->first->bo->bdev->lru_lock);
> dma_resv_assert_held(pos->first->bo->base.resv);
> dma_resv_assert_held(pos->last->bo->base.resv);
>
> @@ -148,7 +148,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res)
> struct ttm_buffer_object *bo = res->bo;
> struct ttm_device *bdev = bo->bdev;
>
> - lockdep_assert_held(&bo->bdev->lru_lock);
> + lockdep_assert_held(bo->bdev->lru_lock);
>
> if (bo->pin_count) {
> list_move_tail(&res->lru, &bdev->pinned);
> @@ -191,13 +191,13 @@ void ttm_resource_init(struct ttm_buffer_object *bo,
> res->bo = bo;
>
> man = ttm_manager_type(bo->bdev, place->mem_type);
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> if (bo->pin_count)
> list_add_tail(&res->lru, &bo->bdev->pinned);
> else
> list_add_tail(&res->lru, &man->lru[bo->priority]);
> man->usage += res->size;
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> }
> EXPORT_SYMBOL(ttm_resource_init);
>
> @@ -216,10 +216,10 @@ void ttm_resource_fini(struct ttm_resource_manager *man,
> {
> struct ttm_device *bdev = man->bdev;
>
> - spin_lock(&bdev->lru_lock);
> + spin_lock(bdev->lru_lock);
> list_del_init(&res->lru);
> man->usage -= res->size;
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
> }
> EXPORT_SYMBOL(ttm_resource_fini);
>
> @@ -235,9 +235,9 @@ int ttm_resource_alloc(struct ttm_buffer_object *bo,
> if (ret)
> return ret;
>
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> ttm_resource_add_bulk_move(*res_ptr, bo);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> return 0;
> }
>
> @@ -248,9 +248,9 @@ void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res)
> if (!*res)
> return;
>
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> ttm_resource_del_bulk_move(*res, bo);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> man = ttm_manager_type(bo->bdev, (*res)->mem_type);
> man->func->free(man, *res);
> *res = NULL;
> @@ -368,9 +368,9 @@ bool ttm_resource_compat(struct ttm_resource *res,
> void ttm_resource_set_bo(struct ttm_resource *res,
> struct ttm_buffer_object *bo)
> {
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> res->bo = bo;
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> }
>
> /**
> @@ -424,18 +424,18 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
> * Can't use standard list traversal since we're unlocking.
> */
>
> - spin_lock(&bdev->lru_lock);
> + spin_lock(bdev->lru_lock);
> for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) {
> while (!list_empty(&man->lru[i])) {
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
> ret = ttm_mem_evict_first(bdev, man, NULL, &ctx,
> NULL);
> if (ret)
> return ret;
> - spin_lock(&bdev->lru_lock);
> + spin_lock(bdev->lru_lock);
> }
> }
> - spin_unlock(&bdev->lru_lock);
> + spin_unlock(bdev->lru_lock);
>
> spin_lock(&man->move_lock);
> fence = dma_fence_get(man->move);
> @@ -463,9 +463,9 @@ uint64_t ttm_resource_manager_usage(struct ttm_resource_manager *man)
> {
> uint64_t usage;
>
> - spin_lock(&man->bdev->lru_lock);
> + spin_lock(man->bdev->lru_lock);
> usage = man->usage;
> - spin_unlock(&man->bdev->lru_lock);
> + spin_unlock(man->bdev->lru_lock);
> return usage;
> }
> EXPORT_SYMBOL(ttm_resource_manager_usage);
> @@ -502,7 +502,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man,
> {
> struct ttm_resource *res;
>
> - lockdep_assert_held(&man->bdev->lru_lock);
> + lockdep_assert_held(man->bdev->lru_lock);
>
> for (cursor->priority = 0; cursor->priority < TTM_MAX_BO_PRIORITY;
> ++cursor->priority)
> @@ -526,7 +526,7 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,
> struct ttm_resource_cursor *cursor,
> struct ttm_resource *res)
> {
> - lockdep_assert_held(&man->bdev->lru_lock);
> + lockdep_assert_held(man->bdev->lru_lock);
>
> list_for_each_entry_continue(res, &man->lru[cursor->priority], lru)
> return res;
> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 25fdc04627ca..827f798cccc0 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
> @@ -946,9 +946,9 @@ static bool xe_ttm_bo_lock_in_destructor(struct ttm_buffer_object *ttm_bo)
> * the ttm_bo refcount is zero at this point. So trylocking *should*
> * always succeed here, as long as we hold the lru lock.
> */
> - spin_lock(&ttm_bo->bdev->lru_lock);
> + spin_lock(ttm_bo->bdev->lru_lock);
> locked = dma_resv_trylock(ttm_bo->base.resv);
> - spin_unlock(&ttm_bo->bdev->lru_lock);
> + spin_unlock(ttm_bo->bdev->lru_lock);
> XE_WARN_ON(!locked);
>
> return locked;
> diff --git a/drivers/gpu/drm/xe/xe_exec.c b/drivers/gpu/drm/xe/xe_exec.c
> index 890fadb0a93e..dafebdfb2368 100644
> --- a/drivers/gpu/drm/xe/xe_exec.c
> +++ b/drivers/gpu/drm/xe/xe_exec.c
> @@ -370,9 +370,9 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
> xe_vm_reactivate_rebind(vm);
>
> if (!err && !xe_vm_no_dma_fences(vm)) {
> - spin_lock(&xe->ttm.lru_lock);
> + spin_lock(xe->ttm.lru_lock);
> ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
> - spin_unlock(&xe->ttm.lru_lock);
> + spin_unlock(xe->ttm.lru_lock);
> }
>
> err_repin:
> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> index a6a0f17fec1d..44e038276d41 100644
> --- a/drivers/gpu/drm/xe/xe_vm.c
> +++ b/drivers/gpu/drm/xe/xe_vm.c
> @@ -651,9 +651,9 @@ static void preempt_rebind_work_func(struct work_struct *w)
>
> #undef retry_required
>
> - spin_lock(&vm->xe->ttm.lru_lock);
> + spin_lock(vm->xe->ttm.lru_lock);
> ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
> - spin_unlock(&vm->xe->ttm.lru_lock);
> + spin_unlock(vm->xe->ttm.lru_lock);
>
> /* Point of no return. */
> arm_preempt_fences(vm, &preempt_fences);
> diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
> index 7cf4afae2e79..d0b5f42786be 100644
> --- a/include/drm/drm_device.h
> +++ b/include/drm/drm_device.h
> @@ -326,6 +326,11 @@ struct drm_device {
> */
> struct list_head debugfs_list;
>
> + /**
> + * @lru_lock: Protection for the per manager LRU and destroy lists.
> + */
> + spinlock_t lru_lock;
> +
> /* Everything below here is for legacy driver, never use! */
> /* private: */
> #if IS_ENABLED(CONFIG_DRM_LEGACY)
> diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
> index 0223a41a64b2..49f32df32204 100644
> --- a/include/drm/ttm/ttm_bo.h
> +++ b/include/drm/ttm/ttm_bo.h
> @@ -290,9 +290,9 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo);
> static inline void
> ttm_bo_move_to_lru_tail_unlocked(struct ttm_buffer_object *bo)
> {
> - spin_lock(&bo->bdev->lru_lock);
> + spin_lock(bo->bdev->lru_lock);
> ttm_bo_move_to_lru_tail(bo);
> - spin_unlock(&bo->bdev->lru_lock);
> + spin_unlock(bo->bdev->lru_lock);
> }
>
> static inline void ttm_bo_assign_mem(struct ttm_buffer_object *bo,
> diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h
> index bab868d55383..4d29e96bd892 100644
> --- a/include/drm/ttm/ttm_device.h
> +++ b/include/drm/ttm/ttm_device.h
> @@ -248,9 +248,9 @@ struct ttm_device {
> struct ttm_pool pool;
>
> /**
> - * @lru_lock: Protection for the per manager LRU and ddestroy lists.
> + * @lru_lock: Weak reference to drm_device::lru_lock.
> */
> - spinlock_t lru_lock;
> + spinlock_t *lru_lock;
>
> /**
> * @pinned: Buffer objects which are pinned and so not on any LRU list.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [RFC 03/11] drm: introduce drm evictable LRU
2023-11-02 4:32 ` [RFC 03/11] drm: introduce drm evictable LRU Oak Zeng
@ 2023-11-02 13:23 ` Christian König
2023-11-03 4:04 ` Zeng, Oak
0 siblings, 1 reply; 19+ messages in thread
From: Christian König @ 2023-11-02 13:23 UTC (permalink / raw)
To: Oak Zeng, dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, brian.welty
Am 02.11.23 um 05:32 schrieb Oak Zeng:
> drm LRU manager is introuced for resource eviction purpose. It maintains
> a LRU list per resource type.
Shouldn't we first add the possible resource types in a separate patch?
> It provides functions to add or remove
> resource to or from the list. It also provides function to retrieve the
> first entity on the LRU list.
+ functions to iterate over them.
>
> drm LRU manager also provides functions for bulk moving resources
> on the LRU lists.
>
> drm LRU manager also does very basic memory accounting function, i.e.,
> LRU manager keeps a size of this resource type and a usage member
> for how much of resource has been added to this LRU manager's LRU
> list. TTM resource manager memory accounting functoins such as
> struct ttm_resource_manager::size and struct ttm_resource_manger::usage
> are still kept. In the future, when SVM codes are in the picture,
> those memory accounting functions need some rework to consider
> the memory used by both TTM and SVM.
Please keep in mind that this structure needs to extremely small to be
usable for SVM. E.g. struct page size small :)
At least HMM based implementations ideally wants to have one for each
page or something like that.
> For one device, a global drm LRU manager per resource type should be
> created/initialized at device initialization time. Drm LRU manager
> instances are embedded in struct drm_device.
>
> It is pretty much moving some of the ttm resource manager functions
> to the drm layer. The reason of this code refactory is, we want to
> create a single LRU list for memory allocated from BO(buffer object)
> based driver and hmm/svm(shared virtual memory) based driver, thus BO
> driver and svm driver can evict memory from each other.
>
> Previously the LRU list in TTM resource manager (lru field in struct
> ttm_reource_manager) is coupled with ttm_buffer_object concept, i.e.,
> each ttm resource is backed by a ttm_buffer_object and the LRU list
> is essentially a list of ttm_buffer_object.
Actually it's the other way around. The resource provides the backing of
the BO.
And when a BO moves around it can temporary be that multiple resource
point to the same BO.
I also want to have a more advanced iterator at some point where we grab
the BO lock for keeping a reference into the LRU list. Not sure how to
do this if we don't have the BO here any more.
Need to think about that further,
Christian.
> Due to this behavior, the
> TTM resource manager can't be used by hmm/svm driver as we don't plan
> to have the BO concept for the hmm/svm implemenation. So we decouple
> the evictable LRU list from the BO concept in this series.
>
> The design goal of drm lru manager is to make it as lean as possible.
> So each lru entity only has a list node member used to link this entity
> to the evictable LRU list, and the basic resource size/type/priority
> of this entity. It doesn't have any driver specify information. A lru
> entity also has a function pointer of evict function. This is used to
> implement ttm or svm specific eviction function. A lru entity is supposed
> to be embedded in a driver specific structure such as struct
> ttm_resource, see the usage in the next patch of this series.
>
> The ttm resource manager, and some of the ttm_bo functions such as
> ttm_mem_evict_first will be rewriten using the new drm lru manager
> library, see the next patch in this series.
>
> The future hmm/svm implemenation will call lru manager function to add
> hmm/svm allocations to the shared evictable lru list.
>
> Lock design: previously ttm_resource LRU list is protected by a device
> global ttm_device::lru_lock (bdev->lru_lock in codes). This lock also
> protects ttm_buffer_object::pin_count, ttm_resource_manager::usage,
> ttm_resource::bo, ttm_device::pinned list etc. With this refactory,
> lru_lock is moved out of ttm_device and is added to struct drm_deive, so
> it can be shared b/t ttm code and svm code.
>
> Signed-off-by: Oak Zeng <oak.zeng@intel.com>
> ---
> drivers/gpu/drm/Makefile | 1 +
> drivers/gpu/drm/drm_evictable_lru.c | 232 ++++++++++++++++++++++++++++
> include/drm/drm_device.h | 7 +
> include/drm/drm_evictable_lru.h | 188 ++++++++++++++++++++++
> 4 files changed, 428 insertions(+)
> create mode 100644 drivers/gpu/drm/drm_evictable_lru.c
> create mode 100644 include/drm/drm_evictable_lru.h
>
> diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
> index 1ad88efb1752..13953b0d271b 100644
> --- a/drivers/gpu/drm/Makefile
> +++ b/drivers/gpu/drm/Makefile
> @@ -46,6 +46,7 @@ drm-y := \
> drm_vblank_work.o \
> drm_vma_manager.o \
> drm_gpuva_mgr.o \
> + drm_evictable_lru.o \
> drm_writeback.o
> drm-$(CONFIG_DRM_LEGACY) += \
> drm_agpsupport.o \
> diff --git a/drivers/gpu/drm/drm_evictable_lru.c b/drivers/gpu/drm/drm_evictable_lru.c
> new file mode 100644
> index 000000000000..2ba9105cca03
> --- /dev/null
> +++ b/drivers/gpu/drm/drm_evictable_lru.c
> @@ -0,0 +1,232 @@
> +// SPDX-License-Identifier: MIT
> +/*
> + * Copyright © 2023 Intel Corporation
> + */
> +
> +#include <linux/lockdep.h>
> +#include <linux/container_of.h>
> +#include <drm/drm_evictable_lru.h>
> +#include <drm/drm_device.h>
> +
> +static inline struct drm_lru_mgr *entity_to_mgr(struct drm_lru_entity *entity)
> +{
> + struct drm_lru_mgr *mgr;
> +
> + mgr = &entity->drm->lru_mgr[entity->mem_type];
> + BUG_ON(!mgr->used);
> +
> + return mgr;
> +}
> +
> +void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm,
> + uint32_t mem_type, uint64_t size, uint32_t priority)
> +{
> + entity->drm = drm;
> + entity->mem_type = mem_type;
> + entity->size = size;
> + entity->priority = priority;
> + INIT_LIST_HEAD(&entity->lru);
> +}
> +
> +/**
> + * drm_lru_mgr_init
> + *
> + * @mgr: drm lru manager to init
> + * @size: size of the resource managed by this manager
> + * @lock: pointer of the global lru_lock
> + *
> + * Initialize a drm lru manager
> + */
> +void drm_lru_mgr_init(struct drm_lru_mgr *mgr, uint64_t size, spinlock_t *lock)
> +{
> + unsigned j;
> +
> + mgr->used = true;
> + mgr->size = size;
> + mgr->usage = 0;
> + mgr->lru_lock = lock;
> +
> + for(j = 0; j < DRM_MAX_LRU_PRIORITY; j++)
> + INIT_LIST_HEAD(&mgr->lru[j]);
> +}
> +
> +void drm_lru_bulk_move_init(struct drm_lru_bulk_move *bulk_move)
> +{
> + memset(bulk_move, 0, sizeof(*bulk_move));
> +}
> +
> +/**
> + * drm_lru_first
> + *
> + * @mgr: drm lru manager to iterate over
> + * @cursor: cursor of the current position
> + *
> + * Returns the first entity in drm lru manager
> + */
> +struct drm_lru_entity *
> +drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor)
> +{
> + struct drm_lru_entity *entity;
> +
> + lockdep_assert_held(mgr->lru_lock);
> +
> + for(cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY; ++cursor->priority)
> + list_for_each_entry(entity, &mgr->lru[cursor->priority], lru)
> + return entity;
> +
> + return NULL;
> +}
> +
> +/**
> + * drm_lru_next
> + *
> + * @mgr: drm lru manager to iterate over
> + * @cursor: cursor of the current position
> + * @entity: the current lru entity pointer
> + *
> + * Returns the next entity from drm lru manager
> + */
> +struct drm_lru_entity *
> +drm_lru_next(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor,
> + struct drm_lru_entity *entity)
> +{
> + lockdep_assert_held(mgr->lru_lock);
> +
> + list_for_each_entry_continue(entity, &mgr->lru[cursor->priority], lru)
> + return entity;
> +
> + for(++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY; ++cursor->priority)
> + list_for_each_entry(entity, &mgr->lru[cursor->priority], lru)
> + return entity;
> +
> + return NULL;
> +}
> +
> +/**
> + * drm_lru_move_to_tail
> + *
> + * @entity: the lru entity to move to lru tail
> + *
> + * Move a lru entity to lru tail
> + */
> +void drm_lru_move_to_tail(struct drm_lru_entity * entity)
> +{
> + struct list_head *lru;
> + struct drm_lru_mgr *mgr;
> +
> + mgr = entity_to_mgr(entity);
> + lockdep_assert_held(mgr->lru_lock);
> + lru = &mgr->lru[entity->priority];
> + list_move_tail(&entity->lru, lru);
> +}
> +
> +/**
> + * drm_lru_bulk_move_range_tail
> + *
> + * @range: bulk move range
> + * @entity: lru_entity to move
> + *
> + * Move a lru_entity to the tail of a bulk move range
> + */
> +void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range,
> + struct drm_lru_entity *entity)
> +{
> + if (entity == range->last)
> + return;
> +
> + if (entity == range->first)
> + range->first = container_of(entity->lru.next, struct drm_lru_entity, lru);
> +
> + if (range->last)
> + list_move(&entity->lru, &range->last->lru);
> +
> + range->last = entity;
> +}
> +EXPORT_SYMBOL(drm_lru_bulk_move_range_tail);
> +
> +/**
> + * drm_lru_bulk_move_tail - bulk move range of entities to the LRU tail.
> + *
> + * @bulk: bulk_move structure
> + *
> + * Bulk move entities to the LRU tail, only valid to use when driver makes sure that
> + * resource order never changes.
> + */
> +void drm_lru_bulk_move_tail(struct drm_lru_bulk_move *bulk)
> +{
> +
> + unsigned i, j;
> +
> + for (i = 0; i < DRM_NUM_MEM_TYPES; ++i) {
> + for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) {
> + struct drm_lru_bulk_move_range *range = &bulk->range[i][j];
> + struct drm_lru_mgr *mgr;
> +
> + if (!range->first)
> + continue;
> +
> + mgr = entity_to_mgr(range->first);
> + lockdep_assert_held(mgr->lru_lock);
> + list_bulk_move_tail(&mgr->lru[range->first->priority], &range->first->lru,
> + &range->last->lru);
> + }
> + }
> +}
> +EXPORT_SYMBOL(drm_lru_bulk_move_tail);
> +
> +/**
> + * drm_lru_add_bulk_move
> + *
> + * @entity: the lru entity to add to the bulk move range
> + * @bulk_move: the bulk move ranges to add the entity
> + *
> + * Add a lru entity to the tail of a bulk move range
> + */
> +void drm_lru_add_bulk_move(struct drm_lru_entity *entity,
> + struct drm_lru_bulk_move *bulk_move)
> +{
> + struct drm_lru_bulk_move_range *range;
> +
> + range = &bulk_move->range[entity->mem_type][entity->priority];
> +
> + if (!range->first) {
> + range->first = entity;
> + range->last = entity;
> + return;
> + }
> +
> + drm_lru_bulk_move_range_tail(range, entity);
> +}
> +
> +EXPORT_SYMBOL(drm_lru_add_bulk_move);
> +/**
> + * drm_lru_del_bulk_move
> + *
> + * @entity: the lru entity to move from the bulk move range
> + * @bulk_move: the bulk move ranges to move the entity out of
> + *
> + * Move a lru entity out of bulk move range. This doesn't
> + * delete entity from lru manager's lru list.
> + */
> +void drm_lru_del_bulk_move(struct drm_lru_entity *entity,
> + struct drm_lru_bulk_move *bulk_move)
> +{
> + struct drm_lru_bulk_move_range *range;
> +
> + range = &bulk_move->range[entity->mem_type][entity->priority];
> +
> + if (unlikely(WARN_ON(!range->first || !range->last) ||
> + (range->first == entity && range->last == entity))) {
> + range->first = NULL;
> + range->last = NULL;
> + } else if (range->first == entity) {
> + range->first = container_of(entity->lru.next,
> + struct drm_lru_entity, lru);
> + } else if (range->last == entity) {
> + range->last = container_of(entity->lru.prev,
> + struct drm_lru_entity, lru);
> + } else {
> + list_move(&entity->lru, &range->last->lru);
> + }
> +}
> +EXPORT_SYMBOL(drm_lru_del_bulk_move);
> diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
> index d0b5f42786be..1bdcd34d3f6b 100644
> --- a/include/drm/drm_device.h
> +++ b/include/drm/drm_device.h
> @@ -8,6 +8,7 @@
>
> #include <drm/drm_legacy.h>
> #include <drm/drm_mode_config.h>
> +#include <drm/drm_evictable_lru.h>
>
> struct drm_driver;
> struct drm_minor;
> @@ -331,6 +332,12 @@ struct drm_device {
> */
> spinlock_t lru_lock;
>
> + /**
> + * @lru_mgr: Device global lru managers per memory type or memory
> + * region. Each lru manager manages a lru list of this memory type.
> + */
> + struct drm_lru_mgr lru_mgr[DRM_NUM_MEM_TYPES];
> +
> /* Everything below here is for legacy driver, never use! */
> /* private: */
> #if IS_ENABLED(CONFIG_DRM_LEGACY)
> diff --git a/include/drm/drm_evictable_lru.h b/include/drm/drm_evictable_lru.h
> new file mode 100644
> index 000000000000..3fd6bd2475d9
> --- /dev/null
> +++ b/include/drm/drm_evictable_lru.h
> @@ -0,0 +1,188 @@
> +// SPDX-License-Identifier: MIT
> +/*
> + * Copyright © 2023 Intel Corporation
> + */
> +
> +#ifndef _DRM_EVICTABLE_LRU_H_
> +#define _DRM_EVICTABLE_LRU_H_
> +
> +#include <linux/list.h>
> +#include <linux/spinlock_types.h>
> +#include <linux/spinlock.h>
> +
> +struct drm_device;
> +
> +#define DRM_MAX_LRU_PRIORITY 4
> +#define DRM_NUM_MEM_TYPES 8
> +
> +/**
> + * struct drm_lru_entity
> + *
> + * @drm: drm device that this entity belongs to
> + * @mem_type: The memory type that this entity belongs to
> + * @size: resource size of this entity
> + * @priority: The priority of this entity
> + * @lru: least recent used list node, see &drm_lru_mgr.lru
> + *
> + * This structure represents an entity in drm_lru_mgr's
> + * list. This structure is supposed to be embedded in
> + * user's data structure.
> + */
> +struct drm_lru_entity {
> + struct drm_device *drm;
> + uint32_t mem_type;
> + uint64_t size;
> + uint32_t priority;
> + struct list_head lru;
> +};
> +
> +/**
> + * struct drm_lru_mgr
> + *
> + * @used: whether this lru manager is used or not
> + * @size: size of the resource
> + * @usage: how much resource has been used
> + * @lru_lock: a weak reference to the global lru_lock
> + * @lru: least recent used list, per priority
> + *
> + * This structure maintains all the buffer allocations
> + * in a least recent used list, so a victim for eviction
> + * can be easily found.
> + */
> +struct drm_lru_mgr {
> + bool used;
> + uint64_t size;
> + uint64_t usage;
> + spinlock_t *lru_lock;
> + struct list_head lru[DRM_MAX_LRU_PRIORITY];
> +};
> +
> +/**
> + * struct drm_lru_cursor
> + *
> + * @priority: the current priority
> + *
> + * Cursor to iterate over all entities in lru manager.
> + */
> +struct drm_lru_cursor {
> + unsigned priority;
> +};
> +
> +/**
> + * struct drm_lru_bulk_move_range
> + *
> + * @first: the first entity in the range
> + * @last: the last entity in the range
> + *
> + * Range of entities on a lru list.
> + */
> +struct drm_lru_bulk_move_range
> +{
> + struct drm_lru_entity *first;
> + struct drm_lru_entity *last;
> +};
> +
> +/**
> + * struct drm_lru_bulk_move
> + *
> + * @range: An array of bulk move range, each corelates to the drm_lru_mgr's
> + * lru list of the same memory type and same priority.
> + *
> + * A collection of bulk_move range which can be used to move drm_lru_entity
> + * on the lru list in a bulk way. It should be initialized through
> + * drm_lru_bulk_move_init. Add/delete a drm_lru_entity to bulk move should call
> + * drm_lru_add_bulk_move/drm_lru_del_bulk_move.
> + */
> +struct drm_lru_bulk_move {
> + struct drm_lru_bulk_move_range range[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY];
> +};
> +
> +
> +
> +/**
> + * drm_lru_add_entity
> + *
> + * @entity: the lru entity to add
> + * @mgr: the drm lru manager
> + * @priority: specify which priority list to add
> + *
> + * Add an entity to lru list
> + */
> +static inline void drm_lru_add_entity(struct drm_lru_entity *entity,
> + struct drm_lru_mgr *mgr, unsigned priority)
> +{
> + lockdep_assert_held(mgr->lru_lock);
> + list_add_tail(&entity->lru, &mgr->lru[priority]);
> + mgr->usage += entity->size;
> +}
> +
> +/**
> + * drm_lru_remove_entity
> + *
> + * @entity: the lru entity to remove
> + * @mgr: the drm lru manager
> + *
> + * Remove an entity from lru list
> + */
> +static inline void drm_lru_remove_entity(struct drm_lru_entity *entity,
> + struct drm_lru_mgr *mgr)
> +{
> + lockdep_assert_held(mgr->lru_lock);
> + list_del_init(&entity->lru);
> + mgr->usage -= entity->size;
> +}
> +
> +/**
> + * drm_lru_mgr_fini
> + *
> + * @mgr: the drm lru manager
> + *
> + * de-initialize a lru manager
> + */
> +static inline void drm_lru_mgr_fini(struct drm_lru_mgr *mgr)
> +{
> + mgr->used = false;
> +}
> +
> +void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm,
> + uint32_t mem_type, uint64_t size, uint32_t priority);
> +
> +struct drm_lru_entity *
> +drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor);
> +
> +struct drm_lru_entity *
> +drm_lru_next(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor,
> + struct drm_lru_entity *entity);
> +
> +void drm_lru_mgr_init(struct drm_lru_mgr *mgr, uint64_t size,
> + spinlock_t *lru_lock);
> +
> +void drm_lru_move_to_tail(struct drm_lru_entity * entity);
> +
> +void drm_lru_bulk_move_init(struct drm_lru_bulk_move *bulk_move);
> +
> +
> +void drm_lru_bulk_move_tail(struct drm_lru_bulk_move *bulk);
> +
> +void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range,
> + struct drm_lru_entity *entity);
> +
> +void drm_lru_add_bulk_move(struct drm_lru_entity *entity,
> + struct drm_lru_bulk_move *bulk_move);
> +
> +void drm_lru_del_bulk_move(struct drm_lru_entity *entity,
> + struct drm_lru_bulk_move *bulk_move);
> +/**
> + * drm_lru_for_each_entity
> + *
> + * @mgr: the drm lru manager
> + * @cursor: cursor for the current position
> + * @entity: the current drm_lru_entity
> + *
> + * Iterate over all entities in drm lru manager
> + */
> +#define drm_lru_for_each_entity(mgr, cursor, entity) \
> + for (entity = drm_lru_first(mgr, cursor); entity; \
> + entity = drm_lru_next(mgr, cursor, entity))
> +
> +#endif
^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [RFC 02/11] drm: move lru_lock from ttm_device to drm_device
2023-11-02 12:53 ` Christian König
@ 2023-11-03 3:26 ` Zeng, Oak
0 siblings, 0 replies; 19+ messages in thread
From: Zeng, Oak @ 2023-11-03 3:26 UTC (permalink / raw)
To: Christian König, dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, Welty, Brian
> -----Original Message-----
> From: Christian König <christian.koenig@amd.com>
> Sent: Thursday, November 2, 2023 8:53 AM
> To: Zeng, Oak <oak.zeng@intel.com>; dri-devel@lists.freedesktop.org; intel-
> xe@lists.freedesktop.org
> Cc: Thomas.Hellstrom@linux.intel.com; felix.kuehling@amd.com;
> airlied@gmail.com; Welty, Brian <brian.welty@intel.com>
> Subject: Re: [RFC 02/11] drm: move lru_lock from ttm_device to drm_device
>
> Am 02.11.23 um 05:32 schrieb Oak Zeng:
> > In the coming patches, we will share the lru list b/t
> > ttm bo based memory allocator and hmm/svm based memory
> > allocator. Thus lru_lock (which is used mainly to protect
> > the lru list) is moved from struct ttm_device to struct
> > drm_device, so this lock can be shared b/t those two
> > memory allocators.
> >
> > To minimize code change, struct ttm_device still hold
> > a weak reference of lru_lock, so ttm layer can still
> > reference to this lock easily.
>
> I would rather like to see drm_device to become the base class of
> ttm_device.
Yah...so drm_dev is the base of ttm_device, and ttm_device is the base of amdgpu_device or xe_device...
>
> Similar to how drm_gem_object is the base class of ttm_buffer_object.
And ttm_buffer_object is base of amdgpu_bo
Pretty uniformed structure 😊
>
> That is probably a bit more work, but would also eliminate some of the
> duplicate house keeping we currently have (e.g. bdev pointer in
> ttm_buffer_object etc...).
Right, if we do that, we can cast a ttm buffer object to amdgpu_bo, then get amdgpu_device of this amdgpu_bo, then get the ttm_device. Need to write a function to get this. Yes, this way we only need to keep a amdgpu_device pointer in amdgpu_bo. Cleaner than keep bdev pointer in tbo.
>
> Moving then stuff from the ttm_device into the drm_device becomes trivial.
Agree.
Oak
>
> Regards,
> Christian.
>
> >
> > Signed-off-by: Oak Zeng <oak.zeng@intel.com>
> > ---
> > drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +-
> > drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 4 +-
> > drivers/gpu/drm/drm_drv.c | 1 +
> > drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 4 +-
> > drivers/gpu/drm/ttm/ttm_bo.c | 40 +++++++++----------
> > drivers/gpu/drm/ttm/ttm_device.c | 18 ++++-----
> > drivers/gpu/drm/ttm/ttm_resource.c | 42 ++++++++++----------
> > drivers/gpu/drm/xe/xe_bo.c | 4 +-
> > drivers/gpu/drm/xe/xe_exec.c | 4 +-
> > drivers/gpu/drm/xe/xe_vm.c | 4 +-
> > include/drm/drm_device.h | 5 +++
> > include/drm/ttm/ttm_bo.h | 4 +-
> > include/drm/ttm/ttm_device.h | 4 +-
> > 13 files changed, 72 insertions(+), 66 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > index f5daadcec865..747bcad86d5d 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > @@ -368,9 +368,9 @@ int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct
> drm_exec *exec,
> > void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev,
> > struct amdgpu_vm *vm)
> > {
> > - spin_lock(&adev->mman.bdev.lru_lock);
> > + spin_lock(adev->mman.bdev.lru_lock);
> > ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
> > - spin_unlock(&adev->mman.bdev.lru_lock);
> > + spin_unlock(adev->mman.bdev.lru_lock);
> > }
> >
> > /* Create scheduler entities for page table updates */
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> > index c7085a747b03..b83e1741905e 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> > @@ -290,9 +290,9 @@ static void amdgpu_vram_mgr_do_reserve(struct
> ttm_resource_manager *man)
> >
> > vis_usage = amdgpu_vram_mgr_vis_size(adev, block);
> > atomic64_add(vis_usage, &mgr->vis_usage);
> > - spin_lock(&man->bdev->lru_lock);
> > + spin_lock(man->bdev->lru_lock);
> > man->usage += rsv->size;
> > - spin_unlock(&man->bdev->lru_lock);
> > + spin_unlock(man->bdev->lru_lock);
> > list_move(&rsv->blocks, &mgr->reserved_pages);
> > }
> > }
> > diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> > index 3eda026ffac6..1943c38815aa 100644
> > --- a/drivers/gpu/drm/drm_drv.c
> > +++ b/drivers/gpu/drm/drm_drv.c
> > @@ -623,6 +623,7 @@ static int drm_dev_init(struct drm_device *dev,
> >
> > INIT_LIST_HEAD(&dev->managed.resources);
> > spin_lock_init(&dev->managed.lock);
> > + spin_lock_init(&dev->lru_lock);
> >
> > /* no per-device feature limits by default */
> > dev->driver_features = ~0u;
> > diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> > index 9227f8146a58..c46f54f83f54 100644
> > --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> > +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> > @@ -984,7 +984,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object
> *obj)
> > /*
> > * Put on the correct LRU list depending on the MADV status
> > */
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > if (shrinkable) {
> > /* Try to keep shmem_tt from being considered for shrinking. */
> > bo->priority = TTM_MAX_BO_PRIORITY - 1;
> > @@ -1013,7 +1013,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object
> *obj)
> > }
> >
> > ttm_bo_move_to_lru_tail(bo);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > }
> >
> > /*
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> > index e58b7e249816..26e0555bad0c 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > @@ -68,7 +68,7 @@ static void ttm_bo_mem_space_debug(struct
> ttm_buffer_object *bo,
> > * @bo: The buffer object.
> > *
> > * Move this BO to the tail of all lru lists used to lookup and reserve an
> > - * object. This function must be called with struct ttm_global::lru_lock
> > + * object. This function must be called with struct drm_device::lru_lock
> > * held, and is used to make a BO less likely to be considered for eviction.
> > */
> > void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo)
> > @@ -102,13 +102,13 @@ void ttm_bo_set_bulk_move(struct ttm_buffer_object
> *bo,
> > if (bo->bulk_move == bulk)
> > return;
> >
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > if (bo->resource)
> > ttm_resource_del_bulk_move(bo->resource, bo);
> > bo->bulk_move = bulk;
> > if (bo->resource)
> > ttm_resource_add_bulk_move(bo->resource, bo);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > }
> > EXPORT_SYMBOL(ttm_bo_set_bulk_move);
> >
> > @@ -202,9 +202,9 @@ static int ttm_bo_individualize_resv(struct
> ttm_buffer_object *bo)
> > * reference it any more. The only tricky case is the trylock on
> > * the resv object while holding the lru_lock.
> > */
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > bo->base.resv = &bo->base._resv;
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > }
> >
> > return r;
> > @@ -255,7 +255,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object
> *bo,
> >
> > if (unlock_resv)
> > dma_resv_unlock(bo->base.resv);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> >
> > lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP,
> > interruptible,
> > @@ -266,7 +266,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object
> *bo,
> > else if (lret == 0)
> > return -EBUSY;
> >
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > if (unlock_resv && !dma_resv_trylock(bo->base.resv)) {
> > /*
> > * We raced, and lost, someone else holds the reservation
> now,
> > @@ -276,7 +276,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object
> *bo,
> > * delayed destruction would succeed, so just return success
> > * here.
> > */
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > return 0;
> > }
> > ret = 0;
> > @@ -285,11 +285,11 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object
> *bo,
> > if (ret) {
> > if (unlock_resv)
> > dma_resv_unlock(bo->base.resv);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > return ret;
> > }
> >
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > ttm_bo_cleanup_memtype_use(bo);
> >
> > if (unlock_resv)
> > @@ -351,7 +351,7 @@ static void ttm_bo_release(struct kref *kref)
> > ttm_bo_flush_all_fences(bo);
> > bo->deleted = true;
> >
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> >
> > /*
> > * Make pinned bos immediately available to
> > @@ -367,7 +367,7 @@ static void ttm_bo_release(struct kref *kref)
> > }
> >
> > kref_init(&bo->kref);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> >
> > INIT_WORK(&bo->delayed_delete, ttm_bo_delayed_delete);
> > queue_work(bdev->wq, &bo->delayed_delete);
> > @@ -598,7 +598,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
> > bool locked = false;
> > int ret;
> >
> > - spin_lock(&bdev->lru_lock);
> > + spin_lock(bdev->lru_lock);
> > ttm_resource_manager_for_each_res(man, &cursor, res) {
> > bool busy;
> >
> > @@ -621,7 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
> > if (!bo) {
> > if (busy_bo && !ttm_bo_get_unless_zero(busy_bo))
> > busy_bo = NULL;
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> > ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket);
> > if (busy_bo)
> > ttm_bo_put(busy_bo);
> > @@ -635,7 +635,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
> > return ret;
> > }
> >
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> >
> > ret = ttm_bo_evict(bo, ctx);
> > if (locked)
> > @@ -658,11 +658,11 @@ void ttm_bo_pin(struct ttm_buffer_object *bo)
> > {
> > dma_resv_assert_held(bo->base.resv);
> > WARN_ON_ONCE(!kref_read(&bo->kref));
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > if (bo->resource)
> > ttm_resource_del_bulk_move(bo->resource, bo);
> > ++bo->pin_count;
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > }
> > EXPORT_SYMBOL(ttm_bo_pin);
> >
> > @@ -679,11 +679,11 @@ void ttm_bo_unpin(struct ttm_buffer_object *bo)
> > if (WARN_ON_ONCE(!bo->pin_count))
> > return;
> >
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > --bo->pin_count;
> > if (bo->resource)
> > ttm_resource_add_bulk_move(bo->resource, bo);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > }
> > EXPORT_SYMBOL(ttm_bo_unpin);
> >
> > @@ -1156,7 +1156,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo,
> struct ttm_operation_ctx *ctx,
> > }
> >
> > /* TODO: Cleanup the locking */
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> >
> > /*
> > * Move to system cached
> > diff --git a/drivers/gpu/drm/ttm/ttm_device.c
> b/drivers/gpu/drm/ttm/ttm_device.c
> > index 12014788b595..d18eca86ebd6 100644
> > --- a/drivers/gpu/drm/ttm/ttm_device.c
> > +++ b/drivers/gpu/drm/ttm/ttm_device.c
> > @@ -147,7 +147,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct
> ttm_operation_ctx *ctx,
> > unsigned i;
> > int ret;
> >
> > - spin_lock(&bdev->lru_lock);
> > + spin_lock(bdev->lru_lock);
> > for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) {
> > man = ttm_manager_type(bdev, i);
> > if (!man || !man->use_tt)
> > @@ -169,7 +169,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct
> ttm_operation_ctx *ctx,
> > return ret;
> > }
> > }
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> > return 0;
> > }
> > EXPORT_SYMBOL(ttm_device_swapout);
> > @@ -217,7 +217,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct
> ttm_device_funcs *func
> > use_dma_alloc, use_dma32);
> >
> > bdev->vma_manager = vma_manager;
> > - spin_lock_init(&bdev->lru_lock);
> > + bdev->lru_lock = &drm->lru_lock;
> > INIT_LIST_HEAD(&bdev->pinned);
> > bdev->dev_mapping = mapping;
> > mutex_lock(&ttm_global_mutex);
> > @@ -244,11 +244,11 @@ void ttm_device_fini(struct ttm_device *bdev)
> > drain_workqueue(bdev->wq);
> > destroy_workqueue(bdev->wq);
> >
> > - spin_lock(&bdev->lru_lock);
> > + spin_lock(bdev->lru_lock);
> > for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i)
> > if (list_empty(&man->lru[0]))
> > pr_debug("Swap list %d was clean\n", i);
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> >
> > ttm_pool_fini(&bdev->pool);
> > ttm_global_release();
> > @@ -260,7 +260,7 @@ static void ttm_device_clear_lru_dma_mappings(struct
> ttm_device *bdev,
> > {
> > struct ttm_resource *res;
> >
> > - spin_lock(&bdev->lru_lock);
> > + spin_lock(bdev->lru_lock);
> > while ((res = list_first_entry_or_null(list, typeof(*res), lru))) {
> > struct ttm_buffer_object *bo = res->bo;
> >
> > @@ -269,15 +269,15 @@ static void ttm_device_clear_lru_dma_mappings(struct
> ttm_device *bdev,
> > continue;
> >
> > list_del_init(&res->lru);
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> >
> > if (bo->ttm)
> > ttm_tt_unpopulate(bo->bdev, bo->ttm);
> >
> > ttm_bo_put(bo);
> > - spin_lock(&bdev->lru_lock);
> > + spin_lock(bdev->lru_lock);
> > }
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> > }
> >
> > void ttm_device_clear_dma_mappings(struct ttm_device *bdev)
> > diff --git a/drivers/gpu/drm/ttm/ttm_resource.c
> b/drivers/gpu/drm/ttm/ttm_resource.c
> > index 46ff9c75bb12..6ada77f51fba 100644
> > --- a/drivers/gpu/drm/ttm/ttm_resource.c
> > +++ b/drivers/gpu/drm/ttm/ttm_resource.c
> > @@ -48,7 +48,7 @@ EXPORT_SYMBOL(ttm_lru_bulk_move_init);
> > * @bulk: bulk move structure
> > *
> > * Bulk move BOs to the LRU tail, only valid to use when driver makes sure that
> > - * resource order never changes. Should be called with &ttm_device.lru_lock held.
> > + * resource order never changes. Should be called with &drm_device.lru_lock held.
> > */
> > void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
> > {
> > @@ -62,7 +62,7 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
> > if (!pos->first)
> > continue;
> >
> > - lockdep_assert_held(&pos->first->bo->bdev->lru_lock);
> > + lockdep_assert_held(pos->first->bo->bdev->lru_lock);
> > dma_resv_assert_held(pos->first->bo->base.resv);
> > dma_resv_assert_held(pos->last->bo->base.resv);
> >
> > @@ -148,7 +148,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource
> *res)
> > struct ttm_buffer_object *bo = res->bo;
> > struct ttm_device *bdev = bo->bdev;
> >
> > - lockdep_assert_held(&bo->bdev->lru_lock);
> > + lockdep_assert_held(bo->bdev->lru_lock);
> >
> > if (bo->pin_count) {
> > list_move_tail(&res->lru, &bdev->pinned);
> > @@ -191,13 +191,13 @@ void ttm_resource_init(struct ttm_buffer_object *bo,
> > res->bo = bo;
> >
> > man = ttm_manager_type(bo->bdev, place->mem_type);
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > if (bo->pin_count)
> > list_add_tail(&res->lru, &bo->bdev->pinned);
> > else
> > list_add_tail(&res->lru, &man->lru[bo->priority]);
> > man->usage += res->size;
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > }
> > EXPORT_SYMBOL(ttm_resource_init);
> >
> > @@ -216,10 +216,10 @@ void ttm_resource_fini(struct ttm_resource_manager
> *man,
> > {
> > struct ttm_device *bdev = man->bdev;
> >
> > - spin_lock(&bdev->lru_lock);
> > + spin_lock(bdev->lru_lock);
> > list_del_init(&res->lru);
> > man->usage -= res->size;
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> > }
> > EXPORT_SYMBOL(ttm_resource_fini);
> >
> > @@ -235,9 +235,9 @@ int ttm_resource_alloc(struct ttm_buffer_object *bo,
> > if (ret)
> > return ret;
> >
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > ttm_resource_add_bulk_move(*res_ptr, bo);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > return 0;
> > }
> >
> > @@ -248,9 +248,9 @@ void ttm_resource_free(struct ttm_buffer_object *bo,
> struct ttm_resource **res)
> > if (!*res)
> > return;
> >
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > ttm_resource_del_bulk_move(*res, bo);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > man = ttm_manager_type(bo->bdev, (*res)->mem_type);
> > man->func->free(man, *res);
> > *res = NULL;
> > @@ -368,9 +368,9 @@ bool ttm_resource_compat(struct ttm_resource *res,
> > void ttm_resource_set_bo(struct ttm_resource *res,
> > struct ttm_buffer_object *bo)
> > {
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > res->bo = bo;
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > }
> >
> > /**
> > @@ -424,18 +424,18 @@ int ttm_resource_manager_evict_all(struct ttm_device
> *bdev,
> > * Can't use standard list traversal since we're unlocking.
> > */
> >
> > - spin_lock(&bdev->lru_lock);
> > + spin_lock(bdev->lru_lock);
> > for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) {
> > while (!list_empty(&man->lru[i])) {
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> > ret = ttm_mem_evict_first(bdev, man, NULL, &ctx,
> > NULL);
> > if (ret)
> > return ret;
> > - spin_lock(&bdev->lru_lock);
> > + spin_lock(bdev->lru_lock);
> > }
> > }
> > - spin_unlock(&bdev->lru_lock);
> > + spin_unlock(bdev->lru_lock);
> >
> > spin_lock(&man->move_lock);
> > fence = dma_fence_get(man->move);
> > @@ -463,9 +463,9 @@ uint64_t ttm_resource_manager_usage(struct
> ttm_resource_manager *man)
> > {
> > uint64_t usage;
> >
> > - spin_lock(&man->bdev->lru_lock);
> > + spin_lock(man->bdev->lru_lock);
> > usage = man->usage;
> > - spin_unlock(&man->bdev->lru_lock);
> > + spin_unlock(man->bdev->lru_lock);
> > return usage;
> > }
> > EXPORT_SYMBOL(ttm_resource_manager_usage);
> > @@ -502,7 +502,7 @@ ttm_resource_manager_first(struct
> ttm_resource_manager *man,
> > {
> > struct ttm_resource *res;
> >
> > - lockdep_assert_held(&man->bdev->lru_lock);
> > + lockdep_assert_held(man->bdev->lru_lock);
> >
> > for (cursor->priority = 0; cursor->priority < TTM_MAX_BO_PRIORITY;
> > ++cursor->priority)
> > @@ -526,7 +526,7 @@ ttm_resource_manager_next(struct
> ttm_resource_manager *man,
> > struct ttm_resource_cursor *cursor,
> > struct ttm_resource *res)
> > {
> > - lockdep_assert_held(&man->bdev->lru_lock);
> > + lockdep_assert_held(man->bdev->lru_lock);
> >
> > list_for_each_entry_continue(res, &man->lru[cursor->priority], lru)
> > return res;
> > diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> > index 25fdc04627ca..827f798cccc0 100644
> > --- a/drivers/gpu/drm/xe/xe_bo.c
> > +++ b/drivers/gpu/drm/xe/xe_bo.c
> > @@ -946,9 +946,9 @@ static bool xe_ttm_bo_lock_in_destructor(struct
> ttm_buffer_object *ttm_bo)
> > * the ttm_bo refcount is zero at this point. So trylocking *should*
> > * always succeed here, as long as we hold the lru lock.
> > */
> > - spin_lock(&ttm_bo->bdev->lru_lock);
> > + spin_lock(ttm_bo->bdev->lru_lock);
> > locked = dma_resv_trylock(ttm_bo->base.resv);
> > - spin_unlock(&ttm_bo->bdev->lru_lock);
> > + spin_unlock(ttm_bo->bdev->lru_lock);
> > XE_WARN_ON(!locked);
> >
> > return locked;
> > diff --git a/drivers/gpu/drm/xe/xe_exec.c b/drivers/gpu/drm/xe/xe_exec.c
> > index 890fadb0a93e..dafebdfb2368 100644
> > --- a/drivers/gpu/drm/xe/xe_exec.c
> > +++ b/drivers/gpu/drm/xe/xe_exec.c
> > @@ -370,9 +370,9 @@ int xe_exec_ioctl(struct drm_device *dev, void *data,
> struct drm_file *file)
> > xe_vm_reactivate_rebind(vm);
> >
> > if (!err && !xe_vm_no_dma_fences(vm)) {
> > - spin_lock(&xe->ttm.lru_lock);
> > + spin_lock(xe->ttm.lru_lock);
> > ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
> > - spin_unlock(&xe->ttm.lru_lock);
> > + spin_unlock(xe->ttm.lru_lock);
> > }
> >
> > err_repin:
> > diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> > index a6a0f17fec1d..44e038276d41 100644
> > --- a/drivers/gpu/drm/xe/xe_vm.c
> > +++ b/drivers/gpu/drm/xe/xe_vm.c
> > @@ -651,9 +651,9 @@ static void preempt_rebind_work_func(struct work_struct
> *w)
> >
> > #undef retry_required
> >
> > - spin_lock(&vm->xe->ttm.lru_lock);
> > + spin_lock(vm->xe->ttm.lru_lock);
> > ttm_lru_bulk_move_tail(&vm->lru_bulk_move);
> > - spin_unlock(&vm->xe->ttm.lru_lock);
> > + spin_unlock(vm->xe->ttm.lru_lock);
> >
> > /* Point of no return. */
> > arm_preempt_fences(vm, &preempt_fences);
> > diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
> > index 7cf4afae2e79..d0b5f42786be 100644
> > --- a/include/drm/drm_device.h
> > +++ b/include/drm/drm_device.h
> > @@ -326,6 +326,11 @@ struct drm_device {
> > */
> > struct list_head debugfs_list;
> >
> > + /**
> > + * @lru_lock: Protection for the per manager LRU and destroy lists.
> > + */
> > + spinlock_t lru_lock;
> > +
> > /* Everything below here is for legacy driver, never use! */
> > /* private: */
> > #if IS_ENABLED(CONFIG_DRM_LEGACY)
> > diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
> > index 0223a41a64b2..49f32df32204 100644
> > --- a/include/drm/ttm/ttm_bo.h
> > +++ b/include/drm/ttm/ttm_bo.h
> > @@ -290,9 +290,9 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object
> *bo);
> > static inline void
> > ttm_bo_move_to_lru_tail_unlocked(struct ttm_buffer_object *bo)
> > {
> > - spin_lock(&bo->bdev->lru_lock);
> > + spin_lock(bo->bdev->lru_lock);
> > ttm_bo_move_to_lru_tail(bo);
> > - spin_unlock(&bo->bdev->lru_lock);
> > + spin_unlock(bo->bdev->lru_lock);
> > }
> >
> > static inline void ttm_bo_assign_mem(struct ttm_buffer_object *bo,
> > diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h
> > index bab868d55383..4d29e96bd892 100644
> > --- a/include/drm/ttm/ttm_device.h
> > +++ b/include/drm/ttm/ttm_device.h
> > @@ -248,9 +248,9 @@ struct ttm_device {
> > struct ttm_pool pool;
> >
> > /**
> > - * @lru_lock: Protection for the per manager LRU and ddestroy lists.
> > + * @lru_lock: Weak reference to drm_device::lru_lock.
> > */
> > - spinlock_t lru_lock;
> > + spinlock_t *lru_lock;
> >
> > /**
> > * @pinned: Buffer objects which are pinned and so not on any LRU list.
^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [RFC 03/11] drm: introduce drm evictable LRU
2023-11-02 13:23 ` Christian König
@ 2023-11-03 4:04 ` Zeng, Oak
2023-11-03 9:36 ` Christian König
0 siblings, 1 reply; 19+ messages in thread
From: Zeng, Oak @ 2023-11-03 4:04 UTC (permalink / raw)
To: Christian König, dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, Welty, Brian
> -----Original Message-----
> From: Christian König <christian.koenig@amd.com>
> Sent: Thursday, November 2, 2023 9:24 AM
> To: Zeng, Oak <oak.zeng@intel.com>; dri-devel@lists.freedesktop.org; intel-
> xe@lists.freedesktop.org
> Cc: Thomas.Hellstrom@linux.intel.com; felix.kuehling@amd.com;
> airlied@gmail.com; Welty, Brian <brian.welty@intel.com>
> Subject: Re: [RFC 03/11] drm: introduce drm evictable LRU
>
> Am 02.11.23 um 05:32 schrieb Oak Zeng:
> > drm LRU manager is introuced for resource eviction purpose. It maintains
> > a LRU list per resource type.
>
> Shouldn't we first add the possible resource types in a separate patch?
Resource type in my description message is not a good name. for resource type we have:
System memory
Gpu stolen memory
Normal gpu vram
Some device such as Intel's pvc has sub-device concept (it is called tile in xe driver). For such device, we create multiple vram type ttm resource manager and multiple lru manager, one for each sub-device...So currently we only defined a DRM_NUM_MEM_TYPES in lru manager, but not defining each resource (memory) type.
>
> > It provides functions to add or remove
> > resource to or from the list. It also provides function to retrieve the
> > first entity on the LRU list.
>
> + functions to iterate over them.
Yes basic iterate functions are implemented in this patch. Will add it to description message.
>
> >
> > drm LRU manager also provides functions for bulk moving resources
> > on the LRU lists.
> >
> > drm LRU manager also does very basic memory accounting function, i.e.,
> > LRU manager keeps a size of this resource type and a usage member
> > for how much of resource has been added to this LRU manager's LRU
> > list. TTM resource manager memory accounting functoins such as
> > struct ttm_resource_manager::size and struct ttm_resource_manger::usage
> > are still kept. In the future, when SVM codes are in the picture,
> > those memory accounting functions need some rework to consider
> > the memory used by both TTM and SVM.
>
> Please keep in mind that this structure needs to extremely small to be
> usable for SVM. E.g. struct page size small :)
>
> At least HMM based implementations ideally wants to have one for each
> page or something like that.
Very good point. List node and eviction function pointer are necessary for drm_lru_entity. I will look whether we can remove other members. At least we can remove the drm_device pointer if we make drm_device base class of ttm_device as you suggested in previous patch comment.
And mem_type and priority can use bitfield, so a dword is enough.
>
> > For one device, a global drm LRU manager per resource type should be
> > created/initialized at device initialization time. Drm LRU manager
> > instances are embedded in struct drm_device.
> >
> > It is pretty much moving some of the ttm resource manager functions
> > to the drm layer. The reason of this code refactory is, we want to
> > create a single LRU list for memory allocated from BO(buffer object)
> > based driver and hmm/svm(shared virtual memory) based driver, thus BO
> > driver and svm driver can evict memory from each other.
> >
> > Previously the LRU list in TTM resource manager (lru field in struct
> > ttm_reource_manager) is coupled with ttm_buffer_object concept, i.e.,
> > each ttm resource is backed by a ttm_buffer_object and the LRU list
> > is essentially a list of ttm_buffer_object.
>
> Actually it's the other way around. The resource provides the backing of
> the BO.
You are right. Will fix this description.
>
> And when a BO moves around it can temporary be that multiple resource
> point to the same BO.
>
> I also want to have a more advanced iterator at some point where we grab
> the BO lock for keeping a reference into the LRU list. Not sure how to
> do this if we don't have the BO here any more.
>
> Need to think about that further,
Don't quite get the what you want to do with the advanced iterator. But with this work, the lru entity is a base class of ttm_resource or any other resource struct in hmm/svm. Lru is decoupled from bo concept - this is why this lru can be shared with svm code which is bo-less.
Oak
> Christian.
>
> > Due to this behavior, the
> > TTM resource manager can't be used by hmm/svm driver as we don't plan
> > to have the BO concept for the hmm/svm implemenation. So we decouple
> > the evictable LRU list from the BO concept in this series.
> >
> > The design goal of drm lru manager is to make it as lean as possible.
> > So each lru entity only has a list node member used to link this entity
> > to the evictable LRU list, and the basic resource size/type/priority
> > of this entity. It doesn't have any driver specify information. A lru
> > entity also has a function pointer of evict function. This is used to
> > implement ttm or svm specific eviction function. A lru entity is supposed
> > to be embedded in a driver specific structure such as struct
> > ttm_resource, see the usage in the next patch of this series.
> >
> > The ttm resource manager, and some of the ttm_bo functions such as
> > ttm_mem_evict_first will be rewriten using the new drm lru manager
> > library, see the next patch in this series.
> >
> > The future hmm/svm implemenation will call lru manager function to add
> > hmm/svm allocations to the shared evictable lru list.
> >
> > Lock design: previously ttm_resource LRU list is protected by a device
> > global ttm_device::lru_lock (bdev->lru_lock in codes). This lock also
> > protects ttm_buffer_object::pin_count, ttm_resource_manager::usage,
> > ttm_resource::bo, ttm_device::pinned list etc. With this refactory,
> > lru_lock is moved out of ttm_device and is added to struct drm_deive, so
> > it can be shared b/t ttm code and svm code.
> >
> > Signed-off-by: Oak Zeng <oak.zeng@intel.com>
> > ---
> > drivers/gpu/drm/Makefile | 1 +
> > drivers/gpu/drm/drm_evictable_lru.c | 232 ++++++++++++++++++++++++++++
> > include/drm/drm_device.h | 7 +
> > include/drm/drm_evictable_lru.h | 188 ++++++++++++++++++++++
> > 4 files changed, 428 insertions(+)
> > create mode 100644 drivers/gpu/drm/drm_evictable_lru.c
> > create mode 100644 include/drm/drm_evictable_lru.h
> >
> > diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
> > index 1ad88efb1752..13953b0d271b 100644
> > --- a/drivers/gpu/drm/Makefile
> > +++ b/drivers/gpu/drm/Makefile
> > @@ -46,6 +46,7 @@ drm-y := \
> > drm_vblank_work.o \
> > drm_vma_manager.o \
> > drm_gpuva_mgr.o \
> > + drm_evictable_lru.o \
> > drm_writeback.o
> > drm-$(CONFIG_DRM_LEGACY) += \
> > drm_agpsupport.o \
> > diff --git a/drivers/gpu/drm/drm_evictable_lru.c
> b/drivers/gpu/drm/drm_evictable_lru.c
> > new file mode 100644
> > index 000000000000..2ba9105cca03
> > --- /dev/null
> > +++ b/drivers/gpu/drm/drm_evictable_lru.c
> > @@ -0,0 +1,232 @@
> > +// SPDX-License-Identifier: MIT
> > +/*
> > + * Copyright © 2023 Intel Corporation
> > + */
> > +
> > +#include <linux/lockdep.h>
> > +#include <linux/container_of.h>
> > +#include <drm/drm_evictable_lru.h>
> > +#include <drm/drm_device.h>
> > +
> > +static inline struct drm_lru_mgr *entity_to_mgr(struct drm_lru_entity *entity)
> > +{
> > + struct drm_lru_mgr *mgr;
> > +
> > + mgr = &entity->drm->lru_mgr[entity->mem_type];
> > + BUG_ON(!mgr->used);
> > +
> > + return mgr;
> > +}
> > +
> > +void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm,
> > + uint32_t mem_type, uint64_t size, uint32_t priority)
> > +{
> > + entity->drm = drm;
> > + entity->mem_type = mem_type;
> > + entity->size = size;
> > + entity->priority = priority;
> > + INIT_LIST_HEAD(&entity->lru);
> > +}
> > +
> > +/**
> > + * drm_lru_mgr_init
> > + *
> > + * @mgr: drm lru manager to init
> > + * @size: size of the resource managed by this manager
> > + * @lock: pointer of the global lru_lock
> > + *
> > + * Initialize a drm lru manager
> > + */
> > +void drm_lru_mgr_init(struct drm_lru_mgr *mgr, uint64_t size, spinlock_t *lock)
> > +{
> > + unsigned j;
> > +
> > + mgr->used = true;
> > + mgr->size = size;
> > + mgr->usage = 0;
> > + mgr->lru_lock = lock;
> > +
> > + for(j = 0; j < DRM_MAX_LRU_PRIORITY; j++)
> > + INIT_LIST_HEAD(&mgr->lru[j]);
> > +}
> > +
> > +void drm_lru_bulk_move_init(struct drm_lru_bulk_move *bulk_move)
> > +{
> > + memset(bulk_move, 0, sizeof(*bulk_move));
> > +}
> > +
> > +/**
> > + * drm_lru_first
> > + *
> > + * @mgr: drm lru manager to iterate over
> > + * @cursor: cursor of the current position
> > + *
> > + * Returns the first entity in drm lru manager
> > + */
> > +struct drm_lru_entity *
> > +drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor)
> > +{
> > + struct drm_lru_entity *entity;
> > +
> > + lockdep_assert_held(mgr->lru_lock);
> > +
> > + for(cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY;
> ++cursor->priority)
> > + list_for_each_entry(entity, &mgr->lru[cursor->priority], lru)
> > + return entity;
> > +
> > + return NULL;
> > +}
> > +
> > +/**
> > + * drm_lru_next
> > + *
> > + * @mgr: drm lru manager to iterate over
> > + * @cursor: cursor of the current position
> > + * @entity: the current lru entity pointer
> > + *
> > + * Returns the next entity from drm lru manager
> > + */
> > +struct drm_lru_entity *
> > +drm_lru_next(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor,
> > + struct drm_lru_entity *entity)
> > +{
> > + lockdep_assert_held(mgr->lru_lock);
> > +
> > + list_for_each_entry_continue(entity, &mgr->lru[cursor->priority], lru)
> > + return entity;
> > +
> > + for(++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY;
> ++cursor->priority)
> > + list_for_each_entry(entity, &mgr->lru[cursor->priority], lru)
> > + return entity;
> > +
> > + return NULL;
> > +}
> > +
> > +/**
> > + * drm_lru_move_to_tail
> > + *
> > + * @entity: the lru entity to move to lru tail
> > + *
> > + * Move a lru entity to lru tail
> > + */
> > +void drm_lru_move_to_tail(struct drm_lru_entity * entity)
> > +{
> > + struct list_head *lru;
> > + struct drm_lru_mgr *mgr;
> > +
> > + mgr = entity_to_mgr(entity);
> > + lockdep_assert_held(mgr->lru_lock);
> > + lru = &mgr->lru[entity->priority];
> > + list_move_tail(&entity->lru, lru);
> > +}
> > +
> > +/**
> > + * drm_lru_bulk_move_range_tail
> > + *
> > + * @range: bulk move range
> > + * @entity: lru_entity to move
> > + *
> > + * Move a lru_entity to the tail of a bulk move range
> > + */
> > +void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range,
> > + struct
> drm_lru_entity *entity)
> > +{
> > + if (entity == range->last)
> > + return;
> > +
> > + if (entity == range->first)
> > + range->first = container_of(entity->lru.next, struct drm_lru_entity,
> lru);
> > +
> > + if (range->last)
> > + list_move(&entity->lru, &range->last->lru);
> > +
> > + range->last = entity;
> > +}
> > +EXPORT_SYMBOL(drm_lru_bulk_move_range_tail);
> > +
> > +/**
> > + * drm_lru_bulk_move_tail - bulk move range of entities to the LRU tail.
> > + *
> > + * @bulk: bulk_move structure
> > + *
> > + * Bulk move entities to the LRU tail, only valid to use when driver makes sure that
> > + * resource order never changes.
> > + */
> > +void drm_lru_bulk_move_tail(struct drm_lru_bulk_move *bulk)
> > +{
> > +
> > + unsigned i, j;
> > +
> > + for (i = 0; i < DRM_NUM_MEM_TYPES; ++i) {
> > + for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) {
> > + struct drm_lru_bulk_move_range *range = &bulk-
> >range[i][j];
> > + struct drm_lru_mgr *mgr;
> > +
> > + if (!range->first)
> > + continue;
> > +
> > + mgr = entity_to_mgr(range->first);
> > + lockdep_assert_held(mgr->lru_lock);
> > + list_bulk_move_tail(&mgr->lru[range->first->priority],
> &range->first->lru,
> > + &range->last->lru);
> > + }
> > + }
> > +}
> > +EXPORT_SYMBOL(drm_lru_bulk_move_tail);
> > +
> > +/**
> > + * drm_lru_add_bulk_move
> > + *
> > + * @entity: the lru entity to add to the bulk move range
> > + * @bulk_move: the bulk move ranges to add the entity
> > + *
> > + * Add a lru entity to the tail of a bulk move range
> > + */
> > +void drm_lru_add_bulk_move(struct drm_lru_entity *entity,
> > + struct drm_lru_bulk_move
> *bulk_move)
> > +{
> > + struct drm_lru_bulk_move_range *range;
> > +
> > + range = &bulk_move->range[entity->mem_type][entity->priority];
> > +
> > + if (!range->first) {
> > + range->first = entity;
> > + range->last = entity;
> > + return;
> > + }
> > +
> > + drm_lru_bulk_move_range_tail(range, entity);
> > +}
> > +
> > +EXPORT_SYMBOL(drm_lru_add_bulk_move);
> > +/**
> > + * drm_lru_del_bulk_move
> > + *
> > + * @entity: the lru entity to move from the bulk move range
> > + * @bulk_move: the bulk move ranges to move the entity out of
> > + *
> > + * Move a lru entity out of bulk move range. This doesn't
> > + * delete entity from lru manager's lru list.
> > + */
> > +void drm_lru_del_bulk_move(struct drm_lru_entity *entity,
> > + struct drm_lru_bulk_move *bulk_move)
> > +{
> > + struct drm_lru_bulk_move_range *range;
> > +
> > + range = &bulk_move->range[entity->mem_type][entity->priority];
> > +
> > + if (unlikely(WARN_ON(!range->first || !range->last) ||
> > + (range->first == entity && range->last == entity))) {
> > + range->first = NULL;
> > + range->last = NULL;
> > + } else if (range->first == entity) {
> > + range->first = container_of(entity->lru.next,
> > + struct drm_lru_entity, lru);
> > + } else if (range->last == entity) {
> > + range->last = container_of(entity->lru.prev,
> > + struct drm_lru_entity, lru);
> > + } else {
> > + list_move(&entity->lru, &range->last->lru);
> > + }
> > +}
> > +EXPORT_SYMBOL(drm_lru_del_bulk_move);
> > diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
> > index d0b5f42786be..1bdcd34d3f6b 100644
> > --- a/include/drm/drm_device.h
> > +++ b/include/drm/drm_device.h
> > @@ -8,6 +8,7 @@
> >
> > #include <drm/drm_legacy.h>
> > #include <drm/drm_mode_config.h>
> > +#include <drm/drm_evictable_lru.h>
> >
> > struct drm_driver;
> > struct drm_minor;
> > @@ -331,6 +332,12 @@ struct drm_device {
> > */
> > spinlock_t lru_lock;
> >
> > + /**
> > + * @lru_mgr: Device global lru managers per memory type or memory
> > + * region. Each lru manager manages a lru list of this memory type.
> > + */
> > + struct drm_lru_mgr lru_mgr[DRM_NUM_MEM_TYPES];
> > +
> > /* Everything below here is for legacy driver, never use! */
> > /* private: */
> > #if IS_ENABLED(CONFIG_DRM_LEGACY)
> > diff --git a/include/drm/drm_evictable_lru.h b/include/drm/drm_evictable_lru.h
> > new file mode 100644
> > index 000000000000..3fd6bd2475d9
> > --- /dev/null
> > +++ b/include/drm/drm_evictable_lru.h
> > @@ -0,0 +1,188 @@
> > +// SPDX-License-Identifier: MIT
> > +/*
> > + * Copyright © 2023 Intel Corporation
> > + */
> > +
> > +#ifndef _DRM_EVICTABLE_LRU_H_
> > +#define _DRM_EVICTABLE_LRU_H_
> > +
> > +#include <linux/list.h>
> > +#include <linux/spinlock_types.h>
> > +#include <linux/spinlock.h>
> > +
> > +struct drm_device;
> > +
> > +#define DRM_MAX_LRU_PRIORITY 4
> > +#define DRM_NUM_MEM_TYPES 8
> > +
> > +/**
> > + * struct drm_lru_entity
> > + *
> > + * @drm: drm device that this entity belongs to
> > + * @mem_type: The memory type that this entity belongs to
> > + * @size: resource size of this entity
> > + * @priority: The priority of this entity
> > + * @lru: least recent used list node, see &drm_lru_mgr.lru
> > + *
> > + * This structure represents an entity in drm_lru_mgr's
> > + * list. This structure is supposed to be embedded in
> > + * user's data structure.
> > + */
> > +struct drm_lru_entity {
> > + struct drm_device *drm;
> > + uint32_t mem_type;
> > + uint64_t size;
> > + uint32_t priority;
> > + struct list_head lru;
> > +};
> > +
> > +/**
> > + * struct drm_lru_mgr
> > + *
> > + * @used: whether this lru manager is used or not
> > + * @size: size of the resource
> > + * @usage: how much resource has been used
> > + * @lru_lock: a weak reference to the global lru_lock
> > + * @lru: least recent used list, per priority
> > + *
> > + * This structure maintains all the buffer allocations
> > + * in a least recent used list, so a victim for eviction
> > + * can be easily found.
> > + */
> > +struct drm_lru_mgr {
> > + bool used;
> > + uint64_t size;
> > + uint64_t usage;
> > + spinlock_t *lru_lock;
> > + struct list_head lru[DRM_MAX_LRU_PRIORITY];
> > +};
> > +
> > +/**
> > + * struct drm_lru_cursor
> > + *
> > + * @priority: the current priority
> > + *
> > + * Cursor to iterate over all entities in lru manager.
> > + */
> > +struct drm_lru_cursor {
> > + unsigned priority;
> > +};
> > +
> > +/**
> > + * struct drm_lru_bulk_move_range
> > + *
> > + * @first: the first entity in the range
> > + * @last: the last entity in the range
> > + *
> > + * Range of entities on a lru list.
> > + */
> > +struct drm_lru_bulk_move_range
> > +{
> > + struct drm_lru_entity *first;
> > + struct drm_lru_entity *last;
> > +};
> > +
> > +/**
> > + * struct drm_lru_bulk_move
> > + *
> > + * @range: An array of bulk move range, each corelates to the drm_lru_mgr's
> > + * lru list of the same memory type and same priority.
> > + *
> > + * A collection of bulk_move range which can be used to move drm_lru_entity
> > + * on the lru list in a bulk way. It should be initialized through
> > + * drm_lru_bulk_move_init. Add/delete a drm_lru_entity to bulk move should call
> > + * drm_lru_add_bulk_move/drm_lru_del_bulk_move.
> > + */
> > +struct drm_lru_bulk_move {
> > + struct drm_lru_bulk_move_range
> range[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY];
> > +};
> > +
> > +
> > +
> > +/**
> > + * drm_lru_add_entity
> > + *
> > + * @entity: the lru entity to add
> > + * @mgr: the drm lru manager
> > + * @priority: specify which priority list to add
> > + *
> > + * Add an entity to lru list
> > + */
> > +static inline void drm_lru_add_entity(struct drm_lru_entity *entity,
> > + struct drm_lru_mgr *mgr, unsigned priority)
> > +{
> > + lockdep_assert_held(mgr->lru_lock);
> > + list_add_tail(&entity->lru, &mgr->lru[priority]);
> > + mgr->usage += entity->size;
> > +}
> > +
> > +/**
> > + * drm_lru_remove_entity
> > + *
> > + * @entity: the lru entity to remove
> > + * @mgr: the drm lru manager
> > + *
> > + * Remove an entity from lru list
> > + */
> > +static inline void drm_lru_remove_entity(struct drm_lru_entity *entity,
> > + struct drm_lru_mgr *mgr)
> > +{
> > + lockdep_assert_held(mgr->lru_lock);
> > + list_del_init(&entity->lru);
> > + mgr->usage -= entity->size;
> > +}
> > +
> > +/**
> > + * drm_lru_mgr_fini
> > + *
> > + * @mgr: the drm lru manager
> > + *
> > + * de-initialize a lru manager
> > + */
> > +static inline void drm_lru_mgr_fini(struct drm_lru_mgr *mgr)
> > +{
> > + mgr->used = false;
> > +}
> > +
> > +void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm,
> > + uint32_t mem_type, uint64_t size, uint32_t priority);
> > +
> > +struct drm_lru_entity *
> > +drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor);
> > +
> > +struct drm_lru_entity *
> > +drm_lru_next(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor,
> > + struct drm_lru_entity *entity);
> > +
> > +void drm_lru_mgr_init(struct drm_lru_mgr *mgr, uint64_t size,
> > + spinlock_t *lru_lock);
> > +
> > +void drm_lru_move_to_tail(struct drm_lru_entity * entity);
> > +
> > +void drm_lru_bulk_move_init(struct drm_lru_bulk_move *bulk_move);
> > +
> > +
> > +void drm_lru_bulk_move_tail(struct drm_lru_bulk_move *bulk);
> > +
> > +void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range,
> > + struct drm_lru_entity *entity);
> > +
> > +void drm_lru_add_bulk_move(struct drm_lru_entity *entity,
> > + struct drm_lru_bulk_move *bulk_move);
> > +
> > +void drm_lru_del_bulk_move(struct drm_lru_entity *entity,
> > + struct drm_lru_bulk_move *bulk_move);
> > +/**
> > + * drm_lru_for_each_entity
> > + *
> > + * @mgr: the drm lru manager
> > + * @cursor: cursor for the current position
> > + * @entity: the current drm_lru_entity
> > + *
> > + * Iterate over all entities in drm lru manager
> > + */
> > +#define drm_lru_for_each_entity(mgr, cursor, entity) \
> > + for (entity = drm_lru_first(mgr, cursor); entity; \
> > + entity = drm_lru_next(mgr, cursor, entity))
> > +
> > +#endif
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [RFC 03/11] drm: introduce drm evictable LRU
2023-11-03 4:04 ` Zeng, Oak
@ 2023-11-03 9:36 ` Christian König
2023-11-03 14:36 ` Zeng, Oak
0 siblings, 1 reply; 19+ messages in thread
From: Christian König @ 2023-11-03 9:36 UTC (permalink / raw)
To: Zeng, Oak, dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, Welty, Brian
[-- Attachment #1: Type: text/plain, Size: 1329 bytes --]
Am 03.11.23 um 05:04 schrieb Zeng, Oak:[SNIP]
> I also want to have a more advanced iterator at some point where we grab
> the BO lock for keeping a reference into the LRU list. Not sure how to
> do this if we don't have the BO here any more.
>
> Need to think about that further,
> Don't quite get the what you want to do with the advanced iterator. But with this work, the lru entity is a base class of ttm_resource or any other resource struct in hmm/svm. Lru is decoupled from bo concept - this is why this lru can be shared with svm code which is bo-less.
This is just a crazy idea I had because TTM tends to perform bad on
certain tasks.
When we start to evict something we use a callback which indicates if an
eviction is valuable or not. So it can happen that we have to skip quite
a bunch of BOs on the LRU until we found one which is worth evicting.
Not it can be that the first eviction doesn't make enough room to
fulfill the allocation requirement, in this case we currently start over
at the beginning searching for some BO to evict.
I want to avoid this by being able to have cursors into the LRU, e.g.
the next BO which can't move until we have evicted the current one.
BTW: How do you handle eviction here? I mean we can't call the evict
callback with the spinlock held easily?
Christian.
>
> Oak
>
[-- Attachment #2: Type: text/html, Size: 2050 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [RFC 03/11] drm: introduce drm evictable LRU
2023-11-03 9:36 ` Christian König
@ 2023-11-03 14:36 ` Zeng, Oak
0 siblings, 0 replies; 19+ messages in thread
From: Zeng, Oak @ 2023-11-03 14:36 UTC (permalink / raw)
To: Christian König, dri-devel, intel-xe
Cc: Thomas.Hellstrom, felix.kuehling, Welty, Brian
[-- Attachment #1: Type: text/plain, Size: 3128 bytes --]
From: Christian König <christian.koenig@amd.com>
Sent: Friday, November 3, 2023 5:36 AM
To: Zeng, Oak <oak.zeng@intel.com>; dri-devel@lists.freedesktop.org; intel-xe@lists.freedesktop.org
Cc: Thomas.Hellstrom@linux.intel.com; felix.kuehling@amd.com; airlied@gmail.com; Welty, Brian <brian.welty@intel.com>
Subject: Re: [RFC 03/11] drm: introduce drm evictable LRU
Am 03.11.23 um 05:04 schrieb Zeng, Oak:[SNIP]
I also want to have a more advanced iterator at some point where we grab
the BO lock for keeping a reference into the LRU list. Not sure how to
do this if we don't have the BO here any more.
Need to think about that further,
Don't quite get the what you want to do with the advanced iterator. But with this work, the lru entity is a base class of ttm_resource or any other resource struct in hmm/svm. Lru is decoupled from bo concept - this is why this lru can be shared with svm code which is bo-less.
This is just a crazy idea I had because TTM tends to perform bad on certain tasks.
When we start to evict something we use a callback which indicates if an eviction is valuable or not. So it can happen that we have to skip quite a bunch of BOs on the LRU until we found one which is worth evicting.
Not it can be that the first eviction doesn't make enough room to fulfill the allocation requirement, in this case we currently start over at the beginning searching for some BO to evict.
I want to avoid this by being able to have cursors into the LRU, e.g. the next BO which can't move until we have evicted the current one.
Got you now. I didn’t know this problem so I didn’t try to fix this efficiency problem in this series. Theoretically I think we can fix this issue this way: change ttm_mem_evict_first to ttm_mem_evict_first_n and add a parameter to this function to specify how much room we want to yield; then we evict the first n objects to make enough room before return, or fail if we can’t make enough room. This scheme would need the caller of ttm_mem_evict_first to tell how much room he need – I think reasonable.
BTW: How do you handle eviction here? I mean we can't call the evict callback with the spinlock held easily?
I was actually struggling when I refactored ttm_mem_evict_first function. I moved this function to lru manager and abstracted 3 callback functions (evict_allowable/valuable, evict_entity, evict_busy_entity) – those need to relook when hmm/svm codes come in picture. I tried not to change any logic of this function – I know people worked on this function in the past 15 years so better to be very careful.
So in my current implementation, spinlock is held calling the evict_entity callback. Spinlock is unlocked before calling ttm_bo_evict in the evict_entity callback and re-held if we need to move entity in lru list. See details in patch 4 and patch 10. So it keeps exactly the original call sequence but does look awkward.
But I think you are right. We can release the spinlock in the drm_lru_evict_first function before calling evict callback.
Oak
Christian.
Oak
[-- Attachment #2: Type: text/html, Size: 6659 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 00/11] Introduce drm evictable lru
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
` (10 preceding siblings ...)
2023-11-02 4:33 ` [RFC 11/11] drm/ttm: Write ttm functions using drm lru manager functions Oak Zeng
@ 2023-12-21 13:12 ` Thomas Hellström
11 siblings, 0 replies; 19+ messages in thread
From: Thomas Hellström @ 2023-12-21 13:12 UTC (permalink / raw)
To: Oak Zeng, dri-devel, intel-xe, Christian König
Cc: felix.kuehling, brian.welty, airlied
Hi Oak, Christian
On 11/2/23 05:32, Oak Zeng wrote:
> We plan to implement xe driver's shared virtual memory
> manager (aka SVM) without buffer object concept. This
> means we won't build our shared virtual memory manager
> upon TTM infrastructure like amdgpu does.
>
> Even though this approach is more efficient, it does
> create a problem for memory eviction when there is
> memory pressure: memory allocated by SVM and memory
> allocated by TTM should be able to mutually evict
> from each other. TTM's resource manager maintains
> a LRU list for each memory type and this list is used
> to pick up the memory eviction victim. Since we don't
> use TTM for SVM implementation, SVM allocated memory
> can't be added to TTM resource manager's LRU list. Thus
> SVM allocated memory and TTM allocated memory are not
> mutually evictable.
>
> See more discussion on this topic here:
> https://www.spinics.net/lists/dri-devel/msg410740.html
>
> This series solve this problem by creating a shared
> LRU list b/t SVM and TTM, or any other resource manager.
>
> The basic idea is, abstract a drm_lru_entity structure
> which is supposed to be embedded in ttm_resource structure,
> or any other resource manager. The resource LRU list is a
> list of drm_lru_entity. drm_lru_entity has eviction function
> pointers which can be used to call back drivers' specific
> eviction function to evict a memory resource.
>
> Introduce global drm_lru_manager to struct drm_device
> to manage LRU lists. Each memory type or memory region
> can have a LRU list. TTM resource manager's LRU list functions
> including bulk move functions are moved to drm lru manager.
> drm lru manager provides a evict_first function to evict
> the first memory resource from LRU list. This function can
> be called from TTM, SVM or any other resource manager, so
> all the memory allocated in the drm sub-system can be mutually
> evicted.
>
> The lru_lock is also moved from struct ttm_device to struct
> drm_device.
>
> Opens:
> 1) memory accounting: currently the ttm resource manager's
> memory accounting functions is kept at ttm resource manager.
> Since memory accounting should be cross TTM and SVM, it should
> be ideally in the drm lru manager layer. This will be polished
> in the future.
>
> 2) eviction callback function interface: The current eviction
> function interface is designed to meet TTM memory eviction
> requirements. When SVM is in the picture, this interface
> need to be futher tunned to meet SVM requirement also.
>
> This series is not tested and it is only compiled for xe
> driver. Some minor changes are needed for other driver
> such as amdgpu, nouveau etc. I intended to send this out
> as a request for comment series to get some early feedback,
> to see whether this is the right direction to go. I will
> futher polish this series after a direction is agreed.
>
> Oak Zeng (11):
> drm/ttm: re-parameter ttm_device_init
> drm: move lru_lock from ttm_device to drm_device
> drm: introduce drm evictable LRU
> drm: Add evict function pointer to drm lru entity
> drm: Replace ttm macros with drm macros
> drm/ttm: Set lru manager to ttm resource manager
> drm/ttm: re-parameterize a few ttm functions
> drm: Initialize drm lru manager
> drm/ttm: Use drm LRU manager iterator
> drm/ttm: Implement ttm memory evict functions
> drm/ttm: Write ttm functions using drm lru manager functions
>
> drivers/gpu/drm/Makefile | 1 +
> drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 +
> .../gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c | 6 +
> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 10 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 6 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 10 +-
> drivers/gpu/drm/drm_drv.c | 1 +
> drivers/gpu/drm/drm_evictable_lru.c | 266 ++++++++++++++++++
> drivers/gpu/drm/drm_gem_vram_helper.c | 10 +-
> drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 6 +-
> drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 10 +
> drivers/gpu/drm/i915/intel_region_ttm.c | 4 +-
> drivers/gpu/drm/i915/selftests/mock_region.c | 2 +-
> drivers/gpu/drm/loongson/lsdc_ttm.c | 10 +-
> drivers/gpu/drm/nouveau/nouveau_ttm.c | 22 +-
> drivers/gpu/drm/qxl/qxl_ttm.c | 6 +-
> drivers/gpu/drm/radeon/radeon_ttm.c | 10 +-
> drivers/gpu/drm/ttm/tests/ttm_device_test.c | 2 +-
> drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c | 2 +-
> drivers/gpu/drm/ttm/ttm_bo.c | 247 ++++++++++++----
> drivers/gpu/drm/ttm/ttm_bo_util.c | 20 +-
> drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
> drivers/gpu/drm/ttm/ttm_device.c | 55 ++--
> drivers/gpu/drm/ttm/ttm_module.h | 3 +-
> drivers/gpu/drm/ttm/ttm_range_manager.c | 14 +-
> drivers/gpu/drm/ttm/ttm_resource.c | 242 +++-------------
> drivers/gpu/drm/ttm/ttm_sys_manager.c | 8 +-
> drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 2 +-
> drivers/gpu/drm/vmwgfx/vmwgfx_bo.h | 2 +-
> drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 6 +-
> .../gpu/drm/vmwgfx/vmwgfx_system_manager.c | 6 +
> drivers/gpu/drm/xe/xe_bo.c | 48 ++--
> drivers/gpu/drm/xe/xe_bo.h | 5 +-
> drivers/gpu/drm/xe/xe_device.c | 2 +-
> drivers/gpu/drm/xe/xe_dma_buf.c | 4 +-
> drivers/gpu/drm/xe/xe_exec.c | 6 +-
> drivers/gpu/drm/xe/xe_migrate.c | 6 +-
> drivers/gpu/drm/xe/xe_res_cursor.h | 10 +-
> drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 8 +-
> drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 18 +-
> drivers/gpu/drm/xe/xe_vm.c | 6 +-
> drivers/gpu/drm/xe/xe_vm_types.h | 2 +-
> include/drm/drm_device.h | 12 +
> include/drm/drm_evictable_lru.h | 260 +++++++++++++++++
> include/drm/ttm/ttm_bo.h | 10 +-
> include/drm/ttm/ttm_device.h | 13 +-
> include/drm/ttm/ttm_range_manager.h | 17 +-
> include/drm/ttm/ttm_resource.h | 117 +++-----
> 49 files changed, 1042 insertions(+), 501 deletions(-)
> create mode 100644 drivers/gpu/drm/drm_evictable_lru.c
> create mode 100644 include/drm/drm_evictable_lru.h
>
Some additional comments after looking through this patchset and the
comments.
1) General: IMO a good start.
2) Where should this reside? I'm not a big fan of *prescribing* that a
component should be part of a specific device structure. IMO that leads
to dumping the whole drm namespace into this component rather than to
keep it isolated with as few dependencies as possible. I would make the
part that should reside in the drm device a "struct drm_lru_device", and
the resource base class a "struct drm_lru_resource". Whether the drm
device then wants to embed the struct drm_lru_device (nice if you need
this component) or have a pointer to it (nice if you don't need this
component) and whether the ttm_device should subclass the drm device
becomes pretty orthogonal to the actual implementation, and we'd avoid
hard to follow code cross-references all over the place. The drm_gpuvm
IMO did a good job with this, but I know there are other opinions on
this, and I don't want it to become a blocker.
3) Christian's comment about cursors into the LRU for LRU traversal restart
This is quickly becoming a must for the Xe driver, and a very-nice to
have for a working TTM shrinker, but I think we should start to bring
what we have in TTM currently over and do the cursors as a separate
task. I have it pretty much on top of my priority list currently. Both
downstream i915 and drm_gpuvm have a way to handle this nicely by
permutating the LRU list just before the lock needs to be released, so I
was thinking something similar. The complication would be the bulk move
tracking.
4) The struct drm_lru_evict_ctx - This one needs to be common across all
users since it's set up by the evictor and used by the evictee ops.
/Thomas
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2023-12-21 13:12 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-11-02 4:32 [PATCH 00/11] Introduce drm evictable lru Oak Zeng
2023-11-02 4:32 ` [RFC 01/11] drm/ttm: re-parameter ttm_device_init Oak Zeng
2023-11-02 4:32 ` [RFC 02/11] drm: move lru_lock from ttm_device to drm_device Oak Zeng
2023-11-02 12:53 ` Christian König
2023-11-03 3:26 ` Zeng, Oak
2023-11-02 4:32 ` [RFC 03/11] drm: introduce drm evictable LRU Oak Zeng
2023-11-02 13:23 ` Christian König
2023-11-03 4:04 ` Zeng, Oak
2023-11-03 9:36 ` Christian König
2023-11-03 14:36 ` Zeng, Oak
2023-11-02 4:32 ` [RFC 04/11] drm: Add evict function pointer to drm lru entity Oak Zeng
2023-11-02 4:33 ` [RFC 05/11] drm: Replace ttm macros with drm macros Oak Zeng
2023-11-02 4:33 ` [RFC 06/11] drm/ttm: Set lru manager to ttm resource manager Oak Zeng
2023-11-02 4:33 ` [RFC 07/11] drm/ttm: re-parameterize a few ttm functions Oak Zeng
2023-11-02 4:33 ` [RFC 08/11] drm: Initialize drm lru manager Oak Zeng
2023-11-02 4:33 ` [RFC 09/11] drm/ttm: Use drm LRU manager iterator Oak Zeng
2023-11-02 4:33 ` [RFC 10/11] drm/ttm: Implement ttm memory evict functions Oak Zeng
2023-11-02 4:33 ` [RFC 11/11] drm/ttm: Write ttm functions using drm lru manager functions Oak Zeng
2023-12-21 13:12 ` [PATCH 00/11] Introduce drm evictable lru Thomas Hellström
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).