* [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature
@ 2024-03-18 21:40 Arunpravin Paneer Selvam
2024-03-18 21:40 ` [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality Arunpravin Paneer Selvam
` (3 more replies)
0 siblings, 4 replies; 21+ messages in thread
From: Arunpravin Paneer Selvam @ 2024-03-18 21:40 UTC (permalink / raw)
To: dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, matthew.auld,
mario.limonciello, felix.kuehling, Arunpravin Paneer Selvam
- Add tracking clear page feature.
- Driver should enable the DRM_BUDDY_CLEARED flag if it
successfully clears the blocks in the free path. On the otherhand,
DRM buddy marks each block as cleared.
- Track the available cleared pages size
- If driver requests cleared memory we prefer cleared memory
but fallback to uncleared if we can't find the cleared blocks.
when driver requests uncleared memory we try to use uncleared but
fallback to cleared memory if necessary.
- When a block gets freed we clear it and mark the freed block as cleared,
when there are buddies which are cleared as well we can merge them.
Otherwise, we prefer to keep the blocks as separated.
- Add a function to support defragmentation.
v1:
- Depends on the flag check DRM_BUDDY_CLEARED, enable the block as
cleared. Else, reset the clear flag for each block in the list(Christian)
- For merging the 2 cleared blocks compare as below,
drm_buddy_is_clear(block) != drm_buddy_is_clear(buddy)(Christian)
- Defragment the memory beginning from min_order
till the required memory space is available.
v2: (Matthew)
- Add a wrapper drm_buddy_free_list_internal for the freeing of blocks
operation within drm buddy.
- Write a macro block_incompatible() to allocate the required blocks.
- Update the xe driver for the drm_buddy_free_list change in arguments.
- add a warning if the two blocks are incompatible on
defragmentation
- call full defragmentation in the fini() function
- place a condition to test if min_order is equal to 0
- replace the list with safe_reverse() variant as we might
remove the block from the list.
v3:
- fix Gitlab user reported lockup issue.
- Keep DRM_BUDDY_HEADER_CLEAR define sorted(Matthew)
- modify to pass the root order instead max_order in fini()
function(Matthew)
- change bool 1 to true(Matthew)
- add check if min_block_size is power of 2(Matthew)
- modify the min_block_size datatype to u64(Matthew)
v4:
- rename the function drm_buddy_defrag with __force_merge.
- Include __force_merge directly in drm buddy file and remove
the defrag use in amdgpu driver.
- Remove list_empty() check(Matthew)
- Remove unnecessary space, headers and placement of new variables(Matthew)
- Add a unit test case(Matthew)
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Matthew Auld <matthew.auld@intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
drivers/gpu/drm/drm_buddy.c | 427 ++++++++++++++----
drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 6 +-
drivers/gpu/drm/tests/drm_buddy_test.c | 18 +-
drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 4 +-
include/drm/drm_buddy.h | 16 +-
6 files changed, 360 insertions(+), 117 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 8db880244324..c0c851409241 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
@@ -571,7 +571,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
return 0;
error_free_blocks:
- drm_buddy_free_list(mm, &vres->blocks);
+ drm_buddy_free_list(mm, &vres->blocks, 0);
mutex_unlock(&mgr->lock);
error_fini:
ttm_resource_fini(man, &vres->base);
@@ -604,7 +604,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
amdgpu_vram_mgr_do_reserve(man);
- drm_buddy_free_list(mm, &vres->blocks);
+ drm_buddy_free_list(mm, &vres->blocks, 0);
mutex_unlock(&mgr->lock);
atomic64_sub(vis_usage, &mgr->vis_usage);
@@ -912,7 +912,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev)
kfree(rsv);
list_for_each_entry_safe(rsv, temp, &mgr->reserved_pages, blocks) {
- drm_buddy_free_list(&mgr->mm, &rsv->allocated);
+ drm_buddy_free_list(&mgr->mm, &rsv->allocated, 0);
kfree(rsv);
}
if (!adev->gmc.is_app_apu)
diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c
index c4222b886db7..625a30a6b855 100644
--- a/drivers/gpu/drm/drm_buddy.c
+++ b/drivers/gpu/drm/drm_buddy.c
@@ -38,8 +38,8 @@ static void drm_block_free(struct drm_buddy *mm,
kmem_cache_free(slab_blocks, block);
}
-static void list_insert_sorted(struct drm_buddy *mm,
- struct drm_buddy_block *block)
+static void list_insert(struct drm_buddy *mm,
+ struct drm_buddy_block *block)
{
struct drm_buddy_block *node;
struct list_head *head;
@@ -57,6 +57,16 @@ static void list_insert_sorted(struct drm_buddy *mm,
__list_add(&block->link, node->link.prev, &node->link);
}
+static void clear_reset(struct drm_buddy_block *block)
+{
+ block->header &= ~DRM_BUDDY_HEADER_CLEAR;
+}
+
+static void mark_cleared(struct drm_buddy_block *block)
+{
+ block->header |= DRM_BUDDY_HEADER_CLEAR;
+}
+
static void mark_allocated(struct drm_buddy_block *block)
{
block->header &= ~DRM_BUDDY_HEADER_STATE;
@@ -71,7 +81,7 @@ static void mark_free(struct drm_buddy *mm,
block->header &= ~DRM_BUDDY_HEADER_STATE;
block->header |= DRM_BUDDY_FREE;
- list_insert_sorted(mm, block);
+ list_insert(mm, block);
}
static void mark_split(struct drm_buddy_block *block)
@@ -82,6 +92,114 @@ static void mark_split(struct drm_buddy_block *block)
list_del(&block->link);
}
+static struct drm_buddy_block *
+__get_buddy(struct drm_buddy_block *block)
+{
+ struct drm_buddy_block *parent;
+
+ parent = block->parent;
+ if (!parent)
+ return NULL;
+
+ if (parent->left == block)
+ return parent->right;
+
+ return parent->left;
+}
+
+static unsigned int __drm_buddy_free(struct drm_buddy *mm,
+ struct drm_buddy_block *block,
+ bool force_merge)
+{
+ struct drm_buddy_block *parent;
+ unsigned int order;
+
+ while ((parent = block->parent)) {
+ struct drm_buddy_block *buddy;
+
+ buddy = __get_buddy(block);
+
+ if (!drm_buddy_block_is_free(buddy))
+ break;
+
+ if (!force_merge) {
+ /**
+ * Check the block and its buddy clear state and exit
+ * the loop if they both have the dissimilar state.
+ */
+ if (drm_buddy_block_is_clear(block) !=
+ drm_buddy_block_is_clear(buddy))
+ break;
+
+ if (drm_buddy_block_is_clear(block))
+ mark_cleared(parent);
+ }
+
+ list_del(&buddy->link);
+ if (force_merge && drm_buddy_block_is_clear(buddy))
+ mm->clear_avail -= drm_buddy_block_size(mm, buddy);
+
+ drm_block_free(mm, block);
+ drm_block_free(mm, buddy);
+
+ block = parent;
+ }
+
+ order = drm_buddy_block_order(block);
+ mark_free(mm, block);
+
+ return order;
+}
+
+static int __force_merge(struct drm_buddy *mm,
+ unsigned int min_order)
+{
+ unsigned int order;
+ int i;
+
+ if (!min_order)
+ return -ENOMEM;
+
+ if (min_order > mm->max_order)
+ return -EINVAL;
+
+ for (i = min_order - 1; i >= 0; i--) {
+ struct drm_buddy_block *block, *prev;
+
+ list_for_each_entry_safe_reverse(block, prev, &mm->free_list[i], link) {
+ struct drm_buddy_block *buddy;
+
+ if (!block->parent)
+ continue;
+
+ buddy = __get_buddy(block);
+ if (!drm_buddy_block_is_free(buddy))
+ continue;
+
+ WARN_ON(drm_buddy_block_is_clear(block) ==
+ drm_buddy_block_is_clear(buddy));
+
+ /**
+ * If the prev block is same as buddy, don't access the
+ * block in the next iteration as we would free the
+ * buddy block as part of the free function.
+ */
+ if (prev == buddy)
+ prev = list_prev_entry(prev, link);
+
+ list_del(&block->link);
+ if (drm_buddy_block_is_clear(block))
+ mm->clear_avail -= drm_buddy_block_size(mm, block);
+
+ order = __drm_buddy_free(mm, block, true);
+ if (order >= min_order)
+ return 0;
+ }
+ }
+
+ return -ENOMEM;
+}
+
/**
* drm_buddy_init - init memory manager
*
@@ -137,7 +255,7 @@ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size)
offset = 0;
i = 0;
- /*
+ /**
* Split into power-of-two blocks, in case we are given a size that is
* not itself a power-of-two.
*/
@@ -186,11 +304,21 @@ EXPORT_SYMBOL(drm_buddy_init);
*/
void drm_buddy_fini(struct drm_buddy *mm)
{
+ u64 root_size, size;
+ unsigned int order;
int i;
+ size = mm->size;
+
for (i = 0; i < mm->n_roots; ++i) {
+ order = ilog2(size) - ilog2(mm->chunk_size);
+ __force_merge(mm, order);
+
WARN_ON(!drm_buddy_block_is_free(mm->roots[i]));
drm_block_free(mm, mm->roots[i]);
+
+ root_size = mm->chunk_size << order;
+ size -= root_size;
}
WARN_ON(mm->avail != mm->size);
@@ -223,26 +351,17 @@ static int split_block(struct drm_buddy *mm,
mark_free(mm, block->left);
mark_free(mm, block->right);
+ if (drm_buddy_block_is_clear(block)) {
+ mark_cleared(block->left);
+ mark_cleared(block->right);
+ clear_reset(block);
+ }
+
mark_split(block);
return 0;
}
-static struct drm_buddy_block *
-__get_buddy(struct drm_buddy_block *block)
-{
- struct drm_buddy_block *parent;
-
- parent = block->parent;
- if (!parent)
- return NULL;
-
- if (parent->left == block)
- return parent->right;
-
- return parent->left;
-}
-
/**
* drm_get_buddy - get buddy address
*
@@ -260,30 +379,6 @@ drm_get_buddy(struct drm_buddy_block *block)
}
EXPORT_SYMBOL(drm_get_buddy);
-static void __drm_buddy_free(struct drm_buddy *mm,
- struct drm_buddy_block *block)
-{
- struct drm_buddy_block *parent;
-
- while ((parent = block->parent)) {
- struct drm_buddy_block *buddy;
-
- buddy = __get_buddy(block);
-
- if (!drm_buddy_block_is_free(buddy))
- break;
-
- list_del(&buddy->link);
-
- drm_block_free(mm, block);
- drm_block_free(mm, buddy);
-
- block = parent;
- }
-
- mark_free(mm, block);
-}
-
/**
* drm_buddy_free_block - free a block
*
@@ -295,26 +390,59 @@ void drm_buddy_free_block(struct drm_buddy *mm,
{
BUG_ON(!drm_buddy_block_is_allocated(block));
mm->avail += drm_buddy_block_size(mm, block);
- __drm_buddy_free(mm, block);
+ if (drm_buddy_block_is_clear(block))
+ mm->clear_avail += drm_buddy_block_size(mm, block);
+
+ __drm_buddy_free(mm, block, false);
}
EXPORT_SYMBOL(drm_buddy_free_block);
-/**
- * drm_buddy_free_list - free blocks
- *
- * @mm: DRM buddy manager
- * @objects: input list head to free blocks
- */
-void drm_buddy_free_list(struct drm_buddy *mm, struct list_head *objects)
+static void __drm_buddy_free_list(struct drm_buddy *mm,
+ struct list_head *objects,
+ bool mark_clear,
+ bool mark_dirty)
{
struct drm_buddy_block *block, *on;
+ WARN_ON(mark_dirty && mark_clear);
+
list_for_each_entry_safe(block, on, objects, link) {
+ if (mark_clear)
+ mark_cleared(block);
+ else if (mark_dirty)
+ clear_reset(block);
drm_buddy_free_block(mm, block);
cond_resched();
}
INIT_LIST_HEAD(objects);
}
+
+static void drm_buddy_free_list_internal(struct drm_buddy *mm,
+ struct list_head *objects)
+{
+ /**
+ * Don't touch the clear/dirty bit, since allocation is still internal
+ * at this point. For example we might have just failed part of the
+ * allocation.
+ */
+ __drm_buddy_free_list(mm, objects, false, false);
+}
+
+/**
+ * drm_buddy_free_list - free blocks
+ *
+ * @mm: DRM buddy manager
+ * @objects: input list head to free blocks
+ * @flags: optional flags like DRM_BUDDY_CLEARED
+ */
+void drm_buddy_free_list(struct drm_buddy *mm,
+ struct list_head *objects,
+ unsigned int flags)
+{
+ bool mark_clear = flags & DRM_BUDDY_CLEARED;
+
+ __drm_buddy_free_list(mm, objects, mark_clear, !mark_clear);
+}
EXPORT_SYMBOL(drm_buddy_free_list);
static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
@@ -327,10 +455,19 @@ static inline bool contains(u64 s1, u64 e1, u64 s2, u64 e2)
return s1 <= s2 && e1 >= e2;
}
+static bool block_incompatible(struct drm_buddy_block *block, unsigned int flags)
+{
+ bool needs_clear = flags & DRM_BUDDY_CLEAR_ALLOCATION;
+
+ return needs_clear != drm_buddy_block_is_clear(block);
+}
+
static struct drm_buddy_block *
-alloc_range_bias(struct drm_buddy *mm,
- u64 start, u64 end,
- unsigned int order)
+__alloc_range_bias(struct drm_buddy *mm,
+ u64 start, u64 end,
+ unsigned int order,
+ unsigned long flags,
+ bool fallback)
{
struct drm_buddy_block *block;
struct drm_buddy_block *buddy;
@@ -369,7 +506,10 @@ alloc_range_bias(struct drm_buddy *mm,
if (contains(start, end, block_start, block_end) &&
order == drm_buddy_block_order(block)) {
- /*
+ if (!fallback && block_incompatible(block, flags))
+ continue;
+
+ /**
* Find the free block within the range.
*/
if (drm_buddy_block_is_free(block))
@@ -391,7 +531,7 @@ alloc_range_bias(struct drm_buddy *mm,
return ERR_PTR(-ENOSPC);
err_undo:
- /*
+ /**
* We really don't want to leave around a bunch of split blocks, since
* bigger is better, so make sure we merge everything back before we
* free the allocated blocks.
@@ -400,30 +540,57 @@ alloc_range_bias(struct drm_buddy *mm,
if (buddy &&
(drm_buddy_block_is_free(block) &&
drm_buddy_block_is_free(buddy)))
- __drm_buddy_free(mm, block);
+ __drm_buddy_free(mm, block, false);
return ERR_PTR(err);
}
static struct drm_buddy_block *
-get_maxblock(struct drm_buddy *mm, unsigned int order)
+__drm_buddy_alloc_range_bias(struct drm_buddy *mm,
+ u64 start, u64 end,
+ unsigned int order,
+ unsigned long flags)
+{
+ struct drm_buddy_block *block;
+ bool fallback = false;
+
+ block = __alloc_range_bias(mm, start, end, order,
+ flags, fallback);
+ if (IS_ERR(block) && mm->clear_avail)
+ return __alloc_range_bias(mm, start, end, order,
+ flags, !fallback);
+
+ return block;
+}
+
+static struct drm_buddy_block *
+get_maxblock(struct drm_buddy *mm, unsigned int order,
+ unsigned long flags)
{
- struct drm_buddy_block *max_block = NULL, *node;
+ struct drm_buddy_block *max_block = NULL, *block = NULL;
unsigned int i;
for (i = order; i <= mm->max_order; ++i) {
- if (!list_empty(&mm->free_list[i])) {
- node = list_last_entry(&mm->free_list[i],
- struct drm_buddy_block,
- link);
- if (!max_block) {
- max_block = node;
+ struct drm_buddy_block *tmp_block;
+
+ list_for_each_entry_reverse(tmp_block, &mm->free_list[i], link) {
+ if (block_incompatible(tmp_block, flags))
continue;
- }
- if (drm_buddy_block_offset(node) >
- drm_buddy_block_offset(max_block)) {
- max_block = node;
- }
+ block = tmp_block;
+ break;
+ }
+
+ if (!block)
+ continue;
+
+ if (!max_block) {
+ max_block = block;
+ continue;
+ }
+
+ if (drm_buddy_block_offset(block) >
+ drm_buddy_block_offset(max_block)) {
+ max_block = block;
}
}
@@ -440,11 +607,29 @@ alloc_from_freelist(struct drm_buddy *mm,
int err;
if (flags & DRM_BUDDY_TOPDOWN_ALLOCATION) {
- block = get_maxblock(mm, order);
+ block = get_maxblock(mm, order, flags);
if (block)
/* Store the obtained block order */
tmp = drm_buddy_block_order(block);
} else {
+ for (tmp = order; tmp <= mm->max_order; ++tmp) {
+ struct drm_buddy_block *tmp_block;
+
+ list_for_each_entry_reverse(tmp_block, &mm->free_list[tmp], link) {
+ if (block_incompatible(tmp_block, flags))
+ continue;
+
+ block = tmp_block;
+ break;
+ }
+
+ if (block)
+ break;
+ }
+ }
+
+ if (!block) {
+ /* Fallback method */
for (tmp = order; tmp <= mm->max_order; ++tmp) {
if (!list_empty(&mm->free_list[tmp])) {
block = list_last_entry(&mm->free_list[tmp],
@@ -454,10 +639,10 @@ alloc_from_freelist(struct drm_buddy *mm,
break;
}
}
- }
- if (!block)
- return ERR_PTR(-ENOSPC);
+ if (!block)
+ return ERR_PTR(-ENOSPC);
+ }
BUG_ON(!drm_buddy_block_is_free(block));
@@ -473,7 +658,7 @@ alloc_from_freelist(struct drm_buddy *mm,
err_undo:
if (tmp != order)
- __drm_buddy_free(mm, block);
+ __drm_buddy_free(mm, block, false);
return ERR_PTR(err);
}
@@ -524,6 +709,8 @@ static int __alloc_range(struct drm_buddy *mm,
mark_allocated(block);
total_allocated += drm_buddy_block_size(mm, block);
mm->avail -= drm_buddy_block_size(mm, block);
+ if (drm_buddy_block_is_clear(block))
+ mm->clear_avail -= drm_buddy_block_size(mm, block);
list_add_tail(&block->link, &allocated);
continue;
}
@@ -548,7 +735,7 @@ static int __alloc_range(struct drm_buddy *mm,
return 0;
err_undo:
- /*
+ /**
* We really don't want to leave around a bunch of split blocks, since
* bigger is better, so make sure we merge everything back before we
* free the allocated blocks.
@@ -557,14 +744,14 @@ static int __alloc_range(struct drm_buddy *mm,
if (buddy &&
(drm_buddy_block_is_free(block) &&
drm_buddy_block_is_free(buddy)))
- __drm_buddy_free(mm, block);
+ __drm_buddy_free(mm, block, false);
err_free:
if (err == -ENOSPC && total_allocated_on_err) {
list_splice_tail(&allocated, blocks);
*total_allocated_on_err = total_allocated;
} else {
- drm_buddy_free_list(mm, &allocated);
+ drm_buddy_free_list_internal(mm, &allocated);
}
return err;
@@ -630,11 +817,11 @@ static int __alloc_contig_try_harder(struct drm_buddy *mm,
list_splice(&blocks_lhs, blocks);
return 0;
} else if (err != -ENOSPC) {
- drm_buddy_free_list(mm, blocks);
+ drm_buddy_free_list_internal(mm, blocks);
return err;
}
/* Free blocks for the next iteration */
- drm_buddy_free_list(mm, blocks);
+ drm_buddy_free_list_internal(mm, blocks);
}
return -ENOSPC;
@@ -690,6 +877,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
list_del(&block->link);
mark_free(mm, block);
mm->avail += drm_buddy_block_size(mm, block);
+ if (drm_buddy_block_is_clear(block))
+ mm->clear_avail += drm_buddy_block_size(mm, block);
/* Prevent recursively freeing this node */
parent = block->parent;
@@ -701,6 +890,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
if (err) {
mark_allocated(block);
mm->avail -= drm_buddy_block_size(mm, block);
+ if (drm_buddy_block_is_clear(block))
+ mm->clear_avail -= drm_buddy_block_size(mm, block);
list_add(&block->link, blocks);
}
@@ -709,13 +900,28 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
}
EXPORT_SYMBOL(drm_buddy_block_trim);
+static struct drm_buddy_block *
+__drm_buddy_alloc_blocks(struct drm_buddy *mm,
+ u64 start, u64 end,
+ unsigned int order,
+ unsigned long flags)
+{
+ if (flags & DRM_BUDDY_RANGE_ALLOCATION)
+ /* Allocate traversing within the range */
+ return __drm_buddy_alloc_range_bias(mm, start, end,
+ order, flags);
+ else
+ /* Allocate from freelist */
+ return alloc_from_freelist(mm, order, flags);
+}
+
/**
* drm_buddy_alloc_blocks - allocate power-of-two blocks
*
* @mm: DRM buddy manager to allocate from
* @start: start of the allowed range for this block
* @end: end of the allowed range for this block
- * @size: size of the allocation
+ * @size: size of the allocation in bytes
* @min_block_size: alignment of the allocation
* @blocks: output list head to add allocated blocks
* @flags: DRM_BUDDY_*_ALLOCATION flags
@@ -761,8 +967,18 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
return -EINVAL;
/* Actual range allocation */
- if (start + size == end)
- return __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
+ if (start + size == end) {
+ err = __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
+ if (err) {
+ order = ilog2(size) - ilog2(mm->chunk_size);
+ if (mm->clear_avail && !__force_merge(mm, order))
+ return __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
+
+ return err;
+ }
+
+ return err;
+ }
original_size = size;
original_min_size = min_block_size;
@@ -786,23 +1002,34 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
BUG_ON(order < min_order);
do {
- if (flags & DRM_BUDDY_RANGE_ALLOCATION)
- /* Allocate traversing within the range */
- block = alloc_range_bias(mm, start, end, order);
- else
- /* Allocate from freelist */
- block = alloc_from_freelist(mm, order, flags);
-
+ block = __drm_buddy_alloc_blocks(mm, start,
+ end,
+ order,
+ flags);
if (!IS_ERR(block))
break;
if (order-- == min_order) {
+ /**
+ * Try allocation through force merge method
+ */
+ if (mm->clear_avail && !__force_merge(mm, min_order)) {
+ block = __drm_buddy_alloc_blocks(mm, start,
+ end,
+ min_order,
+ flags);
+ if (!IS_ERR(block)) {
+ order = min_order;
+ break;
+ }
+ }
+
+ /**
+ * Try contiguous block allocation through
+ * try harder method.
+ */
if (flags & DRM_BUDDY_CONTIGUOUS_ALLOCATION &&
!(flags & DRM_BUDDY_RANGE_ALLOCATION))
- /*
- * Try contiguous block allocation through
- * try harder method
- */
return __alloc_contig_try_harder(mm,
original_size,
original_min_size,
@@ -814,6 +1041,8 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
mark_allocated(block);
mm->avail -= drm_buddy_block_size(mm, block);
+ if (drm_buddy_block_is_clear(block))
+ mm->clear_avail -= drm_buddy_block_size(mm, block);
kmemleak_update_trace(block);
list_add_tail(&block->link, &allocated);
@@ -823,7 +1052,9 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
break;
} while (1);
- /* Trim the allocated block to the required size */
+ /**
+ * Trim the allocated block to the required size
+ */
if (original_size != size) {
struct list_head *trim_list;
LIST_HEAD(temp);
@@ -852,7 +1083,7 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
return 0;
err_free:
- drm_buddy_free_list(mm, &allocated);
+ drm_buddy_free_list_internal(mm, &allocated);
return err;
}
EXPORT_SYMBOL(drm_buddy_alloc_blocks);
@@ -885,8 +1116,8 @@ void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p)
{
int order;
- drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free: %lluMiB\n",
- mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20);
+ drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free: %lluMiB, clear_free: %lluMiB\n",
+ mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20, mm->clear_avail >> 20);
for (order = mm->max_order; order >= 0; order--) {
struct drm_buddy_block *block;
diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
index 0d735d5c2b35..942345548bc3 100644
--- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
+++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
@@ -126,7 +126,7 @@ static int i915_ttm_buddy_man_alloc(struct ttm_resource_manager *man,
return 0;
err_free_blocks:
- drm_buddy_free_list(mm, &bman_res->blocks);
+ drm_buddy_free_list(mm, &bman_res->blocks, 0);
mutex_unlock(&bman->lock);
err_free_res:
ttm_resource_fini(man, &bman_res->base);
@@ -141,7 +141,7 @@ static void i915_ttm_buddy_man_free(struct ttm_resource_manager *man,
struct i915_ttm_buddy_manager *bman = to_buddy_manager(man);
mutex_lock(&bman->lock);
- drm_buddy_free_list(&bman->mm, &bman_res->blocks);
+ drm_buddy_free_list(&bman->mm, &bman_res->blocks, 0);
bman->visible_avail += bman_res->used_visible_size;
mutex_unlock(&bman->lock);
@@ -345,7 +345,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type)
ttm_set_driver_manager(bdev, type, NULL);
mutex_lock(&bman->lock);
- drm_buddy_free_list(mm, &bman->reserved);
+ drm_buddy_free_list(mm, &bman->reserved, 0);
drm_buddy_fini(mm);
bman->visible_avail += bman->visible_reserved;
WARN_ON_ONCE(bman->visible_avail != bman->visible_size);
diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
index 2f32fb2f12e7..454ad9952f56 100644
--- a/drivers/gpu/drm/tests/drm_buddy_test.c
+++ b/drivers/gpu/drm/tests/drm_buddy_test.c
@@ -64,7 +64,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
DRM_BUDDY_CONTIGUOUS_ALLOCATION),
"buddy_alloc didn't error size=%u\n", 3 * ps);
- drm_buddy_free_list(&mm, &middle);
+ drm_buddy_free_list(&mm, &middle, 0);
KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
3 * ps, ps, &allocated,
DRM_BUDDY_CONTIGUOUS_ALLOCATION),
@@ -74,7 +74,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
DRM_BUDDY_CONTIGUOUS_ALLOCATION),
"buddy_alloc didn't error size=%u\n", 2 * ps);
- drm_buddy_free_list(&mm, &right);
+ drm_buddy_free_list(&mm, &right, 0);
KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
3 * ps, ps, &allocated,
DRM_BUDDY_CONTIGUOUS_ALLOCATION),
@@ -89,7 +89,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
DRM_BUDDY_CONTIGUOUS_ALLOCATION),
"buddy_alloc hit an error size=%u\n", 2 * ps);
- drm_buddy_free_list(&mm, &left);
+ drm_buddy_free_list(&mm, &left, 0);
KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
3 * ps, ps, &allocated,
DRM_BUDDY_CONTIGUOUS_ALLOCATION),
@@ -101,7 +101,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
KUNIT_ASSERT_EQ(test, total, ps * 2 + ps * 3);
- drm_buddy_free_list(&mm, &allocated);
+ drm_buddy_free_list(&mm, &allocated, 0);
drm_buddy_fini(&mm);
}
@@ -170,7 +170,7 @@ static void drm_test_buddy_alloc_pathological(struct kunit *test)
top, max_order);
}
- drm_buddy_free_list(&mm, &holes);
+ drm_buddy_free_list(&mm, &holes, 0);
/* Nothing larger than blocks of chunk_size now available */
for (order = 1; order <= max_order; order++) {
@@ -182,7 +182,7 @@ static void drm_test_buddy_alloc_pathological(struct kunit *test)
}
list_splice_tail(&holes, &blocks);
- drm_buddy_free_list(&mm, &blocks);
+ drm_buddy_free_list(&mm, &blocks, 0);
drm_buddy_fini(&mm);
}
@@ -277,7 +277,7 @@ static void drm_test_buddy_alloc_pessimistic(struct kunit *test)
list_del(&block->link);
drm_buddy_free_block(&mm, block);
- drm_buddy_free_list(&mm, &blocks);
+ drm_buddy_free_list(&mm, &blocks, 0);
drm_buddy_fini(&mm);
}
@@ -323,7 +323,7 @@ static void drm_test_buddy_alloc_optimistic(struct kunit *test)
size, size, &tmp, flags),
"buddy_alloc unexpectedly succeeded, it should be full!");
- drm_buddy_free_list(&mm, &blocks);
+ drm_buddy_free_list(&mm, &blocks, 0);
drm_buddy_fini(&mm);
}
@@ -358,7 +358,7 @@ static void drm_test_buddy_alloc_limit(struct kunit *test)
drm_buddy_block_size(&mm, block),
BIT_ULL(mm.max_order) * PAGE_SIZE);
- drm_buddy_free_list(&mm, &allocated);
+ drm_buddy_free_list(&mm, &allocated, 0);
drm_buddy_fini(&mm);
}
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index 115ec745e502..1ad678b62c4a 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -196,7 +196,7 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
return 0;
error_free_blocks:
- drm_buddy_free_list(mm, &vres->blocks);
+ drm_buddy_free_list(mm, &vres->blocks, 0);
mutex_unlock(&mgr->lock);
error_fini:
ttm_resource_fini(man, &vres->base);
@@ -214,7 +214,7 @@ static void xe_ttm_vram_mgr_del(struct ttm_resource_manager *man,
struct drm_buddy *mm = &mgr->mm;
mutex_lock(&mgr->lock);
- drm_buddy_free_list(mm, &vres->blocks);
+ drm_buddy_free_list(mm, &vres->blocks, 0);
mgr->visible_avail += vres->used_visible_size;
mutex_unlock(&mgr->lock);
diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h
index a5b39fc01003..82570f77e817 100644
--- a/include/drm/drm_buddy.h
+++ b/include/drm/drm_buddy.h
@@ -25,6 +25,8 @@
#define DRM_BUDDY_RANGE_ALLOCATION BIT(0)
#define DRM_BUDDY_TOPDOWN_ALLOCATION BIT(1)
#define DRM_BUDDY_CONTIGUOUS_ALLOCATION BIT(2)
+#define DRM_BUDDY_CLEAR_ALLOCATION BIT(3)
+#define DRM_BUDDY_CLEARED BIT(4)
struct drm_buddy_block {
#define DRM_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
@@ -32,8 +34,9 @@ struct drm_buddy_block {
#define DRM_BUDDY_ALLOCATED (1 << 10)
#define DRM_BUDDY_FREE (2 << 10)
#define DRM_BUDDY_SPLIT (3 << 10)
+#define DRM_BUDDY_HEADER_CLEAR GENMASK_ULL(9, 9)
/* Free to be used, if needed in the future */
-#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
+#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(8, 6)
#define DRM_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
u64 header;
@@ -86,6 +89,7 @@ struct drm_buddy {
u64 chunk_size;
u64 size;
u64 avail;
+ u64 clear_avail;
};
static inline u64
@@ -112,6 +116,12 @@ drm_buddy_block_is_allocated(struct drm_buddy_block *block)
return drm_buddy_block_state(block) == DRM_BUDDY_ALLOCATED;
}
+static inline bool
+drm_buddy_block_is_clear(struct drm_buddy_block *block)
+{
+ return block->header & DRM_BUDDY_HEADER_CLEAR;
+}
+
static inline bool
drm_buddy_block_is_free(struct drm_buddy_block *block)
{
@@ -150,7 +160,9 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
void drm_buddy_free_block(struct drm_buddy *mm, struct drm_buddy_block *block);
-void drm_buddy_free_list(struct drm_buddy *mm, struct list_head *objects);
+void drm_buddy_free_list(struct drm_buddy *mm,
+ struct list_head *objects,
+ unsigned int flags);
void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p);
void drm_buddy_block_print(struct drm_buddy *mm,
--
2.25.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-18 21:40 [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature Arunpravin Paneer Selvam
@ 2024-03-18 21:40 ` Arunpravin Paneer Selvam
2024-03-19 10:28 ` Christian König
2024-03-26 13:38 ` Alex Deucher
2024-03-18 21:40 ` [PATCH v9 3/3] drm/tests: Add a test case for drm buddy clear allocation Arunpravin Paneer Selvam
` (2 subsequent siblings)
3 siblings, 2 replies; 21+ messages in thread
From: Arunpravin Paneer Selvam @ 2024-03-18 21:40 UTC (permalink / raw)
To: dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, matthew.auld,
mario.limonciello, felix.kuehling, Arunpravin Paneer Selvam
Add clear page support in vram memory region.
v1(Christian):
- Dont handle clear page as TTM flag since when moving the BO back
in from GTT again we don't need that.
- Make a specialized version of amdgpu_fill_buffer() which only
clears the VRAM areas which are not already cleared
- Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
amdgpu_object.c
v2:
- Modify the function name amdgpu_ttm_* (Alex)
- Drop the delayed parameter (Christian)
- handle amdgpu_res_cleared(&cursor) just above the size
calculation (Christian)
- Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
in the free path to properly wait for fences etc.. (Christian)
v3(Christian):
- Remove buffer clear code in VRAM manager instead change the
AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
the DRM_BUDDY_CLEARED flag.
- Remove ! from amdgpu_res_cleared(&cursor) check.
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Suggested-by: Christian König <christian.koenig@amd.com>
Acked-by: Felix Kuehling <felix.kuehling@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
.../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
6 files changed, 111 insertions(+), 13 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 8bc79924d171..c92d92b28a57 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -39,6 +39,7 @@
#include "amdgpu.h"
#include "amdgpu_trace.h"
#include "amdgpu_amdkfd.h"
+#include "amdgpu_vram_mgr.h"
/**
* DOC: amdgpu_object
@@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
if (!amdgpu_bo_support_uswc(bo->flags))
bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
- if (adev->ras_enabled)
- bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
+ bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
bo->tbo.bdev = &adev->mman.bdev;
if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
@@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
bo->tbo.resource->mem_type == TTM_PL_VRAM) {
- struct dma_fence *fence;
+ struct dma_fence *fence = NULL;
- r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence, true);
+ r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
if (unlikely(r))
goto fail_unreserve;
- dma_resv_add_fence(bo->tbo.base.resv, fence,
- DMA_RESV_USAGE_KERNEL);
- dma_fence_put(fence);
+ if (fence) {
+ dma_resv_add_fence(bo->tbo.base.resv, fence,
+ DMA_RESV_USAGE_KERNEL);
+ dma_fence_put(fence);
+ }
}
if (!bp->resv)
amdgpu_bo_unreserve(bo);
@@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo)
if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
return;
- r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv, &fence, true);
+ r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
if (!WARN_ON(r)) {
+ struct amdgpu_vram_mgr_resource *vres;
+
+ vres = to_amdgpu_vram_mgr_resource(bo->resource);
+ vres->flags |= DRM_BUDDY_CLEARED;
amdgpu_bo_fence(abo, fence, false);
dma_fence_put(fence);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
index 381101d2bf05..50fcd86e1033 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
@@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct amdgpu_res_cursor *cur, uint64_t size)
}
}
+/**
+ * amdgpu_res_cleared - check if blocks are cleared
+ *
+ * @cur: the cursor to extract the block
+ *
+ * Check if the @cur block is cleared
+ */
+static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
+{
+ struct drm_buddy_block *block;
+
+ switch (cur->mem_type) {
+ case TTM_PL_VRAM:
+ block = cur->node;
+
+ if (!amdgpu_vram_mgr_is_cleared(block))
+ return false;
+ break;
+ default:
+ return false;
+ }
+
+ return true;
+}
+
#endif
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 8722beba494e..bcbffe909b47 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
(abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
struct dma_fence *wipe_fence = NULL;
- r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
- false);
+ r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
+ false);
if (r) {
goto error;
} else if (wipe_fence) {
+ struct amdgpu_vram_mgr_resource *vres;
+
+ vres = to_amdgpu_vram_mgr_resource(bo->resource);
+ vres->flags |= DRM_BUDDY_CLEARED;
dma_fence_put(fence);
fence = wipe_fence;
}
@@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_ring *ring, uint32_t src_data,
return 0;
}
+int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
+ struct dma_resv *resv,
+ struct dma_fence **fence)
+{
+ struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
+ struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
+ struct amdgpu_res_cursor cursor;
+ struct dma_fence *f = NULL;
+ u64 addr;
+ int r;
+
+ if (!adev->mman.buffer_funcs_enabled)
+ return -EINVAL;
+
+ amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
+
+ mutex_lock(&adev->mman.gtt_window_lock);
+ while (cursor.remaining) {
+ struct dma_fence *next = NULL;
+ u64 size;
+
+ if (amdgpu_res_cleared(&cursor)) {
+ amdgpu_res_next(&cursor, cursor.size);
+ continue;
+ }
+
+ /* Never clear more than 256MiB at once to avoid timeouts */
+ size = min(cursor.size, 256ULL << 20);
+
+ r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
+ 1, ring, false, &size, &addr);
+ if (r)
+ goto err;
+
+ r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
+ &next, true, true);
+ if (r)
+ goto err;
+
+ dma_fence_put(f);
+ f = next;
+
+ amdgpu_res_next(&cursor, size);
+ }
+err:
+ mutex_unlock(&adev->mman.gtt_window_lock);
+ if (fence)
+ *fence = dma_fence_get(f);
+ dma_fence_put(f);
+
+ return r;
+}
+
int amdgpu_fill_buffer(struct amdgpu_bo *bo,
uint32_t src_data,
struct dma_resv *resv,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
index 65ec82141a8e..b404d89d52e5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
@@ -38,8 +38,6 @@
#define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
#define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
-#define AMDGPU_POISON 0xd0bed0be
-
extern const struct attribute_group amdgpu_vram_mgr_attr_group;
extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
@@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
uint64_t size, bool tmz,
struct dma_resv *resv,
struct dma_fence **f);
+int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
+ struct dma_resv *resv,
+ struct dma_fence **fence);
int amdgpu_fill_buffer(struct amdgpu_bo *bo,
uint32_t src_data,
struct dma_resv *resv,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index c0c851409241..e494f5bf136a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
@@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
{
struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
struct amdgpu_device *adev = to_amdgpu_device(mgr);
+ struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
u64 vis_usage = 0, max_bytes, min_block_size;
struct amdgpu_vram_mgr_resource *vres;
u64 size, remaining_size, lpfn, fpfn;
@@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
+ if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
+ vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
+
if (fpfn || lpfn != mgr->mm.size)
/* Allocate blocks in desired range */
vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
@@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
amdgpu_vram_mgr_do_reserve(man);
- drm_buddy_free_list(mm, &vres->blocks, 0);
+ drm_buddy_free_list(mm, &vres->blocks, vres->flags);
mutex_unlock(&mgr->lock);
atomic64_sub(vis_usage, &mgr->vis_usage);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
index 0e04e42cf809..8478522d7366 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
@@ -53,6 +53,11 @@ static inline u64 amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
return (u64)PAGE_SIZE << drm_buddy_block_order(block);
}
+static inline bool amdgpu_vram_mgr_is_cleared(struct drm_buddy_block *block)
+{
+ return drm_buddy_block_is_clear(block);
+}
+
static inline struct amdgpu_vram_mgr_resource *
to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
{
--
2.25.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 3/3] drm/tests: Add a test case for drm buddy clear allocation
2024-03-18 21:40 [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature Arunpravin Paneer Selvam
2024-03-18 21:40 ` [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality Arunpravin Paneer Selvam
@ 2024-03-18 21:40 ` Arunpravin Paneer Selvam
2024-03-26 17:46 ` Matthew Auld
2024-03-25 13:37 ` [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature Paneer Selvam, Arunpravin
2024-03-26 18:09 ` Matthew Auld
3 siblings, 1 reply; 21+ messages in thread
From: Arunpravin Paneer Selvam @ 2024-03-18 21:40 UTC (permalink / raw)
To: dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, matthew.auld,
mario.limonciello, felix.kuehling, Arunpravin Paneer Selvam
Add a new test case for the drm buddy clear and dirty
allocation.
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Suggested-by: Matthew Auld <matthew.auld@intel.com>
---
drivers/gpu/drm/tests/drm_buddy_test.c | 127 +++++++++++++++++++++++++
1 file changed, 127 insertions(+)
diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
index 454ad9952f56..d355a6e61893 100644
--- a/drivers/gpu/drm/tests/drm_buddy_test.c
+++ b/drivers/gpu/drm/tests/drm_buddy_test.c
@@ -19,6 +19,132 @@ static inline u64 get_size(int order, u64 chunk_size)
return (1 << order) * chunk_size;
}
+static void drm_test_buddy_alloc_clear(struct kunit *test)
+{
+ unsigned long n_pages, total, i = 0;
+ const unsigned long ps = SZ_4K;
+ struct drm_buddy_block *block;
+ const int max_order = 12;
+ LIST_HEAD(allocated);
+ struct drm_buddy mm;
+ unsigned int order;
+ u64 mm_size, size;
+ LIST_HEAD(dirty);
+ LIST_HEAD(clean);
+
+ mm_size = PAGE_SIZE << max_order;
+ KUNIT_EXPECT_FALSE(test, drm_buddy_init(&mm, mm_size, ps));
+
+ KUNIT_EXPECT_EQ(test, mm.max_order, max_order);
+
+ /**
+ * Idea is to allocate and free some random portion of the address space,
+ * returning those pages as non-dirty and randomly alternate between
+ * requesting dirty and non-dirty pages (not going over the limit
+ * we freed as non-dirty), putting that into two separate lists.
+ * Loop over both lists at the end checking that the dirty list
+ * is indeed all dirty pages and vice versa. Free it all again,
+ * keeping the dirty/clear status.
+ */
+ KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
+ 5 * ps, ps, &allocated,
+ DRM_BUDDY_TOPDOWN_ALLOCATION),
+ "buddy_alloc hit an error size=%u\n", 5 * ps);
+ drm_buddy_free_list(&mm, &allocated, DRM_BUDDY_CLEARED);
+
+ n_pages = 10;
+ do {
+ unsigned long flags;
+ struct list_head *list;
+ int slot = i % 2;
+
+ if (slot == 0) {
+ list = &dirty;
+ flags = 0;
+ } else if (slot == 1) {
+ list = &clean;
+ flags = DRM_BUDDY_CLEAR_ALLOCATION;
+ }
+
+ KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
+ ps, ps, list,
+ flags),
+ "buddy_alloc hit an error size=%u\n", ps);
+ } while (++i < n_pages);
+
+ list_for_each_entry(block, &clean, link)
+ KUNIT_EXPECT_EQ(test, drm_buddy_block_is_clear(block), true);
+
+ list_for_each_entry(block, &dirty, link)
+ KUNIT_EXPECT_EQ(test, drm_buddy_block_is_clear(block), false);
+
+ drm_buddy_free_list(&mm, &clean, DRM_BUDDY_CLEARED);
+
+ /**
+ * Trying to go over the clear limit for some allocation.
+ * The allocation should never fail with reasonable page-size.
+ */
+ KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
+ 10 * ps, ps, &clean,
+ DRM_BUDDY_CLEAR_ALLOCATION),
+ "buddy_alloc hit an error size=%u\n", 10 * ps);
+
+ drm_buddy_free_list(&mm, &clean, DRM_BUDDY_CLEARED);
+ drm_buddy_free_list(&mm, &dirty, 0);
+ drm_buddy_fini(&mm);
+
+ KUNIT_EXPECT_FALSE(test, drm_buddy_init(&mm, mm_size, ps));
+
+ /**
+ * Create a new mm. Intentionally fragment the address space by creating
+ * two alternating lists. Free both lists, one as dirty the other as clean.
+ * Try to allocate double the previous size with matching min_page_size. The
+ * allocation should never fail as it calls the force_merge. Also check that
+ * the page is always dirty after force_merge. Free the page as dirty, then
+ * repeat the whole thing, increment the order until we hit the max_order.
+ */
+
+ order = 1;
+ do {
+ size = PAGE_SIZE << order;
+ i = 0;
+ n_pages = mm_size / ps;
+ do {
+ struct list_head *list;
+ int slot = i % 2;
+
+ if (slot == 0)
+ list = &dirty;
+ else if (slot == 1)
+ list = &clean;
+
+ KUNIT_ASSERT_FALSE_MSG(test,
+ drm_buddy_alloc_blocks(&mm, 0, mm_size,
+ ps, ps, list, 0),
+ "buddy_alloc hit an error size=%u\n",
+ ps);
+ } while (++i < n_pages);
+
+ drm_buddy_free_list(&mm, &clean, DRM_BUDDY_CLEARED);
+ drm_buddy_free_list(&mm, &dirty, 0);
+
+ KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
+ size, size, &allocated,
+ DRM_BUDDY_CLEAR_ALLOCATION),
+ "buddy_alloc hit an error size=%u\n", size);
+ total = 0;
+ list_for_each_entry(block, &allocated, link) {
+ KUNIT_EXPECT_EQ(test, drm_buddy_block_is_clear(block), false);
+ total += drm_buddy_block_size(&mm, block);
+ }
+ KUNIT_EXPECT_EQ(test, total, size);
+
+ drm_buddy_free_list(&mm, &allocated, 0);
+ } while (++order <= max_order);
+
+ drm_buddy_fini(&mm);
+}
+
static void drm_test_buddy_alloc_contiguous(struct kunit *test)
{
const unsigned long ps = SZ_4K, mm_size = 16 * 3 * SZ_4K;
@@ -368,6 +494,7 @@ static struct kunit_case drm_buddy_tests[] = {
KUNIT_CASE(drm_test_buddy_alloc_pessimistic),
KUNIT_CASE(drm_test_buddy_alloc_pathological),
KUNIT_CASE(drm_test_buddy_alloc_contiguous),
+ KUNIT_CASE(drm_test_buddy_alloc_clear),
{}
};
--
2.25.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-18 21:40 ` [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality Arunpravin Paneer Selvam
@ 2024-03-19 10:28 ` Christian König
2024-03-19 11:41 ` Paneer Selvam, Arunpravin
2024-03-26 13:38 ` Alex Deucher
1 sibling, 1 reply; 21+ messages in thread
From: Christian König @ 2024-03-19 10:28 UTC (permalink / raw)
To: Arunpravin Paneer Selvam, dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, matthew.auld,
mario.limonciello, felix.kuehling
Am 18.03.24 um 22:40 schrieb Arunpravin Paneer Selvam:
> Add clear page support in vram memory region.
>
> v1(Christian):
> - Dont handle clear page as TTM flag since when moving the BO back
> in from GTT again we don't need that.
> - Make a specialized version of amdgpu_fill_buffer() which only
> clears the VRAM areas which are not already cleared
> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
> amdgpu_object.c
>
> v2:
> - Modify the function name amdgpu_ttm_* (Alex)
> - Drop the delayed parameter (Christian)
> - handle amdgpu_res_cleared(&cursor) just above the size
> calculation (Christian)
> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
> in the free path to properly wait for fences etc.. (Christian)
>
> v3(Christian):
> - Remove buffer clear code in VRAM manager instead change the
> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
> the DRM_BUDDY_CLEARED flag.
> - Remove ! from amdgpu_res_cleared(&cursor) check.
>
> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
Just a few nit picks below, but in general already looks good to me.
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
> 6 files changed, 111 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> index 8bc79924d171..c92d92b28a57 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> @@ -39,6 +39,7 @@
> #include "amdgpu.h"
> #include "amdgpu_trace.h"
> #include "amdgpu_amdkfd.h"
> +#include "amdgpu_vram_mgr.h"
>
> /**
> * DOC: amdgpu_object
> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
> if (!amdgpu_bo_support_uswc(bo->flags))
> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
>
> - if (adev->ras_enabled)
> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>
> bo->tbo.bdev = &adev->mman.bdev;
> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>
> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
> - struct dma_fence *fence;
> + struct dma_fence *fence = NULL;
>
> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence, true);
> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
> if (unlikely(r))
> goto fail_unreserve;
>
> - dma_resv_add_fence(bo->tbo.base.resv, fence,
> - DMA_RESV_USAGE_KERNEL);
> - dma_fence_put(fence);
> + if (fence) {
> + dma_resv_add_fence(bo->tbo.base.resv, fence,
> + DMA_RESV_USAGE_KERNEL);
> + dma_fence_put(fence);
> + }
> }
> if (!bp->resv)
> amdgpu_bo_unreserve(bo);
> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo)
> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
> return;
>
> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv, &fence, true);
> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
> if (!WARN_ON(r)) {
> + struct amdgpu_vram_mgr_resource *vres;
> +
> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
> + vres->flags |= DRM_BUDDY_CLEARED;
Those lines should probably be in the VRAM manager.
> amdgpu_bo_fence(abo, fence, false);
> dma_fence_put(fence);
> }
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> index 381101d2bf05..50fcd86e1033 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct amdgpu_res_cursor *cur, uint64_t size)
> }
> }
>
> +/**
> + * amdgpu_res_cleared - check if blocks are cleared
> + *
> + * @cur: the cursor to extract the block
> + *
> + * Check if the @cur block is cleared
> + */
> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
> +{
> + struct drm_buddy_block *block;
> +
> + switch (cur->mem_type) {
> + case TTM_PL_VRAM:
> + block = cur->node;
> +
> + if (!amdgpu_vram_mgr_is_cleared(block))
> + return false;
> + break;
> + default:
> + return false;
> + }
> +
> + return true;
> +}
> +
> #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> index 8722beba494e..bcbffe909b47 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
> struct dma_fence *wipe_fence = NULL;
>
> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
> - false);
> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
> + false);
> if (r) {
> goto error;
> } else if (wipe_fence) {
> + struct amdgpu_vram_mgr_resource *vres;
> +
> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
> + vres->flags |= DRM_BUDDY_CLEARED;
> dma_fence_put(fence);
> fence = wipe_fence;
> }
> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_ring *ring, uint32_t src_data,
> return 0;
> }
>
Some kerneldoc here what the function does would ne nice to have.
> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
> + struct dma_resv *resv,
> + struct dma_fence **fence)
> +{
> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
> + struct amdgpu_res_cursor cursor;
> + struct dma_fence *f = NULL;
It might be cleaner to just use the stub fence here (see
dma_fence_get_stub()).
This would avoid to local variable init in the caller and the if which
checks if the function returned a fence or not.
> + u64 addr;
> + int r;
> +
> + if (!adev->mman.buffer_funcs_enabled)
> + return -EINVAL;
> +
> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
> +
> + mutex_lock(&adev->mman.gtt_window_lock);
> + while (cursor.remaining) {
> + struct dma_fence *next = NULL;
> + u64 size;
> +
> + if (amdgpu_res_cleared(&cursor)) {
> + amdgpu_res_next(&cursor, cursor.size);
> + continue;
> + }
> +
> + /* Never clear more than 256MiB at once to avoid timeouts */
> + size = min(cursor.size, 256ULL << 20);
> +
> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
> + 1, ring, false, &size, &addr);
> + if (r)
> + goto err;
> +
> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
> + &next, true, true);
> + if (r)
> + goto err;
> +
> + dma_fence_put(f);
> + f = next;
> +
> + amdgpu_res_next(&cursor, size);
> + }
> +err:
> + mutex_unlock(&adev->mman.gtt_window_lock);
> + if (fence)
Just make fence a mandatory parameter and drop the if and the get/put dance.
> + *fence = dma_fence_get(f);
> + dma_fence_put(f);
> +
> + return r;
> +}
> +
> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
> uint32_t src_data,
> struct dma_resv *resv,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> index 65ec82141a8e..b404d89d52e5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> @@ -38,8 +38,6 @@
> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
>
> -#define AMDGPU_POISON 0xd0bed0be
> -
> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
>
> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
> uint64_t size, bool tmz,
> struct dma_resv *resv,
> struct dma_fence **f);
> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
> + struct dma_resv *resv,
> + struct dma_fence **fence);
> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
> uint32_t src_data,
> struct dma_resv *resv,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> index c0c851409241..e494f5bf136a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> {
> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
> struct amdgpu_device *adev = to_amdgpu_device(mgr);
> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
> u64 vis_usage = 0, max_bytes, min_block_size;
> struct amdgpu_vram_mgr_resource *vres;
> u64 size, remaining_size, lpfn, fpfn;
> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>
> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
> +
Mhm, you should probably *not* store this flags in the vres structure.
As soon as the BO is used the VRAM wouldn't be cleared any more.
Regards,
Christian.
> if (fpfn || lpfn != mgr->mm.size)
> /* Allocate blocks in desired range */
> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
>
> amdgpu_vram_mgr_do_reserve(man);
>
> - drm_buddy_free_list(mm, &vres->blocks, 0);
> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
> mutex_unlock(&mgr->lock);
>
> atomic64_sub(vis_usage, &mgr->vis_usage);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> index 0e04e42cf809..8478522d7366 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> @@ -53,6 +53,11 @@ static inline u64 amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
> }
>
> +static inline bool amdgpu_vram_mgr_is_cleared(struct drm_buddy_block *block)
> +{
> + return drm_buddy_block_is_clear(block);
> +}
> +
> static inline struct amdgpu_vram_mgr_resource *
> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
> {
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-19 10:28 ` Christian König
@ 2024-03-19 11:41 ` Paneer Selvam, Arunpravin
2024-03-19 13:47 ` Christian König
0 siblings, 1 reply; 21+ messages in thread
From: Paneer Selvam, Arunpravin @ 2024-03-19 11:41 UTC (permalink / raw)
To: Christian König, dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, matthew.auld,
mario.limonciello, felix.kuehling
Hi Christian,
On 3/19/2024 3:58 PM, Christian König wrote:
>
>
> Am 18.03.24 um 22:40 schrieb Arunpravin Paneer Selvam:
>> Add clear page support in vram memory region.
>>
>> v1(Christian):
>> - Dont handle clear page as TTM flag since when moving the BO back
>> in from GTT again we don't need that.
>> - Make a specialized version of amdgpu_fill_buffer() which only
>> clears the VRAM areas which are not already cleared
>> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
>> amdgpu_object.c
>>
>> v2:
>> - Modify the function name amdgpu_ttm_* (Alex)
>> - Drop the delayed parameter (Christian)
>> - handle amdgpu_res_cleared(&cursor) just above the size
>> calculation (Christian)
>> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
>> in the free path to properly wait for fences etc.. (Christian)
>>
>> v3(Christian):
>> - Remove buffer clear code in VRAM manager instead change the
>> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
>> the DRM_BUDDY_CLEARED flag.
>> - Remove ! from amdgpu_res_cleared(&cursor) check.
>>
>> Signed-off-by: Arunpravin Paneer Selvam
>> <Arunpravin.PaneerSelvam@amd.com>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
>
> Just a few nit picks below, but in general already looks good to me.
>
>> ---
>> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
>> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
>> 6 files changed, 111 insertions(+), 13 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>> index 8bc79924d171..c92d92b28a57 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>> @@ -39,6 +39,7 @@
>> #include "amdgpu.h"
>> #include "amdgpu_trace.h"
>> #include "amdgpu_amdkfd.h"
>> +#include "amdgpu_vram_mgr.h"
>> /**
>> * DOC: amdgpu_object
>> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>> if (!amdgpu_bo_support_uswc(bo->flags))
>> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
>> - if (adev->ras_enabled)
>> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>> bo->tbo.bdev = &adev->mman.bdev;
>> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
>> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
>> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
>> - struct dma_fence *fence;
>> + struct dma_fence *fence = NULL;
>> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence,
>> true);
>> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
>> if (unlikely(r))
>> goto fail_unreserve;
>> - dma_resv_add_fence(bo->tbo.base.resv, fence,
>> - DMA_RESV_USAGE_KERNEL);
>> - dma_fence_put(fence);
>> + if (fence) {
>> + dma_resv_add_fence(bo->tbo.base.resv, fence,
>> + DMA_RESV_USAGE_KERNEL);
>> + dma_fence_put(fence);
>> + }
>> }
>> if (!bp->resv)
>> amdgpu_bo_unreserve(bo);
>> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct
>> ttm_buffer_object *bo)
>> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
>> return;
>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv,
>> &fence, true);
>> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
>> if (!WARN_ON(r)) {
>> + struct amdgpu_vram_mgr_resource *vres;
>> +
>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>> + vres->flags |= DRM_BUDDY_CLEARED;
>
> Those lines should probably be in the VRAM manager.
This flag is set based on the amdgpu_fill_buffer() function return
value, can we move the amdgpu_fill_buffer() function call and
DRM_BUDDY_CLEARED flag set lines to amdgpu_vram_mgr_del() in VRAM
manager and does it require to wipe the VRAM buffers here as well
without setting the DRM_BUDDY_CLEARED.
>
>> amdgpu_bo_fence(abo, fence, false);
>> dma_fence_put(fence);
>> }
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>> index 381101d2bf05..50fcd86e1033 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct
>> amdgpu_res_cursor *cur, uint64_t size)
>> }
>> }
>> +/**
>> + * amdgpu_res_cleared - check if blocks are cleared
>> + *
>> + * @cur: the cursor to extract the block
>> + *
>> + * Check if the @cur block is cleared
>> + */
>> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
>> +{
>> + struct drm_buddy_block *block;
>> +
>> + switch (cur->mem_type) {
>> + case TTM_PL_VRAM:
>> + block = cur->node;
>> +
>> + if (!amdgpu_vram_mgr_is_cleared(block))
>> + return false;
>> + break;
>> + default:
>> + return false;
>> + }
>> +
>> + return true;
>> +}
>> +
>> #endif
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> index 8722beba494e..bcbffe909b47 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct
>> ttm_buffer_object *bo,
>> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
>> struct dma_fence *wipe_fence = NULL;
>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
>> - false);
>> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
>> + false);
>> if (r) {
>> goto error;
>> } else if (wipe_fence) {
>> + struct amdgpu_vram_mgr_resource *vres;
>> +
>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>> + vres->flags |= DRM_BUDDY_CLEARED;
Does it require to set the DRM_BUDDY_CLEARED flag as we are not freeing
the VRAM buffers?
>> dma_fence_put(fence);
>> fence = wipe_fence;
>> }
>> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct
>> amdgpu_ring *ring, uint32_t src_data,
>> return 0;
>> }
>
> Some kerneldoc here what the function does would ne nice to have.
>
>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>> + struct dma_resv *resv,
>> + struct dma_fence **fence)
>> +{
>> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
>> + struct amdgpu_res_cursor cursor;
>> + struct dma_fence *f = NULL;
>
> It might be cleaner to just use the stub fence here (see
> dma_fence_get_stub()).
>
> This would avoid to local variable init in the caller and the if which
> checks if the function returned a fence or not.
>
>> + u64 addr;
>> + int r;
>> +
>> + if (!adev->mman.buffer_funcs_enabled)
>> + return -EINVAL;
>> +
>> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
>> +
>> + mutex_lock(&adev->mman.gtt_window_lock);
>> + while (cursor.remaining) {
>> + struct dma_fence *next = NULL;
>> + u64 size;
>> +
>> + if (amdgpu_res_cleared(&cursor)) {
>> + amdgpu_res_next(&cursor, cursor.size);
>> + continue;
>> + }
>> +
>> + /* Never clear more than 256MiB at once to avoid timeouts */
>> + size = min(cursor.size, 256ULL << 20);
>> +
>> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
>> + 1, ring, false, &size, &addr);
>> + if (r)
>> + goto err;
>> +
>> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
>> + &next, true, true);
>> + if (r)
>> + goto err;
>> +
>> + dma_fence_put(f);
>> + f = next;
>> +
>> + amdgpu_res_next(&cursor, size);
>> + }
>> +err:
>> + mutex_unlock(&adev->mman.gtt_window_lock);
>> + if (fence)
>
> Just make fence a mandatory parameter and drop the if and the get/put
> dance.
>
>> + *fence = dma_fence_get(f);
>> + dma_fence_put(f);
>> +
>> + return r;
>> +}
>> +
>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>> uint32_t src_data,
>> struct dma_resv *resv,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>> index 65ec82141a8e..b404d89d52e5 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>> @@ -38,8 +38,6 @@
>> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
>> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
>> -#define AMDGPU_POISON 0xd0bed0be
>> -
>> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
>> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
>> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct
>> amdgpu_device *adev,
>> uint64_t size, bool tmz,
>> struct dma_resv *resv,
>> struct dma_fence **f);
>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>> + struct dma_resv *resv,
>> + struct dma_fence **fence);
>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>> uint32_t src_data,
>> struct dma_resv *resv,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> index c0c851409241..e494f5bf136a 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct
>> ttm_resource_manager *man,
>> {
>> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
>> struct amdgpu_device *adev = to_amdgpu_device(mgr);
>> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
>> u64 vis_usage = 0, max_bytes, min_block_size;
>> struct amdgpu_vram_mgr_resource *vres;
>> u64 size, remaining_size, lpfn, fpfn;
>> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct
>> ttm_resource_manager *man,
>> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
>> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
>> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
>> +
>
> Mhm, you should probably *not* store this flags in the vres structure.
>
> As soon as the BO is used the VRAM wouldn't be cleared any more.
>
> Regards,
> Christian.
>
>> if (fpfn || lpfn != mgr->mm.size)
>> /* Allocate blocks in desired range */
>> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
>> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct
>> ttm_resource_manager *man,
>> amdgpu_vram_mgr_do_reserve(man);
>> - drm_buddy_free_list(mm, &vres->blocks, 0);
>> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
>> mutex_unlock(&mgr->lock);
>> atomic64_sub(vis_usage, &mgr->vis_usage);
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>> index 0e04e42cf809..8478522d7366 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>> @@ -53,6 +53,11 @@ static inline u64
>> amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
>> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
>> }
>> +static inline bool amdgpu_vram_mgr_is_cleared(struct
>> drm_buddy_block *block)
>> +{
>> + return drm_buddy_block_is_clear(block);
>> +}
>> +
>> static inline struct amdgpu_vram_mgr_resource *
>> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
>> {
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-19 11:41 ` Paneer Selvam, Arunpravin
@ 2024-03-19 13:47 ` Christian König
2024-03-19 13:57 ` Paneer Selvam, Arunpravin
0 siblings, 1 reply; 21+ messages in thread
From: Christian König @ 2024-03-19 13:47 UTC (permalink / raw)
To: Paneer Selvam, Arunpravin, Christian König, dri-devel, amd-gfx
Cc: alexander.deucher, matthew.auld, mario.limonciello, felix.kuehling
Am 19.03.24 um 12:41 schrieb Paneer Selvam, Arunpravin:
> Hi Christian,
>
> On 3/19/2024 3:58 PM, Christian König wrote:
>>
>>
>> Am 18.03.24 um 22:40 schrieb Arunpravin Paneer Selvam:
>>> Add clear page support in vram memory region.
>>>
>>> v1(Christian):
>>> - Dont handle clear page as TTM flag since when moving the BO back
>>> in from GTT again we don't need that.
>>> - Make a specialized version of amdgpu_fill_buffer() which only
>>> clears the VRAM areas which are not already cleared
>>> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
>>> amdgpu_object.c
>>>
>>> v2:
>>> - Modify the function name amdgpu_ttm_* (Alex)
>>> - Drop the delayed parameter (Christian)
>>> - handle amdgpu_res_cleared(&cursor) just above the size
>>> calculation (Christian)
>>> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the
>>> buffers
>>> in the free path to properly wait for fences etc.. (Christian)
>>>
>>> v3(Christian):
>>> - Remove buffer clear code in VRAM manager instead change the
>>> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
>>> the DRM_BUDDY_CLEARED flag.
>>> - Remove ! from amdgpu_res_cleared(&cursor) check.
>>>
>>> Signed-off-by: Arunpravin Paneer Selvam
>>> <Arunpravin.PaneerSelvam@amd.com>
>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
>>
>> Just a few nit picks below, but in general already looks good to me.
>>
>>> ---
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
>>> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61
>>> ++++++++++++++++++-
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
>>> 6 files changed, 111 insertions(+), 13 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>> index 8bc79924d171..c92d92b28a57 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>> @@ -39,6 +39,7 @@
>>> #include "amdgpu.h"
>>> #include "amdgpu_trace.h"
>>> #include "amdgpu_amdkfd.h"
>>> +#include "amdgpu_vram_mgr.h"
>>> /**
>>> * DOC: amdgpu_object
>>> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>> if (!amdgpu_bo_support_uswc(bo->flags))
>>> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
>>> - if (adev->ras_enabled)
>>> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>> bo->tbo.bdev = &adev->mman.bdev;
>>> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
>>> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
>>> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
>>> - struct dma_fence *fence;
>>> + struct dma_fence *fence = NULL;
>>> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence,
>>> true);
>>> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
>>> if (unlikely(r))
>>> goto fail_unreserve;
>>> - dma_resv_add_fence(bo->tbo.base.resv, fence,
>>> - DMA_RESV_USAGE_KERNEL);
>>> - dma_fence_put(fence);
>>> + if (fence) {
>>> + dma_resv_add_fence(bo->tbo.base.resv, fence,
>>> + DMA_RESV_USAGE_KERNEL);
>>> + dma_fence_put(fence);
>>> + }
>>> }
>>> if (!bp->resv)
>>> amdgpu_bo_unreserve(bo);
>>> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct
>>> ttm_buffer_object *bo)
>>> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
>>> return;
>>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv,
>>> &fence, true);
>>> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
>>> if (!WARN_ON(r)) {
>>> + struct amdgpu_vram_mgr_resource *vres;
>>> +
>>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>>> + vres->flags |= DRM_BUDDY_CLEARED;
>>
>> Those lines should probably be in the VRAM manager.
> This flag is set based on the amdgpu_fill_buffer() function return
> value, can we move the amdgpu_fill_buffer() function call and
> DRM_BUDDY_CLEARED flag set lines to amdgpu_vram_mgr_del() in VRAM
> manager and does it require to wipe the VRAM buffers here as well
> without setting the DRM_BUDDY_CLEARED.
No that won't work. The call to amdgpu_fill_buffer() *must* be here
because that here is called before the resource is actually freed.
Only setting the vres->flags should probably be moved into
amdgpu_vram_mgr.c.
So instead of calling that manually here just make a function for it to
keep the code related to the buddy allocator in one place.
Regards,
Christian.
>>
>>> amdgpu_bo_fence(abo, fence, false);
>>> dma_fence_put(fence);
>>> }
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>> index 381101d2bf05..50fcd86e1033 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct
>>> amdgpu_res_cursor *cur, uint64_t size)
>>> }
>>> }
>>> +/**
>>> + * amdgpu_res_cleared - check if blocks are cleared
>>> + *
>>> + * @cur: the cursor to extract the block
>>> + *
>>> + * Check if the @cur block is cleared
>>> + */
>>> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
>>> +{
>>> + struct drm_buddy_block *block;
>>> +
>>> + switch (cur->mem_type) {
>>> + case TTM_PL_VRAM:
>>> + block = cur->node;
>>> +
>>> + if (!amdgpu_vram_mgr_is_cleared(block))
>>> + return false;
>>> + break;
>>> + default:
>>> + return false;
>>> + }
>>> +
>>> + return true;
>>> +}
>>> +
>>> #endif
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> index 8722beba494e..bcbffe909b47 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct
>>> ttm_buffer_object *bo,
>>> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
>>> struct dma_fence *wipe_fence = NULL;
>>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL,
>>> &wipe_fence,
>>> - false);
>>> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
>>> + false);
>>> if (r) {
>>> goto error;
>>> } else if (wipe_fence) {
>>> + struct amdgpu_vram_mgr_resource *vres;
>>> +
>>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>>> + vres->flags |= DRM_BUDDY_CLEARED;
> Does it require to set the DRM_BUDDY_CLEARED flag as we are not
> freeing the VRAM buffers?
>>> dma_fence_put(fence);
>>> fence = wipe_fence;
>>> }
>>> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct
>>> amdgpu_ring *ring, uint32_t src_data,
>>> return 0;
>>> }
>>
>> Some kerneldoc here what the function does would ne nice to have.
>>
>>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>>> + struct dma_resv *resv,
>>> + struct dma_fence **fence)
>>> +{
>>> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>>> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
>>> + struct amdgpu_res_cursor cursor;
>>> + struct dma_fence *f = NULL;
>>
>> It might be cleaner to just use the stub fence here (see
>> dma_fence_get_stub()).
>>
>> This would avoid to local variable init in the caller and the if
>> which checks if the function returned a fence or not.
>>
>>> + u64 addr;
>>> + int r;
>>> +
>>> + if (!adev->mman.buffer_funcs_enabled)
>>> + return -EINVAL;
>>> +
>>> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo),
>>> &cursor);
>>> +
>>> + mutex_lock(&adev->mman.gtt_window_lock);
>>> + while (cursor.remaining) {
>>> + struct dma_fence *next = NULL;
>>> + u64 size;
>>> +
>>> + if (amdgpu_res_cleared(&cursor)) {
>>> + amdgpu_res_next(&cursor, cursor.size);
>>> + continue;
>>> + }
>>> +
>>> + /* Never clear more than 256MiB at once to avoid timeouts */
>>> + size = min(cursor.size, 256ULL << 20);
>>> +
>>> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
>>> + 1, ring, false, &size, &addr);
>>> + if (r)
>>> + goto err;
>>> +
>>> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
>>> + &next, true, true);
>>> + if (r)
>>> + goto err;
>>> +
>>> + dma_fence_put(f);
>>> + f = next;
>>> +
>>> + amdgpu_res_next(&cursor, size);
>>> + }
>>> +err:
>>> + mutex_unlock(&adev->mman.gtt_window_lock);
>>> + if (fence)
>>
>> Just make fence a mandatory parameter and drop the if and the get/put
>> dance.
>>
>>> + *fence = dma_fence_get(f);
>>> + dma_fence_put(f);
>>> +
>>> + return r;
>>> +}
>>> +
>>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>>> uint32_t src_data,
>>> struct dma_resv *resv,
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>> index 65ec82141a8e..b404d89d52e5 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>> @@ -38,8 +38,6 @@
>>> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
>>> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
>>> -#define AMDGPU_POISON 0xd0bed0be
>>> -
>>> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
>>> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
>>> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct
>>> amdgpu_device *adev,
>>> uint64_t size, bool tmz,
>>> struct dma_resv *resv,
>>> struct dma_fence **f);
>>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>>> + struct dma_resv *resv,
>>> + struct dma_fence **fence);
>>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>>> uint32_t src_data,
>>> struct dma_resv *resv,
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>> index c0c851409241..e494f5bf136a 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct
>>> ttm_resource_manager *man,
>>> {
>>> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
>>> struct amdgpu_device *adev = to_amdgpu_device(mgr);
>>> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
>>> u64 vis_usage = 0, max_bytes, min_block_size;
>>> struct amdgpu_vram_mgr_resource *vres;
>>> u64 size, remaining_size, lpfn, fpfn;
>>> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct
>>> ttm_resource_manager *man,
>>> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
>>> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>>> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
>>> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
>>> +
>>
>> Mhm, you should probably *not* store this flags in the vres structure.
>>
>> As soon as the BO is used the VRAM wouldn't be cleared any more.
>>
>> Regards,
>> Christian.
>>
>>> if (fpfn || lpfn != mgr->mm.size)
>>> /* Allocate blocks in desired range */
>>> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
>>> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct
>>> ttm_resource_manager *man,
>>> amdgpu_vram_mgr_do_reserve(man);
>>> - drm_buddy_free_list(mm, &vres->blocks, 0);
>>> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
>>> mutex_unlock(&mgr->lock);
>>> atomic64_sub(vis_usage, &mgr->vis_usage);
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>> index 0e04e42cf809..8478522d7366 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>> @@ -53,6 +53,11 @@ static inline u64
>>> amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
>>> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
>>> }
>>> +static inline bool amdgpu_vram_mgr_is_cleared(struct
>>> drm_buddy_block *block)
>>> +{
>>> + return drm_buddy_block_is_clear(block);
>>> +}
>>> +
>>> static inline struct amdgpu_vram_mgr_resource *
>>> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
>>> {
>>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-19 13:47 ` Christian König
@ 2024-03-19 13:57 ` Paneer Selvam, Arunpravin
0 siblings, 0 replies; 21+ messages in thread
From: Paneer Selvam, Arunpravin @ 2024-03-19 13:57 UTC (permalink / raw)
To: Christian König, Christian König, dri-devel, amd-gfx
Cc: alexander.deucher, matthew.auld, mario.limonciello, felix.kuehling
Hi Christian,
On 3/19/2024 7:17 PM, Christian König wrote:
> Am 19.03.24 um 12:41 schrieb Paneer Selvam, Arunpravin:
>> Hi Christian,
>>
>> On 3/19/2024 3:58 PM, Christian König wrote:
>>>
>>>
>>> Am 18.03.24 um 22:40 schrieb Arunpravin Paneer Selvam:
>>>> Add clear page support in vram memory region.
>>>>
>>>> v1(Christian):
>>>> - Dont handle clear page as TTM flag since when moving the BO back
>>>> in from GTT again we don't need that.
>>>> - Make a specialized version of amdgpu_fill_buffer() which only
>>>> clears the VRAM areas which are not already cleared
>>>> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
>>>> amdgpu_object.c
>>>>
>>>> v2:
>>>> - Modify the function name amdgpu_ttm_* (Alex)
>>>> - Drop the delayed parameter (Christian)
>>>> - handle amdgpu_res_cleared(&cursor) just above the size
>>>> calculation (Christian)
>>>> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the
>>>> buffers
>>>> in the free path to properly wait for fences etc.. (Christian)
>>>>
>>>> v3(Christian):
>>>> - Remove buffer clear code in VRAM manager instead change the
>>>> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
>>>> the DRM_BUDDY_CLEARED flag.
>>>> - Remove ! from amdgpu_res_cleared(&cursor) check.
>>>>
>>>> Signed-off-by: Arunpravin Paneer Selvam
>>>> <Arunpravin.PaneerSelvam@amd.com>
>>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>>> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
>>>
>>> Just a few nit picks below, but in general already looks good to me.
>>>
>>>> ---
>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
>>>> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61
>>>> ++++++++++++++++++-
>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
>>>> 6 files changed, 111 insertions(+), 13 deletions(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>> index 8bc79924d171..c92d92b28a57 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>> @@ -39,6 +39,7 @@
>>>> #include "amdgpu.h"
>>>> #include "amdgpu_trace.h"
>>>> #include "amdgpu_amdkfd.h"
>>>> +#include "amdgpu_vram_mgr.h"
>>>> /**
>>>> * DOC: amdgpu_object
>>>> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>>> if (!amdgpu_bo_support_uswc(bo->flags))
>>>> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
>>>> - if (adev->ras_enabled)
>>>> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>>> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>>> bo->tbo.bdev = &adev->mman.bdev;
>>>> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
>>>> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>>> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
>>>> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
>>>> - struct dma_fence *fence;
>>>> + struct dma_fence *fence = NULL;
>>>> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence,
>>>> true);
>>>> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
>>>> if (unlikely(r))
>>>> goto fail_unreserve;
>>>> - dma_resv_add_fence(bo->tbo.base.resv, fence,
>>>> - DMA_RESV_USAGE_KERNEL);
>>>> - dma_fence_put(fence);
>>>> + if (fence) {
>>>> + dma_resv_add_fence(bo->tbo.base.resv, fence,
>>>> + DMA_RESV_USAGE_KERNEL);
>>>> + dma_fence_put(fence);
>>>> + }
>>>> }
>>>> if (!bp->resv)
>>>> amdgpu_bo_unreserve(bo);
>>>> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct
>>>> ttm_buffer_object *bo)
>>>> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
>>>> return;
>>>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv,
>>>> &fence, true);
>>>> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
>>>> if (!WARN_ON(r)) {
>>>> + struct amdgpu_vram_mgr_resource *vres;
>>>> +
>>>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>>>> + vres->flags |= DRM_BUDDY_CLEARED;
>>>
>>> Those lines should probably be in the VRAM manager.
>> This flag is set based on the amdgpu_fill_buffer() function return
>> value, can we move the amdgpu_fill_buffer() function call and
>> DRM_BUDDY_CLEARED flag set lines to amdgpu_vram_mgr_del() in VRAM
>> manager and does it require to wipe the VRAM buffers here as well
>> without setting the DRM_BUDDY_CLEARED.
>
> No that won't work. The call to amdgpu_fill_buffer() *must* be here
> because that here is called before the resource is actually freed.
>
> Only setting the vres->flags should probably be moved into
> amdgpu_vram_mgr.c.
>
> So instead of calling that manually here just make a function for it
> to keep the code related to the buddy allocator in one place.
I got it, Thanks.
Regards,
Arun.
>
> Regards,
> Christian.
>
>>>
>>>> amdgpu_bo_fence(abo, fence, false);
>>>> dma_fence_put(fence);
>>>> }
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>> index 381101d2bf05..50fcd86e1033 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct
>>>> amdgpu_res_cursor *cur, uint64_t size)
>>>> }
>>>> }
>>>> +/**
>>>> + * amdgpu_res_cleared - check if blocks are cleared
>>>> + *
>>>> + * @cur: the cursor to extract the block
>>>> + *
>>>> + * Check if the @cur block is cleared
>>>> + */
>>>> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
>>>> +{
>>>> + struct drm_buddy_block *block;
>>>> +
>>>> + switch (cur->mem_type) {
>>>> + case TTM_PL_VRAM:
>>>> + block = cur->node;
>>>> +
>>>> + if (!amdgpu_vram_mgr_is_cleared(block))
>>>> + return false;
>>>> + break;
>>>> + default:
>>>> + return false;
>>>> + }
>>>> +
>>>> + return true;
>>>> +}
>>>> +
>>>> #endif
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>> index 8722beba494e..bcbffe909b47 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct
>>>> ttm_buffer_object *bo,
>>>> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
>>>> struct dma_fence *wipe_fence = NULL;
>>>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL,
>>>> &wipe_fence,
>>>> - false);
>>>> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
>>>> + false);
>>>> if (r) {
>>>> goto error;
>>>> } else if (wipe_fence) {
>>>> + struct amdgpu_vram_mgr_resource *vres;
>>>> +
>>>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>>>> + vres->flags |= DRM_BUDDY_CLEARED;
>> Does it require to set the DRM_BUDDY_CLEARED flag as we are not
>> freeing the VRAM buffers?
>>>> dma_fence_put(fence);
>>>> fence = wipe_fence;
>>>> }
>>>> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct
>>>> amdgpu_ring *ring, uint32_t src_data,
>>>> return 0;
>>>> }
>>>
>>> Some kerneldoc here what the function does would ne nice to have.
>>>
>>>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>>>> + struct dma_resv *resv,
>>>> + struct dma_fence **fence)
>>>> +{
>>>> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>>>> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
>>>> + struct amdgpu_res_cursor cursor;
>>>> + struct dma_fence *f = NULL;
>>>
>>> It might be cleaner to just use the stub fence here (see
>>> dma_fence_get_stub()).
>>>
>>> This would avoid to local variable init in the caller and the if
>>> which checks if the function returned a fence or not.
>>>
>>>> + u64 addr;
>>>> + int r;
>>>> +
>>>> + if (!adev->mman.buffer_funcs_enabled)
>>>> + return -EINVAL;
>>>> +
>>>> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo),
>>>> &cursor);
>>>> +
>>>> + mutex_lock(&adev->mman.gtt_window_lock);
>>>> + while (cursor.remaining) {
>>>> + struct dma_fence *next = NULL;
>>>> + u64 size;
>>>> +
>>>> + if (amdgpu_res_cleared(&cursor)) {
>>>> + amdgpu_res_next(&cursor, cursor.size);
>>>> + continue;
>>>> + }
>>>> +
>>>> + /* Never clear more than 256MiB at once to avoid timeouts */
>>>> + size = min(cursor.size, 256ULL << 20);
>>>> +
>>>> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource,
>>>> &cursor,
>>>> + 1, ring, false, &size, &addr);
>>>> + if (r)
>>>> + goto err;
>>>> +
>>>> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
>>>> + &next, true, true);
>>>> + if (r)
>>>> + goto err;
>>>> +
>>>> + dma_fence_put(f);
>>>> + f = next;
>>>> +
>>>> + amdgpu_res_next(&cursor, size);
>>>> + }
>>>> +err:
>>>> + mutex_unlock(&adev->mman.gtt_window_lock);
>>>> + if (fence)
>>>
>>> Just make fence a mandatory parameter and drop the if and the
>>> get/put dance.
>>>
>>>> + *fence = dma_fence_get(f);
>>>> + dma_fence_put(f);
>>>> +
>>>> + return r;
>>>> +}
>>>> +
>>>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>>>> uint32_t src_data,
>>>> struct dma_resv *resv,
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>> index 65ec82141a8e..b404d89d52e5 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>> @@ -38,8 +38,6 @@
>>>> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
>>>> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
>>>> -#define AMDGPU_POISON 0xd0bed0be
>>>> -
>>>> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
>>>> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
>>>> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct
>>>> amdgpu_device *adev,
>>>> uint64_t size, bool tmz,
>>>> struct dma_resv *resv,
>>>> struct dma_fence **f);
>>>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>>>> + struct dma_resv *resv,
>>>> + struct dma_fence **fence);
>>>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>>>> uint32_t src_data,
>>>> struct dma_resv *resv,
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>> index c0c851409241..e494f5bf136a 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct
>>>> ttm_resource_manager *man,
>>>> {
>>>> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
>>>> struct amdgpu_device *adev = to_amdgpu_device(mgr);
>>>> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
>>>> u64 vis_usage = 0, max_bytes, min_block_size;
>>>> struct amdgpu_vram_mgr_resource *vres;
>>>> u64 size, remaining_size, lpfn, fpfn;
>>>> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct
>>>> ttm_resource_manager *man,
>>>> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
>>>> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>>>> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
>>>> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
>>>> +
>>>
>>> Mhm, you should probably *not* store this flags in the vres structure.
>>>
>>> As soon as the BO is used the VRAM wouldn't be cleared any more.
>>>
>>> Regards,
>>> Christian.
>>>
>>>> if (fpfn || lpfn != mgr->mm.size)
>>>> /* Allocate blocks in desired range */
>>>> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
>>>> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct
>>>> ttm_resource_manager *man,
>>>> amdgpu_vram_mgr_do_reserve(man);
>>>> - drm_buddy_free_list(mm, &vres->blocks, 0);
>>>> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
>>>> mutex_unlock(&mgr->lock);
>>>> atomic64_sub(vis_usage, &mgr->vis_usage);
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>> index 0e04e42cf809..8478522d7366 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>> @@ -53,6 +53,11 @@ static inline u64
>>>> amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
>>>> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
>>>> }
>>>> +static inline bool amdgpu_vram_mgr_is_cleared(struct
>>>> drm_buddy_block *block)
>>>> +{
>>>> + return drm_buddy_block_is_clear(block);
>>>> +}
>>>> +
>>>> static inline struct amdgpu_vram_mgr_resource *
>>>> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
>>>> {
>>>
>>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature
2024-03-18 21:40 [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature Arunpravin Paneer Selvam
2024-03-18 21:40 ` [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality Arunpravin Paneer Selvam
2024-03-18 21:40 ` [PATCH v9 3/3] drm/tests: Add a test case for drm buddy clear allocation Arunpravin Paneer Selvam
@ 2024-03-25 13:37 ` Paneer Selvam, Arunpravin
2024-03-26 18:09 ` Matthew Auld
3 siblings, 0 replies; 21+ messages in thread
From: Paneer Selvam, Arunpravin @ 2024-03-25 13:37 UTC (permalink / raw)
To: dri-devel, amd-gfx, Matthew Auld
Cc: christian.koenig, alexander.deucher, mario.limonciello, felix.kuehling
Hi Matthew,
ping?
Thanks,
Arun.
On 3/19/2024 3:10 AM, Arunpravin Paneer Selvam wrote:
> - Add tracking clear page feature.
>
> - Driver should enable the DRM_BUDDY_CLEARED flag if it
> successfully clears the blocks in the free path. On the otherhand,
> DRM buddy marks each block as cleared.
>
> - Track the available cleared pages size
>
> - If driver requests cleared memory we prefer cleared memory
> but fallback to uncleared if we can't find the cleared blocks.
> when driver requests uncleared memory we try to use uncleared but
> fallback to cleared memory if necessary.
>
> - When a block gets freed we clear it and mark the freed block as cleared,
> when there are buddies which are cleared as well we can merge them.
> Otherwise, we prefer to keep the blocks as separated.
>
> - Add a function to support defragmentation.
>
> v1:
> - Depends on the flag check DRM_BUDDY_CLEARED, enable the block as
> cleared. Else, reset the clear flag for each block in the list(Christian)
> - For merging the 2 cleared blocks compare as below,
> drm_buddy_is_clear(block) != drm_buddy_is_clear(buddy)(Christian)
> - Defragment the memory beginning from min_order
> till the required memory space is available.
>
> v2: (Matthew)
> - Add a wrapper drm_buddy_free_list_internal for the freeing of blocks
> operation within drm buddy.
> - Write a macro block_incompatible() to allocate the required blocks.
> - Update the xe driver for the drm_buddy_free_list change in arguments.
> - add a warning if the two blocks are incompatible on
> defragmentation
> - call full defragmentation in the fini() function
> - place a condition to test if min_order is equal to 0
> - replace the list with safe_reverse() variant as we might
> remove the block from the list.
>
> v3:
> - fix Gitlab user reported lockup issue.
> - Keep DRM_BUDDY_HEADER_CLEAR define sorted(Matthew)
> - modify to pass the root order instead max_order in fini()
> function(Matthew)
> - change bool 1 to true(Matthew)
> - add check if min_block_size is power of 2(Matthew)
> - modify the min_block_size datatype to u64(Matthew)
>
> v4:
> - rename the function drm_buddy_defrag with __force_merge.
> - Include __force_merge directly in drm buddy file and remove
> the defrag use in amdgpu driver.
> - Remove list_empty() check(Matthew)
> - Remove unnecessary space, headers and placement of new variables(Matthew)
> - Add a unit test case(Matthew)
>
> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
> Signed-off-by: Matthew Auld <matthew.auld@intel.com>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Matthew Auld <matthew.auld@intel.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
> drivers/gpu/drm/drm_buddy.c | 427 ++++++++++++++----
> drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 6 +-
> drivers/gpu/drm/tests/drm_buddy_test.c | 18 +-
> drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 4 +-
> include/drm/drm_buddy.h | 16 +-
> 6 files changed, 360 insertions(+), 117 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> index 8db880244324..c0c851409241 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> @@ -571,7 +571,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> return 0;
>
> error_free_blocks:
> - drm_buddy_free_list(mm, &vres->blocks);
> + drm_buddy_free_list(mm, &vres->blocks, 0);
> mutex_unlock(&mgr->lock);
> error_fini:
> ttm_resource_fini(man, &vres->base);
> @@ -604,7 +604,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
>
> amdgpu_vram_mgr_do_reserve(man);
>
> - drm_buddy_free_list(mm, &vres->blocks);
> + drm_buddy_free_list(mm, &vres->blocks, 0);
> mutex_unlock(&mgr->lock);
>
> atomic64_sub(vis_usage, &mgr->vis_usage);
> @@ -912,7 +912,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev)
> kfree(rsv);
>
> list_for_each_entry_safe(rsv, temp, &mgr->reserved_pages, blocks) {
> - drm_buddy_free_list(&mgr->mm, &rsv->allocated);
> + drm_buddy_free_list(&mgr->mm, &rsv->allocated, 0);
> kfree(rsv);
> }
> if (!adev->gmc.is_app_apu)
> diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c
> index c4222b886db7..625a30a6b855 100644
> --- a/drivers/gpu/drm/drm_buddy.c
> +++ b/drivers/gpu/drm/drm_buddy.c
> @@ -38,8 +38,8 @@ static void drm_block_free(struct drm_buddy *mm,
> kmem_cache_free(slab_blocks, block);
> }
>
> -static void list_insert_sorted(struct drm_buddy *mm,
> - struct drm_buddy_block *block)
> +static void list_insert(struct drm_buddy *mm,
> + struct drm_buddy_block *block)
> {
> struct drm_buddy_block *node;
> struct list_head *head;
> @@ -57,6 +57,16 @@ static void list_insert_sorted(struct drm_buddy *mm,
> __list_add(&block->link, node->link.prev, &node->link);
> }
>
> +static void clear_reset(struct drm_buddy_block *block)
> +{
> + block->header &= ~DRM_BUDDY_HEADER_CLEAR;
> +}
> +
> +static void mark_cleared(struct drm_buddy_block *block)
> +{
> + block->header |= DRM_BUDDY_HEADER_CLEAR;
> +}
> +
> static void mark_allocated(struct drm_buddy_block *block)
> {
> block->header &= ~DRM_BUDDY_HEADER_STATE;
> @@ -71,7 +81,7 @@ static void mark_free(struct drm_buddy *mm,
> block->header &= ~DRM_BUDDY_HEADER_STATE;
> block->header |= DRM_BUDDY_FREE;
>
> - list_insert_sorted(mm, block);
> + list_insert(mm, block);
> }
>
> static void mark_split(struct drm_buddy_block *block)
> @@ -82,6 +92,114 @@ static void mark_split(struct drm_buddy_block *block)
> list_del(&block->link);
> }
>
> +static struct drm_buddy_block *
> +__get_buddy(struct drm_buddy_block *block)
> +{
> + struct drm_buddy_block *parent;
> +
> + parent = block->parent;
> + if (!parent)
> + return NULL;
> +
> + if (parent->left == block)
> + return parent->right;
> +
> + return parent->left;
> +}
> +
> +static unsigned int __drm_buddy_free(struct drm_buddy *mm,
> + struct drm_buddy_block *block,
> + bool force_merge)
> +{
> + struct drm_buddy_block *parent;
> + unsigned int order;
> +
> + while ((parent = block->parent)) {
> + struct drm_buddy_block *buddy;
> +
> + buddy = __get_buddy(block);
> +
> + if (!drm_buddy_block_is_free(buddy))
> + break;
> +
> + if (!force_merge) {
> + /**
> + * Check the block and its buddy clear state and exit
> + * the loop if they both have the dissimilar state.
> + */
> + if (drm_buddy_block_is_clear(block) !=
> + drm_buddy_block_is_clear(buddy))
> + break;
> +
> + if (drm_buddy_block_is_clear(block))
> + mark_cleared(parent);
> + }
> +
> + list_del(&buddy->link);
> + if (force_merge && drm_buddy_block_is_clear(buddy))
> + mm->clear_avail -= drm_buddy_block_size(mm, buddy);
> +
> + drm_block_free(mm, block);
> + drm_block_free(mm, buddy);
> +
> + block = parent;
> + }
> +
> + order = drm_buddy_block_order(block);
> + mark_free(mm, block);
> +
> + return order;
> +}
> +
> +static int __force_merge(struct drm_buddy *mm,
> + unsigned int min_order)
> +{
> + unsigned int order;
> + int i;
> +
> + if (!min_order)
> + return -ENOMEM;
> +
> + if (min_order > mm->max_order)
> + return -EINVAL;
> +
> + for (i = min_order - 1; i >= 0; i--) {
> + struct drm_buddy_block *block, *prev;
> +
> + list_for_each_entry_safe_reverse(block, prev, &mm->free_list[i], link) {
> + struct drm_buddy_block *buddy;
> +
> + if (!block->parent)
> + continue;
> +
> + buddy = __get_buddy(block);
> + if (!drm_buddy_block_is_free(buddy))
> + continue;
> +
> + WARN_ON(drm_buddy_block_is_clear(block) ==
> + drm_buddy_block_is_clear(buddy));
> +
> + /**
> + * If the prev block is same as buddy, don't access the
> + * block in the next iteration as we would free the
> + * buddy block as part of the free function.
> + */
> + if (prev == buddy)
> + prev = list_prev_entry(prev, link);
> +
> + list_del(&block->link);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail -= drm_buddy_block_size(mm, block);
> +
> + order = __drm_buddy_free(mm, block, true);
> + if (order >= min_order)
> + return 0;
> + }
> + }
> +
> + return -ENOMEM;
> +}
> +
> /**
> * drm_buddy_init - init memory manager
> *
> @@ -137,7 +255,7 @@ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size)
> offset = 0;
> i = 0;
>
> - /*
> + /**
> * Split into power-of-two blocks, in case we are given a size that is
> * not itself a power-of-two.
> */
> @@ -186,11 +304,21 @@ EXPORT_SYMBOL(drm_buddy_init);
> */
> void drm_buddy_fini(struct drm_buddy *mm)
> {
> + u64 root_size, size;
> + unsigned int order;
> int i;
>
> + size = mm->size;
> +
> for (i = 0; i < mm->n_roots; ++i) {
> + order = ilog2(size) - ilog2(mm->chunk_size);
> + __force_merge(mm, order);
> +
> WARN_ON(!drm_buddy_block_is_free(mm->roots[i]));
> drm_block_free(mm, mm->roots[i]);
> +
> + root_size = mm->chunk_size << order;
> + size -= root_size;
> }
>
> WARN_ON(mm->avail != mm->size);
> @@ -223,26 +351,17 @@ static int split_block(struct drm_buddy *mm,
> mark_free(mm, block->left);
> mark_free(mm, block->right);
>
> + if (drm_buddy_block_is_clear(block)) {
> + mark_cleared(block->left);
> + mark_cleared(block->right);
> + clear_reset(block);
> + }
> +
> mark_split(block);
>
> return 0;
> }
>
> -static struct drm_buddy_block *
> -__get_buddy(struct drm_buddy_block *block)
> -{
> - struct drm_buddy_block *parent;
> -
> - parent = block->parent;
> - if (!parent)
> - return NULL;
> -
> - if (parent->left == block)
> - return parent->right;
> -
> - return parent->left;
> -}
> -
> /**
> * drm_get_buddy - get buddy address
> *
> @@ -260,30 +379,6 @@ drm_get_buddy(struct drm_buddy_block *block)
> }
> EXPORT_SYMBOL(drm_get_buddy);
>
> -static void __drm_buddy_free(struct drm_buddy *mm,
> - struct drm_buddy_block *block)
> -{
> - struct drm_buddy_block *parent;
> -
> - while ((parent = block->parent)) {
> - struct drm_buddy_block *buddy;
> -
> - buddy = __get_buddy(block);
> -
> - if (!drm_buddy_block_is_free(buddy))
> - break;
> -
> - list_del(&buddy->link);
> -
> - drm_block_free(mm, block);
> - drm_block_free(mm, buddy);
> -
> - block = parent;
> - }
> -
> - mark_free(mm, block);
> -}
> -
> /**
> * drm_buddy_free_block - free a block
> *
> @@ -295,26 +390,59 @@ void drm_buddy_free_block(struct drm_buddy *mm,
> {
> BUG_ON(!drm_buddy_block_is_allocated(block));
> mm->avail += drm_buddy_block_size(mm, block);
> - __drm_buddy_free(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail += drm_buddy_block_size(mm, block);
> +
> + __drm_buddy_free(mm, block, false);
> }
> EXPORT_SYMBOL(drm_buddy_free_block);
>
> -/**
> - * drm_buddy_free_list - free blocks
> - *
> - * @mm: DRM buddy manager
> - * @objects: input list head to free blocks
> - */
> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head *objects)
> +static void __drm_buddy_free_list(struct drm_buddy *mm,
> + struct list_head *objects,
> + bool mark_clear,
> + bool mark_dirty)
> {
> struct drm_buddy_block *block, *on;
>
> + WARN_ON(mark_dirty && mark_clear);
> +
> list_for_each_entry_safe(block, on, objects, link) {
> + if (mark_clear)
> + mark_cleared(block);
> + else if (mark_dirty)
> + clear_reset(block);
> drm_buddy_free_block(mm, block);
> cond_resched();
> }
> INIT_LIST_HEAD(objects);
> }
> +
> +static void drm_buddy_free_list_internal(struct drm_buddy *mm,
> + struct list_head *objects)
> +{
> + /**
> + * Don't touch the clear/dirty bit, since allocation is still internal
> + * at this point. For example we might have just failed part of the
> + * allocation.
> + */
> + __drm_buddy_free_list(mm, objects, false, false);
> +}
> +
> +/**
> + * drm_buddy_free_list - free blocks
> + *
> + * @mm: DRM buddy manager
> + * @objects: input list head to free blocks
> + * @flags: optional flags like DRM_BUDDY_CLEARED
> + */
> +void drm_buddy_free_list(struct drm_buddy *mm,
> + struct list_head *objects,
> + unsigned int flags)
> +{
> + bool mark_clear = flags & DRM_BUDDY_CLEARED;
> +
> + __drm_buddy_free_list(mm, objects, mark_clear, !mark_clear);
> +}
> EXPORT_SYMBOL(drm_buddy_free_list);
>
> static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
> @@ -327,10 +455,19 @@ static inline bool contains(u64 s1, u64 e1, u64 s2, u64 e2)
> return s1 <= s2 && e1 >= e2;
> }
>
> +static bool block_incompatible(struct drm_buddy_block *block, unsigned int flags)
> +{
> + bool needs_clear = flags & DRM_BUDDY_CLEAR_ALLOCATION;
> +
> + return needs_clear != drm_buddy_block_is_clear(block);
> +}
> +
> static struct drm_buddy_block *
> -alloc_range_bias(struct drm_buddy *mm,
> - u64 start, u64 end,
> - unsigned int order)
> +__alloc_range_bias(struct drm_buddy *mm,
> + u64 start, u64 end,
> + unsigned int order,
> + unsigned long flags,
> + bool fallback)
> {
> struct drm_buddy_block *block;
> struct drm_buddy_block *buddy;
> @@ -369,7 +506,10 @@ alloc_range_bias(struct drm_buddy *mm,
>
> if (contains(start, end, block_start, block_end) &&
> order == drm_buddy_block_order(block)) {
> - /*
> + if (!fallback && block_incompatible(block, flags))
> + continue;
> +
> + /**
> * Find the free block within the range.
> */
> if (drm_buddy_block_is_free(block))
> @@ -391,7 +531,7 @@ alloc_range_bias(struct drm_buddy *mm,
> return ERR_PTR(-ENOSPC);
>
> err_undo:
> - /*
> + /**
> * We really don't want to leave around a bunch of split blocks, since
> * bigger is better, so make sure we merge everything back before we
> * free the allocated blocks.
> @@ -400,30 +540,57 @@ alloc_range_bias(struct drm_buddy *mm,
> if (buddy &&
> (drm_buddy_block_is_free(block) &&
> drm_buddy_block_is_free(buddy)))
> - __drm_buddy_free(mm, block);
> + __drm_buddy_free(mm, block, false);
> return ERR_PTR(err);
> }
>
> static struct drm_buddy_block *
> -get_maxblock(struct drm_buddy *mm, unsigned int order)
> +__drm_buddy_alloc_range_bias(struct drm_buddy *mm,
> + u64 start, u64 end,
> + unsigned int order,
> + unsigned long flags)
> +{
> + struct drm_buddy_block *block;
> + bool fallback = false;
> +
> + block = __alloc_range_bias(mm, start, end, order,
> + flags, fallback);
> + if (IS_ERR(block) && mm->clear_avail)
> + return __alloc_range_bias(mm, start, end, order,
> + flags, !fallback);
> +
> + return block;
> +}
> +
> +static struct drm_buddy_block *
> +get_maxblock(struct drm_buddy *mm, unsigned int order,
> + unsigned long flags)
> {
> - struct drm_buddy_block *max_block = NULL, *node;
> + struct drm_buddy_block *max_block = NULL, *block = NULL;
> unsigned int i;
>
> for (i = order; i <= mm->max_order; ++i) {
> - if (!list_empty(&mm->free_list[i])) {
> - node = list_last_entry(&mm->free_list[i],
> - struct drm_buddy_block,
> - link);
> - if (!max_block) {
> - max_block = node;
> + struct drm_buddy_block *tmp_block;
> +
> + list_for_each_entry_reverse(tmp_block, &mm->free_list[i], link) {
> + if (block_incompatible(tmp_block, flags))
> continue;
> - }
>
> - if (drm_buddy_block_offset(node) >
> - drm_buddy_block_offset(max_block)) {
> - max_block = node;
> - }
> + block = tmp_block;
> + break;
> + }
> +
> + if (!block)
> + continue;
> +
> + if (!max_block) {
> + max_block = block;
> + continue;
> + }
> +
> + if (drm_buddy_block_offset(block) >
> + drm_buddy_block_offset(max_block)) {
> + max_block = block;
> }
> }
>
> @@ -440,11 +607,29 @@ alloc_from_freelist(struct drm_buddy *mm,
> int err;
>
> if (flags & DRM_BUDDY_TOPDOWN_ALLOCATION) {
> - block = get_maxblock(mm, order);
> + block = get_maxblock(mm, order, flags);
> if (block)
> /* Store the obtained block order */
> tmp = drm_buddy_block_order(block);
> } else {
> + for (tmp = order; tmp <= mm->max_order; ++tmp) {
> + struct drm_buddy_block *tmp_block;
> +
> + list_for_each_entry_reverse(tmp_block, &mm->free_list[tmp], link) {
> + if (block_incompatible(tmp_block, flags))
> + continue;
> +
> + block = tmp_block;
> + break;
> + }
> +
> + if (block)
> + break;
> + }
> + }
> +
> + if (!block) {
> + /* Fallback method */
> for (tmp = order; tmp <= mm->max_order; ++tmp) {
> if (!list_empty(&mm->free_list[tmp])) {
> block = list_last_entry(&mm->free_list[tmp],
> @@ -454,10 +639,10 @@ alloc_from_freelist(struct drm_buddy *mm,
> break;
> }
> }
> - }
>
> - if (!block)
> - return ERR_PTR(-ENOSPC);
> + if (!block)
> + return ERR_PTR(-ENOSPC);
> + }
>
> BUG_ON(!drm_buddy_block_is_free(block));
>
> @@ -473,7 +658,7 @@ alloc_from_freelist(struct drm_buddy *mm,
>
> err_undo:
> if (tmp != order)
> - __drm_buddy_free(mm, block);
> + __drm_buddy_free(mm, block, false);
> return ERR_PTR(err);
> }
>
> @@ -524,6 +709,8 @@ static int __alloc_range(struct drm_buddy *mm,
> mark_allocated(block);
> total_allocated += drm_buddy_block_size(mm, block);
> mm->avail -= drm_buddy_block_size(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail -= drm_buddy_block_size(mm, block);
> list_add_tail(&block->link, &allocated);
> continue;
> }
> @@ -548,7 +735,7 @@ static int __alloc_range(struct drm_buddy *mm,
> return 0;
>
> err_undo:
> - /*
> + /**
> * We really don't want to leave around a bunch of split blocks, since
> * bigger is better, so make sure we merge everything back before we
> * free the allocated blocks.
> @@ -557,14 +744,14 @@ static int __alloc_range(struct drm_buddy *mm,
> if (buddy &&
> (drm_buddy_block_is_free(block) &&
> drm_buddy_block_is_free(buddy)))
> - __drm_buddy_free(mm, block);
> + __drm_buddy_free(mm, block, false);
>
> err_free:
> if (err == -ENOSPC && total_allocated_on_err) {
> list_splice_tail(&allocated, blocks);
> *total_allocated_on_err = total_allocated;
> } else {
> - drm_buddy_free_list(mm, &allocated);
> + drm_buddy_free_list_internal(mm, &allocated);
> }
>
> return err;
> @@ -630,11 +817,11 @@ static int __alloc_contig_try_harder(struct drm_buddy *mm,
> list_splice(&blocks_lhs, blocks);
> return 0;
> } else if (err != -ENOSPC) {
> - drm_buddy_free_list(mm, blocks);
> + drm_buddy_free_list_internal(mm, blocks);
> return err;
> }
> /* Free blocks for the next iteration */
> - drm_buddy_free_list(mm, blocks);
> + drm_buddy_free_list_internal(mm, blocks);
> }
>
> return -ENOSPC;
> @@ -690,6 +877,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
> list_del(&block->link);
> mark_free(mm, block);
> mm->avail += drm_buddy_block_size(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail += drm_buddy_block_size(mm, block);
>
> /* Prevent recursively freeing this node */
> parent = block->parent;
> @@ -701,6 +890,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
> if (err) {
> mark_allocated(block);
> mm->avail -= drm_buddy_block_size(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail -= drm_buddy_block_size(mm, block);
> list_add(&block->link, blocks);
> }
>
> @@ -709,13 +900,28 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
> }
> EXPORT_SYMBOL(drm_buddy_block_trim);
>
> +static struct drm_buddy_block *
> +__drm_buddy_alloc_blocks(struct drm_buddy *mm,
> + u64 start, u64 end,
> + unsigned int order,
> + unsigned long flags)
> +{
> + if (flags & DRM_BUDDY_RANGE_ALLOCATION)
> + /* Allocate traversing within the range */
> + return __drm_buddy_alloc_range_bias(mm, start, end,
> + order, flags);
> + else
> + /* Allocate from freelist */
> + return alloc_from_freelist(mm, order, flags);
> +}
> +
> /**
> * drm_buddy_alloc_blocks - allocate power-of-two blocks
> *
> * @mm: DRM buddy manager to allocate from
> * @start: start of the allowed range for this block
> * @end: end of the allowed range for this block
> - * @size: size of the allocation
> + * @size: size of the allocation in bytes
> * @min_block_size: alignment of the allocation
> * @blocks: output list head to add allocated blocks
> * @flags: DRM_BUDDY_*_ALLOCATION flags
> @@ -761,8 +967,18 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
> return -EINVAL;
>
> /* Actual range allocation */
> - if (start + size == end)
> - return __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
> + if (start + size == end) {
> + err = __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
> + if (err) {
> + order = ilog2(size) - ilog2(mm->chunk_size);
> + if (mm->clear_avail && !__force_merge(mm, order))
> + return __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
> +
> + return err;
> + }
> +
> + return err;
> + }
>
> original_size = size;
> original_min_size = min_block_size;
> @@ -786,23 +1002,34 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
> BUG_ON(order < min_order);
>
> do {
> - if (flags & DRM_BUDDY_RANGE_ALLOCATION)
> - /* Allocate traversing within the range */
> - block = alloc_range_bias(mm, start, end, order);
> - else
> - /* Allocate from freelist */
> - block = alloc_from_freelist(mm, order, flags);
> -
> + block = __drm_buddy_alloc_blocks(mm, start,
> + end,
> + order,
> + flags);
> if (!IS_ERR(block))
> break;
>
> if (order-- == min_order) {
> + /**
> + * Try allocation through force merge method
> + */
> + if (mm->clear_avail && !__force_merge(mm, min_order)) {
> + block = __drm_buddy_alloc_blocks(mm, start,
> + end,
> + min_order,
> + flags);
> + if (!IS_ERR(block)) {
> + order = min_order;
> + break;
> + }
> + }
> +
> + /**
> + * Try contiguous block allocation through
> + * try harder method.
> + */
> if (flags & DRM_BUDDY_CONTIGUOUS_ALLOCATION &&
> !(flags & DRM_BUDDY_RANGE_ALLOCATION))
> - /*
> - * Try contiguous block allocation through
> - * try harder method
> - */
> return __alloc_contig_try_harder(mm,
> original_size,
> original_min_size,
> @@ -814,6 +1041,8 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>
> mark_allocated(block);
> mm->avail -= drm_buddy_block_size(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail -= drm_buddy_block_size(mm, block);
> kmemleak_update_trace(block);
> list_add_tail(&block->link, &allocated);
>
> @@ -823,7 +1052,9 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
> break;
> } while (1);
>
> - /* Trim the allocated block to the required size */
> + /**
> + * Trim the allocated block to the required size
> + */
> if (original_size != size) {
> struct list_head *trim_list;
> LIST_HEAD(temp);
> @@ -852,7 +1083,7 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
> return 0;
>
> err_free:
> - drm_buddy_free_list(mm, &allocated);
> + drm_buddy_free_list_internal(mm, &allocated);
> return err;
> }
> EXPORT_SYMBOL(drm_buddy_alloc_blocks);
> @@ -885,8 +1116,8 @@ void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p)
> {
> int order;
>
> - drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free: %lluMiB\n",
> - mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20);
> + drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free: %lluMiB, clear_free: %lluMiB\n",
> + mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20, mm->clear_avail >> 20);
>
> for (order = mm->max_order; order >= 0; order--) {
> struct drm_buddy_block *block;
> diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
> index 0d735d5c2b35..942345548bc3 100644
> --- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
> +++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
> @@ -126,7 +126,7 @@ static int i915_ttm_buddy_man_alloc(struct ttm_resource_manager *man,
> return 0;
>
> err_free_blocks:
> - drm_buddy_free_list(mm, &bman_res->blocks);
> + drm_buddy_free_list(mm, &bman_res->blocks, 0);
> mutex_unlock(&bman->lock);
> err_free_res:
> ttm_resource_fini(man, &bman_res->base);
> @@ -141,7 +141,7 @@ static void i915_ttm_buddy_man_free(struct ttm_resource_manager *man,
> struct i915_ttm_buddy_manager *bman = to_buddy_manager(man);
>
> mutex_lock(&bman->lock);
> - drm_buddy_free_list(&bman->mm, &bman_res->blocks);
> + drm_buddy_free_list(&bman->mm, &bman_res->blocks, 0);
> bman->visible_avail += bman_res->used_visible_size;
> mutex_unlock(&bman->lock);
>
> @@ -345,7 +345,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type)
> ttm_set_driver_manager(bdev, type, NULL);
>
> mutex_lock(&bman->lock);
> - drm_buddy_free_list(mm, &bman->reserved);
> + drm_buddy_free_list(mm, &bman->reserved, 0);
> drm_buddy_fini(mm);
> bman->visible_avail += bman->visible_reserved;
> WARN_ON_ONCE(bman->visible_avail != bman->visible_size);
> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
> index 2f32fb2f12e7..454ad9952f56 100644
> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> @@ -64,7 +64,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> "buddy_alloc didn't error size=%u\n", 3 * ps);
>
> - drm_buddy_free_list(&mm, &middle);
> + drm_buddy_free_list(&mm, &middle, 0);
> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> 3 * ps, ps, &allocated,
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> @@ -74,7 +74,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> "buddy_alloc didn't error size=%u\n", 2 * ps);
>
> - drm_buddy_free_list(&mm, &right);
> + drm_buddy_free_list(&mm, &right, 0);
> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> 3 * ps, ps, &allocated,
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> @@ -89,7 +89,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> "buddy_alloc hit an error size=%u\n", 2 * ps);
>
> - drm_buddy_free_list(&mm, &left);
> + drm_buddy_free_list(&mm, &left, 0);
> KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> 3 * ps, ps, &allocated,
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> @@ -101,7 +101,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
>
> KUNIT_ASSERT_EQ(test, total, ps * 2 + ps * 3);
>
> - drm_buddy_free_list(&mm, &allocated);
> + drm_buddy_free_list(&mm, &allocated, 0);
> drm_buddy_fini(&mm);
> }
>
> @@ -170,7 +170,7 @@ static void drm_test_buddy_alloc_pathological(struct kunit *test)
> top, max_order);
> }
>
> - drm_buddy_free_list(&mm, &holes);
> + drm_buddy_free_list(&mm, &holes, 0);
>
> /* Nothing larger than blocks of chunk_size now available */
> for (order = 1; order <= max_order; order++) {
> @@ -182,7 +182,7 @@ static void drm_test_buddy_alloc_pathological(struct kunit *test)
> }
>
> list_splice_tail(&holes, &blocks);
> - drm_buddy_free_list(&mm, &blocks);
> + drm_buddy_free_list(&mm, &blocks, 0);
> drm_buddy_fini(&mm);
> }
>
> @@ -277,7 +277,7 @@ static void drm_test_buddy_alloc_pessimistic(struct kunit *test)
>
> list_del(&block->link);
> drm_buddy_free_block(&mm, block);
> - drm_buddy_free_list(&mm, &blocks);
> + drm_buddy_free_list(&mm, &blocks, 0);
> drm_buddy_fini(&mm);
> }
>
> @@ -323,7 +323,7 @@ static void drm_test_buddy_alloc_optimistic(struct kunit *test)
> size, size, &tmp, flags),
> "buddy_alloc unexpectedly succeeded, it should be full!");
>
> - drm_buddy_free_list(&mm, &blocks);
> + drm_buddy_free_list(&mm, &blocks, 0);
> drm_buddy_fini(&mm);
> }
>
> @@ -358,7 +358,7 @@ static void drm_test_buddy_alloc_limit(struct kunit *test)
> drm_buddy_block_size(&mm, block),
> BIT_ULL(mm.max_order) * PAGE_SIZE);
>
> - drm_buddy_free_list(&mm, &allocated);
> + drm_buddy_free_list(&mm, &allocated, 0);
> drm_buddy_fini(&mm);
> }
>
> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> index 115ec745e502..1ad678b62c4a 100644
> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> @@ -196,7 +196,7 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
> return 0;
>
> error_free_blocks:
> - drm_buddy_free_list(mm, &vres->blocks);
> + drm_buddy_free_list(mm, &vres->blocks, 0);
> mutex_unlock(&mgr->lock);
> error_fini:
> ttm_resource_fini(man, &vres->base);
> @@ -214,7 +214,7 @@ static void xe_ttm_vram_mgr_del(struct ttm_resource_manager *man,
> struct drm_buddy *mm = &mgr->mm;
>
> mutex_lock(&mgr->lock);
> - drm_buddy_free_list(mm, &vres->blocks);
> + drm_buddy_free_list(mm, &vres->blocks, 0);
> mgr->visible_avail += vres->used_visible_size;
> mutex_unlock(&mgr->lock);
>
> diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h
> index a5b39fc01003..82570f77e817 100644
> --- a/include/drm/drm_buddy.h
> +++ b/include/drm/drm_buddy.h
> @@ -25,6 +25,8 @@
> #define DRM_BUDDY_RANGE_ALLOCATION BIT(0)
> #define DRM_BUDDY_TOPDOWN_ALLOCATION BIT(1)
> #define DRM_BUDDY_CONTIGUOUS_ALLOCATION BIT(2)
> +#define DRM_BUDDY_CLEAR_ALLOCATION BIT(3)
> +#define DRM_BUDDY_CLEARED BIT(4)
>
> struct drm_buddy_block {
> #define DRM_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
> @@ -32,8 +34,9 @@ struct drm_buddy_block {
> #define DRM_BUDDY_ALLOCATED (1 << 10)
> #define DRM_BUDDY_FREE (2 << 10)
> #define DRM_BUDDY_SPLIT (3 << 10)
> +#define DRM_BUDDY_HEADER_CLEAR GENMASK_ULL(9, 9)
> /* Free to be used, if needed in the future */
> -#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
> +#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(8, 6)
> #define DRM_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
> u64 header;
>
> @@ -86,6 +89,7 @@ struct drm_buddy {
> u64 chunk_size;
> u64 size;
> u64 avail;
> + u64 clear_avail;
> };
>
> static inline u64
> @@ -112,6 +116,12 @@ drm_buddy_block_is_allocated(struct drm_buddy_block *block)
> return drm_buddy_block_state(block) == DRM_BUDDY_ALLOCATED;
> }
>
> +static inline bool
> +drm_buddy_block_is_clear(struct drm_buddy_block *block)
> +{
> + return block->header & DRM_BUDDY_HEADER_CLEAR;
> +}
> +
> static inline bool
> drm_buddy_block_is_free(struct drm_buddy_block *block)
> {
> @@ -150,7 +160,9 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>
> void drm_buddy_free_block(struct drm_buddy *mm, struct drm_buddy_block *block);
>
> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head *objects);
> +void drm_buddy_free_list(struct drm_buddy *mm,
> + struct list_head *objects,
> + unsigned int flags);
>
> void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p);
> void drm_buddy_block_print(struct drm_buddy *mm,
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-18 21:40 ` [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality Arunpravin Paneer Selvam
2024-03-19 10:28 ` Christian König
@ 2024-03-26 13:38 ` Alex Deucher
2024-03-26 13:59 ` Paneer Selvam, Arunpravin
1 sibling, 1 reply; 21+ messages in thread
From: Alex Deucher @ 2024-03-26 13:38 UTC (permalink / raw)
To: Arunpravin Paneer Selvam
Cc: dri-devel, amd-gfx, christian.koenig, alexander.deucher,
matthew.auld, mario.limonciello, felix.kuehling
On Mon, Mar 18, 2024 at 5:47 PM Arunpravin Paneer Selvam
<Arunpravin.PaneerSelvam@amd.com> wrote:
>
> Add clear page support in vram memory region.
>
> v1(Christian):
> - Dont handle clear page as TTM flag since when moving the BO back
> in from GTT again we don't need that.
> - Make a specialized version of amdgpu_fill_buffer() which only
> clears the VRAM areas which are not already cleared
> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
> amdgpu_object.c
>
> v2:
> - Modify the function name amdgpu_ttm_* (Alex)
> - Drop the delayed parameter (Christian)
> - handle amdgpu_res_cleared(&cursor) just above the size
> calculation (Christian)
> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
> in the free path to properly wait for fences etc.. (Christian)
>
> v3(Christian):
> - Remove buffer clear code in VRAM manager instead change the
> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
> the DRM_BUDDY_CLEARED flag.
> - Remove ! from amdgpu_res_cleared(&cursor) check.
>
> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
> 6 files changed, 111 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> index 8bc79924d171..c92d92b28a57 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> @@ -39,6 +39,7 @@
> #include "amdgpu.h"
> #include "amdgpu_trace.h"
> #include "amdgpu_amdkfd.h"
> +#include "amdgpu_vram_mgr.h"
>
> /**
> * DOC: amdgpu_object
> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
> if (!amdgpu_bo_support_uswc(bo->flags))
> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
>
> - if (adev->ras_enabled)
> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>
> bo->tbo.bdev = &adev->mman.bdev;
> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>
> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
> - struct dma_fence *fence;
> + struct dma_fence *fence = NULL;
>
> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence, true);
> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
> if (unlikely(r))
> goto fail_unreserve;
>
> - dma_resv_add_fence(bo->tbo.base.resv, fence,
> - DMA_RESV_USAGE_KERNEL);
> - dma_fence_put(fence);
> + if (fence) {
> + dma_resv_add_fence(bo->tbo.base.resv, fence,
> + DMA_RESV_USAGE_KERNEL);
> + dma_fence_put(fence);
> + }
> }
> if (!bp->resv)
> amdgpu_bo_unreserve(bo);
> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo)
> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
> return;
>
> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv, &fence, true);
> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
> if (!WARN_ON(r)) {
> + struct amdgpu_vram_mgr_resource *vres;
> +
> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
> + vres->flags |= DRM_BUDDY_CLEARED;
> amdgpu_bo_fence(abo, fence, false);
> dma_fence_put(fence);
> }
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> index 381101d2bf05..50fcd86e1033 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct amdgpu_res_cursor *cur, uint64_t size)
> }
> }
>
> +/**
> + * amdgpu_res_cleared - check if blocks are cleared
> + *
> + * @cur: the cursor to extract the block
> + *
> + * Check if the @cur block is cleared
> + */
> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
> +{
> + struct drm_buddy_block *block;
> +
> + switch (cur->mem_type) {
> + case TTM_PL_VRAM:
> + block = cur->node;
> +
> + if (!amdgpu_vram_mgr_is_cleared(block))
> + return false;
> + break;
> + default:
> + return false;
> + }
> +
> + return true;
> +}
> +
> #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> index 8722beba494e..bcbffe909b47 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
> struct dma_fence *wipe_fence = NULL;
>
> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
> - false);
> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
> + false);
> if (r) {
> goto error;
> } else if (wipe_fence) {
> + struct amdgpu_vram_mgr_resource *vres;
> +
> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
> + vres->flags |= DRM_BUDDY_CLEARED;
> dma_fence_put(fence);
> fence = wipe_fence;
> }
> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_ring *ring, uint32_t src_data,
> return 0;
> }
>
> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
> + struct dma_resv *resv,
> + struct dma_fence **fence)
> +{
> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
> + struct amdgpu_res_cursor cursor;
> + struct dma_fence *f = NULL;
> + u64 addr;
> + int r;
> +
> + if (!adev->mman.buffer_funcs_enabled)
> + return -EINVAL;
> +
> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
> +
> + mutex_lock(&adev->mman.gtt_window_lock);
> + while (cursor.remaining) {
> + struct dma_fence *next = NULL;
> + u64 size;
> +
> + if (amdgpu_res_cleared(&cursor)) {
> + amdgpu_res_next(&cursor, cursor.size);
> + continue;
> + }
> +
> + /* Never clear more than 256MiB at once to avoid timeouts */
> + size = min(cursor.size, 256ULL << 20);
> +
> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
> + 1, ring, false, &size, &addr);
> + if (r)
> + goto err;
> +
> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
> + &next, true, true);
> + if (r)
> + goto err;
> +
> + dma_fence_put(f);
> + f = next;
> +
> + amdgpu_res_next(&cursor, size);
> + }
> +err:
> + mutex_unlock(&adev->mman.gtt_window_lock);
> + if (fence)
> + *fence = dma_fence_get(f);
> + dma_fence_put(f);
> +
> + return r;
> +}
> +
> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
> uint32_t src_data,
> struct dma_resv *resv,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> index 65ec82141a8e..b404d89d52e5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> @@ -38,8 +38,6 @@
> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
>
> -#define AMDGPU_POISON 0xd0bed0be
> -
> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
>
> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
> uint64_t size, bool tmz,
> struct dma_resv *resv,
> struct dma_fence **f);
> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
> + struct dma_resv *resv,
> + struct dma_fence **fence);
> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
> uint32_t src_data,
> struct dma_resv *resv,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> index c0c851409241..e494f5bf136a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> {
> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
> struct amdgpu_device *adev = to_amdgpu_device(mgr);
> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
> u64 vis_usage = 0, max_bytes, min_block_size;
> struct amdgpu_vram_mgr_resource *vres;
> u64 size, remaining_size, lpfn, fpfn;
> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>
> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
Is there any reason to not always do this?
Alex
> +
> if (fpfn || lpfn != mgr->mm.size)
> /* Allocate blocks in desired range */
> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
>
> amdgpu_vram_mgr_do_reserve(man);
>
> - drm_buddy_free_list(mm, &vres->blocks, 0);
> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
> mutex_unlock(&mgr->lock);
>
> atomic64_sub(vis_usage, &mgr->vis_usage);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> index 0e04e42cf809..8478522d7366 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> @@ -53,6 +53,11 @@ static inline u64 amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
> }
>
> +static inline bool amdgpu_vram_mgr_is_cleared(struct drm_buddy_block *block)
> +{
> + return drm_buddy_block_is_clear(block);
> +}
> +
> static inline struct amdgpu_vram_mgr_resource *
> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
> {
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-26 13:38 ` Alex Deucher
@ 2024-03-26 13:59 ` Paneer Selvam, Arunpravin
2024-03-26 14:01 ` Alex Deucher
0 siblings, 1 reply; 21+ messages in thread
From: Paneer Selvam, Arunpravin @ 2024-03-26 13:59 UTC (permalink / raw)
To: Alex Deucher
Cc: dri-devel, amd-gfx, christian.koenig, alexander.deucher,
matthew.auld, mario.limonciello, felix.kuehling
Hi Alex,
On 3/26/2024 7:08 PM, Alex Deucher wrote:
> On Mon, Mar 18, 2024 at 5:47 PM Arunpravin Paneer Selvam
> <Arunpravin.PaneerSelvam@amd.com> wrote:
>> Add clear page support in vram memory region.
>>
>> v1(Christian):
>> - Dont handle clear page as TTM flag since when moving the BO back
>> in from GTT again we don't need that.
>> - Make a specialized version of amdgpu_fill_buffer() which only
>> clears the VRAM areas which are not already cleared
>> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
>> amdgpu_object.c
>>
>> v2:
>> - Modify the function name amdgpu_ttm_* (Alex)
>> - Drop the delayed parameter (Christian)
>> - handle amdgpu_res_cleared(&cursor) just above the size
>> calculation (Christian)
>> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
>> in the free path to properly wait for fences etc.. (Christian)
>>
>> v3(Christian):
>> - Remove buffer clear code in VRAM manager instead change the
>> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
>> the DRM_BUDDY_CLEARED flag.
>> - Remove ! from amdgpu_res_cleared(&cursor) check.
>>
>> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
>> ---
>> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
>> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
>> 6 files changed, 111 insertions(+), 13 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>> index 8bc79924d171..c92d92b28a57 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>> @@ -39,6 +39,7 @@
>> #include "amdgpu.h"
>> #include "amdgpu_trace.h"
>> #include "amdgpu_amdkfd.h"
>> +#include "amdgpu_vram_mgr.h"
>>
>> /**
>> * DOC: amdgpu_object
>> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>> if (!amdgpu_bo_support_uswc(bo->flags))
>> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
>>
>> - if (adev->ras_enabled)
>> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>
>> bo->tbo.bdev = &adev->mman.bdev;
>> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
>> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>
>> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
>> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
>> - struct dma_fence *fence;
>> + struct dma_fence *fence = NULL;
>>
>> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence, true);
>> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
>> if (unlikely(r))
>> goto fail_unreserve;
>>
>> - dma_resv_add_fence(bo->tbo.base.resv, fence,
>> - DMA_RESV_USAGE_KERNEL);
>> - dma_fence_put(fence);
>> + if (fence) {
>> + dma_resv_add_fence(bo->tbo.base.resv, fence,
>> + DMA_RESV_USAGE_KERNEL);
>> + dma_fence_put(fence);
>> + }
>> }
>> if (!bp->resv)
>> amdgpu_bo_unreserve(bo);
>> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo)
>> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
>> return;
>>
>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv, &fence, true);
>> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
>> if (!WARN_ON(r)) {
>> + struct amdgpu_vram_mgr_resource *vres;
>> +
>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>> + vres->flags |= DRM_BUDDY_CLEARED;
>> amdgpu_bo_fence(abo, fence, false);
>> dma_fence_put(fence);
>> }
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>> index 381101d2bf05..50fcd86e1033 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct amdgpu_res_cursor *cur, uint64_t size)
>> }
>> }
>>
>> +/**
>> + * amdgpu_res_cleared - check if blocks are cleared
>> + *
>> + * @cur: the cursor to extract the block
>> + *
>> + * Check if the @cur block is cleared
>> + */
>> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
>> +{
>> + struct drm_buddy_block *block;
>> +
>> + switch (cur->mem_type) {
>> + case TTM_PL_VRAM:
>> + block = cur->node;
>> +
>> + if (!amdgpu_vram_mgr_is_cleared(block))
>> + return false;
>> + break;
>> + default:
>> + return false;
>> + }
>> +
>> + return true;
>> +}
>> +
>> #endif
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> index 8722beba494e..bcbffe909b47 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
>> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
>> struct dma_fence *wipe_fence = NULL;
>>
>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
>> - false);
>> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
>> + false);
>> if (r) {
>> goto error;
>> } else if (wipe_fence) {
>> + struct amdgpu_vram_mgr_resource *vres;
>> +
>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>> + vres->flags |= DRM_BUDDY_CLEARED;
>> dma_fence_put(fence);
>> fence = wipe_fence;
>> }
>> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_ring *ring, uint32_t src_data,
>> return 0;
>> }
>>
>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>> + struct dma_resv *resv,
>> + struct dma_fence **fence)
>> +{
>> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
>> + struct amdgpu_res_cursor cursor;
>> + struct dma_fence *f = NULL;
>> + u64 addr;
>> + int r;
>> +
>> + if (!adev->mman.buffer_funcs_enabled)
>> + return -EINVAL;
>> +
>> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
>> +
>> + mutex_lock(&adev->mman.gtt_window_lock);
>> + while (cursor.remaining) {
>> + struct dma_fence *next = NULL;
>> + u64 size;
>> +
>> + if (amdgpu_res_cleared(&cursor)) {
>> + amdgpu_res_next(&cursor, cursor.size);
>> + continue;
>> + }
>> +
>> + /* Never clear more than 256MiB at once to avoid timeouts */
>> + size = min(cursor.size, 256ULL << 20);
>> +
>> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
>> + 1, ring, false, &size, &addr);
>> + if (r)
>> + goto err;
>> +
>> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
>> + &next, true, true);
>> + if (r)
>> + goto err;
>> +
>> + dma_fence_put(f);
>> + f = next;
>> +
>> + amdgpu_res_next(&cursor, size);
>> + }
>> +err:
>> + mutex_unlock(&adev->mman.gtt_window_lock);
>> + if (fence)
>> + *fence = dma_fence_get(f);
>> + dma_fence_put(f);
>> +
>> + return r;
>> +}
>> +
>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>> uint32_t src_data,
>> struct dma_resv *resv,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>> index 65ec82141a8e..b404d89d52e5 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>> @@ -38,8 +38,6 @@
>> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
>> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
>>
>> -#define AMDGPU_POISON 0xd0bed0be
>> -
>> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
>> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
>>
>> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
>> uint64_t size, bool tmz,
>> struct dma_resv *resv,
>> struct dma_fence **f);
>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>> + struct dma_resv *resv,
>> + struct dma_fence **fence);
>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>> uint32_t src_data,
>> struct dma_resv *resv,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> index c0c851409241..e494f5bf136a 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
>> {
>> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
>> struct amdgpu_device *adev = to_amdgpu_device(mgr);
>> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
>> u64 vis_usage = 0, max_bytes, min_block_size;
>> struct amdgpu_vram_mgr_resource *vres;
>> u64 size, remaining_size, lpfn, fpfn;
>> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
>> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
>> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>>
>> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
>> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
> Is there any reason to not always do this?
Here we are trying to keep a pool of cleared memory which are cleared on
free path and that can given
to the application which requires a zeroed memory. I think here if we
set always clear the memory,
we would go over the limit of cleared memory in that particular instance
and the application should wait until
the hardware clears the memory and this might impact the overall
performance.
Thanks,
Arun.
>
> Alex
>
>
>> +
>> if (fpfn || lpfn != mgr->mm.size)
>> /* Allocate blocks in desired range */
>> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
>> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
>>
>> amdgpu_vram_mgr_do_reserve(man);
>>
>> - drm_buddy_free_list(mm, &vres->blocks, 0);
>> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
>> mutex_unlock(&mgr->lock);
>>
>> atomic64_sub(vis_usage, &mgr->vis_usage);
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>> index 0e04e42cf809..8478522d7366 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>> @@ -53,6 +53,11 @@ static inline u64 amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
>> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
>> }
>>
>> +static inline bool amdgpu_vram_mgr_is_cleared(struct drm_buddy_block *block)
>> +{
>> + return drm_buddy_block_is_clear(block);
>> +}
>> +
>> static inline struct amdgpu_vram_mgr_resource *
>> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
>> {
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-26 13:59 ` Paneer Selvam, Arunpravin
@ 2024-03-26 14:01 ` Alex Deucher
2024-03-26 14:53 ` Alex Deucher
0 siblings, 1 reply; 21+ messages in thread
From: Alex Deucher @ 2024-03-26 14:01 UTC (permalink / raw)
To: Paneer Selvam, Arunpravin
Cc: dri-devel, amd-gfx, christian.koenig, alexander.deucher,
matthew.auld, mario.limonciello, felix.kuehling
On Tue, Mar 26, 2024 at 9:59 AM Paneer Selvam, Arunpravin
<arunpravin.paneerselvam@amd.com> wrote:
>
> Hi Alex,
>
> On 3/26/2024 7:08 PM, Alex Deucher wrote:
> > On Mon, Mar 18, 2024 at 5:47 PM Arunpravin Paneer Selvam
> > <Arunpravin.PaneerSelvam@amd.com> wrote:
> >> Add clear page support in vram memory region.
> >>
> >> v1(Christian):
> >> - Dont handle clear page as TTM flag since when moving the BO back
> >> in from GTT again we don't need that.
> >> - Make a specialized version of amdgpu_fill_buffer() which only
> >> clears the VRAM areas which are not already cleared
> >> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
> >> amdgpu_object.c
> >>
> >> v2:
> >> - Modify the function name amdgpu_ttm_* (Alex)
> >> - Drop the delayed parameter (Christian)
> >> - handle amdgpu_res_cleared(&cursor) just above the size
> >> calculation (Christian)
> >> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
> >> in the free path to properly wait for fences etc.. (Christian)
> >>
> >> v3(Christian):
> >> - Remove buffer clear code in VRAM manager instead change the
> >> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
> >> the DRM_BUDDY_CLEARED flag.
> >> - Remove ! from amdgpu_res_cleared(&cursor) check.
> >>
> >> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
> >> Suggested-by: Christian König <christian.koenig@amd.com>
> >> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
> >> ---
> >> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
> >> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
> >> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
> >> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
> >> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
> >> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
> >> 6 files changed, 111 insertions(+), 13 deletions(-)
> >>
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> >> index 8bc79924d171..c92d92b28a57 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> >> @@ -39,6 +39,7 @@
> >> #include "amdgpu.h"
> >> #include "amdgpu_trace.h"
> >> #include "amdgpu_amdkfd.h"
> >> +#include "amdgpu_vram_mgr.h"
> >>
> >> /**
> >> * DOC: amdgpu_object
> >> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
> >> if (!amdgpu_bo_support_uswc(bo->flags))
> >> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
> >>
> >> - if (adev->ras_enabled)
> >> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
> >> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
> >>
> >> bo->tbo.bdev = &adev->mman.bdev;
> >> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
> >> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
> >>
> >> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
> >> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
> >> - struct dma_fence *fence;
> >> + struct dma_fence *fence = NULL;
> >>
> >> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence, true);
> >> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
> >> if (unlikely(r))
> >> goto fail_unreserve;
> >>
> >> - dma_resv_add_fence(bo->tbo.base.resv, fence,
> >> - DMA_RESV_USAGE_KERNEL);
> >> - dma_fence_put(fence);
> >> + if (fence) {
> >> + dma_resv_add_fence(bo->tbo.base.resv, fence,
> >> + DMA_RESV_USAGE_KERNEL);
> >> + dma_fence_put(fence);
> >> + }
> >> }
> >> if (!bp->resv)
> >> amdgpu_bo_unreserve(bo);
> >> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo)
> >> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
> >> return;
> >>
> >> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv, &fence, true);
> >> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
> >> if (!WARN_ON(r)) {
> >> + struct amdgpu_vram_mgr_resource *vres;
> >> +
> >> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
> >> + vres->flags |= DRM_BUDDY_CLEARED;
> >> amdgpu_bo_fence(abo, fence, false);
> >> dma_fence_put(fence);
> >> }
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> >> index 381101d2bf05..50fcd86e1033 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> >> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct amdgpu_res_cursor *cur, uint64_t size)
> >> }
> >> }
> >>
> >> +/**
> >> + * amdgpu_res_cleared - check if blocks are cleared
> >> + *
> >> + * @cur: the cursor to extract the block
> >> + *
> >> + * Check if the @cur block is cleared
> >> + */
> >> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
> >> +{
> >> + struct drm_buddy_block *block;
> >> +
> >> + switch (cur->mem_type) {
> >> + case TTM_PL_VRAM:
> >> + block = cur->node;
> >> +
> >> + if (!amdgpu_vram_mgr_is_cleared(block))
> >> + return false;
> >> + break;
> >> + default:
> >> + return false;
> >> + }
> >> +
> >> + return true;
> >> +}
> >> +
> >> #endif
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> >> index 8722beba494e..bcbffe909b47 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> >> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
> >> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
> >> struct dma_fence *wipe_fence = NULL;
> >>
> >> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
> >> - false);
> >> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
> >> + false);
> >> if (r) {
> >> goto error;
> >> } else if (wipe_fence) {
> >> + struct amdgpu_vram_mgr_resource *vres;
> >> +
> >> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
> >> + vres->flags |= DRM_BUDDY_CLEARED;
> >> dma_fence_put(fence);
> >> fence = wipe_fence;
> >> }
> >> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_ring *ring, uint32_t src_data,
> >> return 0;
> >> }
> >>
> >> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
> >> + struct dma_resv *resv,
> >> + struct dma_fence **fence)
> >> +{
> >> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
> >> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
> >> + struct amdgpu_res_cursor cursor;
> >> + struct dma_fence *f = NULL;
> >> + u64 addr;
> >> + int r;
> >> +
> >> + if (!adev->mman.buffer_funcs_enabled)
> >> + return -EINVAL;
> >> +
> >> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
> >> +
> >> + mutex_lock(&adev->mman.gtt_window_lock);
> >> + while (cursor.remaining) {
> >> + struct dma_fence *next = NULL;
> >> + u64 size;
> >> +
> >> + if (amdgpu_res_cleared(&cursor)) {
> >> + amdgpu_res_next(&cursor, cursor.size);
> >> + continue;
> >> + }
> >> +
> >> + /* Never clear more than 256MiB at once to avoid timeouts */
> >> + size = min(cursor.size, 256ULL << 20);
> >> +
> >> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
> >> + 1, ring, false, &size, &addr);
> >> + if (r)
> >> + goto err;
> >> +
> >> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
> >> + &next, true, true);
> >> + if (r)
> >> + goto err;
> >> +
> >> + dma_fence_put(f);
> >> + f = next;
> >> +
> >> + amdgpu_res_next(&cursor, size);
> >> + }
> >> +err:
> >> + mutex_unlock(&adev->mman.gtt_window_lock);
> >> + if (fence)
> >> + *fence = dma_fence_get(f);
> >> + dma_fence_put(f);
> >> +
> >> + return r;
> >> +}
> >> +
> >> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
> >> uint32_t src_data,
> >> struct dma_resv *resv,
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> >> index 65ec82141a8e..b404d89d52e5 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> >> @@ -38,8 +38,6 @@
> >> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
> >> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
> >>
> >> -#define AMDGPU_POISON 0xd0bed0be
> >> -
> >> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
> >> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
> >>
> >> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
> >> uint64_t size, bool tmz,
> >> struct dma_resv *resv,
> >> struct dma_fence **f);
> >> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
> >> + struct dma_resv *resv,
> >> + struct dma_fence **fence);
> >> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
> >> uint32_t src_data,
> >> struct dma_resv *resv,
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> >> index c0c851409241..e494f5bf136a 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> >> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> >> {
> >> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
> >> struct amdgpu_device *adev = to_amdgpu_device(mgr);
> >> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
> >> u64 vis_usage = 0, max_bytes, min_block_size;
> >> struct amdgpu_vram_mgr_resource *vres;
> >> u64 size, remaining_size, lpfn, fpfn;
> >> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> >> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
> >> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
> >>
> >> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
> >> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
> > Is there any reason to not always do this?
> Here we are trying to keep a pool of cleared memory which are cleared on
> free path and that can given
> to the application which requires a zeroed memory. I think here if we
> set always clear the memory,
> we would go over the limit of cleared memory in that particular instance
> and the application should wait until
> the hardware clears the memory and this might impact the overall
> performance.
I'd like to have the driver always deliver cleared memory.
Alex
>
> Thanks,
> Arun.
> >
> > Alex
> >
> >
> >> +
> >> if (fpfn || lpfn != mgr->mm.size)
> >> /* Allocate blocks in desired range */
> >> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
> >> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
> >>
> >> amdgpu_vram_mgr_do_reserve(man);
> >>
> >> - drm_buddy_free_list(mm, &vres->blocks, 0);
> >> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
> >> mutex_unlock(&mgr->lock);
> >>
> >> atomic64_sub(vis_usage, &mgr->vis_usage);
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> >> index 0e04e42cf809..8478522d7366 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> >> @@ -53,6 +53,11 @@ static inline u64 amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
> >> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
> >> }
> >>
> >> +static inline bool amdgpu_vram_mgr_is_cleared(struct drm_buddy_block *block)
> >> +{
> >> + return drm_buddy_block_is_clear(block);
> >> +}
> >> +
> >> static inline struct amdgpu_vram_mgr_resource *
> >> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
> >> {
> >> --
> >> 2.25.1
> >>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-26 14:01 ` Alex Deucher
@ 2024-03-26 14:53 ` Alex Deucher
2024-03-27 8:23 ` Paneer Selvam, Arunpravin
2024-04-02 8:17 ` Christian König
0 siblings, 2 replies; 21+ messages in thread
From: Alex Deucher @ 2024-03-26 14:53 UTC (permalink / raw)
To: Paneer Selvam, Arunpravin
Cc: dri-devel, amd-gfx, christian.koenig, alexander.deucher,
matthew.auld, mario.limonciello, felix.kuehling
On Tue, Mar 26, 2024 at 10:01 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 9:59 AM Paneer Selvam, Arunpravin
> <arunpravin.paneerselvam@amd.com> wrote:
> >
> > Hi Alex,
> >
> > On 3/26/2024 7:08 PM, Alex Deucher wrote:
> > > On Mon, Mar 18, 2024 at 5:47 PM Arunpravin Paneer Selvam
> > > <Arunpravin.PaneerSelvam@amd.com> wrote:
> > >> Add clear page support in vram memory region.
> > >>
> > >> v1(Christian):
> > >> - Dont handle clear page as TTM flag since when moving the BO back
> > >> in from GTT again we don't need that.
> > >> - Make a specialized version of amdgpu_fill_buffer() which only
> > >> clears the VRAM areas which are not already cleared
> > >> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
> > >> amdgpu_object.c
> > >>
> > >> v2:
> > >> - Modify the function name amdgpu_ttm_* (Alex)
> > >> - Drop the delayed parameter (Christian)
> > >> - handle amdgpu_res_cleared(&cursor) just above the size
> > >> calculation (Christian)
> > >> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
> > >> in the free path to properly wait for fences etc.. (Christian)
> > >>
> > >> v3(Christian):
> > >> - Remove buffer clear code in VRAM manager instead change the
> > >> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
> > >> the DRM_BUDDY_CLEARED flag.
> > >> - Remove ! from amdgpu_res_cleared(&cursor) check.
> > >>
> > >> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
> > >> Suggested-by: Christian König <christian.koenig@amd.com>
> > >> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
> > >> ---
> > >> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
> > >> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
> > >> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
> > >> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
> > >> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
> > >> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
> > >> 6 files changed, 111 insertions(+), 13 deletions(-)
> > >>
> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> > >> index 8bc79924d171..c92d92b28a57 100644
> > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> > >> @@ -39,6 +39,7 @@
> > >> #include "amdgpu.h"
> > >> #include "amdgpu_trace.h"
> > >> #include "amdgpu_amdkfd.h"
> > >> +#include "amdgpu_vram_mgr.h"
> > >>
> > >> /**
> > >> * DOC: amdgpu_object
> > >> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
> > >> if (!amdgpu_bo_support_uswc(bo->flags))
> > >> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
> > >>
> > >> - if (adev->ras_enabled)
> > >> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
> > >> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
> > >>
> > >> bo->tbo.bdev = &adev->mman.bdev;
> > >> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
> > >> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
> > >>
> > >> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
> > >> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
> > >> - struct dma_fence *fence;
> > >> + struct dma_fence *fence = NULL;
> > >>
> > >> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence, true);
> > >> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
> > >> if (unlikely(r))
> > >> goto fail_unreserve;
> > >>
> > >> - dma_resv_add_fence(bo->tbo.base.resv, fence,
> > >> - DMA_RESV_USAGE_KERNEL);
> > >> - dma_fence_put(fence);
> > >> + if (fence) {
> > >> + dma_resv_add_fence(bo->tbo.base.resv, fence,
> > >> + DMA_RESV_USAGE_KERNEL);
> > >> + dma_fence_put(fence);
> > >> + }
> > >> }
> > >> if (!bp->resv)
> > >> amdgpu_bo_unreserve(bo);
> > >> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo)
> > >> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
> > >> return;
> > >>
> > >> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv, &fence, true);
> > >> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
> > >> if (!WARN_ON(r)) {
> > >> + struct amdgpu_vram_mgr_resource *vres;
> > >> +
> > >> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
> > >> + vres->flags |= DRM_BUDDY_CLEARED;
> > >> amdgpu_bo_fence(abo, fence, false);
> > >> dma_fence_put(fence);
> > >> }
> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> > >> index 381101d2bf05..50fcd86e1033 100644
> > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
> > >> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct amdgpu_res_cursor *cur, uint64_t size)
> > >> }
> > >> }
> > >>
> > >> +/**
> > >> + * amdgpu_res_cleared - check if blocks are cleared
> > >> + *
> > >> + * @cur: the cursor to extract the block
> > >> + *
> > >> + * Check if the @cur block is cleared
> > >> + */
> > >> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
> > >> +{
> > >> + struct drm_buddy_block *block;
> > >> +
> > >> + switch (cur->mem_type) {
> > >> + case TTM_PL_VRAM:
> > >> + block = cur->node;
> > >> +
> > >> + if (!amdgpu_vram_mgr_is_cleared(block))
> > >> + return false;
> > >> + break;
> > >> + default:
> > >> + return false;
> > >> + }
> > >> +
> > >> + return true;
> > >> +}
> > >> +
> > >> #endif
> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > >> index 8722beba494e..bcbffe909b47 100644
> > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > >> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
> > >> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
> > >> struct dma_fence *wipe_fence = NULL;
> > >>
> > >> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
> > >> - false);
> > >> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
> > >> + false);
> > >> if (r) {
> > >> goto error;
> > >> } else if (wipe_fence) {
> > >> + struct amdgpu_vram_mgr_resource *vres;
> > >> +
> > >> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
> > >> + vres->flags |= DRM_BUDDY_CLEARED;
> > >> dma_fence_put(fence);
> > >> fence = wipe_fence;
> > >> }
> > >> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_ring *ring, uint32_t src_data,
> > >> return 0;
> > >> }
> > >>
> > >> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
> > >> + struct dma_resv *resv,
> > >> + struct dma_fence **fence)
> > >> +{
> > >> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
> > >> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
> > >> + struct amdgpu_res_cursor cursor;
> > >> + struct dma_fence *f = NULL;
> > >> + u64 addr;
> > >> + int r;
> > >> +
> > >> + if (!adev->mman.buffer_funcs_enabled)
> > >> + return -EINVAL;
> > >> +
> > >> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
> > >> +
> > >> + mutex_lock(&adev->mman.gtt_window_lock);
> > >> + while (cursor.remaining) {
> > >> + struct dma_fence *next = NULL;
> > >> + u64 size;
> > >> +
> > >> + if (amdgpu_res_cleared(&cursor)) {
> > >> + amdgpu_res_next(&cursor, cursor.size);
> > >> + continue;
> > >> + }
> > >> +
> > >> + /* Never clear more than 256MiB at once to avoid timeouts */
> > >> + size = min(cursor.size, 256ULL << 20);
> > >> +
> > >> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
> > >> + 1, ring, false, &size, &addr);
> > >> + if (r)
> > >> + goto err;
> > >> +
> > >> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
> > >> + &next, true, true);
> > >> + if (r)
> > >> + goto err;
> > >> +
> > >> + dma_fence_put(f);
> > >> + f = next;
> > >> +
> > >> + amdgpu_res_next(&cursor, size);
> > >> + }
> > >> +err:
> > >> + mutex_unlock(&adev->mman.gtt_window_lock);
> > >> + if (fence)
> > >> + *fence = dma_fence_get(f);
> > >> + dma_fence_put(f);
> > >> +
> > >> + return r;
> > >> +}
> > >> +
> > >> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
> > >> uint32_t src_data,
> > >> struct dma_resv *resv,
> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> > >> index 65ec82141a8e..b404d89d52e5 100644
> > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> > >> @@ -38,8 +38,6 @@
> > >> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
> > >> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
> > >>
> > >> -#define AMDGPU_POISON 0xd0bed0be
> > >> -
> > >> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
> > >> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
> > >>
> > >> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
> > >> uint64_t size, bool tmz,
> > >> struct dma_resv *resv,
> > >> struct dma_fence **f);
> > >> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
> > >> + struct dma_resv *resv,
> > >> + struct dma_fence **fence);
> > >> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
> > >> uint32_t src_data,
> > >> struct dma_resv *resv,
> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> > >> index c0c851409241..e494f5bf136a 100644
> > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> > >> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> > >> {
> > >> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
> > >> struct amdgpu_device *adev = to_amdgpu_device(mgr);
> > >> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
> > >> u64 vis_usage = 0, max_bytes, min_block_size;
> > >> struct amdgpu_vram_mgr_resource *vres;
> > >> u64 size, remaining_size, lpfn, fpfn;
> > >> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> > >> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
> > >> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
> > >>
> > >> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
> > >> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
> > > Is there any reason to not always do this?
> > Here we are trying to keep a pool of cleared memory which are cleared on
> > free path and that can given
> > to the application which requires a zeroed memory. I think here if we
> > set always clear the memory,
> > we would go over the limit of cleared memory in that particular instance
> > and the application should wait until
> > the hardware clears the memory and this might impact the overall
> > performance.
>
> I'd like to have the driver always deliver cleared memory.
Actually, I think we can just always set the flag in the allocation IOCTLs.
Alex
>
> Alex
>
> >
> > Thanks,
> > Arun.
> > >
> > > Alex
> > >
> > >
> > >> +
> > >> if (fpfn || lpfn != mgr->mm.size)
> > >> /* Allocate blocks in desired range */
> > >> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
> > >> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
> > >>
> > >> amdgpu_vram_mgr_do_reserve(man);
> > >>
> > >> - drm_buddy_free_list(mm, &vres->blocks, 0);
> > >> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
> > >> mutex_unlock(&mgr->lock);
> > >>
> > >> atomic64_sub(vis_usage, &mgr->vis_usage);
> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> > >> index 0e04e42cf809..8478522d7366 100644
> > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
> > >> @@ -53,6 +53,11 @@ static inline u64 amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
> > >> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
> > >> }
> > >>
> > >> +static inline bool amdgpu_vram_mgr_is_cleared(struct drm_buddy_block *block)
> > >> +{
> > >> + return drm_buddy_block_is_clear(block);
> > >> +}
> > >> +
> > >> static inline struct amdgpu_vram_mgr_resource *
> > >> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
> > >> {
> > >> --
> > >> 2.25.1
> > >>
> >
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 3/3] drm/tests: Add a test case for drm buddy clear allocation
2024-03-18 21:40 ` [PATCH v9 3/3] drm/tests: Add a test case for drm buddy clear allocation Arunpravin Paneer Selvam
@ 2024-03-26 17:46 ` Matthew Auld
0 siblings, 0 replies; 21+ messages in thread
From: Matthew Auld @ 2024-03-26 17:46 UTC (permalink / raw)
To: Arunpravin Paneer Selvam, dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, mario.limonciello, felix.kuehling
On 18/03/2024 21:40, Arunpravin Paneer Selvam wrote:
> Add a new test case for the drm buddy clear and dirty
> allocation.
>
> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
> Suggested-by: Matthew Auld <matthew.auld@intel.com>
> ---
> drivers/gpu/drm/tests/drm_buddy_test.c | 127 +++++++++++++++++++++++++
> 1 file changed, 127 insertions(+)
>
> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
> index 454ad9952f56..d355a6e61893 100644
> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> @@ -19,6 +19,132 @@ static inline u64 get_size(int order, u64 chunk_size)
> return (1 << order) * chunk_size;
> }
>
> +static void drm_test_buddy_alloc_clear(struct kunit *test)
> +{
> + unsigned long n_pages, total, i = 0;
> + const unsigned long ps = SZ_4K;
> + struct drm_buddy_block *block;
> + const int max_order = 12;
> + LIST_HEAD(allocated);
> + struct drm_buddy mm;
> + unsigned int order;
> + u64 mm_size, size;
Maybe just make these two u32 or unsigned long. That should be big
enough, plus avoids any kind of 32b compilation bugs below.
> + LIST_HEAD(dirty);
> + LIST_HEAD(clean);
> +
> + mm_size = PAGE_SIZE << max_order;
s/PAGE_SIZE/SZ_4K/ below also.
> + KUNIT_EXPECT_FALSE(test, drm_buddy_init(&mm, mm_size, ps));
> +
> + KUNIT_EXPECT_EQ(test, mm.max_order, max_order);
> +
> + /**
Drop the extra *, since is not actual kernel-doc. Below also.
> + * Idea is to allocate and free some random portion of the address space,
> + * returning those pages as non-dirty and randomly alternate between
> + * requesting dirty and non-dirty pages (not going over the limit
> + * we freed as non-dirty), putting that into two separate lists.
> + * Loop over both lists at the end checking that the dirty list
> + * is indeed all dirty pages and vice versa. Free it all again,
> + * keeping the dirty/clear status.
> + */
> + KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> + 5 * ps, ps, &allocated,
> + DRM_BUDDY_TOPDOWN_ALLOCATION),
> + "buddy_alloc hit an error size=%u\n", 5 * ps);
> + drm_buddy_free_list(&mm, &allocated, DRM_BUDDY_CLEARED);
> +
> + n_pages = 10;
> + do {
> + unsigned long flags;
> + struct list_head *list;
> + int slot = i % 2;
> +
> + if (slot == 0) {
> + list = &dirty;
> + flags = 0;
> + } else if (slot == 1) {
Could just be else {
> + list = &clean;
> + flags = DRM_BUDDY_CLEAR_ALLOCATION;
> + }
> +
> + KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> + ps, ps, list,
> + flags),
> + "buddy_alloc hit an error size=%u\n", ps);
> + } while (++i < n_pages);
> +
> + list_for_each_entry(block, &clean, link)
> + KUNIT_EXPECT_EQ(test, drm_buddy_block_is_clear(block), true);
> +
> + list_for_each_entry(block, &dirty, link)
> + KUNIT_EXPECT_EQ(test, drm_buddy_block_is_clear(block), false);
> +
> + drm_buddy_free_list(&mm, &clean, DRM_BUDDY_CLEARED);
> +
> + /**
> + * Trying to go over the clear limit for some allocation.
> + * The allocation should never fail with reasonable page-size.
> + */
> + KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> + 10 * ps, ps, &clean,
> + DRM_BUDDY_CLEAR_ALLOCATION),
> + "buddy_alloc hit an error size=%u\n", 10 * ps);
> +
> + drm_buddy_free_list(&mm, &clean, DRM_BUDDY_CLEARED);
> + drm_buddy_free_list(&mm, &dirty, 0);
> + drm_buddy_fini(&mm);
> +
> + KUNIT_EXPECT_FALSE(test, drm_buddy_init(&mm, mm_size, ps));
> +
> + /**
> + * Create a new mm. Intentionally fragment the address space by creating
> + * two alternating lists. Free both lists, one as dirty the other as clean.
> + * Try to allocate double the previous size with matching min_page_size. The
> + * allocation should never fail as it calls the force_merge. Also check that
> + * the page is always dirty after force_merge. Free the page as dirty, then
> + * repeat the whole thing, increment the order until we hit the max_order.
> + */
> +
> + order = 1;
> + do {
> + size = PAGE_SIZE << order;
> + i = 0;
> + n_pages = mm_size / ps;
> + do {
> + struct list_head *list;
> + int slot = i % 2;
> +
> + if (slot == 0)
> + list = &dirty;
> + else if (slot == 1)
else
> + list = &clean;
> +
> + KUNIT_ASSERT_FALSE_MSG(test,
> + drm_buddy_alloc_blocks(&mm, 0, mm_size,
> + ps, ps, list, 0),
> + "buddy_alloc hit an error size=%u\n",
> + ps);
> + } while (++i < n_pages);
I think we only need to do this once at the beginning, and then just
loop over each order starting from one? Otherwise on the first iteration
here we fragment the entire address space, but then only allocate single
order=1. And then we repeat the whole fragmentation again, which seems
unnecessary.
> +
> + drm_buddy_free_list(&mm, &clean, DRM_BUDDY_CLEARED);
> + drm_buddy_free_list(&mm, &dirty, 0);
> +
> + KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> + size, size, &allocated,
> + DRM_BUDDY_CLEAR_ALLOCATION),
> + "buddy_alloc hit an error size=%u\n", size);
size=%llu or better just make size u32.
> + total = 0;
> + list_for_each_entry(block, &allocated, link) {
> + KUNIT_EXPECT_EQ(test, drm_buddy_block_is_clear(block), false);
> + total += drm_buddy_block_size(&mm, block);
> + }
> + KUNIT_EXPECT_EQ(test, total, size);
> +
> + drm_buddy_free_list(&mm, &allocated, 0);
> + } while (++order <= max_order);
I think would be good to also do some non-power-of-two mm size. Just to
ensure we get some coverage for the multi-root force_merge during fini.
Something simple like create new mm here, allocate random size, free as
cleared, then call fini.
Looks good otherwise.
> +
> + drm_buddy_fini(&mm);
> +}
> +
> static void drm_test_buddy_alloc_contiguous(struct kunit *test)
> {
> const unsigned long ps = SZ_4K, mm_size = 16 * 3 * SZ_4K;
> @@ -368,6 +494,7 @@ static struct kunit_case drm_buddy_tests[] = {
> KUNIT_CASE(drm_test_buddy_alloc_pessimistic),
> KUNIT_CASE(drm_test_buddy_alloc_pathological),
> KUNIT_CASE(drm_test_buddy_alloc_contiguous),
> + KUNIT_CASE(drm_test_buddy_alloc_clear),
> {}
> };
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature
2024-03-18 21:40 [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature Arunpravin Paneer Selvam
` (2 preceding siblings ...)
2024-03-25 13:37 ` [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature Paneer Selvam, Arunpravin
@ 2024-03-26 18:09 ` Matthew Auld
2024-03-28 16:07 ` Paneer Selvam, Arunpravin
3 siblings, 1 reply; 21+ messages in thread
From: Matthew Auld @ 2024-03-26 18:09 UTC (permalink / raw)
To: Arunpravin Paneer Selvam, dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, mario.limonciello, felix.kuehling
On 18/03/2024 21:40, Arunpravin Paneer Selvam wrote:
> - Add tracking clear page feature.
>
> - Driver should enable the DRM_BUDDY_CLEARED flag if it
> successfully clears the blocks in the free path. On the otherhand,
> DRM buddy marks each block as cleared.
>
> - Track the available cleared pages size
>
> - If driver requests cleared memory we prefer cleared memory
> but fallback to uncleared if we can't find the cleared blocks.
> when driver requests uncleared memory we try to use uncleared but
> fallback to cleared memory if necessary.
>
> - When a block gets freed we clear it and mark the freed block as cleared,
> when there are buddies which are cleared as well we can merge them.
> Otherwise, we prefer to keep the blocks as separated.
>
> - Add a function to support defragmentation.
>
> v1:
> - Depends on the flag check DRM_BUDDY_CLEARED, enable the block as
> cleared. Else, reset the clear flag for each block in the list(Christian)
> - For merging the 2 cleared blocks compare as below,
> drm_buddy_is_clear(block) != drm_buddy_is_clear(buddy)(Christian)
> - Defragment the memory beginning from min_order
> till the required memory space is available.
>
> v2: (Matthew)
> - Add a wrapper drm_buddy_free_list_internal for the freeing of blocks
> operation within drm buddy.
> - Write a macro block_incompatible() to allocate the required blocks.
> - Update the xe driver for the drm_buddy_free_list change in arguments.
> - add a warning if the two blocks are incompatible on
> defragmentation
> - call full defragmentation in the fini() function
> - place a condition to test if min_order is equal to 0
> - replace the list with safe_reverse() variant as we might
> remove the block from the list.
>
> v3:
> - fix Gitlab user reported lockup issue.
> - Keep DRM_BUDDY_HEADER_CLEAR define sorted(Matthew)
> - modify to pass the root order instead max_order in fini()
> function(Matthew)
> - change bool 1 to true(Matthew)
> - add check if min_block_size is power of 2(Matthew)
> - modify the min_block_size datatype to u64(Matthew)
>
> v4:
> - rename the function drm_buddy_defrag with __force_merge.
> - Include __force_merge directly in drm buddy file and remove
> the defrag use in amdgpu driver.
> - Remove list_empty() check(Matthew)
> - Remove unnecessary space, headers and placement of new variables(Matthew)
> - Add a unit test case(Matthew)
>
> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
> Signed-off-by: Matthew Auld <matthew.auld@intel.com>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Matthew Auld <matthew.auld@intel.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
> drivers/gpu/drm/drm_buddy.c | 427 ++++++++++++++----
> drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 6 +-
> drivers/gpu/drm/tests/drm_buddy_test.c | 18 +-
> drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 4 +-
> include/drm/drm_buddy.h | 16 +-
> 6 files changed, 360 insertions(+), 117 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> index 8db880244324..c0c851409241 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> @@ -571,7 +571,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
> return 0;
>
> error_free_blocks:
> - drm_buddy_free_list(mm, &vres->blocks);
> + drm_buddy_free_list(mm, &vres->blocks, 0);
> mutex_unlock(&mgr->lock);
> error_fini:
> ttm_resource_fini(man, &vres->base);
> @@ -604,7 +604,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
>
> amdgpu_vram_mgr_do_reserve(man);
>
> - drm_buddy_free_list(mm, &vres->blocks);
> + drm_buddy_free_list(mm, &vres->blocks, 0);
> mutex_unlock(&mgr->lock);
>
> atomic64_sub(vis_usage, &mgr->vis_usage);
> @@ -912,7 +912,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev)
> kfree(rsv);
>
> list_for_each_entry_safe(rsv, temp, &mgr->reserved_pages, blocks) {
> - drm_buddy_free_list(&mgr->mm, &rsv->allocated);
> + drm_buddy_free_list(&mgr->mm, &rsv->allocated, 0);
> kfree(rsv);
> }
> if (!adev->gmc.is_app_apu)
> diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c
> index c4222b886db7..625a30a6b855 100644
> --- a/drivers/gpu/drm/drm_buddy.c
> +++ b/drivers/gpu/drm/drm_buddy.c
> @@ -38,8 +38,8 @@ static void drm_block_free(struct drm_buddy *mm,
> kmem_cache_free(slab_blocks, block);
> }
>
> -static void list_insert_sorted(struct drm_buddy *mm,
> - struct drm_buddy_block *block)
> +static void list_insert(struct drm_buddy *mm,
> + struct drm_buddy_block *block)
> {
> struct drm_buddy_block *node;
> struct list_head *head;
> @@ -57,6 +57,16 @@ static void list_insert_sorted(struct drm_buddy *mm,
> __list_add(&block->link, node->link.prev, &node->link);
> }
>
> +static void clear_reset(struct drm_buddy_block *block)
> +{
> + block->header &= ~DRM_BUDDY_HEADER_CLEAR;
> +}
> +
> +static void mark_cleared(struct drm_buddy_block *block)
> +{
> + block->header |= DRM_BUDDY_HEADER_CLEAR;
> +}
> +
> static void mark_allocated(struct drm_buddy_block *block)
> {
> block->header &= ~DRM_BUDDY_HEADER_STATE;
> @@ -71,7 +81,7 @@ static void mark_free(struct drm_buddy *mm,
> block->header &= ~DRM_BUDDY_HEADER_STATE;
> block->header |= DRM_BUDDY_FREE;
>
> - list_insert_sorted(mm, block);
> + list_insert(mm, block);
> }
>
> static void mark_split(struct drm_buddy_block *block)
> @@ -82,6 +92,114 @@ static void mark_split(struct drm_buddy_block *block)
> list_del(&block->link);
> }
>
> +static struct drm_buddy_block *
> +__get_buddy(struct drm_buddy_block *block)
> +{
> + struct drm_buddy_block *parent;
> +
> + parent = block->parent;
> + if (!parent)
> + return NULL;
> +
> + if (parent->left == block)
> + return parent->right;
> +
> + return parent->left;
> +}
> +
> +static unsigned int __drm_buddy_free(struct drm_buddy *mm,
> + struct drm_buddy_block *block,
> + bool force_merge)
> +{
> + struct drm_buddy_block *parent;
> + unsigned int order;
> +
> + while ((parent = block->parent)) {
> + struct drm_buddy_block *buddy;
> +
> + buddy = __get_buddy(block);
> +
> + if (!drm_buddy_block_is_free(buddy))
> + break;
> +
> + if (!force_merge) {
> + /**
Not really valid kernel-doc AFAIK. I think drop the extra *. Below also.
> + * Check the block and its buddy clear state and exit
> + * the loop if they both have the dissimilar state.
> + */
> + if (drm_buddy_block_is_clear(block) !=
> + drm_buddy_block_is_clear(buddy))
> + break;
> +
> + if (drm_buddy_block_is_clear(block))
> + mark_cleared(parent);
> + }
> +
> + list_del(&buddy->link);
> + if (force_merge && drm_buddy_block_is_clear(buddy))
> + mm->clear_avail -= drm_buddy_block_size(mm, buddy);
> +
> + drm_block_free(mm, block);
> + drm_block_free(mm, buddy);
> +
> + block = parent;
> + }
> +
> + order = drm_buddy_block_order(block);
> + mark_free(mm, block);
> +
> + return order;
> +}
> +
> +static int __force_merge(struct drm_buddy *mm,
> + unsigned int min_order)
> +{
> + unsigned int order;
> + int i;
> +
> + if (!min_order)
> + return -ENOMEM;
> +
> + if (min_order > mm->max_order)
> + return -EINVAL;
> +
> + for (i = min_order - 1; i >= 0; i--) {
> + struct drm_buddy_block *block, *prev;
> +
> + list_for_each_entry_safe_reverse(block, prev, &mm->free_list[i], link) {
> + struct drm_buddy_block *buddy;
> +
> + if (!block->parent)
> + continue;
> +
> + buddy = __get_buddy(block);
> + if (!drm_buddy_block_is_free(buddy))
> + continue;
> +
> + WARN_ON(drm_buddy_block_is_clear(block) ==
> + drm_buddy_block_is_clear(buddy));
> +
> + /**
> + * If the prev block is same as buddy, don't access the
> + * block in the next iteration as we would free the
> + * buddy block as part of the free function.
> + */
> + if (prev == buddy)
> + prev = list_prev_entry(prev, link);
> +
> + list_del(&block->link);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail -= drm_buddy_block_size(mm, block);
> +
> + order = __drm_buddy_free(mm, block, true);
> + if (order >= min_order)
> + return 0;
> + }
> + }
> +
> + return -ENOMEM;
> +}
> +
> /**
> * drm_buddy_init - init memory manager
> *
> @@ -137,7 +255,7 @@ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size)
> offset = 0;
> i = 0;
>
> - /*
> + /**
> * Split into power-of-two blocks, in case we are given a size that is
> * not itself a power-of-two.
> */
> @@ -186,11 +304,21 @@ EXPORT_SYMBOL(drm_buddy_init);
> */
> void drm_buddy_fini(struct drm_buddy *mm)
> {
> + u64 root_size, size;
> + unsigned int order;
> int i;
>
> + size = mm->size;
> +
> for (i = 0; i < mm->n_roots; ++i) {
> + order = ilog2(size) - ilog2(mm->chunk_size);
> + __force_merge(mm, order);
> +
> WARN_ON(!drm_buddy_block_is_free(mm->roots[i]));
> drm_block_free(mm, mm->roots[i]);
> +
> + root_size = mm->chunk_size << order;
> + size -= root_size;
> }
>
> WARN_ON(mm->avail != mm->size);
> @@ -223,26 +351,17 @@ static int split_block(struct drm_buddy *mm,
> mark_free(mm, block->left);
> mark_free(mm, block->right);
>
> + if (drm_buddy_block_is_clear(block)) {
> + mark_cleared(block->left);
> + mark_cleared(block->right);
> + clear_reset(block);
> + }
> +
> mark_split(block);
>
> return 0;
> }
>
> -static struct drm_buddy_block *
> -__get_buddy(struct drm_buddy_block *block)
> -{
> - struct drm_buddy_block *parent;
> -
> - parent = block->parent;
> - if (!parent)
> - return NULL;
> -
> - if (parent->left == block)
> - return parent->right;
> -
> - return parent->left;
> -}
> -
> /**
> * drm_get_buddy - get buddy address
> *
> @@ -260,30 +379,6 @@ drm_get_buddy(struct drm_buddy_block *block)
> }
> EXPORT_SYMBOL(drm_get_buddy);
>
> -static void __drm_buddy_free(struct drm_buddy *mm,
> - struct drm_buddy_block *block)
> -{
> - struct drm_buddy_block *parent;
> -
> - while ((parent = block->parent)) {
> - struct drm_buddy_block *buddy;
> -
> - buddy = __get_buddy(block);
> -
> - if (!drm_buddy_block_is_free(buddy))
> - break;
> -
> - list_del(&buddy->link);
> -
> - drm_block_free(mm, block);
> - drm_block_free(mm, buddy);
> -
> - block = parent;
> - }
> -
> - mark_free(mm, block);
> -}
> -
> /**
> * drm_buddy_free_block - free a block
> *
> @@ -295,26 +390,59 @@ void drm_buddy_free_block(struct drm_buddy *mm,
> {
> BUG_ON(!drm_buddy_block_is_allocated(block));
> mm->avail += drm_buddy_block_size(mm, block);
> - __drm_buddy_free(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail += drm_buddy_block_size(mm, block);
> +
> + __drm_buddy_free(mm, block, false);
> }
> EXPORT_SYMBOL(drm_buddy_free_block);
>
> -/**
> - * drm_buddy_free_list - free blocks
> - *
> - * @mm: DRM buddy manager
> - * @objects: input list head to free blocks
> - */
> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head *objects)
> +static void __drm_buddy_free_list(struct drm_buddy *mm,
> + struct list_head *objects,
> + bool mark_clear,
> + bool mark_dirty)
> {
> struct drm_buddy_block *block, *on;
>
> + WARN_ON(mark_dirty && mark_clear);
> +
> list_for_each_entry_safe(block, on, objects, link) {
> + if (mark_clear)
> + mark_cleared(block);
> + else if (mark_dirty)
> + clear_reset(block);
> drm_buddy_free_block(mm, block);
> cond_resched();
> }
> INIT_LIST_HEAD(objects);
> }
> +
> +static void drm_buddy_free_list_internal(struct drm_buddy *mm,
> + struct list_head *objects)
> +{
> + /**
> + * Don't touch the clear/dirty bit, since allocation is still internal
> + * at this point. For example we might have just failed part of the
> + * allocation.
> + */
> + __drm_buddy_free_list(mm, objects, false, false);
> +}
> +
> +/**
> + * drm_buddy_free_list - free blocks
> + *
> + * @mm: DRM buddy manager
> + * @objects: input list head to free blocks
> + * @flags: optional flags like DRM_BUDDY_CLEARED
> + */
> +void drm_buddy_free_list(struct drm_buddy *mm,
> + struct list_head *objects,
> + unsigned int flags)
> +{
> + bool mark_clear = flags & DRM_BUDDY_CLEARED;
> +
> + __drm_buddy_free_list(mm, objects, mark_clear, !mark_clear);
> +}
> EXPORT_SYMBOL(drm_buddy_free_list);
>
> static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
> @@ -327,10 +455,19 @@ static inline bool contains(u64 s1, u64 e1, u64 s2, u64 e2)
> return s1 <= s2 && e1 >= e2;
> }
>
> +static bool block_incompatible(struct drm_buddy_block *block, unsigned int flags)
> +{
> + bool needs_clear = flags & DRM_BUDDY_CLEAR_ALLOCATION;
> +
> + return needs_clear != drm_buddy_block_is_clear(block);
> +}
> +
> static struct drm_buddy_block *
> -alloc_range_bias(struct drm_buddy *mm,
> - u64 start, u64 end,
> - unsigned int order)
> +__alloc_range_bias(struct drm_buddy *mm,
> + u64 start, u64 end,
> + unsigned int order,
> + unsigned long flags,
> + bool fallback)
> {
> struct drm_buddy_block *block;
> struct drm_buddy_block *buddy;
> @@ -369,7 +506,10 @@ alloc_range_bias(struct drm_buddy *mm,
>
> if (contains(start, end, block_start, block_end) &&
> order == drm_buddy_block_order(block)) {
> - /*
> + if (!fallback && block_incompatible(block, flags))
> + continue;
> +
> + /**
> * Find the free block within the range.
> */
> if (drm_buddy_block_is_free(block))
> @@ -391,7 +531,7 @@ alloc_range_bias(struct drm_buddy *mm,
> return ERR_PTR(-ENOSPC);
>
> err_undo:
> - /*
> + /**
> * We really don't want to leave around a bunch of split blocks, since
> * bigger is better, so make sure we merge everything back before we
> * free the allocated blocks.
> @@ -400,30 +540,57 @@ alloc_range_bias(struct drm_buddy *mm,
> if (buddy &&
> (drm_buddy_block_is_free(block) &&
> drm_buddy_block_is_free(buddy)))
> - __drm_buddy_free(mm, block);
> + __drm_buddy_free(mm, block, false);
> return ERR_PTR(err);
> }
>
> static struct drm_buddy_block *
> -get_maxblock(struct drm_buddy *mm, unsigned int order)
> +__drm_buddy_alloc_range_bias(struct drm_buddy *mm,
> + u64 start, u64 end,
> + unsigned int order,
> + unsigned long flags)
> +{
> + struct drm_buddy_block *block;
> + bool fallback = false;
> +
> + block = __alloc_range_bias(mm, start, end, order,
> + flags, fallback);
> + if (IS_ERR(block) && mm->clear_avail)
> + return __alloc_range_bias(mm, start, end, order,
> + flags, !fallback);
> +
> + return block;
> +}
> +
> +static struct drm_buddy_block *
> +get_maxblock(struct drm_buddy *mm, unsigned int order,
> + unsigned long flags)
> {
> - struct drm_buddy_block *max_block = NULL, *node;
> + struct drm_buddy_block *max_block = NULL, *block = NULL;
> unsigned int i;
>
> for (i = order; i <= mm->max_order; ++i) {
> - if (!list_empty(&mm->free_list[i])) {
> - node = list_last_entry(&mm->free_list[i],
> - struct drm_buddy_block,
> - link);
> - if (!max_block) {
> - max_block = node;
> + struct drm_buddy_block *tmp_block;
> +
> + list_for_each_entry_reverse(tmp_block, &mm->free_list[i], link) {
> + if (block_incompatible(tmp_block, flags))
> continue;
> - }
>
> - if (drm_buddy_block_offset(node) >
> - drm_buddy_block_offset(max_block)) {
> - max_block = node;
> - }
> + block = tmp_block;
> + break;
> + }
> +
> + if (!block)
> + continue;
> +
> + if (!max_block) {
> + max_block = block;
> + continue;
> + }
> +
> + if (drm_buddy_block_offset(block) >
> + drm_buddy_block_offset(max_block)) {
> + max_block = block;
> }
> }
>
> @@ -440,11 +607,29 @@ alloc_from_freelist(struct drm_buddy *mm,
> int err;
>
> if (flags & DRM_BUDDY_TOPDOWN_ALLOCATION) {
> - block = get_maxblock(mm, order);
> + block = get_maxblock(mm, order, flags);
> if (block)
> /* Store the obtained block order */
> tmp = drm_buddy_block_order(block);
> } else {
> + for (tmp = order; tmp <= mm->max_order; ++tmp) {
> + struct drm_buddy_block *tmp_block;
> +
> + list_for_each_entry_reverse(tmp_block, &mm->free_list[tmp], link) {
> + if (block_incompatible(tmp_block, flags))
> + continue;
> +
> + block = tmp_block;
> + break;
> + }
> +
> + if (block)
> + break;
> + }
> + }
> +
> + if (!block) {
> + /* Fallback method */
> for (tmp = order; tmp <= mm->max_order; ++tmp) {
> if (!list_empty(&mm->free_list[tmp])) {
> block = list_last_entry(&mm->free_list[tmp],
> @@ -454,10 +639,10 @@ alloc_from_freelist(struct drm_buddy *mm,
> break;
> }
> }
> - }
>
> - if (!block)
> - return ERR_PTR(-ENOSPC);
> + if (!block)
> + return ERR_PTR(-ENOSPC);
> + }
>
> BUG_ON(!drm_buddy_block_is_free(block));
>
> @@ -473,7 +658,7 @@ alloc_from_freelist(struct drm_buddy *mm,
>
> err_undo:
> if (tmp != order)
> - __drm_buddy_free(mm, block);
> + __drm_buddy_free(mm, block, false);
> return ERR_PTR(err);
> }
>
> @@ -524,6 +709,8 @@ static int __alloc_range(struct drm_buddy *mm,
> mark_allocated(block);
> total_allocated += drm_buddy_block_size(mm, block);
> mm->avail -= drm_buddy_block_size(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail -= drm_buddy_block_size(mm, block);
> list_add_tail(&block->link, &allocated);
> continue;
> }
> @@ -548,7 +735,7 @@ static int __alloc_range(struct drm_buddy *mm,
> return 0;
>
> err_undo:
> - /*
> + /**
> * We really don't want to leave around a bunch of split blocks, since
> * bigger is better, so make sure we merge everything back before we
> * free the allocated blocks.
> @@ -557,14 +744,14 @@ static int __alloc_range(struct drm_buddy *mm,
> if (buddy &&
> (drm_buddy_block_is_free(block) &&
> drm_buddy_block_is_free(buddy)))
> - __drm_buddy_free(mm, block);
> + __drm_buddy_free(mm, block, false);
>
> err_free:
> if (err == -ENOSPC && total_allocated_on_err) {
> list_splice_tail(&allocated, blocks);
> *total_allocated_on_err = total_allocated;
> } else {
> - drm_buddy_free_list(mm, &allocated);
> + drm_buddy_free_list_internal(mm, &allocated);
> }
>
> return err;
> @@ -630,11 +817,11 @@ static int __alloc_contig_try_harder(struct drm_buddy *mm,
> list_splice(&blocks_lhs, blocks);
> return 0;
> } else if (err != -ENOSPC) {
> - drm_buddy_free_list(mm, blocks);
> + drm_buddy_free_list_internal(mm, blocks);
> return err;
> }
> /* Free blocks for the next iteration */
> - drm_buddy_free_list(mm, blocks);
> + drm_buddy_free_list_internal(mm, blocks);
> }
>
> return -ENOSPC;
> @@ -690,6 +877,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
> list_del(&block->link);
> mark_free(mm, block);
> mm->avail += drm_buddy_block_size(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail += drm_buddy_block_size(mm, block);
>
> /* Prevent recursively freeing this node */
> parent = block->parent;
> @@ -701,6 +890,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
> if (err) {
> mark_allocated(block);
> mm->avail -= drm_buddy_block_size(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail -= drm_buddy_block_size(mm, block);
> list_add(&block->link, blocks);
> }
>
> @@ -709,13 +900,28 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
> }
> EXPORT_SYMBOL(drm_buddy_block_trim);
>
> +static struct drm_buddy_block *
> +__drm_buddy_alloc_blocks(struct drm_buddy *mm,
> + u64 start, u64 end,
> + unsigned int order,
> + unsigned long flags)
> +{
> + if (flags & DRM_BUDDY_RANGE_ALLOCATION)
> + /* Allocate traversing within the range */
> + return __drm_buddy_alloc_range_bias(mm, start, end,
> + order, flags);
> + else
> + /* Allocate from freelist */
> + return alloc_from_freelist(mm, order, flags);
> +}
> +
> /**
> * drm_buddy_alloc_blocks - allocate power-of-two blocks
> *
> * @mm: DRM buddy manager to allocate from
> * @start: start of the allowed range for this block
> * @end: end of the allowed range for this block
> - * @size: size of the allocation
> + * @size: size of the allocation in bytes
> * @min_block_size: alignment of the allocation
> * @blocks: output list head to add allocated blocks
> * @flags: DRM_BUDDY_*_ALLOCATION flags
> @@ -761,8 +967,18 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
> return -EINVAL;
>
> /* Actual range allocation */
> - if (start + size == end)
> - return __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
> + if (start + size == end) {
> + err = __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
> + if (err) {
> + order = ilog2(size) - ilog2(mm->chunk_size);
> + if (mm->clear_avail && !__force_merge(mm, order))
> + return __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
That seems strange at a glance. With an actual range allocation like
with intitial fb or whatever it should just find all overlapping pages,
splitting down where needed on the edges. Not sure why force_merge would
factor in here?
> +
> + return err;
> + }
> +
> + return err;
> + }
>
> original_size = size;
> original_min_size = min_block_size;
> @@ -786,23 +1002,34 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
> BUG_ON(order < min_order);
>
> do {
> - if (flags & DRM_BUDDY_RANGE_ALLOCATION)
> - /* Allocate traversing within the range */
> - block = alloc_range_bias(mm, start, end, order);
> - else
> - /* Allocate from freelist */
> - block = alloc_from_freelist(mm, order, flags);
> -
> + block = __drm_buddy_alloc_blocks(mm, start,
> + end,
> + order,
> + flags);
> if (!IS_ERR(block))
> break;
>
> if (order-- == min_order) {
> + /**
> + * Try allocation through force merge method
> + */
> + if (mm->clear_avail && !__force_merge(mm, min_order)) {
> + block = __drm_buddy_alloc_blocks(mm, start,
> + end,
> + min_order,
> + flags);
> + if (!IS_ERR(block)) {
> + order = min_order;
> + break;
> + }
> + }
> +
> + /**
> + * Try contiguous block allocation through
> + * try harder method.
> + */
> if (flags & DRM_BUDDY_CONTIGUOUS_ALLOCATION &&
> !(flags & DRM_BUDDY_RANGE_ALLOCATION))
> - /*
> - * Try contiguous block allocation through
> - * try harder method
> - */
> return __alloc_contig_try_harder(mm,
> original_size,
> original_min_size,
> @@ -814,6 +1041,8 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>
> mark_allocated(block);
> mm->avail -= drm_buddy_block_size(mm, block);
> + if (drm_buddy_block_is_clear(block))
> + mm->clear_avail -= drm_buddy_block_size(mm, block);
> kmemleak_update_trace(block);
> list_add_tail(&block->link, &allocated);
>
> @@ -823,7 +1052,9 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
> break;
> } while (1);
>
> - /* Trim the allocated block to the required size */
> + /**
> + * Trim the allocated block to the required size
> + */
> if (original_size != size) {
> struct list_head *trim_list;
> LIST_HEAD(temp);
> @@ -852,7 +1083,7 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
> return 0;
>
> err_free:
> - drm_buddy_free_list(mm, &allocated);
> + drm_buddy_free_list_internal(mm, &allocated);
> return err;
> }
> EXPORT_SYMBOL(drm_buddy_alloc_blocks);
> @@ -885,8 +1116,8 @@ void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p)
> {
> int order;
>
> - drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free: %lluMiB\n",
> - mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20);
> + drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free: %lluMiB, clear_free: %lluMiB\n",
> + mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20, mm->clear_avail >> 20);
>
> for (order = mm->max_order; order >= 0; order--) {
> struct drm_buddy_block *block;
> diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
> index 0d735d5c2b35..942345548bc3 100644
> --- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
> +++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
> @@ -126,7 +126,7 @@ static int i915_ttm_buddy_man_alloc(struct ttm_resource_manager *man,
> return 0;
>
> err_free_blocks:
> - drm_buddy_free_list(mm, &bman_res->blocks);
> + drm_buddy_free_list(mm, &bman_res->blocks, 0);
> mutex_unlock(&bman->lock);
> err_free_res:
> ttm_resource_fini(man, &bman_res->base);
> @@ -141,7 +141,7 @@ static void i915_ttm_buddy_man_free(struct ttm_resource_manager *man,
> struct i915_ttm_buddy_manager *bman = to_buddy_manager(man);
>
> mutex_lock(&bman->lock);
> - drm_buddy_free_list(&bman->mm, &bman_res->blocks);
> + drm_buddy_free_list(&bman->mm, &bman_res->blocks, 0);
> bman->visible_avail += bman_res->used_visible_size;
> mutex_unlock(&bman->lock);
>
> @@ -345,7 +345,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type)
> ttm_set_driver_manager(bdev, type, NULL);
>
> mutex_lock(&bman->lock);
> - drm_buddy_free_list(mm, &bman->reserved);
> + drm_buddy_free_list(mm, &bman->reserved, 0);
> drm_buddy_fini(mm);
> bman->visible_avail += bman->visible_reserved;
> WARN_ON_ONCE(bman->visible_avail != bman->visible_size);
> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
> index 2f32fb2f12e7..454ad9952f56 100644
> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> @@ -64,7 +64,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> "buddy_alloc didn't error size=%u\n", 3 * ps);
>
> - drm_buddy_free_list(&mm, &middle);
> + drm_buddy_free_list(&mm, &middle, 0);
> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> 3 * ps, ps, &allocated,
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> @@ -74,7 +74,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> "buddy_alloc didn't error size=%u\n", 2 * ps);
>
> - drm_buddy_free_list(&mm, &right);
> + drm_buddy_free_list(&mm, &right, 0);
> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> 3 * ps, ps, &allocated,
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> @@ -89,7 +89,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> "buddy_alloc hit an error size=%u\n", 2 * ps);
>
> - drm_buddy_free_list(&mm, &left);
> + drm_buddy_free_list(&mm, &left, 0);
> KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0, mm_size,
> 3 * ps, ps, &allocated,
> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
> @@ -101,7 +101,7 @@ static void drm_test_buddy_alloc_contiguous(struct kunit *test)
>
> KUNIT_ASSERT_EQ(test, total, ps * 2 + ps * 3);
>
> - drm_buddy_free_list(&mm, &allocated);
> + drm_buddy_free_list(&mm, &allocated, 0);
> drm_buddy_fini(&mm);
> }
>
> @@ -170,7 +170,7 @@ static void drm_test_buddy_alloc_pathological(struct kunit *test)
> top, max_order);
> }
>
> - drm_buddy_free_list(&mm, &holes);
> + drm_buddy_free_list(&mm, &holes, 0);
>
> /* Nothing larger than blocks of chunk_size now available */
> for (order = 1; order <= max_order; order++) {
> @@ -182,7 +182,7 @@ static void drm_test_buddy_alloc_pathological(struct kunit *test)
> }
>
> list_splice_tail(&holes, &blocks);
> - drm_buddy_free_list(&mm, &blocks);
> + drm_buddy_free_list(&mm, &blocks, 0);
> drm_buddy_fini(&mm);
> }
>
> @@ -277,7 +277,7 @@ static void drm_test_buddy_alloc_pessimistic(struct kunit *test)
>
> list_del(&block->link);
> drm_buddy_free_block(&mm, block);
> - drm_buddy_free_list(&mm, &blocks);
> + drm_buddy_free_list(&mm, &blocks, 0);
> drm_buddy_fini(&mm);
> }
>
> @@ -323,7 +323,7 @@ static void drm_test_buddy_alloc_optimistic(struct kunit *test)
> size, size, &tmp, flags),
> "buddy_alloc unexpectedly succeeded, it should be full!");
>
> - drm_buddy_free_list(&mm, &blocks);
> + drm_buddy_free_list(&mm, &blocks, 0);
> drm_buddy_fini(&mm);
> }
>
> @@ -358,7 +358,7 @@ static void drm_test_buddy_alloc_limit(struct kunit *test)
> drm_buddy_block_size(&mm, block),
> BIT_ULL(mm.max_order) * PAGE_SIZE);
>
> - drm_buddy_free_list(&mm, &allocated);
> + drm_buddy_free_list(&mm, &allocated, 0);
> drm_buddy_fini(&mm);
> }
>
> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> index 115ec745e502..1ad678b62c4a 100644
> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> @@ -196,7 +196,7 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
> return 0;
>
> error_free_blocks:
> - drm_buddy_free_list(mm, &vres->blocks);
> + drm_buddy_free_list(mm, &vres->blocks, 0);
> mutex_unlock(&mgr->lock);
> error_fini:
> ttm_resource_fini(man, &vres->base);
> @@ -214,7 +214,7 @@ static void xe_ttm_vram_mgr_del(struct ttm_resource_manager *man,
> struct drm_buddy *mm = &mgr->mm;
>
> mutex_lock(&mgr->lock);
> - drm_buddy_free_list(mm, &vres->blocks);
> + drm_buddy_free_list(mm, &vres->blocks, 0);
> mgr->visible_avail += vres->used_visible_size;
> mutex_unlock(&mgr->lock);
>
> diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h
> index a5b39fc01003..82570f77e817 100644
> --- a/include/drm/drm_buddy.h
> +++ b/include/drm/drm_buddy.h
> @@ -25,6 +25,8 @@
> #define DRM_BUDDY_RANGE_ALLOCATION BIT(0)
> #define DRM_BUDDY_TOPDOWN_ALLOCATION BIT(1)
> #define DRM_BUDDY_CONTIGUOUS_ALLOCATION BIT(2)
> +#define DRM_BUDDY_CLEAR_ALLOCATION BIT(3)
> +#define DRM_BUDDY_CLEARED BIT(4)
>
> struct drm_buddy_block {
> #define DRM_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
> @@ -32,8 +34,9 @@ struct drm_buddy_block {
> #define DRM_BUDDY_ALLOCATED (1 << 10)
> #define DRM_BUDDY_FREE (2 << 10)
> #define DRM_BUDDY_SPLIT (3 << 10)
> +#define DRM_BUDDY_HEADER_CLEAR GENMASK_ULL(9, 9)
> /* Free to be used, if needed in the future */
> -#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
> +#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(8, 6)
> #define DRM_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
> u64 header;
>
> @@ -86,6 +89,7 @@ struct drm_buddy {
> u64 chunk_size;
> u64 size;
> u64 avail;
> + u64 clear_avail;
> };
>
> static inline u64
> @@ -112,6 +116,12 @@ drm_buddy_block_is_allocated(struct drm_buddy_block *block)
> return drm_buddy_block_state(block) == DRM_BUDDY_ALLOCATED;
> }
>
> +static inline bool
> +drm_buddy_block_is_clear(struct drm_buddy_block *block)
> +{
> + return block->header & DRM_BUDDY_HEADER_CLEAR;
> +}
> +
> static inline bool
> drm_buddy_block_is_free(struct drm_buddy_block *block)
> {
> @@ -150,7 +160,9 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>
> void drm_buddy_free_block(struct drm_buddy *mm, struct drm_buddy_block *block);
>
> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head *objects);
> +void drm_buddy_free_list(struct drm_buddy *mm,
> + struct list_head *objects,
> + unsigned int flags);
>
> void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p);
> void drm_buddy_block_print(struct drm_buddy *mm,
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-26 14:53 ` Alex Deucher
@ 2024-03-27 8:23 ` Paneer Selvam, Arunpravin
2024-04-02 8:17 ` Christian König
1 sibling, 0 replies; 21+ messages in thread
From: Paneer Selvam, Arunpravin @ 2024-03-27 8:23 UTC (permalink / raw)
To: Alex Deucher
Cc: dri-devel, amd-gfx, christian.koenig, alexander.deucher,
matthew.auld, mario.limonciello, felix.kuehling
Hi Alex,
On 3/26/2024 8:23 PM, Alex Deucher wrote:
> On Tue, Mar 26, 2024 at 10:01 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>> On Tue, Mar 26, 2024 at 9:59 AM Paneer Selvam, Arunpravin
>> <arunpravin.paneerselvam@amd.com> wrote:
>>> Hi Alex,
>>>
>>> On 3/26/2024 7:08 PM, Alex Deucher wrote:
>>>> On Mon, Mar 18, 2024 at 5:47 PM Arunpravin Paneer Selvam
>>>> <Arunpravin.PaneerSelvam@amd.com> wrote:
>>>>> Add clear page support in vram memory region.
>>>>>
>>>>> v1(Christian):
>>>>> - Dont handle clear page as TTM flag since when moving the BO back
>>>>> in from GTT again we don't need that.
>>>>> - Make a specialized version of amdgpu_fill_buffer() which only
>>>>> clears the VRAM areas which are not already cleared
>>>>> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
>>>>> amdgpu_object.c
>>>>>
>>>>> v2:
>>>>> - Modify the function name amdgpu_ttm_* (Alex)
>>>>> - Drop the delayed parameter (Christian)
>>>>> - handle amdgpu_res_cleared(&cursor) just above the size
>>>>> calculation (Christian)
>>>>> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
>>>>> in the free path to properly wait for fences etc.. (Christian)
>>>>>
>>>>> v3(Christian):
>>>>> - Remove buffer clear code in VRAM manager instead change the
>>>>> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
>>>>> the DRM_BUDDY_CLEARED flag.
>>>>> - Remove ! from amdgpu_res_cleared(&cursor) check.
>>>>>
>>>>> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
>>>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>>>> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
>>>>> ---
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
>>>>> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
>>>>> 6 files changed, 111 insertions(+), 13 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>>> index 8bc79924d171..c92d92b28a57 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>>> @@ -39,6 +39,7 @@
>>>>> #include "amdgpu.h"
>>>>> #include "amdgpu_trace.h"
>>>>> #include "amdgpu_amdkfd.h"
>>>>> +#include "amdgpu_vram_mgr.h"
>>>>>
>>>>> /**
>>>>> * DOC: amdgpu_object
>>>>> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>>>> if (!amdgpu_bo_support_uswc(bo->flags))
>>>>> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
>>>>>
>>>>> - if (adev->ras_enabled)
>>>>> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>>>> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>>>>
>>>>> bo->tbo.bdev = &adev->mman.bdev;
>>>>> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
>>>>> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>>>>
>>>>> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
>>>>> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
>>>>> - struct dma_fence *fence;
>>>>> + struct dma_fence *fence = NULL;
>>>>>
>>>>> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence, true);
>>>>> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
>>>>> if (unlikely(r))
>>>>> goto fail_unreserve;
>>>>>
>>>>> - dma_resv_add_fence(bo->tbo.base.resv, fence,
>>>>> - DMA_RESV_USAGE_KERNEL);
>>>>> - dma_fence_put(fence);
>>>>> + if (fence) {
>>>>> + dma_resv_add_fence(bo->tbo.base.resv, fence,
>>>>> + DMA_RESV_USAGE_KERNEL);
>>>>> + dma_fence_put(fence);
>>>>> + }
>>>>> }
>>>>> if (!bp->resv)
>>>>> amdgpu_bo_unreserve(bo);
>>>>> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo)
>>>>> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
>>>>> return;
>>>>>
>>>>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv, &fence, true);
>>>>> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
>>>>> if (!WARN_ON(r)) {
>>>>> + struct amdgpu_vram_mgr_resource *vres;
>>>>> +
>>>>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>>>>> + vres->flags |= DRM_BUDDY_CLEARED;
>>>>> amdgpu_bo_fence(abo, fence, false);
>>>>> dma_fence_put(fence);
>>>>> }
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>>> index 381101d2bf05..50fcd86e1033 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>>> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct amdgpu_res_cursor *cur, uint64_t size)
>>>>> }
>>>>> }
>>>>>
>>>>> +/**
>>>>> + * amdgpu_res_cleared - check if blocks are cleared
>>>>> + *
>>>>> + * @cur: the cursor to extract the block
>>>>> + *
>>>>> + * Check if the @cur block is cleared
>>>>> + */
>>>>> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
>>>>> +{
>>>>> + struct drm_buddy_block *block;
>>>>> +
>>>>> + switch (cur->mem_type) {
>>>>> + case TTM_PL_VRAM:
>>>>> + block = cur->node;
>>>>> +
>>>>> + if (!amdgpu_vram_mgr_is_cleared(block))
>>>>> + return false;
>>>>> + break;
>>>>> + default:
>>>>> + return false;
>>>>> + }
>>>>> +
>>>>> + return true;
>>>>> +}
>>>>> +
>>>>> #endif
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>>> index 8722beba494e..bcbffe909b47 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>>> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
>>>>> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
>>>>> struct dma_fence *wipe_fence = NULL;
>>>>>
>>>>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
>>>>> - false);
>>>>> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
>>>>> + false);
>>>>> if (r) {
>>>>> goto error;
>>>>> } else if (wipe_fence) {
>>>>> + struct amdgpu_vram_mgr_resource *vres;
>>>>> +
>>>>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>>>>> + vres->flags |= DRM_BUDDY_CLEARED;
>>>>> dma_fence_put(fence);
>>>>> fence = wipe_fence;
>>>>> }
>>>>> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_ring *ring, uint32_t src_data,
>>>>> return 0;
>>>>> }
>>>>>
>>>>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>>>>> + struct dma_resv *resv,
>>>>> + struct dma_fence **fence)
>>>>> +{
>>>>> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>>>>> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
>>>>> + struct amdgpu_res_cursor cursor;
>>>>> + struct dma_fence *f = NULL;
>>>>> + u64 addr;
>>>>> + int r;
>>>>> +
>>>>> + if (!adev->mman.buffer_funcs_enabled)
>>>>> + return -EINVAL;
>>>>> +
>>>>> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
>>>>> +
>>>>> + mutex_lock(&adev->mman.gtt_window_lock);
>>>>> + while (cursor.remaining) {
>>>>> + struct dma_fence *next = NULL;
>>>>> + u64 size;
>>>>> +
>>>>> + if (amdgpu_res_cleared(&cursor)) {
>>>>> + amdgpu_res_next(&cursor, cursor.size);
>>>>> + continue;
>>>>> + }
>>>>> +
>>>>> + /* Never clear more than 256MiB at once to avoid timeouts */
>>>>> + size = min(cursor.size, 256ULL << 20);
>>>>> +
>>>>> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
>>>>> + 1, ring, false, &size, &addr);
>>>>> + if (r)
>>>>> + goto err;
>>>>> +
>>>>> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
>>>>> + &next, true, true);
>>>>> + if (r)
>>>>> + goto err;
>>>>> +
>>>>> + dma_fence_put(f);
>>>>> + f = next;
>>>>> +
>>>>> + amdgpu_res_next(&cursor, size);
>>>>> + }
>>>>> +err:
>>>>> + mutex_unlock(&adev->mman.gtt_window_lock);
>>>>> + if (fence)
>>>>> + *fence = dma_fence_get(f);
>>>>> + dma_fence_put(f);
>>>>> +
>>>>> + return r;
>>>>> +}
>>>>> +
>>>>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>>>>> uint32_t src_data,
>>>>> struct dma_resv *resv,
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>>> index 65ec82141a8e..b404d89d52e5 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>>> @@ -38,8 +38,6 @@
>>>>> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
>>>>> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
>>>>>
>>>>> -#define AMDGPU_POISON 0xd0bed0be
>>>>> -
>>>>> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
>>>>> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
>>>>>
>>>>> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
>>>>> uint64_t size, bool tmz,
>>>>> struct dma_resv *resv,
>>>>> struct dma_fence **f);
>>>>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>>>>> + struct dma_resv *resv,
>>>>> + struct dma_fence **fence);
>>>>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>>>>> uint32_t src_data,
>>>>> struct dma_resv *resv,
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> index c0c851409241..e494f5bf136a 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
>>>>> {
>>>>> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
>>>>> struct amdgpu_device *adev = to_amdgpu_device(mgr);
>>>>> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
>>>>> u64 vis_usage = 0, max_bytes, min_block_size;
>>>>> struct amdgpu_vram_mgr_resource *vres;
>>>>> u64 size, remaining_size, lpfn, fpfn;
>>>>> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
>>>>> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
>>>>> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>>>>>
>>>>> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
>>>>> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
>>>> Is there any reason to not always do this?
>>> Here we are trying to keep a pool of cleared memory which are cleared on
>>> free path and that can given
>>> to the application which requires a zeroed memory. I think here if we
>>> set always clear the memory,
>>> we would go over the limit of cleared memory in that particular instance
>>> and the application should wait until
>>> the hardware clears the memory and this might impact the overall
>>> performance.
>> I'd like to have the driver always deliver cleared memory.
> Actually, I think we can just always set the flag in the allocation IOCTLs.
Sure, we can set the flag in the allocation IOCTLs.
Thanks,
Arun.
>
> Alex
>
>> Alex
>>
>>> Thanks,
>>> Arun.
>>>> Alex
>>>>
>>>>
>>>>> +
>>>>> if (fpfn || lpfn != mgr->mm.size)
>>>>> /* Allocate blocks in desired range */
>>>>> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
>>>>> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
>>>>>
>>>>> amdgpu_vram_mgr_do_reserve(man);
>>>>>
>>>>> - drm_buddy_free_list(mm, &vres->blocks, 0);
>>>>> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
>>>>> mutex_unlock(&mgr->lock);
>>>>>
>>>>> atomic64_sub(vis_usage, &mgr->vis_usage);
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>>> index 0e04e42cf809..8478522d7366 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>>> @@ -53,6 +53,11 @@ static inline u64 amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
>>>>> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
>>>>> }
>>>>>
>>>>> +static inline bool amdgpu_vram_mgr_is_cleared(struct drm_buddy_block *block)
>>>>> +{
>>>>> + return drm_buddy_block_is_clear(block);
>>>>> +}
>>>>> +
>>>>> static inline struct amdgpu_vram_mgr_resource *
>>>>> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
>>>>> {
>>>>> --
>>>>> 2.25.1
>>>>>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature
2024-03-26 18:09 ` Matthew Auld
@ 2024-03-28 16:07 ` Paneer Selvam, Arunpravin
2024-03-28 16:48 ` Matthew Auld
0 siblings, 1 reply; 21+ messages in thread
From: Paneer Selvam, Arunpravin @ 2024-03-28 16:07 UTC (permalink / raw)
To: Matthew Auld, dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, mario.limonciello, felix.kuehling
Hi Matthew,
On 3/26/2024 11:39 PM, Matthew Auld wrote:
> On 18/03/2024 21:40, Arunpravin Paneer Selvam wrote:
>> - Add tracking clear page feature.
>>
>> - Driver should enable the DRM_BUDDY_CLEARED flag if it
>> successfully clears the blocks in the free path. On the otherhand,
>> DRM buddy marks each block as cleared.
>>
>> - Track the available cleared pages size
>>
>> - If driver requests cleared memory we prefer cleared memory
>> but fallback to uncleared if we can't find the cleared blocks.
>> when driver requests uncleared memory we try to use uncleared but
>> fallback to cleared memory if necessary.
>>
>> - When a block gets freed we clear it and mark the freed block as
>> cleared,
>> when there are buddies which are cleared as well we can merge them.
>> Otherwise, we prefer to keep the blocks as separated.
>>
>> - Add a function to support defragmentation.
>>
>> v1:
>> - Depends on the flag check DRM_BUDDY_CLEARED, enable the block as
>> cleared. Else, reset the clear flag for each block in the
>> list(Christian)
>> - For merging the 2 cleared blocks compare as below,
>> drm_buddy_is_clear(block) != drm_buddy_is_clear(buddy)(Christian)
>> - Defragment the memory beginning from min_order
>> till the required memory space is available.
>>
>> v2: (Matthew)
>> - Add a wrapper drm_buddy_free_list_internal for the freeing of
>> blocks
>> operation within drm buddy.
>> - Write a macro block_incompatible() to allocate the required blocks.
>> - Update the xe driver for the drm_buddy_free_list change in
>> arguments.
>> - add a warning if the two blocks are incompatible on
>> defragmentation
>> - call full defragmentation in the fini() function
>> - place a condition to test if min_order is equal to 0
>> - replace the list with safe_reverse() variant as we might
>> remove the block from the list.
>>
>> v3:
>> - fix Gitlab user reported lockup issue.
>> - Keep DRM_BUDDY_HEADER_CLEAR define sorted(Matthew)
>> - modify to pass the root order instead max_order in fini()
>> function(Matthew)
>> - change bool 1 to true(Matthew)
>> - add check if min_block_size is power of 2(Matthew)
>> - modify the min_block_size datatype to u64(Matthew)
>>
>> v4:
>> - rename the function drm_buddy_defrag with __force_merge.
>> - Include __force_merge directly in drm buddy file and remove
>> the defrag use in amdgpu driver.
>> - Remove list_empty() check(Matthew)
>> - Remove unnecessary space, headers and placement of new
>> variables(Matthew)
>> - Add a unit test case(Matthew)
>>
>> Signed-off-by: Arunpravin Paneer Selvam
>> <Arunpravin.PaneerSelvam@amd.com>
>> Signed-off-by: Matthew Auld <matthew.auld@intel.com>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Matthew Auld <matthew.auld@intel.com>
>> ---
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>> drivers/gpu/drm/drm_buddy.c | 427 ++++++++++++++----
>> drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 6 +-
>> drivers/gpu/drm/tests/drm_buddy_test.c | 18 +-
>> drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 4 +-
>> include/drm/drm_buddy.h | 16 +-
>> 6 files changed, 360 insertions(+), 117 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> index 8db880244324..c0c851409241 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>> @@ -571,7 +571,7 @@ static int amdgpu_vram_mgr_new(struct
>> ttm_resource_manager *man,
>> return 0;
>> error_free_blocks:
>> - drm_buddy_free_list(mm, &vres->blocks);
>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>> mutex_unlock(&mgr->lock);
>> error_fini:
>> ttm_resource_fini(man, &vres->base);
>> @@ -604,7 +604,7 @@ static void amdgpu_vram_mgr_del(struct
>> ttm_resource_manager *man,
>> amdgpu_vram_mgr_do_reserve(man);
>> - drm_buddy_free_list(mm, &vres->blocks);
>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>> mutex_unlock(&mgr->lock);
>> atomic64_sub(vis_usage, &mgr->vis_usage);
>> @@ -912,7 +912,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device
>> *adev)
>> kfree(rsv);
>> list_for_each_entry_safe(rsv, temp, &mgr->reserved_pages,
>> blocks) {
>> - drm_buddy_free_list(&mgr->mm, &rsv->allocated);
>> + drm_buddy_free_list(&mgr->mm, &rsv->allocated, 0);
>> kfree(rsv);
>> }
>> if (!adev->gmc.is_app_apu)
>> diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c
>> index c4222b886db7..625a30a6b855 100644
>> --- a/drivers/gpu/drm/drm_buddy.c
>> +++ b/drivers/gpu/drm/drm_buddy.c
>> @@ -38,8 +38,8 @@ static void drm_block_free(struct drm_buddy *mm,
>> kmem_cache_free(slab_blocks, block);
>> }
>> -static void list_insert_sorted(struct drm_buddy *mm,
>> - struct drm_buddy_block *block)
>> +static void list_insert(struct drm_buddy *mm,
>> + struct drm_buddy_block *block)
>> {
>> struct drm_buddy_block *node;
>> struct list_head *head;
>> @@ -57,6 +57,16 @@ static void list_insert_sorted(struct drm_buddy *mm,
>> __list_add(&block->link, node->link.prev, &node->link);
>> }
>> +static void clear_reset(struct drm_buddy_block *block)
>> +{
>> + block->header &= ~DRM_BUDDY_HEADER_CLEAR;
>> +}
>> +
>> +static void mark_cleared(struct drm_buddy_block *block)
>> +{
>> + block->header |= DRM_BUDDY_HEADER_CLEAR;
>> +}
>> +
>> static void mark_allocated(struct drm_buddy_block *block)
>> {
>> block->header &= ~DRM_BUDDY_HEADER_STATE;
>> @@ -71,7 +81,7 @@ static void mark_free(struct drm_buddy *mm,
>> block->header &= ~DRM_BUDDY_HEADER_STATE;
>> block->header |= DRM_BUDDY_FREE;
>> - list_insert_sorted(mm, block);
>> + list_insert(mm, block);
>> }
>> static void mark_split(struct drm_buddy_block *block)
>> @@ -82,6 +92,114 @@ static void mark_split(struct drm_buddy_block
>> *block)
>> list_del(&block->link);
>> }
>> +static struct drm_buddy_block *
>> +__get_buddy(struct drm_buddy_block *block)
>> +{
>> + struct drm_buddy_block *parent;
>> +
>> + parent = block->parent;
>> + if (!parent)
>> + return NULL;
>> +
>> + if (parent->left == block)
>> + return parent->right;
>> +
>> + return parent->left;
>> +}
>> +
>> +static unsigned int __drm_buddy_free(struct drm_buddy *mm,
>> + struct drm_buddy_block *block,
>> + bool force_merge)
>> +{
>> + struct drm_buddy_block *parent;
>> + unsigned int order;
>> +
>> + while ((parent = block->parent)) {
>> + struct drm_buddy_block *buddy;
>> +
>> + buddy = __get_buddy(block);
>> +
>> + if (!drm_buddy_block_is_free(buddy))
>> + break;
>> +
>> + if (!force_merge) {
>> + /**
>
> Not really valid kernel-doc AFAIK. I think drop the extra *. Below also.
>
>> + * Check the block and its buddy clear state and exit
>> + * the loop if they both have the dissimilar state.
>> + */
>> + if (drm_buddy_block_is_clear(block) !=
>> + drm_buddy_block_is_clear(buddy))
>> + break;
>> +
>> + if (drm_buddy_block_is_clear(block))
>> + mark_cleared(parent);
>> + }
>> +
>> + list_del(&buddy->link);
>> + if (force_merge && drm_buddy_block_is_clear(buddy))
>> + mm->clear_avail -= drm_buddy_block_size(mm, buddy);
>> +
>> + drm_block_free(mm, block);
>> + drm_block_free(mm, buddy);
>> +
>> + block = parent;
>> + }
>> +
>> + order = drm_buddy_block_order(block);
>> + mark_free(mm, block);
>> +
>> + return order;
>> +}
>> +
>> +static int __force_merge(struct drm_buddy *mm,
>> + unsigned int min_order)
>> +{
>> + unsigned int order;
>> + int i;
>> +
>> + if (!min_order)
>> + return -ENOMEM;
>> +
>> + if (min_order > mm->max_order)
>> + return -EINVAL;
>> +
>> + for (i = min_order - 1; i >= 0; i--) {
>> + struct drm_buddy_block *block, *prev;
>> +
>> + list_for_each_entry_safe_reverse(block, prev,
>> &mm->free_list[i], link) {
>> + struct drm_buddy_block *buddy;
>> +
>> + if (!block->parent)
>> + continue;
>> +
>> + buddy = __get_buddy(block);
>> + if (!drm_buddy_block_is_free(buddy))
>> + continue;
>> +
>> + WARN_ON(drm_buddy_block_is_clear(block) ==
>> + drm_buddy_block_is_clear(buddy));
>> +
>> + /**
>> + * If the prev block is same as buddy, don't access the
>> + * block in the next iteration as we would free the
>> + * buddy block as part of the free function.
>> + */
>> + if (prev == buddy)
>> + prev = list_prev_entry(prev, link);
>> +
>> + list_del(&block->link);
>> + if (drm_buddy_block_is_clear(block))
>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>> +
>> + order = __drm_buddy_free(mm, block, true);
>> + if (order >= min_order)
>> + return 0;
>> + }
>> + }
>> +
>> + return -ENOMEM;
>> +}
>> +
>> /**
>> * drm_buddy_init - init memory manager
>> *
>> @@ -137,7 +255,7 @@ int drm_buddy_init(struct drm_buddy *mm, u64
>> size, u64 chunk_size)
>> offset = 0;
>> i = 0;
>> - /*
>> + /**
>> * Split into power-of-two blocks, in case we are given a size
>> that is
>> * not itself a power-of-two.
>> */
>> @@ -186,11 +304,21 @@ EXPORT_SYMBOL(drm_buddy_init);
>> */
>> void drm_buddy_fini(struct drm_buddy *mm)
>> {
>> + u64 root_size, size;
>> + unsigned int order;
>> int i;
>> + size = mm->size;
>> +
>> for (i = 0; i < mm->n_roots; ++i) {
>> + order = ilog2(size) - ilog2(mm->chunk_size);
>> + __force_merge(mm, order);
>> +
>> WARN_ON(!drm_buddy_block_is_free(mm->roots[i]));
>> drm_block_free(mm, mm->roots[i]);
>> +
>> + root_size = mm->chunk_size << order;
>> + size -= root_size;
>> }
>> WARN_ON(mm->avail != mm->size);
>> @@ -223,26 +351,17 @@ static int split_block(struct drm_buddy *mm,
>> mark_free(mm, block->left);
>> mark_free(mm, block->right);
>> + if (drm_buddy_block_is_clear(block)) {
>> + mark_cleared(block->left);
>> + mark_cleared(block->right);
>> + clear_reset(block);
>> + }
>> +
>> mark_split(block);
>> return 0;
>> }
>> -static struct drm_buddy_block *
>> -__get_buddy(struct drm_buddy_block *block)
>> -{
>> - struct drm_buddy_block *parent;
>> -
>> - parent = block->parent;
>> - if (!parent)
>> - return NULL;
>> -
>> - if (parent->left == block)
>> - return parent->right;
>> -
>> - return parent->left;
>> -}
>> -
>> /**
>> * drm_get_buddy - get buddy address
>> *
>> @@ -260,30 +379,6 @@ drm_get_buddy(struct drm_buddy_block *block)
>> }
>> EXPORT_SYMBOL(drm_get_buddy);
>> -static void __drm_buddy_free(struct drm_buddy *mm,
>> - struct drm_buddy_block *block)
>> -{
>> - struct drm_buddy_block *parent;
>> -
>> - while ((parent = block->parent)) {
>> - struct drm_buddy_block *buddy;
>> -
>> - buddy = __get_buddy(block);
>> -
>> - if (!drm_buddy_block_is_free(buddy))
>> - break;
>> -
>> - list_del(&buddy->link);
>> -
>> - drm_block_free(mm, block);
>> - drm_block_free(mm, buddy);
>> -
>> - block = parent;
>> - }
>> -
>> - mark_free(mm, block);
>> -}
>> -
>> /**
>> * drm_buddy_free_block - free a block
>> *
>> @@ -295,26 +390,59 @@ void drm_buddy_free_block(struct drm_buddy *mm,
>> {
>> BUG_ON(!drm_buddy_block_is_allocated(block));
>> mm->avail += drm_buddy_block_size(mm, block);
>> - __drm_buddy_free(mm, block);
>> + if (drm_buddy_block_is_clear(block))
>> + mm->clear_avail += drm_buddy_block_size(mm, block);
>> +
>> + __drm_buddy_free(mm, block, false);
>> }
>> EXPORT_SYMBOL(drm_buddy_free_block);
>> -/**
>> - * drm_buddy_free_list - free blocks
>> - *
>> - * @mm: DRM buddy manager
>> - * @objects: input list head to free blocks
>> - */
>> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head
>> *objects)
>> +static void __drm_buddy_free_list(struct drm_buddy *mm,
>> + struct list_head *objects,
>> + bool mark_clear,
>> + bool mark_dirty)
>> {
>> struct drm_buddy_block *block, *on;
>> + WARN_ON(mark_dirty && mark_clear);
>> +
>> list_for_each_entry_safe(block, on, objects, link) {
>> + if (mark_clear)
>> + mark_cleared(block);
>> + else if (mark_dirty)
>> + clear_reset(block);
>> drm_buddy_free_block(mm, block);
>> cond_resched();
>> }
>> INIT_LIST_HEAD(objects);
>> }
>> +
>> +static void drm_buddy_free_list_internal(struct drm_buddy *mm,
>> + struct list_head *objects)
>> +{
>> + /**
>> + * Don't touch the clear/dirty bit, since allocation is still
>> internal
>> + * at this point. For example we might have just failed part of the
>> + * allocation.
>> + */
>> + __drm_buddy_free_list(mm, objects, false, false);
>> +}
>> +
>> +/**
>> + * drm_buddy_free_list - free blocks
>> + *
>> + * @mm: DRM buddy manager
>> + * @objects: input list head to free blocks
>> + * @flags: optional flags like DRM_BUDDY_CLEARED
>> + */
>> +void drm_buddy_free_list(struct drm_buddy *mm,
>> + struct list_head *objects,
>> + unsigned int flags)
>> +{
>> + bool mark_clear = flags & DRM_BUDDY_CLEARED;
>> +
>> + __drm_buddy_free_list(mm, objects, mark_clear, !mark_clear);
>> +}
>> EXPORT_SYMBOL(drm_buddy_free_list);
>> static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
>> @@ -327,10 +455,19 @@ static inline bool contains(u64 s1, u64 e1, u64
>> s2, u64 e2)
>> return s1 <= s2 && e1 >= e2;
>> }
>> +static bool block_incompatible(struct drm_buddy_block *block,
>> unsigned int flags)
>> +{
>> + bool needs_clear = flags & DRM_BUDDY_CLEAR_ALLOCATION;
>> +
>> + return needs_clear != drm_buddy_block_is_clear(block);
>> +}
>> +
>> static struct drm_buddy_block *
>> -alloc_range_bias(struct drm_buddy *mm,
>> - u64 start, u64 end,
>> - unsigned int order)
>> +__alloc_range_bias(struct drm_buddy *mm,
>> + u64 start, u64 end,
>> + unsigned int order,
>> + unsigned long flags,
>> + bool fallback)
>> {
>> struct drm_buddy_block *block;
>> struct drm_buddy_block *buddy;
>> @@ -369,7 +506,10 @@ alloc_range_bias(struct drm_buddy *mm,
>> if (contains(start, end, block_start, block_end) &&
>> order == drm_buddy_block_order(block)) {
>> - /*
>> + if (!fallback && block_incompatible(block, flags))
>> + continue;
>> +
>> + /**
>> * Find the free block within the range.
>> */
>> if (drm_buddy_block_is_free(block))
>> @@ -391,7 +531,7 @@ alloc_range_bias(struct drm_buddy *mm,
>> return ERR_PTR(-ENOSPC);
>> err_undo:
>> - /*
>> + /**
>> * We really don't want to leave around a bunch of split
>> blocks, since
>> * bigger is better, so make sure we merge everything back
>> before we
>> * free the allocated blocks.
>> @@ -400,30 +540,57 @@ alloc_range_bias(struct drm_buddy *mm,
>> if (buddy &&
>> (drm_buddy_block_is_free(block) &&
>> drm_buddy_block_is_free(buddy)))
>> - __drm_buddy_free(mm, block);
>> + __drm_buddy_free(mm, block, false);
>> return ERR_PTR(err);
>> }
>> static struct drm_buddy_block *
>> -get_maxblock(struct drm_buddy *mm, unsigned int order)
>> +__drm_buddy_alloc_range_bias(struct drm_buddy *mm,
>> + u64 start, u64 end,
>> + unsigned int order,
>> + unsigned long flags)
>> +{
>> + struct drm_buddy_block *block;
>> + bool fallback = false;
>> +
>> + block = __alloc_range_bias(mm, start, end, order,
>> + flags, fallback);
>> + if (IS_ERR(block) && mm->clear_avail)
>> + return __alloc_range_bias(mm, start, end, order,
>> + flags, !fallback);
>> +
>> + return block;
>> +}
>> +
>> +static struct drm_buddy_block *
>> +get_maxblock(struct drm_buddy *mm, unsigned int order,
>> + unsigned long flags)
>> {
>> - struct drm_buddy_block *max_block = NULL, *node;
>> + struct drm_buddy_block *max_block = NULL, *block = NULL;
>> unsigned int i;
>> for (i = order; i <= mm->max_order; ++i) {
>> - if (!list_empty(&mm->free_list[i])) {
>> - node = list_last_entry(&mm->free_list[i],
>> - struct drm_buddy_block,
>> - link);
>> - if (!max_block) {
>> - max_block = node;
>> + struct drm_buddy_block *tmp_block;
>> +
>> + list_for_each_entry_reverse(tmp_block, &mm->free_list[i],
>> link) {
>> + if (block_incompatible(tmp_block, flags))
>> continue;
>> - }
>> - if (drm_buddy_block_offset(node) >
>> - drm_buddy_block_offset(max_block)) {
>> - max_block = node;
>> - }
>> + block = tmp_block;
>> + break;
>> + }
>> +
>> + if (!block)
>> + continue;
>> +
>> + if (!max_block) {
>> + max_block = block;
>> + continue;
>> + }
>> +
>> + if (drm_buddy_block_offset(block) >
>> + drm_buddy_block_offset(max_block)) {
>> + max_block = block;
>> }
>> }
>> @@ -440,11 +607,29 @@ alloc_from_freelist(struct drm_buddy *mm,
>> int err;
>> if (flags & DRM_BUDDY_TOPDOWN_ALLOCATION) {
>> - block = get_maxblock(mm, order);
>> + block = get_maxblock(mm, order, flags);
>> if (block)
>> /* Store the obtained block order */
>> tmp = drm_buddy_block_order(block);
>> } else {
>> + for (tmp = order; tmp <= mm->max_order; ++tmp) {
>> + struct drm_buddy_block *tmp_block;
>> +
>> + list_for_each_entry_reverse(tmp_block,
>> &mm->free_list[tmp], link) {
>> + if (block_incompatible(tmp_block, flags))
>> + continue;
>> +
>> + block = tmp_block;
>> + break;
>> + }
>> +
>> + if (block)
>> + break;
>> + }
>> + }
>> +
>> + if (!block) {
>> + /* Fallback method */
>> for (tmp = order; tmp <= mm->max_order; ++tmp) {
>> if (!list_empty(&mm->free_list[tmp])) {
>> block = list_last_entry(&mm->free_list[tmp],
>> @@ -454,10 +639,10 @@ alloc_from_freelist(struct drm_buddy *mm,
>> break;
>> }
>> }
>> - }
>> - if (!block)
>> - return ERR_PTR(-ENOSPC);
>> + if (!block)
>> + return ERR_PTR(-ENOSPC);
>> + }
>> BUG_ON(!drm_buddy_block_is_free(block));
>> @@ -473,7 +658,7 @@ alloc_from_freelist(struct drm_buddy *mm,
>> err_undo:
>> if (tmp != order)
>> - __drm_buddy_free(mm, block);
>> + __drm_buddy_free(mm, block, false);
>> return ERR_PTR(err);
>> }
>> @@ -524,6 +709,8 @@ static int __alloc_range(struct drm_buddy *mm,
>> mark_allocated(block);
>> total_allocated += drm_buddy_block_size(mm, block);
>> mm->avail -= drm_buddy_block_size(mm, block);
>> + if (drm_buddy_block_is_clear(block))
>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>> list_add_tail(&block->link, &allocated);
>> continue;
>> }
>> @@ -548,7 +735,7 @@ static int __alloc_range(struct drm_buddy *mm,
>> return 0;
>> err_undo:
>> - /*
>> + /**
>> * We really don't want to leave around a bunch of split
>> blocks, since
>> * bigger is better, so make sure we merge everything back
>> before we
>> * free the allocated blocks.
>> @@ -557,14 +744,14 @@ static int __alloc_range(struct drm_buddy *mm,
>> if (buddy &&
>> (drm_buddy_block_is_free(block) &&
>> drm_buddy_block_is_free(buddy)))
>> - __drm_buddy_free(mm, block);
>> + __drm_buddy_free(mm, block, false);
>> err_free:
>> if (err == -ENOSPC && total_allocated_on_err) {
>> list_splice_tail(&allocated, blocks);
>> *total_allocated_on_err = total_allocated;
>> } else {
>> - drm_buddy_free_list(mm, &allocated);
>> + drm_buddy_free_list_internal(mm, &allocated);
>> }
>> return err;
>> @@ -630,11 +817,11 @@ static int __alloc_contig_try_harder(struct
>> drm_buddy *mm,
>> list_splice(&blocks_lhs, blocks);
>> return 0;
>> } else if (err != -ENOSPC) {
>> - drm_buddy_free_list(mm, blocks);
>> + drm_buddy_free_list_internal(mm, blocks);
>> return err;
>> }
>> /* Free blocks for the next iteration */
>> - drm_buddy_free_list(mm, blocks);
>> + drm_buddy_free_list_internal(mm, blocks);
>> }
>> return -ENOSPC;
>> @@ -690,6 +877,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>> list_del(&block->link);
>> mark_free(mm, block);
>> mm->avail += drm_buddy_block_size(mm, block);
>> + if (drm_buddy_block_is_clear(block))
>> + mm->clear_avail += drm_buddy_block_size(mm, block);
>> /* Prevent recursively freeing this node */
>> parent = block->parent;
>> @@ -701,6 +890,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>> if (err) {
>> mark_allocated(block);
>> mm->avail -= drm_buddy_block_size(mm, block);
>> + if (drm_buddy_block_is_clear(block))
>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>> list_add(&block->link, blocks);
>> }
>> @@ -709,13 +900,28 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>> }
>> EXPORT_SYMBOL(drm_buddy_block_trim);
>> +static struct drm_buddy_block *
>> +__drm_buddy_alloc_blocks(struct drm_buddy *mm,
>> + u64 start, u64 end,
>> + unsigned int order,
>> + unsigned long flags)
>> +{
>> + if (flags & DRM_BUDDY_RANGE_ALLOCATION)
>> + /* Allocate traversing within the range */
>> + return __drm_buddy_alloc_range_bias(mm, start, end,
>> + order, flags);
>> + else
>> + /* Allocate from freelist */
>> + return alloc_from_freelist(mm, order, flags);
>> +}
>> +
>> /**
>> * drm_buddy_alloc_blocks - allocate power-of-two blocks
>> *
>> * @mm: DRM buddy manager to allocate from
>> * @start: start of the allowed range for this block
>> * @end: end of the allowed range for this block
>> - * @size: size of the allocation
>> + * @size: size of the allocation in bytes
>> * @min_block_size: alignment of the allocation
>> * @blocks: output list head to add allocated blocks
>> * @flags: DRM_BUDDY_*_ALLOCATION flags
>> @@ -761,8 +967,18 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>> return -EINVAL;
>> /* Actual range allocation */
>> - if (start + size == end)
>> - return __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
>> + if (start + size == end) {
>> + err = __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
>> + if (err) {
>> + order = ilog2(size) - ilog2(mm->chunk_size);
>> + if (mm->clear_avail && !__force_merge(mm, order))
>> + return __drm_buddy_alloc_range(mm, start, size,
>> NULL, blocks);
>
> That seems strange at a glance. With an actual range allocation like
> with intitial fb or whatever it should just find all overlapping
> pages, splitting down where needed on the edges. Not sure why
> force_merge would factor in here?
In case of not merged (fragmentation due to dirty and clear pages
split), if the memory request goes into that range, we might see the
split blocks as not free and end up returning -ENOSPC.
I found this problem when I tried to allocate the max_order where the
start + size == end and it ends up entering into this code block and it
returns -ENOSPC as there is no force_merge call
and the requested range max order block seen as not free. And we have an
another issue though if we use force_merge to defragment a specific
ordered block, there is a less probability that
we merge back the required blocks in the specific range. For now, we can
go ahead with no force merge support for actual range allocation if we
are sure that we use this only for initial fb etc.
If there are no major concerns, can we push these patches?
Regards,
Arun.
>> +
>> + return err;
>> + }
>> +
>> + return err;
>> + }
>> original_size = size;
>> original_min_size = min_block_size;
>> @@ -786,23 +1002,34 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>> BUG_ON(order < min_order);
>> do {
>> - if (flags & DRM_BUDDY_RANGE_ALLOCATION)
>> - /* Allocate traversing within the range */
>> - block = alloc_range_bias(mm, start, end, order);
>> - else
>> - /* Allocate from freelist */
>> - block = alloc_from_freelist(mm, order, flags);
>> -
>> + block = __drm_buddy_alloc_blocks(mm, start,
>> + end,
>> + order,
>> + flags);
>> if (!IS_ERR(block))
>> break;
>> if (order-- == min_order) {
>> + /**
>> + * Try allocation through force merge method
>> + */
>> + if (mm->clear_avail && !__force_merge(mm, min_order)) {
>> + block = __drm_buddy_alloc_blocks(mm, start,
>> + end,
>> + min_order,
>> + flags);
>> + if (!IS_ERR(block)) {
>> + order = min_order;
>> + break;
>> + }
>> + }
>> +
>> + /**
>> + * Try contiguous block allocation through
>> + * try harder method.
>> + */
>> if (flags & DRM_BUDDY_CONTIGUOUS_ALLOCATION &&
>> !(flags & DRM_BUDDY_RANGE_ALLOCATION))
>> - /*
>> - * Try contiguous block allocation through
>> - * try harder method
>> - */
>> return __alloc_contig_try_harder(mm,
>> original_size,
>> original_min_size,
>> @@ -814,6 +1041,8 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>> mark_allocated(block);
>> mm->avail -= drm_buddy_block_size(mm, block);
>> + if (drm_buddy_block_is_clear(block))
>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>> kmemleak_update_trace(block);
>> list_add_tail(&block->link, &allocated);
>> @@ -823,7 +1052,9 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>> break;
>> } while (1);
>> - /* Trim the allocated block to the required size */
>> + /**
>> + * Trim the allocated block to the required size
>> + */
>> if (original_size != size) {
>> struct list_head *trim_list;
>> LIST_HEAD(temp);
>> @@ -852,7 +1083,7 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>> return 0;
>> err_free:
>> - drm_buddy_free_list(mm, &allocated);
>> + drm_buddy_free_list_internal(mm, &allocated);
>> return err;
>> }
>> EXPORT_SYMBOL(drm_buddy_alloc_blocks);
>> @@ -885,8 +1116,8 @@ void drm_buddy_print(struct drm_buddy *mm,
>> struct drm_printer *p)
>> {
>> int order;
>> - drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free:
>> %lluMiB\n",
>> - mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20);
>> + drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free:
>> %lluMiB, clear_free: %lluMiB\n",
>> + mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20,
>> mm->clear_avail >> 20);
>> for (order = mm->max_order; order >= 0; order--) {
>> struct drm_buddy_block *block;
>> diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>> b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>> index 0d735d5c2b35..942345548bc3 100644
>> --- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>> +++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>> @@ -126,7 +126,7 @@ static int i915_ttm_buddy_man_alloc(struct
>> ttm_resource_manager *man,
>> return 0;
>> err_free_blocks:
>> - drm_buddy_free_list(mm, &bman_res->blocks);
>> + drm_buddy_free_list(mm, &bman_res->blocks, 0);
>> mutex_unlock(&bman->lock);
>> err_free_res:
>> ttm_resource_fini(man, &bman_res->base);
>> @@ -141,7 +141,7 @@ static void i915_ttm_buddy_man_free(struct
>> ttm_resource_manager *man,
>> struct i915_ttm_buddy_manager *bman = to_buddy_manager(man);
>> mutex_lock(&bman->lock);
>> - drm_buddy_free_list(&bman->mm, &bman_res->blocks);
>> + drm_buddy_free_list(&bman->mm, &bman_res->blocks, 0);
>> bman->visible_avail += bman_res->used_visible_size;
>> mutex_unlock(&bman->lock);
>> @@ -345,7 +345,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device
>> *bdev, unsigned int type)
>> ttm_set_driver_manager(bdev, type, NULL);
>> mutex_lock(&bman->lock);
>> - drm_buddy_free_list(mm, &bman->reserved);
>> + drm_buddy_free_list(mm, &bman->reserved, 0);
>> drm_buddy_fini(mm);
>> bman->visible_avail += bman->visible_reserved;
>> WARN_ON_ONCE(bman->visible_avail != bman->visible_size);
>> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c
>> b/drivers/gpu/drm/tests/drm_buddy_test.c
>> index 2f32fb2f12e7..454ad9952f56 100644
>> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
>> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
>> @@ -64,7 +64,7 @@ static void drm_test_buddy_alloc_contiguous(struct
>> kunit *test)
>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>> "buddy_alloc didn't error size=%u\n", 3 * ps);
>> - drm_buddy_free_list(&mm, &middle);
>> + drm_buddy_free_list(&mm, &middle, 0);
>> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>> mm_size,
>> 3 * ps, ps, &allocated,
>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>> @@ -74,7 +74,7 @@ static void drm_test_buddy_alloc_contiguous(struct
>> kunit *test)
>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>> "buddy_alloc didn't error size=%u\n", 2 * ps);
>> - drm_buddy_free_list(&mm, &right);
>> + drm_buddy_free_list(&mm, &right, 0);
>> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>> mm_size,
>> 3 * ps, ps, &allocated,
>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>> @@ -89,7 +89,7 @@ static void drm_test_buddy_alloc_contiguous(struct
>> kunit *test)
>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>> "buddy_alloc hit an error size=%u\n", 2 * ps);
>> - drm_buddy_free_list(&mm, &left);
>> + drm_buddy_free_list(&mm, &left, 0);
>> KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>> mm_size,
>> 3 * ps, ps, &allocated,
>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>> @@ -101,7 +101,7 @@ static void
>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>> KUNIT_ASSERT_EQ(test, total, ps * 2 + ps * 3);
>> - drm_buddy_free_list(&mm, &allocated);
>> + drm_buddy_free_list(&mm, &allocated, 0);
>> drm_buddy_fini(&mm);
>> }
>> @@ -170,7 +170,7 @@ static void
>> drm_test_buddy_alloc_pathological(struct kunit *test)
>> top, max_order);
>> }
>> - drm_buddy_free_list(&mm, &holes);
>> + drm_buddy_free_list(&mm, &holes, 0);
>> /* Nothing larger than blocks of chunk_size now available */
>> for (order = 1; order <= max_order; order++) {
>> @@ -182,7 +182,7 @@ static void
>> drm_test_buddy_alloc_pathological(struct kunit *test)
>> }
>> list_splice_tail(&holes, &blocks);
>> - drm_buddy_free_list(&mm, &blocks);
>> + drm_buddy_free_list(&mm, &blocks, 0);
>> drm_buddy_fini(&mm);
>> }
>> @@ -277,7 +277,7 @@ static void
>> drm_test_buddy_alloc_pessimistic(struct kunit *test)
>> list_del(&block->link);
>> drm_buddy_free_block(&mm, block);
>> - drm_buddy_free_list(&mm, &blocks);
>> + drm_buddy_free_list(&mm, &blocks, 0);
>> drm_buddy_fini(&mm);
>> }
>> @@ -323,7 +323,7 @@ static void
>> drm_test_buddy_alloc_optimistic(struct kunit *test)
>> size, size, &tmp, flags),
>> "buddy_alloc unexpectedly succeeded, it
>> should be full!");
>> - drm_buddy_free_list(&mm, &blocks);
>> + drm_buddy_free_list(&mm, &blocks, 0);
>> drm_buddy_fini(&mm);
>> }
>> @@ -358,7 +358,7 @@ static void drm_test_buddy_alloc_limit(struct
>> kunit *test)
>> drm_buddy_block_size(&mm, block),
>> BIT_ULL(mm.max_order) * PAGE_SIZE);
>> - drm_buddy_free_list(&mm, &allocated);
>> + drm_buddy_free_list(&mm, &allocated, 0);
>> drm_buddy_fini(&mm);
>> }
>> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>> b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>> index 115ec745e502..1ad678b62c4a 100644
>> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>> @@ -196,7 +196,7 @@ static int xe_ttm_vram_mgr_new(struct
>> ttm_resource_manager *man,
>> return 0;
>> error_free_blocks:
>> - drm_buddy_free_list(mm, &vres->blocks);
>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>> mutex_unlock(&mgr->lock);
>> error_fini:
>> ttm_resource_fini(man, &vres->base);
>> @@ -214,7 +214,7 @@ static void xe_ttm_vram_mgr_del(struct
>> ttm_resource_manager *man,
>> struct drm_buddy *mm = &mgr->mm;
>> mutex_lock(&mgr->lock);
>> - drm_buddy_free_list(mm, &vres->blocks);
>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>> mgr->visible_avail += vres->used_visible_size;
>> mutex_unlock(&mgr->lock);
>> diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h
>> index a5b39fc01003..82570f77e817 100644
>> --- a/include/drm/drm_buddy.h
>> +++ b/include/drm/drm_buddy.h
>> @@ -25,6 +25,8 @@
>> #define DRM_BUDDY_RANGE_ALLOCATION BIT(0)
>> #define DRM_BUDDY_TOPDOWN_ALLOCATION BIT(1)
>> #define DRM_BUDDY_CONTIGUOUS_ALLOCATION BIT(2)
>> +#define DRM_BUDDY_CLEAR_ALLOCATION BIT(3)
>> +#define DRM_BUDDY_CLEARED BIT(4)
>> struct drm_buddy_block {
>> #define DRM_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
>> @@ -32,8 +34,9 @@ struct drm_buddy_block {
>> #define DRM_BUDDY_ALLOCATED (1 << 10)
>> #define DRM_BUDDY_FREE (2 << 10)
>> #define DRM_BUDDY_SPLIT (3 << 10)
>> +#define DRM_BUDDY_HEADER_CLEAR GENMASK_ULL(9, 9)
>> /* Free to be used, if needed in the future */
>> -#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
>> +#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(8, 6)
>> #define DRM_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
>> u64 header;
>> @@ -86,6 +89,7 @@ struct drm_buddy {
>> u64 chunk_size;
>> u64 size;
>> u64 avail;
>> + u64 clear_avail;
>> };
>> static inline u64
>> @@ -112,6 +116,12 @@ drm_buddy_block_is_allocated(struct
>> drm_buddy_block *block)
>> return drm_buddy_block_state(block) == DRM_BUDDY_ALLOCATED;
>> }
>> +static inline bool
>> +drm_buddy_block_is_clear(struct drm_buddy_block *block)
>> +{
>> + return block->header & DRM_BUDDY_HEADER_CLEAR;
>> +}
>> +
>> static inline bool
>> drm_buddy_block_is_free(struct drm_buddy_block *block)
>> {
>> @@ -150,7 +160,9 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>> void drm_buddy_free_block(struct drm_buddy *mm, struct
>> drm_buddy_block *block);
>> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head
>> *objects);
>> +void drm_buddy_free_list(struct drm_buddy *mm,
>> + struct list_head *objects,
>> + unsigned int flags);
>> void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p);
>> void drm_buddy_block_print(struct drm_buddy *mm,
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature
2024-03-28 16:07 ` Paneer Selvam, Arunpravin
@ 2024-03-28 16:48 ` Matthew Auld
2024-04-01 11:07 ` Paneer Selvam, Arunpravin
0 siblings, 1 reply; 21+ messages in thread
From: Matthew Auld @ 2024-03-28 16:48 UTC (permalink / raw)
To: Paneer Selvam, Arunpravin, dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, mario.limonciello, felix.kuehling
On 28/03/2024 16:07, Paneer Selvam, Arunpravin wrote:
> Hi Matthew,
>
> On 3/26/2024 11:39 PM, Matthew Auld wrote:
>> On 18/03/2024 21:40, Arunpravin Paneer Selvam wrote:
>>> - Add tracking clear page feature.
>>>
>>> - Driver should enable the DRM_BUDDY_CLEARED flag if it
>>> successfully clears the blocks in the free path. On the otherhand,
>>> DRM buddy marks each block as cleared.
>>>
>>> - Track the available cleared pages size
>>>
>>> - If driver requests cleared memory we prefer cleared memory
>>> but fallback to uncleared if we can't find the cleared blocks.
>>> when driver requests uncleared memory we try to use uncleared but
>>> fallback to cleared memory if necessary.
>>>
>>> - When a block gets freed we clear it and mark the freed block as
>>> cleared,
>>> when there are buddies which are cleared as well we can merge them.
>>> Otherwise, we prefer to keep the blocks as separated.
>>>
>>> - Add a function to support defragmentation.
>>>
>>> v1:
>>> - Depends on the flag check DRM_BUDDY_CLEARED, enable the block as
>>> cleared. Else, reset the clear flag for each block in the
>>> list(Christian)
>>> - For merging the 2 cleared blocks compare as below,
>>> drm_buddy_is_clear(block) != drm_buddy_is_clear(buddy)(Christian)
>>> - Defragment the memory beginning from min_order
>>> till the required memory space is available.
>>>
>>> v2: (Matthew)
>>> - Add a wrapper drm_buddy_free_list_internal for the freeing of
>>> blocks
>>> operation within drm buddy.
>>> - Write a macro block_incompatible() to allocate the required blocks.
>>> - Update the xe driver for the drm_buddy_free_list change in
>>> arguments.
>>> - add a warning if the two blocks are incompatible on
>>> defragmentation
>>> - call full defragmentation in the fini() function
>>> - place a condition to test if min_order is equal to 0
>>> - replace the list with safe_reverse() variant as we might
>>> remove the block from the list.
>>>
>>> v3:
>>> - fix Gitlab user reported lockup issue.
>>> - Keep DRM_BUDDY_HEADER_CLEAR define sorted(Matthew)
>>> - modify to pass the root order instead max_order in fini()
>>> function(Matthew)
>>> - change bool 1 to true(Matthew)
>>> - add check if min_block_size is power of 2(Matthew)
>>> - modify the min_block_size datatype to u64(Matthew)
>>>
>>> v4:
>>> - rename the function drm_buddy_defrag with __force_merge.
>>> - Include __force_merge directly in drm buddy file and remove
>>> the defrag use in amdgpu driver.
>>> - Remove list_empty() check(Matthew)
>>> - Remove unnecessary space, headers and placement of new
>>> variables(Matthew)
>>> - Add a unit test case(Matthew)
>>>
>>> Signed-off-by: Arunpravin Paneer Selvam
>>> <Arunpravin.PaneerSelvam@amd.com>
>>> Signed-off-by: Matthew Auld <matthew.auld@intel.com>
>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>> Suggested-by: Matthew Auld <matthew.auld@intel.com>
>>> ---
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>>> drivers/gpu/drm/drm_buddy.c | 427 ++++++++++++++----
>>> drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 6 +-
>>> drivers/gpu/drm/tests/drm_buddy_test.c | 18 +-
>>> drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 4 +-
>>> include/drm/drm_buddy.h | 16 +-
>>> 6 files changed, 360 insertions(+), 117 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>> index 8db880244324..c0c851409241 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>> @@ -571,7 +571,7 @@ static int amdgpu_vram_mgr_new(struct
>>> ttm_resource_manager *man,
>>> return 0;
>>> error_free_blocks:
>>> - drm_buddy_free_list(mm, &vres->blocks);
>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>> mutex_unlock(&mgr->lock);
>>> error_fini:
>>> ttm_resource_fini(man, &vres->base);
>>> @@ -604,7 +604,7 @@ static void amdgpu_vram_mgr_del(struct
>>> ttm_resource_manager *man,
>>> amdgpu_vram_mgr_do_reserve(man);
>>> - drm_buddy_free_list(mm, &vres->blocks);
>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>> mutex_unlock(&mgr->lock);
>>> atomic64_sub(vis_usage, &mgr->vis_usage);
>>> @@ -912,7 +912,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device
>>> *adev)
>>> kfree(rsv);
>>> list_for_each_entry_safe(rsv, temp, &mgr->reserved_pages,
>>> blocks) {
>>> - drm_buddy_free_list(&mgr->mm, &rsv->allocated);
>>> + drm_buddy_free_list(&mgr->mm, &rsv->allocated, 0);
>>> kfree(rsv);
>>> }
>>> if (!adev->gmc.is_app_apu)
>>> diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c
>>> index c4222b886db7..625a30a6b855 100644
>>> --- a/drivers/gpu/drm/drm_buddy.c
>>> +++ b/drivers/gpu/drm/drm_buddy.c
>>> @@ -38,8 +38,8 @@ static void drm_block_free(struct drm_buddy *mm,
>>> kmem_cache_free(slab_blocks, block);
>>> }
>>> -static void list_insert_sorted(struct drm_buddy *mm,
>>> - struct drm_buddy_block *block)
>>> +static void list_insert(struct drm_buddy *mm,
>>> + struct drm_buddy_block *block)
>>> {
>>> struct drm_buddy_block *node;
>>> struct list_head *head;
>>> @@ -57,6 +57,16 @@ static void list_insert_sorted(struct drm_buddy *mm,
>>> __list_add(&block->link, node->link.prev, &node->link);
>>> }
>>> +static void clear_reset(struct drm_buddy_block *block)
>>> +{
>>> + block->header &= ~DRM_BUDDY_HEADER_CLEAR;
>>> +}
>>> +
>>> +static void mark_cleared(struct drm_buddy_block *block)
>>> +{
>>> + block->header |= DRM_BUDDY_HEADER_CLEAR;
>>> +}
>>> +
>>> static void mark_allocated(struct drm_buddy_block *block)
>>> {
>>> block->header &= ~DRM_BUDDY_HEADER_STATE;
>>> @@ -71,7 +81,7 @@ static void mark_free(struct drm_buddy *mm,
>>> block->header &= ~DRM_BUDDY_HEADER_STATE;
>>> block->header |= DRM_BUDDY_FREE;
>>> - list_insert_sorted(mm, block);
>>> + list_insert(mm, block);
>>> }
>>> static void mark_split(struct drm_buddy_block *block)
>>> @@ -82,6 +92,114 @@ static void mark_split(struct drm_buddy_block
>>> *block)
>>> list_del(&block->link);
>>> }
>>> +static struct drm_buddy_block *
>>> +__get_buddy(struct drm_buddy_block *block)
>>> +{
>>> + struct drm_buddy_block *parent;
>>> +
>>> + parent = block->parent;
>>> + if (!parent)
>>> + return NULL;
>>> +
>>> + if (parent->left == block)
>>> + return parent->right;
>>> +
>>> + return parent->left;
>>> +}
>>> +
>>> +static unsigned int __drm_buddy_free(struct drm_buddy *mm,
>>> + struct drm_buddy_block *block,
>>> + bool force_merge)
>>> +{
>>> + struct drm_buddy_block *parent;
>>> + unsigned int order;
>>> +
>>> + while ((parent = block->parent)) {
>>> + struct drm_buddy_block *buddy;
>>> +
>>> + buddy = __get_buddy(block);
>>> +
>>> + if (!drm_buddy_block_is_free(buddy))
>>> + break;
>>> +
>>> + if (!force_merge) {
>>> + /**
>>
>> Not really valid kernel-doc AFAIK. I think drop the extra *. Below also.
>>
>>> + * Check the block and its buddy clear state and exit
>>> + * the loop if they both have the dissimilar state.
>>> + */
>>> + if (drm_buddy_block_is_clear(block) !=
>>> + drm_buddy_block_is_clear(buddy))
>>> + break;
>>> +
>>> + if (drm_buddy_block_is_clear(block))
>>> + mark_cleared(parent);
>>> + }
>>> +
>>> + list_del(&buddy->link);
>>> + if (force_merge && drm_buddy_block_is_clear(buddy))
>>> + mm->clear_avail -= drm_buddy_block_size(mm, buddy);
>>> +
>>> + drm_block_free(mm, block);
>>> + drm_block_free(mm, buddy);
>>> +
>>> + block = parent;
>>> + }
>>> +
>>> + order = drm_buddy_block_order(block);
>>> + mark_free(mm, block);
>>> +
>>> + return order;
>>> +}
>>> +
>>> +static int __force_merge(struct drm_buddy *mm,
>>> + unsigned int min_order)
>>> +{
>>> + unsigned int order;
>>> + int i;
>>> +
>>> + if (!min_order)
>>> + return -ENOMEM;
>>> +
>>> + if (min_order > mm->max_order)
>>> + return -EINVAL;
>>> +
>>> + for (i = min_order - 1; i >= 0; i--) {
>>> + struct drm_buddy_block *block, *prev;
>>> +
>>> + list_for_each_entry_safe_reverse(block, prev,
>>> &mm->free_list[i], link) {
>>> + struct drm_buddy_block *buddy;
>>> +
>>> + if (!block->parent)
>>> + continue;
>>> +
>>> + buddy = __get_buddy(block);
>>> + if (!drm_buddy_block_is_free(buddy))
>>> + continue;
>>> +
>>> + WARN_ON(drm_buddy_block_is_clear(block) ==
>>> + drm_buddy_block_is_clear(buddy));
>>> +
>>> + /**
>>> + * If the prev block is same as buddy, don't access the
>>> + * block in the next iteration as we would free the
>>> + * buddy block as part of the free function.
>>> + */
>>> + if (prev == buddy)
>>> + prev = list_prev_entry(prev, link);
>>> +
>>> + list_del(&block->link);
>>> + if (drm_buddy_block_is_clear(block))
>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>> +
>>> + order = __drm_buddy_free(mm, block, true);
>>> + if (order >= min_order)
>>> + return 0;
>>> + }
>>> + }
>>> +
>>> + return -ENOMEM;
>>> +}
>>> +
>>> /**
>>> * drm_buddy_init - init memory manager
>>> *
>>> @@ -137,7 +255,7 @@ int drm_buddy_init(struct drm_buddy *mm, u64
>>> size, u64 chunk_size)
>>> offset = 0;
>>> i = 0;
>>> - /*
>>> + /**
>>> * Split into power-of-two blocks, in case we are given a size
>>> that is
>>> * not itself a power-of-two.
>>> */
>>> @@ -186,11 +304,21 @@ EXPORT_SYMBOL(drm_buddy_init);
>>> */
>>> void drm_buddy_fini(struct drm_buddy *mm)
>>> {
>>> + u64 root_size, size;
>>> + unsigned int order;
>>> int i;
>>> + size = mm->size;
>>> +
>>> for (i = 0; i < mm->n_roots; ++i) {
>>> + order = ilog2(size) - ilog2(mm->chunk_size);
>>> + __force_merge(mm, order);
>>> +
>>> WARN_ON(!drm_buddy_block_is_free(mm->roots[i]));
>>> drm_block_free(mm, mm->roots[i]);
>>> +
>>> + root_size = mm->chunk_size << order;
>>> + size -= root_size;
>>> }
>>> WARN_ON(mm->avail != mm->size);
>>> @@ -223,26 +351,17 @@ static int split_block(struct drm_buddy *mm,
>>> mark_free(mm, block->left);
>>> mark_free(mm, block->right);
>>> + if (drm_buddy_block_is_clear(block)) {
>>> + mark_cleared(block->left);
>>> + mark_cleared(block->right);
>>> + clear_reset(block);
>>> + }
>>> +
>>> mark_split(block);
>>> return 0;
>>> }
>>> -static struct drm_buddy_block *
>>> -__get_buddy(struct drm_buddy_block *block)
>>> -{
>>> - struct drm_buddy_block *parent;
>>> -
>>> - parent = block->parent;
>>> - if (!parent)
>>> - return NULL;
>>> -
>>> - if (parent->left == block)
>>> - return parent->right;
>>> -
>>> - return parent->left;
>>> -}
>>> -
>>> /**
>>> * drm_get_buddy - get buddy address
>>> *
>>> @@ -260,30 +379,6 @@ drm_get_buddy(struct drm_buddy_block *block)
>>> }
>>> EXPORT_SYMBOL(drm_get_buddy);
>>> -static void __drm_buddy_free(struct drm_buddy *mm,
>>> - struct drm_buddy_block *block)
>>> -{
>>> - struct drm_buddy_block *parent;
>>> -
>>> - while ((parent = block->parent)) {
>>> - struct drm_buddy_block *buddy;
>>> -
>>> - buddy = __get_buddy(block);
>>> -
>>> - if (!drm_buddy_block_is_free(buddy))
>>> - break;
>>> -
>>> - list_del(&buddy->link);
>>> -
>>> - drm_block_free(mm, block);
>>> - drm_block_free(mm, buddy);
>>> -
>>> - block = parent;
>>> - }
>>> -
>>> - mark_free(mm, block);
>>> -}
>>> -
>>> /**
>>> * drm_buddy_free_block - free a block
>>> *
>>> @@ -295,26 +390,59 @@ void drm_buddy_free_block(struct drm_buddy *mm,
>>> {
>>> BUG_ON(!drm_buddy_block_is_allocated(block));
>>> mm->avail += drm_buddy_block_size(mm, block);
>>> - __drm_buddy_free(mm, block);
>>> + if (drm_buddy_block_is_clear(block))
>>> + mm->clear_avail += drm_buddy_block_size(mm, block);
>>> +
>>> + __drm_buddy_free(mm, block, false);
>>> }
>>> EXPORT_SYMBOL(drm_buddy_free_block);
>>> -/**
>>> - * drm_buddy_free_list - free blocks
>>> - *
>>> - * @mm: DRM buddy manager
>>> - * @objects: input list head to free blocks
>>> - */
>>> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head
>>> *objects)
>>> +static void __drm_buddy_free_list(struct drm_buddy *mm,
>>> + struct list_head *objects,
>>> + bool mark_clear,
>>> + bool mark_dirty)
>>> {
>>> struct drm_buddy_block *block, *on;
>>> + WARN_ON(mark_dirty && mark_clear);
>>> +
>>> list_for_each_entry_safe(block, on, objects, link) {
>>> + if (mark_clear)
>>> + mark_cleared(block);
>>> + else if (mark_dirty)
>>> + clear_reset(block);
>>> drm_buddy_free_block(mm, block);
>>> cond_resched();
>>> }
>>> INIT_LIST_HEAD(objects);
>>> }
>>> +
>>> +static void drm_buddy_free_list_internal(struct drm_buddy *mm,
>>> + struct list_head *objects)
>>> +{
>>> + /**
>>> + * Don't touch the clear/dirty bit, since allocation is still
>>> internal
>>> + * at this point. For example we might have just failed part of the
>>> + * allocation.
>>> + */
>>> + __drm_buddy_free_list(mm, objects, false, false);
>>> +}
>>> +
>>> +/**
>>> + * drm_buddy_free_list - free blocks
>>> + *
>>> + * @mm: DRM buddy manager
>>> + * @objects: input list head to free blocks
>>> + * @flags: optional flags like DRM_BUDDY_CLEARED
>>> + */
>>> +void drm_buddy_free_list(struct drm_buddy *mm,
>>> + struct list_head *objects,
>>> + unsigned int flags)
>>> +{
>>> + bool mark_clear = flags & DRM_BUDDY_CLEARED;
>>> +
>>> + __drm_buddy_free_list(mm, objects, mark_clear, !mark_clear);
>>> +}
>>> EXPORT_SYMBOL(drm_buddy_free_list);
>>> static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
>>> @@ -327,10 +455,19 @@ static inline bool contains(u64 s1, u64 e1, u64
>>> s2, u64 e2)
>>> return s1 <= s2 && e1 >= e2;
>>> }
>>> +static bool block_incompatible(struct drm_buddy_block *block,
>>> unsigned int flags)
>>> +{
>>> + bool needs_clear = flags & DRM_BUDDY_CLEAR_ALLOCATION;
>>> +
>>> + return needs_clear != drm_buddy_block_is_clear(block);
>>> +}
>>> +
>>> static struct drm_buddy_block *
>>> -alloc_range_bias(struct drm_buddy *mm,
>>> - u64 start, u64 end,
>>> - unsigned int order)
>>> +__alloc_range_bias(struct drm_buddy *mm,
>>> + u64 start, u64 end,
>>> + unsigned int order,
>>> + unsigned long flags,
>>> + bool fallback)
>>> {
>>> struct drm_buddy_block *block;
>>> struct drm_buddy_block *buddy;
>>> @@ -369,7 +506,10 @@ alloc_range_bias(struct drm_buddy *mm,
>>> if (contains(start, end, block_start, block_end) &&
>>> order == drm_buddy_block_order(block)) {
>>> - /*
>>> + if (!fallback && block_incompatible(block, flags))
>>> + continue;
>>> +
>>> + /**
>>> * Find the free block within the range.
>>> */
>>> if (drm_buddy_block_is_free(block))
>>> @@ -391,7 +531,7 @@ alloc_range_bias(struct drm_buddy *mm,
>>> return ERR_PTR(-ENOSPC);
>>> err_undo:
>>> - /*
>>> + /**
>>> * We really don't want to leave around a bunch of split
>>> blocks, since
>>> * bigger is better, so make sure we merge everything back
>>> before we
>>> * free the allocated blocks.
>>> @@ -400,30 +540,57 @@ alloc_range_bias(struct drm_buddy *mm,
>>> if (buddy &&
>>> (drm_buddy_block_is_free(block) &&
>>> drm_buddy_block_is_free(buddy)))
>>> - __drm_buddy_free(mm, block);
>>> + __drm_buddy_free(mm, block, false);
>>> return ERR_PTR(err);
>>> }
>>> static struct drm_buddy_block *
>>> -get_maxblock(struct drm_buddy *mm, unsigned int order)
>>> +__drm_buddy_alloc_range_bias(struct drm_buddy *mm,
>>> + u64 start, u64 end,
>>> + unsigned int order,
>>> + unsigned long flags)
>>> +{
>>> + struct drm_buddy_block *block;
>>> + bool fallback = false;
>>> +
>>> + block = __alloc_range_bias(mm, start, end, order,
>>> + flags, fallback);
>>> + if (IS_ERR(block) && mm->clear_avail)
>>> + return __alloc_range_bias(mm, start, end, order,
>>> + flags, !fallback);
>>> +
>>> + return block;
>>> +}
>>> +
>>> +static struct drm_buddy_block *
>>> +get_maxblock(struct drm_buddy *mm, unsigned int order,
>>> + unsigned long flags)
>>> {
>>> - struct drm_buddy_block *max_block = NULL, *node;
>>> + struct drm_buddy_block *max_block = NULL, *block = NULL;
>>> unsigned int i;
>>> for (i = order; i <= mm->max_order; ++i) {
>>> - if (!list_empty(&mm->free_list[i])) {
>>> - node = list_last_entry(&mm->free_list[i],
>>> - struct drm_buddy_block,
>>> - link);
>>> - if (!max_block) {
>>> - max_block = node;
>>> + struct drm_buddy_block *tmp_block;
>>> +
>>> + list_for_each_entry_reverse(tmp_block, &mm->free_list[i],
>>> link) {
>>> + if (block_incompatible(tmp_block, flags))
>>> continue;
>>> - }
>>> - if (drm_buddy_block_offset(node) >
>>> - drm_buddy_block_offset(max_block)) {
>>> - max_block = node;
>>> - }
>>> + block = tmp_block;
>>> + break;
>>> + }
>>> +
>>> + if (!block)
>>> + continue;
>>> +
>>> + if (!max_block) {
>>> + max_block = block;
>>> + continue;
>>> + }
>>> +
>>> + if (drm_buddy_block_offset(block) >
>>> + drm_buddy_block_offset(max_block)) {
>>> + max_block = block;
>>> }
>>> }
>>> @@ -440,11 +607,29 @@ alloc_from_freelist(struct drm_buddy *mm,
>>> int err;
>>> if (flags & DRM_BUDDY_TOPDOWN_ALLOCATION) {
>>> - block = get_maxblock(mm, order);
>>> + block = get_maxblock(mm, order, flags);
>>> if (block)
>>> /* Store the obtained block order */
>>> tmp = drm_buddy_block_order(block);
>>> } else {
>>> + for (tmp = order; tmp <= mm->max_order; ++tmp) {
>>> + struct drm_buddy_block *tmp_block;
>>> +
>>> + list_for_each_entry_reverse(tmp_block,
>>> &mm->free_list[tmp], link) {
>>> + if (block_incompatible(tmp_block, flags))
>>> + continue;
>>> +
>>> + block = tmp_block;
>>> + break;
>>> + }
>>> +
>>> + if (block)
>>> + break;
>>> + }
>>> + }
>>> +
>>> + if (!block) {
>>> + /* Fallback method */
>>> for (tmp = order; tmp <= mm->max_order; ++tmp) {
>>> if (!list_empty(&mm->free_list[tmp])) {
>>> block = list_last_entry(&mm->free_list[tmp],
>>> @@ -454,10 +639,10 @@ alloc_from_freelist(struct drm_buddy *mm,
>>> break;
>>> }
>>> }
>>> - }
>>> - if (!block)
>>> - return ERR_PTR(-ENOSPC);
>>> + if (!block)
>>> + return ERR_PTR(-ENOSPC);
>>> + }
>>> BUG_ON(!drm_buddy_block_is_free(block));
>>> @@ -473,7 +658,7 @@ alloc_from_freelist(struct drm_buddy *mm,
>>> err_undo:
>>> if (tmp != order)
>>> - __drm_buddy_free(mm, block);
>>> + __drm_buddy_free(mm, block, false);
>>> return ERR_PTR(err);
>>> }
>>> @@ -524,6 +709,8 @@ static int __alloc_range(struct drm_buddy *mm,
>>> mark_allocated(block);
>>> total_allocated += drm_buddy_block_size(mm, block);
>>> mm->avail -= drm_buddy_block_size(mm, block);
>>> + if (drm_buddy_block_is_clear(block))
>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>> list_add_tail(&block->link, &allocated);
>>> continue;
>>> }
>>> @@ -548,7 +735,7 @@ static int __alloc_range(struct drm_buddy *mm,
>>> return 0;
>>> err_undo:
>>> - /*
>>> + /**
>>> * We really don't want to leave around a bunch of split
>>> blocks, since
>>> * bigger is better, so make sure we merge everything back
>>> before we
>>> * free the allocated blocks.
>>> @@ -557,14 +744,14 @@ static int __alloc_range(struct drm_buddy *mm,
>>> if (buddy &&
>>> (drm_buddy_block_is_free(block) &&
>>> drm_buddy_block_is_free(buddy)))
>>> - __drm_buddy_free(mm, block);
>>> + __drm_buddy_free(mm, block, false);
>>> err_free:
>>> if (err == -ENOSPC && total_allocated_on_err) {
>>> list_splice_tail(&allocated, blocks);
>>> *total_allocated_on_err = total_allocated;
>>> } else {
>>> - drm_buddy_free_list(mm, &allocated);
>>> + drm_buddy_free_list_internal(mm, &allocated);
>>> }
>>> return err;
>>> @@ -630,11 +817,11 @@ static int __alloc_contig_try_harder(struct
>>> drm_buddy *mm,
>>> list_splice(&blocks_lhs, blocks);
>>> return 0;
>>> } else if (err != -ENOSPC) {
>>> - drm_buddy_free_list(mm, blocks);
>>> + drm_buddy_free_list_internal(mm, blocks);
>>> return err;
>>> }
>>> /* Free blocks for the next iteration */
>>> - drm_buddy_free_list(mm, blocks);
>>> + drm_buddy_free_list_internal(mm, blocks);
>>> }
>>> return -ENOSPC;
>>> @@ -690,6 +877,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>> list_del(&block->link);
>>> mark_free(mm, block);
>>> mm->avail += drm_buddy_block_size(mm, block);
>>> + if (drm_buddy_block_is_clear(block))
>>> + mm->clear_avail += drm_buddy_block_size(mm, block);
>>> /* Prevent recursively freeing this node */
>>> parent = block->parent;
>>> @@ -701,6 +890,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>> if (err) {
>>> mark_allocated(block);
>>> mm->avail -= drm_buddy_block_size(mm, block);
>>> + if (drm_buddy_block_is_clear(block))
>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>> list_add(&block->link, blocks);
>>> }
>>> @@ -709,13 +900,28 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>> }
>>> EXPORT_SYMBOL(drm_buddy_block_trim);
>>> +static struct drm_buddy_block *
>>> +__drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>> + u64 start, u64 end,
>>> + unsigned int order,
>>> + unsigned long flags)
>>> +{
>>> + if (flags & DRM_BUDDY_RANGE_ALLOCATION)
>>> + /* Allocate traversing within the range */
>>> + return __drm_buddy_alloc_range_bias(mm, start, end,
>>> + order, flags);
>>> + else
>>> + /* Allocate from freelist */
>>> + return alloc_from_freelist(mm, order, flags);
>>> +}
>>> +
>>> /**
>>> * drm_buddy_alloc_blocks - allocate power-of-two blocks
>>> *
>>> * @mm: DRM buddy manager to allocate from
>>> * @start: start of the allowed range for this block
>>> * @end: end of the allowed range for this block
>>> - * @size: size of the allocation
>>> + * @size: size of the allocation in bytes
>>> * @min_block_size: alignment of the allocation
>>> * @blocks: output list head to add allocated blocks
>>> * @flags: DRM_BUDDY_*_ALLOCATION flags
>>> @@ -761,8 +967,18 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>> return -EINVAL;
>>> /* Actual range allocation */
>>> - if (start + size == end)
>>> - return __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
>>> + if (start + size == end) {
>>> + err = __drm_buddy_alloc_range(mm, start, size, NULL, blocks);
>>> + if (err) {
>>> + order = ilog2(size) - ilog2(mm->chunk_size);
>>> + if (mm->clear_avail && !__force_merge(mm, order))
>>> + return __drm_buddy_alloc_range(mm, start, size,
>>> NULL, blocks);
>>
>> That seems strange at a glance. With an actual range allocation like
>> with intitial fb or whatever it should just find all overlapping
>> pages, splitting down where needed on the edges. Not sure why
>> force_merge would factor in here?
> In case of not merged (fragmentation due to dirty and clear pages
> split), if the memory request goes into that range, we might see the
> split blocks as not free and end up returning -ENOSPC.
> I found this problem when I tried to allocate the max_order where the
> start + size == end and it ends up entering into this code block and it
> returns -ENOSPC as there is no force_merge call
> and the requested range max order block seen as not free. And we have an
> another issue though if we use force_merge to defragment a specific
> ordered block, there is a less probability that
> we merge back the required blocks in the specific range. For now, we can
> go ahead with no force merge support for actual range allocation if we
> are sure that we use this only for initial fb etc.
Ah right, now I see. But AFAICT trouble is if we say allocate 8K range
at some specific offset, like above, which fails due to clear/dirty
split then calling force_merge(order(8K)) is not always going to help.
It will just force merge enough for 8K, but that might not be the 8K at
the offset we were looking for, so it still fails. I think perhaps
update the dfs in alloc_range to not bail when contains && split. Or I
think other option is to force merge the entire address space on err.
Other ideas?
>
> If there are no major concerns, can we push these patches?
>
> Regards,
> Arun.
>>> +
>>> + return err;
>>> + }
>>> +
>>> + return err;
>>> + }
>>> original_size = size;
>>> original_min_size = min_block_size;
>>> @@ -786,23 +1002,34 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>> BUG_ON(order < min_order);
>>> do {
>>> - if (flags & DRM_BUDDY_RANGE_ALLOCATION)
>>> - /* Allocate traversing within the range */
>>> - block = alloc_range_bias(mm, start, end, order);
>>> - else
>>> - /* Allocate from freelist */
>>> - block = alloc_from_freelist(mm, order, flags);
>>> -
>>> + block = __drm_buddy_alloc_blocks(mm, start,
>>> + end,
>>> + order,
>>> + flags);
>>> if (!IS_ERR(block))
>>> break;
>>> if (order-- == min_order) {
>>> + /**
>>> + * Try allocation through force merge method
>>> + */
>>> + if (mm->clear_avail && !__force_merge(mm, min_order)) {
>>> + block = __drm_buddy_alloc_blocks(mm, start,
>>> + end,
>>> + min_order,
>>> + flags);
>>> + if (!IS_ERR(block)) {
>>> + order = min_order;
>>> + break;
>>> + }
>>> + }
>>> +
>>> + /**
>>> + * Try contiguous block allocation through
>>> + * try harder method.
>>> + */
>>> if (flags & DRM_BUDDY_CONTIGUOUS_ALLOCATION &&
>>> !(flags & DRM_BUDDY_RANGE_ALLOCATION))
>>> - /*
>>> - * Try contiguous block allocation through
>>> - * try harder method
>>> - */
>>> return __alloc_contig_try_harder(mm,
>>> original_size,
>>> original_min_size,
>>> @@ -814,6 +1041,8 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>> mark_allocated(block);
>>> mm->avail -= drm_buddy_block_size(mm, block);
>>> + if (drm_buddy_block_is_clear(block))
>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>> kmemleak_update_trace(block);
>>> list_add_tail(&block->link, &allocated);
>>> @@ -823,7 +1052,9 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>> break;
>>> } while (1);
>>> - /* Trim the allocated block to the required size */
>>> + /**
>>> + * Trim the allocated block to the required size
>>> + */
>>> if (original_size != size) {
>>> struct list_head *trim_list;
>>> LIST_HEAD(temp);
>>> @@ -852,7 +1083,7 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>> return 0;
>>> err_free:
>>> - drm_buddy_free_list(mm, &allocated);
>>> + drm_buddy_free_list_internal(mm, &allocated);
>>> return err;
>>> }
>>> EXPORT_SYMBOL(drm_buddy_alloc_blocks);
>>> @@ -885,8 +1116,8 @@ void drm_buddy_print(struct drm_buddy *mm,
>>> struct drm_printer *p)
>>> {
>>> int order;
>>> - drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free:
>>> %lluMiB\n",
>>> - mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20);
>>> + drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free:
>>> %lluMiB, clear_free: %lluMiB\n",
>>> + mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20,
>>> mm->clear_avail >> 20);
>>> for (order = mm->max_order; order >= 0; order--) {
>>> struct drm_buddy_block *block;
>>> diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>> b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>> index 0d735d5c2b35..942345548bc3 100644
>>> --- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>> +++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>> @@ -126,7 +126,7 @@ static int i915_ttm_buddy_man_alloc(struct
>>> ttm_resource_manager *man,
>>> return 0;
>>> err_free_blocks:
>>> - drm_buddy_free_list(mm, &bman_res->blocks);
>>> + drm_buddy_free_list(mm, &bman_res->blocks, 0);
>>> mutex_unlock(&bman->lock);
>>> err_free_res:
>>> ttm_resource_fini(man, &bman_res->base);
>>> @@ -141,7 +141,7 @@ static void i915_ttm_buddy_man_free(struct
>>> ttm_resource_manager *man,
>>> struct i915_ttm_buddy_manager *bman = to_buddy_manager(man);
>>> mutex_lock(&bman->lock);
>>> - drm_buddy_free_list(&bman->mm, &bman_res->blocks);
>>> + drm_buddy_free_list(&bman->mm, &bman_res->blocks, 0);
>>> bman->visible_avail += bman_res->used_visible_size;
>>> mutex_unlock(&bman->lock);
>>> @@ -345,7 +345,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device
>>> *bdev, unsigned int type)
>>> ttm_set_driver_manager(bdev, type, NULL);
>>> mutex_lock(&bman->lock);
>>> - drm_buddy_free_list(mm, &bman->reserved);
>>> + drm_buddy_free_list(mm, &bman->reserved, 0);
>>> drm_buddy_fini(mm);
>>> bman->visible_avail += bman->visible_reserved;
>>> WARN_ON_ONCE(bman->visible_avail != bman->visible_size);
>>> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c
>>> b/drivers/gpu/drm/tests/drm_buddy_test.c
>>> index 2f32fb2f12e7..454ad9952f56 100644
>>> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
>>> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
>>> @@ -64,7 +64,7 @@ static void drm_test_buddy_alloc_contiguous(struct
>>> kunit *test)
>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>> "buddy_alloc didn't error size=%u\n", 3 * ps);
>>> - drm_buddy_free_list(&mm, &middle);
>>> + drm_buddy_free_list(&mm, &middle, 0);
>>> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>> mm_size,
>>> 3 * ps, ps, &allocated,
>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>> @@ -74,7 +74,7 @@ static void drm_test_buddy_alloc_contiguous(struct
>>> kunit *test)
>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>> "buddy_alloc didn't error size=%u\n", 2 * ps);
>>> - drm_buddy_free_list(&mm, &right);
>>> + drm_buddy_free_list(&mm, &right, 0);
>>> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>> mm_size,
>>> 3 * ps, ps, &allocated,
>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>> @@ -89,7 +89,7 @@ static void drm_test_buddy_alloc_contiguous(struct
>>> kunit *test)
>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>> "buddy_alloc hit an error size=%u\n", 2 * ps);
>>> - drm_buddy_free_list(&mm, &left);
>>> + drm_buddy_free_list(&mm, &left, 0);
>>> KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>> mm_size,
>>> 3 * ps, ps, &allocated,
>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>> @@ -101,7 +101,7 @@ static void
>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>> KUNIT_ASSERT_EQ(test, total, ps * 2 + ps * 3);
>>> - drm_buddy_free_list(&mm, &allocated);
>>> + drm_buddy_free_list(&mm, &allocated, 0);
>>> drm_buddy_fini(&mm);
>>> }
>>> @@ -170,7 +170,7 @@ static void
>>> drm_test_buddy_alloc_pathological(struct kunit *test)
>>> top, max_order);
>>> }
>>> - drm_buddy_free_list(&mm, &holes);
>>> + drm_buddy_free_list(&mm, &holes, 0);
>>> /* Nothing larger than blocks of chunk_size now available */
>>> for (order = 1; order <= max_order; order++) {
>>> @@ -182,7 +182,7 @@ static void
>>> drm_test_buddy_alloc_pathological(struct kunit *test)
>>> }
>>> list_splice_tail(&holes, &blocks);
>>> - drm_buddy_free_list(&mm, &blocks);
>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>> drm_buddy_fini(&mm);
>>> }
>>> @@ -277,7 +277,7 @@ static void
>>> drm_test_buddy_alloc_pessimistic(struct kunit *test)
>>> list_del(&block->link);
>>> drm_buddy_free_block(&mm, block);
>>> - drm_buddy_free_list(&mm, &blocks);
>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>> drm_buddy_fini(&mm);
>>> }
>>> @@ -323,7 +323,7 @@ static void
>>> drm_test_buddy_alloc_optimistic(struct kunit *test)
>>> size, size, &tmp, flags),
>>> "buddy_alloc unexpectedly succeeded, it
>>> should be full!");
>>> - drm_buddy_free_list(&mm, &blocks);
>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>> drm_buddy_fini(&mm);
>>> }
>>> @@ -358,7 +358,7 @@ static void drm_test_buddy_alloc_limit(struct
>>> kunit *test)
>>> drm_buddy_block_size(&mm, block),
>>> BIT_ULL(mm.max_order) * PAGE_SIZE);
>>> - drm_buddy_free_list(&mm, &allocated);
>>> + drm_buddy_free_list(&mm, &allocated, 0);
>>> drm_buddy_fini(&mm);
>>> }
>>> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>> b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>> index 115ec745e502..1ad678b62c4a 100644
>>> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>> @@ -196,7 +196,7 @@ static int xe_ttm_vram_mgr_new(struct
>>> ttm_resource_manager *man,
>>> return 0;
>>> error_free_blocks:
>>> - drm_buddy_free_list(mm, &vres->blocks);
>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>> mutex_unlock(&mgr->lock);
>>> error_fini:
>>> ttm_resource_fini(man, &vres->base);
>>> @@ -214,7 +214,7 @@ static void xe_ttm_vram_mgr_del(struct
>>> ttm_resource_manager *man,
>>> struct drm_buddy *mm = &mgr->mm;
>>> mutex_lock(&mgr->lock);
>>> - drm_buddy_free_list(mm, &vres->blocks);
>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>> mgr->visible_avail += vres->used_visible_size;
>>> mutex_unlock(&mgr->lock);
>>> diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h
>>> index a5b39fc01003..82570f77e817 100644
>>> --- a/include/drm/drm_buddy.h
>>> +++ b/include/drm/drm_buddy.h
>>> @@ -25,6 +25,8 @@
>>> #define DRM_BUDDY_RANGE_ALLOCATION BIT(0)
>>> #define DRM_BUDDY_TOPDOWN_ALLOCATION BIT(1)
>>> #define DRM_BUDDY_CONTIGUOUS_ALLOCATION BIT(2)
>>> +#define DRM_BUDDY_CLEAR_ALLOCATION BIT(3)
>>> +#define DRM_BUDDY_CLEARED BIT(4)
>>> struct drm_buddy_block {
>>> #define DRM_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
>>> @@ -32,8 +34,9 @@ struct drm_buddy_block {
>>> #define DRM_BUDDY_ALLOCATED (1 << 10)
>>> #define DRM_BUDDY_FREE (2 << 10)
>>> #define DRM_BUDDY_SPLIT (3 << 10)
>>> +#define DRM_BUDDY_HEADER_CLEAR GENMASK_ULL(9, 9)
>>> /* Free to be used, if needed in the future */
>>> -#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
>>> +#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(8, 6)
>>> #define DRM_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
>>> u64 header;
>>> @@ -86,6 +89,7 @@ struct drm_buddy {
>>> u64 chunk_size;
>>> u64 size;
>>> u64 avail;
>>> + u64 clear_avail;
>>> };
>>> static inline u64
>>> @@ -112,6 +116,12 @@ drm_buddy_block_is_allocated(struct
>>> drm_buddy_block *block)
>>> return drm_buddy_block_state(block) == DRM_BUDDY_ALLOCATED;
>>> }
>>> +static inline bool
>>> +drm_buddy_block_is_clear(struct drm_buddy_block *block)
>>> +{
>>> + return block->header & DRM_BUDDY_HEADER_CLEAR;
>>> +}
>>> +
>>> static inline bool
>>> drm_buddy_block_is_free(struct drm_buddy_block *block)
>>> {
>>> @@ -150,7 +160,9 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>> void drm_buddy_free_block(struct drm_buddy *mm, struct
>>> drm_buddy_block *block);
>>> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head
>>> *objects);
>>> +void drm_buddy_free_list(struct drm_buddy *mm,
>>> + struct list_head *objects,
>>> + unsigned int flags);
>>> void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p);
>>> void drm_buddy_block_print(struct drm_buddy *mm,
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature
2024-03-28 16:48 ` Matthew Auld
@ 2024-04-01 11:07 ` Paneer Selvam, Arunpravin
2024-04-05 15:43 ` Matthew Auld
0 siblings, 1 reply; 21+ messages in thread
From: Paneer Selvam, Arunpravin @ 2024-04-01 11:07 UTC (permalink / raw)
To: Matthew Auld, dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, mario.limonciello, felix.kuehling
Hi Matthew,
On 3/28/2024 10:18 PM, Matthew Auld wrote:
> On 28/03/2024 16:07, Paneer Selvam, Arunpravin wrote:
>> Hi Matthew,
>>
>> On 3/26/2024 11:39 PM, Matthew Auld wrote:
>>> On 18/03/2024 21:40, Arunpravin Paneer Selvam wrote:
>>>> - Add tracking clear page feature.
>>>>
>>>> - Driver should enable the DRM_BUDDY_CLEARED flag if it
>>>> successfully clears the blocks in the free path. On the otherhand,
>>>> DRM buddy marks each block as cleared.
>>>>
>>>> - Track the available cleared pages size
>>>>
>>>> - If driver requests cleared memory we prefer cleared memory
>>>> but fallback to uncleared if we can't find the cleared blocks.
>>>> when driver requests uncleared memory we try to use uncleared but
>>>> fallback to cleared memory if necessary.
>>>>
>>>> - When a block gets freed we clear it and mark the freed block as
>>>> cleared,
>>>> when there are buddies which are cleared as well we can merge them.
>>>> Otherwise, we prefer to keep the blocks as separated.
>>>>
>>>> - Add a function to support defragmentation.
>>>>
>>>> v1:
>>>> - Depends on the flag check DRM_BUDDY_CLEARED, enable the block as
>>>> cleared. Else, reset the clear flag for each block in the
>>>> list(Christian)
>>>> - For merging the 2 cleared blocks compare as below,
>>>> drm_buddy_is_clear(block) != drm_buddy_is_clear(buddy)(Christian)
>>>> - Defragment the memory beginning from min_order
>>>> till the required memory space is available.
>>>>
>>>> v2: (Matthew)
>>>> - Add a wrapper drm_buddy_free_list_internal for the freeing of
>>>> blocks
>>>> operation within drm buddy.
>>>> - Write a macro block_incompatible() to allocate the required
>>>> blocks.
>>>> - Update the xe driver for the drm_buddy_free_list change in
>>>> arguments.
>>>> - add a warning if the two blocks are incompatible on
>>>> defragmentation
>>>> - call full defragmentation in the fini() function
>>>> - place a condition to test if min_order is equal to 0
>>>> - replace the list with safe_reverse() variant as we might
>>>> remove the block from the list.
>>>>
>>>> v3:
>>>> - fix Gitlab user reported lockup issue.
>>>> - Keep DRM_BUDDY_HEADER_CLEAR define sorted(Matthew)
>>>> - modify to pass the root order instead max_order in fini()
>>>> function(Matthew)
>>>> - change bool 1 to true(Matthew)
>>>> - add check if min_block_size is power of 2(Matthew)
>>>> - modify the min_block_size datatype to u64(Matthew)
>>>>
>>>> v4:
>>>> - rename the function drm_buddy_defrag with __force_merge.
>>>> - Include __force_merge directly in drm buddy file and remove
>>>> the defrag use in amdgpu driver.
>>>> - Remove list_empty() check(Matthew)
>>>> - Remove unnecessary space, headers and placement of new
>>>> variables(Matthew)
>>>> - Add a unit test case(Matthew)
>>>>
>>>> Signed-off-by: Arunpravin Paneer Selvam
>>>> <Arunpravin.PaneerSelvam@amd.com>
>>>> Signed-off-by: Matthew Auld <matthew.auld@intel.com>
>>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>>> Suggested-by: Matthew Auld <matthew.auld@intel.com>
>>>> ---
>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>>>> drivers/gpu/drm/drm_buddy.c | 427
>>>> ++++++++++++++----
>>>> drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 6 +-
>>>> drivers/gpu/drm/tests/drm_buddy_test.c | 18 +-
>>>> drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 4 +-
>>>> include/drm/drm_buddy.h | 16 +-
>>>> 6 files changed, 360 insertions(+), 117 deletions(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>> index 8db880244324..c0c851409241 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>> @@ -571,7 +571,7 @@ static int amdgpu_vram_mgr_new(struct
>>>> ttm_resource_manager *man,
>>>> return 0;
>>>> error_free_blocks:
>>>> - drm_buddy_free_list(mm, &vres->blocks);
>>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>>> mutex_unlock(&mgr->lock);
>>>> error_fini:
>>>> ttm_resource_fini(man, &vres->base);
>>>> @@ -604,7 +604,7 @@ static void amdgpu_vram_mgr_del(struct
>>>> ttm_resource_manager *man,
>>>> amdgpu_vram_mgr_do_reserve(man);
>>>> - drm_buddy_free_list(mm, &vres->blocks);
>>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>>> mutex_unlock(&mgr->lock);
>>>> atomic64_sub(vis_usage, &mgr->vis_usage);
>>>> @@ -912,7 +912,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device
>>>> *adev)
>>>> kfree(rsv);
>>>> list_for_each_entry_safe(rsv, temp, &mgr->reserved_pages,
>>>> blocks) {
>>>> - drm_buddy_free_list(&mgr->mm, &rsv->allocated);
>>>> + drm_buddy_free_list(&mgr->mm, &rsv->allocated, 0);
>>>> kfree(rsv);
>>>> }
>>>> if (!adev->gmc.is_app_apu)
>>>> diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c
>>>> index c4222b886db7..625a30a6b855 100644
>>>> --- a/drivers/gpu/drm/drm_buddy.c
>>>> +++ b/drivers/gpu/drm/drm_buddy.c
>>>> @@ -38,8 +38,8 @@ static void drm_block_free(struct drm_buddy *mm,
>>>> kmem_cache_free(slab_blocks, block);
>>>> }
>>>> -static void list_insert_sorted(struct drm_buddy *mm,
>>>> - struct drm_buddy_block *block)
>>>> +static void list_insert(struct drm_buddy *mm,
>>>> + struct drm_buddy_block *block)
>>>> {
>>>> struct drm_buddy_block *node;
>>>> struct list_head *head;
>>>> @@ -57,6 +57,16 @@ static void list_insert_sorted(struct drm_buddy
>>>> *mm,
>>>> __list_add(&block->link, node->link.prev, &node->link);
>>>> }
>>>> +static void clear_reset(struct drm_buddy_block *block)
>>>> +{
>>>> + block->header &= ~DRM_BUDDY_HEADER_CLEAR;
>>>> +}
>>>> +
>>>> +static void mark_cleared(struct drm_buddy_block *block)
>>>> +{
>>>> + block->header |= DRM_BUDDY_HEADER_CLEAR;
>>>> +}
>>>> +
>>>> static void mark_allocated(struct drm_buddy_block *block)
>>>> {
>>>> block->header &= ~DRM_BUDDY_HEADER_STATE;
>>>> @@ -71,7 +81,7 @@ static void mark_free(struct drm_buddy *mm,
>>>> block->header &= ~DRM_BUDDY_HEADER_STATE;
>>>> block->header |= DRM_BUDDY_FREE;
>>>> - list_insert_sorted(mm, block);
>>>> + list_insert(mm, block);
>>>> }
>>>> static void mark_split(struct drm_buddy_block *block)
>>>> @@ -82,6 +92,114 @@ static void mark_split(struct drm_buddy_block
>>>> *block)
>>>> list_del(&block->link);
>>>> }
>>>> +static struct drm_buddy_block *
>>>> +__get_buddy(struct drm_buddy_block *block)
>>>> +{
>>>> + struct drm_buddy_block *parent;
>>>> +
>>>> + parent = block->parent;
>>>> + if (!parent)
>>>> + return NULL;
>>>> +
>>>> + if (parent->left == block)
>>>> + return parent->right;
>>>> +
>>>> + return parent->left;
>>>> +}
>>>> +
>>>> +static unsigned int __drm_buddy_free(struct drm_buddy *mm,
>>>> + struct drm_buddy_block *block,
>>>> + bool force_merge)
>>>> +{
>>>> + struct drm_buddy_block *parent;
>>>> + unsigned int order;
>>>> +
>>>> + while ((parent = block->parent)) {
>>>> + struct drm_buddy_block *buddy;
>>>> +
>>>> + buddy = __get_buddy(block);
>>>> +
>>>> + if (!drm_buddy_block_is_free(buddy))
>>>> + break;
>>>> +
>>>> + if (!force_merge) {
>>>> + /**
>>>
>>> Not really valid kernel-doc AFAIK. I think drop the extra *. Below
>>> also.
>>>
>>>> + * Check the block and its buddy clear state and exit
>>>> + * the loop if they both have the dissimilar state.
>>>> + */
>>>> + if (drm_buddy_block_is_clear(block) !=
>>>> + drm_buddy_block_is_clear(buddy))
>>>> + break;
>>>> +
>>>> + if (drm_buddy_block_is_clear(block))
>>>> + mark_cleared(parent);
>>>> + }
>>>> +
>>>> + list_del(&buddy->link);
>>>> + if (force_merge && drm_buddy_block_is_clear(buddy))
>>>> + mm->clear_avail -= drm_buddy_block_size(mm, buddy);
>>>> +
>>>> + drm_block_free(mm, block);
>>>> + drm_block_free(mm, buddy);
>>>> +
>>>> + block = parent;
>>>> + }
>>>> +
>>>> + order = drm_buddy_block_order(block);
>>>> + mark_free(mm, block);
>>>> +
>>>> + return order;
>>>> +}
>>>> +
>>>> +static int __force_merge(struct drm_buddy *mm,
>>>> + unsigned int min_order)
>>>> +{
>>>> + unsigned int order;
>>>> + int i;
>>>> +
>>>> + if (!min_order)
>>>> + return -ENOMEM;
>>>> +
>>>> + if (min_order > mm->max_order)
>>>> + return -EINVAL;
>>>> +
>>>> + for (i = min_order - 1; i >= 0; i--) {
>>>> + struct drm_buddy_block *block, *prev;
>>>> +
>>>> + list_for_each_entry_safe_reverse(block, prev,
>>>> &mm->free_list[i], link) {
>>>> + struct drm_buddy_block *buddy;
>>>> +
>>>> + if (!block->parent)
>>>> + continue;
>>>> +
>>>> + buddy = __get_buddy(block);
>>>> + if (!drm_buddy_block_is_free(buddy))
>>>> + continue;
>>>> +
>>>> + WARN_ON(drm_buddy_block_is_clear(block) ==
>>>> + drm_buddy_block_is_clear(buddy));
>>>> +
>>>> + /**
>>>> + * If the prev block is same as buddy, don't access the
>>>> + * block in the next iteration as we would free the
>>>> + * buddy block as part of the free function.
>>>> + */
>>>> + if (prev == buddy)
>>>> + prev = list_prev_entry(prev, link);
>>>> +
>>>> + list_del(&block->link);
>>>> + if (drm_buddy_block_is_clear(block))
>>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>>> +
>>>> + order = __drm_buddy_free(mm, block, true);
>>>> + if (order >= min_order)
>>>> + return 0;
>>>> + }
>>>> + }
>>>> +
>>>> + return -ENOMEM;
>>>> +}
>>>> +
>>>> /**
>>>> * drm_buddy_init - init memory manager
>>>> *
>>>> @@ -137,7 +255,7 @@ int drm_buddy_init(struct drm_buddy *mm, u64
>>>> size, u64 chunk_size)
>>>> offset = 0;
>>>> i = 0;
>>>> - /*
>>>> + /**
>>>> * Split into power-of-two blocks, in case we are given a
>>>> size that is
>>>> * not itself a power-of-two.
>>>> */
>>>> @@ -186,11 +304,21 @@ EXPORT_SYMBOL(drm_buddy_init);
>>>> */
>>>> void drm_buddy_fini(struct drm_buddy *mm)
>>>> {
>>>> + u64 root_size, size;
>>>> + unsigned int order;
>>>> int i;
>>>> + size = mm->size;
>>>> +
>>>> for (i = 0; i < mm->n_roots; ++i) {
>>>> + order = ilog2(size) - ilog2(mm->chunk_size);
>>>> + __force_merge(mm, order);
>>>> +
>>>> WARN_ON(!drm_buddy_block_is_free(mm->roots[i]));
>>>> drm_block_free(mm, mm->roots[i]);
>>>> +
>>>> + root_size = mm->chunk_size << order;
>>>> + size -= root_size;
>>>> }
>>>> WARN_ON(mm->avail != mm->size);
>>>> @@ -223,26 +351,17 @@ static int split_block(struct drm_buddy *mm,
>>>> mark_free(mm, block->left);
>>>> mark_free(mm, block->right);
>>>> + if (drm_buddy_block_is_clear(block)) {
>>>> + mark_cleared(block->left);
>>>> + mark_cleared(block->right);
>>>> + clear_reset(block);
>>>> + }
>>>> +
>>>> mark_split(block);
>>>> return 0;
>>>> }
>>>> -static struct drm_buddy_block *
>>>> -__get_buddy(struct drm_buddy_block *block)
>>>> -{
>>>> - struct drm_buddy_block *parent;
>>>> -
>>>> - parent = block->parent;
>>>> - if (!parent)
>>>> - return NULL;
>>>> -
>>>> - if (parent->left == block)
>>>> - return parent->right;
>>>> -
>>>> - return parent->left;
>>>> -}
>>>> -
>>>> /**
>>>> * drm_get_buddy - get buddy address
>>>> *
>>>> @@ -260,30 +379,6 @@ drm_get_buddy(struct drm_buddy_block *block)
>>>> }
>>>> EXPORT_SYMBOL(drm_get_buddy);
>>>> -static void __drm_buddy_free(struct drm_buddy *mm,
>>>> - struct drm_buddy_block *block)
>>>> -{
>>>> - struct drm_buddy_block *parent;
>>>> -
>>>> - while ((parent = block->parent)) {
>>>> - struct drm_buddy_block *buddy;
>>>> -
>>>> - buddy = __get_buddy(block);
>>>> -
>>>> - if (!drm_buddy_block_is_free(buddy))
>>>> - break;
>>>> -
>>>> - list_del(&buddy->link);
>>>> -
>>>> - drm_block_free(mm, block);
>>>> - drm_block_free(mm, buddy);
>>>> -
>>>> - block = parent;
>>>> - }
>>>> -
>>>> - mark_free(mm, block);
>>>> -}
>>>> -
>>>> /**
>>>> * drm_buddy_free_block - free a block
>>>> *
>>>> @@ -295,26 +390,59 @@ void drm_buddy_free_block(struct drm_buddy *mm,
>>>> {
>>>> BUG_ON(!drm_buddy_block_is_allocated(block));
>>>> mm->avail += drm_buddy_block_size(mm, block);
>>>> - __drm_buddy_free(mm, block);
>>>> + if (drm_buddy_block_is_clear(block))
>>>> + mm->clear_avail += drm_buddy_block_size(mm, block);
>>>> +
>>>> + __drm_buddy_free(mm, block, false);
>>>> }
>>>> EXPORT_SYMBOL(drm_buddy_free_block);
>>>> -/**
>>>> - * drm_buddy_free_list - free blocks
>>>> - *
>>>> - * @mm: DRM buddy manager
>>>> - * @objects: input list head to free blocks
>>>> - */
>>>> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head
>>>> *objects)
>>>> +static void __drm_buddy_free_list(struct drm_buddy *mm,
>>>> + struct list_head *objects,
>>>> + bool mark_clear,
>>>> + bool mark_dirty)
>>>> {
>>>> struct drm_buddy_block *block, *on;
>>>> + WARN_ON(mark_dirty && mark_clear);
>>>> +
>>>> list_for_each_entry_safe(block, on, objects, link) {
>>>> + if (mark_clear)
>>>> + mark_cleared(block);
>>>> + else if (mark_dirty)
>>>> + clear_reset(block);
>>>> drm_buddy_free_block(mm, block);
>>>> cond_resched();
>>>> }
>>>> INIT_LIST_HEAD(objects);
>>>> }
>>>> +
>>>> +static void drm_buddy_free_list_internal(struct drm_buddy *mm,
>>>> + struct list_head *objects)
>>>> +{
>>>> + /**
>>>> + * Don't touch the clear/dirty bit, since allocation is still
>>>> internal
>>>> + * at this point. For example we might have just failed part
>>>> of the
>>>> + * allocation.
>>>> + */
>>>> + __drm_buddy_free_list(mm, objects, false, false);
>>>> +}
>>>> +
>>>> +/**
>>>> + * drm_buddy_free_list - free blocks
>>>> + *
>>>> + * @mm: DRM buddy manager
>>>> + * @objects: input list head to free blocks
>>>> + * @flags: optional flags like DRM_BUDDY_CLEARED
>>>> + */
>>>> +void drm_buddy_free_list(struct drm_buddy *mm,
>>>> + struct list_head *objects,
>>>> + unsigned int flags)
>>>> +{
>>>> + bool mark_clear = flags & DRM_BUDDY_CLEARED;
>>>> +
>>>> + __drm_buddy_free_list(mm, objects, mark_clear, !mark_clear);
>>>> +}
>>>> EXPORT_SYMBOL(drm_buddy_free_list);
>>>> static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
>>>> @@ -327,10 +455,19 @@ static inline bool contains(u64 s1, u64 e1,
>>>> u64 s2, u64 e2)
>>>> return s1 <= s2 && e1 >= e2;
>>>> }
>>>> +static bool block_incompatible(struct drm_buddy_block *block,
>>>> unsigned int flags)
>>>> +{
>>>> + bool needs_clear = flags & DRM_BUDDY_CLEAR_ALLOCATION;
>>>> +
>>>> + return needs_clear != drm_buddy_block_is_clear(block);
>>>> +}
>>>> +
>>>> static struct drm_buddy_block *
>>>> -alloc_range_bias(struct drm_buddy *mm,
>>>> - u64 start, u64 end,
>>>> - unsigned int order)
>>>> +__alloc_range_bias(struct drm_buddy *mm,
>>>> + u64 start, u64 end,
>>>> + unsigned int order,
>>>> + unsigned long flags,
>>>> + bool fallback)
>>>> {
>>>> struct drm_buddy_block *block;
>>>> struct drm_buddy_block *buddy;
>>>> @@ -369,7 +506,10 @@ alloc_range_bias(struct drm_buddy *mm,
>>>> if (contains(start, end, block_start, block_end) &&
>>>> order == drm_buddy_block_order(block)) {
>>>> - /*
>>>> + if (!fallback && block_incompatible(block, flags))
>>>> + continue;
>>>> +
>>>> + /**
>>>> * Find the free block within the range.
>>>> */
>>>> if (drm_buddy_block_is_free(block))
>>>> @@ -391,7 +531,7 @@ alloc_range_bias(struct drm_buddy *mm,
>>>> return ERR_PTR(-ENOSPC);
>>>> err_undo:
>>>> - /*
>>>> + /**
>>>> * We really don't want to leave around a bunch of split
>>>> blocks, since
>>>> * bigger is better, so make sure we merge everything back
>>>> before we
>>>> * free the allocated blocks.
>>>> @@ -400,30 +540,57 @@ alloc_range_bias(struct drm_buddy *mm,
>>>> if (buddy &&
>>>> (drm_buddy_block_is_free(block) &&
>>>> drm_buddy_block_is_free(buddy)))
>>>> - __drm_buddy_free(mm, block);
>>>> + __drm_buddy_free(mm, block, false);
>>>> return ERR_PTR(err);
>>>> }
>>>> static struct drm_buddy_block *
>>>> -get_maxblock(struct drm_buddy *mm, unsigned int order)
>>>> +__drm_buddy_alloc_range_bias(struct drm_buddy *mm,
>>>> + u64 start, u64 end,
>>>> + unsigned int order,
>>>> + unsigned long flags)
>>>> +{
>>>> + struct drm_buddy_block *block;
>>>> + bool fallback = false;
>>>> +
>>>> + block = __alloc_range_bias(mm, start, end, order,
>>>> + flags, fallback);
>>>> + if (IS_ERR(block) && mm->clear_avail)
>>>> + return __alloc_range_bias(mm, start, end, order,
>>>> + flags, !fallback);
>>>> +
>>>> + return block;
>>>> +}
>>>> +
>>>> +static struct drm_buddy_block *
>>>> +get_maxblock(struct drm_buddy *mm, unsigned int order,
>>>> + unsigned long flags)
>>>> {
>>>> - struct drm_buddy_block *max_block = NULL, *node;
>>>> + struct drm_buddy_block *max_block = NULL, *block = NULL;
>>>> unsigned int i;
>>>> for (i = order; i <= mm->max_order; ++i) {
>>>> - if (!list_empty(&mm->free_list[i])) {
>>>> - node = list_last_entry(&mm->free_list[i],
>>>> - struct drm_buddy_block,
>>>> - link);
>>>> - if (!max_block) {
>>>> - max_block = node;
>>>> + struct drm_buddy_block *tmp_block;
>>>> +
>>>> + list_for_each_entry_reverse(tmp_block, &mm->free_list[i],
>>>> link) {
>>>> + if (block_incompatible(tmp_block, flags))
>>>> continue;
>>>> - }
>>>> - if (drm_buddy_block_offset(node) >
>>>> - drm_buddy_block_offset(max_block)) {
>>>> - max_block = node;
>>>> - }
>>>> + block = tmp_block;
>>>> + break;
>>>> + }
>>>> +
>>>> + if (!block)
>>>> + continue;
>>>> +
>>>> + if (!max_block) {
>>>> + max_block = block;
>>>> + continue;
>>>> + }
>>>> +
>>>> + if (drm_buddy_block_offset(block) >
>>>> + drm_buddy_block_offset(max_block)) {
>>>> + max_block = block;
>>>> }
>>>> }
>>>> @@ -440,11 +607,29 @@ alloc_from_freelist(struct drm_buddy *mm,
>>>> int err;
>>>> if (flags & DRM_BUDDY_TOPDOWN_ALLOCATION) {
>>>> - block = get_maxblock(mm, order);
>>>> + block = get_maxblock(mm, order, flags);
>>>> if (block)
>>>> /* Store the obtained block order */
>>>> tmp = drm_buddy_block_order(block);
>>>> } else {
>>>> + for (tmp = order; tmp <= mm->max_order; ++tmp) {
>>>> + struct drm_buddy_block *tmp_block;
>>>> +
>>>> + list_for_each_entry_reverse(tmp_block,
>>>> &mm->free_list[tmp], link) {
>>>> + if (block_incompatible(tmp_block, flags))
>>>> + continue;
>>>> +
>>>> + block = tmp_block;
>>>> + break;
>>>> + }
>>>> +
>>>> + if (block)
>>>> + break;
>>>> + }
>>>> + }
>>>> +
>>>> + if (!block) {
>>>> + /* Fallback method */
>>>> for (tmp = order; tmp <= mm->max_order; ++tmp) {
>>>> if (!list_empty(&mm->free_list[tmp])) {
>>>> block = list_last_entry(&mm->free_list[tmp],
>>>> @@ -454,10 +639,10 @@ alloc_from_freelist(struct drm_buddy *mm,
>>>> break;
>>>> }
>>>> }
>>>> - }
>>>> - if (!block)
>>>> - return ERR_PTR(-ENOSPC);
>>>> + if (!block)
>>>> + return ERR_PTR(-ENOSPC);
>>>> + }
>>>> BUG_ON(!drm_buddy_block_is_free(block));
>>>> @@ -473,7 +658,7 @@ alloc_from_freelist(struct drm_buddy *mm,
>>>> err_undo:
>>>> if (tmp != order)
>>>> - __drm_buddy_free(mm, block);
>>>> + __drm_buddy_free(mm, block, false);
>>>> return ERR_PTR(err);
>>>> }
>>>> @@ -524,6 +709,8 @@ static int __alloc_range(struct drm_buddy *mm,
>>>> mark_allocated(block);
>>>> total_allocated += drm_buddy_block_size(mm, block);
>>>> mm->avail -= drm_buddy_block_size(mm, block);
>>>> + if (drm_buddy_block_is_clear(block))
>>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>>> list_add_tail(&block->link, &allocated);
>>>> continue;
>>>> }
>>>> @@ -548,7 +735,7 @@ static int __alloc_range(struct drm_buddy *mm,
>>>> return 0;
>>>> err_undo:
>>>> - /*
>>>> + /**
>>>> * We really don't want to leave around a bunch of split
>>>> blocks, since
>>>> * bigger is better, so make sure we merge everything back
>>>> before we
>>>> * free the allocated blocks.
>>>> @@ -557,14 +744,14 @@ static int __alloc_range(struct drm_buddy *mm,
>>>> if (buddy &&
>>>> (drm_buddy_block_is_free(block) &&
>>>> drm_buddy_block_is_free(buddy)))
>>>> - __drm_buddy_free(mm, block);
>>>> + __drm_buddy_free(mm, block, false);
>>>> err_free:
>>>> if (err == -ENOSPC && total_allocated_on_err) {
>>>> list_splice_tail(&allocated, blocks);
>>>> *total_allocated_on_err = total_allocated;
>>>> } else {
>>>> - drm_buddy_free_list(mm, &allocated);
>>>> + drm_buddy_free_list_internal(mm, &allocated);
>>>> }
>>>> return err;
>>>> @@ -630,11 +817,11 @@ static int __alloc_contig_try_harder(struct
>>>> drm_buddy *mm,
>>>> list_splice(&blocks_lhs, blocks);
>>>> return 0;
>>>> } else if (err != -ENOSPC) {
>>>> - drm_buddy_free_list(mm, blocks);
>>>> + drm_buddy_free_list_internal(mm, blocks);
>>>> return err;
>>>> }
>>>> /* Free blocks for the next iteration */
>>>> - drm_buddy_free_list(mm, blocks);
>>>> + drm_buddy_free_list_internal(mm, blocks);
>>>> }
>>>> return -ENOSPC;
>>>> @@ -690,6 +877,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>>> list_del(&block->link);
>>>> mark_free(mm, block);
>>>> mm->avail += drm_buddy_block_size(mm, block);
>>>> + if (drm_buddy_block_is_clear(block))
>>>> + mm->clear_avail += drm_buddy_block_size(mm, block);
>>>> /* Prevent recursively freeing this node */
>>>> parent = block->parent;
>>>> @@ -701,6 +890,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>>> if (err) {
>>>> mark_allocated(block);
>>>> mm->avail -= drm_buddy_block_size(mm, block);
>>>> + if (drm_buddy_block_is_clear(block))
>>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>>> list_add(&block->link, blocks);
>>>> }
>>>> @@ -709,13 +900,28 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>>> }
>>>> EXPORT_SYMBOL(drm_buddy_block_trim);
>>>> +static struct drm_buddy_block *
>>>> +__drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>>> + u64 start, u64 end,
>>>> + unsigned int order,
>>>> + unsigned long flags)
>>>> +{
>>>> + if (flags & DRM_BUDDY_RANGE_ALLOCATION)
>>>> + /* Allocate traversing within the range */
>>>> + return __drm_buddy_alloc_range_bias(mm, start, end,
>>>> + order, flags);
>>>> + else
>>>> + /* Allocate from freelist */
>>>> + return alloc_from_freelist(mm, order, flags);
>>>> +}
>>>> +
>>>> /**
>>>> * drm_buddy_alloc_blocks - allocate power-of-two blocks
>>>> *
>>>> * @mm: DRM buddy manager to allocate from
>>>> * @start: start of the allowed range for this block
>>>> * @end: end of the allowed range for this block
>>>> - * @size: size of the allocation
>>>> + * @size: size of the allocation in bytes
>>>> * @min_block_size: alignment of the allocation
>>>> * @blocks: output list head to add allocated blocks
>>>> * @flags: DRM_BUDDY_*_ALLOCATION flags
>>>> @@ -761,8 +967,18 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>>> return -EINVAL;
>>>> /* Actual range allocation */
>>>> - if (start + size == end)
>>>> - return __drm_buddy_alloc_range(mm, start, size, NULL,
>>>> blocks);
>>>> + if (start + size == end) {
>>>> + err = __drm_buddy_alloc_range(mm, start, size, NULL,
>>>> blocks);
>>>> + if (err) {
>>>> + order = ilog2(size) - ilog2(mm->chunk_size);
>>>> + if (mm->clear_avail && !__force_merge(mm, order))
>>>> + return __drm_buddy_alloc_range(mm, start, size,
>>>> NULL, blocks);
>>>
>>> That seems strange at a glance. With an actual range allocation like
>>> with intitial fb or whatever it should just find all overlapping
>>> pages, splitting down where needed on the edges. Not sure why
>>> force_merge would factor in here?
>> In case of not merged (fragmentation due to dirty and clear pages
>> split), if the memory request goes into that range, we might see the
>> split blocks as not free and end up returning -ENOSPC.
>> I found this problem when I tried to allocate the max_order where the
>> start + size == end and it ends up entering into this code block and
>> it returns -ENOSPC as there is no force_merge call
>> and the requested range max order block seen as not free. And we have
>> an another issue though if we use force_merge to defragment a
>> specific ordered block, there is a less probability that
>> we merge back the required blocks in the specific range. For now, we
>> can go ahead with no force merge support for actual range allocation
>> if we are sure that we use this only for initial fb etc.
>
> Ah right, now I see. But AFAICT trouble is if we say allocate 8K range
> at some specific offset, like above, which fails due to clear/dirty
> split then calling force_merge(order(8K)) is not always going to help.
> It will just force merge enough for 8K, but that might not be the 8K
> at the offset we were looking for, so it still fails. I think perhaps
> update the dfs in alloc_range to not bail when contains && split. Or I
> think other option is to force merge the entire address space on err.
> Other ideas?
For actual range allocation, if the min_block_size == mm->chunk_size, I
think we can proceed with your first idea (i.e) update the dfs in
alloc_range to not bail when contains && split.
For bias range allocation, we need to add range support to the
force_merge, for instance we can just continue in the force_merge for
loop if the contains(start, end, block_start, block_end) returns false.
Thoughts?
Thanks,
Arun.
>
>>
>> If there are no major concerns, can we push these patches?
>>
>> Regards,
>> Arun.
>>>> +
>>>> + return err;
>>>> + }
>>>> +
>>>> + return err;
>>>> + }
>>>> original_size = size;
>>>> original_min_size = min_block_size;
>>>> @@ -786,23 +1002,34 @@ int drm_buddy_alloc_blocks(struct drm_buddy
>>>> *mm,
>>>> BUG_ON(order < min_order);
>>>> do {
>>>> - if (flags & DRM_BUDDY_RANGE_ALLOCATION)
>>>> - /* Allocate traversing within the range */
>>>> - block = alloc_range_bias(mm, start, end, order);
>>>> - else
>>>> - /* Allocate from freelist */
>>>> - block = alloc_from_freelist(mm, order, flags);
>>>> -
>>>> + block = __drm_buddy_alloc_blocks(mm, start,
>>>> + end,
>>>> + order,
>>>> + flags);
>>>> if (!IS_ERR(block))
>>>> break;
>>>> if (order-- == min_order) {
>>>> + /**
>>>> + * Try allocation through force merge method
>>>> + */
>>>> + if (mm->clear_avail && !__force_merge(mm,
>>>> min_order)) {
>>>> + block = __drm_buddy_alloc_blocks(mm, start,
>>>> + end,
>>>> + min_order,
>>>> + flags);
>>>> + if (!IS_ERR(block)) {
>>>> + order = min_order;
>>>> + break;
>>>> + }
>>>> + }
>>>> +
>>>> + �� /**
>>>> + * Try contiguous block allocation through
>>>> + * try harder method.
>>>> + */
>>>> if (flags & DRM_BUDDY_CONTIGUOUS_ALLOCATION &&
>>>> !(flags & DRM_BUDDY_RANGE_ALLOCATION))
>>>> - /*
>>>> - * Try contiguous block allocation through
>>>> - * try harder method
>>>> - */
>>>> return __alloc_contig_try_harder(mm,
>>>> original_size,
>>>> original_min_size,
>>>> @@ -814,6 +1041,8 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>>> mark_allocated(block);
>>>> mm->avail -= drm_buddy_block_size(mm, block);
>>>> + if (drm_buddy_block_is_clear(block))
>>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>>> kmemleak_update_trace(block);
>>>> list_add_tail(&block->link, &allocated);
>>>> @@ -823,7 +1052,9 @@ int drm_buddy_alloc_blocks(struct drm_buddy
>>>> *mm,
>>>> break;
>>>> } while (1);
>>>> - /* Trim the allocated block to the required size */
>>>> + /**
>>>> + * Trim the allocated block to the required size
>>>> + */
>>>> if (original_size != size) {
>>>> struct list_head *trim_list;
>>>> LIST_HEAD(temp);
>>>> @@ -852,7 +1083,7 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>>> return 0;
>>>> err_free:
>>>> - drm_buddy_free_list(mm, &allocated);
>>>> + drm_buddy_free_list_internal(mm, &allocated);
>>>> return err;
>>>> }
>>>> EXPORT_SYMBOL(drm_buddy_alloc_blocks);
>>>> @@ -885,8 +1116,8 @@ void drm_buddy_print(struct drm_buddy *mm,
>>>> struct drm_printer *p)
>>>> {
>>>> int order;
>>>> - drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free:
>>>> %lluMiB\n",
>>>> - mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20);
>>>> + drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free:
>>>> %lluMiB, clear_free: %lluMiB\n",
>>>> + mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20,
>>>> mm->clear_avail >> 20);
>>>> for (order = mm->max_order; order >= 0; order--) {
>>>> struct drm_buddy_block *block;
>>>> diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>>> b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>>> index 0d735d5c2b35..942345548bc3 100644
>>>> --- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>>> +++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>>> @@ -126,7 +126,7 @@ static int i915_ttm_buddy_man_alloc(struct
>>>> ttm_resource_manager *man,
>>>> return 0;
>>>> err_free_blocks:
>>>> - drm_buddy_free_list(mm, &bman_res->blocks);
>>>> + drm_buddy_free_list(mm, &bman_res->blocks, 0);
>>>> mutex_unlock(&bman->lock);
>>>> err_free_res:
>>>> ttm_resource_fini(man, &bman_res->base);
>>>> @@ -141,7 +141,7 @@ static void i915_ttm_buddy_man_free(struct
>>>> ttm_resource_manager *man,
>>>> struct i915_ttm_buddy_manager *bman = to_buddy_manager(man);
>>>> mutex_lock(&bman->lock);
>>>> - drm_buddy_free_list(&bman->mm, &bman_res->blocks);
>>>> + drm_buddy_free_list(&bman->mm, &bman_res->blocks, 0);
>>>> bman->visible_avail += bman_res->used_visible_size;
>>>> mutex_unlock(&bman->lock);
>>>> @@ -345,7 +345,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device
>>>> *bdev, unsigned int type)
>>>> ttm_set_driver_manager(bdev, type, NULL);
>>>> mutex_lock(&bman->lock);
>>>> - drm_buddy_free_list(mm, &bman->reserved);
>>>> + drm_buddy_free_list(mm, &bman->reserved, 0);
>>>> drm_buddy_fini(mm);
>>>> bman->visible_avail += bman->visible_reserved;
>>>> WARN_ON_ONCE(bman->visible_avail != bman->visible_size);
>>>> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c
>>>> b/drivers/gpu/drm/tests/drm_buddy_test.c
>>>> index 2f32fb2f12e7..454ad9952f56 100644
>>>> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
>>>> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
>>>> @@ -64,7 +64,7 @@ static void
>>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>> "buddy_alloc didn't error size=%u\n", 3 * ps);
>>>> - drm_buddy_free_list(&mm, &middle);
>>>> + drm_buddy_free_list(&mm, &middle, 0);
>>>> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>>> mm_size,
>>>> 3 * ps, ps, &allocated,
>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>> @@ -74,7 +74,7 @@ static void
>>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>> "buddy_alloc didn't error size=%u\n", 2 * ps);
>>>> - drm_buddy_free_list(&mm, &right);
>>>> + drm_buddy_free_list(&mm, &right, 0);
>>>> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>>> mm_size,
>>>> 3 * ps, ps, &allocated,
>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>> @@ -89,7 +89,7 @@ static void
>>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>> "buddy_alloc hit an error size=%u\n", 2 * ps);
>>>> - drm_buddy_free_list(&mm, &left);
>>>> + drm_buddy_free_list(&mm, &left, 0);
>>>> KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>>> mm_size,
>>>> 3 * ps, ps, &allocated,
>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>> @@ -101,7 +101,7 @@ static void
>>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>>> KUNIT_ASSERT_EQ(test, total, ps * 2 + ps * 3);
>>>> - drm_buddy_free_list(&mm, &allocated);
>>>> + drm_buddy_free_list(&mm, &allocated, 0);
>>>> drm_buddy_fini(&mm);
>>>> }
>>>> @@ -170,7 +170,7 @@ static void
>>>> drm_test_buddy_alloc_pathological(struct kunit *test)
>>>> top, max_order);
>>>> }
>>>> - drm_buddy_free_list(&mm, &holes);
>>>> + drm_buddy_free_list(&mm, &holes, 0);
>>>> /* Nothing larger than blocks of chunk_size now available */
>>>> for (order = 1; order <= max_order; order++) {
>>>> @@ -182,7 +182,7 @@ static void
>>>> drm_test_buddy_alloc_pathological(struct kunit *test)
>>>> }
>>>> list_splice_tail(&holes, &blocks);
>>>> - drm_buddy_free_list(&mm, &blocks);
>>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>>> drm_buddy_fini(&mm);
>>>> }
>>>> @@ -277,7 +277,7 @@ static void
>>>> drm_test_buddy_alloc_pessimistic(struct kunit *test)
>>>> list_del(&block->link);
>>>> drm_buddy_free_block(&mm, block);
>>>> - drm_buddy_free_list(&mm, &blocks);
>>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>>> drm_buddy_fini(&mm);
>>>> }
>>>> @@ -323,7 +323,7 @@ static void
>>>> drm_test_buddy_alloc_optimistic(struct kunit *test)
>>>> size, size, &tmp, flags),
>>>> "buddy_alloc unexpectedly succeeded, it
>>>> should be full!");
>>>> - drm_buddy_free_list(&mm, &blocks);
>>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>>> drm_buddy_fini(&mm);
>>>> }
>>>> @@ -358,7 +358,7 @@ static void drm_test_buddy_alloc_limit(struct
>>>> kunit *test)
>>>> drm_buddy_block_size(&mm, block),
>>>> BIT_ULL(mm.max_order) * PAGE_SIZE);
>>>> - drm_buddy_free_list(&mm, &allocated);
>>>> + drm_buddy_free_list(&mm, &allocated, 0);
>>>> drm_buddy_fini(&mm);
>>>> }
>>>> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>>> b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>>> index 115ec745e502..1ad678b62c4a 100644
>>>> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>>> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>>> @@ -196,7 +196,7 @@ static int xe_ttm_vram_mgr_new(struct
>>>> ttm_resource_manager *man,
>>>> return 0;
>>>> error_free_blocks:
>>>> - drm_buddy_free_list(mm, &vres->blocks);
>>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>>> mutex_unlock(&mgr->lock);
>>>> error_fini:
>>>> ttm_resource_fini(man, &vres->base);
>>>> @@ -214,7 +214,7 @@ static void xe_ttm_vram_mgr_del(struct
>>>> ttm_resource_manager *man,
>>>> struct drm_buddy *mm = &mgr->mm;
>>>> mutex_lock(&mgr->lock);
>>>> - drm_buddy_free_list(mm, &vres->blocks);
>>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>>> mgr->visible_avail += vres->used_visible_size;
>>>> mutex_unlock(&mgr->lock);
>>>> diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h
>>>> index a5b39fc01003..82570f77e817 100644
>>>> --- a/include/drm/drm_buddy.h
>>>> +++ b/include/drm/drm_buddy.h
>>>> @@ -25,6 +25,8 @@
>>>> #define DRM_BUDDY_RANGE_ALLOCATION BIT(0)
>>>> #define DRM_BUDDY_TOPDOWN_ALLOCATION BIT(1)
>>>> #define DRM_BUDDY_CONTIGUOUS_ALLOCATION BIT(2)
>>>> +#define DRM_BUDDY_CLEAR_ALLOCATION BIT(3)
>>>> +#define DRM_BUDDY_CLEARED BIT(4)
>>>> struct drm_buddy_block {
>>>> #define DRM_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
>>>> @@ -32,8 +34,9 @@ struct drm_buddy_block {
>>>> #define DRM_BUDDY_ALLOCATED (1 << 10)
>>>> #define DRM_BUDDY_FREE (2 << 10)
>>>> #define DRM_BUDDY_SPLIT (3 << 10)
>>>> +#define DRM_BUDDY_HEADER_CLEAR GENMASK_ULL(9, 9)
>>>> /* Free to be used, if needed in the future */
>>>> -#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
>>>> +#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(8, 6)
>>>> #define DRM_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
>>>> u64 header;
>>>> @@ -86,6 +89,7 @@ struct drm_buddy {
>>>> u64 chunk_size;
>>>> u64 size;
>>>> u64 avail;
>>>> + u64 clear_avail;
>>>> };
>>>> static inline u64
>>>> @@ -112,6 +116,12 @@ drm_buddy_block_is_allocated(struct
>>>> drm_buddy_block *block)
>>>> return drm_buddy_block_state(block) == DRM_BUDDY_ALLOCATED;
>>>> }
>>>> +static inline bool
>>>> +drm_buddy_block_is_clear(struct drm_buddy_block *block)
>>>> +{
>>>> + return block->header & DRM_BUDDY_HEADER_CLEAR;
>>>> +}
>>>> +
>>>> static inline bool
>>>> drm_buddy_block_is_free(struct drm_buddy_block *block)
>>>> {
>>>> @@ -150,7 +160,9 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>>> void drm_buddy_free_block(struct drm_buddy *mm, struct
>>>> drm_buddy_block *block);
>>>> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head
>>>> *objects);
>>>> +void drm_buddy_free_list(struct drm_buddy *mm,
>>>> + struct list_head *objects,
>>>> + unsigned int flags);
>>>> void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p);
>>>> void drm_buddy_block_print(struct drm_buddy *mm,
>>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-03-26 14:53 ` Alex Deucher
2024-03-27 8:23 ` Paneer Selvam, Arunpravin
@ 2024-04-02 8:17 ` Christian König
2024-04-03 9:01 ` Michel Dänzer
1 sibling, 1 reply; 21+ messages in thread
From: Christian König @ 2024-04-02 8:17 UTC (permalink / raw)
To: Alex Deucher, Paneer Selvam, Arunpravin
Cc: dri-devel, amd-gfx, alexander.deucher, matthew.auld,
mario.limonciello, felix.kuehling
Am 26.03.24 um 15:53 schrieb Alex Deucher:
> On Tue, Mar 26, 2024 at 10:01 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>> On Tue, Mar 26, 2024 at 9:59 AM Paneer Selvam, Arunpravin
>> <arunpravin.paneerselvam@amd.com> wrote:
>>> Hi Alex,
>>>
>>> On 3/26/2024 7:08 PM, Alex Deucher wrote:
>>>> On Mon, Mar 18, 2024 at 5:47 PM Arunpravin Paneer Selvam
>>>> <Arunpravin.PaneerSelvam@amd.com> wrote:
>>>>> Add clear page support in vram memory region.
>>>>>
>>>>> v1(Christian):
>>>>> - Dont handle clear page as TTM flag since when moving the BO back
>>>>> in from GTT again we don't need that.
>>>>> - Make a specialized version of amdgpu_fill_buffer() which only
>>>>> clears the VRAM areas which are not already cleared
>>>>> - Drop the TTM_PL_FLAG_WIPE_ON_RELEASE check in
>>>>> amdgpu_object.c
>>>>>
>>>>> v2:
>>>>> - Modify the function name amdgpu_ttm_* (Alex)
>>>>> - Drop the delayed parameter (Christian)
>>>>> - handle amdgpu_res_cleared(&cursor) just above the size
>>>>> calculation (Christian)
>>>>> - Use AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE for clearing the buffers
>>>>> in the free path to properly wait for fences etc.. (Christian)
>>>>>
>>>>> v3(Christian):
>>>>> - Remove buffer clear code in VRAM manager instead change the
>>>>> AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE handling to set
>>>>> the DRM_BUDDY_CLEARED flag.
>>>>> - Remove ! from amdgpu_res_cleared(&cursor) check.
>>>>>
>>>>> Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
>>>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>>>> Acked-by: Felix Kuehling <felix.kuehling@amd.com>
>>>>> ---
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++---
>>>>> .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h | 25 ++++++++
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 61 ++++++++++++++++++-
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 5 +-
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h | 5 ++
>>>>> 6 files changed, 111 insertions(+), 13 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>>> index 8bc79924d171..c92d92b28a57 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
>>>>> @@ -39,6 +39,7 @@
>>>>> #include "amdgpu.h"
>>>>> #include "amdgpu_trace.h"
>>>>> #include "amdgpu_amdkfd.h"
>>>>> +#include "amdgpu_vram_mgr.h"
>>>>>
>>>>> /**
>>>>> * DOC: amdgpu_object
>>>>> @@ -601,8 +602,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>>>> if (!amdgpu_bo_support_uswc(bo->flags))
>>>>> bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
>>>>>
>>>>> - if (adev->ras_enabled)
>>>>> - bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>>>> + bo->flags |= AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
>>>>>
>>>>> bo->tbo.bdev = &adev->mman.bdev;
>>>>> if (bp->domain & (AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA |
>>>>> @@ -632,15 +632,17 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>>>>>
>>>>> if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED &&
>>>>> bo->tbo.resource->mem_type == TTM_PL_VRAM) {
>>>>> - struct dma_fence *fence;
>>>>> + struct dma_fence *fence = NULL;
>>>>>
>>>>> - r = amdgpu_fill_buffer(bo, 0, bo->tbo.base.resv, &fence, true);
>>>>> + r = amdgpu_ttm_clear_buffer(bo, bo->tbo.base.resv, &fence);
>>>>> if (unlikely(r))
>>>>> goto fail_unreserve;
>>>>>
>>>>> - dma_resv_add_fence(bo->tbo.base.resv, fence,
>>>>> - DMA_RESV_USAGE_KERNEL);
>>>>> - dma_fence_put(fence);
>>>>> + if (fence) {
>>>>> + dma_resv_add_fence(bo->tbo.base.resv, fence,
>>>>> + DMA_RESV_USAGE_KERNEL);
>>>>> + dma_fence_put(fence);
>>>>> + }
>>>>> }
>>>>> if (!bp->resv)
>>>>> amdgpu_bo_unreserve(bo);
>>>>> @@ -1365,8 +1367,12 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo)
>>>>> if (WARN_ON_ONCE(!dma_resv_trylock(bo->base.resv)))
>>>>> return;
>>>>>
>>>>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, bo->base.resv, &fence, true);
>>>>> + r = amdgpu_fill_buffer(abo, 0, bo->base.resv, &fence, true);
>>>>> if (!WARN_ON(r)) {
>>>>> + struct amdgpu_vram_mgr_resource *vres;
>>>>> +
>>>>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>>>>> + vres->flags |= DRM_BUDDY_CLEARED;
>>>>> amdgpu_bo_fence(abo, fence, false);
>>>>> dma_fence_put(fence);
>>>>> }
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>>> index 381101d2bf05..50fcd86e1033 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h
>>>>> @@ -164,4 +164,29 @@ static inline void amdgpu_res_next(struct amdgpu_res_cursor *cur, uint64_t size)
>>>>> }
>>>>> }
>>>>>
>>>>> +/**
>>>>> + * amdgpu_res_cleared - check if blocks are cleared
>>>>> + *
>>>>> + * @cur: the cursor to extract the block
>>>>> + *
>>>>> + * Check if the @cur block is cleared
>>>>> + */
>>>>> +static inline bool amdgpu_res_cleared(struct amdgpu_res_cursor *cur)
>>>>> +{
>>>>> + struct drm_buddy_block *block;
>>>>> +
>>>>> + switch (cur->mem_type) {
>>>>> + case TTM_PL_VRAM:
>>>>> + block = cur->node;
>>>>> +
>>>>> + if (!amdgpu_vram_mgr_is_cleared(block))
>>>>> + return false;
>>>>> + break;
>>>>> + default:
>>>>> + return false;
>>>>> + }
>>>>> +
>>>>> + return true;
>>>>> +}
>>>>> +
>>>>> #endif
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>>> index 8722beba494e..bcbffe909b47 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
>>>>> @@ -378,11 +378,15 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
>>>>> (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) {
>>>>> struct dma_fence *wipe_fence = NULL;
>>>>>
>>>>> - r = amdgpu_fill_buffer(abo, AMDGPU_POISON, NULL, &wipe_fence,
>>>>> - false);
>>>>> + r = amdgpu_fill_buffer(abo, 0, NULL, &wipe_fence,
>>>>> + false);
>>>>> if (r) {
>>>>> goto error;
>>>>> } else if (wipe_fence) {
>>>>> + struct amdgpu_vram_mgr_resource *vres;
>>>>> +
>>>>> + vres = to_amdgpu_vram_mgr_resource(bo->resource);
>>>>> + vres->flags |= DRM_BUDDY_CLEARED;
>>>>> dma_fence_put(fence);
>>>>> fence = wipe_fence;
>>>>> }
>>>>> @@ -2214,6 +2218,59 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_ring *ring, uint32_t src_data,
>>>>> return 0;
>>>>> }
>>>>>
>>>>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>>>>> + struct dma_resv *resv,
>>>>> + struct dma_fence **fence)
>>>>> +{
>>>>> + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>>>>> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
>>>>> + struct amdgpu_res_cursor cursor;
>>>>> + struct dma_fence *f = NULL;
>>>>> + u64 addr;
>>>>> + int r;
>>>>> +
>>>>> + if (!adev->mman.buffer_funcs_enabled)
>>>>> + return -EINVAL;
>>>>> +
>>>>> + amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor);
>>>>> +
>>>>> + mutex_lock(&adev->mman.gtt_window_lock);
>>>>> + while (cursor.remaining) {
>>>>> + struct dma_fence *next = NULL;
>>>>> + u64 size;
>>>>> +
>>>>> + if (amdgpu_res_cleared(&cursor)) {
>>>>> + amdgpu_res_next(&cursor, cursor.size);
>>>>> + continue;
>>>>> + }
>>>>> +
>>>>> + /* Never clear more than 256MiB at once to avoid timeouts */
>>>>> + size = min(cursor.size, 256ULL << 20);
>>>>> +
>>>>> + r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
>>>>> + 1, ring, false, &size, &addr);
>>>>> + if (r)
>>>>> + goto err;
>>>>> +
>>>>> + r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
>>>>> + &next, true, true);
>>>>> + if (r)
>>>>> + goto err;
>>>>> +
>>>>> + dma_fence_put(f);
>>>>> + f = next;
>>>>> +
>>>>> + amdgpu_res_next(&cursor, size);
>>>>> + }
>>>>> +err:
>>>>> + mutex_unlock(&adev->mman.gtt_window_lock);
>>>>> + if (fence)
>>>>> + *fence = dma_fence_get(f);
>>>>> + dma_fence_put(f);
>>>>> +
>>>>> + return r;
>>>>> +}
>>>>> +
>>>>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>>>>> uint32_t src_data,
>>>>> struct dma_resv *resv,
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>>> index 65ec82141a8e..b404d89d52e5 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
>>>>> @@ -38,8 +38,6 @@
>>>>> #define AMDGPU_GTT_MAX_TRANSFER_SIZE 512
>>>>> #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS 2
>>>>>
>>>>> -#define AMDGPU_POISON 0xd0bed0be
>>>>> -
>>>>> extern const struct attribute_group amdgpu_vram_mgr_attr_group;
>>>>> extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
>>>>>
>>>>> @@ -155,6 +153,9 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
>>>>> uint64_t size, bool tmz,
>>>>> struct dma_resv *resv,
>>>>> struct dma_fence **f);
>>>>> +int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
>>>>> + struct dma_resv *resv,
>>>>> + struct dma_fence **fence);
>>>>> int amdgpu_fill_buffer(struct amdgpu_bo *bo,
>>>>> uint32_t src_data,
>>>>> struct dma_resv *resv,
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> index c0c851409241..e494f5bf136a 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> @@ -450,6 +450,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
>>>>> {
>>>>> struct amdgpu_vram_mgr *mgr = to_vram_mgr(man);
>>>>> struct amdgpu_device *adev = to_amdgpu_device(mgr);
>>>>> + struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
>>>>> u64 vis_usage = 0, max_bytes, min_block_size;
>>>>> struct amdgpu_vram_mgr_resource *vres;
>>>>> u64 size, remaining_size, lpfn, fpfn;
>>>>> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
>>>>> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
>>>>> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>>>>>
>>>>> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
>>>>> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
>>>> Is there any reason to not always do this?
>>> Here we are trying to keep a pool of cleared memory which are cleared on
>>> free path and that can given
>>> to the application which requires a zeroed memory. I think here if we
>>> set always clear the memory,
>>> we would go over the limit of cleared memory in that particular instance
>>> and the application should wait until
>>> the hardware clears the memory and this might impact the overall
>>> performance.
>> I'd like to have the driver always deliver cleared memory.
> Actually, I think we can just always set the flag in the allocation IOCTLs.
We have use cases where that hurts as. Especially during boot when the
backing VRAM isn't cleared yet.
That's one of the reasons why we never always cleared the memory.
Christian.
>
> Alex
>
>> Alex
>>
>>> Thanks,
>>> Arun.
>>>> Alex
>>>>
>>>>
>>>>> +
>>>>> if (fpfn || lpfn != mgr->mm.size)
>>>>> /* Allocate blocks in desired range */
>>>>> vres->flags |= DRM_BUDDY_RANGE_ALLOCATION;
>>>>> @@ -604,7 +608,7 @@ static void amdgpu_vram_mgr_del(struct ttm_resource_manager *man,
>>>>>
>>>>> amdgpu_vram_mgr_do_reserve(man);
>>>>>
>>>>> - drm_buddy_free_list(mm, &vres->blocks, 0);
>>>>> + drm_buddy_free_list(mm, &vres->blocks, vres->flags);
>>>>> mutex_unlock(&mgr->lock);
>>>>>
>>>>> atomic64_sub(vis_usage, &mgr->vis_usage);
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>>> index 0e04e42cf809..8478522d7366 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.h
>>>>> @@ -53,6 +53,11 @@ static inline u64 amdgpu_vram_mgr_block_size(struct drm_buddy_block *block)
>>>>> return (u64)PAGE_SIZE << drm_buddy_block_order(block);
>>>>> }
>>>>>
>>>>> +static inline bool amdgpu_vram_mgr_is_cleared(struct drm_buddy_block *block)
>>>>> +{
>>>>> + return drm_buddy_block_is_clear(block);
>>>>> +}
>>>>> +
>>>>> static inline struct amdgpu_vram_mgr_resource *
>>>>> to_amdgpu_vram_mgr_resource(struct ttm_resource *res)
>>>>> {
>>>>> --
>>>>> 2.25.1
>>>>>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality
2024-04-02 8:17 ` Christian König
@ 2024-04-03 9:01 ` Michel Dänzer
0 siblings, 0 replies; 21+ messages in thread
From: Michel Dänzer @ 2024-04-03 9:01 UTC (permalink / raw)
To: Christian König, Alex Deucher, Paneer Selvam, Arunpravin
Cc: dri-devel, amd-gfx, alexander.deucher, matthew.auld,
mario.limonciello, felix.kuehling
On 2024-04-02 10:17, Christian König wrote:
> Am 26.03.24 um 15:53 schrieb Alex Deucher:
>> On Tue, Mar 26, 2024 at 10:01 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>>> On Tue, Mar 26, 2024 at 9:59 AM Paneer Selvam, Arunpravin
>>>>>> @@ -501,6 +502,9 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
>>>>>> if (place->flags & TTM_PL_FLAG_CONTIGUOUS)
>>>>>> vres->flags |= DRM_BUDDY_CONTIGUOUS_ALLOCATION;
>>>>>>
>>>>>> + if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED)
>>>>>> + vres->flags |= DRM_BUDDY_CLEAR_ALLOCATION;
>>>>> Is there any reason to not always do this?
>>>> Here we are trying to keep a pool of cleared memory which are cleared on
>>>> free path and that can given
>>>> to the application which requires a zeroed memory. I think here if we
>>>> set always clear the memory,
>>>> we would go over the limit of cleared memory in that particular instance
>>>> and the application should wait until
>>>> the hardware clears the memory and this might impact the overall
>>>> performance.
>>> I'd like to have the driver always deliver cleared memory.
>> Actually, I think we can just always set the flag in the allocation IOCTLs.
>
> We have use cases where that hurts as. Especially during boot when the backing VRAM isn't cleared yet.
>
> That's one of the reasons why we never always cleared the memory.
Any such performance gain was only valid in the first place if the kernel delivering non-cleared memory to user space was considered acceptable, which it quite clearly isn't.
--
Earthling Michel Dänzer | https://redhat.com
Libre software enthusiast | Mesa and Xwayland developer
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature
2024-04-01 11:07 ` Paneer Selvam, Arunpravin
@ 2024-04-05 15:43 ` Matthew Auld
0 siblings, 0 replies; 21+ messages in thread
From: Matthew Auld @ 2024-04-05 15:43 UTC (permalink / raw)
To: Paneer Selvam, Arunpravin, dri-devel, amd-gfx
Cc: christian.koenig, alexander.deucher, mario.limonciello, felix.kuehling
On 01/04/2024 12:07, Paneer Selvam, Arunpravin wrote:
> Hi Matthew,
>
> On 3/28/2024 10:18 PM, Matthew Auld wrote:
>> On 28/03/2024 16:07, Paneer Selvam, Arunpravin wrote:
>>> Hi Matthew,
>>>
>>> On 3/26/2024 11:39 PM, Matthew Auld wrote:
>>>> On 18/03/2024 21:40, Arunpravin Paneer Selvam wrote:
>>>>> - Add tracking clear page feature.
>>>>>
>>>>> - Driver should enable the DRM_BUDDY_CLEARED flag if it
>>>>> successfully clears the blocks in the free path. On the otherhand,
>>>>> DRM buddy marks each block as cleared.
>>>>>
>>>>> - Track the available cleared pages size
>>>>>
>>>>> - If driver requests cleared memory we prefer cleared memory
>>>>> but fallback to uncleared if we can't find the cleared blocks.
>>>>> when driver requests uncleared memory we try to use uncleared but
>>>>> fallback to cleared memory if necessary.
>>>>>
>>>>> - When a block gets freed we clear it and mark the freed block as
>>>>> cleared,
>>>>> when there are buddies which are cleared as well we can merge them.
>>>>> Otherwise, we prefer to keep the blocks as separated.
>>>>>
>>>>> - Add a function to support defragmentation.
>>>>>
>>>>> v1:
>>>>> - Depends on the flag check DRM_BUDDY_CLEARED, enable the block as
>>>>> cleared. Else, reset the clear flag for each block in the
>>>>> list(Christian)
>>>>> - For merging the 2 cleared blocks compare as below,
>>>>> drm_buddy_is_clear(block) != drm_buddy_is_clear(buddy)(Christian)
>>>>> - Defragment the memory beginning from min_order
>>>>> till the required memory space is available.
>>>>>
>>>>> v2: (Matthew)
>>>>> - Add a wrapper drm_buddy_free_list_internal for the freeing of
>>>>> blocks
>>>>> operation within drm buddy.
>>>>> - Write a macro block_incompatible() to allocate the required
>>>>> blocks.
>>>>> - Update the xe driver for the drm_buddy_free_list change in
>>>>> arguments.
>>>>> - add a warning if the two blocks are incompatible on
>>>>> defragmentation
>>>>> - call full defragmentation in the fini() function
>>>>> - place a condition to test if min_order is equal to 0
>>>>> - replace the list with safe_reverse() variant as we might
>>>>> remove the block from the list.
>>>>>
>>>>> v3:
>>>>> - fix Gitlab user reported lockup issue.
>>>>> - Keep DRM_BUDDY_HEADER_CLEAR define sorted(Matthew)
>>>>> - modify to pass the root order instead max_order in fini()
>>>>> function(Matthew)
>>>>> - change bool 1 to true(Matthew)
>>>>> - add check if min_block_size is power of 2(Matthew)
>>>>> - modify the min_block_size datatype to u64(Matthew)
>>>>>
>>>>> v4:
>>>>> - rename the function drm_buddy_defrag with __force_merge.
>>>>> - Include __force_merge directly in drm buddy file and remove
>>>>> the defrag use in amdgpu driver.
>>>>> - Remove list_empty() check(Matthew)
>>>>> - Remove unnecessary space, headers and placement of new
>>>>> variables(Matthew)
>>>>> - Add a unit test case(Matthew)
>>>>>
>>>>> Signed-off-by: Arunpravin Paneer Selvam
>>>>> <Arunpravin.PaneerSelvam@amd.com>
>>>>> Signed-off-by: Matthew Auld <matthew.auld@intel.com>
>>>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>>>> Suggested-by: Matthew Auld <matthew.auld@intel.com>
>>>>> ---
>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 +-
>>>>> drivers/gpu/drm/drm_buddy.c | 427
>>>>> ++++++++++++++----
>>>>> drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 6 +-
>>>>> drivers/gpu/drm/tests/drm_buddy_test.c | 18 +-
>>>>> drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 4 +-
>>>>> include/drm/drm_buddy.h | 16 +-
>>>>> 6 files changed, 360 insertions(+), 117 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> index 8db880244324..c0c851409241 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
>>>>> @@ -571,7 +571,7 @@ static int amdgpu_vram_mgr_new(struct
>>>>> ttm_resource_manager *man,
>>>>> return 0;
>>>>> error_free_blocks:
>>>>> - drm_buddy_free_list(mm, &vres->blocks);
>>>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>>>> mutex_unlock(&mgr->lock);
>>>>> error_fini:
>>>>> ttm_resource_fini(man, &vres->base);
>>>>> @@ -604,7 +604,7 @@ static void amdgpu_vram_mgr_del(struct
>>>>> ttm_resource_manager *man,
>>>>> amdgpu_vram_mgr_do_reserve(man);
>>>>> - drm_buddy_free_list(mm, &vres->blocks);
>>>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>>>> mutex_unlock(&mgr->lock);
>>>>> atomic64_sub(vis_usage, &mgr->vis_usage);
>>>>> @@ -912,7 +912,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device
>>>>> *adev)
>>>>> kfree(rsv);
>>>>> list_for_each_entry_safe(rsv, temp, &mgr->reserved_pages,
>>>>> blocks) {
>>>>> - drm_buddy_free_list(&mgr->mm, &rsv->allocated);
>>>>> + drm_buddy_free_list(&mgr->mm, &rsv->allocated, 0);
>>>>> kfree(rsv);
>>>>> }
>>>>> if (!adev->gmc.is_app_apu)
>>>>> diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c
>>>>> index c4222b886db7..625a30a6b855 100644
>>>>> --- a/drivers/gpu/drm/drm_buddy.c
>>>>> +++ b/drivers/gpu/drm/drm_buddy.c
>>>>> @@ -38,8 +38,8 @@ static void drm_block_free(struct drm_buddy *mm,
>>>>> kmem_cache_free(slab_blocks, block);
>>>>> }
>>>>> -static void list_insert_sorted(struct drm_buddy *mm,
>>>>> - struct drm_buddy_block *block)
>>>>> +static void list_insert(struct drm_buddy *mm,
>>>>> + struct drm_buddy_block *block)
>>>>> {
>>>>> struct drm_buddy_block *node;
>>>>> struct list_head *head;
>>>>> @@ -57,6 +57,16 @@ static void list_insert_sorted(struct drm_buddy
>>>>> *mm,
>>>>> __list_add(&block->link, node->link.prev, &node->link);
>>>>> }
>>>>> +static void clear_reset(struct drm_buddy_block *block)
>>>>> +{
>>>>> + block->header &= ~DRM_BUDDY_HEADER_CLEAR;
>>>>> +}
>>>>> +
>>>>> +static void mark_cleared(struct drm_buddy_block *block)
>>>>> +{
>>>>> + block->header |= DRM_BUDDY_HEADER_CLEAR;
>>>>> +}
>>>>> +
>>>>> static void mark_allocated(struct drm_buddy_block *block)
>>>>> {
>>>>> block->header &= ~DRM_BUDDY_HEADER_STATE;
>>>>> @@ -71,7 +81,7 @@ static void mark_free(struct drm_buddy *mm,
>>>>> block->header &= ~DRM_BUDDY_HEADER_STATE;
>>>>> block->header |= DRM_BUDDY_FREE;
>>>>> - list_insert_sorted(mm, block);
>>>>> + list_insert(mm, block);
>>>>> }
>>>>> static void mark_split(struct drm_buddy_block *block)
>>>>> @@ -82,6 +92,114 @@ static void mark_split(struct drm_buddy_block
>>>>> *block)
>>>>> list_del(&block->link);
>>>>> }
>>>>> +static struct drm_buddy_block *
>>>>> +__get_buddy(struct drm_buddy_block *block)
>>>>> +{
>>>>> + struct drm_buddy_block *parent;
>>>>> +
>>>>> + parent = block->parent;
>>>>> + if (!parent)
>>>>> + return NULL;
>>>>> +
>>>>> + if (parent->left == block)
>>>>> + return parent->right;
>>>>> +
>>>>> + return parent->left;
>>>>> +}
>>>>> +
>>>>> +static unsigned int __drm_buddy_free(struct drm_buddy *mm,
>>>>> + struct drm_buddy_block *block,
>>>>> + bool force_merge)
>>>>> +{
>>>>> + struct drm_buddy_block *parent;
>>>>> + unsigned int order;
>>>>> +
>>>>> + while ((parent = block->parent)) {
>>>>> + struct drm_buddy_block *buddy;
>>>>> +
>>>>> + buddy = __get_buddy(block);
>>>>> +
>>>>> + if (!drm_buddy_block_is_free(buddy))
>>>>> + break;
>>>>> +
>>>>> + if (!force_merge) {
>>>>> + /**
>>>>
>>>> Not really valid kernel-doc AFAIK. I think drop the extra *. Below
>>>> also.
>>>>
>>>>> + * Check the block and its buddy clear state and exit
>>>>> + * the loop if they both have the dissimilar state.
>>>>> + */
>>>>> + if (drm_buddy_block_is_clear(block) !=
>>>>> + drm_buddy_block_is_clear(buddy))
>>>>> + break;
>>>>> +
>>>>> + if (drm_buddy_block_is_clear(block))
>>>>> + mark_cleared(parent);
>>>>> + }
>>>>> +
>>>>> + list_del(&buddy->link);
>>>>> + if (force_merge && drm_buddy_block_is_clear(buddy))
>>>>> + mm->clear_avail -= drm_buddy_block_size(mm, buddy);
>>>>> +
>>>>> + drm_block_free(mm, block);
>>>>> + drm_block_free(mm, buddy);
>>>>> +
>>>>> + block = parent;
>>>>> + }
>>>>> +
>>>>> + order = drm_buddy_block_order(block);
>>>>> + mark_free(mm, block);
>>>>> +
>>>>> + return order;
>>>>> +}
>>>>> +
>>>>> +static int __force_merge(struct drm_buddy *mm,
>>>>> + unsigned int min_order)
>>>>> +{
>>>>> + unsigned int order;
>>>>> + int i;
>>>>> +
>>>>> + if (!min_order)
>>>>> + return -ENOMEM;
>>>>> +
>>>>> + if (min_order > mm->max_order)
>>>>> + return -EINVAL;
>>>>> +
>>>>> + for (i = min_order - 1; i >= 0; i--) {
>>>>> + struct drm_buddy_block *block, *prev;
>>>>> +
>>>>> + list_for_each_entry_safe_reverse(block, prev,
>>>>> &mm->free_list[i], link) {
>>>>> + struct drm_buddy_block *buddy;
>>>>> +
>>>>> + if (!block->parent)
>>>>> + continue;
>>>>> +
>>>>> + buddy = __get_buddy(block);
>>>>> + if (!drm_buddy_block_is_free(buddy))
>>>>> + continue;
>>>>> +
>>>>> + WARN_ON(drm_buddy_block_is_clear(block) ==
>>>>> + drm_buddy_block_is_clear(buddy));
>>>>> +
>>>>> + /**
>>>>> + * If the prev block is same as buddy, don't access the
>>>>> + * block in the next iteration as we would free the
>>>>> + * buddy block as part of the free function.
>>>>> + */
>>>>> + if (prev == buddy)
>>>>> + prev = list_prev_entry(prev, link);
>>>>> +
>>>>> + list_del(&block->link);
>>>>> + if (drm_buddy_block_is_clear(block))
>>>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>>>> +
>>>>> + order = __drm_buddy_free(mm, block, true);
>>>>> + if (order >= min_order)
>>>>> + return 0;
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + return -ENOMEM;
>>>>> +}
>>>>> +
>>>>> /**
>>>>> * drm_buddy_init - init memory manager
>>>>> *
>>>>> @@ -137,7 +255,7 @@ int drm_buddy_init(struct drm_buddy *mm, u64
>>>>> size, u64 chunk_size)
>>>>> offset = 0;
>>>>> i = 0;
>>>>> - /*
>>>>> + /**
>>>>> * Split into power-of-two blocks, in case we are given a
>>>>> size that is
>>>>> * not itself a power-of-two.
>>>>> */
>>>>> @@ -186,11 +304,21 @@ EXPORT_SYMBOL(drm_buddy_init);
>>>>> */
>>>>> void drm_buddy_fini(struct drm_buddy *mm)
>>>>> {
>>>>> + u64 root_size, size;
>>>>> + unsigned int order;
>>>>> int i;
>>>>> + size = mm->size;
>>>>> +
>>>>> for (i = 0; i < mm->n_roots; ++i) {
>>>>> + order = ilog2(size) - ilog2(mm->chunk_size);
>>>>> + __force_merge(mm, order);
>>>>> +
>>>>> WARN_ON(!drm_buddy_block_is_free(mm->roots[i]));
>>>>> drm_block_free(mm, mm->roots[i]);
>>>>> +
>>>>> + root_size = mm->chunk_size << order;
>>>>> + size -= root_size;
>>>>> }
>>>>> WARN_ON(mm->avail != mm->size);
>>>>> @@ -223,26 +351,17 @@ static int split_block(struct drm_buddy *mm,
>>>>> mark_free(mm, block->left);
>>>>> mark_free(mm, block->right);
>>>>> + if (drm_buddy_block_is_clear(block)) {
>>>>> + mark_cleared(block->left);
>>>>> + mark_cleared(block->right);
>>>>> + clear_reset(block);
>>>>> + }
>>>>> +
>>>>> mark_split(block);
>>>>> return 0;
>>>>> }
>>>>> -static struct drm_buddy_block *
>>>>> -__get_buddy(struct drm_buddy_block *block)
>>>>> -{
>>>>> - struct drm_buddy_block *parent;
>>>>> -
>>>>> - parent = block->parent;
>>>>> - if (!parent)
>>>>> - return NULL;
>>>>> -
>>>>> - if (parent->left == block)
>>>>> - return parent->right;
>>>>> -
>>>>> - return parent->left;
>>>>> -}
>>>>> -
>>>>> /**
>>>>> * drm_get_buddy - get buddy address
>>>>> *
>>>>> @@ -260,30 +379,6 @@ drm_get_buddy(struct drm_buddy_block *block)
>>>>> }
>>>>> EXPORT_SYMBOL(drm_get_buddy);
>>>>> -static void __drm_buddy_free(struct drm_buddy *mm,
>>>>> - struct drm_buddy_block *block)
>>>>> -{
>>>>> - struct drm_buddy_block *parent;
>>>>> -
>>>>> - while ((parent = block->parent)) {
>>>>> - struct drm_buddy_block *buddy;
>>>>> -
>>>>> - buddy = __get_buddy(block);
>>>>> -
>>>>> - if (!drm_buddy_block_is_free(buddy))
>>>>> - break;
>>>>> -
>>>>> - list_del(&buddy->link);
>>>>> -
>>>>> - drm_block_free(mm, block);
>>>>> - drm_block_free(mm, buddy);
>>>>> -
>>>>> - block = parent;
>>>>> - }
>>>>> -
>>>>> - mark_free(mm, block);
>>>>> -}
>>>>> -
>>>>> /**
>>>>> * drm_buddy_free_block - free a block
>>>>> *
>>>>> @@ -295,26 +390,59 @@ void drm_buddy_free_block(struct drm_buddy *mm,
>>>>> {
>>>>> BUG_ON(!drm_buddy_block_is_allocated(block));
>>>>> mm->avail += drm_buddy_block_size(mm, block);
>>>>> - __drm_buddy_free(mm, block);
>>>>> + if (drm_buddy_block_is_clear(block))
>>>>> + mm->clear_avail += drm_buddy_block_size(mm, block);
>>>>> +
>>>>> + __drm_buddy_free(mm, block, false);
>>>>> }
>>>>> EXPORT_SYMBOL(drm_buddy_free_block);
>>>>> -/**
>>>>> - * drm_buddy_free_list - free blocks
>>>>> - *
>>>>> - * @mm: DRM buddy manager
>>>>> - * @objects: input list head to free blocks
>>>>> - */
>>>>> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head
>>>>> *objects)
>>>>> +static void __drm_buddy_free_list(struct drm_buddy *mm,
>>>>> + struct list_head *objects,
>>>>> + bool mark_clear,
>>>>> + bool mark_dirty)
>>>>> {
>>>>> struct drm_buddy_block *block, *on;
>>>>> + WARN_ON(mark_dirty && mark_clear);
>>>>> +
>>>>> list_for_each_entry_safe(block, on, objects, link) {
>>>>> + if (mark_clear)
>>>>> + mark_cleared(block);
>>>>> + else if (mark_dirty)
>>>>> + clear_reset(block);
>>>>> drm_buddy_free_block(mm, block);
>>>>> cond_resched();
>>>>> }
>>>>> INIT_LIST_HEAD(objects);
>>>>> }
>>>>> +
>>>>> +static void drm_buddy_free_list_internal(struct drm_buddy *mm,
>>>>> + struct list_head *objects)
>>>>> +{
>>>>> + /**
>>>>> + * Don't touch the clear/dirty bit, since allocation is still
>>>>> internal
>>>>> + * at this point. For example we might have just failed part
>>>>> of the
>>>>> + * allocation.
>>>>> + */
>>>>> + __drm_buddy_free_list(mm, objects, false, false);
>>>>> +}
>>>>> +
>>>>> +/**
>>>>> + * drm_buddy_free_list - free blocks
>>>>> + *
>>>>> + * @mm: DRM buddy manager
>>>>> + * @objects: input list head to free blocks
>>>>> + * @flags: optional flags like DRM_BUDDY_CLEARED
>>>>> + */
>>>>> +void drm_buddy_free_list(struct drm_buddy *mm,
>>>>> + struct list_head *objects,
>>>>> + unsigned int flags)
>>>>> +{
>>>>> + bool mark_clear = flags & DRM_BUDDY_CLEARED;
>>>>> +
>>>>> + __drm_buddy_free_list(mm, objects, mark_clear, !mark_clear);
>>>>> +}
>>>>> EXPORT_SYMBOL(drm_buddy_free_list);
>>>>> static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
>>>>> @@ -327,10 +455,19 @@ static inline bool contains(u64 s1, u64 e1,
>>>>> u64 s2, u64 e2)
>>>>> return s1 <= s2 && e1 >= e2;
>>>>> }
>>>>> +static bool block_incompatible(struct drm_buddy_block *block,
>>>>> unsigned int flags)
>>>>> +{
>>>>> + bool needs_clear = flags & DRM_BUDDY_CLEAR_ALLOCATION;
>>>>> +
>>>>> + return needs_clear != drm_buddy_block_is_clear(block);
>>>>> +}
>>>>> +
>>>>> static struct drm_buddy_block *
>>>>> -alloc_range_bias(struct drm_buddy *mm,
>>>>> - u64 start, u64 end,
>>>>> - unsigned int order)
>>>>> +__alloc_range_bias(struct drm_buddy *mm,
>>>>> + u64 start, u64 end,
>>>>> + unsigned int order,
>>>>> + unsigned long flags,
>>>>> + bool fallback)
>>>>> {
>>>>> struct drm_buddy_block *block;
>>>>> struct drm_buddy_block *buddy;
>>>>> @@ -369,7 +506,10 @@ alloc_range_bias(struct drm_buddy *mm,
>>>>> if (contains(start, end, block_start, block_end) &&
>>>>> order == drm_buddy_block_order(block)) {
>>>>> - /*
>>>>> + if (!fallback && block_incompatible(block, flags))
>>>>> + continue;
>>>>> +
>>>>> + /**
>>>>> * Find the free block within the range.
>>>>> */
>>>>> if (drm_buddy_block_is_free(block))
>>>>> @@ -391,7 +531,7 @@ alloc_range_bias(struct drm_buddy *mm,
>>>>> return ERR_PTR(-ENOSPC);
>>>>> err_undo:
>>>>> - /*
>>>>> + /**
>>>>> * We really don't want to leave around a bunch of split
>>>>> blocks, since
>>>>> * bigger is better, so make sure we merge everything back
>>>>> before we
>>>>> * free the allocated blocks.
>>>>> @@ -400,30 +540,57 @@ alloc_range_bias(struct drm_buddy *mm,
>>>>> if (buddy &&
>>>>> (drm_buddy_block_is_free(block) &&
>>>>> drm_buddy_block_is_free(buddy)))
>>>>> - __drm_buddy_free(mm, block);
>>>>> + __drm_buddy_free(mm, block, false);
>>>>> return ERR_PTR(err);
>>>>> }
>>>>> static struct drm_buddy_block *
>>>>> -get_maxblock(struct drm_buddy *mm, unsigned int order)
>>>>> +__drm_buddy_alloc_range_bias(struct drm_buddy *mm,
>>>>> + u64 start, u64 end,
>>>>> + unsigned int order,
>>>>> + unsigned long flags)
>>>>> +{
>>>>> + struct drm_buddy_block *block;
>>>>> + bool fallback = false;
>>>>> +
>>>>> + block = __alloc_range_bias(mm, start, end, order,
>>>>> + flags, fallback);
>>>>> + if (IS_ERR(block) && mm->clear_avail)
>>>>> + return __alloc_range_bias(mm, start, end, order,
>>>>> + flags, !fallback);
>>>>> +
>>>>> + return block;
>>>>> +}
>>>>> +
>>>>> +static struct drm_buddy_block *
>>>>> +get_maxblock(struct drm_buddy *mm, unsigned int order,
>>>>> + unsigned long flags)
>>>>> {
>>>>> - struct drm_buddy_block *max_block = NULL, *node;
>>>>> + struct drm_buddy_block *max_block = NULL, *block = NULL;
>>>>> unsigned int i;
>>>>> for (i = order; i <= mm->max_order; ++i) {
>>>>> - if (!list_empty(&mm->free_list[i])) {
>>>>> - node = list_last_entry(&mm->free_list[i],
>>>>> - struct drm_buddy_block,
>>>>> - link);
>>>>> - if (!max_block) {
>>>>> - max_block = node;
>>>>> + struct drm_buddy_block *tmp_block;
>>>>> +
>>>>> + list_for_each_entry_reverse(tmp_block, &mm->free_list[i],
>>>>> link) {
>>>>> + if (block_incompatible(tmp_block, flags))
>>>>> continue;
>>>>> - }
>>>>> - if (drm_buddy_block_offset(node) >
>>>>> - drm_buddy_block_offset(max_block)) {
>>>>> - max_block = node;
>>>>> - }
>>>>> + block = tmp_block;
>>>>> + break;
>>>>> + }
>>>>> +
>>>>> + if (!block)
>>>>> + continue;
>>>>> +
>>>>> + if (!max_block) {
>>>>> + max_block = block;
>>>>> + continue;
>>>>> + }
>>>>> +
>>>>> + if (drm_buddy_block_offset(block) >
>>>>> + drm_buddy_block_offset(max_block)) {
>>>>> + max_block = block;
>>>>> }
>>>>> }
>>>>> @@ -440,11 +607,29 @@ alloc_from_freelist(struct drm_buddy *mm,
>>>>> int err;
>>>>> if (flags & DRM_BUDDY_TOPDOWN_ALLOCATION) {
>>>>> - block = get_maxblock(mm, order);
>>>>> + block = get_maxblock(mm, order, flags);
>>>>> if (block)
>>>>> /* Store the obtained block order */
>>>>> tmp = drm_buddy_block_order(block);
>>>>> } else {
>>>>> + for (tmp = order; tmp <= mm->max_order; ++tmp) {
>>>>> + struct drm_buddy_block *tmp_block;
>>>>> +
>>>>> + list_for_each_entry_reverse(tmp_block,
>>>>> &mm->free_list[tmp], link) {
>>>>> + if (block_incompatible(tmp_block, flags))
>>>>> + continue;
>>>>> +
>>>>> + block = tmp_block;
>>>>> + break;
>>>>> + }
>>>>> +
>>>>> + if (block)
>>>>> + break;
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + if (!block) {
>>>>> + /* Fallback method */
>>>>> for (tmp = order; tmp <= mm->max_order; ++tmp) {
>>>>> if (!list_empty(&mm->free_list[tmp])) {
>>>>> block = list_last_entry(&mm->free_list[tmp],
>>>>> @@ -454,10 +639,10 @@ alloc_from_freelist(struct drm_buddy *mm,
>>>>> break;
>>>>> }
>>>>> }
>>>>> - }
>>>>> - if (!block)
>>>>> - return ERR_PTR(-ENOSPC);
>>>>> + if (!block)
>>>>> + return ERR_PTR(-ENOSPC);
>>>>> + }
>>>>> BUG_ON(!drm_buddy_block_is_free(block));
>>>>> @@ -473,7 +658,7 @@ alloc_from_freelist(struct drm_buddy *mm,
>>>>> err_undo:
>>>>> if (tmp != order)
>>>>> - __drm_buddy_free(mm, block);
>>>>> + __drm_buddy_free(mm, block, false);
>>>>> return ERR_PTR(err);
>>>>> }
>>>>> @@ -524,6 +709,8 @@ static int __alloc_range(struct drm_buddy *mm,
>>>>> mark_allocated(block);
>>>>> total_allocated += drm_buddy_block_size(mm, block);
>>>>> mm->avail -= drm_buddy_block_size(mm, block);
>>>>> + if (drm_buddy_block_is_clear(block))
>>>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>>>> list_add_tail(&block->link, &allocated);
>>>>> continue;
>>>>> }
>>>>> @@ -548,7 +735,7 @@ static int __alloc_range(struct drm_buddy *mm,
>>>>> return 0;
>>>>> err_undo:
>>>>> - /*
>>>>> + /**
>>>>> * We really don't want to leave around a bunch of split
>>>>> blocks, since
>>>>> * bigger is better, so make sure we merge everything back
>>>>> before we
>>>>> * free the allocated blocks.
>>>>> @@ -557,14 +744,14 @@ static int __alloc_range(struct drm_buddy *mm,
>>>>> if (buddy &&
>>>>> (drm_buddy_block_is_free(block) &&
>>>>> drm_buddy_block_is_free(buddy)))
>>>>> - __drm_buddy_free(mm, block);
>>>>> + __drm_buddy_free(mm, block, false);
>>>>> err_free:
>>>>> if (err == -ENOSPC && total_allocated_on_err) {
>>>>> list_splice_tail(&allocated, blocks);
>>>>> *total_allocated_on_err = total_allocated;
>>>>> } else {
>>>>> - drm_buddy_free_list(mm, &allocated);
>>>>> + drm_buddy_free_list_internal(mm, &allocated);
>>>>> }
>>>>> return err;
>>>>> @@ -630,11 +817,11 @@ static int __alloc_contig_try_harder(struct
>>>>> drm_buddy *mm,
>>>>> list_splice(&blocks_lhs, blocks);
>>>>> return 0;
>>>>> } else if (err != -ENOSPC) {
>>>>> - drm_buddy_free_list(mm, blocks);
>>>>> + drm_buddy_free_list_internal(mm, blocks);
>>>>> return err;
>>>>> }
>>>>> /* Free blocks for the next iteration */
>>>>> - drm_buddy_free_list(mm, blocks);
>>>>> + drm_buddy_free_list_internal(mm, blocks);
>>>>> }
>>>>> return -ENOSPC;
>>>>> @@ -690,6 +877,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>>>> list_del(&block->link);
>>>>> mark_free(mm, block);
>>>>> mm->avail += drm_buddy_block_size(mm, block);
>>>>> + if (drm_buddy_block_is_clear(block))
>>>>> + mm->clear_avail += drm_buddy_block_size(mm, block);
>>>>> /* Prevent recursively freeing this node */
>>>>> parent = block->parent;
>>>>> @@ -701,6 +890,8 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>>>> if (err) {
>>>>> mark_allocated(block);
>>>>> mm->avail -= drm_buddy_block_size(mm, block);
>>>>> + if (drm_buddy_block_is_clear(block))
>>>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>>>> list_add(&block->link, blocks);
>>>>> }
>>>>> @@ -709,13 +900,28 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>>>> }
>>>>> EXPORT_SYMBOL(drm_buddy_block_trim);
>>>>> +static struct drm_buddy_block *
>>>>> +__drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>>>> + u64 start, u64 end,
>>>>> + unsigned int order,
>>>>> + unsigned long flags)
>>>>> +{
>>>>> + if (flags & DRM_BUDDY_RANGE_ALLOCATION)
>>>>> + /* Allocate traversing within the range */
>>>>> + return __drm_buddy_alloc_range_bias(mm, start, end,
>>>>> + order, flags);
>>>>> + else
>>>>> + /* Allocate from freelist */
>>>>> + return alloc_from_freelist(mm, order, flags);
>>>>> +}
>>>>> +
>>>>> /**
>>>>> * drm_buddy_alloc_blocks - allocate power-of-two blocks
>>>>> *
>>>>> * @mm: DRM buddy manager to allocate from
>>>>> * @start: start of the allowed range for this block
>>>>> * @end: end of the allowed range for this block
>>>>> - * @size: size of the allocation
>>>>> + * @size: size of the allocation in bytes
>>>>> * @min_block_size: alignment of the allocation
>>>>> * @blocks: output list head to add allocated blocks
>>>>> * @flags: DRM_BUDDY_*_ALLOCATION flags
>>>>> @@ -761,8 +967,18 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>>>> return -EINVAL;
>>>>> /* Actual range allocation */
>>>>> - if (start + size == end)
>>>>> - return __drm_buddy_alloc_range(mm, start, size, NULL,
>>>>> blocks);
>>>>> + if (start + size == end) {
>>>>> + err = __drm_buddy_alloc_range(mm, start, size, NULL,
>>>>> blocks);
>>>>> + if (err) {
>>>>> + order = ilog2(size) - ilog2(mm->chunk_size);
>>>>> + if (mm->clear_avail && !__force_merge(mm, order))
>>>>> + return __drm_buddy_alloc_range(mm, start, size,
>>>>> NULL, blocks);
>>>>
>>>> That seems strange at a glance. With an actual range allocation like
>>>> with intitial fb or whatever it should just find all overlapping
>>>> pages, splitting down where needed on the edges. Not sure why
>>>> force_merge would factor in here?
>>> In case of not merged (fragmentation due to dirty and clear pages
>>> split), if the memory request goes into that range, we might see the
>>> split blocks as not free and end up returning -ENOSPC.
>>> I found this problem when I tried to allocate the max_order where the
>>> start + size == end and it ends up entering into this code block and
>>> it returns -ENOSPC as there is no force_merge call
>>> and the requested range max order block seen as not free. And we have
>>> an another issue though if we use force_merge to defragment a
>>> specific ordered block, there is a less probability that
>>> we merge back the required blocks in the specific range. For now, we
>>> can go ahead with no force merge support for actual range allocation
>>> if we are sure that we use this only for initial fb etc.
>>
>> Ah right, now I see. But AFAICT trouble is if we say allocate 8K range
>> at some specific offset, like above, which fails due to clear/dirty
>> split then calling force_merge(order(8K)) is not always going to help.
>> It will just force merge enough for 8K, but that might not be the 8K
>> at the offset we were looking for, so it still fails. I think perhaps
>> update the dfs in alloc_range to not bail when contains && split. Or I
>> think other option is to force merge the entire address space on err.
>> Other ideas?
> For actual range allocation, if the min_block_size == mm->chunk_size, I
> think we can proceed with your first idea (i.e) update the dfs in
> alloc_range to not bail when contains && split.
> For bias range allocation, we need to add range support to the
> force_merge, for instance we can just continue in the force_merge for
> loop if the contains(start, end, block_start, block_end) returns false.
> Thoughts?
Yeah, I think something like that should work.
>
> Thanks,
> Arun.
>>
>>>
>>> If there are no major concerns, can we push these patches?
>>>
>>> Regards,
>>> Arun.
>>>>> +
>>>>> + return err;
>>>>> + }
>>>>> +
>>>>> + return err;
>>>>> + }
>>>>> original_size = size;
>>>>> original_min_size = min_block_size;
>>>>> @@ -786,23 +1002,34 @@ int drm_buddy_alloc_blocks(struct drm_buddy
>>>>> *mm,
>>>>> BUG_ON(order < min_order);
>>>>> do {
>>>>> - if (flags & DRM_BUDDY_RANGE_ALLOCATION)
>>>>> - /* Allocate traversing within the range */
>>>>> - block = alloc_range_bias(mm, start, end, order);
>>>>> - else
>>>>> - /* Allocate from freelist */
>>>>> - block = alloc_from_freelist(mm, order, flags);
>>>>> -
>>>>> + block = __drm_buddy_alloc_blocks(mm, start,
>>>>> + end,
>>>>> + order,
>>>>> + flags);
>>>>> if (!IS_ERR(block))
>>>>> break;
>>>>> if (order-- == min_order) {
>>>>> + /**
>>>>> + * Try allocation through force merge method
>>>>> + */
>>>>> + if (mm->clear_avail && !__force_merge(mm,
>>>>> min_order)) {
>>>>> + block = __drm_buddy_alloc_blocks(mm, start,
>>>>> + end,
>>>>> + min_order,
>>>>> + flags);
>>>>> + if (!IS_ERR(block)) {
>>>>> + order = min_order;
>>>>> + break;
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + �� /**
>>>>> + * Try contiguous block allocation through
>>>>> + * try harder method.
>>>>> + */
>>>>> if (flags & DRM_BUDDY_CONTIGUOUS_ALLOCATION &&
>>>>> !(flags & DRM_BUDDY_RANGE_ALLOCATION))
>>>>> - /*
>>>>> - * Try contiguous block allocation through
>>>>> - * try harder method
>>>>> - */
>>>>> return __alloc_contig_try_harder(mm,
>>>>> original_size,
>>>>> original_min_size,
>>>>> @@ -814,6 +1041,8 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>>>> mark_allocated(block);
>>>>> mm->avail -= drm_buddy_block_size(mm, block);
>>>>> + if (drm_buddy_block_is_clear(block))
>>>>> + mm->clear_avail -= drm_buddy_block_size(mm, block);
>>>>> kmemleak_update_trace(block);
>>>>> list_add_tail(&block->link, &allocated);
>>>>> @@ -823,7 +1052,9 @@ int drm_buddy_alloc_blocks(struct drm_buddy
>>>>> *mm,
>>>>> break;
>>>>> } while (1);
>>>>> - /* Trim the allocated block to the required size */
>>>>> + /**
>>>>> + * Trim the allocated block to the required size
>>>>> + */
>>>>> if (original_size != size) {
>>>>> struct list_head *trim_list;
>>>>> LIST_HEAD(temp);
>>>>> @@ -852,7 +1083,7 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm,
>>>>> return 0;
>>>>> err_free:
>>>>> - drm_buddy_free_list(mm, &allocated);
>>>>> + drm_buddy_free_list_internal(mm, &allocated);
>>>>> return err;
>>>>> }
>>>>> EXPORT_SYMBOL(drm_buddy_alloc_blocks);
>>>>> @@ -885,8 +1116,8 @@ void drm_buddy_print(struct drm_buddy *mm,
>>>>> struct drm_printer *p)
>>>>> {
>>>>> int order;
>>>>> - drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free:
>>>>> %lluMiB\n",
>>>>> - mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20);
>>>>> + drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free:
>>>>> %lluMiB, clear_free: %lluMiB\n",
>>>>> + mm->chunk_size >> 10, mm->size >> 20, mm->avail >> 20,
>>>>> mm->clear_avail >> 20);
>>>>> for (order = mm->max_order; order >= 0; order--) {
>>>>> struct drm_buddy_block *block;
>>>>> diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>>>> b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>>>> index 0d735d5c2b35..942345548bc3 100644
>>>>> --- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>>>> +++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c
>>>>> @@ -126,7 +126,7 @@ static int i915_ttm_buddy_man_alloc(struct
>>>>> ttm_resource_manager *man,
>>>>> return 0;
>>>>> err_free_blocks:
>>>>> - drm_buddy_free_list(mm, &bman_res->blocks);
>>>>> + drm_buddy_free_list(mm, &bman_res->blocks, 0);
>>>>> mutex_unlock(&bman->lock);
>>>>> err_free_res:
>>>>> ttm_resource_fini(man, &bman_res->base);
>>>>> @@ -141,7 +141,7 @@ static void i915_ttm_buddy_man_free(struct
>>>>> ttm_resource_manager *man,
>>>>> struct i915_ttm_buddy_manager *bman = to_buddy_manager(man);
>>>>> mutex_lock(&bman->lock);
>>>>> - drm_buddy_free_list(&bman->mm, &bman_res->blocks);
>>>>> + drm_buddy_free_list(&bman->mm, &bman_res->blocks, 0);
>>>>> bman->visible_avail += bman_res->used_visible_size;
>>>>> mutex_unlock(&bman->lock);
>>>>> @@ -345,7 +345,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device
>>>>> *bdev, unsigned int type)
>>>>> ttm_set_driver_manager(bdev, type, NULL);
>>>>> mutex_lock(&bman->lock);
>>>>> - drm_buddy_free_list(mm, &bman->reserved);
>>>>> + drm_buddy_free_list(mm, &bman->reserved, 0);
>>>>> drm_buddy_fini(mm);
>>>>> bman->visible_avail += bman->visible_reserved;
>>>>> WARN_ON_ONCE(bman->visible_avail != bman->visible_size);
>>>>> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c
>>>>> b/drivers/gpu/drm/tests/drm_buddy_test.c
>>>>> index 2f32fb2f12e7..454ad9952f56 100644
>>>>> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
>>>>> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
>>>>> @@ -64,7 +64,7 @@ static void
>>>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>>> "buddy_alloc didn't error size=%u\n", 3 * ps);
>>>>> - drm_buddy_free_list(&mm, &middle);
>>>>> + drm_buddy_free_list(&mm, &middle, 0);
>>>>> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>>>> mm_size,
>>>>> 3 * ps, ps, &allocated,
>>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>>> @@ -74,7 +74,7 @@ static void
>>>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>>> "buddy_alloc didn't error size=%u\n", 2 * ps);
>>>>> - drm_buddy_free_list(&mm, &right);
>>>>> + drm_buddy_free_list(&mm, &right, 0);
>>>>> KUNIT_ASSERT_TRUE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>>>> mm_size,
>>>>> 3 * ps, ps, &allocated,
>>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>>> @@ -89,7 +89,7 @@ static void
>>>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>>> "buddy_alloc hit an error size=%u\n", 2 * ps);
>>>>> - drm_buddy_free_list(&mm, &left);
>>>>> + drm_buddy_free_list(&mm, &left, 0);
>>>>> KUNIT_ASSERT_FALSE_MSG(test, drm_buddy_alloc_blocks(&mm, 0,
>>>>> mm_size,
>>>>> 3 * ps, ps, &allocated,
>>>>> DRM_BUDDY_CONTIGUOUS_ALLOCATION),
>>>>> @@ -101,7 +101,7 @@ static void
>>>>> drm_test_buddy_alloc_contiguous(struct kunit *test)
>>>>> KUNIT_ASSERT_EQ(test, total, ps * 2 + ps * 3);
>>>>> - drm_buddy_free_list(&mm, &allocated);
>>>>> + drm_buddy_free_list(&mm, &allocated, 0);
>>>>> drm_buddy_fini(&mm);
>>>>> }
>>>>> @@ -170,7 +170,7 @@ static void
>>>>> drm_test_buddy_alloc_pathological(struct kunit *test)
>>>>> top, max_order);
>>>>> }
>>>>> - drm_buddy_free_list(&mm, &holes);
>>>>> + drm_buddy_free_list(&mm, &holes, 0);
>>>>> /* Nothing larger than blocks of chunk_size now available */
>>>>> for (order = 1; order <= max_order; order++) {
>>>>> @@ -182,7 +182,7 @@ static void
>>>>> drm_test_buddy_alloc_pathological(struct kunit *test)
>>>>> }
>>>>> list_splice_tail(&holes, &blocks);
>>>>> - drm_buddy_free_list(&mm, &blocks);
>>>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>>>> drm_buddy_fini(&mm);
>>>>> }
>>>>> @@ -277,7 +277,7 @@ static void
>>>>> drm_test_buddy_alloc_pessimistic(struct kunit *test)
>>>>> list_del(&block->link);
>>>>> drm_buddy_free_block(&mm, block);
>>>>> - drm_buddy_free_list(&mm, &blocks);
>>>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>>>> drm_buddy_fini(&mm);
>>>>> }
>>>>> @@ -323,7 +323,7 @@ static void
>>>>> drm_test_buddy_alloc_optimistic(struct kunit *test)
>>>>> size, size, &tmp, flags),
>>>>> "buddy_alloc unexpectedly succeeded, it
>>>>> should be full!");
>>>>> - drm_buddy_free_list(&mm, &blocks);
>>>>> + drm_buddy_free_list(&mm, &blocks, 0);
>>>>> drm_buddy_fini(&mm);
>>>>> }
>>>>> @@ -358,7 +358,7 @@ static void drm_test_buddy_alloc_limit(struct
>>>>> kunit *test)
>>>>> drm_buddy_block_size(&mm, block),
>>>>> BIT_ULL(mm.max_order) * PAGE_SIZE);
>>>>> - drm_buddy_free_list(&mm, &allocated);
>>>>> + drm_buddy_free_list(&mm, &allocated, 0);
>>>>> drm_buddy_fini(&mm);
>>>>> }
>>>>> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>>>> b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>>>> index 115ec745e502..1ad678b62c4a 100644
>>>>> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>>>> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
>>>>> @@ -196,7 +196,7 @@ static int xe_ttm_vram_mgr_new(struct
>>>>> ttm_resource_manager *man,
>>>>> return 0;
>>>>> error_free_blocks:
>>>>> - drm_buddy_free_list(mm, &vres->blocks);
>>>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>>>> mutex_unlock(&mgr->lock);
>>>>> error_fini:
>>>>> ttm_resource_fini(man, &vres->base);
>>>>> @@ -214,7 +214,7 @@ static void xe_ttm_vram_mgr_del(struct
>>>>> ttm_resource_manager *man,
>>>>> struct drm_buddy *mm = &mgr->mm;
>>>>> mutex_lock(&mgr->lock);
>>>>> - drm_buddy_free_list(mm, &vres->blocks);
>>>>> + drm_buddy_free_list(mm, &vres->blocks, 0);
>>>>> mgr->visible_avail += vres->used_visible_size;
>>>>> mutex_unlock(&mgr->lock);
>>>>> diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h
>>>>> index a5b39fc01003..82570f77e817 100644
>>>>> --- a/include/drm/drm_buddy.h
>>>>> +++ b/include/drm/drm_buddy.h
>>>>> @@ -25,6 +25,8 @@
>>>>> #define DRM_BUDDY_RANGE_ALLOCATION BIT(0)
>>>>> #define DRM_BUDDY_TOPDOWN_ALLOCATION BIT(1)
>>>>> #define DRM_BUDDY_CONTIGUOUS_ALLOCATION BIT(2)
>>>>> +#define DRM_BUDDY_CLEAR_ALLOCATION BIT(3)
>>>>> +#define DRM_BUDDY_CLEARED BIT(4)
>>>>> struct drm_buddy_block {
>>>>> #define DRM_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
>>>>> @@ -32,8 +34,9 @@ struct drm_buddy_block {
>>>>> #define DRM_BUDDY_ALLOCATED (1 << 10)
>>>>> #define DRM_BUDDY_FREE (2 << 10)
>>>>> #define DRM_BUDDY_SPLIT (3 << 10)
>>>>> +#define DRM_BUDDY_HEADER_CLEAR GENMASK_ULL(9, 9)
>>>>> /* Free to be used, if needed in the future */
>>>>> -#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
>>>>> +#define DRM_BUDDY_HEADER_UNUSED GENMASK_ULL(8, 6)
>>>>> #define DRM_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
>>>>> u64 header;
>>>>> @@ -86,6 +89,7 @@ struct drm_buddy {
>>>>> u64 chunk_size;
>>>>> u64 size;
>>>>> u64 avail;
>>>>> + u64 clear_avail;
>>>>> };
>>>>> static inline u64
>>>>> @@ -112,6 +116,12 @@ drm_buddy_block_is_allocated(struct
>>>>> drm_buddy_block *block)
>>>>> return drm_buddy_block_state(block) == DRM_BUDDY_ALLOCATED;
>>>>> }
>>>>> +static inline bool
>>>>> +drm_buddy_block_is_clear(struct drm_buddy_block *block)
>>>>> +{
>>>>> + return block->header & DRM_BUDDY_HEADER_CLEAR;
>>>>> +}
>>>>> +
>>>>> static inline bool
>>>>> drm_buddy_block_is_free(struct drm_buddy_block *block)
>>>>> {
>>>>> @@ -150,7 +160,9 @@ int drm_buddy_block_trim(struct drm_buddy *mm,
>>>>> void drm_buddy_free_block(struct drm_buddy *mm, struct
>>>>> drm_buddy_block *block);
>>>>> -void drm_buddy_free_list(struct drm_buddy *mm, struct list_head
>>>>> *objects);
>>>>> +void drm_buddy_free_list(struct drm_buddy *mm,
>>>>> + struct list_head *objects,
>>>>> + unsigned int flags);
>>>>> void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p);
>>>>> void drm_buddy_block_print(struct drm_buddy *mm,
>>>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2024-04-05 15:43 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-18 21:40 [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature Arunpravin Paneer Selvam
2024-03-18 21:40 ` [PATCH v9 2/3] drm/amdgpu: Enable clear page functionality Arunpravin Paneer Selvam
2024-03-19 10:28 ` Christian König
2024-03-19 11:41 ` Paneer Selvam, Arunpravin
2024-03-19 13:47 ` Christian König
2024-03-19 13:57 ` Paneer Selvam, Arunpravin
2024-03-26 13:38 ` Alex Deucher
2024-03-26 13:59 ` Paneer Selvam, Arunpravin
2024-03-26 14:01 ` Alex Deucher
2024-03-26 14:53 ` Alex Deucher
2024-03-27 8:23 ` Paneer Selvam, Arunpravin
2024-04-02 8:17 ` Christian König
2024-04-03 9:01 ` Michel Dänzer
2024-03-18 21:40 ` [PATCH v9 3/3] drm/tests: Add a test case for drm buddy clear allocation Arunpravin Paneer Selvam
2024-03-26 17:46 ` Matthew Auld
2024-03-25 13:37 ` [PATCH v9 1/3] drm/buddy: Implement tracking clear page feature Paneer Selvam, Arunpravin
2024-03-26 18:09 ` Matthew Auld
2024-03-28 16:07 ` Paneer Selvam, Arunpravin
2024-03-28 16:48 ` Matthew Auld
2024-04-01 11:07 ` Paneer Selvam, Arunpravin
2024-04-05 15:43 ` Matthew Auld
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.