* [PATCH v2 0/5] More unit test for mballoc
@ 2024-01-03 10:48 Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 1/5] ext4: Add unit test for test_free_blocks_simple Kemeng Shi
` (6 more replies)
0 siblings, 7 replies; 15+ messages in thread
From: Kemeng Shi @ 2024-01-03 10:48 UTC (permalink / raw)
To: tytso, adilger.kernel; +Cc: linux-ext4, linux-kernel
This series covers more function to mark on-disk bitmap. Besides, some
code which is relevant to buddy cache is also tested.
Before more work is done, I want to be sure I'm not on a wrong
direction!
v1->v2:
-Fix unused variable warning which is reported at
https://lore.kernel.org/lkml/202311260042.kMxL6DnL-lkp@intel.com/T/
Kunit test result is as following:
# ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig --raw_output
[18:39:42] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
[18:39:45] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make ARCH=um O=.kunit --jobs=88
[18:39:57] Starting KUnit Kernel (1/1)...
KTAP version 1
1..2
KTAP version 1
# Subtest: ext4_mballoc_test
# module: ext4
1..6
KTAP version 1
# Subtest: test_new_blocks_simple
ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# test_new_blocks_simple: pass:3 fail:0 skip:0 total:3
ok 1 test_new_blocks_simple
KTAP version 1
# Subtest: test_free_blocks_simple
ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# test_free_blocks_simple: pass:3 fail:0 skip:0 total:3
ok 2 test_free_blocks_simple
KTAP version 1
# Subtest: test_mb_generate_buddy
ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# test_mb_generate_buddy: pass:3 fail:0 skip:0 total:3
ok 3 test_mb_generate_buddy
KTAP version 1
# Subtest: test_mb_mark_used
ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# SKIP blocksize exceeds pagesize
# test_mb_mark_used: pass:2 fail:0 skip:1 total:3
ok 4 test_mb_mark_used
KTAP version 1
# Subtest: test_mb_free_blocks
ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# SKIP blocksize exceeds pagesize
# test_mb_free_blocks: pass:2 fail:0 skip:1 total:3
ok 5 test_mb_free_blocks
KTAP version 1
# Subtest: test_mark_diskspace_used
ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# test_mark_diskspace_used: pass:3 fail:0 skip:0 total:3
ok 6 test_mark_diskspace_used
# ext4_mballoc_test: pass:6 fail:0 skip:0 total:6
# Totals: pass:16 fail:0 skip:2 total:18
ok 1 ext4_mballoc_test
KTAP version 1
# Subtest: ext4_inode_test
# module: ext4_inode_test
1..1
KTAP version 1
# Subtest: inode_test_xtimestamp_decoding
ok 1 1901-12-13 Lower bound of 32bit < 0 timestamp, no extra bits
ok 2 1969-12-31 Upper bound of 32bit < 0 timestamp, no extra bits
ok 3 1970-01-01 Lower bound of 32bit >=0 timestamp, no extra bits
ok 4 2038-01-19 Upper bound of 32bit >=0 timestamp, no extra bits
ok 5 2038-01-19 Lower bound of 32bit <0 timestamp, lo extra sec bit on
ok 6 2106-02-07 Upper bound of 32bit <0 timestamp, lo extra sec bit on
ok 7 2106-02-07 Lower bound of 32bit >=0 timestamp, lo extra sec bit on
ok 8 2174-02-25 Upper bound of 32bit >=0 timestamp, lo extra sec bit on
ok 9 2174-02-25 Lower bound of 32bit <0 timestamp, hi extra sec bit on
ok 10 2242-03-16 Upper bound of 32bit <0 timestamp, hi extra sec bit on
ok 11 2242-03-16 Lower bound of 32bit >=0 timestamp, hi extra sec bit on
ok 12 2310-04-04 Upper bound of 32bit >=0 timestamp, hi extra sec bit on
ok 13 2310-04-04 Upper bound of 32bit>=0 timestamp, hi extra sec bit 1. 1 ns
ok 14 2378-04-22 Lower bound of 32bit>= timestamp. Extra sec bits 1. Max ns
ok 15 2378-04-22 Lower bound of 32bit >=0 timestamp. All extra sec bits on
ok 16 2446-05-10 Upper bound of 32bit >=0 timestamp. All extra sec bits on
# inode_test_xtimestamp_decoding: pass:16 fail:0 skip:0 total:16
ok 1 inode_test_xtimestamp_decoding
# Totals: pass:16 fail:0 skip:0 total:16
ok 2 ext4_inode_test
[18:39:57] Elapsed time: 14.400s total, 2.940s configuring, 11.343s building, 0.074s running
Kemeng Shi (5):
ext4: Add unit test for test_free_blocks_simple
ext4: Add unit test of ext4_mb_generate_buddy
ext4: Add unit test for mb_mark_used
ext4: Add unit test for mb_free_blocks
ext4: Add unit test for ext4_mb_mark_diskspace_used
fs/ext4/mballoc-test.c | 502 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 502 insertions(+)
--
2.30.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v2 1/5] ext4: Add unit test for test_free_blocks_simple
2024-01-03 10:48 [PATCH v2 0/5] More unit test for mballoc Kemeng Shi
@ 2024-01-03 10:48 ` Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy Kemeng Shi
` (5 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Kemeng Shi @ 2024-01-03 10:48 UTC (permalink / raw)
To: tytso, adilger.kernel; +Cc: linux-ext4, linux-kernel
Add unit test for test_free_blocks_simple.
Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
---
fs/ext4/mballoc-test.c | 96 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 96 insertions(+)
diff --git a/fs/ext4/mballoc-test.c b/fs/ext4/mballoc-test.c
index f94901fd3..3aac42ea6 100644
--- a/fs/ext4/mballoc-test.c
+++ b/fs/ext4/mballoc-test.c
@@ -5,6 +5,7 @@
#include <kunit/test.h>
#include <kunit/static_stub.h>
+#include <linux/random.h>
#include "ext4.h"
@@ -112,6 +113,13 @@ static void mbt_ctx_mark_used(struct super_block *sb, ext4_group_t group,
mb_set_bits(grp_ctx->bitmap_bh.b_data, start, len);
}
+static void *mbt_ctx_bitmap(struct super_block *sb, ext4_group_t group)
+{
+ struct mbt_grp_ctx *grp_ctx = MBT_GRP_CTX(sb, group);
+
+ return grp_ctx->bitmap_bh.b_data;
+}
+
/* called after mbt_init_sb_layout */
static int mbt_ctx_init(struct super_block *sb)
{
@@ -297,6 +305,93 @@ static void test_new_blocks_simple(struct kunit *test)
"unexpectedly get block when no block is available");
}
+#define TEST_RANGE_COUNT 8
+
+struct test_range {
+ ext4_grpblk_t start;
+ ext4_grpblk_t len;
+};
+
+static void
+mbt_generate_test_ranges(struct super_block *sb, struct test_range *ranges,
+ int count)
+{
+ ext4_grpblk_t start, len, max;
+ int i;
+
+ max = EXT4_CLUSTERS_PER_GROUP(sb) / count;
+ for (i = 0; i < count; i++) {
+ start = get_random_u32() % max;
+ len = get_random_u32() % max;
+ len = min(len, max - start);
+
+ ranges[i].start = start + i * max;
+ ranges[i].len = len;
+ }
+}
+
+static void
+validate_free_blocks_simple(struct kunit *test, struct super_block *sb,
+ ext4_group_t goal_group, ext4_grpblk_t start,
+ ext4_grpblk_t len)
+{
+ void *bitmap;
+ ext4_grpblk_t bit, max = EXT4_CLUSTERS_PER_GROUP(sb);
+ ext4_group_t i;
+
+ for (i = 0; i < ext4_get_groups_count(sb); i++) {
+ if (i == goal_group)
+ continue;
+
+ bitmap = mbt_ctx_bitmap(sb, i);
+ bit = mb_find_next_zero_bit(bitmap, max, 0);
+ KUNIT_ASSERT_EQ_MSG(test, bit, max,
+ "free block on unexpected group %d", i);
+ }
+
+ bitmap = mbt_ctx_bitmap(sb, goal_group);
+ bit = mb_find_next_zero_bit(bitmap, max, 0);
+ KUNIT_ASSERT_EQ(test, bit, start);
+
+ bit = mb_find_next_bit(bitmap, max, bit + 1);
+ KUNIT_ASSERT_EQ(test, bit, start + len);
+}
+
+static void
+test_free_blocks_simple_range(struct kunit *test, ext4_group_t goal_group,
+ ext4_grpblk_t start, ext4_grpblk_t len)
+{
+ struct super_block *sb = (struct super_block *)test->priv;
+ struct ext4_sb_info *sbi = EXT4_SB(sb);
+ struct inode inode = { .i_sb = sb, };
+ ext4_fsblk_t block;
+
+ if (len == 0)
+ return;
+
+ block = ext4_group_first_block_no(sb, goal_group) +
+ EXT4_C2B(sbi, start);
+ ext4_free_blocks_simple(&inode, block, len);
+ validate_free_blocks_simple(test, sb, goal_group, start, len);
+ mbt_ctx_mark_used(sb, goal_group, 0, EXT4_CLUSTERS_PER_GROUP(sb));
+}
+
+static void test_free_blocks_simple(struct kunit *test)
+{
+ struct super_block *sb = (struct super_block *)test->priv;
+ ext4_grpblk_t max = EXT4_CLUSTERS_PER_GROUP(sb);
+ ext4_group_t i;
+ struct test_range ranges[TEST_RANGE_COUNT];
+
+ for (i = 0; i < ext4_get_groups_count(sb); i++)
+ mbt_ctx_mark_used(sb, i, 0, max);
+
+ mbt_generate_test_ranges(sb, ranges, TEST_RANGE_COUNT);
+ for (i = 0; i < TEST_RANGE_COUNT; i++)
+ test_free_blocks_simple_range(test, TEST_GOAL_GROUP,
+ ranges[i].start, ranges[i].len);
+}
+
static const struct mbt_ext4_block_layout mbt_test_layouts[] = {
{
.blocksize_bits = 10,
@@ -334,6 +429,7 @@ KUNIT_ARRAY_PARAM(mbt_layouts, mbt_test_layouts, mbt_show_layout);
static struct kunit_case mbt_test_cases[] = {
KUNIT_CASE_PARAM(test_new_blocks_simple, mbt_layouts_gen_params),
+ KUNIT_CASE_PARAM(test_free_blocks_simple, mbt_layouts_gen_params),
{}
};
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy
2024-01-03 10:48 [PATCH v2 0/5] More unit test for mballoc Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 1/5] ext4: Add unit test for test_free_blocks_simple Kemeng Shi
@ 2024-01-03 10:48 ` Kemeng Shi
2024-03-20 16:23 ` Guenter Roeck
2024-01-03 10:48 ` [PATCH v2 3/5] ext4: Add unit test for mb_mark_used Kemeng Shi
` (4 subsequent siblings)
6 siblings, 1 reply; 15+ messages in thread
From: Kemeng Shi @ 2024-01-03 10:48 UTC (permalink / raw)
To: tytso, adilger.kernel; +Cc: linux-ext4, linux-kernel
Add unit test of ext4_mb_generate_buddy
Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
---
fs/ext4/mballoc-test.c | 207 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 207 insertions(+)
diff --git a/fs/ext4/mballoc-test.c b/fs/ext4/mballoc-test.c
index 3aac42ea6..6964974fa 100644
--- a/fs/ext4/mballoc-test.c
+++ b/fs/ext4/mballoc-test.c
@@ -28,6 +28,50 @@ struct mbt_ext4_super_block {
#define MBT_CTX(_sb) (&(container_of((_sb), struct mbt_ext4_super_block, sb)->mbt_ctx))
#define MBT_GRP_CTX(_sb, _group) (&MBT_CTX(_sb)->grp_ctx[_group])
+static const struct super_operations mbt_sops = {
+};
+
+static int mbt_mb_init(struct super_block *sb)
+{
+ int ret;
+
+ /* needed by ext4_mb_init->bdev_nonrot(sb->s_bdev) */
+ sb->s_bdev = kzalloc(sizeof(*sb->s_bdev), GFP_KERNEL);
+ if (sb->s_bdev == NULL)
+ return -ENOMEM;
+
+ sb->s_bdev->bd_queue = kzalloc(sizeof(struct request_queue), GFP_KERNEL);
+ if (sb->s_bdev->bd_queue == NULL) {
+ kfree(sb->s_bdev);
+ return -ENOMEM;
+ }
+
+ /*
+ * needed by ext4_mb_init->ext4_mb_init_backend-> sbi->s_buddy_cache =
+ * new_inode(sb);
+ */
+ INIT_LIST_HEAD(&sb->s_inodes);
+ sb->s_op = &mbt_sops;
+
+ ret = ext4_mb_init(sb);
+ if (ret != 0)
+ goto err_out;
+
+ return 0;
+
+err_out:
+ kfree(sb->s_bdev->bd_queue);
+ kfree(sb->s_bdev);
+ return ret;
+}
+
+static void mbt_mb_release(struct super_block *sb)
+{
+ ext4_mb_release(sb);
+ kfree(sb->s_bdev->bd_queue);
+ kfree(sb->s_bdev);
+}
+
static struct super_block *mbt_ext4_alloc_super_block(void)
{
struct ext4_super_block *es = kzalloc(sizeof(*es), GFP_KERNEL);
@@ -37,8 +81,16 @@ static struct super_block *mbt_ext4_alloc_super_block(void)
if (fsb == NULL || sbi == NULL || es == NULL)
goto out;
+ sbi->s_blockgroup_lock =
+ kzalloc(sizeof(struct blockgroup_lock), GFP_KERNEL);
+ if (!sbi->s_blockgroup_lock)
+ goto out;
+
+ bgl_lock_init(sbi->s_blockgroup_lock);
+
sbi->s_es = es;
fsb->sb.s_fs_info = sbi;
+
return &fsb->sb;
out:
@@ -54,6 +106,7 @@ static void mbt_ext4_free_super_block(struct super_block *sb)
container_of(sb, struct mbt_ext4_super_block, sb);
struct ext4_sb_info *sbi = EXT4_SB(sb);
+ kfree(sbi->s_blockgroup_lock);
kfree(sbi->s_es);
kfree(sbi);
kfree(fsb);
@@ -83,6 +136,9 @@ static void mbt_init_sb_layout(struct super_block *sb,
sbi->s_clusters_per_group = layout->blocks_per_group >>
layout->cluster_bits;
sbi->s_desc_size = layout->desc_size;
+ sbi->s_desc_per_block_bits =
+ sb->s_blocksize_bits - (fls(layout->desc_size) - 1);
+ sbi->s_desc_per_block = 1 << sbi->s_desc_per_block_bits;
es->s_first_data_block = cpu_to_le32(0);
es->s_blocks_count_lo = cpu_to_le32(layout->blocks_per_group *
@@ -240,6 +296,14 @@ static int mbt_kunit_init(struct kunit *test)
kunit_activate_static_stub(test,
ext4_mb_mark_context,
ext4_mb_mark_context_stub);
+
+ /* stub function will be called in mt_mb_init->ext4_mb_init */
+ if (mbt_mb_init(sb) != 0) {
+ mbt_ctx_release(sb);
+ mbt_ext4_free_super_block(sb);
+ return -ENOMEM;
+ }
+
return 0;
}
@@ -247,6 +311,7 @@ static void mbt_kunit_exit(struct kunit *test)
{
struct super_block *sb = (struct super_block *)test->priv;
+ mbt_mb_release(sb);
mbt_ctx_release(sb);
mbt_ext4_free_super_block(sb);
}
@@ -392,6 +457,147 @@ static void test_free_blocks_simple(struct kunit *test)
ranges[i].start, ranges[i].len);
}
+static void mbt_generate_buddy(struct super_block *sb, void *buddy,
+ void *bitmap, struct ext4_group_info *grp)
+{
+ struct ext4_sb_info *sbi = EXT4_SB(sb);
+ uint32_t order, off;
+ void *bb, *bb_h;
+ int max;
+
+ memset(buddy, 0xff, sb->s_blocksize);
+ memset(grp, 0, offsetof(struct ext4_group_info,
+ bb_counters[MB_NUM_ORDERS(sb)]));
+
+ bb = bitmap;
+ max = EXT4_CLUSTERS_PER_GROUP(sb);
+ bb_h = buddy + sbi->s_mb_offsets[1];
+
+ off = mb_find_next_zero_bit(bb, max, 0);
+ grp->bb_first_free = off;
+ while (off < max) {
+ grp->bb_counters[0]++;
+ grp->bb_free++;
+
+ if (!(off & 1) && !mb_test_bit(off + 1, bb)) {
+ grp->bb_free++;
+ grp->bb_counters[0]--;
+ mb_clear_bit(off >> 1, bb_h);
+ grp->bb_counters[1]++;
+ grp->bb_largest_free_order = 1;
+ off++;
+ }
+
+ off = mb_find_next_zero_bit(bb, max, off + 1);
+ }
+
+ for (order = 1; order < MB_NUM_ORDERS(sb) - 1; order++) {
+ bb = buddy + sbi->s_mb_offsets[order];
+ bb_h = buddy + sbi->s_mb_offsets[order + 1];
+ max = max >> 1;
+ off = mb_find_next_zero_bit(bb, max, 0);
+
+ while (off < max) {
+ if (!(off & 1) && !mb_test_bit(off + 1, bb)) {
+ mb_set_bits(bb, off, 2);
+ grp->bb_counters[order] -= 2;
+ mb_clear_bit(off >> 1, bb_h);
+ grp->bb_counters[order + 1]++;
+ grp->bb_largest_free_order = order + 1;
+ off++;
+ }
+
+ off = mb_find_next_zero_bit(bb, max, off + 1);
+ }
+ }
+
+ max = EXT4_CLUSTERS_PER_GROUP(sb);
+ off = mb_find_next_zero_bit(bitmap, max, 0);
+ while (off < max) {
+ grp->bb_fragments++;
+
+ off = mb_find_next_bit(bitmap, max, off + 1);
+ if (off + 1 >= max)
+ break;
+
+ off = mb_find_next_zero_bit(bitmap, max, off + 1);
+ }
+}
+
+static void
+mbt_validate_group_info(struct kunit *test, struct ext4_group_info *grp1,
+ struct ext4_group_info *grp2)
+{
+ struct super_block *sb = (struct super_block *)test->priv;
+ int i;
+
+ KUNIT_ASSERT_EQ(test, grp1->bb_first_free,
+ grp2->bb_first_free);
+ KUNIT_ASSERT_EQ(test, grp1->bb_fragments,
+ grp2->bb_fragments);
+ KUNIT_ASSERT_EQ(test, grp1->bb_free, grp2->bb_free);
+ KUNIT_ASSERT_EQ(test, grp1->bb_largest_free_order,
+ grp2->bb_largest_free_order);
+
+ for (i = 1; i < MB_NUM_ORDERS(sb); i++) {
+ KUNIT_ASSERT_EQ_MSG(test, grp1->bb_counters[i],
+ grp2->bb_counters[i],
+ "bb_counters[%d] diffs, expected %d, generated %d",
+ i, grp1->bb_counters[i],
+ grp2->bb_counters[i]);
+ }
+}
+
+static void
+do_test_generate_buddy(struct kunit *test, struct super_block *sb, void *bitmap,
+ void *mbt_buddy, struct ext4_group_info *mbt_grp,
+ void *ext4_buddy, struct ext4_group_info *ext4_grp)
+{
+ int i;
+
+ mbt_generate_buddy(sb, mbt_buddy, bitmap, mbt_grp);
+
+ for (i = 0; i < MB_NUM_ORDERS(sb); i++)
+ ext4_grp->bb_counters[i] = 0;
+ /* needed by validation in ext4_mb_generate_buddy */
+ ext4_grp->bb_free = mbt_grp->bb_free;
+ memset(ext4_buddy, 0xff, sb->s_blocksize);
+ ext4_mb_generate_buddy(sb, ext4_buddy, bitmap, TEST_GOAL_GROUP,
+ ext4_grp);
+
+ KUNIT_ASSERT_EQ(test, memcmp(mbt_buddy, ext4_buddy, sb->s_blocksize),
+ 0);
+ mbt_validate_group_info(test, mbt_grp, ext4_grp);
+}
+
+static void test_mb_generate_buddy(struct kunit *test)
+{
+ struct super_block *sb = (struct super_block *)test->priv;
+ void *bitmap, *expected_bb, *generate_bb;
+ struct ext4_group_info *expected_grp, *generate_grp;
+ struct test_range ranges[TEST_RANGE_COUNT];
+ int i;
+
+ bitmap = kunit_kzalloc(test, sb->s_blocksize, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bitmap);
+ expected_bb = kunit_kzalloc(test, sb->s_blocksize, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, expected_bb);
+ generate_bb = kunit_kzalloc(test, sb->s_blocksize, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, generate_bb);
+ expected_grp = kunit_kzalloc(test, offsetof(struct ext4_group_info,
+ bb_counters[MB_NUM_ORDERS(sb)]), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, expected_grp);
+ generate_grp = ext4_get_group_info(sb, TEST_GOAL_GROUP);
+ KUNIT_ASSERT_NOT_NULL(test, generate_grp);
+
+ mbt_generate_test_ranges(sb, ranges, TEST_RANGE_COUNT);
+ for (i = 0; i < TEST_RANGE_COUNT; i++) {
+ mb_set_bits(bitmap, ranges[i].start, ranges[i].len);
+ do_test_generate_buddy(test, sb, bitmap, expected_bb,
+ expected_grp, generate_bb, generate_grp);
+ }
+}
+
static const struct mbt_ext4_block_layout mbt_test_layouts[] = {
{
.blocksize_bits = 10,
@@ -430,6 +636,7 @@ KUNIT_ARRAY_PARAM(mbt_layouts, mbt_test_layouts, mbt_show_layout);
static struct kunit_case mbt_test_cases[] = {
KUNIT_CASE_PARAM(test_new_blocks_simple, mbt_layouts_gen_params),
KUNIT_CASE_PARAM(test_free_blocks_simple, mbt_layouts_gen_params),
+ KUNIT_CASE_PARAM(test_mb_generate_buddy, mbt_layouts_gen_params),
{}
};
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 3/5] ext4: Add unit test for mb_mark_used
2024-01-03 10:48 [PATCH v2 0/5] More unit test for mballoc Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 1/5] ext4: Add unit test for test_free_blocks_simple Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy Kemeng Shi
@ 2024-01-03 10:48 ` Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 4/5] ext4: Add unit test for mb_free_blocks Kemeng Shi
` (3 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Kemeng Shi @ 2024-01-03 10:48 UTC (permalink / raw)
To: tytso, adilger.kernel; +Cc: linux-ext4, linux-kernel
Add unit test for mb_mark_used
Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
---
fs/ext4/mballoc-test.c | 78 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 78 insertions(+)
diff --git a/fs/ext4/mballoc-test.c b/fs/ext4/mballoc-test.c
index 6964974fa..38e3644cb 100644
--- a/fs/ext4/mballoc-test.c
+++ b/fs/ext4/mballoc-test.c
@@ -148,9 +148,13 @@ static void mbt_init_sb_layout(struct super_block *sb,
static int mbt_grp_ctx_init(struct super_block *sb,
struct mbt_grp_ctx *grp_ctx)
{
+ ext4_grpblk_t max = EXT4_CLUSTERS_PER_GROUP(sb);
+
grp_ctx->bitmap_bh.b_data = kzalloc(EXT4_BLOCK_SIZE(sb), GFP_KERNEL);
if (grp_ctx->bitmap_bh.b_data == NULL)
return -ENOMEM;
+ mb_set_bits(grp_ctx->bitmap_bh.b_data, max, sb->s_blocksize * 8 - max);
+ ext4_free_group_clusters_set(sb, &grp_ctx->desc, max);
return 0;
}
@@ -197,6 +201,8 @@ static int mbt_ctx_init(struct super_block *sb)
* block which will fail ext4_sb_block_valid check.
*/
mb_set_bits(ctx->grp_ctx[0].bitmap_bh.b_data, 0, 1);
+ ext4_free_group_clusters_set(sb, &ctx->grp_ctx[0].desc,
+ EXT4_CLUSTERS_PER_GROUP(sb) - 1);
return 0;
out:
@@ -231,6 +237,13 @@ static int ext4_wait_block_bitmap_stub(struct super_block *sb,
ext4_group_t block_group,
struct buffer_head *bh)
{
+ /*
+ * real ext4_wait_block_bitmap will set these flags and
+ * functions like ext4_mb_init_cache will verify the flags.
+ */
+ set_buffer_uptodate(bh);
+ set_bitmap_uptodate(bh);
+ set_buffer_verified(bh);
return 0;
}
@@ -598,6 +611,70 @@ static void test_mb_generate_buddy(struct kunit *test)
}
}
+static void
+test_mb_mark_used_range(struct kunit *test, struct ext4_buddy *e4b,
+ ext4_grpblk_t start, ext4_grpblk_t len, void *bitmap,
+ void *buddy, struct ext4_group_info *grp)
+{
+ struct super_block *sb = (struct super_block *)test->priv;
+ struct ext4_free_extent ex;
+ int i;
+
+ /* mb_mark_used only accepts non-zero len */
+ if (len == 0)
+ return;
+
+ ex.fe_start = start;
+ ex.fe_len = len;
+ ex.fe_group = TEST_GOAL_GROUP;
+ mb_mark_used(e4b, &ex);
+
+ mb_set_bits(bitmap, start, len);
+ /* bypass bb_free validatoin in ext4_mb_generate_buddy */
+ grp->bb_free -= len;
+ memset(buddy, 0xff, sb->s_blocksize);
+ for (i = 0; i < MB_NUM_ORDERS(sb); i++)
+ grp->bb_counters[i] = 0;
+ ext4_mb_generate_buddy(sb, buddy, bitmap, 0, grp);
+
+ KUNIT_ASSERT_EQ(test, memcmp(buddy, e4b->bd_buddy, sb->s_blocksize),
+ 0);
+ mbt_validate_group_info(test, grp, e4b->bd_info);
+}
+
+static void test_mb_mark_used(struct kunit *test)
+{
+ struct ext4_buddy e4b;
+ struct super_block *sb = (struct super_block *)test->priv;
+ void *bitmap, *buddy;
+ struct ext4_group_info *grp;
+ int ret;
+ struct test_range ranges[TEST_RANGE_COUNT];
+ int i;
+
+ /* buddy cache assumes that each page contains at least one block */
+ if (sb->s_blocksize > PAGE_SIZE)
+ kunit_skip(test, "blocksize exceeds pagesize");
+
+ bitmap = kunit_kzalloc(test, sb->s_blocksize, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bitmap);
+ buddy = kunit_kzalloc(test, sb->s_blocksize, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buddy);
+ grp = kunit_kzalloc(test, offsetof(struct ext4_group_info,
+ bb_counters[MB_NUM_ORDERS(sb)]), GFP_KERNEL);
+
+ ret = ext4_mb_load_buddy(sb, TEST_GOAL_GROUP, &e4b);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+
+ grp->bb_free = EXT4_CLUSTERS_PER_GROUP(sb);
+ mbt_generate_test_ranges(sb, ranges, TEST_RANGE_COUNT);
+ for (i = 0; i < TEST_RANGE_COUNT; i++)
+ test_mb_mark_used_range(test, &e4b, ranges[i].start,
+ ranges[i].len, bitmap, buddy, grp);
+
+ ext4_mb_unload_buddy(&e4b);
+}
+
static const struct mbt_ext4_block_layout mbt_test_layouts[] = {
{
.blocksize_bits = 10,
@@ -637,6 +714,7 @@ static struct kunit_case mbt_test_cases[] = {
KUNIT_CASE_PARAM(test_new_blocks_simple, mbt_layouts_gen_params),
KUNIT_CASE_PARAM(test_free_blocks_simple, mbt_layouts_gen_params),
KUNIT_CASE_PARAM(test_mb_generate_buddy, mbt_layouts_gen_params),
+ KUNIT_CASE_PARAM(test_mb_mark_used, mbt_layouts_gen_params),
{}
};
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 4/5] ext4: Add unit test for mb_free_blocks
2024-01-03 10:48 [PATCH v2 0/5] More unit test for mballoc Kemeng Shi
` (2 preceding siblings ...)
2024-01-03 10:48 ` [PATCH v2 3/5] ext4: Add unit test for mb_mark_used Kemeng Shi
@ 2024-01-03 10:48 ` Kemeng Shi
2024-01-03 10:49 ` [PATCH v2 5/5] ext4: Add unit test for ext4_mb_mark_diskspace_used Kemeng Shi
` (2 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Kemeng Shi @ 2024-01-03 10:48 UTC (permalink / raw)
To: tytso, adilger.kernel; +Cc: linux-ext4, linux-kernel
Add unit test for mb_free_blocks.
Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
---
fs/ext4/mballoc-test.c | 69 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 69 insertions(+)
diff --git a/fs/ext4/mballoc-test.c b/fs/ext4/mballoc-test.c
index 38e3644cb..b68f44740 100644
--- a/fs/ext4/mballoc-test.c
+++ b/fs/ext4/mballoc-test.c
@@ -675,6 +675,74 @@ static void test_mb_mark_used(struct kunit *test)
ext4_mb_unload_buddy(&e4b);
}
+static void
+test_mb_free_blocks_range(struct kunit *test, struct ext4_buddy *e4b,
+ ext4_grpblk_t start, ext4_grpblk_t len, void *bitmap,
+ void *buddy, struct ext4_group_info *grp)
+{
+ struct super_block *sb = (struct super_block *)test->priv;
+ int i;
+
+ /* mb_free_blocks will WARN if len is 0 */
+ if (len == 0)
+ return;
+
+ mb_free_blocks(NULL, e4b, start, len);
+
+ mb_clear_bits(bitmap, start, len);
+ /* bypass bb_free validatoin in ext4_mb_generate_buddy */
+ grp->bb_free += len;
+ memset(buddy, 0xff, sb->s_blocksize);
+ for (i = 0; i < MB_NUM_ORDERS(sb); i++)
+ grp->bb_counters[i] = 0;
+ ext4_mb_generate_buddy(sb, buddy, bitmap, 0, grp);
+
+ KUNIT_ASSERT_EQ(test, memcmp(buddy, e4b->bd_buddy, sb->s_blocksize),
+ 0);
+ mbt_validate_group_info(test, grp, e4b->bd_info);
+
+}
+
+static void test_mb_free_blocks(struct kunit *test)
+{
+ struct ext4_buddy e4b;
+ struct super_block *sb = (struct super_block *)test->priv;
+ void *bitmap, *buddy;
+ struct ext4_group_info *grp;
+ struct ext4_free_extent ex;
+ int ret;
+ int i;
+ struct test_range ranges[TEST_RANGE_COUNT];
+
+ /* buddy cache assumes that each page contains at least one block */
+ if (sb->s_blocksize > PAGE_SIZE)
+ kunit_skip(test, "blocksize exceeds pagesize");
+
+ bitmap = kunit_kzalloc(test, sb->s_blocksize, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bitmap);
+ buddy = kunit_kzalloc(test, sb->s_blocksize, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buddy);
+ grp = kunit_kzalloc(test, offsetof(struct ext4_group_info,
+ bb_counters[MB_NUM_ORDERS(sb)]), GFP_KERNEL);
+
+ ret = ext4_mb_load_buddy(sb, TEST_GOAL_GROUP, &e4b);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+
+ ex.fe_start = 0;
+ ex.fe_len = EXT4_CLUSTERS_PER_GROUP(sb);
+ ex.fe_group = TEST_GOAL_GROUP;
+ mb_mark_used(&e4b, &ex);
+ grp->bb_free = 0;
+ memset(bitmap, 0xff, sb->s_blocksize);
+
+ mbt_generate_test_ranges(sb, ranges, TEST_RANGE_COUNT);
+ for (i = 0; i < TEST_RANGE_COUNT; i++)
+ test_mb_free_blocks_range(test, &e4b, ranges[i].start,
+ ranges[i].len, bitmap, buddy, grp);
+
+ ext4_mb_unload_buddy(&e4b);
+}
+
static const struct mbt_ext4_block_layout mbt_test_layouts[] = {
{
.blocksize_bits = 10,
@@ -715,6 +783,7 @@ static struct kunit_case mbt_test_cases[] = {
KUNIT_CASE_PARAM(test_free_blocks_simple, mbt_layouts_gen_params),
KUNIT_CASE_PARAM(test_mb_generate_buddy, mbt_layouts_gen_params),
KUNIT_CASE_PARAM(test_mb_mark_used, mbt_layouts_gen_params),
+ KUNIT_CASE_PARAM(test_mb_free_blocks, mbt_layouts_gen_params),
{}
};
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 5/5] ext4: Add unit test for ext4_mb_mark_diskspace_used
2024-01-03 10:48 [PATCH v2 0/5] More unit test for mballoc Kemeng Shi
` (3 preceding siblings ...)
2024-01-03 10:48 ` [PATCH v2 4/5] ext4: Add unit test for mb_free_blocks Kemeng Shi
@ 2024-01-03 10:49 ` Kemeng Shi
2024-02-22 15:54 ` [PATCH v2 0/5] More unit test for mballoc Theodore Ts'o
2024-02-25 16:07 ` Guenter Roeck
6 siblings, 0 replies; 15+ messages in thread
From: Kemeng Shi @ 2024-01-03 10:49 UTC (permalink / raw)
To: tytso, adilger.kernel; +Cc: linux-ext4, linux-kernel
Add unit test for ext4_mb_mark_diskspace_used
Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
---
fs/ext4/mballoc-test.c | 52 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 52 insertions(+)
diff --git a/fs/ext4/mballoc-test.c b/fs/ext4/mballoc-test.c
index b68f44740..12d0b22ca 100644
--- a/fs/ext4/mballoc-test.c
+++ b/fs/ext4/mballoc-test.c
@@ -470,6 +470,57 @@ static void test_free_blocks_simple(struct kunit *test)
ranges[i].start, ranges[i].len);
}
+static void
+test_mark_diskspace_used_range(struct kunit *test,
+ struct ext4_allocation_context *ac,
+ ext4_grpblk_t start,
+ ext4_grpblk_t len)
+{
+ struct super_block *sb = (struct super_block *)test->priv;
+ int ret;
+ void *bitmap;
+ ext4_grpblk_t i, max;
+
+ /* ext4_mb_mark_diskspace_used will BUG if len is 0 */
+ if (len == 0)
+ return;
+
+ ac->ac_b_ex.fe_group = TEST_GOAL_GROUP;
+ ac->ac_b_ex.fe_start = start;
+ ac->ac_b_ex.fe_len = len;
+
+ bitmap = mbt_ctx_bitmap(sb, TEST_GOAL_GROUP);
+ memset(bitmap, 0, sb->s_blocksize);
+ ret = ext4_mb_mark_diskspace_used(ac, NULL, 0);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+
+ max = EXT4_CLUSTERS_PER_GROUP(sb);
+ i = mb_find_next_bit(bitmap, max, 0);
+ KUNIT_ASSERT_EQ(test, i, start);
+ i = mb_find_next_zero_bit(bitmap, max, i + 1);
+ KUNIT_ASSERT_EQ(test, i, start + len);
+ i = mb_find_next_bit(bitmap, max, i + 1);
+ KUNIT_ASSERT_EQ(test, max, i);
+}
+
+static void test_mark_diskspace_used(struct kunit *test)
+{
+ struct super_block *sb = (struct super_block *)test->priv;
+ struct inode inode = { .i_sb = sb, };
+ struct ext4_allocation_context ac;
+ struct test_range ranges[TEST_RANGE_COUNT];
+ int i;
+
+ mbt_generate_test_ranges(sb, ranges, TEST_RANGE_COUNT);
+
+ ac.ac_status = AC_STATUS_FOUND;
+ ac.ac_sb = sb;
+ ac.ac_inode = &inode;
+ for (i = 0; i < TEST_RANGE_COUNT; i++)
+ test_mark_diskspace_used_range(test, &ac, ranges[i].start,
+ ranges[i].len);
+}
+
static void mbt_generate_buddy(struct super_block *sb, void *buddy,
void *bitmap, struct ext4_group_info *grp)
{
@@ -784,6 +835,7 @@ static struct kunit_case mbt_test_cases[] = {
KUNIT_CASE_PARAM(test_mb_generate_buddy, mbt_layouts_gen_params),
KUNIT_CASE_PARAM(test_mb_mark_used, mbt_layouts_gen_params),
KUNIT_CASE_PARAM(test_mb_free_blocks, mbt_layouts_gen_params),
+ KUNIT_CASE_PARAM(test_mark_diskspace_used, mbt_layouts_gen_params),
{}
};
--
2.30.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v2 0/5] More unit test for mballoc
2024-01-03 10:48 [PATCH v2 0/5] More unit test for mballoc Kemeng Shi
` (4 preceding siblings ...)
2024-01-03 10:49 ` [PATCH v2 5/5] ext4: Add unit test for ext4_mb_mark_diskspace_used Kemeng Shi
@ 2024-02-22 15:54 ` Theodore Ts'o
2024-02-25 16:07 ` Guenter Roeck
6 siblings, 0 replies; 15+ messages in thread
From: Theodore Ts'o @ 2024-02-22 15:54 UTC (permalink / raw)
To: adilger.kernel, Kemeng Shi; +Cc: Theodore Ts'o, linux-ext4, linux-kernel
On Wed, 03 Jan 2024 18:48:55 +0800, Kemeng Shi wrote:
> This series covers more function to mark on-disk bitmap. Besides, some
> code which is relevant to buddy cache is also tested.
> Before more work is done, I want to be sure I'm not on a wrong
> direction!
>
> v1->v2:
> -Fix unused variable warning which is reported at
> https://lore.kernel.org/lkml/202311260042.kMxL6DnL-lkp@intel.com/T/
>
> [...]
Applied, thanks!
[1/5] ext4: Add unit test for test_free_blocks_simple
commit: 6c5e0c9c21456fb561d0997fe2c4d3cf59745ba7
[2/5] ext4: Add unit test of ext4_mb_generate_buddy
commit: 67d2a11b22b4d520072db62620851615df13183e
[3/5] ext4: Add unit test for mb_mark_used
commit: ac96b56a2fbd9e05b28488bdc5d3bd8006b61d35
[4/5] ext4: Add unit test for mb_free_blocks
commit: b7098e1fa7bcf350e089af9500c4f9992a1bd6cb
[5/5] ext4: Add unit test for ext4_mb_mark_diskspace_used
commit: 2b81493f8eb6fc0c263dbca0064e10e4c00e0f91
Best regards,
--
Theodore Ts'o <tytso@mit.edu>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 0/5] More unit test for mballoc
2024-01-03 10:48 [PATCH v2 0/5] More unit test for mballoc Kemeng Shi
` (5 preceding siblings ...)
2024-02-22 15:54 ` [PATCH v2 0/5] More unit test for mballoc Theodore Ts'o
@ 2024-02-25 16:07 ` Guenter Roeck
2024-02-26 13:42 ` Kemeng Shi
6 siblings, 1 reply; 15+ messages in thread
From: Guenter Roeck @ 2024-02-25 16:07 UTC (permalink / raw)
To: Kemeng Shi; +Cc: tytso, adilger.kernel, linux-ext4, linux-kernel
On Wed, Jan 03, 2024 at 06:48:55PM +0800, Kemeng Shi wrote:
> This series covers more function to mark on-disk bitmap. Besides, some
> code which is relevant to buddy cache is also tested.
> Before more work is done, I want to be sure I'm not on a wrong
> direction!
>
> v1->v2:
> -Fix unused variable warning which is reported at
> https://lore.kernel.org/lkml/202311260042.kMxL6DnL-lkp@intel.com/T/
>
With this patch series in linux-text, and with various debug options
enabled, unit tests for ext4 fail widely and result in crashes.
[ 4.798582] # Subtest: test_new_blocks_simple
[ 4.803166] BUG: key 00000000000000a8 has not been registered!
[ 4.803443] ------------[ cut here ]------------
ILLOPC: ffffffff9050cbf4: 0f 0b
[ 4.803521] DEBUG_LOCKS_WARN_ON(1)
[ 4.803720] WARNING: CPU: 0 PID: 154 at kernel/locking/lockdep.c:4895 lockdep_init_map_type+0x224/0x250
...
[ 4.833277] # Subtest: test_mb_mark_used
[ 4.835875] ------------[ cut here ]------------
[ 4.836055] kernel BUG at fs/ext4/mballoc.c:2053!
Guenter
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 0/5] More unit test for mballoc
2024-02-25 16:07 ` Guenter Roeck
@ 2024-02-26 13:42 ` Kemeng Shi
0 siblings, 0 replies; 15+ messages in thread
From: Kemeng Shi @ 2024-02-26 13:42 UTC (permalink / raw)
To: Guenter Roeck; +Cc: tytso, adilger.kernel, linux-ext4, linux-kernel
on 2/26/2024 12:07 AM, Guenter Roeck wrote:
> On Wed, Jan 03, 2024 at 06:48:55PM +0800, Kemeng Shi wrote:
>> This series covers more function to mark on-disk bitmap. Besides, some
>> code which is relevant to buddy cache is also tested.
>> Before more work is done, I want to be sure I'm not on a wrong
>> direction!
>>
>> v1->v2:
>> -Fix unused variable warning which is reported at
>> https://lore.kernel.org/lkml/202311260042.kMxL6DnL-lkp@intel.com/T/
>>
>
> With this patch series in linux-text, and with various debug options
> enabled, unit tests for ext4 fail widely and result in crashes.
>
> [ 4.798582] # Subtest: test_new_blocks_simple
> [ 4.803166] BUG: key 00000000000000a8 has not been registered!
> [ 4.803443] ------------[ cut here ]------------
> ILLOPC: ffffffff9050cbf4: 0f 0b
> [ 4.803521] DEBUG_LOCKS_WARN_ON(1)
> [ 4.803720] WARNING: CPU: 0 PID: 154 at kernel/locking/lockdep.c:4895 lockdep_init_map_type+0x224/0x250
Not sure how this is triggerred.
> ...
> [ 4.833277] # Subtest: test_mb_mark_used
> [ 4.835875] ------------[ cut here ]------------
> [ 4.836055] kernel BUG at fs/ext4/mballoc.c:2053!
Internal functions mb_mark_used assumes group lock is held but unit
test code doesn't acquire it as there is no concurrent block
allocation/free in test code. Will add lock in unit test to fix this.
>
> Guenter
>
Hi Guenter, thanks for test and report this. I will appreciate it if
you could tell me how to reproduce this as I need to debug the first
issue and verify the fix. Thanks!
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy
2024-01-03 10:48 ` [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy Kemeng Shi
@ 2024-03-20 16:23 ` Guenter Roeck
2024-03-20 18:42 ` Guenter Roeck
2024-03-21 7:16 ` Kemeng Shi
0 siblings, 2 replies; 15+ messages in thread
From: Guenter Roeck @ 2024-03-20 16:23 UTC (permalink / raw)
To: Kemeng Shi; +Cc: tytso, adilger.kernel, linux-ext4, linux-kernel
Hi,
On Wed, Jan 03, 2024 at 06:48:57PM +0800, Kemeng Shi wrote:
> Add unit test of ext4_mb_generate_buddy
>
> Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
With this and other new ext4 tests test in the tree, I see a variety
of backtraces in the upstream kernel if debug options are enabled.
An example is
[ 6.821447] KTAP version 1
[ 6.821769] # Subtest: test_mb_generate_buddy
[ 6.824787] =============================================================================
[ 6.825568] BUG inode_cache (Tainted: G N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
...
[ 6.894341] ok 7 ext4_inode_test
[ 6.895411] =============================================================================
[ 6.895777] BUG inode_cache (Tainted: G B N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
Another example, from another test run, is
[ 3.938551] # Subtest: test_new_blocks_simple
[ 3.947171] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
[ 3.952988] ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
[ 3.958403] ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
[ 3.958890] =============================================================================
[ 3.959159] BUG inode_cache (Tainted: G N): Padding overwritten. 0xffff8de881adbf68-0xffff8de881adbf6f @offset=16232
Another one:
[ 18.730473] # Subtest: test_new_blocks_simple
[ 18.760547] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
[ 18.778477] ==================================================================
[ 18.778950] BUG: KFENCE: out-of-bounds write in ext4_mb_init+0x5d7/0xa60
This is just a sample, taken from a quick look at test results.
Are those backtraces expected ? If so, would it be possible to execute the
tests without generating such backtraces ? The backtraces, if intentional,
hide real problems in the noise.
Thanks,
Guenter
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy
2024-03-20 16:23 ` Guenter Roeck
@ 2024-03-20 18:42 ` Guenter Roeck
2024-03-21 7:16 ` Kemeng Shi
1 sibling, 0 replies; 15+ messages in thread
From: Guenter Roeck @ 2024-03-20 18:42 UTC (permalink / raw)
To: Kemeng Shi
Cc: tytso, adilger.kernel, linux-ext4, linux-kernel, daniel.diaz,
vmartensson
[ Adding more interested parties]
On Wed, Mar 20, 2024 at 09:23:24AM -0700, Guenter Roeck wrote:
> Hi,
>
> On Wed, Jan 03, 2024 at 06:48:57PM +0800, Kemeng Shi wrote:
> > Add unit test of ext4_mb_generate_buddy
> >
> > Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
>
> With this and other new ext4 tests test in the tree, I see a variety
> of backtraces in the upstream kernel if debug options are enabled.
> An example is
>
> [ 6.821447] KTAP version 1
> [ 6.821769] # Subtest: test_mb_generate_buddy
> [ 6.824787] =============================================================================
> [ 6.825568] BUG inode_cache (Tainted: G N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
> ...
> [ 6.894341] ok 7 ext4_inode_test
> [ 6.895411] =============================================================================
> [ 6.895777] BUG inode_cache (Tainted: G B N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
>
> Another example, from another test run, is
>
> [ 3.938551] # Subtest: test_new_blocks_simple
> [ 3.947171] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
> [ 3.952988] ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
> [ 3.958403] ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
> [ 3.958890] =============================================================================
> [ 3.959159] BUG inode_cache (Tainted: G N): Padding overwritten. 0xffff8de881adbf68-0xffff8de881adbf6f @offset=16232
>
> Another one:
>
> [ 18.730473] # Subtest: test_new_blocks_simple
> [ 18.760547] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
> [ 18.778477] ==================================================================
> [ 18.778950] BUG: KFENCE: out-of-bounds write in ext4_mb_init+0x5d7/0xa60
>
> This is just a sample, taken from a quick look at test results.
>
> Are those backtraces expected ? If so, would it be possible to execute the
> tests without generating such backtraces ? The backtraces, if intentional,
> hide real problems in the noise.
>
> Thanks,
> Guenter
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy
2024-03-20 16:23 ` Guenter Roeck
2024-03-20 18:42 ` Guenter Roeck
@ 2024-03-21 7:16 ` Kemeng Shi
2024-03-22 9:27 ` Kemeng Shi
1 sibling, 1 reply; 15+ messages in thread
From: Kemeng Shi @ 2024-03-21 7:16 UTC (permalink / raw)
To: Guenter Roeck; +Cc: tytso, adilger.kernel, linux-ext4, linux-kernel
on 3/21/2024 12:23 AM, Guenter Roeck wrote:
> Hi,
>
> On Wed, Jan 03, 2024 at 06:48:57PM +0800, Kemeng Shi wrote:
>> Add unit test of ext4_mb_generate_buddy
>>
>> Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
>
> With this and other new ext4 tests test in the tree, I see a variety
> of backtraces in the upstream kernel if debug options are enabled.
> An example is
>
> [ 6.821447] KTAP version 1
> [ 6.821769] # Subtest: test_mb_generate_buddy
> [ 6.824787] =============================================================================
> [ 6.825568] BUG inode_cache (Tainted: G N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
> ...
> [ 6.894341] ok 7 ext4_inode_test
> [ 6.895411] =============================================================================
> [ 6.895777] BUG inode_cache (Tainted: G B N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
>
> Another example, from another test run, is
>
> [ 3.938551] # Subtest: test_new_blocks_simple
> [ 3.947171] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
> [ 3.952988] ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
> [ 3.958403] ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
> [ 3.958890] =============================================================================
> [ 3.959159] BUG inode_cache (Tainted: G N): Padding overwritten. 0xffff8de881adbf68-0xffff8de881adbf6f @offset=16232
>
> Another one:
>
> [ 18.730473] # Subtest: test_new_blocks_simple
> [ 18.760547] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
> [ 18.778477] ==================================================================
> [ 18.778950] BUG: KFENCE: out-of-bounds write in ext4_mb_init+0x5d7/0xa60
>
> This is just a sample, taken from a quick look at test results.
>
> Are those backtraces expected ? If so, would it be possible to execute the
> tests without generating such backtraces ? The backtraces, if intentional,
> hide real problems in the noise.
Thanks for the report. The backtrace is not expected, I will look into this. Thansk!
>
> Thanks,
> Guenter
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy
2024-03-21 7:16 ` Kemeng Shi
@ 2024-03-22 9:27 ` Kemeng Shi
2024-03-22 14:49 ` Guenter Roeck
0 siblings, 1 reply; 15+ messages in thread
From: Kemeng Shi @ 2024-03-22 9:27 UTC (permalink / raw)
To: Guenter Roeck; +Cc: tytso, adilger.kernel, linux-ext4, linux-kernel
on 3/21/2024 3:16 PM, Kemeng Shi wrote:
>
>
> on 3/21/2024 12:23 AM, Guenter Roeck wrote:
>> Hi,
>>
>> On Wed, Jan 03, 2024 at 06:48:57PM +0800, Kemeng Shi wrote:
>>> Add unit test of ext4_mb_generate_buddy
>>>
>>> Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
>>
>> With this and other new ext4 tests test in the tree, I see a variety
>> of backtraces in the upstream kernel if debug options are enabled.
>> An example is
>>
>> [ 6.821447] KTAP version 1
>> [ 6.821769] # Subtest: test_mb_generate_buddy
>> [ 6.824787] =============================================================================
>> [ 6.825568] BUG inode_cache (Tainted: G N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
>> ...
>> [ 6.894341] ok 7 ext4_inode_test
>> [ 6.895411] =============================================================================
>> [ 6.895777] BUG inode_cache (Tainted: G B N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
>>
>> Another example, from another test run, is
>>
>> [ 3.938551] # Subtest: test_new_blocks_simple
>> [ 3.947171] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>> [ 3.952988] ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>> [ 3.958403] ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>> [ 3.958890] =============================================================================
>> [ 3.959159] BUG inode_cache (Tainted: G N): Padding overwritten. 0xffff8de881adbf68-0xffff8de881adbf6f @offset=16232
>>
>> Another one:
>>
>> [ 18.730473] # Subtest: test_new_blocks_simple
>> [ 18.760547] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>> [ 18.778477] ==================================================================
>> [ 18.778950] BUG: KFENCE: out-of-bounds write in ext4_mb_init+0x5d7/0xa60
>>
>> This is just a sample, taken from a quick look at test results.
>>
>> Are those backtraces expected ? If so, would it be possible to execute the
>> tests without generating such backtraces ? The backtraces, if intentional,
>> hide real problems in the noise.
> Thanks for the report. The backtrace is not expected, I will look into this. Thansk!
>>
Hi Guenter, I could not reproduce this in my local vm. From the reported backtraces, it's
likely there is a out-of-bounds write to sbi->s_buddy_cache. I try to fix this in [1] and
it works fine in my local vm. I wish this work for you to elimate the reported nosie.
Look forward to your reply, Thanks.
Kemeng
[1] https://lore.kernel.org/linux-ext4/20240322165518.8147-1-shikemeng@huaweicloud.com/T/#u
>> Thanks,
>> Guenter
>>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy
2024-03-22 9:27 ` Kemeng Shi
@ 2024-03-22 14:49 ` Guenter Roeck
2024-03-26 13:09 ` Kemeng Shi
0 siblings, 1 reply; 15+ messages in thread
From: Guenter Roeck @ 2024-03-22 14:49 UTC (permalink / raw)
To: Kemeng Shi; +Cc: tytso, adilger.kernel, linux-ext4, linux-kernel
On 3/22/24 02:27, Kemeng Shi wrote:
>
>
> on 3/21/2024 3:16 PM, Kemeng Shi wrote:
>>
>>
>> on 3/21/2024 12:23 AM, Guenter Roeck wrote:
>>> Hi,
>>>
>>> On Wed, Jan 03, 2024 at 06:48:57PM +0800, Kemeng Shi wrote:
>>>> Add unit test of ext4_mb_generate_buddy
>>>>
>>>> Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
>>>
>>> With this and other new ext4 tests test in the tree, I see a variety
>>> of backtraces in the upstream kernel if debug options are enabled.
>>> An example is
>>>
>>> [ 6.821447] KTAP version 1
>>> [ 6.821769] # Subtest: test_mb_generate_buddy
>>> [ 6.824787] =============================================================================
>>> [ 6.825568] BUG inode_cache (Tainted: G N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
>>> ...
>>> [ 6.894341] ok 7 ext4_inode_test
>>> [ 6.895411] =============================================================================
>>> [ 6.895777] BUG inode_cache (Tainted: G B N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
>>>
>>> Another example, from another test run, is
>>>
>>> [ 3.938551] # Subtest: test_new_blocks_simple
>>> [ 3.947171] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>>> [ 3.952988] ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>>> [ 3.958403] ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>>> [ 3.958890] =============================================================================
>>> [ 3.959159] BUG inode_cache (Tainted: G N): Padding overwritten. 0xffff8de881adbf68-0xffff8de881adbf6f @offset=16232
>>>
>>> Another one:
>>>
>>> [ 18.730473] # Subtest: test_new_blocks_simple
>>> [ 18.760547] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>>> [ 18.778477] ==================================================================
>>> [ 18.778950] BUG: KFENCE: out-of-bounds write in ext4_mb_init+0x5d7/0xa60
>>>
>>> This is just a sample, taken from a quick look at test results.
>>>
>>> Are those backtraces expected ? If so, would it be possible to execute the
>>> tests without generating such backtraces ? The backtraces, if intentional,
>>> hide real problems in the noise.
>> Thanks for the report. The backtrace is not expected, I will look into this. Thansk!
>>>
> Hi Guenter, I could not reproduce this in my local vm. From the reported backtraces, it's
> likely there is a out-of-bounds write to sbi->s_buddy_cache. I try to fix this in [1] and
> it works fine in my local vm. I wish this work for you to elimate the reported nosie.
> Look forward to your reply, Thanks.
>
You would need to have CONFIG_SLUB_DEBUG=y, CONFIG_SLUB_DEBUG_ON=y, and CONFIG_KFENCE=y
to see the problems.
Guenter
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy
2024-03-22 14:49 ` Guenter Roeck
@ 2024-03-26 13:09 ` Kemeng Shi
0 siblings, 0 replies; 15+ messages in thread
From: Kemeng Shi @ 2024-03-26 13:09 UTC (permalink / raw)
To: Guenter Roeck; +Cc: tytso, adilger.kernel, linux-ext4, linux-kernel
on 3/22/2024 10:49 PM, Guenter Roeck wrote:
> On 3/22/24 02:27, Kemeng Shi wrote:
>>
>>
>> on 3/21/2024 3:16 PM, Kemeng Shi wrote:
>>>
>>>
>>> on 3/21/2024 12:23 AM, Guenter Roeck wrote:
>>>> Hi,
>>>>
>>>> On Wed, Jan 03, 2024 at 06:48:57PM +0800, Kemeng Shi wrote:
>>>>> Add unit test of ext4_mb_generate_buddy
>>>>>
>>>>> Signed-off-by: Kemeng Shi <shikemeng@huaweicloud.com>
>>>>
>>>> With this and other new ext4 tests test in the tree, I see a variety
>>>> of backtraces in the upstream kernel if debug options are enabled.
>>>> An example is
>>>>
>>>> [ 6.821447] KTAP version 1
>>>> [ 6.821769] # Subtest: test_mb_generate_buddy
>>>> [ 6.824787] =============================================================================
>>>> [ 6.825568] BUG inode_cache (Tainted: G N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
>>>> ...
>>>> [ 6.894341] ok 7 ext4_inode_test
>>>> [ 6.895411] =============================================================================
>>>> [ 6.895777] BUG inode_cache (Tainted: G B N): Padding overwritten. 0xfffff80006223f68-0xfffff80006223f6f @offset=16232
>>>>
>>>> Another example, from another test run, is
>>>>
>>>> [ 3.938551] # Subtest: test_new_blocks_simple
>>>> [ 3.947171] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>>>> [ 3.952988] ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>>>> [ 3.958403] ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>>>> [ 3.958890] =============================================================================
>>>> [ 3.959159] BUG inode_cache (Tainted: G N): Padding overwritten. 0xffff8de881adbf68-0xffff8de881adbf6f @offset=16232
>>>>
>>>> Another one:
>>>>
>>>> [ 18.730473] # Subtest: test_new_blocks_simple
>>>> [ 18.760547] ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
>>>> [ 18.778477] ==================================================================
>>>> [ 18.778950] BUG: KFENCE: out-of-bounds write in ext4_mb_init+0x5d7/0xa60
>>>>
>>>> This is just a sample, taken from a quick look at test results.
>>>>
>>>> Are those backtraces expected ? If so, would it be possible to execute the
>>>> tests without generating such backtraces ? The backtraces, if intentional,
>>>> hide real problems in the noise.
>>> Thanks for the report. The backtrace is not expected, I will look into this. Thansk!
>>>>
>> Hi Guenter, I could not reproduce this in my local vm. From the reported backtraces, it's
>> likely there is a out-of-bounds write to sbi->s_buddy_cache. I try to fix this in [1] and
>> it works fine in my local vm. I wish this work for you to elimate the reported nosie.
>> Look forward to your reply, Thanks.
>>
>
> You would need to have CONFIG_SLUB_DEBUG=y, CONFIG_SLUB_DEBUG_ON=y, and CONFIG_KFENCE=y
> to see the problems.
Thanks for sharing this. I have already turn these configs on, but I use the ext4 tree
https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/log/?h=dev which may be
the reason why I can't reproduce the issue.
I see you have already tested the fix in upstream. Thanks a lot for that!
Kemeng
>
> Guenter
>
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2024-03-26 13:09 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-01-03 10:48 [PATCH v2 0/5] More unit test for mballoc Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 1/5] ext4: Add unit test for test_free_blocks_simple Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 2/5] ext4: Add unit test of ext4_mb_generate_buddy Kemeng Shi
2024-03-20 16:23 ` Guenter Roeck
2024-03-20 18:42 ` Guenter Roeck
2024-03-21 7:16 ` Kemeng Shi
2024-03-22 9:27 ` Kemeng Shi
2024-03-22 14:49 ` Guenter Roeck
2024-03-26 13:09 ` Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 3/5] ext4: Add unit test for mb_mark_used Kemeng Shi
2024-01-03 10:48 ` [PATCH v2 4/5] ext4: Add unit test for mb_free_blocks Kemeng Shi
2024-01-03 10:49 ` [PATCH v2 5/5] ext4: Add unit test for ext4_mb_mark_diskspace_used Kemeng Shi
2024-02-22 15:54 ` [PATCH v2 0/5] More unit test for mballoc Theodore Ts'o
2024-02-25 16:07 ` Guenter Roeck
2024-02-26 13:42 ` Kemeng Shi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).