* [PATCH 00/15] Fast commits support for e2fsprogs
@ 2020-11-20 19:15 Harshad Shirwadkar
2020-11-20 19:15 ` [PATCH 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
` (14 more replies)
0 siblings, 15 replies; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
This patch series adds fast commits support in e2fsprogs. This
includes fast commit recovery support in e2fsck and fast commit
configuration support in mke2fs and tune2fs. Along with that this
patch series also makes e2fsck/recovery.c identical with
jbd2/recovery.c in kernel. In addition, this patch imports and makes
fast_commit.h (the file that contains on-disk formats for fast
commits) byte identical with the kernel.
The recovery logic for fast commits follows the same steps as that of
the recovery logic in kernel. The general guidining principle for the
fast commit replay is that the individual tags found in fast commit
area store the result of the operation as their paylod instead of
storing the procedure itself. The recovery logic enforces this result
onto the filesystem thereby making the fast commit replay
idempotent. Unlike kernel, there's no atomic oepration support in
e2fsprogs yet. So, it's possible that we may crash while we are in the
middle of replaying of a fast commit tag. The only way to recover from
that situation would be to run fsck. That's why we mark the file
system as unclean before the fast commit replay and make it clean at
the end.
This series adds new regression test that performs fast commit
replay. I ensured that all the regressions tests pass.
Verified that all the tests pass:
367 tests succeeded 0 tests failed
New fast commit recovery test:
j_recover_fast_commit: ok
The patch series invalidates the initial version of the patch series
which was sent back in Mar 2020. Since then the fast commit code in
kernel has evolved a lot (including the on-disk format change). So,
this patch series is based on the new fast commit kernel code (which
is available in upstream kernel now). This patch series is a complete
revamp of the original series.
Github: https://github.com/harshadjs/e2fsprogs/tree/fast-commit-v2
Harshad Shirwadkar (15):
ext2fs: move calculate_summary_stats to ext2fs lib
ext2fs, e2fsck: add kernel endian-ness conversion macros
e2fsck: port fc changes from kernel's recovery.c to e2fsck
mke2fs, dumpe2fs: make fast commit blocks configurable
mke2fs, tune2fs: update man page with fast commit info
ext2fs: add new APIs needed for fast commits
e2fsck: add function to rewrite extent tree
e2fsck: add fast commit setup code
e2fsck: add fast commit scan pass
e2fsck: add fast commit replay skeleton
e2fsck: add fc replay for link, unlink, creat tags
e2fsck: add replay for add_range, del_range, and inode tags
debugfs: add fast commit support to logdump
tests: add fast commit recovery tests
ext4: fix tests to account for new dumpe2fs output
debugfs/journal.c | 10 +-
debugfs/logdump.c | 122 ++++-
e2fsck/e2fsck.h | 32 ++
e2fsck/extents.c | 168 +++---
e2fsck/journal.c | 656 +++++++++++++++++++++++-
e2fsck/recovery.c | 232 ++++++---
e2fsck/unix.c | 8 +-
lib/e2p/e2p.h | 1 +
lib/e2p/ljs.c | 16 +-
lib/ext2fs/bitops.h | 8 +
lib/ext2fs/ext2_fs.h | 1 +
lib/ext2fs/ext2fs.h | 33 +-
lib/ext2fs/extent.c | 56 ++
lib/ext2fs/fast_commit.h | 201 ++++++++
lib/ext2fs/initialize.c | 94 ++++
lib/ext2fs/jfs_compat.h | 19 +-
lib/ext2fs/kernel-jbd.h | 19 +-
lib/ext2fs/mkjournal.c | 99 ++--
lib/ext2fs/unlink.c | 6 +-
misc/dumpe2fs.c | 10 +-
misc/mke2fs.8.in | 21 +
misc/mke2fs.c | 24 +-
misc/tune2fs.8.in | 25 +
misc/tune2fs.c | 67 +--
misc/util.c | 63 ++-
misc/util.h | 4 +-
resize/resize2fs.c | 6 +-
tests/d_corrupt_journal_nr_users/expect | 6 +-
tests/f_jnl_errno/expect.0 | 6 +-
tests/f_opt_extent/expect | 2 +-
tests/i_bitmaps/expect | 8 +-
tests/j_ext_dumpe2fs/expect | 6 +-
tests/j_recover_fast_commit/commands | 4 +
tests/j_recover_fast_commit/expect | 23 +
tests/j_recover_fast_commit/image.gz | Bin 0 -> 3595 bytes
tests/j_recover_fast_commit/script | 26 +
tests/m_bigjournal/expect.1 | 6 +-
tests/m_extent_journal/expect.1 | 6 +-
tests/m_resize_inode_meta_bg/expect.1 | 6 +-
tests/m_rootdir/expect | 6 +-
tests/r_32to64bit/expect | 6 +-
tests/r_32to64bit_meta/expect | 4 +-
tests/r_32to64bit_move_itable/expect | 8 +-
tests/r_64to32bit/expect | 6 +-
tests/r_64to32bit_meta/expect | 4 +-
tests/r_move_itable_nostride/expect | 6 +-
tests/r_move_itable_realloc/expect | 6 +-
tests/t_disable_mcsum/expect | 4 +-
tests/t_disable_mcsum_noinitbg/expect | 6 +-
tests/t_disable_mcsum_yesinitbg/expect | 4 +-
tests/t_enable_mcsum/expect | 6 +-
tests/t_enable_mcsum_ext3/expect | 10 +-
tests/t_enable_mcsum_initbg/expect | 6 +-
53 files changed, 1829 insertions(+), 353 deletions(-)
create mode 100644 lib/ext2fs/fast_commit.h
create mode 100644 tests/j_recover_fast_commit/commands
create mode 100644 tests/j_recover_fast_commit/expect
create mode 100644 tests/j_recover_fast_commit/image.gz
create mode 100755 tests/j_recover_fast_commit/script
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH 01/15] ext2fs: move calculate_summary_stats to ext2fs lib
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
@ 2020-11-20 19:15 ` Harshad Shirwadkar
2020-12-02 16:47 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 02/15] ext2fs, e2fsck: add kernel endian-ness conversion macros Harshad Shirwadkar
` (13 subsequent siblings)
14 siblings, 1 reply; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
The function calculate_summary_stats sets the global metadata of the
file system. Tune2fs had this function defined statically in
tune2fs.c. Fast commit replay needs this function to set global
metadata at the end of the replay phase. So, move this function to
libext2fs.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
lib/ext2fs/ext2fs.h | 1 +
lib/ext2fs/initialize.c | 94 +++++++++++++++++++++++++++++++++++++++++
misc/tune2fs.c | 59 +-------------------------
resize/resize2fs.c | 6 +--
4 files changed, 99 insertions(+), 61 deletions(-)
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 69c8a3ff..a8a6e091 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1484,6 +1484,7 @@ errcode_t ext2fs_write_ind_block(ext2_filsys fs, blk_t blk, void *buf);
extern errcode_t ext2fs_initialize(const char *name, int flags,
struct ext2_super_block *param,
io_manager manager, ext2_filsys *ret_fs);
+extern errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs, int super_only);
/* icount.c */
extern void ext2fs_free_icount(ext2_icount_t icount);
diff --git a/lib/ext2fs/initialize.c b/lib/ext2fs/initialize.c
index 96ec1cff..edd692bb 100644
--- a/lib/ext2fs/initialize.c
+++ b/lib/ext2fs/initialize.c
@@ -575,3 +575,97 @@ cleanup:
ext2fs_free(fs);
return retval;
}
+
+errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs, int super_only)
+{
+ blk64_t blk;
+ ext2_ino_t ino;
+ unsigned int group = 0;
+ unsigned int count = 0;
+ int total_free = 0;
+ int group_free = 0;
+ int last_allocated = 0;
+ int uninit;
+
+ /*
+ * First calculate the block statistics
+ */
+ uninit = 1;
+ for (blk = fs->super->s_first_data_block;
+ blk < ext2fs_blocks_count(fs->super); blk++) {
+ if (!ext2fs_fast_test_block_bitmap2(fs->block_map, blk)) {
+ group_free++;
+ total_free++;
+ } else {
+ uninit = 0;
+ }
+ count++;
+ if ((count == fs->super->s_blocks_per_group) ||
+ (blk == ext2fs_blocks_count(fs->super)-1)) {
+ ext2fs_bg_free_blocks_count_set(fs, group,
+ group_free);
+ if (!super_only) {
+ if (uninit && blk !=
+ ext2fs_blocks_count(fs->super) - 1)
+ ext2fs_bg_flags_set(fs, group,
+ EXT2_BG_BLOCK_UNINIT);
+ else
+ ext2fs_bg_flags_clear(fs, group,
+ EXT2_BG_BLOCK_UNINIT);
+ }
+ count = 0;
+ group_free = 0;
+ uninit = 1;
+ group++;
+ }
+ }
+ total_free = EXT2FS_C2B(fs, total_free);
+ ext2fs_free_blocks_count_set(fs->super, total_free);
+
+ /*
+ * Next, calculate the inode statistics
+ */
+ group_free = 0;
+ total_free = 0;
+ last_allocated = 0;
+ count = 0;
+ group = 0;
+
+ /* Protect loop from wrap-around if s_inodes_count maxed */
+ for (ino = 1; ino <= fs->super->s_inodes_count && ino > 0; ino++) {
+ if (!ext2fs_test_inode_bitmap2(fs->inode_map, ino)) {
+ group_free++;
+ total_free++;
+ } else {
+ last_allocated = ino;
+ }
+ count++;
+ if ((count == fs->super->s_inodes_per_group) ||
+ (ino == fs->super->s_inodes_count)) {
+ if (!super_only) {
+ if (last_allocated) {
+ ext2fs_bg_flags_clear(fs, group,
+ EXT2_BG_INODE_UNINIT);
+ ext2fs_bg_itable_unused_set(fs, group,
+ fs->super->s_inodes_per_group -
+ (last_allocated %
+ fs->super->s_inodes_per_group));
+ } else {
+ ext2fs_bg_flags_set(fs, group,
+ EXT2_BG_INODE_UNINIT);
+ ext2fs_bg_itable_unused_set(fs, group,
+ 0);
+ }
+ ext2fs_bg_free_inodes_count_set(fs, group,
+ group_free);
+ }
+ group++;
+ count = 0;
+ group_free = 0;
+ last_allocated = 0;
+ }
+ }
+ fs->super->s_free_inodes_count = total_free;
+ ext2fs_mark_super_dirty(fs);
+ return 0;
+}
diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index f942c698..670ed9e0 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -2603,63 +2603,6 @@ err_out:
return retval;
}
-static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs)
-{
- blk64_t blk;
- ext2_ino_t ino;
- unsigned int group = 0;
- unsigned int count = 0;
- int total_free = 0;
- int group_free = 0;
-
- /*
- * First calculate the block statistics
- */
- for (blk = fs->super->s_first_data_block;
- blk < ext2fs_blocks_count(fs->super); blk++) {
- if (!ext2fs_fast_test_block_bitmap2(fs->block_map, blk)) {
- group_free++;
- total_free++;
- }
- count++;
- if ((count == fs->super->s_blocks_per_group) ||
- (blk == ext2fs_blocks_count(fs->super)-1)) {
- ext2fs_bg_free_blocks_count_set(fs, group++,
- group_free);
- count = 0;
- group_free = 0;
- }
- }
- total_free = EXT2FS_C2B(fs, total_free);
- ext2fs_free_blocks_count_set(fs->super, total_free);
-
- /*
- * Next, calculate the inode statistics
- */
- group_free = 0;
- total_free = 0;
- count = 0;
- group = 0;
-
- /* Protect loop from wrap-around if s_inodes_count maxed */
- for (ino = 1; ino <= fs->super->s_inodes_count && ino > 0; ino++) {
- if (!ext2fs_fast_test_inode_bitmap2(fs->inode_map, ino)) {
- group_free++;
- total_free++;
- }
- count++;
- if ((count == fs->super->s_inodes_per_group) ||
- (ino == fs->super->s_inodes_count)) {
- ext2fs_bg_free_inodes_count_set(fs, group++,
- group_free);
- count = 0;
- group_free = 0;
- }
- }
- fs->super->s_free_inodes_count = total_free;
- ext2fs_mark_super_dirty(fs);
- return 0;
-}
#define list_for_each_safe(pos, pnext, head) \
for (pos = (head)->next, pnext = pos->next; pos != (head); \
@@ -2738,7 +2681,7 @@ static int resize_inode(ext2_filsys fs, unsigned long new_size)
if (retval)
goto err_out_undo;
- ext2fs_calculate_summary_stats(fs);
+ ext2fs_calculate_summary_stats(fs, 1 /* super only */);
fs->super->s_state |= EXT2_VALID_FS;
/* mark super block and block bitmap as dirty */
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 2443ff67..270e4deb 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -49,7 +49,7 @@ static errcode_t inode_scan_and_fix(ext2_resize_t rfs);
static errcode_t inode_ref_fix(ext2_resize_t rfs);
static errcode_t move_itables(ext2_resize_t rfs);
static errcode_t fix_resize_inode(ext2_filsys fs);
-static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs);
+static errcode_t resize2fs_calculate_summary_stats(ext2_filsys fs);
static errcode_t fix_sb_journal_backup(ext2_filsys fs);
static errcode_t mark_table_blocks(ext2_filsys fs,
ext2fs_block_bitmap bmap);
@@ -211,7 +211,7 @@ errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags,
goto errout;
init_resource_track(&rtrack, "calculate_summary_stats", fs->io);
- retval = ext2fs_calculate_summary_stats(rfs->new_fs);
+ retval = resize2fs_calculate_summary_stats(rfs->new_fs);
if (retval)
goto errout;
print_resource_track(rfs, &rtrack, fs->io);
@@ -2740,7 +2740,7 @@ errout:
/*
* Finally, recalculate the summary information
*/
-static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs)
+static errcode_t resize2fs_calculate_summary_stats(ext2_filsys fs)
{
blk64_t blk;
ext2_ino_t ino;
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 02/15] ext2fs, e2fsck: add kernel endian-ness conversion macros
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
2020-11-20 19:15 ` [PATCH 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
@ 2020-11-20 19:15 ` Harshad Shirwadkar
2020-12-02 16:50 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Harshad Shirwadkar
` (12 subsequent siblings)
14 siblings, 1 reply; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
In order to make recovery.c identical with kernel, we need endianness
conversion macros (such as cpu_to_be32 and friends) defined in
e2fsprogs. This patch defines these macros and also fixes recovery.c
to use these. These macros are also needed for fast commit recovery
patches later in this series.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
e2fsck/recovery.c | 42 ++++++++++--------------------------------
lib/ext2fs/bitops.h | 8 ++++++++
2 files changed, 18 insertions(+), 32 deletions(-)
diff --git a/e2fsck/recovery.c b/e2fsck/recovery.c
index 5df690ad..6c3b7bb4 100644
--- a/e2fsck/recovery.c
+++ b/e2fsck/recovery.c
@@ -121,27 +121,6 @@ failed:
#endif /* __KERNEL__ */
-static inline __u32 get_be32(__be32 *p)
-{
- unsigned char *cp = (unsigned char *) p;
- __u32 ret;
-
- ret = *cp++;
- ret = (ret << 8) + *cp++;
- ret = (ret << 8) + *cp++;
- ret = (ret << 8) + *cp++;
- return ret;
-}
-
-static inline __u16 get_be16(__be16 *p)
-{
- unsigned char *cp = (unsigned char *) p;
- __u16 ret;
-
- ret = *cp++;
- ret = (ret << 8) + *cp++;
- return ret;
-}
/*
* Read a block from the journal
@@ -232,10 +211,10 @@ static int count_tags(journal_t *journal, struct buffer_head *bh)
nr++;
tagp += tag_bytes;
- if (!(get_be16(&tag->t_flags) & JBD2_FLAG_SAME_UUID))
+ if (!(tag->t_flags & cpu_to_be16(JBD2_FLAG_SAME_UUID)))
tagp += 16;
- if (get_be16(&tag->t_flags) & JBD2_FLAG_LAST_TAG)
+ if (tag->t_flags & cpu_to_be16(JBD2_FLAG_LAST_TAG))
break;
}
@@ -358,9 +337,9 @@ int jbd2_journal_skip_recovery(journal_t *journal)
static inline unsigned long long read_tag_block(journal_t *journal,
journal_block_tag_t *tag)
{
- unsigned long long block = get_be32(&tag->t_blocknr);
+ unsigned long long block = be32_to_cpu(tag->t_blocknr);
if (jbd2_has_feature_64bit(journal))
- block |= (u64)get_be32(&tag->t_blocknr_high) << 32;
+ block |= (u64)be32_to_cpu(tag->t_blocknr_high) << 32;
return block;
}
@@ -429,9 +408,9 @@ static int jbd2_block_tag_csum_verify(journal_t *j, journal_block_tag_t *tag,
csum32 = jbd2_chksum(j, csum32, buf, j->j_blocksize);
if (jbd2_has_feature_csum3(j))
- return get_be32(&tag3->t_checksum) == csum32;
-
- return get_be16(&tag->t_checksum) == (csum32 & 0xFFFF);
+ return tag3->t_checksum == cpu_to_be32(csum32);
+ else
+ return tag->t_checksum == cpu_to_be16(csum32);
}
static int do_one_pass(journal_t *journal,
@@ -579,7 +558,7 @@ static int do_one_pass(journal_t *journal,
unsigned long io_block;
tag = (journal_block_tag_t *) tagp;
- flags = get_be16(&tag->t_flags);
+ flags = be16_to_cpu(tag->t_flags);
io_block = next_log_block++;
wrap(journal, next_log_block);
@@ -643,9 +622,8 @@ static int do_one_pass(journal_t *journal,
memcpy(nbh->b_data, obh->b_data,
journal->j_blocksize);
if (flags & JBD2_FLAG_ESCAPE) {
- __be32 magic = cpu_to_be32(JBD2_MAGIC_NUMBER);
- memcpy(nbh->b_data, &magic,
- sizeof(magic));
+ *((__be32 *)nbh->b_data) =
+ cpu_to_be32(JBD2_MAGIC_NUMBER);
}
BUFFER_TRACE(nbh, "marking dirty");
diff --git a/lib/ext2fs/bitops.h b/lib/ext2fs/bitops.h
index 505b3c9c..3c7b2496 100644
--- a/lib/ext2fs/bitops.h
+++ b/lib/ext2fs/bitops.h
@@ -247,6 +247,14 @@ extern errcode_t ext2fs_find_first_set_generic_bmap(ext2fs_generic_bitmap bitmap
#endif /* __STDC_VERSION__ >= 199901L */
#endif /* INCLUDE_INLINE_FUNCS */
+/* Macros for kernel compatibility */
+#define be32_to_cpu(x) ext2fs_be32_to_cpu(x)
+#define le32_to_cpu(x) ext2fs_le32_to_cpu(x)
+#define le16_to_cpu(x) ext2fs_le16_to_cpu(x)
+
+#define cpu_to_be32(x) ext2fs_cpu_to_be32(x)
+#define cpu_to_be16(x) ext2fs_cpu_to_be16(x)
+#define cpu_to_le16(x) ext2fs_cpu_to_le16(x)
/*
* Fast bit set/clear functions that doesn't need to return the
* previous bit value.
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
2020-11-20 19:15 ` [PATCH 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
2020-11-20 19:15 ` [PATCH 02/15] ext2fs, e2fsck: add kernel endian-ness conversion macros Harshad Shirwadkar
@ 2020-11-20 19:15 ` Harshad Shirwadkar
2020-12-02 16:54 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 04/15] mke2fs, dumpe2fs: make fast commit blocks configurable Harshad Shirwadkar
` (11 subsequent siblings)
14 siblings, 1 reply; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
This patch makes recovery.c identical with fast commit kernel changes.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
debugfs/journal.c | 10 +--
e2fsck/journal.c | 28 ++++--
e2fsck/recovery.c | 190 +++++++++++++++++++++++++++++-----------
lib/ext2fs/jfs_compat.h | 19 +++-
lib/ext2fs/kernel-jbd.h | 19 +++-
5 files changed, 197 insertions(+), 69 deletions(-)
diff --git a/debugfs/journal.c b/debugfs/journal.c
index fa72ec57..e8872f05 100644
--- a/debugfs/journal.c
+++ b/debugfs/journal.c
@@ -378,7 +378,7 @@ try_backup_journal:
goto errout;
}
- journal->j_maxlen = EXT2_I_SIZE(&j_inode->i_ext2) /
+ journal->j_total_len = EXT2_I_SIZE(&j_inode->i_ext2) /
journal->j_blocksize;
#ifdef USE_INODE_IO
@@ -493,7 +493,7 @@ try_backup_journal:
brelse(bh);
maxlen = ext2fs_blocks_count(&jsuper);
- journal->j_maxlen = (maxlen < 1ULL << 32) ? maxlen :
+ journal->j_total_len = (maxlen < 1ULL << 32) ? maxlen :
(1ULL << 32) - 1;
start++;
}
@@ -629,9 +629,9 @@ static errcode_t ext2fs_journal_load(journal_t *journal)
if (jsb->s_blocksize != htonl(journal->j_blocksize))
return EXT2_ET_CORRUPT_JOURNAL_SB;
- if (ntohl(jsb->s_maxlen) < journal->j_maxlen)
- journal->j_maxlen = ntohl(jsb->s_maxlen);
- else if (ntohl(jsb->s_maxlen) > journal->j_maxlen)
+ if (ntohl(jsb->s_maxlen) < journal->j_total_len)
+ journal->j_total_len = ntohl(jsb->s_maxlen);
+ else if (ntohl(jsb->s_maxlen) > journal->j_total_len)
return EXT2_ET_CORRUPT_JOURNAL_SB;
journal->j_tail_sequence = ntohl(jsb->s_sequence);
diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 7d9f1b40..75fefcde 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -379,7 +379,7 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal)
goto errout;
}
- journal->j_maxlen = EXT2_I_SIZE(&j_inode->i_ext2) /
+ journal->j_total_len = EXT2_I_SIZE(&j_inode->i_ext2) /
journal->j_blocksize;
#ifdef USE_INODE_IO
@@ -503,7 +503,7 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal)
brelse(bh);
maxlen = ext2fs_blocks_count(&jsuper);
- journal->j_maxlen = (maxlen < 1ULL << 32) ? maxlen : (1ULL << 32) - 1;
+ journal->j_total_len = (maxlen < 1ULL << 32) ? maxlen : (1ULL << 32) - 1;
start++;
}
@@ -675,9 +675,9 @@ static errcode_t e2fsck_journal_load(journal_t *journal)
return EXT2_ET_CORRUPT_JOURNAL_SB;
}
- if (ntohl(jsb->s_maxlen) < journal->j_maxlen)
- journal->j_maxlen = ntohl(jsb->s_maxlen);
- else if (ntohl(jsb->s_maxlen) > journal->j_maxlen) {
+ if (ntohl(jsb->s_maxlen) < journal->j_total_len)
+ journal->j_total_len = ntohl(jsb->s_maxlen);
+ else if (ntohl(jsb->s_maxlen) > journal->j_total_len) {
com_err(ctx->program_name, EXT2_ET_CORRUPT_JOURNAL_SB,
_("%s: journal too short\n"),
ctx->device_name);
@@ -688,7 +688,21 @@ static errcode_t e2fsck_journal_load(journal_t *journal)
journal->j_transaction_sequence = journal->j_tail_sequence;
journal->j_tail = ntohl(jsb->s_start);
journal->j_first = ntohl(jsb->s_first);
- journal->j_last = ntohl(jsb->s_maxlen);
+ if (jbd2_has_feature_fast_commit(journal)) {
+ if (ntohl(jsb->s_maxlen) - jbd2_journal_get_num_fc_blks(jsb)
+ < JBD2_MIN_JOURNAL_BLOCKS) {
+ com_err(ctx->program_name, EXT2_ET_CORRUPT_JOURNAL_SB,
+ _("%s: incorrect fast commit blocks\n"),
+ ctx->device_name);
+ return EXT2_ET_CORRUPT_JOURNAL_SB;
+ }
+ journal->j_fc_last = ntohl(jsb->s_maxlen);
+ journal->j_last = journal->j_fc_last -
+ jbd2_journal_get_num_fc_blks(jsb);
+ journal->j_fc_first = journal->j_last + 1;
+ } else {
+ journal->j_last = ntohl(jsb->s_maxlen);
+ }
return 0;
}
@@ -720,7 +734,7 @@ static void e2fsck_journal_reset_super(e2fsck_t ctx, journal_superblock_t *jsb,
memset (p, 0, ctx->fs->blocksize-sizeof(journal_header_t));
jsb->s_blocksize = htonl(ctx->fs->blocksize);
- jsb->s_maxlen = htonl(journal->j_maxlen);
+ jsb->s_maxlen = htonl(journal->j_total_len);
jsb->s_first = htonl(1);
/* Initialize the journal sequence number so that there is "no"
diff --git a/e2fsck/recovery.c b/e2fsck/recovery.c
index 6c3b7bb4..dc0694fc 100644
--- a/e2fsck/recovery.c
+++ b/e2fsck/recovery.c
@@ -35,7 +35,6 @@ struct recovery_info
int nr_revoke_hits;
};
-enum passtype {PASS_SCAN, PASS_REVOKE, PASS_REPLAY};
static int do_one_pass(journal_t *journal,
struct recovery_info *info, enum passtype pass);
static int scan_revoke_records(journal_t *, struct buffer_head *,
@@ -75,8 +74,8 @@ static int do_readahead(journal_t *journal, unsigned int start)
/* Do up to 128K of readahead */
max = start + (128 * 1024 / journal->j_blocksize);
- if (max > journal->j_maxlen)
- max = journal->j_maxlen;
+ if (max > journal->j_total_len)
+ max = journal->j_total_len;
/* Do the readahead itself. We'll submit MAXBUF buffer_heads at
* a time to the block device IO layer. */
@@ -135,7 +134,7 @@ static int jread(struct buffer_head **bhp, journal_t *journal,
*bhp = NULL;
- if (offset >= journal->j_maxlen) {
+ if (offset >= journal->j_total_len) {
printk(KERN_ERR "JBD2: corrupted journal superblock\n");
return -EFSCORRUPTED;
}
@@ -180,7 +179,7 @@ static int jbd2_descriptor_block_csum_verify(journal_t *j, void *buf)
if (!jbd2_journal_has_csum_v2or3(j))
return 1;
- tail = (struct jbd2_journal_block_tail *)((char *)buf + j->j_blocksize -
+ tail = (struct jbd2_journal_block_tail *)(buf + j->j_blocksize -
sizeof(struct jbd2_journal_block_tail));
provided = tail->t_checksum;
tail->t_checksum = 0;
@@ -225,10 +224,51 @@ static int count_tags(journal_t *journal, struct buffer_head *bh)
/* Make sure we wrap around the log correctly! */
#define wrap(journal, var) \
do { \
- if (var >= (journal)->j_last) \
- var -= ((journal)->j_last - (journal)->j_first); \
+ unsigned long _wrap_last = \
+ jbd2_has_feature_fast_commit(journal) ? \
+ (journal)->j_fc_last : (journal)->j_last; \
+ \
+ if (var >= _wrap_last) \
+ var -= (_wrap_last - (journal)->j_first); \
} while (0)
+static int fc_do_one_pass(journal_t *journal,
+ struct recovery_info *info, enum passtype pass)
+{
+ unsigned int expected_commit_id = info->end_transaction;
+ unsigned long next_fc_block;
+ struct buffer_head *bh;
+ int err = 0;
+
+ next_fc_block = journal->j_fc_first;
+ if (!journal->j_fc_replay_callback)
+ return 0;
+
+ while (next_fc_block <= journal->j_fc_last) {
+ jbd_debug(3, "Fast commit replay: next block %ld",
+ next_fc_block);
+ err = jread(&bh, journal, next_fc_block);
+ if (err) {
+ jbd_debug(3, "Fast commit replay: read error");
+ break;
+ }
+
+ jbd_debug(3, "Processing fast commit blk with seq %d");
+ err = journal->j_fc_replay_callback(journal, bh, pass,
+ next_fc_block - journal->j_fc_first,
+ expected_commit_id);
+ next_fc_block++;
+ if (err < 0 || err == JBD2_FC_REPLAY_STOP)
+ break;
+ err = 0;
+ }
+
+ if (err)
+ jbd_debug(3, "Fast commit replay failed, err = %d\n", err);
+
+ return err;
+}
+
/**
* jbd2_journal_recover - recovers a on-disk journal
* @journal: the journal to recover
@@ -286,7 +326,7 @@ int jbd2_journal_recover(journal_t *journal)
err = err2;
/* Make sure all replayed data is on permanent storage */
if (journal->j_flags & JBD2_BARRIER) {
- err2 = blkdev_issue_flush(journal->j_fs_dev, GFP_KERNEL, NULL);
+ err2 = blkdev_issue_flush(journal->j_fs_dev, GFP_KERNEL);
if (!err)
err = err2;
}
@@ -428,6 +468,8 @@ static int do_one_pass(journal_t *journal,
__u32 crc32_sum = ~0; /* Transactional Checksums */
int descr_csum_size = 0;
int block_error = 0;
+ bool need_check_commit_time = false;
+ __u64 last_trans_commit_time = 0, commit_time;
/*
* First thing is to establish what we expect to find in the log
@@ -470,7 +512,9 @@ static int do_one_pass(journal_t *journal,
break;
jbd_debug(2, "Scanning for sequence ID %u at %lu/%lu\n",
- next_commit_ID, next_log_block, journal->j_last);
+ next_commit_ID, next_log_block,
+ jbd2_has_feature_fast_commit(journal) ?
+ journal->j_fc_last : journal->j_last);
/* Skip over each chunk of the transaction looking
* either the next descriptor block or the final commit
@@ -520,9 +564,21 @@ static int do_one_pass(journal_t *journal,
if (descr_csum_size > 0 &&
!jbd2_descriptor_block_csum_verify(journal,
bh->b_data)) {
- err = -EFSBADCRC;
- brelse(bh);
- goto failed;
+ /*
+ * PASS_SCAN can see stale blocks due to lazy
+ * journal init. Don't error out on those yet.
+ */
+ if (pass != PASS_SCAN) {
+ pr_err("JBD2: Invalid checksum recovering block %lu in log\n",
+ next_log_block);
+ err = -EFSBADCRC;
+ brelse(bh);
+ goto failed;
+ }
+ need_check_commit_time = true;
+ jbd_debug(1,
+ "invalid descriptor block found in %lu\n",
+ next_log_block);
}
/* If it is a valid descriptor block, replay it
@@ -532,6 +588,7 @@ static int do_one_pass(journal_t *journal,
if (pass != PASS_REPLAY) {
if (pass == PASS_SCAN &&
jbd2_has_feature_checksum(journal) &&
+ !need_check_commit_time &&
!info->end_transaction) {
if (calc_chksums(journal, bh,
&next_log_block,
@@ -663,7 +720,7 @@ static int do_one_pass(journal_t *journal,
* | GO TO NEXT "Journal Corruption"
* | TRANSACTION
* |
- * {(n+1)th transaction}
+ * {(n+1)th transanction}
* |
* _______|______________
* | |
@@ -680,21 +737,48 @@ static int do_one_pass(journal_t *journal,
* mentioned conditions. Hence assume
* "Interrupted Commit".)
*/
+ commit_time = be64_to_cpu(
+ ((struct commit_header *)bh->b_data)->h_commit_sec);
+ /*
+ * If need_check_commit_time is set, it means we are in
+ * PASS_SCAN and csum verify failed before. If
+ * commit_time is increasing, it's the same journal,
+ * otherwise it is stale journal block, just end this
+ * recovery.
+ */
+ if (need_check_commit_time) {
+ if (commit_time >= last_trans_commit_time) {
+ pr_err("JBD2: Invalid checksum found in transaction %u\n",
+ next_commit_ID);
+ err = -EFSBADCRC;
+ brelse(bh);
+ goto failed;
+ }
+ ignore_crc_mismatch:
+ /*
+ * It likely does not belong to same journal,
+ * just end this recovery with success.
+ */
+ jbd_debug(1, "JBD2: Invalid checksum ignored in transaction %u, likely stale data\n",
+ next_commit_ID);
+ err = 0;
+ brelse(bh);
+ goto done;
+ }
- /* Found an expected commit block: if checksums
- * are present verify them in PASS_SCAN; else not
+ /*
+ * Found an expected commit block: if checksums
+ * are present, verify them in PASS_SCAN; else not
* much to do other than move on to the next sequence
- * number. */
+ * number.
+ */
if (pass == PASS_SCAN &&
jbd2_has_feature_checksum(journal)) {
- int chksum_err, chksum_seen;
struct commit_header *cbh =
(struct commit_header *)bh->b_data;
unsigned found_chksum =
be32_to_cpu(cbh->h_chksum[0]);
- chksum_err = chksum_seen = 0;
-
if (info->end_transaction) {
journal->j_failed_commit =
info->end_transaction;
@@ -702,42 +786,25 @@ static int do_one_pass(journal_t *journal,
break;
}
- if (crc32_sum == found_chksum &&
- cbh->h_chksum_type == JBD2_CRC32_CHKSUM &&
- cbh->h_chksum_size ==
- JBD2_CRC32_CHKSUM_SIZE)
- chksum_seen = 1;
- else if (!(cbh->h_chksum_type == 0 &&
- cbh->h_chksum_size == 0 &&
- found_chksum == 0 &&
- !chksum_seen))
- /*
- * If fs is mounted using an old kernel and then
- * kernel with journal_chksum is used then we
- * get a situation where the journal flag has
- * checksum flag set but checksums are not
- * present i.e chksum = 0, in the individual
- * commit blocks.
- * Hence to avoid checksum failures, in this
- * situation, this extra check is added.
- */
- chksum_err = 1;
-
- if (chksum_err) {
- info->end_transaction = next_commit_ID;
+ /* Neither checksum match nor unused? */
+ if (!((crc32_sum == found_chksum &&
+ cbh->h_chksum_type ==
+ JBD2_CRC32_CHKSUM &&
+ cbh->h_chksum_size ==
+ JBD2_CRC32_CHKSUM_SIZE) ||
+ (cbh->h_chksum_type == 0 &&
+ cbh->h_chksum_size == 0 &&
+ found_chksum == 0)))
+ goto chksum_error;
- if (!jbd2_has_feature_async_commit(journal)) {
- journal->j_failed_commit =
- next_commit_ID;
- brelse(bh);
- break;
- }
- }
crc32_sum = ~0;
}
if (pass == PASS_SCAN &&
!jbd2_commit_block_csum_verify(journal,
bh->b_data)) {
+ chksum_error:
+ if (commit_time < last_trans_commit_time)
+ goto ignore_crc_mismatch;
info->end_transaction = next_commit_ID;
if (!jbd2_has_feature_async_commit(journal)) {
@@ -747,11 +814,24 @@ static int do_one_pass(journal_t *journal,
break;
}
}
+ if (pass == PASS_SCAN)
+ last_trans_commit_time = commit_time;
brelse(bh);
next_commit_ID++;
continue;
case JBD2_REVOKE_BLOCK:
+ /*
+ * Check revoke block crc in pass_scan, if csum verify
+ * failed, check commit block time later.
+ */
+ if (pass == PASS_SCAN &&
+ !jbd2_descriptor_block_csum_verify(journal,
+ bh->b_data)) {
+ jbd_debug(1, "JBD2: invalid revoke block found in %lu\n",
+ next_log_block);
+ need_check_commit_time = true;
+ }
/* If we aren't in the REVOKE pass, then we can
* just skip over this block. */
if (pass != PASS_REVOKE) {
@@ -796,6 +876,13 @@ static int do_one_pass(journal_t *journal,
success = -EIO;
}
}
+
+ if (jbd2_has_feature_fast_commit(journal) && pass != PASS_REVOKE) {
+ err = fc_do_one_pass(journal, info, pass);
+ if (err)
+ success = err;
+ }
+
if (block_error && success == 0)
success = -EIO;
return success;
@@ -811,7 +898,7 @@ static int scan_revoke_records(journal_t *journal, struct buffer_head *bh,
{
jbd2_journal_revoke_header_t *header;
int offset, max;
- unsigned csum_size = 0;
+ int csum_size = 0;
__u32 rcount;
int record_len = 4;
@@ -819,9 +906,6 @@ static int scan_revoke_records(journal_t *journal, struct buffer_head *bh,
offset = sizeof(jbd2_journal_revoke_header_t);
rcount = be32_to_cpu(header->r_count);
- if (!jbd2_descriptor_block_csum_verify(journal, header))
- return -EFSBADCRC;
-
if (jbd2_journal_has_csum_v2or3(journal))
csum_size = sizeof(struct jbd2_journal_block_tail);
if (rcount > journal->j_blocksize - csum_size)
diff --git a/lib/ext2fs/jfs_compat.h b/lib/ext2fs/jfs_compat.h
index 2bda521d..d6da6bb1 100644
--- a/lib/ext2fs/jfs_compat.h
+++ b/lib/ext2fs/jfs_compat.h
@@ -12,6 +12,7 @@
#else
#include <arpa/inet.h>
#endif
+#include <stdbool.h>
#define printk printf
#define KERN_ERR ""
@@ -60,9 +61,15 @@ static inline __u32 jbd2_chksum(journal_t *j EXT2FS_ATTR((unused)),
sizeof(struct __struct), __alignof__(struct __struct),\
(__flags), NULL)
-#define blkdev_issue_flush(kdev, a, b) sync_blockdev(kdev)
+#define blkdev_issue_flush(kdev, a) sync_blockdev(kdev)
#define is_power_of_2(x) ((x) != 0 && (((x) & ((x) - 1)) == 0))
#define pr_emerg(fmt)
+#define pr_err(...)
+
+enum passtype {PASS_SCAN, PASS_REVOKE, PASS_REPLAY};
+
+#define JBD2_FC_REPLAY_STOP 0
+#define JBD2_FC_REPLAY_CONTINUE 1
struct journal_s
{
@@ -73,13 +80,16 @@ struct journal_s
int j_format_version;
unsigned long j_head;
unsigned long j_tail;
+ unsigned long j_fc_first;
+ unsigned long j_fc_off;
+ unsigned long j_fc_last;
unsigned long j_free;
unsigned long j_first, j_last;
kdev_t j_dev;
kdev_t j_fs_dev;
int j_blocksize;
unsigned int j_blk_offset;
- unsigned int j_maxlen;
+ unsigned int j_total_len;
struct inode * j_inode;
tid_t j_tail_sequence;
tid_t j_transaction_sequence;
@@ -88,6 +98,11 @@ struct journal_s
struct jbd2_revoke_table_s *j_revoke_table[2];
tid_t j_failed_commit;
__u32 j_csum_seed;
+ int (*j_fc_replay_callback)(struct journal_s *journal,
+ struct buffer_head *bh,
+ enum passtype pass, int off,
+ tid_t expected_tid);
+
};
#define is_journal_abort(x) 0
diff --git a/lib/ext2fs/kernel-jbd.h b/lib/ext2fs/kernel-jbd.h
index cb1bc308..2978ccb6 100644
--- a/lib/ext2fs/kernel-jbd.h
+++ b/lib/ext2fs/kernel-jbd.h
@@ -74,6 +74,7 @@ extern void * __jbd_kmalloc (char *where, size_t size, int flags, int retry);
__jbd_kmalloc(__FUNCTION__, (size), (flags), 1)
#define JBD2_MIN_JOURNAL_BLOCKS 1024
+#define JBD2_DEFAULT_FAST_COMMIT_BLOCKS 256
/*
* Internal structures used by the logging mechanism:
@@ -94,6 +95,7 @@ extern void * __jbd_kmalloc (char *where, size_t size, int flags, int retry);
#define JBD2_SUPERBLOCK_V1 3
#define JBD2_SUPERBLOCK_V2 4
#define JBD2_REVOKE_BLOCK 5
+#define JBD2_FC_BLOCK 6
/*
* Standard header for all descriptor blocks:
@@ -233,7 +235,10 @@ typedef struct journal_superblock_s
/* 0x0050 */
__u8 s_checksum_type; /* checksum type */
__u8 s_padding2[3];
- __be32 s_padding[42];
+/* 0x0054 */
+ __be32 s_num_fc_blks; /* Number of fast commit blocks */
+/* 0x0058 */
+ __be32 s_padding[41];
__be32 s_checksum; /* crc32c(superblock) */
/* 0x0100 */
@@ -259,6 +264,7 @@ typedef struct journal_superblock_s
#define JBD2_FEATURE_INCOMPAT_ASYNC_COMMIT 0x00000004
#define JBD2_FEATURE_INCOMPAT_CSUM_V2 0x00000008
#define JBD2_FEATURE_INCOMPAT_CSUM_V3 0x00000010
+#define JBD2_FEATURE_INCOMPAT_FAST_COMMIT 0x00000020
/* Features known to this kernel version: */
#define JBD2_KNOWN_COMPAT_FEATURES 0
@@ -267,7 +273,8 @@ typedef struct journal_superblock_s
JBD2_FEATURE_INCOMPAT_ASYNC_COMMIT| \
JBD2_FEATURE_INCOMPAT_64BIT|\
JBD2_FEATURE_INCOMPAT_CSUM_V2| \
- JBD2_FEATURE_INCOMPAT_CSUM_V3)
+ JBD2_FEATURE_INCOMPAT_CSUM_V3 | \
+ JBD2_FEATURE_INCOMPAT_FAST_COMMIT)
#ifdef NO_INLINE_FUNCS
extern size_t journal_tag_bytes(journal_t *journal);
@@ -384,6 +391,7 @@ JBD2_FEATURE_INCOMPAT_FUNCS(64bit, 64BIT)
JBD2_FEATURE_INCOMPAT_FUNCS(async_commit, ASYNC_COMMIT)
JBD2_FEATURE_INCOMPAT_FUNCS(csum2, CSUM_V2)
JBD2_FEATURE_INCOMPAT_FUNCS(csum3, CSUM_V3)
+JBD2_FEATURE_INCOMPAT_FUNCS(fast_commit, FAST_COMMIT)
#if (defined(E2FSCK_INCLUDE_INLINE_FUNCS) || !defined(NO_INLINE_FUNCS))
/*
@@ -415,6 +423,13 @@ _INLINE_ int jbd2_journal_has_csum_v2or3(journal_t *journal)
return 0;
}
+_INLINE_ int jbd2_journal_get_num_fc_blks(journal_superblock_t *jsb)
+{
+ int num_fc_blocks = be32_to_cpu(jsb->s_num_fc_blks);
+
+ return num_fc_blocks ? num_fc_blocks : JBD2_DEFAULT_FAST_COMMIT_BLOCKS;
+}
+
/* Comparison functions for transaction IDs: perform comparisons using
* modulo arithmetic so that they work over sequence number wraps. */
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 04/15] mke2fs, dumpe2fs: make fast commit blocks configurable
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (2 preceding siblings ...)
2020-11-20 19:15 ` [PATCH 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Harshad Shirwadkar
@ 2020-11-20 19:15 ` Harshad Shirwadkar
2020-12-02 18:29 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 05/15] mke2fs, tune2fs: update man page with fast commit info Harshad Shirwadkar
` (10 subsequent siblings)
14 siblings, 1 reply; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
This patch makes number of fast commit blocks configurable. Also, the
number of fast commit blocks can now be seen in dumpe2fs output.
$ ./misc/mke2fs -O fast_commit -t ext4 image
mke2fs 1.46-WIP (20-Mar-2020)
Discarding device blocks: done
Creating filesystem with 5120 1k blocks and 1280 inodes
Allocating group tables: done
Writing inode tables: done
Creating journal (1040 blocks): done
Writing superblocks and filesystem accounting information: done
$ ./misc/dumpe2fs image
dumpe2fs 1.46-WIP (20-Mar-2020)
...
Journal features: (none)
Total journal size: 1040k
Total journal blocks: 1040
Max transaction length: 1024
Fast commit length: 16
Journal sequence: 0x00000001
Journal start: 0
$ ./misc/mke2fs -O fast_commit -t ext4 image -J fast_commit_size=256,size=1
mke2fs 1.46-WIP (20-Mar-2020)
Creating filesystem with 5120 1k blocks and 1280 inodes
Allocating group tables: done
Writing inode tables: done
Creating journal (1280 blocks): done
Writing superblocks and filesystem accounting information: done
$ ./misc/dumpe2fs image
dumpe2fs 1.46-WIP (20-Mar-2020)
...
Journal features: (none)
Total journal size: 1280k
Total journal blocks: 1280
Max transaction length: 1024
Fast commit length: 256
Journal sequence: 0x00000001
Journal start: 0
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
e2fsck/unix.c | 8 ++--
lib/e2p/e2p.h | 1 +
lib/e2p/ljs.c | 16 +++++--
lib/ext2fs/ext2fs.h | 21 ++++++---
lib/ext2fs/mkjournal.c | 99 +++++++++++++++++++++++++++++-------------
misc/dumpe2fs.c | 10 ++++-
misc/mke2fs.c | 24 +++++++---
misc/tune2fs.c | 8 ++--
misc/util.c | 63 ++++++++++++++++++++-------
misc/util.h | 4 +-
10 files changed, 185 insertions(+), 69 deletions(-)
diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 1cb51672..3162896a 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -1388,7 +1388,7 @@ int main (int argc, char *argv[])
blk64_t orig_superblock = ~(blk64_t)0;
struct problem_context pctx;
int flags, run_result, was_changed;
- int journal_size;
+ int journal_size, journal_fc_size;
int sysval, sys_page_size = 4096;
int old_bitmaps;
__u32 features[3];
@@ -1912,7 +1912,7 @@ print_unsupp_features:
(ctx->flags & E2F_FLAG_JOURNAL_INODE)) {
if (fix_problem(ctx, PR_6_RECREATE_JOURNAL, &pctx)) {
if (journal_size < 1024)
- journal_size = ext2fs_default_journal_size(ext2fs_blocks_count(fs->super));
+ ext2fs_default_journal_size(&journal_size, &journal_fc_size, fs);
if (journal_size < 0) {
ext2fs_clear_feature_journal(fs->super);
fs->flags &= ~EXT2_FLAG_MASTER_SB_ONLY;
@@ -1923,8 +1923,8 @@ print_unsupp_features:
log_out(ctx, _("Creating journal (%d blocks): "),
journal_size);
fflush(stdout);
- retval = ext2fs_add_journal_inode(fs,
- journal_size, 0);
+ retval = ext2fs_add_journal_inode(fs, journal_size,
+ journal_fc_size, 0);
if (retval) {
log_out(ctx, "%s: while trying to create "
"journal\n", error_message(retval));
diff --git a/lib/e2p/e2p.h b/lib/e2p/e2p.h
index 90efb624..65702a7e 100644
--- a/lib/e2p/e2p.h
+++ b/lib/e2p/e2p.h
@@ -47,6 +47,7 @@ void print_fs_state (FILE * f, unsigned short state);
int setflags (int fd, unsigned long flags);
int setversion (int fd, unsigned long version);
+#define E2P_LIST_JOURNAL_FLAG_FC 0x1
void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
int exp_block_size, int flags);
diff --git a/lib/e2p/ljs.c b/lib/e2p/ljs.c
index 4ffe9b61..9f866c7e 100644
--- a/lib/e2p/ljs.c
+++ b/lib/e2p/ljs.c
@@ -54,7 +54,12 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
unsigned int size;
int j, printed = 0;
unsigned int i, nr_users;
+ int num_fc_blks = 0;
+ int journal_blks = 0;
+ if (flags & E2P_LIST_JOURNAL_FLAG_FC)
+ num_fc_blks = jbd2_journal_get_num_fc_blks((journal_superblock_t *)journal_sb_buf);
+ journal_blks = ntohl(jsb->s_maxlen) - num_fc_blks;
fprintf(f, "%s", "Journal features: ");
for (i=0, mask_ptr=&jsb->s_feature_compat; i <3; i++,mask_ptr++) {
mask = e2p_be32(*mask_ptr);
@@ -68,7 +73,7 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
if (printed == 0)
fprintf(f, " (none)");
fputc('\n', f);
- fputs("Journal size: ", f);
+ fputs("Total journal size: ", f);
size = (ntohl(jsb->s_blocksize) / 1024) * ntohl(jsb->s_maxlen);
if (size < 8192)
fprintf(f, "%uk\n", size);
@@ -78,8 +83,13 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
if (exp_block_size != (int) ntohl(jsb->s_blocksize))
fprintf(f, "Journal block size: %u\n",
(unsigned int)ntohl(jsb->s_blocksize));
- fprintf(f, "Journal length: %u\n",
- (unsigned int)ntohl(jsb->s_maxlen));
+ fprintf(f, "Total journal blocks: %u\n",
+ (unsigned int)(journal_blks + num_fc_blks));
+ fprintf(f, "Max transaction length: %u\n",
+ (unsigned int)journal_blks);
+ fprintf(f, "Fast commit length: %u\n",
+ (unsigned int)num_fc_blks);
+
if (ntohl(jsb->s_first) != 1)
fprintf(f, "Journal first block: %u\n",
(unsigned int)ntohl(jsb->s_first));
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index a8a6e091..01132245 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -220,6 +220,12 @@ typedef struct ext2_file *ext2_file_t;
#define EXT2_MKJOURNAL_LAZYINIT 0x0000002 /* don't zero journal inode before use*/
#define EXT2_MKJOURNAL_NO_MNT_CHECK 0x0000004 /* don't check mount status */
+/*
+ * Normal journal area size to fast commit area size ratio. This is used to
+ * set default size of fast commit area.
+ */
+#define EXT2_JOURNAL_TO_FC_BLKS_RATIO 64
+
struct blk_alloc_ctx;
struct opaque_ext2_group_desc;
@@ -1625,15 +1631,18 @@ extern errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
extern errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
blk64_t *ret_blk, int *ret_count);
extern errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
- __u32 num_blocks, int flags,
- char **ret_jsb);
+ __u32 num_blocks, __u32 num_fc_blks,
+ int flags, char **ret_jsb);
+extern errcode_t ext2fs_split_journal_size(ext2_filsys fs, blk_t *journal_blks,
+ blk_t *fc_blks, blk_t total_blks);
extern errcode_t ext2fs_add_journal_device(ext2_filsys fs,
ext2_filsys journal_dev);
extern errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks,
- int flags);
+ blk_t num_fc_blocks, int flags);
extern errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
- blk64_t goal, int flags);
-extern int ext2fs_default_journal_size(__u64 num_blocks);
+ blk_t num_fc_blocks,
+ blk64_t goal, int flags);
+extern errcode_t ext2fs_default_journal_size(int *journal_size, int *fc_size, ext2_filsys fs);
extern int ext2fs_journal_sb_start(int blocksize);
/* openfs.c */
@@ -2122,6 +2131,8 @@ static inline unsigned int ext2_dir_htree_level(ext2_filsys fs)
return EXT4_HTREE_LEVEL_COMPAT;
}
+#define max(a, b) ((a) > (b) ? (a) : (b))
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c
index f47f71e6..74d0c7fc 100644
--- a/lib/ext2fs/mkjournal.c
+++ b/lib/ext2fs/mkjournal.c
@@ -12,6 +12,7 @@
#include "config.h"
#include <stdio.h>
#include <string.h>
+#include <assert.h>
#if HAVE_UNISTD_H
#include <unistd.h>
#endif
@@ -39,13 +40,32 @@
#include "kernel-jbd.h"
+errcode_t ext2fs_split_journal_size(ext2_filsys fs, blk_t *journal_blks,
+ blk_t *fc_blks, blk_t total_blks)
+{
+ if (total_blks < JBD2_MIN_JOURNAL_BLOCKS)
+ return EXT2_ET_JOURNAL_TOO_SMALL;
+
+ if (!ext2fs_has_feature_fast_commit(fs->super)) {
+ *journal_blks = total_blks;
+ *fc_blks = 0;
+ return 0;
+ }
+ *journal_blks = ext2fs_blocks_count(fs->super) *
+ EXT2_JOURNAL_TO_FC_BLKS_RATIO /
+ (EXT2_JOURNAL_TO_FC_BLKS_RATIO + 1);
+ *journal_blks = max(JBD2_MIN_JOURNAL_BLOCKS, *journal_blks);
+ *fc_blks = total_blks - *journal_blks;
+ return 0;
+}
+
/*
* This function automatically sets up the journal superblock and
* returns it as an allocated block.
*/
errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
- __u32 num_blocks, int flags,
- char **ret_jsb)
+ __u32 num_blocks, __u32 num_fc_blks,
+ int flags, char **ret_jsb)
{
errcode_t retval;
journal_superblock_t *jsb;
@@ -64,10 +84,11 @@ errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
else
jsb->s_header.h_blocktype = htonl(JBD2_SUPERBLOCK_V2);
jsb->s_blocksize = htonl(fs->blocksize);
- jsb->s_maxlen = htonl(num_blocks);
+ jsb->s_maxlen = htonl(num_blocks + num_fc_blks);
jsb->s_nr_users = htonl(1);
jsb->s_first = htonl(1);
jsb->s_sequence = htonl(1);
+ jsb->s_num_fc_blks = htonl(num_fc_blks);
memcpy(jsb->s_uuid, fs->super->s_uuid, sizeof(fs->super->s_uuid));
/*
* If we're creating an external journal device, we need to
@@ -88,14 +109,16 @@ errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
* filesystems.
*/
static errcode_t write_journal_file(ext2_filsys fs, char *filename,
- blk_t num_blocks, int flags)
+ blk_t num_blocks, blk_t num_fc_blocks,
+ int flags)
{
errcode_t retval;
char *buf = 0;
int fd, ret_size;
blk_t i;
- if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags,
+ if ((retval = ext2fs_create_journal_superblock(fs, num_blocks,
+ num_fc_blocks, flags,
&buf)))
return retval;
@@ -119,7 +142,7 @@ static errcode_t write_journal_file(ext2_filsys fs, char *filename,
if (flags & EXT2_MKJOURNAL_LAZYINIT)
goto success;
- for (i = 1; i < num_blocks; i++) {
+ for (i = 1; i < num_blocks + num_fc_blocks; i++) {
ret_size = write(fd, buf, fs->blocksize);
if (ret_size < 0) {
retval = errno;
@@ -262,7 +285,8 @@ static blk64_t get_midpoint_journal_block(ext2_filsys fs)
* This function creates a journal using direct I/O routines.
*/
static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
- blk_t num_blocks, blk64_t goal, int flags)
+ blk_t num_blocks, blk_t num_fc_blocks,
+ blk64_t goal, int flags)
{
char *buf;
errcode_t retval;
@@ -271,7 +295,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
int falloc_flags = EXT2_FALLOCATE_FORCE_INIT;
blk64_t zblk;
- if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags,
+ if ((retval = ext2fs_create_journal_superblock(fs, num_blocks,
+ num_fc_blocks, flags,
&buf)))
return retval;
@@ -295,7 +320,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
if (!(flags & EXT2_MKJOURNAL_LAZYINIT))
falloc_flags |= EXT2_FALLOCATE_ZERO_BLOCKS;
- inode_size = (unsigned long long)fs->blocksize * num_blocks;
+ inode_size = (unsigned long long)fs->blocksize *
+ (num_blocks + num_fc_blocks);
inode.i_mtime = inode.i_ctime = fs->now ? fs->now : time(0);
inode.i_links_count = 1;
inode.i_mode = LINUX_S_IFREG | 0600;
@@ -304,7 +330,7 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
goto out2;
retval = ext2fs_fallocate(fs, falloc_flags, journal_ino,
- &inode, goal, 0, num_blocks);
+ &inode, goal, 0, num_blocks + num_fc_blocks);
if (retval)
goto out2;
@@ -337,25 +363,34 @@ out2:
*
* n.b. comments assume 4k blocks
*/
-int ext2fs_default_journal_size(__u64 num_blocks)
+errcode_t ext2fs_default_journal_size(int *journal_size, int *fc_size, ext2_filsys fs)
{
+ __u64 num_blocks = ext2fs_blocks_count(fs->super);
+
if (num_blocks < 2048)
- return -1;
+ return EXT2_ET_TOOSMALL;
if (num_blocks < 32768) /* 128 MB */
- return (1024); /* 4 MB */
- if (num_blocks < 256*1024) /* 1 GB */
- return (4096); /* 16 MB */
- if (num_blocks < 512*1024) /* 2 GB */
- return (8192); /* 32 MB */
- if (num_blocks < 4096*1024) /* 16 GB */
- return (16384); /* 64 MB */
- if (num_blocks < 8192*1024) /* 32 GB */
- return (32768); /* 128 MB */
- if (num_blocks < 16384*1024) /* 64 GB */
- return (65536); /* 256 MB */
- if (num_blocks < 32768*1024) /* 128 GB */
- return (131072); /* 512 MB */
- return 262144; /* 1 GB */
+ *journal_size = 1024; /* 4 MB */
+ else if (num_blocks < 256*1024) /* 1 GB */
+ *journal_size = 4096; /* 16 MB */
+ else if (num_blocks < 512*1024) /* 2 GB */
+ *journal_size = 8192; /* 32 MB */
+ else if (num_blocks < 4096*1024) /* 16 GB */
+ *journal_size = 16384; /* 64 MB */
+ else if (num_blocks < 8192*1024) /* 32 GB */
+ *journal_size = 32768; /* 128 MB */
+ else if (num_blocks < 16384*1024) /* 64 GB */
+ *journal_size = 65536; /* 256 MB */
+ else if (num_blocks < 32768*1024) /* 128 GB */
+ *journal_size = 131072; /* 512 MB */
+ else
+ *journal_size = 262144; /* 1 GB */
+
+ *fc_size = ext2fs_has_feature_fast_commit(fs->super) ?
+ *journal_size / EXT2_JOURNAL_TO_FC_BLKS_RATIO : 0;
+ assert(*journal_size + *fc_size < num_blocks);
+
+ return 0;
}
int ext2fs_journal_sb_start(int blocksize)
@@ -435,6 +470,7 @@ errcode_t ext2fs_add_journal_device(ext2_filsys fs, ext2_filsys journal_dev)
* functions if it is not.
*/
errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
+ blk_t num_fc_blocks,
blk64_t goal, int flags)
{
errcode_t retval;
@@ -486,7 +522,8 @@ errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
* filesystems is extremely rare these days... Ignore it. */
flags &= ~EXT2_MKJOURNAL_LAZYINIT;
- if ((retval = write_journal_file(fs, jfile, num_blocks, flags)))
+ if ((retval = write_journal_file(fs, jfile,
+ num_blocks, num_fc_blocks, flags)))
goto errout;
/* Get inode number of the journal file */
@@ -528,7 +565,8 @@ errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
}
journal_ino = EXT2_JOURNAL_INO;
if ((retval = write_journal_inode(fs, journal_ino,
- num_blocks, goal, flags)))
+ num_blocks, num_fc_blocks,
+ goal, flags)))
return retval;
}
@@ -546,9 +584,10 @@ errout:
return retval;
}
-errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks, int flags)
+errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks,
+ blk_t num_fc_blocks, int flags)
{
- return ext2fs_add_journal_inode2(fs, num_blocks, ~0ULL, flags);
+ return ext2fs_add_journal_inode2(fs, num_blocks, num_fc_blocks, ~0ULL, flags);
}
diff --git a/misc/dumpe2fs.c b/misc/dumpe2fs.c
index d295ba4d..e24dc4e6 100644
--- a/misc/dumpe2fs.c
+++ b/misc/dumpe2fs.c
@@ -364,6 +364,7 @@ static void print_inline_journal_information(ext2_filsys fs)
errcode_t retval;
ext2_ino_t ino = fs->super->s_journal_inum;
char buf[1024];
+ int flags;
if (fs->flags & EXT2_FLAG_IMAGE_FILE)
return;
@@ -392,7 +393,9 @@ static void print_inline_journal_information(ext2_filsys fs)
_("Journal superblock magic number invalid!\n"));
exit(1);
}
- e2p_list_journal_super(stdout, buf, fs->blocksize, 0);
+ flags = ext2fs_has_feature_fast_commit(fs->super) ?
+ E2P_LIST_JOURNAL_FLAG_FC : 0;
+ e2p_list_journal_super(stdout, buf, fs->blocksize, flags);
}
static void print_journal_information(ext2_filsys fs)
@@ -400,6 +403,7 @@ static void print_journal_information(ext2_filsys fs)
errcode_t retval;
char buf[1024];
journal_superblock_t *jsb;
+ int flags;
/* Get the journal superblock */
if ((retval = io_channel_read_blk64(fs->io,
@@ -417,7 +421,9 @@ static void print_journal_information(ext2_filsys fs)
_("Couldn't find journal superblock magic numbers"));
exit(1);
}
- e2p_list_journal_super(stdout, buf, fs->blocksize, 0);
+ flags = ext2fs_has_feature_fast_commit(fs->super) ?
+ E2P_LIST_JOURNAL_FLAG_FC : 0;
+ e2p_list_journal_super(stdout, buf, fs->blocksize, flags);
}
static int check_mmp(ext2_filsys fs)
diff --git a/misc/mke2fs.c b/misc/mke2fs.c
index 8c8f5ea4..2d9a5449 100644
--- a/misc/mke2fs.c
+++ b/misc/mke2fs.c
@@ -93,6 +93,7 @@ static uid_t root_uid;
static gid_t root_gid;
int journal_size;
int journal_flags;
+int journal_fc_size;
static int lazy_itable_init;
static int packed_meta_blocks;
int no_copy_xattrs;
@@ -604,9 +605,18 @@ static void create_journal_dev(ext2_filsys fs)
char *buf;
blk64_t blk, err_blk;
int c, count, err_count;
+ int num_journal_blks = 0, num_fc_blks = 0;
+
+ retval = ext2fs_split_journal_size(fs, &num_journal_blks, &num_fc_blks,
+ ext2fs_blocks_count(fs->super));
+ if (retval) {
+ com_err("create_journal_dev", retval, "%s",
+ _("while splitting the journal size"));
+ exit(1);
+ }
retval = ext2fs_create_journal_superblock(fs,
- ext2fs_blocks_count(fs->super), 0, &buf);
+ num_journal_blks, num_fc_blks, 0, &buf);
if (retval) {
com_err("create_journal_dev", retval, "%s",
_("while initializing journal superblock"));
@@ -1753,6 +1763,8 @@ profile_error:
case 'j':
if (!journal_size)
journal_size = -1;
+ if (!journal_fc_size)
+ journal_fc_size = -1;
break;
case 'J':
parse_journal_opts(optarg);
@@ -2937,7 +2949,7 @@ int main (int argc, char *argv[])
badblocks_list bb_list = 0;
badblocks_iterate bb_iter;
blk_t blk;
- unsigned int journal_blocks = 0;
+ blk_t journal_blocks = 0, journal_fc_blocks = 0;
unsigned int i, checkinterval;
int max_mnt_count;
int val, hash_alg;
@@ -3047,7 +3059,8 @@ int main (int argc, char *argv[])
/* Calculate journal blocks */
if (!journal_device && ((journal_size) ||
ext2fs_has_feature_journal(&fs_param)))
- journal_blocks = figure_journal_size(journal_size, fs);
+ figure_journal_size(&journal_blocks, &journal_fc_blocks,
+ journal_size, journal_fc_size, fs);
sprintf(opt_string, "tdb_data_size=%d", fs->blocksize <= 4096 ?
32768 : fs->blocksize * 8);
@@ -3382,7 +3395,7 @@ int main (int argc, char *argv[])
free(journal_device);
} else if ((journal_size) ||
ext2fs_has_feature_journal(&fs_param)) {
- overhead += EXT2FS_NUM_B2C(fs, journal_blocks);
+ overhead += EXT2FS_NUM_B2C(fs, journal_blocks + journal_fc_blocks);
if (super_only) {
printf("%s", _("Skipping journal creation in super-only mode\n"));
fs->super->s_journal_inum = EXT2_JOURNAL_INO;
@@ -3395,10 +3408,11 @@ int main (int argc, char *argv[])
}
if (!quiet) {
printf(_("Creating journal (%u blocks): "),
- journal_blocks);
+ journal_blocks + journal_fc_blocks);
fflush(stdout);
}
retval = ext2fs_add_journal_inode2(fs, journal_blocks,
+ journal_fc_blocks,
journal_location,
journal_flags);
if (retval) {
diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index 670ed9e0..daee154d 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -102,7 +102,7 @@ static int feature_64bit;
static int fsck_requested;
static char *undo_file;
-int journal_size, journal_flags;
+int journal_size, journal_fc_size, journal_flags;
char *journal_device;
static blk64_t journal_location = ~0LL;
@@ -1543,7 +1543,7 @@ mmp_error:
*/
static int add_journal(ext2_filsys fs)
{
- unsigned long journal_blocks;
+ blk_t journal_blocks, journal_fc_blocks;
errcode_t retval;
ext2_filsys jfs;
io_manager io_ptr;
@@ -1589,13 +1589,15 @@ static int add_journal(ext2_filsys fs)
} else if (journal_size) {
fputs(_("Creating journal inode: "), stdout);
fflush(stdout);
- journal_blocks = figure_journal_size(journal_size, fs);
+ figure_journal_size(&journal_blocks, &journal_fc_blocks,
+ journal_size, journal_fc_size, fs);
if (journal_location_string)
journal_location =
parse_num_blocks2(journal_location_string,
fs->super->s_log_block_size);
retval = ext2fs_add_journal_inode2(fs, journal_blocks,
+ journal_fc_blocks,
journal_location,
journal_flags);
if (retval) {
diff --git a/misc/util.c b/misc/util.c
index dcd2f0a7..85f10a70 100644
--- a/misc/util.c
+++ b/misc/util.c
@@ -200,6 +200,14 @@ void parse_journal_opts(const char *opts)
journal_size = strtoul(arg, &p, 0);
if (*p)
journal_usage++;
+ } else if (strcmp(token, "fast_commit_size") == 0) {
+ if (!arg) {
+ journal_usage++;
+ continue;
+ }
+ journal_fc_size = strtoul(arg, &p, 0);
+ if (*p)
+ journal_usage++;
} else if (!strcmp(token, "location")) {
if (!arg) {
journal_usage++;
@@ -229,42 +237,65 @@ void parse_journal_opts(const char *opts)
free(buf);
}
+static inline int jsize_to_blks(ext2_filsys fs, int size)
+{
+ return (size * 1024) / (fs->blocksize / 1024);
+}
+
+/* Fast commit size is in KBs */
+static inline int fcsize_to_blks(ext2_filsys fs, int size)
+{
+ return (size * 1024) / (fs->blocksize);
+}
+
/*
* Determine the number of journal blocks to use, either via
* user-specified # of megabytes, or via some intelligently selected
* defaults.
*
- * Find a reasonable journal file size (in blocks) given the number of blocks
- * in the filesystem. For very small filesystems, it is not reasonable to
- * have a journal that fills more than half of the filesystem.
+ * Find a reasonable journal file size (in blocks) given the number of blocks in
+ * the filesystem. For very small filesystems, it is not reasonable to have a
+ * journal that fills more than half of the filesystem.
*/
-unsigned int figure_journal_size(int size, ext2_filsys fs)
+void figure_journal_size(blk_t *j_blocks, blk_t *fc_blocks,
+ int requested_j_size, int requested_fc_size, ext2_filsys fs)
{
- int j_blocks;
+ int def_j_blocks, def_fc_blocks, total_blocks;
+ int ret;
- j_blocks = ext2fs_default_journal_size(ext2fs_blocks_count(fs->super));
- if (j_blocks < 0) {
+ ret = ext2fs_default_journal_size(&def_j_blocks, &def_fc_blocks, fs);
+ if (ret) {
fputs(_("\nFilesystem too small for a journal\n"), stderr);
- return 0;
+ return;
}
- if (size > 0) {
- j_blocks = size * 1024 / (fs->blocksize / 1024);
- if (j_blocks < 1024 || j_blocks > 10240000) {
- fprintf(stderr, _("\nThe requested journal "
+ *j_blocks = def_j_blocks;
+ *fc_blocks = def_fc_blocks;
+
+ if (requested_j_size > 0 ||
+ (ext2fs_has_feature_fast_commit(fs->super) && requested_fc_size > 0)) {
+ *j_blocks = requested_j_size > 0 ?
+ jsize_to_blks(fs, requested_j_size) : def_j_blocks;
+
+ *fc_blocks = 0;
+ if (ext2fs_has_feature_fast_commit(fs->super))
+ *fc_blocks = requested_fc_size > 0 ?
+ fcsize_to_blks(fs, requested_fc_size) : def_fc_blocks;
+ total_blocks = *j_blocks + *fc_blocks;
+ if (total_blocks < 1024 || total_blocks > 10240000) {
+ fprintf(stderr, _("\nThe total requested journal "
"size is %d blocks; it must be\n"
"between 1024 and 10240000 blocks. "
"Aborting.\n"),
- j_blocks);
+ total_blocks);
exit(1);
}
- if ((unsigned) j_blocks > ext2fs_free_blocks_count(fs->super) / 2) {
- fputs(_("\nJournal size too big for filesystem.\n"),
+ if ((unsigned int) total_blocks > ext2fs_free_blocks_count(fs->super) / 2) {
+ fputs(_("\nTotal journal size too big for filesystem.\n"),
stderr);
exit(1);
}
}
- return j_blocks;
}
void print_check_message(int mnt, unsigned int check)
diff --git a/misc/util.h b/misc/util.h
index 49b4b9c1..cee812b4 100644
--- a/misc/util.h
+++ b/misc/util.h
@@ -11,6 +11,7 @@
*/
extern int journal_size;
+extern int journal_fc_size;
extern int journal_flags;
extern char *journal_device;
extern char *journal_location_string;
@@ -22,6 +23,7 @@ extern char *get_progname(char *argv_zero);
extern void proceed_question(int delay);
extern void parse_journal_opts(const char *opts);
extern void check_mount(const char *device, int force, const char *type);
-extern unsigned int figure_journal_size(int size, ext2_filsys fs);
+extern void figure_journal_size(blk_t *j_blocks, blk_t *fc_blocks,
+ int requested_j_size, int requested_fc_size, ext2_filsys fs);
extern void print_check_message(int, unsigned int);
extern void dump_mmp_msg(struct mmp_struct *mmp, const char *msg);
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 05/15] mke2fs, tune2fs: update man page with fast commit info
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (3 preceding siblings ...)
2020-11-20 19:15 ` [PATCH 04/15] mke2fs, dumpe2fs: make fast commit blocks configurable Harshad Shirwadkar
@ 2020-11-20 19:15 ` Harshad Shirwadkar
2020-12-02 18:33 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 06/15] ext2fs: add new APIs needed for fast commits Harshad Shirwadkar
` (9 subsequent siblings)
14 siblings, 1 reply; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
This patch adds information about fast commit feature in mke2fs and
tune2fs man pages.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
misc/mke2fs.8.in | 21 +++++++++++++++++++++
misc/tune2fs.8.in | 25 +++++++++++++++++++++++++
2 files changed, 46 insertions(+)
diff --git a/misc/mke2fs.8.in b/misc/mke2fs.8.in
index e6bfc6d6..2833b408 100644
--- a/misc/mke2fs.8.in
+++ b/misc/mke2fs.8.in
@@ -521,6 +521,27 @@ The size of the journal must be at least 1024 filesystem blocks
and may be no more than 10,240,000 filesystem blocks or half the total
file system size (whichever is smaller)
.TP
+.BI fast_commit_size= fast-commit-size
+Create an additional fast commit journal area of size
+.I fast-commit-size
+kilobytes.
+This option is only valid if
+.B fast_commit
+feature is enabled
+on the file system. If this option is not specified and if
+.B fast_commit
+feature is turned on, fast commit area size defaults to
+.I journal-size
+/ 64 megabytes. The total size of the journal with
+.B fast_commit
+feature set is
+.I journal-size
++ (
+.I fast-commit-size
+* 1024) megabytes. The total journal size may be no more than
+10,240,000 filesystem blocks or half the total file system size
+(whichever is smaller).
+.TP
.BI location =journal-location
Specify the location of the journal. The argument
.I journal-location
diff --git a/misc/tune2fs.8.in b/misc/tune2fs.8.in
index 582d1da5..2114c623 100644
--- a/misc/tune2fs.8.in
+++ b/misc/tune2fs.8.in
@@ -357,6 +357,27 @@ and may be no more than 10,240,000 filesystem blocks.
There must be enough free space in the filesystem to create a journal of
that size.
.TP
+.BI fast_commit_size= fast-commit-size
+Create an additional fast commit journal area of size
+.I fast-commit-size
+kilobytes.
+This option is only valid if
+.B fast_commit
+feature is enabled
+on the file system. If this option is not specified and if
+.B fast_commit
+feature is turned on, fast commit area size defaults to
+.I journal-size
+/ 64 megabytes. The total size of the journal with
+.B fast_commit
+feature set is
+.I journal-size
++ (
+.I fast-commit-size
+* 1024) megabytes. The total journal size may be no more than
+10,240,000 filesystem blocks or half the total file system size
+(whichever is smaller).
+.TP
.BI location =journal-location
Specify the location of the journal. The argument
.I journal-location
@@ -586,6 +607,10 @@ Setting the filesystem feature is equivalent to using the
.B \-j
option.
.TP
+.TP
+.B fast_commit
+Enable fast commit journaling feature to improve fsync latency.
+.TP
.B large_dir
Increase the limit on the number of files per directory.
.B Tune2fs
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 06/15] ext2fs: add new APIs needed for fast commits
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (4 preceding siblings ...)
2020-11-20 19:15 ` [PATCH 05/15] mke2fs, tune2fs: update man page with fast commit info Harshad Shirwadkar
@ 2020-11-20 19:15 ` Harshad Shirwadkar
2020-12-02 18:44 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 07/15] e2fsck: add function to rewrite extent tree Harshad Shirwadkar
` (8 subsequent siblings)
14 siblings, 1 reply; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
This patch adds the following new APIs:
Count the total number of blocks occupied by inode including
intermediate extent tree nodes.
extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
struct ext2_inode *inode);
Convert ext3_extent to ext2fs_extent.
extern void ext2fs_convert_extent(struct ext2fs_extent *to,
struct ext3_extent *from);
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
lib/ext2fs/ext2fs.h | 4 ++++
lib/ext2fs/extent.c | 56 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 60 insertions(+)
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 01132245..afa9c5e4 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1332,6 +1332,10 @@ extern errcode_t ext2fs_extent_fix_parents(ext2_extent_handle_t handle);
extern size_t ext2fs_max_extent_depth(ext2_extent_handle_t handle);
extern errcode_t ext2fs_fix_extents_checksums(ext2_filsys fs, ext2_ino_t ino,
struct ext2_inode *inode);
+extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
+ struct ext2_inode *inode);
+extern void ext2fs_convert_extent(struct ext2fs_extent *to,
+ struct ext3_extent *from);
/* fallocate.c */
#define EXT2_FALLOCATE_ZERO_BLOCKS (0x1)
diff --git a/lib/ext2fs/extent.c b/lib/ext2fs/extent.c
index ac3dbfec..43feea0a 100644
--- a/lib/ext2fs/extent.c
+++ b/lib/ext2fs/extent.c
@@ -1785,6 +1785,62 @@ out:
return errcode;
}
+void ext2fs_convert_extent(struct ext2fs_extent *to, struct ext3_extent *from)
+{
+ to->e_pblk = ext2fs_le32_to_cpu(from->ee_start) +
+ ((__u64) ext2fs_le16_to_cpu(from->ee_start_hi)
+ << 32);
+ to->e_lblk = ext2fs_le32_to_cpu(from->ee_block);
+ to->e_len = ext2fs_le16_to_cpu(from->ee_len);
+ to->e_flags |= EXT2_EXTENT_FLAGS_LEAF;
+ if (to->e_len > EXT_INIT_MAX_LEN) {
+ to->e_len -= EXT_INIT_MAX_LEN;
+ to->e_flags |= EXT2_EXTENT_FLAGS_UNINIT;
+ }
+}
+
+blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
+ struct ext2_inode *inode)
+{
+ ext2_extent_handle_t handle;
+ struct ext2fs_extent extent;
+ errcode_t errcode;
+ int i;
+ blk64_t blkcount = 0;
+ blk64_t *intermediate_nodes;
+
+ errcode = ext2fs_extent_open2(fs, ino, inode, &handle);
+ if (errcode)
+ goto out;
+
+ errcode = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
+ if (errcode)
+ goto out;
+
+ ext2fs_get_array(handle->max_depth, sizeof(blk64_t),
+ &intermediate_nodes);
+ blkcount = handle->level;
+ while (!errcode) {
+ if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) {
+ blkcount += extent.e_len;
+ for (i = 0; i < handle->level; i++) {
+ if (intermediate_nodes[i] !=
+ handle->path[i].end_blk) {
+ blkcount++;
+ intermediate_nodes[i] =
+ handle->path[i].end_blk;
+ }
+ }
+ }
+ errcode = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent);
+ }
+ ext2fs_free_mem(&intermediate_nodes);
+out:
+ ext2fs_extent_free(handle);
+
+ return blkcount;
+}
+
#ifdef DEBUG
/*
* Override debugfs's prompt
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 07/15] e2fsck: add function to rewrite extent tree
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (5 preceding siblings ...)
2020-11-20 19:15 ` [PATCH 06/15] ext2fs: add new APIs needed for fast commits Harshad Shirwadkar
@ 2020-11-20 19:15 ` Harshad Shirwadkar
2020-12-02 18:46 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 08/15] e2fsck: add fast commit setup code Harshad Shirwadkar
` (7 subsequent siblings)
14 siblings, 1 reply; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
Fast commit replay needs to rewrite the entire extent tree for inodes
found in fast commit area. This patch makes e2fsck's rewrite extent
tree path visible.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
e2fsck/e2fsck.h | 16 +++++
e2fsck/extents.c | 168 ++++++++++++++++++++++++++++++-----------------
2 files changed, 124 insertions(+), 60 deletions(-)
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 85f953b2..3b9c1874 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -226,6 +226,19 @@ typedef struct e2fsck_struct *e2fsck_t;
#define MAX_EXTENT_DEPTH_COUNT 5
+/*
+ * This strucutre is used to manage the list of extents in a file. Placing
+ * it here since this is used by fast_commit.h.
+ */
+struct extent_list {
+ blk64_t blocks_freed;
+ struct ext2fs_extent *extents;
+ unsigned int count;
+ unsigned int size;
+ unsigned int ext_read;
+ errcode_t retval;
+ ext2_ino_t ino;
+};
struct e2fsck_struct {
ext2_filsys fs;
const char *program_name;
@@ -536,6 +549,9 @@ errcode_t e2fsck_should_rebuild_extents(e2fsck_t ctx,
struct problem_context *pctx,
struct extent_tree_info *eti,
struct ext2_extent_info *info);
+errcode_t e2fsck_read_extents(e2fsck_t ctx, struct extent_list *extents);
+errcode_t e2fsck_rewrite_extent_tree(e2fsck_t ctx,
+ struct extent_list *extents);
/* journal.c */
extern errcode_t e2fsck_check_ext3_journal(e2fsck_t ctx);
diff --git a/e2fsck/extents.c b/e2fsck/extents.c
index e9139326..d6c74834 100644
--- a/e2fsck/extents.c
+++ b/e2fsck/extents.c
@@ -58,16 +58,6 @@ int e2fsck_ino_will_be_rebuilt(e2fsck_t ctx, ext2_ino_t ino)
return ext2fs_test_inode_bitmap2(ctx->inodes_to_rebuild, ino);
}
-struct extent_list {
- blk64_t blocks_freed;
- struct ext2fs_extent *extents;
- unsigned int count;
- unsigned int size;
- unsigned int ext_read;
- errcode_t retval;
- ext2_ino_t ino;
-};
-
static errcode_t load_extents(e2fsck_t ctx, struct extent_list *list)
{
ext2_filsys fs = ctx->fs;
@@ -206,66 +196,40 @@ static int find_blocks(ext2_filsys fs, blk64_t *blocknr, e2_blkcnt_t blockcnt,
return 0;
}
-static errcode_t rebuild_extent_tree(e2fsck_t ctx, struct extent_list *list,
- ext2_ino_t ino)
+errcode_t rewrite_extent_replay(e2fsck_t ctx, struct extent_list *list,
+ struct ext2_inode_large *inode)
{
- struct ext2_inode_large inode;
errcode_t retval;
ext2_extent_handle_t handle;
unsigned int i, ext_written;
struct ext2fs_extent *ex, extent;
- blk64_t start_val, delta;
-
- list->count = 0;
- list->blocks_freed = 0;
- list->ino = ino;
- list->ext_read = 0;
- e2fsck_read_inode_full(ctx, ino, EXT2_INODE(&inode), sizeof(inode),
- "rebuild_extents");
+ blk64_t start_val, delta, blkcount;
- /* Skip deleted inodes and inline data files */
- if (inode.i_links_count == 0 ||
- inode.i_flags & EXT4_INLINE_DATA_FL)
- return 0;
-
- /* Collect lblk->pblk mappings */
- if (inode.i_flags & EXT4_EXTENTS_FL) {
- retval = load_extents(ctx, list);
- if (retval)
- goto err;
- goto extents_loaded;
- }
-
- retval = ext2fs_block_iterate3(ctx->fs, ino, BLOCK_FLAG_READ_ONLY, 0,
- find_blocks, list);
- if (retval)
- goto err;
- if (list->retval) {
- retval = list->retval;
- goto err;
- }
-
-extents_loaded:
/* Reset extent tree */
- inode.i_flags &= ~EXT4_EXTENTS_FL;
- memset(inode.i_block, 0, sizeof(inode.i_block));
+ inode->i_flags &= ~EXT4_EXTENTS_FL;
+ memset(inode->i_block, 0, sizeof(inode->i_block));
/* Make a note of freed blocks */
- quota_data_sub(ctx->qctx, &inode, ino,
+ quota_data_sub(ctx->qctx, inode, list->ino,
list->blocks_freed * ctx->fs->blocksize);
- retval = ext2fs_iblk_sub_blocks(ctx->fs, EXT2_INODE(&inode),
+ retval = ext2fs_iblk_sub_blocks(ctx->fs, EXT2_INODE(inode),
list->blocks_freed);
if (retval)
- goto err;
+ return retval;
/* Now stuff extents into the file */
- retval = ext2fs_extent_open2(ctx->fs, ino, EXT2_INODE(&inode), &handle);
+ retval = ext2fs_extent_open2(ctx->fs, list->ino, EXT2_INODE(inode),
+ &handle);
if (retval)
- goto err;
+ return retval;
ext_written = 0;
- start_val = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(&inode));
+
+ start_val = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(inode));
+
for (i = 0, ex = list->extents; i < list->count; i++, ex++) {
+ if (ex->e_len == 0)
+ continue;
memcpy(&extent, ex, sizeof(struct ext2fs_extent));
extent.e_flags &= EXT2_EXTENT_FLAGS_UNINIT;
if (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) {
@@ -289,36 +253,120 @@ extents_loaded:
}
#ifdef DEBUG
- printf("W: ino=%d pblk=%llu lblk=%llu len=%u\n", ino,
+ printf("W: ino=%d pblk=%llu lblk=%llu len=%u\n", list->ino,
extent.e_pblk, extent.e_lblk, extent.e_len);
#endif
retval = ext2fs_extent_insert(handle, EXT2_EXTENT_INSERT_AFTER,
&extent);
if (retval)
- goto err2;
+ goto err;
retval = ext2fs_extent_fix_parents(handle);
if (retval)
- goto err2;
+ goto err;
ext_written++;
}
- delta = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(&inode)) -
+ delta = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(inode)) -
start_val;
if (delta)
- quota_data_add(ctx->qctx, &inode, ino, delta << 9);
+ quota_data_add(ctx->qctx, inode, list->ino, delta << 9);
#if defined(DEBUG) || defined(DEBUG_SUMMARY)
printf("rebuild: ino=%d extents=%d->%d\n", ino, list->ext_read,
ext_written);
#endif
- e2fsck_write_inode(ctx, ino, EXT2_INODE(&inode), "rebuild_extents");
+ e2fsck_write_inode(ctx, list->ino, EXT2_INODE(inode),
+ "rebuild_extents");
-err2:
- ext2fs_extent_free(handle);
err:
+ ext2fs_extent_free(handle);
+ return retval;
+}
+
+errcode_t e2fsck_rewrite_extent_tree(e2fsck_t ctx, struct extent_list *list)
+{
+ struct ext2_inode_large inode;
+ int ret;
+
+ memset(&inode, 0, sizeof(inode));
+ ext2fs_read_inode_full(ctx->fs, list->ino, EXT2_INODE(&inode),
+ sizeof(inode));
+
+ /* Skip deleted inodes and inline data files */
+ if (inode.i_flags & EXT4_INLINE_DATA_FL)
+ return 0;
+
+ ret = rewrite_extent_replay(ctx, list, &inode);
+ ext2fs_iblk_set(ctx->fs, EXT2_INODE(&inode),
+ ext2fs_count_blocks(ctx->fs, list->ino, EXT2_INODE(&inode)));
+ ext2fs_write_inode_full(ctx->fs, list->ino, EXT2_INODE(&inode),
+ sizeof(inode));
+
+ return ret;
+}
+
+errcode_t e2fsck_read_extents(e2fsck_t ctx, struct extent_list *extents)
+{
+ struct ext2_inode_large inode;
+ errcode_t retval;
+
+ extents->extents = NULL;
+ extents->count = 0;
+ extents->blocks_freed = 0;
+ extents->ext_read = 0;
+ extents->size = NUM_EXTENTS;
+ retval = ext2fs_get_array(NUM_EXTENTS, sizeof(struct ext2fs_extent),
+ &extents->extents);
+ if (retval)
+ return -ENOMEM;
+
+ retval = ext2fs_read_inode(ctx->fs, extents->ino, EXT2_INODE(&inode));
+ if (retval)
+ goto err_out;
+
+ retval = load_extents(ctx, extents);
+ if (!retval)
+ return 0;
+err_out:
+ ext2fs_free_mem(&extents->extents);
+ extents->size = 0;
+ extents->count = 0;
return retval;
}
+static errcode_t rebuild_extent_tree(e2fsck_t ctx, struct extent_list *list,
+ ext2_ino_t ino)
+{
+ struct ext2_inode_large inode;
+ errcode_t retval;
+
+ list->count = 0;
+ list->blocks_freed = 0;
+ list->ino = ino;
+ list->ext_read = 0;
+ e2fsck_read_inode_full(ctx, ino, EXT2_INODE(&inode), sizeof(inode),
+ "rebuild_extents");
+
+ /* Skip deleted inodes and inline data files */
+ if (inode.i_links_count == 0 ||
+ inode.i_flags & EXT4_INLINE_DATA_FL)
+ return 0;
+
+ /* Collect lblk->pblk mappings */
+ if (inode.i_flags & EXT4_EXTENTS_FL) {
+ retval = load_extents(ctx, list);
+ if (retval)
+ return retval;
+ return rewrite_extent_replay(ctx, list, &inode);
+ }
+
+ retval = ext2fs_block_iterate3(ctx->fs, ino, BLOCK_FLAG_READ_ONLY, 0,
+ find_blocks, list);
+
+ return retval || list->retval ||
+ rewrite_extent_replay(ctx, list, &inode);
+}
+
/* Rebuild the extents immediately */
static errcode_t e2fsck_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino)
{
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 08/15] e2fsck: add fast commit setup code
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (6 preceding siblings ...)
2020-11-20 19:15 ` [PATCH 07/15] e2fsck: add function to rewrite extent tree Harshad Shirwadkar
@ 2020-11-20 19:15 ` Harshad Shirwadkar
2020-12-02 18:48 ` Theodore Y. Ts'o
2020-11-20 19:16 ` [PATCH 09/15] e2fsck: add fast commit scan pass Harshad Shirwadkar
` (6 subsequent siblings)
14 siblings, 1 reply; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:15 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
Add fast_commit.h that contains the necessary helpers needed for fast
commit replay. Note that this file is also byte by byte identical with
kernel's fast_commit.h. Also, we introduce the
"e2fsck_fc_replay_state" structure which is needed for ext4 fast
commit replay.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
e2fsck/e2fsck.h | 16 ++++
e2fsck/journal.c | 15 +++
lib/ext2fs/ext2_fs.h | 1 +
lib/ext2fs/ext2fs.h | 3 +
lib/ext2fs/fast_commit.h | 201 +++++++++++++++++++++++++++++++++++++++
5 files changed, 236 insertions(+)
create mode 100644 lib/ext2fs/fast_commit.h
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 3b9c1874..f75cc343 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -68,6 +68,7 @@
#endif
#include "support/quotaio.h"
+#include "ext2fs/fast_commit.h"
/*
* Exit codes used by fsck-type programs
@@ -239,6 +240,18 @@ struct extent_list {
errcode_t retval;
ext2_ino_t ino;
};
+
+/* State structure for fast commit replay */
+struct e2fsck_fc_replay_state {
+ struct extent_list fc_extent_list;
+ int fc_replay_num_tags;
+ int fc_replay_expected_off;
+ int fc_current_pass;
+ int fc_cur_tag;
+ int fc_crc;
+ __u16 fc_super_state;
+};
+
struct e2fsck_struct {
ext2_filsys fs;
const char *program_name;
@@ -431,6 +444,9 @@ struct e2fsck_struct {
/* Undo file */
char *undo_file;
+
+ /* Fast commit replay state */
+ struct e2fsck_fc_replay_state fc_replay_state;
};
/* Data structures to evaluate whether an extent tree needs rebuilding. */
diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 75fefcde..2c8e3441 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -278,6 +278,17 @@ static int process_journal_block(ext2_filsys fs,
return 0;
}
+/*
+ * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE
+ * to indicate that it is expecting more fast commit blocks. It returns
+ * JBD2_FC_REPLAY_STOP to indicate that replay is done.
+ */
+static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
+ enum passtype pass, int off, tid_t expected_tid)
+{
+ return JBD2_FC_REPLAY_STOP;
+}
+
static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal)
{
struct process_block_struct pb;
@@ -514,6 +525,10 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal)
journal->j_sb_buffer = bh;
journal->j_superblock = (journal_superblock_t *)bh->b_data;
+ if (ext2fs_has_feature_fast_commit(ctx->fs->super))
+ journal->j_fc_replay_callback = ext4_fc_replay;
+ else
+ journal->j_fc_replay_callback = NULL;
#ifdef USE_INODE_IO
if (j_inode)
diff --git a/lib/ext2fs/ext2_fs.h b/lib/ext2fs/ext2_fs.h
index bfc30c29..b1e4329c 100644
--- a/lib/ext2fs/ext2_fs.h
+++ b/lib/ext2fs/ext2_fs.h
@@ -543,6 +543,7 @@ struct ext2_inode *EXT2_INODE(struct ext2_inode_large *large_inode)
#define EXT2_VALID_FS 0x0001 /* Unmounted cleanly */
#define EXT2_ERROR_FS 0x0002 /* Errors detected */
#define EXT3_ORPHAN_FS 0x0004 /* Orphans being recovered */
+#define EXT4_FC_REPLAY 0x0020 /* Ext4 fast commit replay ongoing */
/*
* Misc. filesystem flags
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index afa9c5e4..bde369ef 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -2141,4 +2141,7 @@ static inline unsigned int ext2_dir_htree_level(ext2_filsys fs)
}
#endif
+/* Commonly used helpers */
+#define max(a, b) ((a) > (b) ? (a) : (b))
+
#endif /* _EXT2FS_EXT2FS_H */
diff --git a/lib/ext2fs/fast_commit.h b/lib/ext2fs/fast_commit.h
new file mode 100644
index 00000000..b77f70f5
--- /dev/null
+++ b/lib/ext2fs/fast_commit.h
@@ -0,0 +1,201 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __FAST_COMMIT_H__
+#define __FAST_COMMIT_H__
+
+/*
+ * Note this file is present in e2fsprogs/lib/ext2fs/fast_commit.h and
+ * linux/fs/ext4/fast_commit.h. These file should always be byte identical.
+ */
+
+/* Fast commit tags */
+#define EXT4_FC_TAG_ADD_RANGE 0x0001
+#define EXT4_FC_TAG_DEL_RANGE 0x0002
+#define EXT4_FC_TAG_CREAT 0x0003
+#define EXT4_FC_TAG_LINK 0x0004
+#define EXT4_FC_TAG_UNLINK 0x0005
+#define EXT4_FC_TAG_INODE 0x0006
+#define EXT4_FC_TAG_PAD 0x0007
+#define EXT4_FC_TAG_TAIL 0x0008
+#define EXT4_FC_TAG_HEAD 0x0009
+
+#define EXT4_FC_SUPPORTED_FEATURES 0x0
+
+/* On disk fast commit tlv value structures */
+
+/* Fast commit on disk tag length structure */
+struct ext4_fc_tl {
+ __le16 fc_tag;
+ __le16 fc_len;
+};
+
+/* Value structure for tag EXT4_FC_TAG_HEAD. */
+struct ext4_fc_head {
+ __le32 fc_features;
+ __le32 fc_tid;
+};
+
+/* Value structure for EXT4_FC_TAG_ADD_RANGE. */
+struct ext4_fc_add_range {
+ __le32 fc_ino;
+ __u8 fc_ex[12];
+};
+
+/* Value structure for tag EXT4_FC_TAG_DEL_RANGE. */
+struct ext4_fc_del_range {
+ __le32 fc_ino;
+ __le32 fc_lblk;
+ __le32 fc_len;
+};
+
+/*
+ * This is the value structure for tags EXT4_FC_TAG_CREAT, EXT4_FC_TAG_LINK
+ * and EXT4_FC_TAG_UNLINK.
+ */
+struct ext4_fc_dentry_info {
+ __le32 fc_parent_ino;
+ __le32 fc_ino;
+ __u8 fc_dname[0];
+};
+
+/* Value structure for EXT4_FC_TAG_INODE and EXT4_FC_TAG_INODE_PARTIAL. */
+struct ext4_fc_inode {
+ __le32 fc_ino;
+ __u8 fc_raw_inode[0];
+};
+
+/* Value structure for tag EXT4_FC_TAG_TAIL. */
+struct ext4_fc_tail {
+ __le32 fc_tid;
+ __le32 fc_crc;
+};
+
+/*
+ * Fast commit reason codes
+ */
+enum {
+ /*
+ * Commit status codes:
+ */
+ EXT4_FC_REASON_OK = 0,
+ EXT4_FC_REASON_INELIGIBLE,
+ EXT4_FC_REASON_ALREADY_COMMITTED,
+ EXT4_FC_REASON_FC_START_FAILED,
+ EXT4_FC_REASON_FC_FAILED,
+
+ /*
+ * Fast commit ineligiblity reasons:
+ */
+ EXT4_FC_REASON_XATTR = 0,
+ EXT4_FC_REASON_CROSS_RENAME,
+ EXT4_FC_REASON_JOURNAL_FLAG_CHANGE,
+ EXT4_FC_REASON_NOMEM,
+ EXT4_FC_REASON_SWAP_BOOT,
+ EXT4_FC_REASON_RESIZE,
+ EXT4_FC_REASON_RENAME_DIR,
+ EXT4_FC_REASON_FALLOC_RANGE,
+ EXT4_FC_REASON_INODE_JOURNAL_DATA,
+ EXT4_FC_COMMIT_FAILED,
+ EXT4_FC_REASON_MAX
+};
+
+#ifdef __KERNEL__
+/*
+ * In memory list of dentry updates that are performed on the file
+ * system used by fast commit code.
+ */
+struct ext4_fc_dentry_update {
+ int fcd_op; /* Type of update create / unlink / link */
+ int fcd_parent; /* Parent inode number */
+ int fcd_ino; /* Inode number */
+ struct qstr fcd_name; /* Dirent name */
+ unsigned char fcd_iname[DNAME_INLINE_LEN]; /* Dirent name string */
+ struct list_head fcd_list;
+};
+
+struct ext4_fc_stats {
+ unsigned int fc_ineligible_reason_count[EXT4_FC_REASON_MAX];
+ unsigned long fc_num_commits;
+ unsigned long fc_ineligible_commits;
+ unsigned long fc_numblks;
+};
+
+#define EXT4_FC_REPLAY_REALLOC_INCREMENT 4
+
+/*
+ * Physical block regions added to different inodes due to fast commit
+ * recovery. These are set during the SCAN phase. During the replay phase,
+ * our allocator excludes these from its allocation. This ensures that
+ * we don't accidentally allocating a block that is going to be used by
+ * another inode.
+ */
+struct ext4_fc_alloc_region {
+ ext4_lblk_t lblk;
+ ext4_fsblk_t pblk;
+ int ino, len;
+};
+
+/*
+ * Fast commit replay state.
+ */
+struct ext4_fc_replay_state {
+ int fc_replay_num_tags;
+ int fc_replay_expected_off;
+ int fc_current_pass;
+ int fc_cur_tag;
+ int fc_crc;
+ struct ext4_fc_alloc_region *fc_regions;
+ int fc_regions_size, fc_regions_used, fc_regions_valid;
+ int *fc_modified_inodes;
+ int fc_modified_inodes_used, fc_modified_inodes_size;
+};
+
+#define region_last(__region) (((__region)->lblk) + ((__region)->len) - 1)
+#endif
+
+#define fc_for_each_tl(__start, __end, __tl) \
+ for (tl = (struct ext4_fc_tl *)(__start); \
+ (__u8 *)tl < (__u8 *)(__end); \
+ tl = (struct ext4_fc_tl *)((__u8 *)tl + \
+ sizeof(struct ext4_fc_tl) + \
+ + le16_to_cpu(tl->fc_len)))
+
+static inline const char *tag2str(__u16 tag)
+{
+ switch (tag) {
+ case EXT4_FC_TAG_LINK:
+ return "ADD_ENTRY";
+ case EXT4_FC_TAG_UNLINK:
+ return "DEL_ENTRY";
+ case EXT4_FC_TAG_ADD_RANGE:
+ return "ADD_RANGE";
+ case EXT4_FC_TAG_CREAT:
+ return "CREAT_DENTRY";
+ case EXT4_FC_TAG_DEL_RANGE:
+ return "DEL_RANGE";
+ case EXT4_FC_TAG_INODE:
+ return "INODE";
+ case EXT4_FC_TAG_PAD:
+ return "PAD";
+ case EXT4_FC_TAG_TAIL:
+ return "TAIL";
+ case EXT4_FC_TAG_HEAD:
+ return "HEAD";
+ default:
+ return "ERROR";
+ }
+}
+
+/* Get length of a particular tlv */
+static inline int ext4_fc_tag_len(struct ext4_fc_tl *tl)
+{
+ return le16_to_cpu(tl->fc_len);
+}
+
+/* Get a pointer to "value" of a tlv */
+static inline __u8 *ext4_fc_tag_val(struct ext4_fc_tl *tl)
+{
+ return (__u8 *)tl + sizeof(*tl);
+}
+
+#endif /* __FAST_COMMIT_H__ */
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 09/15] e2fsck: add fast commit scan pass
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (7 preceding siblings ...)
2020-11-20 19:15 ` [PATCH 08/15] e2fsck: add fast commit setup code Harshad Shirwadkar
@ 2020-11-20 19:16 ` Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 10/15] e2fsck: add fast commit replay skeleton Harshad Shirwadkar
` (5 subsequent siblings)
14 siblings, 0 replies; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:16 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
Add fast commit scan pass. Scan pass is responsible for following
things:
* Count total number of fast commit tags that need to be replayed
during the replay phase.
* Validate whether the fast commit area is valid for a given
transaction ID.
* Verify the CRC of fast commit area.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
e2fsck/journal.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 107 insertions(+)
diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 2c8e3441..1072dfe8 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -278,6 +278,106 @@ static int process_journal_block(ext2_filsys fs,
return 0;
}
+static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh,
+ int off, tid_t expected_tid)
+{
+ e2fsck_t ctx = j->j_fs_dev->k_ctx;
+ struct e2fsck_fc_replay_state *state;
+ int ret = JBD2_FC_REPLAY_CONTINUE;
+ struct ext4_fc_add_range *ext;
+ struct ext4_fc_tl *tl;
+ struct ext4_fc_tail *tail;
+ __u8 *start, *end;
+ struct ext4_fc_head *head;
+ struct ext3_extent *ex;
+ struct ext2fs_extent ext2fs_ex;
+
+ state = &ctx->fc_replay_state;
+
+ start = (__u8 *)bh->b_data;
+ end = (__u8 *)bh->b_data + j->j_blocksize - 1;
+
+ jbd_debug(1, "Scan phase starting, expected %d", expected_tid);
+ if (state->fc_replay_expected_off == 0) {
+ memset(state, 0, sizeof(*state));
+ /* Check if we can stop early */
+ if (le16_to_cpu(((struct ext4_fc_tl *)start)->fc_tag)
+ != EXT4_FC_TAG_HEAD) {
+ jbd_debug(1, "Ending early!, not a head tag");
+ return 0;
+ }
+ }
+
+ if (off != state->fc_replay_expected_off) {
+ ret = -EFSCORRUPTED;
+ goto out_err;
+ }
+
+ state->fc_replay_expected_off++;
+ fc_for_each_tl(start, end, tl) {
+ jbd_debug(3, "Scan phase, tag:%s, blk %lld\n",
+ tag2str(le16_to_cpu(tl->fc_tag)), bh->b_blocknr);
+ switch (le16_to_cpu(tl->fc_tag)) {
+ case EXT4_FC_TAG_ADD_RANGE:
+ ext = (struct ext4_fc_add_range *)ext4_fc_tag_val(tl);
+ ex = (struct ext3_extent *)&ext->fc_ex;
+ ext2fs_convert_extent(&ext2fs_ex, ex);
+ ret = JBD2_FC_REPLAY_CONTINUE;
+ case EXT4_FC_TAG_DEL_RANGE:
+ case EXT4_FC_TAG_LINK:
+ case EXT4_FC_TAG_UNLINK:
+ case EXT4_FC_TAG_CREAT:
+ case EXT4_FC_TAG_INODE:
+ case EXT4_FC_TAG_PAD:
+ state->fc_cur_tag++;
+ state->fc_crc = jbd2_chksum(j, state->fc_crc, tl,
+ sizeof(*tl) + ext4_fc_tag_len(tl));
+ break;
+ case EXT4_FC_TAG_TAIL:
+ state->fc_cur_tag++;
+ tail = (struct ext4_fc_tail *)ext4_fc_tag_val(tl);
+ state->fc_crc = jbd2_chksum(j, state->fc_crc, tl,
+ sizeof(*tl) +
+ offsetof(struct ext4_fc_tail,
+ fc_crc));
+ jbd_debug(1, "tail tid %d, expected %d\n",
+ le32_to_cpu(tail->fc_tid),
+ expected_tid);
+ if (le32_to_cpu(tail->fc_tid) == expected_tid &&
+ le32_to_cpu(tail->fc_crc) == state->fc_crc) {
+ state->fc_replay_num_tags = state->fc_cur_tag;
+ } else {
+ ret = state->fc_replay_num_tags ?
+ JBD2_FC_REPLAY_STOP : -EFSBADCRC;
+ }
+ state->fc_crc = 0;
+ break;
+ case EXT4_FC_TAG_HEAD:
+ head = (struct ext4_fc_head *)ext4_fc_tag_val(tl);
+ if (le32_to_cpu(head->fc_features) &
+ ~EXT4_FC_SUPPORTED_FEATURES) {
+ ret = -EOPNOTSUPP;
+ break;
+ }
+ if (le32_to_cpu(head->fc_tid) != expected_tid) {
+ ret = -EINVAL;
+ break;
+ }
+ state->fc_cur_tag++;
+ state->fc_crc = jbd2_chksum(j, state->fc_crc, tl,
+ sizeof(*tl) + ext4_fc_tag_len(tl));
+ break;
+ default:
+ ret = state->fc_replay_num_tags ?
+ JBD2_FC_REPLAY_STOP : -ECANCELED;
+ }
+ if (ret < 0 || ret == JBD2_FC_REPLAY_STOP)
+ break;
+ }
+
+out_err:
+ return ret;
+}
/*
* Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE
* to indicate that it is expecting more fast commit blocks. It returns
@@ -286,6 +386,13 @@ static int process_journal_block(ext2_filsys fs,
static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
enum passtype pass, int off, tid_t expected_tid)
{
+ e2fsck_t ctx = journal->j_fs_dev->k_ctx;
+ struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state;
+
+ if (pass == PASS_SCAN) {
+ state->fc_current_pass = PASS_SCAN;
+ return ext4_fc_replay_scan(journal, bh, off, expected_tid);
+ }
return JBD2_FC_REPLAY_STOP;
}
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 10/15] e2fsck: add fast commit replay skeleton
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (8 preceding siblings ...)
2020-11-20 19:16 ` [PATCH 09/15] e2fsck: add fast commit scan pass Harshad Shirwadkar
@ 2020-11-20 19:16 ` Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 11/15] e2fsck: add fc replay for link, unlink, creat tags Harshad Shirwadkar
` (4 subsequent siblings)
14 siblings, 0 replies; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:16 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
This function adds the skeleton for the replay path. Following patches
in the series implement the handling for individual tags.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
e2fsck/journal.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 72 insertions(+)
diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 1072dfe8..d44f777b 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -388,11 +388,83 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
{
e2fsck_t ctx = journal->j_fs_dev->k_ctx;
struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state;
+ int ret = JBD2_FC_REPLAY_CONTINUE;
+ struct ext4_fc_tl *tl;
+ __u8 *start, *end;
if (pass == PASS_SCAN) {
state->fc_current_pass = PASS_SCAN;
return ext4_fc_replay_scan(journal, bh, off, expected_tid);
}
+
+ if (state->fc_replay_num_tags == 0)
+ goto replay_done;
+
+ if (state->fc_current_pass != pass) {
+ /* Starting replay phase */
+ state->fc_current_pass = pass;
+ /* We will reset checksums */
+ ctx->fs->flags |= EXT2_FLAG_IGNORE_CSUM_ERRORS;
+ ret = ext2fs_read_bitmaps(ctx->fs);
+ if (ret) {
+ jbd_debug(1, "Error %d while reading bitmaps\n", ret);
+ return ret;
+ }
+ state->fc_super_state = ctx->fs->super->s_state;
+ /*
+ * Mark the file system to indicate it contains errors. That's
+ * because the updates performed by fast commit replay code are
+ * not atomic and may result in incosistent file system if it
+ * crashes before the replay is complete.
+ */
+ ctx->fs->super->s_state |= EXT2_ERROR_FS;
+ ctx->fs->super->s_state |= EXT4_FC_REPLAY;
+ ext2fs_mark_super_dirty(ctx->fs);
+ ext2fs_flush(ctx->fs);
+ }
+
+ start = (__u8 *)bh->b_data;
+ end = (__u8 *)bh->b_data + journal->j_blocksize - 1;
+
+ fc_for_each_tl(start, end, tl) {
+ if (state->fc_replay_num_tags == 0)
+ goto replay_done;
+ jbd_debug(3, "Replay phase processing %s tag\n",
+ tag2str(le16_to_cpu(tl->fc_tag)));
+ state->fc_replay_num_tags--;
+ switch (le16_to_cpu(tl->fc_tag)) {
+ case EXT4_FC_TAG_CREAT:
+ case EXT4_FC_TAG_LINK:
+ case EXT4_FC_TAG_UNLINK:
+ case EXT4_FC_TAG_ADD_RANGE:
+ case EXT4_FC_TAG_DEL_RANGE:
+ case EXT4_FC_TAG_INODE:
+ case EXT4_FC_TAG_TAIL:
+ case EXT4_FC_TAG_PAD:
+ case EXT4_FC_TAG_HEAD:
+ break;
+ default:
+ ret = -ECANCELED;
+ break;
+ }
+ if (ret < 0)
+ break;
+ ret = JBD2_FC_REPLAY_CONTINUE;
+ }
+ return ret;
+replay_done:
+ jbd_debug(1, "End of fast commit replay\n");
+ if (state->fc_current_pass != pass)
+ return JBD2_FC_REPLAY_STOP;
+
+ ext2fs_calculate_summary_stats(ctx->fs, 0 /* update bg also */);
+ ext2fs_write_block_bitmap(ctx->fs);
+ ext2fs_write_inode_bitmap(ctx->fs);
+ ext2fs_mark_super_dirty(ctx->fs);
+ ext2fs_set_gdt_csum(ctx->fs);
+ ctx->fs->super->s_state = state->fc_super_state;
+ ext2fs_flush(ctx->fs);
+
return JBD2_FC_REPLAY_STOP;
}
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 11/15] e2fsck: add fc replay for link, unlink, creat tags
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (9 preceding siblings ...)
2020-11-20 19:16 ` [PATCH 10/15] e2fsck: add fast commit replay skeleton Harshad Shirwadkar
@ 2020-11-20 19:16 ` Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 12/15] e2fsck: add replay for add_range, del_range, and inode tags Harshad Shirwadkar
` (3 subsequent siblings)
14 siblings, 0 replies; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:16 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
Add fast commit replay for directory entry updates.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
e2fsck/journal.c | 93 +++++++++++++++++++++++++++++++++++++++++++++
lib/ext2fs/ext2fs.h | 4 ++
lib/ext2fs/unlink.c | 6 +--
3 files changed, 100 insertions(+), 3 deletions(-)
diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index d44f777b..9f4c4a70 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -378,6 +378,95 @@ static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh,
out_err:
return ret;
}
+/* Helper struct for dentry replay routines */
+struct dentry_info_args {
+ int parent_ino, dname_len, ino, inode_len;
+ char *dname;
+};
+
+static inline void tl_to_darg(struct dentry_info_args *darg,
+ struct ext4_fc_tl *tl)
+{
+ struct ext4_fc_dentry_info *fcd;
+ int tag = le16_to_cpu(tl->fc_tag);
+
+ fcd = (struct ext4_fc_dentry_info *)ext4_fc_tag_val(tl);
+
+ darg->parent_ino = le32_to_cpu(fcd->fc_parent_ino);
+ darg->ino = le32_to_cpu(fcd->fc_ino);
+ darg->dname = fcd->fc_dname;
+ darg->dname_len = ext4_fc_tag_len(tl) -
+ sizeof(struct ext4_fc_dentry_info);
+ darg->dname = malloc(darg->dname_len + 1);
+ memcpy(darg->dname, fcd->fc_dname, darg->dname_len);
+ darg->dname[darg->dname_len] = 0;
+ jbd_debug(1, "%s: %s, ino %d, parent %d\n",
+ tag == EXT4_FC_TAG_CREAT ? "create" :
+ (tag == EXT4_FC_TAG_LINK ? "link" :
+ (tag == EXT4_FC_TAG_UNLINK ? "unlink" : "error")),
+ darg->dname, darg->ino, darg->parent_ino);
+}
+
+static int ext4_fc_handle_unlink(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+ struct ext2_inode inode;
+ struct dentry_info_args darg;
+ ext2_filsys fs = ctx->fs;
+
+ tl_to_darg(&darg, tl);
+ ext2fs_unlink(ctx->fs, darg.parent_ino, darg.dname, darg.ino, 0);
+ /* It's okay if the above call fails */
+ free(darg.dname);
+}
+
+static int ext4_fc_handle_link_and_create(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+ struct dentry_info_args darg;
+ ext2_filsys fs = ctx->fs;
+ struct ext2_inode_large inode_large;
+ int ret, filetype, mode;
+
+ tl_to_darg(&darg, tl);
+ ret = ext2fs_read_inode(fs, darg.ino,
+ (struct ext2_inode *)&inode_large);
+ if (ret)
+ goto out;
+
+ mode = inode_large.i_mode;
+
+ if (LINUX_S_ISREG(mode))
+ filetype = EXT2_FT_REG_FILE;
+ else if (LINUX_S_ISDIR(mode))
+ filetype = EXT2_FT_DIR;
+ else if (LINUX_S_ISCHR(mode))
+ filetype = EXT2_FT_CHRDEV;
+ else if (LINUX_S_ISBLK(mode))
+ filetype = EXT2_FT_BLKDEV;
+ else if (LINUX_S_ISLNK(mode))
+ return EXT2_FT_SYMLINK;
+ else if (LINUX_S_ISFIFO(mode))
+ filetype = EXT2_FT_FIFO;
+ else if (LINUX_S_ISSOCK(mode))
+ filetype = EXT2_FT_SOCK;
+ else {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /*
+ * Forcefully unlink if the same name is present and ignore the error
+ * if any, since this dirent might not exist
+ */
+ ext2fs_unlink(fs, darg.parent_ino, darg.dname, darg.ino,
+ EXT2FS_UNLINK_FORCE);
+
+ ret = ext2fs_link(fs, darg.parent_ino, darg.dname, darg.ino,
+ filetype);
+out:
+ free(darg.dname);
+ return ret;
+
+}
/*
* Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE
* to indicate that it is expecting more fast commit blocks. It returns
@@ -435,7 +524,11 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
switch (le16_to_cpu(tl->fc_tag)) {
case EXT4_FC_TAG_CREAT:
case EXT4_FC_TAG_LINK:
+ ret = ext4_fc_handle_link_and_create(ctx, tl);
+ break;
case EXT4_FC_TAG_UNLINK:
+ ret = ext4_fc_handle_unlink(ctx, tl);
+ break;
case EXT4_FC_TAG_ADD_RANGE:
case EXT4_FC_TAG_DEL_RANGE:
case EXT4_FC_TAG_INODE:
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index bde369ef..7d9430fd 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1675,6 +1675,10 @@ extern errcode_t ext2fs_get_pathname(ext2_filsys fs, ext2_ino_t dir, ext2_ino_t
char **name);
/* link.c */
+#define EXT2FS_UNLINK_FORCE 0x1 /* Forcefully unlink even if
+ * the inode number doesn't
+ * match the dirent
+ */
errcode_t ext2fs_link(ext2_filsys fs, ext2_ino_t dir, const char *name,
ext2_ino_t ino, int flags);
errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir, const char *name,
diff --git a/lib/ext2fs/unlink.c b/lib/ext2fs/unlink.c
index 8ab27ee2..3ec04cfb 100644
--- a/lib/ext2fs/unlink.c
+++ b/lib/ext2fs/unlink.c
@@ -49,7 +49,7 @@ static int unlink_proc(struct ext2_dir_entry *dirent,
if (strncmp(ls->name, dirent->name, ext2fs_dirent_name_len(dirent)))
return 0;
}
- if (ls->inode) {
+ if (!(ls->flags & EXT2FS_UNLINK_FORCE) && ls->inode) {
if (dirent->inode != ls->inode)
return 0;
} else {
@@ -70,7 +70,7 @@ static int unlink_proc(struct ext2_dir_entry *dirent,
#endif
errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir,
const char *name, ext2_ino_t ino,
- int flags EXT2FS_ATTR((unused)))
+ int flags)
{
errcode_t retval;
struct link_struct ls;
@@ -86,7 +86,7 @@ errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir,
ls.name = name;
ls.namelen = name ? strlen(name) : 0;
ls.inode = ino;
- ls.flags = 0;
+ ls.flags = flags;
ls.done = 0;
ls.prev = 0;
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 12/15] e2fsck: add replay for add_range, del_range, and inode tags
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (10 preceding siblings ...)
2020-11-20 19:16 ` [PATCH 11/15] e2fsck: add fc replay for link, unlink, creat tags Harshad Shirwadkar
@ 2020-11-20 19:16 ` Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 13/15] debugfs: add fast commit support to logdump Harshad Shirwadkar
` (2 subsequent siblings)
14 siblings, 0 replies; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:16 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
Add replay for inode's extent trees and inode itself.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
e2fsck/journal.c | 341 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 341 insertions(+)
diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 9f4c4a70..ccd70186 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -378,6 +378,219 @@ static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh,
out_err:
return ret;
}
+
+#define ex_end(__ex) ((__ex)->e_lblk + (__ex)->e_len - 1)
+#define ex_pend(__ex) ((__ex)->e_pblk + (__ex)->e_len - 1)
+
+static int make_room(struct extent_list *list, int i)
+{
+ int ret;
+
+ if (list->count == list->size) {
+ unsigned int new_size = (list->size + 341) *
+ sizeof(struct ext2fs_extent);
+ ret = ext2fs_resize_mem(0, new_size, &list->extents);
+ if (ret)
+ return ret;
+ list->size += 341;
+ }
+
+ memmove(&list->extents[i + 1], &list->extents[i],
+ sizeof(list->extents[0]) * (list->count - i));
+ list->count++;
+ return 0;
+}
+
+static int ex_compar(const void *arg1, const void *arg2)
+{
+ struct ext2fs_extent *ex1 = (struct ext2fs_extent *)arg1;
+ struct ext2fs_extent *ex2 = (struct ext2fs_extent *)arg2;
+
+ if (ex1->e_lblk < ex2->e_lblk)
+ return -1;
+ if (ex1->e_lblk > ex2->e_lblk)
+ return 1;
+ return ex1->e_len - ex2->e_len;
+}
+
+static int ex_len_compar(const void *arg1, const void *arg2)
+{
+ struct ext2fs_extent *ex1 = (struct ext2fs_extent *)arg1;
+ struct ext2fs_extent *ex2 = (struct ext2fs_extent *)arg2;
+
+ if (ex1->e_len < ex2->e_len)
+ return 1;
+
+ if (ex1->e_lblk > ex2->e_lblk)
+ return -1;
+
+ return 0;
+}
+
+static void ex_sort_and_merge(e2fsck_t ctx, struct extent_list *list)
+{
+ blk64_t ex_end;
+ int i, j;
+
+ if (list->count < 2)
+ return;
+
+ /*
+ * Reverse sort by length, that way we strip off all the 0 length
+ * extents
+ */
+ qsort(list->extents, list->count, sizeof(struct ext2fs_extent),
+ ex_len_compar);
+
+ for (i = 0; i < list->count; i++) {
+ if (list->extents[i].e_len == 0) {
+ list->count = i;
+ break;
+ }
+ }
+
+ /* Now sort by logical offset */
+ qsort(list->extents, list->count, sizeof(list->extents[0]),
+ ex_compar);
+
+ /* Merge adjacent extents if they are logically and physically contiguous */
+ i = 0;
+ while (i < list->count - 1) {
+ if (ex_end(&list->extents[i]) + 1 != list->extents[i + 1].e_lblk ||
+ ex_pend(&list->extents[i]) + 1 != list->extents[i + 1].e_pblk ||
+ (list->extents[i].e_flags & EXT2_EXTENT_FLAGS_UNINIT) !=
+ (list->extents[i + 1].e_flags & EXT2_EXTENT_FLAGS_UNINIT)) {
+ i++;
+ continue;
+ }
+
+ list->extents[i].e_len += list->extents[i + 1].e_len;
+ for (j = i + 1; j < list->count - 1; j++)
+ list->extents[j] = list->extents[j + 1];
+ list->count--;
+ }
+}
+
+/* must free blocks that are released */
+static int ext2fs_modify_extent_list(e2fsck_t ctx, struct extent_list *list,
+ struct ext2fs_extent *ex, int del)
+{
+ int ret;
+ int i, offset;
+ struct ext2fs_extent add_ex = *ex, add_ex2;
+
+ /* First let's create a hole from ex->e_lblk of length ex->e_len */
+ for (i = 0; i < list->count; i++) {
+ if (ex_end(&list->extents[i]) < add_ex.e_lblk)
+ continue;
+
+ /* Case 1: No overlap */
+ if (list->extents[i].e_lblk > ex_end(&add_ex))
+ break;
+ /*
+ * Unmark all the blocks in bb now. All the blocks get marked
+ * before we exit this function.
+ */
+ ext2fs_unmark_block_bitmap_range2(ctx->fs->block_map,
+ list->extents[i].e_pblk, list->extents[i].e_len);
+ /* Case 2: Split */
+ if (list->extents[i].e_lblk < add_ex.e_lblk &&
+ ex_end(&list->extents[i]) > ex_end(&add_ex)) {
+ ret = make_room(list, i + 1);
+ if (ret)
+ return ret;
+ list->extents[i + 1] = list->extents[i];
+ offset = ex_end(&add_ex) + 1 - list->extents[i].e_lblk;
+ list->extents[i + 1].e_lblk += offset;
+ list->extents[i + 1].e_pblk += offset;
+ list->extents[i + 1].e_len -= offset;
+ list->extents[i].e_len =
+ add_ex.e_lblk - list->extents[i].e_lblk;
+ break;
+ }
+
+ /* Case 3: Exact overlap */
+ if (add_ex.e_lblk <= list->extents[i].e_lblk &&
+ ex_end(&list->extents[i]) <= ex_end(&add_ex)) {
+
+ list->extents[i].e_len = 0;
+ continue;
+ }
+
+ /* Case 4: Partial overlap */
+ if (ex_end(&list->extents[i]) > ex_end(&add_ex)) {
+ offset = ex_end(&add_ex) + 1 - list->extents[i].e_lblk;
+ list->extents[i].e_lblk += offset;
+ list->extents[i].e_pblk += offset;
+ list->extents[i].e_len -= offset;
+ break;
+ }
+
+ if (ex_end(&add_ex) >= ex_end(&list->extents[i])) {
+ list->extents[i].e_len =
+ max(add_ex.e_lblk - list->extents[i].e_lblk,
+ 0);
+ }
+ }
+
+ if (add_ex.e_len && !del) {
+ make_room(list, list->count);
+ list->extents[list->count - 1] = add_ex;
+ }
+
+ ex_sort_and_merge(ctx, list);
+
+ /* Mark all occupied blocks allocated */
+ for (i = 0; i < list->count; i++)
+ ext2fs_mark_block_bitmap_range2(ctx->fs->block_map,
+ list->extents[i].e_pblk, list->extents[i].e_len);
+ ext2fs_mark_bb_dirty(ctx->fs);
+
+ return 0;
+}
+
+static int ext2fs_add_extent_to_list(e2fsck_t ctx, struct extent_list *list,
+ struct ext2fs_extent *ex)
+{
+ return ext2fs_modify_extent_list(ctx, list, ex, 0 /* add */);
+}
+
+static int ext2fs_del_extent_to_list(e2fsck_t ctx, struct extent_list *list,
+ struct ext2fs_extent *ex)
+{
+ return ext2fs_modify_extent_list(ctx, list, ex, 1 /* delete */);
+}
+
+static int e2fsck_fc_read_extents(e2fsck_t ctx, int ino)
+{
+ struct extent_list *extent_list = &ctx->fc_replay_state.fc_extent_list;
+
+ if (extent_list->ino == ino)
+ return 0;
+
+ extent_list->ino = ino;
+ return e2fsck_read_extents(ctx, extent_list);
+}
+
+/*
+ * Flush extents in replay state on disk. @ino is the inode that is going
+ * to be processed next. So, we hold back flushing of the extent list
+ * if the next inode that's going to be processed is same as the one with
+ * cached extents in our replay state. That allows us to gather multiple extents
+ * for the inode so that we can flush all of them at once and it also saves us
+ * from continuously growing and shrinking the extent tree.
+ */
+static void e2fsck_fc_flush_extents(e2fsck_t ctx, int ino)
+{
+ struct extent_list *extent_list = &ctx->fc_replay_state.fc_extent_list;
+
+ if (extent_list->ino == ino || extent_list->ino == 0)
+ return;
+ e2fsck_rewrite_extent_tree(ctx, extent_list);
+ ext2fs_free_mem(&extent_list->extents);
+ memset(extent_list, 0, sizeof(*extent_list));
+}
+
/* Helper struct for dentry replay routines */
struct dentry_info_args {
int parent_ino, dname_len, ino, inode_len;
@@ -414,6 +627,7 @@ static int ext4_fc_handle_unlink(e2fsck_t ctx, struct ext4_fc_tl *tl)
ext2_filsys fs = ctx->fs;
tl_to_darg(&darg, tl);
+ e2fsck_fc_flush_extents(ctx, darg.ino);
ext2fs_unlink(ctx->fs, darg.parent_ino, darg.dname, darg.ino, 0);
/* It's okay if the above call fails */
free(darg.dname);
@@ -427,6 +641,8 @@ static int ext4_fc_handle_link_and_create(e2fsck_t ctx, struct ext4_fc_tl *tl)
int ret, filetype, mode;
tl_to_darg(&darg, tl);
+
+ e2fsck_fc_flush_extents(ctx, 0);
ret = ext2fs_read_inode(fs, darg.ino,
(struct ext2_inode *)&inode_large);
if (ret)
@@ -467,6 +683,124 @@ out:
return ret;
}
+
+/* This function fixes the i_blocks field in the replayed indoe */
+static void ext4_fc_replay_fixup_iblocks(struct ext2_inode_large *ondisk_inode,
+ struct ext2_inode_large *fc_inode)
+{
+ if (le32_to_cpu(ondisk_inode->i_flags) & EXT4_EXTENTS_FL) {
+ struct ext3_extent_header *eh;
+
+ eh = (struct ext3_extent_header *)(&ondisk_inode->i_block[0]);
+ if (eh->eh_magic != EXT3_EXT_MAGIC) {
+ memset(eh, 0, sizeof(*eh));
+ eh->eh_magic = EXT3_EXT_MAGIC;
+ eh->eh_max = cpu_to_le16(
+ (sizeof(ondisk_inode->i_block) -
+ sizeof(struct ext3_extent_header)) /
+ sizeof(struct ext3_extent));
+ }
+ } else if (le32_to_cpu(ondisk_inode->i_flags) & EXT4_INLINE_DATA_FL) {
+ memcpy(ondisk_inode->i_block, fc_inode->i_block,
+ sizeof(fc_inode->i_block));
+ }
+}
+
+static int ext4_fc_handle_inode(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+ struct ext4_fc_inode *fc_inode;
+ struct ext2_inode_large *inode = NULL;
+ int ino, inode_len = EXT2_GOOD_OLD_INODE_SIZE, ret;
+ struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state;
+
+ fc_inode = (struct ext4_fc_inode *)ext4_fc_tag_val(tl);
+ ino = le32_to_cpu(fc_inode->fc_ino);
+
+ if (EXT2_INODE_SIZE(ctx->fs->super) > EXT2_GOOD_OLD_INODE_SIZE)
+ inode_len += ext2fs_le16_to_cpu(
+ ((struct ext2_inode_large *)fc_inode->fc_raw_inode)
+ ->i_extra_isize);
+ ret = ext2fs_get_mem(inode_len, &inode);
+ if (ret)
+ return ret;
+ e2fsck_fc_flush_extents(ctx, ino);
+
+ ret = ext2fs_read_inode_full(ctx->fs, ino, (struct ext2_inode *)inode,
+ inode_len);
+ if (ret)
+ goto out;
+ memcpy(inode, fc_inode->fc_raw_inode,
+ offsetof(struct ext2_inode_large, i_block));
+ memcpy(&inode->i_generation,
+ &((struct ext2_inode_large *)(fc_inode->fc_raw_inode))->i_generation,
+ inode_len - offsetof(struct ext2_inode_large, i_generation));
+ ext4_fc_replay_fixup_iblocks(inode,
+ (struct ext2_inode_large *)fc_inode->fc_raw_inode);
+ ext2fs_iblk_set(ctx->fs, EXT2_INODE(inode),
+ ext2fs_count_blocks(ctx->fs, ino, EXT2_INODE(inode)));
+
+ ext2fs_inode_csum_set(ctx->fs, ino, inode);
+
+ ret = ext2fs_write_inode_full(ctx->fs, ino, (struct ext2_inode *)inode,
+ inode_len);
+ if (inode->i_links_count)
+ ext2fs_mark_inode_bitmap2(ctx->fs->inode_map, ino);
+ else
+ ext2fs_unmark_inode_bitmap2(ctx->fs->inode_map, ino);
+ ext2fs_mark_ib_dirty(ctx->fs);
+
+out:
+ ext2fs_free_mem(&inode);
+ return ret;
+}
+
+/*
+ * Handle add extent replay tag.
+ */
+static int ext4_fc_handle_add_extent(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+ struct ext2fs_extent extent;
+ struct ext4_fc_add_range *add_range;
+ struct ext4_fc_del_range *del_range;
+ int ret = 0, ino;
+
+ add_range = (struct ext4_fc_add_range *)ext4_fc_tag_val(tl);
+ ino = le32_to_cpu(add_range->fc_ino);
+ e2fsck_fc_flush_extents(ctx, ino);
+
+ ret = e2fsck_fc_read_extents(ctx, ino);
+ if (ret)
+ return ret;
+ memset(&extent, 0, sizeof(extent));
+ ext2fs_convert_extent(&extent, (struct ext3_extent *)(
+ add_range->fc_ex));
+ return ext2fs_add_extent_to_list(ctx,
+ &ctx->fc_replay_state.fc_extent_list, &extent);
+}
+
+/*
+ * Handle delete logical range replay tag.
+ */
+static int ext4_fc_handle_del_range(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+ struct ext2fs_extent extent;
+ struct ext4_fc_del_range *del_range;
+ int ret, ino;
+
+ del_range = (struct ext4_fc_del_range *)ext4_fc_tag_val(tl);
+ ino = le32_to_cpu(del_range->fc_ino);
+ e2fsck_fc_flush_extents(ctx, ino);
+
+ memset(&extent, 0, sizeof(extent));
+ extent.e_lblk = ext2fs_le32_to_cpu(del_range->fc_lblk);
+ extent.e_len = ext2fs_le16_to_cpu(del_range->fc_len);
+ ret = e2fsck_fc_read_extents(ctx, ino);
+ if (ret)
+ return ret;
+ return ext2fs_del_extent_to_list(ctx,
+ &ctx->fc_replay_state.fc_extent_list, &extent);
+}
+
/*
* Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE
* to indicate that it is expecting more fast commit blocks. It returns
@@ -530,9 +864,16 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
ret = ext4_fc_handle_unlink(ctx, tl);
break;
case EXT4_FC_TAG_ADD_RANGE:
+ ret = ext4_fc_handle_add_extent(ctx, tl);
+ break;
case EXT4_FC_TAG_DEL_RANGE:
+ ret = ext4_fc_handle_del_range(ctx, tl);
+ break;
case EXT4_FC_TAG_INODE:
+ ret = ext4_fc_handle_inode(ctx, tl);
+ break;
case EXT4_FC_TAG_TAIL:
+ e2fsck_fc_flush_extents(ctx, 0);
case EXT4_FC_TAG_PAD:
case EXT4_FC_TAG_HEAD:
break;
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 13/15] debugfs: add fast commit support to logdump
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (11 preceding siblings ...)
2020-11-20 19:16 ` [PATCH 12/15] e2fsck: add replay for add_range, del_range, and inode tags Harshad Shirwadkar
@ 2020-11-20 19:16 ` Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 14/15] tests: add fast commit recovery tests Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 15/15] ext4: fix tests to account for new dumpe2fs output Harshad Shirwadkar
14 siblings, 0 replies; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:16 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
Add fast commit support for debugfs logdump. The debugfs output looks
like this:
debugfs 1.46-WIP (20-Mar-2020)
debugfs: logdump
Journal starts at block 1, transaction 2
Found expected sequence 2, type 1 (descriptor block) at block 1
Found expected sequence 2, type 2 (commit block) at block 10
No magic number at block 11: end of journal.
*** Fast Commit Area ***
tag HEAD, features 0x0, tid 3
tag INODE, inode 14
tag ADD_RANGE, inode 14, lblk 0, pblk 1091, len 1
tag DEL_RANGE, inode 14, lblk 1, len 1
tag CREAT_DENTRY, parent 12, ino 14, name "new"
tag DEL_ENTRY, parent 12, ino 13, name "old"
tag INODE, inode 13
tag ADD_RANGE, inode 13, lblk 0, pblk 1603, len 16
tag ADD_RANGE, inode 13, lblk 16, pblk 1092, len 240
tag CREAT_DENTRY, parent 12, ino 13, name "data"
tag INODE, inode 14
tag INODE, inode 13
tag TAIL, tid 3
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
debugfs/logdump.c | 122 ++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 117 insertions(+), 5 deletions(-)
diff --git a/debugfs/logdump.c b/debugfs/logdump.c
index 16889954..151be3e2 100644
--- a/debugfs/logdump.c
+++ b/debugfs/logdump.c
@@ -33,6 +33,7 @@ extern char *optarg;
#include "debugfs.h"
#include "blkid/blkid.h"
#include "jfs_user.h"
+#include "ext2fs/fast_commit.h"
#include <uuid/uuid.h>
enum journal_location {JOURNAL_IS_INTERNAL, JOURNAL_IS_EXTERNAL};
@@ -65,6 +66,9 @@ static void dump_metadata_block(FILE *, struct journal_source *,
unsigned int, unsigned int, unsigned int,
int, tid_t);
+static void dump_fc_block(FILE *out_file, char *buf, int blocksize,
+ int transaction, int *fc_done, int dump_old);
+
static void do_hexdump (FILE *, char *, int);
#define WRAP(jsb, blocknr) \
@@ -353,6 +357,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
journal_header_t *header;
tid_t transaction;
unsigned int blocknr = 0;
+ int fc_done;
/* First, check to see if there's an ext2 superblock header */
retval = read_journal_block(cmdname, source, 0, buf, 2048);
@@ -410,7 +415,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
if (!blocknr) {
/* Empty journal, nothing to do. */
if (!dump_old)
- return;
+ goto fc;
else
blocknr = 1;
}
@@ -420,7 +425,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
((ext2_loff_t) blocknr) * blocksize,
buf, blocksize);
if (retval)
- return;
+ break;
header = (journal_header_t *) buf;
@@ -431,7 +436,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
if (magic != JBD2_MAGIC_NUMBER) {
fprintf (out_file, "No magic number at block %u: "
"end of journal.\n", blocknr);
- return;
+ break;
}
if (sequence != transaction) {
@@ -439,7 +444,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
"block %u: end of journal.\n",
sequence, transaction, blocknr);
if (!dump_old)
- return;
+ break;
}
if (dump_descriptors) {
@@ -473,9 +478,25 @@ static void dump_journal(char *cmdname, FILE *out_file,
default:
fprintf (out_file, "Unexpected block type %u at "
"block %u.\n", blocktype, blocknr);
- return;
+ break;
}
}
+
+fc:
+ blocknr = be32_to_cpu(jsb->s_maxlen) - jbd2_journal_get_num_fc_blks(jsb) + 1;
+ while (blocknr <= be32_to_cpu(jsb->s_maxlen)) {
+ retval = read_journal_block(cmdname, source,
+ ((ext2_loff_t) blocknr) * blocksize,
+ buf, blocksize);
+ if (retval)
+ return;
+
+ dump_fc_block(out_file, buf, blocksize, transaction, &fc_done,
+ dump_old);
+ if (!dump_old && fc_done)
+ break;
+ blocknr++;
+ }
}
static inline size_t journal_super_tag_bytes(journal_superblock_t *jsb)
@@ -496,6 +517,97 @@ static inline size_t journal_super_tag_bytes(journal_superblock_t *jsb)
return sz - sizeof(__u32);
}
+static void dump_fc_block(FILE *out_file, char *buf, int blocksize,
+ int transaction, int *fc_done, int dump_old)
+{
+ struct ext4_fc_tl *tl;
+ struct ext4_fc_head *head;
+ struct ext4_fc_add_range *add_range;
+ struct ext4_fc_del_range *del_range;
+ struct ext4_fc_dentry_info *dentry_info;
+ struct ext4_fc_tail *tail;
+ struct ext3_extent *ex;
+
+ *fc_done = 0;
+ fc_for_each_tl(buf, buf + blocksize, tl) {
+ switch (le16_to_cpu(tl->fc_tag)) {
+ case EXT4_FC_TAG_ADD_RANGE:
+ add_range =
+ (struct ext4_fc_add_range *)ext4_fc_tag_val(tl);
+ ex = (struct ext3_extent *)add_range->fc_ex;
+ fprintf(out_file,
+ "tag %s, inode %d, lblk %d, pblk %ld, len %d\n",
+ tag2str(tl->fc_tag),
+ le32_to_cpu(add_range->fc_ino),
+ le32_to_cpu(ex->ee_block),
+ le32_to_cpu(ex->ee_start) +
+ (((__u64) le16_to_cpu(ex->ee_start_hi)) << 32),
+ le16_to_cpu(ex->ee_len) > EXT_INIT_MAX_LEN ?
+ le16_to_cpu(ex->ee_len) - EXT_INIT_MAX_LEN :
+ le16_to_cpu(ex->ee_len));
+ break;
+ case EXT4_FC_TAG_DEL_RANGE:
+ del_range =
+ (struct ext4_fc_del_range *)ext4_fc_tag_val(tl);
+ fprintf(out_file, "tag %s, inode %d, lblk %d, len %d\n",
+ tag2str(tl->fc_tag),
+ le32_to_cpu(del_range->fc_ino),
+ le32_to_cpu(del_range->fc_lblk),
+ le32_to_cpu(del_range->fc_len));
+ break;
+ case EXT4_FC_TAG_LINK:
+ case EXT4_FC_TAG_UNLINK:
+ case EXT4_FC_TAG_CREAT:
+ dentry_info =
+ (struct ext4_fc_dentry_info *)
+ ext4_fc_tag_val(tl);
+ fprintf(out_file,
+ "tag %s, parent %d, ino %d, name \"%s\"\n",
+ tag2str(tl->fc_tag),
+ le32_to_cpu(dentry_info->fc_parent_ino),
+ le32_to_cpu(dentry_info->fc_ino),
+ dentry_info->fc_dname);
+ break;
+ case EXT4_FC_TAG_INODE:
+ fprintf(out_file, "tag %s, inode %d\n",
+ tag2str(tl->fc_tag),
+ le32_to_cpu(((struct ext4_fc_inode *)
+ ext4_fc_tag_val(tl))->fc_ino));
+ break;
+ case EXT4_FC_TAG_PAD:
+ fprintf(out_file, "tag %s\n", tag2str(tl->fc_tag));
+ break;
+ case EXT4_FC_TAG_TAIL:
+ tail = (struct ext4_fc_tail *)ext4_fc_tag_val(tl);
+ fprintf(out_file, "tag %s, tid %d\n",
+ tag2str(tl->fc_tag),
+ le32_to_cpu(tail->fc_tid));
+ if (!dump_old &&
+ le32_to_cpu(tail->fc_tid) < transaction) {
+ *fc_done = 1;
+ return;
+ }
+ break;
+ case EXT4_FC_TAG_HEAD:
+ fprintf(out_file, "\n*** Fast Commit Area ***\n");
+ head = (struct ext4_fc_head *)ext4_fc_tag_val(tl);
+ fprintf(out_file, "tag %s, features 0x%x, tid %d\n",
+ tag2str(tl->fc_tag),
+ le32_to_cpu(head->fc_features),
+ le32_to_cpu(head->fc_tid));
+ if (!dump_old &&
+ le32_to_cpu(head->fc_tid) < transaction) {
+ *fc_done = 1;
+ return;
+ }
+ break;
+ default:
+ *fc_done = 1;
+ break;
+ }
+ }
+}
+
static void dump_descriptor_block(FILE *out_file,
struct journal_source *source,
char *buf,
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 14/15] tests: add fast commit recovery tests
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (12 preceding siblings ...)
2020-11-20 19:16 ` [PATCH 13/15] debugfs: add fast commit support to logdump Harshad Shirwadkar
@ 2020-11-20 19:16 ` Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 15/15] ext4: fix tests to account for new dumpe2fs output Harshad Shirwadkar
14 siblings, 0 replies; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:16 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
Add j_recover_fast_commit test that ensure that e2fsck is able to
recover a disk from fast commit log.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
tests/j_recover_fast_commit/commands | 4 ++++
tests/j_recover_fast_commit/expect | 23 +++++++++++++++++++++++
tests/j_recover_fast_commit/image.gz | Bin 0 -> 3595 bytes
tests/j_recover_fast_commit/script | 26 ++++++++++++++++++++++++++
4 files changed, 53 insertions(+)
create mode 100644 tests/j_recover_fast_commit/commands
create mode 100644 tests/j_recover_fast_commit/expect
create mode 100644 tests/j_recover_fast_commit/image.gz
create mode 100755 tests/j_recover_fast_commit/script
diff --git a/tests/j_recover_fast_commit/commands b/tests/j_recover_fast_commit/commands
new file mode 100644
index 00000000..74e20e4e
--- /dev/null
+++ b/tests/j_recover_fast_commit/commands
@@ -0,0 +1,4 @@
+ls
+ls a/
+ex a/new
+ex a/data
diff --git a/tests/j_recover_fast_commit/expect b/tests/j_recover_fast_commit/expect
new file mode 100644
index 00000000..2fc1e53f
--- /dev/null
+++ b/tests/j_recover_fast_commit/expect
@@ -0,0 +1,23 @@
+test_filesys: recovering journal
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 14/256 files (14.3% non-contiguous), 1365/2048 blocks
+Exit status is 0
+debugfs: ls
+ 2 (12) . 2 (12) .. 11 (20) lost+found 12 (968) a
+debugfs: ls a/
+ 12 (12) . 2 (12) .. 13 (12) data 14 (976) new
+debugfs: ex a/new
+Level Entries Logical Physical Length Flags
+ 0/ 0 1/ 1 0 - 0 1107 - 1107 1
+debugfs: ex a/data
+Level Entries Logical Physical Length Flags
+ 0/ 1 1/ 1 0 - 255 1618 256
+ 1/ 1 1/ 5 0 - 15 1619 - 1634 16
+ 1/ 1 2/ 5 16 - 31 1601 - 1616 16
+ 1/ 1 3/ 5 32 - 63 1985 - 2016 32
+ 1/ 1 4/ 5 64 - 127 1537 - 1600 64
+ 1/ 1 5/ 5 128 - 255 1793 - 1920 128
diff --git a/tests/j_recover_fast_commit/image.gz b/tests/j_recover_fast_commit/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..b7357afc46bec8a0154d5746fa2fce86c2341efc
GIT binary patch
literal 3595
zcmeH}eNfT~8ppY9Z*$KxYh$%EC0B221!CQ{-F$;y_Nv9SE-STSSvB{>*GvmjkZCXT
z?f_j=!B-kB-%S%0H3Y6v!7^5Yd<&I|MMB;Z6jb2%yKeTccIN)Sx$&8K{`k&!zW;ol
z`OZ8rs*@8`&c;JF5K?3RngG$yQc})Dc|qP}Dsm5ehAgEclfGm;JbtkgO84-68YdB4
zsDm<_xB0<`_Mfe5Ehc<(^ZmG2x5R<(yD|IUh}iq9FWN?(5XZU@N2~<_h=?GsN5^Ot
zwZ0W$-7K4dx71_oJoDV-Z!~|)54Ez0bAf6;KQnhu>)7(1gnesUs!s~_(bLOL?&h|z
z2j;(5SZe|~f%H?`Jhc>LGeZuYOm4*@|JfNVe2#S*tlw;)X@adgK;_GgnB@`IN$()O
zeJdVTF+(3Gx@-!k#510$y2lYkS(vN9#GrQLCa;-mV?qZU{;kwK$Rv{L6Jf{#Z20Qb
z@<IdCR2elC?GcaQM>yz}Ms3R|)&tr)^yDbq#d`(A0X*cavzL<Ih>NvxIgvD1VK$rO
z1ND7TxV7GM&y8Z#1w(&&^>=;7g{6-m?@l{8KJ!Tb_9_8)`n-#{^gDTjuU|duC!D&$
zw-g&1I=zp?$hSx>UPRYfcB-iNgN=ul&jpT^l}H&s&`y6Z9O@)lFlKin4*F>!XjY}z
zH%zu?*}+&9T@cR1r$2t?naFsn7j8!KH0(v!d&4UQ6AAXi+Ao0GE89WI0~#d}C&YB`
zQNqqK-`b;Z?K3p61O`q$?5kAp0>k|^;cYVc*)iv>>G+ztPx_vc2X@W83hHPAj!b1r
zY6XV1#4Z~|Tv`Oi4Jc0kf>D_H1)flpYK!H<%0qTJsVSQBvck24`sm8=TEiXV>bV)u
zl-(02z{HCOM=U<$%Q$!e#;Zh2T~=hF><M#4mMpSnp{MtWj`3a(FmBaZh3M6Zgi3PW
z7oSt-lI@3EvAcAdyO^4COpLdS%D&a^wF?Lh^2l3Lj_Td#_T$hdf%q$kaMo^{=YS<j
zpX%d+p4Tq1vWoaB>WDWopLklvSs1fCy}Odu<<E1IvI!iDX>l`9%z?r4c5`xf7S%aU
z&E+*8v}xS`<mn%a<)r5>Qt75B{U2YB@TN`IA6TdzSLAF|!jZ$}ahX?GtJAfpeUB+=
zijU8Z&<GZLe6<#QIL-jPlCvCwpZn~m`m1$QXbCOj!zXo1AXuZmiAPRo)q<`L(hnq{
zieo$JJ$<Jv%&bTOVlW^tuTpP}2@?OIK~qHs26H~<-52!hiR8w9Jva~=0g;Vicx70J
z4fElwtc@}#58*Fs1|0nM50oel-tjty1RzdigN^Zbf=@7qqPO^xgBxY9fzK?)NeUd8
z=K`qiTd&px<Cwb{?p?@Fk1F}fxbmz6c%pZJnctU=y#o~r*YII2swB1ZuvtnL8MY-W
zHzn^koHn$58IpGC$6uS%`rn%qHQSg^3U1&P>02?n0K&<(kM_cNQj~1IX?b$W(*0xP
z-_i?z5rOsTi%Tnl`>$~7v^=gl6@7L7z-8Ol{1CZg!>|2lrzyd9v<TeSOBR9Qoi7j>
zfMUoDdQ$?Zb&fbNiXZ`wSA*6!T&fxL@5%ub|6)+N9;(~X%%0oc*_U@JfYDyOz~PA0
zNh(zxZAqoz5<M~({cibwqa-<xeYm}uIk5TF0Jj^u_DSdXZc#LpZNm|T(L2~SIx%0D
zr~FN5U9J;YC-DDBK+OKU`;2ULe9F^Jv^2u=c|z4(-z)mcy|^smP;(_`xPJ5Vqa`Ps
zo8e+@hdA3gnIT}e`~{e1%5omGIGXNt?QqL4?5m!4O><ZVNIiRnh58MIv0yW+V5;$E
zC@S1=?Q-)t{aOdt6zz^LH2gH&<$UP1bBtdA<@?cL96g6`B*u&d4`$;WW|28byT_Cy
z-~D#|O;^>$h-3JF6oyry?<XXVURb>+E>j#lpG`UEw?qP#1vezN7g2eO#qvYh3)d0)
zpTpb;=`8TsO9V3HWC%cFD7c@5S<xo-E@_tE<oVK3852`dW}C5sOW&lD|Ge4};ZLCu
z;<zQVm}Zhc5<ifpabBC<2S;Br8Dm00dkSOE?PqDVP4U;>J^Fl0VbXtImzuDtE!cV%
zOQ(}EAFD2kEZ66+ZZ6Nrn7OSF`P#l9ncx;YubSQdZg^d-6IdtkKS!WE7X1D1l(1q5
HWGCd`W!a1I
literal 0
HcmV?d00001
diff --git a/tests/j_recover_fast_commit/script b/tests/j_recover_fast_commit/script
new file mode 100755
index 00000000..6bdb8258
--- /dev/null
+++ b/tests/j_recover_fast_commit/script
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+FSCK_OPT=-fy
+IMAGE=$test_dir/image.gz
+CMDS=$test_dir/commands
+
+gunzip < $IMAGE > $TMPFILE
+
+# Run fsck to fix things?
+EXP=$test_dir/expect
+OUT=$test_name.log
+
+cp $TMPFILE /tmp/debugthis
+$FSCK $FSCK_OPT -E journal_only -N test_filesys $TMPFILE 2>&1 | head -n 1000 | tail -n +2 > $OUT
+echo "Exit status is $?" >> $OUT
+
+$DEBUGFS -f $CMDS $TMPFILE >> $OUT 2>/dev/null
+
+# Figure out what happened
+if cmp -s $EXP $OUT; then
+ echo "$test_name: $test_description: ok"
+ touch $test_name.ok
+else
+ echo "$test_name: $test_description: failed"
+ diff -u $EXP $OUT >> $test_name.failed
+fi
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 15/15] ext4: fix tests to account for new dumpe2fs output
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
` (13 preceding siblings ...)
2020-11-20 19:16 ` [PATCH 14/15] tests: add fast commit recovery tests Harshad Shirwadkar
@ 2020-11-20 19:16 ` Harshad Shirwadkar
14 siblings, 0 replies; 28+ messages in thread
From: Harshad Shirwadkar @ 2020-11-20 19:16 UTC (permalink / raw)
To: linux-ext4; +Cc: tytso, Harshad Shirwadkar
dumpe2fs tool now is capable of reporting number of fast commit
blocks. There were slight changes in the output of dumpe2fs outside of
fast commits. This patch fixes the regression tests to expect the new
output.
Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
tests/d_corrupt_journal_nr_users/expect | 6 ++++--
tests/f_jnl_errno/expect.0 | 6 ++++--
tests/f_opt_extent/expect | 2 +-
tests/i_bitmaps/expect | 8 +++++---
tests/j_ext_dumpe2fs/expect | 6 ++++--
tests/m_bigjournal/expect.1 | 6 ++++--
tests/m_extent_journal/expect.1 | 6 ++++--
tests/m_resize_inode_meta_bg/expect.1 | 6 ++++--
tests/m_rootdir/expect | 6 ++++--
tests/r_32to64bit/expect | 6 +++---
tests/r_32to64bit_meta/expect | 4 ++--
tests/r_32to64bit_move_itable/expect | 8 ++++----
tests/r_64to32bit/expect | 6 +++---
tests/r_64to32bit_meta/expect | 4 ++--
tests/r_move_itable_nostride/expect | 6 ++++--
tests/r_move_itable_realloc/expect | 6 ++++--
tests/t_disable_mcsum/expect | 4 ++--
tests/t_disable_mcsum_noinitbg/expect | 6 +++---
tests/t_disable_mcsum_yesinitbg/expect | 4 ++--
tests/t_enable_mcsum/expect | 6 +++---
tests/t_enable_mcsum_ext3/expect | 10 +++++-----
tests/t_enable_mcsum_initbg/expect | 6 +++---
22 files changed, 74 insertions(+), 54 deletions(-)
diff --git a/tests/d_corrupt_journal_nr_users/expect b/tests/d_corrupt_journal_nr_users/expect
index cdfb49a0..656d35c3 100644
--- a/tests/d_corrupt_journal_nr_users/expect
+++ b/tests/d_corrupt_journal_nr_users/expect
@@ -34,8 +34,10 @@ Default directory hash: half_md4
Journal backup: inode blocks
Checksum type: crc32c
Journal features: (none)
-Journal size: 4096k
-Journal length: 1024
+Total journal size: 4096k
+Total journal blocks: 1024
+Max transaction length: 1024
+Fast commit length: 0
Journal sequence: 0x00000001
Journal start: 0
Journal number of users: 9999
diff --git a/tests/f_jnl_errno/expect.0 b/tests/f_jnl_errno/expect.0
index 2a9426d9..96fb01a7 100644
--- a/tests/f_jnl_errno/expect.0
+++ b/tests/f_jnl_errno/expect.0
@@ -31,8 +31,10 @@ Journal inode: 8
Default directory hash: half_md4
Journal backup: inode blocks
Journal features: (none)
-Journal size: 1024k
-Journal length: 1024
+Total journal size: 1024k
+Total journal blocks: 1024
+Max transaction length: 1024
+Fast commit length: 0
Journal sequence: 0x00000005
Journal start: 0
diff --git a/tests/f_opt_extent/expect b/tests/f_opt_extent/expect
index 6d1f9d11..9d2021e8 100644
--- a/tests/f_opt_extent/expect
+++ b/tests/f_opt_extent/expect
@@ -35,7 +35,7 @@ Change in FS metadata:
Free inodes: 65047
First block: 1
Block size: 1024
-@@ -48,8 +48,8 @@
+@@ -50,8 +50,8 @@
Block bitmap at 262 (+261)
Inode bitmap at 278 (+277)
Inode table at 294-549 (+293)
diff --git a/tests/i_bitmaps/expect b/tests/i_bitmaps/expect
index fb9d8f1f..6199bb7c 100644
--- a/tests/i_bitmaps/expect
+++ b/tests/i_bitmaps/expect
@@ -1,6 +1,8 @@
-46,50d45
+46,52d45
< Journal features: (none)
-< Journal size: 1024k
-< Journal length: 1024
+< Total journal size: 1024k
+< Total journal blocks: 1024
+< Max transaction length: 1024
+< Fast commit length: 0
< Journal sequence: 0x00000001
< Journal start: 0
diff --git a/tests/j_ext_dumpe2fs/expect b/tests/j_ext_dumpe2fs/expect
index db77a36d..2838bbd1 100644
--- a/tests/j_ext_dumpe2fs/expect
+++ b/tests/j_ext_dumpe2fs/expect
@@ -43,8 +43,10 @@ Desired extra isize: 28
Default directory hash: half_md4
Checksum type: crc32c
Journal features: journal_64bit journal_checksum_v3
-Journal size: 2048k
-Journal length: 2048
+Total journal size: 2048k
+Total journal blocks: 2048
+Max transaction length: 2048
+Fast commit length: 0
Journal first block: 3
Journal sequence: 0x00000003
Journal start: 0
diff --git a/tests/m_bigjournal/expect.1 b/tests/m_bigjournal/expect.1
index 80f71d1f..eb0e3bc3 100644
--- a/tests/m_bigjournal/expect.1
+++ b/tests/m_bigjournal/expect.1
@@ -42,8 +42,10 @@ Journal inode: 8
Default directory hash: half_md4
Journal backup: inode blocks
Journal features: (none)
-Journal size: 5000M
-Journal length: 1280000
+Total journal size: 5000M
+Total journal blocks: 1280000
+Max transaction length: 1280000
+Fast commit length: 0
Journal sequence: 0x00000001
Journal start: 0
diff --git a/tests/m_extent_journal/expect.1 b/tests/m_extent_journal/expect.1
index cfc052a8..9a9219e9 100644
--- a/tests/m_extent_journal/expect.1
+++ b/tests/m_extent_journal/expect.1
@@ -48,8 +48,10 @@ Journal inode: 8
Default directory hash: half_md4
Journal backup: inode blocks
Journal features: (none)
-Journal size: 4096k
-Journal length: 4096
+Total journal size: 4096k
+Total journal blocks: 4096
+Max transaction length: 4096
+Fast commit length: 0
Journal sequence: 0x00000001
Journal start: 0
diff --git a/tests/m_resize_inode_meta_bg/expect.1 b/tests/m_resize_inode_meta_bg/expect.1
index 6a7f3993..7feaed9d 100644
--- a/tests/m_resize_inode_meta_bg/expect.1
+++ b/tests/m_resize_inode_meta_bg/expect.1
@@ -51,8 +51,10 @@ Journal inode: 8
Default directory hash: half_md4
Journal backup: inode blocks
Journal features: (none)
-Journal size: 4096k
-Journal length: 1024
+Total journal size: 4096k
+Total journal blocks: 1024
+Max transaction length: 1024
+Fast commit length: 0
Journal sequence: 0x00000001
Journal start: 0
diff --git a/tests/m_rootdir/expect b/tests/m_rootdir/expect
index 113ffc64..dbc79772 100644
--- a/tests/m_rootdir/expect
+++ b/tests/m_rootdir/expect
@@ -36,8 +36,10 @@ Default directory hash: half_md4
Journal backup: inode blocks
Checksum type: crc32c
Journal features: (none)
-Journal size: 1024k
-Journal length: 1024
+Total journal size: 1024k
+Total journal blocks: 1024
+Max transaction length: 1024
+Fast commit length: 0
Journal sequence: 0x00000001
Journal start: 0
diff --git a/tests/r_32to64bit/expect b/tests/r_32to64bit/expect
index c6816b7f..de573b3e 100644
--- a/tests/r_32to64bit/expect
+++ b/tests/r_32to64bit/expect
@@ -47,7 +47,7 @@ Change in FS metadata:
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 1024
-@@ -41,16 +41,16 @@
+@@ -43,16 +43,16 @@
group:block:super:gdt:bbitmap:ibitmap:itable
@@ -70,7 +70,7 @@ Change in FS metadata:
10:81921:-1:-1:270:286:2852
11:90113:-1:-1:271:287:3108
12:98305:-1:-1:272:288:3364
-@@ -66,9 +66,9 @@
+@@ -68,9 +68,9 @@
22:180225:-1:-1:131079:131095:132641
23:188417:-1:-1:131080:131096:132897
24:196609:-1:-1:131081:131097:133153
@@ -82,7 +82,7 @@ Change in FS metadata:
28:229377:-1:-1:131085:131101:134177
29:237569:-1:-1:131086:131102:134433
30:245761:-1:-1:131087:131103:134689
-@@ -90,7 +90,7 @@
+@@ -92,7 +92,7 @@
46:376833:-1:-1:262159:262175:265761
47:385025:-1:-1:262160:262176:266017
48:393217:-1:-1:393217:393233:393249
diff --git a/tests/r_32to64bit_meta/expect b/tests/r_32to64bit_meta/expect
index c4f39266..70babbd9 100644
--- a/tests/r_32to64bit_meta/expect
+++ b/tests/r_32to64bit_meta/expect
@@ -46,7 +46,7 @@ Change in FS metadata:
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 1024
-@@ -55,9 +55,9 @@
+@@ -57,9 +57,9 @@
12:98305:-1:-1:15:31:3107
13:106497:-1:-1:16:32:3363
14:114689:-1:-1:17:33:3619
@@ -59,7 +59,7 @@ Change in FS metadata:
18:147457:-1:-1:131075:131091:131617
19:155649:-1:-1:131076:131092:131873
20:163841:-1:-1:131077:131093:132129
-@@ -87,9 +87,9 @@
+@@ -89,9 +89,9 @@
44:360449:-1:-1:262158:262174:265250
45:368641:-1:-1:262159:262175:265506
46:376833:-1:-1:262160:262176:265762
diff --git a/tests/r_32to64bit_move_itable/expect b/tests/r_32to64bit_move_itable/expect
index 0a3b78e7..a1d2aebc 100644
--- a/tests/r_32to64bit_move_itable/expect
+++ b/tests/r_32to64bit_move_itable/expect
@@ -46,7 +46,7 @@ Change in FS metadata:
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 1024
-@@ -39,16 +39,16 @@
+@@ -41,16 +41,16 @@
group:block:super:gdt:bbitmap:ibitmap:itable
@@ -69,7 +69,7 @@ Change in FS metadata:
10:81921:-1:-1:81921:81922:81923
11:90113:-1:-1:90113:90114:90115
12:98305:-1:-1:98305:98306:98307
-@@ -64,9 +64,9 @@
+@@ -66,9 +66,9 @@
22:180225:-1:-1:180225:180226:180227
23:188417:-1:-1:188417:188418:188419
24:196609:-1:-1:196609:196610:196611
@@ -81,7 +81,7 @@ Change in FS metadata:
28:229377:-1:-1:229377:229378:229379
29:237569:-1:-1:237569:237570:237571
30:245761:-1:-1:245761:245762:245763
-@@ -88,7 +88,7 @@
+@@ -90,7 +90,7 @@
46:376833:-1:-1:376833:376834:376835
47:385025:-1:-1:385025:385026:385027
48:393217:-1:-1:393217:393218:393219
@@ -90,7 +90,7 @@ Change in FS metadata:
50:409601:-1:-1:409601:409602:409603
51:417793:-1:-1:417793:417794:417795
52:425985:-1:-1:425985:425986:425987
-@@ -120,7 +120,7 @@
+@@ -122,7 +122,7 @@
78:638977:-1:-1:638977:638978:638979
79:647169:-1:-1:647169:647170:647171
80:655361:-1:-1:655361:655362:655363
diff --git a/tests/r_64to32bit/expect b/tests/r_64to32bit/expect
index 7dff2a05..eb609fbd 100644
--- a/tests/r_64to32bit/expect
+++ b/tests/r_64to32bit/expect
@@ -46,7 +46,7 @@ Change in FS metadata:
Reserved GDT blocks: 256
Blocks per group: 8192
Fragments per group: 8192
-@@ -42,16 +40,16 @@
+@@ -44,16 +42,16 @@
group:block:super:gdt:bbitmap:ibitmap:itable
@@ -69,7 +69,7 @@ Change in FS metadata:
10:81921:-1:-1:272:288:2854
11:90113:-1:-1:273:289:3110
12:98305:-1:-1:274:290:3366
-@@ -67,9 +65,9 @@
+@@ -69,9 +67,9 @@
22:180225:-1:-1:131079:131095:132641
23:188417:-1:-1:131080:131096:132897
24:196609:-1:-1:131081:131097:133153
@@ -81,7 +81,7 @@ Change in FS metadata:
28:229377:-1:-1:131085:131101:134177
29:237569:-1:-1:131086:131102:134433
30:245761:-1:-1:131087:131103:134689
-@@ -91,7 +89,7 @@
+@@ -93,7 +91,7 @@
46:376833:-1:-1:262159:262175:265761
47:385025:-1:-1:262160:262176:266017
48:393217:-1:-1:393217:393233:393249
diff --git a/tests/r_64to32bit_meta/expect b/tests/r_64to32bit_meta/expect
index b17a8784..70655909 100644
--- a/tests/r_64to32bit_meta/expect
+++ b/tests/r_64to32bit_meta/expect
@@ -46,7 +46,7 @@ Change in FS metadata:
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 1024
-@@ -56,9 +54,9 @@
+@@ -58,9 +56,9 @@
12:98305:-1:-1:15:31:3107
13:106497:-1:-1:16:32:3363
14:114689:-1:-1:17:33:3619
@@ -59,7 +59,7 @@ Change in FS metadata:
18:147457:-1:-1:131076:131092:131618
19:155649:-1:-1:131077:131093:131874
20:163841:-1:-1:131078:131094:132130
-@@ -88,9 +86,9 @@
+@@ -90,9 +88,9 @@
44:360449:-1:-1:262158:262174:265250
45:368641:-1:-1:262159:262175:265506
46:376833:-1:-1:262160:262176:265762
diff --git a/tests/r_move_itable_nostride/expect b/tests/r_move_itable_nostride/expect
index 098cbfc5..74c2cc0a 100644
--- a/tests/r_move_itable_nostride/expect
+++ b/tests/r_move_itable_nostride/expect
@@ -52,8 +52,10 @@ Journal inode: 8
Default directory hash: half_md4
Journal backup: inode blocks
Journal features: (none)
-Journal size: 16M
-Journal length: 16384
+Total journal size: 16M
+Total journal blocks: 16384
+Max transaction length: 16384
+Fast commit length: 0
Journal sequence: 0x00000001
Journal start: 0
diff --git a/tests/r_move_itable_realloc/expect b/tests/r_move_itable_realloc/expect
index f9a7f515..67f2fe4a 100644
--- a/tests/r_move_itable_realloc/expect
+++ b/tests/r_move_itable_realloc/expect
@@ -51,8 +51,10 @@ Journal inode: 8
Default directory hash: half_md4
Journal backup: inode blocks
Journal features: (none)
-Journal size: 16M
-Journal length: 16384
+Total journal size: 16M
+Total journal blocks: 16384
+Max transaction length: 16384
+Fast commit length: 0
Journal sequence: 0x00000001
Journal start: 0
diff --git a/tests/t_disable_mcsum/expect b/tests/t_disable_mcsum/expect
index 3341ad71..bfa21b89 100644
--- a/tests/t_disable_mcsum/expect
+++ b/tests/t_disable_mcsum/expect
@@ -34,8 +34,8 @@ Change in FS metadata:
Journal backup: inode blocks
-Checksum type: crc32c
Journal features: (none)
- Journal size: 16M
- Journal length: 16384
+ Total journal size: 16M
+ Total journal blocks: 16384
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
diff --git a/tests/t_disable_mcsum_noinitbg/expect b/tests/t_disable_mcsum_noinitbg/expect
index 62eca4e9..fe61fcbc 100644
--- a/tests/t_disable_mcsum_noinitbg/expect
+++ b/tests/t_disable_mcsum_noinitbg/expect
@@ -34,9 +34,9 @@ Change in FS metadata:
Journal backup: inode blocks
-Checksum type: crc32c
Journal features: (none)
- Journal size: 16M
- Journal length: 16384
-@@ -48,18 +47,18 @@
+ Total journal size: 16M
+ Total journal blocks: 16384
+@@ -50,18 +49,18 @@
Block bitmap at 262 (+261)
Inode bitmap at 278 (+277)
Inode table at 294-549 (+293)
diff --git a/tests/t_disable_mcsum_yesinitbg/expect b/tests/t_disable_mcsum_yesinitbg/expect
index 7e3485fe..b9062489 100644
--- a/tests/t_disable_mcsum_yesinitbg/expect
+++ b/tests/t_disable_mcsum_yesinitbg/expect
@@ -34,8 +34,8 @@ Change in FS metadata:
Journal backup: inode blocks
-Checksum type: crc32c
Journal features: (none)
- Journal size: 16M
- Journal length: 16384
+ Total journal size: 16M
+ Total journal blocks: 16384
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
diff --git a/tests/t_enable_mcsum/expect b/tests/t_enable_mcsum/expect
index cb0aef62..fcb0ed16 100644
--- a/tests/t_enable_mcsum/expect
+++ b/tests/t_enable_mcsum/expect
@@ -58,9 +58,9 @@ Change in FS metadata:
Journal backup: inode blocks
+Checksum type: crc32c
Journal features: (none)
- Journal size: 16M
- Journal length: 16384
-@@ -47,8 +48,8 @@
+ Total journal size: 16M
+ Total journal blocks: 16384
+@@ -49,8 +50,8 @@
Block bitmap at 262 (+261)
Inode bitmap at 278 (+277)
Inode table at 294-549 (+293)
diff --git a/tests/t_enable_mcsum_ext3/expect b/tests/t_enable_mcsum_ext3/expect
index 11c5a26d..549e60e9 100644
--- a/tests/t_enable_mcsum_ext3/expect
+++ b/tests/t_enable_mcsum_ext3/expect
@@ -37,9 +37,9 @@ Change in FS metadata:
Journal backup: inode blocks
+Checksum type: crc32c
Journal features: (none)
- Journal size: 16M
- Journal length: 16384
-@@ -37,7 +38,7 @@
+ Total journal size: 16M
+ Total journal blocks: 16384
+@@ -39,7 +40,7 @@
Journal start: 0
@@ -48,7 +48,7 @@ Change in FS metadata:
Primary superblock at 1, Group descriptors at 2-3
Reserved GDT blocks at 4-259
Block bitmap at 260 (+259)
-@@ -46,7 +47,7 @@
+@@ -48,7 +49,7 @@
0 free blocks, 1013 free inodes, 2 directories
Free blocks:
Free inodes: 12-1024
@@ -57,7 +57,7 @@ Change in FS metadata:
Backup superblock at 8193, Group descriptors at 8194-8195
Reserved GDT blocks at 8196-8451
Block bitmap at 8452 (+259)
-@@ -55,6 +56,6 @@
+@@ -57,6 +58,6 @@
0 free blocks, 1024 free inodes, 0 directories
Free blocks:
Free inodes: 1025-2048
diff --git a/tests/t_enable_mcsum_initbg/expect b/tests/t_enable_mcsum_initbg/expect
index a37648bf..987141f1 100644
--- a/tests/t_enable_mcsum_initbg/expect
+++ b/tests/t_enable_mcsum_initbg/expect
@@ -58,9 +58,9 @@ Change in FS metadata:
Journal backup: inode blocks
+Checksum type: crc32c
Journal features: (none)
- Journal size: 16M
- Journal length: 16384
-@@ -41,24 +42,24 @@
+ Total journal size: 16M
+ Total journal blocks: 16384
+@@ -43,24 +44,24 @@
Journal start: 0
--
2.29.2.454.gaff20da3a2-goog
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: [PATCH 01/15] ext2fs: move calculate_summary_stats to ext2fs lib
2020-11-20 19:15 ` [PATCH 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
@ 2020-12-02 16:47 ` Theodore Y. Ts'o
0 siblings, 0 replies; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-02 16:47 UTC (permalink / raw)
To: Harshad Shirwadkar; +Cc: linux-ext4
On Fri, Nov 20, 2020 at 11:15:52AM -0800, Harshad Shirwadkar wrote:
> The function calculate_summary_stats sets the global metadata of the
> file system. Tune2fs had this function defined statically in
> tune2fs.c. Fast commit replay needs this function to set global
> metadata at the end of the replay phase. So, move this function to
> libext2fs.
>
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Looks good,
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
- Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 02/15] ext2fs, e2fsck: add kernel endian-ness conversion macros
2020-11-20 19:15 ` [PATCH 02/15] ext2fs, e2fsck: add kernel endian-ness conversion macros Harshad Shirwadkar
@ 2020-12-02 16:50 ` Theodore Y. Ts'o
2020-12-03 18:10 ` harshad shirwadkar
0 siblings, 1 reply; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-02 16:50 UTC (permalink / raw)
To: Harshad Shirwadkar; +Cc: linux-ext4
On Fri, Nov 20, 2020 at 11:15:53AM -0800, Harshad Shirwadkar wrote:
> diff --git a/lib/ext2fs/bitops.h b/lib/ext2fs/bitops.h
> index 505b3c9c..3c7b2496 100644
> --- a/lib/ext2fs/bitops.h
> +++ b/lib/ext2fs/bitops.h
> @@ -247,6 +247,14 @@ extern errcode_t ext2fs_find_first_set_generic_bmap(ext2fs_generic_bitmap bitmap
> #endif /* __STDC_VERSION__ >= 199901L */
> #endif /* INCLUDE_INLINE_FUNCS */
>
> +/* Macros for kernel compatibility */
> +#define be32_to_cpu(x) ext2fs_be32_to_cpu(x)
> +#define le32_to_cpu(x) ext2fs_le32_to_cpu(x)
> +#define le16_to_cpu(x) ext2fs_le16_to_cpu(x)
> +
> +#define cpu_to_be32(x) ext2fs_cpu_to_be32(x)
> +#define cpu_to_be16(x) ext2fs_cpu_to_be16(x)
> +#define cpu_to_le16(x) ext2fs_cpu_to_le16(x)
> /*
> * Fast bit set/clear functions that doesn't need to return the
> * previous bit value.
Kernel compatibility #define's should be in e2fsck/jfs_user.h.
The problem with putting them in lib/ext2fs/bitops.h is that this a
published header file which will be pulled in by external userspace
applications which #include <ext2fs/ext2fs.h>. And we don't want to
have namespace leakage which might interfere with other header files
or the application's definition of these cpp macros.
- Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck
2020-11-20 19:15 ` [PATCH 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Harshad Shirwadkar
@ 2020-12-02 16:54 ` Theodore Y. Ts'o
0 siblings, 0 replies; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-02 16:54 UTC (permalink / raw)
To: Harshad Shirwadkar; +Cc: linux-ext4
On Fri, Nov 20, 2020 at 11:15:54AM -0800, Harshad Shirwadkar wrote:
> This patch makes recovery.c identical with fast commit kernel changes.
>
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Looks good,
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 04/15] mke2fs, dumpe2fs: make fast commit blocks configurable
2020-11-20 19:15 ` [PATCH 04/15] mke2fs, dumpe2fs: make fast commit blocks configurable Harshad Shirwadkar
@ 2020-12-02 18:29 ` Theodore Y. Ts'o
0 siblings, 0 replies; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-02 18:29 UTC (permalink / raw)
To: Harshad Shirwadkar; +Cc: linux-ext4
On Fri, Nov 20, 2020 at 11:15:55AM -0800, Harshad Shirwadkar wrote:
> diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
> index a8a6e091..01132245 100644
> --- a/lib/ext2fs/ext2fs.h
> +++ b/lib/ext2fs/ext2fs.h
> @@ -1625,15 +1631,18 @@ extern errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
> extern errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
> blk64_t *ret_blk, int *ret_count);
> extern errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
> - __u32 num_blocks, int flags,
> - char **ret_jsb);
> + __u32 num_blocks, __u32 num_fc_blks,
> + int flags, char **ret_jsb);
> +extern errcode_t ext2fs_split_journal_size(ext2_filsys fs, blk_t *journal_blks,
> + blk_t *fc_blks, blk_t total_blks);
> extern errcode_t ext2fs_add_journal_device(ext2_filsys fs,
> ext2_filsys journal_dev);
> extern errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks,
> - int flags);
> + blk_t num_fc_blocks, int flags);
> extern errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
> - blk64_t goal, int flags);
> -extern int ext2fs_default_journal_size(__u64 num_blocks);
> + blk_t num_fc_blocks,
> + blk64_t goal, int flags);
> +extern errcode_t ext2fs_default_journal_size(int *journal_size, int *fc_size, ext2_filsys fs);
> extern int ext2fs_journal_sb_start(int blocksize);
>
We must never change the type or function signature of anything which
is exported via a shared library. Otherwise, if someone grabs a new
mke2fs binary, and somehow fails to run against an older version of
libext2fs.so, Much Hilarity will ensue.
It's also possible that there may be some other userspace application
which is shipped separately from e2fsprogs --- maybe in some company's
userspace program which has never been published and might be living
in some Perforce depot for all we know --- that might be using a
published interface. So even without shared libraries, we don't want
to break those applications when that company imports the newer
version of e2fsprogs into their code base.
That means that we can define new functions (and they should be
prefixed with ext2fs_ to avoid namespace polution), but we must not
modify existing functions. We can either do something like, say,
ext2fs_default_journal_size2() or perhaps better in this case, we
could define a new function ext2fs_default_journal_params(), and then
define ext2fs_default_journal_size() in terms of the new function.
> @@ -2122,6 +2131,8 @@ static inline unsigned int ext2_dir_htree_level(ext2_filsys fs)
> return EXT4_HTREE_LEVEL_COMPAT;
> }
>
> +#define max(a, b) ((a) > (b) ? (a) : (b))
> +
> #ifdef __cplusplus
> }
> #endif
Please don't define max() in ext2fs.h, since that's a public header
file, and we don't want cause problems for applciations which may have
their own max() definition.
There is the ext2fsP.h header file which is private to the ext2fs
library, or you could define a new function or cpp macros in
libsupport, if it's really necessary for multiple e2fsprogs
applications. Or maybe max() is so simple that we can just have it
defined in those .c files where it's needed....
> diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c
> index f47f71e6..74d0c7fc 100644
> --- a/lib/ext2fs/mkjournal.c
> +++ b/lib/ext2fs/mkjournal.c
> +errcode_t ext2fs_split_journal_size(ext2_filsys fs, blk_t *journal_blks,
> + blk_t *fc_blks, blk_t total_blks)
> +{
> + if (total_blks < JBD2_MIN_JOURNAL_BLOCKS)
> + return EXT2_ET_JOURNAL_TOO_SMALL;
> +
> + if (!ext2fs_has_feature_fast_commit(fs->super)) {
> + *journal_blks = total_blks;
> + *fc_blks = 0;
> + return 0;
> + }
> + *journal_blks = ext2fs_blocks_count(fs->super) *
> + EXT2_JOURNAL_TO_FC_BLKS_RATIO /
> + (EXT2_JOURNAL_TO_FC_BLKS_RATIO + 1);
> + *journal_blks = max(JBD2_MIN_JOURNAL_BLOCKS, *journal_blks);
> + *fc_blks = total_blks - *journal_blks;
> + return 0;
> +}
Maybe we should just have a ext2fs_default_journal_params structure,
and do this as part of a new "ext2fs_get_journal_params"? If the
number of journal blocks or fast commit blocks is zero, then we can
have the function fill in an appropriate default value, perhaps?
Cheers,
- Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 05/15] mke2fs, tune2fs: update man page with fast commit info
2020-11-20 19:15 ` [PATCH 05/15] mke2fs, tune2fs: update man page with fast commit info Harshad Shirwadkar
@ 2020-12-02 18:33 ` Theodore Y. Ts'o
2020-12-10 5:24 ` harshad shirwadkar
0 siblings, 1 reply; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-02 18:33 UTC (permalink / raw)
To: Harshad Shirwadkar; +Cc: linux-ext4
On Fri, Nov 20, 2020 at 11:15:56AM -0800, Harshad Shirwadkar wrote:
> This patch adds information about fast commit feature in mke2fs and
> tune2fs man pages.
>
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
So this is a bit more of a personal preference thing, but I like to
keep the libext2fs changes from the changes to the userspace
applications, and then combine the changes to the userspace progams
(mke2fs and tune2fs in this case) with the man page updates.
So you might want to consider moving the mke2fs and tune2fs changes
from the previous patch and then combining them with this patch, and
adjusting the commit message appropriately?
> diff --git a/misc/mke2fs.8.in b/misc/mke2fs.8.in
> index e6bfc6d6..2833b408 100644
> --- a/misc/mke2fs.8.in
> +++ b/misc/mke2fs.8.in
> @@ -521,6 +521,27 @@ The size of the journal must be at least 1024 filesystem blocks
> and may be no more than 10,240,000 filesystem blocks or half the total
> file system size (whichever is smaller)
> .TP
> +.BI fast_commit_size= fast-commit-size
> +Create an additional fast commit journal area of size
> +.I fast-commit-size
> +kilobytes.
> +This option is only valid if
> +.B fast_commit
> +feature is enabled
> +on the file system. If this option is not specified and if
> +.B fast_commit
> +feature is turned on, fast commit area size defaults to
> +.I journal-size
> +/ 64 megabytes. The total size of the journal with
> +.B fast_commit
> +feature set is
> +.I journal-size
> ++ (
> +.I fast-commit-size
> +* 1024) megabytes. The total journal size may be no more than
> +10,240,000 filesystem blocks or half the total file system size
> +(whichever is smaller).
> +.TP
So as I recall, aren't we currently calculating the fast commit size
as a fraction of the total journal size? I'm not sure this is in sync
with was in the last patch.
- Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 06/15] ext2fs: add new APIs needed for fast commits
2020-11-20 19:15 ` [PATCH 06/15] ext2fs: add new APIs needed for fast commits Harshad Shirwadkar
@ 2020-12-02 18:44 ` Theodore Y. Ts'o
2020-12-10 1:45 ` harshad shirwadkar
0 siblings, 1 reply; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-02 18:44 UTC (permalink / raw)
To: Harshad Shirwadkar; +Cc: linux-ext4
On Fri, Nov 20, 2020 at 11:15:57AM -0800, Harshad Shirwadkar wrote:
> This patch adds the following new APIs:
>
> Count the total number of blocks occupied by inode including
> intermediate extent tree nodes.
> extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
> struct ext2_inode *inode);
>
> Convert ext3_extent to ext2fs_extent.
> extern void ext2fs_convert_extent(struct ext2fs_extent *to,
> struct ext3_extent *from);
So one of the reasons why I've intentionally never exposed "struct
ext3_extent" in the libext2fs interface is because that's an on-disk
structure which I keep hoping we might change someday --- for example,
to allow for 64-bit logical block numbers so we can create ext4 files
greater than 2**32 blocks. It might be that some other future
enhancement, such as say, reflinks (depending on how we implement
them), or reverse pointers, might also require making changes to the
on-disk format.
The kernel code has the on-disk format and the various logical
manipulations of the extent tree hopelessly entangled with each other,
which means changing the kernel code to support more than one on-disk
extent structure is going to be **hard**. But in the userspace code,
all of the knowledge about the on-disk structure is abstracted away
inside lib/ext2fs/extent.c.
It may very well be that for fast commit, we're going to need to crack
open that abstraction barrier a bit. But let's make sure the function
name makes it clear that what we are doing here is converting between
a particular on-disk encoding and the ext2fs abtract extent type.
"ext2fs_convert_extent" doesn't exactly make this clear.
It might also be that what should do is include a pointer to the fs
and inode structures, and call this something like
"ext2fs_{decode,encode}_extent()", and pass in the on-disk format via
a void *. We might also want to have some kind of
ext2fs_validate_extent() function which takes a void * and validates
the on-disk structure to make sure it's sane.
What do you think?
- Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 07/15] e2fsck: add function to rewrite extent tree
2020-11-20 19:15 ` [PATCH 07/15] e2fsck: add function to rewrite extent tree Harshad Shirwadkar
@ 2020-12-02 18:46 ` Theodore Y. Ts'o
0 siblings, 0 replies; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-02 18:46 UTC (permalink / raw)
To: Harshad Shirwadkar; +Cc: linux-ext4
On Fri, Nov 20, 2020 at 11:15:58AM -0800, Harshad Shirwadkar wrote:
> Fast commit replay needs to rewrite the entire extent tree for inodes
> found in fast commit area. This patch makes e2fsck's rewrite extent
> tree path visible.
>
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Looks good,
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
- Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 08/15] e2fsck: add fast commit setup code
2020-11-20 19:15 ` [PATCH 08/15] e2fsck: add fast commit setup code Harshad Shirwadkar
@ 2020-12-02 18:48 ` Theodore Y. Ts'o
0 siblings, 0 replies; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-02 18:48 UTC (permalink / raw)
To: Harshad Shirwadkar; +Cc: linux-ext4
On Fri, Nov 20, 2020 at 11:15:59AM -0800, Harshad Shirwadkar wrote:
> Add fast_commit.h that contains the necessary helpers needed for fast
> commit replay. Note that this file is also byte by byte identical with
> kernel's fast_commit.h. Also, we introduce the
> "e2fsck_fc_replay_state" structure which is needed for ext4 fast
> commit replay.
>
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Looks good,
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 02/15] ext2fs, e2fsck: add kernel endian-ness conversion macros
2020-12-02 16:50 ` Theodore Y. Ts'o
@ 2020-12-03 18:10 ` harshad shirwadkar
0 siblings, 0 replies; 28+ messages in thread
From: harshad shirwadkar @ 2020-12-03 18:10 UTC (permalink / raw)
To: Theodore Y. Ts'o; +Cc: Ext4 Developers List
Ah makes sense, sorry I missed that entirely in the patch series, I'll
revisit and will be more careful about what goes in libext2fs.
Thanks,
Harshad
On Wed, Dec 2, 2020 at 8:50 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Fri, Nov 20, 2020 at 11:15:53AM -0800, Harshad Shirwadkar wrote:
> > diff --git a/lib/ext2fs/bitops.h b/lib/ext2fs/bitops.h
> > index 505b3c9c..3c7b2496 100644
> > --- a/lib/ext2fs/bitops.h
> > +++ b/lib/ext2fs/bitops.h
> > @@ -247,6 +247,14 @@ extern errcode_t ext2fs_find_first_set_generic_bmap(ext2fs_generic_bitmap bitmap
> > #endif /* __STDC_VERSION__ >= 199901L */
> > #endif /* INCLUDE_INLINE_FUNCS */
> >
> > +/* Macros for kernel compatibility */
> > +#define be32_to_cpu(x) ext2fs_be32_to_cpu(x)
> > +#define le32_to_cpu(x) ext2fs_le32_to_cpu(x)
> > +#define le16_to_cpu(x) ext2fs_le16_to_cpu(x)
> > +
> > +#define cpu_to_be32(x) ext2fs_cpu_to_be32(x)
> > +#define cpu_to_be16(x) ext2fs_cpu_to_be16(x)
> > +#define cpu_to_le16(x) ext2fs_cpu_to_le16(x)
> > /*
> > * Fast bit set/clear functions that doesn't need to return the
> > * previous bit value.
>
> Kernel compatibility #define's should be in e2fsck/jfs_user.h.
>
> The problem with putting them in lib/ext2fs/bitops.h is that this a
> published header file which will be pulled in by external userspace
> applications which #include <ext2fs/ext2fs.h>. And we don't want to
> have namespace leakage which might interfere with other header files
> or the application's definition of these cpp macros.
>
> - Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 06/15] ext2fs: add new APIs needed for fast commits
2020-12-02 18:44 ` Theodore Y. Ts'o
@ 2020-12-10 1:45 ` harshad shirwadkar
2020-12-10 15:48 ` Theodore Y. Ts'o
0 siblings, 1 reply; 28+ messages in thread
From: harshad shirwadkar @ 2020-12-10 1:45 UTC (permalink / raw)
To: Theodore Y. Ts'o; +Cc: Ext4 Developers List
I see that makes sense. In that case, I'll rename the function to
errcode_t ext2fs_decode_extent(struct ext2fs_extent *dst, void *src).
I wonder if it's okay if we make this function return an error in case
the on-disk format is not sane. If we do that, we can add
ext2fs_validate_extent() later. Does that make sense?
On Wed, Dec 2, 2020 at 10:45 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Fri, Nov 20, 2020 at 11:15:57AM -0800, Harshad Shirwadkar wrote:
> > This patch adds the following new APIs:
> >
> > Count the total number of blocks occupied by inode including
> > intermediate extent tree nodes.
> > extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
> > struct ext2_inode *inode);
> >
> > Convert ext3_extent to ext2fs_extent.
> > extern void ext2fs_convert_extent(struct ext2fs_extent *to,
> > struct ext3_extent *from);
>
> So one of the reasons why I've intentionally never exposed "struct
> ext3_extent" in the libext2fs interface is because that's an on-disk
> structure which I keep hoping we might change someday --- for example,
> to allow for 64-bit logical block numbers so we can create ext4 files
> greater than 2**32 blocks. It might be that some other future
> enhancement, such as say, reflinks (depending on how we implement
> them), or reverse pointers, might also require making changes to the
> on-disk format.
>
> The kernel code has the on-disk format and the various logical
> manipulations of the extent tree hopelessly entangled with each other,
> which means changing the kernel code to support more than one on-disk
> extent structure is going to be **hard**. But in the userspace code,
> all of the knowledge about the on-disk structure is abstracted away
> inside lib/ext2fs/extent.c.
>
> It may very well be that for fast commit, we're going to need to crack
> open that abstraction barrier a bit. But let's make sure the function
> name makes it clear that what we are doing here is converting between
> a particular on-disk encoding and the ext2fs abtract extent type.
> "ext2fs_convert_extent" doesn't exactly make this clear.
>
> It might also be that what should do is include a pointer to the fs
> and inode structures, and call this something like
> "ext2fs_{decode,encode}_extent()", and pass in the on-disk format via
> a void *. We might also want to have some kind of
> ext2fs_validate_extent() function which takes a void * and validates
> the on-disk structure to make sure it's sane.
>
> What do you think?
>
> - Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 05/15] mke2fs, tune2fs: update man page with fast commit info
2020-12-02 18:33 ` Theodore Y. Ts'o
@ 2020-12-10 5:24 ` harshad shirwadkar
0 siblings, 0 replies; 28+ messages in thread
From: harshad shirwadkar @ 2020-12-10 5:24 UTC (permalink / raw)
To: Theodore Y. Ts'o; +Cc: Ext4 Developers List
On Wed, Dec 2, 2020 at 10:33 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Fri, Nov 20, 2020 at 11:15:56AM -0800, Harshad Shirwadkar wrote:
> > This patch adds information about fast commit feature in mke2fs and
> > tune2fs man pages.
> >
> > Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
>
> So this is a bit more of a personal preference thing, but I like to
> keep the libext2fs changes from the changes to the userspace
> applications, and then combine the changes to the userspace progams
> (mke2fs and tune2fs in this case) with the man page updates.
>
> So you might want to consider moving the mke2fs and tune2fs changes
> from the previous patch and then combining them with this patch, and
> adjusting the commit message appropriately?
Sounds good, will do that.
>
> > diff --git a/misc/mke2fs.8.in b/misc/mke2fs.8.in
> > index e6bfc6d6..2833b408 100644
> > --- a/misc/mke2fs.8.in
> > +++ b/misc/mke2fs.8.in
> > @@ -521,6 +521,27 @@ The size of the journal must be at least 1024 filesystem blocks
> > and may be no more than 10,240,000 filesystem blocks or half the total
> > file system size (whichever is smaller)
> > .TP
> > +.BI fast_commit_size= fast-commit-size
> > +Create an additional fast commit journal area of size
> > +.I fast-commit-size
> > +kilobytes.
> > +This option is only valid if
> > +.B fast_commit
> > +feature is enabled
> > +on the file system. If this option is not specified and if
> > +.B fast_commit
> > +feature is turned on, fast commit area size defaults to
> > +.I journal-size
> > +/ 64 megabytes. The total size of the journal with
> > +.B fast_commit
> > +feature set is
> > +.I journal-size
> > ++ (
> > +.I fast-commit-size
> > +* 1024) megabytes. The total journal size may be no more than
> > +10,240,000 filesystem blocks or half the total file system size
> > +(whichever is smaller).
> > +.TP
>
> So as I recall, aren't we currently calculating the fast commit size
> as a fraction of the total journal size? I'm not sure this is in sync
> with was in the last patch.
So there are following three cases of journal area configuration:
1) User provides fast commit size and journal size as arguments to mke2fs
2) We are using internal journal and user asks mke2fs to calculate
journal size by default
3) We are using external journal and user asks mke2fs to calculate
journal size by default
So, for (1), we just provide an option "fast-commit-size" which is an
added area on top of the normal journal area. That's why total journal
size becomes the normal journal size + fast commit area size. However,
things become tricky for option (2) and (3). For (2), I'm *adding*
1/64th of the total journal area as a fast commit area on top of
existing journal. So, with fast commits enabled, the default total
journal size becomes 65/64 times the journal default journal area that
would have been created by mke2fs before these changes. For (3)
however, we don't have an option to use above logic since the external
device size is fixed. So, we have to divide the total journal area
into two parts. We split the external journal as 1:64 (fast commit :
normal commit).
Does this make sense?
Thanks,
Harshad
>
> - Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 06/15] ext2fs: add new APIs needed for fast commits
2020-12-10 1:45 ` harshad shirwadkar
@ 2020-12-10 15:48 ` Theodore Y. Ts'o
0 siblings, 0 replies; 28+ messages in thread
From: Theodore Y. Ts'o @ 2020-12-10 15:48 UTC (permalink / raw)
To: harshad shirwadkar; +Cc: Ext4 Developers List
On Wed, Dec 09, 2020 at 05:45:27PM -0800, harshad shirwadkar wrote:
> I see that makes sense. In that case, I'll rename the function to
> errcode_t ext2fs_decode_extent(struct ext2fs_extent *dst, void *src).
> I wonder if it's okay if we make this function return an error in case
> the on-disk format is not sane. If we do that, we can add
> ext2fs_validate_extent() later. Does that make sense?
Sure, that works for me.
Something that you should think about at some point is how much impact
would be supporting an alternate on-disk extent node structure (for
the leaf and/or intermediate nodes) have on Fast Commit? Obviously
doing this would a new an INCOMPAT feature at the file system level,
so we probably won't need any additional version negotiation in the
fast commit journal header itself, but how many tags would need to be
changed if we were to extend the extent tree structure sometime in the
future?
Cheers,
- Ted
^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2020-12-10 15:50 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-20 19:15 [PATCH 00/15] Fast commits support for e2fsprogs Harshad Shirwadkar
2020-11-20 19:15 ` [PATCH 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
2020-12-02 16:47 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 02/15] ext2fs, e2fsck: add kernel endian-ness conversion macros Harshad Shirwadkar
2020-12-02 16:50 ` Theodore Y. Ts'o
2020-12-03 18:10 ` harshad shirwadkar
2020-11-20 19:15 ` [PATCH 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Harshad Shirwadkar
2020-12-02 16:54 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 04/15] mke2fs, dumpe2fs: make fast commit blocks configurable Harshad Shirwadkar
2020-12-02 18:29 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 05/15] mke2fs, tune2fs: update man page with fast commit info Harshad Shirwadkar
2020-12-02 18:33 ` Theodore Y. Ts'o
2020-12-10 5:24 ` harshad shirwadkar
2020-11-20 19:15 ` [PATCH 06/15] ext2fs: add new APIs needed for fast commits Harshad Shirwadkar
2020-12-02 18:44 ` Theodore Y. Ts'o
2020-12-10 1:45 ` harshad shirwadkar
2020-12-10 15:48 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 07/15] e2fsck: add function to rewrite extent tree Harshad Shirwadkar
2020-12-02 18:46 ` Theodore Y. Ts'o
2020-11-20 19:15 ` [PATCH 08/15] e2fsck: add fast commit setup code Harshad Shirwadkar
2020-12-02 18:48 ` Theodore Y. Ts'o
2020-11-20 19:16 ` [PATCH 09/15] e2fsck: add fast commit scan pass Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 10/15] e2fsck: add fast commit replay skeleton Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 11/15] e2fsck: add fc replay for link, unlink, creat tags Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 12/15] e2fsck: add replay for add_range, del_range, and inode tags Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 13/15] debugfs: add fast commit support to logdump Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 14/15] tests: add fast commit recovery tests Harshad Shirwadkar
2020-11-20 19:16 ` [PATCH 15/15] ext4: fix tests to account for new dumpe2fs output Harshad Shirwadkar
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.