All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v3 00/15] Fast commit changes for e2fsprogs
@ 2021-01-20 21:26 Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
                   ` (15 more replies)
  0 siblings, 16 replies; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

This patch series adds fast commits support in e2fsprogs. This
includes fast commit recovery support in e2fsck and fast commit
configuration support in mke2fs and tune2fs. Along with that this
patch series also makes e2fsck/recovery.c identical with
jbd2/recovery.c in kernel. In addition, this patch imports and makes
fast_commit.h (the file that contains on-disk formats for fast
commits) byte identical with the kernel.

The recovery logic for fast commits follows the same steps as that of
the recovery logic in kernel. The general guidining principle for the
fast commit replay is that the individual tags found in fast commit
area store the result of the operation as their paylod instead of
storing the procedure itself. The recovery logic enforces this result
onto the filesystem thereby making the fast commit replay
idempotent. Unlike kernel, there's no atomic oepration support in
e2fsprogs yet. So, it's possible that we may crash while we are in the
middle of replaying of a fast commit tag. The only way to recover from
that situation would be to run fsck. That's why we mark the file
system as unclean before the fast commit replay and make it clean at
the end.

This series adds new regression test that performs fast commit
replay. I ensured that all the regressions tests pass.

Verified that all the tests pass:
367 tests succeeded     0 tests failed

New fast commit recovery test:
j_recover_fast_commit: ok

The patch series invalidates the initial version of the patch series
which was sent back in Mar 2020. Since then the fast commit code in
kernel has evolved a lot (including the on-disk format change). So,
this patch series is based on the new fast commit kernel code (which
is available in upstream kernel now). This patch series is a complete
revamp of the original series.

Changes Since V2:
----------------

- Fix compilation error by defining jbd2_journal_get_fc_num_blks (also
  rename it to jbd_get_fc_num_blks) as a preprocessor macro instead of
  inline function which gets compiled out when "-g" is passed

Harshad Shirwadkar (15):
  ext2fs: move calculate_summary_stats to ext2fs lib
  e2fsck: add kernel endian-ness conversion macros
  e2fsck: port fc changes from kernel's recovery.c to e2fsck
  libext2fs: provide APIs to configure fast commit blocks
  e2fsprogs: make userspace tools number of fast commits blocks aware
  ext2fs: add new APIs needed for fast commits
  e2fsck: add function to rewrite extent tree
  e2fsck: add fast commit setup code
  e2fsck: add fast commit scan pass
  e2fsck: add fast commit replay skeleton
  e2fsck: add fc replay for link, unlink, creat tags
  e2fsck: add replay for add_range, del_range, and inode tags
  debugfs: add fast commit support to logdump
  tests: add fast commit recovery tests
  ext4: fix tests to account for new dumpe2fs output

 debugfs/journal.c                       |  10 +-
 debugfs/logdump.c                       | 122 ++++-
 e2fsck/e2fsck.h                         |  32 ++
 e2fsck/extents.c                        | 168 +++---
 e2fsck/journal.c                        | 660 +++++++++++++++++++++++-
 e2fsck/recovery.c                       | 232 ++++++---
 e2fsck/unix.c                           |  26 +-
 lib/e2p/e2p.h                           |   1 +
 lib/e2p/ljs.c                           |  16 +-
 lib/ext2fs/ext2_fs.h                    |   1 +
 lib/ext2fs/ext2fs.h                     |  30 ++
 lib/ext2fs/extent.c                     |  63 +++
 lib/ext2fs/fast_commit.h                | 203 ++++++++
 lib/ext2fs/initialize.c                 |  94 ++++
 lib/ext2fs/jfs_compat.h                 |  25 +-
 lib/ext2fs/kernel-jbd.h                 |  16 +-
 lib/ext2fs/mkjournal.c                  |  96 +++-
 lib/ext2fs/unlink.c                     |   6 +-
 misc/dumpe2fs.c                         |  10 +-
 misc/mke2fs.8.in                        |  21 +
 misc/mke2fs.c                           |  26 +-
 misc/tune2fs.8.in                       |  25 +
 misc/tune2fs.c                          |  67 +--
 misc/util.c                             |  61 ++-
 misc/util.h                             |   4 +-
 resize/resize2fs.c                      |   6 +-
 tests/d_corrupt_journal_nr_users/expect |   6 +-
 tests/f_jnl_errno/expect.0              |   6 +-
 tests/f_opt_extent/expect               |   2 +-
 tests/i_bitmaps/expect                  |   8 +-
 tests/j_ext_dumpe2fs/expect             |   6 +-
 tests/j_recover_fast_commit/commands    |   4 +
 tests/j_recover_fast_commit/expect      |  23 +
 tests/j_recover_fast_commit/image.gz    | Bin 0 -> 3595 bytes
 tests/j_recover_fast_commit/script      |  26 +
 tests/m_bigjournal/expect.1             |   6 +-
 tests/m_extent_journal/expect.1         |   6 +-
 tests/m_resize_inode_meta_bg/expect.1   |   6 +-
 tests/m_rootdir/expect                  |   6 +-
 tests/r_32to64bit/expect                |   6 +-
 tests/r_32to64bit_meta/expect           |   4 +-
 tests/r_32to64bit_move_itable/expect    |   8 +-
 tests/r_64to32bit/expect                |   6 +-
 tests/r_64to32bit_meta/expect           |   4 +-
 tests/r_move_itable_nostride/expect     |   6 +-
 tests/r_move_itable_realloc/expect      |   6 +-
 tests/t_disable_mcsum/expect            |   4 +-
 tests/t_disable_mcsum_noinitbg/expect   |   6 +-
 tests/t_disable_mcsum_yesinitbg/expect  |   4 +-
 tests/t_enable_mcsum/expect             |   6 +-
 tests/t_enable_mcsum_ext3/expect        |  10 +-
 tests/t_enable_mcsum_initbg/expect      |   6 +-
 52 files changed, 1861 insertions(+), 341 deletions(-)
 create mode 100644 lib/ext2fs/fast_commit.h
 create mode 100644 tests/j_recover_fast_commit/commands
 create mode 100644 tests/j_recover_fast_commit/expect
 create mode 100644 tests/j_recover_fast_commit/image.gz
 create mode 100755 tests/j_recover_fast_commit/script

-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply	[flat|nested] 26+ messages in thread
* [PATCH v3 01/15] ext2fs: move calculate_summary_stats to ext2fs lib
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21 15:54   ` Theodore Ts'o
  2021-01-20 21:26 ` [PATCH v3 02/15] e2fsck: add kernel endian-ness conversion macros Harshad Shirwadkar
                   ` (14 subsequent siblings)
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

The function calculate_summary_stats sets the global metadata of the
file system. Tune2fs had this function defined statically in
tune2fs.c. Fast commit replay needs this function to set global
metadata at the end of the replay phase. So, move this function to
libext2fs.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
---
 lib/ext2fs/ext2fs.h     |  1 +
 lib/ext2fs/initialize.c | 94 +++++++++++++++++++++++++++++++++++++++++
 misc/tune2fs.c          | 59 +-------------------------
 resize/resize2fs.c      |  6 +--
 4 files changed, 99 insertions(+), 61 deletions(-)

diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 69c8a3ff..a8a6e091 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1484,6 +1484,7 @@ errcode_t ext2fs_write_ind_block(ext2_filsys fs, blk_t blk, void *buf);
 extern errcode_t ext2fs_initialize(const char *name, int flags,
 				   struct ext2_super_block *param,
 				   io_manager manager, ext2_filsys *ret_fs);
+extern errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs, int super_only);
 
 /* icount.c */
 extern void ext2fs_free_icount(ext2_icount_t icount);
diff --git a/lib/ext2fs/initialize.c b/lib/ext2fs/initialize.c
index 96ec1cff..edd692bb 100644
--- a/lib/ext2fs/initialize.c
+++ b/lib/ext2fs/initialize.c
@@ -575,3 +575,97 @@ cleanup:
 	ext2fs_free(fs);
 	return retval;
 }
+
+errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs, int super_only)
+{
+	blk64_t		blk;
+	ext2_ino_t	ino;
+	unsigned int	group = 0;
+	unsigned int	count = 0;
+	int		total_free = 0;
+	int		group_free = 0;
+	int		last_allocated = 0;
+	int		uninit;
+
+	/*
+	 * First calculate the block statistics
+	 */
+	uninit = 1;
+	for (blk = fs->super->s_first_data_block;
+	     blk < ext2fs_blocks_count(fs->super); blk++) {
+		if (!ext2fs_fast_test_block_bitmap2(fs->block_map, blk)) {
+			group_free++;
+			total_free++;
+		} else {
+			uninit = 0;
+		}
+		count++;
+		if ((count == fs->super->s_blocks_per_group) ||
+		    (blk == ext2fs_blocks_count(fs->super)-1)) {
+			ext2fs_bg_free_blocks_count_set(fs, group,
+							group_free);
+			if (!super_only) {
+				if (uninit && blk !=
+					ext2fs_blocks_count(fs->super) - 1)
+					ext2fs_bg_flags_set(fs, group,
+							EXT2_BG_BLOCK_UNINIT);
+				else
+					ext2fs_bg_flags_clear(fs, group,
+							EXT2_BG_BLOCK_UNINIT);
+			}
+			count = 0;
+			group_free = 0;
+			uninit = 1;
+			group++;
+		}
+	}
+	total_free = EXT2FS_C2B(fs, total_free);
+	ext2fs_free_blocks_count_set(fs->super, total_free);
+
+	/*
+	 * Next, calculate the inode statistics
+	 */
+	group_free = 0;
+	total_free = 0;
+	last_allocated = 0;
+	count = 0;
+	group = 0;
+
+	/* Protect loop from wrap-around if s_inodes_count maxed */
+	for (ino = 1; ino <= fs->super->s_inodes_count && ino > 0; ino++) {
+		if (!ext2fs_test_inode_bitmap2(fs->inode_map, ino)) {
+			group_free++;
+			total_free++;
+		} else {
+			last_allocated = ino;
+		}
+		count++;
+		if ((count == fs->super->s_inodes_per_group) ||
+		    (ino == fs->super->s_inodes_count)) {
+			if (!super_only) {
+				if (last_allocated) {
+					ext2fs_bg_flags_clear(fs, group,
+						EXT2_BG_INODE_UNINIT);
+					ext2fs_bg_itable_unused_set(fs, group,
+						fs->super->s_inodes_per_group -
+						(last_allocated %
+						fs->super->s_inodes_per_group));
+				} else {
+					ext2fs_bg_flags_set(fs, group,
+						EXT2_BG_INODE_UNINIT);
+					ext2fs_bg_itable_unused_set(fs, group,
+									0);
+				}
+				ext2fs_bg_free_inodes_count_set(fs, group,
+								group_free);
+			}
+			group++;
+			count = 0;
+			group_free = 0;
+			last_allocated = 0;
+		}
+	}
+	fs->super->s_free_inodes_count = total_free;
+	ext2fs_mark_super_dirty(fs);
+	return 0;
+}
diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index f942c698..670ed9e0 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -2603,63 +2603,6 @@ err_out:
 	return retval;
 }
 
-static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs)
-{
-	blk64_t		blk;
-	ext2_ino_t	ino;
-	unsigned int	group = 0;
-	unsigned int	count = 0;
-	int		total_free = 0;
-	int		group_free = 0;
-
-	/*
-	 * First calculate the block statistics
-	 */
-	for (blk = fs->super->s_first_data_block;
-	     blk < ext2fs_blocks_count(fs->super); blk++) {
-		if (!ext2fs_fast_test_block_bitmap2(fs->block_map, blk)) {
-			group_free++;
-			total_free++;
-		}
-		count++;
-		if ((count == fs->super->s_blocks_per_group) ||
-		    (blk == ext2fs_blocks_count(fs->super)-1)) {
-			ext2fs_bg_free_blocks_count_set(fs, group++,
-							group_free);
-			count = 0;
-			group_free = 0;
-		}
-	}
-	total_free = EXT2FS_C2B(fs, total_free);
-	ext2fs_free_blocks_count_set(fs->super, total_free);
-
-	/*
-	 * Next, calculate the inode statistics
-	 */
-	group_free = 0;
-	total_free = 0;
-	count = 0;
-	group = 0;
-
-	/* Protect loop from wrap-around if s_inodes_count maxed */
-	for (ino = 1; ino <= fs->super->s_inodes_count && ino > 0; ino++) {
-		if (!ext2fs_fast_test_inode_bitmap2(fs->inode_map, ino)) {
-			group_free++;
-			total_free++;
-		}
-		count++;
-		if ((count == fs->super->s_inodes_per_group) ||
-		    (ino == fs->super->s_inodes_count)) {
-			ext2fs_bg_free_inodes_count_set(fs, group++,
-							group_free);
-			count = 0;
-			group_free = 0;
-		}
-	}
-	fs->super->s_free_inodes_count = total_free;
-	ext2fs_mark_super_dirty(fs);
-	return 0;
-}
 
 #define list_for_each_safe(pos, pnext, head) \
 	for (pos = (head)->next, pnext = pos->next; pos != (head); \
@@ -2738,7 +2681,7 @@ static int resize_inode(ext2_filsys fs, unsigned long new_size)
 	if (retval)
 		goto err_out_undo;
 
-	ext2fs_calculate_summary_stats(fs);
+	ext2fs_calculate_summary_stats(fs, 1 /* super only */);
 
 	fs->super->s_state |= EXT2_VALID_FS;
 	/* mark super block and block bitmap as dirty */
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 2443ff67..270e4deb 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -49,7 +49,7 @@ static errcode_t inode_scan_and_fix(ext2_resize_t rfs);
 static errcode_t inode_ref_fix(ext2_resize_t rfs);
 static errcode_t move_itables(ext2_resize_t rfs);
 static errcode_t fix_resize_inode(ext2_filsys fs);
-static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs);
+static errcode_t resize2fs_calculate_summary_stats(ext2_filsys fs);
 static errcode_t fix_sb_journal_backup(ext2_filsys fs);
 static errcode_t mark_table_blocks(ext2_filsys fs,
 				   ext2fs_block_bitmap bmap);
@@ -211,7 +211,7 @@ errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags,
 		goto errout;
 
 	init_resource_track(&rtrack, "calculate_summary_stats", fs->io);
-	retval = ext2fs_calculate_summary_stats(rfs->new_fs);
+	retval = resize2fs_calculate_summary_stats(rfs->new_fs);
 	if (retval)
 		goto errout;
 	print_resource_track(rfs, &rtrack, fs->io);
@@ -2740,7 +2740,7 @@ errout:
 /*
  * Finally, recalculate the summary information
  */
-static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs)
+static errcode_t resize2fs_calculate_summary_stats(ext2_filsys fs)
 {
 	blk64_t		blk;
 	ext2_ino_t	ino;
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 02/15] e2fsck: add kernel endian-ness conversion macros
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21 15:54   ` Theodore Ts'o
  2021-01-20 21:26 ` [PATCH v3 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Harshad Shirwadkar
                   ` (13 subsequent siblings)
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar, Andreas Dilger

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

In order to make recovery.c identical with kernel, we need endianness
conversion macros (such as cpu_to_be32 and friends) defined in
e2fsprogs. This patch defines these macros and also fixes recovery.c
to use these. These macros are also needed for fast commit recovery
patches later in this series.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
---
 e2fsck/recovery.c       | 42 ++++++++++-------------------------------
 lib/ext2fs/jfs_compat.h |  6 ++++++
 2 files changed, 16 insertions(+), 32 deletions(-)

diff --git a/e2fsck/recovery.c b/e2fsck/recovery.c
index 5df690ad..6c3b7bb4 100644
--- a/e2fsck/recovery.c
+++ b/e2fsck/recovery.c
@@ -121,27 +121,6 @@ failed:
 
 #endif /* __KERNEL__ */
 
-static inline __u32 get_be32(__be32 *p)
-{
-	unsigned char *cp = (unsigned char *) p;
-	__u32 ret;
-
-	ret = *cp++;
-	ret = (ret << 8) + *cp++;
-	ret = (ret << 8) + *cp++;
-	ret = (ret << 8) + *cp++;
-	return ret;
-}
-
-static inline __u16 get_be16(__be16 *p)
-{
-	unsigned char *cp = (unsigned char *) p;
-	__u16 ret;
-
-	ret = *cp++;
-	ret = (ret << 8) + *cp++;
-	return ret;
-}
 
 /*
  * Read a block from the journal
@@ -232,10 +211,10 @@ static int count_tags(journal_t *journal, struct buffer_head *bh)
 
 		nr++;
 		tagp += tag_bytes;
-		if (!(get_be16(&tag->t_flags) & JBD2_FLAG_SAME_UUID))
+		if (!(tag->t_flags & cpu_to_be16(JBD2_FLAG_SAME_UUID)))
 			tagp += 16;
 
-		if (get_be16(&tag->t_flags) & JBD2_FLAG_LAST_TAG)
+		if (tag->t_flags & cpu_to_be16(JBD2_FLAG_LAST_TAG))
 			break;
 	}
 
@@ -358,9 +337,9 @@ int jbd2_journal_skip_recovery(journal_t *journal)
 static inline unsigned long long read_tag_block(journal_t *journal,
 						journal_block_tag_t *tag)
 {
-	unsigned long long block = get_be32(&tag->t_blocknr);
+	unsigned long long block = be32_to_cpu(tag->t_blocknr);
 	if (jbd2_has_feature_64bit(journal))
-		block |= (u64)get_be32(&tag->t_blocknr_high) << 32;
+		block |= (u64)be32_to_cpu(tag->t_blocknr_high) << 32;
 	return block;
 }
 
@@ -429,9 +408,9 @@ static int jbd2_block_tag_csum_verify(journal_t *j, journal_block_tag_t *tag,
 	csum32 = jbd2_chksum(j, csum32, buf, j->j_blocksize);
 
 	if (jbd2_has_feature_csum3(j))
-		return get_be32(&tag3->t_checksum) == csum32;
-
-	return get_be16(&tag->t_checksum) == (csum32 & 0xFFFF);
+		return tag3->t_checksum == cpu_to_be32(csum32);
+	else
+		return tag->t_checksum == cpu_to_be16(csum32);
 }
 
 static int do_one_pass(journal_t *journal,
@@ -579,7 +558,7 @@ static int do_one_pass(journal_t *journal,
 				unsigned long io_block;
 
 				tag = (journal_block_tag_t *) tagp;
-				flags = get_be16(&tag->t_flags);
+				flags = be16_to_cpu(tag->t_flags);
 
 				io_block = next_log_block++;
 				wrap(journal, next_log_block);
@@ -643,9 +622,8 @@ static int do_one_pass(journal_t *journal,
 					memcpy(nbh->b_data, obh->b_data,
 							journal->j_blocksize);
 					if (flags & JBD2_FLAG_ESCAPE) {
-						__be32 magic = cpu_to_be32(JBD2_MAGIC_NUMBER);
-						memcpy(nbh->b_data, &magic,
-						       sizeof(magic));
+						*((__be32 *)nbh->b_data) =
+						cpu_to_be32(JBD2_MAGIC_NUMBER);
 					}
 
 					BUFFER_TRACE(nbh, "marking dirty");
diff --git a/lib/ext2fs/jfs_compat.h b/lib/ext2fs/jfs_compat.h
index 2bda521d..63ebef99 100644
--- a/lib/ext2fs/jfs_compat.h
+++ b/lib/ext2fs/jfs_compat.h
@@ -20,12 +20,18 @@
 #define REQ_OP_READ 0
 #define REQ_OP_WRITE 1
 
+#define cpu_to_le16(x)	ext2fs_cpu_to_le16(x)
 #define cpu_to_be16(x)	ext2fs_cpu_to_be16(x)
+#define cpu_to_le32(x)	ext2fs_cpu_to_le32(x)
 #define cpu_to_be32(x)	ext2fs_cpu_to_be32(x)
+#define cpu_to_le64(x)	ext2fs_cpu_to_le64(x)
 #define cpu_to_be64(x)	ext2fs_cpu_to_be64(x)
 
+#define le16_to_cpu(x)	ext2fs_le16_to_cpu(x)
 #define be16_to_cpu(x)	ext2fs_be16_to_cpu(x)
+#define le32_to_cpu(x)	ext2fs_le32_to_cpu(x)
 #define be32_to_cpu(x)	ext2fs_be32_to_cpu(x)
+#define le64_to_cpu(x)	ext2fs_le64_to_cpu(x)
 #define be64_to_cpu(x)	ext2fs_be64_to_cpu(x)
 
 typedef unsigned int tid_t;
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 02/15] e2fsck: add kernel endian-ness conversion macros Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21 15:54   ` Theodore Ts'o
  2021-01-20 21:26 ` [PATCH v3 04/15] libext2fs: provide APIs to configure fast commit blocks Harshad Shirwadkar
                   ` (12 subsequent siblings)
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

This patch makes recovery.c identical with fast commit kernel changes.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
---
 debugfs/journal.c       |  10 +--
 e2fsck/journal.c        |  28 ++++--
 e2fsck/recovery.c       | 190 +++++++++++++++++++++++++++++-----------
 lib/ext2fs/jfs_compat.h |  19 +++-
 lib/ext2fs/kernel-jbd.h |  16 +++-
 5 files changed, 194 insertions(+), 69 deletions(-)

diff --git a/debugfs/journal.c b/debugfs/journal.c
index fa72ec57..e8872f05 100644
--- a/debugfs/journal.c
+++ b/debugfs/journal.c
@@ -378,7 +378,7 @@ try_backup_journal:
 				goto errout;
 		}
 
-		journal->j_maxlen = EXT2_I_SIZE(&j_inode->i_ext2) /
+		journal->j_total_len = EXT2_I_SIZE(&j_inode->i_ext2) /
 			journal->j_blocksize;
 
 #ifdef USE_INODE_IO
@@ -493,7 +493,7 @@ try_backup_journal:
 		brelse(bh);
 
 		maxlen = ext2fs_blocks_count(&jsuper);
-		journal->j_maxlen = (maxlen < 1ULL << 32) ? maxlen :
+		journal->j_total_len = (maxlen < 1ULL << 32) ? maxlen :
 				    (1ULL << 32) - 1;
 		start++;
 	}
@@ -629,9 +629,9 @@ static errcode_t ext2fs_journal_load(journal_t *journal)
 	if (jsb->s_blocksize != htonl(journal->j_blocksize))
 		return EXT2_ET_CORRUPT_JOURNAL_SB;
 
-	if (ntohl(jsb->s_maxlen) < journal->j_maxlen)
-		journal->j_maxlen = ntohl(jsb->s_maxlen);
-	else if (ntohl(jsb->s_maxlen) > journal->j_maxlen)
+	if (ntohl(jsb->s_maxlen) < journal->j_total_len)
+		journal->j_total_len = ntohl(jsb->s_maxlen);
+	else if (ntohl(jsb->s_maxlen) > journal->j_total_len)
 		return EXT2_ET_CORRUPT_JOURNAL_SB;
 
 	journal->j_tail_sequence = ntohl(jsb->s_sequence);
diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 7d9f1b40..2adef76a 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -379,7 +379,7 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal)
 				goto errout;
 		}
 
-		journal->j_maxlen = EXT2_I_SIZE(&j_inode->i_ext2) /
+		journal->j_total_len = EXT2_I_SIZE(&j_inode->i_ext2) /
 			journal->j_blocksize;
 
 #ifdef USE_INODE_IO
@@ -503,7 +503,7 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal)
 		brelse(bh);
 
 		maxlen = ext2fs_blocks_count(&jsuper);
-		journal->j_maxlen = (maxlen < 1ULL << 32) ? maxlen : (1ULL << 32) - 1;
+		journal->j_total_len = (maxlen < 1ULL << 32) ? maxlen : (1ULL << 32) - 1;
 		start++;
 	}
 
@@ -675,9 +675,9 @@ static errcode_t e2fsck_journal_load(journal_t *journal)
 		return EXT2_ET_CORRUPT_JOURNAL_SB;
 	}
 
-	if (ntohl(jsb->s_maxlen) < journal->j_maxlen)
-		journal->j_maxlen = ntohl(jsb->s_maxlen);
-	else if (ntohl(jsb->s_maxlen) > journal->j_maxlen) {
+	if (ntohl(jsb->s_maxlen) < journal->j_total_len)
+		journal->j_total_len = ntohl(jsb->s_maxlen);
+	else if (ntohl(jsb->s_maxlen) > journal->j_total_len) {
 		com_err(ctx->program_name, EXT2_ET_CORRUPT_JOURNAL_SB,
 			_("%s: journal too short\n"),
 			ctx->device_name);
@@ -688,7 +688,21 @@ static errcode_t e2fsck_journal_load(journal_t *journal)
 	journal->j_transaction_sequence = journal->j_tail_sequence;
 	journal->j_tail = ntohl(jsb->s_start);
 	journal->j_first = ntohl(jsb->s_first);
-	journal->j_last = ntohl(jsb->s_maxlen);
+	if (jbd2_has_feature_fast_commit(journal)) {
+		if (ntohl(jsb->s_maxlen) - jbd_get_num_fc_blks(jsb)
+			< JBD2_MIN_JOURNAL_BLOCKS) {
+			com_err(ctx->program_name, EXT2_ET_CORRUPT_JOURNAL_SB,
+				_("%s: incorrect fast commit blocks\n"),
+				ctx->device_name);
+			return EXT2_ET_CORRUPT_JOURNAL_SB;
+		}
+		journal->j_fc_last = ntohl(jsb->s_maxlen);
+		journal->j_last = journal->j_fc_last -
+					jbd_get_num_fc_blks(jsb);
+		journal->j_fc_first = journal->j_last + 1;
+	} else {
+		journal->j_last = ntohl(jsb->s_maxlen);
+	}
 
 	return 0;
 }
@@ -720,7 +734,7 @@ static void e2fsck_journal_reset_super(e2fsck_t ctx, journal_superblock_t *jsb,
 	memset (p, 0, ctx->fs->blocksize-sizeof(journal_header_t));
 
 	jsb->s_blocksize = htonl(ctx->fs->blocksize);
-	jsb->s_maxlen = htonl(journal->j_maxlen);
+	jsb->s_maxlen = htonl(journal->j_total_len);
 	jsb->s_first = htonl(1);
 
 	/* Initialize the journal sequence number so that there is "no"
diff --git a/e2fsck/recovery.c b/e2fsck/recovery.c
index 6c3b7bb4..dc0694fc 100644
--- a/e2fsck/recovery.c
+++ b/e2fsck/recovery.c
@@ -35,7 +35,6 @@ struct recovery_info
 	int		nr_revoke_hits;
 };
 
-enum passtype {PASS_SCAN, PASS_REVOKE, PASS_REPLAY};
 static int do_one_pass(journal_t *journal,
 				struct recovery_info *info, enum passtype pass);
 static int scan_revoke_records(journal_t *, struct buffer_head *,
@@ -75,8 +74,8 @@ static int do_readahead(journal_t *journal, unsigned int start)
 
 	/* Do up to 128K of readahead */
 	max = start + (128 * 1024 / journal->j_blocksize);
-	if (max > journal->j_maxlen)
-		max = journal->j_maxlen;
+	if (max > journal->j_total_len)
+		max = journal->j_total_len;
 
 	/* Do the readahead itself.  We'll submit MAXBUF buffer_heads at
 	 * a time to the block device IO layer. */
@@ -135,7 +134,7 @@ static int jread(struct buffer_head **bhp, journal_t *journal,
 
 	*bhp = NULL;
 
-	if (offset >= journal->j_maxlen) {
+	if (offset >= journal->j_total_len) {
 		printk(KERN_ERR "JBD2: corrupted journal superblock\n");
 		return -EFSCORRUPTED;
 	}
@@ -180,7 +179,7 @@ static int jbd2_descriptor_block_csum_verify(journal_t *j, void *buf)
 	if (!jbd2_journal_has_csum_v2or3(j))
 		return 1;
 
-	tail = (struct jbd2_journal_block_tail *)((char *)buf + j->j_blocksize -
+	tail = (struct jbd2_journal_block_tail *)(buf + j->j_blocksize -
 			sizeof(struct jbd2_journal_block_tail));
 	provided = tail->t_checksum;
 	tail->t_checksum = 0;
@@ -225,10 +224,51 @@ static int count_tags(journal_t *journal, struct buffer_head *bh)
 /* Make sure we wrap around the log correctly! */
 #define wrap(journal, var)						\
 do {									\
-	if (var >= (journal)->j_last)					\
-		var -= ((journal)->j_last - (journal)->j_first);	\
+	unsigned long _wrap_last =					\
+		jbd2_has_feature_fast_commit(journal) ?			\
+			(journal)->j_fc_last : (journal)->j_last;	\
+									\
+	if (var >= _wrap_last)						\
+		var -= (_wrap_last - (journal)->j_first);		\
 } while (0)
 
+static int fc_do_one_pass(journal_t *journal,
+			  struct recovery_info *info, enum passtype pass)
+{
+	unsigned int expected_commit_id = info->end_transaction;
+	unsigned long next_fc_block;
+	struct buffer_head *bh;
+	int err = 0;
+
+	next_fc_block = journal->j_fc_first;
+	if (!journal->j_fc_replay_callback)
+		return 0;
+
+	while (next_fc_block <= journal->j_fc_last) {
+		jbd_debug(3, "Fast commit replay: next block %ld",
+			  next_fc_block);
+		err = jread(&bh, journal, next_fc_block);
+		if (err) {
+			jbd_debug(3, "Fast commit replay: read error");
+			break;
+		}
+
+		jbd_debug(3, "Processing fast commit blk with seq %d");
+		err = journal->j_fc_replay_callback(journal, bh, pass,
+					next_fc_block - journal->j_fc_first,
+					expected_commit_id);
+		next_fc_block++;
+		if (err < 0 || err == JBD2_FC_REPLAY_STOP)
+			break;
+		err = 0;
+	}
+
+	if (err)
+		jbd_debug(3, "Fast commit replay failed, err = %d\n", err);
+
+	return err;
+}
+
 /**
  * jbd2_journal_recover - recovers a on-disk journal
  * @journal: the journal to recover
@@ -286,7 +326,7 @@ int jbd2_journal_recover(journal_t *journal)
 		err = err2;
 	/* Make sure all replayed data is on permanent storage */
 	if (journal->j_flags & JBD2_BARRIER) {
-		err2 = blkdev_issue_flush(journal->j_fs_dev, GFP_KERNEL, NULL);
+		err2 = blkdev_issue_flush(journal->j_fs_dev, GFP_KERNEL);
 		if (!err)
 			err = err2;
 	}
@@ -428,6 +468,8 @@ static int do_one_pass(journal_t *journal,
 	__u32			crc32_sum = ~0; /* Transactional Checksums */
 	int			descr_csum_size = 0;
 	int			block_error = 0;
+	bool			need_check_commit_time = false;
+	__u64			last_trans_commit_time = 0, commit_time;
 
 	/*
 	 * First thing is to establish what we expect to find in the log
@@ -470,7 +512,9 @@ static int do_one_pass(journal_t *journal,
 				break;
 
 		jbd_debug(2, "Scanning for sequence ID %u at %lu/%lu\n",
-			  next_commit_ID, next_log_block, journal->j_last);
+			  next_commit_ID, next_log_block,
+			  jbd2_has_feature_fast_commit(journal) ?
+			  journal->j_fc_last : journal->j_last);
 
 		/* Skip over each chunk of the transaction looking
 		 * either the next descriptor block or the final commit
@@ -520,9 +564,21 @@ static int do_one_pass(journal_t *journal,
 			if (descr_csum_size > 0 &&
 			    !jbd2_descriptor_block_csum_verify(journal,
 							       bh->b_data)) {
-				err = -EFSBADCRC;
-				brelse(bh);
-				goto failed;
+				/*
+				 * PASS_SCAN can see stale blocks due to lazy
+				 * journal init. Don't error out on those yet.
+				 */
+				if (pass != PASS_SCAN) {
+					pr_err("JBD2: Invalid checksum recovering block %lu in log\n",
+					       next_log_block);
+					err = -EFSBADCRC;
+					brelse(bh);
+					goto failed;
+				}
+				need_check_commit_time = true;
+				jbd_debug(1,
+					"invalid descriptor block found in %lu\n",
+					next_log_block);
 			}
 
 			/* If it is a valid descriptor block, replay it
@@ -532,6 +588,7 @@ static int do_one_pass(journal_t *journal,
 			if (pass != PASS_REPLAY) {
 				if (pass == PASS_SCAN &&
 				    jbd2_has_feature_checksum(journal) &&
+				    !need_check_commit_time &&
 				    !info->end_transaction) {
 					if (calc_chksums(journal, bh,
 							&next_log_block,
@@ -663,7 +720,7 @@ static int do_one_pass(journal_t *journal,
 			 *		| GO TO NEXT    "Journal Corruption"
 			 *		| TRANSACTION
 			 *		|
-			 * {(n+1)th transaction}
+			 * {(n+1)th transanction}
 			 *		|
 			 * 	 _______|______________
 			 * 	|	 	      |
@@ -680,21 +737,48 @@ static int do_one_pass(journal_t *journal,
 			 *	 mentioned conditions. Hence assume
 			 *	 "Interrupted Commit".)
 			 */
+			commit_time = be64_to_cpu(
+				((struct commit_header *)bh->b_data)->h_commit_sec);
+			/*
+			 * If need_check_commit_time is set, it means we are in
+			 * PASS_SCAN and csum verify failed before. If
+			 * commit_time is increasing, it's the same journal,
+			 * otherwise it is stale journal block, just end this
+			 * recovery.
+			 */
+			if (need_check_commit_time) {
+				if (commit_time >= last_trans_commit_time) {
+					pr_err("JBD2: Invalid checksum found in transaction %u\n",
+					       next_commit_ID);
+					err = -EFSBADCRC;
+					brelse(bh);
+					goto failed;
+				}
+			ignore_crc_mismatch:
+				/*
+				 * It likely does not belong to same journal,
+				 * just end this recovery with success.
+				 */
+				jbd_debug(1, "JBD2: Invalid checksum ignored in transaction %u, likely stale data\n",
+					  next_commit_ID);
+				err = 0;
+				brelse(bh);
+				goto done;
+			}
 
-			/* Found an expected commit block: if checksums
-			 * are present verify them in PASS_SCAN; else not
+			/*
+			 * Found an expected commit block: if checksums
+			 * are present, verify them in PASS_SCAN; else not
 			 * much to do other than move on to the next sequence
-			 * number. */
+			 * number.
+			 */
 			if (pass == PASS_SCAN &&
 			    jbd2_has_feature_checksum(journal)) {
-				int chksum_err, chksum_seen;
 				struct commit_header *cbh =
 					(struct commit_header *)bh->b_data;
 				unsigned found_chksum =
 					be32_to_cpu(cbh->h_chksum[0]);
 
-				chksum_err = chksum_seen = 0;
-
 				if (info->end_transaction) {
 					journal->j_failed_commit =
 						info->end_transaction;
@@ -702,42 +786,25 @@ static int do_one_pass(journal_t *journal,
 					break;
 				}
 
-				if (crc32_sum == found_chksum &&
-				    cbh->h_chksum_type == JBD2_CRC32_CHKSUM &&
-				    cbh->h_chksum_size ==
-						JBD2_CRC32_CHKSUM_SIZE)
-				       chksum_seen = 1;
-				else if (!(cbh->h_chksum_type == 0 &&
-					     cbh->h_chksum_size == 0 &&
-					     found_chksum == 0 &&
-					     !chksum_seen))
-				/*
-				 * If fs is mounted using an old kernel and then
-				 * kernel with journal_chksum is used then we
-				 * get a situation where the journal flag has
-				 * checksum flag set but checksums are not
-				 * present i.e chksum = 0, in the individual
-				 * commit blocks.
-				 * Hence to avoid checksum failures, in this
-				 * situation, this extra check is added.
-				 */
-						chksum_err = 1;
-
-				if (chksum_err) {
-					info->end_transaction = next_commit_ID;
+				/* Neither checksum match nor unused? */
+				if (!((crc32_sum == found_chksum &&
+				       cbh->h_chksum_type ==
+						JBD2_CRC32_CHKSUM &&
+				       cbh->h_chksum_size ==
+						JBD2_CRC32_CHKSUM_SIZE) ||
+				      (cbh->h_chksum_type == 0 &&
+				       cbh->h_chksum_size == 0 &&
+				       found_chksum == 0)))
+					goto chksum_error;
 
-					if (!jbd2_has_feature_async_commit(journal)) {
-						journal->j_failed_commit =
-							next_commit_ID;
-						brelse(bh);
-						break;
-					}
-				}
 				crc32_sum = ~0;
 			}
 			if (pass == PASS_SCAN &&
 			    !jbd2_commit_block_csum_verify(journal,
 							   bh->b_data)) {
+			chksum_error:
+				if (commit_time < last_trans_commit_time)
+					goto ignore_crc_mismatch;
 				info->end_transaction = next_commit_ID;
 
 				if (!jbd2_has_feature_async_commit(journal)) {
@@ -747,11 +814,24 @@ static int do_one_pass(journal_t *journal,
 					break;
 				}
 			}
+			if (pass == PASS_SCAN)
+				last_trans_commit_time = commit_time;
 			brelse(bh);
 			next_commit_ID++;
 			continue;
 
 		case JBD2_REVOKE_BLOCK:
+			/*
+			 * Check revoke block crc in pass_scan, if csum verify
+			 * failed, check commit block time later.
+			 */
+			if (pass == PASS_SCAN &&
+			    !jbd2_descriptor_block_csum_verify(journal,
+							       bh->b_data)) {
+				jbd_debug(1, "JBD2: invalid revoke block found in %lu\n",
+					  next_log_block);
+				need_check_commit_time = true;
+			}
 			/* If we aren't in the REVOKE pass, then we can
 			 * just skip over this block. */
 			if (pass != PASS_REVOKE) {
@@ -796,6 +876,13 @@ static int do_one_pass(journal_t *journal,
 				success = -EIO;
 		}
 	}
+
+	if (jbd2_has_feature_fast_commit(journal) &&  pass != PASS_REVOKE) {
+		err = fc_do_one_pass(journal, info, pass);
+		if (err)
+			success = err;
+	}
+
 	if (block_error && success == 0)
 		success = -EIO;
 	return success;
@@ -811,7 +898,7 @@ static int scan_revoke_records(journal_t *journal, struct buffer_head *bh,
 {
 	jbd2_journal_revoke_header_t *header;
 	int offset, max;
-	unsigned csum_size = 0;
+	int csum_size = 0;
 	__u32 rcount;
 	int record_len = 4;
 
@@ -819,9 +906,6 @@ static int scan_revoke_records(journal_t *journal, struct buffer_head *bh,
 	offset = sizeof(jbd2_journal_revoke_header_t);
 	rcount = be32_to_cpu(header->r_count);
 
-	if (!jbd2_descriptor_block_csum_verify(journal, header))
-		return -EFSBADCRC;
-
 	if (jbd2_journal_has_csum_v2or3(journal))
 		csum_size = sizeof(struct jbd2_journal_block_tail);
 	if (rcount > journal->j_blocksize - csum_size)
diff --git a/lib/ext2fs/jfs_compat.h b/lib/ext2fs/jfs_compat.h
index 63ebef99..96fe34a4 100644
--- a/lib/ext2fs/jfs_compat.h
+++ b/lib/ext2fs/jfs_compat.h
@@ -12,6 +12,7 @@
 #else
 #include <arpa/inet.h>
 #endif
+#include <stdbool.h>
 
 #define printk printf
 #define KERN_ERR ""
@@ -66,9 +67,15 @@ static inline __u32 jbd2_chksum(journal_t *j EXT2FS_ATTR((unused)),
                 sizeof(struct __struct), __alignof__(struct __struct),\
                 (__flags), NULL)
 
-#define blkdev_issue_flush(kdev, a, b)	sync_blockdev(kdev)
+#define blkdev_issue_flush(kdev, a)	sync_blockdev(kdev)
 #define is_power_of_2(x)	((x) != 0 && (((x) & ((x) - 1)) == 0))
 #define pr_emerg(fmt)
+#define pr_err(...)
+
+enum passtype {PASS_SCAN, PASS_REVOKE, PASS_REPLAY};
+
+#define JBD2_FC_REPLAY_STOP		0
+#define JBD2_FC_REPLAY_CONTINUE		1
 
 struct journal_s
 {
@@ -79,13 +86,16 @@ struct journal_s
 	int			j_format_version;
 	unsigned long		j_head;
 	unsigned long		j_tail;
+	unsigned long		j_fc_first;
+	unsigned long		j_fc_off;
+	unsigned long		j_fc_last;
 	unsigned long		j_free;
 	unsigned long		j_first, j_last;
 	kdev_t			j_dev;
 	kdev_t			j_fs_dev;
 	int			j_blocksize;
 	unsigned int		j_blk_offset;
-	unsigned int		j_maxlen;
+	unsigned int		j_total_len;
 	struct inode *		j_inode;
 	tid_t			j_tail_sequence;
 	tid_t			j_transaction_sequence;
@@ -94,6 +104,11 @@ struct journal_s
 	struct jbd2_revoke_table_s *j_revoke_table[2];
 	tid_t			j_failed_commit;
 	__u32			j_csum_seed;
+	int (*j_fc_replay_callback)(struct journal_s *journal,
+				    struct buffer_head *bh,
+				    enum passtype pass, int off,
+				    tid_t expected_tid);
+
 };
 
 #define is_journal_abort(x) 0
diff --git a/lib/ext2fs/kernel-jbd.h b/lib/ext2fs/kernel-jbd.h
index cb1bc308..6ec1a8c9 100644
--- a/lib/ext2fs/kernel-jbd.h
+++ b/lib/ext2fs/kernel-jbd.h
@@ -72,8 +72,13 @@ extern void * __jbd_kmalloc (char *where, size_t size, int flags, int retry);
 	__jbd_kmalloc(__FUNCTION__, (size), (flags), journal_oom_retry)
 #define jbd_rep_kmalloc(size, flags) \
 	__jbd_kmalloc(__FUNCTION__, (size), (flags), 1)
+#define jbd_get_num_fc_blks(jsb)					\
+	(be32_to_cpu((jsb)->s_num_fc_blks) ?				\
+	 be32_to_cpu((jsb)->s_num_fc_blks) : JBD2_DEFAULT_FAST_COMMIT_BLOCKS)
+
 
 #define JBD2_MIN_JOURNAL_BLOCKS 1024
+#define JBD2_DEFAULT_FAST_COMMIT_BLOCKS 256
 
 /*
  * Internal structures used by the logging mechanism:
@@ -94,6 +99,7 @@ extern void * __jbd_kmalloc (char *where, size_t size, int flags, int retry);
 #define JBD2_SUPERBLOCK_V1	3
 #define JBD2_SUPERBLOCK_V2	4
 #define JBD2_REVOKE_BLOCK	5
+#define JBD2_FC_BLOCK		6
 
 /*
  * Standard header for all descriptor blocks:
@@ -233,7 +239,10 @@ typedef struct journal_superblock_s
 /* 0x0050 */
 	__u8	s_checksum_type;	/* checksum type */
 	__u8	s_padding2[3];
-	__be32	s_padding[42];
+/* 0x0054 */
+	__be32	s_num_fc_blks;		/* Number of fast commit blocks */
+/* 0x0058 */
+	__be32	s_padding[41];
 	__be32	s_checksum;		/* crc32c(superblock) */
 
 /* 0x0100 */
@@ -259,6 +268,7 @@ typedef struct journal_superblock_s
 #define JBD2_FEATURE_INCOMPAT_ASYNC_COMMIT	0x00000004
 #define JBD2_FEATURE_INCOMPAT_CSUM_V2		0x00000008
 #define JBD2_FEATURE_INCOMPAT_CSUM_V3		0x00000010
+#define JBD2_FEATURE_INCOMPAT_FAST_COMMIT	0x00000020
 
 /* Features known to this kernel version: */
 #define JBD2_KNOWN_COMPAT_FEATURES	0
@@ -267,7 +277,8 @@ typedef struct journal_superblock_s
 					 JBD2_FEATURE_INCOMPAT_ASYNC_COMMIT| \
 					 JBD2_FEATURE_INCOMPAT_64BIT|\
 					 JBD2_FEATURE_INCOMPAT_CSUM_V2|	\
-					 JBD2_FEATURE_INCOMPAT_CSUM_V3)
+					 JBD2_FEATURE_INCOMPAT_CSUM_V3 | \
+					 JBD2_FEATURE_INCOMPAT_FAST_COMMIT)
 
 #ifdef NO_INLINE_FUNCS
 extern size_t journal_tag_bytes(journal_t *journal);
@@ -384,6 +395,7 @@ JBD2_FEATURE_INCOMPAT_FUNCS(64bit,		64BIT)
 JBD2_FEATURE_INCOMPAT_FUNCS(async_commit,	ASYNC_COMMIT)
 JBD2_FEATURE_INCOMPAT_FUNCS(csum2,		CSUM_V2)
 JBD2_FEATURE_INCOMPAT_FUNCS(csum3,		CSUM_V3)
+JBD2_FEATURE_INCOMPAT_FUNCS(fast_commit,	FAST_COMMIT)
 
 #if (defined(E2FSCK_INCLUDE_INLINE_FUNCS) || !defined(NO_INLINE_FUNCS))
 /*
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 04/15] libext2fs: provide APIs to configure fast commit blocks
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (2 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21 15:56   ` Theodore Ts'o
  2021-01-20 21:26 ` [PATCH v3 05/15] e2fsprogs: make userspace tools number of fast commits blocks aware Harshad Shirwadkar
                   ` (11 subsequent siblings)
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

This patch adds new libext2fs that allow configuring number of fast
commit blocks in journal superblock. We also add a struct
ext2fs_journal_params which contains number of fast commit blocks and
number of normal journal blocks. With this patch, the preferred way
for configuring number of blocks with and without fast commits is:

struct ext2fs_journal_params params;

ext2fs_get_journal_params(&params, ...);
params.num_journal_blocks = ...;
params.num_fc_blocks = ...;
ext2fs_create_journal_superblock2(..., &params, ...);
         OR
ext2fs_add_journal_inode3(..., &params, ...);

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 lib/e2p/e2p.h          |  1 +
 lib/e2p/ljs.c          | 16 +++++--
 lib/ext2fs/ext2fs.h    | 18 ++++++++
 lib/ext2fs/mkjournal.c | 96 +++++++++++++++++++++++++++++++++++-------
 4 files changed, 113 insertions(+), 18 deletions(-)

diff --git a/lib/e2p/e2p.h b/lib/e2p/e2p.h
index 90efb624..65702a7e 100644
--- a/lib/e2p/e2p.h
+++ b/lib/e2p/e2p.h
@@ -47,6 +47,7 @@ void print_fs_state (FILE * f, unsigned short state);
 int setflags (int fd, unsigned long flags);
 int setversion (int fd, unsigned long version);
 
+#define E2P_LIST_JOURNAL_FLAG_FC		0x1
 void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
 			    int exp_block_size, int flags);
 
diff --git a/lib/e2p/ljs.c b/lib/e2p/ljs.c
index 4ffe9b61..d118ec44 100644
--- a/lib/e2p/ljs.c
+++ b/lib/e2p/ljs.c
@@ -54,7 +54,12 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
 	unsigned int size;
 	int j, printed = 0;
 	unsigned int i, nr_users;
+	int num_fc_blks = 0;
+	int journal_blks = 0;
 
+	if (flags & E2P_LIST_JOURNAL_FLAG_FC)
+		num_fc_blks = jbd_get_num_fc_blks((journal_superblock_t *)journal_sb_buf);
+	journal_blks = ntohl(jsb->s_maxlen) - num_fc_blks;
 	fprintf(f, "%s", "Journal features:        ");
 	for (i=0, mask_ptr=&jsb->s_feature_compat; i <3; i++,mask_ptr++) {
 		mask = e2p_be32(*mask_ptr);
@@ -68,7 +73,7 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
 	if (printed == 0)
 		fprintf(f, " (none)");
 	fputc('\n', f);
-	fputs("Journal size:             ", f);
+	fputs("Total journal size:       ", f);
 	size = (ntohl(jsb->s_blocksize) / 1024) * ntohl(jsb->s_maxlen);
 	if (size < 8192)
 		fprintf(f, "%uk\n", size);
@@ -78,8 +83,13 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
 	if (exp_block_size != (int) ntohl(jsb->s_blocksize))
 		fprintf(f, "Journal block size:       %u\n",
 			(unsigned int)ntohl(jsb->s_blocksize));
-	fprintf(f, "Journal length:           %u\n",
-		(unsigned int)ntohl(jsb->s_maxlen));
+	fprintf(f, "Total journal blocks:     %u\n",
+		(unsigned int)(journal_blks + num_fc_blks));
+	fprintf(f, "Max transaction length:   %u\n",
+		(unsigned int)journal_blks);
+	fprintf(f, "Fast commit length:       %u\n",
+		(unsigned int)num_fc_blks);
+
 	if (ntohl(jsb->s_first) != 1)
 		fprintf(f, "Journal first block:      %u\n",
 			(unsigned int)ntohl(jsb->s_first));
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index a8a6e091..ec841006 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -220,6 +220,12 @@ typedef struct ext2_file *ext2_file_t;
 #define EXT2_MKJOURNAL_LAZYINIT	0x0000002 /* don't zero journal inode before use*/
 #define EXT2_MKJOURNAL_NO_MNT_CHECK 0x0000004 /* don't check mount status */
 
+/*
+ * Normal journal area size to fast commit area size ratio. This is used to
+ * set default size of fast commit area.
+ */
+#define EXT2_JOURNAL_TO_FC_BLKS_RATIO		64
+
 struct blk_alloc_ctx;
 struct opaque_ext2_group_desc;
 
@@ -1620,6 +1626,12 @@ extern errcode_t ext2fs_mkdir(ext2_filsys fs, ext2_ino_t parent, ext2_ino_t inum
 			      const char *name);
 
 /* mkjournal.c */
+struct ext2fs_journal_params {
+	blk_t num_journal_blocks;
+	blk_t num_fc_blocks;
+};
+extern errcode_t ext2fs_get_journal_params(
+		struct ext2fs_journal_params *params, ext2_filsys fs);
 extern errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
 				    blk_t *ret_blk, int *ret_count);
 extern errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
@@ -1627,12 +1639,18 @@ extern errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
 extern errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
 						  __u32 num_blocks, int flags,
 						  char  **ret_jsb);
+extern errcode_t ext2fs_create_journal_superblock2(ext2_filsys fs,
+						  struct ext2fs_journal_params *params,
+						  int flags, char  **ret_jsb);
 extern errcode_t ext2fs_add_journal_device(ext2_filsys fs,
 					   ext2_filsys journal_dev);
 extern errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks,
 					  int flags);
 extern errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
 					   blk64_t goal, int flags);
+extern errcode_t ext2fs_add_journal_inode3(ext2_filsys fs,
+				    struct ext2fs_journal_params *params,
+				    blk64_t goal, int flags);
 extern int ext2fs_default_journal_size(__u64 num_blocks);
 extern int ext2fs_journal_sb_start(int blocksize);
 
diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c
index f47f71e6..732ba7d6 100644
--- a/lib/ext2fs/mkjournal.c
+++ b/lib/ext2fs/mkjournal.c
@@ -12,6 +12,7 @@
 #include "config.h"
 #include <stdio.h>
 #include <string.h>
+#include <assert.h>
 #if HAVE_UNISTD_H
 #include <unistd.h>
 #endif
@@ -43,14 +44,14 @@
  * This function automatically sets up the journal superblock and
  * returns it as an allocated block.
  */
-errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
-					   __u32 num_blocks, int flags,
-					   char  **ret_jsb)
+errcode_t ext2fs_create_journal_superblock2(ext2_filsys fs,
+					   struct ext2fs_journal_params *jparams,
+					   int flags, char **ret_jsb)
 {
 	errcode_t		retval;
 	journal_superblock_t	*jsb;
 
-	if (num_blocks < JBD2_MIN_JOURNAL_BLOCKS)
+	if (jparams->num_journal_blocks < JBD2_MIN_JOURNAL_BLOCKS)
 		return EXT2_ET_JOURNAL_TOO_SMALL;
 
 	if ((retval = ext2fs_get_mem(fs->blocksize, &jsb)))
@@ -64,10 +65,11 @@ errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
 	else
 		jsb->s_header.h_blocktype = htonl(JBD2_SUPERBLOCK_V2);
 	jsb->s_blocksize = htonl(fs->blocksize);
-	jsb->s_maxlen = htonl(num_blocks);
+	jsb->s_maxlen = htonl(jparams->num_journal_blocks + jparams->num_fc_blocks);
 	jsb->s_nr_users = htonl(1);
 	jsb->s_first = htonl(1);
 	jsb->s_sequence = htonl(1);
+	jsb->s_num_fc_blks = htonl(jparams->num_fc_blocks);
 	memcpy(jsb->s_uuid, fs->super->s_uuid, sizeof(fs->super->s_uuid));
 	/*
 	 * If we're creating an external journal device, we need to
@@ -82,20 +84,32 @@ errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
 	return 0;
 }
 
+errcode_t ext2fs_create_journal_superblock(ext2_filsys fs, __u32 num_blocks,
+					int flags, char **ret_sb)
+{
+	struct ext2fs_journal_params jparams;
+
+	jparams.num_journal_blocks = num_blocks;
+	jparams.num_fc_blocks = 0;
+
+	return ext2fs_create_journal_superblock2(fs, &jparams, flags, ret_sb);
+}
+
 /*
  * This function writes a journal using POSIX routines.  It is used
  * for creating external journals and creating journals on live
  * filesystems.
  */
 static errcode_t write_journal_file(ext2_filsys fs, char *filename,
-				    blk_t num_blocks, int flags)
+				    struct ext2fs_journal_params *jparams,
+				    int flags)
 {
 	errcode_t	retval;
 	char		*buf = 0;
 	int		fd, ret_size;
 	blk_t		i;
 
-	if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags,
+	if ((retval = ext2fs_create_journal_superblock2(fs, jparams, flags,
 						       &buf)))
 		return retval;
 
@@ -119,7 +133,7 @@ static errcode_t write_journal_file(ext2_filsys fs, char *filename,
 	if (flags & EXT2_MKJOURNAL_LAZYINIT)
 		goto success;
 
-	for (i = 1; i < num_blocks; i++) {
+	for (i = 1; i < jparams->num_journal_blocks + jparams->num_fc_blocks; i++) {
 		ret_size = write(fd, buf, fs->blocksize);
 		if (ret_size < 0) {
 			retval = errno;
@@ -262,7 +276,8 @@ static blk64_t get_midpoint_journal_block(ext2_filsys fs)
  * This function creates a journal using direct I/O routines.
  */
 static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
-				     blk_t num_blocks, blk64_t goal, int flags)
+				     struct ext2fs_journal_params *jparams,
+				     blk64_t goal, int flags)
 {
 	char			*buf;
 	errcode_t		retval;
@@ -271,7 +286,7 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	int			falloc_flags = EXT2_FALLOCATE_FORCE_INIT;
 	blk64_t			zblk;
 
-	if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags,
+	if ((retval = ext2fs_create_journal_superblock2(fs, jparams, flags,
 						       &buf)))
 		return retval;
 
@@ -295,7 +310,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	if (!(flags & EXT2_MKJOURNAL_LAZYINIT))
 		falloc_flags |= EXT2_FALLOCATE_ZERO_BLOCKS;
 
-	inode_size = (unsigned long long)fs->blocksize * num_blocks;
+	inode_size = (unsigned long long)fs->blocksize *
+			(jparams->num_journal_blocks + jparams->num_fc_blocks);
 	inode.i_mtime = inode.i_ctime = fs->now ? fs->now : time(0);
 	inode.i_links_count = 1;
 	inode.i_mode = LINUX_S_IFREG | 0600;
@@ -304,7 +320,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 		goto out2;
 
 	retval = ext2fs_fallocate(fs, falloc_flags, journal_ino,
-				  &inode, goal, 0, num_blocks);
+				  &inode, goal, 0,
+				  jparams->num_journal_blocks + jparams->num_fc_blocks);
 	if (retval)
 		goto out2;
 
@@ -358,6 +375,43 @@ int ext2fs_default_journal_size(__u64 num_blocks)
 	return 262144;				/* 1 GB */
 }
 
+errcode_t ext2fs_get_journal_params(struct ext2fs_journal_params *params,
+		ext2_filsys fs)
+{
+	blk_t total_blks;
+	int ret;
+
+	memset(params, 0, sizeof(*params));
+	if (ext2fs_has_feature_journal_dev(fs->super)) {
+		total_blks = ext2fs_blocks_count(fs->super);
+		if (total_blks < JBD2_MIN_JOURNAL_BLOCKS)
+			return EXT2_ET_JOURNAL_TOO_SMALL;
+
+		if (!ext2fs_has_feature_fast_commit(fs->super)) {
+			params->num_journal_blocks = total_blks;
+			params->num_fc_blocks = 0;
+			return 0;
+		}
+		params->num_journal_blocks = ext2fs_blocks_count(fs->super) *
+				EXT2_JOURNAL_TO_FC_BLKS_RATIO /
+				(EXT2_JOURNAL_TO_FC_BLKS_RATIO + 1);
+		if (JBD2_MIN_JOURNAL_BLOCKS > params->num_journal_blocks)
+			params->num_journal_blocks = JBD2_MIN_JOURNAL_BLOCKS;
+		params->num_fc_blocks = total_blks - params->num_journal_blocks;
+		return 0;
+	}
+
+	ret = ext2fs_default_journal_size(ext2fs_blocks_count(fs->super));
+	if (ret < 0)
+		return EXT2_ET_JOURNAL_TOO_SMALL;
+
+	params->num_journal_blocks = ret;
+	if (ext2fs_has_feature_fast_commit(fs->super))
+		params->num_fc_blocks = params->num_journal_blocks /
+			EXT2_JOURNAL_TO_FC_BLKS_RATIO;
+	return 0;
+}
+
 int ext2fs_journal_sb_start(int blocksize)
 {
 	if (blocksize == EXT2_MIN_BLOCK_SIZE)
@@ -434,7 +488,7 @@ errcode_t ext2fs_add_journal_device(ext2_filsys fs, ext2_filsys journal_dev)
  * POSIX routines if the filesystem is mounted, or using direct I/O
  * functions if it is not.
  */
-errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
+errcode_t ext2fs_add_journal_inode3(ext2_filsys fs, struct ext2fs_journal_params *jparams,
 				    blk64_t goal, int flags)
 {
 	errcode_t		retval;
@@ -486,7 +540,7 @@ errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
 		 * filesystems is extremely rare these days...  Ignore it. */
 		flags &= ~EXT2_MKJOURNAL_LAZYINIT;
 
-		if ((retval = write_journal_file(fs, jfile, num_blocks, flags)))
+		if ((retval = write_journal_file(fs, jfile, jparams, flags)))
 			goto errout;
 
 		/* Get inode number of the journal file */
@@ -528,7 +582,7 @@ errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
 		}
 		journal_ino = EXT2_JOURNAL_INO;
 		if ((retval = write_journal_inode(fs, journal_ino,
-						  num_blocks, goal, flags)))
+						  jparams, goal, flags)))
 			return retval;
 	}
 
@@ -546,6 +600,18 @@ errout:
 	return retval;
 }
 
+errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks,
+				    blk64_t goal, int flags)
+{
+	struct ext2fs_journal_params jparams;
+	errcode_t ret;
+
+	jparams.num_journal_blocks = num_blocks;
+	jparams.num_fc_blocks = 0;
+
+	return ext2fs_add_journal_inode3(fs, &jparams, goal, flags);
+}
+
 errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks, int flags)
 {
 	return ext2fs_add_journal_inode2(fs, num_blocks, ~0ULL, flags);
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 05/15] e2fsprogs: make userspace tools number of fast commits blocks aware
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (3 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 04/15] libext2fs: provide APIs to configure fast commit blocks Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21 15:58   ` Theodore Ts'o
  2021-01-20 21:26 ` [PATCH v3 06/15] ext2fs: add new APIs needed for fast commits Harshad Shirwadkar
                   ` (10 subsequent siblings)
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

This patch makes number of fast commit blocks configurable. Also, the
number of fast commit blocks can now be seen in dumpe2fs output.

$ ./misc/mke2fs -O fast_commit -t ext4 image
mke2fs 1.46-WIP (20-Mar-2020)
Discarding device blocks: done
Creating filesystem with 5120 1k blocks and 1280 inodes
Allocating group tables: done
Writing inode tables: done
Creating journal (1040 blocks): done
Writing superblocks and filesystem accounting information: done

$ ./misc/dumpe2fs image
dumpe2fs 1.46-WIP (20-Mar-2020)
...
Journal features:         (none)
Total journal size:       1040k
Total journal blocks:     1040
Max transaction length:   1024
Fast commit length:       16
Journal sequence:         0x00000001
Journal start:            0

$ ./misc/mke2fs -O fast_commit -t ext4 image -J fast_commit_size=256,size=1
mke2fs 1.46-WIP (20-Mar-2020)
Creating filesystem with 5120 1k blocks and 1280 inodes
Allocating group tables: done
Writing inode tables: done
Creating journal (1280 blocks): done
Writing superblocks and filesystem accounting information: done

$ ./misc/dumpe2fs image
dumpe2fs 1.46-WIP (20-Mar-2020)
...
Journal features:         (none)
Total journal size:       1280k
Total journal blocks:     1280
Max transaction length:   1024
Fast commit length:       256
Journal sequence:         0x00000001
Journal start:            0

This patch also adds information about fast commit feature in mke2fs
and tune2fs man pages.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 e2fsck/unix.c     | 26 +++++++++++++-------
 misc/dumpe2fs.c   | 10 ++++++--
 misc/mke2fs.8.in  | 21 ++++++++++++++++
 misc/mke2fs.c     | 26 +++++++++++++-------
 misc/tune2fs.8.in | 25 +++++++++++++++++++
 misc/tune2fs.c    |  8 +++----
 misc/util.c       | 61 ++++++++++++++++++++++++++++++++++-------------
 misc/util.h       |  4 +++-
 8 files changed, 142 insertions(+), 39 deletions(-)

diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 1cb51672..bf66605a 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -1394,6 +1394,7 @@ int main (int argc, char *argv[])
 	__u32 features[3];
 	char *cp;
 	enum quota_type qtype;
+	struct ext2fs_journal_params jparams;
 
 	clear_problem_context(&pctx);
 	sigcatcher_setup();
@@ -1887,9 +1888,15 @@ print_unsupp_features:
 	/*
 	 * Save the journal size in megabytes.
 	 * Try and use the journal size from the backup else let e2fsck
-	 * find the default journal size.
+	 * find the default journal size. If fast commit feature is enabled,
+	 * it is not clear how many of the journal blocks were fast commit
+	 * blocks. So, ignore the size of journal found in backup.
+	 *
+	 * TODO: Add a new backup type that captures fast commit info as
+	 * well.
 	 */
-	if (sb->s_jnl_backup_type == EXT3_JNL_BACKUP_BLOCKS)
+	if (sb->s_jnl_backup_type == EXT3_JNL_BACKUP_BLOCKS &&
+		!ext2fs_has_feature_fast_commit(sb))
 		journal_size = (sb->s_jnl_blocks[15] << (32 - 20)) |
 			       (sb->s_jnl_blocks[16] >> 20);
 	else
@@ -1911,9 +1918,13 @@ print_unsupp_features:
 	if (!ctx->invalid_bitmaps &&
 	    (ctx->flags & E2F_FLAG_JOURNAL_INODE)) {
 		if (fix_problem(ctx, PR_6_RECREATE_JOURNAL, &pctx)) {
-			if (journal_size < 1024)
-				journal_size = ext2fs_default_journal_size(ext2fs_blocks_count(fs->super));
-			if (journal_size < 0) {
+			if (journal_size < 1024) {
+				ext2fs_get_journal_params(&jparams, fs);
+			} else {
+				jparams.num_journal_blocks = journal_size;
+				jparams.num_fc_blocks = 0;
+			}
+			if (jparams.num_journal_blocks < 0) {
 				ext2fs_clear_feature_journal(fs->super);
 				fs->flags &= ~EXT2_FLAG_MASTER_SB_ONLY;
 				log_out(ctx, "%s: Couldn't determine "
@@ -1921,10 +1932,9 @@ print_unsupp_features:
 				goto no_journal;
 			}
 			log_out(ctx, _("Creating journal (%d blocks): "),
-			       journal_size);
+			       jparams.num_journal_blocks);
 			fflush(stdout);
-			retval = ext2fs_add_journal_inode(fs,
-							  journal_size, 0);
+			retval = ext2fs_add_journal_inode3(fs, &jparams, ~0ULL, 0);
 			if (retval) {
 				log_out(ctx, "%s: while trying to create "
 					"journal\n", error_message(retval));
diff --git a/misc/dumpe2fs.c b/misc/dumpe2fs.c
index d295ba4d..e24dc4e6 100644
--- a/misc/dumpe2fs.c
+++ b/misc/dumpe2fs.c
@@ -364,6 +364,7 @@ static void print_inline_journal_information(ext2_filsys fs)
 	errcode_t		retval;
 	ext2_ino_t		ino = fs->super->s_journal_inum;
 	char			buf[1024];
+	int			flags;
 
 	if (fs->flags & EXT2_FLAG_IMAGE_FILE)
 		return;
@@ -392,7 +393,9 @@ static void print_inline_journal_information(ext2_filsys fs)
 			_("Journal superblock magic number invalid!\n"));
 		exit(1);
 	}
-	e2p_list_journal_super(stdout, buf, fs->blocksize, 0);
+	flags = ext2fs_has_feature_fast_commit(fs->super) ?
+			E2P_LIST_JOURNAL_FLAG_FC : 0;
+	e2p_list_journal_super(stdout, buf, fs->blocksize, flags);
 }
 
 static void print_journal_information(ext2_filsys fs)
@@ -400,6 +403,7 @@ static void print_journal_information(ext2_filsys fs)
 	errcode_t	retval;
 	char		buf[1024];
 	journal_superblock_t	*jsb;
+	int		flags;
 
 	/* Get the journal superblock */
 	if ((retval = io_channel_read_blk64(fs->io,
@@ -417,7 +421,9 @@ static void print_journal_information(ext2_filsys fs)
 			_("Couldn't find journal superblock magic numbers"));
 		exit(1);
 	}
-	e2p_list_journal_super(stdout, buf, fs->blocksize, 0);
+	flags = ext2fs_has_feature_fast_commit(fs->super) ?
+			E2P_LIST_JOURNAL_FLAG_FC : 0;
+	e2p_list_journal_super(stdout, buf, fs->blocksize, flags);
 }
 
 static int check_mmp(ext2_filsys fs)
diff --git a/misc/mke2fs.8.in b/misc/mke2fs.8.in
index e6bfc6d6..2833b408 100644
--- a/misc/mke2fs.8.in
+++ b/misc/mke2fs.8.in
@@ -521,6 +521,27 @@ The size of the journal must be at least 1024 filesystem blocks
 and may be no more than 10,240,000 filesystem blocks or half the total
 file system size (whichever is smaller)
 .TP
+.BI fast_commit_size= fast-commit-size
+Create an additional fast commit journal area of size
+.I fast-commit-size
+kilobytes.
+This option is only valid if
+.B fast_commit
+feature is enabled
+on the file system. If this option is not specified and if
+.B fast_commit
+feature is turned on, fast commit area size defaults to
+.I journal-size
+/ 64 megabytes. The total size of the journal with
+.B fast_commit
+feature set is
+.I journal-size
++ (
+.I fast-commit-size
+* 1024) megabytes. The total journal size may be no more than
+10,240,000 filesystem blocks or half the total file system size
+(whichever is smaller).
+.TP
 .BI location =journal-location
 Specify the location of the journal.  The argument
 .I journal-location
diff --git a/misc/mke2fs.c b/misc/mke2fs.c
index 8c8f5ea4..d7ce7c9d 100644
--- a/misc/mke2fs.c
+++ b/misc/mke2fs.c
@@ -93,6 +93,7 @@ static uid_t	root_uid;
 static gid_t	root_gid;
 int	journal_size;
 int	journal_flags;
+int	journal_fc_size;
 static int	lazy_itable_init;
 static int	packed_meta_blocks;
 int		no_copy_xattrs;
@@ -604,9 +605,16 @@ static void create_journal_dev(ext2_filsys fs)
 	char			*buf;
 	blk64_t			blk, err_blk;
 	int			c, count, err_count;
+	struct ext2fs_journal_params	jparams;
 
-	retval = ext2fs_create_journal_superblock(fs,
-				  ext2fs_blocks_count(fs->super), 0, &buf);
+	retval = ext2fs_get_journal_params(&jparams, fs);
+	if (retval) {
+		com_err("create_journal_dev", retval, "%s",
+			_("while splitting the journal size"));
+		exit(1);
+	}
+
+	retval = ext2fs_create_journal_superblock2(fs, &jparams, 0, &buf);
 	if (retval) {
 		com_err("create_journal_dev", retval, "%s",
 			_("while initializing journal superblock"));
@@ -1753,6 +1761,8 @@ profile_error:
 		case 'j':
 			if (!journal_size)
 				journal_size = -1;
+			if (!journal_fc_size)
+				journal_fc_size = -1;
 			break;
 		case 'J':
 			parse_journal_opts(optarg);
@@ -2937,7 +2947,7 @@ int main (int argc, char *argv[])
 	badblocks_list	bb_list = 0;
 	badblocks_iterate	bb_iter;
 	blk_t		blk;
-	unsigned int	journal_blocks = 0;
+	struct ext2fs_journal_params	jparams = {0};
 	unsigned int	i, checkinterval;
 	int		max_mnt_count;
 	int		val, hash_alg;
@@ -3047,7 +3057,7 @@ int main (int argc, char *argv[])
 	/* Calculate journal blocks */
 	if (!journal_device && ((journal_size) ||
 	    ext2fs_has_feature_journal(&fs_param)))
-		journal_blocks = figure_journal_size(journal_size, fs);
+		figure_journal_size(&jparams, journal_size, journal_fc_size, fs);
 
 	sprintf(opt_string, "tdb_data_size=%d", fs->blocksize <= 4096 ?
 		32768 : fs->blocksize * 8);
@@ -3382,23 +3392,23 @@ int main (int argc, char *argv[])
 		free(journal_device);
 	} else if ((journal_size) ||
 		   ext2fs_has_feature_journal(&fs_param)) {
-		overhead += EXT2FS_NUM_B2C(fs, journal_blocks);
+		overhead += EXT2FS_NUM_B2C(fs, jparams.num_journal_blocks + jparams.num_fc_blocks);
 		if (super_only) {
 			printf("%s", _("Skipping journal creation in super-only mode\n"));
 			fs->super->s_journal_inum = EXT2_JOURNAL_INO;
 			goto no_journal;
 		}
 
-		if (!journal_blocks) {
+		if (!jparams.num_journal_blocks) {
 			ext2fs_clear_feature_journal(fs->super);
 			goto no_journal;
 		}
 		if (!quiet) {
 			printf(_("Creating journal (%u blocks): "),
-			       journal_blocks);
+			       jparams.num_journal_blocks + jparams.num_fc_blocks);
 			fflush(stdout);
 		}
-		retval = ext2fs_add_journal_inode2(fs, journal_blocks,
+		retval = ext2fs_add_journal_inode3(fs, &jparams,
 						   journal_location,
 						   journal_flags);
 		if (retval) {
diff --git a/misc/tune2fs.8.in b/misc/tune2fs.8.in
index 582d1da5..2114c623 100644
--- a/misc/tune2fs.8.in
+++ b/misc/tune2fs.8.in
@@ -357,6 +357,27 @@ and may be no more than 10,240,000 filesystem blocks.
 There must be enough free space in the filesystem to create a journal of
 that size.
 .TP
+.BI fast_commit_size= fast-commit-size
+Create an additional fast commit journal area of size
+.I fast-commit-size
+kilobytes.
+This option is only valid if
+.B fast_commit
+feature is enabled
+on the file system. If this option is not specified and if
+.B fast_commit
+feature is turned on, fast commit area size defaults to
+.I journal-size
+/ 64 megabytes. The total size of the journal with
+.B fast_commit
+feature set is
+.I journal-size
++ (
+.I fast-commit-size
+* 1024) megabytes. The total journal size may be no more than
+10,240,000 filesystem blocks or half the total file system size
+(whichever is smaller).
+.TP
 .BI location =journal-location
 Specify the location of the journal.  The argument
 .I journal-location
@@ -586,6 +607,10 @@ Setting the filesystem feature is equivalent to using the
 .B \-j
 option.
 .TP
+.TP
+.B fast_commit
+Enable fast commit journaling feature to improve fsync latency.
+.TP
 .B large_dir
 Increase the limit on the number of files per directory.
 .B Tune2fs
diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index 670ed9e0..2307f18a 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -102,7 +102,7 @@ static int feature_64bit;
 static int fsck_requested;
 static char *undo_file;
 
-int journal_size, journal_flags;
+int journal_size, journal_fc_size, journal_flags;
 char *journal_device;
 static blk64_t journal_location = ~0LL;
 
@@ -1543,7 +1543,7 @@ mmp_error:
  */
 static int add_journal(ext2_filsys fs)
 {
-	unsigned long journal_blocks;
+	struct ext2fs_journal_params	jparams;
 	errcode_t	retval;
 	ext2_filsys	jfs;
 	io_manager	io_ptr;
@@ -1589,13 +1589,13 @@ static int add_journal(ext2_filsys fs)
 	} else if (journal_size) {
 		fputs(_("Creating journal inode: "), stdout);
 		fflush(stdout);
-		journal_blocks = figure_journal_size(journal_size, fs);
+		figure_journal_size(&jparams, journal_size, journal_fc_size, fs);
 
 		if (journal_location_string)
 			journal_location =
 				parse_num_blocks2(journal_location_string,
 						  fs->super->s_log_block_size);
-		retval = ext2fs_add_journal_inode2(fs, journal_blocks,
+		retval = ext2fs_add_journal_inode3(fs, &jparams,
 						   journal_location,
 						   journal_flags);
 		if (retval) {
diff --git a/misc/util.c b/misc/util.c
index dcd2f0a7..48e623dc 100644
--- a/misc/util.c
+++ b/misc/util.c
@@ -200,6 +200,14 @@ void parse_journal_opts(const char *opts)
 			journal_size = strtoul(arg, &p, 0);
 			if (*p)
 				journal_usage++;
+		} else if (strcmp(token, "fast_commit_size") == 0) {
+			if (!arg) {
+				journal_usage++;
+				continue;
+			}
+			journal_fc_size = strtoul(arg, &p, 0);
+			if (*p)
+				journal_usage++;
 		} else if (!strcmp(token, "location")) {
 			if (!arg) {
 				journal_usage++;
@@ -229,42 +237,63 @@ void parse_journal_opts(const char *opts)
 	free(buf);
 }
 
+static inline int jsize_to_blks(ext2_filsys fs, int size)
+{
+	return (size * 1024) / (fs->blocksize / 1024);
+}
+
+/* Fast commit size is in KBs */
+static inline int fcsize_to_blks(ext2_filsys fs, int size)
+{
+	return (size * 1024) / (fs->blocksize);
+}
+
 /*
  * Determine the number of journal blocks to use, either via
  * user-specified # of megabytes, or via some intelligently selected
  * defaults.
  *
- * Find a reasonable journal file size (in blocks) given the number of blocks
- * in the filesystem.  For very small filesystems, it is not reasonable to
- * have a journal that fills more than half of the filesystem.
+ * Find a reasonable journal file size (in blocks) given the number of blocks in
+ * the filesystem. For very small filesystems, it is not reasonable to have a
+ * journal that fills more than half of the filesystem.
  */
-unsigned int figure_journal_size(int size, ext2_filsys fs)
+void figure_journal_size(struct ext2fs_journal_params *jparams,
+		int requested_j_size, int requested_fc_size, ext2_filsys fs)
 {
-	int j_blocks;
+	int total_blocks, ret;
 
-	j_blocks = ext2fs_default_journal_size(ext2fs_blocks_count(fs->super));
-	if (j_blocks < 0) {
+	ret = ext2fs_get_journal_params(jparams, fs);
+	if (ret) {
 		fputs(_("\nFilesystem too small for a journal\n"), stderr);
-		return 0;
+		return;
 	}
 
-	if (size > 0) {
-		j_blocks = size * 1024 / (fs->blocksize	/ 1024);
-		if (j_blocks < 1024 || j_blocks > 10240000) {
-			fprintf(stderr, _("\nThe requested journal "
+	if (requested_j_size > 0 ||
+		(ext2fs_has_feature_fast_commit(fs->super) && requested_fc_size > 0)) {
+		if (requested_j_size > 0)
+			jparams->num_journal_blocks =
+				jsize_to_blks(fs, requested_j_size);
+		if (ext2fs_has_feature_fast_commit(fs->super) &&
+			requested_fc_size > 0)
+			jparams->num_fc_blocks =
+				fcsize_to_blks(fs, requested_fc_size);
+		else if (!ext2fs_has_feature_fast_commit(fs->super))
+			jparams->num_fc_blocks = 0;
+		total_blocks = jparams->num_journal_blocks + jparams->num_fc_blocks;
+		if (total_blocks < 1024 || total_blocks > 10240000) {
+			fprintf(stderr, _("\nThe total requested journal "
 				"size is %d blocks; it must be\n"
 				"between 1024 and 10240000 blocks.  "
 				"Aborting.\n"),
-				j_blocks);
+				total_blocks);
 			exit(1);
 		}
-		if ((unsigned) j_blocks > ext2fs_free_blocks_count(fs->super) / 2) {
-			fputs(_("\nJournal size too big for filesystem.\n"),
+		if ((unsigned int) total_blocks > ext2fs_free_blocks_count(fs->super) / 2) {
+			fputs(_("\nTotal journal size too big for filesystem.\n"),
 			      stderr);
 			exit(1);
 		}
 	}
-	return j_blocks;
 }
 
 void print_check_message(int mnt, unsigned int check)
diff --git a/misc/util.h b/misc/util.h
index 49b4b9c1..ccdc7fbc 100644
--- a/misc/util.h
+++ b/misc/util.h
@@ -11,6 +11,7 @@
  */
 
 extern int	 journal_size;
+extern int	 journal_fc_size;
 extern int	 journal_flags;
 extern char	*journal_device;
 extern char	*journal_location_string;
@@ -22,6 +23,7 @@ extern char *get_progname(char *argv_zero);
 extern void proceed_question(int delay);
 extern void parse_journal_opts(const char *opts);
 extern void check_mount(const char *device, int force, const char *type);
-extern unsigned int figure_journal_size(int size, ext2_filsys fs);
+extern void figure_journal_size(struct ext2fs_journal_params *jparams,
+		int requested_j_size, int requested_fc_size, ext2_filsys fs);
 extern void print_check_message(int, unsigned int);
 extern void dump_mmp_msg(struct mmp_struct *mmp, const char *msg);
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 06/15] ext2fs: add new APIs needed for fast commits
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (4 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 05/15] e2fsprogs: make userspace tools number of fast commits blocks aware Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21  5:58   ` Theodore Ts'o
  2021-01-20 21:26 ` [PATCH v3 07/15] e2fsck: add function to rewrite extent tree Harshad Shirwadkar
                   ` (9 subsequent siblings)
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

This patch adds the following new APIs:

Count the total number of blocks occupied by inode including
intermediate extent tree nodes.
extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
                                       struct ext2_inode *inode);

Convert ext3_extent to ext2fs_extent.
extern void ext2fs_convert_extent(struct ext2fs_extent *to,
                                       struct ext3_extent *from);

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 lib/ext2fs/ext2fs.h |  4 +++
 lib/ext2fs/extent.c | 63 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)

diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index ec841006..fdcb28f6 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1332,6 +1332,10 @@ extern errcode_t ext2fs_extent_fix_parents(ext2_extent_handle_t handle);
 extern size_t ext2fs_max_extent_depth(ext2_extent_handle_t handle);
 extern errcode_t ext2fs_fix_extents_checksums(ext2_filsys fs, ext2_ino_t ino,
 					      struct ext2_inode *inode);
+extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
+					struct ext2_inode *inode);
+extern errcode_t ext2fs_decode_extent(struct ext2fs_extent *to, void *from,
+				      int len);
 
 /* fallocate.c */
 #define EXT2_FALLOCATE_ZERO_BLOCKS	(0x1)
diff --git a/lib/ext2fs/extent.c b/lib/ext2fs/extent.c
index ac3dbfec..8d5fc1ab 100644
--- a/lib/ext2fs/extent.c
+++ b/lib/ext2fs/extent.c
@@ -1785,6 +1785,69 @@ out:
 	return errcode;
 }
 
+errcode_t ext2fs_decode_extent(struct ext2fs_extent *to, void *addr, int len)
+{
+	struct ext3_extent *from = (struct ext3_extent *)addr;
+
+	if (len != sizeof(struct ext3_extent))
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	to->e_pblk = ext2fs_le32_to_cpu(from->ee_start) +
+		((__u64) ext2fs_le16_to_cpu(from->ee_start_hi)
+			<< 32);
+	to->e_lblk = ext2fs_le32_to_cpu(from->ee_block);
+	to->e_len = ext2fs_le16_to_cpu(from->ee_len);
+	to->e_flags |= EXT2_EXTENT_FLAGS_LEAF;
+	if (to->e_len > EXT_INIT_MAX_LEN) {
+		to->e_len -= EXT_INIT_MAX_LEN;
+		to->e_flags |= EXT2_EXTENT_FLAGS_UNINIT;
+	}
+
+	return 0;
+}
+
+blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
+			struct ext2_inode *inode)
+{
+	ext2_extent_handle_t	handle;
+	struct ext2fs_extent	extent;
+	errcode_t		errcode;
+	int			i;
+	blk64_t			blkcount = 0;
+	blk64_t			*intermediate_nodes;
+
+	errcode = ext2fs_extent_open2(fs, ino, inode, &handle);
+	if (errcode)
+		goto out;
+
+	errcode = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
+	if (errcode)
+		goto out;
+
+	ext2fs_get_array(handle->max_depth, sizeof(blk64_t),
+				&intermediate_nodes);
+	blkcount = handle->level;
+	while (!errcode) {
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) {
+			blkcount += extent.e_len;
+			for (i = 0; i < handle->level; i++) {
+				if (intermediate_nodes[i] !=
+					handle->path[i].end_blk) {
+					blkcount++;
+					intermediate_nodes[i] =
+						handle->path[i].end_blk;
+				}
+			}
+		}
+		errcode = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent);
+	}
+	ext2fs_free_mem(&intermediate_nodes);
+out:
+	ext2fs_extent_free(handle);
+
+	return blkcount;
+}
+
 #ifdef DEBUG
 /*
  * Override debugfs's prompt
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 07/15] e2fsck: add function to rewrite extent tree
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (5 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 06/15] ext2fs: add new APIs needed for fast commits Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 08/15] e2fsck: add fast commit setup code Harshad Shirwadkar
                   ` (8 subsequent siblings)
  15 siblings, 0 replies; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Fast commit replay needs to rewrite the entire extent tree for inodes
found in fast commit area. This patch makes e2fsck's rewrite extent
tree path visible.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
---
 e2fsck/e2fsck.h  |  16 +++++
 e2fsck/extents.c | 168 ++++++++++++++++++++++++++++++-----------------
 2 files changed, 124 insertions(+), 60 deletions(-)

diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 85f953b2..3b9c1874 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -226,6 +226,19 @@ typedef struct e2fsck_struct *e2fsck_t;
 
 #define MAX_EXTENT_DEPTH_COUNT 5
 
+/*
+ * This strucutre is used to manage the list of extents in a file. Placing
+ * it here since this is used by fast_commit.h.
+ */
+struct extent_list {
+	blk64_t blocks_freed;
+	struct ext2fs_extent *extents;
+	unsigned int count;
+	unsigned int size;
+	unsigned int ext_read;
+	errcode_t retval;
+	ext2_ino_t ino;
+};
 struct e2fsck_struct {
 	ext2_filsys fs;
 	const char *program_name;
@@ -536,6 +549,9 @@ errcode_t e2fsck_should_rebuild_extents(e2fsck_t ctx,
 					struct problem_context *pctx,
 					struct extent_tree_info *eti,
 					struct ext2_extent_info *info);
+errcode_t e2fsck_read_extents(e2fsck_t ctx, struct extent_list *extents);
+errcode_t e2fsck_rewrite_extent_tree(e2fsck_t ctx,
+				     struct extent_list *extents);
 
 /* journal.c */
 extern errcode_t e2fsck_check_ext3_journal(e2fsck_t ctx);
diff --git a/e2fsck/extents.c b/e2fsck/extents.c
index e9139326..d6c74834 100644
--- a/e2fsck/extents.c
+++ b/e2fsck/extents.c
@@ -58,16 +58,6 @@ int e2fsck_ino_will_be_rebuilt(e2fsck_t ctx, ext2_ino_t ino)
 	return ext2fs_test_inode_bitmap2(ctx->inodes_to_rebuild, ino);
 }
 
-struct extent_list {
-	blk64_t blocks_freed;
-	struct ext2fs_extent *extents;
-	unsigned int count;
-	unsigned int size;
-	unsigned int ext_read;
-	errcode_t retval;
-	ext2_ino_t ino;
-};
-
 static errcode_t load_extents(e2fsck_t ctx, struct extent_list *list)
 {
 	ext2_filsys		fs = ctx->fs;
@@ -206,66 +196,40 @@ static int find_blocks(ext2_filsys fs, blk64_t *blocknr, e2_blkcnt_t blockcnt,
 	return 0;
 }
 
-static errcode_t rebuild_extent_tree(e2fsck_t ctx, struct extent_list *list,
-				     ext2_ino_t ino)
+errcode_t rewrite_extent_replay(e2fsck_t ctx, struct extent_list *list,
+				       struct ext2_inode_large *inode)
 {
-	struct ext2_inode_large	inode;
 	errcode_t		retval;
 	ext2_extent_handle_t	handle;
 	unsigned int		i, ext_written;
 	struct ext2fs_extent	*ex, extent;
-	blk64_t			start_val, delta;
-
-	list->count = 0;
-	list->blocks_freed = 0;
-	list->ino = ino;
-	list->ext_read = 0;
-	e2fsck_read_inode_full(ctx, ino, EXT2_INODE(&inode), sizeof(inode),
-			       "rebuild_extents");
+	blk64_t			start_val, delta, blkcount;
 
-	/* Skip deleted inodes and inline data files */
-	if (inode.i_links_count == 0 ||
-	    inode.i_flags & EXT4_INLINE_DATA_FL)
-		return 0;
-
-	/* Collect lblk->pblk mappings */
-	if (inode.i_flags & EXT4_EXTENTS_FL) {
-		retval = load_extents(ctx, list);
-		if (retval)
-			goto err;
-		goto extents_loaded;
-	}
-
-	retval = ext2fs_block_iterate3(ctx->fs, ino, BLOCK_FLAG_READ_ONLY, 0,
-				       find_blocks, list);
-	if (retval)
-		goto err;
-	if (list->retval) {
-		retval = list->retval;
-		goto err;
-	}
-
-extents_loaded:
 	/* Reset extent tree */
-	inode.i_flags &= ~EXT4_EXTENTS_FL;
-	memset(inode.i_block, 0, sizeof(inode.i_block));
+	inode->i_flags &= ~EXT4_EXTENTS_FL;
+	memset(inode->i_block, 0, sizeof(inode->i_block));
 
 	/* Make a note of freed blocks */
-	quota_data_sub(ctx->qctx, &inode, ino,
+	quota_data_sub(ctx->qctx, inode, list->ino,
 		       list->blocks_freed * ctx->fs->blocksize);
-	retval = ext2fs_iblk_sub_blocks(ctx->fs, EXT2_INODE(&inode),
+	retval = ext2fs_iblk_sub_blocks(ctx->fs, EXT2_INODE(inode),
 					list->blocks_freed);
 	if (retval)
-		goto err;
+		return retval;
 
 	/* Now stuff extents into the file */
-	retval = ext2fs_extent_open2(ctx->fs, ino, EXT2_INODE(&inode), &handle);
+	retval = ext2fs_extent_open2(ctx->fs, list->ino, EXT2_INODE(inode),
+					&handle);
 	if (retval)
-		goto err;
+		return retval;
 
 	ext_written = 0;
-	start_val = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(&inode));
+
+	start_val = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(inode));
+
 	for (i = 0, ex = list->extents; i < list->count; i++, ex++) {
+		if (ex->e_len == 0)
+			continue;
 		memcpy(&extent, ex, sizeof(struct ext2fs_extent));
 		extent.e_flags &= EXT2_EXTENT_FLAGS_UNINIT;
 		if (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) {
@@ -289,36 +253,120 @@ extents_loaded:
 		}
 
 #ifdef DEBUG
-		printf("W: ino=%d pblk=%llu lblk=%llu len=%u\n", ino,
+		printf("W: ino=%d pblk=%llu lblk=%llu len=%u\n", list->ino,
 				extent.e_pblk, extent.e_lblk, extent.e_len);
 #endif
 		retval = ext2fs_extent_insert(handle, EXT2_EXTENT_INSERT_AFTER,
 					      &extent);
 		if (retval)
-			goto err2;
+			goto err;
 		retval = ext2fs_extent_fix_parents(handle);
 		if (retval)
-			goto err2;
+			goto err;
 		ext_written++;
 	}
 
-	delta = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(&inode)) -
+	delta = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(inode)) -
 		start_val;
 	if (delta)
-		quota_data_add(ctx->qctx, &inode, ino, delta << 9);
+		quota_data_add(ctx->qctx, inode, list->ino, delta << 9);
 
 #if defined(DEBUG) || defined(DEBUG_SUMMARY)
 	printf("rebuild: ino=%d extents=%d->%d\n", ino, list->ext_read,
 	       ext_written);
 #endif
-	e2fsck_write_inode(ctx, ino, EXT2_INODE(&inode), "rebuild_extents");
+	e2fsck_write_inode(ctx, list->ino, EXT2_INODE(inode),
+				"rebuild_extents");
 
-err2:
-	ext2fs_extent_free(handle);
 err:
+	ext2fs_extent_free(handle);
+	return retval;
+}
+
+errcode_t e2fsck_rewrite_extent_tree(e2fsck_t ctx, struct extent_list *list)
+{
+	struct ext2_inode_large inode;
+	int ret;
+
+	memset(&inode, 0, sizeof(inode));
+	ext2fs_read_inode_full(ctx->fs, list->ino, EXT2_INODE(&inode),
+				sizeof(inode));
+
+	/* Skip deleted inodes and inline data files */
+	if (inode.i_flags & EXT4_INLINE_DATA_FL)
+		return 0;
+
+	ret = rewrite_extent_replay(ctx, list, &inode);
+	ext2fs_iblk_set(ctx->fs, EXT2_INODE(&inode),
+		ext2fs_count_blocks(ctx->fs, list->ino, EXT2_INODE(&inode)));
+	ext2fs_write_inode_full(ctx->fs, list->ino, EXT2_INODE(&inode),
+		sizeof(inode));
+
+	return ret;
+}
+
+errcode_t e2fsck_read_extents(e2fsck_t ctx, struct extent_list *extents)
+{
+	struct ext2_inode_large	inode;
+	errcode_t		retval;
+
+	extents->extents = NULL;
+	extents->count = 0;
+	extents->blocks_freed = 0;
+	extents->ext_read = 0;
+	extents->size = NUM_EXTENTS;
+	retval = ext2fs_get_array(NUM_EXTENTS, sizeof(struct ext2fs_extent),
+				  &extents->extents);
+	if (retval)
+		return -ENOMEM;
+
+	retval = ext2fs_read_inode(ctx->fs, extents->ino, EXT2_INODE(&inode));
+	if (retval)
+		goto err_out;
+
+	retval = load_extents(ctx, extents);
+	if (!retval)
+		return 0;
+err_out:
+	ext2fs_free_mem(&extents->extents);
+	extents->size = 0;
+	extents->count = 0;
 	return retval;
 }
 
+static errcode_t rebuild_extent_tree(e2fsck_t ctx, struct extent_list *list,
+				     ext2_ino_t ino)
+{
+	struct ext2_inode_large	inode;
+	errcode_t		retval;
+
+	list->count = 0;
+	list->blocks_freed = 0;
+	list->ino = ino;
+	list->ext_read = 0;
+	e2fsck_read_inode_full(ctx, ino, EXT2_INODE(&inode), sizeof(inode),
+			       "rebuild_extents");
+
+	/* Skip deleted inodes and inline data files */
+	if (inode.i_links_count == 0 ||
+	    inode.i_flags & EXT4_INLINE_DATA_FL)
+		return 0;
+
+	/* Collect lblk->pblk mappings */
+	if (inode.i_flags & EXT4_EXTENTS_FL) {
+		retval = load_extents(ctx, list);
+		if (retval)
+			return retval;
+		return rewrite_extent_replay(ctx, list, &inode);
+	}
+
+	retval = ext2fs_block_iterate3(ctx->fs, ino, BLOCK_FLAG_READ_ONLY, 0,
+				       find_blocks, list);
+
+	return retval || list->retval ||
+		rewrite_extent_replay(ctx, list, &inode);
+}
+
 /* Rebuild the extents immediately */
 static errcode_t e2fsck_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino)
 {
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 08/15] e2fsck: add fast commit setup code
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (6 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 07/15] e2fsck: add function to rewrite extent tree Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21 16:54   ` Theodore Ts'o
  2021-01-20 21:26 ` [PATCH v3 09/15] e2fsck: add fast commit scan pass Harshad Shirwadkar
                   ` (7 subsequent siblings)
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Add fast_commit.h that contains the necessary helpers needed for fast
commit replay. Note that this file is also byte by byte identical with
kernel's fast_commit.h. Also, we introduce the
"e2fsck_fc_replay_state" structure which is needed for ext4 fast
commit replay.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
---
 e2fsck/e2fsck.h          |  16 +++
 e2fsck/journal.c         |  15 +++
 lib/ext2fs/ext2_fs.h     |   1 +
 lib/ext2fs/ext2fs.h      |   3 +
 lib/ext2fs/fast_commit.h | 203 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 238 insertions(+)
 create mode 100644 lib/ext2fs/fast_commit.h

diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 3b9c1874..f75cc343 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -68,6 +68,7 @@
 #endif
 
 #include "support/quotaio.h"
+#include "ext2fs/fast_commit.h"
 
 /*
  * Exit codes used by fsck-type programs
@@ -239,6 +240,18 @@ struct extent_list {
 	errcode_t retval;
 	ext2_ino_t ino;
 };
+
+/* State structure for fast commit replay */
+struct e2fsck_fc_replay_state {
+	struct extent_list fc_extent_list;
+	int fc_replay_num_tags;
+	int fc_replay_expected_off;
+	int fc_current_pass;
+	int fc_cur_tag;
+	int fc_crc;
+	__u16 fc_super_state;
+};
+
 struct e2fsck_struct {
 	ext2_filsys fs;
 	const char *program_name;
@@ -431,6 +444,9 @@ struct e2fsck_struct {
 
 	/* Undo file */
 	char *undo_file;
+
+	/* Fast commit replay state */
+	struct e2fsck_fc_replay_state fc_replay_state;
 };
 
 /* Data structures to evaluate whether an extent tree needs rebuilding. */
diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 2adef76a..56e48385 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -278,6 +278,17 @@ static int process_journal_block(ext2_filsys fs,
 	return 0;
 }
 
+/*
+ * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE
+ * to indicate that it is expecting more fast commit blocks. It returns
+ * JBD2_FC_REPLAY_STOP to indicate that replay is done.
+ */
+static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
+				enum passtype pass, int off, tid_t expected_tid)
+{
+	return JBD2_FC_REPLAY_STOP;
+}
+
 static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal)
 {
 	struct process_block_struct pb;
@@ -514,6 +525,10 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal)
 
 	journal->j_sb_buffer = bh;
 	journal->j_superblock = (journal_superblock_t *)bh->b_data;
+	if (ext2fs_has_feature_fast_commit(ctx->fs->super))
+		journal->j_fc_replay_callback = ext4_fc_replay;
+	else
+		journal->j_fc_replay_callback = NULL;
 
 #ifdef USE_INODE_IO
 	if (j_inode)
diff --git a/lib/ext2fs/ext2_fs.h b/lib/ext2fs/ext2_fs.h
index bfc30c29..b1e4329c 100644
--- a/lib/ext2fs/ext2_fs.h
+++ b/lib/ext2fs/ext2_fs.h
@@ -543,6 +543,7 @@ struct ext2_inode *EXT2_INODE(struct ext2_inode_large *large_inode)
 #define EXT2_VALID_FS			0x0001	/* Unmounted cleanly */
 #define EXT2_ERROR_FS			0x0002	/* Errors detected */
 #define EXT3_ORPHAN_FS			0x0004	/* Orphans being recovered */
+#define EXT4_FC_REPLAY			0x0020	/* Ext4 fast commit replay ongoing */
 
 /*
  * Misc. filesystem flags
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index fdcb28f6..eb2e6549 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -2148,4 +2148,7 @@ static inline unsigned int ext2_dir_htree_level(ext2_filsys fs)
 }
 #endif
 
+/* Commonly used helpers */
+#define max(a, b) ((a) > (b) ? (a) : (b))
+
 #endif /* _EXT2FS_EXT2FS_H */
diff --git a/lib/ext2fs/fast_commit.h b/lib/ext2fs/fast_commit.h
new file mode 100644
index 00000000..b83e1810
--- /dev/null
+++ b/lib/ext2fs/fast_commit.h
@@ -0,0 +1,203 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __FAST_COMMIT_H__
+#define __FAST_COMMIT_H__
+
+#include "jfs_compat.h"
+
+/*
+ * Note this file is present in e2fsprogs/lib/ext2fs/fast_commit.h and
+ * linux/fs/ext4/fast_commit.h. These file should always be byte identical.
+ */
+
+/* Fast commit tags */
+#define EXT4_FC_TAG_ADD_RANGE		0x0001
+#define EXT4_FC_TAG_DEL_RANGE		0x0002
+#define EXT4_FC_TAG_CREAT		0x0003
+#define EXT4_FC_TAG_LINK		0x0004
+#define EXT4_FC_TAG_UNLINK		0x0005
+#define EXT4_FC_TAG_INODE		0x0006
+#define EXT4_FC_TAG_PAD			0x0007
+#define EXT4_FC_TAG_TAIL		0x0008
+#define EXT4_FC_TAG_HEAD		0x0009
+
+#define EXT4_FC_SUPPORTED_FEATURES	0x0
+
+/* On disk fast commit tlv value structures */
+
+/* Fast commit on disk tag length structure */
+struct ext4_fc_tl {
+	__le16 fc_tag;
+	__le16 fc_len;
+};
+
+/* Value structure for tag EXT4_FC_TAG_HEAD. */
+struct ext4_fc_head {
+	__le32 fc_features;
+	__le32 fc_tid;
+};
+
+/* Value structure for EXT4_FC_TAG_ADD_RANGE. */
+struct ext4_fc_add_range {
+	__le32 fc_ino;
+	__u8 fc_ex[12];
+};
+
+/* Value structure for tag EXT4_FC_TAG_DEL_RANGE. */
+struct ext4_fc_del_range {
+	__le32 fc_ino;
+	__le32 fc_lblk;
+	__le32 fc_len;
+};
+
+/*
+ * This is the value structure for tags EXT4_FC_TAG_CREAT, EXT4_FC_TAG_LINK
+ * and EXT4_FC_TAG_UNLINK.
+ */
+struct ext4_fc_dentry_info {
+	__le32 fc_parent_ino;
+	__le32 fc_ino;
+	__u8 fc_dname[0];
+};
+
+/* Value structure for EXT4_FC_TAG_INODE and EXT4_FC_TAG_INODE_PARTIAL. */
+struct ext4_fc_inode {
+	__le32 fc_ino;
+	__u8 fc_raw_inode[0];
+};
+
+/* Value structure for tag EXT4_FC_TAG_TAIL. */
+struct ext4_fc_tail {
+	__le32 fc_tid;
+	__le32 fc_crc;
+};
+
+/*
+ * Fast commit reason codes
+ */
+enum {
+	/*
+	 * Commit status codes:
+	 */
+	EXT4_FC_REASON_OK = 0,
+	EXT4_FC_REASON_INELIGIBLE,
+	EXT4_FC_REASON_ALREADY_COMMITTED,
+	EXT4_FC_REASON_FC_START_FAILED,
+	EXT4_FC_REASON_FC_FAILED,
+
+	/*
+	 * Fast commit ineligiblity reasons:
+	 */
+	EXT4_FC_REASON_XATTR = 0,
+	EXT4_FC_REASON_CROSS_RENAME,
+	EXT4_FC_REASON_JOURNAL_FLAG_CHANGE,
+	EXT4_FC_REASON_NOMEM,
+	EXT4_FC_REASON_SWAP_BOOT,
+	EXT4_FC_REASON_RESIZE,
+	EXT4_FC_REASON_RENAME_DIR,
+	EXT4_FC_REASON_FALLOC_RANGE,
+	EXT4_FC_REASON_INODE_JOURNAL_DATA,
+	EXT4_FC_COMMIT_FAILED,
+	EXT4_FC_REASON_MAX
+};
+
+#ifdef __KERNEL__
+/*
+ * In memory list of dentry updates that are performed on the file
+ * system used by fast commit code.
+ */
+struct ext4_fc_dentry_update {
+	int fcd_op;		/* Type of update create / unlink / link */
+	int fcd_parent;		/* Parent inode number */
+	int fcd_ino;		/* Inode number */
+	struct qstr fcd_name;	/* Dirent name */
+	unsigned char fcd_iname[DNAME_INLINE_LEN];	/* Dirent name string */
+	struct list_head fcd_list;
+};
+
+struct ext4_fc_stats {
+	unsigned int fc_ineligible_reason_count[EXT4_FC_REASON_MAX];
+	unsigned long fc_num_commits;
+	unsigned long fc_ineligible_commits;
+	unsigned long fc_numblks;
+};
+
+#define EXT4_FC_REPLAY_REALLOC_INCREMENT	4
+
+/*
+ * Physical block regions added to different inodes due to fast commit
+ * recovery. These are set during the SCAN phase. During the replay phase,
+ * our allocator excludes these from its allocation. This ensures that
+ * we don't accidentally allocating a block that is going to be used by
+ * another inode.
+ */
+struct ext4_fc_alloc_region {
+	ext4_lblk_t lblk;
+	ext4_fsblk_t pblk;
+	int ino, len;
+};
+
+/*
+ * Fast commit replay state.
+ */
+struct ext4_fc_replay_state {
+	int fc_replay_num_tags;
+	int fc_replay_expected_off;
+	int fc_current_pass;
+	int fc_cur_tag;
+	int fc_crc;
+	struct ext4_fc_alloc_region *fc_regions;
+	int fc_regions_size, fc_regions_used, fc_regions_valid;
+	int *fc_modified_inodes;
+	int fc_modified_inodes_used, fc_modified_inodes_size;
+};
+
+#define region_last(__region) (((__region)->lblk) + ((__region)->len) - 1)
+#endif
+
+#define fc_for_each_tl(__start, __end, __tl)				\
+	for (tl = (struct ext4_fc_tl *)(__start);			\
+	     (__u8 *)tl < (__u8 *)(__end);				\
+		tl = (struct ext4_fc_tl *)((__u8 *)tl +			\
+					sizeof(struct ext4_fc_tl) +	\
+					+ le16_to_cpu(tl->fc_len)))
+
+static inline const char *tag2str(__u16 tag)
+{
+	switch (tag) {
+	case EXT4_FC_TAG_LINK:
+		return "ADD_ENTRY";
+	case EXT4_FC_TAG_UNLINK:
+		return "DEL_ENTRY";
+	case EXT4_FC_TAG_ADD_RANGE:
+		return "ADD_RANGE";
+	case EXT4_FC_TAG_CREAT:
+		return "CREAT_DENTRY";
+	case EXT4_FC_TAG_DEL_RANGE:
+		return "DEL_RANGE";
+	case EXT4_FC_TAG_INODE:
+		return "INODE";
+	case EXT4_FC_TAG_PAD:
+		return "PAD";
+	case EXT4_FC_TAG_TAIL:
+		return "TAIL";
+	case EXT4_FC_TAG_HEAD:
+		return "HEAD";
+	default:
+		return "ERROR";
+	}
+}
+
+/* Get length of a particular tlv */
+static inline int ext4_fc_tag_len(struct ext4_fc_tl *tl)
+{
+	return le16_to_cpu(tl->fc_len);
+}
+
+/* Get a pointer to "value" of a tlv */
+static inline __u8 *ext4_fc_tag_val(struct ext4_fc_tl *tl)
+{
+	return (__u8 *)tl + sizeof(*tl);
+}
+
+#endif /* __FAST_COMMIT_H__ */
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 09/15] e2fsck: add fast commit scan pass
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (7 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 08/15] e2fsck: add fast commit setup code Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 10/15] e2fsck: add fast commit replay skeleton Harshad Shirwadkar
                   ` (6 subsequent siblings)
  15 siblings, 0 replies; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Add fast commit scan pass. Scan pass is responsible for following
things:

* Count total number of fast commit tags that need to be replayed
  during the replay phase.

* Validate whether the fast commit area is valid for a given
  transaction ID.

* Verify the CRC of fast commit area.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 e2fsck/journal.c | 109 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 109 insertions(+)

diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 56e48385..ba38124e 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -278,6 +278,108 @@ static int process_journal_block(ext2_filsys fs,
 	return 0;
 }
 
+static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh,
+				int off, tid_t expected_tid)
+{
+	e2fsck_t ctx = j->j_fs_dev->k_ctx;
+	struct e2fsck_fc_replay_state *state;
+	int ret = JBD2_FC_REPLAY_CONTINUE;
+	struct ext4_fc_add_range *ext;
+	struct ext4_fc_tl *tl;
+	struct ext4_fc_tail *tail;
+	__u8 *start, *end;
+	struct ext4_fc_head *head;
+	struct ext2fs_extent ext2fs_ex;
+
+	state = &ctx->fc_replay_state;
+
+	start = (__u8 *)bh->b_data;
+	end = (__u8 *)bh->b_data + j->j_blocksize - 1;
+
+	jbd_debug(1, "Scan phase starting, expected %d", expected_tid);
+	if (state->fc_replay_expected_off == 0) {
+		memset(state, 0, sizeof(*state));
+		/* Check if we can stop early */
+		if (le16_to_cpu(((struct ext4_fc_tl *)start)->fc_tag)
+			!= EXT4_FC_TAG_HEAD) {
+			jbd_debug(1, "Ending early!, not a head tag");
+			return 0;
+		}
+	}
+
+	if (off != state->fc_replay_expected_off) {
+		ret = -EFSCORRUPTED;
+		goto out_err;
+	}
+
+	state->fc_replay_expected_off++;
+	fc_for_each_tl(start, end, tl) {
+		jbd_debug(3, "Scan phase, tag:%s, blk %lld\n",
+			  tag2str(le16_to_cpu(tl->fc_tag)), bh->b_blocknr);
+		switch (le16_to_cpu(tl->fc_tag)) {
+		case EXT4_FC_TAG_ADD_RANGE:
+			ext = (struct ext4_fc_add_range *)ext4_fc_tag_val(tl);
+			ret = ext2fs_decode_extent(&ext2fs_ex, (void *)&ext->fc_ex,
+						   sizeof(ext->fc_ex));
+			if (ret)
+				ret = JBD2_FC_REPLAY_STOP;
+			else
+				ret = JBD2_FC_REPLAY_CONTINUE;
+		case EXT4_FC_TAG_DEL_RANGE:
+		case EXT4_FC_TAG_LINK:
+		case EXT4_FC_TAG_UNLINK:
+		case EXT4_FC_TAG_CREAT:
+		case EXT4_FC_TAG_INODE:
+		case EXT4_FC_TAG_PAD:
+			state->fc_cur_tag++;
+			state->fc_crc = jbd2_chksum(j, state->fc_crc, tl,
+					sizeof(*tl) + ext4_fc_tag_len(tl));
+			break;
+		case EXT4_FC_TAG_TAIL:
+			state->fc_cur_tag++;
+			tail = (struct ext4_fc_tail *)ext4_fc_tag_val(tl);
+			state->fc_crc = jbd2_chksum(j, state->fc_crc, tl,
+						sizeof(*tl) +
+						offsetof(struct ext4_fc_tail,
+						fc_crc));
+			jbd_debug(1, "tail tid %d, expected %d\n",
+					le32_to_cpu(tail->fc_tid),
+					expected_tid);
+			if (le32_to_cpu(tail->fc_tid) == expected_tid &&
+				le32_to_cpu(tail->fc_crc) == state->fc_crc) {
+				state->fc_replay_num_tags = state->fc_cur_tag;
+			} else {
+				ret = state->fc_replay_num_tags ?
+					JBD2_FC_REPLAY_STOP : -EFSBADCRC;
+			}
+			state->fc_crc = 0;
+			break;
+		case EXT4_FC_TAG_HEAD:
+			head = (struct ext4_fc_head *)ext4_fc_tag_val(tl);
+			if (le32_to_cpu(head->fc_features) &
+				~EXT4_FC_SUPPORTED_FEATURES) {
+				ret = -EOPNOTSUPP;
+				break;
+			}
+			if (le32_to_cpu(head->fc_tid) != expected_tid) {
+				ret = -EINVAL;
+				break;
+			}
+			state->fc_cur_tag++;
+			state->fc_crc = jbd2_chksum(j, state->fc_crc, tl,
+					sizeof(*tl) + ext4_fc_tag_len(tl));
+			break;
+		default:
+			ret = state->fc_replay_num_tags ?
+				JBD2_FC_REPLAY_STOP : -ECANCELED;
+		}
+		if (ret < 0 || ret == JBD2_FC_REPLAY_STOP)
+			break;
+	}
+
+out_err:
+	return ret;
+}
 /*
  * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE
  * to indicate that it is expecting more fast commit blocks. It returns
@@ -286,6 +388,13 @@ static int process_journal_block(ext2_filsys fs,
 static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
 				enum passtype pass, int off, tid_t expected_tid)
 {
+	e2fsck_t ctx = journal->j_fs_dev->k_ctx;
+	struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state;
+
+	if (pass == PASS_SCAN) {
+		state->fc_current_pass = PASS_SCAN;
+		return ext4_fc_replay_scan(journal, bh, off, expected_tid);
+	}
 	return JBD2_FC_REPLAY_STOP;
 }
 
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 10/15] e2fsck: add fast commit replay skeleton
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (8 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 09/15] e2fsck: add fast commit scan pass Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 11/15] e2fsck: add fc replay for link, unlink, creat tags Harshad Shirwadkar
                   ` (5 subsequent siblings)
  15 siblings, 0 replies; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

This function adds the skeleton for the replay path. Following patches
in the series implement the handling for individual tags.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 e2fsck/journal.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index ba38124e..9d0637e3 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -390,11 +390,83 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
 {
 	e2fsck_t ctx = journal->j_fs_dev->k_ctx;
 	struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state;
+	int ret = JBD2_FC_REPLAY_CONTINUE;
+	struct ext4_fc_tl *tl;
+	__u8 *start, *end;
 
 	if (pass == PASS_SCAN) {
 		state->fc_current_pass = PASS_SCAN;
 		return ext4_fc_replay_scan(journal, bh, off, expected_tid);
 	}
+
+	if (state->fc_replay_num_tags == 0)
+		goto replay_done;
+
+	if (state->fc_current_pass != pass) {
+		/* Starting replay phase */
+		state->fc_current_pass = pass;
+		/* We will reset checksums */
+		ctx->fs->flags |= EXT2_FLAG_IGNORE_CSUM_ERRORS;
+		ret = ext2fs_read_bitmaps(ctx->fs);
+		if (ret) {
+			jbd_debug(1, "Error %d while reading bitmaps\n", ret);
+			return ret;
+		}
+		state->fc_super_state = ctx->fs->super->s_state;
+		/*
+		 * Mark the file system to indicate it contains errors. That's
+		 * because the updates performed by fast commit replay code are
+		 * not atomic and may result in incosistent file system if it
+		 * crashes before the replay is complete.
+		 */
+		ctx->fs->super->s_state |= EXT2_ERROR_FS;
+		ctx->fs->super->s_state |= EXT4_FC_REPLAY;
+		ext2fs_mark_super_dirty(ctx->fs);
+		ext2fs_flush(ctx->fs);
+	}
+
+	start = (__u8 *)bh->b_data;
+	end = (__u8 *)bh->b_data + journal->j_blocksize - 1;
+
+	fc_for_each_tl(start, end, tl) {
+		if (state->fc_replay_num_tags == 0)
+			goto replay_done;
+		jbd_debug(3, "Replay phase processing %s tag\n",
+				tag2str(le16_to_cpu(tl->fc_tag)));
+		state->fc_replay_num_tags--;
+		switch (le16_to_cpu(tl->fc_tag)) {
+		case EXT4_FC_TAG_CREAT:
+		case EXT4_FC_TAG_LINK:
+		case EXT4_FC_TAG_UNLINK:
+		case EXT4_FC_TAG_ADD_RANGE:
+		case EXT4_FC_TAG_DEL_RANGE:
+		case EXT4_FC_TAG_INODE:
+		case EXT4_FC_TAG_TAIL:
+		case EXT4_FC_TAG_PAD:
+		case EXT4_FC_TAG_HEAD:
+			break;
+		default:
+			ret = -ECANCELED;
+			break;
+		}
+		if (ret < 0)
+			break;
+		ret = JBD2_FC_REPLAY_CONTINUE;
+	}
+	return ret;
+replay_done:
+	jbd_debug(1, "End of fast commit replay\n");
+	if (state->fc_current_pass != pass)
+		return JBD2_FC_REPLAY_STOP;
+
+	ext2fs_calculate_summary_stats(ctx->fs, 0 /* update bg also */);
+	ext2fs_write_block_bitmap(ctx->fs);
+	ext2fs_write_inode_bitmap(ctx->fs);
+	ext2fs_mark_super_dirty(ctx->fs);
+	ext2fs_set_gdt_csum(ctx->fs);
+	ctx->fs->super->s_state = state->fc_super_state;
+	ext2fs_flush(ctx->fs);
+
 	return JBD2_FC_REPLAY_STOP;
 }
 
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 11/15] e2fsck: add fc replay for link, unlink, creat tags
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (9 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 10/15] e2fsck: add fast commit replay skeleton Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 12/15] e2fsck: add replay for add_range, del_range, and inode tags Harshad Shirwadkar
                   ` (4 subsequent siblings)
  15 siblings, 0 replies; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Add fast commit replay for directory entry updates.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 e2fsck/journal.c    | 93 +++++++++++++++++++++++++++++++++++++++++++++
 lib/ext2fs/ext2fs.h |  4 ++
 lib/ext2fs/unlink.c |  6 +--
 3 files changed, 100 insertions(+), 3 deletions(-)

diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 9d0637e3..27ff5a76 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -380,6 +380,95 @@ static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh,
 out_err:
 	return ret;
 }
+/* Helper struct for dentry replay routines */
+struct dentry_info_args {
+	int parent_ino, dname_len, ino, inode_len;
+	char *dname;
+};
+
+static inline void tl_to_darg(struct dentry_info_args *darg,
+				struct  ext4_fc_tl *tl)
+{
+	struct ext4_fc_dentry_info *fcd;
+	int tag = le16_to_cpu(tl->fc_tag);
+
+	fcd = (struct ext4_fc_dentry_info *)ext4_fc_tag_val(tl);
+
+	darg->parent_ino = le32_to_cpu(fcd->fc_parent_ino);
+	darg->ino = le32_to_cpu(fcd->fc_ino);
+	darg->dname = fcd->fc_dname;
+	darg->dname_len = ext4_fc_tag_len(tl) -
+			sizeof(struct ext4_fc_dentry_info);
+	darg->dname = malloc(darg->dname_len + 1);
+	memcpy(darg->dname, fcd->fc_dname, darg->dname_len);
+	darg->dname[darg->dname_len] = 0;
+	jbd_debug(1, "%s: %s, ino %d, parent %d\n",
+		tag == EXT4_FC_TAG_CREAT ? "create" :
+		(tag == EXT4_FC_TAG_LINK ? "link" :
+		(tag == EXT4_FC_TAG_UNLINK ? "unlink" : "error")),
+		darg->dname, darg->ino, darg->parent_ino);
+}
+
+static int ext4_fc_handle_unlink(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+	struct ext2_inode inode;
+	struct dentry_info_args darg;
+	ext2_filsys fs = ctx->fs;
+
+	tl_to_darg(&darg, tl);
+	ext2fs_unlink(ctx->fs, darg.parent_ino, darg.dname, darg.ino, 0);
+	/* It's okay if the above call fails */
+	free(darg.dname);
+}
+
+static int ext4_fc_handle_link_and_create(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+	struct dentry_info_args darg;
+	ext2_filsys fs = ctx->fs;
+	struct ext2_inode_large inode_large;
+	int ret, filetype, mode;
+
+	tl_to_darg(&darg, tl);
+	ret = ext2fs_read_inode(fs, darg.ino,
+					(struct ext2_inode *)&inode_large);
+	if (ret)
+		goto out;
+
+	mode = inode_large.i_mode;
+
+	if (LINUX_S_ISREG(mode))
+		filetype = EXT2_FT_REG_FILE;
+	else if (LINUX_S_ISDIR(mode))
+		filetype = EXT2_FT_DIR;
+	else if (LINUX_S_ISCHR(mode))
+		filetype = EXT2_FT_CHRDEV;
+	else if (LINUX_S_ISBLK(mode))
+		filetype = EXT2_FT_BLKDEV;
+	else if (LINUX_S_ISLNK(mode))
+		return EXT2_FT_SYMLINK;
+	else if (LINUX_S_ISFIFO(mode))
+		filetype = EXT2_FT_FIFO;
+	else if (LINUX_S_ISSOCK(mode))
+		filetype = EXT2_FT_SOCK;
+	else {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/*
+	 * Forcefully unlink if the same name is present and ignore the error
+	 * if any, since this dirent might not exist
+	 */
+	ext2fs_unlink(fs, darg.parent_ino, darg.dname, darg.ino,
+			EXT2FS_UNLINK_FORCE);
+
+	ret = ext2fs_link(fs, darg.parent_ino, darg.dname, darg.ino,
+				filetype);
+out:
+	free(darg.dname);
+	return ret;
+
+}
 /*
  * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE
  * to indicate that it is expecting more fast commit blocks. It returns
@@ -437,7 +526,11 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
 		switch (le16_to_cpu(tl->fc_tag)) {
 		case EXT4_FC_TAG_CREAT:
 		case EXT4_FC_TAG_LINK:
+			ret = ext4_fc_handle_link_and_create(ctx, tl);
+			break;
 		case EXT4_FC_TAG_UNLINK:
+			ret = ext4_fc_handle_unlink(ctx, tl);
+			break;
 		case EXT4_FC_TAG_ADD_RANGE:
 		case EXT4_FC_TAG_DEL_RANGE:
 		case EXT4_FC_TAG_INODE:
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index eb2e6549..931cd569 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1684,6 +1684,10 @@ extern errcode_t ext2fs_get_pathname(ext2_filsys fs, ext2_ino_t dir, ext2_ino_t
 			       char **name);
 
 /* link.c */
+#define EXT2FS_UNLINK_FORCE		0x1	/* Forcefully unlink even if
+						 * the inode number doesn't
+						 * match the dirent
+						 */
 errcode_t ext2fs_link(ext2_filsys fs, ext2_ino_t dir, const char *name,
 		      ext2_ino_t ino, int flags);
 errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir, const char *name,
diff --git a/lib/ext2fs/unlink.c b/lib/ext2fs/unlink.c
index 8ab27ee2..3ec04cfb 100644
--- a/lib/ext2fs/unlink.c
+++ b/lib/ext2fs/unlink.c
@@ -49,7 +49,7 @@ static int unlink_proc(struct ext2_dir_entry *dirent,
 		if (strncmp(ls->name, dirent->name, ext2fs_dirent_name_len(dirent)))
 			return 0;
 	}
-	if (ls->inode) {
+	if (!(ls->flags & EXT2FS_UNLINK_FORCE) && ls->inode) {
 		if (dirent->inode != ls->inode)
 			return 0;
 	} else {
@@ -70,7 +70,7 @@ static int unlink_proc(struct ext2_dir_entry *dirent,
 #endif
 errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir,
 			const char *name, ext2_ino_t ino,
-			int flags EXT2FS_ATTR((unused)))
+			int flags)
 {
 	errcode_t	retval;
 	struct link_struct ls;
@@ -86,7 +86,7 @@ errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir,
 	ls.name = name;
 	ls.namelen = name ? strlen(name) : 0;
 	ls.inode = ino;
-	ls.flags = 0;
+	ls.flags = flags;
 	ls.done = 0;
 	ls.prev = 0;
 
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 12/15] e2fsck: add replay for add_range, del_range, and inode tags
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (10 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 11/15] e2fsck: add fc replay for link, unlink, creat tags Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 13/15] debugfs: add fast commit support to logdump Harshad Shirwadkar
                   ` (3 subsequent siblings)
  15 siblings, 0 replies; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Add replay for inode's extent trees and inode itself.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 e2fsck/journal.c | 343 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 343 insertions(+)

diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 27ff5a76..581b519b 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -380,6 +380,219 @@ static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh,
 out_err:
 	return ret;
 }
+
+#define ex_end(__ex) ((__ex)->e_lblk + (__ex)->e_len - 1)
+#define ex_pend(__ex) ((__ex)->e_pblk + (__ex)->e_len - 1)
+
+static int make_room(struct extent_list *list, int i)
+{
+	int ret;
+
+	if (list->count == list->size) {
+		unsigned int new_size = (list->size + 341) *
+					sizeof(struct ext2fs_extent);
+		ret = ext2fs_resize_mem(0, new_size, &list->extents);
+		if (ret)
+			return ret;
+		list->size += 341;
+	}
+
+	memmove(&list->extents[i + 1], &list->extents[i],
+			sizeof(list->extents[0]) * (list->count - i));
+	list->count++;
+	return 0;
+}
+
+static int ex_compar(const void *arg1, const void *arg2)
+{
+	struct ext2fs_extent *ex1 = (struct ext2fs_extent *)arg1;
+	struct ext2fs_extent *ex2 = (struct ext2fs_extent *)arg2;
+
+	if (ex1->e_lblk < ex2->e_lblk)
+		return -1;
+	if (ex1->e_lblk > ex2->e_lblk)
+		return 1;
+	return ex1->e_len - ex2->e_len;
+}
+
+static int ex_len_compar(const void *arg1, const void *arg2)
+{
+	struct ext2fs_extent *ex1 = (struct ext2fs_extent *)arg1;
+	struct ext2fs_extent *ex2 = (struct ext2fs_extent *)arg2;
+
+	if (ex1->e_len < ex2->e_len)
+		return 1;
+
+	if (ex1->e_lblk > ex2->e_lblk)
+		return -1;
+
+	return 0;
+}
+
+static void ex_sort_and_merge(e2fsck_t ctx, struct extent_list *list)
+{
+	blk64_t ex_end;
+	int i, j;
+
+	if (list->count < 2)
+		return;
+
+	/*
+	 * Reverse sort by length, that way we strip off all the 0 length
+	 * extents
+	 */
+	qsort(list->extents, list->count, sizeof(struct ext2fs_extent),
+		ex_len_compar);
+
+	for (i = 0; i < list->count; i++) {
+		if (list->extents[i].e_len == 0) {
+			list->count = i;
+			break;
+		}
+	}
+
+	/* Now sort by logical offset */
+	qsort(list->extents, list->count, sizeof(list->extents[0]),
+		ex_compar);
+
+	/* Merge adjacent extents if they are logically and physically contiguous */
+	i = 0;
+	while (i < list->count - 1) {
+		if (ex_end(&list->extents[i]) + 1 != list->extents[i + 1].e_lblk ||
+			ex_pend(&list->extents[i]) + 1 != list->extents[i + 1].e_pblk ||
+			(list->extents[i].e_flags & EXT2_EXTENT_FLAGS_UNINIT) !=
+				(list->extents[i + 1].e_flags & EXT2_EXTENT_FLAGS_UNINIT)) {
+			i++;
+			continue;
+		}
+
+		list->extents[i].e_len += list->extents[i + 1].e_len;
+		for (j = i + 1; j < list->count - 1; j++)
+			list->extents[j] = list->extents[j + 1];
+		list->count--;
+	}
+}
+
+/* must free blocks that are released */
+static int ext2fs_modify_extent_list(e2fsck_t ctx, struct extent_list *list,
+					struct ext2fs_extent *ex, int del)
+{
+	int ret;
+	int i, offset;
+	struct ext2fs_extent add_ex = *ex, add_ex2;
+
+	/* First let's create a hole from ex->e_lblk of length ex->e_len */
+	for (i = 0; i < list->count; i++) {
+		if (ex_end(&list->extents[i]) < add_ex.e_lblk)
+			continue;
+
+		/* Case 1: No overlap */
+		if (list->extents[i].e_lblk > ex_end(&add_ex))
+			break;
+		/*
+		 * Unmark all the blocks in bb now. All the blocks get marked
+		 * before we exit this function.
+		 */
+		ext2fs_unmark_block_bitmap_range2(ctx->fs->block_map,
+			list->extents[i].e_pblk, list->extents[i].e_len);
+		/* Case 2: Split */
+		if (list->extents[i].e_lblk < add_ex.e_lblk &&
+			ex_end(&list->extents[i]) > ex_end(&add_ex)) {
+			ret = make_room(list, i + 1);
+			if (ret)
+				return ret;
+			list->extents[i + 1] = list->extents[i];
+			offset = ex_end(&add_ex) + 1 - list->extents[i].e_lblk;
+			list->extents[i + 1].e_lblk += offset;
+			list->extents[i + 1].e_pblk += offset;
+			list->extents[i + 1].e_len -= offset;
+			list->extents[i].e_len =
+				add_ex.e_lblk - list->extents[i].e_lblk;
+			break;
+		}
+
+		/* Case 3: Exact overlap */
+		if (add_ex.e_lblk <= list->extents[i].e_lblk  &&
+			ex_end(&list->extents[i]) <= ex_end(&add_ex)) {
+
+			list->extents[i].e_len = 0;
+			continue;
+		}
+
+		/* Case 4: Partial overlap */
+		if (ex_end(&list->extents[i]) > ex_end(&add_ex)) {
+			offset = ex_end(&add_ex) + 1 - list->extents[i].e_lblk;
+			list->extents[i].e_lblk += offset;
+			list->extents[i].e_pblk += offset;
+			list->extents[i].e_len -= offset;
+			break;
+		}
+
+		if (ex_end(&add_ex) >= ex_end(&list->extents[i])) {
+			list->extents[i].e_len =
+				max(add_ex.e_lblk - list->extents[i].e_lblk,
+					0);
+		}
+	}
+
+	if (add_ex.e_len && !del) {
+		make_room(list, list->count);
+		list->extents[list->count - 1] = add_ex;
+	}
+
+	ex_sort_and_merge(ctx, list);
+
+	/* Mark all occupied blocks allocated */
+	for (i = 0; i < list->count; i++)
+		ext2fs_mark_block_bitmap_range2(ctx->fs->block_map,
+			list->extents[i].e_pblk, list->extents[i].e_len);
+	ext2fs_mark_bb_dirty(ctx->fs);
+
+	return 0;
+}
+
+static int ext2fs_add_extent_to_list(e2fsck_t ctx, struct extent_list *list,
+					struct ext2fs_extent *ex)
+{
+	return ext2fs_modify_extent_list(ctx, list, ex, 0 /* add */);
+}
+
+static int ext2fs_del_extent_to_list(e2fsck_t ctx, struct extent_list *list,
+					struct ext2fs_extent *ex)
+{
+	return ext2fs_modify_extent_list(ctx, list, ex, 1 /* delete */);
+}
+
+static int e2fsck_fc_read_extents(e2fsck_t ctx, int ino)
+{
+	struct extent_list *extent_list = &ctx->fc_replay_state.fc_extent_list;
+
+	if (extent_list->ino == ino)
+		return 0;
+
+	extent_list->ino = ino;
+	return e2fsck_read_extents(ctx, extent_list);
+}
+
+/*
+ * Flush extents in replay state on disk. @ino is the inode that is going
+ * to be processed next. So, we hold back flushing of the extent list
+ * if the next inode that's going to be processed is same as the one with
+ * cached extents in our replay state. That allows us to gather multiple extents
+ * for the inode so that we can flush all of them at once and it also saves us
+ * from continuously growing and shrinking the extent tree.
+ */
+static void e2fsck_fc_flush_extents(e2fsck_t ctx, int ino)
+{
+	struct extent_list *extent_list = &ctx->fc_replay_state.fc_extent_list;
+
+	if (extent_list->ino == ino || extent_list->ino == 0)
+		return;
+	e2fsck_rewrite_extent_tree(ctx, extent_list);
+	ext2fs_free_mem(&extent_list->extents);
+	memset(extent_list, 0, sizeof(*extent_list));
+}
+
 /* Helper struct for dentry replay routines */
 struct dentry_info_args {
 	int parent_ino, dname_len, ino, inode_len;
@@ -416,6 +629,7 @@ static int ext4_fc_handle_unlink(e2fsck_t ctx, struct ext4_fc_tl *tl)
 	ext2_filsys fs = ctx->fs;
 
 	tl_to_darg(&darg, tl);
+	e2fsck_fc_flush_extents(ctx, darg.ino);
 	ext2fs_unlink(ctx->fs, darg.parent_ino, darg.dname, darg.ino, 0);
 	/* It's okay if the above call fails */
 	free(darg.dname);
@@ -429,6 +643,8 @@ static int ext4_fc_handle_link_and_create(e2fsck_t ctx, struct ext4_fc_tl *tl)
 	int ret, filetype, mode;
 
 	tl_to_darg(&darg, tl);
+
+	e2fsck_fc_flush_extents(ctx, 0);
 	ret = ext2fs_read_inode(fs, darg.ino,
 					(struct ext2_inode *)&inode_large);
 	if (ret)
@@ -469,6 +685,126 @@ out:
 	return ret;
 
 }
+
+/* This function fixes the i_blocks field in the replayed indoe */
+static void ext4_fc_replay_fixup_iblocks(struct ext2_inode_large *ondisk_inode,
+	struct ext2_inode_large *fc_inode)
+{
+	if (le32_to_cpu(ondisk_inode->i_flags) & EXT4_EXTENTS_FL) {
+		struct ext3_extent_header *eh;
+
+		eh = (struct ext3_extent_header *)(&ondisk_inode->i_block[0]);
+		if (eh->eh_magic != EXT3_EXT_MAGIC) {
+			memset(eh, 0, sizeof(*eh));
+			eh->eh_magic = EXT3_EXT_MAGIC;
+			eh->eh_max = cpu_to_le16(
+				(sizeof(ondisk_inode->i_block) -
+					sizeof(struct ext3_extent_header)) /
+					sizeof(struct ext3_extent));
+		}
+	} else if (le32_to_cpu(ondisk_inode->i_flags) & EXT4_INLINE_DATA_FL) {
+		memcpy(ondisk_inode->i_block, fc_inode->i_block,
+			sizeof(fc_inode->i_block));
+	}
+}
+
+static int ext4_fc_handle_inode(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+	struct ext4_fc_inode *fc_inode;
+	struct ext2_inode_large *inode = NULL;
+	int ino, inode_len = EXT2_GOOD_OLD_INODE_SIZE, ret;
+	struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state;
+
+	fc_inode = (struct ext4_fc_inode *)ext4_fc_tag_val(tl);
+	ino = le32_to_cpu(fc_inode->fc_ino);
+
+	if (EXT2_INODE_SIZE(ctx->fs->super) > EXT2_GOOD_OLD_INODE_SIZE)
+		inode_len += ext2fs_le16_to_cpu(
+			((struct ext2_inode_large *)fc_inode->fc_raw_inode)
+				->i_extra_isize);
+	ret = ext2fs_get_mem(inode_len, &inode);
+	if (ret)
+		return ret;
+	e2fsck_fc_flush_extents(ctx, ino);
+
+	ret = ext2fs_read_inode_full(ctx->fs, ino, (struct ext2_inode *)inode,
+					inode_len);
+	if (ret)
+		goto out;
+	memcpy(inode, fc_inode->fc_raw_inode,
+		offsetof(struct ext2_inode_large, i_block));
+	memcpy(&inode->i_generation,
+		&((struct ext2_inode_large *)(fc_inode->fc_raw_inode))->i_generation,
+		inode_len - offsetof(struct ext2_inode_large, i_generation));
+	ext4_fc_replay_fixup_iblocks(inode,
+		(struct ext2_inode_large *)fc_inode->fc_raw_inode);
+	ext2fs_iblk_set(ctx->fs, EXT2_INODE(inode),
+		ext2fs_count_blocks(ctx->fs, ino, EXT2_INODE(inode)));
+
+	ext2fs_inode_csum_set(ctx->fs, ino, inode);
+
+	ret = ext2fs_write_inode_full(ctx->fs, ino, (struct ext2_inode *)inode,
+					inode_len);
+	if (inode->i_links_count)
+		ext2fs_mark_inode_bitmap2(ctx->fs->inode_map, ino);
+	else
+		ext2fs_unmark_inode_bitmap2(ctx->fs->inode_map, ino);
+	ext2fs_mark_ib_dirty(ctx->fs);
+
+out:
+	ext2fs_free_mem(&inode);
+	return ret;
+}
+
+/*
+ * Handle add extent replay tag.
+ */
+static int ext4_fc_handle_add_extent(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+	struct ext2fs_extent extent;
+	struct ext4_fc_add_range *add_range;
+	struct ext4_fc_del_range *del_range;
+	int ret = 0, ino;
+
+	add_range = (struct ext4_fc_add_range *)ext4_fc_tag_val(tl);
+	ino = le32_to_cpu(add_range->fc_ino);
+	e2fsck_fc_flush_extents(ctx, ino);
+
+	ret = e2fsck_fc_read_extents(ctx, ino);
+	if (ret)
+		return ret;
+	memset(&extent, 0, sizeof(extent));
+	ret = ext2fs_decode_extent(&extent, (void *)(add_range->fc_ex),
+				   sizeof(add_range->fc_ex));
+	if (ret)
+		return ret;
+	return ext2fs_add_extent_to_list(ctx,
+		&ctx->fc_replay_state.fc_extent_list, &extent);
+}
+
+/*
+ * Handle delete logical range replay tag.
+ */
+static int ext4_fc_handle_del_range(e2fsck_t ctx, struct ext4_fc_tl *tl)
+{
+	struct ext2fs_extent extent;
+	struct ext4_fc_del_range *del_range;
+	int ret, ino;
+
+	del_range = (struct ext4_fc_del_range *)ext4_fc_tag_val(tl);
+	ino = le32_to_cpu(del_range->fc_ino);
+	e2fsck_fc_flush_extents(ctx, ino);
+
+	memset(&extent, 0, sizeof(extent));
+	extent.e_lblk = ext2fs_le32_to_cpu(del_range->fc_lblk);
+	extent.e_len = ext2fs_le16_to_cpu(del_range->fc_len);
+	ret = e2fsck_fc_read_extents(ctx, ino);
+	if (ret)
+		return ret;
+	return ext2fs_del_extent_to_list(ctx,
+		&ctx->fc_replay_state.fc_extent_list, &extent);
+}
+
 /*
  * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE
  * to indicate that it is expecting more fast commit blocks. It returns
@@ -532,9 +868,16 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh,
 			ret = ext4_fc_handle_unlink(ctx, tl);
 			break;
 		case EXT4_FC_TAG_ADD_RANGE:
+			ret = ext4_fc_handle_add_extent(ctx, tl);
+			break;
 		case EXT4_FC_TAG_DEL_RANGE:
+			ret = ext4_fc_handle_del_range(ctx, tl);
+			break;
 		case EXT4_FC_TAG_INODE:
+			ret = ext4_fc_handle_inode(ctx, tl);
+			break;
 		case EXT4_FC_TAG_TAIL:
+			e2fsck_fc_flush_extents(ctx, 0);
 		case EXT4_FC_TAG_PAD:
 		case EXT4_FC_TAG_HEAD:
 			break;
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 13/15] debugfs: add fast commit support to logdump
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (11 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 12/15] e2fsck: add replay for add_range, del_range, and inode tags Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21 16:49   ` Theodore Ts'o
  2021-01-20 21:26 ` [PATCH v3 14/15] tests: add fast commit recovery tests Harshad Shirwadkar
                   ` (2 subsequent siblings)
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Add fast commit support for debugfs logdump. The debugfs output looks
like this:

debugfs 1.46-WIP (20-Mar-2020)
debugfs:  logdump
Journal starts at block 1, transaction 2
Found expected sequence 2, type 1 (descriptor block) at block 1
Found expected sequence 2, type 2 (commit block) at block 10
No magic number at block 11: end of journal.

*** Fast Commit Area ***
tag HEAD, features 0x0, tid 3
tag INODE, inode 14
tag ADD_RANGE, inode 14, lblk 0, pblk 1091, len 1
tag DEL_RANGE, inode 14, lblk 1, len 1
tag CREAT_DENTRY, parent 12, ino 14, name "new"
tag DEL_ENTRY, parent 12, ino 13, name "old"
tag INODE, inode 13
tag ADD_RANGE, inode 13, lblk 0, pblk 1603, len 16
tag ADD_RANGE, inode 13, lblk 16, pblk 1092, len 240
tag CREAT_DENTRY, parent 12, ino 13, name "data"
tag INODE, inode 14
tag INODE, inode 13
tag TAIL, tid 3

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 debugfs/logdump.c | 122 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 117 insertions(+), 5 deletions(-)

diff --git a/debugfs/logdump.c b/debugfs/logdump.c
index 16889954..dbcc2fd3 100644
--- a/debugfs/logdump.c
+++ b/debugfs/logdump.c
@@ -33,6 +33,7 @@ extern char *optarg;
 #include "debugfs.h"
 #include "blkid/blkid.h"
 #include "jfs_user.h"
+#include "ext2fs/fast_commit.h"
 #include <uuid/uuid.h>
 
 enum journal_location {JOURNAL_IS_INTERNAL, JOURNAL_IS_EXTERNAL};
@@ -65,6 +66,9 @@ static void dump_metadata_block(FILE *, struct journal_source *,
 				unsigned int, unsigned int, unsigned int,
 				int, tid_t);
 
+static void dump_fc_block(FILE *out_file, char *buf, int blocksize,
+			  int transaction, int *fc_done, int dump_old);
+
 static void do_hexdump (FILE *, char *, int);
 
 #define WRAP(jsb, blocknr)					\
@@ -353,6 +357,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
 	journal_header_t	*header;
 	tid_t			transaction;
 	unsigned int		blocknr = 0;
+	int			fc_done;
 
 	/* First, check to see if there's an ext2 superblock header */
 	retval = read_journal_block(cmdname, source, 0, buf, 2048);
@@ -410,7 +415,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
 	if (!blocknr) {
 		/* Empty journal, nothing to do. */
 		if (!dump_old)
-			return;
+			goto fc;
 		else
 			blocknr = 1;
 	}
@@ -420,7 +425,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
 				((ext2_loff_t) blocknr) * blocksize,
 				buf, blocksize);
 		if (retval)
-			return;
+			break;
 
 		header = (journal_header_t *) buf;
 
@@ -431,7 +436,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
 		if (magic != JBD2_MAGIC_NUMBER) {
 			fprintf (out_file, "No magic number at block %u: "
 				 "end of journal.\n", blocknr);
-			return;
+			break;
 		}
 
 		if (sequence != transaction) {
@@ -439,7 +444,7 @@ static void dump_journal(char *cmdname, FILE *out_file,
 				 "block %u: end of journal.\n",
 				 sequence, transaction, blocknr);
 			if (!dump_old)
-				return;
+				break;
 		}
 
 		if (dump_descriptors) {
@@ -473,9 +478,25 @@ static void dump_journal(char *cmdname, FILE *out_file,
 		default:
 			fprintf (out_file, "Unexpected block type %u at "
 				 "block %u.\n", blocktype, blocknr);
-			return;
+			break;
 		}
 	}
+
+fc:
+	blocknr = be32_to_cpu(jsb->s_maxlen) - jbd_get_num_fc_blks(jsb) + 1;
+	while (blocknr <= be32_to_cpu(jsb->s_maxlen)) {
+		retval = read_journal_block(cmdname, source,
+				((ext2_loff_t) blocknr) * blocksize,
+				buf, blocksize);
+		if (retval)
+			return;
+
+		dump_fc_block(out_file, buf, blocksize, transaction, &fc_done,
+			dump_old);
+		if (!dump_old && fc_done)
+			break;
+		blocknr++;
+	}
 }
 
 static inline size_t journal_super_tag_bytes(journal_superblock_t *jsb)
@@ -496,6 +517,97 @@ static inline size_t journal_super_tag_bytes(journal_superblock_t *jsb)
 	return sz - sizeof(__u32);
 }
 
+static void dump_fc_block(FILE *out_file, char *buf, int blocksize,
+	int transaction, int *fc_done, int dump_old)
+{
+	struct ext4_fc_tl	*tl;
+	struct ext4_fc_head	*head;
+	struct ext4_fc_add_range	*add_range;
+	struct ext4_fc_del_range	*del_range;
+	struct ext4_fc_dentry_info	*dentry_info;
+	struct ext4_fc_tail		*tail;
+	struct ext3_extent	*ex;
+
+	*fc_done = 0;
+	fc_for_each_tl(buf, buf + blocksize, tl) {
+		switch (le16_to_cpu(tl->fc_tag)) {
+		case EXT4_FC_TAG_ADD_RANGE:
+			add_range =
+				(struct ext4_fc_add_range *)ext4_fc_tag_val(tl);
+			ex = (struct ext3_extent *)add_range->fc_ex;
+			fprintf(out_file,
+				"tag %s, inode %d, lblk %d, pblk %ld, len %d\n",
+				tag2str(tl->fc_tag),
+				le32_to_cpu(add_range->fc_ino),
+				le32_to_cpu(ex->ee_block),
+				le32_to_cpu(ex->ee_start) +
+				(((__u64) le16_to_cpu(ex->ee_start_hi)) << 32),
+				le16_to_cpu(ex->ee_len) > EXT_INIT_MAX_LEN ?
+				le16_to_cpu(ex->ee_len) - EXT_INIT_MAX_LEN :
+				le16_to_cpu(ex->ee_len));
+			break;
+		case EXT4_FC_TAG_DEL_RANGE:
+			del_range =
+				(struct ext4_fc_del_range *)ext4_fc_tag_val(tl);
+			fprintf(out_file, "tag %s, inode %d, lblk %d, len %d\n",
+				tag2str(tl->fc_tag),
+				le32_to_cpu(del_range->fc_ino),
+				le32_to_cpu(del_range->fc_lblk),
+				le32_to_cpu(del_range->fc_len));
+			break;
+		case EXT4_FC_TAG_LINK:
+		case EXT4_FC_TAG_UNLINK:
+		case EXT4_FC_TAG_CREAT:
+			dentry_info =
+				(struct ext4_fc_dentry_info *)
+					ext4_fc_tag_val(tl);
+			fprintf(out_file,
+				"tag %s, parent %d, ino %d, name \"%s\"\n",
+				tag2str(tl->fc_tag),
+				le32_to_cpu(dentry_info->fc_parent_ino),
+				le32_to_cpu(dentry_info->fc_ino),
+				dentry_info->fc_dname);
+			break;
+		case EXT4_FC_TAG_INODE:
+			fprintf(out_file, "tag %s, inode %d\n",
+				tag2str(tl->fc_tag),
+				le32_to_cpu(((struct ext4_fc_inode *)
+					ext4_fc_tag_val(tl))->fc_ino));
+			break;
+		case EXT4_FC_TAG_PAD:
+			fprintf(out_file, "tag %s\n", tag2str(tl->fc_tag));
+			break;
+		case EXT4_FC_TAG_TAIL:
+			tail = (struct ext4_fc_tail *)ext4_fc_tag_val(tl);
+			fprintf(out_file, "tag %s, tid %d\n",
+				tag2str(tl->fc_tag),
+				le32_to_cpu(tail->fc_tid));
+			if (!dump_old &&
+				le32_to_cpu(tail->fc_tid) < transaction) {
+				*fc_done = 1;
+				return;
+			}
+			break;
+		case EXT4_FC_TAG_HEAD:
+			fprintf(out_file, "\n*** Fast Commit Area ***\n");
+			head = (struct ext4_fc_head *)ext4_fc_tag_val(tl);
+			fprintf(out_file, "tag %s, features 0x%x, tid %d\n",
+				tag2str(tl->fc_tag),
+				le32_to_cpu(head->fc_features),
+				le32_to_cpu(head->fc_tid));
+			if (!dump_old &&
+				le32_to_cpu(head->fc_tid) < transaction) {
+				*fc_done = 1;
+				return;
+			}
+			break;
+		default:
+			*fc_done = 1;
+			break;
+		}
+	}
+}
+
 static void dump_descriptor_block(FILE *out_file,
 				  struct journal_source *source,
 				  char *buf,
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 14/15] tests: add fast commit recovery tests
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (12 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 13/15] debugfs: add fast commit support to logdump Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-20 21:26 ` [PATCH v3 15/15] ext4: fix tests to account for new dumpe2fs output Harshad Shirwadkar
  2021-01-21  5:22 ` [PATCH v3 00/15] Fast commit changes for e2fsprogs Theodore Ts'o
  15 siblings, 0 replies; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Add j_recover_fast_commit test that ensure that e2fsck is able to
recover a disk from fast commit log.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 tests/j_recover_fast_commit/commands |   4 ++++
 tests/j_recover_fast_commit/expect   |  23 +++++++++++++++++++++++
 tests/j_recover_fast_commit/image.gz | Bin 0 -> 3595 bytes
 tests/j_recover_fast_commit/script   |  26 ++++++++++++++++++++++++++
 4 files changed, 53 insertions(+)
 create mode 100644 tests/j_recover_fast_commit/commands
 create mode 100644 tests/j_recover_fast_commit/expect
 create mode 100644 tests/j_recover_fast_commit/image.gz
 create mode 100755 tests/j_recover_fast_commit/script

diff --git a/tests/j_recover_fast_commit/commands b/tests/j_recover_fast_commit/commands
new file mode 100644
index 00000000..74e20e4e
--- /dev/null
+++ b/tests/j_recover_fast_commit/commands
@@ -0,0 +1,4 @@
+ls
+ls a/
+ex a/new
+ex a/data
diff --git a/tests/j_recover_fast_commit/expect b/tests/j_recover_fast_commit/expect
new file mode 100644
index 00000000..2fc1e53f
--- /dev/null
+++ b/tests/j_recover_fast_commit/expect
@@ -0,0 +1,23 @@
+test_filesys: recovering journal
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 14/256 files (14.3% non-contiguous), 1365/2048 blocks
+Exit status is 0
+debugfs: ls
+ 2  (12) .    2  (12) ..    11  (20) lost+found    12  (968) a   
+debugfs: ls a/
+ 12  (12) .    2  (12) ..    13  (12) data    14  (976) new   
+debugfs: ex a/new
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  1     0 -     0  1107 -  1107      1 
+debugfs: ex a/data
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 -   255  1618            256
+ 1/ 1   1/  5     0 -    15  1619 -  1634     16 
+ 1/ 1   2/  5    16 -    31  1601 -  1616     16 
+ 1/ 1   3/  5    32 -    63  1985 -  2016     32 
+ 1/ 1   4/  5    64 -   127  1537 -  1600     64 
+ 1/ 1   5/  5   128 -   255  1793 -  1920    128 
diff --git a/tests/j_recover_fast_commit/image.gz b/tests/j_recover_fast_commit/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..b7357afc46bec8a0154d5746fa2fce86c2341efc
GIT binary patch
literal 3595
zcmeH}eNfT~8ppY9Z*$KxYh$%EC0B221!CQ{-F$;y_Nv9SE-STSSvB{>*GvmjkZCXT
z?f_j=!B-kB-%S%0H3Y6v!7^5Yd<&I|MMB;Z6jb2%yKeTccIN)Sx$&8K{`k&!zW;ol
z`OZ8rs*@8`&c;JF5K?3RngG$yQc})Dc|qP}Dsm5ehAgEclfGm;JbtkgO84-68YdB4
zsDm<_xB0<`_Mfe5Ehc<(^ZmG2x5R<(yD|IUh}iq9FWN?(5XZU@N2~<_h=?GsN5^Ot
zwZ0W$-7K4dx71_oJoDV-Z!~|)54Ez0bAf6;KQnhu>)7(1gnesUs!s~_(bLOL?&h|z
z2j;(5SZe|~f%H?`Jhc>LGeZuYOm4*@|JfNVe2#S*tlw;)X@adgK;_GgnB@`IN$()O
zeJdVTF+(3Gx@-!k#510$y2lYkS(vN9#GrQLCa;-mV?qZU{;kwK$Rv{L6Jf{#Z20Qb
z@<IdCR2elC?GcaQM>yz}Ms3R|)&tr)^yDbq#d`(A0X*cavzL<Ih>NvxIgvD1VK$rO
z1ND7TxV7GM&y8Z#1w(&&^>=;7g{6-m?@l{8KJ!Tb_9_8)`n-#{^gDTjuU|duC!D&$
zw-g&1I=zp?$hSx>UPRYfcB-iNgN=ul&jpT^l}H&s&`y6Z9O@)lFlKin4*F>!XjY}z
zH%zu?*}+&9T@cR1r$2t?naFsn7j8!KH0(v!d&4UQ6AAXi+Ao0GE89WI0~#d}C&YB`
zQNqqK-`b;Z?K3p61O`q$?5kAp0>k|^;cYVc*)iv>>G+ztPx_vc2X@W83hHPAj!b1r
zY6XV1#4Z~|Tv`Oi4Jc0kf>D_H1)flpYK!H<%0qTJsVSQBvck24`sm8=TEiXV>bV)u
zl-(02z{HCOM=U<$%Q$!e#;Zh2T~=hF><M#4mMpSnp{MtWj`3a(FmBaZh3M6Zgi3PW
z7oSt-lI@3EvAcAdyO^4COpLdS%D&a^wF?Lh^2l3Lj_Td#_T$hdf%q$kaMo^{=YS<j
zpX%d+p4Tq1vWoaB>WDWopLklvSs1fCy}Odu<<E1IvI!iDX>l`9%z?r4c5`xf7S%aU
z&E+*8v}xS`<mn%a<)r5>Qt75B{U2YB@TN`IA6TdzSLAF|!jZ$}ahX?GtJAfpeUB+=
zijU8Z&<GZLe6<#QIL-jPlCvCwpZn~m`m1$QXbCOj!zXo1AXuZmiAPRo)q<`L(hnq{
zieo$JJ$<Jv%&bTOVlW^tuTpP}2@?OIK~qHs26H~<-52!hiR8w9Jva~=0g;Vicx70J
z4fElwtc@}#58*Fs1|0nM50oel-tjty1RzdigN^Zbf=@7qqPO^xgBxY9fzK?)NeUd8
z=K`qiTd&px<Cwb{?p?@Fk1F}fxbmz6c%pZJnctU=y#o~r*YII2swB1ZuvtnL8MY-W
zHzn^koHn$58IpGC$6uS%`rn%qHQSg^3U1&P>02?n0K&<(kM_cNQj~1IX?b$W(*0xP
z-_i?z5rOsTi%Tnl`>$~7v^=gl6@7L7z-8Ol{1CZg!>|2lrzyd9v<TeSOBR9Qoi7j>
zfMUoDdQ$?Zb&fbNiXZ`wSA*6!T&fxL@5%ub|6)+N9;(~X%%0oc*_U@JfYDyOz~PA0
zNh(zxZAqoz5<M~({cibwqa-<xeYm}uIk5TF0Jj^u_DSdXZc#LpZNm|T(L2~SIx%0D
zr~FN5U9J;YC-DDBK+OKU`;2ULe9F^Jv^2u=c|z4(-z)mcy|^smP;(_`xPJ5Vqa`Ps
zo8e+@hdA3gnIT}e`~{e1%5omGIGXNt?QqL4?5m!4O><ZVNIiRnh58MIv0yW+V5;$E
zC@S1=?Q-)t{aOdt6zz^LH2gH&<$UP1bBtdA<@?cL96g6`B*u&d4`$;WW|28byT_Cy
z-~D#|O;^>$h-3JF6oyry?<XXVURb>+E>j#lpG`UEw?qP#1vezN7g2eO#qvYh3)d0)
zpTpb;=`8TsO9V3HWC%cFD7c@5S<xo-E@_tE<oVK3852`dW}C5sOW&lD|Ge4};ZLCu
z;<zQVm}Zhc5<ifpabBC<2S;Br8Dm00dkSOE?PqDVP4U;>J^Fl0VbXtImzuDtE!cV%
zOQ(}EAFD2kEZ66+ZZ6Nrn7OSF`P#l9ncx;YubSQdZg^d-6IdtkKS!WE7X1D1l(1q5
HWGCd`W!a1I

literal 0
HcmV?d00001

diff --git a/tests/j_recover_fast_commit/script b/tests/j_recover_fast_commit/script
new file mode 100755
index 00000000..6bdb8258
--- /dev/null
+++ b/tests/j_recover_fast_commit/script
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+FSCK_OPT=-fy
+IMAGE=$test_dir/image.gz
+CMDS=$test_dir/commands
+
+gunzip < $IMAGE > $TMPFILE
+
+# Run fsck to fix things?
+EXP=$test_dir/expect
+OUT=$test_name.log
+
+cp $TMPFILE /tmp/debugthis
+$FSCK $FSCK_OPT -E journal_only -N test_filesys $TMPFILE 2>&1 | head -n 1000 | tail -n +2 > $OUT
+echo "Exit status is $?" >> $OUT
+
+$DEBUGFS -f $CMDS $TMPFILE >> $OUT 2>/dev/null
+
+# Figure out what happened
+if cmp -s $EXP $OUT; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $OUT >> $test_name.failed
+fi
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* [PATCH v3 15/15] ext4: fix tests to account for new dumpe2fs output
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (13 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 14/15] tests: add fast commit recovery tests Harshad Shirwadkar
@ 2021-01-20 21:26 ` Harshad Shirwadkar
  2021-01-21 16:01   ` Theodore Ts'o
  2021-01-21  5:22 ` [PATCH v3 00/15] Fast commit changes for e2fsprogs Theodore Ts'o
  15 siblings, 1 reply; 26+ messages in thread
From: Harshad Shirwadkar @ 2021-01-20 21:26 UTC (permalink / raw)
  To: linux-ext4; +Cc: tytso, Harshad Shirwadkar

From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

dumpe2fs tool now is capable of reporting number of fast commit
blocks. There were slight changes in the output of dumpe2fs outside of
fast commits. This patch fixes the regression tests to expect the new
output.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
---
 tests/d_corrupt_journal_nr_users/expect |  6 ++++--
 tests/f_jnl_errno/expect.0              |  6 ++++--
 tests/f_opt_extent/expect               |  2 +-
 tests/i_bitmaps/expect                  |  8 +++++---
 tests/j_ext_dumpe2fs/expect             |  6 ++++--
 tests/m_bigjournal/expect.1             |  6 ++++--
 tests/m_extent_journal/expect.1         |  6 ++++--
 tests/m_resize_inode_meta_bg/expect.1   |  6 ++++--
 tests/m_rootdir/expect                  |  6 ++++--
 tests/r_32to64bit/expect                |  6 +++---
 tests/r_32to64bit_meta/expect           |  4 ++--
 tests/r_32to64bit_move_itable/expect    |  8 ++++----
 tests/r_64to32bit/expect                |  6 +++---
 tests/r_64to32bit_meta/expect           |  4 ++--
 tests/r_move_itable_nostride/expect     |  6 ++++--
 tests/r_move_itable_realloc/expect      |  6 ++++--
 tests/t_disable_mcsum/expect            |  4 ++--
 tests/t_disable_mcsum_noinitbg/expect   |  6 +++---
 tests/t_disable_mcsum_yesinitbg/expect  |  4 ++--
 tests/t_enable_mcsum/expect             |  6 +++---
 tests/t_enable_mcsum_ext3/expect        | 10 +++++-----
 tests/t_enable_mcsum_initbg/expect      |  6 +++---
 22 files changed, 74 insertions(+), 54 deletions(-)

diff --git a/tests/d_corrupt_journal_nr_users/expect b/tests/d_corrupt_journal_nr_users/expect
index cdfb49a0..656d35c3 100644
--- a/tests/d_corrupt_journal_nr_users/expect
+++ b/tests/d_corrupt_journal_nr_users/expect
@@ -34,8 +34,10 @@ Default directory hash:   half_md4
 Journal backup:           inode blocks
 Checksum type:            crc32c
 Journal features:         (none)
-Journal size:             4096k
-Journal length:           1024
+Total journal size:       4096k
+Total journal blocks:     1024
+Max transaction length:   1024
+Fast commit length:       0
 Journal sequence:         0x00000001
 Journal start:            0
 Journal number of users:  9999
diff --git a/tests/f_jnl_errno/expect.0 b/tests/f_jnl_errno/expect.0
index 2a9426d9..96fb01a7 100644
--- a/tests/f_jnl_errno/expect.0
+++ b/tests/f_jnl_errno/expect.0
@@ -31,8 +31,10 @@ Journal inode:            8
 Default directory hash:   half_md4
 Journal backup:           inode blocks
 Journal features:         (none)
-Journal size:             1024k
-Journal length:           1024
+Total journal size:       1024k
+Total journal blocks:     1024
+Max transaction length:   1024
+Fast commit length:       0
 Journal sequence:         0x00000005
 Journal start:            0
 
diff --git a/tests/f_opt_extent/expect b/tests/f_opt_extent/expect
index 6d1f9d11..9d2021e8 100644
--- a/tests/f_opt_extent/expect
+++ b/tests/f_opt_extent/expect
@@ -35,7 +35,7 @@ Change in FS metadata:
  Free inodes:              65047
  First block:              1
  Block size:               1024
-@@ -48,8 +48,8 @@
+@@ -50,8 +50,8 @@
    Block bitmap at 262 (+261)
    Inode bitmap at 278 (+277)
    Inode table at 294-549 (+293)
diff --git a/tests/i_bitmaps/expect b/tests/i_bitmaps/expect
index fb9d8f1f..6199bb7c 100644
--- a/tests/i_bitmaps/expect
+++ b/tests/i_bitmaps/expect
@@ -1,6 +1,8 @@
-46,50d45
+46,52d45
 < Journal features:         (none)
-< Journal size:             1024k
-< Journal length:           1024
+< Total journal size:       1024k
+< Total journal blocks:     1024
+< Max transaction length:   1024
+< Fast commit length:       0
 < Journal sequence:         0x00000001
 < Journal start:            0
diff --git a/tests/j_ext_dumpe2fs/expect b/tests/j_ext_dumpe2fs/expect
index db77a36d..2838bbd1 100644
--- a/tests/j_ext_dumpe2fs/expect
+++ b/tests/j_ext_dumpe2fs/expect
@@ -43,8 +43,10 @@ Desired extra isize:      28
 Default directory hash:   half_md4
 Checksum type:            crc32c
 Journal features:         journal_64bit journal_checksum_v3
-Journal size:             2048k
-Journal length:           2048
+Total journal size:       2048k
+Total journal blocks:     2048
+Max transaction length:   2048
+Fast commit length:       0
 Journal first block:      3
 Journal sequence:         0x00000003
 Journal start:            0
diff --git a/tests/m_bigjournal/expect.1 b/tests/m_bigjournal/expect.1
index 80f71d1f..eb0e3bc3 100644
--- a/tests/m_bigjournal/expect.1
+++ b/tests/m_bigjournal/expect.1
@@ -42,8 +42,10 @@ Journal inode:            8
 Default directory hash:   half_md4
 Journal backup:           inode blocks
 Journal features:         (none)
-Journal size:             5000M
-Journal length:           1280000
+Total journal size:       5000M
+Total journal blocks:     1280000
+Max transaction length:   1280000
+Fast commit length:       0
 Journal sequence:         0x00000001
 Journal start:            0
 
diff --git a/tests/m_extent_journal/expect.1 b/tests/m_extent_journal/expect.1
index cfc052a8..9a9219e9 100644
--- a/tests/m_extent_journal/expect.1
+++ b/tests/m_extent_journal/expect.1
@@ -48,8 +48,10 @@ Journal inode:            8
 Default directory hash:   half_md4
 Journal backup:           inode blocks
 Journal features:         (none)
-Journal size:             4096k
-Journal length:           4096
+Total journal size:       4096k
+Total journal blocks:     4096
+Max transaction length:   4096
+Fast commit length:       0
 Journal sequence:         0x00000001
 Journal start:            0
 
diff --git a/tests/m_resize_inode_meta_bg/expect.1 b/tests/m_resize_inode_meta_bg/expect.1
index 6a7f3993..7feaed9d 100644
--- a/tests/m_resize_inode_meta_bg/expect.1
+++ b/tests/m_resize_inode_meta_bg/expect.1
@@ -51,8 +51,10 @@ Journal inode:            8
 Default directory hash:   half_md4
 Journal backup:           inode blocks
 Journal features:         (none)
-Journal size:             4096k
-Journal length:           1024
+Total journal size:       4096k
+Total journal blocks:     1024
+Max transaction length:   1024
+Fast commit length:       0
 Journal sequence:         0x00000001
 Journal start:            0
 
diff --git a/tests/m_rootdir/expect b/tests/m_rootdir/expect
index 113ffc64..dbc79772 100644
--- a/tests/m_rootdir/expect
+++ b/tests/m_rootdir/expect
@@ -36,8 +36,10 @@ Default directory hash:   half_md4
 Journal backup:           inode blocks
 Checksum type:            crc32c
 Journal features:         (none)
-Journal size:             1024k
-Journal length:           1024
+Total journal size:       1024k
+Total journal blocks:     1024
+Max transaction length:   1024
+Fast commit length:       0
 Journal sequence:         0x00000001
 Journal start:            0
 
diff --git a/tests/r_32to64bit/expect b/tests/r_32to64bit/expect
index c6816b7f..de573b3e 100644
--- a/tests/r_32to64bit/expect
+++ b/tests/r_32to64bit/expect
@@ -47,7 +47,7 @@ Change in FS metadata:
  Blocks per group:         8192
  Fragments per group:      8192
  Inodes per group:         1024
-@@ -41,16 +41,16 @@
+@@ -43,16 +43,16 @@
  
  
  group:block:super:gdt:bbitmap:ibitmap:itable
@@ -70,7 +70,7 @@ Change in FS metadata:
  10:81921:-1:-1:270:286:2852
  11:90113:-1:-1:271:287:3108
  12:98305:-1:-1:272:288:3364
-@@ -66,9 +66,9 @@
+@@ -68,9 +68,9 @@
  22:180225:-1:-1:131079:131095:132641
  23:188417:-1:-1:131080:131096:132897
  24:196609:-1:-1:131081:131097:133153
@@ -82,7 +82,7 @@ Change in FS metadata:
  28:229377:-1:-1:131085:131101:134177
  29:237569:-1:-1:131086:131102:134433
  30:245761:-1:-1:131087:131103:134689
-@@ -90,7 +90,7 @@
+@@ -92,7 +92,7 @@
  46:376833:-1:-1:262159:262175:265761
  47:385025:-1:-1:262160:262176:266017
  48:393217:-1:-1:393217:393233:393249
diff --git a/tests/r_32to64bit_meta/expect b/tests/r_32to64bit_meta/expect
index c4f39266..70babbd9 100644
--- a/tests/r_32to64bit_meta/expect
+++ b/tests/r_32to64bit_meta/expect
@@ -46,7 +46,7 @@ Change in FS metadata:
  Blocks per group:         8192
  Fragments per group:      8192
  Inodes per group:         1024
-@@ -55,9 +55,9 @@
+@@ -57,9 +57,9 @@
  12:98305:-1:-1:15:31:3107
  13:106497:-1:-1:16:32:3363
  14:114689:-1:-1:17:33:3619
@@ -59,7 +59,7 @@ Change in FS metadata:
  18:147457:-1:-1:131075:131091:131617
  19:155649:-1:-1:131076:131092:131873
  20:163841:-1:-1:131077:131093:132129
-@@ -87,9 +87,9 @@
+@@ -89,9 +89,9 @@
  44:360449:-1:-1:262158:262174:265250
  45:368641:-1:-1:262159:262175:265506
  46:376833:-1:-1:262160:262176:265762
diff --git a/tests/r_32to64bit_move_itable/expect b/tests/r_32to64bit_move_itable/expect
index 0a3b78e7..a1d2aebc 100644
--- a/tests/r_32to64bit_move_itable/expect
+++ b/tests/r_32to64bit_move_itable/expect
@@ -46,7 +46,7 @@ Change in FS metadata:
  Blocks per group:         8192
  Fragments per group:      8192
  Inodes per group:         1024
-@@ -39,16 +39,16 @@
+@@ -41,16 +41,16 @@
  
  
  group:block:super:gdt:bbitmap:ibitmap:itable
@@ -69,7 +69,7 @@ Change in FS metadata:
  10:81921:-1:-1:81921:81922:81923
  11:90113:-1:-1:90113:90114:90115
  12:98305:-1:-1:98305:98306:98307
-@@ -64,9 +64,9 @@
+@@ -66,9 +66,9 @@
  22:180225:-1:-1:180225:180226:180227
  23:188417:-1:-1:188417:188418:188419
  24:196609:-1:-1:196609:196610:196611
@@ -81,7 +81,7 @@ Change in FS metadata:
  28:229377:-1:-1:229377:229378:229379
  29:237569:-1:-1:237569:237570:237571
  30:245761:-1:-1:245761:245762:245763
-@@ -88,7 +88,7 @@
+@@ -90,7 +90,7 @@
  46:376833:-1:-1:376833:376834:376835
  47:385025:-1:-1:385025:385026:385027
  48:393217:-1:-1:393217:393218:393219
@@ -90,7 +90,7 @@ Change in FS metadata:
  50:409601:-1:-1:409601:409602:409603
  51:417793:-1:-1:417793:417794:417795
  52:425985:-1:-1:425985:425986:425987
-@@ -120,7 +120,7 @@
+@@ -122,7 +122,7 @@
  78:638977:-1:-1:638977:638978:638979
  79:647169:-1:-1:647169:647170:647171
  80:655361:-1:-1:655361:655362:655363
diff --git a/tests/r_64to32bit/expect b/tests/r_64to32bit/expect
index 7dff2a05..eb609fbd 100644
--- a/tests/r_64to32bit/expect
+++ b/tests/r_64to32bit/expect
@@ -46,7 +46,7 @@ Change in FS metadata:
  Reserved GDT blocks:      256
  Blocks per group:         8192
  Fragments per group:      8192
-@@ -42,16 +40,16 @@
+@@ -44,16 +42,16 @@
  
  
  group:block:super:gdt:bbitmap:ibitmap:itable
@@ -69,7 +69,7 @@ Change in FS metadata:
  10:81921:-1:-1:272:288:2854
  11:90113:-1:-1:273:289:3110
  12:98305:-1:-1:274:290:3366
-@@ -67,9 +65,9 @@
+@@ -69,9 +67,9 @@
  22:180225:-1:-1:131079:131095:132641
  23:188417:-1:-1:131080:131096:132897
  24:196609:-1:-1:131081:131097:133153
@@ -81,7 +81,7 @@ Change in FS metadata:
  28:229377:-1:-1:131085:131101:134177
  29:237569:-1:-1:131086:131102:134433
  30:245761:-1:-1:131087:131103:134689
-@@ -91,7 +89,7 @@
+@@ -93,7 +91,7 @@
  46:376833:-1:-1:262159:262175:265761
  47:385025:-1:-1:262160:262176:266017
  48:393217:-1:-1:393217:393233:393249
diff --git a/tests/r_64to32bit_meta/expect b/tests/r_64to32bit_meta/expect
index b17a8784..70655909 100644
--- a/tests/r_64to32bit_meta/expect
+++ b/tests/r_64to32bit_meta/expect
@@ -46,7 +46,7 @@ Change in FS metadata:
  Blocks per group:         8192
  Fragments per group:      8192
  Inodes per group:         1024
-@@ -56,9 +54,9 @@
+@@ -58,9 +56,9 @@
  12:98305:-1:-1:15:31:3107
  13:106497:-1:-1:16:32:3363
  14:114689:-1:-1:17:33:3619
@@ -59,7 +59,7 @@ Change in FS metadata:
  18:147457:-1:-1:131076:131092:131618
  19:155649:-1:-1:131077:131093:131874
  20:163841:-1:-1:131078:131094:132130
-@@ -88,9 +86,9 @@
+@@ -90,9 +88,9 @@
  44:360449:-1:-1:262158:262174:265250
  45:368641:-1:-1:262159:262175:265506
  46:376833:-1:-1:262160:262176:265762
diff --git a/tests/r_move_itable_nostride/expect b/tests/r_move_itable_nostride/expect
index 098cbfc5..74c2cc0a 100644
--- a/tests/r_move_itable_nostride/expect
+++ b/tests/r_move_itable_nostride/expect
@@ -52,8 +52,10 @@ Journal inode:            8
 Default directory hash:   half_md4
 Journal backup:           inode blocks
 Journal features:         (none)
-Journal size:             16M
-Journal length:           16384
+Total journal size:       16M
+Total journal blocks:     16384
+Max transaction length:   16384
+Fast commit length:       0
 Journal sequence:         0x00000001
 Journal start:            0
 
diff --git a/tests/r_move_itable_realloc/expect b/tests/r_move_itable_realloc/expect
index f9a7f515..67f2fe4a 100644
--- a/tests/r_move_itable_realloc/expect
+++ b/tests/r_move_itable_realloc/expect
@@ -51,8 +51,10 @@ Journal inode:            8
 Default directory hash:   half_md4
 Journal backup:           inode blocks
 Journal features:         (none)
-Journal size:             16M
-Journal length:           16384
+Total journal size:       16M
+Total journal blocks:     16384
+Max transaction length:   16384
+Fast commit length:       0
 Journal sequence:         0x00000001
 Journal start:            0
 
diff --git a/tests/t_disable_mcsum/expect b/tests/t_disable_mcsum/expect
index 3341ad71..bfa21b89 100644
--- a/tests/t_disable_mcsum/expect
+++ b/tests/t_disable_mcsum/expect
@@ -34,8 +34,8 @@ Change in FS metadata:
  Journal backup:           inode blocks
 -Checksum type:            crc32c
  Journal features:         (none)
- Journal size:             16M
- Journal length:           16384
+ Total journal size:       16M
+ Total journal blocks:     16384
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
diff --git a/tests/t_disable_mcsum_noinitbg/expect b/tests/t_disable_mcsum_noinitbg/expect
index 62eca4e9..fe61fcbc 100644
--- a/tests/t_disable_mcsum_noinitbg/expect
+++ b/tests/t_disable_mcsum_noinitbg/expect
@@ -34,9 +34,9 @@ Change in FS metadata:
  Journal backup:           inode blocks
 -Checksum type:            crc32c
  Journal features:         (none)
- Journal size:             16M
- Journal length:           16384
-@@ -48,18 +47,18 @@
+ Total journal size:       16M
+ Total journal blocks:     16384
+@@ -50,18 +49,18 @@
    Block bitmap at 262 (+261)
    Inode bitmap at 278 (+277)
    Inode table at 294-549 (+293)
diff --git a/tests/t_disable_mcsum_yesinitbg/expect b/tests/t_disable_mcsum_yesinitbg/expect
index 7e3485fe..b9062489 100644
--- a/tests/t_disable_mcsum_yesinitbg/expect
+++ b/tests/t_disable_mcsum_yesinitbg/expect
@@ -34,8 +34,8 @@ Change in FS metadata:
  Journal backup:           inode blocks
 -Checksum type:            crc32c
  Journal features:         (none)
- Journal size:             16M
- Journal length:           16384
+ Total journal size:       16M
+ Total journal blocks:     16384
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
diff --git a/tests/t_enable_mcsum/expect b/tests/t_enable_mcsum/expect
index cb0aef62..fcb0ed16 100644
--- a/tests/t_enable_mcsum/expect
+++ b/tests/t_enable_mcsum/expect
@@ -58,9 +58,9 @@ Change in FS metadata:
  Journal backup:           inode blocks
 +Checksum type:            crc32c
  Journal features:         (none)
- Journal size:             16M
- Journal length:           16384
-@@ -47,8 +48,8 @@
+ Total journal size:       16M
+ Total journal blocks:     16384
+@@ -49,8 +50,8 @@
    Block bitmap at 262 (+261)
    Inode bitmap at 278 (+277)
    Inode table at 294-549 (+293)
diff --git a/tests/t_enable_mcsum_ext3/expect b/tests/t_enable_mcsum_ext3/expect
index 11c5a26d..549e60e9 100644
--- a/tests/t_enable_mcsum_ext3/expect
+++ b/tests/t_enable_mcsum_ext3/expect
@@ -37,9 +37,9 @@ Change in FS metadata:
  Journal backup:           inode blocks
 +Checksum type:            crc32c
  Journal features:         (none)
- Journal size:             16M
- Journal length:           16384
-@@ -37,7 +38,7 @@
+ Total journal size:       16M
+ Total journal blocks:     16384
+@@ -39,7 +40,7 @@
  Journal start:            0
  
  
@@ -48,7 +48,7 @@ Change in FS metadata:
    Primary superblock at 1, Group descriptors at 2-3
    Reserved GDT blocks at 4-259
    Block bitmap at 260 (+259)
-@@ -46,7 +47,7 @@
+@@ -48,7 +49,7 @@
    0 free blocks, 1013 free inodes, 2 directories
    Free blocks: 
    Free inodes: 12-1024
@@ -57,7 +57,7 @@ Change in FS metadata:
    Backup superblock at 8193, Group descriptors at 8194-8195
    Reserved GDT blocks at 8196-8451
    Block bitmap at 8452 (+259)
-@@ -55,6 +56,6 @@
+@@ -57,6 +58,6 @@
    0 free blocks, 1024 free inodes, 0 directories
    Free blocks: 
    Free inodes: 1025-2048
diff --git a/tests/t_enable_mcsum_initbg/expect b/tests/t_enable_mcsum_initbg/expect
index a37648bf..987141f1 100644
--- a/tests/t_enable_mcsum_initbg/expect
+++ b/tests/t_enable_mcsum_initbg/expect
@@ -58,9 +58,9 @@ Change in FS metadata:
  Journal backup:           inode blocks
 +Checksum type:            crc32c
  Journal features:         (none)
- Journal size:             16M
- Journal length:           16384
-@@ -41,24 +42,24 @@
+ Total journal size:       16M
+ Total journal blocks:     16384
+@@ -43,24 +44,24 @@
  Journal start:            0
  
  
-- 
2.30.0.284.gd98b1dd5eaa7-goog


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 00/15] Fast commit changes for e2fsprogs
  2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
                   ` (14 preceding siblings ...)
  2021-01-20 21:26 ` [PATCH v3 15/15] ext4: fix tests to account for new dumpe2fs output Harshad Shirwadkar
@ 2021-01-21  5:22 ` Theodore Ts'o
  15 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21  5:22 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:26PM -0800, Harshad Shirwadkar wrote:
> 
> - Fix compilation error by defining jbd2_journal_get_fc_num_blks (also
>   rename it to jbd_get_fc_num_blks) as a preprocessor macro instead of
>   inline function which gets compiled out when "-g" is passed

Unfortunately, this still doesn't quite fix things.  The problem with
this approach is the preprocessor macro uses be32_to_cpu(), which gets
turned into ext2fs_swab32().  And the libe2p shared library is not
allowed to depend on libext2fs.  Otherwise when we build with
configure --enable-elf-shlibs, the link of misc/chattr fails with:

<tytso@cwcc> {/build/e2fsprogs/misc}  
1078% make V=1
../util/subst -f ../util/subst.conf /usr/projects/e2fsprogs/e2fsprogs/lib/dirpaths.h.in ../lib/dirpaths.h
gcc   -Wl,-rpath-link,../lib -o chattr chattr.o ../lib/libe2p.so ../lib/libcom_err.so  \
	 
/bin/ld: ../lib/libe2p.so: undefined reference to `ext2fs_swab32'
collect2: error: ld returned 1 exit status
make: *** [Makefile:636: chattr] Error 1

We had already defined e2p_be32() and use it in other places in
lib/e2p/ljs.c.

So I fixed this by taking your Dec 10th version of the patch series,
and making the following change to patch 4/15:

diff --git a/lib/e2p/ljs.c b/lib/e2p/ljs.c
index 9f866c7e..59728198 100644
--- a/lib/e2p/ljs.c
+++ b/lib/e2p/ljs.c
@@ -36,6 +36,17 @@ static __u32 e2p_swab32(__u32 val)
 #define e2p_be32(x) e2p_swab32(x)
 #endif
 
+/*
+ * This function is copied from kernel-jbd.h's function
+ * jbd2_journal_get_num_fc_blks() to avoid inter-library dependencies.
+ */
+static inline int get_num_fc_blks(journal_superblock_t *jsb)
+{
+	int num_fc_blocks = e2p_be32(jsb->s_num_fc_blks);
+
+	return num_fc_blocks ? num_fc_blocks : JBD2_DEFAULT_FAST_COMMIT_BLOCKS;
+}
+
 static const char *journal_checksum_type_str(__u8 type)
 {
 	switch (type) {
@@ -58,7 +69,7 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf,
 	int journal_blks = 0;
 
 	if (flags & E2P_LIST_JOURNAL_FLAG_FC)
-		num_fc_blks = jbd2_journal_get_num_fc_blks((journal_superblock_t *)journal_sb_buf);
+		num_fc_blks = get_num_fc_blks((journal_superblock_t *)journal_sb_buf);
 	journal_blks = ntohl(jsb->s_maxlen) - num_fc_blks;
 	fprintf(f, "%s", "Journal features:        ");
 	for (i=0, mask_ptr=&jsb->s_feature_compat; i <3; i++,mask_ptr++) {


^ permalink raw reply related	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 06/15] ext2fs: add new APIs needed for fast commits
  2021-01-20 21:26 ` [PATCH v3 06/15] ext2fs: add new APIs needed for fast commits Harshad Shirwadkar
@ 2021-01-21  5:58   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21  5:58 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:32PM -0800, Harshad Shirwadkar wrote:
> From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> 
> This patch adds the following new APIs:
> 
> Count the total number of blocks occupied by inode including
> intermediate extent tree nodes.
> extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
>                                        struct ext2_inode *inode);

I wonder if this should be something like this instead:

extern errcode_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino,
                                     struct ext2_inode *inode, blk64_t *ret_count);

The problem is that ext2fs_count_blocks() calls a whole series of
ext2fs functions which could return errors:

> +	errcode = ext2fs_extent_open2(fs, ino, inode, &handle);
> +	if (errcode)
> +		goto out;
> +
> +	errcode = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
> +	if (errcode)
> +		goto out;

... and any of these functions could return an error.  So we need to
make sure errors are faithfully returned to the caller and handled
correctly, instead of just having ext2fs_count_blocks returning a
value of 0.


I then started taking a look at the users of ext2fs_count_blocks() in
e2fsck, and I ran into more concerns.  One of the problems here is
that some of these functions get called by kernel code --- and kernel
code has a different error handling convetion of negative errno's.

And in some cases, I see we are doing this:

static int ext4_fc_handle_inode(e2fsck_t ctx, struct ext4_fc_tl *tl)
{
	...
	
	ret = ext2fs_read_inode_full(ctx->fs, ino, (struct ext2_inode *)inode,
					inode_len);
	if (ret)
		goto out;
	...
out:
	ext2fs_free_mem(&inode);
	return ret;
}

The problem here is that ext2fs_read_inode_full() returns an
errcode_t, and this is getting cast to an int and returned as if it
were a kernel error code.

Also note that ext4_fc_replay() can return 0 or 1:

#define JBD2_FC_REPLAY_STOP		0
#define JBD2_FC_REPLAY_CONTINUE		1

Fortunately, none of the functions that ext4_fc_*() call seem to be
ones which could return in an ext2fs library returning EPERM (which is
errno 1), but you see the potential risks of conflating an errcode_t
and kernel negative errno convention.

This is going to be a bit tricky to deal with, since an errcode_t can
be a errno code, but it can also be one of the codes defined in
lib/ext2fs/ext2_err.et, which get translated to numbers like:

#define EXT2_ET_DIR_CORRUPTED                    (2133571363L)
#define EXT2_ET_SHORT_READ                       (2133571364L)
#define EXT2_ET_SHORT_WRITE                      (2133571365L)

(See lib/ext2fs/ext2_err.h in the build directory of e2fsprogs and the
com_err library found in lib/et.)

So what we may need to do is to create a function which does a simple
mapping of errcode_t values to negative errno's.  It doesn't need to
be exact; in fact, a first pass might just map all errcode_t's greater
than 256 to something like -EFAULT, and all normal errno's to -errno.

We might also want to have it print a diagnistic message to stderr
that prints error_message(retval) was encoutered in function __func__
at line __LINE__.  Hopefully in actual practice they won't happen
(unless a malicious attacker is feeding us a fuzzed file sytem), but
if it does, it would be good if there is a useful message so we can
actually debug what happened.

      	   	  	     	    	     - Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 01/15] ext2fs: move calculate_summary_stats to ext2fs lib
  2021-01-20 21:26 ` [PATCH v3 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
@ 2021-01-21 15:54   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21 15:54 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:27PM -0800, Harshad Shirwadkar wrote:
> From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> 
> The function calculate_summary_stats sets the global metadata of the
> file system. Tune2fs had this function defined statically in
> tune2fs.c. Fast commit replay needs this function to set global
> metadata at the end of the replay phase. So, move this function to
> libext2fs.
> 
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> Reviewed-by: Theodore Ts'o <tytso@mit.edu>

Thanks, applied.

						- Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 02/15] e2fsck: add kernel endian-ness conversion macros
  2021-01-20 21:26 ` [PATCH v3 02/15] e2fsck: add kernel endian-ness conversion macros Harshad Shirwadkar
@ 2021-01-21 15:54   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21 15:54 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4, Andreas Dilger

On Wed, Jan 20, 2021 at 01:26:28PM -0800, Harshad Shirwadkar wrote:
> From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> 
> In order to make recovery.c identical with kernel, we need endianness
> conversion macros (such as cpu_to_be32 and friends) defined in
> e2fsprogs. This patch defines these macros and also fixes recovery.c
> to use these. These macros are also needed for fast commit recovery
> patches later in this series.
> 
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> Reviewed-by: Andreas Dilger <adilger@dilger.ca>

Thanks, applied.

					- Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck
  2021-01-20 21:26 ` [PATCH v3 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Harshad Shirwadkar
@ 2021-01-21 15:54   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21 15:54 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:29PM -0800, Harshad Shirwadkar wrote:
> From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> 
> This patch makes recovery.c identical with fast commit kernel changes.
> 
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> Reviewed-by: Theodore Ts'o <tytso@mit.edu>

Thanks, applied.

					- Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 04/15] libext2fs: provide APIs to configure fast commit blocks
  2021-01-20 21:26 ` [PATCH v3 04/15] libext2fs: provide APIs to configure fast commit blocks Harshad Shirwadkar
@ 2021-01-21 15:56   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21 15:56 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:30PM -0800, Harshad Shirwadkar wrote:
> From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> 
> This patch adds new libext2fs that allow configuring number of fast
> commit blocks in journal superblock. We also add a struct
> ext2fs_journal_params which contains number of fast commit blocks and
> number of normal journal blocks. With this patch, the preferred way
> for configuring number of blocks with and without fast commits is:
> 
> struct ext2fs_journal_params params;
> 
> ext2fs_get_journal_params(&params, ...);
> params.num_journal_blocks = ...;
> params.num_fc_blocks = ...;
> ext2fs_create_journal_superblock2(..., &params, ...);
>          OR
> ext2fs_add_journal_inode3(..., &params, ...);
> 
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

I've applied to the next branch a modified version of this which is
based on the Dec. 20th version of this change, with a local static
inline copy of jbd2_journal_get_num_fc_blks() in ljs.c, as I had
mentioned earlier.

			       	  	    - Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 05/15] e2fsprogs: make userspace tools number of fast commits blocks aware
  2021-01-20 21:26 ` [PATCH v3 05/15] e2fsprogs: make userspace tools number of fast commits blocks aware Harshad Shirwadkar
@ 2021-01-21 15:58   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21 15:58 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:31PM -0800, Harshad Shirwadkar wrote:
> From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> 
> This patch makes number of fast commit blocks configurable. Also, the
> number of fast commit blocks can now be seen in dumpe2fs output.
> 
> $ ./misc/mke2fs -O fast_commit -t ext4 image
> mke2fs 1.46-WIP (20-Mar-2020)
> Discarding device blocks: done
> Creating filesystem with 5120 1k blocks and 1280 inodes
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (1040 blocks): done
> Writing superblocks and filesystem accounting information: done
> 
> $ ./misc/dumpe2fs image
> dumpe2fs 1.46-WIP (20-Mar-2020)
> ...
> Journal features:         (none)
> Total journal size:       1040k
> Total journal blocks:     1040
> Max transaction length:   1024
> Fast commit length:       16
> Journal sequence:         0x00000001
> Journal start:            0
> 
> $ ./misc/mke2fs -O fast_commit -t ext4 image -J fast_commit_size=256,size=1
> mke2fs 1.46-WIP (20-Mar-2020)
> Creating filesystem with 5120 1k blocks and 1280 inodes
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (1280 blocks): done
> Writing superblocks and filesystem accounting information: done
> 
> $ ./misc/dumpe2fs image
> dumpe2fs 1.46-WIP (20-Mar-2020)
> ...
> Journal features:         (none)
> Total journal size:       1280k
> Total journal blocks:     1280
> Max transaction length:   1024
> Fast commit length:       256
> Journal sequence:         0x00000001
> Journal start:            0
> 
> This patch also adds information about fast commit feature in mke2fs
> and tune2fs man pages.
> 
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Thanks, applied.

						- Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 15/15] ext4: fix tests to account for new dumpe2fs output
  2021-01-20 21:26 ` [PATCH v3 15/15] ext4: fix tests to account for new dumpe2fs output Harshad Shirwadkar
@ 2021-01-21 16:01   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21 16:01 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:41PM -0800, Harshad Shirwadkar wrote:
> From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> 
> dumpe2fs tool now is capable of reporting number of fast commit
> blocks. There were slight changes in the output of dumpe2fs outside of
> fast commits. This patch fixes the regression tests to expect the new
> output.
> 
> Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>

Thanks. applied.  I was tempted to merge this with patch #5, which
contains the changes that made the test results necessary, but keeping
this adjacent with patch #5 should be good enough for people who might
need to run bisects.

				- Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 13/15] debugfs: add fast commit support to logdump
  2021-01-20 21:26 ` [PATCH v3 13/15] debugfs: add fast commit support to logdump Harshad Shirwadkar
@ 2021-01-21 16:49   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21 16:49 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:39PM -0800, Harshad Shirwadkar wrote:
> From: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
> 
> Add fast commit support for debugfs logdump. The debugfs output looks
> like this:

I've applied this patch set, with slight changes.  In particular this
commit relies on lib/ext2fs/fast_commit.h which is byte-for-byte
identical with the kernel fast_commit.h, and which originally added in
patch #8 of this series ("e2fsck: add fast commit setup code"), so
I've added that file to this commit.

					- Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
* Re: [PATCH v3 08/15] e2fsck: add fast commit setup code
  2021-01-20 21:26 ` [PATCH v3 08/15] e2fsck: add fast commit setup code Harshad Shirwadkar
@ 2021-01-21 16:54   ` Theodore Ts'o
  0 siblings, 0 replies; 26+ messages in thread
From: Theodore Ts'o @ 2021-01-21 16:54 UTC (permalink / raw)
  To: Harshad Shirwadkar; +Cc: linux-ext4

On Wed, Jan 20, 2021 at 01:26:34PM -0800, Harshad Shirwadkar wrote:
> diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
> index fdcb28f6..eb2e6549 100644
> --- a/lib/ext2fs/ext2fs.h
> +++ b/lib/ext2fs/ext2fs.h
> @@ -2148,4 +2148,7 @@ static inline unsigned int ext2_dir_htree_level(ext2_filsys fs)
>  }
>  #endif
>  
> +/* Commonly used helpers */
> +#define max(a, b) ((a) > (b) ? (a) : (b))
> +
>  #endif /* _EXT2FS_EXT2FS_H */

It's better not to add this to a publically exported file, such as
lib/ext2fs/ext2fs.h, since it may conflict with other userspace
application which may define their own max file.

Something which we might want to do is to add something like
linux/minmax.h from the kernel sources to libsupport.  This defines a
max() macro which does paranoid typechecking to make sure we don't
accidentally compare an unsigned int to an signed int, which can
potential security problems with maliciously fuzzed file system
images.  :-)

						- Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread
end of thread, other threads:[~2021-01-21 16:56 UTC | newest]

Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-20 21:26 [PATCH v3 00/15] Fast commit changes for e2fsprogs Harshad Shirwadkar
2021-01-20 21:26 ` [PATCH v3 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Harshad Shirwadkar
2021-01-21 15:54   ` Theodore Ts'o
2021-01-20 21:26 ` [PATCH v3 02/15] e2fsck: add kernel endian-ness conversion macros Harshad Shirwadkar
2021-01-21 15:54   ` Theodore Ts'o
2021-01-20 21:26 ` [PATCH v3 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Harshad Shirwadkar
2021-01-21 15:54   ` Theodore Ts'o
2021-01-20 21:26 ` [PATCH v3 04/15] libext2fs: provide APIs to configure fast commit blocks Harshad Shirwadkar
2021-01-21 15:56   ` Theodore Ts'o
2021-01-20 21:26 ` [PATCH v3 05/15] e2fsprogs: make userspace tools number of fast commits blocks aware Harshad Shirwadkar
2021-01-21 15:58   ` Theodore Ts'o
2021-01-20 21:26 ` [PATCH v3 06/15] ext2fs: add new APIs needed for fast commits Harshad Shirwadkar
2021-01-21  5:58   ` Theodore Ts'o
2021-01-20 21:26 ` [PATCH v3 07/15] e2fsck: add function to rewrite extent tree Harshad Shirwadkar
2021-01-20 21:26 ` [PATCH v3 08/15] e2fsck: add fast commit setup code Harshad Shirwadkar
2021-01-21 16:54   ` Theodore Ts'o
2021-01-20 21:26 ` [PATCH v3 09/15] e2fsck: add fast commit scan pass Harshad Shirwadkar
2021-01-20 21:26 ` [PATCH v3 10/15] e2fsck: add fast commit replay skeleton Harshad Shirwadkar
2021-01-20 21:26 ` [PATCH v3 11/15] e2fsck: add fc replay for link, unlink, creat tags Harshad Shirwadkar
2021-01-20 21:26 ` [PATCH v3 12/15] e2fsck: add replay for add_range, del_range, and inode tags Harshad Shirwadkar
2021-01-20 21:26 ` [PATCH v3 13/15] debugfs: add fast commit support to logdump Harshad Shirwadkar
2021-01-21 16:49   ` Theodore Ts'o
2021-01-20 21:26 ` [PATCH v3 14/15] tests: add fast commit recovery tests Harshad Shirwadkar
2021-01-20 21:26 ` [PATCH v3 15/15] ext4: fix tests to account for new dumpe2fs output Harshad Shirwadkar
2021-01-21 16:01   ` Theodore Ts'o
2021-01-21  5:22 ` [PATCH v3 00/15] Fast commit changes for e2fsprogs Theodore Ts'o

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.