[PATCH 00/54] e2fsprogs January 2015 patchbomb

[PATCH 00/54] e2fsprogs January 2015 patchbomb

[Darrick J. Wong]

Hi all,

Here's the monthly patchbomb for 1.43.  As usual, I've fixed a number
of bugs discovered since the last patchbomb in December, enhanced the
e2fsck peformance improvements, continued improving the rewrite of
e2undo, fixed up the mke2fs -d functionality to allow for the creation
of minified readonly images, and revised all the new features for 1.43
per our discussions here.

The first nineteen patches fix documentation problems and various
fuzzer bugs in the library, resize2fs, tune2fs, and e2fsck.

Patch 20 implements a "yes to all" response to e2fsck's prompts.  The
user will be notified of the existence of yes-to-all if the 'y'
response is given more than eight times consecutively.

Patches 21-22 are enhancements to the library -- partial zero-out
support (I've decided not to support block device zero-out until I get
that mess cleaned up); and block allocator hook fixes for e2fsck so it
doesn't accidentally crosslink blocks.  These are unchanged from
previous months.

Patch 23 provides a new 'readonly' rocompat feature.  This new feature
flag can be used to create ext4 images for containers where neither
the kernel nor the userspace tools (except tune2fs) can write
anything to the filesystem.

Patches 24-25 are the e2fsck metadata readahead patches, unchanged
from previous months.

Patch 26 changes e2fsck to use a bitmap instead of a u32 list when
building the list of directories to rehash.  This enables some code
cleanup and makes it so we can free the dirinfo structure earlier.
No changes from December.

Patches 27-28 rebuild extent trees.  This can be used to convert block
mapped files to extent files (-E bmap2extent), and it can also detect
sparse extent trees that could be reduced in size by either a full ETB
block or a full level.  The code is now smart enough to put off
detecting and rebuilding the extent trees of directories that are
going to be rehashed in part 3A until after the rehash because the
rehash process can shrink a directory enough to trigger the rebuilder
during the next e2fsck run.  No changes from December.

Patches 29-32 prepare the undo IO manager and e2undo for heavier use
by adding discard, zeroout, and readahead call pass through support;
allow user programs to provide an undo IO file block size that differs
from the filesystem; and speeds up block writeout considerably by
tracking which blocks we've already written in a bitmap (instead of
repeatedly bashing on the tdb keystore).

Patch 33 replaces e2undo's tdb file with a dumb flat file format,
which greatly improves the insane performance losses when using undo
files while fixing a lot of endianness bugs, database size
limitations, and the totally broken detection of undo files that
should not be applied to the filesystem.

Patch 34 implements atexit() handlers so that the undo IO manager has
a chance to finish writing the undo file if the program exits without
explicitly cleaning up the IO managers.

Patches 35-41 enable the creation of e2undo files for all modern
e2fsprogs utilities and adds simple test cases for e2undo and supply
test cases for the new undo features.

Patches 42-46 fix some bugs in the copy-in support for mke2fs and
change the file copy-in algorithm to use SEEK_DATA and SEEK_HOLE to
skip pointless reads on sparse files.  Rudimentary feature testing is
provided, and I added a contrib/ script to generate the minimum-sized
ext4 image of a particular directory.

Patches 47-53 are new API calls in the library, primarily to support
the new fallocate feature in patch 27.  This isn't really new; these
patches have been out for review for quite some time.  There are two
new patches to add rudimentary tests for fallocate and punch.  No
changes from previous months.

Patch 54 implements fuse2fs, a FUSE server based on libext2fs.
Primarily I've been using it to shake out bugs in the library via
xfstests and the metadata checksumming test program.  It can also be
used to mount ext4 on any OS supporting FUSE, and it can also mount
64k-block filesystems on x86, though I'd be wary of using rw mode.
fuse2fs depends on these new APIs: xattr editing, uninit extent
handling, and the new fallocate call.  No changes from previous
months.

I've tested these e2fsprogs changes against the -next branch as of
12/15, though the patches have been rebased to reflect the minor
changes in this morning's -next.  The patches have been tested against
the 'make check' suite and a week's worth of e2fuzz testing on x86_64,
ppc64, armv7l, i386, and aarch64.  Github, for crazy testers:
https://github.com/djwong/e2fsprogs/commits/next

Comments and questions are, as always, welcome.

--D

[PATCH 01/54] misc: fix minor testcase problems

[Darrick J. Wong]

Don't write debugfs headers to stdout...

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/f_badcluster/script |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/tests/f_badcluster/script b/tests/f_badcluster/script
index 56b4b7f..ea9e488 100644
--- a/tests/f_badcluster/script
+++ b/tests/f_badcluster/script
@@ -8,7 +8,7 @@ if test -x $DEBUGFS_EXE; then
 	$FSCK -fy $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
 	for i in a b c d e f g; do echo "stat /$i"; done > $TMPFILE.tmp
 	echo "quit" >> $TMPFILE.tmp
-	$DEBUGFS_EXE -f $TMPFILE.tmp $TMPFILE >> $OUT
+	$DEBUGFS_EXE -f $TMPFILE.tmp $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
 	rm -f $TMPFILE.tmp
 
 	cmp -s $OUT $EXP

[PATCH 02/54] debugfs: document new commands

[Darrick J. Wong]

Document the new journal and xattr commands in the debugfs manpage.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debugfs/debugfs.8.in |   47 ++++++++++++++++++++++++++++++++++++++++++++++-
 debugfs/do_journal.c |    3 ++-
 2 files changed, 48 insertions(+), 2 deletions(-)


diff --git a/debugfs/debugfs.8.in b/debugfs/debugfs.8.in
index 8f44ced..04c280d 100644
--- a/debugfs/debugfs.8.in
+++ b/debugfs/debugfs.8.in
@@ -262,6 +262,32 @@ not stored in filesystem data structures.   Hence, the values displayed
 may not necessarily by accurate and does not indicate a problem or
 corruption in the file system.)
 .TP
+.BI ea_get " [-f outfile] filespec attr_name"
+Retrieve the value of the extended attribute
+.I attr_name
+in the file
+.I filespec
+and write it either to stdout or to \fIoutfile\fR.
+.TP
+.BI ea_list " filespec
+List the extended attributes associated with the file
+.I filespec
+to standard output.
+.TP
+.BI ea_set " [-f infile] filespec attr_name attr_value
+Set the value of the extended attribute
+.I attr_name
+in the file
+.I filespec
+to the string value
+.I attr_value
+or read it from \fIinfile\fR.
+.TP
+.BI ea_rm " filespec attr_names...
+Remove the extended attribute
+.I attr_name
+from the file \fIfilespec\fR.
+.TP
 .BI expand_dir " filespec"
 Expand the directory
 .IR filespec .
@@ -373,6 +399,26 @@ to do this, use the
 program.  This is just a call to the low-level library, which sets up
 the superblock and block descriptors.
 .TP
+.BI journal_close
+Close the open journal.
+.TP
+.BI journal_open " [-c] [-v ver] [-j ext_jnl]
+Opens the journal for reading and writing.  Journal checksumming can
+be enabled by supplying \fI-c\fR; checksum formats 2 and 3 can be
+selected with the \fI-v\fR option.  An external journal can be loaded
+from \fIext_jnl\fR.
+.TP
+.BI journal_run
+Replay all transactions in the open journal.
+.TP
+.BI journal_write " [-b blocks] [-r revoke] [-c] file
+Write a transaction to the open journal.  The list of blocks to write
+should be supplied as a comma-separated list in \fIblocks\fR; the
+blocks themselves should be readable from \fIfile\fR.  A list of
+blocks to revoke can be supplied as a comma-separated list in
+\fIrevoke\fR.  By default, a commit record is written at the end; the
+\fI-c\fR switch writes an uncommitted transaction.
+.TP
 .BI kill_file " filespec"
 Deallocate the inode
 .I filespec
@@ -667,7 +713,6 @@ into a newly-created file in the filesystem named
 .IR out_file .
 .TP
 .BI zap_block " [-f filespec] [-o offset] [-l length] [-p pattern] block_num"
-.TP
 Overwrite the block specified by
 .I block_num
 with zero (NUL) bytes, or if
diff --git a/debugfs/do_journal.c b/debugfs/do_journal.c
index a17af6e..46d1793 100644
--- a/debugfs/do_journal.c
+++ b/debugfs/do_journal.c
@@ -925,9 +925,10 @@ void do_journal_open(int argc, char *argv[])
 			}
 			break;
 		default:
-			printf("%s: [-c] [-v ver]\n", argv[0]);
+			printf("%s: [-c] [-v ver] [-f ext_jnl]\n", argv[0]);
 			printf("-c: Enable journal checksumming.\n");
 			printf("-v: Use this version checksum format.\n");
+			printf("-j: Load this external journal.\n");
 		}
 	}
 

[PATCH 03/54] debugfs: fix crash in ea_set argument handling

[Darrick J. Wong]

Fix an incorrect check in ea_set that would crash debugfs if someone
runs 'ea_set / foo.bar' (i.e. with no value argument)

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debugfs/xattrs.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/debugfs/xattrs.c b/debugfs/xattrs.c
index a80fd0b..e71bc15 100644
--- a/debugfs/xattrs.c
+++ b/debugfs/xattrs.c
@@ -202,7 +202,7 @@ void do_set_xattr(int argc, char **argv)
 		}
 	}
 
-	if (optind != argc - 2 && optind != argc - 3) {
+	if (!(fp && optind == argc - 2) && !(!fp && optind == argc - 3)) {
 		printf("%s: Usage: %s <file> <attr> [-f infile | value>]\n",
 		       argv[0], argv[0]);
 		goto out2;

[PATCH 04/54] libext2fs: initialize i_extra_isize when writing EAs

[Darrick J. Wong]

If i_extra_isize is zero when we try to write extended attributes,
we'll end up writing the EA magic into the i_extra_isize field, which
causes a subsequent crash on big endian systems (when we try to write
0xEA02 bytes past the inode!).  Therefore when the field is zero, set
i_extra_isize to the desired extra_isize size, zero those bytes, and
write the EAs after the end of the extended inode.

v2: Don't bother if we have 128b inodes, and ensure that the value
is 32b-aligned so that the EA magic starts on a 32b boundary.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/ext_attr.c                    |   12 ++++++++++++
 tests/f_write_ea_no_extra_isize/expect.1 |   12 ++++++++++++
 tests/f_write_ea_no_extra_isize/expect.2 |    7 +++++++
 tests/f_write_ea_no_extra_isize/image.gz |  Bin
 tests/f_write_ea_no_extra_isize/name     |    1 +
 5 files changed, 32 insertions(+)
 create mode 100644 tests/f_write_ea_no_extra_isize/expect.1
 create mode 100644 tests/f_write_ea_no_extra_isize/expect.2
 create mode 100644 tests/f_write_ea_no_extra_isize/image.gz
 create mode 100644 tests/f_write_ea_no_extra_isize/name


diff --git a/lib/ext2fs/ext_attr.c b/lib/ext2fs/ext_attr.c
index 70bc3f9..c6fcf54 100644
--- a/lib/ext2fs/ext_attr.c
+++ b/lib/ext2fs/ext_attr.c
@@ -519,6 +519,18 @@ errcode_t ext2fs_xattrs_write(struct ext2_xattr_handle *handle)
 	if (err)
 		goto out;
 
+	/* If extra_isize isn't set, we need to set it now */
+	if (inode->i_extra_isize == 0 &&
+	    EXT2_INODE_SIZE(handle->fs->super) > EXT2_GOOD_OLD_INODE_SIZE) {
+		char *p = (char *)inode;
+		size_t extra = handle->fs->super->s_want_extra_isize;
+
+		if (extra == 0)
+			extra = sizeof(__u32);
+		memset(p + EXT2_GOOD_OLD_INODE_SIZE, 0, extra);
+		inode->i_extra_isize = extra;
+	}
+
 	move_inline_data_to_front(handle);
 
 	x = handle->attrs;
diff --git a/tests/f_write_ea_no_extra_isize/expect.1 b/tests/f_write_ea_no_extra_isize/expect.1
new file mode 100644
index 0000000..b7e7438
--- /dev/null
+++ b/tests/f_write_ea_no_extra_isize/expect.1
@@ -0,0 +1,12 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Directory inode 12, block #0, offset 4: directory corrupted
+Salvage? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 1
diff --git a/tests/f_write_ea_no_extra_isize/expect.2 b/tests/f_write_ea_no_extra_isize/expect.2
new file mode 100644
index 0000000..3b6073e
--- /dev/null
+++ b/tests/f_write_ea_no_extra_isize/expect.2
@@ -0,0 +1,7 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 0
diff --git a/tests/f_write_ea_no_extra_isize/image.gz b/tests/f_write_ea_no_extra_isize/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..928daff1f344824d357e816883a98b2cdfdaffb3
GIT binary patch
literal 2516
zcmb2|=3qFkI6Z`k`Ry&+Y!OEZh6m-}^`s^_@O3Vjpj4;eVQ?ceQSj)oL#Gnz1=cLm
zv~lFfFtI=2=98SCs6XwXQ}-6Ju%@X>9fI9126HxlZ#2Be**0UwlghsG_L*~cr<Q$x
zcJJ<Oj)e8Ibj2!<JPfHhr<Aq!+aA$gzRhQoDpTqfm88@ubyxlMdNpU)@t~~i`mk$S
zzoVyaURhCSpQJX`e|`A<yOq_6FK*5jUa#jL8+)&B<F~W5)<2(II(+ff@7nX~@@wXt
zO0r@5^laDG$2W8iznog0p}+h^;VVXl4ZK}{zBqI(?)|f0Y^ht>t&nbG@wbW$3=B7Z
zzWX-qadqb7_v@K~-2ZzY{r}&1;?u?qJ8lDkN;|V_HErb@zdo&eIeY8H?91Epe8gTW
z#qp>8%-!w_RG#4e@_o_Y_#Xxt|AD*%>jF1&gXpjJK#~;jC;IEb(o&#~gzJUuETrf}
zRok%tV=6ElWvesq%s;o>RW(Vqw|cASS<|O4a;NN_xHR?tb)Bt$LbgmbOZ<IUQ%o$p
zUwq@2A5YK3-Tr^z+swV;J`Rs%pZ#BB`ad`3|E{WQssEz?TR&NG;97*nr}>xXe%f!t
zxAoKf7Zd(Zk&?Xh?1}x|{r`U+Uwh*Jxur#4xBt=4FJGCGo%NMBXYuokNl*KaC%k*G
zu}tK){;6+2PXDfdYaO{ZbnaGE_m1*LLtr!nMnhmU1V%$(Gz6#@0(bv1g#8Yk!N8!v
F003fX5hwrv

literal 0
HcmV?d00001

diff --git a/tests/f_write_ea_no_extra_isize/name b/tests/f_write_ea_no_extra_isize/name
new file mode 100644
index 0000000..200e365
--- /dev/null
+++ b/tests/f_write_ea_no_extra_isize/name
@@ -0,0 +1 @@
+write EA when i_extra_size is zero

[PATCH 05/54] libext2fs: avoid pointless EA block allocation

[Darrick J. Wong]

Use qsort to move the inlinedata attribute to the front of the list
and the empty entries to the end.  Then we can use handle->count to
decide if we're done writing xattrs, which helps us to avoid the
situation where we're midway through the attribute list, so we
allocate an EA block to store more, but have no idea that there's
actually nothing left in the list.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/ext_attr.c |   40 +++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 19 deletions(-)


diff --git a/lib/ext2fs/ext_attr.c b/lib/ext2fs/ext_attr.c
index c6fcf54..e8544dc 100644
--- a/lib/ext2fs/ext_attr.c
+++ b/lib/ext2fs/ext_attr.c
@@ -254,22 +254,20 @@ static struct ea_name_index ea_names[] = {
 	{0, NULL},
 };
 
-static void move_inline_data_to_front(struct ext2_xattr_handle *h)
+/* Push empty attributes to the end and inlinedata to the front. */
+static int attr_compare(const void *a, const void *b)
 {
-	struct ext2_xattr *x;
-	struct ext2_xattr tmp;
-
-	for (x = h->attrs + 1; x < h->attrs + h->length; x++) {
-		if (!x->name)
-			continue;
-
-		if (strcmp(x->name, "system.data") == 0) {
-			memcpy(&tmp, x, sizeof(tmp));
-			memcpy(x, h->attrs, sizeof(tmp));
-			memcpy(h->attrs, &tmp, sizeof(tmp));
-			return;
-		}
-	}
+	const struct ext2_xattr *xa = a, *xb = b;
+
+	if (xa->name == NULL)
+		return +1;
+	else if (xb->name == NULL)
+		return -1;
+	else if (!strcmp(xa->name, "system.data"))
+		return -1;
+	else if (!strcmp(xb->name, "system.data"))
+		return +1;
+	return 0;
 }
 
 static const char *find_ea_prefix(int index)
@@ -531,9 +529,13 @@ errcode_t ext2fs_xattrs_write(struct ext2_xattr_handle *handle)
 		inode->i_extra_isize = extra;
 	}
 
-	move_inline_data_to_front(handle);
-
+	/*
+	 * Force the inlinedata attr to the front and the empty entries
+	 * to the end.
+	 */
 	x = handle->attrs;
+	qsort(x, handle->length, sizeof(struct ext2_xattr), attr_compare);
+
 	/* Does the inode have size for EA? */
 	if (EXT2_INODE_SIZE(handle->fs->super) <= EXT2_GOOD_OLD_INODE_SIZE +
 						  inode->i_extra_isize +
@@ -554,11 +556,11 @@ errcode_t ext2fs_xattrs_write(struct ext2_xattr_handle *handle)
 	if (err)
 		goto out;
 
+write_ea_block:
 	/* Are we done? */
-	if (x == handle->attrs + handle->length)
+	if (x >= handle->attrs + handle->count)
 		goto skip_ea_block;
 
-write_ea_block:
 	/* Write the EA block */
 	err = ext2fs_get_memzero(handle->fs->blocksize, &block_buf);
 	if (err)

[PATCH 06/54] libext2fs: strengthen i_extra_isize checks when reading/writing xattrs

[Darrick J. Wong]

Strengthen the i_extra_isize checks to look for obviously too-small
values before trying to operate on inode EAs.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/ext_attr.c                          |   10 ++++++----
 tests/f_write_ea_toobig_extra_isize/expect.1   |   12 ++++++++++++
 tests/f_write_ea_toobig_extra_isize/expect.2   |    7 +++++++
 tests/f_write_ea_toobig_extra_isize/image.gz   |  Bin
 tests/f_write_ea_toobig_extra_isize/name       |    1 +
 tests/f_write_ea_toosmall_extra_isize/expect.1 |   15 +++++++++++++++
 tests/f_write_ea_toosmall_extra_isize/expect.2 |    7 +++++++
 tests/f_write_ea_toosmall_extra_isize/image.gz |  Bin
 tests/f_write_ea_toosmall_extra_isize/name     |    1 +
 9 files changed, 49 insertions(+), 4 deletions(-)
 create mode 100644 tests/f_write_ea_toobig_extra_isize/expect.1
 create mode 100644 tests/f_write_ea_toobig_extra_isize/expect.2
 create mode 100644 tests/f_write_ea_toobig_extra_isize/image.gz
 create mode 100644 tests/f_write_ea_toobig_extra_isize/name
 create mode 100644 tests/f_write_ea_toosmall_extra_isize/expect.1
 create mode 100644 tests/f_write_ea_toosmall_extra_isize/expect.2
 create mode 100644 tests/f_write_ea_toosmall_extra_isize/image.gz
 create mode 100644 tests/f_write_ea_toosmall_extra_isize/name


diff --git a/lib/ext2fs/ext_attr.c b/lib/ext2fs/ext_attr.c
index e8544dc..df512d8 100644
--- a/lib/ext2fs/ext_attr.c
+++ b/lib/ext2fs/ext_attr.c
@@ -536,8 +536,9 @@ errcode_t ext2fs_xattrs_write(struct ext2_xattr_handle *handle)
 	x = handle->attrs;
 	qsort(x, handle->length, sizeof(struct ext2_xattr), attr_compare);
 
-	/* Does the inode have size for EA? */
-	if (EXT2_INODE_SIZE(handle->fs->super) <= EXT2_GOOD_OLD_INODE_SIZE +
+	/* Does the inode have space for EA? */
+	if (inode->i_extra_isize < sizeof(inode->i_extra_isize) ||
+	    EXT2_INODE_SIZE(handle->fs->super) <= EXT2_GOOD_OLD_INODE_SIZE +
 						  inode->i_extra_isize +
 						  sizeof(__u32))
 		goto write_ea_block;
@@ -773,8 +774,9 @@ errcode_t ext2fs_xattrs_read(struct ext2_xattr_handle *handle)
 
 	xattrs_free_keys(handle);
 
-	/* Does the inode have size for EA? */
-	if (EXT2_INODE_SIZE(handle->fs->super) <= EXT2_GOOD_OLD_INODE_SIZE +
+	/* Does the inode have space for EA? */
+	if (inode->i_extra_isize < sizeof(inode->i_extra_isize) ||
+	    EXT2_INODE_SIZE(handle->fs->super) <= EXT2_GOOD_OLD_INODE_SIZE +
 						  inode->i_extra_isize +
 						  sizeof(__u32))
 		goto read_ea_block;
diff --git a/tests/f_write_ea_toobig_extra_isize/expect.1 b/tests/f_write_ea_toobig_extra_isize/expect.1
new file mode 100644
index 0000000..b7e7438
--- /dev/null
+++ b/tests/f_write_ea_toobig_extra_isize/expect.1
@@ -0,0 +1,12 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Directory inode 12, block #0, offset 4: directory corrupted
+Salvage? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 1
diff --git a/tests/f_write_ea_toobig_extra_isize/expect.2 b/tests/f_write_ea_toobig_extra_isize/expect.2
new file mode 100644
index 0000000..3b6073e
--- /dev/null
+++ b/tests/f_write_ea_toobig_extra_isize/expect.2
@@ -0,0 +1,7 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 0
diff --git a/tests/f_write_ea_toobig_extra_isize/image.gz b/tests/f_write_ea_toobig_extra_isize/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..291924bf62477e5f9f18c198c9d478972590f345
GIT binary patch
literal 2518
zcmb2|=3tmxGd+Zf`Ry&+Y!OEZh6m-}^`s^_@O3Vjpj4;eVQ?ceQSj)oL#Gnz1=cLm
zv~lD(l2O;tJLA~BLvcFItzwzFYgn{h1(d})6D-o-GbiuS<}oxp`B`T3;hH<T#=FhG
zzq{wg@cU}a#*@l@%ieg41!iA;zV^xsgX?F)+{Dzpj*F>nT5)~-(sL`Tb(Zh?mY%wH
z&;C%0`J1NmE2Rbfdz1F{@0;D;&gbvXtEw%1_3M;h&({2Zzh5p7m+K9)m#HxPS@6|I
z?6Khl^V`?@lUMUB55Ko?-<+WSV19-M*?=R<+k>p`b^Ol`%yL=oyVaunCJ!S6!-03@
z>$l6;d(Ez|mt|mJ`0-x!|9^g$Y5lIBBO4k|eBQY6neiUyi_`sm@2~5<H)CIM_NI*P
z3ga11%kP>2bvMK=-GBOP{lmu#eu8+mi<7v3h8*~64<tzef1<x0<V^+YNH|}}&O(Y#
zRJ9HJKbEE>E}Hx8@(%lWpV?kVvZTI8X`4o$x_J9UwX*l^efg2AelA_1oqO<KdvH`t
zly&5ir|Rp|zTN)AY@PnDZX-i)dBp$C*M4oU{40Gba{AZ&$L}XBX1blcv7`Q6S=awh
z7MHr}7dih4&T{dH7XDxI|KIcZ+m!$9p73d_{iFSR?k=6QR{N^?#{PN7y{G2oZa!)_
z!@g$G+^)akmDPWqHt+vFf6cX3=b})ZJjxplfzc2c4S~@R7!85Z5TIHJG`y>0x~TET
Joq<7t0RSPO2<HF*

literal 0
HcmV?d00001

diff --git a/tests/f_write_ea_toobig_extra_isize/name b/tests/f_write_ea_toobig_extra_isize/name
new file mode 100644
index 0000000..a5ed718
--- /dev/null
+++ b/tests/f_write_ea_toobig_extra_isize/name
@@ -0,0 +1 @@
+write EA when i_extra_size is too big for EA
diff --git a/tests/f_write_ea_toosmall_extra_isize/expect.1 b/tests/f_write_ea_toosmall_extra_isize/expect.1
new file mode 100644
index 0000000..eecfc9d
--- /dev/null
+++ b/tests/f_write_ea_toosmall_extra_isize/expect.1
@@ -0,0 +1,15 @@
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 has a extra size (1) which is invalid
+Fix? yes
+
+Pass 2: Checking directory structure
+Directory inode 12, block #0, offset 4: directory corrupted
+Salvage? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 1
diff --git a/tests/f_write_ea_toosmall_extra_isize/expect.2 b/tests/f_write_ea_toosmall_extra_isize/expect.2
new file mode 100644
index 0000000..3b6073e
--- /dev/null
+++ b/tests/f_write_ea_toosmall_extra_isize/expect.2
@@ -0,0 +1,7 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 0
diff --git a/tests/f_write_ea_toosmall_extra_isize/image.gz b/tests/f_write_ea_toosmall_extra_isize/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..78a01497ec729dabc9406afb5914e76ce018cbb3
GIT binary patch
literal 2517
zcmb2|=3uBxpAo{u{Pvb@wuqwy!-MkgdQy`d_&OI%P^we#Fu0MKD0uYPp;HO<0&5m&
z+BkA#nAjh1n{@2np*Wr9R<TUoH7we$0?J~Y2^Q(^nUnWu^B5YQ{4BHiaLpY(<L&0(
z-`#U#_<c2J<4NVdWpBL20<*6^Uwh?+!R<3)ZenU)$Hmk(t+>8^>A98FI?H!`+qWs~
z-hHh({x?sz3mR+Imu>#_@7`{2=kxdHRn?Zh`gO{$XKVhu-!GSk%k_uZ%TyTt%=qdf
z_SkTO`R!}{$*Xymhu>ScZ%$BuFh4_sY`~G_?Lk)eI{s$|X1Of)-D**OlZTOk;lR7{
z{o7^iy=K?f%Q7%9{CF?=|3AOWw0_smkqwO}K5yLk%y^IU#p(XO_t*8_o3XDrds9Yt
zh4GB1<#)}1x*KAb?mzvt{^8>VKS4a(#YtR1Lk@hk2a=?KKha+g@}>fHB%Ci~XCXx=
zs@jJAA4^jb7tQ^4d53+x&up(FSyJDlv`wQ=UA%pwTG{*dzWm5lKbNl1&OP|CJvb^R
z$~tn%Q}y*}-){e5wod=%xRIf^JmUZ6YrnQv{*^uzIsI$?<M$K%S+^Ze68t~st;qkU
zg-b;LyR`pUex+l=Hno2*|NT90pX>goT=8jk{gL|0-QKFv;j8i=N!Wh&o|>0?bf5Th
zzWEu;AN_E>Cs+UT!TZ1JtGBM&EQadjQQl|>jE2By2#kinXb6mk0M$ZZ!S4Ay_uDlX
H7!())`D6t8

literal 0
HcmV?d00001

diff --git a/tests/f_write_ea_toosmall_extra_isize/name b/tests/f_write_ea_toosmall_extra_isize/name
new file mode 100644
index 0000000..718c12c
--- /dev/null
+++ b/tests/f_write_ea_toosmall_extra_isize/name
@@ -0,0 +1 @@
+write EA when i_extra_size is too small to make sense

[PATCH 07/54] libext2fs: fix tdb.c mmap leak

[Darrick J. Wong]

When undoing an expansion of an mmap'd database while cancelling a
transaction, the tdb code prematurely decreases the variable that
tracks the file size, which leads to a region leak during the
subsequent unmap.  Fix this by maintaining a separate counter for the
region size.

(This is probably unnecessary since e2undo was the only user of tdb
transactions, but I suppose we could be proactive.)

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/tdb.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)


diff --git a/lib/ext2fs/tdb.c b/lib/ext2fs/tdb.c
index 61e30ed..1d97685 100644
--- a/lib/ext2fs/tdb.c
+++ b/lib/ext2fs/tdb.c
@@ -246,6 +246,7 @@ struct tdb_context {
 	int page_size;
 	int max_dead_records;
 	bool have_transaction_lock;
+	tdb_len_t real_map_size; /* how much space has been mapped */
 };
 
 
@@ -970,9 +971,10 @@ int tdb_munmap(struct tdb_context *tdb)
 
 #ifdef HAVE_MMAP
 	if (tdb->map_ptr) {
-		int ret = munmap(tdb->map_ptr, tdb->map_size);
+		int ret = munmap(tdb->map_ptr, tdb->real_map_size);
 		if (ret != 0)
 			return ret;
+		tdb->real_map_size = 0;
 	}
 #endif
 	tdb->map_ptr = NULL;
@@ -995,10 +997,12 @@ void tdb_mmap(struct tdb_context *tdb)
 		 */
 
 		if (tdb->map_ptr == MAP_FAILED) {
+			tdb->real_map_size = 0;
 			tdb->map_ptr = NULL;
 			TDB_LOG((tdb, TDB_DEBUG_WARNING, "tdb_mmap failed for size %d (%s)\n",
 				 tdb->map_size, strerror(errno)));
 		}
+		tdb->real_map_size = tdb->map_size;
 	} else {
 		tdb->map_ptr = NULL;
 	}

[PATCH 08/54] resize2fs: fix regression test to not depend on ext4.ko being loaded

[Darrick J. Wong]

The behavior of the r_fixup_lastbg_big test varies depending on
whether or not ext4.ko is loaded and supports lazy_itable_init.  This
makes checking the bg flags after resize2fs hard to predict, so put in
a way to force resize2fs to zero the inode tables, and compare the
output based on lazy_itable_init == 0.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 resize/resize2fs.c              |    3 ++-
 tests/r_fixup_lastbg_big/expect |    4 +++-
 tests/r_fixup_lastbg_big/script |    2 +-
 3 files changed, 6 insertions(+), 3 deletions(-)


diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 2febfde..041ff75 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -904,7 +904,8 @@ retry:
 	group_block = ext2fs_group_first_block2(fs,
 						old_fs->group_desc_count);
 	csum_flag = ext2fs_has_group_desc_csum(fs);
-	if (access("/sys/fs/ext4/features/lazy_itable_init", F_OK) == 0)
+	if (!getenv("RESIZE2FS_FORCE_ITABLE_INIT") &&
+	    access("/sys/fs/ext4/features/lazy_itable_init", F_OK) == 0)
 		lazy_itable_init = 1;
 	if (fs->super->s_feature_incompat & EXT2_FEATURE_INCOMPAT_META_BG)
 		old_desc_blocks = fs->super->s_first_meta_bg;
diff --git a/tests/r_fixup_lastbg_big/expect b/tests/r_fixup_lastbg_big/expect
index 8f302a3..edaabaf 100644
--- a/tests/r_fixup_lastbg_big/expect
+++ b/tests/r_fixup_lastbg_big/expect
@@ -22,6 +22,8 @@ Group 2: (Blocks 16385-19999)
   Free blocks: 16385-19999
   Free inodes: 833-1248
 Resizing the filesystem on test.img to 40000 (1k) blocks.
+Begin pass 1 (max = 2)
+Extending the inode table     ----------------------------------------\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 The filesystem on test.img is now 40000 (1k) blocks long.
 
 Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
@@ -31,7 +33,7 @@ Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
   8192 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-24576
   Free inodes: 833-1248
-Group 3: (Blocks 24577-32768) [INODE_UNINIT]
+Group 3: (Blocks 24577-32768) [INODE_UNINIT, ITABLE_ZEROED]
   Backup superblock at 24577, Group descriptors at 24578-24578
   Reserved GDT blocks at 24579-24656
   Block bitmap at 413 (bg #0 + 412)
diff --git a/tests/r_fixup_lastbg_big/script b/tests/r_fixup_lastbg_big/script
index 4c4a351..97d9fd4 100755
--- a/tests/r_fixup_lastbg_big/script
+++ b/tests/r_fixup_lastbg_big/script
@@ -13,7 +13,7 @@ $DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
 $DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
 $DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
 dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 40000)) conv=notrunc >> $OUT 2> /dev/null
-$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+RESIZE2FS_FORCE_ITABLE_INIT=1 $RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
 $DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
 $E2FSCK -fy $TMPFILE >> $OUT 2>&1
 

[PATCH 09/54] tune2fs: disable csum verification before resizing inode

[Darrick J. Wong]

When we're turning on metadata checksumming /and/ resizing the inode
at the same time, disable checksum verification during the
resize_inode() call because the subroutines it calls will try to
verify the checksums (which have not yet been set), causing the
operation to fail unnecessarily.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/tune2fs.c               |    7 +++-
 tests/t_iexpand_full/script  |    2 +
 tests/t_iexpand_mcsum/expect |   54 ++++++++++++++++++++++++++++
 tests/t_iexpand_mcsum/name   |    1 +
 tests/t_iexpand_mcsum/script |   80 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 142 insertions(+), 2 deletions(-)
 create mode 100644 tests/t_iexpand_mcsum/expect
 create mode 100644 tests/t_iexpand_mcsum/name
 create mode 100644 tests/t_iexpand_mcsum/script


diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index b510c49..f01b05b 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -2961,8 +2961,13 @@ retry_open:
 		 * We want to update group descriptor also
 		 * with the new free inode count
 		 */
+		if (rewrite_checksums)
+			fs->flags |= EXT2_FLAG_IGNORE_CSUM_ERRORS;
 		fs->flags &= ~EXT2_FLAG_SUPER_ONLY;
-		if (resize_inode(fs, new_inode_size) == 0) {
+		retval = resize_inode(fs, new_inode_size);
+		if (rewrite_checksums)
+			fs->flags &= ~EXT2_FLAG_IGNORE_CSUM_ERRORS;
+		if (retval == 0) {
 			printf(_("Setting inode size %lu\n"),
 							new_inode_size);
 			rewrite_checksums = 1;
diff --git a/tests/t_iexpand_full/script b/tests/t_iexpand_full/script
index 9e34090..779c66d 100644
--- a/tests/t_iexpand_full/script
+++ b/tests/t_iexpand_full/script
@@ -44,7 +44,7 @@ echo Exit status is $status >> $OUT
 # dump and check
 ($DUMPE2FS -h $TMPFILE; $DUMPE2FS -g $TMPFILE) 2>&1 | sed -f $cmd_dir/filter.sed -e '/^Checksum:.*/d' >> $OUT.after 2> /dev/null
 echo "Change in FS metadata:" >> $OUT
-diff -u0 $OUT.before $OUT.after | sed -e '/^---.*/d' -e '/^+++.*/d' >> $OUT
+diff -U 0 $OUT.before $OUT.after | sed -e '/^---.*/d' -e '/^+++.*/d' >> $OUT
 $FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
 status=$?
 echo Exit status is $status >> $OUT
diff --git a/tests/t_iexpand_mcsum/expect b/tests/t_iexpand_mcsum/expect
new file mode 100644
index 0000000..2a6d705
--- /dev/null
+++ b/tests/t_iexpand_mcsum/expect
@@ -0,0 +1,54 @@
+tune2fs test
+Creating filesystem with 786432 1k blocks and 98304 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409, 663553
+
+Allocating group tables:      \b\b\b\b\bdone                            
+Writing inode tables:      \b\b\b\b\bdone                            
+Creating journal (16384 blocks): done
+Creating 6334 huge file(s) with 117 blocks each: done
+Writing superblocks and filesystem accounting information:      \b\b\b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+Exit status is 0
+tune2fs -I 256 -O metadata_csum test.img
+Setting inode size 256
+
+Please run e2fsck -D on the filesystem.
+
+Exit status is 0
+Backing up journal inode block information.
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 3A: Optimizing directories
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+
+
+Change in FS metadata:
+@@ -5 +5 @@
+-Filesystem features:      has_journal ext_attr dir_index filetype extent 64bit sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
++Filesystem features:      has_journal ext_attr dir_index filetype extent 64bit sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
+@@ -21 +21 @@
+-Inode blocks per group:   128
++Inode blocks per group:   256
+@@ -27 +27 @@
+-Inode size:	          128
++Inode size:	          256
+@@ -30,0 +31 @@
++Checksum type:            crc32c
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+Exit status is 0
diff --git a/tests/t_iexpand_mcsum/name b/tests/t_iexpand_mcsum/name
new file mode 100644
index 0000000..e767715
--- /dev/null
+++ b/tests/t_iexpand_mcsum/name
@@ -0,0 +1 @@
+expand inodes and turn on metadata_csum
diff --git a/tests/t_iexpand_mcsum/script b/tests/t_iexpand_mcsum/script
new file mode 100644
index 0000000..3057dba
--- /dev/null
+++ b/tests/t_iexpand_mcsum/script
@@ -0,0 +1,80 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fn
+OUT=$test_name.log
+EXP=$test_dir/expect
+CONF=$TMPFILE.conf
+
+#gzip -d < $EXP.gz > $EXP
+
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,^resize_inode,^meta_bg,^flex_bg,^metadata_csum,64bit
+		blocksize = 1024
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 16000K
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 117K
+		zero_hugefiles = false
+	}
+ENDL
+
+echo "tune2fs test" > $OUT
+
+MKE2FS_CONFIG=$CONF $MKE2FS -F -T ext4h -I 128 $TMPFILE 786432 >> $OUT 2>&1
+rm -rf $CONF
+
+# dump and check
+($DUMPE2FS -h $TMPFILE | grep -v '^Free blocks:'; $DUMPE2FS -g $TMPFILE) 2>&1 | sed -f $cmd_dir/filter.sed -e '/^Checksum:.*/d' >> $OUT.before 2> /dev/null
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# convert it
+echo "tune2fs -I 256 -O metadata_csum test.img" >> $OUT
+dd if=/dev/zero of=$TMPFILE conv=notrunc bs=1 count=1 seek=3221225471 2> /dev/null
+$TUNE2FS -I 256 -O metadata_csum $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+$FSCK -N test_filesys -y -f -D $TMPFILE >> $OUT 2>&1
+
+# dump and check
+($DUMPE2FS -h $TMPFILE | grep -v '^Free blocks:'; $DUMPE2FS -g $TMPFILE) 2>&1 | sed -f $cmd_dir/filter.sed -e '/^Checksum:.*/d' >> $OUT.after 2> /dev/null
+echo "Change in FS metadata:" >> $OUT
+diff -U 0 $OUT.before $OUT.after | sed -e '/^---.*/d' -e '/^+++.*/d' >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+rm $TMPFILE
+
+#
+# Do the verification
+#
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" -e 's/test_filesys:.*//g' < $OUT > $OUT.new
+mv $OUT.new $OUT
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+rm $OUT.before $OUT.after
+
+unset IMAGE FSCK_OPT OUT EXP CONF
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
+

[PATCH 10/54] tune2fs: abort when trying to enable/disable metadata_csum on mounted fs

[Darrick J. Wong]

Earlier, I tried to make tune2fs abort if the user tried to enable or
disable metadata_csum on a mounted FS, but forgot the exit() call.
Supply it now.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/tune2fs.c |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)


diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index f01b05b..e33b0d3 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -1110,9 +1110,11 @@ mmp_error:
 		       EXT4_FEATURE_RO_COMPAT_METADATA_CSUM)) {
 		if (check_fsck_needed(fs))
 			exit(1);
-		if (mount_flags & EXT2_MF_MOUNTED)
+		if (mount_flags & EXT2_MF_MOUNTED) {
 			fputs(_("Cannot enable metadata_csum on a mounted "
 				"filesystem!\n"), stderr);
+			exit(1);
+		}
 		if (!EXT2_HAS_INCOMPAT_FEATURE(fs->super,
 				EXT3_FEATURE_INCOMPAT_EXTENTS))
 			printf("%s",
@@ -1152,9 +1154,11 @@ mmp_error:
 
 		if (check_fsck_needed(fs))
 			exit(1);
-		if (mount_flags & EXT2_MF_MOUNTED)
+		if (mount_flags & EXT2_MF_MOUNTED) {
 			fputs(_("Cannot disable metadata_csum on a mounted "
 				"filesystem!\n"), stderr);
+			exit(1);
+		}
 		rewrite_checksums = 1;
 
 		/* Enable uninit_bg unless the user expressly turned it off */

[PATCH 11/54] tune2fs: call out to resize2fs for 64bit conversion

[Darrick J. Wong]

If the user tries to enable or disable the 64bit feature via tune2fs,
tell them how to use resize2fs to effect the conversion.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/tune2fs.c |   64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 62 insertions(+), 2 deletions(-)


diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index e33b0d3..258121f 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -97,6 +97,8 @@ static unsigned long new_inode_size;
 static char *ext_mount_opts;
 static int usrquota, grpquota;
 static int rewrite_checksums;
+static int feature_64bit;
+static int fsck_requested;
 
 int journal_size, journal_flags;
 char *journal_device;
@@ -146,7 +148,8 @@ static __u32 ok_features[3] = {
 	EXT2_FEATURE_INCOMPAT_FILETYPE |
 		EXT3_FEATURE_INCOMPAT_EXTENTS |
 		EXT4_FEATURE_INCOMPAT_FLEX_BG |
-		EXT4_FEATURE_INCOMPAT_MMP,
+		EXT4_FEATURE_INCOMPAT_MMP |
+		EXT4_FEATURE_INCOMPAT_64BIT,
 	/* R/O compat */
 	EXT2_FEATURE_RO_COMPAT_LARGE_FILE |
 		EXT4_FEATURE_RO_COMPAT_HUGE_FILE|
@@ -168,7 +171,8 @@ static __u32 clear_ok_features[3] = {
 	/* Incompat */
 	EXT2_FEATURE_INCOMPAT_FILETYPE |
 		EXT4_FEATURE_INCOMPAT_FLEX_BG |
-		EXT4_FEATURE_INCOMPAT_MMP,
+		EXT4_FEATURE_INCOMPAT_MMP |
+		EXT4_FEATURE_INCOMPAT_64BIT,
 	/* R/O compat */
 	EXT2_FEATURE_RO_COMPAT_LARGE_FILE |
 		EXT4_FEATURE_RO_COMPAT_HUGE_FILE|
@@ -418,6 +422,7 @@ static void request_dir_fsck_afterwards(ext2_filsys fs)
 
 	if (requested++)
 		return;
+	fsck_requested++;
 	fs->super->s_state &= ~EXT2_VALID_FS;
 	printf("\n%s\n", _(please_dir_fsck));
 	if (mount_flags & EXT2_MF_READONLY)
@@ -430,12 +435,40 @@ static void request_fsck_afterwards(ext2_filsys fs)
 
 	if (requested++)
 		return;
+	fsck_requested++;
 	fs->super->s_state &= ~EXT2_VALID_FS;
 	printf("\n%s\n", _(please_fsck));
 	if (mount_flags & EXT2_MF_READONLY)
 		printf("%s", _("(and reboot afterwards!)\n"));
 }
 
+static void convert_64bit(ext2_filsys fs, int direction)
+{
+	if (!direction)
+		return;
+
+	/*
+	 * Is resize2fs going to demand a fsck run? Might as well tell the
+	 * user now.
+	 */
+	if (!fsck_requested &&
+	    ((fs->super->s_state & EXT2_ERROR_FS) ||
+	     !(fs->super->s_state & EXT2_VALID_FS) ||
+	     fs->super->s_lastcheck < fs->super->s_mtime))
+		request_fsck_afterwards(fs);
+	if (fsck_requested)
+		fprintf(stderr, _("After running e2fsck, please run `resize2fs %s %s"),
+			direction > 0 ? "-b" : "-s", fs->device_name);
+	else
+		fprintf(stderr, _("Please run `resize2fs %s %s"),
+			direction > 0 ? "-b" : "-s", fs->device_name);
+
+	if (direction > 0)
+		fprintf(stderr, _("' to enable 64-bit mode.\n"));
+	else
+		fprintf(stderr, _("' to disable 64-bit mode.\n"));
+}
+
 /* Rewrite extents */
 static errcode_t rewrite_extents(ext2_filsys fs, ext2_ino_t ino,
 				 struct ext2_inode *inode)
@@ -1212,6 +1245,32 @@ mmp_error:
 			return 1;
 	}
 
+	/*
+	 * We don't actually toggle 64bit; resize2fs does that.  But this
+	 * must come after the metadata_csum feature_on so that it won't
+	 * complain about the lack of 64bit.
+	 */
+	if (FEATURE_ON(E2P_FEATURE_INCOMPAT,
+		       EXT4_FEATURE_INCOMPAT_64BIT)) {
+		if (mount_flags & EXT2_MF_MOUNTED) {
+			fprintf(stderr, _("Cannot enable 64-bit mode "
+					  "while mounted!\n"));
+			exit(1);
+		}
+		sb->s_feature_incompat &= ~EXT4_FEATURE_INCOMPAT_64BIT;
+		feature_64bit = 1;
+	}
+	if (FEATURE_OFF(E2P_FEATURE_INCOMPAT,
+			EXT4_FEATURE_INCOMPAT_64BIT)) {
+		if (mount_flags & EXT2_MF_MOUNTED) {
+			fprintf(stderr, _("Cannot disable 64-bit mode "
+					  "while mounted!\n"));
+			exit(1);
+		}
+		sb->s_feature_incompat |= EXT4_FEATURE_INCOMPAT_64BIT;
+		feature_64bit = -1;
+	}
+
 	if (FEATURE_ON(E2P_FEATURE_RO_INCOMPAT,
 				EXT4_FEATURE_RO_COMPAT_QUOTA)) {
 		/*
@@ -3015,5 +3074,6 @@ closefs:
 		exit(1);
 	}
 
+	convert_64bit(fs, feature_64bit);
 	return (ext2fs_close_free(&fs) ? 1 : 0);
 }

[PATCH 12/54] e2fsck: clear i_block[] when there are too many bad mappings on a special inode

[Darrick J. Wong]

If we decide to clear a special inode because of bad mappings, we need
to zero the i_block array.  The clearing routine depends on setting
i_links_count to zero to keep us from re-checking the block maps,
but that field isn't checked for special inodes.  Therefore, if we
haven't erased the mappings, check_blocks will restart fsck and fsck
will try to check the blocks again, leading to an infinite loop.

(This seems easy to trigger if the bootloader inode extent map is
corrupted.)

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c |   10 ++++++++++
 1 file changed, 10 insertions(+)


diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 14877d7..faafa7c 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -2285,6 +2285,16 @@ void e2fsck_clear_inode(e2fsck_t ctx, ext2_ino_t ino,
 	ext2fs_icount_store(ctx->inode_link_info, ino, 0);
 	inode->i_dtime = ctx->now;
 
+	/*
+	 * If a special inode has such rotten block mappings that we
+	 * want to clear the whole inode, be sure to actually zap
+	 * the block maps because i_links_count isn't checked for
+	 * special inodes, and we'll end up right back here the next
+	 * time we run fsck.
+	 */
+	if (ino < EXT2_FIRST_INODE(ctx->fs->super))
+		memset(inode->i_block, 0, sizeof(inode->i_block));
+
 	ext2fs_unmark_inode_bitmap2(ctx->inode_dir_map, ino);
 	ext2fs_unmark_inode_bitmap2(ctx->inode_used_map, ino);
 	if (ctx->inode_reg_map)

[PATCH 13/54] e2fsck: on read error, don't rewrite blocks past the end of the fs

[Darrick J. Wong]

If e2fsck encounters a read error on a block past the end of the
filesystem, don't bother trying to "rewrite" the block.  We might
still want to re-try the read to capture FS data marooned past the end
of the filesystem, but in that case e2fsck ought to move the block
back inside the filesystem.

This enables e2fuzz to detect writes past the end of the FS due to
software bugs.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/ehandler.c |    5 +++++
 misc/e2fuzz.sh    |    7 +++++++
 2 files changed, 12 insertions(+)


diff --git a/e2fsck/ehandler.c b/e2fsck/ehandler.c
index 6dddf9c..71ca301 100644
--- a/e2fsck/ehandler.c
+++ b/e2fsck/ehandler.c
@@ -58,6 +58,11 @@ static errcode_t e2fsck_handle_read_error(io_channel channel,
 		printf(_("Error reading block %lu (%s).  "), block,
 		       error_message(error));
 	preenhalt(ctx);
+
+	/* Don't rewrite a block past the end of the FS. */
+	if (block >= ext2fs_blocks_count(fs->super))
+		return 0;
+
 	if (ask(ctx, _("Ignore error"), 1)) {
 		if (ask(ctx, _("Force rewrite"), 1))
 			io_channel_write_blk64(channel, block, count, data);
diff --git a/misc/e2fuzz.sh b/misc/e2fuzz.sh
index 4cb7b61..d8d9a82 100755
--- a/misc/e2fuzz.sh
+++ b/misc/e2fuzz.sh
@@ -219,6 +219,7 @@ seq 1 "${PASSES}" | while read pass; do
 	fi
 	if [ "${RUN_FSCK}" -gt 0 ]; then
 		cp "${PASS_IMG}" "${FSCK_IMG}"
+		pass_img_sz="$(stat -c '%s' "${PASS_IMG}")"
 
 		seq 1 "${MAX_FSCK}" | while read fsck_pass; do
 			echo "++ fsck pass ${fsck_pass}: $(which e2fsck) -fy ${FSCK_IMG} ${EXTENDED_FSCK_OPTS}"
@@ -250,6 +251,12 @@ seq 1 "${PASSES}" | while read pass; do
 					exit 2
 				fi
 			fi
+
+			fsck_img_sz="$(stat -c '%s' "${FSCK_IMG}")"
+			if [ "${fsck_img_sz}" -ne "${pass_img_sz}" ]; then
+				echo "++ fsck image size changed"
+				exit 3
+			fi
 		done
 		fsck_loop_ret=$?
 		if [ "${fsck_loop_ret}" -gt 0 ]; then

[PATCH 14/54] e2fsck: fix the journal recreation message

[Darrick J. Wong]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 12310 bytes --]

When we recreate the journal, don't say that the FS "is now ext3
again", since we could be fixing a damaged ext4 FS journal, which does
not magically convert the FS back to ext3.

Fix the po files too, though this string hasn't been translated.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/unix.c                         |    2 +-
 po/ca.po                              |    2 +-
 po/cs.po                              |    2 +-
 po/de.po                              |    2 +-
 po/eo.po                              |    2 +-
 po/es.po                              |    2 +-
 po/fr.po                              |    2 +-
 po/id.po                              |    2 +-
 po/it.po                              |    2 +-
 po/nl.po                              |    2 +-
 po/pl.po                              |    2 +-
 po/sv.po                              |    2 +-
 po/tr.po                              |    2 +-
 po/uk.po                              |    2 +-
 po/vi.po                              |    2 +-
 po/zh_CN.po                           |    2 +-
 tests/f_badjour_indblks/expect.1      |    2 +-
 tests/f_badjourblks/expect.1          |    2 +-
 tests/f_miss_journal/expect.1         |    2 +-
 tests/j_corrupt_sb_magic/expect       |    2 +-
 tests/j_long_trans/expect             |    2 +-
 tests/j_long_trans_mcsum_32bit/expect |    2 +-
 tests/j_long_trans_mcsum_64bit/expect |    2 +-
 23 files changed, 23 insertions(+), 23 deletions(-)


diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 615d690..5cba013 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -1675,7 +1675,7 @@ print_unsupp_features:
 			log_out(ctx, "%s", _(" Done.\n"));
 			log_out(ctx, "%s",
 				_("\n*** journal has been re-created - "
-				  "filesystem is now ext3 again ***\n"));
+				  "filesystem is journalled again ***\n"));
 		}
 	}
 no_journal:
diff --git a/po/ca.po b/po/ca.po
index 02388da..c9ff513 100644
--- a/po/ca.po
+++ b/po/ca.po
@@ -3178,7 +3178,7 @@ msgstr " Fet.\n"
 #: e2fsck/unix.c:1663
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** registre de canvis recreat - el sist. de fitxers torna a ser ext3 ***\n"
diff --git a/po/cs.po b/po/cs.po
index be0a410..298c94e 100644
--- a/po/cs.po
+++ b/po/cs.po
@@ -3190,7 +3190,7 @@ msgstr " Hotovo.\n"
 #: e2fsck/unix.c:1664
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** žurnál by znovu vytvořen – souborový systém se opět stal ext3 ***\n"
diff --git a/po/de.po b/po/de.po
index fd96bb1..8c2078b 100644
--- a/po/de.po
+++ b/po/de.po
@@ -3159,7 +3159,7 @@ msgstr " Erledigt.\n"
 #: e2fsck/unix.c:1663
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** Journal wurde wiederhergestellt - Dateisystem ist nun wieder ext3 ***\n"
diff --git a/po/eo.po b/po/eo.po
index 33e1635..6ce44ad 100644
--- a/po/eo.po
+++ b/po/eo.po
@@ -3113,7 +3113,7 @@ msgstr " Pretas.\n"
 #: e2fsck/unix.c:1663
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** kaŝprotokolo rekreiĝis -- dosiersistemo estas denove ext3 ***\n"
diff --git a/po/es.po b/po/es.po
index 4137940..8dd4fe4 100644
--- a/po/es.po
+++ b/po/es.po
@@ -3235,7 +3235,7 @@ msgstr " Hecho.\n"
 #: e2fsck/unix.c:1663
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** el fichero de transacciones se ha creado de nuevo ***\n"
diff --git a/po/fr.po b/po/fr.po
index 44a482f..9670f86 100644
--- a/po/fr.po
+++ b/po/fr.po
@@ -3173,7 +3173,7 @@ msgstr "Compl
 #: e2fsck/unix.c:1664
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** le journal a été re-créé - le système de fichiers est de nouveau ext3 ***\n"
diff --git a/po/id.po b/po/id.po
index 0b747ff..7d56ceb 100644
--- a/po/id.po
+++ b/po/id.po
@@ -3170,7 +3170,7 @@ msgstr " Selesai.\n"
 #: e2fsck/unix.c:1663
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** jurnal telah dibuat kembali - sistem berkas sekarang ext3 lagi ***\n"
diff --git a/po/it.po b/po/it.po
index f6c7374..3736d90 100644
--- a/po/it.po
+++ b/po/it.po
@@ -3220,7 +3220,7 @@ msgstr ""
 #: e2fsck/unix.c:1663
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 
 #: e2fsck/unix.c:1687
diff --git a/po/nl.po b/po/nl.po
index dcc716d..033ef05 100644
--- a/po/nl.po
+++ b/po/nl.po
@@ -3177,7 +3177,7 @@ msgstr " voltooid.\n"
 #: e2fsck/unix.c:1664
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** journal is opnieuw aangemaakt -- het bestandssysteem is nu weer ext3 ***\n"
diff --git a/po/pl.po b/po/pl.po
index 4fef6fb..7d71292 100644
--- a/po/pl.po
+++ b/po/pl.po
@@ -3177,7 +3177,7 @@ msgstr " Wykonano.\n"
 #: e2fsck/unix.c:1664
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** kronika została ponownie utworzona - system plików to znowu ext3 ***\n"
diff --git a/po/sv.po b/po/sv.po
index 3391aee..a8d171f 100644
--- a/po/sv.po
+++ b/po/sv.po
@@ -3176,7 +3176,7 @@ msgstr " Klar.\n"
 #: e2fsck/unix.c:1664
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** journalen har återskapats - filsystemet är nu ext3 igen ***\n"
diff --git a/po/tr.po b/po/tr.po
index 0c715c0..4cd64ba 100644
--- a/po/tr.po
+++ b/po/tr.po
@@ -3249,7 +3249,7 @@ msgstr " Tamamlandı.\n"
 #: e2fsck/unix.c:1663
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** günlük yeniden oluşturuldu - dosya sistemi yeniden ext3 ***\n"
diff --git a/po/uk.po b/po/uk.po
index f93282a..7deeafb 100644
--- a/po/uk.po
+++ b/po/uk.po
@@ -3177,7 +3177,7 @@ msgstr " Виконано.\n"
 #: e2fsck/unix.c:1664
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** журнал створено повторно - тепер це знову файлова система ext3 ***\n"
diff --git a/po/vi.po b/po/vi.po
index 2300c19..e08e7f5 100644
--- a/po/vi.po
+++ b/po/vi.po
@@ -3139,7 +3139,7 @@ msgstr " Xong.\n"
 #: e2fsck/unix.c:1664
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 "\n"
 "*** nhật ký đã được tạo lại — hệ thống tập tin lúc này là ext3 lại ***\n"
diff --git a/po/zh_CN.po b/po/zh_CN.po
index 37c4aa7..d79f348 100644
--- a/po/zh_CN.po
+++ b/po/zh_CN.po
@@ -3003,7 +3003,7 @@ msgstr "完成.\n"
 #: e2fsck/unix.c:1663
 msgid ""
 "\n"
-"*** journal has been re-created - filesystem is now ext3 again ***\n"
+"*** journal has been re-created - filesystem is journalled again ***\n"
 msgstr ""
 
 #: e2fsck/unix.c:1687
diff --git a/tests/f_badjour_indblks/expect.1 b/tests/f_badjour_indblks/expect.1
index 7ccc59b..a326900 100644
--- a/tests/f_badjour_indblks/expect.1
+++ b/tests/f_badjour_indblks/expect.1
@@ -25,7 +25,7 @@ Recreate journal? yes
 
 Creating journal (1024 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been re-created - filesystem is journalled again ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/256 files (0.0% non-contiguous), 1111/8192 blocks
diff --git a/tests/f_badjourblks/expect.1 b/tests/f_badjourblks/expect.1
index 34c6658..dc38b2b 100644
--- a/tests/f_badjourblks/expect.1
+++ b/tests/f_badjourblks/expect.1
@@ -23,7 +23,7 @@ Recreate journal? yes
 
 Creating journal (1024 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been re-created - filesystem is journalled again ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/256 files (0.0% non-contiguous), 1079/8192 blocks
diff --git a/tests/f_miss_journal/expect.1 b/tests/f_miss_journal/expect.1
index 6ec8b38..77d3223 100644
--- a/tests/f_miss_journal/expect.1
+++ b/tests/f_miss_journal/expect.1
@@ -21,7 +21,7 @@ Recreate journal? yes
 
 Creating journal (1024 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been re-created - filesystem is journalled again ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/256 files (0.0% non-contiguous), 1079/2048 blocks
diff --git a/tests/j_corrupt_sb_magic/expect b/tests/j_corrupt_sb_magic/expect
index 2169a15..0d2e9b4 100644
--- a/tests/j_corrupt_sb_magic/expect
+++ b/tests/j_corrupt_sb_magic/expect
@@ -26,7 +26,7 @@ Recreate journal? yes
 
 Creating journal (1024 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been re-created - filesystem is journalled again ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 12/128 files (0.0% non-contiguous), 1092/2048 blocks
diff --git a/tests/j_long_trans/expect b/tests/j_long_trans/expect
index 7638ef1..09da441 100644
--- a/tests/j_long_trans/expect
+++ b/tests/j_long_trans/expect
@@ -100,7 +100,7 @@ Recreate journal? yes
 
 Creating journal (8192 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been re-created - filesystem is journalled again ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/16384 files (0.0% non-contiguous), 14420/262144 blocks
diff --git a/tests/j_long_trans_mcsum_32bit/expect b/tests/j_long_trans_mcsum_32bit/expect
index 0d141c1..0f35126 100644
--- a/tests/j_long_trans_mcsum_32bit/expect
+++ b/tests/j_long_trans_mcsum_32bit/expect
@@ -139,7 +139,7 @@ Recreate journal? yes
 
 Creating journal (16384 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been re-created - filesystem is journalled again ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/32768 files (0.0% non-contiguous), 27039/524288 blocks
diff --git a/tests/j_long_trans_mcsum_64bit/expect b/tests/j_long_trans_mcsum_64bit/expect
index 94e9925..17c2517 100644
--- a/tests/j_long_trans_mcsum_64bit/expect
+++ b/tests/j_long_trans_mcsum_64bit/expect
@@ -138,7 +138,7 @@ Recreate journal? yes
 
 Creating journal (16384 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been re-created - filesystem is journalled again ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/32768 files (0.0% non-contiguous), 27057/524288 blocks

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 15/54] e2fsck: handle multiple *ind block collisions with critical metadata

[Darrick J. Wong]

An earlier patch tried to detect indirect blocks that conflicted with
critical FS metadata for the purpose of preventing corrections being
made to those indirect blocks.  Unfortunately, that patch cannot
handle more than one conflicting *ind block per file; therefore, use
the ref_block parameter to test the metadata block map to decide if
we need to avoid fixing the *ind block when we're iterating the
block's entries.  (We have to iterate the block to capture any blocks
that the block points to, as they could be in use.)

As a side note, in 1B we'll reallocate all those conflicting *ind
blocks and restart fsck, so the contents will be checked eventually.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c                       |   17 +++-
 tests/f_ind_inode_collision/expect.1 |  147 ++++++++++++++++++++++++++++++++++
 tests/f_ind_inode_collision/expect.2 |    7 ++
 tests/f_ind_inode_collision/image.gz |  Bin
 tests/f_ind_inode_collision/name     |    1 
 5 files changed, 167 insertions(+), 5 deletions(-)
 create mode 100644 tests/f_ind_inode_collision/expect.1
 create mode 100644 tests/f_ind_inode_collision/expect.2
 create mode 100644 tests/f_ind_inode_collision/image.gz
 create mode 100644 tests/f_ind_inode_collision/name


diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index faafa7c..2089943 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -93,7 +93,6 @@ struct process_block_struct {
 	struct problem_context *pctx;
 	ext2fs_block_bitmap fs_meta_blocks;
 	e2fsck_t	ctx;
-	blk64_t		bad_ref;
 	region_t	region;
 };
 
@@ -2791,7 +2790,6 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx,
 	pb.pctx = pctx;
 	pb.ctx = ctx;
 	pb.inode_modified = 0;
-	pb.bad_ref = 0;
 	pctx->ino = ino;
 	pctx->errcode = 0;
 
@@ -3175,7 +3173,6 @@ static int process_block(ext2_filsys fs,
 	if (blockcnt < 0 &&
 	    p->ino != EXT2_RESIZE_INO &&
 	    ext2fs_test_block_bitmap2(ctx->block_metadata_map, blk)) {
-		p->bad_ref = blk;
 		pctx->blk = blk;
 		fix_problem(ctx, PR_1_CRITICAL_METADATA_COLLISION, pctx);
 		ctx->flags |= E2F_FLAG_RESTART_LATER;
@@ -3185,13 +3182,23 @@ static int process_block(ext2_filsys fs,
 		p->num_illegal_blocks++;
 		/*
 		 * A bit of subterfuge here -- we're trying to fix a block
-		 * mapping, but know that the IND/DIND/TIND block has collided
+		 * mapping, but the IND/DIND/TIND block could have collided
 		 * with some critical metadata.  So, fix the in-core mapping so
 		 * iterate won't go insane, but return 0 instead of
 		 * BLOCK_CHANGED so that it won't write the remapping out to
 		 * our multiply linked block.
+		 *
+		 * Even if we previously determined that an *IND block
+		 * conflicts with critical metadata, we must still try to
+		 * iterate the *IND block as if it is an *IND block to find and
+		 * mark the blocks it points to.  Better to be overly cautious
+		 * with the used_blocks map so that we don't move the *IND
+		 * block to a block that's really in use!
 		 */
-		if (p->bad_ref && ref_block == p->bad_ref) {
+		if (p->ino != EXT2_RESIZE_INO &&
+		    ref_block != 0 &&
+		    ext2fs_test_block_bitmap2(ctx->block_metadata_map,
+					      ref_block)) {
 			*block_nr = 0;
 			return 0;
 		}
diff --git a/tests/f_ind_inode_collision/expect.1 b/tests/f_ind_inode_collision/expect.1
new file mode 100644
index 0000000..012cd9f
--- /dev/null
+++ b/tests/f_ind_inode_collision/expect.1
@@ -0,0 +1,147 @@
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 block 41 conflicts with critical metadata, skipping block checks.
+Inode 12 block 40 conflicts with critical metadata, skipping block checks.
+Inode 12 block 34 conflicts with critical metadata, skipping block checks.
+Illegal block number passed to ext2fs_test_block_bitmap #16777215 for metadata block map
+Inode 12 block 1 conflicts with critical metadata, skipping block checks.
+Inode 12, i_size is 33, should be 25227264.  Fix? yes
+
+Inode 12, i_blocks is 999, should be 184.  Fix? yes
+
+
+Running additional passes to resolve blocks claimed by more than one inode...
+Pass 1B: Rescanning for multiply-claimed blocks
+Multiply-claimed block(s) in inode 2: 3
+Multiply-claimed block(s) in inode 7: 11
+Multiply-claimed block(s) in inode 11: 4--7
+Multiply-claimed block(s) in inode 12: 41 40Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #16877 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #4096 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #196608 for multiply claimed block map
+ 34 8 3Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #33152 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #4243456 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #65536 for multiply claimed block map
+ 28 8 11 1Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #16832 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #16384 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #131072 for multiply claimed block map
+ 28 4--7Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #33206 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #25227264 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529397 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529397 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529397 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #65536 for multiply claimed block map
+ 28 41Illegal block number passed to ext2fs_test_block_bitmap #1421529397 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #16777215 for multiply claimed block map
+ 28
+Error while iterating over blocks in inode 12 (pass1b): Illegal indirect block found
+Pass 1C: Scanning directories for inodes with multiply-claimed blocks
+Pass 1D: Reconciling multiply-claimed blocks
+(There are 3 inodes containing multiply-claimed blocks.)
+
+File / (inode #2, mod time Sat Jan 17 21:16:16 2015) 
+  has 1 multiply-claimed block(s), shared with 1 file(s):
+	/a (inode #12, mod time Sat Jan 17 21:16:37 2015)
+Clone multiply-claimed blocks? yes
+
+File /lost+found (inode #11, mod time Sat Jan 17 21:16:16 2015) 
+  has 4 multiply-claimed block(s), shared with 1 file(s):
+	/a (inode #12, mod time Sat Jan 17 21:16:37 2015)
+Clone multiply-claimed blocks? yes
+
+File /a (inode #12, mod time Sat Jan 17 21:16:37 2015) 
+  has 17 multiply-claimed block(s), shared with 4 file(s):
+	<filesystem metadata>
+	/lost+found (inode #11, mod time Sat Jan 17 21:16:16 2015)
+	<The group descriptor inode> (inode #7, mod time Sat Jan 17 21:16:16 2015)
+	/ (inode #2, mod time Sat Jan 17 21:16:16 2015)
+Clone multiply-claimed blocks? yes
+
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #16877 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #4096 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #196608 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #33152 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #4243456 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #65536 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #16832 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #16384 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #131072 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529376 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #33206 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #25227264 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529397 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529397 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529397 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #65536 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #1421529397 for multiply claimed block map
+Illegal block number passed to ext2fs_test_block_bitmap #16777215 for multiply claimed block map
+Pass 2: Checking directory structure
+Restarting e2fsck from the beginning...
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 has illegal block(s).  Clear? yes
+
+Illegal block #1038 (1421529376) in inode 12.  CLEARED.
+Illegal block #1039 (1421529376) in inode 12.  CLEARED.
+Illegal block #1040 (1421529376) in inode 12.  CLEARED.
+Illegal block #1100 (16877) in inode 12.  CLEARED.
+Illegal block #1101 (4096) in inode 12.  CLEARED.
+Illegal block #1102 (1421529376) in inode 12.  CLEARED.
+Illegal block #1103 (1421529376) in inode 12.  CLEARED.
+Illegal block #1104 (1421529376) in inode 12.  CLEARED.
+Illegal block #1106 (196608) in inode 12.  CLEARED.
+Illegal block #1136 (1421529376) in inode 12.  CLEARED.
+Illegal block #1420 (33152) in inode 12.  CLEARED.
+Too many illegal blocks in inode 12.
+Clear inode? yes
+
+Restarting e2fsck from the beginning...
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Entry 'a' in / (2) has deleted/unused inode 12.  Clear? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Block bitmap differences:  -(3--7) -(15--17) -(19--24)
+Fix? yes
+
+Inode bitmap differences:  -12
+Fix? yes
+
+Free inodes count wrong for group #0 (116, counted=117).
+Fix? yes
+
+Free inodes count wrong (116, counted=117).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 11/128 files (9.1% non-contiguous), 18/512 blocks
+Exit status is 1
diff --git a/tests/f_ind_inode_collision/expect.2 b/tests/f_ind_inode_collision/expect.2
new file mode 100644
index 0000000..d0a6dac
--- /dev/null
+++ b/tests/f_ind_inode_collision/expect.2
@@ -0,0 +1,7 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/128 files (9.1% non-contiguous), 18/512 blocks
+Exit status is 0
diff --git a/tests/f_ind_inode_collision/image.gz b/tests/f_ind_inode_collision/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..924d220f6887030e2584b344847814b02fe055e1
GIT binary patch
literal 2456
zcmb2|=3o%Myeou>`R%R!*}{P`3?Is;N37;t;vv4!`Amx7CQ+Hk&+^P11&%l#<y$t{
zJ0>Qir0lDGzz6r07kp*7^ZqgVY5kLE{Ty4QX?Q0o+jnxE-NWkRzbEeAQ-1vM-qWoN
zGNFHKCbtz>z1ydv_@(R5mJY|(x<LJ^J)3XxDxTJ>^>tb!H>>$eK+v0P*GDyVJrxsn
z|5Oyacc<e2uFqfIzWVf3e*G%`<rl9#|DIo$Z@cqG?Z%G}ZZ2M{{r=PaX#e)q$8Udx
zhdol)Jh}du#lvgN3<+GHUcO%`vG(Af{o<Fe-rW@U<69s<BLl;MH>br`KMYTmzt7IV
zz`(cn&;S2b9+%1__HhUBXWq)=zyC0nPyKTA>+-7S>(=Mr**9%^>xYFCRl;q}fSMa-
z2F;H6|38Q4`hTG219KuTu>$FYzx+Tl;y*KxBoZ7rcuGn*6R0@hT?G>pk(#h5J8(ba
zT~3*M;gnl{e>&aKnYwkqkl688p?oj?Xscd7c$ec3|KIAu&382=|4*_%wtw#m-zCD2
z<z=*P|NOH&*zCWjbMcxu(_<SYxDETxU;OhtRp$S>BY&f8-fOrmzPh~D?^oFO3;#?X
z%VV(^EOe%4vtg9sCY{@7y-T=pDH;_Y4S~@R7!85Z5Eu=C(GVam1P)yJ%Vzdsc>n{0
G0s{axvFe2Y

literal 0
HcmV?d00001

diff --git a/tests/f_ind_inode_collision/name b/tests/f_ind_inode_collision/name
new file mode 100644
index 0000000..11feee3
--- /dev/null
+++ b/tests/f_ind_inode_collision/name
@@ -0,0 +1 @@
+multiple *ind collisions with critical metadata

[PATCH 16/54] e2fsck: decrement bad count _after_ remapping a duplicate block

[Darrick J. Wong]

Decrement the bad count *after* we've shown that (a) we can allocate a
replacement block and (b) remap the file block.  Unfortunately,
the only way to tell if the remapping succeeded is to wait until the
next clone_file_block() call or block_iterate3() returns.

Otherwise, there's a corruption error: we decrease the badcount once in
preparation to remap, then the remap fails (either we can't find a
replacement block or we have to split the extent tree and can't find a
new extent block), so we delete the file, which decreases the badcount
on the block a second time.  Later on e2fsck will think that it's
straightened out all the duplicate blocks, which isn't true.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1b.c                         |   32 +++++++++++++++++++++++++---
 tests/f_cloneblock_alloc_error/expect.1 |   36 +++++++++++++++++++++++++++++++
 tests/f_cloneblock_alloc_error/expect.2 |    7 ++++++
 tests/f_cloneblock_alloc_error/image.gz |  Bin
 tests/f_cloneblock_alloc_error/name     |    1 +
 5 files changed, 73 insertions(+), 3 deletions(-)
 create mode 100644 tests/f_cloneblock_alloc_error/expect.1
 create mode 100644 tests/f_cloneblock_alloc_error/expect.2
 create mode 100644 tests/f_cloneblock_alloc_error/image.gz
 create mode 100644 tests/f_cloneblock_alloc_error/name


diff --git a/e2fsck/pass1b.c b/e2fsck/pass1b.c
index 2d1b448..cd967f4 100644
--- a/e2fsck/pass1b.c
+++ b/e2fsck/pass1b.c
@@ -725,8 +725,30 @@ struct clone_struct {
 	char	*buf;
 	e2fsck_t ctx;
 	struct ext2_inode	*inode;
+
+	struct dup_cluster *save_dup_cluster;
+	blk64_t save_blocknr;
 };
 
+/*
+ * Decrement the bad count *after* we've shown that (a) we can allocate a
+ * replacement block and (b) remap the file blocks.  Unfortunately, there's no
+ * way to find out if the remap succeeded until either the next
+ * clone_file_block() call (an error when remapping the block after returning
+ * BLOCK_CHANGED will halt the iteration) or after block_iterate() returns.
+ * Otherwise, it's possible that we decrease the badcount once in preparation
+ * to remap, then the remap fails (either we can't find a replacement block or
+ * we have to split the extent tree and can't find a new extent block), so we
+ * delete the file, which decreases the badcount again.
+ */
+static void deferred_dec_badcount(struct clone_struct *cs)
+{
+	if (!cs->save_dup_cluster)
+		return;
+	decrement_badcount(cs->ctx, cs->save_blocknr, cs->save_dup_cluster);
+	cs->save_dup_cluster = NULL;
+}
+
 static int clone_file_block(ext2_filsys fs,
 			    blk64_t	*block_nr,
 			    e2_blkcnt_t blockcnt,
@@ -734,7 +756,7 @@ static int clone_file_block(ext2_filsys fs,
 			    int ref_offset EXT2FS_ATTR((unused)),
 			    void *priv_data)
 {
-	struct dup_cluster *p;
+	struct dup_cluster *p = NULL;
 	blk64_t	new_block;
 	errcode_t	retval;
 	struct clone_struct *cs = (struct clone_struct *) priv_data;
@@ -744,6 +766,7 @@ static int clone_file_block(ext2_filsys fs,
 	int is_meta = 0;
 
 	ctx = cs->ctx;
+	deferred_dec_badcount(cs);
 
 	if (HOLE_BLKADDR(*block_nr))
 		return 0;
@@ -768,8 +791,6 @@ static int clone_file_block(ext2_filsys fs,
 		}
 
 		p = (struct dup_cluster *) dnode_get(n);
-		if (!is_meta)
-			decrement_badcount(ctx, *block_nr, p);
 
 		cs->dup_cluster = c;
 		/*
@@ -819,6 +840,8 @@ cluster_alloc_ok:
 			cs->errcode = retval;
 			return BLOCK_ABORT;
 		}
+		cs->save_dup_cluster = (is_meta ? NULL : p);
+		cs->save_blocknr = *block_nr;
 		*block_nr = new_block;
 		ext2fs_mark_block_bitmap2(ctx->block_found_map, new_block);
 		ext2fs_mark_block_bitmap2(fs->block_map, new_block);
@@ -848,6 +871,8 @@ static errcode_t clone_file(e2fsck_t ctx, ext2_ino_t ino,
 	cs.ctx = ctx;
 	cs.ino = ino;
 	cs.inode = &dp->inode;
+	cs.save_dup_cluster = NULL;
+	cs.save_blocknr = 0;
 	retval = ext2fs_get_mem(fs->blocksize, &cs.buf);
 	if (retval)
 		return retval;
@@ -860,6 +885,7 @@ static errcode_t clone_file(e2fsck_t ctx, ext2_ino_t ino,
 	if (ext2fs_inode_has_valid_blocks2(fs, &dp->inode))
 		pctx.errcode = ext2fs_block_iterate3(fs, ino, 0, block_buf,
 						     clone_file_block, &cs);
+	deferred_dec_badcount(&cs);
 	ext2fs_mark_bb_dirty(fs);
 	if (pctx.errcode) {
 		fix_problem(ctx, PR_1B_BLOCK_ITERATE, &pctx);
diff --git a/tests/f_cloneblock_alloc_error/expect.1 b/tests/f_cloneblock_alloc_error/expect.1
new file mode 100644
index 0000000..24fe1ff
--- /dev/null
+++ b/tests/f_cloneblock_alloc_error/expect.1
@@ -0,0 +1,36 @@
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12, i_size is 0, should be 2015232.  Fix? yes
+
+
+Running additional passes to resolve blocks claimed by more than one inode...
+Pass 1B: Rescanning for multiply-claimed blocks
+Multiply-claimed block(s) in inode 13: 8
+Multiply-claimed block(s) in inode 14: 8
+Pass 1C: Scanning directories for inodes with multiply-claimed blocks
+Pass 1D: Reconciling multiply-claimed blocks
+(There are 2 inodes containing multiply-claimed blocks.)
+
+File /b (inode #13, mod time Wed Jan 21 03:41:55 2015) 
+  has 1 multiply-claimed block(s), shared with 1 file(s):
+	/c (inode #14, mod time Wed Jan 21 03:42:37 2015)
+Clone multiply-claimed blocks? yes
+
+clone_file: Could not allocate block in ext2 filesystem returned from clone_file_block
+Couldn't clone file: Could not allocate block in ext2 filesystem
+Delete file? yes
+
+File /c (inode #14, mod time Wed Jan 21 03:42:37 2015) 
+  has 1 multiply-claimed block(s), shared with 1 file(s):
+	/b (inode #13, mod time Wed Jan 21 03:41:55 2015)
+Multiply-claimed blocks already reassigned or cloned.
+
+Pass 2: Checking directory structure
+Entry 'b' in / (2) has deleted/unused inode 13.  Clear? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 13/128 files (7.7% non-contiguous), 512/512 blocks
+Exit status is 1
diff --git a/tests/f_cloneblock_alloc_error/expect.2 b/tests/f_cloneblock_alloc_error/expect.2
new file mode 100644
index 0000000..f7781ec
--- /dev/null
+++ b/tests/f_cloneblock_alloc_error/expect.2
@@ -0,0 +1,7 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 13/128 files (7.7% non-contiguous), 512/512 blocks
+Exit status is 0
diff --git a/tests/f_cloneblock_alloc_error/image.gz b/tests/f_cloneblock_alloc_error/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..ed01df1b9fb7a10f49338cd994ca4ef4cf646bba
GIT binary patch
literal 3441
zcmeH{X;4#l5Ws^6bgBfaRi{$GYNyn(k~#_sX`qVASmjU*F@X>Ulq2EL(wE?)AyO;e
zF!hQDhoC}|mjpG22$)2xLXiqZ-%$cVl<|7cc;O%FQ}T+_Z~fFS{l<RSot-^q=Qq0>
zu<rv__H6rxapPUR8HbZLdu+ew^!~7R%7hhHGslH2{^F<It5!@tbvR{bO60ui-Lv+5
z`{mx$RFCP)4$qOtY2_1EteiT<ckjf+r{3p^KVEiq`H$-C(qo4tb36tHpW5<=pY<+M
z*pG)Q>|Ab!m*;jyFj&GV@$|&~gb^v5LKol8@%`PmJIEBc;8^I_=L%lN5N!7323zL+
z7xdH}bK4iroqwk!@3e2I{(&bqPkv{=DI*)c`Cz8O(tfc`-8((ax#^{rTPl)K!oVtf
z*Cp@PY}>;Zom_cl8)OM+Rki;d?x`Le;gh|5-Y|=`@nuP!UB>S0j7n!BEw^ISm6O&U
zqx{FQSlb!pK95mzr0>ODrj<RJ9qqS`bLG$C>S}Gg`tmM?tGFxD^yt&2idW(JY)4-G
zJu0_HzNPHwz$#lM_C07ydDq`xnH=-JMf2H(?d8nfrA2LTB3LZexr4%M;!(@$U`C^?
zN6e_Y;_S{;y!ZZ#%rCwXD@ykpP2KpUVB&;-l{|xo5lCijQblXL(DAa=XdR$F3U2fl
z#!IC<U5M@g)n#7Is|_)<;sa6<&lZ9Nq7T7!M@<O>`d>RPH0TUe8#wH!Z7{f`Z%?e_
zqVZG~cnz=OnJW2QJSNyAH)as|WFxo?7rEARG&6CdInk9|i0U1tA)QrL1+Q}@a*S$p
zjZALU4<Vu8mBY@~jF;v5ZSW#KgEFl($--S2LSj&Xwe}(Q4$iR}lO=hYx#VwfnpNAE
z6on;{kHc#FM6Gxa>4K8{B!TE2P#UJ~6F%1|h(gi|Y_O^#sSfobcYx|Ja}08a+8<`>
z(D@Ki<Y{o8E>PBl3p0p1vOLVvfy5GTz;QZTX$;bI5<AVOX5cHKt;*^kEeBVV?cfM4
z6^kTDJ+T=apwpFzO{^vqWHYeRMI)zXU_Mk77@${=Tr5h6sF$D|N&u*)8lxT|mw+Bx
zJYsXl35a0O+SFh~D5`{xcWzvQRZ$!$-??68uw!?r07&K>>RjUaR$o1fkDxrV(kwM3
z4C}grYg_P3B5MmV3wkax;W{%FZjNzUo(U)%hL{s^J{m9l`iTt`;R#N`TX<*uMLvi(
zbr0-5kyNIi7i{+D`(iENvEz(?{7F2G+8S)|mz>0=NeAK7oOQ9fC+5vL{6yjqHP0NK
zlh}g|1)r8{>hL?{6R;mn=OJ=^g*4rj)M<Q#=2P3iJIC4NIF4Q{vpB3n8lgGE#V;hn
zL5;(boW#+p@u#rRCCDcqm{oppUU+YqhHZ=_|A1FoFR&2-hEOly=v&Ae;DE(eTVGrh
z`U_}*<E@RQIt$tbt+kp<4Hm42x)f%M(M+TKp!HT$AOE>=7%ilfz#Y~YqxC_(s7Od!
zkx)l`0;N~*1B_*86qN>E(kqk~I*hScD3n_vs4@;xi$FJ>B#x`9pCnthK);W?PcIj*
zm*_c|nwk%=ri-=&Nqsx%3vST~%I2W@=cs}z1S9ma1-bydjBEvm=&TWaQL<el!Rtu^
zS}7I^usqTNnrY=owb3Xe%b*;wfFQg<UDNqoT?8Q^A2ywJi&Ti6h#+y%H<|ALdaW#$
z;(E=VN$Ei@Caago?!Gar-Ivi6W4RWmhMUDqJ36?uAY|P9UoXEAe^qWPAGq6Z_ha1C
z(vAuPet5T|`N+;|gH=!3PcoCe*ETuj|7;KW-C`y*&wcO#=2B#>DNA@;$`JDR`_#~a
zk-<f6=+^!|bAmxUtc6G$bMLSE<f{V_8SaAvuQaPiC&}5?%3#mVjBvPal}mMl=e{;j
ub-PF{7nG`Qer|vNkH%z-z!-rs0%HWm2#gW<KM3@f(t)g$<=rgSY}ViS*fH|}

literal 0
HcmV?d00001

diff --git a/tests/f_cloneblock_alloc_error/name b/tests/f_cloneblock_alloc_error/name
new file mode 100644
index 0000000..9196e89
--- /dev/null
+++ b/tests/f_cloneblock_alloc_error/name
@@ -0,0 +1 @@
+decrement badcount after remapping duplicate block

[PATCH 17/54] e2fsck: inspect inline dir data as two directory blocks

[Darrick J. Wong]

The design of inline directories (apparently) calls for the i_block[]
region and the EA regions to be treated as if they were two separate
blocks of dirents.  Effectively this means that it is impossible for a
directory entry to straddle both areas.  e2fsck doesn't enforce this,
so teach it to do so.  e2fslib already knows to do this....

Cc: Zheng Liu <gnehzuil.liu@gmail.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c                        |   34 ++++++++++-
 e2fsck/pass2.c                        |  104 ++++++++++++++++++++++-----------
 tests/f_inlinedata_dirblocks/expect.1 |   25 ++++++++
 tests/f_inlinedata_dirblocks/expect.2 |    7 ++
 tests/f_inlinedata_dirblocks/image.gz |  Bin
 tests/f_inlinedata_dirblocks/name     |    1 
 6 files changed, 131 insertions(+), 40 deletions(-)
 create mode 100644 tests/f_inlinedata_dirblocks/expect.1
 create mode 100644 tests/f_inlinedata_dirblocks/expect.2
 create mode 100644 tests/f_inlinedata_dirblocks/image.gz
 create mode 100644 tests/f_inlinedata_dirblocks/name


diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 2089943..69a9ac1 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -2742,18 +2742,44 @@ static void check_blocks_extents(e2fsck_t ctx, struct problem_context *pctx,
 static void check_blocks_inline_data(e2fsck_t ctx, struct problem_context *pctx,
 				     struct process_block_struct *pb)
 {
+	int	flags;
+	size_t	inline_data_size = 0;
+
 	if (!pb->is_dir) {
 		pctx->errcode = 0;
 		return;
 	}
 
+	/* Process the dirents in i_block[] as the "first" block. */
 	pctx->errcode = ext2fs_add_dir_block2(ctx->fs->dblist, pb->ino, 0, 0);
+	if (pctx->errcode)
+		goto err;
+
+	/* Process the dirents in the EA as a "second" block. */
+	flags = ctx->fs->flags;
+	ctx->fs->flags |= EXT2_FLAG_IGNORE_CSUM_ERRORS;
+	pctx->errcode = ext2fs_inline_data_size(ctx->fs, pb->ino,
+						&inline_data_size);
+	ctx->fs->flags = (flags & EXT2_FLAG_IGNORE_CSUM_ERRORS) |
+			 (ctx->fs->flags & ~EXT2_FLAG_IGNORE_CSUM_ERRORS);
 	if (pctx->errcode) {
-		pctx->blk = 0;
-		pctx->num = 0;
-		fix_problem(ctx, PR_1_ADD_DBLOCK, pctx);
-		ctx->flags |= E2F_FLAG_ABORT;
+		pctx->errcode = 0;
+		return;
 	}
+
+	if (inline_data_size <= EXT4_MIN_INLINE_DATA_SIZE)
+		return;
+
+	pctx->errcode = ext2fs_add_dir_block2(ctx->fs->dblist, pb->ino, 0, 1);
+	if (pctx->errcode)
+		goto err;
+
+	return;
+err:
+	pctx->blk = 0;
+	pctx->num = 0;
+	fix_problem(ctx, PR_1_ADD_DBLOCK, pctx);
+	ctx->flags |= E2F_FLAG_ABORT;
 }
 
 /*
diff --git a/e2fsck/pass2.c b/e2fsck/pass2.c
index 7aaebce..26db2e6 100644
--- a/e2fsck/pass2.c
+++ b/e2fsck/pass2.c
@@ -732,15 +732,6 @@ static void salvage_directory(ext2_filsys fs,
 	}
 }
 
-static int is_last_entry(ext2_filsys fs, int inline_data_size,
-			 unsigned int offset, int csum_size)
-{
-	if (inline_data_size)
-		return (offset < inline_data_size);
-	else
-		return (offset < fs->blocksize - csum_size);
-}
-
 #define NEXT_DIRENT(d)	((void *)((char *)(d) + (d)->rec_len))
 static errcode_t insert_dirent_tail(ext2_filsys fs, void *dirbuf)
 {
@@ -783,10 +774,22 @@ static errcode_t fix_inline_dir_size(e2fsck_t ctx, ext2_ino_t ino,
 	errcode_t retval;
 
 	old_size = *inline_data_size;
-	new_size = old_size + (4 - (old_size & 3));
+	/*
+	 * If there's not enough bytes to start the "second" dir block
+	 * (in the EA space) then truncate everything to the first block.
+	 */
+	if (old_size > EXT4_MIN_INLINE_DATA_SIZE &&
+	    old_size < EXT4_MIN_INLINE_DATA_SIZE +
+		       EXT2_DIR_REC_LEN(1)) {
+		old_size = EXT4_MIN_INLINE_DATA_SIZE;
+		new_size = old_size;
+	} else
+		/* Increase to the next four-byte boundary for salvaging */
+		new_size = old_size + (4 - (old_size & 3));
 	memset(buf + old_size, 0, new_size - old_size);
 	retval = ext2fs_inline_data_set(fs, ino, 0, buf, new_size);
 	if (retval == EXT2_ET_INLINE_DATA_NO_SPACE) {
+		/* Or we can't, so truncate. */
 		new_size -= 4;
 		retval = ext2fs_inline_data_set(fs, ino, 0, buf, new_size);
 		if (retval) {
@@ -844,7 +847,7 @@ static int check_dir_block(ext2_filsys fs,
 	ext2_ino_t 		subdir_parent;
 	__u16			links;
 	struct check_dir_struct	*cd;
-	char 			*buf;
+	char			*buf, *ibuf;
 	e2fsck_t		ctx;
 	problem_t		problem;
 	struct ext2_dx_root_info *root;
@@ -858,9 +861,10 @@ static int check_dir_block(ext2_filsys fs,
 	int	is_leaf = 1;
 	size_t	inline_data_size = 0;
 	int	filetype = 0;
+	size_t	max_block_size;
 
 	cd = (struct check_dir_struct *) priv_data;
-	buf = cd->buf;
+	ibuf = buf = cd->buf;
 	ctx = cd->ctx;
 
 	if (ctx->flags & E2F_FLAG_SIGNAL_MASK || ctx->flags & E2F_FLAG_RESTART)
@@ -928,10 +932,22 @@ static int check_dir_block(ext2_filsys fs,
 		if (cd->pctx.errcode)
 			goto inline_read_fail;
 #ifdef WORDS_BIGENDIAN
+		if (db->blockcnt)
+			goto skip_first_read_swab;
 		*((__u32 *)buf) = ext2fs_le32_to_cpu(*((__u32 *)buf));
 		cd->pctx.errcode = ext2fs_dirent_swab_in2(fs,
 				buf + EXT4_INLINE_DATA_DOTDOT_SIZE,
-				inline_data_size - EXT4_INLINE_DATA_DOTDOT_SIZE,
+				EXT4_MIN_INLINE_DATA_SIZE - EXT4_INLINE_DATA_DOTDOT_SIZE,
+				0);
+		if (cd->pctx.errcode)
+			goto inline_read_fail;
+skip_first_read_swab:
+		if (inline_data_size <= EXT4_MIN_INLINE_DATA_SIZE ||
+		    !db->blockcnt)
+			goto inline_read_fail;
+		cd->pctx.errcode = ext2fs_dirent_swab_in2(fs,
+				buf + EXT4_MIN_INLINE_DATA_SIZE,
+				inline_data_size - EXT4_MIN_INLINE_DATA_SIZE,
 				0);
 #endif
 	} else
@@ -940,7 +956,10 @@ static int check_dir_block(ext2_filsys fs,
 inline_read_fail:
 	pctx.ino = ino;
 	pctx.num = inline_data_size;
-	if ((inline_data_size & 3) &&
+	if (((inline_data_size & 3) ||
+	     (inline_data_size > EXT4_MIN_INLINE_DATA_SIZE &&
+	      inline_data_size < EXT4_MIN_INLINE_DATA_SIZE +
+				 EXT2_DIR_REC_LEN(1))) &&
 	    fix_problem(ctx, PR_2_BAD_INLINE_DIR_SIZE, &pctx)) {
 		errcode_t err = fix_inline_dir_size(ctx, ino,
 						    &inline_data_size, &pctx,
@@ -1039,6 +1058,19 @@ out_htree:
 		de_csum_size = 0;
 
 skip_checksum:
+	if (inline_data_size) {
+		if (db->blockcnt) {
+			buf += EXT4_MIN_INLINE_DATA_SIZE;
+			max_block_size = inline_data_size - EXT4_MIN_INLINE_DATA_SIZE;
+			/* Zero-length second block, just exit */
+			if (max_block_size == 0)
+				return 0;
+		} else {
+			max_block_size = EXT4_MIN_INLINE_DATA_SIZE;
+		}
+	} else
+		max_block_size = fs->blocksize - de_csum_size;
+
 	dict_init(&de_dict, DICTCOUNT_T_MAX, dict_de_cmp);
 	prev = 0;
 	do {
@@ -1048,10 +1080,6 @@ skip_checksum:
 
 		problem = 0;
 		if (!inline_data_size || dot_state > 1) {
-			size_t max_block_size = fs->blocksize - de_csum_size;
-
-			if (inline_data_size)
-				max_block_size = inline_data_size;
 			dirent = (struct ext2_dir_entry *) (buf + offset);
 			(void) ext2fs_get_rec_len(fs, dirent, &rec_len);
 			cd->pctx.dirent = dirent;
@@ -1388,7 +1416,7 @@ skip_checksum:
 			}
 		}
 		dot_state++;
-	} while (is_last_entry(fs, inline_data_size, offset, de_csum_size));
+	} while (offset < max_block_size);
 #if 0
 	printf("\n");
 #endif
@@ -1406,22 +1434,11 @@ skip_checksum:
 	}
 #endif /* ENABLE_HTREE */
 
-	if (inline_data_size) {
-		if (offset != inline_data_size) {
-			cd->pctx.num = rec_len + offset - inline_data_size;
-			if (fix_problem(ctx, PR_2_FINAL_RECLEN, &cd->pctx)) {
-				dirent->rec_len = cd->pctx.num;
-				dir_modified++;
-			}
-		}
-	} else {
-		if (offset != fs->blocksize - de_csum_size) {
-			cd->pctx.num = rec_len - (fs->blocksize - de_csum_size) +
-				       offset;
-			if (fix_problem(ctx, PR_2_FINAL_RECLEN, &cd->pctx)) {
-				dirent->rec_len = cd->pctx.num;
-				dir_modified++;
-			}
+	if (offset != max_block_size) {
+		cd->pctx.num = rec_len + offset - max_block_size;
+		if (fix_problem(ctx, PR_2_FINAL_RECLEN, &cd->pctx)) {
+			dirent->rec_len = cd->pctx.num;
+			dir_modified++;
 		}
 	}
 	if (dir_modified) {
@@ -1444,13 +1461,28 @@ write_and_fix:
 			ctx->fs->flags |= EXT2_FLAG_IGNORE_CSUM_ERRORS;
 		}
 		if (inline_data_size) {
+			buf = ibuf;
 #ifdef WORDS_BIGENDIAN
+			if (db->blockcnt)
+				goto skip_first_write_swab;
 			*((__u32 *)buf) = ext2fs_le32_to_cpu(*((__u32 *)buf));
 			cd->pctx.errcode = ext2fs_dirent_swab_out2(fs,
 					buf + EXT4_INLINE_DATA_DOTDOT_SIZE,
-					inline_data_size -
+					EXT4_MIN_INLINE_DATA_SIZE -
 					EXT4_INLINE_DATA_DOTDOT_SIZE,
 					0);
+			if (cd->pctx.errcode)
+				goto skip_second_write_swab;
+skip_first_write_swab:
+			if (inline_data_size <= EXT4_MIN_INLINE_DATA_SIZE ||
+			    !db->blockcnt)
+				goto skip_second_write_swab;
+			cd->pctx.errcode = ext2fs_dirent_swab_out2(fs,
+					buf + EXT4_MIN_INLINE_DATA_SIZE,
+					inline_data_size -
+					EXT4_MIN_INLINE_DATA_SIZE,
+					0);
+skip_second_write_swab:
 			if (cd->pctx.errcode &&
 			    !fix_problem(ctx, PR_2_WRITE_DIRBLOCK, &cd->pctx))
 				goto abort_free_dict;
diff --git a/tests/f_inlinedata_dirblocks/expect.1 b/tests/f_inlinedata_dirblocks/expect.1
new file mode 100644
index 0000000..e3d0ee2
--- /dev/null
+++ b/tests/f_inlinedata_dirblocks/expect.1
@@ -0,0 +1,25 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Entry '..' in ??? (12) has invalid inode #: 1752440867.
+Clear? yes
+
+Directory inode 12, block #0, offset 4: directory corrupted
+Salvage? yes
+
+Directory inode 12, block #1, offset 0: directory corrupted
+Salvage? yes
+
+Pass 3: Checking directory connectivity
+'..' in /aoo (12) is ??? (1752440867), should be / (2).
+Fix? yes
+
+Error while adjusting inode count on inode 0
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Directories count wrong for group #0 (2, counted=3).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 1
diff --git a/tests/f_inlinedata_dirblocks/expect.2 b/tests/f_inlinedata_dirblocks/expect.2
new file mode 100644
index 0000000..3b6073e
--- /dev/null
+++ b/tests/f_inlinedata_dirblocks/expect.2
@@ -0,0 +1,7 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 0
diff --git a/tests/f_inlinedata_dirblocks/image.gz b/tests/f_inlinedata_dirblocks/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..4cd64e01348c4370f856ac847287a14d0e3463ba
GIT binary patch
literal 2597
zcmb2|=3qGOxIcu6`R%QP*&^XG3=hmFi>3;DepoW$u(Ito>8%<SnkgR@L_}BKOEuXo
z6|q~<`k)iP?-dyv5uGnjmQLpix+PZ~TBY^Fyh4*hX^LFc+9T$>r6+C>+v1k|XW!Gk
z_p6Qb%YL3c!<S#C&ho%{Yfxw<2Yb*DYwm?Idgt4h+9#x1=v7|WQ?A$Y_3P%$0x7v`
zzy3DnNE~mUo2>b#PPb)!@SeJF*Ke<Ox4&PX{Pm0e(Yfya_gvSp%ipW5Do%QLTV77?
z?$^Kb_y3*Jy3=4H|Fyj{j?Xrl8+Wtg>DG`k%dZR!73`~CzTX#;_UrS1qp#WLuTOpW
zfU%l`fq~&({rxv9TkDU<)&DL9auaT^{`Y@AN2HxJ+j%CR7MuBVWo-^iJKMf5seXF=
z?YmFYKVRLoswY)^*WJ3`{U<l>@yG>fs+O)l`M3V*$0dK6fn4ijQ(}P>!@K{CAc}a9
zkiFJF?-kgHzIX@X^<k*25Y0YY{oL$S`;_nTXG45@Pu1(Zd3oG1_P;v!R`G48e<}al
z|5EVy!J8a=c(qrr++}b6=+|`K^)>7BYl7{I|2F47tuy^^X_EDCa{J%^hW}SSv)igW
z|Nj&JZC3x~ga37F-t(HjJJ9Tt{H#^0`jaXv7mDBJo*VXN>#4Xo`CGf6`ekj|?Y%K#
zkBIoK<_q28-G0kW<=5*?nZEmUYD~@4Kl!<*_HX+=x$d=xR<*T#y?OkUqo?n!j?h}S
z<<zyH^>O+0EDQhneC`gOf7ZnB=;h=7$J_+@kGkobtviyu@nfjXpUj*o#$R{kWB7Sg
jU^E0qLtr!nMnhmU1V%%EN+Gb}vJG?isTY$O7!())I%6w0

literal 0
HcmV?d00001

diff --git a/tests/f_inlinedata_dirblocks/name b/tests/f_inlinedata_dirblocks/name
new file mode 100644
index 0000000..a226758
--- /dev/null
+++ b/tests/f_inlinedata_dirblocks/name
@@ -0,0 +1 @@
+check inline dir as two dirent blocks

[PATCH 18/54] e2fsck: improve the inline directory detector

[Darrick J. Wong]

Strengthen the checks that guess if the inode we're looking at is an
inline directory.  The current check sweeps up any inline inode if
its length is a multiple of four; now we'll at least try to see if
there's the beginning of a valid directory entry.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c                      |   23 +++++++++++++++++++++++
 tests/f_inlinedata_repair/expect.1  |   26 +++-----------------------
 tests/f_inlinedata_repair/expect.2  |    2 +-
 tests/f_inlinedir_detector/expect.1 |   20 ++++++++++++++++++++
 tests/f_inlinedir_detector/expect.2 |    7 +++++++
 tests/f_inlinedir_detector/image.gz |  Bin
 tests/f_inlinedir_detector/name     |    1 +
 7 files changed, 55 insertions(+), 24 deletions(-)
 create mode 100644 tests/f_inlinedir_detector/expect.1
 create mode 100644 tests/f_inlinedir_detector/expect.2
 create mode 100644 tests/f_inlinedir_detector/image.gz
 create mode 100644 tests/f_inlinedir_detector/name


diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 69a9ac1..9f22826 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -537,6 +537,9 @@ static void check_is_really_dir(e2fsck_t ctx, struct problem_context *pctx,
 			 EXT4_FEATURE_INCOMPAT_INLINE_DATA);
 	if (inlinedata_fs && (inode->i_flags & EXT4_INLINE_DATA_FL)) {
 		size_t size;
+		__u32 dotdot;
+		unsigned int rec_len;
+		struct ext2_dir_entry de;
 
 		if (ext2fs_inline_data_size(ctx->fs, pctx->ino, &size))
 			return;
@@ -546,6 +549,26 @@ static void check_is_really_dir(e2fsck_t ctx, struct problem_context *pctx,
 		 */
 		if (size & 3)
 			return;
+		/*
+		 * If the first 10 bytes don't look like a directory entry,
+		 * it's probably not a directory.
+		 */
+		memcpy(&dotdot, inode->i_block, sizeof(dotdot));
+		memcpy(&de, ((char *)inode->i_block) + EXT4_INLINE_DATA_DOTDOT_SIZE,
+		       EXT2_DIR_REC_LEN(0));
+		dotdot = ext2fs_le32_to_cpu(dotdot);
+		de.inode = ext2fs_le32_to_cpu(de.inode);
+		de.rec_len = ext2fs_le16_to_cpu(de.rec_len);
+		ext2fs_get_rec_len(ctx->fs, &de, &rec_len);
+		if (dotdot >= ctx->fs->super->s_inodes_count ||
+		    (dotdot < EXT2_FIRST_INO(ctx->fs->super) &&
+		     dotdot != EXT2_ROOT_INO) ||
+		    de.inode >= ctx->fs->super->s_inodes_count ||
+		    (de.inode < EXT2_FIRST_INO(ctx->fs->super) &&
+		     de.inode != 0) ||
+		    rec_len > EXT4_MIN_INLINE_DATA_SIZE -
+			      EXT4_INLINE_DATA_DOTDOT_SIZE)
+			return;
 		/* device files never have a "system.data" entry */
 		goto isdir;
 	} else if (extent_fs && (inode->i_flags & EXT4_EXTENTS_FL)) {
diff --git a/tests/f_inlinedata_repair/expect.1 b/tests/f_inlinedata_repair/expect.1
index faba192..cc220ba 100644
--- a/tests/f_inlinedata_repair/expect.1
+++ b/tests/f_inlinedata_repair/expect.1
@@ -8,11 +8,6 @@ Inode 24, i_size is 59, should be 60.  Fix? yes
 Inode 28 is a unknown file type with mode 00 but it looks like it is really a directory.
 Fix? yes
 
-Inode 36 is a unknown file type with mode 00 but it looks like it is really a directory.
-Fix? yes
-
-Inode 36, i_size is 5, should be 60.  Fix? yes
-
 Pass 2: Checking directory structure
 Directory inode 20, block #0, offset 4: directory corrupted
 Salvage? yes
@@ -26,26 +21,16 @@ Salvage? yes
 Directory inode 32, block #0, offset 4: directory corrupted
 Salvage? yes
 
-Entry '..' in ??? (36) has invalid inode #: 1633774699.
-Clear? yes
-
-Directory inode 36, block #0, offset 4: directory corrupted
-Salvage? yes
-
 Symlink /3 (inode #14) is invalid.
 Clear? yes
 
 Inode 38 (/B) has invalid mode (00).
 Clear? yes
 
-Entry 'A' in / (2) has an incorrect filetype (was 1, should be 2).
-Fix? yes
+Inode 36 (/A) has invalid mode (00).
+Clear? yes
 
 Pass 3: Checking directory connectivity
-'..' in /A (36) is ??? (1633774699), should be / (2).
-Fix? yes
-
-Error while adjusting inode count on inode 0
 Pass 4: Checking reference counts
 Unattached zero-length inode 22.  Clear? yes
 
@@ -63,13 +48,8 @@ Unattached zero-length inode 34.  Clear? yes
 
 Unattached zero-length inode 35.  Clear? yes
 
-Inode 36 ref count is 1, should be 2.  Fix? yes
-
 Pass 5: Checking group summary information
-Directories count wrong for group #0 (7, counted=8).
-Fix? yes
-
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
-test_filesys: 28/128 files (0.0% non-contiguous), 18/512 blocks
+test_filesys: 27/128 files (0.0% non-contiguous), 18/512 blocks
 Exit status is 1
diff --git a/tests/f_inlinedata_repair/expect.2 b/tests/f_inlinedata_repair/expect.2
index 519f21d..2c400a5 100644
--- a/tests/f_inlinedata_repair/expect.2
+++ b/tests/f_inlinedata_repair/expect.2
@@ -3,5 +3,5 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-test_filesys: 28/128 files (0.0% non-contiguous), 18/512 blocks
+test_filesys: 27/128 files (0.0% non-contiguous), 18/512 blocks
 Exit status is 0
diff --git a/tests/f_inlinedir_detector/expect.1 b/tests/f_inlinedir_detector/expect.1
new file mode 100644
index 0000000..72b7519
--- /dev/null
+++ b/tests/f_inlinedir_detector/expect.1
@@ -0,0 +1,20 @@
+Pass 1: Checking inodes, blocks, and sizes
+Special (device/socket/fifo) file (inode 12) has extents
+or inline-data flag set.  Clear? yes
+
+Special (device/socket/fifo) inode 12 has non-zero size.  Fix? yes
+
+Inode 13 is a named pipe but it looks like it is really a directory.
+Fix? yes
+
+Pass 2: Checking directory structure
+Entry 'moo' in / (2) has an incorrect filetype (was 1, should be 5).
+Fix? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 13/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 1
diff --git a/tests/f_inlinedir_detector/expect.2 b/tests/f_inlinedir_detector/expect.2
new file mode 100644
index 0000000..06886a4
--- /dev/null
+++ b/tests/f_inlinedir_detector/expect.2
@@ -0,0 +1,7 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 13/128 files (0.0% non-contiguous), 17/512 blocks
+Exit status is 0
diff --git a/tests/f_inlinedir_detector/image.gz b/tests/f_inlinedir_detector/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..34b4518849cd60d83890b5c69ff6a0aefaa54d57
GIT binary patch
literal 2640
zcmb2|=3qD$xIcu6`R%Rq*)pLr3?IxlYj2y~vTS8L7pwfo71CL^ompDhH082Xu2x-J
zdNY?<+C0T9M8L30!*$}L`UeZYT#5dA=fTO<wpEvAbG&lg(;;yG4S)2t+=Z$eGk^co
zx&Qsy**!NOS{vKkJCnz8U~<-GolgSue6PNb<T#p8Yu6Fie%5jSlDcQlZVGJMe&26y
z@b?Y-ufIFZ_VvgD!}+W7?Rg&_4#}VU>*J*>XC9f)zaRDc)u-$7v6T`#>;B)qdb&Q;
ze}8fA%k%nvcYY<>ajPFrxp%)TUC#Qc$FG#+YvI8*Pp>gEJW$kFzTa3gt@`tSrI4!4
zm-i{%skkZ0#K6$te*gIIRS(~vYJR_;mw|!dP36<~|0gS+e(v;tKf`3j^Ut6C`v0(2
z?aHUJ<uCSKc$Kqfe@veC;z?`EtP-_!>&q8Rn*TcHix4YN=eHf5yKVkYud{pb2go~+
zoP0|YNHd)O4<bnb8+LD-FP+H@)X^`S??8e^bfr7wf1kf&W8I^!`95Ae$k0~rf7Xrb
ztAD4y?blhmQ`bB2X5(Xd$zs_%y$Aj$ex4kXGgrp=UuyHj_XmA@wirK_f1?@w^w0Ab
zJAV4h@!e1U7yR<<gtF>?lmBnJ{WIQH;!xD8^h<^9hDK4o(~o}MSe>}EW9`&3W3jV&
zg41WOJ?3VuyT$7D#*Jlm>$dy~6+CZx>+Q3mJv%P#y>Rozv8}&P)~Bibe|1^^+cdu=
zGqU3U7wx>5d*|(Lsn@+%)3)!7yOPE2y?+1hEg!BuTD|$}(I3yU^N&<1%k_R$x)%Pp
zW=*|7yp`DNyEmiOp7}p@%lm~TcM^Hl|DXGlKl-zE%|G|g_Fv}zx&M3n*URb05AXfN
mjvg+f{Lv5?4S~@R7!85Z5Eu;sDusZ?U*<Usuk0BZ6c_*iEnI&9

literal 0
HcmV?d00001

diff --git a/tests/f_inlinedir_detector/name b/tests/f_inlinedir_detector/name
new file mode 100644
index 0000000..3368af5
--- /dev/null
+++ b/tests/f_inlinedir_detector/name
@@ -0,0 +1 @@
+detect inline dirs correctly

[PATCH 19/54] e2fsck: salvage under-sized dirents by removing them

[Darrick J. Wong]

If the directory processing code ends up pointing to a directory entry
that's so close to the end of the block that there's not even space
for a rec_len/name_len, just substitute dummy values that will force
e2fsck to extend the previous entry to cover the remaining space.  We
can't use the helper methods to extract rec_len because that's reading
off the end of the buffer.

This isn't an issue with non-inline directories because the directory
check buffer is zero-extended so that fsck won't blow up.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass2.c                       |   49 +++++++++++++++++++++++-----------
 lib/ext2fs/ext2_fs.h                 |    5 +++
 tests/f_trunc_dirent_header/expect.1 |   15 ++++++++++
 tests/f_trunc_dirent_header/expect.2 |    7 +++++
 tests/f_trunc_dirent_header/image.gz |  Bin
 tests/f_trunc_dirent_header/name     |    1 +
 6 files changed, 60 insertions(+), 17 deletions(-)
 create mode 100644 tests/f_trunc_dirent_header/expect.1
 create mode 100644 tests/f_trunc_dirent_header/expect.2
 create mode 100644 tests/f_trunc_dirent_header/image.gz
 create mode 100644 tests/f_trunc_dirent_header/name


diff --git a/e2fsck/pass2.c b/e2fsck/pass2.c
index 26db2e6..78bc24e 100644
--- a/e2fsck/pass2.c
+++ b/e2fsck/pass2.c
@@ -672,18 +672,29 @@ static void salvage_directory(ext2_filsys fs,
 	char	*cp = (char *) dirent;
 	int left;
 	unsigned int rec_len, prev_rec_len;
-	unsigned int name_len = ext2fs_dirent_name_len(dirent);
+	unsigned int name_len;
 
-	(void) ext2fs_get_rec_len(fs, dirent, &rec_len);
+	/*
+	 * If the space left for the entry is too small to be an entry,
+	 * we can't access dirent's fields, so plumb in the values needed
+	 * so that the previous entry absorbs this one.
+	 */
+	if (block_len - *offset < EXT2_DIR_ENTRY_HEADER_LEN) {
+		name_len = 0;
+		rec_len = block_len - *offset;
+	} else {
+		name_len = ext2fs_dirent_name_len(dirent);
+		(void) ext2fs_get_rec_len(fs, dirent, &rec_len);
+	}
 	left = block_len - *offset - rec_len;
 
 	/*
 	 * Special case of directory entry of size 8: copy what's left
 	 * of the directory block up to cover up the invalid hole.
 	 */
-	if ((left >= 12) && (rec_len == 8)) {
-		memmove(cp, cp+8, left);
-		memset(cp + left, 0, 8);
+	if ((left >= 12) && (rec_len == EXT2_DIR_ENTRY_HEADER_LEN)) {
+		memmove(cp, cp+EXT2_DIR_ENTRY_HEADER_LEN, left);
+		memset(cp + left, 0, EXT2_DIR_ENTRY_HEADER_LEN);
 		return;
 	}
 	/*
@@ -692,8 +703,8 @@ static void salvage_directory(ext2_filsys fs,
 	 * record length.
 	 */
 	if ((left < 0) &&
-	    ((int) rec_len + left > 8) &&
-	    ((int) name_len + 8 <= (int) rec_len + left) &&
+	    ((int) rec_len + left > EXT2_DIR_ENTRY_HEADER_LEN) &&
+	    ((int) name_len + EXT2_DIR_ENTRY_HEADER_LEN <= (int) rec_len + left) &&
 	    dirent->inode <= fs->super->s_inodes_count &&
 	    strnlen(dirent->name, name_len) == name_len) {
 		(void) ext2fs_set_rec_len(fs, (int) rec_len + left, dirent);
@@ -928,6 +939,7 @@ static int check_dir_block(ext2_filsys fs,
 
 	ehandler_operation(_("reading directory block"));
 	if (inline_data_size) {
+		memset(buf, 0, fs->blocksize - inline_data_size);
 		cd->pctx.errcode = ext2fs_inline_data_get(fs, ino, 0, buf, 0);
 		if (cd->pctx.errcode)
 			goto inline_read_fail;
@@ -1081,15 +1093,20 @@ skip_checksum:
 		problem = 0;
 		if (!inline_data_size || dot_state > 1) {
 			dirent = (struct ext2_dir_entry *) (buf + offset);
-			(void) ext2fs_get_rec_len(fs, dirent, &rec_len);
+			/*
+			 * If there's not even space for the entry header,
+			 * force salvaging this dir.
+			 */
+			if (max_block_size - offset < EXT2_DIR_ENTRY_HEADER_LEN)
+				rec_len = EXT2_DIR_REC_LEN(1);
+			else
+				(void) ext2fs_get_rec_len(fs, dirent, &rec_len);
 			cd->pctx.dirent = dirent;
 			cd->pctx.num = offset;
-			if (((offset + rec_len) > fs->blocksize) ||
-			    (inline_data_size > 0 &&
-			     (offset + rec_len) > inline_data_size) ||
+			if ((offset + rec_len > max_block_size) ||
 			    (rec_len < 12) ||
 			    ((rec_len % 4) != 0) ||
-			    ((ext2fs_dirent_name_len(dirent) + 8) > rec_len)) {
+			    ((ext2fs_dirent_name_len(dirent) + EXT2_DIR_ENTRY_HEADER_LEN) > rec_len)) {
 				if (fix_problem(ctx, PR_2_DIR_CORRUPTED,
 						&cd->pctx)) {
 #ifdef WORDS_BIGENDIAN
@@ -1098,7 +1115,7 @@ skip_checksum:
 					 * swap routine finds a rec_len that it
 					 * doesn't like, it continues
 					 * processing the block as if rec_len
-					 * == 8.  This means that the name
+					 * == EXT2_DIR_ENTRY_HEADER_LEN.  This means that the name
 					 * field gets byte swapped, which means
 					 * that salvage will not detect the
 					 * correct name length (unless the name
@@ -1112,11 +1129,11 @@ skip_checksum:
 					 * the salvaged dirent.
 					 */
 					int need_reswab = 0;
-					if (rec_len < 8 || rec_len % 4) {
+					if (rec_len < EXT2_DIR_ENTRY_HEADER_LEN || rec_len % 4) {
 						need_reswab = 1;
 						ext2fs_dirent_swab_in2(fs,
-							((char *)dirent) + 8,
-							max_block_size - offset - 8,
+							((char *)dirent) + EXT2_DIR_ENTRY_HEADER_LEN,
+							max_block_size - offset - EXT2_DIR_ENTRY_HEADER_LEN,
 							0);
 					}
 #endif
diff --git a/lib/ext2fs/ext2_fs.h b/lib/ext2fs/ext2_fs.h
index 953cbc4..10cb650 100644
--- a/lib/ext2fs/ext2_fs.h
+++ b/lib/ext2fs/ext2_fs.h
@@ -875,9 +875,12 @@ struct ext2_dir_entry_tail {
  *
  * NOTE: It must be a multiple of 4
  */
+#define EXT2_DIR_ENTRY_HEADER_LEN	8
 #define EXT2_DIR_PAD			4
 #define EXT2_DIR_ROUND			(EXT2_DIR_PAD - 1)
-#define EXT2_DIR_REC_LEN(name_len)	(((name_len) + 8 + EXT2_DIR_ROUND) & \
+#define EXT2_DIR_REC_LEN(name_len)	(((name_len) + \
+					  EXT2_DIR_ENTRY_HEADER_LEN + \
+					  EXT2_DIR_ROUND) & \
 					 ~EXT2_DIR_ROUND)
 
 /*
diff --git a/tests/f_trunc_dirent_header/expect.1 b/tests/f_trunc_dirent_header/expect.1
new file mode 100644
index 0000000..33ce473
--- /dev/null
+++ b/tests/f_trunc_dirent_header/expect.1
@@ -0,0 +1,15 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Directory inode 15, block #0, offset 4092: directory corrupted
+Salvage? yes
+
+Directory inode 13, block #1, offset 28: directory corrupted
+Salvage? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 32/128 files (0.0% non-contiguous), 18/512 blocks
+Exit status is 1
diff --git a/tests/f_trunc_dirent_header/expect.2 b/tests/f_trunc_dirent_header/expect.2
new file mode 100644
index 0000000..df81c9d
--- /dev/null
+++ b/tests/f_trunc_dirent_header/expect.2
@@ -0,0 +1,7 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 32/128 files (0.0% non-contiguous), 18/512 blocks
+Exit status is 0
diff --git a/tests/f_trunc_dirent_header/image.gz b/tests/f_trunc_dirent_header/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..9c59bf4a9e4e7fa07d501e4445a55d872a2e4715
GIT binary patch
literal 2873
zcmb2|=3sa(cRYlN`R$#v{xYdD>>tvXO%PhZn%VvI>ao>rGb$cFVDvxsYHhVp?t%+h
z-if!j^sW%kx^yW(?;rP9{SQe=UoKQUe0DQCf8UYm$GX{tYtqWyrS9C?!o>f!#_owZ
z|KT&wroXp2{p@C{6Nf|ST9Mf{EekFF%xX2b{HyZbs$-nmYd<eKf80DvF(+@{FJ11B
zUkZwr+{ix_mF%CLZp3%LPU?_);huu^)rG%qNiPrEKmX^W<LjsG_nY|jL;2U=^XJz7
ze)@HOe`#r4<>$3JKCSomRK7VVW1VVJmzrC5PjL0+V19-MmaeDo9oC3Tzy3ed;6}t;
zv$HMlzQ-ssFfdg7$}KBx+0XZ<|2Q*{`#bLA_y6}MJQZG8sqc_~;&W%`Guu7x&gFF*
z>z=;ye*WtEv}fJPy~Uf23vI4c-N}*pom6{y`qN@NKSl<IhT4_S72p5ow*Ruz4rtVe
zGcU?Hf%Jjj^+57LX4sua_uu{d|46yf{_y*scNXi|PwfA;`uVDvzaKYdJ)376boNeq
zUzqXtI~##K`88?pHlGhS{=W0^)iZzp1(+7s8^oNw^V}V%{7{Bzv3^ge@%N)2v!2D(
zES$A-y5EsIpC8Vx*%&u}b^Q)$-C6(7^8@W92Fzofeeqi@Q00T0kAww?(Sbva-0_nW
zd!r&lPyIdpy72OSz54VQSv_ZSzspCysCZBqxrW*LLj9JrHWg+})rn85?%bK1_4(KI
zt#Lo+#hl;v_snbgH`?XhU(aj5y=Gr&@lXEs{G_*+H@<#7|IhNf=l(za`YfWm_O#aL
z?O)!f{eSy2`}U`DnK?hx-`~5nI<fBCt2ch@>+knIE_oAu?HW)1Q~h<v9!5+~)QGQ}
zA$!99@A^N%z3=^w?AmX(^~;W*)zY?Xn~we}JDcqK@A{((t8coi@BWd`dh7G<)1Uc%
z2LD|T9oz`?$5~}}xu5@@KhS$}|J8o;{r{i-;Xl4Fb?fZ^`)p_Zf4Fh){QpY-e|`ii
z)cl<P_5STY+#B<OOnsG8`z3x)n`-nwY0)2Vh>+^3`2O0{UNiUW)ld2}J?Z|YplAE1
zX#9VAWBZyZKlRm5)hmC~)lU5<-TCQ$4@lkrv_*ebgSD<#J5{d@mW=NFwEpP6)TL+k
zPt*AS7OXR0{nUOruvJxy{**(E-F*BfFddG9(GVC7fzc2c4S~@R7!3hxhrp^=`8(bu
KePv)!U;qGw8T+09

literal 0
HcmV?d00001

diff --git a/tests/f_trunc_dirent_header/name b/tests/f_trunc_dirent_header/name
new file mode 100644
index 0000000..e244dbf
--- /dev/null
+++ b/tests/f_trunc_dirent_header/name
@@ -0,0 +1 @@
+no space for dirent header at end of buf

[PATCH 20/54] e2fsck: add a 'yes to all' response in interactive mode

[Darrick J. Wong]

Provide a mechanism for a user to switch fsck into '-y' mode if they
start an interactive session and then get tired of pressing 'y' in
response to numerous prompts.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/e2fsck.8.in        |   13 +++++++++
 e2fsck/unix.c             |    2 +
 e2fsck/util.c             |   36 ++++++++++++++++++++++----
 tests/f_no/expect         |   48 +++++++++++++++++++++++++++++++++++
 tests/f_no/script         |   27 ++++++++++++++++++++
 tests/f_yes/expect        |   45 +++++++++++++++++++++++++++++++++
 tests/f_yes/script        |   27 ++++++++++++++++++++
 tests/f_yesall/expect     |   62 +++++++++++++++++++++++++++++++++++++++++++++
 tests/f_yesall/image.gz   |  Bin
 tests/f_yesall/script     |   27 ++++++++++++++++++++
 tests/f_yesthenall/expect |   52 ++++++++++++++++++++++++++++++++++++++
 tests/f_yesthenall/script |   27 ++++++++++++++++++++
 tests/f_yesthenno/expect  |   50 ++++++++++++++++++++++++++++++++++++
 tests/f_yesthenno/script  |   27 ++++++++++++++++++++
 14 files changed, 436 insertions(+), 7 deletions(-)
 create mode 100644 tests/f_no/expect
 create mode 100644 tests/f_no/script
 create mode 100644 tests/f_yes/expect
 create mode 100644 tests/f_yes/script
 create mode 100644 tests/f_yesall/expect
 create mode 100644 tests/f_yesall/image.gz
 create mode 100644 tests/f_yesall/script
 create mode 100644 tests/f_yesthenall/expect
 create mode 100644 tests/f_yesthenall/script
 create mode 100644 tests/f_yesthenno/expect
 create mode 100644 tests/f_yesthenno/script


diff --git a/e2fsck/e2fsck.8.in b/e2fsck/e2fsck.8.in
index f5ed758..3367f4f 100644
--- a/e2fsck/e2fsck.8.in
+++ b/e2fsck/e2fsck.8.in
@@ -68,6 +68,19 @@ are not valid if the filesystem is mounted.   If
 asks whether or not you should check a filesystem which is mounted, 
 the only correct answer is ``no''.  Only experts who really know what
 they are doing should consider answering this question in any other way.
+.PP
+If
+.B e2fsck
+is run in interactive mode (meaning that none of
+.BR \-y ,
+.BR \-n ,
+or
+.BR \-p
+are specified), the program will ask the user to fix each problem found in the
+filesystem.  A response of 'y' will fix the error; 'n' will leave the error
+unfixed; and 'a' will fix the problem and all subsequent problems; pressing
+Enter will proceed with the default response, which is printed before the
+question mark.  Pressing Control-C terminates e2fsck immediately.
 .SH OPTIONS
 .TP
 .B \-a 
diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 5cba013..7377e01 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -760,7 +760,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 
 	setvbuf(stdout, NULL, _IONBF, BUFSIZ);
 	setvbuf(stderr, NULL, _IONBF, BUFSIZ);
-	if (isatty(0) && isatty(1)) {
+	if (getenv("E2FSCK_FORCE_INTERACTIVE") || (isatty(0) && isatty(1))) {
 		ctx->interactive = 1;
 	} else {
 		ctx->start_meta[0] = '\001';
diff --git a/e2fsck/util.c b/e2fsck/util.c
index 2de45f8..e2fb982 100644
--- a/e2fsck/util.c
+++ b/e2fsck/util.c
@@ -189,6 +189,10 @@ int ask_yn(e2fsck_t ctx, const char * string, int def)
 	const char	*defstr;
 	const char	*short_yes = _("yY");
 	const char	*short_no = _("nN");
+	const char	*short_yesall = _("aA");
+	const char	*yesall_prompt = _(" ('a' enables 'yes' to all) ");
+	const char	*extra_prompt = "";
+	static int	yes_answers;
 
 #ifdef HAVE_TERMIOS_H
 	struct termios	termios, tmp;
@@ -207,7 +211,16 @@ int ask_yn(e2fsck_t ctx, const char * string, int def)
 		defstr = _(_("<n>"));
 	else
 		defstr = _(" (y/n)");
-	log_out(ctx, "%s%s? ", string, defstr);
+	/*
+	 * If the user presses 'y' more than 8 (but less than 12) times in
+	 * succession without pressing anything else, display a hint about
+	 * yes-to-all mode.
+	 */
+	if (yes_answers > 12)
+		yes_answers = -1;
+	else if (yes_answers > 8)
+		extra_prompt = yesall_prompt;
+	log_out(ctx, "%s%s%s? ", string, extra_prompt, defstr);
 	while (1) {
 		fflush (stdout);
 		if ((c = read_a_char()) == EOF)
@@ -221,20 +234,31 @@ int ask_yn(e2fsck_t ctx, const char * string, int def)
 				longjmp(e2fsck_global_ctx->abort_loc, 1);
 			}
 			log_out(ctx, "%s", _("cancelled!\n"));
+			yes_answers = 0;
 			return 0;
 		}
 		if (strchr(short_yes, (char) c)) {
 			def = 1;
+			if (yes_answers >= 0)
+				yes_answers++;
 			break;
-		}
-		else if (strchr(short_no, (char) c)) {
+		} else if (strchr(short_no, (char) c)) {
 			def = 0;
+			yes_answers = -1;
 			break;
-		}
-		else if ((c == 27 || c == ' ' || c == '\n') && (def != -1))
+		} else if (strchr(short_yesall, (char)c)) {
+			def = 2;
+			yes_answers = -1;
+			ctx->options |= E2F_OPT_YES;
 			break;
+		} else if ((c == 27 || c == ' ' || c == '\n') && (def != -1)) {
+			yes_answers = -1;
+			break;
+		}
 	}
-	if (def)
+	if (def == 2)
+		log_out(ctx, "%s", _("yes to all\n"));
+	else if (def)
 		log_out(ctx, "%s", _("yes\n"));
 	else
 		log_out(ctx, "%s", _("no\n"));
diff --git a/tests/f_no/expect b/tests/f_no/expect
new file mode 100644
index 0000000..e7b619d
--- /dev/null
+++ b/tests/f_no/expect
@@ -0,0 +1,48 @@
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 has an invalid extent
+	(logical block 0, invalid physical block 999999999, len 1)
+Clear<y>? no
+Inode 12 has an invalid extent
+	(logical block 1, invalid physical block 9999999999, len 1)
+Clear<y>? no
+Inode 13 is in use, but has dtime set.  Fix<y>? no
+Inode 13 has an invalid extent
+	(logical block 1, invalid physical block 8888888888888, len 1)
+Clear<y>? no
+Inode 13 has an invalid extent
+	(logical block 0, invalid physical block 888888888888, len 1)
+Clear<y>? no
+Inode 14 is in use, but has dtime set.  Fix<y>? no
+Inode 14 has an invalid extent
+	(logical block 300, invalid physical block 777777777777, len 300)
+Clear<y>? no
+Inode 14 has an invalid extent
+	(logical block 0, invalid physical block 7777777777, len 1)
+Clear<y>? no
+Inode 14, i_blocks is 52574694748113, should be 0.  Fix<y>? no
+Pass 2: Checking directory structure
+Extended attribute block for inode 12 (/a) is invalid (999999).
+Clear<y>? no
+Extended attribute block for inode 13 (/b) is invalid (298954296).
+Clear<y>? no
+Extended attribute block for inode 14 (/c) is invalid (388697201).
+Clear<y>? no
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Inode 12 ref count is 34463, should be 1.  Fix<y>? no
+Inode 13 ref count is 9999, should be 1.  Fix<y>? no
+Inode 14 ref count is 12241, should be 1.  Fix<y>? no
+Pass 5: Checking group summary information
+Block bitmap differences:  -202 -381 -457
+Fix<y>? no
+Free blocks count wrong for group #0 (0, counted=491).
+Fix<y>? no
+Free blocks count wrong (494, counted=491).
+Fix<y>? no
+Free inodes count wrong for group #0 (4294967293, counted=114).
+Fix<y>? no
+
+test_filesys: ********** WARNING: Filesystem still has errors **********
+
+test_filesys: 14/128 files (0.0% non-contiguous), 18/512 blocks
+Exit status is 4
diff --git a/tests/f_no/script b/tests/f_no/script
new file mode 100644
index 0000000..2a67e77
--- /dev/null
+++ b/tests/f_no/script
@@ -0,0 +1,27 @@
+test_description="e2fsck with repeated no"
+FSCK_OPT=-f
+OUT=$test_name.log
+EXP=$test_dir/expect
+
+gunzip < $test_dir/../f_yesall/image.gz > $TMPFILE
+
+rm -rf $OUT
+echo "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn" | E2FSCK_FORCE_INTERACTIVE=y $FSCK $FSCK_OPT -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed $OUT.new >> $OUT
+rm -f $OUT.new
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f tmp_expect
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
diff --git a/tests/f_yes/expect b/tests/f_yes/expect
new file mode 100644
index 0000000..c73e620
--- /dev/null
+++ b/tests/f_yes/expect
@@ -0,0 +1,45 @@
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 has an invalid extent
+	(logical block 0, invalid physical block 999999999, len 1)
+Clear<y>? yes
+Inode 12 has an invalid extent
+	(logical block 1, invalid physical block 9999999999, len 1)
+Clear<y>? yes
+Inode 13 is in use, but has dtime set.  Fix<y>? yes
+Inode 13 has an invalid extent
+	(logical block 1, invalid physical block 8888888888888, len 1)
+Clear<y>? yes
+Inode 13 has an invalid extent
+	(logical block 0, invalid physical block 888888888888, len 1)
+Clear<y>? yes
+Inode 14 is in use, but has dtime set.  Fix<y>? yes
+Inode 14 has an invalid extent
+	(logical block 300, invalid physical block 777777777777, len 300)
+Clear<y>? yes
+Inode 14 has an invalid extent
+	(logical block 0, invalid physical block 7777777777, len 1)
+Clear<y>? yes
+Inode 14, i_blocks is 52574694748113, should be 0.  Fix<y>? yes
+Pass 2: Checking directory structure
+Extended attribute block for inode 12 (/a) is invalid (999999).
+Clear ('a' enables 'yes' to all) <y>? yes
+Extended attribute block for inode 13 (/b) is invalid (298954296).
+Clear ('a' enables 'yes' to all) <y>? yes
+Extended attribute block for inode 14 (/c) is invalid (388697201).
+Clear ('a' enables 'yes' to all) <y>? yes
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Inode 12 ref count is 34463, should be 1.  Fix ('a' enables 'yes' to all) <y>? yes
+Inode 13 ref count is 9999, should be 1.  Fix<y>? yes
+Inode 14 ref count is 12241, should be 1.  Fix<y>? yes
+Pass 5: Checking group summary information
+Block bitmap differences:  -202 -381 -457
+Fix<y>? yes
+Free blocks count wrong for group #0 (0, counted=494).
+Fix<y>? yes
+Free inodes count wrong for group #0 (4294967293, counted=114).
+Fix<y>? yes
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 14/128 files (0.0% non-contiguous), 18/512 blocks
+Exit status is 1
diff --git a/tests/f_yes/script b/tests/f_yes/script
new file mode 100644
index 0000000..4e114c5
--- /dev/null
+++ b/tests/f_yes/script
@@ -0,0 +1,27 @@
+test_description="e2fsck with repeated yes"
+FSCK_OPT=-f
+OUT=$test_name.log
+EXP=$test_dir/expect
+
+gunzip < $test_dir/../f_yesall/image.gz > $TMPFILE
+
+rm -rf $OUT
+echo "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" | E2FSCK_FORCE_INTERACTIVE=y $FSCK $FSCK_OPT -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed $OUT.new >> $OUT
+rm -f $OUT.new
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f tmp_expect
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
diff --git a/tests/f_yesall/expect b/tests/f_yesall/expect
new file mode 100644
index 0000000..f6d3c2b
--- /dev/null
+++ b/tests/f_yesall/expect
@@ -0,0 +1,62 @@
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 has an invalid extent
+	(logical block 0, invalid physical block 999999999, len 1)
+Clear<y>? yes to all
+Inode 12 has an invalid extent
+	(logical block 1, invalid physical block 9999999999, len 1)
+Clear? yes
+
+Inode 13 is in use, but has dtime set.  Fix? yes
+
+Inode 13 has an invalid extent
+	(logical block 1, invalid physical block 8888888888888, len 1)
+Clear? yes
+
+Inode 13 has an invalid extent
+	(logical block 0, invalid physical block 888888888888, len 1)
+Clear? yes
+
+Inode 14 is in use, but has dtime set.  Fix? yes
+
+Inode 14 has an invalid extent
+	(logical block 300, invalid physical block 777777777777, len 300)
+Clear? yes
+
+Inode 14 has an invalid extent
+	(logical block 0, invalid physical block 7777777777, len 1)
+Clear? yes
+
+Inode 14, i_blocks is 52574694748113, should be 0.  Fix? yes
+
+Pass 2: Checking directory structure
+Extended attribute block for inode 12 (/a) is invalid (999999).
+Clear? yes
+
+Extended attribute block for inode 13 (/b) is invalid (298954296).
+Clear? yes
+
+Extended attribute block for inode 14 (/c) is invalid (388697201).
+Clear? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Inode 12 ref count is 34463, should be 1.  Fix? yes
+
+Inode 13 ref count is 9999, should be 1.  Fix? yes
+
+Inode 14 ref count is 12241, should be 1.  Fix? yes
+
+Pass 5: Checking group summary information
+Block bitmap differences:  -202 -381 -457
+Fix? yes
+
+Free blocks count wrong for group #0 (0, counted=494).
+Fix? yes
+
+Free inodes count wrong for group #0 (4294967293, counted=114).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 14/128 files (0.0% non-contiguous), 18/512 blocks
+Exit status is 1
diff --git a/tests/f_yesall/image.gz b/tests/f_yesall/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..0f8b205d56977277a5c511d880cfda9209f6c540
GIT binary patch
literal 2597
zcmb2|=3sD**%89T{Pvc0hKQpC!-Jb=MGeJ^J6&8haF~gCU1HPeX?nn%v&TeAWd2l%
znngE`IC*nBvWV(5-MH!Mzkx%nsdrM4PHs+3Ll?)ZUDdmn8_rYQH)+NF@;}zryNj*A
zzt^ocKHbXjXT94dSD8bXwp7Tq7VNkoCmxt@5VXa6t^BG`XOBs1^%qSr;}>V!T&1UV
zswnWKfA{W}j($s@_H|yo_^vwI>`qm6>hl|q_nj@C`1)l3_4~5#Z<SPkIrDQ}_V2vq
zk~}TnEbg0I_TNv9{M9upM}O(Yi%Z!V4#=En+uwaDB5r-H$D>~_Ov)|x#C$J(tx|Z5
zfq~({`^j6T9r`^@&DzU%_V=2&ZfSoOpm4)az1hdMw`;u9UH0XzVqbOj=l>td+PD2U
z@L}R&?eF~Z);r8^hv!Sb^*Y^p^w+yBZ#*9Rsdjy<n8O6r`#Grc)Sv&Qs`LLd19|^v
z_$>!g3}^l`g5>H!Y$Cyr&67-nUjY>#us<#=K%^#Y%Ic1^9*o*5_&@mG`D@F%&!zp`
z-g2#KrvB|X*M%l8YXkQlT_1hzNp*oK)8@n{CA%c#Gm8FqcUj**ukF9m{awAm`kwU1
z^BdD1&$syEWBgB3^yBva`n-APPP^aEy6W~MdfhzU_^0yq#{azE#Yf3rz5aNDji&$a
zo!8f_KBMtp|L^&{f8yJ(ubQmfa;oe7)A`T$?AZE0=K0e-{}jHI+teDz+W$3I+m$w{
z{@(XD&-W*+2+MW(XKi}X{Gzz&+4|5aM<Xn*>?`t|S1-ahcgK%{U3F5qpOx$F|F5Yp
pSH<SlQL)hw7!85Z5Eu=C(GVC70V;&R57YS!Sqyg$GB7AG001R&FuDK$

literal 0
HcmV?d00001

diff --git a/tests/f_yesall/script b/tests/f_yesall/script
new file mode 100644
index 0000000..c3721ff
--- /dev/null
+++ b/tests/f_yesall/script
@@ -0,0 +1,27 @@
+test_description="e2fsck with yes-to-all"
+FSCK_OPT=-f
+OUT=$test_name.log
+EXP=$test_dir/expect
+
+gunzip < $test_dir/image.gz > $TMPFILE
+
+rm -rf $OUT
+echo "annnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn" | E2FSCK_FORCE_INTERACTIVE=y $FSCK $FSCK_OPT -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed $OUT.new >> $OUT
+rm -f $OUT.new
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f tmp_expect
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
diff --git a/tests/f_yesthenall/expect b/tests/f_yesthenall/expect
new file mode 100644
index 0000000..1fc3bde
--- /dev/null
+++ b/tests/f_yesthenall/expect
@@ -0,0 +1,52 @@
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 has an invalid extent
+	(logical block 0, invalid physical block 999999999, len 1)
+Clear<y>? yes
+Inode 12 has an invalid extent
+	(logical block 1, invalid physical block 9999999999, len 1)
+Clear<y>? yes
+Inode 13 is in use, but has dtime set.  Fix<y>? yes
+Inode 13 has an invalid extent
+	(logical block 1, invalid physical block 8888888888888, len 1)
+Clear<y>? yes
+Inode 13 has an invalid extent
+	(logical block 0, invalid physical block 888888888888, len 1)
+Clear<y>? yes
+Inode 14 is in use, but has dtime set.  Fix<y>? yes
+Inode 14 has an invalid extent
+	(logical block 300, invalid physical block 777777777777, len 300)
+Clear<y>? yes
+Inode 14 has an invalid extent
+	(logical block 0, invalid physical block 7777777777, len 1)
+Clear<y>? yes
+Inode 14, i_blocks is 52574694748113, should be 0.  Fix<y>? yes
+Pass 2: Checking directory structure
+Extended attribute block for inode 12 (/a) is invalid (999999).
+Clear ('a' enables 'yes' to all) <y>? yes
+Extended attribute block for inode 13 (/b) is invalid (298954296).
+Clear ('a' enables 'yes' to all) <y>? yes to all
+Extended attribute block for inode 14 (/c) is invalid (388697201).
+Clear? yes
+
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Inode 12 ref count is 34463, should be 1.  Fix? yes
+
+Inode 13 ref count is 9999, should be 1.  Fix? yes
+
+Inode 14 ref count is 12241, should be 1.  Fix? yes
+
+Pass 5: Checking group summary information
+Block bitmap differences:  -202 -381 -457
+Fix? yes
+
+Free blocks count wrong for group #0 (0, counted=494).
+Fix? yes
+
+Free inodes count wrong for group #0 (4294967293, counted=114).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 14/128 files (0.0% non-contiguous), 18/512 blocks
+Exit status is 1
diff --git a/tests/f_yesthenall/script b/tests/f_yesthenall/script
new file mode 100644
index 0000000..eb11c23
--- /dev/null
+++ b/tests/f_yesthenall/script
@@ -0,0 +1,27 @@
+test_description="e2fsck with yes then yes-to-all"
+FSCK_OPT=-f
+OUT=$test_name.log
+EXP=$test_dir/expect
+
+gunzip < $test_dir/../f_yesall/image.gz > $TMPFILE
+
+rm -rf $OUT
+echo "yyyyyyyyyyannnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn" | E2FSCK_FORCE_INTERACTIVE=y $FSCK $FSCK_OPT -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed $OUT.new >> $OUT
+rm -f $OUT.new
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f tmp_expect
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
diff --git a/tests/f_yesthenno/expect b/tests/f_yesthenno/expect
new file mode 100644
index 0000000..de55f47
--- /dev/null
+++ b/tests/f_yesthenno/expect
@@ -0,0 +1,50 @@
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 has an invalid extent
+	(logical block 0, invalid physical block 999999999, len 1)
+Clear<y>? yes
+Inode 12 has an invalid extent
+	(logical block 1, invalid physical block 9999999999, len 1)
+Clear<y>? yes
+Inode 13 is in use, but has dtime set.  Fix<y>? yes
+Inode 13 has an invalid extent
+	(logical block 1, invalid physical block 8888888888888, len 1)
+Clear<y>? yes
+Inode 13 has an invalid extent
+	(logical block 0, invalid physical block 888888888888, len 1)
+Clear<y>? yes
+Inode 14 is in use, but has dtime set.  Fix<y>? yes
+Inode 14 has an invalid extent
+	(logical block 300, invalid physical block 777777777777, len 300)
+Clear<y>? yes
+Inode 14 has an invalid extent
+	(logical block 0, invalid physical block 7777777777, len 1)
+Clear<y>? yes
+Inode 14, i_blocks is 52574694748113, should be 0.  Fix<y>? yes
+Pass 2: Checking directory structure
+Extended attribute block for inode 12 (/a) is invalid (999999).
+Clear ('a' enables 'yes' to all) <y>? yes
+Extended attribute block for inode 13 (/b) is invalid (298954296).
+Clear ('a' enables 'yes' to all) <y>? no
+Extended attribute block for inode 14 (/c) is invalid (388697201).
+Clear<y>? no
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Inode 12 ref count is 34463, should be 1.  Fix<y>? no
+Inode 13 ref count is 9999, should be 1.  Fix<y>? no
+Inode 14 ref count is 12241, should be 1.  Fix<y>? no
+Pass 5: Checking group summary information
+Block bitmap differences:  -202 -381 -457
+Fix<y>? no
+Free blocks count wrong for group #0 (0, counted=491).
+Fix<y>? no
+Free blocks count wrong (494, counted=491).
+Fix<y>? no
+Free inodes count wrong for group #0 (4294967293, counted=114).
+Fix<y>? no
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+
+test_filesys: ********** WARNING: Filesystem still has errors **********
+
+test_filesys: 14/128 files (0.0% non-contiguous), 18/512 blocks
+Exit status is 4
diff --git a/tests/f_yesthenno/script b/tests/f_yesthenno/script
new file mode 100644
index 0000000..f41b78b
--- /dev/null
+++ b/tests/f_yesthenno/script
@@ -0,0 +1,27 @@
+test_description="e2fsck with yes then no"
+FSCK_OPT=-f
+OUT=$test_name.log
+EXP=$test_dir/expect
+
+gunzip < $test_dir/../f_yesall/image.gz > $TMPFILE
+
+rm -rf $OUT
+echo "yyyyyyyyyynnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn" | E2FSCK_FORCE_INTERACTIVE=y $FSCK $FSCK_OPT -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed $OUT.new >> $OUT
+rm -f $OUT.new
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f tmp_expect
+fi
+
+unset IMAGE FSCK_OPT OUT EXP

[PATCH 21/54] libext2fs: zero blocks via FALLOC_FL_ZERO_RANGE in ext2fs_zero_blocks

[Darrick J. Wong]

Plumb a new call into the IO manager to support translating
ext2fs_zero_blocks calls into the equivalent FALLOC_FL_ZERO_RANGE
fallocate flag primitive when possible.  This patch provides _only_
support for file-based images.

v2: Remove zero-out for block devices until BLKZEROOUT is cleaned up.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/ext2_io.h    |    7 ++++-
 lib/ext2fs/io_manager.c |   11 +++++++
 lib/ext2fs/mkjournal.c  |    5 +++
 lib/ext2fs/test_io.c    |   21 ++++++++++++++
 lib/ext2fs/unix_io.c    |   70 +++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 113 insertions(+), 1 deletion(-)


diff --git a/lib/ext2fs/ext2_io.h b/lib/ext2fs/ext2_io.h
index 4c5a5c5..1faa720 100644
--- a/lib/ext2fs/ext2_io.h
+++ b/lib/ext2fs/ext2_io.h
@@ -93,7 +93,9 @@ struct struct_io_manager {
 	errcode_t (*cache_readahead)(io_channel channel,
 				     unsigned long long block,
 				     unsigned long long count);
-	long	reserved[15];
+	errcode_t (*zeroout)(io_channel channel, unsigned long long block,
+			     unsigned long long count);
+	long	reserved[14];
 };
 
 #define IO_FLAG_RW		0x0001
@@ -125,6 +127,9 @@ extern errcode_t io_channel_write_blk64(io_channel channel,
 extern errcode_t io_channel_discard(io_channel channel,
 				    unsigned long long block,
 				    unsigned long long count);
+extern errcode_t io_channel_zeroout(io_channel channel,
+				    unsigned long long block,
+				    unsigned long long count);
 extern errcode_t io_channel_alloc_buf(io_channel channel,
 				      int count, void *ptr);
 extern errcode_t io_channel_cache_readahead(io_channel io,
diff --git a/lib/ext2fs/io_manager.c b/lib/ext2fs/io_manager.c
index dc5888d..c395d61 100644
--- a/lib/ext2fs/io_manager.c
+++ b/lib/ext2fs/io_manager.c
@@ -112,6 +112,17 @@ errcode_t io_channel_discard(io_channel channel, unsigned long long block,
 	return EXT2_ET_UNIMPLEMENTED;
 }
 
+errcode_t io_channel_zeroout(io_channel channel, unsigned long long block,
+			     unsigned long long count)
+{
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+
+	if (channel->manager->zeroout)
+		return (channel->manager->zeroout)(channel, block, count);
+
+	return EXT2_ET_UNIMPLEMENTED;
+}
+
 errcode_t io_channel_alloc_buf(io_channel io, int count, void *ptr)
 {
 	size_t	size;
diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c
index fcc6741..c42cb98 100644
--- a/lib/ext2fs/mkjournal.c
+++ b/lib/ext2fs/mkjournal.c
@@ -170,6 +170,11 @@ errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
 	if (num <= 0)
 		return 0;
 
+	/* Try a zero out command, if supported */
+	retval = io_channel_zeroout(fs->io, blk, num);
+	if (retval == 0)
+		return 0;
+
 	/* Allocate the zeroizing buffer if necessary */
 	if (num > stride_length && stride_length < MAX_STRIDE_LENGTH) {
 		void *p;
diff --git a/lib/ext2fs/test_io.c b/lib/ext2fs/test_io.c
index b03a939..f7c50d1 100644
--- a/lib/ext2fs/test_io.c
+++ b/lib/ext2fs/test_io.c
@@ -86,6 +86,7 @@ void (*test_io_cb_write_byte)
 #define TEST_FLAG_SET_OPTION		0x20
 #define TEST_FLAG_DISCARD		0x40
 #define TEST_FLAG_READAHEAD		0x80
+#define TEST_FLAG_ZEROOUT		0x100
 
 static void test_dump_block(io_channel channel,
 			    struct test_private_data *data,
@@ -507,6 +508,25 @@ static errcode_t test_cache_readahead(io_channel channel,
 	return retval;
 }
 
+static errcode_t test_zeroout(io_channel channel, unsigned long long block,
+			      unsigned long long count)
+{
+	struct test_private_data *data;
+	errcode_t	retval = 0;
+
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+	data = (struct test_private_data *) channel->private_data;
+	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_TEST_IO_CHANNEL);
+
+	if (data->real)
+		retval = io_channel_zeroout(data->real, block, count);
+	if (data->flags & TEST_FLAG_ZEROOUT)
+		fprintf(data->outfile,
+			"Test_io: zeroout(%llu, %llu) returned %s\n",
+			block, count, retval ? error_message(retval) : "OK");
+	return retval;
+}
+
 static struct struct_io_manager struct_test_manager = {
 	.magic		= EXT2_ET_MAGIC_IO_MANAGER,
 	.name		= "Test I/O Manager",
@@ -523,6 +543,7 @@ static struct struct_io_manager struct_test_manager = {
 	.write_blk64	= test_write_blk64,
 	.discard	= test_discard,
 	.cache_readahead	= test_cache_readahead,
+	.zeroout	= test_zeroout,
 };
 
 io_manager test_io_manager = &struct_test_manager;
diff --git a/lib/ext2fs/unix_io.c b/lib/ext2fs/unix_io.c
index c3a8ea5..3408599 100644
--- a/lib/ext2fs/unix_io.c
+++ b/lib/ext2fs/unix_io.c
@@ -56,6 +56,9 @@
 #if HAVE_LINUX_FALLOC_H
 #include <linux/falloc.h>
 #endif
+#if HAVE_STDINT_H
+#include <stdint.h>
+#endif
 
 #if defined(__linux__) && defined(_IO) && !defined(BLKROGET)
 #define BLKROGET   _IO(0x12, 94) /* Get read-only status (0 = read_write).  */
@@ -987,6 +990,72 @@ unimplemented:
 	return EXT2_ET_UNIMPLEMENTED;
 }
 
+static errcode_t unix_zeroout(io_channel channel, unsigned long long block,
+			      unsigned long long count)
+{
+	struct unix_private_data *data;
+	int		ret;
+
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+	data = (struct unix_private_data *) channel->private_data;
+	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
+
+	if (getenv("UNIX_IO_NOZEROOUT"))
+		goto unimplemented;
+
+	if (channel->flags & CHANNEL_FLAGS_BLOCK_DEVICE) {
+		/* Not implemented until the BLKZEROOUT mess is fixed */
+		goto unimplemented;
+	} else {
+		/* Regular file, try to use truncate/punch/zero. */
+#if defined(HAVE_FALLOCATE) && (defined(FALLOC_FL_ZERO_RANGE) || \
+	(defined(FALLOC_FL_PUNCH_HOLE) && defined(FALLOC_FL_KEEP_SIZE)))
+		struct stat statbuf;
+
+		if (count == 0)
+			return 0;
+		/*
+		 * If we're trying to zero a range past the end of the file,
+		 * extend the file size, then punch (or zero_range) everything.
+		 */
+		ret = fstat(data->dev, &statbuf);
+		if (ret)
+			goto err;
+		if (statbuf.st_size < (block + count) * channel->block_size) {
+			ret = ftruncate(data->dev,
+					(block + count) * channel->block_size);
+			if (ret)
+				goto err;
+		}
+#if defined(FALLOC_FL_PUNCH_HOLE) && defined(FALLOC_FL_KEEP_SIZE)
+		ret = fallocate(data->dev,
+				FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
+				(off_t)(block) * channel->block_size,
+				(off_t)(count) * channel->block_size);
+		if (ret == 0)
+			goto err;
+#endif
+#ifdef FALLOC_FL_ZERO_RANGE
+		ret = fallocate(data->dev,
+				FALLOC_FL_ZERO_RANGE,
+				(off_t)(block) * channel->block_size,
+				(off_t)(count) * channel->block_size);
+#endif
+#else
+		goto unimplemented;
+#endif /* HAVE_FALLOCATE && (ZERO_RANGE || (PUNCH_HOLE && KEEP_SIZE)) */
+	}
+err:
+	if (ret < 0) {
+		if (errno == EOPNOTSUPP)
+			goto unimplemented;
+		return errno;
+	}
+	return 0;
+unimplemented:
+	return EXT2_ET_UNIMPLEMENTED;
+}
+
 static struct struct_io_manager struct_unix_manager = {
 	.magic		= EXT2_ET_MAGIC_IO_MANAGER,
 	.name		= "Unix I/O Manager",
@@ -1003,6 +1072,7 @@ static struct struct_io_manager struct_unix_manager = {
 	.write_blk64	= unix_write_blk64,
 	.discard	= unix_discard,
 	.cache_readahead	= unix_cache_readahead,
+	.zeroout	= unix_zeroout,
 };
 
 io_manager unix_io_manager = &struct_unix_manager;

[PATCH 22/54] libext2fs: ext2fs_new_block2() should call alloc_block hook

[Darrick J. Wong]

If ext2fs_new_block2() is called without a specific block map, we
should call the alloc_block hook before checking fs->block_map.  This
helps us to avoid a bug in e2fsck where we need to allocate a block
but instead of consulting block_found_map, we use the FS bitmaps,
which (prior to pass 5) could be wrong.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c     |    2 +-
 lib/ext2fs/alloc.c |   15 +++++++++++++++
 2 files changed, 16 insertions(+), 1 deletion(-)


diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 9f22826..7955138 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -3795,7 +3795,7 @@ static errcode_t e2fsck_get_alloc_block(ext2_filsys fs, blk64_t goal,
 				return retval;
 		}
 
-		retval = ext2fs_new_block2(fs, goal, 0, &new_block);
+		retval = ext2fs_new_block2(fs, goal, fs->block_map, &new_block);
 		if (retval)
 			return retval;
 	}
diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 62b36fe..9901ca5 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -137,9 +137,23 @@ errcode_t ext2fs_new_block2(ext2_filsys fs, blk64_t goal,
 {
 	errcode_t retval;
 	blk64_t	b = 0;
+	errcode_t (*gab)(ext2_filsys fs, blk64_t goal, blk64_t *ret);
 
 	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
 
+	if (!map && fs->get_alloc_block) {
+		/*
+		 * In case there are clients out there whose get_alloc_block
+		 * handlers call ext2fs_new_block2 with a NULL block map,
+		 * temporarily swap out the function pointer so that we don't
+		 * end up in an infinite loop.
+		 */
+		gab = fs->get_alloc_block;
+		fs->get_alloc_block = NULL;
+		retval = gab(fs, goal, &b);
+		fs->get_alloc_block = gab;
+		goto allocated;
+	}
 	if (!map)
 		map = fs->block_map;
 	if (!map)
@@ -153,6 +167,7 @@ errcode_t ext2fs_new_block2(ext2_filsys fs, blk64_t goal,
 	if ((retval == ENOENT) && (goal != fs->super->s_first_data_block))
 		retval = ext2fs_find_first_zero_block_bitmap2(map,
 			fs->super->s_first_data_block, goal - 1, &b);
+allocated:
 	if (retval == ENOENT)
 		return EXT2_ET_BLOCK_ALLOC_FAIL;
 	if (retval)

[PATCH 23/54] libext2fs: Support readonly filesystem images

[Darrick J. Wong]

Create a new rocompat feature, "readonly", which marks a filesystem
image read-only.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/unix.c           |    7 +++++-
 lib/blkid/probe.h       |    1 +
 lib/e2p/feature.c       |    2 ++
 lib/ext2fs/ext2_fs.h    |    1 +
 lib/ext2fs/ext2fs.h     |    6 ++++-
 misc/tune2fs.c          |    8 ++++--
 resize/main.c           |    8 ++++++
 tests/t_readonly/script |   58 +++++++++++++++++++++++++++++++++++++++++++++++
 8 files changed, 85 insertions(+), 6 deletions(-)
 create mode 100644 tests/t_readonly/script


diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 7377e01..2fa0139 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -1280,7 +1280,8 @@ restart:
 		if ((ctx->mount_flags & EXT2_MF_READONLY) &&
 		    (ctx->options & E2F_OPT_FORCE))
 			flags &= ~EXT2_FLAG_EXCLUSIVE;
-	}
+	} else
+		flags |= EXT2_FLAG_SOFTSUPP_FEATURES;
 
 	ctx->openfs_flags = flags;
 	retval = try_open_fs(ctx, flags, io_ptr, &fs);
@@ -1539,7 +1540,11 @@ failure:
 	features[1] = sb->s_feature_incompat & ~EXT2_LIB_FEATURE_INCOMPAT_SUPP;
 	features[2] = (sb->s_feature_ro_compat &
 		       ~EXT2_LIB_FEATURE_RO_COMPAT_SUPP);
+	if (ctx->options & E2F_OPT_READONLY)
+		features[2] &= ~EXT4_FEATURE_RO_COMPAT_READONLY;
 print_unsupp_features:
+	if (features[2] & EXT4_FEATURE_RO_COMPAT_READONLY)
+		fatal_error(ctx, _("This is a read-only filesystem."));
 	if (features[0] || features[1] || features[2]) {
 		int	i, j;
 		__u32	*mask = features, m;
diff --git a/lib/blkid/probe.h b/lib/blkid/probe.h
index d6809e1..c578b87 100644
--- a/lib/blkid/probe.h
+++ b/lib/blkid/probe.h
@@ -111,6 +111,7 @@ struct ext2_super_block {
 #define EXT4_FEATURE_RO_COMPAT_EXTRA_ISIZE	0x0040
 #define EXT4_FEATURE_RO_COMPAT_QUOTA		0x0100
 #define EXT4_FEATURE_RO_COMPAT_METADATA_CSUM	0x0400
+#define EXT4_FEATURE_RO_COMPAT_READONLY		0x1000
 
 /* for s_feature_incompat */
 #define EXT2_FEATURE_INCOMPAT_FILETYPE		0x0002
diff --git a/lib/e2p/feature.c b/lib/e2p/feature.c
index 30d2db1..02cb572 100644
--- a/lib/e2p/feature.c
+++ b/lib/e2p/feature.c
@@ -68,6 +68,8 @@ static struct feature feature_list[] = {
 			"metadata_csum"},
 	{	E2P_FEATURE_RO_INCOMPAT, EXT4_FEATURE_RO_COMPAT_REPLICA,
 			"replica" },
+	{	E2P_FEATURE_RO_INCOMPAT, EXT4_FEATURE_RO_COMPAT_READONLY,
+			"readonly" },
 
 	{	E2P_FEATURE_INCOMPAT, EXT2_FEATURE_INCOMPAT_COMPRESSION,
 			"compression" },
diff --git a/lib/ext2fs/ext2_fs.h b/lib/ext2fs/ext2_fs.h
index 10cb650..e1f7c67 100644
--- a/lib/ext2fs/ext2_fs.h
+++ b/lib/ext2fs/ext2_fs.h
@@ -754,6 +754,7 @@ struct ext2_super_block {
  */
 #define EXT4_FEATURE_RO_COMPAT_METADATA_CSUM	0x0400
 #define EXT4_FEATURE_RO_COMPAT_REPLICA		0x0800
+#define EXT4_FEATURE_RO_COMPAT_READONLY		0x1000
 
 #define EXT2_FEATURE_INCOMPAT_COMPRESSION	0x0001
 #define EXT2_FEATURE_INCOMPAT_FILETYPE		0x0002
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index f090df1..6546834 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -613,9 +613,13 @@ typedef struct ext2_icount *ext2_icount_t;
 /*
  * These features are only allowed if EXT2_FLAG_SOFTSUPP_FEATURES is passed
  * to ext2fs_openfs()
+ *
+ * Note that the READONLY feature is integral to debugfs and friends being
+ * able to write to the FS.
  */
 #define EXT2_LIB_SOFTSUPP_INCOMPAT	(EXT4_FEATURE_INCOMPAT_ENCRYPT)
-#define EXT2_LIB_SOFTSUPP_RO_COMPAT	(EXT4_FEATURE_RO_COMPAT_REPLICA)
+#define EXT2_LIB_SOFTSUPP_RO_COMPAT	(EXT4_FEATURE_RO_COMPAT_REPLICA | \
+					 EXT4_FEATURE_RO_COMPAT_READONLY)
 
 
 /* Translate a block number to a cluster number */
diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index 258121f..66b2ba3 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -160,7 +160,8 @@ static __u32 ok_features[3] = {
 #ifdef CONFIG_QUOTA
 		EXT4_FEATURE_RO_COMPAT_QUOTA |
 #endif
-		EXT4_FEATURE_RO_COMPAT_METADATA_CSUM
+		EXT4_FEATURE_RO_COMPAT_METADATA_CSUM |
+		EXT4_FEATURE_RO_COMPAT_READONLY
 };
 
 static __u32 clear_ok_features[3] = {
@@ -182,7 +183,8 @@ static __u32 clear_ok_features[3] = {
 #ifdef CONFIG_QUOTA
 		EXT4_FEATURE_RO_COMPAT_QUOTA |
 #endif
-		EXT4_FEATURE_RO_COMPAT_METADATA_CSUM
+		EXT4_FEATURE_RO_COMPAT_METADATA_CSUM |
+		EXT4_FEATURE_RO_COMPAT_READONLY
 };
 
 /**
@@ -2673,7 +2675,7 @@ retry_open:
 	open_flag |= EXT2_FLAG_64BITS | EXT2_FLAG_JOURNAL_DEV_OK;
 
 	/* keep the filesystem struct around to dump MMP data */
-	open_flag |= EXT2_FLAG_NOFREE_ON_ERROR;
+	open_flag |= EXT2_FLAG_NOFREE_ON_ERROR | EXT2_FLAG_SOFTSUPP_FEATURES;
 
 	retval = ext2fs_open2(device_name, io_options, open_flag,
 			      0, 0, io_ptr, &fs);
diff --git a/resize/main.c b/resize/main.c
index c25de61..5c4aa9f 100644
--- a/resize/main.c
+++ b/resize/main.c
@@ -318,9 +318,15 @@ int main (int argc, char ** argv)
 		io_flags = EXT2_FLAG_RW | EXT2_FLAG_EXCLUSIVE;
 
 	io_flags |= EXT2_FLAG_64BITS;
-
+	io_flags |= EXT2_FLAG_NOFREE_ON_ERROR;
 	retval = ext2fs_open2(device_name, io_options, io_flags,
 			      0, 0, io_ptr, &fs);
+	if (retval == EXT2_ET_RO_UNSUPP_FEATURE &&
+	    EXT2_HAS_RO_COMPAT_FEATURE(fs->super,
+					EXT4_FEATURE_RO_COMPAT_READONLY)) {
+		printf("%s", _("This is a read-only filesystem.\n"));
+		exit(1);
+	}
 	if (retval) {
 		com_err(program_name, retval, _("while trying to open %s"),
 			device_name);
diff --git a/tests/t_readonly/script b/tests/t_readonly/script
new file mode 100644
index 0000000..8b65500
--- /dev/null
+++ b/tests/t_readonly/script
@@ -0,0 +1,58 @@
+test_description="readonly image test"
+if test -x $RESIZE2FS_EXE; then
+
+OUT=$test_name.log
+
+echo readonly image > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 -z $TDB_FILE.0 $TMPFILE 256 >> $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE 256 >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+
+echo enable readonly mode >> $OUT
+$TUNE2FS -O readonly $TMPFILE >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs -O readonly $crc2 >> $OUT
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -s $TMPFILE >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc3 >> $OUT
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D $TMPFILE >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc4 >> $OUT
+
+echo disable readonly mode >> $OUT
+$TUNE2FS -O ^readonly $TMPFILE >> $OUT 2>&1
+crc5=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs -O ^readonly $crc5 >> $OUT
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -s $TMPFILE >> $OUT 2>&1
+crc6=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc6 >> $OUT
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D $TMPFILE >> $OUT 2>&1
+crc7=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc7 >> $OUT
+
+echo $crc0 $crc1 $crc2 $crc3 $crc4 $crc5 $crc6 $crc7 >> $OUT
+
+
+if [ $crc0 != $crc1 ] && [ $crc1 != $crc2 ] && [ $crc2 = $crc3 ] && [ $crc2 = $crc4 ] && [ $crc6 != $crc5 ] && [ $crc7 != $crc6 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TMPFILE
+fi

[PATCH 24/54] libext2fs/e2fsck: provide routines to read-ahead metadata

[Darrick J. Wong]

This patch adds to e2fsck the ability to pre-fetch metadata into the
page cache in the hopes of speeding up fsck runs.  There are two new
functions -- the first allows a caller to readahead a list of blocks,
and the second is a helper function that uses that first mechanism to
load group data (bitmaps, inode tables).

These new e2fsck routines require the addition of a dblist API to
allow us to iterate a subset of a dblist.  This will enable
incremental directory block readahead in e2fsck pass 2.

There's also a function to estimate the readahead given a FS.

v2: Add an API to create a dblist with a given number of list elements
pre-allocated.  This enables us to save ~2ms per call to
e2fsck_readahead() (assuming a 2MB RA buffer) by not having to
repeatedly call ext2_resize_mem as we add blocks to the list.

v3: Instead of creating dblists of arbitrary size, change the dblist
iterator to allow iterating a sub-range.  This eliminates a lot of
unnecessary list copying during e2fsck part2.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 configure           |    2 
 configure.in        |    1 
 e2fsck/Makefile.in  |    9 +-
 e2fsck/e2fsck.h     |   18 ++++
 e2fsck/readahead.c  |  252 +++++++++++++++++++++++++++++++++++++++++++++++++++
 e2fsck/util.c       |   51 ++++++++++
 lib/config.h.in     |    3 +
 lib/ext2fs/dblist.c |   21 ++++
 lib/ext2fs/ext2fs.h |   10 ++
 9 files changed, 359 insertions(+), 8 deletions(-)
 create mode 100644 e2fsck/readahead.c


diff --git a/configure b/configure
index f59d232..fdc93c0 100755
--- a/configure
+++ b/configure
@@ -12414,7 +12414,7 @@ fi
 done
 
 fi
-for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
+for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysctl.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default"
diff --git a/configure.in b/configure.in
index 9069234..73cfeb4 100644
--- a/configure.in
+++ b/configure.in
@@ -949,6 +949,7 @@ AC_CHECK_HEADERS(m4_flatten([
 	sys/sockio.h
 	sys/stat.h
 	sys/syscall.h
+	sys/sysctl.h
 	sys/sysmacros.h
 	sys/time.h
 	sys/types.h
diff --git a/e2fsck/Makefile.in b/e2fsck/Makefile.in
index d0e64eb..e40e51b 100644
--- a/e2fsck/Makefile.in
+++ b/e2fsck/Makefile.in
@@ -62,7 +62,7 @@ OBJS= dict.o unix.o e2fsck.o super.o pass1.o pass1b.o pass2.o \
 	pass3.o pass4.o pass5.o journal.o badblocks.o util.o dirinfo.o \
 	dx_dirinfo.o ehandler.o problem.o message.o quota.o recovery.o \
 	region.o revoke.o ea_refcount.o rehash.o profile.o prof_err.o \
-	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o
+	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o readahead.o
 
 PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/super.o profiled/pass1.o profiled/pass1b.o \
@@ -73,7 +73,8 @@ PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/recovery.o profiled/region.o profiled/revoke.o \
 	profiled/ea_refcount.o profiled/rehash.o profiled/profile.o \
 	profiled/prof_err.o profiled/logfile.o \
-	profiled/sigcatcher.o profiled/plausible.o
+	profiled/sigcatcher.o profiled/plausible.o \
+	profiled/sigcatcher.o profiled/readahead.o
 
 SRCS= $(srcdir)/e2fsck.c \
 	$(srcdir)/dict.c \
@@ -97,6 +98,7 @@ SRCS= $(srcdir)/e2fsck.c \
 	$(srcdir)/message.c \
 	$(srcdir)/ea_refcount.c \
 	$(srcdir)/rehash.c \
+	$(srcdir)/readahead.c \
 	$(srcdir)/region.c \
 	$(srcdir)/profile.c \
 	$(srcdir)/sigcatcher.c \
@@ -541,3 +543,6 @@ plausible.o: $(srcdir)/../misc/plausible.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h \
  $(top_srcdir)/lib/ext2fs/ext2_ext_attr.h $(top_srcdir)/lib/ext2fs/bitops.h \
  $(srcdir)/../misc/nls-enable.h $(srcdir)/../misc/plausible.h
+readahead.o: $(srcdir)/readahead.c $(top_builddir)/lib/config.h \
+ $(top_srcdir)/lib/ext2fs/ext2fs.h $(top_srcdir)/lib/ext2fs/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/e2fsck.h prof_err.h
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 615ad75..0252824 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -490,6 +490,23 @@ extern ext2_ino_t e2fsck_get_lost_and_found(e2fsck_t ctx, int fix);
 extern errcode_t e2fsck_adjust_inode_count(e2fsck_t ctx, ext2_ino_t ino,
 					   int adj);
 
+/* readahead.c */
+#define E2FSCK_READA_SUPER	(0x01)
+#define E2FSCK_READA_GDT	(0x02)
+#define E2FSCK_READA_BBITMAP	(0x04)
+#define E2FSCK_READA_IBITMAP	(0x08)
+#define E2FSCK_READA_ITABLE	(0x10)
+#define E2FSCK_READA_ALL_FLAGS	(0x1F)
+errcode_t e2fsck_readahead(ext2_filsys fs, int flags, dgrp_t start,
+			   dgrp_t ngroups);
+#define E2FSCK_RA_DBLIST_IGNORE_BLOCKCNT	(0x01)
+#define E2FSCK_RA_DBLIST_ALL_FLAGS		(0x01)
+errcode_t e2fsck_readahead_dblist(ext2_filsys fs, int flags,
+				  ext2_dblist dblist,
+				  unsigned long long start,
+				  unsigned long long count);
+int e2fsck_can_readahead(ext2_filsys fs);
+unsigned long long e2fsck_guess_readahead(ext2_filsys fs);
 
 /* region.c */
 extern region_t region_create(region_addr_t min, region_addr_t max);
@@ -577,6 +594,7 @@ extern errcode_t e2fsck_allocate_subcluster_bitmap(ext2_filsys fs,
 						   int default_type,
 						   const char *profile_name,
 						   ext2fs_block_bitmap *ret);
+unsigned long long get_memory_size(void);
 
 /* unix.c */
 extern void e2fsck_clear_progbar(e2fsck_t ctx);
diff --git a/e2fsck/readahead.c b/e2fsck/readahead.c
new file mode 100644
index 0000000..8190d1f
--- /dev/null
+++ b/e2fsck/readahead.c
@@ -0,0 +1,252 @@
+/*
+ * readahead.c -- Prefetch filesystem metadata to speed up fsck.
+ *
+ * Copyright (C) 2014 Oracle.
+ *
+ * %Begin-Header%
+ * This file may be redistributed under the terms of the GNU Library
+ * General Public License, version 2.
+ * %End-Header%
+ */
+
+#include "config.h"
+#include <string.h>
+
+#include "e2fsck.h"
+
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
+struct read_dblist {
+	errcode_t err;
+	blk64_t run_start;
+	blk64_t run_len;
+	int flags;
+};
+
+static int readahead_dir_block(ext2_filsys fs, struct ext2_db_entry2 *db,
+			       void *priv_data)
+{
+	struct read_dblist *pr = priv_data;
+	e2_blkcnt_t count = (pr->flags & E2FSCK_RA_DBLIST_IGNORE_BLOCKCNT ?
+			     1 : db->blockcnt);
+
+	if (!pr->run_len || db->blk != pr->run_start + pr->run_len) {
+		if (pr->run_len) {
+			pr->err = io_channel_cache_readahead(fs->io,
+							     pr->run_start,
+							     pr->run_len);
+			dbg_printf("readahead start=%llu len=%llu err=%d\n",
+				   pr->run_start, pr->run_len,
+				   (int)pr->err);
+		}
+		pr->run_start = db->blk;
+		pr->run_len = 0;
+	}
+	pr->run_len += count;
+
+	return pr->err ? DBLIST_ABORT : 0;
+}
+
+errcode_t e2fsck_readahead_dblist(ext2_filsys fs, int flags,
+				  ext2_dblist dblist,
+				  unsigned long long start,
+				  unsigned long long count)
+{
+	errcode_t err;
+	struct read_dblist pr;
+
+	dbg_printf("%s: flags=0x%x\n", __func__, flags);
+	if (flags & ~E2FSCK_RA_DBLIST_ALL_FLAGS)
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	memset(&pr, 0, sizeof(pr));
+	pr.flags = flags;
+	err = ext2fs_dblist_iterate3(dblist, readahead_dir_block, start,
+				     count, &pr);
+	if (pr.err)
+		return pr.err;
+	if (err)
+		return err;
+
+	if (pr.run_len)
+		err = io_channel_cache_readahead(fs->io, pr.run_start,
+						 pr.run_len);
+
+	return err;
+}
+
+static errcode_t e2fsck_readahead_bitmap(ext2_filsys fs,
+					 ext2fs_block_bitmap ra_map)
+{
+	blk64_t start, end, out;
+	errcode_t err;
+
+	start = 1;
+	end = ext2fs_blocks_count(fs->super) - 1;
+
+	err = ext2fs_find_first_set_block_bitmap2(ra_map, start, end, &out);
+	while (err == 0) {
+		start = out;
+		err = ext2fs_find_first_zero_block_bitmap2(ra_map, start, end,
+							   &out);
+		if (err == ENOENT) {
+			out = end;
+			err = 0;
+		} else if (err)
+			break;
+
+		err = io_channel_cache_readahead(fs->io, start, out - start);
+		if (err)
+			break;
+		start = out;
+		err = ext2fs_find_first_set_block_bitmap2(ra_map, start, end,
+							  &out);
+	}
+
+	if (err == ENOENT)
+		err = 0;
+
+	return err;
+}
+
+/* Try not to spew bitmap range errors for readahead */
+static errcode_t mark_bmap_range(ext2_filsys fs, ext2fs_block_bitmap map,
+				 blk64_t blk, unsigned int num)
+{
+	if (blk >= ext2fs_get_generic_bmap_start(map) &&
+	    blk + num <= ext2fs_get_generic_bmap_end(map))
+		ext2fs_mark_block_bitmap_range2(map, blk, num);
+	else
+		return EXT2_ET_INVALID_ARGUMENT;
+	return 0;
+}
+
+static errcode_t mark_bmap(ext2_filsys fs, ext2fs_block_bitmap map, blk64_t blk)
+{
+	if (blk >= ext2fs_get_generic_bmap_start(map) &&
+	    blk <= ext2fs_get_generic_bmap_end(map))
+		ext2fs_mark_block_bitmap2(map, blk);
+	else
+		return EXT2_ET_INVALID_ARGUMENT;
+	return 0;
+}
+
+errcode_t e2fsck_readahead(ext2_filsys fs, int flags, dgrp_t start,
+			   dgrp_t ngroups)
+{
+	blk64_t		super, old_gdt, new_gdt;
+	blk_t		blocks;
+	dgrp_t		i;
+	ext2fs_block_bitmap		ra_map = NULL;
+	dgrp_t		end = start + ngroups;
+	errcode_t	err = 0;
+
+	dbg_printf("%s: flags=0x%x start=%d groups=%d\n", __func__, flags,
+		   start, ngroups);
+	if (flags & ~E2FSCK_READA_ALL_FLAGS)
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (end > fs->group_desc_count)
+		end = fs->group_desc_count;
+
+	if (flags == 0)
+		return 0;
+
+	err = ext2fs_allocate_block_bitmap(fs, "readahead bitmap",
+					   &ra_map);
+	if (err)
+		return err;
+
+	for (i = start; i < end; i++) {
+		err = ext2fs_super_and_bgd_loc2(fs, i, &super, &old_gdt,
+						&new_gdt, &blocks);
+		if (err)
+			break;
+
+		if (flags & E2FSCK_READA_SUPER) {
+			err = mark_bmap(fs, ra_map, super);
+			if (err)
+				break;
+		}
+
+		if (flags & E2FSCK_READA_GDT) {
+			err = mark_bmap_range(fs, ra_map,
+					      old_gdt ? old_gdt : new_gdt,
+					      blocks);
+			if (err)
+				break;
+		}
+
+		if ((flags & E2FSCK_READA_BBITMAP) &&
+		    !ext2fs_bg_flags_test(fs, i, EXT2_BG_BLOCK_UNINIT) &&
+		    ext2fs_bg_free_blocks_count(fs, i) <
+				fs->super->s_blocks_per_group) {
+			super = ext2fs_block_bitmap_loc(fs, i);
+			err = mark_bmap(fs, ra_map, super);
+			if (err)
+				break;
+		}
+
+		if ((flags & E2FSCK_READA_IBITMAP) &&
+		    !ext2fs_bg_flags_test(fs, i, EXT2_BG_INODE_UNINIT) &&
+		    ext2fs_bg_free_inodes_count(fs, i) <
+				fs->super->s_inodes_per_group) {
+			super = ext2fs_inode_bitmap_loc(fs, i);
+			err = mark_bmap(fs, ra_map, super);
+			if (err)
+				break;
+		}
+
+		if ((flags & E2FSCK_READA_ITABLE) &&
+		    ext2fs_bg_free_inodes_count(fs, i) <
+				fs->super->s_inodes_per_group) {
+			super = ext2fs_inode_table_loc(fs, i);
+			blocks = fs->inode_blocks_per_group -
+				 (ext2fs_bg_itable_unused(fs, i) *
+				  EXT2_INODE_SIZE(fs->super) / fs->blocksize);
+			err = mark_bmap_range(fs, ra_map, super, blocks);
+			if (err)
+				break;
+		}
+	}
+
+	if (!err)
+		err = e2fsck_readahead_bitmap(fs, ra_map);
+
+	ext2fs_free_block_bitmap(ra_map);
+	return err;
+}
+
+int e2fsck_can_readahead(ext2_filsys fs)
+{
+	errcode_t err;
+
+	err = io_channel_cache_readahead(fs->io, 0, 1);
+	dbg_printf("%s: supp=%d\n", __func__, err != EXT2_ET_OP_NOT_SUPPORTED);
+	return err != EXT2_ET_OP_NOT_SUPPORTED;
+}
+
+unsigned long long e2fsck_guess_readahead(ext2_filsys fs)
+{
+	unsigned long long guess;
+
+	/*
+	 * The optimal readahead sizes were experimentally determined by
+	 * djwong in August 2014.  Setting the RA size to two block groups'
+	 * worth of inode table blocks seems to yield the largest reductions
+	 * in e2fsck runtime.
+	 */
+	guess = 2 * fs->blocksize * fs->inode_blocks_per_group;
+
+	/* Disable RA if it'd use more 1/50th of RAM. */
+	if (get_memory_size() > (guess * 50))
+		return guess / 1024;
+
+	return 0;
+}
diff --git a/e2fsck/util.c b/e2fsck/util.c
index e2fb982..9e217e6 100644
--- a/e2fsck/util.c
+++ b/e2fsck/util.c
@@ -37,6 +37,10 @@
 #include <errno.h>
 #endif
 
+#ifdef HAVE_SYS_SYSCTL_H
+#include <sys/sysctl.h>
+#endif
+
 #include "e2fsck.h"
 
 extern e2fsck_t e2fsck_global_ctx;   /* Try your very best not to use this! */
@@ -819,3 +823,50 @@ errcode_t e2fsck_allocate_subcluster_bitmap(ext2_filsys fs, const char *descr,
 	fs->default_bitmap_type = save_type;
 	return retval;
 }
+
+/* Return memory size in bytes */
+unsigned long long get_memory_size(void)
+{
+#if defined(_SC_PHYS_PAGES)
+# if defined(_SC_PAGESIZE)
+	return (unsigned long long)sysconf(_SC_PHYS_PAGES) *
+	       (unsigned long long)sysconf(_SC_PAGESIZE);
+# elif defined(_SC_PAGE_SIZE)
+	return (unsigned long long)sysconf(_SC_PHYS_PAGES) *
+	       (unsigned long long)sysconf(_SC_PAGE_SIZE);
+# endif
+#elif defined(CTL_HW)
+# if (defined(HW_MEMSIZE) || defined(HW_PHYSMEM64))
+#  define CTL_HW_INT64
+# elif (defined(HW_PHYSMEM) || defined(HW_REALMEM))
+#  define CTL_HW_UINT
+# endif
+	int mib[2];
+
+	mib[0] = CTL_HW;
+# if defined(HW_MEMSIZE)
+	mib[1] = HW_MEMSIZE;
+# elif defined(HW_PHYSMEM64)
+	mib[1] = HW_PHYSMEM64;
+# elif defined(HW_REALMEM)
+	mib[1] = HW_REALMEM;
+# elif defined(HW_PYSMEM)
+	mib[1] = HW_PHYSMEM;
+# endif
+# if defined(CTL_HW_INT64)
+	unsigned long long size = 0;
+# elif defined(CTL_HW_UINT)
+	unsigned int size = 0;
+# endif
+# if defined(CTL_HW_INT64) || defined(CTL_HW_UINT)
+	size_t len = sizeof(size);
+
+	if (sysctl(mib, 2, &size, &len, NULL, 0) == 0)
+		return (unsigned long long)size;
+# endif
+	return 0;
+#else
+# warning "Don't know how to detect memory on your platform?"
+	return 0;
+#endif
+}
diff --git a/lib/config.h.in b/lib/config.h.in
index 0db010f..cd7ec90 100644
--- a/lib/config.h.in
+++ b/lib/config.h.in
@@ -509,6 +509,9 @@
 /* Define to 1 if you have the <sys/syscall.h> header file. */
 #undef HAVE_SYS_SYSCALL_H
 
+/* Define to 1 if you have the <sys/sysctl.h> header file. */
+#undef HAVE_SYS_SYSCTL_H
+
 /* Define to 1 if you have the <sys/sysmacros.h> header file. */
 #undef HAVE_SYS_SYSMACROS_H
 
diff --git a/lib/ext2fs/dblist.c b/lib/ext2fs/dblist.c
index 942c4f0..bbdb221 100644
--- a/lib/ext2fs/dblist.c
+++ b/lib/ext2fs/dblist.c
@@ -194,20 +194,25 @@ void ext2fs_dblist_sort2(ext2_dblist dblist,
 /*
  * This function iterates over the directory block list
  */
-errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
+errcode_t ext2fs_dblist_iterate3(ext2_dblist dblist,
 				 int (*func)(ext2_filsys fs,
 					     struct ext2_db_entry2 *db_info,
 					     void	*priv_data),
+				 unsigned long long start,
+				 unsigned long long count,
 				 void *priv_data)
 {
-	unsigned long long	i;
+	unsigned long long	i, end;
 	int		ret;
 
 	EXT2_CHECK_MAGIC(dblist, EXT2_ET_MAGIC_DBLIST);
 
+	end = start + count;
 	if (!dblist->sorted)
 		ext2fs_dblist_sort2(dblist, 0);
-	for (i=0; i < dblist->count; i++) {
+	if (end > dblist->count)
+		end = dblist->count;
+	for (i = start; i < end; i++) {
 		ret = (*func)(dblist->fs, &dblist->list[i], priv_data);
 		if (ret & DBLIST_ABORT)
 			return 0;
@@ -215,6 +220,16 @@ errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
 	return 0;
 }
 
+errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
+				 int (*func)(ext2_filsys fs,
+					     struct ext2_db_entry2 *db_info,
+					     void	*priv_data),
+				 void *priv_data)
+{
+	return ext2fs_dblist_iterate3(dblist, func, 0, dblist->count,
+				      priv_data);
+}
+
 static EXT2_QSORT_TYPE dir_block_cmp2(const void *a, const void *b)
 {
 	const struct ext2_db_entry2 *db_a =
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 6546834..d1a7ee9 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1055,11 +1055,17 @@ extern void ext2fs_dblist_sort2(ext2_dblist dblist,
 extern errcode_t ext2fs_dblist_iterate(ext2_dblist dblist,
 	int (*func)(ext2_filsys fs, struct ext2_db_entry *db_info,
 		    void	*priv_data),
-       void *priv_data);
+	void *priv_data);
 extern errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
 	int (*func)(ext2_filsys fs, struct ext2_db_entry2 *db_info,
 		    void	*priv_data),
-       void *priv_data);
+	void *priv_data);
+extern errcode_t ext2fs_dblist_iterate3(ext2_dblist dblist,
+	int (*func)(ext2_filsys fs, struct ext2_db_entry2 *db_info,
+		    void	*priv_data),
+	unsigned long long start,
+	unsigned long long count,
+	void *priv_data);
 extern errcode_t ext2fs_set_dir_block(ext2_dblist dblist, ext2_ino_t ino,
 				      blk_t blk, int blockcnt);
 extern errcode_t ext2fs_set_dir_block2(ext2_dblist dblist, ext2_ino_t ino,

[PATCH 25/54] e2fsck: read-ahead metadata during passes 1, 2, and 4

[Darrick J. Wong]

e2fsck pass1 is modified to use the block group data prefetch function
to try to fetch the inode tables into the pagecache before it is
needed.  We iterate through the blockgroups until we have enough inode
tables that need reading such that we can issue readahead; then we sit
and wait until the last inode table block read of the last group to
start fetching the next bunch.

pass2 is modified to use the dirblock prefetching function to prefetch
the list of directory blocks that are assembled in pass1.  We use the
"iterate a subset of a dblist" and avoid copying the dblist.  Directory
blocks are fetched incrementally as we walk through the directory
block list.  In previous iterations of this patch we would free the
directory blocks after processing, but the performance hit to e2fsck
itself wasn't worth it.  Furthermore, it is anticipated that most
users will then mount the FS and start using the directories, so they
may as well remain in the page cache.

pass4 is modified to prefetch the block and inode bitmaps in
anticipation of pass 5, because pass4 is entirely CPU bound.

In general, these mechanisms can decrease fsck time by 10-40%, if the
host system has sufficient memory and the storage system can provide a
lot of IOPs.  Pretty much any storage system capable of handling
multiple IOs in-flight at any time will see a fairly large performance
boost.  (Single-issue USB mass storage disks seem to suffer badly.)

By default, the readahead buffer size will be set to the size of a block
group's inode table (which is 2MiB for a regular ext4 FS).  The -E
readahead_kb= option can be given to specify the amount of memory to
use for readahead or zero to disable it entirely; or an option can be
given in e2fsck.conf.

v2: Fix an off-by-one error in the pass1 readahead which made the
readahead trigger one inode too late if the block groups are full.

v3: Use the dblist partial iterator function to read ahead parts
of the directory block list in pass 2, instead of making sublists.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/e2fsck.8.in      |    7 +++++
 e2fsck/e2fsck.conf.5.in |   15 +++++++++++
 e2fsck/e2fsck.h         |    3 ++
 e2fsck/pass1.c          |   65 +++++++++++++++++++++++++++++++++++++++++++++++
 e2fsck/pass2.c          |   38 +++++++++++++++++++++++++++
 e2fsck/pass4.c          |    9 +++++++
 e2fsck/unix.c           |   28 ++++++++++++++++++++
 lib/ext2fs/ext2fs.h     |    1 +
 lib/ext2fs/inode.c      |    3 +-
 9 files changed, 167 insertions(+), 2 deletions(-)


diff --git a/e2fsck/e2fsck.8.in b/e2fsck/e2fsck.8.in
index 3367f4f..270727a 100644
--- a/e2fsck/e2fsck.8.in
+++ b/e2fsck/e2fsck.8.in
@@ -220,6 +220,13 @@ option may prevent you from further manual data recovery.
 .BI nodiscard
 Do not attempt to discard free blocks and unused inode blocks. This option is
 exactly the opposite of discard option. This is set as default.
+.TP
+.BI readahead_kb
+Use this many KiB of memory to pre-fetch metadata in the hopes of reducing
+e2fsck runtime.  By default, this is set to the size of two block groups' inode
+tables (typically 4MiB on a regular ext4 filesystem); if this amount is more
+than 1/50th of total physical memory, readahead is disabled.  Set this to zero
+to disable readahead entirely.
 .RE
 .TP
 .B \-f
diff --git a/e2fsck/e2fsck.conf.5.in b/e2fsck/e2fsck.conf.5.in
index 9ebfbbf..ab83180 100644
--- a/e2fsck/e2fsck.conf.5.in
+++ b/e2fsck/e2fsck.conf.5.in
@@ -205,6 +205,21 @@ of that type are squelched.  This can be useful if the console is slow
 (i.e., connected to a serial port) and so a large amount of output could
 end up delaying the boot process for a long time (potentially hours).
 .TP
+.I readahead_mem_pct
+Use this percentage of memory to try to read in metadata blocks ahead of the
+main e2fsck thread.  This should reduce run times, depending on the speed of
+the underlying storage and the amount of free memory.  There is no default, but
+see
+.B readahead_mem_pct
+for more details.
+.TP
+.I readahead_kb
+Use this amount of memory to read in metadata blocks ahead of the main checking
+thread.  Setting this value to zero disables readahead entirely.  By default,
+this is set the size of two block groups' inode tables (typically 4MiB on a
+regular ext4 filesystem); if this amount is more than 1/50th of total physical
+memory, readahead is disabled.
+.TP
 .I report_features
 If this boolean relation is true, e2fsck will print the file system
 features as part of its verbose reporting (i.e., if the
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 0252824..e359515 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -378,6 +378,9 @@ struct e2fsck_struct {
 	 */
 	void *priv_data;
 	ext2fs_block_bitmap block_metadata_map; /* Metadata blocks */
+
+	/* How much are we allowed to readahead? */
+	unsigned long long readahead_kb;
 };
 
 /* Used by the region allocation code */
diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 7955138..3ac3dfe 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -891,6 +891,60 @@ out:
 	return 0;
 }
 
+static void pass1_readahead(e2fsck_t ctx, dgrp_t *group, ext2_ino_t *next_ino)
+{
+	ext2_ino_t inodes_in_group = 0, inodes_per_block, inodes_per_buffer;
+	dgrp_t start = *group, grp;
+	blk64_t blocks_to_read = 0;
+	errcode_t err = EXT2_ET_INVALID_ARGUMENT;
+
+	if (ctx->readahead_kb == 0)
+		goto out;
+
+	/* Keep iterating groups until we have enough to readahead */
+	inodes_per_block = EXT2_INODES_PER_BLOCK(ctx->fs->super);
+	for (grp = start; grp < ctx->fs->group_desc_count; grp++) {
+		if (ext2fs_bg_flags_test(ctx->fs, grp, EXT2_BG_INODE_UNINIT))
+			continue;
+		inodes_in_group = ctx->fs->super->s_inodes_per_group -
+					ext2fs_bg_itable_unused(ctx->fs, grp);
+		blocks_to_read += (inodes_in_group + inodes_per_block - 1) /
+					inodes_per_block;
+		if (blocks_to_read * ctx->fs->blocksize >
+		    ctx->readahead_kb * 1024)
+			break;
+	}
+
+	err = e2fsck_readahead(ctx->fs, E2FSCK_READA_ITABLE, start,
+			       grp - start + 1);
+	if (err == EAGAIN) {
+		ctx->readahead_kb /= 2;
+		err = 0;
+	}
+
+out:
+	if (err) {
+		/* Error; disable itable readahead */
+		*group = ctx->fs->group_desc_count;
+		*next_ino = ctx->fs->super->s_inodes_count;
+	} else {
+		/*
+		 * Don't do more readahead until we've reached the first inode
+		 * of the last inode scan buffer block for the last group.
+		 */
+		*group = grp + 1;
+		inodes_per_buffer = (ctx->inode_buffer_blocks ?
+				     ctx->inode_buffer_blocks :
+				     EXT2_INODE_SCAN_DEFAULT_BUFFER_BLOCKS) *
+				    ctx->fs->blocksize /
+				    EXT2_INODE_SIZE(ctx->fs->super);
+		inodes_in_group--;
+		*next_ino = inodes_in_group -
+			    (inodes_in_group % inodes_per_buffer) + 1 +
+			    (grp * ctx->fs->super->s_inodes_per_group);
+	}
+}
+
 void e2fsck_pass1(e2fsck_t ctx)
 {
 	int	i;
@@ -913,10 +967,19 @@ void e2fsck_pass1(e2fsck_t ctx)
 	int		low_dtime_check = 1;
 	int		inode_size;
 	int		failed_csum = 0;
+	ext2_ino_t	ino_threshold = 0;
+	dgrp_t		ra_group = 0;
 
 	init_resource_track(&rtrack, ctx->fs->io);
 	clear_problem_context(&pctx);
 
+	/* If we can do readahead, figure out how many groups to pull in. */
+	if (!e2fsck_can_readahead(ctx->fs))
+		ctx->readahead_kb = 0;
+	else if (ctx->readahead_kb == ~0ULL)
+		ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs);
+	pass1_readahead(ctx, &ra_group, &ino_threshold);
+
 	if (!(ctx->options & E2F_OPT_PREEN))
 		fix_problem(ctx, PR_1_PASS_HEADER, &pctx);
 
@@ -1096,6 +1159,8 @@ void e2fsck_pass1(e2fsck_t ctx)
 		old_op = ehandler_operation(_("getting next inode from scan"));
 		pctx.errcode = ext2fs_get_next_inode_full(scan, &ino,
 							  inode, inode_size);
+		if (ino > ino_threshold)
+			pass1_readahead(ctx, &ra_group, &ino_threshold);
 		ehandler_operation(old_op);
 		if (ctx->flags & E2F_FLAG_SIGNAL_MASK)
 			return;
diff --git a/e2fsck/pass2.c b/e2fsck/pass2.c
index 78bc24e..768e5cd 100644
--- a/e2fsck/pass2.c
+++ b/e2fsck/pass2.c
@@ -61,6 +61,9 @@
  * Keeps track of how many times an inode is referenced.
  */
 static void deallocate_inode(e2fsck_t ctx, ext2_ino_t ino, char* block_buf);
+static int check_dir_block2(ext2_filsys fs,
+			   struct ext2_db_entry2 *dir_blocks_info,
+			   void *priv_data);
 static int check_dir_block(ext2_filsys fs,
 			   struct ext2_db_entry2 *dir_blocks_info,
 			   void *priv_data);
@@ -77,6 +80,9 @@ struct check_dir_struct {
 	struct problem_context	pctx;
 	int	count, max;
 	e2fsck_t ctx;
+	unsigned long long list_offset;
+	unsigned long long ra_entries;
+	unsigned long long next_ra_off;
 };
 
 void e2fsck_pass2(e2fsck_t ctx)
@@ -96,6 +102,9 @@ void e2fsck_pass2(e2fsck_t ctx)
 	int			i, depth;
 	problem_t		code;
 	int			bad_dir;
+	int (*check_dir_func)(ext2_filsys fs,
+			      struct ext2_db_entry2 *dir_blocks_info,
+			      void *priv_data);
 
 	init_resource_track(&rtrack, ctx->fs->io);
 	clear_problem_context(&cd.pctx);
@@ -139,6 +148,9 @@ void e2fsck_pass2(e2fsck_t ctx)
 	cd.ctx = ctx;
 	cd.count = 1;
 	cd.max = ext2fs_dblist_count2(fs->dblist);
+	cd.list_offset = 0;
+	cd.ra_entries = ctx->readahead_kb * 1024 / ctx->fs->blocksize;
+	cd.next_ra_off = 0;
 
 	if (ctx->progress)
 		(void) (ctx->progress)(ctx, 2, 0, cd.max);
@@ -146,7 +158,8 @@ void e2fsck_pass2(e2fsck_t ctx)
 	if (fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_DIR_INDEX)
 		ext2fs_dblist_sort2(fs->dblist, special_dir_block_cmp);
 
-	cd.pctx.errcode = ext2fs_dblist_iterate2(fs->dblist, check_dir_block,
+	check_dir_func = cd.ra_entries ? check_dir_block2 : check_dir_block;
+	cd.pctx.errcode = ext2fs_dblist_iterate2(fs->dblist, check_dir_func,
 						 &cd);
 	if (ctx->flags & E2F_FLAG_SIGNAL_MASK || ctx->flags & E2F_FLAG_RESTART)
 		return;
@@ -839,6 +852,29 @@ err:
 	return retval;
 }
 
+static int check_dir_block2(ext2_filsys fs,
+			   struct ext2_db_entry2 *db,
+			   void *priv_data)
+{
+	int err;
+	struct check_dir_struct *cd = priv_data;
+
+	if (cd->ra_entries && cd->list_offset >= cd->next_ra_off) {
+		err = e2fsck_readahead_dblist(fs,
+					E2FSCK_RA_DBLIST_IGNORE_BLOCKCNT,
+					fs->dblist,
+					cd->list_offset + cd->ra_entries / 8,
+					cd->ra_entries);
+		if (err)
+			cd->ra_entries = 0;
+		cd->next_ra_off = cd->list_offset + (cd->ra_entries * 7 / 8);
+	}
+
+	err = check_dir_block(fs, db, priv_data);
+	cd->list_offset++;
+	return err;
+}
+
 static int check_dir_block(ext2_filsys fs,
 			   struct ext2_db_entry2 *db,
 			   void *priv_data)
diff --git a/e2fsck/pass4.c b/e2fsck/pass4.c
index 21d93f0..bc9a2c4 100644
--- a/e2fsck/pass4.c
+++ b/e2fsck/pass4.c
@@ -106,6 +106,15 @@ void e2fsck_pass4(e2fsck_t ctx)
 #ifdef MTRACE
 	mtrace_print("Pass 4");
 #endif
+	/*
+	 * Since pass4 is mostly CPU bound, start readahead of bitmaps
+	 * ahead of pass 5 if we haven't already loaded them.
+	 */
+	if (ctx->readahead_kb &&
+	    (fs->block_map == NULL || fs->inode_map == NULL))
+		e2fsck_readahead(fs, E2FSCK_READA_BBITMAP |
+				     E2FSCK_READA_IBITMAP,
+				 0, fs->group_desc_count);
 
 	clear_problem_context(&pctx);
 
diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 2fa0139..c366c78 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -650,6 +650,7 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 	char	*buf, *token, *next, *p, *arg;
 	int	ea_ver;
 	int	extended_usage = 0;
+	unsigned long long reada_kb;
 
 	buf = string_copy(ctx, opts, 0);
 	for (token = buf; token && *token; token = next) {
@@ -678,6 +679,15 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 				continue;
 			}
 			ctx->ext_attr_ver = ea_ver;
+		} else if (strcmp(token, "readahead_kb") == 0) {
+			reada_kb = strtoull(arg, &p, 0);
+			if (*p) {
+				fprintf(stderr, "%s",
+					_("Invalid readahead buffer size.\n"));
+				extended_usage++;
+				continue;
+			}
+			ctx->readahead_kb = reada_kb;
 		} else if (strcmp(token, "fragcheck") == 0) {
 			ctx->options |= E2F_OPT_FRAGCHECK;
 			continue;
@@ -717,6 +727,7 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 		fputs(("\tjournal_only\n"), stderr);
 		fputs(("\tdiscard\n"), stderr);
 		fputs(("\tnodiscard\n"), stderr);
+		fputs(("\treadahead_kb=<buffer size>\n"), stderr);
 		fputc('\n', stderr);
 		exit(1);
 	}
@@ -750,6 +761,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 #ifdef CONFIG_JBD_DEBUG
 	char 		*jbd_debug;
 #endif
+	unsigned long long phys_mem_kb;
 
 	retval = e2fsck_allocate_context(&ctx);
 	if (retval)
@@ -777,6 +789,8 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 	else
 		ctx->program_name = "e2fsck";
 
+	phys_mem_kb = get_memory_size() / 1024;
+	ctx->readahead_kb = ~0ULL;
 	while ((c = getopt (argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDk")) != EOF)
 		switch (c) {
 		case 'C':
@@ -961,6 +975,20 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 	if (c)
 		verbose = 1;
 
+	if (ctx->readahead_kb == ~0ULL) {
+		profile_get_integer(ctx->profile, "options",
+				    "readahead_mem_pct", 0, -1, &c);
+		if (c >= 0 && c <= 100)
+			ctx->readahead_kb = phys_mem_kb * c / 100;
+		profile_get_integer(ctx->profile, "options",
+				    "readahead_kb", 0, -1, &c);
+		if (c >= 0)
+			ctx->readahead_kb = c;
+		if (ctx->readahead_kb != ~0ULL &&
+		    ctx->readahead_kb > phys_mem_kb)
+			ctx->readahead_kb = phys_mem_kb;
+	}
+
 	/* Turn off discard in read-only mode */
 	if ((ctx->options & E2F_OPT_NO) &&
 	    (ctx->options & E2F_OPT_DISCARD))
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index d1a7ee9..ee8f184 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1422,6 +1422,7 @@ extern errcode_t ext2fs_get_next_inode_full(ext2_inode_scan scan,
 					    ext2_ino_t *ino,
 					    struct ext2_inode *inode,
 					    int bufsize);
+#define EXT2_INODE_SCAN_DEFAULT_BUFFER_BLOCKS	8
 extern errcode_t ext2fs_open_inode_scan(ext2_filsys fs, int buffer_blocks,
 				  ext2_inode_scan *ret_scan);
 extern void ext2fs_close_inode_scan(ext2_inode_scan scan);
diff --git a/lib/ext2fs/inode.c b/lib/ext2fs/inode.c
index 8cc0eb8..58389a8 100644
--- a/lib/ext2fs/inode.c
+++ b/lib/ext2fs/inode.c
@@ -175,7 +175,8 @@ errcode_t ext2fs_open_inode_scan(ext2_filsys fs, int buffer_blocks,
 	scan->bytes_left = 0;
 	scan->current_group = 0;
 	scan->groups_left = fs->group_desc_count - 1;
-	scan->inode_buffer_blocks = buffer_blocks ? buffer_blocks : 8;
+	scan->inode_buffer_blocks = buffer_blocks ? buffer_blocks :
+				    EXT2_INODE_SCAN_DEFAULT_BUFFER_BLOCKS;
 	scan->current_block = ext2fs_inode_table_loc(scan->fs,
 						     scan->current_group);
 	scan->inodes_left = EXT2_INODES_PER_GROUP(scan->fs->super);

[PATCH 26/54] e2fsck: track directories to be rehashed with a bitmap

[Darrick J. Wong]

Use a bitmap to track which directories we want to rehash, since
bitmaps will use less memory.  This enables us to clean up the
rehash-all case to use inode_dir_map, and we can free the dirinfo
memory sooner.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/e2fsck.c |    4 ++--
 e2fsck/e2fsck.h |    2 +-
 e2fsck/pass1.c  |    8 ++++++-
 e2fsck/pass2.c  |    4 ++--
 e2fsck/pass3.c  |    4 ++++
 e2fsck/rehash.c |   60 ++++++++++++++++++-------------------------------------
 6 files changed, 35 insertions(+), 47 deletions(-)


diff --git a/e2fsck/e2fsck.c b/e2fsck/e2fsck.c
index fcda7d7..7483072 100644
--- a/e2fsck/e2fsck.c
+++ b/e2fsck/e2fsck.c
@@ -125,8 +125,8 @@ errcode_t e2fsck_reset_context(e2fsck_t ctx)
 		ctx->inode_imagic_map = 0;
 	}
 	if (ctx->dirs_to_hash) {
-		ext2fs_u32_list_free(ctx->dirs_to_hash);
-		ctx->dirs_to_hash = 0;
+		ext2fs_free_inode_bitmap(ctx->dirs_to_hash);
+		ctx->dirs_to_hash = NULL;
 	}
 
 	/*
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index e359515..33dbcad 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -304,7 +304,7 @@ struct e2fsck_struct {
 	/*
 	 * Directories to hash
 	 */
-	ext2_u32_list	dirs_to_hash;
+	ext2fs_inode_bitmap dirs_to_hash;
 
 	/*
 	 * Tuning parameters
diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 3ac3dfe..da4d3cf 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -985,8 +985,12 @@ void e2fsck_pass1(e2fsck_t ctx)
 
 	if ((fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_DIR_INDEX) &&
 	    !(ctx->options & E2F_OPT_NO)) {
-		if (ext2fs_u32_list_create(&ctx->dirs_to_hash, 50))
-			ctx->dirs_to_hash = 0;
+		if (e2fsck_allocate_inode_bitmap(fs,
+						 _("directories to rehash"),
+						 EXT2FS_BMAP64_AUTODIR,
+						 "dirs_to_hash",
+						 &ctx->dirs_to_hash))
+			ctx->dirs_to_hash = NULL;
 	}
 
 #ifdef MTRACE
diff --git a/e2fsck/pass2.c b/e2fsck/pass2.c
index 768e5cd..661260c 100644
--- a/e2fsck/pass2.c
+++ b/e2fsck/pass2.c
@@ -965,7 +965,7 @@ static int check_dir_block(ext2_filsys fs,
 		dot_state = 0;
 
 	if (ctx->dirs_to_hash &&
-	    ext2fs_u32_list_test(ctx->dirs_to_hash, ino))
+	    ext2fs_fast_test_block_bitmap2(ctx->dirs_to_hash, ino))
 		dups_found++;
 
 #if 0
@@ -1684,7 +1684,7 @@ static void clear_htree(e2fsck_t ctx, ext2_ino_t ino)
 	inode.i_flags = inode.i_flags & ~EXT2_INDEX_FL;
 	e2fsck_write_inode(ctx, ino, &inode, "clear_htree");
 	if (ctx->dirs_to_hash)
-		ext2fs_u32_list_add(ctx->dirs_to_hash, ino);
+		ext2fs_mark_inode_bitmap2(ctx->dirs_to_hash, ino);
 }
 
 
diff --git a/e2fsck/pass3.c b/e2fsck/pass3.c
index 1d5255f..c331b98 100644
--- a/e2fsck/pass3.c
+++ b/e2fsck/pass3.c
@@ -119,6 +119,10 @@ void e2fsck_pass3(e2fsck_t ctx)
 	 * If there are any directories that need to be indexed or
 	 * optimized, do it here.
 	 */
+	if (iter)
+		e2fsck_dir_info_iter_end(ctx, iter);
+	iter = NULL;
+	e2fsck_free_dir_info(ctx);
 	e2fsck_rehash_directories(ctx);
 
 abort_exit:
diff --git a/e2fsck/rehash.c b/e2fsck/rehash.c
index e37e871..348923e 100644
--- a/e2fsck/rehash.c
+++ b/e2fsck/rehash.c
@@ -56,9 +56,13 @@
 void e2fsck_rehash_dir_later(e2fsck_t ctx, ext2_ino_t ino)
 {
 	if (!ctx->dirs_to_hash)
-		ext2fs_u32_list_create(&ctx->dirs_to_hash, 50);
+		e2fsck_allocate_inode_bitmap(ctx->fs,
+					     _("directories to rehash"),
+					     EXT2FS_BMAP64_AUTODIR,
+					     "dirs_to_hash",
+					     &ctx->dirs_to_hash);
 	if (ctx->dirs_to_hash)
-		ext2fs_u32_list_add(ctx->dirs_to_hash, ino);
+		ext2fs_mark_inode_bitmap2(ctx->dirs_to_hash, ino);
 }
 
 /* Ask if a dir will be rebuilt during pass 3A. */
@@ -68,7 +72,7 @@ int e2fsck_dir_will_be_rehashed(e2fsck_t ctx, ext2_ino_t ino)
 		return 1;
 	if (!ctx->dirs_to_hash)
 		return 0;
-	return ext2fs_u32_list_test(ctx->dirs_to_hash, ino);
+	return ext2fs_test_inode_bitmap2(ctx->dirs_to_hash, ino);
 }
 
 struct fill_dir_struct {
@@ -911,12 +915,9 @@ void e2fsck_rehash_directories(e2fsck_t ctx)
 #ifdef RESOURCE_TRACK
 	struct resource_track	rtrack;
 #endif
-	struct dir_info		*dir;
-	ext2_u32_iterate 	iter;
-	struct dir_info_iter *	dirinfo_iter = 0;
-	ext2_ino_t		ino;
-	errcode_t		retval;
-	int			cur, max, all_dirs, first = 1;
+	ext2_ino_t		ino = 0;
+	int			all_dirs, first = 1;
+	ext2fs_inode_bitmap	hmap;
 
 	init_resource_track(&rtrack, ctx->fs->io);
 	all_dirs = ctx->options & E2F_OPT_COMPRESS_DIRS;
@@ -928,30 +929,12 @@ void e2fsck_rehash_directories(e2fsck_t ctx)
 
 	clear_problem_context(&pctx);
 
-	cur = 0;
-	if (all_dirs) {
-		dirinfo_iter = e2fsck_dir_info_iter_begin(ctx);
-		max = e2fsck_get_num_dirinfo(ctx);
-	} else {
-		retval = ext2fs_u32_list_iterate_begin(ctx->dirs_to_hash,
-						       &iter);
-		if (retval) {
-			pctx.errcode = retval;
-			fix_problem(ctx, PR_3A_OPTIMIZE_ITER, &pctx);
-			return;
-		}
-		max = ext2fs_u32_list_count(ctx->dirs_to_hash);
-	}
+	hmap = (all_dirs ? ctx->inode_dir_map : ctx->dirs_to_hash);
 	while (1) {
-		if (all_dirs) {
-			if ((dir = e2fsck_dir_info_iter(ctx,
-							dirinfo_iter)) == 0)
-				break;
-			ino = dir->ino;
-		} else {
-			if (!ext2fs_u32_list_iterate(iter, &ino))
-				break;
-		}
+		if (ext2fs_find_first_set_inode_bitmap2(
+				hmap, ino + 1,
+				ctx->fs->super->s_inodes_count, &ino))
+			break;
 
 		pctx.dir = ino;
 		if (first) {
@@ -968,17 +951,14 @@ void e2fsck_rehash_directories(e2fsck_t ctx)
 		}
 		if (ctx->progress && !ctx->progress_fd)
 			e2fsck_simple_progress(ctx, "Rebuilding directory",
-			       100.0 * (float) (++cur) / (float) max, ino);
+					100.0 * (float) ino /
+					(float) ctx->fs->super->s_inodes_count,
+					ino);
 	}
 	end_problem_latch(ctx, PR_LATCH_OPTIMIZE_DIR);
-	if (all_dirs)
-		e2fsck_dir_info_iter_end(ctx, dirinfo_iter);
-	else
-		ext2fs_u32_list_iterate_end(iter);

[PATCH 27/54] e2fsck: rebuild sparse extent trees/convert non-extent ext3 files

[Darrick J. Wong]

Teach e2fsck to (re)construct extent trees.  This enables us to do
either of the following: compress a highly sparse extent tree into
fewer ETB blocks; or convert a ext3-style block mapped file to an
extent file.  The reconstruction is performed during pass 1E or 3A,
as detailed below.

For files that are already extent based, this algorithm will
automatically run (pending user approval) if pass1 determines either
(1) that a whole level of extent tree will fit into a higher level of
the tree; (2) that the size of any level can be reduced by at least
one ETB block; or (3) the extent tree is unnecessarily deep.  It will
not run at all if errors are found and the user declines to fix the
errors.

The option "-E bmap2extent" can be used to force e2fsck to convert all
block map files to extent trees, and to rebuild all extent files'
extent trees.  After conversion, files larger than 12 blocks should be
defragmented to eliminate empty holes where a block lives.

The extent tree constructor is pretty dumb -- it creates a list of
leaf extents (adjacent extents are collapsed), marks all indirect
blocks / ETB blocks free, installs a new extent tree root in the
inode, then loads the leaf extents into the tree.

v2: Account for extent tree block slack that we create when splitting
a block, so that we don't repeatedly annoy the user to rebuild a tree
that we can't optimize further.

v3: For any directory being rebuilt during pass 3A, defer any extent
tree rebuilding until after the rehash.  It's quite possible that the
act of compressing an aged directory will cause it to shrink far
enough to enable us to knock a level off the dir's extent tree.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/Makefile.in                     |   16 +
 e2fsck/e2fsck.8.in                     |    3 
 e2fsck/e2fsck.c                        |    4 
 e2fsck/e2fsck.h                        |   35 ++
 e2fsck/extents.c                       |  532 ++++++++++++++++++++++++++++++++
 e2fsck/pass1.c                         |   62 ++++
 e2fsck/problem.c                       |   48 +++
 e2fsck/problem.h                       |   33 ++
 e2fsck/rehash.c                        |   27 +-
 e2fsck/super.c                         |    7 
 e2fsck/unix.c                          |    4 
 tests/f_extent_bad_node/expect.1       |   11 -
 tests/f_extent_bad_node/expect.2       |    2 
 tests/f_extent_int_bad_magic/expect.1  |    5 
 tests/f_extent_leaf_bad_magic/expect.1 |    5 
 tests/f_extent_oobounds/expect.1       |   11 -
 tests/f_extent_oobounds/expect.2       |    2 
 tests/f_extents/expect.1               |    9 +
 18 files changed, 789 insertions(+), 27 deletions(-)
 create mode 100644 e2fsck/extents.c


diff --git a/e2fsck/Makefile.in b/e2fsck/Makefile.in
index e40e51b..a4413d9 100644
--- a/e2fsck/Makefile.in
+++ b/e2fsck/Makefile.in
@@ -62,7 +62,8 @@ OBJS= dict.o unix.o e2fsck.o super.o pass1.o pass1b.o pass2.o \
 	pass3.o pass4.o pass5.o journal.o badblocks.o util.o dirinfo.o \
 	dx_dirinfo.o ehandler.o problem.o message.o quota.o recovery.o \
 	region.o revoke.o ea_refcount.o rehash.o profile.o prof_err.o \
-	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o readahead.o
+	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o readahead.o \
+	extents.o
 
 PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/super.o profiled/pass1.o profiled/pass1b.o \
@@ -74,7 +75,7 @@ PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/ea_refcount.o profiled/rehash.o profiled/profile.o \
 	profiled/prof_err.o profiled/logfile.o \
 	profiled/sigcatcher.o profiled/plausible.o \
-	profiled/sigcatcher.o profiled/readahead.o
+	profiled/sigcatcher.o profiled/readahead.o profiled/extents.o
 
 SRCS= $(srcdir)/e2fsck.c \
 	$(srcdir)/dict.c \
@@ -106,6 +107,7 @@ SRCS= $(srcdir)/e2fsck.c \
 	prof_err.c \
 	$(srcdir)/quota.c \
 	$(srcdir)/../misc/plausible.c \
+	$(srcdir)/extents.c \
 	$(MTRACE_SRC)
 
 all:: profiled $(PROGS) e2fsck $(MANPAGES) $(FMANPAGES)
@@ -308,6 +310,16 @@ pass1.o: $(srcdir)/pass1.c $(top_builddir)/lib/config.h \
  $(srcdir)/profile.h prof_err.h $(top_srcdir)/lib/quota/quotaio.h \
  $(top_srcdir)/lib/quota/dqblk_v2.h $(top_srcdir)/lib/quota/quotaio_tree.h \
  $(top_srcdir)/lib/../e2fsck/dict.h $(srcdir)/problem.h
+extents.o: $(srcdir)/extents.c $(top_builddir)/lib/config.h \
+ $(top_builddir)/lib/dirpaths.h $(top_srcdir)/lib/et/com_err.h \
+ $(srcdir)/e2fsck.h $(top_srcdir)/lib/ext2fs/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(top_srcdir)/lib/ext2fs/ext2fs.h \
+ $(top_srcdir)/lib/ext2fs/ext3_extents.h $(top_srcdir)/lib/ext2fs/ext2_io.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h \
+ $(top_srcdir)/lib/ext2fs/ext2_ext_attr.h $(top_srcdir)/lib/ext2fs/bitops.h \
+ $(srcdir)/profile.h prof_err.h $(top_srcdir)/lib/quota/quotaio.h \
+ $(top_srcdir)/lib/quota/dqblk_v2.h $(top_srcdir)/lib/quota/quotaio_tree.h \
+ $(top_srcdir)/lib/../e2fsck/dict.h $(srcdir)/problem.h $(srcdir)/dict.h
 pass1b.o: $(srcdir)/pass1b.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/dirpaths.h $(top_srcdir)/lib/et/com_err.h \
  $(srcdir)/e2fsck.h $(top_srcdir)/lib/ext2fs/ext2_fs.h \
diff --git a/e2fsck/e2fsck.8.in b/e2fsck/e2fsck.8.in
index 270727a..6e6a2cd 100644
--- a/e2fsck/e2fsck.8.in
+++ b/e2fsck/e2fsck.8.in
@@ -227,6 +227,9 @@ e2fsck runtime.  By default, this is set to the size of two block groups' inode
 tables (typically 4MiB on a regular ext4 filesystem); if this amount is more
 than 1/50th of total physical memory, readahead is disabled.  Set this to zero
 to disable readahead entirely.
+.TP
+.BI bmap2extent
+Convert block-mapped files to extent-mapped files.
 .RE
 .TP
 .B \-f
diff --git a/e2fsck/e2fsck.c b/e2fsck/e2fsck.c
index 7483072..8a7b041 100644
--- a/e2fsck/e2fsck.c
+++ b/e2fsck/e2fsck.c
@@ -204,8 +204,8 @@ void e2fsck_free_context(e2fsck_t ctx)
 typedef void (*pass_t)(e2fsck_t ctx);
 
 static pass_t e2fsck_passes[] = {
-	e2fsck_pass1, e2fsck_pass2, e2fsck_pass3, e2fsck_pass4,
-	e2fsck_pass5, 0 };
+	e2fsck_pass1, e2fsck_pass1e, e2fsck_pass2, e2fsck_pass3,
+	e2fsck_pass4, e2fsck_pass5, 0 };
 
 #define E2F_FLAG_RUN_RETURN	(E2F_FLAG_SIGNAL_MASK|E2F_FLAG_RESTART)
 
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 33dbcad..15d968e 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -167,6 +167,7 @@ struct resource_track {
 #define E2F_OPT_FRAGCHECK	0x0800
 #define E2F_OPT_JOURNAL_ONLY	0x1000 /* only replay the journal */
 #define E2F_OPT_DISCARD		0x2000
+#define E2F_OPT_CONVERT_BMAP	0x4000 /* convert blockmap to extent */
 
 /*
  * E2fsck flags
@@ -190,6 +191,7 @@ struct resource_track {
 #define E2F_FLAG_EXITING	0x1000 /* E2fsck exiting due to errors */
 #define E2F_FLAG_TIME_INSANE	0x2000 /* Time is insane */
 #define E2F_FLAG_PROBLEMS_FIXED	0x4000 /* At least one problem was fixed */
+#define E2F_FLAG_ALLOC_OK	0x8000 /* Can we allocate blocks? */
 
 #define E2F_RESET_FLAGS (E2F_FLAG_TIME_INSANE | E2F_FLAG_PROBLEMS_FIXED)
 
@@ -381,6 +383,23 @@ struct e2fsck_struct {
 
 	/* How much are we allowed to readahead? */
 	unsigned long long readahead_kb;
+
+	/*
+	 * Inodes to rebuild extent trees
+	 */
+	ext2fs_inode_bitmap inodes_to_rebuild;
+};
+
+/* Data structures to evaluate whether an extent tree needs rebuilding. */
+struct extent_tree_level {
+	unsigned int	num_extents;
+	unsigned int	max_extents;
+};
+
+struct extent_tree_info {
+	int force_rebuild:1;
+	ext2_ino_t ino;
+	struct extent_tree_level	ext_info[MAX_EXTENT_DEPTH_COUNT];
 };
 
 /* Used by the region allocation code */
@@ -456,6 +475,19 @@ extern blk64_t ea_refcount_intr_next(ext2_refcount_t refcount, int *ret);
 extern const char *ehandler_operation(const char *op);
 extern void ehandler_init(io_channel channel);
 
+/* extents.c */
+struct problem_context;
+errcode_t e2fsck_rebuild_extents_later(e2fsck_t ctx, ext2_ino_t ino);
+int e2fsck_ino_will_be_rebuilt(e2fsck_t ctx, ext2_ino_t ino);
+void e2fsck_pass1e(e2fsck_t ctx);
+errcode_t e2fsck_check_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino,
+				       struct ext2_inode *inode,
+				       struct problem_context *pctx);
+errcode_t e2fsck_should_rebuild_extents(e2fsck_t ctx,
+					struct problem_context *pctx,
+					struct extent_tree_info *eti,
+					struct ext2_extent_info *info);
+
 /* journal.c */
 extern errcode_t e2fsck_check_ext3_journal(e2fsck_t ctx);
 extern errcode_t e2fsck_run_ext3_journal(e2fsck_t ctx);
@@ -519,7 +551,8 @@ extern int region_allocate(region_t region, region_addr_t start, int n);
 /* rehash.c */
 void e2fsck_rehash_dir_later(e2fsck_t ctx, ext2_ino_t ino);
 int e2fsck_dir_will_be_rehashed(e2fsck_t ctx, ext2_ino_t ino);
-errcode_t e2fsck_rehash_dir(e2fsck_t ctx, ext2_ino_t ino);
+errcode_t e2fsck_rehash_dir(e2fsck_t ctx, ext2_ino_t ino,
+			    struct problem_context *pctx);
 void e2fsck_rehash_directories(e2fsck_t ctx);
 
 /* sigcatcher.c */
diff --git a/e2fsck/extents.c b/e2fsck/extents.c
new file mode 100644
index 0000000..9c2ae42
--- /dev/null
+++ b/e2fsck/extents.c
@@ -0,0 +1,532 @@
+/*
+ * extents.c --- rebuild extent tree
+ *
+ * Copyright (C) 2014 Oracle.
+ *
+ * %Begin-Header%
+ * This file may be redistributed under the terms of the GNU Public
+ * License, version 2.
+ * %End-Header%
+ */
+
+#include "config.h"
+#include <string.h>
+#include <ctype.h>
+#include <errno.h>
+#include "e2fsck.h"
+#include "problem.h"
+
+#undef DEBUG
+#undef DEBUG_SUMMARY
+#undef DEBUG_FREE
+
+#define NUM_EXTENTS	341	/* about one ETB' worth of extents */
+
+static errcode_t e2fsck_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino);
+
+/* Schedule an inode to have its extent tree rebuilt during pass 1E. */
+errcode_t e2fsck_rebuild_extents_later(e2fsck_t ctx, ext2_ino_t ino)
+{
+	if (!EXT2_HAS_INCOMPAT_FEATURE(ctx->fs->super,
+				       EXT3_FEATURE_INCOMPAT_EXTENTS) ||
+	    (ctx->options & E2F_OPT_NO) ||
+	    (ino != EXT2_ROOT_INO && ino < ctx->fs->super->s_first_ino))
+		return 0;
+
+	if (ctx->flags & E2F_FLAG_ALLOC_OK)
+		return e2fsck_rebuild_extents(ctx, ino);
+
+	if (!ctx->inodes_to_rebuild)
+		e2fsck_allocate_inode_bitmap(ctx->fs,
+					     _("extent rebuild inode map"),
+					     EXT2FS_BMAP64_RBTREE,
+					     "inodes_to_rebuild",
+					     &ctx->inodes_to_rebuild);
+	if (ctx->inodes_to_rebuild)
+		ext2fs_mark_inode_bitmap2(ctx->inodes_to_rebuild, ino);
+	return 0;
+}
+
+/* Ask if an inode will have its extents rebuilt during pass 1E. */
+int e2fsck_ino_will_be_rebuilt(e2fsck_t ctx, ext2_ino_t ino)
+{
+	if (!ctx->inodes_to_rebuild)
+		return 0;
+	return ext2fs_test_inode_bitmap2(ctx->inodes_to_rebuild, ino);
+}
+
+struct extent_list {
+	blk64_t blocks_freed;
+	struct ext2fs_extent *extents;
+	unsigned int count;
+	unsigned int size;
+	unsigned int ext_read;
+	errcode_t retval;
+	ext2_ino_t ino;
+};
+
+static errcode_t load_extents(e2fsck_t ctx, struct extent_list *list)
+{
+	ext2_filsys		fs = ctx->fs;
+	ext2_extent_handle_t	handle;
+	struct ext2fs_extent	extent;
+	errcode_t		retval;
+
+	retval = ext2fs_extent_open(fs, list->ino, &handle);
+	if (retval)
+		return retval;
+
+	retval = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
+	if (retval)
+		goto out;
+
+	do {
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)
+			goto next;
+
+		/* Internal node; free it and we'll re-allocate it later */
+		if (!(extent.e_flags & EXT2_EXTENT_FLAGS_LEAF)) {
+#if defined(DEBUG) || defined(DEBUG_FREE)
+			printf("ino=%d free=%llu bf=%llu\n", list->ino,
+					extent.e_pblk, list->blocks_freed + 1);
+#endif
+			list->blocks_freed++;
+			ext2fs_block_alloc_stats2(fs, extent.e_pblk, -1);
+			goto next;
+		}
+
+		list->ext_read++;
+		/* Can we attach it to the previous extent? */
+		if (list->count) {
+			struct ext2fs_extent *last = list->extents +
+						     list->count - 1;
+			blk64_t end = last->e_len + extent.e_len;
+
+			if (last->e_pblk + last->e_len == extent.e_pblk &&
+			    last->e_lblk + last->e_len == extent.e_lblk &&
+			    (last->e_flags & EXT2_EXTENT_FLAGS_UNINIT) ==
+			    (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+			    end < (1ULL << 32)) {
+				last->e_len += extent.e_len;
+#ifdef DEBUG
+				printf("R: ino=%d len=%u\n", list->ino,
+						last->e_len);
+#endif
+				goto next;
+			}
+		}
+
+		/* Do we need to expand? */
+		if (list->count == list->size) {
+			unsigned int new_size = (list->size + NUM_EXTENTS) *
+						sizeof(struct ext2fs_extent);
+			retval = ext2fs_resize_mem(0, new_size, &list->extents);
+			if (retval)
+				goto out;
+			list->size += NUM_EXTENTS;
+		}
+
+		/* Add a new extent */
+		memcpy(list->extents + list->count, &extent, sizeof(extent));
+#ifdef DEBUG
+		printf("R: ino=%d pblk=%llu lblk=%llu len=%u\n", list->ino,
+				extent.e_pblk, extent.e_lblk, extent.e_len);
+#endif
+		list->count++;
+next:
+		retval = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent);
+	} while (retval == 0);
+
+out:
+	/* Ok if we run off the end */
+	if (retval == EXT2_ET_EXTENT_NO_NEXT)
+		retval = 0;
+	ext2fs_extent_free(handle);
+	return retval;
+}
+
+static int find_blocks(ext2_filsys fs, blk64_t *blocknr, e2_blkcnt_t blockcnt,
+		       blk64_t ref_blk, int ref_offset, void *priv_data)
+{
+	struct extent_list *list = priv_data;
+
+	/* Internal node? */
+	if (blockcnt < 0) {
+#if defined(DEBUG) || defined(DEBUG_FREE)
+		printf("ino=%d free=%llu bf=%llu\n", list->ino, *blocknr,
+				list->blocks_freed + 1);
+#endif
+		list->blocks_freed++;
+		ext2fs_block_alloc_stats2(fs, *blocknr, -1);
+		return 0;
+	}
+
+	/* Can we attach it to the previous extent? */
+	if (list->count) {
+		struct ext2fs_extent *last = list->extents +
+					     list->count - 1;
+		blk64_t end = last->e_len + 1;
+
+		if (last->e_pblk + last->e_len == *blocknr &&
+		    end < (1ULL << 32)) {
+			last->e_len++;
+#ifdef DEBUG
+			printf("R: ino=%d len=%u\n", list->ino, last->e_len);
+#endif
+			return 0;
+		}
+	}
+
+	/* Do we need to expand? */
+	if (list->count == list->size) {
+		unsigned int new_size = (list->size + NUM_EXTENTS) *
+					sizeof(struct ext2fs_extent);
+		list->retval = ext2fs_resize_mem(0, new_size, &list->extents);
+		if (list->retval)
+			return BLOCK_ABORT;
+		list->size += NUM_EXTENTS;
+	}
+
+	/* Add a new extent */
+	list->extents[list->count].e_pblk = *blocknr;
+	list->extents[list->count].e_lblk = blockcnt;
+	list->extents[list->count].e_len = 1;
+	list->extents[list->count].e_flags = 0;
+#ifdef DEBUG
+	printf("R: ino=%d pblk=%llu lblk=%llu len=%u\n", list->ino, *blocknr,
+			blockcnt, 1);
+#endif
+	list->count++;
+
+	return 0;
+}
+
+static errcode_t rebuild_extent_tree(e2fsck_t ctx, struct extent_list *list,
+				     ext2_ino_t ino)
+{
+	struct ext2_inode	inode;
+	errcode_t		retval;
+	ext2_extent_handle_t	handle;
+	unsigned int		i, ext_written;
+	struct ext2fs_extent	*ex, extent;
+
+	list->count = 0;
+	list->blocks_freed = 0;
+	list->ino = ino;
+	list->ext_read = 0;
+	e2fsck_read_inode(ctx, ino, &inode, "rebuild_extents");
+
+	/* Skip deleted inodes and inline data files */
+	if (inode.i_links_count == 0 ||
+	    inode.i_flags & EXT4_INLINE_DATA_FL)
+		return 0;
+
+	/* Collect lblk->pblk mappings */
+	if (inode.i_flags & EXT4_EXTENTS_FL) {
+		retval = load_extents(ctx, list);
+		goto extents_loaded;
+	}
+
+	retval = ext2fs_block_iterate3(ctx->fs, ino, BLOCK_FLAG_READ_ONLY, 0,
+				       find_blocks, list);
+	if (retval)
+		goto err;
+	if (list->retval) {
+		retval = list->retval;
+		goto err;
+	}
+
+extents_loaded:
+	/* Reset extent tree */
+	inode.i_flags &= ~EXT4_EXTENTS_FL;
+	memset(inode.i_block, 0, sizeof(inode.i_block));
+
+	/* Make a note of freed blocks */
+	retval = ext2fs_iblk_sub_blocks(ctx->fs, &inode, list->blocks_freed);
+	if (retval)
+		goto err;
+
+	/* Now stuff extents into the file */
+	retval = ext2fs_extent_open2(ctx->fs, ino, &inode, &handle);
+	if (retval)
+		goto err;
+
+	ext_written = 0;
+	for (i = 0, ex = list->extents; i < list->count; i++, ex++) {
+		memcpy(&extent, ex, sizeof(struct ext2fs_extent));
+		extent.e_flags &= EXT2_EXTENT_FLAGS_UNINIT;
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) {
+			if (extent.e_len > EXT_UNINIT_MAX_LEN) {
+				extent.e_len = EXT_UNINIT_MAX_LEN;
+				ex->e_pblk += EXT_UNINIT_MAX_LEN;
+				ex->e_lblk += EXT_UNINIT_MAX_LEN;
+				ex->e_len -= EXT_UNINIT_MAX_LEN;
+				ex--;
+				i--;
+			}
+		} else {
+			if (extent.e_len > EXT_INIT_MAX_LEN) {
+				extent.e_len = EXT_INIT_MAX_LEN;
+				ex->e_pblk += EXT_INIT_MAX_LEN;
+				ex->e_lblk += EXT_INIT_MAX_LEN;
+				ex->e_len -= EXT_INIT_MAX_LEN;
+				ex--;
+				i--;
+			}
+		}
+
+#ifdef DEBUG
+		printf("W: ino=%d pblk=%llu lblk=%llu len=%u\n", ino,
+				extent.e_pblk, extent.e_lblk, extent.e_len);
+#endif
+		retval = ext2fs_extent_insert(handle, EXT2_EXTENT_INSERT_AFTER,
+					      &extent);
+		if (retval)
+			goto err2;
+		retval = ext2fs_extent_fix_parents(handle);
+		if (retval)
+			goto err2;
+		ext_written++;
+	}
+
+#if defined(DEBUG) || defined(DEBUG_SUMMARY)
+	printf("rebuild: ino=%d extents=%d->%d\n", ino, list->ext_read,
+	       ext_written);
+#endif
+	e2fsck_write_inode(ctx, ino, &inode, "rebuild_extents");
+
+err2:
+	ext2fs_extent_free(handle);
+err:
+	return retval;
+}
+
+/* Rebuild the extents immediately */
+static errcode_t e2fsck_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino)
+{
+	struct extent_list	list;
+	errcode_t err;
+
+	if (!EXT2_HAS_INCOMPAT_FEATURE(ctx->fs->super,
+				       EXT3_FEATURE_INCOMPAT_EXTENTS) ||
+	    (ctx->options & E2F_OPT_NO) ||
+	    (ino != EXT2_ROOT_INO && ino < ctx->fs->super->s_first_ino))
+		return 0;
+
+	e2fsck_read_bitmaps(ctx);
+	memset(&list, 0, sizeof(list));
+	err = ext2fs_get_mem(sizeof(struct ext2fs_extent) * NUM_EXTENTS,
+				&list.extents);
+	if (err)
+		return err;
+	list.size = NUM_EXTENTS;
+	err = rebuild_extent_tree(ctx, &list, ino);
+	ext2fs_free_mem(&list.extents);
+
+	return err;
+}
+
+static void rebuild_extents(e2fsck_t ctx, const char *pass_name, int pr_header)
+{
+	struct problem_context	pctx;
+#ifdef RESOURCE_TRACK
+	struct resource_track	rtrack;
+#endif
+	struct extent_list	list;
+	int			first = 1;
+	ext2_ino_t		ino = 0;
+	errcode_t		retval;
+
+	if (!EXT2_HAS_INCOMPAT_FEATURE(ctx->fs->super,
+				       EXT3_FEATURE_INCOMPAT_EXTENTS) ||
+	    !ext2fs_test_valid(ctx->fs) ||
+	    ctx->invalid_bitmaps) {
+		if (ctx->inodes_to_rebuild)
+			ext2fs_free_inode_bitmap(ctx->inodes_to_rebuild);
+		ctx->inodes_to_rebuild = NULL;
+	}
+
+	if (ctx->inodes_to_rebuild == NULL)
+		return;
+
+	init_resource_track(&rtrack, ctx->fs->io);
+	clear_problem_context(&pctx);
+	e2fsck_read_bitmaps(ctx);
+
+	memset(&list, 0, sizeof(list));
+	retval = ext2fs_get_mem(sizeof(struct ext2fs_extent) * NUM_EXTENTS,
+				&list.extents);
+	list.size = NUM_EXTENTS;
+	while (1) {
+		retval = ext2fs_find_first_set_inode_bitmap2(
+				ctx->inodes_to_rebuild, ino + 1,
+				ctx->fs->super->s_inodes_count, &ino);
+		if (retval)
+			break;
+		pctx.ino = ino;
+		if (first) {
+			fix_problem(ctx, pr_header, &pctx);
+			first = 0;
+		}
+		pctx.errcode = rebuild_extent_tree(ctx, &list, ino);
+		if (pctx.errcode) {
+			end_problem_latch(ctx, PR_LATCH_OPTIMIZE_EXT);
+			fix_problem(ctx, PR_1E_OPTIMIZE_EXT_ERR, &pctx);
+		}
+		if (ctx->progress && !ctx->progress_fd)
+			e2fsck_simple_progress(ctx, "Rebuilding extents",
+					100.0 * (float) ino /
+					(float) ctx->fs->super->s_inodes_count,
+					ino);
+	}
+	end_problem_latch(ctx, PR_LATCH_OPTIMIZE_EXT);
+
+	ext2fs_free_inode_bitmap(ctx->inodes_to_rebuild);
+	ctx->inodes_to_rebuild = NULL;
+	ext2fs_free_mem(&list.extents);
+
+	print_resource_track(ctx, pass_name, &rtrack, ctx->fs->io);
+}
+
+/* Scan a file to see if we should rebuild its extent tree */
+errcode_t e2fsck_check_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino,
+				  struct ext2_inode *inode,
+				  struct problem_context *pctx)
+{
+	struct extent_tree_info	eti;
+	struct ext2_extent_info	info, top_info;
+	struct ext2fs_extent	extent;
+	ext2_extent_handle_t	ehandle;
+	ext2_filsys		fs = ctx->fs;
+	errcode_t		retval;
+
+	/* block map file and we want extent conversion */
+	if (!(inode->i_flags & EXT4_EXTENTS_FL) &&
+	    !(inode->i_flags & EXT4_INLINE_DATA_FL) &&
+	    (ctx->options & E2F_OPT_CONVERT_BMAP)) {
+		return e2fsck_rebuild_extents_later(ctx, ino);
+	}
+
+	if (!(inode->i_flags & EXT4_EXTENTS_FL))
+		return 0;
+	memset(&eti, 0, sizeof(eti));
+	eti.ino = ino;
+
+	/* Otherwise, go scan the extent tree... */
+	retval = ext2fs_extent_open2(fs, ino, inode, &ehandle);
+	if (retval)
+		return 0;
+
+	retval = ext2fs_extent_get_info(ehandle, &top_info);
+	if (retval)
+		goto out;
+
+	/* Check maximum extent depth */
+	pctx->ino = ino;
+	pctx->blk = top_info.max_depth;
+	pctx->blk2 = ext2fs_max_extent_depth(ehandle);
+	if (pctx->blk2 < pctx->blk &&
+	    fix_problem(ctx, PR_1_EXTENT_BAD_MAX_DEPTH, pctx))
+		eti.force_rebuild = 1;
+
+	/* Can we collect extent tree level stats? */
+	pctx->blk = MAX_EXTENT_DEPTH_COUNT;
+	if (pctx->blk2 > pctx->blk)
+		fix_problem(ctx, PR_1E_MAX_EXTENT_TREE_DEPTH, pctx);
+
+	retval = ext2fs_extent_get(ehandle, EXT2_EXTENT_ROOT, &extent);
+	if (retval)
+		goto out;
+
+	do {
+		retval = ext2fs_extent_get_info(ehandle, &info);
+		if (retval)
+			break;
+
+		/*
+		 * If this is the first extent in an extent block that we
+		 * haven't visited, collect stats on the block.
+		 */
+		if (info.curr_entry == 1 &&
+		    !(extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT) &&
+		    !eti.force_rebuild) {
+			struct extent_tree_level *etl;
+
+			etl = eti.ext_info + info.curr_level;
+			etl->num_extents += info.num_entries;
+			etl->max_extents += info.max_entries;
+			/*
+			 * Implementation wart: Splitting extent blocks when
+			 * appending will leave the old block with one free
+			 * entry.  Therefore unless the node is totally full,
+			 * pretend that a non-root extent block can hold one
+			 * fewer entry than it actually does, so that we don't
+			 * repeatedly rebuild the extent tree.
+			 */
+			if (info.curr_level &&
+			    info.num_entries < info.max_entries)
+				etl->max_extents--;
+		}
+
+		/* Skip to the end of a block of leaf nodes */
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) {
+			retval = ext2fs_extent_get(ehandle,
+						    EXT2_EXTENT_LAST_SIB,
+						    &extent);
+			if (retval)
+				break;
+		}
+
+		retval = ext2fs_extent_get(ehandle, EXT2_EXTENT_NEXT, &extent);
+	} while (retval == 0);
+out:
+	ext2fs_extent_free(ehandle);
+	return e2fsck_should_rebuild_extents(ctx, pctx, &eti, &top_info);
+}
+
+/* Having scanned a file's extent tree, decide if we should rebuild it */
+errcode_t e2fsck_should_rebuild_extents(e2fsck_t ctx,
+				   struct problem_context *pctx,
+				   struct extent_tree_info *eti,
+				   struct ext2_extent_info *info)
+{
+	struct extent_tree_level *ei;
+	int i, j, op;
+	unsigned int extents_per_block;
+
+	if (eti->force_rebuild)
+		goto rebuild;
+
+	extents_per_block = (ctx->fs->blocksize -
+			     sizeof(struct ext3_extent_header)) /
+			    sizeof(struct ext3_extent);
+	/*
+	 * If we can consolidate a level or shorten the tree, schedule the
+	 * extent tree to be rebuilt.
+	 */
+	for (i = 0, ei = eti->ext_info; i < info->max_depth + 1; i++, ei++) {
+		if (ei->max_extents - ei->num_extents > extents_per_block) {
+			pctx->blk = i;
+			op = PR_1E_CAN_NARROW_EXTENT_TREE;
+			goto rebuild;
+		}
+		for (j = 0; j < i; j++) {
+			if (ei->num_extents < eti->ext_info[j].max_extents) {
+				pctx->blk = i;
+				op = PR_1E_CAN_COLLAPSE_EXTENT_TREE;
+				goto rebuild;
+			}
+		}
+	}
+	return 0;
+
+rebuild:
+	if (eti->force_rebuild || fix_problem(ctx, op, pctx))
+		return e2fsck_rebuild_extents_later(ctx, eti->ino);
+	return 0;
+}
+
+void e2fsck_pass1e(e2fsck_t ctx)
+{
+	rebuild_extents(ctx, "Pass 1E", PR_1E_PASS_HEADER);
+}
diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index da4d3cf..2658f39 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -56,6 +56,8 @@
 #define _INLINE_ inline
 #endif
 
+#undef DEBUG
+
 static int process_block(ext2_filsys fs, blk64_t	*blocknr,
 			 e2_blkcnt_t blockcnt, blk64_t ref_blk,
 			 int ref_offset, void *priv_data);
@@ -94,6 +96,7 @@ struct process_block_struct {
 	ext2fs_block_bitmap fs_meta_blocks;
 	e2fsck_t	ctx;
 	region_t	region;
+	struct extent_tree_info	eti;
 };
 
 struct process_inode_block {
@@ -1827,6 +1830,7 @@ void e2fsck_pass1(e2fsck_t ctx)
 		}
 		e2fsck_pass1_dupblocks(ctx, block_buf);
 	}
+	ctx->flags |= E2F_FLAG_ALLOC_OK;
 	ext2fs_free_mem(&inodes_to_process);
 endit:
 	e2fsck_use_inode_shortcuts(ctx, 0);
@@ -2461,6 +2465,22 @@ static void scan_extent_node(e2fsck_t ctx, struct problem_context *pctx,
 	pctx->errcode = ext2fs_extent_get_info(ehandle, &info);
 	if (pctx->errcode)
 		return;
+	if (!pb->eti.force_rebuild) {
+		struct extent_tree_level *etl;
+
+		etl = pb->eti.ext_info + info.curr_level;
+		etl->num_extents += info.num_entries;
+		etl->max_extents += info.max_entries;
+		/*
+		 * Implementation wart: Splitting extent blocks when appending
+		 * will leave the old block with one free entry.  Therefore
+		 * unless the node is totally full, pretend that a non-root
+		 * extent block can hold one fewer entry than it actually does,
+		 * so that we don't repeatedly rebuild the extent tree.
+		 */
+		if (info.curr_level && info.num_entries < info.max_entries)
+			etl->max_extents--;
+	}
 
 	pctx->errcode = ext2fs_extent_get(ehandle, EXT2_EXTENT_FIRST_SIB,
 					  &extent);
@@ -2797,11 +2817,27 @@ static void check_blocks_extents(e2fsck_t ctx, struct problem_context *pctx,
 
 	retval = ext2fs_extent_get_info(ehandle, &info);
 	if (retval == 0) {
-		if (info.max_depth >= MAX_EXTENT_DEPTH_COUNT)
-			info.max_depth = MAX_EXTENT_DEPTH_COUNT-1;
-		ctx->extent_depth_count[info.max_depth]++;
+		int max_depth = info.max_depth;
+
+		if (max_depth >= MAX_EXTENT_DEPTH_COUNT)
+			max_depth = MAX_EXTENT_DEPTH_COUNT-1;
+		ctx->extent_depth_count[max_depth]++;
 	}
 
+	/* Check maximum extent depth */
+	pctx->blk = info.max_depth;
+	pctx->blk2 = ext2fs_max_extent_depth(ehandle);
+	if (pctx->blk2 < pctx->blk &&
+	    fix_problem(ctx, PR_1_EXTENT_BAD_MAX_DEPTH, pctx))
+		pb->eti.force_rebuild = 1;
+
+	/* Can we collect extent tree level stats? */
+	pctx->blk = MAX_EXTENT_DEPTH_COUNT;
+	if (pctx->blk2 > pctx->blk)
+		fix_problem(ctx, PR_1E_MAX_EXTENT_TREE_DEPTH, pctx);
+	memset(pb->eti.ext_info, 0, sizeof(pb->eti.ext_info));
+	pb->eti.ino = pb->ino;
+
 	pb->region = region_create(0, info.max_lblk);
 	if (!pb->region) {
 		ext2fs_extent_free(ehandle);
@@ -2824,6 +2860,16 @@ static void check_blocks_extents(e2fsck_t ctx, struct problem_context *pctx,
 	region_free(pb->region);
 	pb->region = NULL;
 	ext2fs_extent_free(ehandle);
+
+	/* Rebuild unless it's a dir and we're rehashing it */
+	if (LINUX_S_ISDIR(inode->i_mode) &&
+	    e2fsck_dir_will_be_rehashed(ctx, ino))
+		return;
+
+	if (ctx->options & E2F_OPT_CONVERT_BMAP)
+		e2fsck_rebuild_extents_later(ctx, ino);
+	else
+		e2fsck_should_rebuild_extents(ctx, pctx, &pb->eti, &info);
 }
 
 /*
@@ -2908,6 +2954,7 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx,
 	pb.pctx = pctx;
 	pb.ctx = ctx;
 	pb.inode_modified = 0;
+	pb.eti.force_rebuild = 0;
 	pctx->ino = ino;
 	pctx->errcode = 0;
 
@@ -2971,6 +3018,15 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx,
 						  "check_blocks");
 			fs->flags = (flags & EXT2_FLAG_IGNORE_CSUM_ERRORS) |
 				    (fs->flags & ~EXT2_FLAG_IGNORE_CSUM_ERRORS);
+
+			if (ctx->options & E2F_OPT_CONVERT_BMAP) {
+#ifdef DEBUG
+				printf("bmap rebuild ino=%d\n", ino);
+#endif
+				if (!LINUX_S_ISDIR(inode->i_mode) ||
+				    !e2fsck_dir_will_be_rehashed(ctx, ino))
+					e2fsck_rebuild_extents_later(ctx, ino);
+			}
 		}
 	}
 	end_problem_latch(ctx, PR_LATCH_BLOCK);
diff --git a/e2fsck/problem.c b/e2fsck/problem.c
index a63e61c..07d79f3 100644
--- a/e2fsck/problem.c
+++ b/e2fsck/problem.c
@@ -1101,6 +1101,11 @@ static struct e2fsck_problem problem_table[] = {
 	  N_("@i %i has a duplicate @x mapping\n\t(logical @b %c, @n physical @b %b, len %N)\n"),
 	  PROMPT_CLEAR, 0 },
 
+	/* Inode extent tree could be more shallow */
+	{ PR_1_EXTENT_BAD_MAX_DEPTH,
+	  N_("@i %i @x tree could be more shallow (%b; could be <= %c)\n"),
+	  PROMPT_FIX, PR_NO_OK | PR_PREEN_NO | PR_PREEN_OK },
+
 	/* Pass 1b errors */
 
 	/* Pass 1B: Rescan for duplicate/bad blocks */
@@ -1198,6 +1203,48 @@ static struct e2fsck_problem problem_table[] = {
 	{ PR_1D_CLONE_ERROR,
 	  N_("Couldn't clone file: %m\n"), PROMPT_NONE, 0 },
 
+	/* Pass 1E Extent tree optimization	*/
+
+	/* Pass 1E: Optimizing extent trees */
+	{ PR_1E_PASS_HEADER,
+	  N_("Pass 1E: Optimizing @x trees\n"),
+	  PROMPT_NONE, PR_PREEN_NOMSG },
+
+	/* Failed to optimize extent tree */
+	{ PR_1E_OPTIMIZE_EXT_ERR,
+	  N_("Failed to optimize @x tree %p (%i): %m\n"),
+	  PROMPT_NONE, 0 },
+
+	/* Optimizing extent trees */
+	{ PR_1E_OPTIMIZE_EXT_HEADER,
+	  N_("Optimizing @x trees: "),
+	  PROMPT_NONE, PR_MSG_ONLY },
+
+	/* Rebuilding extent tree %d */
+	{ PR_1E_OPTIMIZE_EXT,
+	  " %i",
+	  PROMPT_NONE, PR_LATCH_OPTIMIZE_EXT | PR_PREEN_NOHDR},
+
+	/* Rebuilding extent tree end */
+	{ PR_1E_OPTIMIZE_EXT_END,
+	  "\n",
+	  PROMPT_NONE, PR_PREEN_NOHDR },
+
+	/* Internal error: extent tree depth too large */
+	{ PR_1E_MAX_EXTENT_TREE_DEPTH,
+	  N_("Internal error: max extent tree depth too large (%b; expected=%c).\n"),
+	  PROMPT_NONE, PR_FATAL },
+
+	/* Inode extent tree could be shorter */
+	{ PR_1E_CAN_COLLAPSE_EXTENT_TREE,
+	  N_("@i %i @x tree could be shorter.\n\t(level %b is unnecessary)\n"),
+	  PROMPT_FIX, PR_NO_OK | PR_PREEN_NO | PR_PREEN_OK },
+
+	/* Inode extent tree could be narrower */
+	{ PR_1E_CAN_NARROW_EXTENT_TREE,
+	  N_("@i %i @x tree could be narrower.\n\t(level %b has unnecessary nodes)\n"),
+	  PROMPT_FIX, PR_NO_OK | PR_PREEN_NO | PR_PREEN_OK },
+
 	/* Pass 2 errors */
 
 	/* Pass 2: Checking directory structure */
@@ -1946,6 +1993,7 @@ static struct latch_descr pr_latch_info[] = {
 	{ PR_LATCH_TOOBIG, PR_1_INODE_TOOBIG, 0 },
 	{ PR_LATCH_OPTIMIZE_DIR, PR_3A_OPTIMIZE_DIR_HEADER, PR_3A_OPTIMIZE_DIR_END },
 	{ PR_LATCH_BG_CHECKSUM, PR_0_GDT_CSUM_LATCH, 0 },
+	{ PR_LATCH_OPTIMIZE_EXT, PR_1E_OPTIMIZE_EXT_HEADER, PR_1E_OPTIMIZE_EXT_END },
 	{ -1, 0, 0 },
 };
 
diff --git a/e2fsck/problem.h b/e2fsck/problem.h
index 3c28166..4678b23 100644
--- a/e2fsck/problem.h
+++ b/e2fsck/problem.h
@@ -40,6 +40,7 @@ struct problem_context {
 #define PR_LATCH_TOOBIG	0x0080	/* Latch for file to big errors */
 #define PR_LATCH_OPTIMIZE_DIR 0x0090 /* Latch for optimize directories */
 #define PR_LATCH_BG_CHECKSUM 0x00A0  /* Latch for block group checksums */
+#define PR_LATCH_OPTIMIZE_EXT 0x00B0  /* Latch for rebuild extents */
 
 #define PR_LATCH(x)	((((x) & PR_LATCH_MASK) >> 4) - 1)
 
@@ -641,6 +642,9 @@ struct problem_context {
 /* leaf extent collision */
 #define PR_1_EXTENT_COLLISION			0x01007D
 
+/* extent tree max depth too big */
+#define PR_1_EXTENT_BAD_MAX_DEPTH		0x01007E
+
 /*
  * Pass 1b errors
  */
@@ -704,6 +708,33 @@ struct problem_context {
 #define PR_1D_CLONE_ERROR	0x013008
 
 /*
+ * Pass 1e --- rebuilding extent trees
+ */
+/* Pass 1e: Rebuilding extent trees */
+#define PR_1E_PASS_HEADER		0x014000
+
+/* Error rehash directory */
+#define PR_1E_OPTIMIZE_EXT_ERR		0x014001
+
+/* Rebuilding extent trees */
+#define PR_1E_OPTIMIZE_EXT_HEADER	0x014002
+
+/* Rebuilding extent %d */
+#define PR_1E_OPTIMIZE_EXT		0x014003
+
+/* Rebuilding extent tree end */
+#define PR_1E_OPTIMIZE_EXT_END		0x014004
+
+/* Internal error: extent tree depth too large */
+#define PR_1E_MAX_EXTENT_TREE_DEPTH	0x014005
+
+/* Inode extent tree could be shorter */
+#define PR_1E_CAN_COLLAPSE_EXTENT_TREE	0x014006
+
+/* Inode extent tree could be narrower */
+#define PR_1E_CAN_NARROW_EXTENT_TREE	0x014007
+
+/*
  * Pass 2 errors
  */
 
@@ -1032,6 +1063,8 @@ struct problem_context {
 /* Rehashing dir end */
 #define PR_3A_OPTIMIZE_DIR_END		0x031005
 
+/* Pass 3B is really just 1E */
+
 /*
  * Pass 4 errors
  */
diff --git a/e2fsck/rehash.c b/e2fsck/rehash.c
index 348923e..dadcfc7 100644
--- a/e2fsck/rehash.c
+++ b/e2fsck/rehash.c
@@ -749,11 +749,11 @@ static int write_dir_block(ext2_filsys fs,
 
 static errcode_t write_directory(e2fsck_t ctx, ext2_filsys fs,
 				 struct out_dir *outdir,
-				 ext2_ino_t ino, int compress)
+				 ext2_ino_t ino, struct ext2_inode *inode,
+				 int compress)
 {
 	struct write_dir_struct wd;
 	errcode_t	retval;
-	struct ext2_inode 	inode;
 
 	retval = e2fsck_expand_directory(ctx, ino, -1, outdir->num);
 	if (retval)
@@ -772,22 +772,23 @@ static errcode_t write_directory(e2fsck_t ctx, ext2_filsys fs,
 	if (wd.err)
 		return wd.err;
 
-	e2fsck_read_inode(ctx, ino, &inode, "rehash_dir");
+	e2fsck_read_inode(ctx, ino, inode, "rehash_dir");
 	if (compress)
-		inode.i_flags &= ~EXT2_INDEX_FL;
+		inode->i_flags &= ~EXT2_INDEX_FL;
 	else
-		inode.i_flags |= EXT2_INDEX_FL;
-	retval = ext2fs_inode_size_set(fs, &inode,
+		inode->i_flags |= EXT2_INDEX_FL;
+	retval = ext2fs_inode_size_set(fs, inode,
 				       outdir->num * fs->blocksize);
 	if (retval)
 		return retval;
-	ext2fs_iblk_sub_blocks(fs, &inode, wd.cleared);
-	e2fsck_write_inode(ctx, ino, &inode, "rehash_dir");
+	ext2fs_iblk_sub_blocks(fs, inode, wd.cleared);
+	e2fsck_write_inode(ctx, ino, inode, "rehash_dir");
 
 	return 0;
 }
 
-errcode_t e2fsck_rehash_dir(e2fsck_t ctx, ext2_ino_t ino)
+errcode_t e2fsck_rehash_dir(e2fsck_t ctx, ext2_ino_t ino,
+			    struct problem_context *pctx)
 {
 	ext2_filsys 		fs = ctx->fs;
 	errcode_t		retval;
@@ -897,10 +898,14 @@ resort:
 			goto errout;
 	}
 
-	retval = write_directory(ctx, fs, &outdir, ino, fd.compress);
+	retval = write_directory(ctx, fs, &outdir, ino, &inode, fd.compress);
 	if (retval)
 		goto errout;
 
+	if (ctx->options & E2F_OPT_CONVERT_BMAP)
+		retval = e2fsck_rebuild_extents_later(ctx, ino);
+	else
+		retval = e2fsck_check_rebuild_extents(ctx, ino, &inode, pctx);
 errout:
 	free(dir_buf);
 	free(fd.harray);
@@ -944,7 +949,7 @@ void e2fsck_rehash_directories(e2fsck_t ctx)
 #if 0
 		fix_problem(ctx, PR_3A_OPTIMIZE_DIR, &pctx);
 #endif
-		pctx.errcode = e2fsck_rehash_dir(ctx, ino);
+		pctx.errcode = e2fsck_rehash_dir(ctx, ino, &pctx);
 		if (pctx.errcode) {
 			end_problem_latch(ctx, PR_LATCH_OPTIMIZE_DIR);
 			fix_problem(ctx, PR_3A_OPTIMIZE_DIR_ERR, &pctx);
diff --git a/e2fsck/super.c b/e2fsck/super.c
index 1e7e749..e64262a 100644
--- a/e2fsck/super.c
+++ b/e2fsck/super.c
@@ -606,6 +606,13 @@ void check_super_block(e2fsck_t ctx)
 		ext2fs_mark_super_dirty(fs);
 	}
 
+	/* Did user ask us to convert files to extents? */
+	if (ctx->options & E2F_OPT_CONVERT_BMAP) {
+		fs->super->s_feature_incompat |=
+			EXT3_FEATURE_INCOMPAT_EXTENTS;
+		ext2fs_mark_super_dirty(fs);
+	}
+
 	if ((fs->super->s_feature_incompat & EXT2_FEATURE_INCOMPAT_META_BG) &&
 	    (fs->super->s_first_meta_bg > fs->desc_blocks)) {
 		pctx.group = fs->desc_blocks;
diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index c366c78..b84bffe 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -709,6 +709,9 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 			else
 				ctx->log_fn = string_copy(ctx, arg, 0);
 			continue;
+		} else if (strcmp(token, "bmap2extent") == 0) {
+			ctx->options |= E2F_OPT_CONVERT_BMAP;
+			continue;
 		} else {
 			fprintf(stderr, _("Unknown extended option: %s\n"),
 				token);
@@ -728,6 +731,7 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 		fputs(("\tdiscard\n"), stderr);
 		fputs(("\tnodiscard\n"), stderr);
 		fputs(("\treadahead_kb=<buffer size>\n"), stderr);
+		fputs(("\tbmap2extent\n"), stderr);
 		fputc('\n', stderr);
 		exit(1);
 	}
diff --git a/tests/f_extent_bad_node/expect.1 b/tests/f_extent_bad_node/expect.1
index 0c0bc28..c9643a1 100644
--- a/tests/f_extent_bad_node/expect.1
+++ b/tests/f_extent_bad_node/expect.1
@@ -2,8 +2,13 @@ Pass 1: Checking inodes, blocks, and sizes
 Inode 12 has an invalid extent node (blk 22, lblk 0)
 Clear? yes
 
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 12, i_blocks is 16, should be 8.  Fix? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
@@ -11,13 +16,13 @@ Pass 5: Checking group summary information
 Block bitmap differences:  -(21--23) -25
 Fix? yes
 
-Free blocks count wrong for group #0 (71, counted=75).
+Free blocks count wrong for group #0 (73, counted=77).
 Fix? yes
 
-Free blocks count wrong (71, counted=75).
+Free blocks count wrong (73, counted=77).
 Fix? yes
 
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
-test_filesys: 12/16 files (0.0% non-contiguous), 25/100 blocks
+test_filesys: 12/16 files (0.0% non-contiguous), 23/100 blocks
 Exit status is 1
diff --git a/tests/f_extent_bad_node/expect.2 b/tests/f_extent_bad_node/expect.2
index 568c792..b78b193 100644
--- a/tests/f_extent_bad_node/expect.2
+++ b/tests/f_extent_bad_node/expect.2
@@ -3,5 +3,5 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-test_filesys: 12/16 files (0.0% non-contiguous), 25/100 blocks
+test_filesys: 12/16 files (0.0% non-contiguous), 23/100 blocks
 Exit status is 0
diff --git a/tests/f_extent_int_bad_magic/expect.1 b/tests/f_extent_int_bad_magic/expect.1
index 0e82e2b..3529636 100644
--- a/tests/f_extent_int_bad_magic/expect.1
+++ b/tests/f_extent_int_bad_magic/expect.1
@@ -2,8 +2,13 @@ Pass 1: Checking inodes, blocks, and sizes
 Inode 12 has an invalid extent node (blk 1295, lblk 0)
 Clear? yes
 
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 12, i_blocks is 712, should be 0.  Fix? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
diff --git a/tests/f_extent_leaf_bad_magic/expect.1 b/tests/f_extent_leaf_bad_magic/expect.1
index 7b6dbf1..ae27ecc 100644
--- a/tests/f_extent_leaf_bad_magic/expect.1
+++ b/tests/f_extent_leaf_bad_magic/expect.1
@@ -2,8 +2,13 @@ Pass 1: Checking inodes, blocks, and sizes
 Inode 12 has an invalid extent node (blk 1604, lblk 0)
 Clear? yes
 
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 12, i_blocks is 18, should be 0.  Fix? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
diff --git a/tests/f_extent_oobounds/expect.1 b/tests/f_extent_oobounds/expect.1
index 3164ea0..f0e282e 100644
--- a/tests/f_extent_oobounds/expect.1
+++ b/tests/f_extent_oobounds/expect.1
@@ -3,8 +3,13 @@ Inode 12, end of extent exceeds allowed value
 	(logical block 15, physical block 200, len 30)
 Clear? yes
 
+Inode 12 extent tree could be narrower.
+	(level 1 has unnecessary nodes)
+Fix? yes
+
 Inode 12, i_blocks is 154, should be 94.  Fix? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
@@ -12,13 +17,13 @@ Pass 5: Checking group summary information
 Block bitmap differences:  -(200--229)
 Fix? yes
 
-Free blocks count wrong for group #0 (156, counted=186).
+Free blocks count wrong for group #0 (158, counted=188).
 Fix? yes
 
-Free blocks count wrong (156, counted=186).
+Free blocks count wrong (158, counted=188).
 Fix? yes
 
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
-test_filesys: 12/32 files (8.3% non-contiguous), 70/256 blocks
+test_filesys: 12/32 files (8.3% non-contiguous), 68/256 blocks
 Exit status is 1
diff --git a/tests/f_extent_oobounds/expect.2 b/tests/f_extent_oobounds/expect.2
index 22c4f2c..0729283 100644
--- a/tests/f_extent_oobounds/expect.2
+++ b/tests/f_extent_oobounds/expect.2
@@ -3,5 +3,5 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-test_filesys: 12/32 files (8.3% non-contiguous), 70/256 blocks
+test_filesys: 12/32 files (8.3% non-contiguous), 68/256 blocks
 Exit status is 0
diff --git a/tests/f_extents/expect.1 b/tests/f_extents/expect.1
index aeebc7b..2751eb9 100644
--- a/tests/f_extents/expect.1
+++ b/tests/f_extents/expect.1
@@ -6,6 +6,10 @@ Inode 12 has an invalid extent
 	(logical block 0, invalid physical block 21994527527949, len 17)
 Clear? yes
 
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 12, i_blocks is 34, should be 0.  Fix? yes
 
 Inode 13 missing EXTENT_FL, but is in extents format
@@ -21,6 +25,10 @@ Inode 17 has an invalid extent
 	(logical block 0, invalid physical block 22011707397135, len 15)
 Clear? yes
 
+Inode 17 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 17, i_blocks is 32, should be 0.  Fix? yes
 
 Error while reading over extent tree in inode 18: Corrupt extent header
@@ -31,6 +39,7 @@ Inode 18, i_blocks is 2, should be 0.  Fix? yes
 Special (device/socket/fifo) file (inode 19) has extents
 or inline-data flag set.  Clear? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Entry 'fbad-flag' in / (2) has deleted/unused inode 18.  Clear? yes
 

[PATCH 28/54] tests: verify proper rebuilding of sparse extent trees and block map file conversion

[Darrick J. Wong]

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/f_collapse_extent_tree/expect.1       |   18 ++++
 tests/f_collapse_extent_tree/expect.2       |   10 ++
 tests/f_collapse_extent_tree/image.gz       |  Bin
 tests/f_collapse_extent_tree/name           |    1 
 tests/f_collapse_extent_tree/script         |  118 +++++++++++++++++++++++++++
 tests/f_compress_extent_tree_level/expect.1 |   25 ++++++
 tests/f_compress_extent_tree_level/expect.2 |   17 ++++
 tests/f_compress_extent_tree_level/image.gz |  Bin
 tests/f_compress_extent_tree_level/name     |    1 
 tests/f_compress_extent_tree_level/script   |  118 +++++++++++++++++++++++++++
 tests/f_convert_bmap/expect.1               |   26 ++++++
 tests/f_convert_bmap/expect.2               |   10 ++
 tests/f_convert_bmap/image.gz               |  Bin
 tests/f_convert_bmap/name                   |    1 
 tests/f_convert_bmap/script                 |  117 +++++++++++++++++++++++++++
 tests/f_convert_bmap_and_extent/expect.1    |   33 +++++++
 tests/f_convert_bmap_and_extent/expect.2    |   16 ++++
 tests/f_convert_bmap_and_extent/image.gz    |  Bin
 tests/f_convert_bmap_and_extent/name        |    1 
 tests/f_convert_bmap_and_extent/script      |  119 +++++++++++++++++++++++++++
 tests/f_extent_too_deep/expect.1            |   23 +++++
 tests/f_extent_too_deep/expect.2            |   10 ++
 tests/f_extent_too_deep/image.gz            |  Bin
 tests/f_extent_too_deep/name                |    1 
 tests/f_extent_too_deep/script              |  118 +++++++++++++++++++++++++++
 tests/f_opt_extent/expect                   |   55 ++++++++++++
 tests/f_opt_extent/name                     |    1 
 tests/f_opt_extent/script                   |   64 +++++++++++++++
 tests/f_opt_extent_ext3/expect              |   44 ++++++++++
 tests/f_opt_extent_ext3/name                |    1 
 tests/f_opt_extent_ext3/script              |   65 +++++++++++++++
 31 files changed, 1013 insertions(+)
 create mode 100644 tests/f_collapse_extent_tree/expect.1
 create mode 100644 tests/f_collapse_extent_tree/expect.2
 create mode 100644 tests/f_collapse_extent_tree/image.gz
 create mode 100644 tests/f_collapse_extent_tree/name
 create mode 100644 tests/f_collapse_extent_tree/script
 create mode 100644 tests/f_compress_extent_tree_level/expect.1
 create mode 100644 tests/f_compress_extent_tree_level/expect.2
 create mode 100644 tests/f_compress_extent_tree_level/image.gz
 create mode 100644 tests/f_compress_extent_tree_level/name
 create mode 100644 tests/f_compress_extent_tree_level/script
 create mode 100644 tests/f_convert_bmap/expect.1
 create mode 100644 tests/f_convert_bmap/expect.2
 create mode 100644 tests/f_convert_bmap/image.gz
 create mode 100644 tests/f_convert_bmap/name
 create mode 100644 tests/f_convert_bmap/script
 create mode 100644 tests/f_convert_bmap_and_extent/expect.1
 create mode 100644 tests/f_convert_bmap_and_extent/expect.2
 create mode 100644 tests/f_convert_bmap_and_extent/image.gz
 create mode 100644 tests/f_convert_bmap_and_extent/name
 create mode 100644 tests/f_convert_bmap_and_extent/script
 create mode 100644 tests/f_extent_too_deep/expect.1
 create mode 100644 tests/f_extent_too_deep/expect.2
 create mode 100644 tests/f_extent_too_deep/image.gz
 create mode 100644 tests/f_extent_too_deep/name
 create mode 100644 tests/f_extent_too_deep/script
 create mode 100644 tests/f_opt_extent/expect
 create mode 100644 tests/f_opt_extent/name
 create mode 100644 tests/f_opt_extent/script
 create mode 100644 tests/f_opt_extent_ext3/expect
 create mode 100644 tests/f_opt_extent_ext3/name
 create mode 100644 tests/f_opt_extent_ext3/script


diff --git a/tests/f_collapse_extent_tree/expect.1 b/tests/f_collapse_extent_tree/expect.1
new file mode 100644
index 0000000..d76880c
--- /dev/null
+++ b/tests/f_collapse_extent_tree/expect.1
@@ -0,0 +1,18 @@
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 -     0     9              1
+ 1/ 1   1/  1     0 -     0    10 -    10      1 
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Pass 1E: Optimizing extent trees
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 12/128 files (0.0% non-contiguous), 19/512 blocks
+Exit status is 1
diff --git a/tests/f_collapse_extent_tree/expect.2 b/tests/f_collapse_extent_tree/expect.2
new file mode 100644
index 0000000..a1d28b1
--- /dev/null
+++ b/tests/f_collapse_extent_tree/expect.2
@@ -0,0 +1,10 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 12/128 files (0.0% non-contiguous), 19/512 blocks
+Exit status is 0
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  1     0 -     0    10 -    10      1 
diff --git a/tests/f_collapse_extent_tree/image.gz b/tests/f_collapse_extent_tree/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..97036cc597c5d2bcf6da6ffa68039b743b46aa20
GIT binary patch
literal 2537
zcmb2|=3r2~7aqdI{Pxc7?BGBdh6lxY##=IfImEQ^-3nbX`9VZ0->#??lNCe@FW4FQ
z#YVL}TII*Z*HrHjAfO&wvUze}my@ja$2)hVrIT2)v*+IzulqiySl;uUviknXEiDW(
zp{IYUa6fd>vO6yD>gk^~J=eHSyKdZJX8c#hmFMiUr|S)uTs`cRwf(*Jw5JI-Ev9#g
zzx10H^z+x-+1&PezM<j%wN-x#UY^?f?s?&}Z!v%C-)BF5eDl?n_HOO-adCV1e|+^f
zi)Zq}X`-jCz9%p5SsMMuaQ~#F154Q%4#=Ijd_P(9SoF_&+pXIBr(Hc<aafRvfuW&!
zy=4ribnWdMatsU%H@<&*|36zXO?crCT?gHin@3zeO5c0DaMs;f`-AOklmETD^R*_D
z<Gowbo@dFzK=l>@Y`c&Bzwh4l(H_VvIMHLp0;CiE@B>MS|I9!V1gcl=k~>^-C+GRq
zERBES2J6qaJ*xkI`|HBKUj^>6K=~WTAJ?~oRpKF>ikGbB1<K1<*)cNWQAa>3L0jxv
z+Ope)Q@;H^{nJEJ_gnqOo+Qus7x`2FDlW?`wQch{Ubb=Tv2VRgcl<QbdwS81f1O)`
zxz73}p2?fs|I45L_x?U_+1AJMzxKcTu9(hR=B|9cUak6ReWXR?Gy8?^|AIA>)j!SO
zTl3dGUh~(|qxV1UpRv3(_UHXHezOY~k3TI=558jc%31d7spkCte`^0vE7$)i)8ioE
m=uzp>5Eu=C(GVC7fzc2c4FPgP;J~%NEDm#Conv57U;qF<XAwLA

literal 0
HcmV?d00001

diff --git a/tests/f_collapse_extent_tree/name b/tests/f_collapse_extent_tree/name
new file mode 100644
index 0000000..83e506f
--- /dev/null
+++ b/tests/f_collapse_extent_tree/name
@@ -0,0 +1 @@
+extent tree can be collapsed one level
diff --git a/tests/f_collapse_extent_tree/script b/tests/f_collapse_extent_tree/script
new file mode 100644
index 0000000..ee18438
--- /dev/null
+++ b/tests/f_collapse_extent_tree/script
@@ -0,0 +1,118 @@
+if [ "$DESCRIPTION"x != x ]; then
+	test_description="$DESCRIPTION"
+fi
+if [ "$IMAGE"x = x ]; then
+	IMAGE=$test_dir/image.gz
+fi
+
+if [ "$FSCK_OPT"x = x ]; then
+	FSCK_OPT=-yf
+fi
+
+if [ "$SECOND_FSCK_OPT"x = x ]; then
+	SECOND_FSCK_OPT=-yf
+fi
+
+if [ "$OUT1"x = x ]; then
+	OUT1=$test_name.1.log
+fi
+
+if [ "$OUT2"x = x ]; then
+	OUT2=$test_name.2.log
+fi
+
+if [ "$EXP1"x = x ]; then
+	if [ -f $test_dir/expect.1.gz ]; then
+		EXP1=$test_name.1.tmp
+		gunzip < $test_dir/expect.1.gz > $EXP1
+	else
+		EXP1=$test_dir/expect.1
+	fi
+fi
+
+if [ "$EXP2"x = x ]; then
+	if [ -f $test_dir/expect.2.gz ]; then
+		EXP2=$test_name.2.tmp
+		gunzip < $test_dir/expect.2.gz > $EXP2
+	else
+		EXP2=$test_dir/expect.2
+	fi
+fi
+
+if [ "$SKIP_GUNZIP" != "true" ] ; then
+	gunzip < $IMAGE > $TMPFILE
+fi
+
+cp /dev/null $OUT1
+
+eval $PREP_CMD
+
+echo 'ex /a' > $TMPFILE.cmd
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE > $OUT1.new 2>&1
+rm -rf $TMPFILE.cmd
+$FSCK $FSCK_OPT  -N test_filesys $TMPFILE >> $OUT1.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT1.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT1.new >> $OUT1
+rm -f $OUT1.new
+
+if [ "$ONE_PASS_ONLY" != "true" ]; then
+	$FSCK $SECOND_FSCK_OPT -N test_filesys $TMPFILE > $OUT2.new 2>&1 
+	status=$?
+	echo Exit status is $status >> $OUT2.new
+	echo 'ex /a' > $TMPFILE.cmd
+	$DEBUGFS -f $TMPFILE.cmd $TMPFILE >> $OUT2.new 2>&1
+	rm -rf $TMPFILE.cmd
+	sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT2.new > $OUT2
+	rm -f $OUT2.new
+fi
+
+eval $AFTER_CMD
+
+if [ "$SKIP_VERIFY" != "true" ] ; then
+	rm -f $test_name.ok $test_name.failed
+	cmp -s $OUT1 $EXP1
+	status1=$?
+	if [ "$ONE_PASS_ONLY" != "true" ]; then
+		cmp -s $OUT2 $EXP2
+		status2=$?
+	else
+		status2=0
+	fi
+	if [ "$PASS_ZERO" = "true" ]; then
+		cmp -s $test_name.0.log	$test_dir/expect.0
+		status3=$?
+	else
+		status3=0
+	fi
+
+	if [ -z "$test_description" ] ; then
+		description="$test_name"
+	else
+		description="$test_name: $test_description"
+	fi
+
+	if [ "$status1" -eq 0 -a "$status2" -eq 0 -a "$status3" -eq 0 ] ; then
+		echo "$description: ok"
+		touch $test_name.ok
+	else
+		echo "$description: failed"
+		rm -f $test_name.failed
+		if [ "$PASS_ZERO" = "true" ]; then
+			diff $DIFF_OPTS $test_dir/expect.0 \
+				$test_name.0.log >> $test_name.failed
+		fi
+		diff $DIFF_OPTS $EXP1 $OUT1 >> $test_name.failed
+		if [ "$ONE_PASS_ONLY" != "true" ]; then
+			diff $DIFF_OPTS $EXP2 $OUT2 >> $test_name.failed
+		fi
+	fi
+	rm -f tmp_expect
+fi
+
+if [ "$SKIP_CLEANUP" != "true" ] ; then
+	unset IMAGE FSCK_OPT SECOND_FSCK_OPT OUT1 OUT2 EXP1 EXP2 
+	unset SKIP_VERIFY SKIP_CLEANUP SKIP_GUNZIP ONE_PASS_ONLY PREP_CMD
+	unset DESCRIPTION SKIP_UNLINK AFTER_CMD PASS_ZERO
+fi
+
diff --git a/tests/f_compress_extent_tree_level/expect.1 b/tests/f_compress_extent_tree_level/expect.1
new file mode 100644
index 0000000..45dcb39
--- /dev/null
+++ b/tests/f_compress_extent_tree_level/expect.1
@@ -0,0 +1,25 @@
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  2     0 -    16     9             17
+ 1/ 1   1/  4     0 -     0    10 -    10      1 
+ 1/ 1   2/  4    11 -    11   100 -   100      1 
+ 1/ 1   3/  4    13 -    13   101 -   101      1 
+ 1/ 1   4/  4    15 -    15   102 -   102      1 
+ 0/ 1   2/  2    17 -    21    12              5
+ 1/ 1   1/  3    17 -    17   103 -   103      1 
+ 1/ 1   2/  3    19 -    19   104 -   104      1 
+ 1/ 1   3/  3    21 -    21   105 -   105      1 
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 extent tree could be narrower.
+	(level 1 has unnecessary nodes)
+Fix? yes
+
+Pass 1E: Optimizing extent trees
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 12/128 files (8.3% non-contiguous), 26/512 blocks
+Exit status is 1
diff --git a/tests/f_compress_extent_tree_level/expect.2 b/tests/f_compress_extent_tree_level/expect.2
new file mode 100644
index 0000000..07d1082
--- /dev/null
+++ b/tests/f_compress_extent_tree_level/expect.2
@@ -0,0 +1,17 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 12/128 files (8.3% non-contiguous), 26/512 blocks
+Exit status is 0
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 -    21     9             22
+ 1/ 1   1/  7     0 -     0    10 -    10      1 
+ 1/ 1   2/  7    11 -    11   100 -   100      1 
+ 1/ 1   3/  7    13 -    13   101 -   101      1 
+ 1/ 1   4/  7    15 -    15   102 -   102      1 
+ 1/ 1   5/  7    17 -    17   103 -   103      1 
+ 1/ 1   6/  7    19 -    19   104 -   104      1 
+ 1/ 1   7/  7    21 -    21   105 -   105      1 
diff --git a/tests/f_compress_extent_tree_level/image.gz b/tests/f_compress_extent_tree_level/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..a552a586ce20e4fbc01d6f332ec800c5c5b6011f
GIT binary patch
literal 2581
zcmb2|=3rR!C_IFT`RyHR9}!0hh6gkEYCmX@`O&3zM1osdIA^Jlw(gQiEytLI<J9c9
z&L3%E^Y!W3JGZlUZ`ds<>4n?Hq`jLtg`|b&q_WGat95BG-FN>tx4!H<U)lS-AJ6lO
z6&ZG4&Dp8pWHDvdcEgsixcDa#cN4c|Jrj3xXS?6icVxq+qB_I2E!8u;UtfEt{cM|d
zx!<y9uDd4Byy90XBb_r>eqZ_jr)RIH>*?If{BP|Sv-8%k+sAkPd2;YL_u=Paa{ab8
z7R65wT$X5aDi6&4{msJ6yZ%?_>ox0_zxZ;6nIVCD%G3ArMBKw){||e0eSc1N>&EnR
z0t^fcIrZA@uO9}lJ$3)OSwS^W;KBazx_|d&lMDq`Wi#$_4&>npx&GlDckiQ5TVJnz
zJo$QFjI93l)(@*BAG_wy2C6@>?b21V|Nk$ndG;T~<GcJ?1V}ghv<H$2|M-C<2;?k1
z7G&FVWc^>G+?H3>jHhdCT>a0f2bmPx&0M7aJXrJFvl^?T|K`LORWSmc@PEdCYgULx
zh7E5`r)9;j`(JJgG7%fdIpei?Ey$ub7WWu|+Q|hE%FpQb^sPCXXZ5ap`Q=61t^WO#
zxjHTCMMSQ(lh5g6K|Ejc^Tfly|0w>exlI4=-q>Te9zVFr@rQf!_Dhd`Ua)#xf9CQ1
zW2IZaKK4Jh|J#3OInf{H^QM12f1&DYykOsxU+Qau|6eK2ar?V@>iYlgPk)$;9sa}r
zHJ&GZ|LXr;C41uCH=MikZRv}1mHu*%)xaV1=-A3zpVnWgPo1*Aeytb_x&9i}HyQ$?
fAut*OqaiRF0;3^7AOs$KuVp*dzh0JsL4g4PCSfgc

literal 0
HcmV?d00001

diff --git a/tests/f_compress_extent_tree_level/name b/tests/f_compress_extent_tree_level/name
new file mode 100644
index 0000000..fde4f4a
--- /dev/null
+++ b/tests/f_compress_extent_tree_level/name
@@ -0,0 +1 @@
+compress an extent tree level
diff --git a/tests/f_compress_extent_tree_level/script b/tests/f_compress_extent_tree_level/script
new file mode 100644
index 0000000..ee18438
--- /dev/null
+++ b/tests/f_compress_extent_tree_level/script
@@ -0,0 +1,118 @@
+if [ "$DESCRIPTION"x != x ]; then
+	test_description="$DESCRIPTION"
+fi
+if [ "$IMAGE"x = x ]; then
+	IMAGE=$test_dir/image.gz
+fi
+
+if [ "$FSCK_OPT"x = x ]; then
+	FSCK_OPT=-yf
+fi
+
+if [ "$SECOND_FSCK_OPT"x = x ]; then
+	SECOND_FSCK_OPT=-yf
+fi
+
+if [ "$OUT1"x = x ]; then
+	OUT1=$test_name.1.log
+fi
+
+if [ "$OUT2"x = x ]; then
+	OUT2=$test_name.2.log
+fi
+
+if [ "$EXP1"x = x ]; then
+	if [ -f $test_dir/expect.1.gz ]; then
+		EXP1=$test_name.1.tmp
+		gunzip < $test_dir/expect.1.gz > $EXP1
+	else
+		EXP1=$test_dir/expect.1
+	fi
+fi
+
+if [ "$EXP2"x = x ]; then
+	if [ -f $test_dir/expect.2.gz ]; then
+		EXP2=$test_name.2.tmp
+		gunzip < $test_dir/expect.2.gz > $EXP2
+	else
+		EXP2=$test_dir/expect.2
+	fi
+fi
+
+if [ "$SKIP_GUNZIP" != "true" ] ; then
+	gunzip < $IMAGE > $TMPFILE
+fi
+
+cp /dev/null $OUT1
+
+eval $PREP_CMD
+
+echo 'ex /a' > $TMPFILE.cmd
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE > $OUT1.new 2>&1
+rm -rf $TMPFILE.cmd
+$FSCK $FSCK_OPT  -N test_filesys $TMPFILE >> $OUT1.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT1.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT1.new >> $OUT1
+rm -f $OUT1.new
+
+if [ "$ONE_PASS_ONLY" != "true" ]; then
+	$FSCK $SECOND_FSCK_OPT -N test_filesys $TMPFILE > $OUT2.new 2>&1 
+	status=$?
+	echo Exit status is $status >> $OUT2.new
+	echo 'ex /a' > $TMPFILE.cmd
+	$DEBUGFS -f $TMPFILE.cmd $TMPFILE >> $OUT2.new 2>&1
+	rm -rf $TMPFILE.cmd
+	sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT2.new > $OUT2
+	rm -f $OUT2.new
+fi
+
+eval $AFTER_CMD
+
+if [ "$SKIP_VERIFY" != "true" ] ; then
+	rm -f $test_name.ok $test_name.failed
+	cmp -s $OUT1 $EXP1
+	status1=$?
+	if [ "$ONE_PASS_ONLY" != "true" ]; then
+		cmp -s $OUT2 $EXP2
+		status2=$?
+	else
+		status2=0
+	fi
+	if [ "$PASS_ZERO" = "true" ]; then
+		cmp -s $test_name.0.log	$test_dir/expect.0
+		status3=$?
+	else
+		status3=0
+	fi
+
+	if [ -z "$test_description" ] ; then
+		description="$test_name"
+	else
+		description="$test_name: $test_description"
+	fi
+
+	if [ "$status1" -eq 0 -a "$status2" -eq 0 -a "$status3" -eq 0 ] ; then
+		echo "$description: ok"
+		touch $test_name.ok
+	else
+		echo "$description: failed"
+		rm -f $test_name.failed
+		if [ "$PASS_ZERO" = "true" ]; then
+			diff $DIFF_OPTS $test_dir/expect.0 \
+				$test_name.0.log >> $test_name.failed
+		fi
+		diff $DIFF_OPTS $EXP1 $OUT1 >> $test_name.failed
+		if [ "$ONE_PASS_ONLY" != "true" ]; then
+			diff $DIFF_OPTS $EXP2 $OUT2 >> $test_name.failed
+		fi
+	fi
+	rm -f tmp_expect
+fi
+
+if [ "$SKIP_CLEANUP" != "true" ] ; then
+	unset IMAGE FSCK_OPT SECOND_FSCK_OPT OUT1 OUT2 EXP1 EXP2 
+	unset SKIP_VERIFY SKIP_CLEANUP SKIP_GUNZIP ONE_PASS_ONLY PREP_CMD
+	unset DESCRIPTION SKIP_UNLINK AFTER_CMD PASS_ZERO
+fi
+
diff --git a/tests/f_convert_bmap/expect.1 b/tests/f_convert_bmap/expect.1
new file mode 100644
index 0000000..7d2ca86
--- /dev/null
+++ b/tests/f_convert_bmap/expect.1
@@ -0,0 +1,26 @@
+debugfs: stat /a
+Inode: 12   Type: regular    Mode:  0644   Flags: 0x0
+Generation: 1573716129    Version: 0x00000000:00000001
+User:     0   Group:     0   Size: 524288
+File ACL: 0    Directory ACL: 0
+Links: 1   Blockcount: 1030
+Fragment:  Address: 0    Number: 0    Size: 0
+ ctime: 0x5457f87a:62ae2980 -- Mon Nov  3 21:49:46 2014
+ atime: 0x5457f87a:61ba0598 -- Mon Nov  3 21:49:46 2014
+ mtime: 0x5457f87a:62ae2980 -- Mon Nov  3 21:49:46 2014
+crtime: 0x5457f87a:61ba0598 -- Mon Nov  3 21:49:46 2014
+Size of extra inode fields: 28
+BLOCKS:
+(0-11):1025-1036, (IND):24, (12-267):1037-1292, (DIND):25, (IND):41, (268-511):1293-1536
+TOTAL: 515
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 1E: Optimizing extent trees
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 12/128 files (8.3% non-contiguous), 570/2048 blocks
+Exit status is 1
diff --git a/tests/f_convert_bmap/expect.2 b/tests/f_convert_bmap/expect.2
new file mode 100644
index 0000000..632d411
--- /dev/null
+++ b/tests/f_convert_bmap/expect.2
@@ -0,0 +1,10 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 12/128 files (0.0% non-contiguous), 570/2048 blocks
+Exit status is 0
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  1     0 -   511  1025 -  1536    512 
diff --git a/tests/f_convert_bmap/image.gz b/tests/f_convert_bmap/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..7c22532397ba8d42e928f75190dc545cae2140cf
GIT binary patch
literal 3548
zcmeH{TTD|27{|LAMF+@(0Tox+B<ivR0l73*3mPNpq{>AKJy;kpC>$)zXsxs>&77h{
z5d{R1Tf|{+oO3i%(L%x52w{Q@ZBMl=7l%+hhteTCTuLv?;vP1$_wAwI!}sN%d{4js
zKY#y=D@kV;l8$`5%sjNJl`~FyG$XVKkeS>DH}6f}NR0H7XZeLMHZ~n7S%IBv`1#=N
zvthAe6*sSJFDc(vQMvN!_C4jFe<j65!*^OUm={)9=G$@69F1o=F+fw#*9&#jyvB6W
zL}KW^E$$xb{J^bs+dN*RTpr^8kjSR!^SeC_oE_k?>(^gS=<hNObc@YMH>Kvk-p-E5
z35ksKaBDTb(PT!@CVwq0qb~h&i*n<*7OXbTS(15(N<89733W!X1W$_EjVBkiwdO+O
zgdm1$#XPIm+f@lYY^0qqFBCAP$X(HIv%QKlVn5zT?)`;ZVOpN;uzxq3w!?I(<aK$X
zaS!SJe5<2`goNateEzncYj1mM?y%OZ3HI8cvU9NDh)8IB%F@##rq|m2I-85nb<Z2w
zU2pNJ`vVSvg$htSs_DKP485(ZoNIkL`xm<MpHIzr^#yEhcYn0tc1iSaYjk?luhhO5
z90Xc?kWFAOAghA2z*hiVg?NKc04W0&pkFFLh%qk{=rBd}kQi-QLIK4oC}Cc{q-zD*
zP`R6C8NQns<?n~mydY0dt(W*QypVmG06dPE)IW%j((~@|V>qd0P#|_(kwxolh)vP1
z!4D|7w1*Ax2F-m!!v{D?hY_}Bj_Bhv%&`NoNpo3|U_RQefeE7?I)#|DeuRy`+sqVb
zhZToacT$fmp+(`UIb^NwAlA&?)i0^me$S019}PlkxcgO2tDz)Rj%kS-d=8m`$kjMO
z6!SC5aRssfTtZb|mQr*n*h?xr3>3)6@Uzsrhh!CaB~>w;YLUFa>Is<7Q;82Dp_q#3
z<csQ={t_?rJf<eDT62Ug&tzq~S40U_9La2yoxnS+C+pK1rS8~c>oFl5nU*FALaVW-
z#5F6(I+7;8h~wM?EMP?P6ssj5Wk)A#L~23336-^o#f*~pqh(kRQDM!sx4^PST@Y4H
z$d$?>mQGWE>8%_)T$Cn~;8+<UR(9q~qI9L)1bcHnoC0S2?i-o4t{d#wtbKDMepYK!
z><UbF*FFwcpxc%|-kFy%XsC0WSf9P?uV}qL5AcUQmFdYp`an;>8K9}Er5St++^&Mg
z!A0OEqvaYnu4F^;aunnuN*<J&rxAkvO3UQwWiHGAd&`@O7ag;OUt9kyt+v8kM|<{I
zIyy5D-6rU+%SLL#@YyT-x{F^mR@l1kJP-CwGEIen(_^EFs~!aO?fhmP`84O<$bd6b
zozXVSD_9JACTbIs-$@>k$83X+0lqWGS@2!*G{ZnA+mYmpn#n71w;rbdEhKizylA>_
zMx@ViZ17o^Z5wioybK<B(`9y+686({S#NlAG;TtcgFgWhA6R-TGwW2%xanw*RK0hH
z?Vcg(1Qg2i3)j1S%5oZoyZ;f(55(v*iZt4vOKqNzFP<OV(zIHAX4?0IAD46ya1n43
aa1n43a1r?55SY$(e6qN4MIedv8R-w=i-Ai3

literal 0
HcmV?d00001

diff --git a/tests/f_convert_bmap/name b/tests/f_convert_bmap/name
new file mode 100644
index 0000000..67e0d47
--- /dev/null
+++ b/tests/f_convert_bmap/name
@@ -0,0 +1 @@
+convert blockmap file to extents file
diff --git a/tests/f_convert_bmap/script b/tests/f_convert_bmap/script
new file mode 100644
index 0000000..f6b6f62
--- /dev/null
+++ b/tests/f_convert_bmap/script
@@ -0,0 +1,117 @@
+if [ "$DESCRIPTION"x != x ]; then
+	test_description="$DESCRIPTION"
+fi
+if [ "$IMAGE"x = x ]; then
+	IMAGE=$test_dir/image.gz
+fi
+
+if [ "$FSCK_OPT"x = x ]; then
+	FSCK_OPT=-yf
+fi
+
+if [ "$SECOND_FSCK_OPT"x = x ]; then
+	SECOND_FSCK_OPT=-yf
+fi
+
+if [ "$OUT1"x = x ]; then
+	OUT1=$test_name.1.log
+fi
+
+if [ "$OUT2"x = x ]; then
+	OUT2=$test_name.2.log
+fi
+
+if [ "$EXP1"x = x ]; then
+	if [ -f $test_dir/expect.1.gz ]; then
+		EXP1=$test_name.1.tmp
+		gunzip < $test_dir/expect.1.gz > $EXP1
+	else
+		EXP1=$test_dir/expect.1
+	fi
+fi
+
+if [ "$EXP2"x = x ]; then
+	if [ -f $test_dir/expect.2.gz ]; then
+		EXP2=$test_name.2.tmp
+		gunzip < $test_dir/expect.2.gz > $EXP2
+	else
+		EXP2=$test_dir/expect.2
+	fi
+fi
+
+if [ "$SKIP_GUNZIP" != "true" ] ; then
+	gunzip < $IMAGE > $TMPFILE
+fi
+
+cp /dev/null $OUT1
+
+eval $PREP_CMD
+
+echo 'stat /a' > $TMPFILE.cmd
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE > $OUT1.new 2>&1
+rm -rf $TMPFILE.cmd
+$TUNE2FS -O extent $TMPFILE >> $OUT1.new 2>&1
+$FSCK $FSCK_OPT -E bmap2extent -N test_filesys $TMPFILE >> $OUT1.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT1.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT1.new >> $OUT1
+rm -f $OUT1.new
+
+$FSCK $SECOND_FSCK_OPT -N test_filesys $TMPFILE > $OUT2.new 2>&1 
+status=$?
+echo Exit status is $status >> $OUT2.new
+echo 'ex /a' > $TMPFILE.cmd
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE >> $OUT2.new 2>&1
+rm -rf $TMPFILE.cmd
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT2.new > $OUT2
+rm -f $OUT2.new
+
+eval $AFTER_CMD
+
+if [ "$SKIP_VERIFY" != "true" ] ; then
+	rm -f $test_name.ok $test_name.failed
+	cmp -s $OUT1 $EXP1
+	status1=$?
+	if [ "$ONE_PASS_ONLY" != "true" ]; then
+		cmp -s $OUT2 $EXP2
+		status2=$?
+	else
+		status2=0
+	fi
+	if [ "$PASS_ZERO" = "true" ]; then
+		cmp -s $test_name.0.log	$test_dir/expect.0
+		status3=$?
+	else
+		status3=0
+	fi
+
+	if [ -z "$test_description" ] ; then
+		description="$test_name"
+	else
+		description="$test_name: $test_description"
+	fi
+
+	if [ "$status1" -eq 0 -a "$status2" -eq 0 -a "$status3" -eq 0 ] ; then
+		echo "$description: ok"
+		touch $test_name.ok
+	else
+		echo "$description: failed"
+		rm -f $test_name.failed
+		if [ "$PASS_ZERO" = "true" ]; then
+			diff $DIFF_OPTS $test_dir/expect.0 \
+				$test_name.0.log >> $test_name.failed
+		fi
+		diff $DIFF_OPTS $EXP1 $OUT1 >> $test_name.failed
+		if [ "$ONE_PASS_ONLY" != "true" ]; then
+			diff $DIFF_OPTS $EXP2 $OUT2 >> $test_name.failed
+		fi
+	fi
+	rm -f tmp_expect
+fi
+
+if [ "$SKIP_CLEANUP" != "true" ] ; then
+	unset IMAGE FSCK_OPT SECOND_FSCK_OPT OUT1 OUT2 EXP1 EXP2 
+	unset SKIP_VERIFY SKIP_CLEANUP SKIP_GUNZIP ONE_PASS_ONLY PREP_CMD
+	unset DESCRIPTION SKIP_UNLINK AFTER_CMD PASS_ZERO
+fi
+
diff --git a/tests/f_convert_bmap_and_extent/expect.1 b/tests/f_convert_bmap_and_extent/expect.1
new file mode 100644
index 0000000..7af91aa
--- /dev/null
+++ b/tests/f_convert_bmap_and_extent/expect.1
@@ -0,0 +1,33 @@
+debugfs: stat /a
+Inode: 12   Type: regular    Mode:  0644   Flags: 0x0
+Generation: 1573716129    Version: 0x00000000:00000001
+User:     0   Group:     0   Size: 524288
+File ACL: 0    Directory ACL: 0
+Links: 1   Blockcount: 1030
+Fragment:  Address: 0    Number: 0    Size: 0
+ ctime: 0x5457f87a:62ae2980 -- Mon Nov  3 21:49:46 2014
+ atime: 0x5457f87a:61ba0598 -- Mon Nov  3 21:49:46 2014
+ mtime: 0x5457f87a:62ae2980 -- Mon Nov  3 21:49:46 2014
+crtime: 0x5457f87a:61ba0598 -- Mon Nov  3 21:49:46 2014
+Size of extra inode fields: 28
+BLOCKS:
+(0-11):1025-1036, (IND):24, (12-267):1037-1292, (DIND):25, (IND):41, (268-511):1293-1536
+TOTAL: 515
+
+debugfs: ex /zero
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 -     8    28              9
+ 1/ 1   1/  4     0 -     0    27 -    27      1 
+ 1/ 1   2/  4     2 -     2    29 -    29      1 
+ 1/ 1   3/  4     4 -     4    31 -    31      1 
+ 1/ 1   4/  4     6 -     6    33 -    33      1 
+Pass 1: Checking inodes, blocks, and sizes
+Pass 1E: Optimizing extent trees
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 13/128 files (15.4% non-contiguous), 574/2048 blocks
+Exit status is 1
diff --git a/tests/f_convert_bmap_and_extent/expect.2 b/tests/f_convert_bmap_and_extent/expect.2
new file mode 100644
index 0000000..73765ea
--- /dev/null
+++ b/tests/f_convert_bmap_and_extent/expect.2
@@ -0,0 +1,16 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 13/128 files (7.7% non-contiguous), 574/2048 blocks
+Exit status is 0
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  1     0 -   511  1025 -  1536    512 
+debugfs: ex /zero
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  4     0 -     0    27 -    27      1 
+ 0/ 0   2/  4     2 -     2    29 -    29      1 
+ 0/ 0   3/  4     4 -     4    31 -    31      1 
+ 0/ 0   4/  4     6 -     6    33 -    33      1 
diff --git a/tests/f_convert_bmap_and_extent/image.gz b/tests/f_convert_bmap_and_extent/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..916b493c710843030b453b548be0b6adcd12c48d
GIT binary patch
literal 3657
zcmeIyTToMX9tUvLscfwayK5@~0__Zy*%bt#ks1VQS!7xUxg-RLBtct56D}f#m|%je
z4(XO+RzbXANW}$m;UwG+mq0*^Ra<Uj&B>8)$s$(poDBi>PvC?kJG0$K-}i;z!}sAg
zzo*Z9=jT`PI~P@THFe`A^VGW?Il$%alCvc2Nobha)|k?9a!YJ>-XH&z5>s=9R_{gH
z=JkvJRiWZw+oAU_2l@M#P_IDM(l-Cwf0$Wym}>Wve?8-$k`pE4^;@Q`4(o`}>MR|w
z4HS&pOc6U=DwO;|Ui{Y0uAl7Z{hSrj`|?qOM|w{E%;}<nr@|MWujj^gdVW`FE;c_}
zIbB3d{t)XSn(BDRFBkc`_wl`deKYf>dUe8&ztkCUOE~=ZiD4akDn8#K%q?F0s3_lQ
z2o$Nhn0rf1-oZM7_`+f1(%mltm^$$DpoK@w-_Hx5y5y`Om3Xgu$Yh!L>f_nNS!{^u
zEs;3KFFan7jlSFNs`8ol-RSb)IUh05+S;l+BnaR4(%I>l__bIqn%)ukLwTuVpD-kp
zeCOr;wfXt)Jhzn&c0J3&<`1?X9~*yuK11x+W?JlA%8pyk*HWd?Ctp39w7b`42XEme
zJjvhZ(8xa7>nMx8|21#J4(ItL_qWC52Z6{_mbONb&FP`{)8rp;8t6SVMJflT>kM)d
z2htva7`R@`wbC^PO^H5|bHUiMPg%*S(x&Cn!`SV{z6FL=VKZ!pv!^r*tRsj4PKf;8
zH{D(7ipF5K;k>-G3Dga=7+Zar5_z2AFpwL2d>InOL&F}>ZG?+eH^7SEb2BrsIRL4`
z{|v{OGZG;kb`s{9DK>)$F7-)D)a2;Pz_5VkLv=+zBB|;aV#85V?+g75xFyJs)LfGH
zfHlEEpN!z<J8Y&8BUt$r_9HjfC+#5`k2Qcc0bf_o%+M&)P!0xjKbNMz#<_qH#7L<H
z%~f~;cp-?_!PB$?4F&T7qhxqG)uMiay%8kpx<=(z#D)(Op9x!{sD5gRJ_9@^(uIAa
z>O}p1@Do85YDb|lJs*q{nZo}5Bvx}Uat&W2z7)cH<nQ721f7w|(!^ulV1%Fvd;2NZ
z4B22x+ozUW5G(Eq28m>Qw<skVk|9^QsrEim@<1~SK>)686j4w^0nP(9ty-;#*8Aa!
z+!%Y0BPHCx!V7@a%R5S(=(+U}yJHvf{ANl8@(liKPzHrVPta8Ud*ji|q|W7$sLk6w
zm(RD%-!uk$h3giPML{ZO>#osvLT<4*_e$N}=EUMH6+zcoyb#fN%Mu@L`(Gh{Yo)WY
zp>gZxcr+Bw4N4nAx4_3B>TdKeFdL$&P%rp6l*&`V94WnPfr%poSE9|yOY(Lb3s3S!
zcX2hwj^p~{Fn_9dfntS%*h5;@)Ig^a;?xR0i)`_wx~s+d7|w)I-Kp`xj>2C>_W34D
znx$AWTokGGrAQzKRtR&<v@m=LTr}$H(i4>dD1|LCw%5^1kSA=5v1u8yvz_Gc>zHHm
zP52GaOqRQ;W7xFFUN>lvT|A-~#P@?q0n;bB2ww$%BeyZr<B&eGmPvi6ufU&@)t5Ba
z^znE$cqNF{wJ}pQYE;j`FOpkdpg0CVIax8C9)Q+jBcMjW(LvM9CVd1Z<+63EX+{(B
z64VI_q{$YQ6}<t<h%BKzD$`HVj@<#R#91jrCNILOfPy$9Jt9-Q#@ay-aZY%<Kb@rv
zMpwaAg3U<sFvJ2_+uhH+rVqzz0YnrFoAy9MSTC2)NH(b~s5>SGcZfp!KSlIt)oxr#
z@a^@Yi~*$zN4W|1Mm1zISTHM>!C;ljqYW~woG7w4<Tno(Oqhj4n++bdxsUW5wcdE|
zNcD0lnj5jBv~2cN@7$gf&uYGy)s*e@>VHsZbe^zt+{$OMTRL&jxt#}gZ{S_z$GMjW
zFl)~Z*`1HF#wS(_EN==HocWFe&y&j{9`FZeWg|{x$uysZ)CPtNS9*^HF^2S~Of8u~
ztlj!=;S?xSHThsx=A2uJ=5{;11NUApshAzOeD7G5lfM@J%i4(q<xixB@~&WJy#5La
zZbaV9^n}`OBHNp1tMA2@OdFTSCgz8Q<-eC~5aX<~foJ1B<#gL|tL^#PYM4joDOHo(
zu7%p$3bgXf^#i9X9-0lK+d@x<6xRJ?;?;_?$o|_?eS4mo3+X#Nw`i17Hg7}Cndd?Z
pk+F}bNN+!0->wU+3#<#Q3#<#Q3;h2DCVyPpWW8%MxVU`i@*iE}y4(N&

literal 0
HcmV?d00001

diff --git a/tests/f_convert_bmap_and_extent/name b/tests/f_convert_bmap_and_extent/name
new file mode 100644
index 0000000..c9394c6
--- /dev/null
+++ b/tests/f_convert_bmap_and_extent/name
@@ -0,0 +1 @@
+convert blockmap and extents files to extents files
diff --git a/tests/f_convert_bmap_and_extent/script b/tests/f_convert_bmap_and_extent/script
new file mode 100644
index 0000000..203ab25
--- /dev/null
+++ b/tests/f_convert_bmap_and_extent/script
@@ -0,0 +1,119 @@
+if [ "$DESCRIPTION"x != x ]; then
+	test_description="$DESCRIPTION"
+fi
+if [ "$IMAGE"x = x ]; then
+	IMAGE=$test_dir/image.gz
+fi
+
+if [ "$FSCK_OPT"x = x ]; then
+	FSCK_OPT=-yf
+fi
+
+if [ "$SECOND_FSCK_OPT"x = x ]; then
+	SECOND_FSCK_OPT=-yf
+fi
+
+if [ "$OUT1"x = x ]; then
+	OUT1=$test_name.1.log
+fi
+
+if [ "$OUT2"x = x ]; then
+	OUT2=$test_name.2.log
+fi
+
+if [ "$EXP1"x = x ]; then
+	if [ -f $test_dir/expect.1.gz ]; then
+		EXP1=$test_name.1.tmp
+		gunzip < $test_dir/expect.1.gz > $EXP1
+	else
+		EXP1=$test_dir/expect.1
+	fi
+fi
+
+if [ "$EXP2"x = x ]; then
+	if [ -f $test_dir/expect.2.gz ]; then
+		EXP2=$test_name.2.tmp
+		gunzip < $test_dir/expect.2.gz > $EXP2
+	else
+		EXP2=$test_dir/expect.2
+	fi
+fi
+
+if [ "$SKIP_GUNZIP" != "true" ] ; then
+	gunzip < $IMAGE > $TMPFILE
+fi
+
+cp /dev/null $OUT1
+
+eval $PREP_CMD
+
+echo 'stat /a' > $TMPFILE.cmd
+echo 'ex /zero' >> $TMPFILE.cmd
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE > $OUT1.new 2>&1
+rm -rf $TMPFILE.cmd
+$TUNE2FS -O extent $TMPFILE >> $OUT1.new 2>&1
+$FSCK $FSCK_OPT -E bmap2extent -N test_filesys $TMPFILE >> $OUT1.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT1.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT1.new >> $OUT1
+rm -f $OUT1.new
+
+$FSCK $SECOND_FSCK_OPT -N test_filesys $TMPFILE > $OUT2.new 2>&1 
+status=$?
+echo Exit status is $status >> $OUT2.new
+echo 'ex /a' > $TMPFILE.cmd
+echo 'ex /zero' >> $TMPFILE.cmd
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE >> $OUT2.new 2>&1
+rm -rf $TMPFILE.cmd
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT2.new > $OUT2
+rm -f $OUT2.new
+
+eval $AFTER_CMD
+
+if [ "$SKIP_VERIFY" != "true" ] ; then
+	rm -f $test_name.ok $test_name.failed
+	cmp -s $OUT1 $EXP1
+	status1=$?
+	if [ "$ONE_PASS_ONLY" != "true" ]; then
+		cmp -s $OUT2 $EXP2
+		status2=$?
+	else
+		status2=0
+	fi
+	if [ "$PASS_ZERO" = "true" ]; then
+		cmp -s $test_name.0.log	$test_dir/expect.0
+		status3=$?
+	else
+		status3=0
+	fi
+
+	if [ -z "$test_description" ] ; then
+		description="$test_name"
+	else
+		description="$test_name: $test_description"
+	fi
+
+	if [ "$status1" -eq 0 -a "$status2" -eq 0 -a "$status3" -eq 0 ] ; then
+		echo "$description: ok"
+		touch $test_name.ok
+	else
+		echo "$description: failed"
+		rm -f $test_name.failed
+		if [ "$PASS_ZERO" = "true" ]; then
+			diff $DIFF_OPTS $test_dir/expect.0 \
+				$test_name.0.log >> $test_name.failed
+		fi
+		diff $DIFF_OPTS $EXP1 $OUT1 >> $test_name.failed
+		if [ "$ONE_PASS_ONLY" != "true" ]; then
+			diff $DIFF_OPTS $EXP2 $OUT2 >> $test_name.failed
+		fi
+	fi
+	rm -f tmp_expect
+fi
+
+if [ "$SKIP_CLEANUP" != "true" ] ; then
+	unset IMAGE FSCK_OPT SECOND_FSCK_OPT OUT1 OUT2 EXP1 EXP2 
+	unset SKIP_VERIFY SKIP_CLEANUP SKIP_GUNZIP ONE_PASS_ONLY PREP_CMD
+	unset DESCRIPTION SKIP_UNLINK AFTER_CMD PASS_ZERO
+fi
+
diff --git a/tests/f_extent_too_deep/expect.1 b/tests/f_extent_too_deep/expect.1
new file mode 100644
index 0000000..a595482
--- /dev/null
+++ b/tests/f_extent_too_deep/expect.1
@@ -0,0 +1,23 @@
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+ 0/ 7   1/  1     0 -     0    12              1
+ 1/ 7   1/  1     0 -     0    13              1
+ 2/ 7   1/  1     0 -     0    14              1
+ 3/ 7   1/  1     0 -     0    15              1
+ 4/ 7   1/  1     0 -     0    16              1
+ 5/ 7   1/  1     0 -     0    17              1
+ 6/ 7   1/  1     0 -     0     9              1
+ 7/ 7   1/  1     0 -     0    10 -    10      1 
+Pass 1: Checking inodes, blocks, and sizes
+Inode 12 extent tree could be more shallow (7; could be <= 4)
+Fix? yes
+
+Pass 1E: Optimizing extent trees
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 12/128 files (0.0% non-contiguous), 19/512 blocks
+Exit status is 1
diff --git a/tests/f_extent_too_deep/expect.2 b/tests/f_extent_too_deep/expect.2
new file mode 100644
index 0000000..a1d28b1
--- /dev/null
+++ b/tests/f_extent_too_deep/expect.2
@@ -0,0 +1,10 @@
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 12/128 files (0.0% non-contiguous), 19/512 blocks
+Exit status is 0
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  1     0 -     0    10 -    10      1 
diff --git a/tests/f_extent_too_deep/image.gz b/tests/f_extent_too_deep/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..0f5adff562c7f45f275a4401344e784b6c61ef0b
GIT binary patch
literal 2592
zcmb2|=3wx?6duCF{PymCf8jtGh7aG@g)R_rT-?mre9>1}CPu>N$ctAhoXss6N_=iF
z5@Jgn)7j0{g?;~U$MjrWSXgN2BJ`haZVz)`*522$TUX9#P_RGs-1*+#-}m|t)ke$R
z5APFT+7Par(yh|6IP<$^TY&yHb=Pd!4HL>YcLz^?KYf{6a)z0HeXpXe&6y>)!{oHZ
z{w4_aEWdmEi{rbce>>ZI-7<e&n#^wgT>k9*+V6{hzqYV7kN<!F`meNSFQ+Q6*V|Y3
z^T*9EtryRSC(q!xs&fB&rR?lflfNCSbM%+z9`NR8Xvm!~b$|P%vOS^yuV3DA=UzC!
zdV2~70|P_Z{%7a+eb~FD@iQX>LxZgT@4xk?M^bd2d}j)Ap48L(^!o=nzNt^b-j@Bn
z(tB&}&(}Bi3E1%+uQ+>K4XAm;YM!#Q|M&Ag|6C8`EjZre0W^eR#(!oY`RqTC0D%K#
zt8|VVNY)>}nWdV0b3e-(v5!&LK$gwTUKeJq&J9t;uwnPp@BiQDA9oI70*csL0e#oU
z@4*RVe|hm||DC?}r7S>pRms2jtmDcsHU76gI%<N<lAHVgc$tlYDp1LV;#slvAMS%y
z_mw*UMaTsa{WFiQJUMmS^mV^a|12?_6#qZx;wRgs7w*6M?d&ss>QjkRvd@oR6O=Ze
zwT(YrEiNVZY2}A3mTiX){VlmTBO>)r|Ir`c-?Qz0^ke$R^}nA_NN3&Vro6gdEo}Aw
zNd;?G)F0CN>c2B%cHsY<=imQ-{{3@@$#3_+;+waWT&=Ht`+1dYa@5urll#5~9k0rL
z`R+&5tNm+I=hrV419@x|jE2By2#kinXb6mkz-S22A_Nlt)-hg8%bdf&puhkC@B%oQ

literal 0
HcmV?d00001

diff --git a/tests/f_extent_too_deep/name b/tests/f_extent_too_deep/name
new file mode 100644
index 0000000..7e8654a
--- /dev/null
+++ b/tests/f_extent_too_deep/name
@@ -0,0 +1 @@
+extent tree is deeper than it needs to be
diff --git a/tests/f_extent_too_deep/script b/tests/f_extent_too_deep/script
new file mode 100644
index 0000000..ee18438
--- /dev/null
+++ b/tests/f_extent_too_deep/script
@@ -0,0 +1,118 @@
+if [ "$DESCRIPTION"x != x ]; then
+	test_description="$DESCRIPTION"
+fi
+if [ "$IMAGE"x = x ]; then
+	IMAGE=$test_dir/image.gz
+fi
+
+if [ "$FSCK_OPT"x = x ]; then
+	FSCK_OPT=-yf
+fi
+
+if [ "$SECOND_FSCK_OPT"x = x ]; then
+	SECOND_FSCK_OPT=-yf
+fi
+
+if [ "$OUT1"x = x ]; then
+	OUT1=$test_name.1.log
+fi
+
+if [ "$OUT2"x = x ]; then
+	OUT2=$test_name.2.log
+fi
+
+if [ "$EXP1"x = x ]; then
+	if [ -f $test_dir/expect.1.gz ]; then
+		EXP1=$test_name.1.tmp
+		gunzip < $test_dir/expect.1.gz > $EXP1
+	else
+		EXP1=$test_dir/expect.1
+	fi
+fi
+
+if [ "$EXP2"x = x ]; then
+	if [ -f $test_dir/expect.2.gz ]; then
+		EXP2=$test_name.2.tmp
+		gunzip < $test_dir/expect.2.gz > $EXP2
+	else
+		EXP2=$test_dir/expect.2
+	fi
+fi
+
+if [ "$SKIP_GUNZIP" != "true" ] ; then
+	gunzip < $IMAGE > $TMPFILE
+fi
+
+cp /dev/null $OUT1
+
+eval $PREP_CMD
+
+echo 'ex /a' > $TMPFILE.cmd
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE > $OUT1.new 2>&1
+rm -rf $TMPFILE.cmd
+$FSCK $FSCK_OPT  -N test_filesys $TMPFILE >> $OUT1.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT1.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT1.new >> $OUT1
+rm -f $OUT1.new
+
+if [ "$ONE_PASS_ONLY" != "true" ]; then
+	$FSCK $SECOND_FSCK_OPT -N test_filesys $TMPFILE > $OUT2.new 2>&1 
+	status=$?
+	echo Exit status is $status >> $OUT2.new
+	echo 'ex /a' > $TMPFILE.cmd
+	$DEBUGFS -f $TMPFILE.cmd $TMPFILE >> $OUT2.new 2>&1
+	rm -rf $TMPFILE.cmd
+	sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT2.new > $OUT2
+	rm -f $OUT2.new
+fi
+
+eval $AFTER_CMD
+
+if [ "$SKIP_VERIFY" != "true" ] ; then
+	rm -f $test_name.ok $test_name.failed
+	cmp -s $OUT1 $EXP1
+	status1=$?
+	if [ "$ONE_PASS_ONLY" != "true" ]; then
+		cmp -s $OUT2 $EXP2
+		status2=$?
+	else
+		status2=0
+	fi
+	if [ "$PASS_ZERO" = "true" ]; then
+		cmp -s $test_name.0.log	$test_dir/expect.0
+		status3=$?
+	else
+		status3=0
+	fi
+
+	if [ -z "$test_description" ] ; then
+		description="$test_name"
+	else
+		description="$test_name: $test_description"
+	fi
+
+	if [ "$status1" -eq 0 -a "$status2" -eq 0 -a "$status3" -eq 0 ] ; then
+		echo "$description: ok"
+		touch $test_name.ok
+	else
+		echo "$description: failed"
+		rm -f $test_name.failed
+		if [ "$PASS_ZERO" = "true" ]; then
+			diff $DIFF_OPTS $test_dir/expect.0 \
+				$test_name.0.log >> $test_name.failed
+		fi
+		diff $DIFF_OPTS $EXP1 $OUT1 >> $test_name.failed
+		if [ "$ONE_PASS_ONLY" != "true" ]; then
+			diff $DIFF_OPTS $EXP2 $OUT2 >> $test_name.failed
+		fi
+	fi
+	rm -f tmp_expect
+fi
+
+if [ "$SKIP_CLEANUP" != "true" ] ; then
+	unset IMAGE FSCK_OPT SECOND_FSCK_OPT OUT1 OUT2 EXP1 EXP2 
+	unset SKIP_VERIFY SKIP_CLEANUP SKIP_GUNZIP ONE_PASS_ONLY PREP_CMD
+	unset DESCRIPTION SKIP_UNLINK AFTER_CMD PASS_ZERO
+fi
+
diff --git a/tests/f_opt_extent/expect b/tests/f_opt_extent/expect
new file mode 100644
index 0000000..6d4863b
--- /dev/null
+++ b/tests/f_opt_extent/expect
@@ -0,0 +1,55 @@
+tune2fs metadata_csum test
+Creating filesystem with 524288 1k blocks and 65536 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
+
+Allocating group tables:      \b\b\b\b\bdone                            
+Writing inode tables:      \b\b\b\b\bdone                            
+Creating journal (16384 blocks): done
+Creating 477 huge file(s) with 1024 blocks each: done
+Writing superblocks and filesystem accounting information:      \b\b\b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+Exit status is 0
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 3A: Optimizing directories
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+
+
+Change in FS metadata:
+@@ -10,7 +10,7 @@
+ Inode count:              65536
+ Block count:              524288
+ Reserved block count:     26214
+-Free blocks:              570
++Free blocks:              567
+ Free inodes:              65047
+ First block:              1
+ Block size:               1024
+@@ -47,8 +47,8 @@
+   Block bitmap at 262 (+261)
+   Inode bitmap at 278 (+277)
+   Inode table at 294-549 (+293)
+-  21 free blocks, 535 free inodes, 3 directories, 535 unused inodes
+-  Free blocks: 4414-4434
++  18 free blocks, 535 free inodes, 3 directories, 535 unused inodes
++  Free blocks: 4417-4434
+   Free inodes: 490-1024
+ Group 1: (Blocks 8193-16384) [INODE_UNINIT]
+   Backup superblock at 8193, Group descriptors at 8194-8197
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+Exit status is 0
diff --git a/tests/f_opt_extent/name b/tests/f_opt_extent/name
new file mode 100644
index 0000000..7d4389c
--- /dev/null
+++ b/tests/f_opt_extent/name
@@ -0,0 +1 @@
+optimize extent tree
diff --git a/tests/f_opt_extent/script b/tests/f_opt_extent/script
new file mode 100644
index 0000000..2da5e91
--- /dev/null
+++ b/tests/f_opt_extent/script
@@ -0,0 +1,64 @@
+FSCK_OPT=-fn
+OUT=$test_name.log
+EXP=$test_dir/expect
+CONF=$TMPFILE.conf
+
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode,64bit,metadata_csum
+		blocksize = 1024
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /xyz
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo "tune2fs metadata_csum test" > $OUT
+
+MKE2FS_CONFIG=$CONF $MKE2FS -F -T ext4h $TMPFILE 524288 >> $OUT 2>&1
+rm -rf $CONF
+
+# dump and check
+$DUMPE2FS $TMPFILE 2> /dev/null | grep '^Group 0:' -B99 -A20 | sed -f $cmd_dir/filter.sed > $OUT.before
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# check
+$FSCK -fyD -N test_filesys $TMPFILE >> $OUT 2>&1
+
+# dump and check
+$DUMPE2FS $TMPFILE 2> /dev/null | grep '^Group 0:' -B99 -A20 | sed -f $cmd_dir/filter.sed > $OUT.after
+echo "Change in FS metadata:" >> $OUT
+diff -u $OUT.before $OUT.after | tail -n +3 >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+rm $TMPFILE $OUT.before $OUT.after
+
+#
+# Do the verification
+#
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" -e 's/test_filesys:.*//g' < $OUT > $OUT.new
+mv $OUT.new $OUT
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+unset IMAGE FSCK_OPT OUT EXP CONF
diff --git a/tests/f_opt_extent_ext3/expect b/tests/f_opt_extent_ext3/expect
new file mode 100644
index 0000000..1761471
--- /dev/null
+++ b/tests/f_opt_extent_ext3/expect
@@ -0,0 +1,44 @@
+rebuild extent metadata_csum test
+Creating filesystem with 524288 1k blocks and 65536 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
+
+Allocating group tables:      \b\b\b\b\bdone                            
+Writing inode tables:      \b\b\b\b\bdone                            
+Creating journal (16384 blocks): done
+mke2fs: Operation not supported for inodes containing extents while creating huge files
+Writing superblocks and filesystem accounting information:      \b\b\b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+Exit status is 0
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 3A: Optimizing directories
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+
+
+Change in FS metadata:
+@@ -2,7 +2,7 @@
+ Last mounted on:          <not available>
+ Filesystem magic number:  0xEF53
+ Filesystem revision #:    1 (dynamic)
+-Filesystem features:      has_journal ext_attr resize_inode dir_index filetype sparse_super large_file huge_file dir_nlink
++Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent sparse_super large_file huge_file dir_nlink
+ Default mount options:    user_xattr acl
+ Filesystem state:         clean
+ Errors behavior:          Continue
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+
+Exit status is 0
diff --git a/tests/f_opt_extent_ext3/name b/tests/f_opt_extent_ext3/name
new file mode 100644
index 0000000..b369685
--- /dev/null
+++ b/tests/f_opt_extent_ext3/name
@@ -0,0 +1 @@
+convert ext3 to extent tree
diff --git a/tests/f_opt_extent_ext3/script b/tests/f_opt_extent_ext3/script
new file mode 100644
index 0000000..931eae7
--- /dev/null
+++ b/tests/f_opt_extent_ext3/script
@@ -0,0 +1,65 @@
+FSCK_OPT=-fn
+OUT=$test_name.log
+EXP=$test_dir/expect
+CONF=$TMPFILE.conf
+
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,^extent,huge_file,^flex_bg,^uninit_bg,dir_nlink,^extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode,^64bit,^metadata_csum
+		blocksize = 1024
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		num_hugefiles = 100
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo "rebuild extent metadata_csum test" > $OUT
+
+MKE2FS_CONFIG=$CONF $MKE2FS -F -T ext4h $TMPFILE 524288 >> $OUT 2>&1
+rm -rf $CONF
+
+# dump and check
+$DUMPE2FS $TMPFILE 2> /dev/null | grep '^Group 0:' -B99 -A20 | sed -f $cmd_dir/filter.sed > $OUT.before
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# check
+$FSCK -fyD -N test_filesys -E bmap2extent $TMPFILE >> $OUT 2>&1
+
+# dump and check
+$DUMPE2FS $TMPFILE 2> /dev/null | grep '^Group 0:' -B99 -A20 | sed -f $cmd_dir/filter.sed > $OUT.after
+echo "Change in FS metadata:" >> $OUT
+diff -u $OUT.before $OUT.after | tail -n +3 >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+rm $TMPFILE $OUT.before $OUT.after
+
+#
+# Do the verification
+#
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" -e 's/test_filesys:.*//g' < $OUT > $OUT.new
+mv $OUT.new $OUT
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+unset IMAGE FSCK_OPT OUT EXP CONF

[PATCH 29/54] undo-io: add new calls to and speed up the undo io manager

[Darrick J. Wong]

Implement pass-through calls for discard, zero-out, and readahead in
the IO manager so that we can take advantage of any underlying
support.

Furthermore, improve tdb write-out speed by disabling locking and only
fsyncing at the end -- we don't care about locking because having
multiple writers to the undo file will produce an undo database full
of garbage blocks; and we only need to fsync at the end because if we
fail before the end, our undo file will lack the necessary superblock
data that e2undo requires to do replay safely.  Without this, we call
fsync four times per tdb update(!)  This reduces the overhead of using
undo_io while converting a 2TB FS to metadata_csum from 3+ hours to 55
minutes.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/tdb.c     |   10 ++++++
 lib/ext2fs/tdb.h     |    2 +
 lib/ext2fs/undo_io.c |   87 +++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 97 insertions(+), 2 deletions(-)


diff --git a/lib/ext2fs/tdb.c b/lib/ext2fs/tdb.c
index 1d97685..7317288 100644
--- a/lib/ext2fs/tdb.c
+++ b/lib/ext2fs/tdb.c
@@ -4142,3 +4142,13 @@ int tdb_reopen_all(int parent_longlived)
 
 	return 0;
 }
+
+/**
+ * Flush a database file from the page cache.
+ **/
+int tdb_flush(struct tdb_context *tdb)
+{
+	if (tdb->fd != -1)
+		return fsync(tdb->fd);
+	return 0;
+}
diff --git a/lib/ext2fs/tdb.h b/lib/ext2fs/tdb.h
index 732ef0e..6a4086c 100644
--- a/lib/ext2fs/tdb.h
+++ b/lib/ext2fs/tdb.h
@@ -129,6 +129,7 @@ typedef struct TDB_DATA {
 #define tdb_lockall_nonblock ext2fs_tdb_lockall_nonblock
 #define tdb_lockall_read_nonblock ext2fs_tdb_lockall_read_nonblock
 #define tdb_lockall_unmark ext2fs_tdb_lockall_unmark
+#define tdb_flush ext2fs_tdb_flush
 
 /* this is the context structure that is returned from a db open */
 typedef struct tdb_context TDB_CONTEXT;
@@ -191,6 +192,7 @@ size_t tdb_map_size(struct tdb_context *tdb);
 int tdb_get_flags(struct tdb_context *tdb);
 void tdb_enable_seqnum(struct tdb_context *tdb);
 void tdb_increment_seqnum_nonblock(struct tdb_context *tdb);
+int tdb_flush(struct tdb_context *tdb);
 
 /* Low level locking functions: use with care */
 int tdb_chainlock(struct tdb_context *tdb, TDB_DATA key);
diff --git a/lib/ext2fs/undo_io.c b/lib/ext2fs/undo_io.c
index d6beb02..94317cb 100644
--- a/lib/ext2fs/undo_io.c
+++ b/lib/ext2fs/undo_io.c
@@ -37,6 +37,7 @@
 #if HAVE_SYS_RESOURCE_H
 #include <sys/resource.h>
 #endif
+#include <limits.h>
 
 #include "tdb.h"
 
@@ -354,8 +355,12 @@ static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 		data->real = 0;
 	}
 
+	if (data->real)
+		io->flags = (io->flags & ~CHANNEL_FLAGS_DISCARD_ZEROES) |
+			    (data->real->flags & CHANNEL_FLAGS_DISCARD_ZEROES);
+
 	/* setup the tdb file */
-	data->tdb = tdb_open(tdb_file, 0, TDB_CLEAR_IF_FIRST,
+	data->tdb = tdb_open(tdb_file, 0, TDB_CLEAR_IF_FIRST | TDB_NOLOCK | TDB_NOSYNC,
 			     O_RDWR | O_CREAT | O_TRUNC | O_EXCL, 0600);
 	if (!data->tdb) {
 		retval = errno;
@@ -399,8 +404,10 @@ static errcode_t undo_close(io_channel channel)
 		return retval;
 	if (data->real)
 		retval = io_channel_close(data->real);
-	if (data->tdb)
+	if (data->tdb) {
+		tdb_flush(data->tdb);
 		tdb_close(data->tdb);
+	}
 	ext2fs_free_mem(&channel->private_data);
 	if (channel->name)
 		ext2fs_free_mem(&channel->name);
@@ -510,6 +517,77 @@ static errcode_t undo_write_byte(io_channel channel, unsigned long offset,
 	return retval;
 }
 
+static errcode_t undo_discard(io_channel channel, unsigned long long block,
+			      unsigned long long count)
+{
+	struct undo_private_data *data;
+	errcode_t	retval = 0;
+	int icount;
+
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+	data = (struct undo_private_data *) channel->private_data;
+	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
+
+	if (count > INT_MAX)
+		return EXT2_ET_UNIMPLEMENTED;
+	icount = count;
+
+	/*
+	 * First write the existing content into database
+	 */
+	retval = undo_write_tdb(channel, block, icount);
+	if (retval)
+		return retval;
+	if (data->real)
+		retval = io_channel_discard(data->real, block, count);
+
+	return retval;
+}
+
+static errcode_t undo_zeroout(io_channel channel, unsigned long long block,
+			      unsigned long long count)
+{
+	struct undo_private_data *data;
+	errcode_t	retval = 0;
+	int icount;
+
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+	data = (struct undo_private_data *) channel->private_data;
+	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
+
+	if (count > INT_MAX)
+		return EXT2_ET_UNIMPLEMENTED;
+	icount = count;
+
+	/*
+	 * First write the existing content into database
+	 */
+	retval = undo_write_tdb(channel, block, icount);
+	if (retval)
+		return retval;
+	if (data->real)
+		retval = io_channel_zeroout(data->real, block, count);
+
+	return retval;
+}
+
+static errcode_t undo_cache_readahead(io_channel channel,
+				      unsigned long long block,
+				      unsigned long long count)
+{
+	struct undo_private_data *data;
+	errcode_t	retval = 0;
+
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+	data = (struct undo_private_data *) channel->private_data;
+	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
+
+	if (data->real)
+		retval = io_channel_cache_readahead(data->real, block, count);
+
+	return retval;
+}
+
 /*
  * Flush data buffers to disk.
  */
@@ -522,6 +600,8 @@ static errcode_t undo_flush(io_channel channel)
 	data = (struct undo_private_data *) channel->private_data;
 	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
 
+	if (data->tdb)
+		tdb_flush(data->tdb);
 	if (data->real)
 		retval = io_channel_flush(data->real);
 
@@ -601,6 +681,9 @@ static struct struct_io_manager struct_undo_manager = {
 	.get_stats	= undo_get_stats,
 	.read_blk64	= undo_read_blk64,
 	.write_blk64	= undo_write_blk64,
+	.discard	= undo_discard,
+	.zeroout	= undo_zeroout,
+	.cache_readahead	= undo_cache_readahead,
 };
 
 io_manager undo_io_manager = &struct_undo_manager;

[PATCH 30/54] undo-io: be more flexible about setting block size

[Darrick J. Wong]

Most of the e2fsprogs utilities set the IO block size multiple times
(once to 1k to read the superblock, then again to set the real block
size if we find a real superblock).  Unfortunately, the undo IO
manager only lets the block size be set once.  For the non-mke2fs
utilities we'd rather catch the real block size and use that.  mke2fs
of course wants to use a really large block size since it's probably
writing a lot of data.

Therefore, if we haven't written any blocks to the undo file, it's
perfectly fine to allow block size changes.  For mke2fs, we'll modify
the IO channel option that lets us set the huge size to lock that
in place.  This greatly reduces index overhead for undo files for
e2fsck/tune2fs/resize2fs while continuing the practice of reducing
it even more for mke2fs.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/undo_io.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)


diff --git a/lib/ext2fs/undo_io.c b/lib/ext2fs/undo_io.c
index 94317cb..70b90d5 100644
--- a/lib/ext2fs/undo_io.c
+++ b/lib/ext2fs/undo_io.c
@@ -265,7 +265,7 @@ static errcode_t undo_write_tdb(io_channel channel,
 		       tdb_data.dptr,
 		       tdb_data.dsize);
 #endif
-		if (!data->tdb_written) {
+		if (data->tdb_written != 1) {
 			data->tdb_written = 1;
 			/* Write the blocksize to tdb file */
 			retval = write_block_size(data->tdb,
@@ -430,9 +430,8 @@ static errcode_t undo_set_blksize(io_channel channel, int blksize)
 	/*
 	 * Set the block size used for tdb
 	 */
-	if (!data->tdb_data_size) {
+	if (!data->tdb_data_size || !data->tdb_written)
 		data->tdb_data_size = blksize;
-	}
 	channel->block_size = blksize;
 	return retval;
 }
@@ -628,6 +627,7 @@ static errcode_t undo_set_option(io_channel channel, const char *option,
 		if (*end)
 			return EXT2_ET_INVALID_ARGUMENT;
 		if (!data->tdb_data_size || !data->tdb_written) {
+			data->tdb_written = -1;
 			data->tdb_data_size = tmp;
 		}
 		return 0;

[PATCH 31/54] undo-io: use a bitmap to track what we've already written

[Darrick J. Wong]

It's really inefficient to (ab)use the TDB key store as a bitmap to
find out if we've already written a block to the undo file, because
the tdb code is reads the database key btree disk blocks for *every*
query.  Changing that logic to a bitmap reduces overhead by a large
margin -- the overhead of using undo_io while converting a 2TB FS to
metadata_csum is reduced from 55 minutes to 45.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/undo_io.c |   69 +++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 51 insertions(+), 18 deletions(-)


diff --git a/lib/ext2fs/undo_io.c b/lib/ext2fs/undo_io.c
index 70b90d5..9a01e30 100644
--- a/lib/ext2fs/undo_io.c
+++ b/lib/ext2fs/undo_io.c
@@ -70,6 +70,9 @@ struct undo_private_data {
 
 	/* to support offset in unix I/O manager */
 	ext2_loff_t offset;
+
+	ext2fs_block_bitmap written_block_map;
+	struct struct_ext2_filsys fake_fs;
 };
 
 static io_manager undo_io_backing_manager;
@@ -164,6 +167,38 @@ static errcode_t write_block_size(TDB_CONTEXT *tdb, int block_size)
 	return retval;
 }
 
+static errcode_t undo_setup_tdb(struct undo_private_data *data)
+{
+	errcode_t retval;
+
+	if (data->tdb_written == 1)
+		return 0;
+
+	data->tdb_written = 1;
+
+	/* Make a bitmap to track what we've written */
+	memset(&data->fake_fs, 0, sizeof(data->fake_fs));
+	data->fake_fs.blocksize = data->tdb_data_size;
+	retval = ext2fs_alloc_generic_bmap(&data->fake_fs,
+				EXT2_ET_MAGIC_BLOCK_BITMAP64,
+				EXT2FS_BMAP64_RBTREE,
+				0, ~1ULL, ~1ULL,
+				"undo block map", &data->written_block_map);
+	if (retval)
+		return retval;
+
+	/* Write the blocksize to tdb file */
+	tdb_transaction_start(data->tdb);
+	retval = write_block_size(data->tdb,
+				  data->tdb_data_size);
+	if (retval) {
+		tdb_transaction_cancel(data->tdb);
+		return EXT2_ET_TDB_ERR_IO;
+	}
+	tdb_transaction_commit(data->tdb);
+	return 0;
+}
+
 static errcode_t undo_write_tdb(io_channel channel,
 				unsigned long long block, int count)
 
@@ -194,6 +229,10 @@ static errcode_t undo_write_tdb(io_channel channel,
 		else
 			size = count * channel->block_size;
 	}
+
+	retval = undo_setup_tdb(data);
+	if (retval)
+		return retval;
 	/*
 	 * Data is stored in tdb database as blocks of tdb_data_size size
 	 * This helps in efficient lookup further.
@@ -212,11 +251,14 @@ static errcode_t undo_write_tdb(io_channel channel,
 		/*
 		 * Check if we have the record already
 		 */
-		if (tdb_exists(data->tdb, tdb_key)) {
+		if (ext2fs_test_block_bitmap2(data->written_block_map,
+						   block_num)) {
 			/* Try the next block */
 			block_num++;
 			continue;
 		}
+		ext2fs_mark_block_bitmap2(data->written_block_map, block_num);
+
 		/*
 		 * Read one block using the backing I/O manager
 		 * The backing I/O manager block size may be
@@ -265,19 +307,7 @@ static errcode_t undo_write_tdb(io_channel channel,
 		       tdb_data.dptr,
 		       tdb_data.dsize);
 #endif
-		if (data->tdb_written != 1) {
-			data->tdb_written = 1;
-			/* Write the blocksize to tdb file */
-			retval = write_block_size(data->tdb,
-						  data->tdb_data_size);
-			if (retval) {
-				tdb_transaction_cancel(data->tdb);
-				retval = EXT2_ET_TDB_ERR_IO;
-				free(read_ptr);
-				return retval;
-			}
-		}
-		retval = tdb_store(data->tdb, tdb_key, tdb_data, TDB_INSERT);
+		retval = tdb_store(data->tdb, tdb_key, tdb_data, TDB_REPLACE);
 		if (retval == -1) {
 			/*
 			 * TDB_ERR_EXISTS cannot happen because we
@@ -345,6 +375,7 @@ static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 
 	memset(data, 0, sizeof(struct undo_private_data));
 	data->magic = EXT2_ET_MAGIC_UNIX_IO_CHANNEL;
+	data->written_block_map = NULL;
 
 	if (undo_io_backing_manager) {
 		retval = undo_io_backing_manager->open(name, flags,
@@ -390,7 +421,7 @@ cleanup:
 static errcode_t undo_close(io_channel channel)
 {
 	struct undo_private_data *data;
-	errcode_t	retval = 0;
+	errcode_t	err, retval = 0;
 
 	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
 	data = (struct undo_private_data *) channel->private_data;
@@ -399,20 +430,22 @@ static errcode_t undo_close(io_channel channel)
 	if (--channel->refcount > 0)
 		return 0;
 	/* Before closing write the file system identity */
-	retval = write_file_system_identity(channel, data->tdb);
-	if (retval)
-		return retval;
+	err = write_file_system_identity(channel, data->tdb);
 	if (data->real)
 		retval = io_channel_close(data->real);
 	if (data->tdb) {
 		tdb_flush(data->tdb);
 		tdb_close(data->tdb);
 	}
+	if (data->written_block_map)
+		ext2fs_free_generic_bitmap(data->written_block_map);
 	ext2fs_free_mem(&channel->private_data);
 	if (channel->name)
 		ext2fs_free_mem(&channel->name);
 	ext2fs_free_mem(&channel);
 
+	if (err)
+		return err;
 	return retval;
 }
 

[PATCH 32/54] e2undo: fix memory leaks and tweak the error messages somewhat

[Darrick J. Wong]

Fix memory leaks and improve the error messages to make it easier
to figure out why e2undo went wrong.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/e2undo.c |   43 ++++++++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 19 deletions(-)


diff --git a/misc/e2undo.c b/misc/e2undo.c
index a43c26f..d828d3b 100644
--- a/misc/e2undo.c
+++ b/misc/e2undo.c
@@ -49,7 +49,7 @@ static int check_filesystem(TDB_CONTEXT *tdb, io_channel channel)
 	retval = io_channel_read_blk64(channel, 1, -SUPERBLOCK_SIZE, &super);
 	if (retval) {
 		com_err(prg_name, retval,
-			"%s", _("Failed to read the file system data \n"));
+			"%s", _("while reading filesystem superblock."));
 		return retval;
 	}
 
@@ -58,16 +58,16 @@ static int check_filesystem(TDB_CONTEXT *tdb, io_channel channel)
 	tdb_data = tdb_fetch(tdb, tdb_key);
 	if (!tdb_data.dptr) {
 		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-		com_err(prg_name, retval,
-			_("Failed tdb_fetch %s\n"), tdb_errorstr(tdb));
+		com_err(prg_name, retval, "%s",
+			_("while fetching last mount time."));
 		return retval;
 	}
 
 	s_mtime = *(__u32 *)tdb_data.dptr;
+	free(tdb_data.dptr);
 	if (super.s_mtime != s_mtime) {
-
 		com_err(prg_name, 0,
-			_("The file system Mount time didn't match %u\n"),
+			_("The filesystem last mount time didn't match %u."),
 			s_mtime);
 
 		return  -1;
@@ -79,14 +79,14 @@ static int check_filesystem(TDB_CONTEXT *tdb, io_channel channel)
 	tdb_data = tdb_fetch(tdb, tdb_key);
 	if (!tdb_data.dptr) {
 		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-		com_err(prg_name, retval,
-			_("Failed tdb_fetch %s\n"), tdb_errorstr(tdb));
+		com_err(prg_name, retval, "%s", _("while fetching UUID"));
 		return retval;
 	}
 	memcpy(s_uuid, tdb_data.dptr, sizeof(s_uuid));
+	free(tdb_data.dptr);
 	if (memcmp(s_uuid, super.s_uuid, sizeof(s_uuid))) {
 		com_err(prg_name, 0, "%s",
-			_("The file system UUID didn't match \n"));
+			_("The filesystem UUID didn't match."));
 		return -1;
 	}
 
@@ -104,12 +104,12 @@ static int set_blk_size(TDB_CONTEXT *tdb, io_channel channel)
 	tdb_data = tdb_fetch(tdb, tdb_key);
 	if (!tdb_data.dptr) {
 		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-		com_err(prg_name, retval,
-			_("Failed tdb_fetch %s\n"), tdb_errorstr(tdb));
+		com_err(prg_name, retval, "%s", _("while fetching block size"));
 		return retval;
 	}
 
 	block_size = *(int *)tdb_data.dptr;
+	free(tdb_data.dptr);
 #ifdef DEBUG
 	printf("Block size %d\n", block_size);
 #endif
@@ -129,6 +129,7 @@ int main(int argc, char *argv[])
 	blk64_t  blk_num;
 	char *device_name, *tdb_file;
 	io_manager manager = unix_io_manager;
+	void *old_dptr = NULL;
 
 #ifdef ENABLE_NLS
 	setlocale(LC_MESSAGES, "");
@@ -160,20 +161,20 @@ int main(int argc, char *argv[])
 
 	if (!tdb) {
 		com_err(prg_name, errno,
-				_("Failed tdb_open %s\n"), tdb_file);
+				_("while opening undo file `%s'\n"), tdb_file);
 		exit(1);
 	}
 
 	retval = ext2fs_check_if_mounted(device_name, &mount_flags);
 	if (retval) {
 		com_err(prg_name, retval, _("Error while determining whether "
-				"%s is mounted.\n"), device_name);
+				"%s is mounted."), device_name);
 		exit(1);
 	}
 
 	if (mount_flags & EXT2_MF_MOUNTED) {
 		com_err(prg_name, retval, "%s", _("e2undo should only be run "
-						"on unmounted file system\n"));
+						"on unmounted filesystems"));
 		exit(1);
 	}
 
@@ -181,7 +182,7 @@ int main(int argc, char *argv[])
 				IO_FLAG_EXCLUSIVE | IO_FLAG_RW,  &channel);
 	if (retval) {
 		com_err(prg_name, retval,
-				_("Failed to open %s\n"), device_name);
+				_("while opening `%s'"), device_name);
 		exit(1);
 	}
 
@@ -194,30 +195,34 @@ int main(int argc, char *argv[])
 	}
 
 	for (key = tdb_firstkey(tdb); key.dptr; key = tdb_nextkey(tdb, key)) {
+		free(old_dptr);
+		old_dptr = key.dptr;
 		if (!strcmp((char *) key.dptr, (char *) mtime_key) ||
 		    !strcmp((char *) key.dptr, (char *) uuid_key) ||
 		    !strcmp((char *) key.dptr, (char *) blksize_key)) {
 			continue;
 		}
 
+		blk_num = *(blk64_t *)key.dptr;
 		data = tdb_fetch(tdb, key);
 		if (!data.dptr) {
-			com_err(prg_name, 0,
-				_("Failed tdb_fetch %s\n"), tdb_errorstr(tdb));
+			retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
+			com_err(prg_name, retval,
+				_("while fetching block %llu."), blk_num);
 			exit(1);
 		}
-		blk_num = *(blk64_t *)key.dptr;
 		printf(_("Replayed transaction of size %zd at location %llu\n"),
 							data.dsize, blk_num);
 		retval = io_channel_write_blk64(channel, blk_num,
 						-data.dsize, data.dptr);
+		free(data.dptr);
 		if (retval == -1) {
 			com_err(prg_name, retval,
-					_("Failed write %s\n"),
-					strerror(errno));
+				_("while writing block %llu."), blk_num);
 			exit(1);
 		}
 	}
+	free(old_dptr);
 	io_channel_close(channel);
 	tdb_close(tdb);
 

[PATCH 33/54] e2undo: ditch tdb file, write everything to a flat file

[Darrick J. Wong]

The existing undo file format (which is based on tdb) has many
problems.  First, its comparison of superblock fields is ineffective,
since the last mount time is only written by the kernel, not the tools
(which means that undo files can be applied out of order, thus
corrupting the filesystem); block numbers are written in CPU byte
order, which will cause silent failures if an undo file is moved from
one type of system to another; using the tdb database costs us an
enormous amount of CPU overhead to maintain the key data structure,
and finally, the tdb database is unable to deal with databases larger
than 2GB.  (Upstream tdb 1.2.12 can handle 4GB, but upgrading a 2TB FS
to 64bit,metadata_csum easily produces 2.9GB of undo files, so we
might as well move off of tdb now.)

The last problem is fatal if you want to use tune2fs to turn on
metadata checksumming, since that rewrites every block on the
filesystem, which can easily produce a many-gigabyte undo file, which
of course is unreadable and therefore the operation cannot be undone.

Therefore, rip all of that out in favor of writing to a flat file.
Old blocks are appended to a file and the index is written to the end
when we're done.  This implementation is much faster than wasting a
considerable amount of time trying to maintain a hash index, which
drops the runtime overhead of tune2fs -O metadata_csum from ~45min
to ~20 seconds on a 2TB filesystem.

I have a few reasons that factored in my decision not to repurpose the
jbd2 file format for undo files.  First, undo files are limited to
2^32 blocks (16TB) which some day might not serve us well.  Second,
the journal block size is tied to the file system block size, but
mke2fs wants to be able to back up big chunks of old device contents.
This would require large changes to the e2fsck journal replay code,
which itself is derived from the kernel jbd2 driver, which I'd rather
not destabilize.  Third, I want to require undo files to store the FS
superblock at the end of undo file creation so that e2undo can be
reasonably sure that an undo file is supposed to apply against the
given block device, and doing so would require changes to the jbd2
format.  Fourth, it didn't seem like a good idea that external
journals should resemble undo files so closely.

v2: Provide a state bit that is only set when the undo channel is
closed correctly so we can warn the user about potentially incomplete
undo files.  Straighten out the superblock handling so that undo files
won't be confused for real ext* FS images.  Record multi-block runs in
each block key to reduce overhead even further.  Support reopening an
undo file so that we can combine multiple FS operations into one
(overall smaller) transaction file, which will be easier to manage.
Flush the undo index data if the program should terminate
unexpectedly.  Update the ext4 superblock bits if errors or -f is
found to encourage fsck to do a full run the next time it's invoked.
Enable undoing the undo.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/ext2_err.et.in |    6 
 lib/ext2fs/undo_io.c      |  550 ++++++++++++++++++++++++++++++++++++--------
 misc/e2undo.8.in          |   17 +
 misc/e2undo.c             |  560 +++++++++++++++++++++++++++++++++++++--------
 4 files changed, 923 insertions(+), 210 deletions(-)


diff --git a/lib/ext2fs/ext2_err.et.in b/lib/ext2fs/ext2_err.et.in
index 790d135..894789e 100644
--- a/lib/ext2fs/ext2_err.et.in
+++ b/lib/ext2fs/ext2_err.et.in
@@ -524,4 +524,10 @@ ec	EXT2_ET_EA_BAD_VALUE_OFFSET,
 ec	EXT2_ET_JOURNAL_FLAGS_WRONG,
 	"Journal flags inconsistent"
 
+ec	EXT2_ET_UNDO_FILE_CORRUPT,
+	"Undo file corrupt"
+
+ec	EXT2_ET_UNDO_FILE_WRONG,
+	"Wrong undo file for this filesystem"
+
 	end
diff --git a/lib/ext2fs/undo_io.c b/lib/ext2fs/undo_io.c
index 9a01e30..f1c107a 100644
--- a/lib/ext2fs/undo_io.c
+++ b/lib/ext2fs/undo_io.c
@@ -39,8 +39,6 @@
 #endif
 #include <limits.h>
 
-#include "tdb.h"
-
 #include "ext2_fs.h"
 #include "ext2fs.h"
 
@@ -50,22 +48,86 @@
 #define ATTR(x)
 #endif
 
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
 /*
  * For checking structure magic numbers...
  */
 
 #define EXT2_CHECK_MAGIC(struct, code) \
 	  if ((struct)->magic != (code)) return (code)
+/*
+ * Undo file format: The file is cut up into undo_header.block_size blocks.
+ * The first block contains the header.
+ * The second block contains the superblock.
+ * There is then a repeating series of blocks as follows:
+ *   A key block, which contains undo_keys to map the following data blocks.
+ *   Data blocks
+ * (Note that there are pointers to the first key block and the sb, so this
+ * order isn't strictly necessary.)
+ */
+#define E2UNDO_MAGIC "E2UNDO02"
+#define KEYBLOCK_MAGIC 0xCADECADE
+
+#define E2UNDO_STATE_FINISHED	0x1	/* undo file is complete */
+
+#define E2UNDO_MIN_BLOCK_SIZE	1024	/* undo blocks are no less than 1KB */
+#define E2UNDO_MAX_BLOCK_SIZE	1048576	/* undo blocks are no more than 1MB */
+
+struct undo_header {
+	char magic[8];		/* "E2UNDO02" */
+	__le64 num_keys;	/* how many keys? */
+	__le64 super_offset;	/* where in the file is the superblock copy? */
+	__le64 key_offset;	/* where do the key/data block chunks start? */
+	__le32 block_size;	/* block size of the undo file */
+	__le32 fs_block_size;	/* block size of the target device */
+	__le32 sb_crc;		/* crc32c of the superblock */
+	__le32 state;		/* e2undo state flags */
+	__le32 f_compat;	/* compatible features (none so far) */
+	__le32 f_incompat;	/* incompatible features (none so far) */
+	__le32 f_rocompat;	/* ro compatible features (none so far) */
+	__u8 padding[448];	/* padding */
+	__le32 header_crc;	/* crc32c of this header (but not this field) */
+};
+
+#define E2UNDO_MAX_EXTENT_BLOCKS	512	/* max extent size, in blocks */
+
+struct undo_key {
+	__le64 fsblk;		/* where in the fs does the block go */
+	__le32 blk_crc;		/* crc32c of the block */
+	__le32 size;		/* how many bytes in this block? */
+};
+
+struct undo_key_block {
+	__le32 magic;		/* KEYBLOCK_MAGIC number */
+	__le32 crc;		/* block checksum */
+	__le64 reserved;	/* zero */
+
+	struct undo_key keys[0];	/* keys, which come immediately after */
+};
 
 struct undo_private_data {
 	int	magic;
-	TDB_CONTEXT *tdb;
-	char *tdb_file;
+
+	/* the undo file io channel */
+	io_channel undo_file;
+	blk64_t undo_blk_num;			/* next free block */
+	blk64_t key_blk_num;			/* current key block location */
+	blk64_t super_blk_num;			/* superblock location */
+	blk64_t first_key_blk;			/* first key block location */
+	struct undo_key_block *keyb;
+	size_t num_keys, keys_in_block;
 
 	/* The backing io channel */
 	io_channel real;
 
-	int tdb_data_size;
+	unsigned long long tdb_data_size;
 	int tdb_written;
 
 	/* to support offset in unix I/O manager */
@@ -73,16 +135,15 @@ struct undo_private_data {
 
 	ext2fs_block_bitmap written_block_map;
 	struct struct_ext2_filsys fake_fs;
+
+	struct undo_header hdr;
 };
+#define KEYS_PER_BLOCK(d) (((d)->tdb_data_size / sizeof(struct undo_key)) - 1)
 
 static io_manager undo_io_backing_manager;
 static char *tdb_file;
 static int actual_size;
 
-static unsigned char mtime_key[] = "filesystem MTIME";
-static unsigned char blksize_key[] = "filesystem BLKSIZE";
-static unsigned char uuid_key[] = "filesystem UUID";
-
 errcode_t set_undo_io_backing_manager(io_manager manager)
 {
 	/*
@@ -103,17 +164,34 @@ errcode_t set_undo_io_backup_file(char *file_name)
 	return 0;
 }
 
-static errcode_t write_file_system_identity(io_channel undo_channel,
-							TDB_CONTEXT *tdb)
+static errcode_t write_undo_indexes(struct undo_private_data *data)
 {
 	errcode_t retval;
 	struct ext2_super_block super;
-	TDB_DATA tdb_key, tdb_data;
-	struct undo_private_data *data;
 	io_channel channel;
-	int block_size ;
+	int block_size;
+	__u32 sb_crc, hdr_crc;
+
+	/* Spit out a key block, if there's any data */
+	if (data->keys_in_block) {
+		data->keyb->magic = ext2fs_cpu_to_le32(KEYBLOCK_MAGIC);
+		data->keyb->crc = 0;
+		data->keyb->crc = ext2fs_cpu_to_le32(
+					 ext2fs_crc32c_le(~0,
+					 (unsigned char *)data->keyb,
+					 data->tdb_data_size));
+		dbg_printf("Writing keyblock to blk %llu\n", data->key_blk_num);
+		retval = io_channel_write_blk64(data->undo_file,
+						data->key_blk_num,
+						1, data->keyb);
+		if (retval)
+			return retval;
+		memset(data->keyb, 0, data->tdb_data_size);
+		data->keys_in_block = 0;
+		data->key_blk_num = data->undo_blk_num;
+	}
 
-	data = (struct undo_private_data *) undo_channel->private_data;
+	/* Prepare superblock for write */
 	channel = data->real;
 	block_size = channel->block_size;
 
@@ -121,54 +199,45 @@ static errcode_t write_file_system_identity(io_channel undo_channel,
 	retval = io_channel_read_blk64(channel, 1, -SUPERBLOCK_SIZE, &super);
 	if (retval)
 		goto err_out;
-
-	/* Write to tdb file in the file system byte order */
-	tdb_key.dptr = mtime_key;
-	tdb_key.dsize = sizeof(mtime_key);
-	tdb_data.dptr = (unsigned char *) &(super.s_mtime);
-	tdb_data.dsize = sizeof(super.s_mtime);
-
-	retval = tdb_store(tdb, tdb_key, tdb_data, TDB_INSERT);
-	if (retval == -1) {
-		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
+	sb_crc = ext2fs_crc32c_le(~0, (unsigned char *)&super, SUPERBLOCK_SIZE);
+	super.s_magic = ~super.s_magic;
+
+	/* Write the undo header to disk. */
+	memcpy(data->hdr.magic, E2UNDO_MAGIC, sizeof(data->hdr.magic));
+	data->hdr.num_keys = ext2fs_cpu_to_le64(data->num_keys);
+	data->hdr.super_offset = ext2fs_cpu_to_le64(data->super_blk_num);
+	data->hdr.key_offset = ext2fs_cpu_to_le64(data->first_key_blk);
+	data->hdr.fs_block_size = ext2fs_cpu_to_le32(block_size);
+	data->hdr.sb_crc = ext2fs_cpu_to_le32(sb_crc);
+	hdr_crc = ext2fs_crc32c_le(~0, (unsigned char *)&data->hdr,
+				   sizeof(data->hdr) -
+				   sizeof(data->hdr.header_crc));
+	data->hdr.header_crc = ext2fs_cpu_to_le32(hdr_crc);
+	retval = io_channel_write_blk64(data->undo_file, 0,
+					-(int)sizeof(data->hdr),
+					&data->hdr);
+	if (retval)
 		goto err_out;
-	}
 
-	tdb_key.dptr = uuid_key;
-	tdb_key.dsize = sizeof(uuid_key);
-	tdb_data.dptr = (unsigned char *)&(super.s_uuid);
-	tdb_data.dsize = sizeof(super.s_uuid);
-
-	retval = tdb_store(tdb, tdb_key, tdb_data, TDB_INSERT);
-	if (retval == -1) {
-		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-	}
+	/*
+	 * Record the entire superblock (in FS byte order) so that we can't
+	 * apply e2undo files to the wrong FS or out of order.
+	 */
+	dbg_printf("Writing superblock to block %llu\n", data->super_blk_num);
+	retval = io_channel_write_blk64(data->undo_file, data->super_blk_num,
+					-SUPERBLOCK_SIZE, &super);
+	if (retval)
+		goto err_out;
 
+	retval = io_channel_flush(data->undo_file);
 err_out:
 	io_channel_set_blksize(channel, block_size);
 	return retval;
 }
 
-static errcode_t write_block_size(TDB_CONTEXT *tdb, int block_size)
-{
-	errcode_t retval;
-	TDB_DATA tdb_key, tdb_data;
-
-	tdb_key.dptr = blksize_key;
-	tdb_key.dsize = sizeof(blksize_key);
-	tdb_data.dptr = (unsigned char *)&(block_size);
-	tdb_data.dsize = sizeof(block_size);
-
-	retval = tdb_store(tdb, tdb_key, tdb_data, TDB_INSERT);
-	if (retval == -1) {
-		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-	}
-
-	return retval;
-}
-
 static errcode_t undo_setup_tdb(struct undo_private_data *data)
 {
+	int i;
 	errcode_t retval;
 
 	if (data->tdb_written == 1)
@@ -187,15 +256,33 @@ static errcode_t undo_setup_tdb(struct undo_private_data *data)
 	if (retval)
 		return retval;
 
-	/* Write the blocksize to tdb file */
-	tdb_transaction_start(data->tdb);
-	retval = write_block_size(data->tdb,
-				  data->tdb_data_size);
-	if (retval) {
-		tdb_transaction_cancel(data->tdb);
-		return EXT2_ET_TDB_ERR_IO;
+	/* Allocate key block */
+	retval = ext2fs_get_mem(data->tdb_data_size, &data->keyb);
+	if (retval)
+		return retval;
+	data->key_blk_num = data->undo_blk_num;
+
+	/* Record block size */
+	dbg_printf("Undo block size %llu\n", data->tdb_data_size);
+	dbg_printf("Keys per block %llu\n", KEYS_PER_BLOCK(data));
+	data->hdr.block_size = ext2fs_cpu_to_le32(data->tdb_data_size);
+	io_channel_set_blksize(data->undo_file, data->tdb_data_size);
+
+	/* Ensure that we have space for header blocks */
+	for (i = 0; i <= 2; i++) {
+		retval = io_channel_read_blk64(data->undo_file, i, 1,
+					       data->keyb);
+		if (retval)
+			memset(data->keyb, 0, data->tdb_data_size);
+		retval = io_channel_write_blk64(data->undo_file, i, 1,
+						data->keyb);
+		if (retval)
+			return retval;
+		retval = io_channel_flush(data->undo_file);
+		if (retval)
+			return retval;
 	}
-	tdb_transaction_commit(data->tdb);
+	memset(data->keyb, 0, data->tdb_data_size);
 	return 0;
 }
 
@@ -208,13 +295,16 @@ static errcode_t undo_write_tdb(io_channel channel,
 	errcode_t retval = 0;
 	ext2_loff_t offset;
 	struct undo_private_data *data;
-	TDB_DATA tdb_key, tdb_data;
 	unsigned char *read_ptr;
 	unsigned long long end_block;
+	unsigned long long data_size;
+	void *data_ptr;
+	struct undo_key *key;
+	__u32 blk_crc;
 
 	data = (struct undo_private_data *) channel->private_data;
 
-	if (data->tdb == NULL) {
+	if (data->undo_file == NULL) {
 		/*
 		 * Transaction database not initialized
 		 */
@@ -241,13 +331,11 @@ static errcode_t undo_write_tdb(io_channel channel,
 	 */
 	offset = (block * channel->block_size) + data->offset ;
 	block_num = offset / data->tdb_data_size;
-	end_block = (offset + size) / data->tdb_data_size;
+	end_block = (offset + size - 1) / data->tdb_data_size;
 
-	tdb_transaction_start(data->tdb);
-	while (block_num <= end_block ) {
+	while (block_num <= end_block) {
+		__u32 keysz;
 
-		tdb_key.dptr = (unsigned char *)&block_num;
-		tdb_key.dsize = sizeof(block_num);
 		/*
 		 * Check if we have the record already
 		 */
@@ -259,6 +347,22 @@ static errcode_t undo_write_tdb(io_channel channel,
 		}
 		ext2fs_mark_block_bitmap2(data->written_block_map, block_num);
 
+		/* Spit out a key block */
+		if (data->keys_in_block == KEYS_PER_BLOCK(data)) {
+			retval = write_undo_indexes(data);
+			if (retval)
+				return retval;
+			retval = io_channel_write_blk64(data->undo_file,
+							data->key_blk_num, 1,
+							data->keyb);
+			if (retval)
+				return retval;
+		}
+
+		/* Allocate new key block */
+		if (data->keys_in_block == 0)
+			data->undo_blk_num++;
+
 		/*
 		 * Read one block using the backing I/O manager
 		 * The backing I/O manager block size may be
@@ -273,7 +377,6 @@ static errcode_t undo_write_tdb(io_channel channel,
 				((offset - data->offset) % channel->block_size);
 		retval = ext2fs_get_mem(count, &read_ptr);
 		if (retval) {
-			tdb_transaction_cancel(data->tdb);
 			return retval;
 		}
 
@@ -288,41 +391,75 @@ static errcode_t undo_write_tdb(io_channel channel,
 		if (retval) {
 			if (retval != EXT2_ET_SHORT_READ) {
 				free(read_ptr);
-				tdb_transaction_cancel(data->tdb);
 				return retval;
 			}
 			/*
 			 * short read so update the record size
 			 * accordingly
 			 */
-			tdb_data.dsize = actual_size;
+			data_size = actual_size;
 		} else {
-			tdb_data.dsize = data->tdb_data_size;
+			data_size = data->tdb_data_size;
 		}
-		tdb_data.dptr = read_ptr +
-				((offset - data->offset) % channel->block_size);
-#ifdef DEBUG
-		printf("Printing with key %lld data %x and size %d\n",
+		if (data_size == 0) {
+			free(read_ptr);
+			block_num++;
+			continue;
+		}
+		dbg_printf("Read %llu bytes from FS block %llu (blk=%llu cnt=%u)\n",
+		       data_size, backing_blk_num, block, count);
+		if ((data_size % data->undo_file->block_size) == 0)
+			sz = data_size / data->undo_file->block_size;
+		else
+			sz = -actual_size;
+		data_ptr = read_ptr + ((offset - data->offset) %
+				       data->undo_file->block_size);
+		/* extend this key? */
+		if (data->keys_in_block) {
+			key = data->keyb->keys + data->keys_in_block - 1;
+			keysz = ext2fs_le32_to_cpu(key->size);
+		} else {
+			key = NULL;
+			keysz = 0;
+		}
+		if (key != NULL &&
+		    ext2fs_le64_to_cpu(key->fsblk) +
+		    ((keysz + data->tdb_data_size - 1) /
+		     data->tdb_data_size) == backing_blk_num &&
+		    E2UNDO_MAX_EXTENT_BLOCKS * data->tdb_data_size >
+		    keysz + sz) {
+			blk_crc = ext2fs_le32_to_cpu(key->blk_crc);
+			blk_crc = ext2fs_crc32c_le(blk_crc,
+						   (unsigned char *)data_ptr,
+						   data_size);
+			key->blk_crc = ext2fs_cpu_to_le32(blk_crc);
+			key->size = ext2fs_cpu_to_le32(keysz + data_size);
+		} else {
+			data->num_keys++;
+			key = data->keyb->keys + data->keys_in_block;
+			data->keys_in_block++;
+			key->fsblk = ext2fs_cpu_to_le64(backing_blk_num);
+			blk_crc = ext2fs_crc32c_le(~0,
+						   (unsigned char *)data_ptr,
+						   data_size);
+			key->blk_crc = ext2fs_cpu_to_le32(blk_crc);
+			key->size = ext2fs_cpu_to_le32(data_size);
+		}
+		dbg_printf("Writing block %llu to offset %llu size %d key %zu\n",
 		       block_num,
-		       tdb_data.dptr,
-		       tdb_data.dsize);
-#endif
-		retval = tdb_store(data->tdb, tdb_key, tdb_data, TDB_REPLACE);
-		if (retval == -1) {
-			/*
-			 * TDB_ERR_EXISTS cannot happen because we
-			 * have already verified it doesn't exist
-			 */
-			tdb_transaction_cancel(data->tdb);
-			retval = EXT2_ET_TDB_ERR_IO;
+		       data->undo_blk_num,
+		       sz, data->num_keys - 1);
+		retval = io_channel_write_blk64(data->undo_file,
+					data->undo_blk_num, sz, data_ptr);
+		if (retval) {
 			free(read_ptr);
 			return retval;
 		}
+		data->undo_blk_num++;
 		free(read_ptr);
 		/* Next block */
 		block_num++;
 	}
-	tdb_transaction_commit(data->tdb);
 
 	return retval;
 }
@@ -344,10 +481,192 @@ static void undo_err_handler_init(io_channel channel)
 	channel->read_error = undo_io_read_error;
 }
 
+static int check_filesystem(struct undo_header *hdr, io_channel undo_file,
+			    unsigned int blocksize, blk64_t super_block,
+			    io_channel channel)
+{
+	struct ext2_super_block super, *sb;
+	char *buf;
+	__u32 sb_crc;
+	errcode_t retval;
+
+	io_channel_set_blksize(channel, SUPERBLOCK_OFFSET);
+	retval = io_channel_read_blk64(channel, 1, -SUPERBLOCK_SIZE, &super);
+	if (retval)
+		return retval;
+
+	/*
+	 * Compare the FS and the undo file superblock so that we don't
+	 * append to something that doesn't match this FS.
+	 */
+	retval = ext2fs_get_mem(blocksize, &buf);
+	if (retval)
+		return retval;
+	retval = io_channel_read_blk64(undo_file, super_block,
+				       -SUPERBLOCK_SIZE, buf);
+	if (retval)
+		goto out;
+	sb = (struct ext2_super_block *)buf;
+	sb->s_magic = ~sb->s_magic;
+	if (memcmp(&super, buf, sizeof(super))) {
+		retval = -1;
+		goto out;
+	}
+	sb_crc = ext2fs_crc32c_le(~0, (unsigned char *)buf, SUPERBLOCK_SIZE);
+	if (ext2fs_le32_to_cpu(hdr->sb_crc) != sb_crc) {
+		retval = -1;
+		goto out;
+	}
+
+out:
+	ext2fs_free_mem(&buf);
+	return retval;
+}
+
+/*
+ * Try to re-open the undo file, so that we can resume where we left off.
+ * That way, the user can pass the same undo file to various programs as
+ * part of an FS upgrade instead of having to create multiple files and
+ * then apply them in correct order.
+ */
+static errcode_t try_reopen_undo_file(int undo_fd,
+				      struct undo_private_data *data)
+{
+	struct undo_header hdr;
+	struct undo_key *dkey;
+	ext2fs_struct_stat statbuf;
+	unsigned int blocksize, fs_blocksize;
+	blk64_t super_block, lblk;
+	size_t num_keys, keys_per_block, i;
+	__u32 hdr_crc, key_crc;
+	errcode_t retval;
+
+	/* Zero size already? */
+	retval = ext2fs_fstat(undo_fd, &statbuf);
+	if (retval)
+		goto bad_file;
+	if (statbuf.st_size == 0)
+		goto out;
+
+	/* check the file header */
+	retval = io_channel_read_blk64(data->undo_file, 0, -(int)sizeof(hdr),
+				       &hdr);
+	if (retval)
+		goto bad_file;
+
+	if (memcmp(hdr.magic, E2UNDO_MAGIC,
+		    sizeof(hdr.magic)))
+		goto bad_file;
+	hdr_crc = ext2fs_crc32c_le(~0, (unsigned char *)&hdr,
+				   sizeof(struct undo_header) -
+				   sizeof(__u32));
+	if (ext2fs_le32_to_cpu(hdr.header_crc) != hdr_crc)
+		goto bad_file;
+	blocksize = ext2fs_le32_to_cpu(hdr.block_size);
+	fs_blocksize = ext2fs_le32_to_cpu(hdr.fs_block_size);
+	if (blocksize > E2UNDO_MAX_BLOCK_SIZE ||
+	    blocksize < E2UNDO_MIN_BLOCK_SIZE ||
+	    !blocksize || !fs_blocksize)
+		goto bad_file;
+	super_block = ext2fs_le64_to_cpu(hdr.super_offset);
+	num_keys = ext2fs_le64_to_cpu(hdr.num_keys);
+	io_channel_set_blksize(data->undo_file, blocksize);
+	if (hdr.f_compat || hdr.f_incompat || hdr.f_rocompat)
+		goto bad_file;
+
+	/* Superblock matches this FS? */
+	if (check_filesystem(&hdr, data->undo_file, blocksize, super_block,
+			     data->real) != 0) {
+		retval = EXT2_ET_UNDO_FILE_WRONG;
+		goto out;
+	}
+
+	/* Try to set ourselves up */
+	data->tdb_data_size = blocksize;
+	retval = undo_setup_tdb(data);
+	if (retval)
+		goto bad_file;
+	data->num_keys = num_keys;
+	data->super_blk_num = super_block;
+	data->first_key_blk = ext2fs_le64_to_cpu(hdr.key_offset);
+
+	/* load the written block map */
+	keys_per_block = KEYS_PER_BLOCK(data);
+	lblk = data->first_key_blk;
+	dbg_printf("nr_keys=%lu, kpb=%zu, blksz=%u\n",
+		   num_keys, keys_per_block, blocksize);
+	for (i = 0; i < num_keys; i += keys_per_block) {
+		size_t j, max_j;
+		__le32 crc;
+
+		data->key_blk_num = lblk;
+		retval = io_channel_read_blk64(data->undo_file,
+					       lblk, 1, data->keyb);
+		if (retval)
+			goto bad_key_replay;
+
+		/* check keys */
+		if (ext2fs_le32_to_cpu(data->keyb->magic) != KEYBLOCK_MAGIC) {
+			retval = EXT2_ET_UNDO_FILE_CORRUPT;
+			goto bad_key_replay;
+		}
+		crc = data->keyb->crc;
+		data->keyb->crc = 0;
+		key_crc = ext2fs_crc32c_le(~0, (unsigned char *)data->keyb,
+					   blocksize);
+		if (ext2fs_le32_to_cpu(crc) != key_crc) {
+			retval = EXT2_ET_UNDO_FILE_CORRUPT;
+			goto bad_key_replay;
+		}
+
+		/* load keys from key block */
+		lblk++;
+		max_j = data->num_keys - i;
+		if (max_j > keys_per_block)
+			max_j = keys_per_block;
+		for (j = 0, dkey = data->keyb->keys;
+		     j < max_j;
+		     j++, dkey++) {
+			blk64_t fsblk = ext2fs_le64_to_cpu(dkey->fsblk);
+			blk64_t undo_blk = fsblk * fs_blocksize / blocksize;
+			size_t size = ext2fs_le32_to_cpu(dkey->size);
+
+			ext2fs_mark_block_bitmap_range2(data->written_block_map,
+					 undo_blk,
+					(size + blocksize - 1) / blocksize);
+			lblk += (size + blocksize - 1) / blocksize;
+			data->undo_blk_num = lblk;
+			data->keys_in_block = j + 1;
+		}
+	}
+	dbg_printf("Reopen undo, keyblk=%llu undoblk=%llu nrkeys=%zu kib=%zu\n",
+		   data->key_blk_num, data->undo_blk_num, data->num_keys,
+		   data->keys_in_block);
+
+	data->hdr.state = hdr.state & ~E2UNDO_STATE_FINISHED;
+	data->hdr.f_compat = hdr.f_compat;
+	data->hdr.f_incompat = hdr.f_incompat;
+	data->hdr.f_rocompat = hdr.f_rocompat;
+	return retval;
+
+bad_key_replay:
+	data->key_blk_num = data->undo_blk_num = 0;
+	data->keys_in_block = 0;
+	ext2fs_free_mem(&data->keyb);
+	ext2fs_free_generic_bitmap(data->written_block_map);
+	data->tdb_written = 0;
+	goto out;
+bad_file:
+	retval = EXT2_ET_UNDO_FILE_CORRUPT;
+out:
+	return retval;
+}
+
 static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 {
 	io_channel	io = NULL;
 	struct undo_private_data *data = NULL;
+	int		undo_fd = -1;
 	errcode_t	retval;
 
 	if (name == 0)
@@ -375,29 +694,32 @@ static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 
 	memset(data, 0, sizeof(struct undo_private_data));
 	data->magic = EXT2_ET_MAGIC_UNIX_IO_CHANNEL;
-	data->written_block_map = NULL;
+	data->super_blk_num = 1;
+	data->undo_blk_num = data->first_key_blk = 2;
 
 	if (undo_io_backing_manager) {
 		retval = undo_io_backing_manager->open(name, flags,
 						       &data->real);
 		if (retval)
 			goto cleanup;
+
+		undo_fd = ext2fs_open_file(tdb_file, O_RDWR | O_CREAT, 0600);
+		if (undo_fd < 0)
+			goto cleanup;
+
+		retval = undo_io_backing_manager->open(tdb_file, IO_FLAG_RW,
+						       &data->undo_file);
+		if (retval)
+			goto cleanup;
 	} else {
-		data->real = 0;
+		data->real = NULL;
+		data->undo_file = NULL;
 	}
 
 	if (data->real)
 		io->flags = (io->flags & ~CHANNEL_FLAGS_DISCARD_ZEROES) |
 			    (data->real->flags & CHANNEL_FLAGS_DISCARD_ZEROES);
 
-	/* setup the tdb file */
-	data->tdb = tdb_open(tdb_file, 0, TDB_CLEAR_IF_FIRST | TDB_NOLOCK | TDB_NOSYNC,
-			     O_RDWR | O_CREAT | O_TRUNC | O_EXCL, 0600);
-	if (!data->tdb) {
-		retval = errno;
-		goto cleanup;
-	}
-
 	/*
 	 * setup err handler for read so that we know
 	 * when the backing manager fails do short read
@@ -405,10 +727,22 @@ static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 	if (data->real)
 		undo_err_handler_init(data->real);
 
+	if (data->undo_file) {
+		retval = try_reopen_undo_file(undo_fd, data);
+		if (retval)
+			goto cleanup;
+	}
+
 	*channel = io;
-	return 0;
+	if (undo_fd >= 0)
+		close(undo_fd);
+	return retval;
 
 cleanup:
+	if (undo_fd >= 0)
+		close(undo_fd);
+	if (data && data->undo_file)
+		io_channel_close(data->undo_file);
 	if (data && data->real)
 		io_channel_close(data->real);
 	if (data)
@@ -430,13 +764,14 @@ static errcode_t undo_close(io_channel channel)
 	if (--channel->refcount > 0)
 		return 0;
 	/* Before closing write the file system identity */
-	err = write_file_system_identity(channel, data->tdb);
+	if (!getenv("UNDO_IO_SIMULATE_UNFINISHED"))
+		data->hdr.state = ext2fs_cpu_to_le32(E2UNDO_STATE_FINISHED);
+	err = write_undo_indexes(data);
 	if (data->real)
 		retval = io_channel_close(data->real);
-	if (data->tdb) {
-		tdb_flush(data->tdb);
-		tdb_close(data->tdb);
-	}
+	if (data->undo_file)
+		io_channel_close(data->undo_file);
+	ext2fs_free_mem(&data->keyb);
 	if (data->written_block_map)
 		ext2fs_free_generic_bitmap(data->written_block_map);
 	ext2fs_free_mem(&channel->private_data);
@@ -458,6 +793,9 @@ static errcode_t undo_set_blksize(io_channel channel, int blksize)
 	data = (struct undo_private_data *) channel->private_data;
 	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
 
+	if (blksize > E2UNDO_MAX_BLOCK_SIZE || blksize < E2UNDO_MIN_BLOCK_SIZE)
+		return EXT2_ET_INVALID_ARGUMENT;
+
 	if (data->real)
 		retval = io_channel_set_blksize(data->real, blksize);
 	/*
@@ -632,8 +970,6 @@ static errcode_t undo_flush(io_channel channel)
 	data = (struct undo_private_data *) channel->private_data;
 	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
 
-	if (data->tdb)
-		tdb_flush(data->tdb);
 	if (data->real)
 		retval = io_channel_flush(data->real);
 
@@ -659,6 +995,8 @@ static errcode_t undo_set_option(io_channel channel, const char *option,
 		tmp = strtoul(arg, &end, 0);
 		if (*end)
 			return EXT2_ET_INVALID_ARGUMENT;
+		if (tmp > E2UNDO_MAX_BLOCK_SIZE || tmp < E2UNDO_MIN_BLOCK_SIZE)
+			return EXT2_ET_INVALID_ARGUMENT;
 		if (!data->tdb_data_size || !data->tdb_written) {
 			data->tdb_written = -1;
 			data->tdb_data_size = tmp;
diff --git a/misc/e2undo.8.in b/misc/e2undo.8.in
index 4bf0798..71e8a7b 100644
--- a/misc/e2undo.8.in
+++ b/misc/e2undo.8.in
@@ -10,6 +10,12 @@ e2undo \- Replay an undo log for an ext2/ext3/ext4 filesystem
 [
 .B \-f
 ]
+[
+.B \-n
+]
+[
+.B \-v
+]
 .I undo_log device
 .SH DESCRIPTION
 .B e2undo
@@ -24,13 +30,18 @@ used to undo a failed operation by an e2fsprogs program.
 .B \-f
 Normally,
 .B e2undo
-will check the filesystem UUID and last modified time to make sure the
-undo log matches with the filesystem on the device.  If they do not
-match,
+will check the filesystem superblock to make sure the undo log matches
+with the filesystem on the device.  If they do not match,
 .B e2undo
 will refuse to apply the undo log as a safety mechanism.  The
 .B \-f
 option disables this safety mechanism.
+.TP
+.B \-n
+Dry-run; do not actually write blocks back to the filesystem.
+.TP
+.B \-v
+Report which block we're currently replaying.
 .SH AUTHOR
 .B e2undo
 was written by Aneesh Kumar K.V. (aneesh.kumar@linux.vnet.ibm.com)
diff --git a/misc/e2undo.c b/misc/e2undo.c
index d828d3b..3f312c6 100644
--- a/misc/e2undo.c
+++ b/misc/e2undo.c
@@ -20,30 +20,132 @@
 #if HAVE_ERRNO_H
 #include <errno.h>
 #endif
-#include "ext2fs/tdb.h"
+#include <unistd.h>
 #include "ext2fs/ext2fs.h"
 #include "nls-enable.h"
 
-static unsigned char mtime_key[] = "filesystem MTIME";
-static unsigned char uuid_key[] = "filesystem UUID";
-static unsigned char blksize_key[] = "filesystem BLKSIZE";
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
+/*
+ * Undo file format: The file is cut up into undo_header.block_size blocks.
+ * The first block contains the header.
+ * The second block contains the superblock.
+ * There is then a repeating series of blocks as follows:
+ *   A key block, which contains undo_keys to map the following data blocks.
+ *   Data blocks
+ * (Note that there are pointers to the first key block and the sb, so this
+ * order isn't strictly necessary.)
+ */
+#define E2UNDO_MAGIC "E2UNDO02"
+#define KEYBLOCK_MAGIC 0xCADECADE
+
+#define E2UNDO_STATE_FINISHED	0x1	/* undo file is complete */
+
+#define E2UNDO_MIN_BLOCK_SIZE	1024	/* undo blocks are no less than 1KB */
+#define E2UNDO_MAX_BLOCK_SIZE	1048576	/* undo blocks are no more than 1MB */
+
+struct undo_header {
+	char magic[8];		/* "E2UNDO02" */
+	__le64 num_keys;	/* how many keys? */
+	__le64 super_offset;	/* where in the file is the superblock copy? */
+	__le64 key_offset;	/* where do the key/data block chunks start? */
+	__le32 block_size;	/* block size of the undo file */
+	__le32 fs_block_size;	/* block size of the target device */
+	__le32 sb_crc;		/* crc32c of the superblock */
+	__le32 state;		/* e2undo state flags */
+	__le32 f_compat;	/* compatible features (none so far) */
+	__le32 f_incompat;	/* incompatible features (none so far) */
+	__le32 f_rocompat;	/* ro compatible features (none so far) */
+	__u8 padding[448];	/* padding */
+	__le32 header_crc;	/* crc32c of the header (but not this field) */
+};
+
+#define E2UNDO_MAX_EXTENT_BLOCKS	512	/* max extent size, in blocks */
+
+struct undo_key {
+	__le64 fsblk;		/* where in the fs does the block go */
+	__le32 blk_crc;		/* crc32c of the block */
+	__le32 size;		/* how many bytes in this block? */
+};
+
+struct undo_key_block {
+	__le32 magic;		/* KEYBLOCK_MAGIC number */
+	__le32 crc;		/* block checksum */
+	__le64 reserved;	/* zero */
+
+	struct undo_key keys[0];	/* keys, which come immediately after */
+};
+
+struct undo_key_info {
+	blk64_t fsblk;
+	blk64_t fileblk;
+	__u32 blk_crc;
+	unsigned int size;
+};
+
+struct undo_context {
+	struct undo_header hdr;
+	io_channel undo_file;
+	unsigned int blocksize, fs_blocksize;
+	blk64_t super_block;
+	size_t num_keys;
+	struct undo_key_info *keys;
+};
+#define KEYS_PER_BLOCK(d) (((d)->blocksize / sizeof(struct undo_key)) - 1)
 
 static char *prg_name;
+static char *undo_file;
 
 static void usage(void)
 {
 	fprintf(stderr,
-		_("Usage: %s <transaction file> <filesystem>\n"), prg_name);
+		_("Usage: %s [-f] [-h] [-n] [-v] <transaction file> <filesystem>\n"), prg_name);
 	exit(1);
 }
 
-static int check_filesystem(TDB_CONTEXT *tdb, io_channel channel)
+static void dump_header(struct undo_header *hdr)
+{
+	printf("nr keys:\t%llu\n", ext2fs_le64_to_cpu(hdr->num_keys));
+	printf("super block:\t%llu\n", ext2fs_le64_to_cpu(hdr->super_offset));
+	printf("key block:\t%llu\n", ext2fs_le64_to_cpu(hdr->key_offset));
+	printf("block size:\t%u\n", ext2fs_le32_to_cpu(hdr->block_size));
+	printf("fs block size:\t%u\n", ext2fs_le32_to_cpu(hdr->fs_block_size));
+	printf("super crc:\t0x%x\n", ext2fs_le32_to_cpu(hdr->sb_crc));
+	printf("state:\t\t0x%x\n", ext2fs_le32_to_cpu(hdr->state));
+	printf("compat:\t\t0x%x\n", ext2fs_le32_to_cpu(hdr->f_compat));
+	printf("incompat:\t0x%x\n", ext2fs_le32_to_cpu(hdr->f_incompat));
+	printf("rocompat:\t0x%x\n", ext2fs_le32_to_cpu(hdr->f_rocompat));
+	printf("header crc:\t0x%x\n", ext2fs_le32_to_cpu(hdr->header_crc));
+}
+
+static void print_undo_mismatch(struct ext2_super_block *fs_super,
+				struct ext2_super_block *undo_super)
+{
+	printf("%s",
+	       _("The file system superblock doesn't match the undo file.\n"));
+	if (memcmp(fs_super->s_uuid, undo_super->s_uuid,
+		   sizeof(fs_super->s_uuid)))
+		printf("%s", _("UUID does not match.\n"));
+	if (fs_super->s_mtime != undo_super->s_mtime)
+		printf("%s", _("Last mount time does not match.\n"));
+	if (fs_super->s_wtime != undo_super->s_wtime)
+		printf("%s", _("Last write time does not match.\n"));
+	if (fs_super->s_kbytes_written != undo_super->s_kbytes_written)
+		printf("%s", _("Lifetime write counter does not match.\n"));
+}
+
+static int check_filesystem(struct undo_context *ctx, io_channel channel)
 {
-	__u32   s_mtime;
-	__u8    s_uuid[16];
+	struct ext2_super_block super, *sb;
+	char *buf;
+	__u32 sb_crc;
 	errcode_t retval;
-	TDB_DATA tdb_key, tdb_data;
-	struct ext2_super_block super;
 
 	io_channel_set_blksize(channel, SUPERBLOCK_OFFSET);
 	retval = io_channel_read_blk64(channel, 1, -SUPERBLOCK_SIZE, &super);
@@ -53,83 +155,127 @@ static int check_filesystem(TDB_CONTEXT *tdb, io_channel channel)
 		return retval;
 	}
 
-	tdb_key.dptr = mtime_key;
-	tdb_key.dsize = sizeof(mtime_key);
-	tdb_data = tdb_fetch(tdb, tdb_key);
-	if (!tdb_data.dptr) {
-		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-		com_err(prg_name, retval, "%s",
-			_("while fetching last mount time."));
+	/*
+	 * Compare the FS and the undo file superblock so that we can't apply
+	 * e2undo "patches" out of order.
+	 */
+	retval = ext2fs_get_mem(ctx->blocksize, &buf);
+	if (retval) {
+		com_err(prg_name, retval, "%s", _("while allocating memory"));
 		return retval;
 	}
+	retval = io_channel_read_blk64(ctx->undo_file, ctx->super_block,
+				       -SUPERBLOCK_SIZE, buf);
+	if (retval) {
+		com_err(prg_name, retval, "%s", _("while fetching superblock"));
+		goto out;
+	}
+	sb = (struct ext2_super_block *)buf;
+	sb->s_magic = ~sb->s_magic;
+	if (memcmp(&super, buf, sizeof(super))) {
+		print_undo_mismatch(&super, (struct ext2_super_block *)buf);
+		retval = -1;
+		goto out;
+	}
+	sb_crc = ext2fs_crc32c_le(~0, (unsigned char *)buf, SUPERBLOCK_SIZE);
+	if (ext2fs_le32_to_cpu(ctx->hdr.sb_crc) != sb_crc) {
+		fprintf(stderr,
+			_("Undo file superblock checksum doesn't match.\n"));
+		retval = -1;
+		goto out;
+	}
 
-	s_mtime = *(__u32 *)tdb_data.dptr;
-	free(tdb_data.dptr);
-	if (super.s_mtime != s_mtime) {
-		com_err(prg_name, 0,
-			_("The filesystem last mount time didn't match %u."),
-			s_mtime);
+out:
+	ext2fs_free_mem(&buf);
+	return retval;
+}
 
-		return  -1;
-	}
+static int key_compare(const void *a, const void *b)
+{
+	const struct undo_key_info *ka, *kb;
 
+	ka = a;
+	kb = b;
+	return ext2fs_le64_to_cpu(ka->fsblk) -
+	       ext2fs_le64_to_cpu(kb->fsblk);
+}
+
+static int e2undo_setup_tdb(const char *name, io_manager *io_ptr)
+{
+	errcode_t retval = 0;
+	const char *tdb_dir;
+	char *tdb_file;
+	char *dev_name, *tmp_name;
 
-	tdb_key.dptr = uuid_key;
-	tdb_key.dsize = sizeof(uuid_key);
-	tdb_data = tdb_fetch(tdb, tdb_key);
-	if (!tdb_data.dptr) {
-		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-		com_err(prg_name, retval, "%s", _("while fetching UUID"));
+	/* (re)open a specific undo file */
+	if (undo_file && undo_file[0] != 0) {
+		set_undo_io_backing_manager(*io_ptr);
+		*io_ptr = undo_io_manager;
+		set_undo_io_backup_file(undo_file);
+		printf(_("To undo the e2undo operation please run "
+			 "the command\n    e2undo %s %s\n\n"),
+			 undo_file, name);
 		return retval;
 	}
-	memcpy(s_uuid, tdb_data.dptr, sizeof(s_uuid));
-	free(tdb_data.dptr);
-	if (memcmp(s_uuid, super.s_uuid, sizeof(s_uuid))) {
-		com_err(prg_name, 0, "%s",
-			_("The filesystem UUID didn't match."));
-		return -1;
+
+	tmp_name = strdup(name);
+	if (!tmp_name) {
+	alloc_fn_fail:
+		com_err(prg_name, ENOMEM, "%s",
+			_("Couldn't allocate memory for tdb filename\n"));
+		return ENOMEM;
 	}
+	dev_name = basename(tmp_name);
 
-	return 0;
-}
+	tdb_dir = getenv("E2FSPROGS_UNDO_DIR");
+	if (!tdb_dir)
+		tdb_dir = "/var/lib/e2fsprogs";
 
-static int set_blk_size(TDB_CONTEXT *tdb, io_channel channel)
-{
-	int block_size;
-	errcode_t retval;
-	TDB_DATA tdb_key, tdb_data;
-
-	tdb_key.dptr = blksize_key;
-	tdb_key.dsize = sizeof(blksize_key);
-	tdb_data = tdb_fetch(tdb, tdb_key);
-	if (!tdb_data.dptr) {
-		retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-		com_err(prg_name, retval, "%s", _("while fetching block size"));
+	if (!strcmp(tdb_dir, "none") || (tdb_dir[0] == 0) ||
+	    access(tdb_dir, W_OK))
+		return 0;
+
+	tdb_file = malloc(strlen(tdb_dir) + 9 + strlen(dev_name) + 7 + 1);
+	if (!tdb_file)
+		goto alloc_fn_fail;
+	sprintf(tdb_file, "%s/e2undo-%s.e2undo", tdb_dir, dev_name);
+
+	if ((unlink(tdb_file) < 0) && (errno != ENOENT)) {
+		retval = errno;
+		com_err(prg_name, retval,
+			_("while trying to delete %s"), tdb_file);
+		free(tdb_file);
 		return retval;
 	}
 
-	block_size = *(int *)tdb_data.dptr;
-	free(tdb_data.dptr);
-#ifdef DEBUG
-	printf("Block size %d\n", block_size);
-#endif
-	io_channel_set_blksize(channel, block_size);
-
-	return 0;
+	set_undo_io_backing_manager(*io_ptr);
+	*io_ptr = undo_io_manager;
+	set_undo_io_backup_file(tdb_file);
+	printf(_("To undo the e2undo operation please run "
+		 "the command\n    e2undo %s %s\n\n"),
+		 tdb_file, name);
+	free(tdb_file);
+	free(tmp_name);
+	return retval;
 }
 
 int main(int argc, char *argv[])
 {
-	int c,force = 0;
-	TDB_CONTEXT *tdb;
-	TDB_DATA key, data;
+	int c, force = 0, dry_run = 0, verbose = 0, dump = 0;
 	io_channel channel;
 	errcode_t retval;
-	int  mount_flags;
-	blk64_t  blk_num;
+	int mount_flags, csum_error = 0, io_error = 0;
+	size_t i, keys_per_block;
 	char *device_name, *tdb_file;
 	io_manager manager = unix_io_manager;
-	void *old_dptr = NULL;
+	struct undo_context undo_ctx;
+	char *buf;
+	struct undo_key_block *keyb;
+	struct undo_key *dkey;
+	struct undo_key_info *ikey;
+	__u32 key_crc, blk_crc, hdr_crc;
+	blk64_t lblk;
+	ext2_filsys fs;
 
 #ifdef ENABLE_NLS
 	setlocale(LC_MESSAGES, "");
@@ -141,13 +287,25 @@ int main(int argc, char *argv[])
 	add_error_table(&et_ext2_error_table);
 
 	prg_name = argv[0];
-	while((c = getopt(argc, argv, "f")) != EOF) {
+	while ((c = getopt(argc, argv, "fhnvz:")) != EOF) {
 		switch (c) {
-			case 'f':
-				force = 1;
-				break;
-			default:
-				usage();
+		case 'f':
+			force = 1;
+			break;
+		case 'h':
+			dump = 1;
+			break;
+		case 'n':
+			dry_run = 1;
+			break;
+		case 'v':
+			verbose = 1;
+			break;
+		case 'z':
+			undo_file = optarg;
+			break;
+		default:
+			usage();
 		}
 	}
 
@@ -157,14 +315,70 @@ int main(int argc, char *argv[])
 	tdb_file = argv[optind];
 	device_name = argv[optind+1];
 
-	tdb = tdb_open(tdb_file, 0, 0, O_RDONLY, 0600);
+	if (undo_file && strcmp(tdb_file, undo_file) == 0) {
+		printf(_("Will not write to an undo file while replaying it.\n"));
+		exit(1);
+	}
 
-	if (!tdb) {
+	/* Interpret the undo file */
+	retval = manager->open(tdb_file, IO_FLAG_EXCLUSIVE,
+			       &undo_ctx.undo_file);
+	if (retval) {
 		com_err(prg_name, errno,
 				_("while opening undo file `%s'\n"), tdb_file);
 		exit(1);
 	}
+	retval = io_channel_read_blk64(undo_ctx.undo_file, 0,
+				       -(int)sizeof(undo_ctx.hdr),
+				       &undo_ctx.hdr);
+	if (retval) {
+		com_err(prg_name, retval, _("while reading undo file"));
+		exit(1);
+	}
+	if (memcmp(undo_ctx.hdr.magic, E2UNDO_MAGIC,
+		    sizeof(undo_ctx.hdr.magic))) {
+		fprintf(stderr, _("%s: Not an undo file.\n"), tdb_file);
+		exit(1);
+	}
+	if (dump) {
+		dump_header(&undo_ctx.hdr);
+		exit(1);
+	}
+	hdr_crc = ext2fs_crc32c_le(~0, (unsigned char *)&undo_ctx.hdr,
+				   sizeof(struct undo_header) -
+				   sizeof(__u32));
+	if (!force && ext2fs_le32_to_cpu(undo_ctx.hdr.header_crc) != hdr_crc) {
+		fprintf(stderr, _("%s: Header checksum doesn't match.\n"),
+			tdb_file);
+		exit(1);
+	}
+	undo_ctx.blocksize = ext2fs_le32_to_cpu(undo_ctx.hdr.block_size);
+	undo_ctx.fs_blocksize = ext2fs_le32_to_cpu(undo_ctx.hdr.fs_block_size);
+	if (undo_ctx.blocksize == 0 || undo_ctx.fs_blocksize == 0) {
+		fprintf(stderr, _("%s: Corrupt undo file header.\n"), tdb_file);
+		exit(1);
+	}
+	if (!force && undo_ctx.blocksize > E2UNDO_MAX_BLOCK_SIZE) {
+		fprintf(stderr, _("%s: Undo block size too large.\n"),
+			tdb_file);
+		exit(1);
+	}
+	if (!force && undo_ctx.blocksize < E2UNDO_MIN_BLOCK_SIZE) {
+		fprintf(stderr, _("%s: Undo block size too small.\n"),
+			tdb_file);
+		exit(1);
+	}
+	undo_ctx.super_block = ext2fs_le64_to_cpu(undo_ctx.hdr.super_offset);
+	undo_ctx.num_keys = ext2fs_le64_to_cpu(undo_ctx.hdr.num_keys);
+	io_channel_set_blksize(undo_ctx.undo_file, undo_ctx.blocksize);
+	if (!force && (undo_ctx.hdr.f_compat || undo_ctx.hdr.f_incompat ||
+		       undo_ctx.hdr.f_rocompat)) {
+		fprintf(stderr, _("%s: Unknown undo file feature set.\n"),
+			tdb_file);
+		exit(1);
+	}
 
+	/* open the fs */
 	retval = ext2fs_check_if_mounted(device_name, &mount_flags);
 	if (retval) {
 		com_err(prg_name, retval, _("Error while determining whether "
@@ -178,53 +392,197 @@ int main(int argc, char *argv[])
 		exit(1);
 	}
 
+	if (undo_file) {
+		retval = e2undo_setup_tdb(device_name, &manager);
+		if (retval)
+			exit(1);
+	}
+
 	retval = manager->open(device_name,
-				IO_FLAG_EXCLUSIVE | IO_FLAG_RW,  &channel);
+			       IO_FLAG_EXCLUSIVE | (dry_run ? 0 : IO_FLAG_RW),
+			       &channel);
 	if (retval) {
 		com_err(prg_name, retval,
 				_("while opening `%s'"), device_name);
 		exit(1);
 	}
 
-	if (!force && check_filesystem(tdb, channel)) {
+	if (!force && check_filesystem(&undo_ctx, channel))
 		exit(1);
-	}
 
-	if (set_blk_size(tdb, channel)) {
+	/* prepare to read keys */
+	retval = ext2fs_get_mem(sizeof(struct undo_key_info) * undo_ctx.num_keys,
+				&undo_ctx.keys);
+	if (retval) {
+		com_err(prg_name, retval, "%s", _("while allocating memory"));
+		exit(1);
+	}
+	ikey = undo_ctx.keys;
+	retval = ext2fs_get_mem(undo_ctx.blocksize, &keyb);
+	if (retval) {
+		com_err(prg_name, retval, "%s", _("while allocating memory"));
+		exit(1);
+	}
+	retval = ext2fs_get_mem(E2UNDO_MAX_EXTENT_BLOCKS * undo_ctx.blocksize,
+				&buf);
+	if (retval) {
+		com_err(prg_name, retval, "%s", _("while allocating memory"));
 		exit(1);
 	}
 
-	for (key = tdb_firstkey(tdb); key.dptr; key = tdb_nextkey(tdb, key)) {
-		free(old_dptr);
-		old_dptr = key.dptr;
-		if (!strcmp((char *) key.dptr, (char *) mtime_key) ||
-		    !strcmp((char *) key.dptr, (char *) uuid_key) ||
-		    !strcmp((char *) key.dptr, (char *) blksize_key)) {
-			continue;
+	/* load keys */
+	keys_per_block = KEYS_PER_BLOCK(&undo_ctx);
+	lblk = ext2fs_le64_to_cpu(undo_ctx.hdr.key_offset);
+	dbg_printf("nr_keys=%lu, kpb=%zu, blksz=%u\n",
+		   undo_ctx.num_keys, keys_per_block, undo_ctx.blocksize);
+	for (i = 0; i < undo_ctx.num_keys; i += keys_per_block) {
+		size_t j, max_j;
+		__le32 crc;
+
+		retval = io_channel_read_blk64(undo_ctx.undo_file,
+					       lblk, 1, keyb);
+		if (retval) {
+			com_err(prg_name, retval, "%s", _("while reading keys"));
+			if (force) {
+				io_error = 1;
+				undo_ctx.num_keys = i - 1;
+				break;
+			}
+			exit(1);
 		}
 
-		blk_num = *(blk64_t *)key.dptr;
-		data = tdb_fetch(tdb, key);
-		if (!data.dptr) {
-			retval = EXT2_ET_TDB_SUCCESS + tdb_error(tdb);
-			com_err(prg_name, retval,
-				_("while fetching block %llu."), blk_num);
+		/* check keys */
+		if (!force &&
+		    ext2fs_le32_to_cpu(keyb->magic) != KEYBLOCK_MAGIC) {
+			fprintf(stderr, _("%s: wrong key magic at %llu\n"),
+				tdb_file, lblk);
 			exit(1);
 		}
-		printf(_("Replayed transaction of size %zd at location %llu\n"),
-							data.dsize, blk_num);
-		retval = io_channel_write_blk64(channel, blk_num,
-						-data.dsize, data.dptr);
-		free(data.dptr);
-		if (retval == -1) {
-			com_err(prg_name, retval,
-				_("while writing block %llu."), blk_num);
+		crc = keyb->crc;
+		keyb->crc = 0;
+		key_crc = ext2fs_crc32c_le(~0, (unsigned char *)keyb,
+					   undo_ctx.blocksize);
+		if (!force && ext2fs_le32_to_cpu(crc) != key_crc) {
+			fprintf(stderr,
+				_("%s: key block checksum error at %llu.\n"),
+				tdb_file, lblk);
 			exit(1);
 		}
+
+		/* load keys from key block */
+		lblk++;
+		max_j = undo_ctx.num_keys - i;
+		if (max_j > keys_per_block)
+			max_j = keys_per_block;
+		for (j = 0, dkey = keyb->keys;
+		     j < max_j;
+		     j++, ikey++, dkey++) {
+			ikey->fsblk = ext2fs_le64_to_cpu(dkey->fsblk);
+			ikey->fileblk = lblk;
+			ikey->blk_crc = ext2fs_le32_to_cpu(dkey->blk_crc);
+			ikey->size = ext2fs_le32_to_cpu(dkey->size);
+			lblk += (ikey->size + undo_ctx.blocksize - 1) /
+				undo_ctx.blocksize;
+
+			if (E2UNDO_MAX_EXTENT_BLOCKS * undo_ctx.blocksize <
+			    ikey->size) {
+				com_err(prg_name, retval,
+					_("%s: block %llu is too long."),
+					tdb_file, ikey->fsblk);
+				exit(1);
+			}
+
+			/* check each block's crc */
+			retval = io_channel_read_blk64(undo_ctx.undo_file,
+						       ikey->fileblk,
+						       -(int)ikey->size,
+						       buf);
+			if (retval) {
+				com_err(prg_name, retval,
+					_("while fetching block %llu."),
+					ikey->fileblk);
+				if (!force)
+					exit(1);
+				io_error = 1;
+				continue;
+			}
+
+			blk_crc = ext2fs_crc32c_le(~0, (unsigned char *)buf,
+						   ikey->size);
+			if (blk_crc != ikey->blk_crc) {
+				fprintf(stderr,
+					_("checksum error in filesystem block "
+					  "%llu (undo blk %llu)\n"),
+					ikey->fsblk, ikey->fileblk);
+				if (!force)
+					exit(1);
+				csum_error = 1;
+			}
+		}
 	}
-	free(old_dptr);
+	ext2fs_free_mem(&keyb);
+
+	/* sort keys in fs block order */
+	qsort(undo_ctx.keys, undo_ctx.num_keys, sizeof(struct undo_key_info),
+	      key_compare);
+
+	/* replay */
+	io_channel_set_blksize(channel, undo_ctx.fs_blocksize);
+	for (i = 0, ikey = undo_ctx.keys; i < undo_ctx.num_keys; i++, ikey++) {
+		retval = io_channel_read_blk64(undo_ctx.undo_file,
+					       ikey->fileblk,
+					       -(int)ikey->size,
+					       buf);
+		if (retval) {
+			com_err(prg_name, retval,
+				_("while fetching block %llu."),
+				ikey->fileblk);
+			io_error = 1;
+			continue;
+		}
+
+		if (verbose)
+			printf("Replayed block of size %u from %llu to %llu\n",
+				ikey->size, ikey->fileblk, ikey->fsblk);
+		if (dry_run)
+			continue;
+		retval = io_channel_write_blk64(channel, ikey->fsblk,
+						-(int)ikey->size, buf);
+		if (retval) {
+			com_err(prg_name, retval,
+				_("while writing block %llu."), ikey->fsblk);
+			io_error = 1;
+		}
+	}
+
+	if (csum_error)
+		fprintf(stderr, _("Undo file corruption; run e2fsck NOW!\n"));
+	if (io_error)
+		fprintf(stderr, _("IO error during replay; run e2fsck NOW!\n"));
+	if (!(ext2fs_le32_to_cpu(undo_ctx.hdr.state) & E2UNDO_STATE_FINISHED)) {
+		force = 1;
+		fprintf(stderr, _("Incomplete undo record; run e2fsck.\n"));
+	}
+	ext2fs_free_mem(&buf);
+	ext2fs_free_mem(&undo_ctx.keys);
 	io_channel_close(channel);
-	tdb_close(tdb);
 
-	return 0;
+	/* If there were problems, try to force a fsck */
+	if (!dry_run && (force || csum_error || io_error)) {
+		retval = ext2fs_open2(device_name, NULL,
+				   EXT2_FLAG_RW | EXT2_FLAG_64BITS, 0, 0,
+				   manager, &fs);
+		if (retval)
+			goto out;
+		fs->super->s_state &= ~EXT2_VALID_FS;
+		if (csum_error || io_error)
+			fs->super->s_state |= EXT2_ERROR_FS;
+		ext2fs_mark_super_dirty(fs);
+		ext2fs_close_free(&fs);
+	}
+
+out:
+	io_channel_close(undo_ctx.undo_file);
+
+	return csum_error;
 }

[PATCH 34/54] libext2fs: support atexit cleanups

[Darrick J. Wong]

Use the atexit() function to provide a means for the library to clean
itself up on program exit.  This will be used by the undo IO manager
to flush the undo file state to disk if the program should terminate
without closing the io channel, since most e2fsprogs clients will
simply exit() when they hit errors.

This won't help for signal termination; client programs must set
up signal handlers.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/Makefile.in |    8 +++
 lib/ext2fs/atexit.c    |  112 ++++++++++++++++++++++++++++++++++++++++++++++++
 lib/ext2fs/ext2fsP.h   |    5 ++
 lib/ext2fs/undo_io.c   |   32 ++++++++++++--
 4 files changed, 154 insertions(+), 3 deletions(-)
 create mode 100644 lib/ext2fs/atexit.c


diff --git a/lib/ext2fs/Makefile.in b/lib/ext2fs/Makefile.in
index 2706bfa..4b64101 100644
--- a/lib/ext2fs/Makefile.in
+++ b/lib/ext2fs/Makefile.in
@@ -57,6 +57,7 @@ OBJS= $(DEBUGFS_LIB_OBJS) $(RESIZE_LIB_OBJS) $(E2IMAGE_LIB_OBJS) \
 	alloc_sb.o \
 	alloc_stats.o \
 	alloc_tables.o \
+	atexit.o \
 	badblocks.o \
 	bb_inode.o \
 	bitmaps.o \
@@ -130,6 +131,7 @@ SRCS= ext2_err.c \
 	$(srcdir)/alloc_sb.c \
 	$(srcdir)/alloc_stats.c \
 	$(srcdir)/alloc_tables.c \
+	$(srcdir)/atexit.c \
 	$(srcdir)/badblocks.c \
 	$(srcdir)/bb_compat.c \
 	$(srcdir)/bb_inode.c \
@@ -615,6 +617,12 @@ alloc_tables.o: $(srcdir)/alloc_tables.c $(top_builddir)/lib/config.h \
  $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h $(top_srcdir)/lib/et/com_err.h \
  $(srcdir)/ext2_io.h $(top_builddir)/lib/ext2fs/ext2_err.h \
  $(srcdir)/ext2_ext_attr.h $(srcdir)/bitops.h $(srcdir)/ext2fsP.h
+atexit.o: $(srcdir)/atexit.c $(top_builddir)/lib/config.h \
+ $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
+ $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h $(top_srcdir)/lib/et/com_err.h \
+ $(srcdir)/ext2_io.h $(top_builddir)/lib/ext2fs/ext2_err.h \
+ $(srcdir)/ext2_ext_attr.h $(srcdir)/bitops.h $(srcdir)/ext2fsP.h
 badblocks.o: $(srcdir)/badblocks.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
  $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fsP.h \
diff --git a/lib/ext2fs/atexit.c b/lib/ext2fs/atexit.c
new file mode 100644
index 0000000..5eba993
--- /dev/null
+++ b/lib/ext2fs/atexit.c
@@ -0,0 +1,112 @@
+/*
+ * atexit.c --- Clean things up when we exit normally.
+ *
+ * Copyright Oracle, 2014
+ * Author Darrick J. Wong <darrick.wong@oracle.com>
+ *
+ * %Begin-Header%
+ * This file may be redistributed under the terms of the GNU Library
+ * General Public License, version 2.
+ * %End-Header%
+ */
+
+#define _LARGEFILE_SOURCE
+#define _LARGEFILE64_SOURCE
+
+#include "config.h"
+#include <stdlib.h>
+
+#include "ext2_fs.h"
+#include "ext2fs.h"
+#include "ext2fsP.h"
+
+struct exit_data {
+	ext2_exit_fn func;
+	void *data;
+};
+
+static struct exit_data *items;
+static size_t nr_items;
+
+static void handle_exit(void)
+{
+	struct exit_data *ed;
+
+	for (ed = items + nr_items - 1; ed >= items; ed--) {
+		if (ed->func == NULL)
+			continue;
+		ed->func(ed->data);
+	}
+
+	ext2fs_free_mem(&items);
+	nr_items = 0;
+}
+
+/*
+ * Schedule a function to be called at (normal) program termination.
+ * If you want this to be called during a signal exit, you must capture
+ * the signal and call exit() yourself!
+ */
+errcode_t ext2fs_add_exit_fn(ext2_exit_fn func, void *data)
+{
+	struct exit_data *ed, *free_ed = NULL;
+	size_t x;
+	errcode_t ret;
+
+	if (func == NULL)
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	for (x = 0, ed = items; x < nr_items; x++, ed++) {
+		if (ed->func == func && ed->data == data)
+			return EXT2_ET_FILE_EXISTS;
+		if (ed->func == NULL)
+			free_ed = ed;
+	}
+
+	if (free_ed) {
+		free_ed->func = func;
+		free_ed->data = data;
+		return 0;
+	}
+
+	if (nr_items == 0) {
+		ret = atexit(handle_exit);
+		if (ret)
+			return ret;
+	}
+
+	ret = ext2fs_resize_mem(0, (nr_items + 1) * sizeof(struct exit_data),
+				&items);
+	if (ret)
+		return ret;
+
+	items[nr_items].func = func;
+	items[nr_items].data = data;
+	nr_items++;
+
+	return 0;
+}
+
+/* Remove a function from the exit cleanup list. */
+errcode_t ext2fs_remove_exit_fn(ext2_exit_fn func, void *data)
+{
+	struct exit_data *ed;
+	size_t x;
+
+	if (func == NULL)
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	for (x = 0, ed = items; x < nr_items; x++, ed++) {
+		if (ed->func == NULL)
+			return 0;
+		if (ed->func == func && ed->data == data) {
+			size_t sz = (nr_items - (x + 1)) *
+				    sizeof(struct exit_data);
+			memmove(ed, ed + 1, sz);
+			memset(items + nr_items - 1, 0,
+			       sizeof(struct exit_data));
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/ext2fs/ext2fsP.h b/lib/ext2fs/ext2fsP.h
index f8c61e6..8de9d33 100644
--- a/lib/ext2fs/ext2fsP.h
+++ b/lib/ext2fs/ext2fsP.h
@@ -169,3 +169,8 @@ extern int ext2fs_mem_is_zero(const char *mem, size_t len);
 extern int ext2fs_file_block_offset_too_big(ext2_filsys fs,
 					    struct ext2_inode *inode,
 					    blk64_t offset);
+
+/* atexit support */
+typedef void (*ext2_exit_fn)(void *);
+errcode_t ext2fs_add_exit_fn(ext2_exit_fn fn, void *data);
+errcode_t ext2fs_remove_exit_fn(ext2_exit_fn fn, void *data);
diff --git a/lib/ext2fs/undo_io.c b/lib/ext2fs/undo_io.c
index f1c107a..df26e3e 100644
--- a/lib/ext2fs/undo_io.c
+++ b/lib/ext2fs/undo_io.c
@@ -41,6 +41,7 @@
 
 #include "ext2_fs.h"
 #include "ext2fs.h"
+#include "ext2fsP.h"
 
 #ifdef __GNUC__
 #define ATTR(x) __attribute__(x)
@@ -135,7 +136,7 @@ struct undo_private_data {
 
 	ext2fs_block_bitmap written_block_map;
 	struct struct_ext2_filsys fake_fs;
-
+	char *tdb_file;
 	struct undo_header hdr;
 };
 #define KEYS_PER_BLOCK(d) (((d)->tdb_data_size / sizeof(struct undo_key)) - 1)
@@ -662,6 +663,17 @@ out:
 	return retval;
 }
 
+static void undo_atexit(void *p)
+{
+	struct undo_private_data *data = p;
+	errcode_t err;
+
+	err = write_undo_indexes(data);
+	io_channel_close(data->undo_file);
+
+	com_err(data->tdb_file, err, "while force-closing undo file");
+}
+
 static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 {
 	io_channel	io = NULL;
@@ -703,11 +715,16 @@ static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 		if (retval)
 			goto cleanup;
 
-		undo_fd = ext2fs_open_file(tdb_file, O_RDWR | O_CREAT, 0600);
+		data->tdb_file = strdup(tdb_file);
+		if (data->tdb_file == NULL)
+			goto cleanup;
+		undo_fd = ext2fs_open_file(data->tdb_file, O_RDWR | O_CREAT,
+					   0600);
 		if (undo_fd < 0)
 			goto cleanup;
 
-		retval = undo_io_backing_manager->open(tdb_file, IO_FLAG_RW,
+		retval = undo_io_backing_manager->open(data->tdb_file,
+						       IO_FLAG_RW,
 						       &data->undo_file);
 		if (retval)
 			goto cleanup;
@@ -732,6 +749,9 @@ static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 		if (retval)
 			goto cleanup;
 	}
+	retval = ext2fs_add_exit_fn(undo_atexit, data);
+	if (retval)
+		goto cleanup;
 
 	*channel = io;
 	if (undo_fd >= 0)
@@ -739,10 +759,13 @@ static errcode_t undo_open(const char *name, int flags, io_channel *channel)
 	return retval;
 
 cleanup:
+	ext2fs_remove_exit_fn(undo_atexit, data);
 	if (undo_fd >= 0)
 		close(undo_fd);
 	if (data && data->undo_file)
 		io_channel_close(data->undo_file);
+	if (data && data->tdb_file)
+		free(data->tdb_file);
 	if (data && data->real)
 		io_channel_close(data->real);
 	if (data)
@@ -769,11 +792,14 @@ static errcode_t undo_close(io_channel channel)
 	err = write_undo_indexes(data);
 	if (data->real)
 		retval = io_channel_close(data->real);
+	if (data->tdb_file)
+		free(data->tdb_file);
 	if (data->undo_file)
 		io_channel_close(data->undo_file);
 	ext2fs_free_mem(&data->keyb);
 	if (data->written_block_map)
 		ext2fs_free_generic_bitmap(data->written_block_map);
+	ext2fs_remove_exit_fn(undo_atexit, data);
 	ext2fs_free_mem(&channel->private_data);
 	if (channel->name)
 		ext2fs_free_mem(&channel->name);

[PATCH 35/54] e2fsck: optionally create an undo file

[Darrick J. Wong]

Provide the user with an option to create an undo file so that they
can roll back a failed repair operation.

v2: Support reopening undo files.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/e2fsck.8.in |   10 +++++
 e2fsck/e2fsck.h    |    3 ++
 e2fsck/unix.c      |   96 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 107 insertions(+), 2 deletions(-)


diff --git a/e2fsck/e2fsck.8.in b/e2fsck/e2fsck.8.in
index 6e6a2cd..364c7f1 100644
--- a/e2fsck/e2fsck.8.in
+++ b/e2fsck/e2fsck.8.in
@@ -334,6 +334,16 @@ may not be specified at the same time as the
 or
 .B \-p
 options.
+.TP
+.BI \-z " undo_file"
+Before overwriting a file system block, write the old contents of the block to
+an undo file.  This undo file can be used with e2undo(8) to restore the old
+contents of the file system should something go wrong.  If the empty string is
+passed as the undo_file argument, the undo file will be written to a file named
+e2fsck-\fIdevice\fR.e2undo in the directory specified via the
+\fIE2FSPROGS_UNDO_DIR\fR environment variable.
+
+WARNING: The undo file cannot be used to recover from a power or system crash.
 .SH EXIT CODE
 The exit code returned by
 .B e2fsck
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 15d968e..04f7bac 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -388,6 +388,9 @@ struct e2fsck_struct {
 	 * Inodes to rebuild extent trees
 	 */
 	ext2fs_inode_bitmap inodes_to_rebuild;
+
+	/* Undo file */
+	char *undo_file;
 };
 
 /* Data structures to evaluate whether an extent tree needs rebuilding. */
diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index b84bffe..d2ef50f 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -45,6 +45,7 @@ extern int optind;
 #ifdef HAVE_DIRENT_H
 #include <dirent.h>
 #endif
+#include <libgen.h>
 
 #include "e2p/e2p.h"
 #include "et/com_err.h"
@@ -75,7 +76,7 @@ static void usage(e2fsck_t ctx)
 		_("Usage: %s [-panyrcdfvtDFV] [-b superblock] [-B blocksize]\n"
 		"\t\t[-I inode_buffer_blocks] [-P process_inode_size]\n"
 		"\t\t[-l|-L bad_blocks_file] [-C fd] [-j external_journal]\n"
-		"\t\t[-E extended-options] device\n"),
+		"\t\t[-E extended-options] [-z undo_file] device\n"),
 		ctx->program_name);
 
 	fprintf(stderr, "%s", _("\nEmergency help:\n"
@@ -91,6 +92,7 @@ static void usage(e2fsck_t ctx)
 		" -j external_journal  Set location of the external journal\n"
 		" -l bad_blocks_file   Add to badblocks list\n"
 		" -L bad_blocks_file   Set badblocks list\n"
+		" -z undo_file         Create an undo file\n"
 		));
 
 	exit(FSCK_USAGE);
@@ -795,7 +797,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 
 	phys_mem_kb = get_memory_size() / 1024;
 	ctx->readahead_kb = ~0ULL;
-	while ((c = getopt (argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDk")) != EOF)
+	while ((c = getopt(argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF)
 		switch (c) {
 		case 'C':
 			ctx->progress = e2fsck_update_progress;
@@ -927,6 +929,9 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 		case 'k':
 			keep_bad_blocks++;
 			break;
+		case 'z':
+			ctx->undo_file = optarg;
+			break;
 		default:
 			usage(ctx);
 		}
@@ -1205,6 +1210,87 @@ check_error:
 	return retval;
 }
 
+static int e2fsck_setup_tdb(e2fsck_t ctx, io_manager *io_ptr)
+{
+	errcode_t retval = ENOMEM;
+	char *tdb_dir = NULL, *tdb_file = NULL;
+	char *dev_name, *tmp_name;
+	int free_tdb_dir = 0;
+
+	/* (re)open a specific undo file */
+	if (ctx->undo_file && ctx->undo_file[0] != 0) {
+		set_undo_io_backing_manager(*io_ptr);
+		*io_ptr = undo_io_manager;
+		retval = set_undo_io_backup_file(ctx->undo_file);
+		if (retval)
+			goto err;
+		printf(_("Overwriting existing filesystem; this can be undone "
+			 "using the command:\n"
+			 "    e2undo %s %s\n\n"),
+			ctx->undo_file, ctx->filesystem_name);
+		return 0;
+	}
+
+	/*
+	 * Configuration via a conf file would be
+	 * nice
+	 */
+	tdb_dir = getenv("E2FSPROGS_UNDO_DIR");
+	if (!tdb_dir) {
+		profile_get_string(ctx->profile, "defaults",
+				   "undo_dir", 0, "/var/lib/e2fsprogs",
+				   &tdb_dir);
+		free_tdb_dir = 1;
+	}
+
+	if (!strcmp(tdb_dir, "none") || (tdb_dir[0] == 0) ||
+	    access(tdb_dir, W_OK)) {
+		if (free_tdb_dir)
+			free(tdb_dir);
+		return 0;
+	}
+
+	tmp_name = strdup(ctx->filesystem_name);
+	if (!tmp_name)
+		goto errout;
+	dev_name = basename(tmp_name);
+	tdb_file = malloc(strlen(tdb_dir) + 8 + strlen(dev_name) + 7 + 1);
+	if (!tdb_file) {
+		free(tmp_name);
+		goto errout;
+	}
+	sprintf(tdb_file, "%s/e2fsck-%s.e2undo", tdb_dir, dev_name);
+	free(tmp_name);
+
+	if ((unlink(tdb_file) < 0) && (errno != ENOENT)) {
+		retval = errno;
+		goto errout;
+	}
+
+	set_undo_io_backing_manager(*io_ptr);
+	*io_ptr = undo_io_manager;
+	retval = set_undo_io_backup_file(tdb_file);
+	if (retval)
+		goto errout;
+	printf(_("Overwriting existing filesystem; this can be undone "
+		 "using the command:\n"
+		 "    e2undo %s %s\n\n"), tdb_file, ctx->filesystem_name);
+
+	if (free_tdb_dir)
+		free(tdb_dir);
+	free(tdb_file);
+	return 0;
+
+errout:
+	if (free_tdb_dir)
+		free(tdb_dir);
+	free(tdb_file);
+err:
+	com_err(ctx->program_name, retval, "%s",
+		_("while trying to setup undo file\n"));
+	return retval;
+}
+
 int main (int argc, char *argv[])
 {
 	errcode_t	retval = 0, retval2 = 0, orig_retval = 0;
@@ -1315,6 +1401,12 @@ restart:
 	} else
 		flags |= EXT2_FLAG_SOFTSUPP_FEATURES;
 
+	if (ctx->undo_file) {
+		retval = e2fsck_setup_tdb(ctx, &io_ptr);
+		if (retval)
+			exit(FSCK_ERROR);
+	}
+
 	ctx->openfs_flags = flags;
 	retval = try_open_fs(ctx, flags, io_ptr, &fs);
 

[PATCH 36/54] resize2fs: optionally create undo file

[Darrick J. Wong]

Provide the user with an option to create an undo file so that they
can roll back a failed resize operation.

v2: Allow reopening of undo files.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 resize/main.c         |   92 +++++++++++++++++++++++++++++++++++++++++++++++--
 resize/resize2fs.8.in |   14 +++++++
 2 files changed, 103 insertions(+), 3 deletions(-)


diff --git a/resize/main.c b/resize/main.c
index 5c4aa9f..5316d66 100644
--- a/resize/main.c
+++ b/resize/main.c
@@ -29,6 +29,7 @@ extern int optind;
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <fcntl.h>
+#include <libgen.h>
 
 #include "e2p/e2p.h"
 
@@ -42,7 +43,8 @@ static char *device_name, *io_options;
 static void usage (char *prog)
 {
 	fprintf (stderr, _("Usage: %s [-d debug_flags] [-f] [-F] [-M] [-P] "
-			   "[-p] device [-b|-s|new_size]\n\n"), prog);
+			   "[-p] device [-b|-s|new_size] [-z undo_file]\n\n"),
+		 prog);
 
 	exit (1);
 }
@@ -162,6 +164,82 @@ static void bigalloc_check(ext2_filsys fs, int force)
 	}
 }
 
+static int resize2fs_setup_tdb(const char *device_name, char *undo_file,
+			       io_manager *io_ptr)
+{
+	errcode_t retval = ENOMEM;
+	char *tdb_dir = NULL, *tdb_file = NULL;
+	char *dev_name, *tmp_name;
+	int free_tdb_dir = 0;
+
+	/* (re)open a specific undo file */
+	if (undo_file && undo_file[0] != 0) {
+		set_undo_io_backing_manager(*io_ptr);
+		*io_ptr = undo_io_manager;
+		retval = set_undo_io_backup_file(undo_file);
+		if (retval)
+			goto err;
+		printf(_("Overwriting existing filesystem; this can be undone "
+			 "using the command:\n"
+			 "    e2undo %s %s\n\n"),
+			undo_file, device_name);
+		return 0;
+	}
+
+	/*
+	 * Configuration via a conf file would be
+	 * nice
+	 */
+	tdb_dir = getenv("E2FSPROGS_UNDO_DIR");
+
+	if (tdb_dir == NULL || !strcmp(tdb_dir, "none") || (tdb_dir[0] == 0) ||
+	    access(tdb_dir, W_OK)) {
+		if (free_tdb_dir)
+			free(tdb_dir);
+		return 0;
+	}
+
+	tmp_name = strdup(device_name);
+	if (!tmp_name)
+		goto errout;
+	dev_name = basename(tmp_name);
+	tdb_file = malloc(strlen(tdb_dir) + 8 + strlen(dev_name) + 7 + 1);
+	if (!tdb_file) {
+		free(tmp_name);
+		goto errout;
+	}
+	sprintf(tdb_file, "%s/resize2fs-%s.e2undo", tdb_dir, dev_name);
+	free(tmp_name);
+
+	if ((unlink(tdb_file) < 0) && (errno != ENOENT)) {
+		retval = errno;
+		goto errout;
+	}
+
+	set_undo_io_backing_manager(*io_ptr);
+	*io_ptr = undo_io_manager;
+	retval = set_undo_io_backup_file(tdb_file);
+	if (retval)
+		goto errout;
+	printf(_("Overwriting existing filesystem; this can be undone "
+		 "using the command:\n"
+		 "    e2undo %s %s\n\n"), tdb_file, device_name);
+
+	if (free_tdb_dir)
+		free(tdb_dir);
+	free(tdb_file);
+	return 0;
+
+errout:
+	if (free_tdb_dir)
+		free(tdb_dir);
+	free(tdb_file);
+err:
+	com_err(program_name, retval, "%s",
+		_("while trying to setup undo file\n"));
+	return retval;
+}
+
 int main (int argc, char ** argv)
 {
 	errcode_t	retval;
@@ -186,7 +264,7 @@ int main (int argc, char ** argv)
 	unsigned int	blocksize;
 	long		sysval;
 	int		len, mount_flags;
-	char		*mtpt;
+	char		*mtpt, *undo_file = NULL;
 
 #ifdef ENABLE_NLS
 	setlocale(LC_MESSAGES, "");
@@ -203,7 +281,7 @@ int main (int argc, char ** argv)
 	if (argc && *argv)
 		program_name = *argv;
 
-	while ((c = getopt(argc, argv, "d:fFhMPpS:bs")) != EOF) {
+	while ((c = getopt(argc, argv, "d:fFhMPpS:bsz:")) != EOF) {
 		switch (c) {
 		case 'h':
 			usage(program_name);
@@ -235,6 +313,9 @@ int main (int argc, char ** argv)
 		case 's':
 			flags |= RESIZE_DISABLE_64BIT;
 			break;
+		case 'z':
+			undo_file = optarg;
+			break;
 		default:
 			usage(program_name);
 		}
@@ -319,6 +400,11 @@ int main (int argc, char ** argv)
 
 	io_flags |= EXT2_FLAG_64BITS;
 	io_flags |= EXT2_FLAG_NOFREE_ON_ERROR;
+	if (undo_file) {
+		retval = resize2fs_setup_tdb(device_name, undo_file, &io_ptr);
+		if (retval)
+			exit(1);
+	}
 	retval = ext2fs_open2(device_name, io_options, io_flags,
 			      0, 0, io_ptr, &fs);
 	if (retval == EXT2_ET_RO_UNSUPP_FEATURE &&
diff --git a/resize/resize2fs.8.in b/resize/resize2fs.8.in
index 0129bfc..d2738e9 100644
--- a/resize/resize2fs.8.in
+++ b/resize/resize2fs.8.in
@@ -18,6 +18,10 @@ resize2fs \- ext2/ext3/ext4 file system resizer
 .B \-S
 .I RAID-stride
 ]
+[
+.B \-z
+.I undo_file
+]
 .I device
 [
 .I size
@@ -149,6 +153,16 @@ The
 program will heuristically determine the RAID stride that was specified
 when the filesystem was created.  This option allows the user to
 explicitly specify a RAID stride setting to be used by resize2fs instead.
+.TP
+.BI \-z " undo_file"
+Before overwriting a file system block, write the old contents of the block to
+an undo file.  This undo file can be used with e2undo(8) to restore the old
+contents of the file system should something go wrong.  If the empty string is
+passed as the undo_file argument, the undo file will be written to a file named
+resize2fs-\fIdevice\fR.e2undo in the directory specified via the
+\fIE2FSPROGS_UNDO_DIR\fR environment variable.
+
+WARNING: The undo file cannot be used to recover from a power or system crash.
 .SH KNOWN BUGS
 The minimum size of the filesystem as estimated by resize2fs may be
 incorrect, especially for filesystems with 1k and 2k blocksizes.

[PATCH 37/54] tune2fs: optionally create undo file

[Darrick J. Wong]

Provide the user with an option to create an undo file so that they
can roll back a failed tuning operation.  Previously, one would be
created for inode resize if a bunch of (undocumented) conditions were
met.

v2: Enable re-opening of undo files.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/tune2fs.8.in |   14 ++++++++++++++
 misc/tune2fs.c    |   30 ++++++++++++++++++++++++++----
 2 files changed, 40 insertions(+), 4 deletions(-)


diff --git a/misc/tune2fs.8.in b/misc/tune2fs.8.in
index c50d475..f6a475d 100644
--- a/misc/tune2fs.8.in
+++ b/misc/tune2fs.8.in
@@ -88,6 +88,10 @@ tune2fs \- adjust tunable filesystem parameters on ext2/ext3/ext4 filesystems
 .B \-U
 .I UUID
 ]
+[
+.B \-z
+.I undo_file
+]
 device
 .SH DESCRIPTION
 .BI tune2fs
@@ -684,6 +688,16 @@ or
 .IR /dev/urandom ,
 .B tune2fs
 will automatically use a time-based UUID instead of a randomly-generated UUID.
+.TP
+.BI \-z " undo_file"
+Before overwriting a file system block, write the old contents of the block to
+an undo file.  This undo file can be used with e2undo(8) to restore the old
+contents of the file system should something go wrong.  If the empty string is
+passed as the undo_file argument, the undo file will be written to a file named
+tune2fs-\fIdevice\fR.e2undo in the directory specified via the
+\fIE2FSPROGS_UNDO_DIR\fR environment variable.
+
+WARNING: The undo file cannot be used to recover from a power or system crash.
 .SH BUGS
 We haven't found any bugs yet.  That doesn't mean there aren't any...
 .SH AUTHOR
diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index 66b2ba3..2a9d36d 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -99,6 +99,7 @@ static int usrquota, grpquota;
 static int rewrite_checksums;
 static int feature_64bit;
 static int fsck_requested;
+static char *undo_file;
 
 int journal_size, journal_flags;
 char *journal_device;
@@ -136,7 +137,8 @@ static void usage(void)
 		  "\t[-Q quota_options]\n"
 #endif
 		  "\t[-E extended-option[,...]] [-T last_check_time] "
-		  "[-U UUID]\n\t[ -I new_inode_size ] device\n"), program_name);
+		  "[-U UUID]\n\t[-I new_inode_size] [-z undo_file] device\n"),
+		program_name);
 	exit(1);
 }
 
@@ -465,6 +467,8 @@ static void convert_64bit(ext2_filsys fs, int direction)
 		fprintf(stderr, _("Please run `resize2fs %s %s"),
 			direction > 0 ? "-b" : "-s", fs->device_name);
 
+	if (undo_file)
+		fprintf(stderr, _(" -z \"%s\""), undo_file);
 	if (direction > 0)
 		fprintf(stderr, _("' to enable 64-bit mode.\n"));
 	else
@@ -1563,7 +1567,7 @@ static void parse_tune2fs_options(int argc, char **argv)
 	char *tmp;
 	struct group *gr;
 	struct passwd *pw;
-	char optstring[100] = "c:e:fg:i:jlm:o:r:s:u:C:E:I:J:L:M:O:T:U:";
+	char optstring[100] = "c:e:fg:i:jlm:o:r:s:u:C:E:I:J:L:M:O:T:U:z:";
 
 #ifdef CONFIG_QUOTA
 	strcat(optstring, "Q:");
@@ -1797,6 +1801,9 @@ static void parse_tune2fs_options(int argc, char **argv)
 			open_flag = EXT2_FLAG_RW;
 			I_flag = 1;
 			break;
+		case 'z':
+			undo_file = optarg;
+			break;
 		default:
 			usage();
 		}
@@ -2517,6 +2524,17 @@ static int tune2fs_setup_tdb(const char *name, io_manager *io_ptr)
 	char *tdb_file;
 	char *dev_name, *tmp_name;
 
+	/* (re)open a specific undo file */
+	if (undo_file && undo_file[0] != 0) {
+		set_undo_io_backing_manager(*io_ptr);
+		*io_ptr = undo_io_manager;
+		set_undo_io_backup_file(undo_file);
+		printf(_("To undo the tune2fs operation please run "
+			 "the command\n    e2undo %s %s\n\n"),
+			 undo_file, name);
+		return retval;
+	}
+
 #if 0 /* FIXME!! */
 	/*
 	 * Configuration via a conf file would be
@@ -2712,7 +2730,7 @@ retry_open:
 	}
 	fs->default_bitmap_type = EXT2FS_BMAP64_RBTREE;
 
-	if (I_flag && !io_ptr_orig) {
+	if (I_flag) {
 		/*
 		 * Check the inode size is right so we can issue an
 		 * error message and bail before setting up the tdb
@@ -2736,11 +2754,15 @@ retry_open:
 			rc = 1;
 			goto closefs;
 		}

[PATCH 38/54] mke2fs: optionally create undo file

[Darrick J. Wong]

Provide the user with an option to create an undo file so that they
can roll back a failed tuning operation.  Previously, one would be
created if force_undo was set in the configuration file and a bunch of
(undocumented) conditions were met.

v2: Support reopening undo files.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/mke2fs.8.in |   15 +++++++++++++++
 misc/mke2fs.c    |   25 ++++++++++++++++++++++---
 2 files changed, 37 insertions(+), 3 deletions(-)


diff --git a/misc/mke2fs.8.in b/misc/mke2fs.8.in
index aeb5caf..3230f65 100644
--- a/misc/mke2fs.8.in
+++ b/misc/mke2fs.8.in
@@ -117,6 +117,10 @@ mke2fs \- create an ext2/ext3/ext4 filesystem
 .B \-e
 .I errors-behavior
 ]
+[
+.B \-z
+.I undo_file
+]
 .I device
 [
 .I fs-size
@@ -738,6 +742,17 @@ Verbose execution.
 Print the version number of
 .B mke2fs
 and exit.
+.TP
+.BI \-z " undo_file"
+Before overwriting a file system block, write the old contents of the block to
+an undo file.  This undo file can be used with e2undo(8) to restore the old
+contents of the file system should something go wrong.  If the empty string is
+passed as the undo_file argument, the undo file will be written to a file named
+mke2fs-\fIdevice\fR.e2undo in the directory specified via the
+\fIE2FSPROGS_UNDO_DIR\fR environment variable or the \fIundo_dir\fR directive
+in the configuration file.
+
+WARNING: The undo file cannot be used to recover from a power or system crash.
 .SH ENVIRONMENT
 .TP
 .BI MKE2FS_SYNC
diff --git a/misc/mke2fs.c b/misc/mke2fs.c
index aeb852f..47b0b29 100644
--- a/misc/mke2fs.c
+++ b/misc/mke2fs.c
@@ -110,6 +110,7 @@ char *journal_device;
 static int sync_kludge;	/* Set using the MKE2FS_SYNC env. option */
 char **fs_types;
 const char *root_dir;  /* Copy files from the specified directory */
+static char *undo_file;
 
 static profile_t	profile;
 
@@ -129,7 +130,8 @@ static void usage(void)
 	"[-M last-mounted-directory]\n\t[-O feature[,...]] "
 	"[-r fs-revision] [-E extended-option[,...]]\n"
 	"\t[-t fs-type] [-T usage-type ] [-U UUID] [-e errors_behavior]"
-	"[-jnqvDFKSV] device [blocks-count]\n"),
+	"[-z undo_file]\n"
+	"\t[-jnqvDFKSV] device [blocks-count]\n"),
 		program_name);
 	exit(1);
 }
@@ -1551,7 +1553,7 @@ profile_error:
 	}
 
 	while ((c = getopt (argc, argv,
-		    "b:ce:g:i:jl:m:no:qr:s:t:d:vC:DE:FG:I:J:KL:M:N:O:R:ST:U:V")) != EOF) {
+		    "b:ce:g:i:jl:m:no:qr:s:t:d:vC:DE:FG:I:J:KL:M:N:O:R:ST:U:Vz:")) != EOF) {
 		switch (c) {
 		case 'b':
 			blocksize = parse_num_blocks2(optarg, -1);
@@ -1774,6 +1776,9 @@ profile_error:
 			/* Print version number and exit */
 			show_version_only++;
 			break;
+		case 'z':
+			undo_file = optarg;
+			break;
 		default:
 			usage();
 		}
@@ -2492,6 +2497,19 @@ static int mke2fs_setup_tdb(const char *name, io_manager *io_ptr)
 	char *dev_name, *tmp_name;
 	int free_tdb_dir = 0;
 
+	/* (re)open a specific undo file */
+	if (undo_file && undo_file[0] != 0) {
+		set_undo_io_backing_manager(*io_ptr);
+		*io_ptr = undo_io_manager;
+		retval = set_undo_io_backup_file(undo_file);
+		if (retval)
+			goto err;
+		printf(_("Overwriting existing filesystem; this can be undone "
+			 "using the command:\n"
+			 "    e2undo %s %s\n\n"), undo_file, name);
+		return 0;
+	}
+
 	/*
 	 * Configuration via a conf file would be
 	 * nice
@@ -2546,6 +2564,7 @@ errout:
 	if (free_tdb_dir)
 		free(tdb_dir);
 	free(tdb_file);
+err:
 	com_err(program_name, retval, "%s",
 		_("while trying to setup undo file\n"));
 	return retval;
@@ -2717,7 +2736,7 @@ int main (int argc, char *argv[])
 #endif
 		io_ptr = unix_io_manager;
 
-	if (should_do_undo(device_name)) {
+	if (undo_file != NULL || should_do_undo(device_name)) {
 		retval = mke2fs_setup_tdb(device_name, &io_ptr);
 		if (retval)
 			exit(1);

[PATCH 39/54] debugfs: optionally create undo file

[Darrick J. Wong]

Provide the user with an option to create an undo file so that they
can roll back a failed debugfs expedition.

v2: Support reopening undo files.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debugfs/debugfs.8.in |   16 +++++++-
 debugfs/debugfs.c    |  105 +++++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 113 insertions(+), 8 deletions(-)


diff --git a/debugfs/debugfs.8.in b/debugfs/debugfs.8.in
index 04c280d..81899a3 100644
--- a/debugfs/debugfs.8.in
+++ b/debugfs/debugfs.8.in
@@ -31,6 +31,10 @@ request
 data_source_device
 ]
 [
+.B \-z
+.I undo_file
+]
+[
 device
 ]
 .SH DESCRIPTION
@@ -130,6 +134,16 @@ and then exit.
 print the version number of
 .B debugfs
 and exit.
+.TP
+.BI \-z " undo_file"
+Before overwriting a file system block, write the old contents of the block to
+an undo file.  This undo file can be used with e2undo(8) to restore the old
+contents of the file system should something go wrong.  If the empty string is
+passed as the undo_file argument, the undo file will be written to a file named
+resize2fs-\fIdevice\fR.e2undo in the directory specified via the
+\fIE2FSPROGS_UNDO_DIR\fR environment variable.
+
+WARNING: The undo file cannot be used to recover from a power or system crash.
 .SH SPECIFYING FILES
 Many
 .B debugfs
@@ -532,7 +546,7 @@ to those inodes.  The
 flag will enable checking the file type information in the directory
 entry to make sure it matches the inode's type.
 .TP
-.BI open " [-weficD] [-b blocksize] [-s superblock] device"
+.BI open " [-weficD] [-b blocksize] [-s superblock] [-z undo_file] device"
 Open a filesystem for editing.  The
 .I -f
 flag forces the filesystem to be opened even if there are some unknown
diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
index fe57366..4b88f73 100644
--- a/debugfs/debugfs.c
+++ b/debugfs/debugfs.c
@@ -15,6 +15,7 @@
 #include <ctype.h>
 #include <string.h>
 #include <time.h>
+#include <libgen.h>
 #ifdef HAVE_GETOPT_H
 #include <getopt.h>
 #else
@@ -48,12 +49,88 @@ ext2_filsys	current_fs;
 quota_ctx_t	current_qctx;
 ext2_ino_t	root, cwd;
 
+static int debugfs_setup_tdb(const char *device_name, char *undo_file,
+			     io_manager *io_ptr)
+{
+	errcode_t retval = ENOMEM;
+	char *tdb_dir = NULL, *tdb_file = NULL;
+	char *dev_name, *tmp_name;
+	int free_tdb_dir = 0;
+
+	/* (re)open a specific undo file */
+	if (undo_file && undo_file[0] != 0) {
+		set_undo_io_backing_manager(*io_ptr);
+		*io_ptr = undo_io_manager;
+		retval = set_undo_io_backup_file(undo_file);
+		if (retval)
+			goto err;
+		printf("Overwriting existing filesystem; this can be undone "
+			"using the command:\n"
+			"    e2undo %s %s\n\n",
+			undo_file, device_name);
+		return 0;
+	}
+
+	/*
+	 * Configuration via a conf file would be
+	 * nice
+	 */
+	tdb_dir = getenv("E2FSPROGS_UNDO_DIR");
+
+	if (tdb_dir == NULL || !strcmp(tdb_dir, "none") || (tdb_dir[0] == 0) ||
+	    access(tdb_dir, W_OK)) {
+		if (free_tdb_dir)
+			free(tdb_dir);
+		return 0;
+	}
+
+	tmp_name = strdup(device_name);
+	if (!tmp_name)
+		goto errout;
+	dev_name = basename(tmp_name);
+	tdb_file = malloc(strlen(tdb_dir) + 8 + strlen(dev_name) + 7 + 1);
+	if (!tdb_file) {
+		free(tmp_name);
+		goto errout;
+	}
+	sprintf(tdb_file, "%s/debugfs-%s.e2undo", tdb_dir, dev_name);
+	free(tmp_name);
+
+	if ((unlink(tdb_file) < 0) && (errno != ENOENT)) {
+		retval = errno;
+		goto errout;
+	}
+
+	set_undo_io_backing_manager(*io_ptr);
+	*io_ptr = undo_io_manager;
+	retval = set_undo_io_backup_file(tdb_file);
+	if (retval)
+		goto errout;
+	printf("Overwriting existing filesystem; this can be undone "
+		"using the command:\n"
+		"    e2undo %s %s\n\n", tdb_file, device_name);
+
+	if (free_tdb_dir)
+		free(tdb_dir);
+	free(tdb_file);
+	return 0;
+
+errout:
+	if (free_tdb_dir)
+		free(tdb_dir);
+	free(tdb_file);
+err:
+	com_err("debugfs", retval, "while trying to setup undo file\n");
+	return retval;
+}
+
 static void open_filesystem(char *device, int open_flags, blk64_t superblock,
 			    blk64_t blocksize, int catastrophic,
-			    char *data_filename)
+			    char *data_filename, char *undo_file)
 {
 	int	retval;
 	io_channel data_io = 0;
+	io_manager io_ptr = unix_io_manager;
 
 	if (superblock != 0 && blocksize == 0) {
 		com_err(device, 0, "if you specify the superblock, you must also specify the block size");
@@ -84,8 +161,14 @@ static void open_filesystem(char *device, int open_flags, blk64_t superblock,
 	if (catastrophic)
 		open_flags |= EXT2_FLAG_SKIP_MMP;
 
+	if (undo_file) {
+		retval = debugfs_setup_tdb(device, undo_file, &io_ptr);
+		if (retval)
+			exit(1);
+	}
+
 	retval = ext2fs_open(device, open_flags, superblock, blocksize,
-			     unix_io_manager, &current_fs);
+			     io_ptr, &current_fs);
 	if (retval) {
 		com_err(device, retval, "while opening filesystem");
 		if (retval == EXT2_ET_BAD_MAGIC)
@@ -136,9 +219,10 @@ void do_open_filesys(int argc, char **argv)
 	blk64_t	blocksize = 0;
 	int	open_flags = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS; 
 	char	*data_filename = 0;
+	char	*undo_file = NULL;
 
 	reset_getopt();
-	while ((c = getopt (argc, argv, "iwfecb:s:d:D")) != EOF) {
+	while ((c = getopt(argc, argv, "iwfecb:s:d:Dz:")) != EOF) {
 		switch (c) {
 		case 'i':
 			open_flags |= EXT2_FLAG_IMAGE_FILE;
@@ -177,6 +261,9 @@ void do_open_filesys(int argc, char **argv)
 			if (err)
 				return;
 			break;
+		case 'z':
+			undo_file = optarg;
+			break;
 		default:
 			goto print_usage;
 		}
@@ -188,7 +275,7 @@ void do_open_filesys(int argc, char **argv)
 		return;
 	open_filesystem(argv[optind], open_flags,
 			superblock, blocksize, catastrophic,
-			data_filename);
+			data_filename, undo_file);
 	return;
 
 print_usage:
@@ -2219,7 +2306,7 @@ int main(int argc, char **argv)
 		"Usage: %s [-b blocksize] [-s superblock] [-f cmd_file] "
 		"[-R request] [-V] ["
 #ifndef READ_ONLY
-		"[-w] "
+		"[-w] [-z undo_file] "
 #endif
 		"[-c] device]";
 	int		c;
@@ -2234,7 +2321,8 @@ int main(int argc, char **argv)
 #ifdef READ_ONLY
 	const char	*opt_string = "nicR:f:b:s:Vd:D";
 #else
-	const char	*opt_string = "niwcR:f:b:s:Vd:D";
+	const char	*opt_string = "niwcR:f:b:s:Vd:Dz:";
+	char		*undo_file = NULL;
 #endif
 
 	if (debug_prog_name == 0)
@@ -2291,6 +2379,9 @@ int main(int argc, char **argv)
 			fprintf(stderr, "\tUsing %s\n",
 				error_message(EXT2_ET_BASE));
 			exit(0);
+		case 'z':
+			undo_file = optarg;
+			break;
 		default:
 			com_err(argv[0], 0, usage, debug_prog_name);
 			return 1;
@@ -2299,7 +2390,7 @@ int main(int argc, char **argv)
 	if (optind < argc)
 		open_filesystem(argv[optind], open_flags,
 				superblock, blocksize, catastrophic,
-				data_filename);
+				data_filename, undo_file);
 
 	sci_idx = ss_create_invocation(debug_prog_name, "0.0", (char *) NULL,
 				       &debug_cmds, &retval);

[PATCH 40/54] tests: test undo file creation in e2fsck/resize2fs/tune2fs/mke2fs

[Darrick J. Wong]

Regression tests to ensure that we can create undo files and roll
things back if need be.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/test_config                           |    1 
 tests/u_compound_rollback/script            |   62 ++++++++++++
 tests/u_debugfs_opt/script                  |   32 ++++++
 tests/u_e2fsck_opt/script                   |   32 ++++++
 tests/u_mke2fs/script                       |    4 -
 tests/u_mke2fs_opt/script                   |   32 ++++++
 tests/u_mke2fs_opt_oddsize/script           |   31 ++++++
 tests/u_resize2fs_opt/script                |   32 ++++++
 tests/u_revert_upgrade_to_64bitmcsum/script |  136 +++++++++++++++++++++++++++
 tests/u_tune2fs/script                      |    4 -
 tests/u_tune2fs_opt/script                  |   32 ++++++
 11 files changed, 394 insertions(+), 4 deletions(-)
 create mode 100644 tests/u_compound_rollback/script
 create mode 100644 tests/u_debugfs_opt/script
 create mode 100644 tests/u_e2fsck_opt/script
 create mode 100644 tests/u_mke2fs_opt/script
 create mode 100644 tests/u_mke2fs_opt_oddsize/script
 create mode 100644 tests/u_resize2fs_opt/script
 create mode 100644 tests/u_revert_upgrade_to_64bitmcsum/script
 create mode 100644 tests/u_tune2fs_opt/script


diff --git a/tests/test_config b/tests/test_config
index 2e3af6b..7f39157 100644
--- a/tests/test_config
+++ b/tests/test_config
@@ -17,6 +17,7 @@ TEST_BITS="../debugfs/debugfs"
 RESIZE2FS_EXE="../resize/resize2fs"
 RESIZE2FS="$USE_VALGRIND $RESIZE2FS_EXE"
 E2UNDO_EXE="../misc/e2undo"
+E2UNDO="$USE_VALGRIND $E2UNDO_EXE"
 TEST_REL=../tests/progs/test_rel
 TEST_ICOUNT=../tests/progs/test_icount
 CRCSUM=../tests/progs/crcsum
diff --git a/tests/u_compound_rollback/script b/tests/u_compound_rollback/script
new file mode 100644
index 0000000..0c1fbcc
--- /dev/null
+++ b/tests/u_compound_rollback/script
@@ -0,0 +1,62 @@
+test_description="e2undo with mke2fs/tune2fs/resize2fs/e2fsck -z"
+if test -x $RESIZE2FS_EXE -a -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/resize2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+echo compound e2undo rollback test > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 -z $TDB_FILE.0 $TMPFILE 256 >> $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 -z $TDB_FILE.0 $TMPFILE 256 >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE.1 $TMPFILE >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc2 >> $OUT
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -z $TDB_FILE.2 $TMPFILE 512 >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc3 >> $OUT
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D -z $TDB_FILE.3 $TMPFILE >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc4 >> $OUT
+
+echo roll back e2fsck >> $OUT
+$E2UNDO $TDB_FILE.3 $TMPFILE  >> $OUT 2>&1
+crc3_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo e2fsck $crc3_2 >> $OUT
+
+echo roll back resize2fs >> $OUT
+$E2UNDO $TDB_FILE.2 $TMPFILE  >> $OUT 2>&1
+crc2_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo resize2fs $crc2_2 >> $OUT
+
+echo roll back tune2fs >> $OUT
+$E2UNDO $TDB_FILE.1 $TMPFILE  >> $OUT 2>&1
+crc1_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo tune2fs $crc1_2 >> $OUT
+
+echo roll back mke2fs >> $OUT
+$E2UNDO $TDB_FILE.0 $TMPFILE  >> $OUT 2>&1
+crc0_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo mke2fs $crc0_2 >> $OUT
+
+if [ $crc0 = $crc0_2 ] && [ $crc1 = $crc1_2 ] && [ $crc2 = $crc2_2 ] && [ $crc3 = $crc3_2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE.0 $TDB_FILE.1 $TDB_FILE.2 $TDB_FILE.3 $TMPFILE
+fi
diff --git a/tests/u_debugfs_opt/script b/tests/u_debugfs_opt/script
new file mode 100644
index 0000000..bb93917
--- /dev/null
+++ b/tests/u_debugfs_opt/script
@@ -0,0 +1,32 @@
+test_description="e2undo with debugfs -z"
+if test -x $E2UNDO_EXE -a -x $DEBUGFS_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before debugfs $crc0 >> $OUT
+
+echo using debugfs to test e2undo >> $OUT
+$DEBUGFS -w -z $TDB_FILE -R 'zap -p 0x55 0' $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after debugfs $crc1 >> $OUT
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc0 = $crc2 ] && [ $crc1 != $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_e2fsck_opt/script b/tests/u_e2fsck_opt/script
new file mode 100644
index 0000000..d61cd2b
--- /dev/null
+++ b/tests/u_e2fsck_opt/script
@@ -0,0 +1,32 @@
+test_description="e2undo with e2fsck -z"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/e2fsck-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before e2fsck $crc0 >> $OUT
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -z $TDB_FILE $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc1 >> $OUT
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc0 = $crc2 ] && [ $crc1 != $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_mke2fs/script b/tests/u_mke2fs/script
index d249ddd..f1041a9 100644
--- a/tests/u_mke2fs/script
+++ b/tests/u_mke2fs/script
@@ -1,7 +1,7 @@
 test_description="e2undo with mke2fs"
 if test -x $E2UNDO_EXE; then
 
-E2FSPROGS_UNDO_DIR=/tmp
+E2FSPROGS_UNDO_DIR=${TMPDIR:-/tmp}
 export E2FSPROGS_UNDO_DIR
 TDB_FILE=$E2FSPROGS_UNDO_DIR/mke2fs-$(basename $TMPFILE).e2undo
 OUT=$test_name.log
@@ -19,7 +19,7 @@ $MKE2FS -q -F -o Linux -I 256 -O uninit_bg -E lazy_itable_init=1 -b 1024 $TMPFIL
 new_crc=`$CRCSUM $TMPFILE`
 echo $CRCSUM after mke2fs $new_crc >> $OUT
 
-$E2UNDO_EXE  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
 new_crc=`$CRCSUM $TMPFILE`
 echo $CRCSUM after e2undo $new_crc >> $OUT
 
diff --git a/tests/u_mke2fs_opt/script b/tests/u_mke2fs_opt/script
new file mode 100644
index 0000000..db62ab2
--- /dev/null
+++ b/tests/u_mke2fs_opt/script
@@ -0,0 +1,32 @@
+test_description="e2undo with mke2fs -z"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/mke2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -F -o Linux -I 128 -b 1024 test.img  > $OUT
+$MKE2FS -F -o Linux -I 128 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+echo using mke2fs to test e2undo >> $OUT
+$MKE2FS -q -F -o Linux -T ext4 -E lazy_itable_init=1 -b 1024 -z $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc0 = $crc2 ] && [ $crc1 != $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_mke2fs_opt_oddsize/script b/tests/u_mke2fs_opt_oddsize/script
new file mode 100644
index 0000000..23e0b9e
--- /dev/null
+++ b/tests/u_mke2fs_opt_oddsize/script
@@ -0,0 +1,31 @@
+test_description="e2undo with mke2fs -z and non-32k-aligned bdev size"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/mke2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+yes "abc123abc123abc" | dd bs=1k count=8 >> $TMPFILE 2> /dev/null
+
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 > $OUT
+
+echo using mke2fs to test e2undo >> $OUT
+$MKE2FS -q -F -o Linux -T ext4 -E lazy_itable_init=1 -b 1024 -z $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc0 = $crc2 ] && [ $crc1 != $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_resize2fs_opt/script b/tests/u_resize2fs_opt/script
new file mode 100644
index 0000000..fe1e04d
--- /dev/null
+++ b/tests/u_resize2fs_opt/script
@@ -0,0 +1,32 @@
+test_description="e2undo with resize2fs -z"
+if test -x $RESIZE2FS_EXE -a -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/resize2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE 256 > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE 256 >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before resize2fs $crc0 >> $OUT
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -z $TDB_FILE $TMPFILE 512 >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc1 >> $OUT
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc0 = $crc2 ] && [ $crc1 != $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_revert_upgrade_to_64bitmcsum/script b/tests/u_revert_upgrade_to_64bitmcsum/script
new file mode 100644
index 0000000..6120d00
--- /dev/null
+++ b/tests/u_revert_upgrade_to_64bitmcsum/script
@@ -0,0 +1,136 @@
+test_description="convert fs to 64bit,metadata_csum and revert both changes"
+if test -x $RESIZE2FS_EXE -a -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/resize2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+fail=0
+
+echo convert fs to 64bit,metadata_csum and revert both changes > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+CONF=$TMPFILE.conf
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode
+		blocksize = 4096
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo mke2fs -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 4096 -z $TDB_FILE.0 $TMPFILE 524288 >> $OUT
+MKE2FS_CONFIG=$CONF $MKE2FS -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 4096 -z $TDB_FILE.0 $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -gt 0 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have 64bit or metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -z $TDB_FILE.1 -b $TMPFILE >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE.2 $TMPFILE >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc3 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -lt 1 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit and metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D -z $TDB_FILE.3 $TMPFILE >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc4 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -lt 1 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit and metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back e2fsck >> $OUT
+$E2UNDO $TDB_FILE.3 $TMPFILE  >> $OUT 2>&1
+crc3_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo e2fsck $crc3_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -lt 1 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit and metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back tune2fs >> $OUT
+$E2UNDO $TDB_FILE.2 $TMPFILE  >> $OUT 2>&1
+crc2_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo tune2fs $crc2_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back resize2fs >> $OUT
+$E2UNDO $TDB_FILE.1 $TMPFILE  >> $OUT 2>&1
+crc1_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo resize2fs $crc1_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -gt 0 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have 64bit or metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back mke2fs >> $OUT
+$E2UNDO $TDB_FILE.0 $TMPFILE  >> $OUT 2>&1
+crc0_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo mke2fs $crc0_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ -n "${features}" ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have any features set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 && fail=1
+
+if [ $fail -eq 0 ] && [ $crc0 = $crc0_2 ] && [ $crc1 = $crc1_2 ] && [ $crc2 = $crc2_2 ] && [ $crc3 = $crc3_2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE.0 $TDB_FILE.1 $TDB_FILE.2 $TDB_FILE.3 $TMPFILE $CONF
+fi
diff --git a/tests/u_tune2fs/script b/tests/u_tune2fs/script
index a443f5a..aa5f379 100644
--- a/tests/u_tune2fs/script
+++ b/tests/u_tune2fs/script
@@ -1,7 +1,7 @@
 test_description="e2undo with tune2fs"
 if test -x $E2UNDO_EXE; then
 
-E2FSPROGS_UNDO_DIR=/tmp
+E2FSPROGS_UNDO_DIR=${TMPDIR:-/tmp}
 export E2FSPROGS_UNDO_DIR
 TDB_FILE=$E2FSPROGS_UNDO_DIR/tune2fs-$(basename $TMPFILE).e2undo
 OUT=$test_name.log
@@ -19,7 +19,7 @@ $TUNE2FS -I 256 $TMPFILE  >> $OUT 2>&1
 new_crc=`$CRCSUM $TMPFILE`
 echo $CRCSUM after tune2fs $new_crc >> $OUT
 
-$E2UNDO_EXE  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
 new_crc=`$CRCSUM $TMPFILE`
 echo $CRCSUM after e2undo $new_crc >> $OUT
 
diff --git a/tests/u_tune2fs_opt/script b/tests/u_tune2fs_opt/script
new file mode 100644
index 0000000..c4810b9
--- /dev/null
+++ b/tests/u_tune2fs_opt/script
@@ -0,0 +1,32 @@
+test_description="e2undo with tune2fs -z"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc0 = $crc2 ] && [ $crc1 != $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi

[PATCH 41/54] tests: test various features of the new e2undo format

[Darrick J. Wong]

Verify that the header, checksum, and wrong-order rollback detection
features of the new e2undo actually work.

v2: Collect more tests for the v2 of the e2undo flat file patch.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/e2fuzz.sh                           |    0 
 tests/u_compound_bad_rollback/script     |   62 ++++++++++++++++
 tests/u_corrupt_blk_csum/script          |   38 ++++++++++
 tests/u_corrupt_blk_csum_force/script    |   38 ++++++++++
 tests/u_corrupt_hdr_csum/script          |   37 ++++++++++
 tests/u_corrupt_key_csum/script          |   37 ++++++++++
 tests/u_dryrun/script                    |   32 ++++++++
 tests/u_errorout/script                  |   49 +++++++++++++
 tests/u_force/script                     |   40 ++++++++++
 tests/u_force_dryrun/script              |   38 ++++++++++
 tests/u_incomplete/script                |   38 ++++++++++
 tests/u_not_undo/script                  |   28 +++++++
 tests/u_onefile_bad/script               |  115 ++++++++++++++++++++++++++++++
 tests/u_revert_64bitmcsum_onefile/script |  112 +++++++++++++++++++++++++++++
 tests/u_revert_all_onefile/script        |  100 ++++++++++++++++++++++++++
 tests/u_undo_undo/script                 |   54 ++++++++++++++
 tests/u_wrong_fs/script                  |   36 +++++++++
 17 files changed, 854 insertions(+)
 mode change 100755 => 100644 misc/e2fuzz.sh
 create mode 100644 tests/u_compound_bad_rollback/script
 create mode 100644 tests/u_corrupt_blk_csum/script
 create mode 100644 tests/u_corrupt_blk_csum_force/script
 create mode 100644 tests/u_corrupt_hdr_csum/script
 create mode 100644 tests/u_corrupt_key_csum/script
 create mode 100644 tests/u_dryrun/script
 create mode 100644 tests/u_errorout/script
 create mode 100644 tests/u_force/script
 create mode 100644 tests/u_force_dryrun/script
 create mode 100644 tests/u_incomplete/script
 create mode 100644 tests/u_not_undo/script
 create mode 100644 tests/u_onefile_bad/script
 create mode 100644 tests/u_revert_64bitmcsum_onefile/script
 create mode 100644 tests/u_revert_all_onefile/script
 create mode 100644 tests/u_undo_undo/script
 create mode 100644 tests/u_wrong_fs/script


diff --git a/misc/e2fuzz.sh b/misc/e2fuzz.sh
old mode 100755
new mode 100644
diff --git a/tests/u_compound_bad_rollback/script b/tests/u_compound_bad_rollback/script
new file mode 100644
index 0000000..f54da7f
--- /dev/null
+++ b/tests/u_compound_bad_rollback/script
@@ -0,0 +1,62 @@
+test_description="e2undo with mke2fs/tune2fs/resize2fs/e2fsck -z"
+if test -x $RESIZE2FS_EXE -a -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/resize2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+echo compound e2undo rollback test > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 -z $TDB_FILE.0 $TMPFILE 256 >> $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 -z $TDB_FILE.0 $TMPFILE 256 >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE.1 $TMPFILE >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc2 >> $OUT
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -z $TDB_FILE.2 $TMPFILE 512 >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc3 >> $OUT
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D -z $TDB_FILE.3 $TMPFILE >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc4 >> $OUT
+
+echo roll back mke2fs >> $OUT
+$E2UNDO $TDB_FILE.0 $TMPFILE  >> $OUT 2>&1
+crc0_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo mke2fs $crc0_2 >> $OUT
+
+echo roll back tune2fs >> $OUT
+$E2UNDO $TDB_FILE.1 $TMPFILE  >> $OUT 2>&1
+crc1_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo tune2fs $crc1_2 >> $OUT
+
+echo roll back resize2fs >> $OUT
+$E2UNDO $TDB_FILE.2 $TMPFILE  >> $OUT 2>&1
+crc2_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo resize2fs $crc2_2 >> $OUT
+
+echo roll back e2fsck >> $OUT
+$E2UNDO $TDB_FILE.3 $TMPFILE  >> $OUT 2>&1
+crc3_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo e2fsck $crc3_2 >> $OUT
+
+if [ $crc4 = $crc0_2 ] && [ $crc4 = $crc1_2 ] && [ $crc4 = $crc2_2 ] && [ $crc3 = $crc3_2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE.0 $TDB_FILE.1 $TDB_FILE.2 $TDB_FILE.3 $TMPFILE
+fi
diff --git a/tests/u_corrupt_blk_csum/script b/tests/u_corrupt_blk_csum/script
new file mode 100644
index 0000000..ee16552
--- /dev/null
+++ b/tests/u_corrupt_blk_csum/script
@@ -0,0 +1,38 @@
+test_description="corrupt e2undo block data"
+if test -x $E2UNDO_EXE; then
+
+E2FSPROGS_UNDO_DIR=${TMPDIR:-/tmp}
+export E2FSPROGS_UNDO_DIR
+TDB_FILE=$E2FSPROGS_UNDO_DIR/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -I 128 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -I 256 $TMPFILE  >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+undo_blks=$(( $(stat -c '%s' $TDB_FILE 2>/dev/null || stat -f '%z' $TDB_FILE 2>/dev/null) / 1024 ))
+dd if=/dev/zero of=$TDB_FILE bs=1024 count=1 seek=$((undo_blks - 2)) conv=notrunc > /dev/null 2>&1
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+res=$?
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $res -ne 0 ] && [ $crc2 = $crc1 ] && [ $crc2 != $crc0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_corrupt_blk_csum_force/script b/tests/u_corrupt_blk_csum_force/script
new file mode 100644
index 0000000..ba82726
--- /dev/null
+++ b/tests/u_corrupt_blk_csum_force/script
@@ -0,0 +1,38 @@
+test_description="force replay of corrupt e2undo block data"
+if test -x $E2UNDO_EXE; then
+
+E2FSPROGS_UNDO_DIR=${TMPDIR:-/tmp}
+export E2FSPROGS_UNDO_DIR
+TDB_FILE=$E2FSPROGS_UNDO_DIR/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -I 128 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -I 256 $TMPFILE  >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+undo_blks=$(( $(stat -c '%s' $TDB_FILE 2>/dev/null || stat -f '%z' $TDB_FILE 2>/dev/null) / 1024 ))
+dd if=/dev/zero of=$TDB_FILE bs=1024 count=1 seek=$((undo_blks - 2)) conv=notrunc > /dev/null 2>&1
+
+$E2UNDO -f $TDB_FILE $TMPFILE  >> $OUT 2>&1
+res=$?
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc2 != $crc1 ] && [ $crc2 != $crc0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_corrupt_hdr_csum/script b/tests/u_corrupt_hdr_csum/script
new file mode 100644
index 0000000..32c38c8
--- /dev/null
+++ b/tests/u_corrupt_hdr_csum/script
@@ -0,0 +1,37 @@
+test_description="corrupt e2undo header"
+if test -x $E2UNDO_EXE; then
+
+E2FSPROGS_UNDO_DIR=${TMPDIR:-/tmp}
+export E2FSPROGS_UNDO_DIR
+TDB_FILE=$E2FSPROGS_UNDO_DIR/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -I 128 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -I 256 $TMPFILE  >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+dd if=/dev/zero of=$TDB_FILE bs=256 count=1 seek=1 conv=notrunc > /dev/null 2>&1
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+res=$?
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $res -ne 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_corrupt_key_csum/script b/tests/u_corrupt_key_csum/script
new file mode 100644
index 0000000..d07556b
--- /dev/null
+++ b/tests/u_corrupt_key_csum/script
@@ -0,0 +1,37 @@
+test_description="corrupt e2undo key data"
+if test -x $E2UNDO_EXE; then
+
+E2FSPROGS_UNDO_DIR=${TMPDIR:-/tmp}
+export E2FSPROGS_UNDO_DIR
+TDB_FILE=$E2FSPROGS_UNDO_DIR/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -I 128 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -I 256 $TMPFILE  >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+undo_blks=$(( $(stat -c '%s' $TDB_FILE 2>/dev/null || stat -f '%z' $TDB_FILE 2>/dev/null) / 1024 ))
+dd if=/dev/zero of=$TDB_FILE bs=1024 count=1 seek=$((undo_blks - 1)) conv=notrunc > /dev/null 2>&1
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc0 != $crc1 ] && [ $crc1 = $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_dryrun/script b/tests/u_dryrun/script
new file mode 100644
index 0000000..b90ef47
--- /dev/null
+++ b/tests/u_dryrun/script
@@ -0,0 +1,32 @@
+test_description="e2undo dry run"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+$E2UNDO -n $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc1 = $crc2 ] && [ $crc1 != $crc0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_errorout/script b/tests/u_errorout/script
new file mode 100644
index 0000000..20c53de
--- /dev/null
+++ b/tests/u_errorout/script
@@ -0,0 +1,49 @@
+test_description="e2undo a failed command"
+if test -x $RESIZE2FS_EXE -a -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/resize2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+echo check that we cant append a bad undo file > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+CONF=$TMPFILE.conf
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = ^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode
+		blocksize = 4096
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1K
+		zero_hugefiles = false
+	}
+ENDL
+
+echo mke2fs -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 1024 -z $TDB_FILE.0 $TMPFILE 524288 >> $OUT
+MKE2FS_CONFIG=$CONF $MKE2FS -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 1024 -z $TDB_FILE.0 -d /etc/ $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+
+echo roll back mke2fs >> $OUT
+$E2UNDO $TDB_FILE.0 $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo mke2fs $crc2 >> $OUT
+
+if [ $crc0 != $crc1 ] && [ $crc1 != $crc2 ] && [ $crc0 = $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE.0 $TMPFILE $CONF
+fi
diff --git a/tests/u_force/script b/tests/u_force/script
new file mode 100644
index 0000000..ef39e24
--- /dev/null
+++ b/tests/u_force/script
@@ -0,0 +1,40 @@
+test_description="e2undo force"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+dd if=/dev/zero of=$TDB_FILE bs=4 count=1 seek=127 conv=notrunc 2> /dev/null
+
+$E2UNDO $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+$E2UNDO -f $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo -f $crc3 >> $OUT
+
+MUST_FSCK=$($DUMPE2FS $TMPFILE 2> /dev/null | grep 'Filesystem state:.*not clean' -c )
+
+if [ $MUST_FSCK -eq 1 ] && [ $crc0 != $crc3 ] && [ $crc1 = $crc2 ] && [ $crc2 != $crc0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_force_dryrun/script b/tests/u_force_dryrun/script
new file mode 100644
index 0000000..92d7624
--- /dev/null
+++ b/tests/u_force_dryrun/script
@@ -0,0 +1,38 @@
+test_description="force dry-run replay of corrupt e2undo block data"
+if test -x $E2UNDO_EXE; then
+
+E2FSPROGS_UNDO_DIR=${TMPDIR:-/tmp}
+export E2FSPROGS_UNDO_DIR
+TDB_FILE=$E2FSPROGS_UNDO_DIR/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -I 128 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -I 256 $TMPFILE  >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+undo_blks=$(( $(stat -c '%s' $TDB_FILE 2>/dev/null || stat -f '%z' $TDB_FILE 2>/dev/null) / 1024 ))
+dd if=/dev/zero of=$TDB_FILE bs=1024 count=1 seek=$((undo_blks - 2)) conv=notrunc > /dev/null 2>&1
+
+$E2UNDO -f -n $TDB_FILE $TMPFILE  >> $OUT 2>&1
+res=$?
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+if [ $crc2 = $crc1 ] && [ $crc2 != $crc0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_incomplete/script b/tests/u_incomplete/script
new file mode 100644
index 0000000..7bc7858
--- /dev/null
+++ b/tests/u_incomplete/script
@@ -0,0 +1,38 @@
+test_description="e2undo with incomplete undo file"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+UNDO_IO_SIMULATE_UNFINISHED=1 $TUNE2FS -O metadata_csum -z $TDB_FILE $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+$E2UNDO  $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+$FSCK -y $TMPFILE >> $OUT 2>&1
+fsck_res=$?
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after fsck $crc3 >> $OUT
+echo fsck result $fsck_res >> $OUT
+
+if [ $crc0 != $crc2 ] && [ $crc1 != $crc2 ] && [ $crc0 != $crc1 ] && [ $fsck_res -eq 1 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_not_undo/script b/tests/u_not_undo/script
new file mode 100644
index 0000000..2f07d1b
--- /dev/null
+++ b/tests/u_not_undo/script
@@ -0,0 +1,28 @@
+test_description="e2undo a non-undo file"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+dd if=/dev/zero of=$TDB_FILE bs=1k count=512 > /dev/null 2>&1
+
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before e2undo $crc0 > $OUT
+
+od -tx1 -Ad -c < $TDB_FILE >> $OUT
+
+$E2UNDO $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc3 >> $OUT
+
+if [ $crc3 = $crc0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi
diff --git a/tests/u_onefile_bad/script b/tests/u_onefile_bad/script
new file mode 100644
index 0000000..60c9a2e
--- /dev/null
+++ b/tests/u_onefile_bad/script
@@ -0,0 +1,115 @@
+test_description="check that we cant append a bad undo file"
+if test -x $RESIZE2FS_EXE -a -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/resize2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+fail=0
+
+echo check that we cant append a bad undo file > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+CONF=$TMPFILE.conf
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode
+		blocksize = 4096
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo mke2fs -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 4096 -z $TDB_FILE.0 $TMPFILE 524288 >> $OUT
+MKE2FS_CONFIG=$CONF $MKE2FS -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 4096 -z $TDB_FILE.0 $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -gt 0 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have 64bit or metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -z $TDB_FILE.1 -b $TMPFILE >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo corrupt $TDB_FILE.1 >> $OUT
+dd if=/dev/zero of=$TDB_FILE.1 bs=4096 count=1 skip=1 2> /dev/null
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE.1 $TMPFILE >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc3 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D -z $TDB_FILE.1 $TMPFILE >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc4 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back e2fsck/tune2fs/resize2fs >> $OUT
+$E2UNDO $TDB_FILE.1 $TMPFILE  >> $OUT 2>&1
+crc1_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo e2fsck/tune2fs/resize2fs $crc1_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back mke2fs >> $OUT
+$E2UNDO $TDB_FILE.0 $TMPFILE  >> $OUT 2>&1
+crc0_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo mke2fs $crc0_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+if [ $fail -eq 0 ] && [ $crc0 != $crc1 ] && [ $crc1 != $crc2 ] && [ $crc2 = $crc3 ] && [ $crc3 = $crc4 ] && [ $crc1_2 = $crc2 ] && [ $crc0_2 = $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE.0 $TDB_FILE.1 $TMPFILE $CONF
+fi
diff --git a/tests/u_revert_64bitmcsum_onefile/script b/tests/u_revert_64bitmcsum_onefile/script
new file mode 100644
index 0000000..f1d7c2b
--- /dev/null
+++ b/tests/u_revert_64bitmcsum_onefile/script
@@ -0,0 +1,112 @@
+test_description="convert fs to 64bit,metadata_csum and revert as one undo file"
+if test -x $RESIZE2FS_EXE -a -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/resize2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+fail=0
+
+echo convert fs to 64bit,metadata_csum and revert both changes as one undo file > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+CONF=$TMPFILE.conf
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode
+		blocksize = 4096
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo mke2fs -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 4096 -z $TDB_FILE.0 $TMPFILE 524288 >> $OUT
+MKE2FS_CONFIG=$CONF $MKE2FS -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 4096 -z $TDB_FILE.0 $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -gt 0 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have 64bit or metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -z $TDB_FILE.1 -b $TMPFILE >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE.1 $TMPFILE >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc3 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -lt 1 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit and metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D -z $TDB_FILE.1 $TMPFILE >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc4 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -lt 1 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit and metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back e2fsck/tune2fs/resize2fs >> $OUT
+$E2UNDO $TDB_FILE.1 $TMPFILE  >> $OUT 2>&1
+crc1_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo e2fsck/tune2fs/resize2fs $crc1_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -gt 0 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have 64bit or metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back mke2fs >> $OUT
+$E2UNDO $TDB_FILE.0 $TMPFILE  >> $OUT 2>&1
+crc0_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo mke2fs $crc0_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ -n "${features}" ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have any features set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 && fail=1
+
+if [ $fail -eq 0 ] && [ $crc0 = $crc0_2 ] && [ $crc1 = $crc1_2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE.0 $TDB_FILE.1 $TMPFILE $CONF
+fi
diff --git a/tests/u_revert_all_onefile/script b/tests/u_revert_all_onefile/script
new file mode 100644
index 0000000..27b3b23
--- /dev/null
+++ b/tests/u_revert_all_onefile/script
@@ -0,0 +1,100 @@
+test_description="convert fs to 64bit,metadata_csum and revert as one undo file"
+if test -x $RESIZE2FS_EXE -a -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/resize2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+fail=0
+
+echo convert fs to 64bit,metadata_csum and revert both changes as one undo file > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+CONF=$TMPFILE.conf
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode
+		blocksize = 4096
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo mke2fs -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 4096 -z $TDB_FILE.0 $TMPFILE 524288 >> $OUT
+MKE2FS_CONFIG=$CONF $MKE2FS -q -F -o Linux -T ext4h -O ^metadata_csum,^64bit -E lazy_itable_init=1 -b 4096 -z $TDB_FILE.0 $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -gt 0 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have 64bit or metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -z $TDB_FILE.0 -b $TMPFILE >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -gt 0 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit but not metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE.0 $TMPFILE >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc3 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -lt 1 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit and metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D -z $TDB_FILE.0 $TMPFILE >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc4 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ "$(echo "${features}" | grep "metadata_csum" -c)" -lt 1 ] || [ "$(echo "${features}" | grep 64bit -c)" -lt 1 ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should have 64bit and metadata_csum set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 || fail=1
+
+echo roll back e2fsck/tune2fs/resize2fs/mke2fs >> $OUT
+$E2UNDO $TDB_FILE.0 $TMPFILE  >> $OUT 2>&1
+crc0_2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo e2fsck/tune2fs/resize2fs/mke2fs $crc1_2 >> $OUT
+features="$($DUMPE2FS -h $TMPFILE 2> /dev/null | grep 'Filesystem features:')"
+if [ -n "${features}" ]; then
+	echo "FS features: ${features}" >> $OUT
+	echo "Should not have any features set" >> $OUT
+	fail=1
+fi
+$FSCK -f -n $TMPFILE >> $OUT 2>&1 && fail=1
+
+if [ $fail -eq 0 ] && [ $crc0 = $crc0_2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE.0 $TDB_FILE.1 $TMPFILE $CONF
+fi
diff --git a/tests/u_undo_undo/script b/tests/u_undo_undo/script
new file mode 100644
index 0000000..726a453
--- /dev/null
+++ b/tests/u_undo_undo/script
@@ -0,0 +1,54 @@
+test_description="undo e2undo"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/e2fsck-$(basename $TMPFILE).e2undo
+TDB_FILE2=${TMPDIR:-/tmp}/e2undo-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE $TDB_FILE2 >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before e2fsck $crc0 >> $OUT
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D -z $TDB_FILE $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc1 >> $OUT
+
+echo e2undo the e2fsck >> $OUT
+$E2UNDO -z $TDB_FILE2 $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc2 >> $OUT
+
+echo e2undo the e2undo >> $OUT
+$E2UNDO $TDB_FILE2 $TMPFILE  >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc3 >> $OUT
+
+echo e2undo the e2undo the e2undo >> $OUT
+$E2UNDO $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc4 >> $OUT
+
+$E2UNDO -h $TDB_FILE $TMPFILE >> $OUT 2>&1
+$E2UNDO -h $TDB_FILE2 $TMPFILE >> $OUT 2>&1
+
+$E2UNDO -z $TDB_FILE2 $TDB_FILE2 $TMPFILE >> $OUT 2>&1
+
+crc5=`$CRCSUM $TMPFILE`
+echo $CRCSUM after failed e2undo $crc5 >> $OUT
+
+echo $crc0 $crc1 $crc2 $crc3 $crc4 $crc5 >> $OUT
+
+if [ $crc0 = $crc2 ] && [ $crc2 = $crc4 ] && [ $crc5 = $crc4 ] && [ $crc1 = $crc3 ] && [ $crc1 != $crc2 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TDB_FILE2 $TMPFILE
+fi
diff --git a/tests/u_wrong_fs/script b/tests/u_wrong_fs/script
new file mode 100644
index 0000000..dbf0d6b
--- /dev/null
+++ b/tests/u_wrong_fs/script
@@ -0,0 +1,36 @@
+test_description="e2undo on the wrong fs"
+if test -x $E2UNDO_EXE; then
+
+TDB_FILE=${TMPDIR:-/tmp}/tune2fs-$(basename $TMPFILE).e2undo
+OUT=$test_name.log
+rm -f $TDB_FILE >/dev/null 2>&1
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before tune2fs $crc0 >> $OUT
+
+echo using tune2fs to test e2undo >> $OUT
+$TUNE2FS -O metadata_csum -z $TDB_FILE $TMPFILE >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs $crc1 >> $OUT
+
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE  >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after re-mke2fs $crc2 >> $OUT
+
+$E2UNDO $TDB_FILE $TMPFILE  >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2undo $crc3 >> $OUT
+
+if [ $crc3 = $crc2 ] && [ $crc2 != $crc1 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TDB_FILE $TMPFILE
+fi

[PATCH 42/54] copy-in: create hardlinks with the correct directory filetype

[Darrick J. Wong]

When we're creating hard links via ext2fs_link, the (misnamed?) flags
argument specifies the filetype for the directory entry.  This is
*derived* from i_mode, so provide a translator.  Otherwise, fsck will
complain about unset file types.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/create_inode.c |   32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)


diff --git a/misc/create_inode.c b/misc/create_inode.c
index a024d1c..3bc0515 100644
--- a/misc/create_inode.c
+++ b/misc/create_inode.c
@@ -37,6 +37,32 @@
 #define S_BLKSIZE 512
 #endif
 
+static int ext2_file_type(unsigned int mode)
+{
+	if (LINUX_S_ISREG(mode))
+		return EXT2_FT_REG_FILE;
+
+	if (LINUX_S_ISDIR(mode))
+		return EXT2_FT_DIR;
+
+	if (LINUX_S_ISCHR(mode))
+		return EXT2_FT_CHRDEV;
+
+	if (LINUX_S_ISBLK(mode))
+		return EXT2_FT_BLKDEV;
+
+	if (LINUX_S_ISLNK(mode))
+		return EXT2_FT_SYMLINK;
+
+	if (LINUX_S_ISFIFO(mode))
+		return EXT2_FT_FIFO;
+
+	if (LINUX_S_ISSOCK(mode))
+		return EXT2_FT_SOCK;
+
+	return 0;
+}
+
 /* Link an inode number to a directory */
 static errcode_t add_link(ext2_filsys fs, ext2_ino_t parent_ino,
 			  ext2_ino_t ino, const char *name)
@@ -50,14 +76,16 @@ static errcode_t add_link(ext2_filsys fs, ext2_ino_t parent_ino,
 		return retval;
 	}
 
-	retval = ext2fs_link(fs, parent_ino, name, ino, inode.i_flags);
+	retval = ext2fs_link(fs, parent_ino, name, ino,
+			     ext2_file_type(inode.i_mode));
 	if (retval == EXT2_ET_DIR_NO_SPACE) {
 		retval = ext2fs_expand_dir(fs, parent_ino);
 		if (retval) {
 			com_err(__func__, retval, "while expanding directory");
 			return retval;
 		}
-		retval = ext2fs_link(fs, parent_ino, name, ino, inode.i_flags);
+		retval = ext2fs_link(fs, parent_ino, name, ino,
+				     ext2_file_type(inode.i_mode));
 	}
 	if (retval) {
 		com_err(__func__, retval, "while linking %s", name);

[PATCH 43/54] copy-in: for files, only iterate file blocks that are mapped

[Darrick J. Wong]

Rewrite the file copy-in algorithm to detect smaller holes in the
files we're copying in.  Use SEEK_DATA/SEEK_HOLE/FIEMAP when available
to skip known empty parts.  This fixes the particular bug where zeroed
blocks on a system with 64k pages are needlessly copied into a
4k-block filesystem.  It also saves time by skipping parts we know to
be zeroed.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/create_inode.c |  280 ++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 196 insertions(+), 84 deletions(-)


diff --git a/misc/create_inode.c b/misc/create_inode.c
index 3bc0515..8ab546e 100644
--- a/misc/create_inode.c
+++ b/misc/create_inode.c
@@ -9,12 +9,22 @@
  * %End-Header%
  */
 
+#define _FILE_OFFSET_BITS       64
+#define _LARGEFILE64_SOURCE     1
+#define _GNU_SOURCE		1
+
+#include "config.h"
 #include <time.h>
+#include <sys/types.h>
 #include <unistd.h>
 #include <limits.h> /* for PATH_MAX */
 #ifdef HAVE_ATTR_XATTR_H
 #include <attr/xattr.h>
 #endif
+#include <sys/ioctl.h>
+#include <ext2fs/ext2fs.h>
+#include <ext2fs/ext2_types.h>
+#include <ext2fs/fiemap.h>
 
 #include "create_inode.h"
 #include "nls-enable.h"
@@ -28,14 +38,7 @@
 #endif
 
 /* 64KiB is the minimium blksize to best minimize system call overhead. */
-#ifndef IO_BUFSIZE
-#define IO_BUFSIZE 64*1024
-#endif
-
-/* Block size for `st_blocks' */
-#ifndef S_BLKSIZE
-#define S_BLKSIZE 512
-#endif
+#define COPY_FILE_BUFLEN	65536
 
 static int ext2_file_type(unsigned int mode)
 {
@@ -139,10 +142,9 @@ static errcode_t set_inode_xattr(ext2_filsys fs, ext2_ino_t ino, const char *fil
 {
 #ifdef HAVE_LLISTXATTR
 	errcode_t			retval, close_retval;
-	struct ext2_inode		inode;
 	struct ext2_xattr_handle	*handle;
 	ssize_t				size, value_size;
-	char				*list;
+	char				*list = NULL;
 	int				i;
 
 	size = llistxattr(filename, NULL, 0);
@@ -382,82 +384,202 @@ try_again:
 	return retval;
 }
 
-static errcode_t copy_file(ext2_filsys fs, int fd, ext2_ino_t newfile,
-			   int bufsize, int make_holes)
+#if !defined HAVE_PREAD64 && !defined HAVE_PREAD
+static ssize_t my_pread(int fd, const void *buf, size_t count, off_t offset)
 {
-	ext2_file_t	e2_file;
-	errcode_t	retval, close_ret;
-	int		got;
-	unsigned int	written;
-	char		*buf;
-	char		*ptr;
-	char		*zero_buf;
-	int		cmp;
-
-	retval = ext2fs_file_open(fs, newfile,
-				  EXT2_FILE_WRITE, &e2_file);
-	if (retval)
-		return retval;
-
-	retval = ext2fs_get_mem(bufsize, &buf);
-	if (retval) {
-		com_err("copy_file", retval, "can't allocate buffer\n");
-		goto out_close;
-	}
+	if (lseek(fd, offset, SEEK_SET) < 0)
+		return 0;
 
-	/* This is used for checking whether the whole block is zero */
-	retval = ext2fs_get_memzero(bufsize, &zero_buf);
-	if (retval) {
-		com_err("copy_file", retval, "can't allocate zero buffer\n");
-		goto out_free_buf;
-	}
+	return read(fd, buf, count);
+}
+#endif /* !defined HAVE_PREAD64 && !defined HAVE_PREAD */
 
-	while (1) {
-		got = read(fd, buf, bufsize);
-		if (got == 0)
-			break;
+static errcode_t copy_file_range(ext2_filsys fs, int fd, ext2_file_t e2_file,
+				 off_t start, off_t end, char *buf,
+				 char *zerobuf)
+{
+	off_t off, bpos;
+	ssize_t got, blen;
+	unsigned int written;
+	char *ptr;
+	errcode_t err = 0;
+
+	for (off = start; off < end; off += COPY_FILE_BUFLEN) {
+#ifdef HAVE_PREAD64
+		got = pread64(fd, buf, COPY_FILE_BUFLEN, off);
+#elif HAVE_PREAD
+		got = pread(fd, buf, COPY_FILE_BUFLEN, off);
+#else
+		got = my_pread(fd, buf, COPY_FILE_BUFLEN, off);
+#endif
 		if (got < 0) {
-			retval = errno;
+			err = errno;
 			goto fail;
 		}
-		ptr = buf;
-
-		/* Sparse copy */
-		if (make_holes) {
-			/* Check whether all is zero */
-			cmp = memcmp(ptr, zero_buf, got);
-			if (cmp == 0) {
-				 /* The whole block is zero, make a hole */
-				retval = ext2fs_file_lseek(e2_file, got,
-							   EXT2_SEEK_CUR,
-							   NULL);
-				if (retval)
+		for (bpos = 0, ptr = buf; bpos < got; bpos += fs->blocksize) {
+			blen = fs->blocksize;
+			if (blen > got - bpos)
+				blen = got - bpos;
+			if (memcmp(ptr, zerobuf, blen) == 0) {
+				ptr += blen;
+				continue;
+			}
+			err = ext2fs_file_lseek(e2_file, off + bpos,
+						EXT2_SEEK_SET, NULL);
+			if (err)
+				goto fail;
+			while (blen > 0) {
+				err = ext2fs_file_write(e2_file, ptr, blen,
+							&written);
+				if (err)
+					goto fail;
+				if (written == 0) {
+					err = EIO;
 					goto fail;
-				got = 0;
+				}
+				blen -= written;
+				ptr += written;
 			}
 		}
+	}
+fail:
+	return err;
+}
 
-		/* Normal copy */
-		while (got > 0) {
-			retval = ext2fs_file_write(e2_file, ptr,
-						   got, &written);
-			if (retval)
-				goto fail;
-
-			got -= written;
-			ptr += written;
+static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
+				ext2_file_t e2_file, char *buf, char *zerobuf)
+{
+#if defined(SEEK_DATA) && defined(SEEK_HOLE)
+	off_t data = 0, hole;
+	off_t data_blk, hole_blk;
+	errcode_t err;
+
+	/* Try to use SEEK_DATA and SEEK_HOLE */
+	while (data < statbuf->st_size) {
+		data = lseek(fd, data, SEEK_DATA);
+		if (data < 0) {
+			if (errno == ENXIO)
+				break;
+			return EXT2_ET_UNIMPLEMENTED;
 		}
+		hole = lseek(fd, data, SEEK_HOLE);
+		if (hole < 0)
+			return EXT2_ET_UNIMPLEMENTED;
+
+		data_blk = data & ~(fs->blocksize - 1);
+		hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
+		err = copy_file_range(fs, fd, e2_file, data_blk, hole_blk, buf,
+				      zerobuf);
+		if (err)
+			return err;
+
+		data = hole;
 	}
 
-fail:
-	ext2fs_free_mem(&zero_buf);
-out_free_buf:
+	return err;
+#else
+	return EXT2_ET_UNIMPLEMENTED;
+#endif /* SEEK_DATA and SEEK_HOLE */
+}
+
+static errcode_t try_fiemap_copy(ext2_filsys fs, int fd, ext2_file_t e2_file,
+				 char *buf, char *zerobuf)
+{
+#if defined(FS_IOC_FIEMAP)
+#define EXTENT_MAX_COUNT 512
+	struct fiemap *fiemap_buf;
+	struct fiemap_extent *ext_buf, *ext;
+	int ext_buf_size, fie_buf_size;
+	off_t pos = 0;
+	unsigned int i;
+	errcode_t err;
+
+	ext_buf_size = EXTENT_MAX_COUNT * sizeof(struct fiemap_extent);
+	fie_buf_size = sizeof(struct fiemap) + ext_buf_size;
+
+	err = ext2fs_get_memzero(fie_buf_size, &fiemap_buf);
+	if (err)
+		return err;
+
+	ext_buf = fiemap_buf->fm_extents;
+	memset(fiemap_buf, 0, fie_buf_size);
+	fiemap_buf->fm_length = FIEMAP_MAX_OFFSET;
+	fiemap_buf->fm_flags |= FIEMAP_FLAG_SYNC;
+	fiemap_buf->fm_extent_count = EXTENT_MAX_COUNT;
+
+	do {
+		fiemap_buf->fm_start = pos;
+		memset(ext_buf, 0, ext_buf_size);
+		err = ioctl(fd, FS_IOC_FIEMAP, fiemap_buf);
+		if (err < 0 && (errno == EOPNOTSUPP || errno == ENOTTY)) {
+			err = EXT2_ET_UNIMPLEMENTED;
+			goto out;
+		} else if (err < 0 || fiemap_buf->fm_mapped_extents == 0) {
+			err = errno;
+			goto out;
+		}
+		for (i = 0, ext = ext_buf; i < fiemap_buf->fm_mapped_extents;
+		     i++, ext++) {
+			err = copy_file_range(fs, fd, e2_file, ext->fe_logical,
+					      ext->fe_logical + ext->fe_length,
+					      buf, zerobuf);
+			if (err)
+				goto out;
+		}
+
+		ext--;
+		/* Record file's logical offset this time */
+		pos = ext->fe_logical + ext->fe_length;
+		/*
+		 * If fm_extents array has been filled and
+		 * there are extents left, continue to cycle.
+		 */
+	} while (fiemap_buf->fm_mapped_extents == EXTENT_MAX_COUNT &&
+		 !(ext->fe_flags & FIEMAP_EXTENT_LAST));
+out:
+	ext2fs_free_mem(&fiemap_buf);
+	return err;
+#else
+	return EXT2_ET_UNIMPLEMENTED;
+#endif /* FS_IOC_FIEMAP */
+}
+
+static errcode_t copy_file(ext2_filsys fs, int fd, struct stat *statbuf,
+			   ext2_ino_t ino)
+{
+	ext2_file_t e2_file;
+	char *buf = NULL, *zerobuf = NULL;
+	errcode_t err, close_err;
+
+	err = ext2fs_file_open(fs, ino, EXT2_FILE_WRITE, &e2_file);
+	if (err)
+		return err;
+
+	err = ext2fs_get_mem(COPY_FILE_BUFLEN, &buf);
+	if (err)
+		goto out;
+
+	err = ext2fs_get_memzero(fs->blocksize, &zerobuf);
+	if (err)
+		goto out;
+
+	err = try_lseek_copy(fs, fd, statbuf, e2_file, buf, zerobuf);
+	if (err != EXT2_ET_UNIMPLEMENTED)
+		goto out;
+
+	err = try_fiemap_copy(fs, fd, e2_file, buf, zerobuf);
+	if (err != EXT2_ET_UNIMPLEMENTED)
+		goto out;
+
+	err = copy_file_range(fs, fd, e2_file, 0, statbuf->st_size, buf,
+			      zerobuf);
+out:
+	ext2fs_free_mem(&zerobuf);
 	ext2fs_free_mem(&buf);
-out_close:
-	close_ret = ext2fs_file_close(e2_file);
-	if (retval == 0)
-		retval = close_ret;
-	return retval;
+	close_err = ext2fs_file_close(e2_file);
+	if (err == 0)
+		err = close_err;
+	return err;
 }
 
 static int is_hardlink(struct hdlinks_s *hdlinks, dev_t dev, ino_t ino)
@@ -481,8 +603,6 @@ errcode_t do_write_internal(ext2_filsys fs, ext2_ino_t cwd, const char *src,
 	ext2_ino_t	newfile;
 	errcode_t	retval;
 	struct ext2_inode inode;
-	int		bufsize = IO_BUFSIZE;
-	int		make_holes = 0;
 
 	fd = ext2fs_open_file(src, O_RDONLY, 0);
 	if (fd < 0) {
@@ -570,17 +690,9 @@ errcode_t do_write_internal(ext2_filsys fs, ext2_ino_t cwd, const char *src,
 		}
 	}
 	if (LINUX_S_ISREG(inode.i_mode)) {
-		if (statbuf.st_blocks < statbuf.st_size / S_BLKSIZE) {
-			make_holes = 1;
-			/*
-			 * Use I/O blocksize as buffer size when
-			 * copying sparse files.
-			 */
-			bufsize = statbuf.st_blksize;
-		}
-		retval = copy_file(fs, fd, newfile, bufsize, make_holes);
+		retval = copy_file(fs, fd, &statbuf, newfile);
 		if (retval)
-			com_err("copy_file", retval, 0);
+			com_err("copy_file", retval, _("while copying %s"), src);
 	}
 	close(fd);
 

[PATCH 44/54] copyin: fix error handling

[Darrick J. Wong]

Save errno (in retval) before doing anything else, because the
"anything else" (usually com_err()) can call library functions, which
will reset errno.

Fix the error messages to use the message catalog, and don't _ever_
print an error without providing context.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/create_inode.c            |  212 ++++++++++++++++++++--------------------
 misc/mke2fs.c                  |    2 
 tests/f_create_symlinks/expect |    4 -
 3 files changed, 107 insertions(+), 111 deletions(-)


diff --git a/misc/create_inode.c b/misc/create_inode.c
index 8ab546e..7d3c9e6 100644
--- a/misc/create_inode.c
+++ b/misc/create_inode.c
@@ -75,7 +75,7 @@ static errcode_t add_link(ext2_filsys fs, ext2_ino_t parent_ino,
 
 	retval = ext2fs_read_inode(fs, ino, &inode);
         if (retval) {
-		com_err(__func__, retval, "while reading inode %u", ino);
+		com_err(__func__, retval, _("while reading inode %u"), ino);
 		return retval;
 	}
 
@@ -84,14 +84,15 @@ static errcode_t add_link(ext2_filsys fs, ext2_ino_t parent_ino,
 	if (retval == EXT2_ET_DIR_NO_SPACE) {
 		retval = ext2fs_expand_dir(fs, parent_ino);
 		if (retval) {
-			com_err(__func__, retval, "while expanding directory");
+			com_err(__func__, retval,
+				_("while expanding directory"));
 			return retval;
 		}
 		retval = ext2fs_link(fs, parent_ino, name, ino,
 				     ext2_file_type(inode.i_mode));
 	}
 	if (retval) {
-		com_err(__func__, retval, "while linking %s", name);
+		com_err(__func__, retval, _("while linking \"%s\""), name);
 		return retval;
 	}
 
@@ -99,24 +100,11 @@ static errcode_t add_link(ext2_filsys fs, ext2_ino_t parent_ino,
 
 	retval = ext2fs_write_inode(fs, ino, &inode);
 	if (retval)
-		com_err(__func__, retval, "while writing inode %u", ino);
+		com_err(__func__, retval, _("while writing inode %u"), ino);
 
 	return retval;
 }
 
-/* Fill the uid, gid, mode and time for the inode */
-static void fill_inode(struct ext2_inode *inode, struct stat *st)
-{
-	if (st != NULL) {
-		inode->i_uid = st->st_uid;
-		inode->i_gid = st->st_gid;
-		inode->i_mode |= st->st_mode;
-		inode->i_atime = st->st_atime;
-		inode->i_mtime = st->st_mtime;
-		inode->i_ctime = st->st_ctime;
-	}
-}
-
 /* Set the uid, gid, mode and time for the inode */
 static errcode_t set_inode_extra(ext2_filsys fs, ext2_ino_t cwd,
 				 ext2_ino_t ino, struct stat *st)
@@ -126,19 +114,25 @@ static errcode_t set_inode_extra(ext2_filsys fs, ext2_ino_t cwd,
 
 	retval = ext2fs_read_inode(fs, ino, &inode);
         if (retval) {
-		com_err(__func__, retval, "while reading inode %u", ino);
+		com_err(__func__, retval, _("while reading inode %u"), ino);
 		return retval;
 	}
 
-	fill_inode(&inode, st);
+	inode.i_uid = st->st_uid;
+	inode.i_gid = st->st_gid;
+	inode.i_mode |= st->st_mode;
+	inode.i_atime = st->st_atime;
+	inode.i_mtime = st->st_mtime;
+	inode.i_ctime = st->st_ctime;
 
 	retval = ext2fs_write_inode(fs, ino, &inode);
 	if (retval)
-		com_err(__func__, retval, "while writing inode %u", ino);
+		com_err(__func__, retval, _("while writing inode %u"), ino);
 	return retval;
 }
 
-static errcode_t set_inode_xattr(ext2_filsys fs, ext2_ino_t ino, const char *filename)
+static errcode_t set_inode_xattr(ext2_filsys fs, ext2_ino_t ino,
+				 const char *filename)
 {
 #ifdef HAVE_LLISTXATTR
 	errcode_t			retval, close_retval;
@@ -149,8 +143,10 @@ static errcode_t set_inode_xattr(ext2_filsys fs, ext2_ino_t ino, const char *fil
 
 	size = llistxattr(filename, NULL, 0);
 	if (size == -1) {
-		com_err(__func__, errno, "llistxattr failed on %s", filename);
-		return errno;
+		retval = errno;
+		com_err(__func__, retval, _("while listing attributes of \"%s\""),
+			filename);
+		return retval;
 	} else if (size == 0) {
 		return 0;
 	}
@@ -159,20 +155,21 @@ static errcode_t set_inode_xattr(ext2_filsys fs, ext2_ino_t ino, const char *fil
 	if (retval) {
 		if (retval == EXT2_ET_MISSING_EA_FEATURE)
 			return 0;
-		com_err(__func__, retval, "while opening inode %u", ino);
+		com_err(__func__, retval, _("while opening inode %u"), ino);
 		return retval;
 	}
 
 	retval = ext2fs_get_mem(size, &list);
 	if (retval) {
-		com_err(__func__, retval, "whilst allocating memory");
+		com_err(__func__, retval, _("while allocating memory"));
 		goto out;
 	}
 
 	size = llistxattr(filename, list, size);
 	if (size == -1) {
-		com_err(__func__, errno, "llistxattr failed on %s", filename);
 		retval = errno;
+		com_err(__func__, retval, _("while listing attributes of \"%s\""),
+			filename);
 		goto out;
         }
 
@@ -182,24 +179,26 @@ static errcode_t set_inode_xattr(ext2_filsys fs, ext2_ino_t ino, const char *fil
 
 		value_size = getxattr(filename, name, NULL, 0);
 		if (value_size == -1) {
-			com_err(__func__, errno, "getxattr failed on %s",
-				filename);
 			retval = errno;
+			com_err(__func__, retval,
+				_("while reading attribute \"%s\" of \"%s\""),
+				name, filename);
 			break;
 		}
 
 		retval = ext2fs_get_mem(value_size, &value);
 		if (retval) {
-			com_err(__func__, retval, "whilst allocating memory");
+			com_err(__func__, retval, _("while allocating memory"));
 			break;
 		}
 
 		value_size = getxattr(filename, name, value, value_size);
 		if (value_size == -1) {
 			ext2fs_free_mem(&value);
-			com_err(__func__, errno, "getxattr failed on %s",
-				filename);
 			retval = errno;
+			com_err(__func__, retval,
+				_("while reading attribute \"%s\" of \"%s\""),
+				name, filename);
 			break;
 		}
 
@@ -207,7 +206,8 @@ static errcode_t set_inode_xattr(ext2_filsys fs, ext2_ino_t ino, const char *fil
 		ext2fs_free_mem(&value);
 		if (retval) {
 			com_err(__func__, retval,
-				"while writing xattr %u", ino);
+				_("while writing attribute \"%s\" to inode %u"),
+				name, ino);
 			break;
 		}
 
@@ -216,7 +216,7 @@ static errcode_t set_inode_xattr(ext2_filsys fs, ext2_ino_t ino, const char *fil
 	ext2fs_free_mem(&list);
 	close_retval = ext2fs_xattrs_close(&handle);
 	if (close_retval) {
-		com_err(__func__, retval, "while closing inode %u", ino);
+		com_err(__func__, retval, _("while closing inode %u"), ino);
 		retval = retval ? retval : close_retval;
 	}
 	return retval;
@@ -256,13 +256,10 @@ errcode_t do_mknod_internal(ext2_filsys fs, ext2_ino_t cwd, const char *name,
 		return EXT2_ET_INVALID_ARGUMENT;
 	}
 
-	if (!(fs->flags & EXT2_FLAG_RW)) {
-		com_err(__func__, 0, "Filesystem opened read/only");
-		return EROFS;
-	}
 	retval = ext2fs_new_inode(fs, cwd, 010755, 0, &ino);
 	if (retval) {
-		com_err(__func__, retval, 0);
+		com_err(__func__, retval, _("while allocating inode \"%s\""),
+			name);
 		return retval;
 	}
 
@@ -273,13 +270,14 @@ errcode_t do_mknod_internal(ext2_filsys fs, ext2_ino_t cwd, const char *name,
 	if (retval == EXT2_ET_DIR_NO_SPACE) {
 		retval = ext2fs_expand_dir(fs, cwd);
 		if (retval) {
-			com_err(__func__, retval, "while expanding directory");
+			com_err(__func__, retval,
+				_("while expanding directory"));
 			return retval;
 		}
 		retval = ext2fs_link(fs, cwd, name, ino, filetype);
 	}
 	if (retval) {
-		com_err(name, retval, 0);
+		com_err(name, retval, _("while creating inode \"%s\""), name);
 		return retval;
 	}
 	if (ext2fs_test_inode_bitmap2(fs->inode_map, ino))
@@ -307,7 +305,7 @@ errcode_t do_mknod_internal(ext2_filsys fs, ext2_ino_t cwd, const char *name,
 
 	retval = ext2fs_write_new_inode(fs, ino, &inode);
 	if (retval)
-		com_err(__func__, retval, "while creating inode %u", ino);
+		com_err(__func__, retval, _("while writing inode %u"), ino);
 
 	return retval;
 }
@@ -332,19 +330,19 @@ errcode_t do_symlink_internal(ext2_filsys fs, ext2_ino_t cwd, const char *name,
 	} else
 		parent_ino = cwd;
 
-try_again:
 	retval = ext2fs_symlink(fs, parent_ino, 0, name, target);
 	if (retval == EXT2_ET_DIR_NO_SPACE) {
 		retval = ext2fs_expand_dir(fs, parent_ino);
 		if (retval) {
 			com_err("do_symlink_internal", retval,
-				"while expanding directory");
+				_("while expanding directory"));
 			return retval;
 		}
-		goto try_again;
+		retval = ext2fs_symlink(fs, parent_ino, 0, name, target);
 	}
 	if (retval)
-		com_err("ext2fs_symlink", retval, 0);
+		com_err("ext2fs_symlink", retval,
+			_("while creating symlink \"%s\""), name);
 	return retval;
 }
 
@@ -362,25 +360,27 @@ errcode_t do_mkdir_internal(ext2_filsys fs, ext2_ino_t cwd, const char *name,
 		*cp = 0;
 		retval = ext2fs_namei(fs, root, cwd, name, &parent_ino);
 		if (retval) {
-			com_err(name, retval, 0);
+			com_err(name, retval, _("while looking up \"%s\""),
+				name);
 			return retval;
 		}
 		name = cp+1;
 	} else
 		parent_ino = cwd;
 
-try_again:
 	retval = ext2fs_mkdir(fs, parent_ino, 0, name);
 	if (retval == EXT2_ET_DIR_NO_SPACE) {
 		retval = ext2fs_expand_dir(fs, parent_ino);
 		if (retval) {
-			com_err(__func__, retval, "while expanding directory");
+			com_err(__func__, retval,
+				_("while expanding directory"));
 			return retval;
 		}
-		goto try_again;
+		retval = ext2fs_mkdir(fs, parent_ino, 0, name);
 	}
 	if (retval)
-		com_err("ext2fs_mkdir", retval, 0);
+		com_err("ext2fs_mkdir", retval,
+			_("while creating directory \"%s\""), name);
 	return retval;
 }
 
@@ -606,27 +606,25 @@ errcode_t do_write_internal(ext2_filsys fs, ext2_ino_t cwd, const char *src,
 
 	fd = ext2fs_open_file(src, O_RDONLY, 0);
 	if (fd < 0) {
-		com_err(src, errno, 0);
-		return errno;
+		retval = errno;
+		com_err(__func__, retval, _("while opening \"%s\" to copy"),
+			src);
+		return retval;
 	}
 	if (fstat(fd, &statbuf) < 0) {
-		com_err(src, errno, 0);
-		close(fd);
-		return errno;
+		retval = errno;
+		goto out;
 	}
 
 	retval = ext2fs_namei(fs, root, cwd, dest, &newfile);
 	if (retval == 0) {
-		close(fd);
-		return EXT2_ET_FILE_EXISTS;
+		retval = EXT2_ET_FILE_EXISTS;
+		goto out;
 	}
 
 	retval = ext2fs_new_inode(fs, cwd, 010755, 0, &newfile);
-	if (retval) {
-		com_err(__func__, retval, 0);
-		close(fd);
-		return retval;
-	}
+	if (retval)
+		goto out;
 #ifdef DEBUGFS
 	printf("Allocated inode: %u\n", newfile);
 #endif
@@ -634,19 +632,13 @@ errcode_t do_write_internal(ext2_filsys fs, ext2_ino_t cwd, const char *src,
 				EXT2_FT_REG_FILE);
 	if (retval == EXT2_ET_DIR_NO_SPACE) {
 		retval = ext2fs_expand_dir(fs, cwd);
-		if (retval) {
-			com_err(__func__, retval, "while expanding directory");
-			close(fd);
-			return retval;
-		}
+		if (retval)
+			goto out;
 		retval = ext2fs_link(fs, cwd, dest, newfile,
 					EXT2_FT_REG_FILE);
 	}
-	if (retval) {
-		com_err(dest, retval, 0);
-		close(fd);
-		return errno;
-	}
+	if (retval)
+		goto out;
 	if (ext2fs_test_inode_bitmap2(fs->inode_map, newfile))
 		com_err(__func__, 0, "Warning: inode already set");
 	ext2fs_inode_alloc_stats2(fs, newfile, +1, 0);
@@ -656,11 +648,8 @@ errcode_t do_write_internal(ext2_filsys fs, ext2_ino_t cwd, const char *src,
 		fs->now ? fs->now : time(0);
 	inode.i_links_count = 1;
 	retval = ext2fs_inode_size_set(fs, &inode, statbuf.st_size);
-	if (retval) {
-		com_err(dest, retval, 0);
-		close(fd);
-		return retval;
-	}
+	if (retval)
+		goto out;
 	if (EXT2_HAS_INCOMPAT_FEATURE(fs->super,
 				      EXT4_FEATURE_INCOMPAT_INLINE_DATA)) {
 		inode.i_flags |= EXT4_INLINE_DATA_FL;
@@ -671,31 +660,25 @@ errcode_t do_write_internal(ext2_filsys fs, ext2_ino_t cwd, const char *src,
 		inode.i_flags &= ~EXT4_EXTENTS_FL;
 		retval = ext2fs_extent_open2(fs, newfile, &inode, &handle);
 		if (retval)
-			return retval;
+			goto out;
 		ext2fs_extent_free(handle);
 	}
 
 	retval = ext2fs_write_new_inode(fs, newfile, &inode);
-	if (retval) {
-		com_err(__func__, retval, "while creating inode %u", newfile);
-		close(fd);
-		return retval;
-	}
+	if (retval)
+		goto out;
 	if (inode.i_flags & EXT4_INLINE_DATA_FL) {
 		retval = ext2fs_inline_data_init(fs, newfile);
-		if (retval) {
-			com_err("copy_file", retval, 0);
-			close(fd);
-			return retval;
-		}
+		if (retval)
+			goto out;
 	}
 	if (LINUX_S_ISREG(inode.i_mode)) {
 		retval = copy_file(fs, fd, &statbuf, newfile);
 		if (retval)
-			com_err("copy_file", retval, _("while copying %s"), src);
+			goto out;
 	}
+out:
 	close(fd);
-
 	return retval;
 }
 
@@ -716,16 +699,18 @@ static errcode_t __populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 	int		hdlink;
 
 	if (chdir(source_dir) < 0) {
-		com_err(__func__, errno,
+		retval = errno;
+		com_err(__func__, retval,
 			_("while changing working directory to \"%s\""),
 			source_dir);
-		return errno;
+		return retval;
 	}
 
 	if (!(dh = opendir("."))) {
-		com_err(__func__, errno,
+		retval = errno;
+		com_err(__func__, retval,
 			_("while opening directory \"%s\""), source_dir);
-		return errno;
+		return retval;
 	}
 
 	while ((dent = readdir(dh))) {
@@ -733,7 +718,8 @@ static errcode_t __populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 		    (!strcmp(dent->d_name, "..")))
 			continue;
 		if (lstat(dent->d_name, &st)) {
-			com_err(__func__, errno, _("while lstat \"%s\""),
+			retval = errno;
+			com_err(__func__, retval, _("while lstat \"%s\""),
 				dent->d_name);
 			goto out;
 		}
@@ -775,10 +761,10 @@ static errcode_t __populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 			read_cnt = readlink(name, ln_target,
 					    sizeof(ln_target) - 1);
 			if (read_cnt == -1) {
-				com_err(__func__, errno,
-					_("while trying to readlink \"%s\""),
-					name);
 				retval = errno;
+				com_err(__func__, retval,
+					_("while trying to read link \"%s\""),
+					name);
 				goto out;
 			}
 			ln_target[read_cnt] = '\0';
@@ -801,6 +787,10 @@ static errcode_t __populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 			}
 			break;
 		case S_IFDIR:
+			/* Don't choke on /lost+found */
+			if (parent_ino == EXT2_ROOT_INO &&
+			    strcmp(name, "lost+found") == 0)
+				goto find_lnf;
 			retval = do_mkdir_internal(fs, parent_ino, name, &st,
 						   root);
 			if (retval) {
@@ -808,6 +798,7 @@ static errcode_t __populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 					_("while making dir \"%s\""), name);
 				goto out;
 			}
+find_lnf:
 			retval = ext2fs_namei(fs, root, parent_ino,
 					      name, &ino);
 			if (retval) {
@@ -816,14 +807,12 @@ static errcode_t __populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 			}
 			/* Populate the dir recursively*/
 			retval = __populate_fs(fs, ino, name, root, hdlinks);
-			if (retval) {
-				com_err(__func__, retval,
-					_("while adding dir \"%s\""), name);
+			if (retval)
 				goto out;
-			}
 			if (chdir("..")) {
-				com_err(__func__, errno, _("during cd .."));
 				retval = errno;
+				com_err(__func__, retval,
+					_("while changing directory"));
 				goto out;
 			}
 			break;
@@ -834,7 +823,8 @@ static errcode_t __populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 
 		retval =  ext2fs_namei(fs, root, parent_ino, name, &ino);
 		if (retval) {
-			com_err(name, retval, 0);
+			com_err(name, retval, _("while looking up \"%s\""),
+				name);
 			goto out;
 		}
 
@@ -864,9 +854,9 @@ static errcode_t __populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 						(hdlinks->size + HDLINK_CNT) *
 						sizeof(struct hdlink_s));
 				if (p == NULL) {
-					com_err(name, errno,
-						_("Not enough memory"));
 					retval = EXT2_ET_NO_MEMORY;
+					com_err(name, retval,
+						_("while saving inode data"));
 					goto out;
 				}
 				hdlinks->hdl = p;
@@ -890,12 +880,18 @@ errcode_t populate_fs(ext2_filsys fs, ext2_ino_t parent_ino,
 	struct hdlinks_s hdlinks;
 	errcode_t retval;
 
+	if (!(fs->flags & EXT2_FLAG_RW)) {
+		com_err(__func__, 0, "Filesystem opened readonly");
+		return EROFS;
+	}
+
 	hdlinks.count = 0;
 	hdlinks.size = HDLINK_CNT;
 	hdlinks.hdl = realloc(NULL, hdlinks.size * sizeof(struct hdlink_s));
 	if (hdlinks.hdl == NULL) {
-		com_err(__func__, errno, "Not enough memory");
-		return errno;
+		retval = errno;
+		com_err(__func__, retval, _("while allocating memory"));
+		return retval;
 	}
 
 	retval = __populate_fs(fs, parent_ino, source_dir, root, &hdlinks);
diff --git a/misc/mke2fs.c b/misc/mke2fs.c
index 47b0b29..5572ec0 100644
--- a/misc/mke2fs.c
+++ b/misc/mke2fs.c
@@ -3131,7 +3131,7 @@ no_journal:
 				     EXT2_ROOT_INO);
 		if (retval) {
 			com_err(program_name, retval, "%s",
-				_("\nError while populating file system\n"));
+				_("while populating file system"));
 			exit(1);
 		} else if (!quiet)
 			printf("%s", _("done\n"));
diff --git a/tests/f_create_symlinks/expect b/tests/f_create_symlinks/expect
index 096495c..47fa468 100644
--- a/tests/f_create_symlinks/expect
+++ b/tests/f_create_symlinks/expect
@@ -11,10 +11,10 @@ debugfs -R "symlink /l_70 /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 debugfs -R "symlink /l_500 /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" test.img
 debugfs -R "symlink /l_1023 /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx!
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" test.img
 debugfs -R "symlink /l_1024 /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx!
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" test.img
-ext2fs_symlink: Invalid argument passed to ext2 library 
+ext2fs_symlink: Invalid argument passed to ext2 library while creating symlink "l_1024"
 symlink: Invalid argument passed to ext2 library 
 debugfs -R "symlink /l_1500 /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx!
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" test.img
-ext2fs_symlink: Invalid argument passed to ext2 library 
+ext2fs_symlink: Invalid argument passed to ext2 library while creating symlink "l_1500"
 symlink: Invalid argument passed to ext2 library 
 debugfs -R "stat /l_30" test.img
 Inode: 12   Type: symlink    Mode:  0777   Flags: 0x0

[PATCH 45/54] mke2fs: add simple tests and re-alphabetize mke2fs manpage options

[Darrick J. Wong]

Add some simple tests for mke2fs -d (create image from dir) and make
the manpage options appear in alphabetic order.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/mke2fs.8.in              |   15 ++-
 tests/m_devdir/script         |   33 ++++++
 tests/m_minrootdir/expect     |  216 +++++++++++++++++++++++++++++++++++++++++
 tests/m_minrootdir/output.sed |    5 +
 tests/m_minrootdir/script     |   81 +++++++++++++++
 tests/m_rootdir/expect        |  117 ++++++++++++++++++++++
 tests/m_rootdir/output.sed    |    5 +
 tests/m_rootdir/script        |   71 +++++++++++++
 8 files changed, 536 insertions(+), 7 deletions(-)
 create mode 100644 tests/m_devdir/script
 create mode 100644 tests/m_minrootdir/expect
 create mode 100644 tests/m_minrootdir/output.sed
 create mode 100644 tests/m_minrootdir/script
 create mode 100644 tests/m_rootdir/expect
 create mode 100644 tests/m_rootdir/output.sed
 create mode 100644 tests/m_rootdir/script


diff --git a/misc/mke2fs.8.in b/misc/mke2fs.8.in
index 3230f65..40c40d3 100644
--- a/misc/mke2fs.8.in
+++ b/misc/mke2fs.8.in
@@ -18,6 +18,10 @@ mke2fs \- create an ext2/ext3/ext4 filesystem
 .I block-size
 ]
 [
+.B \-d
+.I root-directory
+]
+[
 .B \-D
 ]
 [
@@ -52,10 +56,6 @@ mke2fs \- create an ext2/ext3/ext4 filesystem
 .I number-of-inodes
 ]
 [
-.B \-d
-.I root-directory
-]
-[
 .B \-n
 ]
 [
@@ -237,6 +237,10 @@ enabled.  (See the
 man page for more details about bigalloc.)   The default cluster size if
 bigalloc is enabled is 16 times the block size.
 .TP
+.BI \-d " root-directory"
+Copy the contents of the given directory into the root directory of the
+filesystem.
+.TP
 .B \-D
 Use direct I/O when writing to the disk.  This avoids mke2fs dirtying a
 lot of buffer cache memory, which may impact other applications running
@@ -589,9 +593,6 @@ the
 ratio).  This allows the user to specify the number
 of desired inodes directly.
 .TP
-.BI \-d " root-directory"
-Add the files from the root-directory to the filesystem.
-.TP
 .BI \-o " creator-os"
 Overrides the default value of the "creator operating system" field of the
 filesystem.  The creator field is set by default to the name of the OS the
diff --git a/tests/m_devdir/script b/tests/m_devdir/script
new file mode 100644
index 0000000..5f26699
--- /dev/null
+++ b/tests/m_devdir/script
@@ -0,0 +1,33 @@
+if test -x $DEBUGFS_EXE; then
+
+test_description="create fs image from /dev"
+MKFS_DIR=/dev
+OUT=$test_name.log
+
+$MKE2FS -q -F -o Linux -T ext4 -O metadata_csum,64bit -E lazy_itable_init=1 -b 1024 -d $MKFS_DIR $TMPFILE 16384 > $OUT 2>&1
+mkfs_status=$?
+
+$DUMPE2FS $TMPFILE >> $OUT 2>&1
+$DEBUGFS -R 'ls /' $TMPFILE >> $OUT 2>&1
+
+$FSCK -f -n $TMPFILE >> $OUT 2>&1
+fsck_status=$?
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $OUT.tmp
+mv $OUT.tmp $OUT
+
+if [ $mkfs_status -ne 0 ]; then
+	echo "$test_name: $test_description: skipped"
+elif [ $mkfs_status -eq 0 ] && [ $fsck_status -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+        echo "$test_name: $test_description: failed"
+fi
+
+rm -rf $TMPFILE.cmd $OUT.sed
+unset MKFS_DIR OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/m_minrootdir/expect b/tests/m_minrootdir/expect
new file mode 100644
index 0000000..41a713f
--- /dev/null
+++ b/tests/m_minrootdir/expect
@@ -0,0 +1,216 @@
+create fs
+Filesystem volume name:   <none>
+Last mounted on:          <not available>
+Filesystem magic number:  0xEF53
+Filesystem revision #:    1 (dynamic)
+Filesystem features:      ext_attr dir_index filetype extent 64bit flex_bg sparse_super huge_file dir_nlink extra_isize metadata_csum
+Default mount options:    (none)
+Filesystem state:         clean
+Errors behavior:          Continue
+Filesystem OS type:       Linux
+Inode count:              1024
+Block count:              16384
+Reserved block count:     819
+Free blocks:              16065
+Free inodes:              1006
+First block:              1
+Block size:               1024
+Fragment size:            1024
+Group descriptor size:    64
+Blocks per group:         8192
+Fragments per group:      8192
+Inodes per group:         512
+Inode blocks per group:   128
+Flex block group size:    16
+Mount count:              0
+Check interval:           15552000 (6 months)
+Reserved blocks uid:      0
+Reserved blocks gid:      0
+First inode:              11
+Inode size:	          256
+Required extra isize:     28
+Desired extra isize:      28
+Default directory hash:   half_md4
+Checksum type:            crc32c
+
+
+Group 0: (Blocks 1-8192)
+  Primary superblock at 1, Group descriptors at 2-2
+  Block bitmap at 3 (+2)
+  Inode bitmap at 5 (+4)
+  Inode table at 7-134 (+6)
+  7876 free blocks, 494 free inodes, 4 directories, 494 unused inodes
+  Free blocks: 317-8192
+  Free inodes: 19-512
+Group 1: (Blocks 8193-16383) [INODE_UNINIT]
+  Backup superblock at 8193, Group descriptors at 8194-8194
+  Block bitmap at 4 (bg #0 + 3)
+  Inode bitmap at 6 (bg #0 + 5)
+  Inode table at 135-262 (bg #0 + 134)
+  8189 free blocks, 512 free inodes, 0 directories, 512 unused inodes
+  Free blocks: 8195-16383
+  Free inodes: 513-1024
+debugfs: stat /emptyfile
+Inode: III   Type: regular    
+Size: 0
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /bigfile
+Inode: III   Type: regular    
+Size: 32768
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /sparsefile
+Inode: III   Type: regular    
+Size: 1073741825
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /bigzerofile
+Inode: III   Type: regular    
+Size: 1073741825
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /fifo
+debugfs: stat /emptydir
+Inode: III   Type: directory    
+Size: 1024
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /dir
+Inode: III   Type: directory    
+Size: 1024
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /dir/file
+Inode: III   Type: regular    
+Size: 8
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: ex /emptyfile
+Level Entries       Logical      Physical Length Flags
+debugfs: ex /bigfile
+Level Entries       Logical      Physical Length Flags
+X 0/0 1/1 0-31 AAA-BBB 32 
+debugfs: ex /sparsefile
+Level Entries           Logical      Physical Length Flags
+Y 0/1 1/1 1-1048576 AAA 1048576
+X 1/1 1/5 1-1 AAA-BBB 1 
+X 1/1 2/5 512-512 AAA-BBB 1 
+X 1/1 3/5 1024-1024 AAA-BBB 1 
+X 1/1 4/5 524288-524288 AAA-BBB 1 
+X 1/1 5/5 1048576-1048576 AAA-BBB 1 
+debugfs: ex /bigzerofile
+Level Entries           Logical      Physical Length Flags
+debugfs: ex /dir
+Level Entries       Logical      Physical Length Flags
+X 0/0 1/1 0-0 AAA-BBB 1 
+debugfs: ex /dir/file
+Level Entries       Logical      Physical Length Flags
+X 0/0 1/1 0-0 AAA-BBB 1 
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 18/1024 files (0.0% non-contiguous), 319/16384 blocks
+minify fs
+Setting reserved blocks percentage to 0% (0 blocks)
+Resizing the filesystem on test.img to 338 (1k) blocks.
+The filesystem on test.img is now 338 (1k) blocks long.
+
+Filesystem volume name:   <none>
+Last mounted on:          <not available>
+Filesystem magic number:  0xEF53
+Filesystem revision #:    1 (dynamic)
+Filesystem features:      ext_attr dir_index filetype extent 64bit flex_bg sparse_super huge_file dir_nlink extra_isize metadata_csum
+Default mount options:    (none)
+Filesystem state:         clean
+Errors behavior:          Continue
+Filesystem OS type:       Linux
+Inode count:              512
+Block count:              338
+Reserved block count:     0
+Free blocks:              151
+Free inodes:              494
+First block:              1
+Block size:               1024
+Fragment size:            1024
+Group descriptor size:    64
+Blocks per group:         8192
+Fragments per group:      8192
+Inodes per group:         512
+Inode blocks per group:   128
+Flex block group size:    16
+Mount count:              0
+Check interval:           15552000 (6 months)
+Reserved blocks uid:      0
+Reserved blocks gid:      0
+First inode:              11
+Inode size:	          256
+Required extra isize:     28
+Desired extra isize:      28
+Default directory hash:   half_md4
+Checksum type:            crc32c
+
+
+Group 0: (Blocks 1-337)
+  Primary superblock at 1, Group descriptors at 2-2
+  Block bitmap at 3 (+2)
+  Inode bitmap at 5 (+4)
+  Inode table at 7-134 (+6)
+  151 free blocks, 494 free inodes, 4 directories, 494 unused inodes
+  Free blocks: 4, 6, 135-262, 317-337
+  Free inodes: 19-512
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 18/512 files (0.0% non-contiguous), 187/338 blocks
+minify fs (2)
+Setting reserved blocks percentage to 0% (0 blocks)
+Resizing the filesystem on test.img to 188 (1k) blocks.
+The filesystem on test.img is now 188 (1k) blocks long.
+
+Filesystem volume name:   <none>
+Last mounted on:          <not available>
+Filesystem magic number:  0xEF53
+Filesystem revision #:    1 (dynamic)
+Filesystem features:      ext_attr dir_index filetype extent 64bit flex_bg sparse_super huge_file dir_nlink extra_isize metadata_csum
+Default mount options:    (none)
+Filesystem state:         clean
+Errors behavior:          Continue
+Filesystem OS type:       Linux
+Inode count:              512
+Block count:              188
+Reserved block count:     0
+Free blocks:              1
+Free inodes:              494
+First block:              1
+Block size:               1024
+Fragment size:            1024
+Group descriptor size:    64
+Blocks per group:         8192
+Fragments per group:      8192
+Inodes per group:         512
+Inode blocks per group:   128
+Flex block group size:    16
+Mount count:              0
+Check interval:           15552000 (6 months)
+Reserved blocks uid:      0
+Reserved blocks gid:      0
+First inode:              11
+Inode size:	          256
+Required extra isize:     28
+Desired extra isize:      28
+Default directory hash:   half_md4
+Checksum type:            crc32c
+
+
+Group 0: (Blocks 1-187)
+  Primary superblock at 1, Group descriptors at 2-2
+  Block bitmap at 3 (+2)
+  Inode bitmap at 5 (+4)
+  Inode table at 7-134 (+6)
+  1 free blocks, 494 free inodes, 4 directories, 494 unused inodes
+  Free blocks: 187
+  Free inodes: 19-512
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 18/512 files (5.6% non-contiguous), 187/188 blocks
diff --git a/tests/m_minrootdir/output.sed b/tests/m_minrootdir/output.sed
new file mode 100644
index 0000000..2e76967
--- /dev/null
+++ b/tests/m_minrootdir/output.sed
@@ -0,0 +1,5 @@
+s/^[[:space:]]*\([0-9]*\)\/[[:space:]]*\([0-9]*\)[[:space:]]*\([0-9]*\)\/[[:space:]]*\([0-9]*\)[[:space:]]*\([0-9]*\)[[:space:]]*-[[:space:]]*\([0-9]*\)[[:space:]]*[0-9]*[[:space:]]*-[[:space:]]*[0-9]*[[:space:]]*\([0-9]*\)/X \1\/\2 \3\/\4 \5-\6 AAA-BBB \7/g
+s/^[[:space:]]*\([0-9]*\)\/[[:space:]]*\([0-9]*\)[[:space:]]*\([0-9]*\)\/[[:space:]]*\([0-9]*\)[[:space:]]*\([0-9]*\)[[:space:]]*-[[:space:]]*\([0-9]*\)[[:space:]]*[0-9]*[[:space:]]*\([0-9]*\)/Y \1\/\2 \3\/\4 \5-\6 AAA \7/g
+s/Mode:.*$//g
+s/User:.*Size:/Size:/g
+s/^Inode: [0-9]*/Inode: III/g
diff --git a/tests/m_minrootdir/script b/tests/m_minrootdir/script
new file mode 100644
index 0000000..662e76f
--- /dev/null
+++ b/tests/m_minrootdir/script
@@ -0,0 +1,81 @@
+if test -x $DEBUGFS_EXE -a -x $RESIZE2FS_EXE; then
+
+test_description="create fs image from dir, then minimize it"
+MKFS_DIR=$TMPFILE.dir
+OUT=$test_name.log
+EXP=$test_dir/expect
+
+rm -rf $MKFS_DIR
+mkdir -p $MKFS_DIR
+mkdir $MKFS_DIR/dir
+mkdir $MKFS_DIR/emptydir
+dd if=/dev/zero of=$MKFS_DIR/bigzerofile bs=1 count=1 seek=1073741824 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=1024 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=524288 conv=notrunc 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=1048576 conv=notrunc 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=536870912 conv=notrunc 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=1073741824 conv=notrunc 2> /dev/null
+dd if=/dev/zero bs=1024 count=32 2> /dev/null | tr '\0' 'a' > $MKFS_DIR/bigfile
+touch $MKFS_DIR/emptyfile
+echo "Test me" > $MKFS_DIR/dir/file
+
+echo "create fs" > $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^has_journal,metadata_csum,64bit,^resize_inode -E lazy_itable_init=1 -b 1024 -d $MKFS_DIR $TMPFILE 16384 >> $OUT 2>&1
+
+$DUMPE2FS $TMPFILE >> $OUT 2>&1
+cat > $TMPFILE.cmd << ENDL
+stat /emptyfile
+stat /bigfile
+stat /sparsefile
+stat /bigzerofile
+stat /fifo
+stat /emptydir
+stat /dir
+stat /dir/file
+ENDL
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE 2>&1 | egrep "(stat|Size:|Type:)" | sed -f $test_dir/output.sed >> $OUT
+
+cat > $TMPFILE.cmd << ENDL
+ex /emptyfile
+ex /bigfile
+ex /sparsefile
+ex /bigzerofile
+ex /dir
+ex /dir/file
+ENDL
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE 2>&1 | sed -f $test_dir/output.sed >> $OUT
+$FSCK -f -n $TMPFILE >> $OUT 2>&1
+
+echo "minify fs" >> $OUT
+$TUNE2FS -m 0 $TMPFILE >> $OUT 2>&1
+$RESIZE2FS -M $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE >> $OUT 2>&1
+$FSCK -f -n $TMPFILE >> $OUT 2>&1
+
+echo "minify fs (2)" >> $OUT
+$TUNE2FS -m 0 $TMPFILE >> $OUT 2>&1
+$RESIZE2FS -M $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE >> $OUT 2>&1
+$FSCK -f -n $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $OUT.tmp
+mv $OUT.tmp $OUT
+
+# Do the verification
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+        echo "$test_name: $test_description: failed"
+        diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+rm -rf $TMPFILE.cmd $MKFS_DIR $OUT.sed
+unset MKFS_DIR OUT EXP
+
+else #if test -x $DEBUGFS_EXE -a -x RESIZE2FS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/m_rootdir/expect b/tests/m_rootdir/expect
new file mode 100644
index 0000000..a5314f1
--- /dev/null
+++ b/tests/m_rootdir/expect
@@ -0,0 +1,117 @@
+Filesystem volume name:   <none>
+Last mounted on:          <not available>
+Filesystem magic number:  0xEF53
+Filesystem revision #:    1 (dynamic)
+Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super huge_file dir_nlink extra_isize metadata_csum
+Default mount options:    (none)
+Filesystem state:         clean
+Errors behavior:          Continue
+Filesystem OS type:       Linux
+Inode count:              1024
+Block count:              16384
+Reserved block count:     819
+Free blocks:              14786
+Free inodes:              1005
+First block:              1
+Block size:               1024
+Fragment size:            1024
+Group descriptor size:    64
+Reserved GDT blocks:      127
+Blocks per group:         8192
+Fragments per group:      8192
+Inodes per group:         512
+Inode blocks per group:   128
+Flex block group size:    16
+Mount count:              0
+Check interval:           15552000 (6 months)
+Reserved blocks uid:      0
+Reserved blocks gid:      0
+First inode:              11
+Inode size:	          256
+Required extra isize:     28
+Desired extra isize:      28
+Journal inode:            8
+Default directory hash:   half_md4
+Journal backup:           inode blocks
+Checksum type:            crc32c
+Journal features:         (none)
+Journal size:             1024k
+Journal length:           1024
+Journal sequence:         0x00000001
+Journal start:            0
+
+
+Group 0: (Blocks 1-8192)
+  Primary superblock at 1, Group descriptors at 2-2
+  Reserved GDT blocks at 3-129
+  Block bitmap at 130 (+129)
+  Inode bitmap at 132 (+131)
+  Inode table at 134-261 (+133)
+  7748 free blocks, 493 free inodes, 4 directories, 493 unused inodes
+  Free blocks: 445-8192
+  Free inodes: 20-512
+Group 1: (Blocks 8193-16383) [INODE_UNINIT]
+  Backup superblock at 8193, Group descriptors at 8194-8194
+  Reserved GDT blocks at 8195-8321
+  Block bitmap at 131 (bg #0 + 130)
+  Inode bitmap at 133 (bg #0 + 132)
+  Inode table at 262-389 (bg #0 + 261)
+  7038 free blocks, 512 free inodes, 0 directories, 512 unused inodes
+  Free blocks: 9346-16383
+  Free inodes: 513-1024
+debugfs: stat /emptyfile
+Inode: III   Type: regular    
+Size: 0
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /bigfile
+Inode: III   Type: regular    
+Size: 32768
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /sparsefile
+Inode: III   Type: regular    
+Size: 1073741825
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /bigzerofile
+Inode: III   Type: regular    
+Size: 1073741825
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /fifo
+debugfs: stat /emptydir
+Inode: III   Type: directory    
+Size: 1024
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /dir
+Inode: III   Type: directory    
+Size: 1024
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: stat /dir/file
+Inode: III   Type: regular    
+Size: 8
+Fragment:  Address: 0    Number: 0    Size: 0
+debugfs: ex /emptyfile
+Level Entries       Logical      Physical Length Flags
+debugfs: ex /bigfile
+Level Entries       Logical      Physical Length Flags
+X 0/0 1/1 0-31 AAA-BBB 32 
+debugfs: ex /sparsefile
+Level Entries           Logical      Physical Length Flags
+Y 0/1 1/1 1-1048576 AAA 1048576
+X 1/1 1/5 1-1 AAA-BBB 1 
+X 1/1 2/5 512-512 AAA-BBB 1 
+X 1/1 3/5 1024-1024 AAA-BBB 1 
+X 1/1 4/5 524288-524288 AAA-BBB 1 
+X 1/1 5/5 1048576-1048576 AAA-BBB 1 
+debugfs: ex /bigzerofile
+Level Entries           Logical      Physical Length Flags
+debugfs: ex /dir
+Level Entries       Logical      Physical Length Flags
+X 0/0 1/1 0-0 AAA-BBB 1 
+debugfs: ex /dir/file
+Level Entries       Logical      Physical Length Flags
+X 0/0 1/1 0-0 AAA-BBB 1 
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 19/1024 files (0.0% non-contiguous), 1598/16384 blocks
diff --git a/tests/m_rootdir/output.sed b/tests/m_rootdir/output.sed
new file mode 100644
index 0000000..2e76967
--- /dev/null
+++ b/tests/m_rootdir/output.sed
@@ -0,0 +1,5 @@
+s/^[[:space:]]*\([0-9]*\)\/[[:space:]]*\([0-9]*\)[[:space:]]*\([0-9]*\)\/[[:space:]]*\([0-9]*\)[[:space:]]*\([0-9]*\)[[:space:]]*-[[:space:]]*\([0-9]*\)[[:space:]]*[0-9]*[[:space:]]*-[[:space:]]*[0-9]*[[:space:]]*\([0-9]*\)/X \1\/\2 \3\/\4 \5-\6 AAA-BBB \7/g
+s/^[[:space:]]*\([0-9]*\)\/[[:space:]]*\([0-9]*\)[[:space:]]*\([0-9]*\)\/[[:space:]]*\([0-9]*\)[[:space:]]*\([0-9]*\)[[:space:]]*-[[:space:]]*\([0-9]*\)[[:space:]]*[0-9]*[[:space:]]*\([0-9]*\)/Y \1\/\2 \3\/\4 \5-\6 AAA \7/g
+s/Mode:.*$//g
+s/User:.*Size:/Size:/g
+s/^Inode: [0-9]*/Inode: III/g
diff --git a/tests/m_rootdir/script b/tests/m_rootdir/script
new file mode 100644
index 0000000..fbe1b31
--- /dev/null
+++ b/tests/m_rootdir/script
@@ -0,0 +1,71 @@
+if test -x $DEBUGFS_EXE; then
+
+test_description="create fs image from dir"
+MKFS_DIR=$TMPFILE.dir
+OUT=$test_name.log
+EXP=$test_dir/expect
+
+rm -rf $MKFS_DIR
+mkdir -p $MKFS_DIR
+touch $MKFS_DIR/emptyfile
+dd if=/dev/zero bs=1024 count=32 2> /dev/null | tr '\0' 'a' > $MKFS_DIR/bigfile
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=1024 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=524288 conv=notrunc 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=1048576 conv=notrunc 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=536870912 conv=notrunc 2> /dev/null
+echo "M" | dd of=$MKFS_DIR/sparsefile bs=1 count=1 seek=1073741824 conv=notrunc 2> /dev/null
+dd if=/dev/zero of=$MKFS_DIR/bigzerofile bs=1 count=1 seek=1073741824 2> /dev/null
+ln $MKFS_DIR/bigzerofile $MKFS_DIR/bigzerofile_hardlink
+ln -s /silly_bs_link $MKFS_DIR/silly_bs_link
+mkdir $MKFS_DIR/emptydir
+mkdir $MKFS_DIR/dir
+echo "Test me" > $MKFS_DIR/dir/file
+
+$MKE2FS -q -F -o Linux -T ext4 -O metadata_csum,64bit -E lazy_itable_init=1 -b 1024 -d $MKFS_DIR $TMPFILE 16384 > $OUT 2>&1
+
+$DUMPE2FS $TMPFILE >> $OUT 2>&1
+cat > $TMPFILE.cmd << ENDL
+stat /emptyfile
+stat /bigfile
+stat /sparsefile
+stat /bigzerofile
+stat /fifo
+stat /emptydir
+stat /dir
+stat /dir/file
+ENDL
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE 2>&1 | egrep "(stat|Size:|Type:)" | sed -f $test_dir/output.sed >> $OUT
+
+cat > $TMPFILE.cmd << ENDL
+ex /emptyfile
+ex /bigfile
+ex /sparsefile
+ex /bigzerofile
+ex /dir
+ex /dir/file
+ENDL
+$DEBUGFS -f $TMPFILE.cmd $TMPFILE 2>&1 | sed -f $test_dir/output.sed >> $OUT 2>&1
+
+$FSCK -f -n $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $OUT.tmp
+mv $OUT.tmp $OUT
+
+# Do the verification
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+        echo "$test_name: $test_description: failed"
+        diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+rm -rf $TMPFILE.cmd $MKFS_DIR $OUT.sed
+unset MKFS_DIR OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

[PATCH 46/54] contrib: script to create minified ext4 image from a directory

[Darrick J. Wong]

The dir2fs script converts a directory into a minimized ext4 filesystem.
FS creation parameters are tweaked to reduce as much FS overhead as
possible, and to leave as few unused blocks and inodes as possible.
Given that mke2fs -d lays out files linearly from the beginning of the
FS, using resize2fs -M is not as horrible as it usually is.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 contrib/dir2fs |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100755 contrib/dir2fs


diff --git a/contrib/dir2fs b/contrib/dir2fs
new file mode 100755
index 0000000..abcecb3
--- /dev/null
+++ b/contrib/dir2fs
@@ -0,0 +1,66 @@
+#!/bin/sh
+
+dir="$1"
+dev="$2"
+
+if [ "$1" = "--help" ] || [ ! -d "${dir}" ]; then
+	echo "Usage: $0 dir [mke2fs args] dev"
+	exit 1
+fi
+
+shift
+
+# Goal: Put all the files at the beginning (which mke2fs does) and minimize
+# the number of free inodes given the minimum number of blocks required.
+# Hence all this math to get the inode ratio just right.
+
+bytes="$(du -ks "${dir}" | awk '{print $1}')"
+bytes="$((bytes * 1024))"
+inodes="$(find "${dir}" -print0 | xargs -0 stat -c '%i' | sort -g | uniq | wc -l)"
+block_sz=4096
+inode_sz=256
+sb_overhead=4096
+blocks_per_group="$((block_sz * 8))"
+bytes_per_group="$((blocks_per_group * block_sz))"
+inode_bytes="$((inodes * inode_sz))"
+
+# Estimate overhead with the minimum number of groups...
+nr_groups="$(( (bytes + inode_bytes + bytes_per_group - 1) / bytes_per_group))"
+inode_bytes_per_group="$((inode_bytes / nr_groups))"
+inode_blocks_per_group="$(( (inode_bytes_per_group + (block_sz - 1)) / block_sz ))"
+per_grp_overhead="$(( ((3 + inode_blocks_per_group) * block_sz) + 64 ))"
+overhead="$(( sb_overhead + (per_grp_overhead * nr_groups) ))"
+used_bytes="$((bytes + overhead))"
+
+# Then do it again with the real number of groups.
+nr_groups="$(( (used_bytes + (bytes_per_group - 1)) / bytes_per_group))"
+tot_blocks="$((nr_groups * blocks_per_group))"
+tot_bytes="$((tot_blocks * block_sz))"
+
+ratio="$((bytes / inodes))"
+mkfs_blocks="$((tot_blocks * 4 / 3))"
+
+mke2fs -i "${ratio}" -T ext4 -d "${dir}" -O ^resize_inode,sparse_super2,metadata_csum,64bit,^has_journal -E packed_meta_blocks=1,num_backup_sb=0 -b "${block_sz}" -I "${inodesz}" -F "${dev}" "${mkfs_blocks}" || exit
+
+e2fsck -fyD "${dev}"
+
+blocks="$(dumpe2fs -h "${dev}" 2>&1 | grep 'Block count:' | awk '{print $3}')"
+while resize2fs -f -M "${dev}"; do
+	new_blocks="$(dumpe2fs -h "${dev}" 2>&1 | grep 'Block count:' | awk '{print $3}')"
+	if [ "${new_blocks}" -eq "${blocks}" ]; then
+		break;
+	fi
+	blocks="${new_blocks}"
+done
+
+if [ ! -b "${dev}" ]; then
+    truncate -s "$((blocks * block_sz))" "${dev}" || (e2image -ar "${dev}" "${dev}.min"; mv "${dev}.min" "${dev}")
+fi
+
+e2fsck -fy "${dev}"
+
+dir_blocks="$((bytes / block_sz))"
+overhead="$((blocks - dir_blocks))"
+echo "Minimized image overhead: $((100 * overhead / dir_blocks))%"
+
+exit 0

[PATCH 47/54] libext2fs: support allocating uninit blocks in bmap2()

[Darrick J. Wong]

As part of supporting fallocate-like functionality, extend
ext2fs_bmap() with two flags -- BMAP_UNINIT and BMAP_ZERO.  The first
will cause it to mark/set a block uninitialized, if it's part of an
extent based file.  For a block mapped file, the mapping is put in,
but there is no way to remember the uninitialized status.  The second
flag causes the block to be zeroed to support the use case of
emulating uninitialized blocks on a block-map file by zeroing them.

Eventually fallocate or fuse2fs or somebody will use these.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/bmap.c   |    9 +++++++--
 lib/ext2fs/ext2fs.h |    2 ++
 2 files changed, 9 insertions(+), 2 deletions(-)


diff --git a/lib/ext2fs/bmap.c b/lib/ext2fs/bmap.c
index cb3f5a1..c18f742 100644
--- a/lib/ext2fs/bmap.c
+++ b/lib/ext2fs/bmap.c
@@ -214,10 +214,13 @@ static errcode_t extent_bmap(ext2_filsys fs, ext2_ino_t ino,
 	errcode_t		retval = 0;
 	blk64_t			blk64 = 0;
 	int			alloc = 0;
+	int			set_flags;
+
+	set_flags = bmap_flags & BMAP_UNINIT ? EXT2_EXTENT_SET_BMAP_UNINIT : 0;
 
 	if (bmap_flags & BMAP_SET) {
 		retval = ext2fs_extent_set_bmap(handle, block,
-						*phys_blk, 0);
+						*phys_blk, set_flags);
 		return retval;
 	}
 	retval = ext2fs_extent_goto(handle, block);
@@ -254,7 +257,7 @@ got_block:
 		alloc++;
 	set_extent:
 		retval = ext2fs_extent_set_bmap(handle, block,
-						blk64, 0);
+						blk64, set_flags);
 		if (retval) {
 			ext2fs_block_alloc_stats2(fs, blk64, -1);
 			return retval;
@@ -441,6 +444,8 @@ errcode_t ext2fs_bmap2(ext2_filsys fs, ext2_ino_t ino, struct ext2_inode *inode,
 	if (retval == 0)
 		*phys_blk = blk32;
 done:
+	if (*phys_blk && retval == 0 && (bmap_flags & BMAP_ZERO))
+		retval = ext2fs_zero_blocks2(fs, *phys_blk, 1, NULL, NULL);
 	if (buf)
 		ext2fs_free_mem(&buf);
 	if (handle)
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index ee8f184..540a263 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -527,6 +527,8 @@ typedef struct ext2_icount *ext2_icount_t;
  */
 #define BMAP_ALLOC	0x0001
 #define BMAP_SET	0x0002
+#define BMAP_UNINIT	0x0004
+#define BMAP_ZERO	0x0008
 
 /*
  * Returned flags from ext2fs_bmap

[PATCH 48/54] libext2fs: find/alloc a range of empty blocks

[Darrick J. Wong]

Provide a function that, given a goal pblk and a range, will try to
find a run of free blocks to satisfy the allocation.  By default the
function will look anywhere in the filesystem for the run, though this
can be constrained with optional flags.  One flag indicates that the
range must start at the goal block; the other flag indicates that we
should not return a range shorter than len.

v2: Add a second function to allocate a range of blocks.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/alloc.c  |  141 +++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/ext2fs/ext2fs.h |   11 ++++
 2 files changed, 152 insertions(+)


diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 9901ca5..4c3b620 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -26,6 +26,16 @@
 #include "ext2_fs.h"
 #include "ext2fs.h"
 
+#define min(a, b) ((a) < (b) ? (a) : (b))
+
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
 /*
  * Clear the uninit block bitmap flag if necessary
  */
@@ -346,3 +356,134 @@ no_blocks:
 		group = group & ~((1 << (log_flex)) - 1);
 	return ext2fs_group_first_block2(fs, group);
 }
+
+/*
+ * Starting at _goal_, scan around the filesystem to find a run of free blocks
+ * that's at least _len_ blocks long.  Possible flags:
+ * - EXT2_NEWRANGE_EXACT_GOAL: The range of blocks must start at _goal_.
+ * - EXT2_NEWRANGE_MIN_LENGTH: do not return a allocation shorter than _len_.
+ * - EXT2_NEWRANGE_ZERO_BLOCKS: Zero blocks pblk to pblk+plen before returning.
+ *
+ * The starting block is returned in _pblk_ and the length is returned via
+ * _plen_.  The blocks are not marked in the bitmap; the caller must mark
+ * however much of the returned run they actually use, hopefully via
+ * ext2fs_block_alloc_stats_range().
+ *
+ * This function can return a range that is longer than what was requested.
+ */
+errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
+			   blk64_t len, ext2fs_block_bitmap map, blk64_t *pblk,
+			   blk64_t *plen)
+{
+	errcode_t retval;
+	blk64_t start, end, b;
+	int looped = 0;
+	blk64_t max_blocks = ext2fs_blocks_count(fs->super);
+
+	dbg_printf("%s: flags=0x%x goal=%llu len=%llu\n", __func__, flags,
+		   goal, len);
+	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
+	if (len == 0 || (flags & ~EXT2_NEWRANGE_ALL_FLAGS))
+		return EXT2_ET_INVALID_ARGUMENT;
+	if (!map)
+		map = fs->block_map;
+	if (!map)
+		return EXT2_ET_NO_BLOCK_BITMAP;
+	if (!goal || goal >= ext2fs_blocks_count(fs->super))
+		goal = fs->super->s_first_data_block;
+
+	start = goal;
+	while (!looped || start <= goal) {
+		retval = ext2fs_find_first_zero_block_bitmap2(map, start,
+							      max_blocks - 1,
+							      &start);
+		if (retval == ENOENT) {
+			/*
+			 * If there are no free blocks beyond the starting
+			 * point, try scanning the whole filesystem, unless the
+			 * user told us only to allocate from _goal_, or if
+			 * we're already scanning the whole filesystem.
+			 */
+			if (flags & EXT2_NEWRANGE_FIXED_GOAL ||
+			    start == fs->super->s_first_data_block)
+				goto fail;
+			start = fs->super->s_first_data_block;
+			continue;
+		} else if (retval)
+			goto errout;
+
+		if (flags & EXT2_NEWRANGE_FIXED_GOAL && start != goal)
+			goto fail;
+
+		b = min(start + len - 1, max_blocks - 1);
+		retval =  ext2fs_find_first_set_block_bitmap2(map, start, b,
+							      &end);
+		if (retval == ENOENT)
+			end = b + 1;
+		else if (retval)
+			goto errout;
+
+		if (!(flags & EXT2_NEWRANGE_MIN_LENGTH) ||
+		    (end - start) >= len) {
+			/* Success! */
+			*pblk = start;
+			*plen = end - start;
+			dbg_printf("%s: new_range goal=%llu--%llu "
+				   "blk=%llu--%llu %llu\n",
+				   __func__, goal, goal + len - 1,
+				   *pblk, *pblk + *plen - 1, *plen);
+
+			for (b = start; b < end;
+			     b += fs->super->s_blocks_per_group)
+				clear_block_uninit(fs,
+						ext2fs_group_of_blk2(fs, b));
+			return 0;
+		}
+
+		if (flags & EXT2_NEWRANGE_FIXED_GOAL)
+			goto fail;
+		start = end;
+		if (start >= max_blocks) {
+			if (looped)
+				goto fail;
+			looped = 1;
+			start = fs->super->s_first_data_block;
+		}
+	}
+
+fail:
+	retval = EXT2_ET_BLOCK_ALLOC_FAIL;
+errout:
+	return retval;
+}
+
+errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
+			     blk_t len, blk64_t *ret)
+{
+	int newr_flags = EXT2_NEWRANGE_MIN_LENGTH;
+	errcode_t retval;
+	blk64_t plen;
+
+	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
+	if (len == 0 || (flags & ~EXT2_ALLOCRANGE_ALL_FLAGS))
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (flags & EXT2_ALLOCRANGE_FIXED_GOAL)
+		newr_flags |= EXT2_NEWRANGE_FIXED_GOAL;
+
+	retval = ext2fs_new_range(fs, newr_flags, goal, len, NULL, ret, &plen);
+	if (retval)
+		return retval;
+
+	if (plen < len)
+		return EXT2_ET_BLOCK_ALLOC_FAIL;
+
+	if (flags & EXT2_ALLOCRANGE_ZERO_BLOCKS) {
+		retval = ext2fs_zero_blocks2(fs, *ret, len, NULL, NULL);
+		if (retval)
+			return retval;
+	}
+
+	ext2fs_block_alloc_stats_range(fs, *ret, len, +1);
+	return retval;
+}
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 540a263..a8c3992 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -697,6 +697,17 @@ extern void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 							      blk64_t *ret));
 blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino,
 			       struct ext2_inode *inode, blk64_t lblk);
+#define EXT2_NEWRANGE_FIXED_GOAL	(0x1)
+#define EXT2_NEWRANGE_MIN_LENGTH	(0x2)
+#define EXT2_NEWRANGE_ALL_FLAGS		(0x3)
+errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
+			   blk64_t len, ext2fs_block_bitmap map, blk64_t *pblk,
+			   blk64_t *plen);
+#define EXT2_ALLOCRANGE_FIXED_GOAL	(0x1)
+#define EXT2_ALLOCRANGE_ZERO_BLOCKS	(0x2)
+#define EXT2_ALLOCRANGE_ALL_FLAGS	(0x3)
+errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
+			     blk_t len, blk64_t *ret);
 
 /* alloc_sb.c */
 extern int ext2fs_reserve_super_and_bgd(ext2_filsys fs,

[PATCH 49/54] libext2fs: add new hooks to support large allocations

[Darrick J. Wong]

Add a new get_alloc_blocks hook and a block_alloc_stats_range hook so
that e2fsck can capture allocation requests spanning more than a
block to its block_found_map.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c           |   45 +++++++++++++++++++++++++++++++++++++++++++++
 lib/ext2fs/alloc.c       |   37 ++++++++++++++++++++++++++++++++++++-
 lib/ext2fs/alloc_stats.c |   16 ++++++++++++++++
 lib/ext2fs/ext2fs.h      |   16 ++++++++++++++++
 4 files changed, 113 insertions(+), 1 deletion(-)


diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 2658f39..139daef 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -3929,6 +3929,26 @@ static errcode_t e2fsck_get_alloc_block(ext2_filsys fs, blk64_t goal,
 	return (0);
 }
 
+static errcode_t e2fsck_new_range(ext2_filsys fs, int flags, blk64_t goal,
+				  blk64_t len, blk64_t *pblk, blk64_t *plen)
+{
+	e2fsck_t ctx = (e2fsck_t) fs->priv_data;
+	errcode_t	retval;
+
+	if (ctx->block_found_map)
+		return ext2fs_new_range(fs, flags, goal, len,
+					ctx->block_found_map, pblk, plen);
+
+	if (!fs->block_map) {
+		retval = ext2fs_read_block_bitmap(fs);
+		if (retval)
+			return retval;
+	}
+
+	return ext2fs_new_range(fs, flags, goal, len, fs->block_map,
+				pblk, plen);
+}
+
 static void e2fsck_block_alloc_stats(ext2_filsys fs, blk64_t blk, int inuse)
 {
 	e2fsck_t ctx = (e2fsck_t) fs->priv_data;
@@ -3948,6 +3968,28 @@ static void e2fsck_block_alloc_stats(ext2_filsys fs, blk64_t blk, int inuse)
 	}
 }
 
+static void e2fsck_block_alloc_stats_range(ext2_filsys fs, blk64_t blk,
+					   blk_t num, int inuse)
+{
+	e2fsck_t ctx = (e2fsck_t) fs->priv_data;
+
+	/* Never free a critical metadata block */
+	if (ctx->block_found_map &&
+	    ctx->block_metadata_map &&
+	    inuse < 0 &&
+	    ext2fs_test_block_bitmap_range2(ctx->block_metadata_map, blk, num))
+		return;
+
+	if (ctx->block_found_map) {
+		if (inuse > 0)
+			ext2fs_mark_block_bitmap_range2(ctx->block_found_map,
+							blk, num);
+		else
+			ext2fs_unmark_block_bitmap_range2(ctx->block_found_map,
+							blk, num);
+	}
+}
+
 void e2fsck_use_inode_shortcuts(e2fsck_t ctx, int use_shortcuts)
 {
 	ext2_filsys fs = ctx->fs;
@@ -3971,4 +4013,7 @@ void e2fsck_intercept_block_allocations(e2fsck_t ctx)
 	ext2fs_set_alloc_block_callback(ctx->fs, e2fsck_get_alloc_block, 0);
 	ext2fs_set_block_alloc_stats_callback(ctx->fs,
 						e2fsck_block_alloc_stats, 0);
+	ext2fs_set_new_range_callback(ctx->fs, e2fsck_new_range, NULL);
+	ext2fs_set_block_alloc_stats_range_callback(ctx->fs,
+					e2fsck_block_alloc_stats_range, NULL);
 }
diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 4c3b620..86e7f99 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -379,12 +379,32 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
 	blk64_t start, end, b;
 	int looped = 0;
 	blk64_t max_blocks = ext2fs_blocks_count(fs->super);
+	errcode_t (*nrf)(ext2_filsys fs, int flags, blk64_t goal,
+			 blk64_t len, blk64_t *pblk, blk64_t *plen);
 
 	dbg_printf("%s: flags=0x%x goal=%llu len=%llu\n", __func__, flags,
 		   goal, len);
 	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
 	if (len == 0 || (flags & ~EXT2_NEWRANGE_ALL_FLAGS))
 		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (!map && fs->new_range) {
+		/*
+		 * In case there are clients out there whose new_range
+		 * handlers call ext2fs_new_range with a NULL block map,
+		 * temporarily swap out the function pointer so that we don't
+		 * end up in an infinite loop.
+		 */
+		nrf = fs->new_range;
+		fs->new_range = NULL;
+		retval = nrf(fs, flags, goal, len, pblk, plen);
+		fs->new_range = nrf;
+		if (retval)
+			return retval;
+		start = *pblk;
+		end = *pblk + *plen;
+		goto allocated;
+	}
 	if (!map)
 		map = fs->block_map;
 	if (!map)
@@ -432,7 +452,7 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
 				   "blk=%llu--%llu %llu\n",
 				   __func__, goal, goal + len - 1,
 				   *pblk, *pblk + *plen - 1, *plen);
-
+allocated:
 			for (b = start; b < end;
 			     b += fs->super->s_blocks_per_group)
 				clear_block_uninit(fs,
@@ -457,6 +477,21 @@ errout:
 	return retval;
 }
 
+void ext2fs_set_new_range_callback(ext2_filsys fs,
+	errcode_t (*func)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen),
+	errcode_t (**old)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen))
+{
+	if (!fs || fs->magic != EXT2_ET_MAGIC_EXT2FS_FILSYS)
+		return;
+
+	if (old)
+		*old = fs->new_range;
+
+	fs->new_range = func;
+}
+
 errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
 			     blk_t len, blk64_t *ret)
 {
diff --git a/lib/ext2fs/alloc_stats.c b/lib/ext2fs/alloc_stats.c
index aca5004..3949f61 100644
--- a/lib/ext2fs/alloc_stats.c
+++ b/lib/ext2fs/alloc_stats.c
@@ -145,4 +145,20 @@ void ext2fs_block_alloc_stats_range(ext2_filsys fs, blk64_t blk,
 	}
 	ext2fs_mark_super_dirty(fs);
 	ext2fs_mark_bb_dirty(fs);
+	if (fs->block_alloc_stats_range)
+		(fs->block_alloc_stats_range)(fs, blk, num, inuse);
+}
+
+void ext2fs_set_block_alloc_stats_range_callback(ext2_filsys fs,
+	void (*func)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse),
+	void (**old)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse))
+{
+	if (!fs || fs->magic != EXT2_ET_MAGIC_EXT2FS_FILSYS)
+		return;
+	if (old)
+		*old = fs->block_alloc_stats_range;
+
+	fs->block_alloc_stats_range = func;
 }
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index a8c3992..d4ab853 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -279,6 +279,12 @@ struct struct_ext2_filsys {
 
 	io_channel			journal_io;
 	char				*journal_name;
+
+	/* New block range allocation hooks */
+	errcode_t (*new_range)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen);
+	void (*block_alloc_stats_range)(ext2_filsys fs, blk64_t blk, blk_t num,
+					int inuse);
 };
 
 #if EXT2_FLAT_INCLUDES
@@ -697,6 +703,16 @@ extern void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 							      blk64_t *ret));
 blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino,
 			       struct ext2_inode *inode, blk64_t lblk);
+extern void ext2fs_set_new_range_callback(ext2_filsys fs,
+	errcode_t (*func)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen),
+	errcode_t (**old)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen));
+extern void ext2fs_set_block_alloc_stats_range_callback(ext2_filsys fs,
+	void (*func)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse),
+	void (**old)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse));
 #define EXT2_NEWRANGE_FIXED_GOAL	(0x1)
 #define EXT2_NEWRANGE_MIN_LENGTH	(0x2)
 #define EXT2_NEWRANGE_ALL_FLAGS		(0x3)

[PATCH 50/54] libext2fs: implement fallocate

[Darrick J. Wong]

Create a library function to perform fallocation on arbitrary files,
and wire up a few users for this function.  This is a bit more intense
than Ted's original mk_hugefiles implementation since we have to honor
any blocks that may already be allocated to the file.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/Makefile.in |    8 
 lib/ext2fs/ext2fs.h    |   10 +
 lib/ext2fs/fallocate.c |  853 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 871 insertions(+)
 create mode 100644 lib/ext2fs/fallocate.c


diff --git a/lib/ext2fs/Makefile.in b/lib/ext2fs/Makefile.in
index 4b64101..19296ba 100644
--- a/lib/ext2fs/Makefile.in
+++ b/lib/ext2fs/Makefile.in
@@ -80,6 +80,7 @@ OBJS= $(DEBUGFS_LIB_OBJS) $(RESIZE_LIB_OBJS) $(E2IMAGE_LIB_OBJS) \
 	expanddir.o \
 	ext_attr.o \
 	extent.o \
+	fallocate.o \
 	fileio.o \
 	finddev.o \
 	flushb.o \
@@ -771,6 +772,13 @@ extent.o: $(srcdir)/extent.c $(top_builddir)/lib/config.h \
  $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/ext2_ext_attr.h \
  $(srcdir)/bitops.h $(srcdir)/e2image.h
+fallocate.o: $(srcdir)/fallocate.c $(top_builddir)/lib/config.h \
+ $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fsP.h \
+ $(srcdir)/ext2fs.h $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h \
+ $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/ext2_ext_attr.h \
+ $(srcdir)/bitops.h $(srcdir)/e2image.h
 fileio.o: $(srcdir)/fileio.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
  $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index d4ab853..cdd2945 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1259,6 +1259,16 @@ extern errcode_t ext2fs_extent_goto2(ext2_extent_handle_t handle,
 extern errcode_t ext2fs_extent_fix_parents(ext2_extent_handle_t handle);
 size_t ext2fs_max_extent_depth(ext2_extent_handle_t handle);
 
+/* fallocate.c */
+#define EXT2_FALLOCATE_ZERO_BLOCKS	(0x1)
+#define EXT2_FALLOCATE_FORCE_INIT	(0x2)
+#define EXT2_FALLOCATE_FORCE_UNINIT	(0x4)
+#define EXT2_FALLOCATE_INIT_BEYOND_EOF	(0x8)
+#define EXT2_FALLOCATE_ALL_FLAGS	(0xF)
+errcode_t ext2fs_fallocate(ext2_filsys fs, int flags, ext2_ino_t ino,
+			   struct ext2_inode *inode, blk64_t goal,
+			   blk64_t start, blk64_t len);
+
 /* fileio.c */
 extern errcode_t ext2fs_file_open2(ext2_filsys fs, ext2_ino_t ino,
 				   struct ext2_inode *inode,
diff --git a/lib/ext2fs/fallocate.c b/lib/ext2fs/fallocate.c
new file mode 100644
index 0000000..8b502d5
--- /dev/null
+++ b/lib/ext2fs/fallocate.c
@@ -0,0 +1,853 @@
+/*
+ * fallocate.c -- Allocate large chunks of file.
+ *
+ * Copyright (C) 2014 Oracle.
+ *
+ * %Begin-Header%
+ * This file may be redistributed under the terms of the GNU Library
+ * General Public License, version 2.
+ * %End-Header%
+ */
+
+#include "config.h"
+
+#include "ext2_fs.h"
+#include "ext2fs.h"
+#define min(a, b) ((a) < (b) ? (a) : (b))
+
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
+/*
+ * Extent-based fallocate code.
+ *
+ * Find runs of unmapped logical blocks by starting at start and walking the
+ * extents until we reach the end of the range we want.
+ *
+ * For each run of unmapped blocks, try to find the extents on either side of
+ * the range.  If there's a left extent that can grow by at least a cluster and
+ * there are lblocks between start and the next lcluster after start, see if
+ * there's an implied cluster allocation; if so, zero the blocks (if the left
+ * extent is initialized) and adjust the extent.  Ditto for the blocks between
+ * the end of the last full lcluster and end, if there's a right extent.
+ *
+ * Try to attach as much as we can to the left extent, then try to attach as
+ * much as we can to the right extent.  For the remainder, try to allocate the
+ * whole range; map in whatever we get; and repeat until we're done.
+ *
+ * To attach to a left extent, figure out the maximum amount we can add to the
+ * extent and try to allocate that much, and append if successful.  To attach
+ * to a right extent, figure out the max we can add to the extent, try to
+ * allocate that much, and prepend if successful.
+ *
+ * We need an alloc_range function that tells us how much we can allocate given
+ * a maximum length and one of a suggested start, a fixed start, or a fixed end
+ * point.
+ *
+ * Every time we modify the extent tree we also need to update the block stats.
+ *
+ * At the end, update i_blocks and i_size appropriately.
+ */
+
+static void dbg_print_extent(char *desc, struct ext2fs_extent *extent)
+{
+#ifdef DEBUG
+	if (desc)
+		printf("%s: ", desc);
+	printf("extent: lblk %llu--%llu, len %u, pblk %llu, flags: ",
+	       extent->e_lblk, extent->e_lblk + extent->e_len - 1,
+	       extent->e_len, extent->e_pblk);
+	if (extent->e_flags & EXT2_EXTENT_FLAGS_LEAF)
+		fputs("LEAF ", stdout);
+	if (extent->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+		fputs("UNINIT ", stdout);
+	if (extent->e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)
+		fputs("2ND_VISIT ", stdout);
+	if (!extent->e_flags)
+		fputs("(none)", stdout);
+	fputc('\n', stdout);
+	fflush(stdout);
+#endif
+}
+
+static errcode_t claim_range(ext2_filsys fs, struct ext2_inode *inode,
+			     blk64_t blk, blk64_t len)
+{
+	blk64_t	clusters;
+
+	clusters = (len + EXT2FS_CLUSTER_RATIO(fs) - 1) /
+		   EXT2FS_CLUSTER_RATIO(fs);
+	ext2fs_block_alloc_stats_range(fs, blk,
+			clusters * EXT2FS_CLUSTER_RATIO(fs), +1);
+	return ext2fs_iblk_add_blocks(fs, inode, clusters);
+}
+
+static errcode_t ext_falloc_helper(ext2_filsys fs,
+				   int flags,
+				   ext2_ino_t ino,
+				   struct ext2_inode *inode,
+				   ext2_extent_handle_t handle,
+				   struct ext2fs_extent *left_ext,
+				   struct ext2fs_extent *right_ext,
+				   blk64_t range_start, blk64_t range_len,
+				   blk64_t alloc_goal)
+{
+	struct ext2fs_extent	newex, ex;
+	int			op;
+	blk64_t			fillable, pblk, plen, x, y;
+	blk64_t			eof_blk = 0, cluster_fill = 0;
+	errcode_t		err;
+	blk_t			max_extent_len, max_uninit_len, max_init_len;
+
+#ifdef DEBUG
+	printf("%s: ", __func__);
+	if (left_ext)
+		printf("left_ext=%llu--%llu, ", left_ext->e_lblk,
+		       left_ext->e_lblk + left_ext->e_len - 1);
+	if (right_ext)
+		printf("right_ext=%llu--%llu, ", right_ext->e_lblk,
+		       right_ext->e_lblk + right_ext->e_len - 1);
+	printf("start=%llu len=%llu, goal=%llu\n", range_start, range_len,
+	       alloc_goal);
+	fflush(stdout);
+#endif
+	/* Can't create initialized extents past EOF? */
+	if (!(flags & EXT2_FALLOCATE_INIT_BEYOND_EOF))
+		eof_blk = EXT2_I_SIZE(inode) / fs->blocksize;
+
+	/* The allocation goal must be as far into a cluster as range_start. */
+	alloc_goal = (alloc_goal & ~EXT2FS_CLUSTER_MASK(fs)) |
+		     (range_start & EXT2FS_CLUSTER_MASK(fs));
+
+	max_uninit_len = EXT_UNINIT_MAX_LEN & ~EXT2FS_CLUSTER_MASK(fs);
+	max_init_len = EXT_INIT_MAX_LEN & ~EXT2FS_CLUSTER_MASK(fs);
+
+	/* We must lengthen the left extent to the end of the cluster */
+	if (left_ext && EXT2FS_CLUSTER_RATIO(fs) > 1) {
+		/* How many more blocks can be attached to left_ext? */
+		if (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+			fillable = max_uninit_len - left_ext->e_len;
+		else
+			fillable = max_init_len - left_ext->e_len;
+
+		if (fillable > range_len)
+			fillable = range_len;
+		if (fillable == 0)
+			goto expand_right;
+
+		/*
+		 * If range_start isn't on a cluster boundary, try an
+		 * implied cluster allocation for left_ext.
+		 */
+		cluster_fill = EXT2FS_CLUSTER_RATIO(fs) -
+			       (range_start & EXT2FS_CLUSTER_MASK(fs));
+		cluster_fill &= EXT2FS_CLUSTER_MASK(fs);
+		if (cluster_fill == 0)
+			goto expand_right;
+
+		if (cluster_fill > fillable)
+			cluster_fill = fillable;
+
+		/* Don't expand an initialized left_ext beyond EOF */
+		if (!(flags & EXT2_FALLOCATE_INIT_BEYOND_EOF)) {
+			x = left_ext->e_lblk + left_ext->e_len - 1;
+			dbg_printf("%s: lend=%llu newlend=%llu eofblk=%llu\n",
+				   __func__, x, x + cluster_fill, eof_blk);
+			if (eof_blk >= x && eof_blk <= x + cluster_fill)
+				cluster_fill = eof_blk - x;
+			if (cluster_fill == 0)
+				goto expand_right;
+		}
+
+		err = ext2fs_extent_goto(handle, left_ext->e_lblk);
+		if (err)
+			goto expand_right;
+		left_ext->e_len += cluster_fill;
+		range_start += cluster_fill;
+		range_len -= cluster_fill;
+		alloc_goal += cluster_fill;
+
+		dbg_print_extent("ext_falloc clus left+", left_ext);
+		err = ext2fs_extent_replace(handle, 0, left_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		/* Zero blocks */
+		if (!(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) {
+			err = ext2fs_zero_blocks2(fs, left_ext->e_pblk +
+						  left_ext->e_len -
+						  cluster_fill, cluster_fill,
+						  NULL, NULL);
+			if (err)
+				goto out;
+		}
+	}
+
+expand_right:
+	/* We must lengthen the right extent to the beginning of the cluster */
+	if (right_ext && EXT2FS_CLUSTER_RATIO(fs) > 1) {
+		/* How much can we attach to right_ext? */
+		if (right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+			fillable = max_uninit_len - right_ext->e_len;
+		else
+			fillable = max_init_len - right_ext->e_len;
+
+		if (fillable > range_len)
+			fillable = range_len;
+		if (fillable == 0)
+			goto try_merge;
+
+		/*
+		 * If range_end isn't on a cluster boundary, try an implied
+		 * cluster allocation for right_ext.
+		 */
+		cluster_fill = right_ext->e_lblk & EXT2FS_CLUSTER_MASK(fs);
+		if (cluster_fill == 0)
+			goto try_merge;
+
+		err = ext2fs_extent_goto(handle, right_ext->e_lblk);
+		if (err)
+			goto out;
+
+		if (cluster_fill > fillable)
+			cluster_fill = fillable;
+		right_ext->e_lblk -= cluster_fill;
+		right_ext->e_pblk -= cluster_fill;
+		right_ext->e_len += cluster_fill;
+		range_len -= cluster_fill;
+
+		dbg_print_extent("ext_falloc clus right+", right_ext);
+		err = ext2fs_extent_replace(handle, 0, right_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		/* Zero blocks if necessary */
+		if (!(right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) {
+			err = ext2fs_zero_blocks2(fs, right_ext->e_pblk,
+						  cluster_fill, NULL, NULL);
+			if (err)
+				goto out;
+		}
+	}
+
+try_merge:
+	/* Merge both extents together, perhaps? */
+	if (left_ext && right_ext) {
+		/* Are the two extents mergeable? */
+		if ((left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) !=
+		    (right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT))
+			goto try_left;
+
+		/* User requires init/uninit but extent is uninit/init. */
+		if (((flags & EXT2_FALLOCATE_FORCE_INIT) &&
+		     (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) ||
+		    ((flags & EXT2_FALLOCATE_FORCE_UNINIT) &&
+		     !(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)))
+			goto try_left;
+
+		/*
+		 * Skip initialized extent unless user wants to zero blocks
+		 * or requires init extent.
+		 */
+		if (!(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (!(flags & EXT2_FALLOCATE_ZERO_BLOCKS) ||
+		     !(flags & EXT2_FALLOCATE_FORCE_INIT)))
+			goto try_left;
+
+		/* Will it even fit? */
+		x = left_ext->e_len + range_len + right_ext->e_len;
+		if (x > (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT ?
+				max_uninit_len : max_init_len))
+			goto try_left;
+
+		err = ext2fs_extent_goto(handle, left_ext->e_lblk);
+		if (err)
+			goto try_left;
+
+		/* Allocate blocks */
+		y = left_ext->e_pblk + left_ext->e_len;
+		err = ext2fs_new_range(fs, EXT2_NEWRANGE_FIXED_GOAL |
+				       EXT2_NEWRANGE_MIN_LENGTH, y,
+				       right_ext->e_pblk - y + 1, NULL,
+				       &pblk, &plen);
+		if (err)
+			goto try_left;
+		if (pblk + plen != right_ext->e_pblk)
+			goto try_left;
+		err = claim_range(fs, inode, pblk, plen);
+		if (err)
+			goto out;
+
+		/* Modify extents */
+		left_ext->e_len = x;
+		dbg_print_extent("ext_falloc merge", left_ext);
+		err = ext2fs_extent_replace(handle, 0, left_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT_LEAF, &newex);
+		if (err)
+			goto out;
+		err = ext2fs_extent_delete(handle, 0);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+		*right_ext = *left_ext;
+
+		/* Zero blocks */
+		if (!(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, range_start, range_len,
+						  NULL, NULL);
+			if (err)
+				goto out;
+		}
+
+		return 0;
+	}
+
+try_left:
+	/* Extend the left extent */
+	if (left_ext) {
+		/* How many more blocks can be attached to left_ext? */
+		if (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+			fillable = max_uninit_len - left_ext->e_len;
+		else if (flags & EXT2_FALLOCATE_ZERO_BLOCKS)
+			fillable = max_init_len - left_ext->e_len;
+		else
+			fillable = 0;
+
+		/* User requires init/uninit but extent is uninit/init. */
+		if (((flags & EXT2_FALLOCATE_FORCE_INIT) &&
+		     (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) ||
+		    ((flags & EXT2_FALLOCATE_FORCE_UNINIT) &&
+		     !(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)))
+			goto try_right;
+
+		if (fillable > range_len)
+			fillable = range_len;
+
+		/* Don't expand an initialized left_ext beyond EOF */
+		x = left_ext->e_lblk + left_ext->e_len - 1;
+		if (!(flags & EXT2_FALLOCATE_INIT_BEYOND_EOF)) {
+			dbg_printf("%s: lend=%llu newlend=%llu eofblk=%llu\n",
+				   __func__, x, x + fillable, eof_blk);
+			if (eof_blk >= x && eof_blk <= x + fillable)
+				fillable = eof_blk - x;
+		}
+
+		if (fillable == 0)
+			goto try_right;
+
+		/* Test if the right edge of the range is already mapped? */
+		if (EXT2FS_CLUSTER_RATIO(fs) > 1) {
+			err = ext2fs_map_cluster_block(fs, ino, inode,
+					x + fillable, &pblk);
+			if (err)
+				goto out;
+			if (pblk)
+				fillable -= 1 + ((x + fillable)
+						 & EXT2FS_CLUSTER_MASK(fs));
+			if (fillable == 0)
+				goto try_right;
+		}
+
+		/* Allocate range of blocks */
+		x = left_ext->e_pblk + left_ext->e_len;
+		err = ext2fs_new_range(fs, EXT2_NEWRANGE_FIXED_GOAL |
+				EXT2_NEWRANGE_MIN_LENGTH,
+				x, fillable, NULL, &pblk, &plen);
+		if (err)
+			goto try_right;
+		err = claim_range(fs, inode, pblk, plen);
+		if (err)
+			goto out;
+
+		/* Modify left_ext */
+		err = ext2fs_extent_goto(handle, left_ext->e_lblk);
+		if (err)
+			goto out;
+		range_start += plen;
+		range_len -= plen;
+		left_ext->e_len += plen;
+		dbg_print_extent("ext_falloc left+", left_ext);
+		err = ext2fs_extent_replace(handle, 0, left_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		/* Zero blocks if necessary */
+		if (!(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, pblk, plen, NULL, NULL);
+			if (err)
+				goto out;
+		}
+	}
+
+try_right:
+	/* Extend the right extent */
+	if (right_ext) {
+		/* How much can we attach to right_ext? */
+		if (right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+			fillable = max_uninit_len - right_ext->e_len;
+		else if (flags & EXT2_FALLOCATE_ZERO_BLOCKS)
+			fillable = max_init_len - right_ext->e_len;
+		else
+			fillable = 0;
+
+		/* User requires init/uninit but extent is uninit/init. */
+		if (((flags & EXT2_FALLOCATE_FORCE_INIT) &&
+		     (right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) ||
+		    ((flags & EXT2_FALLOCATE_FORCE_UNINIT) &&
+		     !(right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)))
+			goto try_anywhere;
+
+		if (fillable > range_len)
+			fillable = range_len;
+		if (fillable == 0)
+			goto try_anywhere;
+
+		/* Test if the left edge of the range is already mapped? */
+		if (EXT2FS_CLUSTER_RATIO(fs) > 1) {
+			err = ext2fs_map_cluster_block(fs, ino, inode,
+					right_ext->e_lblk - fillable, &pblk);
+			if (err)
+				goto out;
+			if (pblk)
+				fillable -= EXT2FS_CLUSTER_RATIO(fs) -
+						((right_ext->e_lblk - fillable)
+						 & EXT2FS_CLUSTER_MASK(fs));
+			if (fillable == 0)
+				goto try_anywhere;
+		}
+
+		/*
+		 * FIXME: It would be nice if we could handle allocating a
+		 * variable range from a fixed end point instead of just
+		 * skipping to the general allocator if the whole range is
+		 * unavailable.
+		 */
+		err = ext2fs_new_range(fs, EXT2_NEWRANGE_FIXED_GOAL |
+				EXT2_NEWRANGE_MIN_LENGTH,
+				right_ext->e_pblk - fillable,
+				fillable, NULL, &pblk, &plen);
+		if (err)
+			goto try_anywhere;
+		err = claim_range(fs, inode,
+			      pblk & ~EXT2FS_CLUSTER_MASK(fs),
+			      plen + (pblk & EXT2FS_CLUSTER_MASK(fs)));
+		if (err)
+			goto out;
+
+		/* Modify right_ext */
+		err = ext2fs_extent_goto(handle, right_ext->e_lblk);
+		if (err)
+			goto out;
+		range_len -= plen;
+		right_ext->e_lblk -= plen;
+		right_ext->e_pblk -= plen;
+		right_ext->e_len += plen;
+		dbg_print_extent("ext_falloc right+", right_ext);
+		err = ext2fs_extent_replace(handle, 0, right_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		/* Zero blocks if necessary */
+		if (!(right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, pblk,
+					plen + cluster_fill, NULL, NULL);
+			if (err)
+				goto out;
+		}
+	}
+
+try_anywhere:
+	/* Try implied cluster alloc on the left and right ends */
+	if (range_len > 0 && (range_start & EXT2FS_CLUSTER_MASK(fs))) {
+		cluster_fill = EXT2FS_CLUSTER_RATIO(fs) -
+			       (range_start & EXT2FS_CLUSTER_MASK(fs));
+		cluster_fill &= EXT2FS_CLUSTER_MASK(fs);
+		if (cluster_fill > range_len)
+			cluster_fill = range_len;
+		newex.e_lblk = range_start;
+		err = ext2fs_map_cluster_block(fs, ino, inode, newex.e_lblk,
+					       &pblk);
+		if (err)
+			goto out;
+		if (pblk == 0)
+			goto try_right_implied;
+		newex.e_pblk = pblk;
+		newex.e_len = cluster_fill;
+		newex.e_flags = (flags & EXT2_FALLOCATE_FORCE_INIT ? 0 :
+				 EXT2_EXTENT_FLAGS_UNINIT);
+		dbg_print_extent("ext_falloc iclus left+", &newex);
+		ext2fs_extent_goto(handle, newex.e_lblk);
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_CURRENT,
+					&ex);
+		if (err == EXT2_ET_NO_CURRENT_NODE)
+			ex.e_lblk = 0;
+		else if (err)
+			goto out;
+
+		if (ex.e_lblk > newex.e_lblk)
+			op = 0; /* insert before */
+		else
+			op = EXT2_EXTENT_INSERT_AFTER;
+		dbg_printf("%s: inserting %s lblk %llu newex=%llu\n",
+			   __func__, op ? "after" : "before", ex.e_lblk,
+			   newex.e_lblk);
+		err = ext2fs_extent_insert(handle, op, &newex);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		if (!(newex.e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, newex.e_pblk,
+						  newex.e_len, NULL, NULL);
+			if (err)
+				goto out;
+		}
+
+		range_start += cluster_fill;
+		range_len -= cluster_fill;
+	}
+
+try_right_implied:
+	y = range_start + range_len;
+	if (range_len > 0 && (y & EXT2FS_CLUSTER_MASK(fs))) {
+		cluster_fill = y & EXT2FS_CLUSTER_MASK(fs);
+		if (cluster_fill > range_len)
+			cluster_fill = range_len;
+		newex.e_lblk = y & ~EXT2FS_CLUSTER_MASK(fs);
+		err = ext2fs_map_cluster_block(fs, ino, inode, newex.e_lblk,
+					       &pblk);
+		if (err)
+			goto out;
+		if (pblk == 0)
+			goto no_implied;
+		newex.e_pblk = pblk;
+		newex.e_len = cluster_fill;
+		newex.e_flags = (flags & EXT2_FALLOCATE_FORCE_INIT ? 0 :
+				 EXT2_EXTENT_FLAGS_UNINIT);
+		dbg_print_extent("ext_falloc iclus right+", &newex);
+		ext2fs_extent_goto(handle, newex.e_lblk);
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_CURRENT,
+					&ex);
+		if (err == EXT2_ET_NO_CURRENT_NODE)
+			ex.e_lblk = 0;
+		else if (err)
+			goto out;
+
+		if (ex.e_lblk > newex.e_lblk)
+			op = 0; /* insert before */
+		else
+			op = EXT2_EXTENT_INSERT_AFTER;
+		dbg_printf("%s: inserting %s lblk %llu newex=%llu\n",
+			   __func__, op ? "after" : "before", ex.e_lblk,
+			   newex.e_lblk);
+		err = ext2fs_extent_insert(handle, op, &newex);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		if (!(newex.e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, newex.e_pblk,
+						  newex.e_len, NULL, NULL);
+			if (err)
+				goto out;
+		}
+
+		range_len -= cluster_fill;
+	}
+
+no_implied:
+	if (range_len == 0)
+		return 0;
+
+	newex.e_lblk = range_start;
+	if (flags & EXT2_FALLOCATE_FORCE_INIT) {
+		max_extent_len = max_init_len;
+		newex.e_flags = 0;
+	} else {
+		max_extent_len = max_uninit_len;
+		newex.e_flags = EXT2_EXTENT_FLAGS_UNINIT;
+	}
+	pblk = alloc_goal;
+	y = range_len;
+	for (x = 0; x < y;) {
+		cluster_fill = newex.e_lblk & EXT2FS_CLUSTER_MASK(fs);
+		fillable = min(range_len + cluster_fill, max_extent_len);
+		err = ext2fs_new_range(fs, 0, pblk & ~EXT2FS_CLUSTER_MASK(fs),
+				       fillable,
+				       NULL, &pblk, &plen);
+		if (err)
+			goto out;
+		err = claim_range(fs, inode, pblk, plen);
+		if (err)
+			goto out;
+
+		/* Create extent */
+		newex.e_pblk = pblk + cluster_fill;
+		newex.e_len = plen - cluster_fill;
+		dbg_print_extent("ext_falloc create", &newex);
+		ext2fs_extent_goto(handle, newex.e_lblk);
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_CURRENT,
+					&ex);
+		if (err == EXT2_ET_NO_CURRENT_NODE)
+			ex.e_lblk = 0;
+		else if (err)
+			goto out;
+
+		if (ex.e_lblk > newex.e_lblk)
+			op = 0; /* insert before */
+		else
+			op = EXT2_EXTENT_INSERT_AFTER;
+		dbg_printf("%s: inserting %s lblk %llu newex=%llu\n",
+			   __func__, op ? "after" : "before", ex.e_lblk,
+			   newex.e_lblk);
+		err = ext2fs_extent_insert(handle, op, &newex);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		if (!(newex.e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, pblk, plen, NULL, NULL);
+			if (err)
+				goto out;
+		}
+
+		/* Update variables at end of loop */
+		x += plen - cluster_fill;
+		range_len -= plen - cluster_fill;
+		newex.e_lblk += plen - cluster_fill;
+		pblk += plen - cluster_fill;
+		if (pblk >= ext2fs_blocks_count(fs->super))
+			pblk = fs->super->s_first_data_block;
+	}
+
+out:
+	return err;
+}
+
+static errcode_t extent_fallocate(ext2_filsys fs, int flags, ext2_ino_t ino,
+				      struct ext2_inode *inode, blk64_t goal,
+				      blk64_t start, blk64_t len)
+{
+	ext2_extent_handle_t	handle;
+	struct ext2fs_extent	left_extent, right_extent;
+	struct ext2fs_extent	*left_adjacent, *right_adjacent;
+	errcode_t		err;
+	blk64_t			range_start, range_end = 0, end, next;
+	blk64_t			count, goal_distance;
+
+	end = start + len - 1;
+	err = ext2fs_extent_open2(fs, ino, inode, &handle);
+	if (err)
+		return err;
+
+	/*
+	 * Find the extent closest to the start of the alloc range.  We don't
+	 * check the return value because _goto() sets the current node to the
+	 * next-lowest extent if 'start' is in a hole; or the next-highest
+	 * extent if there aren't any lower ones; or doesn't set a current node
+	 * if there was a real error reading the extent tree.  In that case,
+	 * _get() will error out.
+	 */
+start_again:
+	ext2fs_extent_goto(handle, start);
+	err = ext2fs_extent_get(handle, EXT2_EXTENT_CURRENT, &left_extent);
+	if (err == EXT2_ET_NO_CURRENT_NODE) {
+		blk64_t max_blocks = ext2fs_blocks_count(fs->super);
+
+		if (goal == ~0ULL)
+			goal = ext2fs_find_inode_goal(fs, ino, inode, start);
+		err = ext2fs_find_first_zero_block_bitmap2(fs->block_map,
+						goal, max_blocks - 1, &goal);
+		goal += start;
+		err = ext_falloc_helper(fs, flags, ino, inode, handle, NULL,
+					NULL, start, len, goal);
+		goto errout;
+	} else if (err)
+		goto errout;
+
+	dbg_print_extent("ext_falloc initial", &left_extent);
+	next = left_extent.e_lblk + left_extent.e_len;
+	if (left_extent.e_lblk > start) {
+		/* The nearest extent we found was beyond start??? */
+		goal = left_extent.e_pblk - (left_extent.e_lblk - start);
+		err = ext_falloc_helper(fs, flags, ino, inode, handle, NULL,
+					&left_extent, start,
+					left_extent.e_lblk - start, goal);
+		if (err)
+			goto errout;
+
+		goto start_again;
+	} else if (next >= start) {
+		range_start = next;
+		left_adjacent = &left_extent;
+	} else {
+		range_start = start;
+		left_adjacent = NULL;
+	}
+	goal = left_extent.e_pblk + (range_start - left_extent.e_lblk);
+	goal_distance = range_start - next;
+
+	do {
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT_LEAF,
+					   &right_extent);
+		dbg_printf("%s: ino=%d get next =%d\n", __func__, ino,
+			   (int)err);
+		dbg_print_extent("ext_falloc next", &right_extent);
+		/* Stop if we've seen this extent before */
+		if (!err && right_extent.e_lblk <= left_extent.e_lblk)
+			err = EXT2_ET_EXTENT_NO_NEXT;
+
+		if (err && err != EXT2_ET_EXTENT_NO_NEXT)
+			goto errout;
+		if (err == EXT2_ET_EXTENT_NO_NEXT ||
+		    right_extent.e_lblk > end + 1) {
+			range_end = end;
+			right_adjacent = NULL;
+		} else {
+			/* Handle right_extent.e_lblk <= end */
+			range_end = right_extent.e_lblk - 1;
+			right_adjacent = &right_extent;
+		}
+		if (err != EXT2_ET_EXTENT_NO_NEXT &&
+		    goal_distance > (range_end - right_extent.e_lblk)) {
+			goal = right_extent.e_pblk -
+					(right_extent.e_lblk - range_start);
+			goal_distance = range_end - right_extent.e_lblk;
+		}
+
+		dbg_printf("%s: ino=%d rstart=%llu rend=%llu\n", __func__, ino,
+			   range_start, range_end);
+		err = 0;
+		if (range_start <= range_end) {
+			count = range_end - range_start + 1;
+			err = ext_falloc_helper(fs, flags, ino, inode, handle,
+						left_adjacent, right_adjacent,
+						range_start, count, goal);
+			if (err)
+				goto errout;
+		}
+
+		if (range_end == end)
+			break;
+
+		err = ext2fs_extent_goto(handle, right_extent.e_lblk);
+		if (err)
+			goto errout;
+		next = right_extent.e_lblk + right_extent.e_len;
+		left_extent = right_extent;
+		left_adjacent = &left_extent;
+		range_start = next;
+		goal = left_extent.e_pblk + (range_start - left_extent.e_lblk);
+		goal_distance = range_start - next;
+	} while (range_end < end);
+
+errout:
+	ext2fs_extent_free(handle);
+	return err;
+}
+
+/*
+ * Map physical blocks to a range of logical blocks within a file.  The range
+ * of logical blocks are (start, start + len).  If there are already extents,
+ * the mappings will try to extend the mappings; otherwise, it will try to map
+ * start as if logical block 0 points to goal.  If goal is ~0ULL, then the goal
+ * is calculated based on the inode group.
+ *
+ * Flags:
+ * - EXT2_FALLOCATE_ZERO_BLOCKS: Zero the blocks that are allocated.
+ * - EXT2_FALLOCATE_FORCE_INIT: Create only initialized extents.
+ * - EXT2_FALLOCATE_FORCE_UNINIT: Create only uninitialized extents.
+ * - EXT2_FALLOCATE_INIT_BEYOND_EOF: Create extents beyond EOF.
+ *
+ * If neither FORCE_INIT nor FORCE_UNINIT are specified, this function will
+ * try to expand any extents it finds, zeroing blocks as necessary.
+ */
+errcode_t ext2fs_fallocate(ext2_filsys fs, int flags, ext2_ino_t ino,
+			   struct ext2_inode *inode, blk64_t goal,
+			   blk64_t start, blk64_t len)
+{
+	struct ext2_inode	inode_buf;
+	blk64_t			blk, x;
+	errcode_t		err;
+
+	if (((flags & EXT2_FALLOCATE_FORCE_INIT) &&
+	    (flags & EXT2_FALLOCATE_FORCE_UNINIT)) ||
+	   (flags & ~EXT2_FALLOCATE_ALL_FLAGS))
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (len > ext2fs_blocks_count(fs->super))
+		return EXT2_ET_BLOCK_ALLOC_FAIL;
+	else if (len == 0)
+		return 0;
+
+	/* Read inode structure if necessary */
+	if (!inode) {
+		err = ext2fs_read_inode(fs, ino, &inode_buf);
+		if (err)
+			return err;
+		inode = &inode_buf;
+	}
+	dbg_printf("%s: ino=%d start=%llu len=%llu goal=%llu\n", __func__, ino,
+		   start, len, goal);
+
+	if (inode->i_flags & EXT4_EXTENTS_FL) {
+		err = extent_fallocate(fs, flags, ino, inode, goal, start, len);
+		goto out;
+	}
+
+	/* XXX: Allocate a bunch of blocks the slow way */
+	for (blk = start; blk < start + len; blk++) {
+		err = ext2fs_bmap2(fs, ino, inode, NULL, 0, blk, 0, &x);
+		if (err)
+			return err;
+		if (x)
+			continue;
+
+		err = ext2fs_bmap2(fs, ino, inode, NULL,
+				   BMAP_ALLOC | BMAP_UNINIT | BMAP_ZERO, blk,
+				   0, &x);
+		if (err)
+			return err;
+	}
+
+out:
+	if (inode == &inode_buf)
+		ext2fs_write_inode(fs, ino, inode);
+	return err;
+}

[PATCH 51/54] libext2fs: use fallocate for creating journals and hugefiles

[Darrick J. Wong]

Use the new fallocate API for creating the journal and the mk_hugefile
feature.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/mkjournal.c                |  134 +++++----------------------------
 misc/mk_hugefiles.c                   |   96 ++----------------------
 tests/f_opt_extent/expect             |   15 ----
 tests/r_32to64bit_meta/expect         |    4 -
 tests/r_32to64bit_move_itable/expect  |    4 -
 tests/r_64to32bit/expect              |    4 -
 tests/r_64to32bit_meta/expect         |    4 -
 tests/t_disable_mcsum_noinitbg/expect |    6 +
 tests/t_enable_mcsum/expect           |   15 ----
 tests/t_enable_mcsum_ext3/expect      |    8 +-
 tests/t_enable_mcsum_initbg/expect    |   11 +--
 tests/t_iexpand_full/expect           |    4 -
 12 files changed, 56 insertions(+), 249 deletions(-)


diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c
index c42cb98..02a65cb 100644
--- a/lib/ext2fs/mkjournal.c
+++ b/lib/ext2fs/mkjournal.c
@@ -227,89 +227,6 @@ errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
 }
 
 /*
- * Helper function for creating the journal using direct I/O routines
- */
-struct mkjournal_struct {
-	int		num_blocks;
-	int		newblocks;
-	blk64_t		goal;
-	blk64_t		blk_to_zero;
-	int		zero_count;
-	int		flags;
-	char		*buf;
-	errcode_t	err;
-};
-
-static int mkjournal_proc(ext2_filsys	fs,
-			  blk64_t	*blocknr,
-			  e2_blkcnt_t	blockcnt,
-			  blk64_t	ref_block EXT2FS_ATTR((unused)),
-			  int		ref_offset EXT2FS_ATTR((unused)),
-			  void		*priv_data)
-{
-	struct mkjournal_struct *es = (struct mkjournal_struct *) priv_data;
-	blk64_t	new_blk;
-	errcode_t	retval;
-
-	if (*blocknr) {
-		es->goal = *blocknr;
-		return 0;
-	}
-	if (blockcnt &&
-	    (EXT2FS_B2C(fs, es->goal) == EXT2FS_B2C(fs, es->goal+1)))
-		new_blk = es->goal+1;
-	else {
-		es->goal &= ~EXT2FS_CLUSTER_MASK(fs);
-		retval = ext2fs_new_block2(fs, es->goal, 0, &new_blk);
-		if (retval) {
-			es->err = retval;
-			return BLOCK_ABORT;
-		}
-		ext2fs_block_alloc_stats2(fs, new_blk, +1);
-		es->newblocks++;
-	}
-	if (blockcnt >= 0)
-		es->num_blocks--;
-
-	retval = 0;
-	if (blockcnt <= 0)
-		retval = io_channel_write_blk64(fs->io, new_blk, 1, es->buf);
-	else if (!(es->flags & EXT2_MKJOURNAL_LAZYINIT)) {
-		if (es->zero_count) {
-			if ((es->blk_to_zero + es->zero_count == new_blk) &&
-			    (es->zero_count < 1024))
-				es->zero_count++;
-			else {
-				retval = ext2fs_zero_blocks2(fs,
-							     es->blk_to_zero,
-							     es->zero_count,
-							     0, 0);
-				es->zero_count = 0;
-			}
-		}
-		if (es->zero_count == 0) {
-			es->blk_to_zero = new_blk;
-			es->zero_count = 1;
-		}
-	}
-
-	if (blockcnt == 0)
-		memset(es->buf, 0, fs->blocksize);
-
-	if (retval) {
-		es->err = retval;
-		return BLOCK_ABORT;
-	}
-	*blocknr = es->goal = new_blk;
-
-	if (es->num_blocks == 0)
-		return (BLOCK_CHANGED | BLOCK_ABORT);
-	else
-		return BLOCK_CHANGED;
-
-}
-
-/*
  * Calculate the initial goal block to be roughly at the middle of the
  * filesystem.  Pick a group that has the largest number of free
  * blocks.
@@ -350,7 +267,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	errcode_t		retval;
 	struct ext2_inode	inode;
 	unsigned long long	inode_size;
-	struct mkjournal_struct	es;
+	int			falloc_flags = EXT2_FALLOCATE_FORCE_INIT;
+	blk64_t			zblk;
 
 	if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags,
 						       &buf)))
@@ -367,40 +285,16 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 		goto out2;
 	}
 
-	es.num_blocks = num_blocks;
-	es.newblocks = 0;
-	es.buf = buf;
-	es.err = 0;
-	es.flags = flags;
-	es.zero_count = 0;
-	es.goal = (goal != ~0ULL) ? goal : get_midpoint_journal_block(fs);
+	if (goal == ~0ULL)
+		goal = get_midpoint_journal_block(fs);
 
-	if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS) {
+	if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS)
 		inode.i_flags |= EXT4_EXTENTS_FL;
-		if ((retval = ext2fs_write_inode(fs, journal_ino, &inode)))
-			goto out2;
-	}
 
-	retval = ext2fs_block_iterate3(fs, journal_ino, BLOCK_FLAG_APPEND,
-				       0, mkjournal_proc, &es);
-	if (retval)
-		goto out2;
-	if (es.err) {
-		retval = es.err;
-		goto out2;
-	}
-	if (es.zero_count) {
-		retval = ext2fs_zero_blocks2(fs, es.blk_to_zero,
-					    es.zero_count, 0, 0);
-		if (retval)
-			goto out2;
-	}
-
-	if ((retval = ext2fs_read_inode(fs, journal_ino, &inode)))
-		goto out2;
+	if (!(flags & EXT2_MKJOURNAL_LAZYINIT))
+		falloc_flags |= EXT2_FALLOCATE_ZERO_BLOCKS;
 
 	inode_size = (unsigned long long)fs->blocksize * num_blocks;
-	ext2fs_iblk_add_blocks(fs, &inode, es.newblocks);
 	inode.i_mtime = inode.i_ctime = fs->now ? fs->now : time(0);
 	inode.i_links_count = 1;
 	inode.i_mode = LINUX_S_IFREG | 0600;
@@ -408,9 +302,21 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	if (retval)
 		goto out2;
 
+	retval = ext2fs_fallocate(fs, falloc_flags, journal_ino,
+				  &inode, goal, 0, num_blocks);
+	if (retval)
+		goto out2;
+
 	if ((retval = ext2fs_write_new_inode(fs, journal_ino, &inode)))
 		goto out2;
-	retval = 0;
+
+	retval = ext2fs_bmap2(fs, journal_ino, &inode, NULL, 0, 0, NULL, &zblk);
+	if (retval)
+		goto out2;
+
+	retval = io_channel_write_blk64(fs->io, zblk, 1, buf);
+	if (retval)
+		goto out2;
 
 	memcpy(fs->super->s_jnl_blocks, inode.i_block, EXT2_N_BLOCKS*4);
 	fs->super->s_jnl_blocks[15] = inode.i_size_high;
diff --git a/misc/mk_hugefiles.c b/misc/mk_hugefiles.c
index e42c0b9..0978d55 100644
--- a/misc/mk_hugefiles.c
+++ b/misc/mk_hugefiles.c
@@ -258,12 +258,7 @@ static errcode_t mk_hugefile(ext2_filsys fs, blk64_t num,
 
 {
 	errcode_t		retval;
-	blk64_t			lblk, bend = 0;
-	__u64			size;
-	blk64_t			left;
-	blk64_t			count = 0;
 	struct ext2_inode	inode;
-	ext2_extent_handle_t	handle;
 
 	retval = ext2fs_new_inode(fs, 0, LINUX_S_IFREG, NULL, ino);
 	if (retval)
@@ -283,85 +278,20 @@ static errcode_t mk_hugefile(ext2_filsys fs, blk64_t num,
 
 	ext2fs_inode_alloc_stats2(fs, *ino, +1, 0);
 
-	retval = ext2fs_extent_open2(fs, *ino, &inode, &handle);
+	if (EXT2_HAS_INCOMPAT_FEATURE(fs->super,
+				      EXT3_FEATURE_INCOMPAT_EXTENTS))
+		inode.i_flags |= EXT4_EXTENTS_FL;
+	retval = ext2fs_fallocate(fs,
+				  EXT2_FALLOCATE_FORCE_INIT |
+				  EXT2_FALLOCATE_ZERO_BLOCKS,
+				  *ino, &inode, ~0ULL, 0, num);
 	if (retval)
 		return retval;
-
-	lblk = 0;
-	left = num ? num : 1;
-	while (left) {
-		blk64_t pblk, end;
-		blk64_t n = left;
-
-		retval =  ext2fs_find_first_zero_block_bitmap2(fs->block_map,
-			goal, ext2fs_blocks_count(fs->super) - 1, &end);
-		if (retval)
-			goto errout;
-		goal = end;
-
-		retval =  ext2fs_find_first_set_block_bitmap2(fs->block_map, goal,
-			       ext2fs_blocks_count(fs->super) - 1, &bend);
-		if (retval == ENOENT) {
-			bend = ext2fs_blocks_count(fs->super);
-			if (num == 0)
-				left = 0;
-		}
-		if (!num || bend - goal < left)
-			n = bend - goal;
-		pblk = goal;
-		if (num)
-			left -= n;
-		goal += n;
-		count += n;
-		ext2fs_block_alloc_stats_range(fs, pblk, n, +1);
-
-		if (zero_hugefile) {
-			blk64_t ret_blk;
-			retval = ext2fs_zero_blocks2(fs, pblk, n,
-						     &ret_blk, NULL);
-
-			if (retval)
-				com_err(program_name, retval,
-					_("while zeroing block %llu "
-					  "for hugefile"), ret_blk);
-		}
-
-		while (n) {
-			blk64_t l = n;
-			struct ext2fs_extent newextent;
-
-			if (l > EXT_INIT_MAX_LEN)
-				l = EXT_INIT_MAX_LEN;
-
-			newextent.e_len = l;
-			newextent.e_pblk = pblk;
-			newextent.e_lblk = lblk;
-			newextent.e_flags = 0;
-
-			retval = ext2fs_extent_insert(handle,
-					EXT2_EXTENT_INSERT_AFTER, &newextent);
-			if (retval)
-				return retval;
-			pblk += l;
-			lblk += l;
-			n -= l;
-		}
-	}
-
-	retval = ext2fs_read_inode(fs, *ino, &inode);
-	if (retval)
-		goto errout;
-
-	retval = ext2fs_iblk_add_blocks(fs, &inode,
-					count / EXT2FS_CLUSTER_RATIO(fs));
-	if (retval)
-		goto errout;
-	size = (__u64) count * fs->blocksize;
-	retval = ext2fs_inode_size_set(fs, &inode, size);
+	retval = ext2fs_inode_size_set(fs, &inode, num * fs->blocksize);
 	if (retval)
-		goto errout;
+		return retval;
 
-	retval = ext2fs_write_new_inode(fs, *ino, &inode);
+	retval = ext2fs_write_inode(fs, *ino, &inode);
 	if (retval)
 		goto errout;
 
@@ -379,13 +309,7 @@ retry:
 		goto retry;
 	}
 
-	if (retval)
-		goto errout;
-
 errout:
-	if (handle)
-		ext2fs_extent_free(handle);
-
 	return retval;
 }
 
diff --git a/tests/f_opt_extent/expect b/tests/f_opt_extent/expect
index 6d4863b..f4ed7ff 100644
--- a/tests/f_opt_extent/expect
+++ b/tests/f_opt_extent/expect
@@ -30,22 +30,11 @@ Change in FS metadata:
  Inode count:              65536
  Block count:              524288
  Reserved block count:     26214
--Free blocks:              570
-+Free blocks:              567
+-Free blocks:              569
++Free blocks:              566
  Free inodes:              65047
  First block:              1
  Block size:               1024
-@@ -47,8 +47,8 @@
-   Block bitmap at 262 (+261)
-   Inode bitmap at 278 (+277)
-   Inode table at 294-549 (+293)
--  21 free blocks, 535 free inodes, 3 directories, 535 unused inodes
--  Free blocks: 4414-4434
-+  18 free blocks, 535 free inodes, 3 directories, 535 unused inodes
-+  Free blocks: 4417-4434
-   Free inodes: 490-1024
- Group 1: (Blocks 8193-16384) [INODE_UNINIT]
-   Backup superblock at 8193, Group descriptors at 8194-8197
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
diff --git a/tests/r_32to64bit_meta/expect b/tests/r_32to64bit_meta/expect
index 0eacd45..8796503 100644
--- a/tests/r_32to64bit_meta/expect
+++ b/tests/r_32to64bit_meta/expect
@@ -35,8 +35,8 @@ Change in FS metadata:
  Inode count:              65536
  Block count:              524288
  Reserved block count:     26214
--Free blocks:              858
-+Free blocks:              852
+-Free blocks:              857
++Free blocks:              851
  Free inodes:              65046
  First block:              1
  Block size:               1024
diff --git a/tests/r_32to64bit_move_itable/expect b/tests/r_32to64bit_move_itable/expect
index b51663d..999bb8d 100644
--- a/tests/r_32to64bit_move_itable/expect
+++ b/tests/r_32to64bit_move_itable/expect
@@ -35,8 +35,8 @@ Change in FS metadata:
  Inode count:              98304
  Block count:              786432
  Reserved block count:     39321
--Free blocks:              764
-+Free blocks:              734
+-Free blocks:              763
++Free blocks:              733
  Free inodes:              97566
  First block:              1
  Block size:               1024
diff --git a/tests/r_64to32bit/expect b/tests/r_64to32bit/expect
index 13e94a2..5d2ea4b 100644
--- a/tests/r_64to32bit/expect
+++ b/tests/r_64to32bit/expect
@@ -35,8 +35,8 @@ Change in FS metadata:
  Inode count:              65536
  Block count:              524288
  Reserved block count:     26214
--Free blocks:              571
-+Free blocks:              589
+-Free blocks:              570
++Free blocks:              588
  Free inodes:              65048
  First block:              1
  Block size:               1024
diff --git a/tests/r_64to32bit_meta/expect b/tests/r_64to32bit_meta/expect
index d6e2dcc..1400c6b 100644
--- a/tests/r_64to32bit_meta/expect
+++ b/tests/r_64to32bit_meta/expect
@@ -35,8 +35,8 @@ Change in FS metadata:
  Inode count:              65536
  Block count:              524288
  Reserved block count:     26214
--Free blocks:              852
-+Free blocks:              858
+-Free blocks:              851
++Free blocks:              857
  Free inodes:              65046
  First block:              1
  Block size:               1024
diff --git a/tests/t_disable_mcsum_noinitbg/expect b/tests/t_disable_mcsum_noinitbg/expect
index a022631..09e4ff1 100644
--- a/tests/t_disable_mcsum_noinitbg/expect
+++ b/tests/t_disable_mcsum_noinitbg/expect
@@ -40,9 +40,9 @@ Change in FS metadata:
    Block bitmap at 262 (+261)
    Inode bitmap at 278 (+277)
    Inode table at 294-549 (+293)
--  21 free blocks, 536 free inodes, 2 directories, 536 unused inodes
-+  21 free blocks, 536 free inodes, 2 directories
-   Free blocks: 4413-4433
+-  0 free blocks, 536 free inodes, 2 directories, 536 unused inodes
++  0 free blocks, 536 free inodes, 2 directories
+   Free blocks: 
    Free inodes: 489-1024
 -Group 1: (Blocks 8193-16384) [INODE_UNINIT]
 +Group 1: (Blocks 8193-16384)
diff --git a/tests/t_enable_mcsum/expect b/tests/t_enable_mcsum/expect
index 2ee3c27..81e1125 100644
--- a/tests/t_enable_mcsum/expect
+++ b/tests/t_enable_mcsum/expect
@@ -45,8 +45,8 @@ Change in FS metadata:
  Inode count:              65536
  Block count:              524288
  Reserved block count:     26214
--Free blocks:              571
-+Free blocks:              568
+-Free blocks:              570
++Free blocks:              567
  Free inodes:              65048
  First block:              1
  Block size:               1024
@@ -58,17 +58,6 @@ Change in FS metadata:
  Journal features:         (none)
  Journal size:             16M
  Journal length:           16384
-@@ -46,8 +47,8 @@
-   Block bitmap at 262 (+261)
-   Inode bitmap at 278 (+277)
-   Inode table at 294-549 (+293)
--  21 free blocks, 536 free inodes, 2 directories, 536 unused inodes
--  Free blocks: 4413-4433
-+  18 free blocks, 536 free inodes, 2 directories, 536 unused inodes
-+  Free blocks: 4413, 4417-4433
-   Free inodes: 489-1024
- Group 1: (Blocks 8193-16384) [INODE_UNINIT]
-   Backup superblock at 8193, Group descriptors at 8194-8197
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
diff --git a/tests/t_enable_mcsum_ext3/expect b/tests/t_enable_mcsum_ext3/expect
index 5460482..0f761a9 100644
--- a/tests/t_enable_mcsum_ext3/expect
+++ b/tests/t_enable_mcsum_ext3/expect
@@ -49,8 +49,8 @@ Change in FS metadata:
    Reserved GDT blocks at 4-259
    Block bitmap at 260 (+259)
 @@ -45,7 +46,7 @@
-   7789 free blocks, 1013 free inodes, 2 directories
-   Free blocks: 404-8192
+   0 free blocks, 1013 free inodes, 2 directories
+   Free blocks: 
    Free inodes: 12-1024
 -Group 1: (Blocks 8193-16384)
 +Group 1: (Blocks 8193-16384) [ITABLE_ZEROED]
@@ -58,8 +58,8 @@ Change in FS metadata:
    Reserved GDT blocks at 8196-8451
    Block bitmap at 8452 (+259)
 @@ -54,6 +55,6 @@
-   7803 free blocks, 1024 free inodes, 0 directories
-   Free blocks: 8582-16384
+   0 free blocks, 1024 free inodes, 0 directories
+   Free blocks: 
    Free inodes: 1025-2048
 -Group 2: (Blocks 16385-24576)
 +Group 2: (Blocks 16385-24576) [ITABLE_ZEROED]
diff --git a/tests/t_enable_mcsum_initbg/expect b/tests/t_enable_mcsum_initbg/expect
index d3b4444..3cbb98f 100644
--- a/tests/t_enable_mcsum_initbg/expect
+++ b/tests/t_enable_mcsum_initbg/expect
@@ -45,8 +45,8 @@ Change in FS metadata:
  Inode count:              65536
  Block count:              524288
  Reserved block count:     26214
--Free blocks:              571
-+Free blocks:              568
+-Free blocks:              570
++Free blocks:              567
  Free inodes:              65048
  First block:              1
  Block size:               1024
@@ -69,10 +69,9 @@ Change in FS metadata:
    Block bitmap at 262 (+261)
    Inode bitmap at 278 (+277)
    Inode table at 294-549 (+293)
--  21 free blocks, 536 free inodes, 2 directories
--  Free blocks: 4413-4433
-+  18 free blocks, 536 free inodes, 2 directories, 536 unused inodes
-+  Free blocks: 4413, 4417-4433
+-  0 free blocks, 536 free inodes, 2 directories
++  0 free blocks, 536 free inodes, 2 directories, 536 unused inodes
+   Free blocks: 
    Free inodes: 489-1024
 -Group 1: (Blocks 8193-16384)
 +Group 1: (Blocks 8193-16384) [INODE_UNINIT, ITABLE_ZEROED]
diff --git a/tests/t_iexpand_full/expect b/tests/t_iexpand_full/expect
index 3eb1715..0474827 100644
--- a/tests/t_iexpand_full/expect
+++ b/tests/t_iexpand_full/expect
@@ -21,8 +21,8 @@ Setting inode size 256
 Exit status is 0
 Change in FS metadata:
 @@ -13 +13 @@
--Free blocks:              12301
-+Free blocks:              12
+-Free blocks:              12299
++Free blocks:              10
 @@ -22 +22 @@
 -Inode blocks per group:   128
 +Inode blocks per group:   256

[PATCH 52/54] debugfs: implement fallocate

[Darrick J. Wong]

Implement a fallocate function for debugfs, and add some tests to
demonstrate that it works (more or less).

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debugfs/debug_cmds.ct                |    3 +
 debugfs/debugfs.8.in                 |    7 +
 debugfs/debugfs.c                    |   36 +++++++
 debugfs/debugfs.h                    |    1 
 tests/d_fallocate/expect.gz          |  Bin
 tests/d_fallocate/name               |    1 
 tests/d_fallocate/script             |  175 ++++++++++++++++++++++++++++++++++
 tests/d_fallocate_bigalloc/expect.gz |  Bin
 tests/d_fallocate_bigalloc/name      |    1 
 tests/d_fallocate_bigalloc/script    |  176 ++++++++++++++++++++++++++++++++++
 tests/d_fallocate_blkmap/expect      |   58 +++++++++++
 tests/d_fallocate_blkmap/name        |    1 
 tests/d_fallocate_blkmap/script      |   85 ++++++++++++++++
 13 files changed, 544 insertions(+)
 create mode 100644 tests/d_fallocate/expect.gz
 create mode 100644 tests/d_fallocate/name
 create mode 100644 tests/d_fallocate/script
 create mode 100644 tests/d_fallocate_bigalloc/expect.gz
 create mode 100644 tests/d_fallocate_bigalloc/name
 create mode 100644 tests/d_fallocate_bigalloc/script
 create mode 100644 tests/d_fallocate_blkmap/expect
 create mode 100644 tests/d_fallocate_blkmap/name
 create mode 100644 tests/d_fallocate_blkmap/script


diff --git a/debugfs/debug_cmds.ct b/debugfs/debug_cmds.ct
index c6f6d6c..34dad9e 100644
--- a/debugfs/debug_cmds.ct
+++ b/debugfs/debug_cmds.ct
@@ -157,6 +157,9 @@ request do_dirsearch, "Search a directory for a particular filename",
 request do_bmap, "Calculate the logical->physical block mapping for an inode",
 	bmap;
 
+request do_fallocate, "Allocate uninitialized blocks to an inode",
+	fallocate;
+
 request do_punch, "Punch (or truncate) blocks from an inode by deallocating them",
 	punch, truncate;
 
diff --git a/debugfs/debugfs.8.in b/debugfs/debugfs.8.in
index 81899a3..1342609 100644
--- a/debugfs/debugfs.8.in
+++ b/debugfs/debugfs.8.in
@@ -306,6 +306,13 @@ from the file \fIfilespec\fR.
 Expand the directory
 .IR filespec .
 .TP
+.BI fallocate " filespec start_block [end_block]
+Allocate and map uninitialized blocks into \fIfilespec\fR between
+logical block \fIstart_block\fR and \fIend_block\fR, inclusive.  If
+\fIend_block\fR is not supplied, this function maps until it runs out
+of free disk blocks or the maximum file size is reached.  Existing
+mappings are left alone.
+.TP
 .BI feature " [fs_feature] [-fs_feature] ..."
 Set or clear various filesystem features in the superblock.  After setting
 or clearing any filesystem features that were requested, print the current
diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
index 4b88f73..2af6b71 100644
--- a/debugfs/debugfs.c
+++ b/debugfs/debugfs.c
@@ -2196,6 +2196,42 @@ void do_punch(int argc, char *argv[])
 		return;
 	}
 }
+
+void do_fallocate(int argc, char *argv[])
+{
+	ext2_ino_t	ino;
+	blk64_t		start, end;
+	int		err;
+	errcode_t	errcode;
+
+	if (common_args_process(argc, argv, 3, 4, argv[0],
+				"<file> start_blk [end_blk]",
+				CHECK_FS_RW | CHECK_FS_BITMAPS))
+		return;
+
+	ino = string_to_inode(argv[1]);
+	if (!ino)
+		return;
+	err = strtoblk(argv[0], argv[2], "logical block", &start);
+	if (err)
+		return;
+	if (argc == 4) {
+		err = strtoblk(argv[0], argv[3], "logical block", &end);
+		if (err)
+			return;
+	} else
+		end = ~0;
+
+	errcode = ext2fs_fallocate(current_fs, EXT2_FALLOCATE_INIT_BEYOND_EOF,
+				   ino, NULL, ~0ULL, start, end - start + 1);
+
+	if (errcode) {
+		com_err(argv[0], errcode,
+			"while fallocating inode %u from %llu to %llu\n", ino,
+			(unsigned long long) start, (unsigned long long) end);
+		return;
+	}
+}
 #endif /* READ_ONLY */
 
 void do_symlink(int argc, char *argv[])
diff --git a/debugfs/debugfs.h b/debugfs/debugfs.h
index e163d0a..76bb22c 100644
--- a/debugfs/debugfs.h
+++ b/debugfs/debugfs.h
@@ -166,6 +166,7 @@ extern void do_imap(int argc, char **argv);
 extern void do_set_current_time(int argc, char **argv);
 extern void do_supported_features(int argc, char **argv);
 extern void do_punch(int argc, char **argv);
+extern void do_fallocate(int argc, char **argv);
 extern void do_symlink(int argc, char **argv);
 
 extern void do_dump_mmp(int argc, char **argv);
diff --git a/tests/d_fallocate/expect.gz b/tests/d_fallocate/expect.gz
new file mode 100644
index 0000000000000000000000000000000000000000..3e6ffc38595b7c451e3de0df3a16f795cf33421b
GIT binary patch
literal 3770
zcmZ9Nc{~$-{KqA}=E^ZmjxiFcMP=1u5+X-hxpK>W6gKA^k(*pYteD9aQf!j@-ZF9@
zIa|z%<Svs4zxDn8{`ft9|GXaW_xt(29`DEJ@%Tt50odT_7N+b+X*TO=5_LlRIp3=+
zUYwp2x!ot`yF<Qymm`s%12x3vEwp8VZp|Eg<<)~-o3D>}6x37KoE6w$b-7eoz5PW1
zRlzvTCtl@Ca_ekW)!OBmnT6E_^6ti0RNcX3*ym3z4_+tK^0)YW4x6-X(}3NZ$?Yn;
zpW7c8^!D0oyV(e<b;fMYm3tHG>l)x3GR?cK)g#dQKH*A*28L(XgRF}W72n*d5h_i2
zZ9e<RJ+!35=U3AnvY+-bH}KeC?)G9uRcqsv*F=Bc;CI}<d=w+*;NFHzMDmsp7tF@i
zetoU(#QduFAt`U!TTRm8-R|%2DEDu{KlhuM%ss?&vdQ@kSnHz0&-`rSWEPjL&*eR4
zfZ-G_yMBM4N!t(4Q(fx0J<+sUyT86L(NDvT4f5@-PWVjhF}Fi#6n!VS*p{B8^qNL^
zv*)bWMxuGpGMU!4(UE`U#^Agv)gFc%tNV`Z53JjN2|o1p3vFxC2%{;vH$@F*+!#@?
zZQ1ein+V5NH0^1>y$bUTcpL5)v9~=G=Kb8RG-x^ESF`2eIHJj_#p<x}V5oWW^Xiq=
zJ4=fn*8=?>F=bkIe$#N;VNB1K7B8LOKgb_f9X^)1rFb413{FRo@;zIZ#KF_{?jy9b
zBb}Q*x|a?oy=!xu7{dbhq%|H{Xmp9(nn*bxTkCbB<Q^$SV+>h9MR$43`L9$(j&kDH
z&<}x?UNZ+vW)Dlz@(gyA%(g&#Ebx<%na`i1;W12@jrVioSD_zkpPIi)e+wf~4piyW
z_jVYTIbkb4kv}~UkFsBy-?cr&qEv@(aOQL}hDG9FVtfLcb+*~VaxZ$!N1Qb?-xT*a
z61)lW#%YIRI)o?#2A5J_Q@e>CBK;L3@hN9zs)BK%LZJ#`*aah-hmp0V0zEdI&jeqb
z>oNVoLCeI`6O|p_&!E8=-^Dj@fnZ-eN(%8j77@Jh<!tbeIYm5O-#$*^B?YKFyDE$1
zQyIu|#XEF0j3le{!SgRLoRuh<j}^+la0lYguENQwekEYHN%S>0>Q^m9M{3Zj?|S*N
z0HM=zJ<++_1e-8QWH1&3auoR0Di>FGE@==5IR)SeJZ6N*sA*5loz&PYHVPTs?gwF>
z+eYx(q~NBQTu9gP-e%VZK_m##@D-uIB0%`_V^Qe`huY_NX8+EBBDXIh#Ii6KO{tlY
z;43!&N5kS2f@BDMm<$8LeX3C`IEbWysa7(|5&gIPE#f!t=9dnWGS>PwXZF#N8cMzU
zo86i1ti_>7`b7lyO8qxg0>^(4ENapNT&c&C7*@6k3X|ya8V0+XQftOvbAxKe+AHKR
zmiO<StBH)-kyPtRwDnTjU!|v+tmI>=b8e$<B4`LHL)Y$%3m#hsrl+9J+~%LmLl_{I
zVi;(I1$ub|lalMa6qM&YvT;6ZteyI<ML8;edulCYn@#NMI%uE?thU~c?&8l2;w|!_
z4yd7286ibay9~GB0lP#de3X%j7l`jKd{9cwOqLMWEZyC7O_`gyX`Om7ye2oV4Yx~;
zi|Z|5DniQKd5x`Yj?JBx%02tlF~Y#jgI2NCyXor>H;AvNZAPcKWx=DDe`TL1Npjy*
zqR;!D-r?a1IeCfYQriRSOr8*fgmI`^s>j?dii5I@b!UT=TL?ZWQy$NKHxikTp3_m4
zB+I`&lOh^|->N^61f)NVE&XcriuxWYlAlniF&=5;fGCt&eC#JOD1M&sw2P6<i1{fL
z&t#13-W^X`M=QEgF$FNY@uVy5&nIWu++I#ZR@*hRi8`r9TS#9UJ%M8i2OU-hklKz8
zh%P*5&XdAr=1~hpB^1MP!*6Is5Czi;v#Sy{K*9q`KX^qvvj_7q&{d+ca|w>nuYxAW
zyPQ7M*Mw7q;@>(K88Vn5GHd7ymx)rO#C$uhaWP!w&&JX3Ycdo}C94b0--=c%zncm#
z5;;Q;X~7=3EqiQPp@hk>Q<iJDd?KLS!NoTkhz)JQaw;o%N2}(6t&5O@u8)_QT1x5y
zv~u(!xV`DI-9L8RC`;wfM?_nM`$n?~8hK3Nmg!o4kEXa^B7c@Xf?4p$Prhs75}n(M
zoHpZzdW?YVx6A8~&wexc6+(w=O`lOK0!<8-J{xKfl&@R1KE&+38(cF{3n+B@l^DNw
z>7go{5eJ1<^FX^Jk1llu)zlbi+NqB35D4;w002+a#Q;hx&hiZSUcYU`QM{ts?RIrI
zCfRUij4&|d8q>4*)E23qa80TpQC*d8Q62x88PwkJLyG@Xv|)F>)a}(JK5gClu#4nP
z+20a1qfLn);Q3CGwQXIVOHuc9Ex|_@GtokM)0tw%<FV%2M?4JxMYmTbNC#xbk_KNq
z4!A!Wb3ndx_)k8{zYC>%$gb-`I^f*J8SL|JDQE49ezONH+rNmgE|1{dpE20C{x+*L
z-WGe{RdN6}?=|~As|5}u9e}si7nz59zQP8ZDvOwxf*N}y?9noACmBE8p*z?k(xjSL
zYWGwcrx>LBJgB$_pdw<fH!_$gj5w~cmrgV;yy2-(ED$3Bh<SRnkVp#};qpF>eQK3H
zJ(?bt1{Vf6FpATiFtUP;$=7$=14!2ap<GKP>@q7yuFg6YT+iUjn|5KB;g5V~5P1SX
zfr1huuiq-UPHe+kLIqc#a&8=*qQVJpjsudHAzm(85c0Wmr(PG=hHYP#5#)-jf6tB<
zOsEn`h`;{FF%koy+~c8WL&8bd?~@eqLJ6&`kta){u7A>l@MThA=LD}<xyP*!09%^|
z66jhgg)#UUm0;+yE^q%vM{p+>F8PEKj6AbA(xW4iAaM>x#x0PP!6kGSCpNzxgw`|+
zY2{AG#w|HoHV>hUEkVk#Pjo!K#=n66u-cXtdf=xzIuaW<+8wy5MUJXgdpqP?5WD@3
zmmf%RU`2j2&esi*%Vb^j(Gu0!h3QMN<EVogtHvSmfqe4--Z%6=9`F1ME<p<jK?k2n
z!CT9CSnIW8Y5kPXgepJT#~0}H0bIyk4)K?U%RL)A6p!M`%*bclX$oQ7{G!AMy}vx2
z?6t^KV^+AJouUnk2Y`{j7%H$eChl#y0^Uet3Fr<_iRP#^iR_N$n0MtA?qVldxT`5(
z&w~KB2!W&Q+)k109%J6OAr4T%`p34`AiWd390M=^BX9s^kXtT!01309R-Pl|vjxdP
z;pNjvmW5_nxzWjOCJs}*fZjVLL}TCk6XV>y?G$mVGtlU05AA@?i>RyoA6cP>OEsnz
z|8R}QEyuXby%MhkW=EPl#FP{H@H3iEzLXO`#m1cS>x=@V#<)NPRZhN+Q3HYS)+X>x
zGbaZ+WulL;^NK{Ao6ng7+FxQ3kN;1^bN`p1{*oe~eKgQsC3L=`D=G$fty@R^cySvR
zLVgYLfa(c2{JyF=Vu3|<8a)HarTqK674*b;g%^qPecTzWicz1=n!lOw{lWzf{gawJ
zHkiA9QN`&H&W@-u56uxhY!vS(eoIYJPux}$uzjka;VeL$+fvY(FC!KiE2(tSc{oc8
z#KGGc@K+0QJeGpcgi2sF#6r+k!C8|C2`R_2T0Jw6Vc;7cRKz4hwEG-oBmh&UJYst<
z^h&ToV|o|LhQDiR5%@jlm|$)(&6GaQOF&#_)C=8DBZdx|cTQHW8CwXBB~&ztfH*GA
zXWQ}cMu~3*{F@O*D<MM4gxa=@Ef6s)%Sy4Fd!;drtSZ4%;XNg&F;4GWr-p2ft*aT~
zg4v*E*VsTEII{qO$M5CcMBO+oIqYjMp6!<1sm`+0WD#5Zac7~hsSmFqNprpCu77zT
z+8nE6j(yX|Q#~2v<-&4^ZlW%n@6EbWh+rB(TAeMIn%O4y;Si?SeY<;>@2d}_GQ6Ui
z_iNglVWrzV)c_$c7h|kWg8SLbD5%zrCb(j^?2Vc%eNB)|;hS53`RaTzBuQ}CT%OZ%
zWxVzxi`Ud7Wkkt}>hXNfa7|<S%N<78F9eoW7suT-CX+LArX0m}FYpT_dARnvpKyWM
zSn85b7ttNW`X3(QFM1po&k3W}v_*U|y5ho~|8&5g^TN#%l`vyf?bCS$4oiJ^hxzhF
z`URi3?%W#YlRn;0X+ATm;4z=LX`-W&ESqZz1FF=A<$p5o)K7B~L4JF-9@&nqH!XGw
zmCX3oBzEzMYh!JUpZdUyZgTM-VL3=Ys9#R93;5(YfBm>=$);8sFj+y-HWFKT{vIq7
zHTT}P`92j>GJmR2%IyO3!8{n=m3uBnA1+c<4@cr*B;^zwGs-#Q>xdcjKk)7Sd$l0E
zM87Dj?Gu7cGge2?Mk%}}1bswDMJRs1&gM@Bb3W9j5wuwkQe1^tDr<f%6w}@0#U<I7
zP4sFH-$FD*lU)stNmL}C6zRVqDOu~gY54c|E+k!=T>78`ZLw>V&5cP)Kju#pB_$^r
zgtQdPbyOPN<Y?FDcDr~?M@qgc9G>5&0}X%WE<b-q?xwIho8;gZSy0ZIoJ6=tQ-HoV
zEu2rfMA?yEUidsgd)UhD?g<^Jjgq57KKc>KXv=E@A(CVPUw+TjQ4y(A>I9jeaqJ1v
zBuNB=_tfSLbfA{X<&#E@{(-tehQGi$H2I2_0M_j#_cTP>YRsYFkl@7t|7*AGejO%N
zy$NfuuiEobNcpz#GkoGv=90I-n6%kXN=OmmWm}ccfn!UpeMD18!|&Cqe%$uxxnQ8@
zY|Ii}ow74sH@A>zvA+S5FVpeaq@1XYUXmEKBAoovg9|!n*z>SE*!iyWb2q>0@L;}l
y0k>{<MPq+8UfTX(XFh&w5ttwuF<UycB^qw5mO16~r}$&Uk<ipHmv5P~vHcHq|7W-W

literal 0
HcmV?d00001

diff --git a/tests/d_fallocate/name b/tests/d_fallocate/name
new file mode 100644
index 0000000..72d0ed3
--- /dev/null
+++ b/tests/d_fallocate/name
@@ -0,0 +1 @@
+fallocate sparse files and big files
diff --git a/tests/d_fallocate/script b/tests/d_fallocate/script
new file mode 100644
index 0000000..ae8956e
--- /dev/null
+++ b/tests/d_fallocate/script
@@ -0,0 +1,175 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O ^bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+make_file() {
+	name="$1"
+	start="$2"
+	flag="$3"
+
+	cat << ENDL
+write /dev/null $name
+sif /$name size 40960
+eo /$name
+set_bmap $flag 10 $((start + 10))
+set_bmap $flag 13 $((start + 13))
+set_bmap $flag 26 $((start + 26))
+set_bmap $flag 29 $((start + 29))
+ec
+sif /$name blocks 8
+setb $((start + 10))
+setb $((start + 13))
+setb $((start + 26))
+setb $((start + 29))
+ENDL
+}
+
+#Files we create:
+# a: fallocate a 40k file
+# b*: falloc sparse file starting at b*
+# c*: falloc spare file ending at c*
+# d: midcluster to midcluster, surrounding sparse
+# e: partial middle cluster alloc
+# f: one big file
+# g*: falloc sparse init file starting at g*
+# h*: falloc sparse init file ending at h*
+# i: midcluster to midcluster, surrounding sparse init
+# j: partial middle cluster alloc
+# k: one big init file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+sif /a size 40960
+fallocate /a 0 39
+ENDL
+echo "ex /a" >> $TMPFILE.cmd2
+
+make_file sample $base --uninit >> $TMPFILE.cmd
+echo "ex /sample" >> $TMPFILE.cmd2
+base=10000
+
+for i in 8 9 10 11 12 13 14 15; do
+	make_file b$i $(($base + (40 * ($i - 8)))) --uninit >> $TMPFILE.cmd
+	echo "fallocate /b$i $i 39" >> $TMPFILE.cmd
+	echo "ex /b$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file c$i $(($base + 320 + (40 * ($i - 24)))) --uninit >> $TMPFILE.cmd
+	echo "fallocate /c$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /c$i" >> $TMPFILE.cmd2
+done
+
+make_file d $(($base + 640)) --uninit >> $TMPFILE.cmd
+echo "fallocate /d 4 35" >> $TMPFILE.cmd
+echo "ex /d" >> $TMPFILE.cmd2
+
+make_file e $(($base + 680)) --uninit >> $TMPFILE.cmd
+echo "fallocate /e 19 20" >> $TMPFILE.cmd
+echo "ex /e" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null f
+sif /f size 1024
+eo /f
+set_bmap --uninit 0 9000
+ec
+sif /f blocks 2
+setb 9000
+fallocate /f 0 8999
+ENDL
+echo "ex /f" >> $TMPFILE.cmd2
+
+# Now do it again, but with initialized blocks
+base=20000
+for i in 8 9 10 11 12 13 14 15; do
+	make_file g$i $(($base + (40 * ($i - 8)))) >> $TMPFILE.cmd
+	echo "fallocate /g$i $i 39" >> $TMPFILE.cmd
+	echo "ex /g$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file h$i $(($base + 320 + (40 * ($i - 24)))) >> $TMPFILE.cmd
+	echo "fallocate /h$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /h$i" >> $TMPFILE.cmd2
+done
+
+make_file i $(($base + 640)) >> $TMPFILE.cmd
+echo "fallocate /i 4 35" >> $TMPFILE.cmd
+echo "ex /i" >> $TMPFILE.cmd2
+
+make_file j $(($base + 680)) >> $TMPFILE.cmd
+echo "fallocate /j 19 20" >> $TMPFILE.cmd
+echo "ex /j" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null k
+sif /k size 1024
+eo /k
+set_bmap 0 19000
+ec
+sif /k blocks 2
+setb 19000
+fallocate /k 0 8999
+sif /k size 9216000
+ENDL
+echo "ex /k" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w -f $TMPFILE.cmd $TMPFILE > /dev/null 2>&1
+$DEBUGFS_EXE -f $TMPFILE.cmd2 $TMPFILE >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/d_fallocate_bigalloc/expect.gz b/tests/d_fallocate_bigalloc/expect.gz
new file mode 100644
index 0000000000000000000000000000000000000000..8640bc29dc71b09810f110d07d84d883b1109b88
GIT binary patch
literal 2673
zcmV-%3Xb(3iwFQXkyum$1KnF)Z``;QeGTSU5GV@l7Fk=o<fTLdMX}vYfB;!^fwTzv
zR7`BeR&CE1GviItUtftl8dDm5kloQs2MB0octq(O>GE7&&g$cOx2ZO_SK@G2ici(;
z`r%=FEk2j)!}G2b)n4om)x(2$-rSTsaZ~<QK5U;J%gtf+p)AGS;qbJ-y1XpE9`wbR
z>b|<TFL#^r;bObHy{u};*NdmSr^~+{PrKcT$8B@oP5tNN`cQ2*tJP1tdbIi)#OLav
z+<)62%17~~I@}4vtc;NN;!_R2-;4F;M(E6%8os?L_p8;9%~?J6cDH?g5{LDt`V?29
z{=9ni>Snts#l&azaaTF8JKz5b-akK;yWk3gyI5aeZ=W|9;&{j52&>hv>-}EHEAi7^
zd3}GpnEh+q^lNv{ef6LEeqC1zy53Z~^7^pdeXCX4JzqDH`A%xE)Ae?<ssF5gs}A3M
zKOOYjm7mL9xw$TdyJ!r@K@ZRI{`v87U7sz~@KEj#|H6%52`Mj~OzuC#clJX5QEaxG
zAL{WB)$Q~4dH?<Ex|7zIUSWTyt2bY(LwzIllecdkyliz-etN$Bycb_;rb~ae_W!O#
z`Bhx5SMQrAQoPw5c2zx;|GeMcR@dtXfAH(wx4k=hUv6&eXX@?4`gXq(*`>(pZ{=nE
z&-Kg14^5-2a97qGNm(fOozBER>X)QCbVuK>AD<q|b6kRMx=CC@-3re9k!#&YwwKW2
z=!jFmDc4^Cr$%t9#gXy2x~=>BBi9b&^bu!)v#!4d&h~H}sy-EGqe&FFU8<SNvgW96
z>m&E!S(-?>{>tOX(<CQo(!dq2zXq-t8SjLq`iW}#TaQz=H(r+}dp??MNR!6!44QN#
zO$?e$B25M~nM9flt}~yeNoMDN>l{xu^W$6aq#}8;o+p{5@gy@1TpCX@^E^p8!;@Tw
zC#gJ7Ql;@E6?oE#JZbQx6M4dOFL{P1S|W<q<L2NgpO4}c@|2UuVFIeZjN_>QPeme6
z1$eSEJgH@PBI$Xe_}R|!L<r-Q7{&=<oD#!0A&k>Z7?*k>o)|oFB2S3plo-beahww4
zI3bSHTpZ`$!;=gRoXo=Ui>%|-Rkmt+wg5gQ7J<KMd&B}qN8F3uYWjm%t446@h-2W`
z^=IIC1gDQU6AEID!zoM{QuT4%;yQvTOc-i<!nh`VHbF!vh-!@oK6*e95za7*ydVlQ
z%;}Au2;!b}g&9Vj&M<vJj!(LF{WWlUPLRMiH~k@qHY2zbLC6t;nW<?feDgK&`5a+~
zUju&g6MiQniORum5g$n$PnelhVMZLoP=3+$2T#QePi7gObmn=|GL0vVnMqR`PZ~3m
zrZk>3W+qKDJmm}Vq`?y>@`Ra5C(KN_>kpnxK9bRl@PwJEX(xFq7UHP@Pd1S!Of;OB
zVK6cbj?XY0PmBqpX(xHI%kadA|E$t@V#I$=%rF@7pLHTnh~qpH$5plvPl)537{?iL
zoQp)B5XU(o!^m8J@YEVC8sSMU!;?WAH;HlFAdZ{F48tIfn}iHQy8fOgqhv4scX+xv
zA4@u932%XE?V62b2~pf6<`)K0+$7`|?D~VH9QHXMWT{*xyD+KQg+X><60-|~pl>3w
zi#90v@jN|w)_>Wi&1{|-;L^-&p0(pm2Eor~&n=oqq_DP|TL>A-Ax#HRy9JxMSkSca
z4#j{Rz`4a{>3Z7ZVza|F?SLE&PWqm1;mNJOVpUO(ji+0jPxhJx26qzRiZL9wIR92P
z`CW4o%$<_PaHhp60?w@hVJ?wo441b!cSlXzcrik3rBObMBM!W<OM<}*jq+k!9D^6*
zauDlR#}rnpPH-}7aR$8P4KG@GoK|Bv+2V5Wk~`0c)i|Ax;i$zG;H7AIVd-%!$8g8z
zr2sFM9XY`981DGISkFtd?%d3+z>V_KJ}-XVxo)MKbHIxk<)wXIq+g}1Upe>M`}-^W
zczID2gRlK<alZGyIELZ+q04$<m^xwD^om7<ViASKqD}xdodC?Prvif8*|-yWO=p2d
zxt_)&;1>qpb&m^XLPBAQ!opG~=$f8WAe5Bq_QffL@3%)N3JG&ZbSy%N5xwmB^s?db
zx~G>M(94E;Y24vU)oM;JhF%=47eg<TsF#6WCR#6pyUC;V%Ar?2t=I3XhK>0ga`VrG
z&oRBS5TSfVVfk!MFblzKv|!c?)|T301@j?~!cyCeV7|y!w=z~RA0FvgmKzmJE+beo
z@h1u9>v$B_@n!_`6}+a66-+7!rV|Cz5KKo4hIk=4S}=xSJR=w_B$!DMENcKc1bZnc
z4hTk}U<e|zO_lVlU^xUU5(O(DSP?B)0l{pvU>0{{X9QEr2u4|=V7{_RVP$jX4I_9C
ziuN1?&q0ZvgWx$R+H(**2Ss}hg6E(a&mr|fg6UMj7=m%MU@#pNZ8`|1gAz>#!E{ix
z=^&U6iZ&gDctLZfgBKEP+H~Zng28kIrQ42Rd2d;TP_(76XxkAi@0Du_<yx94*XE}}
z!>p)Lv%)Q|Bea59Q3Gbh91)?=3T8zOn3cRmbc9wgD{9)T+$~cQuE$koswABEmTd?n
z;ijEfwgE0o`xN=KuPr5qWU7+~gk4Vs#OWJ7EVP1=P-7|kfY7?>50f)qc7BojWu=Z#
zcBZiGd@6UaMn@<(Q&@2B3(X4&olgpFPK7oQ+Dr+3n%^q}q4Qy(PeZq_W;gAy(5InW
z!GMgeJq3YShl0Tz1@-UKK-?7!<|wFq_XRV{2&QGCV7^G+v}uBAEXZgXE0~5k(lS;s
z4FfW|=r<ymhB*of@iT&Hn4_Qs-xn-jNH9(nOhYi8D42#|94#2kk>+T@U_eG!TL%Pl
z_XKkk><E|>4CW}<3(yy=SV%CNDp&!*ibTN*2xg-NgE`VRRxpN<7+vNb5Uepd43onn
zCg)TzhRF#wCG-We%Lv9QQ80#CVM-H>VOChh3dS%itfB>jSz%5T3}%IOv|unDJZd^r
zwvb?{rh{QRIMH-4Ob17s4u<LAXw$(k9c&W?gX!QRLNJxNo)GM1n?_$Sxr|^Y)pQt`
z4wGm)3`~cKHXR10!$g}71JhwtnqUT|!%&Q1()F|$=_T*$2v(g>E*;9%vmD9{WN$i@
z3&F$alMQ>5*_(moFdUn`8CVX(vDuq}<uJKTO4AK2hcU5fx`E{|c}$w_?glfzX;)+Q
zV4D8#WmaQyNhw=h2wPn!-0Cv9$%U}Vg~ClPliOPe+gm7cdkbNE3x(TTCO5VaHnvc>
zv87A<{W<nwsa|Z8*r6iB9V(q2SSol;TrIxcn_{h>BAPuh13R!3wym@^{QAzkPbqR|
z-ilyP%m2milHcyivi<eALyIrFZSza>?GC6vWa2x{GkWd6FK_<LhRpXDtGCtHzld+m
zugu3z@?D*GN%I2KT&vJQ_I?-rr{*KxzI*>heE8>wzrXoKeEji)_~mauzkB=c&CmYp
f9)!C;rDnRY1HYwqzH#!$)N=J-JwLQM>OTMg)z2kH

literal 0
HcmV?d00001

diff --git a/tests/d_fallocate_bigalloc/name b/tests/d_fallocate_bigalloc/name
new file mode 100644
index 0000000..915645c
--- /dev/null
+++ b/tests/d_fallocate_bigalloc/name
@@ -0,0 +1 @@
+fallocate sparse files and big files with bigalloc
diff --git a/tests/d_fallocate_bigalloc/script b/tests/d_fallocate_bigalloc/script
new file mode 100644
index 0000000..6b6bf97
--- /dev/null
+++ b/tests/d_fallocate_bigalloc/script
@@ -0,0 +1,176 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+	cluster_size = 8192
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+make_file() {
+	name="$1"
+	start="$2"
+	flag="$3"
+
+	cat << ENDL
+write /dev/null $name
+sif /$name size 40960
+eo /$name
+set_bmap $flag 10 $((start + 10))
+set_bmap $flag 13 $((start + 13))
+set_bmap $flag 26 $((start + 26))
+set_bmap $flag 29 $((start + 29))
+ec
+sif /$name blocks 32
+setb $((start + 10))
+setb $((start + 13))
+setb $((start + 26))
+setb $((start + 29))
+ENDL
+}
+
+#Files we create:
+# a: fallocate a 40k file
+# b*: falloc sparse file starting at b*
+# c*: falloc spare file ending at c*
+# d: midcluster to midcluster, surrounding sparse
+# e: partial middle cluster alloc
+# f: one big file
+# g*: falloc sparse init file starting at g*
+# h*: falloc sparse init file ending at h*
+# i: midcluster to midcluster, surrounding sparse init
+# j: partial middle cluster alloc
+# k: one big init file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+sif /a size 40960
+fallocate /a 0 39
+ENDL
+echo "ex /a" >> $TMPFILE.cmd2
+
+make_file sample $base --uninit >> $TMPFILE.cmd
+echo "ex /sample" >> $TMPFILE.cmd2
+base=10000
+
+for i in 8 9 10 11 12 13 14 15; do
+	make_file b$i $(($base + (40 * ($i - 8)))) --uninit >> $TMPFILE.cmd
+	echo "fallocate /b$i $i 39" >> $TMPFILE.cmd
+	echo "ex /b$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file c$i $(($base + 320 + (40 * ($i - 24)))) --uninit >> $TMPFILE.cmd
+	echo "fallocate /c$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /c$i" >> $TMPFILE.cmd2
+done
+
+make_file d $(($base + 640)) --uninit >> $TMPFILE.cmd
+echo "fallocate /d 4 35" >> $TMPFILE.cmd
+echo "ex /d" >> $TMPFILE.cmd2
+
+make_file e $(($base + 680)) --uninit >> $TMPFILE.cmd
+echo "fallocate /e 19 20" >> $TMPFILE.cmd
+echo "ex /e" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null f
+sif /f size 1024
+eo /f
+set_bmap --uninit 0 9000
+ec
+sif /f blocks 16
+setb 9000
+fallocate /f 0 8999
+ENDL
+echo "ex /f" >> $TMPFILE.cmd2
+
+# Now do it again, but with initialized blocks
+base=20000
+for i in 8 9 10 11 12 13 14 15; do
+	make_file g$i $(($base + (40 * ($i - 8)))) >> $TMPFILE.cmd
+	echo "fallocate /g$i $i 39" >> $TMPFILE.cmd
+	echo "ex /g$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file h$i $(($base + 320 + (40 * ($i - 24)))) >> $TMPFILE.cmd
+	echo "fallocate /h$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /h$i" >> $TMPFILE.cmd2
+done
+
+make_file i $(($base + 640)) >> $TMPFILE.cmd
+echo "fallocate /i 4 35" >> $TMPFILE.cmd
+echo "ex /i" >> $TMPFILE.cmd2
+
+make_file j $(($base + 680)) >> $TMPFILE.cmd
+echo "fallocate /j 19 20" >> $TMPFILE.cmd
+echo "ex /j" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null k
+sif /k size 1024
+eo /k
+set_bmap 0 19000
+ec
+sif /k blocks 16
+setb 19000
+fallocate /k 0 8999
+sif /k size 9216000
+ENDL
+echo "ex /k" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w -f $TMPFILE.cmd $TMPFILE > /dev/null 2>&1
+$DEBUGFS_EXE -f $TMPFILE.cmd2 $TMPFILE >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/d_fallocate_blkmap/expect b/tests/d_fallocate_blkmap/expect
new file mode 100644
index 0000000..f7ae606
--- /dev/null
+++ b/tests/d_fallocate_blkmap/expect
@@ -0,0 +1,58 @@
+Creating filesystem with 65536 1k blocks and 4096 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/4096 files (0.0% non-contiguous), 2340/65536 blocks
+Exit status is 0
+debugfs write files
+debugfs: stat /a
+Inode: 12   Type: regular    Mode:  0666   Flags: 0x0
+Generation: 0    Version: 0x00000000:00000000
+User:     0   Group:     0   Size: 40960
+File ACL: 0    Directory ACL: 0
+Links: 1   Blockcount: 82
+Fragment:  Address: 0    Number: 0    Size: 0
+Size of extra inode fields: 28
+BLOCKS:
+(0-1):1312-1313, (2-11):8000-8009, (IND):8010, (12-39):8011-8038
+TOTAL: 41
+
+debugfs: stat /b
+Inode: 13   Type: regular    Mode:  0666   Flags: 0x0
+Generation: 0    Version: 0x00000000:00000000
+User:     0   Group:     0   Size: 10240000
+File ACL: 0    Directory ACL: 0
+Links: 1   Blockcount: 20082
+Fragment:  Address: 0    Number: 0    Size: 0
+Size of extra inode fields: 28
+BLOCKS:
+(0-11):10000-10011, (IND):10012, (12-267):10013-10268, (DIND):10269, (IND):10270, (268-523):10271-10526, (IND):10527, (524-779):10528-10783, (IND):10784, (780-1035):10785-11040, (IND):11041, (1036-1291):11042-11297, (IND):11298, (1292-1547):11299-11554, (IND):11555, (1548-1803):11556-11811, (IND):11812, (1804-2059):11813-12068, (IND):12069, (2060-2315):12070-12325, (IND):12326, (2316-2571):12327-12582, (IND):12583, (2572-2827):12584-12839, (IND):12840, (2828-3083):12841-13096, (IND):13097, (3084-3339):13098-13353, (IND):13354, (3340-3595):13355-13610, (IND):13611, (3596-3851):13612-13867, (IND):13868, (3852-4107):13869-14124, (IND):14125, (4108-4363):14126-14381, (IND):14382, (4364-4619):14383-14638, (IND):14639, (4620-4875):14640-14895, (IND):14896, (4876-5131):14897-15152, (IND):15153, 
 (5132-5387):15154-15409, (IND):15410, (5388-5643):15411-15666, (IND):15667, (5644-5899):15668-15923, (IND):15924, (5900-6155):15925-16180, (IND):16181, (6156-6411):16182-16437, (IND):16438,!
  (6412-6667):16439-16694, (IND):16695, (6668-6923):16696-16951, (IND):16952, (6924-7179):16953-17208, (IND):17209, (7180-7435):17210-17465, (IND):17466, (7436-7691):17467-17722, (IND):17723, (7692-7947):17724-17979, (IND):17980, (7948-8203):17981-18236, (IND):18237, (8204-8459):18238-18493, (IND):18494, (8460-8715):18495-18750, (IND):18751, (8716-8971):18752-19007, (IND):19008, (8972-9227):19009-19264, (IND):19265, (9228-9483):19266-19521, (IND):19522, (9484-9739):19523-19778, (IND):19779, (9740-9995):19780-20035, (IND):20036, (9996-9999):20037-20040
+TOTAL: 10041
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong for group #0 (6841, counted=6840).
+Fix? yes
+
+Free blocks count wrong for group #1 (1551, counted=1550).
+Fix? yes
+
+Free blocks count wrong (53116, counted=53114).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 13/4096 files (7.7% non-contiguous), 12422/65536 blocks
+Exit status is 1
diff --git a/tests/d_fallocate_blkmap/name b/tests/d_fallocate_blkmap/name
new file mode 100644
index 0000000..ba2b61d
--- /dev/null
+++ b/tests/d_fallocate_blkmap/name
@@ -0,0 +1 @@
+fallocate sparse files and big files on a blockmap fs
diff --git a/tests/d_fallocate_blkmap/script b/tests/d_fallocate_blkmap/script
new file mode 100644
index 0000000..9c48cbc
--- /dev/null
+++ b/tests/d_fallocate_blkmap/script
@@ -0,0 +1,85 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP1
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,^extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O ^bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+
+#Files we create:
+# a: fallocate a 40k file
+# k: one big file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+sif /a bmap[2] 8000
+sif /a size 40960
+sif /a i_blocks 2
+setb 8000
+fallocate /a 0 39
+
+write /dev/null b
+sif /b size 10240000
+sif /b bmap[0] 10000
+sif /b i_blocks 2
+setb 10000
+fallocate /b 0 9999
+ENDL
+echo "stat /a" >> $TMPFILE.cmd2
+echo "stat /b" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w -f $TMPFILE.cmd $TMPFILE > /dev/null 2>&1
+$DEBUGFS_EXE -f $TMPFILE.cmd2 $TMPFILE >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed -e '/^.*time:.*$/d' < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

[PATCH 53/54] tests: test debugfs punch command

[Darrick J. Wong]

Test punching out various parts of sparse files.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/d_punch/expect          |  208 +++++++++++++++++++++++++++++++++++++++++
 tests/d_punch/name            |    1 
 tests/d_punch/script          |  129 +++++++++++++++++++++++++
 tests/d_punch_bigalloc/expect |  207 +++++++++++++++++++++++++++++++++++++++++
 tests/d_punch_bigalloc/name   |    1 
 tests/d_punch_bigalloc/script |  130 ++++++++++++++++++++++++++
 6 files changed, 676 insertions(+)
 create mode 100644 tests/d_punch/expect
 create mode 100644 tests/d_punch/name
 create mode 100644 tests/d_punch/script
 create mode 100644 tests/d_punch_bigalloc/expect
 create mode 100644 tests/d_punch_bigalloc/name
 create mode 100644 tests/d_punch_bigalloc/script


diff --git a/tests/d_punch/expect b/tests/d_punch/expect
new file mode 100644
index 0000000..764715e
--- /dev/null
+++ b/tests/d_punch/expect
@@ -0,0 +1,208 @@
+Creating filesystem with 65536 1k blocks and 4096 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/4096 files (0.0% non-contiguous), 2345/65536 blocks
+Exit status is 0
+debugfs write files
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+debugfs: ex /sample
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1323              0
+ 1/ 1   1/  5     0 -     9  1313 -  1322     10 Uninit
+ 1/ 1   2/  5    11 -    12  1324 -  1325      2 Uninit
+ 1/ 1   3/  5    14 -    25  1327 -  1338     12 Uninit
+ 1/ 1   4/  5    27 -    28  1340 -  1341      2 Uninit
+ 1/ 1   5/  5    30 -    39  1343 -  1352     10 Uninit
+debugfs: ex /b8
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1390              0
+ 1/ 1   1/  4     0 -     0  1326 -  1326      1 Uninit
+ 1/ 1   2/  4     1 -     1  1339 -  1339      1 Uninit
+ 1/ 1   3/  4     2 -     2  1342 -  1342      1 Uninit
+ 1/ 1   4/  4     3 -     7  1353 -  1357      5 Uninit
+debugfs: ex /b9
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1368              0
+ 1/ 1   1/  1     0 -     8  1358 -  1366      9 Uninit
+debugfs: ex /b10
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1378              0
+ 1/ 1   1/  2     0 -     0  1367 -  1367      1 Uninit
+ 1/ 1   2/  2     1 -     9  1369 -  1377      9 Uninit
+debugfs: ex /b11
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1389              0
+ 1/ 1   1/  1     0 -     9  1379 -  1388     10 Uninit
+debugfs: ex /b12
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1401              0
+ 1/ 1   1/  2     0 -     9  1391 -  1400     10 Uninit
+ 1/ 1   2/  2    11 -    11  1402 -  1402      1 Uninit
+debugfs: ex /b13
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1413              0
+ 1/ 1   1/  2     0 -     9  1403 -  1412     10 Uninit
+ 1/ 1   2/  2    11 -    12  1414 -  1415      2 Uninit
+debugfs: ex /b14
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1426              0
+ 1/ 1   1/  2     0 -     9  1416 -  1425     10 Uninit
+ 1/ 1   2/  2    11 -    12  1427 -  1428      2 Uninit
+debugfs: ex /b15
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1439              0
+ 1/ 1   1/  3     0 -     9  1429 -  1438     10 Uninit
+ 1/ 1   2/  3    11 -    12  1440 -  1441      2 Uninit
+ 1/ 1   3/  3    14 -    14  1443 -  1443      1 Uninit
+debugfs: ex /c24
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    25 - 4294967295  1453         4294967271
+ 1/ 1   1/  3    25 -    25  1468 -  1468      1 Uninit
+ 1/ 1   2/  3    27 -    28  1470 -  1471      2 Uninit
+ 1/ 1   3/  3    30 -    39  1473 -  1482     10 Uninit
+debugfs: ex /c25
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1483         4294967269
+ 1/ 1   1/  2    27 -    28  1485 -  1486      2 Uninit
+ 1/ 1   2/  2    30 -    39  1488 -  1497     10 Uninit
+debugfs: ex /c26
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1484         4294967269
+ 1/ 1   1/  2    27 -    28  1498 -  1499      2 Uninit
+ 1/ 1   2/  2    30 -    39  1501 -  1510     10 Uninit
+debugfs: ex /c27
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    28 - 4294967295  1487         4294967268
+ 1/ 1   1/  2    28 -    28  1512 -  1512      1 Uninit
+ 1/ 1   2/  2    30 -    39  1514 -  1523     10 Uninit
+debugfs: ex /c28
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1500         4294967266
+ 1/ 1   1/  1    30 -    39  1526 -  1535     10 Uninit
+debugfs: ex /c29
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1511         4294967266
+ 1/ 1   1/  1    30 -    39  1537 -  1546     10 Uninit
+debugfs: ex /c30
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    31 - 4294967295  1513         4294967265
+ 1/ 1   1/  1    31 -    39  1549 -  1557      9 Uninit
+debugfs: ex /c31
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    32 - 4294967295  1524         4294967264
+ 1/ 1   1/  1    32 -    39  1560 -  1567      8 Uninit
+debugfs: ex /d
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1525              0
+ 1/ 1   1/  3     0 -     0  1442 -  1442      1 Uninit
+ 1/ 1   2/  3     1 -     3  1444 -  1446      3 Uninit
+ 1/ 1   3/  3    36 -    39  1573 -  1576      4 Uninit
+debugfs: ex /e
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1547              0
+ 1/ 1   1/ 11     0 -     5  1447 -  1452      6 Uninit
+ 1/ 1   2/ 11     6 -     9  1454 -  1457      4 Uninit
+ 1/ 1   3/ 11    11 -    12  1459 -  1460      2 Uninit
+ 1/ 1   4/ 11    14 -    18  1462 -  1466      5 Uninit
+ 1/ 1   5/ 11    21 -    21  1472 -  1472      1 Uninit
+ 1/ 1   6/ 11    22 -    22  1536 -  1536      1 Uninit
+ 1/ 1   7/ 11    23 -    23  1548 -  1548      1 Uninit
+ 1/ 1   8/ 11    24 -    25  1558 -  1559      2 Uninit
+ 1/ 1   9/ 11    27 -    28  1569 -  1570      2 Uninit
+ 1/ 1  10/ 11    30 -    30  1572 -  1572      1 Uninit
+ 1/ 1  11/ 11    31 -    39  1577 -  1585      9 Uninit
+debugfs: ex /f
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  2     0 -     0  9000 -  9000      1 Uninit
+ 0/ 0   2/  2  8999 -  8999 17999 - 17999      1 Uninit
+Pass 1: Checking inodes, blocks, and sizes
+Inode 15 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 16 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 17 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 18 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 19 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 20 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 21 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 22 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 23 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 24 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 25 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 26 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 27 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 28 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 29 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 30 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Pass 1E: Optimizing extent trees
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong for group #1 (7934, counted=7933).
+Fix? yes
+
+Free blocks count wrong (62939, counted=62938).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 32/4096 files (43.8% non-contiguous), 2598/65536 blocks
+Exit status is 1
diff --git a/tests/d_punch/name b/tests/d_punch/name
new file mode 100644
index 0000000..724639f
--- /dev/null
+++ b/tests/d_punch/name
@@ -0,0 +1 @@
+punch sparse files and big files
diff --git a/tests/d_punch/script b/tests/d_punch/script
new file mode 100644
index 0000000..7a77c69
--- /dev/null
+++ b/tests/d_punch/script
@@ -0,0 +1,129 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP1
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O ^bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+make_file() {
+	name="$1"
+	start="$2"
+	flag="$3"
+
+	cat << ENDL
+write /dev/null $name
+fallocate /$name 0 39
+punch /$name 10 10
+punch /$name 13 13
+punch /$name 26 26
+punch /$name 29 29
+ENDL
+}
+
+#Files we create:
+# a: punch a 40k file
+# b*: punch sparse file starting at b*
+# c*: punch spare file ending at c*
+# d: midcluster to midcluster, surrounding sparse
+# e: partial middle cluster alloc
+# f: one big file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+fallocate /a 0 39
+punch /a 0 39
+ENDL
+echo "ex /a" >> $TMPFILE.cmd2
+
+make_file sample $base --uninit >> $TMPFILE.cmd
+echo "ex /sample" >> $TMPFILE.cmd2
+base=10000
+
+for i in 8 9 10 11 12 13 14 15; do
+	make_file b$i $(($base + (40 * ($i - 8)))) --uninit >> $TMPFILE.cmd
+	echo "punch /b$i $i 39" >> $TMPFILE.cmd
+	echo "ex /b$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file c$i $(($base + 320 + (40 * ($i - 24)))) --uninit >> $TMPFILE.cmd
+	echo "punch /c$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /c$i" >> $TMPFILE.cmd2
+done
+
+make_file d $(($base + 640)) --uninit >> $TMPFILE.cmd
+echo "punch /d 4 35" >> $TMPFILE.cmd
+echo "ex /d" >> $TMPFILE.cmd2
+
+make_file e $(($base + 680)) --uninit >> $TMPFILE.cmd
+echo "punch /e 19 20" >> $TMPFILE.cmd
+echo "ex /e" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null f
+sif /f size 1024
+eo /f
+set_bmap --uninit 0 9000
+ec
+sif /f blocks 2
+setb 9000
+fallocate /f 0 8999
+punch /f 1 8998
+ENDL
+echo "ex /f" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w -f $TMPFILE.cmd $TMPFILE > /dev/null 2>&1
+$DEBUGFS_EXE -f $TMPFILE.cmd2 $TMPFILE >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/d_punch_bigalloc/expect b/tests/d_punch_bigalloc/expect
new file mode 100644
index 0000000..21427d5
--- /dev/null
+++ b/tests/d_punch_bigalloc/expect
@@ -0,0 +1,207 @@
+
+Warning: the bigalloc feature is still under development
+See https://ext4.wiki.kernel.org/index.php/Bigalloc for more information
+
+Creating filesystem with 65536 1k blocks and 4096 inodes
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/4096 files (9.1% non-contiguous), 1144/65536 blocks
+Exit status is 0
+debugfs write files
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+debugfs: ex /sample
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1184              0
+ 1/ 1   1/  5     0 -     9  1144 -  1153     10 Uninit
+ 1/ 1   2/  5    11 -    12  1155 -  1156      2 Uninit
+ 1/ 1   3/  5    14 -    25  1158 -  1169     12 Uninit
+ 1/ 1   4/  5    27 -    28  1171 -  1172      2 Uninit
+ 1/ 1   5/  5    30 -    39  1174 -  1183     10 Uninit
+debugfs: ex /b8
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1232              0
+ 1/ 1   1/  1     0 -     7  1192 -  1199      8 Uninit
+debugfs: ex /b9
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1248              0
+ 1/ 1   1/  1     0 -     8  1200 -  1208      9 Uninit
+debugfs: ex /b10
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1272              0
+ 1/ 1   1/  1     0 -     9  1216 -  1225     10 Uninit
+debugfs: ex /b11
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1296              0
+ 1/ 1   1/  2     0 -     7  1240 -  1247      8 Uninit
+ 1/ 1   2/  2     8 -     9  1256 -  1257      2 Uninit
+debugfs: ex /b12
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1320              0
+ 1/ 1   1/  3     0 -     7  1264 -  1271      8 Uninit
+ 1/ 1   2/  3     8 -     9  1280 -  1281      2 Uninit
+ 1/ 1   3/  3    11 -    11  1283 -  1283      1 Uninit
+debugfs: ex /b13
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1344              0
+ 1/ 1   1/  3     0 -     7  1288 -  1295      8 Uninit
+ 1/ 1   2/  3     8 -     9  1304 -  1305      2 Uninit
+ 1/ 1   3/  3    11 -    12  1307 -  1308      2 Uninit
+debugfs: ex /b14
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1368              0
+ 1/ 1   1/  3     0 -     7  1312 -  1319      8 Uninit
+ 1/ 1   2/  3     8 -     9  1328 -  1329      2 Uninit
+ 1/ 1   3/  3    11 -    12  1331 -  1332      2 Uninit
+debugfs: ex /b15
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1392              0
+ 1/ 1   1/  4     0 -     7  1336 -  1343      8 Uninit
+ 1/ 1   2/  4     8 -     9  1352 -  1353      2 Uninit
+ 1/ 1   3/  4    11 -    12  1355 -  1356      2 Uninit
+ 1/ 1   4/  4    14 -    14  1358 -  1358      1 Uninit
+debugfs: ex /c24
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    25 - 4294967295  1416         4294967271
+ 1/ 1   1/  3    25 -    25  1401 -  1401      1 Uninit
+ 1/ 1   2/  3    27 -    28  1403 -  1404      2 Uninit
+ 1/ 1   3/  3    30 -    39  1406 -  1415     10 Uninit
+debugfs: ex /c25
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1440         4294967269
+ 1/ 1   1/  2    27 -    28  1427 -  1428      2 Uninit
+ 1/ 1   2/  2    30 -    39  1430 -  1439     10 Uninit
+debugfs: ex /c26
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1464         4294967269
+ 1/ 1   1/  2    27 -    28  1451 -  1452      2 Uninit
+ 1/ 1   2/  2    30 -    39  1454 -  1463     10 Uninit
+debugfs: ex /c27
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    28 - 4294967295  1488         4294967268
+ 1/ 1   1/  2    28 -    28  1476 -  1476      1 Uninit
+ 1/ 1   2/  2    30 -    39  1478 -  1487     10 Uninit
+debugfs: ex /c28
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1512         4294967266
+ 1/ 1   1/  1    30 -    39  1502 -  1511     10 Uninit
+debugfs: ex /c29
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1536         4294967266
+ 1/ 1   1/  1    30 -    39  1526 -  1535     10 Uninit
+debugfs: ex /c30
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    31 - 4294967295  1560         4294967265
+ 1/ 1   1/  1    31 -    39  1551 -  1559      9 Uninit
+debugfs: ex /c31
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    32 - 4294967295  1584         4294967264
+ 1/ 1   1/  1    32 -    39  1576 -  1583      8 Uninit
+debugfs: ex /d
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1600              0
+ 1/ 1   1/  2     0 -     3  1360 -  1363      4 Uninit
+ 1/ 1   2/  2    36 -    39  1596 -  1599      4 Uninit
+debugfs: ex /e
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1624              0
+ 1/ 1   1/  8     0 -     9  1376 -  1385     10 Uninit
+ 1/ 1   2/  8    11 -    12  1387 -  1388      2 Uninit
+ 1/ 1   3/  8    14 -    15  1390 -  1391      2 Uninit
+ 1/ 1   4/  8    16 -    18  1568 -  1570      3 Uninit
+ 1/ 1   5/  8    21 -    23  1573 -  1575      3 Uninit
+ 1/ 1   6/  8    24 -    25  1608 -  1609      2 Uninit
+ 1/ 1   7/  8    27 -    28  1611 -  1612      2 Uninit
+ 1/ 1   8/  8    30 -    39  1614 -  1623     10 Uninit
+debugfs: ex /f
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  2     0 -     0  9000 -  9000      1 Uninit
+ 0/ 0   2/  2  8999 -  8999 17999 - 17999      1 Uninit
+Pass 1: Checking inodes, blocks, and sizes
+Inode 14 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 15 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 16 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 17 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 18 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 19 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 20 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 22 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 23 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 24 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 25 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 26 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 27 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 28 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 29 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Inode 30 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
+Pass 1E: Optimizing extent trees
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong for group #0 (8003, counted=8002).
+Fix? yes
+
+Free blocks count wrong (64024, counted=64016).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 32/4096 files (43.8% non-contiguous), 1520/65536 blocks
+Exit status is 1
diff --git a/tests/d_punch_bigalloc/name b/tests/d_punch_bigalloc/name
new file mode 100644
index 0000000..6d61ebe
--- /dev/null
+++ b/tests/d_punch_bigalloc/name
@@ -0,0 +1 @@
+punch sparse files and big files with bigalloc
diff --git a/tests/d_punch_bigalloc/script b/tests/d_punch_bigalloc/script
new file mode 100644
index 0000000..6eb0571
--- /dev/null
+++ b/tests/d_punch_bigalloc/script
@@ -0,0 +1,130 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP1
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+	cluster_size = 8192
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+make_file() {
+	name="$1"
+	start="$2"
+	flag="$3"
+
+	cat << ENDL
+write /dev/null $name
+fallocate /$name 0 39
+punch /$name 10 10
+punch /$name 13 13
+punch /$name 26 26
+punch /$name 29 29
+ENDL
+}
+
+#Files we create:
+# a: punch a 40k file
+# b*: punch sparse file starting at b*
+# c*: punch spare file ending at c*
+# d: midcluster to midcluster, surrounding sparse
+# e: partial middle cluster alloc
+# f: one big file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+fallocate /a 0 39
+punch /a 0 39
+ENDL
+echo "ex /a" >> $TMPFILE.cmd2
+
+make_file sample $base --uninit >> $TMPFILE.cmd
+echo "ex /sample" >> $TMPFILE.cmd2
+base=10000
+
+for i in 8 9 10 11 12 13 14 15; do
+	make_file b$i $(($base + (40 * ($i - 8)))) --uninit >> $TMPFILE.cmd
+	echo "punch /b$i $i 39" >> $TMPFILE.cmd
+	echo "ex /b$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file c$i $(($base + 320 + (40 * ($i - 24)))) --uninit >> $TMPFILE.cmd
+	echo "punch /c$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /c$i" >> $TMPFILE.cmd2
+done
+
+make_file d $(($base + 640)) --uninit >> $TMPFILE.cmd
+echo "punch /d 4 35" >> $TMPFILE.cmd
+echo "ex /d" >> $TMPFILE.cmd2
+
+make_file e $(($base + 680)) --uninit >> $TMPFILE.cmd
+echo "punch /e 19 20" >> $TMPFILE.cmd
+echo "ex /e" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null f
+sif /f size 1024
+eo /f
+set_bmap --uninit 0 9000
+ec
+sif /f blocks 16
+setb 9000
+fallocate /f 0 8999
+punch /f 1 8998
+ENDL
+echo "ex /f" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w -f $TMPFILE.cmd $TMPFILE > /dev/null 2>&1
+$DEBUGFS_EXE -f $TMPFILE.cmd2 $TMPFILE >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

Re: [PATCH 01/54] misc: fix minor testcase problems

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:35:40PM -0800, Darrick J. Wong wrote:
> Don't write debugfs headers to stdout...
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

					- Ted

Re: [PATCH 02/54] debugfs: document new commands

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:35:46PM -0800, Darrick J. Wong wrote:
> Document the new journal and xattr commands in the debugfs manpage.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

					- Ted

Re: [PATCH 03/54] debugfs: fix crash in ea_set argument handling

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:35:53PM -0800, Darrick J. Wong wrote:
> Fix an incorrect check in ea_set that would crash debugfs if someone
> runs 'ea_set / foo.bar' (i.e. with no value argument)
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

					- Ted

Re: [PATCH 04/54] libext2fs: initialize i_extra_isize when writing EAs

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:35:59PM -0800, Darrick J. Wong wrote:
> If i_extra_isize is zero when we try to write extended attributes,
> we'll end up writing the EA magic into the i_extra_isize field, which
> causes a subsequent crash on big endian systems (when we try to write
> 0xEA02 bytes past the inode!).  Therefore when the field is zero, set
> i_extra_isize to the desired extra_isize size, zero those bytes, and
> write the EAs after the end of the extended inode.
> 
> v2: Don't bother if we have 128b inodes, and ensure that the value
> is 32b-aligned so that the EA magic starts on a 32b boundary.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Thanks, applied.

						- Ted

Re: [PATCH 05/54] libext2fs: avoid pointless EA block allocation

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:06PM -0800, Darrick J. Wong wrote:
> Use qsort to move the inlinedata attribute to the front of the list
> and the empty entries to the end.  Then we can use handle->count to
> decide if we're done writing xattrs, which helps us to avoid the
> situation where we're midway through the attribute list, so we
> allocate an EA block to store more, but have no idea that there's
> actually nothing left in the list.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, although I wonder if we might want to be a bit more
sophisticated about how we handle the order of attributes in the
future.  In particular, if we need to spill over into an EA block,
there are certain things like the selinux security id that we would
want to be inline in the inode, and let other xattrs spill over into
the EA block.  And of course, if we can manage to fit all of the
xattrs into the inode, except for system.data, then we should bail on
using the inline data feature at all, and just allocate a normal data
block for the data, and not use an EA block for system.data at all.

It's much more imporant that we get this sort of stuff right for the
kernel implementation, though, and if we occasionally misoptimize
things in libext2fs, that's unfortunate, but it's primarily going to
be a problem for the FUSE driver...

					- Ted

Re: [PATCH 06/54] libext2fs: strengthen i_extra_isize checks when reading/writing xattrs

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:12PM -0800, Darrick J. Wong wrote:
> Strengthen the i_extra_isize checks to look for obviously too-small
> values before trying to operate on inode EAs.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

					- Ted

Re: [PATCH 07/54] libext2fs: fix tdb.c mmap leak

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:18PM -0800, Darrick J. Wong wrote:
> When undoing an expansion of an mmap'd database while cancelling a
> transaction, the tdb code prematurely decreases the variable that
> tracks the file size, which leads to a region leak during the
> subsequent unmap.  Fix this by maintaining a separate counter for the
> region size.
> 
> (This is probably unnecessary since e2undo was the only user of tdb
> transactions, but I suppose we could be proactive.)
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 08/54] resize2fs: fix regression test to not depend on ext4.ko being loaded

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:25PM -0800, Darrick J. Wong wrote:
> The behavior of the r_fixup_lastbg_big test varies depending on
> whether or not ext4.ko is loaded and supports lazy_itable_init.  This
> makes checking the bg flags after resize2fs hard to predict, so put in
> a way to force resize2fs to zero the inode tables, and compare the
> output based on lazy_itable_init == 0.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Thanks, applied.

					- Ted

Re: [PATCH 09/54] tune2fs: disable csum verification before resizing inode

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:31PM -0800, Darrick J. Wong wrote:
> When we're turning on metadata checksumming /and/ resizing the inode
> at the same time, disable checksum verification during the
> resize_inode() call because the subroutines it calls will try to
> verify the checksums (which have not yet been set), causing the
> operation to fail unnecessarily.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 10/54] tune2fs: abort when trying to enable/disable metadata_csum on mounted fs

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:38PM -0800, Darrick J. Wong wrote:
> Earlier, I tried to make tune2fs abort if the user tried to enable or
> disable metadata_csum on a mounted FS, but forgot the exit() call.
> Supply it now.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 11/54] tune2fs: call out to resize2fs for 64bit conversion

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:44PM -0800, Darrick J. Wong wrote:
> If the user tries to enable or disable the 64bit feature via tune2fs,
> tell them how to use resize2fs to effect the conversion.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, although I changed to patch summary to read "direct user to
resize2fs..."

Thanks!!

						- Ted

Re: [PATCH 12/54] e2fsck: clear i_block[] when there are too many bad mappings on a special inode

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:50PM -0800, Darrick J. Wong wrote:
> If we decide to clear a special inode because of bad mappings, we need
> to zero the i_block array.  The clearing routine depends on setting
> i_links_count to zero to keep us from re-checking the block maps,
> but that field isn't checked for special inodes.  Therefore, if we
> haven't erased the mappings, check_blocks will restart fsck and fsck
> will try to check the blocks again, leading to an infinite loop.
> 
> (This seems easy to trigger if the bootloader inode extent map is
> corrupted.)
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 13/54] e2fsck: on read error, don't rewrite blocks past the end of the fs

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:36:57PM -0800, Darrick J. Wong wrote:
> If e2fsck encounters a read error on a block past the end of the
> filesystem, don't bother trying to "rewrite" the block.  We might
> still want to re-try the read to capture FS data marooned past the end
> of the filesystem, but in that case e2fsck ought to move the block
> back inside the filesystem.
> 
> This enables e2fuzz to detect writes past the end of the FS due to
> software bugs.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, but if we have a block which is past the end of the file
system, why did we try to read it in the first place?

Was it because e2fsck was told to ignore a file system inconsistency?
Or because e2fsck didn't notice before trying to read a block?  If so,
we should look to see if there is some way we could have caught this
error in the first place.  So it might be useful to run e2fuzz and
call abort() if we try to read past the end of the file system, so we
can get a stack dump and understand what happened....

    	  	     	 	    	 - Ted

Re: [PATCH 14/54] e2fsck: fix the journal recreation message

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:03PM -0800, Darrick J. Wong wrote:
> When we recreate the journal, don't say that the FS "is now ext3
> again", since we could be fixing a damaged ext4 FS journal, which does
> not magically convert the FS back to ext3.
> 
> Fix the po files too, though this string hasn't been translated.
> 

So a couple of things.  "Journaled" is spelled with a single 'l':

   http://www.oxforddictionaries.com/definition/english/journal

Also, if you're going to fix the journal recreation message, we should
also fix the message which is generated when the journal is removed.

Finally, don't mess with the po files; just before we do a release,
we'll regenerate the po template file (e2fsprogs.pot), and there are
automated tools that will take care of merging the changes into the
.po files.  It's cleaner if we don't modify the po files except via
automated tools, when we get updates from the Translation Project:

	  http://translationproject.org/html/maintainers.html

Here's the patch that I've applied instead.  I ended up simplifying
the message because "journal has been re-created - filesystem is
journaled again" sounds a like an entry from

http://tvtropes.org/pmwiki/pmwiki.php/Main/DepartmentOfRedundancyDepartment

:-)

					- Ted


commit 85068f9d86ab70da6d27384210f1c964f8c09f79
Author: Darrick J. Wong <darrick.wong@oracle.com>
Date:   Mon Jan 26 23:37:03 2015 -0800

    e2fsck: fix message when the journal is deleted and regenerated
    
    When we recreate the journal, don't say that the FS "is now ext3
    again", since we could be fixing a damaged ext4 FS journal, which does
    not magically convert the FS back to ext3.
    
    [ Use "journaled" instead of "journalled", and also fix the message we
      print when deleting the journal --Ted ]
    
    Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
    Signed-off-by: Theodore Ts'o <tytso@mit.edu>

diff --git a/e2fsck/journal.c b/e2fsck/journal.c
index 6119cda..9f32095 100644
--- a/e2fsck/journal.c
+++ b/e2fsck/journal.c
@@ -545,8 +545,7 @@ static errcode_t e2fsck_journal_fix_bad_inode(e2fsck_t ctx,
 		pctx->ino = sb->s_journal_inum;
 		if (fix_problem(ctx, PR_0_JOURNAL_BAD_INODE, pctx)) {
 			if (has_journal && sb->s_journal_inum)
-				printf("*** ext3 journal has been deleted - "
-				       "filesystem is now ext2 only ***\n\n");
+				printf("*** journal has been deleted ***\n\n");
 			sb->s_feature_compat &= ~EXT3_FEATURE_COMPAT_HAS_JOURNAL;
 			sb->s_journal_inum = 0;
 			memset(sb->s_jnl_blocks, 0, sizeof(sb->s_jnl_blocks));
diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 615d690..96551a1 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -1674,8 +1674,7 @@ print_unsupp_features:
 			}
 			log_out(ctx, "%s", _(" Done.\n"));
 			log_out(ctx, "%s",
-				_("\n*** journal has been re-created - "
-				  "filesystem is now ext3 again ***\n"));
+				_("\n*** journal has been regenerated ***\n"));
 		}
 	}
 no_journal:
diff --git a/tests/f_badjour_indblks/expect.1 b/tests/f_badjour_indblks/expect.1
index 7ccc59b..cb8054c 100644
--- a/tests/f_badjour_indblks/expect.1
+++ b/tests/f_badjour_indblks/expect.1
@@ -1,7 +1,7 @@
 Superblock has an invalid journal (inode 8).
 Clear? yes
 
-*** ext3 journal has been deleted - filesystem is now ext2 only ***
+*** journal has been deleted ***
 
 Adding dirhash hint to filesystem.
 
@@ -25,7 +25,7 @@ Recreate journal? yes
 
 Creating journal (1024 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been regenerated ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/256 files (0.0% non-contiguous), 1111/8192 blocks
diff --git a/tests/f_badjourblks/expect.1 b/tests/f_badjourblks/expect.1
index 34c6658..5483a0d 100644
--- a/tests/f_badjourblks/expect.1
+++ b/tests/f_badjourblks/expect.1
@@ -1,7 +1,7 @@
 Superblock has an invalid journal (inode 8).
 Clear? yes
 
-*** ext3 journal has been deleted - filesystem is now ext2 only ***
+*** journal has been deleted ***
 
 Pass 1: Checking inodes, blocks, and sizes
 Journal inode is not in use, but contains data.  Clear? yes
@@ -23,7 +23,7 @@ Recreate journal? yes
 
 Creating journal (1024 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been regenerated ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/256 files (0.0% non-contiguous), 1079/8192 blocks
diff --git a/tests/f_jnl_etb_alloc_fail/expect.1 b/tests/f_jnl_etb_alloc_fail/expect.1
index 8f4ebbb..243a151 100644
--- a/tests/f_jnl_etb_alloc_fail/expect.1
+++ b/tests/f_jnl_etb_alloc_fail/expect.1
@@ -1,7 +1,7 @@
 Superblock has an invalid journal (inode 8).
 Clear? yes
 
-*** ext3 journal has been deleted - filesystem is now ext2 only ***
+*** journal has been deleted ***
 
 Superblock has_journal flag is clear, but a journal is present.
 Clear? yes
diff --git a/tests/f_miss_journal/expect.1 b/tests/f_miss_journal/expect.1
index 6ec8b38..d435952 100644
--- a/tests/f_miss_journal/expect.1
+++ b/tests/f_miss_journal/expect.1
@@ -1,7 +1,7 @@
 Superblock has an invalid journal (inode 8).
 Clear? yes
 
-*** ext3 journal has been deleted - filesystem is now ext2 only ***
+*** journal has been deleted ***
 
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
@@ -21,7 +21,7 @@ Recreate journal? yes
 
 Creating journal (1024 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been regenerated ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/256 files (0.0% non-contiguous), 1079/2048 blocks
diff --git a/tests/j_corrupt_sb_magic/expect b/tests/j_corrupt_sb_magic/expect
index 2169a15..d1dae88 100644
--- a/tests/j_corrupt_sb_magic/expect
+++ b/tests/j_corrupt_sb_magic/expect
@@ -1,7 +1,7 @@
 Superblock has an invalid journal (inode 8).
 Clear? yes
 
-*** ext3 journal has been deleted - filesystem is now ext2 only ***
+*** journal has been deleted ***
 
 Superblock has_journal flag is clear, but a journal is present.
 Clear? yes
@@ -26,7 +26,7 @@ Recreate journal? yes
 
 Creating journal (1024 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been regenerated ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 12/128 files (0.0% non-contiguous), 1092/2048 blocks
diff --git a/tests/j_long_trans/expect b/tests/j_long_trans/expect
index 7638ef1..7a17541 100644
--- a/tests/j_long_trans/expect
+++ b/tests/j_long_trans/expect
@@ -57,7 +57,7 @@ test_filesys: recovering journal
 Superblock has an invalid journal (inode 8).
 Clear? yes
 
-*** ext3 journal has been deleted - filesystem is now ext2 only ***
+*** journal has been deleted ***
 
 Resize inode not valid.  Recreate? yes
 
@@ -100,7 +100,7 @@ Recreate journal? yes
 
 Creating journal (8192 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been regenerated ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/16384 files (0.0% non-contiguous), 14420/262144 blocks
diff --git a/tests/j_long_trans_mcsum_32bit/expect b/tests/j_long_trans_mcsum_32bit/expect
index 0d141c1..a808d9f 100644
--- a/tests/j_long_trans_mcsum_32bit/expect
+++ b/tests/j_long_trans_mcsum_32bit/expect
@@ -93,7 +93,7 @@ test_filesys: recovering journal
 Superblock has an invalid journal (inode 8).
 Clear? yes
 
-*** ext3 journal has been deleted - filesystem is now ext2 only ***
+*** journal has been deleted ***
 
 Resize inode not valid.  Recreate? yes
 
@@ -139,7 +139,7 @@ Recreate journal? yes
 
 Creating journal (16384 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been regenerated ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/32768 files (0.0% non-contiguous), 27039/524288 blocks
diff --git a/tests/j_long_trans_mcsum_64bit/expect b/tests/j_long_trans_mcsum_64bit/expect
index 94e9925..76e109a 100644
--- a/tests/j_long_trans_mcsum_64bit/expect
+++ b/tests/j_long_trans_mcsum_64bit/expect
@@ -92,7 +92,7 @@ test_filesys: recovering journal
 Superblock has an invalid journal (inode 8).
 Clear? yes
 
-*** ext3 journal has been deleted - filesystem is now ext2 only ***
+*** journal has been deleted ***
 
 Resize inode not valid.  Recreate? yes
 
@@ -138,7 +138,7 @@ Recreate journal? yes
 
 Creating journal (16384 blocks):  Done.
 
-*** journal has been re-created - filesystem is now ext3 again ***
+*** journal has been regenerated ***
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/32768 files (0.0% non-contiguous), 27057/524288 blocks

Re: [PATCH 05/54] libext2fs: avoid pointless EA block allocation

[Darrick J. Wong]

On Tue, Jan 27, 2015 at 11:07:36AM -0500, Theodore Ts'o wrote:
> On Mon, Jan 26, 2015 at 11:36:06PM -0800, Darrick J. Wong wrote:
> > Use qsort to move the inlinedata attribute to the front of the list
> > and the empty entries to the end.  Then we can use handle->count to
> > decide if we're done writing xattrs, which helps us to avoid the
> > situation where we're midway through the attribute list, so we
> > allocate an EA block to store more, but have no idea that there's
> > actually nothing left in the list.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Applied, although I wonder if we might want to be a bit more
> sophisticated about how we handle the order of attributes in the
> future.  In particular, if we need to spill over into an EA block,
> there are certain things like the selinux security id that we would

Agreed.

> want to be inline in the inode, and let other xattrs spill over into
> the EA block.  And of course, if we can manage to fit all of the
> xattrs into the inode, except for system.data, then we should bail on
> using the inline data feature at all, and just allocate a normal data
> block for the data, and not use an EA block for system.data at all.

I've been wondering for a while why there isn't a name index for
system.data -- doing that would allow 4 more bytes of inline data per
inode.

> It's much more imporant that we get this sort of stuff right for the
> kernel implementation, though, and if we occasionally misoptimize
> things in libext2fs, that's unfortunate, but it's primarily going to
> be a problem for the FUSE driver...

<shrug> That might (for now) be less of a problem for fuse, since
libext2fs has inline directories spill over to a block as soon as
i_block[] fills up... haven't decided if that's a "feature" or a "bug".

(The kernel implementation seems ok with my not-rigorous 3.19 spot
check.)

--D
> 
> 					- Ted

Re: [PATCH 14/54] e2fsck: fix the journal recreation message

[Darrick J. Wong]

On Tue, Jan 27, 2015 at 01:02:54PM -0500, Theodore Ts'o wrote:
> On Mon, Jan 26, 2015 at 11:37:03PM -0800, Darrick J. Wong wrote:
> > When we recreate the journal, don't say that the FS "is now ext3
> > again", since we could be fixing a damaged ext4 FS journal, which does
> > not magically convert the FS back to ext3.
> > 
> > Fix the po files too, though this string hasn't been translated.
> > 
> 
> So a couple of things.  "Journaled" is spelled with a single 'l':
> 
>    http://www.oxforddictionaries.com/definition/english/journal
> 
> Also, if you're going to fix the journal recreation message, we should
> also fix the message which is generated when the journal is removed.
> 
> Finally, don't mess with the po files; just before we do a release,
> we'll regenerate the po template file (e2fsprogs.pot), and there are
> automated tools that will take care of merging the changes into the
> .po files.  It's cleaner if we don't modify the po files except via
> automated tools, when we get updates from the Translation Project:
> 
> 	  http://translationproject.org/html/maintainers.html
> 
> Here's the patch that I've applied instead.  I ended up simplifying
> the message because "journal has been re-created - filesystem is
> journaled again" sounds a like an entry from
> 
> http://tvtropes.org/pmwiki/pmwiki.php/Main/DepartmentOfRedundancyDepartment
> 
> :-)

All good points; thank you for making the corrections.

I'll find a place (SUBMITTING-PATCHES?) for leaving a note to leave the po
files alone, since I wasn't aware of the translation automation.

--D
> 
> 					- Ted
> 
> 
> commit 85068f9d86ab70da6d27384210f1c964f8c09f79
> Author: Darrick J. Wong <darrick.wong@oracle.com>
> Date:   Mon Jan 26 23:37:03 2015 -0800
> 
>     e2fsck: fix message when the journal is deleted and regenerated
>     
>     When we recreate the journal, don't say that the FS "is now ext3
>     again", since we could be fixing a damaged ext4 FS journal, which does
>     not magically convert the FS back to ext3.
>     
>     [ Use "journaled" instead of "journalled", and also fix the message we
>       print when deleting the journal --Ted ]
>     
>     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
>     Signed-off-by: Theodore Ts'o <tytso@mit.edu>
> 
> diff --git a/e2fsck/journal.c b/e2fsck/journal.c
> index 6119cda..9f32095 100644
> --- a/e2fsck/journal.c
> +++ b/e2fsck/journal.c
> @@ -545,8 +545,7 @@ static errcode_t e2fsck_journal_fix_bad_inode(e2fsck_t ctx,
>  		pctx->ino = sb->s_journal_inum;
>  		if (fix_problem(ctx, PR_0_JOURNAL_BAD_INODE, pctx)) {
>  			if (has_journal && sb->s_journal_inum)
> -				printf("*** ext3 journal has been deleted - "
> -				       "filesystem is now ext2 only ***\n\n");
> +				printf("*** journal has been deleted ***\n\n");
>  			sb->s_feature_compat &= ~EXT3_FEATURE_COMPAT_HAS_JOURNAL;
>  			sb->s_journal_inum = 0;
>  			memset(sb->s_jnl_blocks, 0, sizeof(sb->s_jnl_blocks));
> diff --git a/e2fsck/unix.c b/e2fsck/unix.c
> index 615d690..96551a1 100644
> --- a/e2fsck/unix.c
> +++ b/e2fsck/unix.c
> @@ -1674,8 +1674,7 @@ print_unsupp_features:
>  			}
>  			log_out(ctx, "%s", _(" Done.\n"));
>  			log_out(ctx, "%s",
> -				_("\n*** journal has been re-created - "
> -				  "filesystem is now ext3 again ***\n"));
> +				_("\n*** journal has been regenerated ***\n"));
>  		}
>  	}
>  no_journal:
> diff --git a/tests/f_badjour_indblks/expect.1 b/tests/f_badjour_indblks/expect.1
> index 7ccc59b..cb8054c 100644
> --- a/tests/f_badjour_indblks/expect.1
> +++ b/tests/f_badjour_indblks/expect.1
> @@ -1,7 +1,7 @@
>  Superblock has an invalid journal (inode 8).
>  Clear? yes
>  
> -*** ext3 journal has been deleted - filesystem is now ext2 only ***
> +*** journal has been deleted ***
>  
>  Adding dirhash hint to filesystem.
>  
> @@ -25,7 +25,7 @@ Recreate journal? yes
>  
>  Creating journal (1024 blocks):  Done.
>  
> -*** journal has been re-created - filesystem is now ext3 again ***
> +*** journal has been regenerated ***
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/256 files (0.0% non-contiguous), 1111/8192 blocks
> diff --git a/tests/f_badjourblks/expect.1 b/tests/f_badjourblks/expect.1
> index 34c6658..5483a0d 100644
> --- a/tests/f_badjourblks/expect.1
> +++ b/tests/f_badjourblks/expect.1
> @@ -1,7 +1,7 @@
>  Superblock has an invalid journal (inode 8).
>  Clear? yes
>  
> -*** ext3 journal has been deleted - filesystem is now ext2 only ***
> +*** journal has been deleted ***
>  
>  Pass 1: Checking inodes, blocks, and sizes
>  Journal inode is not in use, but contains data.  Clear? yes
> @@ -23,7 +23,7 @@ Recreate journal? yes
>  
>  Creating journal (1024 blocks):  Done.
>  
> -*** journal has been re-created - filesystem is now ext3 again ***
> +*** journal has been regenerated ***
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/256 files (0.0% non-contiguous), 1079/8192 blocks
> diff --git a/tests/f_jnl_etb_alloc_fail/expect.1 b/tests/f_jnl_etb_alloc_fail/expect.1
> index 8f4ebbb..243a151 100644
> --- a/tests/f_jnl_etb_alloc_fail/expect.1
> +++ b/tests/f_jnl_etb_alloc_fail/expect.1
> @@ -1,7 +1,7 @@
>  Superblock has an invalid journal (inode 8).
>  Clear? yes
>  
> -*** ext3 journal has been deleted - filesystem is now ext2 only ***
> +*** journal has been deleted ***
>  
>  Superblock has_journal flag is clear, but a journal is present.
>  Clear? yes
> diff --git a/tests/f_miss_journal/expect.1 b/tests/f_miss_journal/expect.1
> index 6ec8b38..d435952 100644
> --- a/tests/f_miss_journal/expect.1
> +++ b/tests/f_miss_journal/expect.1
> @@ -1,7 +1,7 @@
>  Superblock has an invalid journal (inode 8).
>  Clear? yes
>  
> -*** ext3 journal has been deleted - filesystem is now ext2 only ***
> +*** journal has been deleted ***
>  
>  Pass 1: Checking inodes, blocks, and sizes
>  Pass 2: Checking directory structure
> @@ -21,7 +21,7 @@ Recreate journal? yes
>  
>  Creating journal (1024 blocks):  Done.
>  
> -*** journal has been re-created - filesystem is now ext3 again ***
> +*** journal has been regenerated ***
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/256 files (0.0% non-contiguous), 1079/2048 blocks
> diff --git a/tests/j_corrupt_sb_magic/expect b/tests/j_corrupt_sb_magic/expect
> index 2169a15..d1dae88 100644
> --- a/tests/j_corrupt_sb_magic/expect
> +++ b/tests/j_corrupt_sb_magic/expect
> @@ -1,7 +1,7 @@
>  Superblock has an invalid journal (inode 8).
>  Clear? yes
>  
> -*** ext3 journal has been deleted - filesystem is now ext2 only ***
> +*** journal has been deleted ***
>  
>  Superblock has_journal flag is clear, but a journal is present.
>  Clear? yes
> @@ -26,7 +26,7 @@ Recreate journal? yes
>  
>  Creating journal (1024 blocks):  Done.
>  
> -*** journal has been re-created - filesystem is now ext3 again ***
> +*** journal has been regenerated ***
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 12/128 files (0.0% non-contiguous), 1092/2048 blocks
> diff --git a/tests/j_long_trans/expect b/tests/j_long_trans/expect
> index 7638ef1..7a17541 100644
> --- a/tests/j_long_trans/expect
> +++ b/tests/j_long_trans/expect
> @@ -57,7 +57,7 @@ test_filesys: recovering journal
>  Superblock has an invalid journal (inode 8).
>  Clear? yes
>  
> -*** ext3 journal has been deleted - filesystem is now ext2 only ***
> +*** journal has been deleted ***
>  
>  Resize inode not valid.  Recreate? yes
>  
> @@ -100,7 +100,7 @@ Recreate journal? yes
>  
>  Creating journal (8192 blocks):  Done.
>  
> -*** journal has been re-created - filesystem is now ext3 again ***
> +*** journal has been regenerated ***
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/16384 files (0.0% non-contiguous), 14420/262144 blocks
> diff --git a/tests/j_long_trans_mcsum_32bit/expect b/tests/j_long_trans_mcsum_32bit/expect
> index 0d141c1..a808d9f 100644
> --- a/tests/j_long_trans_mcsum_32bit/expect
> +++ b/tests/j_long_trans_mcsum_32bit/expect
> @@ -93,7 +93,7 @@ test_filesys: recovering journal
>  Superblock has an invalid journal (inode 8).
>  Clear? yes
>  
> -*** ext3 journal has been deleted - filesystem is now ext2 only ***
> +*** journal has been deleted ***
>  
>  Resize inode not valid.  Recreate? yes
>  
> @@ -139,7 +139,7 @@ Recreate journal? yes
>  
>  Creating journal (16384 blocks):  Done.
>  
> -*** journal has been re-created - filesystem is now ext3 again ***
> +*** journal has been regenerated ***
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/32768 files (0.0% non-contiguous), 27039/524288 blocks
> diff --git a/tests/j_long_trans_mcsum_64bit/expect b/tests/j_long_trans_mcsum_64bit/expect
> index 94e9925..76e109a 100644
> --- a/tests/j_long_trans_mcsum_64bit/expect
> +++ b/tests/j_long_trans_mcsum_64bit/expect
> @@ -92,7 +92,7 @@ test_filesys: recovering journal
>  Superblock has an invalid journal (inode 8).
>  Clear? yes
>  
> -*** ext3 journal has been deleted - filesystem is now ext2 only ***
> +*** journal has been deleted ***
>  
>  Resize inode not valid.  Recreate? yes
>  
> @@ -138,7 +138,7 @@ Recreate journal? yes
>  
>  Creating journal (16384 blocks):  Done.
>  
> -*** journal has been re-created - filesystem is now ext3 again ***
> +*** journal has been regenerated ***
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/32768 files (0.0% non-contiguous), 27057/524288 blocks

Re: [PATCH 15/54] e2fsck: handle multiple *ind block collisions with critical metadata

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:10PM -0800, Darrick J. Wong wrote:
> An earlier patch tried to detect indirect blocks that conflicted with
> critical FS metadata for the purpose of preventing corrections being
> made to those indirect blocks.  Unfortunately, that patch cannot
> handle more than one conflicting *ind block per file; therefore, use
> the ref_block parameter to test the metadata block map to decide if
> we need to avoid fixing the *ind block when we're iterating the
> block's entries.  (We have to iterate the block to capture any blocks
> that the block points to, as they could be in use.)
> 
> As a side note, in 1B we'll reallocate all those conflicting *ind
> blocks and restart fsck, so the contents will be checked eventually.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 16/54] e2fsck: decrement bad count _after_ remapping a duplicate block

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:18PM -0800, Darrick J. Wong wrote:
> Decrement the bad count *after* we've shown that (a) we can allocate a
> replacement block and (b) remap the file block.  Unfortunately,
> the only way to tell if the remapping succeeded is to wait until the
> next clone_file_block() call or block_iterate3() returns.
> 
> Otherwise, there's a corruption error: we decrease the badcount once in
> preparation to remap, then the remap fails (either we can't find a
> replacement block or we have to split the extent tree and can't find a
> new extent block), so we delete the file, which decreases the badcount
> on the block a second time.  Later on e2fsck will think that it's
> straightened out all the duplicate blocks, which isn't true.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 17/54] e2fsck: inspect inline dir data as two directory blocks

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:24PM -0800, Darrick J. Wong wrote:
> The design of inline directories (apparently) calls for the i_block[]
> region and the EA regions to be treated as if they were two separate
> blocks of dirents.  Effectively this means that it is impossible for a
> directory entry to straddle both areas.  e2fsck doesn't enforce this,
> so teach it to do so.  e2fslib already knows to do this....
> 
> Cc: Zheng Liu <gnehzuil.liu@gmail.com>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 18/54] e2fsck: improve the inline directory detector

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:31PM -0800, Darrick J. Wong wrote:
> Strengthen the checks that guess if the inode we're looking at is an
> inline directory.  The current check sweeps up any inline inode if
> its length is a multiple of four; now we'll at least try to see if
> there's the beginning of a valid directory entry.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 13/54] e2fsck: on read error, don't rewrite blocks past the end of the fs

[Darrick J. Wong]

On Tue, Jan 27, 2015 at 12:35:07PM -0500, Theodore Ts'o wrote:
> On Mon, Jan 26, 2015 at 11:36:57PM -0800, Darrick J. Wong wrote:
> > If e2fsck encounters a read error on a block past the end of the
> > filesystem, don't bother trying to "rewrite" the block.  We might
> > still want to re-try the read to capture FS data marooned past the end
> > of the filesystem, but in that case e2fsck ought to move the block
> > back inside the filesystem.
> > 
> > This enables e2fuzz to detect writes past the end of the FS due to
> > software bugs.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Applied, but if we have a block which is past the end of the file
> system, why did we try to read it in the first place?
> 
> Was it because e2fsck was told to ignore a file system inconsistency?
> Or because e2fsck didn't notice before trying to read a block?  If so,
> we should look to see if there is some way we could have caught this
> error in the first place.  So it might be useful to run e2fuzz and
> call abort() if we try to read past the end of the file system, so we
> can get a stack dump and understand what happened....

Ok, so I went a little ways down the path to see what I could see:

print_pathname() will run right into it if we try to display the path
name and one of the directories on that path happens to have a bad
directory block pointer, and we haven't yet gotten to fixing the bad
directory block.  Read errors simply make it print "???", so there's
no need to abort().

The journal code will try to replay the journal without checking block
boundaries; returning an error (or a zeroed block with no journal
magic) aborts journal replay like you'd expect.  No need to abort()
the whole program here either.

check_is_really_dir() tries to read a (potentially badly corrupt) file
to guess if it's a directory before we scan the extent tree for
invalid blocks.  Hitting a read error here isn't the end of the world
either.

pass1[b-d] is where things really get tricky -- if pass1 detects that
an extent block collides with critical metadata, it'll put the ETB on
the list of blocks to clone, refuse to touch any of the data
inside that ETB, and restart fsck from the beginning once all the
critical colliding blocks have been reassigned.  This is to avoid
corrupting bitmap/inode/group data.  However, in iterating the
unchecked ETB during that first run through pass 1b, it's possible for
the block iteration call to try to read garbage contents.

There's possibly more, but so far I've not encountered any scenarios
where fsck actually does the wrong thing as a result of reading past
the end of the FS.  It seems that e2fsck will fix pointers to data
blocks outside the filesystem.  Granted, it "fixes" things by erasing
them... but so far, I haven't found anything needing a code fix.

--D

> 
>     	  	     	 	    	 - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 19/54] e2fsck: salvage under-sized dirents by removing them

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:37PM -0800, Darrick J. Wong wrote:
> If the directory processing code ends up pointing to a directory entry
> that's so close to the end of the block that there's not even space
> for a rec_len/name_len, just substitute dummy values that will force
> e2fsck to extend the previous entry to cover the remaining space.  We
> can't use the helper methods to extract rec_len because that's reading
> off the end of the buffer.
> 
> This isn't an issue with non-inline directories because the directory
> check buffer is zero-extended so that fsck won't blow up.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

[PATCH v2 23/54] libext2fs: Support readonly filesystem images

[Darrick J. Wong]

Finish adding the new rocompat feature, "readonly", which marks a
filesystem image read-only.  This also fixes a bug in Ted's patch to
add the feature flag; RO_COMPAT_READONLY needs to be kept out of the
RO_COMPAT supported feature list.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/unix.c           |    7 +++++-
 lib/blkid/probe.h       |    1 +
 lib/ext2fs/ext2fs.h     |   13 ++++++++---
 misc/tune2fs.c          |    2 +-
 resize/main.c           |    8 ++++++
 tests/t_readonly/script |   58 +++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 83 insertions(+), 6 deletions(-)
 create mode 100644 tests/t_readonly/script

diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index e629136..bddb4fa 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -1280,7 +1280,8 @@ restart:
 		if ((ctx->mount_flags & EXT2_MF_READONLY) &&
 		    (ctx->options & E2F_OPT_FORCE))
 			flags &= ~EXT2_FLAG_EXCLUSIVE;
-	}
+	} else
+		flags |= EXT2_FLAG_SOFTSUPP_FEATURES;
 
 	ctx->openfs_flags = flags;
 	retval = try_open_fs(ctx, flags, io_ptr, &fs);
@@ -1539,7 +1540,11 @@ failure:
 	features[1] = sb->s_feature_incompat & ~EXT2_LIB_FEATURE_INCOMPAT_SUPP;
 	features[2] = (sb->s_feature_ro_compat &
 		       ~EXT2_LIB_FEATURE_RO_COMPAT_SUPP);
+	if (ctx->options & E2F_OPT_READONLY)
+		features[2] &= ~EXT4_FEATURE_RO_COMPAT_READONLY;
 print_unsupp_features:
+	if (features[2] & EXT4_FEATURE_RO_COMPAT_READONLY)
+		fatal_error(ctx, _("This is a read-only filesystem."));
 	if (features[0] || features[1] || features[2]) {
 		int	i, j;
 		__u32	*mask = features, m;
diff --git a/lib/blkid/probe.h b/lib/blkid/probe.h
index d6809e1..c578b87 100644
--- a/lib/blkid/probe.h
+++ b/lib/blkid/probe.h
@@ -111,6 +111,7 @@ struct ext2_super_block {
 #define EXT4_FEATURE_RO_COMPAT_EXTRA_ISIZE	0x0040
 #define EXT4_FEATURE_RO_COMPAT_QUOTA		0x0100
 #define EXT4_FEATURE_RO_COMPAT_METADATA_CSUM	0x0400
+#define EXT4_FEATURE_RO_COMPAT_READONLY		0x1000
 
 /* for s_feature_incompat */
 #define EXT2_FEATURE_INCOMPAT_FILETYPE		0x0002
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index d75dd76..cce2d99 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -609,15 +609,22 @@ typedef struct ext2_icount *ext2_icount_t;
 					 EXT4_FEATURE_RO_COMPAT_GDT_CSUM|\
 					 EXT4_FEATURE_RO_COMPAT_BIGALLOC|\
 					 EXT4_LIB_RO_COMPAT_QUOTA|\
-					 EXT4_FEATURE_RO_COMPAT_METADATA_CSUM|\
-					 EXT4_FEATURE_RO_COMPAT_READONLY)
+					 EXT4_FEATURE_RO_COMPAT_METADATA_CSUM)
+/*
+ * N.B. Do not put EXT4_FEATURE_RO_COMPAT_READONLY in this list; that's how
+ * the readonly feature works!
+ */
 
 /*
  * These features are only allowed if EXT2_FLAG_SOFTSUPP_FEATURES is passed
  * to ext2fs_openfs()
+ *
+ * Note that the READONLY feature is integral to debugfs and friends being
+ * able to write to the FS.
  */
 #define EXT2_LIB_SOFTSUPP_INCOMPAT	(0)
-#define EXT2_LIB_SOFTSUPP_RO_COMPAT	(EXT4_FEATURE_RO_COMPAT_REPLICA)
+#define EXT2_LIB_SOFTSUPP_RO_COMPAT	(EXT4_FEATURE_RO_COMPAT_REPLICA | \
+					 EXT4_FEATURE_RO_COMPAT_READONLY)
 
 
 /* Translate a block number to a cluster number */
diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index 550932d..66b2ba3 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -2675,7 +2675,7 @@ retry_open:
 	open_flag |= EXT2_FLAG_64BITS | EXT2_FLAG_JOURNAL_DEV_OK;
 
 	/* keep the filesystem struct around to dump MMP data */
-	open_flag |= EXT2_FLAG_NOFREE_ON_ERROR;
+	open_flag |= EXT2_FLAG_NOFREE_ON_ERROR | EXT2_FLAG_SOFTSUPP_FEATURES;
 
 	retval = ext2fs_open2(device_name, io_options, open_flag,
 			      0, 0, io_ptr, &fs);
diff --git a/resize/main.c b/resize/main.c
index c25de61..5c4aa9f 100644
--- a/resize/main.c
+++ b/resize/main.c
@@ -318,9 +318,15 @@ int main (int argc, char ** argv)
 		io_flags = EXT2_FLAG_RW | EXT2_FLAG_EXCLUSIVE;
 
 	io_flags |= EXT2_FLAG_64BITS;
-
+	io_flags |= EXT2_FLAG_NOFREE_ON_ERROR;
 	retval = ext2fs_open2(device_name, io_options, io_flags,
 			      0, 0, io_ptr, &fs);
+	if (retval == EXT2_ET_RO_UNSUPP_FEATURE &&
+	    EXT2_HAS_RO_COMPAT_FEATURE(fs->super,
+					EXT4_FEATURE_RO_COMPAT_READONLY)) {
+		printf("%s", _("This is a read-only filesystem.\n"));
+		exit(1);
+	}
 	if (retval) {
 		com_err(program_name, retval, _("while trying to open %s"),
 			device_name);
diff --git a/tests/t_readonly/script b/tests/t_readonly/script
new file mode 100644
index 0000000..07bc548
--- /dev/null
+++ b/tests/t_readonly/script
@@ -0,0 +1,58 @@
+test_description="read-only image test"
+if test -x $RESIZE2FS_EXE; then
+
+OUT=$test_name.log
+
+echo read-only image > $OUT
+
+dd if=/dev/zero of=$TMPFILE bs=1k count=512 > /dev/null 2>&1
+crc0=`$CRCSUM $TMPFILE`
+echo $CRCSUM before mke2fs $crc0 >> $OUT
+
+echo mke2fs -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 -z $TDB_FILE.0 $TMPFILE 256 >> $OUT
+$MKE2FS -q -F -o Linux -T ext4 -O ^metadata_csum,64bit -E lazy_itable_init=1 -b 1024 $TMPFILE 256 >> $OUT 2>&1
+crc1=`$CRCSUM $TMPFILE`
+echo $CRCSUM after mke2fs $crc1 >> $OUT
+
+echo enable read-only mode >> $OUT
+$TUNE2FS -O read-only $TMPFILE >> $OUT 2>&1
+crc2=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs -O read-only $crc2 >> $OUT
+
+echo using resize2fs to test read-only >> $OUT
+$RESIZE2FS -s $TMPFILE >> $OUT 2>&1
+crc3=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc3 >> $OUT
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D $TMPFILE >> $OUT 2>&1
+crc4=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc4 >> $OUT
+
+echo disable read-only mode >> $OUT
+$TUNE2FS -O ^read-only $TMPFILE >> $OUT 2>&1
+crc5=`$CRCSUM $TMPFILE`
+echo $CRCSUM after tune2fs -O ^read-only $crc5 >> $OUT
+
+echo using resize2fs to test e2undo >> $OUT
+$RESIZE2FS -s $TMPFILE >> $OUT 2>&1
+crc6=`$CRCSUM $TMPFILE`
+echo $CRCSUM after resize2fs $crc6 >> $OUT
+
+echo using e2fsck to test e2undo >> $OUT
+$FSCK -f -y -D $TMPFILE >> $OUT 2>&1
+crc7=`$CRCSUM $TMPFILE`
+echo $CRCSUM after e2fsck $crc7 >> $OUT
+
+echo $crc0 $crc1 $crc2 $crc3 $crc4 $crc5 $crc6 $crc7 >> $OUT
+
+
+if [ $crc0 != $crc1 ] && [ $crc1 != $crc2 ] && [ $crc2 = $crc3 ] && [ $crc2 = $crc4 ] && [ $crc6 != $crc5 ] && [ $crc7 != $crc6 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	ln -f $test_name.log $test_name.failed
+	echo "$test_name: $test_description: failed"
+fi
+rm -f $TMPFILE
+fi

[PATCH v4 27/54] e2fsck: rebuild sparse extent trees/convert non-extent ext3 files

[Darrick J. Wong]

Teach e2fsck to (re)construct extent trees.  This enables us to do
either of the following: compress a highly sparse extent tree into
fewer ETB blocks; or convert a ext3-style block mapped file to an
extent file.  The reconstruction is performed during pass 1E or 3A,
as detailed below.

For files that are already extent based, this algorithm will
automatically run (pending user approval) if pass1 determines either
(1) that a whole level of extent tree will fit into a higher level of
the tree; (2) that the size of any level can be reduced by at least
one ETB block; or (3) the extent tree is unnecessarily deep.  It will
not run at all if errors are found and the user declines to fix the
errors.

The option "-E bmap2extent" can be used to force e2fsck to convert all
block map files to extent trees, and to rebuild all extent files'
extent trees.  After conversion, files larger than 12 blocks should be
defragmented to eliminate empty holes where a block lives.

The extent tree constructor is pretty dumb -- it creates a list of
leaf extents (adjacent extents are collapsed), marks all indirect
blocks / ETB blocks free, installs a new extent tree root in the
inode, then loads the leaf extents into the tree.

v2: Account for extent tree block slack that we create when splitting
a block, so that we don't repeatedly annoy the user to rebuild a tree
that we can't optimize further.

v3: For any directory being rebuilt during pass 3A, defer any extent
tree rebuilding until after the rehash.  It's quite possible that the
act of compressing an aged directory will cause it to shrink far
enough to enable us to knock a level off the dir's extent tree.

v4: Add a fixes_only option (and a E2FSCK_FIXES_ONLY environment
variable) that disables optimization activities unless they are
required to make the filesystem consistent.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/Makefile.in                     |   16 +
 e2fsck/e2fsck.8.in                     |    8 
 e2fsck/e2fsck.c                        |    4 
 e2fsck/e2fsck.h                        |   36 ++
 e2fsck/extents.c                       |  536 ++++++++++++++++++++++++++++++++
 e2fsck/pass1.c                         |   63 ++++
 e2fsck/problem.c                       |   48 +++
 e2fsck/problem.h                       |   33 ++
 e2fsck/rehash.c                        |   27 +-
 e2fsck/super.c                         |    7 
 e2fsck/unix.c                          |   23 +
 tests/f_extent_bad_node/expect.1       |   11 -
 tests/f_extent_bad_node/expect.2       |    2 
 tests/f_extent_int_bad_magic/expect.1  |    5 
 tests/f_extent_leaf_bad_magic/expect.1 |    5 
 tests/f_extent_oobounds/expect.1       |   11 -
 tests/f_extent_oobounds/expect.2       |    2 
 tests/f_extents/expect.1               |    9 +
 18 files changed, 819 insertions(+), 27 deletions(-)
 create mode 100644 e2fsck/extents.c

diff --git a/e2fsck/Makefile.in b/e2fsck/Makefile.in
index e40e51b..a4413d9 100644
--- a/e2fsck/Makefile.in
+++ b/e2fsck/Makefile.in
@@ -62,7 +62,8 @@ OBJS= dict.o unix.o e2fsck.o super.o pass1.o pass1b.o pass2.o \
 	pass3.o pass4.o pass5.o journal.o badblocks.o util.o dirinfo.o \
 	dx_dirinfo.o ehandler.o problem.o message.o quota.o recovery.o \
 	region.o revoke.o ea_refcount.o rehash.o profile.o prof_err.o \
-	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o readahead.o
+	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o readahead.o \
+	extents.o
 
 PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/super.o profiled/pass1.o profiled/pass1b.o \
@@ -74,7 +75,7 @@ PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/ea_refcount.o profiled/rehash.o profiled/profile.o \
 	profiled/prof_err.o profiled/logfile.o \
 	profiled/sigcatcher.o profiled/plausible.o \
-	profiled/sigcatcher.o profiled/readahead.o
+	profiled/sigcatcher.o profiled/readahead.o profiled/extents.o
 
 SRCS= $(srcdir)/e2fsck.c \
 	$(srcdir)/dict.c \
@@ -106,6 +107,7 @@ SRCS= $(srcdir)/e2fsck.c \
 	prof_err.c \
 	$(srcdir)/quota.c \
 	$(srcdir)/../misc/plausible.c \
+	$(srcdir)/extents.c \
 	$(MTRACE_SRC)
 
 all:: profiled $(PROGS) e2fsck $(MANPAGES) $(FMANPAGES)
@@ -308,6 +310,16 @@ pass1.o: $(srcdir)/pass1.c $(top_builddir)/lib/config.h \
  $(srcdir)/profile.h prof_err.h $(top_srcdir)/lib/quota/quotaio.h \
  $(top_srcdir)/lib/quota/dqblk_v2.h $(top_srcdir)/lib/quota/quotaio_tree.h \
  $(top_srcdir)/lib/../e2fsck/dict.h $(srcdir)/problem.h
+extents.o: $(srcdir)/extents.c $(top_builddir)/lib/config.h \
+ $(top_builddir)/lib/dirpaths.h $(top_srcdir)/lib/et/com_err.h \
+ $(srcdir)/e2fsck.h $(top_srcdir)/lib/ext2fs/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(top_srcdir)/lib/ext2fs/ext2fs.h \
+ $(top_srcdir)/lib/ext2fs/ext3_extents.h $(top_srcdir)/lib/ext2fs/ext2_io.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h \
+ $(top_srcdir)/lib/ext2fs/ext2_ext_attr.h $(top_srcdir)/lib/ext2fs/bitops.h \
+ $(srcdir)/profile.h prof_err.h $(top_srcdir)/lib/quota/quotaio.h \
+ $(top_srcdir)/lib/quota/dqblk_v2.h $(top_srcdir)/lib/quota/quotaio_tree.h \
+ $(top_srcdir)/lib/../e2fsck/dict.h $(srcdir)/problem.h $(srcdir)/dict.h
 pass1b.o: $(srcdir)/pass1b.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/dirpaths.h $(top_srcdir)/lib/et/com_err.h \
  $(srcdir)/e2fsck.h $(top_srcdir)/lib/ext2fs/ext2_fs.h \
diff --git a/e2fsck/e2fsck.8.in b/e2fsck/e2fsck.8.in
index 270727a..e1bbd27 100644
--- a/e2fsck/e2fsck.8.in
+++ b/e2fsck/e2fsck.8.in
@@ -227,6 +227,14 @@ e2fsck runtime.  By default, this is set to the size of two block groups' inode
 tables (typically 4MiB on a regular ext4 filesystem); if this amount is more
 than 1/50th of total physical memory, readahead is disabled.  Set this to zero
 to disable readahead entirely.
+.TP
+.BI bmap2extent
+Convert block-mapped files to extent-mapped files.
+.TP
+.BI fixes_only
+Only fix damaged metadata; do not optimize htree directories or compress
+extent trees.  This option is incompatible with the -D and -E bmap2extent
+options.
 .RE
 .TP
 .B \-f
diff --git a/e2fsck/e2fsck.c b/e2fsck/e2fsck.c
index 4273060..d8db925 100644
--- a/e2fsck/e2fsck.c
+++ b/e2fsck/e2fsck.c
@@ -208,8 +208,8 @@ void e2fsck_free_context(e2fsck_t ctx)
 typedef void (*pass_t)(e2fsck_t ctx);
 
 static pass_t e2fsck_passes[] = {
-	e2fsck_pass1, e2fsck_pass2, e2fsck_pass3, e2fsck_pass4,
-	e2fsck_pass5, 0 };
+	e2fsck_pass1, e2fsck_pass1e, e2fsck_pass2, e2fsck_pass3,
+	e2fsck_pass4, e2fsck_pass5, 0 };
 
 #define E2F_FLAG_RUN_RETURN	(E2F_FLAG_SIGNAL_MASK|E2F_FLAG_RESTART)
 
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 6f96e55..5fda863 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -167,6 +167,8 @@ struct resource_track {
 #define E2F_OPT_FRAGCHECK	0x0800
 #define E2F_OPT_JOURNAL_ONLY	0x1000 /* only replay the journal */
 #define E2F_OPT_DISCARD		0x2000
+#define E2F_OPT_CONVERT_BMAP	0x4000 /* convert blockmap to extent */
+#define E2F_OPT_FIXES_ONLY	0x8000 /* skip all optimizations */
 
 /*
  * E2fsck flags
@@ -190,6 +192,7 @@ struct resource_track {
 #define E2F_FLAG_EXITING	0x1000 /* E2fsck exiting due to errors */
 #define E2F_FLAG_TIME_INSANE	0x2000 /* Time is insane */
 #define E2F_FLAG_PROBLEMS_FIXED	0x4000 /* At least one problem was fixed */
+#define E2F_FLAG_ALLOC_OK	0x8000 /* Can we allocate blocks? */
 
 #define E2F_RESET_FLAGS (E2F_FLAG_TIME_INSANE | E2F_FLAG_PROBLEMS_FIXED)
 
@@ -382,6 +385,23 @@ struct e2fsck_struct {
 
 	/* How much are we allowed to readahead? */
 	unsigned long long readahead_kb;
+
+	/*
+	 * Inodes to rebuild extent trees
+	 */
+	ext2fs_inode_bitmap inodes_to_rebuild;
+};
+
+/* Data structures to evaluate whether an extent tree needs rebuilding. */
+struct extent_tree_level {
+	unsigned int	num_extents;
+	unsigned int	max_extents;
+};
+
+struct extent_tree_info {
+	int force_rebuild:1;
+	ext2_ino_t ino;
+	struct extent_tree_level	ext_info[MAX_EXTENT_DEPTH_COUNT];
 };
 
 /* Used by the region allocation code */
@@ -457,6 +477,19 @@ extern blk64_t ea_refcount_intr_next(ext2_refcount_t refcount, int *ret);
 extern const char *ehandler_operation(const char *op);
 extern void ehandler_init(io_channel channel);
 
+/* extents.c */
+struct problem_context;
+errcode_t e2fsck_rebuild_extents_later(e2fsck_t ctx, ext2_ino_t ino);
+int e2fsck_ino_will_be_rebuilt(e2fsck_t ctx, ext2_ino_t ino);
+void e2fsck_pass1e(e2fsck_t ctx);
+errcode_t e2fsck_check_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino,
+				       struct ext2_inode *inode,
+				       struct problem_context *pctx);
+errcode_t e2fsck_should_rebuild_extents(e2fsck_t ctx,
+					struct problem_context *pctx,
+					struct extent_tree_info *eti,
+					struct ext2_extent_info *info);
+
 /* journal.c */
 extern errcode_t e2fsck_check_ext3_journal(e2fsck_t ctx);
 extern errcode_t e2fsck_run_ext3_journal(e2fsck_t ctx);
@@ -524,7 +557,8 @@ extern int region_allocate(region_t region, region_addr_t start, int n);
 /* rehash.c */
 void e2fsck_rehash_dir_later(e2fsck_t ctx, ext2_ino_t ino);
 int e2fsck_dir_will_be_rehashed(e2fsck_t ctx, ext2_ino_t ino);
-errcode_t e2fsck_rehash_dir(e2fsck_t ctx, ext2_ino_t ino);
+errcode_t e2fsck_rehash_dir(e2fsck_t ctx, ext2_ino_t ino,
+			    struct problem_context *pctx);
 void e2fsck_rehash_directories(e2fsck_t ctx);
 
 /* sigcatcher.c */
diff --git a/e2fsck/extents.c b/e2fsck/extents.c
new file mode 100644
index 0000000..8465299
--- /dev/null
+++ b/e2fsck/extents.c
@@ -0,0 +1,536 @@
+/*
+ * extents.c --- rebuild extent tree
+ *
+ * Copyright (C) 2014 Oracle.
+ *
+ * %Begin-Header%
+ * This file may be redistributed under the terms of the GNU Public
+ * License, version 2.
+ * %End-Header%
+ */
+
+#include "config.h"
+#include <string.h>
+#include <ctype.h>
+#include <errno.h>
+#include "e2fsck.h"
+#include "problem.h"
+
+#undef DEBUG
+#undef DEBUG_SUMMARY
+#undef DEBUG_FREE
+
+#define NUM_EXTENTS	341	/* about one ETB' worth of extents */
+
+static errcode_t e2fsck_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino);
+
+/* Schedule an inode to have its extent tree rebuilt during pass 1E. */
+errcode_t e2fsck_rebuild_extents_later(e2fsck_t ctx, ext2_ino_t ino)
+{
+	if (!EXT2_HAS_INCOMPAT_FEATURE(ctx->fs->super,
+				       EXT3_FEATURE_INCOMPAT_EXTENTS) ||
+	    (ctx->options & E2F_OPT_NO) ||
+	    (ino != EXT2_ROOT_INO && ino < ctx->fs->super->s_first_ino))
+		return 0;
+
+	if (ctx->flags & E2F_FLAG_ALLOC_OK)
+		return e2fsck_rebuild_extents(ctx, ino);
+
+	if (!ctx->inodes_to_rebuild)
+		e2fsck_allocate_inode_bitmap(ctx->fs,
+					     _("extent rebuild inode map"),
+					     EXT2FS_BMAP64_RBTREE,
+					     "inodes_to_rebuild",
+					     &ctx->inodes_to_rebuild);
+	if (ctx->inodes_to_rebuild)
+		ext2fs_mark_inode_bitmap2(ctx->inodes_to_rebuild, ino);
+	return 0;
+}
+
+/* Ask if an inode will have its extents rebuilt during pass 1E. */
+int e2fsck_ino_will_be_rebuilt(e2fsck_t ctx, ext2_ino_t ino)
+{
+	if (!ctx->inodes_to_rebuild)
+		return 0;
+	return ext2fs_test_inode_bitmap2(ctx->inodes_to_rebuild, ino);
+}
+
+struct extent_list {
+	blk64_t blocks_freed;
+	struct ext2fs_extent *extents;
+	unsigned int count;
+	unsigned int size;
+	unsigned int ext_read;
+	errcode_t retval;
+	ext2_ino_t ino;
+};
+
+static errcode_t load_extents(e2fsck_t ctx, struct extent_list *list)
+{
+	ext2_filsys		fs = ctx->fs;
+	ext2_extent_handle_t	handle;
+	struct ext2fs_extent	extent;
+	errcode_t		retval;
+
+	retval = ext2fs_extent_open(fs, list->ino, &handle);
+	if (retval)
+		return retval;
+
+	retval = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
+	if (retval)
+		goto out;
+
+	do {
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)
+			goto next;
+
+		/* Internal node; free it and we'll re-allocate it later */
+		if (!(extent.e_flags & EXT2_EXTENT_FLAGS_LEAF)) {
+#if defined(DEBUG) || defined(DEBUG_FREE)
+			printf("ino=%d free=%llu bf=%llu\n", list->ino,
+					extent.e_pblk, list->blocks_freed + 1);
+#endif
+			list->blocks_freed++;
+			ext2fs_block_alloc_stats2(fs, extent.e_pblk, -1);
+			goto next;
+		}
+
+		list->ext_read++;
+		/* Can we attach it to the previous extent? */
+		if (list->count) {
+			struct ext2fs_extent *last = list->extents +
+						     list->count - 1;
+			blk64_t end = last->e_len + extent.e_len;
+
+			if (last->e_pblk + last->e_len == extent.e_pblk &&
+			    last->e_lblk + last->e_len == extent.e_lblk &&
+			    (last->e_flags & EXT2_EXTENT_FLAGS_UNINIT) ==
+			    (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+			    end < (1ULL << 32)) {
+				last->e_len += extent.e_len;
+#ifdef DEBUG
+				printf("R: ino=%d len=%u\n", list->ino,
+						last->e_len);
+#endif
+				goto next;
+			}
+		}
+
+		/* Do we need to expand? */
+		if (list->count == list->size) {
+			unsigned int new_size = (list->size + NUM_EXTENTS) *
+						sizeof(struct ext2fs_extent);
+			retval = ext2fs_resize_mem(0, new_size, &list->extents);
+			if (retval)
+				goto out;
+			list->size += NUM_EXTENTS;
+		}
+
+		/* Add a new extent */
+		memcpy(list->extents + list->count, &extent, sizeof(extent));
+#ifdef DEBUG
+		printf("R: ino=%d pblk=%llu lblk=%llu len=%u\n", list->ino,
+				extent.e_pblk, extent.e_lblk, extent.e_len);
+#endif
+		list->count++;
+next:
+		retval = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent);
+	} while (retval == 0);
+
+out:
+	/* Ok if we run off the end */
+	if (retval == EXT2_ET_EXTENT_NO_NEXT)
+		retval = 0;
+	ext2fs_extent_free(handle);
+	return retval;
+}
+
+static int find_blocks(ext2_filsys fs, blk64_t *blocknr, e2_blkcnt_t blockcnt,
+		       blk64_t ref_blk, int ref_offset, void *priv_data)
+{
+	struct extent_list *list = priv_data;
+
+	/* Internal node? */
+	if (blockcnt < 0) {
+#if defined(DEBUG) || defined(DEBUG_FREE)
+		printf("ino=%d free=%llu bf=%llu\n", list->ino, *blocknr,
+				list->blocks_freed + 1);
+#endif
+		list->blocks_freed++;
+		ext2fs_block_alloc_stats2(fs, *blocknr, -1);
+		return 0;
+	}
+
+	/* Can we attach it to the previous extent? */
+	if (list->count) {
+		struct ext2fs_extent *last = list->extents +
+					     list->count - 1;
+		blk64_t end = last->e_len + 1;
+
+		if (last->e_pblk + last->e_len == *blocknr &&
+		    end < (1ULL << 32)) {
+			last->e_len++;
+#ifdef DEBUG
+			printf("R: ino=%d len=%u\n", list->ino, last->e_len);
+#endif
+			return 0;
+		}
+	}
+
+	/* Do we need to expand? */
+	if (list->count == list->size) {
+		unsigned int new_size = (list->size + NUM_EXTENTS) *
+					sizeof(struct ext2fs_extent);
+		list->retval = ext2fs_resize_mem(0, new_size, &list->extents);
+		if (list->retval)
+			return BLOCK_ABORT;
+		list->size += NUM_EXTENTS;
+	}
+
+	/* Add a new extent */
+	list->extents[list->count].e_pblk = *blocknr;
+	list->extents[list->count].e_lblk = blockcnt;
+	list->extents[list->count].e_len = 1;
+	list->extents[list->count].e_flags = 0;
+#ifdef DEBUG
+	printf("R: ino=%d pblk=%llu lblk=%llu len=%u\n", list->ino, *blocknr,
+			blockcnt, 1);
+#endif
+	list->count++;
+
+	return 0;
+}
+
+static errcode_t rebuild_extent_tree(e2fsck_t ctx, struct extent_list *list,
+				     ext2_ino_t ino)
+{
+	struct ext2_inode	inode;
+	errcode_t		retval;
+	ext2_extent_handle_t	handle;
+	unsigned int		i, ext_written;
+	struct ext2fs_extent	*ex, extent;
+
+	list->count = 0;
+	list->blocks_freed = 0;
+	list->ino = ino;
+	list->ext_read = 0;
+	e2fsck_read_inode(ctx, ino, &inode, "rebuild_extents");
+
+	/* Skip deleted inodes and inline data files */
+	if (inode.i_links_count == 0 ||
+	    inode.i_flags & EXT4_INLINE_DATA_FL)
+		return 0;
+
+	/* Collect lblk->pblk mappings */
+	if (inode.i_flags & EXT4_EXTENTS_FL) {
+		retval = load_extents(ctx, list);
+		goto extents_loaded;
+	}
+
+	retval = ext2fs_block_iterate3(ctx->fs, ino, BLOCK_FLAG_READ_ONLY, 0,
+				       find_blocks, list);
+	if (retval)
+		goto err;
+	if (list->retval) {
+		retval = list->retval;
+		goto err;
+	}
+
+extents_loaded:
+	/* Reset extent tree */
+	inode.i_flags &= ~EXT4_EXTENTS_FL;
+	memset(inode.i_block, 0, sizeof(inode.i_block));
+
+	/* Make a note of freed blocks */
+	retval = ext2fs_iblk_sub_blocks(ctx->fs, &inode, list->blocks_freed);
+	if (retval)
+		goto err;
+
+	/* Now stuff extents into the file */
+	retval = ext2fs_extent_open2(ctx->fs, ino, &inode, &handle);
+	if (retval)
+		goto err;
+
+	ext_written = 0;
+	for (i = 0, ex = list->extents; i < list->count; i++, ex++) {
+		memcpy(&extent, ex, sizeof(struct ext2fs_extent));
+		extent.e_flags &= EXT2_EXTENT_FLAGS_UNINIT;
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) {
+			if (extent.e_len > EXT_UNINIT_MAX_LEN) {
+				extent.e_len = EXT_UNINIT_MAX_LEN;
+				ex->e_pblk += EXT_UNINIT_MAX_LEN;
+				ex->e_lblk += EXT_UNINIT_MAX_LEN;
+				ex->e_len -= EXT_UNINIT_MAX_LEN;
+				ex--;
+				i--;
+			}
+		} else {
+			if (extent.e_len > EXT_INIT_MAX_LEN) {
+				extent.e_len = EXT_INIT_MAX_LEN;
+				ex->e_pblk += EXT_INIT_MAX_LEN;
+				ex->e_lblk += EXT_INIT_MAX_LEN;
+				ex->e_len -= EXT_INIT_MAX_LEN;
+				ex--;
+				i--;
+			}
+		}
+
+#ifdef DEBUG
+		printf("W: ino=%d pblk=%llu lblk=%llu len=%u\n", ino,
+				extent.e_pblk, extent.e_lblk, extent.e_len);
+#endif
+		retval = ext2fs_extent_insert(handle, EXT2_EXTENT_INSERT_AFTER,
+					      &extent);
+		if (retval)
+			goto err2;
+		retval = ext2fs_extent_fix_parents(handle);
+		if (retval)
+			goto err2;
+		ext_written++;
+	}
+
+#if defined(DEBUG) || defined(DEBUG_SUMMARY)
+	printf("rebuild: ino=%d extents=%d->%d\n", ino, list->ext_read,
+	       ext_written);
+#endif
+	e2fsck_write_inode(ctx, ino, &inode, "rebuild_extents");
+
+err2:
+	ext2fs_extent_free(handle);
+err:
+	return retval;
+}
+
+/* Rebuild the extents immediately */
+static errcode_t e2fsck_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino)
+{
+	struct extent_list	list;
+	errcode_t err;
+
+	if (!EXT2_HAS_INCOMPAT_FEATURE(ctx->fs->super,
+				       EXT3_FEATURE_INCOMPAT_EXTENTS) ||
+	    (ctx->options & E2F_OPT_NO) ||
+	    (ino != EXT2_ROOT_INO && ino < ctx->fs->super->s_first_ino))
+		return 0;
+
+	e2fsck_read_bitmaps(ctx);
+	memset(&list, 0, sizeof(list));
+	err = ext2fs_get_mem(sizeof(struct ext2fs_extent) * NUM_EXTENTS,
+				&list.extents);
+	if (err)
+		return err;
+	list.size = NUM_EXTENTS;
+	err = rebuild_extent_tree(ctx, &list, ino);
+	ext2fs_free_mem(&list.extents);
+
+	return err;
+}
+
+static void rebuild_extents(e2fsck_t ctx, const char *pass_name, int pr_header)
+{
+	struct problem_context	pctx;
+#ifdef RESOURCE_TRACK
+	struct resource_track	rtrack;
+#endif
+	struct extent_list	list;
+	int			first = 1;
+	ext2_ino_t		ino = 0;
+	errcode_t		retval;
+
+	if (!EXT2_HAS_INCOMPAT_FEATURE(ctx->fs->super,
+				       EXT3_FEATURE_INCOMPAT_EXTENTS) ||
+	    !ext2fs_test_valid(ctx->fs) ||
+	    ctx->invalid_bitmaps) {
+		if (ctx->inodes_to_rebuild)
+			ext2fs_free_inode_bitmap(ctx->inodes_to_rebuild);
+		ctx->inodes_to_rebuild = NULL;
+	}
+
+	if (ctx->inodes_to_rebuild == NULL)
+		return;
+
+	init_resource_track(&rtrack, ctx->fs->io);
+	clear_problem_context(&pctx);
+	e2fsck_read_bitmaps(ctx);
+
+	memset(&list, 0, sizeof(list));
+	retval = ext2fs_get_mem(sizeof(struct ext2fs_extent) * NUM_EXTENTS,
+				&list.extents);
+	list.size = NUM_EXTENTS;
+	while (1) {
+		retval = ext2fs_find_first_set_inode_bitmap2(
+				ctx->inodes_to_rebuild, ino + 1,
+				ctx->fs->super->s_inodes_count, &ino);
+		if (retval)
+			break;
+		pctx.ino = ino;
+		if (first) {
+			fix_problem(ctx, pr_header, &pctx);
+			first = 0;
+		}
+		pctx.errcode = rebuild_extent_tree(ctx, &list, ino);
+		if (pctx.errcode) {
+			end_problem_latch(ctx, PR_LATCH_OPTIMIZE_EXT);
+			fix_problem(ctx, PR_1E_OPTIMIZE_EXT_ERR, &pctx);
+		}
+		if (ctx->progress && !ctx->progress_fd)
+			e2fsck_simple_progress(ctx, "Rebuilding extents",
+					100.0 * (float) ino /
+					(float) ctx->fs->super->s_inodes_count,
+					ino);
+	}
+	end_problem_latch(ctx, PR_LATCH_OPTIMIZE_EXT);
+
+	ext2fs_free_inode_bitmap(ctx->inodes_to_rebuild);
+	ctx->inodes_to_rebuild = NULL;
+	ext2fs_free_mem(&list.extents);
+
+	print_resource_track(ctx, pass_name, &rtrack, ctx->fs->io);
+}
+
+/* Scan a file to see if we should rebuild its extent tree */
+errcode_t e2fsck_check_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino,
+				  struct ext2_inode *inode,
+				  struct problem_context *pctx)
+{
+	struct extent_tree_info	eti;
+	struct ext2_extent_info	info, top_info;
+	struct ext2fs_extent	extent;
+	ext2_extent_handle_t	ehandle;
+	ext2_filsys		fs = ctx->fs;
+	errcode_t		retval;
+
+	/* block map file and we want extent conversion */
+	if (!(inode->i_flags & EXT4_EXTENTS_FL) &&
+	    !(inode->i_flags & EXT4_INLINE_DATA_FL) &&
+	    (ctx->options & E2F_OPT_CONVERT_BMAP)) {
+		return e2fsck_rebuild_extents_later(ctx, ino);
+	}
+
+	if (!(inode->i_flags & EXT4_EXTENTS_FL))
+		return 0;
+	memset(&eti, 0, sizeof(eti));
+	eti.ino = ino;
+
+	/* Otherwise, go scan the extent tree... */
+	retval = ext2fs_extent_open2(fs, ino, inode, &ehandle);
+	if (retval)
+		return 0;
+
+	retval = ext2fs_extent_get_info(ehandle, &top_info);
+	if (retval)
+		goto out;
+
+	/* Check maximum extent depth */
+	pctx->ino = ino;
+	pctx->blk = top_info.max_depth;
+	pctx->blk2 = ext2fs_max_extent_depth(ehandle);
+	if (pctx->blk2 < pctx->blk &&
+	    fix_problem(ctx, PR_1_EXTENT_BAD_MAX_DEPTH, pctx))
+		eti.force_rebuild = 1;
+
+	/* Can we collect extent tree level stats? */
+	pctx->blk = MAX_EXTENT_DEPTH_COUNT;
+	if (pctx->blk2 > pctx->blk)
+		fix_problem(ctx, PR_1E_MAX_EXTENT_TREE_DEPTH, pctx);
+
+	/* We need to fix tree depth problems, but the scan isn't a fix. */
+	if (ctx->options & E2F_OPT_FIXES_ONLY)
+		goto out;
+
+	retval = ext2fs_extent_get(ehandle, EXT2_EXTENT_ROOT, &extent);
+	if (retval)
+		goto out;
+
+	do {
+		retval = ext2fs_extent_get_info(ehandle, &info);
+		if (retval)
+			break;
+
+		/*
+		 * If this is the first extent in an extent block that we
+		 * haven't visited, collect stats on the block.
+		 */
+		if (info.curr_entry == 1 &&
+		    !(extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT) &&
+		    !eti.force_rebuild) {
+			struct extent_tree_level *etl;
+
+			etl = eti.ext_info + info.curr_level;
+			etl->num_extents += info.num_entries;
+			etl->max_extents += info.max_entries;
+			/*
+			 * Implementation wart: Splitting extent blocks when
+			 * appending will leave the old block with one free
+			 * entry.  Therefore unless the node is totally full,
+			 * pretend that a non-root extent block can hold one
+			 * fewer entry than it actually does, so that we don't
+			 * repeatedly rebuild the extent tree.
+			 */
+			if (info.curr_level &&
+			    info.num_entries < info.max_entries)
+				etl->max_extents--;
+		}
+
+		/* Skip to the end of a block of leaf nodes */
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) {
+			retval = ext2fs_extent_get(ehandle,
+						    EXT2_EXTENT_LAST_SIB,
+						    &extent);
+			if (retval)
+				break;
+		}
+
+		retval = ext2fs_extent_get(ehandle, EXT2_EXTENT_NEXT, &extent);
+	} while (retval == 0);
+out:
+	ext2fs_extent_free(ehandle);
+	return e2fsck_should_rebuild_extents(ctx, pctx, &eti, &top_info);
+}
+
+/* Having scanned a file's extent tree, decide if we should rebuild it */
+errcode_t e2fsck_should_rebuild_extents(e2fsck_t ctx,
+				   struct problem_context *pctx,
+				   struct extent_tree_info *eti,
+				   struct ext2_extent_info *info)
+{
+	struct extent_tree_level *ei;
+	int i, j, op;
+	unsigned int extents_per_block;
+
+	if (eti->force_rebuild)
+		goto rebuild;
+
+	extents_per_block = (ctx->fs->blocksize -
+			     sizeof(struct ext3_extent_header)) /
+			    sizeof(struct ext3_extent);
+	/*
+	 * If we can consolidate a level or shorten the tree, schedule the
+	 * extent tree to be rebuilt.
+	 */
+	for (i = 0, ei = eti->ext_info; i < info->max_depth + 1; i++, ei++) {
+		if (ei->max_extents - ei->num_extents > extents_per_block) {
+			pctx->blk = i;
+			op = PR_1E_CAN_NARROW_EXTENT_TREE;
+			goto rebuild;
+		}
+		for (j = 0; j < i; j++) {
+			if (ei->num_extents < eti->ext_info[j].max_extents) {
+				pctx->blk = i;
+				op = PR_1E_CAN_COLLAPSE_EXTENT_TREE;
+				goto rebuild;
+			}
+		}
+	}
+	return 0;
+
+rebuild:
+	if (eti->force_rebuild || fix_problem(ctx, op, pctx))
+		return e2fsck_rebuild_extents_later(ctx, eti->ino);
+	return 0;
+}
+
+void e2fsck_pass1e(e2fsck_t ctx)
+{
+	rebuild_extents(ctx, "Pass 1E", PR_1E_PASS_HEADER);
+}
diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 3308fb6..24e991a 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -56,6 +56,8 @@
 #define _INLINE_ inline
 #endif
 
+#undef DEBUG
+
 static int process_block(ext2_filsys fs, blk64_t	*blocknr,
 			 e2_blkcnt_t blockcnt, blk64_t ref_blk,
 			 int ref_offset, void *priv_data);
@@ -95,6 +97,7 @@ struct process_block_struct {
 	ext2fs_block_bitmap fs_meta_blocks;
 	e2fsck_t	ctx;
 	region_t	region;
+	struct extent_tree_info	eti;
 };
 
 struct process_inode_block {
@@ -1837,6 +1840,7 @@ void e2fsck_pass1(e2fsck_t ctx)
 		}
 		e2fsck_pass1_dupblocks(ctx, block_buf);
 	}
+	ctx->flags |= E2F_FLAG_ALLOC_OK;
 	ext2fs_free_mem(&inodes_to_process);
 endit:
 	e2fsck_use_inode_shortcuts(ctx, 0);
@@ -2488,6 +2492,23 @@ static void scan_extent_node(e2fsck_t ctx, struct problem_context *pctx,
 	pctx->errcode = ext2fs_extent_get_info(ehandle, &info);
 	if (pctx->errcode)
 		return;
+	if (!(ctx->options & E2F_OPT_FIXES_ONLY) &&
+	    !pb->eti.force_rebuild) {
+		struct extent_tree_level *etl;
+
+		etl = pb->eti.ext_info + info.curr_level;
+		etl->num_extents += info.num_entries;
+		etl->max_extents += info.max_entries;
+		/*
+		 * Implementation wart: Splitting extent blocks when appending
+		 * will leave the old block with one free entry.  Therefore
+		 * unless the node is totally full, pretend that a non-root
+		 * extent block can hold one fewer entry than it actually does,
+		 * so that we don't repeatedly rebuild the extent tree.
+		 */
+		if (info.curr_level && info.num_entries < info.max_entries)
+			etl->max_extents--;
+	}
 
 	pctx->errcode = ext2fs_extent_get(ehandle, EXT2_EXTENT_FIRST_SIB,
 					  &extent);
@@ -2824,11 +2845,27 @@ static void check_blocks_extents(e2fsck_t ctx, struct problem_context *pctx,
 
 	retval = ext2fs_extent_get_info(ehandle, &info);
 	if (retval == 0) {
-		if (info.max_depth >= MAX_EXTENT_DEPTH_COUNT)
-			info.max_depth = MAX_EXTENT_DEPTH_COUNT-1;
-		ctx->extent_depth_count[info.max_depth]++;
+		int max_depth = info.max_depth;
+
+		if (max_depth >= MAX_EXTENT_DEPTH_COUNT)
+			max_depth = MAX_EXTENT_DEPTH_COUNT-1;
+		ctx->extent_depth_count[max_depth]++;
 	}
 
+	/* Check maximum extent depth */
+	pctx->blk = info.max_depth;
+	pctx->blk2 = ext2fs_max_extent_depth(ehandle);
+	if (pctx->blk2 < pctx->blk &&
+	    fix_problem(ctx, PR_1_EXTENT_BAD_MAX_DEPTH, pctx))
+		pb->eti.force_rebuild = 1;
+
+	/* Can we collect extent tree level stats? */
+	pctx->blk = MAX_EXTENT_DEPTH_COUNT;
+	if (pctx->blk2 > pctx->blk)
+		fix_problem(ctx, PR_1E_MAX_EXTENT_TREE_DEPTH, pctx);
+	memset(pb->eti.ext_info, 0, sizeof(pb->eti.ext_info));
+	pb->eti.ino = pb->ino;
+
 	pb->region = region_create(0, info.max_lblk);
 	if (!pb->region) {
 		ext2fs_extent_free(ehandle);
@@ -2851,6 +2888,16 @@ static void check_blocks_extents(e2fsck_t ctx, struct problem_context *pctx,
 	region_free(pb->region);
 	pb->region = NULL;
 	ext2fs_extent_free(ehandle);
+
+	/* Rebuild unless it's a dir and we're rehashing it */
+	if (LINUX_S_ISDIR(inode->i_mode) &&
+	    e2fsck_dir_will_be_rehashed(ctx, ino))
+		return;
+
+	if (ctx->options & E2F_OPT_CONVERT_BMAP)
+		e2fsck_rebuild_extents_later(ctx, ino);
+	else
+		e2fsck_should_rebuild_extents(ctx, pctx, &pb->eti, &info);
 }
 
 /*
@@ -2935,6 +2982,7 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx,
 	pb.pctx = pctx;
 	pb.ctx = ctx;
 	pb.inode_modified = 0;
+	pb.eti.force_rebuild = 0;
 	pctx->ino = ino;
 	pctx->errcode = 0;
 
@@ -2998,6 +3046,15 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx,
 						  "check_blocks");
 			fs->flags = (flags & EXT2_FLAG_IGNORE_CSUM_ERRORS) |
 				    (fs->flags & ~EXT2_FLAG_IGNORE_CSUM_ERRORS);
+
+			if (ctx->options & E2F_OPT_CONVERT_BMAP) {
+#ifdef DEBUG
+				printf("bmap rebuild ino=%d\n", ino);
+#endif
+				if (!LINUX_S_ISDIR(inode->i_mode) ||
+				    !e2fsck_dir_will_be_rehashed(ctx, ino))
+					e2fsck_rebuild_extents_later(ctx, ino);
+			}
 		}
 	}
 	end_problem_latch(ctx, PR_LATCH_BLOCK);
diff --git a/e2fsck/problem.c b/e2fsck/problem.c
index 960fb07..7af0b76 100644
--- a/e2fsck/problem.c
+++ b/e2fsck/problem.c
@@ -1106,6 +1106,11 @@ static struct e2fsck_problem problem_table[] = {
 	  N_("@A memory for encrypted @d list\n"),
 	  PROMPT_NONE, PR_FATAL },
 
+	/* Inode extent tree could be more shallow */
+	{ PR_1_EXTENT_BAD_MAX_DEPTH,
+	  N_("@i %i @x tree could be more shallow (%b; could be <= %c)\n"),
+	  PROMPT_FIX, PR_NO_OK | PR_PREEN_NO | PR_PREEN_OK },
+
 	/* Pass 1b errors */
 
 	/* Pass 1B: Rescan for duplicate/bad blocks */
@@ -1203,6 +1208,48 @@ static struct e2fsck_problem problem_table[] = {
 	{ PR_1D_CLONE_ERROR,
 	  N_("Couldn't clone file: %m\n"), PROMPT_NONE, 0 },
 
+	/* Pass 1E Extent tree optimization	*/
+
+	/* Pass 1E: Optimizing extent trees */
+	{ PR_1E_PASS_HEADER,
+	  N_("Pass 1E: Optimizing @x trees\n"),
+	  PROMPT_NONE, PR_PREEN_NOMSG },
+
+	/* Failed to optimize extent tree */
+	{ PR_1E_OPTIMIZE_EXT_ERR,
+	  N_("Failed to optimize @x tree %p (%i): %m\n"),
+	  PROMPT_NONE, 0 },
+
+	/* Optimizing extent trees */
+	{ PR_1E_OPTIMIZE_EXT_HEADER,
+	  N_("Optimizing @x trees: "),
+	  PROMPT_NONE, PR_MSG_ONLY },
+
+	/* Rebuilding extent tree %d */
+	{ PR_1E_OPTIMIZE_EXT,
+	  " %i",
+	  PROMPT_NONE, PR_LATCH_OPTIMIZE_EXT | PR_PREEN_NOHDR},
+
+	/* Rebuilding extent tree end */
+	{ PR_1E_OPTIMIZE_EXT_END,
+	  "\n",
+	  PROMPT_NONE, PR_PREEN_NOHDR },
+
+	/* Internal error: extent tree depth too large */
+	{ PR_1E_MAX_EXTENT_TREE_DEPTH,
+	  N_("Internal error: max extent tree depth too large (%b; expected=%c).\n"),
+	  PROMPT_NONE, PR_FATAL },
+
+	/* Inode extent tree could be shorter */
+	{ PR_1E_CAN_COLLAPSE_EXTENT_TREE,
+	  N_("@i %i @x tree could be shorter.\n\t(level %b is unnecessary)\n"),
+	  PROMPT_FIX, PR_NO_OK | PR_PREEN_NO | PR_PREEN_OK },
+
+	/* Inode extent tree could be narrower */
+	{ PR_1E_CAN_NARROW_EXTENT_TREE,
+	  N_("@i %i @x tree could be narrower.\n\t(level %b has unnecessary nodes)\n"),
+	  PROMPT_FIX, PR_NO_OK | PR_PREEN_NO | PR_PREEN_OK },
+
 	/* Pass 2 errors */
 
 	/* Pass 2: Checking directory structure */
@@ -1951,6 +1998,7 @@ static struct latch_descr pr_latch_info[] = {
 	{ PR_LATCH_TOOBIG, PR_1_INODE_TOOBIG, 0 },
 	{ PR_LATCH_OPTIMIZE_DIR, PR_3A_OPTIMIZE_DIR_HEADER, PR_3A_OPTIMIZE_DIR_END },
 	{ PR_LATCH_BG_CHECKSUM, PR_0_GDT_CSUM_LATCH, 0 },
+	{ PR_LATCH_OPTIMIZE_EXT, PR_1E_OPTIMIZE_EXT_HEADER, PR_1E_OPTIMIZE_EXT_END },
 	{ -1, 0, 0 },
 };
 
diff --git a/e2fsck/problem.h b/e2fsck/problem.h
index 19b2301..ace8d2f 100644
--- a/e2fsck/problem.h
+++ b/e2fsck/problem.h
@@ -40,6 +40,7 @@ struct problem_context {
 #define PR_LATCH_TOOBIG	0x0080	/* Latch for file to big errors */
 #define PR_LATCH_OPTIMIZE_DIR 0x0090 /* Latch for optimize directories */
 #define PR_LATCH_BG_CHECKSUM 0x00A0  /* Latch for block group checksums */
+#define PR_LATCH_OPTIMIZE_EXT 0x00B0  /* Latch for rebuild extents */
 
 #define PR_LATCH(x)	((((x) & PR_LATCH_MASK) >> 4) - 1)
 
@@ -644,6 +645,9 @@ struct problem_context {
 /* Error allocating memory for encrypted directory list */
 #define PR_1_ALLOCATE_ENCRYPTED_DIRLIST		0x01007E
 
+/* extent tree max depth too big */
+#define PR_1_EXTENT_BAD_MAX_DEPTH		0x01007F
+
 /*
  * Pass 1b errors
  */
@@ -707,6 +711,33 @@ struct problem_context {
 #define PR_1D_CLONE_ERROR	0x013008
 
 /*
+ * Pass 1e --- rebuilding extent trees
+ */
+/* Pass 1e: Rebuilding extent trees */
+#define PR_1E_PASS_HEADER		0x014000
+
+/* Error rehash directory */
+#define PR_1E_OPTIMIZE_EXT_ERR		0x014001
+
+/* Rebuilding extent trees */
+#define PR_1E_OPTIMIZE_EXT_HEADER	0x014002
+
+/* Rebuilding extent %d */
+#define PR_1E_OPTIMIZE_EXT		0x014003
+
+/* Rebuilding extent tree end */
+#define PR_1E_OPTIMIZE_EXT_END		0x014004
+
+/* Internal error: extent tree depth too large */
+#define PR_1E_MAX_EXTENT_TREE_DEPTH	0x014005
+
+/* Inode extent tree could be shorter */
+#define PR_1E_CAN_COLLAPSE_EXTENT_TREE	0x014006
+
+/* Inode extent tree could be narrower */
+#define PR_1E_CAN_NARROW_EXTENT_TREE	0x014007
+
+/*
  * Pass 2 errors
  */
 
@@ -1035,6 +1066,8 @@ struct problem_context {
 /* Rehashing dir end */
 #define PR_3A_OPTIMIZE_DIR_END		0x031005
 
+/* Pass 3B is really just 1E */
+
 /*
  * Pass 4 errors
  */
diff --git a/e2fsck/rehash.c b/e2fsck/rehash.c
index 1d720dc..c9da5d4 100644
--- a/e2fsck/rehash.c
+++ b/e2fsck/rehash.c
@@ -766,11 +766,11 @@ static int write_dir_block(ext2_filsys fs,
 
 static errcode_t write_directory(e2fsck_t ctx, ext2_filsys fs,
 				 struct out_dir *outdir,
-				 ext2_ino_t ino, int compress)
+				 ext2_ino_t ino, struct ext2_inode *inode,
+				 int compress)
 {
 	struct write_dir_struct wd;
 	errcode_t	retval;
-	struct ext2_inode 	inode;
 
 	retval = e2fsck_expand_directory(ctx, ino, -1, outdir->num);
 	if (retval)
@@ -789,22 +789,23 @@ static errcode_t write_directory(e2fsck_t ctx, ext2_filsys fs,
 	if (wd.err)
 		return wd.err;
 
-	e2fsck_read_inode(ctx, ino, &inode, "rehash_dir");
+	e2fsck_read_inode(ctx, ino, inode, "rehash_dir");
 	if (compress)
-		inode.i_flags &= ~EXT2_INDEX_FL;
+		inode->i_flags &= ~EXT2_INDEX_FL;
 	else
-		inode.i_flags |= EXT2_INDEX_FL;
-	retval = ext2fs_inode_size_set(fs, &inode,
+		inode->i_flags |= EXT2_INDEX_FL;
+	retval = ext2fs_inode_size_set(fs, inode,
 				       outdir->num * fs->blocksize);
 	if (retval)
 		return retval;
-	ext2fs_iblk_sub_blocks(fs, &inode, wd.cleared);
-	e2fsck_write_inode(ctx, ino, &inode, "rehash_dir");
+	ext2fs_iblk_sub_blocks(fs, inode, wd.cleared);
+	e2fsck_write_inode(ctx, ino, inode, "rehash_dir");
 
 	return 0;
 }
 
-errcode_t e2fsck_rehash_dir(e2fsck_t ctx, ext2_ino_t ino)
+errcode_t e2fsck_rehash_dir(e2fsck_t ctx, ext2_ino_t ino,
+			    struct problem_context *pctx)
 {
 	ext2_filsys 		fs = ctx->fs;
 	errcode_t		retval;
@@ -915,10 +916,14 @@ resort:
 			goto errout;
 	}
 
-	retval = write_directory(ctx, fs, &outdir, ino, fd.compress);
+	retval = write_directory(ctx, fs, &outdir, ino, &inode, fd.compress);
 	if (retval)
 		goto errout;
 
+	if (ctx->options & E2F_OPT_CONVERT_BMAP)
+		retval = e2fsck_rebuild_extents_later(ctx, ino);
+	else
+		retval = e2fsck_check_rebuild_extents(ctx, ino, &inode, pctx);
 errout:
 	free(dir_buf);
 	free(fd.harray);
@@ -962,7 +967,7 @@ void e2fsck_rehash_directories(e2fsck_t ctx)
 #if 0
 		fix_problem(ctx, PR_3A_OPTIMIZE_DIR, &pctx);
 #endif
-		pctx.errcode = e2fsck_rehash_dir(ctx, ino);
+		pctx.errcode = e2fsck_rehash_dir(ctx, ino, &pctx);
 		if (pctx.errcode) {
 			end_problem_latch(ctx, PR_LATCH_OPTIMIZE_DIR);
 			fix_problem(ctx, PR_3A_OPTIMIZE_DIR_ERR, &pctx);
diff --git a/e2fsck/super.c b/e2fsck/super.c
index 1e7e749..e64262a 100644
--- a/e2fsck/super.c
+++ b/e2fsck/super.c
@@ -606,6 +606,13 @@ void check_super_block(e2fsck_t ctx)
 		ext2fs_mark_super_dirty(fs);
 	}
 
+	/* Did user ask us to convert files to extents? */
+	if (ctx->options & E2F_OPT_CONVERT_BMAP) {
+		fs->super->s_feature_incompat |=
+			EXT3_FEATURE_INCOMPAT_EXTENTS;
+		ext2fs_mark_super_dirty(fs);
+	}
+
 	if ((fs->super->s_feature_incompat & EXT2_FEATURE_INCOMPAT_META_BG) &&
 	    (fs->super->s_first_meta_bg > fs->desc_blocks)) {
 		pctx.group = fs->desc_blocks;
diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 576ac17..5206a3e 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -709,6 +709,12 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 			else
 				ctx->log_fn = string_copy(ctx, arg, 0);
 			continue;
+		} else if (strcmp(token, "bmap2extent") == 0) {
+			ctx->options |= E2F_OPT_CONVERT_BMAP;
+			continue;
+		} else if (strcmp(token, "fixes_only") == 0) {
+			ctx->options |= E2F_OPT_FIXES_ONLY;
+			continue;
 		} else {
 			fprintf(stderr, _("Unknown extended option: %s\n"),
 				token);
@@ -728,6 +734,7 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 		fputs(("\tdiscard\n"), stderr);
 		fputs(("\tnodiscard\n"), stderr);
 		fputs(("\treadahead_kb=<buffer size>\n"), stderr);
+		fputs(("\tbmap2extent\n"), stderr);
 		fputc('\n', stderr);
 		exit(1);
 	}
@@ -961,6 +968,22 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 	if (extended_opts)
 		parse_extended_opts(ctx, extended_opts);
 
+	/* Complain about mutually exclusive rebuilding activities */
+	if (getenv("E2FSCK_FIXES_ONLY"))
+		ctx->options |= E2F_OPT_FIXES_ONLY;
+	if ((ctx->options & E2F_OPT_COMPRESS_DIRS) &&
+	    (ctx->options & E2F_OPT_FIXES_ONLY)) {
+		com_err(ctx->program_name, 0, "%s",
+			_("The -D and -E fixes_only options are incompatible."));
+		fatal_error(ctx, 0);
+	}
+	if ((ctx->options & E2F_OPT_CONVERT_BMAP) &&
+	    (ctx->options & E2F_OPT_FIXES_ONLY)) {
+		com_err(ctx->program_name, 0, "%s",
+			_("The -E bmap2extent and fixes_only options are incompatible."));
+		fatal_error(ctx, 0);
+	}
+
 	if ((cp = getenv("E2FSCK_CONFIG")) != NULL)
 		config_fn[0] = cp;
 	profile_set_syntax_err_cb(syntax_err_report);
diff --git a/tests/f_extent_bad_node/expect.1 b/tests/f_extent_bad_node/expect.1
index 0c0bc28..c9643a1 100644
--- a/tests/f_extent_bad_node/expect.1
+++ b/tests/f_extent_bad_node/expect.1
@@ -2,8 +2,13 @@ Pass 1: Checking inodes, blocks, and sizes
 Inode 12 has an invalid extent node (blk 22, lblk 0)
 Clear? yes
 
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 12, i_blocks is 16, should be 8.  Fix? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
@@ -11,13 +16,13 @@ Pass 5: Checking group summary information
 Block bitmap differences:  -(21--23) -25
 Fix? yes
 
-Free blocks count wrong for group #0 (71, counted=75).
+Free blocks count wrong for group #0 (73, counted=77).
 Fix? yes
 
-Free blocks count wrong (71, counted=75).
+Free blocks count wrong (73, counted=77).
 Fix? yes
 
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
-test_filesys: 12/16 files (0.0% non-contiguous), 25/100 blocks
+test_filesys: 12/16 files (0.0% non-contiguous), 23/100 blocks
 Exit status is 1
diff --git a/tests/f_extent_bad_node/expect.2 b/tests/f_extent_bad_node/expect.2
index 568c792..b78b193 100644
--- a/tests/f_extent_bad_node/expect.2
+++ b/tests/f_extent_bad_node/expect.2
@@ -3,5 +3,5 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-test_filesys: 12/16 files (0.0% non-contiguous), 25/100 blocks
+test_filesys: 12/16 files (0.0% non-contiguous), 23/100 blocks
 Exit status is 0
diff --git a/tests/f_extent_int_bad_magic/expect.1 b/tests/f_extent_int_bad_magic/expect.1
index 0e82e2b..3529636 100644
--- a/tests/f_extent_int_bad_magic/expect.1
+++ b/tests/f_extent_int_bad_magic/expect.1
@@ -2,8 +2,13 @@ Pass 1: Checking inodes, blocks, and sizes
 Inode 12 has an invalid extent node (blk 1295, lblk 0)
 Clear? yes
 
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 12, i_blocks is 712, should be 0.  Fix? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
diff --git a/tests/f_extent_leaf_bad_magic/expect.1 b/tests/f_extent_leaf_bad_magic/expect.1
index 7b6dbf1..ae27ecc 100644
--- a/tests/f_extent_leaf_bad_magic/expect.1
+++ b/tests/f_extent_leaf_bad_magic/expect.1
@@ -2,8 +2,13 @@ Pass 1: Checking inodes, blocks, and sizes
 Inode 12 has an invalid extent node (blk 1604, lblk 0)
 Clear? yes
 
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 12, i_blocks is 18, should be 0.  Fix? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
diff --git a/tests/f_extent_oobounds/expect.1 b/tests/f_extent_oobounds/expect.1
index 3164ea0..f0e282e 100644
--- a/tests/f_extent_oobounds/expect.1
+++ b/tests/f_extent_oobounds/expect.1
@@ -3,8 +3,13 @@ Inode 12, end of extent exceeds allowed value
 	(logical block 15, physical block 200, len 30)
 Clear? yes
 
+Inode 12 extent tree could be narrower.
+	(level 1 has unnecessary nodes)
+Fix? yes
+
 Inode 12, i_blocks is 154, should be 94.  Fix? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
@@ -12,13 +17,13 @@ Pass 5: Checking group summary information
 Block bitmap differences:  -(200--229)
 Fix? yes
 
-Free blocks count wrong for group #0 (156, counted=186).
+Free blocks count wrong for group #0 (158, counted=188).
 Fix? yes
 
-Free blocks count wrong (156, counted=186).
+Free blocks count wrong (158, counted=188).
 Fix? yes
 
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
-test_filesys: 12/32 files (8.3% non-contiguous), 70/256 blocks
+test_filesys: 12/32 files (8.3% non-contiguous), 68/256 blocks
 Exit status is 1
diff --git a/tests/f_extent_oobounds/expect.2 b/tests/f_extent_oobounds/expect.2
index 22c4f2c..0729283 100644
--- a/tests/f_extent_oobounds/expect.2
+++ b/tests/f_extent_oobounds/expect.2
@@ -3,5 +3,5 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-test_filesys: 12/32 files (8.3% non-contiguous), 70/256 blocks
+test_filesys: 12/32 files (8.3% non-contiguous), 68/256 blocks
 Exit status is 0
diff --git a/tests/f_extents/expect.1 b/tests/f_extents/expect.1
index aeebc7b..2751eb9 100644
--- a/tests/f_extents/expect.1
+++ b/tests/f_extents/expect.1
@@ -6,6 +6,10 @@ Inode 12 has an invalid extent
 	(logical block 0, invalid physical block 21994527527949, len 17)
 Clear? yes
 
+Inode 12 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 12, i_blocks is 34, should be 0.  Fix? yes
 
 Inode 13 missing EXTENT_FL, but is in extents format
@@ -21,6 +25,10 @@ Inode 17 has an invalid extent
 	(logical block 0, invalid physical block 22011707397135, len 15)
 Clear? yes
 
+Inode 17 extent tree could be shorter.
+	(level 1 is unnecessary)
+Fix? yes
+
 Inode 17, i_blocks is 32, should be 0.  Fix? yes
 
 Error while reading over extent tree in inode 18: Corrupt extent header
@@ -31,6 +39,7 @@ Inode 18, i_blocks is 2, should be 0.  Fix? yes
 Special (device/socket/fifo) file (inode 19) has extents
 or inline-data flag set.  Clear? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Entry 'fbad-flag' in / (2) has deleted/unused inode 18.  Clear? yes
 

[PATCH 55/54] e2fsck: actually fix inline_data flags problems when user says to do so

[Darrick J. Wong]

fix_problem() returning 1 means to fix the fs error, so do that.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c                          |    4 ++--
 tests/f_bad_disconnected_inode/expect.1 |   18 ------------------
 2 files changed, 2 insertions(+), 20 deletions(-)

diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 4a479b6..19e1f33 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -1268,12 +1268,12 @@ void e2fsck_pass1(e2fsck_t ctx)
 
 			pctx.errcode = ext2fs_inline_data_size(fs, ino, &size);
 			if (!pctx.errcode && size &&
-			    !fix_problem(ctx, PR_1_INLINE_DATA_FEATURE, &pctx)) {
+			    fix_problem(ctx, PR_1_INLINE_DATA_FEATURE, &pctx)) {
 				sb->s_feature_incompat |=
 					EXT4_FEATURE_INCOMPAT_INLINE_DATA;
 				ext2fs_mark_super_dirty(fs);
 				inlinedata_fs = 1;
-			} else if (!fix_problem(ctx, PR_1_INLINE_DATA_SET, &pctx)) {
+			} else if (fix_problem(ctx, PR_1_INLINE_DATA_SET, &pctx)) {
 				e2fsck_clear_inode(ctx, ino, inode, 0, "pass1");
 				/* skip FINISH_INODE_LOOP */
 				continue;
diff --git a/tests/f_bad_disconnected_inode/expect.1 b/tests/f_bad_disconnected_inode/expect.1
index 9f9b447..c5ea7bf 100644
--- a/tests/f_bad_disconnected_inode/expect.1
+++ b/tests/f_bad_disconnected_inode/expect.1
@@ -8,15 +8,9 @@ Clear? yes
 Inode 15 has INLINE_DATA_FL flag on filesystem without inline data support.
 Clear? yes
 
-Inode 15 has EXTENTS_FL flag set on filesystem without extents support.
-Clear? yes
-
 Inode 16 has INLINE_DATA_FL flag on filesystem without inline data support.
 Clear? yes
 
-Inode 16 has EXTENTS_FL flag set on filesystem without extents support.
-Clear? yes
-
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 /lost+found not found.  Create? yes
@@ -30,12 +24,6 @@ Clear? yes
 Inode 13 (...) has invalid mode (0117003).
 Clear? yes
 
-i_file_acl for inode 14 (...) is 2892851642, should be zero.
-Clear? yes
-
-Inode 14 (...) has invalid mode (0154247).
-Clear? yes
-
 Pass 5: Checking group summary information
 Block bitmap differences:  -(9--19)
 Fix? yes
@@ -46,15 +34,9 @@ Fix? yes
 Free blocks count wrong (79, counted=91).
 Fix? yes
 
-Free inodes count wrong for group #0 (6, counted=5).
-Fix? yes
-
 Directories count wrong for group #0 (3, counted=2).
 Fix? yes
 
-Free inodes count wrong (6, counted=5).
-Fix? yes

[PATCH 56/54] libext2fs: zero hash in ibody extended attributes

[Darrick J. Wong]

The kernel never updates the extended attribute hash value for
attributes stored in the inode.  However, fsck has always checked this
value (if it's nonzero) and will complain if the hash doesn't match
the xattr.  Therefore, always zero the hash value when writing to
in-ibody xattrs to avoid creating "corrupt" attribute errors
downstream.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/ext_attr.c |   12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/lib/ext2fs/ext_attr.c b/lib/ext2fs/ext_attr.c
index df512d8..f8fe691 100644
--- a/lib/ext2fs/ext_attr.c
+++ b/lib/ext2fs/ext_attr.c
@@ -434,7 +434,8 @@ static errcode_t write_xattrs_to_buffer(struct ext2_xattr_handle *handle,
 					struct ext2_xattr **pos,
 					void *entries_start,
 					unsigned int storage_size,
-					unsigned int value_offset_correction)
+					unsigned int value_offset_correction,
+					int write_hash)
 {
 	struct ext2_xattr *x = *pos;
 	struct ext2_ext_attr_entry *e = entries_start;
@@ -481,7 +482,10 @@ static errcode_t write_xattrs_to_buffer(struct ext2_xattr_handle *handle,
 		memcpy((void *)e + sizeof(*e), shortname, e->e_name_len);
 		memcpy(end, x->value, e->e_value_size);
 
-		e->e_hash = ext2fs_ext_attr_hash_entry(e, end);
+		if (write_hash)
+			e->e_hash = ext2fs_ext_attr_hash_entry(e, end);
+		else
+			e->e_hash = 0;
 
 		e = EXT2_EXT_ATTR_NEXT(e);
 		*(__u32 *)e = 0;
@@ -553,7 +557,7 @@ errcode_t ext2fs_xattrs_write(struct ext2_xattr_handle *handle)
 	start = ((char *) inode) + EXT2_GOOD_OLD_INODE_SIZE +
 		inode->i_extra_isize + sizeof(__u32);
 
-	err = write_xattrs_to_buffer(handle, &x, start, storage_size, 0);
+	err = write_xattrs_to_buffer(handle, &x, start, storage_size, 0, 0);
 	if (err)
 		goto out;
 
@@ -572,7 +576,7 @@ write_ea_block:
 	start = block_buf + sizeof(struct ext2_ext_attr_header);
 
 	err = write_xattrs_to_buffer(handle, &x, start, storage_size,
-				     (void *)start - block_buf);
+				     (void *)start - block_buf, 1);
 	if (err)
 		goto out2;
 

[PATCH 57/54] e2fsck: convert block-mapped files to extents on bigalloc fs

[Darrick J. Wong]

As of v4.0, the Linux kernel won't add blocks to a block-mapped file
on a bigalloc filesystem.  Therefore, convert any such files or
directories we find, to prevent fs errors later on.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c            |   17 +++++++++++++++++
 e2fsck/problem.c          |    5 +++++
 e2fsck/problem.h          |    3 +++
 tests/f_badcluster/expect |   26 +++++++++++++-------------
 4 files changed, 38 insertions(+), 13 deletions(-)

diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 19e1f33..308eb79 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -3201,6 +3201,23 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx,
 		pctx->num = 0;
 	}
 
+	/*
+	 * The kernel gets mad if we ask it to allocate bigalloc clusters to
+	 * a block mapped file, so rebuild it as an extent file.  We can skip
+	 * symlinks because they're never rewritten.
+	 */
+	if (EXT2_HAS_RO_COMPAT_FEATURE(fs->super,
+			EXT4_FEATURE_RO_COMPAT_BIGALLOC) &&
+	    (LINUX_S_ISREG(inode->i_mode) || LINUX_S_ISDIR(inode->i_mode)) &&
+	    ext2fs_inode_data_blocks2(fs, inode) > 0 &&
+	    (ino == EXT2_ROOT_INO || ino >= EXT2_FIRST_INO(fs->super)) &&
+	    !(inode->i_flags & (EXT4_EXTENTS_FL | EXT4_INLINE_DATA_FL)) &&
+	    fix_problem(ctx, PR_1_NO_BIGALLOC_BLOCKMAP_FILES, pctx)) {
+		pctx->errcode = e2fsck_rebuild_extents_later(ctx, ino);
+		if (pctx->errcode)
+			goto out;
+	}
+
 	if (ctx->dirs_to_hash && pb.is_dir &&
 	    !(ctx->lost_and_found && ctx->lost_and_found == ino) &&
 	    !(inode->i_flags & EXT2_INDEX_FL) &&
diff --git a/e2fsck/problem.c b/e2fsck/problem.c
index 7af0b76..df919da 100644
--- a/e2fsck/problem.c
+++ b/e2fsck/problem.c
@@ -1111,6 +1111,11 @@ static struct e2fsck_problem problem_table[] = {
 	  N_("@i %i @x tree could be more shallow (%b; could be <= %c)\n"),
 	  PROMPT_FIX, PR_NO_OK | PR_PREEN_NO | PR_PREEN_OK },
 
+	/* Inode extent tree could be more shallow */
+	{ PR_1_NO_BIGALLOC_BLOCKMAP_FILES,
+	  N_("@i %i on bigalloc @f cannot be @b mapped.  "),
+	  PROMPT_FIX, 0 },
+
 	/* Pass 1b errors */
 
 	/* Pass 1B: Rescan for duplicate/bad blocks */
diff --git a/e2fsck/problem.h b/e2fsck/problem.h
index ace8d2f..dc0c18a 100644
--- a/e2fsck/problem.h
+++ b/e2fsck/problem.h
@@ -648,6 +648,9 @@ struct problem_context {
 /* extent tree max depth too big */
 #define PR_1_EXTENT_BAD_MAX_DEPTH		0x01007F
 
+/* bigalloc fs cannot have blockmap files */
+#define PR_1_NO_BIGALLOC_BLOCKMAP_FILES		0x010080
+
 /*
  * Pass 1b errors
  */
diff --git a/tests/f_badcluster/expect b/tests/f_badcluster/expect
index b8ce19d..65a1641 100644
--- a/tests/f_badcluster/expect
+++ b/tests/f_badcluster/expect
@@ -19,6 +19,8 @@ Inode 18 logical block 3 (physical block 1201) violates cluster allocation rules
 Will fix in pass 1B.
 Inode 18, i_blocks is 32, should be 64.  Fix? yes
 
+Inode 15 on bigalloc filesystem cannot be block mapped.  Fix? yes
+
 
 Running additional passes to resolve blocks claimed by more than one inode...
 Pass 1B: Rescanning for multiply-claimed blocks
@@ -65,19 +67,20 @@ File /g (inode #18, mod time Tue Jun 17 08:00:50 2014)
   has 1 multiply-claimed block(s), shared with 0 file(s):
 Clone multiply-claimed blocks? yes
 
+Pass 1E: Optimizing extent trees
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-Free blocks count wrong for group #0 (50, counted=47).
+Free blocks count wrong for group #0 (51, counted=48).
 Fix? yes
 
-Free blocks count wrong (800, counted=752).
+Free blocks count wrong (816, counted=768).
 Fix? yes
 
 
 test_fs: ***** FILE SYSTEM WAS MODIFIED *****
-test_fs: 18/128 files (22.2% non-contiguous), 1296/2048 blocks
+test_fs: 18/128 files (22.2% non-contiguous), 1280/2048 blocks
 Pass 1: Checking inodes, blocks, and sizes
 Inode 12, i_blocks is 64, should be 32.  Fix? yes
 
@@ -94,21 +97,21 @@ Pass 5: Checking group summary information
 Block bitmap differences:  -(1168--1200)
 Fix? yes
 
-Free blocks count wrong for group #0 (47, counted=50).
+Free blocks count wrong for group #0 (48, counted=51).
 Fix? yes
 
-Free blocks count wrong (752, counted=800).
+Free blocks count wrong (768, counted=816).
 Fix? yes
 
 
 test_fs: ***** FILE SYSTEM WAS MODIFIED *****
-test_fs: 18/128 files (5.6% non-contiguous), 1248/2048 blocks
+test_fs: 18/128 files (5.6% non-contiguous), 1232/2048 blocks
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-test_fs: 18/128 files (5.6% non-contiguous), 1248/2048 blocks
+test_fs: 18/128 files (5.6% non-contiguous), 1232/2048 blocks
 debugfs: stat /a
 Inode: 12   Type: regular    Mode:  0644   Flags: 0x80000
 Generation: 1117152157    Version: 0x00000001
@@ -146,19 +149,16 @@ mtime: 0x539ff5b2 -- Tue Jun 17 08:00:50 2014
 EXTENTS:
 (0-1):1216-1217, (2):1218
 debugfs: stat /d
-Inode: 15   Type: regular    Mode:  0644   Flags: 0x0
+Inode: 15   Type: regular    Mode:  0644   Flags: 0x80000
 Generation: 1117152160    Version: 0x00000001
 User:     0   Group:     0   Size: 3072
 File ACL: 0    Directory ACL: 0
-Links: 1   Blockcount: 32
+Links: 1   Blockcount: 0
 Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x539ff5b2 -- Tue Jun 17 08:00:50 2014
 atime: 0x539ff5b2 -- Tue Jun 17 08:00:50 2014
 mtime: 0x539ff5b2 -- Tue Jun 17 08:00:50 2014
-BLOCKS:
-(TIND):1650
-TOTAL: 1

[PATCH 58/54] e2fsck: turn inline data symlink into a fast symlink when possible

[Darrick J. Wong]

When there's a problem accessing the EA part of an inline data symlink
and we want to truncate the symlink back to 60 characters (hoping the
user can re-establish the link later on, apparently) be sure to turn
off the inline data flag to convert the symlink back to a regular fast
symlink.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 2c5fca6..8c66c6d 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -1327,6 +1327,8 @@ void e2fsck_pass1(e2fsck_t ctx)
 						ctx->flags |= E2F_FLAG_ABORT;
 						goto endit;
 					}
+					if (LINUX_S_ISLNK(inode->i_mode))
+						inode->i_flags &= ~EXT4_INLINE_DATA_FL;
 					e2fsck_write_inode(ctx, ino, inode,
 							   "pass1");
 					failed_csum = 0;

Re: [PATCH 20/54] e2fsck: add a 'yes to all' response in interactive mode

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:44PM -0800, Darrick J. Wong wrote:
> Provide a mechanism for a user to switch fsck into '-y' mode if they
> start an interactive session and then get tired of pressing 'y' in
> response to numerous prompts.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Thanks, applied.  (Actually, this was applied a while ago, but I
didn't ack it apparently.)

					- Ted
					

Re: [PATCH 22/54] libext2fs: ext2fs_new_block2() should call alloc_block hook

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:57PM -0800, Darrick J. Wong wrote:
> If ext2fs_new_block2() is called without a specific block map, we
> should call the alloc_block hook before checking fs->block_map.  This
> helps us to avoid a bug in e2fsck where we need to allocate a block
> but instead of consulting block_found_map, we use the FS bitmaps,
> which (prior to pass 5) could be wrong.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Thanks, applied.

						- Ted

Re: [PATCH v2 23/54] libext2fs: Support readonly filesystem images

[Theodore Ts'o]

On Thu, Mar 19, 2015 at 02:32:37PM -0700, Darrick J. Wong wrote:
> Finish adding the new rocompat feature, "readonly", which marks a
> filesystem image read-only.  This also fixes a bug in Ted's patch to
> add the feature flag; RO_COMPAT_READONLY needs to be kept out of the
> RO_COMPAT supported feature list.

This really goes to the question of how many user space utilities
should be expected to be able to write a read-only file system.  I was
taking the position that the flag was primarily meant for the kernel,
and that it was fair game for e2fsck to repair a corrupted file
system, etc.

I can accept the position that we shouldn't enable applications which
use libext2fs to be able to modify READONLY file systems by default
--- but I'm not really comfortable putting RO_COMPAT_READONLY on
SOFTSUPP and then enabling e2fsck, resize2fs, et. al to manipulate all
SOFTSUPP features.  The original intent of SOFTSUPP was to allow
debugfs to be able to look at a file system which had some feature
that was under development, but where it was *not* yet safe to let
e2fsck to try to handle such a file system.

So if we want to remove RO_COMPAT_READONLY from the RO_COMPAT
supported feature list, we shouldn't add it to SOFTSUPP, but instead
manually enable it for each of the e2fsprogs program where we want to
allow them to work with a read-only file system.  Which begs the
question whether it makes sense to allow e2fsck to repair a corrupted
read-only file system, or to resize a read-only file system, without
first removing the read-only feature flag.  I think the answer is yes,
we should allow it, but it seems you think the answer should be no, at
least for the latter?

   	  	    	   	     - Ted

Re: [PATCH 21/54] libext2fs: zero blocks via FALLOC_FL_ZERO_RANGE in ext2fs_zero_blocks

[Theodore Ts'o]

On Mon, Jan 26, 2015 at 11:37:50PM -0800, Darrick J. Wong wrote:
> Plumb a new call into the IO manager to support translating
> ext2fs_zero_blocks calls into the equivalent FALLOC_FL_ZERO_RANGE
> fallocate flag primitive when possible.  This patch provides _only_
> support for file-based images.
> 
> v2: Remove zero-out for block devices until BLKZEROOUT is cleaned up.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Thanks, applied.

						- Ted

Re: [PATCH 55/54] e2fsck: actually fix inline_data flags problems when user says to do so

[Theodore Ts'o]

On Thu, Mar 19, 2015 at 02:44:49PM -0700, Darrick J. Wong wrote:
> fix_problem() returning 1 means to fix the fs error, so do that.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

					- Ted

Re: [PATCH 56/54] libext2fs: zero hash in ibody extended attributes

[Theodore Ts'o]

On Thu, Mar 19, 2015 at 02:45:51PM -0700, Darrick J. Wong wrote:
> The kernel never updates the extended attribute hash value for
> attributes stored in the inode.  However, fsck has always checked this
> value (if it's nonzero) and will complain if the hash doesn't match
> the xattr.  Therefore, always zero the hash value when writing to
> in-ibody xattrs to avoid creating "corrupt" attribute errors
> downstream.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Thanks, applied.

					- Ted