[PATCH 0/6] replace genext2fs with populate-extfs.sh

[PATCH 0/6] replace genext2fs with populate-extfs.sh

[Robert Yang]

* The benefits:
  - Really support ext4
  - Support the sparse file (we lost the sparse file in the image in the
    past, the sparse file became into the common file)
  - Fix the error reported by fsck: (ext2/ext3)
     "Inode 1025, i_size is 16384, should be 17408."
  - Have a uniform code for ext2/3/4 generation

* Impact
  - Build time:
    a) If we build fresh core-image-sato, there is nearly no impact.

    b) If we do the image generation, which means:
       $ bitbake core-image-sato
       $ bitbake core-image-sato -ccleansstate
       $ bitbake core-image-sato
   About 40 extra seconds are needed, here is my test result:
   Before the patches: 4m53s
   After the patches: 5m35s
   This is because the genext2fs is much faster than the 
   populate-extfs.sh, we will replace this script by the mke2fs when it
   supports create the filesystem from a initial directory.

  - Disk space (take core-image-sato as an example)
    a) The image file size is the same as before (519M)

    b) The disk usage is a little different: (du -sh)
              before       now 
       ext2:  356M         379M
       ext3:  372M         395M
       ext4:  372M         380M

   I have done some simple runtime testing on core-image-sato based on
   ext2/3/4, they worked well.

* The "fsck.extX -fn <image>.extX" shows it is OK.

// Robert

The following changes since commit b2ff1add530b1fec2fb7f385227a03db47015c37:

  poky.conf: Don't force the addition of extra DISTRO_FEATURES (2013-08-20 22:58:04 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib robert/extX
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/extX

Robert Yang (6):
  e2fsprogs: the max length of debugfs argument is too short
  e2fsprogs: let debugfs do sparse copy
  e2fsprogs: only update the icache for ext2_inode
  e2fsprogs: properly set up extent header in do_write
  e2fsprogs: add populate-extfs.sh
  image_types.bbclass: replace genext2fs with populate-extfs.sh

 meta/classes/image_types.bbclass                   |   46 +++---
 .../e2fsprogs-1.42.8/debugfs-extent-header.patch   |   47 +++++++
 .../e2fsprogs-1.42.8/debugfs-sparse-copy.patch     |  147 ++++++++++++++++++++
 .../e2fsprogs-1.42.8/debugfs-too-short.patch       |   41 ++++++
 .../e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch    |   69 +++++++++
 .../e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh   |   96 +++++++++++++
 .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    6 +
 7 files changed, 424 insertions(+), 28 deletions(-)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-extent-header.patch
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh

-- 
1.7.10.4

[PATCH 1/6] e2fsprogs: the max length of debugfs argument is too short

[Robert Yang]

The max length of debugfs argument is 256 which is too short, the
arguments are two paths, the PATH_MAX is 4096 according to
/usr/include/linux/limits.h, so use BUFSIZ (which is 8192 on Linux
systems), that's also what the ss library uses.

This patch has been reviewed by the linux-ext4 mailing list, but isn't
merged atm.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs-1.42.8/debugfs-too-short.patch       |   41 ++++++++++++++++++++
 .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
 2 files changed, 42 insertions(+)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch
new file mode 100644
index 0000000..607305b
--- /dev/null
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch
@@ -0,0 +1,41 @@
+debugfs.c: the max length of debugfs argument is too short
+
+The max length of debugfs argument is 256 which is too short, the
+arguments are two paths, the PATH_MAX is 4096 according to
+/usr/include/linux/limits.h, so use BUFSIZ (which is 8192 on Linux
+systems), that's also what the ss library uses.
+
+Upstream-Status: Submitted
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+Acked-by: Darren Hart <dvhart@linux.intel.com>
+---
+ debugfs/debugfs.c | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
+--- a/debugfs/debugfs.c
++++ b/debugfs/debugfs.c
+@@ -37,6 +37,10 @@ extern char *optarg;
+ #include "../version.h"
+ #include "jfs_user.h"
+ 
++#ifndef BUFSIZ
++#define BUFSIZ 8192
++#endif
++
+ ss_request_table *extra_cmds;
+ const char *debug_prog_name;
+ int sci_idx;
+@@ -2311,7 +2315,7 @@ void do_dump_mmp(int argc EXT2FS_ATTR((unused)), char *argv[])
+ static int source_file(const char *cmd_file, int ss_idx)
+ {
+ 	FILE		*f;
+-	char		buf[256];
++	char		buf[BUFSIZ];
+ 	char		*cp;
+ 	int		exit_status = 0;
+ 	int		retval;
+-- 
+1.8.1.2
+
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
index d231268..f97de7f 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
@@ -3,6 +3,7 @@ require e2fsprogs.inc
 
 SRC_URI += "file://acinclude.m4 \
             file://remove.ldconfig.call.patch \
+            file://debugfs-too-short.patch \
 "
 
 SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
-- 
1.7.10.4

[PATCH 2/6] e2fsprogs: let debugfs do sparse copy

[Robert Yang]

Let debugfs do sparse copy when src is a sparse file, just like
"cp --sparse=auto"

This patch has been reviewed by the linux-ext4 mailing list, but isn't
merged atm.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs-1.42.8/debugfs-sparse-copy.patch     |  147 ++++++++++++++++++++
 .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
 2 files changed, 148 insertions(+)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
new file mode 100644
index 0000000..c87bcbf
--- /dev/null
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
@@ -0,0 +1,147 @@
+debugfs.c: do sparse copy when src is a sparse file
+
+Let debugfs do sparse copy when src is a sparse file, just like
+"cp --sparse=auto"
+
+* For the:
+  #define IO_BUFSIZE 64*1024
+  this is a suggested value from gnu coreutils:
+  http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/ioblksize.h;h=1ae93255e7d0ccf0855208c7ae5888209997bf16;hb=HEAD
+
+* Use malloc() to allocate memory for the buffer since put 64K (or
+  more) on the stack seems not a good idea.
+
+Upstream-Status: Submitted
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+Acked-by: Darren Hart <dvhart@linux.intel.com>
+---
+ debugfs/debugfs.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----
+ 1 file changed, 57 insertions(+), 4 deletions(-)
+
+diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
+--- a/debugfs/debugfs.c
++++ b/debugfs/debugfs.c
+@@ -41,6 +41,16 @@ extern char *optarg;
+ #define BUFSIZ 8192
+ #endif
+ 
++/* 64KiB is the minimium blksize to best minimize system call overhead. */
++#ifndef IO_BUFSIZE
++#define IO_BUFSIZE 64*1024
++#endif
++
++/* Block size for `st_blocks' */
++#ifndef S_BLKSIZE
++#define S_BLKSIZE 512
++#endif
++
+ ss_request_table *extra_cmds;
+ const char *debug_prog_name;
+ int sci_idx;
+@@ -1575,22 +1585,36 @@ void do_find_free_inode(int argc, char *argv[])
+ }
+ 
+ #ifndef READ_ONLY
+-static errcode_t copy_file(int fd, ext2_ino_t newfile)
++static errcode_t copy_file(int fd, ext2_ino_t newfile, int bufsize, int make_holes)
+ {
+ 	ext2_file_t	e2_file;
+ 	errcode_t	retval;
+ 	int		got;
+ 	unsigned int	written;
+-	char		buf[8192];
++	char		*buf;
+ 	char		*ptr;
++	char		*zero_buf;
++	int		cmp;
+ 
+ 	retval = ext2fs_file_open(current_fs, newfile,
+ 				  EXT2_FILE_WRITE, &e2_file);
+ 	if (retval)
+ 		return retval;
+ 
++	if (!(buf = (char *) malloc(bufsize))){
++		com_err("copy_file", errno, "can't allocate buffer\n");
++		return;
++	}
++
++        /* This is used for checking whether the whole block is zero */
++	retval = ext2fs_get_memzero(bufsize, &zero_buf);
++	if (retval) {
++		com_err("copy_file", retval, "can't allocate buffer\n");
++		return retval;
++	}
++
+ 	while (1) {
+-		got = read(fd, buf, sizeof(buf));
++		got = read(fd, buf, bufsize);
+ 		if (got == 0)
+ 			break;
+ 		if (got < 0) {
+@@ -1598,6 +1622,21 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
+ 			goto fail;
+ 		}
+ 		ptr = buf;
++
++		/* Sparse copy */
++		if (make_holes) {
++			/* Check whether all is zero */
++			cmp = memcmp(ptr, zero_buf, got);
++			if (cmp == 0) {
++				 /* The whole block is zero, make a hole */
++				retval = ext2fs_file_lseek(e2_file, got, EXT2_SEEK_CUR, NULL);
++				if (retval)
++					goto fail;
++				got = 0;
++			}
++		}
++
++		/* Normal copy */
+ 		while (got > 0) {
+ 			retval = ext2fs_file_write(e2_file, ptr,
+ 						   got, &written);
+@@ -1608,10 +1647,14 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
+ 			ptr += written;
+ 		}
+ 	}
++	free(buf);
++	ext2fs_free_mem(&zero_buf);
+ 	retval = ext2fs_file_close(e2_file);
+ 	return retval;
+ 
+ fail:
++	free(buf);
++	ext2fs_free_mem(&zero_buf);
+ 	(void) ext2fs_file_close(e2_file);
+ 	return retval;
+ }
+@@ -1624,6 +1667,8 @@ void do_write(int argc, char *argv[])
+ 	ext2_ino_t	newfile;
+ 	errcode_t	retval;
+ 	struct ext2_inode inode;
++	int		bufsize = IO_BUFSIZE;
++	int		make_holes = 0;
+ 
+ 	if (common_args_process(argc, argv, 3, 3, "write",
+ 				"<native file> <new file>", CHECK_FS_RW))
+@@ -1699,7 +1744,15 @@ void do_write(int argc, char *argv[])
+ 		return;
+ 	}
+ 	if (LINUX_S_ISREG(inode.i_mode)) {
+-		retval = copy_file(fd, newfile);
++		if (statbuf.st_blocks < statbuf.st_size / S_BLKSIZE) {
++			make_holes = 1;
++			/*
++			 * Use I/O blocksize as buffer size when
++			 * copying sparse files.
++			 */
++			bufsize = statbuf.st_blksize;
++		}
++		retval = copy_file(fd, newfile, bufsize, make_holes);
+ 		if (retval)
+ 			com_err("copy_file", retval, 0);
+ 	}
+-- 
+1.8.1.2
+
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
index f97de7f..5d65bbc 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
@@ -4,6 +4,7 @@ require e2fsprogs.inc
 SRC_URI += "file://acinclude.m4 \
             file://remove.ldconfig.call.patch \
             file://debugfs-too-short.patch \
+            file://debugfs-sparse-copy.patch \
 "
 
 SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
-- 
1.7.10.4

[PATCH 3/6] e2fsprogs: only update the icache for ext2_inode

[Robert Yang]

We only read the cache when:

bufsize == sizeof(struct ext2_inode)

then we should only update the cache in the same condition, otherwise
there would be errors, for example:

cache[0]: cached ino 14 when bufsize = 128 by ext2fs_write_inode_full()
cache[1]: cached ino 14 when bufsize = 156 by ext2fs_read_inode_full()

Then update the cache:
cache[0]: cached ino 15 when bufsize = 156 by ext2fs_read_inode_full()

Then the ino 14 would hit the cache[1] when bufsize = 128 (but it was
cached by bufsize = 156), so there would be errors.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch    |   69 ++++++++++++++++++++
 .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
 2 files changed, 70 insertions(+)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch
new file mode 100644
index 0000000..ad4e343
--- /dev/null
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch
@@ -0,0 +1,69 @@
+inode.c: only update the icache for ext2_inode
+
+We only read the cache when:
+
+bufsize == sizeof(struct ext2_inode)
+
+then we should only update the cache in the same condition, otherwise
+there would be errors, for example:
+
+cache[0]: cached ino 14 when bufsize = 128 by ext2fs_write_inode_full()
+cache[1]: cached ino 14 when bufsize = 156 by ext2fs_read_inode_full()
+
+Then update the cache:
+cache[0]: cached ino 15 when bufsize = 156 by ext2fs_read_inode_full()
+
+Then the ino 14 would hit the cache[1] when bufsize = 128 (but it was
+cached by bufsize = 156), so there would be errors.
+
+Note: the upstream has changed the icache lot, so this patch is
+inappropriate for the upstream, we can drop this patch when we update
+the package.
+
+Upstream-Status: [Inappropriate]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ lib/ext2fs/inode.c | 20 ++++++++++++--------
+ 1 file changed, 12 insertions(+), 8 deletions(-)
+
+diff --git a/lib/ext2fs/inode.c b/lib/ext2fs/inode.c
+--- a/lib/ext2fs/inode.c
++++ b/lib/ext2fs/inode.c
+@@ -612,10 +612,12 @@ errcode_t ext2fs_read_inode_full(ext2_filsys fs, ext2_ino_t ino,
+ #endif
+ 
+ 	/* Update the inode cache */
+-	fs->icache->cache_last = (fs->icache->cache_last + 1) %
+-		fs->icache->cache_size;
+-	fs->icache->cache[fs->icache->cache_last].ino = ino;
+-	fs->icache->cache[fs->icache->cache_last].inode = *inode;
++	if (bufsize == sizeof(struct ext2_inode)) {
++		fs->icache->cache_last = (fs->icache->cache_last + 1) %
++			fs->icache->cache_size;
++		fs->icache->cache[fs->icache->cache_last].ino = ino;
++		fs->icache->cache[fs->icache->cache_last].inode = *inode;
++	}
+ 
+ 	return 0;
+ }
+@@ -648,10 +650,12 @@ errcode_t ext2fs_write_inode_full(ext2_filsys fs, ext2_ino_t ino,
+ 
+ 	/* Check to see if the inode cache needs to be updated */
+ 	if (fs->icache) {
+-		for (i=0; i < fs->icache->cache_size; i++) {
+-			if (fs->icache->cache[i].ino == ino) {
+-				fs->icache->cache[i].inode = *inode;
+-				break;
++		if (bufsize == sizeof(struct ext2_inode)) {
++			for (i=0; i < fs->icache->cache_size; i++) {
++				if (fs->icache->cache[i].ino == ino) {
++					fs->icache->cache[i].inode = *inode;
++					break;
++				}
+ 			}
+ 		}
+ 	} else {
+-- 
+1.8.1.2
+
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
index 5d65bbc..b063bc0 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
@@ -5,6 +5,7 @@ SRC_URI += "file://acinclude.m4 \
             file://remove.ldconfig.call.patch \
             file://debugfs-too-short.patch \
             file://debugfs-sparse-copy.patch \
+            file://fix-icache.patch \
 "
 
 SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
-- 
1.7.10.4

[PATCH 4/6] e2fsprogs: properly set up extent header in do_write

[Robert Yang]

do_write doesn't fully set up the first extent header on a new
inode, so if we write a 0-length file, and don't write any data
to the new file, we end up creating something that looks corrupt
to kernelspace:

EXT4-fs error (device loop0): ext4_ext_check_inode:464: inode #12: comm
ls: bad header/extent: invalid magic - magic 0, entries 0, max 0(0),
depth 0(0)

Do something similar to ext4_ext_tree_init() here, and
fill out the first extent header upon creation to avoid this.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs-1.42.8/debugfs-extent-header.patch   |   47 ++++++++++++++++++++
 .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    2 +
 2 files changed, 49 insertions(+)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-extent-header.patch

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-extent-header.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-extent-header.patch
new file mode 100644
index 0000000..ae44730
--- /dev/null
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-extent-header.patch
@@ -0,0 +1,47 @@
+debugfs: properly set up extent header in do_write
+
+do_write doesn't fully set up the first extent header on a new
+inode, so if we write a 0-length file, and don't write any data
+to the new file, we end up creating something that looks corrupt
+to kernelspace:
+
+EXT4-fs error (device loop0): ext4_ext_check_inode:464: inode #12: comm ls: bad header/extent: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
+
+Do something similar to ext4_ext_tree_init() here, and
+fill out the first extent header upon creation to avoid this.
+
+Upstream-Status: Backport
+
+Reported-by: Robert Yang <liezhi.yang@windriver.com>
+Signed-off-by: Eric Sandeen <sandeen@redhat.com>
+---
+ debugfs/debugfs.c | 13 ++++++++++++-
+ 1 file changed, 12 insertions(+), 1 deletion(-)
+
+diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
+--- a/debugfs/debugfs.c
++++ b/debugfs/debugfs.c
+@@ -1726,8 +1726,19 @@ void do_write(int argc, char *argv[])
+ 	inode.i_links_count = 1;
+ 	inode.i_size = statbuf.st_size;
+ 	if (current_fs->super->s_feature_incompat &
+-	    EXT3_FEATURE_INCOMPAT_EXTENTS)
++	    EXT3_FEATURE_INCOMPAT_EXTENTS) {
++		int i;
++		struct ext3_extent_header *eh;
++
++		eh = (struct ext3_extent_header *) &inode.i_block[0];
++		eh->eh_depth = 0;
++		eh->eh_entries = 0;
++		eh->eh_magic = EXT3_EXT_MAGIC;
++		i = (sizeof(inode.i_block) - sizeof(*eh)) /
++			sizeof(struct ext3_extent);
++		eh->eh_max = ext2fs_cpu_to_le16(i);
+ 		inode.i_flags |= EXT4_EXTENTS_FL;
++	}
+ 	if (debugfs_write_new_inode(newfile, &inode, argv[0])) {
+ 		close(fd);
+ 		return;
+-- 
+1.8.1.2
+
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
index b063bc0..2dc9dab 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
@@ -6,6 +6,8 @@ SRC_URI += "file://acinclude.m4 \
             file://debugfs-too-short.patch \
             file://debugfs-sparse-copy.patch \
             file://fix-icache.patch \
+            file://debugfs-extent-header.patch \
+            file://populate-extfs.sh \
 "
 
 SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
-- 
1.7.10.4

[PATCH 5/6] e2fsprogs: add populate-extfs.sh

[Robert Yang]

This script is originally from Darren Hart, it will be used for creating
the ext* filesystem from a given directory, which will replace the
genext2fs in image_types.bbclass at the moment, we may use the mke2fs to
replace this script again when it has the initial directory support.

Changes of the script:
* Rename it from mkdebugfs.sh to populate-extfs.sh
* Add a simple usage
* Add checking for the number of the parameters
* Add the "regular empty file" and "fifo" file type
* Set mode, uid and gid for the file
* Save the command lines to a file and batch run them
* Change the error message
* Improve the performance
* Add the support for hardlink

[YOCTO #3848]

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh   |   96 ++++++++++++++++++++
 .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
 2 files changed, 97 insertions(+)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh
new file mode 100644
index 0000000..9eff030
--- /dev/null
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh
@@ -0,0 +1,96 @@
+#!/bin/sh
+
+do_usage () {
+	cat << _EOF
+Usage: populate-extfs.sh <source> <device>
+Create an ext2/ext3/ext4 filesystem from a directory or file
+
+  source: The source directory or file
+  device: The target device
+
+_EOF
+	exit 1
+}
+
+[ $# -ne 2 ] && do_usage
+
+SRCDIR=${1%%/}
+DEVICE=$2
+DEBUGFS="debugfs"
+
+{
+	CWD="/"
+	find $SRCDIR | while read FILE; do
+                TGT="${FILE##*/}"
+                DIR="${FILE#$SRCDIR}"
+                DIR="${DIR%$TGT}"
+
+		# Skip the root dir
+		[ ! -z "$DIR" ] || continue
+		[ ! -z "$TGT" ] || continue
+
+		if [ "$DIR" != "$CWD" ]; then
+			echo "cd $DIR"
+			CWD="$DIR"
+		fi
+
+		# Only stat once since stat is a time consuming command
+		STAT=$(stat -c "TYPE=\"%F\";DEVNO=\"0x%t 0x%T\";MODE=\"%f\";U=\"%u\";G=\"%g\"" $FILE)
+		eval $STAT
+
+		case $TYPE in
+		"directory")
+			echo "mkdir $TGT"
+			;;
+		"regular file" | "regular empty file")
+			echo "write $FILE $TGT"
+			;;
+		"symbolic link")
+			LINK_TGT=$(readlink $FILE)
+			echo "symlink $TGT $LINK_TGT"
+			;;
+		"block special file")
+			echo "mknod $TGT b $DEVNO"
+			;;
+		"character special file")
+			echo "mknod $TGT c $DEVNO"
+			;;
+		"fifo")
+			echo "mknod $TGT p"
+			;;
+		*)
+			echo "Unknown/unhandled file type '$TYPE' file: $FILE" 1>&2
+			;;
+		esac
+
+		# Set the file mode
+		echo "sif $TGT mode 0x$MODE"
+
+		# Set uid and gid
+		echo "sif $TGT uid $U"
+		echo "sif $TGT gid $G"
+	done
+
+	# Handle the hard links.
+	# Save the hard links to a file, use the inode number as the filename, for example:
+	# If a and b's inode number is 6775928, save a and b to /tmp/tmp.VrCwHh5gdt/6775928.
+	INODE_DIR=`mktemp -d` || exit 1
+	for i in `find $SRCDIR -type f -links +1 -printf 'INODE=%i###FN=%p\n'`; do
+		eval `echo $i | sed 's$###$ $'`
+		echo ${FN#$SRCDIR} >>$INODE_DIR/$INODE
+	done
+	# Use the debugfs' ln and "sif links_count" to handle them.
+	for i in `ls $INODE_DIR`; do
+		# The link source
+		SRC=`head -1 $INODE_DIR/$i`
+		# Remove the files and link them again except the first one
+		for TGT in `sed -n -e '1!p' $INODE_DIR/$i`; do
+			echo "rm $TGT"
+			echo "ln $SRC $TGT"
+		done
+		LN_CNT=`cat $INODE_DIR/$i | wc -l`
+		# Set the links count
+		echo "sif $SRC links_count $LN_CNT"
+	done
+	rm -fr $INODE_DIR
+} | $DEBUGFS -w -f - $DEVICE
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
index 2dc9dab..8b69b22 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
@@ -45,6 +45,7 @@ do_install_append () {
 		mv ${D}${base_libdir}/e2initrd_helper ${D}${libdir}
 		mv ${D}${base_libdir}/pkgconfig ${D}${libdir}
 	fi
+	install -m 0755 ${WORKDIR}/populate-extfs.sh ${D}${bindir}
 }
 
 RDEPENDS_e2fsprogs = "e2fsprogs-badblocks"
-- 
1.7.10.4

[PATCH 6/6] image_types.bbclass: replace genext2fs with populate-extfs.sh

[Robert Yang]

* The benefits:
  - Really support ext4

  - Support the sparse file (we lost the sparse file in the image in the
    past, the sparse file became into the common file)

  - Fix the error reported by fsck: (ext2/ext3)
      Inode 1025, i_size is 16384, should be 17408.

  - Have a uniform code for ext2/3/4 generation

* Comments from Darren Hart:
Basically, genext2fs doesn't support creating ext4 filesystems. It
creates, as I understand it, an ext2 filesystem, then adds a journal,
and sets some bits. It can't support the newer features like extents. So
what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated
solution for generating prepopulated filesystem images as many other
mkfs* tools do. One thing missing was symlink support in libext2fs. I
added that support and demonstrated a script which uses the e2fsprogs
debugfs tool that can populate the newly formatted filesystem from a
directory and without root privileges.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/image_types.bbclass |   46 +++++++++++++++-----------------------
 1 file changed, 18 insertions(+), 28 deletions(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 0e5a9a8..c168ed5 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -141,34 +141,24 @@ IMAGE_CMD_sum.jffs2 = "${IMAGE_CMD_jffs2} && sumtool -i ${DEPLOY_DIR_IMAGE}/${IM
 
 IMAGE_CMD_cramfs = "mkcramfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cramfs ${EXTRA_IMAGECMD}"
 
-IMAGE_CMD_ext2 () {
-	rm -rf ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN} && mkdir ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}/${IMAGE_NAME}.rootfs.ext2
-	mv ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}/${IMAGE_NAME}.rootfs.ext2 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext2
-	rmdir ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}
-}
+oe_mkext234fs () {
+	fstype=$1
+	extra_imagecmd=""
 
-IMAGE_CMD_ext3 () {
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext3
-	tune2fs -j ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext3
-}
+	if [ $# -gt 1 ]; then
+		shift
+		extra_imagecmd=$@
+	fi
 
-oe_mkext4fs () {
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} $1
-	tune2fs -O extents,uninit_bg,dir_index,has_journal,filetype $1
-	e2fsck -yfDC0 $1 || chk=$?
-	case $chk in
-	0|1|2)
-	    ;;
-	*)
-	    return $chk
-	    ;;
-	esac
+	# Create a sparse image block
+	dd if=/dev/zero of=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype seek=$ROOTFS_SIZE count=0 bs=1k
+	mkfs.$fstype -F $extra_imagecmd ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype
+	populate-extfs.sh ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype
 }
 
-IMAGE_CMD_ext4 () {
-	oe_mkext4fs ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext4
-}
+IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
+IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
+IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
 
 IMAGE_CMD_btrfs () {
 	mkfs.btrfs -b `expr ${ROOTFS_SIZE} \* 1024` ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.btrfs
@@ -218,7 +208,7 @@ JFFS2_ENDIANNESS ?= "${@base_conditional('SITEINFO_ENDIANNESS', 'le', '--little-
 JFFS2_ERASEBLOCK ?= "0x40000"
 EXTRA_IMAGECMD_jffs2 ?= "--pad ${JFFS2_ENDIANNESS} --eraseblock=${JFFS2_ERASEBLOCK} --no-cleanmarkers"
 
-# Change these if you want default genext2fs behavior (i.e. create minimal inode number)
+# Change these if you want default mkfs behavior (i.e. create minimal inode number)
 EXTRA_IMAGECMD_ext2 ?= "-i 8192"
 EXTRA_IMAGECMD_ext3 ?= "-i 8192"
 EXTRA_IMAGECMD_ext4 ?= "-i 8192"
@@ -229,9 +219,9 @@ IMAGE_DEPENDS = ""
 IMAGE_DEPENDS_jffs2 = "mtd-utils-native"
 IMAGE_DEPENDS_sum.jffs2 = "mtd-utils-native"
 IMAGE_DEPENDS_cramfs = "cramfs-native"
-IMAGE_DEPENDS_ext2 = "genext2fs-native"
-IMAGE_DEPENDS_ext3 = "genext2fs-native e2fsprogs-native"
-IMAGE_DEPENDS_ext4 = "genext2fs-native e2fsprogs-native"
+IMAGE_DEPENDS_ext2 = "e2fsprogs-native"
+IMAGE_DEPENDS_ext3 = "e2fsprogs-native"
+IMAGE_DEPENDS_ext4 = "e2fsprogs-native"
 IMAGE_DEPENDS_btrfs = "btrfs-tools-native"
 IMAGE_DEPENDS_squashfs = "squashfs-tools-native"
 IMAGE_DEPENDS_squashfs-xz = "squashfs-tools-native"
-- 
1.7.10.4

Re: [PATCH 1/6] e2fsprogs: the max length of debugfs argument is too short

[Darren Hart]

On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
> The max length of debugfs argument is 256 which is too short, the
> arguments are two paths, the PATH_MAX is 4096 according to
> /usr/include/linux/limits.h, so use BUFSIZ (which is 8192 on Linux
> systems), that's also what the ss library uses.
> 
> This patch has been reviewed by the linux-ext4 mailing list, but isn't
> merged atm.
> 
> [YOCTO #3848]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>

Acked-by: Darren Hart <dvhart@linux.intel.com>

> ---
>  .../e2fsprogs-1.42.8/debugfs-too-short.patch       |   41 ++++++++++++++++++++
>  .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
>  2 files changed, 42 insertions(+)
>  create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch
> 
> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch
> new file mode 100644
> index 0000000..607305b
> --- /dev/null
> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch
> @@ -0,0 +1,41 @@
> +debugfs.c: the max length of debugfs argument is too short
> +
> +The max length of debugfs argument is 256 which is too short, the
> +arguments are two paths, the PATH_MAX is 4096 according to
> +/usr/include/linux/limits.h, so use BUFSIZ (which is 8192 on Linux
> +systems), that's also what the ss library uses.
> +
> +Upstream-Status: Submitted
> +
> +Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> +Acked-by: Darren Hart <dvhart@linux.intel.com>
> +---
> + debugfs/debugfs.c | 6 +++++-
> + 1 file changed, 5 insertions(+), 1 deletion(-)
> +
> +diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
> +--- a/debugfs/debugfs.c
> ++++ b/debugfs/debugfs.c
> +@@ -37,6 +37,10 @@ extern char *optarg;
> + #include "../version.h"
> + #include "jfs_user.h"
> + 
> ++#ifndef BUFSIZ
> ++#define BUFSIZ 8192
> ++#endif
> ++
> + ss_request_table *extra_cmds;
> + const char *debug_prog_name;
> + int sci_idx;
> +@@ -2311,7 +2315,7 @@ void do_dump_mmp(int argc EXT2FS_ATTR((unused)), char *argv[])
> + static int source_file(const char *cmd_file, int ss_idx)
> + {
> + 	FILE		*f;
> +-	char		buf[256];
> ++	char		buf[BUFSIZ];
> + 	char		*cp;
> + 	int		exit_status = 0;
> + 	int		retval;
> +-- 
> +1.8.1.2
> +
> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> index d231268..f97de7f 100644
> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> @@ -3,6 +3,7 @@ require e2fsprogs.inc
>  
>  SRC_URI += "file://acinclude.m4 \
>              file://remove.ldconfig.call.patch \
> +            file://debugfs-too-short.patch \
>  "
>  
>  SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 2/6] e2fsprogs: let debugfs do sparse copy

[Darren Hart]

On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
> Let debugfs do sparse copy when src is a sparse file, just like
> "cp --sparse=auto"
> 
> This patch has been reviewed by the linux-ext4 mailing list, but isn't
> merged atm.
> 
> [YOCTO #3848]

Minor whitespace nit below, but will likely be caught by upstream.

--
Darren

> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  .../e2fsprogs-1.42.8/debugfs-sparse-copy.patch     |  147 ++++++++++++++++++++
>  .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
>  2 files changed, 148 insertions(+)
>  create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> 
> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> new file mode 100644
> index 0000000..c87bcbf
> --- /dev/null
> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> @@ -0,0 +1,147 @@
> +debugfs.c: do sparse copy when src is a sparse file
> +
> +Let debugfs do sparse copy when src is a sparse file, just like
> +"cp --sparse=auto"
> +
> +* For the:
> +  #define IO_BUFSIZE 64*1024
> +  this is a suggested value from gnu coreutils:
> +  http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/ioblksize.h;h=1ae93255e7d0ccf0855208c7ae5888209997bf16;hb=HEAD
> +
> +* Use malloc() to allocate memory for the buffer since put 64K (or
> +  more) on the stack seems not a good idea.
> +
> +Upstream-Status: Submitted
> +
> +Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> +Acked-by: Darren Hart <dvhart@linux.intel.com>
> +---
> + debugfs/debugfs.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----
> + 1 file changed, 57 insertions(+), 4 deletions(-)
> +
> +diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
> +--- a/debugfs/debugfs.c
> ++++ b/debugfs/debugfs.c
> +@@ -41,6 +41,16 @@ extern char *optarg;
> + #define BUFSIZ 8192
> + #endif
> + 
> ++/* 64KiB is the minimium blksize to best minimize system call overhead. */
> ++#ifndef IO_BUFSIZE
> ++#define IO_BUFSIZE 64*1024
> ++#endif
> ++
> ++/* Block size for `st_blocks' */
> ++#ifndef S_BLKSIZE
> ++#define S_BLKSIZE 512
> ++#endif
> ++
> + ss_request_table *extra_cmds;
> + const char *debug_prog_name;
> + int sci_idx;
> +@@ -1575,22 +1585,36 @@ void do_find_free_inode(int argc, char *argv[])
> + }
> + 
> + #ifndef READ_ONLY
> +-static errcode_t copy_file(int fd, ext2_ino_t newfile)
> ++static errcode_t copy_file(int fd, ext2_ino_t newfile, int bufsize, int make_holes)
> + {
> + 	ext2_file_t	e2_file;
> + 	errcode_t	retval;
> + 	int		got;
> + 	unsigned int	written;
> +-	char		buf[8192];
> ++	char		*buf;
> + 	char		*ptr;
> ++	char		*zero_buf;
> ++	int		cmp;
> + 
> + 	retval = ext2fs_file_open(current_fs, newfile,
> + 				  EXT2_FILE_WRITE, &e2_file);
> + 	if (retval)
> + 		return retval;
> + 
> ++	if (!(buf = (char *) malloc(bufsize))){
> ++		com_err("copy_file", errno, "can't allocate buffer\n");
> ++		return;
> ++	}
> ++
> ++        /* This is used for checking whether the whole block is zero */

Whitespace/indentation error ^


> ++	retval = ext2fs_get_memzero(bufsize, &zero_buf);
> ++	if (retval) {
> ++		com_err("copy_file", retval, "can't allocate buffer\n");
> ++		return retval;
> ++	}
> ++
> + 	while (1) {
> +-		got = read(fd, buf, sizeof(buf));
> ++		got = read(fd, buf, bufsize);
> + 		if (got == 0)
> + 			break;
> + 		if (got < 0) {
> +@@ -1598,6 +1622,21 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
> + 			goto fail;
> + 		}
> + 		ptr = buf;
> ++
> ++		/* Sparse copy */
> ++		if (make_holes) {
> ++			/* Check whether all is zero */
> ++			cmp = memcmp(ptr, zero_buf, got);
> ++			if (cmp == 0) {
> ++				 /* The whole block is zero, make a hole */
> ++				retval = ext2fs_file_lseek(e2_file, got, EXT2_SEEK_CUR, NULL);
> ++				if (retval)
> ++					goto fail;
> ++				got = 0;
> ++			}
> ++		}
> ++
> ++		/* Normal copy */
> + 		while (got > 0) {
> + 			retval = ext2fs_file_write(e2_file, ptr,
> + 						   got, &written);
> +@@ -1608,10 +1647,14 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
> + 			ptr += written;
> + 		}
> + 	}
> ++	free(buf);
> ++	ext2fs_free_mem(&zero_buf);
> + 	retval = ext2fs_file_close(e2_file);
> + 	return retval;
> + 
> + fail:
> ++	free(buf);
> ++	ext2fs_free_mem(&zero_buf);
> + 	(void) ext2fs_file_close(e2_file);
> + 	return retval;
> + }
> +@@ -1624,6 +1667,8 @@ void do_write(int argc, char *argv[])
> + 	ext2_ino_t	newfile;
> + 	errcode_t	retval;
> + 	struct ext2_inode inode;
> ++	int		bufsize = IO_BUFSIZE;
> ++	int		make_holes = 0;
> + 
> + 	if (common_args_process(argc, argv, 3, 3, "write",
> + 				"<native file> <new file>", CHECK_FS_RW))
> +@@ -1699,7 +1744,15 @@ void do_write(int argc, char *argv[])
> + 		return;
> + 	}
> + 	if (LINUX_S_ISREG(inode.i_mode)) {
> +-		retval = copy_file(fd, newfile);
> ++		if (statbuf.st_blocks < statbuf.st_size / S_BLKSIZE) {
> ++			make_holes = 1;
> ++			/*
> ++			 * Use I/O blocksize as buffer size when
> ++			 * copying sparse files.
> ++			 */
> ++			bufsize = statbuf.st_blksize;
> ++		}
> ++		retval = copy_file(fd, newfile, bufsize, make_holes);
> + 		if (retval)
> + 			com_err("copy_file", retval, 0);
> + 	}
> +-- 
> +1.8.1.2
> +
> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> index f97de7f..5d65bbc 100644
> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> @@ -4,6 +4,7 @@ require e2fsprogs.inc
>  SRC_URI += "file://acinclude.m4 \
>              file://remove.ldconfig.call.patch \
>              file://debugfs-too-short.patch \
> +            file://debugfs-sparse-copy.patch \
>  "
>  
>  SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 3/6] e2fsprogs: only update the icache for ext2_inode

[Darren Hart]

On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
> We only read the cache when:
> 
> bufsize == sizeof(struct ext2_inode)
> 
> then we should only update the cache in the same condition, otherwise
> there would be errors, for example:
> 
> cache[0]: cached ino 14 when bufsize = 128 by ext2fs_write_inode_full()
> cache[1]: cached ino 14 when bufsize = 156 by ext2fs_read_inode_full()
> 
> Then update the cache:
> cache[0]: cached ino 15 when bufsize = 156 by ext2fs_read_inode_full()
> 
> Then the ino 14 would hit the cache[1] when bufsize = 128 (but it was
> cached by bufsize = 156), so there would be errors.

> [YOCTO #3848]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>


> +Upstream-Status: [Inappropriate]

Hrm... why is this one inappropriate?

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 4/6] e2fsprogs: properly set up extent header in do_write

[Darren Hart]

On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
> do_write doesn't fully set up the first extent header on a new
> inode, so if we write a 0-length file, and don't write any data
> to the new file, we end up creating something that looks corrupt
> to kernelspace:
> 
> EXT4-fs error (device loop0): ext4_ext_check_inode:464: inode #12: comm
> ls: bad header/extent: invalid magic - magic 0, entries 0, max 0(0),
> depth 0(0)
> 
> Do something similar to ext4_ext_tree_init() here, and
> fill out the first extent header upon creation to avoid this.
> 
> [YOCTO #3848]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  .../e2fsprogs-1.42.8/debugfs-extent-header.patch   |   47 ++++++++++++++++++++
>  .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    2 +

> +Upstream-Status: Backport

Should we backport? Or should we just update the revision we use?

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 5/6] e2fsprogs: add populate-extfs.sh

[Darren Hart]

On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
> This script is originally from Darren Hart, it will be used for creating
> the ext* filesystem from a given directory, which will replace the
> genext2fs in image_types.bbclass at the moment, we may use the mke2fs to
> replace this script again when it has the initial directory support.
> 
> Changes of the script:
> * Rename it from mkdebugfs.sh to populate-extfs.sh
> * Add a simple usage
> * Add checking for the number of the parameters
> * Add the "regular empty file" and "fifo" file type
> * Set mode, uid and gid for the file
> * Save the command lines to a file and batch run them
> * Change the error message
> * Improve the performance
> * Add the support for hardlink
> 
> [YOCTO #3848]

> Signed-off-by: Darren Hart <dvhart@linux.intel.com>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>


Ted has offered to add this to the contrib dir for the ext2 tools. This
is fine to add to yocto, but we should contribute this back upstream as
well.


-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 3/6] e2fsprogs: only update the icache for ext2_inode

[Robert Yang]

On 08/23/2013 01:23 AM, Darren Hart wrote:
> On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
>> We only read the cache when:
>>
>> bufsize == sizeof(struct ext2_inode)
>>
>> then we should only update the cache in the same condition, otherwise
>> there would be errors, for example:
>>
>> cache[0]: cached ino 14 when bufsize = 128 by ext2fs_write_inode_full()
>> cache[1]: cached ino 14 when bufsize = 156 by ext2fs_read_inode_full()
>>
>> Then update the cache:
>> cache[0]: cached ino 15 when bufsize = 156 by ext2fs_read_inode_full()
>>
>> Then the ino 14 would hit the cache[1] when bufsize = 128 (but it was
>> cached by bufsize = 156), so there would be errors.
>
>> [YOCTO #3848]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>
>
>> +Upstream-Status: [Inappropriate]
>
> Hrm... why is this one inappropriate?
>

Hi Darren, the upstream has changed the icache lot, so this patch is
inappropriate for the upstream, we can drop this patch when we update
the package, I had put this in the patch head.

// Robert

Re: [PATCH 4/6] e2fsprogs: properly set up extent header in do_write

[Robert Yang]

On 08/23/2013 01:24 AM, Darren Hart wrote:
> On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
>> do_write doesn't fully set up the first extent header on a new
>> inode, so if we write a 0-length file, and don't write any data
>> to the new file, we end up creating something that looks corrupt
>> to kernelspace:
>>
>> EXT4-fs error (device loop0): ext4_ext_check_inode:464: inode #12: comm
>> ls: bad header/extent: invalid magic - magic 0, entries 0, max 0(0),
>> depth 0(0)
>>
>> Do something similar to ext4_ext_tree_init() here, and
>> fill out the first extent header upon creation to avoid this.
>>
>> [YOCTO #3848]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   .../e2fsprogs-1.42.8/debugfs-extent-header.patch   |   47 ++++++++++++++++++++
>>   .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    2 +
>
>> +Upstream-Status: Backport
>
> Should we backport? Or should we just update the revision we use?
>

Yes, I think so, Ted said that he had merge this patch a few days ago, but
I didn't see where is it, I pulled this patch from the linux ext mailing
list.

// Robert

Re: [PATCH 5/6] e2fsprogs: add populate-extfs.sh

[Robert Yang]

On 08/23/2013 01:29 AM, Darren Hart wrote:
> On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
>> This script is originally from Darren Hart, it will be used for creating
>> the ext* filesystem from a given directory, which will replace the
>> genext2fs in image_types.bbclass at the moment, we may use the mke2fs to
>> replace this script again when it has the initial directory support.
>>
>> Changes of the script:
>> * Rename it from mkdebugfs.sh to populate-extfs.sh
>> * Add a simple usage
>> * Add checking for the number of the parameters
>> * Add the "regular empty file" and "fifo" file type
>> * Set mode, uid and gid for the file
>> * Save the command lines to a file and batch run them
>> * Change the error message
>> * Improve the performance
>> * Add the support for hardlink
>>
>> [YOCTO #3848]
>
>> Signed-off-by: Darren Hart <dvhart@linux.intel.com>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>
>
> Ted has offered to add this to the contrib dir for the ext2 tools. This
> is fine to add to yocto, but we should contribute this back upstream as
> well.
>

Thanks, I will send it to linux ext4 list.

// Robert

>

Re: [PATCH 2/6] e2fsprogs: let debugfs do sparse copy

[Robert Yang]

On 08/22/2013 10:52 PM, Darren Hart wrote:
> On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
>> Let debugfs do sparse copy when src is a sparse file, just like
>> "cp --sparse=auto"
>>
>> This patch has been reviewed by the linux-ext4 mailing list, but isn't
>> merged atm.
>>
>> [YOCTO #3848]
>
> Minor whitespace nit below, but will likely be caught by upstream.
>

Thanks, I upgraded the pull request repo:

git://git.pokylinux.org/poky-contrib robert/extX

// Robert

> --
> Darren
>
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   .../e2fsprogs-1.42.8/debugfs-sparse-copy.patch     |  147 ++++++++++++++++++++
>>   .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
>>   2 files changed, 148 insertions(+)
>>   create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>>
>> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>> new file mode 100644
>> index 0000000..c87bcbf
>> --- /dev/null
>> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>> @@ -0,0 +1,147 @@
>> +debugfs.c: do sparse copy when src is a sparse file
>> +
>> +Let debugfs do sparse copy when src is a sparse file, just like
>> +"cp --sparse=auto"
>> +
>> +* For the:
>> +  #define IO_BUFSIZE 64*1024
>> +  this is a suggested value from gnu coreutils:
>> +  http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/ioblksize.h;h=1ae93255e7d0ccf0855208c7ae5888209997bf16;hb=HEAD
>> +
>> +* Use malloc() to allocate memory for the buffer since put 64K (or
>> +  more) on the stack seems not a good idea.
>> +
>> +Upstream-Status: Submitted
>> +
>> +Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> +Acked-by: Darren Hart <dvhart@linux.intel.com>
>> +---
>> + debugfs/debugfs.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----
>> + 1 file changed, 57 insertions(+), 4 deletions(-)
>> +
>> +diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
>> +--- a/debugfs/debugfs.c
>> ++++ b/debugfs/debugfs.c
>> +@@ -41,6 +41,16 @@ extern char *optarg;
>> + #define BUFSIZ 8192
>> + #endif
>> +
>> ++/* 64KiB is the minimium blksize to best minimize system call overhead. */
>> ++#ifndef IO_BUFSIZE
>> ++#define IO_BUFSIZE 64*1024
>> ++#endif
>> ++
>> ++/* Block size for `st_blocks' */
>> ++#ifndef S_BLKSIZE
>> ++#define S_BLKSIZE 512
>> ++#endif
>> ++
>> + ss_request_table *extra_cmds;
>> + const char *debug_prog_name;
>> + int sci_idx;
>> +@@ -1575,22 +1585,36 @@ void do_find_free_inode(int argc, char *argv[])
>> + }
>> +
>> + #ifndef READ_ONLY
>> +-static errcode_t copy_file(int fd, ext2_ino_t newfile)
>> ++static errcode_t copy_file(int fd, ext2_ino_t newfile, int bufsize, int make_holes)
>> + {
>> + 	ext2_file_t	e2_file;
>> + 	errcode_t	retval;
>> + 	int		got;
>> + 	unsigned int	written;
>> +-	char		buf[8192];
>> ++	char		*buf;
>> + 	char		*ptr;
>> ++	char		*zero_buf;
>> ++	int		cmp;
>> +
>> + 	retval = ext2fs_file_open(current_fs, newfile,
>> + 				  EXT2_FILE_WRITE, &e2_file);
>> + 	if (retval)
>> + 		return retval;
>> +
>> ++	if (!(buf = (char *) malloc(bufsize))){
>> ++		com_err("copy_file", errno, "can't allocate buffer\n");
>> ++		return;
>> ++	}
>> ++
>> ++        /* This is used for checking whether the whole block is zero */
>
> Whitespace/indentation error ^
>
>
>> ++	retval = ext2fs_get_memzero(bufsize, &zero_buf);
>> ++	if (retval) {
>> ++		com_err("copy_file", retval, "can't allocate buffer\n");
>> ++		return retval;
>> ++	}
>> ++
>> + 	while (1) {
>> +-		got = read(fd, buf, sizeof(buf));
>> ++		got = read(fd, buf, bufsize);
>> + 		if (got == 0)
>> + 			break;
>> + 		if (got < 0) {
>> +@@ -1598,6 +1622,21 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
>> + 			goto fail;
>> + 		}
>> + 		ptr = buf;
>> ++
>> ++		/* Sparse copy */
>> ++		if (make_holes) {
>> ++			/* Check whether all is zero */
>> ++			cmp = memcmp(ptr, zero_buf, got);
>> ++			if (cmp == 0) {
>> ++				 /* The whole block is zero, make a hole */
>> ++				retval = ext2fs_file_lseek(e2_file, got, EXT2_SEEK_CUR, NULL);
>> ++				if (retval)
>> ++					goto fail;
>> ++				got = 0;
>> ++			}
>> ++		}
>> ++
>> ++		/* Normal copy */
>> + 		while (got > 0) {
>> + 			retval = ext2fs_file_write(e2_file, ptr,
>> + 						   got, &written);
>> +@@ -1608,10 +1647,14 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
>> + 			ptr += written;
>> + 		}
>> + 	}
>> ++	free(buf);
>> ++	ext2fs_free_mem(&zero_buf);
>> + 	retval = ext2fs_file_close(e2_file);
>> + 	return retval;
>> +
>> + fail:
>> ++	free(buf);
>> ++	ext2fs_free_mem(&zero_buf);
>> + 	(void) ext2fs_file_close(e2_file);
>> + 	return retval;
>> + }
>> +@@ -1624,6 +1667,8 @@ void do_write(int argc, char *argv[])
>> + 	ext2_ino_t	newfile;
>> + 	errcode_t	retval;
>> + 	struct ext2_inode inode;
>> ++	int		bufsize = IO_BUFSIZE;
>> ++	int		make_holes = 0;
>> +
>> + 	if (common_args_process(argc, argv, 3, 3, "write",
>> + 				"<native file> <new file>", CHECK_FS_RW))
>> +@@ -1699,7 +1744,15 @@ void do_write(int argc, char *argv[])
>> + 		return;
>> + 	}
>> + 	if (LINUX_S_ISREG(inode.i_mode)) {
>> +-		retval = copy_file(fd, newfile);
>> ++		if (statbuf.st_blocks < statbuf.st_size / S_BLKSIZE) {
>> ++			make_holes = 1;
>> ++			/*
>> ++			 * Use I/O blocksize as buffer size when
>> ++			 * copying sparse files.
>> ++			 */
>> ++			bufsize = statbuf.st_blksize;
>> ++		}
>> ++		retval = copy_file(fd, newfile, bufsize, make_holes);
>> + 		if (retval)
>> + 			com_err("copy_file", retval, 0);
>> + 	}
>> +--
>> +1.8.1.2
>> +
>> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>> index f97de7f..5d65bbc 100644
>> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>> @@ -4,6 +4,7 @@ require e2fsprogs.inc
>>   SRC_URI += "file://acinclude.m4 \
>>               file://remove.ldconfig.call.patch \
>>               file://debugfs-too-short.patch \
>> +            file://debugfs-sparse-copy.patch \
>>   "
>>
>>   SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
>

Re: [PATCH 2/6] e2fsprogs: let debugfs do sparse copy

[Rongqing Li]

On 08/22/2013 09:13 PM, Robert Yang wrote:
> Let debugfs do sparse copy when src is a sparse file, just like
> "cp --sparse=auto"
>
> This patch has been reviewed by the linux-ext4 mailing list, but isn't
> merged atm.
>
> [YOCTO #3848]
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>   .../e2fsprogs-1.42.8/debugfs-sparse-copy.patch     |  147 ++++++++++++++++++++
>   .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
>   2 files changed, 148 insertions(+)
>   create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>
> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> new file mode 100644
> index 0000000..c87bcbf
> --- /dev/null
> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> @@ -0,0 +1,147 @@
> +debugfs.c: do sparse copy when src is a sparse file
> +
> +Let debugfs do sparse copy when src is a sparse file, just like
> +"cp --sparse=auto"
> +
> +* For the:
> +  #define IO_BUFSIZE 64*1024
> +  this is a suggested value from gnu coreutils:
> +  http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/ioblksize.h;h=1ae93255e7d0ccf0855208c7ae5888209997bf16;hb=HEAD
> +
> +* Use malloc() to allocate memory for the buffer since put 64K (or
> +  more) on the stack seems not a good idea.
> +
> +Upstream-Status: Submitted
> +
> +Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> +Acked-by: Darren Hart <dvhart@linux.intel.com>
> +---
> + debugfs/debugfs.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----
> + 1 file changed, 57 insertions(+), 4 deletions(-)
> +
> +diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
> +--- a/debugfs/debugfs.c
> ++++ b/debugfs/debugfs.c
> +@@ -41,6 +41,16 @@ extern char *optarg;
> + #define BUFSIZ 8192
> + #endif
> +
> ++/* 64KiB is the minimium blksize to best minimize system call overhead. */
> ++#ifndef IO_BUFSIZE
> ++#define IO_BUFSIZE 64*1024
> ++#endif
> ++
> ++/* Block size for `st_blocks' */
> ++#ifndef S_BLKSIZE
> ++#define S_BLKSIZE 512
> ++#endif
> ++
> + ss_request_table *extra_cmds;
> + const char *debug_prog_name;
> + int sci_idx;
> +@@ -1575,22 +1585,36 @@ void do_find_free_inode(int argc, char *argv[])
> + }
> +
> + #ifndef READ_ONLY
> +-static errcode_t copy_file(int fd, ext2_ino_t newfile)
> ++static errcode_t copy_file(int fd, ext2_ino_t newfile, int bufsize, int make_holes)
> + {
> + 	ext2_file_t	e2_file;
> + 	errcode_t	retval;
> + 	int		got;
> + 	unsigned int	written;
> +-	char		buf[8192];
> ++	char		*buf;
> + 	char		*ptr;
> ++	char		*zero_buf;
> ++	int		cmp;
> +
> + 	retval = ext2fs_file_open(current_fs, newfile,
> + 				  EXT2_FILE_WRITE, &e2_file);
> + 	if (retval)
> + 		return retval;
> +
> ++	if (!(buf = (char *) malloc(bufsize))){
> ++		com_err("copy_file", errno, "can't allocate buffer\n");
> ++		return;
> ++	}
> ++
> ++        /* This is used for checking whether the whole block is zero */
> ++	retval = ext2fs_get_memzero(bufsize, &zero_buf);
> ++	if (retval) {
> ++		com_err("copy_file", retval, "can't allocate buffer\n");

should free(buf), or memory leak.

-Roy

> ++		return retval;
> ++	}
> ++
> + 	while (1) {
> +-		got = read(fd, buf, sizeof(buf));
> ++		got = read(fd, buf, bufsize);
> + 		if (got == 0)
> + 			break;
> + 		if (got < 0) {
> +@@ -1598,6 +1622,21 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
> + 			goto fail;
> + 		}
> + 		ptr = buf;
> ++
> ++		/* Sparse copy */
> ++		if (make_holes) {
> ++			/* Check whether all is zero */
> ++			cmp = memcmp(ptr, zero_buf, got);
> ++			if (cmp == 0) {
> ++				 /* The whole block is zero, make a hole */
> ++				retval = ext2fs_file_lseek(e2_file, got, EXT2_SEEK_CUR, NULL);
> ++				if (retval)
> ++					goto fail;
> ++				got = 0;
> ++			}
> ++		}
> ++
> ++		/* Normal copy */
> + 		while (got > 0) {
> + 			retval = ext2fs_file_write(e2_file, ptr,
> + 						   got, &written);
> +@@ -1608,10 +1647,14 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
> + 			ptr += written;
> + 		}
> + 	}
> ++	free(buf);
> ++	ext2fs_free_mem(&zero_buf);
> + 	retval = ext2fs_file_close(e2_file);
> + 	return retval;
> +
> + fail:
> ++	free(buf);
> ++	ext2fs_free_mem(&zero_buf);
> + 	(void) ext2fs_file_close(e2_file);
> + 	return retval;
> + }
> +@@ -1624,6 +1667,8 @@ void do_write(int argc, char *argv[])
> + 	ext2_ino_t	newfile;
> + 	errcode_t	retval;
> + 	struct ext2_inode inode;
> ++	int		bufsize = IO_BUFSIZE;
> ++	int		make_holes = 0;
> +
> + 	if (common_args_process(argc, argv, 3, 3, "write",
> + 				"<native file> <new file>", CHECK_FS_RW))
> +@@ -1699,7 +1744,15 @@ void do_write(int argc, char *argv[])
> + 		return;
> + 	}
> + 	if (LINUX_S_ISREG(inode.i_mode)) {
> +-		retval = copy_file(fd, newfile);
> ++		if (statbuf.st_blocks < statbuf.st_size / S_BLKSIZE) {
> ++			make_holes = 1;
> ++			/*
> ++			 * Use I/O blocksize as buffer size when
> ++			 * copying sparse files.
> ++			 */
> ++			bufsize = statbuf.st_blksize;
> ++		}
> ++		retval = copy_file(fd, newfile, bufsize, make_holes);
> + 		if (retval)
> + 			com_err("copy_file", retval, 0);
> + 	}
> +--
> +1.8.1.2
> +
> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> index f97de7f..5d65bbc 100644
> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> @@ -4,6 +4,7 @@ require e2fsprogs.inc
>   SRC_URI += "file://acinclude.m4 \
>               file://remove.ldconfig.call.patch \
>               file://debugfs-too-short.patch \
> +            file://debugfs-sparse-copy.patch \
>   "
>
>   SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
>

-- 
Best Reagrds,
Roy | RongQing Li

Re: [PATCH 2/6] e2fsprogs: let debugfs do sparse copy

[Robert Yang]

On 08/23/2013 10:33 AM, Rongqing Li wrote:
>
>
> On 08/22/2013 09:13 PM, Robert Yang wrote:
>> Let debugfs do sparse copy when src is a sparse file, just like
>> "cp --sparse=auto"
>>
>> This patch has been reviewed by the linux-ext4 mailing list, but isn't
>> merged atm.
>>
>> [YOCTO #3848]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   .../e2fsprogs-1.42.8/debugfs-sparse-copy.patch     |  147 ++++++++++++++++++++
>>   .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
>>   2 files changed, 148 insertions(+)
>>   create mode 100644
>> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>>
>> diff --git
>> a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>> b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>> new file mode 100644
>> index 0000000..c87bcbf
>> --- /dev/null
>> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>> @@ -0,0 +1,147 @@
>> +debugfs.c: do sparse copy when src is a sparse file
>> +
>> +Let debugfs do sparse copy when src is a sparse file, just like
>> +"cp --sparse=auto"
>> +
>> +* For the:
>> +  #define IO_BUFSIZE 64*1024
>> +  this is a suggested value from gnu coreutils:
>> +
>> http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/ioblksize.h;h=1ae93255e7d0ccf0855208c7ae5888209997bf16;hb=HEAD
>>
>> +
>> +* Use malloc() to allocate memory for the buffer since put 64K (or
>> +  more) on the stack seems not a good idea.
>> +
>> +Upstream-Status: Submitted
>> +
>> +Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> +Acked-by: Darren Hart <dvhart@linux.intel.com>
>> +---
>> + debugfs/debugfs.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----
>> + 1 file changed, 57 insertions(+), 4 deletions(-)
>> +
>> +diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
>> +--- a/debugfs/debugfs.c
>> ++++ b/debugfs/debugfs.c
>> +@@ -41,6 +41,16 @@ extern char *optarg;
>> + #define BUFSIZ 8192
>> + #endif
>> +
>> ++/* 64KiB is the minimium blksize to best minimize system call overhead. */
>> ++#ifndef IO_BUFSIZE
>> ++#define IO_BUFSIZE 64*1024
>> ++#endif
>> ++
>> ++/* Block size for `st_blocks' */
>> ++#ifndef S_BLKSIZE
>> ++#define S_BLKSIZE 512
>> ++#endif
>> ++
>> + ss_request_table *extra_cmds;
>> + const char *debug_prog_name;
>> + int sci_idx;
>> +@@ -1575,22 +1585,36 @@ void do_find_free_inode(int argc, char *argv[])
>> + }
>> +
>> + #ifndef READ_ONLY
>> +-static errcode_t copy_file(int fd, ext2_ino_t newfile)
>> ++static errcode_t copy_file(int fd, ext2_ino_t newfile, int bufsize, int
>> make_holes)
>> + {
>> +     ext2_file_t    e2_file;
>> +     errcode_t    retval;
>> +     int        got;
>> +     unsigned int    written;
>> +-    char        buf[8192];
>> ++    char        *buf;
>> +     char        *ptr;
>> ++    char        *zero_buf;
>> ++    int        cmp;
>> +
>> +     retval = ext2fs_file_open(current_fs, newfile,
>> +                   EXT2_FILE_WRITE, &e2_file);
>> +     if (retval)
>> +         return retval;
>> +
>> ++    if (!(buf = (char *) malloc(bufsize))){
>> ++        com_err("copy_file", errno, "can't allocate buffer\n");
>> ++        return;
>> ++    }
>> ++
>> ++        /* This is used for checking whether the whole block is zero */
>> ++    retval = ext2fs_get_memzero(bufsize, &zero_buf);
>> ++    if (retval) {
>> ++        com_err("copy_file", retval, "can't allocate buffer\n");
>
> should free(buf), or memory leak.
>

Add a free(buf) and updated the PR:

git://git.pokylinux.org/poky-contrib robert/extX

// Robert

> -Roy
>
>> ++        return retval;
>> ++    }
>> ++
>> +     while (1) {
>> +-        got = read(fd, buf, sizeof(buf));
>> ++        got = read(fd, buf, bufsize);
>> +         if (got == 0)
>> +             break;
>> +         if (got < 0) {
>> +@@ -1598,6 +1622,21 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
>> +             goto fail;
>> +         }
>> +         ptr = buf;
>> ++
>> ++        /* Sparse copy */
>> ++        if (make_holes) {
>> ++            /* Check whether all is zero */
>> ++            cmp = memcmp(ptr, zero_buf, got);
>> ++            if (cmp == 0) {
>> ++                 /* The whole block is zero, make a hole */
>> ++                retval = ext2fs_file_lseek(e2_file, got, EXT2_SEEK_CUR, NULL);
>> ++                if (retval)
>> ++                    goto fail;
>> ++                got = 0;
>> ++            }
>> ++        }
>> ++
>> ++        /* Normal copy */
>> +         while (got > 0) {
>> +             retval = ext2fs_file_write(e2_file, ptr,
>> +                            got, &written);
>> +@@ -1608,10 +1647,14 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
>> +             ptr += written;
>> +         }
>> +     }
>> ++    free(buf);
>> ++    ext2fs_free_mem(&zero_buf);
>> +     retval = ext2fs_file_close(e2_file);
>> +     return retval;
>> +
>> + fail:
>> ++    free(buf);
>> ++    ext2fs_free_mem(&zero_buf);
>> +     (void) ext2fs_file_close(e2_file);
>> +     return retval;
>> + }
>> +@@ -1624,6 +1667,8 @@ void do_write(int argc, char *argv[])
>> +     ext2_ino_t    newfile;
>> +     errcode_t    retval;
>> +     struct ext2_inode inode;
>> ++    int        bufsize = IO_BUFSIZE;
>> ++    int        make_holes = 0;
>> +
>> +     if (common_args_process(argc, argv, 3, 3, "write",
>> +                 "<native file> <new file>", CHECK_FS_RW))
>> +@@ -1699,7 +1744,15 @@ void do_write(int argc, char *argv[])
>> +         return;
>> +     }
>> +     if (LINUX_S_ISREG(inode.i_mode)) {
>> +-        retval = copy_file(fd, newfile);
>> ++        if (statbuf.st_blocks < statbuf.st_size / S_BLKSIZE) {
>> ++            make_holes = 1;
>> ++            /*
>> ++             * Use I/O blocksize as buffer size when
>> ++             * copying sparse files.
>> ++             */
>> ++            bufsize = statbuf.st_blksize;
>> ++        }
>> ++        retval = copy_file(fd, newfile, bufsize, make_holes);
>> +         if (retval)
>> +             com_err("copy_file", retval, 0);
>> +     }
>> +--
>> +1.8.1.2
>> +
>> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>> b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>> index f97de7f..5d65bbc 100644
>> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>> @@ -4,6 +4,7 @@ require e2fsprogs.inc
>>   SRC_URI += "file://acinclude.m4 \
>>               file://remove.ldconfig.call.patch \
>>               file://debugfs-too-short.patch \
>> +            file://debugfs-sparse-copy.patch \
>>   "
>>
>>   SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
>>
>

Re: [PATCH 3/6] e2fsprogs: only update the icache for ext2_inode

[Darren Hart]

On Fri, 2013-08-23 at 09:58 +0800, Robert Yang wrote:
> On 08/23/2013 01:23 AM, Darren Hart wrote:
> > On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
> >> We only read the cache when:
> >>
> >> bufsize == sizeof(struct ext2_inode)
> >>
> >> then we should only update the cache in the same condition, otherwise
> >> there would be errors, for example:
> >>
> >> cache[0]: cached ino 14 when bufsize = 128 by ext2fs_write_inode_full()
> >> cache[1]: cached ino 14 when bufsize = 156 by ext2fs_read_inode_full()
> >>
> >> Then update the cache:
> >> cache[0]: cached ino 15 when bufsize = 156 by ext2fs_read_inode_full()
> >>
> >> Then the ino 14 would hit the cache[1] when bufsize = 128 (but it was
> >> cached by bufsize = 156), so there would be errors.
> >
> >> [YOCTO #3848]
> >>
> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> >
> >
> >> +Upstream-Status: [Inappropriate]
> >
> > Hrm... why is this one inappropriate?
> >
> 
> Hi Darren, the upstream has changed the icache lot, so this patch is
> inappropriate for the upstream, we can drop this patch when we update
> the package, I had put this in the patch head.

Sorry, I must have missed that.

Acked-by: Darren Hart <dvhart@linux.intel.com>

Thanks Robert!

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 4/6] e2fsprogs: properly set up extent header in do_write

[Darren Hart]

On Fri, 2013-08-23 at 10:02 +0800, Robert Yang wrote:
> On 08/23/2013 01:24 AM, Darren Hart wrote:
> > On Thu, 2013-08-22 at 09:13 -0400, Robert Yang wrote:
> >> do_write doesn't fully set up the first extent header on a new
> >> inode, so if we write a 0-length file, and don't write any data
> >> to the new file, we end up creating something that looks corrupt
> >> to kernelspace:
> >>
> >> EXT4-fs error (device loop0): ext4_ext_check_inode:464: inode #12: comm
> >> ls: bad header/extent: invalid magic - magic 0, entries 0, max 0(0),
> >> depth 0(0)
> >>
> >> Do something similar to ext4_ext_tree_init() here, and
> >> fill out the first extent header upon creation to avoid this.
> >>
> >> [YOCTO #3848]
> >>
> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> >> ---
> >>   .../e2fsprogs-1.42.8/debugfs-extent-header.patch   |   47 ++++++++++++++++++++
> >>   .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    2 +
> >
> >> +Upstream-Status: Backport
> >
> > Should we backport? Or should we just update the revision we use?
> >
> 
> Yes, I think so, Ted said that he had merge this patch a few days ago, but
> I didn't see where is it, I pulled this patch from the linux ext mailing
> list.

OK, yeah, this is fine. It will be nice to rebase at some point in the
future and drop these patches, but we have to do that anyway, so this is
fine in my opinion.

Acked-by: Darren Hart <dvhart@linux.intel.com>

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 2/6] e2fsprogs: let debugfs do sparse copy

[Darren Hart]

On Fri, 2013-08-23 at 14:45 +0800, Robert Yang wrote:
> 
> On 08/23/2013 10:33 AM, Rongqing Li wrote:
> >
> >
> > On 08/22/2013 09:13 PM, Robert Yang wrote:
> >> Let debugfs do sparse copy when src is a sparse file, just like
> >> "cp --sparse=auto"
> >>
> >> This patch has been reviewed by the linux-ext4 mailing list, but isn't
> >> merged atm.
> >>
> >> [YOCTO #3848]
> >>
> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> >> ---
> >>   .../e2fsprogs-1.42.8/debugfs-sparse-copy.patch     |  147 ++++++++++++++++++++
> >>   .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
> >>   2 files changed, 148 insertions(+)
> >>   create mode 100644
> >> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> >>
> >> diff --git
> >> a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> >> b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> >> new file mode 100644
> >> index 0000000..c87bcbf
> >> --- /dev/null
> >> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
> >> @@ -0,0 +1,147 @@
> >> +debugfs.c: do sparse copy when src is a sparse file
> >> +
> >> +Let debugfs do sparse copy when src is a sparse file, just like
> >> +"cp --sparse=auto"
> >> +
> >> +* For the:
> >> +  #define IO_BUFSIZE 64*1024
> >> +  this is a suggested value from gnu coreutils:
> >> +
> >> http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/ioblksize.h;h=1ae93255e7d0ccf0855208c7ae5888209997bf16;hb=HEAD
> >>
> >> +
> >> +* Use malloc() to allocate memory for the buffer since put 64K (or
> >> +  more) on the stack seems not a good idea.
> >> +
> >> +Upstream-Status: Submitted
> >> +
> >> +Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> >> +Acked-by: Darren Hart <dvhart@linux.intel.com>
> >> +---
> >> + debugfs/debugfs.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----
> >> + 1 file changed, 57 insertions(+), 4 deletions(-)
> >> +
> >> +diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
> >> +--- a/debugfs/debugfs.c
> >> ++++ b/debugfs/debugfs.c
> >> +@@ -41,6 +41,16 @@ extern char *optarg;
> >> + #define BUFSIZ 8192
> >> + #endif
> >> +
> >> ++/* 64KiB is the minimium blksize to best minimize system call overhead. */
> >> ++#ifndef IO_BUFSIZE
> >> ++#define IO_BUFSIZE 64*1024
> >> ++#endif
> >> ++
> >> ++/* Block size for `st_blocks' */
> >> ++#ifndef S_BLKSIZE
> >> ++#define S_BLKSIZE 512
> >> ++#endif
> >> ++
> >> + ss_request_table *extra_cmds;
> >> + const char *debug_prog_name;
> >> + int sci_idx;
> >> +@@ -1575,22 +1585,36 @@ void do_find_free_inode(int argc, char *argv[])
> >> + }
> >> +
> >> + #ifndef READ_ONLY
> >> +-static errcode_t copy_file(int fd, ext2_ino_t newfile)
> >> ++static errcode_t copy_file(int fd, ext2_ino_t newfile, int bufsize, int
> >> make_holes)
> >> + {
> >> +     ext2_file_t    e2_file;
> >> +     errcode_t    retval;
> >> +     int        got;
> >> +     unsigned int    written;
> >> +-    char        buf[8192];
> >> ++    char        *buf;
> >> +     char        *ptr;
> >> ++    char        *zero_buf;
> >> ++    int        cmp;
> >> +
> >> +     retval = ext2fs_file_open(current_fs, newfile,
> >> +                   EXT2_FILE_WRITE, &e2_file);
> >> +     if (retval)
> >> +         return retval;
> >> +
> >> ++    if (!(buf = (char *) malloc(bufsize))){
> >> ++        com_err("copy_file", errno, "can't allocate buffer\n");
> >> ++        return;
> >> ++    }
> >> ++
> >> ++        /* This is used for checking whether the whole block is zero */
> >> ++    retval = ext2fs_get_memzero(bufsize, &zero_buf);
> >> ++    if (retval) {
> >> ++        com_err("copy_file", retval, "can't allocate buffer\n");
> >
> > should free(buf), or memory leak.
> >
> 
> Add a free(buf) and updated the PR:
> 
> git://git.pokylinux.org/poky-contrib robert/extX


Nice catch Rongqing, thank you.

Acked-by: Darren Hart <dvhart@linux.intel.com>

Robert, please resubmit a new version to the ext list with the memory
leak and whitespace fixes. It's a good way to the attention back on the
series.

--
Darren

> 
> // Robert
> 
> > -Roy
> >
> >> ++        return retval;
> >> ++    }
> >> ++
> >> +     while (1) {
> >> +-        got = read(fd, buf, sizeof(buf));
> >> ++        got = read(fd, buf, bufsize);
> >> +         if (got == 0)
> >> +             break;
> >> +         if (got < 0) {
> >> +@@ -1598,6 +1622,21 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
> >> +             goto fail;
> >> +         }
> >> +         ptr = buf;
> >> ++
> >> ++        /* Sparse copy */
> >> ++        if (make_holes) {
> >> ++            /* Check whether all is zero */
> >> ++            cmp = memcmp(ptr, zero_buf, got);
> >> ++            if (cmp == 0) {
> >> ++                 /* The whole block is zero, make a hole */
> >> ++                retval = ext2fs_file_lseek(e2_file, got, EXT2_SEEK_CUR, NULL);
> >> ++                if (retval)
> >> ++                    goto fail;
> >> ++                got = 0;
> >> ++            }
> >> ++        }
> >> ++
> >> ++        /* Normal copy */
> >> +         while (got > 0) {
> >> +             retval = ext2fs_file_write(e2_file, ptr,
> >> +                            got, &written);
> >> +@@ -1608,10 +1647,14 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
> >> +             ptr += written;
> >> +         }
> >> +     }
> >> ++    free(buf);
> >> ++    ext2fs_free_mem(&zero_buf);
> >> +     retval = ext2fs_file_close(e2_file);
> >> +     return retval;
> >> +
> >> + fail:
> >> ++    free(buf);
> >> ++    ext2fs_free_mem(&zero_buf);
> >> +     (void) ext2fs_file_close(e2_file);
> >> +     return retval;
> >> + }
> >> +@@ -1624,6 +1667,8 @@ void do_write(int argc, char *argv[])
> >> +     ext2_ino_t    newfile;
> >> +     errcode_t    retval;
> >> +     struct ext2_inode inode;
> >> ++    int        bufsize = IO_BUFSIZE;
> >> ++    int        make_holes = 0;
> >> +
> >> +     if (common_args_process(argc, argv, 3, 3, "write",
> >> +                 "<native file> <new file>", CHECK_FS_RW))
> >> +@@ -1699,7 +1744,15 @@ void do_write(int argc, char *argv[])
> >> +         return;
> >> +     }
> >> +     if (LINUX_S_ISREG(inode.i_mode)) {
> >> +-        retval = copy_file(fd, newfile);
> >> ++        if (statbuf.st_blocks < statbuf.st_size / S_BLKSIZE) {
> >> ++            make_holes = 1;
> >> ++            /*
> >> ++             * Use I/O blocksize as buffer size when
> >> ++             * copying sparse files.
> >> ++             */
> >> ++            bufsize = statbuf.st_blksize;
> >> ++        }
> >> ++        retval = copy_file(fd, newfile, bufsize, make_holes);
> >> +         if (retval)
> >> +             com_err("copy_file", retval, 0);
> >> +     }
> >> +--
> >> +1.8.1.2
> >> +
> >> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> >> b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> >> index f97de7f..5d65bbc 100644
> >> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> >> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
> >> @@ -4,6 +4,7 @@ require e2fsprogs.inc
> >>   SRC_URI += "file://acinclude.m4 \
> >>               file://remove.ldconfig.call.patch \
> >>               file://debugfs-too-short.patch \
> >> +            file://debugfs-sparse-copy.patch \
> >>   "
> >>
> >>   SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
> >>
> >

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 2/6] e2fsprogs: let debugfs do sparse copy

[Robert Yang]

On 08/24/2013 01:06 AM, Darren Hart wrote:
> On Fri, 2013-08-23 at 14:45 +0800, Robert Yang wrote:
>>
>> On 08/23/2013 10:33 AM, Rongqing Li wrote:
>>>
>>>
>>> On 08/22/2013 09:13 PM, Robert Yang wrote:
>>>> Let debugfs do sparse copy when src is a sparse file, just like
>>>> "cp --sparse=auto"
>>>>
>>>> This patch has been reviewed by the linux-ext4 mailing list, but isn't
>>>> merged atm.
>>>>
>>>> [YOCTO #3848]
>>>>
>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>> ---
>>>>    .../e2fsprogs-1.42.8/debugfs-sparse-copy.patch     |  147 ++++++++++++++++++++
>>>>    .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |    1 +
>>>>    2 files changed, 148 insertions(+)
>>>>    create mode 100644
>>>> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>>>>
>>>> diff --git
>>>> a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>>>> b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>>>> new file mode 100644
>>>> index 0000000..c87bcbf
>>>> --- /dev/null
>>>> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-sparse-copy.patch
>>>> @@ -0,0 +1,147 @@
>>>> +debugfs.c: do sparse copy when src is a sparse file
>>>> +
>>>> +Let debugfs do sparse copy when src is a sparse file, just like
>>>> +"cp --sparse=auto"
>>>> +
>>>> +* For the:
>>>> +  #define IO_BUFSIZE 64*1024
>>>> +  this is a suggested value from gnu coreutils:
>>>> +
>>>> http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/ioblksize.h;h=1ae93255e7d0ccf0855208c7ae5888209997bf16;hb=HEAD
>>>>
>>>> +
>>>> +* Use malloc() to allocate memory for the buffer since put 64K (or
>>>> +  more) on the stack seems not a good idea.
>>>> +
>>>> +Upstream-Status: Submitted
>>>> +
>>>> +Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>> +Acked-by: Darren Hart <dvhart@linux.intel.com>
>>>> +---
>>>> + debugfs/debugfs.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----
>>>> + 1 file changed, 57 insertions(+), 4 deletions(-)
>>>> +
>>>> +diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
>>>> +--- a/debugfs/debugfs.c
>>>> ++++ b/debugfs/debugfs.c
>>>> +@@ -41,6 +41,16 @@ extern char *optarg;
>>>> + #define BUFSIZ 8192
>>>> + #endif
>>>> +
>>>> ++/* 64KiB is the minimium blksize to best minimize system call overhead. */
>>>> ++#ifndef IO_BUFSIZE
>>>> ++#define IO_BUFSIZE 64*1024
>>>> ++#endif
>>>> ++
>>>> ++/* Block size for `st_blocks' */
>>>> ++#ifndef S_BLKSIZE
>>>> ++#define S_BLKSIZE 512
>>>> ++#endif
>>>> ++
>>>> + ss_request_table *extra_cmds;
>>>> + const char *debug_prog_name;
>>>> + int sci_idx;
>>>> +@@ -1575,22 +1585,36 @@ void do_find_free_inode(int argc, char *argv[])
>>>> + }
>>>> +
>>>> + #ifndef READ_ONLY
>>>> +-static errcode_t copy_file(int fd, ext2_ino_t newfile)
>>>> ++static errcode_t copy_file(int fd, ext2_ino_t newfile, int bufsize, int
>>>> make_holes)
>>>> + {
>>>> +     ext2_file_t    e2_file;
>>>> +     errcode_t    retval;
>>>> +     int        got;
>>>> +     unsigned int    written;
>>>> +-    char        buf[8192];
>>>> ++    char        *buf;
>>>> +     char        *ptr;
>>>> ++    char        *zero_buf;
>>>> ++    int        cmp;
>>>> +
>>>> +     retval = ext2fs_file_open(current_fs, newfile,
>>>> +                   EXT2_FILE_WRITE, &e2_file);
>>>> +     if (retval)
>>>> +         return retval;
>>>> +
>>>> ++    if (!(buf = (char *) malloc(bufsize))){
>>>> ++        com_err("copy_file", errno, "can't allocate buffer\n");
>>>> ++        return;
>>>> ++    }
>>>> ++
>>>> ++        /* This is used for checking whether the whole block is zero */
>>>> ++    retval = ext2fs_get_memzero(bufsize, &zero_buf);
>>>> ++    if (retval) {
>>>> ++        com_err("copy_file", retval, "can't allocate buffer\n");
>>>
>>> should free(buf), or memory leak.
>>>
>>
>> Add a free(buf) and updated the PR:
>>
>> git://git.pokylinux.org/poky-contrib robert/extX
>
>
> Nice catch Rongqing, thank you.
>
> Acked-by: Darren Hart <dvhart@linux.intel.com>
>
> Robert, please resubmit a new version to the ext list with the memory
> leak and whitespace fixes. It's a good way to the attention back on the
> series.
>

OK, thanks, I will send it today.

// Robert

> --
> Darren
>
>>
>> // Robert
>>
>>> -Roy
>>>
>>>> ++        return retval;
>>>> ++    }
>>>> ++
>>>> +     while (1) {
>>>> +-        got = read(fd, buf, sizeof(buf));
>>>> ++        got = read(fd, buf, bufsize);
>>>> +         if (got == 0)
>>>> +             break;
>>>> +         if (got < 0) {
>>>> +@@ -1598,6 +1622,21 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
>>>> +             goto fail;
>>>> +         }
>>>> +         ptr = buf;
>>>> ++
>>>> ++        /* Sparse copy */
>>>> ++        if (make_holes) {
>>>> ++            /* Check whether all is zero */
>>>> ++            cmp = memcmp(ptr, zero_buf, got);
>>>> ++            if (cmp == 0) {
>>>> ++                 /* The whole block is zero, make a hole */
>>>> ++                retval = ext2fs_file_lseek(e2_file, got, EXT2_SEEK_CUR, NULL);
>>>> ++                if (retval)
>>>> ++                    goto fail;
>>>> ++                got = 0;
>>>> ++            }
>>>> ++        }
>>>> ++
>>>> ++        /* Normal copy */
>>>> +         while (got > 0) {
>>>> +             retval = ext2fs_file_write(e2_file, ptr,
>>>> +                            got, &written);
>>>> +@@ -1608,10 +1647,14 @@ static errcode_t copy_file(int fd, ext2_ino_t newfile)
>>>> +             ptr += written;
>>>> +         }
>>>> +     }
>>>> ++    free(buf);
>>>> ++    ext2fs_free_mem(&zero_buf);
>>>> +     retval = ext2fs_file_close(e2_file);
>>>> +     return retval;
>>>> +
>>>> + fail:
>>>> ++    free(buf);
>>>> ++    ext2fs_free_mem(&zero_buf);
>>>> +     (void) ext2fs_file_close(e2_file);
>>>> +     return retval;
>>>> + }
>>>> +@@ -1624,6 +1667,8 @@ void do_write(int argc, char *argv[])
>>>> +     ext2_ino_t    newfile;
>>>> +     errcode_t    retval;
>>>> +     struct ext2_inode inode;
>>>> ++    int        bufsize = IO_BUFSIZE;
>>>> ++    int        make_holes = 0;
>>>> +
>>>> +     if (common_args_process(argc, argv, 3, 3, "write",
>>>> +                 "<native file> <new file>", CHECK_FS_RW))
>>>> +@@ -1699,7 +1744,15 @@ void do_write(int argc, char *argv[])
>>>> +         return;
>>>> +     }
>>>> +     if (LINUX_S_ISREG(inode.i_mode)) {
>>>> +-        retval = copy_file(fd, newfile);
>>>> ++        if (statbuf.st_blocks < statbuf.st_size / S_BLKSIZE) {
>>>> ++            make_holes = 1;
>>>> ++            /*
>>>> ++             * Use I/O blocksize as buffer size when
>>>> ++             * copying sparse files.
>>>> ++             */
>>>> ++            bufsize = statbuf.st_blksize;
>>>> ++        }
>>>> ++        retval = copy_file(fd, newfile, bufsize, make_holes);
>>>> +         if (retval)
>>>> +             com_err("copy_file", retval, 0);
>>>> +     }
>>>> +--
>>>> +1.8.1.2
>>>> +
>>>> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>>>> b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>>>> index f97de7f..5d65bbc 100644
>>>> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>>>> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb
>>>> @@ -4,6 +4,7 @@ require e2fsprogs.inc
>>>>    SRC_URI += "file://acinclude.m4 \
>>>>                file://remove.ldconfig.call.patch \
>>>>                file://debugfs-too-short.patch \
>>>> +            file://debugfs-sparse-copy.patch \
>>>>    "
>>>>
>>>>    SRC_URI[md5sum] = "8ef664b6eb698aa6b733df59b17b9ed4"
>>>>
>>>
>